2024-02-21T22:45:33.733 INFO:root:teuthology version: 0.0.1.dev260+gb1dac55 2024-02-21T22:45:33.733 DEBUG:teuthology.run:Teuthology command: teuthology --name leonidus-2024-02-21_22:31:03-fs-wip-lusov-qdb-distro-default-smithi --description fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} --owner scheduled_leonidus@teuthology --verbose --archive /home/teuthworker/archive/leonidus-2024-02-21_22:31:03-fs-wip-lusov-qdb-distro-default-smithi/7570010 -- /home/teuthworker/archive/leonidus-2024-02-21_22:31:03-fs-wip-lusov-qdb-distro-default-smithi/7570010/orig.config.yaml 2024-02-21T22:45:33.777 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-21T22:45:33.859 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/leonidus-2024-02-21_22:31:03-fs-wip-lusov-qdb-distro-default-smithi/7570010 branch: wip-lusov-qdb description: fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} email: null first_in_suite: false job_id: '7570010' kernel: client: branch: testing kdb: true sha1: distro ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: leonidus-2024-02-21_22:31:03-fs-wip-lusov-qdb-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-lusov-qdb ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - Replacing daemon mds - \(MDS_INSUFFICIENT_STANDBY\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(MDS_DAMAGE\) - \(FS_DEGRADED\) sha1: 3bfc837d7b45ce8e095d92fa9c342964cea5e95f ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: disabled: true install: ceph: flavor: default sha1: 3bfc837d7b45ce8e095d92fa9c342964cea5e95f kclient: syntax: v2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-lusov-qdb sha1: 3bfc837d7b45ce8e095d92fa9c342964cea5e95f owner: scheduled_leonidus@teuthology priority: 80 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 seed: 6562 sha1: 3bfc837d7b45ce8e095d92fa9c342964cea5e95f sleep_before_teardown: 0 subset: 111/10000 suite: fs suite_branch: wip-lusov-qdb suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_3bfc837d7b45ce8e095d92fa9c342964cea5e95f/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 3bfc837d7b45ce8e095d92fa9c342964cea5e95f syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi028.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCUSNVaQl5WeiAT0gHEsvl0ygugXPVSClauMigugq2UJ+dAgMEqmIvsZoNWBzBashC9pE97t51XpgAkk2r59kVg= smithi081.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMCOnqfcgZ62VPscyDatzXpS3IE3CO1tSuvz7xPZ2DxeiVsJvN6mJONmlC48XG7K18AySCjdQ+ADLKEeyfcr+TU= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - kclient: null - exec: mon.a: - ceph config set mgr mgr/crash/warn_recent_interval 0 - cephfs_test_runner: fail_on_skip: false modules: - tasks.cephfs.test_failover teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: b1dac5519c57c269ea98a22fb7729016a1d0e4d2 timestamp: 2024-02-21_22:31:03 tube: smithi user: leonidus verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-02-21T22:45:33.860 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_3bfc837d7b45ce8e095d92fa9c342964cea5e95f/qa; will attempt to use it 2024-02-21T22:45:33.860 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_3bfc837d7b45ce8e095d92fa9c342964cea5e95f/qa/tasks 2024-02-21T22:45:33.861 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-02-21T22:45:33.862 INFO:teuthology.task.internal:Checking packages... 2024-02-21T22:45:33.887 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '3bfc837d7b45ce8e095d92fa9c342964cea5e95f' 2024-02-21T22:45:33.887 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-21T22:45:33.887 INFO:teuthology.packaging:ref: None 2024-02-21T22:45:33.887 INFO:teuthology.packaging:tag: None 2024-02-21T22:45:33.887 INFO:teuthology.packaging:branch: wip-lusov-qdb 2024-02-21T22:45:33.887 INFO:teuthology.packaging:sha1: 3bfc837d7b45ce8e095d92fa9c342964cea5e95f 2024-02-21T22:45:33.887 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-lusov-qdb 2024-02-21T22:45:34.042 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1643.g3bfc837d 2024-02-21T22:45:34.044 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-02-21T22:45:34.050 INFO:teuthology.task.internal:no buildpackages task found 2024-02-21T22:45:34.051 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-02-21T22:45:34.061 INFO:teuthology.task.internal:Saving configuration 2024-02-21T22:45:34.078 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-02-21T22:45:34.083 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-21T22:45:34.107 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi028.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-02-21_22:31:03-fs-wip-lusov-qdb-distro-default-smithi/7570010', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-21 22:40:53.153709', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCUSNVaQl5WeiAT0gHEsvl0ygugXPVSClauMigugq2UJ+dAgMEqmIvsZoNWBzBashC9pE97t51XpgAkk2r59kVg='} 2024-02-21T22:45:34.130 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi081.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-02-21_22:31:03-fs-wip-lusov-qdb-distro-default-smithi/7570010', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-21 22:40:53.151886', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMCOnqfcgZ62VPscyDatzXpS3IE3CO1tSuvz7xPZ2DxeiVsJvN6mJONmlC48XG7K18AySCjdQ+ADLKEeyfcr+TU='} 2024-02-21T22:45:34.130 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-02-21T22:45:34.136 INFO:teuthology.task.internal:roles: ubuntu@smithi028.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-02-21T22:45:34.136 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1'] 2024-02-21T22:45:34.136 INFO:teuthology.run_tasks:Running task console_log... 2024-02-21T22:45:34.207 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f6ea367e0d0>, signals=[15]) 2024-02-21T22:45:34.207 INFO:teuthology.run_tasks:Running task internal.connect... 2024-02-21T22:45:34.213 INFO:teuthology.task.internal:Opening connections... 2024-02-21T22:45:34.213 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi028.front.sepia.ceph.com 2024-02-21T22:45:34.215 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-21T22:45:34.323 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi081.front.sepia.ceph.com 2024-02-21T22:45:34.324 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-21T22:45:34.432 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-02-21T22:45:34.439 DEBUG:teuthology.orchestra.run.smithi028:> uname -m 2024-02-21T22:45:34.458 INFO:teuthology.orchestra.run.smithi028.stdout:x86_64 2024-02-21T22:45:34.458 DEBUG:teuthology.orchestra.run.smithi028:> cat /etc/os-release 2024-02-21T22:45:34.517 INFO:teuthology.orchestra.run.smithi028.stdout:NAME="CentOS Stream" 2024-02-21T22:45:34.517 INFO:teuthology.orchestra.run.smithi028.stdout:VERSION="9" 2024-02-21T22:45:34.517 INFO:teuthology.orchestra.run.smithi028.stdout:ID="centos" 2024-02-21T22:45:34.517 INFO:teuthology.orchestra.run.smithi028.stdout:ID_LIKE="rhel fedora" 2024-02-21T22:45:34.517 INFO:teuthology.orchestra.run.smithi028.stdout:VERSION_ID="9" 2024-02-21T22:45:34.517 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM_ID="platform:el9" 2024-02-21T22:45:34.517 INFO:teuthology.orchestra.run.smithi028.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-21T22:45:34.517 INFO:teuthology.orchestra.run.smithi028.stdout:ANSI_COLOR="0;31" 2024-02-21T22:45:34.517 INFO:teuthology.orchestra.run.smithi028.stdout:LOGO="fedora-logo-icon" 2024-02-21T22:45:34.518 INFO:teuthology.orchestra.run.smithi028.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-21T22:45:34.518 INFO:teuthology.orchestra.run.smithi028.stdout:HOME_URL="https://centos.org/" 2024-02-21T22:45:34.518 INFO:teuthology.orchestra.run.smithi028.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-21T22:45:34.518 INFO:teuthology.orchestra.run.smithi028.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-21T22:45:34.518 INFO:teuthology.orchestra.run.smithi028.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-21T22:45:34.519 INFO:teuthology.lock.ops:Updating smithi028.front.sepia.ceph.com on lock server 2024-02-21T22:45:34.551 DEBUG:teuthology.orchestra.run.smithi081:> uname -m 2024-02-21T22:45:34.569 INFO:teuthology.orchestra.run.smithi081.stdout:x86_64 2024-02-21T22:45:34.569 DEBUG:teuthology.orchestra.run.smithi081:> cat /etc/os-release 2024-02-21T22:45:34.625 INFO:teuthology.orchestra.run.smithi081.stdout:NAME="CentOS Stream" 2024-02-21T22:45:34.625 INFO:teuthology.orchestra.run.smithi081.stdout:VERSION="9" 2024-02-21T22:45:34.625 INFO:teuthology.orchestra.run.smithi081.stdout:ID="centos" 2024-02-21T22:45:34.626 INFO:teuthology.orchestra.run.smithi081.stdout:ID_LIKE="rhel fedora" 2024-02-21T22:45:34.626 INFO:teuthology.orchestra.run.smithi081.stdout:VERSION_ID="9" 2024-02-21T22:45:34.626 INFO:teuthology.orchestra.run.smithi081.stdout:PLATFORM_ID="platform:el9" 2024-02-21T22:45:34.626 INFO:teuthology.orchestra.run.smithi081.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-21T22:45:34.626 INFO:teuthology.orchestra.run.smithi081.stdout:ANSI_COLOR="0;31" 2024-02-21T22:45:34.626 INFO:teuthology.orchestra.run.smithi081.stdout:LOGO="fedora-logo-icon" 2024-02-21T22:45:34.626 INFO:teuthology.orchestra.run.smithi081.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-21T22:45:34.626 INFO:teuthology.orchestra.run.smithi081.stdout:HOME_URL="https://centos.org/" 2024-02-21T22:45:34.626 INFO:teuthology.orchestra.run.smithi081.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-21T22:45:34.626 INFO:teuthology.orchestra.run.smithi081.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-21T22:45:34.626 INFO:teuthology.orchestra.run.smithi081.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-21T22:45:34.627 INFO:teuthology.lock.ops:Updating smithi081.front.sepia.ceph.com on lock server 2024-02-21T22:45:34.650 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-02-21T22:45:34.659 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-02-21T22:45:34.663 INFO:teuthology.task.internal:Checking for old test directory... 2024-02-21T22:45:34.663 DEBUG:teuthology.orchestra.run.smithi028:> test '!' -e /home/ubuntu/cephtest 2024-02-21T22:45:34.666 DEBUG:teuthology.orchestra.run.smithi081:> test '!' -e /home/ubuntu/cephtest 2024-02-21T22:45:34.682 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-02-21T22:45:34.690 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-02-21T22:45:34.690 DEBUG:teuthology.orchestra.run.smithi028:> test -z $(ls -A /var/lib/ceph) 2024-02-21T22:45:34.724 DEBUG:teuthology.orchestra.run.smithi081:> test -z $(ls -A /var/lib/ceph) 2024-02-21T22:45:34.749 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-02-21T22:45:34.805 INFO:teuthology.run_tasks:Running task kernel... 2024-02-21T22:45:34.819 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'} 2024-02-21T22:45:34.819 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}, 'client.1': {'branch': 'testing'}} 2024-02-21T22:45:34.819 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}, 'client.1': {'branch': 'testing'}}, timeout 300 2024-02-21T22:45:34.820 DEBUG:teuthology.orchestra.run.smithi028:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-21T22:45:34.820 DEBUG:teuthology.orchestra.run.smithi081:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-21T22:45:34.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-21T22:45:34.836 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-02-21T22:45:34.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-21T22:45:34.840 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-02-21T22:45:35.015 DEBUG:teuthology.task.kernel:sha1 for client.1 is 1130e053228021b1be9cac260b5ed52c5a9027c5 2024-02-21T22:45:35.015 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "1130e053228021b1be9cac260b5ed52c5a9027c5"... 2024-02-21T22:45:35.016 DEBUG:teuthology.orchestra.run.smithi081:> uname -r 2024-02-21T22:45:35.034 INFO:teuthology.orchestra.run.smithi081.stdout:5.14.0-361.el9.x86_64 2024-02-21T22:45:35.034 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 1130e053228021b1be9cac260b5ed52c5a9027c5 2024-02-21T22:45:35.035 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-02-21T22:45:35.035 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum upgrade -y linux-firmware 2024-02-21T22:45:35.052 DEBUG:teuthology.task.kernel:sha1 for client.0 is 1130e053228021b1be9cac260b5ed52c5a9027c5 2024-02-21T22:45:35.052 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "1130e053228021b1be9cac260b5ed52c5a9027c5"... 2024-02-21T22:45:35.053 DEBUG:teuthology.orchestra.run.smithi028:> uname -r 2024-02-21T22:45:35.071 INFO:teuthology.orchestra.run.smithi028.stdout:5.14.0-361.el9.x86_64 2024-02-21T22:45:35.071 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 1130e053228021b1be9cac260b5ed52c5a9027c5 2024-02-21T22:45:35.071 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-02-21T22:45:35.072 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum upgrade -y linux-firmware 2024-02-21T22:45:37.263 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS-9 - AppStream 14 MB/s | 8.1 MB 00:00 2024-02-21T22:45:37.320 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS-9 - AppStream 13 MB/s | 8.1 MB 00:00 2024-02-21T22:45:38.897 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS-9 - BaseOS 7.8 MB/s | 2.5 MB 00:00 2024-02-21T22:45:38.978 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS-9 - BaseOS 8.7 MB/s | 2.5 MB 00:00 2024-02-21T22:45:40.005 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-02-21T22:45:40.430 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - BaseOS 8.3 MB/s | 8.0 MB 00:00 2024-02-21T22:45:42.397 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - AppStream 15 MB/s | 19 MB 00:01 2024-02-21T22:45:48.190 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - CRB 2.8 MB/s | 6.1 MB 00:02 2024-02-21T22:45:49.799 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - Extras packages 32 kB/s | 15 kB 00:00 2024-02-21T22:45:55.669 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - AppStream 1.4 MB/s | 19 MB 00:14 2024-02-21T22:46:00.055 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - CRB 9.8 MB/s | 6.1 MB 00:00 2024-02-21T22:46:01.428 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - Extras packages 90 kB/s | 15 kB 00:00 2024-02-21T22:46:04.171 INFO:teuthology.orchestra.run.smithi028.stdout:Extra Packages for Enterprise Linux 7.9 MB/s | 20 MB 00:02 2024-02-21T22:46:09.430 INFO:teuthology.orchestra.run.smithi028.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-02-21T22:46:11.765 INFO:teuthology.orchestra.run.smithi081.stdout:Extra Packages for Enterprise Linux 960 kB/s | 20 MB 00:21 2024-02-21T22:46:11.769 INFO:teuthology.orchestra.run.smithi028.stderr:Modular dependency problems: 2024-02-21T22:46:11.769 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-02-21T22:46:11.769 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 1: conflicting requests 2024-02-21T22:46:11.769 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240219154654:20125149.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.769 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 2: conflicting requests 2024-02-21T22:46:11.769 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.769 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 3: conflicting requests 2024-02-21T22:46:11.769 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.769 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 4: conflicting requests 2024-02-21T22:46:11.769 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.769 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 5: conflicting requests 2024-02-21T22:46:11.769 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240213202549:b754926a.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.770 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 6: conflicting requests 2024-02-21T22:46:11.770 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.770 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 7: conflicting requests 2024-02-21T22:46:11.770 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.770 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 8: conflicting requests 2024-02-21T22:46:11.770 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.770 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 9: conflicting requests 2024-02-21T22:46:11.770 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.770 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 10: conflicting requests 2024-02-21T22:46:11.770 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.770 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 11: conflicting requests 2024-02-21T22:46:11.770 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.770 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 12: conflicting requests 2024-02-21T22:46:11.770 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.771 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 13: conflicting requests 2024-02-21T22:46:11.771 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.771 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 14: conflicting requests 2024-02-21T22:46:11.771 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.771 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 15: conflicting requests 2024-02-21T22:46:11.771 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.771 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 16: conflicting requests 2024-02-21T22:46:11.771 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.771 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 17: conflicting requests 2024-02-21T22:46:11.771 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.771 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 18: conflicting requests 2024-02-21T22:46:11.771 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.771 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 19: conflicting requests 2024-02-21T22:46:11.772 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.772 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 20: conflicting requests 2024-02-21T22:46:11.772 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.772 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 21: conflicting requests 2024-02-21T22:46:11.772 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.772 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 22: conflicting requests 2024-02-21T22:46:11.772 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.772 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 23: conflicting requests 2024-02-21T22:46:11.772 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.772 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 24: conflicting requests 2024-02-21T22:46:11.772 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.773 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 25: conflicting requests 2024-02-21T22:46:11.773 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.773 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 26: conflicting requests 2024-02-21T22:46:11.773 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.773 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 27: conflicting requests 2024-02-21T22:46:11.773 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.773 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 28: conflicting requests 2024-02-21T22:46:11.773 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.773 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 29: conflicting requests 2024-02-21T22:46:11.773 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.773 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 30: conflicting requests 2024-02-21T22:46:11.773 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.773 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 31: conflicting requests 2024-02-21T22:46:11.773 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240214183150:9edba152.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.773 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 32: conflicting requests 2024-02-21T22:46:11.774 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.774 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 33: conflicting requests 2024-02-21T22:46:11.774 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.774 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 34: conflicting requests 2024-02-21T22:46:11.774 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.774 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 35: conflicting requests 2024-02-21T22:46:11.774 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240203221207:9edba152.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.774 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 36: conflicting requests 2024-02-21T22:46:11.774 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.774 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.774 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.774 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 37: cannot install the best candidate for the job 2024-02-21T22:46:11.774 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.774 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.775 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.775 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.775 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 38: conflicting requests 2024-02-21T22:46:11.775 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.775 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.775 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.775 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.775 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.775 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.775 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 39: cannot install the best candidate for the job 2024-02-21T22:46:11.775 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.775 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.775 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.775 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.776 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 40: cannot install the best candidate for the job 2024-02-21T22:46:11.776 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.776 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.776 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.776 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.776 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 41: cannot install the best candidate for the job 2024-02-21T22:46:11.776 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.776 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.776 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.776 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.776 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 42: cannot install the best candidate for the job 2024-02-21T22:46:11.777 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.777 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.777 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.777 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.777 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 43: cannot install the best candidate for the job 2024-02-21T22:46:11.777 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.777 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.777 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.777 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.777 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 44: cannot install the best candidate for the job 2024-02-21T22:46:11.777 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.777 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.777 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-02-21T22:46:11.777 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-02-21T22:46:12.365 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-02-21T22:46:12.375 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-02-21T22:46:12.376 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2024-02-21T22:46:12.376 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-02-21T22:46:12.376 INFO:teuthology.orchestra.run.smithi028.stdout:Upgrading: 2024-02-21T22:46:12.376 INFO:teuthology.orchestra.run.smithi028.stdout: linux-firmware noarch 20240111-121.gitb3132c18.el8 CentOS-BaseOS 354 M 2024-02-21T22:46:12.376 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T22:46:12.376 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-02-21T22:46:12.376 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-02-21T22:46:12.376 INFO:teuthology.orchestra.run.smithi028.stdout:Upgrade 1 Package 2024-02-21T22:46:12.376 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T22:46:12.376 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 354 M 2024-02-21T22:46:12.376 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-02-21T22:46:16.722 INFO:teuthology.orchestra.run.smithi081.stdout:lab-extras 39 kB/s | 1.7 kB 00:00 2024-02-21T22:46:18.967 INFO:teuthology.orchestra.run.smithi081.stderr:Modular dependency problems: 2024-02-21T22:46:18.967 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-02-21T22:46:18.967 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 1: conflicting requests 2024-02-21T22:46:18.967 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240219154654:20125149.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.968 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 2: conflicting requests 2024-02-21T22:46:18.968 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.968 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 3: conflicting requests 2024-02-21T22:46:18.968 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.968 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 4: conflicting requests 2024-02-21T22:46:18.968 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.968 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 5: conflicting requests 2024-02-21T22:46:18.969 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240213202549:b754926a.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.969 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 6: conflicting requests 2024-02-21T22:46:18.969 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.969 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 7: conflicting requests 2024-02-21T22:46:18.969 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.969 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 8: conflicting requests 2024-02-21T22:46:18.969 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.969 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 9: conflicting requests 2024-02-21T22:46:18.969 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.969 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 10: conflicting requests 2024-02-21T22:46:18.969 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.969 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 11: conflicting requests 2024-02-21T22:46:18.969 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.969 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 12: conflicting requests 2024-02-21T22:46:18.970 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.970 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 13: conflicting requests 2024-02-21T22:46:18.970 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.970 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 14: conflicting requests 2024-02-21T22:46:18.970 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.970 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 15: conflicting requests 2024-02-21T22:46:18.970 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.970 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 16: conflicting requests 2024-02-21T22:46:18.970 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.970 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 17: conflicting requests 2024-02-21T22:46:18.970 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.970 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 18: conflicting requests 2024-02-21T22:46:18.970 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.970 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 19: conflicting requests 2024-02-21T22:46:18.970 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.971 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 20: conflicting requests 2024-02-21T22:46:18.971 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.971 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 21: conflicting requests 2024-02-21T22:46:18.971 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.971 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 22: conflicting requests 2024-02-21T22:46:18.971 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.971 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 23: conflicting requests 2024-02-21T22:46:18.971 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.971 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 24: conflicting requests 2024-02-21T22:46:18.971 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.971 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 25: conflicting requests 2024-02-21T22:46:18.971 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.971 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 26: conflicting requests 2024-02-21T22:46:18.971 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.971 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 27: conflicting requests 2024-02-21T22:46:18.972 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.972 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 28: conflicting requests 2024-02-21T22:46:18.972 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.972 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 29: conflicting requests 2024-02-21T22:46:18.972 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.972 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 30: conflicting requests 2024-02-21T22:46:18.972 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.972 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 31: conflicting requests 2024-02-21T22:46:18.972 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240214183150:9edba152.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.972 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 32: conflicting requests 2024-02-21T22:46:18.972 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.972 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 33: conflicting requests 2024-02-21T22:46:18.972 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.972 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 34: conflicting requests 2024-02-21T22:46:18.973 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.973 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 35: conflicting requests 2024-02-21T22:46:18.973 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240203221207:9edba152.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.973 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 36: conflicting requests 2024-02-21T22:46:18.973 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.973 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.973 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.973 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 37: cannot install the best candidate for the job 2024-02-21T22:46:18.973 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.973 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.973 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.973 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.973 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 38: conflicting requests 2024-02-21T22:46:18.973 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.973 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.974 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.974 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.974 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.974 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.974 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 39: cannot install the best candidate for the job 2024-02-21T22:46:18.974 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.974 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.974 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.974 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.974 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 40: cannot install the best candidate for the job 2024-02-21T22:46:18.974 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.974 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.974 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.974 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.975 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 41: cannot install the best candidate for the job 2024-02-21T22:46:18.975 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.975 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.975 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.975 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.975 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 42: cannot install the best candidate for the job 2024-02-21T22:46:18.975 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.975 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.975 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.975 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.975 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 43: cannot install the best candidate for the job 2024-02-21T22:46:18.975 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.975 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.975 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.975 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.975 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 44: cannot install the best candidate for the job 2024-02-21T22:46:18.976 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.976 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.976 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-02-21T22:46:18.976 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-02-21T22:46:19.570 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-02-21T22:46:19.578 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-02-21T22:46:19.578 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2024-02-21T22:46:19.578 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-02-21T22:46:19.579 INFO:teuthology.orchestra.run.smithi081.stdout:Upgrading: 2024-02-21T22:46:19.579 INFO:teuthology.orchestra.run.smithi081.stdout: linux-firmware noarch 20240111-121.gitb3132c18.el8 CentOS-BaseOS 354 M 2024-02-21T22:46:19.579 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-02-21T22:46:19.579 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-02-21T22:46:19.579 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-02-21T22:46:19.579 INFO:teuthology.orchestra.run.smithi081.stdout:Upgrade 1 Package 2024-02-21T22:46:19.579 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-02-21T22:46:19.579 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 354 M 2024-02-21T22:46:19.579 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-02-21T22:46:19.672 INFO:teuthology.orchestra.run.smithi028.stdout:linux-firmware-20240111-121.gitb3132c18.el8.noa 48 MB/s | 354 MB 00:07 2024-02-21T22:46:19.673 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-02-21T22:46:19.674 INFO:teuthology.orchestra.run.smithi028.stdout:Total 48 MB/s | 354 MB 00:07 2024-02-21T22:46:19.674 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-02-21T22:46:19.769 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-02-21T22:46:19.769 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-02-21T22:46:21.353 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-02-21T22:46:21.353 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-02-21T22:46:28.073 INFO:teuthology.orchestra.run.smithi081.stdout:linux-firmware-20240111-121.gitb3132c18.el8.noa 42 MB/s | 354 MB 00:08 2024-02-21T22:46:28.074 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-02-21T22:46:28.075 INFO:teuthology.orchestra.run.smithi081.stdout:Total 42 MB/s | 354 MB 00:08 2024-02-21T22:46:28.075 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-02-21T22:46:28.167 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-02-21T22:46:28.167 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-02-21T22:46:29.791 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-02-21T22:46:29.791 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-02-21T22:46:43.869 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-02-21T22:46:44.138 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : linux-firmware-20240111-121.gitb3132c18.el8.noarch 1/2 2024-02-21T22:46:44.236 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-02-21T22:46:51.093 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-02-21T22:46:51.339 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : linux-firmware-20240111-121.gitb3132c18.el8.noarch 1/2 2024-02-21T22:46:51.445 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-02-21T22:46:51.556 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-02-21T22:46:51.556 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : linux-firmware-20240111-121.gitb3132c18.el8.noarch 1/2 2024-02-21T22:46:51.983 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-02-21T22:46:51.983 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T22:46:51.983 INFO:teuthology.orchestra.run.smithi028.stdout:Upgraded: 2024-02-21T22:46:51.983 INFO:teuthology.orchestra.run.smithi028.stdout: linux-firmware-20240111-121.gitb3132c18.el8.noarch 2024-02-21T22:46:51.983 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T22:46:51.984 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-02-21T22:46:52.266 INFO:teuthology.task.kernel:Downloading kernel 1130e053228021b1be9cac260b5ed52c5a9027c5 on client.0... 2024-02-21T22:46:52.266 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1130e053228021b1be9cac260b5ed52c5a9027c5 2024-02-21T22:46:52.415 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/1130e053228021b1be9cac260b5ed52c5a9027c5/centos/9/flavors/default/x86_64/ 2024-02-21T22:46:52.415 DEBUG:teuthology.orchestra.run.smithi028:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.8.0_rc4_g1130e0532280-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/1130e053228021b1be9cac260b5ed52c5a9027c5/centos/9/flavors/default/x86_64/ --input-file=- 2024-02-21T22:46:52.420 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi028.front.sepia.ceph.com to complete... 2024-02-21T22:46:59.180 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-02-21T22:46:59.180 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : linux-firmware-20240111-121.gitb3132c18.el8.noarch 1/2 2024-02-21T22:46:59.617 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-02-21T22:46:59.617 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-02-21T22:46:59.617 INFO:teuthology.orchestra.run.smithi081.stdout:Upgraded: 2024-02-21T22:46:59.617 INFO:teuthology.orchestra.run.smithi081.stdout: linux-firmware-20240111-121.gitb3132c18.el8.noarch 2024-02-21T22:46:59.617 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-02-21T22:46:59.617 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-02-21T22:46:59.908 INFO:teuthology.task.kernel:Downloading kernel 1130e053228021b1be9cac260b5ed52c5a9027c5 on client.1... 2024-02-21T22:46:59.908 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1130e053228021b1be9cac260b5ed52c5a9027c5 2024-02-21T22:47:00.058 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/1130e053228021b1be9cac260b5ed52c5a9027c5/centos/9/flavors/default/x86_64/ 2024-02-21T22:47:00.058 DEBUG:teuthology.orchestra.run.smithi081:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.8.0_rc4_g1130e0532280-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/1130e053228021b1be9cac260b5ed52c5a9027c5/centos/9/flavors/default/x86_64/ --input-file=- 2024-02-21T22:47:00.061 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi081.front.sepia.ceph.com to complete... 2024-02-21T22:49:28.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21 22:49:28 URL:https://3.chacra.ceph.com/r/kernel/testing/1130e053228021b1be9cac260b5ed52c5a9027c5/centos/9/flavors/default/x86_64/kernel-6.8.0_rc4_g1130e0532280-1.x86_64.rpm [332924888/332924888] -> "/tmp/kernel.x86_64.rpm" [1] 2024-02-21T22:49:28.842 INFO:teuthology.orchestra.run.smithi081.stderr:FINISHED --2024-02-21 22:49:28-- 2024-02-21T22:49:28.842 INFO:teuthology.orchestra.run.smithi081.stderr:Total wall clock time: 2m 29s 2024-02-21T22:49:28.842 INFO:teuthology.orchestra.run.smithi081.stderr:Downloaded: 1 files, 318M in 2m 28s (2.14 MB/s) 2024-02-21T22:49:28.846 INFO:teuthology.task.kernel:Installing kernel 1130e053228021b1be9cac260b5ed52c5a9027c5 on client.1... 2024-02-21T22:49:28.846 DEBUG:teuthology.orchestra.run.smithi081:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-02-21T22:49:30.764 INFO:teuthology.orchestra.run.smithi081.stdout:Verifying... ######################################## 2024-02-21T22:49:30.896 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing... ######################################## 2024-02-21T22:49:30.975 INFO:teuthology.orchestra.run.smithi081.stdout:Updating / installing... 2024-02-21T22:49:33.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21 22:49:33 URL:https://3.chacra.ceph.com/r/kernel/testing/1130e053228021b1be9cac260b5ed52c5a9027c5/centos/9/flavors/default/x86_64/kernel-6.8.0_rc4_g1130e0532280-1.x86_64.rpm [332924888/332924888] -> "/tmp/kernel.x86_64.rpm" [1] 2024-02-21T22:49:33.577 INFO:teuthology.orchestra.run.smithi028.stderr:FINISHED --2024-02-21 22:49:33-- 2024-02-21T22:49:33.577 INFO:teuthology.orchestra.run.smithi028.stderr:Total wall clock time: 2m 41s 2024-02-21T22:49:33.578 INFO:teuthology.orchestra.run.smithi028.stderr:Downloaded: 1 files, 318M in 2m 41s (1.98 MB/s) 2024-02-21T22:49:33.580 INFO:teuthology.task.kernel:Installing kernel 1130e053228021b1be9cac260b5ed52c5a9027c5 on client.0... 2024-02-21T22:49:33.581 DEBUG:teuthology.orchestra.run.smithi028:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-02-21T22:49:35.595 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying... ######################################## 2024-02-21T22:49:35.687 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing... ######################################## 2024-02-21T22:49:35.772 INFO:teuthology.orchestra.run.smithi028.stdout:Updating / installing... 2024-02-21T22:49:38.428 INFO:teuthology.orchestra.run.smithi081.stdout:kernel-6.8.0_rc4_g1130e0532280-1 ######################################## 2024-02-21T22:49:43.554 INFO:teuthology.orchestra.run.smithi028.stdout:kernel-6.8.0_rc4_g1130e0532280-1 ######################################## 2024-02-21T22:49:52.504 INFO:teuthology.orchestra.run.smithi081.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-21T22:49:56.853 INFO:teuthology.orchestra.run.smithi028.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-21T22:50:14.585 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi081.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-02-21T22:50:14.585 DEBUG:teuthology.orchestra.run.smithi081:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-02-21T22:50:14.675 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280 2024-02-21T22:50:14.675 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/System.map 2024-02-21T22:50:14.675 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/config 2024-02-21T22:50:14.675 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel 2024-02-21T22:50:14.675 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/arch 2024-02-21T22:50:14.675 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/arch/x86 2024-02-21T22:50:14.675 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/arch/x86/crypto 2024-02-21T22:50:14.676 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-02-21T22:50:14.676 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-02-21T22:50:14.676 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-02-21T22:50:14.676 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-02-21T22:50:14.676 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-02-21T22:50:14.676 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-02-21T22:50:14.676 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-02-21T22:50:14.676 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/arch/x86/crypto/crc32c-intel.ko 2024-02-21T22:50:14.676 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-02-21T22:50:14.676 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-02-21T22:50:14.677 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-02-21T22:50:14.677 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-02-21T22:50:14.677 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/arch/x86/crypto/serpent-avx2.ko 2024-02-21T22:50:14.677 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-02-21T22:50:14.677 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-02-21T22:50:14.677 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-02-21T22:50:14.677 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-02-21T22:50:14.677 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-02-21T22:50:14.677 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-02-21T22:50:14.677 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-02-21T22:50:14.677 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/arch/x86/kernel 2024-02-21T22:50:14.678 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/arch/x86/kernel/cpu 2024-02-21T22:50:14.678 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/arch/x86/kernel/cpu/mce 2024-02-21T22:50:14.678 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-02-21T22:50:14.678 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/arch/x86/kvm 2024-02-21T22:50:14.678 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/arch/x86/kvm/kvm-amd.ko 2024-02-21T22:50:14.678 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/arch/x86/kvm/kvm-intel.ko 2024-02-21T22:50:14.678 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/arch/x86/kvm/kvm.ko 2024-02-21T22:50:14.679 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto 2024-02-21T22:50:14.679 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/ansi_cprng.ko 2024-02-21T22:50:14.679 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/anubis.ko 2024-02-21T22:50:14.679 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/arc4.ko 2024-02-21T22:50:14.680 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/async_tx 2024-02-21T22:50:14.680 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/async_tx/async_memcpy.ko 2024-02-21T22:50:14.680 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/async_tx/async_pq.ko 2024-02-21T22:50:14.680 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/async_tx/async_raid6_recov.ko 2024-02-21T22:50:14.681 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/async_tx/async_tx.ko 2024-02-21T22:50:14.681 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/async_tx/async_xor.ko 2024-02-21T22:50:14.681 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/async_tx/raid6test.ko 2024-02-21T22:50:14.681 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/authenc.ko 2024-02-21T22:50:14.681 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/authencesn.ko 2024-02-21T22:50:14.682 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/blake2b_generic.ko 2024-02-21T22:50:14.682 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/blowfish_common.ko 2024-02-21T22:50:14.682 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/blowfish_generic.ko 2024-02-21T22:50:14.682 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/camellia_generic.ko 2024-02-21T22:50:14.682 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/cast5_generic.ko 2024-02-21T22:50:14.682 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/cast6_generic.ko 2024-02-21T22:50:14.683 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/cast_common.ko 2024-02-21T22:50:14.683 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/ccm.ko 2024-02-21T22:50:14.683 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/cmac.ko 2024-02-21T22:50:14.683 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/crc32_generic.ko 2024-02-21T22:50:14.683 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/crypto_engine.ko 2024-02-21T22:50:14.684 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/crypto_user.ko 2024-02-21T22:50:14.684 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/cts.ko 2024-02-21T22:50:14.684 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/des_generic.ko 2024-02-21T22:50:14.685 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/dh_generic.ko 2024-02-21T22:50:14.685 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/echainiv.ko 2024-02-21T22:50:14.685 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/essiv.ko 2024-02-21T22:50:14.685 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/fcrypt.ko 2024-02-21T22:50:14.685 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/khazad.ko 2024-02-21T22:50:14.685 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/lz4.ko 2024-02-21T22:50:14.686 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/lz4hc.ko 2024-02-21T22:50:14.686 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/md4.ko 2024-02-21T22:50:14.686 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/michael_mic.ko 2024-02-21T22:50:14.686 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/pcbc.ko 2024-02-21T22:50:14.686 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/pcrypt.ko 2024-02-21T22:50:14.687 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/rmd160.ko 2024-02-21T22:50:14.687 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/seed.ko 2024-02-21T22:50:14.687 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/serpent_generic.ko 2024-02-21T22:50:14.688 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/tcrypt.ko 2024-02-21T22:50:14.688 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/tea.ko 2024-02-21T22:50:14.688 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/twofish_common.ko 2024-02-21T22:50:14.688 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/twofish_generic.ko 2024-02-21T22:50:14.689 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/vmac.ko 2024-02-21T22:50:14.689 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/wp512.ko 2024-02-21T22:50:14.689 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/xcbc.ko 2024-02-21T22:50:14.689 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/xor.ko 2024-02-21T22:50:14.689 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/xxhash_generic.ko 2024-02-21T22:50:14.689 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers 2024-02-21T22:50:14.691 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/acpi 2024-02-21T22:50:14.691 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/acpi/acpi_ipmi.ko 2024-02-21T22:50:14.691 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/acpi/acpi_pad.ko 2024-02-21T22:50:14.692 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/acpi/custom_method.ko 2024-02-21T22:50:14.692 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/acpi/ec_sys.ko 2024-02-21T22:50:14.692 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/acpi/platform_profile.ko 2024-02-21T22:50:14.692 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/acpi/sbs.ko 2024-02-21T22:50:14.692 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/acpi/sbshc.ko 2024-02-21T22:50:14.692 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/acpi/video.ko 2024-02-21T22:50:14.693 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata 2024-02-21T22:50:14.693 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/acard-ahci.ko 2024-02-21T22:50:14.693 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/ahci_platform.ko 2024-02-21T22:50:14.693 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/ata_generic.ko 2024-02-21T22:50:14.693 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/libahci_platform.ko 2024-02-21T22:50:14.693 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_acpi.ko 2024-02-21T22:50:14.693 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_ali.ko 2024-02-21T22:50:14.694 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_amd.ko 2024-02-21T22:50:14.694 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_artop.ko 2024-02-21T22:50:14.694 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_atiixp.ko 2024-02-21T22:50:14.695 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_atp867x.ko 2024-02-21T22:50:14.695 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_cmd640.ko 2024-02-21T22:50:14.695 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_cmd64x.ko 2024-02-21T22:50:14.695 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_cypress.ko 2024-02-21T22:50:14.695 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_efar.ko 2024-02-21T22:50:14.696 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_hpt366.ko 2024-02-21T22:50:14.696 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_hpt37x.ko 2024-02-21T22:50:14.696 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_hpt3x2n.ko 2024-02-21T22:50:14.696 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_hpt3x3.ko 2024-02-21T22:50:14.696 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_it8213.ko 2024-02-21T22:50:14.696 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_it821x.ko 2024-02-21T22:50:14.696 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_jmicron.ko 2024-02-21T22:50:14.697 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_marvell.ko 2024-02-21T22:50:14.697 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_mpiix.ko 2024-02-21T22:50:14.697 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_netcell.ko 2024-02-21T22:50:14.697 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_ninja32.ko 2024-02-21T22:50:14.697 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_ns87410.ko 2024-02-21T22:50:14.697 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_ns87415.ko 2024-02-21T22:50:14.697 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_oldpiix.ko 2024-02-21T22:50:14.697 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_opti.ko 2024-02-21T22:50:14.698 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_optidma.ko 2024-02-21T22:50:14.698 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_pcmcia.ko 2024-02-21T22:50:14.698 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_pdc2027x.ko 2024-02-21T22:50:14.698 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-02-21T22:50:14.698 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_piccolo.ko 2024-02-21T22:50:14.698 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_rdc.ko 2024-02-21T22:50:14.698 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_sch.ko 2024-02-21T22:50:14.699 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_serverworks.ko 2024-02-21T22:50:14.699 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_sil680.ko 2024-02-21T22:50:14.699 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_sis.ko 2024-02-21T22:50:14.699 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_sl82c105.ko 2024-02-21T22:50:14.699 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_triflex.ko 2024-02-21T22:50:14.700 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_via.ko 2024-02-21T22:50:14.700 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pdc_adma.ko 2024-02-21T22:50:14.700 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/sata_inic162x.ko 2024-02-21T22:50:14.700 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/sata_mv.ko 2024-02-21T22:50:14.700 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/sata_nv.ko 2024-02-21T22:50:14.700 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/sata_promise.ko 2024-02-21T22:50:14.701 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/sata_qstor.ko 2024-02-21T22:50:14.701 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/sata_sil.ko 2024-02-21T22:50:14.701 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/sata_sil24.ko 2024-02-21T22:50:14.702 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/sata_sis.ko 2024-02-21T22:50:14.702 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/sata_svw.ko 2024-02-21T22:50:14.702 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/sata_sx4.ko 2024-02-21T22:50:14.702 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/sata_uli.ko 2024-02-21T22:50:14.702 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/sata_via.ko 2024-02-21T22:50:14.702 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/sata_vsc.ko 2024-02-21T22:50:14.703 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/auxdisplay 2024-02-21T22:50:14.703 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/auxdisplay/ks0108.ko 2024-02-21T22:50:14.703 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/base 2024-02-21T22:50:14.703 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/base/regmap 2024-02-21T22:50:14.703 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/base/regmap/regmap-i2c.ko 2024-02-21T22:50:14.703 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/base/regmap/regmap-mmio.ko 2024-02-21T22:50:14.703 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/bcma 2024-02-21T22:50:14.704 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/bcma/bcma.ko 2024-02-21T22:50:14.704 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/block 2024-02-21T22:50:14.704 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/block/aoe 2024-02-21T22:50:14.704 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/block/aoe/aoe.ko 2024-02-21T22:50:14.704 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/block/brd.ko 2024-02-21T22:50:14.704 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/block/drbd 2024-02-21T22:50:14.704 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/block/drbd/drbd.ko 2024-02-21T22:50:14.704 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/block/floppy.ko 2024-02-21T22:50:14.705 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/block/loop.ko 2024-02-21T22:50:14.705 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/block/mtip32xx 2024-02-21T22:50:14.705 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-02-21T22:50:14.705 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/block/nbd.ko 2024-02-21T22:50:14.705 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/block/null_blk 2024-02-21T22:50:14.705 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/block/null_blk/null_blk.ko 2024-02-21T22:50:14.705 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/block/pktcdvd.ko 2024-02-21T22:50:14.705 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/block/rbd.ko 2024-02-21T22:50:14.705 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/block/virtio_blk.ko 2024-02-21T22:50:14.705 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/block/xen-blkback 2024-02-21T22:50:14.705 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-02-21T22:50:14.706 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/block/xen-blkfront.ko 2024-02-21T22:50:14.706 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/block/zram 2024-02-21T22:50:14.706 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/block/zram/zram.ko 2024-02-21T22:50:14.706 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/char 2024-02-21T22:50:14.706 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/char/hangcheck-timer.ko 2024-02-21T22:50:14.706 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/char/hw_random 2024-02-21T22:50:14.706 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/char/hw_random/amd-rng.ko 2024-02-21T22:50:14.706 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/char/hw_random/intel-rng.ko 2024-02-21T22:50:14.706 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-02-21T22:50:14.706 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/char/hw_random/via-rng.ko 2024-02-21T22:50:14.706 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/char/hw_random/virtio-rng.ko 2024-02-21T22:50:14.706 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/char/ipmi 2024-02-21T22:50:14.706 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-02-21T22:50:14.706 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-02-21T22:50:14.706 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-02-21T22:50:14.707 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/char/ipmi/ipmi_si.ko 2024-02-21T22:50:14.707 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-02-21T22:50:14.707 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-02-21T22:50:14.707 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/char/lp.ko 2024-02-21T22:50:14.707 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/char/mwave 2024-02-21T22:50:14.707 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/char/mwave/mwave.ko 2024-02-21T22:50:14.707 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/char/ppdev.ko 2024-02-21T22:50:14.708 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/char/tlclk.ko 2024-02-21T22:50:14.708 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/char/tpm 2024-02-21T22:50:14.708 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/char/tpm/tpm.ko 2024-02-21T22:50:14.708 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/char/tpm/tpm_atmel.ko 2024-02-21T22:50:14.708 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/char/tpm/tpm_infineon.ko 2024-02-21T22:50:14.708 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/char/tpm/tpm_nsc.ko 2024-02-21T22:50:14.708 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/char/tpm/tpm_tis.ko 2024-02-21T22:50:14.709 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-02-21T22:50:14.709 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/char/uv_mmtimer.ko 2024-02-21T22:50:14.709 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/char/virtio_console.ko 2024-02-21T22:50:14.709 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/cpufreq 2024-02-21T22:50:14.709 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-02-21T22:50:14.709 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-02-21T22:50:14.710 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/cpufreq/p4-clockmod.ko 2024-02-21T22:50:14.710 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-02-21T22:50:14.710 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/cpufreq/powernow-k8.ko 2024-02-21T22:50:14.710 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/cpufreq/speedstep-lib.ko 2024-02-21T22:50:14.710 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/crypto 2024-02-21T22:50:14.710 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/crypto/ccp 2024-02-21T22:50:14.710 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-02-21T22:50:14.711 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/crypto/ccp/ccp.ko 2024-02-21T22:50:14.711 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/crypto/intel 2024-02-21T22:50:14.711 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/crypto/intel/qat 2024-02-21T22:50:14.711 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/crypto/intel/qat/qat_common 2024-02-21T22:50:14.712 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-02-21T22:50:14.712 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-02-21T22:50:14.712 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-02-21T22:50:14.712 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/crypto/padlock-aes.ko 2024-02-21T22:50:14.712 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/crypto/padlock-sha.ko 2024-02-21T22:50:14.712 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/crypto/virtio 2024-02-21T22:50:14.713 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-02-21T22:50:14.713 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/dca 2024-02-21T22:50:14.713 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/dca/dca.ko 2024-02-21T22:50:14.713 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/devfreq 2024-02-21T22:50:14.713 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-02-21T22:50:14.713 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/dma 2024-02-21T22:50:14.714 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/dma/dw 2024-02-21T22:50:14.714 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/dma/dw/dw_dmac.ko 2024-02-21T22:50:14.715 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/dma/ioat 2024-02-21T22:50:14.715 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/dma/ioat/ioatdma.ko 2024-02-21T22:50:14.715 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/edac 2024-02-21T22:50:14.715 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/edac/amd64_edac.ko 2024-02-21T22:50:14.715 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/edac/e752x_edac.ko 2024-02-21T22:50:14.716 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/edac/edac_mce_amd.ko 2024-02-21T22:50:14.716 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/edac/i3000_edac.ko 2024-02-21T22:50:14.716 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/edac/i3200_edac.ko 2024-02-21T22:50:14.716 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/edac/i5100_edac.ko 2024-02-21T22:50:14.716 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/edac/i5400_edac.ko 2024-02-21T22:50:14.716 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/edac/i7300_edac.ko 2024-02-21T22:50:14.717 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/edac/i7core_edac.ko 2024-02-21T22:50:14.717 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/edac/i82975x_edac.ko 2024-02-21T22:50:14.717 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/edac/ie31200_edac.ko 2024-02-21T22:50:14.717 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/edac/sb_edac.ko 2024-02-21T22:50:14.717 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/edac/x38_edac.ko 2024-02-21T22:50:14.717 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/firmware 2024-02-21T22:50:14.718 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/firmware/edd.ko 2024-02-21T22:50:14.718 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/firmware/iscsi_ibft.ko 2024-02-21T22:50:14.718 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/gpio 2024-02-21T22:50:14.718 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/gpio/gpio-ich.ko 2024-02-21T22:50:14.718 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/gpio/gpio-viperboard.ko 2024-02-21T22:50:14.718 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid 2024-02-21T22:50:14.719 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-appleir.ko 2024-02-21T22:50:14.719 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-aureal.ko 2024-02-21T22:50:14.719 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-axff.ko 2024-02-21T22:50:14.719 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-dr.ko 2024-02-21T22:50:14.719 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-elecom.ko 2024-02-21T22:50:14.720 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-elo.ko 2024-02-21T22:50:14.720 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-emsff.ko 2024-02-21T22:50:14.720 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-gaff.ko 2024-02-21T22:50:14.721 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-gt683r.ko 2024-02-21T22:50:14.721 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-gyration.ko 2024-02-21T22:50:14.721 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-holtek-kbd.ko 2024-02-21T22:50:14.721 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-holtek-mouse.ko 2024-02-21T22:50:14.721 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-holtekff.ko 2024-02-21T22:50:14.721 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-hyperv.ko 2024-02-21T22:50:14.722 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-icade.ko 2024-02-21T22:50:14.722 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-keytouch.ko 2024-02-21T22:50:14.722 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-kye.ko 2024-02-21T22:50:14.722 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-lcpower.ko 2024-02-21T22:50:14.723 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-led.ko 2024-02-21T22:50:14.723 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-lenovo.ko 2024-02-21T22:50:14.723 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-logitech-dj.ko 2024-02-21T22:50:14.723 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-02-21T22:50:14.723 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-multitouch.ko 2024-02-21T22:50:14.723 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-ortek.ko 2024-02-21T22:50:14.724 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-penmount.ko 2024-02-21T22:50:14.724 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-petalynx.ko 2024-02-21T22:50:14.724 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-picolcd.ko 2024-02-21T22:50:14.724 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-pl.ko 2024-02-21T22:50:14.724 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-plantronics.ko 2024-02-21T22:50:14.724 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-primax.ko 2024-02-21T22:50:14.725 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-rmi.ko 2024-02-21T22:50:14.725 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-roccat-arvo.ko 2024-02-21T22:50:14.725 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-roccat-common.ko 2024-02-21T22:50:14.725 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-roccat-isku.ko 2024-02-21T22:50:14.725 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-roccat-kone.ko 2024-02-21T22:50:14.725 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-02-21T22:50:14.726 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-roccat-konepure.ko 2024-02-21T22:50:14.726 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-02-21T22:50:14.726 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-roccat-lua.ko 2024-02-21T22:50:14.726 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-roccat-pyra.ko 2024-02-21T22:50:14.726 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-roccat-ryos.ko 2024-02-21T22:50:14.726 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-roccat-savu.ko 2024-02-21T22:50:14.727 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-roccat.ko 2024-02-21T22:50:14.727 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-saitek.ko 2024-02-21T22:50:14.727 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-samsung.ko 2024-02-21T22:50:14.727 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-sensor-hub.ko 2024-02-21T22:50:14.727 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-sjoy.ko 2024-02-21T22:50:14.727 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-sony.ko 2024-02-21T22:50:14.728 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-speedlink.ko 2024-02-21T22:50:14.728 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-steelseries.ko 2024-02-21T22:50:14.728 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-sunplus.ko 2024-02-21T22:50:14.728 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-thrustmaster.ko 2024-02-21T22:50:14.728 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-tivo.ko 2024-02-21T22:50:14.728 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-tmff.ko 2024-02-21T22:50:14.729 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-topseed.ko 2024-02-21T22:50:14.729 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-twinhan.ko 2024-02-21T22:50:14.729 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-uclogic.ko 2024-02-21T22:50:14.729 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-waltop.ko 2024-02-21T22:50:14.729 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-wiimote.ko 2024-02-21T22:50:14.729 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-xinmo.ko 2024-02-21T22:50:14.730 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-zpff.ko 2024-02-21T22:50:14.730 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-zydacron.ko 2024-02-21T22:50:14.730 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/uhid.ko 2024-02-21T22:50:14.730 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/wacom.ko 2024-02-21T22:50:14.730 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hv 2024-02-21T22:50:14.731 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hv/hv_balloon.ko 2024-02-21T22:50:14.731 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hv/hv_utils.ko 2024-02-21T22:50:14.731 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hv/hv_vmbus.ko 2024-02-21T22:50:14.731 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon 2024-02-21T22:50:14.731 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/abituguru.ko 2024-02-21T22:50:14.731 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/abituguru3.ko 2024-02-21T22:50:14.732 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/acpi_power_meter.ko 2024-02-21T22:50:14.732 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/ad7414.ko 2024-02-21T22:50:14.732 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/ad7418.ko 2024-02-21T22:50:14.732 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/adc128d818.ko 2024-02-21T22:50:14.732 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/adm1025.ko 2024-02-21T22:50:14.732 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/adm1026.ko 2024-02-21T22:50:14.733 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/adm1029.ko 2024-02-21T22:50:14.733 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/adm1031.ko 2024-02-21T22:50:14.733 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/adm9240.ko 2024-02-21T22:50:14.733 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/ads7828.ko 2024-02-21T22:50:14.733 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/adt7410.ko 2024-02-21T22:50:14.733 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/adt7411.ko 2024-02-21T22:50:14.734 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/adt7462.ko 2024-02-21T22:50:14.734 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/adt7470.ko 2024-02-21T22:50:14.734 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/adt7475.ko 2024-02-21T22:50:14.734 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/adt7x10.ko 2024-02-21T22:50:14.734 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/amc6821.ko 2024-02-21T22:50:14.734 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/applesmc.ko 2024-02-21T22:50:14.735 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/asb100.ko 2024-02-21T22:50:14.735 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/asc7621.ko 2024-02-21T22:50:14.735 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/asus_atk0110.ko 2024-02-21T22:50:14.735 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/atxp1.ko 2024-02-21T22:50:14.735 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/coretemp.ko 2024-02-21T22:50:14.736 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-02-21T22:50:14.736 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/dme1737.ko 2024-02-21T22:50:14.736 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/ds1621.ko 2024-02-21T22:50:14.736 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/ds620.ko 2024-02-21T22:50:14.736 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/emc1403.ko 2024-02-21T22:50:14.736 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/emc6w201.ko 2024-02-21T22:50:14.737 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/f71805f.ko 2024-02-21T22:50:14.737 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/f71882fg.ko 2024-02-21T22:50:14.737 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/f75375s.ko 2024-02-21T22:50:14.737 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/fam15h_power.ko 2024-02-21T22:50:14.737 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/fschmd.ko 2024-02-21T22:50:14.737 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/g760a.ko 2024-02-21T22:50:14.738 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/g762.ko 2024-02-21T22:50:14.738 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/gl518sm.ko 2024-02-21T22:50:14.738 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/gl520sm.ko 2024-02-21T22:50:14.738 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/hwmon-vid.ko 2024-02-21T22:50:14.738 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/i5500_temp.ko 2024-02-21T22:50:14.738 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/i5k_amb.ko 2024-02-21T22:50:14.739 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/ibmaem.ko 2024-02-21T22:50:14.739 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/ibmpex.ko 2024-02-21T22:50:14.739 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/ina209.ko 2024-02-21T22:50:14.739 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/ina2xx.ko 2024-02-21T22:50:14.739 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/it87.ko 2024-02-21T22:50:14.739 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/k10temp.ko 2024-02-21T22:50:14.740 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/k8temp.ko 2024-02-21T22:50:14.740 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/lineage-pem.ko 2024-02-21T22:50:14.740 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/lm63.ko 2024-02-21T22:50:14.740 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/lm73.ko 2024-02-21T22:50:14.740 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/lm75.ko 2024-02-21T22:50:14.740 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/lm77.ko 2024-02-21T22:50:14.741 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/lm78.ko 2024-02-21T22:50:14.741 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/lm80.ko 2024-02-21T22:50:14.741 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/lm83.ko 2024-02-21T22:50:14.741 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/lm85.ko 2024-02-21T22:50:14.741 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/lm87.ko 2024-02-21T22:50:14.742 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/lm90.ko 2024-02-21T22:50:14.742 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/lm92.ko 2024-02-21T22:50:14.742 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/lm93.ko 2024-02-21T22:50:14.742 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/lm95234.ko 2024-02-21T22:50:14.742 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/lm95241.ko 2024-02-21T22:50:14.742 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/lm95245.ko 2024-02-21T22:50:14.742 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/ltc2945.ko 2024-02-21T22:50:14.743 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/ltc4151.ko 2024-02-21T22:50:14.743 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/ltc4215.ko 2024-02-21T22:50:14.743 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/ltc4222.ko 2024-02-21T22:50:14.743 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/ltc4245.ko 2024-02-21T22:50:14.743 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/ltc4260.ko 2024-02-21T22:50:14.743 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/ltc4261.ko 2024-02-21T22:50:14.743 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/max16065.ko 2024-02-21T22:50:14.743 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/max1619.ko 2024-02-21T22:50:14.743 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/max1668.ko 2024-02-21T22:50:14.744 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/max197.ko 2024-02-21T22:50:14.744 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/max6639.ko 2024-02-21T22:50:14.744 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/max6650.ko 2024-02-21T22:50:14.744 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/max6697.ko 2024-02-21T22:50:14.744 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/mcp3021.ko 2024-02-21T22:50:14.744 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/nct6683.ko 2024-02-21T22:50:14.744 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/nct6775-core.ko 2024-02-21T22:50:14.744 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/nct6775.ko 2024-02-21T22:50:14.744 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/nct7802.ko 2024-02-21T22:50:14.744 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/pc87360.ko 2024-02-21T22:50:14.744 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/pc87427.ko 2024-02-21T22:50:14.744 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/pcf8591.ko 2024-02-21T22:50:14.744 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/pmbus 2024-02-21T22:50:14.744 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-02-21T22:50:14.744 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-02-21T22:50:14.745 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-02-21T22:50:14.745 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/pmbus/max16064.ko 2024-02-21T22:50:14.745 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/pmbus/max34440.ko 2024-02-21T22:50:14.745 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/pmbus/max8688.ko 2024-02-21T22:50:14.745 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-02-21T22:50:14.745 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-02-21T22:50:14.745 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-02-21T22:50:14.745 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-02-21T22:50:14.745 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-02-21T22:50:14.745 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-02-21T22:50:14.745 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/powr1220.ko 2024-02-21T22:50:14.745 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/sch5627.ko 2024-02-21T22:50:14.745 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/sch5636.ko 2024-02-21T22:50:14.745 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/sch56xx-common.ko 2024-02-21T22:50:14.746 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/sht15.ko 2024-02-21T22:50:14.746 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/sht21.ko 2024-02-21T22:50:14.746 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/shtc1.ko 2024-02-21T22:50:14.746 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/sis5595.ko 2024-02-21T22:50:14.746 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/smsc47b397.ko 2024-02-21T22:50:14.746 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/smsc47m1.ko 2024-02-21T22:50:14.746 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/smsc47m192.ko 2024-02-21T22:50:14.746 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/thmc50.ko 2024-02-21T22:50:14.746 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/tmp102.ko 2024-02-21T22:50:14.746 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/tmp103.ko 2024-02-21T22:50:14.746 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/tmp401.ko 2024-02-21T22:50:14.746 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/tmp421.ko 2024-02-21T22:50:14.746 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/via-cputemp.ko 2024-02-21T22:50:14.747 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/via686a.ko 2024-02-21T22:50:14.747 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/vt1211.ko 2024-02-21T22:50:14.747 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/vt8231.ko 2024-02-21T22:50:14.747 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/w83627ehf.ko 2024-02-21T22:50:14.747 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/w83627hf.ko 2024-02-21T22:50:14.747 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/w83781d.ko 2024-02-21T22:50:14.747 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/w83791d.ko 2024-02-21T22:50:14.747 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/w83792d.ko 2024-02-21T22:50:14.747 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/w83793.ko 2024-02-21T22:50:14.748 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/w83795.ko 2024-02-21T22:50:14.748 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/w83l785ts.ko 2024-02-21T22:50:14.748 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/w83l786ng.ko 2024-02-21T22:50:14.748 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/i2c 2024-02-21T22:50:14.748 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/i2c/algos 2024-02-21T22:50:14.748 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-02-21T22:50:14.748 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-02-21T22:50:14.748 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/i2c/busses 2024-02-21T22:50:14.748 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-02-21T22:50:14.748 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-02-21T22:50:14.748 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-02-21T22:50:14.749 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-02-21T22:50:14.749 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-02-21T22:50:14.749 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-02-21T22:50:14.749 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-02-21T22:50:14.749 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-02-21T22:50:14.749 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/i2c/busses/i2c-i801.ko 2024-02-21T22:50:14.749 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/i2c/busses/i2c-isch.ko 2024-02-21T22:50:14.749 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-02-21T22:50:14.749 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-02-21T22:50:14.749 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-02-21T22:50:14.750 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/i2c/busses/i2c-parport.ko 2024-02-21T22:50:14.750 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-02-21T22:50:14.750 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-02-21T22:50:14.750 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-02-21T22:50:14.750 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-02-21T22:50:14.750 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-02-21T22:50:14.750 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-02-21T22:50:14.750 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/i2c/busses/i2c-via.ko 2024-02-21T22:50:14.750 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-02-21T22:50:14.750 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-02-21T22:50:14.750 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/i2c/i2c-dev.ko 2024-02-21T22:50:14.750 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/i2c/i2c-mux.ko 2024-02-21T22:50:14.750 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-02-21T22:50:14.750 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/i2c/i2c-smbus.ko 2024-02-21T22:50:14.750 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/i2c/i2c-stub.ko 2024-02-21T22:50:14.751 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/infiniband 2024-02-21T22:50:14.751 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/infiniband/core 2024-02-21T22:50:14.751 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/infiniband/core/ib_cm.ko 2024-02-21T22:50:14.751 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/infiniband/core/ib_core.ko 2024-02-21T22:50:14.751 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/infiniband/core/ib_umad.ko 2024-02-21T22:50:14.751 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-02-21T22:50:14.751 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/infiniband/core/iw_cm.ko 2024-02-21T22:50:14.751 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/infiniband/core/rdma_cm.ko 2024-02-21T22:50:14.751 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-02-21T22:50:14.751 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/infiniband/hw 2024-02-21T22:50:14.751 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/infiniband/hw/cxgb4 2024-02-21T22:50:14.751 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-02-21T22:50:14.751 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/infiniband/hw/mlx4 2024-02-21T22:50:14.751 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-02-21T22:50:14.751 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/infiniband/hw/mlx5 2024-02-21T22:50:14.752 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-02-21T22:50:14.752 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/infiniband/hw/mthca 2024-02-21T22:50:14.752 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-02-21T22:50:14.752 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/infiniband/ulp 2024-02-21T22:50:14.752 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/infiniband/ulp/ipoib 2024-02-21T22:50:14.752 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-02-21T22:50:14.752 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/infiniband/ulp/iser 2024-02-21T22:50:14.752 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-02-21T22:50:14.752 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/infiniband/ulp/isert 2024-02-21T22:50:14.752 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-02-21T22:50:14.752 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/infiniband/ulp/srp 2024-02-21T22:50:14.752 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-02-21T22:50:14.752 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/infiniband/ulp/srpt 2024-02-21T22:50:14.752 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-02-21T22:50:14.753 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/input 2024-02-21T22:50:14.753 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/input/gameport 2024-02-21T22:50:14.753 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-02-21T22:50:14.753 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/input/gameport/fm801-gp.ko 2024-02-21T22:50:14.753 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/input/gameport/gameport.ko 2024-02-21T22:50:14.753 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/input/joydev.ko 2024-02-21T22:50:14.753 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/input/keyboard 2024-02-21T22:50:14.753 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/input/keyboard/gpio_keys.ko 2024-02-21T22:50:14.753 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/input/mouse 2024-02-21T22:50:14.753 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/input/mouse/appletouch.ko 2024-02-21T22:50:14.754 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/input/mouse/bcm5974.ko 2024-02-21T22:50:14.754 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/input/mouse/cyapatp.ko 2024-02-21T22:50:14.754 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/input/mouse/elan_i2c.ko 2024-02-21T22:50:14.754 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/input/mouse/sermouse.ko 2024-02-21T22:50:14.754 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-02-21T22:50:14.754 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/input/mouse/synaptics_usb.ko 2024-02-21T22:50:14.754 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/input/mouse/vsxxxaa.ko 2024-02-21T22:50:14.754 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/input/rmi4 2024-02-21T22:50:14.754 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/input/rmi4/rmi_core.ko 2024-02-21T22:50:14.754 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/input/serio 2024-02-21T22:50:14.754 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/input/serio/altera_ps2.ko 2024-02-21T22:50:14.754 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/input/serio/arc_ps2.ko 2024-02-21T22:50:14.755 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-02-21T22:50:14.755 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/input/serio/serio_raw.ko 2024-02-21T22:50:14.755 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/input/sparse-keymap.ko 2024-02-21T22:50:14.755 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/leds 2024-02-21T22:50:14.755 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/leds/leds-blinkm.ko 2024-02-21T22:50:14.755 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/leds/leds-lm3530.ko 2024-02-21T22:50:14.755 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/leds/leds-lp3944.ko 2024-02-21T22:50:14.755 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/leds/leds-ss4200.ko 2024-02-21T22:50:14.756 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/leds/trigger 2024-02-21T22:50:14.756 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-02-21T22:50:14.756 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-02-21T22:50:14.756 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-02-21T22:50:14.756 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-02-21T22:50:14.756 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-02-21T22:50:14.756 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-02-21T22:50:14.756 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-02-21T22:50:14.756 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-02-21T22:50:14.756 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-02-21T22:50:14.756 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/md 2024-02-21T22:50:14.756 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/md/bcache 2024-02-21T22:50:14.756 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/md/bcache/bcache.ko 2024-02-21T22:50:14.756 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/md/dm-bio-prison.ko 2024-02-21T22:50:14.757 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/md/dm-cache-smq.ko 2024-02-21T22:50:14.757 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/md/dm-cache.ko 2024-02-21T22:50:14.757 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/md/dm-crypt.ko 2024-02-21T22:50:14.757 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/md/dm-delay.ko 2024-02-21T22:50:14.757 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/md/dm-flakey.ko 2024-02-21T22:50:14.757 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/md/dm-log-userspace.ko 2024-02-21T22:50:14.757 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/md/dm-multipath.ko 2024-02-21T22:50:14.757 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/md/dm-queue-length.ko 2024-02-21T22:50:14.757 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/md/dm-raid.ko 2024-02-21T22:50:14.757 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/md/dm-round-robin.ko 2024-02-21T22:50:14.757 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/md/dm-service-time.ko 2024-02-21T22:50:14.757 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/md/dm-switch.ko 2024-02-21T22:50:14.757 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/md/dm-thin-pool.ko 2024-02-21T22:50:14.757 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/md/dm-verity.ko 2024-02-21T22:50:14.758 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/md/persistent-data 2024-02-21T22:50:14.758 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-02-21T22:50:14.758 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/md/raid0.ko 2024-02-21T22:50:14.758 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/md/raid1.ko 2024-02-21T22:50:14.758 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/md/raid10.ko 2024-02-21T22:50:14.758 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/md/raid456.ko 2024-02-21T22:50:14.758 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/message 2024-02-21T22:50:14.758 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/message/fusion 2024-02-21T22:50:14.758 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/message/fusion/mptbase.ko 2024-02-21T22:50:14.758 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/message/fusion/mptctl.ko 2024-02-21T22:50:14.758 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/message/fusion/mptfc.ko 2024-02-21T22:50:14.758 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/message/fusion/mptlan.ko 2024-02-21T22:50:14.758 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/message/fusion/mptsas.ko 2024-02-21T22:50:14.758 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/message/fusion/mptscsih.ko 2024-02-21T22:50:14.759 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/message/fusion/mptspi.ko 2024-02-21T22:50:14.759 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/mfd 2024-02-21T22:50:14.759 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/mfd/lpc_ich.ko 2024-02-21T22:50:14.759 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/mfd/lpc_sch.ko 2024-02-21T22:50:14.759 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/mfd/mfd-core.ko 2024-02-21T22:50:14.759 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/mfd/sm501.ko 2024-02-21T22:50:14.759 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/mfd/viperboard.ko 2024-02-21T22:50:14.759 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/mfd/vx855.ko 2024-02-21T22:50:14.759 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/mfd/wl1273-core.ko 2024-02-21T22:50:14.759 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/misc 2024-02-21T22:50:14.759 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/misc/altera-stapl 2024-02-21T22:50:14.759 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-02-21T22:50:14.759 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/misc/apds9802als.ko 2024-02-21T22:50:14.759 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/misc/apds990x.ko 2024-02-21T22:50:14.760 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/misc/bh1770glc.ko 2024-02-21T22:50:14.760 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/misc/cb710 2024-02-21T22:50:14.760 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/misc/cb710/cb710.ko 2024-02-21T22:50:14.760 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/misc/eeprom 2024-02-21T22:50:14.760 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/misc/eeprom/at24.ko 2024-02-21T22:50:14.760 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-02-21T22:50:14.760 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/misc/eeprom/max6875.ko 2024-02-21T22:50:14.760 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/misc/enclosure.ko 2024-02-21T22:50:14.760 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/misc/hpilo.ko 2024-02-21T22:50:14.760 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/misc/isl29003.ko 2024-02-21T22:50:14.760 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/misc/isl29020.ko 2024-02-21T22:50:14.760 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/misc/lis3lv02d 2024-02-21T22:50:14.760 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-02-21T22:50:14.760 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-02-21T22:50:14.761 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/misc/mei 2024-02-21T22:50:14.761 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/misc/mei/mei-me.ko 2024-02-21T22:50:14.761 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/misc/mei/mei-txe.ko 2024-02-21T22:50:14.761 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/misc/mei/mei.ko 2024-02-21T22:50:14.761 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/misc/sgi-gru 2024-02-21T22:50:14.761 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/misc/sgi-gru/gru.ko 2024-02-21T22:50:14.761 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/misc/sgi-xp 2024-02-21T22:50:14.761 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/misc/sgi-xp/xp.ko 2024-02-21T22:50:14.761 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/misc/sgi-xp/xpc.ko 2024-02-21T22:50:14.761 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-02-21T22:50:14.761 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/misc/tifm_7xx1.ko 2024-02-21T22:50:14.761 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/misc/tifm_core.ko 2024-02-21T22:50:14.761 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/misc/tsl2550.ko 2024-02-21T22:50:14.761 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/misc/vmw_balloon.ko 2024-02-21T22:50:14.761 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/misc/vmw_vmci 2024-02-21T22:50:14.762 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-02-21T22:50:14.762 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net 2024-02-21T22:50:14.762 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/bonding 2024-02-21T22:50:14.762 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/bonding/bonding.ko 2024-02-21T22:50:14.762 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/dsa 2024-02-21T22:50:14.762 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/dsa/b53 2024-02-21T22:50:14.762 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/dsa/b53/b53_common.ko 2024-02-21T22:50:14.762 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/dsa/bcm-sf2.ko 2024-02-21T22:50:14.762 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/dsa/mv88e6060.ko 2024-02-21T22:50:14.762 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/dsa/mv88e6xxx 2024-02-21T22:50:14.762 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-02-21T22:50:14.762 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/dummy.ko 2024-02-21T22:50:14.762 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/eql.ko 2024-02-21T22:50:14.762 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet 2024-02-21T22:50:14.762 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/3com 2024-02-21T22:50:14.763 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-02-21T22:50:14.763 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-02-21T22:50:14.763 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-02-21T22:50:14.763 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-02-21T22:50:14.763 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/8390 2024-02-21T22:50:14.763 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/8390/8390.ko 2024-02-21T22:50:14.763 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-02-21T22:50:14.763 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-02-21T22:50:14.763 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-02-21T22:50:14.763 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/adaptec 2024-02-21T22:50:14.763 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-02-21T22:50:14.763 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/agere 2024-02-21T22:50:14.763 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/agere/et131x.ko 2024-02-21T22:50:14.763 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/alteon 2024-02-21T22:50:14.764 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-02-21T22:50:14.764 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/altera 2024-02-21T22:50:14.764 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-02-21T22:50:14.764 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/amd 2024-02-21T22:50:14.764 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-02-21T22:50:14.764 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-02-21T22:50:14.764 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-02-21T22:50:14.764 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/atheros 2024-02-21T22:50:14.764 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/atheros/alx 2024-02-21T22:50:14.764 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-02-21T22:50:14.764 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/atheros/atl1c 2024-02-21T22:50:14.764 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-02-21T22:50:14.764 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/atheros/atl1e 2024-02-21T22:50:14.764 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-02-21T22:50:14.764 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/atheros/atlx 2024-02-21T22:50:14.765 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-02-21T22:50:14.765 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-02-21T22:50:14.765 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/broadcom 2024-02-21T22:50:14.765 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-02-21T22:50:14.765 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-02-21T22:50:14.765 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-02-21T22:50:14.765 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-02-21T22:50:14.765 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-02-21T22:50:14.765 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/broadcom/genet 2024-02-21T22:50:14.765 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-02-21T22:50:14.765 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-02-21T22:50:14.765 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/brocade 2024-02-21T22:50:14.765 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/brocade/bna 2024-02-21T22:50:14.765 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-02-21T22:50:14.766 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/chelsio 2024-02-21T22:50:14.766 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/chelsio/cxgb 2024-02-21T22:50:14.766 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-02-21T22:50:14.766 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-02-21T22:50:14.766 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-02-21T22:50:14.766 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-02-21T22:50:14.766 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-02-21T22:50:14.766 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-02-21T22:50:14.766 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-02-21T22:50:14.766 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-02-21T22:50:14.766 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-02-21T22:50:14.766 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/cisco 2024-02-21T22:50:14.766 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/cisco/enic 2024-02-21T22:50:14.766 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-02-21T22:50:14.767 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/dec 2024-02-21T22:50:14.767 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/dec/tulip 2024-02-21T22:50:14.767 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-02-21T22:50:14.767 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-02-21T22:50:14.767 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-02-21T22:50:14.767 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-02-21T22:50:14.767 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-02-21T22:50:14.767 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-02-21T22:50:14.767 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/dlink 2024-02-21T22:50:14.767 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-02-21T22:50:14.767 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-02-21T22:50:14.767 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/dnet.ko 2024-02-21T22:50:14.767 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/emulex 2024-02-21T22:50:14.767 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/emulex/benet 2024-02-21T22:50:14.768 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-02-21T22:50:14.768 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/ethoc.ko 2024-02-21T22:50:14.768 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/fealnx.ko 2024-02-21T22:50:14.768 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/intel 2024-02-21T22:50:14.768 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/intel/e100.ko 2024-02-21T22:50:14.768 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/intel/e1000 2024-02-21T22:50:14.768 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-02-21T22:50:14.768 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/intel/e1000e 2024-02-21T22:50:14.768 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-02-21T22:50:14.768 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/intel/fm10k 2024-02-21T22:50:14.768 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-02-21T22:50:14.768 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/intel/i40e 2024-02-21T22:50:14.768 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-02-21T22:50:14.768 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/intel/iavf 2024-02-21T22:50:14.769 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-02-21T22:50:14.769 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/intel/igb 2024-02-21T22:50:14.769 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-02-21T22:50:14.769 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/intel/igbvf 2024-02-21T22:50:14.769 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-02-21T22:50:14.769 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/intel/ixgbe 2024-02-21T22:50:14.769 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-02-21T22:50:14.769 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/intel/ixgbevf 2024-02-21T22:50:14.769 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-02-21T22:50:14.769 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/jme.ko 2024-02-21T22:50:14.769 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/marvell 2024-02-21T22:50:14.769 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-02-21T22:50:14.769 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/marvell/skge.ko 2024-02-21T22:50:14.769 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-02-21T22:50:14.770 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/mellanox 2024-02-21T22:50:14.770 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/mellanox/mlx4 2024-02-21T22:50:14.770 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-02-21T22:50:14.770 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-02-21T22:50:14.770 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/mellanox/mlx5 2024-02-21T22:50:14.770 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-02-21T22:50:14.770 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-02-21T22:50:14.770 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/micrel 2024-02-21T22:50:14.770 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-02-21T22:50:14.770 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/myricom 2024-02-21T22:50:14.770 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/myricom/myri10ge 2024-02-21T22:50:14.770 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-02-21T22:50:14.770 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/natsemi 2024-02-21T22:50:14.770 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-02-21T22:50:14.770 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-02-21T22:50:14.771 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/neterion 2024-02-21T22:50:14.771 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-02-21T22:50:14.771 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/nvidia 2024-02-21T22:50:14.771 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-02-21T22:50:14.771 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/packetengines 2024-02-21T22:50:14.771 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-02-21T22:50:14.771 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-02-21T22:50:14.771 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/qlogic 2024-02-21T22:50:14.771 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/qlogic/netxen 2024-02-21T22:50:14.771 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-02-21T22:50:14.771 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-02-21T22:50:14.771 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-02-21T22:50:14.771 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-02-21T22:50:14.771 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/rdc 2024-02-21T22:50:14.772 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-02-21T22:50:14.772 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/realtek 2024-02-21T22:50:14.772 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-02-21T22:50:14.772 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-02-21T22:50:14.772 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/realtek/atp.ko 2024-02-21T22:50:14.772 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-02-21T22:50:14.772 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/rocker 2024-02-21T22:50:14.772 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-02-21T22:50:14.772 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/sfc 2024-02-21T22:50:14.772 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-02-21T22:50:14.772 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/silan 2024-02-21T22:50:14.772 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-02-21T22:50:14.772 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/sis 2024-02-21T22:50:14.772 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/sis/sis190.ko 2024-02-21T22:50:14.773 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/sis/sis900.ko 2024-02-21T22:50:14.773 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/smsc 2024-02-21T22:50:14.773 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-02-21T22:50:14.773 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-02-21T22:50:14.773 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-02-21T22:50:14.774 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-02-21T22:50:14.774 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/stmicro 2024-02-21T22:50:14.774 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/stmicro/stmmac 2024-02-21T22:50:14.774 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-02-21T22:50:14.774 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-02-21T22:50:14.774 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/sun 2024-02-21T22:50:14.774 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/sun/cassini.ko 2024-02-21T22:50:14.774 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/sun/niu.ko 2024-02-21T22:50:14.774 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/sun/sungem.ko 2024-02-21T22:50:14.774 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-02-21T22:50:14.774 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/tehuti 2024-02-21T22:50:14.775 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-02-21T22:50:14.775 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/ti 2024-02-21T22:50:14.775 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/ti/tlan.ko 2024-02-21T22:50:14.775 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/via 2024-02-21T22:50:14.775 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-02-21T22:50:14.775 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-02-21T22:50:14.775 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/wiznet 2024-02-21T22:50:14.775 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-02-21T22:50:14.775 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-02-21T22:50:14.775 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/xircom 2024-02-21T22:50:14.775 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-02-21T22:50:14.775 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/geneve.ko 2024-02-21T22:50:14.775 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/hyperv 2024-02-21T22:50:14.775 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-02-21T22:50:14.776 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ieee802154 2024-02-21T22:50:14.776 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ieee802154/fakelb.ko 2024-02-21T22:50:14.776 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ifb.ko 2024-02-21T22:50:14.776 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ipvlan 2024-02-21T22:50:14.776 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ipvlan/ipvlan.ko 2024-02-21T22:50:14.776 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/macvlan.ko 2024-02-21T22:50:14.776 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/macvtap.ko 2024-02-21T22:50:14.776 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/mdio 2024-02-21T22:50:14.776 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/mdio.ko 2024-02-21T22:50:14.776 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-02-21T22:50:14.777 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-02-21T22:50:14.777 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/mdio/mdio-regmap.ko 2024-02-21T22:50:14.777 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/mii.ko 2024-02-21T22:50:14.777 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/net_failover.ko 2024-02-21T22:50:14.777 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/netconsole.ko 2024-02-21T22:50:14.777 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/nlmon.ko 2024-02-21T22:50:14.777 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/pcs 2024-02-21T22:50:14.777 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/pcs/pcs-lynx.ko 2024-02-21T22:50:14.777 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-02-21T22:50:14.777 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/phy 2024-02-21T22:50:14.777 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/phy/amd.ko 2024-02-21T22:50:14.777 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-02-21T22:50:14.777 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-02-21T22:50:14.777 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/phy/bcm7xxx.ko 2024-02-21T22:50:14.778 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/phy/bcm87xx.ko 2024-02-21T22:50:14.778 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/phy/broadcom.ko 2024-02-21T22:50:14.778 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/phy/cicada.ko 2024-02-21T22:50:14.778 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/phy/davicom.ko 2024-02-21T22:50:14.778 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/phy/dp83640.ko 2024-02-21T22:50:14.778 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/phy/et1011c.ko 2024-02-21T22:50:14.778 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/phy/icplus.ko 2024-02-21T22:50:14.778 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/phy/lxt.ko 2024-02-21T22:50:14.778 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/phy/marvell.ko 2024-02-21T22:50:14.778 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/phy/micrel.ko 2024-02-21T22:50:14.778 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/phy/national.ko 2024-02-21T22:50:14.778 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/phy/phylink.ko 2024-02-21T22:50:14.778 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/phy/qsemi.ko 2024-02-21T22:50:14.778 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/phy/realtek.ko 2024-02-21T22:50:14.779 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/phy/smsc.ko 2024-02-21T22:50:14.779 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/phy/ste10Xp.ko 2024-02-21T22:50:14.779 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/phy/vitesse.ko 2024-02-21T22:50:14.779 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ppp 2024-02-21T22:50:14.779 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ppp/bsd_comp.ko 2024-02-21T22:50:14.779 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ppp/ppp_async.ko 2024-02-21T22:50:14.779 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ppp/ppp_deflate.ko 2024-02-21T22:50:14.779 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ppp/ppp_generic.ko 2024-02-21T22:50:14.779 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ppp/ppp_mppe.ko 2024-02-21T22:50:14.779 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ppp/ppp_synctty.ko 2024-02-21T22:50:14.779 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ppp/pppoe.ko 2024-02-21T22:50:14.779 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ppp/pppox.ko 2024-02-21T22:50:14.779 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ppp/pptp.ko 2024-02-21T22:50:14.779 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/slip 2024-02-21T22:50:14.779 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/slip/slhc.ko 2024-02-21T22:50:14.780 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/slip/slip.ko 2024-02-21T22:50:14.780 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/sungem_phy.ko 2024-02-21T22:50:14.780 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/tap.ko 2024-02-21T22:50:14.780 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/team 2024-02-21T22:50:14.780 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/team/team.ko 2024-02-21T22:50:14.780 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/team/team_mode_activebackup.ko 2024-02-21T22:50:14.780 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/team/team_mode_broadcast.ko 2024-02-21T22:50:14.780 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-02-21T22:50:14.780 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/team/team_mode_random.ko 2024-02-21T22:50:14.780 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-02-21T22:50:14.780 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/tun.ko 2024-02-21T22:50:14.780 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/veth.ko 2024-02-21T22:50:14.780 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/virtio_net.ko 2024-02-21T22:50:14.781 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/vmxnet3 2024-02-21T22:50:14.781 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-02-21T22:50:14.781 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/vxlan 2024-02-21T22:50:14.781 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/vxlan/vxlan.ko 2024-02-21T22:50:14.781 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/xen-netback 2024-02-21T22:50:14.781 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/xen-netback/xen-netback.ko 2024-02-21T22:50:14.781 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/xen-netfront.ko 2024-02-21T22:50:14.781 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ntb 2024-02-21T22:50:14.781 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ntb/ntb.ko 2024-02-21T22:50:14.781 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/nvme 2024-02-21T22:50:14.781 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/nvme/host 2024-02-21T22:50:14.781 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/nvme/host/nvme-core.ko 2024-02-21T22:50:14.781 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/nvme/host/nvme.ko 2024-02-21T22:50:14.781 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/parport 2024-02-21T22:50:14.782 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/parport/parport.ko 2024-02-21T22:50:14.782 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/parport/parport_cs.ko 2024-02-21T22:50:14.782 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/parport/parport_pc.ko 2024-02-21T22:50:14.782 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/parport/parport_serial.ko 2024-02-21T22:50:14.782 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/pci 2024-02-21T22:50:14.782 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/pci/hotplug 2024-02-21T22:50:14.782 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-02-21T22:50:14.782 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/pci/pcie 2024-02-21T22:50:14.782 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/pci/pcie/aer_inject.ko 2024-02-21T22:50:14.782 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/pci/xen-pcifront.ko 2024-02-21T22:50:14.782 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/pcmcia 2024-02-21T22:50:14.782 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/pcmcia/i82092.ko 2024-02-21T22:50:14.782 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/pcmcia/pd6729.ko 2024-02-21T22:50:14.782 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/pcmcia/yenta_socket.ko 2024-02-21T22:50:14.783 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/pinctrl 2024-02-21T22:50:14.783 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/pinctrl/intel 2024-02-21T22:50:14.783 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-02-21T22:50:14.783 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/platform 2024-02-21T22:50:14.783 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/platform/x86 2024-02-21T22:50:14.783 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/platform/x86/acer-wmi.ko 2024-02-21T22:50:14.783 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/platform/x86/acerhdf.ko 2024-02-21T22:50:14.783 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/platform/x86/apple-gmux.ko 2024-02-21T22:50:14.783 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/platform/x86/asus-laptop.ko 2024-02-21T22:50:14.783 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-02-21T22:50:14.783 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/platform/x86/asus-wmi.ko 2024-02-21T22:50:14.783 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/platform/x86/classmate-laptop.ko 2024-02-21T22:50:14.783 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-02-21T22:50:14.783 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-02-21T22:50:14.783 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-02-21T22:50:14.784 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-02-21T22:50:14.784 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/platform/x86/hdaps.ko 2024-02-21T22:50:14.784 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/platform/x86/intel 2024-02-21T22:50:14.784 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-02-21T22:50:14.784 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/platform/x86/intel_ips.ko 2024-02-21T22:50:14.784 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/platform/x86/msi-wmi.ko 2024-02-21T22:50:14.784 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/platform/x86/mxm-wmi.ko 2024-02-21T22:50:14.784 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-02-21T22:50:14.784 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/platform/x86/samsung-laptop.ko 2024-02-21T22:50:14.784 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/platform/x86/samsung-q10.ko 2024-02-21T22:50:14.784 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/platform/x86/topstar-laptop.ko 2024-02-21T22:50:14.784 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-02-21T22:50:14.784 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/platform/x86/toshiba_haps.ko 2024-02-21T22:50:14.784 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/platform/x86/wmi-bmof.ko 2024-02-21T22:50:14.785 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/platform/x86/wmi.ko 2024-02-21T22:50:14.785 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/powercap 2024-02-21T22:50:14.785 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/powercap/intel_rapl_common.ko 2024-02-21T22:50:14.785 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/powercap/intel_rapl_msr.ko 2024-02-21T22:50:14.785 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/pps 2024-02-21T22:50:14.785 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/pps/clients 2024-02-21T22:50:14.785 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/pps/clients/pps-gpio.ko 2024-02-21T22:50:14.785 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/pps/clients/pps-ldisc.ko 2024-02-21T22:50:14.785 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/pps/clients/pps_parport.ko 2024-02-21T22:50:14.785 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/pps/pps_core.ko 2024-02-21T22:50:14.785 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ptp 2024-02-21T22:50:14.785 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ptp/ptp.ko 2024-02-21T22:50:14.785 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ptp/ptp_kvm.ko 2024-02-21T22:50:14.785 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/rtc 2024-02-21T22:50:14.786 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/rtc/rtc-bq32k.ko 2024-02-21T22:50:14.786 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/rtc/rtc-ds1286.ko 2024-02-21T22:50:14.786 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/rtc/rtc-ds1307.ko 2024-02-21T22:50:14.786 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/rtc/rtc-ds1374.ko 2024-02-21T22:50:14.786 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/rtc/rtc-ds1511.ko 2024-02-21T22:50:14.786 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/rtc/rtc-ds1553.ko 2024-02-21T22:50:14.786 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/rtc/rtc-ds1672.ko 2024-02-21T22:50:14.786 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/rtc/rtc-ds1742.ko 2024-02-21T22:50:14.786 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/rtc/rtc-ds2404.ko 2024-02-21T22:50:14.786 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/rtc/rtc-ds3232.ko 2024-02-21T22:50:14.786 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/rtc/rtc-em3027.ko 2024-02-21T22:50:14.786 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/rtc/rtc-fm3130.ko 2024-02-21T22:50:14.786 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/rtc/rtc-isl12022.ko 2024-02-21T22:50:14.786 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/rtc/rtc-isl1208.ko 2024-02-21T22:50:14.786 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/rtc/rtc-m41t80.ko 2024-02-21T22:50:14.787 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/rtc/rtc-m48t35.ko 2024-02-21T22:50:14.787 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/rtc/rtc-m48t59.ko 2024-02-21T22:50:14.787 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/rtc/rtc-max6900.ko 2024-02-21T22:50:14.787 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/rtc/rtc-msm6242.ko 2024-02-21T22:50:14.787 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/rtc/rtc-pcf2127.ko 2024-02-21T22:50:14.787 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/rtc/rtc-pcf85063.ko 2024-02-21T22:50:14.787 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/rtc/rtc-pcf8523.ko 2024-02-21T22:50:14.787 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/rtc/rtc-pcf8563.ko 2024-02-21T22:50:14.787 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/rtc/rtc-pcf8583.ko 2024-02-21T22:50:14.787 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/rtc/rtc-rp5c01.ko 2024-02-21T22:50:14.787 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/rtc/rtc-rs5c372.ko 2024-02-21T22:50:14.787 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-02-21T22:50:14.787 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/rtc/rtc-rx8025.ko 2024-02-21T22:50:14.787 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/rtc/rtc-rx8581.ko 2024-02-21T22:50:14.787 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-02-21T22:50:14.788 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/rtc/rtc-x1205.ko 2024-02-21T22:50:14.788 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi 2024-02-21T22:50:14.788 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/3w-9xxx.ko 2024-02-21T22:50:14.788 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/3w-sas.ko 2024-02-21T22:50:14.788 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/3w-xxxx.ko 2024-02-21T22:50:14.788 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/BusLogic.ko 2024-02-21T22:50:14.788 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/a100u2w.ko 2024-02-21T22:50:14.788 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/aacraid 2024-02-21T22:50:14.788 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/aacraid/aacraid.ko 2024-02-21T22:50:14.788 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/advansys.ko 2024-02-21T22:50:14.788 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/aic7xxx 2024-02-21T22:50:14.788 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-02-21T22:50:14.788 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-02-21T22:50:14.788 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/am53c974.ko 2024-02-21T22:50:14.789 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/arcmsr 2024-02-21T22:50:14.789 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-02-21T22:50:14.789 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/atp870u.ko 2024-02-21T22:50:14.789 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/be2iscsi 2024-02-21T22:50:14.789 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-02-21T22:50:14.789 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/bfa 2024-02-21T22:50:14.789 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/bfa/bfa.ko 2024-02-21T22:50:14.789 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/bnx2fc 2024-02-21T22:50:14.789 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-02-21T22:50:14.789 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/bnx2i 2024-02-21T22:50:14.789 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-02-21T22:50:14.789 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/ch.ko 2024-02-21T22:50:14.789 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/csiostor 2024-02-21T22:50:14.789 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/csiostor/csiostor.ko 2024-02-21T22:50:14.789 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/cxgbi 2024-02-21T22:50:14.790 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/cxgbi/cxgb3i 2024-02-21T22:50:14.790 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-02-21T22:50:14.790 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/cxgbi/cxgb4i 2024-02-21T22:50:14.790 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-02-21T22:50:14.790 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-02-21T22:50:14.790 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/dc395x.ko 2024-02-21T22:50:14.790 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/device_handler 2024-02-21T22:50:14.790 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-02-21T22:50:14.790 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-02-21T22:50:14.790 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-02-21T22:50:14.790 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-02-21T22:50:14.790 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/dmx3191d.ko 2024-02-21T22:50:14.790 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/esas2r 2024-02-21T22:50:14.790 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/esas2r/esas2r.ko 2024-02-21T22:50:14.791 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/esp_scsi.ko 2024-02-21T22:50:14.791 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/fcoe 2024-02-21T22:50:14.791 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/fcoe/fcoe.ko 2024-02-21T22:50:14.791 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-02-21T22:50:14.791 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/fnic 2024-02-21T22:50:14.791 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/fnic/fnic.ko 2024-02-21T22:50:14.791 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/hpsa.ko 2024-02-21T22:50:14.791 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/hptiop.ko 2024-02-21T22:50:14.791 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/hv_storvsc.ko 2024-02-21T22:50:14.791 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/initio.ko 2024-02-21T22:50:14.791 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/ipr.ko 2024-02-21T22:50:14.791 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/ips.ko 2024-02-21T22:50:14.791 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/isci 2024-02-21T22:50:14.791 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/isci/isci.ko 2024-02-21T22:50:14.791 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-02-21T22:50:14.792 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/iscsi_tcp.ko 2024-02-21T22:50:14.792 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/libfc 2024-02-21T22:50:14.792 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/libfc/libfc.ko 2024-02-21T22:50:14.792 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/libiscsi.ko 2024-02-21T22:50:14.792 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/libiscsi_tcp.ko 2024-02-21T22:50:14.792 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/libsas 2024-02-21T22:50:14.792 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/libsas/libsas.ko 2024-02-21T22:50:14.792 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/lpfc 2024-02-21T22:50:14.792 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/lpfc/lpfc.ko 2024-02-21T22:50:14.792 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/megaraid 2024-02-21T22:50:14.792 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/megaraid.ko 2024-02-21T22:50:14.792 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-02-21T22:50:14.792 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-02-21T22:50:14.792 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-02-21T22:50:14.793 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/mpt3sas 2024-02-21T22:50:14.793 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-02-21T22:50:14.793 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/mvsas 2024-02-21T22:50:14.793 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/mvsas/mvsas.ko 2024-02-21T22:50:14.793 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/mvumi.ko 2024-02-21T22:50:14.793 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/pm8001 2024-02-21T22:50:14.793 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-02-21T22:50:14.793 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/pmcraid.ko 2024-02-21T22:50:14.793 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/qla1280.ko 2024-02-21T22:50:14.793 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/qla2xxx 2024-02-21T22:50:14.793 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-02-21T22:50:14.793 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-02-21T22:50:14.793 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/qla4xxx 2024-02-21T22:50:14.793 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-02-21T22:50:14.793 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/raid_class.ko 2024-02-21T22:50:14.794 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/scsi_debug.ko 2024-02-21T22:50:14.794 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/scsi_transport_fc.ko 2024-02-21T22:50:14.794 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-02-21T22:50:14.794 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/scsi_transport_sas.ko 2024-02-21T22:50:14.794 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/scsi_transport_spi.ko 2024-02-21T22:50:14.794 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/scsi_transport_srp.ko 2024-02-21T22:50:14.794 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/ses.ko 2024-02-21T22:50:14.794 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/st.ko 2024-02-21T22:50:14.794 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/stex.ko 2024-02-21T22:50:14.794 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/sym53c8xx_2 2024-02-21T22:50:14.794 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-02-21T22:50:14.794 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/virtio_scsi.ko 2024-02-21T22:50:14.794 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/vmw_pvscsi.ko 2024-02-21T22:50:14.794 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/wd719x.ko 2024-02-21T22:50:14.795 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ssb 2024-02-21T22:50:14.795 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ssb/ssb.ko 2024-02-21T22:50:14.795 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/target 2024-02-21T22:50:14.795 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/target/iscsi 2024-02-21T22:50:14.795 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-02-21T22:50:14.795 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/target/loopback 2024-02-21T22:50:14.795 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/target/loopback/tcm_loop.ko 2024-02-21T22:50:14.795 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/target/target_core_file.ko 2024-02-21T22:50:14.795 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/target/target_core_iblock.ko 2024-02-21T22:50:14.795 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/target/target_core_mod.ko 2024-02-21T22:50:14.795 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/target/target_core_pscsi.ko 2024-02-21T22:50:14.795 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/target/target_core_user.ko 2024-02-21T22:50:14.795 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/target/tcm_fc 2024-02-21T22:50:14.795 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-02-21T22:50:14.796 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/thermal 2024-02-21T22:50:14.796 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/thermal/intel 2024-02-21T22:50:14.796 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/thermal/intel/int340x_thermal 2024-02-21T22:50:14.796 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-02-21T22:50:14.796 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-02-21T22:50:14.796 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-02-21T22:50:14.796 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-02-21T22:50:14.796 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-02-21T22:50:14.796 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-02-21T22:50:14.796 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-02-21T22:50:14.796 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-02-21T22:50:14.796 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-02-21T22:50:14.796 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-02-21T22:50:14.797 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-02-21T22:50:14.797 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-02-21T22:50:14.797 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-02-21T22:50:14.797 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-02-21T22:50:14.797 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-02-21T22:50:14.797 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-02-21T22:50:14.797 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-02-21T22:50:14.797 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-02-21T22:50:14.797 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/tty 2024-02-21T22:50:14.797 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/tty/ipwireless 2024-02-21T22:50:14.797 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-02-21T22:50:14.797 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/tty/n_gsm.ko 2024-02-21T22:50:14.797 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/tty/n_hdlc.ko 2024-02-21T22:50:14.797 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/tty/nozomi.ko 2024-02-21T22:50:14.798 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/tty/serial 2024-02-21T22:50:14.798 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/tty/serial/8250 2024-02-21T22:50:14.798 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-02-21T22:50:14.798 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/tty/serial/arc_uart.ko 2024-02-21T22:50:14.798 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/tty/serial/jsm 2024-02-21T22:50:14.798 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/tty/serial/jsm/jsm.ko 2024-02-21T22:50:14.798 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ufs 2024-02-21T22:50:14.798 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ufs/core 2024-02-21T22:50:14.798 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ufs/core/ufshcd-core.ko 2024-02-21T22:50:14.798 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ufs/host 2024-02-21T22:50:14.798 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-02-21T22:50:14.798 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/uio 2024-02-21T22:50:14.798 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/uio/uio.ko 2024-02-21T22:50:14.798 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/uio/uio_aec.ko 2024-02-21T22:50:14.798 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/uio/uio_cif.ko 2024-02-21T22:50:14.799 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/uio/uio_pci_generic.ko 2024-02-21T22:50:14.799 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/uio/uio_sercos3.ko 2024-02-21T22:50:14.799 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb 2024-02-21T22:50:14.799 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/class 2024-02-21T22:50:14.799 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/class/cdc-acm.ko 2024-02-21T22:50:14.799 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/class/cdc-wdm.ko 2024-02-21T22:50:14.799 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/class/usblp.ko 2024-02-21T22:50:14.799 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/class/usbtmc.ko 2024-02-21T22:50:14.799 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/host 2024-02-21T22:50:14.799 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/host/sl811-hcd.ko 2024-02-21T22:50:14.799 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/image 2024-02-21T22:50:14.799 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/image/mdc800.ko 2024-02-21T22:50:14.799 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/image/microtek.ko 2024-02-21T22:50:14.799 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/misc 2024-02-21T22:50:14.800 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/misc/adutux.ko 2024-02-21T22:50:14.800 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/misc/appledisplay.ko 2024-02-21T22:50:14.800 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/misc/emi26.ko 2024-02-21T22:50:14.800 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/misc/emi62.ko 2024-02-21T22:50:14.800 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/misc/ezusb.ko 2024-02-21T22:50:14.800 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/misc/idmouse.ko 2024-02-21T22:50:14.800 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/misc/iowarrior.ko 2024-02-21T22:50:14.800 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/misc/isight_firmware.ko 2024-02-21T22:50:14.800 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/misc/ldusb.ko 2024-02-21T22:50:14.800 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/misc/legousbtower.ko 2024-02-21T22:50:14.800 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/misc/sisusbvga 2024-02-21T22:50:14.800 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-02-21T22:50:14.800 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/misc/trancevibrator.ko 2024-02-21T22:50:14.800 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/misc/usb3503.ko 2024-02-21T22:50:14.800 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/misc/usblcd.ko 2024-02-21T22:50:14.801 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/misc/usbsevseg.ko 2024-02-21T22:50:14.801 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/misc/uss720.ko 2024-02-21T22:50:14.801 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/misc/yurex.ko 2024-02-21T22:50:14.801 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/phy 2024-02-21T22:50:14.801 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/phy/phy-generic.ko 2024-02-21T22:50:14.801 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial 2024-02-21T22:50:14.801 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/aircable.ko 2024-02-21T22:50:14.801 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/ark3116.ko 2024-02-21T22:50:14.801 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/belkin_sa.ko 2024-02-21T22:50:14.801 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/ch341.ko 2024-02-21T22:50:14.801 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/cp210x.ko 2024-02-21T22:50:14.801 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/cyberjack.ko 2024-02-21T22:50:14.801 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/cypress_m8.ko 2024-02-21T22:50:14.801 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/digi_acceleport.ko 2024-02-21T22:50:14.802 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/empeg.ko 2024-02-21T22:50:14.802 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/ftdi_sio.ko 2024-02-21T22:50:14.802 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/garmin_gps.ko 2024-02-21T22:50:14.802 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/io_edgeport.ko 2024-02-21T22:50:14.802 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/io_ti.ko 2024-02-21T22:50:14.802 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/ipaq.ko 2024-02-21T22:50:14.802 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/ipw.ko 2024-02-21T22:50:14.802 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/ir-usb.ko 2024-02-21T22:50:14.802 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-02-21T22:50:14.802 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/keyspan.ko 2024-02-21T22:50:14.802 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/keyspan_pda.ko 2024-02-21T22:50:14.802 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/kl5kusb105.ko 2024-02-21T22:50:14.802 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/kobil_sct.ko 2024-02-21T22:50:14.802 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/mct_u232.ko 2024-02-21T22:50:14.802 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/mos7720.ko 2024-02-21T22:50:14.803 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/mos7840.ko 2024-02-21T22:50:14.803 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/navman.ko 2024-02-21T22:50:14.803 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/omninet.ko 2024-02-21T22:50:14.803 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/opticon.ko 2024-02-21T22:50:14.803 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/option.ko 2024-02-21T22:50:14.803 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/oti6858.ko 2024-02-21T22:50:14.803 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/pl2303.ko 2024-02-21T22:50:14.803 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/qcaux.ko 2024-02-21T22:50:14.803 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/qcserial.ko 2024-02-21T22:50:14.803 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/quatech2.ko 2024-02-21T22:50:14.803 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/safe_serial.ko 2024-02-21T22:50:14.803 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/sierra.ko 2024-02-21T22:50:14.803 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/spcp8x5.ko 2024-02-21T22:50:14.803 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/ssu100.ko 2024-02-21T22:50:14.803 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/symbolserial.ko 2024-02-21T22:50:14.804 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-02-21T22:50:14.804 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-02-21T22:50:14.804 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/usb_debug.ko 2024-02-21T22:50:14.804 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/usb_wwan.ko 2024-02-21T22:50:14.804 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/visor.ko 2024-02-21T22:50:14.804 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/whiteheat.ko 2024-02-21T22:50:14.804 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/xsens_mt.ko 2024-02-21T22:50:14.804 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/storage 2024-02-21T22:50:14.804 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/storage/uas.ko 2024-02-21T22:50:14.804 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/storage/ums-alauda.ko 2024-02-21T22:50:14.804 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/storage/ums-cypress.ko 2024-02-21T22:50:14.804 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/storage/ums-datafab.ko 2024-02-21T22:50:14.804 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-02-21T22:50:14.804 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/storage/ums-freecom.ko 2024-02-21T22:50:14.805 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/storage/ums-isd200.ko 2024-02-21T22:50:14.805 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-02-21T22:50:14.805 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/storage/ums-karma.ko 2024-02-21T22:50:14.805 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/storage/ums-onetouch.ko 2024-02-21T22:50:14.805 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/storage/ums-realtek.ko 2024-02-21T22:50:14.805 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/storage/ums-sddr09.ko 2024-02-21T22:50:14.805 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/storage/ums-sddr55.ko 2024-02-21T22:50:14.805 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/storage/ums-usbat.ko 2024-02-21T22:50:14.805 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/storage/usb-storage.ko 2024-02-21T22:50:14.805 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/usbip 2024-02-21T22:50:14.805 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/usbip/usbip-core.ko 2024-02-21T22:50:14.805 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/usbip/usbip-host.ko 2024-02-21T22:50:14.805 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-02-21T22:50:14.805 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/vfio 2024-02-21T22:50:14.805 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/vfio/pci 2024-02-21T22:50:14.806 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-02-21T22:50:14.806 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/vfio/pci/vfio-pci.ko 2024-02-21T22:50:14.806 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/vfio/vfio.ko 2024-02-21T22:50:14.806 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-02-21T22:50:14.806 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/vhost 2024-02-21T22:50:14.806 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/vhost/vhost.ko 2024-02-21T22:50:14.806 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/vhost/vhost_iotlb.ko 2024-02-21T22:50:14.806 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/vhost/vhost_net.ko 2024-02-21T22:50:14.806 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/vhost/vhost_scsi.ko 2024-02-21T22:50:14.807 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/video 2024-02-21T22:50:14.807 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/video/backlight 2024-02-21T22:50:14.807 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/video/backlight/apple_bl.ko 2024-02-21T22:50:14.807 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/video/backlight/lcd.ko 2024-02-21T22:50:14.807 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/video/backlight/platform_lcd.ko 2024-02-21T22:50:14.807 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/virtio 2024-02-21T22:50:14.807 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/virtio/virtio.ko 2024-02-21T22:50:14.807 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/virtio/virtio_balloon.ko 2024-02-21T22:50:14.807 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/virtio/virtio_mmio.ko 2024-02-21T22:50:14.807 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/virtio/virtio_pci.ko 2024-02-21T22:50:14.807 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-02-21T22:50:14.807 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-02-21T22:50:14.807 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/virtio/virtio_ring.ko 2024-02-21T22:50:14.807 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/w1 2024-02-21T22:50:14.807 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/w1/masters 2024-02-21T22:50:14.808 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/w1/masters/ds2482.ko 2024-02-21T22:50:14.808 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/w1/masters/ds2490.ko 2024-02-21T22:50:14.808 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/w1/slaves 2024-02-21T22:50:14.808 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-02-21T22:50:14.808 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-02-21T22:50:14.808 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-02-21T22:50:14.808 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-02-21T22:50:14.808 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-02-21T22:50:14.808 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-02-21T22:50:14.808 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-02-21T22:50:14.808 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-02-21T22:50:14.808 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-02-21T22:50:14.808 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/w1/slaves/w1_smem.ko 2024-02-21T22:50:14.808 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/w1/slaves/w1_therm.ko 2024-02-21T22:50:14.809 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/w1/wire.ko 2024-02-21T22:50:14.809 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/watchdog 2024-02-21T22:50:14.809 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/watchdog/alim1535_wdt.ko 2024-02-21T22:50:14.809 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/watchdog/alim7101_wdt.ko 2024-02-21T22:50:14.809 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/watchdog/f71808e_wdt.ko 2024-02-21T22:50:14.809 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/watchdog/hpwdt.ko 2024-02-21T22:50:14.809 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/watchdog/i6300esb.ko 2024-02-21T22:50:14.809 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-02-21T22:50:14.809 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/watchdog/iTCO_wdt.ko 2024-02-21T22:50:14.809 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/watchdog/ib700wdt.ko 2024-02-21T22:50:14.809 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/watchdog/ibmasr.ko 2024-02-21T22:50:14.809 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-02-21T22:50:14.809 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/watchdog/it8712f_wdt.ko 2024-02-21T22:50:14.809 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/watchdog/it87_wdt.ko 2024-02-21T22:50:14.810 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/watchdog/machzwd.ko 2024-02-21T22:50:14.810 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/watchdog/nv_tco.ko 2024-02-21T22:50:14.810 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/watchdog/pcwd_pci.ko 2024-02-21T22:50:14.810 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/watchdog/pcwd_usb.ko 2024-02-21T22:50:14.810 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-02-21T22:50:14.810 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/watchdog/sch311x_wdt.ko 2024-02-21T22:50:14.810 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/watchdog/softdog.ko 2024-02-21T22:50:14.810 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/watchdog/sp5100_tco.ko 2024-02-21T22:50:14.810 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/watchdog/via_wdt.ko 2024-02-21T22:50:14.810 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-02-21T22:50:14.810 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/watchdog/w83877f_wdt.ko 2024-02-21T22:50:14.810 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/watchdog/w83977f_wdt.ko 2024-02-21T22:50:14.810 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/watchdog/wdt_pci.ko 2024-02-21T22:50:14.810 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/watchdog/xen_wdt.ko 2024-02-21T22:50:14.811 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/xen 2024-02-21T22:50:14.811 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/xen/xen-acpi-processor.ko 2024-02-21T22:50:14.811 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/xen/xen-evtchn.ko 2024-02-21T22:50:14.811 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/xen/xen-gntalloc.ko 2024-02-21T22:50:14.811 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/xen/xen-gntdev.ko 2024-02-21T22:50:14.811 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/xen/xen-pciback 2024-02-21T22:50:14.811 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-02-21T22:50:14.811 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/xen/xen-privcmd.ko 2024-02-21T22:50:14.811 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/xen/xenfs 2024-02-21T22:50:14.811 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/xen/xenfs/xenfs.ko 2024-02-21T22:50:14.811 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs 2024-02-21T22:50:14.811 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/9p 2024-02-21T22:50:14.811 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/9p/9p.ko 2024-02-21T22:50:14.811 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/affs 2024-02-21T22:50:14.812 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/affs/affs.ko 2024-02-21T22:50:14.812 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/befs 2024-02-21T22:50:14.812 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/befs/befs.ko 2024-02-21T22:50:14.812 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/binfmt_misc.ko 2024-02-21T22:50:14.812 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/btrfs 2024-02-21T22:50:14.812 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/btrfs/btrfs.ko 2024-02-21T22:50:14.812 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/cachefiles 2024-02-21T22:50:14.812 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/cachefiles/cachefiles.ko 2024-02-21T22:50:14.812 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/ceph 2024-02-21T22:50:14.812 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/ceph/ceph.ko 2024-02-21T22:50:14.812 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/coda 2024-02-21T22:50:14.812 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/coda/coda.ko 2024-02-21T22:50:14.812 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/cramfs 2024-02-21T22:50:14.812 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/cramfs/cramfs.ko 2024-02-21T22:50:14.813 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/dlm 2024-02-21T22:50:14.813 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/dlm/dlm.ko 2024-02-21T22:50:14.813 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/ecryptfs 2024-02-21T22:50:14.813 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/ecryptfs/ecryptfs.ko 2024-02-21T22:50:14.813 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/f2fs 2024-02-21T22:50:14.813 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/f2fs/f2fs.ko 2024-02-21T22:50:14.813 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/fat 2024-02-21T22:50:14.813 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/fat/fat.ko 2024-02-21T22:50:14.813 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/fat/msdos.ko 2024-02-21T22:50:14.813 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/fat/vfat.ko 2024-02-21T22:50:14.813 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/fuse 2024-02-21T22:50:14.813 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/fuse/cuse.ko 2024-02-21T22:50:14.813 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/fuse/fuse.ko 2024-02-21T22:50:14.813 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/gfs2 2024-02-21T22:50:14.813 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/gfs2/gfs2.ko 2024-02-21T22:50:14.814 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/hfs 2024-02-21T22:50:14.814 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/hfs/hfs.ko 2024-02-21T22:50:14.814 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/hfsplus 2024-02-21T22:50:14.814 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/hfsplus/hfsplus.ko 2024-02-21T22:50:14.814 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/isofs 2024-02-21T22:50:14.814 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/isofs/isofs.ko 2024-02-21T22:50:14.814 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/jfs 2024-02-21T22:50:14.814 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/jfs/jfs.ko 2024-02-21T22:50:14.814 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/lockd 2024-02-21T22:50:14.814 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/lockd/lockd.ko 2024-02-21T22:50:14.814 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/minix 2024-02-21T22:50:14.814 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/minix/minix.ko 2024-02-21T22:50:14.814 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/netfs 2024-02-21T22:50:14.815 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/netfs/netfs.ko 2024-02-21T22:50:14.815 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nfs 2024-02-21T22:50:14.815 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nfs/blocklayout 2024-02-21T22:50:14.815 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-02-21T22:50:14.815 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nfs/filelayout 2024-02-21T22:50:14.815 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-02-21T22:50:14.815 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nfs/flexfilelayout 2024-02-21T22:50:14.815 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-02-21T22:50:14.815 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nfs/nfs.ko 2024-02-21T22:50:14.815 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nfs/nfsv3.ko 2024-02-21T22:50:14.815 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nfs/nfsv4.ko 2024-02-21T22:50:14.815 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nfs_common 2024-02-21T22:50:14.815 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nfs_common/grace.ko 2024-02-21T22:50:14.815 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nfs_common/nfs_acl.ko 2024-02-21T22:50:14.815 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nfsd 2024-02-21T22:50:14.816 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nfsd/nfsd.ko 2024-02-21T22:50:14.816 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nilfs2 2024-02-21T22:50:14.816 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nilfs2/nilfs2.ko 2024-02-21T22:50:14.816 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls 2024-02-21T22:50:14.816 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/mac-celtic.ko 2024-02-21T22:50:14.816 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/mac-centeuro.ko 2024-02-21T22:50:14.816 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/mac-croatian.ko 2024-02-21T22:50:14.816 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/mac-cyrillic.ko 2024-02-21T22:50:14.816 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/mac-gaelic.ko 2024-02-21T22:50:14.816 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/mac-greek.ko 2024-02-21T22:50:14.816 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/mac-iceland.ko 2024-02-21T22:50:14.816 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/mac-inuit.ko 2024-02-21T22:50:14.816 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/mac-roman.ko 2024-02-21T22:50:14.816 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/mac-romanian.ko 2024-02-21T22:50:14.817 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/mac-turkish.ko 2024-02-21T22:50:14.817 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_cp1250.ko 2024-02-21T22:50:14.817 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_cp1251.ko 2024-02-21T22:50:14.817 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_cp1255.ko 2024-02-21T22:50:14.817 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_cp737.ko 2024-02-21T22:50:14.817 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_cp775.ko 2024-02-21T22:50:14.817 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_cp850.ko 2024-02-21T22:50:14.817 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_cp852.ko 2024-02-21T22:50:14.817 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_cp855.ko 2024-02-21T22:50:14.817 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_cp857.ko 2024-02-21T22:50:14.817 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_cp860.ko 2024-02-21T22:50:14.817 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_cp861.ko 2024-02-21T22:50:14.817 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_cp862.ko 2024-02-21T22:50:14.817 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_cp863.ko 2024-02-21T22:50:14.818 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_cp864.ko 2024-02-21T22:50:14.818 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_cp865.ko 2024-02-21T22:50:14.818 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_cp866.ko 2024-02-21T22:50:14.818 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_cp869.ko 2024-02-21T22:50:14.818 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_cp874.ko 2024-02-21T22:50:14.818 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_cp932.ko 2024-02-21T22:50:14.818 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_cp936.ko 2024-02-21T22:50:14.818 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_cp949.ko 2024-02-21T22:50:14.818 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_cp950.ko 2024-02-21T22:50:14.818 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_euc-jp.ko 2024-02-21T22:50:14.818 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_iso8859-1.ko 2024-02-21T22:50:14.818 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_iso8859-13.ko 2024-02-21T22:50:14.818 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_iso8859-14.ko 2024-02-21T22:50:14.818 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_iso8859-15.ko 2024-02-21T22:50:14.818 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_iso8859-2.ko 2024-02-21T22:50:14.819 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_iso8859-3.ko 2024-02-21T22:50:14.819 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_iso8859-4.ko 2024-02-21T22:50:14.819 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_iso8859-5.ko 2024-02-21T22:50:14.819 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_iso8859-6.ko 2024-02-21T22:50:14.819 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_iso8859-7.ko 2024-02-21T22:50:14.819 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_iso8859-9.ko 2024-02-21T22:50:14.819 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_koi8-r.ko 2024-02-21T22:50:14.819 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_koi8-ru.ko 2024-02-21T22:50:14.819 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_koi8-u.ko 2024-02-21T22:50:14.819 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_ucs2_utils.ko 2024-02-21T22:50:14.819 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_utf8.ko 2024-02-21T22:50:14.819 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/ocfs2 2024-02-21T22:50:14.819 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/ocfs2/cluster 2024-02-21T22:50:14.819 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-02-21T22:50:14.820 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/ocfs2/dlm 2024-02-21T22:50:14.820 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-02-21T22:50:14.820 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/ocfs2/dlmfs 2024-02-21T22:50:14.820 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-02-21T22:50:14.820 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/ocfs2/ocfs2.ko 2024-02-21T22:50:14.820 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-02-21T22:50:14.820 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-02-21T22:50:14.820 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-02-21T22:50:14.820 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/overlayfs 2024-02-21T22:50:14.820 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/overlayfs/overlay.ko 2024-02-21T22:50:14.820 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/pstore 2024-02-21T22:50:14.820 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/pstore/ramoops.ko 2024-02-21T22:50:14.820 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/reiserfs 2024-02-21T22:50:14.820 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/reiserfs/reiserfs.ko 2024-02-21T22:50:14.821 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/romfs 2024-02-21T22:50:14.821 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/romfs/romfs.ko 2024-02-21T22:50:14.821 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/smb 2024-02-21T22:50:14.821 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/smb/client 2024-02-21T22:50:14.821 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/smb/client/cifs.ko 2024-02-21T22:50:14.821 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/smb/common 2024-02-21T22:50:14.821 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/smb/common/cifs_arc4.ko 2024-02-21T22:50:14.821 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/smb/common/cifs_md4.ko 2024-02-21T22:50:14.821 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/squashfs 2024-02-21T22:50:14.821 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/squashfs/squashfs.ko 2024-02-21T22:50:14.821 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/sysv 2024-02-21T22:50:14.821 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/sysv/sysv.ko 2024-02-21T22:50:14.821 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/udf 2024-02-21T22:50:14.821 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/udf/udf.ko 2024-02-21T22:50:14.821 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/ufs 2024-02-21T22:50:14.822 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/ufs/ufs.ko 2024-02-21T22:50:14.822 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/xfs 2024-02-21T22:50:14.822 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/xfs/xfs.ko 2024-02-21T22:50:14.822 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/kernel 2024-02-21T22:50:14.822 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/kernel/rcu 2024-02-21T22:50:14.822 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/kernel/rcu/rcutorture.ko 2024-02-21T22:50:14.822 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/kernel/torture.ko 2024-02-21T22:50:14.822 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/kernel/trace 2024-02-21T22:50:14.822 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-02-21T22:50:14.822 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/lib 2024-02-21T22:50:14.822 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/lib/asn1_encoder.ko 2024-02-21T22:50:14.822 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/lib/crc-itu-t.ko 2024-02-21T22:50:14.822 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/lib/crc8.ko 2024-02-21T22:50:14.822 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/lib/crypto 2024-02-21T22:50:14.822 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/lib/crypto/libarc4.ko 2024-02-21T22:50:14.823 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/lib/crypto/libdes.ko 2024-02-21T22:50:14.823 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/lib/libcrc32c.ko 2024-02-21T22:50:14.823 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/lib/lru_cache.ko 2024-02-21T22:50:14.823 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/lib/lz4 2024-02-21T22:50:14.823 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/lib/lz4/lz4_compress.ko 2024-02-21T22:50:14.823 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/lib/lz4/lz4hc_compress.ko 2024-02-21T22:50:14.823 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/lib/math 2024-02-21T22:50:14.823 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/lib/math/cordic.ko 2024-02-21T22:50:14.823 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/lib/raid6 2024-02-21T22:50:14.823 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/lib/raid6/raid6_pq.ko 2024-02-21T22:50:14.823 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/lib/reed_solomon 2024-02-21T22:50:14.823 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/lib/reed_solomon/reed_solomon.ko 2024-02-21T22:50:14.823 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/lib/ts_bm.ko 2024-02-21T22:50:14.824 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/lib/ts_fsm.ko 2024-02-21T22:50:14.824 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/lib/ts_kmp.ko 2024-02-21T22:50:14.824 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/lib/zstd 2024-02-21T22:50:14.824 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/lib/zstd/zstd_compress.ko 2024-02-21T22:50:14.824 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/mm 2024-02-21T22:50:14.824 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/mm/hwpoison-inject.ko 2024-02-21T22:50:14.824 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net 2024-02-21T22:50:14.824 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/6lowpan 2024-02-21T22:50:14.824 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/6lowpan/6lowpan.ko 2024-02-21T22:50:14.824 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/6lowpan/nhc_dest.ko 2024-02-21T22:50:14.824 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/6lowpan/nhc_fragment.ko 2024-02-21T22:50:14.824 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/6lowpan/nhc_hop.ko 2024-02-21T22:50:14.824 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/6lowpan/nhc_ipv6.ko 2024-02-21T22:50:14.824 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/6lowpan/nhc_mobility.ko 2024-02-21T22:50:14.825 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/6lowpan/nhc_routing.ko 2024-02-21T22:50:14.825 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/6lowpan/nhc_udp.ko 2024-02-21T22:50:14.825 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/802 2024-02-21T22:50:14.825 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/802/garp.ko 2024-02-21T22:50:14.825 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/802/mrp.ko 2024-02-21T22:50:14.825 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/802/p8022.ko 2024-02-21T22:50:14.825 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/802/psnap.ko 2024-02-21T22:50:14.825 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/802/stp.ko 2024-02-21T22:50:14.825 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/8021q 2024-02-21T22:50:14.825 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/8021q/8021q.ko 2024-02-21T22:50:14.825 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/9p 2024-02-21T22:50:14.825 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/9p/9pnet.ko 2024-02-21T22:50:14.825 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/9p/9pnet_fd.ko 2024-02-21T22:50:14.825 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/9p/9pnet_rdma.ko 2024-02-21T22:50:14.825 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/9p/9pnet_virtio.ko 2024-02-21T22:50:14.826 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/batman-adv 2024-02-21T22:50:14.826 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/batman-adv/batman-adv.ko 2024-02-21T22:50:14.826 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/bridge 2024-02-21T22:50:14.826 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/bridge/br_netfilter.ko 2024-02-21T22:50:14.826 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/bridge/bridge.ko 2024-02-21T22:50:14.826 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/bridge/netfilter 2024-02-21T22:50:14.826 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-02-21T22:50:14.826 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/bridge/netfilter/ebt_among.ko 2024-02-21T22:50:14.826 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/bridge/netfilter/ebt_arp.ko 2024-02-21T22:50:14.826 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-02-21T22:50:14.826 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-02-21T22:50:14.826 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/bridge/netfilter/ebt_ip.ko 2024-02-21T22:50:14.826 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-02-21T22:50:14.826 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/bridge/netfilter/ebt_limit.ko 2024-02-21T22:50:14.826 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/bridge/netfilter/ebt_log.ko 2024-02-21T22:50:14.827 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/bridge/netfilter/ebt_mark.ko 2024-02-21T22:50:14.827 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-02-21T22:50:14.827 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-02-21T22:50:14.827 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-02-21T22:50:14.827 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-02-21T22:50:14.827 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/bridge/netfilter/ebt_snat.ko 2024-02-21T22:50:14.827 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/bridge/netfilter/ebt_stp.ko 2024-02-21T22:50:14.827 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-02-21T22:50:14.827 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-02-21T22:50:14.827 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-02-21T22:50:14.827 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-02-21T22:50:14.827 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/bridge/netfilter/ebtables.ko 2024-02-21T22:50:14.827 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-02-21T22:50:14.827 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-02-21T22:50:14.828 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-02-21T22:50:14.828 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ceph 2024-02-21T22:50:14.828 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ceph/libceph.ko 2024-02-21T22:50:14.828 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/core 2024-02-21T22:50:14.828 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/core/failover.ko 2024-02-21T22:50:14.828 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/core/pktgen.ko 2024-02-21T22:50:14.828 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/dccp 2024-02-21T22:50:14.828 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/dccp/dccp.ko 2024-02-21T22:50:14.828 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/dccp/dccp_diag.ko 2024-02-21T22:50:14.828 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/dccp/dccp_ipv4.ko 2024-02-21T22:50:14.828 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/dccp/dccp_ipv6.ko 2024-02-21T22:50:14.828 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/dns_resolver 2024-02-21T22:50:14.828 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/dns_resolver/dns_resolver.ko 2024-02-21T22:50:14.828 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/dsa 2024-02-21T22:50:14.829 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/dsa/dsa_core.ko 2024-02-21T22:50:14.829 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/dsa/tag_brcm.ko 2024-02-21T22:50:14.829 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/dsa/tag_dsa.ko 2024-02-21T22:50:14.829 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/dsa/tag_none.ko 2024-02-21T22:50:14.829 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/dsa/tag_trailer.ko 2024-02-21T22:50:14.829 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ieee802154 2024-02-21T22:50:14.829 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ieee802154/6lowpan 2024-02-21T22:50:14.829 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-02-21T22:50:14.829 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ieee802154/ieee802154.ko 2024-02-21T22:50:14.829 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ieee802154/ieee802154_socket.ko 2024-02-21T22:50:14.829 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4 2024-02-21T22:50:14.829 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/ah4.ko 2024-02-21T22:50:14.829 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/esp4.ko 2024-02-21T22:50:14.829 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/fou.ko 2024-02-21T22:50:14.830 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/gre.ko 2024-02-21T22:50:14.830 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/inet_diag.ko 2024-02-21T22:50:14.830 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/ip_gre.ko 2024-02-21T22:50:14.830 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/ip_tunnel.ko 2024-02-21T22:50:14.830 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/ip_vti.ko 2024-02-21T22:50:14.830 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/ipcomp.ko 2024-02-21T22:50:14.830 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/ipip.ko 2024-02-21T22:50:14.830 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/netfilter 2024-02-21T22:50:14.830 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/netfilter/arp_tables.ko 2024-02-21T22:50:14.830 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-02-21T22:50:14.830 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-02-21T22:50:14.830 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-02-21T22:50:14.830 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-02-21T22:50:14.830 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-02-21T22:50:14.830 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-02-21T22:50:14.831 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-02-21T22:50:14.831 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-02-21T22:50:14.831 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-02-21T22:50:14.831 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/netfilter/iptable_security.ko 2024-02-21T22:50:14.831 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-02-21T22:50:14.831 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-02-21T22:50:14.831 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-02-21T22:50:14.831 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-02-21T22:50:14.831 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-02-21T22:50:14.831 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-02-21T22:50:14.831 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-02-21T22:50:14.831 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-02-21T22:50:14.831 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-02-21T22:50:14.831 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-02-21T22:50:14.832 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/tcp_bic.ko 2024-02-21T22:50:14.832 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/tcp_dctcp.ko 2024-02-21T22:50:14.832 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/tcp_diag.ko 2024-02-21T22:50:14.832 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/tcp_highspeed.ko 2024-02-21T22:50:14.832 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/tcp_htcp.ko 2024-02-21T22:50:14.832 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/tcp_hybla.ko 2024-02-21T22:50:14.832 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/tcp_illinois.ko 2024-02-21T22:50:14.832 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/tcp_lp.ko 2024-02-21T22:50:14.832 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/tcp_scalable.ko 2024-02-21T22:50:14.832 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/tcp_vegas.ko 2024-02-21T22:50:14.832 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/tcp_veno.ko 2024-02-21T22:50:14.832 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/tcp_westwood.ko 2024-02-21T22:50:14.832 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/tcp_yeah.ko 2024-02-21T22:50:14.832 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/tunnel4.ko 2024-02-21T22:50:14.833 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/udp_diag.ko 2024-02-21T22:50:14.833 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/udp_tunnel.ko 2024-02-21T22:50:14.833 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/xfrm4_tunnel.ko 2024-02-21T22:50:14.833 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6 2024-02-21T22:50:14.833 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6/ah6.ko 2024-02-21T22:50:14.833 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6/esp6.ko 2024-02-21T22:50:14.833 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6/fou6.ko 2024-02-21T22:50:14.833 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6/ip6_tunnel.ko 2024-02-21T22:50:14.833 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-02-21T22:50:14.833 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6/ip6_vti.ko 2024-02-21T22:50:14.833 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6/ipcomp6.ko 2024-02-21T22:50:14.833 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6/netfilter 2024-02-21T22:50:14.833 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-02-21T22:50:14.833 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-02-21T22:50:14.834 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-02-21T22:50:14.834 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-02-21T22:50:14.834 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-02-21T22:50:14.834 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-02-21T22:50:14.834 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-02-21T22:50:14.834 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-02-21T22:50:14.834 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-02-21T22:50:14.834 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-02-21T22:50:14.834 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-02-21T22:50:14.834 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-02-21T22:50:14.835 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-02-21T22:50:14.835 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-02-21T22:50:14.835 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-02-21T22:50:14.835 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-02-21T22:50:14.835 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-02-21T22:50:14.835 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-02-21T22:50:14.835 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-02-21T22:50:14.835 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-02-21T22:50:14.835 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-02-21T22:50:14.835 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-02-21T22:50:14.835 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-02-21T22:50:14.835 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-02-21T22:50:14.835 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6/sit.ko 2024-02-21T22:50:14.835 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6/tunnel6.ko 2024-02-21T22:50:14.836 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6/xfrm6_tunnel.ko 2024-02-21T22:50:14.836 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/key 2024-02-21T22:50:14.836 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/key/af_key.ko 2024-02-21T22:50:14.836 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/l2tp 2024-02-21T22:50:14.836 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/l2tp/l2tp_core.ko 2024-02-21T22:50:14.836 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/l2tp/l2tp_debugfs.ko 2024-02-21T22:50:14.836 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/l2tp/l2tp_eth.ko 2024-02-21T22:50:14.836 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/l2tp/l2tp_ip.ko 2024-02-21T22:50:14.836 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/l2tp/l2tp_ip6.ko 2024-02-21T22:50:14.836 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/l2tp/l2tp_netlink.ko 2024-02-21T22:50:14.836 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/l2tp/l2tp_ppp.ko 2024-02-21T22:50:14.836 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/llc 2024-02-21T22:50:14.836 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/llc/llc.ko 2024-02-21T22:50:14.836 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/mac802154 2024-02-21T22:50:14.837 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/mac802154/mac802154.ko 2024-02-21T22:50:14.837 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/mpls 2024-02-21T22:50:14.837 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/mpls/mpls_gso.ko 2024-02-21T22:50:14.837 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter 2024-02-21T22:50:14.837 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/ipset 2024-02-21T22:50:14.837 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/ipset/ip_set.ko 2024-02-21T22:50:14.837 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-02-21T22:50:14.837 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-02-21T22:50:14.837 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-02-21T22:50:14.837 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-02-21T22:50:14.837 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-02-21T22:50:14.837 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-02-21T22:50:14.837 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-02-21T22:50:14.837 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-02-21T22:50:14.838 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-02-21T22:50:14.838 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-02-21T22:50:14.838 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-02-21T22:50:14.838 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-02-21T22:50:14.838 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-02-21T22:50:14.838 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-02-21T22:50:14.838 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-02-21T22:50:14.838 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/ipvs 2024-02-21T22:50:14.838 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/ipvs/ip_vs.ko 2024-02-21T22:50:14.838 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-02-21T22:50:14.838 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-02-21T22:50:14.838 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-02-21T22:50:14.838 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-02-21T22:50:14.838 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-02-21T22:50:14.838 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-02-21T22:50:14.839 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-02-21T22:50:14.839 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-02-21T22:50:14.839 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-02-21T22:50:14.839 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-02-21T22:50:14.839 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-02-21T22:50:14.839 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-02-21T22:50:14.839 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-02-21T22:50:14.839 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nf_conncount.ko 2024-02-21T22:50:14.839 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nf_conntrack.ko 2024-02-21T22:50:14.839 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-02-21T22:50:14.839 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-02-21T22:50:14.839 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-02-21T22:50:14.839 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nf_conntrack_h323.ko 2024-02-21T22:50:14.839 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nf_conntrack_irc.ko 2024-02-21T22:50:14.840 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-02-21T22:50:14.840 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-02-21T22:50:14.840 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-02-21T22:50:14.840 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nf_conntrack_sane.ko 2024-02-21T22:50:14.840 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nf_conntrack_sip.ko 2024-02-21T22:50:14.840 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-02-21T22:50:14.840 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-02-21T22:50:14.840 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nf_dup_netdev.ko 2024-02-21T22:50:14.840 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nf_flow_table.ko 2024-02-21T22:50:14.840 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nf_flow_table_inet.ko 2024-02-21T22:50:14.840 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nf_log_syslog.ko 2024-02-21T22:50:14.840 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nf_nat.ko 2024-02-21T22:50:14.840 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nf_nat_amanda.ko 2024-02-21T22:50:14.841 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nf_nat_ftp.ko 2024-02-21T22:50:14.841 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nf_nat_irc.ko 2024-02-21T22:50:14.841 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nf_nat_sip.ko 2024-02-21T22:50:14.841 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nf_nat_tftp.ko 2024-02-21T22:50:14.841 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nf_synproxy_core.ko 2024-02-21T22:50:14.841 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nf_tables.ko 2024-02-21T22:50:14.841 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nfnetlink.ko 2024-02-21T22:50:14.841 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nfnetlink_acct.ko 2024-02-21T22:50:14.841 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nfnetlink_log.ko 2024-02-21T22:50:14.841 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nfnetlink_osf.ko 2024-02-21T22:50:14.841 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nfnetlink_queue.ko 2024-02-21T22:50:14.841 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nft_chain_nat.ko 2024-02-21T22:50:14.841 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nft_compat.ko 2024-02-21T22:50:14.841 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nft_ct.ko 2024-02-21T22:50:14.841 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nft_dup_netdev.ko 2024-02-21T22:50:14.842 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nft_fib.ko 2024-02-21T22:50:14.842 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nft_fwd_netdev.ko 2024-02-21T22:50:14.842 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nft_hash.ko 2024-02-21T22:50:14.842 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nft_limit.ko 2024-02-21T22:50:14.842 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nft_log.ko 2024-02-21T22:50:14.842 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nft_masq.ko 2024-02-21T22:50:14.842 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nft_nat.ko 2024-02-21T22:50:14.842 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nft_numgen.ko 2024-02-21T22:50:14.842 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nft_queue.ko 2024-02-21T22:50:14.842 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nft_quota.ko 2024-02-21T22:50:14.842 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nft_redir.ko 2024-02-21T22:50:14.842 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nft_reject.ko 2024-02-21T22:50:14.842 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nft_reject_inet.ko 2024-02-21T22:50:14.842 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nft_socket.ko 2024-02-21T22:50:14.843 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nft_tunnel.ko 2024-02-21T22:50:14.843 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_AUDIT.ko 2024-02-21T22:50:14.843 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_CHECKSUM.ko 2024-02-21T22:50:14.843 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_CLASSIFY.ko 2024-02-21T22:50:14.843 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-02-21T22:50:14.843 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_CT.ko 2024-02-21T22:50:14.843 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_DSCP.ko 2024-02-21T22:50:14.843 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_HL.ko 2024-02-21T22:50:14.843 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_HMARK.ko 2024-02-21T22:50:14.843 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_IDLETIMER.ko 2024-02-21T22:50:14.843 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_LED.ko 2024-02-21T22:50:14.843 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_LOG.ko 2024-02-21T22:50:14.843 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_MASQUERADE.ko 2024-02-21T22:50:14.843 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_NETMAP.ko 2024-02-21T22:50:14.844 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_NFLOG.ko 2024-02-21T22:50:14.844 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_NFQUEUE.ko 2024-02-21T22:50:14.844 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_RATEEST.ko 2024-02-21T22:50:14.844 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_REDIRECT.ko 2024-02-21T22:50:14.844 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_SECMARK.ko 2024-02-21T22:50:14.844 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_TCPMSS.ko 2024-02-21T22:50:14.844 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-02-21T22:50:14.844 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_TEE.ko 2024-02-21T22:50:14.844 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_TPROXY.ko 2024-02-21T22:50:14.844 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_TRACE.ko 2024-02-21T22:50:14.844 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_addrtype.ko 2024-02-21T22:50:14.844 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_bpf.ko 2024-02-21T22:50:14.844 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_cgroup.ko 2024-02-21T22:50:14.844 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_cluster.ko 2024-02-21T22:50:14.845 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_comment.ko 2024-02-21T22:50:14.845 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_connbytes.ko 2024-02-21T22:50:14.845 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_connlabel.ko 2024-02-21T22:50:14.845 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_connlimit.ko 2024-02-21T22:50:14.845 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_connmark.ko 2024-02-21T22:50:14.845 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_conntrack.ko 2024-02-21T22:50:14.845 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_cpu.ko 2024-02-21T22:50:14.845 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_dccp.ko 2024-02-21T22:50:14.845 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_devgroup.ko 2024-02-21T22:50:14.845 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_dscp.ko 2024-02-21T22:50:14.845 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_ecn.ko 2024-02-21T22:50:14.845 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_esp.ko 2024-02-21T22:50:14.845 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_hashlimit.ko 2024-02-21T22:50:14.845 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_helper.ko 2024-02-21T22:50:14.846 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_hl.ko 2024-02-21T22:50:14.846 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_ipcomp.ko 2024-02-21T22:50:14.846 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_iprange.ko 2024-02-21T22:50:14.846 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_ipvs.ko 2024-02-21T22:50:14.846 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_l2tp.ko 2024-02-21T22:50:14.846 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_length.ko 2024-02-21T22:50:14.846 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_limit.ko 2024-02-21T22:50:14.846 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_mac.ko 2024-02-21T22:50:14.846 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_mark.ko 2024-02-21T22:50:14.846 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_multiport.ko 2024-02-21T22:50:14.846 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_nat.ko 2024-02-21T22:50:14.846 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_nfacct.ko 2024-02-21T22:50:14.846 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_osf.ko 2024-02-21T22:50:14.846 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_owner.ko 2024-02-21T22:50:14.846 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_physdev.ko 2024-02-21T22:50:14.847 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_pkttype.ko 2024-02-21T22:50:14.847 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_policy.ko 2024-02-21T22:50:14.847 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_quota.ko 2024-02-21T22:50:14.847 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_rateest.ko 2024-02-21T22:50:14.847 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_realm.ko 2024-02-21T22:50:14.847 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_recent.ko 2024-02-21T22:50:14.847 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_sctp.ko 2024-02-21T22:50:14.847 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_set.ko 2024-02-21T22:50:14.847 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_socket.ko 2024-02-21T22:50:14.847 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_state.ko 2024-02-21T22:50:14.847 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_statistic.ko 2024-02-21T22:50:14.847 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_string.ko 2024-02-21T22:50:14.848 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_tcpmss.ko 2024-02-21T22:50:14.848 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_time.ko 2024-02-21T22:50:14.848 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_u32.ko 2024-02-21T22:50:14.848 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netlink 2024-02-21T22:50:14.848 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netlink/netlink_diag.ko 2024-02-21T22:50:14.848 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/nsh 2024-02-21T22:50:14.848 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/nsh/nsh.ko 2024-02-21T22:50:14.848 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/openvswitch 2024-02-21T22:50:14.848 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/openvswitch/openvswitch.ko 2024-02-21T22:50:14.848 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/openvswitch/vport-geneve.ko 2024-02-21T22:50:14.848 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/openvswitch/vport-gre.ko 2024-02-21T22:50:14.848 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/openvswitch/vport-vxlan.ko 2024-02-21T22:50:14.848 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/packet 2024-02-21T22:50:14.848 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/packet/af_packet_diag.ko 2024-02-21T22:50:14.849 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/rds 2024-02-21T22:50:14.849 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/rds/rds.ko 2024-02-21T22:50:14.849 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/rds/rds_rdma.ko 2024-02-21T22:50:14.849 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/rds/rds_tcp.ko 2024-02-21T22:50:14.849 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched 2024-02-21T22:50:14.849 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/act_csum.ko 2024-02-21T22:50:14.849 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/act_gact.ko 2024-02-21T22:50:14.849 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/act_mirred.ko 2024-02-21T22:50:14.849 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/act_nat.ko 2024-02-21T22:50:14.849 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/act_pedit.ko 2024-02-21T22:50:14.849 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/act_police.ko 2024-02-21T22:50:14.849 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/act_simple.ko 2024-02-21T22:50:14.849 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/act_skbedit.ko 2024-02-21T22:50:14.849 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/act_vlan.ko 2024-02-21T22:50:14.849 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/cls_basic.ko 2024-02-21T22:50:14.850 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/cls_bpf.ko 2024-02-21T22:50:14.850 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/cls_flow.ko 2024-02-21T22:50:14.850 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/cls_fw.ko 2024-02-21T22:50:14.850 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/cls_route.ko 2024-02-21T22:50:14.850 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/cls_u32.ko 2024-02-21T22:50:14.850 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/em_cmp.ko 2024-02-21T22:50:14.850 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/em_ipset.ko 2024-02-21T22:50:14.850 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/em_meta.ko 2024-02-21T22:50:14.850 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/em_nbyte.ko 2024-02-21T22:50:14.850 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/em_text.ko 2024-02-21T22:50:14.850 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/em_u32.ko 2024-02-21T22:50:14.850 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/sch_choke.ko 2024-02-21T22:50:14.850 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/sch_codel.ko 2024-02-21T22:50:14.850 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/sch_drr.ko 2024-02-21T22:50:14.851 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/sch_fq.ko 2024-02-21T22:50:14.851 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/sch_gred.ko 2024-02-21T22:50:14.851 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/sch_hfsc.ko 2024-02-21T22:50:14.851 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/sch_hhf.ko 2024-02-21T22:50:14.851 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/sch_htb.ko 2024-02-21T22:50:14.851 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/sch_ingress.ko 2024-02-21T22:50:14.851 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/sch_mqprio.ko 2024-02-21T22:50:14.851 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/sch_mqprio_lib.ko 2024-02-21T22:50:14.851 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/sch_multiq.ko 2024-02-21T22:50:14.851 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/sch_netem.ko 2024-02-21T22:50:14.851 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/sch_pie.ko 2024-02-21T22:50:14.851 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/sch_plug.ko 2024-02-21T22:50:14.851 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/sch_prio.ko 2024-02-21T22:50:14.851 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/sch_qfq.ko 2024-02-21T22:50:14.851 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/sch_red.ko 2024-02-21T22:50:14.852 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/sch_sfb.ko 2024-02-21T22:50:14.852 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/sch_sfq.ko 2024-02-21T22:50:14.852 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/sch_tbf.ko 2024-02-21T22:50:14.852 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/sch_teql.ko 2024-02-21T22:50:14.852 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sctp 2024-02-21T22:50:14.852 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sctp/sctp.ko 2024-02-21T22:50:14.852 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sctp/sctp_diag.ko 2024-02-21T22:50:14.852 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sunrpc 2024-02-21T22:50:14.852 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sunrpc/auth_gss 2024-02-21T22:50:14.852 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-02-21T22:50:14.852 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-02-21T22:50:14.852 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sunrpc/sunrpc.ko 2024-02-21T22:50:14.852 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sunrpc/xprtrdma 2024-02-21T22:50:14.852 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-02-21T22:50:14.853 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/tipc 2024-02-21T22:50:14.853 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/tipc/diag.ko 2024-02-21T22:50:14.853 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/tipc/tipc.ko 2024-02-21T22:50:14.853 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/unix 2024-02-21T22:50:14.853 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/unix/unix_diag.ko 2024-02-21T22:50:14.853 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/vmw_vsock 2024-02-21T22:50:14.853 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-02-21T22:50:14.853 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-02-21T22:50:14.853 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/vmw_vsock/vsock.ko 2024-02-21T22:50:14.853 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/vmw_vsock/vsock_diag.ko 2024-02-21T22:50:14.853 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/vmw_vsock/vsock_loopback.ko 2024-02-21T22:50:14.853 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/xfrm 2024-02-21T22:50:14.853 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/xfrm/xfrm_ipcomp.ko 2024-02-21T22:50:14.853 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/security 2024-02-21T22:50:14.853 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/security/keys 2024-02-21T22:50:14.854 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/security/keys/encrypted-keys 2024-02-21T22:50:14.854 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-02-21T22:50:14.854 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/security/keys/trusted-keys 2024-02-21T22:50:14.854 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/security/keys/trusted-keys/trusted.ko 2024-02-21T22:50:14.854 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/virt 2024-02-21T22:50:14.854 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/virt/lib 2024-02-21T22:50:14.854 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/virt/lib/irqbypass.ko 2024-02-21T22:50:14.854 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/modules.alias 2024-02-21T22:50:14.854 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/modules.alias.bin 2024-02-21T22:50:14.854 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/modules.builtin 2024-02-21T22:50:14.854 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/modules.builtin.alias.bin 2024-02-21T22:50:14.854 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/modules.builtin.bin 2024-02-21T22:50:14.854 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/modules.builtin.modinfo 2024-02-21T22:50:14.854 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/modules.dep 2024-02-21T22:50:14.855 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/modules.dep.bin 2024-02-21T22:50:14.855 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/modules.devname 2024-02-21T22:50:14.855 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/modules.order 2024-02-21T22:50:14.855 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/modules.softdep 2024-02-21T22:50:14.855 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/modules.symbols 2024-02-21T22:50:14.855 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/modules.symbols.bin 2024-02-21T22:50:14.855 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/vmlinuz 2024-02-21T22:50:14.859 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-rc4-g1130e0532280 2024-02-21T22:50:14.859 DEBUG:teuthology.orchestra.run.smithi081:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-02-21T22:50:14.882 INFO:teuthology.orchestra.run.smithi081.stdout:postinstall scriptlet (using /bin/sh): 2024-02-21T22:50:14.882 INFO:teuthology.orchestra.run.smithi081.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-02-21T22:50:14.882 INFO:teuthology.orchestra.run.smithi081.stdout: /usr/bin/kernel-install add 6.8.0-rc4-g1130e0532280 /lib/modules/6.8.0-rc4-g1130e0532280/vmlinuz 2024-02-21T22:50:14.882 INFO:teuthology.orchestra.run.smithi081.stdout:fi 2024-02-21T22:50:14.882 INFO:teuthology.orchestra.run.smithi081.stdout:for file in vmlinuz System.map config; do 2024-02-21T22:50:14.882 INFO:teuthology.orchestra.run.smithi081.stdout: if ! cmp --silent "/lib/modules/6.8.0-rc4-g1130e0532280/${file}" "/boot/${file}-6.8.0-rc4-g1130e0532280"; then 2024-02-21T22:50:14.882 INFO:teuthology.orchestra.run.smithi081.stdout: cp "/lib/modules/6.8.0-rc4-g1130e0532280/${file}" "/boot/${file}-6.8.0-rc4-g1130e0532280" 2024-02-21T22:50:14.883 INFO:teuthology.orchestra.run.smithi081.stdout: fi 2024-02-21T22:50:14.883 INFO:teuthology.orchestra.run.smithi081.stdout:done 2024-02-21T22:50:14.883 INFO:teuthology.orchestra.run.smithi081.stdout:preuninstall scriptlet (using /bin/sh): 2024-02-21T22:50:14.883 INFO:teuthology.orchestra.run.smithi081.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-02-21T22:50:14.883 INFO:teuthology.orchestra.run.smithi081.stdout:new-kernel-pkg --remove 6.8.0-rc4-g1130e0532280 --rminitrd --initrdfile=/boot/initramfs-6.8.0-rc4-g1130e0532280.img 2024-02-21T22:50:14.883 INFO:teuthology.orchestra.run.smithi081.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-02-21T22:50:14.883 INFO:teuthology.orchestra.run.smithi081.stdout:kernel-install remove 6.8.0-rc4-g1130e0532280 2024-02-21T22:50:14.883 INFO:teuthology.orchestra.run.smithi081.stdout:fi 2024-02-21T22:50:14.883 INFO:teuthology.orchestra.run.smithi081.stdout:postuninstall scriptlet (using /bin/sh): 2024-02-21T22:50:14.883 INFO:teuthology.orchestra.run.smithi081.stdout:if [ -x /sbin/update-bootloader ]; then 2024-02-21T22:50:14.883 INFO:teuthology.orchestra.run.smithi081.stdout:/sbin/update-bootloader --remove 6.8.0-rc4-g1130e0532280 2024-02-21T22:50:14.883 INFO:teuthology.orchestra.run.smithi081.stdout:fi 2024-02-21T22:50:14.884 DEBUG:teuthology.orchestra.run.smithi081:> sudo rpm -qi grub2-tools 2024-02-21T22:50:14.961 INFO:teuthology.orchestra.run.smithi081.stdout:Name : grub2-tools 2024-02-21T22:50:14.961 INFO:teuthology.orchestra.run.smithi081.stdout:Epoch : 1 2024-02-21T22:50:14.961 INFO:teuthology.orchestra.run.smithi081.stdout:Version : 2.06 2024-02-21T22:50:14.961 INFO:teuthology.orchestra.run.smithi081.stdout:Release : 68.el9 2024-02-21T22:50:14.961 INFO:teuthology.orchestra.run.smithi081.stdout:Architecture: x86_64 2024-02-21T22:50:14.962 INFO:teuthology.orchestra.run.smithi081.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-21T22:50:14.962 INFO:teuthology.orchestra.run.smithi081.stdout:Group : Unspecified 2024-02-21T22:50:14.962 INFO:teuthology.orchestra.run.smithi081.stdout:Size : 8264254 2024-02-21T22:50:14.962 INFO:teuthology.orchestra.run.smithi081.stdout:License : GPLv3+ 2024-02-21T22:50:14.962 INFO:teuthology.orchestra.run.smithi081.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-21T22:50:14.962 INFO:teuthology.orchestra.run.smithi081.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-21T22:50:14.962 INFO:teuthology.orchestra.run.smithi081.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-21T22:50:14.962 INFO:teuthology.orchestra.run.smithi081.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-21T22:50:14.962 INFO:teuthology.orchestra.run.smithi081.stdout:Packager : builder@centos.org 2024-02-21T22:50:14.962 INFO:teuthology.orchestra.run.smithi081.stdout:Vendor : CentOS 2024-02-21T22:50:14.962 INFO:teuthology.orchestra.run.smithi081.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-21T22:50:14.962 INFO:teuthology.orchestra.run.smithi081.stdout:Summary : Support tools for GRUB. 2024-02-21T22:50:14.962 INFO:teuthology.orchestra.run.smithi081.stdout:Description : 2024-02-21T22:50:14.962 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-02-21T22:50:14.962 INFO:teuthology.orchestra.run.smithi081.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-21T22:50:14.963 INFO:teuthology.orchestra.run.smithi081.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-21T22:50:14.963 INFO:teuthology.orchestra.run.smithi081.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-21T22:50:14.963 INFO:teuthology.orchestra.run.smithi081.stdout:hardware devices. 2024-02-21T22:50:14.963 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-02-21T22:50:14.963 INFO:teuthology.orchestra.run.smithi081.stdout:This subpackage provides tools for support of all platforms. 2024-02-21T22:50:14.964 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-21T22:50:14.964 INFO:teuthology.task.kernel:Updating grub on smithi081 to boot 6.8.0-rc4-g1130e0532280 2024-02-21T22:50:14.964 DEBUG:teuthology.orchestra.run.smithi081:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-21T22:50:15.551 INFO:teuthology.orchestra.run.smithi081.stderr:Generating grub configuration file ... 2024-02-21T22:50:16.953 INFO:teuthology.orchestra.run.smithi081.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-21T22:50:16.987 INFO:teuthology.orchestra.run.smithi081.stderr:done 2024-02-21T22:50:16.989 DEBUG:teuthology.orchestra.run.smithi081:> mktemp 2024-02-21T22:50:17.004 INFO:teuthology.orchestra.run.smithi081.stdout:/tmp/tmp.lEDB4KTPjJ 2024-02-21T22:50:17.005 DEBUG:teuthology.orchestra.run.smithi081:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.lEDB4KTPjJ 2024-02-21T22:50:17.074 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 0666 /tmp/tmp.lEDB4KTPjJ 2024-02-21T22:50:17.225 DEBUG:teuthology.orchestra.remote:smithi081:/tmp/tmp.lEDB4KTPjJ is 6KB 2024-02-21T22:50:17.273 DEBUG:teuthology.orchestra.run.smithi081:> rm -fr /tmp/tmp.lEDB4KTPjJ 2024-02-21T22:50:17.288 DEBUG:teuthology.orchestra.run.smithi081:> sudo /bin/ls /boot/loader/entries || true 2024-02-21T22:50:17.352 INFO:teuthology.orchestra.run.smithi081.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-21T22:50:17.353 INFO:teuthology.orchestra.run.smithi081.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-21T22:50:17.353 INFO:teuthology.orchestra.run.smithi081.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-21T22:50:17.353 INFO:teuthology.orchestra.run.smithi081.stdout:15949e3c88704300882482eddfd7baae-6.8.0-rc4-g1130e0532280.conf 2024-02-21T22:50:17.354 DEBUG:teuthology.orchestra.run.smithi081:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.8.0-rc4-g1130e0532280 2024-02-21T22:50:17.493 DEBUG:teuthology.orchestra.run.smithi081:> sudo shutdown -r now 2024-02-21T22:50:19.900 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi028.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-02-21T22:50:19.900 DEBUG:teuthology.orchestra.run.smithi028:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-02-21T22:50:19.982 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280 2024-02-21T22:50:19.982 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/System.map 2024-02-21T22:50:19.982 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/config 2024-02-21T22:50:19.982 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel 2024-02-21T22:50:19.982 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/arch 2024-02-21T22:50:19.982 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/arch/x86 2024-02-21T22:50:19.982 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/arch/x86/crypto 2024-02-21T22:50:19.982 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-02-21T22:50:19.982 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-02-21T22:50:19.983 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-02-21T22:50:19.983 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-02-21T22:50:19.983 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-02-21T22:50:19.983 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-02-21T22:50:19.983 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-02-21T22:50:19.983 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/arch/x86/crypto/crc32c-intel.ko 2024-02-21T22:50:19.983 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-02-21T22:50:19.983 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-02-21T22:50:19.983 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-02-21T22:50:19.983 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-02-21T22:50:19.983 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/arch/x86/crypto/serpent-avx2.ko 2024-02-21T22:50:19.983 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-02-21T22:50:19.983 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-02-21T22:50:19.983 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-02-21T22:50:19.984 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-02-21T22:50:19.984 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-02-21T22:50:19.984 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-02-21T22:50:19.984 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-02-21T22:50:19.984 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/arch/x86/kernel 2024-02-21T22:50:19.984 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/arch/x86/kernel/cpu 2024-02-21T22:50:19.984 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/arch/x86/kernel/cpu/mce 2024-02-21T22:50:19.984 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-02-21T22:50:19.984 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/arch/x86/kvm 2024-02-21T22:50:19.984 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/arch/x86/kvm/kvm-amd.ko 2024-02-21T22:50:19.984 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/arch/x86/kvm/kvm-intel.ko 2024-02-21T22:50:19.984 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/arch/x86/kvm/kvm.ko 2024-02-21T22:50:19.984 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto 2024-02-21T22:50:19.985 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/ansi_cprng.ko 2024-02-21T22:50:19.985 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/anubis.ko 2024-02-21T22:50:19.985 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/arc4.ko 2024-02-21T22:50:19.985 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/async_tx 2024-02-21T22:50:19.985 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/async_tx/async_memcpy.ko 2024-02-21T22:50:19.985 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/async_tx/async_pq.ko 2024-02-21T22:50:19.985 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/async_tx/async_raid6_recov.ko 2024-02-21T22:50:19.985 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/async_tx/async_tx.ko 2024-02-21T22:50:19.985 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/async_tx/async_xor.ko 2024-02-21T22:50:19.985 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/async_tx/raid6test.ko 2024-02-21T22:50:19.985 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/authenc.ko 2024-02-21T22:50:19.985 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/authencesn.ko 2024-02-21T22:50:19.985 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/blake2b_generic.ko 2024-02-21T22:50:19.985 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/blowfish_common.ko 2024-02-21T22:50:19.985 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/blowfish_generic.ko 2024-02-21T22:50:19.986 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/camellia_generic.ko 2024-02-21T22:50:19.986 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/cast5_generic.ko 2024-02-21T22:50:19.986 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/cast6_generic.ko 2024-02-21T22:50:19.986 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/cast_common.ko 2024-02-21T22:50:19.986 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/ccm.ko 2024-02-21T22:50:19.986 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/cmac.ko 2024-02-21T22:50:19.986 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/crc32_generic.ko 2024-02-21T22:50:19.986 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/crypto_engine.ko 2024-02-21T22:50:19.986 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/crypto_user.ko 2024-02-21T22:50:19.986 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/cts.ko 2024-02-21T22:50:19.986 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/des_generic.ko 2024-02-21T22:50:19.986 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/dh_generic.ko 2024-02-21T22:50:19.986 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/echainiv.ko 2024-02-21T22:50:19.986 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/essiv.ko 2024-02-21T22:50:19.987 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/fcrypt.ko 2024-02-21T22:50:19.987 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/khazad.ko 2024-02-21T22:50:19.987 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/lz4.ko 2024-02-21T22:50:19.987 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/lz4hc.ko 2024-02-21T22:50:19.987 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/md4.ko 2024-02-21T22:50:19.987 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/michael_mic.ko 2024-02-21T22:50:19.987 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/pcbc.ko 2024-02-21T22:50:19.987 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/pcrypt.ko 2024-02-21T22:50:19.987 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/rmd160.ko 2024-02-21T22:50:19.987 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/seed.ko 2024-02-21T22:50:19.987 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/serpent_generic.ko 2024-02-21T22:50:19.987 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/tcrypt.ko 2024-02-21T22:50:19.987 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/tea.ko 2024-02-21T22:50:19.988 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/twofish_common.ko 2024-02-21T22:50:19.988 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/twofish_generic.ko 2024-02-21T22:50:19.988 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/vmac.ko 2024-02-21T22:50:19.988 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/wp512.ko 2024-02-21T22:50:19.988 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/xcbc.ko 2024-02-21T22:50:19.988 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/xor.ko 2024-02-21T22:50:19.988 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/crypto/xxhash_generic.ko 2024-02-21T22:50:19.988 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers 2024-02-21T22:50:19.988 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/acpi 2024-02-21T22:50:19.988 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/acpi/acpi_ipmi.ko 2024-02-21T22:50:19.988 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/acpi/acpi_pad.ko 2024-02-21T22:50:19.988 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/acpi/custom_method.ko 2024-02-21T22:50:19.988 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/acpi/ec_sys.ko 2024-02-21T22:50:19.988 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/acpi/platform_profile.ko 2024-02-21T22:50:19.989 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/acpi/sbs.ko 2024-02-21T22:50:19.989 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/acpi/sbshc.ko 2024-02-21T22:50:19.989 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/acpi/video.ko 2024-02-21T22:50:19.989 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata 2024-02-21T22:50:19.989 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/acard-ahci.ko 2024-02-21T22:50:19.989 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/ahci_platform.ko 2024-02-21T22:50:19.989 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/ata_generic.ko 2024-02-21T22:50:19.989 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/libahci_platform.ko 2024-02-21T22:50:19.989 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_acpi.ko 2024-02-21T22:50:19.989 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_ali.ko 2024-02-21T22:50:19.989 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_amd.ko 2024-02-21T22:50:19.989 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_artop.ko 2024-02-21T22:50:19.989 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_atiixp.ko 2024-02-21T22:50:19.989 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_atp867x.ko 2024-02-21T22:50:19.990 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_cmd640.ko 2024-02-21T22:50:19.990 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_cmd64x.ko 2024-02-21T22:50:19.990 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_cypress.ko 2024-02-21T22:50:19.990 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_efar.ko 2024-02-21T22:50:19.990 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_hpt366.ko 2024-02-21T22:50:19.990 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_hpt37x.ko 2024-02-21T22:50:19.990 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_hpt3x2n.ko 2024-02-21T22:50:19.990 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_hpt3x3.ko 2024-02-21T22:50:19.990 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_it8213.ko 2024-02-21T22:50:19.990 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_it821x.ko 2024-02-21T22:50:19.990 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_jmicron.ko 2024-02-21T22:50:19.990 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_marvell.ko 2024-02-21T22:50:19.990 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_mpiix.ko 2024-02-21T22:50:19.990 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_netcell.ko 2024-02-21T22:50:19.991 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_ninja32.ko 2024-02-21T22:50:19.991 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_ns87410.ko 2024-02-21T22:50:19.991 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_ns87415.ko 2024-02-21T22:50:19.991 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_oldpiix.ko 2024-02-21T22:50:19.991 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_opti.ko 2024-02-21T22:50:19.991 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_optidma.ko 2024-02-21T22:50:19.991 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_pcmcia.ko 2024-02-21T22:50:19.991 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_pdc2027x.ko 2024-02-21T22:50:19.991 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-02-21T22:50:19.991 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_piccolo.ko 2024-02-21T22:50:19.991 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_rdc.ko 2024-02-21T22:50:19.991 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_sch.ko 2024-02-21T22:50:19.991 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_serverworks.ko 2024-02-21T22:50:19.992 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_sil680.ko 2024-02-21T22:50:19.992 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_sis.ko 2024-02-21T22:50:19.992 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_sl82c105.ko 2024-02-21T22:50:19.992 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_triflex.ko 2024-02-21T22:50:19.992 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pata_via.ko 2024-02-21T22:50:19.992 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/pdc_adma.ko 2024-02-21T22:50:19.992 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/sata_inic162x.ko 2024-02-21T22:50:19.992 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/sata_mv.ko 2024-02-21T22:50:19.992 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/sata_nv.ko 2024-02-21T22:50:19.992 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/sata_promise.ko 2024-02-21T22:50:19.992 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/sata_qstor.ko 2024-02-21T22:50:19.992 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/sata_sil.ko 2024-02-21T22:50:19.992 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/sata_sil24.ko 2024-02-21T22:50:19.992 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/sata_sis.ko 2024-02-21T22:50:19.993 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/sata_svw.ko 2024-02-21T22:50:19.993 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/sata_sx4.ko 2024-02-21T22:50:19.993 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/sata_uli.ko 2024-02-21T22:50:19.993 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/sata_via.ko 2024-02-21T22:50:19.993 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ata/sata_vsc.ko 2024-02-21T22:50:19.993 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/auxdisplay 2024-02-21T22:50:19.993 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/auxdisplay/ks0108.ko 2024-02-21T22:50:19.993 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/base 2024-02-21T22:50:19.993 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/base/regmap 2024-02-21T22:50:19.993 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/base/regmap/regmap-i2c.ko 2024-02-21T22:50:19.993 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/base/regmap/regmap-mmio.ko 2024-02-21T22:50:19.993 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/bcma 2024-02-21T22:50:19.993 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/bcma/bcma.ko 2024-02-21T22:50:19.993 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/block 2024-02-21T22:50:19.994 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/block/aoe 2024-02-21T22:50:19.994 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/block/aoe/aoe.ko 2024-02-21T22:50:19.994 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/block/brd.ko 2024-02-21T22:50:19.994 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/block/drbd 2024-02-21T22:50:19.994 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/block/drbd/drbd.ko 2024-02-21T22:50:19.994 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/block/floppy.ko 2024-02-21T22:50:19.994 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/block/loop.ko 2024-02-21T22:50:19.994 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/block/mtip32xx 2024-02-21T22:50:19.994 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-02-21T22:50:19.994 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/block/nbd.ko 2024-02-21T22:50:19.994 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/block/null_blk 2024-02-21T22:50:19.994 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/block/null_blk/null_blk.ko 2024-02-21T22:50:19.994 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/block/pktcdvd.ko 2024-02-21T22:50:19.994 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/block/rbd.ko 2024-02-21T22:50:19.994 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/block/virtio_blk.ko 2024-02-21T22:50:19.995 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/block/xen-blkback 2024-02-21T22:50:19.995 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-02-21T22:50:19.995 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/block/xen-blkfront.ko 2024-02-21T22:50:19.995 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/block/zram 2024-02-21T22:50:19.995 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/block/zram/zram.ko 2024-02-21T22:50:19.995 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/char 2024-02-21T22:50:19.995 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/char/hangcheck-timer.ko 2024-02-21T22:50:19.995 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/char/hw_random 2024-02-21T22:50:19.995 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/char/hw_random/amd-rng.ko 2024-02-21T22:50:19.995 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/char/hw_random/intel-rng.ko 2024-02-21T22:50:19.995 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-02-21T22:50:19.995 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/char/hw_random/via-rng.ko 2024-02-21T22:50:19.995 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/char/hw_random/virtio-rng.ko 2024-02-21T22:50:19.995 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/char/ipmi 2024-02-21T22:50:19.996 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-02-21T22:50:19.996 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-02-21T22:50:19.996 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-02-21T22:50:19.996 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/char/ipmi/ipmi_si.ko 2024-02-21T22:50:19.996 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-02-21T22:50:19.996 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-02-21T22:50:19.996 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/char/lp.ko 2024-02-21T22:50:19.996 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/char/mwave 2024-02-21T22:50:19.996 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/char/mwave/mwave.ko 2024-02-21T22:50:19.996 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/char/ppdev.ko 2024-02-21T22:50:19.996 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/char/tlclk.ko 2024-02-21T22:50:19.997 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/char/tpm 2024-02-21T22:50:19.997 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/char/tpm/tpm.ko 2024-02-21T22:50:19.997 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/char/tpm/tpm_atmel.ko 2024-02-21T22:50:19.997 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/char/tpm/tpm_infineon.ko 2024-02-21T22:50:19.997 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/char/tpm/tpm_nsc.ko 2024-02-21T22:50:19.997 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/char/tpm/tpm_tis.ko 2024-02-21T22:50:19.997 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-02-21T22:50:19.997 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/char/uv_mmtimer.ko 2024-02-21T22:50:19.997 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/char/virtio_console.ko 2024-02-21T22:50:19.997 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/cpufreq 2024-02-21T22:50:19.997 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-02-21T22:50:19.997 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-02-21T22:50:19.997 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/cpufreq/p4-clockmod.ko 2024-02-21T22:50:19.997 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-02-21T22:50:19.998 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/cpufreq/powernow-k8.ko 2024-02-21T22:50:19.998 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/cpufreq/speedstep-lib.ko 2024-02-21T22:50:19.998 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/crypto 2024-02-21T22:50:19.998 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/crypto/ccp 2024-02-21T22:50:19.998 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-02-21T22:50:19.998 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/crypto/ccp/ccp.ko 2024-02-21T22:50:19.998 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/crypto/intel 2024-02-21T22:50:19.998 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/crypto/intel/qat 2024-02-21T22:50:19.998 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/crypto/intel/qat/qat_common 2024-02-21T22:50:19.998 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-02-21T22:50:19.998 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-02-21T22:50:19.998 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-02-21T22:50:19.998 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/crypto/padlock-aes.ko 2024-02-21T22:50:19.998 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/crypto/padlock-sha.ko 2024-02-21T22:50:19.999 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/crypto/virtio 2024-02-21T22:50:19.999 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-02-21T22:50:19.999 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/dca 2024-02-21T22:50:19.999 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/dca/dca.ko 2024-02-21T22:50:19.999 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/devfreq 2024-02-21T22:50:19.999 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-02-21T22:50:19.999 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/dma 2024-02-21T22:50:19.999 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/dma/dw 2024-02-21T22:50:19.999 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/dma/dw/dw_dmac.ko 2024-02-21T22:50:19.999 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/dma/ioat 2024-02-21T22:50:19.999 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/dma/ioat/ioatdma.ko 2024-02-21T22:50:19.999 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/edac 2024-02-21T22:50:19.999 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/edac/amd64_edac.ko 2024-02-21T22:50:20.000 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/edac/e752x_edac.ko 2024-02-21T22:50:20.000 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/edac/edac_mce_amd.ko 2024-02-21T22:50:20.000 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/edac/i3000_edac.ko 2024-02-21T22:50:20.000 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/edac/i3200_edac.ko 2024-02-21T22:50:20.000 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/edac/i5100_edac.ko 2024-02-21T22:50:20.000 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/edac/i5400_edac.ko 2024-02-21T22:50:20.000 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/edac/i7300_edac.ko 2024-02-21T22:50:20.000 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/edac/i7core_edac.ko 2024-02-21T22:50:20.000 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/edac/i82975x_edac.ko 2024-02-21T22:50:20.000 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/edac/ie31200_edac.ko 2024-02-21T22:50:20.000 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/edac/sb_edac.ko 2024-02-21T22:50:20.000 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/edac/x38_edac.ko 2024-02-21T22:50:20.000 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/firmware 2024-02-21T22:50:20.001 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/firmware/edd.ko 2024-02-21T22:50:20.001 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/firmware/iscsi_ibft.ko 2024-02-21T22:50:20.001 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/gpio 2024-02-21T22:50:20.001 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/gpio/gpio-ich.ko 2024-02-21T22:50:20.001 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/gpio/gpio-viperboard.ko 2024-02-21T22:50:20.001 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid 2024-02-21T22:50:20.001 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-appleir.ko 2024-02-21T22:50:20.001 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-aureal.ko 2024-02-21T22:50:20.001 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-axff.ko 2024-02-21T22:50:20.001 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-dr.ko 2024-02-21T22:50:20.001 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-elecom.ko 2024-02-21T22:50:20.001 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-elo.ko 2024-02-21T22:50:20.001 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-emsff.ko 2024-02-21T22:50:20.002 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-gaff.ko 2024-02-21T22:50:20.002 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-gt683r.ko 2024-02-21T22:50:20.002 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-gyration.ko 2024-02-21T22:50:20.002 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-holtek-kbd.ko 2024-02-21T22:50:20.002 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-holtek-mouse.ko 2024-02-21T22:50:20.002 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-holtekff.ko 2024-02-21T22:50:20.002 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-hyperv.ko 2024-02-21T22:50:20.002 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-icade.ko 2024-02-21T22:50:20.002 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-keytouch.ko 2024-02-21T22:50:20.002 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-kye.ko 2024-02-21T22:50:20.002 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-lcpower.ko 2024-02-21T22:50:20.003 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-led.ko 2024-02-21T22:50:20.003 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-lenovo.ko 2024-02-21T22:50:20.003 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-logitech-dj.ko 2024-02-21T22:50:20.003 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-02-21T22:50:20.003 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-multitouch.ko 2024-02-21T22:50:20.003 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-ortek.ko 2024-02-21T22:50:20.003 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-penmount.ko 2024-02-21T22:50:20.003 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-petalynx.ko 2024-02-21T22:50:20.003 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-picolcd.ko 2024-02-21T22:50:20.003 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-pl.ko 2024-02-21T22:50:20.003 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-plantronics.ko 2024-02-21T22:50:20.003 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-primax.ko 2024-02-21T22:50:20.003 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-rmi.ko 2024-02-21T22:50:20.004 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-roccat-arvo.ko 2024-02-21T22:50:20.004 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-roccat-common.ko 2024-02-21T22:50:20.004 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-roccat-isku.ko 2024-02-21T22:50:20.004 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-roccat-kone.ko 2024-02-21T22:50:20.004 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-02-21T22:50:20.004 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-roccat-konepure.ko 2024-02-21T22:50:20.004 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-02-21T22:50:20.004 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-roccat-lua.ko 2024-02-21T22:50:20.004 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-roccat-pyra.ko 2024-02-21T22:50:20.004 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-roccat-ryos.ko 2024-02-21T22:50:20.004 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-roccat-savu.ko 2024-02-21T22:50:20.004 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-roccat.ko 2024-02-21T22:50:20.004 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-saitek.ko 2024-02-21T22:50:20.005 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-samsung.ko 2024-02-21T22:50:20.005 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-sensor-hub.ko 2024-02-21T22:50:20.005 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-sjoy.ko 2024-02-21T22:50:20.005 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-sony.ko 2024-02-21T22:50:20.005 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-speedlink.ko 2024-02-21T22:50:20.005 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-steelseries.ko 2024-02-21T22:50:20.005 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-sunplus.ko 2024-02-21T22:50:20.005 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-thrustmaster.ko 2024-02-21T22:50:20.005 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-tivo.ko 2024-02-21T22:50:20.005 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-tmff.ko 2024-02-21T22:50:20.005 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-topseed.ko 2024-02-21T22:50:20.005 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-twinhan.ko 2024-02-21T22:50:20.005 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-uclogic.ko 2024-02-21T22:50:20.006 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-waltop.ko 2024-02-21T22:50:20.006 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-wiimote.ko 2024-02-21T22:50:20.006 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-xinmo.ko 2024-02-21T22:50:20.006 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-zpff.ko 2024-02-21T22:50:20.006 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/hid-zydacron.ko 2024-02-21T22:50:20.006 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/uhid.ko 2024-02-21T22:50:20.006 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hid/wacom.ko 2024-02-21T22:50:20.006 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hv 2024-02-21T22:50:20.006 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hv/hv_balloon.ko 2024-02-21T22:50:20.006 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hv/hv_utils.ko 2024-02-21T22:50:20.006 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hv/hv_vmbus.ko 2024-02-21T22:50:20.006 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon 2024-02-21T22:50:20.006 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/abituguru.ko 2024-02-21T22:50:20.006 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/abituguru3.ko 2024-02-21T22:50:20.007 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/acpi_power_meter.ko 2024-02-21T22:50:20.007 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/ad7414.ko 2024-02-21T22:50:20.007 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/ad7418.ko 2024-02-21T22:50:20.007 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/adc128d818.ko 2024-02-21T22:50:20.007 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/adm1025.ko 2024-02-21T22:50:20.007 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/adm1026.ko 2024-02-21T22:50:20.007 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/adm1029.ko 2024-02-21T22:50:20.007 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/adm1031.ko 2024-02-21T22:50:20.007 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/adm9240.ko 2024-02-21T22:50:20.007 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/ads7828.ko 2024-02-21T22:50:20.007 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/adt7410.ko 2024-02-21T22:50:20.007 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/adt7411.ko 2024-02-21T22:50:20.007 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/adt7462.ko 2024-02-21T22:50:20.007 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/adt7470.ko 2024-02-21T22:50:20.007 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/adt7475.ko 2024-02-21T22:50:20.008 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/adt7x10.ko 2024-02-21T22:50:20.008 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/amc6821.ko 2024-02-21T22:50:20.008 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/applesmc.ko 2024-02-21T22:50:20.008 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/asb100.ko 2024-02-21T22:50:20.008 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/asc7621.ko 2024-02-21T22:50:20.008 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/asus_atk0110.ko 2024-02-21T22:50:20.008 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/atxp1.ko 2024-02-21T22:50:20.008 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/coretemp.ko 2024-02-21T22:50:20.008 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-02-21T22:50:20.008 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/dme1737.ko 2024-02-21T22:50:20.008 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/ds1621.ko 2024-02-21T22:50:20.008 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/ds620.ko 2024-02-21T22:50:20.008 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/emc1403.ko 2024-02-21T22:50:20.008 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/emc6w201.ko 2024-02-21T22:50:20.009 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/f71805f.ko 2024-02-21T22:50:20.009 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/f71882fg.ko 2024-02-21T22:50:20.009 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/f75375s.ko 2024-02-21T22:50:20.009 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/fam15h_power.ko 2024-02-21T22:50:20.009 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/fschmd.ko 2024-02-21T22:50:20.009 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/g760a.ko 2024-02-21T22:50:20.009 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/g762.ko 2024-02-21T22:50:20.009 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/gl518sm.ko 2024-02-21T22:50:20.009 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/gl520sm.ko 2024-02-21T22:50:20.009 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/hwmon-vid.ko 2024-02-21T22:50:20.009 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/i5500_temp.ko 2024-02-21T22:50:20.009 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/i5k_amb.ko 2024-02-21T22:50:20.009 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/ibmaem.ko 2024-02-21T22:50:20.009 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/ibmpex.ko 2024-02-21T22:50:20.010 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/ina209.ko 2024-02-21T22:50:20.010 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/ina2xx.ko 2024-02-21T22:50:20.010 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/it87.ko 2024-02-21T22:50:20.010 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/k10temp.ko 2024-02-21T22:50:20.010 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/k8temp.ko 2024-02-21T22:50:20.010 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/lineage-pem.ko 2024-02-21T22:50:20.010 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/lm63.ko 2024-02-21T22:50:20.010 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/lm73.ko 2024-02-21T22:50:20.010 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/lm75.ko 2024-02-21T22:50:20.010 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/lm77.ko 2024-02-21T22:50:20.010 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/lm78.ko 2024-02-21T22:50:20.010 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/lm80.ko 2024-02-21T22:50:20.010 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/lm83.ko 2024-02-21T22:50:20.011 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/lm85.ko 2024-02-21T22:50:20.011 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/lm87.ko 2024-02-21T22:50:20.011 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/lm90.ko 2024-02-21T22:50:20.011 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/lm92.ko 2024-02-21T22:50:20.011 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/lm93.ko 2024-02-21T22:50:20.011 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/lm95234.ko 2024-02-21T22:50:20.011 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/lm95241.ko 2024-02-21T22:50:20.011 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/lm95245.ko 2024-02-21T22:50:20.011 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/ltc2945.ko 2024-02-21T22:50:20.012 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/ltc4151.ko 2024-02-21T22:50:20.012 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/ltc4215.ko 2024-02-21T22:50:20.012 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/ltc4222.ko 2024-02-21T22:50:20.012 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/ltc4245.ko 2024-02-21T22:50:20.012 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/ltc4260.ko 2024-02-21T22:50:20.012 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/ltc4261.ko 2024-02-21T22:50:20.012 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/max16065.ko 2024-02-21T22:50:20.012 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/max1619.ko 2024-02-21T22:50:20.012 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/max1668.ko 2024-02-21T22:50:20.012 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/max197.ko 2024-02-21T22:50:20.012 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/max6639.ko 2024-02-21T22:50:20.012 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/max6650.ko 2024-02-21T22:50:20.013 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/max6697.ko 2024-02-21T22:50:20.013 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/mcp3021.ko 2024-02-21T22:50:20.013 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/nct6683.ko 2024-02-21T22:50:20.013 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/nct6775-core.ko 2024-02-21T22:50:20.013 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/nct6775.ko 2024-02-21T22:50:20.013 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/nct7802.ko 2024-02-21T22:50:20.013 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/pc87360.ko 2024-02-21T22:50:20.013 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/pc87427.ko 2024-02-21T22:50:20.013 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/pcf8591.ko 2024-02-21T22:50:20.013 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/pmbus 2024-02-21T22:50:20.013 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-02-21T22:50:20.013 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-02-21T22:50:20.013 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-02-21T22:50:20.013 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/pmbus/max16064.ko 2024-02-21T22:50:20.014 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/pmbus/max34440.ko 2024-02-21T22:50:20.014 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/pmbus/max8688.ko 2024-02-21T22:50:20.015 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-02-21T22:50:20.015 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-02-21T22:50:20.015 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-02-21T22:50:20.015 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-02-21T22:50:20.016 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-02-21T22:50:20.016 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-02-21T22:50:20.016 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/powr1220.ko 2024-02-21T22:50:20.016 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/sch5627.ko 2024-02-21T22:50:20.016 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/sch5636.ko 2024-02-21T22:50:20.016 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/sch56xx-common.ko 2024-02-21T22:50:20.016 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/sht15.ko 2024-02-21T22:50:20.016 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/sht21.ko 2024-02-21T22:50:20.016 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/shtc1.ko 2024-02-21T22:50:20.016 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/sis5595.ko 2024-02-21T22:50:20.016 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/smsc47b397.ko 2024-02-21T22:50:20.016 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/smsc47m1.ko 2024-02-21T22:50:20.016 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/smsc47m192.ko 2024-02-21T22:50:20.016 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/thmc50.ko 2024-02-21T22:50:20.016 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/tmp102.ko 2024-02-21T22:50:20.017 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/tmp103.ko 2024-02-21T22:50:20.017 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/tmp401.ko 2024-02-21T22:50:20.017 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/tmp421.ko 2024-02-21T22:50:20.017 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/via-cputemp.ko 2024-02-21T22:50:20.017 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/via686a.ko 2024-02-21T22:50:20.017 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/vt1211.ko 2024-02-21T22:50:20.017 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/vt8231.ko 2024-02-21T22:50:20.017 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/w83627ehf.ko 2024-02-21T22:50:20.017 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/w83627hf.ko 2024-02-21T22:50:20.017 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/w83781d.ko 2024-02-21T22:50:20.017 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/w83791d.ko 2024-02-21T22:50:20.017 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/w83792d.ko 2024-02-21T22:50:20.017 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/w83793.ko 2024-02-21T22:50:20.018 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/w83795.ko 2024-02-21T22:50:20.018 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/w83l785ts.ko 2024-02-21T22:50:20.018 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/hwmon/w83l786ng.ko 2024-02-21T22:50:20.018 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/i2c 2024-02-21T22:50:20.018 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/i2c/algos 2024-02-21T22:50:20.018 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-02-21T22:50:20.018 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-02-21T22:50:20.018 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/i2c/busses 2024-02-21T22:50:20.018 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-02-21T22:50:20.018 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-02-21T22:50:20.018 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-02-21T22:50:20.018 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-02-21T22:50:20.018 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-02-21T22:50:20.019 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-02-21T22:50:20.019 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-02-21T22:50:20.019 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-02-21T22:50:20.019 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/i2c/busses/i2c-i801.ko 2024-02-21T22:50:20.019 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/i2c/busses/i2c-isch.ko 2024-02-21T22:50:20.019 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-02-21T22:50:20.019 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-02-21T22:50:20.019 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-02-21T22:50:20.019 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/i2c/busses/i2c-parport.ko 2024-02-21T22:50:20.019 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-02-21T22:50:20.019 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-02-21T22:50:20.019 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-02-21T22:50:20.019 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-02-21T22:50:20.020 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-02-21T22:50:20.020 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-02-21T22:50:20.020 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/i2c/busses/i2c-via.ko 2024-02-21T22:50:20.020 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-02-21T22:50:20.020 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-02-21T22:50:20.020 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/i2c/i2c-dev.ko 2024-02-21T22:50:20.020 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/i2c/i2c-mux.ko 2024-02-21T22:50:20.020 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-02-21T22:50:20.020 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/i2c/i2c-smbus.ko 2024-02-21T22:50:20.020 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/i2c/i2c-stub.ko 2024-02-21T22:50:20.020 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/infiniband 2024-02-21T22:50:20.020 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/infiniband/core 2024-02-21T22:50:20.020 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/infiniband/core/ib_cm.ko 2024-02-21T22:50:20.020 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/infiniband/core/ib_core.ko 2024-02-21T22:50:20.021 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/infiniband/core/ib_umad.ko 2024-02-21T22:50:20.021 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-02-21T22:50:20.021 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/infiniband/core/iw_cm.ko 2024-02-21T22:50:20.021 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/infiniband/core/rdma_cm.ko 2024-02-21T22:50:20.021 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-02-21T22:50:20.021 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/infiniband/hw 2024-02-21T22:50:20.021 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/infiniband/hw/cxgb4 2024-02-21T22:50:20.021 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-02-21T22:50:20.021 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/infiniband/hw/mlx4 2024-02-21T22:50:20.021 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-02-21T22:50:20.021 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/infiniband/hw/mlx5 2024-02-21T22:50:20.021 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-02-21T22:50:20.021 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/infiniband/hw/mthca 2024-02-21T22:50:20.021 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-02-21T22:50:20.021 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/infiniband/ulp 2024-02-21T22:50:20.022 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/infiniband/ulp/ipoib 2024-02-21T22:50:20.022 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-02-21T22:50:20.022 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/infiniband/ulp/iser 2024-02-21T22:50:20.022 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-02-21T22:50:20.022 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/infiniband/ulp/isert 2024-02-21T22:50:20.022 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-02-21T22:50:20.022 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/infiniband/ulp/srp 2024-02-21T22:50:20.022 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-02-21T22:50:20.022 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/infiniband/ulp/srpt 2024-02-21T22:50:20.022 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-02-21T22:50:20.022 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/input 2024-02-21T22:50:20.022 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/input/gameport 2024-02-21T22:50:20.022 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-02-21T22:50:20.022 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/input/gameport/fm801-gp.ko 2024-02-21T22:50:20.023 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/input/gameport/gameport.ko 2024-02-21T22:50:20.023 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/input/joydev.ko 2024-02-21T22:50:20.023 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/input/keyboard 2024-02-21T22:50:20.023 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/input/keyboard/gpio_keys.ko 2024-02-21T22:50:20.023 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/input/mouse 2024-02-21T22:50:20.023 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/input/mouse/appletouch.ko 2024-02-21T22:50:20.023 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/input/mouse/bcm5974.ko 2024-02-21T22:50:20.023 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/input/mouse/cyapatp.ko 2024-02-21T22:50:20.023 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/input/mouse/elan_i2c.ko 2024-02-21T22:50:20.023 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/input/mouse/sermouse.ko 2024-02-21T22:50:20.023 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-02-21T22:50:20.023 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/input/mouse/synaptics_usb.ko 2024-02-21T22:50:20.023 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/input/mouse/vsxxxaa.ko 2024-02-21T22:50:20.023 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/input/rmi4 2024-02-21T22:50:20.023 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/input/rmi4/rmi_core.ko 2024-02-21T22:50:20.024 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/input/serio 2024-02-21T22:50:20.024 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/input/serio/altera_ps2.ko 2024-02-21T22:50:20.024 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/input/serio/arc_ps2.ko 2024-02-21T22:50:20.024 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-02-21T22:50:20.024 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/input/serio/serio_raw.ko 2024-02-21T22:50:20.024 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/input/sparse-keymap.ko 2024-02-21T22:50:20.024 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/leds 2024-02-21T22:50:20.024 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/leds/leds-blinkm.ko 2024-02-21T22:50:20.024 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/leds/leds-lm3530.ko 2024-02-21T22:50:20.024 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/leds/leds-lp3944.ko 2024-02-21T22:50:20.024 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/leds/leds-ss4200.ko 2024-02-21T22:50:20.024 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/leds/trigger 2024-02-21T22:50:20.024 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-02-21T22:50:20.024 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-02-21T22:50:20.025 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-02-21T22:50:20.025 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-02-21T22:50:20.025 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-02-21T22:50:20.025 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-02-21T22:50:20.025 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-02-21T22:50:20.025 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-02-21T22:50:20.025 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-02-21T22:50:20.025 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/md 2024-02-21T22:50:20.025 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/md/bcache 2024-02-21T22:50:20.025 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/md/bcache/bcache.ko 2024-02-21T22:50:20.025 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/md/dm-bio-prison.ko 2024-02-21T22:50:20.025 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/md/dm-cache-smq.ko 2024-02-21T22:50:20.025 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/md/dm-cache.ko 2024-02-21T22:50:20.025 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/md/dm-crypt.ko 2024-02-21T22:50:20.025 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/md/dm-delay.ko 2024-02-21T22:50:20.026 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/md/dm-flakey.ko 2024-02-21T22:50:20.026 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/md/dm-log-userspace.ko 2024-02-21T22:50:20.026 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/md/dm-multipath.ko 2024-02-21T22:50:20.026 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/md/dm-queue-length.ko 2024-02-21T22:50:20.026 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/md/dm-raid.ko 2024-02-21T22:50:20.026 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/md/dm-round-robin.ko 2024-02-21T22:50:20.026 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/md/dm-service-time.ko 2024-02-21T22:50:20.026 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/md/dm-switch.ko 2024-02-21T22:50:20.026 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/md/dm-thin-pool.ko 2024-02-21T22:50:20.026 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/md/dm-verity.ko 2024-02-21T22:50:20.026 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/md/persistent-data 2024-02-21T22:50:20.026 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-02-21T22:50:20.026 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/md/raid0.ko 2024-02-21T22:50:20.026 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/md/raid1.ko 2024-02-21T22:50:20.027 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/md/raid10.ko 2024-02-21T22:50:20.027 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/md/raid456.ko 2024-02-21T22:50:20.027 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/message 2024-02-21T22:50:20.027 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/message/fusion 2024-02-21T22:50:20.027 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/message/fusion/mptbase.ko 2024-02-21T22:50:20.027 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/message/fusion/mptctl.ko 2024-02-21T22:50:20.027 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/message/fusion/mptfc.ko 2024-02-21T22:50:20.027 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/message/fusion/mptlan.ko 2024-02-21T22:50:20.027 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/message/fusion/mptsas.ko 2024-02-21T22:50:20.027 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/message/fusion/mptscsih.ko 2024-02-21T22:50:20.028 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/message/fusion/mptspi.ko 2024-02-21T22:50:20.028 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/mfd 2024-02-21T22:50:20.028 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/mfd/lpc_ich.ko 2024-02-21T22:50:20.028 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/mfd/lpc_sch.ko 2024-02-21T22:50:20.028 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/mfd/mfd-core.ko 2024-02-21T22:50:20.028 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/mfd/sm501.ko 2024-02-21T22:50:20.028 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/mfd/viperboard.ko 2024-02-21T22:50:20.028 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/mfd/vx855.ko 2024-02-21T22:50:20.028 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/mfd/wl1273-core.ko 2024-02-21T22:50:20.028 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/misc 2024-02-21T22:50:20.028 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/misc/altera-stapl 2024-02-21T22:50:20.028 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-02-21T22:50:20.028 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/misc/apds9802als.ko 2024-02-21T22:50:20.028 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/misc/apds990x.ko 2024-02-21T22:50:20.029 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/misc/bh1770glc.ko 2024-02-21T22:50:20.029 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/misc/cb710 2024-02-21T22:50:20.029 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/misc/cb710/cb710.ko 2024-02-21T22:50:20.029 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/misc/eeprom 2024-02-21T22:50:20.029 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/misc/eeprom/at24.ko 2024-02-21T22:50:20.029 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-02-21T22:50:20.029 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/misc/eeprom/max6875.ko 2024-02-21T22:50:20.029 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/misc/enclosure.ko 2024-02-21T22:50:20.029 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/misc/hpilo.ko 2024-02-21T22:50:20.029 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/misc/isl29003.ko 2024-02-21T22:50:20.029 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/misc/isl29020.ko 2024-02-21T22:50:20.029 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/misc/lis3lv02d 2024-02-21T22:50:20.029 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-02-21T22:50:20.029 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-02-21T22:50:20.029 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/misc/mei 2024-02-21T22:50:20.030 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/misc/mei/mei-me.ko 2024-02-21T22:50:20.030 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/misc/mei/mei-txe.ko 2024-02-21T22:50:20.030 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/misc/mei/mei.ko 2024-02-21T22:50:20.030 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/misc/sgi-gru 2024-02-21T22:50:20.030 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/misc/sgi-gru/gru.ko 2024-02-21T22:50:20.030 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/misc/sgi-xp 2024-02-21T22:50:20.030 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/misc/sgi-xp/xp.ko 2024-02-21T22:50:20.030 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/misc/sgi-xp/xpc.ko 2024-02-21T22:50:20.030 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-02-21T22:50:20.030 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/misc/tifm_7xx1.ko 2024-02-21T22:50:20.030 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/misc/tifm_core.ko 2024-02-21T22:50:20.030 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/misc/tsl2550.ko 2024-02-21T22:50:20.030 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/misc/vmw_balloon.ko 2024-02-21T22:50:20.030 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/misc/vmw_vmci 2024-02-21T22:50:20.031 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-02-21T22:50:20.031 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net 2024-02-21T22:50:20.031 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/bonding 2024-02-21T22:50:20.031 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/bonding/bonding.ko 2024-02-21T22:50:20.031 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/dsa 2024-02-21T22:50:20.031 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/dsa/b53 2024-02-21T22:50:20.031 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/dsa/b53/b53_common.ko 2024-02-21T22:50:20.031 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/dsa/bcm-sf2.ko 2024-02-21T22:50:20.032 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/dsa/mv88e6060.ko 2024-02-21T22:50:20.032 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/dsa/mv88e6xxx 2024-02-21T22:50:20.032 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-02-21T22:50:20.032 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/dummy.ko 2024-02-21T22:50:20.032 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/eql.ko 2024-02-21T22:50:20.032 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet 2024-02-21T22:50:20.032 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/3com 2024-02-21T22:50:20.032 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-02-21T22:50:20.032 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-02-21T22:50:20.032 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-02-21T22:50:20.032 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-02-21T22:50:20.032 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/8390 2024-02-21T22:50:20.032 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/8390/8390.ko 2024-02-21T22:50:20.033 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-02-21T22:50:20.033 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-02-21T22:50:20.033 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-02-21T22:50:20.033 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/adaptec 2024-02-21T22:50:20.033 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-02-21T22:50:20.033 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/agere 2024-02-21T22:50:20.033 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/agere/et131x.ko 2024-02-21T22:50:20.033 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/alteon 2024-02-21T22:50:20.033 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-02-21T22:50:20.033 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/altera 2024-02-21T22:50:20.033 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-02-21T22:50:20.033 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/amd 2024-02-21T22:50:20.033 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-02-21T22:50:20.033 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-02-21T22:50:20.034 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-02-21T22:50:20.034 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/atheros 2024-02-21T22:50:20.034 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/atheros/alx 2024-02-21T22:50:20.034 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-02-21T22:50:20.034 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/atheros/atl1c 2024-02-21T22:50:20.034 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-02-21T22:50:20.034 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/atheros/atl1e 2024-02-21T22:50:20.034 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-02-21T22:50:20.034 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/atheros/atlx 2024-02-21T22:50:20.034 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-02-21T22:50:20.034 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-02-21T22:50:20.034 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/broadcom 2024-02-21T22:50:20.034 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-02-21T22:50:20.034 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-02-21T22:50:20.035 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-02-21T22:50:20.035 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-02-21T22:50:20.035 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-02-21T22:50:20.035 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/broadcom/genet 2024-02-21T22:50:20.035 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-02-21T22:50:20.035 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-02-21T22:50:20.035 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/brocade 2024-02-21T22:50:20.035 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/brocade/bna 2024-02-21T22:50:20.035 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-02-21T22:50:20.035 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/chelsio 2024-02-21T22:50:20.035 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/chelsio/cxgb 2024-02-21T22:50:20.035 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-02-21T22:50:20.035 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-02-21T22:50:20.036 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-02-21T22:50:20.036 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-02-21T22:50:20.036 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-02-21T22:50:20.036 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-02-21T22:50:20.036 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-02-21T22:50:20.036 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-02-21T22:50:20.036 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-02-21T22:50:20.036 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/cisco 2024-02-21T22:50:20.036 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/cisco/enic 2024-02-21T22:50:20.036 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-02-21T22:50:20.036 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/dec 2024-02-21T22:50:20.036 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/dec/tulip 2024-02-21T22:50:20.036 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-02-21T22:50:20.036 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-02-21T22:50:20.037 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-02-21T22:50:20.037 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-02-21T22:50:20.037 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-02-21T22:50:20.037 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-02-21T22:50:20.037 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/dlink 2024-02-21T22:50:20.037 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-02-21T22:50:20.037 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-02-21T22:50:20.037 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/dnet.ko 2024-02-21T22:50:20.037 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/emulex 2024-02-21T22:50:20.037 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/emulex/benet 2024-02-21T22:50:20.037 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-02-21T22:50:20.037 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/ethoc.ko 2024-02-21T22:50:20.037 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/fealnx.ko 2024-02-21T22:50:20.037 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/intel 2024-02-21T22:50:20.037 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/intel/e100.ko 2024-02-21T22:50:20.038 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/intel/e1000 2024-02-21T22:50:20.038 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-02-21T22:50:20.038 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/intel/e1000e 2024-02-21T22:50:20.038 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-02-21T22:50:20.038 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/intel/fm10k 2024-02-21T22:50:20.038 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-02-21T22:50:20.038 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/intel/i40e 2024-02-21T22:50:20.038 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-02-21T22:50:20.038 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/intel/iavf 2024-02-21T22:50:20.038 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-02-21T22:50:20.038 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/intel/igb 2024-02-21T22:50:20.038 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-02-21T22:50:20.038 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/intel/igbvf 2024-02-21T22:50:20.038 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-02-21T22:50:20.039 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/intel/ixgbe 2024-02-21T22:50:20.039 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-02-21T22:50:20.039 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/intel/ixgbevf 2024-02-21T22:50:20.039 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-02-21T22:50:20.039 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/jme.ko 2024-02-21T22:50:20.039 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/marvell 2024-02-21T22:50:20.039 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-02-21T22:50:20.039 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/marvell/skge.ko 2024-02-21T22:50:20.039 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-02-21T22:50:20.039 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/mellanox 2024-02-21T22:50:20.039 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/mellanox/mlx4 2024-02-21T22:50:20.039 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-02-21T22:50:20.039 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-02-21T22:50:20.039 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/mellanox/mlx5 2024-02-21T22:50:20.039 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-02-21T22:50:20.040 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-02-21T22:50:20.040 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/micrel 2024-02-21T22:50:20.040 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-02-21T22:50:20.040 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/myricom 2024-02-21T22:50:20.040 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/myricom/myri10ge 2024-02-21T22:50:20.040 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-02-21T22:50:20.040 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/natsemi 2024-02-21T22:50:20.040 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-02-21T22:50:20.040 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-02-21T22:50:20.040 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/neterion 2024-02-21T22:50:20.040 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-02-21T22:50:20.040 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/nvidia 2024-02-21T22:50:20.041 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-02-21T22:50:20.041 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/packetengines 2024-02-21T22:50:20.041 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-02-21T22:50:20.041 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-02-21T22:50:20.041 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/qlogic 2024-02-21T22:50:20.041 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/qlogic/netxen 2024-02-21T22:50:20.041 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-02-21T22:50:20.041 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-02-21T22:50:20.041 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-02-21T22:50:20.041 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-02-21T22:50:20.041 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/rdc 2024-02-21T22:50:20.041 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-02-21T22:50:20.041 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/realtek 2024-02-21T22:50:20.041 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-02-21T22:50:20.041 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-02-21T22:50:20.042 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/realtek/atp.ko 2024-02-21T22:50:20.042 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-02-21T22:50:20.042 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/rocker 2024-02-21T22:50:20.042 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-02-21T22:50:20.042 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/sfc 2024-02-21T22:50:20.042 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-02-21T22:50:20.042 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/silan 2024-02-21T22:50:20.042 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-02-21T22:50:20.042 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/sis 2024-02-21T22:50:20.042 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/sis/sis190.ko 2024-02-21T22:50:20.042 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/sis/sis900.ko 2024-02-21T22:50:20.042 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/smsc 2024-02-21T22:50:20.042 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-02-21T22:50:20.042 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-02-21T22:50:20.042 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-02-21T22:50:20.043 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-02-21T22:50:20.043 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/stmicro 2024-02-21T22:50:20.043 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/stmicro/stmmac 2024-02-21T22:50:20.043 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-02-21T22:50:20.043 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-02-21T22:50:20.043 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/sun 2024-02-21T22:50:20.043 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/sun/cassini.ko 2024-02-21T22:50:20.043 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/sun/niu.ko 2024-02-21T22:50:20.043 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/sun/sungem.ko 2024-02-21T22:50:20.043 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-02-21T22:50:20.043 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/tehuti 2024-02-21T22:50:20.043 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-02-21T22:50:20.043 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/ti 2024-02-21T22:50:20.043 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/ti/tlan.ko 2024-02-21T22:50:20.043 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/via 2024-02-21T22:50:20.044 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-02-21T22:50:20.044 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-02-21T22:50:20.044 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/wiznet 2024-02-21T22:50:20.044 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-02-21T22:50:20.044 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-02-21T22:50:20.044 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/xircom 2024-02-21T22:50:20.044 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-02-21T22:50:20.044 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/geneve.ko 2024-02-21T22:50:20.044 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/hyperv 2024-02-21T22:50:20.044 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-02-21T22:50:20.044 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ieee802154 2024-02-21T22:50:20.044 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ieee802154/fakelb.ko 2024-02-21T22:50:20.044 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ifb.ko 2024-02-21T22:50:20.045 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ipvlan 2024-02-21T22:50:20.045 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ipvlan/ipvlan.ko 2024-02-21T22:50:20.045 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/macvlan.ko 2024-02-21T22:50:20.045 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/macvtap.ko 2024-02-21T22:50:20.045 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/mdio 2024-02-21T22:50:20.045 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/mdio.ko 2024-02-21T22:50:20.045 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-02-21T22:50:20.045 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-02-21T22:50:20.045 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/mdio/mdio-regmap.ko 2024-02-21T22:50:20.045 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/mii.ko 2024-02-21T22:50:20.045 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/net_failover.ko 2024-02-21T22:50:20.045 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/netconsole.ko 2024-02-21T22:50:20.045 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/nlmon.ko 2024-02-21T22:50:20.045 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/pcs 2024-02-21T22:50:20.045 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/pcs/pcs-lynx.ko 2024-02-21T22:50:20.046 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-02-21T22:50:20.046 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/phy 2024-02-21T22:50:20.046 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/phy/amd.ko 2024-02-21T22:50:20.046 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-02-21T22:50:20.046 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-02-21T22:50:20.046 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/phy/bcm7xxx.ko 2024-02-21T22:50:20.046 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/phy/bcm87xx.ko 2024-02-21T22:50:20.046 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/phy/broadcom.ko 2024-02-21T22:50:20.046 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/phy/cicada.ko 2024-02-21T22:50:20.046 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/phy/davicom.ko 2024-02-21T22:50:20.046 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/phy/dp83640.ko 2024-02-21T22:50:20.046 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/phy/et1011c.ko 2024-02-21T22:50:20.046 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/phy/icplus.ko 2024-02-21T22:50:20.046 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/phy/lxt.ko 2024-02-21T22:50:20.046 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/phy/marvell.ko 2024-02-21T22:50:20.047 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/phy/micrel.ko 2024-02-21T22:50:20.047 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/phy/national.ko 2024-02-21T22:50:20.047 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/phy/phylink.ko 2024-02-21T22:50:20.047 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/phy/qsemi.ko 2024-02-21T22:50:20.047 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/phy/realtek.ko 2024-02-21T22:50:20.047 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/phy/smsc.ko 2024-02-21T22:50:20.047 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/phy/ste10Xp.ko 2024-02-21T22:50:20.047 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/phy/vitesse.ko 2024-02-21T22:50:20.047 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ppp 2024-02-21T22:50:20.047 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ppp/bsd_comp.ko 2024-02-21T22:50:20.047 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ppp/ppp_async.ko 2024-02-21T22:50:20.047 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ppp/ppp_deflate.ko 2024-02-21T22:50:20.047 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ppp/ppp_generic.ko 2024-02-21T22:50:20.047 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ppp/ppp_mppe.ko 2024-02-21T22:50:20.047 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ppp/ppp_synctty.ko 2024-02-21T22:50:20.048 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ppp/pppoe.ko 2024-02-21T22:50:20.048 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ppp/pppox.ko 2024-02-21T22:50:20.048 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/ppp/pptp.ko 2024-02-21T22:50:20.048 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/slip 2024-02-21T22:50:20.048 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/slip/slhc.ko 2024-02-21T22:50:20.048 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/slip/slip.ko 2024-02-21T22:50:20.048 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/sungem_phy.ko 2024-02-21T22:50:20.048 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/tap.ko 2024-02-21T22:50:20.048 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/team 2024-02-21T22:50:20.048 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/team/team.ko 2024-02-21T22:50:20.048 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/team/team_mode_activebackup.ko 2024-02-21T22:50:20.048 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/team/team_mode_broadcast.ko 2024-02-21T22:50:20.048 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-02-21T22:50:20.049 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/team/team_mode_random.ko 2024-02-21T22:50:20.049 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-02-21T22:50:20.049 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/tun.ko 2024-02-21T22:50:20.049 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/veth.ko 2024-02-21T22:50:20.049 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/virtio_net.ko 2024-02-21T22:50:20.049 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/vmxnet3 2024-02-21T22:50:20.049 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-02-21T22:50:20.049 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/vxlan 2024-02-21T22:50:20.049 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/vxlan/vxlan.ko 2024-02-21T22:50:20.049 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/xen-netback 2024-02-21T22:50:20.049 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/xen-netback/xen-netback.ko 2024-02-21T22:50:20.049 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/net/xen-netfront.ko 2024-02-21T22:50:20.049 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ntb 2024-02-21T22:50:20.050 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ntb/ntb.ko 2024-02-21T22:50:20.050 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/nvme 2024-02-21T22:50:20.050 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/nvme/host 2024-02-21T22:50:20.050 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/nvme/host/nvme-core.ko 2024-02-21T22:50:20.050 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/nvme/host/nvme.ko 2024-02-21T22:50:20.050 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/parport 2024-02-21T22:50:20.050 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/parport/parport.ko 2024-02-21T22:50:20.050 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/parport/parport_cs.ko 2024-02-21T22:50:20.050 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/parport/parport_pc.ko 2024-02-21T22:50:20.050 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/parport/parport_serial.ko 2024-02-21T22:50:20.050 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/pci 2024-02-21T22:50:20.050 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/pci/hotplug 2024-02-21T22:50:20.050 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-02-21T22:50:20.051 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/pci/pcie 2024-02-21T22:50:20.051 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/pci/pcie/aer_inject.ko 2024-02-21T22:50:20.051 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/pci/xen-pcifront.ko 2024-02-21T22:50:20.051 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/pcmcia 2024-02-21T22:50:20.051 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/pcmcia/i82092.ko 2024-02-21T22:50:20.051 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/pcmcia/pd6729.ko 2024-02-21T22:50:20.051 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/pcmcia/yenta_socket.ko 2024-02-21T22:50:20.051 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/pinctrl 2024-02-21T22:50:20.051 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/pinctrl/intel 2024-02-21T22:50:20.051 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-02-21T22:50:20.051 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/platform 2024-02-21T22:50:20.051 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/platform/x86 2024-02-21T22:50:20.051 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/platform/x86/acer-wmi.ko 2024-02-21T22:50:20.052 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/platform/x86/acerhdf.ko 2024-02-21T22:50:20.052 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/platform/x86/apple-gmux.ko 2024-02-21T22:50:20.052 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/platform/x86/asus-laptop.ko 2024-02-21T22:50:20.052 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-02-21T22:50:20.052 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/platform/x86/asus-wmi.ko 2024-02-21T22:50:20.052 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/platform/x86/classmate-laptop.ko 2024-02-21T22:50:20.052 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-02-21T22:50:20.052 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-02-21T22:50:20.052 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-02-21T22:50:20.052 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-02-21T22:50:20.052 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/platform/x86/hdaps.ko 2024-02-21T22:50:20.052 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/platform/x86/intel 2024-02-21T22:50:20.052 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-02-21T22:50:20.052 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/platform/x86/intel_ips.ko 2024-02-21T22:50:20.053 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/platform/x86/msi-wmi.ko 2024-02-21T22:50:20.053 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/platform/x86/mxm-wmi.ko 2024-02-21T22:50:20.053 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-02-21T22:50:20.053 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/platform/x86/samsung-laptop.ko 2024-02-21T22:50:20.053 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/platform/x86/samsung-q10.ko 2024-02-21T22:50:20.053 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/platform/x86/topstar-laptop.ko 2024-02-21T22:50:20.053 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-02-21T22:50:20.053 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/platform/x86/toshiba_haps.ko 2024-02-21T22:50:20.053 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/platform/x86/wmi-bmof.ko 2024-02-21T22:50:20.053 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/platform/x86/wmi.ko 2024-02-21T22:50:20.053 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/powercap 2024-02-21T22:50:20.053 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/powercap/intel_rapl_common.ko 2024-02-21T22:50:20.053 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/powercap/intel_rapl_msr.ko 2024-02-21T22:50:20.053 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/pps 2024-02-21T22:50:20.053 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/pps/clients 2024-02-21T22:50:20.054 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/pps/clients/pps-gpio.ko 2024-02-21T22:50:20.054 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/pps/clients/pps-ldisc.ko 2024-02-21T22:50:20.054 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/pps/clients/pps_parport.ko 2024-02-21T22:50:20.054 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/pps/pps_core.ko 2024-02-21T22:50:20.054 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ptp 2024-02-21T22:50:20.054 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ptp/ptp.ko 2024-02-21T22:50:20.054 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ptp/ptp_kvm.ko 2024-02-21T22:50:20.054 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/rtc 2024-02-21T22:50:20.054 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/rtc/rtc-bq32k.ko 2024-02-21T22:50:20.054 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/rtc/rtc-ds1286.ko 2024-02-21T22:50:20.054 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/rtc/rtc-ds1307.ko 2024-02-21T22:50:20.054 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/rtc/rtc-ds1374.ko 2024-02-21T22:50:20.055 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/rtc/rtc-ds1511.ko 2024-02-21T22:50:20.055 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/rtc/rtc-ds1553.ko 2024-02-21T22:50:20.055 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/rtc/rtc-ds1672.ko 2024-02-21T22:50:20.055 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/rtc/rtc-ds1742.ko 2024-02-21T22:50:20.055 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/rtc/rtc-ds2404.ko 2024-02-21T22:50:20.055 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/rtc/rtc-ds3232.ko 2024-02-21T22:50:20.055 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/rtc/rtc-em3027.ko 2024-02-21T22:50:20.055 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/rtc/rtc-fm3130.ko 2024-02-21T22:50:20.055 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/rtc/rtc-isl12022.ko 2024-02-21T22:50:20.055 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/rtc/rtc-isl1208.ko 2024-02-21T22:50:20.055 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/rtc/rtc-m41t80.ko 2024-02-21T22:50:20.055 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/rtc/rtc-m48t35.ko 2024-02-21T22:50:20.056 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/rtc/rtc-m48t59.ko 2024-02-21T22:50:20.056 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/rtc/rtc-max6900.ko 2024-02-21T22:50:20.056 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/rtc/rtc-msm6242.ko 2024-02-21T22:50:20.056 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/rtc/rtc-pcf2127.ko 2024-02-21T22:50:20.056 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/rtc/rtc-pcf85063.ko 2024-02-21T22:50:20.056 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/rtc/rtc-pcf8523.ko 2024-02-21T22:50:20.056 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/rtc/rtc-pcf8563.ko 2024-02-21T22:50:20.056 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/rtc/rtc-pcf8583.ko 2024-02-21T22:50:20.056 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/rtc/rtc-rp5c01.ko 2024-02-21T22:50:20.056 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/rtc/rtc-rs5c372.ko 2024-02-21T22:50:20.057 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-02-21T22:50:20.057 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/rtc/rtc-rx8025.ko 2024-02-21T22:50:20.057 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/rtc/rtc-rx8581.ko 2024-02-21T22:50:20.057 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-02-21T22:50:20.057 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/rtc/rtc-x1205.ko 2024-02-21T22:50:20.057 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi 2024-02-21T22:50:20.057 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/3w-9xxx.ko 2024-02-21T22:50:20.057 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/3w-sas.ko 2024-02-21T22:50:20.057 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/3w-xxxx.ko 2024-02-21T22:50:20.057 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/BusLogic.ko 2024-02-21T22:50:20.057 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/a100u2w.ko 2024-02-21T22:50:20.057 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/aacraid 2024-02-21T22:50:20.057 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/aacraid/aacraid.ko 2024-02-21T22:50:20.057 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/advansys.ko 2024-02-21T22:50:20.057 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/aic7xxx 2024-02-21T22:50:20.058 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-02-21T22:50:20.058 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-02-21T22:50:20.058 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/am53c974.ko 2024-02-21T22:50:20.058 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/arcmsr 2024-02-21T22:50:20.058 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-02-21T22:50:20.058 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/atp870u.ko 2024-02-21T22:50:20.058 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/be2iscsi 2024-02-21T22:50:20.058 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-02-21T22:50:20.058 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/bfa 2024-02-21T22:50:20.058 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/bfa/bfa.ko 2024-02-21T22:50:20.058 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/bnx2fc 2024-02-21T22:50:20.058 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-02-21T22:50:20.058 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/bnx2i 2024-02-21T22:50:20.058 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-02-21T22:50:20.058 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/ch.ko 2024-02-21T22:50:20.058 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/csiostor 2024-02-21T22:50:20.059 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/csiostor/csiostor.ko 2024-02-21T22:50:20.059 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/cxgbi 2024-02-21T22:50:20.059 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/cxgbi/cxgb3i 2024-02-21T22:50:20.059 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-02-21T22:50:20.059 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/cxgbi/cxgb4i 2024-02-21T22:50:20.059 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-02-21T22:50:20.059 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-02-21T22:50:20.059 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/dc395x.ko 2024-02-21T22:50:20.059 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/device_handler 2024-02-21T22:50:20.059 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-02-21T22:50:20.059 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-02-21T22:50:20.059 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-02-21T22:50:20.059 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-02-21T22:50:20.059 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/dmx3191d.ko 2024-02-21T22:50:20.060 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/esas2r 2024-02-21T22:50:20.060 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/esas2r/esas2r.ko 2024-02-21T22:50:20.060 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/esp_scsi.ko 2024-02-21T22:50:20.060 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/fcoe 2024-02-21T22:50:20.060 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/fcoe/fcoe.ko 2024-02-21T22:50:20.060 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-02-21T22:50:20.060 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/fnic 2024-02-21T22:50:20.060 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/fnic/fnic.ko 2024-02-21T22:50:20.060 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/hpsa.ko 2024-02-21T22:50:20.060 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/hptiop.ko 2024-02-21T22:50:20.060 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/hv_storvsc.ko 2024-02-21T22:50:20.060 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/initio.ko 2024-02-21T22:50:20.060 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/ipr.ko 2024-02-21T22:50:20.061 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/ips.ko 2024-02-21T22:50:20.061 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/isci 2024-02-21T22:50:20.061 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/isci/isci.ko 2024-02-21T22:50:20.061 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-02-21T22:50:20.061 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/iscsi_tcp.ko 2024-02-21T22:50:20.061 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/libfc 2024-02-21T22:50:20.061 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/libfc/libfc.ko 2024-02-21T22:50:20.061 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/libiscsi.ko 2024-02-21T22:50:20.061 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/libiscsi_tcp.ko 2024-02-21T22:50:20.061 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/libsas 2024-02-21T22:50:20.061 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/libsas/libsas.ko 2024-02-21T22:50:20.061 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/lpfc 2024-02-21T22:50:20.061 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/lpfc/lpfc.ko 2024-02-21T22:50:20.062 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/megaraid 2024-02-21T22:50:20.062 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/megaraid.ko 2024-02-21T22:50:20.062 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-02-21T22:50:20.062 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-02-21T22:50:20.062 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-02-21T22:50:20.062 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/mpt3sas 2024-02-21T22:50:20.062 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-02-21T22:50:20.062 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/mvsas 2024-02-21T22:50:20.062 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/mvsas/mvsas.ko 2024-02-21T22:50:20.062 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/mvumi.ko 2024-02-21T22:50:20.062 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/pm8001 2024-02-21T22:50:20.062 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-02-21T22:50:20.062 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/pmcraid.ko 2024-02-21T22:50:20.062 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/qla1280.ko 2024-02-21T22:50:20.063 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/qla2xxx 2024-02-21T22:50:20.063 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-02-21T22:50:20.063 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-02-21T22:50:20.063 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/qla4xxx 2024-02-21T22:50:20.063 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-02-21T22:50:20.063 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/raid_class.ko 2024-02-21T22:50:20.063 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/scsi_debug.ko 2024-02-21T22:50:20.063 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/scsi_transport_fc.ko 2024-02-21T22:50:20.063 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-02-21T22:50:20.063 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/scsi_transport_sas.ko 2024-02-21T22:50:20.063 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/scsi_transport_spi.ko 2024-02-21T22:50:20.063 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/scsi_transport_srp.ko 2024-02-21T22:50:20.063 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/ses.ko 2024-02-21T22:50:20.063 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/st.ko 2024-02-21T22:50:20.063 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/stex.ko 2024-02-21T22:50:20.064 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/sym53c8xx_2 2024-02-21T22:50:20.064 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-02-21T22:50:20.064 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/virtio_scsi.ko 2024-02-21T22:50:20.064 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/vmw_pvscsi.ko 2024-02-21T22:50:20.064 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/scsi/wd719x.ko 2024-02-21T22:50:20.064 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ssb 2024-02-21T22:50:20.064 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ssb/ssb.ko 2024-02-21T22:50:20.064 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/target 2024-02-21T22:50:20.065 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/target/iscsi 2024-02-21T22:50:20.065 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-02-21T22:50:20.065 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/target/loopback 2024-02-21T22:50:20.065 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/target/loopback/tcm_loop.ko 2024-02-21T22:50:20.065 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/target/target_core_file.ko 2024-02-21T22:50:20.065 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/target/target_core_iblock.ko 2024-02-21T22:50:20.065 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/target/target_core_mod.ko 2024-02-21T22:50:20.065 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/target/target_core_pscsi.ko 2024-02-21T22:50:20.065 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/target/target_core_user.ko 2024-02-21T22:50:20.065 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/target/tcm_fc 2024-02-21T22:50:20.065 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-02-21T22:50:20.065 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/thermal 2024-02-21T22:50:20.065 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/thermal/intel 2024-02-21T22:50:20.065 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/thermal/intel/int340x_thermal 2024-02-21T22:50:20.066 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-02-21T22:50:20.066 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-02-21T22:50:20.066 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-02-21T22:50:20.066 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-02-21T22:50:20.066 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-02-21T22:50:20.066 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-02-21T22:50:20.066 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-02-21T22:50:20.066 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-02-21T22:50:20.066 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-02-21T22:50:20.066 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-02-21T22:50:20.066 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-02-21T22:50:20.066 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-02-21T22:50:20.066 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-02-21T22:50:20.066 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-02-21T22:50:20.066 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-02-21T22:50:20.067 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-02-21T22:50:20.067 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-02-21T22:50:20.067 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-02-21T22:50:20.067 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/tty 2024-02-21T22:50:20.067 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/tty/ipwireless 2024-02-21T22:50:20.067 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-02-21T22:50:20.067 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/tty/n_gsm.ko 2024-02-21T22:50:20.067 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/tty/n_hdlc.ko 2024-02-21T22:50:20.067 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/tty/nozomi.ko 2024-02-21T22:50:20.067 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/tty/serial 2024-02-21T22:50:20.067 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/tty/serial/8250 2024-02-21T22:50:20.067 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-02-21T22:50:20.067 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/tty/serial/arc_uart.ko 2024-02-21T22:50:20.067 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/tty/serial/jsm 2024-02-21T22:50:20.068 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/tty/serial/jsm/jsm.ko 2024-02-21T22:50:20.068 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ufs 2024-02-21T22:50:20.068 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ufs/core 2024-02-21T22:50:20.068 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ufs/core/ufshcd-core.ko 2024-02-21T22:50:20.068 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ufs/host 2024-02-21T22:50:20.068 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-02-21T22:50:20.068 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/uio 2024-02-21T22:50:20.068 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/uio/uio.ko 2024-02-21T22:50:20.068 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/uio/uio_aec.ko 2024-02-21T22:50:20.068 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/uio/uio_cif.ko 2024-02-21T22:50:20.068 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/uio/uio_pci_generic.ko 2024-02-21T22:50:20.068 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/uio/uio_sercos3.ko 2024-02-21T22:50:20.068 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb 2024-02-21T22:50:20.069 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/class 2024-02-21T22:50:20.069 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/class/cdc-acm.ko 2024-02-21T22:50:20.069 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/class/cdc-wdm.ko 2024-02-21T22:50:20.069 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/class/usblp.ko 2024-02-21T22:50:20.069 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/class/usbtmc.ko 2024-02-21T22:50:20.069 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/host 2024-02-21T22:50:20.069 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/host/sl811-hcd.ko 2024-02-21T22:50:20.069 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/image 2024-02-21T22:50:20.069 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/image/mdc800.ko 2024-02-21T22:50:20.069 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/image/microtek.ko 2024-02-21T22:50:20.069 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/misc 2024-02-21T22:50:20.069 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/misc/adutux.ko 2024-02-21T22:50:20.069 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/misc/appledisplay.ko 2024-02-21T22:50:20.069 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/misc/emi26.ko 2024-02-21T22:50:20.069 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/misc/emi62.ko 2024-02-21T22:50:20.070 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/misc/ezusb.ko 2024-02-21T22:50:20.070 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/misc/idmouse.ko 2024-02-21T22:50:20.070 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/misc/iowarrior.ko 2024-02-21T22:50:20.070 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/misc/isight_firmware.ko 2024-02-21T22:50:20.070 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/misc/ldusb.ko 2024-02-21T22:50:20.070 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/misc/legousbtower.ko 2024-02-21T22:50:20.070 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/misc/sisusbvga 2024-02-21T22:50:20.070 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-02-21T22:50:20.070 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/misc/trancevibrator.ko 2024-02-21T22:50:20.070 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/misc/usb3503.ko 2024-02-21T22:50:20.070 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/misc/usblcd.ko 2024-02-21T22:50:20.070 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/misc/usbsevseg.ko 2024-02-21T22:50:20.070 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/misc/uss720.ko 2024-02-21T22:50:20.070 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/misc/yurex.ko 2024-02-21T22:50:20.070 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/phy 2024-02-21T22:50:20.071 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/phy/phy-generic.ko 2024-02-21T22:50:20.071 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial 2024-02-21T22:50:20.071 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/aircable.ko 2024-02-21T22:50:20.071 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/ark3116.ko 2024-02-21T22:50:20.071 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/belkin_sa.ko 2024-02-21T22:50:20.071 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/ch341.ko 2024-02-21T22:50:20.071 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/cp210x.ko 2024-02-21T22:50:20.071 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/cyberjack.ko 2024-02-21T22:50:20.071 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/cypress_m8.ko 2024-02-21T22:50:20.071 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/digi_acceleport.ko 2024-02-21T22:50:20.071 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/empeg.ko 2024-02-21T22:50:20.071 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/ftdi_sio.ko 2024-02-21T22:50:20.071 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/garmin_gps.ko 2024-02-21T22:50:20.071 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/io_edgeport.ko 2024-02-21T22:50:20.071 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/io_ti.ko 2024-02-21T22:50:20.072 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/ipaq.ko 2024-02-21T22:50:20.072 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/ipw.ko 2024-02-21T22:50:20.072 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/ir-usb.ko 2024-02-21T22:50:20.072 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-02-21T22:50:20.072 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/keyspan.ko 2024-02-21T22:50:20.072 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/keyspan_pda.ko 2024-02-21T22:50:20.073 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/kl5kusb105.ko 2024-02-21T22:50:20.073 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/kobil_sct.ko 2024-02-21T22:50:20.073 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/mct_u232.ko 2024-02-21T22:50:20.073 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/mos7720.ko 2024-02-21T22:50:20.073 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/mos7840.ko 2024-02-21T22:50:20.073 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/navman.ko 2024-02-21T22:50:20.073 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/omninet.ko 2024-02-21T22:50:20.073 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/opticon.ko 2024-02-21T22:50:20.073 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/option.ko 2024-02-21T22:50:20.073 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/oti6858.ko 2024-02-21T22:50:20.073 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/pl2303.ko 2024-02-21T22:50:20.073 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/qcaux.ko 2024-02-21T22:50:20.073 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/qcserial.ko 2024-02-21T22:50:20.073 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/quatech2.ko 2024-02-21T22:50:20.073 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/safe_serial.ko 2024-02-21T22:50:20.074 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/sierra.ko 2024-02-21T22:50:20.074 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/spcp8x5.ko 2024-02-21T22:50:20.074 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/ssu100.ko 2024-02-21T22:50:20.074 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/symbolserial.ko 2024-02-21T22:50:20.074 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-02-21T22:50:20.074 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-02-21T22:50:20.074 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/usb_debug.ko 2024-02-21T22:50:20.074 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/usb_wwan.ko 2024-02-21T22:50:20.074 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/visor.ko 2024-02-21T22:50:20.074 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/whiteheat.ko 2024-02-21T22:50:20.074 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/serial/xsens_mt.ko 2024-02-21T22:50:20.074 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/storage 2024-02-21T22:50:20.074 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/storage/uas.ko 2024-02-21T22:50:20.074 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/storage/ums-alauda.ko 2024-02-21T22:50:20.074 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/storage/ums-cypress.ko 2024-02-21T22:50:20.075 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/storage/ums-datafab.ko 2024-02-21T22:50:20.075 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-02-21T22:50:20.075 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/storage/ums-freecom.ko 2024-02-21T22:50:20.075 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/storage/ums-isd200.ko 2024-02-21T22:50:20.075 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-02-21T22:50:20.075 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/storage/ums-karma.ko 2024-02-21T22:50:20.075 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/storage/ums-onetouch.ko 2024-02-21T22:50:20.075 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/storage/ums-realtek.ko 2024-02-21T22:50:20.075 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/storage/ums-sddr09.ko 2024-02-21T22:50:20.076 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/storage/ums-sddr55.ko 2024-02-21T22:50:20.076 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/storage/ums-usbat.ko 2024-02-21T22:50:20.076 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/storage/usb-storage.ko 2024-02-21T22:50:20.076 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/usbip 2024-02-21T22:50:20.076 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/usbip/usbip-core.ko 2024-02-21T22:50:20.076 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/usbip/usbip-host.ko 2024-02-21T22:50:20.076 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-02-21T22:50:20.076 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/vfio 2024-02-21T22:50:20.076 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/vfio/pci 2024-02-21T22:50:20.076 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-02-21T22:50:20.076 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/vfio/pci/vfio-pci.ko 2024-02-21T22:50:20.076 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/vfio/vfio.ko 2024-02-21T22:50:20.076 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-02-21T22:50:20.077 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/vhost 2024-02-21T22:50:20.077 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/vhost/vhost.ko 2024-02-21T22:50:20.077 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/vhost/vhost_iotlb.ko 2024-02-21T22:50:20.077 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/vhost/vhost_net.ko 2024-02-21T22:50:20.077 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/vhost/vhost_scsi.ko 2024-02-21T22:50:20.077 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/video 2024-02-21T22:50:20.077 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/video/backlight 2024-02-21T22:50:20.077 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/video/backlight/apple_bl.ko 2024-02-21T22:50:20.077 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/video/backlight/lcd.ko 2024-02-21T22:50:20.077 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/video/backlight/platform_lcd.ko 2024-02-21T22:50:20.077 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/virtio 2024-02-21T22:50:20.077 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/virtio/virtio.ko 2024-02-21T22:50:20.077 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/virtio/virtio_balloon.ko 2024-02-21T22:50:20.077 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/virtio/virtio_mmio.ko 2024-02-21T22:50:20.078 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/virtio/virtio_pci.ko 2024-02-21T22:50:20.078 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-02-21T22:50:20.078 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-02-21T22:50:20.078 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/virtio/virtio_ring.ko 2024-02-21T22:50:20.078 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/w1 2024-02-21T22:50:20.078 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/w1/masters 2024-02-21T22:50:20.078 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/w1/masters/ds2482.ko 2024-02-21T22:50:20.078 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/w1/masters/ds2490.ko 2024-02-21T22:50:20.078 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/w1/slaves 2024-02-21T22:50:20.078 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-02-21T22:50:20.078 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-02-21T22:50:20.078 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-02-21T22:50:20.078 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-02-21T22:50:20.078 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-02-21T22:50:20.078 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-02-21T22:50:20.079 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-02-21T22:50:20.079 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-02-21T22:50:20.079 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-02-21T22:50:20.079 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/w1/slaves/w1_smem.ko 2024-02-21T22:50:20.079 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/w1/slaves/w1_therm.ko 2024-02-21T22:50:20.079 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/w1/wire.ko 2024-02-21T22:50:20.079 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/watchdog 2024-02-21T22:50:20.079 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/watchdog/alim1535_wdt.ko 2024-02-21T22:50:20.079 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/watchdog/alim7101_wdt.ko 2024-02-21T22:50:20.079 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/watchdog/f71808e_wdt.ko 2024-02-21T22:50:20.079 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/watchdog/hpwdt.ko 2024-02-21T22:50:20.079 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/watchdog/i6300esb.ko 2024-02-21T22:50:20.079 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-02-21T22:50:20.079 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/watchdog/iTCO_wdt.ko 2024-02-21T22:50:20.080 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/watchdog/ib700wdt.ko 2024-02-21T22:50:20.080 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/watchdog/ibmasr.ko 2024-02-21T22:50:20.080 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-02-21T22:50:20.080 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/watchdog/it8712f_wdt.ko 2024-02-21T22:50:20.080 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/watchdog/it87_wdt.ko 2024-02-21T22:50:20.080 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/watchdog/machzwd.ko 2024-02-21T22:50:20.080 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/watchdog/nv_tco.ko 2024-02-21T22:50:20.080 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/watchdog/pcwd_pci.ko 2024-02-21T22:50:20.080 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/watchdog/pcwd_usb.ko 2024-02-21T22:50:20.080 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-02-21T22:50:20.080 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/watchdog/sch311x_wdt.ko 2024-02-21T22:50:20.080 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/watchdog/softdog.ko 2024-02-21T22:50:20.081 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/watchdog/sp5100_tco.ko 2024-02-21T22:50:20.081 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/watchdog/via_wdt.ko 2024-02-21T22:50:20.081 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-02-21T22:50:20.081 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/watchdog/w83877f_wdt.ko 2024-02-21T22:50:20.081 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/watchdog/w83977f_wdt.ko 2024-02-21T22:50:20.081 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/watchdog/wdt_pci.ko 2024-02-21T22:50:20.081 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/watchdog/xen_wdt.ko 2024-02-21T22:50:20.081 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/xen 2024-02-21T22:50:20.081 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/xen/xen-acpi-processor.ko 2024-02-21T22:50:20.081 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/xen/xen-evtchn.ko 2024-02-21T22:50:20.081 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/xen/xen-gntalloc.ko 2024-02-21T22:50:20.081 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/xen/xen-gntdev.ko 2024-02-21T22:50:20.081 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/xen/xen-pciback 2024-02-21T22:50:20.081 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-02-21T22:50:20.082 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/xen/xen-privcmd.ko 2024-02-21T22:50:20.082 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/xen/xenfs 2024-02-21T22:50:20.082 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/drivers/xen/xenfs/xenfs.ko 2024-02-21T22:50:20.082 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs 2024-02-21T22:50:20.082 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/9p 2024-02-21T22:50:20.082 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/9p/9p.ko 2024-02-21T22:50:20.082 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/affs 2024-02-21T22:50:20.082 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/affs/affs.ko 2024-02-21T22:50:20.083 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/befs 2024-02-21T22:50:20.083 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/befs/befs.ko 2024-02-21T22:50:20.083 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/binfmt_misc.ko 2024-02-21T22:50:20.083 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/btrfs 2024-02-21T22:50:20.083 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/btrfs/btrfs.ko 2024-02-21T22:50:20.083 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/cachefiles 2024-02-21T22:50:20.083 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/cachefiles/cachefiles.ko 2024-02-21T22:50:20.083 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/ceph 2024-02-21T22:50:20.083 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/ceph/ceph.ko 2024-02-21T22:50:20.083 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/coda 2024-02-21T22:50:20.083 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/coda/coda.ko 2024-02-21T22:50:20.083 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/cramfs 2024-02-21T22:50:20.083 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/cramfs/cramfs.ko 2024-02-21T22:50:20.083 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/dlm 2024-02-21T22:50:20.084 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/dlm/dlm.ko 2024-02-21T22:50:20.084 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/ecryptfs 2024-02-21T22:50:20.084 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/ecryptfs/ecryptfs.ko 2024-02-21T22:50:20.084 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/f2fs 2024-02-21T22:50:20.084 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/f2fs/f2fs.ko 2024-02-21T22:50:20.084 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/fat 2024-02-21T22:50:20.084 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/fat/fat.ko 2024-02-21T22:50:20.084 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/fat/msdos.ko 2024-02-21T22:50:20.084 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/fat/vfat.ko 2024-02-21T22:50:20.084 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/fuse 2024-02-21T22:50:20.084 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/fuse/cuse.ko 2024-02-21T22:50:20.084 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/fuse/fuse.ko 2024-02-21T22:50:20.084 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/gfs2 2024-02-21T22:50:20.085 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/gfs2/gfs2.ko 2024-02-21T22:50:20.085 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/hfs 2024-02-21T22:50:20.085 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/hfs/hfs.ko 2024-02-21T22:50:20.085 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/hfsplus 2024-02-21T22:50:20.085 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/hfsplus/hfsplus.ko 2024-02-21T22:50:20.085 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/isofs 2024-02-21T22:50:20.085 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/isofs/isofs.ko 2024-02-21T22:50:20.085 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/jfs 2024-02-21T22:50:20.085 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/jfs/jfs.ko 2024-02-21T22:50:20.085 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/lockd 2024-02-21T22:50:20.085 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/lockd/lockd.ko 2024-02-21T22:50:20.085 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/minix 2024-02-21T22:50:20.085 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/minix/minix.ko 2024-02-21T22:50:20.085 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/netfs 2024-02-21T22:50:20.085 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/netfs/netfs.ko 2024-02-21T22:50:20.086 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nfs 2024-02-21T22:50:20.086 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nfs/blocklayout 2024-02-21T22:50:20.086 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-02-21T22:50:20.086 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nfs/filelayout 2024-02-21T22:50:20.086 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-02-21T22:50:20.086 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nfs/flexfilelayout 2024-02-21T22:50:20.086 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-02-21T22:50:20.086 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nfs/nfs.ko 2024-02-21T22:50:20.086 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nfs/nfsv3.ko 2024-02-21T22:50:20.086 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nfs/nfsv4.ko 2024-02-21T22:50:20.086 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nfs_common 2024-02-21T22:50:20.086 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nfs_common/grace.ko 2024-02-21T22:50:20.086 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nfs_common/nfs_acl.ko 2024-02-21T22:50:20.086 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nfsd 2024-02-21T22:50:20.086 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nfsd/nfsd.ko 2024-02-21T22:50:20.087 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nilfs2 2024-02-21T22:50:20.087 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nilfs2/nilfs2.ko 2024-02-21T22:50:20.087 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls 2024-02-21T22:50:20.087 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/mac-celtic.ko 2024-02-21T22:50:20.087 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/mac-centeuro.ko 2024-02-21T22:50:20.087 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/mac-croatian.ko 2024-02-21T22:50:20.087 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/mac-cyrillic.ko 2024-02-21T22:50:20.087 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/mac-gaelic.ko 2024-02-21T22:50:20.087 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/mac-greek.ko 2024-02-21T22:50:20.087 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/mac-iceland.ko 2024-02-21T22:50:20.087 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/mac-inuit.ko 2024-02-21T22:50:20.087 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/mac-roman.ko 2024-02-21T22:50:20.087 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/mac-romanian.ko 2024-02-21T22:50:20.087 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/mac-turkish.ko 2024-02-21T22:50:20.087 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_cp1250.ko 2024-02-21T22:50:20.088 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_cp1251.ko 2024-02-21T22:50:20.088 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_cp1255.ko 2024-02-21T22:50:20.088 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_cp737.ko 2024-02-21T22:50:20.088 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_cp775.ko 2024-02-21T22:50:20.088 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_cp850.ko 2024-02-21T22:50:20.088 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_cp852.ko 2024-02-21T22:50:20.088 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_cp855.ko 2024-02-21T22:50:20.088 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_cp857.ko 2024-02-21T22:50:20.088 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_cp860.ko 2024-02-21T22:50:20.088 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_cp861.ko 2024-02-21T22:50:20.088 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_cp862.ko 2024-02-21T22:50:20.088 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_cp863.ko 2024-02-21T22:50:20.088 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_cp864.ko 2024-02-21T22:50:20.088 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_cp865.ko 2024-02-21T22:50:20.089 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_cp866.ko 2024-02-21T22:50:20.089 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_cp869.ko 2024-02-21T22:50:20.089 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_cp874.ko 2024-02-21T22:50:20.089 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_cp932.ko 2024-02-21T22:50:20.089 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_cp936.ko 2024-02-21T22:50:20.089 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_cp949.ko 2024-02-21T22:50:20.089 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_cp950.ko 2024-02-21T22:50:20.089 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_euc-jp.ko 2024-02-21T22:50:20.089 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_iso8859-1.ko 2024-02-21T22:50:20.089 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_iso8859-13.ko 2024-02-21T22:50:20.089 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_iso8859-14.ko 2024-02-21T22:50:20.089 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_iso8859-15.ko 2024-02-21T22:50:20.089 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_iso8859-2.ko 2024-02-21T22:50:20.089 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_iso8859-3.ko 2024-02-21T22:50:20.090 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_iso8859-4.ko 2024-02-21T22:50:20.090 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_iso8859-5.ko 2024-02-21T22:50:20.090 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_iso8859-6.ko 2024-02-21T22:50:20.090 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_iso8859-7.ko 2024-02-21T22:50:20.090 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_iso8859-9.ko 2024-02-21T22:50:20.090 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_koi8-r.ko 2024-02-21T22:50:20.090 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_koi8-ru.ko 2024-02-21T22:50:20.090 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_koi8-u.ko 2024-02-21T22:50:20.091 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_ucs2_utils.ko 2024-02-21T22:50:20.091 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/nls/nls_utf8.ko 2024-02-21T22:50:20.092 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/ocfs2 2024-02-21T22:50:20.092 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/ocfs2/cluster 2024-02-21T22:50:20.092 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-02-21T22:50:20.092 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/ocfs2/dlm 2024-02-21T22:50:20.092 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-02-21T22:50:20.092 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/ocfs2/dlmfs 2024-02-21T22:50:20.092 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-02-21T22:50:20.092 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/ocfs2/ocfs2.ko 2024-02-21T22:50:20.092 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-02-21T22:50:20.092 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-02-21T22:50:20.092 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-02-21T22:50:20.092 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/overlayfs 2024-02-21T22:50:20.092 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/overlayfs/overlay.ko 2024-02-21T22:50:20.093 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/pstore 2024-02-21T22:50:20.093 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/pstore/ramoops.ko 2024-02-21T22:50:20.093 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/reiserfs 2024-02-21T22:50:20.093 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/reiserfs/reiserfs.ko 2024-02-21T22:50:20.093 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/romfs 2024-02-21T22:50:20.093 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/romfs/romfs.ko 2024-02-21T22:50:20.093 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/smb 2024-02-21T22:50:20.093 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/smb/client 2024-02-21T22:50:20.093 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/smb/client/cifs.ko 2024-02-21T22:50:20.093 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/smb/common 2024-02-21T22:50:20.093 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/smb/common/cifs_arc4.ko 2024-02-21T22:50:20.093 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/smb/common/cifs_md4.ko 2024-02-21T22:50:20.093 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/squashfs 2024-02-21T22:50:20.093 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/squashfs/squashfs.ko 2024-02-21T22:50:20.094 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/sysv 2024-02-21T22:50:20.094 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/sysv/sysv.ko 2024-02-21T22:50:20.094 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/udf 2024-02-21T22:50:20.094 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/udf/udf.ko 2024-02-21T22:50:20.094 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/ufs 2024-02-21T22:50:20.094 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/ufs/ufs.ko 2024-02-21T22:50:20.094 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/xfs 2024-02-21T22:50:20.094 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/fs/xfs/xfs.ko 2024-02-21T22:50:20.094 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/kernel 2024-02-21T22:50:20.094 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/kernel/rcu 2024-02-21T22:50:20.094 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/kernel/rcu/rcutorture.ko 2024-02-21T22:50:20.094 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/kernel/torture.ko 2024-02-21T22:50:20.094 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/kernel/trace 2024-02-21T22:50:20.094 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-02-21T22:50:20.095 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/lib 2024-02-21T22:50:20.095 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/lib/asn1_encoder.ko 2024-02-21T22:50:20.095 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/lib/crc-itu-t.ko 2024-02-21T22:50:20.095 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/lib/crc8.ko 2024-02-21T22:50:20.095 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/lib/crypto 2024-02-21T22:50:20.095 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/lib/crypto/libarc4.ko 2024-02-21T22:50:20.095 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/lib/crypto/libdes.ko 2024-02-21T22:50:20.095 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/lib/libcrc32c.ko 2024-02-21T22:50:20.095 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/lib/lru_cache.ko 2024-02-21T22:50:20.095 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/lib/lz4 2024-02-21T22:50:20.095 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/lib/lz4/lz4_compress.ko 2024-02-21T22:50:20.095 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/lib/lz4/lz4hc_compress.ko 2024-02-21T22:50:20.095 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/lib/math 2024-02-21T22:50:20.095 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/lib/math/cordic.ko 2024-02-21T22:50:20.096 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/lib/raid6 2024-02-21T22:50:20.096 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/lib/raid6/raid6_pq.ko 2024-02-21T22:50:20.096 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/lib/reed_solomon 2024-02-21T22:50:20.096 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/lib/reed_solomon/reed_solomon.ko 2024-02-21T22:50:20.096 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/lib/ts_bm.ko 2024-02-21T22:50:20.096 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/lib/ts_fsm.ko 2024-02-21T22:50:20.096 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/lib/ts_kmp.ko 2024-02-21T22:50:20.096 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/lib/zstd 2024-02-21T22:50:20.096 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/lib/zstd/zstd_compress.ko 2024-02-21T22:50:20.096 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/mm 2024-02-21T22:50:20.096 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/mm/hwpoison-inject.ko 2024-02-21T22:50:20.096 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net 2024-02-21T22:50:20.096 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/6lowpan 2024-02-21T22:50:20.097 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/6lowpan/6lowpan.ko 2024-02-21T22:50:20.097 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/6lowpan/nhc_dest.ko 2024-02-21T22:50:20.097 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/6lowpan/nhc_fragment.ko 2024-02-21T22:50:20.097 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/6lowpan/nhc_hop.ko 2024-02-21T22:50:20.098 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/6lowpan/nhc_ipv6.ko 2024-02-21T22:50:20.098 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/6lowpan/nhc_mobility.ko 2024-02-21T22:50:20.098 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/6lowpan/nhc_routing.ko 2024-02-21T22:50:20.098 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/6lowpan/nhc_udp.ko 2024-02-21T22:50:20.098 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/802 2024-02-21T22:50:20.098 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/802/garp.ko 2024-02-21T22:50:20.098 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/802/mrp.ko 2024-02-21T22:50:20.098 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/802/p8022.ko 2024-02-21T22:50:20.098 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/802/psnap.ko 2024-02-21T22:50:20.098 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/802/stp.ko 2024-02-21T22:50:20.098 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/8021q 2024-02-21T22:50:20.098 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/8021q/8021q.ko 2024-02-21T22:50:20.098 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/9p 2024-02-21T22:50:20.098 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/9p/9pnet.ko 2024-02-21T22:50:20.099 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/9p/9pnet_fd.ko 2024-02-21T22:50:20.099 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/9p/9pnet_rdma.ko 2024-02-21T22:50:20.099 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/9p/9pnet_virtio.ko 2024-02-21T22:50:20.099 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/batman-adv 2024-02-21T22:50:20.099 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/batman-adv/batman-adv.ko 2024-02-21T22:50:20.099 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/bridge 2024-02-21T22:50:20.099 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/bridge/br_netfilter.ko 2024-02-21T22:50:20.099 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/bridge/bridge.ko 2024-02-21T22:50:20.099 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/bridge/netfilter 2024-02-21T22:50:20.099 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-02-21T22:50:20.099 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/bridge/netfilter/ebt_among.ko 2024-02-21T22:50:20.099 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/bridge/netfilter/ebt_arp.ko 2024-02-21T22:50:20.099 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-02-21T22:50:20.099 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-02-21T22:50:20.099 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/bridge/netfilter/ebt_ip.ko 2024-02-21T22:50:20.100 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-02-21T22:50:20.100 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/bridge/netfilter/ebt_limit.ko 2024-02-21T22:50:20.100 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/bridge/netfilter/ebt_log.ko 2024-02-21T22:50:20.100 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/bridge/netfilter/ebt_mark.ko 2024-02-21T22:50:20.100 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-02-21T22:50:20.100 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-02-21T22:50:20.100 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-02-21T22:50:20.100 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-02-21T22:50:20.101 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/bridge/netfilter/ebt_snat.ko 2024-02-21T22:50:20.101 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/bridge/netfilter/ebt_stp.ko 2024-02-21T22:50:20.101 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-02-21T22:50:20.101 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-02-21T22:50:20.101 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-02-21T22:50:20.101 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-02-21T22:50:20.101 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/bridge/netfilter/ebtables.ko 2024-02-21T22:50:20.101 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-02-21T22:50:20.102 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-02-21T22:50:20.102 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-02-21T22:50:20.102 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ceph 2024-02-21T22:50:20.102 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ceph/libceph.ko 2024-02-21T22:50:20.102 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/core 2024-02-21T22:50:20.102 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/core/failover.ko 2024-02-21T22:50:20.102 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/core/pktgen.ko 2024-02-21T22:50:20.102 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/dccp 2024-02-21T22:50:20.102 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/dccp/dccp.ko 2024-02-21T22:50:20.102 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/dccp/dccp_diag.ko 2024-02-21T22:50:20.102 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/dccp/dccp_ipv4.ko 2024-02-21T22:50:20.102 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/dccp/dccp_ipv6.ko 2024-02-21T22:50:20.102 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/dns_resolver 2024-02-21T22:50:20.102 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/dns_resolver/dns_resolver.ko 2024-02-21T22:50:20.103 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/dsa 2024-02-21T22:50:20.103 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/dsa/dsa_core.ko 2024-02-21T22:50:20.103 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/dsa/tag_brcm.ko 2024-02-21T22:50:20.103 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/dsa/tag_dsa.ko 2024-02-21T22:50:20.103 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/dsa/tag_none.ko 2024-02-21T22:50:20.103 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/dsa/tag_trailer.ko 2024-02-21T22:50:20.103 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ieee802154 2024-02-21T22:50:20.103 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ieee802154/6lowpan 2024-02-21T22:50:20.103 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-02-21T22:50:20.103 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ieee802154/ieee802154.ko 2024-02-21T22:50:20.103 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ieee802154/ieee802154_socket.ko 2024-02-21T22:50:20.103 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4 2024-02-21T22:50:20.103 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/ah4.ko 2024-02-21T22:50:20.104 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/esp4.ko 2024-02-21T22:50:20.104 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/fou.ko 2024-02-21T22:50:20.104 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/gre.ko 2024-02-21T22:50:20.104 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/inet_diag.ko 2024-02-21T22:50:20.104 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/ip_gre.ko 2024-02-21T22:50:20.104 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/ip_tunnel.ko 2024-02-21T22:50:20.104 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/ip_vti.ko 2024-02-21T22:50:20.104 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/ipcomp.ko 2024-02-21T22:50:20.104 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/ipip.ko 2024-02-21T22:50:20.104 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/netfilter 2024-02-21T22:50:20.104 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/netfilter/arp_tables.ko 2024-02-21T22:50:20.104 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-02-21T22:50:20.105 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-02-21T22:50:20.105 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-02-21T22:50:20.105 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-02-21T22:50:20.105 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-02-21T22:50:20.105 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-02-21T22:50:20.105 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-02-21T22:50:20.105 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-02-21T22:50:20.105 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-02-21T22:50:20.105 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/netfilter/iptable_security.ko 2024-02-21T22:50:20.105 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-02-21T22:50:20.105 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-02-21T22:50:20.105 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-02-21T22:50:20.105 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-02-21T22:50:20.106 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-02-21T22:50:20.106 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-02-21T22:50:20.106 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-02-21T22:50:20.106 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-02-21T22:50:20.106 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-02-21T22:50:20.106 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-02-21T22:50:20.106 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/tcp_bic.ko 2024-02-21T22:50:20.106 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/tcp_dctcp.ko 2024-02-21T22:50:20.106 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/tcp_diag.ko 2024-02-21T22:50:20.106 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/tcp_highspeed.ko 2024-02-21T22:50:20.106 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/tcp_htcp.ko 2024-02-21T22:50:20.106 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/tcp_hybla.ko 2024-02-21T22:50:20.106 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/tcp_illinois.ko 2024-02-21T22:50:20.107 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/tcp_lp.ko 2024-02-21T22:50:20.107 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/tcp_scalable.ko 2024-02-21T22:50:20.107 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/tcp_vegas.ko 2024-02-21T22:50:20.107 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/tcp_veno.ko 2024-02-21T22:50:20.107 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/tcp_westwood.ko 2024-02-21T22:50:20.107 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/tcp_yeah.ko 2024-02-21T22:50:20.107 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/tunnel4.ko 2024-02-21T22:50:20.107 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/udp_diag.ko 2024-02-21T22:50:20.107 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/udp_tunnel.ko 2024-02-21T22:50:20.107 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv4/xfrm4_tunnel.ko 2024-02-21T22:50:20.107 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6 2024-02-21T22:50:20.107 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6/ah6.ko 2024-02-21T22:50:20.108 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6/esp6.ko 2024-02-21T22:50:20.108 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6/fou6.ko 2024-02-21T22:50:20.108 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6/ip6_tunnel.ko 2024-02-21T22:50:20.108 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-02-21T22:50:20.108 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6/ip6_vti.ko 2024-02-21T22:50:20.108 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6/ipcomp6.ko 2024-02-21T22:50:20.108 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6/netfilter 2024-02-21T22:50:20.108 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-02-21T22:50:20.108 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-02-21T22:50:20.108 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-02-21T22:50:20.108 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-02-21T22:50:20.108 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-02-21T22:50:20.108 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-02-21T22:50:20.109 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-02-21T22:50:20.109 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-02-21T22:50:20.109 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-02-21T22:50:20.109 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-02-21T22:50:20.109 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-02-21T22:50:20.109 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-02-21T22:50:20.109 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-02-21T22:50:20.109 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-02-21T22:50:20.109 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-02-21T22:50:20.109 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-02-21T22:50:20.109 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-02-21T22:50:20.109 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-02-21T22:50:20.109 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-02-21T22:50:20.110 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-02-21T22:50:20.110 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-02-21T22:50:20.110 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-02-21T22:50:20.110 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-02-21T22:50:20.110 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-02-21T22:50:20.110 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6/sit.ko 2024-02-21T22:50:20.110 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6/tunnel6.ko 2024-02-21T22:50:20.110 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/ipv6/xfrm6_tunnel.ko 2024-02-21T22:50:20.110 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/key 2024-02-21T22:50:20.110 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/key/af_key.ko 2024-02-21T22:50:20.110 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/l2tp 2024-02-21T22:50:20.110 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/l2tp/l2tp_core.ko 2024-02-21T22:50:20.110 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/l2tp/l2tp_debugfs.ko 2024-02-21T22:50:20.111 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/l2tp/l2tp_eth.ko 2024-02-21T22:50:20.111 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/l2tp/l2tp_ip.ko 2024-02-21T22:50:20.111 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/l2tp/l2tp_ip6.ko 2024-02-21T22:50:20.111 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/l2tp/l2tp_netlink.ko 2024-02-21T22:50:20.111 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/l2tp/l2tp_ppp.ko 2024-02-21T22:50:20.111 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/llc 2024-02-21T22:50:20.111 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/llc/llc.ko 2024-02-21T22:50:20.111 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/mac802154 2024-02-21T22:50:20.111 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/mac802154/mac802154.ko 2024-02-21T22:50:20.111 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/mpls 2024-02-21T22:50:20.111 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/mpls/mpls_gso.ko 2024-02-21T22:50:20.111 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter 2024-02-21T22:50:20.111 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/ipset 2024-02-21T22:50:20.111 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/ipset/ip_set.ko 2024-02-21T22:50:20.111 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-02-21T22:50:20.112 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-02-21T22:50:20.112 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-02-21T22:50:20.112 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-02-21T22:50:20.112 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-02-21T22:50:20.112 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-02-21T22:50:20.112 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-02-21T22:50:20.112 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-02-21T22:50:20.112 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-02-21T22:50:20.112 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-02-21T22:50:20.112 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-02-21T22:50:20.112 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-02-21T22:50:20.112 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-02-21T22:50:20.112 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-02-21T22:50:20.113 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-02-21T22:50:20.113 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/ipvs 2024-02-21T22:50:20.113 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/ipvs/ip_vs.ko 2024-02-21T22:50:20.113 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-02-21T22:50:20.113 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-02-21T22:50:20.113 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-02-21T22:50:20.113 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-02-21T22:50:20.113 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-02-21T22:50:20.113 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-02-21T22:50:20.113 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-02-21T22:50:20.113 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-02-21T22:50:20.113 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-02-21T22:50:20.113 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-02-21T22:50:20.113 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-02-21T22:50:20.113 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-02-21T22:50:20.113 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-02-21T22:50:20.114 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nf_conncount.ko 2024-02-21T22:50:20.114 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nf_conntrack.ko 2024-02-21T22:50:20.114 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-02-21T22:50:20.114 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-02-21T22:50:20.114 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-02-21T22:50:20.114 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nf_conntrack_h323.ko 2024-02-21T22:50:20.114 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nf_conntrack_irc.ko 2024-02-21T22:50:20.114 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-02-21T22:50:20.114 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-02-21T22:50:20.114 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-02-21T22:50:20.114 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nf_conntrack_sane.ko 2024-02-21T22:50:20.114 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nf_conntrack_sip.ko 2024-02-21T22:50:20.114 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-02-21T22:50:20.114 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-02-21T22:50:20.115 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nf_dup_netdev.ko 2024-02-21T22:50:20.115 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nf_flow_table.ko 2024-02-21T22:50:20.115 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nf_flow_table_inet.ko 2024-02-21T22:50:20.115 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nf_log_syslog.ko 2024-02-21T22:50:20.115 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nf_nat.ko 2024-02-21T22:50:20.115 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nf_nat_amanda.ko 2024-02-21T22:50:20.115 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nf_nat_ftp.ko 2024-02-21T22:50:20.115 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nf_nat_irc.ko 2024-02-21T22:50:20.115 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nf_nat_sip.ko 2024-02-21T22:50:20.115 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nf_nat_tftp.ko 2024-02-21T22:50:20.115 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nf_synproxy_core.ko 2024-02-21T22:50:20.115 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nf_tables.ko 2024-02-21T22:50:20.115 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nfnetlink.ko 2024-02-21T22:50:20.115 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nfnetlink_acct.ko 2024-02-21T22:50:20.115 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nfnetlink_log.ko 2024-02-21T22:50:20.116 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nfnetlink_osf.ko 2024-02-21T22:50:20.116 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nfnetlink_queue.ko 2024-02-21T22:50:20.116 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nft_chain_nat.ko 2024-02-21T22:50:20.116 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nft_compat.ko 2024-02-21T22:50:20.116 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nft_ct.ko 2024-02-21T22:50:20.116 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nft_dup_netdev.ko 2024-02-21T22:50:20.116 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nft_fib.ko 2024-02-21T22:50:20.116 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nft_fwd_netdev.ko 2024-02-21T22:50:20.116 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nft_hash.ko 2024-02-21T22:50:20.116 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nft_limit.ko 2024-02-21T22:50:20.116 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nft_log.ko 2024-02-21T22:50:20.116 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nft_masq.ko 2024-02-21T22:50:20.116 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nft_nat.ko 2024-02-21T22:50:20.116 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nft_numgen.ko 2024-02-21T22:50:20.116 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nft_queue.ko 2024-02-21T22:50:20.117 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nft_quota.ko 2024-02-21T22:50:20.117 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nft_redir.ko 2024-02-21T22:50:20.117 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nft_reject.ko 2024-02-21T22:50:20.117 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nft_reject_inet.ko 2024-02-21T22:50:20.117 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nft_socket.ko 2024-02-21T22:50:20.117 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/nft_tunnel.ko 2024-02-21T22:50:20.117 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_AUDIT.ko 2024-02-21T22:50:20.117 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_CHECKSUM.ko 2024-02-21T22:50:20.117 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_CLASSIFY.ko 2024-02-21T22:50:20.117 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-02-21T22:50:20.117 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_CT.ko 2024-02-21T22:50:20.117 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_DSCP.ko 2024-02-21T22:50:20.117 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_HL.ko 2024-02-21T22:50:20.118 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_HMARK.ko 2024-02-21T22:50:20.118 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_IDLETIMER.ko 2024-02-21T22:50:20.118 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_LED.ko 2024-02-21T22:50:20.118 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_LOG.ko 2024-02-21T22:50:20.118 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_MASQUERADE.ko 2024-02-21T22:50:20.118 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_NETMAP.ko 2024-02-21T22:50:20.118 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_NFLOG.ko 2024-02-21T22:50:20.118 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_NFQUEUE.ko 2024-02-21T22:50:20.118 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_RATEEST.ko 2024-02-21T22:50:20.118 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_REDIRECT.ko 2024-02-21T22:50:20.118 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_SECMARK.ko 2024-02-21T22:50:20.118 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_TCPMSS.ko 2024-02-21T22:50:20.118 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-02-21T22:50:20.119 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_TEE.ko 2024-02-21T22:50:20.119 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_TPROXY.ko 2024-02-21T22:50:20.119 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_TRACE.ko 2024-02-21T22:50:20.119 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_addrtype.ko 2024-02-21T22:50:20.119 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_bpf.ko 2024-02-21T22:50:20.119 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_cgroup.ko 2024-02-21T22:50:20.119 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_cluster.ko 2024-02-21T22:50:20.119 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_comment.ko 2024-02-21T22:50:20.119 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_connbytes.ko 2024-02-21T22:50:20.119 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_connlabel.ko 2024-02-21T22:50:20.119 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_connlimit.ko 2024-02-21T22:50:20.119 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_connmark.ko 2024-02-21T22:50:20.119 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_conntrack.ko 2024-02-21T22:50:20.120 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_cpu.ko 2024-02-21T22:50:20.120 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_dccp.ko 2024-02-21T22:50:20.120 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_devgroup.ko 2024-02-21T22:50:20.120 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_dscp.ko 2024-02-21T22:50:20.120 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_ecn.ko 2024-02-21T22:50:20.120 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_esp.ko 2024-02-21T22:50:20.120 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_hashlimit.ko 2024-02-21T22:50:20.120 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_helper.ko 2024-02-21T22:50:20.120 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_hl.ko 2024-02-21T22:50:20.120 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_ipcomp.ko 2024-02-21T22:50:20.120 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_iprange.ko 2024-02-21T22:50:20.120 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_ipvs.ko 2024-02-21T22:50:20.120 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_l2tp.ko 2024-02-21T22:50:20.120 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_length.ko 2024-02-21T22:50:20.120 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_limit.ko 2024-02-21T22:50:20.121 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_mac.ko 2024-02-21T22:50:20.121 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_mark.ko 2024-02-21T22:50:20.121 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_multiport.ko 2024-02-21T22:50:20.121 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_nat.ko 2024-02-21T22:50:20.121 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_nfacct.ko 2024-02-21T22:50:20.121 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_osf.ko 2024-02-21T22:50:20.121 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_owner.ko 2024-02-21T22:50:20.121 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_physdev.ko 2024-02-21T22:50:20.121 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_pkttype.ko 2024-02-21T22:50:20.121 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_policy.ko 2024-02-21T22:50:20.121 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_quota.ko 2024-02-21T22:50:20.121 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_rateest.ko 2024-02-21T22:50:20.121 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_realm.ko 2024-02-21T22:50:20.121 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_recent.ko 2024-02-21T22:50:20.121 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_sctp.ko 2024-02-21T22:50:20.122 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_set.ko 2024-02-21T22:50:20.122 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_socket.ko 2024-02-21T22:50:20.122 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_state.ko 2024-02-21T22:50:20.122 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_statistic.ko 2024-02-21T22:50:20.122 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_string.ko 2024-02-21T22:50:20.122 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_tcpmss.ko 2024-02-21T22:50:20.122 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_time.ko 2024-02-21T22:50:20.122 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netfilter/xt_u32.ko 2024-02-21T22:50:20.122 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netlink 2024-02-21T22:50:20.122 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/netlink/netlink_diag.ko 2024-02-21T22:50:20.122 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/nsh 2024-02-21T22:50:20.122 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/nsh/nsh.ko 2024-02-21T22:50:20.122 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/openvswitch 2024-02-21T22:50:20.123 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/openvswitch/openvswitch.ko 2024-02-21T22:50:20.123 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/openvswitch/vport-geneve.ko 2024-02-21T22:50:20.123 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/openvswitch/vport-gre.ko 2024-02-21T22:50:20.123 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/openvswitch/vport-vxlan.ko 2024-02-21T22:50:20.123 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/packet 2024-02-21T22:50:20.123 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/packet/af_packet_diag.ko 2024-02-21T22:50:20.123 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/rds 2024-02-21T22:50:20.123 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/rds/rds.ko 2024-02-21T22:50:20.123 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/rds/rds_rdma.ko 2024-02-21T22:50:20.123 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/rds/rds_tcp.ko 2024-02-21T22:50:20.123 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched 2024-02-21T22:50:20.123 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/act_csum.ko 2024-02-21T22:50:20.123 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/act_gact.ko 2024-02-21T22:50:20.123 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/act_mirred.ko 2024-02-21T22:50:20.123 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/act_nat.ko 2024-02-21T22:50:20.124 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/act_pedit.ko 2024-02-21T22:50:20.124 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/act_police.ko 2024-02-21T22:50:20.124 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/act_simple.ko 2024-02-21T22:50:20.124 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/act_skbedit.ko 2024-02-21T22:50:20.124 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/act_vlan.ko 2024-02-21T22:50:20.124 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/cls_basic.ko 2024-02-21T22:50:20.124 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/cls_bpf.ko 2024-02-21T22:50:20.124 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/cls_flow.ko 2024-02-21T22:50:20.124 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/cls_fw.ko 2024-02-21T22:50:20.124 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/cls_route.ko 2024-02-21T22:50:20.124 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/cls_u32.ko 2024-02-21T22:50:20.124 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/em_cmp.ko 2024-02-21T22:50:20.124 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/em_ipset.ko 2024-02-21T22:50:20.124 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/em_meta.ko 2024-02-21T22:50:20.124 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/em_nbyte.ko 2024-02-21T22:50:20.125 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/em_text.ko 2024-02-21T22:50:20.125 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/em_u32.ko 2024-02-21T22:50:20.125 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/sch_choke.ko 2024-02-21T22:50:20.125 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/sch_codel.ko 2024-02-21T22:50:20.125 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/sch_drr.ko 2024-02-21T22:50:20.126 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/sch_fq.ko 2024-02-21T22:50:20.126 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/sch_gred.ko 2024-02-21T22:50:20.126 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/sch_hfsc.ko 2024-02-21T22:50:20.127 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/sch_hhf.ko 2024-02-21T22:50:20.127 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/sch_htb.ko 2024-02-21T22:50:20.127 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/sch_ingress.ko 2024-02-21T22:50:20.127 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/sch_mqprio.ko 2024-02-21T22:50:20.127 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/sch_mqprio_lib.ko 2024-02-21T22:50:20.127 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/sch_multiq.ko 2024-02-21T22:50:20.127 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/sch_netem.ko 2024-02-21T22:50:20.127 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/sch_pie.ko 2024-02-21T22:50:20.127 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/sch_plug.ko 2024-02-21T22:50:20.127 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/sch_prio.ko 2024-02-21T22:50:20.127 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/sch_qfq.ko 2024-02-21T22:50:20.127 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/sch_red.ko 2024-02-21T22:50:20.127 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/sch_sfb.ko 2024-02-21T22:50:20.128 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/sch_sfq.ko 2024-02-21T22:50:20.128 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/sch_tbf.ko 2024-02-21T22:50:20.128 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sched/sch_teql.ko 2024-02-21T22:50:20.128 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sctp 2024-02-21T22:50:20.128 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sctp/sctp.ko 2024-02-21T22:50:20.128 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sctp/sctp_diag.ko 2024-02-21T22:50:20.128 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sunrpc 2024-02-21T22:50:20.128 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sunrpc/auth_gss 2024-02-21T22:50:20.128 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-02-21T22:50:20.128 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-02-21T22:50:20.129 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sunrpc/sunrpc.ko 2024-02-21T22:50:20.129 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sunrpc/xprtrdma 2024-02-21T22:50:20.129 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-02-21T22:50:20.129 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/tipc 2024-02-21T22:50:20.129 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/tipc/diag.ko 2024-02-21T22:50:20.129 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/tipc/tipc.ko 2024-02-21T22:50:20.129 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/unix 2024-02-21T22:50:20.129 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/unix/unix_diag.ko 2024-02-21T22:50:20.129 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/vmw_vsock 2024-02-21T22:50:20.129 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-02-21T22:50:20.129 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-02-21T22:50:20.129 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/vmw_vsock/vsock.ko 2024-02-21T22:50:20.129 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/vmw_vsock/vsock_diag.ko 2024-02-21T22:50:20.129 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/vmw_vsock/vsock_loopback.ko 2024-02-21T22:50:20.130 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/xfrm 2024-02-21T22:50:20.130 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/net/xfrm/xfrm_ipcomp.ko 2024-02-21T22:50:20.130 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/security 2024-02-21T22:50:20.130 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/security/keys 2024-02-21T22:50:20.130 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/security/keys/encrypted-keys 2024-02-21T22:50:20.130 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-02-21T22:50:20.130 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/security/keys/trusted-keys 2024-02-21T22:50:20.130 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/security/keys/trusted-keys/trusted.ko 2024-02-21T22:50:20.130 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/virt 2024-02-21T22:50:20.130 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/virt/lib 2024-02-21T22:50:20.130 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/kernel/virt/lib/irqbypass.ko 2024-02-21T22:50:20.130 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/modules.alias 2024-02-21T22:50:20.130 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/modules.alias.bin 2024-02-21T22:50:20.130 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/modules.builtin 2024-02-21T22:50:20.131 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/modules.builtin.alias.bin 2024-02-21T22:50:20.131 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/modules.builtin.bin 2024-02-21T22:50:20.131 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/modules.builtin.modinfo 2024-02-21T22:50:20.131 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/modules.dep 2024-02-21T22:50:20.131 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/modules.dep.bin 2024-02-21T22:50:20.131 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/modules.devname 2024-02-21T22:50:20.131 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/modules.order 2024-02-21T22:50:20.131 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/modules.softdep 2024-02-21T22:50:20.131 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/modules.symbols 2024-02-21T22:50:20.131 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/modules.symbols.bin 2024-02-21T22:50:20.131 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.8.0-rc4-g1130e0532280/vmlinuz 2024-02-21T22:50:20.135 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-rc4-g1130e0532280 2024-02-21T22:50:20.136 DEBUG:teuthology.orchestra.run.smithi028:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-02-21T22:50:20.161 INFO:teuthology.orchestra.run.smithi028.stdout:postinstall scriptlet (using /bin/sh): 2024-02-21T22:50:20.161 INFO:teuthology.orchestra.run.smithi028.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-02-21T22:50:20.161 INFO:teuthology.orchestra.run.smithi028.stdout: /usr/bin/kernel-install add 6.8.0-rc4-g1130e0532280 /lib/modules/6.8.0-rc4-g1130e0532280/vmlinuz 2024-02-21T22:50:20.161 INFO:teuthology.orchestra.run.smithi028.stdout:fi 2024-02-21T22:50:20.162 INFO:teuthology.orchestra.run.smithi028.stdout:for file in vmlinuz System.map config; do 2024-02-21T22:50:20.162 INFO:teuthology.orchestra.run.smithi028.stdout: if ! cmp --silent "/lib/modules/6.8.0-rc4-g1130e0532280/${file}" "/boot/${file}-6.8.0-rc4-g1130e0532280"; then 2024-02-21T22:50:20.162 INFO:teuthology.orchestra.run.smithi028.stdout: cp "/lib/modules/6.8.0-rc4-g1130e0532280/${file}" "/boot/${file}-6.8.0-rc4-g1130e0532280" 2024-02-21T22:50:20.162 INFO:teuthology.orchestra.run.smithi028.stdout: fi 2024-02-21T22:50:20.162 INFO:teuthology.orchestra.run.smithi028.stdout:done 2024-02-21T22:50:20.162 INFO:teuthology.orchestra.run.smithi028.stdout:preuninstall scriptlet (using /bin/sh): 2024-02-21T22:50:20.162 INFO:teuthology.orchestra.run.smithi028.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-02-21T22:50:20.162 INFO:teuthology.orchestra.run.smithi028.stdout:new-kernel-pkg --remove 6.8.0-rc4-g1130e0532280 --rminitrd --initrdfile=/boot/initramfs-6.8.0-rc4-g1130e0532280.img 2024-02-21T22:50:20.162 INFO:teuthology.orchestra.run.smithi028.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-02-21T22:50:20.162 INFO:teuthology.orchestra.run.smithi028.stdout:kernel-install remove 6.8.0-rc4-g1130e0532280 2024-02-21T22:50:20.162 INFO:teuthology.orchestra.run.smithi028.stdout:fi 2024-02-21T22:50:20.162 INFO:teuthology.orchestra.run.smithi028.stdout:postuninstall scriptlet (using /bin/sh): 2024-02-21T22:50:20.162 INFO:teuthology.orchestra.run.smithi028.stdout:if [ -x /sbin/update-bootloader ]; then 2024-02-21T22:50:20.162 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/update-bootloader --remove 6.8.0-rc4-g1130e0532280 2024-02-21T22:50:20.163 INFO:teuthology.orchestra.run.smithi028.stdout:fi 2024-02-21T22:50:20.163 DEBUG:teuthology.orchestra.run.smithi028:> sudo rpm -qi grub2-tools 2024-02-21T22:50:20.243 INFO:teuthology.orchestra.run.smithi028.stdout:Name : grub2-tools 2024-02-21T22:50:20.243 INFO:teuthology.orchestra.run.smithi028.stdout:Epoch : 1 2024-02-21T22:50:20.243 INFO:teuthology.orchestra.run.smithi028.stdout:Version : 2.06 2024-02-21T22:50:20.243 INFO:teuthology.orchestra.run.smithi028.stdout:Release : 68.el9 2024-02-21T22:50:20.243 INFO:teuthology.orchestra.run.smithi028.stdout:Architecture: x86_64 2024-02-21T22:50:20.243 INFO:teuthology.orchestra.run.smithi028.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-21T22:50:20.243 INFO:teuthology.orchestra.run.smithi028.stdout:Group : Unspecified 2024-02-21T22:50:20.243 INFO:teuthology.orchestra.run.smithi028.stdout:Size : 8264254 2024-02-21T22:50:20.243 INFO:teuthology.orchestra.run.smithi028.stdout:License : GPLv3+ 2024-02-21T22:50:20.243 INFO:teuthology.orchestra.run.smithi028.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-21T22:50:20.243 INFO:teuthology.orchestra.run.smithi028.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-21T22:50:20.244 INFO:teuthology.orchestra.run.smithi028.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-21T22:50:20.244 INFO:teuthology.orchestra.run.smithi028.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-21T22:50:20.244 INFO:teuthology.orchestra.run.smithi028.stdout:Packager : builder@centos.org 2024-02-21T22:50:20.244 INFO:teuthology.orchestra.run.smithi028.stdout:Vendor : CentOS 2024-02-21T22:50:20.244 INFO:teuthology.orchestra.run.smithi028.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-21T22:50:20.244 INFO:teuthology.orchestra.run.smithi028.stdout:Summary : Support tools for GRUB. 2024-02-21T22:50:20.244 INFO:teuthology.orchestra.run.smithi028.stdout:Description : 2024-02-21T22:50:20.244 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T22:50:20.244 INFO:teuthology.orchestra.run.smithi028.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-21T22:50:20.244 INFO:teuthology.orchestra.run.smithi028.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-21T22:50:20.244 INFO:teuthology.orchestra.run.smithi028.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-21T22:50:20.244 INFO:teuthology.orchestra.run.smithi028.stdout:hardware devices. 2024-02-21T22:50:20.244 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T22:50:20.244 INFO:teuthology.orchestra.run.smithi028.stdout:This subpackage provides tools for support of all platforms. 2024-02-21T22:50:20.245 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-21T22:50:20.245 INFO:teuthology.task.kernel:Updating grub on smithi028 to boot 6.8.0-rc4-g1130e0532280 2024-02-21T22:50:20.246 DEBUG:teuthology.orchestra.run.smithi028:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-21T22:50:20.885 INFO:teuthology.orchestra.run.smithi028.stderr:Generating grub configuration file ... 2024-02-21T22:50:22.327 INFO:teuthology.orchestra.run.smithi028.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-21T22:50:22.363 INFO:teuthology.orchestra.run.smithi028.stderr:done 2024-02-21T22:50:22.372 DEBUG:teuthology.orchestra.run.smithi028:> mktemp 2024-02-21T22:50:22.388 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/tmp.0UcTYRIKzP 2024-02-21T22:50:22.388 DEBUG:teuthology.orchestra.run.smithi028:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.0UcTYRIKzP 2024-02-21T22:50:22.461 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 0666 /tmp/tmp.0UcTYRIKzP 2024-02-21T22:50:22.605 DEBUG:teuthology.orchestra.remote:smithi028:/tmp/tmp.0UcTYRIKzP is 6KB 2024-02-21T22:50:22.653 DEBUG:teuthology.orchestra.run.smithi028:> rm -fr /tmp/tmp.0UcTYRIKzP 2024-02-21T22:50:22.668 DEBUG:teuthology.orchestra.run.smithi028:> sudo /bin/ls /boot/loader/entries || true 2024-02-21T22:50:22.737 INFO:teuthology.orchestra.run.smithi028.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-21T22:50:22.737 INFO:teuthology.orchestra.run.smithi028.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-21T22:50:22.737 INFO:teuthology.orchestra.run.smithi028.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-21T22:50:22.737 INFO:teuthology.orchestra.run.smithi028.stdout:15949e3c88704300882482eddfd7baae-6.8.0-rc4-g1130e0532280.conf 2024-02-21T22:50:22.739 DEBUG:teuthology.orchestra.run.smithi028:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.8.0-rc4-g1130e0532280 2024-02-21T22:50:22.889 DEBUG:teuthology.orchestra.run.smithi028:> sudo shutdown -r now 2024-02-21T22:50:47.520 INFO:teuthology.task.kernel:Checking client client.1 for new kernel version... 2024-02-21T22:50:47.537 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-02-21T22:50:47.538 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-21T22:50:52.896 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-02-21T22:50:52.896 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-02-21T22:50:52.897 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-21T22:51:05.920 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.81 2024-02-21T22:51:11.296 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.28 2024-02-21T22:51:14.925 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-02-21T22:51:14.926 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-21T22:51:20.300 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-02-21T22:51:20.301 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-21T22:51:38.688 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.28 2024-02-21T22:51:50.738 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-02-21T22:51:50.739 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-21T22:51:51.345 DEBUG:teuthology.orchestra.run.smithi028:> true 2024-02-21T22:51:51.998 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-02-21T22:51:51.999 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.8.0-rc4-g1130e0532280"... 2024-02-21T22:51:51.999 DEBUG:teuthology.orchestra.run.smithi028:> uname -r 2024-02-21T22:51:52.037 INFO:teuthology.orchestra.run.smithi028.stdout:6.8.0-rc4-g1130e0532280 2024-02-21T22:51:52.037 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-rc4-g1130e0532280 vs 6.8.0-rc4-g1130e0532280 2024-02-21T22:51:52.037 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-21T22:51:52.037 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-21T22:51:53.038 DEBUG:teuthology.parallel:result is None 2024-02-21T22:52:14.928 DEBUG:teuthology.orchestra.remote:timed out 2024-02-21T22:52:26.930 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-02-21T22:52:26.932 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-21T22:52:27.365 DEBUG:teuthology.orchestra.run.smithi081:> true 2024-02-21T22:52:28.069 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-02-21T22:52:28.069 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "6.8.0-rc4-g1130e0532280"... 2024-02-21T22:52:28.069 DEBUG:teuthology.orchestra.run.smithi081:> uname -r 2024-02-21T22:52:28.122 INFO:teuthology.orchestra.run.smithi081.stdout:6.8.0-rc4-g1130e0532280 2024-02-21T22:52:28.122 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-rc4-g1130e0532280 vs 6.8.0-rc4-g1130e0532280 2024-02-21T22:52:28.122 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-21T22:52:28.122 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-21T22:52:29.123 DEBUG:teuthology.parallel:result is None 2024-02-21T22:52:29.124 INFO:teuthology.run_tasks:Running task internal.base... 2024-02-21T22:52:29.131 INFO:teuthology.task.internal:Creating test directory... 2024-02-21T22:52:29.131 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-21T22:52:29.140 DEBUG:teuthology.orchestra.run.smithi081:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-21T22:52:29.196 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-02-21T22:52:29.208 INFO:teuthology.run_tasks:Running task internal.archive... 2024-02-21T22:52:29.216 INFO:teuthology.task.internal:Creating archive directory... 2024-02-21T22:52:29.216 DEBUG:teuthology.orchestra.run.smithi028:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-21T22:52:29.224 DEBUG:teuthology.orchestra.run.smithi081:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-21T22:52:29.296 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-02-21T22:52:29.304 INFO:teuthology.task.internal:Enabling coredump saving... 2024-02-21T22:52:29.305 DEBUG:teuthology.orchestra.run.smithi028:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-21T22:52:29.312 DEBUG:teuthology.orchestra.run.smithi081:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-21T22:52:29.442 INFO:teuthology.orchestra.run.smithi081.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-21T22:52:29.483 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-21T22:52:29.519 INFO:teuthology.orchestra.run.smithi081.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-21T22:52:29.572 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-21T22:52:29.577 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-02-21T22:52:29.584 INFO:teuthology.task.internal:Configuring sudo... 2024-02-21T22:52:29.584 DEBUG:teuthology.orchestra.run.smithi028:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-21T22:52:29.590 DEBUG:teuthology.orchestra.run.smithi081:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-21T22:52:29.691 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-02-21T22:52:29.700 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-02-21T22:52:29.701 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-21T22:52:29.711 DEBUG:teuthology.orchestra.run.smithi081:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-21T22:52:29.765 DEBUG:teuthology.orchestra.run.smithi028:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-21T22:52:29.848 DEBUG:teuthology.orchestra.run.smithi028:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-21T22:52:29.969 DEBUG:teuthology.orchestra.run.smithi028:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-21T22:52:30.050 DEBUG:teuthology.orchestra.run.smithi028:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-21T22:52:30.158 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-21T22:52:30.158 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-21T22:52:30.259 DEBUG:teuthology.orchestra.run.smithi081:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-21T22:52:30.348 DEBUG:teuthology.orchestra.run.smithi081:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-21T22:52:30.460 DEBUG:teuthology.orchestra.run.smithi081:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-21T22:52:30.540 DEBUG:teuthology.orchestra.run.smithi081:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-21T22:52:30.636 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-02-21T22:52:30.637 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-21T22:52:30.728 DEBUG:teuthology.orchestra.run.smithi028:> sudo service rsyslog restart 2024-02-21T22:52:30.739 DEBUG:teuthology.orchestra.run.smithi081:> sudo service rsyslog restart 2024-02-21T22:52:30.877 INFO:teuthology.orchestra.run.smithi028.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-21T22:52:30.905 INFO:teuthology.orchestra.run.smithi081.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-21T22:52:31.166 INFO:teuthology.run_tasks:Running task internal.timer... 2024-02-21T22:52:31.174 INFO:teuthology.task.internal:Starting timer... 2024-02-21T22:52:31.175 INFO:teuthology.run_tasks:Running task pcp... 2024-02-21T22:52:31.196 INFO:teuthology.run_tasks:Running task selinux... 2024-02-21T22:52:31.205 DEBUG:teuthology.orchestra.run.smithi028:> sudo service auditd rotate 2024-02-21T22:52:31.338 INFO:teuthology.orchestra.run.smithi028.stdout:Rotating logs: 2024-02-21T22:52:31.343 DEBUG:teuthology.orchestra.run.smithi081:> sudo service auditd rotate 2024-02-21T22:52:31.464 INFO:teuthology.orchestra.run.smithi081.stdout:Rotating logs: 2024-02-21T22:52:31.468 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-02-21T22:52:31.469 DEBUG:teuthology.orchestra.run.smithi028:> /usr/sbin/getenforce 2024-02-21T22:52:31.531 INFO:teuthology.orchestra.run.smithi028.stdout:Permissive 2024-02-21T22:52:31.532 DEBUG:teuthology.orchestra.run.smithi081:> /usr/sbin/getenforce 2024-02-21T22:52:31.588 INFO:teuthology.orchestra.run.smithi081.stdout:Permissive 2024-02-21T22:52:31.588 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi028.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi081.front.sepia.ceph.com': 'permissive'} 2024-02-21T22:52:31.589 DEBUG:teuthology.orchestra.run.smithi028:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-21T22:52:31.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-21T22:52:31.672 DEBUG:teuthology.orchestra.run.smithi081:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-21T22:52:31.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-21T22:52:31.746 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-02-21T22:52:31.746 DEBUG:teuthology.orchestra.run.smithi028:> sudo /usr/sbin/setenforce permissive 2024-02-21T22:52:31.853 DEBUG:teuthology.orchestra.run.smithi081:> sudo /usr/sbin/setenforce permissive 2024-02-21T22:52:31.952 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-02-21T22:52:31.962 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-02-21T22:52:31.967 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-02-21T22:52:32.097 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-02-21T22:52:32.115 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-02-21T22:52:32.116 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi028.front.sepia.ceph.com,smithi081.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-02-21T22:59:29.082 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi028.front.sepia.ceph.com'), Remote(name='ubuntu@smithi081.front.sepia.ceph.com')] 2024-02-21T22:59:29.083 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-02-21T22:59:29.084 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-21T22:59:29.232 DEBUG:teuthology.orchestra.run.smithi028:> true 2024-02-21T22:59:29.388 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-02-21T22:59:29.389 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-02-21T22:59:29.389 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-21T22:59:29.531 DEBUG:teuthology.orchestra.run.smithi081:> true 2024-02-21T22:59:29.685 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-02-21T22:59:29.686 INFO:teuthology.run_tasks:Running task clock... 2024-02-21T22:59:29.696 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-02-21T22:59:29.696 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-21T22:59:29.697 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-21T22:59:29.704 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-21T22:59:29.704 DEBUG:teuthology.orchestra.run.smithi081:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-21T22:59:29.803 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-21T22:59:29.804 INFO:teuthology.orchestra.run.smithi081.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-21T22:59:29.854 INFO:teuthology.orchestra.run.smithi081.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-21T22:59:29.857 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-21T22:59:29.949 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: ntpd: command not found 2024-02-21T22:59:29.962 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: ntpd: command not found 2024-02-21T22:59:29.990 INFO:teuthology.orchestra.run.smithi081.stdout:506 Cannot talk to daemon 2024-02-21T22:59:30.011 INFO:teuthology.orchestra.run.smithi028.stdout:506 Cannot talk to daemon 2024-02-21T22:59:30.042 INFO:teuthology.orchestra.run.smithi081.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-21T22:59:30.066 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-21T22:59:30.095 INFO:teuthology.orchestra.run.smithi081.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-21T22:59:30.124 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-21T22:59:30.220 INFO:teuthology.orchestra.run.smithi081.stderr:bash: line 1: ntpq: command not found 2024-02-21T22:59:30.229 INFO:teuthology.orchestra.run.smithi081.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-21T22:59:30.229 INFO:teuthology.orchestra.run.smithi081.stdout:=============================================================================== 2024-02-21T22:59:30.229 INFO:teuthology.orchestra.run.smithi081.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-21T22:59:30.229 INFO:teuthology.orchestra.run.smithi081.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-21T22:59:30.229 INFO:teuthology.orchestra.run.smithi081.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-21T22:59:30.230 INFO:teuthology.orchestra.run.smithi081.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-21T22:59:30.247 INFO:teuthology.orchestra.run.smithi028.stderr:bash: line 1: ntpq: command not found 2024-02-21T22:59:30.257 INFO:teuthology.orchestra.run.smithi028.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-21T22:59:30.257 INFO:teuthology.orchestra.run.smithi028.stdout:=============================================================================== 2024-02-21T22:59:30.257 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-21T22:59:30.257 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-21T22:59:30.257 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-21T22:59:30.257 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-21T22:59:30.259 INFO:teuthology.run_tasks:Running task install... 2024-02-21T22:59:30.270 DEBUG:teuthology.task.install:project ceph 2024-02-21T22:59:30.270 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '3bfc837d7b45ce8e095d92fa9c342964cea5e95f'}} 2024-02-21T22:59:30.270 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '3bfc837d7b45ce8e095d92fa9c342964cea5e95f'} 2024-02-21T22:59:30.270 INFO:teuthology.task.install:Using flavor: default 2024-02-21T22:59:30.276 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-02-21T22:59:30.276 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-02-21T22:59:30.277 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3bfc837d7b45ce8e095d92fa9c342964cea5e95f', 'tag': None, 'wait_for_package': False} 2024-02-21T22:59:30.277 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3bfc837d7b45ce8e095d92fa9c342964cea5e95f 2024-02-21T22:59:30.279 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3bfc837d7b45ce8e095d92fa9c342964cea5e95f', 'tag': None, 'wait_for_package': False} 2024-02-21T22:59:30.279 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3bfc837d7b45ce8e095d92fa9c342964cea5e95f 2024-02-21T22:59:30.423 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/3bfc837d7b45ce8e095d92fa9c342964cea5e95f/centos/9/flavors/default/ 2024-02-21T22:59:30.423 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1643.g3bfc837d 2024-02-21T22:59:30.428 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/3bfc837d7b45ce8e095d92fa9c342964cea5e95f/centos/9/flavors/default/ 2024-02-21T22:59:30.428 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1643.g3bfc837d 2024-02-21T22:59:30.593 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/3bfc837d7b45ce8e095d92fa9c342964cea5e95f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/3bfc837d7b45ce8e095d92fa9c342964cea5e95f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/3bfc837d7b45ce8e095d92fa9c342964cea5e95f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-21T22:59:30.594 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-21T22:59:30.594 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-21T22:59:30.598 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/3bfc837d7b45ce8e095d92fa9c342964cea5e95f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/3bfc837d7b45ce8e095d92fa9c342964cea5e95f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/3bfc837d7b45ce8e095d92fa9c342964cea5e95f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-21T22:59:30.598 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-02-21T22:59:30.599 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-21T22:59:30.698 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-02-21T22:59:30.698 DEBUG:teuthology.orchestra.run.smithi028:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3bfc837d7b45ce8e095d92fa9c342964cea5e95f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-21T22:59:30.700 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-02-21T22:59:30.700 DEBUG:teuthology.orchestra.run.smithi081:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3bfc837d7b45ce8e095d92fa9c342964cea5e95f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-21T22:59:30.834 DEBUG:teuthology.orchestra.run.smithi081:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-21T22:59:30.840 DEBUG:teuthology.orchestra.run.smithi028:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-21T22:59:30.968 DEBUG:teuthology.orchestra.run.smithi081:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-21T22:59:30.979 DEBUG:teuthology.orchestra.run.smithi028:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-21T22:59:31.056 INFO:teuthology.orchestra.run.smithi081.stdout:check_obsoletes = 1 2024-02-21T22:59:31.061 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum clean all 2024-02-21T22:59:31.076 INFO:teuthology.orchestra.run.smithi028.stdout:check_obsoletes = 1 2024-02-21T22:59:31.081 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum clean all 2024-02-21T22:59:31.414 INFO:teuthology.orchestra.run.smithi081.stdout:74 files removed 2024-02-21T22:59:31.453 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-02-21T22:59:31.461 INFO:teuthology.orchestra.run.smithi028.stdout:74 files removed 2024-02-21T22:59:31.500 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-02-21T22:59:32.412 INFO:teuthology.orchestra.run.smithi081.stdout:ceph packages for x86_64 182 kB/s | 83 kB 00:00 2024-02-21T22:59:32.441 INFO:teuthology.orchestra.run.smithi028.stdout:ceph packages for x86_64 198 kB/s | 83 kB 00:00 2024-02-21T22:59:32.912 INFO:teuthology.orchestra.run.smithi081.stdout:ceph noarch packages 25 kB/s | 10 kB 00:00 2024-02-21T22:59:33.024 INFO:teuthology.orchestra.run.smithi028.stdout:ceph noarch packages 21 kB/s | 10 kB 00:00 2024-02-21T22:59:33.396 INFO:teuthology.orchestra.run.smithi081.stdout:ceph source packages 4.6 kB/s | 1.9 kB 00:00 2024-02-21T22:59:33.592 INFO:teuthology.orchestra.run.smithi028.stdout:ceph source packages 3.9 kB/s | 1.9 kB 00:00 2024-02-21T22:59:34.099 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS-9 - AppStream 13 MB/s | 8.1 MB 00:00 2024-02-21T22:59:34.262 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS-9 - AppStream 14 MB/s | 8.1 MB 00:00 2024-02-21T22:59:35.934 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS-9 - BaseOS 6.8 MB/s | 2.5 MB 00:00 2024-02-21T22:59:36.137 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS-9 - BaseOS 7.4 MB/s | 2.5 MB 00:00 2024-02-21T22:59:37.250 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.0 MB 00:00 2024-02-21T22:59:37.393 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - BaseOS 8.4 MB/s | 8.0 MB 00:00 2024-02-21T22:59:39.785 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - AppStream 17 MB/s | 19 MB 00:01 2024-02-21T22:59:40.926 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - AppStream 8.0 MB/s | 19 MB 00:02 2024-02-21T22:59:47.004 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - CRB 3.0 MB/s | 6.1 MB 00:01 2024-02-21T22:59:48.563 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - Extras packages 69 kB/s | 15 kB 00:00 2024-02-21T22:59:50.203 INFO:teuthology.orchestra.run.smithi028.stdout:Extra Packages for Enterprise Linux 14 MB/s | 20 MB 00:01 2024-02-21T22:59:52.006 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - CRB 737 kB/s | 6.1 MB 00:08 2024-02-21T22:59:53.615 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - Extras packages 61 kB/s | 15 kB 00:00 2024-02-21T22:59:55.159 INFO:teuthology.orchestra.run.smithi081.stdout:Extra Packages for Enterprise Linux 15 MB/s | 20 MB 00:01 2024-02-21T22:59:56.381 INFO:teuthology.orchestra.run.smithi028.stdout:lab-extras 21 kB/s | 1.7 kB 00:00 2024-02-21T22:59:59.135 INFO:teuthology.orchestra.run.smithi028.stderr:Modular dependency problems: 2024-02-21T22:59:59.135 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-02-21T22:59:59.135 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 1: conflicting requests 2024-02-21T22:59:59.135 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240219154654:20125149.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.135 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 2: conflicting requests 2024-02-21T22:59:59.135 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.136 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 3: conflicting requests 2024-02-21T22:59:59.136 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.136 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 4: conflicting requests 2024-02-21T22:59:59.136 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.136 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 5: conflicting requests 2024-02-21T22:59:59.136 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240213202549:b754926a.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.136 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 6: conflicting requests 2024-02-21T22:59:59.136 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.136 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 7: conflicting requests 2024-02-21T22:59:59.136 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.137 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 8: conflicting requests 2024-02-21T22:59:59.137 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.137 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 9: conflicting requests 2024-02-21T22:59:59.137 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.137 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 10: conflicting requests 2024-02-21T22:59:59.137 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.137 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 11: conflicting requests 2024-02-21T22:59:59.137 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.137 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 12: conflicting requests 2024-02-21T22:59:59.137 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.137 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 13: conflicting requests 2024-02-21T22:59:59.137 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.137 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 14: conflicting requests 2024-02-21T22:59:59.138 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.138 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 15: conflicting requests 2024-02-21T22:59:59.138 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.138 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 16: conflicting requests 2024-02-21T22:59:59.138 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.138 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 17: conflicting requests 2024-02-21T22:59:59.138 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.138 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 18: conflicting requests 2024-02-21T22:59:59.138 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.138 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 19: conflicting requests 2024-02-21T22:59:59.138 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.138 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 20: conflicting requests 2024-02-21T22:59:59.138 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.139 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 21: conflicting requests 2024-02-21T22:59:59.139 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.139 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 22: conflicting requests 2024-02-21T22:59:59.139 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.139 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 23: conflicting requests 2024-02-21T22:59:59.139 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.139 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 24: conflicting requests 2024-02-21T22:59:59.139 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.139 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 25: conflicting requests 2024-02-21T22:59:59.139 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.139 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 26: conflicting requests 2024-02-21T22:59:59.140 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.140 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 27: conflicting requests 2024-02-21T22:59:59.140 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.140 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 28: conflicting requests 2024-02-21T22:59:59.140 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.140 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 29: conflicting requests 2024-02-21T22:59:59.140 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.140 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 30: conflicting requests 2024-02-21T22:59:59.140 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.140 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 31: conflicting requests 2024-02-21T22:59:59.140 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240214183150:9edba152.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.140 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 32: conflicting requests 2024-02-21T22:59:59.140 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.140 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 33: conflicting requests 2024-02-21T22:59:59.141 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.141 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 34: conflicting requests 2024-02-21T22:59:59.141 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.141 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 35: conflicting requests 2024-02-21T22:59:59.141 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240203221207:9edba152.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.141 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 36: conflicting requests 2024-02-21T22:59:59.141 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.141 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.141 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.141 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 37: cannot install the best candidate for the job 2024-02-21T22:59:59.141 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.141 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.141 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.141 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.141 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 38: conflicting requests 2024-02-21T22:59:59.142 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.142 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.142 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.142 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.142 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.142 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.142 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 39: cannot install the best candidate for the job 2024-02-21T22:59:59.142 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.142 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.142 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.142 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.142 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 40: cannot install the best candidate for the job 2024-02-21T22:59:59.142 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.142 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.142 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.143 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.143 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 41: cannot install the best candidate for the job 2024-02-21T22:59:59.143 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.143 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.143 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.143 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.143 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 42: cannot install the best candidate for the job 2024-02-21T22:59:59.143 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.143 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.143 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.143 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.143 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 43: cannot install the best candidate for the job 2024-02-21T22:59:59.143 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.143 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.143 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.144 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.144 INFO:teuthology.orchestra.run.smithi028.stderr: Problem 44: cannot install the best candidate for the job 2024-02-21T22:59:59.144 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.144 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.144 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.144 INFO:teuthology.orchestra.run.smithi028.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-02-21T22:59:59.769 INFO:teuthology.orchestra.run.smithi028.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-02-21T22:59:59.771 INFO:teuthology.orchestra.run.smithi028.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-02-21T22:59:59.795 INFO:teuthology.orchestra.run.smithi028.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-02-21T22:59:59.796 INFO:teuthology.orchestra.run.smithi028.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-02-21T22:59:59.798 INFO:teuthology.orchestra.run.smithi028.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-02-21T22:59:59.799 INFO:teuthology.orchestra.run.smithi028.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-02-21T22:59:59.802 INFO:teuthology.orchestra.run.smithi028.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-02-21T22:59:59.806 INFO:teuthology.orchestra.run.smithi028.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-02-21T22:59:59.808 INFO:teuthology.orchestra.run.smithi028.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-02-21T22:59:59.914 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-02-21T22:59:59.945 INFO:teuthology.orchestra.run.smithi028.stdout:========================================================================================= 2024-02-21T22:59:59.945 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2024-02-21T22:59:59.945 INFO:teuthology.orchestra.run.smithi028.stdout:========================================================================================= 2024-02-21T22:59:59.945 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-02-21T22:59:59.945 INFO:teuthology.orchestra.run.smithi028.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-02-21T22:59:59.945 INFO:teuthology.orchestra.run.smithi028.stdout: ceph x86_64 2:19.0.0-1643.g3bfc837d.el9 ceph 6.6 k 2024-02-21T22:59:59.945 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-base x86_64 2:19.0.0-1643.g3bfc837d.el9 ceph 5.3 M 2024-02-21T22:59:59.945 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-fuse x86_64 2:19.0.0-1643.g3bfc837d.el9 ceph 889 k 2024-02-21T22:59:59.945 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1643.g3bfc837d.el9 ceph 145 k 2024-02-21T22:59:59.946 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr x86_64 2:19.0.0-1643.g3bfc837d.el9 ceph 1.5 M 2024-02-21T22:59:59.946 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1643.g3bfc837d.el9 ceph-noarch 138 k 2024-02-21T22:59:59.946 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1643.g3bfc837d.el9 ceph-noarch 1.8 M 2024-02-21T22:59:59.946 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1643.g3bfc837d.el9 ceph-noarch 7.4 M 2024-02-21T22:59:59.946 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-rook noarch 2:19.0.0-1643.g3bfc837d.el9 ceph-noarch 49 k 2024-02-21T22:59:59.946 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-radosgw x86_64 2:19.0.0-1643.g3bfc837d.el9 ceph 9.8 M 2024-02-21T22:59:59.946 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-test x86_64 2:19.0.0-1643.g3bfc837d.el9 ceph 47 M 2024-02-21T22:59:59.946 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-volume noarch 2:19.0.0-1643.g3bfc837d.el9 ceph-noarch 264 k 2024-02-21T22:59:59.946 INFO:teuthology.orchestra.run.smithi028.stdout: cephadm noarch 2:19.0.0-1643.g3bfc837d.el9 ceph-noarch 748 k 2024-02-21T22:59:59.946 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-mirror x86_64 2:19.0.0-1643.g3bfc837d.el9 ceph 220 k 2024-02-21T22:59:59.946 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-shell x86_64 2:19.0.0-1643.g3bfc837d.el9 ceph 27 k 2024-02-21T22:59:59.946 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-top noarch 2:19.0.0-1643.g3bfc837d.el9 ceph-noarch 18 k 2024-02-21T22:59:59.946 INFO:teuthology.orchestra.run.smithi028.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-02-21T22:59:59.947 INFO:teuthology.orchestra.run.smithi028.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-02-21T22:59:59.947 INFO:teuthology.orchestra.run.smithi028.stdout: libaio-devel x86_64 0.3.112-1.el8 CentOS-BaseOS 18 k 2024-02-21T22:59:59.947 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs-devel x86_64 2:19.0.0-1643.g3bfc837d.el9 ceph 31 k 2024-02-21T22:59:59.947 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs2 x86_64 2:19.0.0-1643.g3bfc837d.el9 ceph 745 k 2024-02-21T22:59:59.947 INFO:teuthology.orchestra.run.smithi028.stdout: librados-devel x86_64 2:19.0.0-1643.g3bfc837d.el9 ceph 127 k 2024-02-21T22:59:59.947 INFO:teuthology.orchestra.run.smithi028.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-02-21T22:59:59.947 INFO:teuthology.orchestra.run.smithi028.stdout: libuuid-devel x86_64 2.37.4-15.el9 appstream 26 k 2024-02-21T22:59:59.947 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-devel x86_64 1:3.0.7-25.el9 appstream 4.1 M 2024-02-21T22:59:59.947 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql x86_64 13.11-1.el9 appstream 1.6 M 2024-02-21T22:59:59.947 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-contrib x86_64 13.11-1.el9 appstream 905 k 2024-02-21T22:59:59.947 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-server x86_64 13.11-1.el9 appstream 5.7 M 2024-02-21T22:59:59.947 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cephfs x86_64 2:19.0.0-1643.g3bfc837d.el9 ceph 161 k 2024-02-21T22:59:59.947 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rados x86_64 2:19.0.0-1643.g3bfc837d.el9 ceph 321 k 2024-02-21T22:59:59.947 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rbd x86_64 2:19.0.0-1643.g3bfc837d.el9 ceph 298 k 2024-02-21T22:59:59.948 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rgw x86_64 2:19.0.0-1643.g3bfc837d.el9 ceph 100 k 2024-02-21T22:59:59.948 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-fuse x86_64 2:19.0.0-1643.g3bfc837d.el9 ceph 85 k 2024-02-21T22:59:59.948 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-mirror x86_64 2:19.0.0-1643.g3bfc837d.el9 ceph 3.1 M 2024-02-21T22:59:59.948 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-nbd x86_64 2:19.0.0-1643.g3bfc837d.el9 ceph 170 k 2024-02-21T22:59:59.948 INFO:teuthology.orchestra.run.smithi028.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-02-21T22:59:59.948 INFO:teuthology.orchestra.run.smithi028.stdout:Upgrading: 2024-02-21T22:59:59.948 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager x86_64 1:1.45.91-1.el9 baseos 2.3 M 2024-02-21T22:59:59.948 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager-libnm x86_64 1:1.45.91-1.el9 baseos 1.8 M 2024-02-21T22:59:59.948 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager-team x86_64 1:1.45.91-1.el9 baseos 42 k 2024-02-21T22:59:59.948 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager-tui x86_64 1:1.45.91-1.el9 baseos 247 k 2024-02-21T22:59:59.949 INFO:teuthology.orchestra.run.smithi028.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-02-21T22:59:59.949 INFO:teuthology.orchestra.run.smithi028.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-02-21T22:59:59.949 INFO:teuthology.orchestra.run.smithi028.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-02-21T22:59:59.949 INFO:teuthology.orchestra.run.smithi028.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-02-21T22:59:59.949 INFO:teuthology.orchestra.run.smithi028.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-02-21T22:59:59.949 INFO:teuthology.orchestra.run.smithi028.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-02-21T22:59:59.949 INFO:teuthology.orchestra.run.smithi028.stdout: librados2 x86_64 2:19.0.0-1643.g3bfc837d.el9 ceph 3.4 M 2024-02-21T22:59:59.949 INFO:teuthology.orchestra.run.smithi028.stdout: librbd1 x86_64 2:19.0.0-1643.g3bfc837d.el9 ceph 3.1 M 2024-02-21T22:59:59.949 INFO:teuthology.orchestra.run.smithi028.stdout: openssl x86_64 1:3.0.7-25.el9 baseos 1.2 M 2024-02-21T22:59:59.949 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-libs x86_64 1:3.0.7-25.el9 baseos 2.2 M 2024-02-21T22:59:59.949 INFO:teuthology.orchestra.run.smithi028.stdout: xfsdump x86_64 3.1.12-4.el9 baseos 323 k 2024-02-21T22:59:59.949 INFO:teuthology.orchestra.run.smithi028.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-02-21T22:59:59.949 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-02-21T22:59:59.950 INFO:teuthology.orchestra.run.smithi028.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-02-21T22:59:59.950 INFO:teuthology.orchestra.run.smithi028.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-02-21T22:59:59.950 INFO:teuthology.orchestra.run.smithi028.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-02-21T22:59:59.950 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-common x86_64 2:19.0.0-1643.g3bfc837d.el9 ceph 20 M 2024-02-21T22:59:59.950 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1643.g3bfc837d.el9 ceph-noarch 31 k 2024-02-21T22:59:59.950 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mds x86_64 2:19.0.0-1643.g3bfc837d.el9 ceph 2.3 M 2024-02-21T22:59:59.950 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1643.g3bfc837d.el9 ceph-noarch 248 k 2024-02-21T22:59:59.950 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mon x86_64 2:19.0.0-1643.g3bfc837d.el9 ceph 4.7 M 2024-02-21T22:59:59.950 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-osd x86_64 2:19.0.0-1643.g3bfc837d.el9 ceph 17 M 2024-02-21T22:59:59.950 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1643.g3bfc837d.el9 ceph-noarch 15 k 2024-02-21T22:59:59.951 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-selinux x86_64 2:19.0.0-1643.g3bfc837d.el9 ceph 25 k 2024-02-21T22:59:59.951 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-02-21T22:59:59.951 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-02-21T22:59:59.951 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-02-21T22:59:59.951 INFO:teuthology.orchestra.run.smithi028.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-02-21T22:59:59.951 INFO:teuthology.orchestra.run.smithi028.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-02-21T22:59:59.951 INFO:teuthology.orchestra.run.smithi028.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-02-21T22:59:59.951 INFO:teuthology.orchestra.run.smithi028.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-02-21T22:59:59.951 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-02-21T22:59:59.951 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-02-21T22:59:59.951 INFO:teuthology.orchestra.run.smithi028.stdout: libcephsqlite x86_64 2:19.0.0-1643.g3bfc837d.el9 ceph 163 k 2024-02-21T22:59:59.951 INFO:teuthology.orchestra.run.smithi028.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-02-21T22:59:59.951 INFO:teuthology.orchestra.run.smithi028.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-02-21T22:59:59.951 INFO:teuthology.orchestra.run.smithi028.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-02-21T22:59:59.951 INFO:teuthology.orchestra.run.smithi028.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-02-21T22:59:59.952 INFO:teuthology.orchestra.run.smithi028.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-02-21T22:59:59.952 INFO:teuthology.orchestra.run.smithi028.stdout: libradosstriper1 x86_64 2:19.0.0-1643.g3bfc837d.el9 ceph 502 k 2024-02-21T22:59:59.952 INFO:teuthology.orchestra.run.smithi028.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-02-21T22:59:59.952 INFO:teuthology.orchestra.run.smithi028.stdout: librgw2 x86_64 2:19.0.0-1643.g3bfc837d.el9 ceph 5.1 M 2024-02-21T22:59:59.952 INFO:teuthology.orchestra.run.smithi028.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-02-21T22:59:59.952 INFO:teuthology.orchestra.run.smithi028.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-02-21T22:59:59.952 INFO:teuthology.orchestra.run.smithi028.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-02-21T22:59:59.952 INFO:teuthology.orchestra.run.smithi028.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-02-21T22:59:59.952 INFO:teuthology.orchestra.run.smithi028.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-02-21T22:59:59.952 INFO:teuthology.orchestra.run.smithi028.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-02-21T22:59:59.952 INFO:teuthology.orchestra.run.smithi028.stdout: oniguruma x86_64 6.9.6-1.el9.5 baseos 218 k 2024-02-21T22:59:59.952 INFO:teuthology.orchestra.run.smithi028.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-02-21T22:59:59.952 INFO:teuthology.orchestra.run.smithi028.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-02-21T22:59:59.952 INFO:teuthology.orchestra.run.smithi028.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-02-21T22:59:59.953 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-private-libs x86_64 13.11-1.el9 appstream 135 k 2024-02-21T22:59:59.953 INFO:teuthology.orchestra.run.smithi028.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-02-21T22:59:59.953 INFO:teuthology.orchestra.run.smithi028.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-02-21T22:59:59.953 INFO:teuthology.orchestra.run.smithi028.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-02-21T22:59:59.953 INFO:teuthology.orchestra.run.smithi028.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-02-21T22:59:59.953 INFO:teuthology.orchestra.run.smithi028.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-02-21T22:59:59.953 INFO:teuthology.orchestra.run.smithi028.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-02-21T22:59:59.953 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-02-21T22:59:59.953 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-argparse x86_64 2:19.0.0-1643.g3bfc837d.el9 ceph 45 k 2024-02-21T22:59:59.953 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-common x86_64 2:19.0.0-1643.g3bfc837d.el9 ceph 131 k 2024-02-21T22:59:59.953 INFO:teuthology.orchestra.run.smithi028.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-02-21T22:59:59.953 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-02-21T22:59:59.953 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-02-21T22:59:59.953 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-02-21T22:59:59.954 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-02-21T22:59:59.954 INFO:teuthology.orchestra.run.smithi028.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-02-21T22:59:59.954 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-02-21T22:59:59.954 INFO:teuthology.orchestra.run.smithi028.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-02-21T22:59:59.954 INFO:teuthology.orchestra.run.smithi028.stdout: python3-google-auth noarch 1:2.27.0-1.el9 epel 203 k 2024-02-21T22:59:59.954 INFO:teuthology.orchestra.run.smithi028.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-02-21T22:59:59.954 INFO:teuthology.orchestra.run.smithi028.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-02-21T22:59:59.954 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-02-21T22:59:59.955 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-02-21T22:59:59.955 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-02-21T22:59:59.955 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-02-21T22:59:59.955 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-02-21T22:59:59.955 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-02-21T22:59:59.955 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-02-21T22:59:59.955 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-02-21T22:59:59.955 INFO:teuthology.orchestra.run.smithi028.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-02-21T22:59:59.955 INFO:teuthology.orchestra.run.smithi028.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-02-21T22:59:59.955 INFO:teuthology.orchestra.run.smithi028.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-02-21T22:59:59.955 INFO:teuthology.orchestra.run.smithi028.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-02-21T22:59:59.955 INFO:teuthology.orchestra.run.smithi028.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-02-21T22:59:59.955 INFO:teuthology.orchestra.run.smithi028.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-02-21T22:59:59.955 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-02-21T22:59:59.956 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-02-21T22:59:59.956 INFO:teuthology.orchestra.run.smithi028.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-02-21T22:59:59.956 INFO:teuthology.orchestra.run.smithi028.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-02-21T22:59:59.956 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-02-21T22:59:59.956 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-02-21T22:59:59.956 INFO:teuthology.orchestra.run.smithi028.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-02-21T22:59:59.956 INFO:teuthology.orchestra.run.smithi028.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-02-21T22:59:59.956 INFO:teuthology.orchestra.run.smithi028.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-02-21T22:59:59.956 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-02-21T22:59:59.956 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-02-21T22:59:59.956 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-02-21T22:59:59.956 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-02-21T22:59:59.956 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-02-21T22:59:59.956 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-02-21T22:59:59.956 INFO:teuthology.orchestra.run.smithi028.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-02-21T22:59:59.957 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-02-21T22:59:59.957 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-02-21T22:59:59.957 INFO:teuthology.orchestra.run.smithi028.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-02-21T22:59:59.957 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-02-21T22:59:59.957 INFO:teuthology.orchestra.run.smithi028.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-02-21T22:59:59.957 INFO:teuthology.orchestra.run.smithi028.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-02-21T22:59:59.957 INFO:teuthology.orchestra.run.smithi028.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-02-21T22:59:59.957 INFO:teuthology.orchestra.run.smithi028.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-02-21T22:59:59.957 INFO:teuthology.orchestra.run.smithi028.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-02-21T22:59:59.957 INFO:teuthology.orchestra.run.smithi028.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-02-21T22:59:59.957 INFO:teuthology.orchestra.run.smithi028.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-02-21T22:59:59.957 INFO:teuthology.orchestra.run.smithi028.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-02-21T22:59:59.957 INFO:teuthology.orchestra.run.smithi028.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-02-21T22:59:59.957 INFO:teuthology.orchestra.run.smithi028.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-02-21T22:59:59.957 INFO:teuthology.orchestra.run.smithi028.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-02-21T22:59:59.958 INFO:teuthology.orchestra.run.smithi028.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-02-21T22:59:59.958 INFO:teuthology.orchestra.run.smithi028.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-02-21T22:59:59.958 INFO:teuthology.orchestra.run.smithi028.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-02-21T22:59:59.958 INFO:teuthology.orchestra.run.smithi028.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-02-21T22:59:59.958 INFO:teuthology.orchestra.run.smithi028.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-02-21T22:59:59.958 INFO:teuthology.orchestra.run.smithi028.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-02-21T22:59:59.958 INFO:teuthology.orchestra.run.smithi028.stdout: zlib-devel x86_64 1.2.11-40.el9 appstream 45 k 2024-02-21T22:59:59.958 INFO:teuthology.orchestra.run.smithi028.stdout:Installing weak dependencies: 2024-02-21T22:59:59.958 INFO:teuthology.orchestra.run.smithi028.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-02-21T22:59:59.958 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T22:59:59.958 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-02-21T22:59:59.958 INFO:teuthology.orchestra.run.smithi028.stdout:========================================================================================= 2024-02-21T22:59:59.958 INFO:teuthology.orchestra.run.smithi028.stdout:Install 150 Packages 2024-02-21T22:59:59.958 INFO:teuthology.orchestra.run.smithi028.stdout:Upgrade 16 Packages 2024-02-21T22:59:59.958 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T22:59:59.959 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 222 M 2024-02-21T22:59:59.959 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-02-21T23:00:01.042 INFO:teuthology.orchestra.run.smithi081.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-02-21T23:00:01.291 INFO:teuthology.orchestra.run.smithi028.stdout:(1/166): ceph-19.0.0-1643.g3bfc837d.el9.x86_64. 15 kB/s | 6.6 kB 00:00 2024-02-21T23:00:01.583 INFO:teuthology.orchestra.run.smithi028.stdout:(2/166): ceph-base-19.0.0-1643.g3bfc837d.el9.x8 7.4 MB/s | 5.3 MB 00:00 2024-02-21T23:00:01.658 INFO:teuthology.orchestra.run.smithi028.stdout:(3/166): ceph-immutable-object-cache-19.0.0-164 1.9 MB/s | 145 kB 00:00 2024-02-21T23:00:01.942 INFO:teuthology.orchestra.run.smithi028.stdout:(4/166): ceph-mds-19.0.0-1643.g3bfc837d.el9.x86 8.2 MB/s | 2.3 MB 00:00 2024-02-21T23:00:02.117 INFO:teuthology.orchestra.run.smithi028.stdout:(5/166): ceph-mgr-19.0.0-1643.g3bfc837d.el9.x86 8.6 MB/s | 1.5 MB 00:00 2024-02-21T23:00:02.592 INFO:teuthology.orchestra.run.smithi028.stdout:(6/166): ceph-common-19.0.0-1643.g3bfc837d.el9. 12 MB/s | 20 MB 00:01 2024-02-21T23:00:02.634 INFO:teuthology.orchestra.run.smithi028.stdout:(7/166): ceph-fuse-19.0.0-1643.g3bfc837d.el9.x8 663 kB/s | 889 kB 00:01 2024-02-21T23:00:02.768 INFO:teuthology.orchestra.run.smithi028.stdout:(8/166): ceph-mon-19.0.0-1643.g3bfc837d.el9.x86 7.2 MB/s | 4.7 MB 00:00 2024-02-21T23:00:02.834 INFO:teuthology.orchestra.run.smithi028.stdout:(9/166): ceph-selinux-19.0.0-1643.g3bfc837d.el9 384 kB/s | 25 kB 00:00 2024-02-21T23:00:03.679 INFO:teuthology.orchestra.run.smithi081.stderr:Modular dependency problems: 2024-02-21T23:00:03.679 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-02-21T23:00:03.679 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 1: conflicting requests 2024-02-21T23:00:03.679 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240219154654:20125149.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.680 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 2: conflicting requests 2024-02-21T23:00:03.680 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.680 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 3: conflicting requests 2024-02-21T23:00:03.680 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.680 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 4: conflicting requests 2024-02-21T23:00:03.680 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.680 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 5: conflicting requests 2024-02-21T23:00:03.680 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240213202549:b754926a.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.680 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 6: conflicting requests 2024-02-21T23:00:03.680 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.680 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 7: conflicting requests 2024-02-21T23:00:03.680 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.681 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 8: conflicting requests 2024-02-21T23:00:03.681 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.681 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 9: conflicting requests 2024-02-21T23:00:03.681 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.681 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 10: conflicting requests 2024-02-21T23:00:03.681 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.681 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 11: conflicting requests 2024-02-21T23:00:03.681 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.681 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 12: conflicting requests 2024-02-21T23:00:03.681 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.681 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 13: conflicting requests 2024-02-21T23:00:03.681 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.682 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 14: conflicting requests 2024-02-21T23:00:03.682 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.682 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 15: conflicting requests 2024-02-21T23:00:03.682 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.682 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 16: conflicting requests 2024-02-21T23:00:03.682 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.682 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 17: conflicting requests 2024-02-21T23:00:03.682 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.682 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 18: conflicting requests 2024-02-21T23:00:03.682 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.682 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 19: conflicting requests 2024-02-21T23:00:03.682 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.683 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 20: conflicting requests 2024-02-21T23:00:03.683 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.683 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 21: conflicting requests 2024-02-21T23:00:03.683 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.683 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 22: conflicting requests 2024-02-21T23:00:03.683 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.683 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 23: conflicting requests 2024-02-21T23:00:03.683 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.683 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 24: conflicting requests 2024-02-21T23:00:03.683 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.683 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 25: conflicting requests 2024-02-21T23:00:03.683 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.683 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 26: conflicting requests 2024-02-21T23:00:03.683 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.683 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 27: conflicting requests 2024-02-21T23:00:03.684 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.684 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 28: conflicting requests 2024-02-21T23:00:03.684 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.684 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 29: conflicting requests 2024-02-21T23:00:03.684 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.684 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 30: conflicting requests 2024-02-21T23:00:03.684 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.684 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 31: conflicting requests 2024-02-21T23:00:03.684 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240214183150:9edba152.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.684 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 32: conflicting requests 2024-02-21T23:00:03.684 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.684 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 33: conflicting requests 2024-02-21T23:00:03.684 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.684 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 34: conflicting requests 2024-02-21T23:00:03.684 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.685 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 35: conflicting requests 2024-02-21T23:00:03.685 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240203221207:9edba152.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.685 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 36: conflicting requests 2024-02-21T23:00:03.685 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.685 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.685 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.685 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 37: cannot install the best candidate for the job 2024-02-21T23:00:03.685 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.685 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.685 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.685 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.685 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 38: conflicting requests 2024-02-21T23:00:03.685 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.685 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.685 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.686 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.686 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.686 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.686 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 39: cannot install the best candidate for the job 2024-02-21T23:00:03.686 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.686 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.686 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.686 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.686 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 40: cannot install the best candidate for the job 2024-02-21T23:00:03.686 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.686 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.686 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.686 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.687 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 41: cannot install the best candidate for the job 2024-02-21T23:00:03.687 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.687 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.687 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.687 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.687 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 42: cannot install the best candidate for the job 2024-02-21T23:00:03.687 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.687 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.687 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.687 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.687 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 43: cannot install the best candidate for the job 2024-02-21T23:00:03.687 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.687 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.687 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.687 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.688 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 44: cannot install the best candidate for the job 2024-02-21T23:00:03.688 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.688 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.688 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.688 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-02-21T23:00:03.827 INFO:teuthology.orchestra.run.smithi028.stdout:(10/166): ceph-osd-19.0.0-1643.g3bfc837d.el9.x8 14 MB/s | 17 MB 00:01 2024-02-21T23:00:03.885 INFO:teuthology.orchestra.run.smithi028.stdout:(11/166): cephfs-mirror-19.0.0-1643.g3bfc837d.e 3.8 MB/s | 220 kB 00:00 2024-02-21T23:00:03.927 INFO:teuthology.orchestra.run.smithi028.stdout:(12/166): cephfs-shell-19.0.0-1643.g3bfc837d.el 668 kB/s | 27 kB 00:00 2024-02-21T23:00:04.002 INFO:teuthology.orchestra.run.smithi028.stdout:(13/166): libcephfs-devel-19.0.0-1643.g3bfc837d 424 kB/s | 31 kB 00:00 2024-02-21T23:00:04.086 INFO:teuthology.orchestra.run.smithi028.stdout:(14/166): libcephfs2-19.0.0-1643.g3bfc837d.el9. 8.9 MB/s | 745 kB 00:00 2024-02-21T23:00:04.144 INFO:teuthology.orchestra.run.smithi028.stdout:(15/166): libcephsqlite-19.0.0-1643.g3bfc837d.e 2.8 MB/s | 163 kB 00:00 2024-02-21T23:00:04.220 INFO:teuthology.orchestra.run.smithi028.stdout:(16/166): librados-devel-19.0.0-1643.g3bfc837d. 1.7 MB/s | 127 kB 00:00 2024-02-21T23:00:04.311 INFO:teuthology.orchestra.run.smithi028.stdout:(17/166): libradosstriper1-19.0.0-1643.g3bfc837 5.4 MB/s | 502 kB 00:00 2024-02-21T23:00:04.335 INFO:teuthology.orchestra.run.smithi081.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-02-21T23:00:04.337 INFO:teuthology.orchestra.run.smithi081.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-02-21T23:00:04.360 INFO:teuthology.orchestra.run.smithi081.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-02-21T23:00:04.361 INFO:teuthology.orchestra.run.smithi081.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-02-21T23:00:04.363 INFO:teuthology.orchestra.run.smithi081.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-02-21T23:00:04.364 INFO:teuthology.orchestra.run.smithi081.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-02-21T23:00:04.367 INFO:teuthology.orchestra.run.smithi081.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-02-21T23:00:04.371 INFO:teuthology.orchestra.run.smithi081.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-02-21T23:00:04.372 INFO:teuthology.orchestra.run.smithi081.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-02-21T23:00:04.470 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-02-21T23:00:04.488 INFO:teuthology.orchestra.run.smithi081.stdout:========================================================================================= 2024-02-21T23:00:04.489 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2024-02-21T23:00:04.489 INFO:teuthology.orchestra.run.smithi081.stdout:========================================================================================= 2024-02-21T23:00:04.489 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-02-21T23:00:04.489 INFO:teuthology.orchestra.run.smithi081.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-02-21T23:00:04.489 INFO:teuthology.orchestra.run.smithi081.stdout: ceph x86_64 2:19.0.0-1643.g3bfc837d.el9 ceph 6.6 k 2024-02-21T23:00:04.489 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-base x86_64 2:19.0.0-1643.g3bfc837d.el9 ceph 5.3 M 2024-02-21T23:00:04.489 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-fuse x86_64 2:19.0.0-1643.g3bfc837d.el9 ceph 889 k 2024-02-21T23:00:04.489 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1643.g3bfc837d.el9 ceph 145 k 2024-02-21T23:00:04.489 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr x86_64 2:19.0.0-1643.g3bfc837d.el9 ceph 1.5 M 2024-02-21T23:00:04.489 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1643.g3bfc837d.el9 ceph-noarch 138 k 2024-02-21T23:00:04.489 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1643.g3bfc837d.el9 ceph-noarch 1.8 M 2024-02-21T23:00:04.489 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1643.g3bfc837d.el9 ceph-noarch 7.4 M 2024-02-21T23:00:04.489 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-rook noarch 2:19.0.0-1643.g3bfc837d.el9 ceph-noarch 49 k 2024-02-21T23:00:04.490 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-radosgw x86_64 2:19.0.0-1643.g3bfc837d.el9 ceph 9.8 M 2024-02-21T23:00:04.490 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-test x86_64 2:19.0.0-1643.g3bfc837d.el9 ceph 47 M 2024-02-21T23:00:04.490 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-volume noarch 2:19.0.0-1643.g3bfc837d.el9 ceph-noarch 264 k 2024-02-21T23:00:04.490 INFO:teuthology.orchestra.run.smithi081.stdout: cephadm noarch 2:19.0.0-1643.g3bfc837d.el9 ceph-noarch 748 k 2024-02-21T23:00:04.490 INFO:teuthology.orchestra.run.smithi081.stdout: cephfs-mirror x86_64 2:19.0.0-1643.g3bfc837d.el9 ceph 220 k 2024-02-21T23:00:04.490 INFO:teuthology.orchestra.run.smithi081.stdout: cephfs-shell x86_64 2:19.0.0-1643.g3bfc837d.el9 ceph 27 k 2024-02-21T23:00:04.490 INFO:teuthology.orchestra.run.smithi081.stdout: cephfs-top noarch 2:19.0.0-1643.g3bfc837d.el9 ceph-noarch 18 k 2024-02-21T23:00:04.490 INFO:teuthology.orchestra.run.smithi081.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-02-21T23:00:04.490 INFO:teuthology.orchestra.run.smithi081.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-02-21T23:00:04.490 INFO:teuthology.orchestra.run.smithi081.stdout: libaio-devel x86_64 0.3.112-1.el8 CentOS-BaseOS 18 k 2024-02-21T23:00:04.490 INFO:teuthology.orchestra.run.smithi081.stdout: libcephfs-devel x86_64 2:19.0.0-1643.g3bfc837d.el9 ceph 31 k 2024-02-21T23:00:04.490 INFO:teuthology.orchestra.run.smithi081.stdout: libcephfs2 x86_64 2:19.0.0-1643.g3bfc837d.el9 ceph 745 k 2024-02-21T23:00:04.490 INFO:teuthology.orchestra.run.smithi081.stdout: librados-devel x86_64 2:19.0.0-1643.g3bfc837d.el9 ceph 127 k 2024-02-21T23:00:04.491 INFO:teuthology.orchestra.run.smithi081.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-02-21T23:00:04.491 INFO:teuthology.orchestra.run.smithi081.stdout: libuuid-devel x86_64 2.37.4-15.el9 appstream 26 k 2024-02-21T23:00:04.491 INFO:teuthology.orchestra.run.smithi081.stdout: openssl-devel x86_64 1:3.0.7-25.el9 appstream 4.1 M 2024-02-21T23:00:04.491 INFO:teuthology.orchestra.run.smithi081.stdout: postgresql x86_64 13.11-1.el9 appstream 1.6 M 2024-02-21T23:00:04.491 INFO:teuthology.orchestra.run.smithi081.stdout: postgresql-contrib x86_64 13.11-1.el9 appstream 905 k 2024-02-21T23:00:04.491 INFO:teuthology.orchestra.run.smithi081.stdout: postgresql-server x86_64 13.11-1.el9 appstream 5.7 M 2024-02-21T23:00:04.491 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cephfs x86_64 2:19.0.0-1643.g3bfc837d.el9 ceph 161 k 2024-02-21T23:00:04.491 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rados x86_64 2:19.0.0-1643.g3bfc837d.el9 ceph 321 k 2024-02-21T23:00:04.491 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rbd x86_64 2:19.0.0-1643.g3bfc837d.el9 ceph 298 k 2024-02-21T23:00:04.491 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rgw x86_64 2:19.0.0-1643.g3bfc837d.el9 ceph 100 k 2024-02-21T23:00:04.491 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-fuse x86_64 2:19.0.0-1643.g3bfc837d.el9 ceph 85 k 2024-02-21T23:00:04.491 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-mirror x86_64 2:19.0.0-1643.g3bfc837d.el9 ceph 3.1 M 2024-02-21T23:00:04.491 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-nbd x86_64 2:19.0.0-1643.g3bfc837d.el9 ceph 170 k 2024-02-21T23:00:04.491 INFO:teuthology.orchestra.run.smithi081.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-02-21T23:00:04.491 INFO:teuthology.orchestra.run.smithi081.stdout:Upgrading: 2024-02-21T23:00:04.492 INFO:teuthology.orchestra.run.smithi081.stdout: NetworkManager x86_64 1:1.45.91-1.el9 baseos 2.3 M 2024-02-21T23:00:04.492 INFO:teuthology.orchestra.run.smithi081.stdout: NetworkManager-libnm x86_64 1:1.45.91-1.el9 baseos 1.8 M 2024-02-21T23:00:04.492 INFO:teuthology.orchestra.run.smithi081.stdout: NetworkManager-team x86_64 1:1.45.91-1.el9 baseos 42 k 2024-02-21T23:00:04.492 INFO:teuthology.orchestra.run.smithi081.stdout: NetworkManager-tui x86_64 1:1.45.91-1.el9 baseos 247 k 2024-02-21T23:00:04.492 INFO:teuthology.orchestra.run.smithi081.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-02-21T23:00:04.492 INFO:teuthology.orchestra.run.smithi081.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-02-21T23:00:04.492 INFO:teuthology.orchestra.run.smithi081.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-02-21T23:00:04.492 INFO:teuthology.orchestra.run.smithi081.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-02-21T23:00:04.492 INFO:teuthology.orchestra.run.smithi081.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-02-21T23:00:04.492 INFO:teuthology.orchestra.run.smithi081.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-02-21T23:00:04.492 INFO:teuthology.orchestra.run.smithi081.stdout: librados2 x86_64 2:19.0.0-1643.g3bfc837d.el9 ceph 3.4 M 2024-02-21T23:00:04.492 INFO:teuthology.orchestra.run.smithi081.stdout: librbd1 x86_64 2:19.0.0-1643.g3bfc837d.el9 ceph 3.1 M 2024-02-21T23:00:04.492 INFO:teuthology.orchestra.run.smithi081.stdout: openssl x86_64 1:3.0.7-25.el9 baseos 1.2 M 2024-02-21T23:00:04.493 INFO:teuthology.orchestra.run.smithi081.stdout: openssl-libs x86_64 1:3.0.7-25.el9 baseos 2.2 M 2024-02-21T23:00:04.493 INFO:teuthology.orchestra.run.smithi081.stdout: xfsdump x86_64 3.1.12-4.el9 baseos 323 k 2024-02-21T23:00:04.493 INFO:teuthology.orchestra.run.smithi081.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-02-21T23:00:04.493 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2024-02-21T23:00:04.493 INFO:teuthology.orchestra.run.smithi081.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-02-21T23:00:04.493 INFO:teuthology.orchestra.run.smithi081.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-02-21T23:00:04.493 INFO:teuthology.orchestra.run.smithi081.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-02-21T23:00:04.493 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-common x86_64 2:19.0.0-1643.g3bfc837d.el9 ceph 20 M 2024-02-21T23:00:04.493 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1643.g3bfc837d.el9 ceph-noarch 31 k 2024-02-21T23:00:04.493 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mds x86_64 2:19.0.0-1643.g3bfc837d.el9 ceph 2.3 M 2024-02-21T23:00:04.493 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1643.g3bfc837d.el9 ceph-noarch 248 k 2024-02-21T23:00:04.494 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mon x86_64 2:19.0.0-1643.g3bfc837d.el9 ceph 4.7 M 2024-02-21T23:00:04.494 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-osd x86_64 2:19.0.0-1643.g3bfc837d.el9 ceph 17 M 2024-02-21T23:00:04.494 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1643.g3bfc837d.el9 ceph-noarch 15 k 2024-02-21T23:00:04.494 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-selinux x86_64 2:19.0.0-1643.g3bfc837d.el9 ceph 25 k 2024-02-21T23:00:04.494 INFO:teuthology.orchestra.run.smithi081.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-02-21T23:00:04.494 INFO:teuthology.orchestra.run.smithi081.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-02-21T23:00:04.494 INFO:teuthology.orchestra.run.smithi081.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-02-21T23:00:04.494 INFO:teuthology.orchestra.run.smithi081.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-02-21T23:00:04.494 INFO:teuthology.orchestra.run.smithi081.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-02-21T23:00:04.494 INFO:teuthology.orchestra.run.smithi081.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-02-21T23:00:04.494 INFO:teuthology.orchestra.run.smithi081.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-02-21T23:00:04.494 INFO:teuthology.orchestra.run.smithi081.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-02-21T23:00:04.494 INFO:teuthology.orchestra.run.smithi081.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-02-21T23:00:04.495 INFO:teuthology.orchestra.run.smithi081.stdout: libcephsqlite x86_64 2:19.0.0-1643.g3bfc837d.el9 ceph 163 k 2024-02-21T23:00:04.495 INFO:teuthology.orchestra.run.smithi081.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-02-21T23:00:04.495 INFO:teuthology.orchestra.run.smithi081.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-02-21T23:00:04.495 INFO:teuthology.orchestra.run.smithi081.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-02-21T23:00:04.495 INFO:teuthology.orchestra.run.smithi081.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-02-21T23:00:04.495 INFO:teuthology.orchestra.run.smithi081.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-02-21T23:00:04.495 INFO:teuthology.orchestra.run.smithi081.stdout: libradosstriper1 x86_64 2:19.0.0-1643.g3bfc837d.el9 ceph 502 k 2024-02-21T23:00:04.495 INFO:teuthology.orchestra.run.smithi081.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-02-21T23:00:04.495 INFO:teuthology.orchestra.run.smithi081.stdout: librgw2 x86_64 2:19.0.0-1643.g3bfc837d.el9 ceph 5.1 M 2024-02-21T23:00:04.495 INFO:teuthology.orchestra.run.smithi081.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-02-21T23:00:04.495 INFO:teuthology.orchestra.run.smithi081.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-02-21T23:00:04.496 INFO:teuthology.orchestra.run.smithi081.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-02-21T23:00:04.496 INFO:teuthology.orchestra.run.smithi081.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-02-21T23:00:04.496 INFO:teuthology.orchestra.run.smithi081.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-02-21T23:00:04.496 INFO:teuthology.orchestra.run.smithi081.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-02-21T23:00:04.496 INFO:teuthology.orchestra.run.smithi081.stdout: oniguruma x86_64 6.9.6-1.el9.5 baseos 218 k 2024-02-21T23:00:04.496 INFO:teuthology.orchestra.run.smithi081.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-02-21T23:00:04.496 INFO:teuthology.orchestra.run.smithi081.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-02-21T23:00:04.496 INFO:teuthology.orchestra.run.smithi081.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-02-21T23:00:04.496 INFO:teuthology.orchestra.run.smithi081.stdout: postgresql-private-libs x86_64 13.11-1.el9 appstream 135 k 2024-02-21T23:00:04.496 INFO:teuthology.orchestra.run.smithi081.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-02-21T23:00:04.496 INFO:teuthology.orchestra.run.smithi081.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-02-21T23:00:04.496 INFO:teuthology.orchestra.run.smithi081.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-02-21T23:00:04.496 INFO:teuthology.orchestra.run.smithi081.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-02-21T23:00:04.497 INFO:teuthology.orchestra.run.smithi081.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-02-21T23:00:04.497 INFO:teuthology.orchestra.run.smithi081.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-02-21T23:00:04.497 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-02-21T23:00:04.497 INFO:teuthology.orchestra.run.smithi081.stdout: python3-ceph-argparse x86_64 2:19.0.0-1643.g3bfc837d.el9 ceph 45 k 2024-02-21T23:00:04.497 INFO:teuthology.orchestra.run.smithi081.stdout: python3-ceph-common x86_64 2:19.0.0-1643.g3bfc837d.el9 ceph 131 k 2024-02-21T23:00:04.497 INFO:teuthology.orchestra.run.smithi081.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-02-21T23:00:04.497 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-02-21T23:00:04.497 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-02-21T23:00:04.497 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-02-21T23:00:04.497 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-02-21T23:00:04.497 INFO:teuthology.orchestra.run.smithi081.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-02-21T23:00:04.497 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-02-21T23:00:04.497 INFO:teuthology.orchestra.run.smithi081.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-02-21T23:00:04.498 INFO:teuthology.orchestra.run.smithi081.stdout: python3-google-auth noarch 1:2.27.0-1.el9 epel 203 k 2024-02-21T23:00:04.498 INFO:teuthology.orchestra.run.smithi081.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-02-21T23:00:04.498 INFO:teuthology.orchestra.run.smithi081.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-02-21T23:00:04.498 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-02-21T23:00:04.498 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-02-21T23:00:04.498 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-02-21T23:00:04.498 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-02-21T23:00:04.498 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-02-21T23:00:04.498 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-02-21T23:00:04.499 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-02-21T23:00:04.499 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-02-21T23:00:04.499 INFO:teuthology.orchestra.run.smithi081.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-02-21T23:00:04.499 INFO:teuthology.orchestra.run.smithi081.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-02-21T23:00:04.499 INFO:teuthology.orchestra.run.smithi081.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-02-21T23:00:04.499 INFO:teuthology.orchestra.run.smithi081.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-02-21T23:00:04.499 INFO:teuthology.orchestra.run.smithi081.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-02-21T23:00:04.499 INFO:teuthology.orchestra.run.smithi081.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-02-21T23:00:04.499 INFO:teuthology.orchestra.run.smithi081.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-02-21T23:00:04.499 INFO:teuthology.orchestra.run.smithi081.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-02-21T23:00:04.499 INFO:teuthology.orchestra.run.smithi081.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-02-21T23:00:04.499 INFO:teuthology.orchestra.run.smithi081.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-02-21T23:00:04.499 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-02-21T23:00:04.500 INFO:teuthology.orchestra.run.smithi081.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-02-21T23:00:04.500 INFO:teuthology.orchestra.run.smithi081.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-02-21T23:00:04.500 INFO:teuthology.orchestra.run.smithi081.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-02-21T23:00:04.500 INFO:teuthology.orchestra.run.smithi081.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-02-21T23:00:04.500 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-02-21T23:00:04.500 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-02-21T23:00:04.500 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-02-21T23:00:04.500 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-02-21T23:00:04.500 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-02-21T23:00:04.500 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-02-21T23:00:04.500 INFO:teuthology.orchestra.run.smithi081.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-02-21T23:00:04.500 INFO:teuthology.orchestra.run.smithi081.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-02-21T23:00:04.501 INFO:teuthology.orchestra.run.smithi081.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-02-21T23:00:04.501 INFO:teuthology.orchestra.run.smithi081.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-02-21T23:00:04.501 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-02-21T23:00:04.501 INFO:teuthology.orchestra.run.smithi081.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-02-21T23:00:04.501 INFO:teuthology.orchestra.run.smithi081.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-02-21T23:00:04.501 INFO:teuthology.orchestra.run.smithi081.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-02-21T23:00:04.501 INFO:teuthology.orchestra.run.smithi081.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-02-21T23:00:04.501 INFO:teuthology.orchestra.run.smithi081.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-02-21T23:00:04.501 INFO:teuthology.orchestra.run.smithi081.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-02-21T23:00:04.501 INFO:teuthology.orchestra.run.smithi081.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-02-21T23:00:04.501 INFO:teuthology.orchestra.run.smithi081.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-02-21T23:00:04.501 INFO:teuthology.orchestra.run.smithi081.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-02-21T23:00:04.502 INFO:teuthology.orchestra.run.smithi081.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-02-21T23:00:04.502 INFO:teuthology.orchestra.run.smithi081.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-02-21T23:00:04.502 INFO:teuthology.orchestra.run.smithi081.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-02-21T23:00:04.502 INFO:teuthology.orchestra.run.smithi081.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-02-21T23:00:04.502 INFO:teuthology.orchestra.run.smithi081.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-02-21T23:00:04.502 INFO:teuthology.orchestra.run.smithi081.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-02-21T23:00:04.502 INFO:teuthology.orchestra.run.smithi081.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-02-21T23:00:04.502 INFO:teuthology.orchestra.run.smithi081.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-02-21T23:00:04.502 INFO:teuthology.orchestra.run.smithi081.stdout: zlib-devel x86_64 1.2.11-40.el9 appstream 45 k 2024-02-21T23:00:04.502 INFO:teuthology.orchestra.run.smithi081.stdout:Installing weak dependencies: 2024-02-21T23:00:04.502 INFO:teuthology.orchestra.run.smithi081.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-02-21T23:00:04.502 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-02-21T23:00:04.502 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-02-21T23:00:04.502 INFO:teuthology.orchestra.run.smithi081.stdout:========================================================================================= 2024-02-21T23:00:04.503 INFO:teuthology.orchestra.run.smithi081.stdout:Install 150 Packages 2024-02-21T23:00:04.503 INFO:teuthology.orchestra.run.smithi081.stdout:Upgrade 16 Packages 2024-02-21T23:00:04.503 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-02-21T23:00:04.503 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 222 M 2024-02-21T23:00:04.503 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-02-21T23:00:04.770 INFO:teuthology.orchestra.run.smithi028.stdout:(18/166): librgw2-19.0.0-1643.g3bfc837d.el9.x86 11 MB/s | 5.1 MB 00:00 2024-02-21T23:00:04.837 INFO:teuthology.orchestra.run.smithi028.stdout:(19/166): python3-ceph-argparse-19.0.0-1643.g3b 689 kB/s | 45 kB 00:00 2024-02-21T23:00:04.904 INFO:teuthology.orchestra.run.smithi028.stdout:(20/166): python3-ceph-common-19.0.0-1643.g3bfc 1.9 MB/s | 131 kB 00:00 2024-02-21T23:00:04.971 INFO:teuthology.orchestra.run.smithi028.stdout:(21/166): python3-cephfs-19.0.0-1643.g3bfc837d. 2.4 MB/s | 161 kB 00:00 2024-02-21T23:00:05.046 INFO:teuthology.orchestra.run.smithi028.stdout:(22/166): python3-rados-19.0.0-1643.g3bfc837d.e 4.2 MB/s | 321 kB 00:00 2024-02-21T23:00:05.130 INFO:teuthology.orchestra.run.smithi028.stdout:(23/166): python3-rbd-19.0.0-1643.g3bfc837d.el9 3.5 MB/s | 298 kB 00:00 2024-02-21T23:00:05.197 INFO:teuthology.orchestra.run.smithi028.stdout:(24/166): python3-rgw-19.0.0-1643.g3bfc837d.el9 1.5 MB/s | 100 kB 00:00 2024-02-21T23:00:05.255 INFO:teuthology.orchestra.run.smithi028.stdout:(25/166): rbd-fuse-19.0.0-1643.g3bfc837d.el9.x8 1.5 MB/s | 85 kB 00:00 2024-02-21T23:00:05.419 INFO:teuthology.orchestra.run.smithi081.stdout:(1/166): ceph-19.0.0-1643.g3bfc837d.el9.x86_64. 15 kB/s | 6.6 kB 00:00 2024-02-21T23:00:05.539 INFO:teuthology.orchestra.run.smithi028.stdout:(26/166): rbd-mirror-19.0.0-1643.g3bfc837d.el9. 11 MB/s | 3.1 MB 00:00 2024-02-21T23:00:05.794 INFO:teuthology.orchestra.run.smithi081.stdout:(2/166): ceph-fuse-19.0.0-1643.g3bfc837d.el9.x8 2.3 MB/s | 889 kB 00:00 2024-02-21T23:00:05.886 INFO:teuthology.orchestra.run.smithi081.stdout:(3/166): ceph-immutable-object-cache-19.0.0-164 1.6 MB/s | 145 kB 00:00 2024-02-21T23:00:06.006 INFO:teuthology.orchestra.run.smithi028.stdout:(27/166): ceph-test-19.0.0-1643.g3bfc837d.el9.x 15 MB/s | 47 MB 00:03 2024-02-21T23:00:06.039 INFO:teuthology.orchestra.run.smithi028.stdout:(28/166): rbd-nbd-19.0.0-1643.g3bfc837d.el9.x86 342 kB/s | 170 kB 00:00 2024-02-21T23:00:06.106 INFO:teuthology.orchestra.run.smithi028.stdout:(29/166): ceph-grafana-dashboards-19.0.0-1643.g 317 kB/s | 31 kB 00:00 2024-02-21T23:00:06.415 INFO:teuthology.orchestra.run.smithi028.stdout:(30/166): ceph-mgr-dashboard-19.0.0-1643.g3bfc8 5.8 MB/s | 1.8 MB 00:00 2024-02-21T23:00:06.674 INFO:teuthology.orchestra.run.smithi028.stdout:(31/166): ceph-mgr-cephadm-19.0.0-1643.g3bfc837 218 kB/s | 138 kB 00:00 2024-02-21T23:00:06.832 INFO:teuthology.orchestra.run.smithi028.stdout:(32/166): ceph-mgr-modules-core-19.0.0-1643.g3b 1.5 MB/s | 248 kB 00:00 2024-02-21T23:00:06.861 INFO:teuthology.orchestra.run.smithi081.stdout:(4/166): ceph-mds-19.0.0-1643.g3bfc837d.el9.x86 2.4 MB/s | 2.3 MB 00:00 2024-02-21T23:00:06.916 INFO:teuthology.orchestra.run.smithi028.stdout:(33/166): ceph-mgr-rook-19.0.0-1643.g3bfc837d.e 597 kB/s | 49 kB 00:00 2024-02-21T23:00:06.983 INFO:teuthology.orchestra.run.smithi028.stdout:(34/166): ceph-prometheus-alerts-19.0.0-1643.g3 233 kB/s | 15 kB 00:00 2024-02-21T23:00:07.133 INFO:teuthology.orchestra.run.smithi028.stdout:(35/166): ceph-mgr-diskprediction-local-19.0.0- 10 MB/s | 7.4 MB 00:00 2024-02-21T23:00:07.167 INFO:teuthology.orchestra.run.smithi028.stdout:(36/166): ceph-volume-19.0.0-1643.g3bfc837d.el9 1.4 MB/s | 264 kB 00:00 2024-02-21T23:00:07.170 INFO:teuthology.orchestra.run.smithi081.stdout:(5/166): ceph-common-19.0.0-1643.g3bfc837d.el9. 9.3 MB/s | 20 MB 00:02 2024-02-21T23:00:07.217 INFO:teuthology.orchestra.run.smithi028.stdout:(37/166): cephfs-top-19.0.0-1643.g3bfc837d.el9. 370 kB/s | 18 kB 00:00 2024-02-21T23:00:07.276 INFO:teuthology.orchestra.run.smithi028.stdout:(38/166): cephadm-19.0.0-1643.g3bfc837d.el9.noa 5.2 MB/s | 748 kB 00:00 2024-02-21T23:00:07.287 INFO:teuthology.orchestra.run.smithi081.stdout:(6/166): ceph-base-19.0.0-1643.g3bfc837d.el9.x8 2.3 MB/s | 5.3 MB 00:02 2024-02-21T23:00:07.354 INFO:teuthology.orchestra.run.smithi081.stdout:(7/166): ceph-mgr-19.0.0-1643.g3bfc837d.el9.x86 3.0 MB/s | 1.5 MB 00:00 2024-02-21T23:00:07.401 INFO:teuthology.orchestra.run.smithi028.stdout:(39/166): libaio-devel-0.3.112-1.el8.x86_64.rpm 96 kB/s | 18 kB 00:00 2024-02-21T23:00:07.593 INFO:teuthology.orchestra.run.smithi028.stdout:(40/166): fuse-2.9.9-15.el9.x86_64.rpm 327 kB/s | 80 kB 00:00 2024-02-21T23:00:07.627 INFO:teuthology.orchestra.run.smithi028.stdout:(41/166): jq-1.6-16.el9.x86_64.rpm 834 kB/s | 187 kB 00:00 2024-02-21T23:00:07.687 INFO:teuthology.orchestra.run.smithi081.stdout:(8/166): ceph-mon-19.0.0-1643.g3bfc837d.el9.x86 9.1 MB/s | 4.7 MB 00:00 2024-02-21T23:00:07.710 INFO:teuthology.orchestra.run.smithi028.stdout:(42/166): libquadmath-11.4.1-3.el9.x86_64.rpm 2.3 MB/s | 195 kB 00:00 2024-02-21T23:00:07.752 INFO:teuthology.orchestra.run.smithi028.stdout:(43/166): libgfortran-11.4.1-3.el9.x86_64.rpm 5.0 MB/s | 803 kB 00:00 2024-02-21T23:00:07.754 INFO:teuthology.orchestra.run.smithi081.stdout:(9/166): ceph-selinux-19.0.0-1643.g3bfc837d.el9 382 kB/s | 25 kB 00:00 2024-02-21T23:00:07.828 INFO:teuthology.orchestra.run.smithi028.stdout:(44/166): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.8 MB/s | 218 kB 00:00 2024-02-21T23:00:07.861 INFO:teuthology.orchestra.run.smithi028.stdout:(45/166): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.3 MB/s | 253 kB 00:00 2024-02-21T23:00:07.928 INFO:teuthology.orchestra.run.smithi028.stdout:(46/166): python3-ply-3.11-14.el9.noarch.rpm 1.6 MB/s | 106 kB 00:00 2024-02-21T23:00:07.996 INFO:teuthology.orchestra.run.smithi028.stdout:(47/166): python3-cryptography-36.0.1-4.el9.x86 7.4 MB/s | 1.2 MB 00:00 2024-02-21T23:00:08.037 INFO:teuthology.orchestra.run.smithi028.stdout:(48/166): python3-pycparser-2.20-6.el9.noarch.r 1.2 MB/s | 135 kB 00:00 2024-02-21T23:00:08.079 INFO:teuthology.orchestra.run.smithi028.stdout:(49/166): python3-requests-2.25.1-8.el9.noarch. 1.5 MB/s | 125 kB 00:00 2024-02-21T23:00:08.121 INFO:teuthology.orchestra.run.smithi028.stdout:(50/166): python3-urllib3-1.26.5-5.el9.noarch.r 2.5 MB/s | 215 kB 00:00 2024-02-21T23:00:08.305 INFO:teuthology.orchestra.run.smithi081.stdout:(10/166): ceph-radosgw-19.0.0-1643.g3bfc837d.el 10 MB/s | 9.8 MB 00:00 2024-02-21T23:00:08.388 INFO:teuthology.orchestra.run.smithi081.stdout:(11/166): cephfs-mirror-19.0.0-1643.g3bfc837d.e 2.6 MB/s | 220 kB 00:00 2024-02-21T23:00:08.429 INFO:teuthology.orchestra.run.smithi028.stdout:(51/166): automake-1.16.2-8.el9.noarch.rpm 1.9 MB/s | 689 kB 00:00 2024-02-21T23:00:08.464 INFO:teuthology.orchestra.run.smithi081.stdout:(12/166): cephfs-shell-19.0.0-1643.g3bfc837d.el 365 kB/s | 27 kB 00:00 2024-02-21T23:00:08.513 INFO:teuthology.orchestra.run.smithi028.stdout:(52/166): bison-3.7.4-5.el9.x86_64.rpm 2.4 MB/s | 943 kB 00:00 2024-02-21T23:00:08.538 INFO:teuthology.orchestra.run.smithi028.stdout:(53/166): boost-program-options-1.75.0-8.el9.x8 992 kB/s | 107 kB 00:00 2024-02-21T23:00:08.556 INFO:teuthology.orchestra.run.smithi081.stdout:(13/166): libcephfs-devel-19.0.0-1643.g3bfc837d 343 kB/s | 31 kB 00:00 2024-02-21T23:00:08.588 INFO:teuthology.orchestra.run.smithi028.stdout:(54/166): elfutils-libelf-devel-0.190-2.el9.x86 289 kB/s | 21 kB 00:00 2024-02-21T23:00:08.630 INFO:teuthology.orchestra.run.smithi028.stdout:(55/166): flex-2.6.4-9.el9.x86_64.rpm 3.4 MB/s | 313 kB 00:00 2024-02-21T23:00:08.656 INFO:teuthology.orchestra.run.smithi081.stdout:(14/166): libcephfs2-19.0.0-1643.g3bfc837d.el9. 7.3 MB/s | 745 kB 00:00 2024-02-21T23:00:08.664 INFO:teuthology.orchestra.run.smithi028.stdout:(56/166): flexiblas-3.0.4-8.el9.x86_64.rpm 438 kB/s | 33 kB 00:00 2024-02-21T23:00:08.731 INFO:teuthology.orchestra.run.smithi028.stdout:(57/166): flexiblas-openblas-openmp-3.0.4-8.el9 276 kB/s | 18 kB 00:00 2024-02-21T23:00:08.748 INFO:teuthology.orchestra.run.smithi081.stdout:(15/166): libcephsqlite-19.0.0-1643.g3bfc837d.e 1.8 MB/s | 163 kB 00:00 2024-02-21T23:00:08.799 INFO:teuthology.orchestra.run.smithi028.stdout:(58/166): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.4 MB/s | 160 kB 00:00 2024-02-21T23:00:08.832 INFO:teuthology.orchestra.run.smithi081.stdout:(16/166): librados-devel-19.0.0-1643.g3bfc837d. 1.5 MB/s | 127 kB 00:00 2024-02-21T23:00:08.891 INFO:teuthology.orchestra.run.smithi028.stdout:(59/166): flexiblas-netlib-3.0.4-8.el9.x86_64.r 12 MB/s | 3.0 MB 00:00 2024-02-21T23:00:08.915 INFO:teuthology.orchestra.run.smithi028.stdout:(60/166): librabbitmq-0.11.0-7.el9.x86_64.rpm 389 kB/s | 45 kB 00:00 2024-02-21T23:00:08.940 INFO:teuthology.orchestra.run.smithi081.stdout:(17/166): libradosstriper1-19.0.0-1643.g3bfc837 4.6 MB/s | 502 kB 00:00 2024-02-21T23:00:08.974 INFO:teuthology.orchestra.run.smithi028.stdout:(61/166): librdkafka-1.6.1-102.el9.x86_64.rpm 7.9 MB/s | 662 kB 00:00 2024-02-21T23:00:09.016 INFO:teuthology.orchestra.run.smithi028.stdout:(62/166): libtool-2.4.6-45.el9.x86_64.rpm 5.7 MB/s | 582 kB 00:00 2024-02-21T23:00:09.058 INFO:teuthology.orchestra.run.smithi028.stdout:(63/166): libuuid-devel-2.37.4-15.el9.x86_64.rp 308 kB/s | 26 kB 00:00 2024-02-21T23:00:09.100 INFO:teuthology.orchestra.run.smithi028.stdout:(64/166): libzstd-devel-1.5.1-2.el9.x86_64.rpm 571 kB/s | 47 kB 00:00 2024-02-21T23:00:09.141 INFO:teuthology.orchestra.run.smithi028.stdout:(65/166): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.4 MB/s | 292 kB 00:00 2024-02-21T23:00:09.192 INFO:teuthology.orchestra.run.smithi028.stdout:(66/166): lua-5.4.4-4.el9.x86_64.rpm 2.0 MB/s | 188 kB 00:00 2024-02-21T23:00:09.243 INFO:teuthology.orchestra.run.smithi028.stdout:(67/166): openblas-0.3.21-2.el9.x86_64.rpm 353 kB/s | 35 kB 00:00 2024-02-21T23:00:09.468 INFO:teuthology.orchestra.run.smithi028.stdout:(68/166): openssl-devel-3.0.7-25.el9.x86_64.rpm 18 MB/s | 4.1 MB 00:00 2024-02-21T23:00:09.549 INFO:teuthology.orchestra.run.smithi081.stdout:(18/166): librgw2-19.0.0-1643.g3bfc837d.el9.x86 8.4 MB/s | 5.1 MB 00:00 2024-02-21T23:00:09.585 INFO:teuthology.orchestra.run.smithi028.stdout:(69/166): openblas-openmp-0.3.21-2.el9.x86_64.r 12 MB/s | 4.7 MB 00:00 2024-02-21T23:00:09.641 INFO:teuthology.orchestra.run.smithi081.stdout:(19/166): python3-ceph-argparse-19.0.0-1643.g3b 498 kB/s | 45 kB 00:00 2024-02-21T23:00:09.651 INFO:teuthology.orchestra.run.smithi028.stdout:(70/166): postgresql-13.11-1.el9.x86_64.rpm 8.7 MB/s | 1.6 MB 00:00 2024-02-21T23:00:09.702 INFO:teuthology.orchestra.run.smithi028.stdout:(71/166): postgresql-contrib-13.11-1.el9.x86_64 7.6 MB/s | 905 kB 00:00 2024-02-21T23:00:09.724 INFO:teuthology.orchestra.run.smithi081.stdout:(20/166): python3-ceph-common-19.0.0-1643.g3bfc 1.6 MB/s | 131 kB 00:00 2024-02-21T23:00:09.727 INFO:teuthology.orchestra.run.smithi028.stdout:(72/166): postgresql-private-libs-13.11-1.el9.x 1.8 MB/s | 135 kB 00:00 2024-02-21T23:00:09.816 INFO:teuthology.orchestra.run.smithi081.stdout:(21/166): python3-cephfs-19.0.0-1643.g3bfc837d. 1.7 MB/s | 161 kB 00:00 2024-02-21T23:00:09.836 INFO:teuthology.orchestra.run.smithi028.stdout:(73/166): protobuf-3.14.0-13.el9.x86_64.rpm 9.4 MB/s | 1.0 MB 00:00 2024-02-21T23:00:09.911 INFO:teuthology.orchestra.run.smithi028.stdout:(74/166): python3-attrs-20.3.0-7.el9.noarch.rpm 1.2 MB/s | 87 kB 00:00 2024-02-21T23:00:09.917 INFO:teuthology.orchestra.run.smithi081.stdout:(22/166): python3-rados-19.0.0-1643.g3bfc837d.e 3.1 MB/s | 321 kB 00:00 2024-02-21T23:00:10.026 INFO:teuthology.orchestra.run.smithi081.stdout:(23/166): python3-rbd-19.0.0-1643.g3bfc837d.el9 2.7 MB/s | 298 kB 00:00 2024-02-21T23:00:10.036 INFO:teuthology.orchestra.run.smithi028.stdout:(75/166): postgresql-server-13.11-1.el9.x86_64. 17 MB/s | 5.7 MB 00:00 2024-02-21T23:00:10.117 INFO:teuthology.orchestra.run.smithi081.stdout:(24/166): python3-rgw-19.0.0-1643.g3bfc837d.el9 1.1 MB/s | 100 kB 00:00 2024-02-21T23:00:10.120 INFO:teuthology.orchestra.run.smithi028.stdout:(76/166): python3-devel-3.9.17-2.el9.x86_64.rpm 2.9 MB/s | 245 kB 00:00 2024-02-21T23:00:10.201 INFO:teuthology.orchestra.run.smithi081.stdout:(25/166): rbd-fuse-19.0.0-1643.g3bfc837d.el9.x8 1.0 MB/s | 85 kB 00:00 2024-02-21T23:00:10.228 INFO:teuthology.orchestra.run.smithi028.stdout:(77/166): python3-babel-2.9.1-2.el9.noarch.rpm 19 MB/s | 6.0 MB 00:00 2024-02-21T23:00:10.262 INFO:teuthology.orchestra.run.smithi028.stdout:(78/166): python3-jinja2-2.11.3-5.el9.noarch.rp 1.7 MB/s | 248 kB 00:00 2024-02-21T23:00:10.296 INFO:teuthology.orchestra.run.smithi028.stdout:(79/166): python3-jsonpatch-1.21-16.el9.noarch. 389 kB/s | 26 kB 00:00 2024-02-21T23:00:10.421 INFO:teuthology.orchestra.run.smithi028.stdout:(80/166): ceph-radosgw-19.0.0-1643.g3bfc837d.el 1.3 MB/s | 9.8 MB 00:07 2024-02-21T23:00:10.446 INFO:teuthology.orchestra.run.smithi028.stdout:(81/166): python3-jsonpointer-2.0-4.el9.noarch. 104 kB/s | 19 kB 00:00 2024-02-21T23:00:10.471 INFO:teuthology.orchestra.run.smithi028.stdout:(82/166): python3-mako-1.1.4-6.el9.noarch.rpm 985 kB/s | 172 kB 00:00 2024-02-21T23:00:10.543 INFO:teuthology.orchestra.run.smithi081.stdout:(26/166): rbd-mirror-19.0.0-1643.g3bfc837d.el9. 9.1 MB/s | 3.1 MB 00:00 2024-02-21T23:00:10.572 INFO:teuthology.orchestra.run.smithi028.stdout:(83/166): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.5 MB/s | 250 kB 00:00 2024-02-21T23:00:10.597 INFO:teuthology.orchestra.run.smithi028.stdout:(84/166): python3-markupsafe-1.1.1-12.el9.x86_6 198 kB/s | 35 kB 00:00 2024-02-21T23:00:10.610 INFO:teuthology.orchestra.run.smithi081.stdout:(27/166): rbd-nbd-19.0.0-1643.g3bfc837d.el9.x86 2.5 MB/s | 170 kB 00:00 2024-02-21T23:00:10.647 INFO:teuthology.orchestra.run.smithi028.stdout:(85/166): python3-oauthlib-3.1.1-5.el9.noarch.r 2.9 MB/s | 222 kB 00:00 2024-02-21T23:00:10.694 INFO:teuthology.orchestra.run.smithi081.stdout:(28/166): ceph-grafana-dashboards-19.0.0-1643.g 382 kB/s | 31 kB 00:00 2024-02-21T23:00:10.777 INFO:teuthology.orchestra.run.smithi081.stdout:(29/166): ceph-mgr-cephadm-19.0.0-1643.g3bfc837 1.6 MB/s | 138 kB 00:00 2024-02-21T23:00:10.798 INFO:teuthology.orchestra.run.smithi028.stdout:(86/166): python3-numpy-1.20.1-5.el9.x86_64.rpm 15 MB/s | 5.1 MB 00:00 2024-02-21T23:00:10.839 INFO:teuthology.orchestra.run.smithi028.stdout:(87/166): python3-packaging-20.9-5.el9.noarch.r 319 kB/s | 77 kB 00:00 2024-02-21T23:00:10.856 INFO:teuthology.orchestra.run.smithi028.stdout:(88/166): python3-prettytable-0.7.2-27.el9.noar 203 kB/s | 42 kB 00:00 2024-02-21T23:00:10.915 INFO:teuthology.orchestra.run.smithi028.stdout:(89/166): python3-protobuf-3.14.0-13.el9.noarch 2.3 MB/s | 269 kB 00:00 2024-02-21T23:00:10.965 INFO:teuthology.orchestra.run.smithi028.stdout:(90/166): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 159 kB 00:00 2024-02-21T23:00:10.990 INFO:teuthology.orchestra.run.smithi028.stdout:(91/166): python3-pytz-2021.1-5.el9.noarch.rpm 686 kB/s | 51 kB 00:00 2024-02-21T23:00:11.019 INFO:teuthology.orchestra.run.smithi081.stdout:(30/166): ceph-mgr-dashboard-19.0.0-1643.g3bfc8 7.4 MB/s | 1.8 MB 00:00 2024-02-21T23:00:11.049 INFO:teuthology.orchestra.run.smithi028.stdout:(92/166): python3-pyasn1-modules-0.4.8-6.el9.no 1.4 MB/s | 279 kB 00:00 2024-02-21T23:00:11.074 INFO:teuthology.orchestra.run.smithi028.stdout:(93/166): python3-requests-oauthlib-1.3.0-12.el 497 kB/s | 54 kB 00:00 2024-02-21T23:00:11.116 INFO:teuthology.orchestra.run.smithi028.stdout:(94/166): python3-toml-0.10.2-6.el9.noarch.rpm 630 kB/s | 42 kB 00:00 2024-02-21T23:00:11.150 INFO:teuthology.orchestra.run.smithi028.stdout:(95/166): python3-wcwidth-0.2.5-8.el9.noarch.rp 592 kB/s | 44 kB 00:00 2024-02-21T23:00:11.225 INFO:teuthology.orchestra.run.smithi028.stdout:(96/166): qatlib-23.11.0-1.el9.x86_64.rpm 2.0 MB/s | 216 kB 00:00 2024-02-21T23:00:11.251 INFO:teuthology.orchestra.run.smithi028.stdout:(97/166): qatlib-service-23.11.0-1.el9.x86_64.r 349 kB/s | 35 kB 00:00 2024-02-21T23:00:11.293 INFO:teuthology.orchestra.run.smithi028.stdout:(98/166): qatzip-libs-1.1.2-2.el9.x86_64.rpm 674 kB/s | 44 kB 00:00 2024-02-21T23:00:11.343 INFO:teuthology.orchestra.run.smithi028.stdout:(99/166): socat-1.7.4.1-5.el9.x86_64.rpm 3.3 MB/s | 305 kB 00:00 2024-02-21T23:00:11.376 INFO:teuthology.orchestra.run.smithi028.stdout:(100/166): uuid-1.6.2-55.el9.x86_64.rpm 699 kB/s | 58 kB 00:00 2024-02-21T23:00:11.444 INFO:teuthology.orchestra.run.smithi028.stdout:(101/166): xmlstarlet-1.6.1-20.el9.x86_64.rpm 965 kB/s | 64 kB 00:00 2024-02-21T23:00:11.494 INFO:teuthology.orchestra.run.smithi028.stdout:(102/166): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 625 kB/s | 94 kB 00:00 2024-02-21T23:00:11.527 INFO:teuthology.orchestra.run.smithi028.stdout:(103/166): zlib-devel-1.2.11-40.el9.x86_64.rpm 547 kB/s | 45 kB 00:00 2024-02-21T23:00:11.570 INFO:teuthology.orchestra.run.smithi081.stdout:(31/166): ceph-test-19.0.0-1643.g3bfc837d.el9.x 12 MB/s | 47 MB 00:03 2024-02-21T23:00:11.920 INFO:teuthology.orchestra.run.smithi081.stdout:(32/166): ceph-osd-19.0.0-1643.g3bfc837d.el9.x8 3.6 MB/s | 17 MB 00:04 2024-02-21T23:00:11.953 INFO:teuthology.orchestra.run.smithi028.stdout:(104/166): python3-scipy-1.6.2-8.el9.x86_64.rpm 17 MB/s | 16 MB 00:00 2024-02-21T23:00:11.954 INFO:teuthology.orchestra.run.smithi081.stdout:(33/166): ceph-mgr-modules-core-19.0.0-1643.g3b 650 kB/s | 248 kB 00:00 2024-02-21T23:00:11.979 INFO:teuthology.orchestra.run.smithi081.stdout:(34/166): ceph-mgr-rook-19.0.0-1643.g3bfc837d.e 859 kB/s | 49 kB 00:00 2024-02-21T23:00:12.013 INFO:teuthology.orchestra.run.smithi081.stdout:(35/166): ceph-prometheus-alerts-19.0.0-1643.g3 265 kB/s | 15 kB 00:00 2024-02-21T23:00:12.054 INFO:teuthology.orchestra.run.smithi081.stdout:(36/166): ceph-volume-19.0.0-1643.g3bfc837d.el9 3.5 MB/s | 264 kB 00:00 2024-02-21T23:00:12.105 INFO:teuthology.orchestra.run.smithi081.stdout:(37/166): cephfs-top-19.0.0-1643.g3bfc837d.el9. 372 kB/s | 18 kB 00:00 2024-02-21T23:00:12.205 INFO:teuthology.orchestra.run.smithi081.stdout:(38/166): cephadm-19.0.0-1643.g3bfc837d.el9.noa 3.8 MB/s | 748 kB 00:00 2024-02-21T23:00:12.331 INFO:teuthology.orchestra.run.smithi081.stdout:(39/166): ceph-mgr-diskprediction-local-19.0.0- 5.6 MB/s | 7.4 MB 00:01 2024-02-21T23:00:12.384 INFO:teuthology.orchestra.run.smithi081.stdout:(40/166): libaio-devel-0.3.112-1.el8.x86_64.rpm 63 kB/s | 18 kB 00:00 2024-02-21T23:00:12.403 INFO:teuthology.orchestra.run.smithi028.stdout:(105/166): lua-devel-5.4.4-4.el9.x86_64.rpm 25 kB/s | 22 kB 00:00 2024-02-21T23:00:12.473 INFO:teuthology.orchestra.run.smithi081.stdout:(41/166): fuse-2.9.9-15.el9.x86_64.rpm 304 kB/s | 80 kB 00:00 2024-02-21T23:00:12.553 INFO:teuthology.orchestra.run.smithi028.stdout:(106/166): abseil-cpp-20211102.0-3.el9.x86_64.r 1.0 MB/s | 548 kB 00:00 2024-02-21T23:00:12.573 INFO:teuthology.orchestra.run.smithi081.stdout:(42/166): jq-1.6-16.el9.x86_64.rpm 777 kB/s | 187 kB 00:00 2024-02-21T23:00:12.606 INFO:teuthology.orchestra.run.smithi081.stdout:(43/166): libquadmath-11.4.1-3.el9.x86_64.rpm 1.4 MB/s | 195 kB 00:00 2024-02-21T23:00:12.628 INFO:teuthology.orchestra.run.smithi028.stdout:(107/166): grpc-data-1.46.7-10.el9.noarch.rpm 261 kB/s | 19 kB 00:00 2024-02-21T23:00:12.665 INFO:teuthology.orchestra.run.smithi081.stdout:(44/166): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.3 MB/s | 218 kB 00:00 2024-02-21T23:00:12.699 INFO:teuthology.orchestra.run.smithi081.stdout:(45/166): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.7 MB/s | 253 kB 00:00 2024-02-21T23:00:12.740 INFO:teuthology.orchestra.run.smithi081.stdout:(46/166): libgfortran-11.4.1-3.el9.x86_64.rpm 2.2 MB/s | 803 kB 00:00 2024-02-21T23:00:12.766 INFO:teuthology.orchestra.run.smithi081.stdout:(47/166): python3-ply-3.11-14.el9.noarch.rpm 1.6 MB/s | 106 kB 00:00 2024-02-21T23:00:12.800 INFO:teuthology.orchestra.run.smithi081.stdout:(48/166): python3-pycparser-2.20-6.el9.noarch.r 2.3 MB/s | 135 kB 00:00 2024-02-21T23:00:12.833 INFO:teuthology.orchestra.run.smithi081.stdout:(49/166): python3-requests-2.25.1-8.el9.noarch. 1.8 MB/s | 125 kB 00:00 2024-02-21T23:00:12.867 INFO:teuthology.orchestra.run.smithi081.stdout:(50/166): python3-urllib3-1.26.5-5.el9.noarch.r 3.2 MB/s | 215 kB 00:00 2024-02-21T23:00:12.917 INFO:teuthology.orchestra.run.smithi081.stdout:(51/166): python3-cryptography-36.0.1-4.el9.x86 4.9 MB/s | 1.2 MB 00:00 2024-02-21T23:00:12.920 INFO:teuthology.orchestra.run.smithi028.stdout:(108/166): libarrow-9.0.0-10.el9.x86_64.rpm 15 MB/s | 4.4 MB 00:00 2024-02-21T23:00:12.954 INFO:teuthology.orchestra.run.smithi028.stdout:(109/166): gperftools-libs-2.9.1-2.el9.x86_64.r 562 kB/s | 309 kB 00:00 2024-02-21T23:00:12.976 INFO:teuthology.orchestra.run.smithi081.stdout:(52/166): boost-program-options-1.75.0-8.el9.x8 1.8 MB/s | 107 kB 00:00 2024-02-21T23:00:12.996 INFO:teuthology.orchestra.run.smithi028.stdout:(110/166): libarrow-doc-9.0.0-10.el9.noarch.rpm 349 kB/s | 26 kB 00:00 2024-02-21T23:00:13.029 INFO:teuthology.orchestra.run.smithi028.stdout:(111/166): liboath-2.6.7-2.el9.x86_64.rpm 654 kB/s | 49 kB 00:00 2024-02-21T23:00:13.035 INFO:teuthology.orchestra.run.smithi081.stdout:(53/166): elfutils-libelf-devel-0.190-2.el9.x86 374 kB/s | 21 kB 00:00 2024-02-21T23:00:13.063 INFO:teuthology.orchestra.run.smithi028.stdout:(112/166): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-02-21T23:00:13.077 INFO:teuthology.orchestra.run.smithi081.stdout:(54/166): bison-3.7.4-5.el9.x86_64.rpm 4.4 MB/s | 943 kB 00:00 2024-02-21T23:00:13.122 INFO:teuthology.orchestra.run.smithi028.stdout:(113/166): luarocks-3.9.2-1.el9.noarch.rpm 1.6 MB/s | 151 kB 00:00 2024-02-21T23:00:13.127 INFO:teuthology.orchestra.run.smithi081.stdout:(55/166): flex-2.6.4-9.el9.x86_64.rpm 3.4 MB/s | 313 kB 00:00 2024-02-21T23:00:13.152 INFO:teuthology.orchestra.run.smithi081.stdout:(56/166): flexiblas-3.0.4-8.el9.x86_64.rpm 440 kB/s | 33 kB 00:00 2024-02-21T23:00:13.172 INFO:teuthology.orchestra.run.smithi028.stdout:(114/166): parquet-libs-9.0.0-10.el9.x86_64.rpm 7.6 MB/s | 838 kB 00:00 2024-02-21T23:00:13.194 INFO:teuthology.orchestra.run.smithi081.stdout:(57/166): automake-1.16.2-8.el9.noarch.rpm 1.9 MB/s | 689 kB 00:00 2024-02-21T23:00:13.211 INFO:teuthology.orchestra.run.smithi081.stdout:(58/166): flexiblas-openblas-openmp-3.0.4-8.el9 312 kB/s | 18 kB 00:00 2024-02-21T23:00:13.247 INFO:teuthology.orchestra.run.smithi028.stdout:(115/166): python3-bcrypt-3.2.2-1.el9.x86_64.rp 585 kB/s | 43 kB 00:00 2024-02-21T23:00:13.253 INFO:teuthology.orchestra.run.smithi081.stdout:(59/166): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 160 kB 00:00 2024-02-21T23:00:13.278 INFO:teuthology.orchestra.run.smithi081.stdout:(60/166): librabbitmq-0.11.0-7.el9.x86_64.rpm 683 kB/s | 45 kB 00:00 2024-02-21T23:00:13.281 INFO:teuthology.orchestra.run.smithi028.stdout:(116/166): python3-asyncssh-2.13.2-4.el9.noarch 3.4 MB/s | 548 kB 00:00 2024-02-21T23:00:13.323 INFO:teuthology.orchestra.run.smithi028.stdout:(117/166): python3-cachetools-4.2.4-1.el9.noarc 433 kB/s | 32 kB 00:00 2024-02-21T23:00:13.356 INFO:teuthology.orchestra.run.smithi028.stdout:(118/166): python3-certifi-2021.10.8-2.el9.noar 201 kB/s | 15 kB 00:00 2024-02-21T23:00:13.398 INFO:teuthology.orchestra.run.smithi028.stdout:(119/166): python3-cheroot-8.6.0-4.el9.noarch.r 2.3 MB/s | 172 kB 00:00 2024-02-21T23:00:13.428 INFO:teuthology.orchestra.run.smithi081.stdout:(61/166): libtool-2.4.6-45.el9.x86_64.rpm 3.8 MB/s | 582 kB 00:00 2024-02-21T23:00:13.449 INFO:teuthology.orchestra.run.smithi028.stdout:(120/166): python3-cherrypy-18.6.1-2.el9.noarch 3.8 MB/s | 358 kB 00:00 2024-02-21T23:00:13.471 INFO:teuthology.orchestra.run.smithi081.stdout:(62/166): librdkafka-1.6.1-102.el9.x86_64.rpm 3.0 MB/s | 662 kB 00:00 2024-02-21T23:00:13.482 INFO:teuthology.orchestra.run.smithi028.stdout:(121/166): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.6 MB/s | 303 kB 00:00 2024-02-21T23:00:13.496 INFO:teuthology.orchestra.run.smithi081.stdout:(63/166): libuuid-devel-2.37.4-15.el9.x86_64.rp 384 kB/s | 26 kB 00:00 2024-02-21T23:00:13.524 INFO:teuthology.orchestra.run.smithi028.stdout:(122/166): python3-colorama-0.4.4-12.el9.noarch 475 kB/s | 35 kB 00:00 2024-02-21T23:00:13.529 INFO:teuthology.orchestra.run.smithi081.stdout:(64/166): libzstd-devel-1.5.1-2.el9.x86_64.rpm 822 kB/s | 47 kB 00:00 2024-02-21T23:00:13.558 INFO:teuthology.orchestra.run.smithi028.stdout:(123/166): python3-google-auth-2.27.0-1.el9.noa 2.6 MB/s | 203 kB 00:00 2024-02-21T23:00:13.588 INFO:teuthology.orchestra.run.smithi081.stdout:(65/166): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.1 MB/s | 292 kB 00:00 2024-02-21T23:00:13.613 INFO:teuthology.orchestra.run.smithi081.stdout:(66/166): lua-5.4.4-4.el9.x86_64.rpm 2.2 MB/s | 188 kB 00:00 2024-02-21T23:00:13.641 INFO:teuthology.orchestra.run.smithi028.stdout:(124/166): python3-grpcio-tools-1.46.7-10.el9.x 1.7 MB/s | 144 kB 00:00 2024-02-21T23:00:13.647 INFO:teuthology.orchestra.run.smithi081.stdout:(67/166): openblas-0.3.21-2.el9.x86_64.rpm 611 kB/s | 35 kB 00:00 2024-02-21T23:00:13.717 INFO:teuthology.orchestra.run.smithi028.stdout:(125/166): python3-grpcio-1.46.7-10.el9.x86_64. 11 MB/s | 2.0 MB 00:00 2024-02-21T23:00:13.731 INFO:teuthology.orchestra.run.smithi081.stdout:(68/166): flexiblas-netlib-3.0.4-8.el9.x86_64.r 5.0 MB/s | 3.0 MB 00:00 2024-02-21T23:00:13.750 INFO:teuthology.orchestra.run.smithi028.stdout:(126/166): python3-jaraco-8.2.1-3.el9.noarch.rp 99 kB/s | 11 kB 00:00 2024-02-21T23:00:13.801 INFO:teuthology.orchestra.run.smithi028.stdout:(127/166): protobuf-compiler-3.14.0-13.el9.x86_ 380 kB/s | 863 kB 00:02 2024-02-21T23:00:13.824 INFO:teuthology.orchestra.run.smithi028.stdout:(128/166): python3-jaraco-classes-3.2.1-5.el9.n 166 kB/s | 18 kB 00:00 2024-02-21T23:00:13.849 INFO:teuthology.orchestra.run.smithi028.stdout:(129/166): python3-jaraco-collections-3.0.0-8.e 236 kB/s | 23 kB 00:00 2024-02-21T23:00:13.950 INFO:teuthology.orchestra.run.smithi028.stdout:(130/166): python3-jaraco-text-3.2.0-6.el9.noar 199 kB/s | 20 kB 00:00 2024-02-21T23:00:14.000 INFO:teuthology.orchestra.run.smithi028.stdout:(131/166): python3-kubernetes-26.1.0-2.el9.noar 6.8 MB/s | 1.0 MB 00:00 2024-02-21T23:00:14.006 INFO:teuthology.orchestra.run.smithi081.stdout:(69/166): postgresql-13.11-1.el9.x86_64.rpm 5.8 MB/s | 1.6 MB 00:00 2024-02-21T23:00:14.033 INFO:teuthology.orchestra.run.smithi028.stdout:(132/166): python3-logutils-0.3.5-21.el9.noarch 556 kB/s | 46 kB 00:00 2024-02-21T23:00:14.067 INFO:teuthology.orchestra.run.smithi028.stdout:(133/166): python3-more-itertools-8.12.0-2.el9. 1.2 MB/s | 79 kB 00:00 2024-02-21T23:00:14.109 INFO:teuthology.orchestra.run.smithi028.stdout:(134/166): python3-natsort-7.1.1-5.el9.noarch.r 770 kB/s | 58 kB 00:00 2024-02-21T23:00:14.134 INFO:teuthology.orchestra.run.smithi028.stdout:(135/166): python3-jaraco-functools-3.5.0-2.el9 59 kB/s | 19 kB 00:00 2024-02-21T23:00:14.168 INFO:teuthology.orchestra.run.smithi028.stdout:(136/166): python3-pecan-1.4.2-3.el9.noarch.rpm 2.7 MB/s | 272 kB 00:00 2024-02-21T23:00:14.173 INFO:teuthology.orchestra.run.smithi081.stdout:(70/166): postgresql-contrib-13.11-1.el9.x86_64 5.3 MB/s | 905 kB 00:00 2024-02-21T23:00:14.193 INFO:teuthology.orchestra.run.smithi028.stdout:(137/166): python3-portend-3.1.0-2.el9.noarch.r 197 kB/s | 16 kB 00:00 2024-02-21T23:00:14.240 INFO:teuthology.orchestra.run.smithi081.stdout:(71/166): postgresql-private-libs-13.11-1.el9.x 2.0 MB/s | 135 kB 00:00 2024-02-21T23:00:14.244 INFO:teuthology.orchestra.run.smithi028.stdout:(138/166): python3-pyperclip-1.8.2-2.el9.noarch 376 kB/s | 28 kB 00:00 2024-02-21T23:00:14.269 INFO:teuthology.orchestra.run.smithi028.stdout:(139/166): python3-repoze-lru-0.7-16.el9.noarch 412 kB/s | 31 kB 00:00 2024-02-21T23:00:14.302 INFO:teuthology.orchestra.run.smithi028.stdout:(140/166): python3-pyOpenSSL-21.0.0-1.el9.noarc 539 kB/s | 90 kB 00:00 2024-02-21T23:00:14.336 INFO:teuthology.orchestra.run.smithi028.stdout:(141/166): python3-routes-2.5.1-5.el9.noarch.rp 2.0 MB/s | 188 kB 00:00 2024-02-21T23:00:14.357 INFO:teuthology.orchestra.run.smithi081.stdout:(72/166): openssl-devel-3.0.7-25.el9.x86_64.rpm 5.8 MB/s | 4.1 MB 00:00 2024-02-21T23:00:14.361 INFO:teuthology.orchestra.run.smithi028.stdout:(142/166): python3-rsa-4.9-2.el9.noarch.rpm 647 kB/s | 59 kB 00:00 2024-02-21T23:00:14.395 INFO:teuthology.orchestra.run.smithi028.stdout:(143/166): python3-tempora-5.0.0-2.el9.noarch.r 392 kB/s | 36 kB 00:00 2024-02-21T23:00:14.420 INFO:teuthology.orchestra.run.smithi028.stdout:(144/166): python3-typing-extensions-4.4.0-2.el 613 kB/s | 51 kB 00:00 2024-02-21T23:00:14.454 INFO:teuthology.orchestra.run.smithi028.stdout:(145/166): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-02-21T23:00:14.504 INFO:teuthology.orchestra.run.smithi028.stdout:(146/166): python3-werkzeug-2.0.3-3.el9.1.noarc 5.0 MB/s | 427 kB 00:00 2024-02-21T23:00:14.515 INFO:teuthology.orchestra.run.smithi081.stdout:(73/166): protobuf-3.14.0-13.el9.x86_64.rpm 6.4 MB/s | 1.0 MB 00:00 2024-02-21T23:00:14.529 INFO:teuthology.orchestra.run.smithi028.stdout:(147/166): python3-websocket-client-1.2.3-2.el9 672 kB/s | 90 kB 00:00 2024-02-21T23:00:14.546 INFO:teuthology.orchestra.run.smithi028.stdout:(148/166): python3-zc-lockfile-2.0-10.el9.noarc 219 kB/s | 20 kB 00:00 2024-02-21T23:00:14.566 INFO:teuthology.orchestra.run.smithi081.stdout:(74/166): python3-attrs-20.3.0-7.el9.noarch.rpm 1.7 MB/s | 87 kB 00:00 2024-02-21T23:00:14.580 INFO:teuthology.orchestra.run.smithi028.stdout:(149/166): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-02-21T23:00:14.655 INFO:teuthology.orchestra.run.smithi028.stdout:(150/166): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-02-21T23:00:14.674 INFO:teuthology.orchestra.run.smithi081.stdout:(75/166): openblas-openmp-0.3.21-2.el9.x86_64.r 4.5 MB/s | 4.7 MB 00:01 2024-02-21T23:00:14.783 INFO:teuthology.orchestra.run.smithi081.stdout:(76/166): python3-devel-3.9.17-2.el9.x86_64.rpm 2.2 MB/s | 245 kB 00:00 2024-02-21T23:00:14.822 INFO:teuthology.orchestra.run.smithi028.stdout:(151/166): NetworkManager-1.45.91-1.el9.x86_64. 14 MB/s | 2.3 MB 00:00 2024-02-21T23:00:14.867 INFO:teuthology.orchestra.run.smithi081.stdout:(77/166): python3-jinja2-2.11.3-5.el9.noarch.rp 2.9 MB/s | 248 kB 00:00 2024-02-21T23:00:14.925 INFO:teuthology.orchestra.run.smithi081.stdout:(78/166): python3-jsonpatch-1.21-16.el9.noarch. 443 kB/s | 26 kB 00:00 2024-02-21T23:00:14.939 INFO:teuthology.orchestra.run.smithi028.stdout:(152/166): librbd1-19.0.0-1643.g3bfc837d.el9.x8 8.8 MB/s | 3.1 MB 00:00 2024-02-21T23:00:14.984 INFO:teuthology.orchestra.run.smithi081.stdout:(79/166): python3-jsonpointer-2.0-4.el9.noarch. 333 kB/s | 19 kB 00:00 2024-02-21T23:00:15.031 INFO:teuthology.orchestra.run.smithi028.stdout:(153/166): NetworkManager-libnm-1.45.91-1.el9.x 8.7 MB/s | 1.8 MB 00:00 2024-02-21T23:00:15.043 INFO:teuthology.orchestra.run.smithi081.stdout:(80/166): python3-mako-1.1.4-6.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-02-21T23:00:15.056 INFO:teuthology.orchestra.run.smithi028.stdout:(154/166): NetworkManager-team-1.45.91-1.el9.x8 366 kB/s | 42 kB 00:00 2024-02-21T23:00:15.115 INFO:teuthology.orchestra.run.smithi028.stdout:(155/166): acl-2.3.1-4.el9.x86_64.rpm 1.2 MB/s | 71 kB 00:00 2024-02-21T23:00:15.148 INFO:teuthology.orchestra.run.smithi028.stdout:(156/166): NetworkManager-tui-1.45.91-1.el9.x86 2.1 MB/s | 247 kB 00:00 2024-02-21T23:00:15.151 INFO:teuthology.orchestra.run.smithi081.stdout:(81/166): postgresql-server-13.11-1.el9.x86_64. 6.3 MB/s | 5.7 MB 00:00 2024-02-21T23:00:15.177 INFO:teuthology.orchestra.run.smithi081.stdout:(82/166): python3-markupsafe-1.1.1-12.el9.x86_6 262 kB/s | 35 kB 00:00 2024-02-21T23:00:15.190 INFO:teuthology.orchestra.run.smithi028.stdout:(157/166): elfutils-debuginfod-client-0.190-2.e 492 kB/s | 36 kB 00:00 2024-02-21T23:00:15.240 INFO:teuthology.orchestra.run.smithi028.stdout:(158/166): elfutils-libelf-0.190-2.el9.x86_64.r 2.1 MB/s | 192 kB 00:00 2024-02-21T23:00:15.291 INFO:teuthology.orchestra.run.smithi028.stdout:(159/166): elfutils-libs-0.190-2.el9.x86_64.rpm 2.5 MB/s | 256 kB 00:00 2024-02-21T23:00:15.293 INFO:teuthology.orchestra.run.smithi081.stdout:(83/166): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.1 MB/s | 250 kB 00:00 2024-02-21T23:00:15.316 INFO:teuthology.orchestra.run.smithi028.stdout:(160/166): libacl-2.3.1-4.el9.x86_64.rpm 310 kB/s | 23 kB 00:00 2024-02-21T23:00:15.386 INFO:teuthology.orchestra.run.smithi081.stdout:(84/166): python3-oauthlib-3.1.1-5.el9.noarch.r 2.4 MB/s | 222 kB 00:00 2024-02-21T23:00:15.425 INFO:teuthology.orchestra.run.smithi028.stdout:(161/166): openssl-3.0.7-25.el9.x86_64.rpm 9.1 MB/s | 1.2 MB 00:00 2024-02-21T23:00:15.494 INFO:teuthology.orchestra.run.smithi081.stdout:(85/166): python3-babel-2.9.1-2.el9.noarch.rpm 6.4 MB/s | 6.0 MB 00:00 2024-02-21T23:00:15.500 INFO:teuthology.orchestra.run.smithi028.stdout:(162/166): openssl-libs-3.0.7-25.el9.x86_64.rpm 12 MB/s | 2.2 MB 00:00 2024-02-21T23:00:15.519 INFO:teuthology.orchestra.run.smithi081.stdout:(86/166): python3-packaging-20.9-5.el9.noarch.r 581 kB/s | 77 kB 00:00 2024-02-21T23:00:15.553 INFO:teuthology.orchestra.run.smithi081.stdout:(87/166): python3-prettytable-0.7.2-27.el9.noar 732 kB/s | 42 kB 00:00 2024-02-21T23:00:15.559 INFO:teuthology.orchestra.run.smithi028.stdout:(163/166): xfsdump-3.1.12-4.el9.x86_64.rpm 2.4 MB/s | 323 kB 00:00 2024-02-21T23:00:15.612 INFO:teuthology.orchestra.run.smithi081.stdout:(88/166): python3-protobuf-3.14.0-13.el9.noarch 2.9 MB/s | 269 kB 00:00 2024-02-21T23:00:15.625 INFO:teuthology.orchestra.run.smithi028.stdout:(164/166): xfsprogs-6.3.0-1.el9.x86_64.rpm 8.6 MB/s | 1.1 MB 00:00 2024-02-21T23:00:15.645 INFO:teuthology.orchestra.run.smithi081.stdout:(89/166): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 159 kB 00:00 2024-02-21T23:00:15.704 INFO:teuthology.orchestra.run.smithi081.stdout:(90/166): python3-pyasn1-modules-0.4.8-6.el9.no 3.0 MB/s | 279 kB 00:00 2024-02-21T23:00:15.729 INFO:teuthology.orchestra.run.smithi081.stdout:(91/166): python3-pytz-2021.1-5.el9.noarch.rpm 616 kB/s | 51 kB 00:00 2024-02-21T23:00:15.734 INFO:teuthology.orchestra.run.smithi028.stdout:(165/166): libacl-devel-2.3.1-4.el9.x86_64.rpm 427 kB/s | 74 kB 00:00 2024-02-21T23:00:15.754 INFO:teuthology.orchestra.run.smithi081.stdout:(92/166): python3-requests-oauthlib-1.3.0-12.el 1.1 MB/s | 54 kB 00:00 2024-02-21T23:00:15.871 INFO:teuthology.orchestra.run.smithi081.stdout:(93/166): python3-numpy-1.20.1-5.el9.x86_64.rpm 7.1 MB/s | 5.1 MB 00:00 2024-02-21T23:00:15.897 INFO:teuthology.orchestra.run.smithi081.stdout:(94/166): python3-toml-0.10.2-6.el9.noarch.rpm 294 kB/s | 42 kB 00:00 2024-02-21T23:00:15.930 INFO:teuthology.orchestra.run.smithi081.stdout:(95/166): python3-wcwidth-0.2.5-8.el9.noarch.rp 766 kB/s | 44 kB 00:00 2024-02-21T23:00:15.964 INFO:teuthology.orchestra.run.smithi081.stdout:(96/166): qatlib-23.11.0-1.el9.x86_64.rpm 3.2 MB/s | 216 kB 00:00 2024-02-21T23:00:15.989 INFO:teuthology.orchestra.run.smithi081.stdout:(97/166): qatlib-service-23.11.0-1.el9.x86_64.r 604 kB/s | 35 kB 00:00 2024-02-21T23:00:16.001 INFO:teuthology.orchestra.run.smithi028.stdout:(166/166): librados2-19.0.0-1643.g3bfc837d.el9. 2.3 MB/s | 3.4 MB 00:01 2024-02-21T23:00:16.014 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-02-21T23:00:16.015 INFO:teuthology.orchestra.run.smithi028.stdout:Total 14 MB/s | 222 MB 00:16 2024-02-21T23:00:16.023 INFO:teuthology.orchestra.run.smithi081.stdout:(98/166): qatzip-libs-1.1.2-2.el9.x86_64.rpm 771 kB/s | 44 kB 00:00 2024-02-21T23:00:16.057 INFO:teuthology.orchestra.run.smithi081.stdout:(99/166): socat-1.7.4.1-5.el9.x86_64.rpm 4.5 MB/s | 305 kB 00:00 2024-02-21T23:00:16.107 INFO:teuthology.orchestra.run.smithi081.stdout:(100/166): uuid-1.6.2-55.el9.x86_64.rpm 699 kB/s | 58 kB 00:00 2024-02-21T23:00:16.132 INFO:teuthology.orchestra.run.smithi081.stdout:(101/166): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.2 MB/s | 94 kB 00:00 2024-02-21T23:00:16.166 INFO:teuthology.orchestra.run.smithi081.stdout:(102/166): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-02-21T23:00:16.191 INFO:teuthology.orchestra.run.smithi081.stdout:(103/166): zlib-devel-1.2.11-40.el9.x86_64.rpm 779 kB/s | 45 kB 00:00 2024-02-21T23:00:16.241 INFO:teuthology.orchestra.run.smithi081.stdout:(104/166): lua-devel-5.4.4-4.el9.x86_64.rpm 299 kB/s | 22 kB 00:00 2024-02-21T23:00:16.334 INFO:teuthology.orchestra.run.smithi081.stdout:(105/166): protobuf-compiler-3.14.0-13.el9.x86_ 6.0 MB/s | 863 kB 00:00 2024-02-21T23:00:16.459 INFO:teuthology.orchestra.run.smithi081.stdout:(106/166): abseil-cpp-20211102.0-3.el9.x86_64.r 2.5 MB/s | 548 kB 00:00 2024-02-21T23:00:16.501 INFO:teuthology.orchestra.run.smithi081.stdout:(107/166): gperftools-libs-2.9.1-2.el9.x86_64.r 1.8 MB/s | 309 kB 00:00 2024-02-21T23:00:16.526 INFO:teuthology.orchestra.run.smithi081.stdout:(108/166): grpc-data-1.46.7-10.el9.noarch.rpm 294 kB/s | 19 kB 00:00 2024-02-21T23:00:16.560 INFO:teuthology.orchestra.run.smithi081.stdout:(109/166): libarrow-doc-9.0.0-10.el9.noarch.rpm 801 kB/s | 26 kB 00:00 2024-02-21T23:00:16.602 INFO:teuthology.orchestra.run.smithi081.stdout:(110/166): liboath-2.6.7-2.el9.x86_64.rpm 1.2 MB/s | 49 kB 00:00 2024-02-21T23:00:16.644 INFO:teuthology.orchestra.run.smithi081.stdout:(111/166): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-02-21T23:00:16.686 INFO:teuthology.orchestra.run.smithi081.stdout:(112/166): luarocks-3.9.2-1.el9.noarch.rpm 3.6 MB/s | 151 kB 00:00 2024-02-21T23:00:16.753 INFO:teuthology.orchestra.run.smithi081.stdout:(113/166): parquet-libs-9.0.0-10.el9.x86_64.rpm 12 MB/s | 838 kB 00:00 2024-02-21T23:00:16.811 INFO:teuthology.orchestra.run.smithi081.stdout:(114/166): python3-asyncssh-2.13.2-4.el9.noarch 9.4 MB/s | 548 kB 00:00 2024-02-21T23:00:17.004 INFO:teuthology.orchestra.run.smithi081.stdout:(115/166): libarrow-9.0.0-10.el9.x86_64.rpm 8.8 MB/s | 4.4 MB 00:00 2024-02-21T23:00:17.029 INFO:teuthology.orchestra.run.smithi081.stdout:(116/166): python3-bcrypt-3.2.2-1.el9.x86_64.rp 200 kB/s | 43 kB 00:00 2024-02-21T23:00:17.055 INFO:teuthology.orchestra.run.smithi081.stdout:(117/166): python3-cachetools-4.2.4-1.el9.noarc 652 kB/s | 32 kB 00:00 2024-02-21T23:00:17.079 INFO:teuthology.orchestra.run.smithi081.stdout:(118/166): python3-certifi-2021.10.8-2.el9.noar 302 kB/s | 15 kB 00:00 2024-02-21T23:00:17.113 INFO:teuthology.orchestra.run.smithi081.stdout:(119/166): python3-cheroot-8.6.0-4.el9.noarch.r 2.9 MB/s | 172 kB 00:00 2024-02-21T23:00:17.146 INFO:teuthology.orchestra.run.smithi081.stdout:(120/166): python3-cherrypy-18.6.1-2.el9.noarch 5.3 MB/s | 358 kB 00:00 2024-02-21T23:00:17.197 INFO:teuthology.orchestra.run.smithi081.stdout:(121/166): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.6 MB/s | 303 kB 00:00 2024-02-21T23:00:17.222 INFO:teuthology.orchestra.run.smithi081.stdout:(122/166): python3-colorama-0.4.4-12.el9.noarch 475 kB/s | 35 kB 00:00 2024-02-21T23:00:17.247 INFO:teuthology.orchestra.run.smithi081.stdout:(123/166): python3-google-auth-2.27.0-1.el9.noa 4.0 MB/s | 203 kB 00:00 2024-02-21T23:00:17.481 INFO:teuthology.orchestra.run.smithi081.stdout:(124/166): python3-scipy-1.6.2-8.el9.x86_64.rpm 9.3 MB/s | 16 MB 00:01 2024-02-21T23:00:17.515 INFO:teuthology.orchestra.run.smithi081.stdout:(125/166): python3-grpcio-tools-1.46.7-10.el9.x 542 kB/s | 144 kB 00:00 2024-02-21T23:00:17.582 INFO:teuthology.orchestra.run.smithi081.stdout:(126/166): python3-grpcio-1.46.7-10.el9.x86_64. 5.7 MB/s | 2.0 MB 00:00 2024-02-21T23:00:17.607 INFO:teuthology.orchestra.run.smithi081.stdout:(127/166): python3-jaraco-8.2.1-3.el9.noarch.rp 86 kB/s | 11 kB 00:00 2024-02-21T23:00:17.632 INFO:teuthology.orchestra.run.smithi081.stdout:(128/166): python3-jaraco-classes-3.2.1-5.el9.n 152 kB/s | 18 kB 00:00 2024-02-21T23:00:17.666 INFO:teuthology.orchestra.run.smithi081.stdout:(129/166): python3-jaraco-collections-3.0.0-8.e 725 kB/s | 23 kB 00:00 2024-02-21T23:00:17.691 INFO:teuthology.orchestra.run.smithi081.stdout:(130/166): python3-jaraco-functools-3.5.0-2.el9 341 kB/s | 19 kB 00:00 2024-02-21T23:00:17.708 INFO:teuthology.orchestra.run.smithi081.stdout:(131/166): python3-jaraco-text-3.2.0-6.el9.noar 266 kB/s | 20 kB 00:00 2024-02-21T23:00:17.733 INFO:teuthology.orchestra.run.smithi081.stdout:(132/166): python3-logutils-0.3.5-21.el9.noarch 1.1 MB/s | 46 kB 00:00 2024-02-21T23:00:17.759 INFO:teuthology.orchestra.run.smithi081.stdout:(133/166): python3-natsort-7.1.1-5.el9.noarch.r 2.3 MB/s | 58 kB 00:00 2024-02-21T23:00:17.784 INFO:teuthology.orchestra.run.smithi081.stdout:(134/166): python3-more-itertools-8.12.0-2.el9. 1.0 MB/s | 79 kB 00:00 2024-02-21T23:00:17.834 INFO:teuthology.orchestra.run.smithi081.stdout:(135/166): python3-kubernetes-26.1.0-2.el9.noar 7.2 MB/s | 1.0 MB 00:00 2024-02-21T23:00:17.860 INFO:teuthology.orchestra.run.smithi081.stdout:(136/166): python3-portend-3.1.0-2.el9.noarch.r 219 kB/s | 16 kB 00:00 2024-02-21T23:00:17.894 INFO:teuthology.orchestra.run.smithi081.stdout:(137/166): python3-pecan-1.4.2-3.el9.noarch.rpm 2.0 MB/s | 272 kB 00:00 2024-02-21T23:00:17.919 INFO:teuthology.orchestra.run.smithi081.stdout:(138/166): python3-pyperclip-1.8.2-2.el9.noarch 488 kB/s | 28 kB 00:00 2024-02-21T23:00:17.945 INFO:teuthology.orchestra.run.smithi081.stdout:(139/166): python3-pyOpenSSL-21.0.0-1.el9.noarc 823 kB/s | 90 kB 00:00 2024-02-21T23:00:17.970 INFO:teuthology.orchestra.run.smithi081.stdout:(140/166): python3-repoze-lru-0.7-16.el9.noarch 409 kB/s | 31 kB 00:00 2024-02-21T23:00:17.995 INFO:teuthology.orchestra.run.smithi081.stdout:(141/166): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-02-21T23:00:18.029 INFO:teuthology.orchestra.run.smithi081.stdout:(142/166): python3-tempora-5.0.0-2.el9.noarch.r 619 kB/s | 36 kB 00:00 2024-02-21T23:00:18.054 INFO:teuthology.orchestra.run.smithi081.stdout:(143/166): python3-routes-2.5.1-5.el9.noarch.rp 1.4 MB/s | 188 kB 00:00 2024-02-21T23:00:18.079 INFO:teuthology.orchestra.run.smithi081.stdout:(144/166): python3-typing-extensions-4.4.0-2.el 611 kB/s | 51 kB 00:00 2024-02-21T23:00:18.113 INFO:teuthology.orchestra.run.smithi081.stdout:(145/166): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-02-21T23:00:18.163 INFO:teuthology.orchestra.run.smithi081.stdout:(146/166): python3-websocket-client-1.2.3-2.el9 1.1 MB/s | 90 kB 00:00 2024-02-21T23:00:18.189 INFO:teuthology.orchestra.run.smithi081.stdout:(147/166): python3-zc-lockfile-2.0-10.el9.noarc 268 kB/s | 20 kB 00:00 2024-02-21T23:00:18.222 INFO:teuthology.orchestra.run.smithi081.stdout:(148/166): re2-20211101-3.el9.x86_64.rpm 3.2 MB/s | 192 kB 00:00 2024-02-21T23:00:18.256 INFO:teuthology.orchestra.run.smithi081.stdout:(149/166): python3-werkzeug-2.0.3-3.el9.1.noarc 2.4 MB/s | 427 kB 00:00 2024-02-21T23:00:18.323 INFO:teuthology.orchestra.run.smithi081.stdout:(150/166): thrift-0.15.0-2.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-02-21T23:00:18.665 INFO:teuthology.orchestra.run.smithi081.stdout:(151/166): librados2-19.0.0-1643.g3bfc837d.el9. 7.7 MB/s | 3.4 MB 00:00 2024-02-21T23:00:18.782 INFO:teuthology.orchestra.run.smithi081.stdout:(152/166): librbd1-19.0.0-1643.g3bfc837d.el9.x8 6.0 MB/s | 3.1 MB 00:00 2024-02-21T23:00:18.849 INFO:teuthology.orchestra.run.smithi081.stdout:(153/166): NetworkManager-1.45.91-1.el9.x86_64. 4.3 MB/s | 2.3 MB 00:00 2024-02-21T23:00:18.874 INFO:teuthology.orchestra.run.smithi081.stdout:(154/166): NetworkManager-team-1.45.91-1.el9.x8 466 kB/s | 42 kB 00:00 2024-02-21T23:00:18.916 INFO:teuthology.orchestra.run.smithi081.stdout:(155/166): NetworkManager-tui-1.45.91-1.el9.x86 3.7 MB/s | 247 kB 00:00 2024-02-21T23:00:18.941 INFO:teuthology.orchestra.run.smithi081.stdout:(156/166): acl-2.3.1-4.el9.x86_64.rpm 1.1 MB/s | 71 kB 00:00 2024-02-21T23:00:18.975 INFO:teuthology.orchestra.run.smithi081.stdout:(157/166): elfutils-debuginfod-client-0.190-2.e 631 kB/s | 36 kB 00:00 2024-02-21T23:00:19.042 INFO:teuthology.orchestra.run.smithi081.stdout:(158/166): NetworkManager-libnm-1.45.91-1.el9.x 4.8 MB/s | 1.8 MB 00:00 2024-02-21T23:00:19.067 INFO:teuthology.orchestra.run.smithi081.stdout:(159/166): elfutils-libelf-0.190-2.el9.x86_64.r 1.5 MB/s | 192 kB 00:00 2024-02-21T23:00:19.126 INFO:teuthology.orchestra.run.smithi081.stdout:(160/166): elfutils-libs-0.190-2.el9.x86_64.rpm 1.7 MB/s | 256 kB 00:00 2024-02-21T23:00:19.151 INFO:teuthology.orchestra.run.smithi081.stdout:(161/166): libacl-2.3.1-4.el9.x86_64.rpm 213 kB/s | 23 kB 00:00 2024-02-21T23:00:19.210 INFO:teuthology.orchestra.run.smithi081.stdout:(162/166): xfsdump-3.1.12-4.el9.x86_64.rpm 5.5 MB/s | 323 kB 00:00 2024-02-21T23:00:19.310 INFO:teuthology.orchestra.run.smithi081.stdout:(163/166): openssl-3.0.7-25.el9.x86_64.rpm 5.0 MB/s | 1.2 MB 00:00 2024-02-21T23:00:19.385 INFO:teuthology.orchestra.run.smithi081.stdout:(164/166): xfsprogs-6.3.0-1.el9.x86_64.rpm 6.2 MB/s | 1.1 MB 00:00 2024-02-21T23:00:19.402 INFO:teuthology.orchestra.run.smithi081.stdout:(165/166): libacl-devel-2.3.1-4.el9.x86_64.rpm 817 kB/s | 74 kB 00:00 2024-02-21T23:00:19.527 INFO:teuthology.orchestra.run.smithi081.stdout:(166/166): openssl-libs-3.0.7-25.el9.x86_64.rpm 5.4 MB/s | 2.2 MB 00:00 2024-02-21T23:00:19.533 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-02-21T23:00:19.534 INFO:teuthology.orchestra.run.smithi081.stdout:Total 15 MB/s | 222 MB 00:15 2024-02-21T23:00:19.549 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-02-21T23:00:19.910 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-02-21T23:00:19.910 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-02-21T23:00:22.597 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-02-21T23:00:22.597 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-02-21T23:00:22.999 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-02-21T23:00:23.363 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-02-21T23:00:23.363 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-02-21T23:00:25.981 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-02-21T23:00:25.982 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-02-21T23:00:29.683 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-02-21T23:00:29.808 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : openssl-libs-1:3.0.7-25.el9.x86_64 1/182 2024-02-21T23:00:29.895 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/182 2024-02-21T23:00:30.004 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/182 2024-02-21T23:00:30.071 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/182 2024-02-21T23:00:30.125 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/182 2024-02-21T23:00:30.227 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/182 2024-02-21T23:00:30.662 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/182 2024-02-21T23:00:30.889 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : librados2-2:19.0.0-1643.g3bfc837d.el9.x86_64 8/182 2024-02-21T23:00:31.046 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librados2-2:19.0.0-1643.g3bfc837d.el9.x86_64 8/182 2024-02-21T23:00:31.082 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libcephfs2-2:19.0.0-1643.g3bfc837d.el9.x86_64 9/182 2024-02-21T23:00:31.241 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libcephfs2-2:19.0.0-1643.g3bfc837d.el9.x86_64 9/182 2024-02-21T23:00:31.277 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rados-2:19.0.0-1643.g3bfc837d.el9.x86_64 10/182 2024-02-21T23:00:31.433 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 11/182 2024-02-21T23:00:31.531 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 12/182 2024-02-21T23:00:31.832 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 13/182 2024-02-21T23:00:31.889 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : postgresql-13.11-1.el9.x86_64 14/182 2024-02-21T23:00:31.905 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libcephsqlite-2:19.0.0-1643.g3bfc837d.el9.x86_64 15/182 2024-02-21T23:00:32.065 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1643.g3bfc837d.el9.x86_64 15/182 2024-02-21T23:00:32.089 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libradosstriper1-2:19.0.0-1643.g3bfc837d.el9.x86 16/182 2024-02-21T23:00:32.184 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1643.g3bfc837d.el9.x86 16/182 2024-02-21T23:00:32.199 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 17/182 2024-02-21T23:00:32.256 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 17/182 2024-02-21T23:00:32.579 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 18/182 2024-02-21T23:00:32.618 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 18/182 2024-02-21T23:00:32.957 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 19/182 2024-02-21T23:00:33.097 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : NetworkManager-libnm-1:1.45.91-1.el9.x86_64 20/182 2024-02-21T23:00:33.171 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-02-21T23:00:33.178 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: NetworkManager-libnm-1:1.45.91-1.el9.x86_64 20/182 2024-02-21T23:00:33.301 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : openssl-libs-1:3.0.7-25.el9.x86_64 1/182 2024-02-21T23:00:33.380 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: NetworkManager-1:1.45.91-1.el9.x86_64 21/182 2024-02-21T23:00:33.391 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/182 2024-02-21T23:00:33.466 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : NetworkManager-1:1.45.91-1.el9.x86_64 21/182 2024-02-21T23:00:33.501 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/182 2024-02-21T23:00:33.555 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/182 2024-02-21T23:00:33.586 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/182 2024-02-21T23:00:33.681 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/182 2024-02-21T23:00:33.799 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: NetworkManager-1:1.45.91-1.el9.x86_64 21/182 2024-02-21T23:00:33.996 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : re2-1:20211101-3.el9.x86_64 22/182 2024-02-21T23:00:34.104 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/182 2024-02-21T23:00:34.116 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 23/182 2024-02-21T23:00:34.145 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : librados2-2:19.0.0-1643.g3bfc837d.el9.x86_64 8/182 2024-02-21T23:00:34.164 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 24/182 2024-02-21T23:00:34.213 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : liboath-2.6.7-2.el9.x86_64 25/182 2024-02-21T23:00:34.303 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: librados2-2:19.0.0-1643.g3bfc837d.el9.x86_64 8/182 2024-02-21T23:00:34.322 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 26/182 2024-02-21T23:00:34.338 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libcephfs2-2:19.0.0-1643.g3bfc837d.el9.x86_64 9/182 2024-02-21T23:00:34.368 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 27/182 2024-02-21T23:00:34.433 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/182 2024-02-21T23:00:34.478 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: libcephfs2-2:19.0.0-1643.g3bfc837d.el9.x86_64 9/182 2024-02-21T23:00:34.498 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 29/182 2024-02-21T23:00:34.524 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-rados-2:19.0.0-1643.g3bfc837d.el9.x86_64 10/182 2024-02-21T23:00:34.530 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : lua-5.4.4-4.el9.x86_64 30/182 2024-02-21T23:00:34.642 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 31/182 2024-02-21T23:00:34.674 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 11/182 2024-02-21T23:00:34.709 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 32/182 2024-02-21T23:00:34.747 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 33/182 2024-02-21T23:00:34.784 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 12/182 2024-02-21T23:00:34.808 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 34/182 2024-02-21T23:00:34.918 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 35/182 2024-02-21T23:00:34.959 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-ceph-common-2:19.0.0-1643.g3bfc837d.el9. 36/182 2024-02-21T23:00:35.012 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-ceph-argparse-2:19.0.0-1643.g3bfc837d.el 37/182 2024-02-21T23:00:35.049 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cephfs-2:19.0.0-1643.g3bfc837d.el9.x86_6 38/182 2024-02-21T23:00:35.089 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 13/182 2024-02-21T23:00:35.145 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : postgresql-13.11-1.el9.x86_64 14/182 2024-02-21T23:00:35.172 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libcephsqlite-2:19.0.0-1643.g3bfc837d.el9.x86_64 15/182 2024-02-21T23:00:35.173 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 39/182 2024-02-21T23:00:35.237 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : luarocks-3.9.2-1.el9.noarch 40/182 2024-02-21T23:00:35.323 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1643.g3bfc837d.el9.x86_64 15/182 2024-02-21T23:00:35.338 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 41/182 2024-02-21T23:00:35.357 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libradosstriper1-2:19.0.0-1643.g3bfc837d.el9.x86 16/182 2024-02-21T23:00:35.459 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1643.g3bfc837d.el9.x86 16/182 2024-02-21T23:00:35.483 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 17/182 2024-02-21T23:00:35.521 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 42/182 2024-02-21T23:00:35.546 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 17/182 2024-02-21T23:00:35.598 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 43/182 2024-02-21T23:00:35.816 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 18/182 2024-02-21T23:00:35.860 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 18/182 2024-02-21T23:00:36.167 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 19/182 2024-02-21T23:00:36.328 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rsa-4.9-2.el9.noarch 44/182 2024-02-21T23:00:36.340 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : NetworkManager-libnm-1:1.45.91-1.el9.x86_64 20/182 2024-02-21T23:00:36.441 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 45/182 2024-02-21T23:00:36.445 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: NetworkManager-libnm-1:1.45.91-1.el9.x86_64 20/182 2024-02-21T23:00:36.486 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 46/182 2024-02-21T23:00:36.529 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 47/182 2024-02-21T23:00:36.596 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 48/182 2024-02-21T23:00:36.642 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: NetworkManager-1:1.45.91-1.el9.x86_64 21/182 2024-02-21T23:00:36.665 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librados-devel-2:19.0.0-1643.g3bfc837d.el9.x86_6 49/182 2024-02-21T23:00:36.718 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : NetworkManager-1:1.45.91-1.el9.x86_64 21/182 2024-02-21T23:00:36.743 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 50/182 2024-02-21T23:00:36.963 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 51/182 2024-02-21T23:00:37.019 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : openssl-1:3.0.7-25.el9.x86_64 52/182 2024-02-21T23:00:37.058 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: NetworkManager-1:1.45.91-1.el9.x86_64 21/182 2024-02-21T23:00:37.095 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 53/182 2024-02-21T23:00:37.200 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 54/182 2024-02-21T23:00:37.235 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 55/182 2024-02-21T23:00:37.236 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : re2-1:20211101-3.el9.x86_64 22/182 2024-02-21T23:00:37.266 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 56/182 2024-02-21T23:00:37.332 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 57/182 2024-02-21T23:00:37.359 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 23/182 2024-02-21T23:00:37.374 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 58/182 2024-02-21T23:00:37.407 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 24/182 2024-02-21T23:00:37.438 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 59/182 2024-02-21T23:00:37.456 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : liboath-2.6.7-2.el9.x86_64 25/182 2024-02-21T23:00:37.504 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 60/182 2024-02-21T23:00:37.559 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 26/182 2024-02-21T23:00:37.610 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 27/182 2024-02-21T23:00:37.672 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/182 2024-02-21T23:00:37.699 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 61/182 2024-02-21T23:00:37.715 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 29/182 2024-02-21T23:00:37.747 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : lua-5.4.4-4.el9.x86_64 30/182 2024-02-21T23:00:37.754 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 62/182 2024-02-21T23:00:37.793 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 63/182 2024-02-21T23:00:37.851 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 64/182 2024-02-21T23:00:37.860 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 31/182 2024-02-21T23:00:37.935 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 32/182 2024-02-21T23:00:37.974 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 33/182 2024-02-21T23:00:37.984 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 65/182 2024-02-21T23:00:38.024 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-google-auth-1:2.27.0-1.el9.noarch 66/182 2024-02-21T23:00:38.031 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 34/182 2024-02-21T23:00:38.072 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 67/182 2024-02-21T23:00:38.106 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 68/182 2024-02-21T23:00:38.140 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 69/182 2024-02-21T23:00:38.140 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 35/182 2024-02-21T23:00:38.187 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-ceph-common-2:19.0.0-1643.g3bfc837d.el9. 36/182 2024-02-21T23:00:38.237 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-ceph-argparse-2:19.0.0-1643.g3bfc837d.el 37/182 2024-02-21T23:00:38.268 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cephfs-2:19.0.0-1643.g3bfc837d.el9.x86_6 38/182 2024-02-21T23:00:38.313 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 70/182 2024-02-21T23:00:38.411 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 39/182 2024-02-21T23:00:38.495 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 71/182 2024-02-21T23:00:38.528 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : luarocks-3.9.2-1.el9.noarch 40/182 2024-02-21T23:00:38.537 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 72/182 2024-02-21T23:00:38.576 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 73/182 2024-02-21T23:00:38.623 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 74/182 2024-02-21T23:00:38.640 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 41/182 2024-02-21T23:00:38.684 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : uuid-1.6.2-55.el9.x86_64 75/182 2024-02-21T23:00:38.726 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 76/182 2024-02-21T23:00:38.766 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 77/182 2024-02-21T23:00:38.820 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 42/182 2024-02-21T23:00:38.833 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 78/182 2024-02-21T23:00:38.883 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 43/182 2024-02-21T23:00:38.888 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 79/182 2024-02-21T23:00:38.965 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 80/182 2024-02-21T23:00:39.011 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 81/182 2024-02-21T23:00:39.162 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 82/182 2024-02-21T23:00:39.214 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 83/182 2024-02-21T23:00:39.249 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 84/182 2024-02-21T23:00:39.311 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 85/182 2024-02-21T23:00:39.459 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-packaging-20.9-5.el9.noarch 86/182 2024-02-21T23:00:39.522 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 87/182 2024-02-21T23:00:39.576 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-rsa-4.9-2.el9.noarch 44/182 2024-02-21T23:00:39.690 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 45/182 2024-02-21T23:00:39.731 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 46/182 2024-02-21T23:00:39.772 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 47/182 2024-02-21T23:00:39.838 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 48/182 2024-02-21T23:00:39.885 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : librados-devel-2:19.0.0-1643.g3bfc837d.el9.x86_6 49/182 2024-02-21T23:00:39.956 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 50/182 2024-02-21T23:00:40.163 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 51/182 2024-02-21T23:00:40.234 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : openssl-1:3.0.7-25.el9.x86_64 52/182 2024-02-21T23:00:40.317 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 53/182 2024-02-21T23:00:40.407 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 54/182 2024-02-21T23:00:40.444 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 55/182 2024-02-21T23:00:40.483 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 56/182 2024-02-21T23:00:40.534 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 88/182 2024-02-21T23:00:40.551 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 57/182 2024-02-21T23:00:40.586 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 89/182 2024-02-21T23:00:40.591 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 58/182 2024-02-21T23:00:40.620 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 90/182 2024-02-21T23:00:40.653 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 59/182 2024-02-21T23:00:40.711 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 60/182 2024-02-21T23:00:40.749 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 91/182 2024-02-21T23:00:40.807 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 92/182 2024-02-21T23:00:40.900 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 61/182 2024-02-21T23:00:40.915 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 93/182 2024-02-21T23:00:40.947 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 62/182 2024-02-21T23:00:40.955 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 94/182 2024-02-21T23:00:40.985 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 63/182 2024-02-21T23:00:41.034 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 64/182 2024-02-21T23:00:41.163 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 65/182 2024-02-21T23:00:41.204 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : openblas-0.3.21-2.el9.x86_64 95/182 2024-02-21T23:00:41.218 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-google-auth-1:2.27.0-1.el9.noarch 66/182 2024-02-21T23:00:41.243 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 96/182 2024-02-21T23:00:41.266 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 67/182 2024-02-21T23:00:41.300 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 68/182 2024-02-21T23:00:41.334 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 69/182 2024-02-21T23:00:41.344 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 97/182 2024-02-21T23:00:41.505 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 70/182 2024-02-21T23:00:41.682 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 71/182 2024-02-21T23:00:41.732 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 72/182 2024-02-21T23:00:41.770 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 73/182 2024-02-21T23:00:41.815 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 74/182 2024-02-21T23:00:41.874 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : uuid-1.6.2-55.el9.x86_64 75/182 2024-02-21T23:00:41.925 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 76/182 2024-02-21T23:00:41.968 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 77/182 2024-02-21T23:00:42.014 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 78/182 2024-02-21T23:00:42.064 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 79/182 2024-02-21T23:00:42.140 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 80/182 2024-02-21T23:00:42.187 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 81/182 2024-02-21T23:00:42.191 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 98/182 2024-02-21T23:00:42.330 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 82/182 2024-02-21T23:00:42.331 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 99/182 2024-02-21T23:00:42.402 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 83/182 2024-02-21T23:00:42.444 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 84/182 2024-02-21T23:00:42.518 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 85/182 2024-02-21T23:00:42.659 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-packaging-20.9-5.el9.noarch 86/182 2024-02-21T23:00:42.725 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 87/182 2024-02-21T23:00:43.683 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 88/182 2024-02-21T23:00:43.732 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 89/182 2024-02-21T23:00:43.766 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 90/182 2024-02-21T23:00:43.888 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 91/182 2024-02-21T23:00:43.952 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 92/182 2024-02-21T23:00:43.969 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 100/182 2024-02-21T23:00:44.055 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 93/182 2024-02-21T23:00:44.061 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 101/182 2024-02-21T23:00:44.092 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 94/182 2024-02-21T23:00:44.098 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 102/182 2024-02-21T23:00:44.132 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libuuid-devel-2.37.4-15.el9.x86_64 103/182 2024-02-21T23:00:44.314 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : openblas-0.3.21-2.el9.x86_64 95/182 2024-02-21T23:00:44.348 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 96/182 2024-02-21T23:00:44.439 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 97/182 2024-02-21T23:00:44.510 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 104/182 2024-02-21T23:00:44.540 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : librbd1-2:19.0.0-1643.g3bfc837d.el9.x86_64 105/182 2024-02-21T23:00:44.645 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librbd1-2:19.0.0-1643.g3bfc837d.el9.x86_64 105/182 2024-02-21T23:00:44.680 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rbd-2:19.0.0-1643.g3bfc837d.el9.x86_64 106/182 2024-02-21T23:00:44.734 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 107/182 2024-02-21T23:00:45.227 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 98/182 2024-02-21T23:00:45.259 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 108/182 2024-02-21T23:00:45.293 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librgw2-2:19.0.0-1643.g3bfc837d.el9.x86_64 109/182 2024-02-21T23:00:45.361 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 99/182 2024-02-21T23:00:45.393 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librgw2-2:19.0.0-1643.g3bfc837d.el9.x86_64 109/182 2024-02-21T23:00:45.420 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rgw-2:19.0.0-1643.g3bfc837d.el9.x86_64 110/182 2024-02-21T23:00:46.899 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 100/182 2024-02-21T23:00:46.998 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 101/182 2024-02-21T23:00:47.034 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 102/182 2024-02-21T23:00:47.065 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libuuid-devel-2.37.4-15.el9.x86_64 103/182 2024-02-21T23:00:47.426 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 104/182 2024-02-21T23:00:47.460 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : librbd1-2:19.0.0-1643.g3bfc837d.el9.x86_64 105/182 2024-02-21T23:00:47.580 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: librbd1-2:19.0.0-1643.g3bfc837d.el9.x86_64 105/182 2024-02-21T23:00:47.616 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-rbd-2:19.0.0-1643.g3bfc837d.el9.x86_64 106/182 2024-02-21T23:00:47.670 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 107/182 2024-02-21T23:00:48.114 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-common-2:19.0.0-1643.g3bfc837d.el9.x86_64 111/182 2024-02-21T23:00:48.150 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-common-2:19.0.0-1643.g3bfc837d.el9.x86_64 111/182 2024-02-21T23:00:48.183 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 108/182 2024-02-21T23:00:48.222 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : librgw2-2:19.0.0-1643.g3bfc837d.el9.x86_64 109/182 2024-02-21T23:00:48.312 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: librgw2-2:19.0.0-1643.g3bfc837d.el9.x86_64 109/182 2024-02-21T23:00:48.340 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-rgw-2:19.0.0-1643.g3bfc837d.el9.x86_64 110/182 2024-02-21T23:00:48.859 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-common-2:19.0.0-1643.g3bfc837d.el9.x86_64 111/182 2024-02-21T23:00:48.903 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-base-2:19.0.0-1643.g3bfc837d.el9.x86_64 112/182 2024-02-21T23:00:49.073 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-base-2:19.0.0-1643.g3bfc837d.el9.x86_64 112/182 2024-02-21T23:00:49.074 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-02-21T23:00:49.074 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-02-21T23:00:49.074 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:00:49.120 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-selinux-2:19.0.0-1643.g3bfc837d.el9.x86_64 113/182 2024-02-21T23:00:50.892 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-common-2:19.0.0-1643.g3bfc837d.el9.x86_64 111/182 2024-02-21T23:00:50.928 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-common-2:19.0.0-1643.g3bfc837d.el9.x86_64 111/182 2024-02-21T23:00:51.572 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-common-2:19.0.0-1643.g3bfc837d.el9.x86_64 111/182 2024-02-21T23:00:51.615 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-base-2:19.0.0-1643.g3bfc837d.el9.x86_64 112/182 2024-02-21T23:00:51.769 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-base-2:19.0.0-1643.g3bfc837d.el9.x86_64 112/182 2024-02-21T23:00:51.769 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-02-21T23:00:51.769 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-02-21T23:00:51.769 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-02-21T23:00:51.816 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-selinux-2:19.0.0-1643.g3bfc837d.el9.x86_64 113/182 2024-02-21T23:01:02.294 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1643.g3bfc837d.el9.x86_64 113/182 2024-02-21T23:01:02.294 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /sys 2024-02-21T23:01:02.294 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /proc 2024-02-21T23:01:02.294 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /mnt 2024-02-21T23:01:02.294 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /var/tmp 2024-02-21T23:01:02.294 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /home 2024-02-21T23:01:02.294 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /root 2024-02-21T23:01:02.294 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /tmp 2024-02-21T23:01:02.294 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:01:02.522 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-volume-2:19.0.0-1643.g3bfc837d.el9.noarch 114/182 2024-02-21T23:01:02.575 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-volume-2:19.0.0-1643.g3bfc837d.el9.noarch 114/182 2024-02-21T23:01:02.576 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-21T23:01:02.576 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-02-21T23:01:02.576 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:01:04.153 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1643.g3bfc837d.el9.x86_64 113/182 2024-02-21T23:01:04.153 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /sys 2024-02-21T23:01:04.154 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /proc 2024-02-21T23:01:04.154 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /mnt 2024-02-21T23:01:04.154 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /var/tmp 2024-02-21T23:01:04.154 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /home 2024-02-21T23:01:04.154 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /root 2024-02-21T23:01:04.154 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /tmp 2024-02-21T23:01:04.154 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-02-21T23:01:04.267 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-osd-2:19.0.0-1643.g3bfc837d.el9.x86_64 115/182 2024-02-21T23:01:04.356 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-osd-2:19.0.0-1643.g3bfc837d.el9.x86_64 115/182 2024-02-21T23:01:04.356 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-21T23:01:04.357 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-02-21T23:01:04.357 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-21T23:01:04.357 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-21T23:01:04.357 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:01:04.404 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-volume-2:19.0.0-1643.g3bfc837d.el9.noarch 114/182 2024-02-21T23:01:04.453 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-volume-2:19.0.0-1643.g3bfc837d.el9.noarch 114/182 2024-02-21T23:01:04.453 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-21T23:01:04.453 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-02-21T23:01:04.454 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-02-21T23:01:04.609 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mds-2:19.0.0-1643.g3bfc837d.el9.x86_64 116/182 2024-02-21T23:01:04.683 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mds-2:19.0.0-1643.g3bfc837d.el9.x86_64 116/182 2024-02-21T23:01:04.683 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-21T23:01:04.683 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-02-21T23:01:04.683 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-21T23:01:04.683 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-21T23:01:04.683 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:01:05.193 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mon-2:19.0.0-1643.g3bfc837d.el9.x86_64 117/182 2024-02-21T23:01:05.266 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mon-2:19.0.0-1643.g3bfc837d.el9.x86_64 117/182 2024-02-21T23:01:05.266 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-21T23:01:05.266 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-02-21T23:01:05.266 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-21T23:01:05.266 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-21T23:01:05.267 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:01:05.412 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : automake-1.16.2-8.el9.noarch 118/182 2024-02-21T23:01:05.477 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-ply-3.11-14.el9.noarch 119/182 2024-02-21T23:01:05.550 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 120/182 2024-02-21T23:01:05.804 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 121/182 2024-02-21T23:01:05.857 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 122/182 2024-02-21T23:01:05.950 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 123/182 2024-02-21T23:01:06.054 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-osd-2:19.0.0-1643.g3bfc837d.el9.x86_64 115/182 2024-02-21T23:01:06.070 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 124/182 2024-02-21T23:01:06.133 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-osd-2:19.0.0-1643.g3bfc837d.el9.x86_64 115/182 2024-02-21T23:01:06.133 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-21T23:01:06.133 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-02-21T23:01:06.133 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-21T23:01:06.133 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-21T23:01:06.133 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-02-21T23:01:06.270 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 125/182 2024-02-21T23:01:06.322 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 126/182 2024-02-21T23:01:06.362 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 127/182 2024-02-21T23:01:06.372 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mds-2:19.0.0-1643.g3bfc837d.el9.x86_64 116/182 2024-02-21T23:01:06.442 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mds-2:19.0.0-1643.g3bfc837d.el9.x86_64 116/182 2024-02-21T23:01:06.442 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-21T23:01:06.442 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-02-21T23:01:06.442 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-21T23:01:06.442 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-21T23:01:06.442 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-02-21T23:01:06.445 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 128/182 2024-02-21T23:01:06.487 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : jq-1.6-16.el9.x86_64 129/182 2024-02-21T23:01:06.521 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : fuse-2.9.9-15.el9.x86_64 130/182 2024-02-21T23:01:06.931 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mon-2:19.0.0-1643.g3bfc837d.el9.x86_64 117/182 2024-02-21T23:01:07.011 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mon-2:19.0.0-1643.g3bfc837d.el9.x86_64 117/182 2024-02-21T23:01:07.011 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-21T23:01:07.011 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-02-21T23:01:07.011 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-21T23:01:07.011 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-21T23:01:07.011 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-02-21T23:01:07.147 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : automake-1.16.2-8.el9.noarch 118/182 2024-02-21T23:01:07.157 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: cephadm-2:19.0.0-1643.g3bfc837d.el9.noarch 131/182 2024-02-21T23:01:07.198 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : cephadm-2:19.0.0-1643.g3bfc837d.el9.noarch 131/182 2024-02-21T23:01:07.211 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-ply-3.11-14.el9.noarch 119/182 2024-02-21T23:01:07.252 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1643.g3bfc837d.e 132/182 2024-02-21T23:01:07.286 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 120/182 2024-02-21T23:01:07.367 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1643.g3bfc837d. 133/182 2024-02-21T23:01:07.400 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1643.g3bfc837d.el9.noa 134/182 2024-02-21T23:01:07.517 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 121/182 2024-02-21T23:01:07.576 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 122/182 2024-02-21T23:01:07.661 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 123/182 2024-02-21T23:01:07.741 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1643.g3bfc837d.el9.noa 134/182 2024-02-21T23:01:07.768 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1643.g3bfc837d.el9.n 135/182 2024-02-21T23:01:07.770 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 124/182 2024-02-21T23:01:07.945 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 125/182 2024-02-21T23:01:07.992 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 126/182 2024-02-21T23:01:08.033 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 127/182 2024-02-21T23:01:08.066 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 128/182 2024-02-21T23:01:08.105 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : jq-1.6-16.el9.x86_64 129/182 2024-02-21T23:01:08.135 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : fuse-2.9.9-15.el9.x86_64 130/182 2024-02-21T23:01:08.789 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: cephadm-2:19.0.0-1643.g3bfc837d.el9.noarch 131/182 2024-02-21T23:01:08.821 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : cephadm-2:19.0.0-1643.g3bfc837d.el9.noarch 131/182 2024-02-21T23:01:08.876 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1643.g3bfc837d.e 132/182 2024-02-21T23:01:08.946 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1643.g3bfc837d.el9.n 135/182 2024-02-21T23:01:08.968 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1643.g3bf 136/182 2024-02-21T23:01:08.986 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1643.g3bfc837d. 133/182 2024-02-21T23:01:09.014 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1643.g3bfc837d.el9.noa 134/182 2024-02-21T23:01:09.116 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1643.g3bf 136/182 2024-02-21T23:01:09.298 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1643.g3bfc837d.el 137/182 2024-02-21T23:01:09.328 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-2:19.0.0-1643.g3bfc837d.el9.x86_64 138/182 2024-02-21T23:01:09.337 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1643.g3bfc837d.el9.noa 134/182 2024-02-21T23:01:09.382 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1643.g3bfc837d.el9.n 135/182 2024-02-21T23:01:09.400 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1643.g3bfc837d.el9.x86_64 138/182 2024-02-21T23:01:09.400 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-21T23:01:09.400 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-02-21T23:01:09.400 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-21T23:01:09.400 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-21T23:01:09.400 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:01:09.454 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-rook-2:19.0.0-1643.g3bfc837d.el9.noarch 139/182 2024-02-21T23:01:09.492 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1643.g3bfc837d.el9.noarch 139/182 2024-02-21T23:01:09.607 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-2:19.0.0-1643.g3bfc837d.el9.x86_64 140/182 2024-02-21T23:01:10.506 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1643.g3bfc837d.el9.n 135/182 2024-02-21T23:01:10.540 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1643.g3bf 136/182 2024-02-21T23:01:10.684 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1643.g3bf 136/182 2024-02-21T23:01:10.855 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1643.g3bfc837d.el 137/182 2024-02-21T23:01:10.885 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mgr-2:19.0.0-1643.g3bfc837d.el9.x86_64 138/182 2024-02-21T23:01:10.981 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1643.g3bfc837d.el9.x86_64 138/182 2024-02-21T23:01:10.982 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-21T23:01:10.982 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-02-21T23:01:10.982 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-21T23:01:10.982 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-21T23:01:10.982 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-02-21T23:01:11.044 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mgr-rook-2:19.0.0-1643.g3bfc837d.el9.noarch 139/182 2024-02-21T23:01:11.082 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1643.g3bfc837d.el9.noarch 139/182 2024-02-21T23:01:11.224 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-2:19.0.0-1643.g3bfc837d.el9.x86_64 140/182 2024-02-21T23:01:14.679 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-fuse-2:19.0.0-1643.g3bfc837d.el9.x86_64 141/182 2024-02-21T23:01:14.762 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-test-2:19.0.0-1643.g3bfc837d.el9.x86_64 142/182 2024-02-21T23:01:15.742 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libtool-2.4.6-45.el9.x86_64 143/182 2024-02-21T23:01:15.777 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-radosgw-2:19.0.0-1643.g3bfc837d.el9.x86_64 144/182 2024-02-21T23:01:15.850 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1643.g3bfc837d.el9.x86_64 144/182 2024-02-21T23:01:15.850 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-21T23:01:15.850 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-02-21T23:01:15.850 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-21T23:01:15.851 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-21T23:01:15.851 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:01:15.893 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1643.g3bfc8 145/182 2024-02-21T23:01:15.955 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-fuse-2:19.0.0-1643.g3bfc837d.el9.x86_64 141/182 2024-02-21T23:01:15.969 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1643.g3bfc8 145/182 2024-02-21T23:01:15.969 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-21T23:01:15.969 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-02-21T23:01:15.969 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:01:16.036 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : cephfs-mirror-2:19.0.0-1643.g3bfc837d.el9.x86_64 146/182 2024-02-21T23:01:16.073 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-test-2:19.0.0-1643.g3bfc837d.el9.x86_64 142/182 2024-02-21T23:01:16.115 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1643.g3bfc837d.el9.x86_64 146/182 2024-02-21T23:01:16.116 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-21T23:01:16.116 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-02-21T23:01:16.116 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-02-21T23:01:16.116 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-02-21T23:01:16.116 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:01:16.478 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : rbd-mirror-2:19.0.0-1643.g3bfc837d.el9.x86_64 147/182 2024-02-21T23:01:16.551 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1643.g3bfc837d.el9.x86_64 147/182 2024-02-21T23:01:16.551 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-21T23:01:16.551 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-02-21T23:01:16.551 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-21T23:01:16.551 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-21T23:01:16.551 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:01:16.621 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : rbd-fuse-2:19.0.0-1643.g3bfc837d.el9.x86_64 148/182 2024-02-21T23:01:16.678 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : rbd-nbd-2:19.0.0-1643.g3bfc837d.el9.x86_64 149/182 2024-02-21T23:01:16.714 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 150/182 2024-02-21T23:01:16.752 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 151/182 2024-02-21T23:01:16.989 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libtool-2.4.6-45.el9.x86_64 143/182 2024-02-21T23:01:17.025 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-radosgw-2:19.0.0-1643.g3bfc837d.el9.x86_64 144/182 2024-02-21T23:01:17.055 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : cephfs-shell-2:19.0.0-1643.g3bfc837d.el9.x86_64 152/182 2024-02-21T23:01:17.101 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 153/182 2024-02-21T23:01:17.119 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1643.g3bfc837d.el9.x86_64 144/182 2024-02-21T23:01:17.120 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-21T23:01:17.120 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-02-21T23:01:17.120 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-21T23:01:17.120 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-21T23:01:17.120 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-02-21T23:01:17.135 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libcephfs-devel-2:19.0.0-1643.g3bfc837d.el9.x86_ 154/182 2024-02-21T23:01:17.173 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 155/182 2024-02-21T23:01:17.176 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1643.g3bfc8 145/182 2024-02-21T23:01:17.221 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : NetworkManager-team-1:1.45.91-1.el9.x86_64 156/182 2024-02-21T23:01:17.257 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1643.g3bfc8 145/182 2024-02-21T23:01:17.307 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-21T23:01:17.307 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-02-21T23:01:17.307 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-02-21T23:01:17.311 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : NetworkManager-tui-1:1.45.91-1.el9.x86_64 157/182 2024-02-21T23:01:17.327 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : cephfs-mirror-2:19.0.0-1643.g3bfc837d.el9.x86_64 146/182 2024-02-21T23:01:17.402 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1643.g3bfc837d.el9.x86_64 146/182 2024-02-21T23:01:17.402 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-21T23:01:17.402 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-02-21T23:01:17.403 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-02-21T23:01:17.403 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-02-21T23:01:17.403 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-02-21T23:01:17.413 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 158/182 2024-02-21T23:01:17.460 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 159/182 2024-02-21T23:01:17.777 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : rbd-mirror-2:19.0.0-1643.g3bfc837d.el9.x86_64 147/182 2024-02-21T23:01:17.846 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1643.g3bfc837d.el9.x86_64 147/182 2024-02-21T23:01:17.846 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-21T23:01:17.846 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-02-21T23:01:17.846 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-21T23:01:17.846 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-21T23:01:17.847 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-02-21T23:01:17.942 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : rbd-fuse-2:19.0.0-1643.g3bfc837d.el9.x86_64 148/182 2024-02-21T23:01:18.009 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : rbd-nbd-2:19.0.0-1643.g3bfc837d.el9.x86_64 149/182 2024-02-21T23:01:18.046 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 150/182 2024-02-21T23:01:18.085 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 151/182 2024-02-21T23:01:18.374 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : cephfs-shell-2:19.0.0-1643.g3bfc837d.el9.x86_64 152/182 2024-02-21T23:01:18.425 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 153/182 2024-02-21T23:01:18.459 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libcephfs-devel-2:19.0.0-1643.g3bfc837d.el9.x86_ 154/182 2024-02-21T23:01:18.506 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 155/182 2024-02-21T23:01:18.553 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : NetworkManager-team-1:1.45.91-1.el9.x86_64 156/182 2024-02-21T23:01:18.625 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : NetworkManager-tui-1:1.45.91-1.el9.x86_64 157/182 2024-02-21T23:01:18.699 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 160/182 2024-02-21T23:01:18.725 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 158/182 2024-02-21T23:01:18.748 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 160/182 2024-02-21T23:01:18.759 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 159/182 2024-02-21T23:01:18.838 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 160/182 2024-02-21T23:01:18.908 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/182 2024-02-21T23:01:20.037 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 160/182 2024-02-21T23:01:20.083 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 160/182 2024-02-21T23:01:20.162 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 160/182 2024-02-21T23:01:20.235 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/182 2024-02-21T23:01:20.393 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : cephfs-top-2:19.0.0-1643.g3bfc837d.el9.noarch 162/182 2024-02-21T23:01:20.529 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : openssl-devel-1:3.0.7-25.el9.x86_64 163/182 2024-02-21T23:01:20.653 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flex-2.6.4-9.el9.x86_64 164/182 2024-02-21T23:01:20.690 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : bison-3.7.4-5.el9.x86_64 165/182 2024-02-21T23:01:20.716 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 166/182 2024-02-21T23:01:20.749 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 167/182 2024-02-21T23:01:20.784 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/182 2024-02-21T23:01:20.828 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 169/182 2024-02-21T23:01:20.828 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 170/182 2024-02-21T23:01:20.980 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 170/182 2024-02-21T23:01:21.023 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 171/182 2024-02-21T23:01:21.072 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 172/182 2024-02-21T23:01:21.106 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 173/182 2024-02-21T23:01:21.138 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 174/182 2024-02-21T23:01:21.138 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 175/182 2024-02-21T23:01:21.276 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 175/182 2024-02-21T23:01:21.306 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 176/182 2024-02-21T23:01:21.343 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 177/182 2024-02-21T23:01:21.344 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 177/182 2024-02-21T23:01:21.594 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : cephfs-top-2:19.0.0-1643.g3bfc837d.el9.noarch 162/182 2024-02-21T23:01:21.680 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 177/182 2024-02-21T23:01:21.680 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 178/182 2024-02-21T23:01:21.720 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : openssl-devel-1:3.0.7-25.el9.x86_64 163/182 2024-02-21T23:01:21.822 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 178/182 2024-02-21T23:01:21.859 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 179/182 2024-02-21T23:01:21.859 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : flex-2.6.4-9.el9.x86_64 164/182 2024-02-21T23:01:21.894 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : bison-3.7.4-5.el9.x86_64 165/182 2024-02-21T23:01:21.904 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 180/182 2024-02-21T23:01:21.919 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 166/182 2024-02-21T23:01:21.951 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 181/182 2024-02-21T23:01:21.953 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 167/182 2024-02-21T23:01:21.988 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/182 2024-02-21T23:01:22.038 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 169/182 2024-02-21T23:01:22.038 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 170/182 2024-02-21T23:01:22.074 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 182/182 2024-02-21T23:01:22.182 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 170/182 2024-02-21T23:01:22.218 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 171/182 2024-02-21T23:01:22.250 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 172/182 2024-02-21T23:01:22.284 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 173/182 2024-02-21T23:01:22.317 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 174/182 2024-02-21T23:01:22.317 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 175/182 2024-02-21T23:01:22.335 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 182/182 2024-02-21T23:01:22.335 INFO:teuthology.orchestra.run.smithi028.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-02-21T23:01:22.335 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:01:22.430 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 175/182 2024-02-21T23:01:22.460 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 176/182 2024-02-21T23:01:22.496 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 177/182 2024-02-21T23:01:22.496 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 177/182 2024-02-21T23:01:22.800 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 177/182 2024-02-21T23:01:22.801 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 178/182 2024-02-21T23:01:22.943 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 178/182 2024-02-21T23:01:22.979 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 179/182 2024-02-21T23:01:23.022 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 180/182 2024-02-21T23:01:23.064 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 181/182 2024-02-21T23:01:23.184 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 182/182 2024-02-21T23:01:23.438 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 182/182 2024-02-21T23:01:23.439 INFO:teuthology.orchestra.run.smithi081.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-02-21T23:01:23.439 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-02-21T23:01:27.466 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-2:19.0.0-1643.g3bfc837d.el9.x86_64 1/182 2024-02-21T23:01:27.466 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-base-2:19.0.0-1643.g3bfc837d.el9.x86_64 2/182 2024-02-21T23:01:27.466 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-common-2:19.0.0-1643.g3bfc837d.el9.x86_64 3/182 2024-02-21T23:01:27.466 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-fuse-2:19.0.0-1643.g3bfc837d.el9.x86_64 4/182 2024-02-21T23:01:27.467 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1643.g3bfc8 5/182 2024-02-21T23:01:27.467 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mds-2:19.0.0-1643.g3bfc837d.el9.x86_64 6/182 2024-02-21T23:01:27.467 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-2:19.0.0-1643.g3bfc837d.el9.x86_64 7/182 2024-02-21T23:01:27.467 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mon-2:19.0.0-1643.g3bfc837d.el9.x86_64 8/182 2024-02-21T23:01:27.467 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-osd-2:19.0.0-1643.g3bfc837d.el9.x86_64 9/182 2024-02-21T23:01:27.467 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-radosgw-2:19.0.0-1643.g3bfc837d.el9.x86_64 10/182 2024-02-21T23:01:27.467 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-selinux-2:19.0.0-1643.g3bfc837d.el9.x86_64 11/182 2024-02-21T23:01:27.467 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-test-2:19.0.0-1643.g3bfc837d.el9.x86_64 12/182 2024-02-21T23:01:27.467 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : cephfs-mirror-2:19.0.0-1643.g3bfc837d.el9.x86_64 13/182 2024-02-21T23:01:27.467 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : cephfs-shell-2:19.0.0-1643.g3bfc837d.el9.x86_64 14/182 2024-02-21T23:01:27.467 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libcephfs-devel-2:19.0.0-1643.g3bfc837d.el9.x86_ 15/182 2024-02-21T23:01:27.467 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libcephfs2-2:19.0.0-1643.g3bfc837d.el9.x86_64 16/182 2024-02-21T23:01:27.467 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libcephsqlite-2:19.0.0-1643.g3bfc837d.el9.x86_64 17/182 2024-02-21T23:01:27.467 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librados-devel-2:19.0.0-1643.g3bfc837d.el9.x86_6 18/182 2024-02-21T23:01:27.468 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libradosstriper1-2:19.0.0-1643.g3bfc837d.el9.x86 19/182 2024-02-21T23:01:27.468 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librgw2-2:19.0.0-1643.g3bfc837d.el9.x86_64 20/182 2024-02-21T23:01:27.468 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1643.g3bfc837d.el 21/182 2024-02-21T23:01:27.468 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-ceph-common-2:19.0.0-1643.g3bfc837d.el9. 22/182 2024-02-21T23:01:27.468 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cephfs-2:19.0.0-1643.g3bfc837d.el9.x86_6 23/182 2024-02-21T23:01:27.468 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rados-2:19.0.0-1643.g3bfc837d.el9.x86_64 24/182 2024-02-21T23:01:27.468 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rbd-2:19.0.0-1643.g3bfc837d.el9.x86_64 25/182 2024-02-21T23:01:27.468 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rgw-2:19.0.0-1643.g3bfc837d.el9.x86_64 26/182 2024-02-21T23:01:27.468 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : rbd-fuse-2:19.0.0-1643.g3bfc837d.el9.x86_64 27/182 2024-02-21T23:01:27.468 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : rbd-mirror-2:19.0.0-1643.g3bfc837d.el9.x86_64 28/182 2024-02-21T23:01:27.468 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : rbd-nbd-2:19.0.0-1643.g3bfc837d.el9.x86_64 29/182 2024-02-21T23:01:27.468 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1643.g3bfc837d. 30/182 2024-02-21T23:01:27.481 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1643.g3bfc837d.el9.noa 31/182 2024-02-21T23:01:27.481 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1643.g3bfc837d.el9.n 32/182 2024-02-21T23:01:27.481 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1643.g3bf 33/182 2024-02-21T23:01:27.481 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1643.g3bfc837d.el 34/182 2024-02-21T23:01:27.481 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1643.g3bfc837d.el9.noarch 35/182 2024-02-21T23:01:27.481 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1643.g3bfc837d.e 36/182 2024-02-21T23:01:27.481 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-volume-2:19.0.0-1643.g3bfc837d.el9.noarch 37/182 2024-02-21T23:01:27.481 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : cephadm-2:19.0.0-1643.g3bfc837d.el9.noarch 38/182 2024-02-21T23:01:27.481 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : cephfs-top-2:19.0.0-1643.g3bfc837d.el9.noarch 39/182 2024-02-21T23:01:27.481 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 40/182 2024-02-21T23:01:27.481 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 41/182 2024-02-21T23:01:27.481 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : jq-1.6-16.el9.x86_64 42/182 2024-02-21T23:01:27.482 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 43/182 2024-02-21T23:01:27.482 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 44/182 2024-02-21T23:01:27.482 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 45/182 2024-02-21T23:01:27.482 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 46/182 2024-02-21T23:01:27.482 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 47/182 2024-02-21T23:01:27.482 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-ply-3.11-14.el9.noarch 48/182 2024-02-21T23:01:27.482 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 49/182 2024-02-21T23:01:27.482 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 50/182 2024-02-21T23:01:27.482 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 51/182 2024-02-21T23:01:27.482 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : automake-1.16.2-8.el9.noarch 52/182 2024-02-21T23:01:27.482 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : bison-3.7.4-5.el9.x86_64 53/182 2024-02-21T23:01:27.482 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 54/182 2024-02-21T23:01:27.483 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 55/182 2024-02-21T23:01:27.483 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flex-2.6.4-9.el9.x86_64 56/182 2024-02-21T23:01:27.483 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 57/182 2024-02-21T23:01:27.483 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 58/182 2024-02-21T23:01:27.483 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 59/182 2024-02-21T23:01:27.483 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 60/182 2024-02-21T23:01:27.483 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 61/182 2024-02-21T23:01:27.483 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 62/182 2024-02-21T23:01:27.483 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 63/182 2024-02-21T23:01:27.483 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libuuid-devel-2.37.4-15.el9.x86_64 64/182 2024-02-21T23:01:27.483 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 65/182 2024-02-21T23:01:27.483 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 66/182 2024-02-21T23:01:27.483 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : lua-5.4.4-4.el9.x86_64 67/182 2024-02-21T23:01:27.484 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 68/182 2024-02-21T23:01:27.484 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 69/182 2024-02-21T23:01:27.484 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-devel-1:3.0.7-25.el9.x86_64 70/182 2024-02-21T23:01:27.484 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : postgresql-13.11-1.el9.x86_64 71/182 2024-02-21T23:01:27.484 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 72/182 2024-02-21T23:01:27.484 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 73/182 2024-02-21T23:01:27.484 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 74/182 2024-02-21T23:01:27.484 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 75/182 2024-02-21T23:01:27.484 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 76/182 2024-02-21T23:01:27.484 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 77/182 2024-02-21T23:01:27.484 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 78/182 2024-02-21T23:01:27.484 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 79/182 2024-02-21T23:01:27.484 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 80/182 2024-02-21T23:01:27.485 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 81/182 2024-02-21T23:01:27.485 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 82/182 2024-02-21T23:01:27.485 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 83/182 2024-02-21T23:01:27.485 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 84/182 2024-02-21T23:01:27.485 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 85/182 2024-02-21T23:01:27.485 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 86/182 2024-02-21T23:01:27.485 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 87/182 2024-02-21T23:01:27.485 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 88/182 2024-02-21T23:01:27.485 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 89/182 2024-02-21T23:01:27.485 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 90/182 2024-02-21T23:01:27.485 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 91/182 2024-02-21T23:01:27.485 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 92/182 2024-02-21T23:01:27.485 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 93/182 2024-02-21T23:01:27.486 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 94/182 2024-02-21T23:01:27.486 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 95/182 2024-02-21T23:01:27.486 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 96/182 2024-02-21T23:01:27.486 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 97/182 2024-02-21T23:01:27.486 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 98/182 2024-02-21T23:01:27.486 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 99/182 2024-02-21T23:01:27.486 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 100/182 2024-02-21T23:01:27.486 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 101/182 2024-02-21T23:01:27.486 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 102/182 2024-02-21T23:01:27.486 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 103/182 2024-02-21T23:01:27.486 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 104/182 2024-02-21T23:01:27.486 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/182 2024-02-21T23:01:27.486 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/182 2024-02-21T23:01:27.486 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/182 2024-02-21T23:01:27.486 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/182 2024-02-21T23:01:27.487 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/182 2024-02-21T23:01:27.487 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/182 2024-02-21T23:01:27.487 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/182 2024-02-21T23:01:27.487 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/182 2024-02-21T23:01:27.487 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/182 2024-02-21T23:01:27.487 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/182 2024-02-21T23:01:27.487 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/182 2024-02-21T23:01:27.487 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/182 2024-02-21T23:01:27.487 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/182 2024-02-21T23:01:27.487 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/182 2024-02-21T23:01:27.487 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/182 2024-02-21T23:01:27.487 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/182 2024-02-21T23:01:27.487 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/182 2024-02-21T23:01:27.487 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/182 2024-02-21T23:01:27.488 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/182 2024-02-21T23:01:27.488 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-google-auth-1:2.27.0-1.el9.noarch 124/182 2024-02-21T23:01:27.488 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/182 2024-02-21T23:01:27.488 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/182 2024-02-21T23:01:27.488 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/182 2024-02-21T23:01:27.488 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/182 2024-02-21T23:01:27.488 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/182 2024-02-21T23:01:27.488 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/182 2024-02-21T23:01:27.488 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/182 2024-02-21T23:01:27.488 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/182 2024-02-21T23:01:27.488 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/182 2024-02-21T23:01:27.488 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/182 2024-02-21T23:01:27.488 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/182 2024-02-21T23:01:27.489 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/182 2024-02-21T23:01:27.489 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/182 2024-02-21T23:01:27.489 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/182 2024-02-21T23:01:27.489 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/182 2024-02-21T23:01:27.489 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/182 2024-02-21T23:01:27.489 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/182 2024-02-21T23:01:27.489 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/182 2024-02-21T23:01:27.489 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/182 2024-02-21T23:01:27.489 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/182 2024-02-21T23:01:27.489 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/182 2024-02-21T23:01:27.489 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/182 2024-02-21T23:01:27.489 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/182 2024-02-21T23:01:27.489 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/182 2024-02-21T23:01:27.489 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/182 2024-02-21T23:01:27.490 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/182 2024-02-21T23:01:27.490 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librados2-2:19.0.0-1643.g3bfc837d.el9.x86_64 151/182 2024-02-21T23:01:27.490 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/182 2024-02-21T23:01:27.490 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librbd1-2:19.0.0-1643.g3bfc837d.el9.x86_64 153/182 2024-02-21T23:01:27.490 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/182 2024-02-21T23:01:27.490 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-1:1.45.91-1.el9.x86_64 155/182 2024-02-21T23:01:27.490 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 156/182 2024-02-21T23:01:27.490 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-libnm-1:1.45.91-1.el9.x86_64 157/182 2024-02-21T23:01:27.490 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 158/182 2024-02-21T23:01:27.490 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-team-1:1.45.91-1.el9.x86_64 159/182 2024-02-21T23:01:27.490 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 160/182 2024-02-21T23:01:27.490 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-tui-1:1.45.91-1.el9.x86_64 161/182 2024-02-21T23:01:27.490 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 162/182 2024-02-21T23:01:27.490 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : acl-2.3.1-4.el9.x86_64 163/182 2024-02-21T23:01:27.491 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : acl-2.3.1-3.el9.x86_64 164/182 2024-02-21T23:01:27.491 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 165/182 2024-02-21T23:01:27.491 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 166/182 2024-02-21T23:01:27.491 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 167/182 2024-02-21T23:01:27.491 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 168/182 2024-02-21T23:01:27.491 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 169/182 2024-02-21T23:01:27.491 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 170/182 2024-02-21T23:01:27.491 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 171/182 2024-02-21T23:01:27.491 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 172/182 2024-02-21T23:01:27.491 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-1:3.0.7-25.el9.x86_64 173/182 2024-02-21T23:01:27.491 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 174/182 2024-02-21T23:01:27.491 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-libs-1:3.0.7-25.el9.x86_64 175/182 2024-02-21T23:01:27.491 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 176/182 2024-02-21T23:01:27.492 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 177/182 2024-02-21T23:01:27.492 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 178/182 2024-02-21T23:01:27.492 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 179/182 2024-02-21T23:01:27.492 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 180/182 2024-02-21T23:01:27.492 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 181/182 2024-02-21T23:01:28.610 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-2:19.0.0-1643.g3bfc837d.el9.x86_64 1/182 2024-02-21T23:01:28.611 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-base-2:19.0.0-1643.g3bfc837d.el9.x86_64 2/182 2024-02-21T23:01:28.611 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-common-2:19.0.0-1643.g3bfc837d.el9.x86_64 3/182 2024-02-21T23:01:28.611 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-fuse-2:19.0.0-1643.g3bfc837d.el9.x86_64 4/182 2024-02-21T23:01:28.611 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1643.g3bfc8 5/182 2024-02-21T23:01:28.611 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mds-2:19.0.0-1643.g3bfc837d.el9.x86_64 6/182 2024-02-21T23:01:28.611 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mgr-2:19.0.0-1643.g3bfc837d.el9.x86_64 7/182 2024-02-21T23:01:28.611 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mon-2:19.0.0-1643.g3bfc837d.el9.x86_64 8/182 2024-02-21T23:01:28.611 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-osd-2:19.0.0-1643.g3bfc837d.el9.x86_64 9/182 2024-02-21T23:01:28.611 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-radosgw-2:19.0.0-1643.g3bfc837d.el9.x86_64 10/182 2024-02-21T23:01:28.611 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-selinux-2:19.0.0-1643.g3bfc837d.el9.x86_64 11/182 2024-02-21T23:01:28.611 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-test-2:19.0.0-1643.g3bfc837d.el9.x86_64 12/182 2024-02-21T23:01:28.611 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : cephfs-mirror-2:19.0.0-1643.g3bfc837d.el9.x86_64 13/182 2024-02-21T23:01:28.611 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : cephfs-shell-2:19.0.0-1643.g3bfc837d.el9.x86_64 14/182 2024-02-21T23:01:28.611 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libcephfs-devel-2:19.0.0-1643.g3bfc837d.el9.x86_ 15/182 2024-02-21T23:01:28.612 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libcephfs2-2:19.0.0-1643.g3bfc837d.el9.x86_64 16/182 2024-02-21T23:01:28.612 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libcephsqlite-2:19.0.0-1643.g3bfc837d.el9.x86_64 17/182 2024-02-21T23:01:28.612 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librados-devel-2:19.0.0-1643.g3bfc837d.el9.x86_6 18/182 2024-02-21T23:01:28.612 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libradosstriper1-2:19.0.0-1643.g3bfc837d.el9.x86 19/182 2024-02-21T23:01:28.612 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librgw2-2:19.0.0-1643.g3bfc837d.el9.x86_64 20/182 2024-02-21T23:01:28.612 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1643.g3bfc837d.el 21/182 2024-02-21T23:01:28.612 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-ceph-common-2:19.0.0-1643.g3bfc837d.el9. 22/182 2024-02-21T23:01:28.612 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cephfs-2:19.0.0-1643.g3bfc837d.el9.x86_6 23/182 2024-02-21T23:01:28.612 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-rados-2:19.0.0-1643.g3bfc837d.el9.x86_64 24/182 2024-02-21T23:01:28.612 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-rbd-2:19.0.0-1643.g3bfc837d.el9.x86_64 25/182 2024-02-21T23:01:28.612 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-rgw-2:19.0.0-1643.g3bfc837d.el9.x86_64 26/182 2024-02-21T23:01:28.612 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : rbd-fuse-2:19.0.0-1643.g3bfc837d.el9.x86_64 27/182 2024-02-21T23:01:28.612 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : rbd-mirror-2:19.0.0-1643.g3bfc837d.el9.x86_64 28/182 2024-02-21T23:01:28.613 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : rbd-nbd-2:19.0.0-1643.g3bfc837d.el9.x86_64 29/182 2024-02-21T23:01:28.627 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1643.g3bfc837d. 30/182 2024-02-21T23:01:28.627 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1643.g3bfc837d.el9.noa 31/182 2024-02-21T23:01:28.628 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1643.g3bfc837d.el9.n 32/182 2024-02-21T23:01:28.628 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1643.g3bf 33/182 2024-02-21T23:01:28.628 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1643.g3bfc837d.el 34/182 2024-02-21T23:01:28.628 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1643.g3bfc837d.el9.noarch 35/182 2024-02-21T23:01:28.628 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1643.g3bfc837d.e 36/182 2024-02-21T23:01:28.628 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-volume-2:19.0.0-1643.g3bfc837d.el9.noarch 37/182 2024-02-21T23:01:28.628 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : cephadm-2:19.0.0-1643.g3bfc837d.el9.noarch 38/182 2024-02-21T23:01:28.628 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : cephfs-top-2:19.0.0-1643.g3bfc837d.el9.noarch 39/182 2024-02-21T23:01:28.628 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 40/182 2024-02-21T23:01:28.628 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 41/182 2024-02-21T23:01:28.628 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : jq-1.6-16.el9.x86_64 42/182 2024-02-21T23:01:28.628 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 43/182 2024-02-21T23:01:28.628 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 44/182 2024-02-21T23:01:28.629 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 45/182 2024-02-21T23:01:28.629 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 46/182 2024-02-21T23:01:28.629 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 47/182 2024-02-21T23:01:28.629 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-ply-3.11-14.el9.noarch 48/182 2024-02-21T23:01:28.629 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 49/182 2024-02-21T23:01:28.629 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 50/182 2024-02-21T23:01:28.629 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 51/182 2024-02-21T23:01:28.629 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : automake-1.16.2-8.el9.noarch 52/182 2024-02-21T23:01:28.629 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : bison-3.7.4-5.el9.x86_64 53/182 2024-02-21T23:01:28.629 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 54/182 2024-02-21T23:01:28.629 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 55/182 2024-02-21T23:01:28.629 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : flex-2.6.4-9.el9.x86_64 56/182 2024-02-21T23:01:28.629 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 57/182 2024-02-21T23:01:28.629 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 58/182 2024-02-21T23:01:28.630 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 59/182 2024-02-21T23:01:28.630 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 60/182 2024-02-21T23:01:28.630 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 61/182 2024-02-21T23:01:28.630 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 62/182 2024-02-21T23:01:28.630 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 63/182 2024-02-21T23:01:28.630 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libuuid-devel-2.37.4-15.el9.x86_64 64/182 2024-02-21T23:01:28.630 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 65/182 2024-02-21T23:01:28.630 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 66/182 2024-02-21T23:01:28.630 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : lua-5.4.4-4.el9.x86_64 67/182 2024-02-21T23:01:28.630 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 68/182 2024-02-21T23:01:28.630 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 69/182 2024-02-21T23:01:28.630 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : openssl-devel-1:3.0.7-25.el9.x86_64 70/182 2024-02-21T23:01:28.630 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : postgresql-13.11-1.el9.x86_64 71/182 2024-02-21T23:01:28.630 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 72/182 2024-02-21T23:01:28.631 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 73/182 2024-02-21T23:01:28.631 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 74/182 2024-02-21T23:01:28.631 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 75/182 2024-02-21T23:01:28.631 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 76/182 2024-02-21T23:01:28.631 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 77/182 2024-02-21T23:01:28.631 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 78/182 2024-02-21T23:01:28.631 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 79/182 2024-02-21T23:01:28.631 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 80/182 2024-02-21T23:01:28.631 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 81/182 2024-02-21T23:01:28.631 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 82/182 2024-02-21T23:01:28.631 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 83/182 2024-02-21T23:01:28.631 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 84/182 2024-02-21T23:01:28.632 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 85/182 2024-02-21T23:01:28.632 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 86/182 2024-02-21T23:01:28.632 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 87/182 2024-02-21T23:01:28.632 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 88/182 2024-02-21T23:01:28.632 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 89/182 2024-02-21T23:01:28.632 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 90/182 2024-02-21T23:01:28.632 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 91/182 2024-02-21T23:01:28.632 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 92/182 2024-02-21T23:01:28.632 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 93/182 2024-02-21T23:01:28.632 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 94/182 2024-02-21T23:01:28.632 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 95/182 2024-02-21T23:01:28.632 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 96/182 2024-02-21T23:01:28.632 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 97/182 2024-02-21T23:01:28.633 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 98/182 2024-02-21T23:01:28.633 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 99/182 2024-02-21T23:01:28.633 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 100/182 2024-02-21T23:01:28.633 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 101/182 2024-02-21T23:01:28.633 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 102/182 2024-02-21T23:01:28.633 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 103/182 2024-02-21T23:01:28.633 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 104/182 2024-02-21T23:01:28.633 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/182 2024-02-21T23:01:28.633 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/182 2024-02-21T23:01:28.633 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/182 2024-02-21T23:01:28.633 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/182 2024-02-21T23:01:28.633 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/182 2024-02-21T23:01:28.633 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/182 2024-02-21T23:01:28.633 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/182 2024-02-21T23:01:28.634 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/182 2024-02-21T23:01:28.634 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/182 2024-02-21T23:01:28.634 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/182 2024-02-21T23:01:28.634 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/182 2024-02-21T23:01:28.634 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/182 2024-02-21T23:01:28.634 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/182 2024-02-21T23:01:28.634 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/182 2024-02-21T23:01:28.634 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/182 2024-02-21T23:01:28.634 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/182 2024-02-21T23:01:28.634 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/182 2024-02-21T23:01:28.634 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/182 2024-02-21T23:01:28.634 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/182 2024-02-21T23:01:28.634 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-google-auth-1:2.27.0-1.el9.noarch 124/182 2024-02-21T23:01:28.634 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/182 2024-02-21T23:01:28.635 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/182 2024-02-21T23:01:28.635 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/182 2024-02-21T23:01:28.635 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/182 2024-02-21T23:01:28.635 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/182 2024-02-21T23:01:28.635 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/182 2024-02-21T23:01:28.635 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/182 2024-02-21T23:01:28.635 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/182 2024-02-21T23:01:28.635 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/182 2024-02-21T23:01:28.635 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/182 2024-02-21T23:01:28.635 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/182 2024-02-21T23:01:28.635 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/182 2024-02-21T23:01:28.635 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/182 2024-02-21T23:01:28.635 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/182 2024-02-21T23:01:28.636 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/182 2024-02-21T23:01:28.636 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/182 2024-02-21T23:01:28.636 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/182 2024-02-21T23:01:28.636 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/182 2024-02-21T23:01:28.636 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/182 2024-02-21T23:01:28.636 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/182 2024-02-21T23:01:28.636 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/182 2024-02-21T23:01:28.636 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/182 2024-02-21T23:01:28.636 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/182 2024-02-21T23:01:28.636 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/182 2024-02-21T23:01:28.636 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/182 2024-02-21T23:01:28.636 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/182 2024-02-21T23:01:28.636 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librados2-2:19.0.0-1643.g3bfc837d.el9.x86_64 151/182 2024-02-21T23:01:28.636 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/182 2024-02-21T23:01:28.637 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librbd1-2:19.0.0-1643.g3bfc837d.el9.x86_64 153/182 2024-02-21T23:01:28.637 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/182 2024-02-21T23:01:28.637 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : NetworkManager-1:1.45.91-1.el9.x86_64 155/182 2024-02-21T23:01:28.637 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 156/182 2024-02-21T23:01:28.637 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : NetworkManager-libnm-1:1.45.91-1.el9.x86_64 157/182 2024-02-21T23:01:28.637 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 158/182 2024-02-21T23:01:28.637 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : NetworkManager-team-1:1.45.91-1.el9.x86_64 159/182 2024-02-21T23:01:28.637 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 160/182 2024-02-21T23:01:28.637 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : NetworkManager-tui-1:1.45.91-1.el9.x86_64 161/182 2024-02-21T23:01:28.637 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 162/182 2024-02-21T23:01:28.637 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : acl-2.3.1-4.el9.x86_64 163/182 2024-02-21T23:01:28.637 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : acl-2.3.1-3.el9.x86_64 164/182 2024-02-21T23:01:28.637 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 165/182 2024-02-21T23:01:28.637 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 166/182 2024-02-21T23:01:28.638 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 167/182 2024-02-21T23:01:28.638 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 168/182 2024-02-21T23:01:28.638 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 169/182 2024-02-21T23:01:28.638 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 170/182 2024-02-21T23:01:28.638 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 171/182 2024-02-21T23:01:28.638 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 172/182 2024-02-21T23:01:28.638 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : openssl-1:3.0.7-25.el9.x86_64 173/182 2024-02-21T23:01:28.638 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 174/182 2024-02-21T23:01:28.638 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : openssl-libs-1:3.0.7-25.el9.x86_64 175/182 2024-02-21T23:01:28.638 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 176/182 2024-02-21T23:01:28.638 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 177/182 2024-02-21T23:01:28.638 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 178/182 2024-02-21T23:01:28.638 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 179/182 2024-02-21T23:01:28.638 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 180/182 2024-02-21T23:01:28.639 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 181/182 2024-02-21T23:01:29.953 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 182/182 2024-02-21T23:01:29.953 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:01:29.954 INFO:teuthology.orchestra.run.smithi028.stdout:Upgraded: 2024-02-21T23:01:29.954 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager-1:1.45.91-1.el9.x86_64 2024-02-21T23:01:29.954 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager-libnm-1:1.45.91-1.el9.x86_64 2024-02-21T23:01:29.954 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager-team-1:1.45.91-1.el9.x86_64 2024-02-21T23:01:29.954 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager-tui-1:1.45.91-1.el9.x86_64 2024-02-21T23:01:29.954 INFO:teuthology.orchestra.run.smithi028.stdout: acl-2.3.1-4.el9.x86_64 2024-02-21T23:01:29.954 INFO:teuthology.orchestra.run.smithi028.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-02-21T23:01:29.954 INFO:teuthology.orchestra.run.smithi028.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-02-21T23:01:29.954 INFO:teuthology.orchestra.run.smithi028.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-02-21T23:01:29.954 INFO:teuthology.orchestra.run.smithi028.stdout: libacl-2.3.1-4.el9.x86_64 2024-02-21T23:01:29.955 INFO:teuthology.orchestra.run.smithi028.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-02-21T23:01:29.955 INFO:teuthology.orchestra.run.smithi028.stdout: librados2-2:19.0.0-1643.g3bfc837d.el9.x86_64 2024-02-21T23:01:29.955 INFO:teuthology.orchestra.run.smithi028.stdout: librbd1-2:19.0.0-1643.g3bfc837d.el9.x86_64 2024-02-21T23:01:29.955 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-1:3.0.7-25.el9.x86_64 2024-02-21T23:01:29.955 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-libs-1:3.0.7-25.el9.x86_64 2024-02-21T23:01:29.955 INFO:teuthology.orchestra.run.smithi028.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-02-21T23:01:29.955 INFO:teuthology.orchestra.run.smithi028.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-02-21T23:01:29.955 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-02-21T23:01:29.955 INFO:teuthology.orchestra.run.smithi028.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-02-21T23:01:29.955 INFO:teuthology.orchestra.run.smithi028.stdout: automake-1.16.2-8.el9.noarch 2024-02-21T23:01:29.955 INFO:teuthology.orchestra.run.smithi028.stdout: bison-3.7.4-5.el9.x86_64 2024-02-21T23:01:29.955 INFO:teuthology.orchestra.run.smithi028.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-02-21T23:01:29.955 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-2:19.0.0-1643.g3bfc837d.el9.x86_64 2024-02-21T23:01:29.956 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-base-2:19.0.0-1643.g3bfc837d.el9.x86_64 2024-02-21T23:01:29.956 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-common-2:19.0.0-1643.g3bfc837d.el9.x86_64 2024-02-21T23:01:29.956 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-fuse-2:19.0.0-1643.g3bfc837d.el9.x86_64 2024-02-21T23:01:29.956 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-grafana-dashboards-2:19.0.0-1643.g3bfc837d.el9.noarch 2024-02-21T23:01:29.956 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-immutable-object-cache-2:19.0.0-1643.g3bfc837d.el9.x86_64 2024-02-21T23:01:29.956 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mds-2:19.0.0-1643.g3bfc837d.el9.x86_64 2024-02-21T23:01:29.956 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-2:19.0.0-1643.g3bfc837d.el9.x86_64 2024-02-21T23:01:29.956 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-cephadm-2:19.0.0-1643.g3bfc837d.el9.noarch 2024-02-21T23:01:29.956 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-dashboard-2:19.0.0-1643.g3bfc837d.el9.noarch 2024-02-21T23:01:29.956 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1643.g3bfc837d.el9.noarch 2024-02-21T23:01:29.956 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-modules-core-2:19.0.0-1643.g3bfc837d.el9.noarch 2024-02-21T23:01:29.957 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-rook-2:19.0.0-1643.g3bfc837d.el9.noarch 2024-02-21T23:01:29.957 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mon-2:19.0.0-1643.g3bfc837d.el9.x86_64 2024-02-21T23:01:29.957 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-osd-2:19.0.0-1643.g3bfc837d.el9.x86_64 2024-02-21T23:01:29.957 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-prometheus-alerts-2:19.0.0-1643.g3bfc837d.el9.noarch 2024-02-21T23:01:29.957 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-radosgw-2:19.0.0-1643.g3bfc837d.el9.x86_64 2024-02-21T23:01:29.957 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-selinux-2:19.0.0-1643.g3bfc837d.el9.x86_64 2024-02-21T23:01:29.957 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-test-2:19.0.0-1643.g3bfc837d.el9.x86_64 2024-02-21T23:01:29.957 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-volume-2:19.0.0-1643.g3bfc837d.el9.noarch 2024-02-21T23:01:29.957 INFO:teuthology.orchestra.run.smithi028.stdout: cephadm-2:19.0.0-1643.g3bfc837d.el9.noarch 2024-02-21T23:01:29.957 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-mirror-2:19.0.0-1643.g3bfc837d.el9.x86_64 2024-02-21T23:01:29.957 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-shell-2:19.0.0-1643.g3bfc837d.el9.x86_64 2024-02-21T23:01:29.957 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-top-2:19.0.0-1643.g3bfc837d.el9.noarch 2024-02-21T23:01:29.958 INFO:teuthology.orchestra.run.smithi028.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-02-21T23:01:29.958 INFO:teuthology.orchestra.run.smithi028.stdout: flex-2.6.4-9.el9.x86_64 2024-02-21T23:01:29.958 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-02-21T23:01:29.958 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-02-21T23:01:29.958 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-02-21T23:01:29.958 INFO:teuthology.orchestra.run.smithi028.stdout: fuse-2.9.9-15.el9.x86_64 2024-02-21T23:01:29.958 INFO:teuthology.orchestra.run.smithi028.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-02-21T23:01:29.958 INFO:teuthology.orchestra.run.smithi028.stdout: grpc-data-1.46.7-10.el9.noarch 2024-02-21T23:01:29.958 INFO:teuthology.orchestra.run.smithi028.stdout: jq-1.6-16.el9.x86_64 2024-02-21T23:01:29.958 INFO:teuthology.orchestra.run.smithi028.stdout: libaio-devel-0.3.112-1.el8.x86_64 2024-02-21T23:01:29.958 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow-9.0.0-10.el9.x86_64 2024-02-21T23:01:29.958 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-02-21T23:01:29.959 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs-devel-2:19.0.0-1643.g3bfc837d.el9.x86_64 2024-02-21T23:01:29.959 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs2-2:19.0.0-1643.g3bfc837d.el9.x86_64 2024-02-21T23:01:29.959 INFO:teuthology.orchestra.run.smithi028.stdout: libcephsqlite-2:19.0.0-1643.g3bfc837d.el9.x86_64 2024-02-21T23:01:29.959 INFO:teuthology.orchestra.run.smithi028.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-02-21T23:01:29.959 INFO:teuthology.orchestra.run.smithi028.stdout: liboath-2.6.7-2.el9.x86_64 2024-02-21T23:01:29.959 INFO:teuthology.orchestra.run.smithi028.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-02-21T23:01:29.959 INFO:teuthology.orchestra.run.smithi028.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-02-21T23:01:29.959 INFO:teuthology.orchestra.run.smithi028.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-02-21T23:01:29.959 INFO:teuthology.orchestra.run.smithi028.stdout: librados-devel-2:19.0.0-1643.g3bfc837d.el9.x86_64 2024-02-21T23:01:29.959 INFO:teuthology.orchestra.run.smithi028.stdout: libradosstriper1-2:19.0.0-1643.g3bfc837d.el9.x86_64 2024-02-21T23:01:29.959 INFO:teuthology.orchestra.run.smithi028.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-02-21T23:01:29.959 INFO:teuthology.orchestra.run.smithi028.stdout: librgw2-2:19.0.0-1643.g3bfc837d.el9.x86_64 2024-02-21T23:01:29.959 INFO:teuthology.orchestra.run.smithi028.stdout: libtool-2.4.6-45.el9.x86_64 2024-02-21T23:01:29.960 INFO:teuthology.orchestra.run.smithi028.stdout: libunwind-1.6.2-1.el9.x86_64 2024-02-21T23:01:29.960 INFO:teuthology.orchestra.run.smithi028.stdout: libuuid-devel-2.37.4-15.el9.x86_64 2024-02-21T23:01:29.960 INFO:teuthology.orchestra.run.smithi028.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-02-21T23:01:29.960 INFO:teuthology.orchestra.run.smithi028.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-02-21T23:01:29.960 INFO:teuthology.orchestra.run.smithi028.stdout: lua-5.4.4-4.el9.x86_64 2024-02-21T23:01:29.960 INFO:teuthology.orchestra.run.smithi028.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-02-21T23:01:29.960 INFO:teuthology.orchestra.run.smithi028.stdout: luarocks-3.9.2-1.el9.noarch 2024-02-21T23:01:29.960 INFO:teuthology.orchestra.run.smithi028.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-02-21T23:01:29.960 INFO:teuthology.orchestra.run.smithi028.stdout: openblas-0.3.21-2.el9.x86_64 2024-02-21T23:01:29.960 INFO:teuthology.orchestra.run.smithi028.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-02-21T23:01:29.960 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-devel-1:3.0.7-25.el9.x86_64 2024-02-21T23:01:29.960 INFO:teuthology.orchestra.run.smithi028.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-02-21T23:01:29.961 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-13.11-1.el9.x86_64 2024-02-21T23:01:29.961 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-02-21T23:01:29.961 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-02-21T23:01:29.961 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-server-13.11-1.el9.x86_64 2024-02-21T23:01:29.961 INFO:teuthology.orchestra.run.smithi028.stdout: protobuf-3.14.0-13.el9.x86_64 2024-02-21T23:01:29.961 INFO:teuthology.orchestra.run.smithi028.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-02-21T23:01:29.961 INFO:teuthology.orchestra.run.smithi028.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-02-21T23:01:29.961 INFO:teuthology.orchestra.run.smithi028.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-02-21T23:01:29.961 INFO:teuthology.orchestra.run.smithi028.stdout: python3-babel-2.9.1-2.el9.noarch 2024-02-21T23:01:29.961 INFO:teuthology.orchestra.run.smithi028.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-02-21T23:01:29.961 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-02-21T23:01:29.961 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-argparse-2:19.0.0-1643.g3bfc837d.el9.x86_64 2024-02-21T23:01:29.961 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-common-2:19.0.0-1643.g3bfc837d.el9.x86_64 2024-02-21T23:01:29.962 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cephfs-2:19.0.0-1643.g3bfc837d.el9.x86_64 2024-02-21T23:01:29.962 INFO:teuthology.orchestra.run.smithi028.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-02-21T23:01:29.962 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-02-21T23:01:29.962 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-02-21T23:01:29.962 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-02-21T23:01:29.962 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-02-21T23:01:29.962 INFO:teuthology.orchestra.run.smithi028.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-02-21T23:01:29.962 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-02-21T23:01:29.962 INFO:teuthology.orchestra.run.smithi028.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-02-21T23:01:29.962 INFO:teuthology.orchestra.run.smithi028.stdout: python3-google-auth-1:2.27.0-1.el9.noarch 2024-02-21T23:01:29.962 INFO:teuthology.orchestra.run.smithi028.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-02-21T23:01:29.962 INFO:teuthology.orchestra.run.smithi028.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-02-21T23:01:29.963 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-02-21T23:01:29.963 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-02-21T23:01:29.963 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-02-21T23:01:29.963 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-02-21T23:01:29.963 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-02-21T23:01:29.963 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-02-21T23:01:29.963 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-02-21T23:01:29.963 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-02-21T23:01:29.963 INFO:teuthology.orchestra.run.smithi028.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-02-21T23:01:29.963 INFO:teuthology.orchestra.run.smithi028.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-02-21T23:01:29.963 INFO:teuthology.orchestra.run.smithi028.stdout: python3-mako-1.1.4-6.el9.noarch 2024-02-21T23:01:29.964 INFO:teuthology.orchestra.run.smithi028.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-02-21T23:01:29.964 INFO:teuthology.orchestra.run.smithi028.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-02-21T23:01:29.964 INFO:teuthology.orchestra.run.smithi028.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-02-21T23:01:29.964 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-02-21T23:01:29.964 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-02-21T23:01:29.964 INFO:teuthology.orchestra.run.smithi028.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-02-21T23:01:29.964 INFO:teuthology.orchestra.run.smithi028.stdout: python3-packaging-20.9-5.el9.noarch 2024-02-21T23:01:29.964 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-02-21T23:01:29.964 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ply-3.11-14.el9.noarch 2024-02-21T23:01:29.964 INFO:teuthology.orchestra.run.smithi028.stdout: python3-portend-3.1.0-2.el9.noarch 2024-02-21T23:01:29.964 INFO:teuthology.orchestra.run.smithi028.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-02-21T23:01:29.964 INFO:teuthology.orchestra.run.smithi028.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-02-21T23:01:29.964 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-02-21T23:01:29.965 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-02-21T23:01:29.965 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-02-21T23:01:29.965 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pycparser-2.20-6.el9.noarch 2024-02-21T23:01:29.965 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-02-21T23:01:29.965 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pytz-2021.1-5.el9.noarch 2024-02-21T23:01:29.965 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rados-2:19.0.0-1643.g3bfc837d.el9.x86_64 2024-02-21T23:01:29.965 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rbd-2:19.0.0-1643.g3bfc837d.el9.x86_64 2024-02-21T23:01:29.965 INFO:teuthology.orchestra.run.smithi028.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-02-21T23:01:29.965 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests-2.25.1-8.el9.noarch 2024-02-21T23:01:29.965 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-02-21T23:01:29.965 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rgw-2:19.0.0-1643.g3bfc837d.el9.x86_64 2024-02-21T23:01:29.965 INFO:teuthology.orchestra.run.smithi028.stdout: python3-routes-2.5.1-5.el9.noarch 2024-02-21T23:01:29.965 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rsa-4.9-2.el9.noarch 2024-02-21T23:01:29.966 INFO:teuthology.orchestra.run.smithi028.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-02-21T23:01:29.966 INFO:teuthology.orchestra.run.smithi028.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-02-21T23:01:29.966 INFO:teuthology.orchestra.run.smithi028.stdout: python3-toml-0.10.2-6.el9.noarch 2024-02-21T23:01:29.966 INFO:teuthology.orchestra.run.smithi028.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-02-21T23:01:29.966 INFO:teuthology.orchestra.run.smithi028.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-02-21T23:01:29.966 INFO:teuthology.orchestra.run.smithi028.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-02-21T23:01:29.966 INFO:teuthology.orchestra.run.smithi028.stdout: python3-webob-1.8.7-6.el9.noarch 2024-02-21T23:01:29.966 INFO:teuthology.orchestra.run.smithi028.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-02-21T23:01:29.966 INFO:teuthology.orchestra.run.smithi028.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-02-21T23:01:29.966 INFO:teuthology.orchestra.run.smithi028.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-02-21T23:01:29.966 INFO:teuthology.orchestra.run.smithi028.stdout: qatlib-23.11.0-1.el9.x86_64 2024-02-21T23:01:29.966 INFO:teuthology.orchestra.run.smithi028.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-02-21T23:01:29.967 INFO:teuthology.orchestra.run.smithi028.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-02-21T23:01:29.967 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-fuse-2:19.0.0-1643.g3bfc837d.el9.x86_64 2024-02-21T23:01:29.967 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-mirror-2:19.0.0-1643.g3bfc837d.el9.x86_64 2024-02-21T23:01:29.967 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-nbd-2:19.0.0-1643.g3bfc837d.el9.x86_64 2024-02-21T23:01:29.967 INFO:teuthology.orchestra.run.smithi028.stdout: re2-1:20211101-3.el9.x86_64 2024-02-21T23:01:29.967 INFO:teuthology.orchestra.run.smithi028.stdout: socat-1.7.4.1-5.el9.x86_64 2024-02-21T23:01:29.967 INFO:teuthology.orchestra.run.smithi028.stdout: thrift-0.15.0-2.el9.x86_64 2024-02-21T23:01:29.967 INFO:teuthology.orchestra.run.smithi028.stdout: uuid-1.6.2-55.el9.x86_64 2024-02-21T23:01:29.967 INFO:teuthology.orchestra.run.smithi028.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-02-21T23:01:29.967 INFO:teuthology.orchestra.run.smithi028.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-02-21T23:01:29.968 INFO:teuthology.orchestra.run.smithi028.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-02-21T23:01:29.968 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:01:29.968 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-02-21T23:01:30.612 DEBUG:teuthology.parallel:result is None 2024-02-21T23:01:30.946 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 182/182 2024-02-21T23:01:30.946 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-02-21T23:01:30.946 INFO:teuthology.orchestra.run.smithi081.stdout:Upgraded: 2024-02-21T23:01:30.946 INFO:teuthology.orchestra.run.smithi081.stdout: NetworkManager-1:1.45.91-1.el9.x86_64 2024-02-21T23:01:30.946 INFO:teuthology.orchestra.run.smithi081.stdout: NetworkManager-libnm-1:1.45.91-1.el9.x86_64 2024-02-21T23:01:30.946 INFO:teuthology.orchestra.run.smithi081.stdout: NetworkManager-team-1:1.45.91-1.el9.x86_64 2024-02-21T23:01:30.946 INFO:teuthology.orchestra.run.smithi081.stdout: NetworkManager-tui-1:1.45.91-1.el9.x86_64 2024-02-21T23:01:30.946 INFO:teuthology.orchestra.run.smithi081.stdout: acl-2.3.1-4.el9.x86_64 2024-02-21T23:01:30.946 INFO:teuthology.orchestra.run.smithi081.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-02-21T23:01:30.947 INFO:teuthology.orchestra.run.smithi081.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-02-21T23:01:30.947 INFO:teuthology.orchestra.run.smithi081.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-02-21T23:01:30.947 INFO:teuthology.orchestra.run.smithi081.stdout: libacl-2.3.1-4.el9.x86_64 2024-02-21T23:01:30.947 INFO:teuthology.orchestra.run.smithi081.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-02-21T23:01:30.947 INFO:teuthology.orchestra.run.smithi081.stdout: librados2-2:19.0.0-1643.g3bfc837d.el9.x86_64 2024-02-21T23:01:30.947 INFO:teuthology.orchestra.run.smithi081.stdout: librbd1-2:19.0.0-1643.g3bfc837d.el9.x86_64 2024-02-21T23:01:30.947 INFO:teuthology.orchestra.run.smithi081.stdout: openssl-1:3.0.7-25.el9.x86_64 2024-02-21T23:01:30.947 INFO:teuthology.orchestra.run.smithi081.stdout: openssl-libs-1:3.0.7-25.el9.x86_64 2024-02-21T23:01:30.947 INFO:teuthology.orchestra.run.smithi081.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-02-21T23:01:30.947 INFO:teuthology.orchestra.run.smithi081.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-02-21T23:01:30.947 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-02-21T23:01:30.947 INFO:teuthology.orchestra.run.smithi081.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-02-21T23:01:30.947 INFO:teuthology.orchestra.run.smithi081.stdout: automake-1.16.2-8.el9.noarch 2024-02-21T23:01:30.948 INFO:teuthology.orchestra.run.smithi081.stdout: bison-3.7.4-5.el9.x86_64 2024-02-21T23:01:30.948 INFO:teuthology.orchestra.run.smithi081.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-02-21T23:01:30.948 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-2:19.0.0-1643.g3bfc837d.el9.x86_64 2024-02-21T23:01:30.948 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-base-2:19.0.0-1643.g3bfc837d.el9.x86_64 2024-02-21T23:01:30.948 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-common-2:19.0.0-1643.g3bfc837d.el9.x86_64 2024-02-21T23:01:30.948 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-fuse-2:19.0.0-1643.g3bfc837d.el9.x86_64 2024-02-21T23:01:30.948 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-grafana-dashboards-2:19.0.0-1643.g3bfc837d.el9.noarch 2024-02-21T23:01:30.948 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-immutable-object-cache-2:19.0.0-1643.g3bfc837d.el9.x86_64 2024-02-21T23:01:30.948 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mds-2:19.0.0-1643.g3bfc837d.el9.x86_64 2024-02-21T23:01:30.948 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-2:19.0.0-1643.g3bfc837d.el9.x86_64 2024-02-21T23:01:30.948 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-cephadm-2:19.0.0-1643.g3bfc837d.el9.noarch 2024-02-21T23:01:30.948 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-dashboard-2:19.0.0-1643.g3bfc837d.el9.noarch 2024-02-21T23:01:30.948 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1643.g3bfc837d.el9.noarch 2024-02-21T23:01:30.949 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-modules-core-2:19.0.0-1643.g3bfc837d.el9.noarch 2024-02-21T23:01:30.949 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-rook-2:19.0.0-1643.g3bfc837d.el9.noarch 2024-02-21T23:01:30.949 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mon-2:19.0.0-1643.g3bfc837d.el9.x86_64 2024-02-21T23:01:30.949 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-osd-2:19.0.0-1643.g3bfc837d.el9.x86_64 2024-02-21T23:01:30.949 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-prometheus-alerts-2:19.0.0-1643.g3bfc837d.el9.noarch 2024-02-21T23:01:30.949 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-radosgw-2:19.0.0-1643.g3bfc837d.el9.x86_64 2024-02-21T23:01:30.949 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-selinux-2:19.0.0-1643.g3bfc837d.el9.x86_64 2024-02-21T23:01:30.949 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-test-2:19.0.0-1643.g3bfc837d.el9.x86_64 2024-02-21T23:01:30.949 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-volume-2:19.0.0-1643.g3bfc837d.el9.noarch 2024-02-21T23:01:30.949 INFO:teuthology.orchestra.run.smithi081.stdout: cephadm-2:19.0.0-1643.g3bfc837d.el9.noarch 2024-02-21T23:01:30.949 INFO:teuthology.orchestra.run.smithi081.stdout: cephfs-mirror-2:19.0.0-1643.g3bfc837d.el9.x86_64 2024-02-21T23:01:30.949 INFO:teuthology.orchestra.run.smithi081.stdout: cephfs-shell-2:19.0.0-1643.g3bfc837d.el9.x86_64 2024-02-21T23:01:30.949 INFO:teuthology.orchestra.run.smithi081.stdout: cephfs-top-2:19.0.0-1643.g3bfc837d.el9.noarch 2024-02-21T23:01:30.950 INFO:teuthology.orchestra.run.smithi081.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-02-21T23:01:30.950 INFO:teuthology.orchestra.run.smithi081.stdout: flex-2.6.4-9.el9.x86_64 2024-02-21T23:01:30.950 INFO:teuthology.orchestra.run.smithi081.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-02-21T23:01:30.950 INFO:teuthology.orchestra.run.smithi081.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-02-21T23:01:30.950 INFO:teuthology.orchestra.run.smithi081.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-02-21T23:01:30.950 INFO:teuthology.orchestra.run.smithi081.stdout: fuse-2.9.9-15.el9.x86_64 2024-02-21T23:01:30.950 INFO:teuthology.orchestra.run.smithi081.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-02-21T23:01:30.950 INFO:teuthology.orchestra.run.smithi081.stdout: grpc-data-1.46.7-10.el9.noarch 2024-02-21T23:01:30.950 INFO:teuthology.orchestra.run.smithi081.stdout: jq-1.6-16.el9.x86_64 2024-02-21T23:01:30.950 INFO:teuthology.orchestra.run.smithi081.stdout: libaio-devel-0.3.112-1.el8.x86_64 2024-02-21T23:01:30.950 INFO:teuthology.orchestra.run.smithi081.stdout: libarrow-9.0.0-10.el9.x86_64 2024-02-21T23:01:30.950 INFO:teuthology.orchestra.run.smithi081.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-02-21T23:01:30.950 INFO:teuthology.orchestra.run.smithi081.stdout: libcephfs-devel-2:19.0.0-1643.g3bfc837d.el9.x86_64 2024-02-21T23:01:30.950 INFO:teuthology.orchestra.run.smithi081.stdout: libcephfs2-2:19.0.0-1643.g3bfc837d.el9.x86_64 2024-02-21T23:01:30.951 INFO:teuthology.orchestra.run.smithi081.stdout: libcephsqlite-2:19.0.0-1643.g3bfc837d.el9.x86_64 2024-02-21T23:01:30.951 INFO:teuthology.orchestra.run.smithi081.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-02-21T23:01:30.951 INFO:teuthology.orchestra.run.smithi081.stdout: liboath-2.6.7-2.el9.x86_64 2024-02-21T23:01:30.951 INFO:teuthology.orchestra.run.smithi081.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-02-21T23:01:30.951 INFO:teuthology.orchestra.run.smithi081.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-02-21T23:01:30.951 INFO:teuthology.orchestra.run.smithi081.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-02-21T23:01:30.951 INFO:teuthology.orchestra.run.smithi081.stdout: librados-devel-2:19.0.0-1643.g3bfc837d.el9.x86_64 2024-02-21T23:01:30.951 INFO:teuthology.orchestra.run.smithi081.stdout: libradosstriper1-2:19.0.0-1643.g3bfc837d.el9.x86_64 2024-02-21T23:01:30.951 INFO:teuthology.orchestra.run.smithi081.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-02-21T23:01:30.951 INFO:teuthology.orchestra.run.smithi081.stdout: librgw2-2:19.0.0-1643.g3bfc837d.el9.x86_64 2024-02-21T23:01:30.951 INFO:teuthology.orchestra.run.smithi081.stdout: libtool-2.4.6-45.el9.x86_64 2024-02-21T23:01:30.951 INFO:teuthology.orchestra.run.smithi081.stdout: libunwind-1.6.2-1.el9.x86_64 2024-02-21T23:01:30.952 INFO:teuthology.orchestra.run.smithi081.stdout: libuuid-devel-2.37.4-15.el9.x86_64 2024-02-21T23:01:30.952 INFO:teuthology.orchestra.run.smithi081.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-02-21T23:01:30.952 INFO:teuthology.orchestra.run.smithi081.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-02-21T23:01:30.952 INFO:teuthology.orchestra.run.smithi081.stdout: lua-5.4.4-4.el9.x86_64 2024-02-21T23:01:30.952 INFO:teuthology.orchestra.run.smithi081.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-02-21T23:01:30.952 INFO:teuthology.orchestra.run.smithi081.stdout: luarocks-3.9.2-1.el9.noarch 2024-02-21T23:01:30.952 INFO:teuthology.orchestra.run.smithi081.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-02-21T23:01:30.952 INFO:teuthology.orchestra.run.smithi081.stdout: openblas-0.3.21-2.el9.x86_64 2024-02-21T23:01:30.952 INFO:teuthology.orchestra.run.smithi081.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-02-21T23:01:30.952 INFO:teuthology.orchestra.run.smithi081.stdout: openssl-devel-1:3.0.7-25.el9.x86_64 2024-02-21T23:01:30.952 INFO:teuthology.orchestra.run.smithi081.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-02-21T23:01:30.952 INFO:teuthology.orchestra.run.smithi081.stdout: postgresql-13.11-1.el9.x86_64 2024-02-21T23:01:30.953 INFO:teuthology.orchestra.run.smithi081.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-02-21T23:01:30.953 INFO:teuthology.orchestra.run.smithi081.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-02-21T23:01:30.953 INFO:teuthology.orchestra.run.smithi081.stdout: postgresql-server-13.11-1.el9.x86_64 2024-02-21T23:01:30.953 INFO:teuthology.orchestra.run.smithi081.stdout: protobuf-3.14.0-13.el9.x86_64 2024-02-21T23:01:30.953 INFO:teuthology.orchestra.run.smithi081.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-02-21T23:01:30.953 INFO:teuthology.orchestra.run.smithi081.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-02-21T23:01:30.953 INFO:teuthology.orchestra.run.smithi081.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-02-21T23:01:30.953 INFO:teuthology.orchestra.run.smithi081.stdout: python3-babel-2.9.1-2.el9.noarch 2024-02-21T23:01:30.953 INFO:teuthology.orchestra.run.smithi081.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-02-21T23:01:30.953 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-02-21T23:01:30.953 INFO:teuthology.orchestra.run.smithi081.stdout: python3-ceph-argparse-2:19.0.0-1643.g3bfc837d.el9.x86_64 2024-02-21T23:01:30.953 INFO:teuthology.orchestra.run.smithi081.stdout: python3-ceph-common-2:19.0.0-1643.g3bfc837d.el9.x86_64 2024-02-21T23:01:30.953 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cephfs-2:19.0.0-1643.g3bfc837d.el9.x86_64 2024-02-21T23:01:30.954 INFO:teuthology.orchestra.run.smithi081.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-02-21T23:01:30.954 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-02-21T23:01:30.954 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-02-21T23:01:30.954 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-02-21T23:01:30.954 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-02-21T23:01:30.954 INFO:teuthology.orchestra.run.smithi081.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-02-21T23:01:30.954 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-02-21T23:01:30.954 INFO:teuthology.orchestra.run.smithi081.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-02-21T23:01:30.954 INFO:teuthology.orchestra.run.smithi081.stdout: python3-google-auth-1:2.27.0-1.el9.noarch 2024-02-21T23:01:30.954 INFO:teuthology.orchestra.run.smithi081.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-02-21T23:01:30.954 INFO:teuthology.orchestra.run.smithi081.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-02-21T23:01:30.954 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-02-21T23:01:30.954 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-02-21T23:01:30.954 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-02-21T23:01:30.955 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-02-21T23:01:30.955 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-02-21T23:01:30.955 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-02-21T23:01:30.955 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-02-21T23:01:30.955 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-02-21T23:01:30.955 INFO:teuthology.orchestra.run.smithi081.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-02-21T23:01:30.955 INFO:teuthology.orchestra.run.smithi081.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-02-21T23:01:30.955 INFO:teuthology.orchestra.run.smithi081.stdout: python3-mako-1.1.4-6.el9.noarch 2024-02-21T23:01:30.955 INFO:teuthology.orchestra.run.smithi081.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-02-21T23:01:30.955 INFO:teuthology.orchestra.run.smithi081.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-02-21T23:01:30.955 INFO:teuthology.orchestra.run.smithi081.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-02-21T23:01:30.955 INFO:teuthology.orchestra.run.smithi081.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-02-21T23:01:30.955 INFO:teuthology.orchestra.run.smithi081.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-02-21T23:01:30.956 INFO:teuthology.orchestra.run.smithi081.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-02-21T23:01:30.956 INFO:teuthology.orchestra.run.smithi081.stdout: python3-packaging-20.9-5.el9.noarch 2024-02-21T23:01:30.956 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-02-21T23:01:30.956 INFO:teuthology.orchestra.run.smithi081.stdout: python3-ply-3.11-14.el9.noarch 2024-02-21T23:01:30.956 INFO:teuthology.orchestra.run.smithi081.stdout: python3-portend-3.1.0-2.el9.noarch 2024-02-21T23:01:30.956 INFO:teuthology.orchestra.run.smithi081.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-02-21T23:01:30.956 INFO:teuthology.orchestra.run.smithi081.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-02-21T23:01:30.956 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-02-21T23:01:30.956 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-02-21T23:01:30.956 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-02-21T23:01:30.956 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pycparser-2.20-6.el9.noarch 2024-02-21T23:01:30.956 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-02-21T23:01:30.956 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pytz-2021.1-5.el9.noarch 2024-02-21T23:01:30.957 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rados-2:19.0.0-1643.g3bfc837d.el9.x86_64 2024-02-21T23:01:30.957 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rbd-2:19.0.0-1643.g3bfc837d.el9.x86_64 2024-02-21T23:01:30.957 INFO:teuthology.orchestra.run.smithi081.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-02-21T23:01:30.957 INFO:teuthology.orchestra.run.smithi081.stdout: python3-requests-2.25.1-8.el9.noarch 2024-02-21T23:01:30.957 INFO:teuthology.orchestra.run.smithi081.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-02-21T23:01:30.957 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rgw-2:19.0.0-1643.g3bfc837d.el9.x86_64 2024-02-21T23:01:30.957 INFO:teuthology.orchestra.run.smithi081.stdout: python3-routes-2.5.1-5.el9.noarch 2024-02-21T23:01:30.957 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rsa-4.9-2.el9.noarch 2024-02-21T23:01:30.957 INFO:teuthology.orchestra.run.smithi081.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-02-21T23:01:30.957 INFO:teuthology.orchestra.run.smithi081.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-02-21T23:01:30.957 INFO:teuthology.orchestra.run.smithi081.stdout: python3-toml-0.10.2-6.el9.noarch 2024-02-21T23:01:30.957 INFO:teuthology.orchestra.run.smithi081.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-02-21T23:01:30.957 INFO:teuthology.orchestra.run.smithi081.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-02-21T23:01:30.958 INFO:teuthology.orchestra.run.smithi081.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-02-21T23:01:30.958 INFO:teuthology.orchestra.run.smithi081.stdout: python3-webob-1.8.7-6.el9.noarch 2024-02-21T23:01:30.958 INFO:teuthology.orchestra.run.smithi081.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-02-21T23:01:30.958 INFO:teuthology.orchestra.run.smithi081.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-02-21T23:01:30.958 INFO:teuthology.orchestra.run.smithi081.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-02-21T23:01:30.958 INFO:teuthology.orchestra.run.smithi081.stdout: qatlib-23.11.0-1.el9.x86_64 2024-02-21T23:01:30.958 INFO:teuthology.orchestra.run.smithi081.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-02-21T23:01:30.958 INFO:teuthology.orchestra.run.smithi081.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-02-21T23:01:30.958 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-fuse-2:19.0.0-1643.g3bfc837d.el9.x86_64 2024-02-21T23:01:30.958 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-mirror-2:19.0.0-1643.g3bfc837d.el9.x86_64 2024-02-21T23:01:30.958 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-nbd-2:19.0.0-1643.g3bfc837d.el9.x86_64 2024-02-21T23:01:30.958 INFO:teuthology.orchestra.run.smithi081.stdout: re2-1:20211101-3.el9.x86_64 2024-02-21T23:01:30.958 INFO:teuthology.orchestra.run.smithi081.stdout: socat-1.7.4.1-5.el9.x86_64 2024-02-21T23:01:30.958 INFO:teuthology.orchestra.run.smithi081.stdout: thrift-0.15.0-2.el9.x86_64 2024-02-21T23:01:30.958 INFO:teuthology.orchestra.run.smithi081.stdout: uuid-1.6.2-55.el9.x86_64 2024-02-21T23:01:30.959 INFO:teuthology.orchestra.run.smithi081.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-02-21T23:01:30.959 INFO:teuthology.orchestra.run.smithi081.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-02-21T23:01:30.959 INFO:teuthology.orchestra.run.smithi081.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-02-21T23:01:30.959 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-02-21T23:01:30.959 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-02-21T23:01:31.623 DEBUG:teuthology.parallel:result is None 2024-02-21T23:01:31.624 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3bfc837d7b45ce8e095d92fa9c342964cea5e95f 2024-02-21T23:01:31.768 DEBUG:teuthology.orchestra.run.smithi028:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-21T23:01:31.828 INFO:teuthology.orchestra.run.smithi028.stdout:19.0.0-1643.g3bfc837d.el9 2024-02-21T23:01:31.828 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1643.g3bfc837d.el9 2024-02-21T23:01:31.829 INFO:teuthology.task.install:The correct ceph version 19.0.0-1643.g3bfc837d is installed. 2024-02-21T23:01:31.831 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3bfc837d7b45ce8e095d92fa9c342964cea5e95f 2024-02-21T23:01:31.969 DEBUG:teuthology.orchestra.run.smithi081:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-21T23:01:32.023 INFO:teuthology.orchestra.run.smithi081.stdout:19.0.0-1643.g3bfc837d.el9 2024-02-21T23:01:32.023 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1643.g3bfc837d.el9 2024-02-21T23:01:32.023 INFO:teuthology.task.install:The correct ceph version 19.0.0-1643.g3bfc837d is installed. 2024-02-21T23:01:32.025 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-02-21T23:01:32.026 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-21T23:01:32.026 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-21T23:01:32.106 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-02-21T23:01:32.106 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-21T23:01:32.180 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-02-21T23:01:32.180 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-21T23:01:32.180 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/daemon-helper 2024-02-21T23:01:32.258 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-21T23:01:32.370 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-02-21T23:01:32.370 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/usr/bin/daemon-helper 2024-02-21T23:01:32.448 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-21T23:01:32.560 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-02-21T23:01:32.560 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-21T23:01:32.560 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-21T23:01:32.644 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-21T23:01:32.758 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-02-21T23:01:32.758 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-21T23:01:32.827 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-21T23:01:32.934 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-02-21T23:01:32.935 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-21T23:01:32.935 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/stdin-killer 2024-02-21T23:01:33.012 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-21T23:01:33.129 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-02-21T23:01:33.129 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/usr/bin/stdin-killer 2024-02-21T23:01:33.199 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-21T23:01:33.309 INFO:teuthology.run_tasks:Running task ceph... 2024-02-21T23:01:33.405 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-02-21T23:01:33.406 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 777 /var/log/ceph 2024-02-21T23:01:33.412 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 777 /var/log/ceph 2024-02-21T23:01:33.483 INFO:tasks.ceph:Disabling ceph logrotate... 2024-02-21T23:01:33.483 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f -- /etc/logrotate.d/ceph 2024-02-21T23:01:33.487 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -f -- /etc/logrotate.d/ceph 2024-02-21T23:01:33.555 INFO:tasks.ceph:Creating extra log directories... 2024-02-21T23:01:33.556 DEBUG:teuthology.orchestra.run.smithi028:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-02-21T23:01:33.560 DEBUG:teuthology.orchestra.run.smithi081:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-02-21T23:01:33.633 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-02-21T23:01:33.634 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-02-21T23:01:33.634 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-02-21T23:01:33.634 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-21T23:01:33.634 DEBUG:teuthology.orchestra.run.smithi028:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-02-21T23:01:33.671 DEBUG:teuthology.orchestra.run.smithi028:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-02-21T23:01:33.883 DEBUG:teuthology.orchestra.run.smithi028:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-02-21T23:01:33.951 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-02-21T23:01:33.951 DEBUG:teuthology.orchestra.run.smithi081:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-02-21T23:01:33.996 DEBUG:teuthology.orchestra.run.smithi081:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-02-21T23:01:34.210 DEBUG:teuthology.orchestra.run.smithi081:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-02-21T23:01:34.290 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-02-21T23:01:34.291 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'Replacing daemon mds', '\\(MDS_INSUFFICIENT_STANDBY\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(MDS_DAMAGE\\)', '\\(FS_DEGRADED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-02-21T23:01:34.291 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/leonidus-2024-02-21_22:31:03-fs-wip-lusov-qdb-distro-default-smithi/7570010', 'branch': 'wip-lusov-qdb', 'description': 'fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}', 'email': None, 'first_in_suite': False, 'job_id': '7570010', 'kernel': {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'}, 'ktype': 'upstream', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'leonidus-2024-02-21_22:31:03-fs-wip-lusov-qdb-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-lusov-qdb'}, 'ceph': {'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'Replacing daemon mds', '\\(MDS_INSUFFICIENT_STANDBY\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(MDS_DAMAGE\\)', '\\(FS_DEGRADED\\)'], 'sha1': '3bfc837d7b45ce8e095d92fa9c342964cea5e95f'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'ceph-fuse': {'disabled': True}, 'install': {'ceph': {'flavor': 'default', 'sha1': '3bfc837d7b45ce8e095d92fa9c342964cea5e95f'}}, 'kclient': {'syntax': 'v2'}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-lusov-qdb', 'sha1': '3bfc837d7b45ce8e095d92fa9c342964cea5e95f'}}, 'owner': 'scheduled_leonidus@teuthology', 'priority': 80, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']], 'seed': 6562, 'sha1': '3bfc837d7b45ce8e095d92fa9c342964cea5e95f', 'sleep_before_teardown': 0, 'subset': '111/10000', 'suite': 'fs', 'suite_branch': 'wip-lusov-qdb', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_3bfc837d7b45ce8e095d92fa9c342964cea5e95f/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '3bfc837d7b45ce8e095d92fa9c342964cea5e95f', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi028.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCUSNVaQl5WeiAT0gHEsvl0ygugXPVSClauMigugq2UJ+dAgMEqmIvsZoNWBzBashC9pE97t51XpgAkk2r59kVg=', 'smithi081.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMCOnqfcgZ62VPscyDatzXpS3IE3CO1tSuvz7xPZ2DxeiVsJvN6mJONmlC48XG7K18AySCjdQ+ADLKEeyfcr+TU='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '3bfc837d7b45ce8e095d92fa9c342964cea5e95f'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'Replacing daemon mds', '\\(MDS_INSUFFICIENT_STANDBY\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(MDS_DAMAGE\\)', '\\(FS_DEGRADED\\)'], 'sha1': '3bfc837d7b45ce8e095d92fa9c342964cea5e95f', 'cluster': 'ceph'}}, {'kclient': None}, {'exec': {'mon.a': ['ceph config set mgr mgr/crash/warn_recent_interval 0']}}, {'cephfs_test_runner': {'fail_on_skip': False, 'modules': ['tasks.cephfs.test_failover']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': 'b1dac5519c57c269ea98a22fb7729016a1d0e4d2', 'timestamp': '2024-02-21_22:31:03', 'tube': 'smithi', 'user': 'leonidus', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-02-21T23:01:34.291 DEBUG:teuthology.orchestra.run.smithi028:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-02-21T23:01:34.296 DEBUG:teuthology.orchestra.run.smithi081:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-02-21T23:01:34.335 DEBUG:teuthology.orchestra.run.smithi028:> sudo install -d -m0777 -- /var/run/ceph 2024-02-21T23:01:34.381 DEBUG:teuthology.orchestra.run.smithi081:> sudo install -d -m0777 -- /var/run/ceph 2024-02-21T23:01:34.459 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-21T23:01:34.459 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/scratch_devs of=/dev/stdout 2024-02-21T23:01:34.496 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-21T23:01:34.496 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_1 2024-02-21T23:01:34.576 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-21T23:01:34.576 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-21T23:01:34.576 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 631 Links: 1 2024-02-21T23:01:34.577 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-21T23:01:34.577 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-02-21T23:01:34.577 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-02-21 23:01:22.837208326 +0000 2024-02-21T23:01:34.577 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-02-21 22:58:14.565902540 +0000 2024-02-21T23:01:34.577 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-02-21 22:58:14.565902540 +0000 2024-02-21T23:01:34.577 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-02-21 22:58:14.565902540 +0000 2024-02-21T23:01:34.577 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-21T23:01:34.687 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-02-21T23:01:34.687 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-02-21T23:01:34.687 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000283345 s, 1.8 MB/s 2024-02-21T23:01:34.690 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-21T23:01:34.732 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_2 2024-02-21T23:01:34.814 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-21T23:01:34.815 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-21T23:01:34.815 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 650 Links: 1 2024-02-21T23:01:34.815 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-21T23:01:34.815 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-02-21T23:01:34.815 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-02-21 23:01:22.837208326 +0000 2024-02-21T23:01:34.815 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-02-21 22:58:15.159885200 +0000 2024-02-21T23:01:34.815 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-02-21 22:58:15.159885200 +0000 2024-02-21T23:01:34.815 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-02-21 22:58:15.159885200 +0000 2024-02-21T23:01:34.816 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-21T23:01:34.955 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-02-21T23:01:34.955 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-02-21T23:01:34.955 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000276953 s, 1.8 MB/s 2024-02-21T23:01:34.959 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-21T23:01:35.014 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_3 2024-02-21T23:01:35.061 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-21T23:01:35.061 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-21T23:01:35.062 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 669 Links: 1 2024-02-21T23:01:35.062 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-21T23:01:35.062 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-02-21T23:01:35.062 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-02-21 23:01:22.838208297 +0000 2024-02-21T23:01:35.062 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-02-21 22:58:15.773867273 +0000 2024-02-21T23:01:35.062 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-02-21 22:58:15.773867273 +0000 2024-02-21T23:01:35.062 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-02-21 22:58:15.773867273 +0000 2024-02-21T23:01:35.062 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-21T23:01:35.146 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-02-21T23:01:35.146 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-02-21T23:01:35.146 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000220116 s, 2.3 MB/s 2024-02-21T23:01:35.150 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-21T23:01:35.208 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_4 2024-02-21T23:01:35.257 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-21T23:01:35.257 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-21T23:01:35.257 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 690 Links: 1 2024-02-21T23:01:35.257 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-21T23:01:35.257 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-02-21T23:01:35.257 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-02-21 23:01:22.839208268 +0000 2024-02-21T23:01:35.257 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-02-21 22:58:16.350850427 +0000 2024-02-21T23:01:35.257 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-02-21 22:58:16.350850427 +0000 2024-02-21T23:01:35.257 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-02-21 22:58:16.350850427 +0000 2024-02-21T23:01:35.258 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-21T23:01:35.397 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-02-21T23:01:35.397 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-02-21T23:01:35.397 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000229744 s, 2.2 MB/s 2024-02-21T23:01:35.401 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-21T23:01:35.462 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-02-21T23:01:35.462 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-02-21T23:01:35.462 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/scratch_devs of=/dev/stdout 2024-02-21T23:01:35.519 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-21T23:01:35.519 DEBUG:teuthology.orchestra.run.smithi081:> stat /dev/vg_nvme/lv_1 2024-02-21T23:01:35.571 INFO:teuthology.orchestra.run.smithi081.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-21T23:01:35.571 INFO:teuthology.orchestra.run.smithi081.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-21T23:01:35.571 INFO:teuthology.orchestra.run.smithi081.stdout:Device: 5h/5d Inode: 636 Links: 1 2024-02-21T23:01:35.571 INFO:teuthology.orchestra.run.smithi081.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-21T23:01:35.571 INFO:teuthology.orchestra.run.smithi081.stdout:Context: system_u:object_r:device_t:s0 2024-02-21T23:01:35.571 INFO:teuthology.orchestra.run.smithi081.stdout:Access: 2024-02-21 23:01:23.907058126 +0000 2024-02-21T23:01:35.572 INFO:teuthology.orchestra.run.smithi081.stdout:Modify: 2024-02-21 22:58:06.482395174 +0000 2024-02-21T23:01:35.572 INFO:teuthology.orchestra.run.smithi081.stdout:Change: 2024-02-21 22:58:06.482395174 +0000 2024-02-21T23:01:35.572 INFO:teuthology.orchestra.run.smithi081.stdout: Birth: 2024-02-21 22:58:06.482395174 +0000 2024-02-21T23:01:35.572 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-21T23:01:35.654 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records in 2024-02-21T23:01:35.654 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records out 2024-02-21T23:01:35.654 INFO:teuthology.orchestra.run.smithi081.stderr:512 bytes copied, 0.000392042 s, 1.3 MB/s 2024-02-21T23:01:35.657 DEBUG:teuthology.orchestra.run.smithi081:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-21T23:01:35.711 DEBUG:teuthology.orchestra.run.smithi081:> stat /dev/vg_nvme/lv_2 2024-02-21T23:01:35.756 INFO:teuthology.orchestra.run.smithi081.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-21T23:01:35.756 INFO:teuthology.orchestra.run.smithi081.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-21T23:01:35.756 INFO:teuthology.orchestra.run.smithi081.stdout:Device: 5h/5d Inode: 653 Links: 1 2024-02-21T23:01:35.756 INFO:teuthology.orchestra.run.smithi081.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-21T23:01:35.757 INFO:teuthology.orchestra.run.smithi081.stdout:Context: system_u:object_r:device_t:s0 2024-02-21T23:01:35.757 INFO:teuthology.orchestra.run.smithi081.stdout:Access: 2024-02-21 23:01:23.908058107 +0000 2024-02-21T23:01:35.757 INFO:teuthology.orchestra.run.smithi081.stdout:Modify: 2024-02-21 22:58:07.066383950 +0000 2024-02-21T23:01:35.757 INFO:teuthology.orchestra.run.smithi081.stdout:Change: 2024-02-21 22:58:07.066383950 +0000 2024-02-21T23:01:35.757 INFO:teuthology.orchestra.run.smithi081.stdout: Birth: 2024-02-21 22:58:07.066383950 +0000 2024-02-21T23:01:35.758 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-21T23:01:35.886 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records in 2024-02-21T23:01:35.886 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records out 2024-02-21T23:01:35.886 INFO:teuthology.orchestra.run.smithi081.stderr:512 bytes copied, 0.00034487 s, 1.5 MB/s 2024-02-21T23:01:35.890 DEBUG:teuthology.orchestra.run.smithi081:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-21T23:01:35.941 DEBUG:teuthology.orchestra.run.smithi081:> stat /dev/vg_nvme/lv_3 2024-02-21T23:01:35.985 INFO:teuthology.orchestra.run.smithi081.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-21T23:01:35.985 INFO:teuthology.orchestra.run.smithi081.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-21T23:01:35.985 INFO:teuthology.orchestra.run.smithi081.stdout:Device: 5h/5d Inode: 673 Links: 1 2024-02-21T23:01:35.986 INFO:teuthology.orchestra.run.smithi081.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-21T23:01:35.986 INFO:teuthology.orchestra.run.smithi081.stdout:Context: system_u:object_r:device_t:s0 2024-02-21T23:01:35.986 INFO:teuthology.orchestra.run.smithi081.stdout:Access: 2024-02-21 23:01:23.908058107 +0000 2024-02-21T23:01:35.986 INFO:teuthology.orchestra.run.smithi081.stdout:Modify: 2024-02-21 22:58:07.609373513 +0000 2024-02-21T23:01:35.986 INFO:teuthology.orchestra.run.smithi081.stdout:Change: 2024-02-21 22:58:07.609373513 +0000 2024-02-21T23:01:35.986 INFO:teuthology.orchestra.run.smithi081.stdout: Birth: 2024-02-21 22:58:07.609373513 +0000 2024-02-21T23:01:35.986 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-21T23:01:36.116 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records in 2024-02-21T23:01:36.117 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records out 2024-02-21T23:01:36.117 INFO:teuthology.orchestra.run.smithi081.stderr:512 bytes copied, 0.000341162 s, 1.5 MB/s 2024-02-21T23:01:36.121 DEBUG:teuthology.orchestra.run.smithi081:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-21T23:01:36.174 DEBUG:teuthology.orchestra.run.smithi081:> stat /dev/vg_nvme/lv_4 2024-02-21T23:01:36.222 INFO:teuthology.orchestra.run.smithi081.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-21T23:01:36.222 INFO:teuthology.orchestra.run.smithi081.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-21T23:01:36.222 INFO:teuthology.orchestra.run.smithi081.stdout:Device: 5h/5d Inode: 694 Links: 1 2024-02-21T23:01:36.222 INFO:teuthology.orchestra.run.smithi081.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-21T23:01:36.222 INFO:teuthology.orchestra.run.smithi081.stdout:Context: system_u:object_r:device_t:s0 2024-02-21T23:01:36.222 INFO:teuthology.orchestra.run.smithi081.stdout:Access: 2024-02-21 23:01:23.909058088 +0000 2024-02-21T23:01:36.223 INFO:teuthology.orchestra.run.smithi081.stdout:Modify: 2024-02-21 22:58:08.191362327 +0000 2024-02-21T23:01:36.223 INFO:teuthology.orchestra.run.smithi081.stdout:Change: 2024-02-21 22:58:08.191362327 +0000 2024-02-21T23:01:36.223 INFO:teuthology.orchestra.run.smithi081.stdout: Birth: 2024-02-21 22:58:08.191362327 +0000 2024-02-21T23:01:36.223 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-21T23:01:36.305 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records in 2024-02-21T23:01:36.305 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records out 2024-02-21T23:01:36.305 INFO:teuthology.orchestra.run.smithi081.stderr:512 bytes copied, 0.000306216 s, 1.7 MB/s 2024-02-21T23:01:36.309 DEBUG:teuthology.orchestra.run.smithi081:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-21T23:01:36.364 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-02-21T23:01:36.364 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi028.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi081.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-02-21T23:01:36.364 INFO:tasks.ceph:Generating config... 2024-02-21T23:01:36.365 INFO:tasks.ceph:[client] client mount timeout = 600 2024-02-21T23:01:36.366 INFO:tasks.ceph:[client] debug client = 20 2024-02-21T23:01:36.366 INFO:tasks.ceph:[client] debug ms = 1 2024-02-21T23:01:36.366 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-02-21T23:01:36.366 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-02-21T23:01:36.366 INFO:tasks.ceph:[global] ms die on skipped message = False 2024-02-21T23:01:36.366 INFO:tasks.ceph:[mds] debug mds = 20 2024-02-21T23:01:36.366 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-02-21T23:01:36.366 INFO:tasks.ceph:[mds] debug ms = 1 2024-02-21T23:01:36.366 INFO:tasks.ceph:[mds] mds debug frag = True 2024-02-21T23:01:36.366 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-02-21T23:01:36.366 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-02-21T23:01:36.366 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-02-21T23:01:36.366 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-02-21T23:01:36.367 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-02-21T23:01:36.367 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-02-21T23:01:36.367 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-02-21T23:01:36.367 INFO:tasks.ceph:[mgr] debug ms = 1 2024-02-21T23:01:36.367 INFO:tasks.ceph:[mon] debug mon = 20 2024-02-21T23:01:36.367 INFO:tasks.ceph:[mon] debug ms = 1 2024-02-21T23:01:36.367 INFO:tasks.ceph:[mon] debug paxos = 20 2024-02-21T23:01:36.367 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-02-21T23:01:36.367 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-02-21T23:01:36.367 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-02-21T23:01:36.367 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-02-21T23:01:36.367 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-02-21T23:01:36.367 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-02-21T23:01:36.367 INFO:tasks.ceph:[osd] debug ms = 1 2024-02-21T23:01:36.367 INFO:tasks.ceph:[osd] debug osd = 20 2024-02-21T23:01:36.367 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-02-21T23:01:36.368 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-02-21T23:01:36.368 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-02-21T23:01:36.368 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-02-21T23:01:36.368 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-02-21T23:01:36.368 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-02-21T23:01:36.368 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-02-21T23:01:36.368 INFO:tasks.ceph:Setting up mon.a... 2024-02-21T23:01:36.368 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-02-21T23:01:36.549 INFO:teuthology.orchestra.run.smithi028.stdout:creating /etc/ceph/ceph.keyring 2024-02-21T23:01:36.559 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-02-21T23:01:36.702 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-02-21T23:01:36.803 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.28'), ('mon.b', '172.21.15.81'), ('mon.c', '[v2:172.21.15.81:3301,v1:172.21.15.81:6790]')] 2024-02-21T23:01:36.804 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.28,172.21.15.81,[v2:172.21.15.81:3301,v1:172.21.15.81:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-02-21T23:01:36.805 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-21T23:01:36.805 DEBUG:teuthology.orchestra.run.smithi028:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-02-21T23:01:36.860 DEBUG:teuthology.orchestra.run.smithi028:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.28 --add b 172.21.15.81 --addv c '[v2:172.21.15.81:3301,v1:172.21.15.81:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-02-21T23:01:37.014 INFO:teuthology.orchestra.run.smithi028.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-02-21T23:01:37.015 INFO:teuthology.orchestra.run.smithi028.stdout:monmaptool: generated fsid d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac 2024-02-21T23:01:37.015 INFO:teuthology.orchestra.run.smithi028.stdout:setting min_mon_release = quincy 2024-02-21T23:01:37.015 INFO:teuthology.orchestra.run.smithi028.stdout:epoch 0 2024-02-21T23:01:37.015 INFO:teuthology.orchestra.run.smithi028.stdout:fsid d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac 2024-02-21T23:01:37.015 INFO:teuthology.orchestra.run.smithi028.stdout:last_changed 2024-02-21T23:01:37.026462+0000 2024-02-21T23:01:37.015 INFO:teuthology.orchestra.run.smithi028.stdout:created 2024-02-21T23:01:37.026462+0000 2024-02-21T23:01:37.015 INFO:teuthology.orchestra.run.smithi028.stdout:min_mon_release 17 (quincy) 2024-02-21T23:01:37.015 INFO:teuthology.orchestra.run.smithi028.stdout:election_strategy: 1 2024-02-21T23:01:37.015 INFO:teuthology.orchestra.run.smithi028.stdout:0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-02-21T23:01:37.015 INFO:teuthology.orchestra.run.smithi028.stdout:1: [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] mon.b 2024-02-21T23:01:37.015 INFO:teuthology.orchestra.run.smithi028.stdout:2: [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] mon.c 2024-02-21T23:01:37.016 INFO:teuthology.orchestra.run.smithi028.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-02-21T23:01:37.018 DEBUG:teuthology.orchestra.run.smithi028:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-02-21T23:01:37.068 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac... 2024-02-21T23:01:37.069 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-02-21T23:01:37.074 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-02-21T23:01:37.246 INFO:teuthology.orchestra.run.smithi028.stdout:[global] 2024-02-21T23:01:37.246 INFO:teuthology.orchestra.run.smithi028.stdout: chdir = "" 2024-02-21T23:01:37.247 INFO:teuthology.orchestra.run.smithi028.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-02-21T23:01:37.247 INFO:teuthology.orchestra.run.smithi028.stdout: auth supported = cephx 2024-02-21T23:01:37.247 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:01:37.247 INFO:teuthology.orchestra.run.smithi028.stdout: filestore xattr use omap = true 2024-02-21T23:01:37.247 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:01:37.247 INFO:teuthology.orchestra.run.smithi028.stdout: mon clock drift allowed = 1.000 2024-02-21T23:01:37.247 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:01:37.247 INFO:teuthology.orchestra.run.smithi028.stdout: osd crush chooseleaf type = 0 2024-02-21T23:01:37.247 INFO:teuthology.orchestra.run.smithi028.stdout: auth debug = true 2024-02-21T23:01:37.247 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:01:37.247 INFO:teuthology.orchestra.run.smithi028.stdout: ms die on old message = true 2024-02-21T23:01:37.248 INFO:teuthology.orchestra.run.smithi028.stdout: ms die on bug = true 2024-02-21T23:01:37.248 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:01:37.248 INFO:teuthology.orchestra.run.smithi028.stdout: mon max pg per osd = 10000 # >= luminous 2024-02-21T23:01:37.248 INFO:teuthology.orchestra.run.smithi028.stdout: mon pg warn max object skew = 0 2024-02-21T23:01:37.248 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:01:37.248 INFO:teuthology.orchestra.run.smithi028.stdout: # disable pg_autoscaler by default for new pools 2024-02-21T23:01:37.248 INFO:teuthology.orchestra.run.smithi028.stdout: osd_pool_default_pg_autoscale_mode = off 2024-02-21T23:01:37.248 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:01:37.248 INFO:teuthology.orchestra.run.smithi028.stdout: osd pool default size = 2 2024-02-21T23:01:37.248 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:01:37.248 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd allow primary affinity = true 2024-02-21T23:01:37.249 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd allow pg remap = true 2024-02-21T23:01:37.249 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on legacy crush tunables = false 2024-02-21T23:01:37.249 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on crush straw calc version zero = false 2024-02-21T23:01:37.249 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on no sortbitwise = false 2024-02-21T23:01:37.249 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on osd down out interval zero = false 2024-02-21T23:01:37.249 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on too few osds = false 2024-02-21T23:01:37.249 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-02-21T23:01:37.249 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_pool_no_redundancy = false 2024-02-21T23:01:37.249 INFO:teuthology.orchestra.run.smithi028.stdout: mon_allow_pool_size_one = true 2024-02-21T23:01:37.249 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:01:37.249 INFO:teuthology.orchestra.run.smithi028.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-02-21T23:01:37.249 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:01:37.250 INFO:teuthology.orchestra.run.smithi028.stdout: osd default data pool replay window = 5 2024-02-21T23:01:37.250 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:01:37.250 INFO:teuthology.orchestra.run.smithi028.stdout: mon allow pool delete = true 2024-02-21T23:01:37.250 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:01:37.250 INFO:teuthology.orchestra.run.smithi028.stdout: mon cluster log file level = debug 2024-02-21T23:01:37.250 INFO:teuthology.orchestra.run.smithi028.stdout: debug asserts on shutdown = true 2024-02-21T23:01:37.250 INFO:teuthology.orchestra.run.smithi028.stdout: mon health detail to clog = false 2024-02-21T23:01:37.250 INFO:teuthology.orchestra.run.smithi028.stdout: mon host = "172.21.15.28,172.21.15.81,[v2:172.21.15.81:3301,v1:172.21.15.81:6790]" 2024-02-21T23:01:37.250 INFO:teuthology.orchestra.run.smithi028.stdout: ms die on skipped message = False 2024-02-21T23:01:37.250 INFO:teuthology.orchestra.run.smithi028.stdout: fsid = d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac 2024-02-21T23:01:37.250 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:01:37.250 INFO:teuthology.orchestra.run.smithi028.stdout:[osd] 2024-02-21T23:01:37.250 INFO:teuthology.orchestra.run.smithi028.stdout: osd journal size = 100 2024-02-21T23:01:37.251 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:01:37.251 INFO:teuthology.orchestra.run.smithi028.stdout: osd scrub load threshold = 5.0 2024-02-21T23:01:37.251 INFO:teuthology.orchestra.run.smithi028.stdout: osd scrub max interval = 600 2024-02-21T23:01:37.251 INFO:teuthology.orchestra.run.smithi028.stdout: osd mclock profile = high_recovery_ops 2024-02-21T23:01:37.251 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:01:37.251 INFO:teuthology.orchestra.run.smithi028.stdout: osd recover clone overlap = true 2024-02-21T23:01:37.251 INFO:teuthology.orchestra.run.smithi028.stdout: osd recovery max chunk = 1048576 2024-02-21T23:01:37.251 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:01:37.251 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug shutdown = true 2024-02-21T23:01:37.251 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug op order = true 2024-02-21T23:01:37.251 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug verify stray on activate = true 2024-02-21T23:01:37.251 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug trim objects = true 2024-02-21T23:01:37.251 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:01:37.251 INFO:teuthology.orchestra.run.smithi028.stdout: osd open classes on start = true 2024-02-21T23:01:37.251 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug pg log writeout = true 2024-02-21T23:01:37.252 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:01:37.252 INFO:teuthology.orchestra.run.smithi028.stdout: osd deep scrub update digest min age = 30 2024-02-21T23:01:37.252 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:01:37.252 INFO:teuthology.orchestra.run.smithi028.stdout: osd map max advance = 10 2024-02-21T23:01:37.252 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:01:37.252 INFO:teuthology.orchestra.run.smithi028.stdout: journal zero on create = true 2024-02-21T23:01:37.252 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:01:37.252 INFO:teuthology.orchestra.run.smithi028.stdout: filestore ondisk finisher threads = 3 2024-02-21T23:01:37.252 INFO:teuthology.orchestra.run.smithi028.stdout: filestore apply finisher threads = 3 2024-02-21T23:01:37.252 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:01:37.252 INFO:teuthology.orchestra.run.smithi028.stdout: bdev debug aio = true 2024-02-21T23:01:37.252 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug misdirected ops = true 2024-02-21T23:01:37.252 INFO:teuthology.orchestra.run.smithi028.stdout: bluestore block size = 96636764160 2024-02-21T23:01:37.252 INFO:teuthology.orchestra.run.smithi028.stdout: bluestore compression mode = aggressive 2024-02-21T23:01:37.253 INFO:teuthology.orchestra.run.smithi028.stdout: bluestore fsck on mount = True 2024-02-21T23:01:37.253 INFO:teuthology.orchestra.run.smithi028.stdout: debug bluefs = 20 2024-02-21T23:01:37.253 INFO:teuthology.orchestra.run.smithi028.stdout: debug bluestore = 20 2024-02-21T23:01:37.253 INFO:teuthology.orchestra.run.smithi028.stdout: debug ms = 1 2024-02-21T23:01:37.253 INFO:teuthology.orchestra.run.smithi028.stdout: debug osd = 20 2024-02-21T23:01:37.253 INFO:teuthology.orchestra.run.smithi028.stdout: debug rocksdb = 10 2024-02-21T23:01:37.253 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd backfillfull_ratio = 0.85 2024-02-21T23:01:37.253 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd full ratio = 0.9 2024-02-21T23:01:37.253 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd nearfull ratio = 0.8 2024-02-21T23:01:37.253 INFO:teuthology.orchestra.run.smithi028.stdout: osd failsafe full ratio = 0.95 2024-02-21T23:01:37.254 INFO:teuthology.orchestra.run.smithi028.stdout: osd objectstore = bluestore 2024-02-21T23:01:37.254 INFO:teuthology.orchestra.run.smithi028.stdout: osd op complaint time = 180 2024-02-21T23:01:37.254 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:01:37.254 INFO:teuthology.orchestra.run.smithi028.stdout:[mgr] 2024-02-21T23:01:37.254 INFO:teuthology.orchestra.run.smithi028.stdout: debug ms = 1 2024-02-21T23:01:37.254 INFO:teuthology.orchestra.run.smithi028.stdout: debug mgr = 20 2024-02-21T23:01:37.254 INFO:teuthology.orchestra.run.smithi028.stdout: debug mon = 20 2024-02-21T23:01:37.254 INFO:teuthology.orchestra.run.smithi028.stdout: debug auth = 20 2024-02-21T23:01:37.254 INFO:teuthology.orchestra.run.smithi028.stdout: mon reweight min pgs per osd = 4 2024-02-21T23:01:37.254 INFO:teuthology.orchestra.run.smithi028.stdout: mon reweight min bytes per osd = 10 2024-02-21T23:01:37.254 INFO:teuthology.orchestra.run.smithi028.stdout: mgr/telemetry/nag = false 2024-02-21T23:01:37.254 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:01:37.254 INFO:teuthology.orchestra.run.smithi028.stdout:[mon] 2024-02-21T23:01:37.254 INFO:teuthology.orchestra.run.smithi028.stdout: debug ms = 1 2024-02-21T23:01:37.255 INFO:teuthology.orchestra.run.smithi028.stdout: debug mon = 20 2024-02-21T23:01:37.255 INFO:teuthology.orchestra.run.smithi028.stdout: debug paxos = 20 2024-02-21T23:01:37.255 INFO:teuthology.orchestra.run.smithi028.stdout: debug auth = 20 2024-02-21T23:01:37.255 INFO:teuthology.orchestra.run.smithi028.stdout: mon data avail warn = 5 2024-02-21T23:01:37.255 INFO:teuthology.orchestra.run.smithi028.stdout: mon mgr mkfs grace = 240 2024-02-21T23:01:37.255 INFO:teuthology.orchestra.run.smithi028.stdout: mon reweight min pgs per osd = 4 2024-02-21T23:01:37.255 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd reporter subtree level = osd 2024-02-21T23:01:37.255 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd prime pg temp = true 2024-02-21T23:01:37.255 INFO:teuthology.orchestra.run.smithi028.stdout: mon reweight min bytes per osd = 10 2024-02-21T23:01:37.255 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:01:37.255 INFO:teuthology.orchestra.run.smithi028.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-02-21T23:01:37.255 INFO:teuthology.orchestra.run.smithi028.stdout: auth mon ticket ttl = 660 # 11m 2024-02-21T23:01:37.255 INFO:teuthology.orchestra.run.smithi028.stdout: auth service ticket ttl = 240 # 4m 2024-02-21T23:01:37.255 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:01:37.255 INFO:teuthology.orchestra.run.smithi028.stdout: # don't complain about insecure global_id in the test suite 2024-02-21T23:01:37.256 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-02-21T23:01:37.256 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-02-21T23:01:37.256 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:01:37.256 INFO:teuthology.orchestra.run.smithi028.stdout: # 1m isn't quite enough 2024-02-21T23:01:37.256 INFO:teuthology.orchestra.run.smithi028.stdout: mon_down_mkfs_grace = 2m 2024-02-21T23:01:37.256 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:01:37.256 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_filestore_osds = false 2024-02-21T23:01:37.256 INFO:teuthology.orchestra.run.smithi028.stdout: mon op complaint time = 120 2024-02-21T23:01:37.256 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:01:37.256 INFO:teuthology.orchestra.run.smithi028.stdout:[client] 2024-02-21T23:01:37.256 INFO:teuthology.orchestra.run.smithi028.stdout: rgw cache enabled = true 2024-02-21T23:01:37.256 INFO:teuthology.orchestra.run.smithi028.stdout: rgw enable ops log = true 2024-02-21T23:01:37.256 INFO:teuthology.orchestra.run.smithi028.stdout: rgw enable usage log = true 2024-02-21T23:01:37.256 INFO:teuthology.orchestra.run.smithi028.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-02-21T23:01:37.257 INFO:teuthology.orchestra.run.smithi028.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-02-21T23:01:37.257 INFO:teuthology.orchestra.run.smithi028.stdout: client mount timeout = 600 2024-02-21T23:01:37.257 INFO:teuthology.orchestra.run.smithi028.stdout: debug client = 20 2024-02-21T23:01:37.257 INFO:teuthology.orchestra.run.smithi028.stdout: debug ms = 1 2024-02-21T23:01:37.257 INFO:teuthology.orchestra.run.smithi028.stdout: rados mon op timeout = 900 2024-02-21T23:01:37.259 INFO:teuthology.orchestra.run.smithi028.stdout: rados osd op timeout = 900 2024-02-21T23:01:37.259 INFO:teuthology.orchestra.run.smithi028.stdout:[mon.a] 2024-02-21T23:01:37.259 INFO:teuthology.orchestra.run.smithi028.stdout:[mon.b] 2024-02-21T23:01:37.259 INFO:teuthology.orchestra.run.smithi028.stdout:[mon.c] 2024-02-21T23:01:37.259 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.a] 2024-02-21T23:01:37.259 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.c] 2024-02-21T23:01:37.259 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.b] 2024-02-21T23:01:37.260 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.d] 2024-02-21T23:01:37.260 INFO:teuthology.orchestra.run.smithi028.stdout:[mds] 2024-02-21T23:01:37.260 INFO:teuthology.orchestra.run.smithi028.stdout: debug mds = 20 2024-02-21T23:01:37.260 INFO:teuthology.orchestra.run.smithi028.stdout: debug mds balancer = 20 2024-02-21T23:01:37.260 INFO:teuthology.orchestra.run.smithi028.stdout: debug ms = 1 2024-02-21T23:01:37.260 INFO:teuthology.orchestra.run.smithi028.stdout: mds debug frag = True 2024-02-21T23:01:37.260 INFO:teuthology.orchestra.run.smithi028.stdout: mds debug scatterstat = True 2024-02-21T23:01:37.260 INFO:teuthology.orchestra.run.smithi028.stdout: mds op complaint time = 180 2024-02-21T23:01:37.260 INFO:teuthology.orchestra.run.smithi028.stdout: mds verify scatter = True 2024-02-21T23:01:37.260 INFO:teuthology.orchestra.run.smithi028.stdout: osd op complaint time = 180 2024-02-21T23:01:37.260 INFO:teuthology.orchestra.run.smithi028.stdout: rados mon op timeout = 900 2024-02-21T23:01:37.260 INFO:teuthology.orchestra.run.smithi028.stdout: rados osd op timeout = 900 2024-02-21T23:01:37.261 INFO:teuthology.orchestra.run.smithi081.stdout:[global] 2024-02-21T23:01:37.261 INFO:teuthology.orchestra.run.smithi081.stdout: chdir = "" 2024-02-21T23:01:37.261 INFO:teuthology.orchestra.run.smithi081.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-02-21T23:01:37.261 INFO:teuthology.orchestra.run.smithi081.stdout: auth supported = cephx 2024-02-21T23:01:37.261 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-02-21T23:01:37.261 INFO:teuthology.orchestra.run.smithi081.stdout: filestore xattr use omap = true 2024-02-21T23:01:37.261 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-02-21T23:01:37.261 INFO:teuthology.orchestra.run.smithi081.stdout: mon clock drift allowed = 1.000 2024-02-21T23:01:37.261 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-02-21T23:01:37.261 INFO:teuthology.orchestra.run.smithi081.stdout: osd crush chooseleaf type = 0 2024-02-21T23:01:37.262 INFO:teuthology.orchestra.run.smithi081.stdout: auth debug = true 2024-02-21T23:01:37.262 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-02-21T23:01:37.262 INFO:teuthology.orchestra.run.smithi081.stdout: ms die on old message = true 2024-02-21T23:01:37.262 INFO:teuthology.orchestra.run.smithi081.stdout: ms die on bug = true 2024-02-21T23:01:37.262 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-02-21T23:01:37.262 INFO:teuthology.orchestra.run.smithi081.stdout: mon max pg per osd = 10000 # >= luminous 2024-02-21T23:01:37.262 INFO:teuthology.orchestra.run.smithi081.stdout: mon pg warn max object skew = 0 2024-02-21T23:01:37.262 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-02-21T23:01:37.262 INFO:teuthology.orchestra.run.smithi081.stdout: # disable pg_autoscaler by default for new pools 2024-02-21T23:01:37.262 INFO:teuthology.orchestra.run.smithi081.stdout: osd_pool_default_pg_autoscale_mode = off 2024-02-21T23:01:37.262 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-02-21T23:01:37.262 INFO:teuthology.orchestra.run.smithi081.stdout: osd pool default size = 2 2024-02-21T23:01:37.262 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-02-21T23:01:37.262 INFO:teuthology.orchestra.run.smithi081.stdout: mon osd allow primary affinity = true 2024-02-21T23:01:37.262 INFO:teuthology.orchestra.run.smithi081.stdout: mon osd allow pg remap = true 2024-02-21T23:01:37.263 INFO:teuthology.orchestra.run.smithi081.stdout: mon warn on legacy crush tunables = false 2024-02-21T23:01:37.263 INFO:teuthology.orchestra.run.smithi081.stdout: mon warn on crush straw calc version zero = false 2024-02-21T23:01:37.263 INFO:teuthology.orchestra.run.smithi081.stdout: mon warn on no sortbitwise = false 2024-02-21T23:01:37.263 INFO:teuthology.orchestra.run.smithi081.stdout: mon warn on osd down out interval zero = false 2024-02-21T23:01:37.263 INFO:teuthology.orchestra.run.smithi081.stdout: mon warn on too few osds = false 2024-02-21T23:01:37.263 INFO:teuthology.orchestra.run.smithi081.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-02-21T23:01:37.263 INFO:teuthology.orchestra.run.smithi081.stdout: mon_warn_on_pool_no_redundancy = false 2024-02-21T23:01:37.263 INFO:teuthology.orchestra.run.smithi081.stdout: mon_allow_pool_size_one = true 2024-02-21T23:01:37.263 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-02-21T23:01:37.263 INFO:teuthology.orchestra.run.smithi081.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-02-21T23:01:37.263 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-02-21T23:01:37.263 INFO:teuthology.orchestra.run.smithi081.stdout: osd default data pool replay window = 5 2024-02-21T23:01:37.263 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-02-21T23:01:37.263 INFO:teuthology.orchestra.run.smithi081.stdout: mon allow pool delete = true 2024-02-21T23:01:37.263 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-02-21T23:01:37.264 INFO:teuthology.orchestra.run.smithi081.stdout: mon cluster log file level = debug 2024-02-21T23:01:37.264 INFO:teuthology.orchestra.run.smithi081.stdout: debug asserts on shutdown = true 2024-02-21T23:01:37.264 INFO:teuthology.orchestra.run.smithi081.stdout: mon health detail to clog = false 2024-02-21T23:01:37.264 INFO:teuthology.orchestra.run.smithi081.stdout: mon host = "172.21.15.28,172.21.15.81,[v2:172.21.15.81:3301,v1:172.21.15.81:6790]" 2024-02-21T23:01:37.264 INFO:teuthology.orchestra.run.smithi081.stdout: ms die on skipped message = False 2024-02-21T23:01:37.264 INFO:teuthology.orchestra.run.smithi081.stdout: fsid = d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac 2024-02-21T23:01:37.264 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-02-21T23:01:37.264 INFO:teuthology.orchestra.run.smithi081.stdout:[osd] 2024-02-21T23:01:37.264 INFO:teuthology.orchestra.run.smithi081.stdout: osd journal size = 100 2024-02-21T23:01:37.264 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-02-21T23:01:37.264 INFO:teuthology.orchestra.run.smithi081.stdout: osd scrub load threshold = 5.0 2024-02-21T23:01:37.264 INFO:teuthology.orchestra.run.smithi081.stdout: osd scrub max interval = 600 2024-02-21T23:01:37.264 INFO:teuthology.orchestra.run.smithi081.stdout: osd mclock profile = high_recovery_ops 2024-02-21T23:01:37.264 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-02-21T23:01:37.265 INFO:teuthology.orchestra.run.smithi081.stdout: osd recover clone overlap = true 2024-02-21T23:01:37.265 INFO:teuthology.orchestra.run.smithi081.stdout: osd recovery max chunk = 1048576 2024-02-21T23:01:37.265 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-02-21T23:01:37.265 INFO:teuthology.orchestra.run.smithi081.stdout: osd debug shutdown = true 2024-02-21T23:01:37.265 INFO:teuthology.orchestra.run.smithi081.stdout: osd debug op order = true 2024-02-21T23:01:37.265 INFO:teuthology.orchestra.run.smithi081.stdout: osd debug verify stray on activate = true 2024-02-21T23:01:37.265 INFO:teuthology.orchestra.run.smithi081.stdout: osd debug trim objects = true 2024-02-21T23:01:37.265 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-02-21T23:01:37.265 INFO:teuthology.orchestra.run.smithi081.stdout: osd open classes on start = true 2024-02-21T23:01:37.265 INFO:teuthology.orchestra.run.smithi081.stdout: osd debug pg log writeout = true 2024-02-21T23:01:37.265 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-02-21T23:01:37.265 INFO:teuthology.orchestra.run.smithi081.stdout: osd deep scrub update digest min age = 30 2024-02-21T23:01:37.265 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-02-21T23:01:37.265 INFO:teuthology.orchestra.run.smithi081.stdout: osd map max advance = 10 2024-02-21T23:01:37.265 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-02-21T23:01:37.266 INFO:teuthology.orchestra.run.smithi081.stdout: journal zero on create = true 2024-02-21T23:01:37.266 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-02-21T23:01:37.266 INFO:teuthology.orchestra.run.smithi081.stdout: filestore ondisk finisher threads = 3 2024-02-21T23:01:37.266 INFO:teuthology.orchestra.run.smithi081.stdout: filestore apply finisher threads = 3 2024-02-21T23:01:37.266 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-02-21T23:01:37.266 INFO:teuthology.orchestra.run.smithi081.stdout: bdev debug aio = true 2024-02-21T23:01:37.266 INFO:teuthology.orchestra.run.smithi081.stdout: osd debug misdirected ops = true 2024-02-21T23:01:37.266 INFO:teuthology.orchestra.run.smithi081.stdout: bluestore block size = 96636764160 2024-02-21T23:01:37.266 INFO:teuthology.orchestra.run.smithi081.stdout: bluestore compression mode = aggressive 2024-02-21T23:01:37.266 INFO:teuthology.orchestra.run.smithi081.stdout: bluestore fsck on mount = True 2024-02-21T23:01:37.266 INFO:teuthology.orchestra.run.smithi081.stdout: debug bluefs = 20 2024-02-21T23:01:37.266 INFO:teuthology.orchestra.run.smithi081.stdout: debug bluestore = 20 2024-02-21T23:01:37.266 INFO:teuthology.orchestra.run.smithi081.stdout: debug ms = 1 2024-02-21T23:01:37.266 INFO:teuthology.orchestra.run.smithi081.stdout: debug osd = 20 2024-02-21T23:01:37.266 INFO:teuthology.orchestra.run.smithi081.stdout: debug rocksdb = 10 2024-02-21T23:01:37.267 INFO:teuthology.orchestra.run.smithi081.stdout: mon osd backfillfull_ratio = 0.85 2024-02-21T23:01:37.267 INFO:teuthology.orchestra.run.smithi081.stdout: mon osd full ratio = 0.9 2024-02-21T23:01:37.267 INFO:teuthology.orchestra.run.smithi081.stdout: mon osd nearfull ratio = 0.8 2024-02-21T23:01:37.267 INFO:teuthology.orchestra.run.smithi081.stdout: osd failsafe full ratio = 0.95 2024-02-21T23:01:37.267 INFO:teuthology.orchestra.run.smithi081.stdout: osd objectstore = bluestore 2024-02-21T23:01:37.267 INFO:teuthology.orchestra.run.smithi081.stdout: osd op complaint time = 180 2024-02-21T23:01:37.267 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-02-21T23:01:37.267 INFO:teuthology.orchestra.run.smithi081.stdout:[mgr] 2024-02-21T23:01:37.267 INFO:teuthology.orchestra.run.smithi081.stdout: debug ms = 1 2024-02-21T23:01:37.267 INFO:teuthology.orchestra.run.smithi081.stdout: debug mgr = 20 2024-02-21T23:01:37.267 INFO:teuthology.orchestra.run.smithi081.stdout: debug mon = 20 2024-02-21T23:01:37.267 INFO:teuthology.orchestra.run.smithi081.stdout: debug auth = 20 2024-02-21T23:01:37.267 INFO:teuthology.orchestra.run.smithi081.stdout: mon reweight min pgs per osd = 4 2024-02-21T23:01:37.267 INFO:teuthology.orchestra.run.smithi081.stdout: mon reweight min bytes per osd = 10 2024-02-21T23:01:37.267 INFO:teuthology.orchestra.run.smithi081.stdout: mgr/telemetry/nag = false 2024-02-21T23:01:37.267 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-02-21T23:01:37.268 INFO:teuthology.orchestra.run.smithi081.stdout:[mon] 2024-02-21T23:01:37.268 INFO:teuthology.orchestra.run.smithi081.stdout: debug ms = 1 2024-02-21T23:01:37.268 INFO:teuthology.orchestra.run.smithi081.stdout: debug mon = 20 2024-02-21T23:01:37.268 INFO:teuthology.orchestra.run.smithi081.stdout: debug paxos = 20 2024-02-21T23:01:37.268 INFO:teuthology.orchestra.run.smithi081.stdout: debug auth = 20 2024-02-21T23:01:37.268 INFO:teuthology.orchestra.run.smithi081.stdout: mon data avail warn = 5 2024-02-21T23:01:37.268 INFO:teuthology.orchestra.run.smithi081.stdout: mon mgr mkfs grace = 240 2024-02-21T23:01:37.268 INFO:teuthology.orchestra.run.smithi081.stdout: mon reweight min pgs per osd = 4 2024-02-21T23:01:37.268 INFO:teuthology.orchestra.run.smithi081.stdout: mon osd reporter subtree level = osd 2024-02-21T23:01:37.268 INFO:teuthology.orchestra.run.smithi081.stdout: mon osd prime pg temp = true 2024-02-21T23:01:37.268 INFO:teuthology.orchestra.run.smithi081.stdout: mon reweight min bytes per osd = 10 2024-02-21T23:01:37.268 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-02-21T23:01:37.268 INFO:teuthology.orchestra.run.smithi081.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-02-21T23:01:37.268 INFO:teuthology.orchestra.run.smithi081.stdout: auth mon ticket ttl = 660 # 11m 2024-02-21T23:01:37.269 INFO:teuthology.orchestra.run.smithi081.stdout: auth service ticket ttl = 240 # 4m 2024-02-21T23:01:37.269 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-02-21T23:01:37.269 INFO:teuthology.orchestra.run.smithi081.stdout: # don't complain about insecure global_id in the test suite 2024-02-21T23:01:37.269 INFO:teuthology.orchestra.run.smithi081.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-02-21T23:01:37.269 INFO:teuthology.orchestra.run.smithi081.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-02-21T23:01:37.269 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-02-21T23:01:37.269 INFO:teuthology.orchestra.run.smithi081.stdout: # 1m isn't quite enough 2024-02-21T23:01:37.269 INFO:teuthology.orchestra.run.smithi081.stdout: mon_down_mkfs_grace = 2m 2024-02-21T23:01:37.269 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-02-21T23:01:37.269 INFO:teuthology.orchestra.run.smithi081.stdout: mon_warn_on_filestore_osds = false 2024-02-21T23:01:37.269 INFO:teuthology.orchestra.run.smithi081.stdout: mon op complaint time = 120 2024-02-21T23:01:37.269 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-02-21T23:01:37.269 INFO:teuthology.orchestra.run.smithi081.stdout:[client] 2024-02-21T23:01:37.269 INFO:teuthology.orchestra.run.smithi081.stdout: rgw cache enabled = true 2024-02-21T23:01:37.269 INFO:teuthology.orchestra.run.smithi081.stdout: rgw enable ops log = true 2024-02-21T23:01:37.269 INFO:teuthology.orchestra.run.smithi081.stdout: rgw enable usage log = true 2024-02-21T23:01:37.270 INFO:teuthology.orchestra.run.smithi081.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-02-21T23:01:37.270 INFO:teuthology.orchestra.run.smithi081.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-02-21T23:01:37.270 INFO:teuthology.orchestra.run.smithi081.stdout: client mount timeout = 600 2024-02-21T23:01:37.270 INFO:teuthology.orchestra.run.smithi081.stdout: debug client = 20 2024-02-21T23:01:37.270 INFO:teuthology.orchestra.run.smithi081.stdout: debug ms = 1 2024-02-21T23:01:37.270 INFO:teuthology.orchestra.run.smithi081.stdout: rados mon op timeout = 900 2024-02-21T23:01:37.270 INFO:teuthology.orchestra.run.smithi081.stdout: rados osd op timeout = 900 2024-02-21T23:01:37.270 INFO:teuthology.orchestra.run.smithi081.stdout:[mon.a] 2024-02-21T23:01:37.270 INFO:teuthology.orchestra.run.smithi081.stdout:[mon.b] 2024-02-21T23:01:37.270 INFO:teuthology.orchestra.run.smithi081.stdout:[mon.c] 2024-02-21T23:01:37.270 INFO:teuthology.orchestra.run.smithi081.stdout:[mds.a] 2024-02-21T23:01:37.270 INFO:teuthology.orchestra.run.smithi081.stdout:[mds.c] 2024-02-21T23:01:37.270 INFO:teuthology.orchestra.run.smithi081.stdout:[mds.b] 2024-02-21T23:01:37.270 INFO:teuthology.orchestra.run.smithi081.stdout:[mds.d] 2024-02-21T23:01:37.270 INFO:teuthology.orchestra.run.smithi081.stdout:[mds] 2024-02-21T23:01:37.271 INFO:teuthology.orchestra.run.smithi081.stdout: debug mds = 20 2024-02-21T23:01:37.271 INFO:teuthology.orchestra.run.smithi081.stdout: debug mds balancer = 20 2024-02-21T23:01:37.271 INFO:teuthology.orchestra.run.smithi081.stdout: debug ms = 1 2024-02-21T23:01:37.271 INFO:teuthology.orchestra.run.smithi081.stdout: mds debug frag = True 2024-02-21T23:01:37.271 INFO:teuthology.orchestra.run.smithi081.stdout: mds debug scatterstat = True 2024-02-21T23:01:37.271 INFO:teuthology.orchestra.run.smithi081.stdout: mds op complaint time = 180 2024-02-21T23:01:37.271 INFO:teuthology.orchestra.run.smithi081.stdout: mds verify scatter = True 2024-02-21T23:01:37.271 INFO:teuthology.orchestra.run.smithi081.stdout: osd op complaint time = 180 2024-02-21T23:01:37.271 INFO:teuthology.orchestra.run.smithi081.stdout: rados mon op timeout = 900 2024-02-21T23:01:37.271 INFO:teuthology.orchestra.run.smithi081.stdout: rados osd op timeout = 900 2024-02-21T23:01:37.295 INFO:tasks.ceph:Creating admin key on mon.a... 2024-02-21T23:01:37.295 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-02-21T23:01:37.439 INFO:tasks.ceph:Copying monmap to all nodes... 2024-02-21T23:01:37.439 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-21T23:01:37.439 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-02-21T23:01:37.494 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-21T23:01:37.494 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-02-21T23:01:37.547 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi028.front.sepia.ceph.com 2024-02-21T23:01:37.547 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-21T23:01:37.547 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/ceph/ceph.keyring 2024-02-21T23:01:37.548 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-02-21T23:01:37.709 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-21T23:01:37.709 DEBUG:teuthology.orchestra.run.smithi028:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-02-21T23:01:37.808 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi081.front.sepia.ceph.com 2024-02-21T23:01:37.809 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-02-21T23:01:37.809 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/etc/ceph/ceph.keyring 2024-02-21T23:01:37.809 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-02-21T23:01:37.966 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-02-21T23:01:37.966 DEBUG:teuthology.orchestra.run.smithi081:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-02-21T23:01:38.061 INFO:tasks.ceph:Setting up mon nodes... 2024-02-21T23:01:38.061 INFO:tasks.ceph:Setting up mgr nodes... 2024-02-21T23:01:38.061 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-02-21T23:01:38.257 INFO:teuthology.orchestra.run.smithi028.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-02-21T23:01:38.268 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-02-21T23:01:38.468 INFO:teuthology.orchestra.run.smithi081.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-02-21T23:01:38.479 INFO:tasks.ceph:Setting up mds nodes... 2024-02-21T23:01:38.479 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-02-21T23:01:38.679 INFO:teuthology.orchestra.run.smithi028.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-02-21T23:01:38.689 DEBUG:teuthology.orchestra.run.smithi028:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-02-21T23:01:38.791 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-02-21T23:01:38.983 INFO:teuthology.orchestra.run.smithi028.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-02-21T23:01:38.992 DEBUG:teuthology.orchestra.run.smithi028:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-02-21T23:01:39.091 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-02-21T23:01:39.273 INFO:teuthology.orchestra.run.smithi081.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-02-21T23:01:39.282 DEBUG:teuthology.orchestra.run.smithi081:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-02-21T23:01:39.378 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-02-21T23:01:39.552 INFO:teuthology.orchestra.run.smithi081.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-02-21T23:01:39.561 DEBUG:teuthology.orchestra.run.smithi081:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-02-21T23:01:39.658 INFO:tasks.ceph_client:Setting up client nodes... 2024-02-21T23:01:39.658 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-02-21T23:01:39.790 INFO:teuthology.orchestra.run.smithi028.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-02-21T23:01:39.852 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-02-21T23:01:39.980 INFO:teuthology.orchestra.run.smithi081.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-02-21T23:01:40.037 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-02-21T23:01:40.037 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi028.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi081.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-02-21T23:01:40.037 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-02-21T23:01:40.144 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-02-21T23:01:40.144 INFO:tasks.ceph:role: osd.0 2024-02-21T23:01:40.144 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi028.front.sepia.ceph.com 2024-02-21T23:01:40.145 DEBUG:teuthology.orchestra.run.smithi028:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-02-21T23:01:40.243 INFO:teuthology.orchestra.run.smithi028.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-02-21T23:01:40.243 INFO:teuthology.orchestra.run.smithi028.stdout: = sectsz=512 attr=2, projid32bit=1 2024-02-21T23:01:40.243 INFO:teuthology.orchestra.run.smithi028.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-02-21T23:01:40.243 INFO:teuthology.orchestra.run.smithi028.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-02-21T23:01:40.243 INFO:teuthology.orchestra.run.smithi028.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-02-21T23:01:40.243 INFO:teuthology.orchestra.run.smithi028.stdout: = sunit=0 swidth=0 blks 2024-02-21T23:01:40.243 INFO:teuthology.orchestra.run.smithi028.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-02-21T23:01:40.243 INFO:teuthology.orchestra.run.smithi028.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-02-21T23:01:40.243 INFO:teuthology.orchestra.run.smithi028.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-02-21T23:01:40.243 INFO:teuthology.orchestra.run.smithi028.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-02-21T23:01:40.282 INFO:teuthology.orchestra.run.smithi028.stdout:Discarding blocks...Done. 2024-02-21T23:01:40.288 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi028.front.sepia.ceph.com -o noatime 2024-02-21T23:01:40.289 DEBUG:teuthology.orchestra.run.smithi028:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-02-21T23:01:40.399 DEBUG:teuthology.orchestra.run.smithi028:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-02-21T23:01:40.480 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-02-21T23:01:40.561 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-02-21T23:01:40.562 INFO:tasks.ceph:role: osd.1 2024-02-21T23:01:40.562 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi028.front.sepia.ceph.com 2024-02-21T23:01:40.562 DEBUG:teuthology.orchestra.run.smithi028:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-02-21T23:01:40.646 INFO:teuthology.orchestra.run.smithi028.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-02-21T23:01:40.647 INFO:teuthology.orchestra.run.smithi028.stdout: = sectsz=512 attr=2, projid32bit=1 2024-02-21T23:01:40.647 INFO:teuthology.orchestra.run.smithi028.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-02-21T23:01:40.647 INFO:teuthology.orchestra.run.smithi028.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-02-21T23:01:40.647 INFO:teuthology.orchestra.run.smithi028.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-02-21T23:01:40.647 INFO:teuthology.orchestra.run.smithi028.stdout: = sunit=0 swidth=0 blks 2024-02-21T23:01:40.647 INFO:teuthology.orchestra.run.smithi028.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-02-21T23:01:40.647 INFO:teuthology.orchestra.run.smithi028.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-02-21T23:01:40.647 INFO:teuthology.orchestra.run.smithi028.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-02-21T23:01:40.647 INFO:teuthology.orchestra.run.smithi028.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-02-21T23:01:40.680 INFO:teuthology.orchestra.run.smithi028.stdout:Discarding blocks...Done. 2024-02-21T23:01:40.686 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi028.front.sepia.ceph.com -o noatime 2024-02-21T23:01:40.686 DEBUG:teuthology.orchestra.run.smithi028:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-02-21T23:01:40.797 DEBUG:teuthology.orchestra.run.smithi028:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-02-21T23:01:40.890 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-02-21T23:01:40.961 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-02-21T23:01:40.961 INFO:tasks.ceph:role: osd.2 2024-02-21T23:01:40.961 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi028.front.sepia.ceph.com 2024-02-21T23:01:40.961 DEBUG:teuthology.orchestra.run.smithi028:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-02-21T23:01:41.038 INFO:teuthology.orchestra.run.smithi028.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-02-21T23:01:41.039 INFO:teuthology.orchestra.run.smithi028.stdout: = sectsz=512 attr=2, projid32bit=1 2024-02-21T23:01:41.039 INFO:teuthology.orchestra.run.smithi028.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-02-21T23:01:41.039 INFO:teuthology.orchestra.run.smithi028.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-02-21T23:01:41.039 INFO:teuthology.orchestra.run.smithi028.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-02-21T23:01:41.039 INFO:teuthology.orchestra.run.smithi028.stdout: = sunit=0 swidth=0 blks 2024-02-21T23:01:41.039 INFO:teuthology.orchestra.run.smithi028.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-02-21T23:01:41.039 INFO:teuthology.orchestra.run.smithi028.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-02-21T23:01:41.039 INFO:teuthology.orchestra.run.smithi028.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-02-21T23:01:41.039 INFO:teuthology.orchestra.run.smithi028.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-02-21T23:01:41.073 INFO:teuthology.orchestra.run.smithi028.stdout:Discarding blocks...Done. 2024-02-21T23:01:41.079 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi028.front.sepia.ceph.com -o noatime 2024-02-21T23:01:41.079 DEBUG:teuthology.orchestra.run.smithi028:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-02-21T23:01:41.179 DEBUG:teuthology.orchestra.run.smithi028:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-02-21T23:01:41.255 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-02-21T23:01:41.336 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-02-21T23:01:41.336 INFO:tasks.ceph:role: osd.3 2024-02-21T23:01:41.336 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi028.front.sepia.ceph.com 2024-02-21T23:01:41.336 DEBUG:teuthology.orchestra.run.smithi028:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-02-21T23:01:41.418 INFO:teuthology.orchestra.run.smithi028.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-02-21T23:01:41.418 INFO:teuthology.orchestra.run.smithi028.stdout: = sectsz=512 attr=2, projid32bit=1 2024-02-21T23:01:41.418 INFO:teuthology.orchestra.run.smithi028.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-02-21T23:01:41.418 INFO:teuthology.orchestra.run.smithi028.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-02-21T23:01:41.418 INFO:teuthology.orchestra.run.smithi028.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-02-21T23:01:41.418 INFO:teuthology.orchestra.run.smithi028.stdout: = sunit=0 swidth=0 blks 2024-02-21T23:01:41.418 INFO:teuthology.orchestra.run.smithi028.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-02-21T23:01:41.418 INFO:teuthology.orchestra.run.smithi028.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-02-21T23:01:41.418 INFO:teuthology.orchestra.run.smithi028.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-02-21T23:01:41.418 INFO:teuthology.orchestra.run.smithi028.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-02-21T23:01:41.452 INFO:teuthology.orchestra.run.smithi028.stdout:Discarding blocks...Done. 2024-02-21T23:01:41.457 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi028.front.sepia.ceph.com -o noatime 2024-02-21T23:01:41.457 DEBUG:teuthology.orchestra.run.smithi028:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-02-21T23:01:41.558 DEBUG:teuthology.orchestra.run.smithi028:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-02-21T23:01:41.637 DEBUG:teuthology.orchestra.run.smithi028:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-02-21T23:01:41.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:01:41.792+0000 7fe7b7aae640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-02-21T23:01:41.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:01:41.793+0000 7fe7b7aae640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-02-21T23:01:41.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:01:41.793+0000 7fe7b7aae640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-02-21T23:01:41.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:01:41.793+0000 7fe7b7aae640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-02-21T23:01:41.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:01:41.793+0000 7fe7b7aae640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-02-21T23:01:41.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:01:41.796+0000 7fe7b7aae640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-02-21T23:01:44.716 DEBUG:teuthology.orchestra.run.smithi028:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-02-21T23:01:44.815 DEBUG:teuthology.orchestra.run.smithi028:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-02-21T23:01:44.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:01:44.969+0000 7f86a2136640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-02-21T23:01:44.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:01:44.970+0000 7f86a2136640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-02-21T23:01:44.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:01:44.970+0000 7f86a2136640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-02-21T23:01:44.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:01:44.970+0000 7f86a2136640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-02-21T23:01:44.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:01:44.970+0000 7f86a2136640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-02-21T23:01:44.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:01:44.972+0000 7f86a2136640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-02-21T23:01:48.133 DEBUG:teuthology.orchestra.run.smithi028:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-02-21T23:01:48.238 DEBUG:teuthology.orchestra.run.smithi028:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-02-21T23:01:48.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:01:48.402+0000 7fcd30a11640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-02-21T23:01:48.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:01:48.402+0000 7fcd30a11640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-02-21T23:01:48.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:01:48.402+0000 7fcd30a11640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-02-21T23:01:48.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:01:48.402+0000 7fcd30a11640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-02-21T23:01:48.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:01:48.402+0000 7fcd30a11640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-02-21T23:01:48.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:01:48.405+0000 7fcd30a11640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-02-21T23:01:51.555 DEBUG:teuthology.orchestra.run.smithi028:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-02-21T23:01:51.658 DEBUG:teuthology.orchestra.run.smithi028:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-02-21T23:01:51.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:01:51.818+0000 7f08b0ab5640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-02-21T23:01:51.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:01:51.818+0000 7f08b0ab5640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-02-21T23:01:51.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:01:51.818+0000 7f08b0ab5640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-02-21T23:01:51.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:01:51.818+0000 7f08b0ab5640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-02-21T23:01:51.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:01:51.818+0000 7f08b0ab5640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-02-21T23:01:51.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:01:51.821+0000 7f08b0ab5640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-02-21T23:01:54.984 DEBUG:teuthology.orchestra.run.smithi028:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-02-21T23:01:55.087 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-02-21T23:01:55.190 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-02-21T23:01:55.190 INFO:tasks.ceph:role: osd.4 2024-02-21T23:01:55.191 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi081.front.sepia.ceph.com 2024-02-21T23:01:55.191 DEBUG:teuthology.orchestra.run.smithi081:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-02-21T23:01:55.292 INFO:teuthology.orchestra.run.smithi081.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-02-21T23:01:55.292 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 attr=2, projid32bit=1 2024-02-21T23:01:55.292 INFO:teuthology.orchestra.run.smithi081.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-02-21T23:01:55.292 INFO:teuthology.orchestra.run.smithi081.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-02-21T23:01:55.292 INFO:teuthology.orchestra.run.smithi081.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-02-21T23:01:55.293 INFO:teuthology.orchestra.run.smithi081.stdout: = sunit=0 swidth=0 blks 2024-02-21T23:01:55.293 INFO:teuthology.orchestra.run.smithi081.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-02-21T23:01:55.293 INFO:teuthology.orchestra.run.smithi081.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-02-21T23:01:55.293 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-02-21T23:01:55.293 INFO:teuthology.orchestra.run.smithi081.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-02-21T23:01:55.809 INFO:teuthology.orchestra.run.smithi081.stdout:Discarding blocks...Done. 2024-02-21T23:01:55.816 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi081.front.sepia.ceph.com -o noatime 2024-02-21T23:01:55.816 DEBUG:teuthology.orchestra.run.smithi081:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-02-21T23:01:55.975 DEBUG:teuthology.orchestra.run.smithi081:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-02-21T23:01:56.076 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-02-21T23:01:56.168 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-02-21T23:01:56.169 INFO:tasks.ceph:role: osd.5 2024-02-21T23:01:56.169 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi081.front.sepia.ceph.com 2024-02-21T23:01:56.169 DEBUG:teuthology.orchestra.run.smithi081:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-02-21T23:01:56.264 INFO:teuthology.orchestra.run.smithi081.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-02-21T23:01:56.264 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 attr=2, projid32bit=1 2024-02-21T23:01:56.264 INFO:teuthology.orchestra.run.smithi081.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-02-21T23:01:56.264 INFO:teuthology.orchestra.run.smithi081.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-02-21T23:01:56.264 INFO:teuthology.orchestra.run.smithi081.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-02-21T23:01:56.264 INFO:teuthology.orchestra.run.smithi081.stdout: = sunit=0 swidth=0 blks 2024-02-21T23:01:56.264 INFO:teuthology.orchestra.run.smithi081.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-02-21T23:01:56.264 INFO:teuthology.orchestra.run.smithi081.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-02-21T23:01:56.264 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-02-21T23:01:56.264 INFO:teuthology.orchestra.run.smithi081.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-02-21T23:01:56.742 INFO:teuthology.orchestra.run.smithi081.stdout:Discarding blocks...Done. 2024-02-21T23:01:56.748 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi081.front.sepia.ceph.com -o noatime 2024-02-21T23:01:56.748 DEBUG:teuthology.orchestra.run.smithi081:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-02-21T23:01:56.885 DEBUG:teuthology.orchestra.run.smithi081:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-02-21T23:01:56.975 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-02-21T23:01:57.063 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-02-21T23:01:57.063 INFO:tasks.ceph:role: osd.6 2024-02-21T23:01:57.063 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi081.front.sepia.ceph.com 2024-02-21T23:01:57.063 DEBUG:teuthology.orchestra.run.smithi081:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-02-21T23:01:57.158 INFO:teuthology.orchestra.run.smithi081.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-02-21T23:01:57.159 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 attr=2, projid32bit=1 2024-02-21T23:01:57.159 INFO:teuthology.orchestra.run.smithi081.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-02-21T23:01:57.159 INFO:teuthology.orchestra.run.smithi081.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-02-21T23:01:57.159 INFO:teuthology.orchestra.run.smithi081.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-02-21T23:01:57.159 INFO:teuthology.orchestra.run.smithi081.stdout: = sunit=0 swidth=0 blks 2024-02-21T23:01:57.159 INFO:teuthology.orchestra.run.smithi081.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-02-21T23:01:57.159 INFO:teuthology.orchestra.run.smithi081.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-02-21T23:01:57.159 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-02-21T23:01:57.159 INFO:teuthology.orchestra.run.smithi081.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-02-21T23:01:57.672 INFO:teuthology.orchestra.run.smithi081.stdout:Discarding blocks...Done. 2024-02-21T23:01:57.679 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi081.front.sepia.ceph.com -o noatime 2024-02-21T23:01:57.680 DEBUG:teuthology.orchestra.run.smithi081:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-02-21T23:01:57.848 DEBUG:teuthology.orchestra.run.smithi081:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-02-21T23:01:57.942 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-02-21T23:01:58.032 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-02-21T23:01:58.032 INFO:tasks.ceph:role: osd.7 2024-02-21T23:01:58.032 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi081.front.sepia.ceph.com 2024-02-21T23:01:58.032 DEBUG:teuthology.orchestra.run.smithi081:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-02-21T23:01:58.127 INFO:teuthology.orchestra.run.smithi081.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-02-21T23:01:58.127 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 attr=2, projid32bit=1 2024-02-21T23:01:58.127 INFO:teuthology.orchestra.run.smithi081.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-02-21T23:01:58.127 INFO:teuthology.orchestra.run.smithi081.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-02-21T23:01:58.127 INFO:teuthology.orchestra.run.smithi081.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-02-21T23:01:58.127 INFO:teuthology.orchestra.run.smithi081.stdout: = sunit=0 swidth=0 blks 2024-02-21T23:01:58.127 INFO:teuthology.orchestra.run.smithi081.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-02-21T23:01:58.127 INFO:teuthology.orchestra.run.smithi081.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-02-21T23:01:58.128 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-02-21T23:01:58.128 INFO:teuthology.orchestra.run.smithi081.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-02-21T23:01:58.584 INFO:teuthology.orchestra.run.smithi081.stdout:Discarding blocks...Done. 2024-02-21T23:01:58.590 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi081.front.sepia.ceph.com -o noatime 2024-02-21T23:01:58.590 DEBUG:teuthology.orchestra.run.smithi081:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-02-21T23:01:58.742 DEBUG:teuthology.orchestra.run.smithi081:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-02-21T23:01:58.840 DEBUG:teuthology.orchestra.run.smithi081:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-02-21T23:01:58.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:01:59.008+0000 7f6d853aa640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-02-21T23:01:58.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:01:59.008+0000 7f6d853aa640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-02-21T23:01:58.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:01:59.008+0000 7f6d853aa640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-02-21T23:01:58.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:01:59.008+0000 7f6d853aa640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-02-21T23:01:58.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:01:59.008+0000 7f6d853aa640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-02-21T23:01:59.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:01:59.015+0000 7f6d853aa640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-02-21T23:02:01.939 DEBUG:teuthology.orchestra.run.smithi081:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-02-21T23:02:02.035 DEBUG:teuthology.orchestra.run.smithi081:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-02-21T23:02:02.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:02.178+0000 7f9bf1a8b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-02-21T23:02:02.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:02.179+0000 7f9bf1a8b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-02-21T23:02:02.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:02.179+0000 7f9bf1a8b640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-02-21T23:02:02.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:02.179+0000 7f9bf1a8b640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-02-21T23:02:02.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:02.179+0000 7f9bf1a8b640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-02-21T23:02:02.170 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:02.181+0000 7f9bf1a8b640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-02-21T23:02:04.292 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-21T23:02:04.300 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-21T23:02:05.355 DEBUG:teuthology.orchestra.run.smithi081:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-02-21T23:02:05.448 DEBUG:teuthology.orchestra.run.smithi081:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-02-21T23:02:05.585 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:05.596+0000 7f444f68b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-02-21T23:02:05.586 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:05.597+0000 7f444f68b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-02-21T23:02:05.586 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:05.597+0000 7f444f68b640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-02-21T23:02:05.586 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:05.597+0000 7f444f68b640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-02-21T23:02:05.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:05.597+0000 7f444f68b640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-02-21T23:02:05.588 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:05.599+0000 7f444f68b640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-02-21T23:02:08.779 DEBUG:teuthology.orchestra.run.smithi081:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-02-21T23:02:08.873 DEBUG:teuthology.orchestra.run.smithi081:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-02-21T23:02:09.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:09.021+0000 7f239c773640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-02-21T23:02:09.011 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:09.022+0000 7f239c773640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-02-21T23:02:09.011 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:09.022+0000 7f239c773640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-02-21T23:02:09.011 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:09.022+0000 7f239c773640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-02-21T23:02:09.011 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:09.022+0000 7f239c773640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-02-21T23:02:09.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:09.024+0000 7f239c773640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-02-21T23:02:11.948 DEBUG:teuthology.orchestra.run.smithi081:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-02-21T23:02:12.046 INFO:tasks.ceph:Reading keys from all nodes... 2024-02-21T23:02:12.046 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-21T23:02:12.046 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-02-21T23:02:12.152 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-21T23:02:12.153 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-02-21T23:02:12.251 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-21T23:02:12.251 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-02-21T23:02:12.352 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-21T23:02:12.352 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-02-21T23:02:12.454 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-21T23:02:12.454 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-02-21T23:02:12.548 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-21T23:02:12.549 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-02-21T23:02:12.648 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-21T23:02:12.649 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-02-21T23:02:12.751 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-02-21T23:02:12.751 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-02-21T23:02:12.850 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-02-21T23:02:12.850 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-02-21T23:02:12.941 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-02-21T23:02:12.942 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-02-21T23:02:13.034 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-02-21T23:02:13.034 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-02-21T23:02:13.123 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-02-21T23:02:13.123 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-02-21T23:02:13.217 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-02-21T23:02:13.217 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-02-21T23:02:13.310 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-02-21T23:02:13.310 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-02-21T23:02:13.446 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-21T23:02:13.446 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-02-21T23:02:13.505 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-02-21T23:02:13.505 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-02-21T23:02:13.561 INFO:tasks.ceph:Adding keys to all mons... 2024-02-21T23:02:13.561 DEBUG:teuthology.orchestra.run.smithi028:> sudo tee -a /etc/ceph/ceph.keyring 2024-02-21T23:02:13.568 DEBUG:teuthology.orchestra.run.smithi081:> sudo tee -a /etc/ceph/ceph.keyring 2024-02-21T23:02:13.657 INFO:teuthology.orchestra.run.smithi081.stdout:[mgr.y] 2024-02-21T23:02:13.657 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQDSgNZlSRkOEBAAOhRj9PsZDpjoV6WT4drrZA== 2024-02-21T23:02:13.658 INFO:teuthology.orchestra.run.smithi081.stdout:[mds.a] 2024-02-21T23:02:13.658 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQDSgNZl4FY1KRAAJSK5Ibb0ibKiKXYpTLgtmQ== 2024-02-21T23:02:13.658 INFO:teuthology.orchestra.run.smithi081.stdout:[mds.c] 2024-02-21T23:02:13.658 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQDSgNZloFpQOxAAK2WBArXRVNTgqH9WWDEPDA== 2024-02-21T23:02:13.658 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.0] 2024-02-21T23:02:13.658 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQDVgNZlxNJbLxAAXot7Q3chrLEBsFNO8kT6QQ== 2024-02-21T23:02:13.658 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.1] 2024-02-21T23:02:13.658 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQDYgNZlZZfWORAAr4XkOxLYLmumVbu362N2HQ== 2024-02-21T23:02:13.658 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.2] 2024-02-21T23:02:13.658 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQDcgNZl9mcDGBAAIZGOF8IUMenwo386x95mwg== 2024-02-21T23:02:13.658 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.3] 2024-02-21T23:02:13.658 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQDfgNZl8ALSMBAAag9cmGH0/4xALWSUmrZowg== 2024-02-21T23:02:13.659 INFO:teuthology.orchestra.run.smithi081.stdout:[mgr.x] 2024-02-21T23:02:13.659 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQDSgNZlJYKeHBAAYPskmLRr7Jw44TdKzM/JMg== 2024-02-21T23:02:13.659 INFO:teuthology.orchestra.run.smithi081.stdout:[mds.b] 2024-02-21T23:02:13.659 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQDTgNZlwRcAERAAgsYmmkuobKelBEJVCRpXUg== 2024-02-21T23:02:13.659 INFO:teuthology.orchestra.run.smithi081.stdout:[mds.d] 2024-02-21T23:02:13.659 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQDTgNZlblOmIRAAxe2HfSKGSkkzJDN4aVdzgQ== 2024-02-21T23:02:13.659 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.4] 2024-02-21T23:02:13.659 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQDngNZlBgeKABAAeW99RVfCBvB1lp7w7mQBlQ== 2024-02-21T23:02:13.659 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.5] 2024-02-21T23:02:13.659 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQDqgNZljQG3ChAAY/IZEIcvcGNvlFGIWib3gg== 2024-02-21T23:02:13.659 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.6] 2024-02-21T23:02:13.659 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQDtgNZlKnakIxAAkW5xlkshzvEHJdmpgPxR+g== 2024-02-21T23:02:13.659 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.7] 2024-02-21T23:02:13.659 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQDxgNZlTWxYARAASX1/gmqO+8mQPP8uA4xjiQ== 2024-02-21T23:02:13.660 INFO:teuthology.orchestra.run.smithi081.stdout:[client.0] 2024-02-21T23:02:13.660 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQDTgNZlILvYLxAAoREWxKKT7YcXnhkgk8YvoQ== 2024-02-21T23:02:13.660 INFO:teuthology.orchestra.run.smithi081.stdout:[client.1] 2024-02-21T23:02:13.660 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQDTgNZlm7ImOxAA8iRcXlzFPZPaXQfsKhuO/g== 2024-02-21T23:02:13.660 INFO:teuthology.orchestra.run.smithi028.stdout:[mgr.y] 2024-02-21T23:02:13.661 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQDSgNZlSRkOEBAAOhRj9PsZDpjoV6WT4drrZA== 2024-02-21T23:02:13.661 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.a] 2024-02-21T23:02:13.661 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQDSgNZl4FY1KRAAJSK5Ibb0ibKiKXYpTLgtmQ== 2024-02-21T23:02:13.661 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.c] 2024-02-21T23:02:13.661 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQDSgNZloFpQOxAAK2WBArXRVNTgqH9WWDEPDA== 2024-02-21T23:02:13.661 INFO:teuthology.orchestra.run.smithi028.stdout:[osd.0] 2024-02-21T23:02:13.661 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQDVgNZlxNJbLxAAXot7Q3chrLEBsFNO8kT6QQ== 2024-02-21T23:02:13.661 INFO:teuthology.orchestra.run.smithi028.stdout:[osd.1] 2024-02-21T23:02:13.661 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQDYgNZlZZfWORAAr4XkOxLYLmumVbu362N2HQ== 2024-02-21T23:02:13.661 INFO:teuthology.orchestra.run.smithi028.stdout:[osd.2] 2024-02-21T23:02:13.662 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQDcgNZl9mcDGBAAIZGOF8IUMenwo386x95mwg== 2024-02-21T23:02:13.662 INFO:teuthology.orchestra.run.smithi028.stdout:[osd.3] 2024-02-21T23:02:13.662 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQDfgNZl8ALSMBAAag9cmGH0/4xALWSUmrZowg== 2024-02-21T23:02:13.662 INFO:teuthology.orchestra.run.smithi028.stdout:[mgr.x] 2024-02-21T23:02:13.662 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQDSgNZlJYKeHBAAYPskmLRr7Jw44TdKzM/JMg== 2024-02-21T23:02:13.662 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.b] 2024-02-21T23:02:13.662 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQDTgNZlwRcAERAAgsYmmkuobKelBEJVCRpXUg== 2024-02-21T23:02:13.662 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.d] 2024-02-21T23:02:13.662 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQDTgNZlblOmIRAAxe2HfSKGSkkzJDN4aVdzgQ== 2024-02-21T23:02:13.662 INFO:teuthology.orchestra.run.smithi028.stdout:[osd.4] 2024-02-21T23:02:13.662 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQDngNZlBgeKABAAeW99RVfCBvB1lp7w7mQBlQ== 2024-02-21T23:02:13.662 INFO:teuthology.orchestra.run.smithi028.stdout:[osd.5] 2024-02-21T23:02:13.663 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQDqgNZljQG3ChAAY/IZEIcvcGNvlFGIWib3gg== 2024-02-21T23:02:13.663 INFO:teuthology.orchestra.run.smithi028.stdout:[osd.6] 2024-02-21T23:02:13.663 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQDtgNZlKnakIxAAkW5xlkshzvEHJdmpgPxR+g== 2024-02-21T23:02:13.663 INFO:teuthology.orchestra.run.smithi028.stdout:[osd.7] 2024-02-21T23:02:13.663 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQDxgNZlTWxYARAASX1/gmqO+8mQPP8uA4xjiQ== 2024-02-21T23:02:13.663 INFO:teuthology.orchestra.run.smithi028.stdout:[client.0] 2024-02-21T23:02:13.663 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQDTgNZlILvYLxAAoREWxKKT7YcXnhkgk8YvoQ== 2024-02-21T23:02:13.663 INFO:teuthology.orchestra.run.smithi028.stdout:[client.1] 2024-02-21T23:02:13.663 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQDTgNZlm7ImOxAA8iRcXlzFPZPaXQfsKhuO/g== 2024-02-21T23:02:13.665 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-02-21T23:02:13.672 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-02-21T23:02:13.819 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-02-21T23:02:13.825 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-02-21T23:02:13.973 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-02-21T23:02:13.978 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-02-21T23:02:14.125 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-21T23:02:14.132 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-21T23:02:14.269 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-21T23:02:14.276 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-21T23:02:14.415 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-21T23:02:14.422 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-21T23:02:14.566 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-21T23:02:14.573 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-21T23:02:14.717 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-02-21T23:02:14.721 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-02-21T23:02:14.866 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-02-21T23:02:14.874 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-02-21T23:02:15.022 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-02-21T23:02:15.029 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-02-21T23:02:15.176 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-21T23:02:15.183 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-21T23:02:15.329 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-21T23:02:15.333 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-21T23:02:15.479 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-21T23:02:15.484 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-21T23:02:15.624 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-21T23:02:15.629 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-02-21T23:02:15.771 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-02-21T23:02:15.777 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-02-21T23:02:15.923 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-02-21T23:02:15.933 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-02-21T23:02:16.084 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-02-21T23:02:16.084 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-02-21T23:02:16.183 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-02-21T23:02:16.350 DEBUG:teuthology.orchestra.run.smithi028:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-02-21T23:02:16.438 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-02-21T23:02:16.539 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-02-21T23:02:16.700 DEBUG:teuthology.orchestra.run.smithi081:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-02-21T23:02:16.790 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-02-21T23:02:16.885 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-02-21T23:02:17.039 DEBUG:teuthology.orchestra.run.smithi081:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-02-21T23:02:17.115 DEBUG:teuthology.orchestra.run.smithi028:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-02-21T23:02:17.122 DEBUG:teuthology.orchestra.run.smithi081:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-02-21T23:02:17.173 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-02-21T23:02:17.174 INFO:tasks.ceph.mon.a:Restarting daemon 2024-02-21T23:02:17.174 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-02-21T23:02:17.179 INFO:tasks.ceph.mon.a:Started 2024-02-21T23:02:17.179 INFO:tasks.ceph.mon.b:Restarting daemon 2024-02-21T23:02:17.179 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-02-21T23:02:17.186 INFO:tasks.ceph.mon.b:Started 2024-02-21T23:02:17.186 INFO:tasks.ceph.mon.c:Restarting daemon 2024-02-21T23:02:17.186 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-02-21T23:02:17.194 INFO:tasks.ceph.mon.c:Started 2024-02-21T23:02:17.194 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-02-21T23:02:17.194 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-02-21T23:02:17.194 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-02-21T23:02:17.199 INFO:tasks.ceph.mgr.y:Started 2024-02-21T23:02:17.199 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-02-21T23:02:17.199 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-02-21T23:02:17.203 INFO:tasks.ceph.mgr.x:Started 2024-02-21T23:02:17.203 INFO:tasks.ceph:Setting crush tunables to default 2024-02-21T23:02:17.203 DEBUG:teuthology.orchestra.run.smithi028:> sudo ceph --cluster ceph osd crush tunables default 2024-02-21T23:02:17.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:17.398+0000 7fe3e6800640 1 Processor -- start 2024-02-21T23:02:17.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:17.398+0000 7fe3e6800640 1 -- start start 2024-02-21T23:02:17.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:17.399+0000 7fe3e6800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe3e006b0e0 0x7fe3e00c5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:17.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:17.399+0000 7fe3e6800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fe3e00c5d80 0x7fe3e014fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:17.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:17.400+0000 7fe3e6800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe3e0150530 0x7fe3e0156500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:17.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:17.400+0000 7fe3e6800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3e0153bc0 con 0x7fe3e0150530 2024-02-21T23:02:17.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:17.400+0000 7fe3e6800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3e00c61e0 con 0x7fe3e00c5d80 2024-02-21T23:02:17.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:17.400+0000 7fe3e6800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3e0152fb0 con 0x7fe3e006b0e0 2024-02-21T23:02:17.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:17.400+0000 7fe3dfe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe3e0150530 0x7fe3e0156500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:17.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:17.400+0000 7fe3dfe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe3e0150530 0x7fe3e0156500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57226/0 (socket says 172.21.15.28:57226) 2024-02-21T23:02:17.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:17.400+0000 7fe3dfe00640 1 -- 172.21.15.28:0/3055162543 learned_addr learned my addr 172.21.15.28:0/3055162543 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:02:17.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:17.400+0000 7fe3d7e00640 1 --2- 172.21.15.28:0/3055162543 >> v2:172.21.15.81:3300/0 conn(0x7fe3e00c5d80 0x7fe3e014fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:17.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:17.400+0000 7fe3dfe00640 1 -- 172.21.15.28:0/3055162543 >> v2:172.21.15.28:3300/0 conn(0x7fe3e0150530 msgr2=0x7fe3e0156500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-02-21T23:02:17.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:17.400+0000 7fe3df400640 1 --2- 172.21.15.28:0/3055162543 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe3e006b0e0 0x7fe3e00c5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:17.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:17.400+0000 7fe3dfe00640 1 -- 172.21.15.28:0/3055162543 >> v2:172.21.15.28:3300/0 conn(0x7fe3e0150530 msgr2=0x7fe3e0156500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-02-21T23:02:17.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:17.400+0000 7fe3dfe00640 1 --2- 172.21.15.28:0/3055162543 >> v2:172.21.15.28:3300/0 conn(0x7fe3e0150530 0x7fe3e0156500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-02-21T23:02:17.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:17.400+0000 7fe3dfe00640 1 --2- 172.21.15.28:0/3055162543 >> v2:172.21.15.28:3300/0 conn(0x7fe3e0150530 0x7fe3e0156500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-21T23:02:17.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:17.400+0000 7fe3d7e00640 1 -- 172.21.15.28:0/3055162543 >> v2:172.21.15.81:3300/0 conn(0x7fe3e00c5d80 msgr2=0x7fe3e014fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-02-21T23:02:17.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:17.400+0000 7fe3d7e00640 1 -- 172.21.15.28:0/3055162543 >> v2:172.21.15.81:3300/0 conn(0x7fe3e00c5d80 msgr2=0x7fe3e014fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-02-21T23:02:17.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:17.400+0000 7fe3d7e00640 1 --2- 172.21.15.28:0/3055162543 >> v2:172.21.15.81:3300/0 conn(0x7fe3e00c5d80 0x7fe3e014fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-02-21T23:02:17.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:17.400+0000 7fe3d7e00640 1 --2- 172.21.15.28:0/3055162543 >> v2:172.21.15.81:3300/0 conn(0x7fe3e00c5d80 0x7fe3e014fff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-21T23:02:17.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:17.400+0000 7fe3df400640 1 -- 172.21.15.28:0/3055162543 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe3e006b0e0 msgr2=0x7fe3e00c5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-02-21T23:02:17.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:17.400+0000 7fe3df400640 1 -- 172.21.15.28:0/3055162543 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe3e006b0e0 msgr2=0x7fe3e00c5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-02-21T23:02:17.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:17.400+0000 7fe3df400640 1 --2- 172.21.15.28:0/3055162543 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe3e006b0e0 0x7fe3e00c5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-02-21T23:02:17.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:17.400+0000 7fe3df400640 1 --2- 172.21.15.28:0/3055162543 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe3e006b0e0 0x7fe3e00c5840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-21T23:02:17.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:17.601+0000 7fe3df400640 1 --2- 172.21.15.28:0/3055162543 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe3e006b0e0 0x7fe3e00c5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:17.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:17.601+0000 7fe3dfe00640 1 --2- 172.21.15.28:0/3055162543 >> v2:172.21.15.28:3300/0 conn(0x7fe3e0150530 0x7fe3e0156500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:17.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:17.602+0000 7fe3d7e00640 1 --2- 172.21.15.28:0/3055162543 >> v2:172.21.15.81:3300/0 conn(0x7fe3e00c5d80 0x7fe3e014fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:17.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:17.602+0000 7fe3df400640 1 -- 172.21.15.28:0/3055162543 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe3e006b0e0 msgr2=0x7fe3e00c5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-02-21T23:02:17.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:17.602+0000 7fe3df400640 1 -- 172.21.15.28:0/3055162543 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe3e006b0e0 msgr2=0x7fe3e00c5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-02-21T23:02:17.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:17.602+0000 7fe3dfe00640 1 -- 172.21.15.28:0/3055162543 >> v2:172.21.15.28:3300/0 conn(0x7fe3e0150530 msgr2=0x7fe3e0156500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-02-21T23:02:17.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:17.602+0000 7fe3dfe00640 1 -- 172.21.15.28:0/3055162543 >> v2:172.21.15.28:3300/0 conn(0x7fe3e0150530 msgr2=0x7fe3e0156500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-02-21T23:02:17.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:17.602+0000 7fe3dfe00640 1 --2- 172.21.15.28:0/3055162543 >> v2:172.21.15.28:3300/0 conn(0x7fe3e0150530 0x7fe3e0156500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-02-21T23:02:17.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:17.602+0000 7fe3df400640 1 --2- 172.21.15.28:0/3055162543 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe3e006b0e0 0x7fe3e00c5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-02-21T23:02:17.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:17.602+0000 7fe3dfe00640 1 --2- 172.21.15.28:0/3055162543 >> v2:172.21.15.28:3300/0 conn(0x7fe3e0150530 0x7fe3e0156500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-21T23:02:17.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:17.602+0000 7fe3df400640 1 --2- 172.21.15.28:0/3055162543 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe3e006b0e0 0x7fe3e00c5840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-21T23:02:17.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:17.602+0000 7fe3d7e00640 1 -- 172.21.15.28:0/3055162543 >> v2:172.21.15.81:3300/0 conn(0x7fe3e00c5d80 msgr2=0x7fe3e014fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-02-21T23:02:17.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:17.602+0000 7fe3d7e00640 1 -- 172.21.15.28:0/3055162543 >> v2:172.21.15.81:3300/0 conn(0x7fe3e00c5d80 msgr2=0x7fe3e014fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-02-21T23:02:17.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:17.602+0000 7fe3d7e00640 1 --2- 172.21.15.28:0/3055162543 >> v2:172.21.15.81:3300/0 conn(0x7fe3e00c5d80 0x7fe3e014fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-02-21T23:02:17.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:17.602+0000 7fe3d7e00640 1 --2- 172.21.15.28:0/3055162543 >> v2:172.21.15.81:3300/0 conn(0x7fe3e00c5d80 0x7fe3e014fff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-21T23:02:17.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:18.003+0000 7fe3dfe00640 1 --2- 172.21.15.28:0/3055162543 >> v2:172.21.15.28:3300/0 conn(0x7fe3e0150530 0x7fe3e0156500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:17.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:18.003+0000 7fe3d7e00640 1 --2- 172.21.15.28:0/3055162543 >> v2:172.21.15.81:3300/0 conn(0x7fe3e00c5d80 0x7fe3e014fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:17.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:18.003+0000 7fe3df400640 1 --2- 172.21.15.28:0/3055162543 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe3e006b0e0 0x7fe3e00c5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:17.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:18.004+0000 7fe3d7e00640 1 -- 172.21.15.28:0/3055162543 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe3e006b0e0 msgr2=0x7fe3e00c5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:17.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:18.004+0000 7fe3d7e00640 1 --2- 172.21.15.28:0/3055162543 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe3e006b0e0 0x7fe3e00c5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:17.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:18.004+0000 7fe3d7e00640 1 -- 172.21.15.28:0/3055162543 >> v2:172.21.15.28:3300/0 conn(0x7fe3e0150530 msgr2=0x7fe3e0156500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:17.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:18.004+0000 7fe3d7e00640 1 --2- 172.21.15.28:0/3055162543 >> v2:172.21.15.28:3300/0 conn(0x7fe3e0150530 0x7fe3e0156500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:17.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:18.004+0000 7fe3d7e00640 1 -- 172.21.15.28:0/3055162543 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3e0153290 con 0x7fe3e00c5d80 2024-02-21T23:02:17.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:18.004+0000 7fe3dfe00640 1 --2- 172.21.15.28:0/3055162543 >> v2:172.21.15.28:3300/0 conn(0x7fe3e0150530 0x7fe3e0156500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-21T23:02:17.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:18.007+0000 7fe3d7e00640 1 --2- 172.21.15.28:0/3055162543 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3e00c5d80 0x7fe3e014fff0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe3e006fad0 tx=0x7fe3c802bb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ba6a4a5046cc72e server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:17.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:18.008+0000 7fe3dea00640 1 -- 172.21.15.28:0/3055162543 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3c8032070 con 0x7fe3e00c5d80 2024-02-21T23:02:17.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:18.008+0000 7fe3dea00640 1 -- 172.21.15.28:0/3055162543 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3c802c730 con 0x7fe3e00c5d80 2024-02-21T23:02:17.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:18.008+0000 7fe3dea00640 1 -- 172.21.15.28:0/3055162543 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3c802cb60 con 0x7fe3e00c5d80 2024-02-21T23:02:17.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:18.009+0000 7fe3e6800640 1 -- 172.21.15.28:0/3055162543 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3e00c5d80 msgr2=0x7fe3e014fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:17.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:18.009+0000 7fe3e6800640 1 --2- 172.21.15.28:0/3055162543 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3e00c5d80 0x7fe3e014fff0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe3e006fad0 tx=0x7fe3c802bb90 comp rx=0 tx=0).stop 2024-02-21T23:02:17.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:18.009+0000 7fe3e6800640 1 -- 172.21.15.28:0/3055162543 shutdown_connections 2024-02-21T23:02:17.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:18.009+0000 7fe3e6800640 1 --2- 172.21.15.28:0/3055162543 >> v2:172.21.15.28:3300/0 conn(0x7fe3e0150530 0x7fe3e0156500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:17.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:18.009+0000 7fe3e6800640 1 --2- 172.21.15.28:0/3055162543 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3e00c5d80 0x7fe3e014fff0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:17.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:18.009+0000 7fe3e6800640 1 --2- 172.21.15.28:0/3055162543 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe3e006b0e0 0x7fe3e00c5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:17.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:18.009+0000 7fe3e6800640 1 -- 172.21.15.28:0/3055162543 >> 172.21.15.28:0/3055162543 conn(0x7fe3e00a85a0 msgr2=0x7fe3e00a8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:17.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:18.009+0000 7fe3e6800640 1 -- 172.21.15.28:0/3055162543 shutdown_connections 2024-02-21T23:02:17.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:18.009+0000 7fe3e6800640 1 -- 172.21.15.28:0/3055162543 wait complete. 2024-02-21T23:02:18.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:18.014+0000 7fe3e6800640 1 Processor -- start 2024-02-21T23:02:18.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:18.014+0000 7fe3e6800640 1 -- start start 2024-02-21T23:02:18.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:18.016+0000 7fe3e6800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3e006b0e0 0x7fe3e006e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:18.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:18.016+0000 7fe3e6800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe3e00c5d80 0x7fe3e0152600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:18.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:18.016+0000 7fe3e6800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3e0150530 0x7fe3e0148c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:18.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:18.017+0000 7fe3e6800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3e0150940 con 0x7fe3e0150530 2024-02-21T23:02:18.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:18.017+0000 7fe3e6800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3e0153850 con 0x7fe3e006b0e0 2024-02-21T23:02:18.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:18.017+0000 7fe3e6800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3e006b4c0 con 0x7fe3e00c5d80 2024-02-21T23:02:18.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:18.017+0000 7fe3df400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3e006b0e0 0x7fe3e006e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:18.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:18.017+0000 7fe3df400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3e006b0e0 0x7fe3e006e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.28:37120/0 (socket says 172.21.15.28:37120) 2024-02-21T23:02:18.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:18.017+0000 7fe3dfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3e0150530 0x7fe3e0148c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:18.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:18.017+0000 7fe3df400640 1 -- 172.21.15.28:0/848032906 learned_addr learned my addr 172.21.15.28:0/848032906 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:02:18.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:18.017+0000 7fe3d7e00640 1 --2- 172.21.15.28:0/848032906 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe3e00c5d80 0x7fe3e0152600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:18.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:18.017+0000 7fe3dfe00640 1 -- 172.21.15.28:0/848032906 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe3e00c5d80 msgr2=0x7fe3e0152600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:18.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:18.017+0000 7fe3dfe00640 1 --2- 172.21.15.28:0/848032906 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe3e00c5d80 0x7fe3e0152600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:18.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:18.017+0000 7fe3dfe00640 1 -- 172.21.15.28:0/848032906 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3e006b0e0 msgr2=0x7fe3e006e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:18.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:18.017+0000 7fe3dfe00640 1 --2- 172.21.15.28:0/848032906 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3e006b0e0 0x7fe3e006e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:18.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:18.017+0000 7fe3dfe00640 1 -- 172.21.15.28:0/848032906 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3e01ede00 con 0x7fe3e0150530 2024-02-21T23:02:18.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:18.017+0000 7fe3df400640 1 --2- 172.21.15.28:0/848032906 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3e006b0e0 0x7fe3e006e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-21T23:02:18.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:18.018+0000 7fe3dfe00640 1 --2- 172.21.15.28:0/848032906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3e0150530 0x7fe3e0148c50 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe3d80096d0 tx=0x7fe3d80115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:18.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:18.018+0000 7fe3d7400640 1 -- 172.21.15.28:0/848032906 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3d8019020 con 0x7fe3e0150530 2024-02-21T23:02:18.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:18.018+0000 7fe3d7400640 1 -- 172.21.15.28:0/848032906 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3d8011e60 con 0x7fe3e0150530 2024-02-21T23:02:18.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:18.018+0000 7fe3d7400640 1 -- 172.21.15.28:0/848032906 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3d8003b90 con 0x7fe3e0150530 2024-02-21T23:02:18.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:18.018+0000 7fe3e6800640 1 -- 172.21.15.28:0/848032906 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3e01ee0e0 con 0x7fe3e0150530 2024-02-21T23:02:18.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:18.019+0000 7fe3e6800640 1 -- 172.21.15.28:0/848032906 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3e01ee6a0 con 0x7fe3e0150530 2024-02-21T23:02:18.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:18.019+0000 7fe3d7400640 1 -- 172.21.15.28:0/848032906 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fe3d801c020 con 0x7fe3e0150530 2024-02-21T23:02:18.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:18.019+0000 7fe3d7400640 1 -- 172.21.15.28:0/848032906 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fe3d8003d30 con 0x7fe3e0150530 2024-02-21T23:02:18.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:18.020+0000 7fe3e6800640 1 -- 172.21.15.28:0/848032906 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3e01ee9d0 con 0x7fe3e0150530 2024-02-21T23:02:18.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:18.026+0000 7fe3d7400640 1 -- 172.21.15.28:0/848032906 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fe3d8023070 con 0x7fe3e0150530 2024-02-21T23:02:18.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:18.104+0000 7fe3e6800640 1 -- 172.21.15.28:0/848032906 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fe3e0067f30 con 0x7fe3e0150530 2024-02-21T23:02:18.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:18.107+0000 7fe3d7400640 1 -- 172.21.15.28:0/848032906 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7fe3d80043c0 con 0x7fe3e0150530 2024-02-21T23:02:18.096 INFO:teuthology.orchestra.run.smithi028.stderr:adjusted tunables profile to default 2024-02-21T23:02:18.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:18.109+0000 7fe3e6800640 1 -- 172.21.15.28:0/848032906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3e0150530 msgr2=0x7fe3e0148c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:18.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:18.109+0000 7fe3e6800640 1 --2- 172.21.15.28:0/848032906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3e0150530 0x7fe3e0148c50 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe3d80096d0 tx=0x7fe3d80115a0 comp rx=0 tx=0).stop 2024-02-21T23:02:18.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:18.109+0000 7fe3e6800640 1 -- 172.21.15.28:0/848032906 shutdown_connections 2024-02-21T23:02:18.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:18.109+0000 7fe3e6800640 1 --2- 172.21.15.28:0/848032906 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe3e00c5d80 0x7fe3e0152600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:18.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:18.109+0000 7fe3e6800640 1 --2- 172.21.15.28:0/848032906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3e0150530 0x7fe3e0148c50 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:18.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:18.109+0000 7fe3e6800640 1 --2- 172.21.15.28:0/848032906 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3e006b0e0 0x7fe3e006e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:18.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:18.109+0000 7fe3e6800640 1 -- 172.21.15.28:0/848032906 >> 172.21.15.28:0/848032906 conn(0x7fe3e00a85a0 msgr2=0x7fe3e00c4d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:18.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:18.109+0000 7fe3e6800640 1 -- 172.21.15.28:0/848032906 shutdown_connections 2024-02-21T23:02:18.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:18.109+0000 7fe3e6800640 1 -- 172.21.15.28:0/848032906 wait complete. 2024-02-21T23:02:18.117 INFO:tasks.ceph:check_enable_crimson: False 2024-02-21T23:02:18.117 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-02-21T23:02:18.117 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-21T23:02:18.117 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-02-21T23:02:18.199 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-02-21T23:02:18.210+0000 7f09bd23a040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-21T23:02:18.207 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-21T23:02:18.208 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-02-21T23:02:18.260 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-02-21T23:02:18.271+0000 7f09bd23a040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-21T23:02:18.268 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-02-21T23:02:18.280+0000 7f0d34c3e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-21T23:02:18.301 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-21T23:02:18.301 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-02-21T23:02:18.342 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-02-21T23:02:18.354+0000 7f0d34c3e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-21T23:02:18.395 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-21T23:02:18.395 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-02-21T23:02:18.522 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-02-21T23:02:18.523 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-02-21T23:02:18.612 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-02-21T23:02:18.612 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-02-21T23:02:18.733 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-02-21T23:02:18.733 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-02-21T23:02:18.847 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-02-21T23:02:18.847 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-02-21T23:02:18.907 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-02-21T23:02:18.917+0000 7f09bd23a040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-21T23:02:18.974 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-02-21T23:02:18.984+0000 7f09bd23a040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-21T23:02:18.980 DEBUG:teuthology.orchestra.run.smithi081:> sudo ceph --cluster ceph osd new f6ca1d98-5d2a-413d-870c-d7b9da1ff100 0 2024-02-21T23:02:18.989 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-02-21T23:02:19.000+0000 7f0d34c3e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-21T23:02:19.027 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-02-21T23:02:19.037+0000 7f09bd23a040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-21T23:02:19.055 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-02-21T23:02:19.066+0000 7f0d34c3e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-21T23:02:19.093 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-02-21T23:02:19.103+0000 7f09bd23a040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-21T23:02:19.108 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-02-21T23:02:19.120+0000 7f0d34c3e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-21T23:02:19.146 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-02-21T23:02:19.156+0000 7f09bd23a040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-21T23:02:19.181 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-02-21T23:02:19.192+0000 7f0d34c3e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-21T23:02:19.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.220+0000 7ff0e0600640 1 Processor -- start 2024-02-21T23:02:19.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.220+0000 7ff0e0600640 1 -- start start 2024-02-21T23:02:19.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.221+0000 7ff0e0600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff0d80c75d0 0x7ff0d80af680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:19.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.221+0000 7ff0e0600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7ff0d806d460 0x7ff0d815fad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:19.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.221+0000 7ff0e0600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0d8155230 con 0x7ff0d80c4d60 2024-02-21T23:02:19.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.221+0000 7ff0e0600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0d80c6960 con 0x7ff0d80c75d0 2024-02-21T23:02:19.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.221+0000 7ff0e0600640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0d80c6ca0 con 0x7ff0d806d460 2024-02-21T23:02:19.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.221+0000 7ff0de600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7ff0d806d460 0x7ff0d815fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:19.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.221+0000 7ff0de600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7ff0d806d460 0x7ff0d815fad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:52220/0 (socket says 172.21.15.81:52220) 2024-02-21T23:02:19.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.221+0000 7ff0de600640 1 -- 172.21.15.81:0/2657377092 learned_addr learned my addr 172.21.15.81:0/2657377092 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-02-21T23:02:19.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.221+0000 7ff0dd200640 1 --1- 172.21.15.81:0/2657377092 >> v1:172.21.15.81:6789/0 conn(0x7ff0d80c4d60 0x7ff0d806cd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:39984/0 (socket says 172.21.15.81:39984) 2024-02-21T23:02:19.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.222+0000 7ff0ddc00640 1 --2- 172.21.15.81:0/2657377092 >> v2:172.21.15.28:3300/0 conn(0x7ff0d80c75d0 0x7ff0d80af680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:19.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.222+0000 7ff0de600640 1 -- 172.21.15.81:0/2657377092 >> v2:172.21.15.28:3300/0 conn(0x7ff0d80c75d0 msgr2=0x7ff0d80af680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:19.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.222+0000 7ff0de600640 1 --2- 172.21.15.81:0/2657377092 >> v2:172.21.15.28:3300/0 conn(0x7ff0d80c75d0 0x7ff0d80af680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:19.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.222+0000 7ff0de600640 1 -- 172.21.15.81:0/2657377092 >> v1:172.21.15.81:6789/0 conn(0x7ff0d80c4d60 legacy=0x7ff0d806cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:19.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.222+0000 7ff0de600640 1 -- 172.21.15.81:0/2657377092 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0d80c6f80 con 0x7ff0d806d460 2024-02-21T23:02:19.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.224+0000 7ff0de600640 1 --2- 172.21.15.81:0/2657377092 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0d806d460 0x7ff0d815fad0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff0d80c8100 tx=0x7ff0d002bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aeb771c0603f238d server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:19.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.224+0000 7ff0d7e00640 1 -- 172.21.15.81:0/2657377092 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d002ca80 con 0x7ff0d806d460 2024-02-21T23:02:19.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.224+0000 7ff0d7e00640 1 -- 172.21.15.81:0/2657377092 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0d002cc20 con 0x7ff0d806d460 2024-02-21T23:02:19.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.224+0000 7ff0d7e00640 1 -- 172.21.15.81:0/2657377092 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d00343f0 con 0x7ff0d806d460 2024-02-21T23:02:19.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.225+0000 7ff0e0600640 1 -- 172.21.15.81:0/2657377092 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0d806d460 msgr2=0x7ff0d815fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:19.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.225+0000 7ff0e0600640 1 --2- 172.21.15.81:0/2657377092 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0d806d460 0x7ff0d815fad0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff0d80c8100 tx=0x7ff0d002bee0 comp rx=0 tx=0).stop 2024-02-21T23:02:19.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.225+0000 7ff0e0600640 1 -- 172.21.15.81:0/2657377092 shutdown_connections 2024-02-21T23:02:19.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.225+0000 7ff0e0600640 1 --2- 172.21.15.81:0/2657377092 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0d806d460 0x7ff0d815fad0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:19.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.225+0000 7ff0e0600640 1 --2- 172.21.15.81:0/2657377092 >> v2:172.21.15.28:3300/0 conn(0x7ff0d80c75d0 0x7ff0d80af680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:19.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.225+0000 7ff0e0600640 1 -- 172.21.15.81:0/2657377092 >> 172.21.15.81:0/2657377092 conn(0x7ff0d80725a0 msgr2=0x7ff0d80729b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:19.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.225+0000 7ff0e0600640 1 -- 172.21.15.81:0/2657377092 shutdown_connections 2024-02-21T23:02:19.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.225+0000 7ff0e0600640 1 -- 172.21.15.81:0/2657377092 wait complete. 2024-02-21T23:02:19.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.226+0000 7ff0e0600640 1 Processor -- start 2024-02-21T23:02:19.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.226+0000 7ff0e0600640 1 -- start start 2024-02-21T23:02:19.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.227+0000 7ff0e0600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0d80c4d60 0x7ff0d81e0410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:19.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.227+0000 7ff0e0600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0d80c75d0 0x7ff0d81eb540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:19.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.227+0000 7ff0e0600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff0d8159020 0x7ff0d80c97a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:19.219 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.227+0000 7ff0e0600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d8155a70 con 0x7ff0d80c4d60 2024-02-21T23:02:19.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.227+0000 7ff0e0600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d8155790 con 0x7ff0d80c75d0 2024-02-21T23:02:19.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.227+0000 7ff0e0600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d8155f90 con 0x7ff0d8159020 2024-02-21T23:02:19.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.227+0000 7ff0dd200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0d80c75d0 0x7ff0d81eb540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:19.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.228+0000 7ff0dd200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0d80c75d0 0x7ff0d81eb540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:52230/0 (socket says 172.21.15.81:52230) 2024-02-21T23:02:19.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.228+0000 7ff0dd200640 1 -- 172.21.15.81:0/2591549419 learned_addr learned my addr 172.21.15.81:0/2591549419 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-02-21T23:02:19.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.228+0000 7ff0de600640 1 --2- 172.21.15.81:0/2591549419 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff0d8159020 0x7ff0d80c97a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:19.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.228+0000 7ff0ddc00640 1 --2- 172.21.15.81:0/2591549419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0d80c4d60 0x7ff0d81e0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:19.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.228+0000 7ff0dd200640 1 -- 172.21.15.81:0/2591549419 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff0d8159020 msgr2=0x7ff0d80c97a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:19.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.228+0000 7ff0dd200640 1 --2- 172.21.15.81:0/2591549419 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff0d8159020 0x7ff0d80c97a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:19.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.228+0000 7ff0dd200640 1 -- 172.21.15.81:0/2591549419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0d80c4d60 msgr2=0x7ff0d81e0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:19.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.228+0000 7ff0dd200640 1 --2- 172.21.15.81:0/2591549419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0d80c4d60 0x7ff0d81e0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:19.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.228+0000 7ff0dd200640 1 -- 172.21.15.81:0/2591549419 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0d80c1100 con 0x7ff0d80c75d0 2024-02-21T23:02:19.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.228+0000 7ff0ddc00640 1 --2- 172.21.15.81:0/2591549419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0d80c4d60 0x7ff0d81e0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:02:19.220 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.228+0000 7ff0dd200640 1 --2- 172.21.15.81:0/2591549419 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0d80c75d0 0x7ff0d81eb540 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff0c800c400 tx=0x7ff0c800c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:19.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.229+0000 7ff0d5600640 1 -- 172.21.15.81:0/2591549419 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c80074e0 con 0x7ff0d80c75d0 2024-02-21T23:02:19.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.229+0000 7ff0d5600640 1 -- 172.21.15.81:0/2591549419 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0c8007680 con 0x7ff0d80c75d0 2024-02-21T23:02:19.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.229+0000 7ff0d5600640 1 -- 172.21.15.81:0/2591549419 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c8004a00 con 0x7ff0d80c75d0 2024-02-21T23:02:19.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.229+0000 7ff0e0600640 1 -- 172.21.15.81:0/2591549419 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0d81517d0 con 0x7ff0d80c75d0 2024-02-21T23:02:19.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.229+0000 7ff0e0600640 1 -- 172.21.15.81:0/2591549419 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0d8151d90 con 0x7ff0d80c75d0 2024-02-21T23:02:19.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.229+0000 7ff0d5600640 1 -- 172.21.15.81:0/2591549419 <== mon.1 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7ff0c801a020 con 0x7ff0d80c75d0 2024-02-21T23:02:19.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.229+0000 7ff0d5600640 1 -- 172.21.15.81:0/2591549419 <== mon.1 v2:172.21.15.81:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7ff0c801c5b0 con 0x7ff0d80c75d0 2024-02-21T23:02:19.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.230+0000 7ff0e0600640 1 -- 172.21.15.81:0/2591549419 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0d8068640 con 0x7ff0d80c75d0 2024-02-21T23:02:19.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.233+0000 7ff0d5600640 1 -- 172.21.15.81:0/2591549419 <== mon.1 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7ff0c8021070 con 0x7ff0d80c75d0 2024-02-21T23:02:19.233 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-02-21T23:02:19.244+0000 7f0d34c3e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-21T23:02:19.284 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-02-21T23:02:19.294+0000 7f09bd23a040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-21T23:02:19.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.312+0000 7ff0e0600640 1 -- 172.21.15.81:0/2591549419 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd new", "uuid": "f6ca1d98-5d2a-413d-870c-d7b9da1ff100", "id": 0} v 0) v1 -- 0x7ff0d8155010 con 0x7ff0d80c75d0 2024-02-21T23:02:19.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.319+0000 7ff0d5600640 1 -- 172.21.15.81:0/2591549419 <== mon.1 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f6ca1d98-5d2a-413d-870c-d7b9da1ff100", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7ff0d8155230 con 0x7ff0d80c75d0 2024-02-21T23:02:19.309 INFO:teuthology.orchestra.run.smithi081.stdout:0 2024-02-21T23:02:19.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.320+0000 7ff0e0600640 1 -- 172.21.15.81:0/2591549419 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0d80c75d0 msgr2=0x7ff0d81eb540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:19.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.320+0000 7ff0e0600640 1 --2- 172.21.15.81:0/2591549419 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0d80c75d0 0x7ff0d81eb540 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff0c800c400 tx=0x7ff0c800c8c0 comp rx=0 tx=0).stop 2024-02-21T23:02:19.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.321+0000 7ff0e0600640 1 -- 172.21.15.81:0/2591549419 shutdown_connections 2024-02-21T23:02:19.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.321+0000 7ff0e0600640 1 --2- 172.21.15.81:0/2591549419 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff0d8159020 0x7ff0d80c97a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:19.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.321+0000 7ff0e0600640 1 --2- 172.21.15.81:0/2591549419 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0d80c75d0 0x7ff0d81eb540 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:19.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.321+0000 7ff0e0600640 1 --2- 172.21.15.81:0/2591549419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0d80c4d60 0x7ff0d81e0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:19.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.321+0000 7ff0e0600640 1 -- 172.21.15.81:0/2591549419 >> 172.21.15.81:0/2591549419 conn(0x7ff0d80725a0 msgr2=0x7ff0d80c0190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:19.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.321+0000 7ff0e0600640 1 -- 172.21.15.81:0/2591549419 shutdown_connections 2024-02-21T23:02:19.312 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.321+0000 7ff0e0600640 1 -- 172.21.15.81:0/2591549419 wait complete. 2024-02-21T23:02:19.326 DEBUG:teuthology.orchestra.run.smithi081:> sudo ceph --cluster ceph osd new 2a25dd0c-4c28-4a86-a506-f195f981de12 1 2024-02-21T23:02:19.369 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-02-21T23:02:19.378+0000 7f09bd23a040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-21T23:02:19.370 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-02-21T23:02:19.381+0000 7f0d34c3e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-21T23:02:19.425 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-02-21T23:02:19.435+0000 7f09bd23a040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-21T23:02:19.454 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-02-21T23:02:19.465+0000 7f0d34c3e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-21T23:02:19.487 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-02-21T23:02:19.497+0000 7f09bd23a040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-21T23:02:19.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.518+0000 7f1363e00640 1 Processor -- start 2024-02-21T23:02:19.509 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.518+0000 7f1363e00640 1 -- start start 2024-02-21T23:02:19.510 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-02-21T23:02:19.521+0000 7f0d34c3e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-21T23:02:19.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.520+0000 7f1363e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f136406dbc0 0x7f136406c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:19.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.520+0000 7f1363e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1364158550 con 0x7f136413e520 2024-02-21T23:02:19.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.520+0000 7f1363e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13641587d0 con 0x7f136413e0c0 2024-02-21T23:02:19.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.520+0000 7f1363e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13640c89a0 con 0x7f136406dbc0 2024-02-21T23:02:19.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.520+0000 7f1362a00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f136413e0c0 0x7f13640c71b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:39992/0 (socket says 172.21.15.81:39992) 2024-02-21T23:02:19.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.520+0000 7f1362a00640 1 -- 172.21.15.81:0/933729456 learned_addr learned my addr 172.21.15.81:0/933729456 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-02-21T23:02:19.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.520+0000 7f1362000640 1 --2- 172.21.15.81:0/933729456 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f136406dbc0 0x7f136406c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:19.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.521+0000 7f1361600640 1 -- 172.21.15.81:0/933729456 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2619371130 0 0) 0x7f13641587d0 con 0x7f136413e0c0 2024-02-21T23:02:19.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.521+0000 7f1361600640 1 -- 172.21.15.81:0/933729456 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1348003520 con 0x7f136413e0c0 2024-02-21T23:02:19.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.521+0000 7f1362000640 1 -- 172.21.15.81:0/933729456 >> v1:172.21.15.81:6789/0 conn(0x7f136413e0c0 legacy=0x7f13640c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:19.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.521+0000 7f1362000640 1 -- 172.21.15.81:0/933729456 >> v1:172.21.15.28:6789/0 conn(0x7f136413e520 legacy=0x7f136406c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:19.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.521+0000 7f1362000640 1 -- 172.21.15.81:0/933729456 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1364158ab0 con 0x7f136406dbc0 2024-02-21T23:02:19.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.522+0000 7f1362000640 1 --2- 172.21.15.81:0/933729456 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f136406dbc0 0x7f136406c280 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f134c0097d0 tx=0x7f134c02b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a608874cdbb5f91c server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:19.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.523+0000 7f1361600640 1 -- 172.21.15.81:0/933729456 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f134c031770 con 0x7f136406dbc0 2024-02-21T23:02:19.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.523+0000 7f1361600640 1 -- 172.21.15.81:0/933729456 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f134c037070 con 0x7f136406dbc0 2024-02-21T23:02:19.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.523+0000 7f1361600640 1 -- 172.21.15.81:0/933729456 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f134c031b30 con 0x7f136406dbc0 2024-02-21T23:02:19.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.523+0000 7f1363e00640 1 -- 172.21.15.81:0/933729456 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f136406dbc0 msgr2=0x7f136406c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:19.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.523+0000 7f1363e00640 1 --2- 172.21.15.81:0/933729456 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f136406dbc0 0x7f136406c280 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f134c0097d0 tx=0x7f134c02b810 comp rx=0 tx=0).stop 2024-02-21T23:02:19.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.523+0000 7f1363e00640 1 -- 172.21.15.81:0/933729456 shutdown_connections 2024-02-21T23:02:19.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.523+0000 7f1363e00640 1 --2- 172.21.15.81:0/933729456 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f136406dbc0 0x7f136406c280 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:19.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.523+0000 7f1363e00640 1 -- 172.21.15.81:0/933729456 >> 172.21.15.81:0/933729456 conn(0x7f1364073110 msgr2=0x7f13640ac490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:19.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.523+0000 7f1363e00640 1 -- 172.21.15.81:0/933729456 shutdown_connections 2024-02-21T23:02:19.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.523+0000 7f1363e00640 1 -- 172.21.15.81:0/933729456 wait complete. 2024-02-21T23:02:19.518 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.525+0000 7f1363e00640 1 Processor -- start 2024-02-21T23:02:19.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.525+0000 7f1363e00640 1 -- start start 2024-02-21T23:02:19.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.526+0000 7f1363e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f136406dbc0 0x7f1364135530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:19.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.526+0000 7f1363e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f136413e0c0 0x7f13640ab5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:19.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.526+0000 7f1363e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f136413e520 0x7f13641df940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:19.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.526+0000 7f1363e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f136406ced0 con 0x7f136413e0c0 2024-02-21T23:02:19.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.526+0000 7f1363e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1364159070 con 0x7f136406dbc0 2024-02-21T23:02:19.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.526+0000 7f1363e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13640c78c0 con 0x7f136413e520 2024-02-21T23:02:19.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.526+0000 7f1363400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f136413e520 0x7f13641df940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:19.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.526+0000 7f1363400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f136413e520 0x7f13641df940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3301/0 says I am v2:172.21.15.81:55988/0 (socket says 172.21.15.81:55988) 2024-02-21T23:02:19.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.526+0000 7f1363400640 1 -- 172.21.15.81:0/1498842967 learned_addr learned my addr 172.21.15.81:0/1498842967 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-02-21T23:02:19.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.526+0000 7f1362a00640 1 --2- 172.21.15.81:0/1498842967 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f136406dbc0 0x7f1364135530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:19.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.526+0000 7f1362000640 1 --2- 172.21.15.81:0/1498842967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f136413e0c0 0x7f13640ab5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:19.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.527+0000 7f1363400640 1 -- 172.21.15.81:0/1498842967 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f136406dbc0 msgr2=0x7f1364135530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:19.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.527+0000 7f1363400640 1 --2- 172.21.15.81:0/1498842967 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f136406dbc0 0x7f1364135530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:19.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.527+0000 7f1363400640 1 -- 172.21.15.81:0/1498842967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f136413e0c0 msgr2=0x7f13640ab5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:19.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.527+0000 7f1363400640 1 --2- 172.21.15.81:0/1498842967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f136413e0c0 0x7f13640ab5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:19.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.527+0000 7f1363400640 1 -- 172.21.15.81:0/1498842967 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13641df690 con 0x7f136413e520 2024-02-21T23:02:19.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.527+0000 7f1363400640 1 --2- 172.21.15.81:0/1498842967 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f136413e520 0x7f13641df940 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f135c00c7b0 tx=0x7f135c00cc70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:19.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.527+0000 7f135aa00640 1 -- 172.21.15.81:0/1498842967 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f135c004b60 con 0x7f136413e520 2024-02-21T23:02:19.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.527+0000 7f135aa00640 1 -- 172.21.15.81:0/1498842967 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f135c004d00 con 0x7f136413e520 2024-02-21T23:02:19.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.527+0000 7f1363e00640 1 -- 172.21.15.81:0/1498842967 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13641de5f0 con 0x7f136413e520 2024-02-21T23:02:19.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.527+0000 7f135aa00640 1 -- 172.21.15.81:0/1498842967 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f135c00aa90 con 0x7f136413e520 2024-02-21T23:02:19.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.527+0000 7f1363e00640 1 -- 172.21.15.81:0/1498842967 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13641debb0 con 0x7f136413e520 2024-02-21T23:02:19.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.528+0000 7f135aa00640 1 -- 172.21.15.81:0/1498842967 <== mon.2 v2:172.21.15.81:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f135c006b50 con 0x7f136413e520 2024-02-21T23:02:19.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.528+0000 7f135aa00640 1 -- 172.21.15.81:0/1498842967 <== mon.2 v2:172.21.15.81:3301/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f135c0071a0 con 0x7f136413e520 2024-02-21T23:02:19.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.528+0000 7f1363e00640 1 -- 172.21.15.81:0/1498842967 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13641587d0 con 0x7f136413e520 2024-02-21T23:02:19.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.531+0000 7f135aa00640 1 -- 172.21.15.81:0/1498842967 <== mon.2 v2:172.21.15.81:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f13641587d0 con 0x7f136413e520 2024-02-21T23:02:19.545 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-02-21T23:02:19.555+0000 7f09bd23a040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-21T23:02:19.763 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-02-21T23:02:19.584+0000 7f0d34c3e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-21T23:02:19.763 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-02-21T23:02:19.643+0000 7f0d34c3e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-21T23:02:19.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.611+0000 7f1363e00640 1 -- 172.21.15.81:0/1498842967 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_command({"prefix": "osd new", "uuid": "2a25dd0c-4c28-4a86-a506-f195f981de12", "id": 1} v 0) v1 -- 0x7f136404f290 con 0x7f136413e520 2024-02-21T23:02:19.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.616+0000 7f135aa00640 1 -- 172.21.15.81:0/1498842967 <== mon.2 v2:172.21.15.81:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2a25dd0c-4c28-4a86-a506-f195f981de12", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f135c020070 con 0x7f136413e520 2024-02-21T23:02:19.764 INFO:teuthology.orchestra.run.smithi081.stdout:1 2024-02-21T23:02:19.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.618+0000 7f1363e00640 1 -- 172.21.15.81:0/1498842967 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f136413e520 msgr2=0x7f13641df940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:19.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.618+0000 7f1363e00640 1 --2- 172.21.15.81:0/1498842967 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f136413e520 0x7f13641df940 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f135c00c7b0 tx=0x7f135c00cc70 comp rx=0 tx=0).stop 2024-02-21T23:02:19.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.618+0000 7f1363e00640 1 -- 172.21.15.81:0/1498842967 shutdown_connections 2024-02-21T23:02:19.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.618+0000 7f1363e00640 1 --2- 172.21.15.81:0/1498842967 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f136413e520 0x7f13641df940 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:19.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.618+0000 7f1363e00640 1 --2- 172.21.15.81:0/1498842967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f136413e0c0 0x7f13640ab5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:19.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.618+0000 7f1363e00640 1 --2- 172.21.15.81:0/1498842967 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f136406dbc0 0x7f1364135530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:19.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.618+0000 7f1363e00640 1 -- 172.21.15.81:0/1498842967 >> 172.21.15.81:0/1498842967 conn(0x7f1364073110 msgr2=0x7f13640c5020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:19.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.618+0000 7f1363e00640 1 -- 172.21.15.81:0/1498842967 shutdown_connections 2024-02-21T23:02:19.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:19.618+0000 7f1363e00640 1 -- 172.21.15.81:0/1498842967 wait complete. 2024-02-21T23:02:19.766 DEBUG:teuthology.orchestra.run.smithi081:> sudo ceph --cluster ceph osd new 96493c37-efc1-481f-8e86-972e59ccac3f 2 2024-02-21T23:02:20.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.012+0000 7f952a200640 1 Processor -- start 2024-02-21T23:02:20.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.012+0000 7f952a200640 1 -- start start 2024-02-21T23:02:20.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.014+0000 7f952a200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f952413e0c0 0x7f95240c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:20.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.014+0000 7f952a200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f952406dbc0 0x7f952406c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:20.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.014+0000 7f952a200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9524158550 con 0x7f95240c76f0 2024-02-21T23:02:20.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.014+0000 7f952a200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95240c89a0 con 0x7f952406dbc0 2024-02-21T23:02:20.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.014+0000 7f952a200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95241587d0 con 0x7f952413e0c0 2024-02-21T23:02:20.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.014+0000 7f9528e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f952413e0c0 0x7f95240c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:20.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.014+0000 7f9523e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f952406dbc0 0x7f952406c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:20.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.014+0000 7f9528e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f952413e0c0 0x7f95240c71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3301/0 says I am v2:172.21.15.81:55998/0 (socket says 172.21.15.81:55998) 2024-02-21T23:02:20.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.014+0000 7f9528e00640 1 -- 172.21.15.81:0/1008772429 learned_addr learned my addr 172.21.15.81:0/1008772429 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-02-21T23:02:20.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.015+0000 7f9523e00640 1 -- 172.21.15.81:0/1008772429 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f952413e0c0 msgr2=0x7f95240c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:20.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.015+0000 7f9523e00640 1 --2- 172.21.15.81:0/1008772429 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f952413e0c0 0x7f95240c71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:20.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.015+0000 7f9523e00640 1 -- 172.21.15.81:0/1008772429 >> v1:172.21.15.28:6789/0 conn(0x7f95240c76f0 legacy=0x7f952406c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:20.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.015+0000 7f9523e00640 1 -- 172.21.15.81:0/1008772429 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9524158ab0 con 0x7f952406dbc0 2024-02-21T23:02:20.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.016+0000 7f9523e00640 1 --2- 172.21.15.81:0/1008772429 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f952406dbc0 0x7f952406c280 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f95100097a0 tx=0x7f951002b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=757e582b9da57fd6 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:20.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.016+0000 7f9523400640 1 -- 172.21.15.81:0/1008772429 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9510039070 con 0x7f952406dbc0 2024-02-21T23:02:20.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.016+0000 7f9523400640 1 -- 172.21.15.81:0/1008772429 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95100316f0 con 0x7f952406dbc0 2024-02-21T23:02:20.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.016+0000 7f9523400640 1 -- 172.21.15.81:0/1008772429 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9510031ac0 con 0x7f952406dbc0 2024-02-21T23:02:20.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.017+0000 7f952a200640 1 -- 172.21.15.81:0/1008772429 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f952406dbc0 msgr2=0x7f952406c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:20.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.017+0000 7f952a200640 1 --2- 172.21.15.81:0/1008772429 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f952406dbc0 0x7f952406c280 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f95100097a0 tx=0x7f951002b790 comp rx=0 tx=0).stop 2024-02-21T23:02:20.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.017+0000 7f952a200640 1 -- 172.21.15.81:0/1008772429 shutdown_connections 2024-02-21T23:02:20.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.017+0000 7f952a200640 1 --2- 172.21.15.81:0/1008772429 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f952406dbc0 0x7f952406c280 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:20.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.017+0000 7f952a200640 1 --2- 172.21.15.81:0/1008772429 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f952413e0c0 0x7f95240c71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:20.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.017+0000 7f952a200640 1 -- 172.21.15.81:0/1008772429 >> 172.21.15.81:0/1008772429 conn(0x7f9524073110 msgr2=0x7f95240ac490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:20.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.017+0000 7f952a200640 1 -- 172.21.15.81:0/1008772429 shutdown_connections 2024-02-21T23:02:20.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.017+0000 7f952a200640 1 -- 172.21.15.81:0/1008772429 wait complete. 2024-02-21T23:02:20.008 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-02-21T23:02:20.018+0000 7f09bd23a040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-21T23:02:20.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.019+0000 7f952a200640 1 Processor -- start 2024-02-21T23:02:20.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.019+0000 7f952a200640 1 -- start start 2024-02-21T23:02:20.011 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.020+0000 7f952a200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f952406dbc0 0x7f9524151df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:20.011 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.020+0000 7f952a200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95240c76f0 0x7f95241554f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:20.011 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.020+0000 7f952a200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f952413e0c0 0x7f9524155a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:20.011 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.020+0000 7f952a200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f952406ced0 con 0x7f95240c76f0 2024-02-21T23:02:20.011 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.020+0000 7f952a200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9524159070 con 0x7f952413e0c0 2024-02-21T23:02:20.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.020+0000 7f952a200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f952413e520 con 0x7f952406dbc0 2024-02-21T23:02:20.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.020+0000 7f9528e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f952406dbc0 0x7f9524151df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:20.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.020+0000 7f9529800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f952413e0c0 0x7f9524155a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:20.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.020+0000 7f9528e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f952406dbc0 0x7f9524151df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3301/0 says I am v2:172.21.15.81:56008/0 (socket says 172.21.15.81:56008) 2024-02-21T23:02:20.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.020+0000 7f9528e00640 1 -- 172.21.15.81:0/1508348164 learned_addr learned my addr 172.21.15.81:0/1508348164 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-02-21T23:02:20.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.020+0000 7f9523e00640 1 --2- 172.21.15.81:0/1508348164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95240c76f0 0x7f95241554f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:20.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.020+0000 7f9529800640 1 -- 172.21.15.81:0/1508348164 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f952406dbc0 msgr2=0x7f9524151df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:20.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.020+0000 7f9529800640 1 --2- 172.21.15.81:0/1508348164 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f952406dbc0 0x7f9524151df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:20.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.020+0000 7f9529800640 1 -- 172.21.15.81:0/1508348164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95240c76f0 msgr2=0x7f95241554f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:20.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.020+0000 7f9529800640 1 --2- 172.21.15.81:0/1508348164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95240c76f0 0x7f95241554f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:20.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.020+0000 7f9529800640 1 -- 172.21.15.81:0/1508348164 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9524152330 con 0x7f952413e0c0 2024-02-21T23:02:20.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.021+0000 7f9529800640 1 --2- 172.21.15.81:0/1508348164 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f952413e0c0 0x7f9524155a30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f951c00c780 tx=0x7f951c00cc40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:20.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.021+0000 7f9520c00640 1 -- 172.21.15.81:0/1508348164 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f951c016020 con 0x7f952413e0c0 2024-02-21T23:02:20.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.021+0000 7f9520c00640 1 -- 172.21.15.81:0/1508348164 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f951c00aa50 con 0x7f952413e0c0 2024-02-21T23:02:20.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.021+0000 7f9520c00640 1 -- 172.21.15.81:0/1508348164 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f951c00ae40 con 0x7f952413e0c0 2024-02-21T23:02:20.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.021+0000 7f952a200640 1 -- 172.21.15.81:0/1508348164 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9524157fd0 con 0x7f952413e0c0 2024-02-21T23:02:20.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.021+0000 7f952a200640 1 -- 172.21.15.81:0/1508348164 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95241539a0 con 0x7f952413e0c0 2024-02-21T23:02:20.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.021+0000 7f9520c00640 1 -- 172.21.15.81:0/1508348164 <== mon.1 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f951c019020 con 0x7f952413e0c0 2024-02-21T23:02:20.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.021+0000 7f9520c00640 1 -- 172.21.15.81:0/1508348164 <== mon.1 v2:172.21.15.81:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7f951c005440 con 0x7f952413e0c0 2024-02-21T23:02:20.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.022+0000 7f952a200640 1 -- 172.21.15.81:0/1508348164 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f952413c7b0 con 0x7f952413e0c0 2024-02-21T23:02:20.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.026+0000 7f9520c00640 1 -- 172.21.15.81:0/1508348164 <== mon.1 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f952413c7b0 con 0x7f952413e0c0 2024-02-21T23:02:20.095 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.105+0000 7f952a200640 1 -- 172.21.15.81:0/1508348164 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd new", "uuid": "96493c37-efc1-481f-8e86-972e59ccac3f", "id": 2} v 0) v1 -- 0x7f952404f290 con 0x7f952413e0c0 2024-02-21T23:02:20.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.111+0000 7f9520c00640 1 -- 172.21.15.81:0/1508348164 <== mon.1 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "96493c37-efc1-481f-8e86-972e59ccac3f", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f951c020030 con 0x7f952413e0c0 2024-02-21T23:02:20.101 INFO:teuthology.orchestra.run.smithi081.stdout:2 2024-02-21T23:02:20.102 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.112+0000 7f952a200640 1 -- 172.21.15.81:0/1508348164 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f952413e0c0 msgr2=0x7f9524155a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:20.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.112+0000 7f952a200640 1 --2- 172.21.15.81:0/1508348164 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f952413e0c0 0x7f9524155a30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f951c00c780 tx=0x7f951c00cc40 comp rx=0 tx=0).stop 2024-02-21T23:02:20.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.112+0000 7f952a200640 1 -- 172.21.15.81:0/1508348164 shutdown_connections 2024-02-21T23:02:20.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.112+0000 7f952a200640 1 --2- 172.21.15.81:0/1508348164 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f952413e0c0 0x7f9524155a30 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:20.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.112+0000 7f952a200640 1 --2- 172.21.15.81:0/1508348164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95240c76f0 0x7f95241554f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:20.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.112+0000 7f952a200640 1 --2- 172.21.15.81:0/1508348164 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f952406dbc0 0x7f9524151df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:20.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.112+0000 7f952a200640 1 -- 172.21.15.81:0/1508348164 >> 172.21.15.81:0/1508348164 conn(0x7f9524073110 msgr2=0x7f95240c5020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:20.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.112+0000 7f952a200640 1 -- 172.21.15.81:0/1508348164 shutdown_connections 2024-02-21T23:02:20.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.112+0000 7f952a200640 1 -- 172.21.15.81:0/1508348164 wait complete. 2024-02-21T23:02:20.103 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-02-21T23:02:20.113+0000 7f0d34c3e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-21T23:02:20.119 DEBUG:teuthology.orchestra.run.smithi081:> sudo ceph --cluster ceph osd new 6654ebd8-3251-43f7-8730-4db82e869c98 3 2024-02-21T23:02:20.178 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-02-21T23:02:20.188+0000 7f09bd23a040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-21T23:02:20.270 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-02-21T23:02:20.281+0000 7f0d34c3e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-21T23:02:20.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.311+0000 7f5e7e200640 1 Processor -- start 2024-02-21T23:02:20.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.311+0000 7f5e7e200640 1 -- start start 2024-02-21T23:02:20.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.313+0000 7f5e7e200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f5e7813e0c0 0x7f5e780c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:20.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.313+0000 7f5e7e200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5e780c76f0 0x7f5e7806c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:20.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.313+0000 7f5e7e200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e78158550 con 0x7f5e7806dbc0 2024-02-21T23:02:20.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.313+0000 7f5e7e200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e780c89a0 con 0x7f5e780c76f0 2024-02-21T23:02:20.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.313+0000 7f5e7e200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e781587d0 con 0x7f5e7813e0c0 2024-02-21T23:02:20.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.313+0000 7f5e7ce00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f5e7813e0c0 0x7f5e780c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:20.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.313+0000 7f5e7ce00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f5e7813e0c0 0x7f5e780c71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:52272/0 (socket says 172.21.15.81:52272) 2024-02-21T23:02:20.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.313+0000 7f5e7ce00640 1 -- 172.21.15.81:0/455308748 learned_addr learned my addr 172.21.15.81:0/455308748 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-02-21T23:02:20.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.314+0000 7f5e7d800640 1 --2- 172.21.15.81:0/455308748 >> v2:172.21.15.28:3300/0 conn(0x7f5e780c76f0 0x7f5e7806c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:20.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.314+0000 7f5e7ce00640 1 -- 172.21.15.81:0/455308748 >> v2:172.21.15.28:3300/0 conn(0x7f5e780c76f0 msgr2=0x7f5e7806c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:20.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.314+0000 7f5e7ce00640 1 --2- 172.21.15.81:0/455308748 >> v2:172.21.15.28:3300/0 conn(0x7f5e780c76f0 0x7f5e7806c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:20.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.314+0000 7f5e7ce00640 1 -- 172.21.15.81:0/455308748 >> v1:172.21.15.81:6789/0 conn(0x7f5e7806dbc0 legacy=0x7f5e7806c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:20.305 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.314+0000 7f5e7ce00640 1 -- 172.21.15.81:0/455308748 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e78158ab0 con 0x7f5e7813e0c0 2024-02-21T23:02:20.305 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.315+0000 7f5e7ce00640 1 --2- 172.21.15.81:0/455308748 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e7813e0c0 0x7f5e780c71b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5e780ad180 tx=0x7f5e6802b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cc663dd56735825e server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:20.305 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.315+0000 7f5e77e00640 1 -- 172.21.15.81:0/455308748 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e68035020 con 0x7f5e7813e0c0 2024-02-21T23:02:20.305 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.315+0000 7f5e77e00640 1 -- 172.21.15.81:0/455308748 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e68031870 con 0x7f5e7813e0c0 2024-02-21T23:02:20.305 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.315+0000 7f5e77e00640 1 -- 172.21.15.81:0/455308748 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e68031c40 con 0x7f5e7813e0c0 2024-02-21T23:02:20.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.316+0000 7f5e7e200640 1 -- 172.21.15.81:0/455308748 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e7813e0c0 msgr2=0x7f5e780c71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:20.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.316+0000 7f5e7e200640 1 --2- 172.21.15.81:0/455308748 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e7813e0c0 0x7f5e780c71b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5e780ad180 tx=0x7f5e6802b910 comp rx=0 tx=0).stop 2024-02-21T23:02:20.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.316+0000 7f5e7e200640 1 -- 172.21.15.81:0/455308748 shutdown_connections 2024-02-21T23:02:20.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.316+0000 7f5e7e200640 1 --2- 172.21.15.81:0/455308748 >> v2:172.21.15.28:3300/0 conn(0x7f5e780c76f0 0x7f5e7806c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:20.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.316+0000 7f5e7e200640 1 --2- 172.21.15.81:0/455308748 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e7813e0c0 0x7f5e780c71b0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:20.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.316+0000 7f5e7e200640 1 -- 172.21.15.81:0/455308748 >> 172.21.15.81:0/455308748 conn(0x7f5e78073110 msgr2=0x7f5e780ac490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:20.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.316+0000 7f5e7e200640 1 -- 172.21.15.81:0/455308748 shutdown_connections 2024-02-21T23:02:20.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.316+0000 7f5e7e200640 1 -- 172.21.15.81:0/455308748 wait complete. 2024-02-21T23:02:20.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.317+0000 7f5e7e200640 1 Processor -- start 2024-02-21T23:02:20.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.318+0000 7f5e7e200640 1 -- start start 2024-02-21T23:02:20.310 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.318+0000 7f5e7e200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5e7806dbc0 0x7f5e78138f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:20.310 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.318+0000 7f5e7e200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e780c76f0 0x7f5e78151c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:20.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.318+0000 7f5e7e200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e7813e0c0 0x7f5e78152160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:20.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.318+0000 7f5e7e200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e7806ced0 con 0x7f5e780c76f0 2024-02-21T23:02:20.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.318+0000 7f5e7e200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e78159070 con 0x7f5e7813e0c0 2024-02-21T23:02:20.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.318+0000 7f5e7e200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e7813e520 con 0x7f5e7806dbc0 2024-02-21T23:02:20.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.319+0000 7f5e7d800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e7813e0c0 0x7f5e78152160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:20.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.319+0000 7f5e7ce00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5e7806dbc0 0x7f5e78138f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:20.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.319+0000 7f5e7d800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e7813e0c0 0x7f5e78152160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:52280/0 (socket says 172.21.15.81:52280) 2024-02-21T23:02:20.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.319+0000 7f5e7d800640 1 -- 172.21.15.81:0/3644548744 learned_addr learned my addr 172.21.15.81:0/3644548744 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-02-21T23:02:20.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.319+0000 7f5e6fe00640 1 --2- 172.21.15.81:0/3644548744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e780c76f0 0x7f5e78151c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:20.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.319+0000 7f5e7ce00640 1 -- 172.21.15.81:0/3644548744 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e7813e0c0 msgr2=0x7f5e78152160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:20.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.319+0000 7f5e7ce00640 1 --2- 172.21.15.81:0/3644548744 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e7813e0c0 0x7f5e78152160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:20.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.319+0000 7f5e7ce00640 1 -- 172.21.15.81:0/3644548744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e780c76f0 msgr2=0x7f5e78151c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:20.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.319+0000 7f5e7ce00640 1 --2- 172.21.15.81:0/3644548744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e780c76f0 0x7f5e78151c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:20.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.319+0000 7f5e7ce00640 1 -- 172.21.15.81:0/3644548744 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e781eecd0 con 0x7f5e7806dbc0 2024-02-21T23:02:20.312 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.319+0000 7f5e6fe00640 1 --2- 172.21.15.81:0/3644548744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e780c76f0 0x7f5e78151c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:02:20.312 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.319+0000 7f5e7d800640 1 --2- 172.21.15.81:0/3644548744 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e7813e0c0 0x7f5e78152160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-21T23:02:20.312 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.319+0000 7f5e7ce00640 1 --2- 172.21.15.81:0/3644548744 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5e7806dbc0 0x7f5e78138f40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5e68033000 tx=0x7f5e680046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:20.312 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.320+0000 7f5e75600640 1 -- 172.21.15.81:0/3644548744 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e68040070 con 0x7f5e7806dbc0 2024-02-21T23:02:20.312 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.320+0000 7f5e75600640 1 -- 172.21.15.81:0/3644548744 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e68030d00 con 0x7f5e7806dbc0 2024-02-21T23:02:20.312 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.320+0000 7f5e75600640 1 -- 172.21.15.81:0/3644548744 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e68035040 con 0x7f5e7806dbc0 2024-02-21T23:02:20.312 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.320+0000 7f5e7e200640 1 -- 172.21.15.81:0/3644548744 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e780af710 con 0x7f5e7806dbc0 2024-02-21T23:02:20.312 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.320+0000 7f5e7e200640 1 -- 172.21.15.81:0/3644548744 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e780afcd0 con 0x7f5e7806dbc0 2024-02-21T23:02:20.312 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.320+0000 7f5e75600640 1 -- 172.21.15.81:0/3644548744 <== mon.2 v2:172.21.15.81:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f5e6804b060 con 0x7f5e7806dbc0 2024-02-21T23:02:20.312 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.320+0000 7f5e75600640 1 -- 172.21.15.81:0/3644548744 <== mon.2 v2:172.21.15.81:3301/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f5e68044400 con 0x7f5e7806dbc0 2024-02-21T23:02:20.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.321+0000 7f5e7e200640 1 -- 172.21.15.81:0/3644548744 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e7804f290 con 0x7f5e7806dbc0 2024-02-21T23:02:20.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.323+0000 7f5e75600640 1 -- 172.21.15.81:0/3644548744 <== mon.2 v2:172.21.15.81:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f5e6803a9f0 con 0x7f5e7806dbc0 2024-02-21T23:02:20.323 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-02-21T23:02:20.334+0000 7f09bd23a040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-21T23:02:20.377 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-02-21T23:02:20.387+0000 7f09bd23a040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-21T23:02:20.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.402+0000 7f5e7e200640 1 -- 172.21.15.81:0/3644548744 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_command({"prefix": "osd new", "uuid": "6654ebd8-3251-43f7-8730-4db82e869c98", "id": 3} v 0) v1 -- 0x7f5e7806d9a0 con 0x7f5e7806dbc0 2024-02-21T23:02:20.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.409+0000 7f5e75600640 1 -- 172.21.15.81:0/3644548744 <== mon.2 v2:172.21.15.81:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6654ebd8-3251-43f7-8730-4db82e869c98", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f5e6803acd0 con 0x7f5e7806dbc0 2024-02-21T23:02:20.400 INFO:teuthology.orchestra.run.smithi081.stdout:3 2024-02-21T23:02:20.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.411+0000 7f5e7e200640 1 -- 172.21.15.81:0/3644548744 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5e7806dbc0 msgr2=0x7f5e78138f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:20.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.411+0000 7f5e7e200640 1 --2- 172.21.15.81:0/3644548744 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5e7806dbc0 0x7f5e78138f40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5e68033000 tx=0x7f5e680046c0 comp rx=0 tx=0).stop 2024-02-21T23:02:20.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.411+0000 7f5e7e200640 1 -- 172.21.15.81:0/3644548744 shutdown_connections 2024-02-21T23:02:20.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.411+0000 7f5e7e200640 1 --2- 172.21.15.81:0/3644548744 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e7813e0c0 0x7f5e78152160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:20.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.411+0000 7f5e7e200640 1 --2- 172.21.15.81:0/3644548744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e780c76f0 0x7f5e78151c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:20.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.411+0000 7f5e7e200640 1 --2- 172.21.15.81:0/3644548744 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5e7806dbc0 0x7f5e78138f40 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:20.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.411+0000 7f5e7e200640 1 -- 172.21.15.81:0/3644548744 >> 172.21.15.81:0/3644548744 conn(0x7f5e78073110 msgr2=0x7f5e780c8fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:20.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.411+0000 7f5e7e200640 1 -- 172.21.15.81:0/3644548744 shutdown_connections 2024-02-21T23:02:20.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.411+0000 7f5e7e200640 1 -- 172.21.15.81:0/3644548744 wait complete. 2024-02-21T23:02:20.402 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-02-21T23:02:20.413+0000 7f0d34c3e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-21T23:02:20.418 DEBUG:teuthology.orchestra.run.smithi081:> sudo ceph --cluster ceph osd new 6ac7806e-a5b0-4786-898a-b31607939493 4 2024-02-21T23:02:20.457 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-02-21T23:02:20.468+0000 7f0d34c3e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-21T23:02:20.512 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-02-21T23:02:20.521+0000 7f09bd23a040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-21T23:02:20.589 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-02-21T23:02:20.601+0000 7f0d34c3e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-21T23:02:20.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.619+0000 7fb239000640 1 Processor -- start 2024-02-21T23:02:20.609 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.619+0000 7fb239000640 1 -- start start 2024-02-21T23:02:20.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.620+0000 7fb239000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fb23413e0c0 0x7fb2340c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:20.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.620+0000 7fb239000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb23406dbc0 0x7fb23406c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:20.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.620+0000 7fb239000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb234158550 con 0x7fb2340c76f0 2024-02-21T23:02:20.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.620+0000 7fb239000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2340c89a0 con 0x7fb23413e0c0 2024-02-21T23:02:20.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.620+0000 7fb239000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2341587d0 con 0x7fb23406dbc0 2024-02-21T23:02:20.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.620+0000 7fb233400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fb23413e0c0 0x7fb2340c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:20.611 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.621+0000 7fb232a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb23406dbc0 0x7fb23406c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:20.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.621+0000 7fb233400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fb23413e0c0 0x7fb2340c71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:52288/0 (socket says 172.21.15.81:52288) 2024-02-21T23:02:20.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.621+0000 7fb233400640 1 -- 172.21.15.81:0/3795073081 learned_addr learned my addr 172.21.15.81:0/3795073081 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-02-21T23:02:20.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.621+0000 7fb233400640 1 -- 172.21.15.81:0/3795073081 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb23406dbc0 msgr2=0x7fb23406c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:20.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.621+0000 7fb233400640 1 --2- 172.21.15.81:0/3795073081 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb23406dbc0 0x7fb23406c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:20.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.621+0000 7fb233400640 1 -- 172.21.15.81:0/3795073081 >> v1:172.21.15.28:6789/0 conn(0x7fb2340c76f0 legacy=0x7fb23406c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:20.612 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.621+0000 7fb233400640 1 -- 172.21.15.81:0/3795073081 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb234158ab0 con 0x7fb23413e0c0 2024-02-21T23:02:20.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.622+0000 7fb233400640 1 --2- 172.21.15.81:0/3795073081 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb23413e0c0 0x7fb2340c71b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb224009870 tx=0x7fb22402b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc614afffd84208b server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:20.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.622+0000 7fb232000640 1 -- 172.21.15.81:0/3795073081 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2240317c0 con 0x7fb23413e0c0 2024-02-21T23:02:20.613 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.622+0000 7fb232000640 1 -- 172.21.15.81:0/3795073081 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb224037070 con 0x7fb23413e0c0 2024-02-21T23:02:20.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.622+0000 7fb232000640 1 -- 172.21.15.81:0/3795073081 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb224031b80 con 0x7fb23413e0c0 2024-02-21T23:02:20.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.623+0000 7fb239000640 1 -- 172.21.15.81:0/3795073081 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb23413e0c0 msgr2=0x7fb2340c71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:20.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.623+0000 7fb239000640 1 --2- 172.21.15.81:0/3795073081 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb23413e0c0 0x7fb2340c71b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb224009870 tx=0x7fb22402b860 comp rx=0 tx=0).stop 2024-02-21T23:02:20.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.623+0000 7fb239000640 1 -- 172.21.15.81:0/3795073081 shutdown_connections 2024-02-21T23:02:20.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.623+0000 7fb239000640 1 --2- 172.21.15.81:0/3795073081 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb23406dbc0 0x7fb23406c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:20.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.623+0000 7fb239000640 1 --2- 172.21.15.81:0/3795073081 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb23413e0c0 0x7fb2340c71b0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:20.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.623+0000 7fb239000640 1 -- 172.21.15.81:0/3795073081 >> 172.21.15.81:0/3795073081 conn(0x7fb234073110 msgr2=0x7fb2340ac490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:20.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.623+0000 7fb239000640 1 -- 172.21.15.81:0/3795073081 shutdown_connections 2024-02-21T23:02:20.614 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.623+0000 7fb239000640 1 -- 172.21.15.81:0/3795073081 wait complete. 2024-02-21T23:02:20.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.625+0000 7fb239000640 1 Processor -- start 2024-02-21T23:02:20.615 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.625+0000 7fb239000640 1 -- start start 2024-02-21T23:02:20.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.626+0000 7fb239000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb23406dbc0 0x7fb2340ab5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:20.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.626+0000 7fb239000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb2340c76f0 0x7fb234151920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:20.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.626+0000 7fb239000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb23413e0c0 0x7fb234151e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:20.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.626+0000 7fb239000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb23406ced0 con 0x7fb23406dbc0 2024-02-21T23:02:20.618 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.626+0000 7fb239000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb234159070 con 0x7fb23413e0c0 2024-02-21T23:02:20.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.626+0000 7fb239000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb23413e520 con 0x7fb2340c76f0 2024-02-21T23:02:20.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.626+0000 7fb232a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb2340c76f0 0x7fb234151920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:20.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.626+0000 7fb233e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb23413e0c0 0x7fb234151e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:20.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.626+0000 7fb232a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb2340c76f0 0x7fb234151920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3301/0 says I am v2:172.21.15.81:56040/0 (socket says 172.21.15.81:56040) 2024-02-21T23:02:20.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.626+0000 7fb232a00640 1 -- 172.21.15.81:0/2314686423 learned_addr learned my addr 172.21.15.81:0/2314686423 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-02-21T23:02:20.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.626+0000 7fb233400640 1 --2- 172.21.15.81:0/2314686423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb23406dbc0 0x7fb2340ab5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:20.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.627+0000 7fb232a00640 1 -- 172.21.15.81:0/2314686423 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb23413e0c0 msgr2=0x7fb234151e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:20.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.627+0000 7fb232a00640 1 --2- 172.21.15.81:0/2314686423 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb23413e0c0 0x7fb234151e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:20.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.627+0000 7fb232a00640 1 -- 172.21.15.81:0/2314686423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb23406dbc0 msgr2=0x7fb2340ab5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:20.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.627+0000 7fb232a00640 1 --2- 172.21.15.81:0/2314686423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb23406dbc0 0x7fb2340ab5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:20.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.627+0000 7fb232a00640 1 -- 172.21.15.81:0/2314686423 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2341e0640 con 0x7fb2340c76f0 2024-02-21T23:02:20.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.627+0000 7fb233e00640 1 --2- 172.21.15.81:0/2314686423 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb23413e0c0 0x7fb234151e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-21T23:02:20.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.627+0000 7fb232a00640 1 --2- 172.21.15.81:0/2314686423 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb2340c76f0 0x7fb234151920 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb22000e9b0 tx=0x7fb22000ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:20.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.627+0000 7fb22b400640 1 -- 172.21.15.81:0/2314686423 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb220017070 con 0x7fb2340c76f0 2024-02-21T23:02:20.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.627+0000 7fb22b400640 1 -- 172.21.15.81:0/2314686423 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb22000c9a0 con 0x7fb2340c76f0 2024-02-21T23:02:20.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.627+0000 7fb22b400640 1 -- 172.21.15.81:0/2314686423 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb22000cd90 con 0x7fb2340c76f0 2024-02-21T23:02:20.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.627+0000 7fb239000640 1 -- 172.21.15.81:0/2314686423 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb234155ea0 con 0x7fb2340c76f0 2024-02-21T23:02:20.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.627+0000 7fb239000640 1 -- 172.21.15.81:0/2314686423 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2341e1260 con 0x7fb2340c76f0 2024-02-21T23:02:20.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.628+0000 7fb22b400640 1 -- 172.21.15.81:0/2314686423 <== mon.2 v2:172.21.15.81:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fb22001a070 con 0x7fb2340c76f0 2024-02-21T23:02:20.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.628+0000 7fb22b400640 1 -- 172.21.15.81:0/2314686423 <== mon.2 v2:172.21.15.81:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7fb220006660 con 0x7fb2340c76f0 2024-02-21T23:02:20.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.628+0000 7fb239000640 1 -- 172.21.15.81:0/2314686423 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb23404f290 con 0x7fb2340c76f0 2024-02-21T23:02:20.621 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.631+0000 7fb22b400640 1 -- 172.21.15.81:0/2314686423 <== mon.2 v2:172.21.15.81:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fb220006020 con 0x7fb2340c76f0 2024-02-21T23:02:20.630 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-02-21T23:02:20.640+0000 7f09bd23a040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-21T23:02:20.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.710+0000 7fb239000640 1 -- 172.21.15.81:0/2314686423 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_command({"prefix": "osd new", "uuid": "6ac7806e-a5b0-4786-898a-b31607939493", "id": 4} v 0) v1 -- 0x7fb23406d9a0 con 0x7fb2340c76f0 2024-02-21T23:02:20.707 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-02-21T23:02:20.718+0000 7f0d34c3e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-21T23:02:20.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.718+0000 7fb22b400640 1 -- 172.21.15.81:0/2314686423 <== mon.2 v2:172.21.15.81:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6ac7806e-a5b0-4786-898a-b31607939493", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7fb220006200 con 0x7fb2340c76f0 2024-02-21T23:02:20.708 INFO:teuthology.orchestra.run.smithi081.stdout:4 2024-02-21T23:02:20.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.719+0000 7fb239000640 1 -- 172.21.15.81:0/2314686423 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb2340c76f0 msgr2=0x7fb234151920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:20.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.719+0000 7fb239000640 1 --2- 172.21.15.81:0/2314686423 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb2340c76f0 0x7fb234151920 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb22000e9b0 tx=0x7fb22000ee70 comp rx=0 tx=0).stop 2024-02-21T23:02:20.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.719+0000 7fb239000640 1 -- 172.21.15.81:0/2314686423 shutdown_connections 2024-02-21T23:02:20.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.719+0000 7fb239000640 1 --2- 172.21.15.81:0/2314686423 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb2340c76f0 0x7fb234151920 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:20.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.719+0000 7fb239000640 1 --2- 172.21.15.81:0/2314686423 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb23413e0c0 0x7fb234151e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:20.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.719+0000 7fb239000640 1 --2- 172.21.15.81:0/2314686423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb23406dbc0 0x7fb2340ab5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:20.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.719+0000 7fb239000640 1 -- 172.21.15.81:0/2314686423 >> 172.21.15.81:0/2314686423 conn(0x7fb234073110 msgr2=0x7fb2341de5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:20.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.719+0000 7fb239000640 1 -- 172.21.15.81:0/2314686423 shutdown_connections 2024-02-21T23:02:20.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.719+0000 7fb239000640 1 -- 172.21.15.81:0/2314686423 wait complete. 2024-02-21T23:02:20.725 DEBUG:teuthology.orchestra.run.smithi081:> sudo ceph --cluster ceph osd new ca6f7bb1-5003-4a86-b23b-eb6b23e377a9 5 2024-02-21T23:02:20.816 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-02-21T23:02:20.825+0000 7f09bd23a040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-21T23:02:20.868 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-02-21T23:02:20.877+0000 7f09bd23a040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-21T23:02:20.891 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-02-21T23:02:20.902+0000 7f0d34c3e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-21T23:02:20.912 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.922+0000 7fb2bb200640 1 Processor -- start 2024-02-21T23:02:20.912 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.922+0000 7fb2bb200640 1 -- start start 2024-02-21T23:02:20.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.923+0000 7fb2bb200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb2bc152650 0x7fb2bc0c8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:20.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.923+0000 7fb2bb200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb2bc06daa0 0x7fb2bc0c7160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:20.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.923+0000 7fb2bb200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2bc158920 con 0x7fb2bc0c76a0 2024-02-21T23:02:20.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.923+0000 7fb2bb200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2bc1586d0 con 0x7fb2bc06daa0 2024-02-21T23:02:20.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.923+0000 7fb2bb200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2bc158c00 con 0x7fb2bc152650 2024-02-21T23:02:20.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.923+0000 7fb2b9e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb2bc152650 0x7fb2bc0c8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:20.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.923+0000 7fb2ba800640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fb2bc0c76a0 0x7fb2bc1550e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:40014/0 (socket says 172.21.15.81:40014) 2024-02-21T23:02:20.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.924+0000 7fb2ba800640 1 -- 172.21.15.81:0/3023048087 learned_addr learned my addr 172.21.15.81:0/3023048087 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-02-21T23:02:20.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.924+0000 7fb2b9400640 1 --2- 172.21.15.81:0/3023048087 >> v2:172.21.15.28:3300/0 conn(0x7fb2bc06daa0 0x7fb2bc0c7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:20.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.924+0000 7fb2b9e00640 1 -- 172.21.15.81:0/3023048087 >> v2:172.21.15.28:3300/0 conn(0x7fb2bc06daa0 msgr2=0x7fb2bc0c7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:20.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.924+0000 7fb2b9e00640 1 --2- 172.21.15.81:0/3023048087 >> v2:172.21.15.28:3300/0 conn(0x7fb2bc06daa0 0x7fb2bc0c7160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:20.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.924+0000 7fb2b9e00640 1 -- 172.21.15.81:0/3023048087 >> v1:172.21.15.81:6789/0 conn(0x7fb2bc0c76a0 legacy=0x7fb2bc1550e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:20.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.924+0000 7fb2b9e00640 1 -- 172.21.15.81:0/3023048087 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2bc158ee0 con 0x7fb2bc152650 2024-02-21T23:02:20.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.924+0000 7fb2b8a00640 1 -- 172.21.15.81:0/3023048087 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1494534707 0 0) 0x7fb2bc158920 con 0x7fb2bc0c76a0 2024-02-21T23:02:20.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.925+0000 7fb2b9e00640 1 --2- 172.21.15.81:0/3023048087 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb2bc152650 0x7fb2bc0c8ce0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb2a40097a0 tx=0x7fb2a402b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b5a80bd580adf0c server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:20.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.925+0000 7fb2b8a00640 1 -- 172.21.15.81:0/3023048087 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2a40315a0 con 0x7fb2bc152650 2024-02-21T23:02:20.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.925+0000 7fb2b8a00640 1 -- 172.21.15.81:0/3023048087 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2a4037070 con 0x7fb2bc152650 2024-02-21T23:02:20.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.925+0000 7fb2b8a00640 1 -- 172.21.15.81:0/3023048087 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2a4031960 con 0x7fb2bc152650 2024-02-21T23:02:20.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.926+0000 7fb2bb200640 1 -- 172.21.15.81:0/3023048087 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb2bc152650 msgr2=0x7fb2bc0c8ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:20.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.926+0000 7fb2bb200640 1 --2- 172.21.15.81:0/3023048087 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb2bc152650 0x7fb2bc0c8ce0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb2a40097a0 tx=0x7fb2a402b840 comp rx=0 tx=0).stop 2024-02-21T23:02:20.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.926+0000 7fb2bb200640 1 -- 172.21.15.81:0/3023048087 shutdown_connections 2024-02-21T23:02:20.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.926+0000 7fb2bb200640 1 --2- 172.21.15.81:0/3023048087 >> v2:172.21.15.28:3300/0 conn(0x7fb2bc06daa0 0x7fb2bc0c7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:20.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.926+0000 7fb2bb200640 1 --2- 172.21.15.81:0/3023048087 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb2bc152650 0x7fb2bc0c8ce0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:20.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.926+0000 7fb2bb200640 1 -- 172.21.15.81:0/3023048087 >> 172.21.15.81:0/3023048087 conn(0x7fb2bc072ce0 msgr2=0x7fb2bc0ac340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:20.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.926+0000 7fb2bb200640 1 -- 172.21.15.81:0/3023048087 shutdown_connections 2024-02-21T23:02:20.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.926+0000 7fb2bb200640 1 -- 172.21.15.81:0/3023048087 wait complete. 2024-02-21T23:02:20.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.928+0000 7fb2bb200640 1 Processor -- start 2024-02-21T23:02:20.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.928+0000 7fb2bb200640 1 -- start start 2024-02-21T23:02:20.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.929+0000 7fb2bb200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb2bc06daa0 0x7fb2bc149170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:20.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.929+0000 7fb2bb200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2bc0c76a0 0x7fb2bc0ab430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:20.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.929+0000 7fb2bb200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2bc152650 0x7fb2bc1edd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:20.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.929+0000 7fb2bb200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2bc152a30 con 0x7fb2bc152650 2024-02-21T23:02:20.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.929+0000 7fb2bb200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2bc0bdfb0 con 0x7fb2bc0c76a0 2024-02-21T23:02:20.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.929+0000 7fb2bb200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2bc1557f0 con 0x7fb2bc06daa0 2024-02-21T23:02:20.923 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.929+0000 7fb2b9e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb2bc06daa0 0x7fb2bc149170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:20.923 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.929+0000 7fb2b9400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2bc0c76a0 0x7fb2bc0ab430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:20.923 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.929+0000 7fb2b9400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2bc0c76a0 0x7fb2bc0ab430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:52302/0 (socket says 172.21.15.81:52302) 2024-02-21T23:02:20.923 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.929+0000 7fb2b9e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb2bc06daa0 0x7fb2bc149170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3301/0 says I am v2:172.21.15.81:56056/0 (socket says 172.21.15.81:56056) 2024-02-21T23:02:20.923 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.929+0000 7fb2b9400640 1 -- 172.21.15.81:0/1024468453 learned_addr learned my addr 172.21.15.81:0/1024468453 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-02-21T23:02:20.923 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.929+0000 7fb2ba800640 1 --2- 172.21.15.81:0/1024468453 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2bc152650 0x7fb2bc1edd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:20.923 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.929+0000 7fb2b9400640 1 -- 172.21.15.81:0/1024468453 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb2bc06daa0 msgr2=0x7fb2bc149170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:20.923 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.929+0000 7fb2b9400640 1 --2- 172.21.15.81:0/1024468453 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb2bc06daa0 0x7fb2bc149170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:20.923 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.929+0000 7fb2b9400640 1 -- 172.21.15.81:0/1024468453 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2bc152650 msgr2=0x7fb2bc1edd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:20.923 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.929+0000 7fb2b9400640 1 --2- 172.21.15.81:0/1024468453 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2bc152650 0x7fb2bc1edd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:20.923 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.929+0000 7fb2b9400640 1 -- 172.21.15.81:0/1024468453 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2bc1ea7d0 con 0x7fb2bc0c76a0 2024-02-21T23:02:20.923 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.929+0000 7fb2b9e00640 1 --2- 172.21.15.81:0/1024468453 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb2bc06daa0 0x7fb2bc149170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-21T23:02:20.923 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.930+0000 7fb2b9400640 1 --2- 172.21.15.81:0/1024468453 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2bc0c76a0 0x7fb2bc0ab430 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb2ac007ef0 tx=0x7fb2ac00e4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:20.923 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.930+0000 7fb2b2000640 1 -- 172.21.15.81:0/1024468453 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2ac014620 con 0x7fb2bc0c76a0 2024-02-21T23:02:20.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.930+0000 7fb2b2000640 1 -- 172.21.15.81:0/1024468453 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2ac019070 con 0x7fb2bc0c76a0 2024-02-21T23:02:20.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.930+0000 7fb2b2000640 1 -- 172.21.15.81:0/1024468453 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2ac014a00 con 0x7fb2bc0c76a0 2024-02-21T23:02:20.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.930+0000 7fb2bb200640 1 -- 172.21.15.81:0/1024468453 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2bc1496b0 con 0x7fb2bc0c76a0 2024-02-21T23:02:20.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.930+0000 7fb2bb200640 1 -- 172.21.15.81:0/1024468453 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2bc1ead50 con 0x7fb2bc0c76a0 2024-02-21T23:02:20.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.930+0000 7fb2b2000640 1 -- 172.21.15.81:0/1024468453 <== mon.1 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fb2ac014ba0 con 0x7fb2bc0c76a0 2024-02-21T23:02:20.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.930+0000 7fb2b2000640 1 -- 172.21.15.81:0/1024468453 <== mon.1 v2:172.21.15.81:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7fb2ac01e9e0 con 0x7fb2bc0c76a0 2024-02-21T23:02:20.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.931+0000 7fb2bb200640 1 -- 172.21.15.81:0/1024468453 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2bc04f290 con 0x7fb2bc0c76a0 2024-02-21T23:02:20.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:20.934+0000 7fb2b2000640 1 -- 172.21.15.81:0/1024468453 <== mon.1 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fb2ac01e460 con 0x7fb2bc0c76a0 2024-02-21T23:02:20.943 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-02-21T23:02:20.954+0000 7f0d34c3e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-21T23:02:21.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.012+0000 7fb2bb200640 1 -- 172.21.15.81:0/1024468453 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd new", "uuid": "ca6f7bb1-5003-4a86-b23b-eb6b23e377a9", "id": 5} v 0) v1 -- 0x7fb2bc156080 con 0x7fb2bc0c76a0 2024-02-21T23:02:21.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.018+0000 7fb2b2000640 1 -- 172.21.15.81:0/1024468453 <== mon.1 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ca6f7bb1-5003-4a86-b23b-eb6b23e377a9", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7fb2ac01e640 con 0x7fb2bc0c76a0 2024-02-21T23:02:21.008 INFO:teuthology.orchestra.run.smithi081.stdout:5 2024-02-21T23:02:21.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.019+0000 7fb2bb200640 1 -- 172.21.15.81:0/1024468453 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2bc0c76a0 msgr2=0x7fb2bc0ab430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:21.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.019+0000 7fb2bb200640 1 --2- 172.21.15.81:0/1024468453 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2bc0c76a0 0x7fb2bc0ab430 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb2ac007ef0 tx=0x7fb2ac00e4b0 comp rx=0 tx=0).stop 2024-02-21T23:02:21.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.019+0000 7fb2bb200640 1 -- 172.21.15.81:0/1024468453 shutdown_connections 2024-02-21T23:02:21.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.019+0000 7fb2bb200640 1 --2- 172.21.15.81:0/1024468453 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2bc152650 0x7fb2bc1edd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:21.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.019+0000 7fb2bb200640 1 --2- 172.21.15.81:0/1024468453 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2bc0c76a0 0x7fb2bc0ab430 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:21.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.019+0000 7fb2bb200640 1 --2- 172.21.15.81:0/1024468453 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb2bc06daa0 0x7fb2bc149170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:21.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.019+0000 7fb2bb200640 1 -- 172.21.15.81:0/1024468453 >> 172.21.15.81:0/1024468453 conn(0x7fb2bc072ce0 msgr2=0x7fb2bc150370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:21.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.020+0000 7fb2bb200640 1 -- 172.21.15.81:0/1024468453 shutdown_connections 2024-02-21T23:02:21.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.020+0000 7fb2bb200640 1 -- 172.21.15.81:0/1024468453 wait complete. 2024-02-21T23:02:21.026 DEBUG:teuthology.orchestra.run.smithi081:> sudo ceph --cluster ceph osd new 5231b268-6822-4ba3-8e00-4b374f580251 6 2024-02-21T23:02:21.052 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-02-21T23:02:21.062+0000 7f09bd23a040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-21T23:02:21.126 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-02-21T23:02:21.137+0000 7f0d34c3e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-21T23:02:21.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.216+0000 7f49a5800640 1 Processor -- start 2024-02-21T23:02:21.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.216+0000 7f49a5800640 1 -- start start 2024-02-21T23:02:21.208 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.217+0000 7f49a5800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f49a00ab170 0x7f49a0158c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:21.208 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.217+0000 7f49a5800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f49a00c3b40 0x7f49a01591a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:21.208 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.217+0000 7f49a5800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49a00c8170 con 0x7f49a0152a30 2024-02-21T23:02:21.208 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.217+0000 7f49a5800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49a01561a0 con 0x7f49a00ab170 2024-02-21T23:02:21.208 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.217+0000 7f49a5800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49a00c7f40 con 0x7f49a00c3b40 2024-02-21T23:02:21.209 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.218+0000 7f499e000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f49a00ab170 0x7f49a0158c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:21.209 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.218+0000 7f499f400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f49a00c3b40 0x7f49a01591a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:21.209 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.218+0000 7f499e000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f49a00ab170 0x7f49a0158c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:52306/0 (socket says 172.21.15.81:52306) 2024-02-21T23:02:21.209 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.218+0000 7f499e000640 1 -- 172.21.15.81:0/3424124345 learned_addr learned my addr 172.21.15.81:0/3424124345 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-02-21T23:02:21.209 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.218+0000 7f499e000640 1 -- 172.21.15.81:0/3424124345 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f49a00c3b40 msgr2=0x7f49a01591a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:21.209 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.218+0000 7f499e000640 1 --2- 172.21.15.81:0/3424124345 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f49a00c3b40 0x7f49a01591a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:21.209 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.218+0000 7f499e000640 1 -- 172.21.15.81:0/3424124345 >> v1:172.21.15.28:6789/0 conn(0x7f49a0152a30 legacy=0x7f49a00c8690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:21.209 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.218+0000 7f499e000640 1 -- 172.21.15.81:0/3424124345 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49a00c3130 con 0x7f49a00ab170 2024-02-21T23:02:21.209 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.218+0000 7f499f400640 1 --2- 172.21.15.81:0/3424124345 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f49a00c3b40 0x7f49a01591a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-21T23:02:21.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.219+0000 7f499e000640 1 --2- 172.21.15.81:0/3424124345 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f49a00ab170 0x7f49a0158c60 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f498c009870 tx=0x7f498c02b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b64ceaa60e75750 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:21.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.219+0000 7f499d600640 1 -- 172.21.15.81:0/3424124345 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f498c035020 con 0x7f49a00ab170 2024-02-21T23:02:21.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.220+0000 7f499d600640 1 -- 172.21.15.81:0/3424124345 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f498c031640 con 0x7f49a00ab170 2024-02-21T23:02:21.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.220+0000 7f499d600640 1 -- 172.21.15.81:0/3424124345 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f498c02d040 con 0x7f49a00ab170 2024-02-21T23:02:21.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.220+0000 7f49a5800640 1 -- 172.21.15.81:0/3424124345 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f49a00ab170 msgr2=0x7f49a0158c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:21.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.220+0000 7f49a5800640 1 --2- 172.21.15.81:0/3424124345 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f49a00ab170 0x7f49a0158c60 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f498c009870 tx=0x7f498c02b8f0 comp rx=0 tx=0).stop 2024-02-21T23:02:21.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.220+0000 7f49a5800640 1 -- 172.21.15.81:0/3424124345 shutdown_connections 2024-02-21T23:02:21.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.220+0000 7f49a5800640 1 --2- 172.21.15.81:0/3424124345 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f49a00c3b40 0x7f49a01591a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:21.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.220+0000 7f49a5800640 1 --2- 172.21.15.81:0/3424124345 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f49a00ab170 0x7f49a0158c60 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:21.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.220+0000 7f49a5800640 1 -- 172.21.15.81:0/3424124345 >> 172.21.15.81:0/3424124345 conn(0x7f49a006b4a0 msgr2=0x7f49a00aa4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:21.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.220+0000 7f49a5800640 1 -- 172.21.15.81:0/3424124345 shutdown_connections 2024-02-21T23:02:21.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.220+0000 7f49a5800640 1 -- 172.21.15.81:0/3424124345 wait complete. 2024-02-21T23:02:21.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.222+0000 7f49a5800640 1 Processor -- start 2024-02-21T23:02:21.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.222+0000 7f49a5800640 1 -- start start 2024-02-21T23:02:21.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.223+0000 7f49a5800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49a00ab170 0x7f49a00a8450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:21.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.223+0000 7f49a5800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f49a00c3b40 0x7f49a01eb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:21.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.223+0000 7f49a5800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f49a0152a30 0x7f49a01500e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:21.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.223+0000 7f49a5800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49a00c36f0 con 0x7f49a00ab170 2024-02-21T23:02:21.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.223+0000 7f49a5800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49a01587d0 con 0x7f49a00c3b40 2024-02-21T23:02:21.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.223+0000 7f49a5800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49a00c7c60 con 0x7f49a0152a30 2024-02-21T23:02:21.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.223+0000 7f499e000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f49a00c3b40 0x7f49a01eb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:21.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.223+0000 7f499f400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f49a0152a30 0x7f49a01500e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:21.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.223+0000 7f499f400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f49a0152a30 0x7f49a01500e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3301/0 says I am v2:172.21.15.81:56078/0 (socket says 172.21.15.81:56078) 2024-02-21T23:02:21.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.223+0000 7f499f400640 1 -- 172.21.15.81:0/1969587882 learned_addr learned my addr 172.21.15.81:0/1969587882 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-02-21T23:02:21.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.224+0000 7f499f400640 1 -- 172.21.15.81:0/1969587882 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f49a00c3b40 msgr2=0x7f49a01eb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:21.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.224+0000 7f499f400640 1 --2- 172.21.15.81:0/1969587882 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f49a00c3b40 0x7f49a01eb6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:21.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.224+0000 7f499f400640 1 -- 172.21.15.81:0/1969587882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49a00ab170 msgr2=0x7f49a00a8450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:21.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.224+0000 7f499ea00640 1 --2- 172.21.15.81:0/1969587882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49a00ab170 0x7f49a00a8450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:21.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.224+0000 7f499f400640 1 --2- 172.21.15.81:0/1969587882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49a00ab170 0x7f49a00a8450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:21.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.224+0000 7f499f400640 1 -- 172.21.15.81:0/1969587882 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49a00a9af0 con 0x7f49a0152a30 2024-02-21T23:02:21.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.224+0000 7f499e000640 1 --2- 172.21.15.81:0/1969587882 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f49a00c3b40 0x7f49a01eb6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-21T23:02:21.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.224+0000 7f499f400640 1 --2- 172.21.15.81:0/1969587882 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f49a0152a30 0x7f49a01500e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f499800efc0 tx=0x7f4998009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:21.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.224+0000 7f4996a00640 1 -- 172.21.15.81:0/1969587882 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4998018030 con 0x7f49a0152a30 2024-02-21T23:02:21.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.224+0000 7f4996a00640 1 -- 172.21.15.81:0/1969587882 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f499800cde0 con 0x7f49a0152a30 2024-02-21T23:02:21.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.224+0000 7f4996a00640 1 -- 172.21.15.81:0/1969587882 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4998003bb0 con 0x7f49a0152a30 2024-02-21T23:02:21.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.224+0000 7f49a5800640 1 -- 172.21.15.81:0/1969587882 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49a00a9dd0 con 0x7f49a0152a30 2024-02-21T23:02:21.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.224+0000 7f49a5800640 1 -- 172.21.15.81:0/1969587882 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49a00aa390 con 0x7f49a0152a30 2024-02-21T23:02:21.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.225+0000 7f4996a00640 1 -- 172.21.15.81:0/1969587882 <== mon.2 v2:172.21.15.81:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f499801b070 con 0x7f49a0152a30 2024-02-21T23:02:21.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.225+0000 7f4996a00640 1 -- 172.21.15.81:0/1969587882 <== mon.2 v2:172.21.15.81:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f499801db70 con 0x7f49a0152a30 2024-02-21T23:02:21.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.225+0000 7f49a5800640 1 -- 172.21.15.81:0/1969587882 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49a0067f30 con 0x7f49a0152a30 2024-02-21T23:02:21.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.228+0000 7f4996a00640 1 -- 172.21.15.81:0/1969587882 <== mon.2 v2:172.21.15.81:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f4998010040 con 0x7f49a0152a30 2024-02-21T23:02:21.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.306+0000 7f49a5800640 1 -- 172.21.15.81:0/1969587882 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_command({"prefix": "osd new", "uuid": "5231b268-6822-4ba3-8e00-4b374f580251", "id": 6} v 0) v1 -- 0x7f49a00c6630 con 0x7f49a0152a30 2024-02-21T23:02:21.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.311+0000 7f4996a00640 1 -- 172.21.15.81:0/1969587882 <== mon.2 v2:172.21.15.81:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5231b268-6822-4ba3-8e00-4b374f580251", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f499801d540 con 0x7f49a0152a30 2024-02-21T23:02:21.302 INFO:teuthology.orchestra.run.smithi081.stdout:6 2024-02-21T23:02:21.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.312+0000 7f49a5800640 1 -- 172.21.15.81:0/1969587882 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f49a0152a30 msgr2=0x7f49a01500e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:21.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.312+0000 7f49a5800640 1 --2- 172.21.15.81:0/1969587882 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f49a0152a30 0x7f49a01500e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f499800efc0 tx=0x7f4998009f90 comp rx=0 tx=0).stop 2024-02-21T23:02:21.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.313+0000 7f49a5800640 1 -- 172.21.15.81:0/1969587882 shutdown_connections 2024-02-21T23:02:21.305 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.313+0000 7f49a5800640 1 --2- 172.21.15.81:0/1969587882 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f49a0152a30 0x7f49a01500e0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:21.305 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.313+0000 7f49a5800640 1 --2- 172.21.15.81:0/1969587882 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f49a00c3b40 0x7f49a01eb6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:21.305 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.313+0000 7f49a5800640 1 --2- 172.21.15.81:0/1969587882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49a00ab170 0x7f49a00a8450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:21.305 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.313+0000 7f49a5800640 1 -- 172.21.15.81:0/1969587882 >> 172.21.15.81:0/1969587882 conn(0x7f49a006b4a0 msgr2=0x7f49a0158c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:21.305 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.313+0000 7f49a5800640 1 -- 172.21.15.81:0/1969587882 shutdown_connections 2024-02-21T23:02:21.305 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.313+0000 7f49a5800640 1 -- 172.21.15.81:0/1969587882 wait complete. 2024-02-21T23:02:21.319 DEBUG:teuthology.orchestra.run.smithi081:> sudo ceph --cluster ceph osd new 4b2df9db-d659-4a19-a88d-ea4d759259dc 7 2024-02-21T23:02:21.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.554+0000 7f0173800640 1 Processor -- start 2024-02-21T23:02:21.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.554+0000 7f0173800640 1 -- start start 2024-02-21T23:02:21.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.556+0000 7f0173800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f016c0c5cc0 0x7f016c0c6c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:21.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.556+0000 7f0173800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f016c0c3b50 con 0x7f016c0c7590 2024-02-21T23:02:21.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.556+0000 7f0173800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f016c159580 con 0x7f016c0c3030 2024-02-21T23:02:21.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.556+0000 7f0173800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f016c0c6230 con 0x7f016c0c5cc0 2024-02-21T23:02:21.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.556+0000 7f0171800640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f016c0c3030 0x7f016c0c3440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:40016/0 (socket says 172.21.15.81:40016) 2024-02-21T23:02:21.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.556+0000 7f0171800640 1 -- 172.21.15.81:0/1638007037 learned_addr learned my addr 172.21.15.81:0/1638007037 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-02-21T23:02:21.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.557+0000 7f016be00640 1 --2- 172.21.15.81:0/1638007037 >> v2:172.21.15.28:3300/0 conn(0x7f016c0c5cc0 0x7f016c0c6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:21.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.557+0000 7f016b400640 1 -- 172.21.15.81:0/1638007037 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 502140520 0 0) 0x7f016c159580 con 0x7f016c0c3030 2024-02-21T23:02:21.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.557+0000 7f016b400640 1 -- 172.21.15.81:0/1638007037 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0154003520 con 0x7f016c0c3030 2024-02-21T23:02:21.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.557+0000 7f016b400640 1 -- 172.21.15.81:0/1638007037 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4090935359 0 0) 0x7f0154003520 con 0x7f016c0c3030 2024-02-21T23:02:21.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.557+0000 7f016b400640 1 -- 172.21.15.81:0/1638007037 >> v2:172.21.15.28:3300/0 conn(0x7f016c0c5cc0 msgr2=0x7f016c0c6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:21.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.557+0000 7f016b400640 1 --2- 172.21.15.81:0/1638007037 >> v2:172.21.15.28:3300/0 conn(0x7f016c0c5cc0 0x7f016c0c6c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:21.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.557+0000 7f016b400640 1 -- 172.21.15.81:0/1638007037 >> v1:172.21.15.28:6789/0 conn(0x7f016c0c7590 legacy=0x7f016c1524e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:21.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.557+0000 7f016b400640 1 -- 172.21.15.81:0/1638007037 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f016c1587c0 con 0x7f016c0c3030 2024-02-21T23:02:21.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.557+0000 7f016b400640 1 -- 172.21.15.81:0/1638007037 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f01640019f0 con 0x7f016c0c3030 2024-02-21T23:02:21.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.557+0000 7f016b400640 1 -- 172.21.15.81:0/1638007037 >> v1:172.21.15.81:6789/0 conn(0x7f016c0c3030 legacy=0x7f016c0c3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:21.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.557+0000 7f016b400640 1 --2- 172.21.15.81:0/1638007037 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0154003a00 0x7f0154003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:21.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.557+0000 7f016b400640 1 --2- 172.21.15.81:0/1638007037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0154004800 0x7f0154004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:21.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.557+0000 7f016b400640 1 --2- 172.21.15.81:0/1638007037 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f01540052e0 0x7f015400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:21.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.557+0000 7f016b400640 1 -- 172.21.15.81:0/1638007037 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f016c0c6230 con 0x7f0154004800 2024-02-21T23:02:21.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.557+0000 7f016b400640 1 -- 172.21.15.81:0/1638007037 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f015400c200 con 0x7f0154003a00 2024-02-21T23:02:21.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.557+0000 7f016b400640 1 -- 172.21.15.81:0/1638007037 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f015400c400 con 0x7f01540052e0 2024-02-21T23:02:21.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.557+0000 7f016be00640 1 --2- 172.21.15.81:0/1638007037 >> v2:172.21.15.28:3300/0 conn(0x7f016c0c5cc0 0x7f016c0c6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:02:21.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.557+0000 7f0170e00640 1 --2- 172.21.15.81:0/1638007037 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0154003a00 0x7f0154003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:21.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.558+0000 7f0171800640 1 --2- 172.21.15.81:0/1638007037 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f01540052e0 0x7f015400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:21.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.558+0000 7f016be00640 1 --2- 172.21.15.81:0/1638007037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0154004800 0x7f0154004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:21.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.558+0000 7f0170e00640 1 -- 172.21.15.81:0/1638007037 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f01540052e0 msgr2=0x7f015400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:21.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.558+0000 7f0170e00640 1 --2- 172.21.15.81:0/1638007037 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f01540052e0 0x7f015400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:21.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.558+0000 7f0170e00640 1 -- 172.21.15.81:0/1638007037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0154004800 msgr2=0x7f0154004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:21.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.558+0000 7f0170e00640 1 --2- 172.21.15.81:0/1638007037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0154004800 0x7f0154004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:21.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.558+0000 7f0170e00640 1 -- 172.21.15.81:0/1638007037 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0154003520 con 0x7f0154003a00 2024-02-21T23:02:21.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.558+0000 7f0171800640 1 -- 172.21.15.81:0/1638007037 reap_dead start 2024-02-21T23:02:21.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.558+0000 7f016be00640 1 --2- 172.21.15.81:0/1638007037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0154004800 0x7f0154004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:02:21.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.559+0000 7f0170e00640 1 --2- 172.21.15.81:0/1638007037 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0154003a00 0x7f0154003e20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f0160001750 tx=0x7f0154005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9442eeb51b8202f1 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:21.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.559+0000 7f016b400640 1 -- 172.21.15.81:0/1638007037 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0160001370 con 0x7f0154003a00 2024-02-21T23:02:21.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.559+0000 7f016b400640 1 -- 172.21.15.81:0/1638007037 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01600052d0 con 0x7f0154003a00 2024-02-21T23:02:21.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.559+0000 7f016b400640 1 -- 172.21.15.81:0/1638007037 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0160003830 con 0x7f0154003a00 2024-02-21T23:02:21.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.560+0000 7f0173800640 1 -- 172.21.15.81:0/1638007037 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0154003a00 msgr2=0x7f0154003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:21.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.560+0000 7f0173800640 1 --2- 172.21.15.81:0/1638007037 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0154003a00 0x7f0154003e20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f0160001750 tx=0x7f0154005830 comp rx=0 tx=0).stop 2024-02-21T23:02:21.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.560+0000 7f0173800640 1 -- 172.21.15.81:0/1638007037 shutdown_connections 2024-02-21T23:02:21.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.560+0000 7f0173800640 1 --2- 172.21.15.81:0/1638007037 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0154003a00 0x7f0154003e20 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:21.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.560+0000 7f0173800640 1 -- 172.21.15.81:0/1638007037 >> 172.21.15.81:0/1638007037 conn(0x7f016c08afe0 msgr2=0x7f016c0725a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:21.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.560+0000 7f0173800640 1 -- 172.21.15.81:0/1638007037 shutdown_connections 2024-02-21T23:02:21.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.560+0000 7f0173800640 1 -- 172.21.15.81:0/1638007037 wait complete. 2024-02-21T23:02:21.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.562+0000 7f0173800640 1 Processor -- start 2024-02-21T23:02:21.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.562+0000 7f0173800640 1 -- start start 2024-02-21T23:02:21.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.563+0000 7f0173800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0154003a00 0x7f016c154360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:21.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.563+0000 7f0173800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f016c1eb240 0x7f016c14a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:21.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.563+0000 7f0173800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f016c1ee6e0 0x7f016c1e7910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:21.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.563+0000 7f0173800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f016c159060 con 0x7f016c1ee6e0 2024-02-21T23:02:21.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.563+0000 7f0173800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f016c158d80 con 0x7f016c1eb240 2024-02-21T23:02:21.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.563+0000 7f0173800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f016c0c6720 con 0x7f0154003a00 2024-02-21T23:02:21.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.563+0000 7f0170e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0154003a00 0x7f016c154360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:21.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.563+0000 7f016be00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f016c1eb240 0x7f016c14a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:21.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.563+0000 7f0170e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0154003a00 0x7f016c154360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3301/0 says I am v2:172.21.15.81:56086/0 (socket says 172.21.15.81:56086) 2024-02-21T23:02:21.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.563+0000 7f0170e00640 1 -- 172.21.15.81:0/3614860840 learned_addr learned my addr 172.21.15.81:0/3614860840 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-02-21T23:02:21.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.563+0000 7f0170e00640 1 -- 172.21.15.81:0/3614860840 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f016c1eb240 msgr2=0x7f016c14a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:21.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.563+0000 7f0170e00640 1 --2- 172.21.15.81:0/3614860840 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f016c1eb240 0x7f016c14a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:21.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.563+0000 7f0171800640 1 --2- 172.21.15.81:0/3614860840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f016c1ee6e0 0x7f016c1e7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:21.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.563+0000 7f0170e00640 1 -- 172.21.15.81:0/3614860840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f016c1ee6e0 msgr2=0x7f016c1e7910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:21.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.564+0000 7f0170e00640 1 --2- 172.21.15.81:0/3614860840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f016c1ee6e0 0x7f016c1e7910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:21.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.564+0000 7f0170e00640 1 -- 172.21.15.81:0/3614860840 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f016c0c9ba0 con 0x7f0154003a00 2024-02-21T23:02:21.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.564+0000 7f0171800640 1 --2- 172.21.15.81:0/3614860840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f016c1ee6e0 0x7f016c1e7910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:02:21.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.564+0000 7f0170e00640 1 --2- 172.21.15.81:0/3614860840 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0154003a00 0x7f016c154360 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0160005040 tx=0x7f0160001120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:21.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.564+0000 7f0168c00640 1 -- 172.21.15.81:0/3614860840 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0160041020 con 0x7f0154003a00 2024-02-21T23:02:21.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.564+0000 7f0168c00640 1 -- 172.21.15.81:0/3614860840 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0160005480 con 0x7f0154003a00 2024-02-21T23:02:21.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.564+0000 7f0173800640 1 -- 172.21.15.81:0/3614860840 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f016c0c9e80 con 0x7f0154003a00 2024-02-21T23:02:21.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.564+0000 7f0168c00640 1 -- 172.21.15.81:0/3614860840 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0160001370 con 0x7f0154003a00 2024-02-21T23:02:21.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.564+0000 7f0173800640 1 -- 172.21.15.81:0/3614860840 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f016c14abb0 con 0x7f0154003a00 2024-02-21T23:02:21.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.565+0000 7f0168c00640 1 -- 172.21.15.81:0/3614860840 <== mon.2 v2:172.21.15.81:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f0160044020 con 0x7f0154003a00 2024-02-21T23:02:21.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.565+0000 7f0168c00640 1 -- 172.21.15.81:0/3614860840 <== mon.2 v2:172.21.15.81:3301/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7f016003dd00 con 0x7f0154003a00 2024-02-21T23:02:21.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.565+0000 7f0173800640 1 -- 172.21.15.81:0/3614860840 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f016c067f30 con 0x7f0154003a00 2024-02-21T23:02:21.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.568+0000 7f0168c00640 1 -- 172.21.15.81:0/3614860840 <== mon.2 v2:172.21.15.81:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f016004b030 con 0x7f0154003a00 2024-02-21T23:02:21.638 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.648+0000 7f0173800640 1 -- 172.21.15.81:0/3614860840 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_command({"prefix": "osd new", "uuid": "4b2df9db-d659-4a19-a88d-ea4d759259dc", "id": 7} v 0) v1 -- 0x7f016c152af0 con 0x7f0154003a00 2024-02-21T23:02:21.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.657+0000 7f0168c00640 1 -- 172.21.15.81:0/3614860840 <== mon.2 v2:172.21.15.81:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4b2df9db-d659-4a19-a88d-ea4d759259dc", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f0160003b10 con 0x7f0154003a00 2024-02-21T23:02:21.648 INFO:teuthology.orchestra.run.smithi081.stdout:7 2024-02-21T23:02:21.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.658+0000 7f0173800640 1 -- 172.21.15.81:0/3614860840 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0154003a00 msgr2=0x7f016c154360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:21.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.658+0000 7f0173800640 1 --2- 172.21.15.81:0/3614860840 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0154003a00 0x7f016c154360 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0160005040 tx=0x7f0160001120 comp rx=0 tx=0).stop 2024-02-21T23:02:21.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.659+0000 7f0173800640 1 -- 172.21.15.81:0/3614860840 shutdown_connections 2024-02-21T23:02:21.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.659+0000 7f0173800640 1 --2- 172.21.15.81:0/3614860840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f016c1ee6e0 0x7f016c1e7910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:21.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.659+0000 7f0173800640 1 --2- 172.21.15.81:0/3614860840 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f016c1eb240 0x7f016c14a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:21.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.659+0000 7f0173800640 1 --2- 172.21.15.81:0/3614860840 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0154003a00 0x7f016c154360 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:21.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.659+0000 7f0173800640 1 -- 172.21.15.81:0/3614860840 >> 172.21.15.81:0/3614860840 conn(0x7f016c08afe0 msgr2=0x7f016c150ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:21.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.659+0000 7f0173800640 1 -- 172.21.15.81:0/3614860840 shutdown_connections 2024-02-21T23:02:21.649 INFO:teuthology.orchestra.run.smithi081.stderr:2024-02-21T23:02:21.659+0000 7f0173800640 1 -- 172.21.15.81:0/3614860840 wait complete. 2024-02-21T23:02:21.664 INFO:tasks.ceph.osd.0:Restarting daemon 2024-02-21T23:02:21.665 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-02-21T23:02:21.671 INFO:tasks.ceph.osd.0:Started 2024-02-21T23:02:21.672 INFO:tasks.ceph.osd.1:Restarting daemon 2024-02-21T23:02:21.672 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-02-21T23:02:21.677 INFO:tasks.ceph.osd.1:Started 2024-02-21T23:02:21.677 INFO:tasks.ceph.osd.2:Restarting daemon 2024-02-21T23:02:21.677 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-02-21T23:02:21.681 INFO:tasks.ceph.osd.2:Started 2024-02-21T23:02:21.681 INFO:tasks.ceph.osd.3:Restarting daemon 2024-02-21T23:02:21.681 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-02-21T23:02:21.686 INFO:tasks.ceph.osd.3:Started 2024-02-21T23:02:21.686 INFO:tasks.ceph.osd.4:Restarting daemon 2024-02-21T23:02:21.686 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-02-21T23:02:21.690 INFO:tasks.ceph.osd.4:Started 2024-02-21T23:02:21.690 INFO:tasks.ceph.osd.5:Restarting daemon 2024-02-21T23:02:21.690 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-02-21T23:02:21.694 INFO:tasks.ceph.osd.5:Started 2024-02-21T23:02:21.694 INFO:tasks.ceph.osd.6:Restarting daemon 2024-02-21T23:02:21.694 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-02-21T23:02:21.699 INFO:tasks.ceph.osd.6:Started 2024-02-21T23:02:21.699 INFO:tasks.ceph.osd.7:Restarting daemon 2024-02-21T23:02:21.699 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-02-21T23:02:21.704 INFO:tasks.ceph.osd.7:Started 2024-02-21T23:02:21.704 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:02:21.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:21.946+0000 7f9d77200640 1 Processor -- start 2024-02-21T23:02:21.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:21.946+0000 7f9d77200640 1 -- start start 2024-02-21T23:02:21.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:21.947+0000 7f9d77200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f9d7806dbf0 0x7f9d78158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:21.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:21.947+0000 7f9d77200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9d78155960 0x7f9d78159000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:21.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:21.947+0000 7f9d77200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d780c74c0 con 0x7f9d780aff90 2024-02-21T23:02:21.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:21.947+0000 7f9d77200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d780c7270 con 0x7f9d7806dbf0 2024-02-21T23:02:21.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:21.947+0000 7f9d77200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d78158630 con 0x7f9d78155960 2024-02-21T23:02:21.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:21.948+0000 7f9d76800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9d78155960 0x7f9d78159000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:21.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:21.948+0000 7f9d75400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f9d7806dbf0 0x7f9d78158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:21.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:21.948+0000 7f9d75400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f9d7806dbf0 0x7f9d78158ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.28:37148/0 (socket says 172.21.15.28:37148) 2024-02-21T23:02:21.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:21.948+0000 7f9d75400640 1 -- 172.21.15.28:0/411192541 learned_addr learned my addr 172.21.15.28:0/411192541 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:02:21.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:21.948+0000 7f9d76800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9d78155960 0x7f9d78159000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3301/0 says I am v2:172.21.15.28:47208/0 (socket says 172.21.15.28:47208) 2024-02-21T23:02:21.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:21.948+0000 7f9d75400640 1 -- 172.21.15.28:0/411192541 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9d78155960 msgr2=0x7f9d78159000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:21.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:21.948+0000 7f9d75400640 1 --2- 172.21.15.28:0/411192541 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9d78155960 0x7f9d78159000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:21.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:21.948+0000 7f9d75400640 1 -- 172.21.15.28:0/411192541 >> v1:172.21.15.81:6789/0 conn(0x7f9d780aff90 legacy=0x7f9d780c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:21.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:21.948+0000 7f9d75400640 1 -- 172.21.15.28:0/411192541 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d780c77a0 con 0x7f9d7806dbf0 2024-02-21T23:02:21.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:21.949+0000 7f9d75400640 1 --2- 172.21.15.28:0/411192541 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9d7806dbf0 0x7f9d78158ac0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9d64009870 tx=0x7f9d6402b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf1783a8c52e5b0e server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:21.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:21.950+0000 7f9d74a00640 1 -- 172.21.15.28:0/411192541 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d64034070 con 0x7f9d7806dbf0 2024-02-21T23:02:21.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:21.950+0000 7f9d74a00640 1 -- 172.21.15.28:0/411192541 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d64030840 con 0x7f9d7806dbf0 2024-02-21T23:02:21.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:21.950+0000 7f9d74a00640 1 -- 172.21.15.28:0/411192541 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d64030c10 con 0x7f9d7806dbf0 2024-02-21T23:02:21.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:21.950+0000 7f9d77200640 1 -- 172.21.15.28:0/411192541 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9d7806dbf0 msgr2=0x7f9d78158ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:21.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:21.950+0000 7f9d77200640 1 --2- 172.21.15.28:0/411192541 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9d7806dbf0 0x7f9d78158ac0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9d64009870 tx=0x7f9d6402b980 comp rx=0 tx=0).stop 2024-02-21T23:02:21.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:21.950+0000 7f9d77200640 1 -- 172.21.15.28:0/411192541 shutdown_connections 2024-02-21T23:02:21.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:21.950+0000 7f9d77200640 1 --2- 172.21.15.28:0/411192541 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9d78155960 0x7f9d78159000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:21.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:21.950+0000 7f9d77200640 1 --2- 172.21.15.28:0/411192541 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9d7806dbf0 0x7f9d78158ac0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:21.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:21.950+0000 7f9d77200640 1 -- 172.21.15.28:0/411192541 >> 172.21.15.28:0/411192541 conn(0x7f9d780ac370 msgr2=0x7f9d780ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:21.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:21.950+0000 7f9d77200640 1 -- 172.21.15.28:0/411192541 shutdown_connections 2024-02-21T23:02:21.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:21.950+0000 7f9d77200640 1 -- 172.21.15.28:0/411192541 wait complete. 2024-02-21T23:02:21.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:21.952+0000 7f9d77200640 1 Processor -- start 2024-02-21T23:02:21.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:21.952+0000 7f9d77200640 1 -- start start 2024-02-21T23:02:21.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:21.953+0000 7f9d77200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9d7806dbf0 0x7f9d780ae4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:21.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:21.953+0000 7f9d77200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9d780aff90 0x7f9d781eac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:21.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:21.953+0000 7f9d77200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d78155960 0x7f9d781eb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:21.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:21.953+0000 7f9d77200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d780c58f0 con 0x7f9d78155960 2024-02-21T23:02:21.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:21.953+0000 7f9d77200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d780be0c0 con 0x7f9d780aff90 2024-02-21T23:02:21.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:21.953+0000 7f9d77200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d780c5a70 con 0x7f9d7806dbf0 2024-02-21T23:02:21.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:21.953+0000 7f9d76800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d78155960 0x7f9d781eb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:21.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:21.953+0000 7f9d75400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9d780aff90 0x7f9d781eac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:21.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:21.953+0000 7f9d76800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d78155960 0x7f9d781eb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57284/0 (socket says 172.21.15.28:57284) 2024-02-21T23:02:21.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:21.953+0000 7f9d76800640 1 -- 172.21.15.28:0/1917418312 learned_addr learned my addr 172.21.15.28:0/1917418312 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:02:21.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:21.953+0000 7f9d75e00640 1 --2- 172.21.15.28:0/1917418312 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9d7806dbf0 0x7f9d780ae4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:21.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:21.954+0000 7f9d76800640 1 -- 172.21.15.28:0/1917418312 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9d7806dbf0 msgr2=0x7f9d780ae4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:21.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:21.954+0000 7f9d76800640 1 --2- 172.21.15.28:0/1917418312 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9d7806dbf0 0x7f9d780ae4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:21.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:21.954+0000 7f9d76800640 1 -- 172.21.15.28:0/1917418312 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9d780aff90 msgr2=0x7f9d781eac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:21.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:21.954+0000 7f9d76800640 1 --2- 172.21.15.28:0/1917418312 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9d780aff90 0x7f9d781eac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:21.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:21.954+0000 7f9d76800640 1 -- 172.21.15.28:0/1917418312 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d780be6f0 con 0x7f9d78155960 2024-02-21T23:02:21.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:21.954+0000 7f9d76800640 1 --2- 172.21.15.28:0/1917418312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d78155960 0x7f9d781eb1b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f9d7000efc0 tx=0x7f9d7000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:21.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:21.954+0000 7f9d6e000640 1 -- 172.21.15.28:0/1917418312 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d70018020 con 0x7f9d78155960 2024-02-21T23:02:21.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:21.954+0000 7f9d6e000640 1 -- 172.21.15.28:0/1917418312 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d70014480 con 0x7f9d78155960 2024-02-21T23:02:21.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:21.954+0000 7f9d6e000640 1 -- 172.21.15.28:0/1917418312 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d70014870 con 0x7f9d78155960 2024-02-21T23:02:21.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:21.954+0000 7f9d77200640 1 -- 172.21.15.28:0/1917418312 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d781ee190 con 0x7f9d78155960 2024-02-21T23:02:21.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:21.955+0000 7f9d77200640 1 -- 172.21.15.28:0/1917418312 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d781ee750 con 0x7f9d78155960 2024-02-21T23:02:21.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:21.955+0000 7f9d6e000640 1 -- 172.21.15.28:0/1917418312 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f9d70005d10 con 0x7f9d78155960 2024-02-21T23:02:21.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:21.955+0000 7f9d6e000640 1 -- 172.21.15.28:0/1917418312 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f9d7001ddb0 con 0x7f9d78155960 2024-02-21T23:02:21.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:21.956+0000 7f9d77200640 1 -- 172.21.15.28:0/1917418312 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d7804f290 con 0x7f9d78155960 2024-02-21T23:02:21.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:21.958+0000 7f9d6e000640 1 -- 172.21.15.28:0/1917418312 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f9d70022030 con 0x7f9d78155960 2024-02-21T23:02:22.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:22.024+0000 7f9d77200640 1 -- 172.21.15.28:0/1917418312 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9d7806d9d0 con 0x7f9d78155960 2024-02-21T23:02:22.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:22.026+0000 7f9d6e000640 1 -- 172.21.15.28:0/1917418312 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f9d70010040 con 0x7f9d78155960 2024-02-21T23:02:22.014 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:02:22.015 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":10,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:02:21.651243+0000","last_up_change":"0.000000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:02:22.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:22.027+0000 7f9d77200640 1 -- 172.21.15.28:0/1917418312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d78155960 msgr2=0x7f9d781eb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:22.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:22.027+0000 7f9d77200640 1 --2- 172.21.15.28:0/1917418312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d78155960 0x7f9d781eb1b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f9d7000efc0 tx=0x7f9d7000c490 comp rx=0 tx=0).stop 2024-02-21T23:02:22.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:22.027+0000 7f9d77200640 1 -- 172.21.15.28:0/1917418312 shutdown_connections 2024-02-21T23:02:22.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:22.027+0000 7f9d77200640 1 --2- 172.21.15.28:0/1917418312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d78155960 0x7f9d781eb1b0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:22.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:22.027+0000 7f9d77200640 1 --2- 172.21.15.28:0/1917418312 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9d780aff90 0x7f9d781eac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:22.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:22.027+0000 7f9d77200640 1 --2- 172.21.15.28:0/1917418312 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9d7806dbf0 0x7f9d780ae4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:22.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:22.027+0000 7f9d77200640 1 -- 172.21.15.28:0/1917418312 >> 172.21.15.28:0/1917418312 conn(0x7f9d780ac370 msgr2=0x7f9d780c50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:22.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:22.027+0000 7f9d77200640 1 -- 172.21.15.28:0/1917418312 shutdown_connections 2024-02-21T23:02:22.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:22.027+0000 7f9d77200640 1 -- 172.21.15.28:0/1917418312 wait complete. 2024-02-21T23:02:22.033 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-02-21T23:02:22.033 INFO:tasks.ceph:Waiting for OSDs to come up 2024-02-21T23:02:22.113 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-02-21T23:02:22.123+0000 7f09bd23a040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-21T23:02:22.189 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-02-21T23:02:22.200+0000 7f0d34c3e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-21T23:02:22.370 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-02-21T23:02:22.381+0000 7f4bcdc15640 -1 Falling back to public interface 2024-02-21T23:02:22.374 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-02-21T23:02:22.385+0000 7f41a0593640 -1 Falling back to public interface 2024-02-21T23:02:22.374 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-02-21T23:02:22.386+0000 7f7fb000e640 -1 Falling back to public interface 2024-02-21T23:02:22.376 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-02-21T23:02:22.386+0000 7f97ba013640 -1 Falling back to public interface 2024-02-21T23:02:22.381 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-02-21T23:02:22.392+0000 7f4870921640 -1 Falling back to public interface 2024-02-21T23:02:22.392 INFO:tasks.ceph.osd.7.smithi081.stderr:2024-02-21T23:02:22.403+0000 7f952be11640 -1 Falling back to public interface 2024-02-21T23:02:22.396 INFO:tasks.ceph.osd.4.smithi081.stderr:2024-02-21T23:02:22.407+0000 7f20cd415640 -1 Falling back to public interface 2024-02-21T23:02:22.401 INFO:tasks.ceph.osd.7.smithi081.stderr:2024-02-21T23:02:22.412+0000 7f9529e00640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6825/0: (98) Address already in use 2024-02-21T23:02:22.406 INFO:tasks.ceph.osd.3.smithi028.stderr:2024-02-21T23:02:22.418+0000 7efc70c12640 -1 Falling back to public interface 2024-02-21T23:02:22.433 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-02-21T23:02:22.444+0000 7f09bd23a040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-21T23:02:22.491 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-02-21T23:02:22.502+0000 7f09bd23a040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-21T23:02:22.515 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-02-21T23:02:22.527+0000 7f0d34c3e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-21T23:02:22.577 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-02-21T23:02:22.589+0000 7f0d34c3e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-21T23:02:22.620 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-02-21T23:02:22.631+0000 7f09bd23a040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-21T23:02:22.718 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-02-21T23:02:22.730+0000 7f0d34c3e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-21T23:02:22.838 DEBUG:teuthology.orchestra.run.smithi028:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-02-21T23:02:23.016 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:23.027+0000 7f411a000640 1 Processor -- start 2024-02-21T23:02:23.016 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:23.027+0000 7f411a000640 1 -- start start 2024-02-21T23:02:23.018 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:23.029+0000 7f411a000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f41141502a0 0x7f41140c2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:23.018 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:23.029+0000 7f411a000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f41140c34e0 0x7f41140c2370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:23.018 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:23.029+0000 7f411a000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f41140c28b0 0x7f41140c1050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:23.018 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:23.029+0000 7f411a000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41141539b0 con 0x7f41140c28b0 2024-02-21T23:02:23.018 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:23.029+0000 7f411a000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4114152da0 con 0x7f41141502a0 2024-02-21T23:02:23.018 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:23.029+0000 7f411a000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4114153080 con 0x7f41140c34e0 2024-02-21T23:02:23.018 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:23.029+0000 7f4119600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f41140c28b0 0x7f41140c1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:23.018 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:23.029+0000 7f4119600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f41140c28b0 0x7f41140c1050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57296/0 (socket says 172.21.15.28:57296) 2024-02-21T23:02:23.018 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:23.029+0000 7f4119600640 1 -- 172.21.15.28:0/2750439081 learned_addr learned my addr 172.21.15.28:0/2750439081 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:02:23.018 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:23.029+0000 7f4118c00640 1 --2- 172.21.15.28:0/2750439081 >> v2:172.21.15.81:3300/0 conn(0x7f41141502a0 0x7f41140c2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:23.018 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:23.029+0000 7f4113e00640 1 --2- 172.21.15.28:0/2750439081 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f41140c34e0 0x7f41140c2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:23.019 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:23.029+0000 7f4119600640 1 -- 172.21.15.28:0/2750439081 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f41140c34e0 msgr2=0x7f41140c2370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:23.019 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:23.029+0000 7f4119600640 1 --2- 172.21.15.28:0/2750439081 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f41140c34e0 0x7f41140c2370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:23.019 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:23.029+0000 7f4119600640 1 -- 172.21.15.28:0/2750439081 >> v2:172.21.15.81:3300/0 conn(0x7f41141502a0 msgr2=0x7f41140c2fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:23.019 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:23.029+0000 7f4119600640 1 --2- 172.21.15.28:0/2750439081 >> v2:172.21.15.81:3300/0 conn(0x7f41141502a0 0x7f41140c2fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:23.019 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:23.029+0000 7f4119600640 1 -- 172.21.15.28:0/2750439081 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4114153360 con 0x7f41140c28b0 2024-02-21T23:02:23.019 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:23.030+0000 7f4119600640 1 --2- 172.21.15.28:0/2750439081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41140c28b0 0x7f41140c1050 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f41140aba20 tx=0x7f410c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1153d5b2e899011 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:23.020 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:23.031+0000 7f4113400640 1 -- 172.21.15.28:0/2750439081 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f410c02ca80 con 0x7f41140c28b0 2024-02-21T23:02:23.020 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:23.031+0000 7f4113400640 1 -- 172.21.15.28:0/2750439081 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f410c02cc20 con 0x7f41140c28b0 2024-02-21T23:02:23.021 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:23.031+0000 7f4113400640 1 -- 172.21.15.28:0/2750439081 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f410c0343f0 con 0x7f41140c28b0 2024-02-21T23:02:23.021 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:23.031+0000 7f411a000640 1 -- 172.21.15.28:0/2750439081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41140c28b0 msgr2=0x7f41140c1050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:23.021 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:23.031+0000 7f411a000640 1 --2- 172.21.15.28:0/2750439081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41140c28b0 0x7f41140c1050 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f41140aba20 tx=0x7f410c02bee0 comp rx=0 tx=0).stop 2024-02-21T23:02:23.021 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:23.031+0000 7f411a000640 1 -- 172.21.15.28:0/2750439081 shutdown_connections 2024-02-21T23:02:23.021 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:23.031+0000 7f411a000640 1 --2- 172.21.15.28:0/2750439081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41140c28b0 0x7f41140c1050 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:23.021 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:23.031+0000 7f411a000640 1 --2- 172.21.15.28:0/2750439081 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f41140c34e0 0x7f41140c2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:23.021 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:23.031+0000 7f411a000640 1 --2- 172.21.15.28:0/2750439081 >> v2:172.21.15.81:3300/0 conn(0x7f41141502a0 0x7f41140c2fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:23.021 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:23.031+0000 7f411a000640 1 -- 172.21.15.28:0/2750439081 >> 172.21.15.28:0/2750439081 conn(0x7f411406b9d0 msgr2=0x7f41140c6550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:23.021 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:23.032+0000 7f411a000640 1 -- 172.21.15.28:0/2750439081 shutdown_connections 2024-02-21T23:02:23.021 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:23.032+0000 7f411a000640 1 -- 172.21.15.28:0/2750439081 wait complete. 2024-02-21T23:02:23.022 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:23.033+0000 7f411a000640 1 Processor -- start 2024-02-21T23:02:23.022 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:23.033+0000 7f411a000640 1 -- start start 2024-02-21T23:02:23.024 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:23.034+0000 7f411a000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f41140c28b0 0x7f41140ab340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:23.025 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:23.034+0000 7f411a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41140c34e0 0x7f41140a9fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:23.025 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:23.034+0000 7f411a000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f41141502a0 0x7f41140718f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:23.025 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:23.034+0000 7f411a000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41140c2d20 con 0x7f41140c34e0 2024-02-21T23:02:23.025 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:23.034+0000 7f411a000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4114150680 con 0x7f41141502a0 2024-02-21T23:02:23.025 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:23.034+0000 7f411a000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41140c1590 con 0x7f41140c28b0 2024-02-21T23:02:23.025 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:23.034+0000 7f4113e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41140c34e0 0x7f41140a9fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:23.025 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:23.034+0000 7f4113e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41140c34e0 0x7f41140a9fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57308/0 (socket says 172.21.15.28:57308) 2024-02-21T23:02:23.025 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:23.034+0000 7f4113e00640 1 -- 172.21.15.28:0/3624047963 learned_addr learned my addr 172.21.15.28:0/3624047963 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:02:23.025 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:23.035+0000 7f4118c00640 1 --2- 172.21.15.28:0/3624047963 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f41140c28b0 0x7f41140ab340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:23.025 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:23.035+0000 7f4119600640 1 --2- 172.21.15.28:0/3624047963 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f41141502a0 0x7f41140718f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:23.025 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:23.035+0000 7f4113e00640 1 -- 172.21.15.28:0/3624047963 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f41140c28b0 msgr2=0x7f41140ab340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:23.025 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:23.035+0000 7f4113e00640 1 --2- 172.21.15.28:0/3624047963 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f41140c28b0 0x7f41140ab340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:23.025 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:23.035+0000 7f4113e00640 1 -- 172.21.15.28:0/3624047963 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f41141502a0 msgr2=0x7f41140718f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:23.025 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:23.035+0000 7f4113e00640 1 --2- 172.21.15.28:0/3624047963 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f41141502a0 0x7f41140718f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:23.026 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:23.035+0000 7f4113e00640 1 -- 172.21.15.28:0/3624047963 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41140712f0 con 0x7f41140c34e0 2024-02-21T23:02:23.026 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:23.035+0000 7f4113e00640 1 --2- 172.21.15.28:0/3624047963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41140c34e0 0x7f41140a9fb0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f410000d800 tx=0x7f410000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:23.026 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:23.036+0000 7f4110c00640 1 -- 172.21.15.28:0/3624047963 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4100018020 con 0x7f41140c34e0 2024-02-21T23:02:23.026 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:23.036+0000 7f4110c00640 1 -- 172.21.15.28:0/3624047963 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41000096a0 con 0x7f41140c34e0 2024-02-21T23:02:23.026 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:23.036+0000 7f4110c00640 1 -- 172.21.15.28:0/3624047963 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4100009a70 con 0x7f41140c34e0 2024-02-21T23:02:23.026 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:23.036+0000 7f411a000640 1 -- 172.21.15.28:0/3624047963 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f411414ec50 con 0x7f41140c34e0 2024-02-21T23:02:23.026 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:23.036+0000 7f411a000640 1 -- 172.21.15.28:0/3624047963 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f411414f210 con 0x7f41140c34e0 2024-02-21T23:02:23.026 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:23.037+0000 7f411a000640 1 -- 172.21.15.28:0/3624047963 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4114071080 con 0x7f41140c34e0 2024-02-21T23:02:23.027 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:23.037+0000 7f4110c00640 1 -- 172.21.15.28:0/3624047963 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 2) v1 ==== 44915+0+0 (secure 0 0 0) 0x7f410001b020 con 0x7f41140c34e0 2024-02-21T23:02:23.027 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:23.038+0000 7f4110c00640 1 -- 172.21.15.28:0/3624047963 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f4100049fd0 con 0x7f41140c34e0 2024-02-21T23:02:23.028 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:23.040+0000 7f4110c00640 1 -- 172.21.15.28:0/3624047963 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f4100022030 con 0x7f41140c34e0 2024-02-21T23:02:23.095 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:23.107+0000 7f411a000640 1 -- 172.21.15.28:0/3624047963 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4114074b60 con 0x7f41140c34e0 2024-02-21T23:02:23.096 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:23.107+0000 7f4110c00640 1 -- 172.21.15.28:0/3624047963 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f4100009cc0 con 0x7f41140c34e0 2024-02-21T23:02:23.096 INFO:teuthology.misc.health.smithi028.stdout: 2024-02-21T23:02:23.097 INFO:teuthology.misc.health.smithi028.stdout:{"epoch":10,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:02:21.651243+0000","last_up_change":"0.000000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:02:23.098 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:23.109+0000 7f411a000640 1 -- 172.21.15.28:0/3624047963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41140c34e0 msgr2=0x7f41140a9fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:23.098 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:23.109+0000 7f411a000640 1 --2- 172.21.15.28:0/3624047963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41140c34e0 0x7f41140a9fb0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f410000d800 tx=0x7f410000dcc0 comp rx=0 tx=0).stop 2024-02-21T23:02:23.098 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:23.109+0000 7f411a000640 1 -- 172.21.15.28:0/3624047963 shutdown_connections 2024-02-21T23:02:23.098 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:23.109+0000 7f411a000640 1 --2- 172.21.15.28:0/3624047963 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f41141502a0 0x7f41140718f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:23.098 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:23.109+0000 7f411a000640 1 --2- 172.21.15.28:0/3624047963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41140c34e0 0x7f41140a9fb0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:23.098 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:23.109+0000 7f411a000640 1 --2- 172.21.15.28:0/3624047963 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f41140c28b0 0x7f41140ab340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:23.098 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:23.109+0000 7f411a000640 1 -- 172.21.15.28:0/3624047963 >> 172.21.15.28:0/3624047963 conn(0x7f411406b9d0 msgr2=0x7f41140bf980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:23.098 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:23.109+0000 7f411a000640 1 -- 172.21.15.28:0/3624047963 shutdown_connections 2024-02-21T23:02:23.098 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:23.109+0000 7f411a000640 1 -- 172.21.15.28:0/3624047963 wait complete. 2024-02-21T23:02:23.110 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-02-21T23:02:23.683 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-02-21T23:02:23.694+0000 7f095d800640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-02-21T23:02:23.684 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-02-21T23:02:23.695+0000 7f095d800640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-02-21T23:02:25.887 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-02-21T23:02:25.898+0000 7f97ba013640 -1 osd.6 0 log_to_monitors true 2024-02-21T23:02:25.913 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-02-21T23:02:25.924+0000 7f7fb000e640 -1 osd.1 0 log_to_monitors true 2024-02-21T23:02:25.952 INFO:tasks.ceph.osd.4.smithi081.stderr:2024-02-21T23:02:25.963+0000 7f20cd415640 -1 osd.4 0 log_to_monitors true 2024-02-21T23:02:25.961 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-02-21T23:02:25.973+0000 7f41a0593640 -1 osd.2 0 log_to_monitors true 2024-02-21T23:02:25.962 INFO:tasks.ceph.osd.7.smithi081.stderr:2024-02-21T23:02:25.974+0000 7f952be11640 -1 osd.7 0 log_to_monitors true 2024-02-21T23:02:25.964 INFO:tasks.ceph.osd.3.smithi028.stderr:2024-02-21T23:02:25.975+0000 7efc70c12640 -1 osd.3 0 log_to_monitors true 2024-02-21T23:02:26.178 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-02-21T23:02:26.189+0000 7f4bcdc15640 -1 osd.0 0 log_to_monitors true 2024-02-21T23:02:26.231 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-02-21T23:02:26.242+0000 7f4870921640 -1 osd.5 0 log_to_monitors true 2024-02-21T23:02:29.156 INFO:tasks.ceph.osd.7.smithi081.stderr:2024-02-21T23:02:29.167+0000 7f9526200640 -1 osd.7 0 waiting for initial osdmap 2024-02-21T23:02:29.165 INFO:tasks.ceph.osd.7.smithi081.stderr:2024-02-21T23:02:29.175+0000 7f951c200640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-21T23:02:29.182 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-02-21T23:02:29.193+0000 7f97b4a00640 -1 osd.6 0 waiting for initial osdmap 2024-02-21T23:02:29.192 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-02-21T23:02:29.203+0000 7f97aaa00640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-21T23:02:29.197 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-02-21T23:02:29.209+0000 7f486b400640 -1 osd.5 0 waiting for initial osdmap 2024-02-21T23:02:29.206 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-02-21T23:02:29.217+0000 7f4861400640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-21T23:02:29.213 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-02-21T23:02:29.225+0000 7f7faa400640 -1 osd.1 0 waiting for initial osdmap 2024-02-21T23:02:29.223 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-02-21T23:02:29.235+0000 7f7fa0400640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-21T23:02:29.243 INFO:tasks.ceph.osd.3.smithi028.stderr:2024-02-21T23:02:29.254+0000 7efc6b000640 -1 osd.3 0 waiting for initial osdmap 2024-02-21T23:02:29.251 INFO:tasks.ceph.osd.3.smithi028.stderr:2024-02-21T23:02:29.262+0000 7efc61000640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-21T23:02:29.261 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-02-21T23:02:29.272+0000 7f419ba00640 -1 osd.2 0 waiting for initial osdmap 2024-02-21T23:02:29.269 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-02-21T23:02:29.280+0000 7f4bc8000640 -1 osd.0 0 waiting for initial osdmap 2024-02-21T23:02:29.272 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-02-21T23:02:29.283+0000 7f4191000640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-21T23:02:29.273 INFO:tasks.ceph.osd.4.smithi081.stderr:2024-02-21T23:02:29.284+0000 7f20c8400640 -1 osd.4 0 waiting for initial osdmap 2024-02-21T23:02:29.276 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-02-21T23:02:29.288+0000 7f4bbe000640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-21T23:02:29.282 INFO:tasks.ceph.osd.4.smithi081.stderr:2024-02-21T23:02:29.293+0000 7f20bda00640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-21T23:02:29.917 DEBUG:teuthology.orchestra.run.smithi028:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-02-21T23:02:30.111 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.122+0000 7f367ea00640 1 Processor -- start 2024-02-21T23:02:30.111 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.122+0000 7f367ea00640 1 -- start start 2024-02-21T23:02:30.112 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.123+0000 7f367ea00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f36780a9710 0x7f36780ab760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:30.112 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.123+0000 7f367ea00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f367806e9a0 0x7f36780c30f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:30.112 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.123+0000 7f367ea00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36780c12f0 con 0x7f3678150900 2024-02-21T23:02:30.112 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.123+0000 7f367ea00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f367806e510 con 0x7f367806e9a0 2024-02-21T23:02:30.112 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.123+0000 7f367ea00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36780ac840 con 0x7f36780a9710 2024-02-21T23:02:30.113 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.123+0000 7f367ca00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f367806e9a0 0x7f36780c30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:30.113 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.124+0000 7f367ca00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f367806e9a0 0x7f36780c30f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58764/0 (socket says 172.21.15.28:58764) 2024-02-21T23:02:30.113 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.124+0000 7f367ca00640 1 -- 172.21.15.28:0/3210750418 learned_addr learned my addr 172.21.15.28:0/3210750418 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:02:30.113 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.124+0000 7f3677400640 1 --2- 172.21.15.28:0/3210750418 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f36780a9710 0x7f36780ab760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:30.113 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.124+0000 7f367ca00640 1 -- 172.21.15.28:0/3210750418 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f36780a9710 msgr2=0x7f36780ab760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:30.114 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.124+0000 7f367ca00640 1 --2- 172.21.15.28:0/3210750418 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f36780a9710 0x7f36780ab760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:30.114 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.124+0000 7f367ca00640 1 -- 172.21.15.28:0/3210750418 >> v1:172.21.15.28:6789/0 conn(0x7f3678150900 legacy=0x7f3678153920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:30.114 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.124+0000 7f3676a00640 1 -- 172.21.15.28:0/3210750418 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 196043549 0 0) 0x7f36780c12f0 con 0x7f3678150900 2024-02-21T23:02:30.114 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.124+0000 7f367ca00640 1 -- 172.21.15.28:0/3210750418 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36780a9ba0 con 0x7f367806e9a0 2024-02-21T23:02:30.114 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.124+0000 7f3677400640 1 --2- 172.21.15.28:0/3210750418 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f36780a9710 0x7f36780ab760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:02:30.115 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.125+0000 7f367ca00640 1 --2- 172.21.15.28:0/3210750418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f367806e9a0 0x7f36780c30f0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f3670009e50 tx=0x7f367002bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0d0cce27536ea20 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:30.115 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.125+0000 7f3676a00640 1 -- 172.21.15.28:0/3210750418 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f367002caf0 con 0x7f367806e9a0 2024-02-21T23:02:30.115 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.125+0000 7f3676a00640 1 -- 172.21.15.28:0/3210750418 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f367002cc90 con 0x7f367806e9a0 2024-02-21T23:02:30.115 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.125+0000 7f3676a00640 1 -- 172.21.15.28:0/3210750418 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3670034610 con 0x7f367806e9a0 2024-02-21T23:02:30.115 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.126+0000 7f367ea00640 1 -- 172.21.15.28:0/3210750418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f367806e9a0 msgr2=0x7f36780c30f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:30.115 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.126+0000 7f367ea00640 1 --2- 172.21.15.28:0/3210750418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f367806e9a0 0x7f36780c30f0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f3670009e50 tx=0x7f367002bf50 comp rx=0 tx=0).stop 2024-02-21T23:02:30.115 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.126+0000 7f367ea00640 1 -- 172.21.15.28:0/3210750418 shutdown_connections 2024-02-21T23:02:30.116 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.126+0000 7f367ea00640 1 --2- 172.21.15.28:0/3210750418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f367806e9a0 0x7f36780c30f0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:30.116 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.126+0000 7f367ea00640 1 --2- 172.21.15.28:0/3210750418 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f36780a9710 0x7f36780ab760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:30.116 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.126+0000 7f367ea00640 1 -- 172.21.15.28:0/3210750418 >> 172.21.15.28:0/3210750418 conn(0x7f367806af50 msgr2=0x7f367806b360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:30.116 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.126+0000 7f367ea00640 1 -- 172.21.15.28:0/3210750418 shutdown_connections 2024-02-21T23:02:30.116 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.126+0000 7f367ea00640 1 -- 172.21.15.28:0/3210750418 wait complete. 2024-02-21T23:02:30.116 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.128+0000 7f367ea00640 1 Processor -- start 2024-02-21T23:02:30.116 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.128+0000 7f367ea00640 1 -- start start 2024-02-21T23:02:30.118 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.129+0000 7f367ea00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f367806e9a0 0x7f36781ee040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:30.118 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.129+0000 7f367ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3678150900 0x7f3678155070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:30.118 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.129+0000 7f367ea00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f36781ee580 0x7f36781555b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:30.118 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.129+0000 7f367ea00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f367806e1a0 con 0x7f3678150900 2024-02-21T23:02:30.118 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.129+0000 7f367ea00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36780aa160 con 0x7f36781ee580 2024-02-21T23:02:30.118 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.129+0000 7f367ea00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f367806ee30 con 0x7f367806e9a0 2024-02-21T23:02:30.118 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.129+0000 7f3677400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3678150900 0x7f3678155070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:30.118 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.129+0000 7f3677400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3678150900 0x7f3678155070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58778/0 (socket says 172.21.15.28:58778) 2024-02-21T23:02:30.119 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.129+0000 7f3677400640 1 -- 172.21.15.28:0/28913178 learned_addr learned my addr 172.21.15.28:0/28913178 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:02:30.119 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.129+0000 7f367ca00640 1 --2- 172.21.15.28:0/28913178 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f36781ee580 0x7f36781555b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:30.119 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.129+0000 7f3677400640 1 -- 172.21.15.28:0/28913178 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f367806e9a0 msgr2=0x7f36781ee040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:30.119 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.129+0000 7f3677400640 1 --2- 172.21.15.28:0/28913178 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f367806e9a0 0x7f36781ee040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:30.119 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.129+0000 7f3677400640 1 -- 172.21.15.28:0/28913178 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f36781ee580 msgr2=0x7f36781555b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:30.119 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.129+0000 7f3677400640 1 --2- 172.21.15.28:0/28913178 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f36781ee580 0x7f36781555b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:30.119 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.129+0000 7f3677400640 1 -- 172.21.15.28:0/28913178 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36781eeaa0 con 0x7f3678150900 2024-02-21T23:02:30.119 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.130+0000 7f3677400640 1 --2- 172.21.15.28:0/28913178 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3678150900 0x7f3678155070 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f366400c7f0 tx=0x7f366400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:30.120 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.130+0000 7f366fe00640 1 -- 172.21.15.28:0/28913178 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3664015070 con 0x7f3678150900 2024-02-21T23:02:30.120 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.130+0000 7f366fe00640 1 -- 172.21.15.28:0/28913178 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3664007a30 con 0x7f3678150900 2024-02-21T23:02:30.120 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.130+0000 7f366fe00640 1 -- 172.21.15.28:0/28913178 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3664007e00 con 0x7f3678150900 2024-02-21T23:02:30.120 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.130+0000 7f367ea00640 1 -- 172.21.15.28:0/28913178 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36781dc6a0 con 0x7f3678150900 2024-02-21T23:02:30.120 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.130+0000 7f367ea00640 1 -- 172.21.15.28:0/28913178 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3678154330 con 0x7f3678150900 2024-02-21T23:02:30.120 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.131+0000 7f367ea00640 1 -- 172.21.15.28:0/28913178 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3678068400 con 0x7f3678150900 2024-02-21T23:02:30.121 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.132+0000 7f366fe00640 1 -- 172.21.15.28:0/28913178 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 4) v1 ==== 89724+0+0 (secure 0 0 0) 0x7f3664018070 con 0x7f3678150900 2024-02-21T23:02:30.122 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.133+0000 7f366fe00640 1 --2- 172.21.15.28:0/28913178 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f365806b440 0x7f365806d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:30.122 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.133+0000 7f366fe00640 1 -- 172.21.15.28:0/28913178 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2231+0+0 (secure 0 0 0) 0x7f3664089310 con 0x7f3678150900 2024-02-21T23:02:30.122 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.134+0000 7f3677e00640 1 --2- 172.21.15.28:0/28913178 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f365806b440 0x7f365806d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:30.123 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.135+0000 7f3677e00640 1 --2- 172.21.15.28:0/28913178 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f365806b440 0x7f365806d900 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3668001560 tx=0x7f3668007930 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:30.126 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.138+0000 7f366fe00640 1 -- 172.21.15.28:0/28913178 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3664058ff0 con 0x7f3678150900 2024-02-21T23:02:30.336 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.347+0000 7f367ea00640 1 -- 172.21.15.28:0/28913178 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3678154530 con 0x7f3678150900 2024-02-21T23:02:30.338 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.349+0000 7f366fe00640 1 -- 172.21.15.28:0/28913178 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) v1 ==== 74+0+10849 (secure 0 0 0) 0x7f3678154530 con 0x7f3678150900 2024-02-21T23:02:30.338 INFO:teuthology.misc.health.smithi028.stdout: 2024-02-21T23:02:30.338 INFO:teuthology.misc.health.smithi028.stdout:{"epoch":13,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:02:30.169021+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:02:30.341 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.352+0000 7f367ea00640 1 -- 172.21.15.28:0/28913178 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f365806b440 msgr2=0x7f365806d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:30.341 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.352+0000 7f367ea00640 1 --2- 172.21.15.28:0/28913178 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f365806b440 0x7f365806d900 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3668001560 tx=0x7f3668007930 comp rx=0 tx=0).stop 2024-02-21T23:02:30.341 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.352+0000 7f367ea00640 1 -- 172.21.15.28:0/28913178 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3678150900 msgr2=0x7f3678155070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:30.341 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.352+0000 7f367ea00640 1 --2- 172.21.15.28:0/28913178 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3678150900 0x7f3678155070 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f366400c7f0 tx=0x7f366400cbf0 comp rx=0 tx=0).stop 2024-02-21T23:02:30.341 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.352+0000 7f367ea00640 1 -- 172.21.15.28:0/28913178 shutdown_connections 2024-02-21T23:02:30.341 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.352+0000 7f367ea00640 1 --2- 172.21.15.28:0/28913178 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f365806b440 0x7f365806d900 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:30.341 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.352+0000 7f367ea00640 1 --2- 172.21.15.28:0/28913178 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f36781ee580 0x7f36781555b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:30.341 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.352+0000 7f367ea00640 1 --2- 172.21.15.28:0/28913178 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3678150900 0x7f3678155070 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:30.341 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.352+0000 7f367ea00640 1 --2- 172.21.15.28:0/28913178 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f367806e9a0 0x7f36781ee040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:30.341 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.352+0000 7f367ea00640 1 -- 172.21.15.28:0/28913178 >> 172.21.15.28:0/28913178 conn(0x7f367806af50 msgr2=0x7f3678150d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:30.342 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.352+0000 7f367ea00640 1 -- 172.21.15.28:0/28913178 shutdown_connections 2024-02-21T23:02:30.342 INFO:teuthology.misc.health.smithi028.stderr:2024-02-21T23:02:30.352+0000 7f367ea00640 1 -- 172.21.15.28:0/28913178 wait complete. 2024-02-21T23:02:30.353 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-02-21T23:02:30.354 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-02-21T23:02:30.354 INFO:tasks.ceph.mds.a:Restarting daemon 2024-02-21T23:02:30.354 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-02-21T23:02:30.359 INFO:tasks.ceph.mds.a:Started 2024-02-21T23:02:30.359 INFO:tasks.ceph.mds.c:Restarting daemon 2024-02-21T23:02:30.359 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-02-21T23:02:30.363 INFO:tasks.ceph.mds.c:Started 2024-02-21T23:02:30.364 INFO:tasks.ceph.mds.b:Restarting daemon 2024-02-21T23:02:30.364 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-02-21T23:02:30.372 INFO:tasks.ceph.mds.b:Started 2024-02-21T23:02:30.372 INFO:tasks.ceph.mds.d:Restarting daemon 2024-02-21T23:02:30.372 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-02-21T23:02:30.379 INFO:tasks.ceph.mds.d:Started 2024-02-21T23:02:30.379 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-02-21T23:02:30.382 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:02:30.502 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-02-21T23:02:30.503 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-02-21T23:02:30.532 INFO:tasks.ceph.mds.b.smithi081.stdout:starting mds.b at 2024-02-21T23:02:30.532 INFO:tasks.ceph.mds.d.smithi081.stdout:starting mds.d at 2024-02-21T23:02:30.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.567+0000 7f9496c00640 1 Processor -- start 2024-02-21T23:02:30.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.568+0000 7f9496c00640 1 -- start start 2024-02-21T23:02:30.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.569+0000 7f9496c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f94900cab00 0x7f94900c5300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:30.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.569+0000 7f9496c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94900c5ba0 con 0x7f949013e5e0 2024-02-21T23:02:30.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.569+0000 7f9496c00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94900c5e60 con 0x7f94901587b0 2024-02-21T23:02:30.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.569+0000 7f9496c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f949013ea50 con 0x7f94900cab00 2024-02-21T23:02:30.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.569+0000 7f9494c00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f949013e5e0 0x7f94900bfc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43190/0 (socket says 172.21.15.28:43190) 2024-02-21T23:02:30.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.569+0000 7f9494c00640 1 -- 172.21.15.28:0/547719922 learned_addr learned my addr 172.21.15.28:0/547719922 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:02:30.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.569+0000 7f948fe00640 1 --2- 172.21.15.28:0/547719922 >> v2:172.21.15.28:3300/0 conn(0x7f94900cab00 0x7f94900c5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:30.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.569+0000 7f948f400640 1 -- 172.21.15.28:0/547719922 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1782197908 0 0) 0x7f94900c5ba0 con 0x7f949013e5e0 2024-02-21T23:02:30.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.569+0000 7f948f400640 1 -- 172.21.15.28:0/547719922 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9474003520 con 0x7f949013e5e0 2024-02-21T23:02:30.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.569+0000 7f948f400640 1 -- 172.21.15.28:0/547719922 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3683138234 0 0) 0x7f94900c5e60 con 0x7f94901587b0 2024-02-21T23:02:30.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.569+0000 7f948f400640 1 -- 172.21.15.28:0/547719922 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94900c5ba0 con 0x7f94901587b0 2024-02-21T23:02:30.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.569+0000 7f948fe00640 1 -- 172.21.15.28:0/547719922 >> v1:172.21.15.81:6789/0 conn(0x7f94901587b0 legacy=0x7f949013ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:30.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.569+0000 7f948fe00640 1 -- 172.21.15.28:0/547719922 >> v1:172.21.15.28:6789/0 conn(0x7f949013e5e0 legacy=0x7f94900bfc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:30.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.569+0000 7f948fe00640 1 -- 172.21.15.28:0/547719922 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94900c6140 con 0x7f94900cab00 2024-02-21T23:02:30.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.571+0000 7f948fe00640 1 --2- 172.21.15.28:0/547719922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94900cab00 0x7f94900c5300 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f94800097a0 tx=0x7f948002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c56ab8eaad83378 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:30.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.571+0000 7f948f400640 1 -- 172.21.15.28:0/547719922 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94800315a0 con 0x7f94900cab00 2024-02-21T23:02:30.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.571+0000 7f948f400640 1 -- 172.21.15.28:0/547719922 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9480037070 con 0x7f94900cab00 2024-02-21T23:02:30.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.571+0000 7f948f400640 1 -- 172.21.15.28:0/547719922 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9480031960 con 0x7f94900cab00 2024-02-21T23:02:30.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.571+0000 7f9496c00640 1 -- 172.21.15.28:0/547719922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94900cab00 msgr2=0x7f94900c5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:30.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.571+0000 7f9496c00640 1 --2- 172.21.15.28:0/547719922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94900cab00 0x7f94900c5300 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f94800097a0 tx=0x7f948002b840 comp rx=0 tx=0).stop 2024-02-21T23:02:30.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.571+0000 7f9496c00640 1 -- 172.21.15.28:0/547719922 shutdown_connections 2024-02-21T23:02:30.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.571+0000 7f9496c00640 1 --2- 172.21.15.28:0/547719922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94900cab00 0x7f94900c5300 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:30.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.571+0000 7f9496c00640 1 -- 172.21.15.28:0/547719922 >> 172.21.15.28:0/547719922 conn(0x7f949008b1c0 msgr2=0x7f94900725a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:30.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.571+0000 7f9496c00640 1 -- 172.21.15.28:0/547719922 shutdown_connections 2024-02-21T23:02:30.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.572+0000 7f9496c00640 1 -- 172.21.15.28:0/547719922 wait complete. 2024-02-21T23:02:30.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.573+0000 7f9496c00640 1 Processor -- start 2024-02-21T23:02:30.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.573+0000 7f9496c00640 1 -- start start 2024-02-21T23:02:30.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.574+0000 7f9496c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94900cab00 0x7f94901dd930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:30.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.574+0000 7f9496c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f949013e5e0 0x7f94901eeb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:30.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.574+0000 7f9496c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f94901587b0 0x7f94900c96d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:30.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.574+0000 7f9496c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94900c5840 con 0x7f94900cab00 2024-02-21T23:02:30.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.574+0000 7f9496c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94900c6790 con 0x7f949013e5e0 2024-02-21T23:02:30.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.574+0000 7f9496c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94900caee0 con 0x7f94901587b0 2024-02-21T23:02:30.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.574+0000 7f948fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94900cab00 0x7f94901dd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:30.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.574+0000 7f948fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94900cab00 0x7f94901dd930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58822/0 (socket says 172.21.15.28:58822) 2024-02-21T23:02:30.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.574+0000 7f948fe00640 1 -- 172.21.15.28:0/3488024172 learned_addr learned my addr 172.21.15.28:0/3488024172 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:02:30.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.574+0000 7f9494c00640 1 --2- 172.21.15.28:0/3488024172 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f94901587b0 0x7f94900c96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:30.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.574+0000 7f9487e00640 1 --2- 172.21.15.28:0/3488024172 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f949013e5e0 0x7f94901eeb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:30.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.575+0000 7f948fe00640 1 -- 172.21.15.28:0/3488024172 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f94901587b0 msgr2=0x7f94900c96d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:30.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.575+0000 7f948fe00640 1 --2- 172.21.15.28:0/3488024172 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f94901587b0 0x7f94900c96d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:30.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.575+0000 7f948fe00640 1 -- 172.21.15.28:0/3488024172 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f949013e5e0 msgr2=0x7f94901eeb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:30.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.575+0000 7f948fe00640 1 --2- 172.21.15.28:0/3488024172 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f949013e5e0 0x7f94901eeb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:30.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.575+0000 7f948fe00640 1 -- 172.21.15.28:0/3488024172 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94901ef0d0 con 0x7f94900cab00 2024-02-21T23:02:30.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.575+0000 7f9487e00640 1 --2- 172.21.15.28:0/3488024172 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f949013e5e0 0x7f94901eeb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-21T23:02:30.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.575+0000 7f948fe00640 1 --2- 172.21.15.28:0/3488024172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94900cab00 0x7f94901dd930 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9480034040 tx=0x7f94800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:30.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.575+0000 7f948cc00640 1 -- 172.21.15.28:0/3488024172 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9480032630 con 0x7f94900cab00 2024-02-21T23:02:30.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.575+0000 7f948cc00640 1 -- 172.21.15.28:0/3488024172 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94800327f0 con 0x7f94900cab00 2024-02-21T23:02:30.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.575+0000 7f948cc00640 1 -- 172.21.15.28:0/3488024172 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9480032be0 con 0x7f94900cab00 2024-02-21T23:02:30.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.575+0000 7f9496c00640 1 -- 172.21.15.28:0/3488024172 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94901e0590 con 0x7f94900cab00 2024-02-21T23:02:30.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.575+0000 7f9496c00640 1 -- 172.21.15.28:0/3488024172 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94900c9c10 con 0x7f94900cab00 2024-02-21T23:02:30.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.576+0000 7f9496c00640 1 -- 172.21.15.28:0/3488024172 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9490067f30 con 0x7f94900cab00 2024-02-21T23:02:30.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.577+0000 7f948cc00640 1 -- 172.21.15.28:0/3488024172 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 4) v1 ==== 89724+0+0 (secure 0 0 0) 0x7f948003b3f0 con 0x7f94900cab00 2024-02-21T23:02:30.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.578+0000 7f948cc00640 1 --2- 172.21.15.28:0/3488024172 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f946c06b470 0x7f946c06d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:30.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.578+0000 7f948cc00640 1 -- 172.21.15.28:0/3488024172 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 4471+0+0 (secure 0 0 0) 0x7f948002f030 con 0x7f94900cab00 2024-02-21T23:02:30.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.578+0000 7f9487e00640 1 --2- 172.21.15.28:0/3488024172 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f946c06b470 0x7f946c06d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:30.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.579+0000 7f9487e00640 1 --2- 172.21.15.28:0/3488024172 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f946c06b470 0x7f946c06d930 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9478007fd0 tx=0x7f9478009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:30.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.583+0000 7f948cc00640 1 -- 172.21.15.28:0/3488024172 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f9480040070 con 0x7f94900cab00 2024-02-21T23:02:30.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.780+0000 7f9496c00640 1 -- 172.21.15.28:0/3488024172 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f94901e0790 con 0x7f94900cab00 2024-02-21T23:02:30.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.782+0000 7f948cc00640 1 -- 172.21.15.28:0/3488024172 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) v1 ==== 74+0+10849 (secure 0 0 0) 0x7f948007a8b0 con 0x7f94900cab00 2024-02-21T23:02:30.771 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:02:30.771 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":13,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:02:30.169021+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:02:30.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.784+0000 7f9496c00640 1 -- 172.21.15.28:0/3488024172 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f946c06b470 msgr2=0x7f946c06d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:30.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.784+0000 7f9496c00640 1 --2- 172.21.15.28:0/3488024172 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f946c06b470 0x7f946c06d930 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9478007fd0 tx=0x7f9478009040 comp rx=0 tx=0).stop 2024-02-21T23:02:30.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.784+0000 7f9496c00640 1 -- 172.21.15.28:0/3488024172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94900cab00 msgr2=0x7f94901dd930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:30.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.784+0000 7f9496c00640 1 --2- 172.21.15.28:0/3488024172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94900cab00 0x7f94901dd930 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9480034040 tx=0x7f94800046c0 comp rx=0 tx=0).stop 2024-02-21T23:02:30.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.784+0000 7f9496c00640 1 -- 172.21.15.28:0/3488024172 shutdown_connections 2024-02-21T23:02:30.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.784+0000 7f9496c00640 1 --2- 172.21.15.28:0/3488024172 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f946c06b470 0x7f946c06d930 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:30.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.784+0000 7f9496c00640 1 --2- 172.21.15.28:0/3488024172 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f94901587b0 0x7f94900c96d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:30.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.784+0000 7f9496c00640 1 --2- 172.21.15.28:0/3488024172 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f949013e5e0 0x7f94901eeb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:30.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.784+0000 7f9496c00640 1 --2- 172.21.15.28:0/3488024172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94900cab00 0x7f94901dd930 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:30.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.784+0000 7f9496c00640 1 -- 172.21.15.28:0/3488024172 >> 172.21.15.28:0/3488024172 conn(0x7f949008b1c0 msgr2=0x7f94900c8720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:30.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.785+0000 7f9496c00640 1 -- 172.21.15.28:0/3488024172 shutdown_connections 2024-02-21T23:02:30.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:30.785+0000 7f9496c00640 1 -- 172.21.15.28:0/3488024172 wait complete. 2024-02-21T23:02:30.790 INFO:tasks.cephfs.filesystem.ceph_manager:[] 2024-02-21T23:02:30.790 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:02:30.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.006+0000 7fcffdc00640 1 Processor -- start 2024-02-21T23:02:30.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.006+0000 7fcffdc00640 1 -- start start 2024-02-21T23:02:30.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.007+0000 7fcffdc00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcff80c3490 0x7fcff806dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:30.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.007+0000 7fcffdc00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcff80a5f40 0x7fcff8152140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:30.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.007+0000 7fcffdc00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcff806e750 con 0x7fcff8152680 2024-02-21T23:02:30.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.007+0000 7fcffdc00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcff806eff0 con 0x7fcff80a5f40 2024-02-21T23:02:30.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.007+0000 7fcffdc00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcff80bfde0 con 0x7fcff80c3490 2024-02-21T23:02:30.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.007+0000 7fcfee000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcff80a5f40 0x7fcff8152140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:30.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.008+0000 7fcfee000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcff80a5f40 0x7fcff8152140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58826/0 (socket says 172.21.15.28:58826) 2024-02-21T23:02:30.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.008+0000 7fcfee000640 1 -- 172.21.15.28:0/3287610716 learned_addr learned my addr 172.21.15.28:0/3287610716 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:02:30.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.008+0000 7fcff6a00640 1 --2- 172.21.15.28:0/3287610716 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcff80c3490 0x7fcff806dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:30.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.008+0000 7fcfee000640 1 -- 172.21.15.28:0/3287610716 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcff80c3490 msgr2=0x7fcff806dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:30.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.008+0000 7fcfee000640 1 --2- 172.21.15.28:0/3287610716 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcff80c3490 0x7fcff806dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:30.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.008+0000 7fcfee000640 1 -- 172.21.15.28:0/3287610716 >> v1:172.21.15.81:6789/0 conn(0x7fcff8152680 legacy=0x7fcff80c0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:30.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.008+0000 7fcfee000640 1 -- 172.21.15.28:0/3287610716 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcff806ea30 con 0x7fcff80a5f40 2024-02-21T23:02:30.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.009+0000 7fcfee000640 1 --2- 172.21.15.28:0/3287610716 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcff80a5f40 0x7fcff8152140 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fcfe0009e50 tx=0x7fcfe002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a9ab98e3dafa6c3 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:30.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.010+0000 7fcff6000640 1 -- 172.21.15.28:0/3287610716 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfe002c9e0 con 0x7fcff80a5f40 2024-02-21T23:02:30.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.010+0000 7fcff6000640 1 -- 172.21.15.28:0/3287610716 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfe002cb80 con 0x7fcff80a5f40 2024-02-21T23:02:30.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.010+0000 7fcff6000640 1 -- 172.21.15.28:0/3287610716 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfe00343f0 con 0x7fcff80a5f40 2024-02-21T23:02:30.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.010+0000 7fcffdc00640 1 -- 172.21.15.28:0/3287610716 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcff80a5f40 msgr2=0x7fcff8152140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:30.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.010+0000 7fcffdc00640 1 --2- 172.21.15.28:0/3287610716 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcff80a5f40 0x7fcff8152140 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fcfe0009e50 tx=0x7fcfe002be40 comp rx=0 tx=0).stop 2024-02-21T23:02:31.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.011+0000 7fcffdc00640 1 -- 172.21.15.28:0/3287610716 shutdown_connections 2024-02-21T23:02:31.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.011+0000 7fcffdc00640 1 --2- 172.21.15.28:0/3287610716 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcff80a5f40 0x7fcff8152140 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:31.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.011+0000 7fcffdc00640 1 --2- 172.21.15.28:0/3287610716 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcff80c3490 0x7fcff806dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:31.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.011+0000 7fcffdc00640 1 -- 172.21.15.28:0/3287610716 >> 172.21.15.28:0/3287610716 conn(0x7fcff80c67f0 msgr2=0x7fcff80c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:31.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.011+0000 7fcffdc00640 1 -- 172.21.15.28:0/3287610716 shutdown_connections 2024-02-21T23:02:31.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.011+0000 7fcffdc00640 1 -- 172.21.15.28:0/3287610716 wait complete. 2024-02-21T23:02:31.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.012+0000 7fcffdc00640 1 Processor -- start 2024-02-21T23:02:31.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.012+0000 7fcffdc00640 1 -- start start 2024-02-21T23:02:31.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.013+0000 7fcffdc00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcff80a5f40 0x7fcff814aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:31.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.013+0000 7fcffdc00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcff80c3490 0x7fcff8146cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:31.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.013+0000 7fcffdc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcff8152680 0x7fcff8147210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:31.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.013+0000 7fcffdc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcff80c1c70 con 0x7fcff8152680 2024-02-21T23:02:31.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.013+0000 7fcffdc00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcff806e4f0 con 0x7fcff80a5f40 2024-02-21T23:02:31.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.013+0000 7fcffdc00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcff80c1df0 con 0x7fcff80c3490 2024-02-21T23:02:31.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.014+0000 7fcff7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcff8152680 0x7fcff8147210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:31.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.014+0000 7fcff7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcff8152680 0x7fcff8147210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58830/0 (socket says 172.21.15.28:58830) 2024-02-21T23:02:31.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.014+0000 7fcff7400640 1 -- 172.21.15.28:0/7227013 learned_addr learned my addr 172.21.15.28:0/7227013 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:02:31.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.014+0000 7fcff6a00640 1 --2- 172.21.15.28:0/7227013 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcff80a5f40 0x7fcff814aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:31.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.014+0000 7fcfee000640 1 --2- 172.21.15.28:0/7227013 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcff80c3490 0x7fcff8146cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:31.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.014+0000 7fcff7400640 1 -- 172.21.15.28:0/7227013 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcff80c3490 msgr2=0x7fcff8146cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:31.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.014+0000 7fcff7400640 1 --2- 172.21.15.28:0/7227013 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcff80c3490 0x7fcff8146cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:31.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.014+0000 7fcff7400640 1 -- 172.21.15.28:0/7227013 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcff80a5f40 msgr2=0x7fcff814aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:31.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.014+0000 7fcff7400640 1 --2- 172.21.15.28:0/7227013 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcff80a5f40 0x7fcff814aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:31.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.014+0000 7fcff7400640 1 -- 172.21.15.28:0/7227013 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcff8147750 con 0x7fcff8152680 2024-02-21T23:02:31.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.014+0000 7fcff7400640 1 --2- 172.21.15.28:0/7227013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcff8152680 0x7fcff8147210 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fcff000b410 tx=0x7fcff000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:31.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.015+0000 7fcfef400640 1 -- 172.21.15.28:0/7227013 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff0015030 con 0x7fcff8152680 2024-02-21T23:02:31.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.015+0000 7fcfef400640 1 -- 172.21.15.28:0/7227013 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcff0004020 con 0x7fcff8152680 2024-02-21T23:02:31.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.015+0000 7fcfef400640 1 -- 172.21.15.28:0/7227013 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff0004410 con 0x7fcff8152680 2024-02-21T23:02:31.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.015+0000 7fcffdc00640 1 -- 172.21.15.28:0/7227013 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcff80734a0 con 0x7fcff8152680 2024-02-21T23:02:31.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.015+0000 7fcffdc00640 1 -- 172.21.15.28:0/7227013 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcff8073130 con 0x7fcff8152680 2024-02-21T23:02:31.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.016+0000 7fcffdc00640 1 -- 172.21.15.28:0/7227013 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcff8147ab0 con 0x7fcff8152680 2024-02-21T23:02:31.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.017+0000 7fcfef400640 1 -- 172.21.15.28:0/7227013 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 4) v1 ==== 89724+0+0 (secure 0 0 0) 0x7fcff0018070 con 0x7fcff8152680 2024-02-21T23:02:31.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.018+0000 7fcfef400640 1 --2- 172.21.15.28:0/7227013 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcfd406b470 0x7fcfd406d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:31.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.018+0000 7fcfef400640 1 -- 172.21.15.28:0/7227013 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 4471+0+0 (secure 0 0 0) 0x7fcff00896d0 con 0x7fcff8152680 2024-02-21T23:02:31.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.018+0000 7fcff6a00640 1 --2- 172.21.15.28:0/7227013 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcfd406b470 0x7fcfd406d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:31.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.019+0000 7fcff6a00640 1 --2- 172.21.15.28:0/7227013 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcfd406b470 0x7fcfd406d930 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8006fd0 tx=0x7fcfe8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:31.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.023+0000 7fcfef400640 1 -- 172.21.15.28:0/7227013 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fcff8147ab0 con 0x7fcff8152680 2024-02-21T23:02:31.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.258+0000 7fcffdc00640 1 -- 172.21.15.28:0/7227013 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcff8067f30 con 0x7fcff8152680 2024-02-21T23:02:31.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.259+0000 7fcfef400640 1 -- 172.21.15.28:0/7227013 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1324 (secure 0 0 0) 0x7fcff0058af0 con 0x7fcff8152680 2024-02-21T23:02:31.248 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:02:31.248 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4230,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/2559793999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2559793999},{"type":"v1","addr":"172.21.15.28:6834","nonce":2559793999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-02-21T23:02:31.248 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 2 2024-02-21T23:02:31.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.261+0000 7fcffdc00640 1 -- 172.21.15.28:0/7227013 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcfd406b470 msgr2=0x7fcfd406d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:31.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.261+0000 7fcffdc00640 1 --2- 172.21.15.28:0/7227013 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcfd406b470 0x7fcfd406d930 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8006fd0 tx=0x7fcfe8008040 comp rx=0 tx=0).stop 2024-02-21T23:02:31.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.261+0000 7fcffdc00640 1 -- 172.21.15.28:0/7227013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcff8152680 msgr2=0x7fcff8147210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:31.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.261+0000 7fcffdc00640 1 --2- 172.21.15.28:0/7227013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcff8152680 0x7fcff8147210 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fcff000b410 tx=0x7fcff000b8d0 comp rx=0 tx=0).stop 2024-02-21T23:02:31.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.262+0000 7fcffdc00640 1 -- 172.21.15.28:0/7227013 shutdown_connections 2024-02-21T23:02:31.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.262+0000 7fcffdc00640 1 --2- 172.21.15.28:0/7227013 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcfd406b470 0x7fcfd406d930 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:31.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.262+0000 7fcffdc00640 1 --2- 172.21.15.28:0/7227013 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcff80c3490 0x7fcff8146cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:31.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.262+0000 7fcffdc00640 1 --2- 172.21.15.28:0/7227013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcff8152680 0x7fcff8147210 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:31.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.262+0000 7fcffdc00640 1 --2- 172.21.15.28:0/7227013 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcff80a5f40 0x7fcff814aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:31.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.262+0000 7fcffdc00640 1 -- 172.21.15.28:0/7227013 >> 172.21.15.28:0/7227013 conn(0x7fcff80c67f0 msgr2=0x7fcff814fbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:31.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.262+0000 7fcffdc00640 1 -- 172.21.15.28:0/7227013 shutdown_connections 2024-02-21T23:02:31.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:31.262+0000 7fcffdc00640 1 -- 172.21.15.28:0/7227013 wait complete. 2024-02-21T23:02:33.268 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:02:33.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.476+0000 7efe7ae00640 1 Processor -- start 2024-02-21T23:02:33.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.476+0000 7efe7ae00640 1 -- start start 2024-02-21T23:02:33.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.477+0000 7efe7ae00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7efe740c11f0 0x7efe74153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:33.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.477+0000 7efe7ae00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7efe740c15d0 0x7efe7406dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:33.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.477+0000 7efe7ae00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efe74156350 0x7efe74156780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:33.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.477+0000 7efe7ae00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe740c1a60 con 0x7efe740c15d0 2024-02-21T23:02:33.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.477+0000 7efe7ae00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe740a5ec0 con 0x7efe740c11f0 2024-02-21T23:02:33.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.477+0000 7efe7ae00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe74152e70 con 0x7efe74156350 2024-02-21T23:02:33.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.477+0000 7efe73400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7efe740c15d0 0x7efe7406dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:33.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.477+0000 7efe73400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7efe740c15d0 0x7efe7406dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58844/0 (socket says 172.21.15.28:58844) 2024-02-21T23:02:33.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.477+0000 7efe73400640 1 -- 172.21.15.28:0/2429683660 learned_addr learned my addr 172.21.15.28:0/2429683660 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:02:33.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.478+0000 7efe73e00640 1 --2- 172.21.15.28:0/2429683660 >> v2:172.21.15.81:3300/0 conn(0x7efe740c11f0 0x7efe74153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:33.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.478+0000 7efe78e00640 1 --2- 172.21.15.28:0/2429683660 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efe74156350 0x7efe74156780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:33.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.478+0000 7efe73400640 1 -- 172.21.15.28:0/2429683660 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efe74156350 msgr2=0x7efe74156780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:33.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.478+0000 7efe73400640 1 --2- 172.21.15.28:0/2429683660 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efe74156350 0x7efe74156780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:33.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.478+0000 7efe73400640 1 -- 172.21.15.28:0/2429683660 >> v2:172.21.15.81:3300/0 conn(0x7efe740c11f0 msgr2=0x7efe74153950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:33.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.478+0000 7efe73400640 1 --2- 172.21.15.28:0/2429683660 >> v2:172.21.15.81:3300/0 conn(0x7efe740c11f0 0x7efe74153950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:33.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.478+0000 7efe73400640 1 -- 172.21.15.28:0/2429683660 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe74153150 con 0x7efe740c15d0 2024-02-21T23:02:33.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.479+0000 7efe73400640 1 --2- 172.21.15.28:0/2429683660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe740c15d0 0x7efe7406dce0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7efe740c7590 tx=0x7efe6002b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13272a3993edf66b server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:33.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.479+0000 7efe72a00640 1 -- 172.21.15.28:0/2429683660 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe60038070 con 0x7efe740c15d0 2024-02-21T23:02:33.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.480+0000 7efe72a00640 1 -- 172.21.15.28:0/2429683660 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe6002f050 con 0x7efe740c15d0 2024-02-21T23:02:33.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.480+0000 7efe72a00640 1 -- 172.21.15.28:0/2429683660 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe60031970 con 0x7efe740c15d0 2024-02-21T23:02:33.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.480+0000 7efe7ae00640 1 -- 172.21.15.28:0/2429683660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe740c15d0 msgr2=0x7efe7406dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:33.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.480+0000 7efe7ae00640 1 --2- 172.21.15.28:0/2429683660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe740c15d0 0x7efe7406dce0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7efe740c7590 tx=0x7efe6002b7f0 comp rx=0 tx=0).stop 2024-02-21T23:02:33.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.480+0000 7efe7ae00640 1 -- 172.21.15.28:0/2429683660 shutdown_connections 2024-02-21T23:02:33.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.480+0000 7efe7ae00640 1 --2- 172.21.15.28:0/2429683660 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efe74156350 0x7efe74156780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:33.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.480+0000 7efe7ae00640 1 --2- 172.21.15.28:0/2429683660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe740c15d0 0x7efe7406dce0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:33.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.480+0000 7efe7ae00640 1 --2- 172.21.15.28:0/2429683660 >> v2:172.21.15.81:3300/0 conn(0x7efe740c11f0 0x7efe74153950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:33.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.480+0000 7efe7ae00640 1 -- 172.21.15.28:0/2429683660 >> 172.21.15.28:0/2429683660 conn(0x7efe740c67f0 msgr2=0x7efe740c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:33.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.480+0000 7efe7ae00640 1 -- 172.21.15.28:0/2429683660 shutdown_connections 2024-02-21T23:02:33.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.480+0000 7efe7ae00640 1 -- 172.21.15.28:0/2429683660 wait complete. 2024-02-21T23:02:33.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.482+0000 7efe7ae00640 1 Processor -- start 2024-02-21T23:02:33.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.482+0000 7efe7ae00640 1 -- start start 2024-02-21T23:02:33.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.483+0000 7efe7ae00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe740c11f0 0x7efe74072e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:33.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.483+0000 7efe7ae00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efe740c15d0 0x7efe741ead40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:33.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.483+0000 7efe7ae00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efe74156350 0x7efe741eeab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:33.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.483+0000 7efe7ae00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe740c4c00 con 0x7efe740c11f0 2024-02-21T23:02:33.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.483+0000 7efe7ae00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe7414d1c0 con 0x7efe740c15d0 2024-02-21T23:02:33.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.483+0000 7efe7ae00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe740a6110 con 0x7efe74156350 2024-02-21T23:02:33.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.483+0000 7efe73e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe740c11f0 0x7efe74072e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:33.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.483+0000 7efe73e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe740c11f0 0x7efe74072e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58858/0 (socket says 172.21.15.28:58858) 2024-02-21T23:02:33.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.483+0000 7efe73e00640 1 -- 172.21.15.28:0/933401671 learned_addr learned my addr 172.21.15.28:0/933401671 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:02:33.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.484+0000 7efe73e00640 1 -- 172.21.15.28:0/933401671 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efe74156350 msgr2=0x7efe741eeab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:33.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.484+0000 7efe73400640 1 --2- 172.21.15.28:0/933401671 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efe740c15d0 0x7efe741ead40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:33.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.484+0000 7efe78e00640 1 --2- 172.21.15.28:0/933401671 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efe74156350 0x7efe741eeab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:33.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.484+0000 7efe73e00640 1 --2- 172.21.15.28:0/933401671 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efe74156350 0x7efe741eeab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:33.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.484+0000 7efe73e00640 1 -- 172.21.15.28:0/933401671 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efe740c15d0 msgr2=0x7efe741ead40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:33.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.484+0000 7efe73e00640 1 --2- 172.21.15.28:0/933401671 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efe740c15d0 0x7efe741ead40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:33.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.484+0000 7efe73e00640 1 -- 172.21.15.28:0/933401671 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe74147390 con 0x7efe740c11f0 2024-02-21T23:02:33.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.484+0000 7efe73400640 1 --2- 172.21.15.28:0/933401671 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efe740c15d0 0x7efe741ead40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:02:33.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.484+0000 7efe73e00640 1 --2- 172.21.15.28:0/933401671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe740c11f0 0x7efe74072e30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7efe64007e90 tx=0x7efe6400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:33.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.484+0000 7efe6be00640 1 -- 172.21.15.28:0/933401671 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe64014420 con 0x7efe740c11f0 2024-02-21T23:02:33.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.484+0000 7efe6be00640 1 -- 172.21.15.28:0/933401671 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe64019070 con 0x7efe740c11f0 2024-02-21T23:02:33.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.484+0000 7efe6be00640 1 -- 172.21.15.28:0/933401671 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe640147e0 con 0x7efe740c11f0 2024-02-21T23:02:33.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.484+0000 7efe7ae00640 1 -- 172.21.15.28:0/933401671 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe74144f00 con 0x7efe740c11f0 2024-02-21T23:02:33.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.485+0000 7efe7ae00640 1 -- 172.21.15.28:0/933401671 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe741ee2b0 con 0x7efe740c11f0 2024-02-21T23:02:33.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.486+0000 7efe7ae00640 1 -- 172.21.15.28:0/933401671 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe74067f30 con 0x7efe740c11f0 2024-02-21T23:02:33.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.487+0000 7efe6be00640 1 -- 172.21.15.28:0/933401671 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efe64014980 con 0x7efe740c11f0 2024-02-21T23:02:33.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.488+0000 7efe6be00640 1 --2- 172.21.15.28:0/933401671 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7efe5406b4d0 0x7efe5406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:33.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.488+0000 7efe6be00640 1 -- 172.21.15.28:0/933401671 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7efe6408a640 con 0x7efe740c11f0 2024-02-21T23:02:33.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.489+0000 7efe73400640 1 --2- 172.21.15.28:0/933401671 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7efe5406b4d0 0x7efe5406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:33.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.490+0000 7efe73400640 1 --2- 172.21.15.28:0/933401671 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7efe5406b4d0 0x7efe5406d990 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7efe60035000 tx=0x7efe60008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:33.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.492+0000 7efe6be00640 1 -- 172.21.15.28:0/933401671 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7efe640598a0 con 0x7efe740c11f0 2024-02-21T23:02:33.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.740+0000 7efe7ae00640 1 -- 172.21.15.28:0/933401671 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efe740c0960 con 0x7efe740c11f0 2024-02-21T23:02:33.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.741+0000 7efe6be00640 1 -- 172.21.15.28:0/933401671 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3694 (secure 0 0 0) 0x7efe640591a0 con 0x7efe740c11f0 2024-02-21T23:02:33.730 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:02:33.730 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4230,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/2559793999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2559793999},{"type":"v1","addr":"172.21.15.28:6834","nonce":2559793999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4232,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3930208177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3930208177},{"type":"v1","addr":"172.21.15.28:6835","nonce":3930208177}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4238,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/2875984988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":2875984988},{"type":"v1","addr":"172.21.15.81:6836","nonce":2875984988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4267,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/2328151245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2328151245},{"type":"v1","addr":"172.21.15.81:6837","nonce":2328151245}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-02-21T23:02:33.730 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 3 2024-02-21T23:02:33.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.743+0000 7efe7ae00640 1 -- 172.21.15.28:0/933401671 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7efe5406b4d0 msgr2=0x7efe5406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:33.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.744+0000 7efe7ae00640 1 --2- 172.21.15.28:0/933401671 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7efe5406b4d0 0x7efe5406d990 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7efe60035000 tx=0x7efe60008030 comp rx=0 tx=0).stop 2024-02-21T23:02:33.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.744+0000 7efe7ae00640 1 -- 172.21.15.28:0/933401671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe740c11f0 msgr2=0x7efe74072e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:33.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.744+0000 7efe7ae00640 1 --2- 172.21.15.28:0/933401671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe740c11f0 0x7efe74072e30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7efe64007e90 tx=0x7efe6400e4b0 comp rx=0 tx=0).stop 2024-02-21T23:02:33.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.744+0000 7efe7ae00640 1 -- 172.21.15.28:0/933401671 shutdown_connections 2024-02-21T23:02:33.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.744+0000 7efe7ae00640 1 --2- 172.21.15.28:0/933401671 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7efe5406b4d0 0x7efe5406d990 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:33.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.744+0000 7efe7ae00640 1 --2- 172.21.15.28:0/933401671 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efe74156350 0x7efe741eeab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:33.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.744+0000 7efe7ae00640 1 --2- 172.21.15.28:0/933401671 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efe740c15d0 0x7efe741ead40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:33.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.744+0000 7efe7ae00640 1 --2- 172.21.15.28:0/933401671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe740c11f0 0x7efe74072e30 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:33.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.744+0000 7efe7ae00640 1 -- 172.21.15.28:0/933401671 >> 172.21.15.28:0/933401671 conn(0x7efe740c67f0 msgr2=0x7efe7414a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:33.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.744+0000 7efe7ae00640 1 -- 172.21.15.28:0/933401671 shutdown_connections 2024-02-21T23:02:33.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.744+0000 7efe7ae00640 1 -- 172.21.15.28:0/933401671 wait complete. 2024-02-21T23:02:33.752 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:02:33.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.979+0000 7f3fd3c00640 1 Processor -- start 2024-02-21T23:02:33.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.979+0000 7f3fd3c00640 1 -- start start 2024-02-21T23:02:33.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.980+0000 7f3fd3c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f3fcc072430 0x7f3fcc0c3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:33.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.980+0000 7f3fd3c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3fcc155860 0x7f3fcc155d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:33.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.980+0000 7f3fd3c00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fcc0c6560 con 0x7f3fcc0c3280 2024-02-21T23:02:33.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.980+0000 7f3fd3c00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fcc0c5cc0 con 0x7f3fcc072430 2024-02-21T23:02:33.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.980+0000 7f3fd3c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fcc0c5fa0 con 0x7f3fcc155860 2024-02-21T23:02:33.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.980+0000 7f3fd1200640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f3fcc0c3280 0x7f3fcc1526e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.28:41464/0 (socket says 172.21.15.28:41464) 2024-02-21T23:02:33.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.981+0000 7f3fd1200640 1 -- 172.21.15.28:0/3076050814 learned_addr learned my addr 172.21.15.28:0/3076050814 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:02:33.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.981+0000 7f3fd1c00640 1 --2- 172.21.15.28:0/3076050814 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3fcc155860 0x7f3fcc155d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:33.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.981+0000 7f3fcbe00640 1 --2- 172.21.15.28:0/3076050814 >> v2:172.21.15.81:3300/0 conn(0x7f3fcc072430 0x7f3fcc0c3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:33.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.981+0000 7f3fcbe00640 1 -- 172.21.15.28:0/3076050814 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3fcc155860 msgr2=0x7f3fcc155d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:33.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.981+0000 7f3fcbe00640 1 --2- 172.21.15.28:0/3076050814 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3fcc155860 0x7f3fcc155d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:33.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.981+0000 7f3fcbe00640 1 -- 172.21.15.28:0/3076050814 >> v1:172.21.15.81:6789/0 conn(0x7f3fcc0c3280 legacy=0x7f3fcc1526e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:33.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.981+0000 7f3fcbe00640 1 -- 172.21.15.28:0/3076050814 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fcc0c6280 con 0x7f3fcc072430 2024-02-21T23:02:33.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.983+0000 7f3fcbe00640 1 --2- 172.21.15.28:0/3076050814 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3fcc072430 0x7f3fcc0c3660 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3fb80097a0 tx=0x7f3fb802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=88fdd4b3fd39f018 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:33.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.984+0000 7f3fcb400640 1 -- 172.21.15.28:0/3076050814 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb80315a0 con 0x7f3fcc072430 2024-02-21T23:02:33.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.984+0000 7f3fcb400640 1 -- 172.21.15.28:0/3076050814 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fb8037070 con 0x7f3fcc072430 2024-02-21T23:02:33.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.984+0000 7f3fcb400640 1 -- 172.21.15.28:0/3076050814 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb8031960 con 0x7f3fcc072430 2024-02-21T23:02:33.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.984+0000 7f3fd3c00640 1 -- 172.21.15.28:0/3076050814 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3fcc072430 msgr2=0x7f3fcc0c3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:33.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.984+0000 7f3fd3c00640 1 --2- 172.21.15.28:0/3076050814 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3fcc072430 0x7f3fcc0c3660 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3fb80097a0 tx=0x7f3fb802b840 comp rx=0 tx=0).stop 2024-02-21T23:02:33.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.984+0000 7f3fd3c00640 1 -- 172.21.15.28:0/3076050814 shutdown_connections 2024-02-21T23:02:33.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.984+0000 7f3fd3c00640 1 --2- 172.21.15.28:0/3076050814 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3fcc155860 0x7f3fcc155d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:33.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.984+0000 7f3fd3c00640 1 --2- 172.21.15.28:0/3076050814 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3fcc072430 0x7f3fcc0c3660 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:33.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.984+0000 7f3fd3c00640 1 -- 172.21.15.28:0/3076050814 >> 172.21.15.28:0/3076050814 conn(0x7f3fcc0c67f0 msgr2=0x7f3fcc0c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:33.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.985+0000 7f3fd3c00640 1 -- 172.21.15.28:0/3076050814 shutdown_connections 2024-02-21T23:02:33.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.985+0000 7f3fd3c00640 1 -- 172.21.15.28:0/3076050814 wait complete. 2024-02-21T23:02:33.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.987+0000 7f3fd3c00640 1 Processor -- start 2024-02-21T23:02:33.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.987+0000 7f3fd3c00640 1 -- start start 2024-02-21T23:02:33.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.988+0000 7f3fd3c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3fcc072430 0x7f3fcc1eea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:33.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.988+0000 7f3fd3c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3fcc0c3280 0x7f3fcc146c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:33.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.988+0000 7f3fd3c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3fcc155860 0x7f3fcc140a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:33.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.988+0000 7f3fd3c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fcc0c5aa0 con 0x7f3fcc155860 2024-02-21T23:02:33.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.988+0000 7f3fd3c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fcc0c5920 con 0x7f3fcc072430 2024-02-21T23:02:33.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.988+0000 7f3fd3c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fcc0733e0 con 0x7f3fcc0c3280 2024-02-21T23:02:33.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.988+0000 7f3fd1c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3fcc155860 0x7f3fcc140a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:33.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.988+0000 7f3fd1c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3fcc155860 0x7f3fcc140a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58866/0 (socket says 172.21.15.28:58866) 2024-02-21T23:02:33.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.988+0000 7f3fd1c00640 1 -- 172.21.15.28:0/287651696 learned_addr learned my addr 172.21.15.28:0/287651696 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:02:33.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.988+0000 7f3fcbe00640 1 --2- 172.21.15.28:0/287651696 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3fcc0c3280 0x7f3fcc146c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:33.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.989+0000 7f3fd1200640 1 --2- 172.21.15.28:0/287651696 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3fcc072430 0x7f3fcc1eea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:33.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.989+0000 7f3fd1c00640 1 -- 172.21.15.28:0/287651696 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3fcc0c3280 msgr2=0x7f3fcc146c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:33.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.989+0000 7f3fd1c00640 1 --2- 172.21.15.28:0/287651696 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3fcc0c3280 0x7f3fcc146c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:33.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.989+0000 7f3fd1c00640 1 -- 172.21.15.28:0/287651696 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3fcc072430 msgr2=0x7f3fcc1eea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:33.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.989+0000 7f3fd1c00640 1 --2- 172.21.15.28:0/287651696 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3fcc072430 0x7f3fcc1eea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:33.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.989+0000 7f3fd1c00640 1 -- 172.21.15.28:0/287651696 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fcc1eef90 con 0x7f3fcc155860 2024-02-21T23:02:33.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.989+0000 7f3fd1c00640 1 --2- 172.21.15.28:0/287651696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3fcc155860 0x7f3fcc140a30 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3fc400efc0 tx=0x7f3fc400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:33.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.990+0000 7f3fc8c00640 1 -- 172.21.15.28:0/287651696 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fc4018020 con 0x7f3fcc155860 2024-02-21T23:02:33.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.990+0000 7f3fc8c00640 1 -- 172.21.15.28:0/287651696 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fc4014480 con 0x7f3fcc155860 2024-02-21T23:02:33.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.990+0000 7f3fc8c00640 1 -- 172.21.15.28:0/287651696 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fc4014870 con 0x7f3fcc155860 2024-02-21T23:02:33.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.990+0000 7f3fd3c00640 1 -- 172.21.15.28:0/287651696 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fcc152370 con 0x7f3fcc155860 2024-02-21T23:02:33.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.990+0000 7f3fd3c00640 1 -- 172.21.15.28:0/287651696 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fcc1eb040 con 0x7f3fcc155860 2024-02-21T23:02:33.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.991+0000 7f3fd3c00640 1 -- 172.21.15.28:0/287651696 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fcc1eb480 con 0x7f3fcc155860 2024-02-21T23:02:33.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.992+0000 7f3fc8c00640 1 -- 172.21.15.28:0/287651696 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3fc4005d10 con 0x7f3fcc155860 2024-02-21T23:02:33.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.994+0000 7f3fc8c00640 1 --2- 172.21.15.28:0/287651696 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3fac06b4d0 0x7f3fac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:33.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.994+0000 7f3fc8c00640 1 -- 172.21.15.28:0/287651696 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f3fc408a8d0 con 0x7f3fcc155860 2024-02-21T23:02:33.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.994+0000 7f3fd1200640 1 --2- 172.21.15.28:0/287651696 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3fac06b4d0 0x7f3fac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:33.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.995+0000 7f3fd1200640 1 --2- 172.21.15.28:0/287651696 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3fac06b4d0 0x7f3fac06d990 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3fcc147850 tx=0x7f3fbc009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:33.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:33.999+0000 7f3fc8c00640 1 -- 172.21.15.28:0/287651696 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3fc4022070 con 0x7f3fcc155860 2024-02-21T23:02:34.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.209+0000 7f3fd3c00640 1 -- 172.21.15.28:0/287651696 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3fcc1eab50 con 0x7f3fcc155860 2024-02-21T23:02:34.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.210+0000 7f3fc8c00640 1 -- 172.21.15.28:0/287651696 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12824 (secure 0 0 0) 0x7f3fc4010040 con 0x7f3fcc155860 2024-02-21T23:02:34.200 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:02:34.200 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":16,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:02:33.181561+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:02:34.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.213+0000 7f3fd3c00640 1 -- 172.21.15.28:0/287651696 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3fac06b4d0 msgr2=0x7f3fac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:34.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.213+0000 7f3fd3c00640 1 --2- 172.21.15.28:0/287651696 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3fac06b4d0 0x7f3fac06d990 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3fcc147850 tx=0x7f3fbc009040 comp rx=0 tx=0).stop 2024-02-21T23:02:34.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.213+0000 7f3fd3c00640 1 -- 172.21.15.28:0/287651696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3fcc155860 msgr2=0x7f3fcc140a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:34.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.213+0000 7f3fd3c00640 1 --2- 172.21.15.28:0/287651696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3fcc155860 0x7f3fcc140a30 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3fc400efc0 tx=0x7f3fc400c490 comp rx=0 tx=0).stop 2024-02-21T23:02:34.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.213+0000 7f3fd3c00640 1 -- 172.21.15.28:0/287651696 shutdown_connections 2024-02-21T23:02:34.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.213+0000 7f3fd3c00640 1 --2- 172.21.15.28:0/287651696 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3fac06b4d0 0x7f3fac06d990 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:34.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.213+0000 7f3fd3c00640 1 --2- 172.21.15.28:0/287651696 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3fcc0c3280 0x7f3fcc146c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:34.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.213+0000 7f3fd3c00640 1 --2- 172.21.15.28:0/287651696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3fcc155860 0x7f3fcc140a30 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:34.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.213+0000 7f3fd3c00640 1 --2- 172.21.15.28:0/287651696 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3fcc072430 0x7f3fcc1eea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:34.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.214+0000 7f3fd3c00640 1 -- 172.21.15.28:0/287651696 >> 172.21.15.28:0/287651696 conn(0x7f3fcc0c67f0 msgr2=0x7f3fcc0c10f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:34.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.214+0000 7f3fd3c00640 1 -- 172.21.15.28:0/287651696 shutdown_connections 2024-02-21T23:02:34.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.214+0000 7f3fd3c00640 1 -- 172.21.15.28:0/287651696 wait complete. 2024-02-21T23:02:34.220 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-21T23:02:30.728329+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-21T23:02:34.220 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-21T23:02:34.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.455+0000 7f6fd8c00640 1 Processor -- start 2024-02-21T23:02:34.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.455+0000 7f6fd8c00640 1 -- start start 2024-02-21T23:02:34.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.457+0000 7f6fd8c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6fd40c5f50 0x7f6fd414fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:34.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.457+0000 7f6fd8c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6fd4150530 0x7f6fd4156500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:34.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.457+0000 7f6fd8c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fd41509e0 con 0x7f6fd406b0e0 2024-02-21T23:02:34.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.457+0000 7f6fd8c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fd4153bc0 con 0x7f6fd40c5f50 2024-02-21T23:02:34.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.457+0000 7f6fd8c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd4152fb0 con 0x7f6fd4150530 2024-02-21T23:02:34.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.457+0000 7f6fcbe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6fd40c5f50 0x7f6fd414fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:34.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.457+0000 7f6fd1e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f6fd406b0e0 0x7f6fd40c5840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43202/0 (socket says 172.21.15.28:43202) 2024-02-21T23:02:34.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.457+0000 7f6fd1e00640 1 -- 172.21.15.28:0/4043518307 learned_addr learned my addr 172.21.15.28:0/4043518307 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:02:34.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.458+0000 7f6fd2800640 1 --2- 172.21.15.28:0/4043518307 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6fd4150530 0x7f6fd4156500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:34.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.458+0000 7f6fcbe00640 1 -- 172.21.15.28:0/4043518307 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6fd4150530 msgr2=0x7f6fd4156500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:34.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.458+0000 7f6fcbe00640 1 --2- 172.21.15.28:0/4043518307 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6fd4150530 0x7f6fd4156500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:34.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.458+0000 7f6fcbe00640 1 -- 172.21.15.28:0/4043518307 >> v1:172.21.15.28:6789/0 conn(0x7f6fd406b0e0 legacy=0x7f6fd40c5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:34.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.458+0000 7f6fd1400640 1 -- 172.21.15.28:0/4043518307 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2545265695 0 0) 0x7f6fd41509e0 con 0x7f6fd406b0e0 2024-02-21T23:02:34.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.458+0000 7f6fcbe00640 1 -- 172.21.15.28:0/4043518307 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fd4153290 con 0x7f6fd40c5f50 2024-02-21T23:02:34.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.459+0000 7f6fcbe00640 1 --2- 172.21.15.28:0/4043518307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6fd40c5f50 0x7f6fd414fff0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc0097a0 tx=0x7f6fbc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b31db61ffad3fef server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:34.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.460+0000 7f6fd1400640 1 -- 172.21.15.28:0/4043518307 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc0315a0 con 0x7f6fd40c5f50 2024-02-21T23:02:34.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.460+0000 7f6fd1400640 1 -- 172.21.15.28:0/4043518307 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fbc037070 con 0x7f6fd40c5f50 2024-02-21T23:02:34.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.460+0000 7f6fd1400640 1 -- 172.21.15.28:0/4043518307 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc031960 con 0x7f6fd40c5f50 2024-02-21T23:02:34.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.460+0000 7f6fd8c00640 1 -- 172.21.15.28:0/4043518307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6fd40c5f50 msgr2=0x7f6fd414fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:34.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.460+0000 7f6fd8c00640 1 --2- 172.21.15.28:0/4043518307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6fd40c5f50 0x7f6fd414fff0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc0097a0 tx=0x7f6fbc02b840 comp rx=0 tx=0).stop 2024-02-21T23:02:34.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.461+0000 7f6fd8c00640 1 -- 172.21.15.28:0/4043518307 shutdown_connections 2024-02-21T23:02:34.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.461+0000 7f6fd8c00640 1 --2- 172.21.15.28:0/4043518307 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6fd4150530 0x7f6fd4156500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:34.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.461+0000 7f6fd8c00640 1 --2- 172.21.15.28:0/4043518307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6fd40c5f50 0x7f6fd414fff0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:34.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.461+0000 7f6fd8c00640 1 -- 172.21.15.28:0/4043518307 >> 172.21.15.28:0/4043518307 conn(0x7f6fd40a85a0 msgr2=0x7f6fd40a8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:34.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.461+0000 7f6fd8c00640 1 -- 172.21.15.28:0/4043518307 shutdown_connections 2024-02-21T23:02:34.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.461+0000 7f6fd8c00640 1 -- 172.21.15.28:0/4043518307 wait complete. 2024-02-21T23:02:34.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.462+0000 7f6fd8c00640 1 Processor -- start 2024-02-21T23:02:34.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.463+0000 7f6fd8c00640 1 -- start start 2024-02-21T23:02:34.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.463+0000 7f6fd8c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6fd406b0e0 0x7f6fd40bfe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:34.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.463+0000 7f6fd8c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6fd40c5f50 0x7f6fd41499d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:34.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.463+0000 7f6fd8c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6fd4150530 0x7f6fd4149f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:34.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.463+0000 7f6fd8c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd406b4c0 con 0x7f6fd40c5f50 2024-02-21T23:02:34.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.463+0000 7f6fd8c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd4153850 con 0x7f6fd406b0e0 2024-02-21T23:02:34.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.463+0000 7f6fd8c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd406b640 con 0x7f6fd4150530 2024-02-21T23:02:34.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.464+0000 7f6fcbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6fd40c5f50 0x7f6fd41499d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:34.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.464+0000 7f6fcbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6fd40c5f50 0x7f6fd41499d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58878/0 (socket says 172.21.15.28:58878) 2024-02-21T23:02:34.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.464+0000 7f6fcbe00640 1 -- 172.21.15.28:0/1922930544 learned_addr learned my addr 172.21.15.28:0/1922930544 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:02:34.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.464+0000 7f6fd1e00640 1 --2- 172.21.15.28:0/1922930544 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6fd406b0e0 0x7f6fd40bfe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:34.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.464+0000 7f6fd2800640 1 --2- 172.21.15.28:0/1922930544 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6fd4150530 0x7f6fd4149f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:34.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.464+0000 7f6fcbe00640 1 -- 172.21.15.28:0/1922930544 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6fd4150530 msgr2=0x7f6fd4149f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:34.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.464+0000 7f6fcbe00640 1 --2- 172.21.15.28:0/1922930544 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6fd4150530 0x7f6fd4149f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:34.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.464+0000 7f6fcbe00640 1 -- 172.21.15.28:0/1922930544 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6fd406b0e0 msgr2=0x7f6fd40bfe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:34.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.464+0000 7f6fcbe00640 1 --2- 172.21.15.28:0/1922930544 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6fd406b0e0 0x7f6fd40bfe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:34.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.464+0000 7f6fcbe00640 1 -- 172.21.15.28:0/1922930544 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fd41eabf0 con 0x7f6fd40c5f50 2024-02-21T23:02:34.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.465+0000 7f6fcbe00640 1 --2- 172.21.15.28:0/1922930544 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6fd40c5f50 0x7f6fd41499d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc0098c0 tx=0x7f6fbc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:34.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.465+0000 7f6fca000640 1 -- 172.21.15.28:0/1922930544 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc032630 con 0x7f6fd40c5f50 2024-02-21T23:02:34.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.465+0000 7f6fca000640 1 -- 172.21.15.28:0/1922930544 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fbc0327d0 con 0x7f6fd40c5f50 2024-02-21T23:02:34.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.465+0000 7f6fca000640 1 -- 172.21.15.28:0/1922930544 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc032ba0 con 0x7f6fd40c5f50 2024-02-21T23:02:34.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.465+0000 7f6fd8c00640 1 -- 172.21.15.28:0/1922930544 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fd41eaed0 con 0x7f6fd40c5f50 2024-02-21T23:02:34.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.465+0000 7f6fd8c00640 1 -- 172.21.15.28:0/1922930544 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fd41eb490 con 0x7f6fd40c5f50 2024-02-21T23:02:34.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.466+0000 7f6fd8c00640 1 -- 172.21.15.28:0/1922930544 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fd40c53f0 con 0x7f6fd40c5f50 2024-02-21T23:02:34.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.467+0000 7f6fca000640 1 -- 172.21.15.28:0/1922930544 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6fbc04e430 con 0x7f6fd40c5f50 2024-02-21T23:02:34.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.468+0000 7f6fca000640 1 --2- 172.21.15.28:0/1922930544 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6fb006b500 0x7f6fb006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:34.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.468+0000 7f6fca000640 1 -- 172.21.15.28:0/1922930544 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f6fbc02f030 con 0x7f6fd40c5f50 2024-02-21T23:02:34.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.469+0000 7f6fd1e00640 1 --2- 172.21.15.28:0/1922930544 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6fb006b500 0x7f6fb006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:34.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.470+0000 7f6fd1e00640 1 --2- 172.21.15.28:0/1922930544 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6fb006b500 0x7f6fb006d9c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6fc4000f60 tx=0x7f6fc4007d50 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:34.461 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-21T23:02:34.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.473+0000 7f6fca000640 1 -- 172.21.15.28:0/1922930544 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f6fd40c53f0 con 0x7f6fd40c5f50 2024-02-21T23:02:34.465 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-21T23:02:34.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.677+0000 7f6fd8c00640 1 -- 172.21.15.28:0/1922930544 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6fd4067f30 con 0x7f6fd40c5f50 2024-02-21T23:02:34.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.678+0000 7f6fca000640 1 -- 172.21.15.28:0/1922930544 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6fbc040070 con 0x7f6fd40c5f50 2024-02-21T23:02:34.666 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-02-21T23:02:34.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.680+0000 7f6fd8c00640 1 -- 172.21.15.28:0/1922930544 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6fb006b500 msgr2=0x7f6fb006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:34.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.680+0000 7f6fd8c00640 1 --2- 172.21.15.28:0/1922930544 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6fb006b500 0x7f6fb006d9c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6fc4000f60 tx=0x7f6fc4007d50 comp rx=0 tx=0).stop 2024-02-21T23:02:34.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.680+0000 7f6fd8c00640 1 -- 172.21.15.28:0/1922930544 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6fd40c5f50 msgr2=0x7f6fd41499d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:34.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.680+0000 7f6fd8c00640 1 --2- 172.21.15.28:0/1922930544 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6fd40c5f50 0x7f6fd41499d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc0098c0 tx=0x7f6fbc0046c0 comp rx=0 tx=0).stop 2024-02-21T23:02:34.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.680+0000 7f6fd8c00640 1 -- 172.21.15.28:0/1922930544 shutdown_connections 2024-02-21T23:02:34.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.680+0000 7f6fd8c00640 1 --2- 172.21.15.28:0/1922930544 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6fb006b500 0x7f6fb006d9c0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:34.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.680+0000 7f6fd8c00640 1 --2- 172.21.15.28:0/1922930544 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6fd4150530 0x7f6fd4149f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:34.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.680+0000 7f6fd8c00640 1 --2- 172.21.15.28:0/1922930544 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6fd40c5f50 0x7f6fd41499d0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:34.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.680+0000 7f6fd8c00640 1 --2- 172.21.15.28:0/1922930544 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6fd406b0e0 0x7f6fd40bfe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:34.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.680+0000 7f6fd8c00640 1 -- 172.21.15.28:0/1922930544 >> 172.21.15.28:0/1922930544 conn(0x7f6fd40a85a0 msgr2=0x7f6fd40729d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:34.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.681+0000 7f6fd8c00640 1 -- 172.21.15.28:0/1922930544 shutdown_connections 2024-02-21T23:02:34.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.681+0000 7f6fd8c00640 1 -- 172.21.15.28:0/1922930544 wait complete. 2024-02-21T23:02:34.690 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-02-21T23:02:34.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.918+0000 7ff80d600640 1 Processor -- start 2024-02-21T23:02:34.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.919+0000 7ff80d600640 1 -- start start 2024-02-21T23:02:34.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.920+0000 7ff80d600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff8080c4fe0 0x7ff80806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:34.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.920+0000 7ff80d600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff80806ed10 con 0x7ff808159400 2024-02-21T23:02:34.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.920+0000 7ff80d600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff808159900 con 0x7ff8081569d0 2024-02-21T23:02:34.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.920+0000 7ff80d600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff808159be0 con 0x7ff8080c4fe0 2024-02-21T23:02:34.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.920+0000 7ff806000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff8080c4fe0 0x7ff80806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:34.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.920+0000 7ff807400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff808159400 0x7ff80806e600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:56874/0 (socket says 172.21.15.28:56874) 2024-02-21T23:02:34.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.920+0000 7ff807400640 1 -- 172.21.15.28:0/159944572 learned_addr learned my addr 172.21.15.28:0/159944572 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:02:34.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.921+0000 7ff805600640 1 -- 172.21.15.28:0/159944572 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1437776989 0 0) 0x7ff808159900 con 0x7ff8081569d0 2024-02-21T23:02:34.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.921+0000 7ff805600640 1 -- 172.21.15.28:0/159944572 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7e8003520 con 0x7ff8081569d0 2024-02-21T23:02:34.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.921+0000 7ff806000640 1 -- 172.21.15.28:0/159944572 >> v1:172.21.15.81:6789/0 conn(0x7ff8081569d0 legacy=0x7ff8080c3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:34.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.921+0000 7ff806000640 1 -- 172.21.15.28:0/159944572 >> v1:172.21.15.28:6789/0 conn(0x7ff808159400 legacy=0x7ff80806e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:34.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.921+0000 7ff806000640 1 -- 172.21.15.28:0/159944572 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff808159ec0 con 0x7ff8080c4fe0 2024-02-21T23:02:34.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:34.922+0000 7ff806000640 1 --2- 172.21.15.28:0/159944572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8080c4fe0 0x7ff80806e0c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff7f00097a0 tx=0x7ff7f002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee63ba12f10f4928 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:35.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.031+0000 7ff805600640 1 -- 172.21.15.28:0/159944572 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7f0035020 con 0x7ff8080c4fe0 2024-02-21T23:02:35.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.031+0000 7ff805600640 1 -- 172.21.15.28:0/159944572 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7f0031830 con 0x7ff8080c4fe0 2024-02-21T23:02:35.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.031+0000 7ff805600640 1 -- 172.21.15.28:0/159944572 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7f0031be0 con 0x7ff8080c4fe0 2024-02-21T23:02:35.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.032+0000 7ff80d600640 1 -- 172.21.15.28:0/159944572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8080c4fe0 msgr2=0x7ff80806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:35.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.032+0000 7ff80d600640 1 --2- 172.21.15.28:0/159944572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8080c4fe0 0x7ff80806e0c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff7f00097a0 tx=0x7ff7f002b8d0 comp rx=0 tx=0).stop 2024-02-21T23:02:35.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.033+0000 7ff80d600640 1 -- 172.21.15.28:0/159944572 shutdown_connections 2024-02-21T23:02:35.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.033+0000 7ff80d600640 1 --2- 172.21.15.28:0/159944572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8080c4fe0 0x7ff80806e0c0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:35.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.033+0000 7ff80d600640 1 -- 172.21.15.28:0/159944572 >> 172.21.15.28:0/159944572 conn(0x7ff8080c67f0 msgr2=0x7ff8080c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:35.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.033+0000 7ff80d600640 1 -- 172.21.15.28:0/159944572 shutdown_connections 2024-02-21T23:02:35.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.033+0000 7ff80d600640 1 -- 172.21.15.28:0/159944572 wait complete. 2024-02-21T23:02:35.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.036+0000 7ff80d600640 1 Processor -- start 2024-02-21T23:02:35.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.037+0000 7ff80d600640 1 -- start start 2024-02-21T23:02:35.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.038+0000 7ff80d600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff8080c4fe0 0x7ff8081f2720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:35.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.038+0000 7ff80d600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff8081569d0 0x7ff80814f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:35.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.038+0000 7ff80d600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff808159400 0x7ff80814ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:35.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.038+0000 7ff80d600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8080c3990 con 0x7ff808159400 2024-02-21T23:02:35.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.038+0000 7ff80d600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff808156db0 con 0x7ff8081569d0 2024-02-21T23:02:35.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.039+0000 7ff80d600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8080c3b10 con 0x7ff8080c4fe0 2024-02-21T23:02:35.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.039+0000 7ff807400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff808159400 0x7ff80814ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:35.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.039+0000 7ff807400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff808159400 0x7ff80814ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38476/0 (socket says 172.21.15.28:38476) 2024-02-21T23:02:35.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.039+0000 7ff807400640 1 -- 172.21.15.28:0/2225947345 learned_addr learned my addr 172.21.15.28:0/2225947345 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:02:35.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.039+0000 7ff806a00640 1 --2- 172.21.15.28:0/2225947345 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff8080c4fe0 0x7ff8081f2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:35.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.040+0000 7ff807400640 1 -- 172.21.15.28:0/2225947345 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff8080c4fe0 msgr2=0x7ff8081f2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:35.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.040+0000 7ff807400640 1 --2- 172.21.15.28:0/2225947345 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff8080c4fe0 0x7ff8081f2720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:35.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.040+0000 7ff807400640 1 -- 172.21.15.28:0/2225947345 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff8081569d0 msgr2=0x7ff80814f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:35.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.040+0000 7ff807400640 1 --2- 172.21.15.28:0/2225947345 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff8081569d0 0x7ff80814f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:35.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.040+0000 7ff807400640 1 -- 172.21.15.28:0/2225947345 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8081f2f60 con 0x7ff808159400 2024-02-21T23:02:35.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.040+0000 7ff807400640 1 --2- 172.21.15.28:0/2225947345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff808159400 0x7ff80814ef70 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff80000cab0 tx=0x7ff80000cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:35.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.040+0000 7ff7fea00640 1 -- 172.21.15.28:0/2225947345 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff800017070 con 0x7ff808159400 2024-02-21T23:02:35.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.041+0000 7ff7fea00640 1 -- 172.21.15.28:0/2225947345 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff80000ac30 con 0x7ff808159400 2024-02-21T23:02:35.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.041+0000 7ff7fea00640 1 -- 172.21.15.28:0/2225947345 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff800012400 con 0x7ff808159400 2024-02-21T23:02:35.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.041+0000 7ff80d600640 1 -- 172.21.15.28:0/2225947345 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8081f3240 con 0x7ff808159400 2024-02-21T23:02:35.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.041+0000 7ff80d600640 1 -- 172.21.15.28:0/2225947345 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8081512d0 con 0x7ff808159400 2024-02-21T23:02:35.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.042+0000 7ff80d600640 1 -- 172.21.15.28:0/2225947345 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff808067f30 con 0x7ff808159400 2024-02-21T23:02:35.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.047+0000 7ff7fea00640 1 -- 172.21.15.28:0/2225947345 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff80001a070 con 0x7ff808159400 2024-02-21T23:02:35.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.048+0000 7ff7fea00640 1 --2- 172.21.15.28:0/2225947345 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff7e406b550 0x7ff7e406da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:35.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.048+0000 7ff7fea00640 1 -- 172.21.15.28:0/2225947345 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ff800089f50 con 0x7ff808159400 2024-02-21T23:02:35.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.048+0000 7ff806a00640 1 --2- 172.21.15.28:0/2225947345 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff7e406b550 0x7ff7e406da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:35.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.049+0000 7ff806a00640 1 --2- 172.21.15.28:0/2225947345 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff7e406b550 0x7ff7e406da10 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff7f80011e0 tx=0x7ff7f8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:35.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.053+0000 7ff7fea00640 1 -- 172.21.15.28:0/2225947345 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ff800058bb0 con 0x7ff808159400 2024-02-21T23:02:35.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.269+0000 7ff80d600640 1 -- 172.21.15.28:0/2225947345 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7ff80806f1d0 con 0x7ff808159400 2024-02-21T23:02:35.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.270+0000 7ff7fea00640 1 -- 172.21.15.28:0/2225947345 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) v1 ==== 84+0+65 (secure 0 0 0) 0x7ff80005cc50 con 0x7ff808159400 2024-02-21T23:02:35.259 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:02:35.259 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-02-21T23:02:35.259 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:02:35.259 INFO:teuthology.orchestra.run.smithi028.stdout: "poolnum": 1, 2024-02-21T23:02:35.259 INFO:teuthology.orchestra.run.smithi028.stdout: "poolname": ".mgr" 2024-02-21T23:02:35.259 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:02:35.259 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-02-21T23:02:35.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.272+0000 7ff80d600640 1 -- 172.21.15.28:0/2225947345 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff7e406b550 msgr2=0x7ff7e406da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:35.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.272+0000 7ff80d600640 1 --2- 172.21.15.28:0/2225947345 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff7e406b550 0x7ff7e406da10 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff7f80011e0 tx=0x7ff7f8009040 comp rx=0 tx=0).stop 2024-02-21T23:02:35.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.272+0000 7ff80d600640 1 -- 172.21.15.28:0/2225947345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff808159400 msgr2=0x7ff80814ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:35.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.272+0000 7ff80d600640 1 --2- 172.21.15.28:0/2225947345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff808159400 0x7ff80814ef70 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff80000cab0 tx=0x7ff80000cf70 comp rx=0 tx=0).stop 2024-02-21T23:02:35.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.273+0000 7ff80d600640 1 -- 172.21.15.28:0/2225947345 shutdown_connections 2024-02-21T23:02:35.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.273+0000 7ff80d600640 1 --2- 172.21.15.28:0/2225947345 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff7e406b550 0x7ff7e406da10 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:35.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.273+0000 7ff80d600640 1 --2- 172.21.15.28:0/2225947345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff808159400 0x7ff80814ef70 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:35.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.273+0000 7ff80d600640 1 --2- 172.21.15.28:0/2225947345 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff8081569d0 0x7ff80814f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:35.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.273+0000 7ff80d600640 1 --2- 172.21.15.28:0/2225947345 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff8080c4fe0 0x7ff8081f2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:35.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.273+0000 7ff80d600640 1 -- 172.21.15.28:0/2225947345 >> 172.21.15.28:0/2225947345 conn(0x7ff8080c67f0 msgr2=0x7ff808154c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:35.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.273+0000 7ff80d600640 1 -- 172.21.15.28:0/2225947345 shutdown_connections 2024-02-21T23:02:35.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.273+0000 7ff80d600640 1 -- 172.21.15.28:0/2225947345 wait complete. 2024-02-21T23:02:35.282 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-02-21T23:02:35.282 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-02-21T23:02:35.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.518+0000 7f56dda00640 1 Processor -- start 2024-02-21T23:02:35.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.518+0000 7f56dda00640 1 -- start start 2024-02-21T23:02:35.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.520+0000 7f56dda00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f56d8156c10 0x7f56d80c52c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:35.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.520+0000 7f56dda00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f56d80c5a60 0x7f56d80c27e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:35.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.520+0000 7f56dda00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56d815a4c0 con 0x7f56d80c2d20 2024-02-21T23:02:35.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.520+0000 7f56dda00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56d8159f90 con 0x7f56d8156c10 2024-02-21T23:02:35.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.520+0000 7f56dda00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56d80c4820 con 0x7f56d80c5a60 2024-02-21T23:02:35.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.520+0000 7f56d6a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f56d8156c10 0x7f56d80c52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:35.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.520+0000 7f56d6a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f56d8156c10 0x7f56d80c52c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38486/0 (socket says 172.21.15.28:38486) 2024-02-21T23:02:35.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.520+0000 7f56d6a00640 1 -- 172.21.15.28:0/3020410738 learned_addr learned my addr 172.21.15.28:0/3020410738 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:02:35.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.521+0000 7f56d6000640 1 --2- 172.21.15.28:0/3020410738 >> v2:172.21.15.81:3300/0 conn(0x7f56d80c5a60 0x7f56d80c27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:35.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.521+0000 7f56d6a00640 1 -- 172.21.15.28:0/3020410738 >> v2:172.21.15.81:3300/0 conn(0x7f56d80c5a60 msgr2=0x7f56d80c27e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:35.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.521+0000 7f56d6a00640 1 --2- 172.21.15.28:0/3020410738 >> v2:172.21.15.81:3300/0 conn(0x7f56d80c5a60 0x7f56d80c27e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:35.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.521+0000 7f56d6a00640 1 -- 172.21.15.28:0/3020410738 >> v1:172.21.15.81:6789/0 conn(0x7f56d80c2d20 legacy=0x7f56d806e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:35.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.521+0000 7f56d6a00640 1 -- 172.21.15.28:0/3020410738 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56d815a1e0 con 0x7f56d8156c10 2024-02-21T23:02:35.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.522+0000 7f56d6a00640 1 --2- 172.21.15.28:0/3020410738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56d8156c10 0x7f56d80c52c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f56d80c43b0 tx=0x7f56c802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=873707a2bed2b2b6 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:35.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.522+0000 7f56d5600640 1 -- 172.21.15.28:0/3020410738 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56c8035020 con 0x7f56d8156c10 2024-02-21T23:02:35.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.522+0000 7f56d5600640 1 -- 172.21.15.28:0/3020410738 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56c8031870 con 0x7f56d8156c10 2024-02-21T23:02:35.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.522+0000 7f56d5600640 1 -- 172.21.15.28:0/3020410738 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56c8031c40 con 0x7f56d8156c10 2024-02-21T23:02:35.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.523+0000 7f56dda00640 1 -- 172.21.15.28:0/3020410738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56d8156c10 msgr2=0x7f56d80c52c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:35.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.523+0000 7f56dda00640 1 --2- 172.21.15.28:0/3020410738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56d8156c10 0x7f56d80c52c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f56d80c43b0 tx=0x7f56c802b910 comp rx=0 tx=0).stop 2024-02-21T23:02:35.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.523+0000 7f56dda00640 1 -- 172.21.15.28:0/3020410738 shutdown_connections 2024-02-21T23:02:35.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.523+0000 7f56dda00640 1 --2- 172.21.15.28:0/3020410738 >> v2:172.21.15.81:3300/0 conn(0x7f56d80c5a60 0x7f56d80c27e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:35.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.523+0000 7f56dda00640 1 --2- 172.21.15.28:0/3020410738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56d8156c10 0x7f56d80c52c0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:35.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.523+0000 7f56dda00640 1 -- 172.21.15.28:0/3020410738 >> 172.21.15.28:0/3020410738 conn(0x7f56d806b830 msgr2=0x7f56d80c67f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:35.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.523+0000 7f56dda00640 1 -- 172.21.15.28:0/3020410738 shutdown_connections 2024-02-21T23:02:35.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.523+0000 7f56dda00640 1 -- 172.21.15.28:0/3020410738 wait complete. 2024-02-21T23:02:35.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.525+0000 7f56dda00640 1 Processor -- start 2024-02-21T23:02:35.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.525+0000 7f56dda00640 1 -- start start 2024-02-21T23:02:35.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.526+0000 7f56dda00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56d80c2d20 0x7f56d8154df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:35.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.526+0000 7f56dda00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f56d80c5a60 0x7f56d8151da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:35.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.526+0000 7f56dda00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56d8156c10 0x7f56d8151130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:35.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.526+0000 7f56dda00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56d80c0d00 con 0x7f56d80c2d20 2024-02-21T23:02:35.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.526+0000 7f56dda00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56d80c4da0 con 0x7f56d8156c10 2024-02-21T23:02:35.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.526+0000 7f56dda00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56d80c0e80 con 0x7f56d80c5a60 2024-02-21T23:02:35.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.526+0000 7f56d6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56d80c2d20 0x7f56d8154df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:35.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.526+0000 7f56d6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56d80c2d20 0x7f56d8154df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38490/0 (socket says 172.21.15.28:38490) 2024-02-21T23:02:35.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.526+0000 7f56d6a00640 1 -- 172.21.15.28:0/626581772 learned_addr learned my addr 172.21.15.28:0/626581772 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:02:35.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.526+0000 7f56d7400640 1 --2- 172.21.15.28:0/626581772 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56d8156c10 0x7f56d8151130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:35.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.527+0000 7f56d6a00640 1 -- 172.21.15.28:0/626581772 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f56d80c5a60 msgr2=0x7f56d8151da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:35.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.527+0000 7f56d6a00640 1 --2- 172.21.15.28:0/626581772 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f56d80c5a60 0x7f56d8151da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:35.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.527+0000 7f56d6a00640 1 -- 172.21.15.28:0/626581772 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56d8156c10 msgr2=0x7f56d8151130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:35.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.527+0000 7f56d6a00640 1 --2- 172.21.15.28:0/626581772 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56d8156c10 0x7f56d8151130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:35.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.527+0000 7f56d6a00640 1 -- 172.21.15.28:0/626581772 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56d815b650 con 0x7f56d80c2d20 2024-02-21T23:02:35.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.527+0000 7f56d6a00640 1 --2- 172.21.15.28:0/626581772 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56d80c2d20 0x7f56d8154df0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f56c802b950 tx=0x7f56c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:35.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.527+0000 7f56cea00640 1 -- 172.21.15.28:0/626581772 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56c8040070 con 0x7f56d80c2d20 2024-02-21T23:02:35.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.527+0000 7f56cea00640 1 -- 172.21.15.28:0/626581772 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56c8030d00 con 0x7f56d80c2d20 2024-02-21T23:02:35.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.527+0000 7f56dda00640 1 -- 172.21.15.28:0/626581772 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56d81f2b50 con 0x7f56d80c2d20 2024-02-21T23:02:35.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.527+0000 7f56cea00640 1 -- 172.21.15.28:0/626581772 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56c8035040 con 0x7f56d80c2d20 2024-02-21T23:02:35.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.527+0000 7f56dda00640 1 -- 172.21.15.28:0/626581772 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56d81f3110 con 0x7f56d80c2d20 2024-02-21T23:02:35.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.529+0000 7f56dda00640 1 -- 172.21.15.28:0/626581772 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56d8067f30 con 0x7f56d80c2d20 2024-02-21T23:02:35.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.530+0000 7f56cea00640 1 -- 172.21.15.28:0/626581772 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f56c804b060 con 0x7f56d80c2d20 2024-02-21T23:02:35.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.531+0000 7f56cea00640 1 --2- 172.21.15.28:0/626581772 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f56b806b4d0 0x7f56b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:35.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.531+0000 7f56cea00640 1 -- 172.21.15.28:0/626581772 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f56c80abb90 con 0x7f56d80c2d20 2024-02-21T23:02:35.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.531+0000 7f56d6000640 1 --2- 172.21.15.28:0/626581772 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f56b806b4d0 0x7f56b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:35.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.532+0000 7f56d6000640 1 --2- 172.21.15.28:0/626581772 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f56b806b4d0 0x7f56b806d990 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f56d815b3b0 tx=0x7f56c4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:35.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.535+0000 7f56cea00640 1 -- 172.21.15.28:0/626581772 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f56c807adf0 con 0x7f56d80c2d20 2024-02-21T23:02:35.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:35.739+0000 7f56dda00640 1 -- 172.21.15.28:0/626581772 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f56d80c25c0 con 0x7f56d80c2d20 2024-02-21T23:02:36.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.313+0000 7f56cea00640 1 -- 172.21.15.28:0/626581772 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v17) v1 ==== 138+0+0 (secure 0 0 0) 0x7f56c807a6f0 con 0x7f56d80c2d20 2024-02-21T23:02:36.302 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' created 2024-02-21T23:02:36.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.317+0000 7f56dda00640 1 -- 172.21.15.28:0/626581772 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f56b806b4d0 msgr2=0x7f56b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:36.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.317+0000 7f56dda00640 1 --2- 172.21.15.28:0/626581772 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f56b806b4d0 0x7f56b806d990 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f56d815b3b0 tx=0x7f56c4008040 comp rx=0 tx=0).stop 2024-02-21T23:02:36.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.317+0000 7f56dda00640 1 -- 172.21.15.28:0/626581772 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56d80c2d20 msgr2=0x7f56d8154df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:36.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.317+0000 7f56dda00640 1 --2- 172.21.15.28:0/626581772 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56d80c2d20 0x7f56d8154df0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f56c802b950 tx=0x7f56c80046c0 comp rx=0 tx=0).stop 2024-02-21T23:02:36.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.318+0000 7f56dda00640 1 -- 172.21.15.28:0/626581772 shutdown_connections 2024-02-21T23:02:36.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.318+0000 7f56dda00640 1 --2- 172.21.15.28:0/626581772 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f56b806b4d0 0x7f56b806d990 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:36.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.318+0000 7f56dda00640 1 --2- 172.21.15.28:0/626581772 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f56d80c5a60 0x7f56d8151da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:36.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.318+0000 7f56dda00640 1 --2- 172.21.15.28:0/626581772 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56d8156c10 0x7f56d8151130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:36.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.318+0000 7f56dda00640 1 --2- 172.21.15.28:0/626581772 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56d80c2d20 0x7f56d8154df0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:36.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.318+0000 7f56dda00640 1 -- 172.21.15.28:0/626581772 >> 172.21.15.28:0/626581772 conn(0x7f56d806b830 msgr2=0x7f56d80c1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:36.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.318+0000 7f56dda00640 1 -- 172.21.15.28:0/626581772 shutdown_connections 2024-02-21T23:02:36.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.318+0000 7f56dda00640 1 -- 172.21.15.28:0/626581772 wait complete. 2024-02-21T23:02:36.336 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-02-21T23:02:36.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.564+0000 7f04e8000640 1 Processor -- start 2024-02-21T23:02:36.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.564+0000 7f04e8000640 1 -- start start 2024-02-21T23:02:36.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.566+0000 7f04e8000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f04e014f560 0x7f04e00c77a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:36.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.566+0000 7f04e8000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f04e00c6440 0x7f04e00bfbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:36.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.566+0000 7f04e8000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04e01589b0 con 0x7f04e00c7ce0 2024-02-21T23:02:36.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.566+0000 7f04e8000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04e0158760 con 0x7f04e014f560 2024-02-21T23:02:36.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.566+0000 7f04e8000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04e00c6030 con 0x7f04e00c6440 2024-02-21T23:02:36.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.566+0000 7f04e5600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f04e014f560 0x7f04e00c77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:36.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.566+0000 7f04e5600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f04e014f560 0x7f04e00c77a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38498/0 (socket says 172.21.15.28:38498) 2024-02-21T23:02:36.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.566+0000 7f04e5600640 1 -- 172.21.15.28:0/116471418 learned_addr learned my addr 172.21.15.28:0/116471418 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:02:36.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.567+0000 7f04e4c00640 1 --2- 172.21.15.28:0/116471418 >> v2:172.21.15.81:3300/0 conn(0x7f04e00c6440 0x7f04e00bfbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:36.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.567+0000 7f04e5600640 1 -- 172.21.15.28:0/116471418 >> v2:172.21.15.81:3300/0 conn(0x7f04e00c6440 msgr2=0x7f04e00bfbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:36.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.567+0000 7f04e5600640 1 --2- 172.21.15.28:0/116471418 >> v2:172.21.15.81:3300/0 conn(0x7f04e00c6440 0x7f04e00bfbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:36.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.567+0000 7f04e5600640 1 -- 172.21.15.28:0/116471418 >> v1:172.21.15.28:6789/0 conn(0x7f04e00c7ce0 legacy=0x7f04e00c0100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:36.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.567+0000 7f04e5600640 1 -- 172.21.15.28:0/116471418 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04e0158c90 con 0x7f04e014f560 2024-02-21T23:02:36.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.568+0000 7f04e5600640 1 --2- 172.21.15.28:0/116471418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04e014f560 0x7f04e00c77a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f04e00cb940 tx=0x7f04d402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0e2de57995ec49b server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:36.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.568+0000 7f04dfe00640 1 -- 172.21.15.28:0/116471418 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d4034070 con 0x7f04e014f560 2024-02-21T23:02:36.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.568+0000 7f04dfe00640 1 -- 172.21.15.28:0/116471418 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04d4030860 con 0x7f04e014f560 2024-02-21T23:02:36.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.568+0000 7f04dfe00640 1 -- 172.21.15.28:0/116471418 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d4030c30 con 0x7f04e014f560 2024-02-21T23:02:36.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.569+0000 7f04e8000640 1 -- 172.21.15.28:0/116471418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04e014f560 msgr2=0x7f04e00c77a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:36.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.569+0000 7f04e8000640 1 --2- 172.21.15.28:0/116471418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04e014f560 0x7f04e00c77a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f04e00cb940 tx=0x7f04d402b9a0 comp rx=0 tx=0).stop 2024-02-21T23:02:36.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.569+0000 7f04e8000640 1 -- 172.21.15.28:0/116471418 shutdown_connections 2024-02-21T23:02:36.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.569+0000 7f04e8000640 1 --2- 172.21.15.28:0/116471418 >> v2:172.21.15.81:3300/0 conn(0x7f04e00c6440 0x7f04e00bfbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:36.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.569+0000 7f04e8000640 1 --2- 172.21.15.28:0/116471418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04e014f560 0x7f04e00c77a0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:36.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.569+0000 7f04e8000640 1 -- 172.21.15.28:0/116471418 >> 172.21.15.28:0/116471418 conn(0x7f04e008b670 msgr2=0x7f04e006ce80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:36.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.569+0000 7f04e8000640 1 -- 172.21.15.28:0/116471418 shutdown_connections 2024-02-21T23:02:36.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.569+0000 7f04e8000640 1 -- 172.21.15.28:0/116471418 wait complete. 2024-02-21T23:02:36.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.571+0000 7f04e8000640 1 Processor -- start 2024-02-21T23:02:36.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.571+0000 7f04e8000640 1 -- start start 2024-02-21T23:02:36.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.572+0000 7f04e8000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04e00c7ce0 0x7f04e01ee8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:36.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.572+0000 7f04e8000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f04e014f560 0x7f04e0157080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:36.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.572+0000 7f04e8000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f04e00ca2e0 0x7f04e0156400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:36.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.572+0000 7f04e8000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04e0159530 con 0x7f04e00c7ce0 2024-02-21T23:02:36.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.572+0000 7f04e8000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04e0159250 con 0x7f04e00ca2e0 2024-02-21T23:02:36.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.572+0000 7f04e8000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04e00c0810 con 0x7f04e014f560 2024-02-21T23:02:36.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.572+0000 7f04e5600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04e00c7ce0 0x7f04e01ee8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:36.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.572+0000 7f04e5600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04e00c7ce0 0x7f04e01ee8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38506/0 (socket says 172.21.15.28:38506) 2024-02-21T23:02:36.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.572+0000 7f04e5600640 1 -- 172.21.15.28:0/3797500942 learned_addr learned my addr 172.21.15.28:0/3797500942 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:02:36.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.573+0000 7f04e6000640 1 --2- 172.21.15.28:0/3797500942 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f04e00ca2e0 0x7f04e0156400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:36.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.573+0000 7f04e4c00640 1 --2- 172.21.15.28:0/3797500942 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f04e014f560 0x7f04e0157080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:36.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.573+0000 7f04e5600640 1 -- 172.21.15.28:0/3797500942 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f04e014f560 msgr2=0x7f04e0157080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:36.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.573+0000 7f04e5600640 1 --2- 172.21.15.28:0/3797500942 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f04e014f560 0x7f04e0157080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:36.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.573+0000 7f04e5600640 1 -- 172.21.15.28:0/3797500942 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f04e00ca2e0 msgr2=0x7f04e0156400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:36.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.573+0000 7f04e5600640 1 --2- 172.21.15.28:0/3797500942 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f04e00ca2e0 0x7f04e0156400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:36.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.573+0000 7f04e5600640 1 -- 172.21.15.28:0/3797500942 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04e00c8b90 con 0x7f04e00c7ce0 2024-02-21T23:02:36.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.573+0000 7f04e5600640 1 --2- 172.21.15.28:0/3797500942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04e00c7ce0 0x7f04e01ee8e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f04d402b9e0 tx=0x7f04d4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:36.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.574+0000 7f04dd600640 1 -- 172.21.15.28:0/3797500942 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d4048070 con 0x7f04e00c7ce0 2024-02-21T23:02:36.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.574+0000 7f04dd600640 1 -- 172.21.15.28:0/3797500942 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04d4030660 con 0x7f04e00c7ce0 2024-02-21T23:02:36.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.574+0000 7f04dd600640 1 -- 172.21.15.28:0/3797500942 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d4034040 con 0x7f04e00c7ce0 2024-02-21T23:02:36.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.574+0000 7f04e8000640 1 -- 172.21.15.28:0/3797500942 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04e01ef360 con 0x7f04e00c7ce0 2024-02-21T23:02:36.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.574+0000 7f04e8000640 1 -- 172.21.15.28:0/3797500942 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04e0148a70 con 0x7f04e00c7ce0 2024-02-21T23:02:36.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.575+0000 7f04e8000640 1 -- 172.21.15.28:0/3797500942 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04e00c84e0 con 0x7f04e00c7ce0 2024-02-21T23:02:36.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.576+0000 7f04dd600640 1 -- 172.21.15.28:0/3797500942 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f04d404b070 con 0x7f04e00c7ce0 2024-02-21T23:02:36.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.577+0000 7f04dd600640 1 --2- 172.21.15.28:0/3797500942 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f04c006b4d0 0x7f04c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:36.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.577+0000 7f04dd600640 1 -- 172.21.15.28:0/3797500942 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f04d40ac1c0 con 0x7f04e00c7ce0 2024-02-21T23:02:36.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.577+0000 7f04e4c00640 1 --2- 172.21.15.28:0/3797500942 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f04c006b4d0 0x7f04c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:36.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.579+0000 7f04e4c00640 1 --2- 172.21.15.28:0/3797500942 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f04c006b4d0 0x7f04c006d990 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f04cc006fd0 tx=0x7f04cc008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:36.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.582+0000 7f04dd600640 1 -- 172.21.15.28:0/3797500942 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f04e00c84e0 con 0x7f04e00c7ce0 2024-02-21T23:02:36.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:36.784+0000 7f04e8000640 1 -- 172.21.15.28:0/3797500942 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) v1 -- 0x7f04e01ef530 con 0x7f04e00c7ce0 2024-02-21T23:02:37.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.321+0000 7f04dd600640 1 -- 172.21.15.28:0/3797500942 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v18) v1 ==== 171+0+0 (secure 0 0 0) 0x7f04e00c8b90 con 0x7f04e00c7ce0 2024-02-21T23:02:37.310 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' created 2024-02-21T23:02:37.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.325+0000 7f04e8000640 1 -- 172.21.15.28:0/3797500942 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f04c006b4d0 msgr2=0x7f04c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:37.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.325+0000 7f04e8000640 1 --2- 172.21.15.28:0/3797500942 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f04c006b4d0 0x7f04c006d990 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f04cc006fd0 tx=0x7f04cc008040 comp rx=0 tx=0).stop 2024-02-21T23:02:37.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.325+0000 7f04e8000640 1 -- 172.21.15.28:0/3797500942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04e00c7ce0 msgr2=0x7f04e01ee8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:37.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.325+0000 7f04e8000640 1 --2- 172.21.15.28:0/3797500942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04e00c7ce0 0x7f04e01ee8e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f04d402b9e0 tx=0x7f04d4004730 comp rx=0 tx=0).stop 2024-02-21T23:02:37.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.325+0000 7f04e8000640 1 -- 172.21.15.28:0/3797500942 shutdown_connections 2024-02-21T23:02:37.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.325+0000 7f04e8000640 1 --2- 172.21.15.28:0/3797500942 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f04c006b4d0 0x7f04c006d990 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:37.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.325+0000 7f04e8000640 1 --2- 172.21.15.28:0/3797500942 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f04e014f560 0x7f04e0157080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:37.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.325+0000 7f04e8000640 1 --2- 172.21.15.28:0/3797500942 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f04e00ca2e0 0x7f04e0156400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:37.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.325+0000 7f04e8000640 1 --2- 172.21.15.28:0/3797500942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04e00c7ce0 0x7f04e01ee8e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:37.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.325+0000 7f04e8000640 1 -- 172.21.15.28:0/3797500942 >> 172.21.15.28:0/3797500942 conn(0x7f04e00c6440 msgr2=0x7f04e014ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:37.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.325+0000 7f04e8000640 1 -- 172.21.15.28:0/3797500942 shutdown_connections 2024-02-21T23:02:37.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.325+0000 7f04e8000640 1 -- 172.21.15.28:0/3797500942 wait complete. 2024-02-21T23:02:37.352 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-02-21T23:02:37.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.551+0000 7f2d15a00640 1 Processor -- start 2024-02-21T23:02:37.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.552+0000 7f2d15a00640 1 -- start start 2024-02-21T23:02:37.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.553+0000 7f2d15a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2d100c5f50 0x7f2d1014fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:37.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.553+0000 7f2d15a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2d10150530 0x7f2d10156500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:37.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.553+0000 7f2d15a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d101509e0 con 0x7f2d1006b0e0 2024-02-21T23:02:37.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.553+0000 7f2d15a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d10153bc0 con 0x7f2d100c5f50 2024-02-21T23:02:37.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.553+0000 7f2d15a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d10152fb0 con 0x7f2d10150530 2024-02-21T23:02:37.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.553+0000 7f2d07e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2d100c5f50 0x7f2d1014fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:37.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.553+0000 7f2d07e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2d100c5f50 0x7f2d1014fff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38508/0 (socket says 172.21.15.28:38508) 2024-02-21T23:02:37.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.553+0000 7f2d07e00640 1 -- 172.21.15.28:0/2889147962 learned_addr learned my addr 172.21.15.28:0/2889147962 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:02:37.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.554+0000 7f2d0f400640 1 --2- 172.21.15.28:0/2889147962 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2d10150530 0x7f2d10156500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:37.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.554+0000 7f2d07e00640 1 -- 172.21.15.28:0/2889147962 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2d10150530 msgr2=0x7f2d10156500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:37.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.554+0000 7f2d07e00640 1 --2- 172.21.15.28:0/2889147962 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2d10150530 0x7f2d10156500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:37.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.554+0000 7f2d07e00640 1 -- 172.21.15.28:0/2889147962 >> v1:172.21.15.81:6789/0 conn(0x7f2d1006b0e0 legacy=0x7f2d100c5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:37.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.554+0000 7f2d07e00640 1 -- 172.21.15.28:0/2889147962 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d10153290 con 0x7f2d100c5f50 2024-02-21T23:02:37.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.555+0000 7f2d07e00640 1 --2- 172.21.15.28:0/2889147962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d100c5f50 0x7f2d1014fff0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2cf8009870 tx=0x7f2cf802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f06a990b0b24a6ec server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:37.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.556+0000 7f2d0e000640 1 -- 172.21.15.28:0/2889147962 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cf8035020 con 0x7f2d100c5f50 2024-02-21T23:02:37.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.556+0000 7f2d0e000640 1 -- 172.21.15.28:0/2889147962 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cf8031640 con 0x7f2d100c5f50 2024-02-21T23:02:37.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.556+0000 7f2d0e000640 1 -- 172.21.15.28:0/2889147962 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cf802d040 con 0x7f2d100c5f50 2024-02-21T23:02:37.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.556+0000 7f2d15a00640 1 -- 172.21.15.28:0/2889147962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d100c5f50 msgr2=0x7f2d1014fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:37.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.556+0000 7f2d15a00640 1 --2- 172.21.15.28:0/2889147962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d100c5f50 0x7f2d1014fff0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2cf8009870 tx=0x7f2cf802b8f0 comp rx=0 tx=0).stop 2024-02-21T23:02:37.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.556+0000 7f2d15a00640 1 -- 172.21.15.28:0/2889147962 shutdown_connections 2024-02-21T23:02:37.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.556+0000 7f2d15a00640 1 --2- 172.21.15.28:0/2889147962 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2d10150530 0x7f2d10156500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:37.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.556+0000 7f2d15a00640 1 --2- 172.21.15.28:0/2889147962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d100c5f50 0x7f2d1014fff0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:37.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.556+0000 7f2d15a00640 1 -- 172.21.15.28:0/2889147962 >> 172.21.15.28:0/2889147962 conn(0x7f2d100a85a0 msgr2=0x7f2d100a8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:37.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.556+0000 7f2d15a00640 1 -- 172.21.15.28:0/2889147962 shutdown_connections 2024-02-21T23:02:37.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.557+0000 7f2d15a00640 1 -- 172.21.15.28:0/2889147962 wait complete. 2024-02-21T23:02:37.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.558+0000 7f2d15a00640 1 Processor -- start 2024-02-21T23:02:37.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.558+0000 7f2d15a00640 1 -- start start 2024-02-21T23:02:37.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.559+0000 7f2d15a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2d1006b0e0 0x7f2d10152470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:37.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.559+0000 7f2d15a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2d100c5f50 0x7f2d100c1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:37.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.559+0000 7f2d15a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d10150530 0x7f2d100c1860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:37.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.559+0000 7f2d15a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d1006b4c0 con 0x7f2d10150530 2024-02-21T23:02:37.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.559+0000 7f2d15a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d10153850 con 0x7f2d1006b0e0 2024-02-21T23:02:37.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.559+0000 7f2d15a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d1006b640 con 0x7f2d100c5f50 2024-02-21T23:02:37.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.560+0000 7f2d0f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d10150530 0x7f2d100c1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:37.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.560+0000 7f2d0f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d10150530 0x7f2d100c1860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38512/0 (socket says 172.21.15.28:38512) 2024-02-21T23:02:37.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.560+0000 7f2d0f400640 1 -- 172.21.15.28:0/825653933 learned_addr learned my addr 172.21.15.28:0/825653933 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:02:37.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.560+0000 7f2d07e00640 1 --2- 172.21.15.28:0/825653933 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2d100c5f50 0x7f2d100c1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:37.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.560+0000 7f2d0ea00640 1 --2- 172.21.15.28:0/825653933 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2d1006b0e0 0x7f2d10152470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:37.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.560+0000 7f2d0f400640 1 -- 172.21.15.28:0/825653933 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2d100c5f50 msgr2=0x7f2d100c1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:37.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.560+0000 7f2d0f400640 1 --2- 172.21.15.28:0/825653933 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2d100c5f50 0x7f2d100c1320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:37.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.560+0000 7f2d0f400640 1 -- 172.21.15.28:0/825653933 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2d1006b0e0 msgr2=0x7f2d10152470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:37.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.560+0000 7f2d0f400640 1 --2- 172.21.15.28:0/825653933 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2d1006b0e0 0x7f2d10152470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:37.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.560+0000 7f2d0f400640 1 -- 172.21.15.28:0/825653933 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d100a5eb0 con 0x7f2d10150530 2024-02-21T23:02:37.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.560+0000 7f2d0f400640 1 --2- 172.21.15.28:0/825653933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d10150530 0x7f2d100c1860 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2d08007e90 tx=0x7f2d0800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:37.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.561+0000 7f2d06a00640 1 -- 172.21.15.28:0/825653933 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d08014420 con 0x7f2d10150530 2024-02-21T23:02:37.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.561+0000 7f2d06a00640 1 -- 172.21.15.28:0/825653933 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d08019070 con 0x7f2d10150530 2024-02-21T23:02:37.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.561+0000 7f2d06a00640 1 -- 172.21.15.28:0/825653933 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d08014800 con 0x7f2d10150530 2024-02-21T23:02:37.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.561+0000 7f2d15a00640 1 -- 172.21.15.28:0/825653933 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d100a6190 con 0x7f2d10150530 2024-02-21T23:02:37.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.561+0000 7f2d15a00640 1 -- 172.21.15.28:0/825653933 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d1014db60 con 0x7f2d10150530 2024-02-21T23:02:37.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.562+0000 7f2d15a00640 1 -- 172.21.15.28:0/825653933 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d10067f30 con 0x7f2d10150530 2024-02-21T23:02:37.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.563+0000 7f2d06a00640 1 -- 172.21.15.28:0/825653933 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2d080149a0 con 0x7f2d10150530 2024-02-21T23:02:37.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.564+0000 7f2d06a00640 1 --2- 172.21.15.28:0/825653933 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2cec06b4d0 0x7f2cec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:37.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.564+0000 7f2d06a00640 1 -- 172.21.15.28:0/825653933 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f2d0808aa00 con 0x7f2d10150530 2024-02-21T23:02:37.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.565+0000 7f2d0ea00640 1 --2- 172.21.15.28:0/825653933 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2cec06b4d0 0x7f2cec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:37.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.566+0000 7f2d0ea00640 1 --2- 172.21.15.28:0/825653933 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2cec06b4d0 0x7f2cec06d990 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2d00000f60 tx=0x7f2d00009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:37.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.569+0000 7f2d06a00640 1 -- 172.21.15.28:0/825653933 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f2d080598d0 con 0x7f2d10150530 2024-02-21T23:02:37.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:37.810+0000 7f2d15a00640 1 -- 172.21.15.28:0/825653933 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f2d1014c6a0 con 0x7f2d10150530 2024-02-21T23:02:38.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.366+0000 7f2d06a00640 1 -- 172.21.15.28:0/825653933 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f2d080591d0 con 0x7f2d10150530 2024-02-21T23:02:38.358 INFO:teuthology.orchestra.run.smithi028.stderr:new fs with metadata pool 2 and data pool 3 2024-02-21T23:02:38.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.374+0000 7f2d15a00640 1 -- 172.21.15.28:0/825653933 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2cec06b4d0 msgr2=0x7f2cec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:38.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.374+0000 7f2d15a00640 1 --2- 172.21.15.28:0/825653933 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2cec06b4d0 0x7f2cec06d990 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2d00000f60 tx=0x7f2d00009040 comp rx=0 tx=0).stop 2024-02-21T23:02:38.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.374+0000 7f2d15a00640 1 -- 172.21.15.28:0/825653933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d10150530 msgr2=0x7f2d100c1860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:38.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.374+0000 7f2d15a00640 1 --2- 172.21.15.28:0/825653933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d10150530 0x7f2d100c1860 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2d08007e90 tx=0x7f2d0800e4b0 comp rx=0 tx=0).stop 2024-02-21T23:02:38.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.375+0000 7f2d15a00640 1 -- 172.21.15.28:0/825653933 shutdown_connections 2024-02-21T23:02:38.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.375+0000 7f2d15a00640 1 --2- 172.21.15.28:0/825653933 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2cec06b4d0 0x7f2cec06d990 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:38.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.375+0000 7f2d15a00640 1 --2- 172.21.15.28:0/825653933 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2d100c5f50 0x7f2d100c1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:38.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.375+0000 7f2d15a00640 1 --2- 172.21.15.28:0/825653933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d10150530 0x7f2d100c1860 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:38.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.375+0000 7f2d15a00640 1 --2- 172.21.15.28:0/825653933 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2d1006b0e0 0x7f2d10152470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:38.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.376+0000 7f2d15a00640 1 -- 172.21.15.28:0/825653933 >> 172.21.15.28:0/825653933 conn(0x7f2d100a85a0 msgr2=0x7f2d100729d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:38.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.379+0000 7f2d15a00640 1 -- 172.21.15.28:0/825653933 shutdown_connections 2024-02-21T23:02:38.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.380+0000 7f2d15a00640 1 -- 172.21.15.28:0/825653933 wait complete. 2024-02-21T23:02:38.403 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-02-21T23:02:38.403 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-02-21T23:02:38.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.601+0000 7fc0dac00640 1 Processor -- start 2024-02-21T23:02:38.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.602+0000 7fc0dac00640 1 -- start start 2024-02-21T23:02:38.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.603+0000 7fc0dac00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc0d406e780 0x7fc0d406eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:38.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.603+0000 7fc0dac00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc0d40c5290 0x7fc0d415d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:38.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.603+0000 7fc0dac00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0d4153cb0 con 0x7fc0d40a5fc0 2024-02-21T23:02:38.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.603+0000 7fc0dac00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0d41543d0 con 0x7fc0d406e780 2024-02-21T23:02:38.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.603+0000 7fc0dac00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d40c5ad0 con 0x7fc0d40c5290 2024-02-21T23:02:38.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.603+0000 7fc0d3e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc0d406e780 0x7fc0d406eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:38.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.603+0000 7fc0d3e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc0d406e780 0x7fc0d406eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38514/0 (socket says 172.21.15.28:38514) 2024-02-21T23:02:38.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.603+0000 7fc0d3e00640 1 -- 172.21.15.28:0/674936440 learned_addr learned my addr 172.21.15.28:0/674936440 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:02:38.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.604+0000 7fc0d8c00640 1 --2- 172.21.15.28:0/674936440 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc0d40c5290 0x7fc0d415d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:38.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.604+0000 7fc0d3e00640 1 -- 172.21.15.28:0/674936440 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc0d40c5290 msgr2=0x7fc0d415d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:38.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.604+0000 7fc0d3e00640 1 --2- 172.21.15.28:0/674936440 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc0d40c5290 0x7fc0d415d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:38.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.604+0000 7fc0d3e00640 1 -- 172.21.15.28:0/674936440 >> v1:172.21.15.81:6789/0 conn(0x7fc0d40a5fc0 legacy=0x7fc0d40c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:38.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.604+0000 7fc0d3e00640 1 -- 172.21.15.28:0/674936440 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0d40c5d20 con 0x7fc0d406e780 2024-02-21T23:02:38.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.604+0000 7fc0d8c00640 1 --2- 172.21.15.28:0/674936440 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc0d40c5290 0x7fc0d415d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:02:38.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.605+0000 7fc0d3e00640 1 --2- 172.21.15.28:0/674936440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0d406e780 0x7fc0d406eb60 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc0c4009870 tx=0x7fc0c402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=696b5f3f6ac8bb3c server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:38.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.605+0000 7fc0d2a00640 1 -- 172.21.15.28:0/674936440 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0c40317c0 con 0x7fc0d406e780 2024-02-21T23:02:38.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.605+0000 7fc0d2a00640 1 -- 172.21.15.28:0/674936440 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0c4037070 con 0x7fc0d406e780 2024-02-21T23:02:38.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.605+0000 7fc0d2a00640 1 -- 172.21.15.28:0/674936440 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0c4031b80 con 0x7fc0d406e780 2024-02-21T23:02:38.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.606+0000 7fc0dac00640 1 -- 172.21.15.28:0/674936440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0d406e780 msgr2=0x7fc0d406eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:38.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.606+0000 7fc0dac00640 1 --2- 172.21.15.28:0/674936440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0d406e780 0x7fc0d406eb60 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc0c4009870 tx=0x7fc0c402b860 comp rx=0 tx=0).stop 2024-02-21T23:02:38.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.606+0000 7fc0dac00640 1 -- 172.21.15.28:0/674936440 shutdown_connections 2024-02-21T23:02:38.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.606+0000 7fc0dac00640 1 --2- 172.21.15.28:0/674936440 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc0d40c5290 0x7fc0d415d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:38.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.606+0000 7fc0dac00640 1 --2- 172.21.15.28:0/674936440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0d406e780 0x7fc0d406eb60 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:38.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.606+0000 7fc0dac00640 1 -- 172.21.15.28:0/674936440 >> 172.21.15.28:0/674936440 conn(0x7fc0d406b830 msgr2=0x7fc0d40c67f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:38.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.606+0000 7fc0dac00640 1 -- 172.21.15.28:0/674936440 shutdown_connections 2024-02-21T23:02:38.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.606+0000 7fc0dac00640 1 -- 172.21.15.28:0/674936440 wait complete. 2024-02-21T23:02:38.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.607+0000 7fc0dac00640 1 Processor -- start 2024-02-21T23:02:38.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.608+0000 7fc0dac00640 1 -- start start 2024-02-21T23:02:38.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.608+0000 7fc0dac00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0d406e780 0x7fc0d41e9f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:38.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.609+0000 7fc0dac00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0d40a5fc0 0x7fc0d4149580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:38.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.609+0000 7fc0dac00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc0d40c5290 0x7fc0d414b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:38.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.609+0000 7fc0dac00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d4153960 con 0x7fc0d40a5fc0 2024-02-21T23:02:38.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.609+0000 7fc0dac00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d40c46f0 con 0x7fc0d406e780 2024-02-21T23:02:38.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.609+0000 7fc0dac00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d406d110 con 0x7fc0d40c5290 2024-02-21T23:02:38.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.609+0000 7fc0d3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0d40a5fc0 0x7fc0d4149580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:38.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.609+0000 7fc0d3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0d40a5fc0 0x7fc0d4149580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38526/0 (socket says 172.21.15.28:38526) 2024-02-21T23:02:38.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.609+0000 7fc0d3400640 1 -- 172.21.15.28:0/3546173027 learned_addr learned my addr 172.21.15.28:0/3546173027 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:02:38.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.609+0000 7fc0d3e00640 1 --2- 172.21.15.28:0/3546173027 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0d406e780 0x7fc0d41e9f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:38.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.609+0000 7fc0d8c00640 1 --2- 172.21.15.28:0/3546173027 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc0d40c5290 0x7fc0d414b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:38.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.609+0000 7fc0d3400640 1 -- 172.21.15.28:0/3546173027 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc0d40c5290 msgr2=0x7fc0d414b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:38.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.609+0000 7fc0d3400640 1 --2- 172.21.15.28:0/3546173027 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc0d40c5290 0x7fc0d414b6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:38.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.609+0000 7fc0d3400640 1 -- 172.21.15.28:0/3546173027 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0d406e780 msgr2=0x7fc0d41e9f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:38.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.609+0000 7fc0d3400640 1 --2- 172.21.15.28:0/3546173027 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0d406e780 0x7fc0d41e9f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:38.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.609+0000 7fc0d3400640 1 -- 172.21.15.28:0/3546173027 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0d41511d0 con 0x7fc0d40a5fc0 2024-02-21T23:02:38.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.609+0000 7fc0d3e00640 1 --2- 172.21.15.28:0/3546173027 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0d406e780 0x7fc0d41e9f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:02:38.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.610+0000 7fc0d3400640 1 --2- 172.21.15.28:0/3546173027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0d40a5fc0 0x7fc0d4149580 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc0bc00ba50 tx=0x7fc0bc00bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:38.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.610+0000 7fc0cbe00640 1 -- 172.21.15.28:0/3546173027 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0bc019070 con 0x7fc0d40a5fc0 2024-02-21T23:02:38.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.610+0000 7fc0cbe00640 1 -- 172.21.15.28:0/3546173027 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0bc00cc20 con 0x7fc0d40a5fc0 2024-02-21T23:02:38.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.610+0000 7fc0cbe00640 1 -- 172.21.15.28:0/3546173027 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0bc014680 con 0x7fc0d40a5fc0 2024-02-21T23:02:38.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.610+0000 7fc0dac00640 1 -- 172.21.15.28:0/3546173027 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0d41ea550 con 0x7fc0d40a5fc0 2024-02-21T23:02:38.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.610+0000 7fc0dac00640 1 -- 172.21.15.28:0/3546173027 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0d41eadf0 con 0x7fc0d40a5fc0 2024-02-21T23:02:38.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.611+0000 7fc0dac00640 1 -- 172.21.15.28:0/3546173027 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0d4067f30 con 0x7fc0d40a5fc0 2024-02-21T23:02:38.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.612+0000 7fc0cbe00640 1 -- 172.21.15.28:0/3546173027 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc0bc014820 con 0x7fc0d40a5fc0 2024-02-21T23:02:38.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.613+0000 7fc0cbe00640 1 --2- 172.21.15.28:0/3546173027 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc0b006b4d0 0x7fc0b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:38.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.613+0000 7fc0cbe00640 1 -- 172.21.15.28:0/3546173027 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc0bc08a1e0 con 0x7fc0d40a5fc0 2024-02-21T23:02:38.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.614+0000 7fc0d3e00640 1 --2- 172.21.15.28:0/3546173027 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc0b006b4d0 0x7fc0b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:38.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.615+0000 7fc0d3e00640 1 --2- 172.21.15.28:0/3546173027 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc0b006b4d0 0x7fc0b006d990 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc0c4036000 tx=0x7fc0c4008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:38.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.618+0000 7fc0cbe00640 1 -- 172.21.15.28:0/3546173027 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc0bc0588e0 con 0x7fc0d40a5fc0 2024-02-21T23:02:38.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:38.843+0000 7fc0dac00640 1 -- 172.21.15.28:0/3546173027 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7fc0d40c58b0 con 0x7fc0d40a5fc0 2024-02-21T23:02:39.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.376+0000 7fc0cbe00640 1 -- 172.21.15.28:0/3546173027 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v20) v1 ==== 157+0+0 (secure 0 0 0) 0x7fc0bc05c980 con 0x7fc0d40a5fc0 2024-02-21T23:02:39.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.381+0000 7fc0dac00640 1 -- 172.21.15.28:0/3546173027 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc0b006b4d0 msgr2=0x7fc0b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:39.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.381+0000 7fc0dac00640 1 --2- 172.21.15.28:0/3546173027 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc0b006b4d0 0x7fc0b006d990 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc0c4036000 tx=0x7fc0c4008030 comp rx=0 tx=0).stop 2024-02-21T23:02:39.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.381+0000 7fc0dac00640 1 -- 172.21.15.28:0/3546173027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0d40a5fc0 msgr2=0x7fc0d4149580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:39.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.381+0000 7fc0dac00640 1 --2- 172.21.15.28:0/3546173027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0d40a5fc0 0x7fc0d4149580 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc0bc00ba50 tx=0x7fc0bc00bf10 comp rx=0 tx=0).stop 2024-02-21T23:02:39.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.381+0000 7fc0dac00640 1 -- 172.21.15.28:0/3546173027 shutdown_connections 2024-02-21T23:02:39.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.381+0000 7fc0dac00640 1 --2- 172.21.15.28:0/3546173027 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc0b006b4d0 0x7fc0b006d990 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:39.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.381+0000 7fc0dac00640 1 --2- 172.21.15.28:0/3546173027 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc0d40c5290 0x7fc0d414b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:39.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.381+0000 7fc0dac00640 1 --2- 172.21.15.28:0/3546173027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0d40a5fc0 0x7fc0d4149580 secure :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc0bc00ba50 tx=0x7fc0bc00bf10 comp rx=0 tx=0).stop 2024-02-21T23:02:39.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.381+0000 7fc0dac00640 1 --2- 172.21.15.28:0/3546173027 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0d406e780 0x7fc0d41e9f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:39.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.381+0000 7fc0dac00640 1 -- 172.21.15.28:0/3546173027 >> 172.21.15.28:0/3546173027 conn(0x7fc0d406b830 msgr2=0x7fc0d40c4b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:39.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.381+0000 7fc0dac00640 1 -- 172.21.15.28:0/3546173027 shutdown_connections 2024-02-21T23:02:39.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.381+0000 7fc0dac00640 1 -- 172.21.15.28:0/3546173027 wait complete. 2024-02-21T23:02:39.396 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-02-21T23:02:39.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.625+0000 7f37ce600640 1 Processor -- start 2024-02-21T23:02:39.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.625+0000 7f37ce600640 1 -- start start 2024-02-21T23:02:39.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.627+0000 7f37ce600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f37c80c6550 0x7f37c81558d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:39.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.627+0000 7f37ce600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f37c80c5460 0x7f37c80c58f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:39.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.627+0000 7f37ce600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37c80c1da0 con 0x7f37c8152720 2024-02-21T23:02:39.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.627+0000 7f37ce600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37c8152b00 con 0x7f37c80c5460 2024-02-21T23:02:39.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.627+0000 7f37ce600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37c80c83a0 con 0x7f37c80c6550 2024-02-21T23:02:39.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.627+0000 7f37cd200640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f37c8152720 0x7f37c80c8580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:56892/0 (socket says 172.21.15.28:56892) 2024-02-21T23:02:39.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.627+0000 7f37cd200640 1 -- 172.21.15.28:0/2432160838 learned_addr learned my addr 172.21.15.28:0/2432160838 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:02:39.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.627+0000 7f37cdc00640 1 --2- 172.21.15.28:0/2432160838 >> v2:172.21.15.28:3300/0 conn(0x7f37c80c5460 0x7f37c80c58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:39.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.627+0000 7f37c7e00640 1 --2- 172.21.15.28:0/2432160838 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f37c80c6550 0x7f37c81558d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:39.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.627+0000 7f37cdc00640 1 -- 172.21.15.28:0/2432160838 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f37c80c6550 msgr2=0x7f37c81558d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:39.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.627+0000 7f37cdc00640 1 --2- 172.21.15.28:0/2432160838 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f37c80c6550 0x7f37c81558d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:39.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.627+0000 7f37cdc00640 1 -- 172.21.15.28:0/2432160838 >> v1:172.21.15.28:6789/0 conn(0x7f37c8152720 legacy=0x7f37c80c8580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:39.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.628+0000 7f37cdc00640 1 -- 172.21.15.28:0/2432160838 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37c80c5e30 con 0x7f37c80c5460 2024-02-21T23:02:39.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.627+0000 7f37c7400640 1 -- 172.21.15.28:0/2432160838 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1158211891 0 0) 0x7f37c80c1da0 con 0x7f37c8152720 2024-02-21T23:02:39.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.629+0000 7f37cdc00640 1 --2- 172.21.15.28:0/2432160838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37c80c5460 0x7f37c80c58f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f37c0009d90 tx=0x7f37c002be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e3ce09da48a8ad0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:39.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.629+0000 7f37c7400640 1 -- 172.21.15.28:0/2432160838 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37c0039070 con 0x7f37c80c5460 2024-02-21T23:02:39.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.629+0000 7f37c7400640 1 -- 172.21.15.28:0/2432160838 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37c002cbc0 con 0x7f37c80c5460 2024-02-21T23:02:39.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.629+0000 7f37c7400640 1 -- 172.21.15.28:0/2432160838 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37c0034440 con 0x7f37c80c5460 2024-02-21T23:02:39.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.630+0000 7f37ce600640 1 -- 172.21.15.28:0/2432160838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37c80c5460 msgr2=0x7f37c80c58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:39.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.630+0000 7f37ce600640 1 --2- 172.21.15.28:0/2432160838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37c80c5460 0x7f37c80c58f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f37c0009d90 tx=0x7f37c002be10 comp rx=0 tx=0).stop 2024-02-21T23:02:39.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.630+0000 7f37ce600640 1 -- 172.21.15.28:0/2432160838 shutdown_connections 2024-02-21T23:02:39.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.630+0000 7f37ce600640 1 --2- 172.21.15.28:0/2432160838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37c80c5460 0x7f37c80c58f0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:39.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.630+0000 7f37ce600640 1 --2- 172.21.15.28:0/2432160838 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f37c80c6550 0x7f37c81558d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:39.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.630+0000 7f37ce600640 1 -- 172.21.15.28:0/2432160838 >> 172.21.15.28:0/2432160838 conn(0x7f37c806ae70 msgr2=0x7f37c806b280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:39.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.630+0000 7f37ce600640 1 -- 172.21.15.28:0/2432160838 shutdown_connections 2024-02-21T23:02:39.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.630+0000 7f37ce600640 1 -- 172.21.15.28:0/2432160838 wait complete. 2024-02-21T23:02:39.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.632+0000 7f37ce600640 1 Processor -- start 2024-02-21T23:02:39.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.632+0000 7f37ce600640 1 -- start start 2024-02-21T23:02:39.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.633+0000 7f37ce600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f37c80c5460 0x7f37c81de840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:39.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.633+0000 7f37ce600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f37c80c6550 0x7f37c81500d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:39.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.633+0000 7f37ce600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37c8152720 0x7f37c81531c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:39.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.633+0000 7f37ce600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37c80c4f40 con 0x7f37c8152720 2024-02-21T23:02:39.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.633+0000 7f37ce600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37c80c2320 con 0x7f37c80c5460 2024-02-21T23:02:39.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.633+0000 7f37ce600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37c8070190 con 0x7f37c80c6550 2024-02-21T23:02:39.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.633+0000 7f37cdc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37c8152720 0x7f37c81531c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:39.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.633+0000 7f37cdc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37c8152720 0x7f37c81531c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38546/0 (socket says 172.21.15.28:38546) 2024-02-21T23:02:39.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.633+0000 7f37cdc00640 1 -- 172.21.15.28:0/2873107121 learned_addr learned my addr 172.21.15.28:0/2873107121 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:02:39.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.633+0000 7f37cd200640 1 --2- 172.21.15.28:0/2873107121 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f37c80c5460 0x7f37c81de840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:39.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.633+0000 7f37c7e00640 1 --2- 172.21.15.28:0/2873107121 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f37c80c6550 0x7f37c81500d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:39.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.633+0000 7f37cdc00640 1 -- 172.21.15.28:0/2873107121 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f37c80c6550 msgr2=0x7f37c81500d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:39.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.633+0000 7f37cdc00640 1 --2- 172.21.15.28:0/2873107121 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f37c80c6550 0x7f37c81500d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:39.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.634+0000 7f37cdc00640 1 -- 172.21.15.28:0/2873107121 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f37c80c5460 msgr2=0x7f37c81de840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:39.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.634+0000 7f37cdc00640 1 --2- 172.21.15.28:0/2873107121 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f37c80c5460 0x7f37c81de840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:39.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.634+0000 7f37cdc00640 1 -- 172.21.15.28:0/2873107121 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37c8152cc0 con 0x7f37c8152720 2024-02-21T23:02:39.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.634+0000 7f37cd200640 1 --2- 172.21.15.28:0/2873107121 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f37c80c5460 0x7f37c81de840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:02:39.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.634+0000 7f37cdc00640 1 --2- 172.21.15.28:0/2873107121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37c8152720 0x7f37c81531c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f37c0009d90 tx=0x7f37c0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:39.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.634+0000 7f37c4c00640 1 -- 172.21.15.28:0/2873107121 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37c0039050 con 0x7f37c8152720 2024-02-21T23:02:39.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.634+0000 7f37c4c00640 1 -- 172.21.15.28:0/2873107121 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37c0045070 con 0x7f37c8152720 2024-02-21T23:02:39.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.634+0000 7f37c4c00640 1 -- 172.21.15.28:0/2873107121 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37c003e830 con 0x7f37c8152720 2024-02-21T23:02:39.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.634+0000 7f37ce600640 1 -- 172.21.15.28:0/2873107121 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37c8154140 con 0x7f37c8152720 2024-02-21T23:02:39.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.634+0000 7f37ce600640 1 -- 172.21.15.28:0/2873107121 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37c8154700 con 0x7f37c8152720 2024-02-21T23:02:39.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.636+0000 7f37ce600640 1 -- 172.21.15.28:0/2873107121 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37c8155d50 con 0x7f37c8152720 2024-02-21T23:02:39.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.636+0000 7f37c4c00640 1 -- 172.21.15.28:0/2873107121 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f37c003e9d0 con 0x7f37c8152720 2024-02-21T23:02:39.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.637+0000 7f37c4c00640 1 --2- 172.21.15.28:0/2873107121 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f37a806b4d0 0x7f37a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:39.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.637+0000 7f37c4c00640 1 -- 172.21.15.28:0/2873107121 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6007+0+0 (secure 0 0 0) 0x7f37c00ad060 con 0x7f37c8152720 2024-02-21T23:02:39.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.638+0000 7f37cd200640 1 --2- 172.21.15.28:0/2873107121 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f37a806b4d0 0x7f37a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:39.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.639+0000 7f37cd200640 1 --2- 172.21.15.28:0/2873107121 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f37a806b4d0 0x7f37a806d990 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f37b8000f60 tx=0x7f37b8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:39.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.642+0000 7f37c4c00640 1 -- 172.21.15.28:0/2873107121 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f37c8155d50 con 0x7f37c8152720 2024-02-21T23:02:39.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:39.841+0000 7f37ce600640 1 -- 172.21.15.28:0/2873107121 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f37c804f290 con 0x7f37c8152720 2024-02-21T23:02:41.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.391+0000 7f37c4c00640 1 -- 172.21.15.28:0/2873107121 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v22) v1 ==== 228+0+0 (secure 0 0 0) 0x7f37c0032e20 con 0x7f37c8152720 2024-02-21T23:02:41.382 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data_ec' created 2024-02-21T23:02:41.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.396+0000 7f37ce600640 1 -- 172.21.15.28:0/2873107121 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f37a806b4d0 msgr2=0x7f37a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:41.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.396+0000 7f37ce600640 1 --2- 172.21.15.28:0/2873107121 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f37a806b4d0 0x7f37a806d990 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f37b8000f60 tx=0x7f37b8009040 comp rx=0 tx=0).stop 2024-02-21T23:02:41.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.396+0000 7f37ce600640 1 -- 172.21.15.28:0/2873107121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37c8152720 msgr2=0x7f37c81531c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:41.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.396+0000 7f37ce600640 1 --2- 172.21.15.28:0/2873107121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37c8152720 0x7f37c81531c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f37c0009d90 tx=0x7f37c0004a40 comp rx=0 tx=0).stop 2024-02-21T23:02:41.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.397+0000 7f37ce600640 1 -- 172.21.15.28:0/2873107121 shutdown_connections 2024-02-21T23:02:41.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.397+0000 7f37ce600640 1 --2- 172.21.15.28:0/2873107121 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f37a806b4d0 0x7f37a806d990 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:41.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.397+0000 7f37ce600640 1 --2- 172.21.15.28:0/2873107121 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f37c80c6550 0x7f37c81500d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:41.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.397+0000 7f37ce600640 1 --2- 172.21.15.28:0/2873107121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37c8152720 0x7f37c81531c0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:41.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.397+0000 7f37ce600640 1 --2- 172.21.15.28:0/2873107121 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f37c80c5460 0x7f37c81de840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:41.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.397+0000 7f37ce600640 1 -- 172.21.15.28:0/2873107121 >> 172.21.15.28:0/2873107121 conn(0x7f37c806ae70 msgr2=0x7f37c80c8cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:41.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.399+0000 7f37ce600640 1 -- 172.21.15.28:0/2873107121 shutdown_connections 2024-02-21T23:02:41.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.399+0000 7f37ce600640 1 -- 172.21.15.28:0/2873107121 wait complete. 2024-02-21T23:02:41.417 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-02-21T23:02:41.531 INFO:tasks.ceph.mds.b.smithi081.stderr:2024-02-21T23:02:41.541+0000 7fc410c00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-21T23:02:41.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.624+0000 7f0c06e00640 1 Processor -- start 2024-02-21T23:02:41.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.624+0000 7f0c06e00640 1 -- start start 2024-02-21T23:02:41.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.626+0000 7f0c06e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0c0014ce30 0x7f0c000c3280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:41.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.626+0000 7f0c06e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c000c4990 con 0x7f0c000c0c20 2024-02-21T23:02:41.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.626+0000 7f0c06e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c00153ca0 con 0x7f0c000c37c0 2024-02-21T23:02:41.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.626+0000 7f0c06e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c0014d210 con 0x7f0c0014ce30 2024-02-21T23:02:41.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.626+0000 7f0c04e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f0c000c0c20 0x7f0c00156cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:56900/0 (socket says 172.21.15.28:56900) 2024-02-21T23:02:41.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.626+0000 7f0c04e00640 1 -- 172.21.15.28:0/2505282461 learned_addr learned my addr 172.21.15.28:0/2505282461 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:02:41.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.626+0000 7f0bffe00640 1 --2- 172.21.15.28:0/2505282461 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0c0014ce30 0x7f0c000c3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:41.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.627+0000 7f0bff400640 1 -- 172.21.15.28:0/2505282461 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4259021430 0 0) 0x7f0c000c4990 con 0x7f0c000c0c20 2024-02-21T23:02:41.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.627+0000 7f0bff400640 1 -- 172.21.15.28:0/2505282461 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0be4003520 con 0x7f0c000c0c20 2024-02-21T23:02:41.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.627+0000 7f0bff400640 1 -- 172.21.15.28:0/2505282461 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2392890607 0 0) 0x7f0be4003520 con 0x7f0c000c0c20 2024-02-21T23:02:41.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.627+0000 7f0bff400640 1 -- 172.21.15.28:0/2505282461 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0c0014ce30 msgr2=0x7f0c000c3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:41.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.627+0000 7f0bff400640 1 --2- 172.21.15.28:0/2505282461 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0c0014ce30 0x7f0c000c3280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:41.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.627+0000 7f0bff400640 1 -- 172.21.15.28:0/2505282461 >> v1:172.21.15.81:6789/0 conn(0x7f0c000c37c0 legacy=0x7f0c001565c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:41.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.627+0000 7f0bff400640 1 -- 172.21.15.28:0/2505282461 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c000c1320 con 0x7f0c000c0c20 2024-02-21T23:02:41.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.627+0000 7f0bff400640 1 -- 172.21.15.28:0/2505282461 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f0bf8001670 con 0x7f0c000c0c20 2024-02-21T23:02:41.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.627+0000 7f0bff400640 1 -- 172.21.15.28:0/2505282461 >> v1:172.21.15.28:6789/0 conn(0x7f0c000c0c20 legacy=0x7f0c00156cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:41.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.627+0000 7f0bff400640 1 --2- 172.21.15.28:0/2505282461 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0be40039e0 0x7f0be4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:41.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.627+0000 7f0bff400640 1 --2- 172.21.15.28:0/2505282461 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0c0014ce30 0x7f0be4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:41.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.627+0000 7f0bff400640 1 --2- 172.21.15.28:0/2505282461 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0be4004d40 0x7f0be40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:41.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.627+0000 7f0bff400640 1 -- 172.21.15.28:0/2505282461 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c0014d210 con 0x7f0be40039e0 2024-02-21T23:02:41.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.627+0000 7f0bff400640 1 -- 172.21.15.28:0/2505282461 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0be400baa0 con 0x7f0be4004d40 2024-02-21T23:02:41.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.627+0000 7f0bff400640 1 -- 172.21.15.28:0/2505282461 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0be400bca0 con 0x7f0c0014ce30 2024-02-21T23:02:41.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.628+0000 7f0bffe00640 1 --2- 172.21.15.28:0/2505282461 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0be40039e0 0x7f0be4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:41.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.628+0000 7f0bf7400640 1 --2- 172.21.15.28:0/2505282461 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0c0014ce30 0x7f0be4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:41.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.628+0000 7f0c04e00640 1 --2- 172.21.15.28:0/2505282461 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0be4004d40 0x7f0be40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:41.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.628+0000 7f0bffe00640 1 -- 172.21.15.28:0/2505282461 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0c0014ce30 msgr2=0x7f0be4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:41.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.628+0000 7f0bffe00640 1 --2- 172.21.15.28:0/2505282461 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0c0014ce30 0x7f0be4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:41.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.628+0000 7f0bffe00640 1 -- 172.21.15.28:0/2505282461 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0be4004d40 msgr2=0x7f0be40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:41.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.628+0000 7f0bffe00640 1 --2- 172.21.15.28:0/2505282461 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0be4004d40 0x7f0be40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:41.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.628+0000 7f0bffe00640 1 -- 172.21.15.28:0/2505282461 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0be4003520 con 0x7f0be40039e0 2024-02-21T23:02:41.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.628+0000 7f0c04e00640 1 --2- 172.21.15.28:0/2505282461 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0be4004d40 0x7f0be40051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:02:41.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.629+0000 7f0bffe00640 1 --2- 172.21.15.28:0/2505282461 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0be40039e0 0x7f0be4003dc0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0bf0009fc0 tx=0x7f0be4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=695169569a716ac7 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:41.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.629+0000 7f0bff400640 1 -- 172.21.15.28:0/2505282461 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf0039070 con 0x7f0be40039e0 2024-02-21T23:02:41.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.629+0000 7f0bff400640 1 -- 172.21.15.28:0/2505282461 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bf002cc20 con 0x7f0be40039e0 2024-02-21T23:02:41.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.629+0000 7f0bff400640 1 -- 172.21.15.28:0/2505282461 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf0009c10 con 0x7f0be40039e0 2024-02-21T23:02:41.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.630+0000 7f0c06e00640 1 -- 172.21.15.28:0/2505282461 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0be40039e0 msgr2=0x7f0be4003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:41.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.630+0000 7f0c06e00640 1 --2- 172.21.15.28:0/2505282461 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0be40039e0 0x7f0be4003dc0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0bf0009fc0 tx=0x7f0be4005830 comp rx=0 tx=0).stop 2024-02-21T23:02:41.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.630+0000 7f0c04e00640 1 -- 172.21.15.28:0/2505282461 reap_dead start 2024-02-21T23:02:41.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.630+0000 7f0c06e00640 1 -- 172.21.15.28:0/2505282461 shutdown_connections 2024-02-21T23:02:41.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.630+0000 7f0c06e00640 1 -- 172.21.15.28:0/2505282461 >> 172.21.15.28:0/2505282461 conn(0x7f0c000c67f0 msgr2=0x7f0c000c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:41.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.630+0000 7f0c06e00640 1 -- 172.21.15.28:0/2505282461 shutdown_connections 2024-02-21T23:02:41.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.630+0000 7f0c06e00640 1 -- 172.21.15.28:0/2505282461 wait complete. 2024-02-21T23:02:41.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.632+0000 7f0c06e00640 1 Processor -- start 2024-02-21T23:02:41.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.632+0000 7f0c06e00640 1 -- start start 2024-02-21T23:02:41.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.633+0000 7f0c06e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0be40039e0 0x7f0c001ef890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:41.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.633+0000 7f0c06e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c001f1c00 0x7f0c000aa970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:41.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.633+0000 7f0c06e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0c000aaeb0 0x7f0c001eb520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:41.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.633+0000 7f0c06e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c000c15c0 con 0x7f0c001f1c00 2024-02-21T23:02:41.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.633+0000 7f0c06e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c00153770 con 0x7f0c000aaeb0 2024-02-21T23:02:41.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.633+0000 7f0c06e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c000c18a0 con 0x7f0be40039e0 2024-02-21T23:02:41.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.633+0000 7f0bf7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c001f1c00 0x7f0c000aa970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:41.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.633+0000 7f0bf7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c001f1c00 0x7f0c000aa970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38560/0 (socket says 172.21.15.28:38560) 2024-02-21T23:02:41.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.633+0000 7f0bf7400640 1 -- 172.21.15.28:0/4008610671 learned_addr learned my addr 172.21.15.28:0/4008610671 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:02:41.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.634+0000 7f0bffe00640 1 --2- 172.21.15.28:0/4008610671 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0be40039e0 0x7f0c001ef890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:41.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.634+0000 7f0c04e00640 1 --2- 172.21.15.28:0/4008610671 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0c000aaeb0 0x7f0c001eb520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:41.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.634+0000 7f0bf7400640 1 -- 172.21.15.28:0/4008610671 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0be40039e0 msgr2=0x7f0c001ef890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:41.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.634+0000 7f0bf7400640 1 --2- 172.21.15.28:0/4008610671 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0be40039e0 0x7f0c001ef890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:41.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.634+0000 7f0bf7400640 1 -- 172.21.15.28:0/4008610671 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0c000aaeb0 msgr2=0x7f0c001eb520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:41.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.634+0000 7f0bf7400640 1 --2- 172.21.15.28:0/4008610671 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0c000aaeb0 0x7f0c001eb520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:41.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.634+0000 7f0bf7400640 1 -- 172.21.15.28:0/4008610671 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c001f2440 con 0x7f0c001f1c00 2024-02-21T23:02:41.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.634+0000 7f0bffe00640 1 --2- 172.21.15.28:0/4008610671 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0be40039e0 0x7f0c001ef890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:02:41.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.634+0000 7f0bf7400640 1 --2- 172.21.15.28:0/4008610671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c001f1c00 0x7f0c000aa970 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0be800e800 tx=0x7f0be800ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:41.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.634+0000 7f0bfcc00640 1 -- 172.21.15.28:0/4008610671 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be80050d0 con 0x7f0c001f1c00 2024-02-21T23:02:41.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.634+0000 7f0bfcc00640 1 -- 172.21.15.28:0/4008610671 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0be8018070 con 0x7f0c001f1c00 2024-02-21T23:02:41.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.634+0000 7f0c06e00640 1 -- 172.21.15.28:0/4008610671 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c001f2720 con 0x7f0c001f1c00 2024-02-21T23:02:41.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.634+0000 7f0bfcc00640 1 -- 172.21.15.28:0/4008610671 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be800c6c0 con 0x7f0c001f1c00 2024-02-21T23:02:41.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.635+0000 7f0c06e00640 1 -- 172.21.15.28:0/4008610671 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c000ac540 con 0x7f0c001f1c00 2024-02-21T23:02:41.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.636+0000 7f0c06e00640 1 -- 172.21.15.28:0/4008610671 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c001f28f0 con 0x7f0c001f1c00 2024-02-21T23:02:41.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.637+0000 7f0bfcc00640 1 -- 172.21.15.28:0/4008610671 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0be800c860 con 0x7f0c001f1c00 2024-02-21T23:02:41.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.638+0000 7f0bfcc00640 1 --2- 172.21.15.28:0/4008610671 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0bdc06b520 0x7f0bdc06d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:41.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.639+0000 7f0bfcc00640 1 -- 172.21.15.28:0/4008610671 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f0be808a300 con 0x7f0c001f1c00 2024-02-21T23:02:41.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.639+0000 7f0bffe00640 1 --2- 172.21.15.28:0/4008610671 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0bdc06b520 0x7f0bdc06d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:41.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.640+0000 7f0bffe00640 1 --2- 172.21.15.28:0/4008610671 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0bdc06b520 0x7f0bdc06d9e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0bf0036000 tx=0x7f0bf002e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:41.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.644+0000 7f0bfcc00640 1 -- 172.21.15.28:0/4008610671 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f0be8058ed0 con 0x7f0c001f1c00 2024-02-21T23:02:41.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:41.842+0000 7f0c06e00640 1 -- 172.21.15.28:0/4008610671 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7f0c00067f30 con 0x7f0c001f1c00 2024-02-21T23:02:42.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.404+0000 7f0bfcc00640 1 -- 172.21.15.28:0/4008610671 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v23) v1 ==== 169+0+0 (secure 0 0 0) 0x7f0be80587d0 con 0x7f0c001f1c00 2024-02-21T23:02:42.394 INFO:teuthology.orchestra.run.smithi028.stderr:set pool 4 allow_ec_overwrites to true 2024-02-21T23:02:42.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.409+0000 7f0c06e00640 1 -- 172.21.15.28:0/4008610671 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0bdc06b520 msgr2=0x7f0bdc06d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:42.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.409+0000 7f0c06e00640 1 --2- 172.21.15.28:0/4008610671 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0bdc06b520 0x7f0bdc06d9e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0bf0036000 tx=0x7f0bf002e000 comp rx=0 tx=0).stop 2024-02-21T23:02:42.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.409+0000 7f0c06e00640 1 -- 172.21.15.28:0/4008610671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c001f1c00 msgr2=0x7f0c000aa970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:42.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.409+0000 7f0c06e00640 1 --2- 172.21.15.28:0/4008610671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c001f1c00 0x7f0c000aa970 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0be800e800 tx=0x7f0be800ecc0 comp rx=0 tx=0).stop 2024-02-21T23:02:42.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.410+0000 7f0c06e00640 1 -- 172.21.15.28:0/4008610671 shutdown_connections 2024-02-21T23:02:42.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.410+0000 7f0c06e00640 1 --2- 172.21.15.28:0/4008610671 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0bdc06b520 0x7f0bdc06d9e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:42.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.410+0000 7f0c06e00640 1 --2- 172.21.15.28:0/4008610671 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0c000aaeb0 0x7f0c001eb520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:42.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.410+0000 7f0c06e00640 1 --2- 172.21.15.28:0/4008610671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c001f1c00 0x7f0c000aa970 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:42.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.410+0000 7f0c06e00640 1 --2- 172.21.15.28:0/4008610671 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0be40039e0 0x7f0c001ef890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:42.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.410+0000 7f0c06e00640 1 -- 172.21.15.28:0/4008610671 >> 172.21.15.28:0/4008610671 conn(0x7f0c000c67f0 msgr2=0x7f0c000c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:42.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.419+0000 7f0c06e00640 1 -- 172.21.15.28:0/4008610671 shutdown_connections 2024-02-21T23:02:42.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.420+0000 7f0c06e00640 1 -- 172.21.15.28:0/4008610671 wait complete. 2024-02-21T23:02:42.442 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-02-21T23:02:42.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.644+0000 7f264ca00640 1 Processor -- start 2024-02-21T23:02:42.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.644+0000 7f264ca00640 1 -- start start 2024-02-21T23:02:42.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.645+0000 7f264ca00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f26480a7080 0x7f26480c2b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:42.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.645+0000 7f264ca00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26480c2380 con 0x7f26480c5a00 2024-02-21T23:02:42.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.645+0000 7f264ca00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26480c5e90 con 0x7f26481523f0 2024-02-21T23:02:42.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.645+0000 7f264ca00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26480c6170 con 0x7f26480a7080 2024-02-21T23:02:42.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.645+0000 7f2645e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f26480c5a00 0x7f26480c6dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:56908/0 (socket says 172.21.15.28:56908) 2024-02-21T23:02:42.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.646+0000 7f2645e00640 1 -- 172.21.15.28:0/236368516 learned_addr learned my addr 172.21.15.28:0/236368516 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:02:42.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.646+0000 7f2647200640 1 --2- 172.21.15.28:0/236368516 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f26480a7080 0x7f26480c2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:42.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.646+0000 7f2645400640 1 -- 172.21.15.28:0/236368516 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 111192951 0 0) 0x7f26480c2380 con 0x7f26480c5a00 2024-02-21T23:02:42.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.646+0000 7f2645400640 1 -- 172.21.15.28:0/236368516 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f262c003520 con 0x7f26480c5a00 2024-02-21T23:02:42.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.646+0000 7f2647200640 1 -- 172.21.15.28:0/236368516 >> v1:172.21.15.81:6789/0 conn(0x7f26481523f0 legacy=0x7f2648158460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:42.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.646+0000 7f2647200640 1 -- 172.21.15.28:0/236368516 >> v1:172.21.15.28:6789/0 conn(0x7f26480c5a00 legacy=0x7f26480c6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:42.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.646+0000 7f2647200640 1 -- 172.21.15.28:0/236368516 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26480c25f0 con 0x7f26480a7080 2024-02-21T23:02:42.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.647+0000 7f2647200640 1 --2- 172.21.15.28:0/236368516 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f26480a7080 0x7f26480c2b10 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2640009dc0 tx=0x7f264002bfa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bfeb7b4ffa2e672d server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:42.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.648+0000 7f2645400640 1 -- 172.21.15.28:0/236368516 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2640039070 con 0x7f26480a7080 2024-02-21T23:02:42.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.648+0000 7f2645400640 1 -- 172.21.15.28:0/236368516 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f264002cd50 con 0x7f26480a7080 2024-02-21T23:02:42.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.648+0000 7f2645400640 1 -- 172.21.15.28:0/236368516 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2640034660 con 0x7f26480a7080 2024-02-21T23:02:42.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.648+0000 7f264ca00640 1 -- 172.21.15.28:0/236368516 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f26480a7080 msgr2=0x7f26480c2b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:42.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.648+0000 7f264ca00640 1 --2- 172.21.15.28:0/236368516 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f26480a7080 0x7f26480c2b10 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2640009dc0 tx=0x7f264002bfa0 comp rx=0 tx=0).stop 2024-02-21T23:02:42.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.648+0000 7f264ca00640 1 -- 172.21.15.28:0/236368516 shutdown_connections 2024-02-21T23:02:42.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.648+0000 7f264ca00640 1 --2- 172.21.15.28:0/236368516 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f26480a7080 0x7f26480c2b10 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:42.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.648+0000 7f264ca00640 1 -- 172.21.15.28:0/236368516 >> 172.21.15.28:0/236368516 conn(0x7f26480a9690 msgr2=0x7f26480a9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:42.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.649+0000 7f264ca00640 1 -- 172.21.15.28:0/236368516 shutdown_connections 2024-02-21T23:02:42.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.649+0000 7f264ca00640 1 -- 172.21.15.28:0/236368516 wait complete. 2024-02-21T23:02:42.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.650+0000 7f264ca00640 1 Processor -- start 2024-02-21T23:02:42.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.650+0000 7f264ca00640 1 -- start start 2024-02-21T23:02:42.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.651+0000 7f264ca00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f26480a7080 0x7f26480a8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:42.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.651+0000 7f264ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26481523f0 0x7f2648153d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:42.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.651+0000 7f264ca00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f264814e1b0 0x7f2648154290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:42.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.651+0000 7f264ca00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26480c74e0 con 0x7f26481523f0 2024-02-21T23:02:42.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.651+0000 7f264ca00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26480c6450 con 0x7f264814e1b0 2024-02-21T23:02:42.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.651+0000 7f264ca00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26480c7660 con 0x7f26480a7080 2024-02-21T23:02:42.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.652+0000 7f2645e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26481523f0 0x7f2648153d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:42.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.652+0000 7f2645e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26481523f0 0x7f2648153d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38572/0 (socket says 172.21.15.28:38572) 2024-02-21T23:02:42.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.652+0000 7f2645e00640 1 -- 172.21.15.28:0/2004324977 learned_addr learned my addr 172.21.15.28:0/2004324977 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:02:42.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.652+0000 7f2646800640 1 --2- 172.21.15.28:0/2004324977 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f26480a7080 0x7f26480a8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:42.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.652+0000 7f2647200640 1 --2- 172.21.15.28:0/2004324977 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f264814e1b0 0x7f2648154290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:42.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.652+0000 7f2645e00640 1 -- 172.21.15.28:0/2004324977 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f26480a7080 msgr2=0x7f26480a8ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:42.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.652+0000 7f2645e00640 1 --2- 172.21.15.28:0/2004324977 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f26480a7080 0x7f26480a8ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:42.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.652+0000 7f2645e00640 1 -- 172.21.15.28:0/2004324977 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f264814e1b0 msgr2=0x7f2648154290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:42.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.652+0000 7f2645e00640 1 --2- 172.21.15.28:0/2004324977 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f264814e1b0 0x7f2648154290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:42.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.652+0000 7f2645e00640 1 -- 172.21.15.28:0/2004324977 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26481547d0 con 0x7f26481523f0 2024-02-21T23:02:42.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.652+0000 7f2646800640 1 --2- 172.21.15.28:0/2004324977 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f26480a7080 0x7f26480a8ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:02:42.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.653+0000 7f2645e00640 1 --2- 172.21.15.28:0/2004324977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26481523f0 0x7f2648153d50 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f263400c500 tx=0x7f263400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:42.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.653+0000 7f263ea00640 1 -- 172.21.15.28:0/2004324977 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2634017070 con 0x7f26481523f0 2024-02-21T23:02:42.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.653+0000 7f263ea00640 1 -- 172.21.15.28:0/2004324977 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2634005ce0 con 0x7f26481523f0 2024-02-21T23:02:42.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.653+0000 7f263ea00640 1 -- 172.21.15.28:0/2004324977 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2634007330 con 0x7f26481523f0 2024-02-21T23:02:42.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.653+0000 7f264ca00640 1 -- 172.21.15.28:0/2004324977 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26481494f0 con 0x7f26481523f0 2024-02-21T23:02:42.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.653+0000 7f264ca00640 1 -- 172.21.15.28:0/2004324977 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26481edde0 con 0x7f26481523f0 2024-02-21T23:02:42.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.654+0000 7f264ca00640 1 -- 172.21.15.28:0/2004324977 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2648155870 con 0x7f26481523f0 2024-02-21T23:02:42.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.655+0000 7f263ea00640 1 -- 172.21.15.28:0/2004324977 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f263401a070 con 0x7f26481523f0 2024-02-21T23:02:42.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.656+0000 7f263ea00640 1 --2- 172.21.15.28:0/2004324977 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f262806b4d0 0x7f262806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:42.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.656+0000 7f263ea00640 1 -- 172.21.15.28:0/2004324977 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f26340898b0 con 0x7f26481523f0 2024-02-21T23:02:42.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.657+0000 7f2646800640 1 --2- 172.21.15.28:0/2004324977 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f262806b4d0 0x7f262806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:42.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.657+0000 7f2646800640 1 --2- 172.21.15.28:0/2004324977 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f262806b4d0 0x7f262806d990 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2648158180 tx=0x7f2638009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:42.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.661+0000 7f263ea00640 1 -- 172.21.15.28:0/2004324977 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f2648155870 con 0x7f26481523f0 2024-02-21T23:02:42.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:42.898+0000 7f264ca00640 1 -- 172.21.15.28:0/2004324977 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) v1 -- 0x7f26480008d0 con 0x7f26481523f0 2024-02-21T23:02:43.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.421+0000 7f263ea00640 1 -- 172.21.15.28:0/2004324977 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v7) v1 ==== 137+0+0 (secure 0 0 0) 0x7f2634057f10 con 0x7f26481523f0 2024-02-21T23:02:43.411 INFO:teuthology.orchestra.run.smithi028.stderr:added data pool 4 to fsmap 2024-02-21T23:02:43.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.425+0000 7f264ca00640 1 -- 172.21.15.28:0/2004324977 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f262806b4d0 msgr2=0x7f262806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:43.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.425+0000 7f264ca00640 1 --2- 172.21.15.28:0/2004324977 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f262806b4d0 0x7f262806d990 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2648158180 tx=0x7f2638009040 comp rx=0 tx=0).stop 2024-02-21T23:02:43.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.425+0000 7f264ca00640 1 -- 172.21.15.28:0/2004324977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26481523f0 msgr2=0x7f2648153d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:43.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.425+0000 7f264ca00640 1 --2- 172.21.15.28:0/2004324977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26481523f0 0x7f2648153d50 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f263400c500 tx=0x7f263400c9c0 comp rx=0 tx=0).stop 2024-02-21T23:02:43.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.425+0000 7f264ca00640 1 -- 172.21.15.28:0/2004324977 shutdown_connections 2024-02-21T23:02:43.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.425+0000 7f264ca00640 1 --2- 172.21.15.28:0/2004324977 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f262806b4d0 0x7f262806d990 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:43.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.425+0000 7f264ca00640 1 --2- 172.21.15.28:0/2004324977 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f264814e1b0 0x7f2648154290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:43.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.425+0000 7f264ca00640 1 --2- 172.21.15.28:0/2004324977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26481523f0 0x7f2648153d50 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:43.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.425+0000 7f264ca00640 1 --2- 172.21.15.28:0/2004324977 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f26480a7080 0x7f26480a8ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:43.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.425+0000 7f264ca00640 1 -- 172.21.15.28:0/2004324977 >> 172.21.15.28:0/2004324977 conn(0x7f26480a9690 msgr2=0x7f264814f250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:43.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.425+0000 7f264ca00640 1 -- 172.21.15.28:0/2004324977 shutdown_connections 2024-02-21T23:02:43.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.425+0000 7f264ca00640 1 -- 172.21.15.28:0/2004324977 wait complete. 2024-02-21T23:02:43.438 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:02:43.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.672+0000 7f6391a00640 1 Processor -- start 2024-02-21T23:02:43.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.673+0000 7f6391a00640 1 -- start start 2024-02-21T23:02:43.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.674+0000 7f6391a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f638c0c6460 0x7f638c0ca8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:43.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.674+0000 7f6391a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f638c0cadf0 0x7f638c0bf990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:43.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.674+0000 7f6391a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f638c1551b0 con 0x7f638c13e620 2024-02-21T23:02:43.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.674+0000 7f6391a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f638c0c71d0 con 0x7f638c0cadf0 2024-02-21T23:02:43.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.674+0000 7f6391a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f638c155420 con 0x7f638c0c6460 2024-02-21T23:02:43.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.674+0000 7f638aa00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f638c13e620 0x7f638c159070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:56920/0 (socket says 172.21.15.28:56920) 2024-02-21T23:02:43.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.674+0000 7f638aa00640 1 -- 172.21.15.28:0/184944646 learned_addr learned my addr 172.21.15.28:0/184944646 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:02:43.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.674+0000 7f638b400640 1 --2- 172.21.15.28:0/184944646 >> v2:172.21.15.28:3300/0 conn(0x7f638c0cadf0 0x7f638c0bf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:43.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.675+0000 7f6383e00640 1 --2- 172.21.15.28:0/184944646 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f638c0c6460 0x7f638c0ca8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:43.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.675+0000 7f638b400640 1 -- 172.21.15.28:0/184944646 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f638c0c6460 msgr2=0x7f638c0ca8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:43.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.675+0000 7f638b400640 1 --2- 172.21.15.28:0/184944646 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f638c0c6460 0x7f638c0ca8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:43.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.675+0000 7f638b400640 1 -- 172.21.15.28:0/184944646 >> v1:172.21.15.28:6789/0 conn(0x7f638c13e620 legacy=0x7f638c159070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:43.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.675+0000 7f638b400640 1 -- 172.21.15.28:0/184944646 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f638c155700 con 0x7f638c0cadf0 2024-02-21T23:02:43.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.676+0000 7f638b400640 1 --2- 172.21.15.28:0/184944646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f638c0cadf0 0x7f638c0bf990 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6384009d90 tx=0x7f638402be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=283c6d188d4bbf27 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:43.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.676+0000 7f638a000640 1 -- 172.21.15.28:0/184944646 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6384039070 con 0x7f638c0cadf0 2024-02-21T23:02:43.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.676+0000 7f638a000640 1 -- 172.21.15.28:0/184944646 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f638402cbd0 con 0x7f638c0cadf0 2024-02-21T23:02:43.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.676+0000 7f638a000640 1 -- 172.21.15.28:0/184944646 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6384034440 con 0x7f638c0cadf0 2024-02-21T23:02:43.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.677+0000 7f6391a00640 1 -- 172.21.15.28:0/184944646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f638c0cadf0 msgr2=0x7f638c0bf990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:43.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.677+0000 7f6391a00640 1 --2- 172.21.15.28:0/184944646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f638c0cadf0 0x7f638c0bf990 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6384009d90 tx=0x7f638402be20 comp rx=0 tx=0).stop 2024-02-21T23:02:43.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.677+0000 7f6391a00640 1 -- 172.21.15.28:0/184944646 shutdown_connections 2024-02-21T23:02:43.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.677+0000 7f6391a00640 1 --2- 172.21.15.28:0/184944646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f638c0cadf0 0x7f638c0bf990 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:43.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.677+0000 7f6391a00640 1 --2- 172.21.15.28:0/184944646 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f638c0c6460 0x7f638c0ca8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:43.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.677+0000 7f6391a00640 1 -- 172.21.15.28:0/184944646 >> 172.21.15.28:0/184944646 conn(0x7f638c08b1c0 msgr2=0x7f638c0725a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:43.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.677+0000 7f6391a00640 1 -- 172.21.15.28:0/184944646 shutdown_connections 2024-02-21T23:02:43.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.677+0000 7f6391a00640 1 -- 172.21.15.28:0/184944646 wait complete. 2024-02-21T23:02:43.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.679+0000 7f6391a00640 1 Processor -- start 2024-02-21T23:02:43.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.679+0000 7f6391a00640 1 -- start start 2024-02-21T23:02:43.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.680+0000 7f6391a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f638c0c6460 0x7f638c1536a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:43.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.680+0000 7f6391a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f638c0cadf0 0x7f638c13a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:43.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.680+0000 7f6391a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f638c13e620 0x7f638c13a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:43.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.680+0000 7f6391a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f638c13ea00 con 0x7f638c0cadf0 2024-02-21T23:02:43.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.680+0000 7f6391a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f638c155cc0 con 0x7f638c0c6460 2024-02-21T23:02:43.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.680+0000 7f6391a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f638c0bfed0 con 0x7f638c13e620 2024-02-21T23:02:43.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.680+0000 7f6383e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f638c0cadf0 0x7f638c13a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:43.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.680+0000 7f6383e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f638c0cadf0 0x7f638c13a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38594/0 (socket says 172.21.15.28:38594) 2024-02-21T23:02:43.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.680+0000 7f6383e00640 1 -- 172.21.15.28:0/3856707165 learned_addr learned my addr 172.21.15.28:0/3856707165 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:02:43.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.680+0000 7f638aa00640 1 --2- 172.21.15.28:0/3856707165 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f638c0c6460 0x7f638c1536a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:43.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.680+0000 7f638b400640 1 --2- 172.21.15.28:0/3856707165 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f638c13e620 0x7f638c13a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:43.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.681+0000 7f6383e00640 1 -- 172.21.15.28:0/3856707165 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f638c13e620 msgr2=0x7f638c13a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:43.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.681+0000 7f6383e00640 1 --2- 172.21.15.28:0/3856707165 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f638c13e620 0x7f638c13a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:43.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.681+0000 7f6383e00640 1 -- 172.21.15.28:0/3856707165 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f638c0c6460 msgr2=0x7f638c1536a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:43.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.681+0000 7f6383e00640 1 --2- 172.21.15.28:0/3856707165 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f638c0c6460 0x7f638c1536a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:43.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.681+0000 7f6383e00640 1 -- 172.21.15.28:0/3856707165 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f638c13afc0 con 0x7f638c0cadf0 2024-02-21T23:02:43.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.681+0000 7f6383e00640 1 --2- 172.21.15.28:0/3856707165 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f638c0cadf0 0x7f638c13a120 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f637400d800 tx=0x7f637400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:43.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.681+0000 7f6382a00640 1 -- 172.21.15.28:0/3856707165 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f637400b6c0 con 0x7f638c0cadf0 2024-02-21T23:02:43.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.681+0000 7f6382a00640 1 -- 172.21.15.28:0/3856707165 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6374019070 con 0x7f638c0cadf0 2024-02-21T23:02:43.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.681+0000 7f6382a00640 1 -- 172.21.15.28:0/3856707165 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f637400ba80 con 0x7f638c0cadf0 2024-02-21T23:02:43.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.681+0000 7f6391a00640 1 -- 172.21.15.28:0/3856707165 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f638c1de4f0 con 0x7f638c0cadf0 2024-02-21T23:02:43.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.681+0000 7f6391a00640 1 -- 172.21.15.28:0/3856707165 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f638c153be0 con 0x7f638c0cadf0 2024-02-21T23:02:43.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.683+0000 7f6391a00640 1 -- 172.21.15.28:0/3856707165 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f638c1ee740 con 0x7f638c0cadf0 2024-02-21T23:02:43.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.683+0000 7f6382a00640 1 -- 172.21.15.28:0/3856707165 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f637400bc20 con 0x7f638c0cadf0 2024-02-21T23:02:43.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.684+0000 7f6382a00640 1 --2- 172.21.15.28:0/3856707165 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f636806b4d0 0x7f636806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:43.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.684+0000 7f6382a00640 1 -- 172.21.15.28:0/3856707165 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f637408a960 con 0x7f638c0cadf0 2024-02-21T23:02:43.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.685+0000 7f638aa00640 1 --2- 172.21.15.28:0/3856707165 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f636806b4d0 0x7f636806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:43.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.686+0000 7f638aa00640 1 --2- 172.21.15.28:0/3856707165 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f636806b4d0 0x7f636806d990 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f638c073120 tx=0x7f637c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:43.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.689+0000 7f6382a00640 1 -- 172.21.15.28:0/3856707165 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f6374059530 con 0x7f638c0cadf0 2024-02-21T23:02:43.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.934+0000 7f6391a00640 1 -- 172.21.15.28:0/3856707165 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f638c067f30 con 0x7f638c0cadf0 2024-02-21T23:02:43.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.935+0000 7f6382a00640 1 -- 172.21.15.28:0/3856707165 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4934 (secure 0 0 0) 0x7f6374058e30 con 0x7f638c0cadf0 2024-02-21T23:02:43.925 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:02:43.926 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4230,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/2559793999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2559793999},{"type":"v1","addr":"172.21.15.28:6834","nonce":2559793999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4232,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3930208177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3930208177},{"type":"v1","addr":"172.21.15.28:6835","nonce":3930208177}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4238,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/2875984988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":2875984988},{"type":"v1","addr":"172.21.15.81:6836","nonce":2875984988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:02:37.812604+0000","modified":"2024-02-21T23:02:42.900460+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4267},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4267":{"gid":4267,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6837/2328151245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2328151245},{"type":"v1","addr":"172.21.15.81:6837","nonce":2328151245}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-02-21T23:02:43.926 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 7 2024-02-21T23:02:43.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.938+0000 7f6391a00640 1 -- 172.21.15.28:0/3856707165 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f636806b4d0 msgr2=0x7f636806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:43.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.938+0000 7f6391a00640 1 --2- 172.21.15.28:0/3856707165 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f636806b4d0 0x7f636806d990 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f638c073120 tx=0x7f637c009040 comp rx=0 tx=0).stop 2024-02-21T23:02:43.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.938+0000 7f6391a00640 1 -- 172.21.15.28:0/3856707165 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f638c0cadf0 msgr2=0x7f638c13a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:43.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.938+0000 7f6391a00640 1 --2- 172.21.15.28:0/3856707165 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f638c0cadf0 0x7f638c13a120 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f637400d800 tx=0x7f637400dcc0 comp rx=0 tx=0).stop 2024-02-21T23:02:43.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.938+0000 7f6391a00640 1 -- 172.21.15.28:0/3856707165 shutdown_connections 2024-02-21T23:02:43.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.938+0000 7f6391a00640 1 --2- 172.21.15.28:0/3856707165 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f636806b4d0 0x7f636806d990 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:43.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.938+0000 7f6391a00640 1 --2- 172.21.15.28:0/3856707165 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f638c13e620 0x7f638c13a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:43.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.938+0000 7f6391a00640 1 --2- 172.21.15.28:0/3856707165 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f638c0cadf0 0x7f638c13a120 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:43.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.938+0000 7f6391a00640 1 --2- 172.21.15.28:0/3856707165 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f638c0c6460 0x7f638c1536a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:43.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.938+0000 7f6391a00640 1 -- 172.21.15.28:0/3856707165 >> 172.21.15.28:0/3856707165 conn(0x7f638c08b1c0 msgr2=0x7f638c0c1200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:43.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.938+0000 7f6391a00640 1 -- 172.21.15.28:0/3856707165 shutdown_connections 2024-02-21T23:02:43.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:43.938+0000 7f6391a00640 1 -- 172.21.15.28:0/3856707165 wait complete. 2024-02-21T23:02:43.947 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:02:44.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.180+0000 7f4912000640 1 Processor -- start 2024-02-21T23:02:44.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.181+0000 7f4912000640 1 -- start start 2024-02-21T23:02:44.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.182+0000 7f4912000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f490c0aff90 0x7f490c0c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:44.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.182+0000 7f4912000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f490c06dbf0 0x7f490c158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:44.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.182+0000 7f4912000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f490c0c5720 0x7f490c159000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:44.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.182+0000 7f4912000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f490c0c7270 con 0x7f490c06dbf0 2024-02-21T23:02:44.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.182+0000 7f4912000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f490c158630 con 0x7f490c0aff90 2024-02-21T23:02:44.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.182+0000 7f4912000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f490c0c74c0 con 0x7f490c0c5720 2024-02-21T23:02:44.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.182+0000 7f490be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f490c06dbf0 0x7f490c158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:44.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.182+0000 7f490be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f490c06dbf0 0x7f490c158ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38602/0 (socket says 172.21.15.28:38602) 2024-02-21T23:02:44.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.182+0000 7f490be00640 1 -- 172.21.15.28:0/2342068826 learned_addr learned my addr 172.21.15.28:0/2342068826 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:02:44.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.183+0000 7f4910c00640 1 --2- 172.21.15.28:0/2342068826 >> v2:172.21.15.81:3300/0 conn(0x7f490c0aff90 0x7f490c0c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:44.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.183+0000 7f4911600640 1 --2- 172.21.15.28:0/2342068826 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f490c0c5720 0x7f490c159000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:44.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.183+0000 7f490be00640 1 -- 172.21.15.28:0/2342068826 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f490c0c5720 msgr2=0x7f490c159000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:44.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.183+0000 7f490be00640 1 --2- 172.21.15.28:0/2342068826 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f490c0c5720 0x7f490c159000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:44.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.183+0000 7f490be00640 1 -- 172.21.15.28:0/2342068826 >> v2:172.21.15.81:3300/0 conn(0x7f490c0aff90 msgr2=0x7f490c0c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:44.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.183+0000 7f490be00640 1 --2- 172.21.15.28:0/2342068826 >> v2:172.21.15.81:3300/0 conn(0x7f490c0aff90 0x7f490c0c51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:44.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.183+0000 7f490be00640 1 -- 172.21.15.28:0/2342068826 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f490c0c77a0 con 0x7f490c06dbf0 2024-02-21T23:02:44.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.183+0000 7f4910c00640 1 --2- 172.21.15.28:0/2342068826 >> v2:172.21.15.81:3300/0 conn(0x7f490c0aff90 0x7f490c0c51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:02:44.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.185+0000 7f490be00640 1 --2- 172.21.15.28:0/2342068826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f490c06dbf0 0x7f490c158ac0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f490c0aa1b0 tx=0x7f48f802b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f48cbdbab5c9147 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:44.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.185+0000 7f490b400640 1 -- 172.21.15.28:0/2342068826 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f8035020 con 0x7f490c06dbf0 2024-02-21T23:02:44.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.185+0000 7f490b400640 1 -- 172.21.15.28:0/2342068826 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48f802d040 con 0x7f490c06dbf0 2024-02-21T23:02:44.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.185+0000 7f490b400640 1 -- 172.21.15.28:0/2342068826 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f8031a80 con 0x7f490c06dbf0 2024-02-21T23:02:44.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.186+0000 7f4912000640 1 -- 172.21.15.28:0/2342068826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f490c06dbf0 msgr2=0x7f490c158ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:44.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.186+0000 7f4912000640 1 --2- 172.21.15.28:0/2342068826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f490c06dbf0 0x7f490c158ac0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f490c0aa1b0 tx=0x7f48f802b900 comp rx=0 tx=0).stop 2024-02-21T23:02:44.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.186+0000 7f4912000640 1 -- 172.21.15.28:0/2342068826 shutdown_connections 2024-02-21T23:02:44.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.186+0000 7f4912000640 1 --2- 172.21.15.28:0/2342068826 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f490c0c5720 0x7f490c159000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:44.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.186+0000 7f4912000640 1 --2- 172.21.15.28:0/2342068826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f490c06dbf0 0x7f490c158ac0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:44.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.186+0000 7f4912000640 1 --2- 172.21.15.28:0/2342068826 >> v2:172.21.15.81:3300/0 conn(0x7f490c0aff90 0x7f490c0c51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:44.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.186+0000 7f4912000640 1 -- 172.21.15.28:0/2342068826 >> 172.21.15.28:0/2342068826 conn(0x7f490c0ac370 msgr2=0x7f490c0ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:44.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.186+0000 7f4912000640 1 -- 172.21.15.28:0/2342068826 shutdown_connections 2024-02-21T23:02:44.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.186+0000 7f4912000640 1 -- 172.21.15.28:0/2342068826 wait complete. 2024-02-21T23:02:44.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.187+0000 7f4912000640 1 Processor -- start 2024-02-21T23:02:44.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.188+0000 7f4912000640 1 -- start start 2024-02-21T23:02:44.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.189+0000 7f4912000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f490c06dbf0 0x7f490c0c1ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:44.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.189+0000 7f4912000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f490c0aff90 0x7f490c14f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:44.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.189+0000 7f4912000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f490c0c5720 0x7f490c14f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:44.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.189+0000 7f4912000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f490c0c8760 con 0x7f490c0c5720 2024-02-21T23:02:44.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.189+0000 7f4912000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f490c0be0c0 con 0x7f490c06dbf0 2024-02-21T23:02:44.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.189+0000 7f4912000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f490c0c88e0 con 0x7f490c0aff90 2024-02-21T23:02:44.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.189+0000 7f4911600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f490c0c5720 0x7f490c14f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:44.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.189+0000 7f4911600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f490c0c5720 0x7f490c14f5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38608/0 (socket says 172.21.15.28:38608) 2024-02-21T23:02:44.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.189+0000 7f4911600640 1 -- 172.21.15.28:0/2779270553 learned_addr learned my addr 172.21.15.28:0/2779270553 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:02:44.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.189+0000 7f4910c00640 1 --2- 172.21.15.28:0/2779270553 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f490c06dbf0 0x7f490c0c1ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:44.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.189+0000 7f490be00640 1 --2- 172.21.15.28:0/2779270553 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f490c0aff90 0x7f490c14f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:44.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.189+0000 7f4911600640 1 -- 172.21.15.28:0/2779270553 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f490c0aff90 msgr2=0x7f490c14f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:44.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.189+0000 7f4911600640 1 --2- 172.21.15.28:0/2779270553 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f490c0aff90 0x7f490c14f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:44.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.189+0000 7f4911600640 1 -- 172.21.15.28:0/2779270553 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f490c06dbf0 msgr2=0x7f490c0c1ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:44.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.189+0000 7f4911600640 1 --2- 172.21.15.28:0/2779270553 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f490c06dbf0 0x7f490c0c1ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:44.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.189+0000 7f4911600640 1 -- 172.21.15.28:0/2779270553 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f490c1ee510 con 0x7f490c0c5720 2024-02-21T23:02:44.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.190+0000 7f4911600640 1 --2- 172.21.15.28:0/2779270553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f490c0c5720 0x7f490c14f5b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4904007e90 tx=0x7f490400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:44.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.190+0000 7f4908c00640 1 -- 172.21.15.28:0/2779270553 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4904014420 con 0x7f490c0c5720 2024-02-21T23:02:44.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.190+0000 7f4908c00640 1 -- 172.21.15.28:0/2779270553 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4904019070 con 0x7f490c0c5720 2024-02-21T23:02:44.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.190+0000 7f4908c00640 1 -- 172.21.15.28:0/2779270553 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4904014800 con 0x7f490c0c5720 2024-02-21T23:02:44.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.190+0000 7f4912000640 1 -- 172.21.15.28:0/2779270553 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f490c1ee7f0 con 0x7f490c0c5720 2024-02-21T23:02:44.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.190+0000 7f4912000640 1 -- 172.21.15.28:0/2779270553 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f490c1eedb0 con 0x7f490c0c5720 2024-02-21T23:02:44.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.191+0000 7f4912000640 1 -- 172.21.15.28:0/2779270553 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f490c04f290 con 0x7f490c0c5720 2024-02-21T23:02:44.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.192+0000 7f4908c00640 1 -- 172.21.15.28:0/2779270553 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f49040149a0 con 0x7f490c0c5720 2024-02-21T23:02:44.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.193+0000 7f4908c00640 1 --2- 172.21.15.28:0/2779270553 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f48ec06b4d0 0x7f48ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:44.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.193+0000 7f4908c00640 1 -- 172.21.15.28:0/2779270553 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f490408ae10 con 0x7f490c0c5720 2024-02-21T23:02:44.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.194+0000 7f4910c00640 1 --2- 172.21.15.28:0/2779270553 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f48ec06b4d0 0x7f48ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:44.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.195+0000 7f4910c00640 1 --2- 172.21.15.28:0/2779270553 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f48ec06b4d0 0x7f48ec06d990 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4900006fd0 tx=0x7f4900008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:44.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.198+0000 7f4908c00640 1 -- 172.21.15.28:0/2779270553 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f490408e030 con 0x7f490c0c5720 2024-02-21T23:02:44.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.402+0000 7f4912000640 1 -- 172.21.15.28:0/2779270553 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f490c1559a0 con 0x7f490c0c5720 2024-02-21T23:02:44.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.406+0000 7f4908c00640 1 -- 172.21.15.28:0/2779270553 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18454 (secure 0 0 0) 0x7f49040598b0 con 0x7f490c0c5720 2024-02-21T23:02:44.396 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:02:44.396 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":24,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:02:42.900286+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:02:35.795158+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-21T23:02:36.840110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-02-21T23:02:40.441388+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:02:44.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.408+0000 7f4912000640 1 -- 172.21.15.28:0/2779270553 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f48ec06b4d0 msgr2=0x7f48ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:44.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.408+0000 7f4912000640 1 --2- 172.21.15.28:0/2779270553 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f48ec06b4d0 0x7f48ec06d990 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4900006fd0 tx=0x7f4900008040 comp rx=0 tx=0).stop 2024-02-21T23:02:44.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.408+0000 7f4912000640 1 -- 172.21.15.28:0/2779270553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f490c0c5720 msgr2=0x7f490c14f5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:44.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.408+0000 7f4912000640 1 --2- 172.21.15.28:0/2779270553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f490c0c5720 0x7f490c14f5b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4904007e90 tx=0x7f490400e4b0 comp rx=0 tx=0).stop 2024-02-21T23:02:44.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.409+0000 7f4912000640 1 -- 172.21.15.28:0/2779270553 shutdown_connections 2024-02-21T23:02:44.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.409+0000 7f4912000640 1 --2- 172.21.15.28:0/2779270553 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f48ec06b4d0 0x7f48ec06d990 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:44.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.409+0000 7f4912000640 1 --2- 172.21.15.28:0/2779270553 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f490c0aff90 0x7f490c14f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:44.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.409+0000 7f4912000640 1 --2- 172.21.15.28:0/2779270553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f490c0c5720 0x7f490c14f5b0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:44.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.409+0000 7f4912000640 1 --2- 172.21.15.28:0/2779270553 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f490c06dbf0 0x7f490c0c1ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:44.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.409+0000 7f4912000640 1 -- 172.21.15.28:0/2779270553 >> 172.21.15.28:0/2779270553 conn(0x7f490c0ac370 msgr2=0x7f490c0ac750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:44.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.409+0000 7f4912000640 1 -- 172.21.15.28:0/2779270553 shutdown_connections 2024-02-21T23:02:44.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.409+0000 7f4912000640 1 -- 172.21.15.28:0/2779270553 wait complete. 2024-02-21T23:02:44.418 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:02:44.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.647+0000 7f8013e00640 1 Processor -- start 2024-02-21T23:02:44.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.647+0000 7f8013e00640 1 -- start start 2024-02-21T23:02:44.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.648+0000 7f8013e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f801414f0b0 0x7f8014155180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:44.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.648+0000 7f8013e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f80141521c0 0x7f8014152610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:44.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.648+0000 7f8013e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8014155760 con 0x7f80140a6160 2024-02-21T23:02:44.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.648+0000 7f8013e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8014152b50 con 0x7f80141521c0 2024-02-21T23:02:44.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.648+0000 7f8013e00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8014151d30 con 0x7f801414f0b0 2024-02-21T23:02:44.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.649+0000 7f8013400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f80141521c0 0x7f8014152610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:44.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.649+0000 7f8013400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f80141521c0 0x7f8014152610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38616/0 (socket says 172.21.15.28:38616) 2024-02-21T23:02:44.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.649+0000 7f8013400640 1 -- 172.21.15.28:0/2236014950 learned_addr learned my addr 172.21.15.28:0/2236014950 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:02:44.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.649+0000 7f8012a00640 1 --2- 172.21.15.28:0/2236014950 >> v2:172.21.15.81:3300/0 conn(0x7f801414f0b0 0x7f8014155180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:44.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.649+0000 7f8013400640 1 -- 172.21.15.28:0/2236014950 >> v2:172.21.15.81:3300/0 conn(0x7f801414f0b0 msgr2=0x7f8014155180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:44.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.649+0000 7f8013400640 1 --2- 172.21.15.28:0/2236014950 >> v2:172.21.15.81:3300/0 conn(0x7f801414f0b0 0x7f8014155180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:44.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.649+0000 7f8013400640 1 -- 172.21.15.28:0/2236014950 >> v1:172.21.15.81:6789/0 conn(0x7f80140a6160 legacy=0x7f80140c2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:44.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.649+0000 7f8013400640 1 -- 172.21.15.28:0/2236014950 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8014155a40 con 0x7f80141521c0 2024-02-21T23:02:44.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.649+0000 7f8012a00640 1 --2- 172.21.15.28:0/2236014950 >> v2:172.21.15.81:3300/0 conn(0x7f801414f0b0 0x7f8014155180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:02:44.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.651+0000 7f8013400640 1 --2- 172.21.15.28:0/2236014950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80141521c0 0x7f8014152610 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8014151eb0 tx=0x7f800c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbfa1b2cd236b004 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:44.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.651+0000 7f8011600640 1 -- 172.21.15.28:0/2236014950 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f800c02ca70 con 0x7f80141521c0 2024-02-21T23:02:44.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.651+0000 7f8011600640 1 -- 172.21.15.28:0/2236014950 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f800c02cc10 con 0x7f80141521c0 2024-02-21T23:02:44.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.651+0000 7f8011600640 1 -- 172.21.15.28:0/2236014950 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f800c0343f0 con 0x7f80141521c0 2024-02-21T23:02:44.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.652+0000 7f8013e00640 1 -- 172.21.15.28:0/2236014950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80141521c0 msgr2=0x7f8014152610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:44.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.652+0000 7f8013e00640 1 --2- 172.21.15.28:0/2236014950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80141521c0 0x7f8014152610 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8014151eb0 tx=0x7f800c02bed0 comp rx=0 tx=0).stop 2024-02-21T23:02:44.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.652+0000 7f8013e00640 1 -- 172.21.15.28:0/2236014950 shutdown_connections 2024-02-21T23:02:44.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.652+0000 7f8013e00640 1 --2- 172.21.15.28:0/2236014950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80141521c0 0x7f8014152610 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:44.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.652+0000 7f8013e00640 1 --2- 172.21.15.28:0/2236014950 >> v2:172.21.15.81:3300/0 conn(0x7f801414f0b0 0x7f8014155180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:44.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.652+0000 7f8013e00640 1 -- 172.21.15.28:0/2236014950 >> 172.21.15.28:0/2236014950 conn(0x7f801406b630 msgr2=0x7f80140c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:44.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.652+0000 7f8013e00640 1 -- 172.21.15.28:0/2236014950 shutdown_connections 2024-02-21T23:02:44.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.652+0000 7f8013e00640 1 -- 172.21.15.28:0/2236014950 wait complete. 2024-02-21T23:02:44.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.654+0000 7f8013e00640 1 Processor -- start 2024-02-21T23:02:44.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.654+0000 7f8013e00640 1 -- start start 2024-02-21T23:02:44.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.655+0000 7f8013e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f80140a6160 0x7f801414b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:44.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.655+0000 7f8013e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f801414f0b0 0x7f80140bf930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:44.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.655+0000 7f8013e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f80141521c0 0x7f801415de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:44.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.655+0000 7f8013e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80140a68d0 con 0x7f801414f0b0 2024-02-21T23:02:44.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.655+0000 7f8013e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8014156000 con 0x7f80141521c0 2024-02-21T23:02:44.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.655+0000 7f8013e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80140c2720 con 0x7f80140a6160 2024-02-21T23:02:44.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.656+0000 7f8012000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f801414f0b0 0x7f80140bf930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:44.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.656+0000 7f8012000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f801414f0b0 0x7f80140bf930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38620/0 (socket says 172.21.15.28:38620) 2024-02-21T23:02:44.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.656+0000 7f8012000640 1 -- 172.21.15.28:0/368525147 learned_addr learned my addr 172.21.15.28:0/368525147 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:02:44.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.656+0000 7f8012000640 1 -- 172.21.15.28:0/368525147 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f80140a6160 msgr2=0x7f801414b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:44.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.656+0000 7f8012a00640 1 --2- 172.21.15.28:0/368525147 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f80140a6160 0x7f801414b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:44.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.656+0000 7f8013400640 1 --2- 172.21.15.28:0/368525147 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f80141521c0 0x7f801415de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:44.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.656+0000 7f8012000640 1 --2- 172.21.15.28:0/368525147 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f80140a6160 0x7f801414b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:44.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.656+0000 7f8012000640 1 -- 172.21.15.28:0/368525147 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f80141521c0 msgr2=0x7f801415de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:44.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.656+0000 7f8012000640 1 --2- 172.21.15.28:0/368525147 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f80141521c0 0x7f801415de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:44.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.656+0000 7f8012000640 1 -- 172.21.15.28:0/368525147 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80141eee00 con 0x7f801414f0b0 2024-02-21T23:02:44.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.657+0000 7f8012000640 1 --2- 172.21.15.28:0/368525147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f801414f0b0 0x7f80140bf930 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f800000c430 tx=0x7f800000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:44.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.657+0000 7f800aa00640 1 -- 172.21.15.28:0/368525147 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8000011790 con 0x7f801414f0b0 2024-02-21T23:02:44.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.657+0000 7f800aa00640 1 -- 172.21.15.28:0/368525147 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8000017070 con 0x7f801414f0b0 2024-02-21T23:02:44.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.657+0000 7f800aa00640 1 -- 172.21.15.28:0/368525147 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8000011bb0 con 0x7f801414f0b0 2024-02-21T23:02:44.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.657+0000 7f8013e00640 1 -- 172.21.15.28:0/368525147 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f801415ef30 con 0x7f801414f0b0 2024-02-21T23:02:44.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.657+0000 7f8013e00640 1 -- 172.21.15.28:0/368525147 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f801415f1f0 con 0x7f801414f0b0 2024-02-21T23:02:44.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.658+0000 7f8013e00640 1 -- 172.21.15.28:0/368525147 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f801404f290 con 0x7f801414f0b0 2024-02-21T23:02:44.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.659+0000 7f800aa00640 1 -- 172.21.15.28:0/368525147 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f800001c460 con 0x7f801414f0b0 2024-02-21T23:02:44.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.660+0000 7f800aa00640 1 --2- 172.21.15.28:0/368525147 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7ff406b4d0 0x7f7ff406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:44.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.660+0000 7f800aa00640 1 -- 172.21.15.28:0/368525147 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f800008a440 con 0x7f801414f0b0 2024-02-21T23:02:44.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.660+0000 7f8012a00640 1 --2- 172.21.15.28:0/368525147 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7ff406b4d0 0x7f7ff406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:44.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.661+0000 7f8012a00640 1 --2- 172.21.15.28:0/368525147 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7ff406b4d0 0x7f7ff406d990 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8004006fd0 tx=0x7f8004008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:44.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.664+0000 7f800aa00640 1 -- 172.21.15.28:0/368525147 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f8000058f70 con 0x7f801414f0b0 2024-02-21T23:02:44.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.858+0000 7f8013e00640 1 -- 172.21.15.28:0/368525147 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f80140c2370 con 0x7f801414f0b0 2024-02-21T23:02:44.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.862+0000 7f800aa00640 1 -- 172.21.15.28:0/368525147 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18454 (secure 0 0 0) 0x7f8000058870 con 0x7f801414f0b0 2024-02-21T23:02:44.852 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:02:44.852 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":24,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:02:42.900286+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:02:35.795158+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-21T23:02:36.840110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-02-21T23:02:40.441388+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:02:44.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.864+0000 7f8013e00640 1 -- 172.21.15.28:0/368525147 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7ff406b4d0 msgr2=0x7f7ff406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:44.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.864+0000 7f8013e00640 1 --2- 172.21.15.28:0/368525147 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7ff406b4d0 0x7f7ff406d990 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8004006fd0 tx=0x7f8004008040 comp rx=0 tx=0).stop 2024-02-21T23:02:44.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.864+0000 7f8013e00640 1 -- 172.21.15.28:0/368525147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f801414f0b0 msgr2=0x7f80140bf930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:44.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.864+0000 7f8013e00640 1 --2- 172.21.15.28:0/368525147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f801414f0b0 0x7f80140bf930 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f800000c430 tx=0x7f800000c8f0 comp rx=0 tx=0).stop 2024-02-21T23:02:44.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.865+0000 7f8013e00640 1 -- 172.21.15.28:0/368525147 shutdown_connections 2024-02-21T23:02:44.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.865+0000 7f8013e00640 1 --2- 172.21.15.28:0/368525147 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7ff406b4d0 0x7f7ff406d990 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:44.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.865+0000 7f8013e00640 1 --2- 172.21.15.28:0/368525147 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f80141521c0 0x7f801415de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:44.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.865+0000 7f8013e00640 1 --2- 172.21.15.28:0/368525147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f801414f0b0 0x7f80140bf930 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:44.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.865+0000 7f8013e00640 1 --2- 172.21.15.28:0/368525147 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f80140a6160 0x7f801414b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:44.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.865+0000 7f8013e00640 1 -- 172.21.15.28:0/368525147 >> 172.21.15.28:0/368525147 conn(0x7f801406b630 msgr2=0x7f801415cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:44.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.865+0000 7f8013e00640 1 -- 172.21.15.28:0/368525147 shutdown_connections 2024-02-21T23:02:44.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:44.865+0000 7f8013e00640 1 -- 172.21.15.28:0/368525147 wait complete. 2024-02-21T23:02:44.888 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:02:45.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.112+0000 7f253e000640 1 Processor -- start 2024-02-21T23:02:45.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.112+0000 7f253e000640 1 -- start start 2024-02-21T23:02:45.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.113+0000 7f253e000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2538150450 0x7f25380a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:45.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.113+0000 7f253e000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25380c2760 con 0x7f2538156af0 2024-02-21T23:02:45.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.113+0000 7f253e000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2538156310 con 0x7f253806ee40 2024-02-21T23:02:45.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.113+0000 7f253e000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25380c22c0 con 0x7f2538150450 2024-02-21T23:02:45.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.114+0000 7f253d600640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f2538156af0 0x7f253806dff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42338/0 (socket says 172.21.15.28:42338) 2024-02-21T23:02:45.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.114+0000 7f253d600640 1 -- 172.21.15.28:0/951470400 learned_addr learned my addr 172.21.15.28:0/951470400 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:02:45.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.114+0000 7f253cc00640 1 --2- 172.21.15.28:0/951470400 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2538150450 0x7f25380a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:45.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.114+0000 7f2537400640 1 -- 172.21.15.28:0/951470400 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1712984902 0 0) 0x7f25380c2760 con 0x7f2538156af0 2024-02-21T23:02:45.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.114+0000 7f2537400640 1 -- 172.21.15.28:0/951470400 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f251c003520 con 0x7f2538156af0 2024-02-21T23:02:45.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.115+0000 7f2537400640 1 -- 172.21.15.28:0/951470400 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 776448902 0 0) 0x7f251c003520 con 0x7f2538156af0 2024-02-21T23:02:45.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.115+0000 7f2537400640 1 -- 172.21.15.28:0/951470400 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2538150450 msgr2=0x7f25380a5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:45.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.115+0000 7f2537400640 1 --2- 172.21.15.28:0/951470400 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2538150450 0x7f25380a5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:45.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.115+0000 7f2537400640 1 -- 172.21.15.28:0/951470400 >> v1:172.21.15.81:6789/0 conn(0x7f253806ee40 legacy=0x7f25381536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:45.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.115+0000 7f2537400640 1 -- 172.21.15.28:0/951470400 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25380c29d0 con 0x7f2538156af0 2024-02-21T23:02:45.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.115+0000 7f2537400640 1 -- 172.21.15.28:0/951470400 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f25300019f0 con 0x7f2538156af0 2024-02-21T23:02:45.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.115+0000 7f2537400640 1 -- 172.21.15.28:0/951470400 >> v1:172.21.15.28:6789/0 conn(0x7f2538156af0 legacy=0x7f253806dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:45.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.115+0000 7f2537400640 1 --2- 172.21.15.28:0/951470400 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2538150450 0x7f251c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:45.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.115+0000 7f2537400640 1 --2- 172.21.15.28:0/951470400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f251c003fb0 0x7f251c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:45.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.115+0000 7f2537400640 1 --2- 172.21.15.28:0/951470400 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f251c004d40 0x7f251c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:45.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.115+0000 7f2537400640 1 -- 172.21.15.28:0/951470400 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f251c00bad0 con 0x7f251c003fb0 2024-02-21T23:02:45.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.115+0000 7f2537400640 1 -- 172.21.15.28:0/951470400 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f251c00bc80 con 0x7f251c004d40 2024-02-21T23:02:45.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.115+0000 7f2537400640 1 -- 172.21.15.28:0/951470400 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f251c00be80 con 0x7f2538150450 2024-02-21T23:02:45.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.115+0000 7f2537e00640 1 --2- 172.21.15.28:0/951470400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f251c003fb0 0x7f251c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:45.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.115+0000 7f253cc00640 1 --2- 172.21.15.28:0/951470400 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2538150450 0x7f251c0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:45.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.115+0000 7f253d600640 1 --2- 172.21.15.28:0/951470400 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f251c004d40 0x7f251c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:45.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.115+0000 7f2537e00640 1 -- 172.21.15.28:0/951470400 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2538150450 msgr2=0x7f251c0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:45.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.115+0000 7f2537e00640 1 --2- 172.21.15.28:0/951470400 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2538150450 0x7f251c0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:45.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.116+0000 7f2537e00640 1 -- 172.21.15.28:0/951470400 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f251c004d40 msgr2=0x7f251c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:45.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.116+0000 7f2537e00640 1 --2- 172.21.15.28:0/951470400 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f251c004d40 0x7f251c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:45.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.116+0000 7f2537e00640 1 -- 172.21.15.28:0/951470400 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f251c003520 con 0x7f251c003fb0 2024-02-21T23:02:45.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.117+0000 7f2537e00640 1 --2- 172.21.15.28:0/951470400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f251c003fb0 0x7f251c004800 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2524001750 tx=0x7f251c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=183804442a60a3cc server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:45.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.117+0000 7f2537400640 1 -- 172.21.15.28:0/951470400 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2524001370 con 0x7f251c003fb0 2024-02-21T23:02:45.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.117+0000 7f2537400640 1 -- 172.21.15.28:0/951470400 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25240052d0 con 0x7f251c003fb0 2024-02-21T23:02:45.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.117+0000 7f2537400640 1 -- 172.21.15.28:0/951470400 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2524003830 con 0x7f251c003fb0 2024-02-21T23:02:45.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.117+0000 7f253e000640 1 -- 172.21.15.28:0/951470400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f251c003fb0 msgr2=0x7f251c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:45.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.117+0000 7f253e000640 1 --2- 172.21.15.28:0/951470400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f251c003fb0 0x7f251c004800 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2524001750 tx=0x7f251c005830 comp rx=0 tx=0).stop 2024-02-21T23:02:45.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.117+0000 7f253d600640 1 -- 172.21.15.28:0/951470400 reap_dead start 2024-02-21T23:02:45.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.117+0000 7f253e000640 1 -- 172.21.15.28:0/951470400 shutdown_connections 2024-02-21T23:02:45.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.117+0000 7f253e000640 1 -- 172.21.15.28:0/951470400 >> 172.21.15.28:0/951470400 conn(0x7f253806b630 msgr2=0x7f25380c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:45.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.118+0000 7f253e000640 1 -- 172.21.15.28:0/951470400 shutdown_connections 2024-02-21T23:02:45.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.118+0000 7f253e000640 1 -- 172.21.15.28:0/951470400 wait complete. 2024-02-21T23:02:45.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.119+0000 7f253e000640 1 Processor -- start 2024-02-21T23:02:45.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.119+0000 7f253e000640 1 -- start start 2024-02-21T23:02:45.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.120+0000 7f253e000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f251c003fb0 0x7f25380ab640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:45.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.120+0000 7f253e000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f25380ac170 0x7f25380a9610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:45.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.120+0000 7f253e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25380a9b50 0x7f2538149290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:45.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.120+0000 7f253e000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f253806e700 con 0x7f25380a9b50 2024-02-21T23:02:45.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.120+0000 7f253e000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2538156660 con 0x7f25380ac170 2024-02-21T23:02:45.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.120+0000 7f253e000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f253806e880 con 0x7f251c003fb0 2024-02-21T23:02:45.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.121+0000 7f253d600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25380a9b50 0x7f2538149290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:45.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.121+0000 7f253d600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25380a9b50 0x7f2538149290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58028/0 (socket says 172.21.15.28:58028) 2024-02-21T23:02:45.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.121+0000 7f253d600640 1 -- 172.21.15.28:0/3901886113 learned_addr learned my addr 172.21.15.28:0/3901886113 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:02:45.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.121+0000 7f2537e00640 1 --2- 172.21.15.28:0/3901886113 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f25380ac170 0x7f25380a9610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:45.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.121+0000 7f253cc00640 1 --2- 172.21.15.28:0/3901886113 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f251c003fb0 0x7f25380ab640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:45.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.121+0000 7f253d600640 1 -- 172.21.15.28:0/3901886113 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f251c003fb0 msgr2=0x7f25380ab640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:45.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.121+0000 7f253d600640 1 --2- 172.21.15.28:0/3901886113 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f251c003fb0 0x7f25380ab640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:45.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.121+0000 7f253d600640 1 -- 172.21.15.28:0/3901886113 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f25380ac170 msgr2=0x7f25380a9610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:45.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.121+0000 7f253d600640 1 --2- 172.21.15.28:0/3901886113 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f25380ac170 0x7f25380a9610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:45.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.121+0000 7f253d600640 1 -- 172.21.15.28:0/3901886113 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25380aa000 con 0x7f25380a9b50 2024-02-21T23:02:45.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.121+0000 7f253cc00640 1 --2- 172.21.15.28:0/3901886113 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f251c003fb0 0x7f25380ab640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:02:45.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.121+0000 7f253d600640 1 --2- 172.21.15.28:0/3901886113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25380a9b50 0x7f2538149290 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2538071020 tx=0x7f253000e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:45.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.122+0000 7f2534c00640 1 -- 172.21.15.28:0/3901886113 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f253000ec10 con 0x7f25380a9b50 2024-02-21T23:02:45.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.122+0000 7f2534c00640 1 -- 172.21.15.28:0/3901886113 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2530018070 con 0x7f25380a9b50 2024-02-21T23:02:45.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.122+0000 7f2534c00640 1 -- 172.21.15.28:0/3901886113 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2530013420 con 0x7f25380a9b50 2024-02-21T23:02:45.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.122+0000 7f253e000640 1 -- 172.21.15.28:0/3901886113 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25381e9240 con 0x7f25380a9b50 2024-02-21T23:02:45.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.122+0000 7f253e000640 1 -- 172.21.15.28:0/3901886113 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2538152a50 con 0x7f25380a9b50 2024-02-21T23:02:45.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.123+0000 7f253e000640 1 -- 172.21.15.28:0/3901886113 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25381e9030 con 0x7f25380a9b50 2024-02-21T23:02:45.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.124+0000 7f2534c00640 1 -- 172.21.15.28:0/3901886113 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f25300135c0 con 0x7f25380a9b50 2024-02-21T23:02:45.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.125+0000 7f2534c00640 1 --2- 172.21.15.28:0/3901886113 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f251806b4d0 0x7f251806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:45.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.125+0000 7f2534c00640 1 -- 172.21.15.28:0/3901886113 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f253008a8e0 con 0x7f25380a9b50 2024-02-21T23:02:45.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.125+0000 7f253cc00640 1 --2- 172.21.15.28:0/3901886113 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f251806b4d0 0x7f251806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:45.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.126+0000 7f253cc00640 1 --2- 172.21.15.28:0/3901886113 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f251806b4d0 0x7f251806d990 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2528009a00 tx=0x7f2528008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:45.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.129+0000 7f2534c00640 1 -- 172.21.15.28:0/3901886113 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f2530059410 con 0x7f25380a9b50 2024-02-21T23:02:45.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.372+0000 7f253e000640 1 -- 172.21.15.28:0/3901886113 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f253804f290 con 0x7f25380a9b50 2024-02-21T23:02:45.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.373+0000 7f2534c00640 1 -- 172.21.15.28:0/3901886113 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4934 (secure 0 0 0) 0x7f2530058d10 con 0x7f25380a9b50 2024-02-21T23:02:45.363 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:02:45.364 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4230,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/2559793999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2559793999},{"type":"v1","addr":"172.21.15.28:6834","nonce":2559793999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4232,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3930208177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3930208177},{"type":"v1","addr":"172.21.15.28:6835","nonce":3930208177}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4238,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/2875984988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":2875984988},{"type":"v1","addr":"172.21.15.81:6836","nonce":2875984988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:02:37.812604+0000","modified":"2024-02-21T23:02:42.900460+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4267},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4267":{"gid":4267,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6837/2328151245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2328151245},{"type":"v1","addr":"172.21.15.81:6837","nonce":2328151245}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-02-21T23:02:45.364 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 7 2024-02-21T23:02:45.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.376+0000 7f253e000640 1 -- 172.21.15.28:0/3901886113 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f251806b4d0 msgr2=0x7f251806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:45.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.376+0000 7f253e000640 1 --2- 172.21.15.28:0/3901886113 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f251806b4d0 0x7f251806d990 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2528009a00 tx=0x7f2528008040 comp rx=0 tx=0).stop 2024-02-21T23:02:45.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.376+0000 7f253e000640 1 -- 172.21.15.28:0/3901886113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25380a9b50 msgr2=0x7f2538149290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:45.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.376+0000 7f253e000640 1 --2- 172.21.15.28:0/3901886113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25380a9b50 0x7f2538149290 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2538071020 tx=0x7f253000e530 comp rx=0 tx=0).stop 2024-02-21T23:02:45.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.376+0000 7f253e000640 1 -- 172.21.15.28:0/3901886113 shutdown_connections 2024-02-21T23:02:45.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.376+0000 7f253e000640 1 --2- 172.21.15.28:0/3901886113 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f251806b4d0 0x7f251806d990 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:45.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.376+0000 7f253e000640 1 --2- 172.21.15.28:0/3901886113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25380a9b50 0x7f2538149290 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:45.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.376+0000 7f253e000640 1 --2- 172.21.15.28:0/3901886113 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f25380ac170 0x7f25380a9610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:45.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.376+0000 7f253e000640 1 --2- 172.21.15.28:0/3901886113 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f251c003fb0 0x7f25380ab640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:45.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.376+0000 7f253e000640 1 -- 172.21.15.28:0/3901886113 >> 172.21.15.28:0/3901886113 conn(0x7f253806b630 msgr2=0x7f253814f650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:45.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.376+0000 7f253e000640 1 -- 172.21.15.28:0/3901886113 shutdown_connections 2024-02-21T23:02:45.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.376+0000 7f253e000640 1 -- 172.21.15.28:0/3901886113 wait complete. 2024-02-21T23:02:45.387 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:02:45.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.613+0000 7f6baf200640 1 Processor -- start 2024-02-21T23:02:45.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.614+0000 7f6baf200640 1 -- start start 2024-02-21T23:02:45.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.615+0000 7f6baf200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f6ba806dbf0 0x7f6ba8158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:45.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.615+0000 7f6baf200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ba80c74c0 con 0x7f6ba8155960 2024-02-21T23:02:45.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.615+0000 7f6baf200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ba80c77a0 con 0x7f6ba80aff90 2024-02-21T23:02:45.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.615+0000 7f6baf200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ba80c7270 con 0x7f6ba806dbf0 2024-02-21T23:02:45.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.615+0000 7f6bae800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f6ba8155960 0x7f6ba815f890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42346/0 (socket says 172.21.15.28:42346) 2024-02-21T23:02:45.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.615+0000 7f6bae800640 1 -- 172.21.15.28:0/25544642 learned_addr learned my addr 172.21.15.28:0/25544642 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:02:45.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.616+0000 7f6bad400640 1 --2- 172.21.15.28:0/25544642 >> v2:172.21.15.81:3300/0 conn(0x7f6ba806dbf0 0x7f6ba8158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:45.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.616+0000 7f6baca00640 1 -- 172.21.15.28:0/25544642 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3498458739 0 0) 0x7f6ba80c74c0 con 0x7f6ba8155960 2024-02-21T23:02:45.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.616+0000 7f6baca00640 1 -- 172.21.15.28:0/25544642 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b90003520 con 0x7f6ba8155960 2024-02-21T23:02:45.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.616+0000 7f6baca00640 1 -- 172.21.15.28:0/25544642 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3751446229 0 0) 0x7f6b90003520 con 0x7f6ba8155960 2024-02-21T23:02:45.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.616+0000 7f6baca00640 1 -- 172.21.15.28:0/25544642 >> v2:172.21.15.81:3300/0 conn(0x7f6ba806dbf0 msgr2=0x7f6ba8158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:45.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.616+0000 7f6baca00640 1 --2- 172.21.15.28:0/25544642 >> v2:172.21.15.81:3300/0 conn(0x7f6ba806dbf0 0x7f6ba8158ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:45.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.616+0000 7f6baca00640 1 -- 172.21.15.28:0/25544642 >> v1:172.21.15.81:6789/0 conn(0x7f6ba80aff90 legacy=0x7f6ba80c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:45.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.616+0000 7f6baca00640 1 -- 172.21.15.28:0/25544642 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ba80c7a80 con 0x7f6ba8155960 2024-02-21T23:02:45.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.616+0000 7f6baca00640 1 -- 172.21.15.28:0/25544642 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f6ba40019f0 con 0x7f6ba8155960 2024-02-21T23:02:45.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.616+0000 7f6baca00640 1 -- 172.21.15.28:0/25544642 >> v1:172.21.15.28:6789/0 conn(0x7f6ba8155960 legacy=0x7f6ba815f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:45.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.616+0000 7f6baca00640 1 --2- 172.21.15.28:0/25544642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b900039b0 0x7f6b90003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:45.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.616+0000 7f6baca00640 1 --2- 172.21.15.28:0/25544642 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6b90004800 0x7f6b90004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:45.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.617+0000 7f6baca00640 1 --2- 172.21.15.28:0/25544642 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b900051c0 0x7f6b9000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:45.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.617+0000 7f6baca00640 1 -- 172.21.15.28:0/25544642 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba80c7270 con 0x7f6b900039b0 2024-02-21T23:02:45.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.617+0000 7f6baca00640 1 -- 172.21.15.28:0/25544642 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b9000c060 con 0x7f6b900051c0 2024-02-21T23:02:45.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.617+0000 7f6baca00640 1 -- 172.21.15.28:0/25544642 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b9000c260 con 0x7f6b90004800 2024-02-21T23:02:45.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.617+0000 7f6bade00640 1 --2- 172.21.15.28:0/25544642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b900039b0 0x7f6b90003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:45.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.617+0000 7f6bad400640 1 --2- 172.21.15.28:0/25544642 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6b90004800 0x7f6b90004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:45.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.617+0000 7f6bae800640 1 --2- 172.21.15.28:0/25544642 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b900051c0 0x7f6b9000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:45.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.617+0000 7f6bade00640 1 -- 172.21.15.28:0/25544642 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6b90004800 msgr2=0x7f6b90004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:45.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.617+0000 7f6bade00640 1 --2- 172.21.15.28:0/25544642 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6b90004800 0x7f6b90004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:45.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.617+0000 7f6bade00640 1 -- 172.21.15.28:0/25544642 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b900051c0 msgr2=0x7f6b9000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:45.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.617+0000 7f6bade00640 1 --2- 172.21.15.28:0/25544642 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b900051c0 0x7f6b9000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:45.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.617+0000 7f6bade00640 1 -- 172.21.15.28:0/25544642 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b90003520 con 0x7f6b900039b0 2024-02-21T23:02:45.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.617+0000 7f6bae800640 1 --2- 172.21.15.28:0/25544642 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b900051c0 0x7f6b9000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:02:45.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.617+0000 7f6bae800640 1 -- 172.21.15.28:0/25544642 reap_dead start 2024-02-21T23:02:45.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.618+0000 7f6bade00640 1 --2- 172.21.15.28:0/25544642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b900039b0 0x7f6b90003dd0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6b9c001750 tx=0x7f6b90005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68e9be48eff20ff6 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:45.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.619+0000 7f6baca00640 1 -- 172.21.15.28:0/25544642 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b9c001370 con 0x7f6b900039b0 2024-02-21T23:02:45.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.619+0000 7f6baca00640 1 -- 172.21.15.28:0/25544642 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b9c0052d0 con 0x7f6b900039b0 2024-02-21T23:02:45.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.619+0000 7f6baca00640 1 -- 172.21.15.28:0/25544642 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b9c003830 con 0x7f6b900039b0 2024-02-21T23:02:45.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.619+0000 7f6baf200640 1 -- 172.21.15.28:0/25544642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b900039b0 msgr2=0x7f6b90003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:45.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.619+0000 7f6baf200640 1 --2- 172.21.15.28:0/25544642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b900039b0 0x7f6b90003dd0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6b9c001750 tx=0x7f6b90005830 comp rx=0 tx=0).stop 2024-02-21T23:02:45.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.619+0000 7f6baf200640 1 -- 172.21.15.28:0/25544642 shutdown_connections 2024-02-21T23:02:45.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.619+0000 7f6baf200640 1 --2- 172.21.15.28:0/25544642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b900039b0 0x7f6b90003dd0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:45.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.619+0000 7f6baf200640 1 -- 172.21.15.28:0/25544642 >> 172.21.15.28:0/25544642 conn(0x7f6ba80ac370 msgr2=0x7f6ba80ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:45.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.620+0000 7f6baf200640 1 -- 172.21.15.28:0/25544642 shutdown_connections 2024-02-21T23:02:45.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.620+0000 7f6baf200640 1 -- 172.21.15.28:0/25544642 wait complete. 2024-02-21T23:02:45.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.621+0000 7f6baf200640 1 Processor -- start 2024-02-21T23:02:45.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.621+0000 7f6baf200640 1 -- start start 2024-02-21T23:02:45.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.622+0000 7f6baf200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b900039b0 0x7f6ba80c46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:45.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.622+0000 7f6baf200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6ba80c0f10 0x7f6ba80ab500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:45.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.622+0000 7f6baf200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ba80c1440 0x7f6ba8153580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:45.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.622+0000 7f6baf200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba80c8760 con 0x7f6ba80c1440 2024-02-21T23:02:45.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.622+0000 7f6baf200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba8158630 con 0x7f6b900039b0 2024-02-21T23:02:45.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.622+0000 7f6baf200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba80c88e0 con 0x7f6ba80c0f10 2024-02-21T23:02:45.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.623+0000 7f6bae800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ba80c1440 0x7f6ba8153580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:45.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.623+0000 7f6bae800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ba80c1440 0x7f6ba8153580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58046/0 (socket says 172.21.15.28:58046) 2024-02-21T23:02:45.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.623+0000 7f6bae800640 1 -- 172.21.15.28:0/3337442419 learned_addr learned my addr 172.21.15.28:0/3337442419 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:02:45.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.623+0000 7f6bad400640 1 --2- 172.21.15.28:0/3337442419 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6ba80c0f10 0x7f6ba80ab500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:45.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.623+0000 7f6bade00640 1 --2- 172.21.15.28:0/3337442419 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b900039b0 0x7f6ba80c46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:45.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.623+0000 7f6bae800640 1 -- 172.21.15.28:0/3337442419 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6ba80c0f10 msgr2=0x7f6ba80ab500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:45.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.623+0000 7f6bae800640 1 --2- 172.21.15.28:0/3337442419 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6ba80c0f10 0x7f6ba80ab500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:45.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.623+0000 7f6bae800640 1 -- 172.21.15.28:0/3337442419 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b900039b0 msgr2=0x7f6ba80c46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:45.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.623+0000 7f6bae800640 1 --2- 172.21.15.28:0/3337442419 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b900039b0 0x7f6ba80c46b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:45.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.623+0000 7f6bae800640 1 -- 172.21.15.28:0/3337442419 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ba81eeb40 con 0x7f6ba80c1440 2024-02-21T23:02:45.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.623+0000 7f6bae800640 1 --2- 172.21.15.28:0/3337442419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ba80c1440 0x7f6ba8153580 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6ba80b09a0 tx=0x7f6ba400b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:45.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.624+0000 7f6ba2000640 1 -- 172.21.15.28:0/3337442419 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba4004d60 con 0x7f6ba80c1440 2024-02-21T23:02:45.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.624+0000 7f6ba2000640 1 -- 172.21.15.28:0/3337442419 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ba400bb80 con 0x7f6ba80c1440 2024-02-21T23:02:45.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.624+0000 7f6ba2000640 1 -- 172.21.15.28:0/3337442419 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba4007070 con 0x7f6ba80c1440 2024-02-21T23:02:45.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.624+0000 7f6baf200640 1 -- 172.21.15.28:0/3337442419 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ba81545b0 con 0x7f6ba80c1440 2024-02-21T23:02:45.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.624+0000 7f6baf200640 1 -- 172.21.15.28:0/3337442419 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ba80aba40 con 0x7f6ba80c1440 2024-02-21T23:02:45.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.625+0000 7f6baf200640 1 -- 172.21.15.28:0/3337442419 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ba804f290 con 0x7f6ba80c1440 2024-02-21T23:02:45.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.626+0000 7f6ba2000640 1 -- 172.21.15.28:0/3337442419 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6ba401b020 con 0x7f6ba80c1440 2024-02-21T23:02:45.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.627+0000 7f6ba2000640 1 --2- 172.21.15.28:0/3337442419 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6b8c06b500 0x7f6b8c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:45.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.628+0000 7f6ba2000640 1 -- 172.21.15.28:0/3337442419 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f6ba408a660 con 0x7f6ba80c1440 2024-02-21T23:02:45.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.628+0000 7f6bade00640 1 --2- 172.21.15.28:0/3337442419 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6b8c06b500 0x7f6b8c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:45.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.629+0000 7f6bade00640 1 --2- 172.21.15.28:0/3337442419 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6b8c06b500 0x7f6b8c06d9c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6b9c033000 tx=0x7f6b9c009030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:45.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.632+0000 7f6ba2000640 1 -- 172.21.15.28:0/3337442419 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f6ba4059100 con 0x7f6ba80c1440 2024-02-21T23:02:45.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.834+0000 7f6baf200640 1 -- 172.21.15.28:0/3337442419 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6ba80c8c70 con 0x7f6ba80c1440 2024-02-21T23:02:45.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.837+0000 7f6ba2000640 1 -- 172.21.15.28:0/3337442419 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18454 (secure 0 0 0) 0x7f6ba4058a00 con 0x7f6ba80c1440 2024-02-21T23:02:45.828 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:02:45.828 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":24,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:02:42.900286+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:02:35.795158+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-21T23:02:36.840110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-02-21T23:02:40.441388+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:02:45.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.841+0000 7f6baf200640 1 -- 172.21.15.28:0/3337442419 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6b8c06b500 msgr2=0x7f6b8c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:45.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.841+0000 7f6baf200640 1 --2- 172.21.15.28:0/3337442419 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6b8c06b500 0x7f6b8c06d9c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6b9c033000 tx=0x7f6b9c009030 comp rx=0 tx=0).stop 2024-02-21T23:02:45.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.841+0000 7f6baf200640 1 -- 172.21.15.28:0/3337442419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ba80c1440 msgr2=0x7f6ba8153580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:45.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.841+0000 7f6baf200640 1 --2- 172.21.15.28:0/3337442419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ba80c1440 0x7f6ba8153580 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6ba80b09a0 tx=0x7f6ba400b4a0 comp rx=0 tx=0).stop 2024-02-21T23:02:45.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.841+0000 7f6baf200640 1 -- 172.21.15.28:0/3337442419 shutdown_connections 2024-02-21T23:02:45.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.841+0000 7f6baf200640 1 --2- 172.21.15.28:0/3337442419 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6b8c06b500 0x7f6b8c06d9c0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:45.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.841+0000 7f6baf200640 1 --2- 172.21.15.28:0/3337442419 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6ba80c0f10 0x7f6ba80ab500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:45.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.841+0000 7f6baf200640 1 --2- 172.21.15.28:0/3337442419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ba80c1440 0x7f6ba8153580 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:45.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.841+0000 7f6baf200640 1 --2- 172.21.15.28:0/3337442419 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b900039b0 0x7f6ba80c46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:45.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.841+0000 7f6baf200640 1 -- 172.21.15.28:0/3337442419 >> 172.21.15.28:0/3337442419 conn(0x7f6ba80ac370 msgr2=0x7f6ba80ac750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:45.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.841+0000 7f6baf200640 1 -- 172.21.15.28:0/3337442419 shutdown_connections 2024-02-21T23:02:45.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:45.841+0000 7f6baf200640 1 -- 172.21.15.28:0/3337442419 wait complete. 2024-02-21T23:02:46.252 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 7, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:02:37.812604+0000', 'modified': '2024-02-21T23:02:42.900460+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4267}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4267': {'gid': 4267, 'name': 'b', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.81:6837/2328151245', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6834', 'nonce': 2328151245}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 2328151245}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1} 2024-02-21T23:02:46.253 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-21T23:02:46.253 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-02-21T23:02:46.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.496+0000 7fb0cd400640 1 Processor -- start 2024-02-21T23:02:46.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.496+0000 7fb0cd400640 1 -- start start 2024-02-21T23:02:46.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.498+0000 7fb0cd400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb0c81527a0 0x7fb0c8158f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:46.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.498+0000 7fb0cd400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fb0c806d410 0x7fb0c806d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:46.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.498+0000 7fb0cd400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0c80c2070 con 0x7fb0c80bf9a0 2024-02-21T23:02:46.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.498+0000 7fb0cd400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0c806e790 con 0x7fb0c806d410 2024-02-21T23:02:46.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.498+0000 7fb0cd400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c80c29a0 con 0x7fb0c81527a0 2024-02-21T23:02:46.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.498+0000 7fb0c7400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb0c80bf9a0 0x7fb0c80bfdc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42362/0 (socket says 172.21.15.28:42362) 2024-02-21T23:02:46.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.498+0000 7fb0c7400640 1 -- 172.21.15.28:0/1722385265 learned_addr learned my addr 172.21.15.28:0/1722385265 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:02:46.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.499+0000 7fb0c7e00640 1 --2- 172.21.15.28:0/1722385265 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb0c81527a0 0x7fb0c8158f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:46.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.499+0000 7fb0cca00640 1 --2- 172.21.15.28:0/1722385265 >> v2:172.21.15.81:3300/0 conn(0x7fb0c806d410 0x7fb0c806d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:46.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.499+0000 7fb0c6a00640 1 -- 172.21.15.28:0/1722385265 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3503314776 0 0) 0x7fb0c80c2070 con 0x7fb0c80bf9a0 2024-02-21T23:02:46.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.499+0000 7fb0c6a00640 1 -- 172.21.15.28:0/1722385265 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0ac003520 con 0x7fb0c80bf9a0 2024-02-21T23:02:46.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.499+0000 7fb0c6a00640 1 -- 172.21.15.28:0/1722385265 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2237017437 0 0) 0x7fb0ac003520 con 0x7fb0c80bf9a0 2024-02-21T23:02:46.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.499+0000 7fb0c6a00640 1 -- 172.21.15.28:0/1722385265 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb0c81527a0 msgr2=0x7fb0c8158f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:46.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.499+0000 7fb0c6a00640 1 --2- 172.21.15.28:0/1722385265 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb0c81527a0 0x7fb0c8158f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:46.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.499+0000 7fb0c6a00640 1 -- 172.21.15.28:0/1722385265 >> v2:172.21.15.81:3300/0 conn(0x7fb0c806d410 msgr2=0x7fb0c806d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:46.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.499+0000 7fb0c6a00640 1 --2- 172.21.15.28:0/1722385265 >> v2:172.21.15.81:3300/0 conn(0x7fb0c806d410 0x7fb0c806d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:46.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.499+0000 7fb0c6a00640 1 -- 172.21.15.28:0/1722385265 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0c80c2350 con 0x7fb0c80bf9a0 2024-02-21T23:02:46.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.499+0000 7fb0c6a00640 1 -- 172.21.15.28:0/1722385265 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fb0b0001670 con 0x7fb0c80bf9a0 2024-02-21T23:02:46.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.499+0000 7fb0c6a00640 1 -- 172.21.15.28:0/1722385265 >> v1:172.21.15.28:6789/0 conn(0x7fb0c80bf9a0 legacy=0x7fb0c80bfdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:46.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.500+0000 7fb0c6a00640 1 --2- 172.21.15.28:0/1722385265 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0ac0039b0 0x7fb0ac003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:46.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.500+0000 7fb0c6a00640 1 --2- 172.21.15.28:0/1722385265 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb0c81527a0 0x7fb0ac004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:46.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.500+0000 7fb0c6a00640 1 --2- 172.21.15.28:0/1722385265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0ac004d40 0x7fb0ac0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:46.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.500+0000 7fb0c6a00640 1 -- 172.21.15.28:0/1722385265 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c806e790 con 0x7fb0ac004d40 2024-02-21T23:02:46.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.500+0000 7fb0c6a00640 1 -- 172.21.15.28:0/1722385265 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c80c29a0 con 0x7fb0ac0039b0 2024-02-21T23:02:46.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.500+0000 7fb0c6a00640 1 -- 172.21.15.28:0/1722385265 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0ac00b9f0 con 0x7fb0c81527a0 2024-02-21T23:02:46.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.500+0000 7fb0cca00640 1 --2- 172.21.15.28:0/1722385265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0ac004d40 0x7fb0ac0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:46.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.500+0000 7fb0c7e00640 1 --2- 172.21.15.28:0/1722385265 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0ac0039b0 0x7fb0ac003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:46.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.500+0000 7fb0c7400640 1 --2- 172.21.15.28:0/1722385265 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb0c81527a0 0x7fb0ac004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:46.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.500+0000 7fb0cca00640 1 -- 172.21.15.28:0/1722385265 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb0c81527a0 msgr2=0x7fb0ac004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:46.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.500+0000 7fb0cca00640 1 --2- 172.21.15.28:0/1722385265 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb0c81527a0 0x7fb0ac004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:46.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.500+0000 7fb0cca00640 1 -- 172.21.15.28:0/1722385265 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0ac0039b0 msgr2=0x7fb0ac003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:46.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.500+0000 7fb0cca00640 1 --2- 172.21.15.28:0/1722385265 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0ac0039b0 0x7fb0ac003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:46.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.500+0000 7fb0cca00640 1 -- 172.21.15.28:0/1722385265 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0ac003520 con 0x7fb0ac004d40 2024-02-21T23:02:46.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.500+0000 7fb0c7400640 1 --2- 172.21.15.28:0/1722385265 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb0c81527a0 0x7fb0ac004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:02:46.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.501+0000 7fb0cca00640 1 --2- 172.21.15.28:0/1722385265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0ac004d40 0x7fb0ac0051c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb0c0009fc0 tx=0x7fb0ac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c25db2bb218eb2ea server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:46.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.502+0000 7fb0c6a00640 1 -- 172.21.15.28:0/1722385265 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c0039070 con 0x7fb0ac004d40 2024-02-21T23:02:46.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.502+0000 7fb0c6a00640 1 -- 172.21.15.28:0/1722385265 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0c002cc60 con 0x7fb0ac004d40 2024-02-21T23:02:46.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.502+0000 7fb0c6a00640 1 -- 172.21.15.28:0/1722385265 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c0009c10 con 0x7fb0ac004d40 2024-02-21T23:02:46.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.502+0000 7fb0cd400640 1 -- 172.21.15.28:0/1722385265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0ac004d40 msgr2=0x7fb0ac0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:46.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.502+0000 7fb0cd400640 1 --2- 172.21.15.28:0/1722385265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0ac004d40 0x7fb0ac0051c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb0c0009fc0 tx=0x7fb0ac005830 comp rx=0 tx=0).stop 2024-02-21T23:02:46.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.502+0000 7fb0cca00640 1 -- 172.21.15.28:0/1722385265 reap_dead start 2024-02-21T23:02:46.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.502+0000 7fb0cd400640 1 -- 172.21.15.28:0/1722385265 shutdown_connections 2024-02-21T23:02:46.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.502+0000 7fb0cd400640 1 -- 172.21.15.28:0/1722385265 >> 172.21.15.28:0/1722385265 conn(0x7fb0c80a8320 msgr2=0x7fb0c80a8730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:46.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.502+0000 7fb0cd400640 1 -- 172.21.15.28:0/1722385265 shutdown_connections 2024-02-21T23:02:46.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.502+0000 7fb0cd400640 1 -- 172.21.15.28:0/1722385265 wait complete. 2024-02-21T23:02:46.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.504+0000 7fb0cd400640 1 Processor -- start 2024-02-21T23:02:46.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.504+0000 7fb0cd400640 1 -- start start 2024-02-21T23:02:46.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.505+0000 7fb0cd400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb0ac004d40 0x7fb0c81eb2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:46.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.505+0000 7fb0cd400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0c80722a0 0x7fb0c814d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:46.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.505+0000 7fb0cd400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0c814dcb0 0x7fb0c8154e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:46.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.505+0000 7fb0cd400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c80c1d20 con 0x7fb0c814dcb0 2024-02-21T23:02:46.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.505+0000 7fb0cd400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c80c04d0 con 0x7fb0c80722a0 2024-02-21T23:02:46.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.505+0000 7fb0cd400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c806dda0 con 0x7fb0ac004d40 2024-02-21T23:02:46.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.505+0000 7fb0cca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0c814dcb0 0x7fb0c8154e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:46.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.506+0000 7fb0cca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0c814dcb0 0x7fb0c8154e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58058/0 (socket says 172.21.15.28:58058) 2024-02-21T23:02:46.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.506+0000 7fb0cca00640 1 -- 172.21.15.28:0/3324840409 learned_addr learned my addr 172.21.15.28:0/3324840409 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:02:46.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.506+0000 7fb0c7400640 1 --2- 172.21.15.28:0/3324840409 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0c80722a0 0x7fb0c814d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:46.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.506+0000 7fb0c7e00640 1 --2- 172.21.15.28:0/3324840409 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb0ac004d40 0x7fb0c81eb2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:46.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.506+0000 7fb0cca00640 1 -- 172.21.15.28:0/3324840409 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb0ac004d40 msgr2=0x7fb0c81eb2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:46.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.506+0000 7fb0cca00640 1 --2- 172.21.15.28:0/3324840409 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb0ac004d40 0x7fb0c81eb2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:46.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.506+0000 7fb0cca00640 1 -- 172.21.15.28:0/3324840409 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0c80722a0 msgr2=0x7fb0c814d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:46.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.506+0000 7fb0cca00640 1 --2- 172.21.15.28:0/3324840409 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0c80722a0 0x7fb0c814d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:46.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.506+0000 7fb0cca00640 1 -- 172.21.15.28:0/3324840409 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0c814e160 con 0x7fb0c814dcb0 2024-02-21T23:02:46.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.506+0000 7fb0cca00640 1 --2- 172.21.15.28:0/3324840409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0c814dcb0 0x7fb0c8154e00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb0c0037040 tx=0x7fb0c0009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:46.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.507+0000 7fb0bfe00640 1 -- 172.21.15.28:0/3324840409 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c0042020 con 0x7fb0c814dcb0 2024-02-21T23:02:46.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.507+0000 7fb0bfe00640 1 -- 172.21.15.28:0/3324840409 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0c003a040 con 0x7fb0c814dcb0 2024-02-21T23:02:46.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.507+0000 7fb0bfe00640 1 -- 172.21.15.28:0/3324840409 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c0039030 con 0x7fb0c814dcb0 2024-02-21T23:02:46.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.507+0000 7fb0cd400640 1 -- 172.21.15.28:0/3324840409 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0c81eb7e0 con 0x7fb0c814dcb0 2024-02-21T23:02:46.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.507+0000 7fb0cd400640 1 -- 172.21.15.28:0/3324840409 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0c8153400 con 0x7fb0c814dcb0 2024-02-21T23:02:46.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.509+0000 7fb0bfe00640 1 -- 172.21.15.28:0/3324840409 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb0c0005d10 con 0x7fb0c814dcb0 2024-02-21T23:02:46.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.510+0000 7fb0bfe00640 1 --2- 172.21.15.28:0/3324840409 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb0a406b500 0x7fb0a406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:46.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.510+0000 7fb0bfe00640 1 -- 172.21.15.28:0/3324840409 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fb0c00ac500 con 0x7fb0c814dcb0 2024-02-21T23:02:46.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.510+0000 7fb0c7e00640 1 --2- 172.21.15.28:0/3324840409 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb0a406b500 0x7fb0a406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:46.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.511+0000 7fb0d1991740 1 Processor -- start 2024-02-21T23:02:46.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.511+0000 7fb0d1991740 1 -- start start 2024-02-21T23:02:46.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.512+0000 7fb0d1991740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560178d52020 0x560178d52400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:46.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.512+0000 7fb0d1991740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x560178d53330 0x560178d8e170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:46.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.512+0000 7fb0d1991740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560178d8e6b0 0x560178d8c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:46.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.512+0000 7fb0d1991740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x560178cb6470 con 0x560178d8e6b0 2024-02-21T23:02:46.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.512+0000 7fb0d1991740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x560178c5faf0 con 0x560178d52020 2024-02-21T23:02:46.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.512+0000 7fb0d1991740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x560178cb2510 con 0x560178d53330 2024-02-21T23:02:46.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.512+0000 7fb0c7e00640 1 --2- 172.21.15.28:0/3324840409 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb0a406b500 0x7fb0a406d9c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb0b8009990 tx=0x7fb0b8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:46.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.512+0000 7fb0cca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560178d8e6b0 0x560178d8c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:46.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.512+0000 7fb0cca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560178d8e6b0 0x560178d8c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58074/0 (socket says 172.21.15.28:58074) 2024-02-21T23:02:46.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.512+0000 7fb0cca00640 1 -- 172.21.15.28:0/3959060385 learned_addr learned my addr 172.21.15.28:0/3959060385 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:02:46.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.512+0000 7fb0c7e00640 1 --2- 172.21.15.28:0/3959060385 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560178d52020 0x560178d52400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:46.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.512+0000 7fb0c7400640 1 --2- 172.21.15.28:0/3959060385 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x560178d53330 0x560178d8e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:46.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.512+0000 7fb0cca00640 1 -- 172.21.15.28:0/3959060385 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x560178d53330 msgr2=0x560178d8e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:46.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.512+0000 7fb0cca00640 1 --2- 172.21.15.28:0/3959060385 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x560178d53330 0x560178d8e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:46.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.512+0000 7fb0cca00640 1 -- 172.21.15.28:0/3959060385 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560178d52020 msgr2=0x560178d52400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:46.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.512+0000 7fb0cca00640 1 --2- 172.21.15.28:0/3959060385 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560178d52020 0x560178d52400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:46.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.512+0000 7fb0cca00640 1 -- 172.21.15.28:0/3959060385 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560178c66d90 con 0x560178d8e6b0 2024-02-21T23:02:46.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.513+0000 7fb0cca00640 1 --2- 172.21.15.28:0/3959060385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560178d8e6b0 0x560178d8c720 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb0c007a540 tx=0x7fb0c007a350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e92f1cf7037485b9 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:46.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.513+0000 7fb0b7e00640 1 -- 172.21.15.28:0/3959060385 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c00a9070 con 0x560178d8e6b0 2024-02-21T23:02:46.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.513+0000 7fb0b7e00640 1 -- 172.21.15.28:0/3959060385 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0c0083040 con 0x560178d8e6b0 2024-02-21T23:02:46.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.513+0000 7fb0b7e00640 1 -- 172.21.15.28:0/3959060385 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c0009c10 con 0x560178d8e6b0 2024-02-21T23:02:46.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.513+0000 7fb0d1991740 1 -- 172.21.15.28:0/3959060385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560178d8e6b0 msgr2=0x560178d8c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:46.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.513+0000 7fb0d1991740 1 --2- 172.21.15.28:0/3959060385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560178d8e6b0 0x560178d8c720 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb0c007a540 tx=0x7fb0c007a350 comp rx=0 tx=0).stop 2024-02-21T23:02:46.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.514+0000 7fb0d1991740 1 -- 172.21.15.28:0/3959060385 shutdown_connections 2024-02-21T23:02:46.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.514+0000 7fb0d1991740 1 --2- 172.21.15.28:0/3959060385 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x560178d53330 0x560178d8e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:46.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.514+0000 7fb0d1991740 1 --2- 172.21.15.28:0/3959060385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560178d8e6b0 0x560178d8c720 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:46.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.514+0000 7fb0d1991740 1 --2- 172.21.15.28:0/3959060385 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560178d52020 0x560178d52400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:46.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.514+0000 7fb0d1991740 1 -- 172.21.15.28:0/3959060385 >> 172.21.15.28:0/3959060385 conn(0x560178d4e970 msgr2=0x560178d73380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:46.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.514+0000 7fb0d1991740 1 -- 172.21.15.28:0/3959060385 shutdown_connections 2024-02-21T23:02:46.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.514+0000 7fb0d1991740 1 -- 172.21.15.28:0/3959060385 wait complete. 2024-02-21T23:02:46.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.514+0000 7fb0d1991740 1 -- start start 2024-02-21T23:02:46.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.515+0000 7fb0d1991740 1 Processor -- start 2024-02-21T23:02:46.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.515+0000 7fb0d1991740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560178d53330 0x560178d75990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:46.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.516+0000 7fb0d1991740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x560178d8e6b0 0x560178da1b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:46.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.516+0000 7fb0d1991740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560178d72910 0x560178d8b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:46.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.516+0000 7fb0d1991740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x560178c5faf0 con 0x560178d72910 2024-02-21T23:02:46.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.516+0000 7fb0d1991740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x560178c55040 con 0x560178d53330 2024-02-21T23:02:46.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.516+0000 7fb0d1991740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x560178cb6470 con 0x560178d8e6b0 2024-02-21T23:02:46.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.516+0000 7fb0d1991740 10 client.0 _ll_register_callbacks cb 0x560178c72920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:02:46.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.516+0000 7fb0cca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560178d72910 0x560178d8b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:46.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.516+0000 7fb0cca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560178d72910 0x560178d8b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58090/0 (socket says 172.21.15.28:58090) 2024-02-21T23:02:46.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.516+0000 7fb0cca00640 1 -- 172.21.15.28:0/1616356650 learned_addr learned my addr 172.21.15.28:0/1616356650 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:02:46.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.516+0000 7fb0cca00640 1 -- 172.21.15.28:0/1616356650 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x560178d8e6b0 msgr2=0x560178da1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:46.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.516+0000 7fb0cca00640 1 --2- 172.21.15.28:0/1616356650 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x560178d8e6b0 0x560178da1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:46.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.516+0000 7fb0cca00640 1 -- 172.21.15.28:0/1616356650 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560178d53330 msgr2=0x560178d75990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:46.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.516+0000 7fb0cca00640 1 --2- 172.21.15.28:0/1616356650 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560178d53330 0x560178d75990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:46.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.516+0000 7fb0cca00640 1 -- 172.21.15.28:0/1616356650 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560178c74aa0 con 0x560178d72910 2024-02-21T23:02:46.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.516+0000 7fb0cca00640 1 --2- 172.21.15.28:0/1616356650 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560178d72910 0x560178d8b2e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb0c007e010 tx=0x7fb0c003e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca77755b2798dd79 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:46.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.516+0000 7fb0b4c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:02:46.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.517+0000 7fb0b4c00640 1 -- 172.21.15.28:0/1616356650 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0c00a9040 con 0x560178d72910 2024-02-21T23:02:46.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.517+0000 7fb0b4c00640 1 -- 172.21.15.28:0/1616356650 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fb0c0078c30 con 0x560178d72910 2024-02-21T23:02:46.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.517+0000 7fb0d1991740 1 -- 172.21.15.28:0/1616356650 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb0c0083040 con 0x560178d72910 2024-02-21T23:02:46.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.517+0000 7fb0b4c00640 1 -- 172.21.15.28:0/1616356650 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb0c00a0050 con 0x560178d72910 2024-02-21T23:02:46.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.517+0000 7fb0d1991740 10 client.4392 fetch_fsmap learned FSMap version 7 2024-02-21T23:02:46.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.517+0000 7fb0d1991740 1 -- 172.21.15.28:0/1616356650 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) v3 -- 0x5601789cb2d0 con 0x560178d72910 2024-02-21T23:02:46.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.517+0000 7fb0b4c00640 1 -- 172.21.15.28:0/1616356650 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0c0083040 con 0x560178d72910 2024-02-21T23:02:46.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.517+0000 7fb0b4c00640 1 -- 172.21.15.28:0/1616356650 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 7) v1 ==== 3302+0+0 (secure 0 0 0) 0x7fb0c00a3020 con 0x560178d72910 2024-02-21T23:02:46.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.517+0000 7fb0d1991740 10 client.4392 fetch_fsmap finished waiting for FSMap version 7 2024-02-21T23:02:46.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.517+0000 7fb0d1991740 10 client.4392 resolve_mds: resolved name 'b' to daemon mds.b 2024-02-21T23:02:46.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.517+0000 7fb0d1991740 20 client.4392 populate_metadata read hostname 'smithi028' 2024-02-21T23:02:46.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.517+0000 7fb0d1991740 1 --2- 172.21.15.28:0/1616356650 >> [v2:172.21.15.81:6834/2328151245,v1:172.21.15.81:6837/2328151245] conn(0x560178d9e240 0x560178d9e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:46.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.517+0000 7fb0d1991740 4 client.4392 mds_command: new command op to 4267 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-21T23:02:46.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.517+0000 7fb0d1991740 1 -- 172.21.15.28:0/1616356650 --> [v2:172.21.15.81:6834/2328151245,v1:172.21.15.81:6837/2328151245] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560178cc39b0 con 0x560178d9e240 2024-02-21T23:02:46.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.518+0000 7fb0c7e00640 1 --2- 172.21.15.28:0/1616356650 >> [v2:172.21.15.81:6834/2328151245,v1:172.21.15.81:6837/2328151245] conn(0x560178d9e240 0x560178d9e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:46.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.519+0000 7fb0c7e00640 1 --2- 172.21.15.28:0/1616356650 >> [v2:172.21.15.81:6834/2328151245,v1:172.21.15.81:6837/2328151245] conn(0x560178d9e240 0x560178d9e620 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=644c570b8e9dbaf3 server_cookie=5caff26ed3ec46c2 in_seq=0 out_seq=0 2024-02-21T23:02:46.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.519+0000 7fb0b4c00640 10 client.4392 ms_handle_connect on v2:172.21.15.81:6834/2328151245 2024-02-21T23:02:46.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.521+0000 7fb0b4c00640 1 -- 172.21.15.28:0/1616356650 <== mds.0 v2:172.21.15.81:6834/2328151245 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x560178cc39b0 con 0x560178d9e240 2024-02-21T23:02:46.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.521+0000 7fb0b4c00640 10 client.4392 handle_command_reply: tid=0 2024-02-21T23:02:46.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.521+0000 7fb0d1991740 1 client.4392 shutdown 2024-02-21T23:02:46.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.521+0000 7fb0d1991740 1 -- 172.21.15.28:0/1616356650 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560178d72910 msgr2=0x560178d8b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:46.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.521+0000 7fb0d1991740 1 --2- 172.21.15.28:0/1616356650 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560178d72910 0x560178d8b2e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb0c007e010 tx=0x7fb0c003e000 comp rx=0 tx=0).stop 2024-02-21T23:02:46.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.521+0000 7fb0d1991740 1 -- 172.21.15.28:0/1616356650 shutdown_connections 2024-02-21T23:02:46.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.521+0000 7fb0d1991740 1 --2- 172.21.15.28:0/1616356650 >> [v2:172.21.15.81:6834/2328151245,v1:172.21.15.81:6837/2328151245] conn(0x560178d9e240 0x560178d9e620 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:46.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.521+0000 7fb0d1991740 1 --2- 172.21.15.28:0/1616356650 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x560178d8e6b0 0x560178da1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:46.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.521+0000 7fb0d1991740 1 --2- 172.21.15.28:0/1616356650 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560178d72910 0x560178d8b2e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:46.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.521+0000 7fb0b4c00640 0 client.4392 ms_handle_reset on v2:172.21.15.81:6834/2328151245 2024-02-21T23:02:46.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.521+0000 7fb0d1991740 1 --2- 172.21.15.28:0/1616356650 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560178d53330 0x560178d75990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:46.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.521+0000 7fb0d1991740 1 -- 172.21.15.28:0/1616356650 >> 172.21.15.28:0/1616356650 conn(0x560178d52020 msgr2=0x560178d8cfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:46.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.521+0000 7fb0d1991740 1 -- 172.21.15.28:0/1616356650 shutdown_connections 2024-02-21T23:02:46.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.521+0000 7fb0d1991740 1 -- 172.21.15.28:0/1616356650 wait complete. 2024-02-21T23:02:46.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.521+0000 7fb0d1991740 20 client.4392 trim_cache size 0 max 16384 2024-02-21T23:02:46.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.540+0000 7fb0d1991740 1 Processor -- start 2024-02-21T23:02:46.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.540+0000 7fb0d1991740 1 -- start start 2024-02-21T23:02:46.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.541+0000 7fb0d1991740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560178d53330 0x560178da1970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:46.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.541+0000 7fb0d1991740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560178d72910 0x560178d752f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:46.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.541+0000 7fb0d1991740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x560178d8e6b0 0x560178d76270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:46.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.541+0000 7fb0d1991740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c81f8080 con 0x560178d53330 2024-02-21T23:02:46.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.541+0000 7fb0d1991740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c81f75d0 con 0x560178d72910 2024-02-21T23:02:46.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.541+0000 7fb0d1991740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c81f7ab0 con 0x560178d8e6b0 2024-02-21T23:02:46.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.541+0000 7fb0c7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560178d53330 0x560178da1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:46.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.541+0000 7fb0c7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560178d53330 0x560178da1970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58098/0 (socket says 172.21.15.28:58098) 2024-02-21T23:02:46.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.541+0000 7fb0c7e00640 1 -- 172.21.15.28:0/3804702420 learned_addr learned my addr 172.21.15.28:0/3804702420 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:02:46.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.542+0000 7fb0cca00640 1 --2- 172.21.15.28:0/3804702420 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x560178d8e6b0 0x560178d76270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:46.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.542+0000 7fb0c7400640 1 --2- 172.21.15.28:0/3804702420 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560178d72910 0x560178d752f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:46.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.542+0000 7fb0c7e00640 1 -- 172.21.15.28:0/3804702420 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x560178d8e6b0 msgr2=0x560178d76270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:46.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.542+0000 7fb0c7e00640 1 --2- 172.21.15.28:0/3804702420 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x560178d8e6b0 0x560178d76270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:46.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.542+0000 7fb0c7e00640 1 -- 172.21.15.28:0/3804702420 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560178d72910 msgr2=0x560178d752f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:46.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.542+0000 7fb0c7e00640 1 --2- 172.21.15.28:0/3804702420 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560178d72910 0x560178d752f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:46.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.542+0000 7fb0c7e00640 1 -- 172.21.15.28:0/3804702420 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560178a956b0 con 0x560178d53330 2024-02-21T23:02:46.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.542+0000 7fb0cca00640 1 --2- 172.21.15.28:0/3804702420 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x560178d8e6b0 0x560178d76270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:02:46.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.542+0000 7fb0c7400640 1 --2- 172.21.15.28:0/3804702420 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560178d72910 0x560178d752f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:02:46.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.542+0000 7fb0c7e00640 1 --2- 172.21.15.28:0/3804702420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560178d53330 0x560178da1970 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x560178d72320 tx=0x7fb0b8016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=461000fb712249aa server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:46.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.542+0000 7fb0b7400640 1 -- 172.21.15.28:0/3804702420 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b8031070 con 0x560178d53330 2024-02-21T23:02:46.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.542+0000 7fb0b7400640 1 -- 172.21.15.28:0/3804702420 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0b801c040 con 0x560178d53330 2024-02-21T23:02:46.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.542+0000 7fb0b7400640 1 -- 172.21.15.28:0/3804702420 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b802b2d0 con 0x560178d53330 2024-02-21T23:02:46.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.543+0000 7fb0d1991740 1 -- 172.21.15.28:0/3804702420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560178d53330 msgr2=0x560178da1970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:46.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.543+0000 7fb0d1991740 1 --2- 172.21.15.28:0/3804702420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560178d53330 0x560178da1970 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x560178d72320 tx=0x7fb0b8016000 comp rx=0 tx=0).stop 2024-02-21T23:02:46.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.543+0000 7fb0d1991740 1 -- 172.21.15.28:0/3804702420 shutdown_connections 2024-02-21T23:02:46.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.543+0000 7fb0d1991740 1 --2- 172.21.15.28:0/3804702420 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x560178d8e6b0 0x560178d76270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:46.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.543+0000 7fb0d1991740 1 --2- 172.21.15.28:0/3804702420 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560178d72910 0x560178d752f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:46.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.543+0000 7fb0d1991740 1 --2- 172.21.15.28:0/3804702420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560178d53330 0x560178da1970 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:46.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.543+0000 7fb0d1991740 1 -- 172.21.15.28:0/3804702420 >> 172.21.15.28:0/3804702420 conn(0x560178d52020 msgr2=0x560178d47f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:46.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.543+0000 7fb0d1991740 1 -- 172.21.15.28:0/3804702420 shutdown_connections 2024-02-21T23:02:46.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.543+0000 7fb0d1991740 1 -- 172.21.15.28:0/3804702420 wait complete. 2024-02-21T23:02:46.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.544+0000 7fb0d1991740 1 -- start start 2024-02-21T23:02:46.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.544+0000 7fb0d1991740 1 Processor -- start 2024-02-21T23:02:46.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.545+0000 7fb0d1991740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x560178d72910 0x560178da5c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:46.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.545+0000 7fb0d1991740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560178d8e6b0 0x560178d8fb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:46.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.545+0000 7fb0d1991740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560178e00630 0x560178dfa240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:46.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.545+0000 7fb0d1991740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x560178c5faf0 con 0x560178d8e6b0 2024-02-21T23:02:46.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.545+0000 7fb0d1991740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x560178c55040 con 0x560178e00630 2024-02-21T23:02:46.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.545+0000 7fb0d1991740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x560178d74de0 con 0x560178d72910 2024-02-21T23:02:46.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.545+0000 7fb0d1991740 10 client.0 _ll_register_callbacks cb 0x7fb0c007aa70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:02:46.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.545+0000 7fb0c7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560178d8e6b0 0x560178d8fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:46.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.545+0000 7fb0c7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560178d8e6b0 0x560178d8fb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58104/0 (socket says 172.21.15.28:58104) 2024-02-21T23:02:46.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.545+0000 7fb0c7400640 1 -- 172.21.15.28:0/3406430472 learned_addr learned my addr 172.21.15.28:0/3406430472 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:02:46.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.546+0000 7fb0c7e00640 1 --2- 172.21.15.28:0/3406430472 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x560178d72910 0x560178da5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:46.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.546+0000 7fb0cca00640 1 --2- 172.21.15.28:0/3406430472 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560178e00630 0x560178dfa240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:46.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.546+0000 7fb0c7400640 1 -- 172.21.15.28:0/3406430472 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x560178d72910 msgr2=0x560178da5c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:46.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.546+0000 7fb0c7400640 1 --2- 172.21.15.28:0/3406430472 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x560178d72910 0x560178da5c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:46.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.546+0000 7fb0c7400640 1 -- 172.21.15.28:0/3406430472 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560178e00630 msgr2=0x560178dfa240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:46.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.546+0000 7fb0c7400640 1 --2- 172.21.15.28:0/3406430472 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560178e00630 0x560178dfa240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:46.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.546+0000 7fb0c7400640 1 -- 172.21.15.28:0/3406430472 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560178e00a90 con 0x560178d8e6b0 2024-02-21T23:02:46.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.546+0000 7fb0c7400640 1 --2- 172.21.15.28:0/3406430472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560178d8e6b0 0x560178d8fb20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x560178d4e6f0 tx=0x7fb0b0012920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72c3e2db69b8aa9 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:46.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.546+0000 7fb0b4c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:02:46.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.546+0000 7fb0b4c00640 1 -- 172.21.15.28:0/3406430472 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b00157f0 con 0x560178d8e6b0 2024-02-21T23:02:46.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.546+0000 7fb0b4c00640 1 -- 172.21.15.28:0/3406430472 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fb0b0015990 con 0x560178d8e6b0 2024-02-21T23:02:46.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.546+0000 7fb0d1991740 1 -- 172.21.15.28:0/3406430472 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb0b801c040 con 0x560178d8e6b0 2024-02-21T23:02:46.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.547+0000 7fb0b4c00640 1 -- 172.21.15.28:0/3406430472 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb0b0023050 con 0x560178d8e6b0 2024-02-21T23:02:46.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.547+0000 7fb0d1991740 10 client.4398 fetch_fsmap learned FSMap version 7 2024-02-21T23:02:46.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.547+0000 7fb0d1991740 1 -- 172.21.15.28:0/3406430472 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) v3 -- 0x560178d74510 con 0x560178d8e6b0 2024-02-21T23:02:46.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.547+0000 7fb0b4c00640 1 -- 172.21.15.28:0/3406430472 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0b801c040 con 0x560178d8e6b0 2024-02-21T23:02:46.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.547+0000 7fb0b4c00640 1 -- 172.21.15.28:0/3406430472 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 7) v1 ==== 3302+0+0 (secure 0 0 0) 0x7fb0b0026020 con 0x560178d8e6b0 2024-02-21T23:02:46.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.547+0000 7fb0d1991740 10 client.4398 fetch_fsmap finished waiting for FSMap version 7 2024-02-21T23:02:46.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.547+0000 7fb0d1991740 10 client.4398 resolve_mds: resolved name 'b' to daemon mds.b 2024-02-21T23:02:46.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.547+0000 7fb0d1991740 20 client.4398 populate_metadata read hostname 'smithi028' 2024-02-21T23:02:46.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.547+0000 7fb0d1991740 1 --2- 172.21.15.28:0/3406430472 >> [v2:172.21.15.81:6834/2328151245,v1:172.21.15.81:6837/2328151245] conn(0x560178d76510 0x560178d768f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:46.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.547+0000 7fb0d1991740 4 client.4398 mds_command: new command op to 4267 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-21T23:02:46.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.547+0000 7fb0d1991740 1 -- 172.21.15.28:0/3406430472 --> [v2:172.21.15.81:6834/2328151245,v1:172.21.15.81:6837/2328151245] -- command(tid 0: {"prefix": "status"}) v1 -- 0x560178cbdd00 con 0x560178d76510 2024-02-21T23:02:46.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.548+0000 7fb0c7e00640 1 --2- 172.21.15.28:0/3406430472 >> [v2:172.21.15.81:6834/2328151245,v1:172.21.15.81:6837/2328151245] conn(0x560178d76510 0x560178d768f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:46.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.549+0000 7fb0c7e00640 1 --2- 172.21.15.28:0/3406430472 >> [v2:172.21.15.81:6834/2328151245,v1:172.21.15.81:6837/2328151245] conn(0x560178d76510 0x560178d768f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e62331c888afe34 server_cookie=bcd8644ed1e43abc in_seq=0 out_seq=0 2024-02-21T23:02:46.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.549+0000 7fb0b4c00640 10 client.4398 ms_handle_connect on v2:172.21.15.81:6834/2328151245 2024-02-21T23:02:46.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.549+0000 7fb0b4c00640 1 -- 172.21.15.28:0/3406430472 <== mds.0 v2:172.21.15.81:6834/2328151245 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x560178cbdd00 con 0x560178d76510 2024-02-21T23:02:46.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.549+0000 7fb0b4c00640 10 client.4398 handle_command_reply: tid=0 2024-02-21T23:02:46.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.549+0000 7fb0d1991740 1 client.4398 shutdown 2024-02-21T23:02:46.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.549+0000 7fb0d1991740 1 -- 172.21.15.28:0/3406430472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560178d8e6b0 msgr2=0x560178d8fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:46.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.549+0000 7fb0d1991740 1 --2- 172.21.15.28:0/3406430472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560178d8e6b0 0x560178d8fb20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x560178d4e6f0 tx=0x7fb0b0012920 comp rx=0 tx=0).stop 2024-02-21T23:02:46.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.549+0000 7fb0d1991740 1 -- 172.21.15.28:0/3406430472 shutdown_connections 2024-02-21T23:02:46.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.549+0000 7fb0d1991740 1 --2- 172.21.15.28:0/3406430472 >> [v2:172.21.15.81:6834/2328151245,v1:172.21.15.81:6837/2328151245] conn(0x560178d76510 0x560178d768f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:46.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.549+0000 7fb0d1991740 1 --2- 172.21.15.28:0/3406430472 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560178e00630 0x560178dfa240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:46.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.549+0000 7fb0d1991740 1 --2- 172.21.15.28:0/3406430472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560178d8e6b0 0x560178d8fb20 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:46.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.549+0000 7fb0d1991740 1 --2- 172.21.15.28:0/3406430472 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x560178d72910 0x560178da5c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:46.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.549+0000 7fb0b4c00640 0 client.4398 ms_handle_reset on v2:172.21.15.81:6834/2328151245 2024-02-21T23:02:46.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.549+0000 7fb0d1991740 1 -- 172.21.15.28:0/3406430472 >> 172.21.15.28:0/3406430472 conn(0x560178d52020 msgr2=0x560178d75290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:46.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.550+0000 7fb0d1991740 1 -- 172.21.15.28:0/3406430472 shutdown_connections 2024-02-21T23:02:46.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.550+0000 7fb0d1991740 1 -- 172.21.15.28:0/3406430472 wait complete. 2024-02-21T23:02:46.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.550+0000 7fb0d1991740 20 client.4398 trim_cache size 0 max 16384 2024-02-21T23:02:46.545 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-21T23:02:46.545 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac", 2024-02-21T23:02:46.545 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-02-21T23:02:46.545 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4267, 2024-02-21T23:02:46.545 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-02-21T23:02:46.545 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-02-21T23:02:46.546 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-02-21T23:02:46.546 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 8.1742216079999999, 2024-02-21T23:02:46.546 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 7, 2024-02-21T23:02:46.546 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 24, 2024-02-21T23:02:46.546 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 24, 2024-02-21T23:02:46.546 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 16.003122886 2024-02-21T23:02:46.546 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-21T23:02:46.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.550+0000 7fb0cd400640 1 -- 172.21.15.28:0/3324840409 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb0a406b500 msgr2=0x7fb0a406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:46.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.550+0000 7fb0cd400640 1 --2- 172.21.15.28:0/3324840409 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb0a406b500 0x7fb0a406d9c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb0b8009990 tx=0x7fb0b8008040 comp rx=0 tx=0).stop 2024-02-21T23:02:46.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.550+0000 7fb0cd400640 1 -- 172.21.15.28:0/3324840409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0c814dcb0 msgr2=0x7fb0c8154e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:46.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.550+0000 7fb0cd400640 1 --2- 172.21.15.28:0/3324840409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0c814dcb0 0x7fb0c8154e00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb0c0037040 tx=0x7fb0c0009750 comp rx=0 tx=0).stop 2024-02-21T23:02:46.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.550+0000 7fb0cd400640 1 -- 172.21.15.28:0/3324840409 shutdown_connections 2024-02-21T23:02:46.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.550+0000 7fb0cd400640 1 --2- 172.21.15.28:0/3324840409 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb0a406b500 0x7fb0a406d9c0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:46.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.550+0000 7fb0cd400640 1 --2- 172.21.15.28:0/3324840409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0c814dcb0 0x7fb0c8154e00 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:46.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.550+0000 7fb0cd400640 1 --2- 172.21.15.28:0/3324840409 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0c80722a0 0x7fb0c814d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:46.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.550+0000 7fb0cd400640 1 --2- 172.21.15.28:0/3324840409 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb0ac004d40 0x7fb0c81eb2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:46.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.550+0000 7fb0cd400640 1 -- 172.21.15.28:0/3324840409 >> 172.21.15.28:0/3324840409 conn(0x7fb0c80a8320 msgr2=0x7fb0c80c0650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:46.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.551+0000 7fb0cd400640 1 -- 172.21.15.28:0/3324840409 shutdown_connections 2024-02-21T23:02:46.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.551+0000 7fb0cd400640 1 -- 172.21.15.28:0/3324840409 wait complete. 2024-02-21T23:02:46.557 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-02-21T23:02:46.557 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-02-21T23:02:46.557 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-02-21T23:02:46.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.784+0000 7fb4f8a00640 1 Processor -- start 2024-02-21T23:02:46.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.784+0000 7fb4f8a00640 1 -- start start 2024-02-21T23:02:46.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.786+0000 7fb4f8a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb4f40c6b60 0x7fb4f4158930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:46.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.786+0000 7fb4f8a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb4f40c6190 0x7fb4f4158e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:46.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.786+0000 7fb4f8a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4f41529e0 con 0x7fb4f4152600 2024-02-21T23:02:46.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.786+0000 7fb4f8a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4f40c7200 con 0x7fb4f40c6190 2024-02-21T23:02:46.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.786+0000 7fb4f8a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f40c74e0 con 0x7fb4f40c6b60 2024-02-21T23:02:46.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.787+0000 7fb4f3400640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fb4f4152600 0x7fb4f40c0f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.28:57554/0 (socket says 172.21.15.28:57554) 2024-02-21T23:02:46.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.787+0000 7fb4f3400640 1 -- 172.21.15.28:0/702359426 learned_addr learned my addr 172.21.15.28:0/702359426 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:02:46.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.787+0000 7fb4f3e00640 1 --2- 172.21.15.28:0/702359426 >> v2:172.21.15.28:3300/0 conn(0x7fb4f40c6190 0x7fb4f4158e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:46.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.787+0000 7fb4ebe00640 1 --2- 172.21.15.28:0/702359426 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb4f40c6b60 0x7fb4f4158930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:46.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.787+0000 7fb4f3e00640 1 -- 172.21.15.28:0/702359426 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb4f40c6b60 msgr2=0x7fb4f4158930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:46.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.787+0000 7fb4f3e00640 1 --2- 172.21.15.28:0/702359426 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb4f40c6b60 0x7fb4f4158930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:46.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.787+0000 7fb4f3e00640 1 -- 172.21.15.28:0/702359426 >> v1:172.21.15.81:6789/0 conn(0x7fb4f4152600 legacy=0x7fb4f40c0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:46.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.787+0000 7fb4f3e00640 1 -- 172.21.15.28:0/702359426 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4f40c77c0 con 0x7fb4f40c6190 2024-02-21T23:02:46.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.788+0000 7fb4f3e00640 1 --2- 172.21.15.28:0/702359426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4f40c6190 0x7fb4f4158e70 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb4ec009d90 tx=0x7fb4ec02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4a7ef02e7dd0a7d server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:46.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.789+0000 7fb4f2a00640 1 -- 172.21.15.28:0/702359426 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4ec039070 con 0x7fb4f40c6190 2024-02-21T23:02:46.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.789+0000 7fb4f2a00640 1 -- 172.21.15.28:0/702359426 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4ec02cc50 con 0x7fb4f40c6190 2024-02-21T23:02:46.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.789+0000 7fb4f2a00640 1 -- 172.21.15.28:0/702359426 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4ec034440 con 0x7fb4f40c6190 2024-02-21T23:02:46.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.789+0000 7fb4f8a00640 1 -- 172.21.15.28:0/702359426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4f40c6190 msgr2=0x7fb4f4158e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:46.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.789+0000 7fb4f8a00640 1 --2- 172.21.15.28:0/702359426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4f40c6190 0x7fb4f4158e70 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb4ec009d90 tx=0x7fb4ec02bea0 comp rx=0 tx=0).stop 2024-02-21T23:02:46.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.789+0000 7fb4f8a00640 1 -- 172.21.15.28:0/702359426 shutdown_connections 2024-02-21T23:02:46.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.789+0000 7fb4f8a00640 1 --2- 172.21.15.28:0/702359426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4f40c6190 0x7fb4f4158e70 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:46.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.789+0000 7fb4f8a00640 1 --2- 172.21.15.28:0/702359426 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb4f40c6b60 0x7fb4f4158930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:46.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.789+0000 7fb4f8a00640 1 -- 172.21.15.28:0/702359426 >> 172.21.15.28:0/702359426 conn(0x7fb4f406d1a0 msgr2=0x7fb4f40725a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:46.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.790+0000 7fb4f8a00640 1 -- 172.21.15.28:0/702359426 shutdown_connections 2024-02-21T23:02:46.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.790+0000 7fb4f8a00640 1 -- 172.21.15.28:0/702359426 wait complete. 2024-02-21T23:02:46.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.791+0000 7fb4f8a00640 1 Processor -- start 2024-02-21T23:02:46.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.791+0000 7fb4f8a00640 1 -- start start 2024-02-21T23:02:46.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.792+0000 7fb4f8a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb4f40c6190 0x7fb4f414a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:46.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.792+0000 7fb4f8a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb4f40c6b60 0x7fb4f4145130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:46.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.792+0000 7fb4f8a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4f4152600 0x7fb4f4145670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:46.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.792+0000 7fb4f8a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f40c1680 con 0x7fb4f4152600 2024-02-21T23:02:46.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.792+0000 7fb4f8a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f40c7aa0 con 0x7fb4f40c6190 2024-02-21T23:02:46.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.792+0000 7fb4f8a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f40c1800 con 0x7fb4f40c6b60 2024-02-21T23:02:46.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.793+0000 7fb4f3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4f4152600 0x7fb4f4145670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:46.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.793+0000 7fb4f3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4f4152600 0x7fb4f4145670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58126/0 (socket says 172.21.15.28:58126) 2024-02-21T23:02:46.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.793+0000 7fb4f3e00640 1 -- 172.21.15.28:0/3867562403 learned_addr learned my addr 172.21.15.28:0/3867562403 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:02:46.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.793+0000 7fb4f3400640 1 --2- 172.21.15.28:0/3867562403 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb4f40c6190 0x7fb4f414a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:46.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.793+0000 7fb4ebe00640 1 --2- 172.21.15.28:0/3867562403 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb4f40c6b60 0x7fb4f4145130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:46.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.793+0000 7fb4f3e00640 1 -- 172.21.15.28:0/3867562403 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb4f40c6b60 msgr2=0x7fb4f4145130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:46.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.793+0000 7fb4f3e00640 1 --2- 172.21.15.28:0/3867562403 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb4f40c6b60 0x7fb4f4145130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:46.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.793+0000 7fb4f3e00640 1 -- 172.21.15.28:0/3867562403 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb4f40c6190 msgr2=0x7fb4f414a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:46.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.793+0000 7fb4f3e00640 1 --2- 172.21.15.28:0/3867562403 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb4f40c6190 0x7fb4f414a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:46.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.793+0000 7fb4f3e00640 1 -- 172.21.15.28:0/3867562403 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4f40c9b70 con 0x7fb4f4152600 2024-02-21T23:02:46.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.793+0000 7fb4f3400640 1 --2- 172.21.15.28:0/3867562403 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb4f40c6190 0x7fb4f414a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:02:46.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.793+0000 7fb4f3e00640 1 --2- 172.21.15.28:0/3867562403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4f4152600 0x7fb4f4145670 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb4ec009d90 tx=0x7fb4ec004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:46.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.794+0000 7fb4eb400640 1 -- 172.21.15.28:0/3867562403 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4ec039050 con 0x7fb4f4152600 2024-02-21T23:02:46.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.794+0000 7fb4eb400640 1 -- 172.21.15.28:0/3867562403 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4ec045070 con 0x7fb4f4152600 2024-02-21T23:02:46.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.794+0000 7fb4eb400640 1 -- 172.21.15.28:0/3867562403 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4ec009450 con 0x7fb4f4152600 2024-02-21T23:02:46.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.794+0000 7fb4f8a00640 1 -- 172.21.15.28:0/3867562403 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4f41565f0 con 0x7fb4f4152600 2024-02-21T23:02:46.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.794+0000 7fb4f8a00640 1 -- 172.21.15.28:0/3867562403 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4f40ca0c0 con 0x7fb4f4152600 2024-02-21T23:02:46.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.795+0000 7fb4f8a00640 1 -- 172.21.15.28:0/3867562403 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4f404f290 con 0x7fb4f4152600 2024-02-21T23:02:46.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.796+0000 7fb4eb400640 1 -- 172.21.15.28:0/3867562403 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb4ec03e610 con 0x7fb4f4152600 2024-02-21T23:02:46.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.797+0000 7fb4eb400640 1 --2- 172.21.15.28:0/3867562403 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb4d006b4d0 0x7fb4d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:46.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.797+0000 7fb4eb400640 1 -- 172.21.15.28:0/3867562403 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fb4ec0acfe0 con 0x7fb4f4152600 2024-02-21T23:02:46.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.797+0000 7fb4f3400640 1 --2- 172.21.15.28:0/3867562403 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb4d006b4d0 0x7fb4d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:46.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.798+0000 7fb4f3400640 1 --2- 172.21.15.28:0/3867562403 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb4d006b4d0 0x7fb4d006d990 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb4e4005880 tx=0x7fb4e4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:46.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:46.802+0000 7fb4eb400640 1 -- 172.21.15.28:0/3867562403 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb4ec07bb30 con 0x7fb4f4152600 2024-02-21T23:02:47.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:47.044+0000 7fb4f8a00640 1 -- 172.21.15.28:0/3867562403 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fb4f40c6630 con 0x7fb4f4152600 2024-02-21T23:02:47.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:47.045+0000 7fb4eb400640 1 -- 172.21.15.28:0/3867562403 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) v1 ==== 53+0+88 (secure 0 0 0) 0x7fb4ec07b430 con 0x7fb4f4152600 2024-02-21T23:02:47.035 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-02-21T23:02:47.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:47.048+0000 7fb4f8a00640 1 -- 172.21.15.28:0/3867562403 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb4d006b4d0 msgr2=0x7fb4d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:47.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:47.048+0000 7fb4f8a00640 1 --2- 172.21.15.28:0/3867562403 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb4d006b4d0 0x7fb4d006d990 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb4e4005880 tx=0x7fb4e4009040 comp rx=0 tx=0).stop 2024-02-21T23:02:47.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:47.048+0000 7fb4f8a00640 1 -- 172.21.15.28:0/3867562403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4f4152600 msgr2=0x7fb4f4145670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:47.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:47.048+0000 7fb4f8a00640 1 --2- 172.21.15.28:0/3867562403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4f4152600 0x7fb4f4145670 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb4ec009d90 tx=0x7fb4ec004a40 comp rx=0 tx=0).stop 2024-02-21T23:02:47.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:47.048+0000 7fb4f8a00640 1 -- 172.21.15.28:0/3867562403 shutdown_connections 2024-02-21T23:02:47.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:47.048+0000 7fb4f8a00640 1 --2- 172.21.15.28:0/3867562403 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb4d006b4d0 0x7fb4d006d990 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:47.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:47.048+0000 7fb4f8a00640 1 --2- 172.21.15.28:0/3867562403 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb4f40c6b60 0x7fb4f4145130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:47.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:47.048+0000 7fb4f8a00640 1 --2- 172.21.15.28:0/3867562403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4f4152600 0x7fb4f4145670 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:47.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:47.048+0000 7fb4f8a00640 1 --2- 172.21.15.28:0/3867562403 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb4f40c6190 0x7fb4f414a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:47.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:47.048+0000 7fb4f8a00640 1 -- 172.21.15.28:0/3867562403 >> 172.21.15.28:0/3867562403 conn(0x7fb4f406d1a0 msgr2=0x7fb4f40c0f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:47.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:47.049+0000 7fb4f8a00640 1 -- 172.21.15.28:0/3867562403 shutdown_connections 2024-02-21T23:02:47.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:47.049+0000 7fb4f8a00640 1 -- 172.21.15.28:0/3867562403 wait complete. 2024-02-21T23:02:47.058 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-02-21T23:02:47.058 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-02-21T23:02:47.058 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi028.front.sepia.ceph.com 2024-02-21T23:02:47.059 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-02-21T23:02:47.059 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-02-21T23:02:47.059 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-02-21T23:02:47.059 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-02-21T23:02:47.059 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-02-21T23:02:47.059 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-02-21T23:02:47.059 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:02:47.059 DEBUG:teuthology.orchestra.run.smithi028:> ip addr 2024-02-21T23:02:47.116 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-02-21T23:02:47.116 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-02-21T23:02:47.116 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-02-21T23:02:47.116 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-02-21T23:02:47.116 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-02-21T23:02:47.117 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-02-21T23:02:47.117 INFO:teuthology.orchestra.run.smithi028.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-21T23:02:47.117 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:16 brd ff:ff:ff:ff:ff:ff 2024-02-21T23:02:47.117 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f0 2024-02-21T23:02:47.117 INFO:teuthology.orchestra.run.smithi028.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-21T23:02:47.117 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:17 brd ff:ff:ff:ff:ff:ff 2024-02-21T23:02:47.117 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f1 2024-02-21T23:02:47.117 INFO:teuthology.orchestra.run.smithi028.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-02-21T23:02:47.117 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3c brd ff:ff:ff:ff:ff:ff 2024-02-21T23:02:47.117 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f0 2024-02-21T23:02:47.118 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.21.15.28/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-02-21T23:02:47.118 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft 42543sec preferred_lft 42543sec 2024-02-21T23:02:47.118 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link noprefixroute 2024-02-21T23:02:47.118 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-02-21T23:02:47.118 INFO:teuthology.orchestra.run.smithi028.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-21T23:02:47.118 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3d brd ff:ff:ff:ff:ff:ff 2024-02-21T23:02:47.118 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f1 2024-02-21T23:02:47.120 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-02-21T23:02:47.120 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:02:47.120 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec bash -c ' 2024-02-21T23:02:47.120 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-02-21T23:02:47.120 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link add name ceph-brx type bridge 2024-02-21T23:02:47.120 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip addr flush dev ceph-brx 2024-02-21T23:02:47.120 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set ceph-brx up 2024-02-21T23:02:47.120 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-02-21T23:02:47.121 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-02-21T23:02:47.571 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:02:47.571 DEBUG:teuthology.orchestra.run.smithi028:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-02-21T23:02:47.660 INFO:teuthology.orchestra.run.smithi028.stdout:1 2024-02-21T23:02:47.664 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:02:47.664 DEBUG:teuthology.orchestra.run.smithi028:> route 2024-02-21T23:02:47.723 INFO:teuthology.orchestra.run.smithi028.stdout:Kernel IP routing table 2024-02-21T23:02:47.723 INFO:teuthology.orchestra.run.smithi028.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-02-21T23:02:47.723 INFO:teuthology.orchestra.run.smithi028.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-02-21T23:02:47.724 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-02-21T23:02:47.724 INFO:teuthology.orchestra.run.smithi028.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-02-21T23:02:47.725 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:02:47.725 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec bash -c ' 2024-02-21T23:02:47.725 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-02-21T23:02:47.725 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-02-21T23:02:47.725 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-02-21T23:02:47.725 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-02-21T23:02:47.725 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-02-21T23:02:48.467 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:02:48.467 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-02-21T23:02:48.527 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:02:48.528 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list-id 2024-02-21T23:02:48.583 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:02:48.583 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec bash -c ' 2024-02-21T23:02:48.583 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-02-21T23:02:48.584 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-02-21T23:02:48.584 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-02-21T23:02:48.584 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-02-21T23:02:48.735 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-02-21T23:02:48.735 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:02:48.735 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec bash -c ' 2024-02-21T23:02:48.735 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-02-21T23:02:48.735 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-02-21T23:02:48.735 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-02-21T23:02:48.735 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-02-21T23:02:48.736 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-02-21T23:02:48.736 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-02-21T23:02:48.736 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-02-21T23:02:49.048 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:02:49.048 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec bash -c ' 2024-02-21T23:02:49.048 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-02-21T23:02:49.048 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set brx.0 up 2024-02-21T23:02:49.048 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set dev brx.0 master ceph-brx 2024-02-21T23:02:49.048 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-02-21T23:02:49.223 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-02-21T23:02:49.223 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-21T23:02:49.223 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-02-21T23:02:49.266 INFO:teuthology.orchestra.run.smithi028.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-02-21T23:02:49.267 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-21T23:02:49.267 DEBUG:teuthology.orchestra.run.smithi028:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-02-21T23:02:49.362 DEBUG:teuthology.orchestra.run.smithi028:> sudo modprobe fuse 2024-02-21T23:02:49.464 DEBUG:teuthology.orchestra.run.smithi028:> cat /proc/self/mounts | awk '{print $2}' 2024-02-21T23:02:49.519 INFO:teuthology.orchestra.run.smithi028.stdout:/proc 2024-02-21T23:02:49.519 INFO:teuthology.orchestra.run.smithi028.stdout:/sys 2024-02-21T23:02:49.519 INFO:teuthology.orchestra.run.smithi028.stdout:/dev 2024-02-21T23:02:49.519 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/security 2024-02-21T23:02:49.519 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/shm 2024-02-21T23:02:49.519 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/pts 2024-02-21T23:02:49.519 INFO:teuthology.orchestra.run.smithi028.stdout:/run 2024-02-21T23:02:49.519 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/cgroup 2024-02-21T23:02:49.520 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/pstore 2024-02-21T23:02:49.520 INFO:teuthology.orchestra.run.smithi028.stdout:/ 2024-02-21T23:02:49.520 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/selinux 2024-02-21T23:02:49.520 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/sys/fs/binfmt_misc 2024-02-21T23:02:49.520 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/hugepages 2024-02-21T23:02:49.520 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/mqueue 2024-02-21T23:02:49.520 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/debug 2024-02-21T23:02:49.520 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/tracing 2024-02-21T23:02:49.520 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-02-21T23:02:49.520 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/fuse/connections 2024-02-21T23:02:49.520 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-sysctl.service 2024-02-21T23:02:49.520 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/config 2024-02-21T23:02:49.520 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-02-21T23:02:49.521 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/nfs/rpc_pipefs 2024-02-21T23:02:49.521 INFO:teuthology.orchestra.run.smithi028.stdout:/run/user/1000 2024-02-21T23:02:49.521 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph 2024-02-21T23:02:49.521 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-0 2024-02-21T23:02:49.521 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-1 2024-02-21T23:02:49.521 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-2 2024-02-21T23:02:49.521 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-3 2024-02-21T23:02:49.521 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns 2024-02-21T23:02:49.521 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-02-21T23:02:49.521 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-02-21T23:02:49.522 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:02:49.522 DEBUG:teuthology.orchestra.run.smithi028:> ls /sys/fs/fuse/connections 2024-02-21T23:02:49.575 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-02-21T23:02:49.575 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-02-21T23:02:49.580 DEBUG:teuthology.orchestra.run.smithi028:> sudo modprobe fuse 2024-02-21T23:02:49.657 DEBUG:teuthology.orchestra.run.smithi028:> cat /proc/self/mounts | awk '{print $2}' 2024-02-21T23:02:49.701 INFO:teuthology.orchestra.run.smithi028.stdout:/proc 2024-02-21T23:02:49.701 INFO:teuthology.orchestra.run.smithi028.stdout:/sys 2024-02-21T23:02:49.701 INFO:teuthology.orchestra.run.smithi028.stdout:/dev 2024-02-21T23:02:49.701 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/security 2024-02-21T23:02:49.701 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/shm 2024-02-21T23:02:49.701 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/pts 2024-02-21T23:02:49.702 INFO:teuthology.orchestra.run.smithi028.stdout:/run 2024-02-21T23:02:49.702 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/cgroup 2024-02-21T23:02:49.702 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/pstore 2024-02-21T23:02:49.702 INFO:teuthology.orchestra.run.smithi028.stdout:/ 2024-02-21T23:02:49.702 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/selinux 2024-02-21T23:02:49.702 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/sys/fs/binfmt_misc 2024-02-21T23:02:49.702 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/hugepages 2024-02-21T23:02:49.702 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/mqueue 2024-02-21T23:02:49.702 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/debug 2024-02-21T23:02:49.702 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/tracing 2024-02-21T23:02:49.702 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-02-21T23:02:49.702 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/fuse/connections 2024-02-21T23:02:49.703 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-sysctl.service 2024-02-21T23:02:49.703 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/config 2024-02-21T23:02:49.703 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-02-21T23:02:49.703 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/nfs/rpc_pipefs 2024-02-21T23:02:49.703 INFO:teuthology.orchestra.run.smithi028.stdout:/run/user/1000 2024-02-21T23:02:49.703 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph 2024-02-21T23:02:49.703 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-0 2024-02-21T23:02:49.703 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-1 2024-02-21T23:02:49.703 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-2 2024-02-21T23:02:49.703 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-3 2024-02-21T23:02:49.703 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns 2024-02-21T23:02:49.703 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-02-21T23:02:49.703 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-02-21T23:02:49.704 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:02:49.704 DEBUG:teuthology.orchestra.run.smithi028:> ls /sys/fs/fuse/connections 2024-02-21T23:02:49.810 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi028.stderr:2024-02-21T23:02:49.821+0000 7fab45866480 -1 init, newargv = 0x5582d1efe430 newargc=15 2024-02-21T23:02:49.811 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi028.stderr:ceph-fuse[42191]: starting ceph client 2024-02-21T23:02:49.823 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi028.stderr:ceph-fuse[42191]: starting fuse 2024-02-21T23:02:50.894 DEBUG:teuthology.orchestra.run.smithi028:> sudo modprobe fuse 2024-02-21T23:02:50.991 DEBUG:teuthology.orchestra.run.smithi028:> cat /proc/self/mounts | awk '{print $2}' 2024-02-21T23:02:51.051 INFO:teuthology.orchestra.run.smithi028.stdout:/proc 2024-02-21T23:02:51.051 INFO:teuthology.orchestra.run.smithi028.stdout:/sys 2024-02-21T23:02:51.051 INFO:teuthology.orchestra.run.smithi028.stdout:/dev 2024-02-21T23:02:51.052 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/security 2024-02-21T23:02:51.052 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/shm 2024-02-21T23:02:51.052 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/pts 2024-02-21T23:02:51.052 INFO:teuthology.orchestra.run.smithi028.stdout:/run 2024-02-21T23:02:51.052 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/cgroup 2024-02-21T23:02:51.052 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/pstore 2024-02-21T23:02:51.052 INFO:teuthology.orchestra.run.smithi028.stdout:/ 2024-02-21T23:02:51.052 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/selinux 2024-02-21T23:02:51.052 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/sys/fs/binfmt_misc 2024-02-21T23:02:51.052 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/hugepages 2024-02-21T23:02:51.052 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/mqueue 2024-02-21T23:02:51.052 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/debug 2024-02-21T23:02:51.052 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/tracing 2024-02-21T23:02:51.052 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-02-21T23:02:51.053 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/fuse/connections 2024-02-21T23:02:51.053 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-sysctl.service 2024-02-21T23:02:51.053 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/config 2024-02-21T23:02:51.053 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-02-21T23:02:51.053 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/nfs/rpc_pipefs 2024-02-21T23:02:51.053 INFO:teuthology.orchestra.run.smithi028.stdout:/run/user/1000 2024-02-21T23:02:51.053 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph 2024-02-21T23:02:51.053 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-0 2024-02-21T23:02:51.053 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-1 2024-02-21T23:02:51.053 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-2 2024-02-21T23:02:51.053 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-3 2024-02-21T23:02:51.053 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns 2024-02-21T23:02:51.053 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-02-21T23:02:51.053 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-02-21T23:02:51.053 INFO:teuthology.orchestra.run.smithi028.stdout:/home/ubuntu/cephtest/mnt.admin 2024-02-21T23:02:51.054 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:02:51.054 DEBUG:teuthology.orchestra.run.smithi028:> ls /sys/fs/fuse/connections 2024-02-21T23:02:51.110 INFO:teuthology.orchestra.run.smithi028.stdout:40 2024-02-21T23:02:51.110 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-02-21T23:02:51.111 DEBUG:teuthology.orchestra.run.smithi028:> sudo stdin-killer -- python3 -c ' 2024-02-21T23:02:51.111 DEBUG:teuthology.orchestra.run.smithi028:> import glob 2024-02-21T23:02:51.111 DEBUG:teuthology.orchestra.run.smithi028:> import re 2024-02-21T23:02:51.111 DEBUG:teuthology.orchestra.run.smithi028:> import os 2024-02-21T23:02:51.111 DEBUG:teuthology.orchestra.run.smithi028:> import subprocess 2024-02-21T23:02:51.111 DEBUG:teuthology.orchestra.run.smithi028:> 2024-02-21T23:02:51.111 DEBUG:teuthology.orchestra.run.smithi028:> def _find_admin_socket(client_name): 2024-02-21T23:02:51.111 DEBUG:teuthology.orchestra.run.smithi028:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-02-21T23:02:51.111 DEBUG:teuthology.orchestra.run.smithi028:> files = glob.glob(asok_path) 2024-02-21T23:02:51.111 DEBUG:teuthology.orchestra.run.smithi028:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-02-21T23:02:51.111 DEBUG:teuthology.orchestra.run.smithi028:> 2024-02-21T23:02:51.112 DEBUG:teuthology.orchestra.run.smithi028:> # Given a non-glob path, it better be there 2024-02-21T23:02:51.112 DEBUG:teuthology.orchestra.run.smithi028:> if "*" not in asok_path: 2024-02-21T23:02:51.112 DEBUG:teuthology.orchestra.run.smithi028:> assert(len(files) == 1) 2024-02-21T23:02:51.112 DEBUG:teuthology.orchestra.run.smithi028:> return files[0] 2024-02-21T23:02:51.112 DEBUG:teuthology.orchestra.run.smithi028:> 2024-02-21T23:02:51.112 DEBUG:teuthology.orchestra.run.smithi028:> for f in files: 2024-02-21T23:02:51.112 DEBUG:teuthology.orchestra.run.smithi028:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-02-21T23:02:51.112 DEBUG:teuthology.orchestra.run.smithi028:> if os.path.exists("/proc/{0}".format(pid)): 2024-02-21T23:02:51.112 DEBUG:teuthology.orchestra.run.smithi028:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-02-21T23:02:51.112 DEBUG:teuthology.orchestra.run.smithi028:> contents = proc_f.read() 2024-02-21T23:02:51.112 DEBUG:teuthology.orchestra.run.smithi028:> if mountpoint in contents: 2024-02-21T23:02:51.112 DEBUG:teuthology.orchestra.run.smithi028:> return f 2024-02-21T23:02:51.112 DEBUG:teuthology.orchestra.run.smithi028:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-02-21T23:02:51.112 DEBUG:teuthology.orchestra.run.smithi028:> 2024-02-21T23:02:51.112 DEBUG:teuthology.orchestra.run.smithi028:> print(_find_admin_socket("client.admin")) 2024-02-21T23:02:51.113 DEBUG:teuthology.orchestra.run.smithi028:> ' 2024-02-21T23:02:51.302 INFO:teuthology.orchestra.run.smithi028.stdout:/var/run/ceph/ceph-client.admin.42191.asok 2024-02-21T23:02:51.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-02-21T23:02:51.321 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.42191.asok 2024-02-21T23:02:51.321 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:02:51.321 DEBUG:teuthology.orchestra.run.smithi028:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.42191.asok status 2024-02-21T23:02:51.570 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-21T23:02:51.570 INFO:teuthology.orchestra.run.smithi028.stdout: "metadata": { 2024-02-21T23:02:51.570 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_sha1": "3bfc837d7b45ce8e095d92fa9c342964cea5e95f", 2024-02-21T23:02:51.570 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_version": "ceph version 19.0.0-1643-g3bfc837d (3bfc837d7b45ce8e095d92fa9c342964cea5e95f) squid (dev)", 2024-02-21T23:02:51.570 INFO:teuthology.orchestra.run.smithi028.stdout: "entity_id": "admin", 2024-02-21T23:02:51.570 INFO:teuthology.orchestra.run.smithi028.stdout: "hostname": "smithi028", 2024-02-21T23:02:51.570 INFO:teuthology.orchestra.run.smithi028.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-02-21T23:02:51.570 INFO:teuthology.orchestra.run.smithi028.stdout: "pid": "42191", 2024-02-21T23:02:51.570 INFO:teuthology.orchestra.run.smithi028.stdout: "root": "/" 2024-02-21T23:02:51.571 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:02:51.571 INFO:teuthology.orchestra.run.smithi028.stdout: "dentry_count": 0, 2024-02-21T23:02:51.571 INFO:teuthology.orchestra.run.smithi028.stdout: "dentry_pinned_count": 0, 2024-02-21T23:02:51.571 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4413, 2024-02-21T23:02:51.571 INFO:teuthology.orchestra.run.smithi028.stdout: "inst": { 2024-02-21T23:02:51.571 INFO:teuthology.orchestra.run.smithi028.stdout: "name": { 2024-02-21T23:02:51.571 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "client", 2024-02-21T23:02:51.571 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 4413 2024-02-21T23:02:51.571 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:02:51.571 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": { 2024-02-21T23:02:51.571 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-21T23:02:51.571 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "192.168.0.1:0", 2024-02-21T23:02:51.571 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2469498112 2024-02-21T23:02:51.571 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:02:51.572 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:02:51.572 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": { 2024-02-21T23:02:51.572 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-21T23:02:51.572 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "192.168.0.1:0", 2024-02-21T23:02:51.572 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2469498112 2024-02-21T23:02:51.572 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:02:51.572 INFO:teuthology.orchestra.run.smithi028.stdout: "inst_str": "client.4413 192.168.0.1:0/2469498112", 2024-02-21T23:02:51.572 INFO:teuthology.orchestra.run.smithi028.stdout: "addr_str": "192.168.0.1:0/2469498112", 2024-02-21T23:02:51.572 INFO:teuthology.orchestra.run.smithi028.stdout: "inode_count": 1, 2024-02-21T23:02:51.572 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_epoch": 7, 2024-02-21T23:02:51.572 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_epoch": 24, 2024-02-21T23:02:51.572 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_epoch_barrier": 0, 2024-02-21T23:02:51.572 INFO:teuthology.orchestra.run.smithi028.stdout: "blocklisted": false, 2024-02-21T23:02:51.572 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs" 2024-02-21T23:02:51.573 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-21T23:02:51.586 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:02:51.586 DEBUG:teuthology.orchestra.run.smithi028:> stat --file-system '--printf=%T 2024-02-21T23:02:51.586 DEBUG:teuthology.orchestra.run.smithi028:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-02-21T23:02:51.642 INFO:teuthology.orchestra.run.smithi028.stdout:fuseblk 2024-02-21T23:02:51.642 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-02-21T23:02:51.642 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:02:51.642 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-02-21T23:02:51.741 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:02:51.741 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.admin && exec bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-02-21T23:02:51.857 INFO:teuthology.orchestra.run.smithi028.stdout:# file: . 2024-02-21T23:02:51.857 INFO:teuthology.orchestra.run.smithi028.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-02-21T23:02:51.857 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:02:51.859 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-21T23:02:51.859 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-21T23:02:51.913 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:02:52.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.141+0000 7f9bc7200640 1 Processor -- start 2024-02-21T23:02:52.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.141+0000 7f9bc7200640 1 -- start start 2024-02-21T23:02:52.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.142+0000 7f9bc7200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f9bc8150450 0x7f9bc80a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:52.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.143+0000 7f9bc7200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9bc806ee40 0x7f9bc81536f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:52.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.143+0000 7f9bc7200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bc80c2760 con 0x7f9bc8156af0 2024-02-21T23:02:52.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.143+0000 7f9bc7200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bc80c22c0 con 0x7f9bc8150450 2024-02-21T23:02:52.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.143+0000 7f9bc7200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc8156660 con 0x7f9bc806ee40 2024-02-21T23:02:52.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.143+0000 7f9bc6800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f9bc8156af0 0x7f9bc806dff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42370/0 (socket says 172.21.15.28:42370) 2024-02-21T23:02:52.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.143+0000 7f9bc6800640 1 -- 172.21.15.28:0/1374824167 learned_addr learned my addr 172.21.15.28:0/1374824167 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:02:52.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.143+0000 7f9bc5400640 1 --2- 172.21.15.28:0/1374824167 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9bc806ee40 0x7f9bc81536f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:52.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.143+0000 7f9bc5e00640 1 --2- 172.21.15.28:0/1374824167 >> v2:172.21.15.81:3300/0 conn(0x7f9bc8150450 0x7f9bc80a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:52.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.143+0000 7f9bc4a00640 1 -- 172.21.15.28:0/1374824167 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4129554531 0 0) 0x7f9bc80c2760 con 0x7f9bc8156af0 2024-02-21T23:02:52.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.143+0000 7f9bc4a00640 1 -- 172.21.15.28:0/1374824167 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bac003520 con 0x7f9bc8156af0 2024-02-21T23:02:52.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.144+0000 7f9bc4a00640 1 -- 172.21.15.28:0/1374824167 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 302807405 0 0) 0x7f9bac003520 con 0x7f9bc8156af0 2024-02-21T23:02:52.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.144+0000 7f9bc4a00640 1 -- 172.21.15.28:0/1374824167 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9bc806ee40 msgr2=0x7f9bc81536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:52.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.144+0000 7f9bc4a00640 1 --2- 172.21.15.28:0/1374824167 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9bc806ee40 0x7f9bc81536f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:52.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.144+0000 7f9bc4a00640 1 -- 172.21.15.28:0/1374824167 >> v2:172.21.15.81:3300/0 conn(0x7f9bc8150450 msgr2=0x7f9bc80a5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:52.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.144+0000 7f9bc4a00640 1 --2- 172.21.15.28:0/1374824167 >> v2:172.21.15.81:3300/0 conn(0x7f9bc8150450 0x7f9bc80a5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:52.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.144+0000 7f9bc4a00640 1 -- 172.21.15.28:0/1374824167 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bc8153c30 con 0x7f9bc8156af0 2024-02-21T23:02:52.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.144+0000 7f9bc4a00640 1 -- 172.21.15.28:0/1374824167 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f9bc00019f0 con 0x7f9bc8156af0 2024-02-21T23:02:52.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.144+0000 7f9bc4a00640 1 -- 172.21.15.28:0/1374824167 >> v1:172.21.15.28:6789/0 conn(0x7f9bc8156af0 legacy=0x7f9bc806dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:52.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.144+0000 7f9bc4a00640 1 --2- 172.21.15.28:0/1374824167 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9bac0039b0 0x7f9bac003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:52.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.144+0000 7f9bc4a00640 1 --2- 172.21.15.28:0/1374824167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bac004800 0x7f9bac004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:52.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.144+0000 7f9bc4a00640 1 --2- 172.21.15.28:0/1374824167 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9bc806ee40 0x7f9bac0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:52.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.144+0000 7f9bc4a00640 1 -- 172.21.15.28:0/1374824167 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc8156660 con 0x7f9bac004800 2024-02-21T23:02:52.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.144+0000 7f9bc4a00640 1 -- 172.21.15.28:0/1374824167 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bac00b9c0 con 0x7f9bac0039b0 2024-02-21T23:02:52.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.144+0000 7f9bc4a00640 1 -- 172.21.15.28:0/1374824167 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bac00bb40 con 0x7f9bc806ee40 2024-02-21T23:02:52.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.144+0000 7f9bc5400640 1 --2- 172.21.15.28:0/1374824167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bac004800 0x7f9bac004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:52.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.145+0000 7f9bc5e00640 1 --2- 172.21.15.28:0/1374824167 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9bac0039b0 0x7f9bac003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:52.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.145+0000 7f9bc6800640 1 --2- 172.21.15.28:0/1374824167 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9bc806ee40 0x7f9bac0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:52.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.145+0000 7f9bc5400640 1 -- 172.21.15.28:0/1374824167 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9bc806ee40 msgr2=0x7f9bac0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:52.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.145+0000 7f9bc5400640 1 --2- 172.21.15.28:0/1374824167 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9bc806ee40 0x7f9bac0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:52.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.145+0000 7f9bc5400640 1 -- 172.21.15.28:0/1374824167 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9bac0039b0 msgr2=0x7f9bac003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:52.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.145+0000 7f9bc5400640 1 --2- 172.21.15.28:0/1374824167 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9bac0039b0 0x7f9bac003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:52.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.145+0000 7f9bc5400640 1 -- 172.21.15.28:0/1374824167 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bac003520 con 0x7f9bac004800 2024-02-21T23:02:52.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.145+0000 7f9bc6800640 1 --2- 172.21.15.28:0/1374824167 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9bc806ee40 0x7f9bac0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:02:52.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.146+0000 7f9bc5400640 1 --2- 172.21.15.28:0/1374824167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bac004800 0x7f9bac004c80 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f9bb4009ac0 tx=0x7f9bac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63e423f41e59ef3d server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:52.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.146+0000 7f9bc4a00640 1 -- 172.21.15.28:0/1374824167 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb4039070 con 0x7f9bac004800 2024-02-21T23:02:52.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.146+0000 7f9bc4a00640 1 -- 172.21.15.28:0/1374824167 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bb402c6b0 con 0x7f9bac004800 2024-02-21T23:02:52.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.146+0000 7f9bc4a00640 1 -- 172.21.15.28:0/1374824167 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb4009690 con 0x7f9bac004800 2024-02-21T23:02:52.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.147+0000 7f9bc7200640 1 -- 172.21.15.28:0/1374824167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bac004800 msgr2=0x7f9bac004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:52.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.147+0000 7f9bc7200640 1 --2- 172.21.15.28:0/1374824167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bac004800 0x7f9bac004c80 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f9bb4009ac0 tx=0x7f9bac005830 comp rx=0 tx=0).stop 2024-02-21T23:02:52.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.147+0000 7f9bc6800640 1 -- 172.21.15.28:0/1374824167 reap_dead start 2024-02-21T23:02:52.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.147+0000 7f9bc7200640 1 -- 172.21.15.28:0/1374824167 shutdown_connections 2024-02-21T23:02:52.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.147+0000 7f9bc7200640 1 -- 172.21.15.28:0/1374824167 >> 172.21.15.28:0/1374824167 conn(0x7f9bc806b630 msgr2=0x7f9bc80c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:52.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.147+0000 7f9bc7200640 1 -- 172.21.15.28:0/1374824167 shutdown_connections 2024-02-21T23:02:52.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.147+0000 7f9bc7200640 1 -- 172.21.15.28:0/1374824167 wait complete. 2024-02-21T23:02:52.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.149+0000 7f9bc7200640 1 Processor -- start 2024-02-21T23:02:52.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.149+0000 7f9bc7200640 1 -- start start 2024-02-21T23:02:52.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.150+0000 7f9bc7200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9bac004800 0x7f9bc81550c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:52.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.150+0000 7f9bc7200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9bc8150b00 0x7f9bc8147270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:52.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.150+0000 7f9bc7200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bc8147ca0 0x7f9bc81ead20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:52.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.150+0000 7f9bc7200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc80c2cb0 con 0x7f9bc8147ca0 2024-02-21T23:02:52.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.150+0000 7f9bc7200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc80c29d0 con 0x7f9bac004800 2024-02-21T23:02:52.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.150+0000 7f9bc7200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc806e700 con 0x7f9bc8150b00 2024-02-21T23:02:52.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.150+0000 7f9bc6800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bc8147ca0 0x7f9bc81ead20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:52.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.150+0000 7f9bc6800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bc8147ca0 0x7f9bc81ead20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58140/0 (socket says 172.21.15.28:58140) 2024-02-21T23:02:52.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.150+0000 7f9bc6800640 1 -- 172.21.15.28:0/3961428691 learned_addr learned my addr 172.21.15.28:0/3961428691 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:02:52.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.150+0000 7f9bc5e00640 1 --2- 172.21.15.28:0/3961428691 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9bac004800 0x7f9bc81550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:52.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.150+0000 7f9bc5400640 1 --2- 172.21.15.28:0/3961428691 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9bc8150b00 0x7f9bc8147270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:52.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.151+0000 7f9bc6800640 1 -- 172.21.15.28:0/3961428691 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9bc8150b00 msgr2=0x7f9bc8147270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:52.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.151+0000 7f9bc6800640 1 --2- 172.21.15.28:0/3961428691 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9bc8150b00 0x7f9bc8147270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:52.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.151+0000 7f9bc6800640 1 -- 172.21.15.28:0/3961428691 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9bac004800 msgr2=0x7f9bc81550c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:52.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.151+0000 7f9bc6800640 1 --2- 172.21.15.28:0/3961428691 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9bac004800 0x7f9bc81550c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:52.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.151+0000 7f9bc6800640 1 -- 172.21.15.28:0/3961428691 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bc80ac8b0 con 0x7f9bc8147ca0 2024-02-21T23:02:52.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.151+0000 7f9bc5400640 1 --2- 172.21.15.28:0/3961428691 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9bc8150b00 0x7f9bc8147270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:02:52.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.151+0000 7f9bc6800640 1 --2- 172.21.15.28:0/3961428691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bc8147ca0 0x7f9bc81ead20 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f9bc80c7470 tx=0x7f9bc000b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:52.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.151+0000 7f9bbe000640 1 -- 172.21.15.28:0/3961428691 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc0004d60 con 0x7f9bc8147ca0 2024-02-21T23:02:52.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.151+0000 7f9bbe000640 1 -- 172.21.15.28:0/3961428691 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bc000bb80 con 0x7f9bc8147ca0 2024-02-21T23:02:52.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.151+0000 7f9bc7200640 1 -- 172.21.15.28:0/3961428691 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bc814a940 con 0x7f9bc8147ca0 2024-02-21T23:02:52.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.152+0000 7f9bbe000640 1 -- 172.21.15.28:0/3961428691 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc0007110 con 0x7f9bc8147ca0 2024-02-21T23:02:52.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.152+0000 7f9bc7200640 1 -- 172.21.15.28:0/3961428691 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bc81eeb80 con 0x7f9bc8147ca0 2024-02-21T23:02:52.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.153+0000 7f9bc7200640 1 -- 172.21.15.28:0/3961428691 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bc814ffb0 con 0x7f9bc8147ca0 2024-02-21T23:02:52.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.154+0000 7f9bbe000640 1 -- 172.21.15.28:0/3961428691 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9bc001b020 con 0x7f9bc8147ca0 2024-02-21T23:02:52.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.155+0000 7f9bbe000640 1 --2- 172.21.15.28:0/3961428691 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9ba806b500 0x7f9ba806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:52.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.155+0000 7f9bbe000640 1 -- 172.21.15.28:0/3961428691 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f9bc008a690 con 0x7f9bc8147ca0 2024-02-21T23:02:52.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.155+0000 7f9bc5e00640 1 --2- 172.21.15.28:0/3961428691 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9ba806b500 0x7f9ba806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:52.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.156+0000 7f9bc5e00640 1 --2- 172.21.15.28:0/3961428691 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9ba806b500 0x7f9ba806d9c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9bb8009b40 tx=0x7f9bb8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:52.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.160+0000 7f9bbe000640 1 -- 172.21.15.28:0/3961428691 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f9bc814ffb0 con 0x7f9bc8147ca0 2024-02-21T23:02:52.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.361+0000 7f9bc7200640 1 -- 172.21.15.28:0/3961428691 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9bc804f290 con 0x7f9bc8147ca0 2024-02-21T23:02:52.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.363+0000 7f9bbe000640 1 -- 172.21.15.28:0/3961428691 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18454 (secure 0 0 0) 0x7f9bc00591c0 con 0x7f9bc8147ca0 2024-02-21T23:02:52.352 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:02:52.352 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":24,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:02:42.900286+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:02:35.795158+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-21T23:02:36.840110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-02-21T23:02:40.441388+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:02:52.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.366+0000 7f9bc7200640 1 -- 172.21.15.28:0/3961428691 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9ba806b500 msgr2=0x7f9ba806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:52.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.366+0000 7f9bc7200640 1 --2- 172.21.15.28:0/3961428691 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9ba806b500 0x7f9ba806d9c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9bb8009b40 tx=0x7f9bb8008040 comp rx=0 tx=0).stop 2024-02-21T23:02:52.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.366+0000 7f9bc7200640 1 -- 172.21.15.28:0/3961428691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bc8147ca0 msgr2=0x7f9bc81ead20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:52.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.366+0000 7f9bc7200640 1 --2- 172.21.15.28:0/3961428691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bc8147ca0 0x7f9bc81ead20 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f9bc80c7470 tx=0x7f9bc000b4a0 comp rx=0 tx=0).stop 2024-02-21T23:02:52.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.366+0000 7f9bc7200640 1 -- 172.21.15.28:0/3961428691 shutdown_connections 2024-02-21T23:02:52.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.366+0000 7f9bc7200640 1 --2- 172.21.15.28:0/3961428691 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9ba806b500 0x7f9ba806d9c0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:52.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.366+0000 7f9bc7200640 1 --2- 172.21.15.28:0/3961428691 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9bc8150b00 0x7f9bc8147270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:52.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.366+0000 7f9bc7200640 1 --2- 172.21.15.28:0/3961428691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bc8147ca0 0x7f9bc81ead20 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:52.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.366+0000 7f9bc7200640 1 --2- 172.21.15.28:0/3961428691 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9bac004800 0x7f9bc81550c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:52.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.366+0000 7f9bc7200640 1 -- 172.21.15.28:0/3961428691 >> 172.21.15.28:0/3961428691 conn(0x7f9bc806b630 msgr2=0x7f9bc80a7030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:52.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.366+0000 7f9bc7200640 1 -- 172.21.15.28:0/3961428691 shutdown_connections 2024-02-21T23:02:52.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.366+0000 7f9bc7200640 1 -- 172.21.15.28:0/3961428691 wait complete. 2024-02-21T23:02:52.374 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-21T23:02:30.728329+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-21T23:02:35.795158+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-02-21T23:02:36.840110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-02-21T23:02:40.441388+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-02-21T23:02:52.374 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-21T23:02:52.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.595+0000 7ff12e600640 1 Processor -- start 2024-02-21T23:02:52.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.595+0000 7ff12e600640 1 -- start start 2024-02-21T23:02:52.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.597+0000 7ff12e600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7ff1280c6440 0x7ff1280bfbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:52.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.597+0000 7ff12e600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff128155170 0x7ff1280c0100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:52.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.597+0000 7ff12e600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1281589b0 con 0x7ff12814f560 2024-02-21T23:02:52.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.597+0000 7ff12e600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff128158760 con 0x7ff128155170 2024-02-21T23:02:52.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.597+0000 7ff12e600640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1280c6030 con 0x7ff1280c6440 2024-02-21T23:02:52.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.597+0000 7ff127400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff12814f560 0x7ff1280c77a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42386/0 (socket says 172.21.15.28:42386) 2024-02-21T23:02:52.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.597+0000 7ff127400640 1 -- 172.21.15.28:0/2339406833 learned_addr learned my addr 172.21.15.28:0/2339406833 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:02:52.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.597+0000 7ff127e00640 1 --2- 172.21.15.28:0/2339406833 >> v2:172.21.15.28:3300/0 conn(0x7ff128155170 0x7ff1280c0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:52.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.598+0000 7ff126a00640 1 --2- 172.21.15.28:0/2339406833 >> v2:172.21.15.81:3300/0 conn(0x7ff1280c6440 0x7ff1280bfbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:52.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.598+0000 7ff126000640 1 -- 172.21.15.28:0/2339406833 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1860425588 0 0) 0x7ff1281589b0 con 0x7ff12814f560 2024-02-21T23:02:52.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.598+0000 7ff126000640 1 -- 172.21.15.28:0/2339406833 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff10c003520 con 0x7ff12814f560 2024-02-21T23:02:52.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.598+0000 7ff126a00640 1 -- 172.21.15.28:0/2339406833 >> v2:172.21.15.28:3300/0 conn(0x7ff128155170 msgr2=0x7ff1280c0100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:52.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.598+0000 7ff126a00640 1 --2- 172.21.15.28:0/2339406833 >> v2:172.21.15.28:3300/0 conn(0x7ff128155170 0x7ff1280c0100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:52.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.598+0000 7ff126a00640 1 -- 172.21.15.28:0/2339406833 >> v1:172.21.15.28:6789/0 conn(0x7ff12814f560 legacy=0x7ff1280c77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:52.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.598+0000 7ff126a00640 1 -- 172.21.15.28:0/2339406833 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff128158c90 con 0x7ff1280c6440 2024-02-21T23:02:52.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.598+0000 7ff127e00640 1 --2- 172.21.15.28:0/2339406833 >> v2:172.21.15.28:3300/0 conn(0x7ff128155170 0x7ff1280c0100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-21T23:02:52.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.600+0000 7ff126a00640 1 --2- 172.21.15.28:0/2339406833 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff1280c6440 0x7ff1280bfbc0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff1280cb940 tx=0x7ff11402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d29918bcb7ed4625 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:52.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.601+0000 7ff126000640 1 -- 172.21.15.28:0/2339406833 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff114035020 con 0x7ff1280c6440 2024-02-21T23:02:52.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.601+0000 7ff126000640 1 -- 172.21.15.28:0/2339406833 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff114031830 con 0x7ff1280c6440 2024-02-21T23:02:52.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.601+0000 7ff126000640 1 -- 172.21.15.28:0/2339406833 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff114031c00 con 0x7ff1280c6440 2024-02-21T23:02:52.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.602+0000 7ff12e600640 1 -- 172.21.15.28:0/2339406833 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff1280c6440 msgr2=0x7ff1280bfbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:52.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.602+0000 7ff12e600640 1 --2- 172.21.15.28:0/2339406833 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff1280c6440 0x7ff1280bfbc0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff1280cb940 tx=0x7ff11402b8d0 comp rx=0 tx=0).stop 2024-02-21T23:02:52.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.602+0000 7ff12e600640 1 -- 172.21.15.28:0/2339406833 shutdown_connections 2024-02-21T23:02:52.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.602+0000 7ff12e600640 1 --2- 172.21.15.28:0/2339406833 >> v2:172.21.15.28:3300/0 conn(0x7ff128155170 0x7ff1280c0100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:52.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.602+0000 7ff12e600640 1 --2- 172.21.15.28:0/2339406833 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff1280c6440 0x7ff1280bfbc0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:52.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.602+0000 7ff12e600640 1 -- 172.21.15.28:0/2339406833 >> 172.21.15.28:0/2339406833 conn(0x7ff12808b670 msgr2=0x7ff12806ce80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:52.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.602+0000 7ff12e600640 1 -- 172.21.15.28:0/2339406833 shutdown_connections 2024-02-21T23:02:52.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.602+0000 7ff12e600640 1 -- 172.21.15.28:0/2339406833 wait complete. 2024-02-21T23:02:52.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.604+0000 7ff12e600640 1 Processor -- start 2024-02-21T23:02:52.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.605+0000 7ff12e600640 1 -- start start 2024-02-21T23:02:52.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.606+0000 7ff12e600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff12814f560 0x7ff12814d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:52.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.606+0000 7ff12e600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff128155170 0x7ff128156560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:52.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.606+0000 7ff12e600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff1281527c0 0x7ff128156aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:52.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.606+0000 7ff12e600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff128159530 con 0x7ff128155170 2024-02-21T23:02:52.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.606+0000 7ff12e600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff128159250 con 0x7ff1281527c0 2024-02-21T23:02:52.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.606+0000 7ff12e600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1280c7eb0 con 0x7ff12814f560 2024-02-21T23:02:52.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.606+0000 7ff126a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff128155170 0x7ff128156560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:52.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.607+0000 7ff126a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff128155170 0x7ff128156560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58164/0 (socket says 172.21.15.28:58164) 2024-02-21T23:02:52.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.607+0000 7ff126a00640 1 -- 172.21.15.28:0/2610809219 learned_addr learned my addr 172.21.15.28:0/2610809219 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:02:52.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.607+0000 7ff127400640 1 --2- 172.21.15.28:0/2610809219 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff12814f560 0x7ff12814d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:52.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.607+0000 7ff127e00640 1 --2- 172.21.15.28:0/2610809219 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff1281527c0 0x7ff128156aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:52.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.607+0000 7ff126a00640 1 -- 172.21.15.28:0/2610809219 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff12814f560 msgr2=0x7ff12814d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:52.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.607+0000 7ff126a00640 1 --2- 172.21.15.28:0/2610809219 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff12814f560 0x7ff12814d6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:52.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.607+0000 7ff126a00640 1 -- 172.21.15.28:0/2610809219 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff1281527c0 msgr2=0x7ff128156aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:52.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.607+0000 7ff126a00640 1 --2- 172.21.15.28:0/2610809219 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff1281527c0 0x7ff128156aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:52.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.607+0000 7ff126a00640 1 -- 172.21.15.28:0/2610809219 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1280c49e0 con 0x7ff128155170 2024-02-21T23:02:52.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.608+0000 7ff126a00640 1 --2- 172.21.15.28:0/2610809219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff128155170 0x7ff128156560 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff11402bee0 tx=0x7ff1140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:52.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.608+0000 7ff11f400640 1 -- 172.21.15.28:0/2610809219 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff114035040 con 0x7ff128155170 2024-02-21T23:02:52.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.608+0000 7ff11f400640 1 -- 172.21.15.28:0/2610809219 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff11403f070 con 0x7ff128155170 2024-02-21T23:02:52.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.608+0000 7ff11f400640 1 -- 172.21.15.28:0/2610809219 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff11403baa0 con 0x7ff128155170 2024-02-21T23:02:52.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.608+0000 7ff12e600640 1 -- 172.21.15.28:0/2610809219 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1280c35b0 con 0x7ff128155170 2024-02-21T23:02:52.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.609+0000 7ff12e600640 1 -- 172.21.15.28:0/2610809219 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1280c3b70 con 0x7ff128155170 2024-02-21T23:02:52.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.610+0000 7ff12e600640 1 -- 172.21.15.28:0/2610809219 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1280c3d70 con 0x7ff128155170 2024-02-21T23:02:52.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.611+0000 7ff11f400640 1 -- 172.21.15.28:0/2610809219 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff114038020 con 0x7ff128155170 2024-02-21T23:02:52.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.612+0000 7ff11f400640 1 --2- 172.21.15.28:0/2610809219 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff10806b4d0 0x7ff10806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:52.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.612+0000 7ff11f400640 1 -- 172.21.15.28:0/2610809219 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7ff1140abf40 con 0x7ff128155170 2024-02-21T23:02:52.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.612+0000 7ff127400640 1 --2- 172.21.15.28:0/2610809219 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff10806b4d0 0x7ff10806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:52.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.614+0000 7ff127400640 1 --2- 172.21.15.28:0/2610809219 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff10806b4d0 0x7ff10806d990 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff118005880 tx=0x7ff118009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:52.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.617+0000 7ff11f400640 1 -- 172.21.15.28:0/2610809219 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ff1280c3d70 con 0x7ff128155170 2024-02-21T23:02:52.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.821+0000 7ff12e600640 1 -- 172.21.15.28:0/2610809219 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff128067f30 con 0x7ff128155170 2024-02-21T23:02:52.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.822+0000 7ff11f400640 1 -- 172.21.15.28:0/2610809219 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff11407a420 con 0x7ff128155170 2024-02-21T23:02:52.811 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-02-21T23:02:52.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.825+0000 7ff12e600640 1 -- 172.21.15.28:0/2610809219 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff10806b4d0 msgr2=0x7ff10806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:52.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.825+0000 7ff12e600640 1 --2- 172.21.15.28:0/2610809219 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff10806b4d0 0x7ff10806d990 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff118005880 tx=0x7ff118009040 comp rx=0 tx=0).stop 2024-02-21T23:02:52.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.826+0000 7ff12e600640 1 -- 172.21.15.28:0/2610809219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff128155170 msgr2=0x7ff128156560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:52.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.826+0000 7ff12e600640 1 --2- 172.21.15.28:0/2610809219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff128155170 0x7ff128156560 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff11402bee0 tx=0x7ff1140046c0 comp rx=0 tx=0).stop 2024-02-21T23:02:52.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.826+0000 7ff12e600640 1 -- 172.21.15.28:0/2610809219 shutdown_connections 2024-02-21T23:02:52.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.826+0000 7ff12e600640 1 --2- 172.21.15.28:0/2610809219 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff10806b4d0 0x7ff10806d990 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:52.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.826+0000 7ff12e600640 1 --2- 172.21.15.28:0/2610809219 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff1281527c0 0x7ff128156aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:52.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.826+0000 7ff12e600640 1 --2- 172.21.15.28:0/2610809219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff128155170 0x7ff128156560 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:52.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.826+0000 7ff12e600640 1 --2- 172.21.15.28:0/2610809219 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff12814f560 0x7ff12814d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:52.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.826+0000 7ff12e600640 1 -- 172.21.15.28:0/2610809219 >> 172.21.15.28:0/2610809219 conn(0x7ff1280c6440 msgr2=0x7ff1281549a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:52.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.826+0000 7ff12e600640 1 -- 172.21.15.28:0/2610809219 shutdown_connections 2024-02-21T23:02:52.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:52.826+0000 7ff12e600640 1 -- 172.21.15.28:0/2610809219 wait complete. 2024-02-21T23:02:52.835 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-21T23:02:53.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.075+0000 7fb539c00640 1 Processor -- start 2024-02-21T23:02:53.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.075+0000 7fb539c00640 1 -- start start 2024-02-21T23:02:53.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.076+0000 7fb539c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb5340c22e0 0x7fb53406d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:53.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.076+0000 7fb539c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fb5341561a0 0x7fb53406d660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:53.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.076+0000 7fb539c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb534071970 con 0x7fb5340c5a70 2024-02-21T23:02:53.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.076+0000 7fb539c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb534072a10 con 0x7fb5340c22e0 2024-02-21T23:02:53.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.076+0000 7fb539c00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb534072580 con 0x7fb5341561a0 2024-02-21T23:02:53.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.076+0000 7fb532a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb5340c5a70 0x7fb534155a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42392/0 (socket says 172.21.15.28:42392) 2024-02-21T23:02:53.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.076+0000 7fb532a00640 1 -- 172.21.15.28:0/1813037461 learned_addr learned my addr 172.21.15.28:0/1813037461 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:02:53.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.077+0000 7fb532000640 1 --2- 172.21.15.28:0/1813037461 >> v2:172.21.15.28:3300/0 conn(0x7fb5340c22e0 0x7fb53406d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:53.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.077+0000 7fb533400640 1 --2- 172.21.15.28:0/1813037461 >> v2:172.21.15.81:3300/0 conn(0x7fb5341561a0 0x7fb53406d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:53.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.077+0000 7fb531600640 1 -- 172.21.15.28:0/1813037461 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2453611620 0 0) 0x7fb534071970 con 0x7fb5340c5a70 2024-02-21T23:02:53.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.077+0000 7fb531600640 1 -- 172.21.15.28:0/1813037461 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb518003520 con 0x7fb5340c5a70 2024-02-21T23:02:53.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.077+0000 7fb532000640 1 -- 172.21.15.28:0/1813037461 >> v2:172.21.15.81:3300/0 conn(0x7fb5341561a0 msgr2=0x7fb53406d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:53.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.077+0000 7fb532000640 1 --2- 172.21.15.28:0/1813037461 >> v2:172.21.15.81:3300/0 conn(0x7fb5341561a0 0x7fb53406d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:53.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.077+0000 7fb532000640 1 -- 172.21.15.28:0/1813037461 >> v1:172.21.15.28:6789/0 conn(0x7fb5340c5a70 legacy=0x7fb534155a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:53.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.077+0000 7fb532000640 1 -- 172.21.15.28:0/1813037461 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb534071c50 con 0x7fb5340c22e0 2024-02-21T23:02:53.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.078+0000 7fb532000640 1 --2- 172.21.15.28:0/1813037461 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5340c22e0 0x7fb53406d120 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb534071310 tx=0x7fb52002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=150aa4c487648589 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:53.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.079+0000 7fb531600640 1 -- 172.21.15.28:0/1813037461 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5200315a0 con 0x7fb5340c22e0 2024-02-21T23:02:53.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.079+0000 7fb531600640 1 -- 172.21.15.28:0/1813037461 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb520037070 con 0x7fb5340c22e0 2024-02-21T23:02:53.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.079+0000 7fb531600640 1 -- 172.21.15.28:0/1813037461 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb520031960 con 0x7fb5340c22e0 2024-02-21T23:02:53.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.079+0000 7fb539c00640 1 -- 172.21.15.28:0/1813037461 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5340c22e0 msgr2=0x7fb53406d120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:53.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.079+0000 7fb539c00640 1 --2- 172.21.15.28:0/1813037461 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5340c22e0 0x7fb53406d120 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb534071310 tx=0x7fb52002b840 comp rx=0 tx=0).stop 2024-02-21T23:02:53.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.079+0000 7fb539c00640 1 -- 172.21.15.28:0/1813037461 shutdown_connections 2024-02-21T23:02:53.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.079+0000 7fb539c00640 1 --2- 172.21.15.28:0/1813037461 >> v2:172.21.15.81:3300/0 conn(0x7fb5341561a0 0x7fb53406d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:53.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.079+0000 7fb539c00640 1 --2- 172.21.15.28:0/1813037461 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5340c22e0 0x7fb53406d120 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:53.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.079+0000 7fb539c00640 1 -- 172.21.15.28:0/1813037461 >> 172.21.15.28:0/1813037461 conn(0x7fb5340a85a0 msgr2=0x7fb5340a8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:53.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.080+0000 7fb539c00640 1 -- 172.21.15.28:0/1813037461 shutdown_connections 2024-02-21T23:02:53.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.080+0000 7fb539c00640 1 -- 172.21.15.28:0/1813037461 wait complete. 2024-02-21T23:02:53.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.081+0000 7fb539c00640 1 Processor -- start 2024-02-21T23:02:53.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.081+0000 7fb539c00640 1 -- start start 2024-02-21T23:02:53.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.082+0000 7fb539c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb5340c22e0 0x7fb53414f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:53.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.082+0000 7fb539c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5340c5a70 0x7fb5340bfc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:53.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.082+0000 7fb539c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5341561a0 0x7fb5340a6590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:53.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.082+0000 7fb539c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb53406dba0 con 0x7fb5341561a0 2024-02-21T23:02:53.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.082+0000 7fb539c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb534072210 con 0x7fb5340c5a70 2024-02-21T23:02:53.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.082+0000 7fb539c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5340a6160 con 0x7fb5340c22e0 2024-02-21T23:02:53.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.083+0000 7fb533400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5341561a0 0x7fb5340a6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:53.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.083+0000 7fb533400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5341561a0 0x7fb5340a6590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58170/0 (socket says 172.21.15.28:58170) 2024-02-21T23:02:53.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.083+0000 7fb533400640 1 -- 172.21.15.28:0/627270268 learned_addr learned my addr 172.21.15.28:0/627270268 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:02:53.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.083+0000 7fb532000640 1 --2- 172.21.15.28:0/627270268 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5340c5a70 0x7fb5340bfc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:53.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.083+0000 7fb532a00640 1 --2- 172.21.15.28:0/627270268 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb5340c22e0 0x7fb53414f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:53.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.083+0000 7fb533400640 1 -- 172.21.15.28:0/627270268 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb5340c22e0 msgr2=0x7fb53414f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:53.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.083+0000 7fb533400640 1 --2- 172.21.15.28:0/627270268 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb5340c22e0 0x7fb53414f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:53.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.083+0000 7fb533400640 1 -- 172.21.15.28:0/627270268 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5340c5a70 msgr2=0x7fb5340bfc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:53.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.083+0000 7fb533400640 1 --2- 172.21.15.28:0/627270268 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5340c5a70 0x7fb5340bfc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:53.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.083+0000 7fb533400640 1 -- 172.21.15.28:0/627270268 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5341e80b0 con 0x7fb5341561a0 2024-02-21T23:02:53.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.083+0000 7fb532000640 1 --2- 172.21.15.28:0/627270268 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5340c5a70 0x7fb5340bfc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:02:53.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.083+0000 7fb533400640 1 --2- 172.21.15.28:0/627270268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5341561a0 0x7fb5340a6590 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb52c007e90 tx=0x7fb52c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:53.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.084+0000 7fb52aa00640 1 -- 172.21.15.28:0/627270268 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb52c014420 con 0x7fb5341561a0 2024-02-21T23:02:53.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.084+0000 7fb52aa00640 1 -- 172.21.15.28:0/627270268 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb52c019070 con 0x7fb5341561a0 2024-02-21T23:02:53.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.084+0000 7fb52aa00640 1 -- 172.21.15.28:0/627270268 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb52c014800 con 0x7fb5341561a0 2024-02-21T23:02:53.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.084+0000 7fb539c00640 1 -- 172.21.15.28:0/627270268 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5340aa560 con 0x7fb5341561a0 2024-02-21T23:02:53.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.084+0000 7fb539c00640 1 -- 172.21.15.28:0/627270268 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5341eb290 con 0x7fb5341561a0 2024-02-21T23:02:53.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.085+0000 7fb539c00640 1 -- 172.21.15.28:0/627270268 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb534067f30 con 0x7fb5341561a0 2024-02-21T23:02:53.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.086+0000 7fb52aa00640 1 -- 172.21.15.28:0/627270268 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb52c0149a0 con 0x7fb5341561a0 2024-02-21T23:02:53.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.087+0000 7fb52aa00640 1 --2- 172.21.15.28:0/627270268 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb51406b4d0 0x7fb51406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:53.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.087+0000 7fb52aa00640 1 -- 172.21.15.28:0/627270268 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fb52c08ad80 con 0x7fb5341561a0 2024-02-21T23:02:53.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.088+0000 7fb532a00640 1 --2- 172.21.15.28:0/627270268 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb51406b4d0 0x7fb51406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:53.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.089+0000 7fb532a00640 1 --2- 172.21.15.28:0/627270268 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb51406b4d0 0x7fb51406d990 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb524000f60 tx=0x7fb524009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:53.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.092+0000 7fb52aa00640 1 -- 172.21.15.28:0/627270268 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb52c0598b0 con 0x7fb5341561a0 2024-02-21T23:02:53.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.295+0000 7fb539c00640 1 -- 172.21.15.28:0/627270268 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb5341dda20 con 0x7fb5341561a0 2024-02-21T23:02:53.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.295+0000 7fb52aa00640 1 -- 172.21.15.28:0/627270268 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb52c0591b0 con 0x7fb5341561a0 2024-02-21T23:02:53.284 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-02-21T23:02:53.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.298+0000 7fb539c00640 1 -- 172.21.15.28:0/627270268 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb51406b4d0 msgr2=0x7fb51406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:53.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.298+0000 7fb539c00640 1 --2- 172.21.15.28:0/627270268 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb51406b4d0 0x7fb51406d990 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb524000f60 tx=0x7fb524009040 comp rx=0 tx=0).stop 2024-02-21T23:02:53.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.298+0000 7fb539c00640 1 -- 172.21.15.28:0/627270268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5341561a0 msgr2=0x7fb5340a6590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:53.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.298+0000 7fb539c00640 1 --2- 172.21.15.28:0/627270268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5341561a0 0x7fb5340a6590 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb52c007e90 tx=0x7fb52c00e4b0 comp rx=0 tx=0).stop 2024-02-21T23:02:53.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.298+0000 7fb539c00640 1 -- 172.21.15.28:0/627270268 shutdown_connections 2024-02-21T23:02:53.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.298+0000 7fb539c00640 1 --2- 172.21.15.28:0/627270268 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb51406b4d0 0x7fb51406d990 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:53.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.298+0000 7fb539c00640 1 --2- 172.21.15.28:0/627270268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5341561a0 0x7fb5340a6590 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:53.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.298+0000 7fb539c00640 1 --2- 172.21.15.28:0/627270268 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5340c5a70 0x7fb5340bfc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:53.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.298+0000 7fb539c00640 1 --2- 172.21.15.28:0/627270268 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb5340c22e0 0x7fb53414f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:53.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.298+0000 7fb539c00640 1 -- 172.21.15.28:0/627270268 >> 172.21.15.28:0/627270268 conn(0x7fb5340a85a0 msgr2=0x7fb5340c37f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:53.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.298+0000 7fb539c00640 1 -- 172.21.15.28:0/627270268 shutdown_connections 2024-02-21T23:02:53.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.298+0000 7fb539c00640 1 -- 172.21.15.28:0/627270268 wait complete. 2024-02-21T23:02:53.306 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-21T23:02:53.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.518+0000 7f7d92600640 1 Processor -- start 2024-02-21T23:02:53.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.518+0000 7f7d92600640 1 -- start start 2024-02-21T23:02:53.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.520+0000 7f7d92600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7d8c13e0c0 0x7f7d8c0c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:53.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.520+0000 7f7d92600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7d8c06dbc0 0x7f7d8c06c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:53.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.520+0000 7f7d92600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f7d8c0c76f0 0x7f7d8c06c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:53.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.520+0000 7f7d92600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d8c0c89a0 con 0x7f7d8c06dbc0 2024-02-21T23:02:53.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.520+0000 7f7d92600640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d8c158550 con 0x7f7d8c0c76f0 2024-02-21T23:02:53.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.520+0000 7f7d92600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d8c1587d0 con 0x7f7d8c13e0c0 2024-02-21T23:02:53.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.520+0000 7f7d8be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7d8c06dbc0 0x7f7d8c06c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:53.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.520+0000 7f7d8be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7d8c06dbc0 0x7f7d8c06c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58174/0 (socket says 172.21.15.28:58174) 2024-02-21T23:02:53.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.520+0000 7f7d8be00640 1 -- 172.21.15.28:0/2844689095 learned_addr learned my addr 172.21.15.28:0/2844689095 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:02:53.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.521+0000 7f7d91200640 1 --2- 172.21.15.28:0/2844689095 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7d8c13e0c0 0x7f7d8c0c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:53.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.521+0000 7f7d91c00640 1 --2- 172.21.15.28:0/2844689095 >> v2:172.21.15.81:3300/0 conn(0x7f7d8c0c76f0 0x7f7d8c06c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:53.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.521+0000 7f7d8be00640 1 -- 172.21.15.28:0/2844689095 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7d8c13e0c0 msgr2=0x7f7d8c0c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:53.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.521+0000 7f7d8be00640 1 --2- 172.21.15.28:0/2844689095 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7d8c13e0c0 0x7f7d8c0c71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:53.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.521+0000 7f7d8be00640 1 -- 172.21.15.28:0/2844689095 >> v2:172.21.15.81:3300/0 conn(0x7f7d8c0c76f0 msgr2=0x7f7d8c06c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:53.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.521+0000 7f7d8be00640 1 --2- 172.21.15.28:0/2844689095 >> v2:172.21.15.81:3300/0 conn(0x7f7d8c0c76f0 0x7f7d8c06c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:53.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.521+0000 7f7d8be00640 1 -- 172.21.15.28:0/2844689095 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d8c158ab0 con 0x7f7d8c06dbc0 2024-02-21T23:02:53.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.522+0000 7f7d8be00640 1 --2- 172.21.15.28:0/2844689095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d8c06dbc0 0x7f7d8c06c280 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7d8c06bd10 tx=0x7f7d7c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=880b2c0f0ac8c06d server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:53.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.523+0000 7f7d8b400640 1 -- 172.21.15.28:0/2844689095 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d7c02e070 con 0x7f7d8c06dbc0 2024-02-21T23:02:53.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.523+0000 7f7d8b400640 1 -- 172.21.15.28:0/2844689095 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d7c037070 con 0x7f7d8c06dbc0 2024-02-21T23:02:53.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.523+0000 7f7d8b400640 1 -- 172.21.15.28:0/2844689095 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d7c031a00 con 0x7f7d8c06dbc0 2024-02-21T23:02:53.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.523+0000 7f7d92600640 1 -- 172.21.15.28:0/2844689095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d8c06dbc0 msgr2=0x7f7d8c06c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:53.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.523+0000 7f7d92600640 1 --2- 172.21.15.28:0/2844689095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d8c06dbc0 0x7f7d8c06c280 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7d8c06bd10 tx=0x7f7d7c02b880 comp rx=0 tx=0).stop 2024-02-21T23:02:53.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.523+0000 7f7d92600640 1 -- 172.21.15.28:0/2844689095 shutdown_connections 2024-02-21T23:02:53.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.523+0000 7f7d92600640 1 --2- 172.21.15.28:0/2844689095 >> v2:172.21.15.81:3300/0 conn(0x7f7d8c0c76f0 0x7f7d8c06c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:53.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.523+0000 7f7d92600640 1 --2- 172.21.15.28:0/2844689095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d8c06dbc0 0x7f7d8c06c280 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:53.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.523+0000 7f7d92600640 1 --2- 172.21.15.28:0/2844689095 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7d8c13e0c0 0x7f7d8c0c71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:53.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.523+0000 7f7d92600640 1 -- 172.21.15.28:0/2844689095 >> 172.21.15.28:0/2844689095 conn(0x7f7d8c073110 msgr2=0x7f7d8c0ac490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:53.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.523+0000 7f7d92600640 1 -- 172.21.15.28:0/2844689095 shutdown_connections 2024-02-21T23:02:53.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.524+0000 7f7d92600640 1 -- 172.21.15.28:0/2844689095 wait complete. 2024-02-21T23:02:53.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.525+0000 7f7d92600640 1 Processor -- start 2024-02-21T23:02:53.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.525+0000 7f7d92600640 1 -- start start 2024-02-21T23:02:53.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.526+0000 7f7d92600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d8c06dbc0 0x7f7d8c1ee210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:53.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.526+0000 7f7d92600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7d8c0c76f0 0x7f7d8c136220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:53.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.526+0000 7f7d92600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7d8c13e0c0 0x7f7d8c136760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:53.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.526+0000 7f7d92600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d8c06cd00 con 0x7f7d8c06dbc0 2024-02-21T23:02:53.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.526+0000 7f7d92600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d8c159070 con 0x7f7d8c13e0c0 2024-02-21T23:02:53.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.526+0000 7f7d92600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d8c06ce80 con 0x7f7d8c0c76f0 2024-02-21T23:02:53.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.527+0000 7f7d91200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d8c06dbc0 0x7f7d8c1ee210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:53.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.527+0000 7f7d91200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d8c06dbc0 0x7f7d8c1ee210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58176/0 (socket says 172.21.15.28:58176) 2024-02-21T23:02:53.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.527+0000 7f7d91200640 1 -- 172.21.15.28:0/38954547 learned_addr learned my addr 172.21.15.28:0/38954547 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:02:53.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.527+0000 7f7d91c00640 1 --2- 172.21.15.28:0/38954547 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7d8c13e0c0 0x7f7d8c136760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:53.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.527+0000 7f7d8be00640 1 --2- 172.21.15.28:0/38954547 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7d8c0c76f0 0x7f7d8c136220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:53.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.527+0000 7f7d91200640 1 -- 172.21.15.28:0/38954547 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7d8c0c76f0 msgr2=0x7f7d8c136220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:53.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.527+0000 7f7d91200640 1 --2- 172.21.15.28:0/38954547 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7d8c0c76f0 0x7f7d8c136220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:53.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.527+0000 7f7d91200640 1 -- 172.21.15.28:0/38954547 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7d8c13e0c0 msgr2=0x7f7d8c136760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:53.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.527+0000 7f7d91200640 1 --2- 172.21.15.28:0/38954547 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7d8c13e0c0 0x7f7d8c136760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:53.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.527+0000 7f7d91200640 1 -- 172.21.15.28:0/38954547 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d8c153b60 con 0x7f7d8c06dbc0 2024-02-21T23:02:53.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.527+0000 7f7d91c00640 1 --2- 172.21.15.28:0/38954547 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7d8c13e0c0 0x7f7d8c136760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:02:53.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.527+0000 7f7d8be00640 1 --2- 172.21.15.28:0/38954547 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7d8c0c76f0 0x7f7d8c136220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:02:53.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.527+0000 7f7d91200640 1 --2- 172.21.15.28:0/38954547 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d8c06dbc0 0x7f7d8c1ee210 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f7d8000d800 tx=0x7f7d8000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:53.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.528+0000 7f7d88c00640 1 -- 172.21.15.28:0/38954547 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d8000b6c0 con 0x7f7d8c06dbc0 2024-02-21T23:02:53.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.528+0000 7f7d88c00640 1 -- 172.21.15.28:0/38954547 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d80019070 con 0x7f7d8c06dbc0 2024-02-21T23:02:53.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.528+0000 7f7d88c00640 1 -- 172.21.15.28:0/38954547 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d8000ba80 con 0x7f7d8c06dbc0 2024-02-21T23:02:53.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.528+0000 7f7d92600640 1 -- 172.21.15.28:0/38954547 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d8c136ca0 con 0x7f7d8c06dbc0 2024-02-21T23:02:53.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.528+0000 7f7d92600640 1 -- 172.21.15.28:0/38954547 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d8c1eeb50 con 0x7f7d8c06dbc0 2024-02-21T23:02:53.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.529+0000 7f7d92600640 1 -- 172.21.15.28:0/38954547 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d8c13c360 con 0x7f7d8c06dbc0 2024-02-21T23:02:53.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.530+0000 7f7d88c00640 1 -- 172.21.15.28:0/38954547 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7d8000bc20 con 0x7f7d8c06dbc0 2024-02-21T23:02:53.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.531+0000 7f7d88c00640 1 --2- 172.21.15.28:0/38954547 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7d7006b500 0x7f7d7006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:53.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.531+0000 7f7d88c00640 1 -- 172.21.15.28:0/38954547 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f7d8008aa10 con 0x7f7d8c06dbc0 2024-02-21T23:02:53.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.532+0000 7f7d8be00640 1 --2- 172.21.15.28:0/38954547 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7d7006b500 0x7f7d7006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:53.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.533+0000 7f7d8be00640 1 --2- 172.21.15.28:0/38954547 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7d7006b500 0x7f7d7006d9c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7d7c036000 tx=0x7f7d7c008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:53.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.535+0000 7f7d88c00640 1 -- 172.21.15.28:0/38954547 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7d8c13c360 con 0x7f7d8c06dbc0 2024-02-21T23:02:53.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.733+0000 7f7d92600640 1 -- 172.21.15.28:0/38954547 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7d8c04f290 con 0x7f7d8c06dbc0 2024-02-21T23:02:53.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.734+0000 7f7d88c00640 1 -- 172.21.15.28:0/38954547 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7d8001c360 con 0x7f7d8c06dbc0 2024-02-21T23:02:53.723 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-02-21T23:02:53.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.736+0000 7f7d92600640 1 -- 172.21.15.28:0/38954547 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7d7006b500 msgr2=0x7f7d7006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:53.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.736+0000 7f7d92600640 1 --2- 172.21.15.28:0/38954547 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7d7006b500 0x7f7d7006d9c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7d7c036000 tx=0x7f7d7c008030 comp rx=0 tx=0).stop 2024-02-21T23:02:53.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.737+0000 7f7d92600640 1 -- 172.21.15.28:0/38954547 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d8c06dbc0 msgr2=0x7f7d8c1ee210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:53.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.737+0000 7f7d92600640 1 --2- 172.21.15.28:0/38954547 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d8c06dbc0 0x7f7d8c1ee210 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f7d8000d800 tx=0x7f7d8000dcc0 comp rx=0 tx=0).stop 2024-02-21T23:02:53.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.737+0000 7f7d92600640 1 -- 172.21.15.28:0/38954547 shutdown_connections 2024-02-21T23:02:53.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.737+0000 7f7d92600640 1 --2- 172.21.15.28:0/38954547 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7d7006b500 0x7f7d7006d9c0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:53.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.737+0000 7f7d92600640 1 --2- 172.21.15.28:0/38954547 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7d8c0c76f0 0x7f7d8c136220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:53.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.737+0000 7f7d92600640 1 --2- 172.21.15.28:0/38954547 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7d8c13e0c0 0x7f7d8c136760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:53.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.737+0000 7f7d92600640 1 --2- 172.21.15.28:0/38954547 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d8c06dbc0 0x7f7d8c1ee210 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:53.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.737+0000 7f7d92600640 1 -- 172.21.15.28:0/38954547 >> 172.21.15.28:0/38954547 conn(0x7f7d8c073110 msgr2=0x7f7d8c0c5020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:53.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.737+0000 7f7d92600640 1 -- 172.21.15.28:0/38954547 shutdown_connections 2024-02-21T23:02:53.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.737+0000 7f7d92600640 1 -- 172.21.15.28:0/38954547 wait complete. 2024-02-21T23:02:53.743 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-02-21T23:02:53.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.952+0000 7f026ec00640 1 Processor -- start 2024-02-21T23:02:53.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.952+0000 7f026ec00640 1 -- start start 2024-02-21T23:02:53.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.954+0000 7f026ec00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f026806be80 0x7f026806c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:53.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.954+0000 7f026ec00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0268072830 0x7f026806eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:53.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.954+0000 7f026ec00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0268155d70 con 0x7f02680a84b0 2024-02-21T23:02:53.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.954+0000 7f026ec00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0268155840 con 0x7f0268072830 2024-02-21T23:02:53.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.954+0000 7f026ec00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0268155a90 con 0x7f026806be80 2024-02-21T23:02:53.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.954+0000 7f026e200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0268072830 0x7f026806eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:53.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.954+0000 7f026e200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0268072830 0x7f026806eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58178/0 (socket says 172.21.15.28:58178) 2024-02-21T23:02:53.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.954+0000 7f026e200640 1 -- 172.21.15.28:0/3746925510 learned_addr learned my addr 172.21.15.28:0/3746925510 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:02:53.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.955+0000 7f026d800640 1 --2- 172.21.15.28:0/3746925510 >> v2:172.21.15.81:3300/0 conn(0x7f026806be80 0x7f026806c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:53.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.955+0000 7f026e200640 1 -- 172.21.15.28:0/3746925510 >> v2:172.21.15.81:3300/0 conn(0x7f026806be80 msgr2=0x7f026806c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:53.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.955+0000 7f026e200640 1 --2- 172.21.15.28:0/3746925510 >> v2:172.21.15.81:3300/0 conn(0x7f026806be80 0x7f026806c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:53.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.955+0000 7f026e200640 1 -- 172.21.15.28:0/3746925510 >> v1:172.21.15.81:6789/0 conn(0x7f02680a84b0 legacy=0x7f0268158c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:53.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.955+0000 7f026e200640 1 -- 172.21.15.28:0/3746925510 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0268158480 con 0x7f0268072830 2024-02-21T23:02:53.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.957+0000 7f026e200640 1 --2- 172.21.15.28:0/3746925510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0268072830 0x7f026806eb80 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f02680c8030 tx=0x7f026002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3c4f94aa283e436 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:53.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.957+0000 7f0267e00640 1 -- 172.21.15.28:0/3746925510 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f026002c9f0 con 0x7f0268072830 2024-02-21T23:02:53.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.957+0000 7f0267e00640 1 -- 172.21.15.28:0/3746925510 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f026002cb90 con 0x7f0268072830 2024-02-21T23:02:53.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.957+0000 7f0267e00640 1 -- 172.21.15.28:0/3746925510 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02600343f0 con 0x7f0268072830 2024-02-21T23:02:53.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.957+0000 7f026ec00640 1 -- 172.21.15.28:0/3746925510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0268072830 msgr2=0x7f026806eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:53.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.957+0000 7f026ec00640 1 --2- 172.21.15.28:0/3746925510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0268072830 0x7f026806eb80 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f02680c8030 tx=0x7f026002be50 comp rx=0 tx=0).stop 2024-02-21T23:02:53.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.957+0000 7f026ec00640 1 -- 172.21.15.28:0/3746925510 shutdown_connections 2024-02-21T23:02:53.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.957+0000 7f026ec00640 1 --2- 172.21.15.28:0/3746925510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0268072830 0x7f026806eb80 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:53.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.957+0000 7f026ec00640 1 --2- 172.21.15.28:0/3746925510 >> v2:172.21.15.81:3300/0 conn(0x7f026806be80 0x7f026806c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:53.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.957+0000 7f026ec00640 1 -- 172.21.15.28:0/3746925510 >> 172.21.15.28:0/3746925510 conn(0x7f0268071b20 msgr2=0x7f0268071f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:53.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.958+0000 7f026ec00640 1 -- 172.21.15.28:0/3746925510 shutdown_connections 2024-02-21T23:02:53.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.958+0000 7f026ec00640 1 -- 172.21.15.28:0/3746925510 wait complete. 2024-02-21T23:02:53.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.959+0000 7f026ec00640 1 Processor -- start 2024-02-21T23:02:53.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.959+0000 7f026ec00640 1 -- start start 2024-02-21T23:02:53.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.960+0000 7f026ec00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f026806be80 0x7f02681ede30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:53.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.960+0000 7f026ec00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0268072830 0x7f026813cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:53.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.960+0000 7f026ec00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f02680a84b0 0x7f02680c5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:53.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.960+0000 7f026ec00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f026813b150 con 0x7f0268072830 2024-02-21T23:02:53.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.960+0000 7f026ec00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f026813e280 con 0x7f026806be80 2024-02-21T23:02:53.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.960+0000 7f026ec00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f026813b2d0 con 0x7f02680a84b0 2024-02-21T23:02:53.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.960+0000 7f026ce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0268072830 0x7f026813cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:53.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.960+0000 7f026ce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0268072830 0x7f026813cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58194/0 (socket says 172.21.15.28:58194) 2024-02-21T23:02:53.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.960+0000 7f026ce00640 1 -- 172.21.15.28:0/2532949710 learned_addr learned my addr 172.21.15.28:0/2532949710 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:02:53.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.961+0000 7f026d800640 1 --2- 172.21.15.28:0/2532949710 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f026806be80 0x7f02681ede30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:53.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.961+0000 7f026e200640 1 --2- 172.21.15.28:0/2532949710 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f02680a84b0 0x7f02680c5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:53.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.961+0000 7f026ce00640 1 -- 172.21.15.28:0/2532949710 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f02680a84b0 msgr2=0x7f02680c5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:53.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.961+0000 7f026ce00640 1 --2- 172.21.15.28:0/2532949710 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f02680a84b0 0x7f02680c5440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:53.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.961+0000 7f026ce00640 1 -- 172.21.15.28:0/2532949710 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f026806be80 msgr2=0x7f02681ede30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:53.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.961+0000 7f026ce00640 1 --2- 172.21.15.28:0/2532949710 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f026806be80 0x7f02681ede30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:53.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.961+0000 7f026ce00640 1 -- 172.21.15.28:0/2532949710 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02681ee370 con 0x7f0268072830 2024-02-21T23:02:53.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.961+0000 7f026ce00640 1 --2- 172.21.15.28:0/2532949710 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0268072830 0x7f026813cc20 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f025400c3c0 tx=0x7f025400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:53.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.961+0000 7f0265600640 1 -- 172.21.15.28:0/2532949710 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02540052d0 con 0x7f0268072830 2024-02-21T23:02:53.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.962+0000 7f0265600640 1 -- 172.21.15.28:0/2532949710 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0254005490 con 0x7f0268072830 2024-02-21T23:02:53.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.962+0000 7f0265600640 1 -- 172.21.15.28:0/2532949710 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0254007260 con 0x7f0268072830 2024-02-21T23:02:53.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.962+0000 7f026ec00640 1 -- 172.21.15.28:0/2532949710 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02681ee610 con 0x7f0268072830 2024-02-21T23:02:53.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.962+0000 7f026ec00640 1 -- 172.21.15.28:0/2532949710 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02681eebd0 con 0x7f0268072830 2024-02-21T23:02:53.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.963+0000 7f026ec00640 1 -- 172.21.15.28:0/2532949710 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f026804f290 con 0x7f0268072830 2024-02-21T23:02:53.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.964+0000 7f0265600640 1 -- 172.21.15.28:0/2532949710 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0254007400 con 0x7f0268072830 2024-02-21T23:02:53.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.965+0000 7f0265600640 1 --2- 172.21.15.28:0/2532949710 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f024806b4d0 0x7f024806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:53.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.965+0000 7f0265600640 1 -- 172.21.15.28:0/2532949710 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f0254089ac0 con 0x7f0268072830 2024-02-21T23:02:53.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.966+0000 7f026d800640 1 --2- 172.21.15.28:0/2532949710 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f024806b4d0 0x7f024806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:53.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.967+0000 7f026d800640 1 --2- 172.21.15.28:0/2532949710 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f024806b4d0 0x7f024806d990 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f025c006fd0 tx=0x7f025c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:53.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:53.970+0000 7f0265600640 1 -- 172.21.15.28:0/2532949710 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f0254058560 con 0x7f0268072830 2024-02-21T23:02:54.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.168+0000 7f026ec00640 1 -- 172.21.15.28:0/2532949710 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f0268067200 con 0x7f0268072830 2024-02-21T23:02:54.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.168+0000 7f0265600640 1 -- 172.21.15.28:0/2532949710 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) v1 ==== 103+0+11 (secure 0 0 0) 0x7f0254057e60 con 0x7f0268072830 2024-02-21T23:02:54.157 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-02-21T23:02:54.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.171+0000 7f026ec00640 1 -- 172.21.15.28:0/2532949710 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f024806b4d0 msgr2=0x7f024806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:54.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.171+0000 7f026ec00640 1 --2- 172.21.15.28:0/2532949710 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f024806b4d0 0x7f024806d990 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f025c006fd0 tx=0x7f025c008040 comp rx=0 tx=0).stop 2024-02-21T23:02:54.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.171+0000 7f026ec00640 1 -- 172.21.15.28:0/2532949710 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0268072830 msgr2=0x7f026813cc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:54.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.171+0000 7f026ec00640 1 --2- 172.21.15.28:0/2532949710 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0268072830 0x7f026813cc20 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f025400c3c0 tx=0x7f025400c880 comp rx=0 tx=0).stop 2024-02-21T23:02:54.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.171+0000 7f026ec00640 1 -- 172.21.15.28:0/2532949710 shutdown_connections 2024-02-21T23:02:54.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.171+0000 7f026ec00640 1 --2- 172.21.15.28:0/2532949710 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f024806b4d0 0x7f024806d990 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:54.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.171+0000 7f026ec00640 1 --2- 172.21.15.28:0/2532949710 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f02680a84b0 0x7f02680c5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:54.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.171+0000 7f026ec00640 1 --2- 172.21.15.28:0/2532949710 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0268072830 0x7f026813cc20 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:54.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.171+0000 7f026ec00640 1 --2- 172.21.15.28:0/2532949710 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f026806be80 0x7f02681ede30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:54.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.171+0000 7f026ec00640 1 -- 172.21.15.28:0/2532949710 >> 172.21.15.28:0/2532949710 conn(0x7f0268071b20 msgr2=0x7f02680c0550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:54.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.171+0000 7f026ec00640 1 -- 172.21.15.28:0/2532949710 shutdown_connections 2024-02-21T23:02:54.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.171+0000 7f026ec00640 1 -- 172.21.15.28:0/2532949710 wait complete. 2024-02-21T23:02:54.177 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-02-21T23:02:54.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.414+0000 7fb196a00640 1 Processor -- start 2024-02-21T23:02:54.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.414+0000 7fb196a00640 1 -- start start 2024-02-21T23:02:54.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.415+0000 7fb196a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fb1901524c0 0x7fb190155a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:54.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.415+0000 7fb196a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb1900c5040 0x7fb19006ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:54.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.416+0000 7fb196a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb19006df50 con 0x7fb1900c85c0 2024-02-21T23:02:54.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.416+0000 7fb196a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb19006d3b0 con 0x7fb1901524c0 2024-02-21T23:02:54.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.416+0000 7fb196a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1900c5520 con 0x7fb1900c5040 2024-02-21T23:02:54.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.416+0000 7fb196000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb1900c85c0 0x7fb1900c70d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42394/0 (socket says 172.21.15.28:42394) 2024-02-21T23:02:54.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.416+0000 7fb196000640 1 -- 172.21.15.28:0/543050307 learned_addr learned my addr 172.21.15.28:0/543050307 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:02:54.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.416+0000 7fb195600640 1 --2- 172.21.15.28:0/543050307 >> v2:172.21.15.81:3300/0 conn(0x7fb1901524c0 0x7fb190155a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:54.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.416+0000 7fb194c00640 1 --2- 172.21.15.28:0/543050307 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb1900c5040 0x7fb19006ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:54.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.416+0000 7fb18fe00640 1 -- 172.21.15.28:0/543050307 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 674930227 0 0) 0x7fb19006df50 con 0x7fb1900c85c0 2024-02-21T23:02:54.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.416+0000 7fb18fe00640 1 -- 172.21.15.28:0/543050307 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb178003520 con 0x7fb1900c85c0 2024-02-21T23:02:54.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.417+0000 7fb18fe00640 1 -- 172.21.15.28:0/543050307 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2411506611 0 0) 0x7fb178003520 con 0x7fb1900c85c0 2024-02-21T23:02:54.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.417+0000 7fb18fe00640 1 -- 172.21.15.28:0/543050307 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb1900c5040 msgr2=0x7fb19006ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:54.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.417+0000 7fb18fe00640 1 --2- 172.21.15.28:0/543050307 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb1900c5040 0x7fb19006ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:54.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.417+0000 7fb18fe00640 1 -- 172.21.15.28:0/543050307 >> v2:172.21.15.81:3300/0 conn(0x7fb1901524c0 msgr2=0x7fb190155a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:54.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.417+0000 7fb18fe00640 1 --2- 172.21.15.28:0/543050307 >> v2:172.21.15.81:3300/0 conn(0x7fb1901524c0 0x7fb190155a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:54.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.417+0000 7fb18fe00640 1 -- 172.21.15.28:0/543050307 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1900c5800 con 0x7fb1900c85c0 2024-02-21T23:02:54.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.417+0000 7fb18fe00640 1 -- 172.21.15.28:0/543050307 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fb1880019f0 con 0x7fb1900c85c0 2024-02-21T23:02:54.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.417+0000 7fb18fe00640 1 -- 172.21.15.28:0/543050307 >> v1:172.21.15.28:6789/0 conn(0x7fb1900c85c0 legacy=0x7fb1900c70d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:54.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.417+0000 7fb18fe00640 1 --2- 172.21.15.28:0/543050307 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb1900c5040 0x7fb1780039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:54.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.417+0000 7fb18fe00640 1 --2- 172.21.15.28:0/543050307 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb178003f80 0x7fb178004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:54.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.417+0000 7fb18fe00640 1 --2- 172.21.15.28:0/543050307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb178004d40 0x7fb1780051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:54.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.417+0000 7fb18fe00640 1 -- 172.21.15.28:0/543050307 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb19006d3b0 con 0x7fb178004d40 2024-02-21T23:02:54.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.417+0000 7fb18fe00640 1 -- 172.21.15.28:0/543050307 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1900c5520 con 0x7fb178003f80 2024-02-21T23:02:54.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.417+0000 7fb18fe00640 1 -- 172.21.15.28:0/543050307 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb17800baf0 con 0x7fb1900c5040 2024-02-21T23:02:54.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.417+0000 7fb196000640 1 --2- 172.21.15.28:0/543050307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb178004d40 0x7fb1780051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:54.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.417+0000 7fb195600640 1 --2- 172.21.15.28:0/543050307 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb1900c5040 0x7fb1780039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:54.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.418+0000 7fb194c00640 1 --2- 172.21.15.28:0/543050307 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb178003f80 0x7fb178004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:54.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.418+0000 7fb196000640 1 -- 172.21.15.28:0/543050307 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb1900c5040 msgr2=0x7fb1780039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:54.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.418+0000 7fb196000640 1 --2- 172.21.15.28:0/543050307 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb1900c5040 0x7fb1780039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:54.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.418+0000 7fb196000640 1 -- 172.21.15.28:0/543050307 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb178003f80 msgr2=0x7fb178004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:54.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.418+0000 7fb196000640 1 --2- 172.21.15.28:0/543050307 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb178003f80 0x7fb178004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:54.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.418+0000 7fb196000640 1 -- 172.21.15.28:0/543050307 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb178003520 con 0x7fb178004d40 2024-02-21T23:02:54.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.419+0000 7fb196000640 1 --2- 172.21.15.28:0/543050307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb178004d40 0x7fb1780051c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb18800a7b0 tx=0x7fb178005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa8ca11f431ebca server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:54.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.419+0000 7fb18fe00640 1 -- 172.21.15.28:0/543050307 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb18800a3d0 con 0x7fb178004d40 2024-02-21T23:02:54.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.419+0000 7fb18fe00640 1 -- 172.21.15.28:0/543050307 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb188034650 con 0x7fb178004d40 2024-02-21T23:02:54.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.419+0000 7fb18fe00640 1 -- 172.21.15.28:0/543050307 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb188034a40 con 0x7fb178004d40 2024-02-21T23:02:54.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.420+0000 7fb196a00640 1 -- 172.21.15.28:0/543050307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb178004d40 msgr2=0x7fb1780051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:54.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.420+0000 7fb196a00640 1 --2- 172.21.15.28:0/543050307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb178004d40 0x7fb1780051c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb18800a7b0 tx=0x7fb178005830 comp rx=0 tx=0).stop 2024-02-21T23:02:54.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.420+0000 7fb196000640 1 -- 172.21.15.28:0/543050307 reap_dead start 2024-02-21T23:02:54.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.420+0000 7fb196a00640 1 -- 172.21.15.28:0/543050307 shutdown_connections 2024-02-21T23:02:54.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.420+0000 7fb196a00640 1 -- 172.21.15.28:0/543050307 >> 172.21.15.28:0/543050307 conn(0x7fb1900ac2b0 msgr2=0x7fb1900ac6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:54.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.420+0000 7fb196a00640 1 -- 172.21.15.28:0/543050307 shutdown_connections 2024-02-21T23:02:54.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.420+0000 7fb196a00640 1 -- 172.21.15.28:0/543050307 wait complete. 2024-02-21T23:02:54.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.421+0000 7fb196a00640 1 Processor -- start 2024-02-21T23:02:54.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.422+0000 7fb196a00640 1 -- start start 2024-02-21T23:02:54.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.422+0000 7fb196a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb178004d40 0x7fb1900c40c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:54.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.422+0000 7fb196a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1900bee90 0x7fb1901ee4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:54.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.422+0000 7fb196a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb19014e1f0 0x7fb1901e78b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:54.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.422+0000 7fb196a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1901528a0 con 0x7fb1900bee90 2024-02-21T23:02:54.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.422+0000 7fb196a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb19006c950 con 0x7fb19014e1f0 2024-02-21T23:02:54.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.422+0000 7fb196a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1900c8a70 con 0x7fb178004d40 2024-02-21T23:02:54.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.423+0000 7fb194c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1900bee90 0x7fb1901ee4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:54.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.423+0000 7fb194c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1900bee90 0x7fb1901ee4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58220/0 (socket says 172.21.15.28:58220) 2024-02-21T23:02:54.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.423+0000 7fb194c00640 1 -- 172.21.15.28:0/3602696019 learned_addr learned my addr 172.21.15.28:0/3602696019 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:02:54.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.423+0000 7fb195600640 1 --2- 172.21.15.28:0/3602696019 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb178004d40 0x7fb1900c40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:54.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.423+0000 7fb196000640 1 --2- 172.21.15.28:0/3602696019 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb19014e1f0 0x7fb1901e78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:54.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.423+0000 7fb194c00640 1 -- 172.21.15.28:0/3602696019 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb178004d40 msgr2=0x7fb1900c40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:54.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.423+0000 7fb194c00640 1 --2- 172.21.15.28:0/3602696019 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb178004d40 0x7fb1900c40c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:54.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.423+0000 7fb194c00640 1 -- 172.21.15.28:0/3602696019 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb19014e1f0 msgr2=0x7fb1901e78b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:54.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.423+0000 7fb194c00640 1 --2- 172.21.15.28:0/3602696019 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb19014e1f0 0x7fb1901e78b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:54.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.423+0000 7fb194c00640 1 -- 172.21.15.28:0/3602696019 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1901eea30 con 0x7fb1900bee90 2024-02-21T23:02:54.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.423+0000 7fb196000640 1 --2- 172.21.15.28:0/3602696019 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb19014e1f0 0x7fb1901e78b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:02:54.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.424+0000 7fb194c00640 1 --2- 172.21.15.28:0/3602696019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1900bee90 0x7fb1901ee4f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb17c011600 tx=0x7fb17c011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:54.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.424+0000 7fb18d600640 1 -- 172.21.15.28:0/3602696019 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb17c00f8c0 con 0x7fb1900bee90 2024-02-21T23:02:54.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.424+0000 7fb18d600640 1 -- 172.21.15.28:0/3602696019 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb17c00fa60 con 0x7fb1900bee90 2024-02-21T23:02:54.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.424+0000 7fb18d600640 1 -- 172.21.15.28:0/3602696019 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb17c00fe10 con 0x7fb1900bee90 2024-02-21T23:02:54.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.424+0000 7fb196a00640 1 -- 172.21.15.28:0/3602696019 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb190158f40 con 0x7fb1900bee90 2024-02-21T23:02:54.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.424+0000 7fb196a00640 1 -- 172.21.15.28:0/3602696019 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1900c2980 con 0x7fb1900bee90 2024-02-21T23:02:54.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.425+0000 7fb196a00640 1 -- 172.21.15.28:0/3602696019 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb19004f290 con 0x7fb1900bee90 2024-02-21T23:02:54.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.426+0000 7fb18d600640 1 -- 172.21.15.28:0/3602696019 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb17c015460 con 0x7fb1900bee90 2024-02-21T23:02:54.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.427+0000 7fb18d600640 1 --2- 172.21.15.28:0/3602696019 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb17006b4d0 0x7fb17006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:54.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.427+0000 7fb18d600640 1 -- 172.21.15.28:0/3602696019 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fb17c08aa90 con 0x7fb1900bee90 2024-02-21T23:02:54.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.427+0000 7fb195600640 1 --2- 172.21.15.28:0/3602696019 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb17006b4d0 0x7fb17006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:54.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.429+0000 7fb195600640 1 --2- 172.21.15.28:0/3602696019 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb17006b4d0 0x7fb17006d990 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb19014ff00 tx=0x7fb184008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:54.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.432+0000 7fb18d600640 1 -- 172.21.15.28:0/3602696019 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb17c019030 con 0x7fb1900bee90 2024-02-21T23:02:54.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.646+0000 7fb196a00640 1 -- 172.21.15.28:0/3602696019 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fb19006dcc0 con 0x7fb1900bee90 2024-02-21T23:02:54.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.647+0000 7fb18d600640 1 -- 172.21.15.28:0/3602696019 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) v1 ==== 80+0+0 (secure 0 0 0) 0x7fb17c0595d0 con 0x7fb1900bee90 2024-02-21T23:02:54.635 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-02-21T23:02:54.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.650+0000 7fb196a00640 1 -- 172.21.15.28:0/3602696019 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb17006b4d0 msgr2=0x7fb17006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:54.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.650+0000 7fb196a00640 1 --2- 172.21.15.28:0/3602696019 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb17006b4d0 0x7fb17006d990 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb19014ff00 tx=0x7fb184008040 comp rx=0 tx=0).stop 2024-02-21T23:02:54.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.650+0000 7fb196a00640 1 -- 172.21.15.28:0/3602696019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1900bee90 msgr2=0x7fb1901ee4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:54.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.650+0000 7fb196a00640 1 --2- 172.21.15.28:0/3602696019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1900bee90 0x7fb1901ee4f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb17c011600 tx=0x7fb17c011ac0 comp rx=0 tx=0).stop 2024-02-21T23:02:54.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.650+0000 7fb196a00640 1 -- 172.21.15.28:0/3602696019 shutdown_connections 2024-02-21T23:02:54.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.650+0000 7fb196a00640 1 --2- 172.21.15.28:0/3602696019 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb17006b4d0 0x7fb17006d990 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:54.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.650+0000 7fb196a00640 1 --2- 172.21.15.28:0/3602696019 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb19014e1f0 0x7fb1901e78b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:54.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.650+0000 7fb196a00640 1 --2- 172.21.15.28:0/3602696019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1900bee90 0x7fb1901ee4f0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:54.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.650+0000 7fb196a00640 1 --2- 172.21.15.28:0/3602696019 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb178004d40 0x7fb1900c40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:54.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.650+0000 7fb196a00640 1 -- 172.21.15.28:0/3602696019 >> 172.21.15.28:0/3602696019 conn(0x7fb1900ac2b0 msgr2=0x7fb1900ac6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:54.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.650+0000 7fb196a00640 1 -- 172.21.15.28:0/3602696019 shutdown_connections 2024-02-21T23:02:54.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.650+0000 7fb196a00640 1 -- 172.21.15.28:0/3602696019 wait complete. 2024-02-21T23:02:54.656 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-21T23:02:54.656 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-21T23:02:54.702 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:02:54.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.980+0000 7fcea5c00640 1 Processor -- start 2024-02-21T23:02:54.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.980+0000 7fcea5c00640 1 -- start start 2024-02-21T23:02:54.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.981+0000 7fcea5c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fcea00c0fe0 0x7fcea006e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:54.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.981+0000 7fcea5c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcea0158580 0x7fcea014b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:54.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.981+0000 7fcea5c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcea00a8b40 con 0x7fcea006dff0 2024-02-21T23:02:54.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.981+0000 7fcea5c00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcea00a8e00 con 0x7fcea00c0fe0 2024-02-21T23:02:54.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.981+0000 7fcea5c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcea00a90e0 con 0x7fcea0158580 2024-02-21T23:02:54.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.982+0000 7fce9fe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fcea006dff0 0x7fcea0071350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41102/0 (socket says 172.21.15.28:41102) 2024-02-21T23:02:54.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.982+0000 7fce9fe00640 1 -- 172.21.15.28:0/4154944424 learned_addr learned my addr 172.21.15.28:0/4154944424 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:02:54.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.982+0000 7fce97e00640 1 --2- 172.21.15.28:0/4154944424 >> v2:172.21.15.81:3300/0 conn(0x7fcea00c0fe0 0x7fcea006e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:54.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.982+0000 7fcea5200640 1 --2- 172.21.15.28:0/4154944424 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcea0158580 0x7fcea014b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:54.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.982+0000 7fce9f400640 1 -- 172.21.15.28:0/4154944424 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3500727842 0 0) 0x7fcea00a8b40 con 0x7fcea006dff0 2024-02-21T23:02:54.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.982+0000 7fce9f400640 1 -- 172.21.15.28:0/4154944424 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce84003520 con 0x7fcea006dff0 2024-02-21T23:02:54.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.983+0000 7fcea5200640 1 -- 172.21.15.28:0/4154944424 >> v2:172.21.15.81:3300/0 conn(0x7fcea00c0fe0 msgr2=0x7fcea006e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:54.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.983+0000 7fcea5200640 1 --2- 172.21.15.28:0/4154944424 >> v2:172.21.15.81:3300/0 conn(0x7fcea00c0fe0 0x7fcea006e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:54.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.983+0000 7fcea5200640 1 -- 172.21.15.28:0/4154944424 >> v1:172.21.15.28:6789/0 conn(0x7fcea006dff0 legacy=0x7fcea0071350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:54.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.983+0000 7fcea5200640 1 -- 172.21.15.28:0/4154944424 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcea0072ed0 con 0x7fcea0158580 2024-02-21T23:02:54.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.984+0000 7fcea5200640 1 --2- 172.21.15.28:0/4154944424 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcea0158580 0x7fcea014b2b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fce98009d90 tx=0x7fce9802bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90cfda9238f04449 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:54.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.984+0000 7fce9f400640 1 -- 172.21.15.28:0/4154944424 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce98039070 con 0x7fcea0158580 2024-02-21T23:02:54.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.984+0000 7fce9f400640 1 -- 172.21.15.28:0/4154944424 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce9802cc50 con 0x7fcea0158580 2024-02-21T23:02:54.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.984+0000 7fce9f400640 1 -- 172.21.15.28:0/4154944424 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce98034440 con 0x7fcea0158580 2024-02-21T23:02:54.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.985+0000 7fcea5c00640 1 -- 172.21.15.28:0/4154944424 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcea0158580 msgr2=0x7fcea014b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:54.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.985+0000 7fcea5c00640 1 --2- 172.21.15.28:0/4154944424 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcea0158580 0x7fcea014b2b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fce98009d90 tx=0x7fce9802bea0 comp rx=0 tx=0).stop 2024-02-21T23:02:54.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.985+0000 7fcea5c00640 1 -- 172.21.15.28:0/4154944424 shutdown_connections 2024-02-21T23:02:54.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.985+0000 7fcea5c00640 1 --2- 172.21.15.28:0/4154944424 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcea0158580 0x7fcea014b2b0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:54.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.985+0000 7fcea5c00640 1 --2- 172.21.15.28:0/4154944424 >> v2:172.21.15.81:3300/0 conn(0x7fcea00c0fe0 0x7fcea006e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:54.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.985+0000 7fcea5c00640 1 -- 172.21.15.28:0/4154944424 >> 172.21.15.28:0/4154944424 conn(0x7fcea006b630 msgr2=0x7fcea00c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:54.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.985+0000 7fcea5c00640 1 -- 172.21.15.28:0/4154944424 shutdown_connections 2024-02-21T23:02:54.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.986+0000 7fcea5c00640 1 -- 172.21.15.28:0/4154944424 wait complete. 2024-02-21T23:02:54.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.987+0000 7fcea5c00640 1 Processor -- start 2024-02-21T23:02:54.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.987+0000 7fcea5c00640 1 -- start start 2024-02-21T23:02:54.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.988+0000 7fcea5c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcea006dff0 0x7fcea0153070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:54.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.988+0000 7fcea5c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcea00c0fe0 0x7fcea0153e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:54.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.988+0000 7fcea5c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcea0158580 0x7fcea0147960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:54.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.988+0000 7fcea5c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea01589d0 con 0x7fcea0158580 2024-02-21T23:02:54.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.988+0000 7fcea5c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea0073520 con 0x7fcea006dff0 2024-02-21T23:02:54.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.988+0000 7fcea5c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcea0158b50 con 0x7fcea00c0fe0 2024-02-21T23:02:54.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.989+0000 7fcea5200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcea0158580 0x7fcea0147960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:54.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.989+0000 7fcea5200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcea0158580 0x7fcea0147960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37846/0 (socket says 172.21.15.28:37846) 2024-02-21T23:02:54.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.989+0000 7fcea5200640 1 -- 172.21.15.28:0/1669185324 learned_addr learned my addr 172.21.15.28:0/1669185324 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:02:54.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.989+0000 7fce97e00640 1 --2- 172.21.15.28:0/1669185324 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcea00c0fe0 0x7fcea0153e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:54.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.989+0000 7fce9fe00640 1 --2- 172.21.15.28:0/1669185324 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcea006dff0 0x7fcea0153070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:54.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.989+0000 7fcea5200640 1 -- 172.21.15.28:0/1669185324 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcea00c0fe0 msgr2=0x7fcea0153e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:54.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.989+0000 7fcea5200640 1 --2- 172.21.15.28:0/1669185324 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcea00c0fe0 0x7fcea0153e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:54.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.989+0000 7fcea5200640 1 -- 172.21.15.28:0/1669185324 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcea006dff0 msgr2=0x7fcea0153070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:54.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.989+0000 7fcea5200640 1 --2- 172.21.15.28:0/1669185324 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcea006dff0 0x7fcea0153070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:54.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.989+0000 7fcea5200640 1 -- 172.21.15.28:0/1669185324 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcea00aaba0 con 0x7fcea0158580 2024-02-21T23:02:54.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.989+0000 7fcea5200640 1 --2- 172.21.15.28:0/1669185324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcea0158580 0x7fcea0147960 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fce98009d90 tx=0x7fce98004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:54.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.990+0000 7fce9cc00640 1 -- 172.21.15.28:0/1669185324 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce98039050 con 0x7fcea0158580 2024-02-21T23:02:54.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.990+0000 7fce9cc00640 1 -- 172.21.15.28:0/1669185324 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce98045070 con 0x7fcea0158580 2024-02-21T23:02:54.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.990+0000 7fcea5c00640 1 -- 172.21.15.28:0/1669185324 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcea01ea9e0 con 0x7fcea0158580 2024-02-21T23:02:54.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.990+0000 7fce9cc00640 1 -- 172.21.15.28:0/1669185324 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce98009450 con 0x7fcea0158580 2024-02-21T23:02:54.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.990+0000 7fcea5c00640 1 -- 172.21.15.28:0/1669185324 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcea01eb280 con 0x7fcea0158580 2024-02-21T23:02:54.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.991+0000 7fcea5c00640 1 -- 172.21.15.28:0/1669185324 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcea01eabb0 con 0x7fcea0158580 2024-02-21T23:02:54.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.992+0000 7fce9cc00640 1 -- 172.21.15.28:0/1669185324 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fce9803e610 con 0x7fcea0158580 2024-02-21T23:02:54.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.993+0000 7fce9cc00640 1 --2- 172.21.15.28:0/1669185324 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fce7c06b4d0 0x7fce7c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:54.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.993+0000 7fce9cc00640 1 -- 172.21.15.28:0/1669185324 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fce980acef0 con 0x7fcea0158580 2024-02-21T23:02:54.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.994+0000 7fce9fe00640 1 --2- 172.21.15.28:0/1669185324 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fce7c06b4d0 0x7fce7c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:54.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.994+0000 7fce9fe00640 1 --2- 172.21.15.28:0/1669185324 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fce7c06b4d0 0x7fce7c06d990 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fce90007fd0 tx=0x7fce90009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:54.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:54.997+0000 7fce9cc00640 1 -- 172.21.15.28:0/1669185324 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fce9807ba20 con 0x7fcea0158580 2024-02-21T23:02:55.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.201+0000 7fcea5c00640 1 -- 172.21.15.28:0/1669185324 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcea004f290 con 0x7fcea0158580 2024-02-21T23:02:55.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.204+0000 7fce9cc00640 1 -- 172.21.15.28:0/1669185324 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18454 (secure 0 0 0) 0x7fce9807b320 con 0x7fcea0158580 2024-02-21T23:02:55.196 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:02:55.196 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":24,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:02:42.900286+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:02:35.795158+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-21T23:02:36.840110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-02-21T23:02:40.441388+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:02:55.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.207+0000 7fcea5c00640 1 -- 172.21.15.28:0/1669185324 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fce7c06b4d0 msgr2=0x7fce7c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:55.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.207+0000 7fcea5c00640 1 --2- 172.21.15.28:0/1669185324 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fce7c06b4d0 0x7fce7c06d990 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fce90007fd0 tx=0x7fce90009040 comp rx=0 tx=0).stop 2024-02-21T23:02:55.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.207+0000 7fcea5c00640 1 -- 172.21.15.28:0/1669185324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcea0158580 msgr2=0x7fcea0147960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:55.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.207+0000 7fcea5c00640 1 --2- 172.21.15.28:0/1669185324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcea0158580 0x7fcea0147960 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fce98009d90 tx=0x7fce98004a40 comp rx=0 tx=0).stop 2024-02-21T23:02:55.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.207+0000 7fcea5c00640 1 -- 172.21.15.28:0/1669185324 shutdown_connections 2024-02-21T23:02:55.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.207+0000 7fcea5c00640 1 --2- 172.21.15.28:0/1669185324 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fce7c06b4d0 0x7fce7c06d990 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:55.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.207+0000 7fcea5c00640 1 --2- 172.21.15.28:0/1669185324 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcea00c0fe0 0x7fcea0153e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:55.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.207+0000 7fcea5c00640 1 --2- 172.21.15.28:0/1669185324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcea0158580 0x7fcea0147960 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:55.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.207+0000 7fcea5c00640 1 --2- 172.21.15.28:0/1669185324 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcea006dff0 0x7fcea0153070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:55.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.207+0000 7fcea5c00640 1 -- 172.21.15.28:0/1669185324 >> 172.21.15.28:0/1669185324 conn(0x7fcea006b630 msgr2=0x7fcea00bf930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:55.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.207+0000 7fcea5c00640 1 -- 172.21.15.28:0/1669185324 shutdown_connections 2024-02-21T23:02:55.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.207+0000 7fcea5c00640 1 -- 172.21.15.28:0/1669185324 wait complete. 2024-02-21T23:02:55.214 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-21T23:02:30.728329+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-21T23:02:35.795158+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-02-21T23:02:36.840110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-02-21T23:02:40.441388+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-02-21T23:02:55.215 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-21T23:02:55.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.444+0000 7f9332200640 1 Processor -- start 2024-02-21T23:02:55.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.444+0000 7f9332200640 1 -- start start 2024-02-21T23:02:55.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.445+0000 7f9332200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f932c0c7590 0x7f932c0c6c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:55.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.445+0000 7f9332200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f932c0c3030 0x7f932c0c3440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:55.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.445+0000 7f9332200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f932c159580 con 0x7f932c0c5cc0 2024-02-21T23:02:55.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.445+0000 7f9332200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f932c0c6230 con 0x7f932c0c7590 2024-02-21T23:02:55.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.445+0000 7f9332200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f932c1587c0 con 0x7f932c0c3030 2024-02-21T23:02:55.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.446+0000 7f9323e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f932c0c5cc0 0x7f932c1524e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41116/0 (socket says 172.21.15.28:41116) 2024-02-21T23:02:55.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.446+0000 7f9323e00640 1 -- 172.21.15.28:0/1954247253 learned_addr learned my addr 172.21.15.28:0/1954247253 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:02:55.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.446+0000 7f932b400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f932c0c7590 0x7f932c0c6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:55.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.446+0000 7f932be00640 1 --2- 172.21.15.28:0/1954247253 >> v2:172.21.15.81:3300/0 conn(0x7f932c0c3030 0x7f932c0c3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:55.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.446+0000 7f932b400640 1 -- 172.21.15.28:0/1954247253 >> v2:172.21.15.81:3300/0 conn(0x7f932c0c3030 msgr2=0x7f932c0c3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:55.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.446+0000 7f932b400640 1 --2- 172.21.15.28:0/1954247253 >> v2:172.21.15.81:3300/0 conn(0x7f932c0c3030 0x7f932c0c3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:55.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.446+0000 7f932b400640 1 -- 172.21.15.28:0/1954247253 >> v1:172.21.15.28:6789/0 conn(0x7f932c0c5cc0 legacy=0x7f932c1524e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:55.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.446+0000 7f932b400640 1 -- 172.21.15.28:0/1954247253 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f932c158aa0 con 0x7f932c0c7590 2024-02-21T23:02:55.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.446+0000 7f932aa00640 1 -- 172.21.15.28:0/1954247253 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1222208393 0 0) 0x7f932c159580 con 0x7f932c0c5cc0 2024-02-21T23:02:55.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.446+0000 7f932be00640 1 --2- 172.21.15.28:0/1954247253 >> v2:172.21.15.81:3300/0 conn(0x7f932c0c3030 0x7f932c0c3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:02:55.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.447+0000 7f932b400640 1 --2- 172.21.15.28:0/1954247253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f932c0c7590 0x7f932c0c6c60 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f932c073140 tx=0x7f931c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b613faf45b347022 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:55.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.448+0000 7f932aa00640 1 -- 172.21.15.28:0/1954247253 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f931c0315a0 con 0x7f932c0c7590 2024-02-21T23:02:55.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.448+0000 7f932aa00640 1 -- 172.21.15.28:0/1954247253 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f931c037070 con 0x7f932c0c7590 2024-02-21T23:02:55.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.448+0000 7f932aa00640 1 -- 172.21.15.28:0/1954247253 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f931c031960 con 0x7f932c0c7590 2024-02-21T23:02:55.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.448+0000 7f9332200640 1 -- 172.21.15.28:0/1954247253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f932c0c7590 msgr2=0x7f932c0c6c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:55.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.448+0000 7f9332200640 1 --2- 172.21.15.28:0/1954247253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f932c0c7590 0x7f932c0c6c60 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f932c073140 tx=0x7f931c02b840 comp rx=0 tx=0).stop 2024-02-21T23:02:55.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.448+0000 7f9332200640 1 -- 172.21.15.28:0/1954247253 shutdown_connections 2024-02-21T23:02:55.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.448+0000 7f9332200640 1 --2- 172.21.15.28:0/1954247253 >> v2:172.21.15.81:3300/0 conn(0x7f932c0c3030 0x7f932c0c3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:55.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.448+0000 7f9332200640 1 --2- 172.21.15.28:0/1954247253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f932c0c7590 0x7f932c0c6c60 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:55.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.448+0000 7f9332200640 1 -- 172.21.15.28:0/1954247253 >> 172.21.15.28:0/1954247253 conn(0x7f932c08afe0 msgr2=0x7f932c0725a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:55.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.448+0000 7f9332200640 1 -- 172.21.15.28:0/1954247253 shutdown_connections 2024-02-21T23:02:55.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.448+0000 7f9332200640 1 -- 172.21.15.28:0/1954247253 wait complete. 2024-02-21T23:02:55.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.450+0000 7f9332200640 1 Processor -- start 2024-02-21T23:02:55.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.450+0000 7f9332200640 1 -- start start 2024-02-21T23:02:55.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.451+0000 7f9332200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f932c0c3030 0x7f932c1571b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:55.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.451+0000 7f9332200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f932c0c5cc0 0x7f932c1554d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:55.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.451+0000 7f9332200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f932c0c7590 0x7f932c155a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:55.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.451+0000 7f9332200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f932c0c6720 con 0x7f932c0c5cc0 2024-02-21T23:02:55.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.451+0000 7f9332200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f932c159060 con 0x7f932c0c7590 2024-02-21T23:02:55.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.451+0000 7f9332200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f932c0c68a0 con 0x7f932c0c3030 2024-02-21T23:02:55.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.451+0000 7f9323e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f932c0c5cc0 0x7f932c1554d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:55.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.452+0000 7f9323e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f932c0c5cc0 0x7f932c1554d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37854/0 (socket says 172.21.15.28:37854) 2024-02-21T23:02:55.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.452+0000 7f9323e00640 1 -- 172.21.15.28:0/1149231490 learned_addr learned my addr 172.21.15.28:0/1149231490 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:02:55.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.452+0000 7f932be00640 1 --2- 172.21.15.28:0/1149231490 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f932c0c7590 0x7f932c155a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:55.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.452+0000 7f932b400640 1 --2- 172.21.15.28:0/1149231490 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f932c0c3030 0x7f932c1571b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:55.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.452+0000 7f9323e00640 1 -- 172.21.15.28:0/1149231490 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f932c0c3030 msgr2=0x7f932c1571b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:55.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.452+0000 7f9323e00640 1 --2- 172.21.15.28:0/1149231490 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f932c0c3030 0x7f932c1571b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:55.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.452+0000 7f9323e00640 1 -- 172.21.15.28:0/1149231490 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f932c0c7590 msgr2=0x7f932c155a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:55.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.452+0000 7f9323e00640 1 --2- 172.21.15.28:0/1149231490 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f932c0c7590 0x7f932c155a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:55.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.452+0000 7f9323e00640 1 -- 172.21.15.28:0/1149231490 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f932c152d40 con 0x7f932c0c5cc0 2024-02-21T23:02:55.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.452+0000 7f932be00640 1 --2- 172.21.15.28:0/1149231490 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f932c0c7590 0x7f932c155a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:02:55.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.452+0000 7f9323e00640 1 --2- 172.21.15.28:0/1149231490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f932c0c5cc0 0x7f932c1554d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f931400c500 tx=0x7f931400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:55.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.453+0000 7f9323400640 1 -- 172.21.15.28:0/1149231490 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9314018070 con 0x7f932c0c5cc0 2024-02-21T23:02:55.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.453+0000 7f9323400640 1 -- 172.21.15.28:0/1149231490 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9314007580 con 0x7f932c0c5cc0 2024-02-21T23:02:55.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.453+0000 7f9323400640 1 -- 172.21.15.28:0/1149231490 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9314004a20 con 0x7f932c0c5cc0 2024-02-21T23:02:55.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.453+0000 7f9332200640 1 -- 172.21.15.28:0/1149231490 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f932c0c0630 con 0x7f932c0c5cc0 2024-02-21T23:02:55.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.453+0000 7f9332200640 1 -- 172.21.15.28:0/1149231490 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f932c1569b0 con 0x7f932c0c5cc0 2024-02-21T23:02:55.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.454+0000 7f9332200640 1 -- 172.21.15.28:0/1149231490 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f932c067f30 con 0x7f932c0c5cc0 2024-02-21T23:02:55.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.455+0000 7f9323400640 1 -- 172.21.15.28:0/1149231490 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9314004bc0 con 0x7f932c0c5cc0 2024-02-21T23:02:55.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.456+0000 7f9323400640 1 --2- 172.21.15.28:0/1149231490 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f930806b4d0 0x7f930806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:55.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.456+0000 7f9323400640 1 -- 172.21.15.28:0/1149231490 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f9314089730 con 0x7f932c0c5cc0 2024-02-21T23:02:55.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.456+0000 7f932b400640 1 --2- 172.21.15.28:0/1149231490 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f930806b4d0 0x7f930806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:55.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.457+0000 7f932b400640 1 --2- 172.21.15.28:0/1149231490 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f930806b4d0 0x7f930806d990 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f931c036000 tx=0x7f931c03d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:55.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.460+0000 7f9323400640 1 -- 172.21.15.28:0/1149231490 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f9314058260 con 0x7f932c0c5cc0 2024-02-21T23:02:55.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.662+0000 7f9332200640 1 -- 172.21.15.28:0/1149231490 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f932c0b0800 con 0x7f932c0c5cc0 2024-02-21T23:02:55.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.663+0000 7f9323400640 1 -- 172.21.15.28:0/1149231490 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9314057b60 con 0x7f932c0c5cc0 2024-02-21T23:02:55.652 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-02-21T23:02:55.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.665+0000 7f9332200640 1 -- 172.21.15.28:0/1149231490 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f930806b4d0 msgr2=0x7f930806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:55.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.665+0000 7f9332200640 1 --2- 172.21.15.28:0/1149231490 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f930806b4d0 0x7f930806d990 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f931c036000 tx=0x7f931c03d040 comp rx=0 tx=0).stop 2024-02-21T23:02:55.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.666+0000 7f9332200640 1 -- 172.21.15.28:0/1149231490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f932c0c5cc0 msgr2=0x7f932c1554d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:55.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.666+0000 7f9332200640 1 --2- 172.21.15.28:0/1149231490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f932c0c5cc0 0x7f932c1554d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f931400c500 tx=0x7f931400c9c0 comp rx=0 tx=0).stop 2024-02-21T23:02:55.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.666+0000 7f9332200640 1 -- 172.21.15.28:0/1149231490 shutdown_connections 2024-02-21T23:02:55.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.666+0000 7f9332200640 1 --2- 172.21.15.28:0/1149231490 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f930806b4d0 0x7f930806d990 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:55.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.666+0000 7f9332200640 1 --2- 172.21.15.28:0/1149231490 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f932c0c7590 0x7f932c155a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:55.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.666+0000 7f9332200640 1 --2- 172.21.15.28:0/1149231490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f932c0c5cc0 0x7f932c1554d0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:55.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.666+0000 7f9332200640 1 --2- 172.21.15.28:0/1149231490 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f932c0c3030 0x7f932c1571b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:55.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.666+0000 7f9332200640 1 -- 172.21.15.28:0/1149231490 >> 172.21.15.28:0/1149231490 conn(0x7f932c08afe0 msgr2=0x7f932c151550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:55.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.666+0000 7f9332200640 1 -- 172.21.15.28:0/1149231490 shutdown_connections 2024-02-21T23:02:55.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.666+0000 7f9332200640 1 -- 172.21.15.28:0/1149231490 wait complete. 2024-02-21T23:02:55.672 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-21T23:02:55.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.913+0000 7fe76b200640 1 Processor -- start 2024-02-21T23:02:55.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.913+0000 7fe76b200640 1 -- start start 2024-02-21T23:02:55.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.915+0000 7fe76b200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe76413e520 0x7fe76406c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:55.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.915+0000 7fe76b200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe764158550 con 0x7fe76406dbc0 2024-02-21T23:02:55.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.915+0000 7fe76b200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7641587d0 con 0x7fe76413e0c0 2024-02-21T23:02:55.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.915+0000 7fe76b200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7640c89a0 con 0x7fe76413e520 2024-02-21T23:02:55.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.915+0000 7fe76a800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe76413e520 0x7fe76406c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:55.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.915+0000 7fe769400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe76406dbc0 0x7fe76406c280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41128/0 (socket says 172.21.15.28:41128) 2024-02-21T23:02:55.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.916+0000 7fe769400640 1 -- 172.21.15.28:0/1698481318 learned_addr learned my addr 172.21.15.28:0/1698481318 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:02:55.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.916+0000 7fe76a800640 1 -- 172.21.15.28:0/1698481318 >> v1:172.21.15.81:6789/0 conn(0x7fe76413e0c0 legacy=0x7fe7640c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:55.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.916+0000 7fe76a800640 1 -- 172.21.15.28:0/1698481318 >> v1:172.21.15.28:6789/0 conn(0x7fe76406dbc0 legacy=0x7fe76406c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:55.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.916+0000 7fe76a800640 1 -- 172.21.15.28:0/1698481318 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe764158ab0 con 0x7fe76413e520 2024-02-21T23:02:55.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.917+0000 7fe76a800640 1 --2- 172.21.15.28:0/1698481318 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe76413e520 0x7fe76406c990 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe7600097a0 tx=0x7fe76002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b6c1d17602525ba server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:55.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.918+0000 7fe768a00640 1 -- 172.21.15.28:0/1698481318 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7600315a0 con 0x7fe76413e520 2024-02-21T23:02:55.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.918+0000 7fe768a00640 1 -- 172.21.15.28:0/1698481318 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe760037070 con 0x7fe76413e520 2024-02-21T23:02:55.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.918+0000 7fe768a00640 1 -- 172.21.15.28:0/1698481318 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe760031960 con 0x7fe76413e520 2024-02-21T23:02:55.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.918+0000 7fe76b200640 1 -- 172.21.15.28:0/1698481318 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe76413e520 msgr2=0x7fe76406c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:55.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.918+0000 7fe76b200640 1 --2- 172.21.15.28:0/1698481318 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe76413e520 0x7fe76406c990 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe7600097a0 tx=0x7fe76002b840 comp rx=0 tx=0).stop 2024-02-21T23:02:55.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.918+0000 7fe76b200640 1 -- 172.21.15.28:0/1698481318 shutdown_connections 2024-02-21T23:02:55.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.918+0000 7fe76b200640 1 --2- 172.21.15.28:0/1698481318 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe76413e520 0x7fe76406c990 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:55.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.918+0000 7fe76b200640 1 -- 172.21.15.28:0/1698481318 >> 172.21.15.28:0/1698481318 conn(0x7fe764073110 msgr2=0x7fe7640ac490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:55.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.918+0000 7fe76b200640 1 -- 172.21.15.28:0/1698481318 shutdown_connections 2024-02-21T23:02:55.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.918+0000 7fe76b200640 1 -- 172.21.15.28:0/1698481318 wait complete. 2024-02-21T23:02:55.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.920+0000 7fe76b200640 1 Processor -- start 2024-02-21T23:02:55.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.920+0000 7fe76b200640 1 -- start start 2024-02-21T23:02:55.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.921+0000 7fe76b200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe76406dbc0 0x7fe7641535a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:55.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.921+0000 7fe76b200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe76413e0c0 0x7fe764151dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:55.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.921+0000 7fe76b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe76413e520 0x7fe7641ebe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:55.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.921+0000 7fe76b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe76406ced0 con 0x7fe76413e520 2024-02-21T23:02:55.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.921+0000 7fe76b200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe764159070 con 0x7fe76413e0c0 2024-02-21T23:02:55.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.921+0000 7fe76b200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7640c78c0 con 0x7fe76406dbc0 2024-02-21T23:02:55.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.921+0000 7fe76a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe76413e520 0x7fe7641ebe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:55.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.921+0000 7fe76a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe76413e520 0x7fe7641ebe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37866/0 (socket says 172.21.15.28:37866) 2024-02-21T23:02:55.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.921+0000 7fe76a800640 1 -- 172.21.15.28:0/4190084583 learned_addr learned my addr 172.21.15.28:0/4190084583 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:02:55.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.922+0000 7fe769e00640 1 --2- 172.21.15.28:0/4190084583 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe76406dbc0 0x7fe7641535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:55.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.922+0000 7fe769400640 1 --2- 172.21.15.28:0/4190084583 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe76413e0c0 0x7fe764151dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:55.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.922+0000 7fe76a800640 1 -- 172.21.15.28:0/4190084583 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe76406dbc0 msgr2=0x7fe7641535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:55.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.922+0000 7fe76a800640 1 --2- 172.21.15.28:0/4190084583 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe76406dbc0 0x7fe7641535a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:55.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.922+0000 7fe76a800640 1 -- 172.21.15.28:0/4190084583 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe76413e0c0 msgr2=0x7fe764151dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:55.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.922+0000 7fe76a800640 1 --2- 172.21.15.28:0/4190084583 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe76413e0c0 0x7fe764151dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:55.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.922+0000 7fe76a800640 1 -- 172.21.15.28:0/4190084583 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7640ac940 con 0x7fe76413e520 2024-02-21T23:02:55.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.922+0000 7fe76a800640 1 --2- 172.21.15.28:0/4190084583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe76413e520 0x7fe7641ebe30 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe760034040 tx=0x7fe7600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:55.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.922+0000 7fe75a000640 1 -- 172.21.15.28:0/4190084583 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe760032840 con 0x7fe76413e520 2024-02-21T23:02:55.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.922+0000 7fe75a000640 1 -- 172.21.15.28:0/4190084583 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7600329e0 con 0x7fe76413e520 2024-02-21T23:02:55.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.922+0000 7fe75a000640 1 -- 172.21.15.28:0/4190084583 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe760032dd0 con 0x7fe76413e520 2024-02-21T23:02:55.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.922+0000 7fe76b200640 1 -- 172.21.15.28:0/4190084583 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7641553b0 con 0x7fe76413e520 2024-02-21T23:02:55.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.923+0000 7fe76b200640 1 -- 172.21.15.28:0/4190084583 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe764155970 con 0x7fe76413e520 2024-02-21T23:02:55.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.924+0000 7fe76b200640 1 -- 172.21.15.28:0/4190084583 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe76404f290 con 0x7fe76413e520 2024-02-21T23:02:55.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.925+0000 7fe75a000640 1 -- 172.21.15.28:0/4190084583 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe76004d3f0 con 0x7fe76413e520 2024-02-21T23:02:55.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.926+0000 7fe75a000640 1 --2- 172.21.15.28:0/4190084583 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe74806b4d0 0x7fe74806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:55.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.926+0000 7fe75a000640 1 -- 172.21.15.28:0/4190084583 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fe76002f030 con 0x7fe76413e520 2024-02-21T23:02:55.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.926+0000 7fe769e00640 1 --2- 172.21.15.28:0/4190084583 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe74806b4d0 0x7fe74806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:55.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.927+0000 7fe769e00640 1 --2- 172.21.15.28:0/4190084583 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe74806b4d0 0x7fe74806d990 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe75c0012e0 tx=0x7fe75c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:55.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:55.930+0000 7fe75a000640 1 -- 172.21.15.28:0/4190084583 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe760040070 con 0x7fe76413e520 2024-02-21T23:02:56.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.128+0000 7fe76b200640 1 -- 172.21.15.28:0/4190084583 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe76406d9a0 con 0x7fe76413e520 2024-02-21T23:02:56.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.129+0000 7fe75a000640 1 -- 172.21.15.28:0/4190084583 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe76007a520 con 0x7fe76413e520 2024-02-21T23:02:56.118 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-02-21T23:02:56.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.132+0000 7fe76b200640 1 -- 172.21.15.28:0/4190084583 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe74806b4d0 msgr2=0x7fe74806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:56.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.132+0000 7fe76b200640 1 --2- 172.21.15.28:0/4190084583 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe74806b4d0 0x7fe74806d990 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe75c0012e0 tx=0x7fe75c009040 comp rx=0 tx=0).stop 2024-02-21T23:02:56.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.132+0000 7fe76b200640 1 -- 172.21.15.28:0/4190084583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe76413e520 msgr2=0x7fe7641ebe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:56.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.132+0000 7fe76b200640 1 --2- 172.21.15.28:0/4190084583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe76413e520 0x7fe7641ebe30 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe760034040 tx=0x7fe7600046c0 comp rx=0 tx=0).stop 2024-02-21T23:02:56.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.132+0000 7fe76b200640 1 -- 172.21.15.28:0/4190084583 shutdown_connections 2024-02-21T23:02:56.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.132+0000 7fe76b200640 1 --2- 172.21.15.28:0/4190084583 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe74806b4d0 0x7fe74806d990 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:56.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.132+0000 7fe76b200640 1 --2- 172.21.15.28:0/4190084583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe76413e520 0x7fe7641ebe30 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:56.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.132+0000 7fe76b200640 1 --2- 172.21.15.28:0/4190084583 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe76413e0c0 0x7fe764151dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:56.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.132+0000 7fe76b200640 1 --2- 172.21.15.28:0/4190084583 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe76406dbc0 0x7fe7641535a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:56.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.132+0000 7fe76b200640 1 -- 172.21.15.28:0/4190084583 >> 172.21.15.28:0/4190084583 conn(0x7fe764073110 msgr2=0x7fe764157a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:56.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.132+0000 7fe76b200640 1 -- 172.21.15.28:0/4190084583 shutdown_connections 2024-02-21T23:02:56.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.132+0000 7fe76b200640 1 -- 172.21.15.28:0/4190084583 wait complete. 2024-02-21T23:02:56.141 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-21T23:02:56.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.375+0000 7f5028a00640 1 Processor -- start 2024-02-21T23:02:56.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.375+0000 7f5028a00640 1 -- start start 2024-02-21T23:02:56.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.377+0000 7f5028a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5024158dc0 0x7f50241609f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:56.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.377+0000 7f5028a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5024159230 con 0x7f50240c6220 2024-02-21T23:02:56.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.377+0000 7f5028a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50240c20d0 con 0x7f50240c4960 2024-02-21T23:02:56.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.377+0000 7f5028a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50240a62e0 con 0x7f5024158dc0 2024-02-21T23:02:56.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.377+0000 7f5019400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f50240c6220 0x7f50241586b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41136/0 (socket says 172.21.15.28:41136) 2024-02-21T23:02:56.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.377+0000 7f5019400640 1 -- 172.21.15.28:0/3901699372 learned_addr learned my addr 172.21.15.28:0/3901699372 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:02:56.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.377+0000 7f5022800640 1 --2- 172.21.15.28:0/3901699372 >> v2:172.21.15.28:3300/0 conn(0x7f5024158dc0 0x7f50241609f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:56.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.378+0000 7f5021400640 1 -- 172.21.15.28:0/3901699372 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2263731357 0 0) 0x7f500c001110 con 0x7f50240c6220 2024-02-21T23:02:56.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.378+0000 7f5021400640 1 -- 172.21.15.28:0/3901699372 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5008003520 con 0x7f50240c6220 2024-02-21T23:02:56.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.378+0000 7f5021400640 1 -- 172.21.15.28:0/3901699372 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 268981181 0 0) 0x7f50240c20d0 con 0x7f50240c4960 2024-02-21T23:02:56.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.378+0000 7f5022800640 1 -- 172.21.15.28:0/3901699372 >> v1:172.21.15.81:6789/0 conn(0x7f50240c4960 legacy=0x7f50241524f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:56.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.378+0000 7f5022800640 1 -- 172.21.15.28:0/3901699372 >> v1:172.21.15.28:6789/0 conn(0x7f50240c6220 legacy=0x7f50241586b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:56.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.378+0000 7f5022800640 1 -- 172.21.15.28:0/3901699372 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50240c1290 con 0x7f5024158dc0 2024-02-21T23:02:56.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.380+0000 7f5022800640 1 --2- 172.21.15.28:0/3901699372 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5024158dc0 0x7f50241609f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f501c009d90 tx=0x7f501c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfe676bf676523e3 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:56.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.380+0000 7f5021400640 1 -- 172.21.15.28:0/3901699372 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f501c039070 con 0x7f5024158dc0 2024-02-21T23:02:56.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.380+0000 7f5021400640 1 -- 172.21.15.28:0/3901699372 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f501c02cc60 con 0x7f5024158dc0 2024-02-21T23:02:56.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.380+0000 7f5021400640 1 -- 172.21.15.28:0/3901699372 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f501c034660 con 0x7f5024158dc0 2024-02-21T23:02:56.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.380+0000 7f5028a00640 1 -- 172.21.15.28:0/3901699372 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5024158dc0 msgr2=0x7f50241609f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:56.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.380+0000 7f5028a00640 1 --2- 172.21.15.28:0/3901699372 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5024158dc0 0x7f50241609f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f501c009d90 tx=0x7f501c02beb0 comp rx=0 tx=0).stop 2024-02-21T23:02:56.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.381+0000 7f5028a00640 1 -- 172.21.15.28:0/3901699372 shutdown_connections 2024-02-21T23:02:56.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.381+0000 7f5028a00640 1 --2- 172.21.15.28:0/3901699372 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5024158dc0 0x7f50241609f0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:56.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.381+0000 7f5028a00640 1 -- 172.21.15.28:0/3901699372 >> 172.21.15.28:0/3901699372 conn(0x7f50240a85a0 msgr2=0x7f50240a8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:56.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.381+0000 7f5028a00640 1 -- 172.21.15.28:0/3901699372 shutdown_connections 2024-02-21T23:02:56.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.381+0000 7f5028a00640 1 -- 172.21.15.28:0/3901699372 wait complete. 2024-02-21T23:02:56.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.382+0000 7f5028a00640 1 Processor -- start 2024-02-21T23:02:56.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.383+0000 7f5028a00640 1 -- start start 2024-02-21T23:02:56.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.383+0000 7f5028a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50240c4960 0x7f502414f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:56.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.383+0000 7f5028a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f50240c6220 0x7f50241499a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:56.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.383+0000 7f5028a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5024158dc0 0x7f50240ab6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:56.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.383+0000 7f5028a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50240c1bb0 con 0x7f50240c4960 2024-02-21T23:02:56.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.383+0000 7f5028a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50240c1960 con 0x7f5024158dc0 2024-02-21T23:02:56.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.383+0000 7f5028a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50240a5f20 con 0x7f50240c6220 2024-02-21T23:02:56.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.384+0000 7f5021e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50240c4960 0x7f502414f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:56.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.384+0000 7f5021e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50240c4960 0x7f502414f3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37872/0 (socket says 172.21.15.28:37872) 2024-02-21T23:02:56.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.384+0000 7f5021e00640 1 -- 172.21.15.28:0/3766984770 learned_addr learned my addr 172.21.15.28:0/3766984770 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:02:56.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.384+0000 7f5021e00640 1 -- 172.21.15.28:0/3766984770 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f50240c6220 msgr2=0x7f50241499a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:56.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.384+0000 7f5021e00640 1 --2- 172.21.15.28:0/3766984770 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f50240c6220 0x7f50241499a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:56.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.384+0000 7f5021e00640 1 -- 172.21.15.28:0/3766984770 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5024158dc0 msgr2=0x7f50240ab6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:56.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.384+0000 7f5021e00640 1 --2- 172.21.15.28:0/3766984770 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5024158dc0 0x7f50240ab6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:56.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.384+0000 7f5021e00640 1 -- 172.21.15.28:0/3766984770 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50241ee4f0 con 0x7f50240c4960 2024-02-21T23:02:56.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.385+0000 7f5021e00640 1 --2- 172.21.15.28:0/3766984770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50240c4960 0x7f502414f3a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f501400c4b0 tx=0x7f501400c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:56.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.385+0000 7f501aa00640 1 -- 172.21.15.28:0/3766984770 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5014017070 con 0x7f50240c4960 2024-02-21T23:02:56.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.385+0000 7f501aa00640 1 -- 172.21.15.28:0/3766984770 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5014005480 con 0x7f50240c4960 2024-02-21T23:02:56.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.385+0000 7f501aa00640 1 -- 172.21.15.28:0/3766984770 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5014003830 con 0x7f50240c4960 2024-02-21T23:02:56.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.385+0000 7f5028a00640 1 -- 172.21.15.28:0/3766984770 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50241518d0 con 0x7f50240c4960 2024-02-21T23:02:56.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.385+0000 7f5028a00640 1 -- 172.21.15.28:0/3766984770 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50241589d0 con 0x7f50240c4960 2024-02-21T23:02:56.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.386+0000 7f5028a00640 1 -- 172.21.15.28:0/3766984770 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5024067f30 con 0x7f50240c4960 2024-02-21T23:02:56.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.387+0000 7f501aa00640 1 -- 172.21.15.28:0/3766984770 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f501401a070 con 0x7f50240c4960 2024-02-21T23:02:56.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.388+0000 7f501aa00640 1 --2- 172.21.15.28:0/3766984770 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f500006b4d0 0x7f500006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:56.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.388+0000 7f501aa00640 1 -- 172.21.15.28:0/3766984770 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f50140898f0 con 0x7f50240c4960 2024-02-21T23:02:56.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.388+0000 7f5019400640 1 --2- 172.21.15.28:0/3766984770 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f500006b4d0 0x7f500006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:56.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.389+0000 7f5019400640 1 --2- 172.21.15.28:0/3766984770 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f500006b4d0 0x7f500006d990 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f500c000f60 tx=0x7f500c001110 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:56.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.392+0000 7f501aa00640 1 -- 172.21.15.28:0/3766984770 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f5014058420 con 0x7f50240c4960 2024-02-21T23:02:56.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.593+0000 7f5028a00640 1 -- 172.21.15.28:0/3766984770 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f502406d780 con 0x7f50240c4960 2024-02-21T23:02:56.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.593+0000 7f501aa00640 1 -- 172.21.15.28:0/3766984770 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5014057d20 con 0x7f50240c4960 2024-02-21T23:02:56.582 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-02-21T23:02:56.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.596+0000 7f5028a00640 1 -- 172.21.15.28:0/3766984770 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f500006b4d0 msgr2=0x7f500006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:56.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.596+0000 7f5028a00640 1 --2- 172.21.15.28:0/3766984770 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f500006b4d0 0x7f500006d990 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f500c000f60 tx=0x7f500c001110 comp rx=0 tx=0).stop 2024-02-21T23:02:56.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.596+0000 7f5028a00640 1 -- 172.21.15.28:0/3766984770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50240c4960 msgr2=0x7f502414f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:56.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.596+0000 7f5028a00640 1 --2- 172.21.15.28:0/3766984770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50240c4960 0x7f502414f3a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f501400c4b0 tx=0x7f501400c970 comp rx=0 tx=0).stop 2024-02-21T23:02:56.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.596+0000 7f5028a00640 1 -- 172.21.15.28:0/3766984770 shutdown_connections 2024-02-21T23:02:56.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.596+0000 7f5028a00640 1 --2- 172.21.15.28:0/3766984770 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f500006b4d0 0x7f500006d990 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:56.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.596+0000 7f5028a00640 1 --2- 172.21.15.28:0/3766984770 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f50240c6220 0x7f50241499a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:56.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.596+0000 7f5028a00640 1 --2- 172.21.15.28:0/3766984770 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5024158dc0 0x7f50240ab6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:56.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.596+0000 7f5028a00640 1 --2- 172.21.15.28:0/3766984770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50240c4960 0x7f502414f3a0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:56.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.596+0000 7f5028a00640 1 -- 172.21.15.28:0/3766984770 >> 172.21.15.28:0/3766984770 conn(0x7f50240a85a0 msgr2=0x7f502414c2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:56.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.597+0000 7f5028a00640 1 -- 172.21.15.28:0/3766984770 shutdown_connections 2024-02-21T23:02:56.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.597+0000 7f5028a00640 1 -- 172.21.15.28:0/3766984770 wait complete. 2024-02-21T23:02:56.605 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-02-21T23:02:56.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.838+0000 7fd80e800640 1 Processor -- start 2024-02-21T23:02:56.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.839+0000 7fd80e800640 1 -- start start 2024-02-21T23:02:56.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.840+0000 7fd80e800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fd80806dbc0 0x7fd80806c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:56.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.840+0000 7fd80e800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd808158550 con 0x7fd80813e520 2024-02-21T23:02:56.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.840+0000 7fd80e800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8081587d0 con 0x7fd80813e0c0 2024-02-21T23:02:56.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.840+0000 7fd80e800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8080c89a0 con 0x7fd80806dbc0 2024-02-21T23:02:56.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.841+0000 7fd80ca00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fd80806dbc0 0x7fd80806c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:56.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.841+0000 7fd80de00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd80813e520 0x7fd80806c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41146/0 (socket says 172.21.15.28:41146) 2024-02-21T23:02:56.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.841+0000 7fd80de00640 1 -- 172.21.15.28:0/1280253876 learned_addr learned my addr 172.21.15.28:0/1280253876 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:02:56.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.841+0000 7fd80d400640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fd80813e0c0 0x7fd8080c71b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.28:59254/0 (socket says 172.21.15.28:59254) 2024-02-21T23:02:56.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.841+0000 7fd807e00640 1 -- 172.21.15.28:0/1280253876 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3071129622 0 0) 0x7fd800001110 con 0x7fd80813e520 2024-02-21T23:02:56.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.841+0000 7fd807e00640 1 -- 172.21.15.28:0/1280253876 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7f0003520 con 0x7fd80813e520 2024-02-21T23:02:56.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.841+0000 7fd80ca00640 1 -- 172.21.15.28:0/1280253876 >> v1:172.21.15.81:6789/0 conn(0x7fd80813e0c0 legacy=0x7fd8080c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:56.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.841+0000 7fd80ca00640 1 -- 172.21.15.28:0/1280253876 >> v1:172.21.15.28:6789/0 conn(0x7fd80813e520 legacy=0x7fd80806c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:56.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.841+0000 7fd80ca00640 1 -- 172.21.15.28:0/1280253876 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd808158ab0 con 0x7fd80806dbc0 2024-02-21T23:02:56.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.843+0000 7fd80ca00640 1 --2- 172.21.15.28:0/1280253876 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd80806dbc0 0x7fd80806c280 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc009d90 tx=0x7fd7fc02be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5a40ce7b409deb2 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:56.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.843+0000 7fd807e00640 1 -- 172.21.15.28:0/1280253876 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7fc039070 con 0x7fd80806dbc0 2024-02-21T23:02:56.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.843+0000 7fd807e00640 1 -- 172.21.15.28:0/1280253876 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7fc02cbd0 con 0x7fd80806dbc0 2024-02-21T23:02:56.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.843+0000 7fd807e00640 1 -- 172.21.15.28:0/1280253876 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7fc034440 con 0x7fd80806dbc0 2024-02-21T23:02:56.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.844+0000 7fd80e800640 1 -- 172.21.15.28:0/1280253876 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd80806dbc0 msgr2=0x7fd80806c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:56.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.844+0000 7fd80e800640 1 --2- 172.21.15.28:0/1280253876 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd80806dbc0 0x7fd80806c280 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc009d90 tx=0x7fd7fc02be20 comp rx=0 tx=0).stop 2024-02-21T23:02:56.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.844+0000 7fd80e800640 1 -- 172.21.15.28:0/1280253876 shutdown_connections 2024-02-21T23:02:56.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.844+0000 7fd80e800640 1 --2- 172.21.15.28:0/1280253876 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd80806dbc0 0x7fd80806c280 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:56.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.844+0000 7fd80e800640 1 -- 172.21.15.28:0/1280253876 >> 172.21.15.28:0/1280253876 conn(0x7fd808073110 msgr2=0x7fd8080ac490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:56.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.844+0000 7fd80e800640 1 -- 172.21.15.28:0/1280253876 shutdown_connections 2024-02-21T23:02:56.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.844+0000 7fd80e800640 1 -- 172.21.15.28:0/1280253876 wait complete. 2024-02-21T23:02:56.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.846+0000 7fd80e800640 1 Processor -- start 2024-02-21T23:02:56.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.846+0000 7fd80e800640 1 -- start start 2024-02-21T23:02:56.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.847+0000 7fd80e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd80806dbc0 0x7fd808152e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:56.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.847+0000 7fd80e800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd80813e0c0 0x7fd8081eb780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:56.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.847+0000 7fd80e800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd80813e520 0x7fd8080affb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:56.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.847+0000 7fd80e800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd80806ced0 con 0x7fd80806dbc0 2024-02-21T23:02:56.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.847+0000 7fd80e800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd808159070 con 0x7fd80813e0c0 2024-02-21T23:02:56.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.847+0000 7fd80e800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8080c78c0 con 0x7fd80813e520 2024-02-21T23:02:56.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.847+0000 7fd80d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd80806dbc0 0x7fd808152e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:56.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.847+0000 7fd80d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd80806dbc0 0x7fd808152e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37874/0 (socket says 172.21.15.28:37874) 2024-02-21T23:02:56.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.847+0000 7fd80d400640 1 -- 172.21.15.28:0/2527980952 learned_addr learned my addr 172.21.15.28:0/2527980952 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:02:56.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.847+0000 7fd80ca00640 1 --2- 172.21.15.28:0/2527980952 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd80813e0c0 0x7fd8081eb780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:56.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.847+0000 7fd80de00640 1 --2- 172.21.15.28:0/2527980952 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd80813e520 0x7fd8080affb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:56.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.848+0000 7fd80d400640 1 -- 172.21.15.28:0/2527980952 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd80813e520 msgr2=0x7fd8080affb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:56.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.848+0000 7fd80d400640 1 --2- 172.21.15.28:0/2527980952 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd80813e520 0x7fd8080affb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:56.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.848+0000 7fd80d400640 1 -- 172.21.15.28:0/2527980952 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd80813e0c0 msgr2=0x7fd8081eb780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:56.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.848+0000 7fd80d400640 1 --2- 172.21.15.28:0/2527980952 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd80813e0c0 0x7fd8081eb780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:56.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.848+0000 7fd80d400640 1 -- 172.21.15.28:0/2527980952 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd80813ca60 con 0x7fd80806dbc0 2024-02-21T23:02:56.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.848+0000 7fd80d400640 1 --2- 172.21.15.28:0/2527980952 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd80806dbc0 0x7fd808152e80 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd7f800c3f0 tx=0x7fd7f800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:56.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.848+0000 7fd805600640 1 -- 172.21.15.28:0/2527980952 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7f80074e0 con 0x7fd80806dbc0 2024-02-21T23:02:56.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.848+0000 7fd805600640 1 -- 172.21.15.28:0/2527980952 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7f8007680 con 0x7fd80806dbc0 2024-02-21T23:02:56.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.848+0000 7fd805600640 1 -- 172.21.15.28:0/2527980952 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7f8004a00 con 0x7fd80806dbc0 2024-02-21T23:02:56.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.848+0000 7fd80e800640 1 -- 172.21.15.28:0/2527980952 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8081de5f0 con 0x7fd80806dbc0 2024-02-21T23:02:56.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.849+0000 7fd80e800640 1 -- 172.21.15.28:0/2527980952 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd808155d10 con 0x7fd80806dbc0 2024-02-21T23:02:56.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.850+0000 7fd80e800640 1 -- 172.21.15.28:0/2527980952 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd80804f290 con 0x7fd80806dbc0 2024-02-21T23:02:56.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.850+0000 7fd805600640 1 -- 172.21.15.28:0/2527980952 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd7f801a020 con 0x7fd80806dbc0 2024-02-21T23:02:56.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.852+0000 7fd805600640 1 --2- 172.21.15.28:0/2527980952 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd7ec06b500 0x7fd7ec06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:56.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.852+0000 7fd805600640 1 -- 172.21.15.28:0/2527980952 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fd7f8089ca0 con 0x7fd80806dbc0 2024-02-21T23:02:56.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.852+0000 7fd80ca00640 1 --2- 172.21.15.28:0/2527980952 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd7ec06b500 0x7fd7ec06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:56.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.853+0000 7fd80ca00640 1 --2- 172.21.15.28:0/2527980952 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd7ec06b500 0x7fd7ec06d9c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc036000 tx=0x7fd7fc030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:56.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:56.855+0000 7fd805600640 1 -- 172.21.15.28:0/2527980952 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd7f8058740 con 0x7fd80806dbc0 2024-02-21T23:02:57.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.063+0000 7fd80e800640 1 -- 172.21.15.28:0/2527980952 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fd80806d9a0 con 0x7fd80806dbc0 2024-02-21T23:02:57.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.064+0000 7fd805600640 1 -- 172.21.15.28:0/2527980952 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) v1 ==== 103+0+11 (secure 0 0 0) 0x7fd7f8058040 con 0x7fd80806dbc0 2024-02-21T23:02:57.053 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-02-21T23:02:57.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.066+0000 7fd80e800640 1 -- 172.21.15.28:0/2527980952 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd7ec06b500 msgr2=0x7fd7ec06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:57.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.066+0000 7fd80e800640 1 --2- 172.21.15.28:0/2527980952 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd7ec06b500 0x7fd7ec06d9c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc036000 tx=0x7fd7fc030000 comp rx=0 tx=0).stop 2024-02-21T23:02:57.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.066+0000 7fd80e800640 1 -- 172.21.15.28:0/2527980952 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd80806dbc0 msgr2=0x7fd808152e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:57.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.066+0000 7fd80e800640 1 --2- 172.21.15.28:0/2527980952 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd80806dbc0 0x7fd808152e80 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd7f800c3f0 tx=0x7fd7f800c8b0 comp rx=0 tx=0).stop 2024-02-21T23:02:57.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.067+0000 7fd80e800640 1 -- 172.21.15.28:0/2527980952 shutdown_connections 2024-02-21T23:02:57.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.067+0000 7fd80e800640 1 --2- 172.21.15.28:0/2527980952 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd7ec06b500 0x7fd7ec06d9c0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:57.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.067+0000 7fd80e800640 1 --2- 172.21.15.28:0/2527980952 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd80813e520 0x7fd8080affb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:57.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.067+0000 7fd80e800640 1 --2- 172.21.15.28:0/2527980952 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd80813e0c0 0x7fd8081eb780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:57.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.067+0000 7fd80e800640 1 --2- 172.21.15.28:0/2527980952 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd80806dbc0 0x7fd808152e80 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:57.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.067+0000 7fd80e800640 1 -- 172.21.15.28:0/2527980952 >> 172.21.15.28:0/2527980952 conn(0x7fd808073110 msgr2=0x7fd8080c5020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:57.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.067+0000 7fd80e800640 1 -- 172.21.15.28:0/2527980952 shutdown_connections 2024-02-21T23:02:57.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.067+0000 7fd80e800640 1 -- 172.21.15.28:0/2527980952 wait complete. 2024-02-21T23:02:57.076 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-02-21T23:02:57.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.314+0000 7f37cbe00640 1 Processor -- start 2024-02-21T23:02:57.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.314+0000 7f37cbe00640 1 -- start start 2024-02-21T23:02:57.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.315+0000 7f37cbe00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f37cc149830 0x7f37cc0a8e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:57.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.315+0000 7f37cbe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f37cc0c43e0 0x7f37cc0c5a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:57.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.315+0000 7f37cbe00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37cc149c10 con 0x7f37cc06d410 2024-02-21T23:02:57.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.315+0000 7f37cbe00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37cc1537d0 con 0x7f37cc0c43e0 2024-02-21T23:02:57.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.315+0000 7f37cbe00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37cc153ab0 con 0x7f37cc149830 2024-02-21T23:02:57.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.315+0000 7f37cb400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f37cc0c43e0 0x7f37cc0c5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:57.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.315+0000 7f37cb400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f37cc0c43e0 0x7f37cc0c5a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37878/0 (socket says 172.21.15.28:37878) 2024-02-21T23:02:57.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.315+0000 7f37cb400640 1 -- 172.21.15.28:0/3814652359 learned_addr learned my addr 172.21.15.28:0/3814652359 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:02:57.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.316+0000 7f37caa00640 1 --2- 172.21.15.28:0/3814652359 >> v2:172.21.15.81:3300/0 conn(0x7f37cc149830 0x7f37cc0a8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:57.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.316+0000 7f37cb400640 1 -- 172.21.15.28:0/3814652359 >> v2:172.21.15.81:3300/0 conn(0x7f37cc149830 msgr2=0x7f37cc0a8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:02:57.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.316+0000 7f37cb400640 1 --2- 172.21.15.28:0/3814652359 >> v2:172.21.15.81:3300/0 conn(0x7f37cc149830 0x7f37cc0a8e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:57.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.316+0000 7f37cb400640 1 -- 172.21.15.28:0/3814652359 >> v1:172.21.15.81:6789/0 conn(0x7f37cc06d410 legacy=0x7f37cc0c0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:57.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.316+0000 7f37cb400640 1 -- 172.21.15.28:0/3814652359 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37cc0c4fa0 con 0x7f37cc0c43e0 2024-02-21T23:02:57.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.317+0000 7f37cb400640 1 --2- 172.21.15.28:0/3814652359 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37cc0c43e0 0x7f37cc0c5a80 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f37cc06f7c0 tx=0x7f37c402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b5629953e3045e1 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:57.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.318+0000 7f37ca000640 1 -- 172.21.15.28:0/3814652359 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37c402cb20 con 0x7f37cc0c43e0 2024-02-21T23:02:57.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.318+0000 7f37ca000640 1 -- 172.21.15.28:0/3814652359 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37c402ccc0 con 0x7f37cc0c43e0 2024-02-21T23:02:57.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.318+0000 7f37ca000640 1 -- 172.21.15.28:0/3814652359 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37c4034610 con 0x7f37cc0c43e0 2024-02-21T23:02:57.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.318+0000 7f37cbe00640 1 -- 172.21.15.28:0/3814652359 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37cc0c43e0 msgr2=0x7f37cc0c5a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:57.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.318+0000 7f37cbe00640 1 --2- 172.21.15.28:0/3814652359 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37cc0c43e0 0x7f37cc0c5a80 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f37cc06f7c0 tx=0x7f37c402bf80 comp rx=0 tx=0).stop 2024-02-21T23:02:57.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.318+0000 7f37cbe00640 1 -- 172.21.15.28:0/3814652359 shutdown_connections 2024-02-21T23:02:57.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.318+0000 7f37cbe00640 1 --2- 172.21.15.28:0/3814652359 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37cc0c43e0 0x7f37cc0c5a80 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:57.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.318+0000 7f37cbe00640 1 --2- 172.21.15.28:0/3814652359 >> v2:172.21.15.81:3300/0 conn(0x7f37cc149830 0x7f37cc0a8e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:57.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.318+0000 7f37cbe00640 1 -- 172.21.15.28:0/3814652359 >> 172.21.15.28:0/3814652359 conn(0x7f37cc0a8320 msgr2=0x7f37cc0a8730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:57.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.318+0000 7f37cbe00640 1 -- 172.21.15.28:0/3814652359 shutdown_connections 2024-02-21T23:02:57.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.318+0000 7f37cbe00640 1 -- 172.21.15.28:0/3814652359 wait complete. 2024-02-21T23:02:57.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.320+0000 7f37cbe00640 1 Processor -- start 2024-02-21T23:02:57.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.320+0000 7f37cbe00640 1 -- start start 2024-02-21T23:02:57.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.321+0000 7f37cbe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f37cc06d410 0x7f37cc1472d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:57.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.321+0000 7f37cbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37cc0c43e0 0x7f37cc156300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:57.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.321+0000 7f37cbe00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f37cc149830 0x7f37cc1e6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:57.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.321+0000 7f37cbe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37cc0c15f0 con 0x7f37cc0c43e0 2024-02-21T23:02:57.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.321+0000 7f37cbe00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37cc0c5560 con 0x7f37cc06d410 2024-02-21T23:02:57.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.321+0000 7f37cbe00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37cc0c1770 con 0x7f37cc149830 2024-02-21T23:02:57.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.321+0000 7f37c3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37cc0c43e0 0x7f37cc156300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:57.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.321+0000 7f37c3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37cc0c43e0 0x7f37cc156300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37890/0 (socket says 172.21.15.28:37890) 2024-02-21T23:02:57.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.321+0000 7f37c3e00640 1 -- 172.21.15.28:0/1567891718 learned_addr learned my addr 172.21.15.28:0/1567891718 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:02:57.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.322+0000 7f37caa00640 1 --2- 172.21.15.28:0/1567891718 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f37cc06d410 0x7f37cc1472d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:57.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.322+0000 7f37cb400640 1 --2- 172.21.15.28:0/1567891718 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f37cc149830 0x7f37cc1e6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:57.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.322+0000 7f37c3e00640 1 -- 172.21.15.28:0/1567891718 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f37cc149830 msgr2=0x7f37cc1e6380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:57.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.322+0000 7f37c3e00640 1 --2- 172.21.15.28:0/1567891718 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f37cc149830 0x7f37cc1e6380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:57.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.322+0000 7f37c3e00640 1 -- 172.21.15.28:0/1567891718 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f37cc06d410 msgr2=0x7f37cc1472d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:57.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.322+0000 7f37c3e00640 1 --2- 172.21.15.28:0/1567891718 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f37cc06d410 0x7f37cc1472d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:57.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.322+0000 7f37c3e00640 1 -- 172.21.15.28:0/1567891718 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37cc0a70a0 con 0x7f37cc0c43e0 2024-02-21T23:02:57.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.322+0000 7f37c3e00640 1 --2- 172.21.15.28:0/1567891718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37cc0c43e0 0x7f37cc156300 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f37b400c440 tx=0x7f37b4001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:57.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.322+0000 7f37c3400640 1 -- 172.21.15.28:0/1567891718 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37b40052d0 con 0x7f37cc0c43e0 2024-02-21T23:02:57.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.322+0000 7f37c3400640 1 -- 172.21.15.28:0/1567891718 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37b4005470 con 0x7f37cc0c43e0 2024-02-21T23:02:57.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.322+0000 7f37c3400640 1 -- 172.21.15.28:0/1567891718 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37b4007260 con 0x7f37cc0c43e0 2024-02-21T23:02:57.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.322+0000 7f37cbe00640 1 -- 172.21.15.28:0/1567891718 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37cc1e98f0 con 0x7f37cc0c43e0 2024-02-21T23:02:57.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.323+0000 7f37cbe00640 1 -- 172.21.15.28:0/1567891718 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37cc1e9eb0 con 0x7f37cc0c43e0 2024-02-21T23:02:57.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.324+0000 7f37cbe00640 1 -- 172.21.15.28:0/1567891718 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37cc0c73f0 con 0x7f37cc0c43e0 2024-02-21T23:02:57.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.325+0000 7f37c3400640 1 -- 172.21.15.28:0/1567891718 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f37b4007400 con 0x7f37cc0c43e0 2024-02-21T23:02:57.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.326+0000 7f37c3400640 1 --2- 172.21.15.28:0/1567891718 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f37a806b4d0 0x7f37a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:02:57.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.326+0000 7f37c3400640 1 -- 172.21.15.28:0/1567891718 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f37b40899f0 con 0x7f37cc0c43e0 2024-02-21T23:02:57.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.326+0000 7f37caa00640 1 --2- 172.21.15.28:0/1567891718 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f37a806b4d0 0x7f37a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:02:57.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.327+0000 7f37caa00640 1 --2- 172.21.15.28:0/1567891718 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f37a806b4d0 0x7f37a806d990 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f37bc009a30 tx=0x7f37bc008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:02:57.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.330+0000 7f37c3400640 1 -- 172.21.15.28:0/1567891718 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f37cc0c73f0 con 0x7f37cc0c43e0 2024-02-21T23:02:57.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.542+0000 7f37cbe00640 1 -- 172.21.15.28:0/1567891718 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f37cc04f290 con 0x7f37cc0c43e0 2024-02-21T23:02:57.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.543+0000 7f37c3400640 1 -- 172.21.15.28:0/1567891718 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) v1 ==== 80+0+0 (secure 0 0 0) 0x7f37b40585c0 con 0x7f37cc0c43e0 2024-02-21T23:02:57.532 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-02-21T23:02:57.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.546+0000 7f37cbe00640 1 -- 172.21.15.28:0/1567891718 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f37a806b4d0 msgr2=0x7f37a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:57.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.546+0000 7f37cbe00640 1 --2- 172.21.15.28:0/1567891718 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f37a806b4d0 0x7f37a806d990 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f37bc009a30 tx=0x7f37bc008040 comp rx=0 tx=0).stop 2024-02-21T23:02:57.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.546+0000 7f37cbe00640 1 -- 172.21.15.28:0/1567891718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37cc0c43e0 msgr2=0x7f37cc156300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:02:57.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.546+0000 7f37cbe00640 1 --2- 172.21.15.28:0/1567891718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37cc0c43e0 0x7f37cc156300 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f37b400c440 tx=0x7f37b4001e70 comp rx=0 tx=0).stop 2024-02-21T23:02:57.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.546+0000 7f37cbe00640 1 -- 172.21.15.28:0/1567891718 shutdown_connections 2024-02-21T23:02:57.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.546+0000 7f37cbe00640 1 --2- 172.21.15.28:0/1567891718 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f37a806b4d0 0x7f37a806d990 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:57.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.546+0000 7f37cbe00640 1 --2- 172.21.15.28:0/1567891718 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f37cc149830 0x7f37cc1e6380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:57.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.546+0000 7f37cbe00640 1 --2- 172.21.15.28:0/1567891718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37cc0c43e0 0x7f37cc156300 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:57.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.546+0000 7f37cbe00640 1 --2- 172.21.15.28:0/1567891718 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f37cc06d410 0x7f37cc1472d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:02:57.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.546+0000 7f37cbe00640 1 -- 172.21.15.28:0/1567891718 >> 172.21.15.28:0/1567891718 conn(0x7f37cc0a8320 msgr2=0x7f37cc06e8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:02:57.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.546+0000 7f37cbe00640 1 -- 172.21.15.28:0/1567891718 shutdown_connections 2024-02-21T23:02:57.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:02:57.546+0000 7f37cbe00640 1 -- 172.21.15.28:0/1567891718 wait complete. 2024-02-21T23:02:57.552 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi028.front.sepia.ceph.com... 2024-02-21T23:02:57.552 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:02:57.552 DEBUG:teuthology.orchestra.run.smithi028:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-02-21T23:02:57.646 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi028.stderr:ceph-fuse[42191]: fuse finished with error 0 and tester_r 0 2024-02-21T23:02:57.650 INFO:teuthology.orchestra.run:waiting for 300 2024-02-21T23:03:03.752 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi028.front.sepia.ceph.com 2024-02-21T23:03:03.752 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:03:03.752 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-02-21T23:03:03.824 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:03:04.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.068+0000 7f3edc400640 1 Processor -- start 2024-02-21T23:03:04.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.068+0000 7f3edc400640 1 -- start start 2024-02-21T23:03:04.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.070+0000 7f3edc400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f3ed40c6460 0x7f3ed40ca8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:04.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.070+0000 7f3edc400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ed41551b0 con 0x7f3ed40cadf0 2024-02-21T23:03:04.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.070+0000 7f3edc400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ed4155420 con 0x7f3ed413e620 2024-02-21T23:03:04.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.070+0000 7f3edc400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ed40c71d0 con 0x7f3ed40c6460 2024-02-21T23:03:04.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.070+0000 7f3eda400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f3ed40cadf0 0x7f3ed40bf990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41162/0 (socket says 172.21.15.28:41162) 2024-02-21T23:03:04.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.070+0000 7f3eda400640 1 -- 172.21.15.28:0/3115889718 learned_addr learned my addr 172.21.15.28:0/3115889718 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:03:04.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.071+0000 7f3ed9000640 1 --2- 172.21.15.28:0/3115889718 >> v2:172.21.15.81:3300/0 conn(0x7f3ed40c6460 0x7f3ed40ca8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:04.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.071+0000 7f3ed3e00640 1 -- 172.21.15.28:0/3115889718 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1049757944 0 0) 0x7f3ed41551b0 con 0x7f3ed40cadf0 2024-02-21T23:03:04.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.071+0000 7f3ed3e00640 1 -- 172.21.15.28:0/3115889718 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ebc003520 con 0x7f3ed40cadf0 2024-02-21T23:03:04.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.071+0000 7f3ed3e00640 1 -- 172.21.15.28:0/3115889718 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1345712480 0 0) 0x7f3ed4155420 con 0x7f3ed413e620 2024-02-21T23:03:04.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.071+0000 7f3ed3e00640 1 -- 172.21.15.28:0/3115889718 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ed41551b0 con 0x7f3ed413e620 2024-02-21T23:03:04.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.071+0000 7f3ed3e00640 1 -- 172.21.15.28:0/3115889718 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2695412913 0 0) 0x7f3ebc003520 con 0x7f3ed40cadf0 2024-02-21T23:03:04.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.071+0000 7f3ed3e00640 1 -- 172.21.15.28:0/3115889718 >> v2:172.21.15.81:3300/0 conn(0x7f3ed40c6460 msgr2=0x7f3ed40ca8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:04.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.072+0000 7f3ed3e00640 1 --2- 172.21.15.28:0/3115889718 >> v2:172.21.15.81:3300/0 conn(0x7f3ed40c6460 0x7f3ed40ca8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:04.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.072+0000 7f3ed3e00640 1 -- 172.21.15.28:0/3115889718 >> v1:172.21.15.81:6789/0 conn(0x7f3ed413e620 legacy=0x7f3ed4159070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:04.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.072+0000 7f3ed3e00640 1 -- 172.21.15.28:0/3115889718 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ed4155700 con 0x7f3ed40cadf0 2024-02-21T23:03:04.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.072+0000 7f3ed3e00640 1 -- 172.21.15.28:0/3115889718 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f3ecc0019f0 con 0x7f3ed40cadf0 2024-02-21T23:03:04.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.072+0000 7f3ed3e00640 1 -- 172.21.15.28:0/3115889718 >> v1:172.21.15.28:6789/0 conn(0x7f3ed40cadf0 legacy=0x7f3ed40bf990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:04.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.072+0000 7f3ed3e00640 1 --2- 172.21.15.28:0/3115889718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ebc003b90 0x7f3ebc003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:04.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.072+0000 7f3ed3e00640 1 --2- 172.21.15.28:0/3115889718 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3ebc004800 0x7f3ebc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:04.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.072+0000 7f3ed3e00640 1 --2- 172.21.15.28:0/3115889718 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3ebc005210 0x7f3ebc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:04.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.072+0000 7f3ed3e00640 1 -- 172.21.15.28:0/3115889718 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed40c71d0 con 0x7f3ebc003b90 2024-02-21T23:03:04.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.072+0000 7f3ed3e00640 1 -- 172.21.15.28:0/3115889718 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ebc00c140 con 0x7f3ebc005210 2024-02-21T23:03:04.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.072+0000 7f3ed3e00640 1 -- 172.21.15.28:0/3115889718 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ebc00c340 con 0x7f3ebc004800 2024-02-21T23:03:04.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.072+0000 7f3ed9a00640 1 --2- 172.21.15.28:0/3115889718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ebc003b90 0x7f3ebc003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:04.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.072+0000 7f3eda400640 1 --2- 172.21.15.28:0/3115889718 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3ebc005210 0x7f3ebc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:04.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.072+0000 7f3ed9000640 1 --2- 172.21.15.28:0/3115889718 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3ebc004800 0x7f3ebc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:04.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.073+0000 7f3ed9a00640 1 -- 172.21.15.28:0/3115889718 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3ebc004800 msgr2=0x7f3ebc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:04.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.073+0000 7f3ed9a00640 1 --2- 172.21.15.28:0/3115889718 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3ebc004800 0x7f3ebc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:04.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.073+0000 7f3ed9a00640 1 -- 172.21.15.28:0/3115889718 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3ebc005210 msgr2=0x7f3ebc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:04.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.073+0000 7f3ed9a00640 1 --2- 172.21.15.28:0/3115889718 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3ebc005210 0x7f3ebc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:04.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.073+0000 7f3ed9a00640 1 -- 172.21.15.28:0/3115889718 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ebc003520 con 0x7f3ebc003b90 2024-02-21T23:03:04.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.073+0000 7f3eda400640 1 -- 172.21.15.28:0/3115889718 reap_dead start 2024-02-21T23:03:04.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.074+0000 7f3ed9a00640 1 --2- 172.21.15.28:0/3115889718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ebc003b90 0x7f3ebc003f90 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3ec80018d0 tx=0x7f3ebc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbccd245d8d20c2f server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:04.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.075+0000 7f3ed3e00640 1 -- 172.21.15.28:0/3115889718 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec8037070 con 0x7f3ebc003b90 2024-02-21T23:03:04.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.075+0000 7f3ed3e00640 1 -- 172.21.15.28:0/3115889718 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ec80052d0 con 0x7f3ebc003b90 2024-02-21T23:03:04.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.075+0000 7f3ed3e00640 1 -- 172.21.15.28:0/3115889718 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec80014f0 con 0x7f3ebc003b90 2024-02-21T23:03:04.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.075+0000 7f3edc400640 1 -- 172.21.15.28:0/3115889718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ebc003b90 msgr2=0x7f3ebc003f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:04.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.075+0000 7f3edc400640 1 --2- 172.21.15.28:0/3115889718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ebc003b90 0x7f3ebc003f90 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3ec80018d0 tx=0x7f3ebc005830 comp rx=0 tx=0).stop 2024-02-21T23:03:04.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.076+0000 7f3edc400640 1 -- 172.21.15.28:0/3115889718 shutdown_connections 2024-02-21T23:03:04.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.076+0000 7f3edc400640 1 --2- 172.21.15.28:0/3115889718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ebc003b90 0x7f3ebc003f90 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:04.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.076+0000 7f3edc400640 1 -- 172.21.15.28:0/3115889718 >> 172.21.15.28:0/3115889718 conn(0x7f3ed408b1c0 msgr2=0x7f3ed40725a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:04.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.076+0000 7f3edc400640 1 -- 172.21.15.28:0/3115889718 shutdown_connections 2024-02-21T23:03:04.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.076+0000 7f3edc400640 1 -- 172.21.15.28:0/3115889718 wait complete. 2024-02-21T23:03:04.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.078+0000 7f3edc400640 1 Processor -- start 2024-02-21T23:03:04.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.078+0000 7f3edc400640 1 -- start start 2024-02-21T23:03:04.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.079+0000 7f3edc400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3ebc003b90 0x7f3ed413b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:04.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.079+0000 7f3edc400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3ed41dfbe0 0x7f3ed41ee230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:04.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.079+0000 7f3edc400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ed41ee770 0x7f3ed4156490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:04.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.079+0000 7f3edc400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed413ea00 con 0x7f3ed41ee770 2024-02-21T23:03:04.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.079+0000 7f3edc400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed4155cc0 con 0x7f3ed41dfbe0 2024-02-21T23:03:04.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.079+0000 7f3edc400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed40ca8b0 con 0x7f3ebc003b90 2024-02-21T23:03:04.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.079+0000 7f3eda400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ed41ee770 0x7f3ed4156490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:04.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.079+0000 7f3eda400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ed41ee770 0x7f3ed4156490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37916/0 (socket says 172.21.15.28:37916) 2024-02-21T23:03:04.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.079+0000 7f3eda400640 1 -- 172.21.15.28:0/2060513556 learned_addr learned my addr 172.21.15.28:0/2060513556 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:04.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.079+0000 7f3ed9a00640 1 --2- 172.21.15.28:0/2060513556 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3ebc003b90 0x7f3ed413b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:04.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.079+0000 7f3ed9000640 1 --2- 172.21.15.28:0/2060513556 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3ed41dfbe0 0x7f3ed41ee230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:04.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.080+0000 7f3eda400640 1 -- 172.21.15.28:0/2060513556 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3ebc003b90 msgr2=0x7f3ed413b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:04.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.080+0000 7f3eda400640 1 --2- 172.21.15.28:0/2060513556 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3ebc003b90 0x7f3ed413b1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:04.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.080+0000 7f3eda400640 1 -- 172.21.15.28:0/2060513556 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3ed41dfbe0 msgr2=0x7f3ed41ee230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:04.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.080+0000 7f3eda400640 1 --2- 172.21.15.28:0/2060513556 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3ed41dfbe0 0x7f3ed41ee230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:04.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.080+0000 7f3eda400640 1 -- 172.21.15.28:0/2060513556 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ed4158750 con 0x7f3ed41ee770 2024-02-21T23:03:04.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.080+0000 7f3ed9000640 1 --2- 172.21.15.28:0/2060513556 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3ed41dfbe0 0x7f3ed41ee230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:03:04.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.080+0000 7f3eda400640 1 --2- 172.21.15.28:0/2060513556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ed41ee770 0x7f3ed4156490 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f3ed4152ee0 tx=0x7f3ecc007af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:04.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.081+0000 7f3ed1600640 1 -- 172.21.15.28:0/2060513556 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ecc00ea40 con 0x7f3ed41ee770 2024-02-21T23:03:04.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.081+0000 7f3ed1600640 1 -- 172.21.15.28:0/2060513556 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ecc018070 con 0x7f3ed41ee770 2024-02-21T23:03:04.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.081+0000 7f3ed1600640 1 -- 172.21.15.28:0/2060513556 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ecc00ede0 con 0x7f3ed41ee770 2024-02-21T23:03:04.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.081+0000 7f3edc400640 1 -- 172.21.15.28:0/2060513556 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ed4157aa0 con 0x7f3ed41ee770 2024-02-21T23:03:04.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.081+0000 7f3edc400640 1 -- 172.21.15.28:0/2060513556 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ed41de4f0 con 0x7f3ed41ee770 2024-02-21T23:03:04.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.082+0000 7f3edc400640 1 -- 172.21.15.28:0/2060513556 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ed4067f30 con 0x7f3ed41ee770 2024-02-21T23:03:04.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.083+0000 7f3ed1600640 1 -- 172.21.15.28:0/2060513556 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3ecc01d460 con 0x7f3ed41ee770 2024-02-21T23:03:04.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.084+0000 7f3ed1600640 1 --2- 172.21.15.28:0/2060513556 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3eb406b4d0 0x7f3eb406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:04.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.084+0000 7f3ed1600640 1 -- 172.21.15.28:0/2060513556 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f3ecc08a650 con 0x7f3ed41ee770 2024-02-21T23:03:04.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.085+0000 7f3ed9a00640 1 --2- 172.21.15.28:0/2060513556 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3eb406b4d0 0x7f3eb406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:04.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.086+0000 7f3ed9a00640 1 --2- 172.21.15.28:0/2060513556 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3eb406b4d0 0x7f3eb406d990 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3ec8034000 tx=0x7f3ec8031000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:04.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.089+0000 7f3ed1600640 1 -- 172.21.15.28:0/2060513556 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3ecc059220 con 0x7f3ed41ee770 2024-02-21T23:03:04.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.293+0000 7f3edc400640 1 -- 172.21.15.28:0/2060513556 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3ed40c0ce0 con 0x7f3ed41ee770 2024-02-21T23:03:04.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.296+0000 7f3ed1600640 1 -- 172.21.15.28:0/2060513556 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18454 (secure 0 0 0) 0x7f3ecc058b20 con 0x7f3ed41ee770 2024-02-21T23:03:04.285 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:03:04.285 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":24,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:02:42.900286+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:02:35.795158+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-21T23:02:36.840110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-02-21T23:02:40.441388+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:03:04.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.299+0000 7f3edc400640 1 -- 172.21.15.28:0/2060513556 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3eb406b4d0 msgr2=0x7f3eb406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:04.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.299+0000 7f3edc400640 1 --2- 172.21.15.28:0/2060513556 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3eb406b4d0 0x7f3eb406d990 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3ec8034000 tx=0x7f3ec8031000 comp rx=0 tx=0).stop 2024-02-21T23:03:04.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.299+0000 7f3edc400640 1 -- 172.21.15.28:0/2060513556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ed41ee770 msgr2=0x7f3ed4156490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:04.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.299+0000 7f3edc400640 1 --2- 172.21.15.28:0/2060513556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ed41ee770 0x7f3ed4156490 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f3ed4152ee0 tx=0x7f3ecc007af0 comp rx=0 tx=0).stop 2024-02-21T23:03:04.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.299+0000 7f3edc400640 1 -- 172.21.15.28:0/2060513556 shutdown_connections 2024-02-21T23:03:04.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.299+0000 7f3edc400640 1 --2- 172.21.15.28:0/2060513556 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3eb406b4d0 0x7f3eb406d990 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:04.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.299+0000 7f3edc400640 1 --2- 172.21.15.28:0/2060513556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ed41ee770 0x7f3ed4156490 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:04.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.299+0000 7f3edc400640 1 --2- 172.21.15.28:0/2060513556 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3ed41dfbe0 0x7f3ed41ee230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:04.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.299+0000 7f3edc400640 1 --2- 172.21.15.28:0/2060513556 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3ebc003b90 0x7f3ed413b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:04.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.299+0000 7f3edc400640 1 -- 172.21.15.28:0/2060513556 >> 172.21.15.28:0/2060513556 conn(0x7f3ed408b1c0 msgr2=0x7f3ed413beb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:04.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.299+0000 7f3edc400640 1 -- 172.21.15.28:0/2060513556 shutdown_connections 2024-02-21T23:03:04.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.299+0000 7f3edc400640 1 -- 172.21.15.28:0/2060513556 wait complete. 2024-02-21T23:03:04.307 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:03:04.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.539+0000 7f58b2400640 1 Processor -- start 2024-02-21T23:03:04.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.540+0000 7f58b2400640 1 -- start start 2024-02-21T23:03:04.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.541+0000 7f58b2400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f58ac0c11f0 0x7f58ac153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:04.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.541+0000 7f58b2400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f58ac15b6d0 0x7f58ac15dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:04.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.541+0000 7f58b2400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58ac152e70 con 0x7f58ac0c15d0 2024-02-21T23:03:04.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.541+0000 7f58b2400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58ac0c1a60 con 0x7f58ac15b6d0 2024-02-21T23:03:04.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.541+0000 7f58b2400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58ac0a5ec0 con 0x7f58ac0c11f0 2024-02-21T23:03:04.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.542+0000 7f58abe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f58ac15b6d0 0x7f58ac15dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:04.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.542+0000 7f58abe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f58ac15b6d0 0x7f58ac15dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37920/0 (socket says 172.21.15.28:37920) 2024-02-21T23:03:04.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.542+0000 7f58abe00640 1 -- 172.21.15.28:0/1171205416 learned_addr learned my addr 172.21.15.28:0/1171205416 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:04.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.542+0000 7f58ab400640 1 --2- 172.21.15.28:0/1171205416 >> v2:172.21.15.81:3300/0 conn(0x7f58ac0c11f0 0x7f58ac153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:04.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.542+0000 7f58abe00640 1 -- 172.21.15.28:0/1171205416 >> v2:172.21.15.81:3300/0 conn(0x7f58ac0c11f0 msgr2=0x7f58ac153950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:04.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.542+0000 7f58abe00640 1 --2- 172.21.15.28:0/1171205416 >> v2:172.21.15.81:3300/0 conn(0x7f58ac0c11f0 0x7f58ac153950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:04.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.542+0000 7f58abe00640 1 -- 172.21.15.28:0/1171205416 >> v1:172.21.15.81:6789/0 conn(0x7f58ac0c15d0 legacy=0x7f58ac06dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:04.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.542+0000 7f58abe00640 1 -- 172.21.15.28:0/1171205416 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58ac153150 con 0x7f58ac15b6d0 2024-02-21T23:03:04.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.543+0000 7f58abe00640 1 --2- 172.21.15.28:0/1171205416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58ac15b6d0 0x7f58ac15dac0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f58ac0c7590 tx=0x7f58a402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21c163beb7f1f67d server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:04.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.544+0000 7f58aa000640 1 -- 172.21.15.28:0/1171205416 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a402c9f0 con 0x7f58ac15b6d0 2024-02-21T23:03:04.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.544+0000 7f58aa000640 1 -- 172.21.15.28:0/1171205416 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58a402cb90 con 0x7f58ac15b6d0 2024-02-21T23:03:04.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.544+0000 7f58aa000640 1 -- 172.21.15.28:0/1171205416 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a40343f0 con 0x7f58ac15b6d0 2024-02-21T23:03:04.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.544+0000 7f58b2400640 1 -- 172.21.15.28:0/1171205416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58ac15b6d0 msgr2=0x7f58ac15dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:04.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.544+0000 7f58b2400640 1 --2- 172.21.15.28:0/1171205416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58ac15b6d0 0x7f58ac15dac0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f58ac0c7590 tx=0x7f58a402be50 comp rx=0 tx=0).stop 2024-02-21T23:03:04.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.545+0000 7f58b2400640 1 -- 172.21.15.28:0/1171205416 shutdown_connections 2024-02-21T23:03:04.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.545+0000 7f58b2400640 1 --2- 172.21.15.28:0/1171205416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58ac15b6d0 0x7f58ac15dac0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:04.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.545+0000 7f58b2400640 1 --2- 172.21.15.28:0/1171205416 >> v2:172.21.15.81:3300/0 conn(0x7f58ac0c11f0 0x7f58ac153950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:04.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.545+0000 7f58b2400640 1 -- 172.21.15.28:0/1171205416 >> 172.21.15.28:0/1171205416 conn(0x7f58ac0c67f0 msgr2=0x7f58ac0c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:04.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.545+0000 7f58b2400640 1 -- 172.21.15.28:0/1171205416 shutdown_connections 2024-02-21T23:03:04.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.545+0000 7f58b2400640 1 -- 172.21.15.28:0/1171205416 wait complete. 2024-02-21T23:03:04.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.546+0000 7f58b2400640 1 Processor -- start 2024-02-21T23:03:04.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.546+0000 7f58b2400640 1 -- start start 2024-02-21T23:03:04.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.547+0000 7f58b2400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58ac0c11f0 0x7f58ac0bfe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:04.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.547+0000 7f58b2400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f58ac0c15d0 0x7f58ac0a74e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:04.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.547+0000 7f58b2400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58ac15b6d0 0x7f58ac141ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:04.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.547+0000 7f58b2400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58ac0c4c00 con 0x7f58ac15b6d0 2024-02-21T23:03:04.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.547+0000 7f58b2400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58ac14d1c0 con 0x7f58ac0c11f0 2024-02-21T23:03:04.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.547+0000 7f58b2400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58ac0a6110 con 0x7f58ac0c15d0 2024-02-21T23:03:04.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.548+0000 7f58abe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58ac15b6d0 0x7f58ac141ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:04.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.548+0000 7f58abe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58ac15b6d0 0x7f58ac141ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37930/0 (socket says 172.21.15.28:37930) 2024-02-21T23:03:04.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.548+0000 7f58abe00640 1 -- 172.21.15.28:0/2056521838 learned_addr learned my addr 172.21.15.28:0/2056521838 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:04.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.548+0000 7f58ab400640 1 --2- 172.21.15.28:0/2056521838 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58ac0c11f0 0x7f58ac0bfe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:04.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.548+0000 7f58aaa00640 1 --2- 172.21.15.28:0/2056521838 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f58ac0c15d0 0x7f58ac0a74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:04.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.548+0000 7f58abe00640 1 -- 172.21.15.28:0/2056521838 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f58ac0c15d0 msgr2=0x7f58ac0a74e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:04.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.548+0000 7f58abe00640 1 --2- 172.21.15.28:0/2056521838 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f58ac0c15d0 0x7f58ac0a74e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:04.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.548+0000 7f58abe00640 1 -- 172.21.15.28:0/2056521838 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58ac0c11f0 msgr2=0x7f58ac0bfe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:04.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.548+0000 7f58abe00640 1 --2- 172.21.15.28:0/2056521838 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58ac0c11f0 0x7f58ac0bfe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:04.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.548+0000 7f58abe00640 1 -- 172.21.15.28:0/2056521838 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58ac149480 con 0x7f58ac15b6d0 2024-02-21T23:03:04.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.548+0000 7f58aaa00640 1 --2- 172.21.15.28:0/2056521838 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f58ac0c15d0 0x7f58ac0a74e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:03:04.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.549+0000 7f58abe00640 1 --2- 172.21.15.28:0/2056521838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58ac15b6d0 0x7f58ac141ae0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f58a4009f40 tx=0x7f58a4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:04.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.549+0000 7f58a3400640 1 -- 172.21.15.28:0/2056521838 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a4044020 con 0x7f58ac15b6d0 2024-02-21T23:03:04.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.549+0000 7f58a3400640 1 -- 172.21.15.28:0/2056521838 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58a4040470 con 0x7f58ac15b6d0 2024-02-21T23:03:04.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.549+0000 7f58a3400640 1 -- 172.21.15.28:0/2056521838 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a4040860 con 0x7f58ac15b6d0 2024-02-21T23:03:04.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.549+0000 7f58b2400640 1 -- 172.21.15.28:0/2056521838 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58ac072140 con 0x7f58ac15b6d0 2024-02-21T23:03:04.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.549+0000 7f58b2400640 1 -- 172.21.15.28:0/2056521838 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58ac071790 con 0x7f58ac15b6d0 2024-02-21T23:03:04.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.550+0000 7f58b2400640 1 -- 172.21.15.28:0/2056521838 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58ac067f30 con 0x7f58ac15b6d0 2024-02-21T23:03:04.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.551+0000 7f58a3400640 1 -- 172.21.15.28:0/2056521838 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f58a4047020 con 0x7f58ac15b6d0 2024-02-21T23:03:04.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.552+0000 7f58a3400640 1 --2- 172.21.15.28:0/2056521838 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f588c06b520 0x7f588c06d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:04.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.552+0000 7f58a3400640 1 -- 172.21.15.28:0/2056521838 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f58a40acd60 con 0x7f58ac15b6d0 2024-02-21T23:03:04.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.553+0000 7f58ab400640 1 --2- 172.21.15.28:0/2056521838 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f588c06b520 0x7f588c06d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:04.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.554+0000 7f58ab400640 1 --2- 172.21.15.28:0/2056521838 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f588c06b520 0x7f588c06d9e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f589c006fd0 tx=0x7f589c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:04.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.557+0000 7f58a3400640 1 -- 172.21.15.28:0/2056521838 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f58a404e070 con 0x7f58ac15b6d0 2024-02-21T23:03:04.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.760+0000 7f58b2400640 1 -- 172.21.15.28:0/2056521838 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f58ac06f0e0 con 0x7f58ac15b6d0 2024-02-21T23:03:04.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.763+0000 7f58a3400640 1 -- 172.21.15.28:0/2056521838 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18454 (secure 0 0 0) 0x7f58a407b910 con 0x7f58ac15b6d0 2024-02-21T23:03:04.752 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:03:04.752 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":24,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:02:42.900286+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:02:35.795158+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-21T23:02:36.840110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-02-21T23:02:40.441388+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:03:04.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.766+0000 7f58b2400640 1 -- 172.21.15.28:0/2056521838 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f588c06b520 msgr2=0x7f588c06d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:04.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.766+0000 7f58b2400640 1 --2- 172.21.15.28:0/2056521838 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f588c06b520 0x7f588c06d9e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f589c006fd0 tx=0x7f589c008040 comp rx=0 tx=0).stop 2024-02-21T23:03:04.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.766+0000 7f58b2400640 1 -- 172.21.15.28:0/2056521838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58ac15b6d0 msgr2=0x7f58ac141ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:04.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.766+0000 7f58b2400640 1 --2- 172.21.15.28:0/2056521838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58ac15b6d0 0x7f58ac141ae0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f58a4009f40 tx=0x7f58a4004a40 comp rx=0 tx=0).stop 2024-02-21T23:03:04.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.767+0000 7f58b2400640 1 -- 172.21.15.28:0/2056521838 shutdown_connections 2024-02-21T23:03:04.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.767+0000 7f58b2400640 1 --2- 172.21.15.28:0/2056521838 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f588c06b520 0x7f588c06d9e0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:04.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.767+0000 7f58b2400640 1 --2- 172.21.15.28:0/2056521838 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f58ac0c15d0 0x7f58ac0a74e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:04.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.767+0000 7f58b2400640 1 --2- 172.21.15.28:0/2056521838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58ac15b6d0 0x7f58ac141ae0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:04.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.767+0000 7f58b2400640 1 --2- 172.21.15.28:0/2056521838 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58ac0c11f0 0x7f58ac0bfe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:04.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.767+0000 7f58b2400640 1 -- 172.21.15.28:0/2056521838 >> 172.21.15.28:0/2056521838 conn(0x7f58ac0c67f0 msgr2=0x7f58ac14c4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:04.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.767+0000 7f58b2400640 1 -- 172.21.15.28:0/2056521838 shutdown_connections 2024-02-21T23:03:04.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:04.767+0000 7f58b2400640 1 -- 172.21.15.28:0/2056521838 wait complete. 2024-02-21T23:03:04.775 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-02-21T23:03:04.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.008+0000 7fb3b9c00640 1 Processor -- start 2024-02-21T23:03:04.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.009+0000 7fb3b9c00640 1 -- start start 2024-02-21T23:03:04.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.010+0000 7fb3b9c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb3b41569d0 0x7fb3b406e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:04.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.010+0000 7fb3b9c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb3b406d190 0x7fb3b40c5d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:04.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.010+0000 7fb3b9c00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3b406f370 con 0x7fb3b40c62d0 2024-02-21T23:03:04.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.010+0000 7fb3b9c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3b415ca90 con 0x7fb3b406d190 2024-02-21T23:03:04.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.010+0000 7fb3b9c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b415cd70 con 0x7fb3b41569d0 2024-02-21T23:03:04.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.010+0000 7fb3b2000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb3b406d190 0x7fb3b40c5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:04.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.010+0000 7fb3b2000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb3b406d190 0x7fb3b40c5d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45814/0 (socket says 172.21.15.28:45814) 2024-02-21T23:03:04.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.010+0000 7fb3b2000640 1 -- 172.21.15.28:0/3947284657 learned_addr learned my addr 172.21.15.28:0/3947284657 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:04.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.010+0000 7fb3b2a00640 1 --2- 172.21.15.28:0/3947284657 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb3b41569d0 0x7fb3b406e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:05.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.011+0000 7fb3b2000640 1 -- 172.21.15.28:0/3947284657 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb3b41569d0 msgr2=0x7fb3b406e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:05.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.011+0000 7fb3b2000640 1 --2- 172.21.15.28:0/3947284657 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb3b41569d0 0x7fb3b406e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:05.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.011+0000 7fb3b2000640 1 -- 172.21.15.28:0/3947284657 >> v1:172.21.15.81:6789/0 conn(0x7fb3b40c62d0 legacy=0x7fb3b4159470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:05.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.011+0000 7fb3b2000640 1 -- 172.21.15.28:0/3947284657 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3b415d050 con 0x7fb3b406d190 2024-02-21T23:03:05.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.012+0000 7fb3b2000640 1 --2- 172.21.15.28:0/3947284657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3b406d190 0x7fb3b40c5d90 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb39c009e50 tx=0x7fb39c02bff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84bcb2c46e088e00 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:05.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.012+0000 7fb3b1600640 1 -- 172.21.15.28:0/3947284657 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb39c02cb90 con 0x7fb3b406d190 2024-02-21T23:03:05.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.012+0000 7fb3b1600640 1 -- 172.21.15.28:0/3947284657 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb39c02cd30 con 0x7fb3b406d190 2024-02-21T23:03:05.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.012+0000 7fb3b1600640 1 -- 172.21.15.28:0/3947284657 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb39c034610 con 0x7fb3b406d190 2024-02-21T23:03:05.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.013+0000 7fb3b9c00640 1 -- 172.21.15.28:0/3947284657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3b406d190 msgr2=0x7fb3b40c5d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:05.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.013+0000 7fb3b9c00640 1 --2- 172.21.15.28:0/3947284657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3b406d190 0x7fb3b40c5d90 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb39c009e50 tx=0x7fb39c02bff0 comp rx=0 tx=0).stop 2024-02-21T23:03:05.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.013+0000 7fb3b9c00640 1 -- 172.21.15.28:0/3947284657 shutdown_connections 2024-02-21T23:03:05.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.013+0000 7fb3b9c00640 1 --2- 172.21.15.28:0/3947284657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3b406d190 0x7fb3b40c5d90 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:05.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.013+0000 7fb3b9c00640 1 --2- 172.21.15.28:0/3947284657 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb3b41569d0 0x7fb3b406e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:05.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.013+0000 7fb3b9c00640 1 -- 172.21.15.28:0/3947284657 >> 172.21.15.28:0/3947284657 conn(0x7fb3b40a85a0 msgr2=0x7fb3b40a8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:05.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.013+0000 7fb3b9c00640 1 -- 172.21.15.28:0/3947284657 shutdown_connections 2024-02-21T23:03:05.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.013+0000 7fb3b9c00640 1 -- 172.21.15.28:0/3947284657 wait complete. 2024-02-21T23:03:05.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.015+0000 7fb3b9c00640 1 Processor -- start 2024-02-21T23:03:05.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.015+0000 7fb3b9c00640 1 -- start start 2024-02-21T23:03:05.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.016+0000 7fb3b9c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3b406d190 0x7fb3b4155750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:05.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.016+0000 7fb3b9c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb3b40c62d0 0x7fb3b414faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:05.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.016+0000 7fb3b9c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb3b41569d0 0x7fb3b414ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:05.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.016+0000 7fb3b9c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b415d850 con 0x7fb3b406d190 2024-02-21T23:03:05.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.016+0000 7fb3b9c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b4156db0 con 0x7fb3b40c62d0 2024-02-21T23:03:05.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.016+0000 7fb3b9c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b406ef20 con 0x7fb3b41569d0 2024-02-21T23:03:05.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.016+0000 7fb3b2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3b406d190 0x7fb3b4155750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:05.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.016+0000 7fb3b2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3b406d190 0x7fb3b4155750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45822/0 (socket says 172.21.15.28:45822) 2024-02-21T23:03:05.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.016+0000 7fb3b2a00640 1 -- 172.21.15.28:0/1533123776 learned_addr learned my addr 172.21.15.28:0/1533123776 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:05.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.016+0000 7fb3b2000640 1 --2- 172.21.15.28:0/1533123776 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb3b40c62d0 0x7fb3b414faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:05.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.017+0000 7fb3b3400640 1 --2- 172.21.15.28:0/1533123776 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb3b41569d0 0x7fb3b414ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:05.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.017+0000 7fb3b2a00640 1 -- 172.21.15.28:0/1533123776 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb3b41569d0 msgr2=0x7fb3b414ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:05.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.017+0000 7fb3b2a00640 1 --2- 172.21.15.28:0/1533123776 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb3b41569d0 0x7fb3b414ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:05.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.017+0000 7fb3b2a00640 1 -- 172.21.15.28:0/1533123776 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb3b40c62d0 msgr2=0x7fb3b414faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:05.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.017+0000 7fb3b2a00640 1 --2- 172.21.15.28:0/1533123776 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb3b40c62d0 0x7fb3b414faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:05.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.017+0000 7fb3b2a00640 1 -- 172.21.15.28:0/1533123776 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3b40a98e0 con 0x7fb3b406d190 2024-02-21T23:03:05.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.017+0000 7fb3b2000640 1 --2- 172.21.15.28:0/1533123776 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb3b40c62d0 0x7fb3b414faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:03:05.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.017+0000 7fb3b3400640 1 --2- 172.21.15.28:0/1533123776 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb3b41569d0 0x7fb3b414ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:03:05.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.017+0000 7fb3b2a00640 1 --2- 172.21.15.28:0/1533123776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3b406d190 0x7fb3b4155750 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb3a400e9b0 tx=0x7fb3a400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:05.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.017+0000 7fb3aaa00640 1 -- 172.21.15.28:0/1533123776 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3a4017070 con 0x7fb3b406d190 2024-02-21T23:03:05.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.017+0000 7fb3aaa00640 1 -- 172.21.15.28:0/1533123776 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3a400c9a0 con 0x7fb3b406d190 2024-02-21T23:03:05.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.017+0000 7fb3aaa00640 1 -- 172.21.15.28:0/1533123776 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3a400cd90 con 0x7fb3b406d190 2024-02-21T23:03:05.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.017+0000 7fb3b9c00640 1 -- 172.21.15.28:0/1533123776 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3b415a070 con 0x7fb3b406d190 2024-02-21T23:03:05.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.018+0000 7fb3b9c00640 1 -- 172.21.15.28:0/1533123776 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3b4155c90 con 0x7fb3b406d190 2024-02-21T23:03:05.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.019+0000 7fb3b9c00640 1 -- 172.21.15.28:0/1533123776 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3b406c560 con 0x7fb3b406d190 2024-02-21T23:03:05.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.020+0000 7fb3aaa00640 1 -- 172.21.15.28:0/1533123776 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb3a401a070 con 0x7fb3b406d190 2024-02-21T23:03:05.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.021+0000 7fb3aaa00640 1 --2- 172.21.15.28:0/1533123776 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb39006b4d0 0x7fb39006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:05.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.021+0000 7fb3aaa00640 1 -- 172.21.15.28:0/1533123776 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fb3a408a8e0 con 0x7fb3b406d190 2024-02-21T23:03:05.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.021+0000 7fb3b2000640 1 --2- 172.21.15.28:0/1533123776 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb39006b4d0 0x7fb39006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:05.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.022+0000 7fb3b2000640 1 --2- 172.21.15.28:0/1533123776 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb39006b4d0 0x7fb39006d990 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb39c035000 tx=0x7fb39c030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:05.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.025+0000 7fb3aaa00640 1 -- 172.21.15.28:0/1533123776 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb3b406c560 con 0x7fb3b406d190 2024-02-21T23:03:05.033 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-21T23:03:05.040 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-21T23:03:05.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.273+0000 7fb3b9c00640 1 -- 172.21.15.28:0/1533123776 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fb3b40c2540 con 0x7fb3b406d190 2024-02-21T23:03:05.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.737+0000 7fb3aaa00640 1 -- 172.21.15.28:0/1533123776 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v8) v1 ==== 118+0+0 (secure 0 0 0) 0x7fb3b40c2540 con 0x7fb3b406d190 2024-02-21T23:03:05.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.741+0000 7fb3b9c00640 1 -- 172.21.15.28:0/1533123776 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb39006b4d0 msgr2=0x7fb39006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:05.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.741+0000 7fb3b9c00640 1 --2- 172.21.15.28:0/1533123776 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb39006b4d0 0x7fb39006d990 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb39c035000 tx=0x7fb39c030000 comp rx=0 tx=0).stop 2024-02-21T23:03:05.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.741+0000 7fb3b9c00640 1 -- 172.21.15.28:0/1533123776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3b406d190 msgr2=0x7fb3b4155750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:05.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.741+0000 7fb3b9c00640 1 --2- 172.21.15.28:0/1533123776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3b406d190 0x7fb3b4155750 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb3a400e9b0 tx=0x7fb3a400ee70 comp rx=0 tx=0).stop 2024-02-21T23:03:05.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.741+0000 7fb3b9c00640 1 -- 172.21.15.28:0/1533123776 shutdown_connections 2024-02-21T23:03:05.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.741+0000 7fb3b9c00640 1 --2- 172.21.15.28:0/1533123776 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb39006b4d0 0x7fb39006d990 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:05.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.741+0000 7fb3b9c00640 1 --2- 172.21.15.28:0/1533123776 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb3b41569d0 0x7fb3b414ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:05.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.741+0000 7fb3b9c00640 1 --2- 172.21.15.28:0/1533123776 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb3b40c62d0 0x7fb3b414faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:05.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.741+0000 7fb3b9c00640 1 --2- 172.21.15.28:0/1533123776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3b406d190 0x7fb3b4155750 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:05.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.741+0000 7fb3b9c00640 1 -- 172.21.15.28:0/1533123776 >> 172.21.15.28:0/1533123776 conn(0x7fb3b40a85a0 msgr2=0x7fb3b406ae70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:05.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.741+0000 7fb3b9c00640 1 -- 172.21.15.28:0/1533123776 shutdown_connections 2024-02-21T23:03:05.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.742+0000 7fb3b9c00640 1 -- 172.21.15.28:0/1533123776 wait complete. 2024-02-21T23:03:05.752 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']} 2024-02-21T23:03:05.753 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-02-21T23:03:05.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.983+0000 7f8885400640 1 Processor -- start 2024-02-21T23:03:05.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.983+0000 7f8885400640 1 -- start start 2024-02-21T23:03:05.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.984+0000 7f8885400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f88800c8f20 0x7f8880153a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:05.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.984+0000 7f8885400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f88801564e0 0x7f88800c5680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:05.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.984+0000 7f8885400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88800c80f0 con 0x7f8880071300 2024-02-21T23:03:05.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.984+0000 7f8885400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88800c83d0 con 0x7f88801564e0 2024-02-21T23:03:05.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.984+0000 7f8885400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8880153240 con 0x7f88800c8f20 2024-02-21T23:03:05.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.985+0000 7f887de00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f8880071300 0x7f88800c77f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:48502/0 (socket says 172.21.15.28:48502) 2024-02-21T23:03:05.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.985+0000 7f887de00640 1 -- 172.21.15.28:0/2178142032 learned_addr learned my addr 172.21.15.28:0/2178142032 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:03:05.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.985+0000 7f887e800640 1 --2- 172.21.15.28:0/2178142032 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f88800c8f20 0x7f8880153a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:05.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.985+0000 7f887f200640 1 --2- 172.21.15.28:0/2178142032 >> v2:172.21.15.81:3300/0 conn(0x7f88801564e0 0x7f88800c5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:05.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.985+0000 7f887d400640 1 -- 172.21.15.28:0/2178142032 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1280883913 0 0) 0x7f88800c80f0 con 0x7f8880071300 2024-02-21T23:03:05.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.985+0000 7f887d400640 1 -- 172.21.15.28:0/2178142032 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8864003520 con 0x7f8880071300 2024-02-21T23:03:05.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.986+0000 7f887d400640 1 -- 172.21.15.28:0/2178142032 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3211550733 0 0) 0x7f8864003520 con 0x7f8880071300 2024-02-21T23:03:05.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.986+0000 7f887d400640 1 -- 172.21.15.28:0/2178142032 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f88800c8f20 msgr2=0x7f8880153a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:05.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.986+0000 7f887d400640 1 --2- 172.21.15.28:0/2178142032 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f88800c8f20 0x7f8880153a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:05.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.986+0000 7f887d400640 1 -- 172.21.15.28:0/2178142032 >> v2:172.21.15.81:3300/0 conn(0x7f88801564e0 msgr2=0x7f88800c5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:05.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.986+0000 7f887d400640 1 --2- 172.21.15.28:0/2178142032 >> v2:172.21.15.81:3300/0 conn(0x7f88801564e0 0x7f88800c5680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:05.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.986+0000 7f887d400640 1 -- 172.21.15.28:0/2178142032 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8880153520 con 0x7f8880071300 2024-02-21T23:03:05.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.986+0000 7f887d400640 1 -- 172.21.15.28:0/2178142032 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f886c001670 con 0x7f8880071300 2024-02-21T23:03:05.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.986+0000 7f887d400640 1 -- 172.21.15.28:0/2178142032 >> v1:172.21.15.28:6789/0 conn(0x7f8880071300 legacy=0x7f88800c77f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:05.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.986+0000 7f887d400640 1 --2- 172.21.15.28:0/2178142032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88640039b0 0x7f8864003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:05.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.986+0000 7f887d400640 1 --2- 172.21.15.28:0/2178142032 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8864004800 0x7f8864004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:05.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.986+0000 7f887d400640 1 --2- 172.21.15.28:0/2178142032 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f88800c8f20 0x7f88640051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:05.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.986+0000 7f887d400640 1 -- 172.21.15.28:0/2178142032 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88800c83d0 con 0x7f88640039b0 2024-02-21T23:03:05.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.986+0000 7f887d400640 1 -- 172.21.15.28:0/2178142032 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8880153240 con 0x7f8864004800 2024-02-21T23:03:05.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.986+0000 7f887d400640 1 -- 172.21.15.28:0/2178142032 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f886400b9c0 con 0x7f88800c8f20 2024-02-21T23:03:05.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.986+0000 7f887e800640 1 --2- 172.21.15.28:0/2178142032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88640039b0 0x7f8864003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:05.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.987+0000 7f887de00640 1 --2- 172.21.15.28:0/2178142032 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8864004800 0x7f8864004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:05.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.987+0000 7f887f200640 1 --2- 172.21.15.28:0/2178142032 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f88800c8f20 0x7f88640051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:05.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.987+0000 7f887e800640 1 -- 172.21.15.28:0/2178142032 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f88800c8f20 msgr2=0x7f88640051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:05.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.987+0000 7f887e800640 1 --2- 172.21.15.28:0/2178142032 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f88800c8f20 0x7f88640051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:05.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.987+0000 7f887e800640 1 -- 172.21.15.28:0/2178142032 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8864004800 msgr2=0x7f8864004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:05.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.987+0000 7f887e800640 1 --2- 172.21.15.28:0/2178142032 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8864004800 0x7f8864004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:05.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.987+0000 7f887e800640 1 -- 172.21.15.28:0/2178142032 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8864003520 con 0x7f88640039b0 2024-02-21T23:03:05.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.988+0000 7f887e800640 1 --2- 172.21.15.28:0/2178142032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88640039b0 0x7f8864003dd0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f8870009ac0 tx=0x7f8864005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14736ca9f9f8bdc5 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:05.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.988+0000 7f887d400640 1 -- 172.21.15.28:0/2178142032 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8870039070 con 0x7f88640039b0 2024-02-21T23:03:05.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.988+0000 7f887d400640 1 -- 172.21.15.28:0/2178142032 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f887002c6b0 con 0x7f88640039b0 2024-02-21T23:03:05.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.988+0000 7f887d400640 1 -- 172.21.15.28:0/2178142032 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8870009690 con 0x7f88640039b0 2024-02-21T23:03:05.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.989+0000 7f8885400640 1 -- 172.21.15.28:0/2178142032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88640039b0 msgr2=0x7f8864003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:05.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.989+0000 7f8885400640 1 --2- 172.21.15.28:0/2178142032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88640039b0 0x7f8864003dd0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f8870009ac0 tx=0x7f8864005830 comp rx=0 tx=0).stop 2024-02-21T23:03:05.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.989+0000 7f887f200640 1 -- 172.21.15.28:0/2178142032 reap_dead start 2024-02-21T23:03:05.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.989+0000 7f8885400640 1 -- 172.21.15.28:0/2178142032 shutdown_connections 2024-02-21T23:03:05.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.989+0000 7f8885400640 1 -- 172.21.15.28:0/2178142032 >> 172.21.15.28:0/2178142032 conn(0x7f888006b290 msgr2=0x7f888006b6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:05.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.989+0000 7f8885400640 1 -- 172.21.15.28:0/2178142032 shutdown_connections 2024-02-21T23:03:05.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.989+0000 7f8885400640 1 -- 172.21.15.28:0/2178142032 wait complete. 2024-02-21T23:03:05.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.991+0000 7f8885400640 1 Processor -- start 2024-02-21T23:03:05.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.991+0000 7f8885400640 1 -- start start 2024-02-21T23:03:05.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.992+0000 7f8885400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f88640039b0 0x7f88801e9870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:05.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.992+0000 7f8885400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88801df8c0 0x7f88800c2680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:05.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.992+0000 7f8885400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f888014d590 0x7f88800a8330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:05.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.992+0000 7f8885400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88800c9480 con 0x7f88801df8c0 2024-02-21T23:03:05.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.992+0000 7f8885400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88800c9300 con 0x7f888014d590 2024-02-21T23:03:05.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.992+0000 7f8885400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8880156950 con 0x7f88640039b0 2024-02-21T23:03:05.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.992+0000 7f887de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88801df8c0 0x7f88800c2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:05.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.992+0000 7f887de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88801df8c0 0x7f88800c2680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45838/0 (socket says 172.21.15.28:45838) 2024-02-21T23:03:05.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.992+0000 7f887de00640 1 -- 172.21.15.28:0/3311042996 learned_addr learned my addr 172.21.15.28:0/3311042996 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:05.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.993+0000 7f887e800640 1 --2- 172.21.15.28:0/3311042996 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f88640039b0 0x7f88801e9870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:05.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.993+0000 7f887f200640 1 --2- 172.21.15.28:0/3311042996 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f888014d590 0x7f88800a8330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:05.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.993+0000 7f887de00640 1 -- 172.21.15.28:0/3311042996 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f88640039b0 msgr2=0x7f88801e9870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:05.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.993+0000 7f887de00640 1 --2- 172.21.15.28:0/3311042996 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f88640039b0 0x7f88801e9870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:05.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.993+0000 7f887de00640 1 -- 172.21.15.28:0/3311042996 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f888014d590 msgr2=0x7f88800a8330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:05.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.993+0000 7f887de00640 1 --2- 172.21.15.28:0/3311042996 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f888014d590 0x7f88800a8330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:05.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.993+0000 7f887de00640 1 -- 172.21.15.28:0/3311042996 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88801e0060 con 0x7f88801df8c0 2024-02-21T23:03:05.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.993+0000 7f887e800640 1 --2- 172.21.15.28:0/3311042996 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f88640039b0 0x7f88801e9870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:03:05.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.993+0000 7f887de00640 1 --2- 172.21.15.28:0/3311042996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88801df8c0 0x7f88800c2680 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f886c001050 tx=0x7f886c00eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:05.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.993+0000 7f8876a00640 1 -- 172.21.15.28:0/3311042996 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f886c00c660 con 0x7f88801df8c0 2024-02-21T23:03:05.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.994+0000 7f8876a00640 1 -- 172.21.15.28:0/3311042996 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f886c018070 con 0x7f88801df8c0 2024-02-21T23:03:05.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.994+0000 7f8885400640 1 -- 172.21.15.28:0/3311042996 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88801e0300 con 0x7f88801df8c0 2024-02-21T23:03:05.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.994+0000 7f8876a00640 1 -- 172.21.15.28:0/3311042996 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f886c00ca40 con 0x7f88801df8c0 2024-02-21T23:03:05.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.994+0000 7f8885400640 1 -- 172.21.15.28:0/3311042996 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88801e9070 con 0x7f88801df8c0 2024-02-21T23:03:05.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.995+0000 7f8885400640 1 -- 172.21.15.28:0/3311042996 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8880067f30 con 0x7f88801df8c0 2024-02-21T23:03:05.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.996+0000 7f8876a00640 1 -- 172.21.15.28:0/3311042996 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f886c00cbe0 con 0x7f88801df8c0 2024-02-21T23:03:05.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.997+0000 7f8876a00640 1 --2- 172.21.15.28:0/3311042996 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f886006b500 0x7f886006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:05.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.997+0000 7f8876a00640 1 -- 172.21.15.28:0/3311042996 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f886c08a260 con 0x7f88801df8c0 2024-02-21T23:03:05.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.998+0000 7f887e800640 1 --2- 172.21.15.28:0/3311042996 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f886006b500 0x7f886006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:05.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:05.999+0000 7f887e800640 1 --2- 172.21.15.28:0/3311042996 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f886006b500 0x7f886006d9c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8870036000 tx=0x7f8870030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:05.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:06.002+0000 7f8876a00640 1 -- 172.21.15.28:0/3311042996 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f886c058d90 con 0x7f88801df8c0 2024-02-21T23:03:06.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:06.242+0000 7f8885400640 1 -- 172.21.15.28:0/3311042996 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f88800c75d0 con 0x7f88801df8c0 2024-02-21T23:03:06.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:06.748+0000 7f8876a00640 1 -- 172.21.15.28:0/3311042996 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v9) v1 ==== 122+0+0 (secure 0 0 0) 0x7f886c058690 con 0x7f88801df8c0 2024-02-21T23:03:06.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:06.753+0000 7f8885400640 1 -- 172.21.15.28:0/3311042996 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f886006b500 msgr2=0x7f886006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:06.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:06.753+0000 7f8885400640 1 --2- 172.21.15.28:0/3311042996 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f886006b500 0x7f886006d9c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8870036000 tx=0x7f8870030000 comp rx=0 tx=0).stop 2024-02-21T23:03:06.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:06.753+0000 7f8885400640 1 -- 172.21.15.28:0/3311042996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88801df8c0 msgr2=0x7f88800c2680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:06.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:06.753+0000 7f8885400640 1 --2- 172.21.15.28:0/3311042996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88801df8c0 0x7f88800c2680 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f886c001050 tx=0x7f886c00eb90 comp rx=0 tx=0).stop 2024-02-21T23:03:06.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:06.753+0000 7f8885400640 1 -- 172.21.15.28:0/3311042996 shutdown_connections 2024-02-21T23:03:06.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:06.753+0000 7f8885400640 1 --2- 172.21.15.28:0/3311042996 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f886006b500 0x7f886006d9c0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:06.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:06.753+0000 7f8885400640 1 --2- 172.21.15.28:0/3311042996 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f888014d590 0x7f88800a8330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:06.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:06.753+0000 7f8885400640 1 --2- 172.21.15.28:0/3311042996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88801df8c0 0x7f88800c2680 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:06.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:06.753+0000 7f8885400640 1 --2- 172.21.15.28:0/3311042996 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f88640039b0 0x7f88801e9870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:06.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:06.753+0000 7f8885400640 1 -- 172.21.15.28:0/3311042996 >> 172.21.15.28:0/3311042996 conn(0x7f888006b290 msgr2=0x7f8880074360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:06.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:06.753+0000 7f8885400640 1 -- 172.21.15.28:0/3311042996 shutdown_connections 2024-02-21T23:03:06.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:06.754+0000 7f8885400640 1 -- 172.21.15.28:0/3311042996 wait complete. 2024-02-21T23:03:06.765 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-02-21T23:03:06.765 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:03:06.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:06.999+0000 7fa0da200640 1 Processor -- start 2024-02-21T23:03:06.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:06.999+0000 7fa0da200640 1 -- start start 2024-02-21T23:03:06.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.000+0000 7fa0da200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa0d40aff90 0x7fa0d40c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:06.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.001+0000 7fa0da200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0d40c74c0 con 0x7fa0d406dbf0 2024-02-21T23:03:06.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.001+0000 7fa0da200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0d40c77a0 con 0x7fa0d40c5720 2024-02-21T23:03:06.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.001+0000 7fa0da200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0d40c7270 con 0x7fa0d40aff90 2024-02-21T23:03:06.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.001+0000 7fa0d8e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa0d40aff90 0x7fa0d40c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:06.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.001+0000 7fa0cbe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa0d406dbf0 0x7fa0d4158ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:48514/0 (socket says 172.21.15.28:48514) 2024-02-21T23:03:06.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.001+0000 7fa0cbe00640 1 -- 172.21.15.28:0/829258796 learned_addr learned my addr 172.21.15.28:0/829258796 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:03:06.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.002+0000 7fa0d8e00640 1 -- 172.21.15.28:0/829258796 >> v1:172.21.15.81:6789/0 conn(0x7fa0d40c5720 legacy=0x7fa0d415f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:06.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.002+0000 7fa0d8e00640 1 -- 172.21.15.28:0/829258796 >> v1:172.21.15.28:6789/0 conn(0x7fa0d406dbf0 legacy=0x7fa0d4158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:06.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.002+0000 7fa0d8e00640 1 -- 172.21.15.28:0/829258796 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0d40c7a80 con 0x7fa0d40aff90 2024-02-21T23:03:06.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.003+0000 7fa0d8e00640 1 --2- 172.21.15.28:0/829258796 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0d40aff90 0x7fa0d40c51e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa0c40097a0 tx=0x7fa0c402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7654295d45dde8 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:06.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.004+0000 7fa0d3e00640 1 -- 172.21.15.28:0/829258796 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0c4034070 con 0x7fa0d40aff90 2024-02-21T23:03:06.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.004+0000 7fa0d3e00640 1 -- 172.21.15.28:0/829258796 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0c4030610 con 0x7fa0d40aff90 2024-02-21T23:03:06.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.004+0000 7fa0d3e00640 1 -- 172.21.15.28:0/829258796 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0c40309e0 con 0x7fa0d40aff90 2024-02-21T23:03:06.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.004+0000 7fa0da200640 1 -- 172.21.15.28:0/829258796 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0d40aff90 msgr2=0x7fa0d40c51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:06.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.004+0000 7fa0da200640 1 --2- 172.21.15.28:0/829258796 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0d40aff90 0x7fa0d40c51e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa0c40097a0 tx=0x7fa0c402b960 comp rx=0 tx=0).stop 2024-02-21T23:03:06.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.004+0000 7fa0da200640 1 -- 172.21.15.28:0/829258796 shutdown_connections 2024-02-21T23:03:06.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.004+0000 7fa0da200640 1 --2- 172.21.15.28:0/829258796 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0d40aff90 0x7fa0d40c51e0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:06.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.004+0000 7fa0da200640 1 -- 172.21.15.28:0/829258796 >> 172.21.15.28:0/829258796 conn(0x7fa0d40ac370 msgr2=0x7fa0d40ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:06.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.004+0000 7fa0da200640 1 -- 172.21.15.28:0/829258796 shutdown_connections 2024-02-21T23:03:06.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.004+0000 7fa0da200640 1 -- 172.21.15.28:0/829258796 wait complete. 2024-02-21T23:03:06.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.006+0000 7fa0da200640 1 Processor -- start 2024-02-21T23:03:06.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.006+0000 7fa0da200640 1 -- start start 2024-02-21T23:03:06.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.007+0000 7fa0da200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa0d406dbf0 0x7fa0d40af250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:06.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.007+0000 7fa0da200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa0d40aff90 0x7fa0d41e1c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:06.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.007+0000 7fa0da200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0d40c5720 0x7fa0d40ad8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:06.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.007+0000 7fa0da200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0d40c8760 con 0x7fa0d40c5720 2024-02-21T23:03:06.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.007+0000 7fa0da200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0d4158630 con 0x7fa0d40aff90 2024-02-21T23:03:06.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.007+0000 7fa0da200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0d40c88e0 con 0x7fa0d406dbf0 2024-02-21T23:03:06.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.007+0000 7fa0d9800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0d40c5720 0x7fa0d40ad8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:06.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.007+0000 7fa0d9800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0d40c5720 0x7fa0d40ad8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45860/0 (socket says 172.21.15.28:45860) 2024-02-21T23:03:06.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.007+0000 7fa0d9800640 1 -- 172.21.15.28:0/4153657370 learned_addr learned my addr 172.21.15.28:0/4153657370 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:06.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.008+0000 7fa0d8e00640 1 --2- 172.21.15.28:0/4153657370 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa0d406dbf0 0x7fa0d40af250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:06.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.008+0000 7fa0cbe00640 1 --2- 172.21.15.28:0/4153657370 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa0d40aff90 0x7fa0d41e1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:06.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.008+0000 7fa0d9800640 1 -- 172.21.15.28:0/4153657370 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa0d406dbf0 msgr2=0x7fa0d40af250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:06.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.008+0000 7fa0d9800640 1 --2- 172.21.15.28:0/4153657370 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa0d406dbf0 0x7fa0d40af250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:06.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.008+0000 7fa0d9800640 1 -- 172.21.15.28:0/4153657370 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa0d40aff90 msgr2=0x7fa0d41e1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:06.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.008+0000 7fa0d9800640 1 --2- 172.21.15.28:0/4153657370 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa0d40aff90 0x7fa0d41e1c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:06.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.008+0000 7fa0d9800640 1 -- 172.21.15.28:0/4153657370 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0d414f800 con 0x7fa0d40c5720 2024-02-21T23:03:06.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.008+0000 7fa0d8e00640 1 --2- 172.21.15.28:0/4153657370 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa0d406dbf0 0x7fa0d40af250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:03:06.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.008+0000 7fa0d9800640 1 --2- 172.21.15.28:0/4153657370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0d40c5720 0x7fa0d40ad8e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa0cc00c6b0 tx=0x7fa0cc00cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:06.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.008+0000 7fa0d1600640 1 -- 172.21.15.28:0/4153657370 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0cc003b70 con 0x7fa0d40c5720 2024-02-21T23:03:06.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.009+0000 7fa0d1600640 1 -- 172.21.15.28:0/4153657370 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0cc003d30 con 0x7fa0d40c5720 2024-02-21T23:03:06.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.009+0000 7fa0d1600640 1 -- 172.21.15.28:0/4153657370 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0cc005310 con 0x7fa0d40c5720 2024-02-21T23:03:06.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.009+0000 7fa0da200640 1 -- 172.21.15.28:0/4153657370 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0d4151480 con 0x7fa0d40c5720 2024-02-21T23:03:06.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.009+0000 7fa0da200640 1 -- 172.21.15.28:0/4153657370 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0d40aea30 con 0x7fa0d40c5720 2024-02-21T23:03:06.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.010+0000 7fa0da200640 1 -- 172.21.15.28:0/4153657370 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0d40c50e0 con 0x7fa0d40c5720 2024-02-21T23:03:07.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.011+0000 7fa0d1600640 1 -- 172.21.15.28:0/4153657370 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa0cc0054b0 con 0x7fa0d40c5720 2024-02-21T23:03:07.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.012+0000 7fa0d1600640 1 --2- 172.21.15.28:0/4153657370 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa0b006b4d0 0x7fa0b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:07.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.012+0000 7fa0d1600640 1 -- 172.21.15.28:0/4153657370 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fa0cc08a3d0 con 0x7fa0d40c5720 2024-02-21T23:03:07.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.012+0000 7fa0d8e00640 1 --2- 172.21.15.28:0/4153657370 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa0b006b4d0 0x7fa0b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:07.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.013+0000 7fa0d8e00640 1 --2- 172.21.15.28:0/4153657370 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa0b006b4d0 0x7fa0b006d990 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa0c4036000 tx=0x7fa0c403b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:07.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.016+0000 7fa0d1600640 1 -- 172.21.15.28:0/4153657370 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fa0d40c50e0 con 0x7fa0d40c5720 2024-02-21T23:03:07.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.258+0000 7fa0da200640 1 -- 172.21.15.28:0/4153657370 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa0d404f290 con 0x7fa0d40c5720 2024-02-21T23:03:07.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.260+0000 7fa0d1600640 1 -- 172.21.15.28:0/4153657370 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4934 (secure 0 0 0) 0x7fa0cc058f00 con 0x7fa0d40c5720 2024-02-21T23:03:07.248 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:03:07.249 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4230,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/2559793999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2559793999},{"type":"v1","addr":"172.21.15.28:6834","nonce":2559793999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4232,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3930208177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3930208177},{"type":"v1","addr":"172.21.15.28:6835","nonce":3930208177}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4238,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/2875984988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":2875984988},{"type":"v1","addr":"172.21.15.81:6836","nonce":2875984988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:02:37.812604+0000","modified":"2024-02-21T23:03:06.243463+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4267},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4267":{"gid":4267,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6837/2328151245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2328151245},{"type":"v1","addr":"172.21.15.81:6837","nonce":2328151245}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-02-21T23:03:07.249 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 9 2024-02-21T23:03:07.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.262+0000 7fa0da200640 1 -- 172.21.15.28:0/4153657370 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa0b006b4d0 msgr2=0x7fa0b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:07.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.262+0000 7fa0da200640 1 --2- 172.21.15.28:0/4153657370 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa0b006b4d0 0x7fa0b006d990 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa0c4036000 tx=0x7fa0c403b040 comp rx=0 tx=0).stop 2024-02-21T23:03:07.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.262+0000 7fa0da200640 1 -- 172.21.15.28:0/4153657370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0d40c5720 msgr2=0x7fa0d40ad8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:07.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.262+0000 7fa0da200640 1 --2- 172.21.15.28:0/4153657370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0d40c5720 0x7fa0d40ad8e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa0cc00c6b0 tx=0x7fa0cc00cb70 comp rx=0 tx=0).stop 2024-02-21T23:03:07.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.263+0000 7fa0da200640 1 -- 172.21.15.28:0/4153657370 shutdown_connections 2024-02-21T23:03:07.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.263+0000 7fa0da200640 1 --2- 172.21.15.28:0/4153657370 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa0b006b4d0 0x7fa0b006d990 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:07.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.263+0000 7fa0da200640 1 --2- 172.21.15.28:0/4153657370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0d40c5720 0x7fa0d40ad8e0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:07.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.263+0000 7fa0da200640 1 --2- 172.21.15.28:0/4153657370 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa0d40aff90 0x7fa0d41e1c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:07.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.263+0000 7fa0da200640 1 --2- 172.21.15.28:0/4153657370 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa0d406dbf0 0x7fa0d40af250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:07.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.263+0000 7fa0da200640 1 -- 172.21.15.28:0/4153657370 >> 172.21.15.28:0/4153657370 conn(0x7fa0d40ac370 msgr2=0x7fa0d41de570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:07.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.263+0000 7fa0da200640 1 -- 172.21.15.28:0/4153657370 shutdown_connections 2024-02-21T23:03:07.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.263+0000 7fa0da200640 1 -- 172.21.15.28:0/4153657370 wait complete. 2024-02-21T23:03:07.272 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:03:07.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.504+0000 7fe2b9800640 1 Processor -- start 2024-02-21T23:03:07.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.504+0000 7fe2b9800640 1 -- start start 2024-02-21T23:03:07.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.506+0000 7fe2b9800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fe2b40aff90 0x7fe2b40c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:07.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.506+0000 7fe2b9800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe2b40c5720 0x7fe2b415f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:07.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.506+0000 7fe2b9800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2b40c74c0 con 0x7fe2b406dbf0 2024-02-21T23:03:07.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.506+0000 7fe2b9800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2b40c7270 con 0x7fe2b40c5720 2024-02-21T23:03:07.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.506+0000 7fe2b9800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2b4158630 con 0x7fe2b40aff90 2024-02-21T23:03:07.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.506+0000 7fe2b8e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe2b40c5720 0x7fe2b415f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:07.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.506+0000 7fe2b8e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe2b40c5720 0x7fe2b415f890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45868/0 (socket says 172.21.15.28:45868) 2024-02-21T23:03:07.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.506+0000 7fe2b8e00640 1 -- 172.21.15.28:0/149908744 learned_addr learned my addr 172.21.15.28:0/149908744 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:07.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.506+0000 7fe2b3e00640 1 --2- 172.21.15.28:0/149908744 >> v2:172.21.15.81:3300/0 conn(0x7fe2b40aff90 0x7fe2b40c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:07.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.507+0000 7fe2b8e00640 1 -- 172.21.15.28:0/149908744 >> v2:172.21.15.81:3300/0 conn(0x7fe2b40aff90 msgr2=0x7fe2b40c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:07.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.507+0000 7fe2b8e00640 1 --2- 172.21.15.28:0/149908744 >> v2:172.21.15.81:3300/0 conn(0x7fe2b40aff90 0x7fe2b40c51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:07.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.507+0000 7fe2b8e00640 1 -- 172.21.15.28:0/149908744 >> v1:172.21.15.81:6789/0 conn(0x7fe2b406dbf0 legacy=0x7fe2b4158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:07.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.507+0000 7fe2b8e00640 1 -- 172.21.15.28:0/149908744 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2b40c77a0 con 0x7fe2b40c5720 2024-02-21T23:03:07.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.508+0000 7fe2b8e00640 1 --2- 172.21.15.28:0/149908744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2b40c5720 0x7fe2b415f890 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fe2b40c73f0 tx=0x7fe2ac02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b49c1cab17ce90e1 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:07.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.508+0000 7fe2b2a00640 1 -- 172.21.15.28:0/149908744 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2ac034070 con 0x7fe2b40c5720 2024-02-21T23:03:07.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.508+0000 7fe2b2a00640 1 -- 172.21.15.28:0/149908744 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2ac030860 con 0x7fe2b40c5720 2024-02-21T23:03:07.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.508+0000 7fe2b2a00640 1 -- 172.21.15.28:0/149908744 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2ac030c30 con 0x7fe2b40c5720 2024-02-21T23:03:07.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.509+0000 7fe2b9800640 1 -- 172.21.15.28:0/149908744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2b40c5720 msgr2=0x7fe2b415f890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:07.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.509+0000 7fe2b9800640 1 --2- 172.21.15.28:0/149908744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2b40c5720 0x7fe2b415f890 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fe2b40c73f0 tx=0x7fe2ac02b9a0 comp rx=0 tx=0).stop 2024-02-21T23:03:07.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.509+0000 7fe2b9800640 1 -- 172.21.15.28:0/149908744 shutdown_connections 2024-02-21T23:03:07.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.509+0000 7fe2b9800640 1 --2- 172.21.15.28:0/149908744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2b40c5720 0x7fe2b415f890 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:07.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.509+0000 7fe2b9800640 1 --2- 172.21.15.28:0/149908744 >> v2:172.21.15.81:3300/0 conn(0x7fe2b40aff90 0x7fe2b40c51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:07.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.509+0000 7fe2b9800640 1 -- 172.21.15.28:0/149908744 >> 172.21.15.28:0/149908744 conn(0x7fe2b40ac370 msgr2=0x7fe2b40ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:07.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.509+0000 7fe2b9800640 1 -- 172.21.15.28:0/149908744 shutdown_connections 2024-02-21T23:03:07.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.509+0000 7fe2b9800640 1 -- 172.21.15.28:0/149908744 wait complete. 2024-02-21T23:03:07.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.511+0000 7fe2b9800640 1 Processor -- start 2024-02-21T23:03:07.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.511+0000 7fe2b9800640 1 -- start start 2024-02-21T23:03:07.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.512+0000 7fe2b9800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe2b406dbf0 0x7fe2b40c3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:07.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.512+0000 7fe2b9800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2b40aff90 0x7fe2b40c2900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:07.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.512+0000 7fe2b9800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2b40c5720 0x7fe2b40c2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:07.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.512+0000 7fe2b9800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b40c8760 con 0x7fe2b40aff90 2024-02-21T23:03:07.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.512+0000 7fe2b9800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b40be0c0 con 0x7fe2b40c5720 2024-02-21T23:03:07.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.512+0000 7fe2b9800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b40c88e0 con 0x7fe2b406dbf0 2024-02-21T23:03:07.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.512+0000 7fe2b3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2b40aff90 0x7fe2b40c2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:07.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.512+0000 7fe2b3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2b40aff90 0x7fe2b40c2900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45884/0 (socket says 172.21.15.28:45884) 2024-02-21T23:03:07.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.512+0000 7fe2b3400640 1 -- 172.21.15.28:0/2119788767 learned_addr learned my addr 172.21.15.28:0/2119788767 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:07.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.512+0000 7fe2b3e00640 1 --2- 172.21.15.28:0/2119788767 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe2b406dbf0 0x7fe2b40c3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:07.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.512+0000 7fe2b8e00640 1 --2- 172.21.15.28:0/2119788767 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2b40c5720 0x7fe2b40c2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:07.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.513+0000 7fe2b3400640 1 -- 172.21.15.28:0/2119788767 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe2b406dbf0 msgr2=0x7fe2b40c3670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:07.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.513+0000 7fe2b3400640 1 --2- 172.21.15.28:0/2119788767 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe2b406dbf0 0x7fe2b40c3670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:07.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.513+0000 7fe2b3400640 1 -- 172.21.15.28:0/2119788767 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2b40c5720 msgr2=0x7fe2b40c2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:07.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.513+0000 7fe2b3400640 1 --2- 172.21.15.28:0/2119788767 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2b40c5720 0x7fe2b40c2e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:07.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.513+0000 7fe2b3400640 1 -- 172.21.15.28:0/2119788767 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2b40c1440 con 0x7fe2b40aff90 2024-02-21T23:03:07.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.513+0000 7fe2b3400640 1 --2- 172.21.15.28:0/2119788767 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2b40aff90 0x7fe2b40c2900 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fe29c00c440 tx=0x7fe29c001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:07.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.513+0000 7fe2abe00640 1 -- 172.21.15.28:0/2119788767 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe29c0052d0 con 0x7fe2b40aff90 2024-02-21T23:03:07.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.513+0000 7fe2abe00640 1 -- 172.21.15.28:0/2119788767 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe29c005470 con 0x7fe2b40aff90 2024-02-21T23:03:07.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.513+0000 7fe2b9800640 1 -- 172.21.15.28:0/2119788767 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2b40c16e0 con 0x7fe2b40aff90 2024-02-21T23:03:07.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.513+0000 7fe2abe00640 1 -- 172.21.15.28:0/2119788767 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe29c007260 con 0x7fe2b40aff90 2024-02-21T23:03:07.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.513+0000 7fe2b9800640 1 -- 172.21.15.28:0/2119788767 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2b40c23e0 con 0x7fe2b40aff90 2024-02-21T23:03:07.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.514+0000 7fe2b9800640 1 -- 172.21.15.28:0/2119788767 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2b404f290 con 0x7fe2b40aff90 2024-02-21T23:03:07.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.515+0000 7fe2abe00640 1 -- 172.21.15.28:0/2119788767 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe29c007400 con 0x7fe2b40aff90 2024-02-21T23:03:07.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.516+0000 7fe2abe00640 1 --2- 172.21.15.28:0/2119788767 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe29006b4d0 0x7fe29006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:07.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.516+0000 7fe2abe00640 1 -- 172.21.15.28:0/2119788767 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fe29c089a90 con 0x7fe2b40aff90 2024-02-21T23:03:07.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.517+0000 7fe2b3e00640 1 --2- 172.21.15.28:0/2119788767 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe29006b4d0 0x7fe29006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:07.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.518+0000 7fe2b3e00640 1 --2- 172.21.15.28:0/2119788767 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe29006b4d0 0x7fe29006d990 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe2a4009a60 tx=0x7fe2a4003990 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:07.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.521+0000 7fe2abe00640 1 -- 172.21.15.28:0/2119788767 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe29c0585c0 con 0x7fe2b40aff90 2024-02-21T23:03:07.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.719+0000 7fe2b9800640 1 -- 172.21.15.28:0/2119788767 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe2b406d9d0 con 0x7fe2b40aff90 2024-02-21T23:03:07.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.722+0000 7fe2abe00640 1 -- 172.21.15.28:0/2119788767 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18454 (secure 0 0 0) 0x7fe29c057ec0 con 0x7fe2b40aff90 2024-02-21T23:03:07.711 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:03:07.712 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":24,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:02:42.900286+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:02:35.795158+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-21T23:02:36.840110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-02-21T23:02:40.441388+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:03:07.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.725+0000 7fe2b9800640 1 -- 172.21.15.28:0/2119788767 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe29006b4d0 msgr2=0x7fe29006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:07.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.725+0000 7fe2b9800640 1 --2- 172.21.15.28:0/2119788767 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe29006b4d0 0x7fe29006d990 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe2a4009a60 tx=0x7fe2a4003990 comp rx=0 tx=0).stop 2024-02-21T23:03:07.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.725+0000 7fe2b9800640 1 -- 172.21.15.28:0/2119788767 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2b40aff90 msgr2=0x7fe2b40c2900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:07.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.725+0000 7fe2b9800640 1 --2- 172.21.15.28:0/2119788767 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2b40aff90 0x7fe2b40c2900 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fe29c00c440 tx=0x7fe29c001e70 comp rx=0 tx=0).stop 2024-02-21T23:03:07.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.725+0000 7fe2b9800640 1 -- 172.21.15.28:0/2119788767 shutdown_connections 2024-02-21T23:03:07.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.725+0000 7fe2b9800640 1 --2- 172.21.15.28:0/2119788767 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe29006b4d0 0x7fe29006d990 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:07.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.725+0000 7fe2b9800640 1 --2- 172.21.15.28:0/2119788767 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2b40c5720 0x7fe2b40c2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:07.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.725+0000 7fe2b9800640 1 --2- 172.21.15.28:0/2119788767 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2b40aff90 0x7fe2b40c2900 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:07.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.725+0000 7fe2b9800640 1 --2- 172.21.15.28:0/2119788767 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe2b406dbf0 0x7fe2b40c3670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:07.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.725+0000 7fe2b9800640 1 -- 172.21.15.28:0/2119788767 >> 172.21.15.28:0/2119788767 conn(0x7fe2b40ac370 msgr2=0x7fe2b40ac750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:07.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.725+0000 7fe2b9800640 1 -- 172.21.15.28:0/2119788767 shutdown_connections 2024-02-21T23:03:07.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.725+0000 7fe2b9800640 1 -- 172.21.15.28:0/2119788767 wait complete. 2024-02-21T23:03:07.733 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-02-21T23:03:07.734 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-21T23:03:07.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.964+0000 7fd3be000640 1 Processor -- start 2024-02-21T23:03:07.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.964+0000 7fd3be000640 1 -- start start 2024-02-21T23:03:07.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.966+0000 7fd3be000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fd3b806ee40 0x7fd3b80a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:07.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.966+0000 7fd3be000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd3b8156af0 0x7fd3b806dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:07.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.966+0000 7fd3be000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3b80c2760 con 0x7fd3b8150450 2024-02-21T23:03:07.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.966+0000 7fd3be000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3b80c22c0 con 0x7fd3b806ee40 2024-02-21T23:03:07.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.966+0000 7fd3be000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b8156660 con 0x7fd3b8156af0 2024-02-21T23:03:07.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.966+0000 7fd3bcc00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd3b8150450 0x7fd3b81536f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:48518/0 (socket says 172.21.15.28:48518) 2024-02-21T23:03:07.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.966+0000 7fd3bcc00640 1 -- 172.21.15.28:0/195878200 learned_addr learned my addr 172.21.15.28:0/195878200 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:03:07.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.967+0000 7fd3bd600640 1 --2- 172.21.15.28:0/195878200 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd3b8156af0 0x7fd3b806dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:07.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.966+0000 7fd3b7e00640 1 --2- 172.21.15.28:0/195878200 >> v2:172.21.15.81:3300/0 conn(0x7fd3b806ee40 0x7fd3b80a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:07.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.967+0000 7fd3b7400640 1 -- 172.21.15.28:0/195878200 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3700893887 0 0) 0x7fd3b80c2760 con 0x7fd3b8150450 2024-02-21T23:03:07.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.967+0000 7fd3b7400640 1 -- 172.21.15.28:0/195878200 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3a0003520 con 0x7fd3b8150450 2024-02-21T23:03:07.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.967+0000 7fd3b7400640 1 -- 172.21.15.28:0/195878200 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4063605917 0 0) 0x7fd3a0003520 con 0x7fd3b8150450 2024-02-21T23:03:07.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.967+0000 7fd3b7400640 1 -- 172.21.15.28:0/195878200 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd3b8156af0 msgr2=0x7fd3b806dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:07.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.967+0000 7fd3b7400640 1 --2- 172.21.15.28:0/195878200 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd3b8156af0 0x7fd3b806dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:07.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.967+0000 7fd3b7400640 1 -- 172.21.15.28:0/195878200 >> v2:172.21.15.81:3300/0 conn(0x7fd3b806ee40 msgr2=0x7fd3b80a5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:07.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.967+0000 7fd3b7400640 1 --2- 172.21.15.28:0/195878200 >> v2:172.21.15.81:3300/0 conn(0x7fd3b806ee40 0x7fd3b80a5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:07.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.967+0000 7fd3b7400640 1 -- 172.21.15.28:0/195878200 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3b8156310 con 0x7fd3b8150450 2024-02-21T23:03:07.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.967+0000 7fd3b7400640 1 -- 172.21.15.28:0/195878200 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fd3ac001670 con 0x7fd3b8150450 2024-02-21T23:03:07.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.967+0000 7fd3b7e00640 1 --2- 172.21.15.28:0/195878200 >> v2:172.21.15.81:3300/0 conn(0x7fd3b806ee40 0x7fd3b80a5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-21T23:03:07.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.967+0000 7fd3b7400640 1 -- 172.21.15.28:0/195878200 >> v1:172.21.15.28:6789/0 conn(0x7fd3b8150450 legacy=0x7fd3b81536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:07.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.967+0000 7fd3b7400640 1 --2- 172.21.15.28:0/195878200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3a00039b0 0x7fd3a0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:07.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.968+0000 7fd3b7400640 1 --2- 172.21.15.28:0/195878200 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd3b8156af0 0x7fd3a0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:07.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.968+0000 7fd3b7400640 1 --2- 172.21.15.28:0/195878200 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd3a0004d40 0x7fd3a00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:07.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.968+0000 7fd3b7400640 1 -- 172.21.15.28:0/195878200 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b8156660 con 0x7fd3a00039b0 2024-02-21T23:03:07.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.968+0000 7fd3b7400640 1 -- 172.21.15.28:0/195878200 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a000b9c0 con 0x7fd3a0004d40 2024-02-21T23:03:07.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.968+0000 7fd3b7400640 1 -- 172.21.15.28:0/195878200 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a000bb40 con 0x7fd3b8156af0 2024-02-21T23:03:07.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.968+0000 7fd3bcc00640 1 --2- 172.21.15.28:0/195878200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3a00039b0 0x7fd3a0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:07.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.968+0000 7fd3b7e00640 1 --2- 172.21.15.28:0/195878200 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd3b8156af0 0x7fd3a0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:07.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.968+0000 7fd3bd600640 1 --2- 172.21.15.28:0/195878200 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd3a0004d40 0x7fd3a00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:07.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.968+0000 7fd3bcc00640 1 -- 172.21.15.28:0/195878200 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd3b8156af0 msgr2=0x7fd3a0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:07.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.968+0000 7fd3bcc00640 1 --2- 172.21.15.28:0/195878200 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd3b8156af0 0x7fd3a0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:07.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.968+0000 7fd3bcc00640 1 -- 172.21.15.28:0/195878200 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd3a0004d40 msgr2=0x7fd3a00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:07.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.968+0000 7fd3bcc00640 1 --2- 172.21.15.28:0/195878200 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd3a0004d40 0x7fd3a00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:07.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.968+0000 7fd3bcc00640 1 -- 172.21.15.28:0/195878200 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3a0003520 con 0x7fd3a00039b0 2024-02-21T23:03:07.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.968+0000 7fd3b7e00640 1 --2- 172.21.15.28:0/195878200 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd3b8156af0 0x7fd3a0004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:03:07.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.970+0000 7fd3bcc00640 1 --2- 172.21.15.28:0/195878200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3a00039b0 0x7fd3a0003dd0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac00c2c0 tx=0x7fd3a0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba9ee565c1544e1a server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:07.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.970+0000 7fd3b7400640 1 -- 172.21.15.28:0/195878200 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3ac007d50 con 0x7fd3a00039b0 2024-02-21T23:03:07.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.970+0000 7fd3b7400640 1 -- 172.21.15.28:0/195878200 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3ac030070 con 0x7fd3a00039b0 2024-02-21T23:03:07.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.970+0000 7fd3b7400640 1 -- 172.21.15.28:0/195878200 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3ac02cd60 con 0x7fd3a00039b0 2024-02-21T23:03:07.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.971+0000 7fd3be000640 1 -- 172.21.15.28:0/195878200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3a00039b0 msgr2=0x7fd3a0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:07.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.971+0000 7fd3be000640 1 --2- 172.21.15.28:0/195878200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3a00039b0 0x7fd3a0003dd0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac00c2c0 tx=0x7fd3a0005830 comp rx=0 tx=0).stop 2024-02-21T23:03:07.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.971+0000 7fd3bd600640 1 -- 172.21.15.28:0/195878200 reap_dead start 2024-02-21T23:03:07.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.971+0000 7fd3be000640 1 -- 172.21.15.28:0/195878200 shutdown_connections 2024-02-21T23:03:07.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.971+0000 7fd3be000640 1 -- 172.21.15.28:0/195878200 >> 172.21.15.28:0/195878200 conn(0x7fd3b806b630 msgr2=0x7fd3b80c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:07.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.971+0000 7fd3be000640 1 -- 172.21.15.28:0/195878200 shutdown_connections 2024-02-21T23:03:07.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.971+0000 7fd3be000640 1 -- 172.21.15.28:0/195878200 wait complete. 2024-02-21T23:03:07.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.973+0000 7fd3be000640 1 Processor -- start 2024-02-21T23:03:07.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.973+0000 7fd3be000640 1 -- start start 2024-02-21T23:03:07.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.974+0000 7fd3be000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd3b81ebab0 0x7fd3b8149750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:07.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.974+0000 7fd3be000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3b80c0530 0x7fd3b8149c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:07.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.974+0000 7fd3be000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd3b81ebe90 0x7fd3b81e9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:07.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.974+0000 7fd3be000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b8156f80 con 0x7fd3b80c0530 2024-02-21T23:03:07.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.974+0000 7fd3be000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b80c2cb0 con 0x7fd3b81ebe90 2024-02-21T23:03:07.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.974+0000 7fd3be000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b8157100 con 0x7fd3b81ebab0 2024-02-21T23:03:07.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.974+0000 7fd3b7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3b80c0530 0x7fd3b8149c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:07.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.974+0000 7fd3b7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3b80c0530 0x7fd3b8149c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45900/0 (socket says 172.21.15.28:45900) 2024-02-21T23:03:07.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.974+0000 7fd3b7e00640 1 -- 172.21.15.28:0/1811421028 learned_addr learned my addr 172.21.15.28:0/1811421028 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:07.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.974+0000 7fd3bd600640 1 --2- 172.21.15.28:0/1811421028 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd3b81ebe90 0x7fd3b81e9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:07.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.974+0000 7fd3bcc00640 1 --2- 172.21.15.28:0/1811421028 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd3b81ebab0 0x7fd3b8149750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:07.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.975+0000 7fd3b7e00640 1 -- 172.21.15.28:0/1811421028 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd3b81ebab0 msgr2=0x7fd3b8149750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:07.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.975+0000 7fd3b7e00640 1 --2- 172.21.15.28:0/1811421028 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd3b81ebab0 0x7fd3b8149750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:07.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.975+0000 7fd3b7e00640 1 -- 172.21.15.28:0/1811421028 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd3b81ebe90 msgr2=0x7fd3b81e9fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:07.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.975+0000 7fd3b7e00640 1 --2- 172.21.15.28:0/1811421028 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd3b81ebe90 0x7fd3b81e9fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:07.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.975+0000 7fd3b7e00640 1 -- 172.21.15.28:0/1811421028 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3b81ec3f0 con 0x7fd3b80c0530 2024-02-21T23:03:07.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.975+0000 7fd3bd600640 1 --2- 172.21.15.28:0/1811421028 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd3b81ebe90 0x7fd3b81e9fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:03:07.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.975+0000 7fd3b7e00640 1 --2- 172.21.15.28:0/1811421028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3b80c0530 0x7fd3b8149c90 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd3a4011900 tx=0x7fd3a4011dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:07.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.975+0000 7fd3b4c00640 1 -- 172.21.15.28:0/1811421028 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a400fb70 con 0x7fd3b80c0530 2024-02-21T23:03:07.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.975+0000 7fd3b4c00640 1 -- 172.21.15.28:0/1811421028 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3a400fd10 con 0x7fd3b80c0530 2024-02-21T23:03:07.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.975+0000 7fd3b4c00640 1 -- 172.21.15.28:0/1811421028 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a4015660 con 0x7fd3b80c0530 2024-02-21T23:03:07.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.975+0000 7fd3be000640 1 -- 172.21.15.28:0/1811421028 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3b81ec6d0 con 0x7fd3b80c0530 2024-02-21T23:03:07.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.976+0000 7fd3be000640 1 -- 172.21.15.28:0/1811421028 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3b8152db0 con 0x7fd3b80c0530 2024-02-21T23:03:07.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.977+0000 7fd3be000640 1 -- 172.21.15.28:0/1811421028 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3b804f290 con 0x7fd3b80c0530 2024-02-21T23:03:07.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.978+0000 7fd3b4c00640 1 -- 172.21.15.28:0/1811421028 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd3a4015800 con 0x7fd3b80c0530 2024-02-21T23:03:07.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.979+0000 7fd3b4c00640 1 --2- 172.21.15.28:0/1811421028 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd39806b500 0x7fd39806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:07.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.979+0000 7fd3b4c00640 1 -- 172.21.15.28:0/1811421028 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fd3a408af00 con 0x7fd3b80c0530 2024-02-21T23:03:07.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.979+0000 7fd3bcc00640 1 --2- 172.21.15.28:0/1811421028 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd39806b500 0x7fd39806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:07.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.981+0000 7fd3bcc00640 1 --2- 172.21.15.28:0/1811421028 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd39806b500 0x7fd39806d9c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac034000 tx=0x7fd3ac007000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:07.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:07.984+0000 7fd3b4c00640 1 -- 172.21.15.28:0/1811421028 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd3a4019030 con 0x7fd3b80c0530 2024-02-21T23:03:08.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.180+0000 7fd3be000640 1 -- 172.21.15.28:0/1811421028 --> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd3b80c1f10 con 0x7fd39806b500 2024-02-21T23:03:08.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.210+0000 7fd3b4c00640 1 -- 172.21.15.28:0/1811421028 <== mgr.4101 v2:172.21.15.81:6832/1848538577 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+332615 (secure 0 0 0) 0x7fd3b80c1f10 con 0x7fd39806b500 2024-02-21T23:03:08.204 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:03:08.206 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-21T23:03:08.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.218+0000 7fd3be000640 1 -- 172.21.15.28:0/1811421028 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd39806b500 msgr2=0x7fd39806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:08.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.218+0000 7fd3be000640 1 --2- 172.21.15.28:0/1811421028 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd39806b500 0x7fd39806d9c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac034000 tx=0x7fd3ac007000 comp rx=0 tx=0).stop 2024-02-21T23:03:08.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.218+0000 7fd3be000640 1 -- 172.21.15.28:0/1811421028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3b80c0530 msgr2=0x7fd3b8149c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:08.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.218+0000 7fd3be000640 1 --2- 172.21.15.28:0/1811421028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3b80c0530 0x7fd3b8149c90 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd3a4011900 tx=0x7fd3a4011dc0 comp rx=0 tx=0).stop 2024-02-21T23:03:08.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.218+0000 7fd3be000640 1 -- 172.21.15.28:0/1811421028 shutdown_connections 2024-02-21T23:03:08.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.218+0000 7fd3be000640 1 --2- 172.21.15.28:0/1811421028 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd39806b500 0x7fd39806d9c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:08.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.218+0000 7fd3be000640 1 --2- 172.21.15.28:0/1811421028 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd3b81ebe90 0x7fd3b81e9fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:08.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.218+0000 7fd3be000640 1 --2- 172.21.15.28:0/1811421028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3b80c0530 0x7fd3b8149c90 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:08.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.218+0000 7fd3be000640 1 --2- 172.21.15.28:0/1811421028 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd3b81ebab0 0x7fd3b8149750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:08.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.218+0000 7fd3be000640 1 -- 172.21.15.28:0/1811421028 >> 172.21.15.28:0/1811421028 conn(0x7fd3b806b630 msgr2=0x7fd3b814df20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:08.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.218+0000 7fd3be000640 1 -- 172.21.15.28:0/1811421028 shutdown_connections 2024-02-21T23:03:08.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.218+0000 7fd3be000640 1 -- 172.21.15.28:0/1811421028 wait complete. 2024-02-21T23:03:08.229 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2024-02-21T23:03:06.667116+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":208,"num_write_kb":2435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160,"ondisk_log_size":160,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":222496,"kb_used_data":2176,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754752224,"statfs":{"total":773094113280,"available":772866277376,"internally_reserved":0,"allocated":2228224,"data_stored":1992064,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":225496625},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1781,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005118"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":26,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.598140+0000","last_change":"2024-02-21T23:02:37.367904+0000","last_active":"2024-02-21T23:02:43.598140+0000","last_peered":"2024-02-21T23:02:43.598140+0000","last_clean":"2024-02-21T23:02:43.598140+0000","last_became_active":"2024-02-21T23:02:37.367537+0000","last_became_peered":"2024-02-21T23:02:37.367537+0000","last_unstale":"2024-02-21T23:02:43.598140+0000","last_undegraded":"2024-02-21T23:02:43.598140+0000","last_fullsized":"2024-02-21T23:02:43.598140+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:39:34.395881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420749+0000","last_change":"2024-02-21T23:02:38.385115+0000","last_active":"2024-02-21T23:02:43.420749+0000","last_peered":"2024-02-21T23:02:43.420749+0000","last_clean":"2024-02-21T23:02:43.420749+0000","last_became_active":"2024-02-21T23:02:38.384818+0000","last_became_peered":"2024-02-21T23:02:38.384818+0000","last_unstale":"2024-02-21T23:02:43.420749+0000","last_undegraded":"2024-02-21T23:02:43.420749+0000","last_fullsized":"2024-02-21T23:02:43.420749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:54:40.962238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.419345+0000","last_change":"2024-02-21T23:02:42.500553+0000","last_active":"2024-02-21T23:02:43.419345+0000","last_peered":"2024-02-21T23:02:43.419345+0000","last_clean":"2024-02-21T23:02:43.419345+0000","last_became_active":"2024-02-21T23:02:42.498884+0000","last_became_peered":"2024-02-21T23:02:42.498884+0000","last_unstale":"2024-02-21T23:02:43.419345+0000","last_undegraded":"2024-02-21T23:02:43.419345+0000","last_fullsized":"2024-02-21T23:02:43.419345+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:41:42.562121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.421106+0000","last_change":"2024-02-21T23:02:37.354176+0000","last_active":"2024-02-21T23:02:43.421106+0000","last_peered":"2024-02-21T23:02:43.421106+0000","last_clean":"2024-02-21T23:02:43.421106+0000","last_became_active":"2024-02-21T23:02:37.353425+0000","last_became_peered":"2024-02-21T23:02:37.353425+0000","last_unstale":"2024-02-21T23:02:43.421106+0000","last_undegraded":"2024-02-21T23:02:43.421106+0000","last_fullsized":"2024-02-21T23:02:43.421106+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:07:23.801765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.597346+0000","last_change":"2024-02-21T23:02:38.399361+0000","last_active":"2024-02-21T23:02:43.597346+0000","last_peered":"2024-02-21T23:02:43.597346+0000","last_clean":"2024-02-21T23:02:43.597346+0000","last_became_active":"2024-02-21T23:02:38.399026+0000","last_became_peered":"2024-02-21T23:02:38.399026+0000","last_unstale":"2024-02-21T23:02:43.597346+0000","last_undegraded":"2024-02-21T23:02:43.597346+0000","last_fullsized":"2024-02-21T23:02:43.597346+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:29:12.483912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.419463+0000","last_change":"2024-02-21T23:02:42.441232+0000","last_active":"2024-02-21T23:02:43.419463+0000","last_peered":"2024-02-21T23:02:43.419463+0000","last_clean":"2024-02-21T23:02:43.419463+0000","last_became_active":"2024-02-21T23:02:42.440354+0000","last_became_peered":"2024-02-21T23:02:42.440354+0000","last_unstale":"2024-02-21T23:02:43.419463+0000","last_undegraded":"2024-02-21T23:02:43.419463+0000","last_fullsized":"2024-02-21T23:02:43.419463+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:29:16.551658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.598167+0000","last_change":"2024-02-21T23:02:37.373520+0000","last_active":"2024-02-21T23:02:43.598167+0000","last_peered":"2024-02-21T23:02:43.598167+0000","last_clean":"2024-02-21T23:02:43.598167+0000","last_became_active":"2024-02-21T23:02:37.373127+0000","last_became_peered":"2024-02-21T23:02:37.373127+0000","last_unstale":"2024-02-21T23:02:43.598167+0000","last_undegraded":"2024-02-21T23:02:43.598167+0000","last_fullsized":"2024-02-21T23:02:43.598167+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:12:45.623160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.600822+0000","last_change":"2024-02-21T23:02:38.387666+0000","last_active":"2024-02-21T23:02:43.600822+0000","last_peered":"2024-02-21T23:02:43.600822+0000","last_clean":"2024-02-21T23:02:43.600822+0000","last_became_active":"2024-02-21T23:02:38.387151+0000","last_became_peered":"2024-02-21T23:02:38.387151+0000","last_unstale":"2024-02-21T23:02:43.600822+0000","last_undegraded":"2024-02-21T23:02:43.600822+0000","last_fullsized":"2024-02-21T23:02:43.600822+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:24:40.762404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420258+0000","last_change":"2024-02-21T23:02:42.444676+0000","last_active":"2024-02-21T23:02:43.420258+0000","last_peered":"2024-02-21T23:02:43.420258+0000","last_clean":"2024-02-21T23:02:43.420258+0000","last_became_active":"2024-02-21T23:02:42.444185+0000","last_became_peered":"2024-02-21T23:02:42.444185+0000","last_unstale":"2024-02-21T23:02:43.420258+0000","last_undegraded":"2024-02-21T23:02:43.420258+0000","last_fullsized":"2024-02-21T23:02:43.420258+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:54:45.029984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.597339+0000","last_change":"2024-02-21T23:02:37.372881+0000","last_active":"2024-02-21T23:02:43.597339+0000","last_peered":"2024-02-21T23:02:43.597339+0000","last_clean":"2024-02-21T23:02:43.597339+0000","last_became_active":"2024-02-21T23:02:37.372557+0000","last_became_peered":"2024-02-21T23:02:37.372557+0000","last_unstale":"2024-02-21T23:02:43.597339+0000","last_undegraded":"2024-02-21T23:02:43.597339+0000","last_fullsized":"2024-02-21T23:02:43.597339+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:29:33.262208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.600960+0000","last_change":"2024-02-21T23:02:38.399333+0000","last_active":"2024-02-21T23:02:43.600960+0000","last_peered":"2024-02-21T23:02:43.600960+0000","last_clean":"2024-02-21T23:02:43.600960+0000","last_became_active":"2024-02-21T23:02:38.393010+0000","last_became_peered":"2024-02-21T23:02:38.393010+0000","last_unstale":"2024-02-21T23:02:43.600960+0000","last_undegraded":"2024-02-21T23:02:43.600960+0000","last_fullsized":"2024-02-21T23:02:43.600960+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:54:16.651108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.419916+0000","last_change":"2024-02-21T23:02:42.436700+0000","last_active":"2024-02-21T23:02:43.419916+0000","last_peered":"2024-02-21T23:02:43.419916+0000","last_clean":"2024-02-21T23:02:43.419916+0000","last_became_active":"2024-02-21T23:02:42.434264+0000","last_became_peered":"2024-02-21T23:02:42.434264+0000","last_unstale":"2024-02-21T23:02:43.419916+0000","last_undegraded":"2024-02-21T23:02:43.419916+0000","last_fullsized":"2024-02-21T23:02:43.419916+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:28:47.259045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.594139+0000","last_change":"2024-02-21T23:02:38.391008+0000","last_active":"2024-02-21T23:02:43.594139+0000","last_peered":"2024-02-21T23:02:43.594139+0000","last_clean":"2024-02-21T23:02:43.594139+0000","last_became_active":"2024-02-21T23:02:38.390695+0000","last_became_peered":"2024-02-21T23:02:38.390695+0000","last_unstale":"2024-02-21T23:02:43.594139+0000","last_undegraded":"2024-02-21T23:02:43.594139+0000","last_fullsized":"2024-02-21T23:02:43.594139+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:09:34.130536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.598232+0000","last_change":"2024-02-21T23:02:37.373206+0000","last_active":"2024-02-21T23:02:43.598232+0000","last_peered":"2024-02-21T23:02:43.598232+0000","last_clean":"2024-02-21T23:02:43.598232+0000","last_became_active":"2024-02-21T23:02:37.371805+0000","last_became_peered":"2024-02-21T23:02:37.371805+0000","last_unstale":"2024-02-21T23:02:43.598232+0000","last_undegraded":"2024-02-21T23:02:43.598232+0000","last_fullsized":"2024-02-21T23:02:43.598232+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:29:26.321681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420841+0000","last_change":"2024-02-21T23:02:42.435351+0000","last_active":"2024-02-21T23:02:43.420841+0000","last_peered":"2024-02-21T23:02:43.420841+0000","last_clean":"2024-02-21T23:02:43.420841+0000","last_became_active":"2024-02-21T23:02:42.434903+0000","last_became_peered":"2024-02-21T23:02:42.434903+0000","last_unstale":"2024-02-21T23:02:43.420841+0000","last_undegraded":"2024-02-21T23:02:43.420841+0000","last_fullsized":"2024-02-21T23:02:43.420841+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:49:24.584877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420313+0000","last_change":"2024-02-21T23:02:38.369472+0000","last_active":"2024-02-21T23:02:43.420313+0000","last_peered":"2024-02-21T23:02:43.420313+0000","last_clean":"2024-02-21T23:02:43.420313+0000","last_became_active":"2024-02-21T23:02:38.369189+0000","last_became_peered":"2024-02-21T23:02:38.369189+0000","last_unstale":"2024-02-21T23:02:43.420313+0000","last_undegraded":"2024-02-21T23:02:43.420313+0000","last_fullsized":"2024-02-21T23:02:43.420313+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:00:09.172059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.596428+0000","last_change":"2024-02-21T23:02:37.355429+0000","last_active":"2024-02-21T23:02:43.596428+0000","last_peered":"2024-02-21T23:02:43.596428+0000","last_clean":"2024-02-21T23:02:43.596428+0000","last_became_active":"2024-02-21T23:02:37.354573+0000","last_became_peered":"2024-02-21T23:02:37.354573+0000","last_unstale":"2024-02-21T23:02:43.596428+0000","last_undegraded":"2024-02-21T23:02:43.596428+0000","last_fullsized":"2024-02-21T23:02:43.596428+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:17:46.451150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420650+0000","last_change":"2024-02-21T23:02:42.445184+0000","last_active":"2024-02-21T23:02:43.420650+0000","last_peered":"2024-02-21T23:02:43.420650+0000","last_clean":"2024-02-21T23:02:43.420650+0000","last_became_active":"2024-02-21T23:02:42.444773+0000","last_became_peered":"2024-02-21T23:02:42.444773+0000","last_unstale":"2024-02-21T23:02:43.420650+0000","last_undegraded":"2024-02-21T23:02:43.420650+0000","last_fullsized":"2024-02-21T23:02:43.420650+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:24:44.830150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.419669+0000","last_change":"2024-02-21T23:02:38.379076+0000","last_active":"2024-02-21T23:02:43.419669+0000","last_peered":"2024-02-21T23:02:43.419669+0000","last_clean":"2024-02-21T23:02:43.419669+0000","last_became_active":"2024-02-21T23:02:38.378513+0000","last_became_peered":"2024-02-21T23:02:38.378513+0000","last_unstale":"2024-02-21T23:02:43.419669+0000","last_undegraded":"2024-02-21T23:02:43.419669+0000","last_fullsized":"2024-02-21T23:02:43.419669+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:29:00.068426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.595013+0000","last_change":"2024-02-21T23:02:37.359130+0000","last_active":"2024-02-21T23:02:43.595013+0000","last_peered":"2024-02-21T23:02:43.595013+0000","last_clean":"2024-02-21T23:02:43.595013+0000","last_became_active":"2024-02-21T23:02:37.358812+0000","last_became_peered":"2024-02-21T23:02:37.358812+0000","last_unstale":"2024-02-21T23:02:43.595013+0000","last_undegraded":"2024-02-21T23:02:43.595013+0000","last_fullsized":"2024-02-21T23:02:43.595013+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:29:33.262208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.598892+0000","last_change":"2024-02-21T23:02:42.500216+0000","last_active":"2024-02-21T23:02:43.598892+0000","last_peered":"2024-02-21T23:02:43.598892+0000","last_clean":"2024-02-21T23:02:43.598892+0000","last_became_active":"2024-02-21T23:02:42.499762+0000","last_became_peered":"2024-02-21T23:02:42.499762+0000","last_unstale":"2024-02-21T23:02:43.598892+0000","last_undegraded":"2024-02-21T23:02:43.598892+0000","last_fullsized":"2024-02-21T23:02:43.598892+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:12:30.131037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420590+0000","last_change":"2024-02-21T23:02:38.381297+0000","last_active":"2024-02-21T23:02:43.420590+0000","last_peered":"2024-02-21T23:02:43.420590+0000","last_clean":"2024-02-21T23:02:43.420590+0000","last_became_active":"2024-02-21T23:02:38.381015+0000","last_became_peered":"2024-02-21T23:02:38.381015+0000","last_unstale":"2024-02-21T23:02:43.420590+0000","last_undegraded":"2024-02-21T23:02:43.420590+0000","last_fullsized":"2024-02-21T23:02:43.420590+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:29:34.276860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.596339+0000","last_change":"2024-02-21T23:02:37.370201+0000","last_active":"2024-02-21T23:02:43.596339+0000","last_peered":"2024-02-21T23:02:43.596339+0000","last_clean":"2024-02-21T23:02:43.596339+0000","last_became_active":"2024-02-21T23:02:37.369814+0000","last_became_peered":"2024-02-21T23:02:37.369814+0000","last_unstale":"2024-02-21T23:02:43.596339+0000","last_undegraded":"2024-02-21T23:02:43.596339+0000","last_fullsized":"2024-02-21T23:02:43.596339+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:16:38.578056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.421168+0000","last_change":"2024-02-21T23:02:42.439870+0000","last_active":"2024-02-21T23:02:43.421168+0000","last_peered":"2024-02-21T23:02:43.421168+0000","last_clean":"2024-02-21T23:02:43.421168+0000","last_became_active":"2024-02-21T23:02:42.439470+0000","last_became_peered":"2024-02-21T23:02:42.439470+0000","last_unstale":"2024-02-21T23:02:43.421168+0000","last_undegraded":"2024-02-21T23:02:43.421168+0000","last_fullsized":"2024-02-21T23:02:43.421168+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:54:20.718854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.597460+0000","last_change":"2024-02-21T23:02:38.380559+0000","last_active":"2024-02-21T23:02:43.597460+0000","last_peered":"2024-02-21T23:02:43.597460+0000","last_clean":"2024-02-21T23:02:43.597460+0000","last_became_active":"2024-02-21T23:02:38.380229+0000","last_became_peered":"2024-02-21T23:02:38.380229+0000","last_unstale":"2024-02-21T23:02:43.597460+0000","last_undegraded":"2024-02-21T23:02:43.597460+0000","last_fullsized":"2024-02-21T23:02:43.597460+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:09:34.130536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.595919+0000","last_change":"2024-02-21T23:02:37.367812+0000","last_active":"2024-02-21T23:02:43.595919+0000","last_peered":"2024-02-21T23:02:43.595919+0000","last_clean":"2024-02-21T23:02:43.595919+0000","last_became_active":"2024-02-21T23:02:37.367527+0000","last_became_peered":"2024-02-21T23:02:37.367527+0000","last_unstale":"2024-02-21T23:02:43.595919+0000","last_undegraded":"2024-02-21T23:02:43.595919+0000","last_fullsized":"2024-02-21T23:02:43.595919+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:17:46.451150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420213+0000","last_change":"2024-02-21T23:02:42.497817+0000","last_active":"2024-02-21T23:02:43.420213+0000","last_peered":"2024-02-21T23:02:43.420213+0000","last_clean":"2024-02-21T23:02:43.420213+0000","last_became_active":"2024-02-21T23:02:42.497417+0000","last_became_peered":"2024-02-21T23:02:42.497417+0000","last_unstale":"2024-02-21T23:02:43.420213+0000","last_undegraded":"2024-02-21T23:02:43.420213+0000","last_fullsized":"2024-02-21T23:02:43.420213+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:00:13.239804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420747+0000","last_change":"2024-02-21T23:02:38.389615+0000","last_active":"2024-02-21T23:02:43.420747+0000","last_peered":"2024-02-21T23:02:43.420747+0000","last_clean":"2024-02-21T23:02:43.420747+0000","last_became_active":"2024-02-21T23:02:38.389268+0000","last_became_peered":"2024-02-21T23:02:38.389268+0000","last_unstale":"2024-02-21T23:02:43.420747+0000","last_undegraded":"2024-02-21T23:02:43.420747+0000","last_fullsized":"2024-02-21T23:02:43.420747+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:20:23.836303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.598294+0000","last_change":"2024-02-21T23:02:37.356409+0000","last_active":"2024-02-21T23:02:43.598294+0000","last_peered":"2024-02-21T23:02:43.598294+0000","last_clean":"2024-02-21T23:02:43.598294+0000","last_became_active":"2024-02-21T23:02:37.356096+0000","last_became_peered":"2024-02-21T23:02:37.356096+0000","last_unstale":"2024-02-21T23:02:43.598294+0000","last_undegraded":"2024-02-21T23:02:43.598294+0000","last_fullsized":"2024-02-21T23:02:43.598294+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:07:23.801765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.418871+0000","last_change":"2024-02-21T23:02:42.495050+0000","last_active":"2024-02-21T23:02:43.418871+0000","last_peered":"2024-02-21T23:02:43.418871+0000","last_clean":"2024-02-21T23:02:43.418871+0000","last_became_active":"2024-02-21T23:02:42.494469+0000","last_became_peered":"2024-02-21T23:02:42.494469+0000","last_unstale":"2024-02-21T23:02:43.418871+0000","last_undegraded":"2024-02-21T23:02:43.418871+0000","last_fullsized":"2024-02-21T23:02:43.418871+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:00:12.286244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.596815+0000","last_change":"2024-02-21T23:02:38.378708+0000","last_active":"2024-02-21T23:02:43.596815+0000","last_peered":"2024-02-21T23:02:43.596815+0000","last_clean":"2024-02-21T23:02:43.596815+0000","last_became_active":"2024-02-21T23:02:38.378374+0000","last_became_peered":"2024-02-21T23:02:38.378374+0000","last_unstale":"2024-02-21T23:02:43.596815+0000","last_undegraded":"2024-02-21T23:02:43.596815+0000","last_fullsized":"2024-02-21T23:02:43.596815+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:19:25.522975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.598361+0000","last_change":"2024-02-21T23:02:37.372175+0000","last_active":"2024-02-21T23:02:43.598361+0000","last_peered":"2024-02-21T23:02:43.598361+0000","last_clean":"2024-02-21T23:02:43.598361+0000","last_became_active":"2024-02-21T23:02:37.371900+0000","last_became_peered":"2024-02-21T23:02:37.371900+0000","last_unstale":"2024-02-21T23:02:43.598361+0000","last_undegraded":"2024-02-21T23:02:43.598361+0000","last_fullsized":"2024-02-21T23:02:43.598361+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:12:51.484005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.596406+0000","last_change":"2024-02-21T23:02:42.500502+0000","last_active":"2024-02-21T23:02:43.596406+0000","last_peered":"2024-02-21T23:02:43.596406+0000","last_clean":"2024-02-21T23:02:43.596406+0000","last_became_active":"2024-02-21T23:02:42.497658+0000","last_became_peered":"2024-02-21T23:02:42.497658+0000","last_unstale":"2024-02-21T23:02:43.596406+0000","last_undegraded":"2024-02-21T23:02:43.596406+0000","last_fullsized":"2024-02-21T23:02:43.596406+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:54:20.718854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.601212+0000","last_change":"2024-02-21T23:02:38.399632+0000","last_active":"2024-02-21T23:02:43.601212+0000","last_peered":"2024-02-21T23:02:43.601212+0000","last_clean":"2024-02-21T23:02:43.601212+0000","last_became_active":"2024-02-21T23:02:38.392339+0000","last_became_peered":"2024-02-21T23:02:38.392339+0000","last_unstale":"2024-02-21T23:02:43.601212+0000","last_undegraded":"2024-02-21T23:02:43.601212+0000","last_fullsized":"2024-02-21T23:02:43.601212+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:35:27.966492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.596919+0000","last_change":"2024-02-21T23:02:37.347461+0000","last_active":"2024-02-21T23:02:43.596919+0000","last_peered":"2024-02-21T23:02:43.596919+0000","last_clean":"2024-02-21T23:02:43.596919+0000","last_became_active":"2024-02-21T23:02:37.346584+0000","last_became_peered":"2024-02-21T23:02:37.346584+0000","last_unstale":"2024-02-21T23:02:43.596919+0000","last_undegraded":"2024-02-21T23:02:43.596919+0000","last_fullsized":"2024-02-21T23:02:43.596919+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:32:39.769989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.595619+0000","last_change":"2024-02-21T23:02:42.436480+0000","last_active":"2024-02-21T23:02:43.595619+0000","last_peered":"2024-02-21T23:02:43.595619+0000","last_clean":"2024-02-21T23:02:43.595619+0000","last_became_active":"2024-02-21T23:02:42.436035+0000","last_became_peered":"2024-02-21T23:02:42.436035+0000","last_unstale":"2024-02-21T23:02:43.595619+0000","last_undegraded":"2024-02-21T23:02:43.595619+0000","last_fullsized":"2024-02-21T23:02:43.595619+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:54:20.718854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420857+0000","last_change":"2024-02-21T23:02:38.373673+0000","last_active":"2024-02-21T23:02:43.420857+0000","last_peered":"2024-02-21T23:02:43.420857+0000","last_clean":"2024-02-21T23:02:43.420857+0000","last_became_active":"2024-02-21T23:02:38.373324+0000","last_became_peered":"2024-02-21T23:02:38.373324+0000","last_unstale":"2024-02-21T23:02:43.420857+0000","last_undegraded":"2024-02-21T23:02:43.420857+0000","last_fullsized":"2024-02-21T23:02:43.420857+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:09:48.394584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.600679+0000","last_change":"2024-02-21T23:02:37.346663+0000","last_active":"2024-02-21T23:02:43.600679+0000","last_peered":"2024-02-21T23:02:43.600679+0000","last_clean":"2024-02-21T23:02:43.600679+0000","last_became_active":"2024-02-21T23:02:37.344771+0000","last_became_peered":"2024-02-21T23:02:37.344771+0000","last_unstale":"2024-02-21T23:02:43.600679+0000","last_undegraded":"2024-02-21T23:02:43.600679+0000","last_fullsized":"2024-02-21T23:02:43.600679+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:12:51.484005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.596759+0000","last_change":"2024-02-21T23:02:42.440882+0000","last_active":"2024-02-21T23:02:43.596759+0000","last_peered":"2024-02-21T23:02:43.596759+0000","last_clean":"2024-02-21T23:02:43.596759+0000","last_became_active":"2024-02-21T23:02:42.440534+0000","last_became_peered":"2024-02-21T23:02:42.440534+0000","last_unstale":"2024-02-21T23:02:43.596759+0000","last_undegraded":"2024-02-21T23:02:43.596759+0000","last_fullsized":"2024-02-21T23:02:43.596759+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:35:32.034237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.595207+0000","last_change":"2024-02-21T23:02:38.377238+0000","last_active":"2024-02-21T23:02:43.595207+0000","last_peered":"2024-02-21T23:02:43.595207+0000","last_clean":"2024-02-21T23:02:43.595207+0000","last_became_active":"2024-02-21T23:02:38.376272+0000","last_became_peered":"2024-02-21T23:02:38.376272+0000","last_unstale":"2024-02-21T23:02:43.595207+0000","last_undegraded":"2024-02-21T23:02:43.595207+0000","last_fullsized":"2024-02-21T23:02:43.595207+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:10:48.186239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.415223+0000","last_change":"2024-02-21T23:02:37.360945+0000","last_active":"2024-02-21T23:02:43.415223+0000","last_peered":"2024-02-21T23:02:43.415223+0000","last_clean":"2024-02-21T23:02:43.415223+0000","last_became_active":"2024-02-21T23:02:37.359879+0000","last_became_peered":"2024-02-21T23:02:37.359879+0000","last_unstale":"2024-02-21T23:02:43.415223+0000","last_undegraded":"2024-02-21T23:02:43.415223+0000","last_fullsized":"2024-02-21T23:02:43.415223+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:07:23.801765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.598887+0000","last_change":"2024-02-21T23:02:42.441857+0000","last_active":"2024-02-21T23:02:43.598887+0000","last_peered":"2024-02-21T23:02:43.598887+0000","last_clean":"2024-02-21T23:02:43.598887+0000","last_became_active":"2024-02-21T23:02:42.438695+0000","last_became_peered":"2024-02-21T23:02:42.438695+0000","last_unstale":"2024-02-21T23:02:43.598887+0000","last_undegraded":"2024-02-21T23:02:43.598887+0000","last_fullsized":"2024-02-21T23:02:43.598887+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:28:27.820393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.594261+0000","last_change":"2024-02-21T23:02:38.366727+0000","last_active":"2024-02-21T23:02:43.594261+0000","last_peered":"2024-02-21T23:02:43.594261+0000","last_clean":"2024-02-21T23:02:43.594261+0000","last_became_active":"2024-02-21T23:02:38.366344+0000","last_became_peered":"2024-02-21T23:02:38.366344+0000","last_unstale":"2024-02-21T23:02:43.594261+0000","last_undegraded":"2024-02-21T23:02:43.594261+0000","last_fullsized":"2024-02-21T23:02:43.594261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:08:44.843392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.596392+0000","last_change":"2024-02-21T23:02:37.372506+0000","last_active":"2024-02-21T23:02:43.596392+0000","last_peered":"2024-02-21T23:02:43.596392+0000","last_clean":"2024-02-21T23:02:43.596392+0000","last_became_active":"2024-02-21T23:02:37.371806+0000","last_became_peered":"2024-02-21T23:02:37.371806+0000","last_unstale":"2024-02-21T23:02:43.596392+0000","last_undegraded":"2024-02-21T23:02:43.596392+0000","last_fullsized":"2024-02-21T23:02:43.596392+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:09:47.379932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.597397+0000","last_change":"2024-02-21T23:02:42.500339+0000","last_active":"2024-02-21T23:02:43.597397+0000","last_peered":"2024-02-21T23:02:43.597397+0000","last_clean":"2024-02-21T23:02:43.597397+0000","last_became_active":"2024-02-21T23:02:42.498471+0000","last_became_peered":"2024-02-21T23:02:42.498471+0000","last_unstale":"2024-02-21T23:02:43.597397+0000","last_undegraded":"2024-02-21T23:02:43.597397+0000","last_fullsized":"2024-02-21T23:02:43.597397+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:12:30.131037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.419210+0000","last_change":"2024-02-21T23:02:38.400850+0000","last_active":"2024-02-21T23:02:43.419210+0000","last_peered":"2024-02-21T23:02:43.419210+0000","last_clean":"2024-02-21T23:02:43.419210+0000","last_became_active":"2024-02-21T23:02:38.400367+0000","last_became_peered":"2024-02-21T23:02:38.400367+0000","last_unstale":"2024-02-21T23:02:43.419210+0000","last_undegraded":"2024-02-21T23:02:43.419210+0000","last_fullsized":"2024-02-21T23:02:43.419210+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:29:12.483912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.600646+0000","last_change":"2024-02-21T23:02:37.349152+0000","last_active":"2024-02-21T23:02:43.600646+0000","last_peered":"2024-02-21T23:02:43.600646+0000","last_clean":"2024-02-21T23:02:43.600646+0000","last_became_active":"2024-02-21T23:02:37.348647+0000","last_became_peered":"2024-02-21T23:02:37.348647+0000","last_unstale":"2024-02-21T23:02:43.600646+0000","last_undegraded":"2024-02-21T23:02:43.600646+0000","last_fullsized":"2024-02-21T23:02:43.600646+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:17:46.451150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":178,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.596598+0000","last_change":"2024-02-21T23:02:32.289335+0000","last_active":"2024-02-21T23:02:43.596598+0000","last_peered":"2024-02-21T23:02:43.596598+0000","last_clean":"2024-02-21T23:02:43.596598+0000","last_became_active":"2024-02-21T23:02:32.286913+0000","last_became_peered":"2024-02-21T23:02:32.286913+0000","last_unstale":"2024-02-21T23:02:43.596598+0000","last_undegraded":"2024-02-21T23:02:43.596598+0000","last_fullsized":"2024-02-21T23:02:43.596598+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_clean_scrub_stamp":"2024-02-21T23:02:31.172569+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:36:23.987374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.594895+0000","last_change":"2024-02-21T23:02:42.498307+0000","last_active":"2024-02-21T23:02:43.594895+0000","last_peered":"2024-02-21T23:02:43.594895+0000","last_clean":"2024-02-21T23:02:43.594895+0000","last_became_active":"2024-02-21T23:02:42.497846+0000","last_became_peered":"2024-02-21T23:02:42.497846+0000","last_unstale":"2024-02-21T23:02:43.594895+0000","last_undegraded":"2024-02-21T23:02:43.594895+0000","last_fullsized":"2024-02-21T23:02:43.594895+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:24:44.830150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.597925+0000","last_change":"2024-02-21T23:02:38.401025+0000","last_active":"2024-02-21T23:02:43.597925+0000","last_peered":"2024-02-21T23:02:43.597925+0000","last_clean":"2024-02-21T23:02:43.597925+0000","last_became_active":"2024-02-21T23:02:38.400639+0000","last_became_peered":"2024-02-21T23:02:38.400639+0000","last_unstale":"2024-02-21T23:02:43.597925+0000","last_undegraded":"2024-02-21T23:02:43.597925+0000","last_fullsized":"2024-02-21T23:02:43.597925+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:49:20.517131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.416472+0000","last_change":"2024-02-21T23:02:37.360670+0000","last_active":"2024-02-21T23:02:43.416472+0000","last_peered":"2024-02-21T23:02:43.416472+0000","last_clean":"2024-02-21T23:02:43.416472+0000","last_became_active":"2024-02-21T23:02:37.358188+0000","last_became_peered":"2024-02-21T23:02:37.358188+0000","last_unstale":"2024-02-21T23:02:43.416472+0000","last_undegraded":"2024-02-21T23:02:43.416472+0000","last_fullsized":"2024-02-21T23:02:43.416472+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:17:46.451150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.595571+0000","last_change":"2024-02-21T23:02:42.495923+0000","last_active":"2024-02-21T23:02:43.595571+0000","last_peered":"2024-02-21T23:02:43.595571+0000","last_clean":"2024-02-21T23:02:43.595571+0000","last_became_active":"2024-02-21T23:02:42.495507+0000","last_became_peered":"2024-02-21T23:02:42.495507+0000","last_unstale":"2024-02-21T23:02:43.595571+0000","last_undegraded":"2024-02-21T23:02:43.595571+0000","last_fullsized":"2024-02-21T23:02:43.595571+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:54:20.718854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.600858+0000","last_change":"2024-02-21T23:02:38.386757+0000","last_active":"2024-02-21T23:02:43.600858+0000","last_peered":"2024-02-21T23:02:43.600858+0000","last_clean":"2024-02-21T23:02:43.600858+0000","last_became_active":"2024-02-21T23:02:38.384691+0000","last_became_peered":"2024-02-21T23:02:38.384691+0000","last_unstale":"2024-02-21T23:02:43.600858+0000","last_undegraded":"2024-02-21T23:02:43.600858+0000","last_fullsized":"2024-02-21T23:02:43.600858+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:49:20.517131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.597791+0000","last_change":"2024-02-21T23:02:37.356479+0000","last_active":"2024-02-21T23:02:43.597791+0000","last_peered":"2024-02-21T23:02:43.597791+0000","last_clean":"2024-02-21T23:02:43.597791+0000","last_became_active":"2024-02-21T23:02:37.356132+0000","last_became_peered":"2024-02-21T23:02:37.356132+0000","last_unstale":"2024-02-21T23:02:43.597791+0000","last_undegraded":"2024-02-21T23:02:43.597791+0000","last_fullsized":"2024-02-21T23:02:43.597791+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:16:38.578056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.597135+0000","last_change":"2024-02-21T23:02:42.440448+0000","last_active":"2024-02-21T23:02:43.597135+0000","last_peered":"2024-02-21T23:02:43.597135+0000","last_clean":"2024-02-21T23:02:43.597135+0000","last_became_active":"2024-02-21T23:02:42.439847+0000","last_became_peered":"2024-02-21T23:02:42.439847+0000","last_unstale":"2024-02-21T23:02:43.597135+0000","last_undegraded":"2024-02-21T23:02:43.597135+0000","last_fullsized":"2024-02-21T23:02:43.597135+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:28:27.820393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.598913+0000","last_change":"2024-02-21T23:02:38.401714+0000","last_active":"2024-02-21T23:02:43.598913+0000","last_peered":"2024-02-21T23:02:43.598913+0000","last_clean":"2024-02-21T23:02:43.598913+0000","last_became_active":"2024-02-21T23:02:38.401166+0000","last_became_peered":"2024-02-21T23:02:38.401166+0000","last_unstale":"2024-02-21T23:02:43.598913+0000","last_undegraded":"2024-02-21T23:02:43.598913+0000","last_fullsized":"2024-02-21T23:02:43.598913+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:33:11.255331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.416538+0000","last_change":"2024-02-21T23:02:37.356667+0000","last_active":"2024-02-21T23:02:43.416538+0000","last_peered":"2024-02-21T23:02:43.416538+0000","last_clean":"2024-02-21T23:02:43.416538+0000","last_became_active":"2024-02-21T23:02:37.354832+0000","last_became_peered":"2024-02-21T23:02:37.354832+0000","last_unstale":"2024-02-21T23:02:43.416538+0000","last_undegraded":"2024-02-21T23:02:43.416538+0000","last_fullsized":"2024-02-21T23:02:43.416538+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:01:17.207079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.599759+0000","last_change":"2024-02-21T23:02:42.494816+0000","last_active":"2024-02-21T23:02:43.599759+0000","last_peered":"2024-02-21T23:02:43.599759+0000","last_clean":"2024-02-21T23:02:43.599759+0000","last_became_active":"2024-02-21T23:02:42.494397+0000","last_became_peered":"2024-02-21T23:02:42.494397+0000","last_unstale":"2024-02-21T23:02:43.599759+0000","last_undegraded":"2024-02-21T23:02:43.599759+0000","last_fullsized":"2024-02-21T23:02:43.599759+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:35:47.803796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.597807+0000","last_change":"2024-02-21T23:02:38.402995+0000","last_active":"2024-02-21T23:02:43.597807+0000","last_peered":"2024-02-21T23:02:43.597807+0000","last_clean":"2024-02-21T23:02:43.597807+0000","last_became_active":"2024-02-21T23:02:38.400263+0000","last_became_peered":"2024-02-21T23:02:38.400263+0000","last_unstale":"2024-02-21T23:02:43.597807+0000","last_undegraded":"2024-02-21T23:02:43.597807+0000","last_fullsized":"2024-02-21T23:02:43.597807+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:28:23.752647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.421321+0000","last_change":"2024-02-21T23:02:37.348962+0000","last_active":"2024-02-21T23:02:43.421321+0000","last_peered":"2024-02-21T23:02:43.421321+0000","last_clean":"2024-02-21T23:02:43.421321+0000","last_became_active":"2024-02-21T23:02:37.348550+0000","last_became_peered":"2024-02-21T23:02:37.348550+0000","last_unstale":"2024-02-21T23:02:43.421321+0000","last_undegraded":"2024-02-21T23:02:43.421321+0000","last_fullsized":"2024-02-21T23:02:43.421321+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:20:15.139033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.419867+0000","last_change":"2024-02-21T23:02:42.442477+0000","last_active":"2024-02-21T23:02:43.419867+0000","last_peered":"2024-02-21T23:02:43.419867+0000","last_clean":"2024-02-21T23:02:43.419867+0000","last_became_active":"2024-02-21T23:02:42.442091+0000","last_became_peered":"2024-02-21T23:02:42.442091+0000","last_unstale":"2024-02-21T23:02:43.419867+0000","last_undegraded":"2024-02-21T23:02:43.419867+0000","last_fullsized":"2024-02-21T23:02:43.419867+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:09:38.198282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.596717+0000","last_change":"2024-02-21T23:02:38.401181+0000","last_active":"2024-02-21T23:02:43.596717+0000","last_peered":"2024-02-21T23:02:43.596717+0000","last_clean":"2024-02-21T23:02:43.596717+0000","last_became_active":"2024-02-21T23:02:38.400463+0000","last_became_peered":"2024-02-21T23:02:38.400463+0000","last_unstale":"2024-02-21T23:02:43.596717+0000","last_undegraded":"2024-02-21T23:02:43.596717+0000","last_fullsized":"2024-02-21T23:02:43.596717+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:05:40.265121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.414478+0000","last_change":"2024-02-21T23:02:37.348047+0000","last_active":"2024-02-21T23:02:43.414478+0000","last_peered":"2024-02-21T23:02:43.414478+0000","last_clean":"2024-02-21T23:02:43.414478+0000","last_became_active":"2024-02-21T23:02:37.347664+0000","last_became_peered":"2024-02-21T23:02:37.347664+0000","last_unstale":"2024-02-21T23:02:43.414478+0000","last_undegraded":"2024-02-21T23:02:43.414478+0000","last_fullsized":"2024-02-21T23:02:43.414478+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:39:34.395881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.596934+0000","last_change":"2024-02-21T23:02:42.447743+0000","last_active":"2024-02-21T23:02:43.596934+0000","last_peered":"2024-02-21T23:02:43.596934+0000","last_clean":"2024-02-21T23:02:43.596934+0000","last_became_active":"2024-02-21T23:02:42.447181+0000","last_became_peered":"2024-02-21T23:02:42.447181+0000","last_unstale":"2024-02-21T23:02:43.596934+0000","last_undegraded":"2024-02-21T23:02:43.596934+0000","last_fullsized":"2024-02-21T23:02:43.596934+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:07:03.424869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.594280+0000","last_change":"2024-02-21T23:02:38.375362+0000","last_active":"2024-02-21T23:02:43.594280+0000","last_peered":"2024-02-21T23:02:43.594280+0000","last_clean":"2024-02-21T23:02:43.594280+0000","last_became_active":"2024-02-21T23:02:38.375081+0000","last_became_peered":"2024-02-21T23:02:38.375081+0000","last_unstale":"2024-02-21T23:02:43.594280+0000","last_undegraded":"2024-02-21T23:02:43.594280+0000","last_fullsized":"2024-02-21T23:02:43.594280+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:19:25.522975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.595646+0000","last_change":"2024-02-21T23:02:37.359690+0000","last_active":"2024-02-21T23:02:43.595646+0000","last_peered":"2024-02-21T23:02:43.595646+0000","last_clean":"2024-02-21T23:02:43.595646+0000","last_became_active":"2024-02-21T23:02:37.359396+0000","last_became_peered":"2024-02-21T23:02:37.359396+0000","last_unstale":"2024-02-21T23:02:43.595646+0000","last_undegraded":"2024-02-21T23:02:43.595646+0000","last_fullsized":"2024-02-21T23:02:43.595646+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:29:33.262208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420901+0000","last_change":"2024-02-21T23:02:42.499203+0000","last_active":"2024-02-21T23:02:43.420901+0000","last_peered":"2024-02-21T23:02:43.420901+0000","last_clean":"2024-02-21T23:02:43.420901+0000","last_became_active":"2024-02-21T23:02:42.498781+0000","last_became_peered":"2024-02-21T23:02:42.498781+0000","last_unstale":"2024-02-21T23:02:43.420901+0000","last_undegraded":"2024-02-21T23:02:43.420901+0000","last_fullsized":"2024-02-21T23:02:43.420901+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:08:32.584721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.598100+0000","last_change":"2024-02-21T23:02:38.402276+0000","last_active":"2024-02-21T23:02:43.598100+0000","last_peered":"2024-02-21T23:02:43.598100+0000","last_clean":"2024-02-21T23:02:43.598100+0000","last_became_active":"2024-02-21T23:02:38.401976+0000","last_became_peered":"2024-02-21T23:02:38.401976+0000","last_unstale":"2024-02-21T23:02:43.598100+0000","last_undegraded":"2024-02-21T23:02:43.598100+0000","last_fullsized":"2024-02-21T23:02:43.598100+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:12:26.063291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.600414+0000","last_change":"2024-02-21T23:02:37.359735+0000","last_active":"2024-02-21T23:02:43.600414+0000","last_peered":"2024-02-21T23:02:43.600414+0000","last_clean":"2024-02-21T23:02:43.600414+0000","last_became_active":"2024-02-21T23:02:37.359447+0000","last_became_peered":"2024-02-21T23:02:37.359447+0000","last_unstale":"2024-02-21T23:02:43.600414+0000","last_undegraded":"2024-02-21T23:02:43.600414+0000","last_fullsized":"2024-02-21T23:02:43.600414+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:29:26.321681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.421314+0000","last_change":"2024-02-21T23:02:42.494896+0000","last_active":"2024-02-21T23:02:43.421314+0000","last_peered":"2024-02-21T23:02:43.421314+0000","last_clean":"2024-02-21T23:02:43.421314+0000","last_became_active":"2024-02-21T23:02:42.494433+0000","last_became_peered":"2024-02-21T23:02:42.494433+0000","last_unstale":"2024-02-21T23:02:43.421314+0000","last_undegraded":"2024-02-21T23:02:43.421314+0000","last_fullsized":"2024-02-21T23:02:43.421314+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:53:14.069591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.595745+0000","last_change":"2024-02-21T23:02:38.389883+0000","last_active":"2024-02-21T23:02:43.595745+0000","last_peered":"2024-02-21T23:02:43.595745+0000","last_clean":"2024-02-21T23:02:43.595745+0000","last_became_active":"2024-02-21T23:02:38.389132+0000","last_became_peered":"2024-02-21T23:02:38.389132+0000","last_unstale":"2024-02-21T23:02:43.595745+0000","last_undegraded":"2024-02-21T23:02:43.595745+0000","last_fullsized":"2024-02-21T23:02:43.595745+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:49:26.724162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.597566+0000","last_change":"2024-02-21T23:02:37.354560+0000","last_active":"2024-02-21T23:02:43.597566+0000","last_peered":"2024-02-21T23:02:43.597566+0000","last_clean":"2024-02-21T23:02:43.597566+0000","last_became_active":"2024-02-21T23:02:37.354266+0000","last_became_peered":"2024-02-21T23:02:37.354266+0000","last_unstale":"2024-02-21T23:02:43.597566+0000","last_undegraded":"2024-02-21T23:02:43.597566+0000","last_fullsized":"2024-02-21T23:02:43.597566+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:36:43.400580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.601188+0000","last_change":"2024-02-21T23:02:42.496211+0000","last_active":"2024-02-21T23:02:43.601188+0000","last_peered":"2024-02-21T23:02:43.601188+0000","last_clean":"2024-02-21T23:02:43.601188+0000","last_became_active":"2024-02-21T23:02:42.495493+0000","last_became_peered":"2024-02-21T23:02:42.495493+0000","last_unstale":"2024-02-21T23:02:43.601188+0000","last_undegraded":"2024-02-21T23:02:43.601188+0000","last_fullsized":"2024-02-21T23:02:43.601188+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:00:12.286244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.597793+0000","last_change":"2024-02-21T23:02:38.395172+0000","last_active":"2024-02-21T23:02:43.597793+0000","last_peered":"2024-02-21T23:02:43.597793+0000","last_clean":"2024-02-21T23:02:43.597793+0000","last_became_active":"2024-02-21T23:02:38.394809+0000","last_became_peered":"2024-02-21T23:02:38.394809+0000","last_unstale":"2024-02-21T23:02:43.597793+0000","last_undegraded":"2024-02-21T23:02:43.597793+0000","last_fullsized":"2024-02-21T23:02:43.597793+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:20:23.836303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"24'10","reported_seq":32,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:57.662420+0000","last_change":"2024-02-21T23:02:37.362539+0000","last_active":"2024-02-21T23:02:57.662420+0000","last_peered":"2024-02-21T23:02:57.662420+0000","last_clean":"2024-02-21T23:02:57.662420+0000","last_became_active":"2024-02-21T23:02:37.362186+0000","last_became_peered":"2024-02-21T23:02:37.362186+0000","last_unstale":"2024-02-21T23:02:57.662420+0000","last_undegraded":"2024-02-21T23:02:57.662420+0000","last_fullsized":"2024-02-21T23:02:57.662420+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:32:39.769989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.597415+0000","last_change":"2024-02-21T23:02:42.433330+0000","last_active":"2024-02-21T23:02:43.597415+0000","last_peered":"2024-02-21T23:02:43.597415+0000","last_clean":"2024-02-21T23:02:43.597415+0000","last_became_active":"2024-02-21T23:02:42.432944+0000","last_became_peered":"2024-02-21T23:02:42.432944+0000","last_unstale":"2024-02-21T23:02:43.597415+0000","last_undegraded":"2024-02-21T23:02:43.597415+0000","last_fullsized":"2024-02-21T23:02:43.597415+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:07:03.424869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420973+0000","last_change":"2024-02-21T23:02:38.389764+0000","last_active":"2024-02-21T23:02:43.420973+0000","last_peered":"2024-02-21T23:02:43.420973+0000","last_clean":"2024-02-21T23:02:43.420973+0000","last_became_active":"2024-02-21T23:02:38.389176+0000","last_became_peered":"2024-02-21T23:02:38.389176+0000","last_unstale":"2024-02-21T23:02:43.420973+0000","last_undegraded":"2024-02-21T23:02:43.420973+0000","last_fullsized":"2024-02-21T23:02:43.420973+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:12:46.637812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.597677+0000","last_change":"2024-02-21T23:02:37.344126+0000","last_active":"2024-02-21T23:02:43.597677+0000","last_peered":"2024-02-21T23:02:43.597677+0000","last_clean":"2024-02-21T23:02:43.597677+0000","last_became_active":"2024-02-21T23:02:37.343769+0000","last_became_peered":"2024-02-21T23:02:37.343769+0000","last_unstale":"2024-02-21T23:02:43.597677+0000","last_undegraded":"2024-02-21T23:02:43.597677+0000","last_fullsized":"2024-02-21T23:02:43.597677+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:09:47.379932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420591+0000","last_change":"2024-02-21T23:02:42.440359+0000","last_active":"2024-02-21T23:02:43.420591+0000","last_peered":"2024-02-21T23:02:43.420591+0000","last_clean":"2024-02-21T23:02:43.420591+0000","last_became_active":"2024-02-21T23:02:42.439967+0000","last_became_peered":"2024-02-21T23:02:42.439967+0000","last_unstale":"2024-02-21T23:02:43.420591+0000","last_undegraded":"2024-02-21T23:02:43.420591+0000","last_fullsized":"2024-02-21T23:02:43.420591+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:20:27.904049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.597926+0000","last_change":"2024-02-21T23:02:38.393522+0000","last_active":"2024-02-21T23:02:43.597926+0000","last_peered":"2024-02-21T23:02:43.597926+0000","last_clean":"2024-02-21T23:02:43.597926+0000","last_became_active":"2024-02-21T23:02:38.392890+0000","last_became_peered":"2024-02-21T23:02:38.392890+0000","last_unstale":"2024-02-21T23:02:43.597926+0000","last_undegraded":"2024-02-21T23:02:43.597926+0000","last_fullsized":"2024-02-21T23:02:43.597926+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:30:37.791426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.601175+0000","last_change":"2024-02-21T23:02:37.361082+0000","last_active":"2024-02-21T23:02:43.601175+0000","last_peered":"2024-02-21T23:02:43.601175+0000","last_clean":"2024-02-21T23:02:43.601175+0000","last_became_active":"2024-02-21T23:02:37.360751+0000","last_became_peered":"2024-02-21T23:02:37.360751+0000","last_unstale":"2024-02-21T23:02:43.601175+0000","last_undegraded":"2024-02-21T23:02:43.601175+0000","last_fullsized":"2024-02-21T23:02:43.601175+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:09:47.379932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.419487+0000","last_change":"2024-02-21T23:02:42.440260+0000","last_active":"2024-02-21T23:02:43.419487+0000","last_peered":"2024-02-21T23:02:43.419487+0000","last_clean":"2024-02-21T23:02:43.419487+0000","last_became_active":"2024-02-21T23:02:42.439785+0000","last_became_peered":"2024-02-21T23:02:42.439785+0000","last_unstale":"2024-02-21T23:02:43.419487+0000","last_undegraded":"2024-02-21T23:02:43.419487+0000","last_fullsized":"2024-02-21T23:02:43.419487+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:42:58.884647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.600988+0000","last_change":"2024-02-21T23:02:38.400013+0000","last_active":"2024-02-21T23:02:43.600988+0000","last_peered":"2024-02-21T23:02:43.600988+0000","last_clean":"2024-02-21T23:02:43.600988+0000","last_became_active":"2024-02-21T23:02:38.399708+0000","last_became_peered":"2024-02-21T23:02:38.399708+0000","last_unstale":"2024-02-21T23:02:43.600988+0000","last_undegraded":"2024-02-21T23:02:43.600988+0000","last_fullsized":"2024-02-21T23:02:43.600988+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:06:59.357124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.595798+0000","last_change":"2024-02-21T23:02:37.366264+0000","last_active":"2024-02-21T23:02:43.595798+0000","last_peered":"2024-02-21T23:02:43.595798+0000","last_clean":"2024-02-21T23:02:43.595798+0000","last_became_active":"2024-02-21T23:02:37.365889+0000","last_became_peered":"2024-02-21T23:02:37.365889+0000","last_unstale":"2024-02-21T23:02:43.595798+0000","last_undegraded":"2024-02-21T23:02:43.595798+0000","last_fullsized":"2024-02-21T23:02:43.595798+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:07:23.801765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.419552+0000","last_change":"2024-02-21T23:02:42.495489+0000","last_active":"2024-02-21T23:02:43.419552+0000","last_peered":"2024-02-21T23:02:43.419552+0000","last_clean":"2024-02-21T23:02:43.419552+0000","last_became_active":"2024-02-21T23:02:42.495098+0000","last_became_peered":"2024-02-21T23:02:42.495098+0000","last_unstale":"2024-02-21T23:02:43.419552+0000","last_undegraded":"2024-02-21T23:02:43.419552+0000","last_fullsized":"2024-02-21T23:02:43.419552+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:10:21.610343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.596467+0000","last_change":"2024-02-21T23:02:42.500336+0000","last_active":"2024-02-21T23:02:43.596467+0000","last_peered":"2024-02-21T23:02:43.596467+0000","last_clean":"2024-02-21T23:02:43.596467+0000","last_became_active":"2024-02-21T23:02:42.498817+0000","last_became_peered":"2024-02-21T23:02:42.498817+0000","last_unstale":"2024-02-21T23:02:43.596467+0000","last_undegraded":"2024-02-21T23:02:43.596467+0000","last_fullsized":"2024-02-21T23:02:43.596467+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:35:32.034237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.594893+0000","last_change":"2024-02-21T23:02:38.378745+0000","last_active":"2024-02-21T23:02:43.594893+0000","last_peered":"2024-02-21T23:02:43.594893+0000","last_clean":"2024-02-21T23:02:43.594893+0000","last_became_active":"2024-02-21T23:02:38.378423+0000","last_became_peered":"2024-02-21T23:02:38.378423+0000","last_unstale":"2024-02-21T23:02:43.594893+0000","last_undegraded":"2024-02-21T23:02:43.594893+0000","last_fullsized":"2024-02-21T23:02:43.594893+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:30:37.791426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.600043+0000","last_change":"2024-02-21T23:02:37.352231+0000","last_active":"2024-02-21T23:02:43.600043+0000","last_peered":"2024-02-21T23:02:43.600043+0000","last_clean":"2024-02-21T23:02:43.600043+0000","last_became_active":"2024-02-21T23:02:37.351715+0000","last_became_peered":"2024-02-21T23:02:37.351715+0000","last_unstale":"2024-02-21T23:02:43.600043+0000","last_undegraded":"2024-02-21T23:02:43.600043+0000","last_fullsized":"2024-02-21T23:02:43.600043+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:07:23.801765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.595272+0000","last_change":"2024-02-21T23:02:42.449848+0000","last_active":"2024-02-21T23:02:43.595272+0000","last_peered":"2024-02-21T23:02:43.595272+0000","last_clean":"2024-02-21T23:02:43.595272+0000","last_became_active":"2024-02-21T23:02:42.449357+0000","last_became_peered":"2024-02-21T23:02:42.449357+0000","last_unstale":"2024-02-21T23:02:43.595272+0000","last_undegraded":"2024-02-21T23:02:43.595272+0000","last_fullsized":"2024-02-21T23:02:43.595272+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:49:24.584877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.419241+0000","last_change":"2024-02-21T23:02:38.368240+0000","last_active":"2024-02-21T23:02:43.419241+0000","last_peered":"2024-02-21T23:02:43.419241+0000","last_clean":"2024-02-21T23:02:43.419241+0000","last_became_active":"2024-02-21T23:02:38.367852+0000","last_became_peered":"2024-02-21T23:02:38.367852+0000","last_unstale":"2024-02-21T23:02:43.419241+0000","last_undegraded":"2024-02-21T23:02:43.419241+0000","last_fullsized":"2024-02-21T23:02:43.419241+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:30:37.791426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420051+0000","last_change":"2024-02-21T23:02:37.363129+0000","last_active":"2024-02-21T23:02:43.420051+0000","last_peered":"2024-02-21T23:02:43.420051+0000","last_clean":"2024-02-21T23:02:43.420051+0000","last_became_active":"2024-02-21T23:02:37.362829+0000","last_became_peered":"2024-02-21T23:02:37.362829+0000","last_unstale":"2024-02-21T23:02:43.420051+0000","last_undegraded":"2024-02-21T23:02:43.420051+0000","last_fullsized":"2024-02-21T23:02:43.420051+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:39:34.395881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.594502+0000","last_change":"2024-02-21T23:02:37.358568+0000","last_active":"2024-02-21T23:02:43.594502+0000","last_peered":"2024-02-21T23:02:43.594502+0000","last_clean":"2024-02-21T23:02:43.594502+0000","last_became_active":"2024-02-21T23:02:37.358238+0000","last_became_peered":"2024-02-21T23:02:37.358238+0000","last_unstale":"2024-02-21T23:02:43.594502+0000","last_undegraded":"2024-02-21T23:02:43.594502+0000","last_fullsized":"2024-02-21T23:02:43.594502+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:01:17.207079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.597289+0000","last_change":"2024-02-21T23:02:38.375982+0000","last_active":"2024-02-21T23:02:43.597289+0000","last_peered":"2024-02-21T23:02:43.597289+0000","last_clean":"2024-02-21T23:02:43.597289+0000","last_became_active":"2024-02-21T23:02:38.375504+0000","last_became_peered":"2024-02-21T23:02:38.375504+0000","last_unstale":"2024-02-21T23:02:43.597289+0000","last_undegraded":"2024-02-21T23:02:43.597289+0000","last_fullsized":"2024-02-21T23:02:43.597289+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:19:25.522975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.416324+0000","last_change":"2024-02-21T23:02:42.441484+0000","last_active":"2024-02-21T23:02:43.416324+0000","last_peered":"2024-02-21T23:02:43.416324+0000","last_clean":"2024-02-21T23:02:43.416324+0000","last_became_active":"2024-02-21T23:02:42.438094+0000","last_became_peered":"2024-02-21T23:02:42.438094+0000","last_unstale":"2024-02-21T23:02:43.416324+0000","last_undegraded":"2024-02-21T23:02:43.416324+0000","last_fullsized":"2024-02-21T23:02:43.416324+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:04:15.959906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420559+0000","last_change":"2024-02-21T23:02:37.365019+0000","last_active":"2024-02-21T23:02:43.420559+0000","last_peered":"2024-02-21T23:02:43.420559+0000","last_clean":"2024-02-21T23:02:43.420559+0000","last_became_active":"2024-02-21T23:02:37.364761+0000","last_became_peered":"2024-02-21T23:02:37.364761+0000","last_unstale":"2024-02-21T23:02:43.420559+0000","last_undegraded":"2024-02-21T23:02:43.420559+0000","last_fullsized":"2024-02-21T23:02:43.420559+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:01:17.207079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420052+0000","last_change":"2024-02-21T23:02:38.385160+0000","last_active":"2024-02-21T23:02:43.420052+0000","last_peered":"2024-02-21T23:02:43.420052+0000","last_clean":"2024-02-21T23:02:43.420052+0000","last_became_active":"2024-02-21T23:02:38.382830+0000","last_became_peered":"2024-02-21T23:02:38.382830+0000","last_unstale":"2024-02-21T23:02:43.420052+0000","last_undegraded":"2024-02-21T23:02:43.420052+0000","last_fullsized":"2024-02-21T23:02:43.420052+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:16:39.592708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.421894+0000","last_change":"2024-02-21T23:02:42.437091+0000","last_active":"2024-02-21T23:02:43.421894+0000","last_peered":"2024-02-21T23:02:43.421894+0000","last_clean":"2024-02-21T23:02:43.421894+0000","last_became_active":"2024-02-21T23:02:42.436674+0000","last_became_peered":"2024-02-21T23:02:42.436674+0000","last_unstale":"2024-02-21T23:02:43.421894+0000","last_undegraded":"2024-02-21T23:02:43.421894+0000","last_fullsized":"2024-02-21T23:02:43.421894+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:16:29.140022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.594049+0000","last_change":"2024-02-21T23:02:38.379817+0000","last_active":"2024-02-21T23:02:43.594049+0000","last_peered":"2024-02-21T23:02:43.594049+0000","last_clean":"2024-02-21T23:02:43.594049+0000","last_became_active":"2024-02-21T23:02:38.379367+0000","last_became_peered":"2024-02-21T23:02:38.379367+0000","last_unstale":"2024-02-21T23:02:43.594049+0000","last_undegraded":"2024-02-21T23:02:43.594049+0000","last_fullsized":"2024-02-21T23:02:43.594049+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:04:11.892160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.415084+0000","last_change":"2024-02-21T23:02:38.386264+0000","last_active":"2024-02-21T23:02:43.415084+0000","last_peered":"2024-02-21T23:02:43.415084+0000","last_clean":"2024-02-21T23:02:43.415084+0000","last_became_active":"2024-02-21T23:02:38.385976+0000","last_became_peered":"2024-02-21T23:02:38.385976+0000","last_unstale":"2024-02-21T23:02:43.415084+0000","last_undegraded":"2024-02-21T23:02:43.415084+0000","last_fullsized":"2024-02-21T23:02:43.415084+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:09:48.394584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.600386+0000","last_change":"2024-02-21T23:02:38.394227+0000","last_active":"2024-02-21T23:02:43.600386+0000","last_peered":"2024-02-21T23:02:43.600386+0000","last_clean":"2024-02-21T23:02:43.600386+0000","last_became_active":"2024-02-21T23:02:38.393965+0000","last_became_peered":"2024-02-21T23:02:38.393965+0000","last_unstale":"2024-02-21T23:02:43.600386+0000","last_undegraded":"2024-02-21T23:02:43.600386+0000","last_fullsized":"2024-02-21T23:02:43.600386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:20:23.836303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.596804+0000","last_change":"2024-02-21T23:02:38.365717+0000","last_active":"2024-02-21T23:02:43.596804+0000","last_peered":"2024-02-21T23:02:43.596804+0000","last_clean":"2024-02-21T23:02:43.596804+0000","last_became_active":"2024-02-21T23:02:38.365375+0000","last_became_peered":"2024-02-21T23:02:38.365375+0000","last_unstale":"2024-02-21T23:02:43.596804+0000","last_undegraded":"2024-02-21T23:02:43.596804+0000","last_fullsized":"2024-02-21T23:02:43.596804+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:29:00.068426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.421488+0000","last_change":"2024-02-21T23:02:38.376971+0000","last_active":"2024-02-21T23:02:43.421488+0000","last_peered":"2024-02-21T23:02:43.421488+0000","last_clean":"2024-02-21T23:02:43.421488+0000","last_became_active":"2024-02-21T23:02:38.376565+0000","last_became_peered":"2024-02-21T23:02:38.376565+0000","last_unstale":"2024-02-21T23:02:43.421488+0000","last_undegraded":"2024-02-21T23:02:43.421488+0000","last_fullsized":"2024-02-21T23:02:43.421488+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:28:43.191300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.597197+0000","last_change":"2024-02-21T23:02:38.379350+0000","last_active":"2024-02-21T23:02:43.597197+0000","last_peered":"2024-02-21T23:02:43.597197+0000","last_clean":"2024-02-21T23:02:43.597197+0000","last_became_active":"2024-02-21T23:02:38.379030+0000","last_became_peered":"2024-02-21T23:02:38.379030+0000","last_unstale":"2024-02-21T23:02:43.597197+0000","last_undegraded":"2024-02-21T23:02:43.597197+0000","last_fullsized":"2024-02-21T23:02:43.597197+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:09:34.130536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.595248+0000","last_change":"2024-02-21T23:02:38.403472+0000","last_active":"2024-02-21T23:02:43.595248+0000","last_peered":"2024-02-21T23:02:43.595248+0000","last_clean":"2024-02-21T23:02:43.595248+0000","last_became_active":"2024-02-21T23:02:38.403213+0000","last_became_peered":"2024-02-21T23:02:38.403213+0000","last_unstale":"2024-02-21T23:02:43.595248+0000","last_undegraded":"2024-02-21T23:02:43.595248+0000","last_fullsized":"2024-02-21T23:02:43.595248+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:29:12.483912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420337+0000","last_change":"2024-02-21T23:02:38.377455+0000","last_active":"2024-02-21T23:02:43.420337+0000","last_peered":"2024-02-21T23:02:43.420337+0000","last_clean":"2024-02-21T23:02:43.420337+0000","last_became_active":"2024-02-21T23:02:38.376781+0000","last_became_peered":"2024-02-21T23:02:38.376781+0000","last_unstale":"2024-02-21T23:02:43.420337+0000","last_undegraded":"2024-02-21T23:02:43.420337+0000","last_fullsized":"2024-02-21T23:02:43.420337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:24:40.762404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.599882+0000","last_change":"2024-02-21T23:02:38.386554+0000","last_active":"2024-02-21T23:02:43.599882+0000","last_peered":"2024-02-21T23:02:43.599882+0000","last_clean":"2024-02-21T23:02:43.599882+0000","last_became_active":"2024-02-21T23:02:38.384500+0000","last_became_peered":"2024-02-21T23:02:38.384500+0000","last_unstale":"2024-02-21T23:02:43.599882+0000","last_undegraded":"2024-02-21T23:02:43.599882+0000","last_fullsized":"2024-02-21T23:02:43.599882+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:09:34.130536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420337+0000","last_change":"2024-02-21T23:02:38.383893+0000","last_active":"2024-02-21T23:02:43.420337+0000","last_peered":"2024-02-21T23:02:43.420337+0000","last_clean":"2024-02-21T23:02:43.420337+0000","last_became_active":"2024-02-21T23:02:38.383569+0000","last_became_peered":"2024-02-21T23:02:38.383569+0000","last_unstale":"2024-02-21T23:02:43.420337+0000","last_undegraded":"2024-02-21T23:02:43.420337+0000","last_fullsized":"2024-02-21T23:02:43.420337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:19:25.522975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.600302+0000","last_change":"2024-02-21T23:02:38.395013+0000","last_active":"2024-02-21T23:02:43.600302+0000","last_peered":"2024-02-21T23:02:43.600302+0000","last_clean":"2024-02-21T23:02:43.600302+0000","last_became_active":"2024-02-21T23:02:38.394670+0000","last_became_peered":"2024-02-21T23:02:38.394670+0000","last_unstale":"2024-02-21T23:02:43.600302+0000","last_undegraded":"2024-02-21T23:02:43.600302+0000","last_fullsized":"2024-02-21T23:02:43.600302+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:54:40.962238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.596541+0000","last_change":"2024-02-21T23:02:38.385293+0000","last_active":"2024-02-21T23:02:43.596541+0000","last_peered":"2024-02-21T23:02:43.596541+0000","last_clean":"2024-02-21T23:02:43.596541+0000","last_became_active":"2024-02-21T23:02:38.384589+0000","last_became_peered":"2024-02-21T23:02:38.384589+0000","last_unstale":"2024-02-21T23:02:43.596541+0000","last_undegraded":"2024-02-21T23:02:43.596541+0000","last_fullsized":"2024-02-21T23:02:43.596541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:10:48.186239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.595433+0000","last_change":"2024-02-21T23:02:38.385769+0000","last_active":"2024-02-21T23:02:43.595433+0000","last_peered":"2024-02-21T23:02:43.595433+0000","last_clean":"2024-02-21T23:02:43.595433+0000","last_became_active":"2024-02-21T23:02:38.385488+0000","last_became_peered":"2024-02-21T23:02:38.385488+0000","last_unstale":"2024-02-21T23:02:43.595433+0000","last_undegraded":"2024-02-21T23:02:43.595433+0000","last_fullsized":"2024-02-21T23:02:43.595433+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:29:00.068426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.598237+0000","last_change":"2024-02-21T23:02:38.396764+0000","last_active":"2024-02-21T23:02:43.598237+0000","last_peered":"2024-02-21T23:02:43.598237+0000","last_clean":"2024-02-21T23:02:43.598237+0000","last_became_active":"2024-02-21T23:02:38.396438+0000","last_became_peered":"2024-02-21T23:02:38.396438+0000","last_unstale":"2024-02-21T23:02:43.598237+0000","last_undegraded":"2024-02-21T23:02:43.598237+0000","last_fullsized":"2024-02-21T23:02:43.598237+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:06:59.357124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.416220+0000","last_change":"2024-02-21T23:02:38.387876+0000","last_active":"2024-02-21T23:02:43.416220+0000","last_peered":"2024-02-21T23:02:43.416220+0000","last_clean":"2024-02-21T23:02:43.416220+0000","last_became_active":"2024-02-21T23:02:38.383130+0000","last_became_peered":"2024-02-21T23:02:38.383130+0000","last_unstale":"2024-02-21T23:02:43.416220+0000","last_undegraded":"2024-02-21T23:02:43.416220+0000","last_fullsized":"2024-02-21T23:02:43.416220+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:12:46.637812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.601961+0000","last_change":"2024-02-21T23:02:38.403615+0000","last_active":"2024-02-21T23:02:43.601961+0000","last_peered":"2024-02-21T23:02:43.601961+0000","last_clean":"2024-02-21T23:02:43.601961+0000","last_became_active":"2024-02-21T23:02:38.402773+0000","last_became_peered":"2024-02-21T23:02:38.402773+0000","last_unstale":"2024-02-21T23:02:43.601961+0000","last_undegraded":"2024-02-21T23:02:43.601961+0000","last_fullsized":"2024-02-21T23:02:43.601961+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:12:26.063291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.599735+0000","last_change":"2024-02-21T23:02:38.384479+0000","last_active":"2024-02-21T23:02:43.599735+0000","last_peered":"2024-02-21T23:02:43.599735+0000","last_clean":"2024-02-21T23:02:43.599735+0000","last_became_active":"2024-02-21T23:02:38.383995+0000","last_became_peered":"2024-02-21T23:02:38.383995+0000","last_unstale":"2024-02-21T23:02:43.599735+0000","last_undegraded":"2024-02-21T23:02:43.599735+0000","last_fullsized":"2024-02-21T23:02:43.599735+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:28:43.191300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.595040+0000","last_change":"2024-02-21T23:02:38.390823+0000","last_active":"2024-02-21T23:02:43.595040+0000","last_peered":"2024-02-21T23:02:43.595040+0000","last_clean":"2024-02-21T23:02:43.595040+0000","last_became_active":"2024-02-21T23:02:38.390390+0000","last_became_peered":"2024-02-21T23:02:38.390390+0000","last_unstale":"2024-02-21T23:02:43.595040+0000","last_undegraded":"2024-02-21T23:02:43.595040+0000","last_fullsized":"2024-02-21T23:02:43.595040+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:29:00.068426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.595697+0000","last_change":"2024-02-21T23:02:38.380151+0000","last_active":"2024-02-21T23:02:43.595697+0000","last_peered":"2024-02-21T23:02:43.595697+0000","last_clean":"2024-02-21T23:02:43.595697+0000","last_became_active":"2024-02-21T23:02:38.379792+0000","last_became_peered":"2024-02-21T23:02:38.379792+0000","last_unstale":"2024-02-21T23:02:43.595697+0000","last_undegraded":"2024-02-21T23:02:43.595697+0000","last_fullsized":"2024-02-21T23:02:43.595697+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:39:18.270811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420027+0000","last_change":"2024-02-21T23:02:38.379804+0000","last_active":"2024-02-21T23:02:43.420027+0000","last_peered":"2024-02-21T23:02:43.420027+0000","last_clean":"2024-02-21T23:02:43.420027+0000","last_became_active":"2024-02-21T23:02:38.379461+0000","last_became_peered":"2024-02-21T23:02:38.379461+0000","last_unstale":"2024-02-21T23:02:43.420027+0000","last_undegraded":"2024-02-21T23:02:43.420027+0000","last_fullsized":"2024-02-21T23:02:43.420027+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:09:34.130536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420766+0000","last_change":"2024-02-21T23:02:38.381806+0000","last_active":"2024-02-21T23:02:43.420766+0000","last_peered":"2024-02-21T23:02:43.420766+0000","last_clean":"2024-02-21T23:02:43.420766+0000","last_became_active":"2024-02-21T23:02:38.381475+0000","last_became_peered":"2024-02-21T23:02:38.381475+0000","last_unstale":"2024-02-21T23:02:43.420766+0000","last_undegraded":"2024-02-21T23:02:43.420766+0000","last_fullsized":"2024-02-21T23:02:43.420766+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:29:27.336333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.595120+0000","last_change":"2024-02-21T23:02:38.398122+0000","last_active":"2024-02-21T23:02:43.595120+0000","last_peered":"2024-02-21T23:02:43.595120+0000","last_clean":"2024-02-21T23:02:43.595120+0000","last_became_active":"2024-02-21T23:02:38.392050+0000","last_became_peered":"2024-02-21T23:02:38.392050+0000","last_unstale":"2024-02-21T23:02:43.595120+0000","last_undegraded":"2024-02-21T23:02:43.595120+0000","last_fullsized":"2024-02-21T23:02:43.595120+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:10:48.186239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.594981+0000","last_change":"2024-02-21T23:02:38.398482+0000","last_active":"2024-02-21T23:02:43.594981+0000","last_peered":"2024-02-21T23:02:43.594981+0000","last_clean":"2024-02-21T23:02:43.594981+0000","last_became_active":"2024-02-21T23:02:38.392232+0000","last_became_peered":"2024-02-21T23:02:38.392232+0000","last_unstale":"2024-02-21T23:02:43.594981+0000","last_undegraded":"2024-02-21T23:02:43.594981+0000","last_fullsized":"2024-02-21T23:02:43.594981+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:30:37.791426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.598787+0000","last_change":"2024-02-21T23:02:38.400825+0000","last_active":"2024-02-21T23:02:43.598787+0000","last_peered":"2024-02-21T23:02:43.598787+0000","last_clean":"2024-02-21T23:02:43.598787+0000","last_became_active":"2024-02-21T23:02:38.398964+0000","last_became_peered":"2024-02-21T23:02:38.398964+0000","last_unstale":"2024-02-21T23:02:43.598787+0000","last_undegraded":"2024-02-21T23:02:43.598787+0000","last_fullsized":"2024-02-21T23:02:43.598787+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:24:40.762404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.418962+0000","last_change":"2024-02-21T23:02:38.387131+0000","last_active":"2024-02-21T23:02:43.418962+0000","last_peered":"2024-02-21T23:02:43.418962+0000","last_clean":"2024-02-21T23:02:43.418962+0000","last_became_active":"2024-02-21T23:02:38.386835+0000","last_became_peered":"2024-02-21T23:02:38.386835+0000","last_unstale":"2024-02-21T23:02:43.418962+0000","last_undegraded":"2024-02-21T23:02:43.418962+0000","last_fullsized":"2024-02-21T23:02:43.418962+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:10:48.186239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.596722+0000","last_change":"2024-02-21T23:02:38.394292+0000","last_active":"2024-02-21T23:02:43.596722+0000","last_peered":"2024-02-21T23:02:43.596722+0000","last_clean":"2024-02-21T23:02:43.596722+0000","last_became_active":"2024-02-21T23:02:38.394006+0000","last_became_peered":"2024-02-21T23:02:38.394006+0000","last_unstale":"2024-02-21T23:02:43.596722+0000","last_undegraded":"2024-02-21T23:02:43.596722+0000","last_fullsized":"2024-02-21T23:02:43.596722+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:54:40.962238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.600192+0000","last_change":"2024-02-21T23:02:38.394994+0000","last_active":"2024-02-21T23:02:43.600192+0000","last_peered":"2024-02-21T23:02:43.600192+0000","last_clean":"2024-02-21T23:02:43.600192+0000","last_became_active":"2024-02-21T23:02:38.394693+0000","last_became_peered":"2024-02-21T23:02:38.394693+0000","last_unstale":"2024-02-21T23:02:43.600192+0000","last_undegraded":"2024-02-21T23:02:43.600192+0000","last_fullsized":"2024-02-21T23:02:43.600192+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:00:09.172059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.597595+0000","last_change":"2024-02-21T23:02:38.397147+0000","last_active":"2024-02-21T23:02:43.597595+0000","last_peered":"2024-02-21T23:02:43.597595+0000","last_clean":"2024-02-21T23:02:43.597595+0000","last_became_active":"2024-02-21T23:02:38.396787+0000","last_became_peered":"2024-02-21T23:02:38.396787+0000","last_unstale":"2024-02-21T23:02:43.597595+0000","last_undegraded":"2024-02-21T23:02:43.597595+0000","last_fullsized":"2024-02-21T23:02:43.597595+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:35:27.966492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.600709+0000","last_change":"2024-02-21T23:02:38.398132+0000","last_active":"2024-02-21T23:02:43.600709+0000","last_peered":"2024-02-21T23:02:43.600709+0000","last_clean":"2024-02-21T23:02:43.600709+0000","last_became_active":"2024-02-21T23:02:38.397847+0000","last_became_peered":"2024-02-21T23:02:38.397847+0000","last_unstale":"2024-02-21T23:02:43.600709+0000","last_undegraded":"2024-02-21T23:02:43.600709+0000","last_fullsized":"2024-02-21T23:02:43.600709+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:28:23.752647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.596548+0000","last_change":"2024-02-21T23:02:38.372038+0000","last_active":"2024-02-21T23:02:43.596548+0000","last_peered":"2024-02-21T23:02:43.596548+0000","last_clean":"2024-02-21T23:02:43.596548+0000","last_became_active":"2024-02-21T23:02:38.371731+0000","last_became_peered":"2024-02-21T23:02:38.371731+0000","last_unstale":"2024-02-21T23:02:43.596548+0000","last_undegraded":"2024-02-21T23:02:43.596548+0000","last_fullsized":"2024-02-21T23:02:43.596548+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:04:11.892160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.598184+0000","last_change":"2024-02-21T23:02:38.379292+0000","last_active":"2024-02-21T23:02:43.598184+0000","last_peered":"2024-02-21T23:02:43.598184+0000","last_clean":"2024-02-21T23:02:43.598184+0000","last_became_active":"2024-02-21T23:02:38.378980+0000","last_became_peered":"2024-02-21T23:02:38.378980+0000","last_unstale":"2024-02-21T23:02:43.598184+0000","last_undegraded":"2024-02-21T23:02:43.598184+0000","last_fullsized":"2024-02-21T23:02:43.598184+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:29:12.483912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.598604+0000","last_change":"2024-02-21T23:02:38.396837+0000","last_active":"2024-02-21T23:02:43.598604+0000","last_peered":"2024-02-21T23:02:43.598604+0000","last_clean":"2024-02-21T23:02:43.598604+0000","last_became_active":"2024-02-21T23:02:38.396502+0000","last_became_peered":"2024-02-21T23:02:38.396502+0000","last_unstale":"2024-02-21T23:02:43.598604+0000","last_undegraded":"2024-02-21T23:02:43.598604+0000","last_fullsized":"2024-02-21T23:02:43.598604+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:54:16.651108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27340,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":323584,"data_stored":691206,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":258048,"data_stored":100346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":266240,"data_stored":100382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574858,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":253952,"data_stored":100256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":262144,"data_stored":100416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31508,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340332,"statfs":{"total":96636764160,"available":96604499968,"internally_reserved":0,"allocated":331776,"data_stored":691262,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-21T23:03:08.235 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-21T23:03:08.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.510+0000 7f0a92800640 1 Processor -- start 2024-02-21T23:03:08.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.510+0000 7f0a92800640 1 -- start start 2024-02-21T23:03:08.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.511+0000 7f0a92800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0a8c0c11f0 0x7f0a8c153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:08.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.511+0000 7f0a92800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0a8c0c15d0 0x7f0a8c06dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:08.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.511+0000 7f0a92800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0a8c156350 0x7f0a8c156780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:08.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.511+0000 7f0a92800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a8c0c1a60 con 0x7f0a8c0c11f0 2024-02-21T23:03:08.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.511+0000 7f0a92800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a8c0a5ec0 con 0x7f0a8c0c15d0 2024-02-21T23:03:08.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.511+0000 7f0a92800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a8c152e70 con 0x7f0a8c156350 2024-02-21T23:03:08.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.512+0000 7f0a8b400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0a8c0c11f0 0x7f0a8c153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:08.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.512+0000 7f0a8b400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0a8c0c11f0 0x7f0a8c153950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45906/0 (socket says 172.21.15.28:45906) 2024-02-21T23:03:08.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.512+0000 7f0a8b400640 1 -- 172.21.15.28:0/1293427870 learned_addr learned my addr 172.21.15.28:0/1293427870 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:08.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.512+0000 7f0a8be00640 1 --2- 172.21.15.28:0/1293427870 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0a8c156350 0x7f0a8c156780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:08.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.512+0000 7f0a8aa00640 1 --2- 172.21.15.28:0/1293427870 >> v2:172.21.15.81:3300/0 conn(0x7f0a8c0c15d0 0x7f0a8c06dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:08.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.512+0000 7f0a8b400640 1 -- 172.21.15.28:0/1293427870 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0a8c156350 msgr2=0x7f0a8c156780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:08.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.512+0000 7f0a8b400640 1 --2- 172.21.15.28:0/1293427870 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0a8c156350 0x7f0a8c156780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:08.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.512+0000 7f0a8b400640 1 -- 172.21.15.28:0/1293427870 >> v2:172.21.15.81:3300/0 conn(0x7f0a8c0c15d0 msgr2=0x7f0a8c06dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:08.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.512+0000 7f0a8b400640 1 --2- 172.21.15.28:0/1293427870 >> v2:172.21.15.81:3300/0 conn(0x7f0a8c0c15d0 0x7f0a8c06dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:08.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.512+0000 7f0a8b400640 1 -- 172.21.15.28:0/1293427870 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a8c153150 con 0x7f0a8c0c11f0 2024-02-21T23:03:08.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.514+0000 7f0a8b400640 1 --2- 172.21.15.28:0/1293427870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a8c0c11f0 0x7f0a8c153950 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c0c24a0 tx=0x7f0a7c02b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=174830e7abe53499 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:08.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.514+0000 7f0a8a000640 1 -- 172.21.15.28:0/1293427870 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a7c038070 con 0x7f0a8c0c11f0 2024-02-21T23:03:08.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.514+0000 7f0a8a000640 1 -- 172.21.15.28:0/1293427870 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a7c031740 con 0x7f0a8c0c11f0 2024-02-21T23:03:08.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.514+0000 7f0a8a000640 1 -- 172.21.15.28:0/1293427870 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a7c02f050 con 0x7f0a8c0c11f0 2024-02-21T23:03:08.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.515+0000 7f0a92800640 1 -- 172.21.15.28:0/1293427870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a8c0c11f0 msgr2=0x7f0a8c153950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:08.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.515+0000 7f0a92800640 1 --2- 172.21.15.28:0/1293427870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a8c0c11f0 0x7f0a8c153950 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c0c24a0 tx=0x7f0a7c02b7e0 comp rx=0 tx=0).stop 2024-02-21T23:03:08.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.515+0000 7f0a92800640 1 -- 172.21.15.28:0/1293427870 shutdown_connections 2024-02-21T23:03:08.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.515+0000 7f0a92800640 1 --2- 172.21.15.28:0/1293427870 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0a8c156350 0x7f0a8c156780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:08.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.515+0000 7f0a92800640 1 --2- 172.21.15.28:0/1293427870 >> v2:172.21.15.81:3300/0 conn(0x7f0a8c0c15d0 0x7f0a8c06dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:08.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.515+0000 7f0a92800640 1 --2- 172.21.15.28:0/1293427870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a8c0c11f0 0x7f0a8c153950 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:08.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.515+0000 7f0a92800640 1 -- 172.21.15.28:0/1293427870 >> 172.21.15.28:0/1293427870 conn(0x7f0a8c0c67f0 msgr2=0x7f0a8c0c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:08.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.515+0000 7f0a92800640 1 -- 172.21.15.28:0/1293427870 shutdown_connections 2024-02-21T23:03:08.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.515+0000 7f0a92800640 1 -- 172.21.15.28:0/1293427870 wait complete. 2024-02-21T23:03:08.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.517+0000 7f0a92800640 1 Processor -- start 2024-02-21T23:03:08.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.517+0000 7f0a92800640 1 -- start start 2024-02-21T23:03:08.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.518+0000 7f0a92800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a8c0c11f0 0x7f0a8c146960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:08.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.518+0000 7f0a92800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0a8c0c15d0 0x7f0a8c0a63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:08.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.518+0000 7f0a92800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a8c156350 0x7f0a8c0a6910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:08.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.518+0000 7f0a92800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a8c0c4c00 con 0x7f0a8c0c11f0 2024-02-21T23:03:08.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.518+0000 7f0a92800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a8c14d1c0 con 0x7f0a8c156350 2024-02-21T23:03:08.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.518+0000 7f0a92800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a8c0a6110 con 0x7f0a8c0c15d0 2024-02-21T23:03:08.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.518+0000 7f0a8b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a8c0c11f0 0x7f0a8c146960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:08.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.518+0000 7f0a8b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a8c0c11f0 0x7f0a8c146960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45918/0 (socket says 172.21.15.28:45918) 2024-02-21T23:03:08.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.518+0000 7f0a8b400640 1 -- 172.21.15.28:0/1389288035 learned_addr learned my addr 172.21.15.28:0/1389288035 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:08.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.518+0000 7f0a8aa00640 1 --2- 172.21.15.28:0/1389288035 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0a8c0c15d0 0x7f0a8c0a63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:08.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.518+0000 7f0a8be00640 1 --2- 172.21.15.28:0/1389288035 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a8c156350 0x7f0a8c0a6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:08.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.519+0000 7f0a8b400640 1 -- 172.21.15.28:0/1389288035 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0a8c0c15d0 msgr2=0x7f0a8c0a63d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:08.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.519+0000 7f0a8b400640 1 --2- 172.21.15.28:0/1389288035 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0a8c0c15d0 0x7f0a8c0a63d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:08.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.519+0000 7f0a8b400640 1 -- 172.21.15.28:0/1389288035 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a8c156350 msgr2=0x7f0a8c0a6910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:08.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.519+0000 7f0a8b400640 1 --2- 172.21.15.28:0/1389288035 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a8c156350 0x7f0a8c0a6910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:08.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.519+0000 7f0a8b400640 1 -- 172.21.15.28:0/1389288035 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a8c1eacd0 con 0x7f0a8c0c11f0 2024-02-21T23:03:08.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.519+0000 7f0a8aa00640 1 --2- 172.21.15.28:0/1389288035 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0a8c0c15d0 0x7f0a8c0a63d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:03:08.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.519+0000 7f0a8b400640 1 --2- 172.21.15.28:0/1389288035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a8c0c11f0 0x7f0a8c146960 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f0a7c02b820 tx=0x7f0a7c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:08.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.519+0000 7f0a83400640 1 -- 172.21.15.28:0/1389288035 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a7c042070 con 0x7f0a8c0c11f0 2024-02-21T23:03:08.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.519+0000 7f0a83400640 1 -- 172.21.15.28:0/1389288035 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a7c031e80 con 0x7f0a8c0c11f0 2024-02-21T23:03:08.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.519+0000 7f0a83400640 1 -- 172.21.15.28:0/1389288035 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a7c038040 con 0x7f0a8c0c11f0 2024-02-21T23:03:08.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.519+0000 7f0a92800640 1 -- 172.21.15.28:0/1389288035 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a8c1ee3e0 con 0x7f0a8c0c11f0 2024-02-21T23:03:08.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.519+0000 7f0a92800640 1 -- 172.21.15.28:0/1389288035 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a8c1eb270 con 0x7f0a8c0c11f0 2024-02-21T23:03:08.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.520+0000 7f0a92800640 1 -- 172.21.15.28:0/1389288035 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a8c067f30 con 0x7f0a8c0c11f0 2024-02-21T23:03:08.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.521+0000 7f0a83400640 1 -- 172.21.15.28:0/1389288035 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0a7c045070 con 0x7f0a8c0c11f0 2024-02-21T23:03:08.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.522+0000 7f0a83400640 1 --2- 172.21.15.28:0/1389288035 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0a6c06b4d0 0x7f0a6c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:08.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.523+0000 7f0a83400640 1 -- 172.21.15.28:0/1389288035 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f0a7c0ac200 con 0x7f0a8c0c11f0 2024-02-21T23:03:08.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.523+0000 7f0a8aa00640 1 --2- 172.21.15.28:0/1389288035 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0a6c06b4d0 0x7f0a6c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:08.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.524+0000 7f0a8aa00640 1 --2- 172.21.15.28:0/1389288035 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0a6c06b4d0 0x7f0a6c06d990 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c0c6d20 tx=0x7f0a78008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:08.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.527+0000 7f0a83400640 1 -- 172.21.15.28:0/1389288035 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f0a7c04c030 con 0x7f0a8c0c11f0 2024-02-21T23:03:08.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.723+0000 7f0a92800640 1 -- 172.21.15.28:0/1389288035 --> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0a8c0c0960 con 0x7f0a6c06b4d0 2024-02-21T23:03:08.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.747+0000 7f0a83400640 1 -- 172.21.15.28:0/1389288035 <== mgr.4101 v2:172.21.15.81:6832/1848538577 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+332614 (secure 0 0 0) 0x7f0a8c0c0960 con 0x7f0a6c06b4d0 2024-02-21T23:03:08.740 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:03:08.742 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-21T23:03:08.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.753+0000 7f0a92800640 1 -- 172.21.15.28:0/1389288035 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0a6c06b4d0 msgr2=0x7f0a6c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:08.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.753+0000 7f0a92800640 1 --2- 172.21.15.28:0/1389288035 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0a6c06b4d0 0x7f0a6c06d990 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c0c6d20 tx=0x7f0a78008040 comp rx=0 tx=0).stop 2024-02-21T23:03:08.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.754+0000 7f0a92800640 1 -- 172.21.15.28:0/1389288035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a8c0c11f0 msgr2=0x7f0a8c146960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:08.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.754+0000 7f0a92800640 1 --2- 172.21.15.28:0/1389288035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a8c0c11f0 0x7f0a8c146960 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f0a7c02b820 tx=0x7f0a7c0046c0 comp rx=0 tx=0).stop 2024-02-21T23:03:08.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.754+0000 7f0a92800640 1 -- 172.21.15.28:0/1389288035 shutdown_connections 2024-02-21T23:03:08.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.754+0000 7f0a92800640 1 --2- 172.21.15.28:0/1389288035 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0a6c06b4d0 0x7f0a6c06d990 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:08.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.754+0000 7f0a92800640 1 --2- 172.21.15.28:0/1389288035 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0a8c0c15d0 0x7f0a8c0a63d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:08.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.754+0000 7f0a92800640 1 --2- 172.21.15.28:0/1389288035 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a8c156350 0x7f0a8c0a6910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:08.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.754+0000 7f0a92800640 1 --2- 172.21.15.28:0/1389288035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a8c0c11f0 0x7f0a8c146960 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:08.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.754+0000 7f0a92800640 1 -- 172.21.15.28:0/1389288035 >> 172.21.15.28:0/1389288035 conn(0x7f0a8c0c67f0 msgr2=0x7f0a8c1489f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:08.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.754+0000 7f0a92800640 1 -- 172.21.15.28:0/1389288035 shutdown_connections 2024-02-21T23:03:08.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:08.754+0000 7f0a92800640 1 -- 172.21.15.28:0/1389288035 wait complete. 2024-02-21T23:03:08.765 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2024-02-21T23:03:08.668201+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":208,"num_write_kb":2435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160,"ondisk_log_size":160,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":222496,"kb_used_data":2176,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754752224,"statfs":{"total":773094113280,"available":772866277376,"internally_reserved":0,"allocated":2228224,"data_stored":1992064,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":225496625},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005536"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":26,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.598140+0000","last_change":"2024-02-21T23:02:37.367904+0000","last_active":"2024-02-21T23:02:43.598140+0000","last_peered":"2024-02-21T23:02:43.598140+0000","last_clean":"2024-02-21T23:02:43.598140+0000","last_became_active":"2024-02-21T23:02:37.367537+0000","last_became_peered":"2024-02-21T23:02:37.367537+0000","last_unstale":"2024-02-21T23:02:43.598140+0000","last_undegraded":"2024-02-21T23:02:43.598140+0000","last_fullsized":"2024-02-21T23:02:43.598140+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:39:34.395881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420749+0000","last_change":"2024-02-21T23:02:38.385115+0000","last_active":"2024-02-21T23:02:43.420749+0000","last_peered":"2024-02-21T23:02:43.420749+0000","last_clean":"2024-02-21T23:02:43.420749+0000","last_became_active":"2024-02-21T23:02:38.384818+0000","last_became_peered":"2024-02-21T23:02:38.384818+0000","last_unstale":"2024-02-21T23:02:43.420749+0000","last_undegraded":"2024-02-21T23:02:43.420749+0000","last_fullsized":"2024-02-21T23:02:43.420749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:54:40.962238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.419345+0000","last_change":"2024-02-21T23:02:42.500553+0000","last_active":"2024-02-21T23:02:43.419345+0000","last_peered":"2024-02-21T23:02:43.419345+0000","last_clean":"2024-02-21T23:02:43.419345+0000","last_became_active":"2024-02-21T23:02:42.498884+0000","last_became_peered":"2024-02-21T23:02:42.498884+0000","last_unstale":"2024-02-21T23:02:43.419345+0000","last_undegraded":"2024-02-21T23:02:43.419345+0000","last_fullsized":"2024-02-21T23:02:43.419345+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:41:42.562121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.421106+0000","last_change":"2024-02-21T23:02:37.354176+0000","last_active":"2024-02-21T23:02:43.421106+0000","last_peered":"2024-02-21T23:02:43.421106+0000","last_clean":"2024-02-21T23:02:43.421106+0000","last_became_active":"2024-02-21T23:02:37.353425+0000","last_became_peered":"2024-02-21T23:02:37.353425+0000","last_unstale":"2024-02-21T23:02:43.421106+0000","last_undegraded":"2024-02-21T23:02:43.421106+0000","last_fullsized":"2024-02-21T23:02:43.421106+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:07:23.801765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.597346+0000","last_change":"2024-02-21T23:02:38.399361+0000","last_active":"2024-02-21T23:02:43.597346+0000","last_peered":"2024-02-21T23:02:43.597346+0000","last_clean":"2024-02-21T23:02:43.597346+0000","last_became_active":"2024-02-21T23:02:38.399026+0000","last_became_peered":"2024-02-21T23:02:38.399026+0000","last_unstale":"2024-02-21T23:02:43.597346+0000","last_undegraded":"2024-02-21T23:02:43.597346+0000","last_fullsized":"2024-02-21T23:02:43.597346+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:29:12.483912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.419463+0000","last_change":"2024-02-21T23:02:42.441232+0000","last_active":"2024-02-21T23:02:43.419463+0000","last_peered":"2024-02-21T23:02:43.419463+0000","last_clean":"2024-02-21T23:02:43.419463+0000","last_became_active":"2024-02-21T23:02:42.440354+0000","last_became_peered":"2024-02-21T23:02:42.440354+0000","last_unstale":"2024-02-21T23:02:43.419463+0000","last_undegraded":"2024-02-21T23:02:43.419463+0000","last_fullsized":"2024-02-21T23:02:43.419463+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:29:16.551658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.598167+0000","last_change":"2024-02-21T23:02:37.373520+0000","last_active":"2024-02-21T23:02:43.598167+0000","last_peered":"2024-02-21T23:02:43.598167+0000","last_clean":"2024-02-21T23:02:43.598167+0000","last_became_active":"2024-02-21T23:02:37.373127+0000","last_became_peered":"2024-02-21T23:02:37.373127+0000","last_unstale":"2024-02-21T23:02:43.598167+0000","last_undegraded":"2024-02-21T23:02:43.598167+0000","last_fullsized":"2024-02-21T23:02:43.598167+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:12:45.623160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.600822+0000","last_change":"2024-02-21T23:02:38.387666+0000","last_active":"2024-02-21T23:02:43.600822+0000","last_peered":"2024-02-21T23:02:43.600822+0000","last_clean":"2024-02-21T23:02:43.600822+0000","last_became_active":"2024-02-21T23:02:38.387151+0000","last_became_peered":"2024-02-21T23:02:38.387151+0000","last_unstale":"2024-02-21T23:02:43.600822+0000","last_undegraded":"2024-02-21T23:02:43.600822+0000","last_fullsized":"2024-02-21T23:02:43.600822+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:24:40.762404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420258+0000","last_change":"2024-02-21T23:02:42.444676+0000","last_active":"2024-02-21T23:02:43.420258+0000","last_peered":"2024-02-21T23:02:43.420258+0000","last_clean":"2024-02-21T23:02:43.420258+0000","last_became_active":"2024-02-21T23:02:42.444185+0000","last_became_peered":"2024-02-21T23:02:42.444185+0000","last_unstale":"2024-02-21T23:02:43.420258+0000","last_undegraded":"2024-02-21T23:02:43.420258+0000","last_fullsized":"2024-02-21T23:02:43.420258+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:54:45.029984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.597339+0000","last_change":"2024-02-21T23:02:37.372881+0000","last_active":"2024-02-21T23:02:43.597339+0000","last_peered":"2024-02-21T23:02:43.597339+0000","last_clean":"2024-02-21T23:02:43.597339+0000","last_became_active":"2024-02-21T23:02:37.372557+0000","last_became_peered":"2024-02-21T23:02:37.372557+0000","last_unstale":"2024-02-21T23:02:43.597339+0000","last_undegraded":"2024-02-21T23:02:43.597339+0000","last_fullsized":"2024-02-21T23:02:43.597339+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:29:33.262208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.600960+0000","last_change":"2024-02-21T23:02:38.399333+0000","last_active":"2024-02-21T23:02:43.600960+0000","last_peered":"2024-02-21T23:02:43.600960+0000","last_clean":"2024-02-21T23:02:43.600960+0000","last_became_active":"2024-02-21T23:02:38.393010+0000","last_became_peered":"2024-02-21T23:02:38.393010+0000","last_unstale":"2024-02-21T23:02:43.600960+0000","last_undegraded":"2024-02-21T23:02:43.600960+0000","last_fullsized":"2024-02-21T23:02:43.600960+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:54:16.651108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.419916+0000","last_change":"2024-02-21T23:02:42.436700+0000","last_active":"2024-02-21T23:02:43.419916+0000","last_peered":"2024-02-21T23:02:43.419916+0000","last_clean":"2024-02-21T23:02:43.419916+0000","last_became_active":"2024-02-21T23:02:42.434264+0000","last_became_peered":"2024-02-21T23:02:42.434264+0000","last_unstale":"2024-02-21T23:02:43.419916+0000","last_undegraded":"2024-02-21T23:02:43.419916+0000","last_fullsized":"2024-02-21T23:02:43.419916+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:28:47.259045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.594139+0000","last_change":"2024-02-21T23:02:38.391008+0000","last_active":"2024-02-21T23:02:43.594139+0000","last_peered":"2024-02-21T23:02:43.594139+0000","last_clean":"2024-02-21T23:02:43.594139+0000","last_became_active":"2024-02-21T23:02:38.390695+0000","last_became_peered":"2024-02-21T23:02:38.390695+0000","last_unstale":"2024-02-21T23:02:43.594139+0000","last_undegraded":"2024-02-21T23:02:43.594139+0000","last_fullsized":"2024-02-21T23:02:43.594139+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:09:34.130536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.598232+0000","last_change":"2024-02-21T23:02:37.373206+0000","last_active":"2024-02-21T23:02:43.598232+0000","last_peered":"2024-02-21T23:02:43.598232+0000","last_clean":"2024-02-21T23:02:43.598232+0000","last_became_active":"2024-02-21T23:02:37.371805+0000","last_became_peered":"2024-02-21T23:02:37.371805+0000","last_unstale":"2024-02-21T23:02:43.598232+0000","last_undegraded":"2024-02-21T23:02:43.598232+0000","last_fullsized":"2024-02-21T23:02:43.598232+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:29:26.321681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420841+0000","last_change":"2024-02-21T23:02:42.435351+0000","last_active":"2024-02-21T23:02:43.420841+0000","last_peered":"2024-02-21T23:02:43.420841+0000","last_clean":"2024-02-21T23:02:43.420841+0000","last_became_active":"2024-02-21T23:02:42.434903+0000","last_became_peered":"2024-02-21T23:02:42.434903+0000","last_unstale":"2024-02-21T23:02:43.420841+0000","last_undegraded":"2024-02-21T23:02:43.420841+0000","last_fullsized":"2024-02-21T23:02:43.420841+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:49:24.584877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420313+0000","last_change":"2024-02-21T23:02:38.369472+0000","last_active":"2024-02-21T23:02:43.420313+0000","last_peered":"2024-02-21T23:02:43.420313+0000","last_clean":"2024-02-21T23:02:43.420313+0000","last_became_active":"2024-02-21T23:02:38.369189+0000","last_became_peered":"2024-02-21T23:02:38.369189+0000","last_unstale":"2024-02-21T23:02:43.420313+0000","last_undegraded":"2024-02-21T23:02:43.420313+0000","last_fullsized":"2024-02-21T23:02:43.420313+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:00:09.172059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.596428+0000","last_change":"2024-02-21T23:02:37.355429+0000","last_active":"2024-02-21T23:02:43.596428+0000","last_peered":"2024-02-21T23:02:43.596428+0000","last_clean":"2024-02-21T23:02:43.596428+0000","last_became_active":"2024-02-21T23:02:37.354573+0000","last_became_peered":"2024-02-21T23:02:37.354573+0000","last_unstale":"2024-02-21T23:02:43.596428+0000","last_undegraded":"2024-02-21T23:02:43.596428+0000","last_fullsized":"2024-02-21T23:02:43.596428+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:17:46.451150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420650+0000","last_change":"2024-02-21T23:02:42.445184+0000","last_active":"2024-02-21T23:02:43.420650+0000","last_peered":"2024-02-21T23:02:43.420650+0000","last_clean":"2024-02-21T23:02:43.420650+0000","last_became_active":"2024-02-21T23:02:42.444773+0000","last_became_peered":"2024-02-21T23:02:42.444773+0000","last_unstale":"2024-02-21T23:02:43.420650+0000","last_undegraded":"2024-02-21T23:02:43.420650+0000","last_fullsized":"2024-02-21T23:02:43.420650+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:24:44.830150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.419669+0000","last_change":"2024-02-21T23:02:38.379076+0000","last_active":"2024-02-21T23:02:43.419669+0000","last_peered":"2024-02-21T23:02:43.419669+0000","last_clean":"2024-02-21T23:02:43.419669+0000","last_became_active":"2024-02-21T23:02:38.378513+0000","last_became_peered":"2024-02-21T23:02:38.378513+0000","last_unstale":"2024-02-21T23:02:43.419669+0000","last_undegraded":"2024-02-21T23:02:43.419669+0000","last_fullsized":"2024-02-21T23:02:43.419669+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:29:00.068426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.595013+0000","last_change":"2024-02-21T23:02:37.359130+0000","last_active":"2024-02-21T23:02:43.595013+0000","last_peered":"2024-02-21T23:02:43.595013+0000","last_clean":"2024-02-21T23:02:43.595013+0000","last_became_active":"2024-02-21T23:02:37.358812+0000","last_became_peered":"2024-02-21T23:02:37.358812+0000","last_unstale":"2024-02-21T23:02:43.595013+0000","last_undegraded":"2024-02-21T23:02:43.595013+0000","last_fullsized":"2024-02-21T23:02:43.595013+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:29:33.262208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.598892+0000","last_change":"2024-02-21T23:02:42.500216+0000","last_active":"2024-02-21T23:02:43.598892+0000","last_peered":"2024-02-21T23:02:43.598892+0000","last_clean":"2024-02-21T23:02:43.598892+0000","last_became_active":"2024-02-21T23:02:42.499762+0000","last_became_peered":"2024-02-21T23:02:42.499762+0000","last_unstale":"2024-02-21T23:02:43.598892+0000","last_undegraded":"2024-02-21T23:02:43.598892+0000","last_fullsized":"2024-02-21T23:02:43.598892+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:12:30.131037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420590+0000","last_change":"2024-02-21T23:02:38.381297+0000","last_active":"2024-02-21T23:02:43.420590+0000","last_peered":"2024-02-21T23:02:43.420590+0000","last_clean":"2024-02-21T23:02:43.420590+0000","last_became_active":"2024-02-21T23:02:38.381015+0000","last_became_peered":"2024-02-21T23:02:38.381015+0000","last_unstale":"2024-02-21T23:02:43.420590+0000","last_undegraded":"2024-02-21T23:02:43.420590+0000","last_fullsized":"2024-02-21T23:02:43.420590+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:29:34.276860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.596339+0000","last_change":"2024-02-21T23:02:37.370201+0000","last_active":"2024-02-21T23:02:43.596339+0000","last_peered":"2024-02-21T23:02:43.596339+0000","last_clean":"2024-02-21T23:02:43.596339+0000","last_became_active":"2024-02-21T23:02:37.369814+0000","last_became_peered":"2024-02-21T23:02:37.369814+0000","last_unstale":"2024-02-21T23:02:43.596339+0000","last_undegraded":"2024-02-21T23:02:43.596339+0000","last_fullsized":"2024-02-21T23:02:43.596339+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:16:38.578056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.421168+0000","last_change":"2024-02-21T23:02:42.439870+0000","last_active":"2024-02-21T23:02:43.421168+0000","last_peered":"2024-02-21T23:02:43.421168+0000","last_clean":"2024-02-21T23:02:43.421168+0000","last_became_active":"2024-02-21T23:02:42.439470+0000","last_became_peered":"2024-02-21T23:02:42.439470+0000","last_unstale":"2024-02-21T23:02:43.421168+0000","last_undegraded":"2024-02-21T23:02:43.421168+0000","last_fullsized":"2024-02-21T23:02:43.421168+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:54:20.718854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.597460+0000","last_change":"2024-02-21T23:02:38.380559+0000","last_active":"2024-02-21T23:02:43.597460+0000","last_peered":"2024-02-21T23:02:43.597460+0000","last_clean":"2024-02-21T23:02:43.597460+0000","last_became_active":"2024-02-21T23:02:38.380229+0000","last_became_peered":"2024-02-21T23:02:38.380229+0000","last_unstale":"2024-02-21T23:02:43.597460+0000","last_undegraded":"2024-02-21T23:02:43.597460+0000","last_fullsized":"2024-02-21T23:02:43.597460+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:09:34.130536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.595919+0000","last_change":"2024-02-21T23:02:37.367812+0000","last_active":"2024-02-21T23:02:43.595919+0000","last_peered":"2024-02-21T23:02:43.595919+0000","last_clean":"2024-02-21T23:02:43.595919+0000","last_became_active":"2024-02-21T23:02:37.367527+0000","last_became_peered":"2024-02-21T23:02:37.367527+0000","last_unstale":"2024-02-21T23:02:43.595919+0000","last_undegraded":"2024-02-21T23:02:43.595919+0000","last_fullsized":"2024-02-21T23:02:43.595919+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:17:46.451150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420213+0000","last_change":"2024-02-21T23:02:42.497817+0000","last_active":"2024-02-21T23:02:43.420213+0000","last_peered":"2024-02-21T23:02:43.420213+0000","last_clean":"2024-02-21T23:02:43.420213+0000","last_became_active":"2024-02-21T23:02:42.497417+0000","last_became_peered":"2024-02-21T23:02:42.497417+0000","last_unstale":"2024-02-21T23:02:43.420213+0000","last_undegraded":"2024-02-21T23:02:43.420213+0000","last_fullsized":"2024-02-21T23:02:43.420213+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:00:13.239804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420747+0000","last_change":"2024-02-21T23:02:38.389615+0000","last_active":"2024-02-21T23:02:43.420747+0000","last_peered":"2024-02-21T23:02:43.420747+0000","last_clean":"2024-02-21T23:02:43.420747+0000","last_became_active":"2024-02-21T23:02:38.389268+0000","last_became_peered":"2024-02-21T23:02:38.389268+0000","last_unstale":"2024-02-21T23:02:43.420747+0000","last_undegraded":"2024-02-21T23:02:43.420747+0000","last_fullsized":"2024-02-21T23:02:43.420747+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:20:23.836303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.598294+0000","last_change":"2024-02-21T23:02:37.356409+0000","last_active":"2024-02-21T23:02:43.598294+0000","last_peered":"2024-02-21T23:02:43.598294+0000","last_clean":"2024-02-21T23:02:43.598294+0000","last_became_active":"2024-02-21T23:02:37.356096+0000","last_became_peered":"2024-02-21T23:02:37.356096+0000","last_unstale":"2024-02-21T23:02:43.598294+0000","last_undegraded":"2024-02-21T23:02:43.598294+0000","last_fullsized":"2024-02-21T23:02:43.598294+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:07:23.801765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.418871+0000","last_change":"2024-02-21T23:02:42.495050+0000","last_active":"2024-02-21T23:02:43.418871+0000","last_peered":"2024-02-21T23:02:43.418871+0000","last_clean":"2024-02-21T23:02:43.418871+0000","last_became_active":"2024-02-21T23:02:42.494469+0000","last_became_peered":"2024-02-21T23:02:42.494469+0000","last_unstale":"2024-02-21T23:02:43.418871+0000","last_undegraded":"2024-02-21T23:02:43.418871+0000","last_fullsized":"2024-02-21T23:02:43.418871+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:00:12.286244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.596815+0000","last_change":"2024-02-21T23:02:38.378708+0000","last_active":"2024-02-21T23:02:43.596815+0000","last_peered":"2024-02-21T23:02:43.596815+0000","last_clean":"2024-02-21T23:02:43.596815+0000","last_became_active":"2024-02-21T23:02:38.378374+0000","last_became_peered":"2024-02-21T23:02:38.378374+0000","last_unstale":"2024-02-21T23:02:43.596815+0000","last_undegraded":"2024-02-21T23:02:43.596815+0000","last_fullsized":"2024-02-21T23:02:43.596815+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:19:25.522975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.598361+0000","last_change":"2024-02-21T23:02:37.372175+0000","last_active":"2024-02-21T23:02:43.598361+0000","last_peered":"2024-02-21T23:02:43.598361+0000","last_clean":"2024-02-21T23:02:43.598361+0000","last_became_active":"2024-02-21T23:02:37.371900+0000","last_became_peered":"2024-02-21T23:02:37.371900+0000","last_unstale":"2024-02-21T23:02:43.598361+0000","last_undegraded":"2024-02-21T23:02:43.598361+0000","last_fullsized":"2024-02-21T23:02:43.598361+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:12:51.484005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.596406+0000","last_change":"2024-02-21T23:02:42.500502+0000","last_active":"2024-02-21T23:02:43.596406+0000","last_peered":"2024-02-21T23:02:43.596406+0000","last_clean":"2024-02-21T23:02:43.596406+0000","last_became_active":"2024-02-21T23:02:42.497658+0000","last_became_peered":"2024-02-21T23:02:42.497658+0000","last_unstale":"2024-02-21T23:02:43.596406+0000","last_undegraded":"2024-02-21T23:02:43.596406+0000","last_fullsized":"2024-02-21T23:02:43.596406+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:54:20.718854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.601212+0000","last_change":"2024-02-21T23:02:38.399632+0000","last_active":"2024-02-21T23:02:43.601212+0000","last_peered":"2024-02-21T23:02:43.601212+0000","last_clean":"2024-02-21T23:02:43.601212+0000","last_became_active":"2024-02-21T23:02:38.392339+0000","last_became_peered":"2024-02-21T23:02:38.392339+0000","last_unstale":"2024-02-21T23:02:43.601212+0000","last_undegraded":"2024-02-21T23:02:43.601212+0000","last_fullsized":"2024-02-21T23:02:43.601212+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:35:27.966492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.596919+0000","last_change":"2024-02-21T23:02:37.347461+0000","last_active":"2024-02-21T23:02:43.596919+0000","last_peered":"2024-02-21T23:02:43.596919+0000","last_clean":"2024-02-21T23:02:43.596919+0000","last_became_active":"2024-02-21T23:02:37.346584+0000","last_became_peered":"2024-02-21T23:02:37.346584+0000","last_unstale":"2024-02-21T23:02:43.596919+0000","last_undegraded":"2024-02-21T23:02:43.596919+0000","last_fullsized":"2024-02-21T23:02:43.596919+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:32:39.769989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.595619+0000","last_change":"2024-02-21T23:02:42.436480+0000","last_active":"2024-02-21T23:02:43.595619+0000","last_peered":"2024-02-21T23:02:43.595619+0000","last_clean":"2024-02-21T23:02:43.595619+0000","last_became_active":"2024-02-21T23:02:42.436035+0000","last_became_peered":"2024-02-21T23:02:42.436035+0000","last_unstale":"2024-02-21T23:02:43.595619+0000","last_undegraded":"2024-02-21T23:02:43.595619+0000","last_fullsized":"2024-02-21T23:02:43.595619+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:54:20.718854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420857+0000","last_change":"2024-02-21T23:02:38.373673+0000","last_active":"2024-02-21T23:02:43.420857+0000","last_peered":"2024-02-21T23:02:43.420857+0000","last_clean":"2024-02-21T23:02:43.420857+0000","last_became_active":"2024-02-21T23:02:38.373324+0000","last_became_peered":"2024-02-21T23:02:38.373324+0000","last_unstale":"2024-02-21T23:02:43.420857+0000","last_undegraded":"2024-02-21T23:02:43.420857+0000","last_fullsized":"2024-02-21T23:02:43.420857+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:09:48.394584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.600679+0000","last_change":"2024-02-21T23:02:37.346663+0000","last_active":"2024-02-21T23:02:43.600679+0000","last_peered":"2024-02-21T23:02:43.600679+0000","last_clean":"2024-02-21T23:02:43.600679+0000","last_became_active":"2024-02-21T23:02:37.344771+0000","last_became_peered":"2024-02-21T23:02:37.344771+0000","last_unstale":"2024-02-21T23:02:43.600679+0000","last_undegraded":"2024-02-21T23:02:43.600679+0000","last_fullsized":"2024-02-21T23:02:43.600679+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:12:51.484005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.596759+0000","last_change":"2024-02-21T23:02:42.440882+0000","last_active":"2024-02-21T23:02:43.596759+0000","last_peered":"2024-02-21T23:02:43.596759+0000","last_clean":"2024-02-21T23:02:43.596759+0000","last_became_active":"2024-02-21T23:02:42.440534+0000","last_became_peered":"2024-02-21T23:02:42.440534+0000","last_unstale":"2024-02-21T23:02:43.596759+0000","last_undegraded":"2024-02-21T23:02:43.596759+0000","last_fullsized":"2024-02-21T23:02:43.596759+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:35:32.034237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.595207+0000","last_change":"2024-02-21T23:02:38.377238+0000","last_active":"2024-02-21T23:02:43.595207+0000","last_peered":"2024-02-21T23:02:43.595207+0000","last_clean":"2024-02-21T23:02:43.595207+0000","last_became_active":"2024-02-21T23:02:38.376272+0000","last_became_peered":"2024-02-21T23:02:38.376272+0000","last_unstale":"2024-02-21T23:02:43.595207+0000","last_undegraded":"2024-02-21T23:02:43.595207+0000","last_fullsized":"2024-02-21T23:02:43.595207+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:10:48.186239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.415223+0000","last_change":"2024-02-21T23:02:37.360945+0000","last_active":"2024-02-21T23:02:43.415223+0000","last_peered":"2024-02-21T23:02:43.415223+0000","last_clean":"2024-02-21T23:02:43.415223+0000","last_became_active":"2024-02-21T23:02:37.359879+0000","last_became_peered":"2024-02-21T23:02:37.359879+0000","last_unstale":"2024-02-21T23:02:43.415223+0000","last_undegraded":"2024-02-21T23:02:43.415223+0000","last_fullsized":"2024-02-21T23:02:43.415223+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:07:23.801765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.598887+0000","last_change":"2024-02-21T23:02:42.441857+0000","last_active":"2024-02-21T23:02:43.598887+0000","last_peered":"2024-02-21T23:02:43.598887+0000","last_clean":"2024-02-21T23:02:43.598887+0000","last_became_active":"2024-02-21T23:02:42.438695+0000","last_became_peered":"2024-02-21T23:02:42.438695+0000","last_unstale":"2024-02-21T23:02:43.598887+0000","last_undegraded":"2024-02-21T23:02:43.598887+0000","last_fullsized":"2024-02-21T23:02:43.598887+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:28:27.820393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.594261+0000","last_change":"2024-02-21T23:02:38.366727+0000","last_active":"2024-02-21T23:02:43.594261+0000","last_peered":"2024-02-21T23:02:43.594261+0000","last_clean":"2024-02-21T23:02:43.594261+0000","last_became_active":"2024-02-21T23:02:38.366344+0000","last_became_peered":"2024-02-21T23:02:38.366344+0000","last_unstale":"2024-02-21T23:02:43.594261+0000","last_undegraded":"2024-02-21T23:02:43.594261+0000","last_fullsized":"2024-02-21T23:02:43.594261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:08:44.843392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.596392+0000","last_change":"2024-02-21T23:02:37.372506+0000","last_active":"2024-02-21T23:02:43.596392+0000","last_peered":"2024-02-21T23:02:43.596392+0000","last_clean":"2024-02-21T23:02:43.596392+0000","last_became_active":"2024-02-21T23:02:37.371806+0000","last_became_peered":"2024-02-21T23:02:37.371806+0000","last_unstale":"2024-02-21T23:02:43.596392+0000","last_undegraded":"2024-02-21T23:02:43.596392+0000","last_fullsized":"2024-02-21T23:02:43.596392+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:09:47.379932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.597397+0000","last_change":"2024-02-21T23:02:42.500339+0000","last_active":"2024-02-21T23:02:43.597397+0000","last_peered":"2024-02-21T23:02:43.597397+0000","last_clean":"2024-02-21T23:02:43.597397+0000","last_became_active":"2024-02-21T23:02:42.498471+0000","last_became_peered":"2024-02-21T23:02:42.498471+0000","last_unstale":"2024-02-21T23:02:43.597397+0000","last_undegraded":"2024-02-21T23:02:43.597397+0000","last_fullsized":"2024-02-21T23:02:43.597397+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:12:30.131037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.419210+0000","last_change":"2024-02-21T23:02:38.400850+0000","last_active":"2024-02-21T23:02:43.419210+0000","last_peered":"2024-02-21T23:02:43.419210+0000","last_clean":"2024-02-21T23:02:43.419210+0000","last_became_active":"2024-02-21T23:02:38.400367+0000","last_became_peered":"2024-02-21T23:02:38.400367+0000","last_unstale":"2024-02-21T23:02:43.419210+0000","last_undegraded":"2024-02-21T23:02:43.419210+0000","last_fullsized":"2024-02-21T23:02:43.419210+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:29:12.483912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.600646+0000","last_change":"2024-02-21T23:02:37.349152+0000","last_active":"2024-02-21T23:02:43.600646+0000","last_peered":"2024-02-21T23:02:43.600646+0000","last_clean":"2024-02-21T23:02:43.600646+0000","last_became_active":"2024-02-21T23:02:37.348647+0000","last_became_peered":"2024-02-21T23:02:37.348647+0000","last_unstale":"2024-02-21T23:02:43.600646+0000","last_undegraded":"2024-02-21T23:02:43.600646+0000","last_fullsized":"2024-02-21T23:02:43.600646+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:17:46.451150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":178,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.596598+0000","last_change":"2024-02-21T23:02:32.289335+0000","last_active":"2024-02-21T23:02:43.596598+0000","last_peered":"2024-02-21T23:02:43.596598+0000","last_clean":"2024-02-21T23:02:43.596598+0000","last_became_active":"2024-02-21T23:02:32.286913+0000","last_became_peered":"2024-02-21T23:02:32.286913+0000","last_unstale":"2024-02-21T23:02:43.596598+0000","last_undegraded":"2024-02-21T23:02:43.596598+0000","last_fullsized":"2024-02-21T23:02:43.596598+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_clean_scrub_stamp":"2024-02-21T23:02:31.172569+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:36:23.987374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.594895+0000","last_change":"2024-02-21T23:02:42.498307+0000","last_active":"2024-02-21T23:02:43.594895+0000","last_peered":"2024-02-21T23:02:43.594895+0000","last_clean":"2024-02-21T23:02:43.594895+0000","last_became_active":"2024-02-21T23:02:42.497846+0000","last_became_peered":"2024-02-21T23:02:42.497846+0000","last_unstale":"2024-02-21T23:02:43.594895+0000","last_undegraded":"2024-02-21T23:02:43.594895+0000","last_fullsized":"2024-02-21T23:02:43.594895+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:24:44.830150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.597925+0000","last_change":"2024-02-21T23:02:38.401025+0000","last_active":"2024-02-21T23:02:43.597925+0000","last_peered":"2024-02-21T23:02:43.597925+0000","last_clean":"2024-02-21T23:02:43.597925+0000","last_became_active":"2024-02-21T23:02:38.400639+0000","last_became_peered":"2024-02-21T23:02:38.400639+0000","last_unstale":"2024-02-21T23:02:43.597925+0000","last_undegraded":"2024-02-21T23:02:43.597925+0000","last_fullsized":"2024-02-21T23:02:43.597925+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:49:20.517131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.416472+0000","last_change":"2024-02-21T23:02:37.360670+0000","last_active":"2024-02-21T23:02:43.416472+0000","last_peered":"2024-02-21T23:02:43.416472+0000","last_clean":"2024-02-21T23:02:43.416472+0000","last_became_active":"2024-02-21T23:02:37.358188+0000","last_became_peered":"2024-02-21T23:02:37.358188+0000","last_unstale":"2024-02-21T23:02:43.416472+0000","last_undegraded":"2024-02-21T23:02:43.416472+0000","last_fullsized":"2024-02-21T23:02:43.416472+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:17:46.451150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.595571+0000","last_change":"2024-02-21T23:02:42.495923+0000","last_active":"2024-02-21T23:02:43.595571+0000","last_peered":"2024-02-21T23:02:43.595571+0000","last_clean":"2024-02-21T23:02:43.595571+0000","last_became_active":"2024-02-21T23:02:42.495507+0000","last_became_peered":"2024-02-21T23:02:42.495507+0000","last_unstale":"2024-02-21T23:02:43.595571+0000","last_undegraded":"2024-02-21T23:02:43.595571+0000","last_fullsized":"2024-02-21T23:02:43.595571+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:54:20.718854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.600858+0000","last_change":"2024-02-21T23:02:38.386757+0000","last_active":"2024-02-21T23:02:43.600858+0000","last_peered":"2024-02-21T23:02:43.600858+0000","last_clean":"2024-02-21T23:02:43.600858+0000","last_became_active":"2024-02-21T23:02:38.384691+0000","last_became_peered":"2024-02-21T23:02:38.384691+0000","last_unstale":"2024-02-21T23:02:43.600858+0000","last_undegraded":"2024-02-21T23:02:43.600858+0000","last_fullsized":"2024-02-21T23:02:43.600858+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:49:20.517131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.597791+0000","last_change":"2024-02-21T23:02:37.356479+0000","last_active":"2024-02-21T23:02:43.597791+0000","last_peered":"2024-02-21T23:02:43.597791+0000","last_clean":"2024-02-21T23:02:43.597791+0000","last_became_active":"2024-02-21T23:02:37.356132+0000","last_became_peered":"2024-02-21T23:02:37.356132+0000","last_unstale":"2024-02-21T23:02:43.597791+0000","last_undegraded":"2024-02-21T23:02:43.597791+0000","last_fullsized":"2024-02-21T23:02:43.597791+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:16:38.578056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.597135+0000","last_change":"2024-02-21T23:02:42.440448+0000","last_active":"2024-02-21T23:02:43.597135+0000","last_peered":"2024-02-21T23:02:43.597135+0000","last_clean":"2024-02-21T23:02:43.597135+0000","last_became_active":"2024-02-21T23:02:42.439847+0000","last_became_peered":"2024-02-21T23:02:42.439847+0000","last_unstale":"2024-02-21T23:02:43.597135+0000","last_undegraded":"2024-02-21T23:02:43.597135+0000","last_fullsized":"2024-02-21T23:02:43.597135+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:28:27.820393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.598913+0000","last_change":"2024-02-21T23:02:38.401714+0000","last_active":"2024-02-21T23:02:43.598913+0000","last_peered":"2024-02-21T23:02:43.598913+0000","last_clean":"2024-02-21T23:02:43.598913+0000","last_became_active":"2024-02-21T23:02:38.401166+0000","last_became_peered":"2024-02-21T23:02:38.401166+0000","last_unstale":"2024-02-21T23:02:43.598913+0000","last_undegraded":"2024-02-21T23:02:43.598913+0000","last_fullsized":"2024-02-21T23:02:43.598913+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:33:11.255331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.416538+0000","last_change":"2024-02-21T23:02:37.356667+0000","last_active":"2024-02-21T23:02:43.416538+0000","last_peered":"2024-02-21T23:02:43.416538+0000","last_clean":"2024-02-21T23:02:43.416538+0000","last_became_active":"2024-02-21T23:02:37.354832+0000","last_became_peered":"2024-02-21T23:02:37.354832+0000","last_unstale":"2024-02-21T23:02:43.416538+0000","last_undegraded":"2024-02-21T23:02:43.416538+0000","last_fullsized":"2024-02-21T23:02:43.416538+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:01:17.207079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.599759+0000","last_change":"2024-02-21T23:02:42.494816+0000","last_active":"2024-02-21T23:02:43.599759+0000","last_peered":"2024-02-21T23:02:43.599759+0000","last_clean":"2024-02-21T23:02:43.599759+0000","last_became_active":"2024-02-21T23:02:42.494397+0000","last_became_peered":"2024-02-21T23:02:42.494397+0000","last_unstale":"2024-02-21T23:02:43.599759+0000","last_undegraded":"2024-02-21T23:02:43.599759+0000","last_fullsized":"2024-02-21T23:02:43.599759+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:35:47.803796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.597807+0000","last_change":"2024-02-21T23:02:38.402995+0000","last_active":"2024-02-21T23:02:43.597807+0000","last_peered":"2024-02-21T23:02:43.597807+0000","last_clean":"2024-02-21T23:02:43.597807+0000","last_became_active":"2024-02-21T23:02:38.400263+0000","last_became_peered":"2024-02-21T23:02:38.400263+0000","last_unstale":"2024-02-21T23:02:43.597807+0000","last_undegraded":"2024-02-21T23:02:43.597807+0000","last_fullsized":"2024-02-21T23:02:43.597807+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:28:23.752647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.421321+0000","last_change":"2024-02-21T23:02:37.348962+0000","last_active":"2024-02-21T23:02:43.421321+0000","last_peered":"2024-02-21T23:02:43.421321+0000","last_clean":"2024-02-21T23:02:43.421321+0000","last_became_active":"2024-02-21T23:02:37.348550+0000","last_became_peered":"2024-02-21T23:02:37.348550+0000","last_unstale":"2024-02-21T23:02:43.421321+0000","last_undegraded":"2024-02-21T23:02:43.421321+0000","last_fullsized":"2024-02-21T23:02:43.421321+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:20:15.139033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.419867+0000","last_change":"2024-02-21T23:02:42.442477+0000","last_active":"2024-02-21T23:02:43.419867+0000","last_peered":"2024-02-21T23:02:43.419867+0000","last_clean":"2024-02-21T23:02:43.419867+0000","last_became_active":"2024-02-21T23:02:42.442091+0000","last_became_peered":"2024-02-21T23:02:42.442091+0000","last_unstale":"2024-02-21T23:02:43.419867+0000","last_undegraded":"2024-02-21T23:02:43.419867+0000","last_fullsized":"2024-02-21T23:02:43.419867+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:09:38.198282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.596717+0000","last_change":"2024-02-21T23:02:38.401181+0000","last_active":"2024-02-21T23:02:43.596717+0000","last_peered":"2024-02-21T23:02:43.596717+0000","last_clean":"2024-02-21T23:02:43.596717+0000","last_became_active":"2024-02-21T23:02:38.400463+0000","last_became_peered":"2024-02-21T23:02:38.400463+0000","last_unstale":"2024-02-21T23:02:43.596717+0000","last_undegraded":"2024-02-21T23:02:43.596717+0000","last_fullsized":"2024-02-21T23:02:43.596717+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:05:40.265121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.414478+0000","last_change":"2024-02-21T23:02:37.348047+0000","last_active":"2024-02-21T23:02:43.414478+0000","last_peered":"2024-02-21T23:02:43.414478+0000","last_clean":"2024-02-21T23:02:43.414478+0000","last_became_active":"2024-02-21T23:02:37.347664+0000","last_became_peered":"2024-02-21T23:02:37.347664+0000","last_unstale":"2024-02-21T23:02:43.414478+0000","last_undegraded":"2024-02-21T23:02:43.414478+0000","last_fullsized":"2024-02-21T23:02:43.414478+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:39:34.395881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.596934+0000","last_change":"2024-02-21T23:02:42.447743+0000","last_active":"2024-02-21T23:02:43.596934+0000","last_peered":"2024-02-21T23:02:43.596934+0000","last_clean":"2024-02-21T23:02:43.596934+0000","last_became_active":"2024-02-21T23:02:42.447181+0000","last_became_peered":"2024-02-21T23:02:42.447181+0000","last_unstale":"2024-02-21T23:02:43.596934+0000","last_undegraded":"2024-02-21T23:02:43.596934+0000","last_fullsized":"2024-02-21T23:02:43.596934+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:07:03.424869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.594280+0000","last_change":"2024-02-21T23:02:38.375362+0000","last_active":"2024-02-21T23:02:43.594280+0000","last_peered":"2024-02-21T23:02:43.594280+0000","last_clean":"2024-02-21T23:02:43.594280+0000","last_became_active":"2024-02-21T23:02:38.375081+0000","last_became_peered":"2024-02-21T23:02:38.375081+0000","last_unstale":"2024-02-21T23:02:43.594280+0000","last_undegraded":"2024-02-21T23:02:43.594280+0000","last_fullsized":"2024-02-21T23:02:43.594280+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:19:25.522975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.595646+0000","last_change":"2024-02-21T23:02:37.359690+0000","last_active":"2024-02-21T23:02:43.595646+0000","last_peered":"2024-02-21T23:02:43.595646+0000","last_clean":"2024-02-21T23:02:43.595646+0000","last_became_active":"2024-02-21T23:02:37.359396+0000","last_became_peered":"2024-02-21T23:02:37.359396+0000","last_unstale":"2024-02-21T23:02:43.595646+0000","last_undegraded":"2024-02-21T23:02:43.595646+0000","last_fullsized":"2024-02-21T23:02:43.595646+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:29:33.262208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420901+0000","last_change":"2024-02-21T23:02:42.499203+0000","last_active":"2024-02-21T23:02:43.420901+0000","last_peered":"2024-02-21T23:02:43.420901+0000","last_clean":"2024-02-21T23:02:43.420901+0000","last_became_active":"2024-02-21T23:02:42.498781+0000","last_became_peered":"2024-02-21T23:02:42.498781+0000","last_unstale":"2024-02-21T23:02:43.420901+0000","last_undegraded":"2024-02-21T23:02:43.420901+0000","last_fullsized":"2024-02-21T23:02:43.420901+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:08:32.584721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.598100+0000","last_change":"2024-02-21T23:02:38.402276+0000","last_active":"2024-02-21T23:02:43.598100+0000","last_peered":"2024-02-21T23:02:43.598100+0000","last_clean":"2024-02-21T23:02:43.598100+0000","last_became_active":"2024-02-21T23:02:38.401976+0000","last_became_peered":"2024-02-21T23:02:38.401976+0000","last_unstale":"2024-02-21T23:02:43.598100+0000","last_undegraded":"2024-02-21T23:02:43.598100+0000","last_fullsized":"2024-02-21T23:02:43.598100+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:12:26.063291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.600414+0000","last_change":"2024-02-21T23:02:37.359735+0000","last_active":"2024-02-21T23:02:43.600414+0000","last_peered":"2024-02-21T23:02:43.600414+0000","last_clean":"2024-02-21T23:02:43.600414+0000","last_became_active":"2024-02-21T23:02:37.359447+0000","last_became_peered":"2024-02-21T23:02:37.359447+0000","last_unstale":"2024-02-21T23:02:43.600414+0000","last_undegraded":"2024-02-21T23:02:43.600414+0000","last_fullsized":"2024-02-21T23:02:43.600414+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:29:26.321681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.421314+0000","last_change":"2024-02-21T23:02:42.494896+0000","last_active":"2024-02-21T23:02:43.421314+0000","last_peered":"2024-02-21T23:02:43.421314+0000","last_clean":"2024-02-21T23:02:43.421314+0000","last_became_active":"2024-02-21T23:02:42.494433+0000","last_became_peered":"2024-02-21T23:02:42.494433+0000","last_unstale":"2024-02-21T23:02:43.421314+0000","last_undegraded":"2024-02-21T23:02:43.421314+0000","last_fullsized":"2024-02-21T23:02:43.421314+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:53:14.069591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.595745+0000","last_change":"2024-02-21T23:02:38.389883+0000","last_active":"2024-02-21T23:02:43.595745+0000","last_peered":"2024-02-21T23:02:43.595745+0000","last_clean":"2024-02-21T23:02:43.595745+0000","last_became_active":"2024-02-21T23:02:38.389132+0000","last_became_peered":"2024-02-21T23:02:38.389132+0000","last_unstale":"2024-02-21T23:02:43.595745+0000","last_undegraded":"2024-02-21T23:02:43.595745+0000","last_fullsized":"2024-02-21T23:02:43.595745+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:49:26.724162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.597566+0000","last_change":"2024-02-21T23:02:37.354560+0000","last_active":"2024-02-21T23:02:43.597566+0000","last_peered":"2024-02-21T23:02:43.597566+0000","last_clean":"2024-02-21T23:02:43.597566+0000","last_became_active":"2024-02-21T23:02:37.354266+0000","last_became_peered":"2024-02-21T23:02:37.354266+0000","last_unstale":"2024-02-21T23:02:43.597566+0000","last_undegraded":"2024-02-21T23:02:43.597566+0000","last_fullsized":"2024-02-21T23:02:43.597566+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:36:43.400580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.601188+0000","last_change":"2024-02-21T23:02:42.496211+0000","last_active":"2024-02-21T23:02:43.601188+0000","last_peered":"2024-02-21T23:02:43.601188+0000","last_clean":"2024-02-21T23:02:43.601188+0000","last_became_active":"2024-02-21T23:02:42.495493+0000","last_became_peered":"2024-02-21T23:02:42.495493+0000","last_unstale":"2024-02-21T23:02:43.601188+0000","last_undegraded":"2024-02-21T23:02:43.601188+0000","last_fullsized":"2024-02-21T23:02:43.601188+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:00:12.286244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.597793+0000","last_change":"2024-02-21T23:02:38.395172+0000","last_active":"2024-02-21T23:02:43.597793+0000","last_peered":"2024-02-21T23:02:43.597793+0000","last_clean":"2024-02-21T23:02:43.597793+0000","last_became_active":"2024-02-21T23:02:38.394809+0000","last_became_peered":"2024-02-21T23:02:38.394809+0000","last_unstale":"2024-02-21T23:02:43.597793+0000","last_undegraded":"2024-02-21T23:02:43.597793+0000","last_fullsized":"2024-02-21T23:02:43.597793+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:20:23.836303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"24'10","reported_seq":32,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:57.662420+0000","last_change":"2024-02-21T23:02:37.362539+0000","last_active":"2024-02-21T23:02:57.662420+0000","last_peered":"2024-02-21T23:02:57.662420+0000","last_clean":"2024-02-21T23:02:57.662420+0000","last_became_active":"2024-02-21T23:02:37.362186+0000","last_became_peered":"2024-02-21T23:02:37.362186+0000","last_unstale":"2024-02-21T23:02:57.662420+0000","last_undegraded":"2024-02-21T23:02:57.662420+0000","last_fullsized":"2024-02-21T23:02:57.662420+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:32:39.769989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.597415+0000","last_change":"2024-02-21T23:02:42.433330+0000","last_active":"2024-02-21T23:02:43.597415+0000","last_peered":"2024-02-21T23:02:43.597415+0000","last_clean":"2024-02-21T23:02:43.597415+0000","last_became_active":"2024-02-21T23:02:42.432944+0000","last_became_peered":"2024-02-21T23:02:42.432944+0000","last_unstale":"2024-02-21T23:02:43.597415+0000","last_undegraded":"2024-02-21T23:02:43.597415+0000","last_fullsized":"2024-02-21T23:02:43.597415+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:07:03.424869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420973+0000","last_change":"2024-02-21T23:02:38.389764+0000","last_active":"2024-02-21T23:02:43.420973+0000","last_peered":"2024-02-21T23:02:43.420973+0000","last_clean":"2024-02-21T23:02:43.420973+0000","last_became_active":"2024-02-21T23:02:38.389176+0000","last_became_peered":"2024-02-21T23:02:38.389176+0000","last_unstale":"2024-02-21T23:02:43.420973+0000","last_undegraded":"2024-02-21T23:02:43.420973+0000","last_fullsized":"2024-02-21T23:02:43.420973+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:12:46.637812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.597677+0000","last_change":"2024-02-21T23:02:37.344126+0000","last_active":"2024-02-21T23:02:43.597677+0000","last_peered":"2024-02-21T23:02:43.597677+0000","last_clean":"2024-02-21T23:02:43.597677+0000","last_became_active":"2024-02-21T23:02:37.343769+0000","last_became_peered":"2024-02-21T23:02:37.343769+0000","last_unstale":"2024-02-21T23:02:43.597677+0000","last_undegraded":"2024-02-21T23:02:43.597677+0000","last_fullsized":"2024-02-21T23:02:43.597677+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:09:47.379932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420591+0000","last_change":"2024-02-21T23:02:42.440359+0000","last_active":"2024-02-21T23:02:43.420591+0000","last_peered":"2024-02-21T23:02:43.420591+0000","last_clean":"2024-02-21T23:02:43.420591+0000","last_became_active":"2024-02-21T23:02:42.439967+0000","last_became_peered":"2024-02-21T23:02:42.439967+0000","last_unstale":"2024-02-21T23:02:43.420591+0000","last_undegraded":"2024-02-21T23:02:43.420591+0000","last_fullsized":"2024-02-21T23:02:43.420591+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:20:27.904049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.597926+0000","last_change":"2024-02-21T23:02:38.393522+0000","last_active":"2024-02-21T23:02:43.597926+0000","last_peered":"2024-02-21T23:02:43.597926+0000","last_clean":"2024-02-21T23:02:43.597926+0000","last_became_active":"2024-02-21T23:02:38.392890+0000","last_became_peered":"2024-02-21T23:02:38.392890+0000","last_unstale":"2024-02-21T23:02:43.597926+0000","last_undegraded":"2024-02-21T23:02:43.597926+0000","last_fullsized":"2024-02-21T23:02:43.597926+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:30:37.791426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.601175+0000","last_change":"2024-02-21T23:02:37.361082+0000","last_active":"2024-02-21T23:02:43.601175+0000","last_peered":"2024-02-21T23:02:43.601175+0000","last_clean":"2024-02-21T23:02:43.601175+0000","last_became_active":"2024-02-21T23:02:37.360751+0000","last_became_peered":"2024-02-21T23:02:37.360751+0000","last_unstale":"2024-02-21T23:02:43.601175+0000","last_undegraded":"2024-02-21T23:02:43.601175+0000","last_fullsized":"2024-02-21T23:02:43.601175+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:09:47.379932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.419487+0000","last_change":"2024-02-21T23:02:42.440260+0000","last_active":"2024-02-21T23:02:43.419487+0000","last_peered":"2024-02-21T23:02:43.419487+0000","last_clean":"2024-02-21T23:02:43.419487+0000","last_became_active":"2024-02-21T23:02:42.439785+0000","last_became_peered":"2024-02-21T23:02:42.439785+0000","last_unstale":"2024-02-21T23:02:43.419487+0000","last_undegraded":"2024-02-21T23:02:43.419487+0000","last_fullsized":"2024-02-21T23:02:43.419487+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:42:58.884647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.600988+0000","last_change":"2024-02-21T23:02:38.400013+0000","last_active":"2024-02-21T23:02:43.600988+0000","last_peered":"2024-02-21T23:02:43.600988+0000","last_clean":"2024-02-21T23:02:43.600988+0000","last_became_active":"2024-02-21T23:02:38.399708+0000","last_became_peered":"2024-02-21T23:02:38.399708+0000","last_unstale":"2024-02-21T23:02:43.600988+0000","last_undegraded":"2024-02-21T23:02:43.600988+0000","last_fullsized":"2024-02-21T23:02:43.600988+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:06:59.357124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.595798+0000","last_change":"2024-02-21T23:02:37.366264+0000","last_active":"2024-02-21T23:02:43.595798+0000","last_peered":"2024-02-21T23:02:43.595798+0000","last_clean":"2024-02-21T23:02:43.595798+0000","last_became_active":"2024-02-21T23:02:37.365889+0000","last_became_peered":"2024-02-21T23:02:37.365889+0000","last_unstale":"2024-02-21T23:02:43.595798+0000","last_undegraded":"2024-02-21T23:02:43.595798+0000","last_fullsized":"2024-02-21T23:02:43.595798+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:07:23.801765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.419552+0000","last_change":"2024-02-21T23:02:42.495489+0000","last_active":"2024-02-21T23:02:43.419552+0000","last_peered":"2024-02-21T23:02:43.419552+0000","last_clean":"2024-02-21T23:02:43.419552+0000","last_became_active":"2024-02-21T23:02:42.495098+0000","last_became_peered":"2024-02-21T23:02:42.495098+0000","last_unstale":"2024-02-21T23:02:43.419552+0000","last_undegraded":"2024-02-21T23:02:43.419552+0000","last_fullsized":"2024-02-21T23:02:43.419552+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:10:21.610343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.596467+0000","last_change":"2024-02-21T23:02:42.500336+0000","last_active":"2024-02-21T23:02:43.596467+0000","last_peered":"2024-02-21T23:02:43.596467+0000","last_clean":"2024-02-21T23:02:43.596467+0000","last_became_active":"2024-02-21T23:02:42.498817+0000","last_became_peered":"2024-02-21T23:02:42.498817+0000","last_unstale":"2024-02-21T23:02:43.596467+0000","last_undegraded":"2024-02-21T23:02:43.596467+0000","last_fullsized":"2024-02-21T23:02:43.596467+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:35:32.034237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.594893+0000","last_change":"2024-02-21T23:02:38.378745+0000","last_active":"2024-02-21T23:02:43.594893+0000","last_peered":"2024-02-21T23:02:43.594893+0000","last_clean":"2024-02-21T23:02:43.594893+0000","last_became_active":"2024-02-21T23:02:38.378423+0000","last_became_peered":"2024-02-21T23:02:38.378423+0000","last_unstale":"2024-02-21T23:02:43.594893+0000","last_undegraded":"2024-02-21T23:02:43.594893+0000","last_fullsized":"2024-02-21T23:02:43.594893+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:30:37.791426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.600043+0000","last_change":"2024-02-21T23:02:37.352231+0000","last_active":"2024-02-21T23:02:43.600043+0000","last_peered":"2024-02-21T23:02:43.600043+0000","last_clean":"2024-02-21T23:02:43.600043+0000","last_became_active":"2024-02-21T23:02:37.351715+0000","last_became_peered":"2024-02-21T23:02:37.351715+0000","last_unstale":"2024-02-21T23:02:43.600043+0000","last_undegraded":"2024-02-21T23:02:43.600043+0000","last_fullsized":"2024-02-21T23:02:43.600043+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:07:23.801765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.595272+0000","last_change":"2024-02-21T23:02:42.449848+0000","last_active":"2024-02-21T23:02:43.595272+0000","last_peered":"2024-02-21T23:02:43.595272+0000","last_clean":"2024-02-21T23:02:43.595272+0000","last_became_active":"2024-02-21T23:02:42.449357+0000","last_became_peered":"2024-02-21T23:02:42.449357+0000","last_unstale":"2024-02-21T23:02:43.595272+0000","last_undegraded":"2024-02-21T23:02:43.595272+0000","last_fullsized":"2024-02-21T23:02:43.595272+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:49:24.584877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.419241+0000","last_change":"2024-02-21T23:02:38.368240+0000","last_active":"2024-02-21T23:02:43.419241+0000","last_peered":"2024-02-21T23:02:43.419241+0000","last_clean":"2024-02-21T23:02:43.419241+0000","last_became_active":"2024-02-21T23:02:38.367852+0000","last_became_peered":"2024-02-21T23:02:38.367852+0000","last_unstale":"2024-02-21T23:02:43.419241+0000","last_undegraded":"2024-02-21T23:02:43.419241+0000","last_fullsized":"2024-02-21T23:02:43.419241+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:30:37.791426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420051+0000","last_change":"2024-02-21T23:02:37.363129+0000","last_active":"2024-02-21T23:02:43.420051+0000","last_peered":"2024-02-21T23:02:43.420051+0000","last_clean":"2024-02-21T23:02:43.420051+0000","last_became_active":"2024-02-21T23:02:37.362829+0000","last_became_peered":"2024-02-21T23:02:37.362829+0000","last_unstale":"2024-02-21T23:02:43.420051+0000","last_undegraded":"2024-02-21T23:02:43.420051+0000","last_fullsized":"2024-02-21T23:02:43.420051+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:39:34.395881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.594502+0000","last_change":"2024-02-21T23:02:37.358568+0000","last_active":"2024-02-21T23:02:43.594502+0000","last_peered":"2024-02-21T23:02:43.594502+0000","last_clean":"2024-02-21T23:02:43.594502+0000","last_became_active":"2024-02-21T23:02:37.358238+0000","last_became_peered":"2024-02-21T23:02:37.358238+0000","last_unstale":"2024-02-21T23:02:43.594502+0000","last_undegraded":"2024-02-21T23:02:43.594502+0000","last_fullsized":"2024-02-21T23:02:43.594502+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:01:17.207079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.597289+0000","last_change":"2024-02-21T23:02:38.375982+0000","last_active":"2024-02-21T23:02:43.597289+0000","last_peered":"2024-02-21T23:02:43.597289+0000","last_clean":"2024-02-21T23:02:43.597289+0000","last_became_active":"2024-02-21T23:02:38.375504+0000","last_became_peered":"2024-02-21T23:02:38.375504+0000","last_unstale":"2024-02-21T23:02:43.597289+0000","last_undegraded":"2024-02-21T23:02:43.597289+0000","last_fullsized":"2024-02-21T23:02:43.597289+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:19:25.522975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.416324+0000","last_change":"2024-02-21T23:02:42.441484+0000","last_active":"2024-02-21T23:02:43.416324+0000","last_peered":"2024-02-21T23:02:43.416324+0000","last_clean":"2024-02-21T23:02:43.416324+0000","last_became_active":"2024-02-21T23:02:42.438094+0000","last_became_peered":"2024-02-21T23:02:42.438094+0000","last_unstale":"2024-02-21T23:02:43.416324+0000","last_undegraded":"2024-02-21T23:02:43.416324+0000","last_fullsized":"2024-02-21T23:02:43.416324+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:04:15.959906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420559+0000","last_change":"2024-02-21T23:02:37.365019+0000","last_active":"2024-02-21T23:02:43.420559+0000","last_peered":"2024-02-21T23:02:43.420559+0000","last_clean":"2024-02-21T23:02:43.420559+0000","last_became_active":"2024-02-21T23:02:37.364761+0000","last_became_peered":"2024-02-21T23:02:37.364761+0000","last_unstale":"2024-02-21T23:02:43.420559+0000","last_undegraded":"2024-02-21T23:02:43.420559+0000","last_fullsized":"2024-02-21T23:02:43.420559+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:01:17.207079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420052+0000","last_change":"2024-02-21T23:02:38.385160+0000","last_active":"2024-02-21T23:02:43.420052+0000","last_peered":"2024-02-21T23:02:43.420052+0000","last_clean":"2024-02-21T23:02:43.420052+0000","last_became_active":"2024-02-21T23:02:38.382830+0000","last_became_peered":"2024-02-21T23:02:38.382830+0000","last_unstale":"2024-02-21T23:02:43.420052+0000","last_undegraded":"2024-02-21T23:02:43.420052+0000","last_fullsized":"2024-02-21T23:02:43.420052+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:16:39.592708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.421894+0000","last_change":"2024-02-21T23:02:42.437091+0000","last_active":"2024-02-21T23:02:43.421894+0000","last_peered":"2024-02-21T23:02:43.421894+0000","last_clean":"2024-02-21T23:02:43.421894+0000","last_became_active":"2024-02-21T23:02:42.436674+0000","last_became_peered":"2024-02-21T23:02:42.436674+0000","last_unstale":"2024-02-21T23:02:43.421894+0000","last_undegraded":"2024-02-21T23:02:43.421894+0000","last_fullsized":"2024-02-21T23:02:43.421894+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:16:29.140022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.594049+0000","last_change":"2024-02-21T23:02:38.379817+0000","last_active":"2024-02-21T23:02:43.594049+0000","last_peered":"2024-02-21T23:02:43.594049+0000","last_clean":"2024-02-21T23:02:43.594049+0000","last_became_active":"2024-02-21T23:02:38.379367+0000","last_became_peered":"2024-02-21T23:02:38.379367+0000","last_unstale":"2024-02-21T23:02:43.594049+0000","last_undegraded":"2024-02-21T23:02:43.594049+0000","last_fullsized":"2024-02-21T23:02:43.594049+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:04:11.892160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.415084+0000","last_change":"2024-02-21T23:02:38.386264+0000","last_active":"2024-02-21T23:02:43.415084+0000","last_peered":"2024-02-21T23:02:43.415084+0000","last_clean":"2024-02-21T23:02:43.415084+0000","last_became_active":"2024-02-21T23:02:38.385976+0000","last_became_peered":"2024-02-21T23:02:38.385976+0000","last_unstale":"2024-02-21T23:02:43.415084+0000","last_undegraded":"2024-02-21T23:02:43.415084+0000","last_fullsized":"2024-02-21T23:02:43.415084+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:09:48.394584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.600386+0000","last_change":"2024-02-21T23:02:38.394227+0000","last_active":"2024-02-21T23:02:43.600386+0000","last_peered":"2024-02-21T23:02:43.600386+0000","last_clean":"2024-02-21T23:02:43.600386+0000","last_became_active":"2024-02-21T23:02:38.393965+0000","last_became_peered":"2024-02-21T23:02:38.393965+0000","last_unstale":"2024-02-21T23:02:43.600386+0000","last_undegraded":"2024-02-21T23:02:43.600386+0000","last_fullsized":"2024-02-21T23:02:43.600386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:20:23.836303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.596804+0000","last_change":"2024-02-21T23:02:38.365717+0000","last_active":"2024-02-21T23:02:43.596804+0000","last_peered":"2024-02-21T23:02:43.596804+0000","last_clean":"2024-02-21T23:02:43.596804+0000","last_became_active":"2024-02-21T23:02:38.365375+0000","last_became_peered":"2024-02-21T23:02:38.365375+0000","last_unstale":"2024-02-21T23:02:43.596804+0000","last_undegraded":"2024-02-21T23:02:43.596804+0000","last_fullsized":"2024-02-21T23:02:43.596804+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:29:00.068426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.421488+0000","last_change":"2024-02-21T23:02:38.376971+0000","last_active":"2024-02-21T23:02:43.421488+0000","last_peered":"2024-02-21T23:02:43.421488+0000","last_clean":"2024-02-21T23:02:43.421488+0000","last_became_active":"2024-02-21T23:02:38.376565+0000","last_became_peered":"2024-02-21T23:02:38.376565+0000","last_unstale":"2024-02-21T23:02:43.421488+0000","last_undegraded":"2024-02-21T23:02:43.421488+0000","last_fullsized":"2024-02-21T23:02:43.421488+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:28:43.191300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.597197+0000","last_change":"2024-02-21T23:02:38.379350+0000","last_active":"2024-02-21T23:02:43.597197+0000","last_peered":"2024-02-21T23:02:43.597197+0000","last_clean":"2024-02-21T23:02:43.597197+0000","last_became_active":"2024-02-21T23:02:38.379030+0000","last_became_peered":"2024-02-21T23:02:38.379030+0000","last_unstale":"2024-02-21T23:02:43.597197+0000","last_undegraded":"2024-02-21T23:02:43.597197+0000","last_fullsized":"2024-02-21T23:02:43.597197+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:09:34.130536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.595248+0000","last_change":"2024-02-21T23:02:38.403472+0000","last_active":"2024-02-21T23:02:43.595248+0000","last_peered":"2024-02-21T23:02:43.595248+0000","last_clean":"2024-02-21T23:02:43.595248+0000","last_became_active":"2024-02-21T23:02:38.403213+0000","last_became_peered":"2024-02-21T23:02:38.403213+0000","last_unstale":"2024-02-21T23:02:43.595248+0000","last_undegraded":"2024-02-21T23:02:43.595248+0000","last_fullsized":"2024-02-21T23:02:43.595248+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:29:12.483912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420337+0000","last_change":"2024-02-21T23:02:38.377455+0000","last_active":"2024-02-21T23:02:43.420337+0000","last_peered":"2024-02-21T23:02:43.420337+0000","last_clean":"2024-02-21T23:02:43.420337+0000","last_became_active":"2024-02-21T23:02:38.376781+0000","last_became_peered":"2024-02-21T23:02:38.376781+0000","last_unstale":"2024-02-21T23:02:43.420337+0000","last_undegraded":"2024-02-21T23:02:43.420337+0000","last_fullsized":"2024-02-21T23:02:43.420337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:24:40.762404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.599882+0000","last_change":"2024-02-21T23:02:38.386554+0000","last_active":"2024-02-21T23:02:43.599882+0000","last_peered":"2024-02-21T23:02:43.599882+0000","last_clean":"2024-02-21T23:02:43.599882+0000","last_became_active":"2024-02-21T23:02:38.384500+0000","last_became_peered":"2024-02-21T23:02:38.384500+0000","last_unstale":"2024-02-21T23:02:43.599882+0000","last_undegraded":"2024-02-21T23:02:43.599882+0000","last_fullsized":"2024-02-21T23:02:43.599882+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:09:34.130536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420337+0000","last_change":"2024-02-21T23:02:38.383893+0000","last_active":"2024-02-21T23:02:43.420337+0000","last_peered":"2024-02-21T23:02:43.420337+0000","last_clean":"2024-02-21T23:02:43.420337+0000","last_became_active":"2024-02-21T23:02:38.383569+0000","last_became_peered":"2024-02-21T23:02:38.383569+0000","last_unstale":"2024-02-21T23:02:43.420337+0000","last_undegraded":"2024-02-21T23:02:43.420337+0000","last_fullsized":"2024-02-21T23:02:43.420337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:19:25.522975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.600302+0000","last_change":"2024-02-21T23:02:38.395013+0000","last_active":"2024-02-21T23:02:43.600302+0000","last_peered":"2024-02-21T23:02:43.600302+0000","last_clean":"2024-02-21T23:02:43.600302+0000","last_became_active":"2024-02-21T23:02:38.394670+0000","last_became_peered":"2024-02-21T23:02:38.394670+0000","last_unstale":"2024-02-21T23:02:43.600302+0000","last_undegraded":"2024-02-21T23:02:43.600302+0000","last_fullsized":"2024-02-21T23:02:43.600302+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:54:40.962238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.596541+0000","last_change":"2024-02-21T23:02:38.385293+0000","last_active":"2024-02-21T23:02:43.596541+0000","last_peered":"2024-02-21T23:02:43.596541+0000","last_clean":"2024-02-21T23:02:43.596541+0000","last_became_active":"2024-02-21T23:02:38.384589+0000","last_became_peered":"2024-02-21T23:02:38.384589+0000","last_unstale":"2024-02-21T23:02:43.596541+0000","last_undegraded":"2024-02-21T23:02:43.596541+0000","last_fullsized":"2024-02-21T23:02:43.596541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:10:48.186239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.595433+0000","last_change":"2024-02-21T23:02:38.385769+0000","last_active":"2024-02-21T23:02:43.595433+0000","last_peered":"2024-02-21T23:02:43.595433+0000","last_clean":"2024-02-21T23:02:43.595433+0000","last_became_active":"2024-02-21T23:02:38.385488+0000","last_became_peered":"2024-02-21T23:02:38.385488+0000","last_unstale":"2024-02-21T23:02:43.595433+0000","last_undegraded":"2024-02-21T23:02:43.595433+0000","last_fullsized":"2024-02-21T23:02:43.595433+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:29:00.068426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.598237+0000","last_change":"2024-02-21T23:02:38.396764+0000","last_active":"2024-02-21T23:02:43.598237+0000","last_peered":"2024-02-21T23:02:43.598237+0000","last_clean":"2024-02-21T23:02:43.598237+0000","last_became_active":"2024-02-21T23:02:38.396438+0000","last_became_peered":"2024-02-21T23:02:38.396438+0000","last_unstale":"2024-02-21T23:02:43.598237+0000","last_undegraded":"2024-02-21T23:02:43.598237+0000","last_fullsized":"2024-02-21T23:02:43.598237+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:06:59.357124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.416220+0000","last_change":"2024-02-21T23:02:38.387876+0000","last_active":"2024-02-21T23:02:43.416220+0000","last_peered":"2024-02-21T23:02:43.416220+0000","last_clean":"2024-02-21T23:02:43.416220+0000","last_became_active":"2024-02-21T23:02:38.383130+0000","last_became_peered":"2024-02-21T23:02:38.383130+0000","last_unstale":"2024-02-21T23:02:43.416220+0000","last_undegraded":"2024-02-21T23:02:43.416220+0000","last_fullsized":"2024-02-21T23:02:43.416220+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:12:46.637812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.601961+0000","last_change":"2024-02-21T23:02:38.403615+0000","last_active":"2024-02-21T23:02:43.601961+0000","last_peered":"2024-02-21T23:02:43.601961+0000","last_clean":"2024-02-21T23:02:43.601961+0000","last_became_active":"2024-02-21T23:02:38.402773+0000","last_became_peered":"2024-02-21T23:02:38.402773+0000","last_unstale":"2024-02-21T23:02:43.601961+0000","last_undegraded":"2024-02-21T23:02:43.601961+0000","last_fullsized":"2024-02-21T23:02:43.601961+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:12:26.063291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.599735+0000","last_change":"2024-02-21T23:02:38.384479+0000","last_active":"2024-02-21T23:02:43.599735+0000","last_peered":"2024-02-21T23:02:43.599735+0000","last_clean":"2024-02-21T23:02:43.599735+0000","last_became_active":"2024-02-21T23:02:38.383995+0000","last_became_peered":"2024-02-21T23:02:38.383995+0000","last_unstale":"2024-02-21T23:02:43.599735+0000","last_undegraded":"2024-02-21T23:02:43.599735+0000","last_fullsized":"2024-02-21T23:02:43.599735+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:28:43.191300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.595040+0000","last_change":"2024-02-21T23:02:38.390823+0000","last_active":"2024-02-21T23:02:43.595040+0000","last_peered":"2024-02-21T23:02:43.595040+0000","last_clean":"2024-02-21T23:02:43.595040+0000","last_became_active":"2024-02-21T23:02:38.390390+0000","last_became_peered":"2024-02-21T23:02:38.390390+0000","last_unstale":"2024-02-21T23:02:43.595040+0000","last_undegraded":"2024-02-21T23:02:43.595040+0000","last_fullsized":"2024-02-21T23:02:43.595040+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:29:00.068426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.595697+0000","last_change":"2024-02-21T23:02:38.380151+0000","last_active":"2024-02-21T23:02:43.595697+0000","last_peered":"2024-02-21T23:02:43.595697+0000","last_clean":"2024-02-21T23:02:43.595697+0000","last_became_active":"2024-02-21T23:02:38.379792+0000","last_became_peered":"2024-02-21T23:02:38.379792+0000","last_unstale":"2024-02-21T23:02:43.595697+0000","last_undegraded":"2024-02-21T23:02:43.595697+0000","last_fullsized":"2024-02-21T23:02:43.595697+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:39:18.270811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420027+0000","last_change":"2024-02-21T23:02:38.379804+0000","last_active":"2024-02-21T23:02:43.420027+0000","last_peered":"2024-02-21T23:02:43.420027+0000","last_clean":"2024-02-21T23:02:43.420027+0000","last_became_active":"2024-02-21T23:02:38.379461+0000","last_became_peered":"2024-02-21T23:02:38.379461+0000","last_unstale":"2024-02-21T23:02:43.420027+0000","last_undegraded":"2024-02-21T23:02:43.420027+0000","last_fullsized":"2024-02-21T23:02:43.420027+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:09:34.130536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420766+0000","last_change":"2024-02-21T23:02:38.381806+0000","last_active":"2024-02-21T23:02:43.420766+0000","last_peered":"2024-02-21T23:02:43.420766+0000","last_clean":"2024-02-21T23:02:43.420766+0000","last_became_active":"2024-02-21T23:02:38.381475+0000","last_became_peered":"2024-02-21T23:02:38.381475+0000","last_unstale":"2024-02-21T23:02:43.420766+0000","last_undegraded":"2024-02-21T23:02:43.420766+0000","last_fullsized":"2024-02-21T23:02:43.420766+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:29:27.336333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.595120+0000","last_change":"2024-02-21T23:02:38.398122+0000","last_active":"2024-02-21T23:02:43.595120+0000","last_peered":"2024-02-21T23:02:43.595120+0000","last_clean":"2024-02-21T23:02:43.595120+0000","last_became_active":"2024-02-21T23:02:38.392050+0000","last_became_peered":"2024-02-21T23:02:38.392050+0000","last_unstale":"2024-02-21T23:02:43.595120+0000","last_undegraded":"2024-02-21T23:02:43.595120+0000","last_fullsized":"2024-02-21T23:02:43.595120+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:10:48.186239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.594981+0000","last_change":"2024-02-21T23:02:38.398482+0000","last_active":"2024-02-21T23:02:43.594981+0000","last_peered":"2024-02-21T23:02:43.594981+0000","last_clean":"2024-02-21T23:02:43.594981+0000","last_became_active":"2024-02-21T23:02:38.392232+0000","last_became_peered":"2024-02-21T23:02:38.392232+0000","last_unstale":"2024-02-21T23:02:43.594981+0000","last_undegraded":"2024-02-21T23:02:43.594981+0000","last_fullsized":"2024-02-21T23:02:43.594981+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:30:37.791426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.598787+0000","last_change":"2024-02-21T23:02:38.400825+0000","last_active":"2024-02-21T23:02:43.598787+0000","last_peered":"2024-02-21T23:02:43.598787+0000","last_clean":"2024-02-21T23:02:43.598787+0000","last_became_active":"2024-02-21T23:02:38.398964+0000","last_became_peered":"2024-02-21T23:02:38.398964+0000","last_unstale":"2024-02-21T23:02:43.598787+0000","last_undegraded":"2024-02-21T23:02:43.598787+0000","last_fullsized":"2024-02-21T23:02:43.598787+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:24:40.762404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.418962+0000","last_change":"2024-02-21T23:02:38.387131+0000","last_active":"2024-02-21T23:02:43.418962+0000","last_peered":"2024-02-21T23:02:43.418962+0000","last_clean":"2024-02-21T23:02:43.418962+0000","last_became_active":"2024-02-21T23:02:38.386835+0000","last_became_peered":"2024-02-21T23:02:38.386835+0000","last_unstale":"2024-02-21T23:02:43.418962+0000","last_undegraded":"2024-02-21T23:02:43.418962+0000","last_fullsized":"2024-02-21T23:02:43.418962+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:10:48.186239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.596722+0000","last_change":"2024-02-21T23:02:38.394292+0000","last_active":"2024-02-21T23:02:43.596722+0000","last_peered":"2024-02-21T23:02:43.596722+0000","last_clean":"2024-02-21T23:02:43.596722+0000","last_became_active":"2024-02-21T23:02:38.394006+0000","last_became_peered":"2024-02-21T23:02:38.394006+0000","last_unstale":"2024-02-21T23:02:43.596722+0000","last_undegraded":"2024-02-21T23:02:43.596722+0000","last_fullsized":"2024-02-21T23:02:43.596722+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:54:40.962238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.600192+0000","last_change":"2024-02-21T23:02:38.394994+0000","last_active":"2024-02-21T23:02:43.600192+0000","last_peered":"2024-02-21T23:02:43.600192+0000","last_clean":"2024-02-21T23:02:43.600192+0000","last_became_active":"2024-02-21T23:02:38.394693+0000","last_became_peered":"2024-02-21T23:02:38.394693+0000","last_unstale":"2024-02-21T23:02:43.600192+0000","last_undegraded":"2024-02-21T23:02:43.600192+0000","last_fullsized":"2024-02-21T23:02:43.600192+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:00:09.172059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.597595+0000","last_change":"2024-02-21T23:02:38.397147+0000","last_active":"2024-02-21T23:02:43.597595+0000","last_peered":"2024-02-21T23:02:43.597595+0000","last_clean":"2024-02-21T23:02:43.597595+0000","last_became_active":"2024-02-21T23:02:38.396787+0000","last_became_peered":"2024-02-21T23:02:38.396787+0000","last_unstale":"2024-02-21T23:02:43.597595+0000","last_undegraded":"2024-02-21T23:02:43.597595+0000","last_fullsized":"2024-02-21T23:02:43.597595+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:35:27.966492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.600709+0000","last_change":"2024-02-21T23:02:38.398132+0000","last_active":"2024-02-21T23:02:43.600709+0000","last_peered":"2024-02-21T23:02:43.600709+0000","last_clean":"2024-02-21T23:02:43.600709+0000","last_became_active":"2024-02-21T23:02:38.397847+0000","last_became_peered":"2024-02-21T23:02:38.397847+0000","last_unstale":"2024-02-21T23:02:43.600709+0000","last_undegraded":"2024-02-21T23:02:43.600709+0000","last_fullsized":"2024-02-21T23:02:43.600709+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:28:23.752647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.596548+0000","last_change":"2024-02-21T23:02:38.372038+0000","last_active":"2024-02-21T23:02:43.596548+0000","last_peered":"2024-02-21T23:02:43.596548+0000","last_clean":"2024-02-21T23:02:43.596548+0000","last_became_active":"2024-02-21T23:02:38.371731+0000","last_became_peered":"2024-02-21T23:02:38.371731+0000","last_unstale":"2024-02-21T23:02:43.596548+0000","last_undegraded":"2024-02-21T23:02:43.596548+0000","last_fullsized":"2024-02-21T23:02:43.596548+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:04:11.892160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.598184+0000","last_change":"2024-02-21T23:02:38.379292+0000","last_active":"2024-02-21T23:02:43.598184+0000","last_peered":"2024-02-21T23:02:43.598184+0000","last_clean":"2024-02-21T23:02:43.598184+0000","last_became_active":"2024-02-21T23:02:38.378980+0000","last_became_peered":"2024-02-21T23:02:38.378980+0000","last_unstale":"2024-02-21T23:02:43.598184+0000","last_undegraded":"2024-02-21T23:02:43.598184+0000","last_fullsized":"2024-02-21T23:02:43.598184+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:29:12.483912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.598604+0000","last_change":"2024-02-21T23:02:38.396837+0000","last_active":"2024-02-21T23:02:43.598604+0000","last_peered":"2024-02-21T23:02:43.598604+0000","last_clean":"2024-02-21T23:02:43.598604+0000","last_became_active":"2024-02-21T23:02:38.396502+0000","last_became_peered":"2024-02-21T23:02:38.396502+0000","last_unstale":"2024-02-21T23:02:43.598604+0000","last_undegraded":"2024-02-21T23:02:43.598604+0000","last_fullsized":"2024-02-21T23:02:43.598604+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:54:16.651108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27340,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":323584,"data_stored":691206,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":258048,"data_stored":100346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":266240,"data_stored":100382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574858,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":253952,"data_stored":100256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":262144,"data_stored":100416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31508,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340332,"statfs":{"total":96636764160,"available":96604499968,"internally_reserved":0,"allocated":331776,"data_stored":691262,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-21T23:03:08.770 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-02-21T23:03:08.770 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-02-21T23:03:08.770 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-02-21T23:03:08.771 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-02-21T23:03:08.771 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-02-21T23:03:09.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.048+0000 7f5ea2000640 1 Processor -- start 2024-02-21T23:03:09.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.048+0000 7f5ea2000640 1 -- start start 2024-02-21T23:03:09.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.049+0000 7f5ea2000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5e9c0a5fd0 0x7f5e9c06dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:09.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.049+0000 7f5ea2000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e9c0c2760 con 0x7f5e9c06ee40 2024-02-21T23:03:09.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.049+0000 7f5ea2000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e9c156310 con 0x7f5e9c150450 2024-02-21T23:03:09.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.049+0000 7f5ea2000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e9c0c22c0 con 0x7f5e9c0a5fd0 2024-02-21T23:03:09.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.050+0000 7f5e9be00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f5e9c06ee40 0x7f5e9c156af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:48528/0 (socket says 172.21.15.28:48528) 2024-02-21T23:03:09.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.050+0000 7f5e9be00640 1 -- 172.21.15.28:0/1245736484 learned_addr learned my addr 172.21.15.28:0/1245736484 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:03:09.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.050+0000 7f5ea1600640 1 --2- 172.21.15.28:0/1245736484 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5e9c0a5fd0 0x7f5e9c06dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:09.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.050+0000 7f5e9b400640 1 -- 172.21.15.28:0/1245736484 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 68366586 0 0) 0x7f5e9c0c2760 con 0x7f5e9c06ee40 2024-02-21T23:03:09.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.050+0000 7f5e9b400640 1 -- 172.21.15.28:0/1245736484 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e84003520 con 0x7f5e9c06ee40 2024-02-21T23:03:09.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.050+0000 7f5e9b400640 1 -- 172.21.15.28:0/1245736484 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2175179895 0 0) 0x7f5e84003520 con 0x7f5e9c06ee40 2024-02-21T23:03:09.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.050+0000 7f5e9b400640 1 -- 172.21.15.28:0/1245736484 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5e9c0a5fd0 msgr2=0x7f5e9c06dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:09.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.050+0000 7f5e9b400640 1 --2- 172.21.15.28:0/1245736484 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5e9c0a5fd0 0x7f5e9c06dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:09.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.051+0000 7f5e9b400640 1 -- 172.21.15.28:0/1245736484 >> v1:172.21.15.81:6789/0 conn(0x7f5e9c150450 legacy=0x7f5e9c1536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:09.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.051+0000 7f5e9b400640 1 -- 172.21.15.28:0/1245736484 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e9c0c29d0 con 0x7f5e9c06ee40 2024-02-21T23:03:09.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.051+0000 7f5e9b400640 1 -- 172.21.15.28:0/1245736484 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f5e88001670 con 0x7f5e9c06ee40 2024-02-21T23:03:09.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.051+0000 7f5e9b400640 1 -- 172.21.15.28:0/1245736484 >> v1:172.21.15.28:6789/0 conn(0x7f5e9c06ee40 legacy=0x7f5e9c156af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:09.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.051+0000 7f5e9b400640 1 --2- 172.21.15.28:0/1245736484 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5e9c0a5fd0 0x7f5e840039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:09.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.051+0000 7f5e9b400640 1 --2- 172.21.15.28:0/1245736484 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e84003fb0 0x7f5e84004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:09.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.051+0000 7f5e9b400640 1 --2- 172.21.15.28:0/1245736484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e84004d40 0x7f5e840051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:09.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.051+0000 7f5e9b400640 1 -- 172.21.15.28:0/1245736484 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e8400bad0 con 0x7f5e84004d40 2024-02-21T23:03:09.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.051+0000 7f5e9b400640 1 -- 172.21.15.28:0/1245736484 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e8400bc80 con 0x7f5e84003fb0 2024-02-21T23:03:09.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.051+0000 7f5e9b400640 1 -- 172.21.15.28:0/1245736484 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e8400be80 con 0x7f5e9c0a5fd0 2024-02-21T23:03:09.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.051+0000 7f5ea1600640 1 --2- 172.21.15.28:0/1245736484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e84004d40 0x7f5e840051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:09.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.051+0000 7f5ea0c00640 1 --2- 172.21.15.28:0/1245736484 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5e9c0a5fd0 0x7f5e840039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:09.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.051+0000 7f5e9be00640 1 --2- 172.21.15.28:0/1245736484 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e84003fb0 0x7f5e84004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:09.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.051+0000 7f5ea1600640 1 -- 172.21.15.28:0/1245736484 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5e9c0a5fd0 msgr2=0x7f5e840039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:09.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.051+0000 7f5ea1600640 1 --2- 172.21.15.28:0/1245736484 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5e9c0a5fd0 0x7f5e840039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:09.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.052+0000 7f5ea1600640 1 -- 172.21.15.28:0/1245736484 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e84003fb0 msgr2=0x7f5e84004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:09.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.052+0000 7f5ea1600640 1 --2- 172.21.15.28:0/1245736484 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e84003fb0 0x7f5e84004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:09.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.052+0000 7f5ea1600640 1 -- 172.21.15.28:0/1245736484 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e84003520 con 0x7f5e84004d40 2024-02-21T23:03:09.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.053+0000 7f5ea1600640 1 --2- 172.21.15.28:0/1245736484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e84004d40 0x7f5e840051c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f5e94009fc0 tx=0x7f5e84005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e26b8c11fbc76d97 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:09.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.053+0000 7f5e9b400640 1 -- 172.21.15.28:0/1245736484 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e94039070 con 0x7f5e84004d40 2024-02-21T23:03:09.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.053+0000 7f5e9b400640 1 -- 172.21.15.28:0/1245736484 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e9402cc20 con 0x7f5e84004d40 2024-02-21T23:03:09.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.053+0000 7f5e9b400640 1 -- 172.21.15.28:0/1245736484 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e94009c10 con 0x7f5e84004d40 2024-02-21T23:03:09.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.053+0000 7f5ea2000640 1 -- 172.21.15.28:0/1245736484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e84004d40 msgr2=0x7f5e840051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:09.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.053+0000 7f5ea2000640 1 --2- 172.21.15.28:0/1245736484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e84004d40 0x7f5e840051c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f5e94009fc0 tx=0x7f5e84005830 comp rx=0 tx=0).stop 2024-02-21T23:03:09.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.053+0000 7f5ea1600640 1 -- 172.21.15.28:0/1245736484 reap_dead start 2024-02-21T23:03:09.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.054+0000 7f5ea2000640 1 -- 172.21.15.28:0/1245736484 shutdown_connections 2024-02-21T23:03:09.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.054+0000 7f5ea2000640 1 -- 172.21.15.28:0/1245736484 >> 172.21.15.28:0/1245736484 conn(0x7f5e9c06b630 msgr2=0x7f5e9c0c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:09.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.054+0000 7f5ea2000640 1 -- 172.21.15.28:0/1245736484 shutdown_connections 2024-02-21T23:03:09.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.054+0000 7f5ea2000640 1 -- 172.21.15.28:0/1245736484 wait complete. 2024-02-21T23:03:09.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.055+0000 7f5ea2000640 1 Processor -- start 2024-02-21T23:03:09.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.056+0000 7f5ea2000640 1 -- start start 2024-02-21T23:03:09.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.056+0000 7f5ea2000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e84004d40 0x7f5e9c14ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:09.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.056+0000 7f5ea2000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5e9c0a9880 0x7f5e9c155220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:09.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.056+0000 7f5ea2000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e9c0a93a0 0x7f5e9c1e8970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:09.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.056+0000 7f5ea2000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e9c157200 con 0x7f5e84004d40 2024-02-21T23:03:09.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.056+0000 7f5ea2000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e9c156660 con 0x7f5e9c0a93a0 2024-02-21T23:03:09.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.056+0000 7f5ea2000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e9c06dff0 con 0x7f5e9c0a9880 2024-02-21T23:03:09.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.057+0000 7f5ea0c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e84004d40 0x7f5e9c14ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:09.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.057+0000 7f5ea0c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e84004d40 0x7f5e9c14ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45932/0 (socket says 172.21.15.28:45932) 2024-02-21T23:03:09.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.057+0000 7f5ea0c00640 1 -- 172.21.15.28:0/1829048787 learned_addr learned my addr 172.21.15.28:0/1829048787 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:09.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.057+0000 7f5ea1600640 1 --2- 172.21.15.28:0/1829048787 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e9c0a93a0 0x7f5e9c1e8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:09.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.057+0000 7f5e9be00640 1 --2- 172.21.15.28:0/1829048787 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5e9c0a9880 0x7f5e9c155220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:09.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.057+0000 7f5ea0c00640 1 -- 172.21.15.28:0/1829048787 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5e9c0a9880 msgr2=0x7f5e9c155220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:09.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.057+0000 7f5ea0c00640 1 --2- 172.21.15.28:0/1829048787 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5e9c0a9880 0x7f5e9c155220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:09.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.057+0000 7f5ea0c00640 1 -- 172.21.15.28:0/1829048787 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e9c0a93a0 msgr2=0x7f5e9c1e8970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:09.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.057+0000 7f5ea0c00640 1 --2- 172.21.15.28:0/1829048787 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e9c0a93a0 0x7f5e9c1e8970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:09.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.057+0000 7f5ea0c00640 1 -- 172.21.15.28:0/1829048787 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e9c0aa900 con 0x7f5e84004d40 2024-02-21T23:03:09.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.057+0000 7f5e9be00640 1 --2- 172.21.15.28:0/1829048787 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5e9c0a9880 0x7f5e9c155220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:03:09.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.058+0000 7f5ea0c00640 1 --2- 172.21.15.28:0/1829048787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e84004d40 0x7f5e9c14ca40 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f5e9000e570 tx=0x7f5e9000ea30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:09.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.058+0000 7f5e98c00640 1 -- 172.21.15.28:0/1829048787 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e900052e0 con 0x7f5e84004d40 2024-02-21T23:03:09.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.058+0000 7f5e98c00640 1 -- 172.21.15.28:0/1829048787 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e90018070 con 0x7f5e84004d40 2024-02-21T23:03:09.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.058+0000 7f5e98c00640 1 -- 172.21.15.28:0/1829048787 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e90003830 con 0x7f5e84004d40 2024-02-21T23:03:09.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.058+0000 7f5ea2000640 1 -- 172.21.15.28:0/1829048787 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e9c0ac130 con 0x7f5e84004d40 2024-02-21T23:03:09.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.058+0000 7f5ea2000640 1 -- 172.21.15.28:0/1829048787 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e9c155760 con 0x7f5e84004d40 2024-02-21T23:03:09.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.059+0000 7f5ea2000640 1 -- 172.21.15.28:0/1829048787 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e9c04f290 con 0x7f5e84004d40 2024-02-21T23:03:09.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.060+0000 7f5e98c00640 1 -- 172.21.15.28:0/1829048787 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5e900039d0 con 0x7f5e84004d40 2024-02-21T23:03:09.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.061+0000 7f5e98c00640 1 --2- 172.21.15.28:0/1829048787 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5e7c06b500 0x7f5e7c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:09.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.061+0000 7f5e98c00640 1 -- 172.21.15.28:0/1829048787 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f5e9008a2d0 con 0x7f5e84004d40 2024-02-21T23:03:09.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.062+0000 7f5e9be00640 1 --2- 172.21.15.28:0/1829048787 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5e7c06b500 0x7f5e7c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:09.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.063+0000 7f5e9be00640 1 --2- 172.21.15.28:0/1829048787 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5e7c06b500 0x7f5e7c06d9c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5e9c0a6450 tx=0x7f5e88000f70 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:09.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.066+0000 7f5e98c00640 1 -- 172.21.15.28:0/1829048787 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f5e90058d70 con 0x7f5e84004d40 2024-02-21T23:03:09.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.314+0000 7f5ea2000640 1 -- 172.21.15.28:0/1829048787 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f5e9c0c1f10 con 0x7f5e84004d40 2024-02-21T23:03:09.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.323+0000 7f5e98c00640 1 -- 172.21.15.28:0/1829048787 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+171784 (secure 0 0 0) 0x7f5e90058670 con 0x7f5e84004d40 2024-02-21T23:03:09.315 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:03:09.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.328+0000 7f5ea2000640 1 -- 172.21.15.28:0/1829048787 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5e7c06b500 msgr2=0x7f5e7c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:09.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.328+0000 7f5ea2000640 1 --2- 172.21.15.28:0/1829048787 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5e7c06b500 0x7f5e7c06d9c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5e9c0a6450 tx=0x7f5e88000f70 comp rx=0 tx=0).stop 2024-02-21T23:03:09.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.328+0000 7f5ea2000640 1 -- 172.21.15.28:0/1829048787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e84004d40 msgr2=0x7f5e9c14ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:09.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.328+0000 7f5ea2000640 1 --2- 172.21.15.28:0/1829048787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e84004d40 0x7f5e9c14ca40 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f5e9000e570 tx=0x7f5e9000ea30 comp rx=0 tx=0).stop 2024-02-21T23:03:09.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.328+0000 7f5ea2000640 1 -- 172.21.15.28:0/1829048787 shutdown_connections 2024-02-21T23:03:09.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.328+0000 7f5ea2000640 1 --2- 172.21.15.28:0/1829048787 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5e7c06b500 0x7f5e7c06d9c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:09.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.328+0000 7f5ea2000640 1 --2- 172.21.15.28:0/1829048787 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5e9c0a9880 0x7f5e9c155220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:09.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.328+0000 7f5ea2000640 1 --2- 172.21.15.28:0/1829048787 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e9c0a93a0 0x7f5e9c1e8970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:09.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.328+0000 7f5ea2000640 1 --2- 172.21.15.28:0/1829048787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e84004d40 0x7f5e9c14ca40 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:09.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.328+0000 7f5ea2000640 1 -- 172.21.15.28:0/1829048787 >> 172.21.15.28:0/1829048787 conn(0x7f5e9c06b630 msgr2=0x7f5e9c0c2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:09.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.329+0000 7f5ea2000640 1 -- 172.21.15.28:0/1829048787 shutdown_connections 2024-02-21T23:03:09.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.329+0000 7f5ea2000640 1 -- 172.21.15.28:0/1829048787 wait complete. 2024-02-21T23:03:09.337 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":5,"flags":0,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1848538577},{"type":"v1","addr":"172.21.15.81:6833","nonce":1848538577}]},"active_addr":"172.21.15.81:6833/1848538577","active_change":"2024-02-21T23:02:22.635336+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4104,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.81:0","nonce":304310310}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.81:0","nonce":1600676998}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.81:0","nonce":1840618014}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.81:0","nonce":2680805287}]}]} 2024-02-21T23:03:09.341 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-02-21T23:03:09.341 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-02-21T23:03:09.341 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:03:09.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.580+0000 7fb81d800640 1 Processor -- start 2024-02-21T23:03:09.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.580+0000 7fb81d800640 1 -- start start 2024-02-21T23:03:09.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.582+0000 7fb81d800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb818150450 0x7fb8180a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:09.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.582+0000 7fb81d800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb818156af0 0x7fb81806dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:09.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.582+0000 7fb81d800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8180c2760 con 0x7fb81806ee40 2024-02-21T23:03:09.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.582+0000 7fb81d800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8180c22c0 con 0x7fb818156af0 2024-02-21T23:03:09.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.582+0000 7fb81d800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb818156660 con 0x7fb818150450 2024-02-21T23:03:09.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.582+0000 7fb81ce00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb818156af0 0x7fb81806dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:09.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.582+0000 7fb81ce00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb818156af0 0x7fb81806dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45940/0 (socket says 172.21.15.28:45940) 2024-02-21T23:03:09.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.582+0000 7fb817400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb81806ee40 0x7fb8181536f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:48540/0 (socket says 172.21.15.28:48540) 2024-02-21T23:03:09.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.582+0000 7fb81ce00640 1 -- 172.21.15.28:0/319583956 learned_addr learned my addr 172.21.15.28:0/319583956 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:09.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.582+0000 7fb817e00640 1 --2- 172.21.15.28:0/319583956 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb818150450 0x7fb8180a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:09.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.583+0000 7fb81ce00640 1 -- 172.21.15.28:0/319583956 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb818150450 msgr2=0x7fb8180a5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:09.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.583+0000 7fb81ce00640 1 --2- 172.21.15.28:0/319583956 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb818150450 0x7fb8180a5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:09.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.583+0000 7fb81ce00640 1 -- 172.21.15.28:0/319583956 >> v1:172.21.15.28:6789/0 conn(0x7fb81806ee40 legacy=0x7fb8181536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:09.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.583+0000 7fb81ce00640 1 -- 172.21.15.28:0/319583956 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb818156310 con 0x7fb818156af0 2024-02-21T23:03:09.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.583+0000 7fb816a00640 1 -- 172.21.15.28:0/319583956 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4024634634 0 0) 0x7fb8180c2760 con 0x7fb81806ee40 2024-02-21T23:03:09.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.584+0000 7fb81ce00640 1 --2- 172.21.15.28:0/319583956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb818156af0 0x7fb81806dff0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb810009e50 tx=0x7fb81002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40113224d1a2745d server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:09.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.584+0000 7fb816a00640 1 -- 172.21.15.28:0/319583956 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb81002cb00 con 0x7fb818156af0 2024-02-21T23:03:09.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.584+0000 7fb816a00640 1 -- 172.21.15.28:0/319583956 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb81002cca0 con 0x7fb818156af0 2024-02-21T23:03:09.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.584+0000 7fb816a00640 1 -- 172.21.15.28:0/319583956 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb810034610 con 0x7fb818156af0 2024-02-21T23:03:09.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.585+0000 7fb81d800640 1 -- 172.21.15.28:0/319583956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb818156af0 msgr2=0x7fb81806dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:09.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.585+0000 7fb81d800640 1 --2- 172.21.15.28:0/319583956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb818156af0 0x7fb81806dff0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb810009e50 tx=0x7fb81002bf60 comp rx=0 tx=0).stop 2024-02-21T23:03:09.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.585+0000 7fb81d800640 1 -- 172.21.15.28:0/319583956 shutdown_connections 2024-02-21T23:03:09.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.585+0000 7fb81d800640 1 --2- 172.21.15.28:0/319583956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb818156af0 0x7fb81806dff0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:09.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.585+0000 7fb81d800640 1 --2- 172.21.15.28:0/319583956 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb818150450 0x7fb8180a5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:09.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.585+0000 7fb81d800640 1 -- 172.21.15.28:0/319583956 >> 172.21.15.28:0/319583956 conn(0x7fb81806b630 msgr2=0x7fb8180c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:09.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.585+0000 7fb81d800640 1 -- 172.21.15.28:0/319583956 shutdown_connections 2024-02-21T23:03:09.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.585+0000 7fb81d800640 1 -- 172.21.15.28:0/319583956 wait complete. 2024-02-21T23:03:09.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.587+0000 7fb81d800640 1 Processor -- start 2024-02-21T23:03:09.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.587+0000 7fb81d800640 1 -- start start 2024-02-21T23:03:09.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.588+0000 7fb81d800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb81806ee40 0x7fb8180c5220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:09.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.588+0000 7fb81d800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb818150450 0x7fb8180bfba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:09.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.588+0000 7fb81d800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb818156af0 0x7fb8181ea5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:09.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.588+0000 7fb81d800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb818156f80 con 0x7fb818156af0 2024-02-21T23:03:09.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.588+0000 7fb81d800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8180c2cb0 con 0x7fb81806ee40 2024-02-21T23:03:09.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.588+0000 7fb81d800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb818157100 con 0x7fb818150450 2024-02-21T23:03:09.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.588+0000 7fb81ce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb818156af0 0x7fb8181ea5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:09.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.588+0000 7fb81ce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb818156af0 0x7fb8181ea5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45956/0 (socket says 172.21.15.28:45956) 2024-02-21T23:03:09.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.588+0000 7fb81ce00640 1 -- 172.21.15.28:0/1615994484 learned_addr learned my addr 172.21.15.28:0/1615994484 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:09.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.588+0000 7fb817e00640 1 --2- 172.21.15.28:0/1615994484 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb81806ee40 0x7fb8180c5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:09.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.589+0000 7fb817400640 1 --2- 172.21.15.28:0/1615994484 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb818150450 0x7fb8180bfba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:09.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.589+0000 7fb81ce00640 1 -- 172.21.15.28:0/1615994484 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb818150450 msgr2=0x7fb8180bfba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:09.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.589+0000 7fb81ce00640 1 --2- 172.21.15.28:0/1615994484 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb818150450 0x7fb8180bfba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:09.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.589+0000 7fb81ce00640 1 -- 172.21.15.28:0/1615994484 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb81806ee40 msgr2=0x7fb8180c5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:09.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.589+0000 7fb81ce00640 1 --2- 172.21.15.28:0/1615994484 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb81806ee40 0x7fb8180c5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:09.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.589+0000 7fb81ce00640 1 -- 172.21.15.28:0/1615994484 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb818072260 con 0x7fb818156af0 2024-02-21T23:03:09.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.589+0000 7fb81ce00640 1 --2- 172.21.15.28:0/1615994484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb818156af0 0x7fb8181ea5a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb810036040 tx=0x7fb810004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:09.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.589+0000 7fb80fe00640 1 -- 172.21.15.28:0/1615994484 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb810044020 con 0x7fb818156af0 2024-02-21T23:03:09.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.589+0000 7fb80fe00640 1 -- 172.21.15.28:0/1615994484 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb81002ce50 con 0x7fb818156af0 2024-02-21T23:03:09.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.589+0000 7fb80fe00640 1 -- 172.21.15.28:0/1615994484 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb810009510 con 0x7fb818156af0 2024-02-21T23:03:09.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.589+0000 7fb81d800640 1 -- 172.21.15.28:0/1615994484 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8180c5900 con 0x7fb818156af0 2024-02-21T23:03:09.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.590+0000 7fb81d800640 1 -- 172.21.15.28:0/1615994484 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8181edf90 con 0x7fb818156af0 2024-02-21T23:03:09.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.591+0000 7fb81d800640 1 -- 172.21.15.28:0/1615994484 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb81804f290 con 0x7fb818156af0 2024-02-21T23:03:09.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.592+0000 7fb80fe00640 1 -- 172.21.15.28:0/1615994484 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb810005d10 con 0x7fb818156af0 2024-02-21T23:03:09.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.593+0000 7fb80fe00640 1 --2- 172.21.15.28:0/1615994484 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb7f806b4d0 0x7fb7f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:09.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.593+0000 7fb80fe00640 1 -- 172.21.15.28:0/1615994484 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fb8100acfd0 con 0x7fb818156af0 2024-02-21T23:03:09.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.594+0000 7fb817e00640 1 --2- 172.21.15.28:0/1615994484 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb7f806b4d0 0x7fb7f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:09.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.595+0000 7fb817e00640 1 --2- 172.21.15.28:0/1615994484 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb7f806b4d0 0x7fb7f806d990 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb8180c6060 tx=0x7fb808008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:09.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.598+0000 7fb80fe00640 1 -- 172.21.15.28:0/1615994484 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb81007bba0 con 0x7fb818156af0 2024-02-21T23:03:09.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.801+0000 7fb81d800640 1 -- 172.21.15.28:0/1615994484 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb8180ac890 con 0x7fb818156af0 2024-02-21T23:03:09.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.803+0000 7fb80fe00640 1 -- 172.21.15.28:0/1615994484 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18454 (secure 0 0 0) 0x7fb81007b4a0 con 0x7fb818156af0 2024-02-21T23:03:09.792 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:03:09.792 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":24,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:02:42.900286+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:02:35.795158+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-21T23:02:36.840110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-02-21T23:02:40.441388+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:03:09.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.805+0000 7fb81d800640 1 -- 172.21.15.28:0/1615994484 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb7f806b4d0 msgr2=0x7fb7f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:09.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.805+0000 7fb81d800640 1 --2- 172.21.15.28:0/1615994484 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb7f806b4d0 0x7fb7f806d990 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb8180c6060 tx=0x7fb808008040 comp rx=0 tx=0).stop 2024-02-21T23:03:09.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.805+0000 7fb81d800640 1 -- 172.21.15.28:0/1615994484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb818156af0 msgr2=0x7fb8181ea5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:09.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.805+0000 7fb81d800640 1 --2- 172.21.15.28:0/1615994484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb818156af0 0x7fb8181ea5a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb810036040 tx=0x7fb810004a40 comp rx=0 tx=0).stop 2024-02-21T23:03:09.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.805+0000 7fb81d800640 1 -- 172.21.15.28:0/1615994484 shutdown_connections 2024-02-21T23:03:09.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.805+0000 7fb81d800640 1 --2- 172.21.15.28:0/1615994484 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb7f806b4d0 0x7fb7f806d990 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:09.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.805+0000 7fb81d800640 1 --2- 172.21.15.28:0/1615994484 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb818150450 0x7fb8180bfba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:09.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.805+0000 7fb81d800640 1 --2- 172.21.15.28:0/1615994484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb818156af0 0x7fb8181ea5a0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:09.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.805+0000 7fb81d800640 1 --2- 172.21.15.28:0/1615994484 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb81806ee40 0x7fb8180c5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:09.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.806+0000 7fb81d800640 1 -- 172.21.15.28:0/1615994484 >> 172.21.15.28:0/1615994484 conn(0x7fb81806b630 msgr2=0x7fb81814dc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:09.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.806+0000 7fb81d800640 1 -- 172.21.15.28:0/1615994484 shutdown_connections 2024-02-21T23:03:09.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:09.806+0000 7fb81d800640 1 -- 172.21.15.28:0/1615994484 wait complete. 2024-02-21T23:03:09.813 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-02-21T23:03:09.814 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:03:10.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.042+0000 7f2a0a600640 1 Processor -- start 2024-02-21T23:03:10.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.042+0000 7f2a0a600640 1 -- start start 2024-02-21T23:03:10.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.043+0000 7f2a0a600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2a0406dff0 0x7f2a0406e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:10.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.043+0000 7f2a0a600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f2a040c0fe0 0x7f2a04071350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:10.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.043+0000 7f2a0a600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a040a8b40 con 0x7f2a04158580 2024-02-21T23:03:10.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.043+0000 7f2a0a600640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a040a8e00 con 0x7f2a040c0fe0 2024-02-21T23:03:10.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.043+0000 7f2a0a600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a040a90e0 con 0x7f2a0406dff0 2024-02-21T23:03:10.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.043+0000 7f2a09c00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f2a04158580 0x7f2a0414b2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:48542/0 (socket says 172.21.15.28:48542) 2024-02-21T23:03:10.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.043+0000 7f2a09c00640 1 -- 172.21.15.28:0/3048679725 learned_addr learned my addr 172.21.15.28:0/3048679725 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:03:10.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.044+0000 7f2a03e00640 1 --2- 172.21.15.28:0/3048679725 >> v2:172.21.15.81:3300/0 conn(0x7f2a040c0fe0 0x7f2a04071350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:10.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.044+0000 7f2a09200640 1 --2- 172.21.15.28:0/3048679725 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2a0406dff0 0x7f2a0406e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:10.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.044+0000 7f2a03400640 1 -- 172.21.15.28:0/3048679725 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3663579802 0 0) 0x7f2a040a8b40 con 0x7f2a04158580 2024-02-21T23:03:10.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.044+0000 7f2a03400640 1 -- 172.21.15.28:0/3048679725 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29e8003520 con 0x7f2a04158580 2024-02-21T23:03:10.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.044+0000 7f2a03400640 1 -- 172.21.15.28:0/3048679725 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3928709810 0 0) 0x7f29e8003520 con 0x7f2a04158580 2024-02-21T23:03:10.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.044+0000 7f2a03400640 1 -- 172.21.15.28:0/3048679725 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2a0406dff0 msgr2=0x7f2a0406e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:10.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.044+0000 7f2a03400640 1 --2- 172.21.15.28:0/3048679725 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2a0406dff0 0x7f2a0406e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:10.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.044+0000 7f2a03400640 1 -- 172.21.15.28:0/3048679725 >> v2:172.21.15.81:3300/0 conn(0x7f2a040c0fe0 msgr2=0x7f2a04071350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:10.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.044+0000 7f2a03400640 1 --2- 172.21.15.28:0/3048679725 >> v2:172.21.15.81:3300/0 conn(0x7f2a040c0fe0 0x7f2a04071350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:10.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.044+0000 7f2a03400640 1 -- 172.21.15.28:0/3048679725 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a04072ed0 con 0x7f2a04158580 2024-02-21T23:03:10.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.044+0000 7f2a03400640 1 -- 172.21.15.28:0/3048679725 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f29fc001670 con 0x7f2a04158580 2024-02-21T23:03:10.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.044+0000 7f2a03400640 1 -- 172.21.15.28:0/3048679725 >> v1:172.21.15.28:6789/0 conn(0x7f2a04158580 legacy=0x7f2a0414b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:10.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.044+0000 7f2a03400640 1 --2- 172.21.15.28:0/3048679725 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29e80039b0 0x7f29e8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:10.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.044+0000 7f2a03400640 1 --2- 172.21.15.28:0/3048679725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29e8004800 0x7f29e8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:10.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.044+0000 7f2a03400640 1 --2- 172.21.15.28:0/3048679725 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2a0406dff0 0x7f29e80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:10.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.045+0000 7f2a03400640 1 -- 172.21.15.28:0/3048679725 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a040a8e00 con 0x7f29e8004800 2024-02-21T23:03:10.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.045+0000 7f2a03400640 1 -- 172.21.15.28:0/3048679725 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a040a90e0 con 0x7f29e80039b0 2024-02-21T23:03:10.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.045+0000 7f2a03400640 1 -- 172.21.15.28:0/3048679725 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29e800b9c0 con 0x7f2a0406dff0 2024-02-21T23:03:10.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.045+0000 7f2a03e00640 1 --2- 172.21.15.28:0/3048679725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29e8004800 0x7f29e8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:10.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.045+0000 7f2a09200640 1 --2- 172.21.15.28:0/3048679725 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29e80039b0 0x7f29e8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:10.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.045+0000 7f2a09c00640 1 --2- 172.21.15.28:0/3048679725 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2a0406dff0 0x7f29e80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:10.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.045+0000 7f2a03e00640 1 -- 172.21.15.28:0/3048679725 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2a0406dff0 msgr2=0x7f29e80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:10.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.045+0000 7f2a03e00640 1 --2- 172.21.15.28:0/3048679725 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2a0406dff0 0x7f29e80051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:10.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.045+0000 7f2a03e00640 1 -- 172.21.15.28:0/3048679725 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29e80039b0 msgr2=0x7f29e8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:10.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.045+0000 7f2a03e00640 1 --2- 172.21.15.28:0/3048679725 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29e80039b0 0x7f29e8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:10.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.045+0000 7f2a03e00640 1 -- 172.21.15.28:0/3048679725 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29e8003520 con 0x7f29e8004800 2024-02-21T23:03:10.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.045+0000 7f2a09200640 1 --2- 172.21.15.28:0/3048679725 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29e80039b0 0x7f29e8003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:03:10.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.046+0000 7f2a03e00640 1 --2- 172.21.15.28:0/3048679725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29e8004800 0x7f29e8004c80 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f29f0009ac0 tx=0x7f29e8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1025294eaee16e67 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:10.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.046+0000 7f2a03400640 1 -- 172.21.15.28:0/3048679725 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f0039070 con 0x7f29e8004800 2024-02-21T23:03:10.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.047+0000 7f2a03400640 1 -- 172.21.15.28:0/3048679725 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29f002c6d0 con 0x7f29e8004800 2024-02-21T23:03:10.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.047+0000 7f2a03400640 1 -- 172.21.15.28:0/3048679725 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f0009690 con 0x7f29e8004800 2024-02-21T23:03:10.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.047+0000 7f2a0a600640 1 -- 172.21.15.28:0/3048679725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29e8004800 msgr2=0x7f29e8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:10.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.047+0000 7f2a0a600640 1 --2- 172.21.15.28:0/3048679725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29e8004800 0x7f29e8004c80 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f29f0009ac0 tx=0x7f29e8005830 comp rx=0 tx=0).stop 2024-02-21T23:03:10.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.047+0000 7f2a09c00640 1 -- 172.21.15.28:0/3048679725 reap_dead start 2024-02-21T23:03:10.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.047+0000 7f2a0a600640 1 -- 172.21.15.28:0/3048679725 shutdown_connections 2024-02-21T23:03:10.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.047+0000 7f2a0a600640 1 -- 172.21.15.28:0/3048679725 >> 172.21.15.28:0/3048679725 conn(0x7f2a0406b630 msgr2=0x7f2a040c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:10.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.047+0000 7f2a0a600640 1 -- 172.21.15.28:0/3048679725 shutdown_connections 2024-02-21T23:03:10.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.047+0000 7f2a0a600640 1 -- 172.21.15.28:0/3048679725 wait complete. 2024-02-21T23:03:10.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.049+0000 7f2a0a600640 1 Processor -- start 2024-02-21T23:03:10.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.049+0000 7f2a0a600640 1 -- start start 2024-02-21T23:03:10.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.050+0000 7f2a0a600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f29e8004800 0x7f2a04153810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:10.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.050+0000 7f2a0a600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a041e1bc0 0x7f2a040ab7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:10.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.050+0000 7f2a0a600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2a041ee310 0x7f2a041ee7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:10.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.050+0000 7f2a0a600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a04071890 con 0x7f2a041e1bc0 2024-02-21T23:03:10.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.050+0000 7f2a0a600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a04073520 con 0x7f2a041ee310 2024-02-21T23:03:10.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.050+0000 7f2a0a600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a04158a10 con 0x7f29e8004800 2024-02-21T23:03:10.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.050+0000 7f2a03e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a041e1bc0 0x7f2a040ab7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:10.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.050+0000 7f2a03e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a041e1bc0 0x7f2a040ab7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45960/0 (socket says 172.21.15.28:45960) 2024-02-21T23:03:10.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.050+0000 7f2a03e00640 1 -- 172.21.15.28:0/1264552288 learned_addr learned my addr 172.21.15.28:0/1264552288 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:10.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.051+0000 7f2a09c00640 1 --2- 172.21.15.28:0/1264552288 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2a041ee310 0x7f2a041ee7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:10.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.051+0000 7f2a09200640 1 --2- 172.21.15.28:0/1264552288 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f29e8004800 0x7f2a04153810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:10.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.051+0000 7f2a03e00640 1 -- 172.21.15.28:0/1264552288 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f29e8004800 msgr2=0x7f2a04153810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:10.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.051+0000 7f2a03e00640 1 --2- 172.21.15.28:0/1264552288 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f29e8004800 0x7f2a04153810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:10.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.051+0000 7f2a03e00640 1 -- 172.21.15.28:0/1264552288 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2a041ee310 msgr2=0x7f2a041ee7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:10.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.051+0000 7f2a03e00640 1 --2- 172.21.15.28:0/1264552288 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2a041ee310 0x7f2a041ee7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:10.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.051+0000 7f2a03e00640 1 -- 172.21.15.28:0/1264552288 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a04153d50 con 0x7f2a041e1bc0 2024-02-21T23:03:10.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.051+0000 7f2a03e00640 1 --2- 172.21.15.28:0/1264552288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a041e1bc0 0x7f2a040ab7e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f29f0000c00 tx=0x7f29f0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:10.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.052+0000 7f2a00c00640 1 -- 172.21.15.28:0/1264552288 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f0039050 con 0x7f2a041e1bc0 2024-02-21T23:03:10.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.052+0000 7f2a00c00640 1 -- 172.21.15.28:0/1264552288 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29f0045070 con 0x7f2a041e1bc0 2024-02-21T23:03:10.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.052+0000 7f2a00c00640 1 -- 172.21.15.28:0/1264552288 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f00044e0 con 0x7f2a041e1bc0 2024-02-21T23:03:10.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.052+0000 7f2a0a600640 1 -- 172.21.15.28:0/1264552288 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a041deff0 con 0x7f2a041e1bc0 2024-02-21T23:03:10.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.052+0000 7f2a0a600640 1 -- 172.21.15.28:0/1264552288 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a041eddf0 con 0x7f2a041e1bc0 2024-02-21T23:03:10.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.053+0000 7f2a0a600640 1 -- 172.21.15.28:0/1264552288 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a041df610 con 0x7f2a041e1bc0 2024-02-21T23:03:10.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.054+0000 7f2a00c00640 1 -- 172.21.15.28:0/1264552288 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f29f0003a50 con 0x7f2a041e1bc0 2024-02-21T23:03:10.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.055+0000 7f2a00c00640 1 --2- 172.21.15.28:0/1264552288 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f29e406b4d0 0x7f29e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:10.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.055+0000 7f2a00c00640 1 -- 172.21.15.28:0/1264552288 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f29f00ac290 con 0x7f2a041e1bc0 2024-02-21T23:03:10.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.056+0000 7f2a09200640 1 --2- 172.21.15.28:0/1264552288 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f29e406b4d0 0x7f29e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:10.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.057+0000 7f2a09200640 1 --2- 172.21.15.28:0/1264552288 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f29e406b4d0 0x7f29e406d990 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f29f4003be0 tx=0x7f29f400b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:10.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.059+0000 7f2a00c00640 1 -- 172.21.15.28:0/1264552288 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f29f004e020 con 0x7f2a041e1bc0 2024-02-21T23:03:10.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.256+0000 7f2a0a600640 1 -- 172.21.15.28:0/1264552288 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2a04156160 con 0x7f2a041e1bc0 2024-02-21T23:03:10.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.259+0000 7f2a00c00640 1 -- 172.21.15.28:0/1264552288 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18454 (secure 0 0 0) 0x7f2a04153d50 con 0x7f2a041e1bc0 2024-02-21T23:03:10.248 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:03:10.249 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":24,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:02:42.900286+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:02:35.795158+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-21T23:02:36.840110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-02-21T23:02:40.441388+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:03:10.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.261+0000 7f2a0a600640 1 -- 172.21.15.28:0/1264552288 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f29e406b4d0 msgr2=0x7f29e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:10.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.261+0000 7f2a0a600640 1 --2- 172.21.15.28:0/1264552288 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f29e406b4d0 0x7f29e406d990 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f29f4003be0 tx=0x7f29f400b040 comp rx=0 tx=0).stop 2024-02-21T23:03:10.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.261+0000 7f2a0a600640 1 -- 172.21.15.28:0/1264552288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a041e1bc0 msgr2=0x7f2a040ab7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:10.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.261+0000 7f2a0a600640 1 --2- 172.21.15.28:0/1264552288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a041e1bc0 0x7f2a040ab7e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f29f0000c00 tx=0x7f29f0004730 comp rx=0 tx=0).stop 2024-02-21T23:03:10.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.261+0000 7f2a0a600640 1 -- 172.21.15.28:0/1264552288 shutdown_connections 2024-02-21T23:03:10.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.261+0000 7f2a0a600640 1 --2- 172.21.15.28:0/1264552288 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f29e406b4d0 0x7f29e406d990 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:10.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.261+0000 7f2a0a600640 1 --2- 172.21.15.28:0/1264552288 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2a041ee310 0x7f2a041ee7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:10.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.261+0000 7f2a0a600640 1 --2- 172.21.15.28:0/1264552288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a041e1bc0 0x7f2a040ab7e0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:10.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.261+0000 7f2a0a600640 1 --2- 172.21.15.28:0/1264552288 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f29e8004800 0x7f2a04153810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:10.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.261+0000 7f2a0a600640 1 -- 172.21.15.28:0/1264552288 >> 172.21.15.28:0/1264552288 conn(0x7f2a0406b630 msgr2=0x7f2a040c5eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:10.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.262+0000 7f2a0a600640 1 -- 172.21.15.28:0/1264552288 shutdown_connections 2024-02-21T23:03:10.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.262+0000 7f2a0a600640 1 -- 172.21.15.28:0/1264552288 wait complete. 2024-02-21T23:03:10.269 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-02-21T23:03:10.270 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-02-21T23:03:10.270 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-02-21T23:03:10.271 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-02-21T23:03:10.271 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-02-21T23:03:10.272 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-02-21T23:03:10.272 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-02-21T23:03:10.273 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-02-21T23:03:10.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.585+0000 7f39af400640 1 Processor -- start 2024-02-21T23:03:10.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.586+0000 7f39af400640 1 -- start start 2024-02-21T23:03:10.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.588+0000 7f39af400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f39a806b8b0 0x7f39a8151670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:10.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.588+0000 7f39af400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39a806ec50 con 0x7f39a80a6040 2024-02-21T23:03:10.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.588+0000 7f39af400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39a806f1d0 con 0x7f39a806d390 2024-02-21T23:03:10.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.588+0000 7f39af400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39a8151cc0 con 0x7f39a806b8b0 2024-02-21T23:03:10.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.589+0000 7f39a7e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f39a80a6040 0x7f39a80c2c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:48554/0 (socket says 172.21.15.28:48554) 2024-02-21T23:03:10.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.589+0000 7f39a7e00640 1 -- 172.21.15.28:0/1634146062 learned_addr learned my addr 172.21.15.28:0/1634146062 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:03:10.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.589+0000 7f39aca00640 1 --2- 172.21.15.28:0/1634146062 >> v2:172.21.15.81:3300/0 conn(0x7f39a806b8b0 0x7f39a8151670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:10.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.590+0000 7f39a7400640 1 -- 172.21.15.28:0/1634146062 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 914543309 0 0) 0x7f39a806ec50 con 0x7f39a80a6040 2024-02-21T23:03:10.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.590+0000 7f39a7400640 1 -- 172.21.15.28:0/1634146062 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3990003520 con 0x7f39a80a6040 2024-02-21T23:03:10.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.590+0000 7f39aca00640 1 -- 172.21.15.28:0/1634146062 >> v1:172.21.15.81:6789/0 conn(0x7f39a806d390 legacy=0x7f39a80c46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:10.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.590+0000 7f39aca00640 1 -- 172.21.15.28:0/1634146062 >> v1:172.21.15.28:6789/0 conn(0x7f39a80a6040 legacy=0x7f39a80c2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:10.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.590+0000 7f39aca00640 1 -- 172.21.15.28:0/1634146062 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39a8151fa0 con 0x7f39a806b8b0 2024-02-21T23:03:10.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.591+0000 7fc4b5600640 1 Processor -- start 2024-02-21T23:03:10.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.591+0000 7fe87c200640 1 Processor -- start 2024-02-21T23:03:10.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.591+0000 7f7aee600640 1 Processor -- start 2024-02-21T23:03:10.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.591+0000 7f7aee600640 1 -- start start 2024-02-21T23:03:10.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.591+0000 7fe87c200640 1 -- start start 2024-02-21T23:03:10.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.591+0000 7fc4b5600640 1 -- start start 2024-02-21T23:03:10.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.592+0000 7f39aca00640 1 --2- 172.21.15.28:0/1634146062 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39a806b8b0 0x7f39a8151670 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f399c0097a0 tx=0x7f399c02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=871fd0bfd706ac00 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:10.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.593+0000 7f39a7400640 1 -- 172.21.15.28:0/1634146062 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f399c034070 con 0x7f39a806b8b0 2024-02-21T23:03:10.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.593+0000 7f7aee600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f7ae00cda50 0x7f7ae00d79b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:10.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.593+0000 7f7aee600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7ae0006ef0 0x7f7ae0007340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:10.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.593+0000 7f7aee600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7ae00daab0 0x7f7ae00daf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:10.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.593+0000 7f7aee600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ae00d7ef0 con 0x7f7ae00daab0 2024-02-21T23:03:10.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.593+0000 7f7aee600640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ae00db460 con 0x7f7ae00cda50 2024-02-21T23:03:10.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.593+0000 7f7aee600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae00cde30 con 0x7f7ae0006ef0 2024-02-21T23:03:10.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.593+0000 7f39a7400640 1 -- 172.21.15.28:0/1634146062 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f399c030610 con 0x7f39a806b8b0 2024-02-21T23:03:10.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.593+0000 7f39a7400640 1 -- 172.21.15.28:0/1634146062 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f399c0309c0 con 0x7f39a806b8b0 2024-02-21T23:03:10.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.593+0000 7fc4b5600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc4a80dbd30 0x7fc4a800cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:10.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.593+0000 7fc4b5600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc4a800d7d0 0x7fc4a80409c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:10.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.593+0000 7fc4b5600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fc4a8040f00 0x7fc4a80e22d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:10.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.593+0000 7fc4b5600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4a80e1e20 con 0x7fc4a800d7d0 2024-02-21T23:03:10.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.593+0000 7fc4b5600640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4a8049830 con 0x7fc4a8040f00 2024-02-21T23:03:10.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.593+0000 7fc4b5600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a8049b10 con 0x7fc4a80dbd30 2024-02-21T23:03:10.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.594+0000 7fc4b4c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fc4a8040f00 0x7fc4a80e22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:10.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.594+0000 7f7ae7e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7ae0006ef0 0x7f7ae0007340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:10.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.594+0000 7f7aed200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f7ae00cda50 0x7f7ae00d79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:10.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.594+0000 7fc4b4c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fc4a8040f00 0x7fc4a80e22d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.28:53504/0 (socket says 172.21.15.28:53504) 2024-02-21T23:03:10.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.594+0000 7fc4b4c00640 1 -- 172.21.15.28:0/4187761676 learned_addr learned my addr 172.21.15.28:0/4187761676 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:10.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.594+0000 7f7ae7e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7ae0006ef0 0x7f7ae0007340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3301/0 says I am v2:172.21.15.28:40614/0 (socket says 172.21.15.28:40614) 2024-02-21T23:03:10.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.594+0000 7f7ae7e00640 1 -- 172.21.15.28:0/616834024 learned_addr learned my addr 172.21.15.28:0/616834024 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:10.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.595+0000 7fc4b4c00640 1 -- 172.21.15.28:0/4187761676 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc4a80dbd30 msgr2=0x7fc4a800cf70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-02-21T23:03:10.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.595+0000 7fc4b4c00640 1 --2- 172.21.15.28:0/4187761676 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc4a80dbd30 0x7fc4a800cf70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:10.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.595+0000 7fc4b4c00640 1 -- 172.21.15.28:0/4187761676 >> v2:172.21.15.28:3300/0 conn(0x7fc4a800d7d0 msgr2=0x7fc4a80409c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:10.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.595+0000 7fc4b4c00640 1 --2- 172.21.15.28:0/4187761676 >> v2:172.21.15.28:3300/0 conn(0x7fc4a800d7d0 0x7fc4a80409c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:10.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.595+0000 7fc4b4c00640 1 -- 172.21.15.28:0/4187761676 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4a8049df0 con 0x7fc4a8040f00 2024-02-21T23:03:10.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.594+0000 7f39af400640 1 -- 172.21.15.28:0/1634146062 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39a806b8b0 msgr2=0x7f39a8151670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:10.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.594+0000 7f39af400640 1 --2- 172.21.15.28:0/1634146062 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39a806b8b0 0x7f39a8151670 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f399c0097a0 tx=0x7f399c02b960 comp rx=0 tx=0).stop 2024-02-21T23:03:10.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.595+0000 7f7ae7e00640 1 -- 172.21.15.28:0/616834024 >> v2:172.21.15.81:3300/0 conn(0x7f7ae00cda50 msgr2=0x7f7ae00d79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:10.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.596+0000 7fe87c200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe8740c5200 0x7fe8740c55e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:10.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.596+0000 7f7ae7e00640 1 --2- 172.21.15.28:0/616834024 >> v2:172.21.15.81:3300/0 conn(0x7f7ae00cda50 0x7f7ae00d79b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:10.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.596+0000 7f7ae7e00640 1 -- 172.21.15.28:0/616834024 >> v2:172.21.15.28:3300/0 conn(0x7f7ae00daab0 msgr2=0x7f7ae00daf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:10.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.596+0000 7f7ae7e00640 1 --2- 172.21.15.28:0/616834024 >> v2:172.21.15.28:3300/0 conn(0x7f7ae00daab0 0x7f7ae00daf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:10.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.596+0000 7f7ae7e00640 1 -- 172.21.15.28:0/616834024 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ae004ac10 con 0x7f7ae0006ef0 2024-02-21T23:03:10.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.596+0000 7fe87c200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8740c8c00 con 0x7fe874158c60 2024-02-21T23:03:10.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.596+0000 7fe87c200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8740c8e80 con 0x7fe8740c8780 2024-02-21T23:03:10.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.596+0000 7fe87c200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe874155f40 con 0x7fe8740c5200 2024-02-21T23:03:10.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.597+0000 7f7aed200640 1 --2- 172.21.15.28:0/616834024 >> v2:172.21.15.81:3300/0 conn(0x7f7ae00cda50 0x7f7ae00d79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-21T23:03:10.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.597+0000 7fe879800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe8740c5200 0x7fe8740c55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:10.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.597+0000 7fe879800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe8740c5200 0x7fe8740c55e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45980/0 (socket says 172.21.15.28:45980) 2024-02-21T23:03:10.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.597+0000 7fe879800640 1 -- 172.21.15.28:0/2082429258 learned_addr learned my addr 172.21.15.28:0/2082429258 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:10.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.597+0000 7f39af400640 1 -- 172.21.15.28:0/1634146062 shutdown_connections 2024-02-21T23:03:10.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.597+0000 7f39af400640 1 --2- 172.21.15.28:0/1634146062 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39a806b8b0 0x7f39a8151670 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:10.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.597+0000 7f39af400640 1 -- 172.21.15.28:0/1634146062 >> 172.21.15.28:0/1634146062 conn(0x7f39a806b4a0 msgr2=0x7f39a80c67f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:10.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.598+0000 7f39af400640 1 -- 172.21.15.28:0/1634146062 shutdown_connections 2024-02-21T23:03:10.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.598+0000 7f7ae7e00640 1 --2- 172.21.15.28:0/616834024 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7ae0006ef0 0x7f7ae0007340 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7ae000c040 tx=0x7f7adc02bf00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63a8a0db19103ec6 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:10.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.598+0000 7fc4b4c00640 1 --2- 172.21.15.28:0/4187761676 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc4a8040f00 0x7fc4a80e22d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc4b0074070 tx=0x7fc4b00a6850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8168e1b293d1e562 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:10.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.598+0000 7f7ae7400640 1 -- 172.21.15.28:0/616834024 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7adc039070 con 0x7f7ae0006ef0 2024-02-21T23:03:10.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.598+0000 7f7ae7400640 1 -- 172.21.15.28:0/616834024 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7adc02ccb0 con 0x7f7ae0006ef0 2024-02-21T23:03:10.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.598+0000 7f7ae7400640 1 -- 172.21.15.28:0/616834024 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7adc034660 con 0x7f7ae0006ef0 2024-02-21T23:03:10.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.599+0000 7fc4aea00640 1 -- 172.21.15.28:0/4187761676 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b00ac460 con 0x7fc4a8040f00 2024-02-21T23:03:10.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.599+0000 7fc4aea00640 1 -- 172.21.15.28:0/4187761676 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4b00ac600 con 0x7fc4a8040f00 2024-02-21T23:03:10.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.599+0000 7fc4aea00640 1 -- 172.21.15.28:0/4187761676 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b00aca70 con 0x7fc4a8040f00 2024-02-21T23:03:10.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.599+0000 7f7aee600640 1 -- 172.21.15.28:0/616834024 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7ae0006ef0 msgr2=0x7f7ae0007340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:10.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.599+0000 7f7aee600640 1 --2- 172.21.15.28:0/616834024 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7ae0006ef0 0x7f7ae0007340 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7ae000c040 tx=0x7f7adc02bf00 comp rx=0 tx=0).stop 2024-02-21T23:03:10.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.599+0000 7f7aee600640 1 -- 172.21.15.28:0/616834024 shutdown_connections 2024-02-21T23:03:10.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.599+0000 7f7aee600640 1 --2- 172.21.15.28:0/616834024 >> v2:172.21.15.28:3300/0 conn(0x7f7ae00daab0 0x7f7ae00daf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:10.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.599+0000 7f7aee600640 1 --2- 172.21.15.28:0/616834024 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7ae0006ef0 0x7f7ae0007340 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:10.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.599+0000 7f7aee600640 1 --2- 172.21.15.28:0/616834024 >> v2:172.21.15.81:3300/0 conn(0x7f7ae00cda50 0x7f7ae00d79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:10.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.599+0000 7f7aee600640 1 -- 172.21.15.28:0/616834024 >> 172.21.15.28:0/616834024 conn(0x7f7ae0048250 msgr2=0x7f7ae0048660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:10.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.599+0000 7f39af400640 1 -- 172.21.15.28:0/1634146062 wait complete. 2024-02-21T23:03:10.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.601+0000 7f39af400640 1 Processor -- start 2024-02-21T23:03:10.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.602+0000 7f39af400640 1 -- start start 2024-02-21T23:03:10.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.600+0000 7fc4b5600640 1 -- 172.21.15.28:0/4187761676 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc4a8040f00 msgr2=0x7fc4a80e22d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:10.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.600+0000 7fc4b5600640 1 --2- 172.21.15.28:0/4187761676 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc4a8040f00 0x7fc4a80e22d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc4b0074070 tx=0x7fc4b00a6850 comp rx=0 tx=0).stop 2024-02-21T23:03:10.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.600+0000 7fc4b5600640 1 -- 172.21.15.28:0/4187761676 shutdown_connections 2024-02-21T23:03:10.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.600+0000 7fc4b5600640 1 --2- 172.21.15.28:0/4187761676 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc4a8040f00 0x7fc4a80e22d0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:10.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.600+0000 7fc4b5600640 1 --2- 172.21.15.28:0/4187761676 >> v2:172.21.15.28:3300/0 conn(0x7fc4a800d7d0 0x7fc4a80409c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:10.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.600+0000 7fc4b5600640 1 --2- 172.21.15.28:0/4187761676 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc4a80dbd30 0x7fc4a800cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:10.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.600+0000 7fc4b5600640 1 -- 172.21.15.28:0/4187761676 >> 172.21.15.28:0/4187761676 conn(0x7fc4a802f560 msgr2=0x7fc4a802f940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:10.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.600+0000 7f7aee600640 1 -- 172.21.15.28:0/616834024 shutdown_connections 2024-02-21T23:03:10.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.602+0000 7fe873e00640 1 -- 172.21.15.28:0/2082429258 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2165082044 0 0) 0x7fe860001040 con 0x7fe874158c60 2024-02-21T23:03:10.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.602+0000 7fe873e00640 1 -- 172.21.15.28:0/2082429258 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe85c003520 con 0x7fe874158c60 2024-02-21T23:03:10.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.602+0000 7fe873e00640 1 -- 172.21.15.28:0/2082429258 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1022751589 0 0) 0x7fe8740c8e80 con 0x7fe8740c8780 2024-02-21T23:03:10.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.602+0000 7fe873e00640 1 -- 172.21.15.28:0/2082429258 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe860001040 con 0x7fe8740c8780 2024-02-21T23:03:10.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.602+0000 7fe879800640 1 -- 172.21.15.28:0/2082429258 >> v1:172.21.15.81:6789/0 conn(0x7fe8740c8780 legacy=0x7fe874158550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:10.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.602+0000 7fe879800640 1 -- 172.21.15.28:0/2082429258 >> v1:172.21.15.28:6789/0 conn(0x7fe874158c60 legacy=0x7fe87415f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:10.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.602+0000 7fe879800640 1 -- 172.21.15.28:0/2082429258 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe874155180 con 0x7fe8740c5200 2024-02-21T23:03:10.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.600+0000 7f7aee600640 1 -- 172.21.15.28:0/616834024 wait complete. 2024-02-21T23:03:10.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.602+0000 7f7aee600640 1 Processor -- start 2024-02-21T23:03:10.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.603+0000 7f7aee600640 1 -- start start 2024-02-21T23:03:10.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.601+0000 7fc4b5600640 1 -- 172.21.15.28:0/4187761676 shutdown_connections 2024-02-21T23:03:10.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.601+0000 7fc4b5600640 1 -- 172.21.15.28:0/4187761676 wait complete. 2024-02-21T23:03:10.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.603+0000 7fc4b5600640 1 Processor -- start 2024-02-21T23:03:10.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.604+0000 7fc4b5600640 1 -- start start 2024-02-21T23:03:10.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.604+0000 7fe879800640 1 --2- 172.21.15.28:0/2082429258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8740c5200 0x7fe8740c55e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe86c009870 tx=0x7fe86c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26a01b86985e734c server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:10.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.605+0000 7fe873e00640 1 -- 172.21.15.28:0/2082429258 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe86c034070 con 0x7fe8740c5200 2024-02-21T23:03:10.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.605+0000 7f39af400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f39a806b8b0 0x7f39a81e7620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:10.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.605+0000 7f7aee600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7ae0006ef0 0x7f7ae00d4920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:10.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.605+0000 7fe873e00640 1 -- 172.21.15.28:0/2082429258 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe86c030860 con 0x7fe8740c5200 2024-02-21T23:03:10.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.605+0000 7fe873e00640 1 -- 172.21.15.28:0/2082429258 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe86c034070 con 0x7fe8740c5200 2024-02-21T23:03:10.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.605+0000 7f39af400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39a806d390 0x7f39a814dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:10.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.605+0000 7f7aee600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ae00cda50 0x7f7ae0170910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:10.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.605+0000 7f39af400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39a80a6040 0x7f39a80aacd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:10.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.605+0000 7f7aee600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ae00daab0 0x7f7ae016f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:10.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.605+0000 7fd47c000640 1 Processor -- start 2024-02-21T23:03:10.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.605+0000 7f39aca00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f39a806b8b0 0x7f39a81e7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:10.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.606+0000 7f39aca00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f39a806b8b0 0x7f39a81e7620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3301/0 says I am v2:172.21.15.28:40632/0 (socket says 172.21.15.28:40632) 2024-02-21T23:03:10.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.606+0000 7f39aca00640 1 -- 172.21.15.28:0/3058958991 learned_addr learned my addr 172.21.15.28:0/3058958991 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:10.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.606+0000 7f39af400640 1 -- 172.21.15.28:0/3058958991 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39a80c62d0 con 0x7f39a80a6040 2024-02-21T23:03:10.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.606+0000 7f7aed200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7ae0006ef0 0x7f7ae00d4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:10.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.606+0000 7f39af400640 1 -- 172.21.15.28:0/3058958991 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39a80c3310 con 0x7f39a806d390 2024-02-21T23:03:10.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.606+0000 7f39af400640 1 -- 172.21.15.28:0/3058958991 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39a80c4e00 con 0x7f39a806b8b0 2024-02-21T23:03:10.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.606+0000 7fc4b5600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4a800d7d0 0x7fc4a80306c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:10.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.606+0000 7f7ae7e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ae00cda50 0x7f7ae0170910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:10.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.606+0000 7f7aed200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7ae0006ef0 0x7f7ae00d4920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3301/0 says I am v2:172.21.15.28:40630/0 (socket says 172.21.15.28:40630) 2024-02-21T23:03:10.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.606+0000 7f7aed200640 1 -- 172.21.15.28:0/1861662920 learned_addr learned my addr 172.21.15.28:0/1861662920 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:10.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.606+0000 7f39ad400640 1 --2- 172.21.15.28:0/3058958991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39a80a6040 0x7f39a80aacd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:10.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.606+0000 7f7aee600640 1 -- 172.21.15.28:0/1861662920 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae004b1b0 con 0x7f7ae00daab0 2024-02-21T23:03:10.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.606+0000 7f7aee600640 1 -- 172.21.15.28:0/1861662920 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae004aed0 con 0x7f7ae00cda50 2024-02-21T23:03:10.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.606+0000 7f7aee600640 1 -- 172.21.15.28:0/1861662920 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae004b490 con 0x7f7ae0006ef0 2024-02-21T23:03:10.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.606+0000 7f7aedc00640 1 --2- 172.21.15.28:0/1861662920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ae00daab0 0x7f7ae016f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:10.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.606+0000 7fc4b5600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc4a8040f00 0x7fc4a80dc640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:10.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.606+0000 7fc4b5600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc4a80dbd30 0x7fc4a80dcb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:10.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.606+0000 7fd47c000640 1 -- start start 2024-02-21T23:03:10.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.607+0000 7fc4b5600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a80dc110 con 0x7fc4a800d7d0 2024-02-21T23:03:10.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.607+0000 7fc4b5600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a804a0d0 con 0x7fc4a80dbd30 2024-02-21T23:03:10.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.607+0000 7fc4b5600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a80e2810 con 0x7fc4a8040f00 2024-02-21T23:03:10.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.607+0000 7fc4b4c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc4a80dbd30 0x7fc4a80dcb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:10.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.607+0000 7fc4b4c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc4a80dbd30 0x7fc4a80dcb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.28:53524/0 (socket says 172.21.15.28:53524) 2024-02-21T23:03:10.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.607+0000 7fc4b4c00640 1 -- 172.21.15.28:0/2439353557 learned_addr learned my addr 172.21.15.28:0/2439353557 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:10.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.607+0000 7f7ae7e00640 1 -- 172.21.15.28:0/1861662920 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7ae0006ef0 msgr2=0x7f7ae00d4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:10.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.607+0000 7f7ae7e00640 1 --2- 172.21.15.28:0/1861662920 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7ae0006ef0 0x7f7ae00d4920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:10.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.607+0000 7f7ae7e00640 1 -- 172.21.15.28:0/1861662920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ae00daab0 msgr2=0x7f7ae016f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:10.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.607+0000 7f7ae7e00640 1 --2- 172.21.15.28:0/1861662920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ae00daab0 0x7f7ae016f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:10.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.607+0000 7f7ae7e00640 1 -- 172.21.15.28:0/1861662920 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ae00d4f90 con 0x7f7ae00cda50 2024-02-21T23:03:10.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.607+0000 7f39a7e00640 1 --2- 172.21.15.28:0/3058958991 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39a806d390 0x7f39a814dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:10.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.608+0000 7fc4afe00640 1 --2- 172.21.15.28:0/2439353557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4a800d7d0 0x7fc4a80306c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:10.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.608+0000 7f39aca00640 1 -- 172.21.15.28:0/3058958991 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39a806d390 msgr2=0x7f39a814dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:10.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.608+0000 7fc4af400640 1 --2- 172.21.15.28:0/2439353557 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc4a8040f00 0x7fc4a80dc640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:10.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.608+0000 7f39aca00640 1 --2- 172.21.15.28:0/3058958991 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39a806d390 0x7f39a814dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:10.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.608+0000 7f39aca00640 1 -- 172.21.15.28:0/3058958991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39a80a6040 msgr2=0x7f39a80aacd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:10.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.608+0000 7f39aca00640 1 --2- 172.21.15.28:0/3058958991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39a80a6040 0x7f39a80aacd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:10.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.608+0000 7f39aca00640 1 -- 172.21.15.28:0/3058958991 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39a8140030 con 0x7f39a806b8b0 2024-02-21T23:03:10.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.608+0000 7f39ad400640 1 --2- 172.21.15.28:0/3058958991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39a80a6040 0x7f39a80aacd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:03:10.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.608+0000 7f7ae7e00640 1 --2- 172.21.15.28:0/1861662920 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ae00cda50 0x7f7ae0170910 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7adc033000 tx=0x7f7adc0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:10.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.608+0000 7fe87c200640 1 -- 172.21.15.28:0/2082429258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8740c5200 msgr2=0x7fe8740c55e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:10.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.608+0000 7fe87c200640 1 --2- 172.21.15.28:0/2082429258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8740c5200 0x7fe8740c55e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe86c009870 tx=0x7fe86c02b9a0 comp rx=0 tx=0).stop 2024-02-21T23:03:10.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.608+0000 7fd47c000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd4741550d0 0x7fd4740c5320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:10.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.608+0000 7fd47c000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd4740bfa40 0x7fd4740bfeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:10.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.608+0000 7fd47c000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd47414f5a0 con 0x7fd47414f1c0 2024-02-21T23:03:10.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.608+0000 7fd47c000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4740c5a20 con 0x7fd4740bfa40 2024-02-21T23:03:10.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.608+0000 7fd47c000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4740c5d00 con 0x7fd4741550d0 2024-02-21T23:03:10.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.609+0000 7fe87c200640 1 -- 172.21.15.28:0/2082429258 shutdown_connections 2024-02-21T23:03:10.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.609+0000 7fe87c200640 1 --2- 172.21.15.28:0/2082429258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8740c5200 0x7fe8740c55e0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:10.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.609+0000 7fe87c200640 1 -- 172.21.15.28:0/2082429258 >> 172.21.15.28:0/2082429258 conn(0x7fe8740ac3b0 msgr2=0x7fe8740ac7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:10.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.608+0000 7f7ae4c00640 1 -- 172.21.15.28:0/1861662920 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7adc043030 con 0x7f7ae00cda50 2024-02-21T23:03:10.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.609+0000 7f7ae4c00640 1 -- 172.21.15.28:0/1861662920 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7adc02cab0 con 0x7f7ae00cda50 2024-02-21T23:03:10.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.609+0000 7f7ae4c00640 1 -- 172.21.15.28:0/1861662920 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7adc039040 con 0x7f7ae00cda50 2024-02-21T23:03:10.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.609+0000 7fd479600640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd47414f1c0 0x7fd4740ca890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:48572/0 (socket says 172.21.15.28:48572) 2024-02-21T23:03:10.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.609+0000 7fd479600640 1 -- 172.21.15.28:0/595525915 learned_addr learned my addr 172.21.15.28:0/595525915 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:03:10.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.609+0000 7fc4b4c00640 1 -- 172.21.15.28:0/2439353557 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc4a8040f00 msgr2=0x7fc4a80dc640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:10.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.609+0000 7fc4b4c00640 1 --2- 172.21.15.28:0/2439353557 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc4a8040f00 0x7fc4a80dc640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:10.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.609+0000 7fc4b4c00640 1 -- 172.21.15.28:0/2439353557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4a800d7d0 msgr2=0x7fc4a80306c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:10.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.609+0000 7fc4b4c00640 1 --2- 172.21.15.28:0/2439353557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4a800d7d0 0x7fc4a80306c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:10.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.609+0000 7fc4b4c00640 1 -- 172.21.15.28:0/2439353557 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4a8047b60 con 0x7fc4a80dbd30 2024-02-21T23:03:10.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.609+0000 7fd47a000640 1 --2- 172.21.15.28:0/595525915 >> v2:172.21.15.28:3300/0 conn(0x7fd4740bfa40 0x7fd4740bfeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:10.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.609+0000 7f39aca00640 1 --2- 172.21.15.28:0/3058958991 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f39a806b8b0 0x7f39a81e7620 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f399c02b9a0 tx=0x7f399c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:10.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.609+0000 7fe87c200640 1 -- 172.21.15.28:0/2082429258 shutdown_connections 2024-02-21T23:03:10.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.610+0000 7fd478c00640 1 --2- 172.21.15.28:0/595525915 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd4741550d0 0x7fd4740c5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:10.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.610+0000 7f39a4c00640 1 -- 172.21.15.28:0/3058958991 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f399c047070 con 0x7f39a806b8b0 2024-02-21T23:03:10.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.610+0000 7f178ca00640 1 Processor -- start 2024-02-21T23:03:10.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.611+0000 7f178ca00640 1 -- start start 2024-02-21T23:03:10.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.611+0000 7f39a4c00640 1 -- 172.21.15.28:0/3058958991 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f399c030410 con 0x7f39a806b8b0 2024-02-21T23:03:10.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.611+0000 7f39af400640 1 -- 172.21.15.28:0/3058958991 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39a813f790 con 0x7f39a806b8b0 2024-02-21T23:03:10.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.611+0000 7f39a4c00640 1 -- 172.21.15.28:0/3058958991 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f399c034050 con 0x7f39a806b8b0 2024-02-21T23:03:10.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.611+0000 7f39af400640 1 -- 172.21.15.28:0/3058958991 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39a813fd50 con 0x7f39a806b8b0 2024-02-21T23:03:10.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.611+0000 7fc4b4c00640 1 --2- 172.21.15.28:0/2439353557 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc4a80dbd30 0x7fc4a80dcb80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc4b00685e0 tx=0x7fc4b006a210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:10.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.610+0000 7f7aee600640 1 -- 172.21.15.28:0/1861662920 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ae00d5270 con 0x7f7ae00cda50 2024-02-21T23:03:10.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.610+0000 7f7aee600640 1 -- 172.21.15.28:0/1861662920 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ae00d5830 con 0x7f7ae00cda50 2024-02-21T23:03:10.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.611+0000 7f7aee600640 1 -- 172.21.15.28:0/1861662920 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7ae0171120 con 0x7f7ae00cda50 2024-02-21T23:03:10.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.610+0000 7fe87c200640 1 -- 172.21.15.28:0/2082429258 wait complete. 2024-02-21T23:03:10.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.612+0000 7fe87c200640 1 Processor -- start 2024-02-21T23:03:10.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.613+0000 7f39af400640 1 -- 172.21.15.28:0/3058958991 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f39a806f400 con 0x7f39a806b8b0 2024-02-21T23:03:10.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.612+0000 7fc4a3e00640 1 -- 172.21.15.28:0/2439353557 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b00bc070 con 0x7fc4a80dbd30 2024-02-21T23:03:10.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.612+0000 7fc4a3e00640 1 -- 172.21.15.28:0/2439353557 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4b00ad980 con 0x7fc4a80dbd30 2024-02-21T23:03:10.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.612+0000 7fc4a3e00640 1 -- 172.21.15.28:0/2439353557 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b00aced0 con 0x7fc4a80dbd30 2024-02-21T23:03:10.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.611+0000 7fd47a000640 1 -- 172.21.15.28:0/595525915 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd4741550d0 msgr2=0x7fd4740c5320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:10.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.611+0000 7fd47a000640 1 --2- 172.21.15.28:0/595525915 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd4741550d0 0x7fd4740c5320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:10.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.611+0000 7fd47a000640 1 -- 172.21.15.28:0/595525915 >> v1:172.21.15.28:6789/0 conn(0x7fd47414f1c0 legacy=0x7fd4740ca890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:10.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.611+0000 7fd47a000640 1 -- 172.21.15.28:0/595525915 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4740c5fe0 con 0x7fd4740bfa40 2024-02-21T23:03:10.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.613+0000 7f178ca00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f178813e480 0x7f1788155160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:10.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.613+0000 7f178ca00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f17880c5320 0x7f17880ca850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:10.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.613+0000 7fe87c200640 1 -- start start 2024-02-21T23:03:10.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.613+0000 7fd47a000640 1 --2- 172.21.15.28:0/595525915 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4740bfa40 0x7fd4740bfeb0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd46c009d90 tx=0x7fd46c02be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c0cc7dec5f9524c server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:10.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.613+0000 7fc4b5600640 1 -- 172.21.15.28:0/2439353557 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4a8047e10 con 0x7fc4a80dbd30 2024-02-21T23:03:10.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.613+0000 7f177d400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f17880c5320 0x7f17880ca850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:10.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.614+0000 7f1785e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f178813e480 0x7f1788155160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:10.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.614+0000 7f1785e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f178813e480 0x7f1788155160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.28:53550/0 (socket says 172.21.15.28:53550) 2024-02-21T23:03:10.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.614+0000 7f1785e00640 1 -- 172.21.15.28:0/3728518835 learned_addr learned my addr 172.21.15.28:0/3728518835 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:10.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.614+0000 7f178ca00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1788073380 con 0x7f17880cad90 2024-02-21T23:03:10.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.614+0000 7f178ca00640 1 -- 172.21.15.28:0/3728518835 --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17880c5760 con 0x7f178813e480 2024-02-21T23:03:10.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.614+0000 7f178ca00640 1 -- 172.21.15.28:0/3728518835 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17880c5a00 con 0x7f17880c5320 2024-02-21T23:03:10.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.613+0000 7fc4b5600640 1 -- 172.21.15.28:0/2439353557 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4a80450a0 con 0x7fc4a80dbd30 2024-02-21T23:03:10.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.614+0000 7fd473e00640 1 -- 172.21.15.28:0/595525915 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd46c039070 con 0x7fd4740bfa40 2024-02-21T23:03:10.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.614+0000 7fc4b5600640 1 -- 172.21.15.28:0/2439353557 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc4a8180350 con 0x7fc4a80dbd30 2024-02-21T23:03:10.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.614+0000 7fe87c200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe8740c5200 0x7fe874158090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:10.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.614+0000 7fe87c200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe8740c8780 0x7fe8741ebb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:10.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.614+0000 7fe87c200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe874158c60 0x7fe8741ec090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:10.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.614+0000 7fd473e00640 1 -- 172.21.15.28:0/595525915 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd46c02cbe0 con 0x7fd4740bfa40 2024-02-21T23:03:10.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.615+0000 7fd473e00640 1 -- 172.21.15.28:0/595525915 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd46c039070 con 0x7fd4740bfa40 2024-02-21T23:03:10.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.615+0000 7f177d400640 1 -- 172.21.15.28:0/3728518835 >> v2:172.21.15.81:3300/0 conn(0x7f178813e480 msgr2=0x7f1788155160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:10.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.615+0000 7fe878e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe8740c8780 0x7fe8741ebb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:10.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.615+0000 7fe878e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe8740c8780 0x7fe8741ebb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3301/0 says I am v2:172.21.15.28:40702/0 (socket says 172.21.15.28:40702) 2024-02-21T23:03:10.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.615+0000 7f39a4c00640 1 -- 172.21.15.28:0/3058958991 <== mon.2 v2:172.21.15.81:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f399c04a070 con 0x7f39a806b8b0 2024-02-21T23:03:10.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.616+0000 7fe878e00640 1 -- 172.21.15.28:0/2635171193 learned_addr learned my addr 172.21.15.28:0/2635171193 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:10.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.616+0000 7fe87c200640 1 -- 172.21.15.28:0/2635171193 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe874155740 con 0x7fe874158c60 2024-02-21T23:03:10.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.616+0000 7fe87c200640 1 -- 172.21.15.28:0/2635171193 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe874152a80 con 0x7fe8740c5200 2024-02-21T23:03:10.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.616+0000 7fe87c200640 1 -- 172.21.15.28:0/2635171193 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe874155a20 con 0x7fe8740c8780 2024-02-21T23:03:10.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.616+0000 7f177d400640 1 --2- 172.21.15.28:0/3728518835 >> v2:172.21.15.81:3300/0 conn(0x7f178813e480 0x7f1788155160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:10.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.616+0000 7f177d400640 1 -- 172.21.15.28:0/3728518835 >> v1:172.21.15.81:6789/0 conn(0x7f17880cad90 legacy=0x7f17880bfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:10.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.616+0000 7f177d400640 1 -- 172.21.15.28:0/3728518835 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17880c5ce0 con 0x7f17880c5320 2024-02-21T23:03:10.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.616+0000 7fe879800640 1 --2- 172.21.15.28:0/2635171193 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe8740c5200 0x7fe874158090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:10.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.616+0000 7f39a4c00640 1 --2- 172.21.15.28:0/3058958991 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f398806b520 0x7f398806d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:10.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.617+0000 7f39a4c00640 1 -- 172.21.15.28:0/3058958991 <== mon.2 v2:172.21.15.81:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f399c0ac520 con 0x7f39a806b8b0 2024-02-21T23:03:10.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.616+0000 7fd47c000640 1 -- 172.21.15.28:0/595525915 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4740bfa40 msgr2=0x7fd4740bfeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:10.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.616+0000 7fd47c000640 1 --2- 172.21.15.28:0/595525915 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4740bfa40 0x7fd4740bfeb0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd46c009d90 tx=0x7fd46c02be30 comp rx=0 tx=0).stop 2024-02-21T23:03:10.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.617+0000 7f1785e00640 1 --2- 172.21.15.28:0/3728518835 >> v2:172.21.15.81:3300/0 conn(0x7f178813e480 0x7f1788155160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-21T23:03:10.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.617+0000 7fe878e00640 1 -- 172.21.15.28:0/2635171193 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe8740c5200 msgr2=0x7fe874158090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:10.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.617+0000 7fe878e00640 1 --2- 172.21.15.28:0/2635171193 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe8740c5200 0x7fe874158090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:10.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.617+0000 7fe878e00640 1 -- 172.21.15.28:0/2635171193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe874158c60 msgr2=0x7fe8741ec090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:10.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.617+0000 7fe878e00640 1 --2- 172.21.15.28:0/2635171193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe874158c60 0x7fe8741ec090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:10.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.617+0000 7fe878e00640 1 -- 172.21.15.28:0/2635171193 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8740c0de0 con 0x7fe8740c8780 2024-02-21T23:03:10.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.617+0000 7f39a4c00640 1 --2- 172.21.15.28:0/3058958991 >> [v2:172.21.15.28:6808/2193666734,v1:172.21.15.28:6810/2193666734] conn(0x7f3988074ca0 0x7f39880770e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:10.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.617+0000 7f39a4c00640 1 -- 172.21.15.28:0/3058958991 --> [v2:172.21.15.28:6808/2193666734,v1:172.21.15.28:6810/2193666734] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f399c044ec0 con 0x7f3988074ca0 2024-02-21T23:03:10.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.617+0000 7f39a4c00640 1 -- 172.21.15.28:0/3058958991 <== mon.2 v2:172.21.15.81:3301/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f399c0aca00 con 0x7f39a806b8b0 2024-02-21T23:03:10.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.617+0000 7f39a7e00640 1 --2- 172.21.15.28:0/3058958991 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f398806b520 0x7f398806d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:10.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.617+0000 7fe878e00640 1 --2- 172.21.15.28:0/2635171193 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe8740c8780 0x7fe8741ebb50 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe86800c7a0 tx=0x7fe86800cc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:10.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.617+0000 7fc4a3e00640 1 -- 172.21.15.28:0/2439353557 <== mon.1 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc4b00ca480 con 0x7fc4a80dbd30 2024-02-21T23:03:10.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.617+0000 7f7ae4c00640 1 -- 172.21.15.28:0/1861662920 <== mon.1 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7adc046070 con 0x7f7ae00cda50 2024-02-21T23:03:10.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.618+0000 7f39ad400640 1 --2- 172.21.15.28:0/3058958991 >> [v2:172.21.15.28:6808/2193666734,v1:172.21.15.28:6810/2193666734] conn(0x7f3988074ca0 0x7f39880770e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:10.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.617+0000 7fd47c000640 1 -- 172.21.15.28:0/595525915 shutdown_connections 2024-02-21T23:03:10.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.617+0000 7fd47c000640 1 --2- 172.21.15.28:0/595525915 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4740bfa40 0x7fd4740bfeb0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:10.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.617+0000 7fd47c000640 1 --2- 172.21.15.28:0/595525915 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd4741550d0 0x7fd4740c5320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:10.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.617+0000 7fd47c000640 1 -- 172.21.15.28:0/595525915 >> 172.21.15.28:0/595525915 conn(0x7fd47408b1c0 msgr2=0x7fd4740725a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:10.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.618+0000 7fd47c000640 1 -- 172.21.15.28:0/595525915 shutdown_connections 2024-02-21T23:03:10.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.618+0000 7fd47c000640 1 -- 172.21.15.28:0/595525915 wait complete. 2024-02-21T23:03:10.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.618+0000 7fe871600640 1 -- 172.21.15.28:0/2635171193 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe868003b70 con 0x7fe8740c8780 2024-02-21T23:03:10.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.618+0000 7fe871600640 1 -- 172.21.15.28:0/2635171193 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe868003d10 con 0x7fe8740c8780 2024-02-21T23:03:10.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.618+0000 7fe871600640 1 -- 172.21.15.28:0/2635171193 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe86800aa70 con 0x7fe8740c8780 2024-02-21T23:03:10.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.619+0000 7fc4a3e00640 1 --2- 172.21.15.28:0/2439353557 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc49006b520 0x7fc49006d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:10.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.619+0000 7f7ae4c00640 1 --2- 172.21.15.28:0/1861662920 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7ac806b5a0 0x7f7ac806da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:10.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.619+0000 7fc4a3e00640 1 -- 172.21.15.28:0/2439353557 <== mon.1 v2:172.21.15.81:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fc4b0127560 con 0x7fc4a80dbd30 2024-02-21T23:03:10.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.619+0000 7f7ae4c00640 1 -- 172.21.15.28:0/1861662920 <== mon.1 v2:172.21.15.81:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f7adc0acb00 con 0x7f7ae00cda50 2024-02-21T23:03:10.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.618+0000 7fe87c200640 1 -- 172.21.15.28:0/2635171193 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8741ef630 con 0x7fe8740c8780 2024-02-21T23:03:10.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.618+0000 7fe87c200640 1 -- 172.21.15.28:0/2635171193 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8740c3a20 con 0x7fe8740c8780 2024-02-21T23:03:10.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.619+0000 7fc4a3e00640 1 --2- 172.21.15.28:0/2439353557 >> [v2:172.21.15.28:6809/160098884,v1:172.21.15.28:6812/160098884] conn(0x7fc490074c90 0x7fc4900770d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:10.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.619+0000 7f177d400640 1 --2- 172.21.15.28:0/3728518835 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f17880c5320 0x7f17880ca850 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f17780097a0 tx=0x7f177802b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2eeb5eb70181b953 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:10.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.619+0000 7fc4a3e00640 1 -- 172.21.15.28:0/2439353557 --> [v2:172.21.15.28:6809/160098884,v1:172.21.15.28:6812/160098884] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc4b006de80 con 0x7fc490074c90 2024-02-21T23:03:10.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.619+0000 7fc4a3e00640 1 -- 172.21.15.28:0/2439353557 <== mon.1 v2:172.21.15.81:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc4b0127910 con 0x7fc4a80dbd30 2024-02-21T23:03:10.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.619+0000 7f39a7e00640 1 --2- 172.21.15.28:0/3058958991 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f398806b520 0x7f398806d9e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f39a80aba00 tx=0x7f39940014b0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:10.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.620+0000 7f7ae4c00640 1 --2- 172.21.15.28:0/1861662920 >> [v2:172.21.15.28:6824/3053508071,v1:172.21.15.28:6825/3053508071] conn(0x7f7ac8074d20 0x7f7ac8077160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:10.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.620+0000 7f1785400640 1 -- 172.21.15.28:0/3728518835 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1778039070 con 0x7f17880c5320 2024-02-21T23:03:10.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.620+0000 7fe87c200640 1 -- 172.21.15.28:0/2635171193 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe8740c3e00 con 0x7fe8740c8780 2024-02-21T23:03:10.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.620+0000 7fc4afe00640 1 --2- 172.21.15.28:0/2439353557 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc49006b520 0x7fc49006d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:10.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.620+0000 7f1785400640 1 -- 172.21.15.28:0/3728518835 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17780316f0 con 0x7f17880c5320 2024-02-21T23:03:10.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.620+0000 7fc4af400640 1 --2- 172.21.15.28:0/2439353557 >> [v2:172.21.15.28:6809/160098884,v1:172.21.15.28:6812/160098884] conn(0x7fc490074c90 0x7fc4900770d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:10.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.620+0000 7fd47c000640 1 Processor -- start 2024-02-21T23:03:10.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.620+0000 7f7aedc00640 1 --2- 172.21.15.28:0/1861662920 >> [v2:172.21.15.28:6824/3053508071,v1:172.21.15.28:6825/3053508071] conn(0x7f7ac8074d20 0x7f7ac8077160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:10.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.620+0000 7f7ae4c00640 1 -- 172.21.15.28:0/1861662920 --> [v2:172.21.15.28:6824/3053508071,v1:172.21.15.28:6825/3053508071] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7ac8077830 con 0x7f7ac8074d20 2024-02-21T23:03:10.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.620+0000 7f7aed200640 1 --2- 172.21.15.28:0/1861662920 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7ac806b5a0 0x7f7ac806da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:10.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.621+0000 7f7ae4c00640 1 -- 172.21.15.28:0/1861662920 <== mon.1 v2:172.21.15.81:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7adc0acef0 con 0x7f7ae00cda50 2024-02-21T23:03:10.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.621+0000 7fd47c000640 1 -- start start 2024-02-21T23:03:10.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.621+0000 7f178ca00640 1 -- 172.21.15.28:0/3728518835 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f17880c5320 msgr2=0x7f17880ca850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:10.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.621+0000 7f178ca00640 1 --2- 172.21.15.28:0/3728518835 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f17880c5320 0x7f17880ca850 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f17780097a0 tx=0x7f177802b790 comp rx=0 tx=0).stop 2024-02-21T23:03:10.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.621+0000 7f1785400640 1 -- 172.21.15.28:0/3728518835 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1778031ac0 con 0x7f17880c5320 2024-02-21T23:03:10.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.621+0000 7f39ad400640 1 --2- 172.21.15.28:0/3058958991 >> [v2:172.21.15.28:6808/2193666734,v1:172.21.15.28:6810/2193666734] conn(0x7f3988074ca0 0x7f39880770e0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:10.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.621+0000 7f178ca00640 1 -- 172.21.15.28:0/3728518835 shutdown_connections 2024-02-21T23:03:10.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.621+0000 7f178ca00640 1 --2- 172.21.15.28:0/3728518835 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f17880c5320 0x7f17880ca850 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:10.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.621+0000 7f178ca00640 1 --2- 172.21.15.28:0/3728518835 >> v2:172.21.15.81:3300/0 conn(0x7f178813e480 0x7f1788155160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:10.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.621+0000 7f178ca00640 1 -- 172.21.15.28:0/3728518835 >> 172.21.15.28:0/3728518835 conn(0x7f178808b1c0 msgr2=0x7f17880725a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:10.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.622+0000 7f178ca00640 1 -- 172.21.15.28:0/3728518835 shutdown_connections 2024-02-21T23:03:10.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.622+0000 7f178ca00640 1 -- 172.21.15.28:0/3728518835 wait complete. 2024-02-21T23:03:10.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.622+0000 7f604a200640 1 Processor -- start 2024-02-21T23:03:10.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.622+0000 7f7aed200640 1 --2- 172.21.15.28:0/1861662920 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7ac806b5a0 0x7f7ac806da60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7ad400da00 tx=0x7f7ad400f040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:10.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.622+0000 7fd47c000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd4740bfa40 0x7fd4741eb8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:10.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.622+0000 7fd47c000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd47414f1c0 0x7fd474147f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:10.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.622+0000 7fd47c000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd4741550d0 0x7fd47414ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:10.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.622+0000 7fd47c000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4740cafa0 con 0x7fd47414f1c0 2024-02-21T23:03:10.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.622+0000 7fd47c000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4740c65a0 con 0x7fd4740bfa40 2024-02-21T23:03:10.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.622+0000 7fd47c000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4740cb120 con 0x7fd4741550d0 2024-02-21T23:03:10.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.622+0000 7f7aedc00640 1 --2- 172.21.15.28:0/1861662920 >> [v2:172.21.15.28:6824/3053508071,v1:172.21.15.28:6825/3053508071] conn(0x7f7ac8074d20 0x7f7ac8077160 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:10.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.623+0000 7fd478c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd47414f1c0 0x7fd474147f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:10.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.623+0000 7f39a4c00640 1 -- 172.21.15.28:0/3058958991 <== osd.1 v2:172.21.15.28:6808/2193666734 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f399c044ec0 con 0x7f3988074ca0 2024-02-21T23:03:10.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.623+0000 7fd478c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd47414f1c0 0x7fd474147f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46038/0 (socket says 172.21.15.28:46038) 2024-02-21T23:03:10.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.623+0000 7fd478c00640 1 -- 172.21.15.28:0/2793001054 learned_addr learned my addr 172.21.15.28:0/2793001054 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:10.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.623+0000 7fd479600640 1 --2- 172.21.15.28:0/2793001054 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd4740bfa40 0x7fd4741eb8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:10.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.623+0000 7fd47a000640 1 --2- 172.21.15.28:0/2793001054 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd4741550d0 0x7fd47414ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:10.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.624+0000 7fd478c00640 1 -- 172.21.15.28:0/2793001054 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd4741550d0 msgr2=0x7fd47414ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:10.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.624+0000 7fd478c00640 1 --2- 172.21.15.28:0/2793001054 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd4741550d0 0x7fd47414ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:10.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.624+0000 7fd478c00640 1 -- 172.21.15.28:0/2793001054 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd4740bfa40 msgr2=0x7fd4741eb8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:10.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.624+0000 7fd478c00640 1 --2- 172.21.15.28:0/2793001054 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd4740bfa40 0x7fd4741eb8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:10.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.624+0000 7fd478c00640 1 -- 172.21.15.28:0/2793001054 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd474150f40 con 0x7fd47414f1c0 2024-02-21T23:03:10.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.624+0000 7fc4af400640 1 --2- 172.21.15.28:0/2439353557 >> [v2:172.21.15.28:6809/160098884,v1:172.21.15.28:6812/160098884] conn(0x7fc490074c90 0x7fc4900770d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:10.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.624+0000 7fc4afe00640 1 --2- 172.21.15.28:0/2439353557 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc49006b520 0x7fc49006d9e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc4a817a920 tx=0x7fc4a4009060 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:10.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.624+0000 7f7ae4c00640 1 -- 172.21.15.28:0/1861662920 <== osd.3 v2:172.21.15.28:6824/3053508071 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f7ac8077830 con 0x7f7ac8074d20 2024-02-21T23:03:10.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.625+0000 7fd478c00640 1 --2- 172.21.15.28:0/2793001054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd47414f1c0 0x7fd474147f90 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fd46000d620 tx=0x7fd46000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:10.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.623+0000 7f604a200640 1 -- start start 2024-02-21T23:03:10.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.625+0000 7f604a200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f60441550d0 0x7f60440c5320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:10.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.625+0000 7f604a200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f60440bfa40 0x7f60440bfe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:10.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.625+0000 7f604a200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f604414f5a0 con 0x7f604414f1c0 2024-02-21T23:03:10.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.625+0000 7f604a200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60440c5a20 con 0x7f60441550d0 2024-02-21T23:03:10.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.625+0000 7f604a200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60440c5d00 con 0x7f60440bfa40 2024-02-21T23:03:10.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.624+0000 7f178ca00640 1 Processor -- start 2024-02-21T23:03:10.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.625+0000 7fd471600640 1 -- 172.21.15.28:0/2793001054 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd46000b620 con 0x7fd47414f1c0 2024-02-21T23:03:10.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.624+0000 7f178ca00640 1 -- start start 2024-02-21T23:03:10.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.624+0000 7fe871600640 1 -- 172.21.15.28:0/2635171193 <== mon.2 v2:172.21.15.81:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe86800ac10 con 0x7fe8740c8780 2024-02-21T23:03:10.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.626+0000 7fc4a3e00640 1 -- 172.21.15.28:0/2439353557 <== osd.2 v2:172.21.15.28:6809/160098884 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fc4b006de80 con 0x7fc490074c90 2024-02-21T23:03:10.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.626+0000 7fd471600640 1 -- 172.21.15.28:0/2793001054 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd460019070 con 0x7fd47414f1c0 2024-02-21T23:03:10.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.626+0000 7fd471600640 1 -- 172.21.15.28:0/2793001054 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd46000b9e0 con 0x7fd47414f1c0 2024-02-21T23:03:10.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.626+0000 7f6043e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f60440bfa40 0x7f60440bfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:10.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.627+0000 7fe871600640 1 --2- 172.21.15.28:0/2635171193 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe85406b5a0 0x7fe85406da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:10.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.627+0000 7fe871600640 1 -- 172.21.15.28:0/2635171193 <== mon.2 v2:172.21.15.81:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fe86808af10 con 0x7fe8740c8780 2024-02-21T23:03:10.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.628+0000 7fe879800640 1 --2- 172.21.15.28:0/2635171193 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe85406b5a0 0x7fe85406da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:10.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.628+0000 7fe871600640 1 --2- 172.21.15.28:0/2635171193 >> [v2:172.21.15.28:6800/2069258563,v1:172.21.15.28:6801/2069258563] conn(0x7fe854074d10 0x7fe854077150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:10.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.628+0000 7fe871600640 1 -- 172.21.15.28:0/2635171193 --> [v2:172.21.15.28:6800/2069258563,v1:172.21.15.28:6801/2069258563] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe86801fec0 con 0x7fe854074d10 2024-02-21T23:03:10.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.628+0000 7fe871600640 1 -- 172.21.15.28:0/2635171193 <== mon.2 v2:172.21.15.81:3301/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe86808b300 con 0x7fe8740c8780 2024-02-21T23:03:10.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.628+0000 7fe87a200640 1 --2- 172.21.15.28:0/2635171193 >> [v2:172.21.15.28:6800/2069258563,v1:172.21.15.28:6801/2069258563] conn(0x7fe854074d10 0x7fe854077150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:10.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.627+0000 7fd47c000640 1 -- 172.21.15.28:0/2793001054 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd47414fdf0 con 0x7fd47414f1c0 2024-02-21T23:03:10.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.627+0000 7fd47c000640 1 -- 172.21.15.28:0/2793001054 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4741503b0 con 0x7fd47414f1c0 2024-02-21T23:03:10.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.628+0000 7fd47c000640 1 -- 172.21.15.28:0/2793001054 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd47406b2e0 con 0x7fd47414f1c0 2024-02-21T23:03:10.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.626+0000 7f6043400640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f604414f1c0 0x7f60440ca890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.28:50170/0 (socket says 172.21.15.28:50170) 2024-02-21T23:03:10.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.626+0000 7f6043e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f60440bfa40 0x7f60440bfe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.28:53576/0 (socket says 172.21.15.28:53576) 2024-02-21T23:03:10.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.626+0000 7f6043400640 1 -- 172.21.15.28:0/3669030330 learned_addr learned my addr 172.21.15.28:0/3669030330 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:03:10.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.627+0000 7f6042a00640 1 --2- 172.21.15.28:0/3669030330 >> v2:172.21.15.28:3300/0 conn(0x7f60441550d0 0x7f60440c5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:10.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.627+0000 7f6042000640 1 -- 172.21.15.28:0/3669030330 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3003790853 0 0) 0x7f604414f5a0 con 0x7f604414f1c0 2024-02-21T23:03:10.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.627+0000 7f6042000640 1 -- 172.21.15.28:0/3669030330 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6028003520 con 0x7f604414f1c0 2024-02-21T23:03:10.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.627+0000 7f6043e00640 1 -- 172.21.15.28:0/3669030330 >> v2:172.21.15.28:3300/0 conn(0x7f60441550d0 msgr2=0x7f60440c5320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:10.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.627+0000 7f6043e00640 1 --2- 172.21.15.28:0/3669030330 >> v2:172.21.15.28:3300/0 conn(0x7f60441550d0 0x7f60440c5320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:10.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.627+0000 7f6043e00640 1 -- 172.21.15.28:0/3669030330 >> v1:172.21.15.81:6789/0 conn(0x7f604414f1c0 legacy=0x7f60440ca890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:10.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.627+0000 7f6043e00640 1 -- 172.21.15.28:0/3669030330 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60440c5fe0 con 0x7f60440bfa40 2024-02-21T23:03:10.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.626+0000 7f178ca00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f17880c5320 0x7f17881f2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:10.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.626+0000 7f178ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17880cad90 0x7f17881f34f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:10.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.626+0000 7f178ca00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f178813e480 0x7f1788070470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:10.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.626+0000 7f178ca00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17880c6580 con 0x7f17880cad90 2024-02-21T23:03:10.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.626+0000 7f178ca00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17880c62a0 con 0x7f178813e480 2024-02-21T23:03:10.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.626+0000 7f178ca00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f178813e860 con 0x7f17880c5320 2024-02-21T23:03:10.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.627+0000 7f1785e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f17880c5320 0x7f17881f2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:10.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.627+0000 7f1786800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f178813e480 0x7f1788070470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:10.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.627+0000 7f177d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17880cad90 0x7f17881f34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:10.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.627+0000 7f1786800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f178813e480 0x7f1788070470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.28:53590/0 (socket says 172.21.15.28:53590) 2024-02-21T23:03:10.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.627+0000 7f1786800640 1 -- 172.21.15.28:0/669468594 learned_addr learned my addr 172.21.15.28:0/669468594 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:10.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.628+0000 7f177d400640 1 -- 172.21.15.28:0/669468594 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f17880c5320 msgr2=0x7f17881f2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:10.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.628+0000 7f177d400640 1 --2- 172.21.15.28:0/669468594 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f17880c5320 0x7f17881f2fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:10.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.628+0000 7f177d400640 1 -- 172.21.15.28:0/669468594 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f178813e480 msgr2=0x7f1788070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:10.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.628+0000 7f177d400640 1 --2- 172.21.15.28:0/669468594 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f178813e480 0x7f1788070470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:10.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.628+0000 7f177d400640 1 -- 172.21.15.28:0/669468594 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17881f3a30 con 0x7f17880cad90 2024-02-21T23:03:10.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.629+0000 7f177d400640 1 --2- 172.21.15.28:0/669468594 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17880cad90 0x7f17881f34f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f1778033000 tx=0x7f17780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:10.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.629+0000 7f177ea00640 1 -- 172.21.15.28:0/669468594 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1778043070 con 0x7f17880cad90 2024-02-21T23:03:10.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.630+0000 7f177ea00640 1 -- 172.21.15.28:0/669468594 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1778034480 con 0x7f17880cad90 2024-02-21T23:03:10.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.628+0000 7f8e4e600640 1 Processor -- start 2024-02-21T23:03:10.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.629+0000 7f8e4e600640 1 -- start start 2024-02-21T23:03:10.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.630+0000 7f177ea00640 1 -- 172.21.15.28:0/669468594 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1778039040 con 0x7f17880cad90 2024-02-21T23:03:10.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.631+0000 7f8e4e600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8e4814ea70 0x7f8e4806ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:10.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.631+0000 7f8e4e600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f8e48155190 0x7f8e480c2f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:10.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.631+0000 7f8e4e600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8e480c34d0 0x7f8e48151520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:10.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.631+0000 7f8e4e600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e4806d110 con 0x7f8e4814ea70 2024-02-21T23:03:10.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.631+0000 7fe879800640 1 --2- 172.21.15.28:0/2635171193 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe85406b5a0 0x7fe85406da60 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe86c036000 tx=0x7fe86c008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:10.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.631+0000 7f8e4e600640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e4814ee50 con 0x7f8e48155190 2024-02-21T23:03:10.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.631+0000 7f8e4e600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e480c5e30 con 0x7f8e480c34d0 2024-02-21T23:03:10.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.631+0000 7f178ca00640 1 -- 172.21.15.28:0/669468594 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f178806edf0 con 0x7f17880cad90 2024-02-21T23:03:10.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.631+0000 7f8e47400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8e4814ea70 0x7f8e4806ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:10.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.631+0000 7f8e47400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8e4814ea70 0x7f8e4806ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46084/0 (socket says 172.21.15.28:46084) 2024-02-21T23:03:10.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.631+0000 7f8e47400640 1 -- 172.21.15.28:0/1458819456 learned_addr learned my addr 172.21.15.28:0/1458819456 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:10.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.631+0000 7f178ca00640 1 -- 172.21.15.28:0/669468594 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17880c2fe0 con 0x7f17880cad90 2024-02-21T23:03:10.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.630+0000 7f6043e00640 1 --2- 172.21.15.28:0/3669030330 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f60440bfa40 0x7f60440bfe70 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6044152ec0 tx=0x7f603402be70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a0ddd999867d2512 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:10.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.631+0000 7f6042000640 1 -- 172.21.15.28:0/3669030330 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6034039070 con 0x7f60440bfa40 2024-02-21T23:03:10.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.631+0000 7f6042000640 1 -- 172.21.15.28:0/3669030330 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f603402cc00 con 0x7f60440bfa40 2024-02-21T23:03:10.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.631+0000 7f6042000640 1 -- 172.21.15.28:0/3669030330 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6034034440 con 0x7f60440bfa40 2024-02-21T23:03:10.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.631+0000 7f8e46a00640 1 --2- 172.21.15.28:0/1458819456 >> v2:172.21.15.81:3300/0 conn(0x7f8e48155190 0x7f8e480c2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:10.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.632+0000 7f8e47e00640 1 --2- 172.21.15.28:0/1458819456 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8e480c34d0 0x7f8e48151520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:10.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.632+0000 7fe87a200640 1 --2- 172.21.15.28:0/2635171193 >> [v2:172.21.15.28:6800/2069258563,v1:172.21.15.28:6801/2069258563] conn(0x7fe854074d10 0x7fe854077150 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:10.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.632+0000 7fd471600640 1 -- 172.21.15.28:0/2793001054 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd46000bb80 con 0x7fd47414f1c0 2024-02-21T23:03:10.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.632+0000 7f8e47400640 1 -- 172.21.15.28:0/1458819456 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8e480c34d0 msgr2=0x7f8e48151520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:10.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.632+0000 7f8e47400640 1 --2- 172.21.15.28:0/1458819456 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8e480c34d0 0x7f8e48151520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:10.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.632+0000 7f8e47400640 1 -- 172.21.15.28:0/1458819456 >> v2:172.21.15.81:3300/0 conn(0x7f8e48155190 msgr2=0x7f8e480c2f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:10.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.632+0000 7f8e47400640 1 --2- 172.21.15.28:0/1458819456 >> v2:172.21.15.81:3300/0 conn(0x7f8e48155190 0x7f8e480c2f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:10.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.632+0000 7f8e47400640 1 -- 172.21.15.28:0/1458819456 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e480c6110 con 0x7f8e4814ea70 2024-02-21T23:03:10.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.633+0000 7f178ca00640 1 -- 172.21.15.28:0/669468594 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f178806eba0 con 0x7f17880cad90 2024-02-21T23:03:10.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.633+0000 7fd471600640 1 --2- 172.21.15.28:0/2793001054 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd45406b5d0 0x7fd45406da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:10.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.633+0000 7fd471600640 1 -- 172.21.15.28:0/2793001054 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fd46008b210 con 0x7fd47414f1c0 2024-02-21T23:03:10.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.634+0000 7fd471600640 1 --2- 172.21.15.28:0/2793001054 >> [v2:172.21.15.81:6819/2091269704,v1:172.21.15.81:6821/2091269704] conn(0x7fd454074d40 0x7fd454077180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:10.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.634+0000 7fd471600640 1 -- 172.21.15.28:0/2793001054 --> [v2:172.21.15.81:6819/2091269704,v1:172.21.15.81:6821/2091269704] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd460023e50 con 0x7fd454074d40 2024-02-21T23:03:10.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.634+0000 7fd471600640 1 -- 172.21.15.28:0/2793001054 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd46008b600 con 0x7fd47414f1c0 2024-02-21T23:03:10.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.634+0000 7fe871600640 1 -- 172.21.15.28:0/2635171193 <== osd.0 v2:172.21.15.28:6800/2069258563 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fe86801fec0 con 0x7fe854074d10 2024-02-21T23:03:10.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.632+0000 7f604a200640 1 -- 172.21.15.28:0/3669030330 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f60440bfa40 msgr2=0x7f60440bfe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:10.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.632+0000 7f604a200640 1 --2- 172.21.15.28:0/3669030330 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f60440bfa40 0x7f60440bfe70 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6044152ec0 tx=0x7f603402be70 comp rx=0 tx=0).stop 2024-02-21T23:03:10.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.634+0000 7f8e47400640 1 --2- 172.21.15.28:0/1458819456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e4814ea70 0x7f8e4806ec50 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8e480c5a60 tx=0x7f8e3402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a274ac58a9ef9a8 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:10.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.634+0000 7fd479600640 1 --2- 172.21.15.28:0/2793001054 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd45406b5d0 0x7fd45406da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:10.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.634+0000 7fd47a000640 1 --2- 172.21.15.28:0/2793001054 >> [v2:172.21.15.81:6819/2091269704,v1:172.21.15.81:6821/2091269704] conn(0x7fd454074d40 0x7fd454077180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:10.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.635+0000 7f604a200640 1 -- 172.21.15.28:0/3669030330 shutdown_connections 2024-02-21T23:03:10.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.635+0000 7f604a200640 1 --2- 172.21.15.28:0/3669030330 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f60440bfa40 0x7f60440bfe70 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:10.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.635+0000 7f604a200640 1 --2- 172.21.15.28:0/3669030330 >> v2:172.21.15.28:3300/0 conn(0x7f60441550d0 0x7f60440c5320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:10.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.635+0000 7f604a200640 1 -- 172.21.15.28:0/3669030330 >> 172.21.15.28:0/3669030330 conn(0x7f604408b1c0 msgr2=0x7f60440725a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:10.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.635+0000 7f8e46000640 1 -- 172.21.15.28:0/1458819456 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e34035020 con 0x7f8e4814ea70 2024-02-21T23:03:10.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.635+0000 7f8e46000640 1 -- 172.21.15.28:0/1458819456 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e34031830 con 0x7f8e4814ea70 2024-02-21T23:03:10.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.635+0000 7f8e46000640 1 -- 172.21.15.28:0/1458819456 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e34031c40 con 0x7f8e4814ea70 2024-02-21T23:03:10.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.635+0000 7f604a200640 1 -- 172.21.15.28:0/3669030330 shutdown_connections 2024-02-21T23:03:10.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.636+0000 7f604a200640 1 -- 172.21.15.28:0/3669030330 wait complete. 2024-02-21T23:03:10.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.636+0000 7f8e4e600640 1 -- 172.21.15.28:0/1458819456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e4814ea70 msgr2=0x7f8e4806ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:10.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.636+0000 7f8e4e600640 1 --2- 172.21.15.28:0/1458819456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e4814ea70 0x7f8e4806ec50 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8e480c5a60 tx=0x7f8e3402b8d0 comp rx=0 tx=0).stop 2024-02-21T23:03:10.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.636+0000 7f177ea00640 1 -- 172.21.15.28:0/669468594 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f177803c070 con 0x7f17880cad90 2024-02-21T23:03:10.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.636+0000 7fd479600640 1 --2- 172.21.15.28:0/2793001054 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd45406b5d0 0x7fd45406da90 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd47414fa30 tx=0x7fd468007e80 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:10.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.636+0000 7fd47a000640 1 --2- 172.21.15.28:0/2793001054 >> [v2:172.21.15.81:6819/2091269704,v1:172.21.15.81:6821/2091269704] conn(0x7fd454074d40 0x7fd454077180 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:10.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.637+0000 7f177ea00640 1 --2- 172.21.15.28:0/669468594 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f176406b520 0x7f176406d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:10.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.637+0000 7f8e4e600640 1 -- 172.21.15.28:0/1458819456 shutdown_connections 2024-02-21T23:03:10.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.637+0000 7f8e4e600640 1 --2- 172.21.15.28:0/1458819456 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8e480c34d0 0x7f8e48151520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:10.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.637+0000 7f8e4e600640 1 --2- 172.21.15.28:0/1458819456 >> v2:172.21.15.81:3300/0 conn(0x7f8e48155190 0x7f8e480c2f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:10.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.637+0000 7f8e4e600640 1 --2- 172.21.15.28:0/1458819456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e4814ea70 0x7f8e4806ec50 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:10.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.637+0000 7f8e4e600640 1 -- 172.21.15.28:0/1458819456 >> 172.21.15.28:0/1458819456 conn(0x7f8e4806b820 msgr2=0x7f8e480c67f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:10.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.637+0000 7f177ea00640 1 -- 172.21.15.28:0/669468594 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f17780abe60 con 0x7f17880cad90 2024-02-21T23:03:10.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.637+0000 7f177ea00640 1 --2- 172.21.15.28:0/669468594 >> [v2:172.21.15.81:6800/3369511790,v1:172.21.15.81:6801/3369511790] conn(0x7f1764074c90 0x7f17640770d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:10.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.638+0000 7f8e4e600640 1 -- 172.21.15.28:0/1458819456 shutdown_connections 2024-02-21T23:03:10.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.638+0000 7f8e4e600640 1 -- 172.21.15.28:0/1458819456 wait complete. 2024-02-21T23:03:10.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.638+0000 7f177ea00640 1 -- 172.21.15.28:0/669468594 --> [v2:172.21.15.81:6800/3369511790,v1:172.21.15.81:6801/3369511790] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1778001d30 con 0x7f1764074c90 2024-02-21T23:03:10.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.638+0000 7f177ea00640 1 -- 172.21.15.28:0/669468594 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f17780ae5b0 con 0x7f17880cad90 2024-02-21T23:03:10.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.638+0000 7f1785e00640 1 --2- 172.21.15.28:0/669468594 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f176406b520 0x7f176406d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:10.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.639+0000 7f1786800640 1 --2- 172.21.15.28:0/669468594 >> [v2:172.21.15.81:6800/3369511790,v1:172.21.15.81:6801/3369511790] conn(0x7f1764074c90 0x7f17640770d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:10.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.639+0000 7fd471600640 1 -- 172.21.15.28:0/2793001054 <== osd.4 v2:172.21.15.81:6819/2091269704 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd460023e50 con 0x7fd454074d40 2024-02-21T23:03:10.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.639+0000 7f604a200640 1 Processor -- start 2024-02-21T23:03:10.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.639+0000 7f604a200640 1 -- start start 2024-02-21T23:03:10.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.640+0000 7f1785e00640 1 --2- 172.21.15.28:0/669468594 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f176406b520 0x7f176406d9e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1770006ca0 tx=0x7f1770009240 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:10.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.641+0000 7f8e4e600640 1 Processor -- start 2024-02-21T23:03:10.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.641+0000 7f1786800640 1 --2- 172.21.15.28:0/669468594 >> [v2:172.21.15.81:6800/3369511790,v1:172.21.15.81:6801/3369511790] conn(0x7f1764074c90 0x7f17640770d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:10.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.641+0000 7f8e4e600640 1 -- start start 2024-02-21T23:03:10.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.642+0000 7f604a200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60440bfa40 0x7f6044144930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:10.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.642+0000 7f604a200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f604414f1c0 0x7f6044149460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:10.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.642+0000 7f604a200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f60441550d0 0x7f60441437f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:10.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.642+0000 7f6043400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60440bfa40 0x7f6044144930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:10.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.642+0000 7f6043400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60440bfa40 0x7f6044144930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46092/0 (socket says 172.21.15.28:46092) 2024-02-21T23:03:10.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.642+0000 7f6042a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f604414f1c0 0x7f6044149460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:10.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.642+0000 7f6042a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f604414f1c0 0x7f6044149460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.28:53618/0 (socket says 172.21.15.28:53618) 2024-02-21T23:03:10.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.643+0000 7f6043400640 1 -- 172.21.15.28:0/711548408 learned_addr learned my addr 172.21.15.28:0/711548408 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:10.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.643+0000 7f8e4e600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8e480c34d0 0x7f8e480a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:10.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.643+0000 7f8e4e600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e4814ea70 0x7f8e481f24a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:10.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.643+0000 7f604a200640 1 -- 172.21.15.28:0/711548408 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60440cafa0 con 0x7f60440bfa40 2024-02-21T23:03:10.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.643+0000 7f604a200640 1 -- 172.21.15.28:0/711548408 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60440c65a0 con 0x7f604414f1c0 2024-02-21T23:03:10.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.643+0000 7f6043e00640 1 --2- 172.21.15.28:0/711548408 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f60441550d0 0x7f60441437f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:10.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.643+0000 7f177ea00640 1 -- 172.21.15.28:0/669468594 <== osd.6 v2:172.21.15.81:6800/3369511790 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f1778001d30 con 0x7f1764074c90 2024-02-21T23:03:10.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.643+0000 7f604a200640 1 -- 172.21.15.28:0/711548408 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60440cb120 con 0x7f60441550d0 2024-02-21T23:03:10.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.643+0000 7f8e47400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8e480c34d0 0x7f8e480a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:10.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.643+0000 7f8e47400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8e480c34d0 0x7f8e480a50f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3301/0 says I am v2:172.21.15.28:40750/0 (socket says 172.21.15.28:40750) 2024-02-21T23:03:10.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.643+0000 7f8e47400640 1 -- 172.21.15.28:0/1163144080 learned_addr learned my addr 172.21.15.28:0/1163144080 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:10.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.644+0000 7f8e46a00640 1 --2- 172.21.15.28:0/1163144080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e4814ea70 0x7f8e481f24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:10.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.644+0000 7f8e4e600640 1 --2- 172.21.15.28:0/1163144080 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8e48155190 0x7f8e48153630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:10.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.644+0000 7f8e4e600640 1 -- 172.21.15.28:0/1163144080 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e4806d640 con 0x7f8e4814ea70 2024-02-21T23:03:10.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.644+0000 7f8e4e600640 1 -- 172.21.15.28:0/1163144080 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e48155610 con 0x7f8e48155190 2024-02-21T23:03:10.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.644+0000 7f8e4e600640 1 -- 172.21.15.28:0/1163144080 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e4806d7c0 con 0x7f8e480c34d0 2024-02-21T23:03:10.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.644+0000 7f8e47e00640 1 --2- 172.21.15.28:0/1163144080 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8e48155190 0x7f8e48153630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:10.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.645+0000 7f8e46a00640 1 -- 172.21.15.28:0/1163144080 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8e480c34d0 msgr2=0x7f8e480a50f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:10.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.645+0000 7f8e46a00640 1 --2- 172.21.15.28:0/1163144080 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8e480c34d0 0x7f8e480a50f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:10.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.645+0000 7f8e46a00640 1 -- 172.21.15.28:0/1163144080 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8e48155190 msgr2=0x7f8e48153630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:10.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.645+0000 7f8e46a00640 1 --2- 172.21.15.28:0/1163144080 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8e48155190 0x7f8e48153630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:10.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.645+0000 7f8e46a00640 1 -- 172.21.15.28:0/1163144080 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e481f2ce0 con 0x7f8e4814ea70 2024-02-21T23:03:10.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.646+0000 7f6043400640 1 -- 172.21.15.28:0/711548408 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f60441550d0 msgr2=0x7f60441437f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:10.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.646+0000 7f6043400640 1 --2- 172.21.15.28:0/711548408 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f60441550d0 0x7f60441437f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:10.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.646+0000 7f6043400640 1 -- 172.21.15.28:0/711548408 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f604414f1c0 msgr2=0x7f6044149460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:10.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.646+0000 7f6043400640 1 --2- 172.21.15.28:0/711548408 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f604414f1c0 0x7f6044149460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:10.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.646+0000 7f6043400640 1 -- 172.21.15.28:0/711548408 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60440c9820 con 0x7f60440bfa40 2024-02-21T23:03:10.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.646+0000 7f8e46a00640 1 --2- 172.21.15.28:0/1163144080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e4814ea70 0x7f8e481f24a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f8e3800e640 tx=0x7f8e3800eb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:10.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.646+0000 7f6042a00640 1 --2- 172.21.15.28:0/711548408 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f604414f1c0 0x7f6044149460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-21T23:03:10.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.646+0000 7f8e3f400640 1 -- 172.21.15.28:0/1163144080 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e38016070 con 0x7f8e4814ea70 2024-02-21T23:03:10.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.647+0000 7f6043400640 1 --2- 172.21.15.28:0/711548408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60440bfa40 0x7f6044144930 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f603c00c530 tx=0x7f603c00c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:10.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.646+0000 7f8e3f400640 1 -- 172.21.15.28:0/1163144080 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e3800c990 con 0x7f8e4814ea70 2024-02-21T23:03:10.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.647+0000 7f8e3f400640 1 -- 172.21.15.28:0/1163144080 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e3800b040 con 0x7f8e4814ea70 2024-02-21T23:03:10.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.647+0000 7f8e4e600640 1 -- 172.21.15.28:0/1163144080 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e481f2fc0 con 0x7f8e4814ea70 2024-02-21T23:03:10.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.647+0000 7f8e4e600640 1 -- 172.21.15.28:0/1163144080 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e481f29e0 con 0x7f8e4814ea70 2024-02-21T23:03:10.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.649+0000 7f603b400640 1 -- 172.21.15.28:0/711548408 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f603c017070 con 0x7f60440bfa40 2024-02-21T23:03:10.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.649+0000 7f603b400640 1 -- 172.21.15.28:0/711548408 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f603c005ce0 con 0x7f60440bfa40 2024-02-21T23:03:10.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.649+0000 7f603b400640 1 -- 172.21.15.28:0/711548408 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f603c0072b0 con 0x7f60440bfa40 2024-02-21T23:03:10.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.649+0000 7f8e4e600640 1 -- 172.21.15.28:0/1163144080 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8e481519e0 con 0x7f8e4814ea70 2024-02-21T23:03:10.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.649+0000 7f604a200640 1 -- 172.21.15.28:0/711548408 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60440c9b00 con 0x7f60440bfa40 2024-02-21T23:03:10.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.649+0000 7f604a200640 1 -- 172.21.15.28:0/711548408 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60440ca0c0 con 0x7f60440bfa40 2024-02-21T23:03:10.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.653+0000 7f604a200640 1 -- 172.21.15.28:0/711548408 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f60440c95d0 con 0x7f60440bfa40 2024-02-21T23:03:10.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.654+0000 7f8e3f400640 1 -- 172.21.15.28:0/1163144080 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8e38019070 con 0x7f8e4814ea70 2024-02-21T23:03:10.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.654+0000 7f603b400640 1 -- 172.21.15.28:0/711548408 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f603c01a070 con 0x7f60440bfa40 2024-02-21T23:03:10.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.655+0000 7f603b400640 1 --2- 172.21.15.28:0/711548408 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f602406b520 0x7f602406d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:10.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.655+0000 7f603b400640 1 -- 172.21.15.28:0/711548408 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f603c089850 con 0x7f60440bfa40 2024-02-21T23:03:10.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.656+0000 7f603b400640 1 --2- 172.21.15.28:0/711548408 >> [v2:172.21.15.81:6808/568898662,v1:172.21.15.81:6809/568898662] conn(0x7f6024074ca0 0x7f60240770e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:10.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.656+0000 7f603b400640 1 -- 172.21.15.28:0/711548408 --> [v2:172.21.15.81:6808/568898662,v1:172.21.15.81:6809/568898662] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f603c00ae50 con 0x7f6024074ca0 2024-02-21T23:03:10.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.656+0000 7f603b400640 1 -- 172.21.15.28:0/711548408 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f603c08e2c0 con 0x7f60440bfa40 2024-02-21T23:03:10.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.655+0000 7f8e3f400640 1 --2- 172.21.15.28:0/1163144080 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8e2806b520 0x7f8e2806d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:10.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.656+0000 7f8e3f400640 1 -- 172.21.15.28:0/1163144080 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f8e3808a130 con 0x7f8e4814ea70 2024-02-21T23:03:10.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.657+0000 7f6042a00640 1 --2- 172.21.15.28:0/711548408 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f602406b520 0x7f602406d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:10.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.657+0000 7f8e47400640 1 --2- 172.21.15.28:0/1163144080 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8e2806b520 0x7f8e2806d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:10.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.656+0000 7fc4b5600640 1 -- 172.21.15.28:0/2439353557 --> [v2:172.21.15.28:6809/160098884,v1:172.21.15.28:6812/160098884] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc4a8005710 con 0x7fc490074c90 2024-02-21T23:03:10.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.657+0000 7f8e3f400640 1 --2- 172.21.15.28:0/1163144080 >> [v2:172.21.15.81:6816/3022012384,v1:172.21.15.81:6817/3022012384] conn(0x7f8e28074ca0 0x7f8e280770e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:10.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.657+0000 7f8e3f400640 1 -- 172.21.15.28:0/1163144080 --> [v2:172.21.15.81:6816/3022012384,v1:172.21.15.81:6817/3022012384] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8e38009e90 con 0x7f8e28074ca0 2024-02-21T23:03:10.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.657+0000 7f8e3f400640 1 -- 172.21.15.28:0/1163144080 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8e38091050 con 0x7f8e4814ea70 2024-02-21T23:03:10.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.657+0000 7f6043e00640 1 --2- 172.21.15.28:0/711548408 >> [v2:172.21.15.81:6808/568898662,v1:172.21.15.81:6809/568898662] conn(0x7f6024074ca0 0x7f60240770e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:10.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.657+0000 7fc4a3e00640 1 -- 172.21.15.28:0/2439353557 <== osd.2 v2:172.21.15.28:6809/160098884 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fc4a8005710 con 0x7fc490074c90 2024-02-21T23:03:10.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.658+0000 7f8e47e00640 1 --2- 172.21.15.28:0/1163144080 >> [v2:172.21.15.81:6816/3022012384,v1:172.21.15.81:6817/3022012384] conn(0x7f8e28074ca0 0x7f8e280770e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:10.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.658+0000 7f6043e00640 1 --2- 172.21.15.28:0/711548408 >> [v2:172.21.15.81:6808/568898662,v1:172.21.15.81:6809/568898662] conn(0x7f6024074ca0 0x7f60240770e0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:10.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.658+0000 7fc4b5600640 1 -- 172.21.15.28:0/2439353557 >> [v2:172.21.15.28:6809/160098884,v1:172.21.15.28:6812/160098884] conn(0x7fc490074c90 msgr2=0x7fc4900770d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:10.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.658+0000 7f6042a00640 1 --2- 172.21.15.28:0/711548408 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f602406b520 0x7f602406d9e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f6044148740 tx=0x7f6030009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:10.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.658+0000 7fc4b5600640 1 --2- 172.21.15.28:0/2439353557 >> [v2:172.21.15.28:6809/160098884,v1:172.21.15.28:6812/160098884] conn(0x7fc490074c90 0x7fc4900770d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:10.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.658+0000 7fc4b5600640 1 -- 172.21.15.28:0/2439353557 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc49006b520 msgr2=0x7fc49006d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:10.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.658+0000 7fc4b5600640 1 --2- 172.21.15.28:0/2439353557 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc49006b520 0x7fc49006d9e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc4a817a920 tx=0x7fc4a4009060 comp rx=0 tx=0).stop 2024-02-21T23:03:10.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.658+0000 7fc4b5600640 1 -- 172.21.15.28:0/2439353557 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc4a80dbd30 msgr2=0x7fc4a80dcb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:10.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.658+0000 7fc4b5600640 1 --2- 172.21.15.28:0/2439353557 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc4a80dbd30 0x7fc4a80dcb80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc4b00685e0 tx=0x7fc4b006a210 comp rx=0 tx=0).stop 2024-02-21T23:03:10.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.658+0000 7f8e47400640 1 --2- 172.21.15.28:0/1163144080 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8e2806b520 0x7f8e2806d9e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8e3402b910 tx=0x7f8e3403c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:10.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.658+0000 7fc4b4c00640 1 -- 172.21.15.28:0/2439353557 reap_dead start 2024-02-21T23:03:10.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.659+0000 7f8e47e00640 1 --2- 172.21.15.28:0/1163144080 >> [v2:172.21.15.81:6816/3022012384,v1:172.21.15.81:6817/3022012384] conn(0x7f8e28074ca0 0x7f8e280770e0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:10.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.659+0000 7fc4b5600640 1 -- 172.21.15.28:0/2439353557 shutdown_connections 2024-02-21T23:03:10.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.659+0000 7fc4b5600640 1 -- 172.21.15.28:0/2439353557 >> 172.21.15.28:0/2439353557 conn(0x7fc4a802f560 msgr2=0x7fc4a800cf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:10.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.659+0000 7fc4b5600640 1 -- 172.21.15.28:0/2439353557 shutdown_connections 2024-02-21T23:03:10.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.659+0000 7fc4b5600640 1 -- 172.21.15.28:0/2439353557 wait complete. 2024-02-21T23:03:10.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.660+0000 7f603b400640 1 -- 172.21.15.28:0/711548408 <== osd.5 v2:172.21.15.81:6808/568898662 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f603c00ae50 con 0x7f6024074ca0 2024-02-21T23:03:10.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.661+0000 7f8e3f400640 1 -- 172.21.15.28:0/1163144080 <== osd.7 v2:172.21.15.81:6816/3022012384 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f8e38009e90 con 0x7f8e28074ca0 2024-02-21T23:03:10.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.663+0000 7fe87c200640 1 -- 172.21.15.28:0/2635171193 --> [v2:172.21.15.28:6800/2069258563,v1:172.21.15.28:6801/2069258563] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe87404f290 con 0x7fe854074d10 2024-02-21T23:03:10.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.665+0000 7fe871600640 1 -- 172.21.15.28:0/2635171193 <== osd.0 v2:172.21.15.28:6800/2069258563 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fe87404f290 con 0x7fe854074d10 2024-02-21T23:03:10.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.665+0000 7fe87c200640 1 -- 172.21.15.28:0/2635171193 >> [v2:172.21.15.28:6800/2069258563,v1:172.21.15.28:6801/2069258563] conn(0x7fe854074d10 msgr2=0x7fe854077150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:10.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.665+0000 7fe87c200640 1 --2- 172.21.15.28:0/2635171193 >> [v2:172.21.15.28:6800/2069258563,v1:172.21.15.28:6801/2069258563] conn(0x7fe854074d10 0x7fe854077150 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:10.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.666+0000 7fe87c200640 1 -- 172.21.15.28:0/2635171193 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe85406b5a0 msgr2=0x7fe85406da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:10.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.666+0000 7fe87c200640 1 --2- 172.21.15.28:0/2635171193 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe85406b5a0 0x7fe85406da60 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe86c036000 tx=0x7fe86c008030 comp rx=0 tx=0).stop 2024-02-21T23:03:10.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.666+0000 7fe87c200640 1 -- 172.21.15.28:0/2635171193 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe8740c8780 msgr2=0x7fe8741ebb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:10.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.666+0000 7fe87c200640 1 --2- 172.21.15.28:0/2635171193 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe8740c8780 0x7fe8741ebb50 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe86800c7a0 tx=0x7fe86800cc60 comp rx=0 tx=0).stop 2024-02-21T23:03:10.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.666+0000 7f39af400640 1 -- 172.21.15.28:0/3058958991 --> [v2:172.21.15.28:6808/2193666734,v1:172.21.15.28:6810/2193666734] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f39a814ebe0 con 0x7f3988074ca0 2024-02-21T23:03:10.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.666+0000 7fe87a200640 1 -- 172.21.15.28:0/2635171193 reap_dead start 2024-02-21T23:03:10.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.666+0000 7fe87c200640 1 -- 172.21.15.28:0/2635171193 shutdown_connections 2024-02-21T23:03:10.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.666+0000 7fe87c200640 1 -- 172.21.15.28:0/2635171193 >> 172.21.15.28:0/2635171193 conn(0x7fe8740ac3b0 msgr2=0x7fe8740ac790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:10.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.666+0000 7f7aee600640 1 -- 172.21.15.28:0/1861662920 --> [v2:172.21.15.28:6824/3053508071,v1:172.21.15.28:6825/3053508071] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7ae00052c0 con 0x7f7ac8074d20 2024-02-21T23:03:10.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.667+0000 7f39a4c00640 1 -- 172.21.15.28:0/3058958991 <== osd.1 v2:172.21.15.28:6808/2193666734 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f39a814ebe0 con 0x7f3988074ca0 2024-02-21T23:03:10.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.667+0000 7f7ae4c00640 1 -- 172.21.15.28:0/1861662920 <== osd.3 v2:172.21.15.28:6824/3053508071 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f7ae00052c0 con 0x7f7ac8074d20 2024-02-21T23:03:10.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.667+0000 7fe87c200640 1 -- 172.21.15.28:0/2635171193 shutdown_connections 2024-02-21T23:03:10.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.667+0000 7fe87c200640 1 -- 172.21.15.28:0/2635171193 wait complete. 2024-02-21T23:03:10.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.667+0000 7f39af400640 1 -- 172.21.15.28:0/3058958991 >> [v2:172.21.15.28:6808/2193666734,v1:172.21.15.28:6810/2193666734] conn(0x7f3988074ca0 msgr2=0x7f39880770e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:10.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.668+0000 7f7aee600640 1 -- 172.21.15.28:0/1861662920 >> [v2:172.21.15.28:6824/3053508071,v1:172.21.15.28:6825/3053508071] conn(0x7f7ac8074d20 msgr2=0x7f7ac8077160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:10.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.668+0000 7f7aee600640 1 --2- 172.21.15.28:0/1861662920 >> [v2:172.21.15.28:6824/3053508071,v1:172.21.15.28:6825/3053508071] conn(0x7f7ac8074d20 0x7f7ac8077160 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:10.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.668+0000 7f39af400640 1 --2- 172.21.15.28:0/3058958991 >> [v2:172.21.15.28:6808/2193666734,v1:172.21.15.28:6810/2193666734] conn(0x7f3988074ca0 0x7f39880770e0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:10.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.668+0000 7f39af400640 1 -- 172.21.15.28:0/3058958991 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f398806b520 msgr2=0x7f398806d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:10.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.668+0000 7f39af400640 1 --2- 172.21.15.28:0/3058958991 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f398806b520 0x7f398806d9e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f39a80aba00 tx=0x7f39940014b0 comp rx=0 tx=0).stop 2024-02-21T23:03:10.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.668+0000 7f39af400640 1 -- 172.21.15.28:0/3058958991 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f39a806b8b0 msgr2=0x7f39a81e7620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:10.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.668+0000 7f39af400640 1 --2- 172.21.15.28:0/3058958991 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f39a806b8b0 0x7f39a81e7620 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f399c02b9a0 tx=0x7f399c0046c0 comp rx=0 tx=0).stop 2024-02-21T23:03:10.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.669+0000 7f39ad400640 1 -- 172.21.15.28:0/3058958991 reap_dead start 2024-02-21T23:03:10.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.669+0000 7f39af400640 1 -- 172.21.15.28:0/3058958991 shutdown_connections 2024-02-21T23:03:10.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.669+0000 7f39af400640 1 -- 172.21.15.28:0/3058958991 >> 172.21.15.28:0/3058958991 conn(0x7f39a806b4a0 msgr2=0x7f39a8150850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:10.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.669+0000 7f39af400640 1 -- 172.21.15.28:0/3058958991 shutdown_connections 2024-02-21T23:03:10.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.670+0000 7f7aee600640 1 -- 172.21.15.28:0/1861662920 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7ac806b5a0 msgr2=0x7f7ac806da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:10.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.670+0000 7f7aee600640 1 --2- 172.21.15.28:0/1861662920 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7ac806b5a0 0x7f7ac806da60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7ad400da00 tx=0x7f7ad400f040 comp rx=0 tx=0).stop 2024-02-21T23:03:10.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.670+0000 7f7aee600640 1 -- 172.21.15.28:0/1861662920 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ae00cda50 msgr2=0x7f7ae0170910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:10.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.670+0000 7f7aee600640 1 --2- 172.21.15.28:0/1861662920 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ae00cda50 0x7f7ae0170910 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7adc033000 tx=0x7f7adc0046c0 comp rx=0 tx=0).stop 2024-02-21T23:03:10.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.670+0000 7f7aedc00640 1 -- 172.21.15.28:0/1861662920 reap_dead start 2024-02-21T23:03:10.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.670+0000 7f7aee600640 1 -- 172.21.15.28:0/1861662920 shutdown_connections 2024-02-21T23:03:10.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.670+0000 7f7aee600640 1 -- 172.21.15.28:0/1861662920 >> 172.21.15.28:0/1861662920 conn(0x7f7ae0048250 msgr2=0x7f7ae0006690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:10.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.671+0000 7f7aee600640 1 -- 172.21.15.28:0/1861662920 shutdown_connections 2024-02-21T23:03:10.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.671+0000 7f39af400640 1 -- 172.21.15.28:0/3058958991 wait complete. 2024-02-21T23:03:10.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.674+0000 7f7aee600640 1 -- 172.21.15.28:0/1861662920 wait complete. 2024-02-21T23:03:10.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.674+0000 7fd47c000640 1 -- 172.21.15.28:0/2793001054 --> [v2:172.21.15.81:6819/2091269704,v1:172.21.15.81:6821/2091269704] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd47404f370 con 0x7fd454074d40 2024-02-21T23:03:10.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.675+0000 7fd471600640 1 -- 172.21.15.28:0/2793001054 <== osd.4 v2:172.21.15.81:6819/2091269704 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fd47404f370 con 0x7fd454074d40 2024-02-21T23:03:10.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.676+0000 7fd47c000640 1 -- 172.21.15.28:0/2793001054 >> [v2:172.21.15.81:6819/2091269704,v1:172.21.15.81:6821/2091269704] conn(0x7fd454074d40 msgr2=0x7fd454077180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:10.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.676+0000 7fd47c000640 1 --2- 172.21.15.28:0/2793001054 >> [v2:172.21.15.81:6819/2091269704,v1:172.21.15.81:6821/2091269704] conn(0x7fd454074d40 0x7fd454077180 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:10.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.677+0000 7fd47c000640 1 -- 172.21.15.28:0/2793001054 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd45406b5d0 msgr2=0x7fd45406da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:10.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.677+0000 7fd47c000640 1 --2- 172.21.15.28:0/2793001054 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd45406b5d0 0x7fd45406da90 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd47414fa30 tx=0x7fd468007e80 comp rx=0 tx=0).stop 2024-02-21T23:03:10.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.677+0000 7fd47c000640 1 -- 172.21.15.28:0/2793001054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd47414f1c0 msgr2=0x7fd474147f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:10.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.677+0000 7fd47c000640 1 --2- 172.21.15.28:0/2793001054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd47414f1c0 0x7fd474147f90 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fd46000d620 tx=0x7fd46000dae0 comp rx=0 tx=0).stop 2024-02-21T23:03:10.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.677+0000 7fd47a000640 1 -- 172.21.15.28:0/2793001054 reap_dead start 2024-02-21T23:03:10.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.677+0000 7fd47c000640 1 -- 172.21.15.28:0/2793001054 shutdown_connections 2024-02-21T23:03:10.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.677+0000 7fd47c000640 1 -- 172.21.15.28:0/2793001054 >> 172.21.15.28:0/2793001054 conn(0x7fd47408b1c0 msgr2=0x7fd4740c4770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:10.691 INFO:teuthology.orchestra.run.smithi028.stdout:55834574859 2024-02-21T23:03:10.691 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-02-21T23:03:10.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.680+0000 7fd47c000640 1 -- 172.21.15.28:0/2793001054 shutdown_connections 2024-02-21T23:03:10.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.680+0000 7fd47c000640 1 -- 172.21.15.28:0/2793001054 wait complete. 2024-02-21T23:03:10.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.680+0000 7f178ca00640 1 -- 172.21.15.28:0/669468594 --> [v2:172.21.15.81:6800/3369511790,v1:172.21.15.81:6801/3369511790] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f17880b0d30 con 0x7f1764074c90 2024-02-21T23:03:10.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.681+0000 7f177ea00640 1 -- 172.21.15.28:0/669468594 <== osd.6 v2:172.21.15.81:6800/3369511790 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f17880b0d30 con 0x7f1764074c90 2024-02-21T23:03:10.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.682+0000 7f178ca00640 1 -- 172.21.15.28:0/669468594 >> [v2:172.21.15.81:6800/3369511790,v1:172.21.15.81:6801/3369511790] conn(0x7f1764074c90 msgr2=0x7f17640770d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:10.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.682+0000 7f178ca00640 1 --2- 172.21.15.28:0/669468594 >> [v2:172.21.15.81:6800/3369511790,v1:172.21.15.81:6801/3369511790] conn(0x7f1764074c90 0x7f17640770d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:10.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.682+0000 7f178ca00640 1 -- 172.21.15.28:0/669468594 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f176406b520 msgr2=0x7f176406d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:10.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.682+0000 7f178ca00640 1 --2- 172.21.15.28:0/669468594 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f176406b520 0x7f176406d9e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1770006ca0 tx=0x7f1770009240 comp rx=0 tx=0).stop 2024-02-21T23:03:10.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.682+0000 7f178ca00640 1 -- 172.21.15.28:0/669468594 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17880cad90 msgr2=0x7f17881f34f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:10.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.682+0000 7f178ca00640 1 --2- 172.21.15.28:0/669468594 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17880cad90 0x7f17881f34f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f1778033000 tx=0x7f17780046c0 comp rx=0 tx=0).stop 2024-02-21T23:03:10.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.682+0000 7f1786800640 1 -- 172.21.15.28:0/669468594 reap_dead start 2024-02-21T23:03:10.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.682+0000 7f178ca00640 1 -- 172.21.15.28:0/669468594 shutdown_connections 2024-02-21T23:03:10.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.682+0000 7f178ca00640 1 -- 172.21.15.28:0/669468594 >> 172.21.15.28:0/669468594 conn(0x7f178808b1c0 msgr2=0x7f178813a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:10.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.683+0000 7f178ca00640 1 -- 172.21.15.28:0/669468594 shutdown_connections 2024-02-21T23:03:10.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.683+0000 7f178ca00640 1 -- 172.21.15.28:0/669468594 wait complete. 2024-02-21T23:03:10.694 INFO:teuthology.orchestra.run.smithi028.stdout:55834574857 2024-02-21T23:03:10.695 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-02-21T23:03:10.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.694+0000 7f8e4e600640 1 -- 172.21.15.28:0/1163144080 --> [v2:172.21.15.81:6816/3022012384,v1:172.21.15.81:6817/3022012384] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8e4806f400 con 0x7f8e28074ca0 2024-02-21T23:03:10.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.694+0000 7f604a200640 1 -- 172.21.15.28:0/711548408 --> [v2:172.21.15.81:6808/568898662,v1:172.21.15.81:6809/568898662] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f604406b2e0 con 0x7f6024074ca0 2024-02-21T23:03:10.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.695+0000 7f8e3f400640 1 -- 172.21.15.28:0/1163144080 <== osd.7 v2:172.21.15.81:6816/3022012384 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f8e4806f400 con 0x7f8e28074ca0 2024-02-21T23:03:10.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.695+0000 7f603b400640 1 -- 172.21.15.28:0/711548408 <== osd.5 v2:172.21.15.81:6808/568898662 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f604406b2e0 con 0x7f6024074ca0 2024-02-21T23:03:10.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.695+0000 7f8e4e600640 1 -- 172.21.15.28:0/1163144080 >> [v2:172.21.15.81:6816/3022012384,v1:172.21.15.81:6817/3022012384] conn(0x7f8e28074ca0 msgr2=0x7f8e280770e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:10.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.695+0000 7f8e4e600640 1 --2- 172.21.15.28:0/1163144080 >> [v2:172.21.15.81:6816/3022012384,v1:172.21.15.81:6817/3022012384] conn(0x7f8e28074ca0 0x7f8e280770e0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:10.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.695+0000 7f8e4e600640 1 -- 172.21.15.28:0/1163144080 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8e2806b520 msgr2=0x7f8e2806d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:10.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.695+0000 7f8e4e600640 1 --2- 172.21.15.28:0/1163144080 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8e2806b520 0x7f8e2806d9e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8e3402b910 tx=0x7f8e3403c040 comp rx=0 tx=0).stop 2024-02-21T23:03:10.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.695+0000 7f8e4e600640 1 -- 172.21.15.28:0/1163144080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e4814ea70 msgr2=0x7f8e481f24a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:10.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.695+0000 7f8e4e600640 1 --2- 172.21.15.28:0/1163144080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e4814ea70 0x7f8e481f24a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f8e3800e640 tx=0x7f8e3800eb00 comp rx=0 tx=0).stop 2024-02-21T23:03:10.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.696+0000 7f604a200640 1 -- 172.21.15.28:0/711548408 >> [v2:172.21.15.81:6808/568898662,v1:172.21.15.81:6809/568898662] conn(0x7f6024074ca0 msgr2=0x7f60240770e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:10.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.696+0000 7f604a200640 1 --2- 172.21.15.28:0/711548408 >> [v2:172.21.15.81:6808/568898662,v1:172.21.15.81:6809/568898662] conn(0x7f6024074ca0 0x7f60240770e0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:10.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.696+0000 7f8e47e00640 1 -- 172.21.15.28:0/1163144080 reap_dead start 2024-02-21T23:03:10.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.696+0000 7f8e4e600640 1 -- 172.21.15.28:0/1163144080 shutdown_connections 2024-02-21T23:03:10.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.696+0000 7f8e4e600640 1 -- 172.21.15.28:0/1163144080 >> 172.21.15.28:0/1163144080 conn(0x7f8e4806b820 msgr2=0x7f8e480c2480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:10.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.696+0000 7f604a200640 1 -- 172.21.15.28:0/711548408 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f602406b520 msgr2=0x7f602406d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:10.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.696+0000 7f604a200640 1 --2- 172.21.15.28:0/711548408 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f602406b520 0x7f602406d9e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f6044148740 tx=0x7f6030009190 comp rx=0 tx=0).stop 2024-02-21T23:03:10.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.696+0000 7f604a200640 1 -- 172.21.15.28:0/711548408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60440bfa40 msgr2=0x7f6044144930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:10.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.696+0000 7f604a200640 1 --2- 172.21.15.28:0/711548408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60440bfa40 0x7f6044144930 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f603c00c530 tx=0x7f603c00c9f0 comp rx=0 tx=0).stop 2024-02-21T23:03:10.699 INFO:teuthology.orchestra.run.smithi028.stdout:55834574858 2024-02-21T23:03:10.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.696+0000 7f6043e00640 1 -- 172.21.15.28:0/711548408 reap_dead start 2024-02-21T23:03:10.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.696+0000 7f604a200640 1 -- 172.21.15.28:0/711548408 shutdown_connections 2024-02-21T23:03:10.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.696+0000 7f604a200640 1 -- 172.21.15.28:0/711548408 >> 172.21.15.28:0/711548408 conn(0x7f604408b1c0 msgr2=0x7f60440c1030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:10.700 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-02-21T23:03:10.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.696+0000 7f604a200640 1 -- 172.21.15.28:0/711548408 shutdown_connections 2024-02-21T23:03:10.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.696+0000 7f8e4e600640 1 -- 172.21.15.28:0/1163144080 shutdown_connections 2024-02-21T23:03:10.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.696+0000 7f8e4e600640 1 -- 172.21.15.28:0/1163144080 wait complete. 2024-02-21T23:03:10.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:10.697+0000 7f604a200640 1 -- 172.21.15.28:0/711548408 wait complete. 2024-02-21T23:03:10.702 INFO:teuthology.orchestra.run.smithi028.stdout:55834574857 2024-02-21T23:03:10.703 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-02-21T23:03:10.703 INFO:teuthology.orchestra.run.smithi028.stdout:55834574858 2024-02-21T23:03:10.704 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-02-21T23:03:10.704 INFO:teuthology.orchestra.run.smithi028.stdout:55834574858 2024-02-21T23:03:10.705 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-02-21T23:03:10.710 INFO:teuthology.orchestra.run.smithi028.stdout:55834574857 2024-02-21T23:03:10.711 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-02-21T23:03:10.711 INFO:teuthology.orchestra.run.smithi028.stdout:55834574857 2024-02-21T23:03:10.714 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-02-21T23:03:10.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.006+0000 7fb552000640 1 Processor -- start 2024-02-21T23:03:10.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.006+0000 7f4586800640 1 Processor -- start 2024-02-21T23:03:10.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.006+0000 7fb552000640 1 -- start start 2024-02-21T23:03:10.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.007+0000 7f4586800640 1 -- start start 2024-02-21T23:03:10.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.009+0000 7f4586800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4580072e60 0x7f45800c4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:10.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.009+0000 7f4586800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f45800c11f0 0x7f45800c5010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:10.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.009+0000 7f4586800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4580159fd0 con 0x7f4580156c80 2024-02-21T23:03:10.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.009+0000 7f4586800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45800c5b60 con 0x7f4580072e60 2024-02-21T23:03:10.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.009+0000 7f4586800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45800c5db0 con 0x7f45800c11f0 2024-02-21T23:03:10.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.009+0000 7fb552000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fb54404e0f0 0x7fb54404e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:10.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.010+0000 7f457f400640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f4580156c80 0x7f458006e840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.28:50176/0 (socket says 172.21.15.28:50176) 2024-02-21T23:03:10.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.010+0000 7f457f400640 1 -- 172.21.15.28:0/2709330297 learned_addr learned my addr 172.21.15.28:0/2709330297 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:03:10.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.010+0000 7fb550c00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb5440d7b60 0x7fb544007200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:48588/0 (socket says 172.21.15.28:48588) 2024-02-21T23:03:10.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.010+0000 7fb552000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb54404ede0 0x7fb5440e4d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:10.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.010+0000 7f457fe00640 1 --2- 172.21.15.28:0/2709330297 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f45800c11f0 0x7f45800c5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:11.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.010+0000 7fb550c00640 1 -- 172.21.15.28:0/3375977556 learned_addr learned my addr 172.21.15.28:0/3375977556 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:03:11.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.011+0000 7f457e000640 1 -- 172.21.15.28:0/2709330297 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 783848054 0 0) 0x7f4580159fd0 con 0x7f4580156c80 2024-02-21T23:03:11.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.011+0000 7fb552000640 1 -- 172.21.15.28:0/3375977556 --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb54400fd60 con 0x7fb5440d7b60 2024-02-21T23:03:11.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.011+0000 7fb552000640 1 -- 172.21.15.28:0/3375977556 --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb54400f8e0 con 0x7fb54404e0f0 2024-02-21T23:03:11.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.011+0000 7fb552000640 1 -- 172.21.15.28:0/3375977556 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5440da500 con 0x7fb54404ede0 2024-02-21T23:03:11.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.011+0000 7fb551600640 1 --2- 172.21.15.28:0/3375977556 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb54404ede0 0x7fb5440e4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:11.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.011+0000 7f457e000640 1 -- 172.21.15.28:0/2709330297 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4564003520 con 0x7f4580156c80 2024-02-21T23:03:11.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.011+0000 7fb54be00640 1 --2- 172.21.15.28:0/3375977556 >> v2:172.21.15.81:3300/0 conn(0x7fb54404e0f0 0x7fb54404e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:11.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.011+0000 7f457ea00640 1 --2- 172.21.15.28:0/2709330297 >> v2:172.21.15.28:3300/0 conn(0x7f4580072e60 0x7f45800c4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:11.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.012+0000 7f457fe00640 1 -- 172.21.15.28:0/2709330297 >> v2:172.21.15.28:3300/0 conn(0x7f4580072e60 msgr2=0x7f45800c4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:11.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.011+0000 7fb54b400640 1 -- 172.21.15.28:0/3375977556 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1546297188 0 0) 0x7fb54400fd60 con 0x7fb5440d7b60 2024-02-21T23:03:11.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.012+0000 7f457fe00640 1 --2- 172.21.15.28:0/2709330297 >> v2:172.21.15.28:3300/0 conn(0x7f4580072e60 0x7f45800c4ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.011+0000 7fb54b400640 1 -- 172.21.15.28:0/3375977556 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb534003520 con 0x7fb5440d7b60 2024-02-21T23:03:11.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.012+0000 7fb551600640 1 -- 172.21.15.28:0/3375977556 >> v2:172.21.15.81:3300/0 conn(0x7fb54404e0f0 msgr2=0x7fb54404e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:11.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.012+0000 7fb551600640 1 --2- 172.21.15.28:0/3375977556 >> v2:172.21.15.81:3300/0 conn(0x7fb54404e0f0 0x7fb54404e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.012+0000 7fb551600640 1 -- 172.21.15.28:0/3375977556 >> v1:172.21.15.28:6789/0 conn(0x7fb5440d7b60 legacy=0x7fb544007200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:11.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.013+0000 7f457fe00640 1 -- 172.21.15.28:0/2709330297 >> v1:172.21.15.81:6789/0 conn(0x7f4580156c80 legacy=0x7f458006e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:11.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.012+0000 7fb551600640 1 -- 172.21.15.28:0/3375977556 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5440da7e0 con 0x7fb54404ede0 2024-02-21T23:03:11.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.013+0000 7f457fe00640 1 -- 172.21.15.28:0/2709330297 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f458015a290 con 0x7f45800c11f0 2024-02-21T23:03:11.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.013+0000 7fb54be00640 1 --2- 172.21.15.28:0/3375977556 >> v2:172.21.15.81:3300/0 conn(0x7fb54404e0f0 0x7fb54404e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:03:11.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.015+0000 7fb551600640 1 --2- 172.21.15.28:0/3375977556 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb54404ede0 0x7fb5440e4d70 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb54c050040 tx=0x7fb54c0a6830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=891f1005882fca49 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:11.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.015+0000 7f457fe00640 1 --2- 172.21.15.28:0/2709330297 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f45800c11f0 0x7f45800c5010 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f45800c5ce0 tx=0x7f457802bc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78edf720ba4116d4 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:11.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.016+0000 7f457e000640 1 -- 172.21.15.28:0/2709330297 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4578039070 con 0x7f45800c11f0 2024-02-21T23:03:11.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.016+0000 7fb54b400640 1 -- 172.21.15.28:0/3375977556 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb54c0ac650 con 0x7fb54404ede0 2024-02-21T23:03:11.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.016+0000 7f457e000640 1 -- 172.21.15.28:0/2709330297 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f457802c7b0 con 0x7f45800c11f0 2024-02-21T23:03:11.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.016+0000 7f457e000640 1 -- 172.21.15.28:0/2709330297 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4570001670 con 0x7f45800c11f0 2024-02-21T23:03:11.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.016+0000 7fb54b400640 1 -- 172.21.15.28:0/3375977556 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb54c0ac7f0 con 0x7fb54404ede0 2024-02-21T23:03:11.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.016+0000 7fb54b400640 1 -- 172.21.15.28:0/3375977556 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb54c0acc80 con 0x7fb54404ede0 2024-02-21T23:03:11.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.017+0000 7f4586800640 1 -- 172.21.15.28:0/2709330297 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f45800c11f0 msgr2=0x7f45800c5010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:11.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.017+0000 7f4586800640 1 --2- 172.21.15.28:0/2709330297 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f45800c11f0 0x7f45800c5010 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f45800c5ce0 tx=0x7f457802bc80 comp rx=0 tx=0).stop 2024-02-21T23:03:11.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.018+0000 7fb552000640 1 -- 172.21.15.28:0/3375977556 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb54404ede0 msgr2=0x7fb5440e4d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:11.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.018+0000 7fb552000640 1 --2- 172.21.15.28:0/3375977556 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb54404ede0 0x7fb5440e4d70 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb54c050040 tx=0x7fb54c0a6830 comp rx=0 tx=0).stop 2024-02-21T23:03:11.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.018+0000 7f4586800640 1 -- 172.21.15.28:0/2709330297 shutdown_connections 2024-02-21T23:03:11.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.018+0000 7f4586800640 1 --2- 172.21.15.28:0/2709330297 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f45800c11f0 0x7f45800c5010 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.018+0000 7f4586800640 1 --2- 172.21.15.28:0/2709330297 >> v2:172.21.15.28:3300/0 conn(0x7f4580072e60 0x7f45800c4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.018+0000 7f4586800640 1 -- 172.21.15.28:0/2709330297 >> 172.21.15.28:0/2709330297 conn(0x7f458006b820 msgr2=0x7f45800c67f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:11.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.018+0000 7fb552000640 1 -- 172.21.15.28:0/3375977556 shutdown_connections 2024-02-21T23:03:11.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.018+0000 7fb552000640 1 --2- 172.21.15.28:0/3375977556 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb54404ede0 0x7fb5440e4d70 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.018+0000 7fb552000640 1 --2- 172.21.15.28:0/3375977556 >> v2:172.21.15.81:3300/0 conn(0x7fb54404e0f0 0x7fb54404e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.018+0000 7fb552000640 1 -- 172.21.15.28:0/3375977556 >> 172.21.15.28:0/3375977556 conn(0x7fb54404c5e0 msgr2=0x7fb54404c9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:11.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.019+0000 7f4586800640 1 -- 172.21.15.28:0/2709330297 shutdown_connections 2024-02-21T23:03:11.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.019+0000 7f4586800640 1 -- 172.21.15.28:0/2709330297 wait complete. 2024-02-21T23:03:11.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.019+0000 7fde93c00640 1 Processor -- start 2024-02-21T23:03:11.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.018+0000 7fb552000640 1 -- 172.21.15.28:0/3375977556 shutdown_connections 2024-02-21T23:03:11.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.019+0000 7fb552000640 1 -- 172.21.15.28:0/3375977556 wait complete. 2024-02-21T23:03:11.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.020+0000 7fde93c00640 1 -- start start 2024-02-21T23:03:11.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.021+0000 7fb552000640 1 Processor -- start 2024-02-21T23:03:11.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.021+0000 7fb552000640 1 -- start start 2024-02-21T23:03:11.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.022+0000 7f4586800640 1 Processor -- start 2024-02-21T23:03:11.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.022+0000 7fd128c00640 1 Processor -- start 2024-02-21T23:03:11.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.022+0000 7f4586800640 1 -- start start 2024-02-21T23:03:11.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.022+0000 7fd128c00640 1 -- start start 2024-02-21T23:03:11.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.022+0000 7fde93c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fde8c1550d0 0x7fde8c0c5320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:11.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.022+0000 7fde93c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde8c14f5a0 con 0x7fde8c14f1c0 2024-02-21T23:03:11.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.022+0000 7fde93c00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde8c0c5a20 con 0x7fde8c0bfa40 2024-02-21T23:03:11.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.022+0000 7fde93c00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde8c0c5d00 con 0x7fde8c1550d0 2024-02-21T23:03:11.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.023+0000 7fde91200640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fde8c14f1c0 0x7fde8c0ca890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:48602/0 (socket says 172.21.15.28:48602) 2024-02-21T23:03:11.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.023+0000 7fde91200640 1 -- 172.21.15.28:0/2645401885 learned_addr learned my addr 172.21.15.28:0/2645401885 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:03:11.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.023+0000 7fde8be00640 1 --2- 172.21.15.28:0/2645401885 >> v2:172.21.15.81:3300/0 conn(0x7fde8c1550d0 0x7fde8c0c5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:11.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.024+0000 7fde8b400640 1 -- 172.21.15.28:0/2645401885 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 711749487 0 0) 0x7fde8c14f5a0 con 0x7fde8c14f1c0 2024-02-21T23:03:11.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.024+0000 7fde8b400640 1 -- 172.21.15.28:0/2645401885 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde74003520 con 0x7fde8c14f1c0 2024-02-21T23:03:11.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.024+0000 7f4586800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45800c11f0 0x7f45800bba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:11.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.024+0000 7fde8b400640 1 -- 172.21.15.28:0/2645401885 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 235103607 0 0) 0x7fde8c0c5a20 con 0x7fde8c0bfa40 2024-02-21T23:03:11.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.024+0000 7f4586800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4580156c80 0x7f45801536a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:11.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.024+0000 7fde8b400640 1 -- 172.21.15.28:0/2645401885 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde8c14f5a0 con 0x7fde8c0bfa40 2024-02-21T23:03:11.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.023+0000 7fb552000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb54404e0f0 0x7fb544030840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:11.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.023+0000 7fb552000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb54404ede0 0x7fb5440dcf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:11.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.023+0000 7fb552000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb5440d7b60 0x7fb5440dd4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:11.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.023+0000 7fb552000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb544007910 con 0x7fb54404ede0 2024-02-21T23:03:11.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.023+0000 7fb552000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5440dada0 con 0x7fb54404e0f0 2024-02-21T23:03:11.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.023+0000 7fb552000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb544007a90 con 0x7fb5440d7b60 2024-02-21T23:03:11.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.024+0000 7fb550c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb54404e0f0 0x7fb544030840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:11.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.025+0000 7fb54be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb54404ede0 0x7fb5440dcf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:11.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.025+0000 7f4586800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f458015d4c0 0x7f4580153be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:11.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.025+0000 7fde8b400640 1 -- 172.21.15.28:0/2645401885 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1278398799 0 0) 0x7fde74003520 con 0x7fde8c14f1c0 2024-02-21T23:03:11.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.025+0000 7f4586800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45800c5550 con 0x7f45800c11f0 2024-02-21T23:03:11.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.025+0000 7f4586800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f458006f160 con 0x7f4580156c80 2024-02-21T23:03:11.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.025+0000 7f4586800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45800c56d0 con 0x7f458015d4c0 2024-02-21T23:03:11.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.025+0000 7fd128c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd1241550d0 0x7fd1240c5320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:11.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.025+0000 7fde8b400640 1 -- 172.21.15.28:0/2645401885 >> v2:172.21.15.81:3300/0 conn(0x7fde8c1550d0 msgr2=0x7fde8c0c5320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:11.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.025+0000 7f457ea00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4580156c80 0x7f45801536a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:11.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.025+0000 7fde8b400640 1 --2- 172.21.15.28:0/2645401885 >> v2:172.21.15.81:3300/0 conn(0x7fde8c1550d0 0x7fde8c0c5320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.025+0000 7fde8b400640 1 -- 172.21.15.28:0/2645401885 >> v1:172.21.15.81:6789/0 conn(0x7fde8c0bfa40 legacy=0x7fde8c0bfe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:11.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.025+0000 7f457fe00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f458015d4c0 0x7f4580153be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:11.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.026+0000 7f457ea00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4580156c80 0x7f45801536a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.28:53658/0 (socket says 172.21.15.28:53658) 2024-02-21T23:03:11.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.026+0000 7f457ea00640 1 -- 172.21.15.28:0/1520875431 learned_addr learned my addr 172.21.15.28:0/1520875431 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:11.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.025+0000 7fde8b400640 1 -- 172.21.15.28:0/2645401885 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde8c0c5fe0 con 0x7fde8c14f1c0 2024-02-21T23:03:11.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.026+0000 7fb551600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb5440d7b60 0x7fb5440dd4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:11.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.025+0000 7fd128c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fd1240bfa40 0x7fd1240bfe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:11.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.025+0000 7fd128c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd12414f5a0 con 0x7fd12414f1c0 2024-02-21T23:03:11.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.025+0000 7fd121e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd12414f1c0 0x7fd1240ca890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:48612/0 (socket says 172.21.15.28:48612) 2024-02-21T23:03:11.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.025+0000 7fd121e00640 1 -- 172.21.15.28:0/3329553164 learned_addr learned my addr 172.21.15.28:0/3329553164 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:03:11.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.025+0000 7fd128c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1240c5a20 con 0x7fd1241550d0 2024-02-21T23:03:11.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.026+0000 7fd128c00640 1 -- 172.21.15.28:0/3329553164 --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1240c5d00 con 0x7fd1240bfa40 2024-02-21T23:03:11.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.026+0000 7fb550c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb54404e0f0 0x7fb544030840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.28:53652/0 (socket says 172.21.15.28:53652) 2024-02-21T23:03:11.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.026+0000 7fb550c00640 1 -- 172.21.15.28:0/1792924823 learned_addr learned my addr 172.21.15.28:0/1792924823 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:11.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.025+0000 7fde8b400640 1 -- 172.21.15.28:0/2645401885 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fde800015f0 con 0x7fde8c14f1c0 2024-02-21T23:03:11.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.025+0000 7fde8b400640 1 -- 172.21.15.28:0/2645401885 >> v1:172.21.15.28:6789/0 conn(0x7fde8c14f1c0 legacy=0x7fde8c0ca890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:11.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.026+0000 7fd122800640 1 --2- 172.21.15.28:0/3329553164 >> v2:172.21.15.81:3300/0 conn(0x7fd1240bfa40 0x7fd1240bfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:11.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.025+0000 7fde8b400640 1 --2- 172.21.15.28:0/2645401885 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fde74003b90 0x7fde74003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:11.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.025+0000 7fde8b400640 1 --2- 172.21.15.28:0/2645401885 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde74004800 0x7fde74004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:11.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.026+0000 7f457f400640 1 --2- 172.21.15.28:0/1520875431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45800c11f0 0x7f45800bba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:11.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.025+0000 7fde8b400640 1 --2- 172.21.15.28:0/2645401885 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde74005370 0x7fde7400b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:11.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.025+0000 7fde8b400640 1 -- 172.21.15.28:0/2645401885 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde8c0c5d00 con 0x7fde74005370 2024-02-21T23:03:11.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.025+0000 7fde8b400640 1 -- 172.21.15.28:0/2645401885 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde7400c2c0 con 0x7fde74004800 2024-02-21T23:03:11.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.025+0000 7fde8b400640 1 -- 172.21.15.28:0/2645401885 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde7400c4c0 con 0x7fde74003b90 2024-02-21T23:03:11.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.025+0000 7fde91200640 1 --2- 172.21.15.28:0/2645401885 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fde74003b90 0x7fde74003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:11.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.026+0000 7fde91200640 1 -- 172.21.15.28:0/2645401885 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde74004800 msgr2=0x7fde74004c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-02-21T23:03:11.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.026+0000 7fde91200640 1 --2- 172.21.15.28:0/2645401885 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde74004800 0x7fde74004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.026+0000 7fde91200640 1 -- 172.21.15.28:0/2645401885 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde74005370 msgr2=0x7fde7400b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:11.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.026+0000 7fde91200640 1 --2- 172.21.15.28:0/2645401885 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde74005370 0x7fde7400b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.026+0000 7fde91200640 1 -- 172.21.15.28:0/2645401885 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde74003520 con 0x7fde74003b90 2024-02-21T23:03:11.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.027+0000 7fde91c00640 1 -- 172.21.15.28:0/2645401885 reap_dead start 2024-02-21T23:03:11.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.028+0000 7f457fe00640 1 -- 172.21.15.28:0/1520875431 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4580156c80 msgr2=0x7f45801536a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:11.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.028+0000 7f457fe00640 1 --2- 172.21.15.28:0/1520875431 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4580156c80 0x7f45801536a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.028+0000 7f457fe00640 1 -- 172.21.15.28:0/1520875431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45800c11f0 msgr2=0x7f45800bba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:11.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.028+0000 7f3d1e400640 1 Processor -- start 2024-02-21T23:03:11.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.028+0000 7f3d1e400640 1 -- start start 2024-02-21T23:03:11.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.027+0000 7fb550c00640 1 -- 172.21.15.28:0/1792924823 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb5440d7b60 msgr2=0x7fb5440dd4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:11.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.027+0000 7fb550c00640 1 --2- 172.21.15.28:0/1792924823 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb5440d7b60 0x7fb5440dd4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.027+0000 7fb550c00640 1 -- 172.21.15.28:0/1792924823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb54404ede0 msgr2=0x7fb5440dcf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:11.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.027+0000 7fb550c00640 1 --2- 172.21.15.28:0/1792924823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb54404ede0 0x7fb5440dcf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.027+0000 7fb550c00640 1 -- 172.21.15.28:0/1792924823 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5440dbf20 con 0x7fb54404e0f0 2024-02-21T23:03:11.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.027+0000 7fb550c00640 1 --2- 172.21.15.28:0/1792924823 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb54404e0f0 0x7fb544030840 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb54000c3f0 tx=0x7fb54000c8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:11.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.028+0000 7fb548c00640 1 -- 172.21.15.28:0/1792924823 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5400074e0 con 0x7fb54404e0f0 2024-02-21T23:03:11.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.028+0000 7fb548c00640 1 -- 172.21.15.28:0/1792924823 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb540007680 con 0x7fb54404e0f0 2024-02-21T23:03:11.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.028+0000 7fb548c00640 1 -- 172.21.15.28:0/1792924823 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb540004a00 con 0x7fb54404e0f0 2024-02-21T23:03:11.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.028+0000 7fb552000640 1 -- 172.21.15.28:0/1792924823 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb544173f60 con 0x7fb54404e0f0 2024-02-21T23:03:11.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.027+0000 7fd122800640 1 -- 172.21.15.28:0/3329553164 >> v2:172.21.15.28:3300/0 conn(0x7fd1241550d0 msgr2=0x7fd1240c5320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:11.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.027+0000 7fd122800640 1 --2- 172.21.15.28:0/3329553164 >> v2:172.21.15.28:3300/0 conn(0x7fd1241550d0 0x7fd1240c5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.027+0000 7fd122800640 1 -- 172.21.15.28:0/3329553164 >> v1:172.21.15.28:6789/0 conn(0x7fd12414f1c0 legacy=0x7fd1240ca890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:11.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.027+0000 7fd122800640 1 -- 172.21.15.28:0/3329553164 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1240c5fe0 con 0x7fd1240bfa40 2024-02-21T23:03:11.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.029+0000 7f457fe00640 1 --2- 172.21.15.28:0/1520875431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45800c11f0 0x7f45800bba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.029+0000 7f457fe00640 1 -- 172.21.15.28:0/1520875431 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45801531c0 con 0x7f458015d4c0 2024-02-21T23:03:11.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.029+0000 7f457f400640 1 --2- 172.21.15.28:0/1520875431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45800c11f0 0x7f45800bba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:03:11.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.029+0000 7fb552000640 1 -- 172.21.15.28:0/1792924823 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb544030d80 con 0x7fb54404e0f0 2024-02-21T23:03:11.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.029+0000 7fd122800640 1 --2- 172.21.15.28:0/3329553164 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1240bfa40 0x7fd1240bfe70 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd124152ec0 tx=0x7fd11c02be50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad902778efe11333 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:11.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.029+0000 7f457fe00640 1 --2- 172.21.15.28:0/1520875431 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f458015d4c0 0x7f4580153be0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4578033000 tx=0x7f4578004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:11.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.030+0000 7fde91200640 1 --2- 172.21.15.28:0/2645401885 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fde74003b90 0x7fde74003f90 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fde80007b20 tx=0x7fde74005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a5b0d85363e9c5f server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:11.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.030+0000 7fd120a00640 1 -- 172.21.15.28:0/3329553164 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd11c039070 con 0x7fd1240bfa40 2024-02-21T23:03:11.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.030+0000 7fd120a00640 1 -- 172.21.15.28:0/3329553164 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd11c02cc00 con 0x7fd1240bfa40 2024-02-21T23:03:11.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.030+0000 7fde8b400640 1 -- 172.21.15.28:0/2645401885 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde80007b60 con 0x7fde74003b90 2024-02-21T23:03:11.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.030+0000 7fb552000640 1 -- 172.21.15.28:0/1792924823 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb528005290 con 0x7fb54404e0f0 2024-02-21T23:03:11.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.030+0000 7f4577400640 1 -- 172.21.15.28:0/1520875431 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4578039050 con 0x7f458015d4c0 2024-02-21T23:03:11.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.030+0000 7fde8b400640 1 -- 172.21.15.28:0/2645401885 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde80005bb0 con 0x7fde74003b90 2024-02-21T23:03:11.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.030+0000 7fde8b400640 1 -- 172.21.15.28:0/2645401885 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde80007070 con 0x7fde74003b90 2024-02-21T23:03:11.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.030+0000 7f3d1e400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3d180b0150 0x7f3d180c8760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:11.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.031+0000 7f3d1e400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d181550d0 con 0x7f3d18158c40 2024-02-21T23:03:11.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.031+0000 7f3d1e400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d18155370 con 0x7f3d180c51e0 2024-02-21T23:03:11.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.031+0000 7f3d1e400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d18152a60 con 0x7f3d180b0150 2024-02-21T23:03:11.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.030+0000 7f4586800640 1 -- 172.21.15.28:0/1520875431 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4580157470 con 0x7f458015d4c0 2024-02-21T23:03:11.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.031+0000 7f4586800640 1 -- 172.21.15.28:0/1520875431 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45800bc930 con 0x7f458015d4c0 2024-02-21T23:03:11.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.031+0000 7fd128c00640 1 -- 172.21.15.28:0/3329553164 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1240bfa40 msgr2=0x7fd1240bfe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:11.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.031+0000 7fd128c00640 1 --2- 172.21.15.28:0/3329553164 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1240bfa40 0x7fd1240bfe70 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd124152ec0 tx=0x7fd11c02be50 comp rx=0 tx=0).stop 2024-02-21T23:03:11.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.031+0000 7fd128c00640 1 -- 172.21.15.28:0/3329553164 shutdown_connections 2024-02-21T23:03:11.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.031+0000 7fd128c00640 1 --2- 172.21.15.28:0/3329553164 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1240bfa40 0x7fd1240bfe70 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.031+0000 7fd128c00640 1 --2- 172.21.15.28:0/3329553164 >> v2:172.21.15.28:3300/0 conn(0x7fd1241550d0 0x7fd1240c5320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.031+0000 7fd128c00640 1 -- 172.21.15.28:0/3329553164 >> 172.21.15.28:0/3329553164 conn(0x7fd12408b1c0 msgr2=0x7fd1240725a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:11.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.032+0000 7fde93c00640 1 -- 172.21.15.28:0/2645401885 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fde74003b90 msgr2=0x7fde74003f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:11.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.032+0000 7fde93c00640 1 --2- 172.21.15.28:0/2645401885 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fde74003b90 0x7fde74003f90 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fde80007b20 tx=0x7fde74005830 comp rx=0 tx=0).stop 2024-02-21T23:03:11.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.032+0000 7fd128c00640 1 -- 172.21.15.28:0/3329553164 shutdown_connections 2024-02-21T23:03:11.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.032+0000 7f4577400640 1 -- 172.21.15.28:0/1520875431 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f457803a040 con 0x7f458015d4c0 2024-02-21T23:03:11.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.032+0000 7f4577400640 1 -- 172.21.15.28:0/1520875431 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4578039050 con 0x7f458015d4c0 2024-02-21T23:03:11.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.032+0000 7fde93c00640 1 -- 172.21.15.28:0/2645401885 shutdown_connections 2024-02-21T23:03:11.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.032+0000 7fde93c00640 1 --2- 172.21.15.28:0/2645401885 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fde74003b90 0x7fde74003f90 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.032+0000 7fde93c00640 1 -- 172.21.15.28:0/2645401885 >> 172.21.15.28:0/2645401885 conn(0x7fde8c08b1c0 msgr2=0x7fde8c0725a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:11.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.032+0000 7f6e61e00640 1 Processor -- start 2024-02-21T23:03:11.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.031+0000 7f3d1da00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f3d18158c40 0x7f3d1815f8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:48624/0 (socket says 172.21.15.28:48624) 2024-02-21T23:03:11.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.031+0000 7f3d1da00640 1 -- 172.21.15.28:0/889248649 learned_addr learned my addr 172.21.15.28:0/889248649 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:03:11.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.032+0000 7f3d1d000640 1 --2- 172.21.15.28:0/889248649 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3d180b0150 0x7f3d180c8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:11.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.032+0000 7fd128c00640 1 -- 172.21.15.28:0/3329553164 wait complete. 2024-02-21T23:03:11.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.033+0000 7fde93c00640 1 -- 172.21.15.28:0/2645401885 shutdown_connections 2024-02-21T23:03:11.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.033+0000 7fde93c00640 1 -- 172.21.15.28:0/2645401885 wait complete. 2024-02-21T23:03:11.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.034+0000 7f4586800640 1 -- 172.21.15.28:0/1520875431 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4558005290 con 0x7f458015d4c0 2024-02-21T23:03:11.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.034+0000 7f98a9c00640 1 Processor -- start 2024-02-21T23:03:11.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.034+0000 7fd128c00640 1 Processor -- start 2024-02-21T23:03:11.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.034+0000 7f3d1d000640 1 -- 172.21.15.28:0/889248649 >> v1:172.21.15.81:6789/0 conn(0x7f3d180c51e0 legacy=0x7f3d18158530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:11.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.034+0000 7f3d1d000640 1 -- 172.21.15.28:0/889248649 >> v1:172.21.15.28:6789/0 conn(0x7f3d18158c40 legacy=0x7f3d1815f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:11.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.034+0000 7f3d1d000640 1 -- 172.21.15.28:0/889248649 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d18155650 con 0x7f3d180b0150 2024-02-21T23:03:11.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.034+0000 7fd128c00640 1 -- start start 2024-02-21T23:03:11.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.034+0000 7fb548c00640 1 -- 172.21.15.28:0/1792924823 <== mon.1 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb540011070 con 0x7fb54404e0f0 2024-02-21T23:03:11.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.032+0000 7f6e61e00640 1 -- start start 2024-02-21T23:03:11.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.034+0000 7f6e61e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f6e5c14f1c0 0x7f6e5c1550d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:11.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.034+0000 7f6e61e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6e5c0c5320 0x7f6e5c0ca890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:11.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.034+0000 7f98a9c00640 1 -- start start 2024-02-21T23:03:11.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.034+0000 7f6e61e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6e5c0cadd0 0x7f6e5c0bfa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:11.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.034+0000 7f6e61e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e5c0c57a0 con 0x7f6e5c0cadd0 2024-02-21T23:03:11.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.034+0000 7f6e61e00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e5c14f5a0 con 0x7f6e5c14f1c0 2024-02-21T23:03:11.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.034+0000 7f6e61e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e5c0c5a20 con 0x7f6e5c0c5320 2024-02-21T23:03:11.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.035+0000 7f6e5b400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6e5c0cadd0 0x7f6e5c0bfa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:11.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.036+0000 7f6e5b400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6e5c0cadd0 0x7f6e5c0bfa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46150/0 (socket says 172.21.15.28:46150) 2024-02-21T23:03:11.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.036+0000 7f6e5b400640 1 -- 172.21.15.28:0/3698429264 learned_addr learned my addr 172.21.15.28:0/3698429264 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:11.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.036+0000 7fde93c00640 1 Processor -- start 2024-02-21T23:03:11.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.036+0000 7fd128c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1240bfa40 0x7fd1241ee4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:11.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.036+0000 7fd128c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd12414f1c0 0x7fd124147660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:11.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.036+0000 7fd128c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1241550d0 0x7fd12414da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:11.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.036+0000 7fd128c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1240cafa0 con 0x7fd1241550d0 2024-02-21T23:03:11.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.036+0000 7fd128c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1240c65a0 con 0x7fd1240bfa40 2024-02-21T23:03:11.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.036+0000 7fd128c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1240cb120 con 0x7fd12414f1c0 2024-02-21T23:03:11.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.036+0000 7fb548c00640 1 --2- 172.21.15.28:0/1792924823 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb52c06b4d0 0x7fb52c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:11.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.036+0000 7fb548c00640 1 -- 172.21.15.28:0/1792924823 <== mon.1 v2:172.21.15.81:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fb54008a690 con 0x7fb54404e0f0 2024-02-21T23:03:11.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.036+0000 7f6e5b400640 1 -- 172.21.15.28:0/3698429264 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6e5c0c5320 msgr2=0x7f6e5c0ca890 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-02-21T23:03:11.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.036+0000 7f6e5b400640 1 --2- 172.21.15.28:0/3698429264 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6e5c0c5320 0x7f6e5c0ca890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.036+0000 7f6e5b400640 1 -- 172.21.15.28:0/3698429264 >> v2:172.21.15.81:3300/0 conn(0x7f6e5c14f1c0 msgr2=0x7f6e5c1550d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-02-21T23:03:11.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.036+0000 7f6e5b400640 1 --2- 172.21.15.28:0/3698429264 >> v2:172.21.15.81:3300/0 conn(0x7f6e5c14f1c0 0x7f6e5c1550d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.036+0000 7f6e5b400640 1 -- 172.21.15.28:0/3698429264 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e5c0c5d00 con 0x7f6e5c0cadd0 2024-02-21T23:03:11.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.036+0000 7f98a9c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f98a40a6040 0x7f98a4151670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:11.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.036+0000 7f98a9c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98a406ec50 con 0x7f98a406b8b0 2024-02-21T23:03:11.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.036+0000 7f98a9c00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98a406f1d0 con 0x7f98a406d390 2024-02-21T23:03:11.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.036+0000 7f98a9c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98a4151cc0 con 0x7f98a40a6040 2024-02-21T23:03:11.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.036+0000 7fde93c00640 1 -- start start 2024-02-21T23:03:11.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.037+0000 7fd121e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1240bfa40 0x7fd1241ee4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:11.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.037+0000 7fd122800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1241550d0 0x7fd12414da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:11.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.037+0000 7fd122800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1241550d0 0x7fd12414da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46164/0 (socket says 172.21.15.28:46164) 2024-02-21T23:03:11.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.037+0000 7fd121e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1240bfa40 0x7fd1241ee4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.28:53676/0 (socket says 172.21.15.28:53676) 2024-02-21T23:03:11.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.037+0000 7fd122800640 1 -- 172.21.15.28:0/3733403901 learned_addr learned my addr 172.21.15.28:0/3733403901 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:11.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.037+0000 7fb54be00640 1 --2- 172.21.15.28:0/1792924823 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb52c06b4d0 0x7fb52c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:11.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.037+0000 7fd121400640 1 --2- 172.21.15.28:0/3733403901 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd12414f1c0 0x7fd124147660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:11.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.036+0000 7f4577400640 1 -- 172.21.15.28:0/1520875431 <== mon.2 v2:172.21.15.81:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4578047020 con 0x7f458015d4c0 2024-02-21T23:03:11.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.037+0000 7f98a2000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f98a40a6040 0x7f98a4151670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:11.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.038+0000 7f98a2000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f98a40a6040 0x7f98a4151670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46170/0 (socket says 172.21.15.28:46170) 2024-02-21T23:03:11.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.038+0000 7f98a2000640 1 -- 172.21.15.28:0/2369592727 learned_addr learned my addr 172.21.15.28:0/2369592727 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:11.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.038+0000 7fd121e00640 1 -- 172.21.15.28:0/3733403901 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd12414f1c0 msgr2=0x7fd124147660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:11.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.038+0000 7fd121e00640 1 --2- 172.21.15.28:0/3733403901 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd12414f1c0 0x7fd124147660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.038+0000 7f98a2000640 1 -- 172.21.15.28:0/2369592727 >> v1:172.21.15.81:6789/0 conn(0x7f98a406d390 legacy=0x7f98a40c46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:11.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.038+0000 7f4577400640 1 --2- 172.21.15.28:0/1520875431 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f456006b700 0x7f456006dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:11.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.038+0000 7f98a2000640 1 -- 172.21.15.28:0/2369592727 >> v1:172.21.15.28:6789/0 conn(0x7f98a406b8b0 legacy=0x7f98a40c2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:11.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.038+0000 7f98a2000640 1 -- 172.21.15.28:0/2369592727 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98a4151fa0 con 0x7f98a40a6040 2024-02-21T23:03:11.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.038+0000 7f4577400640 1 -- 172.21.15.28:0/1520875431 <== mon.2 v2:172.21.15.81:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f45780ac1b0 con 0x7f458015d4c0 2024-02-21T23:03:11.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.038+0000 7fd121e00640 1 -- 172.21.15.28:0/3733403901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1241550d0 msgr2=0x7fd12414da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:11.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.038+0000 7f457f400640 1 --2- 172.21.15.28:0/1520875431 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f456006b700 0x7f456006dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:11.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.038+0000 7fd121e00640 1 --2- 172.21.15.28:0/3733403901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1241550d0 0x7fd12414da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.038+0000 7fd121e00640 1 -- 172.21.15.28:0/3733403901 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1241eef20 con 0x7fd1240bfa40 2024-02-21T23:03:11.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.039+0000 7fb54be00640 1 --2- 172.21.15.28:0/1792924823 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb52c06b4d0 0x7fb52c06d990 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb544049c30 tx=0x7fb538008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:11.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.039+0000 7fde93c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fde8c13ecd0 0x7fde8c13f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:11.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.039+0000 7f6e5b400640 1 --2- 172.21.15.28:0/3698429264 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e5c0cadd0 0x7f6e5c0bfa40 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c08baa0 tx=0x7f6e5402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec2a705bfbc752fa server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:11.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.039+0000 7fde93c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde8c153f20 0x7fde8c1543a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:11.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.039+0000 7f3d1d000640 1 --2- 172.21.15.28:0/889248649 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3d180b0150 0x7f3d180c8760 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3d10009dc0 tx=0x7f3d1002be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f31eac93f9316a4c server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:11.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.039+0000 7fd121e00640 1 --2- 172.21.15.28:0/3733403901 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1240bfa40 0x7fd1241ee4a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fd11400c470 tx=0x7fd114001e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:11.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.039+0000 7f6e5a000640 1 -- 172.21.15.28:0/3698429264 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e54035020 con 0x7f6e5c0cadd0 2024-02-21T23:03:11.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.039+0000 7fde91200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fde8c13ecd0 0x7fde8c13f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:11.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.039+0000 7f6e5a000640 1 -- 172.21.15.28:0/3698429264 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e54031830 con 0x7f6e5c0cadd0 2024-02-21T23:03:11.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.039+0000 7f6e5a000640 1 -- 172.21.15.28:0/3698429264 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e54031c00 con 0x7f6e5c0cadd0 2024-02-21T23:03:11.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.039+0000 7fde93c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde8c1548e0 0x7fde8c0c9470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:11.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.039+0000 7fde93c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde8c0c5320 con 0x7fde8c153f20 2024-02-21T23:03:11.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.039+0000 7fde93c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde8c0c65a0 con 0x7fde8c1548e0 2024-02-21T23:03:11.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.039+0000 7fde93c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde8c0c54a0 con 0x7fde8c13ecd0 2024-02-21T23:03:11.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.040+0000 7fde91200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fde8c13ecd0 0x7fde8c13f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3301/0 says I am v2:172.21.15.28:40818/0 (socket says 172.21.15.28:40818) 2024-02-21T23:03:11.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.040+0000 7fde91200640 1 -- 172.21.15.28:0/155260342 learned_addr learned my addr 172.21.15.28:0/155260342 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:11.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.040+0000 7f457f400640 1 --2- 172.21.15.28:0/1520875431 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f456006b700 0x7f456006dbc0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f458015d120 tx=0x7f4570009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:11.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.040+0000 7fde8be00640 1 --2- 172.21.15.28:0/155260342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde8c153f20 0x7fde8c1543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:11.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.040+0000 7f98a2000640 1 --2- 172.21.15.28:0/2369592727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98a40a6040 0x7f98a4151670 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f9890009870 tx=0x7f989002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa988ea1e118dd02 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:11.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.040+0000 7fde91c00640 1 --2- 172.21.15.28:0/155260342 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde8c1548e0 0x7fde8c0c9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:11.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.040+0000 7fde91200640 1 -- 172.21.15.28:0/155260342 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde8c1548e0 msgr2=0x7fde8c0c9470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:11.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.040+0000 7fde91200640 1 --2- 172.21.15.28:0/155260342 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde8c1548e0 0x7fde8c0c9470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.041+0000 7fde91200640 1 -- 172.21.15.28:0/155260342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde8c153f20 msgr2=0x7fde8c1543a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:11.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.041+0000 7fde91200640 1 --2- 172.21.15.28:0/155260342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde8c153f20 0x7fde8c1543a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.041+0000 7fde91200640 1 -- 172.21.15.28:0/155260342 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde8c14e3e0 con 0x7fde8c13ecd0 2024-02-21T23:03:11.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.041+0000 7f6e61e00640 1 -- 172.21.15.28:0/3698429264 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e5c0cadd0 msgr2=0x7f6e5c0bfa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:11.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.041+0000 7f98a1600640 1 -- 172.21.15.28:0/2369592727 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9890035020 con 0x7f98a40a6040 2024-02-21T23:03:11.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.041+0000 7f98a1600640 1 -- 172.21.15.28:0/2369592727 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9890031870 con 0x7f98a40a6040 2024-02-21T23:03:11.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.041+0000 7f98a1600640 1 -- 172.21.15.28:0/2369592727 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9890031c40 con 0x7f98a40a6040 2024-02-21T23:03:11.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.041+0000 7f6e61e00640 1 --2- 172.21.15.28:0/3698429264 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e5c0cadd0 0x7f6e5c0bfa40 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c08baa0 tx=0x7f6e5402b8d0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.041+0000 7f6e61e00640 1 -- 172.21.15.28:0/3698429264 shutdown_connections 2024-02-21T23:03:11.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.041+0000 7f6e61e00640 1 --2- 172.21.15.28:0/3698429264 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e5c0cadd0 0x7f6e5c0bfa40 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.041+0000 7f6e61e00640 1 --2- 172.21.15.28:0/3698429264 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6e5c0c5320 0x7f6e5c0ca890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.041+0000 7f6e61e00640 1 --2- 172.21.15.28:0/3698429264 >> v2:172.21.15.81:3300/0 conn(0x7f6e5c14f1c0 0x7f6e5c1550d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.041+0000 7f6e61e00640 1 -- 172.21.15.28:0/3698429264 >> 172.21.15.28:0/3698429264 conn(0x7f6e5c08b1c0 msgr2=0x7f6e5c0725a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:11.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.041+0000 7fde91200640 1 --2- 172.21.15.28:0/155260342 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fde8c13ecd0 0x7fde8c13f0b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fde80000c00 tx=0x7fde8c0c5bf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:11.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.041+0000 7f6e61e00640 1 -- 172.21.15.28:0/3698429264 shutdown_connections 2024-02-21T23:03:11.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.042+0000 7f6e61e00640 1 -- 172.21.15.28:0/3698429264 wait complete. 2024-02-21T23:03:11.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.042+0000 7f98a9c00640 1 -- 172.21.15.28:0/2369592727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98a40a6040 msgr2=0x7f98a4151670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:11.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.042+0000 7f98a9c00640 1 --2- 172.21.15.28:0/2369592727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98a40a6040 0x7f98a4151670 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f9890009870 tx=0x7f989002b910 comp rx=0 tx=0).stop 2024-02-21T23:03:11.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.042+0000 7f98a9c00640 1 -- 172.21.15.28:0/2369592727 shutdown_connections 2024-02-21T23:03:11.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.042+0000 7f98a9c00640 1 --2- 172.21.15.28:0/2369592727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98a40a6040 0x7f98a4151670 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.042+0000 7f98a9c00640 1 -- 172.21.15.28:0/2369592727 >> 172.21.15.28:0/2369592727 conn(0x7f98a406b4a0 msgr2=0x7f98a40c67f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:11.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.042+0000 7f98a9c00640 1 -- 172.21.15.28:0/2369592727 shutdown_connections 2024-02-21T23:03:11.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.042+0000 7f98a9c00640 1 -- 172.21.15.28:0/2369592727 wait complete. 2024-02-21T23:03:11.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.043+0000 7f98a9c00640 1 Processor -- start 2024-02-21T23:03:11.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.044+0000 7f98a9c00640 1 -- start start 2024-02-21T23:03:11.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.044+0000 7f6e61e00640 1 Processor -- start 2024-02-21T23:03:11.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.044+0000 7f6e61e00640 1 -- start start 2024-02-21T23:03:11.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.044+0000 7f98a9c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f98a406b8b0 0x7f98a40a7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:11.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.044+0000 7f98a9c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f98a406d390 0x7f98a4152fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:11.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.044+0000 7f98a9c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98a40a6040 0x7f98a4153520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:11.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.044+0000 7f98a9c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98a40c62d0 con 0x7f98a40a6040 2024-02-21T23:03:11.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.044+0000 7f98a9c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98a40c3310 con 0x7f98a406b8b0 2024-02-21T23:03:11.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.044+0000 7f98a9c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98a40c4e00 con 0x7f98a406d390 2024-02-21T23:03:11.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.045+0000 7f98a3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98a40a6040 0x7f98a4153520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:11.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.045+0000 7f98a3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98a40a6040 0x7f98a4153520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46188/0 (socket says 172.21.15.28:46188) 2024-02-21T23:03:11.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.045+0000 7f98a3400640 1 -- 172.21.15.28:0/1431523680 learned_addr learned my addr 172.21.15.28:0/1431523680 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:11.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.045+0000 7f98a2000640 1 --2- 172.21.15.28:0/1431523680 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f98a406d390 0x7f98a4152fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:11.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.045+0000 7f6e61e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e5c0c5320 0x7f6e5c14a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:11.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.045+0000 7f6e61e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e5c0cadd0 0x7f6e5c1ee950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:11.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.045+0000 7f6e61e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6e5c14f1c0 0x7f6e5c1eee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:11.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.045+0000 7f6e61e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e5c0c65a0 con 0x7f6e5c0c5320 2024-02-21T23:03:11.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.045+0000 7f6e61e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e5c0c62c0 con 0x7f6e5c0cadd0 2024-02-21T23:03:11.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.045+0000 7f6e61e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e5c0bff80 con 0x7f6e5c14f1c0 2024-02-21T23:03:11.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.045+0000 7f98a2a00640 1 --2- 172.21.15.28:0/1431523680 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f98a406b8b0 0x7f98a40a7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:11.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.045+0000 7f98a3400640 1 -- 172.21.15.28:0/1431523680 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f98a406d390 msgr2=0x7f98a4152fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:11.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.045+0000 7f98a3400640 1 --2- 172.21.15.28:0/1431523680 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f98a406d390 0x7f98a4152fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.045+0000 7f98a3400640 1 -- 172.21.15.28:0/1431523680 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f98a406b8b0 msgr2=0x7f98a40a7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:11.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.045+0000 7f98a3400640 1 --2- 172.21.15.28:0/1431523680 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f98a406b8b0 0x7f98a40a7a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.045+0000 7f98a3400640 1 -- 172.21.15.28:0/1431523680 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98a4152b80 con 0x7f98a40a6040 2024-02-21T23:03:11.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.046+0000 7f98a3400640 1 --2- 172.21.15.28:0/1431523680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98a40a6040 0x7f98a4153520 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f989c00c3c0 tx=0x7f989c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:11.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.046+0000 7f6e5aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e5c0c5320 0x7f6e5c14a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:11.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.046+0000 7f6e5b400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6e5c14f1c0 0x7f6e5c1eee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:11.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.046+0000 7f6e5b400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6e5c14f1c0 0x7f6e5c1eee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3301/0 says I am v2:172.21.15.28:40832/0 (socket says 172.21.15.28:40832) 2024-02-21T23:03:11.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.046+0000 7f6e5b400640 1 -- 172.21.15.28:0/753814752 learned_addr learned my addr 172.21.15.28:0/753814752 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:11.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.046+0000 7f6e52000640 1 --2- 172.21.15.28:0/753814752 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e5c0cadd0 0x7f6e5c1ee950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:11.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.046+0000 7f989aa00640 1 -- 172.21.15.28:0/1431523680 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f989c017070 con 0x7f98a40a6040 2024-02-21T23:03:11.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.046+0000 7f989aa00640 1 -- 172.21.15.28:0/1431523680 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f989c0035f0 con 0x7f98a40a6040 2024-02-21T23:03:11.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.046+0000 7f989aa00640 1 -- 172.21.15.28:0/1431523680 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f989c0039e0 con 0x7f98a40a6040 2024-02-21T23:03:11.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.046+0000 7f98a9c00640 1 -- 172.21.15.28:0/1431523680 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98a41528e0 con 0x7f98a40a6040 2024-02-21T23:03:11.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.047+0000 7f98a9c00640 1 -- 172.21.15.28:0/1431523680 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98a40a8600 con 0x7f98a40a6040 2024-02-21T23:03:11.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.046+0000 7f6e5b400640 1 -- 172.21.15.28:0/753814752 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e5c0cadd0 msgr2=0x7f6e5c1ee950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:11.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.046+0000 7f6e5b400640 1 --2- 172.21.15.28:0/753814752 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e5c0cadd0 0x7f6e5c1ee950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.046+0000 7f6e5b400640 1 -- 172.21.15.28:0/753814752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e5c0c5320 msgr2=0x7f6e5c14a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:11.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.047+0000 7f6e5b400640 1 --2- 172.21.15.28:0/753814752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e5c0c5320 0x7f6e5c14a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.047+0000 7f6e5b400640 1 -- 172.21.15.28:0/753814752 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e5c1ee470 con 0x7f6e5c14f1c0 2024-02-21T23:03:11.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.047+0000 7f6e52000640 1 --2- 172.21.15.28:0/753814752 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e5c0cadd0 0x7f6e5c1ee950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:03:11.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.047+0000 7f6e5aa00640 1 --2- 172.21.15.28:0/753814752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e5c0c5320 0x7f6e5c14a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-21T23:03:11.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.047+0000 7f6e5b400640 1 --2- 172.21.15.28:0/753814752 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6e5c14f1c0 0x7f6e5c1eee90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6e5402b910 tx=0x7f6e540046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:11.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.048+0000 7f98a9c00640 1 -- 172.21.15.28:0/1431523680 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98a406f400 con 0x7f98a40a6040 2024-02-21T23:03:11.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.050+0000 7f989aa00640 1 -- 172.21.15.28:0/1431523680 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f989c01a070 con 0x7f98a40a6040 2024-02-21T23:03:11.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.051+0000 7f989aa00640 1 --2- 172.21.15.28:0/1431523680 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f988406b520 0x7f988406d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:11.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.051+0000 7f989aa00640 1 -- 172.21.15.28:0/1431523680 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f989c089c60 con 0x7f98a40a6040 2024-02-21T23:03:11.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.052+0000 7f98a2a00640 1 --2- 172.21.15.28:0/1431523680 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f988406b520 0x7f988406d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:11.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.052+0000 7fd11a000640 1 -- 172.21.15.28:0/3733403901 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd114017070 con 0x7fd1240bfa40 2024-02-21T23:03:11.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.052+0000 7fd11a000640 1 -- 172.21.15.28:0/3733403901 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd114005ce0 con 0x7fd1240bfa40 2024-02-21T23:03:11.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.052+0000 7fd11a000640 1 -- 172.21.15.28:0/3733403901 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1140072d0 con 0x7fd1240bfa40 2024-02-21T23:03:11.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.052+0000 7fb548c00640 1 -- 172.21.15.28:0/1792924823 <== mon.1 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb540059220 con 0x7fb54404e0f0 2024-02-21T23:03:11.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.052+0000 7fd128c00640 1 -- 172.21.15.28:0/3733403901 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd12414a800 con 0x7fd1240bfa40 2024-02-21T23:03:11.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.052+0000 7fd128c00640 1 -- 172.21.15.28:0/3733403901 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd124148090 con 0x7fd1240bfa40 2024-02-21T23:03:11.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.053+0000 7f98a2a00640 1 --2- 172.21.15.28:0/1431523680 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f988406b520 0x7f988406d9e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f9894007fd0 tx=0x7f9894009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:11.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.053+0000 7fd128c00640 1 -- 172.21.15.28:0/3733403901 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1241ef0f0 con 0x7fd1240bfa40 2024-02-21T23:03:11.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.054+0000 7f989aa00640 1 -- 172.21.15.28:0/1431523680 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f989c021020 con 0x7f98a40a6040 2024-02-21T23:03:11.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.057+0000 7fd11a000640 1 -- 172.21.15.28:0/3733403901 <== mon.1 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd11401a070 con 0x7fd1240bfa40 2024-02-21T23:03:11.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.058+0000 7fd11a000640 1 --2- 172.21.15.28:0/3733403901 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd10406b4d0 0x7fd10406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:11.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.058+0000 7fd11a000640 1 -- 172.21.15.28:0/3733403901 <== mon.1 v2:172.21.15.81:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fd114089f50 con 0x7fd1240bfa40 2024-02-21T23:03:11.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.058+0000 7fd121400640 1 --2- 172.21.15.28:0/3733403901 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd10406b4d0 0x7fd10406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:11.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.059+0000 7f3d17400640 1 -- 172.21.15.28:0/889248649 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d10039070 con 0x7f3d180b0150 2024-02-21T23:03:11.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.059+0000 7fde88c00640 1 -- 172.21.15.28:0/155260342 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde8004b020 con 0x7fde8c13ecd0 2024-02-21T23:03:11.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.059+0000 7f3d17400640 1 -- 172.21.15.28:0/889248649 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d1002cc40 con 0x7f3d180b0150 2024-02-21T23:03:11.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.059+0000 7f3d17400640 1 -- 172.21.15.28:0/889248649 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d10034440 con 0x7f3d180b0150 2024-02-21T23:03:11.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.059+0000 7fde93c00640 1 -- 172.21.15.28:0/155260342 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde8c13f7a0 con 0x7fde8c13ecd0 2024-02-21T23:03:11.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.059+0000 7fde93c00640 1 -- 172.21.15.28:0/155260342 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde8c152f40 con 0x7fde8c13ecd0 2024-02-21T23:03:11.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.060+0000 7f3d1e400640 1 -- 172.21.15.28:0/889248649 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3d180b0150 msgr2=0x7f3d180c8760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:11.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.060+0000 7f3d1e400640 1 --2- 172.21.15.28:0/889248649 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3d180b0150 0x7f3d180c8760 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3d10009dc0 tx=0x7f3d1002be90 comp rx=0 tx=0).stop 2024-02-21T23:03:11.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.059+0000 7fde88c00640 1 -- 172.21.15.28:0/155260342 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde80042040 con 0x7fde8c13ecd0 2024-02-21T23:03:11.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.059+0000 7fde88c00640 1 -- 172.21.15.28:0/155260342 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde80001aa0 con 0x7fde8c13ecd0 2024-02-21T23:03:11.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.060+0000 7f3d1e400640 1 -- 172.21.15.28:0/889248649 shutdown_connections 2024-02-21T23:03:11.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.060+0000 7f3d1e400640 1 --2- 172.21.15.28:0/889248649 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3d180b0150 0x7f3d180c8760 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.060+0000 7f3d1e400640 1 -- 172.21.15.28:0/889248649 >> 172.21.15.28:0/889248649 conn(0x7f3d180ac390 msgr2=0x7f3d180ac7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:11.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.060+0000 7f6e53e00640 1 -- 172.21.15.28:0/753814752 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e5403aa70 con 0x7f6e5c14f1c0 2024-02-21T23:03:11.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.060+0000 7f6e53e00640 1 -- 172.21.15.28:0/753814752 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e5403ac10 con 0x7f6e5c14f1c0 2024-02-21T23:03:11.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.060+0000 7f6e53e00640 1 -- 172.21.15.28:0/753814752 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e540443f0 con 0x7f6e5c14f1c0 2024-02-21T23:03:11.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.060+0000 7fd121400640 1 --2- 172.21.15.28:0/3733403901 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd10406b4d0 0x7fd10406d990 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd1241ebdf0 tx=0x7fd110009060 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:11.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.060+0000 7f6e61e00640 1 -- 172.21.15.28:0/753814752 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e5c1ebc70 con 0x7f6e5c14f1c0 2024-02-21T23:03:11.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.060+0000 7f3d1e400640 1 -- 172.21.15.28:0/889248649 shutdown_connections 2024-02-21T23:03:11.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.060+0000 7f4577400640 1 -- 172.21.15.28:0/1520875431 <== mon.2 v2:172.21.15.81:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f457807ad90 con 0x7f458015d4c0 2024-02-21T23:03:11.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.061+0000 7f6e61e00640 1 -- 172.21.15.28:0/753814752 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e5c0c9f10 con 0x7f6e5c14f1c0 2024-02-21T23:03:11.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.061+0000 7f3d1e400640 1 -- 172.21.15.28:0/889248649 wait complete. 2024-02-21T23:03:11.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.062+0000 7fde93c00640 1 -- 172.21.15.28:0/155260342 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde68005290 con 0x7fde8c13ecd0 2024-02-21T23:03:11.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.063+0000 7f6e61e00640 1 -- 172.21.15.28:0/753814752 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e5c0bff20 con 0x7f6e5c14f1c0 2024-02-21T23:03:11.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.063+0000 7f3d1e400640 1 Processor -- start 2024-02-21T23:03:11.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.063+0000 7f3d1e400640 1 -- start start 2024-02-21T23:03:11.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.065+0000 7f6e53e00640 1 -- 172.21.15.28:0/753814752 <== mon.2 v2:172.21.15.81:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6e54038020 con 0x7f6e5c14f1c0 2024-02-21T23:03:11.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.064+0000 7fde88c00640 1 -- 172.21.15.28:0/155260342 <== mon.2 v2:172.21.15.81:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fde8004e020 con 0x7fde8c13ecd0 2024-02-21T23:03:11.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.065+0000 7f3d1e400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d180b0150 0x7f3d18157330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:11.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.065+0000 7f3d1e400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d180c51e0 0x7f3d180c0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:11.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.065+0000 7f3d1e400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3d18158c40 0x7f3d180c0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:11.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.065+0000 7f3d1e400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d180c8df0 con 0x7f3d180b0150 2024-02-21T23:03:11.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.065+0000 7f3d1e400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d18155e50 con 0x7f3d180c51e0 2024-02-21T23:03:11.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.065+0000 7f3d1e400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d180be080 con 0x7f3d18158c40 2024-02-21T23:03:11.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.065+0000 7f6e53e00640 1 --2- 172.21.15.28:0/753814752 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6e3806b520 0x7f6e3806d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:11.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.066+0000 7f6e53e00640 1 -- 172.21.15.28:0/753814752 <== mon.2 v2:172.21.15.81:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f6e540ac1b0 con 0x7f6e5c14f1c0 2024-02-21T23:03:11.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.066+0000 7fde88c00640 1 --2- 172.21.15.28:0/155260342 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fde6c06b4d0 0x7fde6c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:11.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.066+0000 7f6e5aa00640 1 --2- 172.21.15.28:0/753814752 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6e3806b520 0x7f6e3806d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:11.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.066+0000 7f3d1da00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3d18158c40 0x7f3d180c0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:11.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.066+0000 7f3d1da00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3d18158c40 0x7f3d180c0b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3301/0 says I am v2:172.21.15.28:40836/0 (socket says 172.21.15.28:40836) 2024-02-21T23:03:11.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.066+0000 7f3d1da00640 1 -- 172.21.15.28:0/345994846 learned_addr learned my addr 172.21.15.28:0/345994846 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:11.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.066+0000 7fde88c00640 1 -- 172.21.15.28:0/155260342 <== mon.2 v2:172.21.15.81:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fde800b4620 con 0x7fde8c13ecd0 2024-02-21T23:03:11.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.066+0000 7f3d17e00640 1 --2- 172.21.15.28:0/345994846 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d180c51e0 0x7f3d180c0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:11.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.066+0000 7f3d1d000640 1 --2- 172.21.15.28:0/345994846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d180b0150 0x7f3d18157330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:11.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.067+0000 7fde8be00640 1 --2- 172.21.15.28:0/155260342 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fde6c06b4d0 0x7fde6c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:11.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.067+0000 7f3d1da00640 1 -- 172.21.15.28:0/345994846 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d180c51e0 msgr2=0x7f3d180c0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:11.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.067+0000 7f3d1da00640 1 --2- 172.21.15.28:0/345994846 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d180c51e0 0x7f3d180c0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.067+0000 7f3d1da00640 1 -- 172.21.15.28:0/345994846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d180b0150 msgr2=0x7f3d18157330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:11.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.067+0000 7f3d1da00640 1 --2- 172.21.15.28:0/345994846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d180b0150 0x7f3d18157330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.067+0000 7f3d1da00640 1 -- 172.21.15.28:0/345994846 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d180af0b0 con 0x7f3d18158c40 2024-02-21T23:03:11.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.067+0000 7f3d17e00640 1 --2- 172.21.15.28:0/345994846 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d180c51e0 0x7f3d180c0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:03:11.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.068+0000 7f6e5aa00640 1 --2- 172.21.15.28:0/753814752 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6e3806b520 0x7f6e3806d9e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c1e1640 tx=0x7f6e4c00d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:11.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.068+0000 7f3d1da00640 1 --2- 172.21.15.28:0/345994846 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3d18158c40 0x7f3d180c0b50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3d0800c450 tx=0x7f3d0800c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:11.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.068+0000 7fde8be00640 1 --2- 172.21.15.28:0/155260342 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fde6c06b4d0 0x7fde6c06d990 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fde8c1550a0 tx=0x7fde78008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:11.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.072+0000 7fd11a000640 1 -- 172.21.15.28:0/3733403901 <== mon.1 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd1241ef0f0 con 0x7fd1240bfa40 2024-02-21T23:03:11.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.073+0000 7f7b03000640 1 Processor -- start 2024-02-21T23:03:11.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.074+0000 7f7b03000640 1 -- start start 2024-02-21T23:03:11.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.075+0000 7f7b03000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7afc14f1c0 0x7f7afc1550d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:11.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.075+0000 7f7b03000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7afc0bfa40 0x7f7afc0bfed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:11.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.075+0000 7f7b03000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7afc14f5a0 con 0x7f7afc0c5320 2024-02-21T23:03:11.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.075+0000 7f7b03000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7afc0c57d0 con 0x7f7afc14f1c0 2024-02-21T23:03:11.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.075+0000 7f7b03000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7afc0c5a20 con 0x7f7afc0bfa40 2024-02-21T23:03:11.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.075+0000 7f7afbe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7afc14f1c0 0x7f7afc1550d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:11.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.075+0000 7f7afb400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f7afc0c5320 0x7f7afc0ca890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:48648/0 (socket says 172.21.15.28:48648) 2024-02-21T23:03:11.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.075+0000 7f7afb400640 1 -- 172.21.15.28:0/3574608970 learned_addr learned my addr 172.21.15.28:0/3574608970 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:03:11.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.075+0000 7f7b01000640 1 --2- 172.21.15.28:0/3574608970 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7afc0bfa40 0x7f7afc0bfed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:11.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.076+0000 7f7afaa00640 1 -- 172.21.15.28:0/3574608970 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3010716494 0 0) 0x7f7afc14f5a0 con 0x7f7afc0c5320 2024-02-21T23:03:11.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.076+0000 7f7afaa00640 1 -- 172.21.15.28:0/3574608970 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ae0003520 con 0x7f7afc0c5320 2024-02-21T23:03:11.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.076+0000 7f7b01000640 1 -- 172.21.15.28:0/3574608970 >> v2:172.21.15.28:3300/0 conn(0x7f7afc14f1c0 msgr2=0x7f7afc1550d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:11.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.076+0000 7f7afaa00640 1 -- 172.21.15.28:0/3574608970 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 595345290 0 0) 0x7f7ae0003520 con 0x7f7afc0c5320 2024-02-21T23:03:11.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.076+0000 7f7b01000640 1 --2- 172.21.15.28:0/3574608970 >> v2:172.21.15.28:3300/0 conn(0x7f7afc14f1c0 0x7f7afc1550d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.076+0000 7f7b01000640 1 -- 172.21.15.28:0/3574608970 >> v1:172.21.15.28:6789/0 conn(0x7f7afc0c5320 legacy=0x7f7afc0ca890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:11.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.076+0000 7f7b01000640 1 -- 172.21.15.28:0/3574608970 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7afc0c5d00 con 0x7f7afc0bfa40 2024-02-21T23:03:11.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.076+0000 7f7afbe00640 1 --2- 172.21.15.28:0/3574608970 >> v2:172.21.15.28:3300/0 conn(0x7f7afc14f1c0 0x7f7afc1550d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-21T23:03:11.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.078+0000 7f7b01000640 1 --2- 172.21.15.28:0/3574608970 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7afc0bfa40 0x7f7afc0bfed0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7af4009d90 tx=0x7f7af402bce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4db8826d62151279 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:11.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.082+0000 7fde88c00640 1 -- 172.21.15.28:0/155260342 <== mon.2 v2:172.21.15.81:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fde80082a80 con 0x7fde8c13ecd0 2024-02-21T23:03:11.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.098+0000 7f7afaa00640 1 -- 172.21.15.28:0/3574608970 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af4039070 con 0x7f7afc0bfa40 2024-02-21T23:03:11.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.098+0000 7f7afaa00640 1 -- 172.21.15.28:0/3574608970 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7af402c810 con 0x7f7afc0bfa40 2024-02-21T23:03:11.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.098+0000 7f3d14c00640 1 -- 172.21.15.28:0/345994846 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d08017070 con 0x7f3d18158c40 2024-02-21T23:03:11.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.098+0000 7f3d14c00640 1 -- 172.21.15.28:0/345994846 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d08005ce0 con 0x7f3d18158c40 2024-02-21T23:03:11.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.098+0000 7f3d14c00640 1 -- 172.21.15.28:0/345994846 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d08007270 con 0x7f3d18158c40 2024-02-21T23:03:11.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.098+0000 7f7afaa00640 1 -- 172.21.15.28:0/3574608970 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ae8001670 con 0x7f7afc0bfa40 2024-02-21T23:03:11.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.099+0000 7f6e53e00640 1 -- 172.21.15.28:0/753814752 <== mon.2 v2:172.21.15.81:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f6e5c0bff20 con 0x7f6e5c14f1c0 2024-02-21T23:03:11.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.099+0000 7f7b03000640 1 -- 172.21.15.28:0/3574608970 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7afc0bfa40 msgr2=0x7f7afc0bfed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:11.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.099+0000 7f7b03000640 1 --2- 172.21.15.28:0/3574608970 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7afc0bfa40 0x7f7afc0bfed0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7af4009d90 tx=0x7f7af402bce0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.100+0000 7f7b03000640 1 -- 172.21.15.28:0/3574608970 shutdown_connections 2024-02-21T23:03:11.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.100+0000 7f7b03000640 1 --2- 172.21.15.28:0/3574608970 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7afc0bfa40 0x7f7afc0bfed0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.100+0000 7f7b03000640 1 --2- 172.21.15.28:0/3574608970 >> v2:172.21.15.28:3300/0 conn(0x7f7afc14f1c0 0x7f7afc1550d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.100+0000 7f7b03000640 1 -- 172.21.15.28:0/3574608970 >> 172.21.15.28:0/3574608970 conn(0x7f7afc08b1c0 msgr2=0x7f7afc0725a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:11.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.100+0000 7f7b03000640 1 -- 172.21.15.28:0/3574608970 shutdown_connections 2024-02-21T23:03:11.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.100+0000 7f7b03000640 1 -- 172.21.15.28:0/3574608970 wait complete. 2024-02-21T23:03:11.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.100+0000 7f3d1e400640 1 -- 172.21.15.28:0/345994846 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d181f27d0 con 0x7f3d18158c40 2024-02-21T23:03:11.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.100+0000 7f3d1e400640 1 -- 172.21.15.28:0/345994846 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d181f2d90 con 0x7f3d18158c40 2024-02-21T23:03:11.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.102+0000 7f3d1e400640 1 -- 172.21.15.28:0/345994846 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d1804f290 con 0x7f3d18158c40 2024-02-21T23:03:11.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.102+0000 7f7b03000640 1 Processor -- start 2024-02-21T23:03:11.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.103+0000 7f7b03000640 1 -- start start 2024-02-21T23:03:11.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.104+0000 7f7b03000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7afc0bfa40 0x7f7afc147e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:11.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.104+0000 7f7b03000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7afc0c5320 0x7f7afc14b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:11.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.104+0000 7f7b03000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7afc14f1c0 0x7f7afc1f28e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:11.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.104+0000 7f7b03000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7afc0c65a0 con 0x7f7afc14f1c0 2024-02-21T23:03:11.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.104+0000 7f7b03000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7afc0c62c0 con 0x7f7afc0c5320 2024-02-21T23:03:11.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.104+0000 7f7b03000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7afc0cafa0 con 0x7f7afc0bfa40 2024-02-21T23:03:11.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.105+0000 7f7b01000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7afc14f1c0 0x7f7afc1f28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:11.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.105+0000 7f7b01000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7afc14f1c0 0x7f7afc1f28e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46214/0 (socket says 172.21.15.28:46214) 2024-02-21T23:03:11.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.105+0000 7f7b01000640 1 -- 172.21.15.28:0/1974893294 learned_addr learned my addr 172.21.15.28:0/1974893294 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:11.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.105+0000 7f7afbe00640 1 --2- 172.21.15.28:0/1974893294 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7afc0bfa40 0x7f7afc147e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:11.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.105+0000 7f7afb400640 1 --2- 172.21.15.28:0/1974893294 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7afc0c5320 0x7f7afc14b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:11.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.105+0000 7f3d14c00640 1 -- 172.21.15.28:0/345994846 <== mon.2 v2:172.21.15.81:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3d0801a070 con 0x7f3d18158c40 2024-02-21T23:03:11.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.106+0000 7f7afbe00640 1 -- 172.21.15.28:0/1974893294 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7afc0c5320 msgr2=0x7f7afc14b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:11.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.106+0000 7f7afbe00640 1 --2- 172.21.15.28:0/1974893294 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7afc0c5320 0x7f7afc14b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.106+0000 7f7afbe00640 1 -- 172.21.15.28:0/1974893294 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7afc14f1c0 msgr2=0x7f7afc1f28e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:11.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.106+0000 7f7afbe00640 1 --2- 172.21.15.28:0/1974893294 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7afc14f1c0 0x7f7afc1f28e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.106+0000 7f7afbe00640 1 -- 172.21.15.28:0/1974893294 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7afc0c4050 con 0x7f7afc0bfa40 2024-02-21T23:03:11.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.106+0000 7f7b01000640 1 --2- 172.21.15.28:0/1974893294 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7afc14f1c0 0x7f7afc1f28e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-21T23:03:11.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.107+0000 7f7afbe00640 1 --2- 172.21.15.28:0/1974893294 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7afc0bfa40 0x7f7afc147e40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7aec00e900 tx=0x7f7aec00edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:11.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.107+0000 7f3d14c00640 1 --2- 172.21.15.28:0/345994846 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3cf806b5a0 0x7f3cf806da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:11.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.108+0000 7f3d14c00640 1 -- 172.21.15.28:0/345994846 <== mon.2 v2:172.21.15.81:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f3d08089710 con 0x7f3d18158c40 2024-02-21T23:03:11.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.108+0000 7f3d1d000640 1 --2- 172.21.15.28:0/345994846 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3cf806b5a0 0x7f3cf806da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:11.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.110+0000 7f3d1d000640 1 --2- 172.21.15.28:0/345994846 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3cf806b5a0 0x7f3cf806da60 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3d10036000 tx=0x7f3d10030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:11.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.123+0000 7f3d14c00640 1 -- 172.21.15.28:0/345994846 <== mon.2 v2:172.21.15.81:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3d08057be0 con 0x7f3d18158c40 2024-02-21T23:03:11.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.125+0000 7f7af3e00640 1 -- 172.21.15.28:0/1974893294 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aec017070 con 0x7f7afc0bfa40 2024-02-21T23:03:11.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.125+0000 7f7af3e00640 1 -- 172.21.15.28:0/1974893294 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7aec00ca00 con 0x7f7afc0bfa40 2024-02-21T23:03:11.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.125+0000 7f7af3e00640 1 -- 172.21.15.28:0/1974893294 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aec00cdf0 con 0x7f7afc0bfa40 2024-02-21T23:03:11.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.125+0000 7f7b03000640 1 -- 172.21.15.28:0/1974893294 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7afc1efce0 con 0x7f7afc0bfa40 2024-02-21T23:03:11.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.125+0000 7f7b03000640 1 -- 172.21.15.28:0/1974893294 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7afc0ca420 con 0x7f7afc0bfa40 2024-02-21T23:03:11.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.127+0000 7f7b03000640 1 -- 172.21.15.28:0/1974893294 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7afc06b2e0 con 0x7f7afc0bfa40 2024-02-21T23:03:11.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.129+0000 7f7af3e00640 1 -- 172.21.15.28:0/1974893294 <== mon.2 v2:172.21.15.81:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7aec01a070 con 0x7f7afc0bfa40 2024-02-21T23:03:11.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.131+0000 7f7af3e00640 1 --2- 172.21.15.28:0/1974893294 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7adc06b550 0x7f7adc06da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:11.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.131+0000 7f7af3e00640 1 -- 172.21.15.28:0/1974893294 <== mon.2 v2:172.21.15.81:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f7aec08a700 con 0x7f7afc0bfa40 2024-02-21T23:03:11.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.132+0000 7f7afb400640 1 --2- 172.21.15.28:0/1974893294 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7adc06b550 0x7f7adc06da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:11.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.133+0000 7f7afb400640 1 --2- 172.21.15.28:0/1974893294 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7adc06b550 0x7f7adc06da10 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f7ae80011e0 tx=0x7f7ae8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:11.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.141+0000 7f7af3e00640 1 -- 172.21.15.28:0/1974893294 <== mon.2 v2:172.21.15.81:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7aec058d60 con 0x7f7afc0bfa40 2024-02-21T23:03:11.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.461+0000 7f98a9c00640 1 -- 172.21.15.28:0/1431523680 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f98a404f370 con 0x7f98a40a6040 2024-02-21T23:03:11.452 INFO:teuthology.orchestra.run.smithi028.stdout:55834574857 2024-02-21T23:03:11.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.462+0000 7f989aa00640 1 -- 172.21.15.28:0/1431523680 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f989c058910 con 0x7f98a40a6040 2024-02-21T23:03:11.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.465+0000 7f98a9c00640 1 -- 172.21.15.28:0/1431523680 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f988406b520 msgr2=0x7f988406d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:11.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.465+0000 7f98a9c00640 1 --2- 172.21.15.28:0/1431523680 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f988406b520 0x7f988406d9e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f9894007fd0 tx=0x7f9894009040 comp rx=0 tx=0).stop 2024-02-21T23:03:11.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.465+0000 7f98a9c00640 1 -- 172.21.15.28:0/1431523680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98a40a6040 msgr2=0x7f98a4153520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:11.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.465+0000 7f98a9c00640 1 --2- 172.21.15.28:0/1431523680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98a40a6040 0x7f98a4153520 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f989c00c3c0 tx=0x7f989c00c880 comp rx=0 tx=0).stop 2024-02-21T23:03:11.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.466+0000 7f98a9c00640 1 -- 172.21.15.28:0/1431523680 shutdown_connections 2024-02-21T23:03:11.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.466+0000 7f98a9c00640 1 --2- 172.21.15.28:0/1431523680 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f988406b520 0x7f988406d9e0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.466+0000 7f98a9c00640 1 --2- 172.21.15.28:0/1431523680 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f98a406d390 0x7f98a4152fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.466+0000 7f98a9c00640 1 --2- 172.21.15.28:0/1431523680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98a40a6040 0x7f98a4153520 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.466+0000 7f98a9c00640 1 --2- 172.21.15.28:0/1431523680 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f98a406b8b0 0x7f98a40a7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.466+0000 7f98a9c00640 1 -- 172.21.15.28:0/1431523680 >> 172.21.15.28:0/1431523680 conn(0x7f98a406b4a0 msgr2=0x7f98a40c3490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:11.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.467+0000 7f98a9c00640 1 -- 172.21.15.28:0/1431523680 shutdown_connections 2024-02-21T23:03:11.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.467+0000 7f98a9c00640 1 -- 172.21.15.28:0/1431523680 wait complete. 2024-02-21T23:03:11.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.475+0000 7fd128c00640 1 -- 172.21.15.28:0/3733403901 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fd12414a5f0 con 0x7fd1240bfa40 2024-02-21T23:03:11.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.476+0000 7fd11a000640 1 -- 172.21.15.28:0/3733403901 <== mon.1 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd1140584b0 con 0x7fd1240bfa40 2024-02-21T23:03:11.468 INFO:teuthology.orchestra.run.smithi028.stdout:55834574857 2024-02-21T23:03:11.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.481+0000 7fd128c00640 1 -- 172.21.15.28:0/3733403901 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd10406b4d0 msgr2=0x7fd10406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:11.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.481+0000 7fd128c00640 1 --2- 172.21.15.28:0/3733403901 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd10406b4d0 0x7fd10406d990 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd1241ebdf0 tx=0x7fd110009060 comp rx=0 tx=0).stop 2024-02-21T23:03:11.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.481+0000 7fd128c00640 1 -- 172.21.15.28:0/3733403901 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1240bfa40 msgr2=0x7fd1241ee4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:11.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.481+0000 7fd128c00640 1 --2- 172.21.15.28:0/3733403901 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1240bfa40 0x7fd1241ee4a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fd11400c470 tx=0x7fd114001e70 comp rx=0 tx=0).stop 2024-02-21T23:03:11.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.481+0000 7fd128c00640 1 -- 172.21.15.28:0/3733403901 shutdown_connections 2024-02-21T23:03:11.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.481+0000 7fd128c00640 1 --2- 172.21.15.28:0/3733403901 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd10406b4d0 0x7fd10406d990 secure :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd1241ebdf0 tx=0x7fd110009060 comp rx=0 tx=0).stop 2024-02-21T23:03:11.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.481+0000 7fd128c00640 1 --2- 172.21.15.28:0/3733403901 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd12414f1c0 0x7fd124147660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.481+0000 7fd128c00640 1 --2- 172.21.15.28:0/3733403901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1241550d0 0x7fd12414da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.481+0000 7fd128c00640 1 --2- 172.21.15.28:0/3733403901 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1240bfa40 0x7fd1241ee4a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.481+0000 7fd128c00640 1 -- 172.21.15.28:0/3733403901 >> 172.21.15.28:0/3733403901 conn(0x7fd12408b1c0 msgr2=0x7fd1240c1cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:11.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.482+0000 7fd128c00640 1 -- 172.21.15.28:0/3733403901 shutdown_connections 2024-02-21T23:03:11.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.484+0000 7fd128c00640 1 -- 172.21.15.28:0/3733403901 wait complete. 2024-02-21T23:03:11.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.488+0000 7fb552000640 1 -- 172.21.15.28:0/1792924823 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fb528005850 con 0x7fb54404e0f0 2024-02-21T23:03:11.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.489+0000 7fb548c00640 1 -- 172.21.15.28:0/1792924823 <== mon.1 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb540058b20 con 0x7fb54404e0f0 2024-02-21T23:03:11.478 INFO:teuthology.orchestra.run.smithi028.stdout:55834574859 2024-02-21T23:03:11.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.493+0000 7fb552000640 1 -- 172.21.15.28:0/1792924823 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb52c06b4d0 msgr2=0x7fb52c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:11.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.493+0000 7fb552000640 1 --2- 172.21.15.28:0/1792924823 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb52c06b4d0 0x7fb52c06d990 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb544049c30 tx=0x7fb538008040 comp rx=0 tx=0).stop 2024-02-21T23:03:11.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.493+0000 7fb552000640 1 -- 172.21.15.28:0/1792924823 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb54404e0f0 msgr2=0x7fb544030840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:11.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.493+0000 7fb552000640 1 --2- 172.21.15.28:0/1792924823 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb54404e0f0 0x7fb544030840 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb54000c3f0 tx=0x7fb54000c8b0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.493+0000 7fb552000640 1 -- 172.21.15.28:0/1792924823 shutdown_connections 2024-02-21T23:03:11.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.493+0000 7fb552000640 1 --2- 172.21.15.28:0/1792924823 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb52c06b4d0 0x7fb52c06d990 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.493+0000 7fb552000640 1 --2- 172.21.15.28:0/1792924823 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb5440d7b60 0x7fb5440dd4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.493+0000 7fb552000640 1 --2- 172.21.15.28:0/1792924823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb54404ede0 0x7fb5440dcf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.493+0000 7fb552000640 1 --2- 172.21.15.28:0/1792924823 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb54404e0f0 0x7fb544030840 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.493+0000 7fb552000640 1 -- 172.21.15.28:0/1792924823 >> 172.21.15.28:0/1792924823 conn(0x7fb54404c5e0 msgr2=0x7fb544007110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:11.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.493+0000 7fb552000640 1 -- 172.21.15.28:0/1792924823 shutdown_connections 2024-02-21T23:03:11.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.493+0000 7fb552000640 1 -- 172.21.15.28:0/1792924823 wait complete. 2024-02-21T23:03:11.484 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574857 for osd.4 2024-02-21T23:03:11.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.509+0000 7f4586800640 1 -- 172.21.15.28:0/1520875431 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f4558005850 con 0x7f458015d4c0 2024-02-21T23:03:11.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.510+0000 7f4577400640 1 -- 172.21.15.28:0/1520875431 <== mon.2 v2:172.21.15.81:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4578034d00 con 0x7f458015d4c0 2024-02-21T23:03:11.500 INFO:teuthology.orchestra.run.smithi028.stdout:55834574857 2024-02-21T23:03:11.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.512+0000 7fde93c00640 1 -- 172.21.15.28:0/155260342 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fde68005580 con 0x7fde8c13ecd0 2024-02-21T23:03:11.501 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.1 2024-02-21T23:03:11.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.512+0000 7fde88c00640 1 -- 172.21.15.28:0/155260342 <== mon.2 v2:172.21.15.81:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fde80086b20 con 0x7fde8c13ecd0 2024-02-21T23:03:11.502 DEBUG:teuthology.parallel:result is None 2024-02-21T23:03:11.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.514+0000 7f4586800640 1 -- 172.21.15.28:0/1520875431 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f456006b700 msgr2=0x7f456006dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:11.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.514+0000 7f4586800640 1 --2- 172.21.15.28:0/1520875431 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f456006b700 0x7f456006dbc0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f458015d120 tx=0x7f4570009040 comp rx=0 tx=0).stop 2024-02-21T23:03:11.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.514+0000 7f4586800640 1 -- 172.21.15.28:0/1520875431 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f458015d4c0 msgr2=0x7f4580153be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:11.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.514+0000 7f4586800640 1 --2- 172.21.15.28:0/1520875431 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f458015d4c0 0x7f4580153be0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4578033000 tx=0x7f4578004a40 comp rx=0 tx=0).stop 2024-02-21T23:03:11.504 INFO:teuthology.orchestra.run.smithi028.stdout:55834574858 2024-02-21T23:03:11.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.514+0000 7f4586800640 1 -- 172.21.15.28:0/1520875431 shutdown_connections 2024-02-21T23:03:11.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.514+0000 7f4586800640 1 --2- 172.21.15.28:0/1520875431 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f456006b700 0x7f456006dbc0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.514+0000 7f4586800640 1 --2- 172.21.15.28:0/1520875431 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f458015d4c0 0x7f4580153be0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.514+0000 7f4586800640 1 --2- 172.21.15.28:0/1520875431 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4580156c80 0x7f45801536a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.514+0000 7f4586800640 1 --2- 172.21.15.28:0/1520875431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45800c11f0 0x7f45800bba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.514+0000 7f4586800640 1 -- 172.21.15.28:0/1520875431 >> 172.21.15.28:0/1520875431 conn(0x7f458006b820 msgr2=0x7f4580158e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:11.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.514+0000 7f4586800640 1 -- 172.21.15.28:0/1520875431 shutdown_connections 2024-02-21T23:03:11.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.515+0000 7f4586800640 1 -- 172.21.15.28:0/1520875431 wait complete. 2024-02-21T23:03:11.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.515+0000 7fde93c00640 1 -- 172.21.15.28:0/155260342 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fde6c06b4d0 msgr2=0x7fde6c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:11.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.515+0000 7fde93c00640 1 --2- 172.21.15.28:0/155260342 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fde6c06b4d0 0x7fde6c06d990 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fde8c1550a0 tx=0x7fde78008040 comp rx=0 tx=0).stop 2024-02-21T23:03:11.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.515+0000 7fde93c00640 1 -- 172.21.15.28:0/155260342 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fde8c13ecd0 msgr2=0x7fde8c13f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:11.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.515+0000 7fde93c00640 1 --2- 172.21.15.28:0/155260342 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fde8c13ecd0 0x7fde8c13f0b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fde80000c00 tx=0x7fde8c0c5bf0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.516+0000 7fde93c00640 1 -- 172.21.15.28:0/155260342 shutdown_connections 2024-02-21T23:03:11.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.516+0000 7fde93c00640 1 --2- 172.21.15.28:0/155260342 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fde6c06b4d0 0x7fde6c06d990 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.516+0000 7fde93c00640 1 --2- 172.21.15.28:0/155260342 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde8c1548e0 0x7fde8c0c9470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.516+0000 7fde93c00640 1 --2- 172.21.15.28:0/155260342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde8c153f20 0x7fde8c1543a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.516+0000 7fde93c00640 1 --2- 172.21.15.28:0/155260342 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fde8c13ecd0 0x7fde8c13f0b0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.516+0000 7fde93c00640 1 -- 172.21.15.28:0/155260342 >> 172.21.15.28:0/155260342 conn(0x7fde8c08b1c0 msgr2=0x7fde8c146ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:11.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.516+0000 7fde93c00640 1 -- 172.21.15.28:0/155260342 shutdown_connections 2024-02-21T23:03:11.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.516+0000 7fde93c00640 1 -- 172.21.15.28:0/155260342 wait complete. 2024-02-21T23:03:11.512 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574859 for osd.2 2024-02-21T23:03:11.512 DEBUG:teuthology.parallel:result is None 2024-02-21T23:03:11.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.524+0000 7f6e61e00640 1 -- 172.21.15.28:0/753814752 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f6e5c1ee640 con 0x7f6e5c14f1c0 2024-02-21T23:03:11.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.524+0000 7f3d1e400640 1 -- 172.21.15.28:0/345994846 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f3d180c76e0 con 0x7f3d18158c40 2024-02-21T23:03:11.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.525+0000 7f6e53e00640 1 -- 172.21.15.28:0/753814752 <== mon.2 v2:172.21.15.81:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6e5403f070 con 0x7f6e5c14f1c0 2024-02-21T23:03:11.515 INFO:teuthology.orchestra.run.smithi028.stdout:55834574856 2024-02-21T23:03:11.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.525+0000 7f3d14c00640 1 -- 172.21.15.28:0/345994846 <== mon.2 v2:172.21.15.81:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3d0805bc80 con 0x7f3d18158c40 2024-02-21T23:03:11.516 INFO:teuthology.orchestra.run.smithi028.stdout:55834574857 2024-02-21T23:03:11.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.527+0000 7f6e61e00640 1 -- 172.21.15.28:0/753814752 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6e3806b520 msgr2=0x7f6e3806d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:11.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.527+0000 7f6e61e00640 1 --2- 172.21.15.28:0/753814752 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6e3806b520 0x7f6e3806d9e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c1e1640 tx=0x7f6e4c00d040 comp rx=0 tx=0).stop 2024-02-21T23:03:11.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.527+0000 7f6e61e00640 1 -- 172.21.15.28:0/753814752 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6e5c14f1c0 msgr2=0x7f6e5c1eee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:11.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.527+0000 7f6e61e00640 1 --2- 172.21.15.28:0/753814752 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6e5c14f1c0 0x7f6e5c1eee90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6e5402b910 tx=0x7f6e540046c0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.527+0000 7f3d1e400640 1 -- 172.21.15.28:0/345994846 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3cf806b5a0 msgr2=0x7f3cf806da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:11.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.527+0000 7f3d1e400640 1 --2- 172.21.15.28:0/345994846 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3cf806b5a0 0x7f3cf806da60 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3d10036000 tx=0x7f3d10030000 comp rx=0 tx=0).stop 2024-02-21T23:03:11.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.527+0000 7f3d1e400640 1 -- 172.21.15.28:0/345994846 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3d18158c40 msgr2=0x7f3d180c0b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:11.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.528+0000 7f3d1e400640 1 --2- 172.21.15.28:0/345994846 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3d18158c40 0x7f3d180c0b50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3d0800c450 tx=0x7f3d0800c910 comp rx=0 tx=0).stop 2024-02-21T23:03:11.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.528+0000 7f6e61e00640 1 -- 172.21.15.28:0/753814752 shutdown_connections 2024-02-21T23:03:11.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.528+0000 7f6e61e00640 1 --2- 172.21.15.28:0/753814752 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6e3806b520 0x7f6e3806d9e0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.528+0000 7f6e61e00640 1 --2- 172.21.15.28:0/753814752 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6e5c14f1c0 0x7f6e5c1eee90 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.528+0000 7f6e61e00640 1 --2- 172.21.15.28:0/753814752 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e5c0cadd0 0x7f6e5c1ee950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.528+0000 7f6e61e00640 1 --2- 172.21.15.28:0/753814752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e5c0c5320 0x7f6e5c14a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.528+0000 7f6e61e00640 1 -- 172.21.15.28:0/753814752 >> 172.21.15.28:0/753814752 conn(0x7f6e5c08b1c0 msgr2=0x7f6e5c1e17d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:11.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.528+0000 7f6e61e00640 1 -- 172.21.15.28:0/753814752 shutdown_connections 2024-02-21T23:03:11.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.528+0000 7f3d1e400640 1 -- 172.21.15.28:0/345994846 shutdown_connections 2024-02-21T23:03:11.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.528+0000 7f3d1e400640 1 --2- 172.21.15.28:0/345994846 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3cf806b5a0 0x7f3cf806da60 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.528+0000 7f3d1e400640 1 --2- 172.21.15.28:0/345994846 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3d18158c40 0x7f3d180c0b50 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.528+0000 7f3d1e400640 1 --2- 172.21.15.28:0/345994846 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d180c51e0 0x7f3d180c0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.528+0000 7f3d1e400640 1 --2- 172.21.15.28:0/345994846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d180b0150 0x7f3d18157330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.528+0000 7f3d1e400640 1 -- 172.21.15.28:0/345994846 >> 172.21.15.28:0/345994846 conn(0x7f3d180ac390 msgr2=0x7f3d180ac770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:11.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.528+0000 7f6e61e00640 1 -- 172.21.15.28:0/753814752 wait complete. 2024-02-21T23:03:11.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.528+0000 7f3d1e400640 1 -- 172.21.15.28:0/345994846 shutdown_connections 2024-02-21T23:03:11.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.528+0000 7f3d1e400640 1 -- 172.21.15.28:0/345994846 wait complete. 2024-02-21T23:03:11.529 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.0 2024-02-21T23:03:11.530 DEBUG:teuthology.parallel:result is None 2024-02-21T23:03:11.531 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574858 for osd.3 2024-02-21T23:03:11.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.542+0000 7f7b03000640 1 -- 172.21.15.28:0/1974893294 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f7afc0cab00 con 0x7f7afc0bfa40 2024-02-21T23:03:11.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.542+0000 7f7af3e00640 1 -- 172.21.15.28:0/1974893294 <== mon.2 v2:172.21.15.81:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7aec05ce00 con 0x7f7afc0bfa40 2024-02-21T23:03:11.532 DEBUG:teuthology.parallel:result is None 2024-02-21T23:03:11.532 INFO:teuthology.orchestra.run.smithi028.stdout:55834574856 2024-02-21T23:03:11.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.545+0000 7f7b03000640 1 -- 172.21.15.28:0/1974893294 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7adc06b550 msgr2=0x7f7adc06da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:11.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.545+0000 7f7b03000640 1 --2- 172.21.15.28:0/1974893294 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7adc06b550 0x7f7adc06da10 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f7ae80011e0 tx=0x7f7ae8009040 comp rx=0 tx=0).stop 2024-02-21T23:03:11.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.545+0000 7f7b03000640 1 -- 172.21.15.28:0/1974893294 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7afc0bfa40 msgr2=0x7f7afc147e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:11.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.545+0000 7f7b03000640 1 --2- 172.21.15.28:0/1974893294 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7afc0bfa40 0x7f7afc147e40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7aec00e900 tx=0x7f7aec00edc0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.545+0000 7f7b03000640 1 -- 172.21.15.28:0/1974893294 shutdown_connections 2024-02-21T23:03:11.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.545+0000 7f7b03000640 1 --2- 172.21.15.28:0/1974893294 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7adc06b550 0x7f7adc06da10 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.545+0000 7f7b03000640 1 --2- 172.21.15.28:0/1974893294 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7afc14f1c0 0x7f7afc1f28e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.545+0000 7f7b03000640 1 --2- 172.21.15.28:0/1974893294 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7afc0c5320 0x7f7afc14b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.545+0000 7f7b03000640 1 --2- 172.21.15.28:0/1974893294 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7afc0bfa40 0x7f7afc147e40 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:11.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.545+0000 7f7b03000640 1 -- 172.21.15.28:0/1974893294 >> 172.21.15.28:0/1974893294 conn(0x7f7afc08b1c0 msgr2=0x7f7afc0c45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:11.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.545+0000 7f7b03000640 1 -- 172.21.15.28:0/1974893294 shutdown_connections 2024-02-21T23:03:11.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:11.545+0000 7f7b03000640 1 -- 172.21.15.28:0/1974893294 wait complete. 2024-02-21T23:03:11.538 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.5 2024-02-21T23:03:11.539 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574857 for osd.6 2024-02-21T23:03:11.552 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.7 2024-02-21T23:03:12.485 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-02-21T23:03:12.539 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-02-21T23:03:12.539 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-02-21T23:03:12.553 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-02-21T23:03:12.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.713+0000 7fef87a00640 1 Processor -- start 2024-02-21T23:03:12.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.714+0000 7fef87a00640 1 -- start start 2024-02-21T23:03:12.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.715+0000 7fef87a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fef8006e110 0x7fef80155c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:12.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.715+0000 7fef87a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fef80156190 0x7fef800c2170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:12.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.715+0000 7fef87a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef800c2e60 con 0x7fef800c5a70 2024-02-21T23:03:12.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.716+0000 7fef87a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef801565b0 con 0x7fef80156190 2024-02-21T23:03:12.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.716+0000 7fef87a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef8006dc80 con 0x7fef8006e110 2024-02-21T23:03:12.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.716+0000 7fef85a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fef80156190 0x7fef800c2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:12.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.716+0000 7fef85a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fef80156190 0x7fef800c2170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46224/0 (socket says 172.21.15.28:46224) 2024-02-21T23:03:12.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.716+0000 7fef85a00640 1 -- 172.21.15.28:0/906858513 learned_addr learned my addr 172.21.15.28:0/906858513 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:12.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.716+0000 7fef7fe00640 1 --2- 172.21.15.28:0/906858513 >> v2:172.21.15.81:3300/0 conn(0x7fef8006e110 0x7fef80155c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:12.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.717+0000 7fef85a00640 1 -- 172.21.15.28:0/906858513 >> v2:172.21.15.81:3300/0 conn(0x7fef8006e110 msgr2=0x7fef80155c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:12.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.717+0000 7fef85a00640 1 --2- 172.21.15.28:0/906858513 >> v2:172.21.15.81:3300/0 conn(0x7fef8006e110 0x7fef80155c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:12.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.717+0000 7fef85a00640 1 -- 172.21.15.28:0/906858513 >> v1:172.21.15.28:6789/0 conn(0x7fef800c5a70 legacy=0x7fef80071900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:12.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.717+0000 7fef85a00640 1 -- 172.21.15.28:0/906858513 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef8006d630 con 0x7fef80156190 2024-02-21T23:03:12.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.718+0000 7fef85a00640 1 --2- 172.21.15.28:0/906858513 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef80156190 0x7fef800c2170 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fef800c4390 tx=0x7fef7802bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ec73628b64ebf34 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:12.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.719+0000 7fef7f400640 1 -- 172.21.15.28:0/906858513 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef7802cb10 con 0x7fef80156190 2024-02-21T23:03:12.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.719+0000 7fef7f400640 1 -- 172.21.15.28:0/906858513 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef7802ccb0 con 0x7fef80156190 2024-02-21T23:03:12.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.719+0000 7fef7f400640 1 -- 172.21.15.28:0/906858513 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef7802ce60 con 0x7fef80156190 2024-02-21T23:03:12.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.720+0000 7fef87a00640 1 -- 172.21.15.28:0/906858513 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef80156190 msgr2=0x7fef800c2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:12.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.720+0000 7fef87a00640 1 --2- 172.21.15.28:0/906858513 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef80156190 0x7fef800c2170 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fef800c4390 tx=0x7fef7802bf70 comp rx=0 tx=0).stop 2024-02-21T23:03:12.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.720+0000 7fef87a00640 1 -- 172.21.15.28:0/906858513 shutdown_connections 2024-02-21T23:03:12.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.720+0000 7fef87a00640 1 --2- 172.21.15.28:0/906858513 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef80156190 0x7fef800c2170 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:12.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.720+0000 7fef87a00640 1 --2- 172.21.15.28:0/906858513 >> v2:172.21.15.81:3300/0 conn(0x7fef8006e110 0x7fef80155c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:12.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.720+0000 7fef87a00640 1 -- 172.21.15.28:0/906858513 >> 172.21.15.28:0/906858513 conn(0x7fef8006b820 msgr2=0x7fef800c67f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:12.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.720+0000 7fef87a00640 1 -- 172.21.15.28:0/906858513 shutdown_connections 2024-02-21T23:03:12.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.720+0000 7fef87a00640 1 -- 172.21.15.28:0/906858513 wait complete. 2024-02-21T23:03:12.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.723+0000 7fef87a00640 1 Processor -- start 2024-02-21T23:03:12.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.723+0000 7fef87a00640 1 -- start start 2024-02-21T23:03:12.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.724+0000 7fef87a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fef8006e110 0x7fef80151200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:12.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.724+0000 7fef87a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef800c5a70 0x7fef8014d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:12.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.724+0000 7fef87a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fef80156190 0x7fef8014a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:12.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.724+0000 7fef87a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef80072190 con 0x7fef800c5a70 2024-02-21T23:03:12.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.724+0000 7fef87a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef80072010 con 0x7fef8006e110 2024-02-21T23:03:12.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.724+0000 7fef87a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef800c5e50 con 0x7fef80156190 2024-02-21T23:03:12.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.725+0000 7fef7fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef800c5a70 0x7fef8014d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:12.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.725+0000 7fef7fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef800c5a70 0x7fef8014d1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46228/0 (socket says 172.21.15.28:46228) 2024-02-21T23:03:12.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.725+0000 7fef7fe00640 1 -- 172.21.15.28:0/3460533489 learned_addr learned my addr 172.21.15.28:0/3460533489 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:12.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.725+0000 7fef85000640 1 --2- 172.21.15.28:0/3460533489 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fef8006e110 0x7fef80151200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:12.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.725+0000 7fef85a00640 1 --2- 172.21.15.28:0/3460533489 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fef80156190 0x7fef8014a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:12.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.725+0000 7fef7fe00640 1 -- 172.21.15.28:0/3460533489 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fef80156190 msgr2=0x7fef8014a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:12.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.725+0000 7fef7fe00640 1 --2- 172.21.15.28:0/3460533489 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fef80156190 0x7fef8014a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:12.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.725+0000 7fef7fe00640 1 -- 172.21.15.28:0/3460533489 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fef8006e110 msgr2=0x7fef80151200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:12.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.725+0000 7fef7fe00640 1 --2- 172.21.15.28:0/3460533489 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fef8006e110 0x7fef80151200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:12.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.725+0000 7fef7fe00640 1 -- 172.21.15.28:0/3460533489 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef8014def0 con 0x7fef800c5a70 2024-02-21T23:03:12.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.725+0000 7fef85a00640 1 --2- 172.21.15.28:0/3460533489 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fef80156190 0x7fef8014a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:03:12.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.726+0000 7fef7fe00640 1 --2- 172.21.15.28:0/3460533489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef800c5a70 0x7fef8014d1c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fef6c00e9b0 tx=0x7fef6c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:12.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.726+0000 7fef7cc00640 1 -- 172.21.15.28:0/3460533489 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef6c017070 con 0x7fef800c5a70 2024-02-21T23:03:12.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.726+0000 7fef7cc00640 1 -- 172.21.15.28:0/3460533489 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef6c00c9a0 con 0x7fef800c5a70 2024-02-21T23:03:12.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.726+0000 7fef7cc00640 1 -- 172.21.15.28:0/3460533489 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef6c00cd90 con 0x7fef800c5a70 2024-02-21T23:03:12.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.727+0000 7fef87a00640 1 -- 172.21.15.28:0/3460533489 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef8014e1d0 con 0x7fef800c5a70 2024-02-21T23:03:12.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.727+0000 7fef87a00640 1 -- 172.21.15.28:0/3460533489 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef800be160 con 0x7fef800c5a70 2024-02-21T23:03:12.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.728+0000 7fef87a00640 1 -- 172.21.15.28:0/3460533489 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef800c5630 con 0x7fef800c5a70 2024-02-21T23:03:12.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.729+0000 7fef7cc00640 1 -- 172.21.15.28:0/3460533489 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fef6c01a070 con 0x7fef800c5a70 2024-02-21T23:03:12.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.731+0000 7fef7cc00640 1 --2- 172.21.15.28:0/3460533489 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fef6006b500 0x7fef6006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:12.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.731+0000 7fef7cc00640 1 -- 172.21.15.28:0/3460533489 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fef6c08a570 con 0x7fef800c5a70 2024-02-21T23:03:12.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.732+0000 7fef85000640 1 --2- 172.21.15.28:0/3460533489 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fef6006b500 0x7fef6006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:12.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.733+0000 7fef85000640 1 --2- 172.21.15.28:0/3460533489 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fef6006b500 0x7fef6006d9c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fef74001640 tx=0x7fef74009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:12.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.735+0000 7fef7cc00640 1 -- 172.21.15.28:0/3460533489 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fef6c059010 con 0x7fef800c5a70 2024-02-21T23:03:12.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.743+0000 7fac89600640 1 Processor -- start 2024-02-21T23:03:12.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.744+0000 7fac89600640 1 -- start start 2024-02-21T23:03:12.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.745+0000 7fac89600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fac841550d0 0x7fac840c5320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:12.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.745+0000 7fac89600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fac840bfa40 0x7fac840bfeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:12.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.745+0000 7fac89600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac8414f5a0 con 0x7fac8414f1c0 2024-02-21T23:03:12.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.745+0000 7fac89600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac840c5a20 con 0x7fac840bfa40 2024-02-21T23:03:12.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.745+0000 7fac89600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac840c5d00 con 0x7fac841550d0 2024-02-21T23:03:12.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.745+0000 7fac83400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fac840bfa40 0x7fac840bfeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:12.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.745+0000 7fac83400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fac840bfa40 0x7fac840bfeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46240/0 (socket says 172.21.15.28:46240) 2024-02-21T23:03:12.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.745+0000 7fac83400640 1 -- 172.21.15.28:0/4093846048 learned_addr learned my addr 172.21.15.28:0/4093846048 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:12.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.746+0000 7fac82000640 1 --2- 172.21.15.28:0/4093846048 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fac841550d0 0x7fac840c5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:12.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.746+0000 7fac83400640 1 -- 172.21.15.28:0/4093846048 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fac841550d0 msgr2=0x7fac840c5320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:12.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.746+0000 7fac83400640 1 --2- 172.21.15.28:0/4093846048 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fac841550d0 0x7fac840c5320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:12.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.746+0000 7fac83400640 1 -- 172.21.15.28:0/4093846048 >> v1:172.21.15.81:6789/0 conn(0x7fac8414f1c0 legacy=0x7fac840ca890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:12.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.746+0000 7fac83400640 1 -- 172.21.15.28:0/4093846048 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac840c5fe0 con 0x7fac840bfa40 2024-02-21T23:03:12.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.748+0000 7fac83400640 1 --2- 172.21.15.28:0/4093846048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac840bfa40 0x7fac840bfeb0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fac7c009e50 tx=0x7fac7c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=795d389dc8ecfc68 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:12.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.748+0000 7fac81600640 1 -- 172.21.15.28:0/4093846048 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac7c02ca80 con 0x7fac840bfa40 2024-02-21T23:03:12.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.748+0000 7fac81600640 1 -- 172.21.15.28:0/4093846048 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac7c02cc20 con 0x7fac840bfa40 2024-02-21T23:03:12.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.748+0000 7fac81600640 1 -- 172.21.15.28:0/4093846048 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac7c0343f0 con 0x7fac840bfa40 2024-02-21T23:03:12.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.749+0000 7fac89600640 1 -- 172.21.15.28:0/4093846048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac840bfa40 msgr2=0x7fac840bfeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:12.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.749+0000 7fac89600640 1 --2- 172.21.15.28:0/4093846048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac840bfa40 0x7fac840bfeb0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fac7c009e50 tx=0x7fac7c02bee0 comp rx=0 tx=0).stop 2024-02-21T23:03:12.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.749+0000 7fac89600640 1 -- 172.21.15.28:0/4093846048 shutdown_connections 2024-02-21T23:03:12.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.749+0000 7fac89600640 1 --2- 172.21.15.28:0/4093846048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac840bfa40 0x7fac840bfeb0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:12.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.749+0000 7fac89600640 1 --2- 172.21.15.28:0/4093846048 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fac841550d0 0x7fac840c5320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:12.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.749+0000 7fac89600640 1 -- 172.21.15.28:0/4093846048 >> 172.21.15.28:0/4093846048 conn(0x7fac8408b1c0 msgr2=0x7fac840725a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:12.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.749+0000 7fac89600640 1 -- 172.21.15.28:0/4093846048 shutdown_connections 2024-02-21T23:03:12.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.749+0000 7fac89600640 1 -- 172.21.15.28:0/4093846048 wait complete. 2024-02-21T23:03:12.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.751+0000 7fac89600640 1 Processor -- start 2024-02-21T23:03:12.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.752+0000 7f3f75c00640 1 Processor -- start 2024-02-21T23:03:12.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.751+0000 7fac89600640 1 -- start start 2024-02-21T23:03:12.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.752+0000 7f3f75c00640 1 -- start start 2024-02-21T23:03:12.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.752+0000 7fac89600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac840bfa40 0x7fac8414e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:12.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.752+0000 7fac89600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac8414f1c0 0x7fac84153510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:12.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.752+0000 7fac89600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fac841550d0 0x7fac840c9530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:12.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.752+0000 7fac89600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac840cafa0 con 0x7fac840bfa40 2024-02-21T23:03:12.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.752+0000 7fac89600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac840c65a0 con 0x7fac8414f1c0 2024-02-21T23:03:12.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.752+0000 7fac89600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac840cb120 con 0x7fac841550d0 2024-02-21T23:03:12.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.753+0000 7fac82a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac840bfa40 0x7fac8414e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:12.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.753+0000 7fac82a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac840bfa40 0x7fac8414e480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46242/0 (socket says 172.21.15.28:46242) 2024-02-21T23:03:12.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.753+0000 7fac82a00640 1 -- 172.21.15.28:0/104452742 learned_addr learned my addr 172.21.15.28:0/104452742 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:12.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.753+0000 7fac82000640 1 --2- 172.21.15.28:0/104452742 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac8414f1c0 0x7fac84153510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:12.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.753+0000 7fac83400640 1 --2- 172.21.15.28:0/104452742 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fac841550d0 0x7fac840c9530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:12.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.753+0000 7fac82a00640 1 -- 172.21.15.28:0/104452742 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fac841550d0 msgr2=0x7fac840c9530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:12.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.753+0000 7fac82a00640 1 --2- 172.21.15.28:0/104452742 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fac841550d0 0x7fac840c9530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:12.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.753+0000 7fac82a00640 1 -- 172.21.15.28:0/104452742 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac8414f1c0 msgr2=0x7fac84153510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:12.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.753+0000 7fac82a00640 1 --2- 172.21.15.28:0/104452742 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac8414f1c0 0x7fac84153510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:12.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.753+0000 7fac82a00640 1 -- 172.21.15.28:0/104452742 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac84153d50 con 0x7fac840bfa40 2024-02-21T23:03:12.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.754+0000 7fac82a00640 1 --2- 172.21.15.28:0/104452742 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac840bfa40 0x7fac8414e480 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fac7400c400 tx=0x7fac7400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:12.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.754+0000 7fac7aa00640 1 -- 172.21.15.28:0/104452742 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac740052d0 con 0x7fac840bfa40 2024-02-21T23:03:12.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.754+0000 7fac7aa00640 1 -- 172.21.15.28:0/104452742 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac74005490 con 0x7fac840bfa40 2024-02-21T23:03:12.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.754+0000 7fac7aa00640 1 -- 172.21.15.28:0/104452742 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac74007260 con 0x7fac840bfa40 2024-02-21T23:03:12.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.754+0000 7f3f75c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3f701550d0 0x7f3f700c5320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:12.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.754+0000 7f3f75c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3f700bfa40 0x7f3f700bfeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:12.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.754+0000 7f3f75c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f7014f5a0 con 0x7f3f7014f1c0 2024-02-21T23:03:12.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.754+0000 7f3f75c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f700c5a20 con 0x7f3f700bfa40 2024-02-21T23:03:12.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.754+0000 7f3f75c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f700c5d00 con 0x7f3f701550d0 2024-02-21T23:03:12.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.754+0000 7f3f6ea00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f3f7014f1c0 0x7f3f700ca890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:48666/0 (socket says 172.21.15.28:48666) 2024-02-21T23:03:12.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.754+0000 7f3f6ea00640 1 -- 172.21.15.28:0/2680434364 learned_addr learned my addr 172.21.15.28:0/2680434364 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:03:12.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.754+0000 7f3f6e000640 1 --2- 172.21.15.28:0/2680434364 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3f701550d0 0x7f3f700c5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:12.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.754+0000 7f3f6f400640 1 --2- 172.21.15.28:0/2680434364 >> v2:172.21.15.28:3300/0 conn(0x7f3f700bfa40 0x7f3f700bfeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:12.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.755+0000 7fac89600640 1 -- 172.21.15.28:0/104452742 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac84154030 con 0x7fac840bfa40 2024-02-21T23:03:12.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.755+0000 7f3f6d600640 1 -- 172.21.15.28:0/2680434364 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4278790381 0 0) 0x7f3f7014f5a0 con 0x7f3f7014f1c0 2024-02-21T23:03:12.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.755+0000 7f3f6d600640 1 -- 172.21.15.28:0/2680434364 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f54003520 con 0x7f3f7014f1c0 2024-02-21T23:03:12.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.755+0000 7fac89600640 1 -- 172.21.15.28:0/104452742 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac84154c40 con 0x7fac840bfa40 2024-02-21T23:03:12.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.755+0000 7f3f6f400640 1 -- 172.21.15.28:0/2680434364 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3f701550d0 msgr2=0x7f3f700c5320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:12.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.755+0000 7f3f6f400640 1 --2- 172.21.15.28:0/2680434364 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3f701550d0 0x7f3f700c5320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:12.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.755+0000 7f3f6f400640 1 -- 172.21.15.28:0/2680434364 >> v1:172.21.15.28:6789/0 conn(0x7f3f7014f1c0 legacy=0x7f3f700ca890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:12.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.755+0000 7f3f6f400640 1 -- 172.21.15.28:0/2680434364 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f700c5fe0 con 0x7f3f700bfa40 2024-02-21T23:03:12.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.756+0000 7fac89600640 1 -- 172.21.15.28:0/104452742 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac8406b2e0 con 0x7fac840bfa40 2024-02-21T23:03:12.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.756+0000 7f3f6f400640 1 --2- 172.21.15.28:0/2680434364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f700bfa40 0x7f3f700bfeb0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f3f68009d90 tx=0x7f3f6802be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18b3caa224aa1577 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:12.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.757+0000 7f3f6d600640 1 -- 172.21.15.28:0/2680434364 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f68039070 con 0x7f3f700bfa40 2024-02-21T23:03:12.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.757+0000 7f3f6d600640 1 -- 172.21.15.28:0/2680434364 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f6802cbc0 con 0x7f3f700bfa40 2024-02-21T23:03:12.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.757+0000 7f3f6d600640 1 -- 172.21.15.28:0/2680434364 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f68034440 con 0x7f3f700bfa40 2024-02-21T23:03:12.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.758+0000 7f3f75c00640 1 -- 172.21.15.28:0/2680434364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f700bfa40 msgr2=0x7f3f700bfeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:12.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.758+0000 7f3f75c00640 1 --2- 172.21.15.28:0/2680434364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f700bfa40 0x7f3f700bfeb0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f3f68009d90 tx=0x7f3f6802be10 comp rx=0 tx=0).stop 2024-02-21T23:03:12.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.758+0000 7f3f75c00640 1 -- 172.21.15.28:0/2680434364 shutdown_connections 2024-02-21T23:03:12.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.758+0000 7f3f75c00640 1 --2- 172.21.15.28:0/2680434364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f700bfa40 0x7f3f700bfeb0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:12.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.758+0000 7f3f75c00640 1 --2- 172.21.15.28:0/2680434364 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3f701550d0 0x7f3f700c5320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:12.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.758+0000 7f3f75c00640 1 -- 172.21.15.28:0/2680434364 >> 172.21.15.28:0/2680434364 conn(0x7f3f7008b1c0 msgr2=0x7f3f700725a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:12.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.758+0000 7f3f75c00640 1 -- 172.21.15.28:0/2680434364 shutdown_connections 2024-02-21T23:03:12.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.758+0000 7f3f75c00640 1 -- 172.21.15.28:0/2680434364 wait complete. 2024-02-21T23:03:12.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.758+0000 7fac7aa00640 1 -- 172.21.15.28:0/104452742 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fac74007400 con 0x7fac840bfa40 2024-02-21T23:03:12.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.759+0000 7fac7aa00640 1 --2- 172.21.15.28:0/104452742 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fac6406b4d0 0x7fac6406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:12.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.759+0000 7fac7aa00640 1 -- 172.21.15.28:0/104452742 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fac740899d0 con 0x7fac840bfa40 2024-02-21T23:03:12.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.760+0000 7f3f75c00640 1 Processor -- start 2024-02-21T23:03:12.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.760+0000 7fac82000640 1 --2- 172.21.15.28:0/104452742 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fac6406b4d0 0x7fac6406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:12.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.760+0000 7f3f75c00640 1 -- start start 2024-02-21T23:03:12.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.761+0000 7f3f75c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3f700bfa40 0x7f3f701efa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:12.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.761+0000 7fac82000640 1 --2- 172.21.15.28:0/104452742 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fac6406b4d0 0x7fac6406d990 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fac8414ebd0 tx=0x7fac70009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:12.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.761+0000 7f3f75c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f7014f1c0 0x7f3f7014da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:12.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.761+0000 7f3f75c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f701550d0 0x7f3f701474e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:12.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.761+0000 7f3f75c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f700cafa0 con 0x7f3f7014f1c0 2024-02-21T23:03:12.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.761+0000 7f3f75c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f700c65a0 con 0x7f3f701550d0 2024-02-21T23:03:12.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.761+0000 7f3f75c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f700cb120 con 0x7f3f700bfa40 2024-02-21T23:03:12.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.761+0000 7f3f6e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f7014f1c0 0x7f3f7014da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:12.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.762+0000 7f3f6e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f7014f1c0 0x7f3f7014da30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46250/0 (socket says 172.21.15.28:46250) 2024-02-21T23:03:12.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.762+0000 7f3f6e000640 1 -- 172.21.15.28:0/403762135 learned_addr learned my addr 172.21.15.28:0/403762135 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:12.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.762+0000 7f3f6ea00640 1 --2- 172.21.15.28:0/403762135 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3f700bfa40 0x7f3f701efa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:12.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.762+0000 7f3f6f400640 1 --2- 172.21.15.28:0/403762135 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f701550d0 0x7f3f701474e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:12.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.762+0000 7f3f6e000640 1 -- 172.21.15.28:0/403762135 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3f700bfa40 msgr2=0x7f3f701efa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:12.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.762+0000 7f3f6e000640 1 --2- 172.21.15.28:0/403762135 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3f700bfa40 0x7f3f701efa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:12.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.762+0000 7f3f6e000640 1 -- 172.21.15.28:0/403762135 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f701550d0 msgr2=0x7f3f701474e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:12.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.762+0000 7f3f6e000640 1 --2- 172.21.15.28:0/403762135 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f701550d0 0x7f3f701474e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:12.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.762+0000 7f3f6e000640 1 -- 172.21.15.28:0/403762135 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f700c9460 con 0x7f3f7014f1c0 2024-02-21T23:03:12.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.763+0000 7f3f6e000640 1 --2- 172.21.15.28:0/403762135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f7014f1c0 0x7f3f7014da30 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f3f5c00e870 tx=0x7f3f5c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:12.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.768+0000 7f3f66a00640 1 -- 172.21.15.28:0/403762135 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f5c017070 con 0x7f3f7014f1c0 2024-02-21T23:03:12.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.768+0000 7f3f66a00640 1 -- 172.21.15.28:0/403762135 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f5c00c940 con 0x7f3f7014f1c0 2024-02-21T23:03:12.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.768+0000 7f3f66a00640 1 -- 172.21.15.28:0/403762135 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f5c00cd30 con 0x7f3f7014f1c0 2024-02-21T23:03:12.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.768+0000 7f3f75c00640 1 -- 172.21.15.28:0/403762135 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f701481d0 con 0x7f3f7014f1c0 2024-02-21T23:03:12.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.768+0000 7f3f75c00640 1 -- 172.21.15.28:0/403762135 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f7014f9e0 con 0x7f3f7014f1c0 2024-02-21T23:03:12.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.768+0000 7fac7aa00640 1 -- 172.21.15.28:0/104452742 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fac74057eb0 con 0x7fac840bfa40 2024-02-21T23:03:12.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.769+0000 7f3f75c00640 1 -- 172.21.15.28:0/403762135 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f7006b2e0 con 0x7f3f7014f1c0 2024-02-21T23:03:12.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.770+0000 7f3f66a00640 1 -- 172.21.15.28:0/403762135 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3f5c01a070 con 0x7f3f7014f1c0 2024-02-21T23:03:12.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.772+0000 7f3f66a00640 1 --2- 172.21.15.28:0/403762135 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3f5006b4d0 0x7f3f5006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:12.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.772+0000 7f3f66a00640 1 -- 172.21.15.28:0/403762135 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f3f5c08a320 con 0x7f3f7014f1c0 2024-02-21T23:03:12.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.773+0000 7f3f6ea00640 1 --2- 172.21.15.28:0/403762135 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3f5006b4d0 0x7f3f5006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:12.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.774+0000 7f3f6ea00640 1 --2- 172.21.15.28:0/403762135 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3f5006b4d0 0x7f3f5006d990 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3f60005880 tx=0x7f3f60009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:12.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.780+0000 7f3f66a00640 1 -- 172.21.15.28:0/403762135 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3f5c058800 con 0x7f3f7014f1c0 2024-02-21T23:03:12.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.790+0000 7fa06b200640 1 Processor -- start 2024-02-21T23:03:12.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.790+0000 7fa06b200640 1 -- start start 2024-02-21T23:03:12.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.791+0000 7fa06b200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fa06c1524c0 0x7fa06c155a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:12.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.791+0000 7fa06b200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa06c0c5040 0x7fa06c06ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:12.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.791+0000 7fa06b200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa06c0c85c0 0x7fa06c0c70d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:12.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.791+0000 7fa06b200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa06c06df50 con 0x7fa06c0c5040 2024-02-21T23:03:12.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.791+0000 7fa06b200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa06c06d3b0 con 0x7fa06c1524c0 2024-02-21T23:03:12.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.791+0000 7fa06b200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa06c0c5520 con 0x7fa06c0c85c0 2024-02-21T23:03:12.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.792+0000 7fa069400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa06c0c5040 0x7fa06c06ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:12.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.792+0000 7fa069e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fa06c1524c0 0x7fa06c155a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:12.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.792+0000 7fa069400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa06c0c5040 0x7fa06c06ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46256/0 (socket says 172.21.15.28:46256) 2024-02-21T23:03:12.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.792+0000 7fa069400640 1 -- 172.21.15.28:0/2495691519 learned_addr learned my addr 172.21.15.28:0/2495691519 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:12.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.792+0000 7fa06a800640 1 --2- 172.21.15.28:0/2495691519 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa06c0c85c0 0x7fa06c0c70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:12.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.792+0000 7fa069400640 1 -- 172.21.15.28:0/2495691519 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa06c0c85c0 msgr2=0x7fa06c0c70d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:12.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.792+0000 7fa069400640 1 --2- 172.21.15.28:0/2495691519 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa06c0c85c0 0x7fa06c0c70d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:12.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.792+0000 7fa069400640 1 -- 172.21.15.28:0/2495691519 >> v2:172.21.15.81:3300/0 conn(0x7fa06c1524c0 msgr2=0x7fa06c155a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:12.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.792+0000 7fa069400640 1 --2- 172.21.15.28:0/2495691519 >> v2:172.21.15.81:3300/0 conn(0x7fa06c1524c0 0x7fa06c155a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:12.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.792+0000 7fa069400640 1 -- 172.21.15.28:0/2495691519 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa06c0c5800 con 0x7fa06c0c5040 2024-02-21T23:03:12.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.794+0000 7fa069400640 1 --2- 172.21.15.28:0/2495691519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa06c0c5040 0x7fa06c06ce70 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fa06c0cb790 tx=0x7fa05802b800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=681dca77d7396015 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:12.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.795+0000 7fa068a00640 1 -- 172.21.15.28:0/2495691519 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa058031760 con 0x7fa06c0c5040 2024-02-21T23:03:12.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.795+0000 7fa068a00640 1 -- 172.21.15.28:0/2495691519 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa058037070 con 0x7fa06c0c5040 2024-02-21T23:03:12.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.795+0000 7fa068a00640 1 -- 172.21.15.28:0/2495691519 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa058031b60 con 0x7fa06c0c5040 2024-02-21T23:03:12.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.795+0000 7fa06b200640 1 -- 172.21.15.28:0/2495691519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa06c0c5040 msgr2=0x7fa06c06ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:12.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.795+0000 7fa06b200640 1 --2- 172.21.15.28:0/2495691519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa06c0c5040 0x7fa06c06ce70 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fa06c0cb790 tx=0x7fa05802b800 comp rx=0 tx=0).stop 2024-02-21T23:03:12.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.796+0000 7fa06b200640 1 -- 172.21.15.28:0/2495691519 shutdown_connections 2024-02-21T23:03:12.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.796+0000 7fa06b200640 1 --2- 172.21.15.28:0/2495691519 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa06c0c85c0 0x7fa06c0c70d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:12.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.796+0000 7fa06b200640 1 --2- 172.21.15.28:0/2495691519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa06c0c5040 0x7fa06c06ce70 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:12.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.796+0000 7fa06b200640 1 --2- 172.21.15.28:0/2495691519 >> v2:172.21.15.81:3300/0 conn(0x7fa06c1524c0 0x7fa06c155a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:12.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.796+0000 7fa06b200640 1 -- 172.21.15.28:0/2495691519 >> 172.21.15.28:0/2495691519 conn(0x7fa06c0ac2b0 msgr2=0x7fa06c0ac6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:12.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.796+0000 7fa06b200640 1 -- 172.21.15.28:0/2495691519 shutdown_connections 2024-02-21T23:03:12.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.796+0000 7fa06b200640 1 -- 172.21.15.28:0/2495691519 wait complete. 2024-02-21T23:03:12.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.798+0000 7fa06b200640 1 Processor -- start 2024-02-21T23:03:12.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.799+0000 7fa06b200640 1 -- start start 2024-02-21T23:03:12.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.800+0000 7fa06b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa06c0c5040 0x7fa06c0ae4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:12.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.800+0000 7fa06b200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa06c0c85c0 0x7fa06c0c2ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:12.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.800+0000 7fa06b200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa06c1524c0 0x7fa06c1ee240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:12.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.800+0000 7fa06b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa06c1528a0 con 0x7fa06c0c5040 2024-02-21T23:03:12.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.800+0000 7fa06b200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa06c06c950 con 0x7fa06c1524c0 2024-02-21T23:03:12.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.800+0000 7fa06b200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa06c0c8a70 con 0x7fa06c0c85c0 2024-02-21T23:03:12.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.800+0000 7fa069e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa06c0c5040 0x7fa06c0ae4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:12.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.801+0000 7fa069e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa06c0c5040 0x7fa06c0ae4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46258/0 (socket says 172.21.15.28:46258) 2024-02-21T23:03:12.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.801+0000 7fa069e00640 1 -- 172.21.15.28:0/2103088840 learned_addr learned my addr 172.21.15.28:0/2103088840 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:12.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.801+0000 7fa06a800640 1 --2- 172.21.15.28:0/2103088840 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa06c1524c0 0x7fa06c1ee240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:12.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.801+0000 7fa069400640 1 --2- 172.21.15.28:0/2103088840 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa06c0c85c0 0x7fa06c0c2ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:12.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.801+0000 7fa069e00640 1 -- 172.21.15.28:0/2103088840 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa06c0c85c0 msgr2=0x7fa06c0c2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:12.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.801+0000 7fa069e00640 1 --2- 172.21.15.28:0/2103088840 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa06c0c85c0 0x7fa06c0c2ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:12.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.801+0000 7fa069e00640 1 -- 172.21.15.28:0/2103088840 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa06c1524c0 msgr2=0x7fa06c1ee240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:12.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.801+0000 7fa069e00640 1 --2- 172.21.15.28:0/2103088840 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa06c1524c0 0x7fa06c1ee240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:12.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.801+0000 7fa069e00640 1 -- 172.21.15.28:0/2103088840 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa06c153310 con 0x7fa06c0c5040 2024-02-21T23:03:12.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.802+0000 7fa069e00640 1 --2- 172.21.15.28:0/2103088840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa06c0c5040 0x7fa06c0ae4c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa05c00e870 tx=0x7fa05c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:12.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.802+0000 7fa062000640 1 -- 172.21.15.28:0/2103088840 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa05c017070 con 0x7fa06c0c5040 2024-02-21T23:03:12.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.802+0000 7fa062000640 1 -- 172.21.15.28:0/2103088840 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa05c00c8e0 con 0x7fa06c0c5040 2024-02-21T23:03:12.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.802+0000 7fa062000640 1 -- 172.21.15.28:0/2103088840 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa05c00ccd0 con 0x7fa06c0c5040 2024-02-21T23:03:12.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.803+0000 7fa06b200640 1 -- 172.21.15.28:0/2103088840 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa06c1eee60 con 0x7fa06c0c5040 2024-02-21T23:03:12.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.803+0000 7fa06b200640 1 -- 172.21.15.28:0/2103088840 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa06c0c26c0 con 0x7fa06c0c5040 2024-02-21T23:03:12.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.805+0000 7fa06b200640 1 -- 172.21.15.28:0/2103088840 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa06c150de0 con 0x7fa06c0c5040 2024-02-21T23:03:12.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.807+0000 7fa062000640 1 -- 172.21.15.28:0/2103088840 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa05c01a070 con 0x7fa06c0c5040 2024-02-21T23:03:12.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.808+0000 7fa062000640 1 --2- 172.21.15.28:0/2103088840 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa04c06b500 0x7fa04c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:12.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.808+0000 7fa062000640 1 -- 172.21.15.28:0/2103088840 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fa05c08a260 con 0x7fa06c0c5040 2024-02-21T23:03:12.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.809+0000 7fa069400640 1 --2- 172.21.15.28:0/2103088840 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa04c06b500 0x7fa04c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:12.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.810+0000 7fa069400640 1 --2- 172.21.15.28:0/2103088840 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa04c06b500 0x7fa04c06d9c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa058036000 tx=0x7fa05803d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:12.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.813+0000 7fa062000640 1 -- 172.21.15.28:0/2103088840 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fa06c150de0 con 0x7fa06c0c5040 2024-02-21T23:03:12.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.948+0000 7fef87a00640 1 -- 172.21.15.28:0/3460533489 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fef800bdf50 con 0x7fef800c5a70 2024-02-21T23:03:12.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.949+0000 7fef7cc00640 1 -- 172.21.15.28:0/3460533489 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fef6c058910 con 0x7fef800c5a70 2024-02-21T23:03:12.937 INFO:teuthology.orchestra.run.smithi028.stdout:55834574859 2024-02-21T23:03:12.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.951+0000 7fef87a00640 1 -- 172.21.15.28:0/3460533489 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fef6006b500 msgr2=0x7fef6006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:12.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.951+0000 7fef87a00640 1 --2- 172.21.15.28:0/3460533489 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fef6006b500 0x7fef6006d9c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fef74001640 tx=0x7fef74009040 comp rx=0 tx=0).stop 2024-02-21T23:03:12.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.951+0000 7fef87a00640 1 -- 172.21.15.28:0/3460533489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef800c5a70 msgr2=0x7fef8014d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:12.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.951+0000 7fef87a00640 1 --2- 172.21.15.28:0/3460533489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef800c5a70 0x7fef8014d1c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fef6c00e9b0 tx=0x7fef6c00ee70 comp rx=0 tx=0).stop 2024-02-21T23:03:12.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.951+0000 7fef87a00640 1 -- 172.21.15.28:0/3460533489 shutdown_connections 2024-02-21T23:03:12.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.951+0000 7fef87a00640 1 --2- 172.21.15.28:0/3460533489 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fef6006b500 0x7fef6006d9c0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:12.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.951+0000 7fef87a00640 1 --2- 172.21.15.28:0/3460533489 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fef80156190 0x7fef8014a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:12.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.951+0000 7fef87a00640 1 --2- 172.21.15.28:0/3460533489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef800c5a70 0x7fef8014d1c0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:12.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.951+0000 7fef87a00640 1 --2- 172.21.15.28:0/3460533489 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fef8006e110 0x7fef80151200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:12.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.951+0000 7fef87a00640 1 -- 172.21.15.28:0/3460533489 >> 172.21.15.28:0/3460533489 conn(0x7fef8006b820 msgr2=0x7fef800c6530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:12.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.951+0000 7fef87a00640 1 -- 172.21.15.28:0/3460533489 shutdown_connections 2024-02-21T23:03:12.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.951+0000 7fef87a00640 1 -- 172.21.15.28:0/3460533489 wait complete. 2024-02-21T23:03:12.956 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574859 for osd.4 2024-02-21T23:03:12.956 DEBUG:teuthology.parallel:result is None 2024-02-21T23:03:12.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.979+0000 7f3f75c00640 1 -- 172.21.15.28:0/403762135 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f3f7004f370 con 0x7f3f7014f1c0 2024-02-21T23:03:12.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.980+0000 7f3f66a00640 1 -- 172.21.15.28:0/403762135 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3f5c08a0e0 con 0x7f3f7014f1c0 2024-02-21T23:03:12.968 INFO:teuthology.orchestra.run.smithi028.stdout:55834574859 2024-02-21T23:03:12.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.982+0000 7f3f75c00640 1 -- 172.21.15.28:0/403762135 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3f5006b4d0 msgr2=0x7f3f5006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:12.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.982+0000 7f3f75c00640 1 --2- 172.21.15.28:0/403762135 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3f5006b4d0 0x7f3f5006d990 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3f60005880 tx=0x7f3f60009040 comp rx=0 tx=0).stop 2024-02-21T23:03:12.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.982+0000 7f3f75c00640 1 -- 172.21.15.28:0/403762135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f7014f1c0 msgr2=0x7f3f7014da30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:12.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.982+0000 7f3f75c00640 1 --2- 172.21.15.28:0/403762135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f7014f1c0 0x7f3f7014da30 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f3f5c00e870 tx=0x7f3f5c00ed30 comp rx=0 tx=0).stop 2024-02-21T23:03:12.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.982+0000 7f3f75c00640 1 -- 172.21.15.28:0/403762135 shutdown_connections 2024-02-21T23:03:12.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.982+0000 7f3f75c00640 1 --2- 172.21.15.28:0/403762135 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3f5006b4d0 0x7f3f5006d990 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:12.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.982+0000 7f3f75c00640 1 --2- 172.21.15.28:0/403762135 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f701550d0 0x7f3f701474e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:12.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.982+0000 7f3f75c00640 1 --2- 172.21.15.28:0/403762135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f7014f1c0 0x7f3f7014da30 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:12.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.982+0000 7f3f75c00640 1 --2- 172.21.15.28:0/403762135 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3f700bfa40 0x7f3f701efa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:12.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.982+0000 7f3f75c00640 1 -- 172.21.15.28:0/403762135 >> 172.21.15.28:0/403762135 conn(0x7f3f7008b1c0 msgr2=0x7f3f700c06b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:12.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.982+0000 7f3f75c00640 1 -- 172.21.15.28:0/403762135 shutdown_connections 2024-02-21T23:03:12.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:12.982+0000 7f3f75c00640 1 -- 172.21.15.28:0/403762135 wait complete. 2024-02-21T23:03:12.988 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574859 for osd.6 2024-02-21T23:03:12.988 DEBUG:teuthology.parallel:result is None 2024-02-21T23:03:12.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.002+0000 7fac89600640 1 -- 172.21.15.28:0/104452742 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fac840c6c70 con 0x7fac840bfa40 2024-02-21T23:03:12.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.002+0000 7fac7aa00640 1 -- 172.21.15.28:0/104452742 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fac7405bf50 con 0x7fac840bfa40 2024-02-21T23:03:12.991 INFO:teuthology.orchestra.run.smithi028.stdout:55834574858 2024-02-21T23:03:12.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.005+0000 7fac89600640 1 -- 172.21.15.28:0/104452742 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fac6406b4d0 msgr2=0x7fac6406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:12.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.005+0000 7fac89600640 1 --2- 172.21.15.28:0/104452742 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fac6406b4d0 0x7fac6406d990 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fac8414ebd0 tx=0x7fac70009190 comp rx=0 tx=0).stop 2024-02-21T23:03:12.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.005+0000 7fac89600640 1 -- 172.21.15.28:0/104452742 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac840bfa40 msgr2=0x7fac8414e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:12.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.005+0000 7fac89600640 1 --2- 172.21.15.28:0/104452742 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac840bfa40 0x7fac8414e480 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fac7400c400 tx=0x7fac7400c8c0 comp rx=0 tx=0).stop 2024-02-21T23:03:12.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.005+0000 7fac89600640 1 -- 172.21.15.28:0/104452742 shutdown_connections 2024-02-21T23:03:12.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.005+0000 7fac89600640 1 --2- 172.21.15.28:0/104452742 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fac6406b4d0 0x7fac6406d990 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:12.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.005+0000 7fac89600640 1 --2- 172.21.15.28:0/104452742 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fac841550d0 0x7fac840c9530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:12.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.005+0000 7fac89600640 1 --2- 172.21.15.28:0/104452742 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac8414f1c0 0x7fac84153510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:12.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.005+0000 7fac89600640 1 --2- 172.21.15.28:0/104452742 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac840bfa40 0x7fac8414e480 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:12.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.005+0000 7fac89600640 1 -- 172.21.15.28:0/104452742 >> 172.21.15.28:0/104452742 conn(0x7fac8408b1c0 msgr2=0x7fac840c3d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:12.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.005+0000 7fac89600640 1 -- 172.21.15.28:0/104452742 shutdown_connections 2024-02-21T23:03:12.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.005+0000 7fac89600640 1 -- 172.21.15.28:0/104452742 wait complete. 2024-02-21T23:03:13.011 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574858 for osd.5 2024-02-21T23:03:13.011 DEBUG:teuthology.parallel:result is None 2024-02-21T23:03:13.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.038+0000 7fa06b200640 1 -- 172.21.15.28:0/2103088840 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fa06c04f290 con 0x7fa06c0c5040 2024-02-21T23:03:13.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.039+0000 7fa062000640 1 -- 172.21.15.28:0/2103088840 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa05c058740 con 0x7fa06c0c5040 2024-02-21T23:03:13.029 INFO:teuthology.orchestra.run.smithi028.stdout:55834574858 2024-02-21T23:03:13.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.041+0000 7fa06b200640 1 -- 172.21.15.28:0/2103088840 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa04c06b500 msgr2=0x7fa04c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:13.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.041+0000 7fa06b200640 1 --2- 172.21.15.28:0/2103088840 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa04c06b500 0x7fa04c06d9c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa058036000 tx=0x7fa05803d040 comp rx=0 tx=0).stop 2024-02-21T23:03:13.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.041+0000 7fa06b200640 1 -- 172.21.15.28:0/2103088840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa06c0c5040 msgr2=0x7fa06c0ae4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:13.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.041+0000 7fa06b200640 1 --2- 172.21.15.28:0/2103088840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa06c0c5040 0x7fa06c0ae4c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa05c00e870 tx=0x7fa05c00ed30 comp rx=0 tx=0).stop 2024-02-21T23:03:13.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.042+0000 7fa06b200640 1 -- 172.21.15.28:0/2103088840 shutdown_connections 2024-02-21T23:03:13.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.042+0000 7fa06b200640 1 --2- 172.21.15.28:0/2103088840 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa04c06b500 0x7fa04c06d9c0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:13.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.042+0000 7fa06b200640 1 --2- 172.21.15.28:0/2103088840 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa06c0c85c0 0x7fa06c0c2ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:13.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.042+0000 7fa06b200640 1 --2- 172.21.15.28:0/2103088840 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa06c1524c0 0x7fa06c1ee240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:13.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.042+0000 7fa06b200640 1 --2- 172.21.15.28:0/2103088840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa06c0c5040 0x7fa06c0ae4c0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:13.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.042+0000 7fa06b200640 1 -- 172.21.15.28:0/2103088840 >> 172.21.15.28:0/2103088840 conn(0x7fa06c0ac2b0 msgr2=0x7fa06c1506d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:13.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.042+0000 7fa06b200640 1 -- 172.21.15.28:0/2103088840 shutdown_connections 2024-02-21T23:03:13.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.042+0000 7fa06b200640 1 -- 172.21.15.28:0/2103088840 wait complete. 2024-02-21T23:03:13.048 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574858 for osd.7 2024-02-21T23:03:13.048 DEBUG:teuthology.parallel:result is None 2024-02-21T23:03:13.048 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-02-21T23:03:13.048 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-21T23:03:13.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.321+0000 7fb967e00640 1 Processor -- start 2024-02-21T23:03:13.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.321+0000 7fb967e00640 1 -- start start 2024-02-21T23:03:13.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.323+0000 7fb967e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb9600c3490 0x7fb96006dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:13.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.323+0000 7fb967e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fb9600c0270 0x7fb9600c0700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:13.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.323+0000 7fb967e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9600c0c40 con 0x7fb9600a5f40 2024-02-21T23:03:13.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.323+0000 7fb967e00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb96006eff0 con 0x7fb9600c0270 2024-02-21T23:03:13.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.323+0000 7fb967e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9600bfde0 con 0x7fb9600c3490 2024-02-21T23:03:13.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.323+0000 7fb964a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb9600a5f40 0x7fb960152140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:48682/0 (socket says 172.21.15.28:48682) 2024-02-21T23:03:13.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.323+0000 7fb964a00640 1 -- 172.21.15.28:0/3205613428 learned_addr learned my addr 172.21.15.28:0/3205613428 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:03:13.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.323+0000 7fb965400640 1 --2- 172.21.15.28:0/3205613428 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb9600c3490 0x7fb96006dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:13.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.324+0000 7fb965e00640 1 --2- 172.21.15.28:0/3205613428 >> v2:172.21.15.81:3300/0 conn(0x7fb9600c0270 0x7fb9600c0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:13.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.324+0000 7fb95fe00640 1 -- 172.21.15.28:0/3205613428 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4012669491 0 0) 0x7fb9600c0c40 con 0x7fb9600a5f40 2024-02-21T23:03:13.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.324+0000 7fb95fe00640 1 -- 172.21.15.28:0/3205613428 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb948003520 con 0x7fb9600a5f40 2024-02-21T23:03:13.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.324+0000 7fb95fe00640 1 -- 172.21.15.28:0/3205613428 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2981203215 0 0) 0x7fb948003520 con 0x7fb9600a5f40 2024-02-21T23:03:13.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.324+0000 7fb95fe00640 1 -- 172.21.15.28:0/3205613428 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb9600c3490 msgr2=0x7fb96006dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:13.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.324+0000 7fb95fe00640 1 --2- 172.21.15.28:0/3205613428 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb9600c3490 0x7fb96006dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:13.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.324+0000 7fb95fe00640 1 -- 172.21.15.28:0/3205613428 >> v2:172.21.15.81:3300/0 conn(0x7fb9600c0270 msgr2=0x7fb9600c0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:13.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.324+0000 7fb95fe00640 1 --2- 172.21.15.28:0/3205613428 >> v2:172.21.15.81:3300/0 conn(0x7fb9600c0270 0x7fb9600c0700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:13.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.324+0000 7fb95fe00640 1 -- 172.21.15.28:0/3205613428 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb96006e750 con 0x7fb9600a5f40 2024-02-21T23:03:13.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.324+0000 7fb95fe00640 1 -- 172.21.15.28:0/3205613428 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fb94c001670 con 0x7fb9600a5f40 2024-02-21T23:03:13.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.324+0000 7fb95fe00640 1 -- 172.21.15.28:0/3205613428 >> v1:172.21.15.28:6789/0 conn(0x7fb9600a5f40 legacy=0x7fb960152140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:13.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.324+0000 7fb95fe00640 1 --2- 172.21.15.28:0/3205613428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9480039b0 0x7fb948003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:13.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.324+0000 7fb95fe00640 1 --2- 172.21.15.28:0/3205613428 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb9600c3490 0x7fb948004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:13.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.324+0000 7fb95fe00640 1 --2- 172.21.15.28:0/3205613428 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb948004d40 0x7fb9480051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:13.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.324+0000 7fb95fe00640 1 -- 172.21.15.28:0/3205613428 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb96006eff0 con 0x7fb9480039b0 2024-02-21T23:03:13.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.324+0000 7fb95fe00640 1 -- 172.21.15.28:0/3205613428 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9600bfde0 con 0x7fb948004d40 2024-02-21T23:03:13.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.324+0000 7fb95fe00640 1 -- 172.21.15.28:0/3205613428 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb94800b9c0 con 0x7fb9600c3490 2024-02-21T23:03:13.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.325+0000 7fb965400640 1 --2- 172.21.15.28:0/3205613428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9480039b0 0x7fb948003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:13.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.325+0000 7fb965e00640 1 --2- 172.21.15.28:0/3205613428 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb948004d40 0x7fb9480051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:13.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.325+0000 7fb964a00640 1 --2- 172.21.15.28:0/3205613428 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb9600c3490 0x7fb948004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:13.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.325+0000 7fb965400640 1 -- 172.21.15.28:0/3205613428 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb9600c3490 msgr2=0x7fb948004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:13.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.325+0000 7fb965400640 1 --2- 172.21.15.28:0/3205613428 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb9600c3490 0x7fb948004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:13.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.325+0000 7fb965400640 1 -- 172.21.15.28:0/3205613428 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb948004d40 msgr2=0x7fb9480051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:13.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.325+0000 7fb965400640 1 --2- 172.21.15.28:0/3205613428 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb948004d40 0x7fb9480051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:13.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.325+0000 7fb965400640 1 -- 172.21.15.28:0/3205613428 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb948003520 con 0x7fb9480039b0 2024-02-21T23:03:13.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.326+0000 7fb965400640 1 --2- 172.21.15.28:0/3205613428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9480039b0 0x7fb948003dd0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fb954009ac0 tx=0x7fb948005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca7fef5300760732 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:13.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.326+0000 7fb95fe00640 1 -- 172.21.15.28:0/3205613428 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb954039070 con 0x7fb9480039b0 2024-02-21T23:03:13.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.326+0000 7fb95fe00640 1 -- 172.21.15.28:0/3205613428 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb95402c6b0 con 0x7fb9480039b0 2024-02-21T23:03:13.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.326+0000 7fb95fe00640 1 -- 172.21.15.28:0/3205613428 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb954009690 con 0x7fb9480039b0 2024-02-21T23:03:13.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.327+0000 7fb967e00640 1 -- 172.21.15.28:0/3205613428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9480039b0 msgr2=0x7fb948003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:13.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.327+0000 7fb967e00640 1 --2- 172.21.15.28:0/3205613428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9480039b0 0x7fb948003dd0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fb954009ac0 tx=0x7fb948005830 comp rx=0 tx=0).stop 2024-02-21T23:03:13.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.327+0000 7fb965e00640 1 -- 172.21.15.28:0/3205613428 reap_dead start 2024-02-21T23:03:13.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.327+0000 7fb967e00640 1 -- 172.21.15.28:0/3205613428 shutdown_connections 2024-02-21T23:03:13.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.327+0000 7fb967e00640 1 -- 172.21.15.28:0/3205613428 >> 172.21.15.28:0/3205613428 conn(0x7fb9600c67f0 msgr2=0x7fb9600c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:13.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.327+0000 7fb967e00640 1 -- 172.21.15.28:0/3205613428 shutdown_connections 2024-02-21T23:03:13.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.327+0000 7fb967e00640 1 -- 172.21.15.28:0/3205613428 wait complete. 2024-02-21T23:03:13.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.329+0000 7fb967e00640 1 Processor -- start 2024-02-21T23:03:13.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.329+0000 7fb967e00640 1 -- start start 2024-02-21T23:03:13.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.330+0000 7fb967e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9480039b0 0x7fb96014da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:13.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.330+0000 7fb967e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb9601556e0 0x7fb960143ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:13.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.330+0000 7fb967e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9600a7fd0 0x7fb9601440e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:13.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.330+0000 7fb967e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9600c1c70 con 0x7fb9480039b0 2024-02-21T23:03:13.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.330+0000 7fb967e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb96006ed10 con 0x7fb9600a7fd0 2024-02-21T23:03:13.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.330+0000 7fb967e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9600c1df0 con 0x7fb9601556e0 2024-02-21T23:03:13.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.330+0000 7fb965400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9480039b0 0x7fb96014da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:13.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.330+0000 7fb965400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9480039b0 0x7fb96014da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46276/0 (socket says 172.21.15.28:46276) 2024-02-21T23:03:13.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.330+0000 7fb965400640 1 -- 172.21.15.28:0/2306920694 learned_addr learned my addr 172.21.15.28:0/2306920694 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:13.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.330+0000 7fb964a00640 1 --2- 172.21.15.28:0/2306920694 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb9601556e0 0x7fb960143ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:13.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.330+0000 7fb965400640 1 -- 172.21.15.28:0/2306920694 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb9601556e0 msgr2=0x7fb960143ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:13.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.330+0000 7fb965400640 1 --2- 172.21.15.28:0/2306920694 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb9601556e0 0x7fb960143ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:13.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.330+0000 7fb965400640 1 -- 172.21.15.28:0/2306920694 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9600a7fd0 msgr2=0x7fb9601440e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:13.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.330+0000 7fb965400640 1 --2- 172.21.15.28:0/2306920694 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9600a7fd0 0x7fb9601440e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:13.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.330+0000 7fb965400640 1 -- 172.21.15.28:0/2306920694 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9600ac900 con 0x7fb9480039b0 2024-02-21T23:03:13.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.331+0000 7fb965400640 1 --2- 172.21.15.28:0/2306920694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9480039b0 0x7fb96014da10 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb954037040 tx=0x7fb954004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:13.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.331+0000 7fb95d600640 1 -- 172.21.15.28:0/2306920694 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb954039050 con 0x7fb9480039b0 2024-02-21T23:03:13.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.331+0000 7fb95d600640 1 -- 172.21.15.28:0/2306920694 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb954045070 con 0x7fb9480039b0 2024-02-21T23:03:13.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.331+0000 7fb95d600640 1 -- 172.21.15.28:0/2306920694 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb954003850 con 0x7fb9480039b0 2024-02-21T23:03:13.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.331+0000 7fb967e00640 1 -- 172.21.15.28:0/2306920694 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb960152f70 con 0x7fb9480039b0 2024-02-21T23:03:13.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.331+0000 7fb967e00640 1 -- 172.21.15.28:0/2306920694 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb96014df50 con 0x7fb9480039b0 2024-02-21T23:03:13.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.332+0000 7fb967e00640 1 -- 172.21.15.28:0/2306920694 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb960067f30 con 0x7fb9480039b0 2024-02-21T23:03:13.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.333+0000 7fb95d600640 1 -- 172.21.15.28:0/2306920694 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb9540039f0 con 0x7fb9480039b0 2024-02-21T23:03:13.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.334+0000 7fb95d600640 1 --2- 172.21.15.28:0/2306920694 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb94006b500 0x7fb94006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:13.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.335+0000 7fb95d600640 1 -- 172.21.15.28:0/2306920694 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fb9540ac590 con 0x7fb9480039b0 2024-02-21T23:03:13.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.335+0000 7fb964a00640 1 --2- 172.21.15.28:0/2306920694 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb94006b500 0x7fb94006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:13.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.336+0000 7fb964a00640 1 --2- 172.21.15.28:0/2306920694 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb94006b500 0x7fb94006d9c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb94c001eb0 tx=0x7fb94c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:13.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.339+0000 7fb95d600640 1 -- 172.21.15.28:0/2306920694 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb95404f020 con 0x7fb9480039b0 2024-02-21T23:03:13.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.526+0000 7fb967e00640 1 -- 172.21.15.28:0/2306920694 --> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb96006dd00 con 0x7fb94006b500 2024-02-21T23:03:13.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.552+0000 7fb95d600640 1 -- 172.21.15.28:0/2306920694 <== mgr.4101 v2:172.21.15.81:6832/1848538577 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+332614 (secure 0 0 0) 0x7fb96006dd00 con 0x7fb94006b500 2024-02-21T23:03:13.544 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:03:13.547 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-21T23:03:13.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.559+0000 7fb967e00640 1 -- 172.21.15.28:0/2306920694 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb94006b500 msgr2=0x7fb94006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:13.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.559+0000 7fb967e00640 1 --2- 172.21.15.28:0/2306920694 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb94006b500 0x7fb94006d9c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb94c001eb0 tx=0x7fb94c009040 comp rx=0 tx=0).stop 2024-02-21T23:03:13.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.559+0000 7fb967e00640 1 -- 172.21.15.28:0/2306920694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9480039b0 msgr2=0x7fb96014da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:13.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.559+0000 7fb967e00640 1 --2- 172.21.15.28:0/2306920694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9480039b0 0x7fb96014da10 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb954037040 tx=0x7fb954004730 comp rx=0 tx=0).stop 2024-02-21T23:03:13.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.559+0000 7fb967e00640 1 -- 172.21.15.28:0/2306920694 shutdown_connections 2024-02-21T23:03:13.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.559+0000 7fb967e00640 1 --2- 172.21.15.28:0/2306920694 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb94006b500 0x7fb94006d9c0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:13.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.559+0000 7fb967e00640 1 --2- 172.21.15.28:0/2306920694 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb9601556e0 0x7fb960143ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:13.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.559+0000 7fb967e00640 1 --2- 172.21.15.28:0/2306920694 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9600a7fd0 0x7fb9601440e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:13.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.559+0000 7fb967e00640 1 --2- 172.21.15.28:0/2306920694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9480039b0 0x7fb96014da10 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:13.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.559+0000 7fb967e00640 1 -- 172.21.15.28:0/2306920694 >> 172.21.15.28:0/2306920694 conn(0x7fb9600c67f0 msgr2=0x7fb9600c5230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:13.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.559+0000 7fb967e00640 1 -- 172.21.15.28:0/2306920694 shutdown_connections 2024-02-21T23:03:13.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.559+0000 7fb967e00640 1 -- 172.21.15.28:0/2306920694 wait complete. 2024-02-21T23:03:13.570 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":38,"stamp":"2024-02-21T23:03:12.669735+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":208,"num_write_kb":2435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160,"ondisk_log_size":160,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":222496,"kb_used_data":2176,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754752224,"statfs":{"total":773094113280,"available":772866277376,"internally_reserved":0,"allocated":2228224,"data_stored":1992064,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":225496625},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005195"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":26,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.598140+0000","last_change":"2024-02-21T23:02:37.367904+0000","last_active":"2024-02-21T23:02:43.598140+0000","last_peered":"2024-02-21T23:02:43.598140+0000","last_clean":"2024-02-21T23:02:43.598140+0000","last_became_active":"2024-02-21T23:02:37.367537+0000","last_became_peered":"2024-02-21T23:02:37.367537+0000","last_unstale":"2024-02-21T23:02:43.598140+0000","last_undegraded":"2024-02-21T23:02:43.598140+0000","last_fullsized":"2024-02-21T23:02:43.598140+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:39:34.395881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420749+0000","last_change":"2024-02-21T23:02:38.385115+0000","last_active":"2024-02-21T23:02:43.420749+0000","last_peered":"2024-02-21T23:02:43.420749+0000","last_clean":"2024-02-21T23:02:43.420749+0000","last_became_active":"2024-02-21T23:02:38.384818+0000","last_became_peered":"2024-02-21T23:02:38.384818+0000","last_unstale":"2024-02-21T23:02:43.420749+0000","last_undegraded":"2024-02-21T23:02:43.420749+0000","last_fullsized":"2024-02-21T23:02:43.420749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:54:40.962238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.419345+0000","last_change":"2024-02-21T23:02:42.500553+0000","last_active":"2024-02-21T23:02:43.419345+0000","last_peered":"2024-02-21T23:02:43.419345+0000","last_clean":"2024-02-21T23:02:43.419345+0000","last_became_active":"2024-02-21T23:02:42.498884+0000","last_became_peered":"2024-02-21T23:02:42.498884+0000","last_unstale":"2024-02-21T23:02:43.419345+0000","last_undegraded":"2024-02-21T23:02:43.419345+0000","last_fullsized":"2024-02-21T23:02:43.419345+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:41:42.562121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.421106+0000","last_change":"2024-02-21T23:02:37.354176+0000","last_active":"2024-02-21T23:02:43.421106+0000","last_peered":"2024-02-21T23:02:43.421106+0000","last_clean":"2024-02-21T23:02:43.421106+0000","last_became_active":"2024-02-21T23:02:37.353425+0000","last_became_peered":"2024-02-21T23:02:37.353425+0000","last_unstale":"2024-02-21T23:02:43.421106+0000","last_undegraded":"2024-02-21T23:02:43.421106+0000","last_fullsized":"2024-02-21T23:02:43.421106+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:07:23.801765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.597346+0000","last_change":"2024-02-21T23:02:38.399361+0000","last_active":"2024-02-21T23:02:43.597346+0000","last_peered":"2024-02-21T23:02:43.597346+0000","last_clean":"2024-02-21T23:02:43.597346+0000","last_became_active":"2024-02-21T23:02:38.399026+0000","last_became_peered":"2024-02-21T23:02:38.399026+0000","last_unstale":"2024-02-21T23:02:43.597346+0000","last_undegraded":"2024-02-21T23:02:43.597346+0000","last_fullsized":"2024-02-21T23:02:43.597346+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:29:12.483912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.419463+0000","last_change":"2024-02-21T23:02:42.441232+0000","last_active":"2024-02-21T23:02:43.419463+0000","last_peered":"2024-02-21T23:02:43.419463+0000","last_clean":"2024-02-21T23:02:43.419463+0000","last_became_active":"2024-02-21T23:02:42.440354+0000","last_became_peered":"2024-02-21T23:02:42.440354+0000","last_unstale":"2024-02-21T23:02:43.419463+0000","last_undegraded":"2024-02-21T23:02:43.419463+0000","last_fullsized":"2024-02-21T23:02:43.419463+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:29:16.551658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.598167+0000","last_change":"2024-02-21T23:02:37.373520+0000","last_active":"2024-02-21T23:02:43.598167+0000","last_peered":"2024-02-21T23:02:43.598167+0000","last_clean":"2024-02-21T23:02:43.598167+0000","last_became_active":"2024-02-21T23:02:37.373127+0000","last_became_peered":"2024-02-21T23:02:37.373127+0000","last_unstale":"2024-02-21T23:02:43.598167+0000","last_undegraded":"2024-02-21T23:02:43.598167+0000","last_fullsized":"2024-02-21T23:02:43.598167+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:12:45.623160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.600822+0000","last_change":"2024-02-21T23:02:38.387666+0000","last_active":"2024-02-21T23:02:43.600822+0000","last_peered":"2024-02-21T23:02:43.600822+0000","last_clean":"2024-02-21T23:02:43.600822+0000","last_became_active":"2024-02-21T23:02:38.387151+0000","last_became_peered":"2024-02-21T23:02:38.387151+0000","last_unstale":"2024-02-21T23:02:43.600822+0000","last_undegraded":"2024-02-21T23:02:43.600822+0000","last_fullsized":"2024-02-21T23:02:43.600822+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:24:40.762404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420258+0000","last_change":"2024-02-21T23:02:42.444676+0000","last_active":"2024-02-21T23:02:43.420258+0000","last_peered":"2024-02-21T23:02:43.420258+0000","last_clean":"2024-02-21T23:02:43.420258+0000","last_became_active":"2024-02-21T23:02:42.444185+0000","last_became_peered":"2024-02-21T23:02:42.444185+0000","last_unstale":"2024-02-21T23:02:43.420258+0000","last_undegraded":"2024-02-21T23:02:43.420258+0000","last_fullsized":"2024-02-21T23:02:43.420258+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:54:45.029984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.597339+0000","last_change":"2024-02-21T23:02:37.372881+0000","last_active":"2024-02-21T23:02:43.597339+0000","last_peered":"2024-02-21T23:02:43.597339+0000","last_clean":"2024-02-21T23:02:43.597339+0000","last_became_active":"2024-02-21T23:02:37.372557+0000","last_became_peered":"2024-02-21T23:02:37.372557+0000","last_unstale":"2024-02-21T23:02:43.597339+0000","last_undegraded":"2024-02-21T23:02:43.597339+0000","last_fullsized":"2024-02-21T23:02:43.597339+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:29:33.262208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.600960+0000","last_change":"2024-02-21T23:02:38.399333+0000","last_active":"2024-02-21T23:02:43.600960+0000","last_peered":"2024-02-21T23:02:43.600960+0000","last_clean":"2024-02-21T23:02:43.600960+0000","last_became_active":"2024-02-21T23:02:38.393010+0000","last_became_peered":"2024-02-21T23:02:38.393010+0000","last_unstale":"2024-02-21T23:02:43.600960+0000","last_undegraded":"2024-02-21T23:02:43.600960+0000","last_fullsized":"2024-02-21T23:02:43.600960+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:54:16.651108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.419916+0000","last_change":"2024-02-21T23:02:42.436700+0000","last_active":"2024-02-21T23:02:43.419916+0000","last_peered":"2024-02-21T23:02:43.419916+0000","last_clean":"2024-02-21T23:02:43.419916+0000","last_became_active":"2024-02-21T23:02:42.434264+0000","last_became_peered":"2024-02-21T23:02:42.434264+0000","last_unstale":"2024-02-21T23:02:43.419916+0000","last_undegraded":"2024-02-21T23:02:43.419916+0000","last_fullsized":"2024-02-21T23:02:43.419916+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:28:47.259045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.594139+0000","last_change":"2024-02-21T23:02:38.391008+0000","last_active":"2024-02-21T23:02:43.594139+0000","last_peered":"2024-02-21T23:02:43.594139+0000","last_clean":"2024-02-21T23:02:43.594139+0000","last_became_active":"2024-02-21T23:02:38.390695+0000","last_became_peered":"2024-02-21T23:02:38.390695+0000","last_unstale":"2024-02-21T23:02:43.594139+0000","last_undegraded":"2024-02-21T23:02:43.594139+0000","last_fullsized":"2024-02-21T23:02:43.594139+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:09:34.130536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.598232+0000","last_change":"2024-02-21T23:02:37.373206+0000","last_active":"2024-02-21T23:02:43.598232+0000","last_peered":"2024-02-21T23:02:43.598232+0000","last_clean":"2024-02-21T23:02:43.598232+0000","last_became_active":"2024-02-21T23:02:37.371805+0000","last_became_peered":"2024-02-21T23:02:37.371805+0000","last_unstale":"2024-02-21T23:02:43.598232+0000","last_undegraded":"2024-02-21T23:02:43.598232+0000","last_fullsized":"2024-02-21T23:02:43.598232+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:29:26.321681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420841+0000","last_change":"2024-02-21T23:02:42.435351+0000","last_active":"2024-02-21T23:02:43.420841+0000","last_peered":"2024-02-21T23:02:43.420841+0000","last_clean":"2024-02-21T23:02:43.420841+0000","last_became_active":"2024-02-21T23:02:42.434903+0000","last_became_peered":"2024-02-21T23:02:42.434903+0000","last_unstale":"2024-02-21T23:02:43.420841+0000","last_undegraded":"2024-02-21T23:02:43.420841+0000","last_fullsized":"2024-02-21T23:02:43.420841+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:49:24.584877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420313+0000","last_change":"2024-02-21T23:02:38.369472+0000","last_active":"2024-02-21T23:02:43.420313+0000","last_peered":"2024-02-21T23:02:43.420313+0000","last_clean":"2024-02-21T23:02:43.420313+0000","last_became_active":"2024-02-21T23:02:38.369189+0000","last_became_peered":"2024-02-21T23:02:38.369189+0000","last_unstale":"2024-02-21T23:02:43.420313+0000","last_undegraded":"2024-02-21T23:02:43.420313+0000","last_fullsized":"2024-02-21T23:02:43.420313+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:00:09.172059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.596428+0000","last_change":"2024-02-21T23:02:37.355429+0000","last_active":"2024-02-21T23:02:43.596428+0000","last_peered":"2024-02-21T23:02:43.596428+0000","last_clean":"2024-02-21T23:02:43.596428+0000","last_became_active":"2024-02-21T23:02:37.354573+0000","last_became_peered":"2024-02-21T23:02:37.354573+0000","last_unstale":"2024-02-21T23:02:43.596428+0000","last_undegraded":"2024-02-21T23:02:43.596428+0000","last_fullsized":"2024-02-21T23:02:43.596428+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:17:46.451150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420650+0000","last_change":"2024-02-21T23:02:42.445184+0000","last_active":"2024-02-21T23:02:43.420650+0000","last_peered":"2024-02-21T23:02:43.420650+0000","last_clean":"2024-02-21T23:02:43.420650+0000","last_became_active":"2024-02-21T23:02:42.444773+0000","last_became_peered":"2024-02-21T23:02:42.444773+0000","last_unstale":"2024-02-21T23:02:43.420650+0000","last_undegraded":"2024-02-21T23:02:43.420650+0000","last_fullsized":"2024-02-21T23:02:43.420650+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:24:44.830150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.419669+0000","last_change":"2024-02-21T23:02:38.379076+0000","last_active":"2024-02-21T23:02:43.419669+0000","last_peered":"2024-02-21T23:02:43.419669+0000","last_clean":"2024-02-21T23:02:43.419669+0000","last_became_active":"2024-02-21T23:02:38.378513+0000","last_became_peered":"2024-02-21T23:02:38.378513+0000","last_unstale":"2024-02-21T23:02:43.419669+0000","last_undegraded":"2024-02-21T23:02:43.419669+0000","last_fullsized":"2024-02-21T23:02:43.419669+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:29:00.068426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.595013+0000","last_change":"2024-02-21T23:02:37.359130+0000","last_active":"2024-02-21T23:02:43.595013+0000","last_peered":"2024-02-21T23:02:43.595013+0000","last_clean":"2024-02-21T23:02:43.595013+0000","last_became_active":"2024-02-21T23:02:37.358812+0000","last_became_peered":"2024-02-21T23:02:37.358812+0000","last_unstale":"2024-02-21T23:02:43.595013+0000","last_undegraded":"2024-02-21T23:02:43.595013+0000","last_fullsized":"2024-02-21T23:02:43.595013+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:29:33.262208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.598892+0000","last_change":"2024-02-21T23:02:42.500216+0000","last_active":"2024-02-21T23:02:43.598892+0000","last_peered":"2024-02-21T23:02:43.598892+0000","last_clean":"2024-02-21T23:02:43.598892+0000","last_became_active":"2024-02-21T23:02:42.499762+0000","last_became_peered":"2024-02-21T23:02:42.499762+0000","last_unstale":"2024-02-21T23:02:43.598892+0000","last_undegraded":"2024-02-21T23:02:43.598892+0000","last_fullsized":"2024-02-21T23:02:43.598892+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:12:30.131037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420590+0000","last_change":"2024-02-21T23:02:38.381297+0000","last_active":"2024-02-21T23:02:43.420590+0000","last_peered":"2024-02-21T23:02:43.420590+0000","last_clean":"2024-02-21T23:02:43.420590+0000","last_became_active":"2024-02-21T23:02:38.381015+0000","last_became_peered":"2024-02-21T23:02:38.381015+0000","last_unstale":"2024-02-21T23:02:43.420590+0000","last_undegraded":"2024-02-21T23:02:43.420590+0000","last_fullsized":"2024-02-21T23:02:43.420590+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:29:34.276860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.596339+0000","last_change":"2024-02-21T23:02:37.370201+0000","last_active":"2024-02-21T23:02:43.596339+0000","last_peered":"2024-02-21T23:02:43.596339+0000","last_clean":"2024-02-21T23:02:43.596339+0000","last_became_active":"2024-02-21T23:02:37.369814+0000","last_became_peered":"2024-02-21T23:02:37.369814+0000","last_unstale":"2024-02-21T23:02:43.596339+0000","last_undegraded":"2024-02-21T23:02:43.596339+0000","last_fullsized":"2024-02-21T23:02:43.596339+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:16:38.578056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.421168+0000","last_change":"2024-02-21T23:02:42.439870+0000","last_active":"2024-02-21T23:02:43.421168+0000","last_peered":"2024-02-21T23:02:43.421168+0000","last_clean":"2024-02-21T23:02:43.421168+0000","last_became_active":"2024-02-21T23:02:42.439470+0000","last_became_peered":"2024-02-21T23:02:42.439470+0000","last_unstale":"2024-02-21T23:02:43.421168+0000","last_undegraded":"2024-02-21T23:02:43.421168+0000","last_fullsized":"2024-02-21T23:02:43.421168+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:54:20.718854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.597460+0000","last_change":"2024-02-21T23:02:38.380559+0000","last_active":"2024-02-21T23:02:43.597460+0000","last_peered":"2024-02-21T23:02:43.597460+0000","last_clean":"2024-02-21T23:02:43.597460+0000","last_became_active":"2024-02-21T23:02:38.380229+0000","last_became_peered":"2024-02-21T23:02:38.380229+0000","last_unstale":"2024-02-21T23:02:43.597460+0000","last_undegraded":"2024-02-21T23:02:43.597460+0000","last_fullsized":"2024-02-21T23:02:43.597460+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:09:34.130536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.595919+0000","last_change":"2024-02-21T23:02:37.367812+0000","last_active":"2024-02-21T23:02:43.595919+0000","last_peered":"2024-02-21T23:02:43.595919+0000","last_clean":"2024-02-21T23:02:43.595919+0000","last_became_active":"2024-02-21T23:02:37.367527+0000","last_became_peered":"2024-02-21T23:02:37.367527+0000","last_unstale":"2024-02-21T23:02:43.595919+0000","last_undegraded":"2024-02-21T23:02:43.595919+0000","last_fullsized":"2024-02-21T23:02:43.595919+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:17:46.451150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420213+0000","last_change":"2024-02-21T23:02:42.497817+0000","last_active":"2024-02-21T23:02:43.420213+0000","last_peered":"2024-02-21T23:02:43.420213+0000","last_clean":"2024-02-21T23:02:43.420213+0000","last_became_active":"2024-02-21T23:02:42.497417+0000","last_became_peered":"2024-02-21T23:02:42.497417+0000","last_unstale":"2024-02-21T23:02:43.420213+0000","last_undegraded":"2024-02-21T23:02:43.420213+0000","last_fullsized":"2024-02-21T23:02:43.420213+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:00:13.239804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420747+0000","last_change":"2024-02-21T23:02:38.389615+0000","last_active":"2024-02-21T23:02:43.420747+0000","last_peered":"2024-02-21T23:02:43.420747+0000","last_clean":"2024-02-21T23:02:43.420747+0000","last_became_active":"2024-02-21T23:02:38.389268+0000","last_became_peered":"2024-02-21T23:02:38.389268+0000","last_unstale":"2024-02-21T23:02:43.420747+0000","last_undegraded":"2024-02-21T23:02:43.420747+0000","last_fullsized":"2024-02-21T23:02:43.420747+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:20:23.836303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.598294+0000","last_change":"2024-02-21T23:02:37.356409+0000","last_active":"2024-02-21T23:02:43.598294+0000","last_peered":"2024-02-21T23:02:43.598294+0000","last_clean":"2024-02-21T23:02:43.598294+0000","last_became_active":"2024-02-21T23:02:37.356096+0000","last_became_peered":"2024-02-21T23:02:37.356096+0000","last_unstale":"2024-02-21T23:02:43.598294+0000","last_undegraded":"2024-02-21T23:02:43.598294+0000","last_fullsized":"2024-02-21T23:02:43.598294+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:07:23.801765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.418871+0000","last_change":"2024-02-21T23:02:42.495050+0000","last_active":"2024-02-21T23:02:43.418871+0000","last_peered":"2024-02-21T23:02:43.418871+0000","last_clean":"2024-02-21T23:02:43.418871+0000","last_became_active":"2024-02-21T23:02:42.494469+0000","last_became_peered":"2024-02-21T23:02:42.494469+0000","last_unstale":"2024-02-21T23:02:43.418871+0000","last_undegraded":"2024-02-21T23:02:43.418871+0000","last_fullsized":"2024-02-21T23:02:43.418871+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:00:12.286244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.596815+0000","last_change":"2024-02-21T23:02:38.378708+0000","last_active":"2024-02-21T23:02:43.596815+0000","last_peered":"2024-02-21T23:02:43.596815+0000","last_clean":"2024-02-21T23:02:43.596815+0000","last_became_active":"2024-02-21T23:02:38.378374+0000","last_became_peered":"2024-02-21T23:02:38.378374+0000","last_unstale":"2024-02-21T23:02:43.596815+0000","last_undegraded":"2024-02-21T23:02:43.596815+0000","last_fullsized":"2024-02-21T23:02:43.596815+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:19:25.522975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.598361+0000","last_change":"2024-02-21T23:02:37.372175+0000","last_active":"2024-02-21T23:02:43.598361+0000","last_peered":"2024-02-21T23:02:43.598361+0000","last_clean":"2024-02-21T23:02:43.598361+0000","last_became_active":"2024-02-21T23:02:37.371900+0000","last_became_peered":"2024-02-21T23:02:37.371900+0000","last_unstale":"2024-02-21T23:02:43.598361+0000","last_undegraded":"2024-02-21T23:02:43.598361+0000","last_fullsized":"2024-02-21T23:02:43.598361+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:12:51.484005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.596406+0000","last_change":"2024-02-21T23:02:42.500502+0000","last_active":"2024-02-21T23:02:43.596406+0000","last_peered":"2024-02-21T23:02:43.596406+0000","last_clean":"2024-02-21T23:02:43.596406+0000","last_became_active":"2024-02-21T23:02:42.497658+0000","last_became_peered":"2024-02-21T23:02:42.497658+0000","last_unstale":"2024-02-21T23:02:43.596406+0000","last_undegraded":"2024-02-21T23:02:43.596406+0000","last_fullsized":"2024-02-21T23:02:43.596406+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:54:20.718854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.601212+0000","last_change":"2024-02-21T23:02:38.399632+0000","last_active":"2024-02-21T23:02:43.601212+0000","last_peered":"2024-02-21T23:02:43.601212+0000","last_clean":"2024-02-21T23:02:43.601212+0000","last_became_active":"2024-02-21T23:02:38.392339+0000","last_became_peered":"2024-02-21T23:02:38.392339+0000","last_unstale":"2024-02-21T23:02:43.601212+0000","last_undegraded":"2024-02-21T23:02:43.601212+0000","last_fullsized":"2024-02-21T23:02:43.601212+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:35:27.966492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.596919+0000","last_change":"2024-02-21T23:02:37.347461+0000","last_active":"2024-02-21T23:02:43.596919+0000","last_peered":"2024-02-21T23:02:43.596919+0000","last_clean":"2024-02-21T23:02:43.596919+0000","last_became_active":"2024-02-21T23:02:37.346584+0000","last_became_peered":"2024-02-21T23:02:37.346584+0000","last_unstale":"2024-02-21T23:02:43.596919+0000","last_undegraded":"2024-02-21T23:02:43.596919+0000","last_fullsized":"2024-02-21T23:02:43.596919+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:32:39.769989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.595619+0000","last_change":"2024-02-21T23:02:42.436480+0000","last_active":"2024-02-21T23:02:43.595619+0000","last_peered":"2024-02-21T23:02:43.595619+0000","last_clean":"2024-02-21T23:02:43.595619+0000","last_became_active":"2024-02-21T23:02:42.436035+0000","last_became_peered":"2024-02-21T23:02:42.436035+0000","last_unstale":"2024-02-21T23:02:43.595619+0000","last_undegraded":"2024-02-21T23:02:43.595619+0000","last_fullsized":"2024-02-21T23:02:43.595619+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:54:20.718854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420857+0000","last_change":"2024-02-21T23:02:38.373673+0000","last_active":"2024-02-21T23:02:43.420857+0000","last_peered":"2024-02-21T23:02:43.420857+0000","last_clean":"2024-02-21T23:02:43.420857+0000","last_became_active":"2024-02-21T23:02:38.373324+0000","last_became_peered":"2024-02-21T23:02:38.373324+0000","last_unstale":"2024-02-21T23:02:43.420857+0000","last_undegraded":"2024-02-21T23:02:43.420857+0000","last_fullsized":"2024-02-21T23:02:43.420857+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:09:48.394584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.600679+0000","last_change":"2024-02-21T23:02:37.346663+0000","last_active":"2024-02-21T23:02:43.600679+0000","last_peered":"2024-02-21T23:02:43.600679+0000","last_clean":"2024-02-21T23:02:43.600679+0000","last_became_active":"2024-02-21T23:02:37.344771+0000","last_became_peered":"2024-02-21T23:02:37.344771+0000","last_unstale":"2024-02-21T23:02:43.600679+0000","last_undegraded":"2024-02-21T23:02:43.600679+0000","last_fullsized":"2024-02-21T23:02:43.600679+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:12:51.484005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.596759+0000","last_change":"2024-02-21T23:02:42.440882+0000","last_active":"2024-02-21T23:02:43.596759+0000","last_peered":"2024-02-21T23:02:43.596759+0000","last_clean":"2024-02-21T23:02:43.596759+0000","last_became_active":"2024-02-21T23:02:42.440534+0000","last_became_peered":"2024-02-21T23:02:42.440534+0000","last_unstale":"2024-02-21T23:02:43.596759+0000","last_undegraded":"2024-02-21T23:02:43.596759+0000","last_fullsized":"2024-02-21T23:02:43.596759+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:35:32.034237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.595207+0000","last_change":"2024-02-21T23:02:38.377238+0000","last_active":"2024-02-21T23:02:43.595207+0000","last_peered":"2024-02-21T23:02:43.595207+0000","last_clean":"2024-02-21T23:02:43.595207+0000","last_became_active":"2024-02-21T23:02:38.376272+0000","last_became_peered":"2024-02-21T23:02:38.376272+0000","last_unstale":"2024-02-21T23:02:43.595207+0000","last_undegraded":"2024-02-21T23:02:43.595207+0000","last_fullsized":"2024-02-21T23:02:43.595207+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:10:48.186239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.415223+0000","last_change":"2024-02-21T23:02:37.360945+0000","last_active":"2024-02-21T23:02:43.415223+0000","last_peered":"2024-02-21T23:02:43.415223+0000","last_clean":"2024-02-21T23:02:43.415223+0000","last_became_active":"2024-02-21T23:02:37.359879+0000","last_became_peered":"2024-02-21T23:02:37.359879+0000","last_unstale":"2024-02-21T23:02:43.415223+0000","last_undegraded":"2024-02-21T23:02:43.415223+0000","last_fullsized":"2024-02-21T23:02:43.415223+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:07:23.801765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.598887+0000","last_change":"2024-02-21T23:02:42.441857+0000","last_active":"2024-02-21T23:02:43.598887+0000","last_peered":"2024-02-21T23:02:43.598887+0000","last_clean":"2024-02-21T23:02:43.598887+0000","last_became_active":"2024-02-21T23:02:42.438695+0000","last_became_peered":"2024-02-21T23:02:42.438695+0000","last_unstale":"2024-02-21T23:02:43.598887+0000","last_undegraded":"2024-02-21T23:02:43.598887+0000","last_fullsized":"2024-02-21T23:02:43.598887+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:28:27.820393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.594261+0000","last_change":"2024-02-21T23:02:38.366727+0000","last_active":"2024-02-21T23:02:43.594261+0000","last_peered":"2024-02-21T23:02:43.594261+0000","last_clean":"2024-02-21T23:02:43.594261+0000","last_became_active":"2024-02-21T23:02:38.366344+0000","last_became_peered":"2024-02-21T23:02:38.366344+0000","last_unstale":"2024-02-21T23:02:43.594261+0000","last_undegraded":"2024-02-21T23:02:43.594261+0000","last_fullsized":"2024-02-21T23:02:43.594261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:08:44.843392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.596392+0000","last_change":"2024-02-21T23:02:37.372506+0000","last_active":"2024-02-21T23:02:43.596392+0000","last_peered":"2024-02-21T23:02:43.596392+0000","last_clean":"2024-02-21T23:02:43.596392+0000","last_became_active":"2024-02-21T23:02:37.371806+0000","last_became_peered":"2024-02-21T23:02:37.371806+0000","last_unstale":"2024-02-21T23:02:43.596392+0000","last_undegraded":"2024-02-21T23:02:43.596392+0000","last_fullsized":"2024-02-21T23:02:43.596392+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:09:47.379932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.597397+0000","last_change":"2024-02-21T23:02:42.500339+0000","last_active":"2024-02-21T23:02:43.597397+0000","last_peered":"2024-02-21T23:02:43.597397+0000","last_clean":"2024-02-21T23:02:43.597397+0000","last_became_active":"2024-02-21T23:02:42.498471+0000","last_became_peered":"2024-02-21T23:02:42.498471+0000","last_unstale":"2024-02-21T23:02:43.597397+0000","last_undegraded":"2024-02-21T23:02:43.597397+0000","last_fullsized":"2024-02-21T23:02:43.597397+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:12:30.131037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.419210+0000","last_change":"2024-02-21T23:02:38.400850+0000","last_active":"2024-02-21T23:02:43.419210+0000","last_peered":"2024-02-21T23:02:43.419210+0000","last_clean":"2024-02-21T23:02:43.419210+0000","last_became_active":"2024-02-21T23:02:38.400367+0000","last_became_peered":"2024-02-21T23:02:38.400367+0000","last_unstale":"2024-02-21T23:02:43.419210+0000","last_undegraded":"2024-02-21T23:02:43.419210+0000","last_fullsized":"2024-02-21T23:02:43.419210+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:29:12.483912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.600646+0000","last_change":"2024-02-21T23:02:37.349152+0000","last_active":"2024-02-21T23:02:43.600646+0000","last_peered":"2024-02-21T23:02:43.600646+0000","last_clean":"2024-02-21T23:02:43.600646+0000","last_became_active":"2024-02-21T23:02:37.348647+0000","last_became_peered":"2024-02-21T23:02:37.348647+0000","last_unstale":"2024-02-21T23:02:43.600646+0000","last_undegraded":"2024-02-21T23:02:43.600646+0000","last_fullsized":"2024-02-21T23:02:43.600646+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:17:46.451150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":178,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.596598+0000","last_change":"2024-02-21T23:02:32.289335+0000","last_active":"2024-02-21T23:02:43.596598+0000","last_peered":"2024-02-21T23:02:43.596598+0000","last_clean":"2024-02-21T23:02:43.596598+0000","last_became_active":"2024-02-21T23:02:32.286913+0000","last_became_peered":"2024-02-21T23:02:32.286913+0000","last_unstale":"2024-02-21T23:02:43.596598+0000","last_undegraded":"2024-02-21T23:02:43.596598+0000","last_fullsized":"2024-02-21T23:02:43.596598+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_clean_scrub_stamp":"2024-02-21T23:02:31.172569+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:36:23.987374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.594895+0000","last_change":"2024-02-21T23:02:42.498307+0000","last_active":"2024-02-21T23:02:43.594895+0000","last_peered":"2024-02-21T23:02:43.594895+0000","last_clean":"2024-02-21T23:02:43.594895+0000","last_became_active":"2024-02-21T23:02:42.497846+0000","last_became_peered":"2024-02-21T23:02:42.497846+0000","last_unstale":"2024-02-21T23:02:43.594895+0000","last_undegraded":"2024-02-21T23:02:43.594895+0000","last_fullsized":"2024-02-21T23:02:43.594895+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:24:44.830150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.597925+0000","last_change":"2024-02-21T23:02:38.401025+0000","last_active":"2024-02-21T23:02:43.597925+0000","last_peered":"2024-02-21T23:02:43.597925+0000","last_clean":"2024-02-21T23:02:43.597925+0000","last_became_active":"2024-02-21T23:02:38.400639+0000","last_became_peered":"2024-02-21T23:02:38.400639+0000","last_unstale":"2024-02-21T23:02:43.597925+0000","last_undegraded":"2024-02-21T23:02:43.597925+0000","last_fullsized":"2024-02-21T23:02:43.597925+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:49:20.517131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.416472+0000","last_change":"2024-02-21T23:02:37.360670+0000","last_active":"2024-02-21T23:02:43.416472+0000","last_peered":"2024-02-21T23:02:43.416472+0000","last_clean":"2024-02-21T23:02:43.416472+0000","last_became_active":"2024-02-21T23:02:37.358188+0000","last_became_peered":"2024-02-21T23:02:37.358188+0000","last_unstale":"2024-02-21T23:02:43.416472+0000","last_undegraded":"2024-02-21T23:02:43.416472+0000","last_fullsized":"2024-02-21T23:02:43.416472+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:17:46.451150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.595571+0000","last_change":"2024-02-21T23:02:42.495923+0000","last_active":"2024-02-21T23:02:43.595571+0000","last_peered":"2024-02-21T23:02:43.595571+0000","last_clean":"2024-02-21T23:02:43.595571+0000","last_became_active":"2024-02-21T23:02:42.495507+0000","last_became_peered":"2024-02-21T23:02:42.495507+0000","last_unstale":"2024-02-21T23:02:43.595571+0000","last_undegraded":"2024-02-21T23:02:43.595571+0000","last_fullsized":"2024-02-21T23:02:43.595571+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:54:20.718854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.600858+0000","last_change":"2024-02-21T23:02:38.386757+0000","last_active":"2024-02-21T23:02:43.600858+0000","last_peered":"2024-02-21T23:02:43.600858+0000","last_clean":"2024-02-21T23:02:43.600858+0000","last_became_active":"2024-02-21T23:02:38.384691+0000","last_became_peered":"2024-02-21T23:02:38.384691+0000","last_unstale":"2024-02-21T23:02:43.600858+0000","last_undegraded":"2024-02-21T23:02:43.600858+0000","last_fullsized":"2024-02-21T23:02:43.600858+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:49:20.517131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.597791+0000","last_change":"2024-02-21T23:02:37.356479+0000","last_active":"2024-02-21T23:02:43.597791+0000","last_peered":"2024-02-21T23:02:43.597791+0000","last_clean":"2024-02-21T23:02:43.597791+0000","last_became_active":"2024-02-21T23:02:37.356132+0000","last_became_peered":"2024-02-21T23:02:37.356132+0000","last_unstale":"2024-02-21T23:02:43.597791+0000","last_undegraded":"2024-02-21T23:02:43.597791+0000","last_fullsized":"2024-02-21T23:02:43.597791+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:16:38.578056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.597135+0000","last_change":"2024-02-21T23:02:42.440448+0000","last_active":"2024-02-21T23:02:43.597135+0000","last_peered":"2024-02-21T23:02:43.597135+0000","last_clean":"2024-02-21T23:02:43.597135+0000","last_became_active":"2024-02-21T23:02:42.439847+0000","last_became_peered":"2024-02-21T23:02:42.439847+0000","last_unstale":"2024-02-21T23:02:43.597135+0000","last_undegraded":"2024-02-21T23:02:43.597135+0000","last_fullsized":"2024-02-21T23:02:43.597135+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:28:27.820393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.598913+0000","last_change":"2024-02-21T23:02:38.401714+0000","last_active":"2024-02-21T23:02:43.598913+0000","last_peered":"2024-02-21T23:02:43.598913+0000","last_clean":"2024-02-21T23:02:43.598913+0000","last_became_active":"2024-02-21T23:02:38.401166+0000","last_became_peered":"2024-02-21T23:02:38.401166+0000","last_unstale":"2024-02-21T23:02:43.598913+0000","last_undegraded":"2024-02-21T23:02:43.598913+0000","last_fullsized":"2024-02-21T23:02:43.598913+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:33:11.255331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.416538+0000","last_change":"2024-02-21T23:02:37.356667+0000","last_active":"2024-02-21T23:02:43.416538+0000","last_peered":"2024-02-21T23:02:43.416538+0000","last_clean":"2024-02-21T23:02:43.416538+0000","last_became_active":"2024-02-21T23:02:37.354832+0000","last_became_peered":"2024-02-21T23:02:37.354832+0000","last_unstale":"2024-02-21T23:02:43.416538+0000","last_undegraded":"2024-02-21T23:02:43.416538+0000","last_fullsized":"2024-02-21T23:02:43.416538+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:01:17.207079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.599759+0000","last_change":"2024-02-21T23:02:42.494816+0000","last_active":"2024-02-21T23:02:43.599759+0000","last_peered":"2024-02-21T23:02:43.599759+0000","last_clean":"2024-02-21T23:02:43.599759+0000","last_became_active":"2024-02-21T23:02:42.494397+0000","last_became_peered":"2024-02-21T23:02:42.494397+0000","last_unstale":"2024-02-21T23:02:43.599759+0000","last_undegraded":"2024-02-21T23:02:43.599759+0000","last_fullsized":"2024-02-21T23:02:43.599759+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:35:47.803796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.597807+0000","last_change":"2024-02-21T23:02:38.402995+0000","last_active":"2024-02-21T23:02:43.597807+0000","last_peered":"2024-02-21T23:02:43.597807+0000","last_clean":"2024-02-21T23:02:43.597807+0000","last_became_active":"2024-02-21T23:02:38.400263+0000","last_became_peered":"2024-02-21T23:02:38.400263+0000","last_unstale":"2024-02-21T23:02:43.597807+0000","last_undegraded":"2024-02-21T23:02:43.597807+0000","last_fullsized":"2024-02-21T23:02:43.597807+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:28:23.752647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.421321+0000","last_change":"2024-02-21T23:02:37.348962+0000","last_active":"2024-02-21T23:02:43.421321+0000","last_peered":"2024-02-21T23:02:43.421321+0000","last_clean":"2024-02-21T23:02:43.421321+0000","last_became_active":"2024-02-21T23:02:37.348550+0000","last_became_peered":"2024-02-21T23:02:37.348550+0000","last_unstale":"2024-02-21T23:02:43.421321+0000","last_undegraded":"2024-02-21T23:02:43.421321+0000","last_fullsized":"2024-02-21T23:02:43.421321+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:20:15.139033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.419867+0000","last_change":"2024-02-21T23:02:42.442477+0000","last_active":"2024-02-21T23:02:43.419867+0000","last_peered":"2024-02-21T23:02:43.419867+0000","last_clean":"2024-02-21T23:02:43.419867+0000","last_became_active":"2024-02-21T23:02:42.442091+0000","last_became_peered":"2024-02-21T23:02:42.442091+0000","last_unstale":"2024-02-21T23:02:43.419867+0000","last_undegraded":"2024-02-21T23:02:43.419867+0000","last_fullsized":"2024-02-21T23:02:43.419867+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:09:38.198282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.596717+0000","last_change":"2024-02-21T23:02:38.401181+0000","last_active":"2024-02-21T23:02:43.596717+0000","last_peered":"2024-02-21T23:02:43.596717+0000","last_clean":"2024-02-21T23:02:43.596717+0000","last_became_active":"2024-02-21T23:02:38.400463+0000","last_became_peered":"2024-02-21T23:02:38.400463+0000","last_unstale":"2024-02-21T23:02:43.596717+0000","last_undegraded":"2024-02-21T23:02:43.596717+0000","last_fullsized":"2024-02-21T23:02:43.596717+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:05:40.265121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.414478+0000","last_change":"2024-02-21T23:02:37.348047+0000","last_active":"2024-02-21T23:02:43.414478+0000","last_peered":"2024-02-21T23:02:43.414478+0000","last_clean":"2024-02-21T23:02:43.414478+0000","last_became_active":"2024-02-21T23:02:37.347664+0000","last_became_peered":"2024-02-21T23:02:37.347664+0000","last_unstale":"2024-02-21T23:02:43.414478+0000","last_undegraded":"2024-02-21T23:02:43.414478+0000","last_fullsized":"2024-02-21T23:02:43.414478+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:39:34.395881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.596934+0000","last_change":"2024-02-21T23:02:42.447743+0000","last_active":"2024-02-21T23:02:43.596934+0000","last_peered":"2024-02-21T23:02:43.596934+0000","last_clean":"2024-02-21T23:02:43.596934+0000","last_became_active":"2024-02-21T23:02:42.447181+0000","last_became_peered":"2024-02-21T23:02:42.447181+0000","last_unstale":"2024-02-21T23:02:43.596934+0000","last_undegraded":"2024-02-21T23:02:43.596934+0000","last_fullsized":"2024-02-21T23:02:43.596934+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:07:03.424869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.594280+0000","last_change":"2024-02-21T23:02:38.375362+0000","last_active":"2024-02-21T23:02:43.594280+0000","last_peered":"2024-02-21T23:02:43.594280+0000","last_clean":"2024-02-21T23:02:43.594280+0000","last_became_active":"2024-02-21T23:02:38.375081+0000","last_became_peered":"2024-02-21T23:02:38.375081+0000","last_unstale":"2024-02-21T23:02:43.594280+0000","last_undegraded":"2024-02-21T23:02:43.594280+0000","last_fullsized":"2024-02-21T23:02:43.594280+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:19:25.522975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.595646+0000","last_change":"2024-02-21T23:02:37.359690+0000","last_active":"2024-02-21T23:02:43.595646+0000","last_peered":"2024-02-21T23:02:43.595646+0000","last_clean":"2024-02-21T23:02:43.595646+0000","last_became_active":"2024-02-21T23:02:37.359396+0000","last_became_peered":"2024-02-21T23:02:37.359396+0000","last_unstale":"2024-02-21T23:02:43.595646+0000","last_undegraded":"2024-02-21T23:02:43.595646+0000","last_fullsized":"2024-02-21T23:02:43.595646+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:29:33.262208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420901+0000","last_change":"2024-02-21T23:02:42.499203+0000","last_active":"2024-02-21T23:02:43.420901+0000","last_peered":"2024-02-21T23:02:43.420901+0000","last_clean":"2024-02-21T23:02:43.420901+0000","last_became_active":"2024-02-21T23:02:42.498781+0000","last_became_peered":"2024-02-21T23:02:42.498781+0000","last_unstale":"2024-02-21T23:02:43.420901+0000","last_undegraded":"2024-02-21T23:02:43.420901+0000","last_fullsized":"2024-02-21T23:02:43.420901+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:08:32.584721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.598100+0000","last_change":"2024-02-21T23:02:38.402276+0000","last_active":"2024-02-21T23:02:43.598100+0000","last_peered":"2024-02-21T23:02:43.598100+0000","last_clean":"2024-02-21T23:02:43.598100+0000","last_became_active":"2024-02-21T23:02:38.401976+0000","last_became_peered":"2024-02-21T23:02:38.401976+0000","last_unstale":"2024-02-21T23:02:43.598100+0000","last_undegraded":"2024-02-21T23:02:43.598100+0000","last_fullsized":"2024-02-21T23:02:43.598100+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:12:26.063291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.600414+0000","last_change":"2024-02-21T23:02:37.359735+0000","last_active":"2024-02-21T23:02:43.600414+0000","last_peered":"2024-02-21T23:02:43.600414+0000","last_clean":"2024-02-21T23:02:43.600414+0000","last_became_active":"2024-02-21T23:02:37.359447+0000","last_became_peered":"2024-02-21T23:02:37.359447+0000","last_unstale":"2024-02-21T23:02:43.600414+0000","last_undegraded":"2024-02-21T23:02:43.600414+0000","last_fullsized":"2024-02-21T23:02:43.600414+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:29:26.321681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.421314+0000","last_change":"2024-02-21T23:02:42.494896+0000","last_active":"2024-02-21T23:02:43.421314+0000","last_peered":"2024-02-21T23:02:43.421314+0000","last_clean":"2024-02-21T23:02:43.421314+0000","last_became_active":"2024-02-21T23:02:42.494433+0000","last_became_peered":"2024-02-21T23:02:42.494433+0000","last_unstale":"2024-02-21T23:02:43.421314+0000","last_undegraded":"2024-02-21T23:02:43.421314+0000","last_fullsized":"2024-02-21T23:02:43.421314+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:53:14.069591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.595745+0000","last_change":"2024-02-21T23:02:38.389883+0000","last_active":"2024-02-21T23:02:43.595745+0000","last_peered":"2024-02-21T23:02:43.595745+0000","last_clean":"2024-02-21T23:02:43.595745+0000","last_became_active":"2024-02-21T23:02:38.389132+0000","last_became_peered":"2024-02-21T23:02:38.389132+0000","last_unstale":"2024-02-21T23:02:43.595745+0000","last_undegraded":"2024-02-21T23:02:43.595745+0000","last_fullsized":"2024-02-21T23:02:43.595745+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:49:26.724162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.597566+0000","last_change":"2024-02-21T23:02:37.354560+0000","last_active":"2024-02-21T23:02:43.597566+0000","last_peered":"2024-02-21T23:02:43.597566+0000","last_clean":"2024-02-21T23:02:43.597566+0000","last_became_active":"2024-02-21T23:02:37.354266+0000","last_became_peered":"2024-02-21T23:02:37.354266+0000","last_unstale":"2024-02-21T23:02:43.597566+0000","last_undegraded":"2024-02-21T23:02:43.597566+0000","last_fullsized":"2024-02-21T23:02:43.597566+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:36:43.400580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.601188+0000","last_change":"2024-02-21T23:02:42.496211+0000","last_active":"2024-02-21T23:02:43.601188+0000","last_peered":"2024-02-21T23:02:43.601188+0000","last_clean":"2024-02-21T23:02:43.601188+0000","last_became_active":"2024-02-21T23:02:42.495493+0000","last_became_peered":"2024-02-21T23:02:42.495493+0000","last_unstale":"2024-02-21T23:02:43.601188+0000","last_undegraded":"2024-02-21T23:02:43.601188+0000","last_fullsized":"2024-02-21T23:02:43.601188+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:00:12.286244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.597793+0000","last_change":"2024-02-21T23:02:38.395172+0000","last_active":"2024-02-21T23:02:43.597793+0000","last_peered":"2024-02-21T23:02:43.597793+0000","last_clean":"2024-02-21T23:02:43.597793+0000","last_became_active":"2024-02-21T23:02:38.394809+0000","last_became_peered":"2024-02-21T23:02:38.394809+0000","last_unstale":"2024-02-21T23:02:43.597793+0000","last_undegraded":"2024-02-21T23:02:43.597793+0000","last_fullsized":"2024-02-21T23:02:43.597793+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:20:23.836303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"24'10","reported_seq":32,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:57.662420+0000","last_change":"2024-02-21T23:02:37.362539+0000","last_active":"2024-02-21T23:02:57.662420+0000","last_peered":"2024-02-21T23:02:57.662420+0000","last_clean":"2024-02-21T23:02:57.662420+0000","last_became_active":"2024-02-21T23:02:37.362186+0000","last_became_peered":"2024-02-21T23:02:37.362186+0000","last_unstale":"2024-02-21T23:02:57.662420+0000","last_undegraded":"2024-02-21T23:02:57.662420+0000","last_fullsized":"2024-02-21T23:02:57.662420+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:32:39.769989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.597415+0000","last_change":"2024-02-21T23:02:42.433330+0000","last_active":"2024-02-21T23:02:43.597415+0000","last_peered":"2024-02-21T23:02:43.597415+0000","last_clean":"2024-02-21T23:02:43.597415+0000","last_became_active":"2024-02-21T23:02:42.432944+0000","last_became_peered":"2024-02-21T23:02:42.432944+0000","last_unstale":"2024-02-21T23:02:43.597415+0000","last_undegraded":"2024-02-21T23:02:43.597415+0000","last_fullsized":"2024-02-21T23:02:43.597415+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:07:03.424869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420973+0000","last_change":"2024-02-21T23:02:38.389764+0000","last_active":"2024-02-21T23:02:43.420973+0000","last_peered":"2024-02-21T23:02:43.420973+0000","last_clean":"2024-02-21T23:02:43.420973+0000","last_became_active":"2024-02-21T23:02:38.389176+0000","last_became_peered":"2024-02-21T23:02:38.389176+0000","last_unstale":"2024-02-21T23:02:43.420973+0000","last_undegraded":"2024-02-21T23:02:43.420973+0000","last_fullsized":"2024-02-21T23:02:43.420973+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:12:46.637812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.597677+0000","last_change":"2024-02-21T23:02:37.344126+0000","last_active":"2024-02-21T23:02:43.597677+0000","last_peered":"2024-02-21T23:02:43.597677+0000","last_clean":"2024-02-21T23:02:43.597677+0000","last_became_active":"2024-02-21T23:02:37.343769+0000","last_became_peered":"2024-02-21T23:02:37.343769+0000","last_unstale":"2024-02-21T23:02:43.597677+0000","last_undegraded":"2024-02-21T23:02:43.597677+0000","last_fullsized":"2024-02-21T23:02:43.597677+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:09:47.379932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420591+0000","last_change":"2024-02-21T23:02:42.440359+0000","last_active":"2024-02-21T23:02:43.420591+0000","last_peered":"2024-02-21T23:02:43.420591+0000","last_clean":"2024-02-21T23:02:43.420591+0000","last_became_active":"2024-02-21T23:02:42.439967+0000","last_became_peered":"2024-02-21T23:02:42.439967+0000","last_unstale":"2024-02-21T23:02:43.420591+0000","last_undegraded":"2024-02-21T23:02:43.420591+0000","last_fullsized":"2024-02-21T23:02:43.420591+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:20:27.904049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.597926+0000","last_change":"2024-02-21T23:02:38.393522+0000","last_active":"2024-02-21T23:02:43.597926+0000","last_peered":"2024-02-21T23:02:43.597926+0000","last_clean":"2024-02-21T23:02:43.597926+0000","last_became_active":"2024-02-21T23:02:38.392890+0000","last_became_peered":"2024-02-21T23:02:38.392890+0000","last_unstale":"2024-02-21T23:02:43.597926+0000","last_undegraded":"2024-02-21T23:02:43.597926+0000","last_fullsized":"2024-02-21T23:02:43.597926+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:30:37.791426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.601175+0000","last_change":"2024-02-21T23:02:37.361082+0000","last_active":"2024-02-21T23:02:43.601175+0000","last_peered":"2024-02-21T23:02:43.601175+0000","last_clean":"2024-02-21T23:02:43.601175+0000","last_became_active":"2024-02-21T23:02:37.360751+0000","last_became_peered":"2024-02-21T23:02:37.360751+0000","last_unstale":"2024-02-21T23:02:43.601175+0000","last_undegraded":"2024-02-21T23:02:43.601175+0000","last_fullsized":"2024-02-21T23:02:43.601175+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:09:47.379932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.419487+0000","last_change":"2024-02-21T23:02:42.440260+0000","last_active":"2024-02-21T23:02:43.419487+0000","last_peered":"2024-02-21T23:02:43.419487+0000","last_clean":"2024-02-21T23:02:43.419487+0000","last_became_active":"2024-02-21T23:02:42.439785+0000","last_became_peered":"2024-02-21T23:02:42.439785+0000","last_unstale":"2024-02-21T23:02:43.419487+0000","last_undegraded":"2024-02-21T23:02:43.419487+0000","last_fullsized":"2024-02-21T23:02:43.419487+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:42:58.884647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.600988+0000","last_change":"2024-02-21T23:02:38.400013+0000","last_active":"2024-02-21T23:02:43.600988+0000","last_peered":"2024-02-21T23:02:43.600988+0000","last_clean":"2024-02-21T23:02:43.600988+0000","last_became_active":"2024-02-21T23:02:38.399708+0000","last_became_peered":"2024-02-21T23:02:38.399708+0000","last_unstale":"2024-02-21T23:02:43.600988+0000","last_undegraded":"2024-02-21T23:02:43.600988+0000","last_fullsized":"2024-02-21T23:02:43.600988+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:06:59.357124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.595798+0000","last_change":"2024-02-21T23:02:37.366264+0000","last_active":"2024-02-21T23:02:43.595798+0000","last_peered":"2024-02-21T23:02:43.595798+0000","last_clean":"2024-02-21T23:02:43.595798+0000","last_became_active":"2024-02-21T23:02:37.365889+0000","last_became_peered":"2024-02-21T23:02:37.365889+0000","last_unstale":"2024-02-21T23:02:43.595798+0000","last_undegraded":"2024-02-21T23:02:43.595798+0000","last_fullsized":"2024-02-21T23:02:43.595798+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:07:23.801765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.419552+0000","last_change":"2024-02-21T23:02:42.495489+0000","last_active":"2024-02-21T23:02:43.419552+0000","last_peered":"2024-02-21T23:02:43.419552+0000","last_clean":"2024-02-21T23:02:43.419552+0000","last_became_active":"2024-02-21T23:02:42.495098+0000","last_became_peered":"2024-02-21T23:02:42.495098+0000","last_unstale":"2024-02-21T23:02:43.419552+0000","last_undegraded":"2024-02-21T23:02:43.419552+0000","last_fullsized":"2024-02-21T23:02:43.419552+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:10:21.610343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.596467+0000","last_change":"2024-02-21T23:02:42.500336+0000","last_active":"2024-02-21T23:02:43.596467+0000","last_peered":"2024-02-21T23:02:43.596467+0000","last_clean":"2024-02-21T23:02:43.596467+0000","last_became_active":"2024-02-21T23:02:42.498817+0000","last_became_peered":"2024-02-21T23:02:42.498817+0000","last_unstale":"2024-02-21T23:02:43.596467+0000","last_undegraded":"2024-02-21T23:02:43.596467+0000","last_fullsized":"2024-02-21T23:02:43.596467+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:35:32.034237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.594893+0000","last_change":"2024-02-21T23:02:38.378745+0000","last_active":"2024-02-21T23:02:43.594893+0000","last_peered":"2024-02-21T23:02:43.594893+0000","last_clean":"2024-02-21T23:02:43.594893+0000","last_became_active":"2024-02-21T23:02:38.378423+0000","last_became_peered":"2024-02-21T23:02:38.378423+0000","last_unstale":"2024-02-21T23:02:43.594893+0000","last_undegraded":"2024-02-21T23:02:43.594893+0000","last_fullsized":"2024-02-21T23:02:43.594893+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:30:37.791426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.600043+0000","last_change":"2024-02-21T23:02:37.352231+0000","last_active":"2024-02-21T23:02:43.600043+0000","last_peered":"2024-02-21T23:02:43.600043+0000","last_clean":"2024-02-21T23:02:43.600043+0000","last_became_active":"2024-02-21T23:02:37.351715+0000","last_became_peered":"2024-02-21T23:02:37.351715+0000","last_unstale":"2024-02-21T23:02:43.600043+0000","last_undegraded":"2024-02-21T23:02:43.600043+0000","last_fullsized":"2024-02-21T23:02:43.600043+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:07:23.801765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.595272+0000","last_change":"2024-02-21T23:02:42.449848+0000","last_active":"2024-02-21T23:02:43.595272+0000","last_peered":"2024-02-21T23:02:43.595272+0000","last_clean":"2024-02-21T23:02:43.595272+0000","last_became_active":"2024-02-21T23:02:42.449357+0000","last_became_peered":"2024-02-21T23:02:42.449357+0000","last_unstale":"2024-02-21T23:02:43.595272+0000","last_undegraded":"2024-02-21T23:02:43.595272+0000","last_fullsized":"2024-02-21T23:02:43.595272+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:49:24.584877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.419241+0000","last_change":"2024-02-21T23:02:38.368240+0000","last_active":"2024-02-21T23:02:43.419241+0000","last_peered":"2024-02-21T23:02:43.419241+0000","last_clean":"2024-02-21T23:02:43.419241+0000","last_became_active":"2024-02-21T23:02:38.367852+0000","last_became_peered":"2024-02-21T23:02:38.367852+0000","last_unstale":"2024-02-21T23:02:43.419241+0000","last_undegraded":"2024-02-21T23:02:43.419241+0000","last_fullsized":"2024-02-21T23:02:43.419241+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:30:37.791426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420051+0000","last_change":"2024-02-21T23:02:37.363129+0000","last_active":"2024-02-21T23:02:43.420051+0000","last_peered":"2024-02-21T23:02:43.420051+0000","last_clean":"2024-02-21T23:02:43.420051+0000","last_became_active":"2024-02-21T23:02:37.362829+0000","last_became_peered":"2024-02-21T23:02:37.362829+0000","last_unstale":"2024-02-21T23:02:43.420051+0000","last_undegraded":"2024-02-21T23:02:43.420051+0000","last_fullsized":"2024-02-21T23:02:43.420051+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:39:34.395881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.594502+0000","last_change":"2024-02-21T23:02:37.358568+0000","last_active":"2024-02-21T23:02:43.594502+0000","last_peered":"2024-02-21T23:02:43.594502+0000","last_clean":"2024-02-21T23:02:43.594502+0000","last_became_active":"2024-02-21T23:02:37.358238+0000","last_became_peered":"2024-02-21T23:02:37.358238+0000","last_unstale":"2024-02-21T23:02:43.594502+0000","last_undegraded":"2024-02-21T23:02:43.594502+0000","last_fullsized":"2024-02-21T23:02:43.594502+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:01:17.207079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.597289+0000","last_change":"2024-02-21T23:02:38.375982+0000","last_active":"2024-02-21T23:02:43.597289+0000","last_peered":"2024-02-21T23:02:43.597289+0000","last_clean":"2024-02-21T23:02:43.597289+0000","last_became_active":"2024-02-21T23:02:38.375504+0000","last_became_peered":"2024-02-21T23:02:38.375504+0000","last_unstale":"2024-02-21T23:02:43.597289+0000","last_undegraded":"2024-02-21T23:02:43.597289+0000","last_fullsized":"2024-02-21T23:02:43.597289+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:19:25.522975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.416324+0000","last_change":"2024-02-21T23:02:42.441484+0000","last_active":"2024-02-21T23:02:43.416324+0000","last_peered":"2024-02-21T23:02:43.416324+0000","last_clean":"2024-02-21T23:02:43.416324+0000","last_became_active":"2024-02-21T23:02:42.438094+0000","last_became_peered":"2024-02-21T23:02:42.438094+0000","last_unstale":"2024-02-21T23:02:43.416324+0000","last_undegraded":"2024-02-21T23:02:43.416324+0000","last_fullsized":"2024-02-21T23:02:43.416324+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:04:15.959906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420559+0000","last_change":"2024-02-21T23:02:37.365019+0000","last_active":"2024-02-21T23:02:43.420559+0000","last_peered":"2024-02-21T23:02:43.420559+0000","last_clean":"2024-02-21T23:02:43.420559+0000","last_became_active":"2024-02-21T23:02:37.364761+0000","last_became_peered":"2024-02-21T23:02:37.364761+0000","last_unstale":"2024-02-21T23:02:43.420559+0000","last_undegraded":"2024-02-21T23:02:43.420559+0000","last_fullsized":"2024-02-21T23:02:43.420559+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:01:17.207079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420052+0000","last_change":"2024-02-21T23:02:38.385160+0000","last_active":"2024-02-21T23:02:43.420052+0000","last_peered":"2024-02-21T23:02:43.420052+0000","last_clean":"2024-02-21T23:02:43.420052+0000","last_became_active":"2024-02-21T23:02:38.382830+0000","last_became_peered":"2024-02-21T23:02:38.382830+0000","last_unstale":"2024-02-21T23:02:43.420052+0000","last_undegraded":"2024-02-21T23:02:43.420052+0000","last_fullsized":"2024-02-21T23:02:43.420052+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:16:39.592708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.421894+0000","last_change":"2024-02-21T23:02:42.437091+0000","last_active":"2024-02-21T23:02:43.421894+0000","last_peered":"2024-02-21T23:02:43.421894+0000","last_clean":"2024-02-21T23:02:43.421894+0000","last_became_active":"2024-02-21T23:02:42.436674+0000","last_became_peered":"2024-02-21T23:02:42.436674+0000","last_unstale":"2024-02-21T23:02:43.421894+0000","last_undegraded":"2024-02-21T23:02:43.421894+0000","last_fullsized":"2024-02-21T23:02:43.421894+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:16:29.140022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.594049+0000","last_change":"2024-02-21T23:02:38.379817+0000","last_active":"2024-02-21T23:02:43.594049+0000","last_peered":"2024-02-21T23:02:43.594049+0000","last_clean":"2024-02-21T23:02:43.594049+0000","last_became_active":"2024-02-21T23:02:38.379367+0000","last_became_peered":"2024-02-21T23:02:38.379367+0000","last_unstale":"2024-02-21T23:02:43.594049+0000","last_undegraded":"2024-02-21T23:02:43.594049+0000","last_fullsized":"2024-02-21T23:02:43.594049+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:04:11.892160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.415084+0000","last_change":"2024-02-21T23:02:38.386264+0000","last_active":"2024-02-21T23:02:43.415084+0000","last_peered":"2024-02-21T23:02:43.415084+0000","last_clean":"2024-02-21T23:02:43.415084+0000","last_became_active":"2024-02-21T23:02:38.385976+0000","last_became_peered":"2024-02-21T23:02:38.385976+0000","last_unstale":"2024-02-21T23:02:43.415084+0000","last_undegraded":"2024-02-21T23:02:43.415084+0000","last_fullsized":"2024-02-21T23:02:43.415084+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:09:48.394584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.600386+0000","last_change":"2024-02-21T23:02:38.394227+0000","last_active":"2024-02-21T23:02:43.600386+0000","last_peered":"2024-02-21T23:02:43.600386+0000","last_clean":"2024-02-21T23:02:43.600386+0000","last_became_active":"2024-02-21T23:02:38.393965+0000","last_became_peered":"2024-02-21T23:02:38.393965+0000","last_unstale":"2024-02-21T23:02:43.600386+0000","last_undegraded":"2024-02-21T23:02:43.600386+0000","last_fullsized":"2024-02-21T23:02:43.600386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:20:23.836303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.596804+0000","last_change":"2024-02-21T23:02:38.365717+0000","last_active":"2024-02-21T23:02:43.596804+0000","last_peered":"2024-02-21T23:02:43.596804+0000","last_clean":"2024-02-21T23:02:43.596804+0000","last_became_active":"2024-02-21T23:02:38.365375+0000","last_became_peered":"2024-02-21T23:02:38.365375+0000","last_unstale":"2024-02-21T23:02:43.596804+0000","last_undegraded":"2024-02-21T23:02:43.596804+0000","last_fullsized":"2024-02-21T23:02:43.596804+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:29:00.068426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.421488+0000","last_change":"2024-02-21T23:02:38.376971+0000","last_active":"2024-02-21T23:02:43.421488+0000","last_peered":"2024-02-21T23:02:43.421488+0000","last_clean":"2024-02-21T23:02:43.421488+0000","last_became_active":"2024-02-21T23:02:38.376565+0000","last_became_peered":"2024-02-21T23:02:38.376565+0000","last_unstale":"2024-02-21T23:02:43.421488+0000","last_undegraded":"2024-02-21T23:02:43.421488+0000","last_fullsized":"2024-02-21T23:02:43.421488+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:28:43.191300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.597197+0000","last_change":"2024-02-21T23:02:38.379350+0000","last_active":"2024-02-21T23:02:43.597197+0000","last_peered":"2024-02-21T23:02:43.597197+0000","last_clean":"2024-02-21T23:02:43.597197+0000","last_became_active":"2024-02-21T23:02:38.379030+0000","last_became_peered":"2024-02-21T23:02:38.379030+0000","last_unstale":"2024-02-21T23:02:43.597197+0000","last_undegraded":"2024-02-21T23:02:43.597197+0000","last_fullsized":"2024-02-21T23:02:43.597197+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:09:34.130536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.595248+0000","last_change":"2024-02-21T23:02:38.403472+0000","last_active":"2024-02-21T23:02:43.595248+0000","last_peered":"2024-02-21T23:02:43.595248+0000","last_clean":"2024-02-21T23:02:43.595248+0000","last_became_active":"2024-02-21T23:02:38.403213+0000","last_became_peered":"2024-02-21T23:02:38.403213+0000","last_unstale":"2024-02-21T23:02:43.595248+0000","last_undegraded":"2024-02-21T23:02:43.595248+0000","last_fullsized":"2024-02-21T23:02:43.595248+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:29:12.483912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420337+0000","last_change":"2024-02-21T23:02:38.377455+0000","last_active":"2024-02-21T23:02:43.420337+0000","last_peered":"2024-02-21T23:02:43.420337+0000","last_clean":"2024-02-21T23:02:43.420337+0000","last_became_active":"2024-02-21T23:02:38.376781+0000","last_became_peered":"2024-02-21T23:02:38.376781+0000","last_unstale":"2024-02-21T23:02:43.420337+0000","last_undegraded":"2024-02-21T23:02:43.420337+0000","last_fullsized":"2024-02-21T23:02:43.420337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:24:40.762404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.599882+0000","last_change":"2024-02-21T23:02:38.386554+0000","last_active":"2024-02-21T23:02:43.599882+0000","last_peered":"2024-02-21T23:02:43.599882+0000","last_clean":"2024-02-21T23:02:43.599882+0000","last_became_active":"2024-02-21T23:02:38.384500+0000","last_became_peered":"2024-02-21T23:02:38.384500+0000","last_unstale":"2024-02-21T23:02:43.599882+0000","last_undegraded":"2024-02-21T23:02:43.599882+0000","last_fullsized":"2024-02-21T23:02:43.599882+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:09:34.130536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420337+0000","last_change":"2024-02-21T23:02:38.383893+0000","last_active":"2024-02-21T23:02:43.420337+0000","last_peered":"2024-02-21T23:02:43.420337+0000","last_clean":"2024-02-21T23:02:43.420337+0000","last_became_active":"2024-02-21T23:02:38.383569+0000","last_became_peered":"2024-02-21T23:02:38.383569+0000","last_unstale":"2024-02-21T23:02:43.420337+0000","last_undegraded":"2024-02-21T23:02:43.420337+0000","last_fullsized":"2024-02-21T23:02:43.420337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:19:25.522975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.600302+0000","last_change":"2024-02-21T23:02:38.395013+0000","last_active":"2024-02-21T23:02:43.600302+0000","last_peered":"2024-02-21T23:02:43.600302+0000","last_clean":"2024-02-21T23:02:43.600302+0000","last_became_active":"2024-02-21T23:02:38.394670+0000","last_became_peered":"2024-02-21T23:02:38.394670+0000","last_unstale":"2024-02-21T23:02:43.600302+0000","last_undegraded":"2024-02-21T23:02:43.600302+0000","last_fullsized":"2024-02-21T23:02:43.600302+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:54:40.962238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.596541+0000","last_change":"2024-02-21T23:02:38.385293+0000","last_active":"2024-02-21T23:02:43.596541+0000","last_peered":"2024-02-21T23:02:43.596541+0000","last_clean":"2024-02-21T23:02:43.596541+0000","last_became_active":"2024-02-21T23:02:38.384589+0000","last_became_peered":"2024-02-21T23:02:38.384589+0000","last_unstale":"2024-02-21T23:02:43.596541+0000","last_undegraded":"2024-02-21T23:02:43.596541+0000","last_fullsized":"2024-02-21T23:02:43.596541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:10:48.186239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.595433+0000","last_change":"2024-02-21T23:02:38.385769+0000","last_active":"2024-02-21T23:02:43.595433+0000","last_peered":"2024-02-21T23:02:43.595433+0000","last_clean":"2024-02-21T23:02:43.595433+0000","last_became_active":"2024-02-21T23:02:38.385488+0000","last_became_peered":"2024-02-21T23:02:38.385488+0000","last_unstale":"2024-02-21T23:02:43.595433+0000","last_undegraded":"2024-02-21T23:02:43.595433+0000","last_fullsized":"2024-02-21T23:02:43.595433+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:29:00.068426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.598237+0000","last_change":"2024-02-21T23:02:38.396764+0000","last_active":"2024-02-21T23:02:43.598237+0000","last_peered":"2024-02-21T23:02:43.598237+0000","last_clean":"2024-02-21T23:02:43.598237+0000","last_became_active":"2024-02-21T23:02:38.396438+0000","last_became_peered":"2024-02-21T23:02:38.396438+0000","last_unstale":"2024-02-21T23:02:43.598237+0000","last_undegraded":"2024-02-21T23:02:43.598237+0000","last_fullsized":"2024-02-21T23:02:43.598237+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:06:59.357124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.416220+0000","last_change":"2024-02-21T23:02:38.387876+0000","last_active":"2024-02-21T23:02:43.416220+0000","last_peered":"2024-02-21T23:02:43.416220+0000","last_clean":"2024-02-21T23:02:43.416220+0000","last_became_active":"2024-02-21T23:02:38.383130+0000","last_became_peered":"2024-02-21T23:02:38.383130+0000","last_unstale":"2024-02-21T23:02:43.416220+0000","last_undegraded":"2024-02-21T23:02:43.416220+0000","last_fullsized":"2024-02-21T23:02:43.416220+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:12:46.637812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.601961+0000","last_change":"2024-02-21T23:02:38.403615+0000","last_active":"2024-02-21T23:02:43.601961+0000","last_peered":"2024-02-21T23:02:43.601961+0000","last_clean":"2024-02-21T23:02:43.601961+0000","last_became_active":"2024-02-21T23:02:38.402773+0000","last_became_peered":"2024-02-21T23:02:38.402773+0000","last_unstale":"2024-02-21T23:02:43.601961+0000","last_undegraded":"2024-02-21T23:02:43.601961+0000","last_fullsized":"2024-02-21T23:02:43.601961+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:12:26.063291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.599735+0000","last_change":"2024-02-21T23:02:38.384479+0000","last_active":"2024-02-21T23:02:43.599735+0000","last_peered":"2024-02-21T23:02:43.599735+0000","last_clean":"2024-02-21T23:02:43.599735+0000","last_became_active":"2024-02-21T23:02:38.383995+0000","last_became_peered":"2024-02-21T23:02:38.383995+0000","last_unstale":"2024-02-21T23:02:43.599735+0000","last_undegraded":"2024-02-21T23:02:43.599735+0000","last_fullsized":"2024-02-21T23:02:43.599735+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:28:43.191300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.595040+0000","last_change":"2024-02-21T23:02:38.390823+0000","last_active":"2024-02-21T23:02:43.595040+0000","last_peered":"2024-02-21T23:02:43.595040+0000","last_clean":"2024-02-21T23:02:43.595040+0000","last_became_active":"2024-02-21T23:02:38.390390+0000","last_became_peered":"2024-02-21T23:02:38.390390+0000","last_unstale":"2024-02-21T23:02:43.595040+0000","last_undegraded":"2024-02-21T23:02:43.595040+0000","last_fullsized":"2024-02-21T23:02:43.595040+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:29:00.068426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.595697+0000","last_change":"2024-02-21T23:02:38.380151+0000","last_active":"2024-02-21T23:02:43.595697+0000","last_peered":"2024-02-21T23:02:43.595697+0000","last_clean":"2024-02-21T23:02:43.595697+0000","last_became_active":"2024-02-21T23:02:38.379792+0000","last_became_peered":"2024-02-21T23:02:38.379792+0000","last_unstale":"2024-02-21T23:02:43.595697+0000","last_undegraded":"2024-02-21T23:02:43.595697+0000","last_fullsized":"2024-02-21T23:02:43.595697+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:39:18.270811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420027+0000","last_change":"2024-02-21T23:02:38.379804+0000","last_active":"2024-02-21T23:02:43.420027+0000","last_peered":"2024-02-21T23:02:43.420027+0000","last_clean":"2024-02-21T23:02:43.420027+0000","last_became_active":"2024-02-21T23:02:38.379461+0000","last_became_peered":"2024-02-21T23:02:38.379461+0000","last_unstale":"2024-02-21T23:02:43.420027+0000","last_undegraded":"2024-02-21T23:02:43.420027+0000","last_fullsized":"2024-02-21T23:02:43.420027+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:09:34.130536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420766+0000","last_change":"2024-02-21T23:02:38.381806+0000","last_active":"2024-02-21T23:02:43.420766+0000","last_peered":"2024-02-21T23:02:43.420766+0000","last_clean":"2024-02-21T23:02:43.420766+0000","last_became_active":"2024-02-21T23:02:38.381475+0000","last_became_peered":"2024-02-21T23:02:38.381475+0000","last_unstale":"2024-02-21T23:02:43.420766+0000","last_undegraded":"2024-02-21T23:02:43.420766+0000","last_fullsized":"2024-02-21T23:02:43.420766+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:29:27.336333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.595120+0000","last_change":"2024-02-21T23:02:38.398122+0000","last_active":"2024-02-21T23:02:43.595120+0000","last_peered":"2024-02-21T23:02:43.595120+0000","last_clean":"2024-02-21T23:02:43.595120+0000","last_became_active":"2024-02-21T23:02:38.392050+0000","last_became_peered":"2024-02-21T23:02:38.392050+0000","last_unstale":"2024-02-21T23:02:43.595120+0000","last_undegraded":"2024-02-21T23:02:43.595120+0000","last_fullsized":"2024-02-21T23:02:43.595120+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:10:48.186239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.594981+0000","last_change":"2024-02-21T23:02:38.398482+0000","last_active":"2024-02-21T23:02:43.594981+0000","last_peered":"2024-02-21T23:02:43.594981+0000","last_clean":"2024-02-21T23:02:43.594981+0000","last_became_active":"2024-02-21T23:02:38.392232+0000","last_became_peered":"2024-02-21T23:02:38.392232+0000","last_unstale":"2024-02-21T23:02:43.594981+0000","last_undegraded":"2024-02-21T23:02:43.594981+0000","last_fullsized":"2024-02-21T23:02:43.594981+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:30:37.791426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.598787+0000","last_change":"2024-02-21T23:02:38.400825+0000","last_active":"2024-02-21T23:02:43.598787+0000","last_peered":"2024-02-21T23:02:43.598787+0000","last_clean":"2024-02-21T23:02:43.598787+0000","last_became_active":"2024-02-21T23:02:38.398964+0000","last_became_peered":"2024-02-21T23:02:38.398964+0000","last_unstale":"2024-02-21T23:02:43.598787+0000","last_undegraded":"2024-02-21T23:02:43.598787+0000","last_fullsized":"2024-02-21T23:02:43.598787+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:24:40.762404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.418962+0000","last_change":"2024-02-21T23:02:38.387131+0000","last_active":"2024-02-21T23:02:43.418962+0000","last_peered":"2024-02-21T23:02:43.418962+0000","last_clean":"2024-02-21T23:02:43.418962+0000","last_became_active":"2024-02-21T23:02:38.386835+0000","last_became_peered":"2024-02-21T23:02:38.386835+0000","last_unstale":"2024-02-21T23:02:43.418962+0000","last_undegraded":"2024-02-21T23:02:43.418962+0000","last_fullsized":"2024-02-21T23:02:43.418962+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:10:48.186239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.596722+0000","last_change":"2024-02-21T23:02:38.394292+0000","last_active":"2024-02-21T23:02:43.596722+0000","last_peered":"2024-02-21T23:02:43.596722+0000","last_clean":"2024-02-21T23:02:43.596722+0000","last_became_active":"2024-02-21T23:02:38.394006+0000","last_became_peered":"2024-02-21T23:02:38.394006+0000","last_unstale":"2024-02-21T23:02:43.596722+0000","last_undegraded":"2024-02-21T23:02:43.596722+0000","last_fullsized":"2024-02-21T23:02:43.596722+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:54:40.962238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.600192+0000","last_change":"2024-02-21T23:02:38.394994+0000","last_active":"2024-02-21T23:02:43.600192+0000","last_peered":"2024-02-21T23:02:43.600192+0000","last_clean":"2024-02-21T23:02:43.600192+0000","last_became_active":"2024-02-21T23:02:38.394693+0000","last_became_peered":"2024-02-21T23:02:38.394693+0000","last_unstale":"2024-02-21T23:02:43.600192+0000","last_undegraded":"2024-02-21T23:02:43.600192+0000","last_fullsized":"2024-02-21T23:02:43.600192+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:00:09.172059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.597595+0000","last_change":"2024-02-21T23:02:38.397147+0000","last_active":"2024-02-21T23:02:43.597595+0000","last_peered":"2024-02-21T23:02:43.597595+0000","last_clean":"2024-02-21T23:02:43.597595+0000","last_became_active":"2024-02-21T23:02:38.396787+0000","last_became_peered":"2024-02-21T23:02:38.396787+0000","last_unstale":"2024-02-21T23:02:43.597595+0000","last_undegraded":"2024-02-21T23:02:43.597595+0000","last_fullsized":"2024-02-21T23:02:43.597595+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:35:27.966492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.600709+0000","last_change":"2024-02-21T23:02:38.398132+0000","last_active":"2024-02-21T23:02:43.600709+0000","last_peered":"2024-02-21T23:02:43.600709+0000","last_clean":"2024-02-21T23:02:43.600709+0000","last_became_active":"2024-02-21T23:02:38.397847+0000","last_became_peered":"2024-02-21T23:02:38.397847+0000","last_unstale":"2024-02-21T23:02:43.600709+0000","last_undegraded":"2024-02-21T23:02:43.600709+0000","last_fullsized":"2024-02-21T23:02:43.600709+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:28:23.752647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.596548+0000","last_change":"2024-02-21T23:02:38.372038+0000","last_active":"2024-02-21T23:02:43.596548+0000","last_peered":"2024-02-21T23:02:43.596548+0000","last_clean":"2024-02-21T23:02:43.596548+0000","last_became_active":"2024-02-21T23:02:38.371731+0000","last_became_peered":"2024-02-21T23:02:38.371731+0000","last_unstale":"2024-02-21T23:02:43.596548+0000","last_undegraded":"2024-02-21T23:02:43.596548+0000","last_fullsized":"2024-02-21T23:02:43.596548+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:04:11.892160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.598184+0000","last_change":"2024-02-21T23:02:38.379292+0000","last_active":"2024-02-21T23:02:43.598184+0000","last_peered":"2024-02-21T23:02:43.598184+0000","last_clean":"2024-02-21T23:02:43.598184+0000","last_became_active":"2024-02-21T23:02:38.378980+0000","last_became_peered":"2024-02-21T23:02:38.378980+0000","last_unstale":"2024-02-21T23:02:43.598184+0000","last_undegraded":"2024-02-21T23:02:43.598184+0000","last_fullsized":"2024-02-21T23:02:43.598184+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:29:12.483912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.598604+0000","last_change":"2024-02-21T23:02:38.396837+0000","last_active":"2024-02-21T23:02:43.598604+0000","last_peered":"2024-02-21T23:02:43.598604+0000","last_clean":"2024-02-21T23:02:43.598604+0000","last_became_active":"2024-02-21T23:02:38.396502+0000","last_became_peered":"2024-02-21T23:02:38.396502+0000","last_unstale":"2024-02-21T23:02:43.598604+0000","last_undegraded":"2024-02-21T23:02:43.598604+0000","last_fullsized":"2024-02-21T23:02:43.598604+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:54:16.651108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574858,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27340,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":323584,"data_stored":691206,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574859,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574858,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":258048,"data_stored":100346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574859,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":266240,"data_stored":100382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574859,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574860,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":253952,"data_stored":100256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574858,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":262144,"data_stored":100416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574858,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31508,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340332,"statfs":{"total":96636764160,"available":96604499968,"internally_reserved":0,"allocated":331776,"data_stored":691262,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-21T23:03:13.575 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-21T23:03:13.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.849+0000 7fa957200640 1 Processor -- start 2024-02-21T23:03:13.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.849+0000 7fa957200640 1 -- start start 2024-02-21T23:03:13.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.851+0000 7fa957200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa958150450 0x7fa9580a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:13.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.851+0000 7fa957200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9580c2760 con 0x7fa95806ee40 2024-02-21T23:03:13.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.851+0000 7fa957200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa958156310 con 0x7fa958156af0 2024-02-21T23:03:13.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.851+0000 7fa957200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9580c22c0 con 0x7fa958150450 2024-02-21T23:03:13.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.851+0000 7fa955e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa958150450 0x7fa9580a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:13.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.851+0000 7fa955e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa958150450 0x7fa9580a5fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46282/0 (socket says 172.21.15.28:46282) 2024-02-21T23:03:13.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.851+0000 7fa955e00640 1 -- 172.21.15.28:0/4225021475 learned_addr learned my addr 172.21.15.28:0/4225021475 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:13.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.852+0000 7fa955e00640 1 -- 172.21.15.28:0/4225021475 >> v1:172.21.15.81:6789/0 conn(0x7fa958156af0 legacy=0x7fa95806dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:13.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.852+0000 7fa955e00640 1 -- 172.21.15.28:0/4225021475 >> v1:172.21.15.28:6789/0 conn(0x7fa95806ee40 legacy=0x7fa9581536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:13.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.852+0000 7fa955400640 1 -- 172.21.15.28:0/4225021475 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3710499887 0 0) 0x7fa9580c2760 con 0x7fa95806ee40 2024-02-21T23:03:13.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.852+0000 7fa955e00640 1 -- 172.21.15.28:0/4225021475 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9580c29d0 con 0x7fa958150450 2024-02-21T23:03:13.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.854+0000 7fa955e00640 1 --2- 172.21.15.28:0/4225021475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa958150450 0x7fa9580a5fd0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa948009870 tx=0x7fa94802ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63a7bc16a3bd9078 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:13.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.854+0000 7fa955400640 1 -- 172.21.15.28:0/4225021475 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa94802f840 con 0x7fa958150450 2024-02-21T23:03:13.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.854+0000 7fa955400640 1 -- 172.21.15.28:0/4225021475 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa94802f9e0 con 0x7fa958150450 2024-02-21T23:03:13.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.854+0000 7fa955400640 1 -- 172.21.15.28:0/4225021475 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa94802fdb0 con 0x7fa958150450 2024-02-21T23:03:13.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.855+0000 7fa957200640 1 -- 172.21.15.28:0/4225021475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa958150450 msgr2=0x7fa9580a5fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:13.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.855+0000 7fa957200640 1 --2- 172.21.15.28:0/4225021475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa958150450 0x7fa9580a5fd0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa948009870 tx=0x7fa94802ba20 comp rx=0 tx=0).stop 2024-02-21T23:03:13.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.855+0000 7fa957200640 1 -- 172.21.15.28:0/4225021475 shutdown_connections 2024-02-21T23:03:13.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.855+0000 7fa957200640 1 --2- 172.21.15.28:0/4225021475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa958150450 0x7fa9580a5fd0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:13.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.855+0000 7fa957200640 1 -- 172.21.15.28:0/4225021475 >> 172.21.15.28:0/4225021475 conn(0x7fa95806b630 msgr2=0x7fa9580c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:13.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.855+0000 7fa957200640 1 -- 172.21.15.28:0/4225021475 shutdown_connections 2024-02-21T23:03:13.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.855+0000 7fa957200640 1 -- 172.21.15.28:0/4225021475 wait complete. 2024-02-21T23:03:13.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.856+0000 7fa957200640 1 Processor -- start 2024-02-21T23:03:13.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.857+0000 7fa957200640 1 -- start start 2024-02-21T23:03:13.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.858+0000 7fa957200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa95806ee40 0x7fa958155d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:13.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.858+0000 7fa957200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa958150450 0x7fa9580bfba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:13.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.858+0000 7fa957200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa958156af0 0x7fa9581ee390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:13.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.858+0000 7fa957200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa95806e700 con 0x7fa958150450 2024-02-21T23:03:13.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.858+0000 7fa957200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa958156660 con 0x7fa95806ee40 2024-02-21T23:03:13.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.858+0000 7fa957200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa95806e880 con 0x7fa958156af0 2024-02-21T23:03:13.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.858+0000 7fa94d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa958150450 0x7fa9580bfba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:13.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.858+0000 7fa94d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa958150450 0x7fa9580bfba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46290/0 (socket says 172.21.15.28:46290) 2024-02-21T23:03:13.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.858+0000 7fa94d400640 1 -- 172.21.15.28:0/1784656965 learned_addr learned my addr 172.21.15.28:0/1784656965 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:13.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.858+0000 7fa94d400640 1 -- 172.21.15.28:0/1784656965 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa958156af0 msgr2=0x7fa9581ee390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:13.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.858+0000 7fa94d400640 1 --2- 172.21.15.28:0/1784656965 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa958156af0 0x7fa9581ee390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:13.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.858+0000 7fa94d400640 1 -- 172.21.15.28:0/1784656965 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa95806ee40 msgr2=0x7fa958155d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:13.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.858+0000 7fa955e00640 1 --2- 172.21.15.28:0/1784656965 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa95806ee40 0x7fa958155d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:13.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.858+0000 7fa94d400640 1 --2- 172.21.15.28:0/1784656965 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa95806ee40 0x7fa958155d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:13.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.858+0000 7fa94d400640 1 -- 172.21.15.28:0/1784656965 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9580c5d40 con 0x7fa958150450 2024-02-21T23:03:13.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.859+0000 7fa94d400640 1 --2- 172.21.15.28:0/1784656965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa958150450 0x7fa9580bfba0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa94000c810 tx=0x7fa94000ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:13.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.859+0000 7fa94ea00640 1 -- 172.21.15.28:0/1784656965 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa940003da0 con 0x7fa958150450 2024-02-21T23:03:13.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.859+0000 7fa94ea00640 1 -- 172.21.15.28:0/1784656965 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa940005070 con 0x7fa958150450 2024-02-21T23:03:13.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.859+0000 7fa94ea00640 1 -- 172.21.15.28:0/1784656965 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa940005420 con 0x7fa958150450 2024-02-21T23:03:13.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.859+0000 7fa957200640 1 -- 172.21.15.28:0/1784656965 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9580abf10 con 0x7fa958150450 2024-02-21T23:03:13.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.859+0000 7fa957200640 1 -- 172.21.15.28:0/1784656965 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9580c00e0 con 0x7fa958150450 2024-02-21T23:03:13.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.860+0000 7fa957200640 1 -- 172.21.15.28:0/1784656965 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9580ac890 con 0x7fa958150450 2024-02-21T23:03:13.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.861+0000 7fa94ea00640 1 -- 172.21.15.28:0/1784656965 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa9400055c0 con 0x7fa958150450 2024-02-21T23:03:13.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.862+0000 7fa94ea00640 1 --2- 172.21.15.28:0/1784656965 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa93406b500 0x7fa93406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:13.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.863+0000 7fa94ea00640 1 -- 172.21.15.28:0/1784656965 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fa94008a4e0 con 0x7fa958150450 2024-02-21T23:03:13.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.863+0000 7fa955e00640 1 --2- 172.21.15.28:0/1784656965 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa93406b500 0x7fa93406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:13.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.864+0000 7fa955e00640 1 --2- 172.21.15.28:0/1784656965 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa93406b500 0x7fa93406d9c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa948036000 tx=0x7fa948039040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:13.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:13.867+0000 7fa94ea00640 1 -- 172.21.15.28:0/1784656965 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fa9580ac890 con 0x7fa958150450 2024-02-21T23:03:14.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.054+0000 7fa957200640 1 -- 172.21.15.28:0/1784656965 --> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa95804f290 con 0x7fa93406b500 2024-02-21T23:03:14.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.097+0000 7fa94ea00640 1 -- 172.21.15.28:0/1784656965 <== mgr.4101 v2:172.21.15.81:6832/1848538577 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+332614 (secure 0 0 0) 0x7fa95804f290 con 0x7fa93406b500 2024-02-21T23:03:14.088 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:03:14.093 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-21T23:03:14.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.104+0000 7fa957200640 1 -- 172.21.15.28:0/1784656965 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa93406b500 msgr2=0x7fa93406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:14.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.104+0000 7fa957200640 1 --2- 172.21.15.28:0/1784656965 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa93406b500 0x7fa93406d9c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa948036000 tx=0x7fa948039040 comp rx=0 tx=0).stop 2024-02-21T23:03:14.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.104+0000 7fa957200640 1 -- 172.21.15.28:0/1784656965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa958150450 msgr2=0x7fa9580bfba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:14.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.104+0000 7fa957200640 1 --2- 172.21.15.28:0/1784656965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa958150450 0x7fa9580bfba0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa94000c810 tx=0x7fa94000ccd0 comp rx=0 tx=0).stop 2024-02-21T23:03:14.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.105+0000 7fa957200640 1 -- 172.21.15.28:0/1784656965 shutdown_connections 2024-02-21T23:03:14.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.105+0000 7fa957200640 1 --2- 172.21.15.28:0/1784656965 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa93406b500 0x7fa93406d9c0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:14.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.105+0000 7fa957200640 1 --2- 172.21.15.28:0/1784656965 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa958156af0 0x7fa9581ee390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:14.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.105+0000 7fa957200640 1 --2- 172.21.15.28:0/1784656965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa958150450 0x7fa9580bfba0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:14.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.105+0000 7fa957200640 1 --2- 172.21.15.28:0/1784656965 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa95806ee40 0x7fa958155d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:14.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.105+0000 7fa957200640 1 -- 172.21.15.28:0/1784656965 >> 172.21.15.28:0/1784656965 conn(0x7fa95806b630 msgr2=0x7fa95814d120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:14.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.105+0000 7fa957200640 1 -- 172.21.15.28:0/1784656965 shutdown_connections 2024-02-21T23:03:14.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.105+0000 7fa957200640 1 -- 172.21.15.28:0/1784656965 wait complete. 2024-02-21T23:03:14.114 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":38,"stamp":"2024-02-21T23:03:12.669735+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":208,"num_write_kb":2435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160,"ondisk_log_size":160,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":222496,"kb_used_data":2176,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754752224,"statfs":{"total":773094113280,"available":772866277376,"internally_reserved":0,"allocated":2228224,"data_stored":1992064,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":225496625},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005195"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":26,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.598140+0000","last_change":"2024-02-21T23:02:37.367904+0000","last_active":"2024-02-21T23:02:43.598140+0000","last_peered":"2024-02-21T23:02:43.598140+0000","last_clean":"2024-02-21T23:02:43.598140+0000","last_became_active":"2024-02-21T23:02:37.367537+0000","last_became_peered":"2024-02-21T23:02:37.367537+0000","last_unstale":"2024-02-21T23:02:43.598140+0000","last_undegraded":"2024-02-21T23:02:43.598140+0000","last_fullsized":"2024-02-21T23:02:43.598140+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:39:34.395881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420749+0000","last_change":"2024-02-21T23:02:38.385115+0000","last_active":"2024-02-21T23:02:43.420749+0000","last_peered":"2024-02-21T23:02:43.420749+0000","last_clean":"2024-02-21T23:02:43.420749+0000","last_became_active":"2024-02-21T23:02:38.384818+0000","last_became_peered":"2024-02-21T23:02:38.384818+0000","last_unstale":"2024-02-21T23:02:43.420749+0000","last_undegraded":"2024-02-21T23:02:43.420749+0000","last_fullsized":"2024-02-21T23:02:43.420749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:54:40.962238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.419345+0000","last_change":"2024-02-21T23:02:42.500553+0000","last_active":"2024-02-21T23:02:43.419345+0000","last_peered":"2024-02-21T23:02:43.419345+0000","last_clean":"2024-02-21T23:02:43.419345+0000","last_became_active":"2024-02-21T23:02:42.498884+0000","last_became_peered":"2024-02-21T23:02:42.498884+0000","last_unstale":"2024-02-21T23:02:43.419345+0000","last_undegraded":"2024-02-21T23:02:43.419345+0000","last_fullsized":"2024-02-21T23:02:43.419345+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:41:42.562121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.421106+0000","last_change":"2024-02-21T23:02:37.354176+0000","last_active":"2024-02-21T23:02:43.421106+0000","last_peered":"2024-02-21T23:02:43.421106+0000","last_clean":"2024-02-21T23:02:43.421106+0000","last_became_active":"2024-02-21T23:02:37.353425+0000","last_became_peered":"2024-02-21T23:02:37.353425+0000","last_unstale":"2024-02-21T23:02:43.421106+0000","last_undegraded":"2024-02-21T23:02:43.421106+0000","last_fullsized":"2024-02-21T23:02:43.421106+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:07:23.801765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.597346+0000","last_change":"2024-02-21T23:02:38.399361+0000","last_active":"2024-02-21T23:02:43.597346+0000","last_peered":"2024-02-21T23:02:43.597346+0000","last_clean":"2024-02-21T23:02:43.597346+0000","last_became_active":"2024-02-21T23:02:38.399026+0000","last_became_peered":"2024-02-21T23:02:38.399026+0000","last_unstale":"2024-02-21T23:02:43.597346+0000","last_undegraded":"2024-02-21T23:02:43.597346+0000","last_fullsized":"2024-02-21T23:02:43.597346+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:29:12.483912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.419463+0000","last_change":"2024-02-21T23:02:42.441232+0000","last_active":"2024-02-21T23:02:43.419463+0000","last_peered":"2024-02-21T23:02:43.419463+0000","last_clean":"2024-02-21T23:02:43.419463+0000","last_became_active":"2024-02-21T23:02:42.440354+0000","last_became_peered":"2024-02-21T23:02:42.440354+0000","last_unstale":"2024-02-21T23:02:43.419463+0000","last_undegraded":"2024-02-21T23:02:43.419463+0000","last_fullsized":"2024-02-21T23:02:43.419463+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:29:16.551658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.598167+0000","last_change":"2024-02-21T23:02:37.373520+0000","last_active":"2024-02-21T23:02:43.598167+0000","last_peered":"2024-02-21T23:02:43.598167+0000","last_clean":"2024-02-21T23:02:43.598167+0000","last_became_active":"2024-02-21T23:02:37.373127+0000","last_became_peered":"2024-02-21T23:02:37.373127+0000","last_unstale":"2024-02-21T23:02:43.598167+0000","last_undegraded":"2024-02-21T23:02:43.598167+0000","last_fullsized":"2024-02-21T23:02:43.598167+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:12:45.623160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.600822+0000","last_change":"2024-02-21T23:02:38.387666+0000","last_active":"2024-02-21T23:02:43.600822+0000","last_peered":"2024-02-21T23:02:43.600822+0000","last_clean":"2024-02-21T23:02:43.600822+0000","last_became_active":"2024-02-21T23:02:38.387151+0000","last_became_peered":"2024-02-21T23:02:38.387151+0000","last_unstale":"2024-02-21T23:02:43.600822+0000","last_undegraded":"2024-02-21T23:02:43.600822+0000","last_fullsized":"2024-02-21T23:02:43.600822+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:24:40.762404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420258+0000","last_change":"2024-02-21T23:02:42.444676+0000","last_active":"2024-02-21T23:02:43.420258+0000","last_peered":"2024-02-21T23:02:43.420258+0000","last_clean":"2024-02-21T23:02:43.420258+0000","last_became_active":"2024-02-21T23:02:42.444185+0000","last_became_peered":"2024-02-21T23:02:42.444185+0000","last_unstale":"2024-02-21T23:02:43.420258+0000","last_undegraded":"2024-02-21T23:02:43.420258+0000","last_fullsized":"2024-02-21T23:02:43.420258+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:54:45.029984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.597339+0000","last_change":"2024-02-21T23:02:37.372881+0000","last_active":"2024-02-21T23:02:43.597339+0000","last_peered":"2024-02-21T23:02:43.597339+0000","last_clean":"2024-02-21T23:02:43.597339+0000","last_became_active":"2024-02-21T23:02:37.372557+0000","last_became_peered":"2024-02-21T23:02:37.372557+0000","last_unstale":"2024-02-21T23:02:43.597339+0000","last_undegraded":"2024-02-21T23:02:43.597339+0000","last_fullsized":"2024-02-21T23:02:43.597339+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:29:33.262208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.600960+0000","last_change":"2024-02-21T23:02:38.399333+0000","last_active":"2024-02-21T23:02:43.600960+0000","last_peered":"2024-02-21T23:02:43.600960+0000","last_clean":"2024-02-21T23:02:43.600960+0000","last_became_active":"2024-02-21T23:02:38.393010+0000","last_became_peered":"2024-02-21T23:02:38.393010+0000","last_unstale":"2024-02-21T23:02:43.600960+0000","last_undegraded":"2024-02-21T23:02:43.600960+0000","last_fullsized":"2024-02-21T23:02:43.600960+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:54:16.651108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.419916+0000","last_change":"2024-02-21T23:02:42.436700+0000","last_active":"2024-02-21T23:02:43.419916+0000","last_peered":"2024-02-21T23:02:43.419916+0000","last_clean":"2024-02-21T23:02:43.419916+0000","last_became_active":"2024-02-21T23:02:42.434264+0000","last_became_peered":"2024-02-21T23:02:42.434264+0000","last_unstale":"2024-02-21T23:02:43.419916+0000","last_undegraded":"2024-02-21T23:02:43.419916+0000","last_fullsized":"2024-02-21T23:02:43.419916+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:28:47.259045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.594139+0000","last_change":"2024-02-21T23:02:38.391008+0000","last_active":"2024-02-21T23:02:43.594139+0000","last_peered":"2024-02-21T23:02:43.594139+0000","last_clean":"2024-02-21T23:02:43.594139+0000","last_became_active":"2024-02-21T23:02:38.390695+0000","last_became_peered":"2024-02-21T23:02:38.390695+0000","last_unstale":"2024-02-21T23:02:43.594139+0000","last_undegraded":"2024-02-21T23:02:43.594139+0000","last_fullsized":"2024-02-21T23:02:43.594139+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:09:34.130536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.598232+0000","last_change":"2024-02-21T23:02:37.373206+0000","last_active":"2024-02-21T23:02:43.598232+0000","last_peered":"2024-02-21T23:02:43.598232+0000","last_clean":"2024-02-21T23:02:43.598232+0000","last_became_active":"2024-02-21T23:02:37.371805+0000","last_became_peered":"2024-02-21T23:02:37.371805+0000","last_unstale":"2024-02-21T23:02:43.598232+0000","last_undegraded":"2024-02-21T23:02:43.598232+0000","last_fullsized":"2024-02-21T23:02:43.598232+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:29:26.321681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420841+0000","last_change":"2024-02-21T23:02:42.435351+0000","last_active":"2024-02-21T23:02:43.420841+0000","last_peered":"2024-02-21T23:02:43.420841+0000","last_clean":"2024-02-21T23:02:43.420841+0000","last_became_active":"2024-02-21T23:02:42.434903+0000","last_became_peered":"2024-02-21T23:02:42.434903+0000","last_unstale":"2024-02-21T23:02:43.420841+0000","last_undegraded":"2024-02-21T23:02:43.420841+0000","last_fullsized":"2024-02-21T23:02:43.420841+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:49:24.584877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420313+0000","last_change":"2024-02-21T23:02:38.369472+0000","last_active":"2024-02-21T23:02:43.420313+0000","last_peered":"2024-02-21T23:02:43.420313+0000","last_clean":"2024-02-21T23:02:43.420313+0000","last_became_active":"2024-02-21T23:02:38.369189+0000","last_became_peered":"2024-02-21T23:02:38.369189+0000","last_unstale":"2024-02-21T23:02:43.420313+0000","last_undegraded":"2024-02-21T23:02:43.420313+0000","last_fullsized":"2024-02-21T23:02:43.420313+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:00:09.172059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.596428+0000","last_change":"2024-02-21T23:02:37.355429+0000","last_active":"2024-02-21T23:02:43.596428+0000","last_peered":"2024-02-21T23:02:43.596428+0000","last_clean":"2024-02-21T23:02:43.596428+0000","last_became_active":"2024-02-21T23:02:37.354573+0000","last_became_peered":"2024-02-21T23:02:37.354573+0000","last_unstale":"2024-02-21T23:02:43.596428+0000","last_undegraded":"2024-02-21T23:02:43.596428+0000","last_fullsized":"2024-02-21T23:02:43.596428+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:17:46.451150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420650+0000","last_change":"2024-02-21T23:02:42.445184+0000","last_active":"2024-02-21T23:02:43.420650+0000","last_peered":"2024-02-21T23:02:43.420650+0000","last_clean":"2024-02-21T23:02:43.420650+0000","last_became_active":"2024-02-21T23:02:42.444773+0000","last_became_peered":"2024-02-21T23:02:42.444773+0000","last_unstale":"2024-02-21T23:02:43.420650+0000","last_undegraded":"2024-02-21T23:02:43.420650+0000","last_fullsized":"2024-02-21T23:02:43.420650+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:24:44.830150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.419669+0000","last_change":"2024-02-21T23:02:38.379076+0000","last_active":"2024-02-21T23:02:43.419669+0000","last_peered":"2024-02-21T23:02:43.419669+0000","last_clean":"2024-02-21T23:02:43.419669+0000","last_became_active":"2024-02-21T23:02:38.378513+0000","last_became_peered":"2024-02-21T23:02:38.378513+0000","last_unstale":"2024-02-21T23:02:43.419669+0000","last_undegraded":"2024-02-21T23:02:43.419669+0000","last_fullsized":"2024-02-21T23:02:43.419669+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:29:00.068426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.595013+0000","last_change":"2024-02-21T23:02:37.359130+0000","last_active":"2024-02-21T23:02:43.595013+0000","last_peered":"2024-02-21T23:02:43.595013+0000","last_clean":"2024-02-21T23:02:43.595013+0000","last_became_active":"2024-02-21T23:02:37.358812+0000","last_became_peered":"2024-02-21T23:02:37.358812+0000","last_unstale":"2024-02-21T23:02:43.595013+0000","last_undegraded":"2024-02-21T23:02:43.595013+0000","last_fullsized":"2024-02-21T23:02:43.595013+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:29:33.262208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.598892+0000","last_change":"2024-02-21T23:02:42.500216+0000","last_active":"2024-02-21T23:02:43.598892+0000","last_peered":"2024-02-21T23:02:43.598892+0000","last_clean":"2024-02-21T23:02:43.598892+0000","last_became_active":"2024-02-21T23:02:42.499762+0000","last_became_peered":"2024-02-21T23:02:42.499762+0000","last_unstale":"2024-02-21T23:02:43.598892+0000","last_undegraded":"2024-02-21T23:02:43.598892+0000","last_fullsized":"2024-02-21T23:02:43.598892+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:12:30.131037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420590+0000","last_change":"2024-02-21T23:02:38.381297+0000","last_active":"2024-02-21T23:02:43.420590+0000","last_peered":"2024-02-21T23:02:43.420590+0000","last_clean":"2024-02-21T23:02:43.420590+0000","last_became_active":"2024-02-21T23:02:38.381015+0000","last_became_peered":"2024-02-21T23:02:38.381015+0000","last_unstale":"2024-02-21T23:02:43.420590+0000","last_undegraded":"2024-02-21T23:02:43.420590+0000","last_fullsized":"2024-02-21T23:02:43.420590+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:29:34.276860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.596339+0000","last_change":"2024-02-21T23:02:37.370201+0000","last_active":"2024-02-21T23:02:43.596339+0000","last_peered":"2024-02-21T23:02:43.596339+0000","last_clean":"2024-02-21T23:02:43.596339+0000","last_became_active":"2024-02-21T23:02:37.369814+0000","last_became_peered":"2024-02-21T23:02:37.369814+0000","last_unstale":"2024-02-21T23:02:43.596339+0000","last_undegraded":"2024-02-21T23:02:43.596339+0000","last_fullsized":"2024-02-21T23:02:43.596339+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:16:38.578056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.421168+0000","last_change":"2024-02-21T23:02:42.439870+0000","last_active":"2024-02-21T23:02:43.421168+0000","last_peered":"2024-02-21T23:02:43.421168+0000","last_clean":"2024-02-21T23:02:43.421168+0000","last_became_active":"2024-02-21T23:02:42.439470+0000","last_became_peered":"2024-02-21T23:02:42.439470+0000","last_unstale":"2024-02-21T23:02:43.421168+0000","last_undegraded":"2024-02-21T23:02:43.421168+0000","last_fullsized":"2024-02-21T23:02:43.421168+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:54:20.718854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.597460+0000","last_change":"2024-02-21T23:02:38.380559+0000","last_active":"2024-02-21T23:02:43.597460+0000","last_peered":"2024-02-21T23:02:43.597460+0000","last_clean":"2024-02-21T23:02:43.597460+0000","last_became_active":"2024-02-21T23:02:38.380229+0000","last_became_peered":"2024-02-21T23:02:38.380229+0000","last_unstale":"2024-02-21T23:02:43.597460+0000","last_undegraded":"2024-02-21T23:02:43.597460+0000","last_fullsized":"2024-02-21T23:02:43.597460+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:09:34.130536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.595919+0000","last_change":"2024-02-21T23:02:37.367812+0000","last_active":"2024-02-21T23:02:43.595919+0000","last_peered":"2024-02-21T23:02:43.595919+0000","last_clean":"2024-02-21T23:02:43.595919+0000","last_became_active":"2024-02-21T23:02:37.367527+0000","last_became_peered":"2024-02-21T23:02:37.367527+0000","last_unstale":"2024-02-21T23:02:43.595919+0000","last_undegraded":"2024-02-21T23:02:43.595919+0000","last_fullsized":"2024-02-21T23:02:43.595919+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:17:46.451150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420213+0000","last_change":"2024-02-21T23:02:42.497817+0000","last_active":"2024-02-21T23:02:43.420213+0000","last_peered":"2024-02-21T23:02:43.420213+0000","last_clean":"2024-02-21T23:02:43.420213+0000","last_became_active":"2024-02-21T23:02:42.497417+0000","last_became_peered":"2024-02-21T23:02:42.497417+0000","last_unstale":"2024-02-21T23:02:43.420213+0000","last_undegraded":"2024-02-21T23:02:43.420213+0000","last_fullsized":"2024-02-21T23:02:43.420213+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:00:13.239804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420747+0000","last_change":"2024-02-21T23:02:38.389615+0000","last_active":"2024-02-21T23:02:43.420747+0000","last_peered":"2024-02-21T23:02:43.420747+0000","last_clean":"2024-02-21T23:02:43.420747+0000","last_became_active":"2024-02-21T23:02:38.389268+0000","last_became_peered":"2024-02-21T23:02:38.389268+0000","last_unstale":"2024-02-21T23:02:43.420747+0000","last_undegraded":"2024-02-21T23:02:43.420747+0000","last_fullsized":"2024-02-21T23:02:43.420747+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:20:23.836303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.598294+0000","last_change":"2024-02-21T23:02:37.356409+0000","last_active":"2024-02-21T23:02:43.598294+0000","last_peered":"2024-02-21T23:02:43.598294+0000","last_clean":"2024-02-21T23:02:43.598294+0000","last_became_active":"2024-02-21T23:02:37.356096+0000","last_became_peered":"2024-02-21T23:02:37.356096+0000","last_unstale":"2024-02-21T23:02:43.598294+0000","last_undegraded":"2024-02-21T23:02:43.598294+0000","last_fullsized":"2024-02-21T23:02:43.598294+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:07:23.801765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.418871+0000","last_change":"2024-02-21T23:02:42.495050+0000","last_active":"2024-02-21T23:02:43.418871+0000","last_peered":"2024-02-21T23:02:43.418871+0000","last_clean":"2024-02-21T23:02:43.418871+0000","last_became_active":"2024-02-21T23:02:42.494469+0000","last_became_peered":"2024-02-21T23:02:42.494469+0000","last_unstale":"2024-02-21T23:02:43.418871+0000","last_undegraded":"2024-02-21T23:02:43.418871+0000","last_fullsized":"2024-02-21T23:02:43.418871+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:00:12.286244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.596815+0000","last_change":"2024-02-21T23:02:38.378708+0000","last_active":"2024-02-21T23:02:43.596815+0000","last_peered":"2024-02-21T23:02:43.596815+0000","last_clean":"2024-02-21T23:02:43.596815+0000","last_became_active":"2024-02-21T23:02:38.378374+0000","last_became_peered":"2024-02-21T23:02:38.378374+0000","last_unstale":"2024-02-21T23:02:43.596815+0000","last_undegraded":"2024-02-21T23:02:43.596815+0000","last_fullsized":"2024-02-21T23:02:43.596815+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:19:25.522975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.598361+0000","last_change":"2024-02-21T23:02:37.372175+0000","last_active":"2024-02-21T23:02:43.598361+0000","last_peered":"2024-02-21T23:02:43.598361+0000","last_clean":"2024-02-21T23:02:43.598361+0000","last_became_active":"2024-02-21T23:02:37.371900+0000","last_became_peered":"2024-02-21T23:02:37.371900+0000","last_unstale":"2024-02-21T23:02:43.598361+0000","last_undegraded":"2024-02-21T23:02:43.598361+0000","last_fullsized":"2024-02-21T23:02:43.598361+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:12:51.484005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.596406+0000","last_change":"2024-02-21T23:02:42.500502+0000","last_active":"2024-02-21T23:02:43.596406+0000","last_peered":"2024-02-21T23:02:43.596406+0000","last_clean":"2024-02-21T23:02:43.596406+0000","last_became_active":"2024-02-21T23:02:42.497658+0000","last_became_peered":"2024-02-21T23:02:42.497658+0000","last_unstale":"2024-02-21T23:02:43.596406+0000","last_undegraded":"2024-02-21T23:02:43.596406+0000","last_fullsized":"2024-02-21T23:02:43.596406+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:54:20.718854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.601212+0000","last_change":"2024-02-21T23:02:38.399632+0000","last_active":"2024-02-21T23:02:43.601212+0000","last_peered":"2024-02-21T23:02:43.601212+0000","last_clean":"2024-02-21T23:02:43.601212+0000","last_became_active":"2024-02-21T23:02:38.392339+0000","last_became_peered":"2024-02-21T23:02:38.392339+0000","last_unstale":"2024-02-21T23:02:43.601212+0000","last_undegraded":"2024-02-21T23:02:43.601212+0000","last_fullsized":"2024-02-21T23:02:43.601212+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:35:27.966492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.596919+0000","last_change":"2024-02-21T23:02:37.347461+0000","last_active":"2024-02-21T23:02:43.596919+0000","last_peered":"2024-02-21T23:02:43.596919+0000","last_clean":"2024-02-21T23:02:43.596919+0000","last_became_active":"2024-02-21T23:02:37.346584+0000","last_became_peered":"2024-02-21T23:02:37.346584+0000","last_unstale":"2024-02-21T23:02:43.596919+0000","last_undegraded":"2024-02-21T23:02:43.596919+0000","last_fullsized":"2024-02-21T23:02:43.596919+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:32:39.769989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.595619+0000","last_change":"2024-02-21T23:02:42.436480+0000","last_active":"2024-02-21T23:02:43.595619+0000","last_peered":"2024-02-21T23:02:43.595619+0000","last_clean":"2024-02-21T23:02:43.595619+0000","last_became_active":"2024-02-21T23:02:42.436035+0000","last_became_peered":"2024-02-21T23:02:42.436035+0000","last_unstale":"2024-02-21T23:02:43.595619+0000","last_undegraded":"2024-02-21T23:02:43.595619+0000","last_fullsized":"2024-02-21T23:02:43.595619+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:54:20.718854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420857+0000","last_change":"2024-02-21T23:02:38.373673+0000","last_active":"2024-02-21T23:02:43.420857+0000","last_peered":"2024-02-21T23:02:43.420857+0000","last_clean":"2024-02-21T23:02:43.420857+0000","last_became_active":"2024-02-21T23:02:38.373324+0000","last_became_peered":"2024-02-21T23:02:38.373324+0000","last_unstale":"2024-02-21T23:02:43.420857+0000","last_undegraded":"2024-02-21T23:02:43.420857+0000","last_fullsized":"2024-02-21T23:02:43.420857+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:09:48.394584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.600679+0000","last_change":"2024-02-21T23:02:37.346663+0000","last_active":"2024-02-21T23:02:43.600679+0000","last_peered":"2024-02-21T23:02:43.600679+0000","last_clean":"2024-02-21T23:02:43.600679+0000","last_became_active":"2024-02-21T23:02:37.344771+0000","last_became_peered":"2024-02-21T23:02:37.344771+0000","last_unstale":"2024-02-21T23:02:43.600679+0000","last_undegraded":"2024-02-21T23:02:43.600679+0000","last_fullsized":"2024-02-21T23:02:43.600679+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:12:51.484005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.596759+0000","last_change":"2024-02-21T23:02:42.440882+0000","last_active":"2024-02-21T23:02:43.596759+0000","last_peered":"2024-02-21T23:02:43.596759+0000","last_clean":"2024-02-21T23:02:43.596759+0000","last_became_active":"2024-02-21T23:02:42.440534+0000","last_became_peered":"2024-02-21T23:02:42.440534+0000","last_unstale":"2024-02-21T23:02:43.596759+0000","last_undegraded":"2024-02-21T23:02:43.596759+0000","last_fullsized":"2024-02-21T23:02:43.596759+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:35:32.034237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.595207+0000","last_change":"2024-02-21T23:02:38.377238+0000","last_active":"2024-02-21T23:02:43.595207+0000","last_peered":"2024-02-21T23:02:43.595207+0000","last_clean":"2024-02-21T23:02:43.595207+0000","last_became_active":"2024-02-21T23:02:38.376272+0000","last_became_peered":"2024-02-21T23:02:38.376272+0000","last_unstale":"2024-02-21T23:02:43.595207+0000","last_undegraded":"2024-02-21T23:02:43.595207+0000","last_fullsized":"2024-02-21T23:02:43.595207+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:10:48.186239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.415223+0000","last_change":"2024-02-21T23:02:37.360945+0000","last_active":"2024-02-21T23:02:43.415223+0000","last_peered":"2024-02-21T23:02:43.415223+0000","last_clean":"2024-02-21T23:02:43.415223+0000","last_became_active":"2024-02-21T23:02:37.359879+0000","last_became_peered":"2024-02-21T23:02:37.359879+0000","last_unstale":"2024-02-21T23:02:43.415223+0000","last_undegraded":"2024-02-21T23:02:43.415223+0000","last_fullsized":"2024-02-21T23:02:43.415223+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:07:23.801765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.598887+0000","last_change":"2024-02-21T23:02:42.441857+0000","last_active":"2024-02-21T23:02:43.598887+0000","last_peered":"2024-02-21T23:02:43.598887+0000","last_clean":"2024-02-21T23:02:43.598887+0000","last_became_active":"2024-02-21T23:02:42.438695+0000","last_became_peered":"2024-02-21T23:02:42.438695+0000","last_unstale":"2024-02-21T23:02:43.598887+0000","last_undegraded":"2024-02-21T23:02:43.598887+0000","last_fullsized":"2024-02-21T23:02:43.598887+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:28:27.820393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.594261+0000","last_change":"2024-02-21T23:02:38.366727+0000","last_active":"2024-02-21T23:02:43.594261+0000","last_peered":"2024-02-21T23:02:43.594261+0000","last_clean":"2024-02-21T23:02:43.594261+0000","last_became_active":"2024-02-21T23:02:38.366344+0000","last_became_peered":"2024-02-21T23:02:38.366344+0000","last_unstale":"2024-02-21T23:02:43.594261+0000","last_undegraded":"2024-02-21T23:02:43.594261+0000","last_fullsized":"2024-02-21T23:02:43.594261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:08:44.843392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.596392+0000","last_change":"2024-02-21T23:02:37.372506+0000","last_active":"2024-02-21T23:02:43.596392+0000","last_peered":"2024-02-21T23:02:43.596392+0000","last_clean":"2024-02-21T23:02:43.596392+0000","last_became_active":"2024-02-21T23:02:37.371806+0000","last_became_peered":"2024-02-21T23:02:37.371806+0000","last_unstale":"2024-02-21T23:02:43.596392+0000","last_undegraded":"2024-02-21T23:02:43.596392+0000","last_fullsized":"2024-02-21T23:02:43.596392+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:09:47.379932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.597397+0000","last_change":"2024-02-21T23:02:42.500339+0000","last_active":"2024-02-21T23:02:43.597397+0000","last_peered":"2024-02-21T23:02:43.597397+0000","last_clean":"2024-02-21T23:02:43.597397+0000","last_became_active":"2024-02-21T23:02:42.498471+0000","last_became_peered":"2024-02-21T23:02:42.498471+0000","last_unstale":"2024-02-21T23:02:43.597397+0000","last_undegraded":"2024-02-21T23:02:43.597397+0000","last_fullsized":"2024-02-21T23:02:43.597397+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:12:30.131037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.419210+0000","last_change":"2024-02-21T23:02:38.400850+0000","last_active":"2024-02-21T23:02:43.419210+0000","last_peered":"2024-02-21T23:02:43.419210+0000","last_clean":"2024-02-21T23:02:43.419210+0000","last_became_active":"2024-02-21T23:02:38.400367+0000","last_became_peered":"2024-02-21T23:02:38.400367+0000","last_unstale":"2024-02-21T23:02:43.419210+0000","last_undegraded":"2024-02-21T23:02:43.419210+0000","last_fullsized":"2024-02-21T23:02:43.419210+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:29:12.483912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.600646+0000","last_change":"2024-02-21T23:02:37.349152+0000","last_active":"2024-02-21T23:02:43.600646+0000","last_peered":"2024-02-21T23:02:43.600646+0000","last_clean":"2024-02-21T23:02:43.600646+0000","last_became_active":"2024-02-21T23:02:37.348647+0000","last_became_peered":"2024-02-21T23:02:37.348647+0000","last_unstale":"2024-02-21T23:02:43.600646+0000","last_undegraded":"2024-02-21T23:02:43.600646+0000","last_fullsized":"2024-02-21T23:02:43.600646+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:17:46.451150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":178,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.596598+0000","last_change":"2024-02-21T23:02:32.289335+0000","last_active":"2024-02-21T23:02:43.596598+0000","last_peered":"2024-02-21T23:02:43.596598+0000","last_clean":"2024-02-21T23:02:43.596598+0000","last_became_active":"2024-02-21T23:02:32.286913+0000","last_became_peered":"2024-02-21T23:02:32.286913+0000","last_unstale":"2024-02-21T23:02:43.596598+0000","last_undegraded":"2024-02-21T23:02:43.596598+0000","last_fullsized":"2024-02-21T23:02:43.596598+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_clean_scrub_stamp":"2024-02-21T23:02:31.172569+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:36:23.987374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.594895+0000","last_change":"2024-02-21T23:02:42.498307+0000","last_active":"2024-02-21T23:02:43.594895+0000","last_peered":"2024-02-21T23:02:43.594895+0000","last_clean":"2024-02-21T23:02:43.594895+0000","last_became_active":"2024-02-21T23:02:42.497846+0000","last_became_peered":"2024-02-21T23:02:42.497846+0000","last_unstale":"2024-02-21T23:02:43.594895+0000","last_undegraded":"2024-02-21T23:02:43.594895+0000","last_fullsized":"2024-02-21T23:02:43.594895+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:24:44.830150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.597925+0000","last_change":"2024-02-21T23:02:38.401025+0000","last_active":"2024-02-21T23:02:43.597925+0000","last_peered":"2024-02-21T23:02:43.597925+0000","last_clean":"2024-02-21T23:02:43.597925+0000","last_became_active":"2024-02-21T23:02:38.400639+0000","last_became_peered":"2024-02-21T23:02:38.400639+0000","last_unstale":"2024-02-21T23:02:43.597925+0000","last_undegraded":"2024-02-21T23:02:43.597925+0000","last_fullsized":"2024-02-21T23:02:43.597925+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:49:20.517131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.416472+0000","last_change":"2024-02-21T23:02:37.360670+0000","last_active":"2024-02-21T23:02:43.416472+0000","last_peered":"2024-02-21T23:02:43.416472+0000","last_clean":"2024-02-21T23:02:43.416472+0000","last_became_active":"2024-02-21T23:02:37.358188+0000","last_became_peered":"2024-02-21T23:02:37.358188+0000","last_unstale":"2024-02-21T23:02:43.416472+0000","last_undegraded":"2024-02-21T23:02:43.416472+0000","last_fullsized":"2024-02-21T23:02:43.416472+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:17:46.451150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.595571+0000","last_change":"2024-02-21T23:02:42.495923+0000","last_active":"2024-02-21T23:02:43.595571+0000","last_peered":"2024-02-21T23:02:43.595571+0000","last_clean":"2024-02-21T23:02:43.595571+0000","last_became_active":"2024-02-21T23:02:42.495507+0000","last_became_peered":"2024-02-21T23:02:42.495507+0000","last_unstale":"2024-02-21T23:02:43.595571+0000","last_undegraded":"2024-02-21T23:02:43.595571+0000","last_fullsized":"2024-02-21T23:02:43.595571+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:54:20.718854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.600858+0000","last_change":"2024-02-21T23:02:38.386757+0000","last_active":"2024-02-21T23:02:43.600858+0000","last_peered":"2024-02-21T23:02:43.600858+0000","last_clean":"2024-02-21T23:02:43.600858+0000","last_became_active":"2024-02-21T23:02:38.384691+0000","last_became_peered":"2024-02-21T23:02:38.384691+0000","last_unstale":"2024-02-21T23:02:43.600858+0000","last_undegraded":"2024-02-21T23:02:43.600858+0000","last_fullsized":"2024-02-21T23:02:43.600858+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:49:20.517131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.597791+0000","last_change":"2024-02-21T23:02:37.356479+0000","last_active":"2024-02-21T23:02:43.597791+0000","last_peered":"2024-02-21T23:02:43.597791+0000","last_clean":"2024-02-21T23:02:43.597791+0000","last_became_active":"2024-02-21T23:02:37.356132+0000","last_became_peered":"2024-02-21T23:02:37.356132+0000","last_unstale":"2024-02-21T23:02:43.597791+0000","last_undegraded":"2024-02-21T23:02:43.597791+0000","last_fullsized":"2024-02-21T23:02:43.597791+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:16:38.578056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.597135+0000","last_change":"2024-02-21T23:02:42.440448+0000","last_active":"2024-02-21T23:02:43.597135+0000","last_peered":"2024-02-21T23:02:43.597135+0000","last_clean":"2024-02-21T23:02:43.597135+0000","last_became_active":"2024-02-21T23:02:42.439847+0000","last_became_peered":"2024-02-21T23:02:42.439847+0000","last_unstale":"2024-02-21T23:02:43.597135+0000","last_undegraded":"2024-02-21T23:02:43.597135+0000","last_fullsized":"2024-02-21T23:02:43.597135+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:28:27.820393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.598913+0000","last_change":"2024-02-21T23:02:38.401714+0000","last_active":"2024-02-21T23:02:43.598913+0000","last_peered":"2024-02-21T23:02:43.598913+0000","last_clean":"2024-02-21T23:02:43.598913+0000","last_became_active":"2024-02-21T23:02:38.401166+0000","last_became_peered":"2024-02-21T23:02:38.401166+0000","last_unstale":"2024-02-21T23:02:43.598913+0000","last_undegraded":"2024-02-21T23:02:43.598913+0000","last_fullsized":"2024-02-21T23:02:43.598913+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:33:11.255331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.416538+0000","last_change":"2024-02-21T23:02:37.356667+0000","last_active":"2024-02-21T23:02:43.416538+0000","last_peered":"2024-02-21T23:02:43.416538+0000","last_clean":"2024-02-21T23:02:43.416538+0000","last_became_active":"2024-02-21T23:02:37.354832+0000","last_became_peered":"2024-02-21T23:02:37.354832+0000","last_unstale":"2024-02-21T23:02:43.416538+0000","last_undegraded":"2024-02-21T23:02:43.416538+0000","last_fullsized":"2024-02-21T23:02:43.416538+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:01:17.207079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.599759+0000","last_change":"2024-02-21T23:02:42.494816+0000","last_active":"2024-02-21T23:02:43.599759+0000","last_peered":"2024-02-21T23:02:43.599759+0000","last_clean":"2024-02-21T23:02:43.599759+0000","last_became_active":"2024-02-21T23:02:42.494397+0000","last_became_peered":"2024-02-21T23:02:42.494397+0000","last_unstale":"2024-02-21T23:02:43.599759+0000","last_undegraded":"2024-02-21T23:02:43.599759+0000","last_fullsized":"2024-02-21T23:02:43.599759+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:35:47.803796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.597807+0000","last_change":"2024-02-21T23:02:38.402995+0000","last_active":"2024-02-21T23:02:43.597807+0000","last_peered":"2024-02-21T23:02:43.597807+0000","last_clean":"2024-02-21T23:02:43.597807+0000","last_became_active":"2024-02-21T23:02:38.400263+0000","last_became_peered":"2024-02-21T23:02:38.400263+0000","last_unstale":"2024-02-21T23:02:43.597807+0000","last_undegraded":"2024-02-21T23:02:43.597807+0000","last_fullsized":"2024-02-21T23:02:43.597807+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:28:23.752647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.421321+0000","last_change":"2024-02-21T23:02:37.348962+0000","last_active":"2024-02-21T23:02:43.421321+0000","last_peered":"2024-02-21T23:02:43.421321+0000","last_clean":"2024-02-21T23:02:43.421321+0000","last_became_active":"2024-02-21T23:02:37.348550+0000","last_became_peered":"2024-02-21T23:02:37.348550+0000","last_unstale":"2024-02-21T23:02:43.421321+0000","last_undegraded":"2024-02-21T23:02:43.421321+0000","last_fullsized":"2024-02-21T23:02:43.421321+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:20:15.139033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.419867+0000","last_change":"2024-02-21T23:02:42.442477+0000","last_active":"2024-02-21T23:02:43.419867+0000","last_peered":"2024-02-21T23:02:43.419867+0000","last_clean":"2024-02-21T23:02:43.419867+0000","last_became_active":"2024-02-21T23:02:42.442091+0000","last_became_peered":"2024-02-21T23:02:42.442091+0000","last_unstale":"2024-02-21T23:02:43.419867+0000","last_undegraded":"2024-02-21T23:02:43.419867+0000","last_fullsized":"2024-02-21T23:02:43.419867+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:09:38.198282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.596717+0000","last_change":"2024-02-21T23:02:38.401181+0000","last_active":"2024-02-21T23:02:43.596717+0000","last_peered":"2024-02-21T23:02:43.596717+0000","last_clean":"2024-02-21T23:02:43.596717+0000","last_became_active":"2024-02-21T23:02:38.400463+0000","last_became_peered":"2024-02-21T23:02:38.400463+0000","last_unstale":"2024-02-21T23:02:43.596717+0000","last_undegraded":"2024-02-21T23:02:43.596717+0000","last_fullsized":"2024-02-21T23:02:43.596717+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:05:40.265121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.414478+0000","last_change":"2024-02-21T23:02:37.348047+0000","last_active":"2024-02-21T23:02:43.414478+0000","last_peered":"2024-02-21T23:02:43.414478+0000","last_clean":"2024-02-21T23:02:43.414478+0000","last_became_active":"2024-02-21T23:02:37.347664+0000","last_became_peered":"2024-02-21T23:02:37.347664+0000","last_unstale":"2024-02-21T23:02:43.414478+0000","last_undegraded":"2024-02-21T23:02:43.414478+0000","last_fullsized":"2024-02-21T23:02:43.414478+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:39:34.395881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.596934+0000","last_change":"2024-02-21T23:02:42.447743+0000","last_active":"2024-02-21T23:02:43.596934+0000","last_peered":"2024-02-21T23:02:43.596934+0000","last_clean":"2024-02-21T23:02:43.596934+0000","last_became_active":"2024-02-21T23:02:42.447181+0000","last_became_peered":"2024-02-21T23:02:42.447181+0000","last_unstale":"2024-02-21T23:02:43.596934+0000","last_undegraded":"2024-02-21T23:02:43.596934+0000","last_fullsized":"2024-02-21T23:02:43.596934+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:07:03.424869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.594280+0000","last_change":"2024-02-21T23:02:38.375362+0000","last_active":"2024-02-21T23:02:43.594280+0000","last_peered":"2024-02-21T23:02:43.594280+0000","last_clean":"2024-02-21T23:02:43.594280+0000","last_became_active":"2024-02-21T23:02:38.375081+0000","last_became_peered":"2024-02-21T23:02:38.375081+0000","last_unstale":"2024-02-21T23:02:43.594280+0000","last_undegraded":"2024-02-21T23:02:43.594280+0000","last_fullsized":"2024-02-21T23:02:43.594280+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:19:25.522975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.595646+0000","last_change":"2024-02-21T23:02:37.359690+0000","last_active":"2024-02-21T23:02:43.595646+0000","last_peered":"2024-02-21T23:02:43.595646+0000","last_clean":"2024-02-21T23:02:43.595646+0000","last_became_active":"2024-02-21T23:02:37.359396+0000","last_became_peered":"2024-02-21T23:02:37.359396+0000","last_unstale":"2024-02-21T23:02:43.595646+0000","last_undegraded":"2024-02-21T23:02:43.595646+0000","last_fullsized":"2024-02-21T23:02:43.595646+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:29:33.262208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420901+0000","last_change":"2024-02-21T23:02:42.499203+0000","last_active":"2024-02-21T23:02:43.420901+0000","last_peered":"2024-02-21T23:02:43.420901+0000","last_clean":"2024-02-21T23:02:43.420901+0000","last_became_active":"2024-02-21T23:02:42.498781+0000","last_became_peered":"2024-02-21T23:02:42.498781+0000","last_unstale":"2024-02-21T23:02:43.420901+0000","last_undegraded":"2024-02-21T23:02:43.420901+0000","last_fullsized":"2024-02-21T23:02:43.420901+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:08:32.584721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.598100+0000","last_change":"2024-02-21T23:02:38.402276+0000","last_active":"2024-02-21T23:02:43.598100+0000","last_peered":"2024-02-21T23:02:43.598100+0000","last_clean":"2024-02-21T23:02:43.598100+0000","last_became_active":"2024-02-21T23:02:38.401976+0000","last_became_peered":"2024-02-21T23:02:38.401976+0000","last_unstale":"2024-02-21T23:02:43.598100+0000","last_undegraded":"2024-02-21T23:02:43.598100+0000","last_fullsized":"2024-02-21T23:02:43.598100+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:12:26.063291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.600414+0000","last_change":"2024-02-21T23:02:37.359735+0000","last_active":"2024-02-21T23:02:43.600414+0000","last_peered":"2024-02-21T23:02:43.600414+0000","last_clean":"2024-02-21T23:02:43.600414+0000","last_became_active":"2024-02-21T23:02:37.359447+0000","last_became_peered":"2024-02-21T23:02:37.359447+0000","last_unstale":"2024-02-21T23:02:43.600414+0000","last_undegraded":"2024-02-21T23:02:43.600414+0000","last_fullsized":"2024-02-21T23:02:43.600414+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:29:26.321681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.421314+0000","last_change":"2024-02-21T23:02:42.494896+0000","last_active":"2024-02-21T23:02:43.421314+0000","last_peered":"2024-02-21T23:02:43.421314+0000","last_clean":"2024-02-21T23:02:43.421314+0000","last_became_active":"2024-02-21T23:02:42.494433+0000","last_became_peered":"2024-02-21T23:02:42.494433+0000","last_unstale":"2024-02-21T23:02:43.421314+0000","last_undegraded":"2024-02-21T23:02:43.421314+0000","last_fullsized":"2024-02-21T23:02:43.421314+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:53:14.069591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.595745+0000","last_change":"2024-02-21T23:02:38.389883+0000","last_active":"2024-02-21T23:02:43.595745+0000","last_peered":"2024-02-21T23:02:43.595745+0000","last_clean":"2024-02-21T23:02:43.595745+0000","last_became_active":"2024-02-21T23:02:38.389132+0000","last_became_peered":"2024-02-21T23:02:38.389132+0000","last_unstale":"2024-02-21T23:02:43.595745+0000","last_undegraded":"2024-02-21T23:02:43.595745+0000","last_fullsized":"2024-02-21T23:02:43.595745+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:49:26.724162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.597566+0000","last_change":"2024-02-21T23:02:37.354560+0000","last_active":"2024-02-21T23:02:43.597566+0000","last_peered":"2024-02-21T23:02:43.597566+0000","last_clean":"2024-02-21T23:02:43.597566+0000","last_became_active":"2024-02-21T23:02:37.354266+0000","last_became_peered":"2024-02-21T23:02:37.354266+0000","last_unstale":"2024-02-21T23:02:43.597566+0000","last_undegraded":"2024-02-21T23:02:43.597566+0000","last_fullsized":"2024-02-21T23:02:43.597566+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:36:43.400580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.601188+0000","last_change":"2024-02-21T23:02:42.496211+0000","last_active":"2024-02-21T23:02:43.601188+0000","last_peered":"2024-02-21T23:02:43.601188+0000","last_clean":"2024-02-21T23:02:43.601188+0000","last_became_active":"2024-02-21T23:02:42.495493+0000","last_became_peered":"2024-02-21T23:02:42.495493+0000","last_unstale":"2024-02-21T23:02:43.601188+0000","last_undegraded":"2024-02-21T23:02:43.601188+0000","last_fullsized":"2024-02-21T23:02:43.601188+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:00:12.286244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.597793+0000","last_change":"2024-02-21T23:02:38.395172+0000","last_active":"2024-02-21T23:02:43.597793+0000","last_peered":"2024-02-21T23:02:43.597793+0000","last_clean":"2024-02-21T23:02:43.597793+0000","last_became_active":"2024-02-21T23:02:38.394809+0000","last_became_peered":"2024-02-21T23:02:38.394809+0000","last_unstale":"2024-02-21T23:02:43.597793+0000","last_undegraded":"2024-02-21T23:02:43.597793+0000","last_fullsized":"2024-02-21T23:02:43.597793+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:20:23.836303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"24'10","reported_seq":32,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:57.662420+0000","last_change":"2024-02-21T23:02:37.362539+0000","last_active":"2024-02-21T23:02:57.662420+0000","last_peered":"2024-02-21T23:02:57.662420+0000","last_clean":"2024-02-21T23:02:57.662420+0000","last_became_active":"2024-02-21T23:02:37.362186+0000","last_became_peered":"2024-02-21T23:02:37.362186+0000","last_unstale":"2024-02-21T23:02:57.662420+0000","last_undegraded":"2024-02-21T23:02:57.662420+0000","last_fullsized":"2024-02-21T23:02:57.662420+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:32:39.769989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.597415+0000","last_change":"2024-02-21T23:02:42.433330+0000","last_active":"2024-02-21T23:02:43.597415+0000","last_peered":"2024-02-21T23:02:43.597415+0000","last_clean":"2024-02-21T23:02:43.597415+0000","last_became_active":"2024-02-21T23:02:42.432944+0000","last_became_peered":"2024-02-21T23:02:42.432944+0000","last_unstale":"2024-02-21T23:02:43.597415+0000","last_undegraded":"2024-02-21T23:02:43.597415+0000","last_fullsized":"2024-02-21T23:02:43.597415+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:07:03.424869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420973+0000","last_change":"2024-02-21T23:02:38.389764+0000","last_active":"2024-02-21T23:02:43.420973+0000","last_peered":"2024-02-21T23:02:43.420973+0000","last_clean":"2024-02-21T23:02:43.420973+0000","last_became_active":"2024-02-21T23:02:38.389176+0000","last_became_peered":"2024-02-21T23:02:38.389176+0000","last_unstale":"2024-02-21T23:02:43.420973+0000","last_undegraded":"2024-02-21T23:02:43.420973+0000","last_fullsized":"2024-02-21T23:02:43.420973+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:12:46.637812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.597677+0000","last_change":"2024-02-21T23:02:37.344126+0000","last_active":"2024-02-21T23:02:43.597677+0000","last_peered":"2024-02-21T23:02:43.597677+0000","last_clean":"2024-02-21T23:02:43.597677+0000","last_became_active":"2024-02-21T23:02:37.343769+0000","last_became_peered":"2024-02-21T23:02:37.343769+0000","last_unstale":"2024-02-21T23:02:43.597677+0000","last_undegraded":"2024-02-21T23:02:43.597677+0000","last_fullsized":"2024-02-21T23:02:43.597677+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:09:47.379932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420591+0000","last_change":"2024-02-21T23:02:42.440359+0000","last_active":"2024-02-21T23:02:43.420591+0000","last_peered":"2024-02-21T23:02:43.420591+0000","last_clean":"2024-02-21T23:02:43.420591+0000","last_became_active":"2024-02-21T23:02:42.439967+0000","last_became_peered":"2024-02-21T23:02:42.439967+0000","last_unstale":"2024-02-21T23:02:43.420591+0000","last_undegraded":"2024-02-21T23:02:43.420591+0000","last_fullsized":"2024-02-21T23:02:43.420591+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:20:27.904049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.597926+0000","last_change":"2024-02-21T23:02:38.393522+0000","last_active":"2024-02-21T23:02:43.597926+0000","last_peered":"2024-02-21T23:02:43.597926+0000","last_clean":"2024-02-21T23:02:43.597926+0000","last_became_active":"2024-02-21T23:02:38.392890+0000","last_became_peered":"2024-02-21T23:02:38.392890+0000","last_unstale":"2024-02-21T23:02:43.597926+0000","last_undegraded":"2024-02-21T23:02:43.597926+0000","last_fullsized":"2024-02-21T23:02:43.597926+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:30:37.791426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.601175+0000","last_change":"2024-02-21T23:02:37.361082+0000","last_active":"2024-02-21T23:02:43.601175+0000","last_peered":"2024-02-21T23:02:43.601175+0000","last_clean":"2024-02-21T23:02:43.601175+0000","last_became_active":"2024-02-21T23:02:37.360751+0000","last_became_peered":"2024-02-21T23:02:37.360751+0000","last_unstale":"2024-02-21T23:02:43.601175+0000","last_undegraded":"2024-02-21T23:02:43.601175+0000","last_fullsized":"2024-02-21T23:02:43.601175+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:09:47.379932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.419487+0000","last_change":"2024-02-21T23:02:42.440260+0000","last_active":"2024-02-21T23:02:43.419487+0000","last_peered":"2024-02-21T23:02:43.419487+0000","last_clean":"2024-02-21T23:02:43.419487+0000","last_became_active":"2024-02-21T23:02:42.439785+0000","last_became_peered":"2024-02-21T23:02:42.439785+0000","last_unstale":"2024-02-21T23:02:43.419487+0000","last_undegraded":"2024-02-21T23:02:43.419487+0000","last_fullsized":"2024-02-21T23:02:43.419487+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:42:58.884647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.600988+0000","last_change":"2024-02-21T23:02:38.400013+0000","last_active":"2024-02-21T23:02:43.600988+0000","last_peered":"2024-02-21T23:02:43.600988+0000","last_clean":"2024-02-21T23:02:43.600988+0000","last_became_active":"2024-02-21T23:02:38.399708+0000","last_became_peered":"2024-02-21T23:02:38.399708+0000","last_unstale":"2024-02-21T23:02:43.600988+0000","last_undegraded":"2024-02-21T23:02:43.600988+0000","last_fullsized":"2024-02-21T23:02:43.600988+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:06:59.357124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.595798+0000","last_change":"2024-02-21T23:02:37.366264+0000","last_active":"2024-02-21T23:02:43.595798+0000","last_peered":"2024-02-21T23:02:43.595798+0000","last_clean":"2024-02-21T23:02:43.595798+0000","last_became_active":"2024-02-21T23:02:37.365889+0000","last_became_peered":"2024-02-21T23:02:37.365889+0000","last_unstale":"2024-02-21T23:02:43.595798+0000","last_undegraded":"2024-02-21T23:02:43.595798+0000","last_fullsized":"2024-02-21T23:02:43.595798+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:07:23.801765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.419552+0000","last_change":"2024-02-21T23:02:42.495489+0000","last_active":"2024-02-21T23:02:43.419552+0000","last_peered":"2024-02-21T23:02:43.419552+0000","last_clean":"2024-02-21T23:02:43.419552+0000","last_became_active":"2024-02-21T23:02:42.495098+0000","last_became_peered":"2024-02-21T23:02:42.495098+0000","last_unstale":"2024-02-21T23:02:43.419552+0000","last_undegraded":"2024-02-21T23:02:43.419552+0000","last_fullsized":"2024-02-21T23:02:43.419552+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:10:21.610343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.596467+0000","last_change":"2024-02-21T23:02:42.500336+0000","last_active":"2024-02-21T23:02:43.596467+0000","last_peered":"2024-02-21T23:02:43.596467+0000","last_clean":"2024-02-21T23:02:43.596467+0000","last_became_active":"2024-02-21T23:02:42.498817+0000","last_became_peered":"2024-02-21T23:02:42.498817+0000","last_unstale":"2024-02-21T23:02:43.596467+0000","last_undegraded":"2024-02-21T23:02:43.596467+0000","last_fullsized":"2024-02-21T23:02:43.596467+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:35:32.034237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.594893+0000","last_change":"2024-02-21T23:02:38.378745+0000","last_active":"2024-02-21T23:02:43.594893+0000","last_peered":"2024-02-21T23:02:43.594893+0000","last_clean":"2024-02-21T23:02:43.594893+0000","last_became_active":"2024-02-21T23:02:38.378423+0000","last_became_peered":"2024-02-21T23:02:38.378423+0000","last_unstale":"2024-02-21T23:02:43.594893+0000","last_undegraded":"2024-02-21T23:02:43.594893+0000","last_fullsized":"2024-02-21T23:02:43.594893+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:30:37.791426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.600043+0000","last_change":"2024-02-21T23:02:37.352231+0000","last_active":"2024-02-21T23:02:43.600043+0000","last_peered":"2024-02-21T23:02:43.600043+0000","last_clean":"2024-02-21T23:02:43.600043+0000","last_became_active":"2024-02-21T23:02:37.351715+0000","last_became_peered":"2024-02-21T23:02:37.351715+0000","last_unstale":"2024-02-21T23:02:43.600043+0000","last_undegraded":"2024-02-21T23:02:43.600043+0000","last_fullsized":"2024-02-21T23:02:43.600043+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:07:23.801765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.595272+0000","last_change":"2024-02-21T23:02:42.449848+0000","last_active":"2024-02-21T23:02:43.595272+0000","last_peered":"2024-02-21T23:02:43.595272+0000","last_clean":"2024-02-21T23:02:43.595272+0000","last_became_active":"2024-02-21T23:02:42.449357+0000","last_became_peered":"2024-02-21T23:02:42.449357+0000","last_unstale":"2024-02-21T23:02:43.595272+0000","last_undegraded":"2024-02-21T23:02:43.595272+0000","last_fullsized":"2024-02-21T23:02:43.595272+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:49:24.584877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.419241+0000","last_change":"2024-02-21T23:02:38.368240+0000","last_active":"2024-02-21T23:02:43.419241+0000","last_peered":"2024-02-21T23:02:43.419241+0000","last_clean":"2024-02-21T23:02:43.419241+0000","last_became_active":"2024-02-21T23:02:38.367852+0000","last_became_peered":"2024-02-21T23:02:38.367852+0000","last_unstale":"2024-02-21T23:02:43.419241+0000","last_undegraded":"2024-02-21T23:02:43.419241+0000","last_fullsized":"2024-02-21T23:02:43.419241+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:30:37.791426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420051+0000","last_change":"2024-02-21T23:02:37.363129+0000","last_active":"2024-02-21T23:02:43.420051+0000","last_peered":"2024-02-21T23:02:43.420051+0000","last_clean":"2024-02-21T23:02:43.420051+0000","last_became_active":"2024-02-21T23:02:37.362829+0000","last_became_peered":"2024-02-21T23:02:37.362829+0000","last_unstale":"2024-02-21T23:02:43.420051+0000","last_undegraded":"2024-02-21T23:02:43.420051+0000","last_fullsized":"2024-02-21T23:02:43.420051+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:39:34.395881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.594502+0000","last_change":"2024-02-21T23:02:37.358568+0000","last_active":"2024-02-21T23:02:43.594502+0000","last_peered":"2024-02-21T23:02:43.594502+0000","last_clean":"2024-02-21T23:02:43.594502+0000","last_became_active":"2024-02-21T23:02:37.358238+0000","last_became_peered":"2024-02-21T23:02:37.358238+0000","last_unstale":"2024-02-21T23:02:43.594502+0000","last_undegraded":"2024-02-21T23:02:43.594502+0000","last_fullsized":"2024-02-21T23:02:43.594502+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:01:17.207079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.597289+0000","last_change":"2024-02-21T23:02:38.375982+0000","last_active":"2024-02-21T23:02:43.597289+0000","last_peered":"2024-02-21T23:02:43.597289+0000","last_clean":"2024-02-21T23:02:43.597289+0000","last_became_active":"2024-02-21T23:02:38.375504+0000","last_became_peered":"2024-02-21T23:02:38.375504+0000","last_unstale":"2024-02-21T23:02:43.597289+0000","last_undegraded":"2024-02-21T23:02:43.597289+0000","last_fullsized":"2024-02-21T23:02:43.597289+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:19:25.522975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.416324+0000","last_change":"2024-02-21T23:02:42.441484+0000","last_active":"2024-02-21T23:02:43.416324+0000","last_peered":"2024-02-21T23:02:43.416324+0000","last_clean":"2024-02-21T23:02:43.416324+0000","last_became_active":"2024-02-21T23:02:42.438094+0000","last_became_peered":"2024-02-21T23:02:42.438094+0000","last_unstale":"2024-02-21T23:02:43.416324+0000","last_undegraded":"2024-02-21T23:02:43.416324+0000","last_fullsized":"2024-02-21T23:02:43.416324+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:04:15.959906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420559+0000","last_change":"2024-02-21T23:02:37.365019+0000","last_active":"2024-02-21T23:02:43.420559+0000","last_peered":"2024-02-21T23:02:43.420559+0000","last_clean":"2024-02-21T23:02:43.420559+0000","last_became_active":"2024-02-21T23:02:37.364761+0000","last_became_peered":"2024-02-21T23:02:37.364761+0000","last_unstale":"2024-02-21T23:02:43.420559+0000","last_undegraded":"2024-02-21T23:02:43.420559+0000","last_fullsized":"2024-02-21T23:02:43.420559+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:36.299061+0000","last_clean_scrub_stamp":"2024-02-21T23:02:36.299061+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:01:17.207079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420052+0000","last_change":"2024-02-21T23:02:38.385160+0000","last_active":"2024-02-21T23:02:43.420052+0000","last_peered":"2024-02-21T23:02:43.420052+0000","last_clean":"2024-02-21T23:02:43.420052+0000","last_became_active":"2024-02-21T23:02:38.382830+0000","last_became_peered":"2024-02-21T23:02:38.382830+0000","last_unstale":"2024-02-21T23:02:43.420052+0000","last_undegraded":"2024-02-21T23:02:43.420052+0000","last_fullsized":"2024-02-21T23:02:43.420052+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:16:39.592708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.421894+0000","last_change":"2024-02-21T23:02:42.437091+0000","last_active":"2024-02-21T23:02:43.421894+0000","last_peered":"2024-02-21T23:02:43.421894+0000","last_clean":"2024-02-21T23:02:43.421894+0000","last_became_active":"2024-02-21T23:02:42.436674+0000","last_became_peered":"2024-02-21T23:02:42.436674+0000","last_unstale":"2024-02-21T23:02:43.421894+0000","last_undegraded":"2024-02-21T23:02:43.421894+0000","last_fullsized":"2024-02-21T23:02:43.421894+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:41.381459+0000","last_clean_scrub_stamp":"2024-02-21T23:02:41.381459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:16:29.140022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.594049+0000","last_change":"2024-02-21T23:02:38.379817+0000","last_active":"2024-02-21T23:02:43.594049+0000","last_peered":"2024-02-21T23:02:43.594049+0000","last_clean":"2024-02-21T23:02:43.594049+0000","last_became_active":"2024-02-21T23:02:38.379367+0000","last_became_peered":"2024-02-21T23:02:38.379367+0000","last_unstale":"2024-02-21T23:02:43.594049+0000","last_undegraded":"2024-02-21T23:02:43.594049+0000","last_fullsized":"2024-02-21T23:02:43.594049+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:04:11.892160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.415084+0000","last_change":"2024-02-21T23:02:38.386264+0000","last_active":"2024-02-21T23:02:43.415084+0000","last_peered":"2024-02-21T23:02:43.415084+0000","last_clean":"2024-02-21T23:02:43.415084+0000","last_became_active":"2024-02-21T23:02:38.385976+0000","last_became_peered":"2024-02-21T23:02:38.385976+0000","last_unstale":"2024-02-21T23:02:43.415084+0000","last_undegraded":"2024-02-21T23:02:43.415084+0000","last_fullsized":"2024-02-21T23:02:43.415084+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:09:48.394584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.600386+0000","last_change":"2024-02-21T23:02:38.394227+0000","last_active":"2024-02-21T23:02:43.600386+0000","last_peered":"2024-02-21T23:02:43.600386+0000","last_clean":"2024-02-21T23:02:43.600386+0000","last_became_active":"2024-02-21T23:02:38.393965+0000","last_became_peered":"2024-02-21T23:02:38.393965+0000","last_unstale":"2024-02-21T23:02:43.600386+0000","last_undegraded":"2024-02-21T23:02:43.600386+0000","last_fullsized":"2024-02-21T23:02:43.600386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:20:23.836303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.596804+0000","last_change":"2024-02-21T23:02:38.365717+0000","last_active":"2024-02-21T23:02:43.596804+0000","last_peered":"2024-02-21T23:02:43.596804+0000","last_clean":"2024-02-21T23:02:43.596804+0000","last_became_active":"2024-02-21T23:02:38.365375+0000","last_became_peered":"2024-02-21T23:02:38.365375+0000","last_unstale":"2024-02-21T23:02:43.596804+0000","last_undegraded":"2024-02-21T23:02:43.596804+0000","last_fullsized":"2024-02-21T23:02:43.596804+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:29:00.068426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.421488+0000","last_change":"2024-02-21T23:02:38.376971+0000","last_active":"2024-02-21T23:02:43.421488+0000","last_peered":"2024-02-21T23:02:43.421488+0000","last_clean":"2024-02-21T23:02:43.421488+0000","last_became_active":"2024-02-21T23:02:38.376565+0000","last_became_peered":"2024-02-21T23:02:38.376565+0000","last_unstale":"2024-02-21T23:02:43.421488+0000","last_undegraded":"2024-02-21T23:02:43.421488+0000","last_fullsized":"2024-02-21T23:02:43.421488+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:28:43.191300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.597197+0000","last_change":"2024-02-21T23:02:38.379350+0000","last_active":"2024-02-21T23:02:43.597197+0000","last_peered":"2024-02-21T23:02:43.597197+0000","last_clean":"2024-02-21T23:02:43.597197+0000","last_became_active":"2024-02-21T23:02:38.379030+0000","last_became_peered":"2024-02-21T23:02:38.379030+0000","last_unstale":"2024-02-21T23:02:43.597197+0000","last_undegraded":"2024-02-21T23:02:43.597197+0000","last_fullsized":"2024-02-21T23:02:43.597197+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:09:34.130536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.595248+0000","last_change":"2024-02-21T23:02:38.403472+0000","last_active":"2024-02-21T23:02:43.595248+0000","last_peered":"2024-02-21T23:02:43.595248+0000","last_clean":"2024-02-21T23:02:43.595248+0000","last_became_active":"2024-02-21T23:02:38.403213+0000","last_became_peered":"2024-02-21T23:02:38.403213+0000","last_unstale":"2024-02-21T23:02:43.595248+0000","last_undegraded":"2024-02-21T23:02:43.595248+0000","last_fullsized":"2024-02-21T23:02:43.595248+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:29:12.483912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420337+0000","last_change":"2024-02-21T23:02:38.377455+0000","last_active":"2024-02-21T23:02:43.420337+0000","last_peered":"2024-02-21T23:02:43.420337+0000","last_clean":"2024-02-21T23:02:43.420337+0000","last_became_active":"2024-02-21T23:02:38.376781+0000","last_became_peered":"2024-02-21T23:02:38.376781+0000","last_unstale":"2024-02-21T23:02:43.420337+0000","last_undegraded":"2024-02-21T23:02:43.420337+0000","last_fullsized":"2024-02-21T23:02:43.420337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:24:40.762404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.599882+0000","last_change":"2024-02-21T23:02:38.386554+0000","last_active":"2024-02-21T23:02:43.599882+0000","last_peered":"2024-02-21T23:02:43.599882+0000","last_clean":"2024-02-21T23:02:43.599882+0000","last_became_active":"2024-02-21T23:02:38.384500+0000","last_became_peered":"2024-02-21T23:02:38.384500+0000","last_unstale":"2024-02-21T23:02:43.599882+0000","last_undegraded":"2024-02-21T23:02:43.599882+0000","last_fullsized":"2024-02-21T23:02:43.599882+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:09:34.130536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420337+0000","last_change":"2024-02-21T23:02:38.383893+0000","last_active":"2024-02-21T23:02:43.420337+0000","last_peered":"2024-02-21T23:02:43.420337+0000","last_clean":"2024-02-21T23:02:43.420337+0000","last_became_active":"2024-02-21T23:02:38.383569+0000","last_became_peered":"2024-02-21T23:02:38.383569+0000","last_unstale":"2024-02-21T23:02:43.420337+0000","last_undegraded":"2024-02-21T23:02:43.420337+0000","last_fullsized":"2024-02-21T23:02:43.420337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:19:25.522975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.600302+0000","last_change":"2024-02-21T23:02:38.395013+0000","last_active":"2024-02-21T23:02:43.600302+0000","last_peered":"2024-02-21T23:02:43.600302+0000","last_clean":"2024-02-21T23:02:43.600302+0000","last_became_active":"2024-02-21T23:02:38.394670+0000","last_became_peered":"2024-02-21T23:02:38.394670+0000","last_unstale":"2024-02-21T23:02:43.600302+0000","last_undegraded":"2024-02-21T23:02:43.600302+0000","last_fullsized":"2024-02-21T23:02:43.600302+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:54:40.962238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.596541+0000","last_change":"2024-02-21T23:02:38.385293+0000","last_active":"2024-02-21T23:02:43.596541+0000","last_peered":"2024-02-21T23:02:43.596541+0000","last_clean":"2024-02-21T23:02:43.596541+0000","last_became_active":"2024-02-21T23:02:38.384589+0000","last_became_peered":"2024-02-21T23:02:38.384589+0000","last_unstale":"2024-02-21T23:02:43.596541+0000","last_undegraded":"2024-02-21T23:02:43.596541+0000","last_fullsized":"2024-02-21T23:02:43.596541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:10:48.186239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.595433+0000","last_change":"2024-02-21T23:02:38.385769+0000","last_active":"2024-02-21T23:02:43.595433+0000","last_peered":"2024-02-21T23:02:43.595433+0000","last_clean":"2024-02-21T23:02:43.595433+0000","last_became_active":"2024-02-21T23:02:38.385488+0000","last_became_peered":"2024-02-21T23:02:38.385488+0000","last_unstale":"2024-02-21T23:02:43.595433+0000","last_undegraded":"2024-02-21T23:02:43.595433+0000","last_fullsized":"2024-02-21T23:02:43.595433+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:29:00.068426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.598237+0000","last_change":"2024-02-21T23:02:38.396764+0000","last_active":"2024-02-21T23:02:43.598237+0000","last_peered":"2024-02-21T23:02:43.598237+0000","last_clean":"2024-02-21T23:02:43.598237+0000","last_became_active":"2024-02-21T23:02:38.396438+0000","last_became_peered":"2024-02-21T23:02:38.396438+0000","last_unstale":"2024-02-21T23:02:43.598237+0000","last_undegraded":"2024-02-21T23:02:43.598237+0000","last_fullsized":"2024-02-21T23:02:43.598237+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:06:59.357124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.416220+0000","last_change":"2024-02-21T23:02:38.387876+0000","last_active":"2024-02-21T23:02:43.416220+0000","last_peered":"2024-02-21T23:02:43.416220+0000","last_clean":"2024-02-21T23:02:43.416220+0000","last_became_active":"2024-02-21T23:02:38.383130+0000","last_became_peered":"2024-02-21T23:02:38.383130+0000","last_unstale":"2024-02-21T23:02:43.416220+0000","last_undegraded":"2024-02-21T23:02:43.416220+0000","last_fullsized":"2024-02-21T23:02:43.416220+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:12:46.637812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.601961+0000","last_change":"2024-02-21T23:02:38.403615+0000","last_active":"2024-02-21T23:02:43.601961+0000","last_peered":"2024-02-21T23:02:43.601961+0000","last_clean":"2024-02-21T23:02:43.601961+0000","last_became_active":"2024-02-21T23:02:38.402773+0000","last_became_peered":"2024-02-21T23:02:38.402773+0000","last_unstale":"2024-02-21T23:02:43.601961+0000","last_undegraded":"2024-02-21T23:02:43.601961+0000","last_fullsized":"2024-02-21T23:02:43.601961+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:12:26.063291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.599735+0000","last_change":"2024-02-21T23:02:38.384479+0000","last_active":"2024-02-21T23:02:43.599735+0000","last_peered":"2024-02-21T23:02:43.599735+0000","last_clean":"2024-02-21T23:02:43.599735+0000","last_became_active":"2024-02-21T23:02:38.383995+0000","last_became_peered":"2024-02-21T23:02:38.383995+0000","last_unstale":"2024-02-21T23:02:43.599735+0000","last_undegraded":"2024-02-21T23:02:43.599735+0000","last_fullsized":"2024-02-21T23:02:43.599735+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:28:43.191300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.595040+0000","last_change":"2024-02-21T23:02:38.390823+0000","last_active":"2024-02-21T23:02:43.595040+0000","last_peered":"2024-02-21T23:02:43.595040+0000","last_clean":"2024-02-21T23:02:43.595040+0000","last_became_active":"2024-02-21T23:02:38.390390+0000","last_became_peered":"2024-02-21T23:02:38.390390+0000","last_unstale":"2024-02-21T23:02:43.595040+0000","last_undegraded":"2024-02-21T23:02:43.595040+0000","last_fullsized":"2024-02-21T23:02:43.595040+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:29:00.068426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.595697+0000","last_change":"2024-02-21T23:02:38.380151+0000","last_active":"2024-02-21T23:02:43.595697+0000","last_peered":"2024-02-21T23:02:43.595697+0000","last_clean":"2024-02-21T23:02:43.595697+0000","last_became_active":"2024-02-21T23:02:38.379792+0000","last_became_peered":"2024-02-21T23:02:38.379792+0000","last_unstale":"2024-02-21T23:02:43.595697+0000","last_undegraded":"2024-02-21T23:02:43.595697+0000","last_fullsized":"2024-02-21T23:02:43.595697+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:39:18.270811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420027+0000","last_change":"2024-02-21T23:02:38.379804+0000","last_active":"2024-02-21T23:02:43.420027+0000","last_peered":"2024-02-21T23:02:43.420027+0000","last_clean":"2024-02-21T23:02:43.420027+0000","last_became_active":"2024-02-21T23:02:38.379461+0000","last_became_peered":"2024-02-21T23:02:38.379461+0000","last_unstale":"2024-02-21T23:02:43.420027+0000","last_undegraded":"2024-02-21T23:02:43.420027+0000","last_fullsized":"2024-02-21T23:02:43.420027+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:09:34.130536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.420766+0000","last_change":"2024-02-21T23:02:38.381806+0000","last_active":"2024-02-21T23:02:43.420766+0000","last_peered":"2024-02-21T23:02:43.420766+0000","last_clean":"2024-02-21T23:02:43.420766+0000","last_became_active":"2024-02-21T23:02:38.381475+0000","last_became_peered":"2024-02-21T23:02:38.381475+0000","last_unstale":"2024-02-21T23:02:43.420766+0000","last_undegraded":"2024-02-21T23:02:43.420766+0000","last_fullsized":"2024-02-21T23:02:43.420766+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:29:27.336333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.595120+0000","last_change":"2024-02-21T23:02:38.398122+0000","last_active":"2024-02-21T23:02:43.595120+0000","last_peered":"2024-02-21T23:02:43.595120+0000","last_clean":"2024-02-21T23:02:43.595120+0000","last_became_active":"2024-02-21T23:02:38.392050+0000","last_became_peered":"2024-02-21T23:02:38.392050+0000","last_unstale":"2024-02-21T23:02:43.595120+0000","last_undegraded":"2024-02-21T23:02:43.595120+0000","last_fullsized":"2024-02-21T23:02:43.595120+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:10:48.186239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.594981+0000","last_change":"2024-02-21T23:02:38.398482+0000","last_active":"2024-02-21T23:02:43.594981+0000","last_peered":"2024-02-21T23:02:43.594981+0000","last_clean":"2024-02-21T23:02:43.594981+0000","last_became_active":"2024-02-21T23:02:38.392232+0000","last_became_peered":"2024-02-21T23:02:38.392232+0000","last_unstale":"2024-02-21T23:02:43.594981+0000","last_undegraded":"2024-02-21T23:02:43.594981+0000","last_fullsized":"2024-02-21T23:02:43.594981+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:30:37.791426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.598787+0000","last_change":"2024-02-21T23:02:38.400825+0000","last_active":"2024-02-21T23:02:43.598787+0000","last_peered":"2024-02-21T23:02:43.598787+0000","last_clean":"2024-02-21T23:02:43.598787+0000","last_became_active":"2024-02-21T23:02:38.398964+0000","last_became_peered":"2024-02-21T23:02:38.398964+0000","last_unstale":"2024-02-21T23:02:43.598787+0000","last_undegraded":"2024-02-21T23:02:43.598787+0000","last_fullsized":"2024-02-21T23:02:43.598787+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:24:40.762404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.418962+0000","last_change":"2024-02-21T23:02:38.387131+0000","last_active":"2024-02-21T23:02:43.418962+0000","last_peered":"2024-02-21T23:02:43.418962+0000","last_clean":"2024-02-21T23:02:43.418962+0000","last_became_active":"2024-02-21T23:02:38.386835+0000","last_became_peered":"2024-02-21T23:02:38.386835+0000","last_unstale":"2024-02-21T23:02:43.418962+0000","last_undegraded":"2024-02-21T23:02:43.418962+0000","last_fullsized":"2024-02-21T23:02:43.418962+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:10:48.186239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.596722+0000","last_change":"2024-02-21T23:02:38.394292+0000","last_active":"2024-02-21T23:02:43.596722+0000","last_peered":"2024-02-21T23:02:43.596722+0000","last_clean":"2024-02-21T23:02:43.596722+0000","last_became_active":"2024-02-21T23:02:38.394006+0000","last_became_peered":"2024-02-21T23:02:38.394006+0000","last_unstale":"2024-02-21T23:02:43.596722+0000","last_undegraded":"2024-02-21T23:02:43.596722+0000","last_fullsized":"2024-02-21T23:02:43.596722+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:54:40.962238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.600192+0000","last_change":"2024-02-21T23:02:38.394994+0000","last_active":"2024-02-21T23:02:43.600192+0000","last_peered":"2024-02-21T23:02:43.600192+0000","last_clean":"2024-02-21T23:02:43.600192+0000","last_became_active":"2024-02-21T23:02:38.394693+0000","last_became_peered":"2024-02-21T23:02:38.394693+0000","last_unstale":"2024-02-21T23:02:43.600192+0000","last_undegraded":"2024-02-21T23:02:43.600192+0000","last_fullsized":"2024-02-21T23:02:43.600192+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:00:09.172059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.597595+0000","last_change":"2024-02-21T23:02:38.397147+0000","last_active":"2024-02-21T23:02:43.597595+0000","last_peered":"2024-02-21T23:02:43.597595+0000","last_clean":"2024-02-21T23:02:43.597595+0000","last_became_active":"2024-02-21T23:02:38.396787+0000","last_became_peered":"2024-02-21T23:02:38.396787+0000","last_unstale":"2024-02-21T23:02:43.597595+0000","last_undegraded":"2024-02-21T23:02:43.597595+0000","last_fullsized":"2024-02-21T23:02:43.597595+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:35:27.966492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.600709+0000","last_change":"2024-02-21T23:02:38.398132+0000","last_active":"2024-02-21T23:02:43.600709+0000","last_peered":"2024-02-21T23:02:43.600709+0000","last_clean":"2024-02-21T23:02:43.600709+0000","last_became_active":"2024-02-21T23:02:38.397847+0000","last_became_peered":"2024-02-21T23:02:38.397847+0000","last_unstale":"2024-02-21T23:02:43.600709+0000","last_undegraded":"2024-02-21T23:02:43.600709+0000","last_fullsized":"2024-02-21T23:02:43.600709+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:28:23.752647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.596548+0000","last_change":"2024-02-21T23:02:38.372038+0000","last_active":"2024-02-21T23:02:43.596548+0000","last_peered":"2024-02-21T23:02:43.596548+0000","last_clean":"2024-02-21T23:02:43.596548+0000","last_became_active":"2024-02-21T23:02:38.371731+0000","last_became_peered":"2024-02-21T23:02:38.371731+0000","last_unstale":"2024-02-21T23:02:43.596548+0000","last_undegraded":"2024-02-21T23:02:43.596548+0000","last_fullsized":"2024-02-21T23:02:43.596548+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:04:11.892160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.598184+0000","last_change":"2024-02-21T23:02:38.379292+0000","last_active":"2024-02-21T23:02:43.598184+0000","last_peered":"2024-02-21T23:02:43.598184+0000","last_clean":"2024-02-21T23:02:43.598184+0000","last_became_active":"2024-02-21T23:02:38.378980+0000","last_became_peered":"2024-02-21T23:02:38.378980+0000","last_unstale":"2024-02-21T23:02:43.598184+0000","last_undegraded":"2024-02-21T23:02:43.598184+0000","last_fullsized":"2024-02-21T23:02:43.598184+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:29:12.483912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-02-21T23:02:43.598604+0000","last_change":"2024-02-21T23:02:38.396837+0000","last_active":"2024-02-21T23:02:43.598604+0000","last_peered":"2024-02-21T23:02:43.598604+0000","last_clean":"2024-02-21T23:02:43.598604+0000","last_became_active":"2024-02-21T23:02:38.396502+0000","last_became_peered":"2024-02-21T23:02:38.396502+0000","last_unstale":"2024-02-21T23:02:43.598604+0000","last_undegraded":"2024-02-21T23:02:43.598604+0000","last_fullsized":"2024-02-21T23:02:43.598604+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:37.313713+0000","last_clean_scrub_stamp":"2024-02-21T23:02:37.313713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:54:16.651108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574858,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27340,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":323584,"data_stored":691206,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574859,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574858,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":258048,"data_stored":100346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574859,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":266240,"data_stored":100382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574859,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574860,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":253952,"data_stored":100256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574858,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":262144,"data_stored":100416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574858,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31508,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340332,"statfs":{"total":96636764160,"available":96604499968,"internally_reserved":0,"allocated":331776,"data_stored":691262,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-21T23:03:14.119 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-02-21T23:03:14.119 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-02-21T23:03:14.119 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-02-21T23:03:14.119 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-02-21T23:03:14.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.383+0000 7fbf9b200640 1 Processor -- start 2024-02-21T23:03:14.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.384+0000 7fbf9b200640 1 -- start start 2024-02-21T23:03:14.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.385+0000 7fbf9b200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbf9c0aff90 0x7fbf9c0c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:14.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.385+0000 7fbf9b200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fbf9c0c5720 0x7fbf9c15f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:14.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.385+0000 7fbf9b200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf9c0c74c0 con 0x7fbf9c06dbf0 2024-02-21T23:03:14.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.385+0000 7fbf9b200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf9c0c7270 con 0x7fbf9c0c5720 2024-02-21T23:03:14.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.385+0000 7fbf9b200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf9c158630 con 0x7fbf9c0aff90 2024-02-21T23:03:14.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.386+0000 7fbf9a800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fbf9c0c5720 0x7fbf9c15f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:14.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.386+0000 7fbf99e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbf9c0aff90 0x7fbf9c0c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:14.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.386+0000 7fbf99400640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fbf9c06dbf0 0x7fbf9c158ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.28:50218/0 (socket says 172.21.15.28:50218) 2024-02-21T23:03:14.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.386+0000 7fbf99400640 1 -- 172.21.15.28:0/3419626767 learned_addr learned my addr 172.21.15.28:0/3419626767 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:03:14.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.386+0000 7fbf99e00640 1 -- 172.21.15.28:0/3419626767 >> v2:172.21.15.81:3300/0 conn(0x7fbf9c0c5720 msgr2=0x7fbf9c15f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:14.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.386+0000 7fbf99e00640 1 --2- 172.21.15.28:0/3419626767 >> v2:172.21.15.81:3300/0 conn(0x7fbf9c0c5720 0x7fbf9c15f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:14.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.386+0000 7fbf98a00640 1 -- 172.21.15.28:0/3419626767 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1635231269 0 0) 0x7fbf9c0c74c0 con 0x7fbf9c06dbf0 2024-02-21T23:03:14.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.386+0000 7fbf99e00640 1 -- 172.21.15.28:0/3419626767 >> v1:172.21.15.81:6789/0 conn(0x7fbf9c06dbf0 legacy=0x7fbf9c158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:14.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.386+0000 7fbf99e00640 1 -- 172.21.15.28:0/3419626767 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf9c0c77a0 con 0x7fbf9c0aff90 2024-02-21T23:03:14.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.388+0000 7fbf99e00640 1 --2- 172.21.15.28:0/3419626767 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbf9c0aff90 0x7fbf9c0c51e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c0097a0 tx=0x7fbf8c02b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ceb1b8c730f0a307 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:14.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.388+0000 7fbf98a00640 1 -- 172.21.15.28:0/3419626767 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf8c035020 con 0x7fbf9c0aff90 2024-02-21T23:03:14.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.388+0000 7fbf98a00640 1 -- 172.21.15.28:0/3419626767 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf8c031810 con 0x7fbf9c0aff90 2024-02-21T23:03:14.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.388+0000 7fbf98a00640 1 -- 172.21.15.28:0/3419626767 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf8c031be0 con 0x7fbf9c0aff90 2024-02-21T23:03:14.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.389+0000 7fbf9b200640 1 -- 172.21.15.28:0/3419626767 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbf9c0aff90 msgr2=0x7fbf9c0c51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:14.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.389+0000 7fbf9b200640 1 --2- 172.21.15.28:0/3419626767 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbf9c0aff90 0x7fbf9c0c51e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c0097a0 tx=0x7fbf8c02b8b0 comp rx=0 tx=0).stop 2024-02-21T23:03:14.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.389+0000 7fbf9b200640 1 -- 172.21.15.28:0/3419626767 shutdown_connections 2024-02-21T23:03:14.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.389+0000 7fbf9b200640 1 --2- 172.21.15.28:0/3419626767 >> v2:172.21.15.81:3300/0 conn(0x7fbf9c0c5720 0x7fbf9c15f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:14.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.389+0000 7fbf9b200640 1 --2- 172.21.15.28:0/3419626767 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbf9c0aff90 0x7fbf9c0c51e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:14.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.389+0000 7fbf9b200640 1 -- 172.21.15.28:0/3419626767 >> 172.21.15.28:0/3419626767 conn(0x7fbf9c0ac370 msgr2=0x7fbf9c0ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:14.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.389+0000 7fbf9b200640 1 -- 172.21.15.28:0/3419626767 shutdown_connections 2024-02-21T23:03:14.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.389+0000 7fbf9b200640 1 -- 172.21.15.28:0/3419626767 wait complete. 2024-02-21T23:03:14.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.390+0000 7fbf9b200640 1 Processor -- start 2024-02-21T23:03:14.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.391+0000 7fbf9b200640 1 -- start start 2024-02-21T23:03:14.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.391+0000 7fbf9b200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf9c06dbf0 0x7fbf9c1e7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:14.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.391+0000 7fbf9b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf9c0aff90 0x7fbf9c156300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:14.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.391+0000 7fbf9b200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbf9c0c5720 0x7fbf9c1edb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:14.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.391+0000 7fbf9b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf9c0c8760 con 0x7fbf9c0aff90 2024-02-21T23:03:14.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.391+0000 7fbf9b200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf9c0be0c0 con 0x7fbf9c06dbf0 2024-02-21T23:03:14.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.391+0000 7fbf9b200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf9c0c88e0 con 0x7fbf9c0c5720 2024-02-21T23:03:14.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.392+0000 7fbf99400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf9c0aff90 0x7fbf9c156300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:14.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.392+0000 7fbf99400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf9c0aff90 0x7fbf9c156300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46306/0 (socket says 172.21.15.28:46306) 2024-02-21T23:03:14.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.392+0000 7fbf99400640 1 -- 172.21.15.28:0/3242145403 learned_addr learned my addr 172.21.15.28:0/3242145403 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:14.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.392+0000 7fbf99e00640 1 --2- 172.21.15.28:0/3242145403 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf9c06dbf0 0x7fbf9c1e7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:14.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.392+0000 7fbf9a800640 1 --2- 172.21.15.28:0/3242145403 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbf9c0c5720 0x7fbf9c1edb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:14.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.392+0000 7fbf99400640 1 -- 172.21.15.28:0/3242145403 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbf9c0c5720 msgr2=0x7fbf9c1edb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:14.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.392+0000 7fbf99400640 1 --2- 172.21.15.28:0/3242145403 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbf9c0c5720 0x7fbf9c1edb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:14.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.392+0000 7fbf99400640 1 -- 172.21.15.28:0/3242145403 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf9c06dbf0 msgr2=0x7fbf9c1e7aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:14.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.392+0000 7fbf99400640 1 --2- 172.21.15.28:0/3242145403 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf9c06dbf0 0x7fbf9c1e7aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:14.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.392+0000 7fbf99400640 1 -- 172.21.15.28:0/3242145403 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf9c0c2ef0 con 0x7fbf9c0aff90 2024-02-21T23:03:14.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.392+0000 7fbf9a800640 1 --2- 172.21.15.28:0/3242145403 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbf9c0c5720 0x7fbf9c1edb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:03:14.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.392+0000 7fbf99400640 1 --2- 172.21.15.28:0/3242145403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf9c0aff90 0x7fbf9c156300 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fbf8400c8c0 tx=0x7fbf8400cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:14.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.393+0000 7fbf92000640 1 -- 172.21.15.28:0/3242145403 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf84018070 con 0x7fbf9c0aff90 2024-02-21T23:03:14.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.393+0000 7fbf92000640 1 -- 172.21.15.28:0/3242145403 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf8400a5e0 con 0x7fbf9c0aff90 2024-02-21T23:03:14.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.393+0000 7fbf9b200640 1 -- 172.21.15.28:0/3242145403 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf9c1e7fe0 con 0x7fbf9c0aff90 2024-02-21T23:03:14.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.393+0000 7fbf92000640 1 -- 172.21.15.28:0/3242145403 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf8400a9d0 con 0x7fbf9c0aff90 2024-02-21T23:03:14.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.393+0000 7fbf9b200640 1 -- 172.21.15.28:0/3242145403 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf9c0c44a0 con 0x7fbf9c0aff90 2024-02-21T23:03:14.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.394+0000 7fbf9b200640 1 -- 172.21.15.28:0/3242145403 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf9c04f290 con 0x7fbf9c0aff90 2024-02-21T23:03:14.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.395+0000 7fbf92000640 1 -- 172.21.15.28:0/3242145403 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbf8401b070 con 0x7fbf9c0aff90 2024-02-21T23:03:14.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.396+0000 7fbf92000640 1 --2- 172.21.15.28:0/3242145403 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbf7806b500 0x7fbf7806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:14.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.396+0000 7fbf92000640 1 -- 172.21.15.28:0/3242145403 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fbf84089b70 con 0x7fbf9c0aff90 2024-02-21T23:03:14.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.396+0000 7fbf99e00640 1 --2- 172.21.15.28:0/3242145403 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbf7806b500 0x7fbf7806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:14.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.398+0000 7fbf99e00640 1 --2- 172.21.15.28:0/3242145403 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbf7806b500 0x7fbf7806d9c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c036000 tx=0x7fbf8c03c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:14.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.400+0000 7fbf92000640 1 -- 172.21.15.28:0/3242145403 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fbf840586a0 con 0x7fbf9c0aff90 2024-02-21T23:03:14.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.643+0000 7fbf9b200640 1 -- 172.21.15.28:0/3242145403 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fbf9c06d9d0 con 0x7fbf9c0aff90 2024-02-21T23:03:14.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.643+0000 7fbf92000640 1 -- 172.21.15.28:0/3242145403 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fbf84057fa0 con 0x7fbf9c0aff90 2024-02-21T23:03:14.633 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:03:14.633 INFO:teuthology.orchestra.run.smithi028.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-02-21T23:03:14.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.646+0000 7fbf9b200640 1 -- 172.21.15.28:0/3242145403 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbf7806b500 msgr2=0x7fbf7806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:14.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.646+0000 7fbf9b200640 1 --2- 172.21.15.28:0/3242145403 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbf7806b500 0x7fbf7806d9c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c036000 tx=0x7fbf8c03c040 comp rx=0 tx=0).stop 2024-02-21T23:03:14.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.646+0000 7fbf9b200640 1 -- 172.21.15.28:0/3242145403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf9c0aff90 msgr2=0x7fbf9c156300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:14.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.646+0000 7fbf9b200640 1 --2- 172.21.15.28:0/3242145403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf9c0aff90 0x7fbf9c156300 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fbf8400c8c0 tx=0x7fbf8400cd80 comp rx=0 tx=0).stop 2024-02-21T23:03:14.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.646+0000 7fbf9b200640 1 -- 172.21.15.28:0/3242145403 shutdown_connections 2024-02-21T23:03:14.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.646+0000 7fbf9b200640 1 --2- 172.21.15.28:0/3242145403 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbf7806b500 0x7fbf7806d9c0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:14.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.646+0000 7fbf9b200640 1 --2- 172.21.15.28:0/3242145403 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbf9c0c5720 0x7fbf9c1edb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:14.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.646+0000 7fbf9b200640 1 --2- 172.21.15.28:0/3242145403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf9c0aff90 0x7fbf9c156300 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:14.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.646+0000 7fbf9b200640 1 --2- 172.21.15.28:0/3242145403 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf9c06dbf0 0x7fbf9c1e7aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:14.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.646+0000 7fbf9b200640 1 -- 172.21.15.28:0/3242145403 >> 172.21.15.28:0/3242145403 conn(0x7fbf9c0ac370 msgr2=0x7fbf9c0ac750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:14.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.646+0000 7fbf9b200640 1 -- 172.21.15.28:0/3242145403 shutdown_connections 2024-02-21T23:03:14.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.646+0000 7fbf9b200640 1 -- 172.21.15.28:0/3242145403 wait complete. 2024-02-21T23:03:14.654 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-02-21T23:03:14.655 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:03:14.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.885+0000 7f24a9e00640 1 Processor -- start 2024-02-21T23:03:14.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.885+0000 7f24a9e00640 1 -- start start 2024-02-21T23:03:14.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.886+0000 7f24a9e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f24a4150450 0x7f24a40a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:14.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.886+0000 7f24a9e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f24a4156af0 0x7f24a406dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:14.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.886+0000 7f24a9e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24a40c2760 con 0x7f24a406ee40 2024-02-21T23:03:14.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.886+0000 7f24a9e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24a40c22c0 con 0x7f24a4156af0 2024-02-21T23:03:14.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.886+0000 7f24a9e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24a4156660 con 0x7f24a4150450 2024-02-21T23:03:14.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.887+0000 7f24a9400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f24a4156af0 0x7f24a406dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:14.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.887+0000 7f24a8a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f24a4150450 0x7f24a40a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:14.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.887+0000 7f24a9400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f24a4156af0 0x7f24a406dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46322/0 (socket says 172.21.15.28:46322) 2024-02-21T23:03:14.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.887+0000 7f24a3e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f24a406ee40 0x7f24a41536f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:48706/0 (socket says 172.21.15.28:48706) 2024-02-21T23:03:14.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.887+0000 7f24a9400640 1 -- 172.21.15.28:0/3664494038 learned_addr learned my addr 172.21.15.28:0/3664494038 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:14.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.887+0000 7f24a9400640 1 -- 172.21.15.28:0/3664494038 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f24a4150450 msgr2=0x7f24a40a5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:14.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.887+0000 7f24a9400640 1 --2- 172.21.15.28:0/3664494038 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f24a4150450 0x7f24a40a5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:14.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.888+0000 7f24a9400640 1 -- 172.21.15.28:0/3664494038 >> v1:172.21.15.28:6789/0 conn(0x7f24a406ee40 legacy=0x7f24a41536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:14.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.888+0000 7f24a9400640 1 -- 172.21.15.28:0/3664494038 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24a4156310 con 0x7f24a4156af0 2024-02-21T23:03:14.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.888+0000 7f24a3400640 1 -- 172.21.15.28:0/3664494038 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 190298327 0 0) 0x7f24a40c2760 con 0x7f24a406ee40 2024-02-21T23:03:14.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.888+0000 7f24a8a00640 1 --2- 172.21.15.28:0/3664494038 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f24a4150450 0x7f24a40a5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-21T23:03:14.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.889+0000 7f24a9400640 1 --2- 172.21.15.28:0/3664494038 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24a4156af0 0x7f24a406dff0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f249c009e50 tx=0x7f249c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f2c4ce6ee2933df server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:14.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.889+0000 7f24a3400640 1 -- 172.21.15.28:0/3664494038 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f249c02cb00 con 0x7f24a4156af0 2024-02-21T23:03:14.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.889+0000 7f24a3400640 1 -- 172.21.15.28:0/3664494038 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f249c02cca0 con 0x7f24a4156af0 2024-02-21T23:03:14.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.889+0000 7f24a3400640 1 -- 172.21.15.28:0/3664494038 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f249c034610 con 0x7f24a4156af0 2024-02-21T23:03:14.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.890+0000 7f24a9e00640 1 -- 172.21.15.28:0/3664494038 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24a4156af0 msgr2=0x7f24a406dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:14.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.890+0000 7f24a9e00640 1 --2- 172.21.15.28:0/3664494038 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24a4156af0 0x7f24a406dff0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f249c009e50 tx=0x7f249c02bf60 comp rx=0 tx=0).stop 2024-02-21T23:03:14.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.890+0000 7f24a9e00640 1 -- 172.21.15.28:0/3664494038 shutdown_connections 2024-02-21T23:03:14.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.890+0000 7f24a9e00640 1 --2- 172.21.15.28:0/3664494038 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24a4156af0 0x7f24a406dff0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:14.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.890+0000 7f24a9e00640 1 --2- 172.21.15.28:0/3664494038 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f24a4150450 0x7f24a40a5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:14.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.890+0000 7f24a9e00640 1 -- 172.21.15.28:0/3664494038 >> 172.21.15.28:0/3664494038 conn(0x7f24a406b630 msgr2=0x7f24a40c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:14.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.890+0000 7f24a9e00640 1 -- 172.21.15.28:0/3664494038 shutdown_connections 2024-02-21T23:03:14.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.890+0000 7f24a9e00640 1 -- 172.21.15.28:0/3664494038 wait complete. 2024-02-21T23:03:14.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.892+0000 7f24a9e00640 1 Processor -- start 2024-02-21T23:03:14.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.892+0000 7f24a9e00640 1 -- start start 2024-02-21T23:03:14.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.893+0000 7f24a9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24a406ee40 0x7f24a4151290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:14.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.893+0000 7f24a9e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f24a4150450 0x7f24a414bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:14.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.893+0000 7f24a9e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f24a4156af0 0x7f24a4145630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:14.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.893+0000 7f24a9e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24a4156f80 con 0x7f24a406ee40 2024-02-21T23:03:14.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.893+0000 7f24a9e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24a40c2cb0 con 0x7f24a4150450 2024-02-21T23:03:14.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.893+0000 7f24a9e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24a4157100 con 0x7f24a4156af0 2024-02-21T23:03:14.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.893+0000 7f24a8a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24a406ee40 0x7f24a4151290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:14.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.893+0000 7f24a8a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24a406ee40 0x7f24a4151290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45872/0 (socket says 172.21.15.28:45872) 2024-02-21T23:03:14.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.894+0000 7f24a8a00640 1 -- 172.21.15.28:0/3765736471 learned_addr learned my addr 172.21.15.28:0/3765736471 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:14.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.894+0000 7f24a3e00640 1 --2- 172.21.15.28:0/3765736471 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f24a4150450 0x7f24a414bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:14.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.894+0000 7f24a9400640 1 --2- 172.21.15.28:0/3765736471 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f24a4156af0 0x7f24a4145630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:14.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.894+0000 7f24a8a00640 1 -- 172.21.15.28:0/3765736471 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f24a4156af0 msgr2=0x7f24a4145630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:14.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.894+0000 7f24a8a00640 1 --2- 172.21.15.28:0/3765736471 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f24a4156af0 0x7f24a4145630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:14.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.894+0000 7f24a8a00640 1 -- 172.21.15.28:0/3765736471 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f24a4150450 msgr2=0x7f24a414bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:14.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.894+0000 7f24a8a00640 1 --2- 172.21.15.28:0/3765736471 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f24a4150450 0x7f24a414bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:14.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.894+0000 7f24a8a00640 1 -- 172.21.15.28:0/3765736471 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24a4154ae0 con 0x7f24a406ee40 2024-02-21T23:03:14.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.894+0000 7f24a8a00640 1 --2- 172.21.15.28:0/3765736471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24a406ee40 0x7f24a4151290 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f249800ebe0 tx=0x7f249800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:14.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.894+0000 7f24a0c00640 1 -- 172.21.15.28:0/3765736471 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2498017070 con 0x7f24a406ee40 2024-02-21T23:03:14.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.895+0000 7f24a0c00640 1 -- 172.21.15.28:0/3765736471 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f249800cc30 con 0x7f24a406ee40 2024-02-21T23:03:14.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.895+0000 7f24a0c00640 1 -- 172.21.15.28:0/3765736471 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2498013470 con 0x7f24a406ee40 2024-02-21T23:03:14.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.895+0000 7f24a9e00640 1 -- 172.21.15.28:0/3765736471 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24a4154dc0 con 0x7f24a406ee40 2024-02-21T23:03:14.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.895+0000 7f24a9e00640 1 -- 172.21.15.28:0/3765736471 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24a4154380 con 0x7f24a406ee40 2024-02-21T23:03:14.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.896+0000 7f24a9e00640 1 -- 172.21.15.28:0/3765736471 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24a4153e90 con 0x7f24a406ee40 2024-02-21T23:03:14.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.897+0000 7f24a0c00640 1 -- 172.21.15.28:0/3765736471 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f249801a070 con 0x7f24a406ee40 2024-02-21T23:03:14.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.897+0000 7f24a0c00640 1 --2- 172.21.15.28:0/3765736471 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f248406b4d0 0x7f248406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:14.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.898+0000 7f24a0c00640 1 -- 172.21.15.28:0/3765736471 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f249808a840 con 0x7f24a406ee40 2024-02-21T23:03:14.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.898+0000 7f24a3e00640 1 --2- 172.21.15.28:0/3765736471 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f248406b4d0 0x7f248406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:14.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.899+0000 7f24a3e00640 1 --2- 172.21.15.28:0/3765736471 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f248406b4d0 0x7f248406d990 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f2490007fd0 tx=0x7f2490009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:14.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:14.902+0000 7f24a0c00640 1 -- 172.21.15.28:0/3765736471 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f2498059370 con 0x7f24a406ee40 2024-02-21T23:03:15.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.105+0000 7f24a9e00640 1 -- 172.21.15.28:0/3765736471 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f24a404f290 con 0x7f24a406ee40 2024-02-21T23:03:15.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.108+0000 7f24a0c00640 1 -- 172.21.15.28:0/3765736471 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18454 (secure 0 0 0) 0x7f2498058c70 con 0x7f24a406ee40 2024-02-21T23:03:15.099 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:03:15.099 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":24,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:02:42.900286+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:02:35.795158+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-21T23:02:36.840110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-02-21T23:02:40.441388+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:03:15.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.111+0000 7f24a9e00640 1 -- 172.21.15.28:0/3765736471 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f248406b4d0 msgr2=0x7f248406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:15.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.111+0000 7f24a9e00640 1 --2- 172.21.15.28:0/3765736471 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f248406b4d0 0x7f248406d990 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f2490007fd0 tx=0x7f2490009040 comp rx=0 tx=0).stop 2024-02-21T23:03:15.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.111+0000 7f24a9e00640 1 -- 172.21.15.28:0/3765736471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24a406ee40 msgr2=0x7f24a4151290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:15.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.111+0000 7f24a9e00640 1 --2- 172.21.15.28:0/3765736471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24a406ee40 0x7f24a4151290 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f249800ebe0 tx=0x7f249800c520 comp rx=0 tx=0).stop 2024-02-21T23:03:15.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.111+0000 7f24a9e00640 1 -- 172.21.15.28:0/3765736471 shutdown_connections 2024-02-21T23:03:15.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.111+0000 7f24a9e00640 1 --2- 172.21.15.28:0/3765736471 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f248406b4d0 0x7f248406d990 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:15.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.111+0000 7f24a9e00640 1 --2- 172.21.15.28:0/3765736471 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f24a4156af0 0x7f24a4145630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:15.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.111+0000 7f24a9e00640 1 --2- 172.21.15.28:0/3765736471 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f24a4150450 0x7f24a414bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:15.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.111+0000 7f24a9e00640 1 --2- 172.21.15.28:0/3765736471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24a406ee40 0x7f24a4151290 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:15.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.111+0000 7f24a9e00640 1 -- 172.21.15.28:0/3765736471 >> 172.21.15.28:0/3765736471 conn(0x7f24a406b630 msgr2=0x7f24a40a69b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:15.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.112+0000 7f24a9e00640 1 -- 172.21.15.28:0/3765736471 shutdown_connections 2024-02-21T23:03:15.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.112+0000 7f24a9e00640 1 -- 172.21.15.28:0/3765736471 wait complete. 2024-02-21T23:03:15.121 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-21T23:02:30.728329+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-21T23:02:35.795158+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-02-21T23:02:36.840110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-02-21T23:02:40.441388+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-02-21T23:03:15.121 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-21T23:03:15.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.359+0000 7fcffbe00640 1 Processor -- start 2024-02-21T23:03:15.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.360+0000 7fcffbe00640 1 -- start start 2024-02-21T23:03:15.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.361+0000 7fcffbe00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcffc152650 0x7fcffc0c8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:15.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.361+0000 7fcffbe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcffc06daa0 0x7fcffc0c7160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:15.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.361+0000 7fcffbe00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcffc158920 con 0x7fcffc0c76a0 2024-02-21T23:03:15.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.361+0000 7fcffbe00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcffc1586d0 con 0x7fcffc06daa0 2024-02-21T23:03:15.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.361+0000 7fcffbe00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcffc158c00 con 0x7fcffc152650 2024-02-21T23:03:15.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.362+0000 7fcffa000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcffc06daa0 0x7fcffc0c7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:15.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.362+0000 7fcffa000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcffc06daa0 0x7fcffc0c7160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45890/0 (socket says 172.21.15.28:45890) 2024-02-21T23:03:15.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.362+0000 7fcffa000640 1 -- 172.21.15.28:0/217498784 learned_addr learned my addr 172.21.15.28:0/217498784 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:15.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.362+0000 7fcffaa00640 1 --2- 172.21.15.28:0/217498784 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcffc152650 0x7fcffc0c8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:15.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.362+0000 7fcffa000640 1 -- 172.21.15.28:0/217498784 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcffc152650 msgr2=0x7fcffc0c8ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:15.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.362+0000 7fcffa000640 1 --2- 172.21.15.28:0/217498784 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcffc152650 0x7fcffc0c8ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:15.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.362+0000 7fcffa000640 1 -- 172.21.15.28:0/217498784 >> v1:172.21.15.81:6789/0 conn(0x7fcffc0c76a0 legacy=0x7fcffc1550e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:15.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.362+0000 7fcffa000640 1 -- 172.21.15.28:0/217498784 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcffc158ee0 con 0x7fcffc06daa0 2024-02-21T23:03:15.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.364+0000 7fcffa000640 1 --2- 172.21.15.28:0/217498784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcffc06daa0 0x7fcffc0c7160 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8009870 tx=0x7fcfe802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d00d9553176a727 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:15.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.364+0000 7fcff9600640 1 -- 172.21.15.28:0/217498784 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfe8035020 con 0x7fcffc06daa0 2024-02-21T23:03:15.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.364+0000 7fcff9600640 1 -- 172.21.15.28:0/217498784 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfe8031640 con 0x7fcffc06daa0 2024-02-21T23:03:15.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.364+0000 7fcff9600640 1 -- 172.21.15.28:0/217498784 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfe802d040 con 0x7fcffc06daa0 2024-02-21T23:03:15.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.365+0000 7fcffbe00640 1 -- 172.21.15.28:0/217498784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcffc06daa0 msgr2=0x7fcffc0c7160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:15.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.365+0000 7fcffbe00640 1 --2- 172.21.15.28:0/217498784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcffc06daa0 0x7fcffc0c7160 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fcfe8009870 tx=0x7fcfe802b8f0 comp rx=0 tx=0).stop 2024-02-21T23:03:15.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.365+0000 7fcffbe00640 1 -- 172.21.15.28:0/217498784 shutdown_connections 2024-02-21T23:03:15.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.365+0000 7fcffbe00640 1 --2- 172.21.15.28:0/217498784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcffc06daa0 0x7fcffc0c7160 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:15.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.365+0000 7fcffbe00640 1 --2- 172.21.15.28:0/217498784 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcffc152650 0x7fcffc0c8ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:15.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.365+0000 7fcffbe00640 1 -- 172.21.15.28:0/217498784 >> 172.21.15.28:0/217498784 conn(0x7fcffc072ce0 msgr2=0x7fcffc0ac340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:15.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.365+0000 7fcffbe00640 1 -- 172.21.15.28:0/217498784 shutdown_connections 2024-02-21T23:03:15.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.365+0000 7fcffbe00640 1 -- 172.21.15.28:0/217498784 wait complete. 2024-02-21T23:03:15.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.367+0000 7fcffbe00640 1 Processor -- start 2024-02-21T23:03:15.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.367+0000 7fcffbe00640 1 -- start start 2024-02-21T23:03:15.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.368+0000 7fcffbe00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcffc06daa0 0x7fcffc1eb930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:15.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.368+0000 7fcffbe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcffc0c76a0 0x7fcffc152bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:15.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.368+0000 7fcffbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcffc152650 0x7fcffc0c34d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:15.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.368+0000 7fcffbe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcffc152a30 con 0x7fcffc152650 2024-02-21T23:03:15.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.368+0000 7fcffbe00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcffc0bdfb0 con 0x7fcffc0c76a0 2024-02-21T23:03:15.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.368+0000 7fcffbe00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcffc1557f0 con 0x7fcffc06daa0 2024-02-21T23:03:15.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.368+0000 7fcffb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcffc152650 0x7fcffc0c34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:15.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.369+0000 7fcffb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcffc152650 0x7fcffc0c34d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45902/0 (socket says 172.21.15.28:45902) 2024-02-21T23:03:15.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.369+0000 7fcffb400640 1 -- 172.21.15.28:0/2684649401 learned_addr learned my addr 172.21.15.28:0/2684649401 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:15.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.369+0000 7fcffaa00640 1 --2- 172.21.15.28:0/2684649401 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcffc06daa0 0x7fcffc1eb930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:15.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.369+0000 7fcffa000640 1 --2- 172.21.15.28:0/2684649401 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcffc0c76a0 0x7fcffc152bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:15.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.369+0000 7fcffb400640 1 -- 172.21.15.28:0/2684649401 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcffc06daa0 msgr2=0x7fcffc1eb930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:15.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.369+0000 7fcffb400640 1 --2- 172.21.15.28:0/2684649401 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcffc06daa0 0x7fcffc1eb930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:15.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.369+0000 7fcffb400640 1 -- 172.21.15.28:0/2684649401 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcffc0c76a0 msgr2=0x7fcffc152bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:15.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.369+0000 7fcffb400640 1 --2- 172.21.15.28:0/2684649401 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcffc0c76a0 0x7fcffc152bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:15.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.369+0000 7fcffb400640 1 -- 172.21.15.28:0/2684649401 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcffc1eae70 con 0x7fcffc152650 2024-02-21T23:03:15.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.369+0000 7fcffa000640 1 --2- 172.21.15.28:0/2684649401 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcffc0c76a0 0x7fcffc152bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:03:15.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.369+0000 7fcffb400640 1 --2- 172.21.15.28:0/2684649401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcffc152650 0x7fcffc0c34d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fcff400c7a0 tx=0x7fcff400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:15.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.370+0000 7fcff3400640 1 -- 172.21.15.28:0/2684649401 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff4003b70 con 0x7fcffc152650 2024-02-21T23:03:15.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.370+0000 7fcff3400640 1 -- 172.21.15.28:0/2684649401 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcff4003d30 con 0x7fcffc152650 2024-02-21T23:03:15.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.370+0000 7fcff3400640 1 -- 172.21.15.28:0/2684649401 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff400aa90 con 0x7fcffc152650 2024-02-21T23:03:15.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.370+0000 7fcffbe00640 1 -- 172.21.15.28:0/2684649401 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcffc0c4390 con 0x7fcffc152650 2024-02-21T23:03:15.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.370+0000 7fcffbe00640 1 -- 172.21.15.28:0/2684649401 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcffc1eb130 con 0x7fcffc152650 2024-02-21T23:03:15.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.371+0000 7fcffbe00640 1 -- 172.21.15.28:0/2684649401 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcffc04f290 con 0x7fcffc152650 2024-02-21T23:03:15.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.372+0000 7fcff3400640 1 -- 172.21.15.28:0/2684649401 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcff400ac30 con 0x7fcffc152650 2024-02-21T23:03:15.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.373+0000 7fcff3400640 1 --2- 172.21.15.28:0/2684649401 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcfdc06b500 0x7fcfdc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:15.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.373+0000 7fcff3400640 1 -- 172.21.15.28:0/2684649401 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fcff408a480 con 0x7fcffc152650 2024-02-21T23:03:15.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.373+0000 7fcffaa00640 1 --2- 172.21.15.28:0/2684649401 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcfdc06b500 0x7fcfdc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:15.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.375+0000 7fcffaa00640 1 --2- 172.21.15.28:0/2684649401 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcfdc06b500 0x7fcfdc06d9c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fcfec009a30 tx=0x7fcfec008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:15.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.379+0000 7fcff3400640 1 -- 172.21.15.28:0/2684649401 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fcff4058fb0 con 0x7fcffc152650 2024-02-21T23:03:15.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.581+0000 7fcffbe00640 1 -- 172.21.15.28:0/2684649401 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcffc156080 con 0x7fcffc152650 2024-02-21T23:03:15.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.582+0000 7fcff3400640 1 -- 172.21.15.28:0/2684649401 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcff40588b0 con 0x7fcffc152650 2024-02-21T23:03:15.572 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-02-21T23:03:15.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.585+0000 7fcffbe00640 1 -- 172.21.15.28:0/2684649401 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcfdc06b500 msgr2=0x7fcfdc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:15.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.585+0000 7fcffbe00640 1 --2- 172.21.15.28:0/2684649401 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcfdc06b500 0x7fcfdc06d9c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fcfec009a30 tx=0x7fcfec008040 comp rx=0 tx=0).stop 2024-02-21T23:03:15.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.585+0000 7fcffbe00640 1 -- 172.21.15.28:0/2684649401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcffc152650 msgr2=0x7fcffc0c34d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:15.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.585+0000 7fcffbe00640 1 --2- 172.21.15.28:0/2684649401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcffc152650 0x7fcffc0c34d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fcff400c7a0 tx=0x7fcff400cc60 comp rx=0 tx=0).stop 2024-02-21T23:03:15.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.585+0000 7fcffbe00640 1 -- 172.21.15.28:0/2684649401 shutdown_connections 2024-02-21T23:03:15.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.585+0000 7fcffbe00640 1 --2- 172.21.15.28:0/2684649401 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcfdc06b500 0x7fcfdc06d9c0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:15.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.585+0000 7fcffbe00640 1 --2- 172.21.15.28:0/2684649401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcffc152650 0x7fcffc0c34d0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:15.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.585+0000 7fcffbe00640 1 --2- 172.21.15.28:0/2684649401 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcffc0c76a0 0x7fcffc152bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:15.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.585+0000 7fcffbe00640 1 --2- 172.21.15.28:0/2684649401 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcffc06daa0 0x7fcffc1eb930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:15.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.585+0000 7fcffbe00640 1 -- 172.21.15.28:0/2684649401 >> 172.21.15.28:0/2684649401 conn(0x7fcffc072ce0 msgr2=0x7fcffc0c5380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:15.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.585+0000 7fcffbe00640 1 -- 172.21.15.28:0/2684649401 shutdown_connections 2024-02-21T23:03:15.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.585+0000 7fcffbe00640 1 -- 172.21.15.28:0/2684649401 wait complete. 2024-02-21T23:03:15.593 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-21T23:03:15.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.827+0000 7fb2d9c00640 1 Processor -- start 2024-02-21T23:03:15.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.827+0000 7fb2d9c00640 1 -- start start 2024-02-21T23:03:15.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.829+0000 7fb2d9c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb2d4155db0 0x7fb2d415d480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:15.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.829+0000 7fb2d9c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2d40c5b80 con 0x7fb2d40c5df0 2024-02-21T23:03:15.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.829+0000 7fb2d9c00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2d4152950 con 0x7fb2d414f000 2024-02-21T23:03:15.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.829+0000 7fb2d9c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d40c4fe0 con 0x7fb2d4155db0 2024-02-21T23:03:15.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.829+0000 7fb2d3400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb2d40c5df0 0x7fb2d41556a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39572/0 (socket says 172.21.15.28:39572) 2024-02-21T23:03:15.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.829+0000 7fb2d3400640 1 -- 172.21.15.28:0/4269729517 learned_addr learned my addr 172.21.15.28:0/4269729517 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:03:15.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.829+0000 7fb2d9200640 1 --2- 172.21.15.28:0/4269729517 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb2d4155db0 0x7fb2d415d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:15.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.829+0000 7fb2d2a00640 1 -- 172.21.15.28:0/4269729517 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2707061786 0 0) 0x7fb2d40c5b80 con 0x7fb2d40c5df0 2024-02-21T23:03:15.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.829+0000 7fb2d2a00640 1 -- 172.21.15.28:0/4269729517 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2b8003520 con 0x7fb2d40c5df0 2024-02-21T23:03:15.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.830+0000 7fb2d2a00640 1 -- 172.21.15.28:0/4269729517 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1836543472 0 0) 0x7fb2b8003520 con 0x7fb2d40c5df0 2024-02-21T23:03:15.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.830+0000 7fb2d2a00640 1 -- 172.21.15.28:0/4269729517 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb2d4155db0 msgr2=0x7fb2d415d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:15.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.830+0000 7fb2d2a00640 1 --2- 172.21.15.28:0/4269729517 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb2d4155db0 0x7fb2d415d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:15.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.830+0000 7fb2d2a00640 1 -- 172.21.15.28:0/4269729517 >> v1:172.21.15.81:6789/0 conn(0x7fb2d414f000 legacy=0x7fb2d406ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:15.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.830+0000 7fb2d2a00640 1 -- 172.21.15.28:0/4269729517 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2d40c58a0 con 0x7fb2d40c5df0 2024-02-21T23:03:15.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.830+0000 7fb2d2a00640 1 -- 172.21.15.28:0/4269729517 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fb2c0001670 con 0x7fb2d40c5df0 2024-02-21T23:03:15.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.830+0000 7fb2d2a00640 1 -- 172.21.15.28:0/4269729517 >> v1:172.21.15.28:6789/0 conn(0x7fb2d40c5df0 legacy=0x7fb2d41556a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:15.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.830+0000 7fb2d2a00640 1 --2- 172.21.15.28:0/4269729517 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2b80039e0 0x7fb2b8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:15.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.830+0000 7fb2d2a00640 1 --2- 172.21.15.28:0/4269729517 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb2d4155db0 0x7fb2b8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:15.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.830+0000 7fb2d2a00640 1 --2- 172.21.15.28:0/4269729517 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2b8004d40 0x7fb2b80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:15.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.830+0000 7fb2d2a00640 1 -- 172.21.15.28:0/4269729517 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2b800ba70 con 0x7fb2b80039e0 2024-02-21T23:03:15.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.830+0000 7fb2d2a00640 1 -- 172.21.15.28:0/4269729517 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2b800bc20 con 0x7fb2b8004d40 2024-02-21T23:03:15.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.830+0000 7fb2d2a00640 1 -- 172.21.15.28:0/4269729517 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2b800e030 con 0x7fb2d4155db0 2024-02-21T23:03:15.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.830+0000 7fb2d3e00640 1 --2- 172.21.15.28:0/4269729517 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2b80039e0 0x7fb2b8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:15.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.831+0000 7fb2d3400640 1 --2- 172.21.15.28:0/4269729517 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb2d4155db0 0x7fb2b8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:15.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.831+0000 7fb2d9200640 1 --2- 172.21.15.28:0/4269729517 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2b8004d40 0x7fb2b80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:15.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.831+0000 7fb2d3e00640 1 -- 172.21.15.28:0/4269729517 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb2d4155db0 msgr2=0x7fb2b8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:15.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.831+0000 7fb2d3e00640 1 --2- 172.21.15.28:0/4269729517 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb2d4155db0 0x7fb2b8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:15.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.831+0000 7fb2d3e00640 1 -- 172.21.15.28:0/4269729517 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2b8004d40 msgr2=0x7fb2b80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:15.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.831+0000 7fb2d3e00640 1 --2- 172.21.15.28:0/4269729517 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2b8004d40 0x7fb2b80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:15.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.831+0000 7fb2d3e00640 1 -- 172.21.15.28:0/4269729517 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2b8003520 con 0x7fb2b80039e0 2024-02-21T23:03:15.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.832+0000 7fb2d3e00640 1 --2- 172.21.15.28:0/4269729517 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2b80039e0 0x7fb2b8003e00 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fb2c4001720 tx=0x7fb2b8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfa4214f6280213f server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:15.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.832+0000 7fb2d2a00640 1 -- 172.21.15.28:0/4269729517 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c4001370 con 0x7fb2b80039e0 2024-02-21T23:03:15.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.832+0000 7fb2d2a00640 1 -- 172.21.15.28:0/4269729517 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2c40052d0 con 0x7fb2b80039e0 2024-02-21T23:03:15.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.832+0000 7fb2d2a00640 1 -- 172.21.15.28:0/4269729517 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c4003830 con 0x7fb2b80039e0 2024-02-21T23:03:15.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.833+0000 7fb2d9c00640 1 -- 172.21.15.28:0/4269729517 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2b80039e0 msgr2=0x7fb2b8003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:15.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.833+0000 7fb2d9c00640 1 --2- 172.21.15.28:0/4269729517 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2b80039e0 0x7fb2b8003e00 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fb2c4001720 tx=0x7fb2b8005830 comp rx=0 tx=0).stop 2024-02-21T23:03:15.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.833+0000 7fb2d9200640 1 -- 172.21.15.28:0/4269729517 reap_dead start 2024-02-21T23:03:15.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.833+0000 7fb2d9c00640 1 -- 172.21.15.28:0/4269729517 shutdown_connections 2024-02-21T23:03:15.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.833+0000 7fb2d9c00640 1 -- 172.21.15.28:0/4269729517 >> 172.21.15.28:0/4269729517 conn(0x7fb2d40a8530 msgr2=0x7fb2d40a8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:15.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.833+0000 7fb2d9c00640 1 -- 172.21.15.28:0/4269729517 shutdown_connections 2024-02-21T23:03:15.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.833+0000 7fb2d9c00640 1 -- 172.21.15.28:0/4269729517 wait complete. 2024-02-21T23:03:15.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.835+0000 7fb2d9c00640 1 Processor -- start 2024-02-21T23:03:15.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.835+0000 7fb2d9c00640 1 -- start start 2024-02-21T23:03:15.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.836+0000 7fb2d9c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb2b80039e0 0x7fb2d41541a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:15.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.836+0000 7fb2d9c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2d40aa400 0x7fb2d40abdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:15.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.836+0000 7fb2d9c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2d414b290 0x7fb2d40ac2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:15.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.836+0000 7fb2d9c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d406f330 con 0x7fb2d40aa400 2024-02-21T23:03:15.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.836+0000 7fb2d9c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d40c5650 con 0x7fb2d414b290 2024-02-21T23:03:15.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.836+0000 7fb2d9c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d406f4b0 con 0x7fb2b80039e0 2024-02-21T23:03:15.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.836+0000 7fb2d3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2d40aa400 0x7fb2d40abdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:15.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.836+0000 7fb2d3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2d40aa400 0x7fb2d40abdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45924/0 (socket says 172.21.15.28:45924) 2024-02-21T23:03:15.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.836+0000 7fb2d3400640 1 -- 172.21.15.28:0/2767555481 learned_addr learned my addr 172.21.15.28:0/2767555481 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:15.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.836+0000 7fb2d3e00640 1 --2- 172.21.15.28:0/2767555481 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb2b80039e0 0x7fb2d41541a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:15.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.836+0000 7fb2d9200640 1 --2- 172.21.15.28:0/2767555481 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2d414b290 0x7fb2d40ac2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:15.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.837+0000 7fb2d3400640 1 -- 172.21.15.28:0/2767555481 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb2b80039e0 msgr2=0x7fb2d41541a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:15.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.837+0000 7fb2d3400640 1 --2- 172.21.15.28:0/2767555481 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb2b80039e0 0x7fb2d41541a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:15.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.837+0000 7fb2d3400640 1 -- 172.21.15.28:0/2767555481 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2d414b290 msgr2=0x7fb2d40ac2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:15.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.837+0000 7fb2d3400640 1 --2- 172.21.15.28:0/2767555481 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2d414b290 0x7fb2d40ac2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:15.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.837+0000 7fb2d3400640 1 -- 172.21.15.28:0/2767555481 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2d414fbb0 con 0x7fb2d40aa400 2024-02-21T23:03:15.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.837+0000 7fb2d9200640 1 --2- 172.21.15.28:0/2767555481 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2d414b290 0x7fb2d40ac2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:03:15.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.837+0000 7fb2d3400640 1 --2- 172.21.15.28:0/2767555481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2d40aa400 0x7fb2d40abdb0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fb2c000e730 tx=0x7fb2c000ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:15.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.837+0000 7fb2cbe00640 1 -- 172.21.15.28:0/2767555481 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c000ee40 con 0x7fb2d40aa400 2024-02-21T23:03:15.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.837+0000 7fb2cbe00640 1 -- 172.21.15.28:0/2767555481 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2c0018070 con 0x7fb2d40aa400 2024-02-21T23:03:15.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.837+0000 7fb2d9c00640 1 -- 172.21.15.28:0/2767555481 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2d414fe90 con 0x7fb2d40aa400 2024-02-21T23:03:15.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.837+0000 7fb2cbe00640 1 -- 172.21.15.28:0/2767555481 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c000ca90 con 0x7fb2d40aa400 2024-02-21T23:03:15.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.837+0000 7fb2d9c00640 1 -- 172.21.15.28:0/2767555481 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2d4150450 con 0x7fb2d40aa400 2024-02-21T23:03:15.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.839+0000 7fb2d9c00640 1 -- 172.21.15.28:0/2767555481 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2d404f290 con 0x7fb2d40aa400 2024-02-21T23:03:15.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.840+0000 7fb2cbe00640 1 -- 172.21.15.28:0/2767555481 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb2c000cc30 con 0x7fb2d40aa400 2024-02-21T23:03:15.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.841+0000 7fb2cbe00640 1 --2- 172.21.15.28:0/2767555481 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb2b406b4d0 0x7fb2b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:15.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.841+0000 7fb2cbe00640 1 -- 172.21.15.28:0/2767555481 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fb2c008a280 con 0x7fb2d40aa400 2024-02-21T23:03:15.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.841+0000 7fb2d3e00640 1 --2- 172.21.15.28:0/2767555481 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb2b406b4d0 0x7fb2b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:15.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.842+0000 7fb2d3e00640 1 --2- 172.21.15.28:0/2767555481 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb2b406b4d0 0x7fb2b406d990 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb2c4033000 tx=0x7fb2c4009030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:15.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:15.847+0000 7fb2cbe00640 1 -- 172.21.15.28:0/2767555481 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb2c0058e50 con 0x7fb2d40aa400 2024-02-21T23:03:16.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.045+0000 7fb2d9c00640 1 -- 172.21.15.28:0/2767555481 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb2d40c51d0 con 0x7fb2d40aa400 2024-02-21T23:03:16.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.046+0000 7fb2cbe00640 1 -- 172.21.15.28:0/2767555481 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb2c0058750 con 0x7fb2d40aa400 2024-02-21T23:03:16.036 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-02-21T23:03:16.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.049+0000 7fb2d9c00640 1 -- 172.21.15.28:0/2767555481 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb2b406b4d0 msgr2=0x7fb2b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:16.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.049+0000 7fb2d9c00640 1 --2- 172.21.15.28:0/2767555481 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb2b406b4d0 0x7fb2b406d990 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb2c4033000 tx=0x7fb2c4009030 comp rx=0 tx=0).stop 2024-02-21T23:03:16.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.049+0000 7fb2d9c00640 1 -- 172.21.15.28:0/2767555481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2d40aa400 msgr2=0x7fb2d40abdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:16.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.049+0000 7fb2d9c00640 1 --2- 172.21.15.28:0/2767555481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2d40aa400 0x7fb2d40abdb0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fb2c000e730 tx=0x7fb2c000ebf0 comp rx=0 tx=0).stop 2024-02-21T23:03:16.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.049+0000 7fb2d9c00640 1 -- 172.21.15.28:0/2767555481 shutdown_connections 2024-02-21T23:03:16.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.049+0000 7fb2d9c00640 1 --2- 172.21.15.28:0/2767555481 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb2b406b4d0 0x7fb2b406d990 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:16.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.049+0000 7fb2d9c00640 1 --2- 172.21.15.28:0/2767555481 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2d414b290 0x7fb2d40ac2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:16.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.049+0000 7fb2d9c00640 1 --2- 172.21.15.28:0/2767555481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2d40aa400 0x7fb2d40abdb0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:16.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.049+0000 7fb2d9c00640 1 --2- 172.21.15.28:0/2767555481 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb2b80039e0 0x7fb2d41541a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:16.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.049+0000 7fb2d9c00640 1 -- 172.21.15.28:0/2767555481 >> 172.21.15.28:0/2767555481 conn(0x7fb2d40a8530 msgr2=0x7fb2d40bfe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:16.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.049+0000 7fb2d9c00640 1 -- 172.21.15.28:0/2767555481 shutdown_connections 2024-02-21T23:03:16.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.049+0000 7fb2d9c00640 1 -- 172.21.15.28:0/2767555481 wait complete. 2024-02-21T23:03:16.059 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-21T23:03:16.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.296+0000 7f87e9600640 1 Processor -- start 2024-02-21T23:03:16.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.296+0000 7f87e9600640 1 -- start start 2024-02-21T23:03:16.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.298+0000 7f87e9600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f87e406dbc0 0x7f87e406c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:16.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.298+0000 7f87e9600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f87e413e520 0x7f87e406c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:16.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.298+0000 7f87e9600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87e4158550 con 0x7f87e413e0c0 2024-02-21T23:03:16.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.298+0000 7f87e9600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87e40c89a0 con 0x7f87e413e520 2024-02-21T23:03:16.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.298+0000 7f87e9600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87e41587d0 con 0x7f87e406dbc0 2024-02-21T23:03:16.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.298+0000 7f87e3e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f87e413e0c0 0x7f87e40c71b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39582/0 (socket says 172.21.15.28:39582) 2024-02-21T23:03:16.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.298+0000 7f87e3e00640 1 -- 172.21.15.28:0/2631507981 learned_addr learned my addr 172.21.15.28:0/2631507981 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:03:16.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.298+0000 7f87e8c00640 1 --2- 172.21.15.28:0/2631507981 >> v2:172.21.15.28:3300/0 conn(0x7f87e413e520 0x7f87e406c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:16.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.299+0000 7f87e3400640 1 --2- 172.21.15.28:0/2631507981 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f87e406dbc0 0x7f87e406c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:16.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.299+0000 7f87e2a00640 1 -- 172.21.15.28:0/2631507981 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3116902582 0 0) 0x7f87d4001490 con 0x7f87e413e0c0 2024-02-21T23:03:16.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.299+0000 7f87e2a00640 1 -- 172.21.15.28:0/2631507981 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87c8003520 con 0x7f87e413e0c0 2024-02-21T23:03:16.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.299+0000 7f87e8c00640 1 -- 172.21.15.28:0/2631507981 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f87e406dbc0 msgr2=0x7f87e406c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:16.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.299+0000 7f87e8c00640 1 --2- 172.21.15.28:0/2631507981 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f87e406dbc0 0x7f87e406c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:16.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.299+0000 7f87e8c00640 1 -- 172.21.15.28:0/2631507981 >> v1:172.21.15.28:6789/0 conn(0x7f87e413e0c0 legacy=0x7f87e40c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:16.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.299+0000 7f87e2a00640 1 -- 172.21.15.28:0/2631507981 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 31289177 0 0) 0x7f87c8003520 con 0x7f87e413e0c0 2024-02-21T23:03:16.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.299+0000 7f87e8c00640 1 -- 172.21.15.28:0/2631507981 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87e4158ab0 con 0x7f87e413e520 2024-02-21T23:03:16.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.301+0000 7f87e8c00640 1 --2- 172.21.15.28:0/2631507981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87e413e520 0x7f87e406c7c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f87dc0097a0 tx=0x7f87dc02b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b50f3cb9229620f server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:16.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.301+0000 7f87e2a00640 1 -- 172.21.15.28:0/2631507981 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87dc037070 con 0x7f87e413e520 2024-02-21T23:03:16.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.301+0000 7f87e2a00640 1 -- 172.21.15.28:0/2631507981 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87dc030510 con 0x7f87e413e520 2024-02-21T23:03:16.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.301+0000 7f87e2a00640 1 -- 172.21.15.28:0/2631507981 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87d4001bc0 con 0x7f87e413e520 2024-02-21T23:03:16.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.302+0000 7f87e9600640 1 -- 172.21.15.28:0/2631507981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87e413e520 msgr2=0x7f87e406c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:16.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.302+0000 7f87e9600640 1 --2- 172.21.15.28:0/2631507981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87e413e520 0x7f87e406c7c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f87dc0097a0 tx=0x7f87dc02b710 comp rx=0 tx=0).stop 2024-02-21T23:03:16.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.302+0000 7f87e9600640 1 -- 172.21.15.28:0/2631507981 shutdown_connections 2024-02-21T23:03:16.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.302+0000 7f87e9600640 1 --2- 172.21.15.28:0/2631507981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87e413e520 0x7f87e406c7c0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:16.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.302+0000 7f87e9600640 1 --2- 172.21.15.28:0/2631507981 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f87e406dbc0 0x7f87e406c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:16.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.302+0000 7f87e9600640 1 -- 172.21.15.28:0/2631507981 >> 172.21.15.28:0/2631507981 conn(0x7f87e4073110 msgr2=0x7f87e40ac490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:16.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.302+0000 7f87e9600640 1 -- 172.21.15.28:0/2631507981 shutdown_connections 2024-02-21T23:03:16.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.302+0000 7f87e9600640 1 -- 172.21.15.28:0/2631507981 wait complete. 2024-02-21T23:03:16.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.304+0000 7f87e9600640 1 Processor -- start 2024-02-21T23:03:16.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.304+0000 7f87e9600640 1 -- start start 2024-02-21T23:03:16.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.305+0000 7f87e9600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f87e406dbc0 0x7f87e40abb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:16.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.305+0000 7f87e9600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87e413e0c0 0x7f87e41ee2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:16.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.305+0000 7f87e9600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f87e413e520 0x7f87e41ee800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:16.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.305+0000 7f87e9600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87e40c78c0 con 0x7f87e413e0c0 2024-02-21T23:03:16.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.305+0000 7f87e9600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87e4159070 con 0x7f87e406dbc0 2024-02-21T23:03:16.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.305+0000 7f87e9600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87e40c7a40 con 0x7f87e413e520 2024-02-21T23:03:16.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.305+0000 7f87e3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87e413e0c0 0x7f87e41ee2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:16.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.305+0000 7f87e3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87e413e0c0 0x7f87e41ee2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45948/0 (socket says 172.21.15.28:45948) 2024-02-21T23:03:16.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.305+0000 7f87e3400640 1 -- 172.21.15.28:0/1139557554 learned_addr learned my addr 172.21.15.28:0/1139557554 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:16.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.306+0000 7f87e3e00640 1 --2- 172.21.15.28:0/1139557554 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f87e406dbc0 0x7f87e40abb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:16.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.306+0000 7f87e8c00640 1 --2- 172.21.15.28:0/1139557554 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f87e413e520 0x7f87e41ee800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:16.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.306+0000 7f87e3400640 1 -- 172.21.15.28:0/1139557554 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f87e413e520 msgr2=0x7f87e41ee800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:16.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.306+0000 7f87e3400640 1 --2- 172.21.15.28:0/1139557554 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f87e413e520 0x7f87e41ee800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:16.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.306+0000 7f87e3400640 1 -- 172.21.15.28:0/1139557554 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f87e406dbc0 msgr2=0x7f87e40abb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:16.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.306+0000 7f87e3400640 1 --2- 172.21.15.28:0/1139557554 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f87e406dbc0 0x7f87e40abb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:16.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.306+0000 7f87e3400640 1 -- 172.21.15.28:0/1139557554 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87e4151d60 con 0x7f87e413e0c0 2024-02-21T23:03:16.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.306+0000 7f87e3400640 1 --2- 172.21.15.28:0/1139557554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87e413e0c0 0x7f87e41ee2c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f87cc00e8a0 tx=0x7f87cc00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:16.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.306+0000 7f87dbe00640 1 -- 172.21.15.28:0/1139557554 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87cc017070 con 0x7f87e413e0c0 2024-02-21T23:03:16.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.306+0000 7f87dbe00640 1 -- 172.21.15.28:0/1139557554 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87cc00c940 con 0x7f87e413e0c0 2024-02-21T23:03:16.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.306+0000 7f87dbe00640 1 -- 172.21.15.28:0/1139557554 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87cc00cd30 con 0x7f87e413e0c0 2024-02-21T23:03:16.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.306+0000 7f87e9600640 1 -- 172.21.15.28:0/1139557554 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87e4151a80 con 0x7f87e413e0c0 2024-02-21T23:03:16.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.307+0000 7f87e9600640 1 -- 172.21.15.28:0/1139557554 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87e4137840 con 0x7f87e413e0c0 2024-02-21T23:03:16.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.308+0000 7f87e9600640 1 -- 172.21.15.28:0/1139557554 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87e404f290 con 0x7f87e413e0c0 2024-02-21T23:03:16.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.309+0000 7f87dbe00640 1 -- 172.21.15.28:0/1139557554 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f87cc01a070 con 0x7f87e413e0c0 2024-02-21T23:03:16.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.310+0000 7f87dbe00640 1 --2- 172.21.15.28:0/1139557554 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f87c006b4d0 0x7f87c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:16.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.310+0000 7f87dbe00640 1 -- 172.21.15.28:0/1139557554 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f87cc08a390 con 0x7f87e413e0c0 2024-02-21T23:03:16.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.310+0000 7f87e3e00640 1 --2- 172.21.15.28:0/1139557554 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f87c006b4d0 0x7f87c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:16.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.311+0000 7f87e3e00640 1 --2- 172.21.15.28:0/1139557554 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f87c006b4d0 0x7f87c006d990 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f87d4002230 tx=0x7f87d4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:16.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.314+0000 7f87dbe00640 1 -- 172.21.15.28:0/1139557554 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f87cc058ed0 con 0x7f87e413e0c0 2024-02-21T23:03:16.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.516+0000 7f87e9600640 1 -- 172.21.15.28:0/1139557554 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f87e40ac0a0 con 0x7f87e413e0c0 2024-02-21T23:03:16.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.517+0000 7f87dbe00640 1 -- 172.21.15.28:0/1139557554 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f87cc0587d0 con 0x7f87e413e0c0 2024-02-21T23:03:16.507 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-02-21T23:03:16.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.519+0000 7f87e9600640 1 -- 172.21.15.28:0/1139557554 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f87c006b4d0 msgr2=0x7f87c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:16.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.519+0000 7f87e9600640 1 --2- 172.21.15.28:0/1139557554 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f87c006b4d0 0x7f87c006d990 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f87d4002230 tx=0x7f87d4009040 comp rx=0 tx=0).stop 2024-02-21T23:03:16.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.519+0000 7f87e9600640 1 -- 172.21.15.28:0/1139557554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87e413e0c0 msgr2=0x7f87e41ee2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:16.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.519+0000 7f87e9600640 1 --2- 172.21.15.28:0/1139557554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87e413e0c0 0x7f87e41ee2c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f87cc00e8a0 tx=0x7f87cc00ed60 comp rx=0 tx=0).stop 2024-02-21T23:03:16.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.519+0000 7f87e9600640 1 -- 172.21.15.28:0/1139557554 shutdown_connections 2024-02-21T23:03:16.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.519+0000 7f87e9600640 1 --2- 172.21.15.28:0/1139557554 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f87c006b4d0 0x7f87c006d990 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:16.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.519+0000 7f87e9600640 1 --2- 172.21.15.28:0/1139557554 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f87e413e520 0x7f87e41ee800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:16.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.519+0000 7f87e9600640 1 --2- 172.21.15.28:0/1139557554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87e413e0c0 0x7f87e41ee2c0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:16.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.519+0000 7f87e9600640 1 --2- 172.21.15.28:0/1139557554 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f87e406dbc0 0x7f87e40abb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:16.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.519+0000 7f87e9600640 1 -- 172.21.15.28:0/1139557554 >> 172.21.15.28:0/1139557554 conn(0x7f87e4073110 msgr2=0x7f87e40c5ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:16.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.520+0000 7f87e9600640 1 -- 172.21.15.28:0/1139557554 shutdown_connections 2024-02-21T23:03:16.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.520+0000 7f87e9600640 1 -- 172.21.15.28:0/1139557554 wait complete. 2024-02-21T23:03:16.529 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-02-21T23:03:16.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.772+0000 7f20d9a00640 1 Processor -- start 2024-02-21T23:03:16.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.773+0000 7f20d9a00640 1 -- start start 2024-02-21T23:03:16.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.774+0000 7f20d9a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f20d40c22e0 0x7f20d406d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:16.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.774+0000 7f20d9a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f20d41561a0 0x7f20d406d660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:16.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.774+0000 7f20d9a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20d4071970 con 0x7f20d40c5a70 2024-02-21T23:03:16.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.774+0000 7f20d9a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20d4072a10 con 0x7f20d40c22e0 2024-02-21T23:03:16.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.774+0000 7f20d9a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20d4072580 con 0x7f20d41561a0 2024-02-21T23:03:16.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.774+0000 7f20d2a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f20d40c5a70 0x7f20d4155a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39590/0 (socket says 172.21.15.28:39590) 2024-02-21T23:03:16.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.774+0000 7f20d2a00640 1 -- 172.21.15.28:0/1938883907 learned_addr learned my addr 172.21.15.28:0/1938883907 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:03:16.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.774+0000 7f20d2000640 1 --2- 172.21.15.28:0/1938883907 >> v2:172.21.15.81:3300/0 conn(0x7f20d40c22e0 0x7f20d406d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:16.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.774+0000 7f20d3400640 1 --2- 172.21.15.28:0/1938883907 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f20d41561a0 0x7f20d406d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:16.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.775+0000 7f20d1600640 1 -- 172.21.15.28:0/1938883907 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3749200654 0 0) 0x7f20d4071970 con 0x7f20d40c5a70 2024-02-21T23:03:16.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.775+0000 7f20d1600640 1 -- 172.21.15.28:0/1938883907 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20b4003520 con 0x7f20d40c5a70 2024-02-21T23:03:16.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.775+0000 7f20d1600640 1 -- 172.21.15.28:0/1938883907 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 347038390 0 0) 0x7f20b4003520 con 0x7f20d40c5a70 2024-02-21T23:03:16.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.775+0000 7f20d1600640 1 -- 172.21.15.28:0/1938883907 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f20d41561a0 msgr2=0x7f20d406d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:16.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.775+0000 7f20d1600640 1 --2- 172.21.15.28:0/1938883907 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f20d41561a0 0x7f20d406d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:16.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.775+0000 7f20d1600640 1 -- 172.21.15.28:0/1938883907 >> v2:172.21.15.81:3300/0 conn(0x7f20d40c22e0 msgr2=0x7f20d406d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:16.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.775+0000 7f20d1600640 1 --2- 172.21.15.28:0/1938883907 >> v2:172.21.15.81:3300/0 conn(0x7f20d40c22e0 0x7f20d406d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:16.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.775+0000 7f20d1600640 1 -- 172.21.15.28:0/1938883907 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20d4071c50 con 0x7f20d40c5a70 2024-02-21T23:03:16.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.775+0000 7f20d1600640 1 -- 172.21.15.28:0/1938883907 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f20c4001670 con 0x7f20d40c5a70 2024-02-21T23:03:16.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.775+0000 7f20d1600640 1 -- 172.21.15.28:0/1938883907 >> v1:172.21.15.28:6789/0 conn(0x7f20d40c5a70 legacy=0x7f20d4155a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:16.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.775+0000 7f20d1600640 1 --2- 172.21.15.28:0/1938883907 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20b40039b0 0x7f20b4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:16.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.775+0000 7f20d1600640 1 --2- 172.21.15.28:0/1938883907 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f20d41561a0 0x7f20b4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:16.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.775+0000 7f20d1600640 1 --2- 172.21.15.28:0/1938883907 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20b4004d40 0x7f20b40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:16.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.775+0000 7f20d1600640 1 -- 172.21.15.28:0/1938883907 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20d4072a10 con 0x7f20b40039b0 2024-02-21T23:03:16.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.775+0000 7f20d1600640 1 -- 172.21.15.28:0/1938883907 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20d4072580 con 0x7f20b4004d40 2024-02-21T23:03:16.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.775+0000 7f20d1600640 1 -- 172.21.15.28:0/1938883907 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20b400b9f0 con 0x7f20d41561a0 2024-02-21T23:03:16.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.776+0000 7f20d2a00640 1 --2- 172.21.15.28:0/1938883907 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20b40039b0 0x7f20b4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:16.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.776+0000 7f20d2000640 1 --2- 172.21.15.28:0/1938883907 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f20d41561a0 0x7f20b4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:16.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.776+0000 7f20d3400640 1 --2- 172.21.15.28:0/1938883907 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20b4004d40 0x7f20b40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:16.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.776+0000 7f20d2a00640 1 -- 172.21.15.28:0/1938883907 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f20d41561a0 msgr2=0x7f20b4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:16.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.776+0000 7f20d2a00640 1 --2- 172.21.15.28:0/1938883907 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f20d41561a0 0x7f20b4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:16.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.776+0000 7f20d2a00640 1 -- 172.21.15.28:0/1938883907 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20b4004d40 msgr2=0x7f20b40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:16.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.776+0000 7f20d2a00640 1 --2- 172.21.15.28:0/1938883907 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20b4004d40 0x7f20b40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:16.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.776+0000 7f20d2a00640 1 -- 172.21.15.28:0/1938883907 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20b4003520 con 0x7f20b40039b0 2024-02-21T23:03:16.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.777+0000 7f20d2a00640 1 --2- 172.21.15.28:0/1938883907 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20b40039b0 0x7f20b4003dd0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f20c400c160 tx=0x7f20b4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f19cf1992724833 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:16.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.777+0000 7f20d1600640 1 -- 172.21.15.28:0/1938883907 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c4007c80 con 0x7f20b40039b0 2024-02-21T23:03:16.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.777+0000 7f20d1600640 1 -- 172.21.15.28:0/1938883907 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20c4030070 con 0x7f20b40039b0 2024-02-21T23:03:16.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.777+0000 7f20d1600640 1 -- 172.21.15.28:0/1938883907 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c4005d30 con 0x7f20b40039b0 2024-02-21T23:03:16.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.778+0000 7f20d9a00640 1 -- 172.21.15.28:0/1938883907 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20b40039b0 msgr2=0x7f20b4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:16.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.778+0000 7f20d9a00640 1 --2- 172.21.15.28:0/1938883907 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20b40039b0 0x7f20b4003dd0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f20c400c160 tx=0x7f20b4005830 comp rx=0 tx=0).stop 2024-02-21T23:03:16.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.778+0000 7f20d3400640 1 -- 172.21.15.28:0/1938883907 reap_dead start 2024-02-21T23:03:16.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.778+0000 7f20d9a00640 1 -- 172.21.15.28:0/1938883907 shutdown_connections 2024-02-21T23:03:16.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.778+0000 7f20d9a00640 1 -- 172.21.15.28:0/1938883907 >> 172.21.15.28:0/1938883907 conn(0x7f20d40a85a0 msgr2=0x7f20d40a8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:16.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.778+0000 7f20d9a00640 1 -- 172.21.15.28:0/1938883907 shutdown_connections 2024-02-21T23:03:16.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.778+0000 7f20d9a00640 1 -- 172.21.15.28:0/1938883907 wait complete. 2024-02-21T23:03:16.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.780+0000 7f20d9a00640 1 Processor -- start 2024-02-21T23:03:16.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.780+0000 7f20d9a00640 1 -- start start 2024-02-21T23:03:16.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.781+0000 7f20d9a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20b40039b0 0x7f20d40aaba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:16.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.781+0000 7f20d9a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20d40ab490 0x7f20d41e7880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:16.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.781+0000 7f20d9a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f20d40bb810 0x7f20d40bfb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:16.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.781+0000 7f20d9a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20d40a6160 con 0x7f20b40039b0 2024-02-21T23:03:16.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.781+0000 7f20d9a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20d4072210 con 0x7f20d40ab490 2024-02-21T23:03:16.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.781+0000 7f20d9a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20d40a62e0 con 0x7f20d40bb810 2024-02-21T23:03:16.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.781+0000 7f20d2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20b40039b0 0x7f20d40aaba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:16.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.781+0000 7f20d2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20b40039b0 0x7f20d40aaba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45964/0 (socket says 172.21.15.28:45964) 2024-02-21T23:03:16.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.781+0000 7f20d2a00640 1 -- 172.21.15.28:0/1533019666 learned_addr learned my addr 172.21.15.28:0/1533019666 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:16.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.782+0000 7f20d3400640 1 --2- 172.21.15.28:0/1533019666 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f20d40bb810 0x7f20d40bfb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:16.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.782+0000 7f20d2000640 1 --2- 172.21.15.28:0/1533019666 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20d40ab490 0x7f20d41e7880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:16.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.782+0000 7f20d2a00640 1 -- 172.21.15.28:0/1533019666 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f20d40bb810 msgr2=0x7f20d40bfb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:16.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.782+0000 7f20d2a00640 1 --2- 172.21.15.28:0/1533019666 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f20d40bb810 0x7f20d40bfb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:16.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.782+0000 7f20d2a00640 1 -- 172.21.15.28:0/1533019666 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20d40ab490 msgr2=0x7f20d41e7880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:16.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.782+0000 7f20d2a00640 1 --2- 172.21.15.28:0/1533019666 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20d40ab490 0x7f20d41e7880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:16.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.782+0000 7f20d2a00640 1 -- 172.21.15.28:0/1533019666 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20d40aa0c0 con 0x7f20b40039b0 2024-02-21T23:03:16.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.782+0000 7f20d2a00640 1 --2- 172.21.15.28:0/1533019666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20b40039b0 0x7f20d40aaba0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f20c400c160 tx=0x7f20d4071b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:16.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.782+0000 7f20caa00640 1 -- 172.21.15.28:0/1533019666 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c4004bf0 con 0x7f20b40039b0 2024-02-21T23:03:16.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.782+0000 7f20caa00640 1 -- 172.21.15.28:0/1533019666 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20c4004d90 con 0x7f20b40039b0 2024-02-21T23:03:16.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.782+0000 7f20d9a00640 1 -- 172.21.15.28:0/1533019666 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20d40aa3a0 con 0x7f20b40039b0 2024-02-21T23:03:16.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.782+0000 7f20caa00640 1 -- 172.21.15.28:0/1533019666 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c40072b0 con 0x7f20b40039b0 2024-02-21T23:03:16.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.783+0000 7f20d9a00640 1 -- 172.21.15.28:0/1533019666 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20d41e7dc0 con 0x7f20b40039b0 2024-02-21T23:03:16.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.784+0000 7f20d9a00640 1 -- 172.21.15.28:0/1533019666 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20d4067f30 con 0x7f20b40039b0 2024-02-21T23:03:16.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.785+0000 7f20caa00640 1 -- 172.21.15.28:0/1533019666 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f20c4047020 con 0x7f20b40039b0 2024-02-21T23:03:16.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.785+0000 7f20caa00640 1 --2- 172.21.15.28:0/1533019666 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f20b006b500 0x7f20b006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:16.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.786+0000 7f20caa00640 1 -- 172.21.15.28:0/1533019666 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f20c40ab7e0 con 0x7f20b40039b0 2024-02-21T23:03:16.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.786+0000 7f20d2000640 1 --2- 172.21.15.28:0/1533019666 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f20b006b500 0x7f20b006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:16.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.787+0000 7f20d2000640 1 --2- 172.21.15.28:0/1533019666 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f20b006b500 0x7f20b006d9c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f20d40abd00 tx=0x7f20bc008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:16.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.790+0000 7f20caa00640 1 -- 172.21.15.28:0/1533019666 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f20c407a310 con 0x7f20b40039b0 2024-02-21T23:03:16.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.990+0000 7f20d9a00640 1 -- 172.21.15.28:0/1533019666 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f20d40c16b0 con 0x7f20b40039b0 2024-02-21T23:03:16.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.991+0000 7f20caa00640 1 -- 172.21.15.28:0/1533019666 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) v1 ==== 103+0+11 (secure 0 0 0) 0x7f20c4034e80 con 0x7f20b40039b0 2024-02-21T23:03:16.981 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-02-21T23:03:16.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.994+0000 7f20d9a00640 1 -- 172.21.15.28:0/1533019666 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f20b006b500 msgr2=0x7f20b006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:16.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.994+0000 7f20d9a00640 1 --2- 172.21.15.28:0/1533019666 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f20b006b500 0x7f20b006d9c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f20d40abd00 tx=0x7f20bc008040 comp rx=0 tx=0).stop 2024-02-21T23:03:16.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.994+0000 7f20d9a00640 1 -- 172.21.15.28:0/1533019666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20b40039b0 msgr2=0x7f20d40aaba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:16.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.994+0000 7f20d9a00640 1 --2- 172.21.15.28:0/1533019666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20b40039b0 0x7f20d40aaba0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f20c400c160 tx=0x7f20d4071b40 comp rx=0 tx=0).stop 2024-02-21T23:03:16.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.994+0000 7f20d9a00640 1 -- 172.21.15.28:0/1533019666 shutdown_connections 2024-02-21T23:03:16.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.994+0000 7f20d9a00640 1 --2- 172.21.15.28:0/1533019666 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f20b006b500 0x7f20b006d9c0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:16.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.994+0000 7f20d9a00640 1 --2- 172.21.15.28:0/1533019666 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f20d40bb810 0x7f20d40bfb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:16.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.994+0000 7f20d9a00640 1 --2- 172.21.15.28:0/1533019666 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20d40ab490 0x7f20d41e7880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:16.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.994+0000 7f20d9a00640 1 --2- 172.21.15.28:0/1533019666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20b40039b0 0x7f20d40aaba0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:16.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.994+0000 7f20d9a00640 1 -- 172.21.15.28:0/1533019666 >> 172.21.15.28:0/1533019666 conn(0x7f20d40a85a0 msgr2=0x7f20d4074570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:16.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.994+0000 7f20d9a00640 1 -- 172.21.15.28:0/1533019666 shutdown_connections 2024-02-21T23:03:16.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:16.994+0000 7f20d9a00640 1 -- 172.21.15.28:0/1533019666 wait complete. 2024-02-21T23:03:17.005 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:03:17.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.235+0000 7f14ecc00640 1 Processor -- start 2024-02-21T23:03:17.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.236+0000 7f14ecc00640 1 -- start start 2024-02-21T23:03:17.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.237+0000 7f14ecc00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f14e80aff90 0x7f14e80c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:17.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.237+0000 7f14ecc00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f14e80c5720 0x7f14e815f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:17.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.237+0000 7f14ecc00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14e80c74c0 con 0x7f14e806dbf0 2024-02-21T23:03:17.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.237+0000 7f14ecc00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14e80c7270 con 0x7f14e80aff90 2024-02-21T23:03:17.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.237+0000 7f14ecc00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14e8158630 con 0x7f14e80c5720 2024-02-21T23:03:17.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.238+0000 7f14e7400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f14e80aff90 0x7f14e80c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:17.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.238+0000 7f14e7400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f14e80aff90 0x7f14e80c51e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45970/0 (socket says 172.21.15.28:45970) 2024-02-21T23:03:17.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.238+0000 7f14e7400640 1 -- 172.21.15.28:0/3249444601 learned_addr learned my addr 172.21.15.28:0/3249444601 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:17.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.238+0000 7f14e7e00640 1 --2- 172.21.15.28:0/3249444601 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f14e80c5720 0x7f14e815f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:17.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.238+0000 7f14e7400640 1 -- 172.21.15.28:0/3249444601 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f14e80c5720 msgr2=0x7f14e815f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:17.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.238+0000 7f14e7400640 1 --2- 172.21.15.28:0/3249444601 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f14e80c5720 0x7f14e815f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:17.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.238+0000 7f14e7400640 1 -- 172.21.15.28:0/3249444601 >> v1:172.21.15.28:6789/0 conn(0x7f14e806dbf0 legacy=0x7f14e8158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:17.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.238+0000 7f14e6000640 1 -- 172.21.15.28:0/3249444601 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 691961479 0 0) 0x7f14e80c74c0 con 0x7f14e806dbf0 2024-02-21T23:03:17.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.238+0000 7f14e7400640 1 -- 172.21.15.28:0/3249444601 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14e80c77a0 con 0x7f14e80aff90 2024-02-21T23:03:17.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.239+0000 7f14e7400640 1 --2- 172.21.15.28:0/3249444601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14e80aff90 0x7f14e80c51e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f14d8009e50 tx=0x7f14d802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84724655430826f8 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:17.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.240+0000 7f14e6000640 1 -- 172.21.15.28:0/3249444601 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d802cb00 con 0x7f14e80aff90 2024-02-21T23:03:17.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.240+0000 7f14e6000640 1 -- 172.21.15.28:0/3249444601 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14d802cca0 con 0x7f14e80aff90 2024-02-21T23:03:17.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.240+0000 7f14e6000640 1 -- 172.21.15.28:0/3249444601 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d8034610 con 0x7f14e80aff90 2024-02-21T23:03:17.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.240+0000 7f14ecc00640 1 -- 172.21.15.28:0/3249444601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14e80aff90 msgr2=0x7f14e80c51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:17.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.240+0000 7f14ecc00640 1 --2- 172.21.15.28:0/3249444601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14e80aff90 0x7f14e80c51e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f14d8009e50 tx=0x7f14d802bf60 comp rx=0 tx=0).stop 2024-02-21T23:03:17.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.240+0000 7f14ecc00640 1 -- 172.21.15.28:0/3249444601 shutdown_connections 2024-02-21T23:03:17.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.240+0000 7f14ecc00640 1 --2- 172.21.15.28:0/3249444601 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f14e80c5720 0x7f14e815f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:17.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.240+0000 7f14ecc00640 1 --2- 172.21.15.28:0/3249444601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14e80aff90 0x7f14e80c51e0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:17.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.240+0000 7f14ecc00640 1 -- 172.21.15.28:0/3249444601 >> 172.21.15.28:0/3249444601 conn(0x7f14e80ac370 msgr2=0x7f14e80ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:17.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.240+0000 7f14ecc00640 1 -- 172.21.15.28:0/3249444601 shutdown_connections 2024-02-21T23:03:17.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.240+0000 7f14ecc00640 1 -- 172.21.15.28:0/3249444601 wait complete. 2024-02-21T23:03:17.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.242+0000 7f14ecc00640 1 Processor -- start 2024-02-21T23:03:17.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.242+0000 7f14ecc00640 1 -- start start 2024-02-21T23:03:17.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.243+0000 7f14ecc00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f14e806dbf0 0x7f14e80ae9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:17.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.243+0000 7f14ecc00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f14e80aff90 0x7f14e81e78c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:17.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.243+0000 7f14ecc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14e80c5720 0x7f14e81e7e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:17.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.243+0000 7f14ecc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14e80c8760 con 0x7f14e80c5720 2024-02-21T23:03:17.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.243+0000 7f14ecc00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14e80be0c0 con 0x7f14e806dbf0 2024-02-21T23:03:17.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.243+0000 7f14ecc00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14e80c88e0 con 0x7f14e80aff90 2024-02-21T23:03:17.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.243+0000 7f14e7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14e80c5720 0x7f14e81e7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:17.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.244+0000 7f14e7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14e80c5720 0x7f14e81e7e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45982/0 (socket says 172.21.15.28:45982) 2024-02-21T23:03:17.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.244+0000 7f14e7e00640 1 -- 172.21.15.28:0/49703835 learned_addr learned my addr 172.21.15.28:0/49703835 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:17.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.244+0000 7f14e6a00640 1 --2- 172.21.15.28:0/49703835 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f14e80aff90 0x7f14e81e78c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:17.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.244+0000 7f14e7400640 1 --2- 172.21.15.28:0/49703835 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f14e806dbf0 0x7f14e80ae9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:17.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.244+0000 7f14e7e00640 1 -- 172.21.15.28:0/49703835 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f14e80aff90 msgr2=0x7f14e81e78c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:17.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.244+0000 7f14e7e00640 1 --2- 172.21.15.28:0/49703835 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f14e80aff90 0x7f14e81e78c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:17.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.244+0000 7f14e7e00640 1 -- 172.21.15.28:0/49703835 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f14e806dbf0 msgr2=0x7f14e80ae9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:17.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.244+0000 7f14e7e00640 1 --2- 172.21.15.28:0/49703835 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f14e806dbf0 0x7f14e80ae9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:17.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.244+0000 7f14e7e00640 1 -- 172.21.15.28:0/49703835 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14e8153500 con 0x7f14e80c5720 2024-02-21T23:03:17.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.244+0000 7f14e7e00640 1 --2- 172.21.15.28:0/49703835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14e80c5720 0x7f14e81e7e00 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f14e000d600 tx=0x7f14e000dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:17.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.245+0000 7f14df400640 1 -- 172.21.15.28:0/49703835 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14e00125f0 con 0x7f14e80c5720 2024-02-21T23:03:17.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.245+0000 7f14df400640 1 -- 172.21.15.28:0/49703835 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14e0019070 con 0x7f14e80c5720 2024-02-21T23:03:17.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.245+0000 7f14ecc00640 1 -- 172.21.15.28:0/49703835 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14e81537e0 con 0x7f14e80c5720 2024-02-21T23:03:17.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.245+0000 7f14df400640 1 -- 172.21.15.28:0/49703835 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14e00129f0 con 0x7f14e80c5720 2024-02-21T23:03:17.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.245+0000 7f14ecc00640 1 -- 172.21.15.28:0/49703835 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14e80aeee0 con 0x7f14e80c5720 2024-02-21T23:03:17.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.246+0000 7f14ecc00640 1 -- 172.21.15.28:0/49703835 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14e81539b0 con 0x7f14e80c5720 2024-02-21T23:03:17.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.247+0000 7f14df400640 1 -- 172.21.15.28:0/49703835 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f14e0012b90 con 0x7f14e80c5720 2024-02-21T23:03:17.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.248+0000 7f14df400640 1 --2- 172.21.15.28:0/49703835 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f14c406b500 0x7f14c406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:17.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.248+0000 7f14df400640 1 -- 172.21.15.28:0/49703835 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f14e008a990 con 0x7f14e80c5720 2024-02-21T23:03:17.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.248+0000 7f14e7400640 1 --2- 172.21.15.28:0/49703835 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f14c406b500 0x7f14c406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:17.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.249+0000 7f14e7400640 1 --2- 172.21.15.28:0/49703835 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f14c406b500 0x7f14c406d9c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f14d8035000 tx=0x7f14d8030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:17.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.252+0000 7f14df400640 1 -- 172.21.15.28:0/49703835 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f14e81539b0 con 0x7f14e80c5720 2024-02-21T23:03:17.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.494+0000 7f14ecc00640 1 -- 172.21.15.28:0/49703835 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f14e80c2090 con 0x7f14e80c5720 2024-02-21T23:03:17.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.496+0000 7f14df400640 1 -- 172.21.15.28:0/49703835 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4934 (secure 0 0 0) 0x7f14e0059440 con 0x7f14e80c5720 2024-02-21T23:03:17.486 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:03:17.486 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4230,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/2559793999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2559793999},{"type":"v1","addr":"172.21.15.28:6834","nonce":2559793999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4232,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3930208177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3930208177},{"type":"v1","addr":"172.21.15.28:6835","nonce":3930208177}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4238,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/2875984988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":2875984988},{"type":"v1","addr":"172.21.15.81:6836","nonce":2875984988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:02:37.812604+0000","modified":"2024-02-21T23:03:06.243463+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4267},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4267":{"gid":4267,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6837/2328151245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2328151245},{"type":"v1","addr":"172.21.15.81:6837","nonce":2328151245}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-02-21T23:03:17.486 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 9 2024-02-21T23:03:17.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.498+0000 7f14ecc00640 1 -- 172.21.15.28:0/49703835 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f14c406b500 msgr2=0x7f14c406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:17.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.498+0000 7f14ecc00640 1 --2- 172.21.15.28:0/49703835 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f14c406b500 0x7f14c406d9c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f14d8035000 tx=0x7f14d8030000 comp rx=0 tx=0).stop 2024-02-21T23:03:17.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.498+0000 7f14ecc00640 1 -- 172.21.15.28:0/49703835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14e80c5720 msgr2=0x7f14e81e7e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:17.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.498+0000 7f14ecc00640 1 --2- 172.21.15.28:0/49703835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14e80c5720 0x7f14e81e7e00 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f14e000d600 tx=0x7f14e000dac0 comp rx=0 tx=0).stop 2024-02-21T23:03:17.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.498+0000 7f14ecc00640 1 -- 172.21.15.28:0/49703835 shutdown_connections 2024-02-21T23:03:17.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.499+0000 7f14ecc00640 1 --2- 172.21.15.28:0/49703835 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f14c406b500 0x7f14c406d9c0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:17.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.499+0000 7f14ecc00640 1 --2- 172.21.15.28:0/49703835 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f14e80aff90 0x7f14e81e78c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:17.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.499+0000 7f14ecc00640 1 --2- 172.21.15.28:0/49703835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14e80c5720 0x7f14e81e7e00 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:17.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.499+0000 7f14ecc00640 1 --2- 172.21.15.28:0/49703835 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f14e806dbf0 0x7f14e80ae9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:17.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.499+0000 7f14ecc00640 1 -- 172.21.15.28:0/49703835 >> 172.21.15.28:0/49703835 conn(0x7f14e80ac370 msgr2=0x7f14e8156190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:17.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.499+0000 7f14ecc00640 1 -- 172.21.15.28:0/49703835 shutdown_connections 2024-02-21T23:03:17.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.499+0000 7f14ecc00640 1 -- 172.21.15.28:0/49703835 wait complete. 2024-02-21T23:03:17.507 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:03:17.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.735+0000 7fe263200640 1 Processor -- start 2024-02-21T23:03:17.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.735+0000 7fe263200640 1 -- start start 2024-02-21T23:03:17.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.737+0000 7fe263200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe264155960 0x7fe2641608a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:17.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.737+0000 7fe263200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2640c74c0 con 0x7fe26406dbf0 2024-02-21T23:03:17.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.737+0000 7fe263200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2640c77a0 con 0x7fe2640aff90 2024-02-21T23:03:17.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.737+0000 7fe263200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2640c7270 con 0x7fe264155960 2024-02-21T23:03:17.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.737+0000 7fe262800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe264155960 0x7fe2641608a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:17.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.737+0000 7fe262800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe264155960 0x7fe2641608a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45990/0 (socket says 172.21.15.28:45990) 2024-02-21T23:03:17.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.737+0000 7fe262800640 1 -- 172.21.15.28:0/3357141523 learned_addr learned my addr 172.21.15.28:0/3357141523 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:17.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.738+0000 7fe262800640 1 -- 172.21.15.28:0/3357141523 >> v1:172.21.15.81:6789/0 conn(0x7fe2640aff90 legacy=0x7fe2640c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:17.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.738+0000 7fe262800640 1 -- 172.21.15.28:0/3357141523 >> v1:172.21.15.28:6789/0 conn(0x7fe26406dbf0 legacy=0x7fe264158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:17.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.738+0000 7fe262800640 1 -- 172.21.15.28:0/3357141523 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2640c7a80 con 0x7fe264155960 2024-02-21T23:03:17.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.739+0000 7fe262800640 1 --2- 172.21.15.28:0/3357141523 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe264155960 0x7fe2641608a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe25c009e50 tx=0x7fe25c02c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ee686d6c5a456ae server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:17.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.739+0000 7fe260a00640 1 -- 172.21.15.28:0/3357141523 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe25c02cbb0 con 0x7fe264155960 2024-02-21T23:03:17.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.739+0000 7fe260a00640 1 -- 172.21.15.28:0/3357141523 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe25c02cd50 con 0x7fe264155960 2024-02-21T23:03:17.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.739+0000 7fe260a00640 1 -- 172.21.15.28:0/3357141523 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe25c034610 con 0x7fe264155960 2024-02-21T23:03:17.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.740+0000 7fe263200640 1 -- 172.21.15.28:0/3357141523 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe264155960 msgr2=0x7fe2641608a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:17.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.740+0000 7fe263200640 1 --2- 172.21.15.28:0/3357141523 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe264155960 0x7fe2641608a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe25c009e50 tx=0x7fe25c02c010 comp rx=0 tx=0).stop 2024-02-21T23:03:17.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.740+0000 7fe263200640 1 -- 172.21.15.28:0/3357141523 shutdown_connections 2024-02-21T23:03:17.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.740+0000 7fe263200640 1 --2- 172.21.15.28:0/3357141523 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe264155960 0x7fe2641608a0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:17.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.740+0000 7fe263200640 1 -- 172.21.15.28:0/3357141523 >> 172.21.15.28:0/3357141523 conn(0x7fe2640ac370 msgr2=0x7fe2640ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:17.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.740+0000 7fe263200640 1 -- 172.21.15.28:0/3357141523 shutdown_connections 2024-02-21T23:03:17.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.740+0000 7fe263200640 1 -- 172.21.15.28:0/3357141523 wait complete. 2024-02-21T23:03:17.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.742+0000 7fe263200640 1 Processor -- start 2024-02-21T23:03:17.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.742+0000 7fe263200640 1 -- start start 2024-02-21T23:03:17.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.743+0000 7fe263200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe26406dbf0 0x7fe2641eb470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:17.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.743+0000 7fe263200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe2640aff90 0x7fe2640ae740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:17.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.743+0000 7fe263200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe264155960 0x7fe264152d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:17.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.743+0000 7fe263200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2640c58f0 con 0x7fe264155960 2024-02-21T23:03:17.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.743+0000 7fe263200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe264158630 con 0x7fe26406dbf0 2024-02-21T23:03:17.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.743+0000 7fe263200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2640c5a70 con 0x7fe2640aff90 2024-02-21T23:03:17.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.743+0000 7fe262800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe264155960 0x7fe264152d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:17.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.743+0000 7fe262800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe264155960 0x7fe264152d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46002/0 (socket says 172.21.15.28:46002) 2024-02-21T23:03:17.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.743+0000 7fe262800640 1 -- 172.21.15.28:0/2353420845 learned_addr learned my addr 172.21.15.28:0/2353420845 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:17.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.744+0000 7fe261400640 1 --2- 172.21.15.28:0/2353420845 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe2640aff90 0x7fe2640ae740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:17.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.744+0000 7fe261e00640 1 --2- 172.21.15.28:0/2353420845 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe26406dbf0 0x7fe2641eb470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:17.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.744+0000 7fe262800640 1 -- 172.21.15.28:0/2353420845 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe2640aff90 msgr2=0x7fe2640ae740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:17.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.744+0000 7fe262800640 1 --2- 172.21.15.28:0/2353420845 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe2640aff90 0x7fe2640ae740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:17.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.744+0000 7fe262800640 1 -- 172.21.15.28:0/2353420845 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe26406dbf0 msgr2=0x7fe2641eb470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:17.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.744+0000 7fe262800640 1 --2- 172.21.15.28:0/2353420845 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe26406dbf0 0x7fe2641eb470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:17.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.744+0000 7fe262800640 1 -- 172.21.15.28:0/2353420845 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2641ea9b0 con 0x7fe264155960 2024-02-21T23:03:17.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.744+0000 7fe262800640 1 --2- 172.21.15.28:0/2353420845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe264155960 0x7fe264152d80 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fe25c009e50 tx=0x7fe25c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:17.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.744+0000 7fe25a000640 1 -- 172.21.15.28:0/2353420845 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe25c043020 con 0x7fe264155960 2024-02-21T23:03:17.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.745+0000 7fe25a000640 1 -- 172.21.15.28:0/2353420845 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe25c03e450 con 0x7fe264155960 2024-02-21T23:03:17.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.745+0000 7fe263200640 1 -- 172.21.15.28:0/2353420845 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe264154560 con 0x7fe264155960 2024-02-21T23:03:17.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.745+0000 7fe25a000640 1 -- 172.21.15.28:0/2353420845 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe25c009510 con 0x7fe264155960 2024-02-21T23:03:17.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.745+0000 7fe263200640 1 -- 172.21.15.28:0/2353420845 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2641eac70 con 0x7fe264155960 2024-02-21T23:03:17.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.746+0000 7fe263200640 1 -- 172.21.15.28:0/2353420845 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe26404f290 con 0x7fe264155960 2024-02-21T23:03:17.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.747+0000 7fe25a000640 1 -- 172.21.15.28:0/2353420845 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe25c046020 con 0x7fe264155960 2024-02-21T23:03:17.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.748+0000 7fe25a000640 1 --2- 172.21.15.28:0/2353420845 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe24406b4d0 0x7fe24406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:17.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.748+0000 7fe25a000640 1 -- 172.21.15.28:0/2353420845 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fe25c0acdd0 con 0x7fe264155960 2024-02-21T23:03:17.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.748+0000 7fe261e00640 1 --2- 172.21.15.28:0/2353420845 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe24406b4d0 0x7fe24406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:17.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.749+0000 7fe261e00640 1 --2- 172.21.15.28:0/2353420845 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe24406b4d0 0x7fe24406d990 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe254001640 tx=0x7fe254009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:17.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.752+0000 7fe25a000640 1 -- 172.21.15.28:0/2353420845 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe25c04d030 con 0x7fe264155960 2024-02-21T23:03:17.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.957+0000 7fe263200640 1 -- 172.21.15.28:0/2353420845 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe26406d9d0 con 0x7fe264155960 2024-02-21T23:03:17.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.959+0000 7fe25a000640 1 -- 172.21.15.28:0/2353420845 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18454 (secure 0 0 0) 0x7fe25c07b9a0 con 0x7fe264155960 2024-02-21T23:03:17.950 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:03:17.950 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":24,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:02:42.900286+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:02:35.795158+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-21T23:02:36.840110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-02-21T23:02:40.441388+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:03:17.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.962+0000 7fe263200640 1 -- 172.21.15.28:0/2353420845 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe24406b4d0 msgr2=0x7fe24406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:17.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.962+0000 7fe263200640 1 --2- 172.21.15.28:0/2353420845 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe24406b4d0 0x7fe24406d990 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe254001640 tx=0x7fe254009040 comp rx=0 tx=0).stop 2024-02-21T23:03:17.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.962+0000 7fe263200640 1 -- 172.21.15.28:0/2353420845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe264155960 msgr2=0x7fe264152d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:17.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.962+0000 7fe263200640 1 --2- 172.21.15.28:0/2353420845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe264155960 0x7fe264152d80 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fe25c009e50 tx=0x7fe25c004a40 comp rx=0 tx=0).stop 2024-02-21T23:03:17.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.962+0000 7fe263200640 1 -- 172.21.15.28:0/2353420845 shutdown_connections 2024-02-21T23:03:17.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.962+0000 7fe263200640 1 --2- 172.21.15.28:0/2353420845 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe24406b4d0 0x7fe24406d990 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:17.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.962+0000 7fe263200640 1 --2- 172.21.15.28:0/2353420845 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe2640aff90 0x7fe2640ae740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:17.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.962+0000 7fe263200640 1 --2- 172.21.15.28:0/2353420845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe264155960 0x7fe264152d80 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:17.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.962+0000 7fe263200640 1 --2- 172.21.15.28:0/2353420845 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe26406dbf0 0x7fe2641eb470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:17.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.962+0000 7fe263200640 1 -- 172.21.15.28:0/2353420845 >> 172.21.15.28:0/2353420845 conn(0x7fe2640ac370 msgr2=0x7fe2640c50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:17.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.963+0000 7fe263200640 1 -- 172.21.15.28:0/2353420845 shutdown_connections 2024-02-21T23:03:17.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:17.963+0000 7fe263200640 1 -- 172.21.15.28:0/2353420845 wait complete. 2024-02-21T23:03:17.973 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-21T23:02:30.728329+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-21T23:02:35.795158+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-02-21T23:02:36.840110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-02-21T23:02:40.441388+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-02-21T23:03:17.973 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-21T23:03:18.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.213+0000 7fc149800640 1 Processor -- start 2024-02-21T23:03:18.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.213+0000 7fc149800640 1 -- start start 2024-02-21T23:03:18.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.215+0000 7fc149800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fc14406b0e0 0x7fc1440c5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:18.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.215+0000 7fc149800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc1440c5d80 0x7fc14414fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:18.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.215+0000 7fc149800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1440c61e0 con 0x7fc144150530 2024-02-21T23:03:18.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.215+0000 7fc149800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc144153bc0 con 0x7fc1440c5d80 2024-02-21T23:03:18.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.215+0000 7fc149800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc144152fb0 con 0x7fc14406b0e0 2024-02-21T23:03:18.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.215+0000 7fc142000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc1440c5d80 0x7fc14414fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:18.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.215+0000 7fc142000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc1440c5d80 0x7fc14414fff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46004/0 (socket says 172.21.15.28:46004) 2024-02-21T23:03:18.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.215+0000 7fc142000640 1 -- 172.21.15.28:0/2121147055 learned_addr learned my addr 172.21.15.28:0/2121147055 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:18.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.215+0000 7fc142a00640 1 --2- 172.21.15.28:0/2121147055 >> v2:172.21.15.81:3300/0 conn(0x7fc14406b0e0 0x7fc1440c5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:18.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.216+0000 7fc142000640 1 -- 172.21.15.28:0/2121147055 >> v2:172.21.15.81:3300/0 conn(0x7fc14406b0e0 msgr2=0x7fc1440c5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:18.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.216+0000 7fc142000640 1 --2- 172.21.15.28:0/2121147055 >> v2:172.21.15.81:3300/0 conn(0x7fc14406b0e0 0x7fc1440c5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:18.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.216+0000 7fc142000640 1 -- 172.21.15.28:0/2121147055 >> v1:172.21.15.28:6789/0 conn(0x7fc144150530 legacy=0x7fc144156500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:18.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.216+0000 7fc142000640 1 -- 172.21.15.28:0/2121147055 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc144153290 con 0x7fc1440c5d80 2024-02-21T23:03:18.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.216+0000 7fc142a00640 1 --2- 172.21.15.28:0/2121147055 >> v2:172.21.15.81:3300/0 conn(0x7fc14406b0e0 0x7fc1440c5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:03:18.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.216+0000 7fc141600640 1 -- 172.21.15.28:0/2121147055 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2316481273 0 0) 0x7fc1440c61e0 con 0x7fc144150530 2024-02-21T23:03:18.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.217+0000 7fc142000640 1 --2- 172.21.15.28:0/2121147055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1440c5d80 0x7fc14414fff0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fc1440c77e0 tx=0x7fc13002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0c47c358461913 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:18.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.217+0000 7fc141600640 1 -- 172.21.15.28:0/2121147055 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc130035020 con 0x7fc1440c5d80 2024-02-21T23:03:18.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.217+0000 7fc141600640 1 -- 172.21.15.28:0/2121147055 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc130031870 con 0x7fc1440c5d80 2024-02-21T23:03:18.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.217+0000 7fc141600640 1 -- 172.21.15.28:0/2121147055 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc130031c40 con 0x7fc1440c5d80 2024-02-21T23:03:18.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.218+0000 7fc149800640 1 -- 172.21.15.28:0/2121147055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1440c5d80 msgr2=0x7fc14414fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:18.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.218+0000 7fc149800640 1 --2- 172.21.15.28:0/2121147055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1440c5d80 0x7fc14414fff0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fc1440c77e0 tx=0x7fc13002b910 comp rx=0 tx=0).stop 2024-02-21T23:03:18.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.218+0000 7fc149800640 1 -- 172.21.15.28:0/2121147055 shutdown_connections 2024-02-21T23:03:18.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.218+0000 7fc149800640 1 --2- 172.21.15.28:0/2121147055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1440c5d80 0x7fc14414fff0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:18.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.218+0000 7fc149800640 1 --2- 172.21.15.28:0/2121147055 >> v2:172.21.15.81:3300/0 conn(0x7fc14406b0e0 0x7fc1440c5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:18.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.218+0000 7fc149800640 1 -- 172.21.15.28:0/2121147055 >> 172.21.15.28:0/2121147055 conn(0x7fc1440a85a0 msgr2=0x7fc1440a8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:18.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.218+0000 7fc149800640 1 -- 172.21.15.28:0/2121147055 shutdown_connections 2024-02-21T23:03:18.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.218+0000 7fc149800640 1 -- 172.21.15.28:0/2121147055 wait complete. 2024-02-21T23:03:18.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.220+0000 7fc149800640 1 Processor -- start 2024-02-21T23:03:18.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.220+0000 7fc149800640 1 -- start start 2024-02-21T23:03:18.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.221+0000 7fc149800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc14406b0e0 0x7fc1441557f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:18.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.221+0000 7fc149800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc1440c5d80 0x7fc1441548b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:18.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.221+0000 7fc149800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc144150530 0x7fc144152000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:18.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.221+0000 7fc149800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc144150970 con 0x7fc14406b0e0 2024-02-21T23:03:18.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.221+0000 7fc149800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc144153850 con 0x7fc144150530 2024-02-21T23:03:18.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.221+0000 7fc149800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc14406b4c0 con 0x7fc1440c5d80 2024-02-21T23:03:18.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.221+0000 7fc142a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc14406b0e0 0x7fc1441557f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:18.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.221+0000 7fc142a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc14406b0e0 0x7fc1441557f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46012/0 (socket says 172.21.15.28:46012) 2024-02-21T23:03:18.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.221+0000 7fc142a00640 1 -- 172.21.15.28:0/2873003179 learned_addr learned my addr 172.21.15.28:0/2873003179 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:18.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.222+0000 7fc142000640 1 --2- 172.21.15.28:0/2873003179 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc1440c5d80 0x7fc1441548b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:18.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.222+0000 7fc142a00640 1 -- 172.21.15.28:0/2873003179 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc1440c5d80 msgr2=0x7fc1441548b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:18.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.222+0000 7fc143400640 1 --2- 172.21.15.28:0/2873003179 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc144150530 0x7fc144152000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:18.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.222+0000 7fc142a00640 1 --2- 172.21.15.28:0/2873003179 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc1440c5d80 0x7fc1441548b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:18.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.222+0000 7fc142a00640 1 -- 172.21.15.28:0/2873003179 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc144150530 msgr2=0x7fc144152000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:18.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.222+0000 7fc142a00640 1 --2- 172.21.15.28:0/2873003179 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc144150530 0x7fc144152000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:18.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.222+0000 7fc142a00640 1 -- 172.21.15.28:0/2873003179 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1440aba90 con 0x7fc14406b0e0 2024-02-21T23:03:18.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.222+0000 7fc143400640 1 --2- 172.21.15.28:0/2873003179 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc144150530 0x7fc144152000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:03:18.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.222+0000 7fc142a00640 1 --2- 172.21.15.28:0/2873003179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc14406b0e0 0x7fc1441557f0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fc13400e730 tx=0x7fc13400ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:18.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.222+0000 7fc13aa00640 1 -- 172.21.15.28:0/2873003179 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc13400f040 con 0x7fc14406b0e0 2024-02-21T23:03:18.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.222+0000 7fc13aa00640 1 -- 172.21.15.28:0/2873003179 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc134004230 con 0x7fc14406b0e0 2024-02-21T23:03:18.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.222+0000 7fc13aa00640 1 -- 172.21.15.28:0/2873003179 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc13400ee40 con 0x7fc14406b0e0 2024-02-21T23:03:18.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.222+0000 7fc149800640 1 -- 172.21.15.28:0/2873003179 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc144155e80 con 0x7fc14406b0e0 2024-02-21T23:03:18.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.222+0000 7fc149800640 1 -- 172.21.15.28:0/2873003179 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1440a9b80 con 0x7fc14406b0e0 2024-02-21T23:03:18.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.224+0000 7fc149800640 1 -- 172.21.15.28:0/2873003179 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc144067f30 con 0x7fc14406b0e0 2024-02-21T23:03:18.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.225+0000 7fc13aa00640 1 -- 172.21.15.28:0/2873003179 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc13400c6d0 con 0x7fc14406b0e0 2024-02-21T23:03:18.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.225+0000 7fc13aa00640 1 --2- 172.21.15.28:0/2873003179 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc12406b500 0x7fc12406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:18.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.226+0000 7fc13aa00640 1 -- 172.21.15.28:0/2873003179 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fc13408a1f0 con 0x7fc14406b0e0 2024-02-21T23:03:18.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.226+0000 7fc142000640 1 --2- 172.21.15.28:0/2873003179 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc12406b500 0x7fc12406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:18.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.227+0000 7fc142000640 1 --2- 172.21.15.28:0/2873003179 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc12406b500 0x7fc12406d9c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc130036000 tx=0x7fc130008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:18.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.231+0000 7fc13aa00640 1 -- 172.21.15.28:0/2873003179 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc134017020 con 0x7fc14406b0e0 2024-02-21T23:03:18.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.436+0000 7fc149800640 1 -- 172.21.15.28:0/2873003179 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc144152880 con 0x7fc14406b0e0 2024-02-21T23:03:18.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.437+0000 7fc13aa00640 1 -- 172.21.15.28:0/2873003179 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc134058dc0 con 0x7fc14406b0e0 2024-02-21T23:03:18.427 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-02-21T23:03:18.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.439+0000 7fc149800640 1 -- 172.21.15.28:0/2873003179 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc12406b500 msgr2=0x7fc12406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:18.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.439+0000 7fc149800640 1 --2- 172.21.15.28:0/2873003179 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc12406b500 0x7fc12406d9c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc130036000 tx=0x7fc130008030 comp rx=0 tx=0).stop 2024-02-21T23:03:18.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.439+0000 7fc149800640 1 -- 172.21.15.28:0/2873003179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc14406b0e0 msgr2=0x7fc1441557f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:18.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.439+0000 7fc149800640 1 --2- 172.21.15.28:0/2873003179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc14406b0e0 0x7fc1441557f0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fc13400e730 tx=0x7fc13400ebf0 comp rx=0 tx=0).stop 2024-02-21T23:03:18.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.440+0000 7fc149800640 1 -- 172.21.15.28:0/2873003179 shutdown_connections 2024-02-21T23:03:18.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.440+0000 7fc149800640 1 --2- 172.21.15.28:0/2873003179 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc12406b500 0x7fc12406d9c0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:18.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.440+0000 7fc149800640 1 --2- 172.21.15.28:0/2873003179 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc1440c5d80 0x7fc1441548b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:18.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.440+0000 7fc149800640 1 --2- 172.21.15.28:0/2873003179 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc144150530 0x7fc144152000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:18.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.440+0000 7fc149800640 1 --2- 172.21.15.28:0/2873003179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc14406b0e0 0x7fc1441557f0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:18.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.440+0000 7fc149800640 1 -- 172.21.15.28:0/2873003179 >> 172.21.15.28:0/2873003179 conn(0x7fc1440a85a0 msgr2=0x7fc14414b030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:18.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.440+0000 7fc149800640 1 -- 172.21.15.28:0/2873003179 shutdown_connections 2024-02-21T23:03:18.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.440+0000 7fc149800640 1 -- 172.21.15.28:0/2873003179 wait complete. 2024-02-21T23:03:18.448 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-21T23:03:18.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.681+0000 7ffbb3000640 1 Processor -- start 2024-02-21T23:03:18.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.681+0000 7ffbb3000640 1 -- start start 2024-02-21T23:03:18.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.683+0000 7ffbb3000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7ffbac150240 0x7ffbac0c12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:18.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.683+0000 7ffbb3000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffbac0c4580 0x7ffbac06db10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:18.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.683+0000 7ffbb3000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbac06f780 con 0x7ffbac0c2c10 2024-02-21T23:03:18.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.683+0000 7ffbb3000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbac150620 con 0x7ffbac150240 2024-02-21T23:03:18.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.683+0000 7ffbb3000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbac153a90 con 0x7ffbac0c4580 2024-02-21T23:03:18.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.683+0000 7ffbb1200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffbac0c4580 0x7ffbac06db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:18.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.683+0000 7ffbb1200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffbac0c4580 0x7ffbac06db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3301/0 says I am v2:172.21.15.28:57512/0 (socket says 172.21.15.28:57512) 2024-02-21T23:03:18.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.683+0000 7ffbb1200640 1 -- 172.21.15.28:0/2035248994 learned_addr learned my addr 172.21.15.28:0/2035248994 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:18.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.683+0000 7ffbb1c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7ffbac150240 0x7ffbac0c12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:18.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.684+0000 7ffbb1200640 1 -- 172.21.15.28:0/2035248994 >> v2:172.21.15.81:3300/0 conn(0x7ffbac150240 msgr2=0x7ffbac0c12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:18.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.684+0000 7ffbb1200640 1 --2- 172.21.15.28:0/2035248994 >> v2:172.21.15.81:3300/0 conn(0x7ffbac150240 0x7ffbac0c12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:18.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.684+0000 7ffbb1200640 1 -- 172.21.15.28:0/2035248994 >> v1:172.21.15.81:6789/0 conn(0x7ffbac0c2c10 legacy=0x7ffbac06ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:18.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.684+0000 7ffbb1200640 1 -- 172.21.15.28:0/2035248994 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbac06f4a0 con 0x7ffbac0c4580 2024-02-21T23:03:18.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.686+0000 7ffbb1200640 1 --2- 172.21.15.28:0/2035248994 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffbac0c4580 0x7ffbac06db10 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ffb98009870 tx=0x7ffb9802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b38840649e7e824 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:18.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.687+0000 7ffbabe00640 1 -- 172.21.15.28:0/2035248994 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb98035020 con 0x7ffbac0c4580 2024-02-21T23:03:18.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.687+0000 7ffbabe00640 1 -- 172.21.15.28:0/2035248994 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb98031640 con 0x7ffbac0c4580 2024-02-21T23:03:18.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.687+0000 7ffbabe00640 1 -- 172.21.15.28:0/2035248994 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb9802d040 con 0x7ffbac0c4580 2024-02-21T23:03:18.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.687+0000 7ffbb3000640 1 -- 172.21.15.28:0/2035248994 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffbac0c4580 msgr2=0x7ffbac06db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:18.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.687+0000 7ffbb3000640 1 --2- 172.21.15.28:0/2035248994 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffbac0c4580 0x7ffbac06db10 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ffb98009870 tx=0x7ffb9802b8f0 comp rx=0 tx=0).stop 2024-02-21T23:03:18.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.688+0000 7ffbb3000640 1 -- 172.21.15.28:0/2035248994 shutdown_connections 2024-02-21T23:03:18.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.688+0000 7ffbb3000640 1 --2- 172.21.15.28:0/2035248994 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffbac0c4580 0x7ffbac06db10 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:18.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.688+0000 7ffbb3000640 1 --2- 172.21.15.28:0/2035248994 >> v2:172.21.15.81:3300/0 conn(0x7ffbac150240 0x7ffbac0c12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:18.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.688+0000 7ffbb3000640 1 -- 172.21.15.28:0/2035248994 >> 172.21.15.28:0/2035248994 conn(0x7ffbac0a8530 msgr2=0x7ffbac0a8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:18.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.688+0000 7ffbb3000640 1 -- 172.21.15.28:0/2035248994 shutdown_connections 2024-02-21T23:03:18.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.688+0000 7ffbb3000640 1 -- 172.21.15.28:0/2035248994 wait complete. 2024-02-21T23:03:18.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.689+0000 7ffbb3000640 1 Processor -- start 2024-02-21T23:03:18.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.690+0000 7ffbb3000640 1 -- start start 2024-02-21T23:03:18.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.690+0000 7ffbb3000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbac0c2c10 0x7ffbac0c5900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:18.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.690+0000 7ffbb3000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffbac0c4580 0x7ffbac0bfe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:18.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.690+0000 7ffbb3000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffbac150240 0x7ffbac0713a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:18.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.690+0000 7ffbb3000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbac0c4b90 con 0x7ffbac0c2c10 2024-02-21T23:03:18.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.690+0000 7ffbb3000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbac0c4a10 con 0x7ffbac150240 2024-02-21T23:03:18.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.690+0000 7ffbb3000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbac06b580 con 0x7ffbac0c4580 2024-02-21T23:03:18.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.691+0000 7ffbb1c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbac0c2c10 0x7ffbac0c5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:18.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.691+0000 7ffbb1c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbac0c2c10 0x7ffbac0c5900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46014/0 (socket says 172.21.15.28:46014) 2024-02-21T23:03:18.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.691+0000 7ffbb1c00640 1 -- 172.21.15.28:0/253359549 learned_addr learned my addr 172.21.15.28:0/253359549 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:18.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.691+0000 7ffbb2600640 1 --2- 172.21.15.28:0/253359549 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffbac150240 0x7ffbac0713a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:18.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.691+0000 7ffbb1200640 1 --2- 172.21.15.28:0/253359549 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffbac0c4580 0x7ffbac0bfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:18.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.691+0000 7ffbb1c00640 1 -- 172.21.15.28:0/253359549 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffbac0c4580 msgr2=0x7ffbac0bfe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:18.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.691+0000 7ffbb1c00640 1 --2- 172.21.15.28:0/253359549 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffbac0c4580 0x7ffbac0bfe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:18.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.691+0000 7ffbb1c00640 1 -- 172.21.15.28:0/253359549 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffbac150240 msgr2=0x7ffbac0713a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:18.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.691+0000 7ffbb1c00640 1 --2- 172.21.15.28:0/253359549 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffbac150240 0x7ffbac0713a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:18.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.691+0000 7ffbb1c00640 1 -- 172.21.15.28:0/253359549 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbac0726d0 con 0x7ffbac0c2c10 2024-02-21T23:03:18.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.691+0000 7ffbb2600640 1 --2- 172.21.15.28:0/253359549 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffbac150240 0x7ffbac0713a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:03:18.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.692+0000 7ffbb1c00640 1 --2- 172.21.15.28:0/253359549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbac0c2c10 0x7ffbac0c5900 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ffba000e9b0 tx=0x7ffba000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:18.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.692+0000 7ffba9600640 1 -- 172.21.15.28:0/253359549 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffba0017070 con 0x7ffbac0c2c10 2024-02-21T23:03:18.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.692+0000 7ffba9600640 1 -- 172.21.15.28:0/253359549 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffba000c9a0 con 0x7ffbac0c2c10 2024-02-21T23:03:18.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.692+0000 7ffba9600640 1 -- 172.21.15.28:0/253359549 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffba000cd90 con 0x7ffbac0c2c10 2024-02-21T23:03:18.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.692+0000 7ffbb3000640 1 -- 172.21.15.28:0/253359549 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbac1eb540 con 0x7ffbac0c2c10 2024-02-21T23:03:18.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.692+0000 7ffbb3000640 1 -- 172.21.15.28:0/253359549 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbac1eb820 con 0x7ffbac0c2c10 2024-02-21T23:03:18.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.693+0000 7ffbb3000640 1 -- 172.21.15.28:0/253359549 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbac074ab0 con 0x7ffbac0c2c10 2024-02-21T23:03:18.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.694+0000 7ffba9600640 1 -- 172.21.15.28:0/253359549 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffba001a070 con 0x7ffbac0c2c10 2024-02-21T23:03:18.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.695+0000 7ffba9600640 1 --2- 172.21.15.28:0/253359549 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ffb8c06b4d0 0x7ffb8c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:18.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.695+0000 7ffba9600640 1 -- 172.21.15.28:0/253359549 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7ffba008a540 con 0x7ffbac0c2c10 2024-02-21T23:03:18.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.696+0000 7ffbb1200640 1 --2- 172.21.15.28:0/253359549 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ffb8c06b4d0 0x7ffb8c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:18.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.697+0000 7ffbb1200640 1 --2- 172.21.15.28:0/253359549 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ffb8c06b4d0 0x7ffb8c06d990 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ffb98036000 tx=0x7ffb98008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:18.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.700+0000 7ffba9600640 1 -- 172.21.15.28:0/253359549 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ffbac074ab0 con 0x7ffbac0c2c10 2024-02-21T23:03:18.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.899+0000 7ffbb3000640 1 -- 172.21.15.28:0/253359549 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ffbac04f290 con 0x7ffbac0c2c10 2024-02-21T23:03:18.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.900+0000 7ffba9600640 1 -- 172.21.15.28:0/253359549 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7ffba0059070 con 0x7ffbac0c2c10 2024-02-21T23:03:18.890 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-02-21T23:03:18.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.902+0000 7ffbb3000640 1 -- 172.21.15.28:0/253359549 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ffb8c06b4d0 msgr2=0x7ffb8c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:18.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.902+0000 7ffbb3000640 1 --2- 172.21.15.28:0/253359549 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ffb8c06b4d0 0x7ffb8c06d990 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ffb98036000 tx=0x7ffb98008030 comp rx=0 tx=0).stop 2024-02-21T23:03:18.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.902+0000 7ffbb3000640 1 -- 172.21.15.28:0/253359549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbac0c2c10 msgr2=0x7ffbac0c5900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:18.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.902+0000 7ffbb3000640 1 --2- 172.21.15.28:0/253359549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbac0c2c10 0x7ffbac0c5900 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ffba000e9b0 tx=0x7ffba000ee70 comp rx=0 tx=0).stop 2024-02-21T23:03:18.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.903+0000 7ffbb3000640 1 -- 172.21.15.28:0/253359549 shutdown_connections 2024-02-21T23:03:18.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.903+0000 7ffbb3000640 1 --2- 172.21.15.28:0/253359549 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ffb8c06b4d0 0x7ffb8c06d990 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:18.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.903+0000 7ffbb3000640 1 --2- 172.21.15.28:0/253359549 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffbac0c4580 0x7ffbac0bfe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:18.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.903+0000 7ffbb3000640 1 --2- 172.21.15.28:0/253359549 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffbac150240 0x7ffbac0713a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:18.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.903+0000 7ffbb3000640 1 --2- 172.21.15.28:0/253359549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbac0c2c10 0x7ffbac0c5900 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:18.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.903+0000 7ffbb3000640 1 -- 172.21.15.28:0/253359549 >> 172.21.15.28:0/253359549 conn(0x7ffbac0a8530 msgr2=0x7ffbac0a8910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:18.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.903+0000 7ffbb3000640 1 -- 172.21.15.28:0/253359549 shutdown_connections 2024-02-21T23:03:18.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:18.903+0000 7ffbb3000640 1 -- 172.21.15.28:0/253359549 wait complete. 2024-02-21T23:03:18.911 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-21T23:03:19.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.135+0000 7fdf11200640 1 Processor -- start 2024-02-21T23:03:19.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.135+0000 7fdf11200640 1 -- start start 2024-02-21T23:03:19.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.137+0000 7fdf11200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdf0c06dbc0 0x7fdf0c06c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:19.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.137+0000 7fdf11200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fdf0c13e520 0x7fdf0c06c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:19.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.137+0000 7fdf11200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf0c158550 con 0x7fdf0c13e0c0 2024-02-21T23:03:19.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.137+0000 7fdf11200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf0c0c89a0 con 0x7fdf0c13e520 2024-02-21T23:03:19.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.137+0000 7fdf11200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf0c1587d0 con 0x7fdf0c06dbc0 2024-02-21T23:03:19.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.137+0000 7fdf0b400640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fdf0c13e0c0 0x7fdf0c0c71b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.28:51988/0 (socket says 172.21.15.28:51988) 2024-02-21T23:03:19.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.137+0000 7fdf0b400640 1 -- 172.21.15.28:0/3535554634 learned_addr learned my addr 172.21.15.28:0/3535554634 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:03:19.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.138+0000 7fdf0aa00640 1 --2- 172.21.15.28:0/3535554634 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdf0c06dbc0 0x7fdf0c06c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:19.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.138+0000 7fdf0be00640 1 --2- 172.21.15.28:0/3535554634 >> v2:172.21.15.81:3300/0 conn(0x7fdf0c13e520 0x7fdf0c06c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:19.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.138+0000 7fdf0a000640 1 -- 172.21.15.28:0/3535554634 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3334564749 0 0) 0x7fdefc001110 con 0x7fdf0c13e0c0 2024-02-21T23:03:19.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.138+0000 7fdf0a000640 1 -- 172.21.15.28:0/3535554634 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdeec003520 con 0x7fdf0c13e0c0 2024-02-21T23:03:19.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.138+0000 7fdf0aa00640 1 -- 172.21.15.28:0/3535554634 >> v2:172.21.15.81:3300/0 conn(0x7fdf0c13e520 msgr2=0x7fdf0c06c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:19.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.138+0000 7fdf0aa00640 1 --2- 172.21.15.28:0/3535554634 >> v2:172.21.15.81:3300/0 conn(0x7fdf0c13e520 0x7fdf0c06c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:19.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.139+0000 7fdf0aa00640 1 -- 172.21.15.28:0/3535554634 >> v1:172.21.15.81:6789/0 conn(0x7fdf0c13e0c0 legacy=0x7fdf0c0c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:19.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.139+0000 7fdf0aa00640 1 -- 172.21.15.28:0/3535554634 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf0c158ab0 con 0x7fdf0c06dbc0 2024-02-21T23:03:19.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.139+0000 7fdf0be00640 1 --2- 172.21.15.28:0/3535554634 >> v2:172.21.15.81:3300/0 conn(0x7fdf0c13e520 0x7fdf0c06c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-21T23:03:19.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.140+0000 7fdf0aa00640 1 --2- 172.21.15.28:0/3535554634 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdf0c06dbc0 0x7fdf0c06c280 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fdef40097a0 tx=0x7fdef402b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78f7bbb8bcadb2ce server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:19.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.140+0000 7fdf0a000640 1 -- 172.21.15.28:0/3535554634 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdef4031780 con 0x7fdf0c06dbc0 2024-02-21T23:03:19.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.140+0000 7fdf0a000640 1 -- 172.21.15.28:0/3535554634 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdef4037070 con 0x7fdf0c06dbc0 2024-02-21T23:03:19.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.140+0000 7fdf0a000640 1 -- 172.21.15.28:0/3535554634 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdef4031b20 con 0x7fdf0c06dbc0 2024-02-21T23:03:19.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.141+0000 7fdf11200640 1 -- 172.21.15.28:0/3535554634 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdf0c06dbc0 msgr2=0x7fdf0c06c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:19.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.141+0000 7fdf11200640 1 --2- 172.21.15.28:0/3535554634 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdf0c06dbc0 0x7fdf0c06c280 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fdef40097a0 tx=0x7fdef402b820 comp rx=0 tx=0).stop 2024-02-21T23:03:19.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.141+0000 7fdf11200640 1 -- 172.21.15.28:0/3535554634 shutdown_connections 2024-02-21T23:03:19.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.141+0000 7fdf11200640 1 --2- 172.21.15.28:0/3535554634 >> v2:172.21.15.81:3300/0 conn(0x7fdf0c13e520 0x7fdf0c06c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:19.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.141+0000 7fdf11200640 1 --2- 172.21.15.28:0/3535554634 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdf0c06dbc0 0x7fdf0c06c280 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:19.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.141+0000 7fdf11200640 1 -- 172.21.15.28:0/3535554634 >> 172.21.15.28:0/3535554634 conn(0x7fdf0c073110 msgr2=0x7fdf0c0ac490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:19.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.141+0000 7fdf11200640 1 -- 172.21.15.28:0/3535554634 shutdown_connections 2024-02-21T23:03:19.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.141+0000 7fdf11200640 1 -- 172.21.15.28:0/3535554634 wait complete. 2024-02-21T23:03:19.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.143+0000 7fdf11200640 1 Processor -- start 2024-02-21T23:03:19.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.143+0000 7fdf11200640 1 -- start start 2024-02-21T23:03:19.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.144+0000 7fdf11200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdf0c06dbc0 0x7fdf0c1eb380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:19.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.144+0000 7fdf11200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf0c13e0c0 0x7fdf0c1eb8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:19.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.144+0000 7fdf11200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf0c13e520 0x7fdf0c135910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:19.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.144+0000 7fdf11200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf0c0c78c0 con 0x7fdf0c13e0c0 2024-02-21T23:03:19.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.144+0000 7fdf11200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf0c159070 con 0x7fdf0c13e520 2024-02-21T23:03:19.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.144+0000 7fdf11200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf0c0c7a40 con 0x7fdf0c06dbc0 2024-02-21T23:03:19.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.144+0000 7fdf0aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf0c13e0c0 0x7fdf0c1eb8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:19.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.144+0000 7fdf0aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf0c13e0c0 0x7fdf0c1eb8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46030/0 (socket says 172.21.15.28:46030) 2024-02-21T23:03:19.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.144+0000 7fdf0aa00640 1 -- 172.21.15.28:0/155640002 learned_addr learned my addr 172.21.15.28:0/155640002 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:19.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.144+0000 7fdf0b400640 1 --2- 172.21.15.28:0/155640002 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdf0c06dbc0 0x7fdf0c1eb380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:19.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.144+0000 7fdf0be00640 1 --2- 172.21.15.28:0/155640002 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf0c13e520 0x7fdf0c135910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:19.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.145+0000 7fdf0aa00640 1 -- 172.21.15.28:0/155640002 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdf0c06dbc0 msgr2=0x7fdf0c1eb380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:19.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.145+0000 7fdf0aa00640 1 --2- 172.21.15.28:0/155640002 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdf0c06dbc0 0x7fdf0c1eb380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:19.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.145+0000 7fdf0aa00640 1 -- 172.21.15.28:0/155640002 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf0c13e520 msgr2=0x7fdf0c135910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:19.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.145+0000 7fdf0aa00640 1 --2- 172.21.15.28:0/155640002 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf0c13e520 0x7fdf0c135910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:19.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.145+0000 7fdf0aa00640 1 -- 172.21.15.28:0/155640002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf0c0ae1c0 con 0x7fdf0c13e0c0 2024-02-21T23:03:19.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.145+0000 7fdf0aa00640 1 --2- 172.21.15.28:0/155640002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf0c13e0c0 0x7fdf0c1eb8c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fdef4000c00 tx=0x7fdef40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:19.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.145+0000 7fdf03400640 1 -- 172.21.15.28:0/155640002 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdef4031780 con 0x7fdf0c13e0c0 2024-02-21T23:03:19.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.145+0000 7fdf03400640 1 -- 172.21.15.28:0/155640002 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdef4031920 con 0x7fdf0c13e0c0 2024-02-21T23:03:19.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.145+0000 7fdf11200640 1 -- 172.21.15.28:0/155640002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf0c1ebf00 con 0x7fdf0c13e0c0 2024-02-21T23:03:19.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.145+0000 7fdf03400640 1 -- 172.21.15.28:0/155640002 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdef4031cf0 con 0x7fdf0c13e0c0 2024-02-21T23:03:19.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.146+0000 7fdf11200640 1 -- 172.21.15.28:0/155640002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf0c0ae760 con 0x7fdf0c13e0c0 2024-02-21T23:03:19.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.147+0000 7fdf11200640 1 -- 172.21.15.28:0/155640002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf0c155970 con 0x7fdf0c13e0c0 2024-02-21T23:03:19.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.148+0000 7fdf03400640 1 -- 172.21.15.28:0/155640002 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdef4032410 con 0x7fdf0c13e0c0 2024-02-21T23:03:19.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.149+0000 7fdf03400640 1 --2- 172.21.15.28:0/155640002 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdee806b4d0 0x7fdee806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:19.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.149+0000 7fdf03400640 1 -- 172.21.15.28:0/155640002 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fdef40acb70 con 0x7fdf0c13e0c0 2024-02-21T23:03:19.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.149+0000 7fdf0b400640 1 --2- 172.21.15.28:0/155640002 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdee806b4d0 0x7fdee806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:19.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.150+0000 7fdf0b400640 1 --2- 172.21.15.28:0/155640002 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdee806b4d0 0x7fdee806d990 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fdefc007fd0 tx=0x7fdefc009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:19.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.153+0000 7fdf03400640 1 -- 172.21.15.28:0/155640002 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fdef402f050 con 0x7fdf0c13e0c0 2024-02-21T23:03:19.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.344+0000 7fdf11200640 1 -- 172.21.15.28:0/155640002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fdf0c04f290 con 0x7fdf0c13e0c0 2024-02-21T23:03:19.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.345+0000 7fdf03400640 1 -- 172.21.15.28:0/155640002 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7fdef4031e90 con 0x7fdf0c13e0c0 2024-02-21T23:03:19.335 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-02-21T23:03:19.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.348+0000 7fdf11200640 1 -- 172.21.15.28:0/155640002 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdee806b4d0 msgr2=0x7fdee806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:19.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.348+0000 7fdf11200640 1 --2- 172.21.15.28:0/155640002 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdee806b4d0 0x7fdee806d990 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fdefc007fd0 tx=0x7fdefc009040 comp rx=0 tx=0).stop 2024-02-21T23:03:19.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.348+0000 7fdf11200640 1 -- 172.21.15.28:0/155640002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf0c13e0c0 msgr2=0x7fdf0c1eb8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:19.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.348+0000 7fdf11200640 1 --2- 172.21.15.28:0/155640002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf0c13e0c0 0x7fdf0c1eb8c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fdef4000c00 tx=0x7fdef40046c0 comp rx=0 tx=0).stop 2024-02-21T23:03:19.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.348+0000 7fdf11200640 1 -- 172.21.15.28:0/155640002 shutdown_connections 2024-02-21T23:03:19.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.348+0000 7fdf11200640 1 --2- 172.21.15.28:0/155640002 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdee806b4d0 0x7fdee806d990 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:19.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.348+0000 7fdf11200640 1 --2- 172.21.15.28:0/155640002 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf0c13e520 0x7fdf0c135910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:19.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.348+0000 7fdf11200640 1 --2- 172.21.15.28:0/155640002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf0c13e0c0 0x7fdf0c1eb8c0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:19.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.348+0000 7fdf11200640 1 --2- 172.21.15.28:0/155640002 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdf0c06dbc0 0x7fdf0c1eb380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:19.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.348+0000 7fdf11200640 1 -- 172.21.15.28:0/155640002 >> 172.21.15.28:0/155640002 conn(0x7fdf0c073110 msgr2=0x7fdf0c13cbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:19.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.349+0000 7fdf11200640 1 -- 172.21.15.28:0/155640002 shutdown_connections 2024-02-21T23:03:19.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.349+0000 7fdf11200640 1 -- 172.21.15.28:0/155640002 wait complete. 2024-02-21T23:03:19.356 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-02-21T23:03:19.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.589+0000 7f89ebe00640 1 Processor -- start 2024-02-21T23:03:19.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.589+0000 7f89ebe00640 1 -- start start 2024-02-21T23:03:19.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.591+0000 7f89ebe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f89ec06c1f0 0x7f89ec0c6260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:19.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.591+0000 7f89ebe00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f89ec14f520 0x7f89ec155450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:19.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.591+0000 7f89ebe00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89ec0c2530 con 0x7f89ec0c70b0 2024-02-21T23:03:19.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.591+0000 7f89ebe00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89ec0c22e0 con 0x7f89ec06c1f0 2024-02-21T23:03:19.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.591+0000 7f89ebe00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89ec0c2810 con 0x7f89ec14f520 2024-02-21T23:03:19.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.591+0000 7f89eaa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f89ec06c1f0 0x7f89ec0c6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:19.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.591+0000 7f89ea000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f89ec0c70b0 0x7f89ec14ee10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39628/0 (socket says 172.21.15.28:39628) 2024-02-21T23:03:19.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.592+0000 7f89ea000640 1 -- 172.21.15.28:0/2794025098 learned_addr learned my addr 172.21.15.28:0/2794025098 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:03:19.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.592+0000 7f89eb400640 1 --2- 172.21.15.28:0/2794025098 >> v2:172.21.15.81:3300/0 conn(0x7f89ec14f520 0x7f89ec155450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:19.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.592+0000 7f89eaa00640 1 -- 172.21.15.28:0/2794025098 >> v2:172.21.15.81:3300/0 conn(0x7f89ec14f520 msgr2=0x7f89ec155450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:19.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.592+0000 7f89eaa00640 1 --2- 172.21.15.28:0/2794025098 >> v2:172.21.15.81:3300/0 conn(0x7f89ec14f520 0x7f89ec155450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:19.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.592+0000 7f89eaa00640 1 -- 172.21.15.28:0/2794025098 >> v1:172.21.15.28:6789/0 conn(0x7f89ec0c70b0 legacy=0x7f89ec14ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:19.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.592+0000 7f89eaa00640 1 -- 172.21.15.28:0/2794025098 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89ec0c2af0 con 0x7f89ec06c1f0 2024-02-21T23:03:19.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.592+0000 7f89e9600640 1 -- 172.21.15.28:0/2794025098 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 106712820 0 0) 0x7f89ec0c2530 con 0x7f89ec0c70b0 2024-02-21T23:03:19.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.593+0000 7f89eaa00640 1 --2- 172.21.15.28:0/2794025098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89ec06c1f0 0x7f89ec0c6260 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f89ec0c3d00 tx=0x7f89dc02b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd95e9c283f6e961 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:19.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.594+0000 7f89e9600640 1 -- 172.21.15.28:0/2794025098 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89dc038070 con 0x7f89ec06c1f0 2024-02-21T23:03:19.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.594+0000 7f89e9600640 1 -- 172.21.15.28:0/2794025098 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89dc031710 con 0x7f89ec06c1f0 2024-02-21T23:03:19.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.594+0000 7f89e9600640 1 -- 172.21.15.28:0/2794025098 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89dc031ae0 con 0x7f89ec06c1f0 2024-02-21T23:03:19.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.594+0000 7f89ebe00640 1 -- 172.21.15.28:0/2794025098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89ec06c1f0 msgr2=0x7f89ec0c6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:19.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.594+0000 7f89ebe00640 1 --2- 172.21.15.28:0/2794025098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89ec06c1f0 0x7f89ec0c6260 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f89ec0c3d00 tx=0x7f89dc02b7b0 comp rx=0 tx=0).stop 2024-02-21T23:03:19.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.594+0000 7f89ebe00640 1 -- 172.21.15.28:0/2794025098 shutdown_connections 2024-02-21T23:03:19.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.594+0000 7f89ebe00640 1 --2- 172.21.15.28:0/2794025098 >> v2:172.21.15.81:3300/0 conn(0x7f89ec14f520 0x7f89ec155450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:19.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.594+0000 7f89ebe00640 1 --2- 172.21.15.28:0/2794025098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89ec06c1f0 0x7f89ec0c6260 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:19.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.594+0000 7f89ebe00640 1 -- 172.21.15.28:0/2794025098 >> 172.21.15.28:0/2794025098 conn(0x7f89ec0a9690 msgr2=0x7f89ec0a9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:19.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.595+0000 7f89ebe00640 1 -- 172.21.15.28:0/2794025098 shutdown_connections 2024-02-21T23:03:19.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.595+0000 7f89ebe00640 1 -- 172.21.15.28:0/2794025098 wait complete. 2024-02-21T23:03:19.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.596+0000 7f89ebe00640 1 Processor -- start 2024-02-21T23:03:19.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.596+0000 7f89ebe00640 1 -- start start 2024-02-21T23:03:19.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.597+0000 7f89ebe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89ec06c1f0 0x7f89ec152680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:19.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.597+0000 7f89ebe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f89ec0c70b0 0x7f89ec1eae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:19.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.597+0000 7f89ebe00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f89ec14f520 0x7f89ec1eb360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:19.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.597+0000 7f89ebe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89ec06c5d0 con 0x7f89ec06c1f0 2024-02-21T23:03:19.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.597+0000 7f89ebe00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89ec0c30b0 con 0x7f89ec0c70b0 2024-02-21T23:03:19.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.597+0000 7f89ebe00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89ec06c750 con 0x7f89ec14f520 2024-02-21T23:03:19.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.597+0000 7f89eaa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89ec06c1f0 0x7f89ec152680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:19.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.597+0000 7f89eaa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89ec06c1f0 0x7f89ec152680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46050/0 (socket says 172.21.15.28:46050) 2024-02-21T23:03:19.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.597+0000 7f89eaa00640 1 -- 172.21.15.28:0/1802267208 learned_addr learned my addr 172.21.15.28:0/1802267208 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:19.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.598+0000 7f89eb400640 1 --2- 172.21.15.28:0/1802267208 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f89ec14f520 0x7f89ec1eb360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:19.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.598+0000 7f89ea000640 1 --2- 172.21.15.28:0/1802267208 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f89ec0c70b0 0x7f89ec1eae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:19.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.598+0000 7f89eaa00640 1 -- 172.21.15.28:0/1802267208 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f89ec14f520 msgr2=0x7f89ec1eb360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:19.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.598+0000 7f89eaa00640 1 --2- 172.21.15.28:0/1802267208 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f89ec14f520 0x7f89ec1eb360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:19.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.598+0000 7f89eaa00640 1 -- 172.21.15.28:0/1802267208 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f89ec0c70b0 msgr2=0x7f89ec1eae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:19.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.598+0000 7f89eaa00640 1 --2- 172.21.15.28:0/1802267208 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f89ec0c70b0 0x7f89ec1eae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:19.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.598+0000 7f89eaa00640 1 -- 172.21.15.28:0/1802267208 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89ec0ac4e0 con 0x7f89ec06c1f0 2024-02-21T23:03:19.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.598+0000 7f89eaa00640 1 --2- 172.21.15.28:0/1802267208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89ec06c1f0 0x7f89ec152680 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f89dc009890 tx=0x7f89dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:19.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.599+0000 7f89e2a00640 1 -- 172.21.15.28:0/1802267208 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89dc042030 con 0x7f89ec06c1f0 2024-02-21T23:03:19.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.599+0000 7f89e2a00640 1 -- 172.21.15.28:0/1802267208 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89dc03d670 con 0x7f89ec06c1f0 2024-02-21T23:03:19.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.599+0000 7f89e2a00640 1 -- 172.21.15.28:0/1802267208 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89dc038040 con 0x7f89ec06c1f0 2024-02-21T23:03:19.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.599+0000 7f89ebe00640 1 -- 172.21.15.28:0/1802267208 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89ec0ac7c0 con 0x7f89ec06c1f0 2024-02-21T23:03:19.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.599+0000 7f89ebe00640 1 -- 172.21.15.28:0/1802267208 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89ec1ea900 con 0x7f89ec06c1f0 2024-02-21T23:03:19.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.600+0000 7f89ebe00640 1 -- 172.21.15.28:0/1802267208 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89ec1eab00 con 0x7f89ec06c1f0 2024-02-21T23:03:19.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.601+0000 7f89e2a00640 1 -- 172.21.15.28:0/1802267208 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f89dc045070 con 0x7f89ec06c1f0 2024-02-21T23:03:19.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.602+0000 7f89e2a00640 1 --2- 172.21.15.28:0/1802267208 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f89cc06b4d0 0x7f89cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:19.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.602+0000 7f89e2a00640 1 -- 172.21.15.28:0/1802267208 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f89dc0abca0 con 0x7f89ec06c1f0 2024-02-21T23:03:19.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.602+0000 7f89ea000640 1 --2- 172.21.15.28:0/1802267208 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f89cc06b4d0 0x7f89cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:19.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.603+0000 7f89ea000640 1 --2- 172.21.15.28:0/1802267208 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f89cc06b4d0 0x7f89cc06d990 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f89d8007fd0 tx=0x7f89d8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:19.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.606+0000 7f89e2a00640 1 -- 172.21.15.28:0/1802267208 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f89ec1eab00 con 0x7f89ec06c1f0 2024-02-21T23:03:19.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.805+0000 7f89ebe00640 1 -- 172.21.15.28:0/1802267208 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f89ec04f290 con 0x7f89ec06c1f0 2024-02-21T23:03:19.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.806+0000 7f89e2a00640 1 -- 172.21.15.28:0/1802267208 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) v1 ==== 103+0+11 (secure 0 0 0) 0x7f89dc04c070 con 0x7f89ec06c1f0 2024-02-21T23:03:19.796 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-02-21T23:03:19.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.808+0000 7f89ebe00640 1 -- 172.21.15.28:0/1802267208 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f89cc06b4d0 msgr2=0x7f89cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:19.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.809+0000 7f89ebe00640 1 --2- 172.21.15.28:0/1802267208 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f89cc06b4d0 0x7f89cc06d990 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f89d8007fd0 tx=0x7f89d8009040 comp rx=0 tx=0).stop 2024-02-21T23:03:19.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.809+0000 7f89ebe00640 1 -- 172.21.15.28:0/1802267208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89ec06c1f0 msgr2=0x7f89ec152680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:19.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.809+0000 7f89ebe00640 1 --2- 172.21.15.28:0/1802267208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89ec06c1f0 0x7f89ec152680 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f89dc009890 tx=0x7f89dc0046c0 comp rx=0 tx=0).stop 2024-02-21T23:03:19.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.809+0000 7f89ebe00640 1 -- 172.21.15.28:0/1802267208 shutdown_connections 2024-02-21T23:03:19.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.809+0000 7f89ebe00640 1 --2- 172.21.15.28:0/1802267208 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f89cc06b4d0 0x7f89cc06d990 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:19.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.809+0000 7f89ebe00640 1 --2- 172.21.15.28:0/1802267208 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f89ec14f520 0x7f89ec1eb360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:19.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.809+0000 7f89ebe00640 1 --2- 172.21.15.28:0/1802267208 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f89ec0c70b0 0x7f89ec1eae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:19.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.809+0000 7f89ebe00640 1 --2- 172.21.15.28:0/1802267208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89ec06c1f0 0x7f89ec152680 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:19.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.809+0000 7f89ebe00640 1 -- 172.21.15.28:0/1802267208 >> 172.21.15.28:0/1802267208 conn(0x7f89ec0a9690 msgr2=0x7f89ec0c4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:19.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.809+0000 7f89ebe00640 1 -- 172.21.15.28:0/1802267208 shutdown_connections 2024-02-21T23:03:19.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:19.809+0000 7f89ebe00640 1 -- 172.21.15.28:0/1802267208 wait complete. 2024-02-21T23:03:19.816 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:03:20.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.051+0000 7fc7d3a00640 1 Processor -- start 2024-02-21T23:03:20.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.051+0000 7fc7d3a00640 1 -- start start 2024-02-21T23:03:20.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.052+0000 7fc7d3a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fc7cc070d20 0x7fc7cc150230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:20.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.052+0000 7fc7d3a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc7cc0c66c0 0x7fc7cc150770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:20.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.052+0000 7fc7d3a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7cc152f50 con 0x7fc7cc0a8090 2024-02-21T23:03:20.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.052+0000 7fc7d3a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7cc0c3340 con 0x7fc7cc070d20 2024-02-21T23:03:20.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.052+0000 7fc7d3a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7cc0c3620 con 0x7fc7cc0c66c0 2024-02-21T23:03:20.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.053+0000 7fc7d1a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fc7cc0a8090 0x7fc7cc0c3e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39644/0 (socket says 172.21.15.28:39644) 2024-02-21T23:03:20.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.053+0000 7fc7d1a00640 1 -- 172.21.15.28:0/3660469737 learned_addr learned my addr 172.21.15.28:0/3660469737 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:03:20.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.053+0000 7fc7d1000640 1 --2- 172.21.15.28:0/3660469737 >> v2:172.21.15.81:3300/0 conn(0x7fc7cc070d20 0x7fc7cc150230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:20.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.053+0000 7fc7cbe00640 1 --2- 172.21.15.28:0/3660469737 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc7cc0c66c0 0x7fc7cc150770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:20.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.053+0000 7fc7cb400640 1 -- 172.21.15.28:0/3660469737 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4024144555 0 0) 0x7fc7cc152f50 con 0x7fc7cc0a8090 2024-02-21T23:03:20.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.053+0000 7fc7cb400640 1 -- 172.21.15.28:0/3660469737 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7b0003520 con 0x7fc7cc0a8090 2024-02-21T23:03:20.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.054+0000 7fc7cb400640 1 -- 172.21.15.28:0/3660469737 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 977170295 0 0) 0x7fc7b0003520 con 0x7fc7cc0a8090 2024-02-21T23:03:20.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.054+0000 7fc7cb400640 1 -- 172.21.15.28:0/3660469737 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc7cc0c66c0 msgr2=0x7fc7cc150770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:20.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.054+0000 7fc7cb400640 1 --2- 172.21.15.28:0/3660469737 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc7cc0c66c0 0x7fc7cc150770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:20.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.054+0000 7fc7cb400640 1 -- 172.21.15.28:0/3660469737 >> v2:172.21.15.81:3300/0 conn(0x7fc7cc070d20 msgr2=0x7fc7cc150230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:20.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.054+0000 7fc7cb400640 1 --2- 172.21.15.28:0/3660469737 >> v2:172.21.15.81:3300/0 conn(0x7fc7cc070d20 0x7fc7cc150230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:20.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.054+0000 7fc7cb400640 1 -- 172.21.15.28:0/3660469737 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7cc0c3900 con 0x7fc7cc0a8090 2024-02-21T23:03:20.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.054+0000 7fc7cb400640 1 -- 172.21.15.28:0/3660469737 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fc7c40019f0 con 0x7fc7cc0a8090 2024-02-21T23:03:20.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.054+0000 7fc7cb400640 1 -- 172.21.15.28:0/3660469737 >> v1:172.21.15.28:6789/0 conn(0x7fc7cc0a8090 legacy=0x7fc7cc0c3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:20.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.054+0000 7fc7cb400640 1 --2- 172.21.15.28:0/3660469737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7b00039b0 0x7fc7b0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:20.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.054+0000 7fc7cb400640 1 --2- 172.21.15.28:0/3660469737 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc7b0004800 0x7fc7b0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:20.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.054+0000 7fc7cb400640 1 --2- 172.21.15.28:0/3660469737 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc7cc0c66c0 0x7fc7b00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:20.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.054+0000 7fc7cb400640 1 -- 172.21.15.28:0/3660469737 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7cc0c3340 con 0x7fc7b00039b0 2024-02-21T23:03:20.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.054+0000 7fc7cb400640 1 -- 172.21.15.28:0/3660469737 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7cc0c3620 con 0x7fc7b0004800 2024-02-21T23:03:20.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.054+0000 7fc7cb400640 1 -- 172.21.15.28:0/3660469737 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b000ba10 con 0x7fc7cc0c66c0 2024-02-21T23:03:20.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.054+0000 7fc7d1000640 1 --2- 172.21.15.28:0/3660469737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7b00039b0 0x7fc7b0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:20.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.054+0000 7fc7cbe00640 1 --2- 172.21.15.28:0/3660469737 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc7b0004800 0x7fc7b0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:20.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.054+0000 7fc7d1a00640 1 --2- 172.21.15.28:0/3660469737 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc7cc0c66c0 0x7fc7b00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:20.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.055+0000 7fc7d1000640 1 -- 172.21.15.28:0/3660469737 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc7cc0c66c0 msgr2=0x7fc7b00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:20.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.055+0000 7fc7d1000640 1 --2- 172.21.15.28:0/3660469737 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc7cc0c66c0 0x7fc7b00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:20.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.055+0000 7fc7d1000640 1 -- 172.21.15.28:0/3660469737 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc7b0004800 msgr2=0x7fc7b0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:20.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.055+0000 7fc7d1000640 1 --2- 172.21.15.28:0/3660469737 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc7b0004800 0x7fc7b0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:20.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.055+0000 7fc7d1000640 1 -- 172.21.15.28:0/3660469737 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7b0003520 con 0x7fc7b00039b0 2024-02-21T23:03:20.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.056+0000 7fc7d1000640 1 --2- 172.21.15.28:0/3660469737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7b00039b0 0x7fc7b0003dd0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fc7bc009af0 tx=0x7fc7b0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c04ff4688132bdc server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:20.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.056+0000 7fc7cb400640 1 -- 172.21.15.28:0/3660469737 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7bc039070 con 0x7fc7b00039b0 2024-02-21T23:03:20.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.056+0000 7fc7cb400640 1 -- 172.21.15.28:0/3660469737 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7bc02c710 con 0x7fc7b00039b0 2024-02-21T23:03:20.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.056+0000 7fc7cb400640 1 -- 172.21.15.28:0/3660469737 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7bc009690 con 0x7fc7b00039b0 2024-02-21T23:03:20.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.057+0000 7fc7d3a00640 1 -- 172.21.15.28:0/3660469737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7b00039b0 msgr2=0x7fc7b0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:20.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.057+0000 7fc7d3a00640 1 --2- 172.21.15.28:0/3660469737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7b00039b0 0x7fc7b0003dd0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fc7bc009af0 tx=0x7fc7b0005830 comp rx=0 tx=0).stop 2024-02-21T23:03:20.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.057+0000 7fc7d1a00640 1 -- 172.21.15.28:0/3660469737 reap_dead start 2024-02-21T23:03:20.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.057+0000 7fc7d3a00640 1 -- 172.21.15.28:0/3660469737 shutdown_connections 2024-02-21T23:03:20.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.057+0000 7fc7d3a00640 1 -- 172.21.15.28:0/3660469737 >> 172.21.15.28:0/3660469737 conn(0x7fc7cc0aa550 msgr2=0x7fc7cc0aa930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:20.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.057+0000 7fc7d3a00640 1 -- 172.21.15.28:0/3660469737 shutdown_connections 2024-02-21T23:03:20.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.057+0000 7fc7d3a00640 1 -- 172.21.15.28:0/3660469737 wait complete. 2024-02-21T23:03:20.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.059+0000 7fc7d3a00640 1 Processor -- start 2024-02-21T23:03:20.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.059+0000 7fc7d3a00640 1 -- start start 2024-02-21T23:03:20.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.060+0000 7fc7d3a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7b00039b0 0x7fc7cc1eb090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:20.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.060+0000 7fc7d3a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc7cc0c8c70 0x7fc7cc06d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:20.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.060+0000 7fc7d3a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc7cc147a50 0x7fc7cc147ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:20.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.060+0000 7fc7d3a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7cc0c2d40 con 0x7fc7b00039b0 2024-02-21T23:03:20.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.060+0000 7fc7d3a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7cc0c2bc0 con 0x7fc7cc0c8c70 2024-02-21T23:03:20.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.060+0000 7fc7d3a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7cc0c2ec0 con 0x7fc7cc147a50 2024-02-21T23:03:20.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.060+0000 7fc7d1000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7b00039b0 0x7fc7cc1eb090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:20.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.060+0000 7fc7d1000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7b00039b0 0x7fc7cc1eb090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46074/0 (socket says 172.21.15.28:46074) 2024-02-21T23:03:20.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.060+0000 7fc7d1000640 1 -- 172.21.15.28:0/2123071598 learned_addr learned my addr 172.21.15.28:0/2123071598 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:20.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.060+0000 7fc7cbe00640 1 --2- 172.21.15.28:0/2123071598 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc7cc0c8c70 0x7fc7cc06d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:20.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.060+0000 7fc7d1a00640 1 --2- 172.21.15.28:0/2123071598 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc7cc147a50 0x7fc7cc147ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:20.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.060+0000 7fc7d1000640 1 -- 172.21.15.28:0/2123071598 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc7cc147a50 msgr2=0x7fc7cc147ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:20.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.060+0000 7fc7d1000640 1 --2- 172.21.15.28:0/2123071598 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc7cc147a50 0x7fc7cc147ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:20.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.060+0000 7fc7d1000640 1 -- 172.21.15.28:0/2123071598 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc7cc0c8c70 msgr2=0x7fc7cc06d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:20.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.060+0000 7fc7d1000640 1 --2- 172.21.15.28:0/2123071598 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc7cc0c8c70 0x7fc7cc06d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:20.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.060+0000 7fc7d1000640 1 -- 172.21.15.28:0/2123071598 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7cc0c7920 con 0x7fc7b00039b0 2024-02-21T23:03:20.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.060+0000 7fc7d1a00640 1 --2- 172.21.15.28:0/2123071598 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc7cc147a50 0x7fc7cc147ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:03:20.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.061+0000 7fc7d1000640 1 --2- 172.21.15.28:0/2123071598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7b00039b0 0x7fc7cc1eb090 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fc7bc009c10 tx=0x7fc7bc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:20.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.061+0000 7fc7c8c00640 1 -- 172.21.15.28:0/2123071598 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7bc039050 con 0x7fc7b00039b0 2024-02-21T23:03:20.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.061+0000 7fc7c8c00640 1 -- 172.21.15.28:0/2123071598 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7bc045070 con 0x7fc7b00039b0 2024-02-21T23:03:20.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.061+0000 7fc7c8c00640 1 -- 172.21.15.28:0/2123071598 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7bc003a50 con 0x7fc7b00039b0 2024-02-21T23:03:20.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.061+0000 7fc7d3a00640 1 -- 172.21.15.28:0/2123071598 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7cc0c7c00 con 0x7fc7b00039b0 2024-02-21T23:03:20.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.061+0000 7fc7d3a00640 1 -- 172.21.15.28:0/2123071598 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7cc06d8b0 con 0x7fc7b00039b0 2024-02-21T23:03:20.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.063+0000 7fc7d3a00640 1 -- 172.21.15.28:0/2123071598 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7cc070590 con 0x7fc7b00039b0 2024-02-21T23:03:20.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.064+0000 7fc7c8c00640 1 -- 172.21.15.28:0/2123071598 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc7bc003bf0 con 0x7fc7b00039b0 2024-02-21T23:03:20.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.065+0000 7fc7c8c00640 1 --2- 172.21.15.28:0/2123071598 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc7ac06b500 0x7fc7ac06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:20.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.065+0000 7fc7c8c00640 1 -- 172.21.15.28:0/2123071598 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fc7bc0abfb0 con 0x7fc7b00039b0 2024-02-21T23:03:20.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.066+0000 7fc7cbe00640 1 --2- 172.21.15.28:0/2123071598 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc7ac06b500 0x7fc7ac06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:20.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.067+0000 7fc7cbe00640 1 --2- 172.21.15.28:0/2123071598 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc7ac06b500 0x7fc7ac06d9c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc0c9070 tx=0x7fc7b8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:20.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.070+0000 7fc7c8c00640 1 -- 172.21.15.28:0/2123071598 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc7cc070590 con 0x7fc7b00039b0 2024-02-21T23:03:20.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.308+0000 7fc7d3a00640 1 -- 172.21.15.28:0/2123071598 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc7cc06e710 con 0x7fc7b00039b0 2024-02-21T23:03:20.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.309+0000 7fc7c8c00640 1 -- 172.21.15.28:0/2123071598 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4934 (secure 0 0 0) 0x7fc7cc06e710 con 0x7fc7b00039b0 2024-02-21T23:03:20.299 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:03:20.299 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4230,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/2559793999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2559793999},{"type":"v1","addr":"172.21.15.28:6834","nonce":2559793999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4232,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3930208177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3930208177},{"type":"v1","addr":"172.21.15.28:6835","nonce":3930208177}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4238,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/2875984988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":2875984988},{"type":"v1","addr":"172.21.15.81:6836","nonce":2875984988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:02:37.812604+0000","modified":"2024-02-21T23:03:06.243463+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4267},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4267":{"gid":4267,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6837/2328151245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2328151245},{"type":"v1","addr":"172.21.15.81:6837","nonce":2328151245}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-02-21T23:03:20.299 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 9 2024-02-21T23:03:20.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.312+0000 7fc7d3a00640 1 -- 172.21.15.28:0/2123071598 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc7ac06b500 msgr2=0x7fc7ac06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:20.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.312+0000 7fc7d3a00640 1 --2- 172.21.15.28:0/2123071598 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc7ac06b500 0x7fc7ac06d9c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc0c9070 tx=0x7fc7b8008040 comp rx=0 tx=0).stop 2024-02-21T23:03:20.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.312+0000 7fc7d3a00640 1 -- 172.21.15.28:0/2123071598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7b00039b0 msgr2=0x7fc7cc1eb090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:20.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.312+0000 7fc7d3a00640 1 --2- 172.21.15.28:0/2123071598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7b00039b0 0x7fc7cc1eb090 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fc7bc009c10 tx=0x7fc7bc004730 comp rx=0 tx=0).stop 2024-02-21T23:03:20.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.312+0000 7fc7d3a00640 1 -- 172.21.15.28:0/2123071598 shutdown_connections 2024-02-21T23:03:20.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.312+0000 7fc7d3a00640 1 --2- 172.21.15.28:0/2123071598 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc7ac06b500 0x7fc7ac06d9c0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:20.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.312+0000 7fc7d3a00640 1 --2- 172.21.15.28:0/2123071598 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc7cc147a50 0x7fc7cc147ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:20.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.312+0000 7fc7d3a00640 1 --2- 172.21.15.28:0/2123071598 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc7cc0c8c70 0x7fc7cc06d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:20.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.312+0000 7fc7d3a00640 1 --2- 172.21.15.28:0/2123071598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7b00039b0 0x7fc7cc1eb090 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:20.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.312+0000 7fc7d3a00640 1 -- 172.21.15.28:0/2123071598 >> 172.21.15.28:0/2123071598 conn(0x7fc7cc0aa550 msgr2=0x7fc7cc153510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:20.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.312+0000 7fc7d3a00640 1 -- 172.21.15.28:0/2123071598 shutdown_connections 2024-02-21T23:03:20.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.312+0000 7fc7d3a00640 1 -- 172.21.15.28:0/2123071598 wait complete. 2024-02-21T23:03:20.321 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:03:20.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.552+0000 7f941e400640 1 Processor -- start 2024-02-21T23:03:20.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.552+0000 7f941e400640 1 -- start start 2024-02-21T23:03:20.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.553+0000 7f941e400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f94180aff90 0x7f94180c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:20.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.553+0000 7f941e400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f94180c5720 0x7f941815f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:20.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.553+0000 7f941e400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94180c74c0 con 0x7f941806dbf0 2024-02-21T23:03:20.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.554+0000 7f941e400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94180c7270 con 0x7f94180c5720 2024-02-21T23:03:20.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.554+0000 7f941e400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9418158630 con 0x7f94180aff90 2024-02-21T23:03:20.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.554+0000 7f941da00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f94180c5720 0x7f941815f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:20.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.554+0000 7f941da00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f94180c5720 0x7f941815f890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46076/0 (socket says 172.21.15.28:46076) 2024-02-21T23:03:20.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.554+0000 7f941da00640 1 -- 172.21.15.28:0/2327179963 learned_addr learned my addr 172.21.15.28:0/2327179963 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:20.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.554+0000 7f941d000640 1 --2- 172.21.15.28:0/2327179963 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f94180aff90 0x7f94180c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:20.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.554+0000 7f941da00640 1 -- 172.21.15.28:0/2327179963 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f94180aff90 msgr2=0x7f94180c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:20.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.554+0000 7f941da00640 1 --2- 172.21.15.28:0/2327179963 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f94180aff90 0x7f94180c51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:20.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.555+0000 7f941da00640 1 -- 172.21.15.28:0/2327179963 >> v1:172.21.15.81:6789/0 conn(0x7f941806dbf0 legacy=0x7f9418158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:20.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.555+0000 7f941da00640 1 -- 172.21.15.28:0/2327179963 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94180c77a0 con 0x7f94180c5720 2024-02-21T23:03:20.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.555+0000 7f941d000640 1 --2- 172.21.15.28:0/2327179963 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f94180aff90 0x7f94180c51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:03:20.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.556+0000 7f941da00640 1 --2- 172.21.15.28:0/2327179963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94180c5720 0x7f941815f890 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f9410009e50 tx=0x7f941002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d6239fa421051b4 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:20.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.556+0000 7f9417400640 1 -- 172.21.15.28:0/2327179963 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f941002cb00 con 0x7f94180c5720 2024-02-21T23:03:20.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.556+0000 7f9417400640 1 -- 172.21.15.28:0/2327179963 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f941002cca0 con 0x7f94180c5720 2024-02-21T23:03:20.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.556+0000 7f9417400640 1 -- 172.21.15.28:0/2327179963 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9410034610 con 0x7f94180c5720 2024-02-21T23:03:20.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.557+0000 7f941e400640 1 -- 172.21.15.28:0/2327179963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94180c5720 msgr2=0x7f941815f890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:20.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.557+0000 7f941e400640 1 --2- 172.21.15.28:0/2327179963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94180c5720 0x7f941815f890 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f9410009e50 tx=0x7f941002bf60 comp rx=0 tx=0).stop 2024-02-21T23:03:20.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.557+0000 7f941e400640 1 -- 172.21.15.28:0/2327179963 shutdown_connections 2024-02-21T23:03:20.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.557+0000 7f941e400640 1 --2- 172.21.15.28:0/2327179963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94180c5720 0x7f941815f890 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:20.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.557+0000 7f941e400640 1 --2- 172.21.15.28:0/2327179963 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f94180aff90 0x7f94180c51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:20.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.557+0000 7f941e400640 1 -- 172.21.15.28:0/2327179963 >> 172.21.15.28:0/2327179963 conn(0x7f94180ac370 msgr2=0x7f94180ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:20.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.557+0000 7f941e400640 1 -- 172.21.15.28:0/2327179963 shutdown_connections 2024-02-21T23:03:20.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.557+0000 7f941e400640 1 -- 172.21.15.28:0/2327179963 wait complete. 2024-02-21T23:03:20.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.558+0000 7f941e400640 1 Processor -- start 2024-02-21T23:03:20.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.559+0000 7f941e400640 1 -- start start 2024-02-21T23:03:20.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.559+0000 7f941e400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f941806dbf0 0x7f94181eaeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:20.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.559+0000 7f941e400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f94180aff90 0x7f94180ae740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:20.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.559+0000 7f941e400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94180c5720 0x7f94181533d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:20.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.559+0000 7f941e400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94180c8760 con 0x7f94180c5720 2024-02-21T23:03:20.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.559+0000 7f941e400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94180be0c0 con 0x7f941806dbf0 2024-02-21T23:03:20.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.559+0000 7f941e400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94180c88e0 con 0x7f94180aff90 2024-02-21T23:03:20.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.560+0000 7f941da00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94180c5720 0x7f94181533d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:20.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.560+0000 7f941da00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94180c5720 0x7f94181533d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46086/0 (socket says 172.21.15.28:46086) 2024-02-21T23:03:20.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.560+0000 7f941da00640 1 -- 172.21.15.28:0/413148545 learned_addr learned my addr 172.21.15.28:0/413148545 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:20.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.560+0000 7f941d000640 1 --2- 172.21.15.28:0/413148545 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f941806dbf0 0x7f94181eaeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:20.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.560+0000 7f9417e00640 1 --2- 172.21.15.28:0/413148545 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f94180aff90 0x7f94180ae740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:20.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.560+0000 7f941da00640 1 -- 172.21.15.28:0/413148545 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f94180aff90 msgr2=0x7f94180ae740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:20.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.560+0000 7f941da00640 1 --2- 172.21.15.28:0/413148545 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f94180aff90 0x7f94180ae740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:20.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.560+0000 7f941da00640 1 -- 172.21.15.28:0/413148545 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f941806dbf0 msgr2=0x7f94181eaeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:20.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.560+0000 7f941da00640 1 --2- 172.21.15.28:0/413148545 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f941806dbf0 0x7f94181eaeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:20.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.560+0000 7f941da00640 1 -- 172.21.15.28:0/413148545 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9418154190 con 0x7f94180c5720 2024-02-21T23:03:20.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.561+0000 7f941da00640 1 --2- 172.21.15.28:0/413148545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94180c5720 0x7f94181533d0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9410009e50 tx=0x7f9410004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:20.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.561+0000 7f9414c00640 1 -- 172.21.15.28:0/413148545 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9410044020 con 0x7f94180c5720 2024-02-21T23:03:20.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.561+0000 7f9414c00640 1 -- 172.21.15.28:0/413148545 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f941002ce50 con 0x7f94180c5720 2024-02-21T23:03:20.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.561+0000 7f9414c00640 1 -- 172.21.15.28:0/413148545 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9410009510 con 0x7f94180c5720 2024-02-21T23:03:20.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.561+0000 7f941e400640 1 -- 172.21.15.28:0/413148545 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9418154470 con 0x7f94180c5720 2024-02-21T23:03:20.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.561+0000 7f941e400640 1 -- 172.21.15.28:0/413148545 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94180aec80 con 0x7f94180c5720 2024-02-21T23:03:20.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.562+0000 7f941e400640 1 -- 172.21.15.28:0/413148545 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f941804f290 con 0x7f94180c5720 2024-02-21T23:03:20.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.563+0000 7f9414c00640 1 -- 172.21.15.28:0/413148545 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9410047020 con 0x7f94180c5720 2024-02-21T23:03:20.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.564+0000 7f9414c00640 1 --2- 172.21.15.28:0/413148545 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f93f806b4d0 0x7f93f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:20.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.564+0000 7f9414c00640 1 -- 172.21.15.28:0/413148545 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f94100acfd0 con 0x7f94180c5720 2024-02-21T23:03:20.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.565+0000 7f941d000640 1 --2- 172.21.15.28:0/413148545 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f93f806b4d0 0x7f93f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:20.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.566+0000 7f941d000640 1 --2- 172.21.15.28:0/413148545 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f93f806b4d0 0x7f93f806d990 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f940c006fd0 tx=0x7f940c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:20.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.569+0000 7f9414c00640 1 -- 172.21.15.28:0/413148545 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f941004e030 con 0x7f94180c5720 2024-02-21T23:03:20.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.769+0000 7f941e400640 1 -- 172.21.15.28:0/413148545 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f94181524f0 con 0x7f94180c5720 2024-02-21T23:03:20.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.772+0000 7f9414c00640 1 -- 172.21.15.28:0/413148545 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18454 (secure 0 0 0) 0x7f941007bc20 con 0x7f94180c5720 2024-02-21T23:03:20.762 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:03:20.763 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":24,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:02:42.900286+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:02:35.795158+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-21T23:02:36.840110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-02-21T23:02:40.441388+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:03:20.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.775+0000 7f941e400640 1 -- 172.21.15.28:0/413148545 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f93f806b4d0 msgr2=0x7f93f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:20.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.775+0000 7f941e400640 1 --2- 172.21.15.28:0/413148545 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f93f806b4d0 0x7f93f806d990 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f940c006fd0 tx=0x7f940c008040 comp rx=0 tx=0).stop 2024-02-21T23:03:20.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.775+0000 7f941e400640 1 -- 172.21.15.28:0/413148545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94180c5720 msgr2=0x7f94181533d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:20.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.775+0000 7f941e400640 1 --2- 172.21.15.28:0/413148545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94180c5720 0x7f94181533d0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9410009e50 tx=0x7f9410004a40 comp rx=0 tx=0).stop 2024-02-21T23:03:20.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.775+0000 7f941e400640 1 -- 172.21.15.28:0/413148545 shutdown_connections 2024-02-21T23:03:20.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.775+0000 7f941e400640 1 --2- 172.21.15.28:0/413148545 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f93f806b4d0 0x7f93f806d990 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:20.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.775+0000 7f941e400640 1 --2- 172.21.15.28:0/413148545 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f94180aff90 0x7f94180ae740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:20.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.775+0000 7f941e400640 1 --2- 172.21.15.28:0/413148545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94180c5720 0x7f94181533d0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:20.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.775+0000 7f941e400640 1 --2- 172.21.15.28:0/413148545 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f941806dbf0 0x7f94181eaeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:20.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.775+0000 7f941e400640 1 -- 172.21.15.28:0/413148545 >> 172.21.15.28:0/413148545 conn(0x7f94180ac370 msgr2=0x7f94180ac750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:20.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.775+0000 7f941e400640 1 -- 172.21.15.28:0/413148545 shutdown_connections 2024-02-21T23:03:20.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:20.775+0000 7f941e400640 1 -- 172.21.15.28:0/413148545 wait complete. 2024-02-21T23:03:20.786 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:03:20.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.008+0000 7f539b200640 1 Processor -- start 2024-02-21T23:03:20.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.008+0000 7f539b200640 1 -- start start 2024-02-21T23:03:20.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.009+0000 7f539b200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f539406f410 0x7f5394153900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:20.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.009+0000 7f539b200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5394071400 0x7f53940c7840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:21.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.009+0000 7f539b200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53940c4b70 con 0x7f5394071020 2024-02-21T23:03:21.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.009+0000 7f539b200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53940c5130 con 0x7f539406f410 2024-02-21T23:03:21.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.009+0000 7f539b200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53940c4e50 con 0x7f5394071400 2024-02-21T23:03:21.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.009+0000 7f5393400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f539406f410 0x7f5394153900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:21.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.009+0000 7f5393e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f5394071020 0x7f5394156550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39652/0 (socket says 172.21.15.28:39652) 2024-02-21T23:03:21.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.009+0000 7f5393400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f539406f410 0x7f5394153900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46088/0 (socket says 172.21.15.28:46088) 2024-02-21T23:03:21.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.009+0000 7f5393e00640 1 -- 172.21.15.28:0/1724185580 learned_addr learned my addr 172.21.15.28:0/1724185580 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:03:21.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.010+0000 7f5399200640 1 --2- 172.21.15.28:0/1724185580 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5394071400 0x7f53940c7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:21.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.010+0000 7f5393400640 1 -- 172.21.15.28:0/1724185580 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5394071400 msgr2=0x7f53940c7840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:21.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.010+0000 7f5393400640 1 --2- 172.21.15.28:0/1724185580 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5394071400 0x7f53940c7840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:21.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.010+0000 7f5393400640 1 -- 172.21.15.28:0/1724185580 >> v1:172.21.15.28:6789/0 conn(0x7f5394071020 legacy=0x7f5394156550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:21.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.010+0000 7f5393400640 1 -- 172.21.15.28:0/1724185580 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5394153100 con 0x7f539406f410 2024-02-21T23:03:21.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.010+0000 7f5392a00640 1 -- 172.21.15.28:0/1724185580 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1605279891 0 0) 0x7f53940c4b70 con 0x7f5394071020 2024-02-21T23:03:21.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.011+0000 7f5393400640 1 --2- 172.21.15.28:0/1724185580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f539406f410 0x7f5394153900 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f53800097a0 tx=0x7f538002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b51cc164114dfae server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:21.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.011+0000 7f5392a00640 1 -- 172.21.15.28:0/1724185580 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53800315a0 con 0x7f539406f410 2024-02-21T23:03:21.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.011+0000 7f5392a00640 1 -- 172.21.15.28:0/1724185580 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5380037070 con 0x7f539406f410 2024-02-21T23:03:21.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.011+0000 7f5392a00640 1 -- 172.21.15.28:0/1724185580 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5380031960 con 0x7f539406f410 2024-02-21T23:03:21.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.012+0000 7f539b200640 1 -- 172.21.15.28:0/1724185580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f539406f410 msgr2=0x7f5394153900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:21.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.012+0000 7f539b200640 1 --2- 172.21.15.28:0/1724185580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f539406f410 0x7f5394153900 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f53800097a0 tx=0x7f538002b840 comp rx=0 tx=0).stop 2024-02-21T23:03:21.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.012+0000 7f539b200640 1 -- 172.21.15.28:0/1724185580 shutdown_connections 2024-02-21T23:03:21.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.012+0000 7f539b200640 1 --2- 172.21.15.28:0/1724185580 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5394071400 0x7f53940c7840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:21.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.012+0000 7f539b200640 1 --2- 172.21.15.28:0/1724185580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f539406f410 0x7f5394153900 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:21.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.012+0000 7f539b200640 1 -- 172.21.15.28:0/1724185580 >> 172.21.15.28:0/1724185580 conn(0x7f53940c8880 msgr2=0x7f53940c8c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:21.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.012+0000 7f539b200640 1 -- 172.21.15.28:0/1724185580 shutdown_connections 2024-02-21T23:03:21.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.012+0000 7f539b200640 1 -- 172.21.15.28:0/1724185580 wait complete. 2024-02-21T23:03:21.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.014+0000 7f539b200640 1 Processor -- start 2024-02-21T23:03:21.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.014+0000 7f539b200640 1 -- start start 2024-02-21T23:03:21.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.015+0000 7f539b200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f539406f410 0x7f53940c1f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:21.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.015+0000 7f539b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5394071020 0x7f53941552a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:21.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.015+0000 7f539b200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5394071400 0x7f539414ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:21.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.015+0000 7f539b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53940c7d80 con 0x7f5394071020 2024-02-21T23:03:21.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.015+0000 7f539b200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5394153e40 con 0x7f539406f410 2024-02-21T23:03:21.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.015+0000 7f539b200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53940c7f00 con 0x7f5394071400 2024-02-21T23:03:21.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.015+0000 7f5393400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5394071020 0x7f53941552a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:21.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.015+0000 7f5393400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5394071020 0x7f53941552a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46098/0 (socket says 172.21.15.28:46098) 2024-02-21T23:03:21.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.015+0000 7f5393400640 1 -- 172.21.15.28:0/4175405965 learned_addr learned my addr 172.21.15.28:0/4175405965 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:21.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.015+0000 7f5399200640 1 --2- 172.21.15.28:0/4175405965 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5394071400 0x7f539414ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:21.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.015+0000 7f5393e00640 1 --2- 172.21.15.28:0/4175405965 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f539406f410 0x7f53940c1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:21.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.016+0000 7f5393400640 1 -- 172.21.15.28:0/4175405965 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5394071400 msgr2=0x7f539414ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:21.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.016+0000 7f5393400640 1 --2- 172.21.15.28:0/4175405965 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5394071400 0x7f539414ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:21.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.016+0000 7f5393400640 1 -- 172.21.15.28:0/4175405965 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f539406f410 msgr2=0x7f53940c1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:21.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.016+0000 7f5393400640 1 --2- 172.21.15.28:0/4175405965 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f539406f410 0x7f53940c1f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:21.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.016+0000 7f5393400640 1 -- 172.21.15.28:0/4175405965 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53940be950 con 0x7f5394071020 2024-02-21T23:03:21.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.016+0000 7f5393400640 1 --2- 172.21.15.28:0/4175405965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5394071020 0x7f53941552a0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f5380034040 tx=0x7f53800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:21.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.016+0000 7f538be00640 1 -- 172.21.15.28:0/4175405965 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5380032840 con 0x7f5394071020 2024-02-21T23:03:21.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.016+0000 7f538be00640 1 -- 172.21.15.28:0/4175405965 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5380032a00 con 0x7f5394071020 2024-02-21T23:03:21.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.016+0000 7f539b200640 1 -- 172.21.15.28:0/4175405965 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53940c26d0 con 0x7f5394071020 2024-02-21T23:03:21.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.017+0000 7f538be00640 1 -- 172.21.15.28:0/4175405965 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5380032df0 con 0x7f5394071020 2024-02-21T23:03:21.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.017+0000 7f539b200640 1 -- 172.21.15.28:0/4175405965 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53941e8a60 con 0x7f5394071020 2024-02-21T23:03:21.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.018+0000 7f539b200640 1 -- 172.21.15.28:0/4175405965 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5394070c40 con 0x7f5394071020 2024-02-21T23:03:21.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.019+0000 7f538be00640 1 -- 172.21.15.28:0/4175405965 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f538003b3f0 con 0x7f5394071020 2024-02-21T23:03:21.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.020+0000 7f538be00640 1 --2- 172.21.15.28:0/4175405965 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f537406b4d0 0x7f537406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:21.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.020+0000 7f538be00640 1 -- 172.21.15.28:0/4175405965 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f538002f030 con 0x7f5394071020 2024-02-21T23:03:21.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.020+0000 7f5393e00640 1 --2- 172.21.15.28:0/4175405965 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f537406b4d0 0x7f537406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:21.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.022+0000 7f5393e00640 1 --2- 172.21.15.28:0/4175405965 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f537406b4d0 0x7f537406d990 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f5384000f60 tx=0x7f5384009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:21.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.024+0000 7f538be00640 1 -- 172.21.15.28:0/4175405965 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f5380040070 con 0x7f5394071020 2024-02-21T23:03:21.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.264+0000 7f539b200640 1 -- 172.21.15.28:0/4175405965 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5394068640 con 0x7f5394071020 2024-02-21T23:03:21.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.265+0000 7f538be00640 1 -- 172.21.15.28:0/4175405965 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4934 (secure 0 0 0) 0x7f538007b1c0 con 0x7f5394071020 2024-02-21T23:03:21.255 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:03:21.255 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4230,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/2559793999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2559793999},{"type":"v1","addr":"172.21.15.28:6834","nonce":2559793999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4232,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3930208177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3930208177},{"type":"v1","addr":"172.21.15.28:6835","nonce":3930208177}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4238,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/2875984988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":2875984988},{"type":"v1","addr":"172.21.15.81:6836","nonce":2875984988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:02:37.812604+0000","modified":"2024-02-21T23:03:06.243463+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4267},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4267":{"gid":4267,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6837/2328151245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2328151245},{"type":"v1","addr":"172.21.15.81:6837","nonce":2328151245}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-02-21T23:03:21.255 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 9 2024-02-21T23:03:21.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.267+0000 7f539b200640 1 -- 172.21.15.28:0/4175405965 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f537406b4d0 msgr2=0x7f537406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:21.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.267+0000 7f539b200640 1 --2- 172.21.15.28:0/4175405965 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f537406b4d0 0x7f537406d990 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f5384000f60 tx=0x7f5384009040 comp rx=0 tx=0).stop 2024-02-21T23:03:21.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.267+0000 7f539b200640 1 -- 172.21.15.28:0/4175405965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5394071020 msgr2=0x7f53941552a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:21.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.267+0000 7f539b200640 1 --2- 172.21.15.28:0/4175405965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5394071020 0x7f53941552a0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f5380034040 tx=0x7f53800046c0 comp rx=0 tx=0).stop 2024-02-21T23:03:21.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.268+0000 7f539b200640 1 -- 172.21.15.28:0/4175405965 shutdown_connections 2024-02-21T23:03:21.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.268+0000 7f539b200640 1 --2- 172.21.15.28:0/4175405965 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f537406b4d0 0x7f537406d990 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:21.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.268+0000 7f539b200640 1 --2- 172.21.15.28:0/4175405965 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5394071400 0x7f539414ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:21.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.268+0000 7f539b200640 1 --2- 172.21.15.28:0/4175405965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5394071020 0x7f53941552a0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:21.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.268+0000 7f539b200640 1 --2- 172.21.15.28:0/4175405965 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f539406f410 0x7f53940c1f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:21.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.268+0000 7f539b200640 1 -- 172.21.15.28:0/4175405965 >> 172.21.15.28:0/4175405965 conn(0x7f53940c8880 msgr2=0x7f53940c7510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:21.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.268+0000 7f539b200640 1 -- 172.21.15.28:0/4175405965 shutdown_connections 2024-02-21T23:03:21.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.268+0000 7f539b200640 1 -- 172.21.15.28:0/4175405965 wait complete. 2024-02-21T23:03:21.680 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 9, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:02:37.812604+0000', 'modified': '2024-02-21T23:03:06.243463+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4267}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4267': {'gid': 4267, 'name': 'b', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.81:6837/2328151245', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6834', 'nonce': 2328151245}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 2328151245}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:03:21.681 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-21T23:03:21.681 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-02-21T23:03:21.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.914+0000 7f2af1400640 1 Processor -- start 2024-02-21T23:03:21.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.915+0000 7f2af1400640 1 -- start start 2024-02-21T23:03:21.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.916+0000 7f2af1400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2aec1524c0 0x7f2aec155a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:21.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.916+0000 7f2af1400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f2aec0c85c0 0x7f2aec0c70d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:21.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.916+0000 7f2af1400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2aec06df50 con 0x7f2aec0c5040 2024-02-21T23:03:21.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.916+0000 7f2af1400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2aec0c5520 con 0x7f2aec0c85c0 2024-02-21T23:03:21.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.916+0000 7f2af1400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2aec0c5800 con 0x7f2aec1524c0 2024-02-21T23:03:21.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.917+0000 7f2aebe00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2aec1524c0 0x7f2aec155a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:21.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.917+0000 7f2aeb400640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f2aec0c5040 0x7f2aec06ce70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.28:51992/0 (socket says 172.21.15.28:51992) 2024-02-21T23:03:21.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.917+0000 7f2aeb400640 1 -- 172.21.15.28:0/623637094 learned_addr learned my addr 172.21.15.28:0/623637094 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:03:21.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.917+0000 7f2af0a00640 1 --2- 172.21.15.28:0/623637094 >> v2:172.21.15.81:3300/0 conn(0x7f2aec0c85c0 0x7f2aec0c70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:21.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.918+0000 7f2af0a00640 1 -- 172.21.15.28:0/623637094 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2aec1524c0 msgr2=0x7f2aec155a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:21.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.918+0000 7f2af0a00640 1 --2- 172.21.15.28:0/623637094 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2aec1524c0 0x7f2aec155a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:21.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.918+0000 7f2af0a00640 1 -- 172.21.15.28:0/623637094 >> v1:172.21.15.81:6789/0 conn(0x7f2aec0c5040 legacy=0x7f2aec06ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:21.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.918+0000 7f2af0a00640 1 -- 172.21.15.28:0/623637094 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aec06c670 con 0x7f2aec0c85c0 2024-02-21T23:03:21.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.918+0000 7f2aebe00640 1 --2- 172.21.15.28:0/623637094 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2aec1524c0 0x7f2aec155a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-21T23:03:21.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.919+0000 7f2af0a00640 1 --2- 172.21.15.28:0/623637094 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2aec0c85c0 0x7f2aec0c70d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2ae40097a0 tx=0x7f2ae402b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=68410699692e280e server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:21.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.920+0000 7f2aeaa00640 1 -- 172.21.15.28:0/623637094 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ae4038070 con 0x7f2aec0c85c0 2024-02-21T23:03:21.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.920+0000 7f2aeaa00640 1 -- 172.21.15.28:0/623637094 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ae4031700 con 0x7f2aec0c85c0 2024-02-21T23:03:21.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.920+0000 7f2aeaa00640 1 -- 172.21.15.28:0/623637094 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ae4031ad0 con 0x7f2aec0c85c0 2024-02-21T23:03:21.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.920+0000 7f2af1400640 1 -- 172.21.15.28:0/623637094 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2aec0c85c0 msgr2=0x7f2aec0c70d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:21.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.920+0000 7f2af1400640 1 --2- 172.21.15.28:0/623637094 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2aec0c85c0 0x7f2aec0c70d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2ae40097a0 tx=0x7f2ae402b7a0 comp rx=0 tx=0).stop 2024-02-21T23:03:21.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.921+0000 7f2af1400640 1 -- 172.21.15.28:0/623637094 shutdown_connections 2024-02-21T23:03:21.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.921+0000 7f2af1400640 1 --2- 172.21.15.28:0/623637094 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2aec0c85c0 0x7f2aec0c70d0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:21.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.921+0000 7f2af1400640 1 --2- 172.21.15.28:0/623637094 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2aec1524c0 0x7f2aec155a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:21.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.921+0000 7f2af1400640 1 -- 172.21.15.28:0/623637094 >> 172.21.15.28:0/623637094 conn(0x7f2aec0ac2b0 msgr2=0x7f2aec0ac6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:21.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.921+0000 7f2af1400640 1 -- 172.21.15.28:0/623637094 shutdown_connections 2024-02-21T23:03:21.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.921+0000 7f2af1400640 1 -- 172.21.15.28:0/623637094 wait complete. 2024-02-21T23:03:21.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.923+0000 7f2af1400640 1 Processor -- start 2024-02-21T23:03:21.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.923+0000 7f2af1400640 1 -- start start 2024-02-21T23:03:21.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.924+0000 7f2af1400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2aec0c5040 0x7f2aec1ee7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:21.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.924+0000 7f2af1400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2aec0c85c0 0x7f2aec154070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:21.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.924+0000 7f2af1400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2aec1524c0 0x7f2aec149bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:21.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.924+0000 7f2af1400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aec0c8a70 con 0x7f2aec0c5040 2024-02-21T23:03:21.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.924+0000 7f2af1400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aec1528a0 con 0x7f2aec0c85c0 2024-02-21T23:03:21.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.924+0000 7f2af1400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2aec0c7610 con 0x7f2aec1524c0 2024-02-21T23:03:21.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.924+0000 7f2aebe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2aec0c5040 0x7f2aec1ee7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:21.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.925+0000 7f2aebe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2aec0c5040 0x7f2aec1ee7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46102/0 (socket says 172.21.15.28:46102) 2024-02-21T23:03:21.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.925+0000 7f2aebe00640 1 -- 172.21.15.28:0/1168652715 learned_addr learned my addr 172.21.15.28:0/1168652715 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:21.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.925+0000 7f2af0a00640 1 --2- 172.21.15.28:0/1168652715 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2aec1524c0 0x7f2aec149bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:21.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.925+0000 7f2aeb400640 1 --2- 172.21.15.28:0/1168652715 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2aec0c85c0 0x7f2aec154070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:21.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.925+0000 7f2aebe00640 1 -- 172.21.15.28:0/1168652715 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2aec1524c0 msgr2=0x7f2aec149bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:21.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.925+0000 7f2aebe00640 1 --2- 172.21.15.28:0/1168652715 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2aec1524c0 0x7f2aec149bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:21.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.925+0000 7f2aebe00640 1 -- 172.21.15.28:0/1168652715 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2aec0c85c0 msgr2=0x7f2aec154070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:21.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.925+0000 7f2aebe00640 1 --2- 172.21.15.28:0/1168652715 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2aec0c85c0 0x7f2aec154070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:21.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.925+0000 7f2aebe00640 1 -- 172.21.15.28:0/1168652715 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aec1eed20 con 0x7f2aec0c5040 2024-02-21T23:03:21.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.925+0000 7f2aebe00640 1 --2- 172.21.15.28:0/1168652715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2aec0c5040 0x7f2aec1ee7e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f2adc009ec0 tx=0x7f2adc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:21.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.926+0000 7f2ae3e00640 1 -- 172.21.15.28:0/1168652715 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2adc018020 con 0x7f2aec0c5040 2024-02-21T23:03:21.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.926+0000 7f2ae3e00640 1 -- 172.21.15.28:0/1168652715 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2adc014680 con 0x7f2aec0c5040 2024-02-21T23:03:21.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.926+0000 7f2ae3e00640 1 -- 172.21.15.28:0/1168652715 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2adc014a50 con 0x7f2aec0c5040 2024-02-21T23:03:21.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.926+0000 7f2af1400640 1 -- 172.21.15.28:0/1168652715 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2aec153c50 con 0x7f2aec0c5040 2024-02-21T23:03:21.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.926+0000 7f2af1400640 1 -- 172.21.15.28:0/1168652715 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2aec1edfe0 con 0x7f2aec0c5040 2024-02-21T23:03:21.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.928+0000 7f2ae3e00640 1 -- 172.21.15.28:0/1168652715 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2adc005d10 con 0x7f2aec0c5040 2024-02-21T23:03:21.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.929+0000 7f2ae3e00640 1 --2- 172.21.15.28:0/1168652715 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2acc06b500 0x7f2acc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:21.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.929+0000 7f2ae3e00640 1 -- 172.21.15.28:0/1168652715 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f2adc08b290 con 0x7f2aec0c5040 2024-02-21T23:03:21.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.929+0000 7f2aeb400640 1 --2- 172.21.15.28:0/1168652715 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2acc06b500 0x7f2acc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:21.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.930+0000 7f2af5a32740 1 Processor -- start 2024-02-21T23:03:21.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.930+0000 7f2af5a32740 1 -- start start 2024-02-21T23:03:21.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.930+0000 7f2af5a32740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558992a14020 0x558992a14400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:21.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.931+0000 7f2af5a32740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x558992a15330 0x558992a50170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:21.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.931+0000 7f2af5a32740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558992a506b0 0x558992a4e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:21.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.931+0000 7f2af5a32740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x558992978470 con 0x558992a14020 2024-02-21T23:03:21.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.931+0000 7f2af5a32740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x558992921af0 con 0x558992a506b0 2024-02-21T23:03:21.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.931+0000 7f2af5a32740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x558992974510 con 0x558992a15330 2024-02-21T23:03:21.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.931+0000 7f2aeb400640 1 --2- 172.21.15.28:0/1168652715 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2acc06b500 0x7f2acc06d9c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f2aec1eb9e0 tx=0x7f2ad8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:21.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.931+0000 7f2aebe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558992a14020 0x558992a14400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:21.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.931+0000 7f2aebe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558992a14020 0x558992a14400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46104/0 (socket says 172.21.15.28:46104) 2024-02-21T23:03:21.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.931+0000 7f2aebe00640 1 -- 172.21.15.28:0/448103629 learned_addr learned my addr 172.21.15.28:0/448103629 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:21.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.931+0000 7f2aeb400640 1 --2- 172.21.15.28:0/448103629 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x558992a15330 0x558992a50170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:21.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.931+0000 7f2af0a00640 1 --2- 172.21.15.28:0/448103629 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558992a506b0 0x558992a4e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:21.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.931+0000 7f2aebe00640 1 -- 172.21.15.28:0/448103629 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x558992a15330 msgr2=0x558992a50170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:21.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.931+0000 7f2aebe00640 1 --2- 172.21.15.28:0/448103629 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x558992a15330 0x558992a50170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:21.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.931+0000 7f2aebe00640 1 -- 172.21.15.28:0/448103629 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558992a506b0 msgr2=0x558992a4e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:21.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.931+0000 7f2aebe00640 1 --2- 172.21.15.28:0/448103629 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558992a506b0 0x558992a4e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:21.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.932+0000 7f2aebe00640 1 -- 172.21.15.28:0/448103629 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558992928d90 con 0x558992a14020 2024-02-21T23:03:21.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.932+0000 7f2af0a00640 1 --2- 172.21.15.28:0/448103629 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558992a506b0 0x558992a4e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:03:21.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.932+0000 7f2aebe00640 1 --2- 172.21.15.28:0/448103629 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558992a14020 0x558992a14400 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f2adc0893f0 tx=0x7f2adc059160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad4d6f003831d4f4 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:21.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.932+0000 7f2ad7e00640 1 -- 172.21.15.28:0/448103629 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2adc08b0d0 con 0x558992a14020 2024-02-21T23:03:21.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.932+0000 7f2ad7e00640 1 -- 172.21.15.28:0/448103629 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2adc061de0 con 0x558992a14020 2024-02-21T23:03:21.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.932+0000 7f2ad7e00640 1 -- 172.21.15.28:0/448103629 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2adc0579d0 con 0x558992a14020 2024-02-21T23:03:21.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.933+0000 7f2af5a32740 1 -- 172.21.15.28:0/448103629 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558992a14020 msgr2=0x558992a14400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:21.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.933+0000 7f2af5a32740 1 --2- 172.21.15.28:0/448103629 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558992a14020 0x558992a14400 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f2adc0893f0 tx=0x7f2adc059160 comp rx=0 tx=0).stop 2024-02-21T23:03:21.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.933+0000 7f2af5a32740 1 -- 172.21.15.28:0/448103629 shutdown_connections 2024-02-21T23:03:21.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.933+0000 7f2af5a32740 1 --2- 172.21.15.28:0/448103629 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x558992a15330 0x558992a50170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:21.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.933+0000 7f2af5a32740 1 --2- 172.21.15.28:0/448103629 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558992a506b0 0x558992a4e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:21.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.933+0000 7f2af5a32740 1 --2- 172.21.15.28:0/448103629 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558992a14020 0x558992a14400 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:21.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.933+0000 7f2af5a32740 1 -- 172.21.15.28:0/448103629 >> 172.21.15.28:0/448103629 conn(0x558992a10970 msgr2=0x558992a35380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:21.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.933+0000 7f2af5a32740 1 -- 172.21.15.28:0/448103629 shutdown_connections 2024-02-21T23:03:21.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.933+0000 7f2af5a32740 1 -- 172.21.15.28:0/448103629 wait complete. 2024-02-21T23:03:21.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.934+0000 7f2af5a32740 1 -- start start 2024-02-21T23:03:21.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.934+0000 7f2af5a32740 1 Processor -- start 2024-02-21T23:03:21.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.935+0000 7f2af5a32740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558992a15330 0x558992a63bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:21.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.935+0000 7f2af5a32740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x558992a506b0 0x558992a33cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:21.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.935+0000 7f2af5a32740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558992a34210 0x558992a2f590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:21.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.935+0000 7f2af5a32740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x558992917040 con 0x558992a34210 2024-02-21T23:03:21.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.935+0000 7f2af5a32740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x558992a35a90 con 0x558992a15330 2024-02-21T23:03:21.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.935+0000 7f2af5a32740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x558992a34980 con 0x558992a506b0 2024-02-21T23:03:21.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.935+0000 7f2af5a32740 10 client.0 _ll_register_callbacks cb 0x558992934920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:03:21.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.935+0000 7f2af0a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558992a34210 0x558992a2f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:21.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.935+0000 7f2af0a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558992a34210 0x558992a2f590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46120/0 (socket says 172.21.15.28:46120) 2024-02-21T23:03:21.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.935+0000 7f2af0a00640 1 -- 172.21.15.28:0/3126946630 learned_addr learned my addr 172.21.15.28:0/3126946630 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:21.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.935+0000 7f2aebe00640 1 --2- 172.21.15.28:0/3126946630 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558992a15330 0x558992a63bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:21.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.935+0000 7f2aeb400640 1 --2- 172.21.15.28:0/3126946630 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x558992a506b0 0x558992a33cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:21.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.935+0000 7f2af0a00640 1 -- 172.21.15.28:0/3126946630 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x558992a506b0 msgr2=0x558992a33cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:21.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.935+0000 7f2af0a00640 1 --2- 172.21.15.28:0/3126946630 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x558992a506b0 0x558992a33cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:21.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.935+0000 7f2af0a00640 1 -- 172.21.15.28:0/3126946630 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558992a15330 msgr2=0x558992a63bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:21.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.935+0000 7f2af0a00640 1 --2- 172.21.15.28:0/3126946630 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558992a15330 0x558992a63bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:21.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.935+0000 7f2af0a00640 1 -- 172.21.15.28:0/3126946630 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558992936aa0 con 0x558992a34210 2024-02-21T23:03:21.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.936+0000 7f2af0a00640 1 --2- 172.21.15.28:0/3126946630 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558992a34210 0x558992a2f590 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f2ae40098c0 tx=0x7f2ae4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=521247b013f923a8 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:21.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.936+0000 7f2ad4c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:03:21.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.936+0000 7f2ad4c00640 1 -- 172.21.15.28:0/3126946630 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ae4038040 con 0x558992a34210 2024-02-21T23:03:21.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.936+0000 7f2ad4c00640 1 -- 172.21.15.28:0/3126946630 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f2ae4047410 con 0x558992a34210 2024-02-21T23:03:21.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.936+0000 7f2af5a32740 1 -- 172.21.15.28:0/3126946630 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2adc061de0 con 0x558992a34210 2024-02-21T23:03:21.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.936+0000 7f2ad4c00640 1 -- 172.21.15.28:0/3126946630 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=9) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2ae404e050 con 0x558992a34210 2024-02-21T23:03:21.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.936+0000 7f2af5a32740 10 client.4818 fetch_fsmap learned FSMap version 9 2024-02-21T23:03:21.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.936+0000 7f2af5a32740 1 -- 172.21.15.28:0/3126946630 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=9,monmap=2+}) v3 -- 0x55899268d2d0 con 0x558992a34210 2024-02-21T23:03:21.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.937+0000 7f2ad4c00640 1 -- 172.21.15.28:0/3126946630 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2adc061de0 con 0x558992a34210 2024-02-21T23:03:21.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.937+0000 7f2ad4c00640 1 -- 172.21.15.28:0/3126946630 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 9) v1 ==== 3302+0+0 (secure 0 0 0) 0x7f2ae4051020 con 0x558992a34210 2024-02-21T23:03:21.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.937+0000 7f2af5a32740 10 client.4818 fetch_fsmap finished waiting for FSMap version 9 2024-02-21T23:03:21.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.937+0000 7f2af5a32740 10 client.4818 resolve_mds: resolved name 'b' to daemon mds.b 2024-02-21T23:03:21.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.937+0000 7f2af5a32740 20 client.4818 populate_metadata read hostname 'smithi028' 2024-02-21T23:03:21.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.937+0000 7f2af5a32740 1 --2- 172.21.15.28:0/3126946630 >> [v2:172.21.15.81:6834/2328151245,v1:172.21.15.81:6837/2328151245] conn(0x558992a60b60 0x558992a6ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:21.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.937+0000 7f2af5a32740 4 client.4818 mds_command: new command op to 4267 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-21T23:03:21.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.937+0000 7f2af5a32740 1 -- 172.21.15.28:0/3126946630 --> [v2:172.21.15.81:6834/2328151245,v1:172.21.15.81:6837/2328151245] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5589929859b0 con 0x558992a60b60 2024-02-21T23:03:21.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.937+0000 7f2aebe00640 1 --2- 172.21.15.28:0/3126946630 >> [v2:172.21.15.81:6834/2328151245,v1:172.21.15.81:6837/2328151245] conn(0x558992a60b60 0x558992a6ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:21.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.938+0000 7f2aebe00640 1 --2- 172.21.15.28:0/3126946630 >> [v2:172.21.15.81:6834/2328151245,v1:172.21.15.81:6837/2328151245] conn(0x558992a60b60 0x558992a6ee30 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c088c5fda4f0857 server_cookie=3956a1d4e7db84af in_seq=0 out_seq=0 2024-02-21T23:03:21.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.938+0000 7f2ad4c00640 10 client.4818 ms_handle_connect on v2:172.21.15.81:6834/2328151245 2024-02-21T23:03:21.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.940+0000 7f2ad4c00640 1 -- 172.21.15.28:0/3126946630 <== mds.0 v2:172.21.15.81:6834/2328151245 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5589929859b0 con 0x558992a60b60 2024-02-21T23:03:21.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.940+0000 7f2ad4c00640 10 client.4818 handle_command_reply: tid=0 2024-02-21T23:03:21.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.940+0000 7f2af5a32740 1 client.4818 shutdown 2024-02-21T23:03:21.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.940+0000 7f2af5a32740 1 -- 172.21.15.28:0/3126946630 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558992a34210 msgr2=0x558992a2f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:21.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.941+0000 7f2af5a32740 1 --2- 172.21.15.28:0/3126946630 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558992a34210 0x558992a2f590 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f2ae40098c0 tx=0x7f2ae4004730 comp rx=0 tx=0).stop 2024-02-21T23:03:21.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.941+0000 7f2af5a32740 1 -- 172.21.15.28:0/3126946630 shutdown_connections 2024-02-21T23:03:21.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.941+0000 7f2af5a32740 1 --2- 172.21.15.28:0/3126946630 >> [v2:172.21.15.81:6834/2328151245,v1:172.21.15.81:6837/2328151245] conn(0x558992a60b60 0x558992a6ee30 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:21.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.941+0000 7f2af5a32740 1 --2- 172.21.15.28:0/3126946630 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x558992a506b0 0x558992a33cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:21.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.941+0000 7f2af5a32740 1 --2- 172.21.15.28:0/3126946630 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558992a34210 0x558992a2f590 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:21.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.941+0000 7f2af5a32740 1 --2- 172.21.15.28:0/3126946630 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558992a15330 0x558992a63bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:21.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.941+0000 7f2af5a32740 1 -- 172.21.15.28:0/3126946630 >> 172.21.15.28:0/3126946630 conn(0x558992a14020 msgr2=0x558992a4f080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:21.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.941+0000 7f2ad4c00640 0 client.4818 ms_handle_reset on v2:172.21.15.81:6834/2328151245 2024-02-21T23:03:21.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.941+0000 7f2af5a32740 1 -- 172.21.15.28:0/3126946630 shutdown_connections 2024-02-21T23:03:21.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.941+0000 7f2af5a32740 1 -- 172.21.15.28:0/3126946630 wait complete. 2024-02-21T23:03:21.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.941+0000 7f2af5a32740 20 client.4818 trim_cache size 0 max 16384 2024-02-21T23:03:21.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.959+0000 7f2af5a32740 1 Processor -- start 2024-02-21T23:03:21.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.960+0000 7f2af5a32740 1 -- start start 2024-02-21T23:03:21.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.960+0000 7f2af5a32740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x558992a15330 0x558992a0d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:21.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.960+0000 7f2af5a32740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558992a34210 0x558992a0e100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:21.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.960+0000 7f2af5a32740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558992a506b0 0x558992a0e640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:21.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.960+0000 7f2af5a32740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aec1f7a10 con 0x558992a34210 2024-02-21T23:03:21.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.960+0000 7f2af5a32740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aec1f6e10 con 0x558992a506b0 2024-02-21T23:03:21.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.960+0000 7f2af5a32740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2aec1f7410 con 0x558992a15330 2024-02-21T23:03:21.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.961+0000 7f2aeb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558992a34210 0x558992a0e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:21.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.961+0000 7f2aeb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558992a34210 0x558992a0e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46128/0 (socket says 172.21.15.28:46128) 2024-02-21T23:03:21.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.961+0000 7f2aeb400640 1 -- 172.21.15.28:0/3347038162 learned_addr learned my addr 172.21.15.28:0/3347038162 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:21.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.961+0000 7f2aebe00640 1 --2- 172.21.15.28:0/3347038162 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x558992a15330 0x558992a0d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:21.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.961+0000 7f2af0a00640 1 --2- 172.21.15.28:0/3347038162 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558992a506b0 0x558992a0e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:21.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.961+0000 7f2aeb400640 1 -- 172.21.15.28:0/3347038162 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x558992a15330 msgr2=0x558992a0d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:21.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.961+0000 7f2aeb400640 1 --2- 172.21.15.28:0/3347038162 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x558992a15330 0x558992a0d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:21.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.961+0000 7f2aeb400640 1 -- 172.21.15.28:0/3347038162 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558992a506b0 msgr2=0x558992a0e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:21.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.961+0000 7f2aeb400640 1 --2- 172.21.15.28:0/3347038162 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558992a506b0 0x558992a0e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:21.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.961+0000 7f2aeb400640 1 -- 172.21.15.28:0/3347038162 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5589927576b0 con 0x558992a34210 2024-02-21T23:03:21.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.961+0000 7f2af0a00640 1 --2- 172.21.15.28:0/3347038162 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558992a506b0 0x558992a0e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:03:21.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.961+0000 7f2aeb400640 1 --2- 172.21.15.28:0/3347038162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558992a34210 0x558992a0e100 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f2ad8005b10 tx=0x7f2ad8012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ce8603ac46c6e8c server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:21.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.962+0000 7f2ad7400640 1 -- 172.21.15.28:0/3347038162 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ad80076e0 con 0x558992a34210 2024-02-21T23:03:21.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.962+0000 7f2ad7400640 1 -- 172.21.15.28:0/3347038162 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ad800e500 con 0x558992a34210 2024-02-21T23:03:21.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.962+0000 7f2ad7400640 1 -- 172.21.15.28:0/3347038162 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ad800e6b0 con 0x558992a34210 2024-02-21T23:03:21.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.962+0000 7f2af5a32740 1 -- 172.21.15.28:0/3347038162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558992a34210 msgr2=0x558992a0e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:21.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.962+0000 7f2af5a32740 1 --2- 172.21.15.28:0/3347038162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558992a34210 0x558992a0e100 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f2ad8005b10 tx=0x7f2ad8012000 comp rx=0 tx=0).stop 2024-02-21T23:03:21.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.962+0000 7f2af5a32740 1 -- 172.21.15.28:0/3347038162 shutdown_connections 2024-02-21T23:03:21.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.962+0000 7f2af5a32740 1 --2- 172.21.15.28:0/3347038162 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558992a506b0 0x558992a0e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:21.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.962+0000 7f2af5a32740 1 --2- 172.21.15.28:0/3347038162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558992a34210 0x558992a0e100 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:21.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.962+0000 7f2af5a32740 1 --2- 172.21.15.28:0/3347038162 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x558992a15330 0x558992a0d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:21.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.962+0000 7f2af5a32740 1 -- 172.21.15.28:0/3347038162 >> 172.21.15.28:0/3347038162 conn(0x558992a14020 msgr2=0x558992a4fd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:21.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.962+0000 7f2af5a32740 1 -- 172.21.15.28:0/3347038162 shutdown_connections 2024-02-21T23:03:21.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.963+0000 7f2af5a32740 1 -- 172.21.15.28:0/3347038162 wait complete. 2024-02-21T23:03:21.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.963+0000 7f2af5a32740 1 -- start start 2024-02-21T23:03:21.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.963+0000 7f2af5a32740 1 Processor -- start 2024-02-21T23:03:21.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.964+0000 7f2af5a32740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558992a15330 0x558992a6a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:21.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.964+0000 7f2af5a32740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x558992a34210 0x558992ac0140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:21.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.964+0000 7f2af5a32740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558992a506b0 0x558992abd090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:21.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.964+0000 7f2af5a32740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x558992a346c0 con 0x558992a15330 2024-02-21T23:03:21.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.964+0000 7f2af5a32740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x558992a54cf0 con 0x558992a506b0 2024-02-21T23:03:21.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.964+0000 7f2af5a32740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x558992a55230 con 0x558992a34210 2024-02-21T23:03:21.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.964+0000 7f2af5a32740 10 client.0 _ll_register_callbacks cb 0x7f2ae4036070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:03:21.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.965+0000 7f2aebe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558992a15330 0x558992a6a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:21.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.965+0000 7f2aeb400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x558992a34210 0x558992ac0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:21.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.965+0000 7f2aebe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558992a15330 0x558992a6a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46144/0 (socket says 172.21.15.28:46144) 2024-02-21T23:03:21.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.965+0000 7f2af0a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558992a506b0 0x558992abd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:21.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.965+0000 7f2aebe00640 1 -- 172.21.15.28:0/4048524405 learned_addr learned my addr 172.21.15.28:0/4048524405 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:21.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.965+0000 7f2aebe00640 1 -- 172.21.15.28:0/4048524405 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x558992a34210 msgr2=0x558992ac0140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:21.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.965+0000 7f2aebe00640 1 --2- 172.21.15.28:0/4048524405 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x558992a34210 0x558992ac0140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:21.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.965+0000 7f2aebe00640 1 -- 172.21.15.28:0/4048524405 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558992a506b0 msgr2=0x558992abd090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:21.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.965+0000 7f2aebe00640 1 --2- 172.21.15.28:0/4048524405 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558992a506b0 0x558992abd090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:21.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.965+0000 7f2aebe00640 1 -- 172.21.15.28:0/4048524405 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558992ac0a30 con 0x558992a15330 2024-02-21T23:03:21.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.965+0000 7f2af0a00640 1 --2- 172.21.15.28:0/4048524405 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558992a506b0 0x558992abd090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:03:21.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.965+0000 7f2aeb400640 1 --2- 172.21.15.28:0/4048524405 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x558992a34210 0x558992ac0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:03:21.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.966+0000 7f2aebe00640 1 --2- 172.21.15.28:0/4048524405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558992a15330 0x558992a6a340 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f2adc05d9e0 tx=0x7f2adc05c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12b11a16078be63 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:21.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.966+0000 7f2ad4c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:03:21.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.966+0000 7f2ad4c00640 1 -- 172.21.15.28:0/4048524405 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2adc061de0 con 0x558992a15330 2024-02-21T23:03:21.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.966+0000 7f2ad4c00640 1 -- 172.21.15.28:0/4048524405 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f2adc057770 con 0x558992a15330 2024-02-21T23:03:21.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.966+0000 7f2af5a32740 1 -- 172.21.15.28:0/4048524405 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2ad800e500 con 0x558992a15330 2024-02-21T23:03:21.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.966+0000 7f2ad4c00640 1 -- 172.21.15.28:0/4048524405 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=9) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2adc071180 con 0x558992a15330 2024-02-21T23:03:21.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.966+0000 7f2af5a32740 10 client.4824 fetch_fsmap learned FSMap version 9 2024-02-21T23:03:21.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.966+0000 7f2af5a32740 1 -- 172.21.15.28:0/4048524405 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=9,monmap=2+}) v3 -- 0x558992a637f0 con 0x558992a15330 2024-02-21T23:03:21.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.966+0000 7f2ad4c00640 1 -- 172.21.15.28:0/4048524405 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ad800e500 con 0x558992a15330 2024-02-21T23:03:21.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.966+0000 7f2ad4c00640 1 -- 172.21.15.28:0/4048524405 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 9) v1 ==== 3302+0+0 (secure 0 0 0) 0x7f2adc055d40 con 0x558992a15330 2024-02-21T23:03:21.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.966+0000 7f2af5a32740 10 client.4824 fetch_fsmap finished waiting for FSMap version 9 2024-02-21T23:03:21.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.966+0000 7f2af5a32740 10 client.4824 resolve_mds: resolved name 'b' to daemon mds.b 2024-02-21T23:03:21.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.966+0000 7f2af5a32740 20 client.4824 populate_metadata read hostname 'smithi028' 2024-02-21T23:03:21.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.966+0000 7f2af5a32740 1 --2- 172.21.15.28:0/4048524405 >> [v2:172.21.15.81:6834/2328151245,v1:172.21.15.81:6837/2328151245] conn(0x558992abd5d0 0x558992ac3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:21.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.966+0000 7f2af5a32740 4 client.4824 mds_command: new command op to 4267 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-21T23:03:21.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.966+0000 7f2af5a32740 1 -- 172.21.15.28:0/4048524405 --> [v2:172.21.15.81:6834/2328151245,v1:172.21.15.81:6837/2328151245] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55899297fd00 con 0x558992abd5d0 2024-02-21T23:03:21.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.967+0000 7f2aeb400640 1 --2- 172.21.15.28:0/4048524405 >> [v2:172.21.15.81:6834/2328151245,v1:172.21.15.81:6837/2328151245] conn(0x558992abd5d0 0x558992ac3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:21.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.968+0000 7f2aeb400640 1 --2- 172.21.15.28:0/4048524405 >> [v2:172.21.15.81:6834/2328151245,v1:172.21.15.81:6837/2328151245] conn(0x558992abd5d0 0x558992ac3240 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=54e083436c12d9ac server_cookie=2a78370ecaed048d in_seq=0 out_seq=0 2024-02-21T23:03:21.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.968+0000 7f2ad4c00640 10 client.4824 ms_handle_connect on v2:172.21.15.81:6834/2328151245 2024-02-21T23:03:21.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.968+0000 7f2ad4c00640 1 -- 172.21.15.28:0/4048524405 <== mds.0 v2:172.21.15.81:6834/2328151245 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x55899297fd00 con 0x558992abd5d0 2024-02-21T23:03:21.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.968+0000 7f2ad4c00640 10 client.4824 handle_command_reply: tid=0 2024-02-21T23:03:21.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.968+0000 7f2af5a32740 1 client.4824 shutdown 2024-02-21T23:03:21.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.969+0000 7f2af5a32740 1 -- 172.21.15.28:0/4048524405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558992a15330 msgr2=0x558992a6a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:21.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.969+0000 7f2af5a32740 1 --2- 172.21.15.28:0/4048524405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558992a15330 0x558992a6a340 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f2adc05d9e0 tx=0x7f2adc05c040 comp rx=0 tx=0).stop 2024-02-21T23:03:21.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.969+0000 7f2af5a32740 1 -- 172.21.15.28:0/4048524405 shutdown_connections 2024-02-21T23:03:21.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.969+0000 7f2af5a32740 1 --2- 172.21.15.28:0/4048524405 >> [v2:172.21.15.81:6834/2328151245,v1:172.21.15.81:6837/2328151245] conn(0x558992abd5d0 0x558992ac3240 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:21.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.969+0000 7f2af5a32740 1 --2- 172.21.15.28:0/4048524405 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x558992a34210 0x558992ac0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:21.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.969+0000 7f2af5a32740 1 --2- 172.21.15.28:0/4048524405 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x558992a506b0 0x558992abd090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:21.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.969+0000 7f2af5a32740 1 --2- 172.21.15.28:0/4048524405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558992a15330 0x558992a6a340 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:21.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.969+0000 7f2ad4c00640 0 client.4824 ms_handle_reset on v2:172.21.15.81:6834/2328151245 2024-02-21T23:03:21.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.969+0000 7f2af5a32740 1 -- 172.21.15.28:0/4048524405 >> 172.21.15.28:0/4048524405 conn(0x558992a14020 msgr2=0x558992a0d5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:21.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.969+0000 7f2af5a32740 1 -- 172.21.15.28:0/4048524405 shutdown_connections 2024-02-21T23:03:21.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.969+0000 7f2af5a32740 1 -- 172.21.15.28:0/4048524405 wait complete. 2024-02-21T23:03:21.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.969+0000 7f2af5a32740 20 client.4824 trim_cache size 0 max 16384 2024-02-21T23:03:21.963 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-21T23:03:21.963 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac", 2024-02-21T23:03:21.964 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-02-21T23:03:21.964 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4267, 2024-02-21T23:03:21.964 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-02-21T23:03:21.964 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-02-21T23:03:21.964 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-02-21T23:03:21.964 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 43.593589209000001, 2024-02-21T23:03:21.964 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 9, 2024-02-21T23:03:21.964 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 24, 2024-02-21T23:03:21.964 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 24, 2024-02-21T23:03:21.964 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 51.422454621 2024-02-21T23:03:21.964 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-21T23:03:21.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.970+0000 7f2af1400640 1 -- 172.21.15.28:0/1168652715 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2acc06b500 msgr2=0x7f2acc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:21.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.970+0000 7f2af1400640 1 --2- 172.21.15.28:0/1168652715 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2acc06b500 0x7f2acc06d9c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f2aec1eb9e0 tx=0x7f2ad8009040 comp rx=0 tx=0).stop 2024-02-21T23:03:21.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.970+0000 7f2af1400640 1 -- 172.21.15.28:0/1168652715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2aec0c5040 msgr2=0x7f2aec1ee7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:21.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.970+0000 7f2af1400640 1 --2- 172.21.15.28:0/1168652715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2aec0c5040 0x7f2aec1ee7e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f2adc009ec0 tx=0x7f2adc00c490 comp rx=0 tx=0).stop 2024-02-21T23:03:21.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.970+0000 7f2af1400640 1 -- 172.21.15.28:0/1168652715 shutdown_connections 2024-02-21T23:03:21.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.970+0000 7f2af1400640 1 --2- 172.21.15.28:0/1168652715 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2acc06b500 0x7f2acc06d9c0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:21.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.970+0000 7f2af1400640 1 --2- 172.21.15.28:0/1168652715 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2aec1524c0 0x7f2aec149bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:21.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.970+0000 7f2af1400640 1 --2- 172.21.15.28:0/1168652715 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2aec0c85c0 0x7f2aec154070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:21.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.970+0000 7f2af1400640 1 --2- 172.21.15.28:0/1168652715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2aec0c5040 0x7f2aec1ee7e0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:21.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.970+0000 7f2af1400640 1 -- 172.21.15.28:0/1168652715 >> 172.21.15.28:0/1168652715 conn(0x7f2aec0ac2b0 msgr2=0x7f2aec1508a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:21.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.970+0000 7f2af1400640 1 -- 172.21.15.28:0/1168652715 shutdown_connections 2024-02-21T23:03:21.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:21.970+0000 7f2af1400640 1 -- 172.21.15.28:0/1168652715 wait complete. 2024-02-21T23:03:21.980 INFO:teuthology.run_tasks:Running task kclient... 2024-02-21T23:03:21.993 INFO:tasks.kclient:Mounting kernel clients... 2024-02-21T23:03:21.994 INFO:tasks.kclient:config is {'client.0': {}, 'client.1': {}} 2024-02-21T23:03:21.994 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:03:21.994 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-02-21T23:03:22.048 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-02-21T23:03:22.049 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:03:22.050 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-02-21T23:03:22.150 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:03:22.151 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link delete ceph-brx 2024-02-21T23:03:22.273 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:03:22.273 DEBUG:teuthology.orchestra.run.smithi081:> ip netns list 2024-02-21T23:03:22.331 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:03:22.331 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip link delete ceph-brx 2024-02-21T23:03:22.426 INFO:teuthology.orchestra.run.smithi081.stderr:Cannot find device "ceph-brx" 2024-02-21T23:03:22.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-21T23:03:22.431 INFO:tasks.kclient:client.0 config is {'syntax': 'v2'} 2024-02-21T23:03:22.431 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-02-21T23:03:22.431 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-02-21T23:03:22.431 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-02-21T23:03:22.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.663+0000 7f1c8e200640 1 Processor -- start 2024-02-21T23:03:22.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.663+0000 7f1c8e200640 1 -- start start 2024-02-21T23:03:22.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.665+0000 7f1c8e200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f1c880c3280 0x7f1c880c3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:22.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.665+0000 7f1c8e200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1c88072430 0x7f1c881526e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:22.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.665+0000 7f1c8e200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c88152c20 con 0x7f1c88155860 2024-02-21T23:03:22.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.665+0000 7f1c8e200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c880c6560 con 0x7f1c880c3280 2024-02-21T23:03:22.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.665+0000 7f1c8e200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c880c5cc0 con 0x7f1c88072430 2024-02-21T23:03:22.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.665+0000 7f1c87e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f1c88155860 0x7f1c880c46f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39654/0 (socket says 172.21.15.28:39654) 2024-02-21T23:03:22.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.665+0000 7f1c87e00640 1 -- 172.21.15.28:0/3041449029 learned_addr learned my addr 172.21.15.28:0/3041449029 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:03:22.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.665+0000 7f1c7fe00640 1 --2- 172.21.15.28:0/3041449029 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1c88072430 0x7f1c881526e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:22.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.665+0000 7f1c87400640 1 --2- 172.21.15.28:0/3041449029 >> v2:172.21.15.81:3300/0 conn(0x7f1c880c3280 0x7f1c880c3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:22.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.666+0000 7f1c86a00640 1 -- 172.21.15.28:0/3041449029 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2102647730 0 0) 0x7f1c80001490 con 0x7f1c88155860 2024-02-21T23:03:22.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.666+0000 7f1c86a00640 1 -- 172.21.15.28:0/3041449029 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c6c003520 con 0x7f1c88155860 2024-02-21T23:03:22.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.666+0000 7f1c86a00640 1 -- 172.21.15.28:0/3041449029 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 683083003 0 0) 0x7f1c6c003520 con 0x7f1c88155860 2024-02-21T23:03:22.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.666+0000 7f1c86a00640 1 -- 172.21.15.28:0/3041449029 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1c88072430 msgr2=0x7f1c881526e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:22.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.666+0000 7f1c86a00640 1 --2- 172.21.15.28:0/3041449029 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1c88072430 0x7f1c881526e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:22.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.666+0000 7f1c86a00640 1 -- 172.21.15.28:0/3041449029 >> v2:172.21.15.81:3300/0 conn(0x7f1c880c3280 msgr2=0x7f1c880c3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:22.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.666+0000 7f1c86a00640 1 --2- 172.21.15.28:0/3041449029 >> v2:172.21.15.81:3300/0 conn(0x7f1c880c3280 0x7f1c880c3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:22.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.666+0000 7f1c86a00640 1 -- 172.21.15.28:0/3041449029 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c880c5fa0 con 0x7f1c88155860 2024-02-21T23:03:22.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.666+0000 7f1c86a00640 1 -- 172.21.15.28:0/3041449029 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f1c80001bc0 con 0x7f1c88155860 2024-02-21T23:03:22.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.666+0000 7f1c86a00640 1 -- 172.21.15.28:0/3041449029 >> v1:172.21.15.28:6789/0 conn(0x7f1c88155860 legacy=0x7f1c880c46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:22.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.666+0000 7f1c86a00640 1 --2- 172.21.15.28:0/3041449029 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1c6c0039b0 0x7f1c6c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:22.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.666+0000 7f1c86a00640 1 --2- 172.21.15.28:0/3041449029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c6c004800 0x7f1c6c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:22.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.666+0000 7f1c86a00640 1 --2- 172.21.15.28:0/3041449029 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c6c0051c0 0x7f1c6c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:22.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.666+0000 7f1c86a00640 1 -- 172.21.15.28:0/3041449029 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c880c6560 con 0x7f1c6c004800 2024-02-21T23:03:22.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.666+0000 7f1c86a00640 1 -- 172.21.15.28:0/3041449029 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c880c5cc0 con 0x7f1c6c0051c0 2024-02-21T23:03:22.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.666+0000 7f1c86a00640 1 -- 172.21.15.28:0/3041449029 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c6c00bf80 con 0x7f1c6c0039b0 2024-02-21T23:03:22.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.666+0000 7f1c7fe00640 1 --2- 172.21.15.28:0/3041449029 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1c88072430 0x7f1c881526e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-21T23:03:22.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.667+0000 7f1c7fe00640 1 --2- 172.21.15.28:0/3041449029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c6c004800 0x7f1c6c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:22.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.667+0000 7f1c87e00640 1 --2- 172.21.15.28:0/3041449029 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c6c0051c0 0x7f1c6c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:22.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.667+0000 7f1c87400640 1 --2- 172.21.15.28:0/3041449029 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1c6c0039b0 0x7f1c6c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:22.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.667+0000 7f1c7fe00640 1 -- 172.21.15.28:0/3041449029 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1c6c0039b0 msgr2=0x7f1c6c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:22.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.667+0000 7f1c7fe00640 1 --2- 172.21.15.28:0/3041449029 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1c6c0039b0 0x7f1c6c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:22.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.667+0000 7f1c7fe00640 1 -- 172.21.15.28:0/3041449029 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c6c0051c0 msgr2=0x7f1c6c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:22.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.667+0000 7f1c7fe00640 1 --2- 172.21.15.28:0/3041449029 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c6c0051c0 0x7f1c6c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:22.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.667+0000 7f1c7fe00640 1 -- 172.21.15.28:0/3041449029 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c6c003520 con 0x7f1c6c004800 2024-02-21T23:03:22.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.667+0000 7f1c87400640 1 --2- 172.21.15.28:0/3041449029 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1c6c0039b0 0x7f1c6c003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:03:22.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.668+0000 7f1c7fe00640 1 --2- 172.21.15.28:0/3041449029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c6c004800 0x7f1c6c004c80 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f1c88071330 tx=0x7f1c6c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80cf807feabeb306 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:22.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.669+0000 7f1c86a00640 1 -- 172.21.15.28:0/3041449029 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c700097d0 con 0x7f1c6c004800 2024-02-21T23:03:22.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.669+0000 7f1c86a00640 1 -- 172.21.15.28:0/3041449029 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c70004030 con 0x7f1c6c004800 2024-02-21T23:03:22.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.669+0000 7f1c86a00640 1 -- 172.21.15.28:0/3041449029 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c70004400 con 0x7f1c6c004800 2024-02-21T23:03:22.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.669+0000 7f1c8e200640 1 -- 172.21.15.28:0/3041449029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c6c004800 msgr2=0x7f1c6c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:22.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.669+0000 7f1c8e200640 1 --2- 172.21.15.28:0/3041449029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c6c004800 0x7f1c6c004c80 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f1c88071330 tx=0x7f1c6c005830 comp rx=0 tx=0).stop 2024-02-21T23:03:22.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.669+0000 7f1c87e00640 1 -- 172.21.15.28:0/3041449029 reap_dead start 2024-02-21T23:03:22.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.669+0000 7f1c8e200640 1 -- 172.21.15.28:0/3041449029 shutdown_connections 2024-02-21T23:03:22.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.669+0000 7f1c8e200640 1 -- 172.21.15.28:0/3041449029 >> 172.21.15.28:0/3041449029 conn(0x7f1c880c67f0 msgr2=0x7f1c880c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:22.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.670+0000 7f1c8e200640 1 -- 172.21.15.28:0/3041449029 shutdown_connections 2024-02-21T23:03:22.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.670+0000 7f1c8e200640 1 -- 172.21.15.28:0/3041449029 wait complete. 2024-02-21T23:03:22.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.671+0000 7f1c8e200640 1 Processor -- start 2024-02-21T23:03:22.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.671+0000 7f1c8e200640 1 -- start start 2024-02-21T23:03:22.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.672+0000 7f1c8e200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1c6c004800 0x7f1c880c0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:22.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.672+0000 7f1c8e200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c880a8160 0x7f1c880a85c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:22.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.672+0000 7f1c8e200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c880a7400 0x7f1c880a7880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:22.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.672+0000 7f1c8e200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c880c5920 con 0x7f1c880a7400 2024-02-21T23:03:22.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.672+0000 7f1c8e200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c880a6100 con 0x7f1c880a8160 2024-02-21T23:03:22.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.672+0000 7f1c8e200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c880c5aa0 con 0x7f1c6c004800 2024-02-21T23:03:22.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.673+0000 7f1c87e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c880a7400 0x7f1c880a7880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:22.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.673+0000 7f1c87e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c880a7400 0x7f1c880a7880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46164/0 (socket says 172.21.15.28:46164) 2024-02-21T23:03:22.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.673+0000 7f1c87e00640 1 -- 172.21.15.28:0/2546179917 learned_addr learned my addr 172.21.15.28:0/2546179917 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:22.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.673+0000 7f1c7fe00640 1 --2- 172.21.15.28:0/2546179917 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c880a8160 0x7f1c880a85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:22.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.673+0000 7f1c87e00640 1 -- 172.21.15.28:0/2546179917 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1c6c004800 msgr2=0x7f1c880c0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:22.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.673+0000 7f1c87e00640 1 --2- 172.21.15.28:0/2546179917 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1c6c004800 0x7f1c880c0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:22.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.673+0000 7f1c87e00640 1 -- 172.21.15.28:0/2546179917 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c880a8160 msgr2=0x7f1c880a85c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:22.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.673+0000 7f1c87e00640 1 --2- 172.21.15.28:0/2546179917 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c880a8160 0x7f1c880a85c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:22.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.673+0000 7f1c87e00640 1 -- 172.21.15.28:0/2546179917 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c880ac1d0 con 0x7f1c880a7400 2024-02-21T23:03:22.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.673+0000 7f1c7fe00640 1 --2- 172.21.15.28:0/2546179917 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c880a8160 0x7f1c880a85c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:03:22.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.673+0000 7f1c87e00640 1 --2- 172.21.15.28:0/2546179917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c880a7400 0x7f1c880a7880 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f1c88155c40 tx=0x7f1c8000e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:22.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.674+0000 7f1c7f400640 1 -- 172.21.15.28:0/2546179917 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c8000eb90 con 0x7f1c880a7400 2024-02-21T23:03:22.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.674+0000 7f1c7f400640 1 -- 172.21.15.28:0/2546179917 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c8000ed30 con 0x7f1c880a7400 2024-02-21T23:03:22.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.674+0000 7f1c8e200640 1 -- 172.21.15.28:0/2546179917 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c880ac4b0 con 0x7f1c880a7400 2024-02-21T23:03:22.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.674+0000 7f1c7f400640 1 -- 172.21.15.28:0/2546179917 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c80014670 con 0x7f1c880a7400 2024-02-21T23:03:22.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.674+0000 7f1c8e200640 1 -- 172.21.15.28:0/2546179917 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c880aca70 con 0x7f1c880a7400 2024-02-21T23:03:22.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.675+0000 7f1c8e200640 1 -- 172.21.15.28:0/2546179917 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c880c4110 con 0x7f1c880a7400 2024-02-21T23:03:22.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.676+0000 7f1c7f400640 1 -- 172.21.15.28:0/2546179917 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1c80014810 con 0x7f1c880a7400 2024-02-21T23:03:22.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.677+0000 7f1c7f400640 1 --2- 172.21.15.28:0/2546179917 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1c6406b4d0 0x7f1c6406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:22.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.677+0000 7f1c7f400640 1 -- 172.21.15.28:0/2546179917 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f1c8008a310 con 0x7f1c880a7400 2024-02-21T23:03:22.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.678+0000 7f1c87400640 1 --2- 172.21.15.28:0/2546179917 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1c6406b4d0 0x7f1c6406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:22.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.679+0000 7f1c87400640 1 --2- 172.21.15.28:0/2546179917 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1c6406b4d0 0x7f1c6406d990 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1c780099b0 tx=0x7f1c78008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:22.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.683+0000 7f1c7f400640 1 -- 172.21.15.28:0/2546179917 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1c880c4110 con 0x7f1c880a7400 2024-02-21T23:03:22.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.935+0000 7f1c8e200640 1 -- 172.21.15.28:0/2546179917 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f1c88067f30 con 0x7f1c880a7400 2024-02-21T23:03:22.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.936+0000 7f1c7f400640 1 -- 172.21.15.28:0/2546179917 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) v1 ==== 53+0+88 (secure 0 0 0) 0x7f1c88152c20 con 0x7f1c880a7400 2024-02-21T23:03:22.926 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-02-21T23:03:22.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.938+0000 7f1c8e200640 1 -- 172.21.15.28:0/2546179917 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1c6406b4d0 msgr2=0x7f1c6406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:22.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.938+0000 7f1c8e200640 1 --2- 172.21.15.28:0/2546179917 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1c6406b4d0 0x7f1c6406d990 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1c780099b0 tx=0x7f1c78008040 comp rx=0 tx=0).stop 2024-02-21T23:03:22.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.938+0000 7f1c8e200640 1 -- 172.21.15.28:0/2546179917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c880a7400 msgr2=0x7f1c880a7880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:22.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.938+0000 7f1c8e200640 1 --2- 172.21.15.28:0/2546179917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c880a7400 0x7f1c880a7880 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f1c88155c40 tx=0x7f1c8000e530 comp rx=0 tx=0).stop 2024-02-21T23:03:22.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.939+0000 7f1c8e200640 1 -- 172.21.15.28:0/2546179917 shutdown_connections 2024-02-21T23:03:22.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.939+0000 7f1c8e200640 1 --2- 172.21.15.28:0/2546179917 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1c6406b4d0 0x7f1c6406d990 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:22.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.939+0000 7f1c8e200640 1 --2- 172.21.15.28:0/2546179917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c880a7400 0x7f1c880a7880 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:22.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.939+0000 7f1c8e200640 1 --2- 172.21.15.28:0/2546179917 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c880a8160 0x7f1c880a85c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:22.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.939+0000 7f1c8e200640 1 --2- 172.21.15.28:0/2546179917 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1c6c004800 0x7f1c880c0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:22.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.939+0000 7f1c8e200640 1 -- 172.21.15.28:0/2546179917 >> 172.21.15.28:0/2546179917 conn(0x7f1c880c67f0 msgr2=0x7f1c88152420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:22.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.939+0000 7f1c8e200640 1 -- 172.21.15.28:0/2546179917 shutdown_connections 2024-02-21T23:03:22.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:22.939+0000 7f1c8e200640 1 -- 172.21.15.28:0/2546179917 wait complete. 2024-02-21T23:03:22.949 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-02-21T23:03:22.949 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-02-21T23:03:22.949 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi028.front.sepia.ceph.com 2024-02-21T23:03:22.950 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-02-21T23:03:22.950 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-02-21T23:03:22.950 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-02-21T23:03:22.950 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-02-21T23:03:22.950 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-02-21T23:03:22.950 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-02-21T23:03:22.950 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:03:22.950 DEBUG:teuthology.orchestra.run.smithi028:> ip addr 2024-02-21T23:03:23.001 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-02-21T23:03:23.001 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-02-21T23:03:23.001 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-02-21T23:03:23.001 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-02-21T23:03:23.001 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-02-21T23:03:23.001 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-02-21T23:03:23.001 INFO:teuthology.orchestra.run.smithi028.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-21T23:03:23.001 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:16 brd ff:ff:ff:ff:ff:ff 2024-02-21T23:03:23.001 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f0 2024-02-21T23:03:23.001 INFO:teuthology.orchestra.run.smithi028.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-21T23:03:23.001 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:17 brd ff:ff:ff:ff:ff:ff 2024-02-21T23:03:23.002 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f1 2024-02-21T23:03:23.002 INFO:teuthology.orchestra.run.smithi028.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-02-21T23:03:23.002 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3c brd ff:ff:ff:ff:ff:ff 2024-02-21T23:03:23.002 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f0 2024-02-21T23:03:23.002 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.21.15.28/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-02-21T23:03:23.002 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft 42507sec preferred_lft 42507sec 2024-02-21T23:03:23.002 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link noprefixroute 2024-02-21T23:03:23.002 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-02-21T23:03:23.002 INFO:teuthology.orchestra.run.smithi028.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-21T23:03:23.002 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3d brd ff:ff:ff:ff:ff:ff 2024-02-21T23:03:23.002 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f1 2024-02-21T23:03:23.003 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-02-21T23:03:23.003 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:03:23.003 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec bash -c ' 2024-02-21T23:03:23.003 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-02-21T23:03:23.004 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link add name ceph-brx type bridge 2024-02-21T23:03:23.004 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip addr flush dev ceph-brx 2024-02-21T23:03:23.004 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set ceph-brx up 2024-02-21T23:03:23.004 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-02-21T23:03:23.004 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-02-21T23:03:23.232 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:03:23.232 DEBUG:teuthology.orchestra.run.smithi028:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-02-21T23:03:23.311 INFO:teuthology.orchestra.run.smithi028.stdout:1 2024-02-21T23:03:23.315 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:03:23.315 DEBUG:teuthology.orchestra.run.smithi028:> route 2024-02-21T23:03:23.368 INFO:teuthology.orchestra.run.smithi028.stdout:Kernel IP routing table 2024-02-21T23:03:23.368 INFO:teuthology.orchestra.run.smithi028.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-02-21T23:03:23.368 INFO:teuthology.orchestra.run.smithi028.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-02-21T23:03:23.368 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-02-21T23:03:23.369 INFO:teuthology.orchestra.run.smithi028.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-02-21T23:03:23.369 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:03:23.370 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec bash -c ' 2024-02-21T23:03:23.370 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-02-21T23:03:23.370 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-02-21T23:03:23.370 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-02-21T23:03:23.370 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-02-21T23:03:23.370 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-02-21T23:03:23.556 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:03:23.556 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-02-21T23:03:23.610 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:03:23.610 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list-id 2024-02-21T23:03:23.666 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:03:23.666 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec bash -c ' 2024-02-21T23:03:23.666 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-02-21T23:03:23.666 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-02-21T23:03:23.667 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-02-21T23:03:23.667 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-02-21T23:03:23.815 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-02-21T23:03:23.815 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:03:23.815 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec bash -c ' 2024-02-21T23:03:23.815 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-02-21T23:03:23.815 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-02-21T23:03:23.815 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-02-21T23:03:23.815 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-02-21T23:03:23.816 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-02-21T23:03:23.816 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-02-21T23:03:23.816 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-02-21T23:03:24.073 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:03:24.073 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec bash -c ' 2024-02-21T23:03:24.073 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-02-21T23:03:24.073 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set brx.0 up 2024-02-21T23:03:24.073 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set dev brx.0 master ceph-brx 2024-02-21T23:03:24.073 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-02-21T23:03:24.250 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-21T23:03:24.251 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-02-21T23:03:24.302 INFO:teuthology.orchestra.run.smithi028.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-02-21T23:03:24.302 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-21T23:03:24.302 DEBUG:teuthology.orchestra.run.smithi028:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-02-21T23:03:24.350 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-02-21T23:03:24.350 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:03:24.350 DEBUG:teuthology.orchestra.run.smithi028:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-02-21T23:03:24.485 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-02-21T23:03:24.486 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-02-21T23:03:25.207 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-02-21T23:03:25.207 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-02-21T23:03:25.207 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.81:3300,172.21.15.81:3301" 2024-02-21T23:03:25.207 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: trying mount with new device syntax: 0@d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac.cephfs=/ 2024-02-21T23:03:25.207 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.28:3300/172.21.15.81:3300/172.21.15.81:3301" will pass to kernel 2024-02-21T23:03:25.210 INFO:teuthology.orchestra.run.smithi028.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-02-21T23:03:25.210 INFO:teuthology.orchestra.run.smithi028.stdout: You just mounted a file system that supports labels which does not 2024-02-21T23:03:25.210 INFO:teuthology.orchestra.run.smithi028.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-02-21T23:03:25.210 INFO:teuthology.orchestra.run.smithi028.stdout: applications will generate AVC messages and not be allowed access to 2024-02-21T23:03:25.210 INFO:teuthology.orchestra.run.smithi028.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-02-21T23:03:25.215 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-02-21T23:03:25.215 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:03:25.215 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-02-21T23:03:25.312 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:03:25.313 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-21T23:03:25.371 INFO:teuthology.orchestra.run.smithi028.stdout:client4592 2024-02-21T23:03:25.371 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-21T23:03:25.371 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-21T23:03:25.420 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:03:25.420 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-21T23:03:25.523 INFO:teuthology.orchestra.run.smithi028.stdout:client4592 2024-02-21T23:03:25.524 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:03:25.524 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-21T23:03:25.579 INFO:teuthology.orchestra.run.smithi028.stdout:client4592 2024-02-21T23:03:25.579 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:03:25.579 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac.client4592/status') 2024-02-21T23:03:25.673 INFO:teuthology.orchestra.run.smithi028.stdout:instance: client.4592 (3)172.21.15.28:0/102708982 2024-02-21T23:03:25.673 INFO:teuthology.orchestra.run.smithi028.stdout:blocklisted: false 2024-02-21T23:03:25.674 INFO:teuthology.orchestra.run.smithi028.stderr:0+1 records in 2024-02-21T23:03:25.674 INFO:teuthology.orchestra.run.smithi028.stderr:0+1 records out 2024-02-21T23:03:25.674 INFO:teuthology.orchestra.run.smithi028.stderr:69 bytes copied, 9.6997e-05 s, 711 kB/s 2024-02-21T23:03:25.678 INFO:tasks.kclient:client.1 config is {'syntax': 'v2'} 2024-02-21T23:03:25.678 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-02-21T23:03:25.678 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-02-21T23:03:25.678 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-02-21T23:03:25.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:25.910+0000 7f0a5a800640 1 Processor -- start 2024-02-21T23:03:25.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:25.910+0000 7f0a5a800640 1 -- start start 2024-02-21T23:03:25.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:25.911+0000 7f0a5a800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0a5406d0b0 0x7f0a5406f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:25.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:25.912+0000 7f0a5a800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a54156d10 con 0x7f0a540c1730 2024-02-21T23:03:25.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:25.912+0000 7f0a5a800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a54152be0 con 0x7f0a54156150 2024-02-21T23:03:25.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:25.912+0000 7f0a5a800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a54152ea0 con 0x7f0a5406d0b0 2024-02-21T23:03:25.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:25.912+0000 7f0a59400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f0a540c1730 0x7f0a5414ff80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59372/0 (socket says 172.21.15.28:59372) 2024-02-21T23:03:25.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:25.912+0000 7f0a59400640 1 -- 172.21.15.28:0/1191409084 learned_addr learned my addr 172.21.15.28:0/1191409084 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:03:25.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:25.912+0000 7f0a58a00640 1 --2- 172.21.15.28:0/1191409084 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0a5406d0b0 0x7f0a5406f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:25.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:25.912+0000 7f0a53e00640 1 -- 172.21.15.28:0/1191409084 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2094718120 0 0) 0x7f0a54156d10 con 0x7f0a540c1730 2024-02-21T23:03:25.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:25.913+0000 7f0a53e00640 1 -- 172.21.15.28:0/1191409084 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a3c003520 con 0x7f0a540c1730 2024-02-21T23:03:25.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:25.913+0000 7f0a58a00640 1 -- 172.21.15.28:0/1191409084 >> v1:172.21.15.81:6789/0 conn(0x7f0a54156150 legacy=0x7f0a54156600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:25.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:25.913+0000 7f0a58a00640 1 -- 172.21.15.28:0/1191409084 >> v1:172.21.15.28:6789/0 conn(0x7f0a540c1730 legacy=0x7f0a5414ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:25.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:25.913+0000 7f0a58a00640 1 -- 172.21.15.28:0/1191409084 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a54153180 con 0x7f0a5406d0b0 2024-02-21T23:03:25.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:25.914+0000 7f0a58a00640 1 --2- 172.21.15.28:0/1191409084 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0a5406d0b0 0x7f0a5406f3f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0a400097d0 tx=0x7f0a4002b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72fd50805d6cc4d7 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:25.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:25.914+0000 7f0a53e00640 1 -- 172.21.15.28:0/1191409084 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a40034070 con 0x7f0a5406d0b0 2024-02-21T23:03:25.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:25.914+0000 7f0a53e00640 1 -- 172.21.15.28:0/1191409084 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a40030800 con 0x7f0a5406d0b0 2024-02-21T23:03:25.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:25.914+0000 7f0a53e00640 1 -- 172.21.15.28:0/1191409084 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a40030bd0 con 0x7f0a5406d0b0 2024-02-21T23:03:25.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:25.915+0000 7f0a5a800640 1 -- 172.21.15.28:0/1191409084 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0a5406d0b0 msgr2=0x7f0a5406f3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:25.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:25.915+0000 7f0a5a800640 1 --2- 172.21.15.28:0/1191409084 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0a5406d0b0 0x7f0a5406f3f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0a400097d0 tx=0x7f0a4002b940 comp rx=0 tx=0).stop 2024-02-21T23:03:25.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:25.915+0000 7f0a5a800640 1 -- 172.21.15.28:0/1191409084 shutdown_connections 2024-02-21T23:03:25.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:25.915+0000 7f0a5a800640 1 --2- 172.21.15.28:0/1191409084 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0a5406d0b0 0x7f0a5406f3f0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:25.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:25.915+0000 7f0a5a800640 1 -- 172.21.15.28:0/1191409084 >> 172.21.15.28:0/1191409084 conn(0x7f0a540a8b60 msgr2=0x7f0a540c3280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:25.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:25.915+0000 7f0a5a800640 1 -- 172.21.15.28:0/1191409084 shutdown_connections 2024-02-21T23:03:25.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:25.915+0000 7f0a5a800640 1 -- 172.21.15.28:0/1191409084 wait complete. 2024-02-21T23:03:25.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:25.917+0000 7f0a5a800640 1 Processor -- start 2024-02-21T23:03:25.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:25.917+0000 7f0a5a800640 1 -- start start 2024-02-21T23:03:25.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:25.918+0000 7f0a5a800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0a5406d0b0 0x7f0a54151460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:25.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:25.918+0000 7f0a5a800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a540c1730 0x7f0a5414b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:25.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:25.918+0000 7f0a5a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a54156150 0x7f0a5414ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:25.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:25.918+0000 7f0a5a800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a540c3ef0 con 0x7f0a54156150 2024-02-21T23:03:25.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:25.918+0000 7f0a5a800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a54153740 con 0x7f0a540c1730 2024-02-21T23:03:25.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:25.918+0000 7f0a5a800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a540c4070 con 0x7f0a5406d0b0 2024-02-21T23:03:25.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:25.918+0000 7f0a59e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a54156150 0x7f0a5414ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:25.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:25.918+0000 7f0a59e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a54156150 0x7f0a5414ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45078/0 (socket says 172.21.15.28:45078) 2024-02-21T23:03:25.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:25.918+0000 7f0a59e00640 1 -- 172.21.15.28:0/3523561765 learned_addr learned my addr 172.21.15.28:0/3523561765 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:25.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:25.918+0000 7f0a59400640 1 --2- 172.21.15.28:0/3523561765 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0a5406d0b0 0x7f0a54151460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:25.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:25.919+0000 7f0a58a00640 1 --2- 172.21.15.28:0/3523561765 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a540c1730 0x7f0a5414b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:25.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:25.919+0000 7f0a59e00640 1 -- 172.21.15.28:0/3523561765 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0a5406d0b0 msgr2=0x7f0a54151460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:25.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:25.919+0000 7f0a59e00640 1 --2- 172.21.15.28:0/3523561765 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0a5406d0b0 0x7f0a54151460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:25.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:25.919+0000 7f0a59e00640 1 -- 172.21.15.28:0/3523561765 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a540c1730 msgr2=0x7f0a5414b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:25.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:25.919+0000 7f0a59e00640 1 --2- 172.21.15.28:0/3523561765 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a540c1730 0x7f0a5414b7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:25.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:25.919+0000 7f0a59e00640 1 -- 172.21.15.28:0/3523561765 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a5413f0e0 con 0x7f0a54156150 2024-02-21T23:03:25.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:25.919+0000 7f0a59e00640 1 --2- 172.21.15.28:0/3523561765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a54156150 0x7f0a5414ef20 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c00c6b0 tx=0x7f0a4c00cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:25.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:25.919+0000 7f0a51600640 1 -- 172.21.15.28:0/3523561765 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a4c003b70 con 0x7f0a54156150 2024-02-21T23:03:25.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:25.919+0000 7f0a51600640 1 -- 172.21.15.28:0/3523561765 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a4c003d30 con 0x7f0a54156150 2024-02-21T23:03:25.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:25.919+0000 7f0a51600640 1 -- 172.21.15.28:0/3523561765 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a4c005310 con 0x7f0a54156150 2024-02-21T23:03:25.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:25.919+0000 7f0a5a800640 1 -- 172.21.15.28:0/3523561765 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a541ec1b0 con 0x7f0a54156150 2024-02-21T23:03:25.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:25.919+0000 7f0a5a800640 1 -- 172.21.15.28:0/3523561765 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a541ec770 con 0x7f0a54156150 2024-02-21T23:03:25.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:25.921+0000 7f0a5a800640 1 -- 172.21.15.28:0/3523561765 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a5404f290 con 0x7f0a54156150 2024-02-21T23:03:25.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:25.921+0000 7f0a51600640 1 -- 172.21.15.28:0/3523561765 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0a4c0054b0 con 0x7f0a54156150 2024-02-21T23:03:25.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:25.922+0000 7f0a51600640 1 --2- 172.21.15.28:0/3523561765 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0a3406b4d0 0x7f0a3406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:25.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:25.923+0000 7f0a51600640 1 -- 172.21.15.28:0/3523561765 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f0a4c08a410 con 0x7f0a54156150 2024-02-21T23:03:25.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:25.923+0000 7f0a59400640 1 --2- 172.21.15.28:0/3523561765 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0a3406b4d0 0x7f0a3406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:25.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:25.924+0000 7f0a59400640 1 --2- 172.21.15.28:0/3523561765 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0a3406b4d0 0x7f0a3406d990 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f0a4800e4a0 tx=0x7f0a48007d30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:25.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:25.927+0000 7f0a51600640 1 -- 172.21.15.28:0/3523561765 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f0a4c058e80 con 0x7f0a54156150 2024-02-21T23:03:26.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:26.169+0000 7f0a5a800640 1 -- 172.21.15.28:0/3523561765 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f0a540ac860 con 0x7f0a54156150 2024-02-21T23:03:26.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:26.170+0000 7f0a51600640 1 -- 172.21.15.28:0/3523561765 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) v1 ==== 53+0+88 (secure 0 0 0) 0x7f0a4c058780 con 0x7f0a54156150 2024-02-21T23:03:26.159 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-02-21T23:03:26.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:26.173+0000 7f0a5a800640 1 -- 172.21.15.28:0/3523561765 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0a3406b4d0 msgr2=0x7f0a3406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:26.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:26.173+0000 7f0a5a800640 1 --2- 172.21.15.28:0/3523561765 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0a3406b4d0 0x7f0a3406d990 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f0a4800e4a0 tx=0x7f0a48007d30 comp rx=0 tx=0).stop 2024-02-21T23:03:26.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:26.173+0000 7f0a5a800640 1 -- 172.21.15.28:0/3523561765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a54156150 msgr2=0x7f0a5414ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:26.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:26.173+0000 7f0a5a800640 1 --2- 172.21.15.28:0/3523561765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a54156150 0x7f0a5414ef20 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c00c6b0 tx=0x7f0a4c00cb70 comp rx=0 tx=0).stop 2024-02-21T23:03:26.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:26.173+0000 7f0a5a800640 1 -- 172.21.15.28:0/3523561765 shutdown_connections 2024-02-21T23:03:26.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:26.173+0000 7f0a5a800640 1 --2- 172.21.15.28:0/3523561765 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0a3406b4d0 0x7f0a3406d990 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:26.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:26.173+0000 7f0a5a800640 1 --2- 172.21.15.28:0/3523561765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a54156150 0x7f0a5414ef20 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:26.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:26.173+0000 7f0a5a800640 1 --2- 172.21.15.28:0/3523561765 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a540c1730 0x7f0a5414b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:26.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:26.173+0000 7f0a5a800640 1 --2- 172.21.15.28:0/3523561765 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0a5406d0b0 0x7f0a54151460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:26.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:26.173+0000 7f0a5a800640 1 -- 172.21.15.28:0/3523561765 >> 172.21.15.28:0/3523561765 conn(0x7f0a540a8b60 msgr2=0x7f0a5414fd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:26.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:26.173+0000 7f0a5a800640 1 -- 172.21.15.28:0/3523561765 shutdown_connections 2024-02-21T23:03:26.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:26.173+0000 7f0a5a800640 1 -- 172.21.15.28:0/3523561765 wait complete. 2024-02-21T23:03:26.182 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-02-21T23:03:26.183 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-02-21T23:03:26.183 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi081.front.sepia.ceph.com 2024-02-21T23:03:26.183 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-02-21T23:03:26.183 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-02-21T23:03:26.183 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-02-21T23:03:26.183 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-02-21T23:03:26.183 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-02-21T23:03:26.183 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-02-21T23:03:26.183 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:03:26.183 DEBUG:teuthology.orchestra.run.smithi081:> ip addr 2024-02-21T23:03:26.239 INFO:teuthology.orchestra.run.smithi081.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-02-21T23:03:26.239 INFO:teuthology.orchestra.run.smithi081.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-02-21T23:03:26.239 INFO:teuthology.orchestra.run.smithi081.stdout: inet 127.0.0.1/8 scope host lo 2024-02-21T23:03:26.239 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-02-21T23:03:26.239 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 ::1/128 scope host 2024-02-21T23:03:26.239 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-02-21T23:03:26.240 INFO:teuthology.orchestra.run.smithi081.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-21T23:03:26.240 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:d8:a1:30 brd ff:ff:ff:ff:ff:ff 2024-02-21T23:03:26.240 INFO:teuthology.orchestra.run.smithi081.stdout: altname enp7s0f0 2024-02-21T23:03:26.240 INFO:teuthology.orchestra.run.smithi081.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-21T23:03:26.240 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:d8:a1:31 brd ff:ff:ff:ff:ff:ff 2024-02-21T23:03:26.240 INFO:teuthology.orchestra.run.smithi081.stdout: altname enp7s0f1 2024-02-21T23:03:26.240 INFO:teuthology.orchestra.run.smithi081.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-21T23:03:26.240 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:8f:d7:52 brd ff:ff:ff:ff:ff:ff 2024-02-21T23:03:26.240 INFO:teuthology.orchestra.run.smithi081.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-02-21T23:03:26.240 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:8f:d7:53 brd ff:ff:ff:ff:ff:ff 2024-02-21T23:03:26.240 INFO:teuthology.orchestra.run.smithi081.stdout: inet 172.21.15.81/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-02-21T23:03:26.240 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft 42512sec preferred_lft 42512sec 2024-02-21T23:03:26.240 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::ec4:7aff:fe8f:d753/64 scope link 2024-02-21T23:03:26.241 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-02-21T23:03:26.241 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-02-21T23:03:26.241 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:03:26.242 DEBUG:teuthology.orchestra.run.smithi081:> (cd / && exec bash -c ' 2024-02-21T23:03:26.242 DEBUG:teuthology.orchestra.run.smithi081:> set -e 2024-02-21T23:03:26.242 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip link add name ceph-brx type bridge 2024-02-21T23:03:26.242 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip addr flush dev ceph-brx 2024-02-21T23:03:26.242 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip link set ceph-brx up 2024-02-21T23:03:26.242 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-02-21T23:03:26.242 DEBUG:teuthology.orchestra.run.smithi081:> ') 2024-02-21T23:03:26.684 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:03:26.684 DEBUG:teuthology.orchestra.run.smithi081:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-02-21T23:03:26.766 INFO:teuthology.orchestra.run.smithi081.stdout:1 2024-02-21T23:03:26.771 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:03:26.771 DEBUG:teuthology.orchestra.run.smithi081:> route 2024-02-21T23:03:26.822 INFO:teuthology.orchestra.run.smithi081.stdout:Kernel IP routing table 2024-02-21T23:03:26.823 INFO:teuthology.orchestra.run.smithi081.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-02-21T23:03:26.823 INFO:teuthology.orchestra.run.smithi081.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-02-21T23:03:26.823 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-02-21T23:03:26.823 INFO:teuthology.orchestra.run.smithi081.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-02-21T23:03:26.823 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:03:26.823 DEBUG:teuthology.orchestra.run.smithi081:> (cd / && exec bash -c ' 2024-02-21T23:03:26.823 DEBUG:teuthology.orchestra.run.smithi081:> set -e 2024-02-21T23:03:26.823 DEBUG:teuthology.orchestra.run.smithi081:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-02-21T23:03:26.824 DEBUG:teuthology.orchestra.run.smithi081:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-02-21T23:03:26.824 DEBUG:teuthology.orchestra.run.smithi081:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-02-21T23:03:26.824 DEBUG:teuthology.orchestra.run.smithi081:> ') 2024-02-21T23:03:27.505 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:03:27.505 DEBUG:teuthology.orchestra.run.smithi081:> ip netns list 2024-02-21T23:03:27.554 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:03:27.554 DEBUG:teuthology.orchestra.run.smithi081:> ip netns list-id 2024-02-21T23:03:27.637 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:03:27.637 DEBUG:teuthology.orchestra.run.smithi081:> (cd / && exec bash -c ' 2024-02-21T23:03:27.637 DEBUG:teuthology.orchestra.run.smithi081:> set -e 2024-02-21T23:03:27.637 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-02-21T23:03:27.637 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-02-21T23:03:27.637 DEBUG:teuthology.orchestra.run.smithi081:> ') 2024-02-21T23:03:27.794 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-02-21T23:03:27.795 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:03:27.795 DEBUG:teuthology.orchestra.run.smithi081:> (cd / && exec bash -c ' 2024-02-21T23:03:27.795 DEBUG:teuthology.orchestra.run.smithi081:> set -e 2024-02-21T23:03:27.795 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-02-21T23:03:27.795 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-02-21T23:03:27.795 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-02-21T23:03:27.795 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-02-21T23:03:27.795 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-02-21T23:03:27.795 DEBUG:teuthology.orchestra.run.smithi081:> ') 2024-02-21T23:03:28.052 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:03:28.053 DEBUG:teuthology.orchestra.run.smithi081:> (cd / && exec bash -c ' 2024-02-21T23:03:28.053 DEBUG:teuthology.orchestra.run.smithi081:> set -e 2024-02-21T23:03:28.053 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip link set brx.0 up 2024-02-21T23:03:28.053 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip link set dev brx.0 master ceph-brx 2024-02-21T23:03:28.053 DEBUG:teuthology.orchestra.run.smithi081:> ') 2024-02-21T23:03:28.200 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-21T23:03:28.200 DEBUG:teuthology.orchestra.run.smithi081:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-02-21T23:03:28.240 INFO:teuthology.orchestra.run.smithi081.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-02-21T23:03:28.240 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-21T23:03:28.240 DEBUG:teuthology.orchestra.run.smithi081:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-02-21T23:03:28.323 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2024-02-21T23:03:28.323 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:03:28.323 DEBUG:teuthology.orchestra.run.smithi081:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-02-21T23:03:28.491 INFO:teuthology.orchestra.run.smithi081.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-02-21T23:03:28.491 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-02-21T23:03:29.165 INFO:teuthology.orchestra.run.smithi081.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-02-21T23:03:29.165 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-02-21T23:03:29.165 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.81:3300,172.21.15.81:3301" 2024-02-21T23:03:29.165 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: trying mount with new device syntax: 1@d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac.cephfs=/ 2024-02-21T23:03:29.165 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.28:3300/172.21.15.81:3300/172.21.15.81:3301" will pass to kernel 2024-02-21T23:03:29.168 INFO:teuthology.orchestra.run.smithi081.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-02-21T23:03:29.168 INFO:teuthology.orchestra.run.smithi081.stdout: You just mounted a file system that supports labels which does not 2024-02-21T23:03:29.169 INFO:teuthology.orchestra.run.smithi081.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-02-21T23:03:29.169 INFO:teuthology.orchestra.run.smithi081.stdout: applications will generate AVC messages and not be allowed access to 2024-02-21T23:03:29.169 INFO:teuthology.orchestra.run.smithi081.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-02-21T23:03:29.172 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-02-21T23:03:29.172 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:03:29.172 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-02-21T23:03:29.257 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:03:29.257 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-21T23:03:29.307 INFO:teuthology.orchestra.run.smithi081.stdout:client4839 2024-02-21T23:03:29.308 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-02-21T23:03:29.308 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-21T23:03:29.350 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:03:29.350 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-21T23:03:29.443 INFO:teuthology.orchestra.run.smithi081.stdout:client4839 2024-02-21T23:03:29.443 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:03:29.443 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-21T23:03:29.489 INFO:teuthology.orchestra.run.smithi081.stdout:client4839 2024-02-21T23:03:29.489 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:03:29.489 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac.client4839/status') 2024-02-21T23:03:29.617 INFO:teuthology.orchestra.run.smithi081.stdout:instance: client.4839 (3)172.21.15.81:0/1805519135 2024-02-21T23:03:29.617 INFO:teuthology.orchestra.run.smithi081.stdout:blocklisted: false 2024-02-21T23:03:29.617 INFO:teuthology.orchestra.run.smithi081.stderr:0+1 records in 2024-02-21T23:03:29.617 INFO:teuthology.orchestra.run.smithi081.stderr:0+1 records out 2024-02-21T23:03:29.617 INFO:teuthology.orchestra.run.smithi081.stderr:70 bytes copied, 6.3579e-05 s, 1.1 MB/s 2024-02-21T23:03:29.622 INFO:teuthology.run_tasks:Running task exec... 2024-02-21T23:03:29.631 INFO:teuthology.task.exec:Executing custom commands... 2024-02-21T23:03:29.631 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi028.front.sepia.ceph.com 2024-02-21T23:03:29.632 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/crash/warn_recent_interval 0' 2024-02-21T23:03:29.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.870+0000 7f8faae00640 1 Processor -- start 2024-02-21T23:03:29.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.870+0000 7f8faae00640 1 -- start start 2024-02-21T23:03:29.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.872+0000 7f8faae00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f8fa40c5950 0x7f8fa40c07b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:29.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.872+0000 7f8faae00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fa4156c10 con 0x7f8fa4156830 2024-02-21T23:03:29.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.872+0000 7f8faae00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fa4159540 con 0x7f8fa40c4670 2024-02-21T23:03:29.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.872+0000 7f8faae00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fa415a150 con 0x7f8fa40c5950 2024-02-21T23:03:29.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.872+0000 7f8fa9a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f8fa4156830 0x7f8fa406aed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59376/0 (socket says 172.21.15.28:59376) 2024-02-21T23:03:29.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.872+0000 7f8fa9a00640 1 -- 172.21.15.28:0/4109875444 learned_addr learned my addr 172.21.15.28:0/4109875444 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:03:29.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.873+0000 7f8fa9000640 1 --2- 172.21.15.28:0/4109875444 >> v2:172.21.15.81:3300/0 conn(0x7f8fa40c5950 0x7f8fa40c07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:29.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.873+0000 7f8fa3e00640 1 -- 172.21.15.28:0/4109875444 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1233516857 0 0) 0x7f8fa4156c10 con 0x7f8fa4156830 2024-02-21T23:03:29.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.873+0000 7f8fa3e00640 1 -- 172.21.15.28:0/4109875444 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f8c003520 con 0x7f8fa4156830 2024-02-21T23:03:29.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.873+0000 7f8fa3e00640 1 -- 172.21.15.28:0/4109875444 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1483639274 0 0) 0x7f8f8c003520 con 0x7f8fa4156830 2024-02-21T23:03:29.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.873+0000 7f8fa3e00640 1 -- 172.21.15.28:0/4109875444 >> v2:172.21.15.81:3300/0 conn(0x7f8fa40c5950 msgr2=0x7f8fa40c07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:29.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.873+0000 7f8fa3e00640 1 --2- 172.21.15.28:0/4109875444 >> v2:172.21.15.81:3300/0 conn(0x7f8fa40c5950 0x7f8fa40c07b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:29.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.873+0000 7f8fa3e00640 1 -- 172.21.15.28:0/4109875444 >> v1:172.21.15.81:6789/0 conn(0x7f8fa40c4670 legacy=0x7f8fa40c4ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:29.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.873+0000 7f8fa3e00640 1 -- 172.21.15.28:0/4109875444 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fa4159820 con 0x7f8fa4156830 2024-02-21T23:03:29.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.873+0000 7f8fa3e00640 1 -- 172.21.15.28:0/4109875444 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f8f98001670 con 0x7f8fa4156830 2024-02-21T23:03:29.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.873+0000 7f8fa3e00640 1 -- 172.21.15.28:0/4109875444 >> v1:172.21.15.28:6789/0 conn(0x7f8fa4156830 legacy=0x7f8fa406aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:29.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.873+0000 7f8fa3e00640 1 --2- 172.21.15.28:0/4109875444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f8c0039b0 0x7f8f8c003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:29.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.873+0000 7f8fa3e00640 1 --2- 172.21.15.28:0/4109875444 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8f8c004800 0x7f8f8c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:29.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.873+0000 7f8fa3e00640 1 --2- 172.21.15.28:0/4109875444 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8f8c005210 0x7f8f8c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:29.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.873+0000 7f8fa3e00640 1 -- 172.21.15.28:0/4109875444 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa415a150 con 0x7f8f8c0039b0 2024-02-21T23:03:29.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.873+0000 7f8fa3e00640 1 -- 172.21.15.28:0/4109875444 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f8c00c140 con 0x7f8f8c004800 2024-02-21T23:03:29.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.873+0000 7f8fa3e00640 1 -- 172.21.15.28:0/4109875444 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f8c00c340 con 0x7f8f8c005210 2024-02-21T23:03:29.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.874+0000 7f8fa9a00640 1 --2- 172.21.15.28:0/4109875444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f8c0039b0 0x7f8f8c003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:29.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.874+0000 7f8fa9000640 1 --2- 172.21.15.28:0/4109875444 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8f8c004800 0x7f8f8c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:29.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.874+0000 7f8fa9a00640 1 -- 172.21.15.28:0/4109875444 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8f8c005210 msgr2=0x7f8f8c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:29.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.874+0000 7f8fa9a00640 1 --2- 172.21.15.28:0/4109875444 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8f8c005210 0x7f8f8c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:29.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.874+0000 7f8fa9a00640 1 -- 172.21.15.28:0/4109875444 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8f8c004800 msgr2=0x7f8f8c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:29.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.874+0000 7f8fa9a00640 1 --2- 172.21.15.28:0/4109875444 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8f8c004800 0x7f8f8c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:29.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.874+0000 7f8fa9a00640 1 -- 172.21.15.28:0/4109875444 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f8c003520 con 0x7f8f8c0039b0 2024-02-21T23:03:29.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.874+0000 7f8faa400640 1 -- 172.21.15.28:0/4109875444 reap_dead start 2024-02-21T23:03:29.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.875+0000 7f8fa9a00640 1 --2- 172.21.15.28:0/4109875444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f8c0039b0 0x7f8f8c003df0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f8f9800c240 tx=0x7f8f8c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a04733edc42a3cbf server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:29.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.876+0000 7f8fa3e00640 1 -- 172.21.15.28:0/4109875444 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f98007cf0 con 0x7f8f8c0039b0 2024-02-21T23:03:29.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.876+0000 7f8fa3e00640 1 -- 172.21.15.28:0/4109875444 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f98030070 con 0x7f8f8c0039b0 2024-02-21T23:03:29.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.876+0000 7f8fa3e00640 1 -- 172.21.15.28:0/4109875444 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f98005d70 con 0x7f8f8c0039b0 2024-02-21T23:03:29.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.876+0000 7f8faae00640 1 -- 172.21.15.28:0/4109875444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f8c0039b0 msgr2=0x7f8f8c003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:29.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.876+0000 7f8faae00640 1 --2- 172.21.15.28:0/4109875444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f8c0039b0 0x7f8f8c003df0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f8f9800c240 tx=0x7f8f8c005830 comp rx=0 tx=0).stop 2024-02-21T23:03:29.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.876+0000 7f8faae00640 1 -- 172.21.15.28:0/4109875444 shutdown_connections 2024-02-21T23:03:29.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.876+0000 7f8faae00640 1 --2- 172.21.15.28:0/4109875444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f8c0039b0 0x7f8f8c003df0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:29.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.876+0000 7f8faae00640 1 -- 172.21.15.28:0/4109875444 >> 172.21.15.28:0/4109875444 conn(0x7f8fa40c6bc0 msgr2=0x7f8fa40a82e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:29.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.877+0000 7f8faae00640 1 -- 172.21.15.28:0/4109875444 shutdown_connections 2024-02-21T23:03:29.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.877+0000 7f8faae00640 1 -- 172.21.15.28:0/4109875444 wait complete. 2024-02-21T23:03:29.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.879+0000 7f8faae00640 1 Processor -- start 2024-02-21T23:03:29.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.879+0000 7f8faae00640 1 -- start start 2024-02-21T23:03:29.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.880+0000 7f8faae00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8f8c0039b0 0x7f8fa41eff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:29.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.880+0000 7f8faae00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fa41e6530 0x7f8fa4153fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:29.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.880+0000 7f8faae00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8fa41544e0 0x7f8fa40aa420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:29.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.880+0000 7f8faae00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa41591f0 con 0x7f8fa41e6530 2024-02-21T23:03:29.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.880+0000 7f8faae00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa4159de0 con 0x7f8fa41544e0 2024-02-21T23:03:29.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.880+0000 7f8faae00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa40c07b0 con 0x7f8f8c0039b0 2024-02-21T23:03:29.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.880+0000 7f8fa9000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fa41e6530 0x7f8fa4153fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:29.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.880+0000 7f8fa9000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fa41e6530 0x7f8fa4153fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45106/0 (socket says 172.21.15.28:45106) 2024-02-21T23:03:29.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.880+0000 7f8fa9000640 1 -- 172.21.15.28:0/3722484013 learned_addr learned my addr 172.21.15.28:0/3722484013 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:29.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.880+0000 7f8faa400640 1 --2- 172.21.15.28:0/3722484013 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8fa41544e0 0x7f8fa40aa420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:29.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.880+0000 7f8fa9a00640 1 --2- 172.21.15.28:0/3722484013 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8f8c0039b0 0x7f8fa41eff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:29.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.881+0000 7f8fa9000640 1 -- 172.21.15.28:0/3722484013 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8f8c0039b0 msgr2=0x7f8fa41eff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:29.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.881+0000 7f8fa9000640 1 --2- 172.21.15.28:0/3722484013 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8f8c0039b0 0x7f8fa41eff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:29.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.881+0000 7f8fa9000640 1 -- 172.21.15.28:0/3722484013 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8fa41544e0 msgr2=0x7f8fa40aa420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:29.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.881+0000 7f8fa9000640 1 --2- 172.21.15.28:0/3722484013 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8fa41544e0 0x7f8fa40aa420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:29.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.881+0000 7f8fa9000640 1 -- 172.21.15.28:0/3722484013 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fa4072480 con 0x7f8fa41e6530 2024-02-21T23:03:29.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.881+0000 7f8fa9000640 1 --2- 172.21.15.28:0/3722484013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fa41e6530 0x7f8fa4153fa0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f8f94011980 tx=0x7f8f94011e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:29.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.881+0000 7f8fa1600640 1 -- 172.21.15.28:0/3722484013 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f94019070 con 0x7f8fa41e6530 2024-02-21T23:03:29.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.881+0000 7f8fa1600640 1 -- 172.21.15.28:0/3722484013 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f94004240 con 0x7f8fa41e6530 2024-02-21T23:03:29.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.881+0000 7f8faae00640 1 -- 172.21.15.28:0/3722484013 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fa41ef430 con 0x7f8fa41e6530 2024-02-21T23:03:29.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.881+0000 7f8fa1600640 1 -- 172.21.15.28:0/3722484013 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f9400f8e0 con 0x7f8fa41e6530 2024-02-21T23:03:29.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.882+0000 7f8faae00640 1 -- 172.21.15.28:0/3722484013 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fa41ef9f0 con 0x7f8fa41e6530 2024-02-21T23:03:29.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.883+0000 7f8faae00640 1 -- 172.21.15.28:0/3722484013 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fa406bec0 con 0x7f8fa41e6530 2024-02-21T23:03:29.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.884+0000 7f8fa1600640 1 -- 172.21.15.28:0/3722484013 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8f9401c070 con 0x7f8fa41e6530 2024-02-21T23:03:29.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.885+0000 7f8fa1600640 1 --2- 172.21.15.28:0/3722484013 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8f8806b4d0 0x7f8f8806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:29.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.885+0000 7f8fa1600640 1 -- 172.21.15.28:0/3722484013 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f8f9408aac0 con 0x7f8fa41e6530 2024-02-21T23:03:29.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.885+0000 7f8fa9a00640 1 --2- 172.21.15.28:0/3722484013 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8f8806b4d0 0x7f8f8806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:29.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.886+0000 7f8fa9a00640 1 --2- 172.21.15.28:0/3722484013 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8f8806b4d0 0x7f8f8806d990 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f8f98034000 tx=0x7f8f98007000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:29.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:29.889+0000 7f8fa1600640 1 -- 172.21.15.28:0/3722484013 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f8fa406bec0 con 0x7f8fa41e6530 2024-02-21T23:03:30.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.092+0000 7f8faae00640 1 -- 172.21.15.28:0/3722484013 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mgr/crash/warn_recent_interval}] v 0) v1 -- 0x7f8fa404f7b0 con 0x7f8fa41e6530 2024-02-21T23:03:30.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.098+0000 7f8fa1600640 1 -- 172.21.15.28:0/3722484013 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/crash/warn_recent_interval}]=0 v10)=0 v10) v1 ==== 128+0+0 (secure 0 0 0) 0x7f8f94059530 con 0x7f8fa41e6530 2024-02-21T23:03:30.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.101+0000 7f8faae00640 1 -- 172.21.15.28:0/3722484013 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8f8806b4d0 msgr2=0x7f8f8806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:30.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.101+0000 7f8faae00640 1 --2- 172.21.15.28:0/3722484013 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8f8806b4d0 0x7f8f8806d990 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f8f98034000 tx=0x7f8f98007000 comp rx=0 tx=0).stop 2024-02-21T23:03:30.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.101+0000 7f8faae00640 1 -- 172.21.15.28:0/3722484013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fa41e6530 msgr2=0x7f8fa4153fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:30.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.101+0000 7f8faae00640 1 --2- 172.21.15.28:0/3722484013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fa41e6530 0x7f8fa4153fa0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f8f94011980 tx=0x7f8f94011e40 comp rx=0 tx=0).stop 2024-02-21T23:03:30.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.101+0000 7f8faae00640 1 -- 172.21.15.28:0/3722484013 shutdown_connections 2024-02-21T23:03:30.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.101+0000 7f8faae00640 1 --2- 172.21.15.28:0/3722484013 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8f8806b4d0 0x7f8f8806d990 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:30.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.101+0000 7f8faae00640 1 --2- 172.21.15.28:0/3722484013 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8fa41544e0 0x7f8fa40aa420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:30.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.101+0000 7f8faae00640 1 --2- 172.21.15.28:0/3722484013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fa41e6530 0x7f8fa4153fa0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:30.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.101+0000 7f8faae00640 1 --2- 172.21.15.28:0/3722484013 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8f8c0039b0 0x7f8fa41eff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:30.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.101+0000 7f8faae00640 1 -- 172.21.15.28:0/3722484013 >> 172.21.15.28:0/3722484013 conn(0x7f8fa40c6bc0 msgr2=0x7f8fa4158480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:30.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.101+0000 7f8faae00640 1 -- 172.21.15.28:0/3722484013 shutdown_connections 2024-02-21T23:03:30.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.101+0000 7f8faae00640 1 -- 172.21.15.28:0/3722484013 wait complete. 2024-02-21T23:03:30.111 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-02-21T23:03:30.126 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:03:30.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.362+0000 7fa941200640 1 Processor -- start 2024-02-21T23:03:30.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.362+0000 7fa941200640 1 -- start start 2024-02-21T23:03:30.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.364+0000 7fa941200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa93c0c3490 0x7fa93c06dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:30.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.364+0000 7fa941200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa93c06e750 con 0x7fa93c0a5f40 2024-02-21T23:03:30.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.364+0000 7fa941200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa93c06ea30 con 0x7fa93c0c0270 2024-02-21T23:03:30.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.364+0000 7fa941200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa93c06eff0 con 0x7fa93c0c3490 2024-02-21T23:03:30.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.364+0000 7fa933e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa93c0a5f40 0x7fa93c152140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59382/0 (socket says 172.21.15.28:59382) 2024-02-21T23:03:30.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.364+0000 7fa933e00640 1 -- 172.21.15.28:0/1594068920 learned_addr learned my addr 172.21.15.28:0/1594068920 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:03:30.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.365+0000 7fa939e00640 1 --2- 172.21.15.28:0/1594068920 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa93c0c3490 0x7fa93c06dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:30.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.365+0000 7fa939400640 1 -- 172.21.15.28:0/1594068920 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 992355853 0 0) 0x7fa93c06e750 con 0x7fa93c0a5f40 2024-02-21T23:03:30.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.365+0000 7fa939400640 1 -- 172.21.15.28:0/1594068920 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa920003520 con 0x7fa93c0a5f40 2024-02-21T23:03:30.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.365+0000 7fa939400640 1 -- 172.21.15.28:0/1594068920 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1245909071 0 0) 0x7fa920003520 con 0x7fa93c0a5f40 2024-02-21T23:03:30.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.365+0000 7fa939400640 1 -- 172.21.15.28:0/1594068920 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa93c0c3490 msgr2=0x7fa93c06dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:30.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.365+0000 7fa939400640 1 --2- 172.21.15.28:0/1594068920 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa93c0c3490 0x7fa93c06dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:30.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.365+0000 7fa939400640 1 -- 172.21.15.28:0/1594068920 >> v1:172.21.15.81:6789/0 conn(0x7fa93c0c0270 legacy=0x7fa93c0c0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:30.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.365+0000 7fa939400640 1 -- 172.21.15.28:0/1594068920 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa93c06ed10 con 0x7fa93c0a5f40 2024-02-21T23:03:30.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.365+0000 7fa939400640 1 -- 172.21.15.28:0/1594068920 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fa924001670 con 0x7fa93c0a5f40 2024-02-21T23:03:30.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.365+0000 7fa939400640 1 -- 172.21.15.28:0/1594068920 >> v1:172.21.15.28:6789/0 conn(0x7fa93c0a5f40 legacy=0x7fa93c152140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:30.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.365+0000 7fa939400640 1 --2- 172.21.15.28:0/1594068920 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa9200039e0 0x7fa920003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:30.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.365+0000 7fa939400640 1 --2- 172.21.15.28:0/1594068920 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa93c0c3490 0x7fa920004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:30.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.365+0000 7fa939400640 1 --2- 172.21.15.28:0/1594068920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa920004d40 0x7fa9200051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:30.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.365+0000 7fa939400640 1 -- 172.21.15.28:0/1594068920 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa93c06eff0 con 0x7fa920004d40 2024-02-21T23:03:30.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.365+0000 7fa939400640 1 -- 172.21.15.28:0/1594068920 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa92000bb50 con 0x7fa9200039e0 2024-02-21T23:03:30.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.365+0000 7fa939400640 1 -- 172.21.15.28:0/1594068920 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa92000bd50 con 0x7fa93c0c3490 2024-02-21T23:03:30.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.366+0000 7fa93a800640 1 --2- 172.21.15.28:0/1594068920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa920004d40 0x7fa9200051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:30.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.366+0000 7fa939e00640 1 --2- 172.21.15.28:0/1594068920 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa9200039e0 0x7fa920003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:30.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.366+0000 7fa933e00640 1 --2- 172.21.15.28:0/1594068920 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa93c0c3490 0x7fa920004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:30.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.366+0000 7fa93a800640 1 -- 172.21.15.28:0/1594068920 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa93c0c3490 msgr2=0x7fa920004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:30.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.366+0000 7fa93a800640 1 --2- 172.21.15.28:0/1594068920 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa93c0c3490 0x7fa920004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:30.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.366+0000 7fa93a800640 1 -- 172.21.15.28:0/1594068920 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa9200039e0 msgr2=0x7fa920003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:30.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.366+0000 7fa93a800640 1 --2- 172.21.15.28:0/1594068920 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa9200039e0 0x7fa920003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:30.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.366+0000 7fa93a800640 1 -- 172.21.15.28:0/1594068920 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa920003520 con 0x7fa920004d40 2024-02-21T23:03:30.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.368+0000 7fa93a800640 1 --2- 172.21.15.28:0/1594068920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa920004d40 0x7fa9200051c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fa934001e50 tx=0x7fa920005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af99605b3ea72c9d server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:30.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.368+0000 7fa939400640 1 -- 172.21.15.28:0/1594068920 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa934001a70 con 0x7fa920004d40 2024-02-21T23:03:30.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.368+0000 7fa939400640 1 -- 172.21.15.28:0/1594068920 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa934003b70 con 0x7fa920004d40 2024-02-21T23:03:30.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.368+0000 7fa939400640 1 -- 172.21.15.28:0/1594068920 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9340050d0 con 0x7fa920004d40 2024-02-21T23:03:30.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.368+0000 7fa941200640 1 -- 172.21.15.28:0/1594068920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa920004d40 msgr2=0x7fa9200051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:30.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.368+0000 7fa941200640 1 --2- 172.21.15.28:0/1594068920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa920004d40 0x7fa9200051c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fa934001e50 tx=0x7fa920005830 comp rx=0 tx=0).stop 2024-02-21T23:03:30.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.368+0000 7fa93a800640 1 -- 172.21.15.28:0/1594068920 reap_dead start 2024-02-21T23:03:30.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.368+0000 7fa941200640 1 -- 172.21.15.28:0/1594068920 shutdown_connections 2024-02-21T23:03:30.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.368+0000 7fa941200640 1 -- 172.21.15.28:0/1594068920 >> 172.21.15.28:0/1594068920 conn(0x7fa93c0c67f0 msgr2=0x7fa93c0c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:30.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.369+0000 7fa941200640 1 -- 172.21.15.28:0/1594068920 shutdown_connections 2024-02-21T23:03:30.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.369+0000 7fa941200640 1 -- 172.21.15.28:0/1594068920 wait complete. 2024-02-21T23:03:30.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.370+0000 7fa941200640 1 Processor -- start 2024-02-21T23:03:30.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.370+0000 7fa941200640 1 -- start start 2024-02-21T23:03:30.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.371+0000 7fa941200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa920004d40 0x7fa93c0ac230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:30.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.371+0000 7fa941200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa93c0a6400 0x7fa93c147560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:30.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.371+0000 7fa941200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa93c0a8de0 0x7fa93c14e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:30.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.371+0000 7fa941200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa93c152850 con 0x7fa93c0a6400 2024-02-21T23:03:30.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.371+0000 7fa941200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa93c0bfde0 con 0x7fa93c0a8de0 2024-02-21T23:03:30.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.371+0000 7fa941200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa93c1529d0 con 0x7fa920004d40 2024-02-21T23:03:30.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.372+0000 7fa933e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa93c0a6400 0x7fa93c147560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:30.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.372+0000 7fa933e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa93c0a6400 0x7fa93c147560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45124/0 (socket says 172.21.15.28:45124) 2024-02-21T23:03:30.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.372+0000 7fa933e00640 1 -- 172.21.15.28:0/781805851 learned_addr learned my addr 172.21.15.28:0/781805851 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:30.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.372+0000 7fa939e00640 1 --2- 172.21.15.28:0/781805851 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa920004d40 0x7fa93c0ac230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:30.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.372+0000 7fa93a800640 1 --2- 172.21.15.28:0/781805851 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa93c0a8de0 0x7fa93c14e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:30.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.372+0000 7fa933e00640 1 -- 172.21.15.28:0/781805851 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa920004d40 msgr2=0x7fa93c0ac230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:30.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.372+0000 7fa933e00640 1 --2- 172.21.15.28:0/781805851 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa920004d40 0x7fa93c0ac230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:30.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.372+0000 7fa933e00640 1 -- 172.21.15.28:0/781805851 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa93c0a8de0 msgr2=0x7fa93c14e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:30.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.372+0000 7fa933e00640 1 --2- 172.21.15.28:0/781805851 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa93c0a8de0 0x7fa93c14e1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:30.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.372+0000 7fa933e00640 1 -- 172.21.15.28:0/781805851 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa93c1eb750 con 0x7fa93c0a6400 2024-02-21T23:03:30.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.372+0000 7fa939e00640 1 --2- 172.21.15.28:0/781805851 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa920004d40 0x7fa93c0ac230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:03:30.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.372+0000 7fa933e00640 1 --2- 172.21.15.28:0/781805851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa93c0a6400 0x7fa93c147560 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fa924007310 tx=0x7fa9240077d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:30.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.373+0000 7fa932000640 1 -- 172.21.15.28:0/781805851 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa92401a070 con 0x7fa93c0a6400 2024-02-21T23:03:30.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.373+0000 7fa932000640 1 -- 172.21.15.28:0/781805851 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa92400f8e0 con 0x7fa93c0a6400 2024-02-21T23:03:30.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.373+0000 7fa932000640 1 -- 172.21.15.28:0/781805851 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa92400fcd0 con 0x7fa93c0a6400 2024-02-21T23:03:30.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.373+0000 7fa941200640 1 -- 172.21.15.28:0/781805851 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa93c0a68f0 con 0x7fa93c0a6400 2024-02-21T23:03:30.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.373+0000 7fa941200640 1 -- 172.21.15.28:0/781805851 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa93c0a7c90 con 0x7fa93c0a6400 2024-02-21T23:03:30.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.374+0000 7fa941200640 1 -- 172.21.15.28:0/781805851 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa93c067f30 con 0x7fa93c0a6400 2024-02-21T23:03:30.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.375+0000 7fa932000640 1 -- 172.21.15.28:0/781805851 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa92401d360 con 0x7fa93c0a6400 2024-02-21T23:03:30.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.376+0000 7fa932000640 1 --2- 172.21.15.28:0/781805851 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa91806b500 0x7fa91806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:30.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.376+0000 7fa932000640 1 -- 172.21.15.28:0/781805851 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6735+0+0 (secure 0 0 0) 0x7fa924089e80 con 0x7fa93c0a6400 2024-02-21T23:03:30.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.377+0000 7fa939e00640 1 --2- 172.21.15.28:0/781805851 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa91806b500 0x7fa91806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:30.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.378+0000 7fa939e00640 1 --2- 172.21.15.28:0/781805851 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa91806b500 0x7fa91806d9c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fa92c009ab0 tx=0x7fa92c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:30.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.383+0000 7fa932000640 1 -- 172.21.15.28:0/781805851 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fa9240588f0 con 0x7fa93c0a6400 2024-02-21T23:03:30.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.595+0000 7fa941200640 1 -- 172.21.15.28:0/781805851 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa93c06dd00 con 0x7fa93c0a6400 2024-02-21T23:03:30.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.597+0000 7fa932000640 1 -- 172.21.15.28:0/781805851 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+18843 (secure 0 0 0) 0x7fa9240581f0 con 0x7fa93c0a6400 2024-02-21T23:03:30.586 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:03:30.586 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":26,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:03:23.860109+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:02:35.795158+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-21T23:02:36.840110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-02-21T23:02:40.441388+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":3,"to":2}]},{"pgid":"3.12","mappings":[{"from":0,"to":1}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:03:30.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.599+0000 7fa941200640 1 -- 172.21.15.28:0/781805851 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa91806b500 msgr2=0x7fa91806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:30.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.599+0000 7fa941200640 1 --2- 172.21.15.28:0/781805851 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa91806b500 0x7fa91806d9c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fa92c009ab0 tx=0x7fa92c008040 comp rx=0 tx=0).stop 2024-02-21T23:03:30.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.599+0000 7fa941200640 1 -- 172.21.15.28:0/781805851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa93c0a6400 msgr2=0x7fa93c147560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:30.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.599+0000 7fa941200640 1 --2- 172.21.15.28:0/781805851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa93c0a6400 0x7fa93c147560 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fa924007310 tx=0x7fa9240077d0 comp rx=0 tx=0).stop 2024-02-21T23:03:30.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.600+0000 7fa941200640 1 -- 172.21.15.28:0/781805851 shutdown_connections 2024-02-21T23:03:30.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.600+0000 7fa941200640 1 --2- 172.21.15.28:0/781805851 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa91806b500 0x7fa91806d9c0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:30.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.600+0000 7fa941200640 1 --2- 172.21.15.28:0/781805851 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa93c0a8de0 0x7fa93c14e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:30.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.600+0000 7fa941200640 1 --2- 172.21.15.28:0/781805851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa93c0a6400 0x7fa93c147560 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:30.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.600+0000 7fa941200640 1 --2- 172.21.15.28:0/781805851 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa920004d40 0x7fa93c0ac230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:30.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.600+0000 7fa941200640 1 -- 172.21.15.28:0/781805851 >> 172.21.15.28:0/781805851 conn(0x7fa93c0c67f0 msgr2=0x7fa93c151080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:30.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.600+0000 7fa941200640 1 -- 172.21.15.28:0/781805851 shutdown_connections 2024-02-21T23:03:30.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.600+0000 7fa941200640 1 -- 172.21.15.28:0/781805851 wait complete. 2024-02-21T23:03:30.607 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-21T23:02:30.728329+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-21T23:02:35.795158+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-02-21T23:02:36.840110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-02-21T23:02:40.441388+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-02-21T23:03:30.608 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-21T23:03:30.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.846+0000 7ff76b200640 1 Processor -- start 2024-02-21T23:03:30.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.847+0000 7ff76b200640 1 -- start start 2024-02-21T23:03:30.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.848+0000 7ff76b200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff7640c22e0 0x7ff76406d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:30.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.848+0000 7ff76b200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff7641561a0 0x7ff76406d660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:30.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.848+0000 7ff76b200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff764071970 con 0x7ff7640c5a70 2024-02-21T23:03:30.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.848+0000 7ff76b200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff764072a10 con 0x7ff7640c22e0 2024-02-21T23:03:30.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.848+0000 7ff76b200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff764072580 con 0x7ff7641561a0 2024-02-21T23:03:30.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.849+0000 7ff763e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff7640c5a70 0x7ff764155a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59390/0 (socket says 172.21.15.28:59390) 2024-02-21T23:03:30.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.849+0000 7ff763e00640 1 -- 172.21.15.28:0/2269985825 learned_addr learned my addr 172.21.15.28:0/2269985825 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:03:30.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.849+0000 7ff763400640 1 --2- 172.21.15.28:0/2269985825 >> v2:172.21.15.28:3300/0 conn(0x7ff7640c22e0 0x7ff76406d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:30.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.849+0000 7ff769200640 1 --2- 172.21.15.28:0/2269985825 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff7641561a0 0x7ff76406d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:30.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.849+0000 7ff763400640 1 -- 172.21.15.28:0/2269985825 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff7641561a0 msgr2=0x7ff76406d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:30.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.849+0000 7ff763400640 1 --2- 172.21.15.28:0/2269985825 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff7641561a0 0x7ff76406d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:30.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.849+0000 7ff763400640 1 -- 172.21.15.28:0/2269985825 >> v1:172.21.15.28:6789/0 conn(0x7ff7640c5a70 legacy=0x7ff764155a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:30.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.849+0000 7ff763400640 1 -- 172.21.15.28:0/2269985825 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff764071c50 con 0x7ff7640c22e0 2024-02-21T23:03:30.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.850+0000 7ff763400640 1 --2- 172.21.15.28:0/2269985825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7640c22e0 0x7ff76406d120 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff7500097a0 tx=0x7ff75002b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ce40a926b555b51 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:30.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.851+0000 7ff762a00640 1 -- 172.21.15.28:0/2269985825 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff750031780 con 0x7ff7640c22e0 2024-02-21T23:03:30.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.851+0000 7ff762a00640 1 -- 172.21.15.28:0/2269985825 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff750037070 con 0x7ff7640c22e0 2024-02-21T23:03:30.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.851+0000 7ff762a00640 1 -- 172.21.15.28:0/2269985825 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff750031b40 con 0x7ff7640c22e0 2024-02-21T23:03:30.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.851+0000 7ff76b200640 1 -- 172.21.15.28:0/2269985825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7640c22e0 msgr2=0x7ff76406d120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:30.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.851+0000 7ff76b200640 1 --2- 172.21.15.28:0/2269985825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7640c22e0 0x7ff76406d120 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff7500097a0 tx=0x7ff75002b820 comp rx=0 tx=0).stop 2024-02-21T23:03:30.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.851+0000 7ff76b200640 1 -- 172.21.15.28:0/2269985825 shutdown_connections 2024-02-21T23:03:30.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.851+0000 7ff76b200640 1 --2- 172.21.15.28:0/2269985825 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff7641561a0 0x7ff76406d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:30.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.851+0000 7ff76b200640 1 --2- 172.21.15.28:0/2269985825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7640c22e0 0x7ff76406d120 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:30.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.851+0000 7ff76b200640 1 -- 172.21.15.28:0/2269985825 >> 172.21.15.28:0/2269985825 conn(0x7ff7640a85a0 msgr2=0x7ff7640a8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:30.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.852+0000 7ff76b200640 1 -- 172.21.15.28:0/2269985825 shutdown_connections 2024-02-21T23:03:30.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.852+0000 7ff76b200640 1 -- 172.21.15.28:0/2269985825 wait complete. 2024-02-21T23:03:30.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.853+0000 7ff76b200640 1 Processor -- start 2024-02-21T23:03:30.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.853+0000 7ff76b200640 1 -- start start 2024-02-21T23:03:30.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.854+0000 7ff76b200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff7640c22e0 0x7ff764151dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:30.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.854+0000 7ff76b200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff7640c5a70 0x7ff76414d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:30.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.854+0000 7ff76b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7641561a0 0x7ff76414bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:30.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.854+0000 7ff76b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff76406dba0 con 0x7ff7641561a0 2024-02-21T23:03:30.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.854+0000 7ff76b200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff764072210 con 0x7ff7640c22e0 2024-02-21T23:03:30.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.854+0000 7ff76b200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7640a6160 con 0x7ff7640c5a70 2024-02-21T23:03:30.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.855+0000 7ff769200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7641561a0 0x7ff76414bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:30.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.855+0000 7ff769200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7641561a0 0x7ff76414bdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45128/0 (socket says 172.21.15.28:45128) 2024-02-21T23:03:30.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.855+0000 7ff769200640 1 -- 172.21.15.28:0/1336511019 learned_addr learned my addr 172.21.15.28:0/1336511019 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:30.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.855+0000 7ff763e00640 1 --2- 172.21.15.28:0/1336511019 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff7640c22e0 0x7ff764151dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:30.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.855+0000 7ff763400640 1 --2- 172.21.15.28:0/1336511019 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff7640c5a70 0x7ff76414d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:30.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.855+0000 7ff769200640 1 -- 172.21.15.28:0/1336511019 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff7640c5a70 msgr2=0x7ff76414d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:30.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.855+0000 7ff769200640 1 --2- 172.21.15.28:0/1336511019 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff7640c5a70 0x7ff76414d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:30.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.855+0000 7ff769200640 1 -- 172.21.15.28:0/1336511019 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff7640c22e0 msgr2=0x7ff764151dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:30.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.855+0000 7ff769200640 1 --2- 172.21.15.28:0/1336511019 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff7640c22e0 0x7ff764151dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:30.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.855+0000 7ff769200640 1 -- 172.21.15.28:0/1336511019 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7641e9880 con 0x7ff7641561a0 2024-02-21T23:03:30.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.855+0000 7ff763400640 1 --2- 172.21.15.28:0/1336511019 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff7640c5a70 0x7ff76414d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:03:30.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.856+0000 7ff769200640 1 --2- 172.21.15.28:0/1336511019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7641561a0 0x7ff76414bdd0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7ff75c007e40 tx=0x7ff75c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:30.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.856+0000 7ff75be00640 1 -- 172.21.15.28:0/1336511019 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff75c014620 con 0x7ff7641561a0 2024-02-21T23:03:30.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.856+0000 7ff75be00640 1 -- 172.21.15.28:0/1336511019 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff75c019070 con 0x7ff7641561a0 2024-02-21T23:03:30.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.856+0000 7ff75be00640 1 -- 172.21.15.28:0/1336511019 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff75c014a20 con 0x7ff7641561a0 2024-02-21T23:03:30.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.856+0000 7ff76b200640 1 -- 172.21.15.28:0/1336511019 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7641e9b60 con 0x7ff7641561a0 2024-02-21T23:03:30.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.856+0000 7ff76b200640 1 -- 172.21.15.28:0/1336511019 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff76414cc60 con 0x7ff7641561a0 2024-02-21T23:03:30.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.857+0000 7ff76b200640 1 -- 172.21.15.28:0/1336511019 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff764067f30 con 0x7ff7641561a0 2024-02-21T23:03:30.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.858+0000 7ff75be00640 1 -- 172.21.15.28:0/1336511019 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff75c014bc0 con 0x7ff7641561a0 2024-02-21T23:03:30.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.859+0000 7ff75be00640 1 --2- 172.21.15.28:0/1336511019 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff74406b4d0 0x7ff74406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:30.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.859+0000 7ff75be00640 1 -- 172.21.15.28:0/1336511019 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6735+0+0 (secure 0 0 0) 0x7ff75c08af50 con 0x7ff7641561a0 2024-02-21T23:03:30.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.860+0000 7ff763e00640 1 --2- 172.21.15.28:0/1336511019 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff74406b4d0 0x7ff74406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:30.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.861+0000 7ff763e00640 1 --2- 172.21.15.28:0/1336511019 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff74406b4d0 0x7ff74406d990 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ff754000f60 tx=0x7ff754009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:30.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:30.864+0000 7ff75be00640 1 -- 172.21.15.28:0/1336511019 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ff75c0599c0 con 0x7ff7641561a0 2024-02-21T23:03:31.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.066+0000 7ff76b200640 1 -- 172.21.15.28:0/1336511019 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff7640a5b10 con 0x7ff7641561a0 2024-02-21T23:03:31.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.067+0000 7ff75be00640 1 -- 172.21.15.28:0/1336511019 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff75c0592c0 con 0x7ff7641561a0 2024-02-21T23:03:31.056 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-02-21T23:03:31.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.069+0000 7ff76b200640 1 -- 172.21.15.28:0/1336511019 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff74406b4d0 msgr2=0x7ff74406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:31.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.069+0000 7ff76b200640 1 --2- 172.21.15.28:0/1336511019 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff74406b4d0 0x7ff74406d990 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ff754000f60 tx=0x7ff754009040 comp rx=0 tx=0).stop 2024-02-21T23:03:31.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.069+0000 7ff76b200640 1 -- 172.21.15.28:0/1336511019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7641561a0 msgr2=0x7ff76414bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:31.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.069+0000 7ff76b200640 1 --2- 172.21.15.28:0/1336511019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7641561a0 0x7ff76414bdd0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7ff75c007e40 tx=0x7ff75c00e4b0 comp rx=0 tx=0).stop 2024-02-21T23:03:31.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.070+0000 7ff76b200640 1 -- 172.21.15.28:0/1336511019 shutdown_connections 2024-02-21T23:03:31.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.070+0000 7ff76b200640 1 --2- 172.21.15.28:0/1336511019 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff74406b4d0 0x7ff74406d990 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:31.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.070+0000 7ff76b200640 1 --2- 172.21.15.28:0/1336511019 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff7640c5a70 0x7ff76414d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:31.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.070+0000 7ff76b200640 1 --2- 172.21.15.28:0/1336511019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7641561a0 0x7ff76414bdd0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:31.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.070+0000 7ff76b200640 1 --2- 172.21.15.28:0/1336511019 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff7640c22e0 0x7ff764151dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:31.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.070+0000 7ff76b200640 1 -- 172.21.15.28:0/1336511019 >> 172.21.15.28:0/1336511019 conn(0x7ff7640a85a0 msgr2=0x7ff76406dd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:31.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.070+0000 7ff76b200640 1 -- 172.21.15.28:0/1336511019 shutdown_connections 2024-02-21T23:03:31.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.070+0000 7ff76b200640 1 -- 172.21.15.28:0/1336511019 wait complete. 2024-02-21T23:03:31.079 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-21T23:03:31.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.296+0000 7f7c6d600640 1 Processor -- start 2024-02-21T23:03:31.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.297+0000 7f7c6d600640 1 -- start start 2024-02-21T23:03:31.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.298+0000 7f7c6d600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7c68158480 0x7f7c68154f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:31.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.298+0000 7f7c6d600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f7c6806e9d0 0x7f7c6806ee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:31.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.298+0000 7f7c6d600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c6806f3a0 con 0x7f7c68152560 2024-02-21T23:03:31.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.298+0000 7f7c6d600640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c681554a0 con 0x7f7c6806e9d0 2024-02-21T23:03:31.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.298+0000 7f7c6d600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c680a6270 con 0x7f7c68158480 2024-02-21T23:03:31.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.299+0000 7f7c6cc00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f7c6806e9d0 0x7f7c6806ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:31.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.299+0000 7f7c67400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7c68158480 0x7f7c68154f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:31.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.299+0000 7f7c6cc00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f7c6806e9d0 0x7f7c6806ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.28:36252/0 (socket says 172.21.15.28:36252) 2024-02-21T23:03:31.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.299+0000 7f7c6cc00640 1 -- 172.21.15.28:0/3603008806 learned_addr learned my addr 172.21.15.28:0/3603008806 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:31.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.300+0000 7f7c6cc00640 1 -- 172.21.15.28:0/3603008806 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7c68158480 msgr2=0x7f7c68154f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:31.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.300+0000 7f7c6cc00640 1 --2- 172.21.15.28:0/3603008806 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7c68158480 0x7f7c68154f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:31.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.300+0000 7f7c6cc00640 1 -- 172.21.15.28:0/3603008806 >> v1:172.21.15.81:6789/0 conn(0x7f7c68152560 legacy=0x7f7c6806ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:31.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.300+0000 7f7c6cc00640 1 -- 172.21.15.28:0/3603008806 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c681556d0 con 0x7f7c6806e9d0 2024-02-21T23:03:31.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.301+0000 7f7c6cc00640 1 --2- 172.21.15.28:0/3603008806 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7c6806e9d0 0x7f7c6806ee60 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7c60009e50 tx=0x7f7c6002bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a9a9bed767bc2c04 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:31.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.301+0000 7f7c66a00640 1 -- 172.21.15.28:0/3603008806 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c6002cb00 con 0x7f7c6806e9d0 2024-02-21T23:03:31.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.301+0000 7f7c66a00640 1 -- 172.21.15.28:0/3603008806 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c6002cca0 con 0x7f7c6806e9d0 2024-02-21T23:03:31.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.302+0000 7f7c66a00640 1 -- 172.21.15.28:0/3603008806 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c60034610 con 0x7f7c6806e9d0 2024-02-21T23:03:31.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.302+0000 7f7c6d600640 1 -- 172.21.15.28:0/3603008806 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7c6806e9d0 msgr2=0x7f7c6806ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:31.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.302+0000 7f7c6d600640 1 --2- 172.21.15.28:0/3603008806 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7c6806e9d0 0x7f7c6806ee60 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7c60009e50 tx=0x7f7c6002bf60 comp rx=0 tx=0).stop 2024-02-21T23:03:31.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.302+0000 7f7c6d600640 1 -- 172.21.15.28:0/3603008806 shutdown_connections 2024-02-21T23:03:31.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.302+0000 7f7c6d600640 1 --2- 172.21.15.28:0/3603008806 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7c6806e9d0 0x7f7c6806ee60 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:31.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.302+0000 7f7c6d600640 1 --2- 172.21.15.28:0/3603008806 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7c68158480 0x7f7c68154f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:31.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.302+0000 7f7c6d600640 1 -- 172.21.15.28:0/3603008806 >> 172.21.15.28:0/3603008806 conn(0x7f7c680a8530 msgr2=0x7f7c680a8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:31.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.302+0000 7f7c6d600640 1 -- 172.21.15.28:0/3603008806 shutdown_connections 2024-02-21T23:03:31.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.302+0000 7f7c6d600640 1 -- 172.21.15.28:0/3603008806 wait complete. 2024-02-21T23:03:31.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.305+0000 7f7c6d600640 1 Processor -- start 2024-02-21T23:03:31.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.305+0000 7f7c6d600640 1 -- start start 2024-02-21T23:03:31.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.306+0000 7f7c6d600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7c6806e9d0 0x7f7c681eaac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:31.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.306+0000 7f7c6d600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7c68152560 0x7f7c68149b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:31.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.306+0000 7f7c6d600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c68158480 0x7f7c68157bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:31.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.306+0000 7f7c6d600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c6806e4b0 con 0x7f7c68158480 2024-02-21T23:03:31.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.306+0000 7f7c6d600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c68155c90 con 0x7f7c6806e9d0 2024-02-21T23:03:31.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.306+0000 7f7c6d600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c68152940 con 0x7f7c68152560 2024-02-21T23:03:31.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.306+0000 7f7c6cc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c68158480 0x7f7c68157bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:31.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.306+0000 7f7c6cc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c68158480 0x7f7c68157bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45130/0 (socket says 172.21.15.28:45130) 2024-02-21T23:03:31.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.306+0000 7f7c6cc00640 1 -- 172.21.15.28:0/1421436825 learned_addr learned my addr 172.21.15.28:0/1421436825 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:31.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.306+0000 7f7c67e00640 1 --2- 172.21.15.28:0/1421436825 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7c6806e9d0 0x7f7c681eaac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:31.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.307+0000 7f7c67400640 1 --2- 172.21.15.28:0/1421436825 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7c68152560 0x7f7c68149b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:31.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.307+0000 7f7c6cc00640 1 -- 172.21.15.28:0/1421436825 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7c68152560 msgr2=0x7f7c68149b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:31.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.307+0000 7f7c6cc00640 1 --2- 172.21.15.28:0/1421436825 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7c68152560 0x7f7c68149b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:31.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.307+0000 7f7c6cc00640 1 -- 172.21.15.28:0/1421436825 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7c6806e9d0 msgr2=0x7f7c681eaac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:31.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.307+0000 7f7c6cc00640 1 --2- 172.21.15.28:0/1421436825 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7c6806e9d0 0x7f7c681eaac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:31.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.307+0000 7f7c6cc00640 1 -- 172.21.15.28:0/1421436825 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c681eec90 con 0x7f7c68158480 2024-02-21T23:03:31.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.307+0000 7f7c6cc00640 1 --2- 172.21.15.28:0/1421436825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c68158480 0x7f7c68157bb0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f7c60036040 tx=0x7f7c60004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:31.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.308+0000 7f7c5fe00640 1 -- 172.21.15.28:0/1421436825 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c60043020 con 0x7f7c68158480 2024-02-21T23:03:31.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.308+0000 7f7c5fe00640 1 -- 172.21.15.28:0/1421436825 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c6002ce50 con 0x7f7c68158480 2024-02-21T23:03:31.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.308+0000 7f7c5fe00640 1 -- 172.21.15.28:0/1421436825 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c60009510 con 0x7f7c68158480 2024-02-21T23:03:31.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.308+0000 7f7c6d600640 1 -- 172.21.15.28:0/1421436825 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c68156b00 con 0x7f7c68158480 2024-02-21T23:03:31.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.308+0000 7f7c6d600640 1 -- 172.21.15.28:0/1421436825 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c681570c0 con 0x7f7c68158480 2024-02-21T23:03:31.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.309+0000 7f7c6d600640 1 -- 172.21.15.28:0/1421436825 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c6804f290 con 0x7f7c68158480 2024-02-21T23:03:31.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.310+0000 7f7c5fe00640 1 -- 172.21.15.28:0/1421436825 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7c60046020 con 0x7f7c68158480 2024-02-21T23:03:31.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.311+0000 7f7c5fe00640 1 --2- 172.21.15.28:0/1421436825 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7c4406b4d0 0x7f7c4406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:31.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.311+0000 7f7c5fe00640 1 -- 172.21.15.28:0/1421436825 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f7c600acb90 con 0x7f7c68158480 2024-02-21T23:03:31.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.311+0000 7f7c67e00640 1 --2- 172.21.15.28:0/1421436825 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7c4406b4d0 0x7f7c4406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:31.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.313+0000 7f7c67e00640 1 --2- 172.21.15.28:0/1421436825 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7c4406b4d0 0x7f7c4406d990 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7c58000f60 tx=0x7f7c58009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:31.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.315+0000 7f7c5fe00640 1 -- 172.21.15.28:0/1421436825 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7c6004d030 con 0x7f7c68158480 2024-02-21T23:03:31.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.523+0000 7f7c6d600640 1 -- 172.21.15.28:0/1421436825 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7c6806e0f0 con 0x7f7c68158480 2024-02-21T23:03:31.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.524+0000 7f7c5fe00640 1 -- 172.21.15.28:0/1421436825 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7c6007b600 con 0x7f7c68158480 2024-02-21T23:03:31.513 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-02-21T23:03:31.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.527+0000 7f7c6d600640 1 -- 172.21.15.28:0/1421436825 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7c4406b4d0 msgr2=0x7f7c4406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:31.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.527+0000 7f7c6d600640 1 --2- 172.21.15.28:0/1421436825 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7c4406b4d0 0x7f7c4406d990 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7c58000f60 tx=0x7f7c58009040 comp rx=0 tx=0).stop 2024-02-21T23:03:31.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.527+0000 7f7c6d600640 1 -- 172.21.15.28:0/1421436825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c68158480 msgr2=0x7f7c68157bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:31.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.527+0000 7f7c6d600640 1 --2- 172.21.15.28:0/1421436825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c68158480 0x7f7c68157bb0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f7c60036040 tx=0x7f7c60004a40 comp rx=0 tx=0).stop 2024-02-21T23:03:31.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.527+0000 7f7c6d600640 1 -- 172.21.15.28:0/1421436825 shutdown_connections 2024-02-21T23:03:31.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.527+0000 7f7c6d600640 1 --2- 172.21.15.28:0/1421436825 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7c4406b4d0 0x7f7c4406d990 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:31.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.527+0000 7f7c6d600640 1 --2- 172.21.15.28:0/1421436825 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7c68152560 0x7f7c68149b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:31.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.527+0000 7f7c6d600640 1 --2- 172.21.15.28:0/1421436825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c68158480 0x7f7c68157bb0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:31.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.527+0000 7f7c6d600640 1 --2- 172.21.15.28:0/1421436825 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7c6806e9d0 0x7f7c681eaac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:31.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.527+0000 7f7c6d600640 1 -- 172.21.15.28:0/1421436825 >> 172.21.15.28:0/1421436825 conn(0x7f7c680a8530 msgr2=0x7f7c680c5640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:31.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.527+0000 7f7c6d600640 1 -- 172.21.15.28:0/1421436825 shutdown_connections 2024-02-21T23:03:31.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.527+0000 7f7c6d600640 1 -- 172.21.15.28:0/1421436825 wait complete. 2024-02-21T23:03:31.535 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-21T23:03:31.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.769+0000 7fb01a000640 1 Processor -- start 2024-02-21T23:03:31.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.769+0000 7fb01a000640 1 -- start start 2024-02-21T23:03:31.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.770+0000 7fb01a000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb014150240 0x7fb0140c12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:31.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.771+0000 7fb01a000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb0140c2c10 0x7fb01406ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:31.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.771+0000 7fb01a000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb01406e220 con 0x7fb0140c4580 2024-02-21T23:03:31.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.771+0000 7fb01a000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb014150620 con 0x7fb014150240 2024-02-21T23:03:31.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.771+0000 7fb01a000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb014153a90 con 0x7fb0140c2c10 2024-02-21T23:03:31.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.771+0000 7fb018c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb014150240 0x7fb0140c12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:31.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.771+0000 7fb013e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb0140c4580 0x7fb01406db10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59394/0 (socket says 172.21.15.28:59394) 2024-02-21T23:03:31.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.771+0000 7fb013e00640 1 -- 172.21.15.28:0/391515757 learned_addr learned my addr 172.21.15.28:0/391515757 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:03:31.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.771+0000 7fb019600640 1 --2- 172.21.15.28:0/391515757 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb0140c2c10 0x7fb01406ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:31.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.772+0000 7fb018c00640 1 -- 172.21.15.28:0/391515757 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb0140c2c10 msgr2=0x7fb01406ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:31.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.772+0000 7fb018c00640 1 --2- 172.21.15.28:0/391515757 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb0140c2c10 0x7fb01406ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:31.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.772+0000 7fb018c00640 1 -- 172.21.15.28:0/391515757 >> v1:172.21.15.28:6789/0 conn(0x7fb0140c4580 legacy=0x7fb01406db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:31.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.772+0000 7fb013400640 1 -- 172.21.15.28:0/391515757 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3038625839 0 0) 0x7fb01406e220 con 0x7fb0140c4580 2024-02-21T23:03:31.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.772+0000 7fb018c00640 1 -- 172.21.15.28:0/391515757 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb01406f780 con 0x7fb014150240 2024-02-21T23:03:31.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.772+0000 7fb019600640 1 --2- 172.21.15.28:0/391515757 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb0140c2c10 0x7fb01406ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:03:31.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.773+0000 7fb018c00640 1 --2- 172.21.15.28:0/391515757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb014150240 0x7fb0140c12c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fb0080097a0 tx=0x7fb00802b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4df70f117621a621 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:31.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.773+0000 7fb013400640 1 -- 172.21.15.28:0/391515757 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb008031780 con 0x7fb014150240 2024-02-21T23:03:31.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.773+0000 7fb013400640 1 -- 172.21.15.28:0/391515757 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb008037070 con 0x7fb014150240 2024-02-21T23:03:31.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.773+0000 7fb013400640 1 -- 172.21.15.28:0/391515757 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb008031b40 con 0x7fb014150240 2024-02-21T23:03:31.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.774+0000 7fb01a000640 1 -- 172.21.15.28:0/391515757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb014150240 msgr2=0x7fb0140c12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:31.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.774+0000 7fb01a000640 1 --2- 172.21.15.28:0/391515757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb014150240 0x7fb0140c12c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fb0080097a0 tx=0x7fb00802b820 comp rx=0 tx=0).stop 2024-02-21T23:03:31.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.774+0000 7fb01a000640 1 -- 172.21.15.28:0/391515757 shutdown_connections 2024-02-21T23:03:31.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.774+0000 7fb01a000640 1 --2- 172.21.15.28:0/391515757 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb0140c2c10 0x7fb01406ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:31.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.774+0000 7fb01a000640 1 --2- 172.21.15.28:0/391515757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb014150240 0x7fb0140c12c0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:31.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.774+0000 7fb01a000640 1 -- 172.21.15.28:0/391515757 >> 172.21.15.28:0/391515757 conn(0x7fb0140a8530 msgr2=0x7fb0140a8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:31.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.774+0000 7fb01a000640 1 -- 172.21.15.28:0/391515757 shutdown_connections 2024-02-21T23:03:31.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.774+0000 7fb01a000640 1 -- 172.21.15.28:0/391515757 wait complete. 2024-02-21T23:03:31.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.776+0000 7fb01a000640 1 Processor -- start 2024-02-21T23:03:31.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.776+0000 7fb01a000640 1 -- start start 2024-02-21T23:03:31.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.777+0000 7fb01a000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb0140c2c10 0x7fb014153fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:31.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.777+0000 7fb01a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0140c4580 0x7fb014150c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:31.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.777+0000 7fb01a000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb014150240 0x7fb014154e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:31.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.777+0000 7fb01a000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0140c4a10 con 0x7fb0140c4580 2024-02-21T23:03:31.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.777+0000 7fb01a000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb014153720 con 0x7fb014150240 2024-02-21T23:03:31.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.777+0000 7fb01a000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0140c4b90 con 0x7fb0140c2c10 2024-02-21T23:03:31.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.777+0000 7fb013e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0140c4580 0x7fb014150c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:31.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.777+0000 7fb013e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0140c4580 0x7fb014150c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45158/0 (socket says 172.21.15.28:45158) 2024-02-21T23:03:31.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.777+0000 7fb013e00640 1 -- 172.21.15.28:0/287640039 learned_addr learned my addr 172.21.15.28:0/287640039 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:31.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.778+0000 7fb019600640 1 --2- 172.21.15.28:0/287640039 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb014150240 0x7fb014154e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:31.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.778+0000 7fb013e00640 1 -- 172.21.15.28:0/287640039 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb0140c2c10 msgr2=0x7fb014153fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:31.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.778+0000 7fb013e00640 1 --2- 172.21.15.28:0/287640039 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb0140c2c10 0x7fb014153fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:31.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.778+0000 7fb013e00640 1 -- 172.21.15.28:0/287640039 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb014150240 msgr2=0x7fb014154e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:31.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.778+0000 7fb013e00640 1 --2- 172.21.15.28:0/287640039 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb014150240 0x7fb014154e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:31.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.778+0000 7fb013e00640 1 -- 172.21.15.28:0/287640039 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb014154510 con 0x7fb0140c4580 2024-02-21T23:03:31.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.778+0000 7fb013e00640 1 --2- 172.21.15.28:0/287640039 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0140c4580 0x7fb014150c80 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fb00000c4e0 tx=0x7fb00000c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:31.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.778+0000 7fb010c00640 1 -- 172.21.15.28:0/287640039 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb000018070 con 0x7fb0140c4580 2024-02-21T23:03:31.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.779+0000 7fb010c00640 1 -- 172.21.15.28:0/287640039 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb000007500 con 0x7fb0140c4580 2024-02-21T23:03:31.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.779+0000 7fb010c00640 1 -- 172.21.15.28:0/287640039 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb000004a20 con 0x7fb0140c4580 2024-02-21T23:03:31.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.779+0000 7fb01a000640 1 -- 172.21.15.28:0/287640039 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb01414b440 con 0x7fb0140c4580 2024-02-21T23:03:31.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.779+0000 7fb01a000640 1 -- 172.21.15.28:0/287640039 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0141511c0 con 0x7fb0140c4580 2024-02-21T23:03:31.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.780+0000 7fb01a000640 1 -- 172.21.15.28:0/287640039 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb01404f290 con 0x7fb0140c4580 2024-02-21T23:03:31.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.781+0000 7fb010c00640 1 -- 172.21.15.28:0/287640039 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb000004bc0 con 0x7fb0140c4580 2024-02-21T23:03:31.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.782+0000 7fb010c00640 1 --2- 172.21.15.28:0/287640039 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7faff406b500 0x7faff406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:31.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.782+0000 7fb010c00640 1 -- 172.21.15.28:0/287640039 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6735+0+0 (secure 0 0 0) 0x7fb000089890 con 0x7fb0140c4580 2024-02-21T23:03:31.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.782+0000 7fb018c00640 1 --2- 172.21.15.28:0/287640039 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7faff406b500 0x7faff406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:31.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.784+0000 7fb018c00640 1 --2- 172.21.15.28:0/287640039 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7faff406b500 0x7faff406d9c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb008036000 tx=0x7fb008030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:31.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.786+0000 7fb010c00640 1 -- 172.21.15.28:0/287640039 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb000058460 con 0x7fb0140c4580 2024-02-21T23:03:31.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.989+0000 7fb01a000640 1 -- 172.21.15.28:0/287640039 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb01406f0e0 con 0x7fb0140c4580 2024-02-21T23:03:31.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.990+0000 7fb010c00640 1 -- 172.21.15.28:0/287640039 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb000057d60 con 0x7fb0140c4580 2024-02-21T23:03:31.979 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-02-21T23:03:31.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.993+0000 7fb01a000640 1 -- 172.21.15.28:0/287640039 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7faff406b500 msgr2=0x7faff406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:31.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.993+0000 7fb01a000640 1 --2- 172.21.15.28:0/287640039 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7faff406b500 0x7faff406d9c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb008036000 tx=0x7fb008030000 comp rx=0 tx=0).stop 2024-02-21T23:03:31.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.993+0000 7fb01a000640 1 -- 172.21.15.28:0/287640039 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0140c4580 msgr2=0x7fb014150c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:31.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.993+0000 7fb01a000640 1 --2- 172.21.15.28:0/287640039 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0140c4580 0x7fb014150c80 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fb00000c4e0 tx=0x7fb00000c9a0 comp rx=0 tx=0).stop 2024-02-21T23:03:31.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.993+0000 7fb01a000640 1 -- 172.21.15.28:0/287640039 shutdown_connections 2024-02-21T23:03:31.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.993+0000 7fb01a000640 1 --2- 172.21.15.28:0/287640039 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7faff406b500 0x7faff406d9c0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:31.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.993+0000 7fb01a000640 1 --2- 172.21.15.28:0/287640039 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb014150240 0x7fb014154e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:31.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.993+0000 7fb01a000640 1 --2- 172.21.15.28:0/287640039 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0140c4580 0x7fb014150c80 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:31.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.993+0000 7fb01a000640 1 --2- 172.21.15.28:0/287640039 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb0140c2c10 0x7fb014153fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:31.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.993+0000 7fb01a000640 1 -- 172.21.15.28:0/287640039 >> 172.21.15.28:0/287640039 conn(0x7fb0140a8530 msgr2=0x7fb01414c5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:31.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.993+0000 7fb01a000640 1 -- 172.21.15.28:0/287640039 shutdown_connections 2024-02-21T23:03:31.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:31.993+0000 7fb01a000640 1 -- 172.21.15.28:0/287640039 wait complete. 2024-02-21T23:03:32.001 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-02-21T23:03:32.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.239+0000 7f5fb2e00640 1 Processor -- start 2024-02-21T23:03:32.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.239+0000 7f5fb2e00640 1 -- start start 2024-02-21T23:03:32.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.240+0000 7f5fb2e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f5fac158480 0x7f5fac154f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:32.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.240+0000 7f5fb2e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5fac06e9d0 0x7f5fac06ee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:32.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.240+0000 7f5fb2e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fac0a6270 con 0x7f5fac152560 2024-02-21T23:03:32.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.240+0000 7f5fb2e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fac1554a0 con 0x7f5fac06e9d0 2024-02-21T23:03:32.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.240+0000 7f5fb2e00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fac1556d0 con 0x7f5fac158480 2024-02-21T23:03:32.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.241+0000 7f5fb1a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f5fac152560 0x7f5fac06ae70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59402/0 (socket says 172.21.15.28:59402) 2024-02-21T23:03:32.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.241+0000 7f5fb1a00640 1 -- 172.21.15.28:0/1130236981 learned_addr learned my addr 172.21.15.28:0/1130236981 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:03:32.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.241+0000 7f5fb2400640 1 --2- 172.21.15.28:0/1130236981 >> v2:172.21.15.28:3300/0 conn(0x7f5fac06e9d0 0x7f5fac06ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:32.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.241+0000 7f5fb1000640 1 --2- 172.21.15.28:0/1130236981 >> v2:172.21.15.81:3300/0 conn(0x7f5fac158480 0x7f5fac154f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:32.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.241+0000 7f5fabe00640 1 -- 172.21.15.28:0/1130236981 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 837676345 0 0) 0x7f5fac0a6270 con 0x7f5fac152560 2024-02-21T23:03:32.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.241+0000 7f5fb2400640 1 -- 172.21.15.28:0/1130236981 >> v2:172.21.15.81:3300/0 conn(0x7f5fac158480 msgr2=0x7f5fac154f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:32.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.241+0000 7f5fb2400640 1 --2- 172.21.15.28:0/1130236981 >> v2:172.21.15.81:3300/0 conn(0x7f5fac158480 0x7f5fac154f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:32.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.241+0000 7f5fb2400640 1 -- 172.21.15.28:0/1130236981 >> v1:172.21.15.28:6789/0 conn(0x7f5fac152560 legacy=0x7f5fac06ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:32.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.241+0000 7f5fb2400640 1 -- 172.21.15.28:0/1130236981 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fac1559b0 con 0x7f5fac06e9d0 2024-02-21T23:03:32.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.243+0000 7f5fb2400640 1 --2- 172.21.15.28:0/1130236981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fac06e9d0 0x7f5fac06ee00 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f5fac06f7f0 tx=0x7f5fa402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26df723f941208d server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:32.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.243+0000 7f5fabe00640 1 -- 172.21.15.28:0/1130236981 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa4039070 con 0x7f5fac06e9d0 2024-02-21T23:03:32.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.243+0000 7f5fabe00640 1 -- 172.21.15.28:0/1130236981 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fa402cc80 con 0x7f5fac06e9d0 2024-02-21T23:03:32.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.243+0000 7f5fabe00640 1 -- 172.21.15.28:0/1130236981 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa4034660 con 0x7f5fac06e9d0 2024-02-21T23:03:32.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.243+0000 7f5fb2e00640 1 -- 172.21.15.28:0/1130236981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fac06e9d0 msgr2=0x7f5fac06ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:32.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.243+0000 7f5fb2e00640 1 --2- 172.21.15.28:0/1130236981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fac06e9d0 0x7f5fac06ee00 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f5fac06f7f0 tx=0x7f5fa402bed0 comp rx=0 tx=0).stop 2024-02-21T23:03:32.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.243+0000 7f5fb2e00640 1 -- 172.21.15.28:0/1130236981 shutdown_connections 2024-02-21T23:03:32.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.243+0000 7f5fb2e00640 1 --2- 172.21.15.28:0/1130236981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fac06e9d0 0x7f5fac06ee00 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:32.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.243+0000 7f5fb2e00640 1 --2- 172.21.15.28:0/1130236981 >> v2:172.21.15.81:3300/0 conn(0x7f5fac158480 0x7f5fac154f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:32.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.243+0000 7f5fb2e00640 1 -- 172.21.15.28:0/1130236981 >> 172.21.15.28:0/1130236981 conn(0x7f5fac0a8530 msgr2=0x7f5fac0a8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:32.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.244+0000 7f5fb2e00640 1 -- 172.21.15.28:0/1130236981 shutdown_connections 2024-02-21T23:03:32.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.244+0000 7f5fb2e00640 1 -- 172.21.15.28:0/1130236981 wait complete. 2024-02-21T23:03:32.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.245+0000 7f5fb2e00640 1 Processor -- start 2024-02-21T23:03:32.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.245+0000 7f5fb2e00640 1 -- start start 2024-02-21T23:03:32.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.246+0000 7f5fb2e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fac06e9d0 0x7f5fac150ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:32.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.246+0000 7f5fb2e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5fac152560 0x7f5fac156510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:32.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.246+0000 7f5fb2e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5fac158480 0x7f5fac156a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:32.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.246+0000 7f5fb2e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fac06f340 con 0x7f5fac06e9d0 2024-02-21T23:03:32.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.246+0000 7f5fb2e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fac06e4b0 con 0x7f5fac152560 2024-02-21T23:03:32.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.246+0000 7f5fb2e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fac152940 con 0x7f5fac158480 2024-02-21T23:03:32.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.246+0000 7f5fb1a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fac06e9d0 0x7f5fac150ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:32.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.247+0000 7f5fb1a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fac06e9d0 0x7f5fac150ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45168/0 (socket says 172.21.15.28:45168) 2024-02-21T23:03:32.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.247+0000 7f5fb1a00640 1 -- 172.21.15.28:0/3450516891 learned_addr learned my addr 172.21.15.28:0/3450516891 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:32.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.247+0000 7f5fb1a00640 1 -- 172.21.15.28:0/3450516891 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5fac158480 msgr2=0x7f5fac156a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:32.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.247+0000 7f5fb1a00640 1 --2- 172.21.15.28:0/3450516891 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5fac158480 0x7f5fac156a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:32.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.247+0000 7f5fb1a00640 1 -- 172.21.15.28:0/3450516891 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5fac152560 msgr2=0x7f5fac156510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:32.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.247+0000 7f5fb1a00640 1 --2- 172.21.15.28:0/3450516891 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5fac152560 0x7f5fac156510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:32.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.247+0000 7f5fb1a00640 1 -- 172.21.15.28:0/3450516891 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fac157e90 con 0x7f5fac06e9d0 2024-02-21T23:03:32.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.247+0000 7f5fb1a00640 1 --2- 172.21.15.28:0/3450516891 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fac06e9d0 0x7f5fac150ff0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f5fa000c500 tx=0x7f5fa000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:32.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.247+0000 7f5fa9600640 1 -- 172.21.15.28:0/3450516891 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa0018070 con 0x7f5fac06e9d0 2024-02-21T23:03:32.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.247+0000 7f5fa9600640 1 -- 172.21.15.28:0/3450516891 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fa0007580 con 0x7f5fac06e9d0 2024-02-21T23:03:32.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.247+0000 7f5fb2e00640 1 -- 172.21.15.28:0/3450516891 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fac072ed0 con 0x7f5fac06e9d0 2024-02-21T23:03:32.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.247+0000 7f5fa9600640 1 -- 172.21.15.28:0/3450516891 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa0004a20 con 0x7f5fac06e9d0 2024-02-21T23:03:32.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.248+0000 7f5fb2e00640 1 -- 172.21.15.28:0/3450516891 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fac151530 con 0x7f5fac06e9d0 2024-02-21T23:03:32.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.249+0000 7f5fb2e00640 1 -- 172.21.15.28:0/3450516891 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fac04f290 con 0x7f5fac06e9d0 2024-02-21T23:03:32.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.250+0000 7f5fa9600640 1 -- 172.21.15.28:0/3450516891 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5fa001b070 con 0x7f5fac06e9d0 2024-02-21T23:03:32.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.251+0000 7f5fa9600640 1 --2- 172.21.15.28:0/3450516891 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5f8c06b4d0 0x7f5f8c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:32.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.251+0000 7f5fa9600640 1 -- 172.21.15.28:0/3450516891 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f5fa00898c0 con 0x7f5fac06e9d0 2024-02-21T23:03:32.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.252+0000 7f5fb1000640 1 --2- 172.21.15.28:0/3450516891 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5f8c06b4d0 0x7f5f8c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:32.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.253+0000 7f5fb1000640 1 --2- 172.21.15.28:0/3450516891 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5f8c06b4d0 0x7f5f8c06d990 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5fac0c5a40 tx=0x7f5f98008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:32.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.255+0000 7f5fa9600640 1 -- 172.21.15.28:0/3450516891 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f5fa00583d0 con 0x7f5fac06e9d0 2024-02-21T23:03:32.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.455+0000 7f5fb2e00640 1 -- 172.21.15.28:0/3450516891 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f5fac06e0f0 con 0x7f5fac06e9d0 2024-02-21T23:03:32.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.456+0000 7f5fa9600640 1 -- 172.21.15.28:0/3450516891 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) v1 ==== 103+0+11 (secure 0 0 0) 0x7f5fa0057cd0 con 0x7f5fac06e9d0 2024-02-21T23:03:32.445 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-02-21T23:03:32.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.459+0000 7f5fb2e00640 1 -- 172.21.15.28:0/3450516891 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5f8c06b4d0 msgr2=0x7f5f8c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:32.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.459+0000 7f5fb2e00640 1 --2- 172.21.15.28:0/3450516891 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5f8c06b4d0 0x7f5f8c06d990 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5fac0c5a40 tx=0x7f5f98008040 comp rx=0 tx=0).stop 2024-02-21T23:03:32.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.459+0000 7f5fb2e00640 1 -- 172.21.15.28:0/3450516891 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fac06e9d0 msgr2=0x7f5fac150ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:32.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.459+0000 7f5fb2e00640 1 --2- 172.21.15.28:0/3450516891 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fac06e9d0 0x7f5fac150ff0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f5fa000c500 tx=0x7f5fa000c9c0 comp rx=0 tx=0).stop 2024-02-21T23:03:32.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.459+0000 7f5fb2e00640 1 -- 172.21.15.28:0/3450516891 shutdown_connections 2024-02-21T23:03:32.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.459+0000 7f5fb2e00640 1 --2- 172.21.15.28:0/3450516891 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5f8c06b4d0 0x7f5f8c06d990 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:32.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.459+0000 7f5fb2e00640 1 --2- 172.21.15.28:0/3450516891 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5fac158480 0x7f5fac156a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:32.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.459+0000 7f5fb2e00640 1 --2- 172.21.15.28:0/3450516891 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5fac152560 0x7f5fac156510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:32.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.459+0000 7f5fb2e00640 1 --2- 172.21.15.28:0/3450516891 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fac06e9d0 0x7f5fac150ff0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:32.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.459+0000 7f5fb2e00640 1 -- 172.21.15.28:0/3450516891 >> 172.21.15.28:0/3450516891 conn(0x7f5fac0a8530 msgr2=0x7f5fac0c1010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:32.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.459+0000 7f5fb2e00640 1 -- 172.21.15.28:0/3450516891 shutdown_connections 2024-02-21T23:03:32.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.459+0000 7f5fb2e00640 1 -- 172.21.15.28:0/3450516891 wait complete. 2024-02-21T23:03:32.467 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:03:32.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.698+0000 7f6b1d000640 1 Processor -- start 2024-02-21T23:03:32.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.698+0000 7f6b1d000640 1 -- start start 2024-02-21T23:03:32.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.700+0000 7f6b1d000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f6b1806ee40 0x7f6b180a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:32.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.700+0000 7f6b1d000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b180c2760 con 0x7f6b18156af0 2024-02-21T23:03:32.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.700+0000 7f6b1d000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b18156310 con 0x7f6b18150450 2024-02-21T23:03:32.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.700+0000 7f6b1d000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b180c22c0 con 0x7f6b1806ee40 2024-02-21T23:03:32.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.700+0000 7f6b17e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f6b18156af0 0x7f6b1806dff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59404/0 (socket says 172.21.15.28:59404) 2024-02-21T23:03:32.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.700+0000 7f6b17e00640 1 -- 172.21.15.28:0/1050231172 learned_addr learned my addr 172.21.15.28:0/1050231172 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:03:32.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.700+0000 7f6b16a00640 1 --2- 172.21.15.28:0/1050231172 >> v2:172.21.15.81:3300/0 conn(0x7f6b1806ee40 0x7f6b180a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:32.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.701+0000 7f6b16000640 1 -- 172.21.15.28:0/1050231172 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3329879428 0 0) 0x7f6b180c2760 con 0x7f6b18156af0 2024-02-21T23:03:32.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.701+0000 7f6b16000640 1 -- 172.21.15.28:0/1050231172 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6afc003520 con 0x7f6b18156af0 2024-02-21T23:03:32.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.701+0000 7f6b16000640 1 -- 172.21.15.28:0/1050231172 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3144561566 0 0) 0x7f6afc003520 con 0x7f6b18156af0 2024-02-21T23:03:32.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.701+0000 7f6b16000640 1 -- 172.21.15.28:0/1050231172 >> v2:172.21.15.81:3300/0 conn(0x7f6b1806ee40 msgr2=0x7f6b180a5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:32.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.701+0000 7f6b16000640 1 --2- 172.21.15.28:0/1050231172 >> v2:172.21.15.81:3300/0 conn(0x7f6b1806ee40 0x7f6b180a5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:32.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.701+0000 7f6b16000640 1 -- 172.21.15.28:0/1050231172 >> v1:172.21.15.81:6789/0 conn(0x7f6b18150450 legacy=0x7f6b181536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:32.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.701+0000 7f6b16000640 1 -- 172.21.15.28:0/1050231172 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b180c29d0 con 0x7f6b18156af0 2024-02-21T23:03:32.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.701+0000 7f6b16000640 1 -- 172.21.15.28:0/1050231172 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f6b100019f0 con 0x7f6b18156af0 2024-02-21T23:03:32.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.701+0000 7f6b16000640 1 -- 172.21.15.28:0/1050231172 >> v1:172.21.15.28:6789/0 conn(0x7f6b18156af0 legacy=0x7f6b1806dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:32.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.701+0000 7f6b16000640 1 --2- 172.21.15.28:0/1050231172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6afc0039b0 0x7f6afc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:32.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.701+0000 7f6b16000640 1 --2- 172.21.15.28:0/1050231172 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6afc004800 0x7f6afc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:32.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.701+0000 7f6b16000640 1 --2- 172.21.15.28:0/1050231172 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6afc0051c0 0x7f6afc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:32.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.701+0000 7f6b16000640 1 -- 172.21.15.28:0/1050231172 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6afc00c030 con 0x7f6afc0039b0 2024-02-21T23:03:32.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.701+0000 7f6b16000640 1 -- 172.21.15.28:0/1050231172 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6afc00c1e0 con 0x7f6afc004800 2024-02-21T23:03:32.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.701+0000 7f6b16000640 1 -- 172.21.15.28:0/1050231172 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6afc0107f0 con 0x7f6afc0051c0 2024-02-21T23:03:32.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.702+0000 7f6b16a00640 1 --2- 172.21.15.28:0/1050231172 >> v2:172.21.15.81:3300/0 conn(0x7f6b1806ee40 0x7f6b180a5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-21T23:03:32.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.702+0000 7f6b17400640 1 --2- 172.21.15.28:0/1050231172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6afc0039b0 0x7f6afc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:32.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.702+0000 7f6b17e00640 1 --2- 172.21.15.28:0/1050231172 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6afc0051c0 0x7f6afc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:32.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.702+0000 7f6b16a00640 1 --2- 172.21.15.28:0/1050231172 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6afc004800 0x7f6afc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:32.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.703+0000 7f6b17400640 1 -- 172.21.15.28:0/1050231172 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6afc0051c0 msgr2=0x7f6afc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:32.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.703+0000 7f6b17400640 1 --2- 172.21.15.28:0/1050231172 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6afc0051c0 0x7f6afc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:32.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.703+0000 7f6b17400640 1 -- 172.21.15.28:0/1050231172 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6afc004800 msgr2=0x7f6afc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:32.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.703+0000 7f6b17400640 1 --2- 172.21.15.28:0/1050231172 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6afc004800 0x7f6afc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:32.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.703+0000 7f6b17e00640 1 -- 172.21.15.28:0/1050231172 reap_dead start 2024-02-21T23:03:32.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.703+0000 7f6b17400640 1 -- 172.21.15.28:0/1050231172 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6afc003520 con 0x7f6afc0039b0 2024-02-21T23:03:32.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.705+0000 7f6b17400640 1 --2- 172.21.15.28:0/1050231172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6afc0039b0 0x7f6afc003dd0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f6b08001750 tx=0x7f6afc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb88d6d33b7f2131 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:32.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.705+0000 7f6b16000640 1 -- 172.21.15.28:0/1050231172 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b08001370 con 0x7f6afc0039b0 2024-02-21T23:03:32.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.705+0000 7f6b16000640 1 -- 172.21.15.28:0/1050231172 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b080052d0 con 0x7f6afc0039b0 2024-02-21T23:03:32.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.705+0000 7f6b16000640 1 -- 172.21.15.28:0/1050231172 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b08003830 con 0x7f6afc0039b0 2024-02-21T23:03:32.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.706+0000 7f6b1d000640 1 -- 172.21.15.28:0/1050231172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6afc0039b0 msgr2=0x7f6afc003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:32.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.706+0000 7f6b1d000640 1 --2- 172.21.15.28:0/1050231172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6afc0039b0 0x7f6afc003dd0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f6b08001750 tx=0x7f6afc005830 comp rx=0 tx=0).stop 2024-02-21T23:03:32.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.706+0000 7f6b1d000640 1 -- 172.21.15.28:0/1050231172 shutdown_connections 2024-02-21T23:03:32.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.706+0000 7f6b1d000640 1 --2- 172.21.15.28:0/1050231172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6afc0039b0 0x7f6afc003dd0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:32.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.706+0000 7f6b1d000640 1 -- 172.21.15.28:0/1050231172 >> 172.21.15.28:0/1050231172 conn(0x7f6b1806b630 msgr2=0x7f6b180c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:32.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.706+0000 7f6b1d000640 1 -- 172.21.15.28:0/1050231172 shutdown_connections 2024-02-21T23:03:32.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.706+0000 7f6b1d000640 1 -- 172.21.15.28:0/1050231172 wait complete. 2024-02-21T23:03:32.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.707+0000 7f6b1d000640 1 Processor -- start 2024-02-21T23:03:32.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.708+0000 7f6b1d000640 1 -- start start 2024-02-21T23:03:32.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.709+0000 7f6b1d000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6afc0039b0 0x7f6b18071f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:32.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.709+0000 7f6b1d000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b180716c0 0x7f6b181436f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:32.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.709+0000 7f6b1d000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b18143c30 0x7f6b18154cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:32.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.709+0000 7f6b1d000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b18156f80 con 0x7f6b180716c0 2024-02-21T23:03:32.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.709+0000 7f6b1d000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b18156660 con 0x7f6b18143c30 2024-02-21T23:03:32.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.709+0000 7f6b1d000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b18157100 con 0x7f6afc0039b0 2024-02-21T23:03:32.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.709+0000 7f6b16a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b180716c0 0x7f6b181436f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:32.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.709+0000 7f6b16a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b180716c0 0x7f6b181436f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45184/0 (socket says 172.21.15.28:45184) 2024-02-21T23:03:32.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.709+0000 7f6b16a00640 1 -- 172.21.15.28:0/1737507776 learned_addr learned my addr 172.21.15.28:0/1737507776 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:32.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.709+0000 7f6b17e00640 1 --2- 172.21.15.28:0/1737507776 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b18143c30 0x7f6b18154cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:32.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.709+0000 7f6b16a00640 1 -- 172.21.15.28:0/1737507776 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6afc0039b0 msgr2=0x7f6b18071f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:32.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.709+0000 7f6b17400640 1 --2- 172.21.15.28:0/1737507776 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6afc0039b0 0x7f6b18071f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:32.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.709+0000 7f6b16a00640 1 --2- 172.21.15.28:0/1737507776 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6afc0039b0 0x7f6b18071f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:32.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.709+0000 7f6b16a00640 1 -- 172.21.15.28:0/1737507776 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b18143c30 msgr2=0x7f6b18154cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:32.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.709+0000 7f6b16a00640 1 --2- 172.21.15.28:0/1737507776 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b18143c30 0x7f6b18154cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:32.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.709+0000 7f6b16a00640 1 -- 172.21.15.28:0/1737507776 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b181440b0 con 0x7f6b180716c0 2024-02-21T23:03:32.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.709+0000 7f6b17e00640 1 --2- 172.21.15.28:0/1737507776 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b18143c30 0x7f6b18154cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:03:32.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.710+0000 7f6b16a00640 1 --2- 172.21.15.28:0/1737507776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b180716c0 0x7f6b181436f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f6b04011c60 tx=0x7f6b0400f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:32.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.710+0000 7f6b0f400640 1 -- 172.21.15.28:0/1737507776 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b0400fd20 con 0x7f6b180716c0 2024-02-21T23:03:32.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.710+0000 7f6b0f400640 1 -- 172.21.15.28:0/1737507776 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b04004030 con 0x7f6b180716c0 2024-02-21T23:03:32.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.710+0000 7f6b0f400640 1 -- 172.21.15.28:0/1737507776 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b04004420 con 0x7f6b180716c0 2024-02-21T23:03:32.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.710+0000 7f6b1d000640 1 -- 172.21.15.28:0/1737507776 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b180c49c0 con 0x7f6b180716c0 2024-02-21T23:03:32.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.710+0000 7f6b1d000640 1 -- 172.21.15.28:0/1737507776 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b181e72d0 con 0x7f6b180716c0 2024-02-21T23:03:32.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.711+0000 7f6b1d000640 1 -- 172.21.15.28:0/1737507776 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b1804f290 con 0x7f6b180716c0 2024-02-21T23:03:32.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.712+0000 7f6b0f400640 1 -- 172.21.15.28:0/1737507776 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6b04012040 con 0x7f6b180716c0 2024-02-21T23:03:32.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.713+0000 7f6b0f400640 1 --2- 172.21.15.28:0/1737507776 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6af806b4d0 0x7f6af806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:32.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.713+0000 7f6b0f400640 1 -- 172.21.15.28:0/1737507776 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f6b0408ab30 con 0x7f6b180716c0 2024-02-21T23:03:32.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.714+0000 7f6b17400640 1 --2- 172.21.15.28:0/1737507776 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6af806b4d0 0x7f6af806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:32.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.715+0000 7f6b17400640 1 --2- 172.21.15.28:0/1737507776 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6af806b4d0 0x7f6af806d990 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f6b08033000 tx=0x7f6b08009030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:32.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.718+0000 7f6b0f400640 1 -- 172.21.15.28:0/1737507776 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f6b04019070 con 0x7f6b180716c0 2024-02-21T23:03:32.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.921+0000 7f6b1d000640 1 -- 172.21.15.28:0/1737507776 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6b18154610 con 0x7f6b180716c0 2024-02-21T23:03:32.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.924+0000 7f6b0f400640 1 -- 172.21.15.28:0/1737507776 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+18843 (secure 0 0 0) 0x7f6b04059660 con 0x7f6b180716c0 2024-02-21T23:03:32.913 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:03:32.914 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":26,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:03:23.860109+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:02:35.795158+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-21T23:02:36.840110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-02-21T23:02:40.441388+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":3,"to":2}]},{"pgid":"3.12","mappings":[{"from":0,"to":1}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:03:32.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.927+0000 7f6b1d000640 1 -- 172.21.15.28:0/1737507776 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6af806b4d0 msgr2=0x7f6af806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:32.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.927+0000 7f6b1d000640 1 --2- 172.21.15.28:0/1737507776 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6af806b4d0 0x7f6af806d990 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f6b08033000 tx=0x7f6b08009030 comp rx=0 tx=0).stop 2024-02-21T23:03:32.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.927+0000 7f6b1d000640 1 -- 172.21.15.28:0/1737507776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b180716c0 msgr2=0x7f6b181436f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:32.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.927+0000 7f6b1d000640 1 --2- 172.21.15.28:0/1737507776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b180716c0 0x7f6b181436f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f6b04011c60 tx=0x7f6b0400f610 comp rx=0 tx=0).stop 2024-02-21T23:03:32.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.927+0000 7f6b1d000640 1 -- 172.21.15.28:0/1737507776 shutdown_connections 2024-02-21T23:03:32.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.927+0000 7f6b1d000640 1 --2- 172.21.15.28:0/1737507776 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6af806b4d0 0x7f6af806d990 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:32.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.927+0000 7f6b1d000640 1 --2- 172.21.15.28:0/1737507776 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b18143c30 0x7f6b18154cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:32.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.927+0000 7f6b1d000640 1 --2- 172.21.15.28:0/1737507776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b180716c0 0x7f6b181436f0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:32.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.927+0000 7f6b1d000640 1 --2- 172.21.15.28:0/1737507776 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6afc0039b0 0x7f6b18071f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:32.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.927+0000 7f6b1d000640 1 -- 172.21.15.28:0/1737507776 >> 172.21.15.28:0/1737507776 conn(0x7f6b1806b630 msgr2=0x7f6b1814f340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:32.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.928+0000 7f6b1d000640 1 -- 172.21.15.28:0/1737507776 shutdown_connections 2024-02-21T23:03:32.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:32.928+0000 7f6b1d000640 1 -- 172.21.15.28:0/1737507776 wait complete. 2024-02-21T23:03:32.937 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-21T23:02:30.728329+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-21T23:02:35.795158+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-02-21T23:02:36.840110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-02-21T23:02:40.441388+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-02-21T23:03:32.937 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-21T23:03:33.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.175+0000 7f2efb600640 1 Processor -- start 2024-02-21T23:03:33.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.175+0000 7f2efb600640 1 -- start start 2024-02-21T23:03:33.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.177+0000 7f2efb600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2ef40c4960 0x7f2ef40c6220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:33.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.177+0000 7f2efb600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2ef41524f0 0x7f2ef4152930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:33.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.177+0000 7f2efb600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f2ef41586b0 0x7f2ef4158b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:33.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.177+0000 7f2efb600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ef40a62e0 con 0x7f2ef41524f0 2024-02-21T23:03:33.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.177+0000 7f2efb600640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ef40c1960 con 0x7f2ef41586b0 2024-02-21T23:03:33.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.177+0000 7f2efb600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef40c20d0 con 0x7f2ef40c4960 2024-02-21T23:03:33.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.178+0000 7f2ef3e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2ef41524f0 0x7f2ef4152930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:33.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.178+0000 7f2ef9600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f2ef41586b0 0x7f2ef4158b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:33.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.178+0000 7f2ef9600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f2ef41586b0 0x7f2ef4158b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.28:36322/0 (socket says 172.21.15.28:36322) 2024-02-21T23:03:33.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.178+0000 7f2ef8c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2ef40c4960 0x7f2ef40c6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:33.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.178+0000 7f2ef9600640 1 -- 172.21.15.28:0/3782627523 learned_addr learned my addr 172.21.15.28:0/3782627523 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:33.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.178+0000 7f2ef9600640 1 -- 172.21.15.28:0/3782627523 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2ef40c4960 msgr2=0x7f2ef40c6220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:33.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.178+0000 7f2ef9600640 1 --2- 172.21.15.28:0/3782627523 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2ef40c4960 0x7f2ef40c6220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:33.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.179+0000 7f2ef9600640 1 -- 172.21.15.28:0/3782627523 >> v2:172.21.15.28:3300/0 conn(0x7f2ef41524f0 msgr2=0x7f2ef4152930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:33.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.179+0000 7f2ef9600640 1 --2- 172.21.15.28:0/3782627523 >> v2:172.21.15.28:3300/0 conn(0x7f2ef41524f0 0x7f2ef4152930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:33.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.179+0000 7f2ef9600640 1 -- 172.21.15.28:0/3782627523 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ef40c1290 con 0x7f2ef41586b0 2024-02-21T23:03:33.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.179+0000 7f2ef3e00640 1 --2- 172.21.15.28:0/3782627523 >> v2:172.21.15.28:3300/0 conn(0x7f2ef41524f0 0x7f2ef4152930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-21T23:03:33.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.179+0000 7f2ef8c00640 1 --2- 172.21.15.28:0/3782627523 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2ef40c4960 0x7f2ef40c6220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-21T23:03:33.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.180+0000 7f2ef9600640 1 --2- 172.21.15.28:0/3782627523 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ef41586b0 0x7f2ef4158b40 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f2ef4073730 tx=0x7f2eec02be50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=19666c84c123c927 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:33.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.181+0000 7f2ef3400640 1 -- 172.21.15.28:0/3782627523 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eec02c9f0 con 0x7f2ef41586b0 2024-02-21T23:03:33.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.181+0000 7f2ef3400640 1 -- 172.21.15.28:0/3782627523 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2eec02cb90 con 0x7f2ef41586b0 2024-02-21T23:03:33.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.181+0000 7f2ef3400640 1 -- 172.21.15.28:0/3782627523 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eec0343f0 con 0x7f2ef41586b0 2024-02-21T23:03:33.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.181+0000 7f2efb600640 1 -- 172.21.15.28:0/3782627523 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ef41586b0 msgr2=0x7f2ef4158b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:33.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.181+0000 7f2efb600640 1 --2- 172.21.15.28:0/3782627523 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ef41586b0 0x7f2ef4158b40 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f2ef4073730 tx=0x7f2eec02be50 comp rx=0 tx=0).stop 2024-02-21T23:03:33.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.182+0000 7f2efb600640 1 -- 172.21.15.28:0/3782627523 shutdown_connections 2024-02-21T23:03:33.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.182+0000 7f2efb600640 1 --2- 172.21.15.28:0/3782627523 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ef41586b0 0x7f2ef4158b40 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:33.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.182+0000 7f2efb600640 1 --2- 172.21.15.28:0/3782627523 >> v2:172.21.15.28:3300/0 conn(0x7f2ef41524f0 0x7f2ef4152930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:33.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.182+0000 7f2efb600640 1 --2- 172.21.15.28:0/3782627523 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2ef40c4960 0x7f2ef40c6220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:33.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.182+0000 7f2efb600640 1 -- 172.21.15.28:0/3782627523 >> 172.21.15.28:0/3782627523 conn(0x7f2ef40a85a0 msgr2=0x7f2ef40a8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:33.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.182+0000 7f2efb600640 1 -- 172.21.15.28:0/3782627523 shutdown_connections 2024-02-21T23:03:33.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.182+0000 7f2efb600640 1 -- 172.21.15.28:0/3782627523 wait complete. 2024-02-21T23:03:33.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.183+0000 7f2efb600640 1 Processor -- start 2024-02-21T23:03:33.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.184+0000 7f2efb600640 1 -- start start 2024-02-21T23:03:33.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.184+0000 7f2efb600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ef40c4960 0x7f2ef40aa8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:33.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.184+0000 7f2efb600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ef41524f0 0x7f2ef4071f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:33.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.184+0000 7f2efb600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2ef41586b0 0x7f2ef40a9b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:33.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.184+0000 7f2efb600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef40a5f20 con 0x7f2ef41524f0 2024-02-21T23:03:33.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.184+0000 7f2efb600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef40c1bb0 con 0x7f2ef40c4960 2024-02-21T23:03:33.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.184+0000 7f2efb600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef40a60a0 con 0x7f2ef41586b0 2024-02-21T23:03:33.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.185+0000 7f2ef3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ef41524f0 0x7f2ef4071f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:33.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.185+0000 7f2ef3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ef41524f0 0x7f2ef4071f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45208/0 (socket says 172.21.15.28:45208) 2024-02-21T23:03:33.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.185+0000 7f2ef3e00640 1 -- 172.21.15.28:0/4232643468 learned_addr learned my addr 172.21.15.28:0/4232643468 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:33.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.185+0000 7f2ef8c00640 1 --2- 172.21.15.28:0/4232643468 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ef40c4960 0x7f2ef40aa8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:33.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.185+0000 7f2ef3e00640 1 -- 172.21.15.28:0/4232643468 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2ef41586b0 msgr2=0x7f2ef40a9b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:33.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.185+0000 7f2ef3e00640 1 --2- 172.21.15.28:0/4232643468 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2ef41586b0 0x7f2ef40a9b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:33.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.185+0000 7f2ef3e00640 1 -- 172.21.15.28:0/4232643468 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ef40c4960 msgr2=0x7f2ef40aa8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:33.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.185+0000 7f2ef3e00640 1 --2- 172.21.15.28:0/4232643468 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ef40c4960 0x7f2ef40aa8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:33.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.185+0000 7f2ef3e00640 1 -- 172.21.15.28:0/4232643468 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ef41eb4b0 con 0x7f2ef41524f0 2024-02-21T23:03:33.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.185+0000 7f2ef3e00640 1 --2- 172.21.15.28:0/4232643468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ef41524f0 0x7f2ef4071f90 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f2ee000ebe0 tx=0x7f2ee000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:33.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.186+0000 7f2ef0c00640 1 -- 172.21.15.28:0/4232643468 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee0017070 con 0x7f2ef41524f0 2024-02-21T23:03:33.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.186+0000 7f2ef0c00640 1 -- 172.21.15.28:0/4232643468 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ee000cc30 con 0x7f2ef41524f0 2024-02-21T23:03:33.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.186+0000 7f2efb600640 1 -- 172.21.15.28:0/4232643468 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ef40aa620 con 0x7f2ef41524f0 2024-02-21T23:03:33.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.186+0000 7f2ef0c00640 1 -- 172.21.15.28:0/4232643468 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee0013470 con 0x7f2ef41524f0 2024-02-21T23:03:33.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.186+0000 7f2efb600640 1 -- 172.21.15.28:0/4232643468 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ef41eb720 con 0x7f2ef41524f0 2024-02-21T23:03:33.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.187+0000 7f2efb600640 1 -- 172.21.15.28:0/4232643468 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ef4067f30 con 0x7f2ef41524f0 2024-02-21T23:03:33.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.188+0000 7f2ef0c00640 1 -- 172.21.15.28:0/4232643468 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2ee001a070 con 0x7f2ef41524f0 2024-02-21T23:03:33.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.189+0000 7f2ef0c00640 1 --2- 172.21.15.28:0/4232643468 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2ed406b4d0 0x7f2ed406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:33.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.189+0000 7f2ef0c00640 1 -- 172.21.15.28:0/4232643468 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f2ee008a910 con 0x7f2ef41524f0 2024-02-21T23:03:33.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.189+0000 7f2ef8c00640 1 --2- 172.21.15.28:0/4232643468 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2ed406b4d0 0x7f2ed406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:33.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.191+0000 7f2ef8c00640 1 --2- 172.21.15.28:0/4232643468 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2ed406b4d0 0x7f2ed406d990 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f2ee8009560 tx=0x7f2ee8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:33.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.193+0000 7f2ef0c00640 1 -- 172.21.15.28:0/4232643468 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f2ee0059380 con 0x7f2ef41524f0 2024-02-21T23:03:33.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.396+0000 7f2efb600640 1 -- 172.21.15.28:0/4232643468 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2ef4152c70 con 0x7f2ef41524f0 2024-02-21T23:03:33.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.397+0000 7f2ef0c00640 1 -- 172.21.15.28:0/4232643468 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2ee0058c80 con 0x7f2ef41524f0 2024-02-21T23:03:33.386 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-02-21T23:03:33.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.400+0000 7f2efb600640 1 -- 172.21.15.28:0/4232643468 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2ed406b4d0 msgr2=0x7f2ed406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:33.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.400+0000 7f2efb600640 1 --2- 172.21.15.28:0/4232643468 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2ed406b4d0 0x7f2ed406d990 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f2ee8009560 tx=0x7f2ee8008040 comp rx=0 tx=0).stop 2024-02-21T23:03:33.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.400+0000 7f2efb600640 1 -- 172.21.15.28:0/4232643468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ef41524f0 msgr2=0x7f2ef4071f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:33.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.400+0000 7f2efb600640 1 --2- 172.21.15.28:0/4232643468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ef41524f0 0x7f2ef4071f90 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f2ee000ebe0 tx=0x7f2ee000c520 comp rx=0 tx=0).stop 2024-02-21T23:03:33.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.400+0000 7f2efb600640 1 -- 172.21.15.28:0/4232643468 shutdown_connections 2024-02-21T23:03:33.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.400+0000 7f2efb600640 1 --2- 172.21.15.28:0/4232643468 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2ed406b4d0 0x7f2ed406d990 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:33.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.400+0000 7f2efb600640 1 --2- 172.21.15.28:0/4232643468 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2ef41586b0 0x7f2ef40a9b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:33.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.400+0000 7f2efb600640 1 --2- 172.21.15.28:0/4232643468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ef41524f0 0x7f2ef4071f90 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:33.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.400+0000 7f2efb600640 1 --2- 172.21.15.28:0/4232643468 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ef40c4960 0x7f2ef40aa8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:33.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.400+0000 7f2efb600640 1 -- 172.21.15.28:0/4232643468 >> 172.21.15.28:0/4232643468 conn(0x7f2ef40a85a0 msgr2=0x7f2ef40c5e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:33.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.400+0000 7f2efb600640 1 -- 172.21.15.28:0/4232643468 shutdown_connections 2024-02-21T23:03:33.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.400+0000 7f2efb600640 1 -- 172.21.15.28:0/4232643468 wait complete. 2024-02-21T23:03:33.409 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-21T23:03:33.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.638+0000 7fbbb2a00640 1 Processor -- start 2024-02-21T23:03:33.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.638+0000 7fbbb2a00640 1 -- start start 2024-02-21T23:03:33.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.640+0000 7fbbb2a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fbbac13ad50 0x7fbbac0c03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:33.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.640+0000 7fbbb2a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbbac1558e0 0x7fbbac158e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:33.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.640+0000 7fbbb2a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbac0671d0 con 0x7fbbac0c0920 2024-02-21T23:03:33.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.640+0000 7fbbb2a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbac159360 con 0x7fbbac1558e0 2024-02-21T23:03:33.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.640+0000 7fbbb2a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbac13b130 con 0x7fbbac13ad50 2024-02-21T23:03:33.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.640+0000 7fbbb2000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbbac1558e0 0x7fbbac158e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:33.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.640+0000 7fbbb2000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbbac1558e0 0x7fbbac158e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45210/0 (socket says 172.21.15.28:45210) 2024-02-21T23:03:33.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.640+0000 7fbbb2000640 1 -- 172.21.15.28:0/3612288186 learned_addr learned my addr 172.21.15.28:0/3612288186 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:33.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.640+0000 7fbbb1600640 1 --2- 172.21.15.28:0/3612288186 >> v2:172.21.15.81:3300/0 conn(0x7fbbac13ad50 0x7fbbac0c03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:33.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.641+0000 7fbbb2000640 1 -- 172.21.15.28:0/3612288186 >> v2:172.21.15.81:3300/0 conn(0x7fbbac13ad50 msgr2=0x7fbbac0c03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:33.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.641+0000 7fbbb2000640 1 --2- 172.21.15.28:0/3612288186 >> v2:172.21.15.81:3300/0 conn(0x7fbbac13ad50 0x7fbbac0c03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:33.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.641+0000 7fbbb2000640 1 -- 172.21.15.28:0/3612288186 >> v1:172.21.15.81:6789/0 conn(0x7fbbac0c0920 legacy=0x7fbbac158710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:33.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.641+0000 7fbbb2000640 1 -- 172.21.15.28:0/3612288186 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbac067450 con 0x7fbbac1558e0 2024-02-21T23:03:33.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.642+0000 7fbbb2000640 1 --2- 172.21.15.28:0/3612288186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbac1558e0 0x7fbbac158e20 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fbbac13b2b0 tx=0x7fbba402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fbf295e3f6e1581 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:33.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.642+0000 7fbbabe00640 1 -- 172.21.15.28:0/3612288186 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbba402cb00 con 0x7fbbac1558e0 2024-02-21T23:03:33.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.642+0000 7fbbabe00640 1 -- 172.21.15.28:0/3612288186 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbba402cca0 con 0x7fbbac1558e0 2024-02-21T23:03:33.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.642+0000 7fbbabe00640 1 -- 172.21.15.28:0/3612288186 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbba4034610 con 0x7fbbac1558e0 2024-02-21T23:03:33.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.643+0000 7fbbb2a00640 1 -- 172.21.15.28:0/3612288186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbac1558e0 msgr2=0x7fbbac158e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:33.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.643+0000 7fbbb2a00640 1 --2- 172.21.15.28:0/3612288186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbac1558e0 0x7fbbac158e20 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fbbac13b2b0 tx=0x7fbba402bf60 comp rx=0 tx=0).stop 2024-02-21T23:03:33.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.643+0000 7fbbb2a00640 1 -- 172.21.15.28:0/3612288186 shutdown_connections 2024-02-21T23:03:33.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.643+0000 7fbbb2a00640 1 --2- 172.21.15.28:0/3612288186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbac1558e0 0x7fbbac158e20 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:33.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.643+0000 7fbbb2a00640 1 --2- 172.21.15.28:0/3612288186 >> v2:172.21.15.81:3300/0 conn(0x7fbbac13ad50 0x7fbbac0c03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:33.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.643+0000 7fbbb2a00640 1 -- 172.21.15.28:0/3612288186 >> 172.21.15.28:0/3612288186 conn(0x7fbbac0c3ab0 msgr2=0x7fbbac0c3e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:33.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.643+0000 7fbbb2a00640 1 -- 172.21.15.28:0/3612288186 shutdown_connections 2024-02-21T23:03:33.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.643+0000 7fbbb2a00640 1 -- 172.21.15.28:0/3612288186 wait complete. 2024-02-21T23:03:33.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.645+0000 7fbbb2a00640 1 Processor -- start 2024-02-21T23:03:33.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.645+0000 7fbbb2a00640 1 -- start start 2024-02-21T23:03:33.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.646+0000 7fbbb2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbac0c0920 0x7fbbac0a8b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:33.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.646+0000 7fbbb2a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbbac13ad50 0x7fbbac154aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:33.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.646+0000 7fbbb2a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbbac1558e0 0x7fbbac137340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:33.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.646+0000 7fbbb2a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbac0c37e0 con 0x7fbbac0c0920 2024-02-21T23:03:33.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.646+0000 7fbbb2a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbac067730 con 0x7fbbac13ad50 2024-02-21T23:03:33.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.646+0000 7fbbb2a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbac0c1ce0 con 0x7fbbac1558e0 2024-02-21T23:03:33.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.647+0000 7fbbb1600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbac0c0920 0x7fbbac0a8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:33.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.647+0000 7fbbb1600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbac0c0920 0x7fbbac0a8b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45218/0 (socket says 172.21.15.28:45218) 2024-02-21T23:03:33.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.647+0000 7fbbb1600640 1 -- 172.21.15.28:0/1190043014 learned_addr learned my addr 172.21.15.28:0/1190043014 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:33.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.647+0000 7fbbb0c00640 1 --2- 172.21.15.28:0/1190043014 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbbac13ad50 0x7fbbac154aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:33.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.647+0000 7fbbb2000640 1 --2- 172.21.15.28:0/1190043014 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbbac1558e0 0x7fbbac137340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:33.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.647+0000 7fbbb1600640 1 -- 172.21.15.28:0/1190043014 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbbac1558e0 msgr2=0x7fbbac137340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:33.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.647+0000 7fbbb1600640 1 --2- 172.21.15.28:0/1190043014 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbbac1558e0 0x7fbbac137340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:33.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.647+0000 7fbbb1600640 1 -- 172.21.15.28:0/1190043014 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbbac13ad50 msgr2=0x7fbbac154aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:33.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.647+0000 7fbbb1600640 1 --2- 172.21.15.28:0/1190043014 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbbac13ad50 0x7fbbac154aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:33.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.647+0000 7fbbb1600640 1 -- 172.21.15.28:0/1190043014 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbac1ee540 con 0x7fbbac0c0920 2024-02-21T23:03:33.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.647+0000 7fbbb2000640 1 --2- 172.21.15.28:0/1190043014 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbbac1558e0 0x7fbbac137340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:03:33.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.648+0000 7fbbb1600640 1 --2- 172.21.15.28:0/1190043014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbac0c0920 0x7fbbac0a8b10 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fbba000d830 tx=0x7fbba000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:33.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.648+0000 7fbba9600640 1 -- 172.21.15.28:0/1190043014 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbba000b6c0 con 0x7fbbac0c0920 2024-02-21T23:03:33.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.648+0000 7fbba9600640 1 -- 172.21.15.28:0/1190043014 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbba0019070 con 0x7fbbac0c0920 2024-02-21T23:03:33.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.648+0000 7fbbb2a00640 1 -- 172.21.15.28:0/1190043014 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbac1ee820 con 0x7fbbac0c0920 2024-02-21T23:03:33.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.648+0000 7fbba9600640 1 -- 172.21.15.28:0/1190043014 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbba000ba80 con 0x7fbbac0c0920 2024-02-21T23:03:33.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.648+0000 7fbbb2a00640 1 -- 172.21.15.28:0/1190043014 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbac1e1ef0 con 0x7fbbac0c0920 2024-02-21T23:03:33.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.649+0000 7fbbb2a00640 1 -- 172.21.15.28:0/1190043014 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbac04f290 con 0x7fbbac0c0920 2024-02-21T23:03:33.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.650+0000 7fbba9600640 1 -- 172.21.15.28:0/1190043014 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbba001e460 con 0x7fbbac0c0920 2024-02-21T23:03:33.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.651+0000 7fbba9600640 1 --2- 172.21.15.28:0/1190043014 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbb9006b4d0 0x7fbb9006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:33.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.651+0000 7fbba9600640 1 -- 172.21.15.28:0/1190043014 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6735+0+0 (secure 0 0 0) 0x7fbba008ac70 con 0x7fbbac0c0920 2024-02-21T23:03:33.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.652+0000 7fbbb0c00640 1 --2- 172.21.15.28:0/1190043014 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbb9006b4d0 0x7fbb9006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:33.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.653+0000 7fbbb0c00640 1 --2- 172.21.15.28:0/1190043014 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbb9006b4d0 0x7fbb9006d990 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fbbac0a9050 tx=0x7fbb9c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:33.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.656+0000 7fbba9600640 1 -- 172.21.15.28:0/1190043014 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fbba0059780 con 0x7fbbac0c0920 2024-02-21T23:03:33.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.860+0000 7fbbb2a00640 1 -- 172.21.15.28:0/1190043014 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fbbac0c0df0 con 0x7fbbac0c0920 2024-02-21T23:03:33.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.861+0000 7fbba9600640 1 -- 172.21.15.28:0/1190043014 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+11 (secure 0 0 0) 0x7fbba0059080 con 0x7fbbac0c0920 2024-02-21T23:03:33.850 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-02-21T23:03:33.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.864+0000 7fbbb2a00640 1 -- 172.21.15.28:0/1190043014 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbb9006b4d0 msgr2=0x7fbb9006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:33.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.864+0000 7fbbb2a00640 1 --2- 172.21.15.28:0/1190043014 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbb9006b4d0 0x7fbb9006d990 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fbbac0a9050 tx=0x7fbb9c009040 comp rx=0 tx=0).stop 2024-02-21T23:03:33.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.864+0000 7fbbb2a00640 1 -- 172.21.15.28:0/1190043014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbac0c0920 msgr2=0x7fbbac0a8b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:33.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.864+0000 7fbbb2a00640 1 --2- 172.21.15.28:0/1190043014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbac0c0920 0x7fbbac0a8b10 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fbba000d830 tx=0x7fbba000dcf0 comp rx=0 tx=0).stop 2024-02-21T23:03:33.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.864+0000 7fbbb2a00640 1 -- 172.21.15.28:0/1190043014 shutdown_connections 2024-02-21T23:03:33.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.864+0000 7fbbb2a00640 1 --2- 172.21.15.28:0/1190043014 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbb9006b4d0 0x7fbb9006d990 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:33.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.864+0000 7fbbb2a00640 1 --2- 172.21.15.28:0/1190043014 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbbac1558e0 0x7fbbac137340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:33.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.864+0000 7fbbb2a00640 1 --2- 172.21.15.28:0/1190043014 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbbac13ad50 0x7fbbac154aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:33.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.864+0000 7fbbb2a00640 1 --2- 172.21.15.28:0/1190043014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbac0c0920 0x7fbbac0a8b10 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:33.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.864+0000 7fbbb2a00640 1 -- 172.21.15.28:0/1190043014 >> 172.21.15.28:0/1190043014 conn(0x7fbbac0c3ab0 msgr2=0x7fbbac139fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:33.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.865+0000 7fbbb2a00640 1 -- 172.21.15.28:0/1190043014 shutdown_connections 2024-02-21T23:03:33.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:33.865+0000 7fbbb2a00640 1 -- 172.21.15.28:0/1190043014 wait complete. 2024-02-21T23:03:33.873 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-21T23:03:34.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.101+0000 7ff3bcc00640 1 Processor -- start 2024-02-21T23:03:34.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.101+0000 7ff3bcc00640 1 -- start start 2024-02-21T23:03:34.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.103+0000 7ff3bcc00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff3b806dbc0 0x7ff3b806c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:34.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.103+0000 7ff3bcc00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7ff3b813e520 0x7ff3b806c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:34.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.103+0000 7ff3bcc00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3b8158550 con 0x7ff3b813e0c0 2024-02-21T23:03:34.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.103+0000 7ff3bcc00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3b80c89a0 con 0x7ff3b813e520 2024-02-21T23:03:34.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.103+0000 7ff3bcc00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b81587d0 con 0x7ff3b806dbc0 2024-02-21T23:03:34.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.103+0000 7ff3b6a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff3b806dbc0 0x7ff3b806c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:34.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.103+0000 7ff3b7e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7ff3b813e520 0x7ff3b806c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:34.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.103+0000 7ff3b7e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7ff3b813e520 0x7ff3b806c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.28:36360/0 (socket says 172.21.15.28:36360) 2024-02-21T23:03:34.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.103+0000 7ff3b6a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff3b806dbc0 0x7ff3b806c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3301/0 says I am v2:172.21.15.28:58256/0 (socket says 172.21.15.28:58256) 2024-02-21T23:03:34.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.103+0000 7ff3b7e00640 1 -- 172.21.15.28:0/2148509745 learned_addr learned my addr 172.21.15.28:0/2148509745 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:34.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.104+0000 7ff3b7e00640 1 -- 172.21.15.28:0/2148509745 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff3b806dbc0 msgr2=0x7ff3b806c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:34.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.104+0000 7ff3b7e00640 1 --2- 172.21.15.28:0/2148509745 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff3b806dbc0 0x7ff3b806c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:34.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.104+0000 7ff3b7e00640 1 -- 172.21.15.28:0/2148509745 >> v1:172.21.15.81:6789/0 conn(0x7ff3b813e0c0 legacy=0x7ff3b80c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:34.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.104+0000 7ff3b7e00640 1 -- 172.21.15.28:0/2148509745 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3b8158ab0 con 0x7ff3b813e520 2024-02-21T23:03:34.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.106+0000 7ff3b7e00640 1 --2- 172.21.15.28:0/2148509745 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3b813e520 0x7ff3b806c7c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff3b0009870 tx=0x7ff3b002b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=57e1f4fe87b06dcf server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:34.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.106+0000 7ff3b6000640 1 -- 172.21.15.28:0/2148509745 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b0035020 con 0x7ff3b813e520 2024-02-21T23:03:34.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.106+0000 7ff3b6000640 1 -- 172.21.15.28:0/2148509745 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3b0031640 con 0x7ff3b813e520 2024-02-21T23:03:34.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.106+0000 7ff3b6000640 1 -- 172.21.15.28:0/2148509745 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b002d040 con 0x7ff3b813e520 2024-02-21T23:03:34.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.107+0000 7ff3bcc00640 1 -- 172.21.15.28:0/2148509745 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3b813e520 msgr2=0x7ff3b806c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:34.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.107+0000 7ff3bcc00640 1 --2- 172.21.15.28:0/2148509745 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3b813e520 0x7ff3b806c7c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff3b0009870 tx=0x7ff3b002b8f0 comp rx=0 tx=0).stop 2024-02-21T23:03:34.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.107+0000 7ff3bcc00640 1 -- 172.21.15.28:0/2148509745 shutdown_connections 2024-02-21T23:03:34.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.107+0000 7ff3bcc00640 1 --2- 172.21.15.28:0/2148509745 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3b813e520 0x7ff3b806c7c0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:34.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.107+0000 7ff3bcc00640 1 --2- 172.21.15.28:0/2148509745 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff3b806dbc0 0x7ff3b806c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:34.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.107+0000 7ff3bcc00640 1 -- 172.21.15.28:0/2148509745 >> 172.21.15.28:0/2148509745 conn(0x7ff3b8073110 msgr2=0x7ff3b80ac490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:34.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.107+0000 7ff3bcc00640 1 -- 172.21.15.28:0/2148509745 shutdown_connections 2024-02-21T23:03:34.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.107+0000 7ff3bcc00640 1 -- 172.21.15.28:0/2148509745 wait complete. 2024-02-21T23:03:34.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.109+0000 7ff3bcc00640 1 Processor -- start 2024-02-21T23:03:34.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.109+0000 7ff3bcc00640 1 -- start start 2024-02-21T23:03:34.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.110+0000 7ff3bcc00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff3b806dbc0 0x7ff3b80adf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:34.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.110+0000 7ff3bcc00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3b813e0c0 0x7ff3b80afd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:34.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.110+0000 7ff3bcc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3b813e520 0x7ff3b81ec150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:34.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.110+0000 7ff3bcc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b80c78c0 con 0x7ff3b813e520 2024-02-21T23:03:34.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.110+0000 7ff3bcc00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b8159070 con 0x7ff3b813e0c0 2024-02-21T23:03:34.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.110+0000 7ff3bcc00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b80c7a40 con 0x7ff3b806dbc0 2024-02-21T23:03:34.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.110+0000 7ff3b7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3b813e520 0x7ff3b81ec150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:34.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.110+0000 7ff3b7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3b813e520 0x7ff3b81ec150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45232/0 (socket says 172.21.15.28:45232) 2024-02-21T23:03:34.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.110+0000 7ff3b7e00640 1 -- 172.21.15.28:0/3813531207 learned_addr learned my addr 172.21.15.28:0/3813531207 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:34.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.110+0000 7ff3b7400640 1 --2- 172.21.15.28:0/3813531207 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff3b806dbc0 0x7ff3b80adf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:34.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.111+0000 7ff3b6a00640 1 --2- 172.21.15.28:0/3813531207 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3b813e0c0 0x7ff3b80afd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:34.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.111+0000 7ff3b7e00640 1 -- 172.21.15.28:0/3813531207 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff3b806dbc0 msgr2=0x7ff3b80adf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:34.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.111+0000 7ff3b7e00640 1 --2- 172.21.15.28:0/3813531207 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff3b806dbc0 0x7ff3b80adf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:34.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.111+0000 7ff3b7e00640 1 -- 172.21.15.28:0/3813531207 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3b813e0c0 msgr2=0x7ff3b80afd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:34.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.111+0000 7ff3b7e00640 1 --2- 172.21.15.28:0/3813531207 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3b813e0c0 0x7ff3b80afd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:34.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.111+0000 7ff3b7e00640 1 -- 172.21.15.28:0/3813531207 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3b80ae570 con 0x7ff3b813e520 2024-02-21T23:03:34.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.111+0000 7ff3b7e00640 1 --2- 172.21.15.28:0/3813531207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3b813e520 0x7ff3b81ec150 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff3b0009870 tx=0x7ff3b00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:34.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.111+0000 7ff3af400640 1 -- 172.21.15.28:0/3813531207 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b003ab60 con 0x7ff3b813e520 2024-02-21T23:03:34.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.111+0000 7ff3af400640 1 -- 172.21.15.28:0/3813531207 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3b003ad20 con 0x7ff3b813e520 2024-02-21T23:03:34.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.111+0000 7ff3af400640 1 -- 172.21.15.28:0/3813531207 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b0044630 con 0x7ff3b813e520 2024-02-21T23:03:34.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.111+0000 7ff3bcc00640 1 -- 172.21.15.28:0/3813531207 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3b80ae850 con 0x7ff3b813e520 2024-02-21T23:03:34.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.112+0000 7ff3bcc00640 1 -- 172.21.15.28:0/3813531207 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3b80aee10 con 0x7ff3b813e520 2024-02-21T23:03:34.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.113+0000 7ff3bcc00640 1 -- 172.21.15.28:0/3813531207 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3b813b800 con 0x7ff3b813e520 2024-02-21T23:03:34.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.114+0000 7ff3af400640 1 -- 172.21.15.28:0/3813531207 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff3b0038020 con 0x7ff3b813e520 2024-02-21T23:03:34.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.114+0000 7ff3af400640 1 --2- 172.21.15.28:0/3813531207 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff39406b4d0 0x7ff39406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:34.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.115+0000 7ff3af400640 1 -- 172.21.15.28:0/3813531207 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6735+0+0 (secure 0 0 0) 0x7ff3b00ac280 con 0x7ff3b813e520 2024-02-21T23:03:34.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.115+0000 7ff3b7400640 1 --2- 172.21.15.28:0/3813531207 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff39406b4d0 0x7ff39406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:34.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.116+0000 7ff3b7400640 1 --2- 172.21.15.28:0/3813531207 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff39406b4d0 0x7ff39406d990 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff3a0007fd0 tx=0x7ff3a0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:34.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.119+0000 7ff3af400640 1 -- 172.21.15.28:0/3813531207 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ff3b813b800 con 0x7ff3b813e520 2024-02-21T23:03:34.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.321+0000 7ff3bcc00640 1 -- 172.21.15.28:0/3813531207 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff3b806d9a0 con 0x7ff3b813e520 2024-02-21T23:03:34.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.322+0000 7ff3af400640 1 -- 172.21.15.28:0/3813531207 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff3b003f070 con 0x7ff3b813e520 2024-02-21T23:03:34.311 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-02-21T23:03:34.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.325+0000 7ff3bcc00640 1 -- 172.21.15.28:0/3813531207 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff39406b4d0 msgr2=0x7ff39406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:34.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.325+0000 7ff3bcc00640 1 --2- 172.21.15.28:0/3813531207 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff39406b4d0 0x7ff39406d990 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff3a0007fd0 tx=0x7ff3a0009040 comp rx=0 tx=0).stop 2024-02-21T23:03:34.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.325+0000 7ff3bcc00640 1 -- 172.21.15.28:0/3813531207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3b813e520 msgr2=0x7ff3b81ec150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:34.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.325+0000 7ff3bcc00640 1 --2- 172.21.15.28:0/3813531207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3b813e520 0x7ff3b81ec150 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff3b0009870 tx=0x7ff3b00046c0 comp rx=0 tx=0).stop 2024-02-21T23:03:34.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.325+0000 7ff3bcc00640 1 -- 172.21.15.28:0/3813531207 shutdown_connections 2024-02-21T23:03:34.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.325+0000 7ff3bcc00640 1 --2- 172.21.15.28:0/3813531207 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff39406b4d0 0x7ff39406d990 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:34.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.325+0000 7ff3bcc00640 1 --2- 172.21.15.28:0/3813531207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3b813e520 0x7ff3b81ec150 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:34.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.325+0000 7ff3bcc00640 1 --2- 172.21.15.28:0/3813531207 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3b813e0c0 0x7ff3b80afd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:34.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.325+0000 7ff3bcc00640 1 --2- 172.21.15.28:0/3813531207 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff3b806dbc0 0x7ff3b80adf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:34.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.325+0000 7ff3bcc00640 1 -- 172.21.15.28:0/3813531207 >> 172.21.15.28:0/3813531207 conn(0x7ff3b8073110 msgr2=0x7ff3b806c180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:34.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.325+0000 7ff3bcc00640 1 -- 172.21.15.28:0/3813531207 shutdown_connections 2024-02-21T23:03:34.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.325+0000 7ff3bcc00640 1 -- 172.21.15.28:0/3813531207 wait complete. 2024-02-21T23:03:34.335 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-02-21T23:03:34.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.563+0000 7ff03f200640 1 Processor -- start 2024-02-21T23:03:34.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.563+0000 7ff03f200640 1 -- start start 2024-02-21T23:03:34.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.564+0000 7ff03f200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7ff04013e0c0 0x7ff0400c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:34.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.564+0000 7ff03f200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff04006dbc0 0x7ff04006c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:34.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.564+0000 7ff03f200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff040158550 con 0x7ff0400c76f0 2024-02-21T23:03:34.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.564+0000 7ff03f200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0400c89a0 con 0x7ff04013e0c0 2024-02-21T23:03:34.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.564+0000 7ff03f200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0401587d0 con 0x7ff04006dbc0 2024-02-21T23:03:34.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.565+0000 7ff03e800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff0400c76f0 0x7ff04006c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59406/0 (socket says 172.21.15.28:59406) 2024-02-21T23:03:34.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.565+0000 7ff03e800640 1 -- 172.21.15.28:0/1080819912 learned_addr learned my addr 172.21.15.28:0/1080819912 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:03:34.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.565+0000 7ff03d400640 1 --2- 172.21.15.28:0/1080819912 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff04006dbc0 0x7ff04006c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:34.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.565+0000 7ff03de00640 1 --2- 172.21.15.28:0/1080819912 >> v2:172.21.15.81:3300/0 conn(0x7ff04013e0c0 0x7ff0400c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:34.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.565+0000 7ff03ca00640 1 -- 172.21.15.28:0/1080819912 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3495835467 0 0) 0x7ff038001490 con 0x7ff0400c76f0 2024-02-21T23:03:34.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.565+0000 7ff03ca00640 1 -- 172.21.15.28:0/1080819912 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff024003520 con 0x7ff0400c76f0 2024-02-21T23:03:34.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.566+0000 7ff03ca00640 1 -- 172.21.15.28:0/1080819912 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 684752523 0 0) 0x7ff024003520 con 0x7ff0400c76f0 2024-02-21T23:03:34.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.566+0000 7ff03ca00640 1 -- 172.21.15.28:0/1080819912 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff04006dbc0 msgr2=0x7ff04006c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:34.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.566+0000 7ff03ca00640 1 --2- 172.21.15.28:0/1080819912 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff04006dbc0 0x7ff04006c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:34.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.566+0000 7ff03ca00640 1 -- 172.21.15.28:0/1080819912 >> v2:172.21.15.81:3300/0 conn(0x7ff04013e0c0 msgr2=0x7ff0400c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:34.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.566+0000 7ff03ca00640 1 --2- 172.21.15.28:0/1080819912 >> v2:172.21.15.81:3300/0 conn(0x7ff04013e0c0 0x7ff0400c71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:34.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.566+0000 7ff03ca00640 1 -- 172.21.15.28:0/1080819912 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff040158ab0 con 0x7ff0400c76f0 2024-02-21T23:03:34.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.566+0000 7ff03ca00640 1 -- 172.21.15.28:0/1080819912 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7ff038001bc0 con 0x7ff0400c76f0 2024-02-21T23:03:34.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.566+0000 7ff03ca00640 1 -- 172.21.15.28:0/1080819912 >> v1:172.21.15.28:6789/0 conn(0x7ff0400c76f0 legacy=0x7ff04006c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:34.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.566+0000 7ff03ca00640 1 --2- 172.21.15.28:0/1080819912 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0240039b0 0x7ff024003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:34.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.566+0000 7ff03ca00640 1 --2- 172.21.15.28:0/1080819912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff024004800 0x7ff024004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:34.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.566+0000 7ff03ca00640 1 --2- 172.21.15.28:0/1080819912 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff04006dbc0 0x7ff0240051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:34.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.566+0000 7ff03ca00640 1 -- 172.21.15.28:0/1080819912 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0401587d0 con 0x7ff024004800 2024-02-21T23:03:34.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.566+0000 7ff03ca00640 1 -- 172.21.15.28:0/1080819912 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff02400b9c0 con 0x7ff0240039b0 2024-02-21T23:03:34.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.566+0000 7ff03ca00640 1 -- 172.21.15.28:0/1080819912 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff02400bb40 con 0x7ff04006dbc0 2024-02-21T23:03:34.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.566+0000 7ff03d400640 1 --2- 172.21.15.28:0/1080819912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff024004800 0x7ff024004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:34.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.567+0000 7ff03de00640 1 --2- 172.21.15.28:0/1080819912 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0240039b0 0x7ff024003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:34.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.567+0000 7ff03d400640 1 -- 172.21.15.28:0/1080819912 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff04006dbc0 msgr2=0x7ff0240051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:34.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.567+0000 7ff03d400640 1 --2- 172.21.15.28:0/1080819912 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff04006dbc0 0x7ff0240051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:34.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.567+0000 7ff03d400640 1 -- 172.21.15.28:0/1080819912 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0240039b0 msgr2=0x7ff024003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:34.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.567+0000 7ff03d400640 1 --2- 172.21.15.28:0/1080819912 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0240039b0 0x7ff024003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:34.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.567+0000 7ff03d400640 1 -- 172.21.15.28:0/1080819912 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff024003520 con 0x7ff024004800 2024-02-21T23:03:34.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.567+0000 7ff03de00640 1 --2- 172.21.15.28:0/1080819912 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0240039b0 0x7ff024003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:03:34.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.568+0000 7ff03d400640 1 --2- 172.21.15.28:0/1080819912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff024004800 0x7ff024004c80 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff02c009af0 tx=0x7ff024005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1d375539c2c683b server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:34.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.568+0000 7ff03ca00640 1 -- 172.21.15.28:0/1080819912 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff02c039070 con 0x7ff024004800 2024-02-21T23:03:34.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.568+0000 7ff03ca00640 1 -- 172.21.15.28:0/1080819912 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff02c02c710 con 0x7ff024004800 2024-02-21T23:03:34.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.568+0000 7ff03ca00640 1 -- 172.21.15.28:0/1080819912 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff02c009690 con 0x7ff024004800 2024-02-21T23:03:34.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.569+0000 7ff03f200640 1 -- 172.21.15.28:0/1080819912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff024004800 msgr2=0x7ff024004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:34.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.569+0000 7ff03f200640 1 --2- 172.21.15.28:0/1080819912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff024004800 0x7ff024004c80 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff02c009af0 tx=0x7ff024005830 comp rx=0 tx=0).stop 2024-02-21T23:03:34.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.569+0000 7ff03e800640 1 -- 172.21.15.28:0/1080819912 reap_dead start 2024-02-21T23:03:34.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.569+0000 7ff03f200640 1 -- 172.21.15.28:0/1080819912 shutdown_connections 2024-02-21T23:03:34.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.569+0000 7ff03f200640 1 -- 172.21.15.28:0/1080819912 >> 172.21.15.28:0/1080819912 conn(0x7ff040073110 msgr2=0x7ff0400ac490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:34.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.569+0000 7ff03f200640 1 -- 172.21.15.28:0/1080819912 shutdown_connections 2024-02-21T23:03:34.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.569+0000 7ff03f200640 1 -- 172.21.15.28:0/1080819912 wait complete. 2024-02-21T23:03:34.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.571+0000 7ff03f200640 1 Processor -- start 2024-02-21T23:03:34.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.571+0000 7ff03f200640 1 -- start start 2024-02-21T23:03:34.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.572+0000 7ff03f200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff024004800 0x7ff040151ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:34.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.572+0000 7ff03f200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff0401df390 0x7ff0400c9a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:34.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.572+0000 7ff03f200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0401ee8d0 0x7ff0400af2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:34.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.572+0000 7ff03f200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff04006ced0 con 0x7ff0401ee8d0 2024-02-21T23:03:34.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.572+0000 7ff03f200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff040159070 con 0x7ff024004800 2024-02-21T23:03:34.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.572+0000 7ff03f200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff04013e520 con 0x7ff0401df390 2024-02-21T23:03:34.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.572+0000 7ff03e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0401ee8d0 0x7ff0400af2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:34.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.572+0000 7ff03e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0401ee8d0 0x7ff0400af2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45242/0 (socket says 172.21.15.28:45242) 2024-02-21T23:03:34.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.572+0000 7ff03e800640 1 -- 172.21.15.28:0/3333603739 learned_addr learned my addr 172.21.15.28:0/3333603739 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:34.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.573+0000 7ff03d400640 1 --2- 172.21.15.28:0/3333603739 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff0401df390 0x7ff0400c9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:34.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.573+0000 7ff03de00640 1 --2- 172.21.15.28:0/3333603739 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff024004800 0x7ff040151ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:34.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.573+0000 7ff03e800640 1 -- 172.21.15.28:0/3333603739 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff0401df390 msgr2=0x7ff0400c9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:34.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.573+0000 7ff03e800640 1 --2- 172.21.15.28:0/3333603739 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff0401df390 0x7ff0400c9a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:34.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.573+0000 7ff03e800640 1 -- 172.21.15.28:0/3333603739 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff024004800 msgr2=0x7ff040151ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:34.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.573+0000 7ff03e800640 1 --2- 172.21.15.28:0/3333603739 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff024004800 0x7ff040151ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:34.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.573+0000 7ff03e800640 1 -- 172.21.15.28:0/3333603739 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0400ad720 con 0x7ff0401ee8d0 2024-02-21T23:03:34.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.573+0000 7ff03e800640 1 --2- 172.21.15.28:0/3333603739 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0401ee8d0 0x7ff0400af2d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7ff0400ad2a0 tx=0x7ff03800cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:34.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.573+0000 7ff036000640 1 -- 172.21.15.28:0/3333603739 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff03800ce50 con 0x7ff0401ee8d0 2024-02-21T23:03:34.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.573+0000 7ff036000640 1 -- 172.21.15.28:0/3333603739 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff03800a610 con 0x7ff0401ee8d0 2024-02-21T23:03:34.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.574+0000 7ff03f200640 1 -- 172.21.15.28:0/3333603739 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0400ada00 con 0x7ff0401ee8d0 2024-02-21T23:03:34.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.574+0000 7ff036000640 1 -- 172.21.15.28:0/3333603739 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff03800aa00 con 0x7ff0401ee8d0 2024-02-21T23:03:34.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.574+0000 7ff03f200640 1 -- 172.21.15.28:0/3333603739 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0401398e0 con 0x7ff0401ee8d0 2024-02-21T23:03:34.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.575+0000 7ff03f200640 1 -- 172.21.15.28:0/3333603739 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff04004f290 con 0x7ff0401ee8d0 2024-02-21T23:03:34.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.576+0000 7ff036000640 1 -- 172.21.15.28:0/3333603739 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff03800aba0 con 0x7ff0401ee8d0 2024-02-21T23:03:34.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.577+0000 7ff036000640 1 --2- 172.21.15.28:0/3333603739 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff02006b4d0 0x7ff02006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:34.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.577+0000 7ff036000640 1 -- 172.21.15.28:0/3333603739 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6735+0+0 (secure 0 0 0) 0x7ff03808a340 con 0x7ff0401ee8d0 2024-02-21T23:03:34.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.577+0000 7ff03de00640 1 --2- 172.21.15.28:0/3333603739 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff02006b4d0 0x7ff02006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:34.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.579+0000 7ff03de00640 1 --2- 172.21.15.28:0/3333603739 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff02006b4d0 0x7ff02006d990 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ff0401560f0 tx=0x7ff030008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:34.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.581+0000 7ff036000640 1 -- 172.21.15.28:0/3333603739 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ff040158550 con 0x7ff0401ee8d0 2024-02-21T23:03:34.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.783+0000 7ff03f200640 1 -- 172.21.15.28:0/3333603739 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7ff04006d9a0 con 0x7ff0401ee8d0 2024-02-21T23:03:34.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.784+0000 7ff036000640 1 -- 172.21.15.28:0/3333603739 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) v1 ==== 103+0+11 (secure 0 0 0) 0x7ff038010300 con 0x7ff0401ee8d0 2024-02-21T23:03:34.773 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-02-21T23:03:34.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.787+0000 7ff03f200640 1 -- 172.21.15.28:0/3333603739 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff02006b4d0 msgr2=0x7ff02006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:34.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.787+0000 7ff03f200640 1 --2- 172.21.15.28:0/3333603739 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff02006b4d0 0x7ff02006d990 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ff0401560f0 tx=0x7ff030008040 comp rx=0 tx=0).stop 2024-02-21T23:03:34.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.787+0000 7ff03f200640 1 -- 172.21.15.28:0/3333603739 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0401ee8d0 msgr2=0x7ff0400af2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:34.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.787+0000 7ff03f200640 1 --2- 172.21.15.28:0/3333603739 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0401ee8d0 0x7ff0400af2d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7ff0400ad2a0 tx=0x7ff03800cc00 comp rx=0 tx=0).stop 2024-02-21T23:03:34.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.787+0000 7ff03f200640 1 -- 172.21.15.28:0/3333603739 shutdown_connections 2024-02-21T23:03:34.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.787+0000 7ff03f200640 1 --2- 172.21.15.28:0/3333603739 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff02006b4d0 0x7ff02006d990 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:34.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.787+0000 7ff03f200640 1 --2- 172.21.15.28:0/3333603739 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff0401df390 0x7ff0400c9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:34.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.787+0000 7ff03f200640 1 --2- 172.21.15.28:0/3333603739 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0401ee8d0 0x7ff0400af2d0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:34.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.787+0000 7ff03f200640 1 --2- 172.21.15.28:0/3333603739 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff024004800 0x7ff040151ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:34.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.787+0000 7ff03f200640 1 -- 172.21.15.28:0/3333603739 >> 172.21.15.28:0/3333603739 conn(0x7ff040073110 msgr2=0x7ff04013d250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:34.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.788+0000 7ff03f200640 1 -- 172.21.15.28:0/3333603739 shutdown_connections 2024-02-21T23:03:34.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:34.788+0000 7ff03f200640 1 -- 172.21.15.28:0/3333603739 wait complete. 2024-02-21T23:03:34.794 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:03:35.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.028+0000 7f8b7be00640 1 Processor -- start 2024-02-21T23:03:35.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.028+0000 7f8b7be00640 1 -- start start 2024-02-21T23:03:35.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.029+0000 7f8b7be00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f8b7c06dff0 0x7f8b7c06e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:35.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.029+0000 7f8b7be00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8b7c0c0fe0 0x7f8b7c071350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:35.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.029+0000 7f8b7be00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b7c0a8b40 con 0x7f8b7c158580 2024-02-21T23:03:35.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.029+0000 7f8b7be00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b7c0a8e00 con 0x7f8b7c06dff0 2024-02-21T23:03:35.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.029+0000 7f8b7be00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b7c0a90e0 con 0x7f8b7c0c0fe0 2024-02-21T23:03:35.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.030+0000 7f8b7aa00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f8b7c06dff0 0x7f8b7c06e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:35.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.030+0000 7f8b7a000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8b7c0c0fe0 0x7f8b7c071350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:35.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.030+0000 7f8b7a000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8b7c0c0fe0 0x7f8b7c071350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3301/0 says I am v2:172.21.15.28:45922/0 (socket says 172.21.15.28:45922) 2024-02-21T23:03:35.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.030+0000 7f8b7aa00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f8b7c06dff0 0x7f8b7c06e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.28:53354/0 (socket says 172.21.15.28:53354) 2024-02-21T23:03:35.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.030+0000 7f8b7a000640 1 -- 172.21.15.28:0/1728281049 learned_addr learned my addr 172.21.15.28:0/1728281049 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:35.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.031+0000 7f8b7a000640 1 -- 172.21.15.28:0/1728281049 >> v2:172.21.15.81:3300/0 conn(0x7f8b7c06dff0 msgr2=0x7f8b7c06e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:35.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.031+0000 7f8b7a000640 1 --2- 172.21.15.28:0/1728281049 >> v2:172.21.15.81:3300/0 conn(0x7f8b7c06dff0 0x7f8b7c06e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:35.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.031+0000 7f8b7a000640 1 -- 172.21.15.28:0/1728281049 >> v1:172.21.15.81:6789/0 conn(0x7f8b7c158580 legacy=0x7f8b7c14b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:35.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.031+0000 7f8b7a000640 1 -- 172.21.15.28:0/1728281049 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b7c072ed0 con 0x7f8b7c0c0fe0 2024-02-21T23:03:35.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.031+0000 7f8b7aa00640 1 --2- 172.21.15.28:0/1728281049 >> v2:172.21.15.81:3300/0 conn(0x7f8b7c06dff0 0x7f8b7c06e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-21T23:03:35.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.032+0000 7f8b7a000640 1 --2- 172.21.15.28:0/1728281049 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8b7c0c0fe0 0x7f8b7c071350 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8b68009870 tx=0x7f8b6802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=710681e3005a70cc server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:35.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.033+0000 7f8b79600640 1 -- 172.21.15.28:0/1728281049 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b68034070 con 0x7f8b7c0c0fe0 2024-02-21T23:03:35.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.033+0000 7f8b79600640 1 -- 172.21.15.28:0/1728281049 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b68030840 con 0x7f8b7c0c0fe0 2024-02-21T23:03:35.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.033+0000 7f8b79600640 1 -- 172.21.15.28:0/1728281049 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b68030c10 con 0x7f8b7c0c0fe0 2024-02-21T23:03:35.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.033+0000 7f8b7be00640 1 -- 172.21.15.28:0/1728281049 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8b7c0c0fe0 msgr2=0x7f8b7c071350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:35.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.033+0000 7f8b7be00640 1 --2- 172.21.15.28:0/1728281049 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8b7c0c0fe0 0x7f8b7c071350 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8b68009870 tx=0x7f8b6802b980 comp rx=0 tx=0).stop 2024-02-21T23:03:35.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.034+0000 7f8b7be00640 1 -- 172.21.15.28:0/1728281049 shutdown_connections 2024-02-21T23:03:35.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.034+0000 7f8b7be00640 1 --2- 172.21.15.28:0/1728281049 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8b7c0c0fe0 0x7f8b7c071350 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:35.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.034+0000 7f8b7be00640 1 --2- 172.21.15.28:0/1728281049 >> v2:172.21.15.81:3300/0 conn(0x7f8b7c06dff0 0x7f8b7c06e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:35.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.034+0000 7f8b7be00640 1 -- 172.21.15.28:0/1728281049 >> 172.21.15.28:0/1728281049 conn(0x7f8b7c06b630 msgr2=0x7f8b7c0c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:35.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.034+0000 7f8b7be00640 1 -- 172.21.15.28:0/1728281049 shutdown_connections 2024-02-21T23:03:35.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.034+0000 7f8b7be00640 1 -- 172.21.15.28:0/1728281049 wait complete. 2024-02-21T23:03:35.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.035+0000 7f8b7be00640 1 Processor -- start 2024-02-21T23:03:35.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.036+0000 7f8b7be00640 1 -- start start 2024-02-21T23:03:35.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.036+0000 7f8b7be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b7c06dff0 0x7f8b7c1ee7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:35.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.036+0000 7f8b7be00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b7c0c0fe0 0x7f8b7c1ea790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:35.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.036+0000 7f8b7be00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8b7c158580 0x7f8b7c141210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:35.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.036+0000 7f8b7be00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b7c071890 con 0x7f8b7c06dff0 2024-02-21T23:03:35.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.036+0000 7f8b7be00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b7c073520 con 0x7f8b7c0c0fe0 2024-02-21T23:03:35.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.036+0000 7f8b7be00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b7c158a10 con 0x7f8b7c158580 2024-02-21T23:03:35.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.037+0000 7f8b7aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b7c06dff0 0x7f8b7c1ee7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:35.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.037+0000 7f8b7aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b7c06dff0 0x7f8b7c1ee7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44272/0 (socket says 172.21.15.28:44272) 2024-02-21T23:03:35.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.037+0000 7f8b7aa00640 1 -- 172.21.15.28:0/2780434062 learned_addr learned my addr 172.21.15.28:0/2780434062 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:35.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.037+0000 7f8b7a000640 1 --2- 172.21.15.28:0/2780434062 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b7c0c0fe0 0x7f8b7c1ea790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:35.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.037+0000 7f8b7b400640 1 --2- 172.21.15.28:0/2780434062 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8b7c158580 0x7f8b7c141210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:35.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.037+0000 7f8b7aa00640 1 -- 172.21.15.28:0/2780434062 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8b7c158580 msgr2=0x7f8b7c141210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:35.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.037+0000 7f8b7aa00640 1 --2- 172.21.15.28:0/2780434062 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8b7c158580 0x7f8b7c141210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:35.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.037+0000 7f8b7aa00640 1 -- 172.21.15.28:0/2780434062 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b7c0c0fe0 msgr2=0x7f8b7c1ea790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:35.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.037+0000 7f8b7aa00640 1 --2- 172.21.15.28:0/2780434062 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b7c0c0fe0 0x7f8b7c1ea790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:35.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.037+0000 7f8b7aa00640 1 -- 172.21.15.28:0/2780434062 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b7c155da0 con 0x7f8b7c06dff0 2024-02-21T23:03:35.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.037+0000 7f8b7b400640 1 --2- 172.21.15.28:0/2780434062 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8b7c158580 0x7f8b7c141210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:03:35.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.038+0000 7f8b7aa00640 1 --2- 172.21.15.28:0/2780434062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b7c06dff0 0x7f8b7c1ee7d0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f8b6c00e960 tx=0x7f8b6c00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:35.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.038+0000 7f8b73400640 1 -- 172.21.15.28:0/2780434062 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b6c017070 con 0x7f8b7c06dff0 2024-02-21T23:03:35.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.038+0000 7f8b73400640 1 -- 172.21.15.28:0/2780434062 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b6c00ca00 con 0x7f8b7c06dff0 2024-02-21T23:03:35.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.038+0000 7f8b73400640 1 -- 172.21.15.28:0/2780434062 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b6c00cdd0 con 0x7f8b7c06dff0 2024-02-21T23:03:35.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.038+0000 7f8b7be00640 1 -- 172.21.15.28:0/2780434062 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b7c156080 con 0x7f8b7c06dff0 2024-02-21T23:03:35.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.038+0000 7f8b7be00640 1 -- 172.21.15.28:0/2780434062 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b7c152790 con 0x7f8b7c06dff0 2024-02-21T23:03:35.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.039+0000 7f8b7be00640 1 -- 172.21.15.28:0/2780434062 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b7c04f290 con 0x7f8b7c06dff0 2024-02-21T23:03:35.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.040+0000 7f8b73400640 1 -- 172.21.15.28:0/2780434062 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8b6c01a070 con 0x7f8b7c06dff0 2024-02-21T23:03:35.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.041+0000 7f8b73400640 1 --2- 172.21.15.28:0/2780434062 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8b5c06b4d0 0x7f8b5c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:35.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.041+0000 7f8b73400640 1 -- 172.21.15.28:0/2780434062 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f8b6c08a4f0 con 0x7f8b7c06dff0 2024-02-21T23:03:35.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.042+0000 7f8b7a000640 1 --2- 172.21.15.28:0/2780434062 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8b5c06b4d0 0x7f8b5c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:35.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.043+0000 7f8b7a000640 1 --2- 172.21.15.28:0/2780434062 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8b5c06b4d0 0x7f8b5c06d990 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f8b68036000 tx=0x7f8b6803b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:35.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.045+0000 7f8b73400640 1 -- 172.21.15.28:0/2780434062 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f8b6c058f60 con 0x7f8b7c06dff0 2024-02-21T23:03:35.169 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-21T23:03:35.175 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-21T23:03:35.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.253+0000 7f8b7be00640 1 -- 172.21.15.28:0/2780434062 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8b7c06ecb0 con 0x7f8b7c06dff0 2024-02-21T23:03:35.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.254+0000 7f8b73400640 1 -- 172.21.15.28:0/2780434062 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+18843 (secure 0 0 0) 0x7f8b6c058860 con 0x7f8b7c06dff0 2024-02-21T23:03:35.243 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:03:35.243 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":26,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:03:23.860109+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:02:35.795158+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-21T23:02:36.840110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-02-21T23:02:40.441388+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":3,"to":2}]},{"pgid":"3.12","mappings":[{"from":0,"to":1}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:03:35.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.257+0000 7f8b7be00640 1 -- 172.21.15.28:0/2780434062 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8b5c06b4d0 msgr2=0x7f8b5c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:35.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.257+0000 7f8b7be00640 1 --2- 172.21.15.28:0/2780434062 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8b5c06b4d0 0x7f8b5c06d990 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f8b68036000 tx=0x7f8b6803b040 comp rx=0 tx=0).stop 2024-02-21T23:03:35.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.257+0000 7f8b7be00640 1 -- 172.21.15.28:0/2780434062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b7c06dff0 msgr2=0x7f8b7c1ee7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:35.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.257+0000 7f8b7be00640 1 --2- 172.21.15.28:0/2780434062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b7c06dff0 0x7f8b7c1ee7d0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f8b6c00e960 tx=0x7f8b6c00ee20 comp rx=0 tx=0).stop 2024-02-21T23:03:35.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.257+0000 7f8b7be00640 1 -- 172.21.15.28:0/2780434062 shutdown_connections 2024-02-21T23:03:35.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.257+0000 7f8b7be00640 1 --2- 172.21.15.28:0/2780434062 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8b5c06b4d0 0x7f8b5c06d990 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:35.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.257+0000 7f8b7be00640 1 --2- 172.21.15.28:0/2780434062 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8b7c158580 0x7f8b7c141210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:35.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.257+0000 7f8b7be00640 1 --2- 172.21.15.28:0/2780434062 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b7c0c0fe0 0x7f8b7c1ea790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:35.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.257+0000 7f8b7be00640 1 --2- 172.21.15.28:0/2780434062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b7c06dff0 0x7f8b7c1ee7d0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:35.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.257+0000 7f8b7be00640 1 -- 172.21.15.28:0/2780434062 >> 172.21.15.28:0/2780434062 conn(0x7f8b7c06b630 msgr2=0x7f8b7c0c26a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:35.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.257+0000 7f8b7be00640 1 -- 172.21.15.28:0/2780434062 shutdown_connections 2024-02-21T23:03:35.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.257+0000 7f8b7be00640 1 -- 172.21.15.28:0/2780434062 wait complete. 2024-02-21T23:03:35.265 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-21T23:02:30.728329+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-21T23:02:35.795158+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-02-21T23:02:36.840110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-02-21T23:02:40.441388+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-02-21T23:03:35.266 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-21T23:03:35.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.479+0000 7f372a000640 1 Processor -- start 2024-02-21T23:03:35.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.479+0000 7f372a000640 1 -- start start 2024-02-21T23:03:35.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.481+0000 7f372a000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f372413e0c0 0x7f37240c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:35.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.481+0000 7f372a000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3724158550 con 0x7f37240c76f0 2024-02-21T23:03:35.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.481+0000 7f372a000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37241587d0 con 0x7f372406dbc0 2024-02-21T23:03:35.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.481+0000 7f372a000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37240c89a0 con 0x7f372413e0c0 2024-02-21T23:03:35.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.481+0000 7f3729600640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f37240c76f0 0x7f372406c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55086/0 (socket says 172.21.15.28:55086) 2024-02-21T23:03:35.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.481+0000 7f3729600640 1 -- 172.21.15.28:0/533021332 learned_addr learned my addr 172.21.15.28:0/533021332 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:03:35.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.482+0000 7f3728c00640 1 --2- 172.21.15.28:0/533021332 >> v2:172.21.15.81:3300/0 conn(0x7f372413e0c0 0x7f37240c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:35.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.482+0000 7f3723e00640 1 -- 172.21.15.28:0/533021332 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4235059531 0 0) 0x7f371c001490 con 0x7f37240c76f0 2024-02-21T23:03:35.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.482+0000 7f3723e00640 1 -- 172.21.15.28:0/533021332 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3708003520 con 0x7f37240c76f0 2024-02-21T23:03:35.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.482+0000 7f3723e00640 1 -- 172.21.15.28:0/533021332 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2489777135 0 0) 0x7f3708003520 con 0x7f37240c76f0 2024-02-21T23:03:35.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.482+0000 7f3723e00640 1 -- 172.21.15.28:0/533021332 >> v2:172.21.15.81:3300/0 conn(0x7f372413e0c0 msgr2=0x7f37240c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:35.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.482+0000 7f3723e00640 1 --2- 172.21.15.28:0/533021332 >> v2:172.21.15.81:3300/0 conn(0x7f372413e0c0 0x7f37240c71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:35.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.482+0000 7f3723e00640 1 -- 172.21.15.28:0/533021332 >> v1:172.21.15.81:6789/0 conn(0x7f372406dbc0 legacy=0x7f372406c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:35.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.482+0000 7f3723e00640 1 -- 172.21.15.28:0/533021332 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3724158ab0 con 0x7f37240c76f0 2024-02-21T23:03:35.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.482+0000 7f3723e00640 1 -- 172.21.15.28:0/533021332 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f371c001bc0 con 0x7f37240c76f0 2024-02-21T23:03:35.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.482+0000 7f3723e00640 1 -- 172.21.15.28:0/533021332 >> v1:172.21.15.28:6789/0 conn(0x7f37240c76f0 legacy=0x7f372406c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:35.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.483+0000 7f3723e00640 1 --2- 172.21.15.28:0/533021332 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f37080039b0 0x7f3708003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:35.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.483+0000 7f3723e00640 1 --2- 172.21.15.28:0/533021332 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3708004800 0x7f3708004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:35.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.483+0000 7f3723e00640 1 --2- 172.21.15.28:0/533021332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37080052e0 0x7f370800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:35.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.483+0000 7f3723e00640 1 -- 172.21.15.28:0/533021332 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f370800c180 con 0x7f37080052e0 2024-02-21T23:03:35.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.483+0000 7f3723e00640 1 -- 172.21.15.28:0/533021332 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f370800c330 con 0x7f37080039b0 2024-02-21T23:03:35.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.483+0000 7f3723e00640 1 -- 172.21.15.28:0/533021332 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f370800c530 con 0x7f3708004800 2024-02-21T23:03:35.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.483+0000 7f3729600640 1 --2- 172.21.15.28:0/533021332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37080052e0 0x7f370800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:35.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.483+0000 7f3728c00640 1 --2- 172.21.15.28:0/533021332 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f37080039b0 0x7f3708003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:35.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.483+0000 7f3729600640 1 -- 172.21.15.28:0/533021332 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3708004800 msgr2=0x7f3708004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:35.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.483+0000 7f371be00640 1 --2- 172.21.15.28:0/533021332 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3708004800 0x7f3708004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:35.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.483+0000 7f3729600640 1 --2- 172.21.15.28:0/533021332 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3708004800 0x7f3708004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:35.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.483+0000 7f3729600640 1 -- 172.21.15.28:0/533021332 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f37080039b0 msgr2=0x7f3708003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:35.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.483+0000 7f3729600640 1 --2- 172.21.15.28:0/533021332 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f37080039b0 0x7f3708003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:35.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.483+0000 7f3729600640 1 -- 172.21.15.28:0/533021332 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3708003520 con 0x7f37080052e0 2024-02-21T23:03:35.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.483+0000 7f371be00640 1 --2- 172.21.15.28:0/533021332 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3708004800 0x7f3708004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:03:35.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.483+0000 7f3728c00640 1 --2- 172.21.15.28:0/533021332 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f37080039b0 0x7f3708003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:03:35.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.484+0000 7f3729600640 1 -- 172.21.15.28:0/533021332 reap_dead start 2024-02-21T23:03:35.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.485+0000 7f3729600640 1 --2- 172.21.15.28:0/533021332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37080052e0 0x7f370800b9c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f371c00aa20 tx=0x7f3708005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=337e3de37c2a0661 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:35.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.485+0000 7f3723e00640 1 -- 172.21.15.28:0/533021332 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f371c038070 con 0x7f37080052e0 2024-02-21T23:03:35.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.485+0000 7f3723e00640 1 -- 172.21.15.28:0/533021332 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f371c033620 con 0x7f37080052e0 2024-02-21T23:03:35.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.485+0000 7f3723e00640 1 -- 172.21.15.28:0/533021332 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f371c00a670 con 0x7f37080052e0 2024-02-21T23:03:35.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.485+0000 7f372a000640 1 -- 172.21.15.28:0/533021332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37080052e0 msgr2=0x7f370800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:35.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.485+0000 7f372a000640 1 --2- 172.21.15.28:0/533021332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37080052e0 0x7f370800b9c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f371c00aa20 tx=0x7f3708005830 comp rx=0 tx=0).stop 2024-02-21T23:03:35.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.486+0000 7f372a000640 1 -- 172.21.15.28:0/533021332 shutdown_connections 2024-02-21T23:03:35.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.486+0000 7f372a000640 1 --2- 172.21.15.28:0/533021332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37080052e0 0x7f370800b9c0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:35.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.486+0000 7f372a000640 1 -- 172.21.15.28:0/533021332 >> 172.21.15.28:0/533021332 conn(0x7f3724073110 msgr2=0x7f37240ac490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:35.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.486+0000 7f372a000640 1 -- 172.21.15.28:0/533021332 shutdown_connections 2024-02-21T23:03:35.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.486+0000 7f372a000640 1 -- 172.21.15.28:0/533021332 wait complete. 2024-02-21T23:03:35.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.487+0000 7f372a000640 1 Processor -- start 2024-02-21T23:03:35.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.488+0000 7f372a000640 1 -- start start 2024-02-21T23:03:35.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.488+0000 7f372a000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f37080052e0 0x7f37240afed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:35.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.488+0000 7f372a000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f37241ede30 0x7f3724152050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:35.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.488+0000 7f372a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3724152590 0x7f37240aad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:35.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.488+0000 7f372a000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f372413e520 con 0x7f3724152590 2024-02-21T23:03:35.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.488+0000 7f372a000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3724159070 con 0x7f37080052e0 2024-02-21T23:03:35.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.488+0000 7f372a000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f372413e6a0 con 0x7f37241ede30 2024-02-21T23:03:35.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.489+0000 7f3729600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3724152590 0x7f37240aad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:35.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.489+0000 7f3729600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3724152590 0x7f37240aad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44300/0 (socket says 172.21.15.28:44300) 2024-02-21T23:03:35.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.489+0000 7f3729600640 1 -- 172.21.15.28:0/1651530550 learned_addr learned my addr 172.21.15.28:0/1651530550 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:35.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.489+0000 7f3728c00640 1 --2- 172.21.15.28:0/1651530550 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f37080052e0 0x7f37240afed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:35.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.489+0000 7f371be00640 1 --2- 172.21.15.28:0/1651530550 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f37241ede30 0x7f3724152050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:35.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.489+0000 7f3729600640 1 -- 172.21.15.28:0/1651530550 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f37241ede30 msgr2=0x7f3724152050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:35.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.489+0000 7f3729600640 1 --2- 172.21.15.28:0/1651530550 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f37241ede30 0x7f3724152050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:35.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.489+0000 7f3729600640 1 -- 172.21.15.28:0/1651530550 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f37080052e0 msgr2=0x7f37240afed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:35.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.489+0000 7f3729600640 1 --2- 172.21.15.28:0/1651530550 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f37080052e0 0x7f37240afed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:35.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.489+0000 7f3729600640 1 -- 172.21.15.28:0/1651530550 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37241ea8c0 con 0x7f3724152590 2024-02-21T23:03:35.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.490+0000 7f3729600640 1 --2- 172.21.15.28:0/1651530550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3724152590 0x7f37240aad40 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f371c036040 tx=0x7f371c001f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:35.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.490+0000 7f3721600640 1 -- 172.21.15.28:0/1651530550 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f371c045070 con 0x7f3724152590 2024-02-21T23:03:35.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.490+0000 7f3721600640 1 -- 172.21.15.28:0/1651530550 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f371c001b80 con 0x7f3724152590 2024-02-21T23:03:35.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.490+0000 7f3721600640 1 -- 172.21.15.28:0/1651530550 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f371c038040 con 0x7f3724152590 2024-02-21T23:03:35.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.490+0000 7f372a000640 1 -- 172.21.15.28:0/1651530550 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37241eab80 con 0x7f3724152590 2024-02-21T23:03:35.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.490+0000 7f372a000640 1 -- 172.21.15.28:0/1651530550 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37241eb140 con 0x7f3724152590 2024-02-21T23:03:35.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.491+0000 7f372a000640 1 -- 172.21.15.28:0/1651530550 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37240c7130 con 0x7f3724152590 2024-02-21T23:03:35.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.492+0000 7f3721600640 1 -- 172.21.15.28:0/1651530550 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f371c048070 con 0x7f3724152590 2024-02-21T23:03:35.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.493+0000 7f3721600640 1 --2- 172.21.15.28:0/1651530550 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f370006b4d0 0x7f370006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:35.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.493+0000 7f3721600640 1 -- 172.21.15.28:0/1651530550 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f371c0ab5e0 con 0x7f3724152590 2024-02-21T23:03:35.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.494+0000 7f3728c00640 1 --2- 172.21.15.28:0/1651530550 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f370006b4d0 0x7f370006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:35.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.495+0000 7f3728c00640 1 --2- 172.21.15.28:0/1651530550 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f370006b4d0 0x7f370006d990 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3714009990 tx=0x7f3714008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:35.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.497+0000 7f3721600640 1 -- 172.21.15.28:0/1651530550 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3724158550 con 0x7f3724152590 2024-02-21T23:03:35.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.696+0000 7f372a000640 1 -- 172.21.15.28:0/1651530550 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f372404f290 con 0x7f3724152590 2024-02-21T23:03:35.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.697+0000 7f3721600640 1 -- 172.21.15.28:0/1651530550 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) v1 ==== 93+0+10 (secure 0 0 0) 0x7f371c079a00 con 0x7f3724152590 2024-02-21T23:03:35.686 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-02-21T23:03:35.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.700+0000 7f372a000640 1 -- 172.21.15.28:0/1651530550 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f370006b4d0 msgr2=0x7f370006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:35.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.700+0000 7f372a000640 1 --2- 172.21.15.28:0/1651530550 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f370006b4d0 0x7f370006d990 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3714009990 tx=0x7f3714008040 comp rx=0 tx=0).stop 2024-02-21T23:03:35.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.700+0000 7f372a000640 1 -- 172.21.15.28:0/1651530550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3724152590 msgr2=0x7f37240aad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:35.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.700+0000 7f372a000640 1 --2- 172.21.15.28:0/1651530550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3724152590 0x7f37240aad40 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f371c036040 tx=0x7f371c001f90 comp rx=0 tx=0).stop 2024-02-21T23:03:35.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.700+0000 7f372a000640 1 -- 172.21.15.28:0/1651530550 shutdown_connections 2024-02-21T23:03:35.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.700+0000 7f372a000640 1 --2- 172.21.15.28:0/1651530550 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f370006b4d0 0x7f370006d990 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:35.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.700+0000 7f372a000640 1 --2- 172.21.15.28:0/1651530550 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f37241ede30 0x7f3724152050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:35.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.700+0000 7f372a000640 1 --2- 172.21.15.28:0/1651530550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3724152590 0x7f37240aad40 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:35.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.700+0000 7f372a000640 1 --2- 172.21.15.28:0/1651530550 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f37080052e0 0x7f37240afed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:35.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.700+0000 7f372a000640 1 -- 172.21.15.28:0/1651530550 >> 172.21.15.28:0/1651530550 conn(0x7f3724073110 msgr2=0x7f37240c5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:35.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.700+0000 7f372a000640 1 -- 172.21.15.28:0/1651530550 shutdown_connections 2024-02-21T23:03:35.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.700+0000 7f372a000640 1 -- 172.21.15.28:0/1651530550 wait complete. 2024-02-21T23:03:35.709 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-21T23:03:35.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.953+0000 7f8129a00640 1 Processor -- start 2024-02-21T23:03:35.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.953+0000 7f8129a00640 1 -- start start 2024-02-21T23:03:35.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.954+0000 7f8129a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f81240c6220 0x7f81241586b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:35.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.954+0000 7f8129a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8124158bf0 0x7f812415f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:35.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.954+0000 7f8129a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81240c20d0 con 0x7f81240c4960 2024-02-21T23:03:35.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.954+0000 7f8129a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81240a62e0 con 0x7f8124158bf0 2024-02-21T23:03:35.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.954+0000 7f8129a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81240c1960 con 0x7f81240c6220 2024-02-21T23:03:35.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.955+0000 7f8123400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8124158bf0 0x7f812415f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:35.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.955+0000 7f8123400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8124158bf0 0x7f812415f9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44308/0 (socket says 172.21.15.28:44308) 2024-02-21T23:03:35.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.955+0000 7f8123400640 1 -- 172.21.15.28:0/141592945 learned_addr learned my addr 172.21.15.28:0/141592945 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:35.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.955+0000 7f8122000640 1 --2- 172.21.15.28:0/141592945 >> v2:172.21.15.81:3300/0 conn(0x7f81240c6220 0x7f81241586b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:35.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.955+0000 7f8123400640 1 -- 172.21.15.28:0/141592945 >> v2:172.21.15.81:3300/0 conn(0x7f81240c6220 msgr2=0x7f81241586b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:35.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.955+0000 7f8123400640 1 --2- 172.21.15.28:0/141592945 >> v2:172.21.15.81:3300/0 conn(0x7f81240c6220 0x7f81241586b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:35.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.955+0000 7f8123400640 1 -- 172.21.15.28:0/141592945 >> v1:172.21.15.28:6789/0 conn(0x7f81240c4960 legacy=0x7f81241524f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:35.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.955+0000 7f8123400640 1 -- 172.21.15.28:0/141592945 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81240c1290 con 0x7f8124158bf0 2024-02-21T23:03:35.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.957+0000 7f8123400640 1 --2- 172.21.15.28:0/141592945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8124158bf0 0x7f812415f9e0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f81240701a0 tx=0x7f811c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a863748c91665007 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:35.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.957+0000 7f8121600640 1 -- 172.21.15.28:0/141592945 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f811c02ca80 con 0x7f8124158bf0 2024-02-21T23:03:35.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.957+0000 7f8121600640 1 -- 172.21.15.28:0/141592945 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f811c02cc20 con 0x7f8124158bf0 2024-02-21T23:03:35.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.957+0000 7f8121600640 1 -- 172.21.15.28:0/141592945 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f811c0343f0 con 0x7f8124158bf0 2024-02-21T23:03:35.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.958+0000 7f8129a00640 1 -- 172.21.15.28:0/141592945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8124158bf0 msgr2=0x7f812415f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:35.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.958+0000 7f8129a00640 1 --2- 172.21.15.28:0/141592945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8124158bf0 0x7f812415f9e0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f81240701a0 tx=0x7f811c02bee0 comp rx=0 tx=0).stop 2024-02-21T23:03:35.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.958+0000 7f8129a00640 1 -- 172.21.15.28:0/141592945 shutdown_connections 2024-02-21T23:03:35.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.958+0000 7f8129a00640 1 --2- 172.21.15.28:0/141592945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8124158bf0 0x7f812415f9e0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:35.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.958+0000 7f8129a00640 1 --2- 172.21.15.28:0/141592945 >> v2:172.21.15.81:3300/0 conn(0x7f81240c6220 0x7f81241586b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:35.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.958+0000 7f8129a00640 1 -- 172.21.15.28:0/141592945 >> 172.21.15.28:0/141592945 conn(0x7f81240a85a0 msgr2=0x7f81240a8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:35.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.958+0000 7f8129a00640 1 -- 172.21.15.28:0/141592945 shutdown_connections 2024-02-21T23:03:35.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.958+0000 7f8129a00640 1 -- 172.21.15.28:0/141592945 wait complete. 2024-02-21T23:03:35.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.960+0000 7f8129a00640 1 Processor -- start 2024-02-21T23:03:35.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.960+0000 7f8129a00640 1 -- start start 2024-02-21T23:03:35.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.961+0000 7f8129a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81240c4960 0x7f8124150670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:35.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.961+0000 7f8129a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f81240c6220 0x7f812413d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:35.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.961+0000 7f8129a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8124158bf0 0x7f812413ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:35.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.961+0000 7f8129a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81240a5f20 con 0x7f8124158bf0 2024-02-21T23:03:35.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.961+0000 7f8129a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81240c1bb0 con 0x7f81240c4960 2024-02-21T23:03:35.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.961+0000 7f8129a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81240a60a0 con 0x7f81240c6220 2024-02-21T23:03:35.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.961+0000 7f8123400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8124158bf0 0x7f812413ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:35.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.961+0000 7f8123400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8124158bf0 0x7f812413ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44314/0 (socket says 172.21.15.28:44314) 2024-02-21T23:03:35.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.961+0000 7f8123400640 1 -- 172.21.15.28:0/2156931366 learned_addr learned my addr 172.21.15.28:0/2156931366 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:35.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.961+0000 7f8122a00640 1 --2- 172.21.15.28:0/2156931366 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81240c4960 0x7f8124150670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:35.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.961+0000 7f8122000640 1 --2- 172.21.15.28:0/2156931366 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f81240c6220 0x7f812413d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:35.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.962+0000 7f8123400640 1 -- 172.21.15.28:0/2156931366 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f81240c6220 msgr2=0x7f812413d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:35.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.962+0000 7f8123400640 1 --2- 172.21.15.28:0/2156931366 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f81240c6220 0x7f812413d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:35.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.962+0000 7f8123400640 1 -- 172.21.15.28:0/2156931366 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81240c4960 msgr2=0x7f8124150670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:35.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.962+0000 7f8123400640 1 --2- 172.21.15.28:0/2156931366 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81240c4960 0x7f8124150670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:35.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.962+0000 7f8123400640 1 -- 172.21.15.28:0/2156931366 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81241575b0 con 0x7f8124158bf0 2024-02-21T23:03:35.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.962+0000 7f8123400640 1 --2- 172.21.15.28:0/2156931366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8124158bf0 0x7f812413ddc0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f811c036040 tx=0x7f811c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:35.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.962+0000 7f811aa00640 1 -- 172.21.15.28:0/2156931366 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f811c042020 con 0x7f8124158bf0 2024-02-21T23:03:35.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.962+0000 7f811aa00640 1 -- 172.21.15.28:0/2156931366 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f811c03e450 con 0x7f8124158bf0 2024-02-21T23:03:35.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.962+0000 7f811aa00640 1 -- 172.21.15.28:0/2156931366 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f811c009510 con 0x7f8124158bf0 2024-02-21T23:03:35.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.962+0000 7f8129a00640 1 -- 172.21.15.28:0/2156931366 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81241eed00 con 0x7f8124158bf0 2024-02-21T23:03:35.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.963+0000 7f8129a00640 1 -- 172.21.15.28:0/2156931366 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8124150bb0 con 0x7f8124158bf0 2024-02-21T23:03:35.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.964+0000 7f8129a00640 1 -- 172.21.15.28:0/2156931366 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8124067f30 con 0x7f8124158bf0 2024-02-21T23:03:35.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.965+0000 7f811aa00640 1 -- 172.21.15.28:0/2156931366 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f811c045020 con 0x7f8124158bf0 2024-02-21T23:03:35.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.966+0000 7f811aa00640 1 --2- 172.21.15.28:0/2156931366 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f810006b4d0 0x7f810006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:35.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.966+0000 7f811aa00640 1 -- 172.21.15.28:0/2156931366 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f811c0acd50 con 0x7f8124158bf0 2024-02-21T23:03:35.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.966+0000 7f8122a00640 1 --2- 172.21.15.28:0/2156931366 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f810006b4d0 0x7f810006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:35.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.967+0000 7f8122a00640 1 --2- 172.21.15.28:0/2156931366 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f810006b4d0 0x7f810006d990 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f810c001640 tx=0x7f810c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:35.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:35.970+0000 7f811aa00640 1 -- 172.21.15.28:0/2156931366 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f811c04c030 con 0x7f8124158bf0 2024-02-21T23:03:36.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.172+0000 7f8129a00640 1 -- 172.21.15.28:0/2156931366 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f81241590a0 con 0x7f8124158bf0 2024-02-21T23:03:36.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.172+0000 7f811aa00640 1 -- 172.21.15.28:0/2156931366 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+11 (secure 0 0 0) 0x7f811c07b7c0 con 0x7f8124158bf0 2024-02-21T23:03:36.161 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-02-21T23:03:36.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.175+0000 7f8129a00640 1 -- 172.21.15.28:0/2156931366 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f810006b4d0 msgr2=0x7f810006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:36.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.175+0000 7f8129a00640 1 --2- 172.21.15.28:0/2156931366 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f810006b4d0 0x7f810006d990 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f810c001640 tx=0x7f810c009040 comp rx=0 tx=0).stop 2024-02-21T23:03:36.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.175+0000 7f8129a00640 1 -- 172.21.15.28:0/2156931366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8124158bf0 msgr2=0x7f812413ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:36.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.175+0000 7f8129a00640 1 --2- 172.21.15.28:0/2156931366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8124158bf0 0x7f812413ddc0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f811c036040 tx=0x7f811c004a40 comp rx=0 tx=0).stop 2024-02-21T23:03:36.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.176+0000 7f8129a00640 1 -- 172.21.15.28:0/2156931366 shutdown_connections 2024-02-21T23:03:36.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.176+0000 7f8129a00640 1 --2- 172.21.15.28:0/2156931366 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f810006b4d0 0x7f810006d990 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:36.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.176+0000 7f8129a00640 1 --2- 172.21.15.28:0/2156931366 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f81240c6220 0x7f812413d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:36.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.176+0000 7f8129a00640 1 --2- 172.21.15.28:0/2156931366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8124158bf0 0x7f812413ddc0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:36.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.176+0000 7f8129a00640 1 --2- 172.21.15.28:0/2156931366 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81240c4960 0x7f8124150670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:36.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.176+0000 7f8129a00640 1 -- 172.21.15.28:0/2156931366 >> 172.21.15.28:0/2156931366 conn(0x7f81240a85a0 msgr2=0x7f81240c5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:36.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.176+0000 7f8129a00640 1 -- 172.21.15.28:0/2156931366 shutdown_connections 2024-02-21T23:03:36.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.176+0000 7f8129a00640 1 -- 172.21.15.28:0/2156931366 wait complete. 2024-02-21T23:03:36.182 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-21T23:03:36.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.408+0000 7f7cf2e00640 1 Processor -- start 2024-02-21T23:03:36.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.409+0000 7f7cf2e00640 1 -- start start 2024-02-21T23:03:36.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.410+0000 7f7cf2e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7cec06daa0 0x7f7cec0c8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:36.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.410+0000 7f7cf2e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f7cec1550e0 0x7f7cec155550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:36.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.410+0000 7f7cf2e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cec158920 con 0x7f7cec152650 2024-02-21T23:03:36.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.410+0000 7f7cf2e00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cec1586d0 con 0x7f7cec1550e0 2024-02-21T23:03:36.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.410+0000 7f7cf2e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cec158c00 con 0x7f7cec06daa0 2024-02-21T23:03:36.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.411+0000 7f7cf1a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f7cec152650 0x7f7cec0c7160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55110/0 (socket says 172.21.15.28:55110) 2024-02-21T23:03:36.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.411+0000 7f7cf1a00640 1 -- 172.21.15.28:0/1194681233 learned_addr learned my addr 172.21.15.28:0/1194681233 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:03:36.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.411+0000 7f7cebe00640 1 -- 172.21.15.28:0/1194681233 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3340899502 0 0) 0x7f7cec158920 con 0x7f7cec152650 2024-02-21T23:03:36.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.411+0000 7f7cebe00640 1 -- 172.21.15.28:0/1194681233 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cd4003520 con 0x7f7cec152650 2024-02-21T23:03:36.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.411+0000 7f7cf2400640 1 --2- 172.21.15.28:0/1194681233 >> v2:172.21.15.81:3300/0 conn(0x7f7cec1550e0 0x7f7cec155550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:36.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.411+0000 7f7cf1000640 1 --2- 172.21.15.28:0/1194681233 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7cec06daa0 0x7f7cec0c8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:36.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.411+0000 7f7cebe00640 1 -- 172.21.15.28:0/1194681233 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 635744652 0 0) 0x7f7cd4003520 con 0x7f7cec152650 2024-02-21T23:03:36.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.412+0000 7f7cebe00640 1 -- 172.21.15.28:0/1194681233 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7cec06daa0 msgr2=0x7f7cec0c8ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:36.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.412+0000 7f7cebe00640 1 --2- 172.21.15.28:0/1194681233 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7cec06daa0 0x7f7cec0c8ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:36.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.412+0000 7f7cebe00640 1 -- 172.21.15.28:0/1194681233 >> v2:172.21.15.81:3300/0 conn(0x7f7cec1550e0 msgr2=0x7f7cec155550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:36.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.412+0000 7f7cebe00640 1 --2- 172.21.15.28:0/1194681233 >> v2:172.21.15.81:3300/0 conn(0x7f7cec1550e0 0x7f7cec155550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:36.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.412+0000 7f7cebe00640 1 -- 172.21.15.28:0/1194681233 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cec158ee0 con 0x7f7cec152650 2024-02-21T23:03:36.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.412+0000 7f7cebe00640 1 -- 172.21.15.28:0/1194681233 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f7ce0001670 con 0x7f7cec152650 2024-02-21T23:03:36.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.412+0000 7f7cebe00640 1 -- 172.21.15.28:0/1194681233 >> v1:172.21.15.28:6789/0 conn(0x7f7cec152650 legacy=0x7f7cec0c7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:36.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.412+0000 7f7cebe00640 1 --2- 172.21.15.28:0/1194681233 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7cd40039b0 0x7f7cd4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:36.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.412+0000 7f7cebe00640 1 --2- 172.21.15.28:0/1194681233 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7cec06daa0 0x7f7cd4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:36.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.412+0000 7f7cebe00640 1 --2- 172.21.15.28:0/1194681233 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cd4004d40 0x7f7cd40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:36.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.412+0000 7f7cebe00640 1 -- 172.21.15.28:0/1194681233 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cec1586d0 con 0x7f7cd4004d40 2024-02-21T23:03:36.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.412+0000 7f7cebe00640 1 -- 172.21.15.28:0/1194681233 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cec158c00 con 0x7f7cd40039b0 2024-02-21T23:03:36.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.412+0000 7f7cebe00640 1 -- 172.21.15.28:0/1194681233 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd400b9c0 con 0x7f7cec06daa0 2024-02-21T23:03:36.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.412+0000 7f7cf2400640 1 --2- 172.21.15.28:0/1194681233 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cd4004d40 0x7f7cd40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:36.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.412+0000 7f7cf1a00640 1 --2- 172.21.15.28:0/1194681233 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7cd40039b0 0x7f7cd4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:36.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.412+0000 7f7cf1000640 1 --2- 172.21.15.28:0/1194681233 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7cec06daa0 0x7f7cd4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:36.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.412+0000 7f7cf2400640 1 -- 172.21.15.28:0/1194681233 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7cec06daa0 msgr2=0x7f7cd4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:36.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.412+0000 7f7cf2400640 1 --2- 172.21.15.28:0/1194681233 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7cec06daa0 0x7f7cd4004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:36.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.412+0000 7f7cf2400640 1 -- 172.21.15.28:0/1194681233 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7cd40039b0 msgr2=0x7f7cd4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:36.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.412+0000 7f7cf2400640 1 --2- 172.21.15.28:0/1194681233 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7cd40039b0 0x7f7cd4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:36.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.412+0000 7f7cf2400640 1 -- 172.21.15.28:0/1194681233 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cd4003520 con 0x7f7cd4004d40 2024-02-21T23:03:36.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.413+0000 7f7cf1a00640 1 --2- 172.21.15.28:0/1194681233 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7cd40039b0 0x7f7cd4003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:03:36.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.414+0000 7f7cf2400640 1 --2- 172.21.15.28:0/1194681233 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cd4004d40 0x7f7cd40051c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f7ce4009fc0 tx=0x7f7cd4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2defd8c88343f5e0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:36.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.414+0000 7f7cebe00640 1 -- 172.21.15.28:0/1194681233 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce4039070 con 0x7f7cd4004d40 2024-02-21T23:03:36.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.414+0000 7f7cebe00640 1 -- 172.21.15.28:0/1194681233 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ce402cc60 con 0x7f7cd4004d40 2024-02-21T23:03:36.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.414+0000 7f7cebe00640 1 -- 172.21.15.28:0/1194681233 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce4009c10 con 0x7f7cd4004d40 2024-02-21T23:03:36.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.414+0000 7f7cf2e00640 1 -- 172.21.15.28:0/1194681233 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cd4004d40 msgr2=0x7f7cd40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:36.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.414+0000 7f7cf2e00640 1 --2- 172.21.15.28:0/1194681233 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cd4004d40 0x7f7cd40051c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f7ce4009fc0 tx=0x7f7cd4005830 comp rx=0 tx=0).stop 2024-02-21T23:03:36.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.414+0000 7f7cf2400640 1 -- 172.21.15.28:0/1194681233 reap_dead start 2024-02-21T23:03:36.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.415+0000 7f7cf2e00640 1 -- 172.21.15.28:0/1194681233 shutdown_connections 2024-02-21T23:03:36.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.415+0000 7f7cf2e00640 1 -- 172.21.15.28:0/1194681233 >> 172.21.15.28:0/1194681233 conn(0x7f7cec072ce0 msgr2=0x7f7cec0ac340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:36.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.415+0000 7f7cf2e00640 1 -- 172.21.15.28:0/1194681233 shutdown_connections 2024-02-21T23:03:36.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.415+0000 7f7cf2e00640 1 -- 172.21.15.28:0/1194681233 wait complete. 2024-02-21T23:03:36.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.417+0000 7f7cf2e00640 1 Processor -- start 2024-02-21T23:03:36.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.417+0000 7f7cf2e00640 1 -- start start 2024-02-21T23:03:36.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.418+0000 7f7cf2e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7cd4004d40 0x7f7cec1514d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:36.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.418+0000 7f7cf2e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7cec0aa920 0x7f7cec148d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:36.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.418+0000 7f7cf2e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cec0aa3d0 0x7f7cec149240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:36.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.418+0000 7f7cf2e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cec152a30 con 0x7f7cec0aa3d0 2024-02-21T23:03:36.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.418+0000 7f7cf2e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cec0bdfb0 con 0x7f7cec0aa920 2024-02-21T23:03:36.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.418+0000 7f7cf2e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cec0c7870 con 0x7f7cd4004d40 2024-02-21T23:03:36.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.418+0000 7f7cf2400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cec0aa3d0 0x7f7cec149240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:36.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.418+0000 7f7cf2400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cec0aa3d0 0x7f7cec149240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44330/0 (socket says 172.21.15.28:44330) 2024-02-21T23:03:36.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.418+0000 7f7cf2400640 1 -- 172.21.15.28:0/961412614 learned_addr learned my addr 172.21.15.28:0/961412614 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:36.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.418+0000 7f7cf1a00640 1 --2- 172.21.15.28:0/961412614 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7cd4004d40 0x7f7cec1514d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:36.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.418+0000 7f7cf1000640 1 --2- 172.21.15.28:0/961412614 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7cec0aa920 0x7f7cec148d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:36.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.418+0000 7f7cf2400640 1 -- 172.21.15.28:0/961412614 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7cd4004d40 msgr2=0x7f7cec1514d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:36.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.418+0000 7f7cf2400640 1 --2- 172.21.15.28:0/961412614 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7cd4004d40 0x7f7cec1514d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:36.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.418+0000 7f7cf2400640 1 -- 172.21.15.28:0/961412614 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7cec0aa920 msgr2=0x7f7cec148d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:36.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.418+0000 7f7cf2400640 1 --2- 172.21.15.28:0/961412614 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7cec0aa920 0x7f7cec148d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:36.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.418+0000 7f7cf2400640 1 -- 172.21.15.28:0/961412614 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cec14e970 con 0x7f7cec0aa3d0 2024-02-21T23:03:36.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.419+0000 7f7cf2400640 1 --2- 172.21.15.28:0/961412614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cec0aa3d0 0x7f7cec149240 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f7ce4037040 tx=0x7f7ce4009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:36.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.419+0000 7f7ce9600640 1 -- 172.21.15.28:0/961412614 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce4039030 con 0x7f7cec0aa3d0 2024-02-21T23:03:36.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.419+0000 7f7ce9600640 1 -- 172.21.15.28:0/961412614 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ce403a040 con 0x7f7cec0aa3d0 2024-02-21T23:03:36.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.419+0000 7f7ce9600640 1 -- 172.21.15.28:0/961412614 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce4009c10 con 0x7f7cec0aa3d0 2024-02-21T23:03:36.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.419+0000 7f7cf2e00640 1 -- 172.21.15.28:0/961412614 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cec14d740 con 0x7f7cec0aa3d0 2024-02-21T23:03:36.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.419+0000 7f7cf2e00640 1 -- 172.21.15.28:0/961412614 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cec14dd00 con 0x7f7cec0aa3d0 2024-02-21T23:03:36.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.421+0000 7f7cf2e00640 1 -- 172.21.15.28:0/961412614 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cec04f290 con 0x7f7cec0aa3d0 2024-02-21T23:03:36.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.422+0000 7f7ce9600640 1 -- 172.21.15.28:0/961412614 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7ce4045030 con 0x7f7cec0aa3d0 2024-02-21T23:03:36.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.423+0000 7f7ce9600640 1 --2- 172.21.15.28:0/961412614 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7cd006b4d0 0x7f7cd006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:36.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.423+0000 7f7ce9600640 1 -- 172.21.15.28:0/961412614 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f7ce40ac210 con 0x7f7cec0aa3d0 2024-02-21T23:03:36.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.423+0000 7f7cf1a00640 1 --2- 172.21.15.28:0/961412614 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7cd006b4d0 0x7f7cd006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:36.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.425+0000 7f7cf1a00640 1 --2- 172.21.15.28:0/961412614 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7cd006b4d0 0x7f7cd006d990 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f7ce0001eb0 tx=0x7f7ce0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:36.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.427+0000 7f7ce9600640 1 -- 172.21.15.28:0/961412614 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7ce404c060 con 0x7f7cec0aa3d0 2024-02-21T23:03:36.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.626+0000 7f7cf2e00640 1 -- 172.21.15.28:0/961412614 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7cec156080 con 0x7f7cec0aa3d0 2024-02-21T23:03:36.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.627+0000 7f7ce9600640 1 -- 172.21.15.28:0/961412614 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7ce407ad20 con 0x7f7cec0aa3d0 2024-02-21T23:03:36.615 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-02-21T23:03:36.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.629+0000 7f7cf2e00640 1 -- 172.21.15.28:0/961412614 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7cd006b4d0 msgr2=0x7f7cd006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:36.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.629+0000 7f7cf2e00640 1 --2- 172.21.15.28:0/961412614 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7cd006b4d0 0x7f7cd006d990 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f7ce0001eb0 tx=0x7f7ce0009040 comp rx=0 tx=0).stop 2024-02-21T23:03:36.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.629+0000 7f7cf2e00640 1 -- 172.21.15.28:0/961412614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cec0aa3d0 msgr2=0x7f7cec149240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:36.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.629+0000 7f7cf2e00640 1 --2- 172.21.15.28:0/961412614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cec0aa3d0 0x7f7cec149240 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f7ce4037040 tx=0x7f7ce4009750 comp rx=0 tx=0).stop 2024-02-21T23:03:36.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.629+0000 7f7cf2e00640 1 -- 172.21.15.28:0/961412614 shutdown_connections 2024-02-21T23:03:36.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.629+0000 7f7cf2e00640 1 --2- 172.21.15.28:0/961412614 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7cd006b4d0 0x7f7cd006d990 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:36.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.629+0000 7f7cf2e00640 1 --2- 172.21.15.28:0/961412614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cec0aa3d0 0x7f7cec149240 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:36.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.629+0000 7f7cf2e00640 1 --2- 172.21.15.28:0/961412614 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7cec0aa920 0x7f7cec148d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:36.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.629+0000 7f7cf2e00640 1 --2- 172.21.15.28:0/961412614 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7cd4004d40 0x7f7cec1514d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:36.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.629+0000 7f7cf2e00640 1 -- 172.21.15.28:0/961412614 >> 172.21.15.28:0/961412614 conn(0x7f7cec072ce0 msgr2=0x7f7cec0c7160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:36.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.630+0000 7f7cf2e00640 1 -- 172.21.15.28:0/961412614 shutdown_connections 2024-02-21T23:03:36.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.630+0000 7f7cf2e00640 1 -- 172.21.15.28:0/961412614 wait complete. 2024-02-21T23:03:36.635 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-02-21T23:03:36.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.868+0000 7f5a37200640 1 Processor -- start 2024-02-21T23:03:36.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.868+0000 7f5a37200640 1 -- start start 2024-02-21T23:03:36.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.869+0000 7f5a37200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5a3014f000 0x7f5a3006ec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:36.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.869+0000 7f5a37200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f5a30155db0 0x7f5a3015c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:36.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.869+0000 7f5a37200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a300c5b80 con 0x7f5a3006f160 2024-02-21T23:03:36.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.870+0000 7f5a37200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a300c4fe0 con 0x7f5a30155db0 2024-02-21T23:03:36.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.870+0000 7f5a37200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a300c5650 con 0x7f5a3014f000 2024-02-21T23:03:36.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.870+0000 7f5a35e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5a3014f000 0x7f5a3006ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:36.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.870+0000 7f5a35e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5a3014f000 0x7f5a3006ec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3301/0 says I am v2:172.21.15.28:45978/0 (socket says 172.21.15.28:45978) 2024-02-21T23:03:36.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.870+0000 7f5a35e00640 1 -- 172.21.15.28:0/2165441922 learned_addr learned my addr 172.21.15.28:0/2165441922 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:36.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.870+0000 7f5a36800640 1 --2- 172.21.15.28:0/2165441922 >> v2:172.21.15.81:3300/0 conn(0x7f5a30155db0 0x7f5a3015c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:36.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.871+0000 7f5a35e00640 1 -- 172.21.15.28:0/2165441922 >> v2:172.21.15.81:3300/0 conn(0x7f5a30155db0 msgr2=0x7f5a3015c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:36.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.871+0000 7f5a35e00640 1 --2- 172.21.15.28:0/2165441922 >> v2:172.21.15.81:3300/0 conn(0x7f5a30155db0 0x7f5a3015c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:36.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.871+0000 7f5a35e00640 1 -- 172.21.15.28:0/2165441922 >> v1:172.21.15.81:6789/0 conn(0x7f5a3006f160 legacy=0x7f5a301556a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:36.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.871+0000 7f5a35e00640 1 -- 172.21.15.28:0/2165441922 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a30152950 con 0x7f5a3014f000 2024-02-21T23:03:36.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.872+0000 7f5a35e00640 1 --2- 172.21.15.28:0/2165441922 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5a3014f000 0x7f5a3006ec20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5a24009870 tx=0x7f5a2402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dbb265ba4ba34941 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:36.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.872+0000 7f5a34a00640 1 -- 172.21.15.28:0/2165441922 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a24034070 con 0x7f5a3014f000 2024-02-21T23:03:36.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.873+0000 7f5a34a00640 1 -- 172.21.15.28:0/2165441922 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a24030840 con 0x7f5a3014f000 2024-02-21T23:03:36.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.873+0000 7f5a34a00640 1 -- 172.21.15.28:0/2165441922 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a24030c10 con 0x7f5a3014f000 2024-02-21T23:03:36.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.873+0000 7f5a37200640 1 -- 172.21.15.28:0/2165441922 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5a3014f000 msgr2=0x7f5a3006ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:36.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.873+0000 7f5a37200640 1 --2- 172.21.15.28:0/2165441922 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5a3014f000 0x7f5a3006ec20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5a24009870 tx=0x7f5a2402b980 comp rx=0 tx=0).stop 2024-02-21T23:03:36.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.873+0000 7f5a37200640 1 -- 172.21.15.28:0/2165441922 shutdown_connections 2024-02-21T23:03:36.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.873+0000 7f5a37200640 1 --2- 172.21.15.28:0/2165441922 >> v2:172.21.15.81:3300/0 conn(0x7f5a30155db0 0x7f5a3015c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:36.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.873+0000 7f5a37200640 1 --2- 172.21.15.28:0/2165441922 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5a3014f000 0x7f5a3006ec20 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:36.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.873+0000 7f5a37200640 1 -- 172.21.15.28:0/2165441922 >> 172.21.15.28:0/2165441922 conn(0x7f5a300a8530 msgr2=0x7f5a300a8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:36.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.873+0000 7f5a37200640 1 -- 172.21.15.28:0/2165441922 shutdown_connections 2024-02-21T23:03:36.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.873+0000 7f5a37200640 1 -- 172.21.15.28:0/2165441922 wait complete. 2024-02-21T23:03:36.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.875+0000 7f5a37200640 1 Processor -- start 2024-02-21T23:03:36.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.875+0000 7f5a37200640 1 -- start start 2024-02-21T23:03:36.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.876+0000 7f5a37200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5a3006f160 0x7f5a30150e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:36.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.876+0000 7f5a37200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a3014f000 0x7f5a301df7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:36.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.876+0000 7f5a37200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5a30155db0 0x7f5a3014d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:36.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.876+0000 7f5a37200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a300a6170 con 0x7f5a3014f000 2024-02-21T23:03:36.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.876+0000 7f5a37200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a30155180 con 0x7f5a3006f160 2024-02-21T23:03:36.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.876+0000 7f5a37200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a300a62f0 con 0x7f5a30155db0 2024-02-21T23:03:36.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.876+0000 7f5a35400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a3014f000 0x7f5a301df7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:36.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.876+0000 7f5a35400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a3014f000 0x7f5a301df7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44336/0 (socket says 172.21.15.28:44336) 2024-02-21T23:03:36.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.876+0000 7f5a35400640 1 -- 172.21.15.28:0/1312947428 learned_addr learned my addr 172.21.15.28:0/1312947428 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:36.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.876+0000 7f5a35e00640 1 --2- 172.21.15.28:0/1312947428 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5a3006f160 0x7f5a30150e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:36.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.876+0000 7f5a36800640 1 --2- 172.21.15.28:0/1312947428 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5a30155db0 0x7f5a3014d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:36.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.877+0000 7f5a35400640 1 -- 172.21.15.28:0/1312947428 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5a30155db0 msgr2=0x7f5a3014d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:36.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.877+0000 7f5a35400640 1 --2- 172.21.15.28:0/1312947428 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5a30155db0 0x7f5a3014d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:36.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.877+0000 7f5a35400640 1 -- 172.21.15.28:0/1312947428 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5a3006f160 msgr2=0x7f5a30150e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:36.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.877+0000 7f5a35400640 1 --2- 172.21.15.28:0/1312947428 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5a3006f160 0x7f5a30150e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:36.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.877+0000 7f5a35400640 1 -- 172.21.15.28:0/1312947428 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a30151480 con 0x7f5a3014f000 2024-02-21T23:03:36.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.877+0000 7f5a35400640 1 --2- 172.21.15.28:0/1312947428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a3014f000 0x7f5a301df7d0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f5a2000c480 tx=0x7f5a2000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:36.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.877+0000 7f5a2a000640 1 -- 172.21.15.28:0/1312947428 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a200052d0 con 0x7f5a3014f000 2024-02-21T23:03:36.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.877+0000 7f5a2a000640 1 -- 172.21.15.28:0/1312947428 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a20005470 con 0x7f5a3014f000 2024-02-21T23:03:36.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.877+0000 7f5a2a000640 1 -- 172.21.15.28:0/1312947428 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a20012870 con 0x7f5a3014f000 2024-02-21T23:03:36.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.877+0000 7f5a37200640 1 -- 172.21.15.28:0/1312947428 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a301ee030 con 0x7f5a3014f000 2024-02-21T23:03:36.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.877+0000 7f5a37200640 1 -- 172.21.15.28:0/1312947428 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a301dfd10 con 0x7f5a3014f000 2024-02-21T23:03:36.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.878+0000 7f5a37200640 1 -- 172.21.15.28:0/1312947428 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a3004f290 con 0x7f5a3014f000 2024-02-21T23:03:36.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.879+0000 7f5a2a000640 1 -- 172.21.15.28:0/1312947428 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5a20012a10 con 0x7f5a3014f000 2024-02-21T23:03:36.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.880+0000 7f5a2a000640 1 --2- 172.21.15.28:0/1312947428 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5a1406b4d0 0x7f5a1406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:36.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.880+0000 7f5a2a000640 1 -- 172.21.15.28:0/1312947428 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f5a2008a2b0 con 0x7f5a3014f000 2024-02-21T23:03:36.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.881+0000 7f5a35e00640 1 --2- 172.21.15.28:0/1312947428 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5a1406b4d0 0x7f5a1406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:36.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.882+0000 7f5a35e00640 1 --2- 172.21.15.28:0/1312947428 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5a1406b4d0 0x7f5a1406d990 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f5a24036000 tx=0x7f5a2403b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:36.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:36.884+0000 7f5a2a000640 1 -- 172.21.15.28:0/1312947428 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f5a20058d20 con 0x7f5a3014f000 2024-02-21T23:03:37.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.074+0000 7f5a37200640 1 -- 172.21.15.28:0/1312947428 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f5a300c51d0 con 0x7f5a3014f000 2024-02-21T23:03:37.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.075+0000 7f5a2a000640 1 -- 172.21.15.28:0/1312947428 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) v1 ==== 103+0+11 (secure 0 0 0) 0x7f5a20058620 con 0x7f5a3014f000 2024-02-21T23:03:37.064 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-02-21T23:03:37.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.077+0000 7f5a37200640 1 -- 172.21.15.28:0/1312947428 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5a1406b4d0 msgr2=0x7f5a1406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:37.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.077+0000 7f5a37200640 1 --2- 172.21.15.28:0/1312947428 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5a1406b4d0 0x7f5a1406d990 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f5a24036000 tx=0x7f5a2403b040 comp rx=0 tx=0).stop 2024-02-21T23:03:37.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.077+0000 7f5a37200640 1 -- 172.21.15.28:0/1312947428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a3014f000 msgr2=0x7f5a301df7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:37.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.077+0000 7f5a37200640 1 --2- 172.21.15.28:0/1312947428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a3014f000 0x7f5a301df7d0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f5a2000c480 tx=0x7f5a2000c940 comp rx=0 tx=0).stop 2024-02-21T23:03:37.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.077+0000 7f5a37200640 1 -- 172.21.15.28:0/1312947428 shutdown_connections 2024-02-21T23:03:37.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.077+0000 7f5a37200640 1 --2- 172.21.15.28:0/1312947428 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5a1406b4d0 0x7f5a1406d990 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:37.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.077+0000 7f5a37200640 1 --2- 172.21.15.28:0/1312947428 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5a30155db0 0x7f5a3014d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:37.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.077+0000 7f5a37200640 1 --2- 172.21.15.28:0/1312947428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a3014f000 0x7f5a301df7d0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:37.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.077+0000 7f5a37200640 1 --2- 172.21.15.28:0/1312947428 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5a3006f160 0x7f5a30150e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:37.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.077+0000 7f5a37200640 1 -- 172.21.15.28:0/1312947428 >> 172.21.15.28:0/1312947428 conn(0x7f5a300a8530 msgr2=0x7f5a30151c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:37.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.078+0000 7f5a37200640 1 -- 172.21.15.28:0/1312947428 shutdown_connections 2024-02-21T23:03:37.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.078+0000 7f5a37200640 1 -- 172.21.15.28:0/1312947428 wait complete. 2024-02-21T23:03:37.084 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:03:37.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.320+0000 7fc1ef200640 1 Processor -- start 2024-02-21T23:03:37.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.320+0000 7fc1ef200640 1 -- start start 2024-02-21T23:03:37.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.321+0000 7fc1ef200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc1f0150450 0x7fc1f00a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:37.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.321+0000 7fc1ef200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc1f0156af0 0x7fc1f006dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:37.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.321+0000 7fc1ef200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1f00c2760 con 0x7fc1f006ee40 2024-02-21T23:03:37.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.321+0000 7fc1ef200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1f00c22c0 con 0x7fc1f0150450 2024-02-21T23:03:37.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.321+0000 7fc1ef200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f0156660 con 0x7fc1f0156af0 2024-02-21T23:03:37.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.322+0000 7fc1ede00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc1f0150450 0x7fc1f00a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:37.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.322+0000 7fc1ede00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc1f0150450 0x7fc1f00a5fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44350/0 (socket says 172.21.15.28:44350) 2024-02-21T23:03:37.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.322+0000 7fc1ede00640 1 -- 172.21.15.28:0/3633553625 learned_addr learned my addr 172.21.15.28:0/3633553625 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:37.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.322+0000 7fc1ede00640 1 -- 172.21.15.28:0/3633553625 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc1f0156af0 msgr2=0x7fc1f006dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:37.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.322+0000 7fc1ee800640 1 --2- 172.21.15.28:0/3633553625 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc1f0156af0 0x7fc1f006dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:37.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.322+0000 7fc1ede00640 1 --2- 172.21.15.28:0/3633553625 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc1f0156af0 0x7fc1f006dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:37.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.322+0000 7fc1ede00640 1 -- 172.21.15.28:0/3633553625 >> v1:172.21.15.28:6789/0 conn(0x7fc1f006ee40 legacy=0x7fc1f01536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:37.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.322+0000 7fc1ede00640 1 -- 172.21.15.28:0/3633553625 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1f0156310 con 0x7fc1f0150450 2024-02-21T23:03:37.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.322+0000 7fc1ee800640 1 --2- 172.21.15.28:0/3633553625 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc1f0156af0 0x7fc1f006dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:03:37.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.323+0000 7fc1ede00640 1 --2- 172.21.15.28:0/3633553625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1f0150450 0x7fc1f00a5fd0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fc1e0009870 tx=0x7fc1e002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ca75e7dadbd9288 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:37.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.324+0000 7fc1eca00640 1 -- 172.21.15.28:0/3633553625 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e0034070 con 0x7fc1f0150450 2024-02-21T23:03:37.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.324+0000 7fc1eca00640 1 -- 172.21.15.28:0/3633553625 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1e0030840 con 0x7fc1f0150450 2024-02-21T23:03:37.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.324+0000 7fc1eca00640 1 -- 172.21.15.28:0/3633553625 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e0030c10 con 0x7fc1f0150450 2024-02-21T23:03:37.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.324+0000 7fc1ef200640 1 -- 172.21.15.28:0/3633553625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1f0150450 msgr2=0x7fc1f00a5fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:37.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.324+0000 7fc1ef200640 1 --2- 172.21.15.28:0/3633553625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1f0150450 0x7fc1f00a5fd0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fc1e0009870 tx=0x7fc1e002b980 comp rx=0 tx=0).stop 2024-02-21T23:03:37.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.324+0000 7fc1ef200640 1 -- 172.21.15.28:0/3633553625 shutdown_connections 2024-02-21T23:03:37.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.324+0000 7fc1ef200640 1 --2- 172.21.15.28:0/3633553625 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc1f0156af0 0x7fc1f006dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:37.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.324+0000 7fc1ef200640 1 --2- 172.21.15.28:0/3633553625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1f0150450 0x7fc1f00a5fd0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:37.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.324+0000 7fc1ef200640 1 -- 172.21.15.28:0/3633553625 >> 172.21.15.28:0/3633553625 conn(0x7fc1f006b630 msgr2=0x7fc1f00c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:37.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.324+0000 7fc1ef200640 1 -- 172.21.15.28:0/3633553625 shutdown_connections 2024-02-21T23:03:37.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.324+0000 7fc1ef200640 1 -- 172.21.15.28:0/3633553625 wait complete. 2024-02-21T23:03:37.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.326+0000 7fc1ef200640 1 Processor -- start 2024-02-21T23:03:37.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.326+0000 7fc1ef200640 1 -- start start 2024-02-21T23:03:37.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.327+0000 7fc1ef200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc1f006ee40 0x7fc1f014ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:37.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.327+0000 7fc1ef200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc1f0150450 0x7fc1f014cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:37.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.327+0000 7fc1ef200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1f0156af0 0x7fc1f01498f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:37.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.327+0000 7fc1ef200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f0156f80 con 0x7fc1f0156af0 2024-02-21T23:03:37.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.327+0000 7fc1ef200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f00c2cb0 con 0x7fc1f0150450 2024-02-21T23:03:37.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.327+0000 7fc1ef200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f0157100 con 0x7fc1f006ee40 2024-02-21T23:03:37.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.327+0000 7fc1ee800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1f0156af0 0x7fc1f01498f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:37.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.328+0000 7fc1ee800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1f0156af0 0x7fc1f01498f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44356/0 (socket says 172.21.15.28:44356) 2024-02-21T23:03:37.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.328+0000 7fc1ee800640 1 -- 172.21.15.28:0/3400797340 learned_addr learned my addr 172.21.15.28:0/3400797340 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:37.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.328+0000 7fc1ede00640 1 --2- 172.21.15.28:0/3400797340 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc1f006ee40 0x7fc1f014ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:37.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.328+0000 7fc1ed400640 1 --2- 172.21.15.28:0/3400797340 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc1f0150450 0x7fc1f014cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:37.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.328+0000 7fc1ee800640 1 -- 172.21.15.28:0/3400797340 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc1f006ee40 msgr2=0x7fc1f014ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:37.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.328+0000 7fc1ee800640 1 --2- 172.21.15.28:0/3400797340 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc1f006ee40 0x7fc1f014ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:37.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.328+0000 7fc1ee800640 1 -- 172.21.15.28:0/3400797340 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc1f0150450 msgr2=0x7fc1f014cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:37.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.328+0000 7fc1ee800640 1 --2- 172.21.15.28:0/3400797340 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc1f0150450 0x7fc1f014cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:37.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.328+0000 7fc1ee800640 1 -- 172.21.15.28:0/3400797340 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1f01eaab0 con 0x7fc1f0156af0 2024-02-21T23:03:37.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.328+0000 7fc1ed400640 1 --2- 172.21.15.28:0/3400797340 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc1f0150450 0x7fc1f014cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:03:37.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.328+0000 7fc1ee800640 1 --2- 172.21.15.28:0/3400797340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1f0156af0 0x7fc1f01498f0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fc1e800cd50 tx=0x7fc1e800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:37.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.329+0000 7fc1e6000640 1 -- 172.21.15.28:0/3400797340 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e800f680 con 0x7fc1f0156af0 2024-02-21T23:03:37.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.329+0000 7fc1e6000640 1 -- 172.21.15.28:0/3400797340 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1e800f840 con 0x7fc1f0156af0 2024-02-21T23:03:37.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.329+0000 7fc1e6000640 1 -- 172.21.15.28:0/3400797340 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e800fc30 con 0x7fc1f0156af0 2024-02-21T23:03:37.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.329+0000 7fc1ef200640 1 -- 172.21.15.28:0/3400797340 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1f01ead90 con 0x7fc1f0156af0 2024-02-21T23:03:37.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.329+0000 7fc1ef200640 1 -- 172.21.15.28:0/3400797340 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1f01eb350 con 0x7fc1f0156af0 2024-02-21T23:03:37.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.330+0000 7fc1ef200640 1 -- 172.21.15.28:0/3400797340 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1f004f290 con 0x7fc1f0156af0 2024-02-21T23:03:37.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.331+0000 7fc1e6000640 1 -- 172.21.15.28:0/3400797340 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc1e800e3e0 con 0x7fc1f0156af0 2024-02-21T23:03:37.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.332+0000 7fc1e6000640 1 --2- 172.21.15.28:0/3400797340 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc1d006b4d0 0x7fc1d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:37.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.332+0000 7fc1e6000640 1 -- 172.21.15.28:0/3400797340 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6735+0+0 (secure 0 0 0) 0x7fc1e808b3d0 con 0x7fc1f0156af0 2024-02-21T23:03:37.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.333+0000 7fc1ede00640 1 --2- 172.21.15.28:0/3400797340 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc1d006b4d0 0x7fc1d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:37.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.334+0000 7fc1ede00640 1 --2- 172.21.15.28:0/3400797340 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc1d006b4d0 0x7fc1d006d990 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fc1e0036000 tx=0x7fc1e003b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:37.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.337+0000 7fc1e6000640 1 -- 172.21.15.28:0/3400797340 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc1e8014030 con 0x7fc1f0156af0 2024-02-21T23:03:37.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.529+0000 7fc1ef200640 1 -- 172.21.15.28:0/3400797340 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc1f00c1f10 con 0x7fc1f0156af0 2024-02-21T23:03:37.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.531+0000 7fc1e6000640 1 -- 172.21.15.28:0/3400797340 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+18843 (secure 0 0 0) 0x7fc1e8059e40 con 0x7fc1f0156af0 2024-02-21T23:03:37.520 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:03:37.520 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":26,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:03:23.860109+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:02:35.795158+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-21T23:02:36.840110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-02-21T23:02:40.441388+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":3,"to":2}]},{"pgid":"3.12","mappings":[{"from":0,"to":1}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:03:37.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.533+0000 7fc1ef200640 1 -- 172.21.15.28:0/3400797340 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc1d006b4d0 msgr2=0x7fc1d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:37.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.533+0000 7fc1ef200640 1 --2- 172.21.15.28:0/3400797340 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc1d006b4d0 0x7fc1d006d990 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fc1e0036000 tx=0x7fc1e003b040 comp rx=0 tx=0).stop 2024-02-21T23:03:37.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.533+0000 7fc1ef200640 1 -- 172.21.15.28:0/3400797340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1f0156af0 msgr2=0x7fc1f01498f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:37.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.533+0000 7fc1ef200640 1 --2- 172.21.15.28:0/3400797340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1f0156af0 0x7fc1f01498f0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fc1e800cd50 tx=0x7fc1e800a650 comp rx=0 tx=0).stop 2024-02-21T23:03:37.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.534+0000 7fc1ef200640 1 -- 172.21.15.28:0/3400797340 shutdown_connections 2024-02-21T23:03:37.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.534+0000 7fc1ef200640 1 --2- 172.21.15.28:0/3400797340 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc1d006b4d0 0x7fc1d006d990 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:37.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.534+0000 7fc1ef200640 1 --2- 172.21.15.28:0/3400797340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1f0156af0 0x7fc1f01498f0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:37.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.534+0000 7fc1ef200640 1 --2- 172.21.15.28:0/3400797340 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc1f0150450 0x7fc1f014cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:37.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.534+0000 7fc1ef200640 1 --2- 172.21.15.28:0/3400797340 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc1f006ee40 0x7fc1f014ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:37.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.534+0000 7fc1ef200640 1 -- 172.21.15.28:0/3400797340 >> 172.21.15.28:0/3400797340 conn(0x7fc1f006b630 msgr2=0x7fc1f00a6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:37.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.534+0000 7fc1ef200640 1 -- 172.21.15.28:0/3400797340 shutdown_connections 2024-02-21T23:03:37.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.534+0000 7fc1ef200640 1 -- 172.21.15.28:0/3400797340 wait complete. 2024-02-21T23:03:37.539 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-21T23:02:30.728329+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-21T23:02:35.795158+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-02-21T23:02:36.840110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-02-21T23:02:40.441388+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-02-21T23:03:37.540 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-21T23:03:37.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.771+0000 7f3c03e00640 1 Processor -- start 2024-02-21T23:03:37.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.772+0000 7f3c03e00640 1 -- start start 2024-02-21T23:03:37.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.773+0000 7f3c03e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3bfc152a30 0x7f3bfc0c8690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:37.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.773+0000 7f3c03e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3bfc159370 0x7f3bfc15fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:37.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.773+0000 7f3c03e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bfc0c8170 con 0x7f3bfc0ab170 2024-02-21T23:03:37.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.773+0000 7f3c03e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bfc1561a0 con 0x7f3bfc159370 2024-02-21T23:03:37.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.773+0000 7f3c03e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bfc0c7f40 con 0x7f3bfc152a30 2024-02-21T23:03:37.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.773+0000 7f3c01e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3bfc159370 0x7f3bfc15fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:37.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.774+0000 7f3c01e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3bfc159370 0x7f3bfc15fa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44358/0 (socket says 172.21.15.28:44358) 2024-02-21T23:03:37.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.774+0000 7f3c01e00640 1 -- 172.21.15.28:0/71579261 learned_addr learned my addr 172.21.15.28:0/71579261 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:37.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.774+0000 7f3c01400640 1 --2- 172.21.15.28:0/71579261 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3bfc152a30 0x7f3bfc0c8690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:37.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.774+0000 7f3c01e00640 1 -- 172.21.15.28:0/71579261 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3bfc152a30 msgr2=0x7f3bfc0c8690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:37.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.774+0000 7f3c01e00640 1 --2- 172.21.15.28:0/71579261 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3bfc152a30 0x7f3bfc0c8690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:37.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.774+0000 7f3c01e00640 1 -- 172.21.15.28:0/71579261 >> v1:172.21.15.81:6789/0 conn(0x7f3bfc0ab170 legacy=0x7f3bfc158c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:37.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.774+0000 7f3c01e00640 1 -- 172.21.15.28:0/71579261 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bfc0c3130 con 0x7f3bfc159370 2024-02-21T23:03:37.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.776+0000 7f3c01e00640 1 --2- 172.21.15.28:0/71579261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bfc159370 0x7f3bfc15fa70 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f3bf40098a0 tx=0x7f3bf402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b643b804d6c08898 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:37.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.776+0000 7f3bfbe00640 1 -- 172.21.15.28:0/71579261 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf4035020 con 0x7f3bfc159370 2024-02-21T23:03:37.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.776+0000 7f3bfbe00640 1 -- 172.21.15.28:0/71579261 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bf4031640 con 0x7f3bfc159370 2024-02-21T23:03:37.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.776+0000 7f3bfbe00640 1 -- 172.21.15.28:0/71579261 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf402d040 con 0x7f3bfc159370 2024-02-21T23:03:37.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.777+0000 7f3c03e00640 1 -- 172.21.15.28:0/71579261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bfc159370 msgr2=0x7f3bfc15fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:37.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.777+0000 7f3c03e00640 1 --2- 172.21.15.28:0/71579261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bfc159370 0x7f3bfc15fa70 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f3bf40098a0 tx=0x7f3bf402b8f0 comp rx=0 tx=0).stop 2024-02-21T23:03:37.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.777+0000 7f3c03e00640 1 -- 172.21.15.28:0/71579261 shutdown_connections 2024-02-21T23:03:37.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.777+0000 7f3c03e00640 1 --2- 172.21.15.28:0/71579261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bfc159370 0x7f3bfc15fa70 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:37.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.777+0000 7f3c03e00640 1 --2- 172.21.15.28:0/71579261 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3bfc152a30 0x7f3bfc0c8690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:37.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.777+0000 7f3c03e00640 1 -- 172.21.15.28:0/71579261 >> 172.21.15.28:0/71579261 conn(0x7f3bfc06b4a0 msgr2=0x7f3bfc0aa4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:37.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.777+0000 7f3c03e00640 1 -- 172.21.15.28:0/71579261 shutdown_connections 2024-02-21T23:03:37.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.777+0000 7f3c03e00640 1 -- 172.21.15.28:0/71579261 wait complete. 2024-02-21T23:03:37.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.778+0000 7f3c03e00640 1 Processor -- start 2024-02-21T23:03:37.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.779+0000 7f3c03e00640 1 -- start start 2024-02-21T23:03:37.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.779+0000 7f3c03e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bfc0ab170 0x7f3bfc14e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:37.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.779+0000 7f3c03e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3bfc152a30 0x7f3bfc150c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:37.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.779+0000 7f3c03e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3bfc159370 0x7f3bfc06f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:37.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.780+0000 7f3c03e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bfc0c36f0 con 0x7f3bfc0ab170 2024-02-21T23:03:37.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.780+0000 7f3c03e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bfc1587d0 con 0x7f3bfc152a30 2024-02-21T23:03:37.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.780+0000 7f3c03e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bfc0c7c60 con 0x7f3bfc159370 2024-02-21T23:03:37.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.780+0000 7f3c01400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bfc0ab170 0x7f3bfc14e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:37.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.780+0000 7f3c01400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bfc0ab170 0x7f3bfc14e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44366/0 (socket says 172.21.15.28:44366) 2024-02-21T23:03:37.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.780+0000 7f3c01400640 1 -- 172.21.15.28:0/2130116672 learned_addr learned my addr 172.21.15.28:0/2130116672 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:37.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.780+0000 7f3c00a00640 1 --2- 172.21.15.28:0/2130116672 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3bfc152a30 0x7f3bfc150c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:37.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.780+0000 7f3c01400640 1 -- 172.21.15.28:0/2130116672 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3bfc159370 msgr2=0x7f3bfc06f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:37.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.780+0000 7f3c01400640 1 --2- 172.21.15.28:0/2130116672 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3bfc159370 0x7f3bfc06f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:37.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.780+0000 7f3c01400640 1 -- 172.21.15.28:0/2130116672 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3bfc152a30 msgr2=0x7f3bfc150c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:37.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.780+0000 7f3c01400640 1 --2- 172.21.15.28:0/2130116672 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3bfc152a30 0x7f3bfc150c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:37.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.780+0000 7f3c01400640 1 -- 172.21.15.28:0/2130116672 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bfc0710e0 con 0x7f3bfc0ab170 2024-02-21T23:03:37.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.781+0000 7f3c01400640 1 --2- 172.21.15.28:0/2130116672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bfc0ab170 0x7f3bfc14e880 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f3bf0007ef0 tx=0x7f3bf000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:37.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.781+0000 7f3bf9600640 1 -- 172.21.15.28:0/2130116672 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf0014620 con 0x7f3bfc0ab170 2024-02-21T23:03:37.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.781+0000 7f3bf9600640 1 -- 172.21.15.28:0/2130116672 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bf0019070 con 0x7f3bfc0ab170 2024-02-21T23:03:37.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.781+0000 7f3bf9600640 1 -- 172.21.15.28:0/2130116672 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf0014a00 con 0x7f3bfc0ab170 2024-02-21T23:03:37.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.781+0000 7f3c03e00640 1 -- 172.21.15.28:0/2130116672 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bfc1ebc80 con 0x7f3bfc0ab170 2024-02-21T23:03:37.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.781+0000 7f3c03e00640 1 -- 172.21.15.28:0/2130116672 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bfc151150 con 0x7f3bfc0ab170 2024-02-21T23:03:37.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.782+0000 7f3c03e00640 1 -- 172.21.15.28:0/2130116672 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bfc067f30 con 0x7f3bfc0ab170 2024-02-21T23:03:37.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.783+0000 7f3bf9600640 1 -- 172.21.15.28:0/2130116672 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3bf0014ba0 con 0x7f3bfc0ab170 2024-02-21T23:03:37.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.784+0000 7f3bf9600640 1 --2- 172.21.15.28:0/2130116672 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3bdc06b4d0 0x7f3bdc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:37.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.784+0000 7f3bf9600640 1 -- 172.21.15.28:0/2130116672 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f3bf008af40 con 0x7f3bfc0ab170 2024-02-21T23:03:37.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.785+0000 7f3c00a00640 1 --2- 172.21.15.28:0/2130116672 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3bdc06b4d0 0x7f3bdc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:37.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.786+0000 7f3c00a00640 1 --2- 172.21.15.28:0/2130116672 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3bdc06b4d0 0x7f3bdc06d990 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc0aa3e0 tx=0x7f3be8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:37.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.789+0000 7f3bf9600640 1 -- 172.21.15.28:0/2130116672 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3bf00599b0 con 0x7f3bfc0ab170 2024-02-21T23:03:37.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.988+0000 7f3c03e00640 1 -- 172.21.15.28:0/2130116672 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3bfc0adc60 con 0x7f3bfc0ab170 2024-02-21T23:03:37.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.989+0000 7f3bf9600640 1 -- 172.21.15.28:0/2130116672 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3bf00592b0 con 0x7f3bfc0ab170 2024-02-21T23:03:37.978 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-02-21T23:03:37.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.991+0000 7f3c03e00640 1 -- 172.21.15.28:0/2130116672 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3bdc06b4d0 msgr2=0x7f3bdc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:37.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.991+0000 7f3c03e00640 1 --2- 172.21.15.28:0/2130116672 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3bdc06b4d0 0x7f3bdc06d990 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc0aa3e0 tx=0x7f3be8009040 comp rx=0 tx=0).stop 2024-02-21T23:03:37.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.991+0000 7f3c03e00640 1 -- 172.21.15.28:0/2130116672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bfc0ab170 msgr2=0x7f3bfc14e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:37.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.991+0000 7f3c03e00640 1 --2- 172.21.15.28:0/2130116672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bfc0ab170 0x7f3bfc14e880 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f3bf0007ef0 tx=0x7f3bf000e4b0 comp rx=0 tx=0).stop 2024-02-21T23:03:37.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.992+0000 7f3c03e00640 1 -- 172.21.15.28:0/2130116672 shutdown_connections 2024-02-21T23:03:37.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.992+0000 7f3c03e00640 1 --2- 172.21.15.28:0/2130116672 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3bdc06b4d0 0x7f3bdc06d990 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:37.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.992+0000 7f3c03e00640 1 --2- 172.21.15.28:0/2130116672 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3bfc159370 0x7f3bfc06f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:37.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.992+0000 7f3c03e00640 1 --2- 172.21.15.28:0/2130116672 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3bfc152a30 0x7f3bfc150c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:37.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.992+0000 7f3c03e00640 1 --2- 172.21.15.28:0/2130116672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bfc0ab170 0x7f3bfc14e880 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:37.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.992+0000 7f3c03e00640 1 -- 172.21.15.28:0/2130116672 >> 172.21.15.28:0/2130116672 conn(0x7f3bfc06b4a0 msgr2=0x7f3bfc0c5150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:37.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.992+0000 7f3c03e00640 1 -- 172.21.15.28:0/2130116672 shutdown_connections 2024-02-21T23:03:37.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:37.992+0000 7f3c03e00640 1 -- 172.21.15.28:0/2130116672 wait complete. 2024-02-21T23:03:38.000 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-21T23:03:38.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.231+0000 7f534e800640 1 Processor -- start 2024-02-21T23:03:38.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.232+0000 7f534e800640 1 -- start start 2024-02-21T23:03:38.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.233+0000 7f534e800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f534813e0c0 0x7f53480c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:38.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.233+0000 7f534e800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5348158550 con 0x7f53480c76f0 2024-02-21T23:03:38.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.233+0000 7f534e800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53481587d0 con 0x7f534806dbc0 2024-02-21T23:03:38.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.233+0000 7f534e800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53480c89a0 con 0x7f534813e0c0 2024-02-21T23:03:38.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.234+0000 7f534d400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f534813e0c0 0x7f53480c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:38.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.234+0000 7f534d400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f534813e0c0 0x7f53480c71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44378/0 (socket says 172.21.15.28:44378) 2024-02-21T23:03:38.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.234+0000 7f534d400640 1 -- 172.21.15.28:0/927320960 learned_addr learned my addr 172.21.15.28:0/927320960 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:38.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.234+0000 7f534d400640 1 -- 172.21.15.28:0/927320960 >> v1:172.21.15.81:6789/0 conn(0x7f534806dbc0 legacy=0x7f534806c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:38.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.234+0000 7f534d400640 1 -- 172.21.15.28:0/927320960 >> v1:172.21.15.28:6789/0 conn(0x7f53480c76f0 legacy=0x7f534806c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:38.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.234+0000 7f534d400640 1 -- 172.21.15.28:0/927320960 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5348158ab0 con 0x7f534813e0c0 2024-02-21T23:03:38.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.235+0000 7f534d400640 1 --2- 172.21.15.28:0/927320960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f534813e0c0 0x7f53480c71b0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f5338009e50 tx=0x7f533802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d00784cafb23a98e server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:38.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.236+0000 7f5347e00640 1 -- 172.21.15.28:0/927320960 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f533802c9f0 con 0x7f534813e0c0 2024-02-21T23:03:38.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.236+0000 7f5347e00640 1 -- 172.21.15.28:0/927320960 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f533802cb90 con 0x7f534813e0c0 2024-02-21T23:03:38.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.236+0000 7f5347e00640 1 -- 172.21.15.28:0/927320960 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53380343f0 con 0x7f534813e0c0 2024-02-21T23:03:38.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.236+0000 7f534e800640 1 -- 172.21.15.28:0/927320960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f534813e0c0 msgr2=0x7f53480c71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:38.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.236+0000 7f534e800640 1 --2- 172.21.15.28:0/927320960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f534813e0c0 0x7f53480c71b0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f5338009e50 tx=0x7f533802be50 comp rx=0 tx=0).stop 2024-02-21T23:03:38.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.236+0000 7f534e800640 1 -- 172.21.15.28:0/927320960 shutdown_connections 2024-02-21T23:03:38.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.236+0000 7f534e800640 1 --2- 172.21.15.28:0/927320960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f534813e0c0 0x7f53480c71b0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:38.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.236+0000 7f534e800640 1 -- 172.21.15.28:0/927320960 >> 172.21.15.28:0/927320960 conn(0x7f5348073110 msgr2=0x7f53480ac490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:38.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.237+0000 7f534e800640 1 -- 172.21.15.28:0/927320960 shutdown_connections 2024-02-21T23:03:38.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.237+0000 7f534e800640 1 -- 172.21.15.28:0/927320960 wait complete. 2024-02-21T23:03:38.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.239+0000 7f534e800640 1 Processor -- start 2024-02-21T23:03:38.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.239+0000 7f534e800640 1 -- start start 2024-02-21T23:03:38.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.240+0000 7f534e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f534806dbc0 0x7f53480abc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:38.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.240+0000 7f534e800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f53480c76f0 0x7f5348154950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:38.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.240+0000 7f534e800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f534813e0c0 0x7f5348154e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:38.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.240+0000 7f534e800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f534813e520 con 0x7f534806dbc0 2024-02-21T23:03:38.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.240+0000 7f534e800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5348159070 con 0x7f53480c76f0 2024-02-21T23:03:38.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.240+0000 7f534e800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f534813e6a0 con 0x7f534813e0c0 2024-02-21T23:03:38.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.240+0000 7f534d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f534806dbc0 0x7f53480abc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:38.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.241+0000 7f534d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f534806dbc0 0x7f53480abc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44388/0 (socket says 172.21.15.28:44388) 2024-02-21T23:03:38.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.241+0000 7f534d400640 1 -- 172.21.15.28:0/532139747 learned_addr learned my addr 172.21.15.28:0/532139747 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:38.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.241+0000 7f534de00640 1 --2- 172.21.15.28:0/532139747 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f534813e0c0 0x7f5348154e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:38.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.241+0000 7f534ca00640 1 --2- 172.21.15.28:0/532139747 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f53480c76f0 0x7f5348154950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:38.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.241+0000 7f534d400640 1 -- 172.21.15.28:0/532139747 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f534813e0c0 msgr2=0x7f5348154e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:38.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.241+0000 7f534d400640 1 --2- 172.21.15.28:0/532139747 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f534813e0c0 0x7f5348154e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:38.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.241+0000 7f534d400640 1 -- 172.21.15.28:0/532139747 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f53480c76f0 msgr2=0x7f5348154950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:38.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.241+0000 7f534d400640 1 --2- 172.21.15.28:0/532139747 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f53480c76f0 0x7f5348154950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:38.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.241+0000 7f534d400640 1 -- 172.21.15.28:0/532139747 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53481553d0 con 0x7f534806dbc0 2024-02-21T23:03:38.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.241+0000 7f534d400640 1 --2- 172.21.15.28:0/532139747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f534806dbc0 0x7f53480abc90 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f5338036040 tx=0x7f5338004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:38.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.242+0000 7f5345600640 1 -- 172.21.15.28:0/532139747 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5338044020 con 0x7f534806dbc0 2024-02-21T23:03:38.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.242+0000 7f5345600640 1 -- 172.21.15.28:0/532139747 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5338040670 con 0x7f534806dbc0 2024-02-21T23:03:38.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.242+0000 7f5345600640 1 -- 172.21.15.28:0/532139747 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5338040a60 con 0x7f534806dbc0 2024-02-21T23:03:38.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.242+0000 7f534e800640 1 -- 172.21.15.28:0/532139747 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5348135e50 con 0x7f534806dbc0 2024-02-21T23:03:38.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.242+0000 7f534e800640 1 -- 172.21.15.28:0/532139747 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5348136410 con 0x7f534806dbc0 2024-02-21T23:03:38.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.243+0000 7f534e800640 1 -- 172.21.15.28:0/532139747 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f534804f290 con 0x7f534806dbc0 2024-02-21T23:03:38.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.244+0000 7f5345600640 1 -- 172.21.15.28:0/532139747 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5338047020 con 0x7f534806dbc0 2024-02-21T23:03:38.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.245+0000 7f5345600640 1 --2- 172.21.15.28:0/532139747 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f532806b500 0x7f532806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:38.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.245+0000 7f5345600640 1 -- 172.21.15.28:0/532139747 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f53380acee0 con 0x7f534806dbc0 2024-02-21T23:03:38.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.246+0000 7f534ca00640 1 --2- 172.21.15.28:0/532139747 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f532806b500 0x7f532806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:38.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.247+0000 7f534ca00640 1 --2- 172.21.15.28:0/532139747 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f532806b500 0x7f532806d9c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f53480ca0c0 tx=0x7f5334008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:38.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.249+0000 7f5345600640 1 -- 172.21.15.28:0/532139747 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f533804e070 con 0x7f534806dbc0 2024-02-21T23:03:38.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.462+0000 7f534e800640 1 -- 172.21.15.28:0/532139747 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f534806d9a0 con 0x7f534806dbc0 2024-02-21T23:03:38.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.463+0000 7f5345600640 1 -- 172.21.15.28:0/532139747 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+11 (secure 0 0 0) 0x7f533807b9e0 con 0x7f534806dbc0 2024-02-21T23:03:38.452 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-02-21T23:03:38.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.466+0000 7f534e800640 1 -- 172.21.15.28:0/532139747 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f532806b500 msgr2=0x7f532806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:38.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.466+0000 7f534e800640 1 --2- 172.21.15.28:0/532139747 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f532806b500 0x7f532806d9c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f53480ca0c0 tx=0x7f5334008040 comp rx=0 tx=0).stop 2024-02-21T23:03:38.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.466+0000 7f534e800640 1 -- 172.21.15.28:0/532139747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f534806dbc0 msgr2=0x7f53480abc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:38.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.466+0000 7f534e800640 1 --2- 172.21.15.28:0/532139747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f534806dbc0 0x7f53480abc90 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f5338036040 tx=0x7f5338004a40 comp rx=0 tx=0).stop 2024-02-21T23:03:38.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.466+0000 7f534e800640 1 -- 172.21.15.28:0/532139747 shutdown_connections 2024-02-21T23:03:38.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.466+0000 7f534e800640 1 --2- 172.21.15.28:0/532139747 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f532806b500 0x7f532806d9c0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:38.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.466+0000 7f534e800640 1 --2- 172.21.15.28:0/532139747 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f534813e0c0 0x7f5348154e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:38.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.466+0000 7f534e800640 1 --2- 172.21.15.28:0/532139747 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f53480c76f0 0x7f5348154950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:38.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.466+0000 7f534e800640 1 --2- 172.21.15.28:0/532139747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f534806dbc0 0x7f53480abc90 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:38.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.466+0000 7f534e800640 1 -- 172.21.15.28:0/532139747 >> 172.21.15.28:0/532139747 conn(0x7f5348073110 msgr2=0x7f534806c280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:38.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.466+0000 7f534e800640 1 -- 172.21.15.28:0/532139747 shutdown_connections 2024-02-21T23:03:38.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.466+0000 7f534e800640 1 -- 172.21.15.28:0/532139747 wait complete. 2024-02-21T23:03:38.474 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-21T23:03:38.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.697+0000 7f31f5400640 1 Processor -- start 2024-02-21T23:03:38.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.697+0000 7f31f5400640 1 -- start start 2024-02-21T23:03:38.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.699+0000 7f31f5400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f31f013e0c0 0x7f31f00c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:38.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.699+0000 7f31f5400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f31f006dbc0 0x7f31f006c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:38.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.699+0000 7f31f5400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31f0158550 con 0x7f31f00c76f0 2024-02-21T23:03:38.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.699+0000 7f31f5400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31f00c89a0 con 0x7f31f013e0c0 2024-02-21T23:03:38.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.699+0000 7f31f5400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31f01587d0 con 0x7f31f006dbc0 2024-02-21T23:03:38.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.699+0000 7f31f4a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f31f00c76f0 0x7f31f006c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55134/0 (socket says 172.21.15.28:55134) 2024-02-21T23:03:38.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.699+0000 7f31f4a00640 1 -- 172.21.15.28:0/2692272836 learned_addr learned my addr 172.21.15.28:0/2692272836 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:03:38.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.700+0000 7f31efe00640 1 --2- 172.21.15.28:0/2692272836 >> v2:172.21.15.81:3300/0 conn(0x7f31f013e0c0 0x7f31f00c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:38.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.700+0000 7f31ef400640 1 --2- 172.21.15.28:0/2692272836 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f31f006dbc0 0x7f31f006c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:38.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.700+0000 7f31eea00640 1 -- 172.21.15.28:0/2692272836 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3043062922 0 0) 0x7f31e8001490 con 0x7f31f00c76f0 2024-02-21T23:03:38.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.700+0000 7f31eea00640 1 -- 172.21.15.28:0/2692272836 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31d4003520 con 0x7f31f00c76f0 2024-02-21T23:03:38.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.700+0000 7f31eea00640 1 -- 172.21.15.28:0/2692272836 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3545852962 0 0) 0x7f31d4003520 con 0x7f31f00c76f0 2024-02-21T23:03:38.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.700+0000 7f31eea00640 1 -- 172.21.15.28:0/2692272836 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f31f006dbc0 msgr2=0x7f31f006c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:38.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.700+0000 7f31eea00640 1 --2- 172.21.15.28:0/2692272836 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f31f006dbc0 0x7f31f006c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:38.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.700+0000 7f31eea00640 1 -- 172.21.15.28:0/2692272836 >> v2:172.21.15.81:3300/0 conn(0x7f31f013e0c0 msgr2=0x7f31f00c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:38.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.700+0000 7f31eea00640 1 --2- 172.21.15.28:0/2692272836 >> v2:172.21.15.81:3300/0 conn(0x7f31f013e0c0 0x7f31f00c71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:38.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.700+0000 7f31eea00640 1 -- 172.21.15.28:0/2692272836 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31f0158ab0 con 0x7f31f00c76f0 2024-02-21T23:03:38.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.700+0000 7f31eea00640 1 -- 172.21.15.28:0/2692272836 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f31e8001bc0 con 0x7f31f00c76f0 2024-02-21T23:03:38.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.700+0000 7f31eea00640 1 -- 172.21.15.28:0/2692272836 >> v1:172.21.15.28:6789/0 conn(0x7f31f00c76f0 legacy=0x7f31f006c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:38.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.700+0000 7f31eea00640 1 --2- 172.21.15.28:0/2692272836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31d40039b0 0x7f31d4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:38.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.700+0000 7f31eea00640 1 --2- 172.21.15.28:0/2692272836 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f31f006dbc0 0x7f31d4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:38.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.700+0000 7f31eea00640 1 --2- 172.21.15.28:0/2692272836 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f31d4004d40 0x7f31d40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:38.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.700+0000 7f31eea00640 1 -- 172.21.15.28:0/2692272836 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31f01587d0 con 0x7f31d40039b0 2024-02-21T23:03:38.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.700+0000 7f31eea00640 1 -- 172.21.15.28:0/2692272836 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31d400b9c0 con 0x7f31d4004d40 2024-02-21T23:03:38.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.700+0000 7f31eea00640 1 -- 172.21.15.28:0/2692272836 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31d400bb70 con 0x7f31f006dbc0 2024-02-21T23:03:38.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.701+0000 7f31efe00640 1 --2- 172.21.15.28:0/2692272836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31d40039b0 0x7f31d4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:38.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.701+0000 7f31ef400640 1 --2- 172.21.15.28:0/2692272836 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f31f006dbc0 0x7f31d4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:38.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.701+0000 7f31efe00640 1 -- 172.21.15.28:0/2692272836 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f31f006dbc0 msgr2=0x7f31d4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:38.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.701+0000 7f31efe00640 1 --2- 172.21.15.28:0/2692272836 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f31f006dbc0 0x7f31d4004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:38.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.701+0000 7f31efe00640 1 -- 172.21.15.28:0/2692272836 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f31d4004d40 msgr2=0x7f31d40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:38.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.701+0000 7f31efe00640 1 --2- 172.21.15.28:0/2692272836 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f31d4004d40 0x7f31d40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:38.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.701+0000 7f31efe00640 1 -- 172.21.15.28:0/2692272836 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31d4003520 con 0x7f31d40039b0 2024-02-21T23:03:38.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.702+0000 7f31efe00640 1 --2- 172.21.15.28:0/2692272836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31d40039b0 0x7f31d4003dd0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f31e0009ac0 tx=0x7f31d4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfde30cdc936eb42 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:38.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.702+0000 7f31eea00640 1 -- 172.21.15.28:0/2692272836 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31e0039070 con 0x7f31d40039b0 2024-02-21T23:03:38.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.702+0000 7f31eea00640 1 -- 172.21.15.28:0/2692272836 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31e002c710 con 0x7f31d40039b0 2024-02-21T23:03:38.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.702+0000 7f31eea00640 1 -- 172.21.15.28:0/2692272836 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31e0009690 con 0x7f31d40039b0 2024-02-21T23:03:38.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.703+0000 7f31f5400640 1 -- 172.21.15.28:0/2692272836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31d40039b0 msgr2=0x7f31d4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:38.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.703+0000 7f31f5400640 1 --2- 172.21.15.28:0/2692272836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31d40039b0 0x7f31d4003dd0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f31e0009ac0 tx=0x7f31d4005830 comp rx=0 tx=0).stop 2024-02-21T23:03:38.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.703+0000 7f31f4a00640 1 -- 172.21.15.28:0/2692272836 reap_dead start 2024-02-21T23:03:38.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.703+0000 7f31f5400640 1 -- 172.21.15.28:0/2692272836 shutdown_connections 2024-02-21T23:03:38.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.703+0000 7f31f5400640 1 -- 172.21.15.28:0/2692272836 >> 172.21.15.28:0/2692272836 conn(0x7f31f0073110 msgr2=0x7f31f00ac490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:38.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.703+0000 7f31f5400640 1 -- 172.21.15.28:0/2692272836 shutdown_connections 2024-02-21T23:03:38.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.703+0000 7f31f5400640 1 -- 172.21.15.28:0/2692272836 wait complete. 2024-02-21T23:03:38.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.705+0000 7f31f5400640 1 Processor -- start 2024-02-21T23:03:38.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.705+0000 7f31f5400640 1 -- start start 2024-02-21T23:03:38.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.706+0000 7f31f5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31d40039b0 0x7f31f0155a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:38.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.706+0000 7f31f5400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f31f0153cf0 0x7f31f01396e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:38.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.706+0000 7f31f5400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f31f0152ee0 0x7f31f0153340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:38.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.706+0000 7f31f5400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31f006ced0 con 0x7f31d40039b0 2024-02-21T23:03:38.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.706+0000 7f31f5400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31f0159070 con 0x7f31f0152ee0 2024-02-21T23:03:38.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.706+0000 7f31f5400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31f013e520 con 0x7f31f0153cf0 2024-02-21T23:03:38.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.707+0000 7f31efe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31d40039b0 0x7f31f0155a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:38.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.707+0000 7f31efe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31d40039b0 0x7f31f0155a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44416/0 (socket says 172.21.15.28:44416) 2024-02-21T23:03:38.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.707+0000 7f31efe00640 1 -- 172.21.15.28:0/1273523729 learned_addr learned my addr 172.21.15.28:0/1273523729 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:38.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.707+0000 7f31f4a00640 1 --2- 172.21.15.28:0/1273523729 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f31f0152ee0 0x7f31f0153340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:38.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.707+0000 7f31ef400640 1 --2- 172.21.15.28:0/1273523729 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f31f0153cf0 0x7f31f01396e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:38.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.707+0000 7f31efe00640 1 -- 172.21.15.28:0/1273523729 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f31f0153cf0 msgr2=0x7f31f01396e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:38.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.707+0000 7f31efe00640 1 --2- 172.21.15.28:0/1273523729 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f31f0153cf0 0x7f31f01396e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:38.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.707+0000 7f31efe00640 1 -- 172.21.15.28:0/1273523729 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f31f0152ee0 msgr2=0x7f31f0153340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:38.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.707+0000 7f31efe00640 1 --2- 172.21.15.28:0/1273523729 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f31f0152ee0 0x7f31f0153340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:38.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.707+0000 7f31efe00640 1 -- 172.21.15.28:0/1273523729 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31f0155670 con 0x7f31d40039b0 2024-02-21T23:03:38.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.708+0000 7f31efe00640 1 --2- 172.21.15.28:0/1273523729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31d40039b0 0x7f31f0155a90 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f31e0037040 tx=0x7f31e0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:38.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.708+0000 7f31e7e00640 1 -- 172.21.15.28:0/1273523729 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31e0042020 con 0x7f31d40039b0 2024-02-21T23:03:38.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.708+0000 7f31e7e00640 1 -- 172.21.15.28:0/1273523729 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31e003a040 con 0x7f31d40039b0 2024-02-21T23:03:38.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.708+0000 7f31f5400640 1 -- 172.21.15.28:0/1273523729 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31f01543a0 con 0x7f31d40039b0 2024-02-21T23:03:38.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.708+0000 7f31e7e00640 1 -- 172.21.15.28:0/1273523729 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31e0039070 con 0x7f31d40039b0 2024-02-21T23:03:38.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.708+0000 7f31f5400640 1 -- 172.21.15.28:0/1273523729 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31f00b0130 con 0x7f31d40039b0 2024-02-21T23:03:38.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.710+0000 7f31f5400640 1 -- 172.21.15.28:0/1273523729 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31f004f290 con 0x7f31d40039b0 2024-02-21T23:03:38.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.710+0000 7f31e7e00640 1 -- 172.21.15.28:0/1273523729 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f31e0045020 con 0x7f31d40039b0 2024-02-21T23:03:38.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.712+0000 7f31e7e00640 1 --2- 172.21.15.28:0/1273523729 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f31d006b4d0 0x7f31d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:38.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.712+0000 7f31e7e00640 1 -- 172.21.15.28:0/1273523729 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f31e00abb30 con 0x7f31d40039b0 2024-02-21T23:03:38.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.712+0000 7f31ef400640 1 --2- 172.21.15.28:0/1273523729 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f31d006b4d0 0x7f31d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:38.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.713+0000 7f31ef400640 1 --2- 172.21.15.28:0/1273523729 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f31d006b4d0 0x7f31d006d990 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f31f01560f0 tx=0x7f31dc008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:38.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.716+0000 7f31e7e00640 1 -- 172.21.15.28:0/1273523729 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f31e004c070 con 0x7f31d40039b0 2024-02-21T23:03:38.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.917+0000 7f31f5400640 1 -- 172.21.15.28:0/1273523729 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f31f006d9a0 con 0x7f31d40039b0 2024-02-21T23:03:38.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.918+0000 7f31e7e00640 1 -- 172.21.15.28:0/1273523729 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+11 (secure 0 0 0) 0x7f31e007a510 con 0x7f31d40039b0 2024-02-21T23:03:38.907 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-02-21T23:03:38.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.920+0000 7f31f5400640 1 -- 172.21.15.28:0/1273523729 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f31d006b4d0 msgr2=0x7f31d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:38.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.920+0000 7f31f5400640 1 --2- 172.21.15.28:0/1273523729 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f31d006b4d0 0x7f31d006d990 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f31f01560f0 tx=0x7f31dc008040 comp rx=0 tx=0).stop 2024-02-21T23:03:38.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.920+0000 7f31f5400640 1 -- 172.21.15.28:0/1273523729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31d40039b0 msgr2=0x7f31f0155a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:38.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.920+0000 7f31f5400640 1 --2- 172.21.15.28:0/1273523729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31d40039b0 0x7f31f0155a90 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f31e0037040 tx=0x7f31e0004730 comp rx=0 tx=0).stop 2024-02-21T23:03:38.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.921+0000 7f31f5400640 1 -- 172.21.15.28:0/1273523729 shutdown_connections 2024-02-21T23:03:38.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.921+0000 7f31f5400640 1 --2- 172.21.15.28:0/1273523729 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f31d006b4d0 0x7f31d006d990 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:38.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.921+0000 7f31f5400640 1 --2- 172.21.15.28:0/1273523729 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f31f0153cf0 0x7f31f01396e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:38.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.921+0000 7f31f5400640 1 --2- 172.21.15.28:0/1273523729 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f31f0152ee0 0x7f31f0153340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:38.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.921+0000 7f31f5400640 1 --2- 172.21.15.28:0/1273523729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31d40039b0 0x7f31f0155a90 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:38.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.921+0000 7f31f5400640 1 -- 172.21.15.28:0/1273523729 >> 172.21.15.28:0/1273523729 conn(0x7f31f0073110 msgr2=0x7f31f013ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:38.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.921+0000 7f31f5400640 1 -- 172.21.15.28:0/1273523729 shutdown_connections 2024-02-21T23:03:38.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:38.921+0000 7f31f5400640 1 -- 172.21.15.28:0/1273523729 wait complete. 2024-02-21T23:03:38.928 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-02-21T23:03:39.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.168+0000 7fecef200640 1 Processor -- start 2024-02-21T23:03:39.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.169+0000 7fecef200640 1 -- start start 2024-02-21T23:03:39.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.170+0000 7fecef200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fecf0155db0 0x7fecf015d480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:39.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.170+0000 7fecef200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecf00c5b80 con 0x7fecf00c5df0 2024-02-21T23:03:39.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.170+0000 7fecef200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecf0152950 con 0x7fecf014f000 2024-02-21T23:03:39.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.170+0000 7fecef200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecf00c4fe0 con 0x7fecf0155db0 2024-02-21T23:03:39.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.170+0000 7feced400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fecf00c5df0 0x7fecf01556a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55148/0 (socket says 172.21.15.28:55148) 2024-02-21T23:03:39.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.171+0000 7feced400640 1 -- 172.21.15.28:0/1766110617 learned_addr learned my addr 172.21.15.28:0/1766110617 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:03:39.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.171+0000 7fecee800640 1 --2- 172.21.15.28:0/1766110617 >> v2:172.21.15.28:3300/0 conn(0x7fecf0155db0 0x7fecf015d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:39.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.171+0000 7fececa00640 1 -- 172.21.15.28:0/1766110617 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 507580336 0 0) 0x7fecf00c5b80 con 0x7fecf00c5df0 2024-02-21T23:03:39.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.171+0000 7fececa00640 1 -- 172.21.15.28:0/1766110617 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecd0003520 con 0x7fecf00c5df0 2024-02-21T23:03:39.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.171+0000 7fecee800640 1 -- 172.21.15.28:0/1766110617 >> v1:172.21.15.81:6789/0 conn(0x7fecf014f000 legacy=0x7fecf006ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:39.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.171+0000 7fecee800640 1 -- 172.21.15.28:0/1766110617 >> v1:172.21.15.28:6789/0 conn(0x7fecf00c5df0 legacy=0x7fecf01556a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:39.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.171+0000 7fecee800640 1 -- 172.21.15.28:0/1766110617 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecf00c58a0 con 0x7fecf0155db0 2024-02-21T23:03:39.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.172+0000 7fecee800640 1 --2- 172.21.15.28:0/1766110617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecf0155db0 0x7fecf015d480 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fece8009d90 tx=0x7fece802bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1965452829aa12e9 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:39.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.173+0000 7fececa00640 1 -- 172.21.15.28:0/1766110617 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece8039070 con 0x7fecf0155db0 2024-02-21T23:03:39.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.173+0000 7fececa00640 1 -- 172.21.15.28:0/1766110617 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fece802ccf0 con 0x7fecf0155db0 2024-02-21T23:03:39.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.173+0000 7fececa00640 1 -- 172.21.15.28:0/1766110617 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece8034660 con 0x7fecf0155db0 2024-02-21T23:03:39.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.173+0000 7fecef200640 1 -- 172.21.15.28:0/1766110617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecf0155db0 msgr2=0x7fecf015d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:39.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.173+0000 7fecef200640 1 --2- 172.21.15.28:0/1766110617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecf0155db0 0x7fecf015d480 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fece8009d90 tx=0x7fece802bf40 comp rx=0 tx=0).stop 2024-02-21T23:03:39.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.173+0000 7fecef200640 1 -- 172.21.15.28:0/1766110617 shutdown_connections 2024-02-21T23:03:39.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.173+0000 7fecef200640 1 --2- 172.21.15.28:0/1766110617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecf0155db0 0x7fecf015d480 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:39.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.173+0000 7fecef200640 1 -- 172.21.15.28:0/1766110617 >> 172.21.15.28:0/1766110617 conn(0x7fecf00a8530 msgr2=0x7fecf00a8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:39.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.173+0000 7fecef200640 1 -- 172.21.15.28:0/1766110617 shutdown_connections 2024-02-21T23:03:39.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.174+0000 7fecef200640 1 -- 172.21.15.28:0/1766110617 wait complete. 2024-02-21T23:03:39.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.175+0000 7fecef200640 1 Processor -- start 2024-02-21T23:03:39.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.175+0000 7fecef200640 1 -- start start 2024-02-21T23:03:39.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.176+0000 7fecef200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecf00c5df0 0x7fecf01eb0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:39.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.176+0000 7fecef200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fecf014f000 0x7fecf00bfe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:39.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.176+0000 7fecef200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fecf0155db0 0x7fecf01536e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:39.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.176+0000 7fecef200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecf006f330 con 0x7fecf00c5df0 2024-02-21T23:03:39.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.176+0000 7fecef200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecf00c5650 con 0x7fecf0155db0 2024-02-21T23:03:39.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.176+0000 7fecef200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecf006f4b0 con 0x7fecf014f000 2024-02-21T23:03:39.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.177+0000 7fecede00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecf00c5df0 0x7fecf01eb0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:39.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.177+0000 7fecede00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecf00c5df0 0x7fecf01eb0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44442/0 (socket says 172.21.15.28:44442) 2024-02-21T23:03:39.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.177+0000 7fecede00640 1 -- 172.21.15.28:0/455007596 learned_addr learned my addr 172.21.15.28:0/455007596 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:39.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.177+0000 7feced400640 1 --2- 172.21.15.28:0/455007596 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fecf014f000 0x7fecf00bfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:39.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.177+0000 7fecee800640 1 --2- 172.21.15.28:0/455007596 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fecf0155db0 0x7fecf01536e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:39.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.177+0000 7fecede00640 1 -- 172.21.15.28:0/455007596 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fecf014f000 msgr2=0x7fecf00bfe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:39.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.177+0000 7fecede00640 1 --2- 172.21.15.28:0/455007596 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fecf014f000 0x7fecf00bfe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:39.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.177+0000 7fecede00640 1 -- 172.21.15.28:0/455007596 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fecf0155db0 msgr2=0x7fecf01536e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:39.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.177+0000 7fecede00640 1 --2- 172.21.15.28:0/455007596 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fecf0155db0 0x7fecf01536e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:39.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.177+0000 7fecede00640 1 -- 172.21.15.28:0/455007596 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecf00a7580 con 0x7fecf00c5df0 2024-02-21T23:03:39.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.177+0000 7feced400640 1 --2- 172.21.15.28:0/455007596 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fecf014f000 0x7fecf00bfe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:03:39.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.178+0000 7fecede00640 1 --2- 172.21.15.28:0/455007596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecf00c5df0 0x7fecf01eb0f0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fece000c450 tx=0x7fece000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:39.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.178+0000 7fece6000640 1 -- 172.21.15.28:0/455007596 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece00052d0 con 0x7fecf00c5df0 2024-02-21T23:03:39.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.178+0000 7fece6000640 1 -- 172.21.15.28:0/455007596 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fece0005470 con 0x7fecf00c5df0 2024-02-21T23:03:39.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.178+0000 7fece6000640 1 -- 172.21.15.28:0/455007596 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece0012870 con 0x7fecf00c5df0 2024-02-21T23:03:39.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.178+0000 7fecef200640 1 -- 172.21.15.28:0/455007596 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecf00a77f0 con 0x7fecf00c5df0 2024-02-21T23:03:39.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.178+0000 7fecef200640 1 -- 172.21.15.28:0/455007596 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecf00a80b0 con 0x7fecf00c5df0 2024-02-21T23:03:39.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.179+0000 7fecef200640 1 -- 172.21.15.28:0/455007596 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecf004f290 con 0x7fecf00c5df0 2024-02-21T23:03:39.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.180+0000 7fece6000640 1 -- 172.21.15.28:0/455007596 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fece0012a10 con 0x7fecf00c5df0 2024-02-21T23:03:39.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.181+0000 7fece6000640 1 --2- 172.21.15.28:0/455007596 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7feccc06b4d0 0x7feccc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:39.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.181+0000 7fece6000640 1 -- 172.21.15.28:0/455007596 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6735+0+0 (secure 0 0 0) 0x7fece008a250 con 0x7fecf00c5df0 2024-02-21T23:03:39.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.182+0000 7feced400640 1 --2- 172.21.15.28:0/455007596 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7feccc06b4d0 0x7feccc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:39.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.183+0000 7feced400640 1 --2- 172.21.15.28:0/455007596 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7feccc06b4d0 0x7feccc06d990 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fecf0153320 tx=0x7fecd8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:39.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.185+0000 7fece6000640 1 -- 172.21.15.28:0/455007596 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fece0058cc0 con 0x7fecf00c5df0 2024-02-21T23:03:39.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.384+0000 7fecef200640 1 -- 172.21.15.28:0/455007596 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fecf006eb20 con 0x7fecf00c5df0 2024-02-21T23:03:39.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.385+0000 7fece6000640 1 -- 172.21.15.28:0/455007596 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) v1 ==== 103+0+11 (secure 0 0 0) 0x7fece00585c0 con 0x7fecf00c5df0 2024-02-21T23:03:39.374 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-02-21T23:03:39.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.388+0000 7fecef200640 1 -- 172.21.15.28:0/455007596 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7feccc06b4d0 msgr2=0x7feccc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:39.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.388+0000 7fecef200640 1 --2- 172.21.15.28:0/455007596 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7feccc06b4d0 0x7feccc06d990 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fecf0153320 tx=0x7fecd8009040 comp rx=0 tx=0).stop 2024-02-21T23:03:39.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.388+0000 7fecef200640 1 -- 172.21.15.28:0/455007596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecf00c5df0 msgr2=0x7fecf01eb0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:39.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.388+0000 7fecef200640 1 --2- 172.21.15.28:0/455007596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecf00c5df0 0x7fecf01eb0f0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fece000c450 tx=0x7fece000c910 comp rx=0 tx=0).stop 2024-02-21T23:03:39.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.388+0000 7fecef200640 1 -- 172.21.15.28:0/455007596 shutdown_connections 2024-02-21T23:03:39.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.388+0000 7fecef200640 1 --2- 172.21.15.28:0/455007596 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7feccc06b4d0 0x7feccc06d990 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:39.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.388+0000 7fecef200640 1 --2- 172.21.15.28:0/455007596 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fecf014f000 0x7fecf00bfe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:39.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.388+0000 7fecef200640 1 --2- 172.21.15.28:0/455007596 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fecf0155db0 0x7fecf01536e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:39.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.388+0000 7fecef200640 1 --2- 172.21.15.28:0/455007596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecf00c5df0 0x7fecf01eb0f0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:39.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.388+0000 7fecef200640 1 -- 172.21.15.28:0/455007596 >> 172.21.15.28:0/455007596 conn(0x7fecf00a8530 msgr2=0x7fecf014be10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:39.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.388+0000 7fecef200640 1 -- 172.21.15.28:0/455007596 shutdown_connections 2024-02-21T23:03:39.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.388+0000 7fecef200640 1 -- 172.21.15.28:0/455007596 wait complete. 2024-02-21T23:03:39.410 INFO:tasks.cephfs_test_runner:Starting test: test_join_fs_drop (tasks.cephfs.test_failover.TestClusterAffinity) That unsetting mds_join_fs will not cause failover if no high-affinity standby exists. 2024-02-21T23:03:39.411 INFO:tasks.cephfs_test_runner:test_join_fs_drop (tasks.cephfs.test_failover.TestClusterAffinity) 2024-02-21T23:03:39.411 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:03:39.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.648+0000 7f29fb200640 1 Processor -- start 2024-02-21T23:03:39.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.648+0000 7f29fb200640 1 -- start start 2024-02-21T23:03:39.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.650+0000 7f29fb200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f29fc0a6160 0x7f29fc155180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:39.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.650+0000 7f29fb200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29fc155760 con 0x7f29fc1521c0 2024-02-21T23:03:39.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.650+0000 7f29fb200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29fc155a40 con 0x7f29fc14f0b0 2024-02-21T23:03:39.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.650+0000 7f29fb200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29fc151d30 con 0x7f29fc0a6160 2024-02-21T23:03:39.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.650+0000 7f29fa800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f29fc1521c0 0x7f29fc15c510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55154/0 (socket says 172.21.15.28:55154) 2024-02-21T23:03:39.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.650+0000 7f29fa800640 1 -- 172.21.15.28:0/3854357937 learned_addr learned my addr 172.21.15.28:0/3854357937 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:03:39.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.650+0000 7f29f3e00640 1 --2- 172.21.15.28:0/3854357937 >> v2:172.21.15.81:3300/0 conn(0x7f29fc0a6160 0x7f29fc155180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:39.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.651+0000 7f29f9400640 1 -- 172.21.15.28:0/3854357937 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3111218365 0 0) 0x7f29fc155760 con 0x7f29fc1521c0 2024-02-21T23:03:39.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.651+0000 7f29f9400640 1 -- 172.21.15.28:0/3854357937 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29e0003520 con 0x7f29fc1521c0 2024-02-21T23:03:39.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.651+0000 7f29f9400640 1 -- 172.21.15.28:0/3854357937 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 174538173 0 0) 0x7f29e0003520 con 0x7f29fc1521c0 2024-02-21T23:03:39.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.651+0000 7f29f9400640 1 -- 172.21.15.28:0/3854357937 >> v2:172.21.15.81:3300/0 conn(0x7f29fc0a6160 msgr2=0x7f29fc155180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:39.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.651+0000 7f29f9400640 1 --2- 172.21.15.28:0/3854357937 >> v2:172.21.15.81:3300/0 conn(0x7f29fc0a6160 0x7f29fc155180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:39.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.651+0000 7f29f9400640 1 -- 172.21.15.28:0/3854357937 >> v1:172.21.15.81:6789/0 conn(0x7f29fc14f0b0 legacy=0x7f29fc0c2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:39.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.651+0000 7f29f9400640 1 -- 172.21.15.28:0/3854357937 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29fc155d20 con 0x7f29fc1521c0 2024-02-21T23:03:39.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.651+0000 7f29f9400640 1 -- 172.21.15.28:0/3854357937 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f29f40019f0 con 0x7f29fc1521c0 2024-02-21T23:03:39.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.651+0000 7f29f9400640 1 -- 172.21.15.28:0/3854357937 >> v1:172.21.15.28:6789/0 conn(0x7f29fc1521c0 legacy=0x7f29fc15c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:39.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.651+0000 7f29f9400640 1 --2- 172.21.15.28:0/3854357937 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29e00039b0 0x7f29e0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:39.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.651+0000 7f29f9400640 1 --2- 172.21.15.28:0/3854357937 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29e0004800 0x7f29e0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:39.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.651+0000 7f29f9400640 1 --2- 172.21.15.28:0/3854357937 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f29e00051c0 0x7f29e000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:39.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.651+0000 7f29f9400640 1 -- 172.21.15.28:0/3854357937 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29fc151d30 con 0x7f29e0004800 2024-02-21T23:03:39.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.651+0000 7f29f9400640 1 -- 172.21.15.28:0/3854357937 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e000c060 con 0x7f29e00039b0 2024-02-21T23:03:39.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.651+0000 7f29f9400640 1 -- 172.21.15.28:0/3854357937 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29e000c260 con 0x7f29e00051c0 2024-02-21T23:03:39.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.651+0000 7f29f3e00640 1 --2- 172.21.15.28:0/3854357937 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29e0004800 0x7f29e0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:39.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.652+0000 7f29f9e00640 1 --2- 172.21.15.28:0/3854357937 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29e00039b0 0x7f29e0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:39.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.652+0000 7f29f3e00640 1 -- 172.21.15.28:0/3854357937 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f29e00051c0 msgr2=0x7f29e000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:39.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.652+0000 7f29f3e00640 1 --2- 172.21.15.28:0/3854357937 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f29e00051c0 0x7f29e000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:39.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.652+0000 7f29f3e00640 1 -- 172.21.15.28:0/3854357937 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29e00039b0 msgr2=0x7f29e0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:39.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.652+0000 7f29f3e00640 1 --2- 172.21.15.28:0/3854357937 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29e00039b0 0x7f29e0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:39.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.652+0000 7f29f3e00640 1 -- 172.21.15.28:0/3854357937 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29e0003520 con 0x7f29e0004800 2024-02-21T23:03:39.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.652+0000 7f29fa800640 1 -- 172.21.15.28:0/3854357937 reap_dead start 2024-02-21T23:03:39.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.653+0000 7f29f3e00640 1 --2- 172.21.15.28:0/3854357937 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29e0004800 0x7f29e0004c80 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f29e4009ac0 tx=0x7f29e0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18cf3f31394207c server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:39.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.653+0000 7f29f9400640 1 -- 172.21.15.28:0/3854357937 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29e4039070 con 0x7f29e0004800 2024-02-21T23:03:39.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.654+0000 7f29f9400640 1 -- 172.21.15.28:0/3854357937 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29e402c6b0 con 0x7f29e0004800 2024-02-21T23:03:39.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.654+0000 7f29f9400640 1 -- 172.21.15.28:0/3854357937 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29e4009690 con 0x7f29e0004800 2024-02-21T23:03:39.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.654+0000 7f29fb200640 1 -- 172.21.15.28:0/3854357937 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29e0004800 msgr2=0x7f29e0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:39.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.654+0000 7f29fb200640 1 --2- 172.21.15.28:0/3854357937 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29e0004800 0x7f29e0004c80 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f29e4009ac0 tx=0x7f29e0005830 comp rx=0 tx=0).stop 2024-02-21T23:03:39.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.654+0000 7f29fb200640 1 -- 172.21.15.28:0/3854357937 shutdown_connections 2024-02-21T23:03:39.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.654+0000 7f29fb200640 1 --2- 172.21.15.28:0/3854357937 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29e0004800 0x7f29e0004c80 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:39.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.654+0000 7f29fb200640 1 -- 172.21.15.28:0/3854357937 >> 172.21.15.28:0/3854357937 conn(0x7f29fc06b630 msgr2=0x7f29fc0c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:39.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.654+0000 7f29fb200640 1 -- 172.21.15.28:0/3854357937 shutdown_connections 2024-02-21T23:03:39.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.654+0000 7f29fb200640 1 -- 172.21.15.28:0/3854357937 wait complete. 2024-02-21T23:03:39.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.656+0000 7f29fb200640 1 Processor -- start 2024-02-21T23:03:39.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.656+0000 7f29fb200640 1 -- start start 2024-02-21T23:03:39.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.657+0000 7f29fb200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29e0004800 0x7f29fc0ab990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:39.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.657+0000 7f29fb200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f29fc0a9e60 0x7f29fc1eb100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:39.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.657+0000 7f29fb200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29fc1541f0 0x7f29fc0aa540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:39.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.657+0000 7f29fb200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29fc0a7f70 con 0x7f29e0004800 2024-02-21T23:03:39.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.657+0000 7f29fb200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29fc0a68d0 con 0x7f29fc1541f0 2024-02-21T23:03:39.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.657+0000 7f29fb200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29fc0a80f0 con 0x7f29fc0a9e60 2024-02-21T23:03:39.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.657+0000 7f29f9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29e0004800 0x7f29fc0ab990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:39.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.657+0000 7f29f9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29e0004800 0x7f29fc0ab990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44458/0 (socket says 172.21.15.28:44458) 2024-02-21T23:03:39.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.657+0000 7f29f9e00640 1 -- 172.21.15.28:0/2857566170 learned_addr learned my addr 172.21.15.28:0/2857566170 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:39.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.658+0000 7f29f3e00640 1 --2- 172.21.15.28:0/2857566170 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f29fc0a9e60 0x7f29fc1eb100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:39.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.658+0000 7f29fa800640 1 --2- 172.21.15.28:0/2857566170 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29fc1541f0 0x7f29fc0aa540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:39.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.658+0000 7f29f9e00640 1 -- 172.21.15.28:0/2857566170 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f29fc0a9e60 msgr2=0x7f29fc1eb100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:39.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.658+0000 7f29f9e00640 1 --2- 172.21.15.28:0/2857566170 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f29fc0a9e60 0x7f29fc1eb100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:39.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.658+0000 7f29f9e00640 1 -- 172.21.15.28:0/2857566170 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29fc1541f0 msgr2=0x7f29fc0aa540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:39.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.658+0000 7f29f9e00640 1 --2- 172.21.15.28:0/2857566170 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29fc1541f0 0x7f29fc0aa540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:39.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.658+0000 7f29f9e00640 1 -- 172.21.15.28:0/2857566170 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29fc1ea640 con 0x7f29e0004800 2024-02-21T23:03:39.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.658+0000 7f29fa800640 1 --2- 172.21.15.28:0/2857566170 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29fc1541f0 0x7f29fc0aa540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:03:39.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.658+0000 7f29f9e00640 1 --2- 172.21.15.28:0/2857566170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29e0004800 0x7f29fc0ab990 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f29ec00e550 tx=0x7f29ec00ea10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:39.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.658+0000 7f29f2000640 1 -- 172.21.15.28:0/2857566170 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29ec0052e0 con 0x7f29e0004800 2024-02-21T23:03:39.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.658+0000 7f29f2000640 1 -- 172.21.15.28:0/2857566170 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29ec018070 con 0x7f29e0004800 2024-02-21T23:03:39.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.658+0000 7f29f2000640 1 -- 172.21.15.28:0/2857566170 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29ec003810 con 0x7f29e0004800 2024-02-21T23:03:39.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.659+0000 7f29fb200640 1 -- 172.21.15.28:0/2857566170 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29fc0c3990 con 0x7f29e0004800 2024-02-21T23:03:39.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.659+0000 7f29fb200640 1 -- 172.21.15.28:0/2857566170 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29fc1ea900 con 0x7f29e0004800 2024-02-21T23:03:39.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.660+0000 7f29fb200640 1 -- 172.21.15.28:0/2857566170 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29fc04f290 con 0x7f29e0004800 2024-02-21T23:03:39.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.661+0000 7f29f2000640 1 -- 172.21.15.28:0/2857566170 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f29ec0039b0 con 0x7f29e0004800 2024-02-21T23:03:39.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.662+0000 7f29f2000640 1 --2- 172.21.15.28:0/2857566170 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f29d806b500 0x7f29d806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:39.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.662+0000 7f29f2000640 1 -- 172.21.15.28:0/2857566170 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f29ec08a200 con 0x7f29e0004800 2024-02-21T23:03:39.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.662+0000 7f29f3e00640 1 --2- 172.21.15.28:0/2857566170 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f29d806b500 0x7f29d806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:39.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.663+0000 7f29f3e00640 1 --2- 172.21.15.28:0/2857566170 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f29d806b500 0x7f29d806d9c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f29e4036000 tx=0x7f29e4030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:39.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.666+0000 7f29f2000640 1 -- 172.21.15.28:0/2857566170 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f29ec058d10 con 0x7f29e0004800 2024-02-21T23:03:39.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.867+0000 7f29fb200640 1 -- 172.21.15.28:0/2857566170 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f29fc0c0be0 con 0x7f29e0004800 2024-02-21T23:03:39.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.869+0000 7f29f2000640 1 -- 172.21.15.28:0/2857566170 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+18843 (secure 0 0 0) 0x7f29ec00ce10 con 0x7f29e0004800 2024-02-21T23:03:39.859 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:03:39.859 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":26,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:03:23.860109+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:02:35.795158+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-21T23:02:36.840110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-02-21T23:02:40.441388+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":3,"to":2}]},{"pgid":"3.12","mappings":[{"from":0,"to":1}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:03:39.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.872+0000 7f29fb200640 1 -- 172.21.15.28:0/2857566170 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f29d806b500 msgr2=0x7f29d806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:39.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.872+0000 7f29fb200640 1 --2- 172.21.15.28:0/2857566170 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f29d806b500 0x7f29d806d9c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f29e4036000 tx=0x7f29e4030000 comp rx=0 tx=0).stop 2024-02-21T23:03:39.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.872+0000 7f29fb200640 1 -- 172.21.15.28:0/2857566170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29e0004800 msgr2=0x7f29fc0ab990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:39.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.872+0000 7f29fb200640 1 --2- 172.21.15.28:0/2857566170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29e0004800 0x7f29fc0ab990 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f29ec00e550 tx=0x7f29ec00ea10 comp rx=0 tx=0).stop 2024-02-21T23:03:39.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.872+0000 7f29fb200640 1 -- 172.21.15.28:0/2857566170 shutdown_connections 2024-02-21T23:03:39.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.872+0000 7f29fb200640 1 --2- 172.21.15.28:0/2857566170 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f29d806b500 0x7f29d806d9c0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:39.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.872+0000 7f29fb200640 1 --2- 172.21.15.28:0/2857566170 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f29fc0a9e60 0x7f29fc1eb100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:39.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.872+0000 7f29fb200640 1 --2- 172.21.15.28:0/2857566170 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29fc1541f0 0x7f29fc0aa540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:39.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.872+0000 7f29fb200640 1 --2- 172.21.15.28:0/2857566170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29e0004800 0x7f29fc0ab990 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:39.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.872+0000 7f29fb200640 1 -- 172.21.15.28:0/2857566170 >> 172.21.15.28:0/2857566170 conn(0x7f29fc06b630 msgr2=0x7f29fc150400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:39.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.873+0000 7f29fb200640 1 -- 172.21.15.28:0/2857566170 shutdown_connections 2024-02-21T23:03:39.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:39.873+0000 7f29fb200640 1 -- 172.21.15.28:0/2857566170 wait complete. 2024-02-21T23:03:39.883 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-21T23:02:30.728329+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-21T23:02:35.795158+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-02-21T23:02:36.840110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-02-21T23:02:40.441388+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-02-21T23:03:39.883 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-21T23:03:40.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.119+0000 7f4601200640 1 Processor -- start 2024-02-21T23:03:40.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.120+0000 7f4601200640 1 -- start start 2024-02-21T23:03:40.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.121+0000 7f4601200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f45fc0c7590 0x7f45fc0c6c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:40.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.121+0000 7f4601200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f45fc0c5cc0 0x7f45fc1524e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:40.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.122+0000 7f4601200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45fc159580 con 0x7f45fc152a20 2024-02-21T23:03:40.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.122+0000 7f4601200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45fc0c6230 con 0x7f45fc0c7590 2024-02-21T23:03:40.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.122+0000 7f4601200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45fc1587c0 con 0x7f45fc0c5cc0 2024-02-21T23:03:40.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.122+0000 7f45f9e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f45fc0c7590 0x7f45fc0c6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:40.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.122+0000 7f45f9e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f45fc0c7590 0x7f45fc0c6c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.28:53582/0 (socket says 172.21.15.28:53582) 2024-02-21T23:03:40.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.122+0000 7f45f9e00640 1 -- 172.21.15.28:0/552575900 learned_addr learned my addr 172.21.15.28:0/552575900 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:40.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.122+0000 7f45f9400640 1 --2- 172.21.15.28:0/552575900 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f45fc0c5cc0 0x7f45fc1524e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:40.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.123+0000 7f45f9e00640 1 -- 172.21.15.28:0/552575900 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f45fc0c5cc0 msgr2=0x7f45fc1524e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:40.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.123+0000 7f45f9e00640 1 --2- 172.21.15.28:0/552575900 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f45fc0c5cc0 0x7f45fc1524e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:40.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.123+0000 7f45f9e00640 1 -- 172.21.15.28:0/552575900 >> v1:172.21.15.81:6789/0 conn(0x7f45fc152a20 legacy=0x7f45fc0c3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:40.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.123+0000 7f45f9e00640 1 -- 172.21.15.28:0/552575900 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45fc158aa0 con 0x7f45fc0c7590 2024-02-21T23:03:40.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.124+0000 7f45f9e00640 1 --2- 172.21.15.28:0/552575900 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f45fc0c7590 0x7f45fc0c6c60 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f45fc0c63b0 tx=0x7f45ec02b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db730f0b4ff86bd server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:40.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.125+0000 7f45f8a00640 1 -- 172.21.15.28:0/552575900 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45ec035020 con 0x7f45fc0c7590 2024-02-21T23:03:40.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.125+0000 7f45f8a00640 1 -- 172.21.15.28:0/552575900 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45ec031640 con 0x7f45fc0c7590 2024-02-21T23:03:40.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.125+0000 7f45f8a00640 1 -- 172.21.15.28:0/552575900 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45ec02d040 con 0x7f45fc0c7590 2024-02-21T23:03:40.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.126+0000 7f4601200640 1 -- 172.21.15.28:0/552575900 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f45fc0c7590 msgr2=0x7f45fc0c6c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:40.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.126+0000 7f4601200640 1 --2- 172.21.15.28:0/552575900 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f45fc0c7590 0x7f45fc0c6c60 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f45fc0c63b0 tx=0x7f45ec02b8f0 comp rx=0 tx=0).stop 2024-02-21T23:03:40.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.126+0000 7f4601200640 1 -- 172.21.15.28:0/552575900 shutdown_connections 2024-02-21T23:03:40.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.126+0000 7f4601200640 1 --2- 172.21.15.28:0/552575900 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f45fc0c5cc0 0x7f45fc1524e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:40.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.126+0000 7f4601200640 1 --2- 172.21.15.28:0/552575900 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f45fc0c7590 0x7f45fc0c6c60 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:40.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.126+0000 7f4601200640 1 -- 172.21.15.28:0/552575900 >> 172.21.15.28:0/552575900 conn(0x7f45fc08afe0 msgr2=0x7f45fc0725a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:40.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.126+0000 7f4601200640 1 -- 172.21.15.28:0/552575900 shutdown_connections 2024-02-21T23:03:40.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.126+0000 7f4601200640 1 -- 172.21.15.28:0/552575900 wait complete. 2024-02-21T23:03:40.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.128+0000 7f4601200640 1 Processor -- start 2024-02-21T23:03:40.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.128+0000 7f4601200640 1 -- start start 2024-02-21T23:03:40.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.130+0000 7f4601200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f45fc0c5cc0 0x7f45fc155a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:40.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.130+0000 7f4601200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45fc0c7590 0x7f45fc0ca190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:40.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.130+0000 7f4601200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f45fc152a20 0x7f45fc0ca6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:40.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.130+0000 7f4601200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45fc0c6720 con 0x7f45fc0c7590 2024-02-21T23:03:40.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.130+0000 7f4601200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45fc159060 con 0x7f45fc0c5cc0 2024-02-21T23:03:40.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.130+0000 7f4601200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45fc0c68a0 con 0x7f45fc152a20 2024-02-21T23:03:40.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.130+0000 7f45f9400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45fc0c7590 0x7f45fc0ca190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:40.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.130+0000 7f45f9400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45fc0c7590 0x7f45fc0ca190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44468/0 (socket says 172.21.15.28:44468) 2024-02-21T23:03:40.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.130+0000 7f45f9400640 1 -- 172.21.15.28:0/2673191418 learned_addr learned my addr 172.21.15.28:0/2673191418 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:40.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.130+0000 7f45f9e00640 1 --2- 172.21.15.28:0/2673191418 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f45fc0c5cc0 0x7f45fc155a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:40.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.130+0000 7f45fa800640 1 --2- 172.21.15.28:0/2673191418 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f45fc152a20 0x7f45fc0ca6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:40.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.131+0000 7f45f9400640 1 -- 172.21.15.28:0/2673191418 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f45fc152a20 msgr2=0x7f45fc0ca6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:40.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.131+0000 7f45f9400640 1 --2- 172.21.15.28:0/2673191418 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f45fc152a20 0x7f45fc0ca6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:40.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.131+0000 7f45f9400640 1 -- 172.21.15.28:0/2673191418 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f45fc0c5cc0 msgr2=0x7f45fc155a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:40.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.131+0000 7f45f9400640 1 --2- 172.21.15.28:0/2673191418 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f45fc0c5cc0 0x7f45fc155a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:40.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.131+0000 7f45f9400640 1 -- 172.21.15.28:0/2673191418 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45fc155f40 con 0x7f45fc0c7590 2024-02-21T23:03:40.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.131+0000 7f45f9400640 1 --2- 172.21.15.28:0/2673191418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45fc0c7590 0x7f45fc0ca190 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f45e4007e40 tx=0x7f45e400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:40.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.131+0000 7f45f2000640 1 -- 172.21.15.28:0/2673191418 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45e4014620 con 0x7f45fc0c7590 2024-02-21T23:03:40.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.131+0000 7f45f2000640 1 -- 172.21.15.28:0/2673191418 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45e4019070 con 0x7f45fc0c7590 2024-02-21T23:03:40.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.131+0000 7f45f2000640 1 -- 172.21.15.28:0/2673191418 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45e4014a00 con 0x7f45fc0c7590 2024-02-21T23:03:40.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.131+0000 7f4601200640 1 -- 172.21.15.28:0/2673191418 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45fc1f3410 con 0x7f45fc0c7590 2024-02-21T23:03:40.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.132+0000 7f4601200640 1 -- 172.21.15.28:0/2673191418 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45fc1495c0 con 0x7f45fc0c7590 2024-02-21T23:03:40.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.133+0000 7f4601200640 1 -- 172.21.15.28:0/2673191418 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45fc067f30 con 0x7f45fc0c7590 2024-02-21T23:03:40.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.134+0000 7f45f2000640 1 -- 172.21.15.28:0/2673191418 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f45e4014ba0 con 0x7f45fc0c7590 2024-02-21T23:03:40.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.135+0000 7f45f2000640 1 --2- 172.21.15.28:0/2673191418 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f45d806b4d0 0x7f45d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:40.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.135+0000 7f45f2000640 1 -- 172.21.15.28:0/2673191418 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f45e408ae00 con 0x7f45fc0c7590 2024-02-21T23:03:40.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.136+0000 7f45f9e00640 1 --2- 172.21.15.28:0/2673191418 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f45d806b4d0 0x7f45d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:40.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.137+0000 7f45f9e00640 1 --2- 172.21.15.28:0/2673191418 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f45d806b4d0 0x7f45d806d990 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f45ec036000 tx=0x7f45ec03c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:40.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.140+0000 7f45f2000640 1 -- 172.21.15.28:0/2673191418 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f45e4059870 con 0x7f45fc0c7590 2024-02-21T23:03:40.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.340+0000 7f4601200640 1 -- 172.21.15.28:0/2673191418 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f45fc14d870 con 0x7f45fc0c7590 2024-02-21T23:03:40.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.340+0000 7f45f2000640 1 -- 172.21.15.28:0/2673191418 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) v1 ==== 93+0+10 (secure 0 0 0) 0x7f45e4059170 con 0x7f45fc0c7590 2024-02-21T23:03:40.329 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-02-21T23:03:40.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.343+0000 7f4601200640 1 -- 172.21.15.28:0/2673191418 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f45d806b4d0 msgr2=0x7f45d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:40.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.343+0000 7f4601200640 1 --2- 172.21.15.28:0/2673191418 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f45d806b4d0 0x7f45d806d990 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f45ec036000 tx=0x7f45ec03c040 comp rx=0 tx=0).stop 2024-02-21T23:03:40.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.343+0000 7f4601200640 1 -- 172.21.15.28:0/2673191418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45fc0c7590 msgr2=0x7f45fc0ca190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:40.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.343+0000 7f4601200640 1 --2- 172.21.15.28:0/2673191418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45fc0c7590 0x7f45fc0ca190 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f45e4007e40 tx=0x7f45e400e4b0 comp rx=0 tx=0).stop 2024-02-21T23:03:40.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.343+0000 7f4601200640 1 -- 172.21.15.28:0/2673191418 shutdown_connections 2024-02-21T23:03:40.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.343+0000 7f4601200640 1 --2- 172.21.15.28:0/2673191418 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f45d806b4d0 0x7f45d806d990 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:40.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.343+0000 7f4601200640 1 --2- 172.21.15.28:0/2673191418 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f45fc152a20 0x7f45fc0ca6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:40.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.343+0000 7f4601200640 1 --2- 172.21.15.28:0/2673191418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45fc0c7590 0x7f45fc0ca190 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:40.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.343+0000 7f4601200640 1 --2- 172.21.15.28:0/2673191418 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f45fc0c5cc0 0x7f45fc155a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:40.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.343+0000 7f4601200640 1 -- 172.21.15.28:0/2673191418 >> 172.21.15.28:0/2673191418 conn(0x7f45fc08afe0 msgr2=0x7f45fc150980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:40.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.343+0000 7f4601200640 1 -- 172.21.15.28:0/2673191418 shutdown_connections 2024-02-21T23:03:40.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.343+0000 7f4601200640 1 -- 172.21.15.28:0/2673191418 wait complete. 2024-02-21T23:03:40.353 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-21T23:03:40.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.585+0000 7f5557400640 1 Processor -- start 2024-02-21T23:03:40.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.586+0000 7f5557400640 1 -- start start 2024-02-21T23:03:40.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.587+0000 7f5557400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f55501569d0 0x7f555006e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:40.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.587+0000 7f5557400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f555006d190 0x7f55500c5d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:40.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.587+0000 7f5557400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f555006f370 con 0x7f55500c62d0 2024-02-21T23:03:40.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.587+0000 7f5557400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f555015ca90 con 0x7f55501569d0 2024-02-21T23:03:40.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.587+0000 7f5557400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f555015cd70 con 0x7f555006d190 2024-02-21T23:03:40.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.588+0000 7f5555400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f55500c62d0 0x7f5550159470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55156/0 (socket says 172.21.15.28:55156) 2024-02-21T23:03:40.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.588+0000 7f5555400640 1 -- 172.21.15.28:0/3196206601 learned_addr learned my addr 172.21.15.28:0/3196206601 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:03:40.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.588+0000 7f5554a00640 1 --2- 172.21.15.28:0/3196206601 >> v2:172.21.15.81:3300/0 conn(0x7f55501569d0 0x7f555006e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:40.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.588+0000 7f554fe00640 1 --2- 172.21.15.28:0/3196206601 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f555006d190 0x7f55500c5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:40.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.588+0000 7f554f400640 1 -- 172.21.15.28:0/3196206601 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3083481066 0 0) 0x7f555006f370 con 0x7f55500c62d0 2024-02-21T23:03:40.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.588+0000 7f554f400640 1 -- 172.21.15.28:0/3196206601 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5534003520 con 0x7f55500c62d0 2024-02-21T23:03:40.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.589+0000 7f554f400640 1 -- 172.21.15.28:0/3196206601 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1021539149 0 0) 0x7f5534003520 con 0x7f55500c62d0 2024-02-21T23:03:40.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.589+0000 7f554f400640 1 -- 172.21.15.28:0/3196206601 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f555006d190 msgr2=0x7f55500c5d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:40.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.589+0000 7f554f400640 1 --2- 172.21.15.28:0/3196206601 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f555006d190 0x7f55500c5d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:40.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.589+0000 7f554f400640 1 -- 172.21.15.28:0/3196206601 >> v2:172.21.15.81:3300/0 conn(0x7f55501569d0 msgr2=0x7f555006e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:40.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.589+0000 7f554f400640 1 --2- 172.21.15.28:0/3196206601 >> v2:172.21.15.81:3300/0 conn(0x7f55501569d0 0x7f555006e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:40.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.589+0000 7f554f400640 1 -- 172.21.15.28:0/3196206601 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f555015d050 con 0x7f55500c62d0 2024-02-21T23:03:40.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.589+0000 7f554f400640 1 -- 172.21.15.28:0/3196206601 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f5548001670 con 0x7f55500c62d0 2024-02-21T23:03:40.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.589+0000 7f5554a00640 1 --2- 172.21.15.28:0/3196206601 >> v2:172.21.15.81:3300/0 conn(0x7f55501569d0 0x7f555006e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-21T23:03:40.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.589+0000 7f554f400640 1 -- 172.21.15.28:0/3196206601 >> v1:172.21.15.28:6789/0 conn(0x7f55500c62d0 legacy=0x7f5550159470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:40.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.589+0000 7f554f400640 1 --2- 172.21.15.28:0/3196206601 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55340039b0 0x7f5534003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:40.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.589+0000 7f554f400640 1 --2- 172.21.15.28:0/3196206601 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f555006d190 0x7f5534004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:40.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.589+0000 7f554f400640 1 --2- 172.21.15.28:0/3196206601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5534004d40 0x7f55340051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:40.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.589+0000 7f554f400640 1 -- 172.21.15.28:0/3196206601 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f555015ca90 con 0x7f5534004d40 2024-02-21T23:03:40.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.589+0000 7f554f400640 1 -- 172.21.15.28:0/3196206601 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f555015cd70 con 0x7f55340039b0 2024-02-21T23:03:40.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.589+0000 7f554f400640 1 -- 172.21.15.28:0/3196206601 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f553400b9c0 con 0x7f555006d190 2024-02-21T23:03:40.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.589+0000 7f5555400640 1 --2- 172.21.15.28:0/3196206601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5534004d40 0x7f55340051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:40.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.589+0000 7f5554a00640 1 --2- 172.21.15.28:0/3196206601 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55340039b0 0x7f5534003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:40.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.589+0000 7f554fe00640 1 --2- 172.21.15.28:0/3196206601 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f555006d190 0x7f5534004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:40.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.590+0000 7f5555400640 1 -- 172.21.15.28:0/3196206601 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f555006d190 msgr2=0x7f5534004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:40.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.590+0000 7f5555400640 1 --2- 172.21.15.28:0/3196206601 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f555006d190 0x7f5534004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:40.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.590+0000 7f5555400640 1 -- 172.21.15.28:0/3196206601 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55340039b0 msgr2=0x7f5534003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:40.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.590+0000 7f5555400640 1 --2- 172.21.15.28:0/3196206601 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55340039b0 0x7f5534003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:40.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.590+0000 7f5555400640 1 -- 172.21.15.28:0/3196206601 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5534003520 con 0x7f5534004d40 2024-02-21T23:03:40.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.590+0000 7f5554a00640 1 --2- 172.21.15.28:0/3196206601 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55340039b0 0x7f5534003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:03:40.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.591+0000 7f5555400640 1 --2- 172.21.15.28:0/3196206601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5534004d40 0x7f55340051c0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f554800c210 tx=0x7f5534005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1242b4aae9bc0ed0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:40.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.591+0000 7f554f400640 1 -- 172.21.15.28:0/3196206601 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5548007d50 con 0x7f5534004d40 2024-02-21T23:03:40.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.591+0000 7f554f400640 1 -- 172.21.15.28:0/3196206601 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5548030070 con 0x7f5534004d40 2024-02-21T23:03:40.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.591+0000 7f554f400640 1 -- 172.21.15.28:0/3196206601 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f554802cda0 con 0x7f5534004d40 2024-02-21T23:03:40.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.592+0000 7f5557400640 1 -- 172.21.15.28:0/3196206601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5534004d40 msgr2=0x7f55340051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:40.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.592+0000 7f5557400640 1 --2- 172.21.15.28:0/3196206601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5534004d40 0x7f55340051c0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f554800c210 tx=0x7f5534005830 comp rx=0 tx=0).stop 2024-02-21T23:03:40.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.592+0000 7f5555400640 1 -- 172.21.15.28:0/3196206601 reap_dead start 2024-02-21T23:03:40.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.592+0000 7f5557400640 1 -- 172.21.15.28:0/3196206601 shutdown_connections 2024-02-21T23:03:40.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.592+0000 7f5557400640 1 -- 172.21.15.28:0/3196206601 >> 172.21.15.28:0/3196206601 conn(0x7f55500a85a0 msgr2=0x7f55500a8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:40.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.592+0000 7f5557400640 1 -- 172.21.15.28:0/3196206601 shutdown_connections 2024-02-21T23:03:40.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.592+0000 7f5557400640 1 -- 172.21.15.28:0/3196206601 wait complete. 2024-02-21T23:03:40.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.593+0000 7f5557400640 1 Processor -- start 2024-02-21T23:03:40.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.594+0000 7f5557400640 1 -- start start 2024-02-21T23:03:40.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.594+0000 7f5557400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f555014fa80 0x7f55501595c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:40.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.594+0000 7f5557400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5550150230 0x7f5550159b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:40.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.594+0000 7f5557400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f555015a040 0x7f55501f2770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:40.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.594+0000 7f5557400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f555015d850 con 0x7f555015a040 2024-02-21T23:03:40.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.594+0000 7f5557400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5550156db0 con 0x7f5550150230 2024-02-21T23:03:40.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.594+0000 7f5557400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f555006ef20 con 0x7f555014fa80 2024-02-21T23:03:40.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.595+0000 7f5555400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f555015a040 0x7f55501f2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:40.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.595+0000 7f5555400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f555015a040 0x7f55501f2770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44480/0 (socket says 172.21.15.28:44480) 2024-02-21T23:03:40.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.595+0000 7f5555400640 1 -- 172.21.15.28:0/2612236010 learned_addr learned my addr 172.21.15.28:0/2612236010 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:40.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.595+0000 7f554fe00640 1 --2- 172.21.15.28:0/2612236010 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5550150230 0x7f5550159b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:40.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.595+0000 7f5554a00640 1 --2- 172.21.15.28:0/2612236010 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f555014fa80 0x7f55501595c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:40.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.595+0000 7f5555400640 1 -- 172.21.15.28:0/2612236010 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f555014fa80 msgr2=0x7f55501595c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:40.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.595+0000 7f5555400640 1 --2- 172.21.15.28:0/2612236010 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f555014fa80 0x7f55501595c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:40.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.595+0000 7f5555400640 1 -- 172.21.15.28:0/2612236010 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5550150230 msgr2=0x7f5550159b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:40.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.595+0000 7f5555400640 1 --2- 172.21.15.28:0/2612236010 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5550150230 0x7f5550159b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:40.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.595+0000 7f5555400640 1 -- 172.21.15.28:0/2612236010 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5550157060 con 0x7f555015a040 2024-02-21T23:03:40.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.595+0000 7f5554a00640 1 --2- 172.21.15.28:0/2612236010 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f555014fa80 0x7f55501595c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:03:40.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.596+0000 7f5555400640 1 --2- 172.21.15.28:0/2612236010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f555015a040 0x7f55501f2770 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f554800c330 tx=0x7f5548001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:40.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.596+0000 7f554cc00640 1 -- 172.21.15.28:0/2612236010 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5548007d50 con 0x7f555015a040 2024-02-21T23:03:40.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.596+0000 7f554cc00640 1 -- 172.21.15.28:0/2612236010 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5548004cb0 con 0x7f555015a040 2024-02-21T23:03:40.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.596+0000 7f5557400640 1 -- 172.21.15.28:0/2612236010 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5550157300 con 0x7f555015a040 2024-02-21T23:03:40.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.596+0000 7f554cc00640 1 -- 172.21.15.28:0/2612236010 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5548001c90 con 0x7f555015a040 2024-02-21T23:03:40.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.596+0000 7f5557400640 1 -- 172.21.15.28:0/2612236010 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5550150c70 con 0x7f555015a040 2024-02-21T23:03:40.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.598+0000 7f5557400640 1 -- 172.21.15.28:0/2612236010 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5550151420 con 0x7f555015a040 2024-02-21T23:03:40.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.598+0000 7f554cc00640 1 -- 172.21.15.28:0/2612236010 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f55480072b0 con 0x7f555015a040 2024-02-21T23:03:40.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.599+0000 7f554cc00640 1 --2- 172.21.15.28:0/2612236010 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f553006b4d0 0x7f553006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:40.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.599+0000 7f554cc00640 1 -- 172.21.15.28:0/2612236010 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f55480ab210 con 0x7f555015a040 2024-02-21T23:03:40.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.600+0000 7f5554a00640 1 --2- 172.21.15.28:0/2612236010 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f553006b4d0 0x7f553006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:40.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.601+0000 7f5554a00640 1 --2- 172.21.15.28:0/2612236010 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f553006b4d0 0x7f553006d990 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5540006e80 tx=0x7f5540008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:40.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.603+0000 7f554cc00640 1 -- 172.21.15.28:0/2612236010 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f5548079d20 con 0x7f555015a040 2024-02-21T23:03:40.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.802+0000 7f5557400640 1 -- 172.21.15.28:0/2612236010 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5550067f30 con 0x7f555015a040 2024-02-21T23:03:40.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.803+0000 7f554cc00640 1 -- 172.21.15.28:0/2612236010 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5548079620 con 0x7f555015a040 2024-02-21T23:03:40.791 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-02-21T23:03:40.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.805+0000 7f5557400640 1 -- 172.21.15.28:0/2612236010 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f553006b4d0 msgr2=0x7f553006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:40.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.805+0000 7f5557400640 1 --2- 172.21.15.28:0/2612236010 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f553006b4d0 0x7f553006d990 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5540006e80 tx=0x7f5540008040 comp rx=0 tx=0).stop 2024-02-21T23:03:40.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.805+0000 7f5557400640 1 -- 172.21.15.28:0/2612236010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f555015a040 msgr2=0x7f55501f2770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:40.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.805+0000 7f5557400640 1 --2- 172.21.15.28:0/2612236010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f555015a040 0x7f55501f2770 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f554800c330 tx=0x7f5548001a40 comp rx=0 tx=0).stop 2024-02-21T23:03:40.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.806+0000 7f5557400640 1 -- 172.21.15.28:0/2612236010 shutdown_connections 2024-02-21T23:03:40.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.806+0000 7f5557400640 1 --2- 172.21.15.28:0/2612236010 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f553006b4d0 0x7f553006d990 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:40.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.806+0000 7f5557400640 1 --2- 172.21.15.28:0/2612236010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f555015a040 0x7f55501f2770 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:40.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.806+0000 7f5557400640 1 --2- 172.21.15.28:0/2612236010 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5550150230 0x7f5550159b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:40.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.806+0000 7f5557400640 1 --2- 172.21.15.28:0/2612236010 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f555014fa80 0x7f55501595c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:40.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.806+0000 7f5557400640 1 -- 172.21.15.28:0/2612236010 >> 172.21.15.28:0/2612236010 conn(0x7f55500a85a0 msgr2=0x7f55500c43e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:40.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.806+0000 7f5557400640 1 -- 172.21.15.28:0/2612236010 shutdown_connections 2024-02-21T23:03:40.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:40.806+0000 7f5557400640 1 -- 172.21.15.28:0/2612236010 wait complete. 2024-02-21T23:03:40.816 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-21T23:03:41.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.049+0000 7f62edc00640 1 Processor -- start 2024-02-21T23:03:41.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.049+0000 7f62edc00640 1 -- start start 2024-02-21T23:03:41.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.051+0000 7f62edc00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f62e80c4960 0x7f62e80c6220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:41.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.051+0000 7f62edc00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f62e8158dc0 0x7f62e8152930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:41.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.051+0000 7f62edc00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62e80c20d0 con 0x7f62e81524f0 2024-02-21T23:03:41.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.051+0000 7f62edc00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62e80a62e0 con 0x7f62e8158dc0 2024-02-21T23:03:41.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.051+0000 7f62edc00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62e80c1960 con 0x7f62e80c4960 2024-02-21T23:03:41.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.051+0000 7f62de000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f62e81524f0 0x7f62e81586b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55164/0 (socket says 172.21.15.28:55164) 2024-02-21T23:03:41.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.051+0000 7f62de000640 1 -- 172.21.15.28:0/2357780760 learned_addr learned my addr 172.21.15.28:0/2357780760 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:03:41.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.051+0000 7f62e7400640 1 --2- 172.21.15.28:0/2357780760 >> v2:172.21.15.28:3300/0 conn(0x7f62e8158dc0 0x7f62e8152930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:41.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.051+0000 7f62e6a00640 1 --2- 172.21.15.28:0/2357780760 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f62e80c4960 0x7f62e80c6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:41.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.052+0000 7f62e6000640 1 -- 172.21.15.28:0/2357780760 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2871979772 0 0) 0x7f62e80c20d0 con 0x7f62e81524f0 2024-02-21T23:03:41.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.052+0000 7f62e6000640 1 -- 172.21.15.28:0/2357780760 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62cc003520 con 0x7f62e81524f0 2024-02-21T23:03:41.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.052+0000 7f62e7400640 1 -- 172.21.15.28:0/2357780760 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f62e80c4960 msgr2=0x7f62e80c6220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:41.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.052+0000 7f62e7400640 1 --2- 172.21.15.28:0/2357780760 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f62e80c4960 0x7f62e80c6220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:41.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.052+0000 7f62e7400640 1 -- 172.21.15.28:0/2357780760 >> v1:172.21.15.28:6789/0 conn(0x7f62e81524f0 legacy=0x7f62e81586b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:41.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.052+0000 7f62e7400640 1 -- 172.21.15.28:0/2357780760 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62e80c1290 con 0x7f62e8158dc0 2024-02-21T23:03:41.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.053+0000 7f62e7400640 1 --2- 172.21.15.28:0/2357780760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62e8158dc0 0x7f62e8152930 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f62e00097a0 tx=0x7f62e002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e40185602fb469c3 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:41.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.053+0000 7f62e6000640 1 -- 172.21.15.28:0/2357780760 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e00315a0 con 0x7f62e8158dc0 2024-02-21T23:03:41.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.053+0000 7f62e6000640 1 -- 172.21.15.28:0/2357780760 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62e0037070 con 0x7f62e8158dc0 2024-02-21T23:03:41.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.053+0000 7f62e6000640 1 -- 172.21.15.28:0/2357780760 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e0031960 con 0x7f62e8158dc0 2024-02-21T23:03:41.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.054+0000 7f62edc00640 1 -- 172.21.15.28:0/2357780760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62e8158dc0 msgr2=0x7f62e8152930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:41.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.054+0000 7f62edc00640 1 --2- 172.21.15.28:0/2357780760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62e8158dc0 0x7f62e8152930 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f62e00097a0 tx=0x7f62e002b840 comp rx=0 tx=0).stop 2024-02-21T23:03:41.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.054+0000 7f62edc00640 1 -- 172.21.15.28:0/2357780760 shutdown_connections 2024-02-21T23:03:41.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.054+0000 7f62edc00640 1 --2- 172.21.15.28:0/2357780760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62e8158dc0 0x7f62e8152930 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:41.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.054+0000 7f62edc00640 1 --2- 172.21.15.28:0/2357780760 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f62e80c4960 0x7f62e80c6220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:41.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.054+0000 7f62edc00640 1 -- 172.21.15.28:0/2357780760 >> 172.21.15.28:0/2357780760 conn(0x7f62e80a85a0 msgr2=0x7f62e80a8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:41.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.054+0000 7f62edc00640 1 -- 172.21.15.28:0/2357780760 shutdown_connections 2024-02-21T23:03:41.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.054+0000 7f62edc00640 1 -- 172.21.15.28:0/2357780760 wait complete. 2024-02-21T23:03:41.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.056+0000 7f62edc00640 1 Processor -- start 2024-02-21T23:03:41.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.056+0000 7f62edc00640 1 -- start start 2024-02-21T23:03:41.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.057+0000 7f62edc00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62e80c4960 0x7f62e81ea680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:41.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.057+0000 7f62edc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62e81524f0 0x7f62e8149930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:41.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.057+0000 7f62edc00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f62e8158dc0 0x7f62e81ee610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:41.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.057+0000 7f62edc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62e80c1bb0 con 0x7f62e81524f0 2024-02-21T23:03:41.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.057+0000 7f62edc00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62e8159250 con 0x7f62e80c4960 2024-02-21T23:03:41.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.057+0000 7f62edc00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62e80a5f20 con 0x7f62e8158dc0 2024-02-21T23:03:41.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.057+0000 7f62de000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62e81524f0 0x7f62e8149930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:41.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.058+0000 7f62de000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62e81524f0 0x7f62e8149930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44498/0 (socket says 172.21.15.28:44498) 2024-02-21T23:03:41.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.058+0000 7f62de000640 1 -- 172.21.15.28:0/1150649776 learned_addr learned my addr 172.21.15.28:0/1150649776 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:41.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.058+0000 7f62e6a00640 1 --2- 172.21.15.28:0/1150649776 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62e80c4960 0x7f62e81ea680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:41.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.058+0000 7f62e7400640 1 --2- 172.21.15.28:0/1150649776 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f62e8158dc0 0x7f62e81ee610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:41.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.058+0000 7f62de000640 1 -- 172.21.15.28:0/1150649776 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f62e8158dc0 msgr2=0x7f62e81ee610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:41.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.058+0000 7f62de000640 1 --2- 172.21.15.28:0/1150649776 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f62e8158dc0 0x7f62e81ee610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:41.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.058+0000 7f62de000640 1 -- 172.21.15.28:0/1150649776 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62e80c4960 msgr2=0x7f62e81ea680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:41.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.058+0000 7f62de000640 1 --2- 172.21.15.28:0/1150649776 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62e80c4960 0x7f62e81ea680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:41.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.058+0000 7f62de000640 1 -- 172.21.15.28:0/1150649776 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62e81518a0 con 0x7f62e81524f0 2024-02-21T23:03:41.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.058+0000 7f62de000640 1 --2- 172.21.15.28:0/1150649776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62e81524f0 0x7f62e8149930 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f62d000c430 tx=0x7f62d000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:41.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.059+0000 7f62df400640 1 -- 172.21.15.28:0/1150649776 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62d0017070 con 0x7f62e81524f0 2024-02-21T23:03:41.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.059+0000 7f62df400640 1 -- 172.21.15.28:0/1150649776 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62d0005ce0 con 0x7f62e81524f0 2024-02-21T23:03:41.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.059+0000 7f62df400640 1 -- 172.21.15.28:0/1150649776 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62d00072f0 con 0x7f62e81524f0 2024-02-21T23:03:41.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.059+0000 7f62edc00640 1 -- 172.21.15.28:0/1150649776 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62e8151b80 con 0x7f62e81524f0 2024-02-21T23:03:41.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.059+0000 7f62edc00640 1 -- 172.21.15.28:0/1150649776 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62e81ee0f0 con 0x7f62e81524f0 2024-02-21T23:03:41.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.060+0000 7f62edc00640 1 -- 172.21.15.28:0/1150649776 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62e8067f30 con 0x7f62e81524f0 2024-02-21T23:03:41.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.061+0000 7f62df400640 1 -- 172.21.15.28:0/1150649776 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f62d001a070 con 0x7f62e81524f0 2024-02-21T23:03:41.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.062+0000 7f62df400640 1 --2- 172.21.15.28:0/1150649776 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f62c406b4d0 0x7f62c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:41.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.063+0000 7f62df400640 1 -- 172.21.15.28:0/1150649776 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f62d0089af0 con 0x7f62e81524f0 2024-02-21T23:03:41.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.063+0000 7f62e6a00640 1 --2- 172.21.15.28:0/1150649776 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f62c406b4d0 0x7f62c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:41.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.064+0000 7f62e6a00640 1 --2- 172.21.15.28:0/1150649776 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f62c406b4d0 0x7f62c406d990 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f62d80074a0 tx=0x7f62d800b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:41.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.067+0000 7f62df400640 1 -- 172.21.15.28:0/1150649776 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f62d0058560 con 0x7f62e81524f0 2024-02-21T23:03:41.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.266+0000 7f62edc00640 1 -- 172.21.15.28:0/1150649776 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f62e8073330 con 0x7f62e81524f0 2024-02-21T23:03:41.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.266+0000 7f62df400640 1 -- 172.21.15.28:0/1150649776 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+11 (secure 0 0 0) 0x7f62d0057e60 con 0x7f62e81524f0 2024-02-21T23:03:41.255 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-02-21T23:03:41.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.269+0000 7f62edc00640 1 -- 172.21.15.28:0/1150649776 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f62c406b4d0 msgr2=0x7f62c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:41.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.269+0000 7f62edc00640 1 --2- 172.21.15.28:0/1150649776 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f62c406b4d0 0x7f62c406d990 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f62d80074a0 tx=0x7f62d800b040 comp rx=0 tx=0).stop 2024-02-21T23:03:41.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.269+0000 7f62edc00640 1 -- 172.21.15.28:0/1150649776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62e81524f0 msgr2=0x7f62e8149930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:41.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.269+0000 7f62edc00640 1 --2- 172.21.15.28:0/1150649776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62e81524f0 0x7f62e8149930 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f62d000c430 tx=0x7f62d000c8f0 comp rx=0 tx=0).stop 2024-02-21T23:03:41.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.269+0000 7f62edc00640 1 -- 172.21.15.28:0/1150649776 shutdown_connections 2024-02-21T23:03:41.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.269+0000 7f62edc00640 1 --2- 172.21.15.28:0/1150649776 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f62c406b4d0 0x7f62c406d990 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:41.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.269+0000 7f62edc00640 1 --2- 172.21.15.28:0/1150649776 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f62e8158dc0 0x7f62e81ee610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:41.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.269+0000 7f62edc00640 1 --2- 172.21.15.28:0/1150649776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62e81524f0 0x7f62e8149930 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:41.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.269+0000 7f62edc00640 1 --2- 172.21.15.28:0/1150649776 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62e80c4960 0x7f62e81ea680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:41.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.269+0000 7f62edc00640 1 -- 172.21.15.28:0/1150649776 >> 172.21.15.28:0/1150649776 conn(0x7f62e80a85a0 msgr2=0x7f62e814c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:41.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.269+0000 7f62edc00640 1 -- 172.21.15.28:0/1150649776 shutdown_connections 2024-02-21T23:03:41.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.269+0000 7f62edc00640 1 -- 172.21.15.28:0/1150649776 wait complete. 2024-02-21T23:03:41.275 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-02-21T23:03:41.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.501+0000 7f5089800640 1 Processor -- start 2024-02-21T23:03:41.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.501+0000 7f5089800640 1 -- start start 2024-02-21T23:03:41.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.503+0000 7f5089800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f508406dbc0 0x7f508406c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:41.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.503+0000 7f5089800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f508413e520 0x7f508406c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:41.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.503+0000 7f5089800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5084158550 con 0x7f508413e0c0 2024-02-21T23:03:41.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.503+0000 7f5089800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50840c89a0 con 0x7f508413e520 2024-02-21T23:03:41.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.503+0000 7f5089800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50841587d0 con 0x7f508406dbc0 2024-02-21T23:03:41.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.503+0000 7f5083e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f508413e0c0 0x7f50840c71b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55178/0 (socket says 172.21.15.28:55178) 2024-02-21T23:03:41.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.503+0000 7f5083e00640 1 -- 172.21.15.28:0/2448878624 learned_addr learned my addr 172.21.15.28:0/2448878624 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:03:41.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.503+0000 7f5088e00640 1 --2- 172.21.15.28:0/2448878624 >> v2:172.21.15.28:3300/0 conn(0x7f508413e520 0x7f508406c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:41.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.504+0000 7f5083400640 1 --2- 172.21.15.28:0/2448878624 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f508406dbc0 0x7f508406c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:41.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.504+0000 7f5082a00640 1 -- 172.21.15.28:0/2448878624 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3412432725 0 0) 0x7f5074001490 con 0x7f508413e0c0 2024-02-21T23:03:41.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.504+0000 7f5082a00640 1 -- 172.21.15.28:0/2448878624 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5068003520 con 0x7f508413e0c0 2024-02-21T23:03:41.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.504+0000 7f5088e00640 1 -- 172.21.15.28:0/2448878624 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f508406dbc0 msgr2=0x7f508406c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:41.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.504+0000 7f5088e00640 1 --2- 172.21.15.28:0/2448878624 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f508406dbc0 0x7f508406c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:41.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.504+0000 7f5088e00640 1 -- 172.21.15.28:0/2448878624 >> v1:172.21.15.28:6789/0 conn(0x7f508413e0c0 legacy=0x7f50840c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:41.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.504+0000 7f5088e00640 1 -- 172.21.15.28:0/2448878624 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5084158ab0 con 0x7f508413e520 2024-02-21T23:03:41.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.505+0000 7f5088e00640 1 --2- 172.21.15.28:0/2448878624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f508413e520 0x7f508406c7c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f507c0097a0 tx=0x7f507c02b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45b891fc1a0f74f5 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:41.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.505+0000 7f5082a00640 1 -- 172.21.15.28:0/2448878624 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f507c031780 con 0x7f508413e520 2024-02-21T23:03:41.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.505+0000 7f5082a00640 1 -- 172.21.15.28:0/2448878624 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f507c037070 con 0x7f508413e520 2024-02-21T23:03:41.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.505+0000 7f5082a00640 1 -- 172.21.15.28:0/2448878624 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f507c031b40 con 0x7f508413e520 2024-02-21T23:03:41.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.506+0000 7f5089800640 1 -- 172.21.15.28:0/2448878624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f508413e520 msgr2=0x7f508406c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:41.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.506+0000 7f5089800640 1 --2- 172.21.15.28:0/2448878624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f508413e520 0x7f508406c7c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f507c0097a0 tx=0x7f507c02b820 comp rx=0 tx=0).stop 2024-02-21T23:03:41.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.506+0000 7f5089800640 1 -- 172.21.15.28:0/2448878624 shutdown_connections 2024-02-21T23:03:41.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.506+0000 7f5089800640 1 --2- 172.21.15.28:0/2448878624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f508413e520 0x7f508406c7c0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:41.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.506+0000 7f5089800640 1 --2- 172.21.15.28:0/2448878624 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f508406dbc0 0x7f508406c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:41.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.506+0000 7f5089800640 1 -- 172.21.15.28:0/2448878624 >> 172.21.15.28:0/2448878624 conn(0x7f5084073110 msgr2=0x7f50840ac490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:41.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.506+0000 7f5089800640 1 -- 172.21.15.28:0/2448878624 shutdown_connections 2024-02-21T23:03:41.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.506+0000 7f5089800640 1 -- 172.21.15.28:0/2448878624 wait complete. 2024-02-21T23:03:41.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.508+0000 7f5089800640 1 Processor -- start 2024-02-21T23:03:41.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.508+0000 7f5089800640 1 -- start start 2024-02-21T23:03:41.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.509+0000 7f5089800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f508406dbc0 0x7f50840ab530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:41.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.509+0000 7f5089800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f508413e0c0 0x7f50840ae060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:41.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.509+0000 7f5089800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f508413e520 0x7f50840ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:41.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.509+0000 7f5089800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50840c78c0 con 0x7f508413e0c0 2024-02-21T23:03:41.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.509+0000 7f5089800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5084159070 con 0x7f508413e520 2024-02-21T23:03:41.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.509+0000 7f5089800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50840c7a40 con 0x7f508406dbc0 2024-02-21T23:03:41.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.509+0000 7f5083400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f508413e0c0 0x7f50840ae060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:41.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.509+0000 7f5083400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f508413e0c0 0x7f50840ae060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44502/0 (socket says 172.21.15.28:44502) 2024-02-21T23:03:41.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.509+0000 7f5083400640 1 -- 172.21.15.28:0/2513452876 learned_addr learned my addr 172.21.15.28:0/2513452876 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:41.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.509+0000 7f5083e00640 1 --2- 172.21.15.28:0/2513452876 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f508406dbc0 0x7f50840ab530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:41.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.510+0000 7f5083400640 1 -- 172.21.15.28:0/2513452876 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f508406dbc0 msgr2=0x7f50840ab530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:41.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.510+0000 7f5083400640 1 --2- 172.21.15.28:0/2513452876 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f508406dbc0 0x7f50840ab530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:41.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.510+0000 7f5083400640 1 -- 172.21.15.28:0/2513452876 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f508413e520 msgr2=0x7f50840ae5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:41.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.510+0000 7f5083400640 1 --2- 172.21.15.28:0/2513452876 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f508413e520 0x7f50840ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:41.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.510+0000 7f5083400640 1 -- 172.21.15.28:0/2513452876 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5084155850 con 0x7f508413e0c0 2024-02-21T23:03:41.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.510+0000 7f5083400640 1 --2- 172.21.15.28:0/2513452876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f508413e0c0 0x7f50840ae060 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f507000d5f0 tx=0x7f507000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:41.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.510+0000 7f507be00640 1 -- 172.21.15.28:0/2513452876 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50700125f0 con 0x7f508413e0c0 2024-02-21T23:03:41.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.510+0000 7f507be00640 1 -- 172.21.15.28:0/2513452876 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5070019070 con 0x7f508413e0c0 2024-02-21T23:03:41.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.510+0000 7f507be00640 1 -- 172.21.15.28:0/2513452876 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50700129d0 con 0x7f508413e0c0 2024-02-21T23:03:41.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.510+0000 7f5089800640 1 -- 172.21.15.28:0/2513452876 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50840b0150 con 0x7f508413e0c0 2024-02-21T23:03:41.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.510+0000 7f5089800640 1 -- 172.21.15.28:0/2513452876 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5084157fe0 con 0x7f508413e0c0 2024-02-21T23:03:41.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.511+0000 7f5089800640 1 -- 172.21.15.28:0/2513452876 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f508404f290 con 0x7f508413e0c0 2024-02-21T23:03:41.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.512+0000 7f507be00640 1 -- 172.21.15.28:0/2513452876 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5070012b70 con 0x7f508413e0c0 2024-02-21T23:03:41.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.513+0000 7f507be00640 1 --2- 172.21.15.28:0/2513452876 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f506406b4d0 0x7f506406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:41.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.513+0000 7f507be00640 1 -- 172.21.15.28:0/2513452876 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f507008aa60 con 0x7f508413e0c0 2024-02-21T23:03:41.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.514+0000 7f5083e00640 1 --2- 172.21.15.28:0/2513452876 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f506406b4d0 0x7f506406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:41.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.515+0000 7f5083e00640 1 --2- 172.21.15.28:0/2513452876 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f506406b4d0 0x7f506406d990 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f5074005c00 tx=0x7f5074009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:41.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.517+0000 7f507be00640 1 -- 172.21.15.28:0/2513452876 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f5070059440 con 0x7f508413e0c0 2024-02-21T23:03:41.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.715+0000 7f5089800640 1 -- 172.21.15.28:0/2513452876 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f508406d9a0 con 0x7f508413e0c0 2024-02-21T23:03:41.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.716+0000 7f507be00640 1 -- 172.21.15.28:0/2513452876 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) v1 ==== 103+0+11 (secure 0 0 0) 0x7f5070058d40 con 0x7f508413e0c0 2024-02-21T23:03:41.705 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-02-21T23:03:41.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.719+0000 7f5089800640 1 -- 172.21.15.28:0/2513452876 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f506406b4d0 msgr2=0x7f506406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:41.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.719+0000 7f5089800640 1 --2- 172.21.15.28:0/2513452876 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f506406b4d0 0x7f506406d990 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f5074005c00 tx=0x7f5074009040 comp rx=0 tx=0).stop 2024-02-21T23:03:41.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.719+0000 7f5089800640 1 -- 172.21.15.28:0/2513452876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f508413e0c0 msgr2=0x7f50840ae060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:41.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.719+0000 7f5089800640 1 --2- 172.21.15.28:0/2513452876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f508413e0c0 0x7f50840ae060 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f507000d5f0 tx=0x7f507000dab0 comp rx=0 tx=0).stop 2024-02-21T23:03:41.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.719+0000 7f5089800640 1 -- 172.21.15.28:0/2513452876 shutdown_connections 2024-02-21T23:03:41.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.719+0000 7f5089800640 1 --2- 172.21.15.28:0/2513452876 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f506406b4d0 0x7f506406d990 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:41.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.719+0000 7f5089800640 1 --2- 172.21.15.28:0/2513452876 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f508413e520 0x7f50840ae5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:41.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.719+0000 7f5089800640 1 --2- 172.21.15.28:0/2513452876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f508413e0c0 0x7f50840ae060 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:41.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.719+0000 7f5089800640 1 --2- 172.21.15.28:0/2513452876 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f508406dbc0 0x7f50840ab530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:41.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.719+0000 7f5089800640 1 -- 172.21.15.28:0/2513452876 >> 172.21.15.28:0/2513452876 conn(0x7f5084073110 msgr2=0x7f508413c850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:41.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.719+0000 7f5089800640 1 -- 172.21.15.28:0/2513452876 shutdown_connections 2024-02-21T23:03:41.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.719+0000 7f5089800640 1 -- 172.21.15.28:0/2513452876 wait complete. 2024-02-21T23:03:41.729 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_drop' 2024-02-21T23:03:41.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.961+0000 7fdf31000640 1 Processor -- start 2024-02-21T23:03:41.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.961+0000 7fdf31000640 1 -- start start 2024-02-21T23:03:41.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.962+0000 7fdf31000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdf2c0c11f0 0x7fdf2c153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:41.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.962+0000 7fdf31000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fdf2c15b6d0 0x7fdf2c15dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:41.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.962+0000 7fdf31000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf2c152e70 con 0x7fdf2c0c15d0 2024-02-21T23:03:41.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.962+0000 7fdf31000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf2c0c1a60 con 0x7fdf2c0c11f0 2024-02-21T23:03:41.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.962+0000 7fdf31000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf2c0a5ec0 con 0x7fdf2c15b6d0 2024-02-21T23:03:41.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.963+0000 7fdf29e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdf2c0c11f0 0x7fdf2c153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:41.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.963+0000 7fdf29e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdf2c0c11f0 0x7fdf2c153950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44518/0 (socket says 172.21.15.28:44518) 2024-02-21T23:03:41.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.963+0000 7fdf29e00640 1 -- 172.21.15.28:0/1474228011 learned_addr learned my addr 172.21.15.28:0/1474228011 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:41.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.963+0000 7fdf2a800640 1 --2- 172.21.15.28:0/1474228011 >> v2:172.21.15.81:3300/0 conn(0x7fdf2c15b6d0 0x7fdf2c15dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:41.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.963+0000 7fdf29e00640 1 -- 172.21.15.28:0/1474228011 >> v2:172.21.15.81:3300/0 conn(0x7fdf2c15b6d0 msgr2=0x7fdf2c15dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:41.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.963+0000 7fdf29e00640 1 --2- 172.21.15.28:0/1474228011 >> v2:172.21.15.81:3300/0 conn(0x7fdf2c15b6d0 0x7fdf2c15dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:41.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.963+0000 7fdf29e00640 1 -- 172.21.15.28:0/1474228011 >> v1:172.21.15.81:6789/0 conn(0x7fdf2c0c15d0 legacy=0x7fdf2c06dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:41.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.963+0000 7fdf29e00640 1 -- 172.21.15.28:0/1474228011 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf2c153150 con 0x7fdf2c0c11f0 2024-02-21T23:03:41.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.964+0000 7fdf29e00640 1 --2- 172.21.15.28:0/1474228011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf2c0c11f0 0x7fdf2c153950 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fdf2c0a5e50 tx=0x7fdf1c02bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfec5cc40dce826c server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:41.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.965+0000 7fdf28a00640 1 -- 172.21.15.28:0/1474228011 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf1c02c960 con 0x7fdf2c0c11f0 2024-02-21T23:03:41.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.965+0000 7fdf28a00640 1 -- 172.21.15.28:0/1474228011 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf1c02cb00 con 0x7fdf2c0c11f0 2024-02-21T23:03:41.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.965+0000 7fdf28a00640 1 -- 172.21.15.28:0/1474228011 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf1c0343f0 con 0x7fdf2c0c11f0 2024-02-21T23:03:41.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.965+0000 7fdf31000640 1 -- 172.21.15.28:0/1474228011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf2c0c11f0 msgr2=0x7fdf2c153950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:41.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.965+0000 7fdf31000640 1 --2- 172.21.15.28:0/1474228011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf2c0c11f0 0x7fdf2c153950 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fdf2c0a5e50 tx=0x7fdf1c02bdc0 comp rx=0 tx=0).stop 2024-02-21T23:03:41.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.966+0000 7fdf31000640 1 -- 172.21.15.28:0/1474228011 shutdown_connections 2024-02-21T23:03:41.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.966+0000 7fdf31000640 1 --2- 172.21.15.28:0/1474228011 >> v2:172.21.15.81:3300/0 conn(0x7fdf2c15b6d0 0x7fdf2c15dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:41.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.966+0000 7fdf31000640 1 --2- 172.21.15.28:0/1474228011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf2c0c11f0 0x7fdf2c153950 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:41.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.966+0000 7fdf31000640 1 -- 172.21.15.28:0/1474228011 >> 172.21.15.28:0/1474228011 conn(0x7fdf2c0c67f0 msgr2=0x7fdf2c0c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:41.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.966+0000 7fdf31000640 1 -- 172.21.15.28:0/1474228011 shutdown_connections 2024-02-21T23:03:41.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.966+0000 7fdf31000640 1 -- 172.21.15.28:0/1474228011 wait complete. 2024-02-21T23:03:41.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.968+0000 7fdf31000640 1 Processor -- start 2024-02-21T23:03:41.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.968+0000 7fdf31000640 1 -- start start 2024-02-21T23:03:41.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.969+0000 7fdf31000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf2c0c11f0 0x7fdf2c0bfc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:41.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.969+0000 7fdf31000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf2c0c15d0 0x7fdf2c072210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:41.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.969+0000 7fdf31000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdf2c15b6d0 0x7fdf2c072750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:41.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.969+0000 7fdf31000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf2c0c4c00 con 0x7fdf2c0c15d0 2024-02-21T23:03:41.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.969+0000 7fdf31000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf2c14d1c0 con 0x7fdf2c0c11f0 2024-02-21T23:03:41.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.969+0000 7fdf31000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf2c0a6110 con 0x7fdf2c15b6d0 2024-02-21T23:03:41.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.969+0000 7fdf29400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf2c0c15d0 0x7fdf2c072210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:41.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.969+0000 7fdf29400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf2c0c15d0 0x7fdf2c072210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44522/0 (socket says 172.21.15.28:44522) 2024-02-21T23:03:41.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.969+0000 7fdf29400640 1 -- 172.21.15.28:0/1421899355 learned_addr learned my addr 172.21.15.28:0/1421899355 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:41.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.969+0000 7fdf29e00640 1 --2- 172.21.15.28:0/1421899355 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf2c0c11f0 0x7fdf2c0bfc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:41.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.970+0000 7fdf2a800640 1 --2- 172.21.15.28:0/1421899355 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdf2c15b6d0 0x7fdf2c072750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:41.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.970+0000 7fdf29400640 1 -- 172.21.15.28:0/1421899355 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdf2c15b6d0 msgr2=0x7fdf2c072750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:41.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.970+0000 7fdf29400640 1 --2- 172.21.15.28:0/1421899355 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdf2c15b6d0 0x7fdf2c072750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:41.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.970+0000 7fdf29400640 1 -- 172.21.15.28:0/1421899355 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf2c0c11f0 msgr2=0x7fdf2c0bfc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:41.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.970+0000 7fdf29400640 1 --2- 172.21.15.28:0/1421899355 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf2c0c11f0 0x7fdf2c0bfc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:41.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.970+0000 7fdf29400640 1 -- 172.21.15.28:0/1421899355 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf2c1eaf70 con 0x7fdf2c0c15d0 2024-02-21T23:03:41.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.970+0000 7fdf2a800640 1 --2- 172.21.15.28:0/1421899355 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdf2c15b6d0 0x7fdf2c072750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:03:41.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.970+0000 7fdf29400640 1 --2- 172.21.15.28:0/1421899355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf2c0c15d0 0x7fdf2c072210 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fdf1400b3f0 tx=0x7fdf1400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:41.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.970+0000 7fdf22000640 1 -- 172.21.15.28:0/1421899355 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf14015020 con 0x7fdf2c0c15d0 2024-02-21T23:03:41.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.970+0000 7fdf22000640 1 -- 172.21.15.28:0/1421899355 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf14011830 con 0x7fdf2c0c15d0 2024-02-21T23:03:41.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.970+0000 7fdf22000640 1 -- 172.21.15.28:0/1421899355 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf14011c20 con 0x7fdf2c0c15d0 2024-02-21T23:03:41.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.970+0000 7fdf31000640 1 -- 172.21.15.28:0/1421899355 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf2c1eb250 con 0x7fdf2c0c15d0 2024-02-21T23:03:41.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.971+0000 7fdf31000640 1 -- 172.21.15.28:0/1421899355 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf2c1eb810 con 0x7fdf2c0c15d0 2024-02-21T23:03:41.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.972+0000 7fdf31000640 1 -- 172.21.15.28:0/1421899355 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf2c1eba10 con 0x7fdf2c0c15d0 2024-02-21T23:03:41.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.972+0000 7fdf22000640 1 -- 172.21.15.28:0/1421899355 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdf14018020 con 0x7fdf2c0c15d0 2024-02-21T23:03:41.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.973+0000 7fdf22000640 1 --2- 172.21.15.28:0/1421899355 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdf0806b4d0 0x7fdf0806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:41.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.974+0000 7fdf22000640 1 -- 172.21.15.28:0/1421899355 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6735+0+0 (secure 0 0 0) 0x7fdf1408a2a0 con 0x7fdf2c0c15d0 2024-02-21T23:03:41.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.974+0000 7fdf29e00640 1 --2- 172.21.15.28:0/1421899355 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdf0806b4d0 0x7fdf0806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:41.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.975+0000 7fdf29e00640 1 --2- 172.21.15.28:0/1421899355 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdf0806b4d0 0x7fdf0806d990 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c035000 tx=0x7fdf1c030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:41.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:41.978+0000 7fdf22000640 1 -- 172.21.15.28:0/1421899355 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fdf2c1eba10 con 0x7fdf2c0c15d0 2024-02-21T23:03:42.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:42.240+0000 7fdf31000640 1 -- 172.21.15.28:0/1421899355 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_drop"]} v 0) v1 -- 0x7fdf2c067f30 con 0x7fdf2c0c15d0 2024-02-21T23:03:43.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.189+0000 7fdf22000640 1 -- 172.21.15.28:0/1421899355 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_drop"]}]=0 v80) v1 ==== 146+0+0 (secure 0 0 0) 0x7fdf1401f030 con 0x7fdf2c0c15d0 2024-02-21T23:03:43.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.192+0000 7fdf31000640 1 -- 172.21.15.28:0/1421899355 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdf0806b4d0 msgr2=0x7fdf0806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:43.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.192+0000 7fdf31000640 1 --2- 172.21.15.28:0/1421899355 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdf0806b4d0 0x7fdf0806d990 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c035000 tx=0x7fdf1c030000 comp rx=0 tx=0).stop 2024-02-21T23:03:43.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.192+0000 7fdf31000640 1 -- 172.21.15.28:0/1421899355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf2c0c15d0 msgr2=0x7fdf2c072210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:43.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.192+0000 7fdf31000640 1 --2- 172.21.15.28:0/1421899355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf2c0c15d0 0x7fdf2c072210 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fdf1400b3f0 tx=0x7fdf1400b8b0 comp rx=0 tx=0).stop 2024-02-21T23:03:43.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.192+0000 7fdf31000640 1 -- 172.21.15.28:0/1421899355 shutdown_connections 2024-02-21T23:03:43.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.192+0000 7fdf31000640 1 --2- 172.21.15.28:0/1421899355 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdf0806b4d0 0x7fdf0806d990 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:43.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.192+0000 7fdf31000640 1 --2- 172.21.15.28:0/1421899355 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdf2c15b6d0 0x7fdf2c072750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:43.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.192+0000 7fdf31000640 1 --2- 172.21.15.28:0/1421899355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf2c0c15d0 0x7fdf2c072210 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:43.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.192+0000 7fdf31000640 1 --2- 172.21.15.28:0/1421899355 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf2c0c11f0 0x7fdf2c0bfc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:43.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.192+0000 7fdf31000640 1 -- 172.21.15.28:0/1421899355 >> 172.21.15.28:0/1421899355 conn(0x7fdf2c0c67f0 msgr2=0x7fdf2c0c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:43.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.193+0000 7fdf31000640 1 -- 172.21.15.28:0/1421899355 shutdown_connections 2024-02-21T23:03:43.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.193+0000 7fdf31000640 1 -- 172.21.15.28:0/1421899355 wait complete. 2024-02-21T23:03:43.205 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-02-21T23:03:43.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.451+0000 7ff859000640 1 Processor -- start 2024-02-21T23:03:43.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.451+0000 7ff859000640 1 -- start start 2024-02-21T23:03:43.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.452+0000 7ff859000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff8540c4580 0x7ff8540c12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:43.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.452+0000 7ff859000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff8540c2c10 0x7ff85406ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:43.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.452+0000 7ff859000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff85406e220 con 0x7ff854150240 2024-02-21T23:03:43.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.452+0000 7ff859000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff854150620 con 0x7ff8540c2c10 2024-02-21T23:03:43.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.452+0000 7ff859000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff854153a90 con 0x7ff8540c4580 2024-02-21T23:03:43.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.453+0000 7ff853e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff8540c2c10 0x7ff85406ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:43.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.453+0000 7ff853e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff8540c2c10 0x7ff85406ae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44526/0 (socket says 172.21.15.28:44526) 2024-02-21T23:03:43.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.453+0000 7ff853e00640 1 -- 172.21.15.28:0/58229670 learned_addr learned my addr 172.21.15.28:0/58229670 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:43.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.453+0000 7ff852a00640 1 --2- 172.21.15.28:0/58229670 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff8540c4580 0x7ff8540c12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:43.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.453+0000 7ff853e00640 1 -- 172.21.15.28:0/58229670 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff8540c4580 msgr2=0x7ff8540c12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:43.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.453+0000 7ff853e00640 1 --2- 172.21.15.28:0/58229670 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff8540c4580 0x7ff8540c12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:43.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.453+0000 7ff853e00640 1 -- 172.21.15.28:0/58229670 >> v1:172.21.15.81:6789/0 conn(0x7ff854150240 legacy=0x7ff85406db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:43.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.453+0000 7ff853e00640 1 -- 172.21.15.28:0/58229670 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff85406f780 con 0x7ff8540c2c10 2024-02-21T23:03:43.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.453+0000 7ff852a00640 1 --2- 172.21.15.28:0/58229670 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff8540c4580 0x7ff8540c12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:03:43.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.454+0000 7ff853e00640 1 --2- 172.21.15.28:0/58229670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8540c2c10 0x7ff85406ae70 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7ff84c009e50 tx=0x7ff84c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e764b2116a3e2d0c server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:43.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.455+0000 7ff852000640 1 -- 172.21.15.28:0/58229670 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff84c02c9e0 con 0x7ff8540c2c10 2024-02-21T23:03:43.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.455+0000 7ff852000640 1 -- 172.21.15.28:0/58229670 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff84c02cb80 con 0x7ff8540c2c10 2024-02-21T23:03:43.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.455+0000 7ff852000640 1 -- 172.21.15.28:0/58229670 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff84c0343f0 con 0x7ff8540c2c10 2024-02-21T23:03:43.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.455+0000 7ff859000640 1 -- 172.21.15.28:0/58229670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8540c2c10 msgr2=0x7ff85406ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:43.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.455+0000 7ff859000640 1 --2- 172.21.15.28:0/58229670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8540c2c10 0x7ff85406ae70 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7ff84c009e50 tx=0x7ff84c02be40 comp rx=0 tx=0).stop 2024-02-21T23:03:43.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.456+0000 7ff859000640 1 -- 172.21.15.28:0/58229670 shutdown_connections 2024-02-21T23:03:43.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.456+0000 7ff859000640 1 --2- 172.21.15.28:0/58229670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8540c2c10 0x7ff85406ae70 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:43.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.456+0000 7ff859000640 1 --2- 172.21.15.28:0/58229670 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff8540c4580 0x7ff8540c12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:43.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.456+0000 7ff859000640 1 -- 172.21.15.28:0/58229670 >> 172.21.15.28:0/58229670 conn(0x7ff8540a8530 msgr2=0x7ff8540a8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:43.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.456+0000 7ff859000640 1 -- 172.21.15.28:0/58229670 shutdown_connections 2024-02-21T23:03:43.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.456+0000 7ff859000640 1 -- 172.21.15.28:0/58229670 wait complete. 2024-02-21T23:03:43.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.457+0000 7ff859000640 1 Processor -- start 2024-02-21T23:03:43.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.458+0000 7ff859000640 1 -- start start 2024-02-21T23:03:43.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.458+0000 7ff859000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8540c2c10 0x7ff8541f3000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:43.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.458+0000 7ff859000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff8540c4580 0x7ff854072710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:43.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.458+0000 7ff859000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff854150240 0x7ff8541ef720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:43.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.458+0000 7ff859000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8540c4a10 con 0x7ff8540c2c10 2024-02-21T23:03:43.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.458+0000 7ff859000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff854153720 con 0x7ff8540c4580 2024-02-21T23:03:43.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.458+0000 7ff859000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8540c4b90 con 0x7ff854150240 2024-02-21T23:03:43.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.459+0000 7ff853400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8540c2c10 0x7ff8541f3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:43.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.459+0000 7ff853400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8540c2c10 0x7ff8541f3000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44528/0 (socket says 172.21.15.28:44528) 2024-02-21T23:03:43.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.459+0000 7ff853400640 1 -- 172.21.15.28:0/3734025838 learned_addr learned my addr 172.21.15.28:0/3734025838 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:43.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.459+0000 7ff853e00640 1 --2- 172.21.15.28:0/3734025838 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff854150240 0x7ff8541ef720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:43.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.459+0000 7ff853400640 1 -- 172.21.15.28:0/3734025838 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff854150240 msgr2=0x7ff8541ef720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:43.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.459+0000 7ff853400640 1 --2- 172.21.15.28:0/3734025838 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff854150240 0x7ff8541ef720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:43.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.459+0000 7ff852a00640 1 --2- 172.21.15.28:0/3734025838 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff8540c4580 0x7ff854072710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:43.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.459+0000 7ff853400640 1 -- 172.21.15.28:0/3734025838 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff8540c4580 msgr2=0x7ff854072710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:43.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.459+0000 7ff853400640 1 --2- 172.21.15.28:0/3734025838 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff8540c4580 0x7ff854072710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:43.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.459+0000 7ff853400640 1 -- 172.21.15.28:0/3734025838 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff854152f80 con 0x7ff8540c2c10 2024-02-21T23:03:43.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.460+0000 7ff853400640 1 --2- 172.21.15.28:0/3734025838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8540c2c10 0x7ff8541f3000 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7ff84400b3f0 tx=0x7ff84400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:43.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.460+0000 7ff84b400640 1 -- 172.21.15.28:0/3734025838 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff844015020 con 0x7ff8540c2c10 2024-02-21T23:03:43.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.460+0000 7ff84b400640 1 -- 172.21.15.28:0/3734025838 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff844011830 con 0x7ff8540c2c10 2024-02-21T23:03:43.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.460+0000 7ff84b400640 1 -- 172.21.15.28:0/3734025838 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff844011c20 con 0x7ff8540c2c10 2024-02-21T23:03:43.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.460+0000 7ff859000640 1 -- 172.21.15.28:0/3734025838 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8541526e0 con 0x7ff8540c2c10 2024-02-21T23:03:43.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.460+0000 7ff859000640 1 -- 172.21.15.28:0/3734025838 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff854152ca0 con 0x7ff8540c2c10 2024-02-21T23:03:43.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.461+0000 7ff859000640 1 -- 172.21.15.28:0/3734025838 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff85404f290 con 0x7ff8540c2c10 2024-02-21T23:03:43.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.462+0000 7ff84b400640 1 -- 172.21.15.28:0/3734025838 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff844018020 con 0x7ff8540c2c10 2024-02-21T23:03:43.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.463+0000 7ff84b400640 1 --2- 172.21.15.28:0/3734025838 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff83406b4d0 0x7ff83406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:43.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.464+0000 7ff84b400640 1 -- 172.21.15.28:0/3734025838 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6735+0+0 (secure 0 0 0) 0x7ff84408a230 con 0x7ff8540c2c10 2024-02-21T23:03:43.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.464+0000 7ff852a00640 1 --2- 172.21.15.28:0/3734025838 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff83406b4d0 0x7ff83406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:43.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.465+0000 7ff852a00640 1 --2- 172.21.15.28:0/3734025838 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff83406b4d0 0x7ff83406d990 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff854073070 tx=0x7ff840008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:43.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.468+0000 7ff84b400640 1 -- 172.21.15.28:0/3734025838 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ff84401f030 con 0x7ff8540c2c10 2024-02-21T23:03:43.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.669+0000 7ff859000640 1 -- 172.21.15.28:0/3734025838 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7ff85406f0e0 con 0x7ff8540c2c10 2024-02-21T23:03:43.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.676+0000 7ff84b400640 1 -- 172.21.15.28:0/3734025838 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v11)=0 v11) v1 ==== 122+0+0 (secure 0 0 0) 0x7ff8440172c0 con 0x7ff8540c2c10 2024-02-21T23:03:43.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.679+0000 7ff859000640 1 -- 172.21.15.28:0/3734025838 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff83406b4d0 msgr2=0x7ff83406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:43.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.679+0000 7ff859000640 1 --2- 172.21.15.28:0/3734025838 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff83406b4d0 0x7ff83406d990 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff854073070 tx=0x7ff840008040 comp rx=0 tx=0).stop 2024-02-21T23:03:43.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.679+0000 7ff859000640 1 -- 172.21.15.28:0/3734025838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8540c2c10 msgr2=0x7ff8541f3000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:43.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.679+0000 7ff859000640 1 --2- 172.21.15.28:0/3734025838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8540c2c10 0x7ff8541f3000 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7ff84400b3f0 tx=0x7ff84400b8b0 comp rx=0 tx=0).stop 2024-02-21T23:03:43.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.679+0000 7ff859000640 1 -- 172.21.15.28:0/3734025838 shutdown_connections 2024-02-21T23:03:43.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.679+0000 7ff859000640 1 --2- 172.21.15.28:0/3734025838 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff83406b4d0 0x7ff83406d990 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:43.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.679+0000 7ff859000640 1 --2- 172.21.15.28:0/3734025838 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff854150240 0x7ff8541ef720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:43.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.679+0000 7ff859000640 1 --2- 172.21.15.28:0/3734025838 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff8540c4580 0x7ff854072710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:43.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.679+0000 7ff859000640 1 --2- 172.21.15.28:0/3734025838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8540c2c10 0x7ff8541f3000 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:43.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.679+0000 7ff859000640 1 -- 172.21.15.28:0/3734025838 >> 172.21.15.28:0/3734025838 conn(0x7ff8540a8530 msgr2=0x7ff85406ae70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:43.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.679+0000 7ff859000640 1 -- 172.21.15.28:0/3734025838 shutdown_connections 2024-02-21T23:03:43.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:43.679+0000 7ff859000640 1 -- 172.21.15.28:0/3734025838 wait complete. 2024-02-21T23:03:43.688 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-02-21T23:03:43.688 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-21T23:03:43.819 DEBUG:teuthology.orchestra.run.smithi081:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-21T23:03:43.942 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-02-21T23:03:43.942 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-21T23:03:43.942 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-21T23:03:44.002 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-21T23:03:44.003 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-21T23:03:44.059 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-21T23:03:44.059 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-21T23:03:44.116 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:03:44.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.353+0000 7f9c5f200640 1 Processor -- start 2024-02-21T23:03:44.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.353+0000 7f9c5f200640 1 -- start start 2024-02-21T23:03:44.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.354+0000 7f9c5f200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9c58150450 0x7f9c580a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:44.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.354+0000 7f9c5f200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9c5806ee40 0x7f9c581536f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:44.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.354+0000 7f9c5f200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c580c2760 con 0x7f9c58156af0 2024-02-21T23:03:44.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.355+0000 7f9c5f200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c580c22c0 con 0x7f9c5806ee40 2024-02-21T23:03:44.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.355+0000 7f9c5f200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c58156660 con 0x7f9c58150450 2024-02-21T23:03:44.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.355+0000 7f9c5d400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9c5806ee40 0x7f9c581536f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:44.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.355+0000 7f9c5d400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9c5806ee40 0x7f9c581536f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44530/0 (socket says 172.21.15.28:44530) 2024-02-21T23:03:44.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.355+0000 7f9c5d400640 1 -- 172.21.15.28:0/2034337794 learned_addr learned my addr 172.21.15.28:0/2034337794 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:44.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.355+0000 7f9c5de00640 1 --2- 172.21.15.28:0/2034337794 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9c58150450 0x7f9c580a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:44.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.355+0000 7f9c5d400640 1 -- 172.21.15.28:0/2034337794 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9c58150450 msgr2=0x7f9c580a5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:44.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.355+0000 7f9c5d400640 1 --2- 172.21.15.28:0/2034337794 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9c58150450 0x7f9c580a5fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:44.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.355+0000 7f9c5d400640 1 -- 172.21.15.28:0/2034337794 >> v1:172.21.15.81:6789/0 conn(0x7f9c58156af0 legacy=0x7f9c5806dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:44.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.355+0000 7f9c5d400640 1 -- 172.21.15.28:0/2034337794 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c58153c30 con 0x7f9c5806ee40 2024-02-21T23:03:44.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.357+0000 7f9c5d400640 1 --2- 172.21.15.28:0/2034337794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c5806ee40 0x7f9c581536f0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f9c48009870 tx=0x7f9c4802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce23a78b550e34c3 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:44.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.357+0000 7f9c5ca00640 1 -- 172.21.15.28:0/2034337794 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c48035020 con 0x7f9c5806ee40 2024-02-21T23:03:44.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.357+0000 7f9c5ca00640 1 -- 172.21.15.28:0/2034337794 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c48031640 con 0x7f9c5806ee40 2024-02-21T23:03:44.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.357+0000 7f9c5ca00640 1 -- 172.21.15.28:0/2034337794 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c4802d040 con 0x7f9c5806ee40 2024-02-21T23:03:44.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.358+0000 7f9c5f200640 1 -- 172.21.15.28:0/2034337794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c5806ee40 msgr2=0x7f9c581536f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:44.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.358+0000 7f9c5f200640 1 --2- 172.21.15.28:0/2034337794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c5806ee40 0x7f9c581536f0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f9c48009870 tx=0x7f9c4802b8f0 comp rx=0 tx=0).stop 2024-02-21T23:03:44.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.358+0000 7f9c5f200640 1 -- 172.21.15.28:0/2034337794 shutdown_connections 2024-02-21T23:03:44.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.358+0000 7f9c5f200640 1 --2- 172.21.15.28:0/2034337794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c5806ee40 0x7f9c581536f0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:44.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.358+0000 7f9c5f200640 1 --2- 172.21.15.28:0/2034337794 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9c58150450 0x7f9c580a5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:44.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.358+0000 7f9c5f200640 1 -- 172.21.15.28:0/2034337794 >> 172.21.15.28:0/2034337794 conn(0x7f9c5806b630 msgr2=0x7f9c580c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:44.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.358+0000 7f9c5f200640 1 -- 172.21.15.28:0/2034337794 shutdown_connections 2024-02-21T23:03:44.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.358+0000 7f9c5f200640 1 -- 172.21.15.28:0/2034337794 wait complete. 2024-02-21T23:03:44.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.360+0000 7f9c5f200640 1 Processor -- start 2024-02-21T23:03:44.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.360+0000 7f9c5f200640 1 -- start start 2024-02-21T23:03:44.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.361+0000 7f9c5f200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9c5806ee40 0x7f9c580a9610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:44.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.361+0000 7f9c5f200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c58150450 0x7f9c58149290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:44.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.361+0000 7f9c5f200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c58156af0 0x7f9c581497d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:44.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.361+0000 7f9c5f200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c580c2cb0 con 0x7f9c58156af0 2024-02-21T23:03:44.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.361+0000 7f9c5f200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c580c29d0 con 0x7f9c58150450 2024-02-21T23:03:44.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.361+0000 7f9c5f200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c5806e700 con 0x7f9c5806ee40 2024-02-21T23:03:44.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.361+0000 7f9c5e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c58156af0 0x7f9c581497d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:44.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.361+0000 7f9c5e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c58156af0 0x7f9c581497d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44536/0 (socket says 172.21.15.28:44536) 2024-02-21T23:03:44.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.361+0000 7f9c5e800640 1 -- 172.21.15.28:0/2146766128 learned_addr learned my addr 172.21.15.28:0/2146766128 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:44.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.361+0000 7f9c5de00640 1 --2- 172.21.15.28:0/2146766128 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9c5806ee40 0x7f9c580a9610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:44.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.361+0000 7f9c5d400640 1 --2- 172.21.15.28:0/2146766128 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c58150450 0x7f9c58149290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:44.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.362+0000 7f9c5e800640 1 -- 172.21.15.28:0/2146766128 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9c5806ee40 msgr2=0x7f9c580a9610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:44.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.362+0000 7f9c5e800640 1 --2- 172.21.15.28:0/2146766128 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9c5806ee40 0x7f9c580a9610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:44.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.362+0000 7f9c5e800640 1 -- 172.21.15.28:0/2146766128 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c58150450 msgr2=0x7f9c58149290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:44.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.362+0000 7f9c5e800640 1 --2- 172.21.15.28:0/2146766128 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c58150450 0x7f9c58149290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:44.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.362+0000 7f9c5e800640 1 -- 172.21.15.28:0/2146766128 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c580a9f50 con 0x7f9c58156af0 2024-02-21T23:03:44.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.362+0000 7f9c5e800640 1 --2- 172.21.15.28:0/2146766128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c58156af0 0x7f9c581497d0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f9c5400b9a0 tx=0x7f9c5400be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:44.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.362+0000 7f9c52000640 1 -- 172.21.15.28:0/2146766128 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c54019070 con 0x7f9c58156af0 2024-02-21T23:03:44.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.362+0000 7f9c52000640 1 -- 172.21.15.28:0/2146766128 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c5400cba0 con 0x7f9c58156af0 2024-02-21T23:03:44.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.362+0000 7f9c52000640 1 -- 172.21.15.28:0/2146766128 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c54014440 con 0x7f9c58156af0 2024-02-21T23:03:44.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.362+0000 7f9c5f200640 1 -- 172.21.15.28:0/2146766128 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c581e9c80 con 0x7f9c58156af0 2024-02-21T23:03:44.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.362+0000 7f9c5f200640 1 -- 172.21.15.28:0/2146766128 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c580a9b50 con 0x7f9c58156af0 2024-02-21T23:03:44.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.363+0000 7f9c5f200640 1 -- 172.21.15.28:0/2146766128 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c580c61b0 con 0x7f9c58156af0 2024-02-21T23:03:44.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.364+0000 7f9c52000640 1 -- 172.21.15.28:0/2146766128 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9c5401c070 con 0x7f9c58156af0 2024-02-21T23:03:44.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.365+0000 7f9c52000640 1 --2- 172.21.15.28:0/2146766128 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9c3c06b520 0x7f9c3c06d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:44.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.365+0000 7f9c52000640 1 -- 172.21.15.28:0/2146766128 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f9c5408a280 con 0x7f9c58156af0 2024-02-21T23:03:44.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.366+0000 7f9c5de00640 1 --2- 172.21.15.28:0/2146766128 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9c3c06b520 0x7f9c3c06d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:44.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.367+0000 7f9c5de00640 1 --2- 172.21.15.28:0/2146766128 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9c3c06b520 0x7f9c3c06d9e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9c4c00a970 tx=0x7f9c4c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:44.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.370+0000 7f9c52000640 1 -- 172.21.15.28:0/2146766128 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f9c580c61b0 con 0x7f9c58156af0 2024-02-21T23:03:44.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.573+0000 7f9c5f200640 1 -- 172.21.15.28:0/2146766128 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9c5814a6f0 con 0x7f9c58156af0 2024-02-21T23:03:44.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.577+0000 7f9c52000640 1 -- 172.21.15.28:0/2146766128 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+18843 (secure 0 0 0) 0x7f9c54058d70 con 0x7f9c58156af0 2024-02-21T23:03:44.566 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:03:44.566 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":26,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:03:23.860109+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:02:35.795158+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-21T23:02:36.840110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-02-21T23:02:40.441388+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":3,"to":2}]},{"pgid":"3.12","mappings":[{"from":0,"to":1}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:03:44.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.580+0000 7f9c5f200640 1 -- 172.21.15.28:0/2146766128 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9c3c06b520 msgr2=0x7f9c3c06d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:44.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.580+0000 7f9c5f200640 1 --2- 172.21.15.28:0/2146766128 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9c3c06b520 0x7f9c3c06d9e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9c4c00a970 tx=0x7f9c4c008040 comp rx=0 tx=0).stop 2024-02-21T23:03:44.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.580+0000 7f9c5f200640 1 -- 172.21.15.28:0/2146766128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c58156af0 msgr2=0x7f9c581497d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:44.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.580+0000 7f9c5f200640 1 --2- 172.21.15.28:0/2146766128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c58156af0 0x7f9c581497d0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f9c5400b9a0 tx=0x7f9c5400be60 comp rx=0 tx=0).stop 2024-02-21T23:03:44.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.580+0000 7f9c5f200640 1 -- 172.21.15.28:0/2146766128 shutdown_connections 2024-02-21T23:03:44.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.580+0000 7f9c5f200640 1 --2- 172.21.15.28:0/2146766128 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9c3c06b520 0x7f9c3c06d9e0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:44.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.580+0000 7f9c5f200640 1 --2- 172.21.15.28:0/2146766128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c58156af0 0x7f9c581497d0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:44.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.580+0000 7f9c5f200640 1 --2- 172.21.15.28:0/2146766128 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c58150450 0x7f9c58149290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:44.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.580+0000 7f9c5f200640 1 --2- 172.21.15.28:0/2146766128 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9c5806ee40 0x7f9c580a9610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:44.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.580+0000 7f9c5f200640 1 -- 172.21.15.28:0/2146766128 >> 172.21.15.28:0/2146766128 conn(0x7f9c5806b630 msgr2=0x7f9c580c5aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:44.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.580+0000 7f9c5f200640 1 -- 172.21.15.28:0/2146766128 shutdown_connections 2024-02-21T23:03:44.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.580+0000 7f9c5f200640 1 -- 172.21.15.28:0/2146766128 wait complete. 2024-02-21T23:03:44.589 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-21T23:02:30.728329+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-21T23:02:35.795158+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-02-21T23:02:36.840110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-02-21T23:02:40.441388+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-02-21T23:03:44.590 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-21T23:03:44.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.834+0000 7f37ca400640 1 Processor -- start 2024-02-21T23:03:44.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.834+0000 7f37ca400640 1 -- start start 2024-02-21T23:03:44.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.835+0000 7f37ca400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f37c40c4580 0x7f37c40c12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:44.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.836+0000 7f37ca400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f37c40c2c10 0x7f37c406ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:44.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.836+0000 7f37ca400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37c406e220 con 0x7f37c4150240 2024-02-21T23:03:44.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.836+0000 7f37ca400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37c4150620 con 0x7f37c40c4580 2024-02-21T23:03:44.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.836+0000 7f37ca400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37c4153a90 con 0x7f37c40c2c10 2024-02-21T23:03:44.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.836+0000 7f37c3e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f37c40c4580 0x7f37c40c12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:44.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.836+0000 7f37c3e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f37c40c4580 0x7f37c40c12c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44550/0 (socket says 172.21.15.28:44550) 2024-02-21T23:03:44.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.836+0000 7f37c3e00640 1 -- 172.21.15.28:0/1876852390 learned_addr learned my addr 172.21.15.28:0/1876852390 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:44.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.836+0000 7f37c9a00640 1 --2- 172.21.15.28:0/1876852390 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f37c40c2c10 0x7f37c406ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:44.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.837+0000 7f37c3e00640 1 -- 172.21.15.28:0/1876852390 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f37c40c2c10 msgr2=0x7f37c406ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:44.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.837+0000 7f37c3e00640 1 --2- 172.21.15.28:0/1876852390 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f37c40c2c10 0x7f37c406ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:44.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.837+0000 7f37c3e00640 1 -- 172.21.15.28:0/1876852390 >> v1:172.21.15.81:6789/0 conn(0x7f37c4150240 legacy=0x7f37c406db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:44.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.837+0000 7f37c3e00640 1 -- 172.21.15.28:0/1876852390 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37c406f780 con 0x7f37c40c4580 2024-02-21T23:03:44.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.838+0000 7f37c3e00640 1 --2- 172.21.15.28:0/1876852390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37c40c4580 0x7f37c40c12c0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f37ac009870 tx=0x7f37ac02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d997aefe0467ff1b server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:44.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.838+0000 7f37c3400640 1 -- 172.21.15.28:0/1876852390 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37ac035020 con 0x7f37c40c4580 2024-02-21T23:03:44.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.838+0000 7f37c3400640 1 -- 172.21.15.28:0/1876852390 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37ac031640 con 0x7f37c40c4580 2024-02-21T23:03:44.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.838+0000 7f37c3400640 1 -- 172.21.15.28:0/1876852390 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37ac02d040 con 0x7f37c40c4580 2024-02-21T23:03:44.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.839+0000 7f37ca400640 1 -- 172.21.15.28:0/1876852390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37c40c4580 msgr2=0x7f37c40c12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:44.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.839+0000 7f37ca400640 1 --2- 172.21.15.28:0/1876852390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37c40c4580 0x7f37c40c12c0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f37ac009870 tx=0x7f37ac02b8f0 comp rx=0 tx=0).stop 2024-02-21T23:03:44.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.839+0000 7f37ca400640 1 -- 172.21.15.28:0/1876852390 shutdown_connections 2024-02-21T23:03:44.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.839+0000 7f37ca400640 1 --2- 172.21.15.28:0/1876852390 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f37c40c2c10 0x7f37c406ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:44.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.839+0000 7f37ca400640 1 --2- 172.21.15.28:0/1876852390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37c40c4580 0x7f37c40c12c0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:44.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.839+0000 7f37ca400640 1 -- 172.21.15.28:0/1876852390 >> 172.21.15.28:0/1876852390 conn(0x7f37c40a8530 msgr2=0x7f37c40a8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:44.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.839+0000 7f37ca400640 1 -- 172.21.15.28:0/1876852390 shutdown_connections 2024-02-21T23:03:44.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.839+0000 7f37ca400640 1 -- 172.21.15.28:0/1876852390 wait complete. 2024-02-21T23:03:44.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.841+0000 7f37ca400640 1 Processor -- start 2024-02-21T23:03:44.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.841+0000 7f37ca400640 1 -- start start 2024-02-21T23:03:44.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.842+0000 7f37ca400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f37c40c2c10 0x7f37c41dcc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:44.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.842+0000 7f37ca400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37c40c4580 0x7f37c4152670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:44.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.842+0000 7f37ca400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f37c4150240 0x7f37c4150f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:44.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.842+0000 7f37ca400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37c40c4a10 con 0x7f37c40c4580 2024-02-21T23:03:44.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.842+0000 7f37ca400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37c4153720 con 0x7f37c4150240 2024-02-21T23:03:44.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.842+0000 7f37ca400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37c40c4b90 con 0x7f37c40c2c10 2024-02-21T23:03:44.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.842+0000 7f37c3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37c40c4580 0x7f37c4152670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:44.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.842+0000 7f37c3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37c40c4580 0x7f37c4152670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44562/0 (socket says 172.21.15.28:44562) 2024-02-21T23:03:44.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.842+0000 7f37c3e00640 1 -- 172.21.15.28:0/1222414065 learned_addr learned my addr 172.21.15.28:0/1222414065 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:44.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.843+0000 7f37c9a00640 1 --2- 172.21.15.28:0/1222414065 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f37c4150240 0x7f37c4150f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:44.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.843+0000 7f37c9000640 1 --2- 172.21.15.28:0/1222414065 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f37c40c2c10 0x7f37c41dcc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:44.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.843+0000 7f37c3e00640 1 -- 172.21.15.28:0/1222414065 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f37c40c2c10 msgr2=0x7f37c41dcc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:44.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.843+0000 7f37c3e00640 1 --2- 172.21.15.28:0/1222414065 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f37c40c2c10 0x7f37c41dcc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:44.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.843+0000 7f37c3e00640 1 -- 172.21.15.28:0/1222414065 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f37c4150240 msgr2=0x7f37c4150f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:44.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.843+0000 7f37c3e00640 1 --2- 172.21.15.28:0/1222414065 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f37c4150240 0x7f37c4150f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:44.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.843+0000 7f37c3e00640 1 -- 172.21.15.28:0/1222414065 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37c41de0b0 con 0x7f37c40c4580 2024-02-21T23:03:44.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.843+0000 7f37c3e00640 1 --2- 172.21.15.28:0/1222414065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37c40c4580 0x7f37c4152670 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f37ac02b930 tx=0x7f37ac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:44.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.843+0000 7f37c0c00640 1 -- 172.21.15.28:0/1222414065 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37ac040070 con 0x7f37c40c4580 2024-02-21T23:03:44.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.844+0000 7f37c0c00640 1 -- 172.21.15.28:0/1222414065 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37ac03e070 con 0x7f37c40c4580 2024-02-21T23:03:44.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.844+0000 7f37c0c00640 1 -- 172.21.15.28:0/1222414065 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37ac035040 con 0x7f37c40c4580 2024-02-21T23:03:44.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.844+0000 7f37ca400640 1 -- 172.21.15.28:0/1222414065 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37c41eaf70 con 0x7f37c40c4580 2024-02-21T23:03:44.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.844+0000 7f37ca400640 1 -- 172.21.15.28:0/1222414065 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37c41eb530 con 0x7f37c40c4580 2024-02-21T23:03:44.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.845+0000 7f37ca400640 1 -- 172.21.15.28:0/1222414065 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37c404f290 con 0x7f37c40c4580 2024-02-21T23:03:44.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.846+0000 7f37c0c00640 1 -- 172.21.15.28:0/1222414065 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f37ac038020 con 0x7f37c40c4580 2024-02-21T23:03:44.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.847+0000 7f37c0c00640 1 --2- 172.21.15.28:0/1222414065 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f37a006b4d0 0x7f37a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:44.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.847+0000 7f37c0c00640 1 -- 172.21.15.28:0/1222414065 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f37ac0ac0a0 con 0x7f37c40c4580 2024-02-21T23:03:44.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.847+0000 7f37c9000640 1 --2- 172.21.15.28:0/1222414065 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f37a006b4d0 0x7f37a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:44.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.848+0000 7f37c9000640 1 --2- 172.21.15.28:0/1222414065 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f37a006b4d0 0x7f37a006d990 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f37b4007fd0 tx=0x7f37b4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:44.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:44.851+0000 7f37c0c00640 1 -- 172.21.15.28:0/1222414065 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f37ac07ab10 con 0x7f37c40c4580 2024-02-21T23:03:45.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.051+0000 7f37ca400640 1 -- 172.21.15.28:0/1222414065 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f37c406f0e0 con 0x7f37c40c4580 2024-02-21T23:03:45.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.052+0000 7f37c0c00640 1 -- 172.21.15.28:0/1222414065 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) v1 ==== 93+0+10 (secure 0 0 0) 0x7f37ac07a410 con 0x7f37c40c4580 2024-02-21T23:03:45.041 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-02-21T23:03:45.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.055+0000 7f37ca400640 1 -- 172.21.15.28:0/1222414065 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f37a006b4d0 msgr2=0x7f37a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:45.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.055+0000 7f37ca400640 1 --2- 172.21.15.28:0/1222414065 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f37a006b4d0 0x7f37a006d990 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f37b4007fd0 tx=0x7f37b4009040 comp rx=0 tx=0).stop 2024-02-21T23:03:45.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.055+0000 7f37ca400640 1 -- 172.21.15.28:0/1222414065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37c40c4580 msgr2=0x7f37c4152670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:45.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.055+0000 7f37ca400640 1 --2- 172.21.15.28:0/1222414065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37c40c4580 0x7f37c4152670 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f37ac02b930 tx=0x7f37ac0046c0 comp rx=0 tx=0).stop 2024-02-21T23:03:45.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.055+0000 7f37ca400640 1 -- 172.21.15.28:0/1222414065 shutdown_connections 2024-02-21T23:03:45.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.055+0000 7f37ca400640 1 --2- 172.21.15.28:0/1222414065 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f37a006b4d0 0x7f37a006d990 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:45.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.055+0000 7f37ca400640 1 --2- 172.21.15.28:0/1222414065 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f37c4150240 0x7f37c4150f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:45.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.055+0000 7f37ca400640 1 --2- 172.21.15.28:0/1222414065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37c40c4580 0x7f37c4152670 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:45.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.055+0000 7f37ca400640 1 --2- 172.21.15.28:0/1222414065 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f37c40c2c10 0x7f37c41dcc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:45.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.055+0000 7f37ca400640 1 -- 172.21.15.28:0/1222414065 >> 172.21.15.28:0/1222414065 conn(0x7f37c40a8530 msgr2=0x7f37c414ebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:45.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.055+0000 7f37ca400640 1 -- 172.21.15.28:0/1222414065 shutdown_connections 2024-02-21T23:03:45.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.055+0000 7f37ca400640 1 -- 172.21.15.28:0/1222414065 wait complete. 2024-02-21T23:03:45.062 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-21T23:03:45.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.296+0000 7fdeade00640 1 Processor -- start 2024-02-21T23:03:45.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.297+0000 7fdeade00640 1 -- start start 2024-02-21T23:03:45.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.298+0000 7fdeade00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdea806ae70 0x7fdea806b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:45.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.298+0000 7fdeade00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdea8159c50 0x7fdea8163b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:45.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.298+0000 7fdeade00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdea80c0d40 con 0x7fdea80c4c10 2024-02-21T23:03:45.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.298+0000 7fdeade00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdea815caf0 con 0x7fdea8159c50 2024-02-21T23:03:45.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.298+0000 7fdeade00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdea815cdd0 con 0x7fdea806ae70 2024-02-21T23:03:45.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.299+0000 7fdead400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdea8159c50 0x7fdea8163b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:45.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.299+0000 7fdead400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdea8159c50 0x7fdea8163b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41666/0 (socket says 172.21.15.28:41666) 2024-02-21T23:03:45.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.299+0000 7fdead400640 1 -- 172.21.15.28:0/258571242 learned_addr learned my addr 172.21.15.28:0/258571242 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:45.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.299+0000 7fdeaca00640 1 --2- 172.21.15.28:0/258571242 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdea806ae70 0x7fdea806b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:45.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.299+0000 7fdead400640 1 -- 172.21.15.28:0/258571242 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdea806ae70 msgr2=0x7fdea806b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:45.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.299+0000 7fdead400640 1 --2- 172.21.15.28:0/258571242 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdea806ae70 0x7fdea806b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:45.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.299+0000 7fdead400640 1 -- 172.21.15.28:0/258571242 >> v1:172.21.15.81:6789/0 conn(0x7fdea80c4c10 legacy=0x7fdea8159540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:45.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.299+0000 7fdead400640 1 -- 172.21.15.28:0/258571242 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdea815d0b0 con 0x7fdea8159c50 2024-02-21T23:03:45.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.300+0000 7fdead400640 1 --2- 172.21.15.28:0/258571242 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdea8159c50 0x7fdea8163b70 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fdea0009e50 tx=0x7fdea002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89761132a6f028a4 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:45.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.301+0000 7fdea7400640 1 -- 172.21.15.28:0/258571242 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea002ca70 con 0x7fdea8159c50 2024-02-21T23:03:45.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.301+0000 7fdea7400640 1 -- 172.21.15.28:0/258571242 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdea002cc10 con 0x7fdea8159c50 2024-02-21T23:03:45.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.301+0000 7fdea7400640 1 -- 172.21.15.28:0/258571242 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea00343f0 con 0x7fdea8159c50 2024-02-21T23:03:45.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.302+0000 7fdeade00640 1 -- 172.21.15.28:0/258571242 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdea8159c50 msgr2=0x7fdea8163b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:45.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.302+0000 7fdeade00640 1 --2- 172.21.15.28:0/258571242 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdea8159c50 0x7fdea8163b70 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fdea0009e50 tx=0x7fdea002bed0 comp rx=0 tx=0).stop 2024-02-21T23:03:45.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.302+0000 7fdeade00640 1 -- 172.21.15.28:0/258571242 shutdown_connections 2024-02-21T23:03:45.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.302+0000 7fdeade00640 1 --2- 172.21.15.28:0/258571242 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdea8159c50 0x7fdea8163b70 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:45.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.302+0000 7fdeade00640 1 --2- 172.21.15.28:0/258571242 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdea806ae70 0x7fdea806b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:45.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.302+0000 7fdeade00640 1 -- 172.21.15.28:0/258571242 >> 172.21.15.28:0/258571242 conn(0x7fdea80a8530 msgr2=0x7fdea80a8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:45.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.302+0000 7fdeade00640 1 -- 172.21.15.28:0/258571242 shutdown_connections 2024-02-21T23:03:45.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.302+0000 7fdeade00640 1 -- 172.21.15.28:0/258571242 wait complete. 2024-02-21T23:03:45.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.303+0000 7fdeade00640 1 Processor -- start 2024-02-21T23:03:45.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.304+0000 7fdeade00640 1 -- start start 2024-02-21T23:03:45.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.305+0000 7fdeade00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdea806ae70 0x7fdea8151520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:45.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.305+0000 7fdeade00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdea80c4c10 0x7fdea81f21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:45.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.305+0000 7fdeade00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdea8159c50 0x7fdea8072200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:45.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.305+0000 7fdeade00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdea815c7e0 con 0x7fdea8159c50 2024-02-21T23:03:45.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.305+0000 7fdeade00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdea815d670 con 0x7fdea80c4c10 2024-02-21T23:03:45.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.305+0000 7fdeade00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdea806f6b0 con 0x7fdea806ae70 2024-02-21T23:03:45.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.305+0000 7fdead400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdea8159c50 0x7fdea8072200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:45.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.305+0000 7fdead400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdea8159c50 0x7fdea8072200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41680/0 (socket says 172.21.15.28:41680) 2024-02-21T23:03:45.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.305+0000 7fdead400640 1 -- 172.21.15.28:0/2861613571 learned_addr learned my addr 172.21.15.28:0/2861613571 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:45.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.305+0000 7fdeaca00640 1 --2- 172.21.15.28:0/2861613571 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdea806ae70 0x7fdea8151520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:45.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.305+0000 7fdea7e00640 1 --2- 172.21.15.28:0/2861613571 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdea80c4c10 0x7fdea81f21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:45.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.305+0000 7fdead400640 1 -- 172.21.15.28:0/2861613571 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdea806ae70 msgr2=0x7fdea8151520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:45.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.305+0000 7fdead400640 1 --2- 172.21.15.28:0/2861613571 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdea806ae70 0x7fdea8151520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:45.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.306+0000 7fdead400640 1 -- 172.21.15.28:0/2861613571 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdea80c4c10 msgr2=0x7fdea81f21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:45.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.306+0000 7fdead400640 1 --2- 172.21.15.28:0/2861613571 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdea80c4c10 0x7fdea81f21e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:45.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.306+0000 7fdead400640 1 -- 172.21.15.28:0/2861613571 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdea8151a60 con 0x7fdea8159c50 2024-02-21T23:03:45.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.306+0000 7fdead400640 1 --2- 172.21.15.28:0/2861613571 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdea8159c50 0x7fdea8072200 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fdea0036040 tx=0x7fdea0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:45.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.306+0000 7fdea4c00640 1 -- 172.21.15.28:0/2861613571 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea0043020 con 0x7fdea8159c50 2024-02-21T23:03:45.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.306+0000 7fdea4c00640 1 -- 172.21.15.28:0/2861613571 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdea003e450 con 0x7fdea8159c50 2024-02-21T23:03:45.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.306+0000 7fdea4c00640 1 -- 172.21.15.28:0/2861613571 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea0009510 con 0x7fdea8159c50 2024-02-21T23:03:45.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.306+0000 7fdeade00640 1 -- 172.21.15.28:0/2861613571 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdea81e28e0 con 0x7fdea8159c50 2024-02-21T23:03:45.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.306+0000 7fdeade00640 1 -- 172.21.15.28:0/2861613571 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdea814f600 con 0x7fdea8159c50 2024-02-21T23:03:45.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.308+0000 7fdeade00640 1 -- 172.21.15.28:0/2861613571 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdea804f290 con 0x7fdea8159c50 2024-02-21T23:03:45.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.309+0000 7fdea4c00640 1 -- 172.21.15.28:0/2861613571 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdea0046020 con 0x7fdea8159c50 2024-02-21T23:03:45.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.310+0000 7fdea4c00640 1 --2- 172.21.15.28:0/2861613571 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fde8806b500 0x7fde8806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:45.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.310+0000 7fdea4c00640 1 -- 172.21.15.28:0/2861613571 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6735+0+0 (secure 0 0 0) 0x7fdea00acad0 con 0x7fdea8159c50 2024-02-21T23:03:45.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.311+0000 7fdeaca00640 1 --2- 172.21.15.28:0/2861613571 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fde8806b500 0x7fde8806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:45.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.312+0000 7fdeaca00640 1 --2- 172.21.15.28:0/2861613571 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fde8806b500 0x7fde8806d9c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fde9c006fd0 tx=0x7fde9c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:45.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.314+0000 7fdea4c00640 1 -- 172.21.15.28:0/2861613571 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fdea004d070 con 0x7fdea8159c50 2024-02-21T23:03:45.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.512+0000 7fdeade00640 1 -- 172.21.15.28:0/2861613571 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fdea80c4580 con 0x7fdea8159c50 2024-02-21T23:03:45.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.513+0000 7fdea4c00640 1 -- 172.21.15.28:0/2861613571 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+11 (secure 0 0 0) 0x7fdea007b540 con 0x7fdea8159c50 2024-02-21T23:03:45.502 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-02-21T23:03:45.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.516+0000 7fdeade00640 1 -- 172.21.15.28:0/2861613571 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fde8806b500 msgr2=0x7fde8806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:45.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.516+0000 7fdeade00640 1 --2- 172.21.15.28:0/2861613571 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fde8806b500 0x7fde8806d9c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fde9c006fd0 tx=0x7fde9c008040 comp rx=0 tx=0).stop 2024-02-21T23:03:45.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.516+0000 7fdeade00640 1 -- 172.21.15.28:0/2861613571 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdea8159c50 msgr2=0x7fdea8072200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:45.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.516+0000 7fdeade00640 1 --2- 172.21.15.28:0/2861613571 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdea8159c50 0x7fdea8072200 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fdea0036040 tx=0x7fdea0004a40 comp rx=0 tx=0).stop 2024-02-21T23:03:45.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.516+0000 7fdeade00640 1 -- 172.21.15.28:0/2861613571 shutdown_connections 2024-02-21T23:03:45.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.516+0000 7fdeade00640 1 --2- 172.21.15.28:0/2861613571 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fde8806b500 0x7fde8806d9c0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:45.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.516+0000 7fdeade00640 1 --2- 172.21.15.28:0/2861613571 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdea8159c50 0x7fdea8072200 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:45.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.516+0000 7fdeade00640 1 --2- 172.21.15.28:0/2861613571 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdea80c4c10 0x7fdea81f21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:45.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.516+0000 7fdeade00640 1 --2- 172.21.15.28:0/2861613571 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdea806ae70 0x7fdea8151520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:45.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.516+0000 7fdeade00640 1 -- 172.21.15.28:0/2861613571 >> 172.21.15.28:0/2861613571 conn(0x7fdea80a8530 msgr2=0x7fdea80a8910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:45.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.516+0000 7fdeade00640 1 -- 172.21.15.28:0/2861613571 shutdown_connections 2024-02-21T23:03:45.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.516+0000 7fdeade00640 1 -- 172.21.15.28:0/2861613571 wait complete. 2024-02-21T23:03:45.525 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-21T23:03:45.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.766+0000 7f8000000640 1 Processor -- start 2024-02-21T23:03:45.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.766+0000 7f8000000640 1 -- start start 2024-02-21T23:03:45.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.767+0000 7f8000000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7ff80c5cc0 0x7f7ff80c6c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:45.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.767+0000 7f8000000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f7ff80c3030 0x7f7ff80c3440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:45.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.767+0000 7f8000000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ff8159580 con 0x7f7ff80c7590 2024-02-21T23:03:45.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.767+0000 7f8000000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ff80c6230 con 0x7f7ff80c5cc0 2024-02-21T23:03:45.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.767+0000 7f8000000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ff81587c0 con 0x7f7ff80c3030 2024-02-21T23:03:45.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.767+0000 7f7ffcc00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7ff80c5cc0 0x7f7ff80c6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:45.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.768+0000 7f7ffcc00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7ff80c5cc0 0x7f7ff80c6c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41690/0 (socket says 172.21.15.28:41690) 2024-02-21T23:03:45.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.768+0000 7f7ffcc00640 1 -- 172.21.15.28:0/929847621 learned_addr learned my addr 172.21.15.28:0/929847621 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:45.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.768+0000 7f7ffe000640 1 --2- 172.21.15.28:0/929847621 >> v2:172.21.15.81:3300/0 conn(0x7f7ff80c3030 0x7f7ff80c3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:45.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.768+0000 7f7ffcc00640 1 -- 172.21.15.28:0/929847621 >> v2:172.21.15.81:3300/0 conn(0x7f7ff80c3030 msgr2=0x7f7ff80c3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:45.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.768+0000 7f7ffcc00640 1 --2- 172.21.15.28:0/929847621 >> v2:172.21.15.81:3300/0 conn(0x7f7ff80c3030 0x7f7ff80c3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:45.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.768+0000 7f7ffcc00640 1 -- 172.21.15.28:0/929847621 >> v1:172.21.15.81:6789/0 conn(0x7f7ff80c7590 legacy=0x7f7ff81524e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:45.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.768+0000 7f7ffcc00640 1 -- 172.21.15.28:0/929847621 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ff8158aa0 con 0x7f7ff80c5cc0 2024-02-21T23:03:45.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.770+0000 7f7ffcc00640 1 --2- 172.21.15.28:0/929847621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ff80c5cc0 0x7f7ff80c6c60 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f7ff8073140 tx=0x7f7fe402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57ba6a2ac0381204 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:45.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.770+0000 7f7ff7e00640 1 -- 172.21.15.28:0/929847621 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fe402e070 con 0x7f7ff80c5cc0 2024-02-21T23:03:45.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.770+0000 7f7ff7e00640 1 -- 172.21.15.28:0/929847621 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fe4037070 con 0x7f7ff80c5cc0 2024-02-21T23:03:45.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.770+0000 7f7ff7e00640 1 -- 172.21.15.28:0/929847621 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fe4031a00 con 0x7f7ff80c5cc0 2024-02-21T23:03:45.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.770+0000 7f8000000640 1 -- 172.21.15.28:0/929847621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ff80c5cc0 msgr2=0x7f7ff80c6c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:45.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.770+0000 7f8000000640 1 --2- 172.21.15.28:0/929847621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ff80c5cc0 0x7f7ff80c6c60 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f7ff8073140 tx=0x7f7fe402b880 comp rx=0 tx=0).stop 2024-02-21T23:03:45.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.771+0000 7f8000000640 1 -- 172.21.15.28:0/929847621 shutdown_connections 2024-02-21T23:03:45.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.771+0000 7f8000000640 1 --2- 172.21.15.28:0/929847621 >> v2:172.21.15.81:3300/0 conn(0x7f7ff80c3030 0x7f7ff80c3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:45.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.771+0000 7f8000000640 1 --2- 172.21.15.28:0/929847621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ff80c5cc0 0x7f7ff80c6c60 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:45.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.771+0000 7f8000000640 1 -- 172.21.15.28:0/929847621 >> 172.21.15.28:0/929847621 conn(0x7f7ff808afe0 msgr2=0x7f7ff80725a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:45.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.771+0000 7f8000000640 1 -- 172.21.15.28:0/929847621 shutdown_connections 2024-02-21T23:03:45.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.771+0000 7f8000000640 1 -- 172.21.15.28:0/929847621 wait complete. 2024-02-21T23:03:45.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.772+0000 7f8000000640 1 Processor -- start 2024-02-21T23:03:45.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.773+0000 7f8000000640 1 -- start start 2024-02-21T23:03:45.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.773+0000 7f8000000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ff80c3030 0x7f7ff81eb0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:45.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.773+0000 7f8000000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ff80c5cc0 0x7f7ff814eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:45.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.773+0000 7f8000000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7ff80c7590 0x7f7ff814f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:45.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.773+0000 7f8000000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff80c6720 con 0x7f7ff80c3030 2024-02-21T23:03:45.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.773+0000 7f8000000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff8159060 con 0x7f7ff80c5cc0 2024-02-21T23:03:45.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.773+0000 7f8000000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff80c68a0 con 0x7f7ff80c7590 2024-02-21T23:03:45.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.774+0000 7f7ffd600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ff80c3030 0x7f7ff81eb0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:45.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.774+0000 7f7ffd600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ff80c3030 0x7f7ff81eb0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41698/0 (socket says 172.21.15.28:41698) 2024-02-21T23:03:45.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.774+0000 7f7ffd600640 1 -- 172.21.15.28:0/1555547426 learned_addr learned my addr 172.21.15.28:0/1555547426 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:45.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.774+0000 7f7ffcc00640 1 --2- 172.21.15.28:0/1555547426 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ff80c5cc0 0x7f7ff814eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:45.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.774+0000 7f7ffe000640 1 --2- 172.21.15.28:0/1555547426 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7ff80c7590 0x7f7ff814f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:45.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.774+0000 7f7ffd600640 1 -- 172.21.15.28:0/1555547426 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7ff80c7590 msgr2=0x7f7ff814f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:45.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.774+0000 7f7ffd600640 1 --2- 172.21.15.28:0/1555547426 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7ff80c7590 0x7f7ff814f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:45.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.774+0000 7f7ffd600640 1 -- 172.21.15.28:0/1555547426 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ff80c5cc0 msgr2=0x7f7ff814eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:45.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.774+0000 7f7ffd600640 1 --2- 172.21.15.28:0/1555547426 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ff80c5cc0 0x7f7ff814eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:45.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.774+0000 7f7ffd600640 1 -- 172.21.15.28:0/1555547426 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ff81ebaa0 con 0x7f7ff80c3030 2024-02-21T23:03:45.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.775+0000 7f7ffd600640 1 --2- 172.21.15.28:0/1555547426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ff80c3030 0x7f7ff81eb0b0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f7fec00b3f0 tx=0x7f7fec00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:45.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.775+0000 7f7ff5600640 1 -- 172.21.15.28:0/1555547426 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fec015020 con 0x7f7ff80c3030 2024-02-21T23:03:45.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.775+0000 7f7ff5600640 1 -- 172.21.15.28:0/1555547426 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fec011830 con 0x7f7ff80c3030 2024-02-21T23:03:45.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.775+0000 7f7ff5600640 1 -- 172.21.15.28:0/1555547426 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fec011c20 con 0x7f7ff80c3030 2024-02-21T23:03:45.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.775+0000 7f8000000640 1 -- 172.21.15.28:0/1555547426 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ff814d020 con 0x7f7ff80c3030 2024-02-21T23:03:45.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.775+0000 7f8000000640 1 -- 172.21.15.28:0/1555547426 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ff8155950 con 0x7f7ff80c3030 2024-02-21T23:03:45.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.776+0000 7f8000000640 1 -- 172.21.15.28:0/1555547426 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ff8067f30 con 0x7f7ff80c3030 2024-02-21T23:03:45.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.777+0000 7f7ff5600640 1 -- 172.21.15.28:0/1555547426 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7fec018020 con 0x7f7ff80c3030 2024-02-21T23:03:45.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.778+0000 7f7ff5600640 1 --2- 172.21.15.28:0/1555547426 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7fd806b4d0 0x7f7fd806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:45.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.778+0000 7f7ff5600640 1 -- 172.21.15.28:0/1555547426 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f7fec08a240 con 0x7f7ff80c3030 2024-02-21T23:03:45.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.779+0000 7f7ffcc00640 1 --2- 172.21.15.28:0/1555547426 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7fd806b4d0 0x7f7fd806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:45.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.780+0000 7f7ffcc00640 1 --2- 172.21.15.28:0/1555547426 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7fd806b4d0 0x7f7fd806d990 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7fe4036000 tx=0x7f7fe403d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:45.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.783+0000 7f7ff5600640 1 -- 172.21.15.28:0/1555547426 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7fec01f030 con 0x7f7ff80c3030 2024-02-21T23:03:45.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.981+0000 7f8000000640 1 -- 172.21.15.28:0/1555547426 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7ff80c3f90 con 0x7f7ff80c3030 2024-02-21T23:03:45.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.982+0000 7f7ff5600640 1 -- 172.21.15.28:0/1555547426 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7fec0172c0 con 0x7f7ff80c3030 2024-02-21T23:03:45.970 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-02-21T23:03:45.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.984+0000 7f8000000640 1 -- 172.21.15.28:0/1555547426 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7fd806b4d0 msgr2=0x7f7fd806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:45.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.984+0000 7f8000000640 1 --2- 172.21.15.28:0/1555547426 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7fd806b4d0 0x7f7fd806d990 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7fe4036000 tx=0x7f7fe403d040 comp rx=0 tx=0).stop 2024-02-21T23:03:45.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.984+0000 7f8000000640 1 -- 172.21.15.28:0/1555547426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ff80c3030 msgr2=0x7f7ff81eb0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:45.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.984+0000 7f8000000640 1 --2- 172.21.15.28:0/1555547426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ff80c3030 0x7f7ff81eb0b0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f7fec00b3f0 tx=0x7f7fec00b8b0 comp rx=0 tx=0).stop 2024-02-21T23:03:45.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.985+0000 7f8000000640 1 -- 172.21.15.28:0/1555547426 shutdown_connections 2024-02-21T23:03:45.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.985+0000 7f8000000640 1 --2- 172.21.15.28:0/1555547426 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7fd806b4d0 0x7f7fd806d990 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:45.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.985+0000 7f8000000640 1 --2- 172.21.15.28:0/1555547426 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7ff80c7590 0x7f7ff814f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:45.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.985+0000 7f8000000640 1 --2- 172.21.15.28:0/1555547426 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ff80c5cc0 0x7f7ff814eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:45.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.985+0000 7f8000000640 1 --2- 172.21.15.28:0/1555547426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ff80c3030 0x7f7ff81eb0b0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:45.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.985+0000 7f8000000640 1 -- 172.21.15.28:0/1555547426 >> 172.21.15.28:0/1555547426 conn(0x7f7ff808afe0 msgr2=0x7f7ff80c44b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:45.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.985+0000 7f8000000640 1 -- 172.21.15.28:0/1555547426 shutdown_connections 2024-02-21T23:03:45.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:45.985+0000 7f8000000640 1 -- 172.21.15.28:0/1555547426 wait complete. 2024-02-21T23:03:45.993 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-02-21T23:03:46.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.227+0000 7f861b200640 1 Processor -- start 2024-02-21T23:03:46.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.227+0000 7f861b200640 1 -- start start 2024-02-21T23:03:46.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.228+0000 7f861b200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f861c152560 0x7f861c154f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:46.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.228+0000 7f861b200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f861c0a6270 con 0x7f861c06e9d0 2024-02-21T23:03:46.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.228+0000 7f861b200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f861c1556d0 con 0x7f861c158480 2024-02-21T23:03:46.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.228+0000 7f861b200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f861c1554a0 con 0x7f861c152560 2024-02-21T23:03:46.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.228+0000 7f861a800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f861c06e9d0 0x7f861c06ee00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:46980/0 (socket says 172.21.15.28:46980) 2024-02-21T23:03:46.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.228+0000 7f861a800640 1 -- 172.21.15.28:0/2793264126 learned_addr learned my addr 172.21.15.28:0/2793264126 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:03:46.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.229+0000 7f8619e00640 1 --2- 172.21.15.28:0/2793264126 >> v2:172.21.15.81:3300/0 conn(0x7f861c152560 0x7f861c154f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:46.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.229+0000 7f8619400640 1 -- 172.21.15.28:0/2793264126 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2891580806 0 0) 0x7f861c0a6270 con 0x7f861c06e9d0 2024-02-21T23:03:46.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.229+0000 7f8619400640 1 -- 172.21.15.28:0/2793264126 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8600003520 con 0x7f861c06e9d0 2024-02-21T23:03:46.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.229+0000 7f8619400640 1 -- 172.21.15.28:0/2793264126 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 152559823 0 0) 0x7f8600003520 con 0x7f861c06e9d0 2024-02-21T23:03:46.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.229+0000 7f8619400640 1 -- 172.21.15.28:0/2793264126 >> v2:172.21.15.81:3300/0 conn(0x7f861c152560 msgr2=0x7f861c154f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:46.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.229+0000 7f8619400640 1 --2- 172.21.15.28:0/2793264126 >> v2:172.21.15.81:3300/0 conn(0x7f861c152560 0x7f861c154f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:46.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.229+0000 7f8619400640 1 -- 172.21.15.28:0/2793264126 >> v1:172.21.15.81:6789/0 conn(0x7f861c158480 legacy=0x7f861c06ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:46.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.229+0000 7f8619400640 1 -- 172.21.15.28:0/2793264126 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f861c1559b0 con 0x7f861c06e9d0 2024-02-21T23:03:46.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.229+0000 7f8619400640 1 -- 172.21.15.28:0/2793264126 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f86140019f0 con 0x7f861c06e9d0 2024-02-21T23:03:46.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.229+0000 7f8619400640 1 -- 172.21.15.28:0/2793264126 >> v1:172.21.15.28:6789/0 conn(0x7f861c06e9d0 legacy=0x7f861c06ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:46.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.229+0000 7f8619400640 1 --2- 172.21.15.28:0/2793264126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86000039b0 0x7f8600003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:46.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.229+0000 7f8619400640 1 --2- 172.21.15.28:0/2793264126 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8600004800 0x7f8600004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:46.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.229+0000 7f8619400640 1 --2- 172.21.15.28:0/2793264126 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f86000051c0 0x7f860000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:46.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.230+0000 7f8619400640 1 -- 172.21.15.28:0/2793264126 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f860000c030 con 0x7f86000039b0 2024-02-21T23:03:46.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.230+0000 7f8619400640 1 -- 172.21.15.28:0/2793264126 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f860000c1e0 con 0x7f8600004800 2024-02-21T23:03:46.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.230+0000 7f8619400640 1 -- 172.21.15.28:0/2793264126 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f860000c3e0 con 0x7f86000051c0 2024-02-21T23:03:46.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.230+0000 7f8619e00640 1 --2- 172.21.15.28:0/2793264126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86000039b0 0x7f8600003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:46.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.230+0000 7f8619e00640 1 -- 172.21.15.28:0/2793264126 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f86000051c0 msgr2=0x7f860000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:46.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.230+0000 7f8619e00640 1 --2- 172.21.15.28:0/2793264126 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f86000051c0 0x7f860000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:46.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.230+0000 7f8613e00640 1 --2- 172.21.15.28:0/2793264126 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8600004800 0x7f8600004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:46.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.230+0000 7f8619e00640 1 -- 172.21.15.28:0/2793264126 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8600004800 msgr2=0x7f8600004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:46.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.230+0000 7f8619e00640 1 --2- 172.21.15.28:0/2793264126 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8600004800 0x7f8600004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:46.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.230+0000 7f861a800640 1 -- 172.21.15.28:0/2793264126 reap_dead start 2024-02-21T23:03:46.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.230+0000 7f8619e00640 1 -- 172.21.15.28:0/2793264126 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8600003520 con 0x7f86000039b0 2024-02-21T23:03:46.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.230+0000 7f8613e00640 1 --2- 172.21.15.28:0/2793264126 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8600004800 0x7f8600004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:03:46.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.231+0000 7f8619e00640 1 --2- 172.21.15.28:0/2793264126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86000039b0 0x7f8600003dd0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f860c009ac0 tx=0x7f8600005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edb696afa5e72333 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:46.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.232+0000 7f8619400640 1 -- 172.21.15.28:0/2793264126 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f860c039070 con 0x7f86000039b0 2024-02-21T23:03:46.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.232+0000 7f8619400640 1 -- 172.21.15.28:0/2793264126 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f860c02c6b0 con 0x7f86000039b0 2024-02-21T23:03:46.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.232+0000 7f8619400640 1 -- 172.21.15.28:0/2793264126 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f860c009690 con 0x7f86000039b0 2024-02-21T23:03:46.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.232+0000 7f861b200640 1 -- 172.21.15.28:0/2793264126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86000039b0 msgr2=0x7f8600003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:46.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.232+0000 7f861b200640 1 --2- 172.21.15.28:0/2793264126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86000039b0 0x7f8600003dd0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f860c009ac0 tx=0x7f8600005830 comp rx=0 tx=0).stop 2024-02-21T23:03:46.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.232+0000 7f861b200640 1 -- 172.21.15.28:0/2793264126 shutdown_connections 2024-02-21T23:03:46.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.232+0000 7f861b200640 1 --2- 172.21.15.28:0/2793264126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86000039b0 0x7f8600003dd0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:46.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.232+0000 7f861b200640 1 -- 172.21.15.28:0/2793264126 >> 172.21.15.28:0/2793264126 conn(0x7f861c0a8530 msgr2=0x7f861c0a8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:46.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.232+0000 7f861b200640 1 -- 172.21.15.28:0/2793264126 shutdown_connections 2024-02-21T23:03:46.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.232+0000 7f861b200640 1 -- 172.21.15.28:0/2793264126 wait complete. 2024-02-21T23:03:46.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.234+0000 7f861b200640 1 Processor -- start 2024-02-21T23:03:46.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.234+0000 7f861b200640 1 -- start start 2024-02-21T23:03:46.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.235+0000 7f861b200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f86000039b0 0x7f861c14ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:46.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.235+0000 7f861b200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f861c1eab90 0x7f861c144c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:46.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.235+0000 7f861b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f861c1451a0 0x7f861c1ede60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:46.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.235+0000 7f861b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f861c152940 con 0x7f861c1451a0 2024-02-21T23:03:46.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.235+0000 7f861b200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f861c06e4b0 con 0x7f861c1eab90 2024-02-21T23:03:46.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.235+0000 7f861b200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f861c0c1bc0 con 0x7f86000039b0 2024-02-21T23:03:46.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.236+0000 7f861a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f861c1451a0 0x7f861c1ede60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:46.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.236+0000 7f861a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f861c1451a0 0x7f861c1ede60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41718/0 (socket says 172.21.15.28:41718) 2024-02-21T23:03:46.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.236+0000 7f861a800640 1 -- 172.21.15.28:0/3892040815 learned_addr learned my addr 172.21.15.28:0/3892040815 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:46.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.236+0000 7f8619e00640 1 --2- 172.21.15.28:0/3892040815 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f86000039b0 0x7f861c14ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:46.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.236+0000 7f8613e00640 1 --2- 172.21.15.28:0/3892040815 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f861c1eab90 0x7f861c144c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:46.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.236+0000 7f861a800640 1 -- 172.21.15.28:0/3892040815 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f86000039b0 msgr2=0x7f861c14ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:46.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.236+0000 7f861a800640 1 --2- 172.21.15.28:0/3892040815 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f86000039b0 0x7f861c14ff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:46.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.236+0000 7f861a800640 1 -- 172.21.15.28:0/3892040815 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f861c1eab90 msgr2=0x7f861c144c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:46.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.236+0000 7f861a800640 1 --2- 172.21.15.28:0/3892040815 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f861c1eab90 0x7f861c144c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:46.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.236+0000 7f861a800640 1 -- 172.21.15.28:0/3892040815 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f861c1eb460 con 0x7f861c1451a0 2024-02-21T23:03:46.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.236+0000 7f861a800640 1 --2- 172.21.15.28:0/3892040815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f861c1451a0 0x7f861c1ede60 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f861c06e0c0 tx=0x7f861400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:46.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.237+0000 7f8612000640 1 -- 172.21.15.28:0/3892040815 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f861400ce70 con 0x7f861c1451a0 2024-02-21T23:03:46.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.237+0000 7f8612000640 1 -- 172.21.15.28:0/3892040815 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f861400a610 con 0x7f861c1451a0 2024-02-21T23:03:46.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.237+0000 7f8612000640 1 -- 172.21.15.28:0/3892040815 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f861400aa20 con 0x7f861c1451a0 2024-02-21T23:03:46.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.237+0000 7f861b200640 1 -- 172.21.15.28:0/3892040815 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f861c150830 con 0x7f861c1451a0 2024-02-21T23:03:46.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.237+0000 7f861b200640 1 -- 172.21.15.28:0/3892040815 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f861c145ae0 con 0x7f861c1451a0 2024-02-21T23:03:46.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.238+0000 7f861b200640 1 -- 172.21.15.28:0/3892040815 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f861c04f290 con 0x7f861c1451a0 2024-02-21T23:03:46.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.239+0000 7f8612000640 1 -- 172.21.15.28:0/3892040815 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f861400abc0 con 0x7f861c1451a0 2024-02-21T23:03:46.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.240+0000 7f8612000640 1 --2- 172.21.15.28:0/3892040815 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f85f806b500 0x7f85f806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:46.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.240+0000 7f8612000640 1 -- 172.21.15.28:0/3892040815 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f8614089c70 con 0x7f861c1451a0 2024-02-21T23:03:46.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.240+0000 7f8619e00640 1 --2- 172.21.15.28:0/3892040815 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f85f806b500 0x7f85f806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:46.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.242+0000 7f8619e00640 1 --2- 172.21.15.28:0/3892040815 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f85f806b500 0x7f85f806d9c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f860c036000 tx=0x7f860c030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:46.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.244+0000 7f8612000640 1 -- 172.21.15.28:0/3892040815 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f861401b070 con 0x7f861c1451a0 2024-02-21T23:03:46.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.449+0000 7f861b200640 1 -- 172.21.15.28:0/3892040815 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f861c06e0f0 con 0x7f861c1451a0 2024-02-21T23:03:46.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.450+0000 7f8612000640 1 -- 172.21.15.28:0/3892040815 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) v1 ==== 103+0+11 (secure 0 0 0) 0x7f86140586e0 con 0x7f861c1451a0 2024-02-21T23:03:46.438 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-02-21T23:03:46.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.452+0000 7f861b200640 1 -- 172.21.15.28:0/3892040815 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f85f806b500 msgr2=0x7f85f806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:46.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.452+0000 7f861b200640 1 --2- 172.21.15.28:0/3892040815 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f85f806b500 0x7f85f806d9c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f860c036000 tx=0x7f860c030000 comp rx=0 tx=0).stop 2024-02-21T23:03:46.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.452+0000 7f861b200640 1 -- 172.21.15.28:0/3892040815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f861c1451a0 msgr2=0x7f861c1ede60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:46.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.452+0000 7f861b200640 1 --2- 172.21.15.28:0/3892040815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f861c1451a0 0x7f861c1ede60 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f861c06e0c0 tx=0x7f861400cc20 comp rx=0 tx=0).stop 2024-02-21T23:03:46.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.453+0000 7f861b200640 1 -- 172.21.15.28:0/3892040815 shutdown_connections 2024-02-21T23:03:46.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.453+0000 7f861b200640 1 --2- 172.21.15.28:0/3892040815 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f85f806b500 0x7f85f806d9c0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:46.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.453+0000 7f861b200640 1 --2- 172.21.15.28:0/3892040815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f861c1451a0 0x7f861c1ede60 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:46.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.453+0000 7f861b200640 1 --2- 172.21.15.28:0/3892040815 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f861c1eab90 0x7f861c144c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:46.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.453+0000 7f861b200640 1 --2- 172.21.15.28:0/3892040815 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f86000039b0 0x7f861c14ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:46.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.453+0000 7f861b200640 1 -- 172.21.15.28:0/3892040815 >> 172.21.15.28:0/3892040815 conn(0x7f861c0a8530 msgr2=0x7f861c0c2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:46.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.453+0000 7f861b200640 1 -- 172.21.15.28:0/3892040815 shutdown_connections 2024-02-21T23:03:46.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.453+0000 7f861b200640 1 -- 172.21.15.28:0/3892040815 wait complete. 2024-02-21T23:03:46.460 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-02-21T23:03:46.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.689+0000 7f25d5000640 1 Processor -- start 2024-02-21T23:03:46.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.689+0000 7f25d5000640 1 -- start start 2024-02-21T23:03:46.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.690+0000 7f25d5000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f25d00c43e0 0x7f25d00a8e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:46.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.690+0000 7f25d5000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25d0149c10 con 0x7f25d0149830 2024-02-21T23:03:46.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.690+0000 7f25d5000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25d01537d0 con 0x7f25d006d410 2024-02-21T23:03:46.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.690+0000 7f25d5000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25d0153ab0 con 0x7f25d00c43e0 2024-02-21T23:03:46.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.690+0000 7f25cf400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f25d0149830 0x7f25d00c0ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:46982/0 (socket says 172.21.15.28:46982) 2024-02-21T23:03:46.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.691+0000 7f25cf400640 1 -- 172.21.15.28:0/4166786200 learned_addr learned my addr 172.21.15.28:0/4166786200 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:03:46.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.691+0000 7f25cfe00640 1 --2- 172.21.15.28:0/4166786200 >> v2:172.21.15.81:3300/0 conn(0x7f25d00c43e0 0x7f25d00a8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:46.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.691+0000 7f25ce000640 1 -- 172.21.15.28:0/4166786200 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 54404665 0 0) 0x7f25d0149c10 con 0x7f25d0149830 2024-02-21T23:03:46.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.691+0000 7f25ce000640 1 -- 172.21.15.28:0/4166786200 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25b4003520 con 0x7f25d0149830 2024-02-21T23:03:46.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.691+0000 7f25ce000640 1 -- 172.21.15.28:0/4166786200 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3267918484 0 0) 0x7f25b4003520 con 0x7f25d0149830 2024-02-21T23:03:46.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.691+0000 7f25ce000640 1 -- 172.21.15.28:0/4166786200 >> v2:172.21.15.81:3300/0 conn(0x7f25d00c43e0 msgr2=0x7f25d00a8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:46.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.691+0000 7f25ce000640 1 --2- 172.21.15.28:0/4166786200 >> v2:172.21.15.81:3300/0 conn(0x7f25d00c43e0 0x7f25d00a8e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:46.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.691+0000 7f25ce000640 1 -- 172.21.15.28:0/4166786200 >> v1:172.21.15.81:6789/0 conn(0x7f25d006d410 legacy=0x7f25d00c5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:46.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.691+0000 7f25ce000640 1 -- 172.21.15.28:0/4166786200 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25d00c4fa0 con 0x7f25d0149830 2024-02-21T23:03:46.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.691+0000 7f25ce000640 1 -- 172.21.15.28:0/4166786200 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f25b8001670 con 0x7f25d0149830 2024-02-21T23:03:46.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.691+0000 7f25ce000640 1 -- 172.21.15.28:0/4166786200 >> v1:172.21.15.28:6789/0 conn(0x7f25d0149830 legacy=0x7f25d00c0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:46.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.692+0000 7f25ce000640 1 --2- 172.21.15.28:0/4166786200 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f25b4003a00 0x7f25b4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:46.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.692+0000 7f25ce000640 1 --2- 172.21.15.28:0/4166786200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25b4004800 0x7f25b4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:46.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.692+0000 7f25ce000640 1 --2- 172.21.15.28:0/4166786200 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f25b40051c0 0x7f25b400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:46.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.692+0000 7f25ce000640 1 -- 172.21.15.28:0/4166786200 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25d0153ab0 con 0x7f25b4004800 2024-02-21T23:03:46.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.692+0000 7f25ce000640 1 -- 172.21.15.28:0/4166786200 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25b400c070 con 0x7f25b4003a00 2024-02-21T23:03:46.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.692+0000 7f25ce000640 1 -- 172.21.15.28:0/4166786200 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25b400c270 con 0x7f25b40051c0 2024-02-21T23:03:46.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.692+0000 7f25cea00640 1 --2- 172.21.15.28:0/4166786200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25b4004800 0x7f25b4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:46.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.692+0000 7f25cf400640 1 --2- 172.21.15.28:0/4166786200 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f25b4003a00 0x7f25b4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:46.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.692+0000 7f25cfe00640 1 --2- 172.21.15.28:0/4166786200 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f25b40051c0 0x7f25b400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:46.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.692+0000 7f25cea00640 1 -- 172.21.15.28:0/4166786200 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f25b40051c0 msgr2=0x7f25b400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:46.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.692+0000 7f25cea00640 1 --2- 172.21.15.28:0/4166786200 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f25b40051c0 0x7f25b400b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:46.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.692+0000 7f25cea00640 1 -- 172.21.15.28:0/4166786200 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f25b4003a00 msgr2=0x7f25b4003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:46.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.692+0000 7f25cea00640 1 --2- 172.21.15.28:0/4166786200 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f25b4003a00 0x7f25b4003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:46.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.692+0000 7f25cea00640 1 -- 172.21.15.28:0/4166786200 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25b4003520 con 0x7f25b4004800 2024-02-21T23:03:46.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.692+0000 7f25cfe00640 1 -- 172.21.15.28:0/4166786200 reap_dead start 2024-02-21T23:03:46.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.694+0000 7f25cea00640 1 --2- 172.21.15.28:0/4166786200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25b4004800 0x7f25b4004c80 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f25c00044b0 tx=0x7f25b4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5faf7293f05a4387 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:46.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.694+0000 7f25ce000640 1 -- 172.21.15.28:0/4166786200 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c0034070 con 0x7f25b4004800 2024-02-21T23:03:46.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.694+0000 7f25ce000640 1 -- 172.21.15.28:0/4166786200 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25c0030400 con 0x7f25b4004800 2024-02-21T23:03:46.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.694+0000 7f25ce000640 1 -- 172.21.15.28:0/4166786200 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c0004080 con 0x7f25b4004800 2024-02-21T23:03:46.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.694+0000 7f25d5000640 1 -- 172.21.15.28:0/4166786200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25b4004800 msgr2=0x7f25b4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:46.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.694+0000 7f25d5000640 1 --2- 172.21.15.28:0/4166786200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25b4004800 0x7f25b4004c80 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f25c00044b0 tx=0x7f25b4005830 comp rx=0 tx=0).stop 2024-02-21T23:03:46.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.694+0000 7f25d5000640 1 -- 172.21.15.28:0/4166786200 shutdown_connections 2024-02-21T23:03:46.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.694+0000 7f25d5000640 1 --2- 172.21.15.28:0/4166786200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25b4004800 0x7f25b4004c80 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:46.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.695+0000 7f25d5000640 1 -- 172.21.15.28:0/4166786200 >> 172.21.15.28:0/4166786200 conn(0x7f25d00a8320 msgr2=0x7f25d00a8730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:46.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.695+0000 7f25d5000640 1 -- 172.21.15.28:0/4166786200 shutdown_connections 2024-02-21T23:03:46.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.695+0000 7f25d5000640 1 -- 172.21.15.28:0/4166786200 wait complete. 2024-02-21T23:03:46.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.696+0000 7f25d5000640 1 Processor -- start 2024-02-21T23:03:46.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.696+0000 7f25d5000640 1 -- start start 2024-02-21T23:03:46.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.697+0000 7f25d5000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25b4004800 0x7f25d014abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:46.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.697+0000 7f25d5000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f25d0149c10 0x7f25d014dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:46.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.697+0000 7f25d5000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f25d0155fc0 0x7f25d01edbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:46.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.697+0000 7f25d5000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25d00a8e40 con 0x7f25b4004800 2024-02-21T23:03:46.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.697+0000 7f25d5000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25d00c5560 con 0x7f25d0155fc0 2024-02-21T23:03:46.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.697+0000 7f25d5000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25d00a8fc0 con 0x7f25d0149c10 2024-02-21T23:03:46.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.698+0000 7f25cf400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25b4004800 0x7f25d014abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:46.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.698+0000 7f25cf400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25b4004800 0x7f25d014abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41724/0 (socket says 172.21.15.28:41724) 2024-02-21T23:03:46.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.698+0000 7f25cf400640 1 -- 172.21.15.28:0/314384896 learned_addr learned my addr 172.21.15.28:0/314384896 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:46.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.698+0000 7f25cfe00640 1 --2- 172.21.15.28:0/314384896 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f25d0155fc0 0x7f25d01edbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:46.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.698+0000 7f25cea00640 1 --2- 172.21.15.28:0/314384896 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f25d0149c10 0x7f25d014dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:46.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.698+0000 7f25cf400640 1 -- 172.21.15.28:0/314384896 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f25d0149c10 msgr2=0x7f25d014dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:46.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.698+0000 7f25cf400640 1 --2- 172.21.15.28:0/314384896 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f25d0149c10 0x7f25d014dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:46.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.698+0000 7f25cf400640 1 -- 172.21.15.28:0/314384896 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f25d0155fc0 msgr2=0x7f25d01edbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:46.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.698+0000 7f25cf400640 1 --2- 172.21.15.28:0/314384896 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f25d0155fc0 0x7f25d01edbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:46.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.698+0000 7f25cf400640 1 -- 172.21.15.28:0/314384896 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25d0148600 con 0x7f25b4004800 2024-02-21T23:03:46.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.698+0000 7f25cf400640 1 --2- 172.21.15.28:0/314384896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25b4004800 0x7f25d014abe0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f25b800e700 tx=0x7f25b800ebc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:46.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.699+0000 7f25c7400640 1 -- 172.21.15.28:0/314384896 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b800c5e0 con 0x7f25b4004800 2024-02-21T23:03:46.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.699+0000 7f25c7400640 1 -- 172.21.15.28:0/314384896 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25b8018070 con 0x7f25b4004800 2024-02-21T23:03:46.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.699+0000 7f25c7400640 1 -- 172.21.15.28:0/314384896 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b800c9c0 con 0x7f25b4004800 2024-02-21T23:03:46.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.699+0000 7f25d5000640 1 -- 172.21.15.28:0/314384896 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25d01ee5a0 con 0x7f25b4004800 2024-02-21T23:03:46.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.699+0000 7f25d5000640 1 -- 172.21.15.28:0/314384896 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25d0148ac0 con 0x7f25b4004800 2024-02-21T23:03:46.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.700+0000 7f25d5000640 1 -- 172.21.15.28:0/314384896 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25d00c73f0 con 0x7f25b4004800 2024-02-21T23:03:46.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.701+0000 7f25c7400640 1 -- 172.21.15.28:0/314384896 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f25b800cb60 con 0x7f25b4004800 2024-02-21T23:03:46.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.702+0000 7f25c7400640 1 --2- 172.21.15.28:0/314384896 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f25ac06b500 0x7f25ac06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:46.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.702+0000 7f25c7400640 1 -- 172.21.15.28:0/314384896 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f25b808a050 con 0x7f25b4004800 2024-02-21T23:03:46.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.703+0000 7f25cea00640 1 --2- 172.21.15.28:0/314384896 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f25ac06b500 0x7f25ac06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:46.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.704+0000 7f25cea00640 1 --2- 172.21.15.28:0/314384896 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f25ac06b500 0x7f25ac06d9c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f25c0036000 tx=0x7f25c0009030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:46.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.706+0000 7f25c7400640 1 -- 172.21.15.28:0/314384896 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f25d00c73f0 con 0x7f25b4004800 2024-02-21T23:03:46.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.921+0000 7f25d5000640 1 -- 172.21.15.28:0/314384896 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f25d004f290 con 0x7f25b4004800 2024-02-21T23:03:46.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.922+0000 7f25c7400640 1 -- 172.21.15.28:0/314384896 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v26) v1 ==== 80+0+0 (secure 0 0 0) 0x7f25b8022250 con 0x7f25b4004800 2024-02-21T23:03:46.911 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-02-21T23:03:46.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.925+0000 7f25d5000640 1 -- 172.21.15.28:0/314384896 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f25ac06b500 msgr2=0x7f25ac06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:46.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.925+0000 7f25d5000640 1 --2- 172.21.15.28:0/314384896 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f25ac06b500 0x7f25ac06d9c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f25c0036000 tx=0x7f25c0009030 comp rx=0 tx=0).stop 2024-02-21T23:03:46.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.925+0000 7f25d5000640 1 -- 172.21.15.28:0/314384896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25b4004800 msgr2=0x7f25d014abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:46.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.925+0000 7f25d5000640 1 --2- 172.21.15.28:0/314384896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25b4004800 0x7f25d014abe0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f25b800e700 tx=0x7f25b800ebc0 comp rx=0 tx=0).stop 2024-02-21T23:03:46.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.925+0000 7f25d5000640 1 -- 172.21.15.28:0/314384896 shutdown_connections 2024-02-21T23:03:46.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.925+0000 7f25d5000640 1 --2- 172.21.15.28:0/314384896 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f25ac06b500 0x7f25ac06d9c0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:46.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.925+0000 7f25d5000640 1 --2- 172.21.15.28:0/314384896 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f25d0149c10 0x7f25d014dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:46.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.925+0000 7f25d5000640 1 --2- 172.21.15.28:0/314384896 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f25d0155fc0 0x7f25d01edbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:46.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.925+0000 7f25d5000640 1 --2- 172.21.15.28:0/314384896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25b4004800 0x7f25d014abe0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:46.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.925+0000 7f25d5000640 1 -- 172.21.15.28:0/314384896 >> 172.21.15.28:0/314384896 conn(0x7f25d00a8320 msgr2=0x7f25d006d860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:46.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.925+0000 7f25d5000640 1 -- 172.21.15.28:0/314384896 shutdown_connections 2024-02-21T23:03:46.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:46.925+0000 7f25d5000640 1 -- 172.21.15.28:0/314384896 wait complete. 2024-02-21T23:03:46.934 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-02-21T23:03:46.934 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:03:46.934 DEBUG:teuthology.orchestra.run.smithi028:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2024-02-21T23:03:47.037 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi028.front.sepia.ceph.com 2024-02-21T23:03:47.037 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:03:47.037 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-02-21T23:03:47.092 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-02-21T23:03:47.092 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-21T23:03:47.144 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-02-21T23:03:47.144 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-21T23:03:47.191 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-02-21T23:03:47.192 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-21T23:03:47.278 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:03:47.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.520+0000 7f750be00640 1 Processor -- start 2024-02-21T23:03:47.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.520+0000 7f750be00640 1 -- start start 2024-02-21T23:03:47.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.522+0000 7f750be00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f750c150450 0x7f750c0a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:47.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.522+0000 7f750be00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f750c0c2760 con 0x7f750c06ee40 2024-02-21T23:03:47.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.522+0000 7f750be00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f750c156310 con 0x7f750c156af0 2024-02-21T23:03:47.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.522+0000 7f750be00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f750c0c22c0 con 0x7f750c150450 2024-02-21T23:03:47.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.522+0000 7f7502000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f750c06ee40 0x7f750c1536f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:46986/0 (socket says 172.21.15.28:46986) 2024-02-21T23:03:47.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.522+0000 7f7502000640 1 -- 172.21.15.28:0/2920499096 learned_addr learned my addr 172.21.15.28:0/2920499096 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:03:47.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.522+0000 7f750aa00640 1 --2- 172.21.15.28:0/2920499096 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f750c150450 0x7f750c0a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:47.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.523+0000 7f750a000640 1 -- 172.21.15.28:0/2920499096 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4178549539 0 0) 0x7f750c0c2760 con 0x7f750c06ee40 2024-02-21T23:03:47.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.523+0000 7f750a000640 1 -- 172.21.15.28:0/2920499096 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74f0003520 con 0x7f750c06ee40 2024-02-21T23:03:47.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.523+0000 7f750a000640 1 -- 172.21.15.28:0/2920499096 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 286749143 0 0) 0x7f74f0003520 con 0x7f750c06ee40 2024-02-21T23:03:47.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.523+0000 7f750a000640 1 -- 172.21.15.28:0/2920499096 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f750c150450 msgr2=0x7f750c0a5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:47.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.523+0000 7f750a000640 1 --2- 172.21.15.28:0/2920499096 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f750c150450 0x7f750c0a5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:47.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.523+0000 7f750a000640 1 -- 172.21.15.28:0/2920499096 >> v1:172.21.15.81:6789/0 conn(0x7f750c156af0 legacy=0x7f750c06dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:47.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.523+0000 7f750a000640 1 -- 172.21.15.28:0/2920499096 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f750c0c29d0 con 0x7f750c06ee40 2024-02-21T23:03:47.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.523+0000 7f750a000640 1 -- 172.21.15.28:0/2920499096 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f74f4001670 con 0x7f750c06ee40 2024-02-21T23:03:47.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.523+0000 7f750a000640 1 -- 172.21.15.28:0/2920499096 >> v1:172.21.15.28:6789/0 conn(0x7f750c06ee40 legacy=0x7f750c1536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:47.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.523+0000 7f750a000640 1 --2- 172.21.15.28:0/2920499096 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74f00039e0 0x7f74f0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:47.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.523+0000 7f750a000640 1 --2- 172.21.15.28:0/2920499096 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f750c150450 0x7f74f0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:47.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.523+0000 7f750a000640 1 --2- 172.21.15.28:0/2920499096 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74f0004d40 0x7f74f00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:47.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.523+0000 7f750a000640 1 -- 172.21.15.28:0/2920499096 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74f000b9c0 con 0x7f74f0004d40 2024-02-21T23:03:47.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.523+0000 7f750a000640 1 -- 172.21.15.28:0/2920499096 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74f000bb40 con 0x7f74f00039e0 2024-02-21T23:03:47.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.523+0000 7f750a000640 1 -- 172.21.15.28:0/2920499096 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74f000bd40 con 0x7f750c150450 2024-02-21T23:03:47.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.524+0000 7f750b400640 1 --2- 172.21.15.28:0/2920499096 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74f0004d40 0x7f74f00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:47.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.524+0000 7f750aa00640 1 --2- 172.21.15.28:0/2920499096 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74f00039e0 0x7f74f0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:47.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.524+0000 7f7502000640 1 --2- 172.21.15.28:0/2920499096 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f750c150450 0x7f74f0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:47.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.524+0000 7f750b400640 1 -- 172.21.15.28:0/2920499096 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f750c150450 msgr2=0x7f74f0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:47.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.524+0000 7f750b400640 1 --2- 172.21.15.28:0/2920499096 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f750c150450 0x7f74f0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:47.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.524+0000 7f750b400640 1 -- 172.21.15.28:0/2920499096 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74f00039e0 msgr2=0x7f74f0003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:47.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.524+0000 7f750b400640 1 --2- 172.21.15.28:0/2920499096 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74f00039e0 0x7f74f0003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:47.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.524+0000 7f750b400640 1 -- 172.21.15.28:0/2920499096 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74f0003520 con 0x7f74f0004d40 2024-02-21T23:03:47.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.525+0000 7f750b400640 1 --2- 172.21.15.28:0/2920499096 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74f0004d40 0x7f74f00051c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f7504001f60 tx=0x7f74f0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d0aaf553cd1483 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:47.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.526+0000 7f750a000640 1 -- 172.21.15.28:0/2920499096 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7504001b30 con 0x7f74f0004d40 2024-02-21T23:03:47.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.526+0000 7f750a000640 1 -- 172.21.15.28:0/2920499096 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f750402d710 con 0x7f74f0004d40 2024-02-21T23:03:47.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.526+0000 7f750a000640 1 -- 172.21.15.28:0/2920499096 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f750402db00 con 0x7f74f0004d40 2024-02-21T23:03:47.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.526+0000 7f750be00640 1 -- 172.21.15.28:0/2920499096 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74f0004d40 msgr2=0x7f74f00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:47.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.526+0000 7f750be00640 1 --2- 172.21.15.28:0/2920499096 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74f0004d40 0x7f74f00051c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f7504001f60 tx=0x7f74f0005830 comp rx=0 tx=0).stop 2024-02-21T23:03:47.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.526+0000 7f750b400640 1 -- 172.21.15.28:0/2920499096 reap_dead start 2024-02-21T23:03:47.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.526+0000 7f750be00640 1 -- 172.21.15.28:0/2920499096 shutdown_connections 2024-02-21T23:03:47.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.526+0000 7f750be00640 1 -- 172.21.15.28:0/2920499096 >> 172.21.15.28:0/2920499096 conn(0x7f750c06b630 msgr2=0x7f750c0c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:47.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.526+0000 7f750be00640 1 -- 172.21.15.28:0/2920499096 shutdown_connections 2024-02-21T23:03:47.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.526+0000 7f750be00640 1 -- 172.21.15.28:0/2920499096 wait complete. 2024-02-21T23:03:47.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.528+0000 7f750be00640 1 Processor -- start 2024-02-21T23:03:47.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.528+0000 7f750be00640 1 -- start start 2024-02-21T23:03:47.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.529+0000 7f750be00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74f0004d40 0x7f750c0ac190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:47.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.529+0000 7f750be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f750c14d0d0 0x7f750c150b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:47.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.529+0000 7f750be00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f750c1529e0 0x7f750c152e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:47.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.529+0000 7f750be00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f750c06e700 con 0x7f750c14d0d0 2024-02-21T23:03:47.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.529+0000 7f750be00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f750c156660 con 0x7f74f0004d40 2024-02-21T23:03:47.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.529+0000 7f750be00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f750c06e880 con 0x7f750c1529e0 2024-02-21T23:03:47.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.530+0000 7f7502000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f750c14d0d0 0x7f750c150b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:47.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.530+0000 7f7502000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f750c14d0d0 0x7f750c150b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41740/0 (socket says 172.21.15.28:41740) 2024-02-21T23:03:47.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.530+0000 7f7502000640 1 -- 172.21.15.28:0/2917443941 learned_addr learned my addr 172.21.15.28:0/2917443941 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:47.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.530+0000 7f750aa00640 1 --2- 172.21.15.28:0/2917443941 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74f0004d40 0x7f750c0ac190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:47.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.530+0000 7f750b400640 1 --2- 172.21.15.28:0/2917443941 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f750c1529e0 0x7f750c152e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:47.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.530+0000 7f7502000640 1 -- 172.21.15.28:0/2917443941 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f750c1529e0 msgr2=0x7f750c152e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:47.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.530+0000 7f7502000640 1 --2- 172.21.15.28:0/2917443941 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f750c1529e0 0x7f750c152e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:47.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.530+0000 7f7502000640 1 -- 172.21.15.28:0/2917443941 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74f0004d40 msgr2=0x7f750c0ac190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:47.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.530+0000 7f7502000640 1 --2- 172.21.15.28:0/2917443941 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74f0004d40 0x7f750c0ac190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:47.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.530+0000 7f7502000640 1 -- 172.21.15.28:0/2917443941 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f750c14b2a0 con 0x7f750c14d0d0 2024-02-21T23:03:47.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.530+0000 7f7502000640 1 --2- 172.21.15.28:0/2917443941 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f750c14d0d0 0x7f750c150b30 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f74f400e840 tx=0x7f74f400ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:47.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.531+0000 7f7503400640 1 -- 172.21.15.28:0/2917443941 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74f400ee90 con 0x7f750c14d0d0 2024-02-21T23:03:47.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.531+0000 7f7503400640 1 -- 172.21.15.28:0/2917443941 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74f4018070 con 0x7f750c14d0d0 2024-02-21T23:03:47.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.531+0000 7f7503400640 1 -- 172.21.15.28:0/2917443941 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74f400ca90 con 0x7f750c14d0d0 2024-02-21T23:03:47.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.531+0000 7f750be00640 1 -- 172.21.15.28:0/2917443941 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f750c14b580 con 0x7f750c14d0d0 2024-02-21T23:03:47.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.531+0000 7f750be00640 1 -- 172.21.15.28:0/2917443941 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f750c14bb40 con 0x7f750c14d0d0 2024-02-21T23:03:47.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.532+0000 7f750be00640 1 -- 172.21.15.28:0/2917443941 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f750c04f290 con 0x7f750c14d0d0 2024-02-21T23:03:47.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.533+0000 7f7503400640 1 -- 172.21.15.28:0/2917443941 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f74f400cc30 con 0x7f750c14d0d0 2024-02-21T23:03:47.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.534+0000 7f7503400640 1 --2- 172.21.15.28:0/2917443941 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f74e806b670 0x7f74e806db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:47.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.534+0000 7f7503400640 1 -- 172.21.15.28:0/2917443941 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f74f408a460 con 0x7f750c14d0d0 2024-02-21T23:03:47.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.534+0000 7f750aa00640 1 --2- 172.21.15.28:0/2917443941 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f74e806b670 0x7f74e806db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:47.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.535+0000 7f750aa00640 1 --2- 172.21.15.28:0/2917443941 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f74e806b670 0x7f74e806db30 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f74fc009bc0 tx=0x7f74fc009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:47.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.539+0000 7f7503400640 1 -- 172.21.15.28:0/2917443941 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f74f4058ed0 con 0x7f750c14d0d0 2024-02-21T23:03:47.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.739+0000 7f750be00640 1 -- 172.21.15.28:0/2917443941 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f750c0c1f10 con 0x7f750c14d0d0 2024-02-21T23:03:47.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.742+0000 7f7503400640 1 -- 172.21.15.28:0/2917443941 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+18843 (secure 0 0 0) 0x7f74f4001630 con 0x7f750c14d0d0 2024-02-21T23:03:47.733 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:03:47.733 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":26,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:03:23.860109+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:02:35.795158+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-21T23:02:36.840110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-02-21T23:02:40.441388+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":3,"to":2}]},{"pgid":"3.12","mappings":[{"from":0,"to":1}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:03:47.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.745+0000 7f750be00640 1 -- 172.21.15.28:0/2917443941 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f74e806b670 msgr2=0x7f74e806db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:47.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.745+0000 7f750be00640 1 --2- 172.21.15.28:0/2917443941 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f74e806b670 0x7f74e806db30 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f74fc009bc0 tx=0x7f74fc009190 comp rx=0 tx=0).stop 2024-02-21T23:03:47.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.745+0000 7f750be00640 1 -- 172.21.15.28:0/2917443941 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f750c14d0d0 msgr2=0x7f750c150b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:47.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.745+0000 7f750be00640 1 --2- 172.21.15.28:0/2917443941 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f750c14d0d0 0x7f750c150b30 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f74f400e840 tx=0x7f74f400ec40 comp rx=0 tx=0).stop 2024-02-21T23:03:47.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.746+0000 7f750be00640 1 -- 172.21.15.28:0/2917443941 shutdown_connections 2024-02-21T23:03:47.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.746+0000 7f750be00640 1 --2- 172.21.15.28:0/2917443941 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f74e806b670 0x7f74e806db30 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:47.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.746+0000 7f750be00640 1 --2- 172.21.15.28:0/2917443941 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f750c1529e0 0x7f750c152e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:47.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.746+0000 7f750be00640 1 --2- 172.21.15.28:0/2917443941 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f750c14d0d0 0x7f750c150b30 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:47.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.746+0000 7f750be00640 1 --2- 172.21.15.28:0/2917443941 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74f0004d40 0x7f750c0ac190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:47.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.746+0000 7f750be00640 1 -- 172.21.15.28:0/2917443941 >> 172.21.15.28:0/2917443941 conn(0x7f750c06b630 msgr2=0x7f750c071070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:47.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.746+0000 7f750be00640 1 -- 172.21.15.28:0/2917443941 shutdown_connections 2024-02-21T23:03:47.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.746+0000 7f750be00640 1 -- 172.21.15.28:0/2917443941 wait complete. 2024-02-21T23:03:47.755 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-21T23:02:30.728329+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-21T23:02:35.795158+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-02-21T23:02:36.840110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-02-21T23:02:40.441388+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-02-21T23:03:47.756 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-21T23:03:47.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.990+0000 7fc4d3200640 1 Processor -- start 2024-02-21T23:03:47.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.990+0000 7fc4d3200640 1 -- start start 2024-02-21T23:03:47.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.991+0000 7fc4d3200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc4d413e0c0 0x7fc4d40c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:47.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.991+0000 7fc4d3200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc4d406dbc0 0x7fc4d406c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:47.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.992+0000 7fc4d3200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4d4158550 con 0x7fc4d40c76f0 2024-02-21T23:03:47.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.992+0000 7fc4d3200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4d40c89a0 con 0x7fc4d413e0c0 2024-02-21T23:03:47.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.992+0000 7fc4d3200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d41587d0 con 0x7fc4d406dbc0 2024-02-21T23:03:47.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.992+0000 7fc4d1e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc4d413e0c0 0x7fc4d40c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:47.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.992+0000 7fc4d1e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc4d413e0c0 0x7fc4d40c71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41750/0 (socket says 172.21.15.28:41750) 2024-02-21T23:03:47.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.992+0000 7fc4d1e00640 1 -- 172.21.15.28:0/129632512 learned_addr learned my addr 172.21.15.28:0/129632512 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:47.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.992+0000 7fc4d1400640 1 --2- 172.21.15.28:0/129632512 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc4d406dbc0 0x7fc4d406c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:47.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.992+0000 7fc4d1e00640 1 -- 172.21.15.28:0/129632512 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc4d406dbc0 msgr2=0x7fc4d406c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:47.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.992+0000 7fc4d1e00640 1 --2- 172.21.15.28:0/129632512 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc4d406dbc0 0x7fc4d406c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:47.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.992+0000 7fc4d1e00640 1 -- 172.21.15.28:0/129632512 >> v1:172.21.15.81:6789/0 conn(0x7fc4d40c76f0 legacy=0x7fc4d406c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:47.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.993+0000 7fc4d1e00640 1 -- 172.21.15.28:0/129632512 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4d4158ab0 con 0x7fc4d413e0c0 2024-02-21T23:03:47.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.994+0000 7fc4d1e00640 1 --2- 172.21.15.28:0/129632512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4d413e0c0 0x7fc4d40c71b0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fc4c4009870 tx=0x7fc4c402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d46b33d0c28457cb server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:47.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.994+0000 7fc4d0a00640 1 -- 172.21.15.28:0/129632512 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c4035020 con 0x7fc4d413e0c0 2024-02-21T23:03:47.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.994+0000 7fc4d0a00640 1 -- 172.21.15.28:0/129632512 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4c4031640 con 0x7fc4d413e0c0 2024-02-21T23:03:47.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.994+0000 7fc4d0a00640 1 -- 172.21.15.28:0/129632512 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c402d040 con 0x7fc4d413e0c0 2024-02-21T23:03:47.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.995+0000 7fc4d3200640 1 -- 172.21.15.28:0/129632512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4d413e0c0 msgr2=0x7fc4d40c71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:47.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.995+0000 7fc4d3200640 1 --2- 172.21.15.28:0/129632512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4d413e0c0 0x7fc4d40c71b0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fc4c4009870 tx=0x7fc4c402b8f0 comp rx=0 tx=0).stop 2024-02-21T23:03:47.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.995+0000 7fc4d3200640 1 -- 172.21.15.28:0/129632512 shutdown_connections 2024-02-21T23:03:47.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.995+0000 7fc4d3200640 1 --2- 172.21.15.28:0/129632512 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc4d406dbc0 0x7fc4d406c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:47.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.995+0000 7fc4d3200640 1 --2- 172.21.15.28:0/129632512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4d413e0c0 0x7fc4d40c71b0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:47.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.995+0000 7fc4d3200640 1 -- 172.21.15.28:0/129632512 >> 172.21.15.28:0/129632512 conn(0x7fc4d4073110 msgr2=0x7fc4d40ac490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:47.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.995+0000 7fc4d3200640 1 -- 172.21.15.28:0/129632512 shutdown_connections 2024-02-21T23:03:47.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.995+0000 7fc4d3200640 1 -- 172.21.15.28:0/129632512 wait complete. 2024-02-21T23:03:47.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.997+0000 7fc4d3200640 1 Processor -- start 2024-02-21T23:03:47.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.997+0000 7fc4d3200640 1 -- start start 2024-02-21T23:03:47.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.998+0000 7fc4d3200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4d406dbc0 0x7fc4d4151730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:47.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.998+0000 7fc4d3200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc4d40c76f0 0x7fc4d40af830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:47.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.998+0000 7fc4d3200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc4d413e0c0 0x7fc4d40afd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:47.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.998+0000 7fc4d3200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d406ced0 con 0x7fc4d406dbc0 2024-02-21T23:03:47.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.998+0000 7fc4d3200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d4159070 con 0x7fc4d40c76f0 2024-02-21T23:03:47.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.998+0000 7fc4d3200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d413e520 con 0x7fc4d413e0c0 2024-02-21T23:03:47.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.998+0000 7fc4d1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4d406dbc0 0x7fc4d4151730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:47.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.998+0000 7fc4d1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4d406dbc0 0x7fc4d4151730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41762/0 (socket says 172.21.15.28:41762) 2024-02-21T23:03:47.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.998+0000 7fc4d1e00640 1 -- 172.21.15.28:0/3093790128 learned_addr learned my addr 172.21.15.28:0/3093790128 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:47.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.998+0000 7fc4d1e00640 1 -- 172.21.15.28:0/3093790128 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc4d413e0c0 msgr2=0x7fc4d40afd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:47.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.998+0000 7fc4d1e00640 1 --2- 172.21.15.28:0/3093790128 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc4d413e0c0 0x7fc4d40afd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:47.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.998+0000 7fc4d1e00640 1 -- 172.21.15.28:0/3093790128 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc4d40c76f0 msgr2=0x7fc4d40af830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:47.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.998+0000 7fc4d1e00640 1 --2- 172.21.15.28:0/3093790128 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc4d40c76f0 0x7fc4d40af830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:47.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.998+0000 7fc4d1e00640 1 -- 172.21.15.28:0/3093790128 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4d40ac8a0 con 0x7fc4d406dbc0 2024-02-21T23:03:47.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.999+0000 7fc4d1e00640 1 --2- 172.21.15.28:0/3093790128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4d406dbc0 0x7fc4d4151730 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fc4c402b930 tx=0x7fc4c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:47.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.999+0000 7fc4ca000640 1 -- 172.21.15.28:0/3093790128 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c4040070 con 0x7fc4d406dbc0 2024-02-21T23:03:47.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.999+0000 7fc4ca000640 1 -- 172.21.15.28:0/3093790128 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4c403e070 con 0x7fc4d406dbc0 2024-02-21T23:03:47.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.999+0000 7fc4ca000640 1 -- 172.21.15.28:0/3093790128 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c4035040 con 0x7fc4d406dbc0 2024-02-21T23:03:47.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.999+0000 7fc4d3200640 1 -- 172.21.15.28:0/3093790128 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4d40b02b0 con 0x7fc4d406dbc0 2024-02-21T23:03:47.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:47.999+0000 7fc4d3200640 1 -- 172.21.15.28:0/3093790128 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4d40ae640 con 0x7fc4d406dbc0 2024-02-21T23:03:47.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.000+0000 7fc4d3200640 1 -- 172.21.15.28:0/3093790128 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4d404f290 con 0x7fc4d406dbc0 2024-02-21T23:03:47.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.001+0000 7fc4ca000640 1 -- 172.21.15.28:0/3093790128 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc4c4038020 con 0x7fc4d406dbc0 2024-02-21T23:03:47.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.002+0000 7fc4ca000640 1 --2- 172.21.15.28:0/3093790128 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc4b006b500 0x7fc4b006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:47.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.002+0000 7fc4ca000640 1 -- 172.21.15.28:0/3093790128 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6735+0+0 (secure 0 0 0) 0x7fc4c40abff0 con 0x7fc4d406dbc0 2024-02-21T23:03:47.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.003+0000 7fc4d1400640 1 --2- 172.21.15.28:0/3093790128 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc4b006b500 0x7fc4b006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:47.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.004+0000 7fc4d1400640 1 --2- 172.21.15.28:0/3093790128 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc4b006b500 0x7fc4b006d9c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc4d40ac3e0 tx=0x7fc4bc008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:47.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.007+0000 7fc4ca000640 1 -- 172.21.15.28:0/3093790128 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc4d40ac8a0 con 0x7fc4d406dbc0 2024-02-21T23:03:48.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.206+0000 7fc4d3200640 1 -- 172.21.15.28:0/3093790128 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc4d40c85a0 con 0x7fc4d406dbc0 2024-02-21T23:03:48.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.207+0000 7fc4ca000640 1 -- 172.21.15.28:0/3093790128 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc4c407aa60 con 0x7fc4d406dbc0 2024-02-21T23:03:48.197 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-02-21T23:03:48.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.210+0000 7fc4d3200640 1 -- 172.21.15.28:0/3093790128 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc4b006b500 msgr2=0x7fc4b006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:48.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.210+0000 7fc4d3200640 1 --2- 172.21.15.28:0/3093790128 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc4b006b500 0x7fc4b006d9c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc4d40ac3e0 tx=0x7fc4bc008040 comp rx=0 tx=0).stop 2024-02-21T23:03:48.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.210+0000 7fc4d3200640 1 -- 172.21.15.28:0/3093790128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4d406dbc0 msgr2=0x7fc4d4151730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:48.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.210+0000 7fc4d3200640 1 --2- 172.21.15.28:0/3093790128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4d406dbc0 0x7fc4d4151730 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fc4c402b930 tx=0x7fc4c40046c0 comp rx=0 tx=0).stop 2024-02-21T23:03:48.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.210+0000 7fc4d3200640 1 -- 172.21.15.28:0/3093790128 shutdown_connections 2024-02-21T23:03:48.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.210+0000 7fc4d3200640 1 --2- 172.21.15.28:0/3093790128 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc4b006b500 0x7fc4b006d9c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:48.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.210+0000 7fc4d3200640 1 --2- 172.21.15.28:0/3093790128 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc4d413e0c0 0x7fc4d40afd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:48.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.210+0000 7fc4d3200640 1 --2- 172.21.15.28:0/3093790128 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc4d40c76f0 0x7fc4d40af830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:48.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.210+0000 7fc4d3200640 1 --2- 172.21.15.28:0/3093790128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4d406dbc0 0x7fc4d4151730 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:48.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.210+0000 7fc4d3200640 1 -- 172.21.15.28:0/3093790128 >> 172.21.15.28:0/3093790128 conn(0x7fc4d4073110 msgr2=0x7fc4d413cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:48.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.210+0000 7fc4d3200640 1 -- 172.21.15.28:0/3093790128 shutdown_connections 2024-02-21T23:03:48.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.210+0000 7fc4d3200640 1 -- 172.21.15.28:0/3093790128 wait complete. 2024-02-21T23:03:48.221 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-21T23:03:48.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.443+0000 7fe42f200640 1 Processor -- start 2024-02-21T23:03:48.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.443+0000 7fe42f200640 1 -- start start 2024-02-21T23:03:48.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.445+0000 7fe42f200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fe4280c4030 0x7fe4280c5d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:48.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.445+0000 7fe42f200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe42806d3b0 0x7fe42815fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:48.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.445+0000 7fe42f200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4281594d0 con 0x7fe4280c4d30 2024-02-21T23:03:48.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.445+0000 7fe42f200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe428155500 con 0x7fe4280c4030 2024-02-21T23:03:48.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.445+0000 7fe42f200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4281557e0 con 0x7fe42806d3b0 2024-02-21T23:03:48.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.446+0000 7fe427e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fe4280c4030 0x7fe4280c5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:48.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.446+0000 7fe427e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fe4280c4030 0x7fe4280c5d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.28:54184/0 (socket says 172.21.15.28:54184) 2024-02-21T23:03:48.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.446+0000 7fe427e00640 1 -- 172.21.15.28:0/51970829 learned_addr learned my addr 172.21.15.28:0/51970829 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:48.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.446+0000 7fe42d200640 1 --2- 172.21.15.28:0/51970829 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe42806d3b0 0x7fe42815fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:48.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.447+0000 7fe427e00640 1 -- 172.21.15.28:0/51970829 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe42806d3b0 msgr2=0x7fe42815fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:48.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.447+0000 7fe427e00640 1 --2- 172.21.15.28:0/51970829 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe42806d3b0 0x7fe42815fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:48.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.447+0000 7fe427e00640 1 -- 172.21.15.28:0/51970829 >> v1:172.21.15.81:6789/0 conn(0x7fe4280c4d30 legacy=0x7fe42806cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:48.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.447+0000 7fe427e00640 1 -- 172.21.15.28:0/51970829 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe428155ac0 con 0x7fe4280c4030 2024-02-21T23:03:48.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.448+0000 7fe427e00640 1 --2- 172.21.15.28:0/51970829 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe4280c4030 0x7fe4280c5d60 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe418009870 tx=0x7fe41802b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ed785e813d3c488 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:48.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.449+0000 7fe426a00640 1 -- 172.21.15.28:0/51970829 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4180317c0 con 0x7fe4280c4030 2024-02-21T23:03:48.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.449+0000 7fe426a00640 1 -- 172.21.15.28:0/51970829 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe418037070 con 0x7fe4280c4030 2024-02-21T23:03:48.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.449+0000 7fe426a00640 1 -- 172.21.15.28:0/51970829 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe418031b80 con 0x7fe4280c4030 2024-02-21T23:03:48.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.450+0000 7fe42f200640 1 -- 172.21.15.28:0/51970829 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe4280c4030 msgr2=0x7fe4280c5d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:48.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.450+0000 7fe42f200640 1 --2- 172.21.15.28:0/51970829 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe4280c4030 0x7fe4280c5d60 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe418009870 tx=0x7fe41802b860 comp rx=0 tx=0).stop 2024-02-21T23:03:48.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.450+0000 7fe42f200640 1 -- 172.21.15.28:0/51970829 shutdown_connections 2024-02-21T23:03:48.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.450+0000 7fe42f200640 1 --2- 172.21.15.28:0/51970829 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe42806d3b0 0x7fe42815fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:48.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.450+0000 7fe42f200640 1 --2- 172.21.15.28:0/51970829 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe4280c4030 0x7fe4280c5d60 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:48.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.450+0000 7fe42f200640 1 -- 172.21.15.28:0/51970829 >> 172.21.15.28:0/51970829 conn(0x7fe4280725a0 msgr2=0x7fe4280729b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:48.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.450+0000 7fe42f200640 1 -- 172.21.15.28:0/51970829 shutdown_connections 2024-02-21T23:03:48.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.450+0000 7fe42f200640 1 -- 172.21.15.28:0/51970829 wait complete. 2024-02-21T23:03:48.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.452+0000 7fe42f200640 1 Processor -- start 2024-02-21T23:03:48.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.452+0000 7fe42f200640 1 -- start start 2024-02-21T23:03:48.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.454+0000 7fe42f200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe42806d3b0 0x7fe4280aec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:48.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.454+0000 7fe42f200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4280c4030 0x7fe4280af1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:48.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.454+0000 7fe42f200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe4280c4d30 0x7fe4281deb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:48.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.454+0000 7fe42f200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4281592b0 con 0x7fe4280c4030 2024-02-21T23:03:48.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.454+0000 7fe42f200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe428159130 con 0x7fe42806d3b0 2024-02-21T23:03:48.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.454+0000 7fe42f200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe428155fe0 con 0x7fe4280c4d30 2024-02-21T23:03:48.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.454+0000 7fe427400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4280c4030 0x7fe4280af1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:48.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.454+0000 7fe427400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4280c4030 0x7fe4280af1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41770/0 (socket says 172.21.15.28:41770) 2024-02-21T23:03:48.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.454+0000 7fe427400640 1 -- 172.21.15.28:0/982808878 learned_addr learned my addr 172.21.15.28:0/982808878 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:48.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.454+0000 7fe427e00640 1 --2- 172.21.15.28:0/982808878 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe42806d3b0 0x7fe4280aec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:48.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.454+0000 7fe42d200640 1 --2- 172.21.15.28:0/982808878 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe4280c4d30 0x7fe4281deb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:48.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.455+0000 7fe427400640 1 -- 172.21.15.28:0/982808878 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe4280c4d30 msgr2=0x7fe4281deb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:48.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.455+0000 7fe427400640 1 --2- 172.21.15.28:0/982808878 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe4280c4d30 0x7fe4281deb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:48.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.455+0000 7fe427400640 1 -- 172.21.15.28:0/982808878 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe42806d3b0 msgr2=0x7fe4280aec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:48.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.455+0000 7fe427400640 1 --2- 172.21.15.28:0/982808878 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe42806d3b0 0x7fe4280aec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:48.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.455+0000 7fe427400640 1 -- 172.21.15.28:0/982808878 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe42814fb60 con 0x7fe4280c4030 2024-02-21T23:03:48.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.455+0000 7fe427400640 1 --2- 172.21.15.28:0/982808878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4280c4030 0x7fe4280af1a0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fe41000c440 tx=0x7fe41000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:48.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.455+0000 7fe41fe00640 1 -- 172.21.15.28:0/982808878 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4100052d0 con 0x7fe4280c4030 2024-02-21T23:03:48.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.455+0000 7fe41fe00640 1 -- 172.21.15.28:0/982808878 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe410005470 con 0x7fe4280c4030 2024-02-21T23:03:48.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.455+0000 7fe41fe00640 1 -- 172.21.15.28:0/982808878 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe410007260 con 0x7fe4280c4030 2024-02-21T23:03:48.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.455+0000 7fe42f200640 1 -- 172.21.15.28:0/982808878 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe42814f880 con 0x7fe4280c4030 2024-02-21T23:03:48.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.456+0000 7fe42f200640 1 -- 172.21.15.28:0/982808878 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4281eba10 con 0x7fe4280c4030 2024-02-21T23:03:48.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.457+0000 7fe42f200640 1 -- 172.21.15.28:0/982808878 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe428067f30 con 0x7fe4280c4030 2024-02-21T23:03:48.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.458+0000 7fe41fe00640 1 -- 172.21.15.28:0/982808878 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe410007400 con 0x7fe4280c4030 2024-02-21T23:03:48.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.460+0000 7fe41fe00640 1 --2- 172.21.15.28:0/982808878 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe40406b500 0x7fe40406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:48.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.460+0000 7fe41fe00640 1 -- 172.21.15.28:0/982808878 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6735+0+0 (secure 0 0 0) 0x7fe410089960 con 0x7fe4280c4030 2024-02-21T23:03:48.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.460+0000 7fe427e00640 1 --2- 172.21.15.28:0/982808878 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe40406b500 0x7fe40406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:48.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.462+0000 7fe427e00640 1 --2- 172.21.15.28:0/982808878 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe40406b500 0x7fe40406d9c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fe418036000 tx=0x7fe41803d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:48.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.464+0000 7fe41fe00640 1 -- 172.21.15.28:0/982808878 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe410057d80 con 0x7fe4280c4030 2024-02-21T23:03:48.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.662+0000 7fe42f200640 1 -- 172.21.15.28:0/982808878 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe4280c5b40 con 0x7fe4280c4030 2024-02-21T23:03:48.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.663+0000 7fe41fe00640 1 -- 172.21.15.28:0/982808878 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe41005e0d0 con 0x7fe4280c4030 2024-02-21T23:03:48.653 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-02-21T23:03:48.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.666+0000 7fe42f200640 1 -- 172.21.15.28:0/982808878 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe40406b500 msgr2=0x7fe40406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:48.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.666+0000 7fe42f200640 1 --2- 172.21.15.28:0/982808878 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe40406b500 0x7fe40406d9c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fe418036000 tx=0x7fe41803d040 comp rx=0 tx=0).stop 2024-02-21T23:03:48.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.666+0000 7fe42f200640 1 -- 172.21.15.28:0/982808878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4280c4030 msgr2=0x7fe4280af1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:48.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.666+0000 7fe42f200640 1 --2- 172.21.15.28:0/982808878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4280c4030 0x7fe4280af1a0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fe41000c440 tx=0x7fe41000c900 comp rx=0 tx=0).stop 2024-02-21T23:03:48.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.666+0000 7fe42f200640 1 -- 172.21.15.28:0/982808878 shutdown_connections 2024-02-21T23:03:48.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.666+0000 7fe42f200640 1 --2- 172.21.15.28:0/982808878 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe40406b500 0x7fe40406d9c0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:48.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.666+0000 7fe42f200640 1 --2- 172.21.15.28:0/982808878 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe4280c4d30 0x7fe4281deb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:48.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.666+0000 7fe42f200640 1 --2- 172.21.15.28:0/982808878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4280c4030 0x7fe4280af1a0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:48.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.666+0000 7fe42f200640 1 --2- 172.21.15.28:0/982808878 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe42806d3b0 0x7fe4280aec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:48.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.666+0000 7fe42f200640 1 -- 172.21.15.28:0/982808878 >> 172.21.15.28:0/982808878 conn(0x7fe4280725a0 msgr2=0x7fe4280c0a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:48.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.666+0000 7fe42f200640 1 -- 172.21.15.28:0/982808878 shutdown_connections 2024-02-21T23:03:48.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.666+0000 7fe42f200640 1 -- 172.21.15.28:0/982808878 wait complete. 2024-02-21T23:03:48.675 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-21T23:03:48.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.908+0000 7f2d22800640 1 Processor -- start 2024-02-21T23:03:48.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.908+0000 7f2d22800640 1 -- start start 2024-02-21T23:03:48.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.910+0000 7f2d22800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f2d1c13e0c0 0x7f2d1c0c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:48.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.910+0000 7f2d22800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2d1c0c76f0 0x7f2d1c06c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:48.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.910+0000 7f2d22800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d1c158550 con 0x7f2d1c06dbc0 2024-02-21T23:03:48.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.910+0000 7f2d22800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d1c0c89a0 con 0x7f2d1c0c76f0 2024-02-21T23:03:48.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.910+0000 7f2d22800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d1c1587d0 con 0x7f2d1c13e0c0 2024-02-21T23:03:48.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.910+0000 7f2d21e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2d1c0c76f0 0x7f2d1c06c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:48.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.910+0000 7f2d21e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2d1c0c76f0 0x7f2d1c06c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41774/0 (socket says 172.21.15.28:41774) 2024-02-21T23:03:48.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.910+0000 7f2d21e00640 1 -- 172.21.15.28:0/2474668864 learned_addr learned my addr 172.21.15.28:0/2474668864 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:48.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.910+0000 7f2d21400640 1 --2- 172.21.15.28:0/2474668864 >> v2:172.21.15.81:3300/0 conn(0x7f2d1c13e0c0 0x7f2d1c0c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:48.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.911+0000 7f2d21e00640 1 -- 172.21.15.28:0/2474668864 >> v2:172.21.15.81:3300/0 conn(0x7f2d1c13e0c0 msgr2=0x7f2d1c0c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:48.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.911+0000 7f2d21e00640 1 --2- 172.21.15.28:0/2474668864 >> v2:172.21.15.81:3300/0 conn(0x7f2d1c13e0c0 0x7f2d1c0c71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:48.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.911+0000 7f2d21e00640 1 -- 172.21.15.28:0/2474668864 >> v1:172.21.15.81:6789/0 conn(0x7f2d1c06dbc0 legacy=0x7f2d1c06c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:48.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.911+0000 7f2d21e00640 1 -- 172.21.15.28:0/2474668864 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d1c158ab0 con 0x7f2d1c0c76f0 2024-02-21T23:03:48.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.912+0000 7f2d21e00640 1 --2- 172.21.15.28:0/2474668864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d1c0c76f0 0x7f2d1c06c990 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c0ad180 tx=0x7f2d1402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42ad0395ced9d4b4 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:48.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.913+0000 7f2d1be00640 1 -- 172.21.15.28:0/2474668864 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d14035020 con 0x7f2d1c0c76f0 2024-02-21T23:03:48.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.913+0000 7f2d1be00640 1 -- 172.21.15.28:0/2474668864 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d14031870 con 0x7f2d1c0c76f0 2024-02-21T23:03:48.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.913+0000 7f2d1be00640 1 -- 172.21.15.28:0/2474668864 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d14031c40 con 0x7f2d1c0c76f0 2024-02-21T23:03:48.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.913+0000 7f2d22800640 1 -- 172.21.15.28:0/2474668864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d1c0c76f0 msgr2=0x7f2d1c06c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:48.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.913+0000 7f2d22800640 1 --2- 172.21.15.28:0/2474668864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d1c0c76f0 0x7f2d1c06c990 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c0ad180 tx=0x7f2d1402b910 comp rx=0 tx=0).stop 2024-02-21T23:03:48.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.913+0000 7f2d22800640 1 -- 172.21.15.28:0/2474668864 shutdown_connections 2024-02-21T23:03:48.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.913+0000 7f2d22800640 1 --2- 172.21.15.28:0/2474668864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d1c0c76f0 0x7f2d1c06c990 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:48.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.913+0000 7f2d22800640 1 --2- 172.21.15.28:0/2474668864 >> v2:172.21.15.81:3300/0 conn(0x7f2d1c13e0c0 0x7f2d1c0c71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:48.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.913+0000 7f2d22800640 1 -- 172.21.15.28:0/2474668864 >> 172.21.15.28:0/2474668864 conn(0x7f2d1c073110 msgr2=0x7f2d1c0ac490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:48.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.913+0000 7f2d22800640 1 -- 172.21.15.28:0/2474668864 shutdown_connections 2024-02-21T23:03:48.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.914+0000 7f2d22800640 1 -- 172.21.15.28:0/2474668864 wait complete. 2024-02-21T23:03:48.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.915+0000 7f2d22800640 1 Processor -- start 2024-02-21T23:03:48.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.915+0000 7f2d22800640 1 -- start start 2024-02-21T23:03:48.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.916+0000 7f2d22800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2d1c06dbc0 0x7f2d1c1ee730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:48.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.916+0000 7f2d22800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d1c0c76f0 0x7f2d1c138360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:48.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.916+0000 7f2d22800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2d1c13e0c0 0x7f2d1c150100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:48.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.916+0000 7f2d22800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d1c06ced0 con 0x7f2d1c0c76f0 2024-02-21T23:03:48.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.916+0000 7f2d22800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d1c159070 con 0x7f2d1c13e0c0 2024-02-21T23:03:48.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.916+0000 7f2d22800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d1c13e520 con 0x7f2d1c06dbc0 2024-02-21T23:03:48.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.916+0000 7f2d20a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d1c0c76f0 0x7f2d1c138360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:48.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.916+0000 7f2d20a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d1c0c76f0 0x7f2d1c138360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41790/0 (socket says 172.21.15.28:41790) 2024-02-21T23:03:48.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.916+0000 7f2d20a00640 1 -- 172.21.15.28:0/518559325 learned_addr learned my addr 172.21.15.28:0/518559325 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:48.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.917+0000 7f2d21e00640 1 --2- 172.21.15.28:0/518559325 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2d1c13e0c0 0x7f2d1c150100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:48.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.917+0000 7f2d21400640 1 --2- 172.21.15.28:0/518559325 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2d1c06dbc0 0x7f2d1c1ee730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:48.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.917+0000 7f2d20a00640 1 -- 172.21.15.28:0/518559325 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2d1c06dbc0 msgr2=0x7f2d1c1ee730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:48.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.917+0000 7f2d20a00640 1 --2- 172.21.15.28:0/518559325 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2d1c06dbc0 0x7f2d1c1ee730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:48.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.917+0000 7f2d20a00640 1 -- 172.21.15.28:0/518559325 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2d1c13e0c0 msgr2=0x7f2d1c150100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:48.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.917+0000 7f2d20a00640 1 --2- 172.21.15.28:0/518559325 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2d1c13e0c0 0x7f2d1c150100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:48.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.917+0000 7f2d20a00640 1 -- 172.21.15.28:0/518559325 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d1c132f30 con 0x7f2d1c0c76f0 2024-02-21T23:03:48.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.917+0000 7f2d20a00640 1 --2- 172.21.15.28:0/518559325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d1c0c76f0 0x7f2d1c138360 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f2d0800c400 tx=0x7f2d0800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:48.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.917+0000 7f2d19600640 1 -- 172.21.15.28:0/518559325 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d080052d0 con 0x7f2d1c0c76f0 2024-02-21T23:03:48.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.918+0000 7f2d19600640 1 -- 172.21.15.28:0/518559325 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d08005490 con 0x7f2d1c0c76f0 2024-02-21T23:03:48.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.918+0000 7f2d19600640 1 -- 172.21.15.28:0/518559325 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d08007260 con 0x7f2d1c0c76f0 2024-02-21T23:03:48.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.918+0000 7f2d22800640 1 -- 172.21.15.28:0/518559325 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d1c133210 con 0x7f2d1c0c76f0 2024-02-21T23:03:48.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.918+0000 7f2d22800640 1 -- 172.21.15.28:0/518559325 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d1c1ee210 con 0x7f2d1c0c76f0 2024-02-21T23:03:48.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.919+0000 7f2d22800640 1 -- 172.21.15.28:0/518559325 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d1c04f290 con 0x7f2d1c0c76f0 2024-02-21T23:03:48.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.920+0000 7f2d19600640 1 -- 172.21.15.28:0/518559325 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2d08007400 con 0x7f2d1c0c76f0 2024-02-21T23:03:48.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.921+0000 7f2d19600640 1 --2- 172.21.15.28:0/518559325 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2cfc06b4d0 0x7f2cfc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:48.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.921+0000 7f2d19600640 1 -- 172.21.15.28:0/518559325 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f2d08089b60 con 0x7f2d1c0c76f0 2024-02-21T23:03:48.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.922+0000 7f2d21400640 1 --2- 172.21.15.28:0/518559325 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2cfc06b4d0 0x7f2cfc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:48.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.923+0000 7f2d21400640 1 --2- 172.21.15.28:0/518559325 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2cfc06b4d0 0x7f2cfc06d990 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c1380e0 tx=0x7f2d100039b0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:48.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:48.925+0000 7f2d19600640 1 -- 172.21.15.28:0/518559325 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f2d1c158550 con 0x7f2d1c0c76f0 2024-02-21T23:03:49.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.123+0000 7f2d22800640 1 -- 172.21.15.28:0/518559325 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2d1c06d9a0 con 0x7f2d1c0c76f0 2024-02-21T23:03:49.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.124+0000 7f2d19600640 1 -- 172.21.15.28:0/518559325 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2d080585d0 con 0x7f2d1c0c76f0 2024-02-21T23:03:49.114 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-02-21T23:03:49.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.127+0000 7f2d22800640 1 -- 172.21.15.28:0/518559325 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2cfc06b4d0 msgr2=0x7f2cfc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:49.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.127+0000 7f2d22800640 1 --2- 172.21.15.28:0/518559325 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2cfc06b4d0 0x7f2cfc06d990 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c1380e0 tx=0x7f2d100039b0 comp rx=0 tx=0).stop 2024-02-21T23:03:49.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.127+0000 7f2d22800640 1 -- 172.21.15.28:0/518559325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d1c0c76f0 msgr2=0x7f2d1c138360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:49.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.127+0000 7f2d22800640 1 --2- 172.21.15.28:0/518559325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d1c0c76f0 0x7f2d1c138360 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f2d0800c400 tx=0x7f2d0800c8c0 comp rx=0 tx=0).stop 2024-02-21T23:03:49.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.127+0000 7f2d22800640 1 -- 172.21.15.28:0/518559325 shutdown_connections 2024-02-21T23:03:49.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.127+0000 7f2d22800640 1 --2- 172.21.15.28:0/518559325 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2cfc06b4d0 0x7f2cfc06d990 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:49.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.127+0000 7f2d22800640 1 --2- 172.21.15.28:0/518559325 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2d1c13e0c0 0x7f2d1c150100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:49.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.127+0000 7f2d22800640 1 --2- 172.21.15.28:0/518559325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d1c0c76f0 0x7f2d1c138360 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:49.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.127+0000 7f2d22800640 1 --2- 172.21.15.28:0/518559325 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2d1c06dbc0 0x7f2d1c1ee730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:49.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.127+0000 7f2d22800640 1 -- 172.21.15.28:0/518559325 >> 172.21.15.28:0/518559325 conn(0x7f2d1c073110 msgr2=0x7f2d1c06c280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:49.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.127+0000 7f2d22800640 1 -- 172.21.15.28:0/518559325 shutdown_connections 2024-02-21T23:03:49.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.127+0000 7f2d22800640 1 -- 172.21.15.28:0/518559325 wait complete. 2024-02-21T23:03:49.137 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-02-21T23:03:49.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.373+0000 7fc3a7e00640 1 Processor -- start 2024-02-21T23:03:49.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.374+0000 7fc3a7e00640 1 -- start start 2024-02-21T23:03:49.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.375+0000 7fc3a7e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fc3a806f530 0x7fc3a8151a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:49.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.375+0000 7fc3a7e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc3a80c5a90 0x7fc3a80c1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:49.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.375+0000 7fc3a7e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3a8154710 con 0x7fc3a806b0e0 2024-02-21T23:03:49.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.375+0000 7fc3a7e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3a81549f0 con 0x7fc3a80c5a90 2024-02-21T23:03:49.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.375+0000 7fc3a7e00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3a8154cd0 con 0x7fc3a806f530 2024-02-21T23:03:49.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.375+0000 7fc3a6000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc3a80c5a90 0x7fc3a80c1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:49.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.376+0000 7fc3a6000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc3a80c5a90 0x7fc3a80c1a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41796/0 (socket says 172.21.15.28:41796) 2024-02-21T23:03:49.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.376+0000 7fc3a6000640 1 -- 172.21.15.28:0/658588353 learned_addr learned my addr 172.21.15.28:0/658588353 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:49.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.376+0000 7fc3a6a00640 1 --2- 172.21.15.28:0/658588353 >> v2:172.21.15.81:3300/0 conn(0x7fc3a806f530 0x7fc3a8151a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:49.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.376+0000 7fc3a6000640 1 -- 172.21.15.28:0/658588353 >> v2:172.21.15.81:3300/0 conn(0x7fc3a806f530 msgr2=0x7fc3a8151a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:49.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.376+0000 7fc3a6000640 1 --2- 172.21.15.28:0/658588353 >> v2:172.21.15.81:3300/0 conn(0x7fc3a806f530 0x7fc3a8151a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:49.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.376+0000 7fc3a6000640 1 -- 172.21.15.28:0/658588353 >> v1:172.21.15.81:6789/0 conn(0x7fc3a806b0e0 legacy=0x7fc3a806db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:49.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.376+0000 7fc3a6000640 1 -- 172.21.15.28:0/658588353 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3a8154fb0 con 0x7fc3a80c5a90 2024-02-21T23:03:49.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.377+0000 7fc3a6000640 1 --2- 172.21.15.28:0/658588353 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3a80c5a90 0x7fc3a80c1a90 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fc3a80c41c0 tx=0x7fc39402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29add88f8212b608 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:49.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.378+0000 7fc3a5600640 1 -- 172.21.15.28:0/658588353 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc39402e070 con 0x7fc3a80c5a90 2024-02-21T23:03:49.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.378+0000 7fc3a5600640 1 -- 172.21.15.28:0/658588353 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc394037070 con 0x7fc3a80c5a90 2024-02-21T23:03:49.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.378+0000 7fc3a5600640 1 -- 172.21.15.28:0/658588353 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc394031a00 con 0x7fc3a80c5a90 2024-02-21T23:03:49.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.378+0000 7fc3a7e00640 1 -- 172.21.15.28:0/658588353 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3a80c5a90 msgr2=0x7fc3a80c1a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:49.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.378+0000 7fc3a7e00640 1 --2- 172.21.15.28:0/658588353 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3a80c5a90 0x7fc3a80c1a90 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fc3a80c41c0 tx=0x7fc39402b880 comp rx=0 tx=0).stop 2024-02-21T23:03:49.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.378+0000 7fc3a7e00640 1 -- 172.21.15.28:0/658588353 shutdown_connections 2024-02-21T23:03:49.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.378+0000 7fc3a7e00640 1 --2- 172.21.15.28:0/658588353 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3a80c5a90 0x7fc3a80c1a90 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:49.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.378+0000 7fc3a7e00640 1 --2- 172.21.15.28:0/658588353 >> v2:172.21.15.81:3300/0 conn(0x7fc3a806f530 0x7fc3a8151a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:49.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.378+0000 7fc3a7e00640 1 -- 172.21.15.28:0/658588353 >> 172.21.15.28:0/658588353 conn(0x7fc3a80a8530 msgr2=0x7fc3a80a8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:49.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.378+0000 7fc3a7e00640 1 -- 172.21.15.28:0/658588353 shutdown_connections 2024-02-21T23:03:49.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.378+0000 7fc3a7e00640 1 -- 172.21.15.28:0/658588353 wait complete. 2024-02-21T23:03:49.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.380+0000 7fc3a7e00640 1 Processor -- start 2024-02-21T23:03:49.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.381+0000 7fc3a7e00640 1 -- start start 2024-02-21T23:03:49.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.382+0000 7fc3a7e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc3a806b0e0 0x7fc3a8147070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:49.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.382+0000 7fc3a7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3a806f530 0x7fc3a8152610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:49.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.382+0000 7fc3a7e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc3a80c5a90 0x7fc3a8072b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:49.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.382+0000 7fc3a7e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a8151330 con 0x7fc3a806f530 2024-02-21T23:03:49.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.382+0000 7fc3a7e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a81511b0 con 0x7fc3a806b0e0 2024-02-21T23:03:49.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.382+0000 7fc3a7e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a81514b0 con 0x7fc3a80c5a90 2024-02-21T23:03:49.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.382+0000 7fc3a6000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3a806f530 0x7fc3a8152610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:49.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.382+0000 7fc3a6000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3a806f530 0x7fc3a8152610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41812/0 (socket says 172.21.15.28:41812) 2024-02-21T23:03:49.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.382+0000 7fc3a6000640 1 -- 172.21.15.28:0/2900826793 learned_addr learned my addr 172.21.15.28:0/2900826793 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:49.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.382+0000 7fc3a6a00640 1 --2- 172.21.15.28:0/2900826793 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc3a806b0e0 0x7fc3a8147070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:49.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.382+0000 7fc3a7400640 1 --2- 172.21.15.28:0/2900826793 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc3a80c5a90 0x7fc3a8072b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:49.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.383+0000 7fc3a6000640 1 -- 172.21.15.28:0/2900826793 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc3a80c5a90 msgr2=0x7fc3a8072b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:49.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.383+0000 7fc3a6000640 1 --2- 172.21.15.28:0/2900826793 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc3a80c5a90 0x7fc3a8072b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:49.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.383+0000 7fc3a6000640 1 -- 172.21.15.28:0/2900826793 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc3a806b0e0 msgr2=0x7fc3a8147070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:49.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.383+0000 7fc3a6000640 1 --2- 172.21.15.28:0/2900826793 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc3a806b0e0 0x7fc3a8147070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:49.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.383+0000 7fc3a6000640 1 -- 172.21.15.28:0/2900826793 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3a81475b0 con 0x7fc3a806f530 2024-02-21T23:03:49.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.383+0000 7fc3a7400640 1 --2- 172.21.15.28:0/2900826793 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc3a80c5a90 0x7fc3a8072b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:03:49.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.383+0000 7fc3a6000640 1 --2- 172.21.15.28:0/2900826793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3a806f530 0x7fc3a8152610 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fc39402b8c0 tx=0x7fc3940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:49.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.383+0000 7fc39ea00640 1 -- 172.21.15.28:0/2900826793 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc394032880 con 0x7fc3a806f530 2024-02-21T23:03:49.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.383+0000 7fc39ea00640 1 -- 172.21.15.28:0/2900826793 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc394032a20 con 0x7fc3a806f530 2024-02-21T23:03:49.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.383+0000 7fc39ea00640 1 -- 172.21.15.28:0/2900826793 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc394032e10 con 0x7fc3a806f530 2024-02-21T23:03:49.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.383+0000 7fc3a7e00640 1 -- 172.21.15.28:0/2900826793 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3a813e9e0 con 0x7fc3a806f530 2024-02-21T23:03:49.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.384+0000 7fc3a7e00640 1 -- 172.21.15.28:0/2900826793 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3a81eeb90 con 0x7fc3a806f530 2024-02-21T23:03:49.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.385+0000 7fc3a7e00640 1 -- 172.21.15.28:0/2900826793 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3a804f290 con 0x7fc3a806f530 2024-02-21T23:03:49.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.386+0000 7fc39ea00640 1 -- 172.21.15.28:0/2900826793 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc3940453f0 con 0x7fc3a806f530 2024-02-21T23:03:49.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.387+0000 7fc39ea00640 1 --2- 172.21.15.28:0/2900826793 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc38806b4d0 0x7fc38806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:49.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.387+0000 7fc39ea00640 1 -- 172.21.15.28:0/2900826793 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6735+0+0 (secure 0 0 0) 0x7fc3940ac4e0 con 0x7fc3a806f530 2024-02-21T23:03:49.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.387+0000 7fc3a6a00640 1 --2- 172.21.15.28:0/2900826793 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc38806b4d0 0x7fc38806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:49.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.388+0000 7fc3a6a00640 1 --2- 172.21.15.28:0/2900826793 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc38806b4d0 0x7fc38806d990 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc398006fd0 tx=0x7fc398008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:49.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.391+0000 7fc39ea00640 1 -- 172.21.15.28:0/2900826793 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc394040070 con 0x7fc3a806f530 2024-02-21T23:03:49.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.594+0000 7fc3a7e00640 1 -- 172.21.15.28:0/2900826793 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fc3a8152b50 con 0x7fc3a806f530 2024-02-21T23:03:49.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.594+0000 7fc39ea00640 1 -- 172.21.15.28:0/2900826793 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) v1 ==== 103+0+11 (secure 0 0 0) 0x7fc39407aff0 con 0x7fc3a806f530 2024-02-21T23:03:49.584 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-02-21T23:03:49.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.597+0000 7fc3a7e00640 1 -- 172.21.15.28:0/2900826793 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc38806b4d0 msgr2=0x7fc38806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:49.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.597+0000 7fc3a7e00640 1 --2- 172.21.15.28:0/2900826793 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc38806b4d0 0x7fc38806d990 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc398006fd0 tx=0x7fc398008040 comp rx=0 tx=0).stop 2024-02-21T23:03:49.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.597+0000 7fc3a7e00640 1 -- 172.21.15.28:0/2900826793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3a806f530 msgr2=0x7fc3a8152610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:49.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.597+0000 7fc3a7e00640 1 --2- 172.21.15.28:0/2900826793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3a806f530 0x7fc3a8152610 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fc39402b8c0 tx=0x7fc3940046c0 comp rx=0 tx=0).stop 2024-02-21T23:03:49.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.597+0000 7fc3a7e00640 1 -- 172.21.15.28:0/2900826793 shutdown_connections 2024-02-21T23:03:49.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.597+0000 7fc3a7e00640 1 --2- 172.21.15.28:0/2900826793 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc38806b4d0 0x7fc38806d990 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:49.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.597+0000 7fc3a7e00640 1 --2- 172.21.15.28:0/2900826793 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc3a80c5a90 0x7fc3a8072b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:49.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.597+0000 7fc3a7e00640 1 --2- 172.21.15.28:0/2900826793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3a806f530 0x7fc3a8152610 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:49.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.597+0000 7fc3a7e00640 1 --2- 172.21.15.28:0/2900826793 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc3a806b0e0 0x7fc3a8147070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:49.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.597+0000 7fc3a7e00640 1 -- 172.21.15.28:0/2900826793 >> 172.21.15.28:0/2900826793 conn(0x7fc3a80a8530 msgr2=0x7fc3a814a5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:49.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.598+0000 7fc3a7e00640 1 -- 172.21.15.28:0/2900826793 shutdown_connections 2024-02-21T23:03:49.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.598+0000 7fc3a7e00640 1 -- 172.21.15.28:0/2900826793 wait complete. 2024-02-21T23:03:49.608 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-02-21T23:03:49.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.831+0000 7fd953e00640 1 Processor -- start 2024-02-21T23:03:49.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.832+0000 7fd953e00640 1 -- start start 2024-02-21T23:03:49.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.833+0000 7fd953e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd954149830 0x7fd9540a8e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:49.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.833+0000 7fd953e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd95406d410 0x7fd9540c0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:49.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.833+0000 7fd953e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd954149c10 con 0x7fd9540c1420 2024-02-21T23:03:49.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.833+0000 7fd953e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9541537d0 con 0x7fd95406d410 2024-02-21T23:03:49.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.833+0000 7fd953e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd954153ab0 con 0x7fd954149830 2024-02-21T23:03:49.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.834+0000 7fd953400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd9540c1420 0x7fd9540c5a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:46994/0 (socket says 172.21.15.28:46994) 2024-02-21T23:03:49.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.834+0000 7fd953400640 1 -- 172.21.15.28:0/370500339 learned_addr learned my addr 172.21.15.28:0/370500339 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:03:49.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.834+0000 7fd952a00640 1 --2- 172.21.15.28:0/370500339 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd954149830 0x7fd9540a8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:49.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.834+0000 7fd952000640 1 --2- 172.21.15.28:0/370500339 >> v2:172.21.15.28:3300/0 conn(0x7fd95406d410 0x7fd9540c0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:49.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.834+0000 7fd951600640 1 -- 172.21.15.28:0/370500339 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1714635075 0 0) 0x7fd954149c10 con 0x7fd9540c1420 2024-02-21T23:03:49.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.834+0000 7fd951600640 1 -- 172.21.15.28:0/370500339 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd938003520 con 0x7fd9540c1420 2024-02-21T23:03:49.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.835+0000 7fd952000640 1 -- 172.21.15.28:0/370500339 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd954149830 msgr2=0x7fd9540a8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:49.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.835+0000 7fd952000640 1 --2- 172.21.15.28:0/370500339 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd954149830 0x7fd9540a8e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:49.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.835+0000 7fd952000640 1 -- 172.21.15.28:0/370500339 >> v1:172.21.15.28:6789/0 conn(0x7fd9540c1420 legacy=0x7fd9540c5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:49.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.835+0000 7fd952000640 1 -- 172.21.15.28:0/370500339 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9540c4fa0 con 0x7fd95406d410 2024-02-21T23:03:49.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.836+0000 7fd952000640 1 --2- 172.21.15.28:0/370500339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd95406d410 0x7fd9540c0ee0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fd9400097a0 tx=0x7fd94002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1ef49d821f84844 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:49.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.836+0000 7fd951600640 1 -- 172.21.15.28:0/370500339 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd940034070 con 0x7fd95406d410 2024-02-21T23:03:49.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.836+0000 7fd951600640 1 -- 172.21.15.28:0/370500339 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd940030610 con 0x7fd95406d410 2024-02-21T23:03:49.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.836+0000 7fd951600640 1 -- 172.21.15.28:0/370500339 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9400309e0 con 0x7fd95406d410 2024-02-21T23:03:49.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.837+0000 7fd953e00640 1 -- 172.21.15.28:0/370500339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd95406d410 msgr2=0x7fd9540c0ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:49.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.837+0000 7fd953e00640 1 --2- 172.21.15.28:0/370500339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd95406d410 0x7fd9540c0ee0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fd9400097a0 tx=0x7fd94002b960 comp rx=0 tx=0).stop 2024-02-21T23:03:49.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.837+0000 7fd953e00640 1 -- 172.21.15.28:0/370500339 shutdown_connections 2024-02-21T23:03:49.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.837+0000 7fd953e00640 1 --2- 172.21.15.28:0/370500339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd95406d410 0x7fd9540c0ee0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:49.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.837+0000 7fd953e00640 1 --2- 172.21.15.28:0/370500339 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd954149830 0x7fd9540a8e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:49.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.837+0000 7fd953e00640 1 -- 172.21.15.28:0/370500339 >> 172.21.15.28:0/370500339 conn(0x7fd9540a8320 msgr2=0x7fd9540a8730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:49.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.837+0000 7fd953e00640 1 -- 172.21.15.28:0/370500339 shutdown_connections 2024-02-21T23:03:49.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.837+0000 7fd953e00640 1 -- 172.21.15.28:0/370500339 wait complete. 2024-02-21T23:03:49.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.839+0000 7fd953e00640 1 Processor -- start 2024-02-21T23:03:49.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.839+0000 7fd953e00640 1 -- start start 2024-02-21T23:03:49.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.840+0000 7fd953e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd95406d410 0x7fd9541500d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:49.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.840+0000 7fd953e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9540c1420 0x7fd9540a9bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:49.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.840+0000 7fd953e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd954149830 0x7fd9541eb9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:49.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.840+0000 7fd953e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9540c6190 con 0x7fd9540c1420 2024-02-21T23:03:49.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.840+0000 7fd953e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9540c5560 con 0x7fd954149830 2024-02-21T23:03:49.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.840+0000 7fd953e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9540c6310 con 0x7fd95406d410 2024-02-21T23:03:49.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.840+0000 7fd952000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9540c1420 0x7fd9540a9bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:49.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.840+0000 7fd952000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9540c1420 0x7fd9540a9bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41824/0 (socket says 172.21.15.28:41824) 2024-02-21T23:03:49.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.840+0000 7fd952000640 1 -- 172.21.15.28:0/2722705866 learned_addr learned my addr 172.21.15.28:0/2722705866 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:49.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.840+0000 7fd952a00640 1 --2- 172.21.15.28:0/2722705866 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd95406d410 0x7fd9541500d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:49.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.840+0000 7fd953400640 1 --2- 172.21.15.28:0/2722705866 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd954149830 0x7fd9541eb9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:49.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.840+0000 7fd952000640 1 -- 172.21.15.28:0/2722705866 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd95406d410 msgr2=0x7fd9541500d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:49.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.840+0000 7fd952000640 1 --2- 172.21.15.28:0/2722705866 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd95406d410 0x7fd9541500d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:49.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.840+0000 7fd952000640 1 -- 172.21.15.28:0/2722705866 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd954149830 msgr2=0x7fd9541eb9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:49.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.840+0000 7fd952000640 1 --2- 172.21.15.28:0/2722705866 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd954149830 0x7fd9541eb9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:49.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.840+0000 7fd952000640 1 -- 172.21.15.28:0/2722705866 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd95414d170 con 0x7fd9540c1420 2024-02-21T23:03:49.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.841+0000 7fd952000640 1 --2- 172.21.15.28:0/2722705866 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9540c1420 0x7fd9540a9bf0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fd9400098c0 tx=0x7fd9400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:49.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.841+0000 7fd94aa00640 1 -- 172.21.15.28:0/2722705866 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd940049070 con 0x7fd9540c1420 2024-02-21T23:03:49.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.841+0000 7fd94aa00640 1 -- 172.21.15.28:0/2722705866 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd940030410 con 0x7fd9540c1420 2024-02-21T23:03:49.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.841+0000 7fd953e00640 1 -- 172.21.15.28:0/2722705866 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd95414d5d0 con 0x7fd9540c1420 2024-02-21T23:03:49.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.841+0000 7fd94aa00640 1 -- 172.21.15.28:0/2722705866 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd940034040 con 0x7fd9540c1420 2024-02-21T23:03:49.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.841+0000 7fd953e00640 1 -- 172.21.15.28:0/2722705866 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd954156c60 con 0x7fd9540c1420 2024-02-21T23:03:49.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.842+0000 7fd953e00640 1 -- 172.21.15.28:0/2722705866 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9540c73f0 con 0x7fd9540c1420 2024-02-21T23:03:49.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.843+0000 7fd94aa00640 1 -- 172.21.15.28:0/2722705866 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd94004c070 con 0x7fd9540c1420 2024-02-21T23:03:49.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.844+0000 7fd94aa00640 1 --2- 172.21.15.28:0/2722705866 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd93406b4d0 0x7fd93406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:49.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.844+0000 7fd94aa00640 1 -- 172.21.15.28:0/2722705866 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6735+0+0 (secure 0 0 0) 0x7fd9400ac290 con 0x7fd9540c1420 2024-02-21T23:03:49.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.845+0000 7fd952a00640 1 --2- 172.21.15.28:0/2722705866 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd93406b4d0 0x7fd93406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:49.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.846+0000 7fd952a00640 1 --2- 172.21.15.28:0/2722705866 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd93406b4d0 0x7fd93406d990 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd944009a30 tx=0x7fd944008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:49.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:49.849+0000 7fd94aa00640 1 -- 172.21.15.28:0/2722705866 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd9540c73f0 con 0x7fd9540c1420 2024-02-21T23:03:50.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.064+0000 7fd953e00640 1 -- 172.21.15.28:0/2722705866 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fd95414d340 con 0x7fd9540c1420 2024-02-21T23:03:50.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.065+0000 7fd94aa00640 1 -- 172.21.15.28:0/2722705866 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v26) v1 ==== 80+0+0 (secure 0 0 0) 0x7fd94007ac70 con 0x7fd9540c1420 2024-02-21T23:03:50.055 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-02-21T23:03:50.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.067+0000 7fd953e00640 1 -- 172.21.15.28:0/2722705866 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd93406b4d0 msgr2=0x7fd93406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:50.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.067+0000 7fd953e00640 1 --2- 172.21.15.28:0/2722705866 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd93406b4d0 0x7fd93406d990 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd944009a30 tx=0x7fd944008040 comp rx=0 tx=0).stop 2024-02-21T23:03:50.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.067+0000 7fd953e00640 1 -- 172.21.15.28:0/2722705866 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9540c1420 msgr2=0x7fd9540a9bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:50.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.067+0000 7fd953e00640 1 --2- 172.21.15.28:0/2722705866 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9540c1420 0x7fd9540a9bf0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fd9400098c0 tx=0x7fd9400046c0 comp rx=0 tx=0).stop 2024-02-21T23:03:50.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.068+0000 7fd953e00640 1 -- 172.21.15.28:0/2722705866 shutdown_connections 2024-02-21T23:03:50.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.068+0000 7fd953e00640 1 --2- 172.21.15.28:0/2722705866 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd93406b4d0 0x7fd93406d990 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:50.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.068+0000 7fd953e00640 1 --2- 172.21.15.28:0/2722705866 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd954149830 0x7fd9541eb9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:50.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.068+0000 7fd953e00640 1 --2- 172.21.15.28:0/2722705866 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9540c1420 0x7fd9540a9bf0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:50.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.068+0000 7fd953e00640 1 --2- 172.21.15.28:0/2722705866 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd95406d410 0x7fd9541500d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:50.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.068+0000 7fd953e00640 1 -- 172.21.15.28:0/2722705866 >> 172.21.15.28:0/2722705866 conn(0x7fd9540a8320 msgr2=0x7fd9540a70b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:50.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.068+0000 7fd953e00640 1 -- 172.21.15.28:0/2722705866 shutdown_connections 2024-02-21T23:03:50.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.068+0000 7fd953e00640 1 -- 172.21.15.28:0/2722705866 wait complete. 2024-02-21T23:03:50.077 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-02-21T23:03:50.077 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:03:50.077 DEBUG:teuthology.orchestra.run.smithi081:> sudo umount /home/ubuntu/cephtest/mnt.1 -f 2024-02-21T23:03:50.192 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi081.front.sepia.ceph.com 2024-02-21T23:03:50.193 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:03:50.193 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-02-21T23:03:50.250 INFO:tasks.cephfs.cephfs_test_case:[, ] 2024-02-21T23:03:50.250 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:03:50.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.491+0000 7f5393a00640 1 Processor -- start 2024-02-21T23:03:50.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.492+0000 7f5393a00640 1 -- start start 2024-02-21T23:03:50.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.493+0000 7f5393a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f538c0c11f0 0x7f538c153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:50.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.493+0000 7f5393a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f538c152e70 con 0x7f538c0c15d0 2024-02-21T23:03:50.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.493+0000 7f5393a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f538c153150 con 0x7f538c15b6d0 2024-02-21T23:03:50.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.493+0000 7f5393a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f538c0c1a60 con 0x7f538c0c11f0 2024-02-21T23:03:50.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.493+0000 7f538be00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f538c0c15d0 0x7f538c06dce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:47008/0 (socket says 172.21.15.28:47008) 2024-02-21T23:03:50.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.493+0000 7f538be00640 1 -- 172.21.15.28:0/2516559271 learned_addr learned my addr 172.21.15.28:0/2516559271 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:03:50.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.494+0000 7f5391000640 1 --2- 172.21.15.28:0/2516559271 >> v2:172.21.15.81:3300/0 conn(0x7f538c0c11f0 0x7f538c153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:50.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.494+0000 7f538b400640 1 -- 172.21.15.28:0/2516559271 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1857371272 0 0) 0x7f538c152e70 con 0x7f538c0c15d0 2024-02-21T23:03:50.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.494+0000 7f538b400640 1 -- 172.21.15.28:0/2516559271 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5374003520 con 0x7f538c0c15d0 2024-02-21T23:03:50.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.494+0000 7f538b400640 1 -- 172.21.15.28:0/2516559271 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1400653211 0 0) 0x7f5374003520 con 0x7f538c0c15d0 2024-02-21T23:03:50.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.494+0000 7f538b400640 1 -- 172.21.15.28:0/2516559271 >> v2:172.21.15.81:3300/0 conn(0x7f538c0c11f0 msgr2=0x7f538c153950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:50.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.494+0000 7f538b400640 1 --2- 172.21.15.28:0/2516559271 >> v2:172.21.15.81:3300/0 conn(0x7f538c0c11f0 0x7f538c153950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:50.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.494+0000 7f538b400640 1 -- 172.21.15.28:0/2516559271 >> v1:172.21.15.81:6789/0 conn(0x7f538c15b6d0 legacy=0x7f538c15dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:50.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.494+0000 7f538b400640 1 -- 172.21.15.28:0/2516559271 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f538c153430 con 0x7f538c0c15d0 2024-02-21T23:03:50.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.494+0000 7f538b400640 1 -- 172.21.15.28:0/2516559271 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f5378001670 con 0x7f538c0c15d0 2024-02-21T23:03:50.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.494+0000 7f538b400640 1 -- 172.21.15.28:0/2516559271 >> v1:172.21.15.28:6789/0 conn(0x7f538c0c15d0 legacy=0x7f538c06dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:50.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.494+0000 7f538b400640 1 --2- 172.21.15.28:0/2516559271 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f53740039b0 0x7f5374003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:50.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.495+0000 7f538b400640 1 --2- 172.21.15.28:0/2516559271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5374004800 0x7f5374004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:50.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.495+0000 7f538b400640 1 --2- 172.21.15.28:0/2516559271 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f53740052e0 0x7f537400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:50.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.495+0000 7f538b400640 1 -- 172.21.15.28:0/2516559271 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f538c0c1a60 con 0x7f5374004800 2024-02-21T23:03:50.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.495+0000 7f538b400640 1 -- 172.21.15.28:0/2516559271 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f537400c1b0 con 0x7f53740052e0 2024-02-21T23:03:50.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.495+0000 7f538b400640 1 -- 172.21.15.28:0/2516559271 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f537400c3b0 con 0x7f53740039b0 2024-02-21T23:03:50.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.495+0000 7f538be00640 1 --2- 172.21.15.28:0/2516559271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5374004800 0x7f5374004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:50.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.495+0000 7f5391a00640 1 --2- 172.21.15.28:0/2516559271 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f53740052e0 0x7f537400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:50.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.495+0000 7f5391000640 1 --2- 172.21.15.28:0/2516559271 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f53740039b0 0x7f5374003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:50.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.495+0000 7f538be00640 1 -- 172.21.15.28:0/2516559271 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f53740039b0 msgr2=0x7f5374003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:50.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.495+0000 7f538be00640 1 --2- 172.21.15.28:0/2516559271 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f53740039b0 0x7f5374003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:50.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.495+0000 7f538be00640 1 -- 172.21.15.28:0/2516559271 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f53740052e0 msgr2=0x7f537400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:50.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.495+0000 7f538be00640 1 --2- 172.21.15.28:0/2516559271 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f53740052e0 0x7f537400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:50.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.495+0000 7f538be00640 1 -- 172.21.15.28:0/2516559271 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5374003520 con 0x7f5374004800 2024-02-21T23:03:50.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.495+0000 7f5391a00640 1 -- 172.21.15.28:0/2516559271 reap_dead start 2024-02-21T23:03:50.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.497+0000 7f538be00640 1 --2- 172.21.15.28:0/2516559271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5374004800 0x7f5374004c80 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f537800c210 tx=0x7f5374005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38046f2076267da7 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:50.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.497+0000 7f538b400640 1 -- 172.21.15.28:0/2516559271 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5378007cf0 con 0x7f5374004800 2024-02-21T23:03:50.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.497+0000 7f538b400640 1 -- 172.21.15.28:0/2516559271 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5378030070 con 0x7f5374004800 2024-02-21T23:03:50.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.497+0000 7f538b400640 1 -- 172.21.15.28:0/2516559271 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5378005d70 con 0x7f5374004800 2024-02-21T23:03:50.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.497+0000 7f5393a00640 1 -- 172.21.15.28:0/2516559271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5374004800 msgr2=0x7f5374004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:50.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.497+0000 7f5393a00640 1 --2- 172.21.15.28:0/2516559271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5374004800 0x7f5374004c80 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f537800c210 tx=0x7f5374005830 comp rx=0 tx=0).stop 2024-02-21T23:03:50.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.497+0000 7f5393a00640 1 -- 172.21.15.28:0/2516559271 shutdown_connections 2024-02-21T23:03:50.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.497+0000 7f5393a00640 1 --2- 172.21.15.28:0/2516559271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5374004800 0x7f5374004c80 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:50.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.497+0000 7f5393a00640 1 -- 172.21.15.28:0/2516559271 >> 172.21.15.28:0/2516559271 conn(0x7f538c0c67f0 msgr2=0x7f538c0c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:50.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.498+0000 7f5393a00640 1 -- 172.21.15.28:0/2516559271 shutdown_connections 2024-02-21T23:03:50.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.498+0000 7f5393a00640 1 -- 172.21.15.28:0/2516559271 wait complete. 2024-02-21T23:03:50.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.499+0000 7f5393a00640 1 Processor -- start 2024-02-21T23:03:50.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.499+0000 7f5393a00640 1 -- start start 2024-02-21T23:03:50.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.500+0000 7f5393a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5374004800 0x7f538c1418e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:50.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.500+0000 7f5393a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f538c14b920 0x7f538c0a7420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:50.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.500+0000 7f5393a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f538c1538c0 0x7f538c0a7960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:50.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.500+0000 7f5393a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f538c0c4c00 con 0x7f5374004800 2024-02-21T23:03:50.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.500+0000 7f5393a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f538c0a5ec0 con 0x7f538c14b920 2024-02-21T23:03:50.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.500+0000 7f5393a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f538c0a6110 con 0x7f538c1538c0 2024-02-21T23:03:50.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.501+0000 7f5391000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5374004800 0x7f538c1418e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:50.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.501+0000 7f5391000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5374004800 0x7f538c1418e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41840/0 (socket says 172.21.15.28:41840) 2024-02-21T23:03:50.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.501+0000 7f5391000640 1 -- 172.21.15.28:0/1020472204 learned_addr learned my addr 172.21.15.28:0/1020472204 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:50.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.501+0000 7f538be00640 1 --2- 172.21.15.28:0/1020472204 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f538c14b920 0x7f538c0a7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:50.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.501+0000 7f5391a00640 1 --2- 172.21.15.28:0/1020472204 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f538c1538c0 0x7f538c0a7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:50.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.501+0000 7f5391000640 1 -- 172.21.15.28:0/1020472204 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f538c1538c0 msgr2=0x7f538c0a7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:50.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.501+0000 7f5391000640 1 --2- 172.21.15.28:0/1020472204 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f538c1538c0 0x7f538c0a7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:50.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.501+0000 7f5391000640 1 -- 172.21.15.28:0/1020472204 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f538c14b920 msgr2=0x7f538c0a7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:50.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.501+0000 7f5391000640 1 --2- 172.21.15.28:0/1020472204 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f538c14b920 0x7f538c0a7420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:50.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.501+0000 7f5391000640 1 -- 172.21.15.28:0/1020472204 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f538c0a6380 con 0x7f5374004800 2024-02-21T23:03:50.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.501+0000 7f5391a00640 1 --2- 172.21.15.28:0/1020472204 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f538c1538c0 0x7f538c0a7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:03:50.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.502+0000 7f5391000640 1 --2- 172.21.15.28:0/1020472204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5374004800 0x7f538c1418e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f53800114e0 tx=0x7f53800119a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:50.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.502+0000 7f5388c00640 1 -- 172.21.15.28:0/1020472204 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5380015870 con 0x7f5374004800 2024-02-21T23:03:50.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.502+0000 7f5388c00640 1 -- 172.21.15.28:0/1020472204 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5380015a10 con 0x7f5374004800 2024-02-21T23:03:50.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.502+0000 7f5388c00640 1 -- 172.21.15.28:0/1020472204 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5380015dc0 con 0x7f5374004800 2024-02-21T23:03:50.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.502+0000 7f5393a00640 1 -- 172.21.15.28:0/1020472204 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f538c0a6660 con 0x7f5374004800 2024-02-21T23:03:50.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.502+0000 7f5393a00640 1 -- 172.21.15.28:0/1020472204 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f538c0a6c20 con 0x7f5374004800 2024-02-21T23:03:50.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.503+0000 7f5393a00640 1 -- 172.21.15.28:0/1020472204 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f538c0c6200 con 0x7f5374004800 2024-02-21T23:03:50.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.504+0000 7f5388c00640 1 -- 172.21.15.28:0/1020472204 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5380013400 con 0x7f5374004800 2024-02-21T23:03:50.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.505+0000 7f5388c00640 1 --2- 172.21.15.28:0/1020472204 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f536c06b4d0 0x7f536c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:50.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.505+0000 7f5388c00640 1 -- 172.21.15.28:0/1020472204 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f538008aef0 con 0x7f5374004800 2024-02-21T23:03:50.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.506+0000 7f538be00640 1 --2- 172.21.15.28:0/1020472204 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f536c06b4d0 0x7f536c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:50.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.507+0000 7f538be00640 1 --2- 172.21.15.28:0/1020472204 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f536c06b4d0 0x7f536c06d990 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f5378034000 tx=0x7f5378007000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:50.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.514+0000 7f5388c00640 1 -- 172.21.15.28:0/1020472204 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f538c0c6200 con 0x7f5374004800 2024-02-21T23:03:50.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.753+0000 7f5393a00640 1 -- 172.21.15.28:0/1020472204 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f538c067f30 con 0x7f5374004800 2024-02-21T23:03:50.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.754+0000 7f5388c00640 1 -- 172.21.15.28:0/1020472204 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4934 (secure 0 0 0) 0x7f5380019030 con 0x7f5374004800 2024-02-21T23:03:50.744 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:03:50.744 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4230,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/2559793999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2559793999},{"type":"v1","addr":"172.21.15.28:6834","nonce":2559793999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4232,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3930208177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3930208177},{"type":"v1","addr":"172.21.15.28:6835","nonce":3930208177}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4238,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/2875984988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":2875984988},{"type":"v1","addr":"172.21.15.81:6836","nonce":2875984988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:02:37.812604+0000","modified":"2024-02-21T23:03:06.243463+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4267},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4267":{"gid":4267,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6837/2328151245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2328151245},{"type":"v1","addr":"172.21.15.81:6837","nonce":2328151245}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-02-21T23:03:50.744 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 9 2024-02-21T23:03:50.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.757+0000 7f5393a00640 1 -- 172.21.15.28:0/1020472204 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f536c06b4d0 msgr2=0x7f536c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:50.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.757+0000 7f5393a00640 1 --2- 172.21.15.28:0/1020472204 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f536c06b4d0 0x7f536c06d990 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f5378034000 tx=0x7f5378007000 comp rx=0 tx=0).stop 2024-02-21T23:03:50.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.757+0000 7f5393a00640 1 -- 172.21.15.28:0/1020472204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5374004800 msgr2=0x7f538c1418e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:50.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.757+0000 7f5393a00640 1 --2- 172.21.15.28:0/1020472204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5374004800 0x7f538c1418e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f53800114e0 tx=0x7f53800119a0 comp rx=0 tx=0).stop 2024-02-21T23:03:50.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.757+0000 7f5393a00640 1 -- 172.21.15.28:0/1020472204 shutdown_connections 2024-02-21T23:03:50.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.757+0000 7f5393a00640 1 --2- 172.21.15.28:0/1020472204 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f536c06b4d0 0x7f536c06d990 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:50.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.757+0000 7f5393a00640 1 --2- 172.21.15.28:0/1020472204 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f538c1538c0 0x7f538c0a7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:50.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.757+0000 7f5393a00640 1 --2- 172.21.15.28:0/1020472204 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f538c14b920 0x7f538c0a7420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:50.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.757+0000 7f5393a00640 1 --2- 172.21.15.28:0/1020472204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5374004800 0x7f538c1418e0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:50.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.757+0000 7f5393a00640 1 -- 172.21.15.28:0/1020472204 >> 172.21.15.28:0/1020472204 conn(0x7f538c0c67f0 msgr2=0x7f538c142f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:50.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.757+0000 7f5393a00640 1 -- 172.21.15.28:0/1020472204 shutdown_connections 2024-02-21T23:03:50.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:50.758+0000 7f5393a00640 1 -- 172.21.15.28:0/1020472204 wait complete. 2024-02-21T23:03:50.769 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:03:50.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.003+0000 7f3e50c00640 1 Processor -- start 2024-02-21T23:03:50.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.003+0000 7f3e50c00640 1 -- start start 2024-02-21T23:03:50.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.005+0000 7f3e50c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f3e4c14f0b0 0x7f3e4c155180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:50.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.005+0000 7f3e50c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3e4c0a6160 0x7f3e4c0c2010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:50.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.005+0000 7f3e50c00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e4c155760 con 0x7f3e4c0c2550 2024-02-21T23:03:50.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.005+0000 7f3e50c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e4c151d30 con 0x7f3e4c0a6160 2024-02-21T23:03:50.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.005+0000 7f3e50c00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e4c155a40 con 0x7f3e4c14f0b0 2024-02-21T23:03:50.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.006+0000 7f3e43e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3e4c0a6160 0x7f3e4c0c2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:50.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.006+0000 7f3e43e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3e4c0a6160 0x7f3e4c0c2010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41848/0 (socket says 172.21.15.28:41848) 2024-02-21T23:03:50.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.006+0000 7f3e43e00640 1 -- 172.21.15.28:0/2169945424 learned_addr learned my addr 172.21.15.28:0/2169945424 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:50.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.006+0000 7f3e4b400640 1 --2- 172.21.15.28:0/2169945424 >> v2:172.21.15.81:3300/0 conn(0x7f3e4c14f0b0 0x7f3e4c155180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:50.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.006+0000 7f3e43e00640 1 -- 172.21.15.28:0/2169945424 >> v2:172.21.15.81:3300/0 conn(0x7f3e4c14f0b0 msgr2=0x7f3e4c155180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:50.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.006+0000 7f3e43e00640 1 --2- 172.21.15.28:0/2169945424 >> v2:172.21.15.81:3300/0 conn(0x7f3e4c14f0b0 0x7f3e4c155180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:50.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.006+0000 7f3e43e00640 1 -- 172.21.15.28:0/2169945424 >> v1:172.21.15.81:6789/0 conn(0x7f3e4c0c2550 legacy=0x7f3e4c1521c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:50.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.006+0000 7f3e43e00640 1 -- 172.21.15.28:0/2169945424 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e4c155d20 con 0x7f3e4c0a6160 2024-02-21T23:03:50.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.007+0000 7f3e43e00640 1 --2- 172.21.15.28:0/2169945424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e4c0a6160 0x7f3e4c0c2010 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c151eb0 tx=0x7f3e3402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bad48358c0b9430 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:50.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.008+0000 7f3e4aa00640 1 -- 172.21.15.28:0/2169945424 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e34034070 con 0x7f3e4c0a6160 2024-02-21T23:03:50.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.008+0000 7f3e4aa00640 1 -- 172.21.15.28:0/2169945424 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e34030860 con 0x7f3e4c0a6160 2024-02-21T23:03:50.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.008+0000 7f3e4aa00640 1 -- 172.21.15.28:0/2169945424 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e34030c30 con 0x7f3e4c0a6160 2024-02-21T23:03:50.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.009+0000 7f3e50c00640 1 -- 172.21.15.28:0/2169945424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e4c0a6160 msgr2=0x7f3e4c0c2010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:50.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.009+0000 7f3e50c00640 1 --2- 172.21.15.28:0/2169945424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e4c0a6160 0x7f3e4c0c2010 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c151eb0 tx=0x7f3e3402b9a0 comp rx=0 tx=0).stop 2024-02-21T23:03:50.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.009+0000 7f3e50c00640 1 -- 172.21.15.28:0/2169945424 shutdown_connections 2024-02-21T23:03:50.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.009+0000 7f3e50c00640 1 --2- 172.21.15.28:0/2169945424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e4c0a6160 0x7f3e4c0c2010 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:50.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.009+0000 7f3e50c00640 1 --2- 172.21.15.28:0/2169945424 >> v2:172.21.15.81:3300/0 conn(0x7f3e4c14f0b0 0x7f3e4c155180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:50.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.009+0000 7f3e50c00640 1 -- 172.21.15.28:0/2169945424 >> 172.21.15.28:0/2169945424 conn(0x7f3e4c06b630 msgr2=0x7f3e4c0c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:50.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.009+0000 7f3e50c00640 1 -- 172.21.15.28:0/2169945424 shutdown_connections 2024-02-21T23:03:50.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.009+0000 7f3e50c00640 1 -- 172.21.15.28:0/2169945424 wait complete. 2024-02-21T23:03:51.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.011+0000 7f3e50c00640 1 Processor -- start 2024-02-21T23:03:51.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.011+0000 7f3e50c00640 1 -- start start 2024-02-21T23:03:51.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.012+0000 7f3e50c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3e4c0a6160 0x7f3e4c1e87a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:51.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.012+0000 7f3e50c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e4c0c2550 0x7f3e4c0c2f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:51.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.012+0000 7f3e50c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3e4c14f0b0 0x7f3e4c0c47b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:51.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.012+0000 7f3e50c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4c0a7f70 con 0x7f3e4c0c2550 2024-02-21T23:03:51.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.012+0000 7f3e50c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4c0a68d0 con 0x7f3e4c14f0b0 2024-02-21T23:03:51.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.012+0000 7f3e50c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4c0a80f0 con 0x7f3e4c0a6160 2024-02-21T23:03:51.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.012+0000 7f3e43e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e4c0c2550 0x7f3e4c0c2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:51.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.012+0000 7f3e43e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e4c0c2550 0x7f3e4c0c2f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41852/0 (socket says 172.21.15.28:41852) 2024-02-21T23:03:51.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.012+0000 7f3e43e00640 1 -- 172.21.15.28:0/4108304055 learned_addr learned my addr 172.21.15.28:0/4108304055 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:51.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.012+0000 7f3e4b400640 1 --2- 172.21.15.28:0/4108304055 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3e4c0a6160 0x7f3e4c1e87a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:51.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.012+0000 7f3e4be00640 1 --2- 172.21.15.28:0/4108304055 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3e4c14f0b0 0x7f3e4c0c47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:51.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.012+0000 7f3e43e00640 1 -- 172.21.15.28:0/4108304055 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3e4c0a6160 msgr2=0x7f3e4c1e87a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:51.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.012+0000 7f3e43e00640 1 --2- 172.21.15.28:0/4108304055 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3e4c0a6160 0x7f3e4c1e87a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:51.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.012+0000 7f3e43e00640 1 -- 172.21.15.28:0/4108304055 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3e4c14f0b0 msgr2=0x7f3e4c0c47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:51.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.012+0000 7f3e43e00640 1 --2- 172.21.15.28:0/4108304055 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3e4c14f0b0 0x7f3e4c0c47b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:51.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.012+0000 7f3e43e00640 1 -- 172.21.15.28:0/4108304055 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e4c1e8ce0 con 0x7f3e4c0c2550 2024-02-21T23:03:51.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.012+0000 7f3e4b400640 1 --2- 172.21.15.28:0/4108304055 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3e4c0a6160 0x7f3e4c1e87a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:03:51.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.013+0000 7f3e43e00640 1 --2- 172.21.15.28:0/4108304055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e4c0c2550 0x7f3e4c0c2f00 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f3e3402b9e0 tx=0x7f3e340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:51.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.013+0000 7f3e43400640 1 -- 172.21.15.28:0/4108304055 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e34034040 con 0x7f3e4c0c2550 2024-02-21T23:03:51.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.013+0000 7f3e43400640 1 -- 172.21.15.28:0/4108304055 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e34039d90 con 0x7f3e4c0c2550 2024-02-21T23:03:51.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.013+0000 7f3e43400640 1 -- 172.21.15.28:0/4108304055 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e34043610 con 0x7f3e4c0c2550 2024-02-21T23:03:51.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.013+0000 7f3e50c00640 1 -- 172.21.15.28:0/4108304055 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e4c0c3b40 con 0x7f3e4c0c2550 2024-02-21T23:03:51.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.013+0000 7f3e50c00640 1 -- 172.21.15.28:0/4108304055 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e4c1e9230 con 0x7f3e4c0c2550 2024-02-21T23:03:51.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.014+0000 7f3e50c00640 1 -- 172.21.15.28:0/4108304055 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e4c04f290 con 0x7f3e4c0c2550 2024-02-21T23:03:51.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.015+0000 7f3e43400640 1 -- 172.21.15.28:0/4108304055 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3e34037070 con 0x7f3e4c0c2550 2024-02-21T23:03:51.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.016+0000 7f3e43400640 1 --2- 172.21.15.28:0/4108304055 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3e2806b4d0 0x7f3e2806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:51.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.016+0000 7f3e43400640 1 -- 172.21.15.28:0/4108304055 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f3e340ac660 con 0x7f3e4c0c2550 2024-02-21T23:03:51.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.017+0000 7f3e4b400640 1 --2- 172.21.15.28:0/4108304055 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3e2806b4d0 0x7f3e2806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:51.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.018+0000 7f3e4b400640 1 --2- 172.21.15.28:0/4108304055 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3e2806b4d0 0x7f3e2806d990 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c0c0720 tx=0x7f3e3c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:51.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.021+0000 7f3e43400640 1 -- 172.21.15.28:0/4108304055 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3e4c0c3b40 con 0x7f3e4c0c2550 2024-02-21T23:03:51.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.221+0000 7f3e50c00640 1 -- 172.21.15.28:0/4108304055 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3e4c0c6220 con 0x7f3e4c0c2550 2024-02-21T23:03:51.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.224+0000 7f3e43400640 1 -- 172.21.15.28:0/4108304055 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+18843 (secure 0 0 0) 0x7f3e3403e070 con 0x7f3e4c0c2550 2024-02-21T23:03:51.214 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:03:51.214 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":26,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:03:23.860109+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:02:35.795158+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-21T23:02:36.840110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-02-21T23:02:40.441388+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":3,"to":2}]},{"pgid":"3.12","mappings":[{"from":0,"to":1}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:03:51.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.227+0000 7f3e50c00640 1 -- 172.21.15.28:0/4108304055 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3e2806b4d0 msgr2=0x7f3e2806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:51.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.227+0000 7f3e50c00640 1 --2- 172.21.15.28:0/4108304055 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3e2806b4d0 0x7f3e2806d990 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c0c0720 tx=0x7f3e3c008040 comp rx=0 tx=0).stop 2024-02-21T23:03:51.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.227+0000 7f3e50c00640 1 -- 172.21.15.28:0/4108304055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e4c0c2550 msgr2=0x7f3e4c0c2f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:51.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.227+0000 7f3e50c00640 1 --2- 172.21.15.28:0/4108304055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e4c0c2550 0x7f3e4c0c2f00 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f3e3402b9e0 tx=0x7f3e340046c0 comp rx=0 tx=0).stop 2024-02-21T23:03:51.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.227+0000 7f3e50c00640 1 -- 172.21.15.28:0/4108304055 shutdown_connections 2024-02-21T23:03:51.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.227+0000 7f3e50c00640 1 --2- 172.21.15.28:0/4108304055 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3e2806b4d0 0x7f3e2806d990 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:51.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.227+0000 7f3e50c00640 1 --2- 172.21.15.28:0/4108304055 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3e4c14f0b0 0x7f3e4c0c47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:51.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.227+0000 7f3e50c00640 1 --2- 172.21.15.28:0/4108304055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e4c0c2550 0x7f3e4c0c2f00 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:51.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.227+0000 7f3e50c00640 1 --2- 172.21.15.28:0/4108304055 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3e4c0a6160 0x7f3e4c1e87a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:51.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.227+0000 7f3e50c00640 1 -- 172.21.15.28:0/4108304055 >> 172.21.15.28:0/4108304055 conn(0x7f3e4c06b630 msgr2=0x7f3e4c1521c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:51.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.227+0000 7f3e50c00640 1 -- 172.21.15.28:0/4108304055 shutdown_connections 2024-02-21T23:03:51.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.227+0000 7f3e50c00640 1 -- 172.21.15.28:0/4108304055 wait complete. 2024-02-21T23:03:51.239 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-21T23:02:30.728329+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-21T23:02:35.795158+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-02-21T23:02:36.840110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-02-21T23:02:40.441388+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-02-21T23:03:51.239 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-21T23:03:51.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.462+0000 7fbb84c00640 1 Processor -- start 2024-02-21T23:03:51.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.462+0000 7fbb84c00640 1 -- start start 2024-02-21T23:03:51.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.464+0000 7fbb84c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbb80150240 0x7fbb800c12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:51.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.464+0000 7fbb84c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbb800c4580 0x7fbb8006db10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:51.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.464+0000 7fbb84c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fbb800c2c10 0x7fbb8006ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:51.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.464+0000 7fbb84c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb80150620 con 0x7fbb80150240 2024-02-21T23:03:51.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.464+0000 7fbb84c00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb80153a90 con 0x7fbb800c2c10 2024-02-21T23:03:51.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.464+0000 7fbb84c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb8006f780 con 0x7fbb800c4580 2024-02-21T23:03:51.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.464+0000 7fbb7f400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbb80150240 0x7fbb800c12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:51.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.464+0000 7fbb7f400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbb80150240 0x7fbb800c12c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41856/0 (socket says 172.21.15.28:41856) 2024-02-21T23:03:51.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.464+0000 7fbb7f400640 1 -- 172.21.15.28:0/53896328 learned_addr learned my addr 172.21.15.28:0/53896328 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:51.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.464+0000 7fbb7ea00640 1 --2- 172.21.15.28:0/53896328 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbb800c4580 0x7fbb8006db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:51.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.465+0000 7fbb7fe00640 1 --2- 172.21.15.28:0/53896328 >> v2:172.21.15.81:3300/0 conn(0x7fbb800c2c10 0x7fbb8006ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:51.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.465+0000 7fbb7f400640 1 -- 172.21.15.28:0/53896328 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbb800c4580 msgr2=0x7fbb8006db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:51.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.465+0000 7fbb7f400640 1 --2- 172.21.15.28:0/53896328 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbb800c4580 0x7fbb8006db10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:51.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.465+0000 7fbb7f400640 1 -- 172.21.15.28:0/53896328 >> v2:172.21.15.81:3300/0 conn(0x7fbb800c2c10 msgr2=0x7fbb8006ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:51.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.465+0000 7fbb7f400640 1 --2- 172.21.15.28:0/53896328 >> v2:172.21.15.81:3300/0 conn(0x7fbb800c2c10 0x7fbb8006ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:51.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.465+0000 7fbb7f400640 1 -- 172.21.15.28:0/53896328 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb8006f4a0 con 0x7fbb80150240 2024-02-21T23:03:51.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.465+0000 7fbb7fe00640 1 --2- 172.21.15.28:0/53896328 >> v2:172.21.15.81:3300/0 conn(0x7fbb800c2c10 0x7fbb8006ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:03:51.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.466+0000 7fbb7f400640 1 --2- 172.21.15.28:0/53896328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb80150240 0x7fbb800c12c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fbb800c6500 tx=0x7fbb7002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6109b9b006d668a8 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:51.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.466+0000 7fbb7e000640 1 -- 172.21.15.28:0/53896328 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb7002e070 con 0x7fbb80150240 2024-02-21T23:03:51.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.466+0000 7fbb7e000640 1 -- 172.21.15.28:0/53896328 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb70037070 con 0x7fbb80150240 2024-02-21T23:03:51.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.466+0000 7fbb7e000640 1 -- 172.21.15.28:0/53896328 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb70031a00 con 0x7fbb80150240 2024-02-21T23:03:51.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.467+0000 7fbb84c00640 1 -- 172.21.15.28:0/53896328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb80150240 msgr2=0x7fbb800c12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:51.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.467+0000 7fbb84c00640 1 --2- 172.21.15.28:0/53896328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb80150240 0x7fbb800c12c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fbb800c6500 tx=0x7fbb7002b880 comp rx=0 tx=0).stop 2024-02-21T23:03:51.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.467+0000 7fbb84c00640 1 -- 172.21.15.28:0/53896328 shutdown_connections 2024-02-21T23:03:51.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.467+0000 7fbb84c00640 1 --2- 172.21.15.28:0/53896328 >> v2:172.21.15.81:3300/0 conn(0x7fbb800c2c10 0x7fbb8006ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:51.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.467+0000 7fbb84c00640 1 --2- 172.21.15.28:0/53896328 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbb800c4580 0x7fbb8006db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:51.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.467+0000 7fbb84c00640 1 --2- 172.21.15.28:0/53896328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb80150240 0x7fbb800c12c0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:51.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.467+0000 7fbb84c00640 1 -- 172.21.15.28:0/53896328 >> 172.21.15.28:0/53896328 conn(0x7fbb800a8530 msgr2=0x7fbb800a8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:51.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.467+0000 7fbb84c00640 1 -- 172.21.15.28:0/53896328 shutdown_connections 2024-02-21T23:03:51.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.467+0000 7fbb84c00640 1 -- 172.21.15.28:0/53896328 wait complete. 2024-02-21T23:03:51.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.468+0000 7fbb84c00640 1 Processor -- start 2024-02-21T23:03:51.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.469+0000 7fbb84c00640 1 -- start start 2024-02-21T23:03:51.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.470+0000 7fbb84c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb800c2c10 0x7fbb80152240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:51.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.470+0000 7fbb84c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb800c4580 0x7fbb801ee630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:51.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.470+0000 7fbb84c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbb80150240 0x7fbb801eeb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:51.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.470+0000 7fbb84c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb800c4b90 con 0x7fbb800c4580 2024-02-21T23:03:51.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.470+0000 7fbb84c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb800c4a10 con 0x7fbb800c2c10 2024-02-21T23:03:51.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.470+0000 7fbb84c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb8006e050 con 0x7fbb80150240 2024-02-21T23:03:51.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.470+0000 7fbb7ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb800c4580 0x7fbb801ee630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:51.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.470+0000 7fbb7ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb800c4580 0x7fbb801ee630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41870/0 (socket says 172.21.15.28:41870) 2024-02-21T23:03:51.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.470+0000 7fbb7ea00640 1 -- 172.21.15.28:0/379929440 learned_addr learned my addr 172.21.15.28:0/379929440 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:51.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.470+0000 7fbb7f400640 1 --2- 172.21.15.28:0/379929440 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb800c2c10 0x7fbb80152240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:51.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.470+0000 7fbb7ea00640 1 -- 172.21.15.28:0/379929440 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbb80150240 msgr2=0x7fbb801eeb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:51.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.470+0000 7fbb7ea00640 1 --2- 172.21.15.28:0/379929440 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbb80150240 0x7fbb801eeb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:51.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.470+0000 7fbb7ea00640 1 -- 172.21.15.28:0/379929440 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb800c2c10 msgr2=0x7fbb80152240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:51.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.470+0000 7fbb7ea00640 1 --2- 172.21.15.28:0/379929440 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb800c2c10 0x7fbb80152240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:51.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.470+0000 7fbb7ea00640 1 -- 172.21.15.28:0/379929440 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb800c51d0 con 0x7fbb800c4580 2024-02-21T23:03:51.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.471+0000 7fbb7ea00640 1 --2- 172.21.15.28:0/379929440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb800c4580 0x7fbb801ee630 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c00d5f0 tx=0x7fbb6c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:51.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.471+0000 7fbb77400640 1 -- 172.21.15.28:0/379929440 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb6c0125f0 con 0x7fbb800c4580 2024-02-21T23:03:51.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.471+0000 7fbb77400640 1 -- 172.21.15.28:0/379929440 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb6c019070 con 0x7fbb800c4580 2024-02-21T23:03:51.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.471+0000 7fbb84c00640 1 -- 172.21.15.28:0/379929440 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb801547f0 con 0x7fbb800c4580 2024-02-21T23:03:51.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.471+0000 7fbb77400640 1 -- 172.21.15.28:0/379929440 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb6c0129d0 con 0x7fbb800c4580 2024-02-21T23:03:51.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.471+0000 7fbb84c00640 1 -- 172.21.15.28:0/379929440 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb80154db0 con 0x7fbb800c4580 2024-02-21T23:03:51.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.472+0000 7fbb84c00640 1 -- 172.21.15.28:0/379929440 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb8004f290 con 0x7fbb800c4580 2024-02-21T23:03:51.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.473+0000 7fbb77400640 1 -- 172.21.15.28:0/379929440 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbb6c012b70 con 0x7fbb800c4580 2024-02-21T23:03:51.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.474+0000 7fbb77400640 1 --2- 172.21.15.28:0/379929440 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbb6006b4d0 0x7fbb6006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:51.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.475+0000 7fbb77400640 1 -- 172.21.15.28:0/379929440 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6735+0+0 (secure 0 0 0) 0x7fbb6c08a9f0 con 0x7fbb800c4580 2024-02-21T23:03:51.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.475+0000 7fbb7f400640 1 --2- 172.21.15.28:0/379929440 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbb6006b4d0 0x7fbb6006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:51.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.476+0000 7fbb7f400640 1 --2- 172.21.15.28:0/379929440 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbb6006b4d0 0x7fbb6006d990 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fbb70036000 tx=0x7fbb70008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:51.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.479+0000 7fbb77400640 1 -- 172.21.15.28:0/379929440 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fbb6c0593d0 con 0x7fbb800c4580 2024-02-21T23:03:51.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.671+0000 7fbb84c00640 1 -- 172.21.15.28:0/379929440 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbb800ac630 con 0x7fbb800c4580 2024-02-21T23:03:51.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.672+0000 7fbb77400640 1 -- 172.21.15.28:0/379929440 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbb6c05f0d0 con 0x7fbb800c4580 2024-02-21T23:03:51.662 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-02-21T23:03:51.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.674+0000 7fbb84c00640 1 -- 172.21.15.28:0/379929440 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbb6006b4d0 msgr2=0x7fbb6006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:51.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.674+0000 7fbb84c00640 1 --2- 172.21.15.28:0/379929440 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbb6006b4d0 0x7fbb6006d990 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fbb70036000 tx=0x7fbb70008030 comp rx=0 tx=0).stop 2024-02-21T23:03:51.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.674+0000 7fbb84c00640 1 -- 172.21.15.28:0/379929440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb800c4580 msgr2=0x7fbb801ee630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:51.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.674+0000 7fbb84c00640 1 --2- 172.21.15.28:0/379929440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb800c4580 0x7fbb801ee630 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c00d5f0 tx=0x7fbb6c00dab0 comp rx=0 tx=0).stop 2024-02-21T23:03:51.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.675+0000 7fbb84c00640 1 -- 172.21.15.28:0/379929440 shutdown_connections 2024-02-21T23:03:51.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.675+0000 7fbb84c00640 1 --2- 172.21.15.28:0/379929440 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbb6006b4d0 0x7fbb6006d990 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:51.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.675+0000 7fbb84c00640 1 --2- 172.21.15.28:0/379929440 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbb80150240 0x7fbb801eeb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:51.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.675+0000 7fbb84c00640 1 --2- 172.21.15.28:0/379929440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb800c4580 0x7fbb801ee630 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:51.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.675+0000 7fbb84c00640 1 --2- 172.21.15.28:0/379929440 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb800c2c10 0x7fbb80152240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:51.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.675+0000 7fbb84c00640 1 -- 172.21.15.28:0/379929440 >> 172.21.15.28:0/379929440 conn(0x7fbb800a8530 msgr2=0x7fbb8006ae70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:51.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.675+0000 7fbb84c00640 1 -- 172.21.15.28:0/379929440 shutdown_connections 2024-02-21T23:03:51.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.675+0000 7fbb84c00640 1 -- 172.21.15.28:0/379929440 wait complete. 2024-02-21T23:03:51.683 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-21T23:03:51.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.914+0000 7f3822600640 1 Processor -- start 2024-02-21T23:03:51.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.914+0000 7f3822600640 1 -- start start 2024-02-21T23:03:51.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.915+0000 7f3822600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f381c06ae70 0x7f381c06b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:51.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.916+0000 7f3822600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f381c0c4c10 0x7f381c159540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:51.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.916+0000 7f3822600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f381c0c0d40 con 0x7f381c159a80 2024-02-21T23:03:51.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.916+0000 7f3822600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f381c15caf0 con 0x7f381c0c4c10 2024-02-21T23:03:51.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.916+0000 7f3822600640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f381c15cdd0 con 0x7f381c06ae70 2024-02-21T23:03:51.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.916+0000 7f381be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f381c0c4c10 0x7f381c159540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:51.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.916+0000 7f3821c00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f381c159a80 0x7f381c06db10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:47022/0 (socket says 172.21.15.28:47022) 2024-02-21T23:03:51.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.916+0000 7f3821c00640 1 -- 172.21.15.28:0/1793532376 learned_addr learned my addr 172.21.15.28:0/1793532376 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:03:51.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.916+0000 7f3821200640 1 --2- 172.21.15.28:0/1793532376 >> v2:172.21.15.81:3300/0 conn(0x7f381c06ae70 0x7f381c06b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:51.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.916+0000 7f381be00640 1 -- 172.21.15.28:0/1793532376 >> v2:172.21.15.81:3300/0 conn(0x7f381c06ae70 msgr2=0x7f381c06b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:51.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.916+0000 7f381be00640 1 --2- 172.21.15.28:0/1793532376 >> v2:172.21.15.81:3300/0 conn(0x7f381c06ae70 0x7f381c06b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:51.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.916+0000 7f381be00640 1 -- 172.21.15.28:0/1793532376 >> v1:172.21.15.28:6789/0 conn(0x7f381c159a80 legacy=0x7f381c06db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:51.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.917+0000 7f381b400640 1 -- 172.21.15.28:0/1793532376 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2770989149 0 0) 0x7f381c0c0d40 con 0x7f381c159a80 2024-02-21T23:03:51.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.917+0000 7f381be00640 1 -- 172.21.15.28:0/1793532376 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f381c15d0b0 con 0x7f381c0c4c10 2024-02-21T23:03:51.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.918+0000 7f381be00640 1 --2- 172.21.15.28:0/1793532376 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f381c0c4c10 0x7f381c159540 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f381c06ed60 tx=0x7f380802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f0eec92bd600d6a server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:51.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.918+0000 7f381b400640 1 -- 172.21.15.28:0/1793532376 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38080315a0 con 0x7f381c0c4c10 2024-02-21T23:03:51.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.918+0000 7f381b400640 1 -- 172.21.15.28:0/1793532376 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3808037070 con 0x7f381c0c4c10 2024-02-21T23:03:51.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.918+0000 7f381b400640 1 -- 172.21.15.28:0/1793532376 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3808031960 con 0x7f381c0c4c10 2024-02-21T23:03:51.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.918+0000 7f3822600640 1 -- 172.21.15.28:0/1793532376 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f381c0c4c10 msgr2=0x7f381c159540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:51.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.918+0000 7f3822600640 1 --2- 172.21.15.28:0/1793532376 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f381c0c4c10 0x7f381c159540 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f381c06ed60 tx=0x7f380802b840 comp rx=0 tx=0).stop 2024-02-21T23:03:51.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.919+0000 7f3822600640 1 -- 172.21.15.28:0/1793532376 shutdown_connections 2024-02-21T23:03:51.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.919+0000 7f3822600640 1 --2- 172.21.15.28:0/1793532376 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f381c0c4c10 0x7f381c159540 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:51.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.919+0000 7f3822600640 1 --2- 172.21.15.28:0/1793532376 >> v2:172.21.15.81:3300/0 conn(0x7f381c06ae70 0x7f381c06b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:51.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.919+0000 7f3822600640 1 -- 172.21.15.28:0/1793532376 >> 172.21.15.28:0/1793532376 conn(0x7f381c0a8530 msgr2=0x7f381c0a8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:51.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.919+0000 7f3822600640 1 -- 172.21.15.28:0/1793532376 shutdown_connections 2024-02-21T23:03:51.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.919+0000 7f3822600640 1 -- 172.21.15.28:0/1793532376 wait complete. 2024-02-21T23:03:51.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.920+0000 7f3822600640 1 Processor -- start 2024-02-21T23:03:51.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.921+0000 7f3822600640 1 -- start start 2024-02-21T23:03:51.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.921+0000 7f3822600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f381c06ae70 0x7f381c0ac540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:51.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.921+0000 7f3822600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f381c0c4c10 0x7f381c0aaa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:51.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.921+0000 7f3822600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f381c159a80 0x7f381c151730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:51.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.921+0000 7f3822600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f381c15c7e0 con 0x7f381c0c4c10 2024-02-21T23:03:51.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.921+0000 7f3822600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f381c15d670 con 0x7f381c06ae70 2024-02-21T23:03:51.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.921+0000 7f3822600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f381c06f6b0 con 0x7f381c159a80 2024-02-21T23:03:51.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.922+0000 7f381be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f381c0c4c10 0x7f381c0aaa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:51.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.922+0000 7f381be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f381c0c4c10 0x7f381c0aaa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41888/0 (socket says 172.21.15.28:41888) 2024-02-21T23:03:51.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.922+0000 7f381be00640 1 -- 172.21.15.28:0/3517657072 learned_addr learned my addr 172.21.15.28:0/3517657072 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:51.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.922+0000 7f3821200640 1 --2- 172.21.15.28:0/3517657072 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f381c06ae70 0x7f381c0ac540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:51.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.922+0000 7f3821c00640 1 --2- 172.21.15.28:0/3517657072 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f381c159a80 0x7f381c151730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:51.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.922+0000 7f381be00640 1 -- 172.21.15.28:0/3517657072 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f381c159a80 msgr2=0x7f381c151730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:51.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.922+0000 7f381be00640 1 --2- 172.21.15.28:0/3517657072 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f381c159a80 0x7f381c151730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:51.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.922+0000 7f381be00640 1 -- 172.21.15.28:0/3517657072 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f381c06ae70 msgr2=0x7f381c0ac540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:51.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.922+0000 7f381be00640 1 --2- 172.21.15.28:0/3517657072 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f381c06ae70 0x7f381c0ac540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:51.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.922+0000 7f381be00640 1 -- 172.21.15.28:0/3517657072 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f381c1f2e10 con 0x7f381c0c4c10 2024-02-21T23:03:51.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.922+0000 7f381be00640 1 --2- 172.21.15.28:0/3517657072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f381c0c4c10 0x7f381c0aaa30 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f3808000c00 tx=0x7f38080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:51.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.923+0000 7f3818c00640 1 -- 172.21.15.28:0/3517657072 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3808032840 con 0x7f381c0c4c10 2024-02-21T23:03:51.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.923+0000 7f3818c00640 1 -- 172.21.15.28:0/3517657072 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3808032a00 con 0x7f381c0c4c10 2024-02-21T23:03:51.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.923+0000 7f3818c00640 1 -- 172.21.15.28:0/3517657072 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3808032df0 con 0x7f381c0c4c10 2024-02-21T23:03:51.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.923+0000 7f3822600640 1 -- 172.21.15.28:0/3517657072 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f381c153b00 con 0x7f381c0c4c10 2024-02-21T23:03:51.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.923+0000 7f3822600640 1 -- 172.21.15.28:0/3517657072 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f381c153de0 con 0x7f381c0c4c10 2024-02-21T23:03:51.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.924+0000 7f3822600640 1 -- 172.21.15.28:0/3517657072 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f381c04f290 con 0x7f381c0c4c10 2024-02-21T23:03:51.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.925+0000 7f3818c00640 1 -- 172.21.15.28:0/3517657072 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f380804c020 con 0x7f381c0c4c10 2024-02-21T23:03:51.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.926+0000 7f3818c00640 1 --2- 172.21.15.28:0/3517657072 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f37fc06b4d0 0x7f37fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:51.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.926+0000 7f3818c00640 1 -- 172.21.15.28:0/3517657072 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f380802f020 con 0x7f381c0c4c10 2024-02-21T23:03:51.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.926+0000 7f3821200640 1 --2- 172.21.15.28:0/3517657072 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f37fc06b4d0 0x7f37fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:51.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.928+0000 7f3821200640 1 --2- 172.21.15.28:0/3517657072 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f37fc06b4d0 0x7f37fc06d990 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f3810006fd0 tx=0x7f3810008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:51.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:51.931+0000 7f3818c00640 1 -- 172.21.15.28:0/3517657072 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3808040070 con 0x7f381c0c4c10 2024-02-21T23:03:52.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.133+0000 7f3822600640 1 -- 172.21.15.28:0/3517657072 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f381c06bcd0 con 0x7f381c0c4c10 2024-02-21T23:03:52.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.134+0000 7f3818c00640 1 -- 172.21.15.28:0/3517657072 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+11 (secure 0 0 0) 0x7f380807af80 con 0x7f381c0c4c10 2024-02-21T23:03:52.124 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-02-21T23:03:52.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.137+0000 7f3822600640 1 -- 172.21.15.28:0/3517657072 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f37fc06b4d0 msgr2=0x7f37fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:52.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.137+0000 7f3822600640 1 --2- 172.21.15.28:0/3517657072 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f37fc06b4d0 0x7f37fc06d990 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f3810006fd0 tx=0x7f3810008040 comp rx=0 tx=0).stop 2024-02-21T23:03:52.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.137+0000 7f3822600640 1 -- 172.21.15.28:0/3517657072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f381c0c4c10 msgr2=0x7f381c0aaa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:52.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.137+0000 7f3822600640 1 --2- 172.21.15.28:0/3517657072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f381c0c4c10 0x7f381c0aaa30 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f3808000c00 tx=0x7f38080046c0 comp rx=0 tx=0).stop 2024-02-21T23:03:52.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.137+0000 7f3822600640 1 -- 172.21.15.28:0/3517657072 shutdown_connections 2024-02-21T23:03:52.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.137+0000 7f3822600640 1 --2- 172.21.15.28:0/3517657072 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f37fc06b4d0 0x7f37fc06d990 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:52.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.137+0000 7f3822600640 1 --2- 172.21.15.28:0/3517657072 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f381c159a80 0x7f381c151730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:52.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.137+0000 7f3822600640 1 --2- 172.21.15.28:0/3517657072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f381c0c4c10 0x7f381c0aaa30 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:52.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.137+0000 7f3822600640 1 --2- 172.21.15.28:0/3517657072 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f381c06ae70 0x7f381c0ac540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:52.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.137+0000 7f3822600640 1 -- 172.21.15.28:0/3517657072 >> 172.21.15.28:0/3517657072 conn(0x7f381c0a8530 msgr2=0x7f381c06b250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:52.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.137+0000 7f3822600640 1 -- 172.21.15.28:0/3517657072 shutdown_connections 2024-02-21T23:03:52.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.137+0000 7f3822600640 1 -- 172.21.15.28:0/3517657072 wait complete. 2024-02-21T23:03:52.144 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-21T23:03:52.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.360+0000 7fed20200640 1 Processor -- start 2024-02-21T23:03:52.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.361+0000 7fed20200640 1 -- start start 2024-02-21T23:03:52.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.362+0000 7fed20200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fed180c75d0 0x7fed180af680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:52.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.362+0000 7fed20200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fed180c4d60 0x7fed1806cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:52.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.362+0000 7fed20200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed18155230 con 0x7fed180bfbf0 2024-02-21T23:03:52.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.362+0000 7fed20200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed180c6960 con 0x7fed180c4d60 2024-02-21T23:03:52.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.362+0000 7fed20200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed180c6ca0 con 0x7fed180c75d0 2024-02-21T23:03:52.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.362+0000 7fed1ce00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fed180c4d60 0x7fed1806cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:52.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.362+0000 7fed1ce00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fed180c4d60 0x7fed1806cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41892/0 (socket says 172.21.15.28:41892) 2024-02-21T23:03:52.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.362+0000 7fed1ce00640 1 -- 172.21.15.28:0/3373464985 learned_addr learned my addr 172.21.15.28:0/3373464985 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:52.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.362+0000 7fed1d800640 1 --2- 172.21.15.28:0/3373464985 >> v2:172.21.15.81:3300/0 conn(0x7fed180c75d0 0x7fed180af680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:52.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.363+0000 7fed1ce00640 1 -- 172.21.15.28:0/3373464985 >> v2:172.21.15.81:3300/0 conn(0x7fed180c75d0 msgr2=0x7fed180af680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:52.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.363+0000 7fed1ce00640 1 --2- 172.21.15.28:0/3373464985 >> v2:172.21.15.81:3300/0 conn(0x7fed180c75d0 0x7fed180af680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:52.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.363+0000 7fed1ce00640 1 -- 172.21.15.28:0/3373464985 >> v1:172.21.15.81:6789/0 conn(0x7fed180bfbf0 legacy=0x7fed181587a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:52.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.363+0000 7fed1ce00640 1 -- 172.21.15.28:0/3373464985 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed180c6f80 con 0x7fed180c4d60 2024-02-21T23:03:52.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.364+0000 7fed1ce00640 1 --2- 172.21.15.28:0/3373464985 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed180c4d60 0x7fed1806cd50 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fed180c8150 tx=0x7fed0402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8fa1e3f95ca9b7f server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:52.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.365+0000 7fed17e00640 1 -- 172.21.15.28:0/3373464985 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed04035020 con 0x7fed180c4d60 2024-02-21T23:03:52.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.365+0000 7fed17e00640 1 -- 172.21.15.28:0/3373464985 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed04031870 con 0x7fed180c4d60 2024-02-21T23:03:52.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.365+0000 7fed17e00640 1 -- 172.21.15.28:0/3373464985 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed04031c40 con 0x7fed180c4d60 2024-02-21T23:03:52.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.365+0000 7fed20200640 1 -- 172.21.15.28:0/3373464985 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed180c4d60 msgr2=0x7fed1806cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:52.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.365+0000 7fed20200640 1 --2- 172.21.15.28:0/3373464985 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed180c4d60 0x7fed1806cd50 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fed180c8150 tx=0x7fed0402b910 comp rx=0 tx=0).stop 2024-02-21T23:03:52.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.366+0000 7fed20200640 1 -- 172.21.15.28:0/3373464985 shutdown_connections 2024-02-21T23:03:52.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.366+0000 7fed20200640 1 --2- 172.21.15.28:0/3373464985 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed180c4d60 0x7fed1806cd50 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:52.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.366+0000 7fed20200640 1 --2- 172.21.15.28:0/3373464985 >> v2:172.21.15.81:3300/0 conn(0x7fed180c75d0 0x7fed180af680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:52.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.366+0000 7fed20200640 1 -- 172.21.15.28:0/3373464985 >> 172.21.15.28:0/3373464985 conn(0x7fed180725a0 msgr2=0x7fed180729b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:52.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.366+0000 7fed20200640 1 -- 172.21.15.28:0/3373464985 shutdown_connections 2024-02-21T23:03:52.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.366+0000 7fed20200640 1 -- 172.21.15.28:0/3373464985 wait complete. 2024-02-21T23:03:52.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.367+0000 7fed20200640 1 Processor -- start 2024-02-21T23:03:52.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.367+0000 7fed20200640 1 -- start start 2024-02-21T23:03:52.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.368+0000 7fed20200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fed180bfbf0 0x7fed1814b150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:52.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.368+0000 7fed20200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed180c4d60 0x7fed180c9fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:52.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.368+0000 7fed20200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed180c75d0 0x7fed18157e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:52.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.368+0000 7fed20200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed18155a70 con 0x7fed180c75d0 2024-02-21T23:03:52.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.368+0000 7fed20200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed18155790 con 0x7fed180c4d60 2024-02-21T23:03:52.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.368+0000 7fed20200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed18155f90 con 0x7fed180bfbf0 2024-02-21T23:03:52.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.369+0000 7fed1e200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed180c75d0 0x7fed18157e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:52.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.369+0000 7fed1e200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed180c75d0 0x7fed18157e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41902/0 (socket says 172.21.15.28:41902) 2024-02-21T23:03:52.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.369+0000 7fed1e200640 1 -- 172.21.15.28:0/1749990399 learned_addr learned my addr 172.21.15.28:0/1749990399 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:52.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.369+0000 7fed1ce00640 1 --2- 172.21.15.28:0/1749990399 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed180c4d60 0x7fed180c9fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:52.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.369+0000 7fed1d800640 1 --2- 172.21.15.28:0/1749990399 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fed180bfbf0 0x7fed1814b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:52.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.369+0000 7fed1e200640 1 -- 172.21.15.28:0/1749990399 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fed180bfbf0 msgr2=0x7fed1814b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:52.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.369+0000 7fed1e200640 1 --2- 172.21.15.28:0/1749990399 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fed180bfbf0 0x7fed1814b150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:52.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.369+0000 7fed1e200640 1 -- 172.21.15.28:0/1749990399 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed180c4d60 msgr2=0x7fed180c9fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:52.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.369+0000 7fed1e200640 1 --2- 172.21.15.28:0/1749990399 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed180c4d60 0x7fed180c9fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:52.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.369+0000 7fed1e200640 1 -- 172.21.15.28:0/1749990399 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed181ec030 con 0x7fed180c75d0 2024-02-21T23:03:52.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.370+0000 7fed1e200640 1 --2- 172.21.15.28:0/1749990399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed180c75d0 0x7fed18157e60 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fed1000c7a0 tx=0x7fed1000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:52.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.370+0000 7fed15600640 1 -- 172.21.15.28:0/1749990399 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed10003b70 con 0x7fed180c75d0 2024-02-21T23:03:52.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.370+0000 7fed15600640 1 -- 172.21.15.28:0/1749990399 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed10003d30 con 0x7fed180c75d0 2024-02-21T23:03:52.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.370+0000 7fed15600640 1 -- 172.21.15.28:0/1749990399 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed1000aa90 con 0x7fed180c75d0 2024-02-21T23:03:52.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.370+0000 7fed20200640 1 -- 172.21.15.28:0/1749990399 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed18159140 con 0x7fed180c75d0 2024-02-21T23:03:52.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.370+0000 7fed20200640 1 -- 172.21.15.28:0/1749990399 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed181ebd50 con 0x7fed180c75d0 2024-02-21T23:03:52.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.371+0000 7fed20200640 1 -- 172.21.15.28:0/1749990399 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed180c79b0 con 0x7fed180c75d0 2024-02-21T23:03:52.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.372+0000 7fed15600640 1 -- 172.21.15.28:0/1749990399 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fed1000ac30 con 0x7fed180c75d0 2024-02-21T23:03:52.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.373+0000 7fed15600640 1 --2- 172.21.15.28:0/1749990399 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fecf806b500 0x7fecf806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:52.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.373+0000 7fed15600640 1 -- 172.21.15.28:0/1749990399 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6735+0+0 (secure 0 0 0) 0x7fed1008a540 con 0x7fed180c75d0 2024-02-21T23:03:52.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.374+0000 7fed1d800640 1 --2- 172.21.15.28:0/1749990399 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fecf806b500 0x7fecf806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:52.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.375+0000 7fed1d800640 1 --2- 172.21.15.28:0/1749990399 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fecf806b500 0x7fecf806d9c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fed0c009a30 tx=0x7fed0c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:52.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.378+0000 7fed15600640 1 -- 172.21.15.28:0/1749990399 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fed180c79b0 con 0x7fed180c75d0 2024-02-21T23:03:52.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.577+0000 7fed20200640 1 -- 172.21.15.28:0/1749990399 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fed18068640 con 0x7fed180c75d0 2024-02-21T23:03:52.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.577+0000 7fed15600640 1 -- 172.21.15.28:0/1749990399 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+11 (secure 0 0 0) 0x7fed18155230 con 0x7fed180c75d0 2024-02-21T23:03:52.568 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-02-21T23:03:52.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.580+0000 7fed20200640 1 -- 172.21.15.28:0/1749990399 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fecf806b500 msgr2=0x7fecf806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:52.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.580+0000 7fed20200640 1 --2- 172.21.15.28:0/1749990399 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fecf806b500 0x7fecf806d9c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fed0c009a30 tx=0x7fed0c008040 comp rx=0 tx=0).stop 2024-02-21T23:03:52.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.580+0000 7fed20200640 1 -- 172.21.15.28:0/1749990399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed180c75d0 msgr2=0x7fed18157e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:52.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.580+0000 7fed20200640 1 --2- 172.21.15.28:0/1749990399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed180c75d0 0x7fed18157e60 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fed1000c7a0 tx=0x7fed1000cc60 comp rx=0 tx=0).stop 2024-02-21T23:03:52.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.580+0000 7fed20200640 1 -- 172.21.15.28:0/1749990399 shutdown_connections 2024-02-21T23:03:52.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.580+0000 7fed20200640 1 --2- 172.21.15.28:0/1749990399 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fecf806b500 0x7fecf806d9c0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:52.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.580+0000 7fed20200640 1 --2- 172.21.15.28:0/1749990399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed180c75d0 0x7fed18157e60 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:52.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.580+0000 7fed20200640 1 --2- 172.21.15.28:0/1749990399 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed180c4d60 0x7fed180c9fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:52.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.580+0000 7fed20200640 1 --2- 172.21.15.28:0/1749990399 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fed180bfbf0 0x7fed1814b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:52.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.580+0000 7fed20200640 1 -- 172.21.15.28:0/1749990399 >> 172.21.15.28:0/1749990399 conn(0x7fed180725a0 msgr2=0x7fed180af450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:52.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.580+0000 7fed20200640 1 -- 172.21.15.28:0/1749990399 shutdown_connections 2024-02-21T23:03:52.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.580+0000 7fed20200640 1 -- 172.21.15.28:0/1749990399 wait complete. 2024-02-21T23:03:52.585 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-02-21T23:03:52.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.787+0000 7fe9b7000640 1 Processor -- start 2024-02-21T23:03:52.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.787+0000 7fe9b7000640 1 -- start start 2024-02-21T23:03:52.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.788+0000 7fe9b7000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe9b0150240 0x7fe9b00c12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:52.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.788+0000 7fe9b7000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe9b00c4580 0x7fe9b006db10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:52.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.788+0000 7fe9b7000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fe9b00c2c10 0x7fe9b006ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:52.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.788+0000 7fe9b7000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9b0150620 con 0x7fe9b00c4580 2024-02-21T23:03:52.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.788+0000 7fe9b7000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9b0153a90 con 0x7fe9b00c2c10 2024-02-21T23:03:52.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.788+0000 7fe9b7000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b006f780 con 0x7fe9b0150240 2024-02-21T23:03:52.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.789+0000 7fe9b5200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe9b00c4580 0x7fe9b006db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:52.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.789+0000 7fe9b5200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe9b00c4580 0x7fe9b006db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41912/0 (socket says 172.21.15.28:41912) 2024-02-21T23:03:52.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.789+0000 7fe9b5200640 1 -- 172.21.15.28:0/3438638033 learned_addr learned my addr 172.21.15.28:0/3438638033 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:52.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.789+0000 7fe9b5c00640 1 --2- 172.21.15.28:0/3438638033 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe9b0150240 0x7fe9b00c12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:52.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.789+0000 7fe9b5200640 1 -- 172.21.15.28:0/3438638033 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe9b0150240 msgr2=0x7fe9b00c12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:52.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.789+0000 7fe9b5200640 1 --2- 172.21.15.28:0/3438638033 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe9b0150240 0x7fe9b00c12c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:52.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.789+0000 7fe9b6600640 1 --2- 172.21.15.28:0/3438638033 >> v2:172.21.15.81:3300/0 conn(0x7fe9b00c2c10 0x7fe9b006ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:52.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.789+0000 7fe9b5200640 1 -- 172.21.15.28:0/3438638033 >> v2:172.21.15.81:3300/0 conn(0x7fe9b00c2c10 msgr2=0x7fe9b006ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:52.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.789+0000 7fe9b5200640 1 --2- 172.21.15.28:0/3438638033 >> v2:172.21.15.81:3300/0 conn(0x7fe9b00c2c10 0x7fe9b006ae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:52.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.789+0000 7fe9b5200640 1 -- 172.21.15.28:0/3438638033 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9b006f4a0 con 0x7fe9b00c4580 2024-02-21T23:03:52.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.790+0000 7fe9b5200640 1 --2- 172.21.15.28:0/3438638033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9b00c4580 0x7fe9b006db10 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fe9b00c6500 tx=0x7fe99c02b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36b1bb2242237ae0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:52.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.790+0000 7fe9afe00640 1 -- 172.21.15.28:0/3438638033 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe99c0317d0 con 0x7fe9b00c4580 2024-02-21T23:03:52.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.790+0000 7fe9afe00640 1 -- 172.21.15.28:0/3438638033 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe99c037070 con 0x7fe9b00c4580 2024-02-21T23:03:52.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.790+0000 7fe9afe00640 1 -- 172.21.15.28:0/3438638033 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe99c031b90 con 0x7fe9b00c4580 2024-02-21T23:03:52.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.791+0000 7fe9b7000640 1 -- 172.21.15.28:0/3438638033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9b00c4580 msgr2=0x7fe9b006db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:52.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.791+0000 7fe9b7000640 1 --2- 172.21.15.28:0/3438638033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9b00c4580 0x7fe9b006db10 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fe9b00c6500 tx=0x7fe99c02b870 comp rx=0 tx=0).stop 2024-02-21T23:03:52.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.791+0000 7fe9b7000640 1 -- 172.21.15.28:0/3438638033 shutdown_connections 2024-02-21T23:03:52.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.791+0000 7fe9b7000640 1 --2- 172.21.15.28:0/3438638033 >> v2:172.21.15.81:3300/0 conn(0x7fe9b00c2c10 0x7fe9b006ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:52.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.791+0000 7fe9b7000640 1 --2- 172.21.15.28:0/3438638033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9b00c4580 0x7fe9b006db10 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:52.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.791+0000 7fe9b7000640 1 --2- 172.21.15.28:0/3438638033 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe9b0150240 0x7fe9b00c12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:52.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.791+0000 7fe9b7000640 1 -- 172.21.15.28:0/3438638033 >> 172.21.15.28:0/3438638033 conn(0x7fe9b00a8530 msgr2=0x7fe9b00a8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:52.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.791+0000 7fe9b7000640 1 -- 172.21.15.28:0/3438638033 shutdown_connections 2024-02-21T23:03:52.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.791+0000 7fe9b7000640 1 -- 172.21.15.28:0/3438638033 wait complete. 2024-02-21T23:03:52.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.793+0000 7fe9b7000640 1 Processor -- start 2024-02-21T23:03:52.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.793+0000 7fe9b7000640 1 -- start start 2024-02-21T23:03:52.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.794+0000 7fe9b7000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe9b00c2c10 0x7fe9b00aa010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:52.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.794+0000 7fe9b7000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9b00c4580 0x7fe9b00aa960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:52.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.794+0000 7fe9b7000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9b0150240 0x7fe9b01e8be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:52.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.794+0000 7fe9b7000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b00c4b90 con 0x7fe9b0150240 2024-02-21T23:03:52.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.794+0000 7fe9b7000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b00c4a10 con 0x7fe9b00c4580 2024-02-21T23:03:52.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.794+0000 7fe9b7000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b006e050 con 0x7fe9b00c2c10 2024-02-21T23:03:52.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.794+0000 7fe9b6600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9b0150240 0x7fe9b01e8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:52.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.794+0000 7fe9b6600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9b0150240 0x7fe9b01e8be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41918/0 (socket says 172.21.15.28:41918) 2024-02-21T23:03:52.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.794+0000 7fe9b6600640 1 -- 172.21.15.28:0/3323924682 learned_addr learned my addr 172.21.15.28:0/3323924682 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:52.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.795+0000 7fe9b6600640 1 -- 172.21.15.28:0/3323924682 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe9b00c2c10 msgr2=0x7fe9b00aa010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:52.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.795+0000 7fe9b6600640 1 --2- 172.21.15.28:0/3323924682 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe9b00c2c10 0x7fe9b00aa010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:52.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.795+0000 7fe9b6600640 1 -- 172.21.15.28:0/3323924682 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9b00c4580 msgr2=0x7fe9b00aa960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:52.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.795+0000 7fe9b6600640 1 --2- 172.21.15.28:0/3323924682 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9b00c4580 0x7fe9b00aa960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:52.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.795+0000 7fe9b6600640 1 -- 172.21.15.28:0/3323924682 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9b0151d70 con 0x7fe9b0150240 2024-02-21T23:03:52.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.795+0000 7fe9b6600640 1 --2- 172.21.15.28:0/3323924682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9b0150240 0x7fe9b01e8be0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fe9a800cc30 tx=0x7fe9a800a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:52.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.795+0000 7fe9ad600640 1 -- 172.21.15.28:0/3323924682 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a800ce40 con 0x7fe9b0150240 2024-02-21T23:03:52.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.795+0000 7fe9ad600640 1 -- 172.21.15.28:0/3323924682 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9a8003990 con 0x7fe9b0150240 2024-02-21T23:03:52.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.795+0000 7fe9b7000640 1 -- 172.21.15.28:0/3323924682 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9b01525a0 con 0x7fe9b0150240 2024-02-21T23:03:52.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.795+0000 7fe9ad600640 1 -- 172.21.15.28:0/3323924682 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a8003d40 con 0x7fe9b0150240 2024-02-21T23:03:52.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.795+0000 7fe9b7000640 1 -- 172.21.15.28:0/3323924682 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9b01522c0 con 0x7fe9b0150240 2024-02-21T23:03:52.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.797+0000 7fe9b7000640 1 -- 172.21.15.28:0/3323924682 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9b004f290 con 0x7fe9b0150240 2024-02-21T23:03:52.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.798+0000 7fe9ad600640 1 -- 172.21.15.28:0/3323924682 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe9a8003ee0 con 0x7fe9b0150240 2024-02-21T23:03:52.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.799+0000 7fe9ad600640 1 --2- 172.21.15.28:0/3323924682 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe99006b4d0 0x7fe99006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:52.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.799+0000 7fe9ad600640 1 -- 172.21.15.28:0/3323924682 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6735+0+0 (secure 0 0 0) 0x7fe9a808ab40 con 0x7fe9b0150240 2024-02-21T23:03:52.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.799+0000 7fe9b5c00640 1 --2- 172.21.15.28:0/3323924682 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe99006b4d0 0x7fe99006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:52.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.800+0000 7fe9b5c00640 1 --2- 172.21.15.28:0/3323924682 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe99006b4d0 0x7fe99006d990 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fe9a4006b10 tx=0x7fe9a4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:52.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.803+0000 7fe9ad600640 1 -- 172.21.15.28:0/3323924682 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe9a8014030 con 0x7fe9b0150240 2024-02-21T23:03:52.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.997+0000 7fe9b7000640 1 -- 172.21.15.28:0/3323924682 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fe9b006f0e0 con 0x7fe9b0150240 2024-02-21T23:03:52.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:52.998+0000 7fe9ad600640 1 -- 172.21.15.28:0/3323924682 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) v1 ==== 103+0+11 (secure 0 0 0) 0x7fe9a8059520 con 0x7fe9b0150240 2024-02-21T23:03:52.988 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-02-21T23:03:52.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.001+0000 7fe9b7000640 1 -- 172.21.15.28:0/3323924682 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe99006b4d0 msgr2=0x7fe99006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:52.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.001+0000 7fe9b7000640 1 --2- 172.21.15.28:0/3323924682 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe99006b4d0 0x7fe99006d990 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fe9a4006b10 tx=0x7fe9a4008040 comp rx=0 tx=0).stop 2024-02-21T23:03:52.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.001+0000 7fe9b7000640 1 -- 172.21.15.28:0/3323924682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9b0150240 msgr2=0x7fe9b01e8be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:52.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.001+0000 7fe9b7000640 1 --2- 172.21.15.28:0/3323924682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9b0150240 0x7fe9b01e8be0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fe9a800cc30 tx=0x7fe9a800a5c0 comp rx=0 tx=0).stop 2024-02-21T23:03:52.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.001+0000 7fe9b7000640 1 -- 172.21.15.28:0/3323924682 shutdown_connections 2024-02-21T23:03:52.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.001+0000 7fe9b7000640 1 --2- 172.21.15.28:0/3323924682 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe99006b4d0 0x7fe99006d990 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:52.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.001+0000 7fe9b7000640 1 --2- 172.21.15.28:0/3323924682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9b0150240 0x7fe9b01e8be0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:52.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.001+0000 7fe9b7000640 1 --2- 172.21.15.28:0/3323924682 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9b00c4580 0x7fe9b00aa960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:52.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.001+0000 7fe9b7000640 1 --2- 172.21.15.28:0/3323924682 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe9b00c2c10 0x7fe9b00aa010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:52.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.001+0000 7fe9b7000640 1 -- 172.21.15.28:0/3323924682 >> 172.21.15.28:0/3323924682 conn(0x7fe9b00a8530 msgr2=0x7fe9b00a8910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:52.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.001+0000 7fe9b7000640 1 -- 172.21.15.28:0/3323924682 shutdown_connections 2024-02-21T23:03:52.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.001+0000 7fe9b7000640 1 -- 172.21.15.28:0/3323924682 wait complete. 2024-02-21T23:03:53.011 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:03:53.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.216+0000 7fe07ca00640 1 Processor -- start 2024-02-21T23:03:53.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.216+0000 7fe07ca00640 1 -- start start 2024-02-21T23:03:53.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.218+0000 7fe07ca00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe0780aff90 0x7fe0780c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:53.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.218+0000 7fe07ca00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe07806dbf0 0x7fe078158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:53.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.218+0000 7fe07ca00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0780c74c0 con 0x7fe0780c5720 2024-02-21T23:03:53.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.218+0000 7fe07ca00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0780c7270 con 0x7fe0780aff90 2024-02-21T23:03:53.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.218+0000 7fe07ca00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe078158630 con 0x7fe07806dbf0 2024-02-21T23:03:53.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.218+0000 7fe076800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe0780aff90 0x7fe0780c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:53.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.218+0000 7fe076800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe0780aff90 0x7fe0780c51e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41920/0 (socket says 172.21.15.28:41920) 2024-02-21T23:03:53.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.218+0000 7fe076800640 1 -- 172.21.15.28:0/1921448749 learned_addr learned my addr 172.21.15.28:0/1921448749 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:53.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.218+0000 7fe06de00640 1 --2- 172.21.15.28:0/1921448749 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe07806dbf0 0x7fe078158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:53.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.218+0000 7fe076800640 1 -- 172.21.15.28:0/1921448749 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe07806dbf0 msgr2=0x7fe078158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:53.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.218+0000 7fe076800640 1 --2- 172.21.15.28:0/1921448749 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe07806dbf0 0x7fe078158ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:53.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.218+0000 7fe076800640 1 -- 172.21.15.28:0/1921448749 >> v1:172.21.15.81:6789/0 conn(0x7fe0780c5720 legacy=0x7fe07806c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:53.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.219+0000 7fe076800640 1 -- 172.21.15.28:0/1921448749 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0780c77a0 con 0x7fe0780aff90 2024-02-21T23:03:53.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.220+0000 7fe076800640 1 --2- 172.21.15.28:0/1921448749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0780aff90 0x7fe0780c51e0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fe068009e50 tx=0x7fe06802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=daee7e5f53a0a2c server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:53.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.220+0000 7fe075e00640 1 -- 172.21.15.28:0/1921448749 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe06802ca70 con 0x7fe0780aff90 2024-02-21T23:03:53.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.220+0000 7fe075e00640 1 -- 172.21.15.28:0/1921448749 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe06802cc10 con 0x7fe0780aff90 2024-02-21T23:03:53.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.220+0000 7fe075e00640 1 -- 172.21.15.28:0/1921448749 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0680343f0 con 0x7fe0780aff90 2024-02-21T23:03:53.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.221+0000 7fe07ca00640 1 -- 172.21.15.28:0/1921448749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0780aff90 msgr2=0x7fe0780c51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:53.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.221+0000 7fe07ca00640 1 --2- 172.21.15.28:0/1921448749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0780aff90 0x7fe0780c51e0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fe068009e50 tx=0x7fe06802bed0 comp rx=0 tx=0).stop 2024-02-21T23:03:53.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.221+0000 7fe07ca00640 1 -- 172.21.15.28:0/1921448749 shutdown_connections 2024-02-21T23:03:53.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.221+0000 7fe07ca00640 1 --2- 172.21.15.28:0/1921448749 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe07806dbf0 0x7fe078158ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:53.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.221+0000 7fe07ca00640 1 --2- 172.21.15.28:0/1921448749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0780aff90 0x7fe0780c51e0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:53.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.221+0000 7fe07ca00640 1 -- 172.21.15.28:0/1921448749 >> 172.21.15.28:0/1921448749 conn(0x7fe0780ac370 msgr2=0x7fe0780ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:53.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.221+0000 7fe07ca00640 1 -- 172.21.15.28:0/1921448749 shutdown_connections 2024-02-21T23:03:53.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.221+0000 7fe07ca00640 1 -- 172.21.15.28:0/1921448749 wait complete. 2024-02-21T23:03:53.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.223+0000 7fe07ca00640 1 Processor -- start 2024-02-21T23:03:53.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.223+0000 7fe07ca00640 1 -- start start 2024-02-21T23:03:53.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.224+0000 7fe07ca00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe07806dbf0 0x7fe0780aea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:53.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.224+0000 7fe07ca00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe0780aff90 0x7fe0781ea9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:53.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.224+0000 7fe07ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0780c5720 0x7fe0781eaf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:53.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.224+0000 7fe07ca00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0780c8760 con 0x7fe0780c5720 2024-02-21T23:03:53.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.224+0000 7fe07ca00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0780be0c0 con 0x7fe0780aff90 2024-02-21T23:03:53.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.224+0000 7fe07ca00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0780c88e0 con 0x7fe07806dbf0 2024-02-21T23:03:53.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.225+0000 7fe077200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0780c5720 0x7fe0781eaf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:53.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.225+0000 7fe077200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0780c5720 0x7fe0781eaf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41934/0 (socket says 172.21.15.28:41934) 2024-02-21T23:03:53.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.225+0000 7fe077200640 1 -- 172.21.15.28:0/3249862831 learned_addr learned my addr 172.21.15.28:0/3249862831 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:53.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.225+0000 7fe06de00640 1 --2- 172.21.15.28:0/3249862831 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe0780aff90 0x7fe0781ea9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:53.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.225+0000 7fe076800640 1 --2- 172.21.15.28:0/3249862831 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe07806dbf0 0x7fe0780aea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:53.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.225+0000 7fe077200640 1 -- 172.21.15.28:0/3249862831 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe07806dbf0 msgr2=0x7fe0780aea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:53.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.225+0000 7fe077200640 1 --2- 172.21.15.28:0/3249862831 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe07806dbf0 0x7fe0780aea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:53.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.225+0000 7fe077200640 1 -- 172.21.15.28:0/3249862831 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe0780aff90 msgr2=0x7fe0781ea9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:53.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.225+0000 7fe077200640 1 --2- 172.21.15.28:0/3249862831 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe0780aff90 0x7fe0781ea9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:53.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.225+0000 7fe077200640 1 -- 172.21.15.28:0/3249862831 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe07814e970 con 0x7fe0780c5720 2024-02-21T23:03:53.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.226+0000 7fe077200640 1 --2- 172.21.15.28:0/3249862831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0780c5720 0x7fe0781eaf30 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fe07000b410 tx=0x7fe07000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:53.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.226+0000 7fe06f400640 1 -- 172.21.15.28:0/3249862831 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe070015030 con 0x7fe0780c5720 2024-02-21T23:03:53.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.226+0000 7fe06f400640 1 -- 172.21.15.28:0/3249862831 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe070004020 con 0x7fe0780c5720 2024-02-21T23:03:53.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.226+0000 7fe06f400640 1 -- 172.21.15.28:0/3249862831 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe070004410 con 0x7fe0780c5720 2024-02-21T23:03:53.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.226+0000 7fe07ca00640 1 -- 172.21.15.28:0/3249862831 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0781ee160 con 0x7fe0780c5720 2024-02-21T23:03:53.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.226+0000 7fe07ca00640 1 -- 172.21.15.28:0/3249862831 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0781574c0 con 0x7fe0780c5720 2024-02-21T23:03:53.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.227+0000 7fe07ca00640 1 -- 172.21.15.28:0/3249862831 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0780c4ab0 con 0x7fe0780c5720 2024-02-21T23:03:53.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.228+0000 7fe06f400640 1 -- 172.21.15.28:0/3249862831 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe070018070 con 0x7fe0780c5720 2024-02-21T23:03:53.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.229+0000 7fe06f400640 1 --2- 172.21.15.28:0/3249862831 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe05406b4d0 0x7fe05406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:53.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.230+0000 7fe06f400640 1 -- 172.21.15.28:0/3249862831 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6735+0+0 (secure 0 0 0) 0x7fe070089ff0 con 0x7fe0780c5720 2024-02-21T23:03:53.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.230+0000 7fe076800640 1 --2- 172.21.15.28:0/3249862831 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe05406b4d0 0x7fe05406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:53.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.231+0000 7fe076800640 1 --2- 172.21.15.28:0/3249862831 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe05406b4d0 0x7fe05406d990 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe068035000 tx=0x7fe068030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:53.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.234+0000 7fe06f400640 1 -- 172.21.15.28:0/3249862831 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe070058ac0 con 0x7fe0780c5720 2024-02-21T23:03:53.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.490+0000 7fe07ca00640 1 -- 172.21.15.28:0/3249862831 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe07804f290 con 0x7fe0780c5720 2024-02-21T23:03:53.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.491+0000 7fe06f400640 1 -- 172.21.15.28:0/3249862831 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4934 (secure 0 0 0) 0x7fe0700583c0 con 0x7fe0780c5720 2024-02-21T23:03:53.481 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:03:53.481 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4230,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/2559793999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2559793999},{"type":"v1","addr":"172.21.15.28:6834","nonce":2559793999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4232,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3930208177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3930208177},{"type":"v1","addr":"172.21.15.28:6835","nonce":3930208177}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4238,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/2875984988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":2875984988},{"type":"v1","addr":"172.21.15.81:6836","nonce":2875984988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:02:37.812604+0000","modified":"2024-02-21T23:03:06.243463+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4267},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4267":{"gid":4267,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6837/2328151245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2328151245},{"type":"v1","addr":"172.21.15.81:6837","nonce":2328151245}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-02-21T23:03:53.482 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 9 2024-02-21T23:03:53.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.495+0000 7fe07ca00640 1 -- 172.21.15.28:0/3249862831 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe05406b4d0 msgr2=0x7fe05406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:53.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.495+0000 7fe07ca00640 1 --2- 172.21.15.28:0/3249862831 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe05406b4d0 0x7fe05406d990 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe068035000 tx=0x7fe068030000 comp rx=0 tx=0).stop 2024-02-21T23:03:53.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.495+0000 7fe07ca00640 1 -- 172.21.15.28:0/3249862831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0780c5720 msgr2=0x7fe0781eaf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:53.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.495+0000 7fe07ca00640 1 --2- 172.21.15.28:0/3249862831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0780c5720 0x7fe0781eaf30 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fe07000b410 tx=0x7fe07000b8d0 comp rx=0 tx=0).stop 2024-02-21T23:03:53.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.495+0000 7fe07ca00640 1 -- 172.21.15.28:0/3249862831 shutdown_connections 2024-02-21T23:03:53.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.495+0000 7fe07ca00640 1 --2- 172.21.15.28:0/3249862831 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe05406b4d0 0x7fe05406d990 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:53.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.495+0000 7fe07ca00640 1 --2- 172.21.15.28:0/3249862831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0780c5720 0x7fe0781eaf30 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:53.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.495+0000 7fe07ca00640 1 --2- 172.21.15.28:0/3249862831 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe0780aff90 0x7fe0781ea9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:53.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.495+0000 7fe07ca00640 1 --2- 172.21.15.28:0/3249862831 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe07806dbf0 0x7fe0780aea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:53.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.495+0000 7fe07ca00640 1 -- 172.21.15.28:0/3249862831 >> 172.21.15.28:0/3249862831 conn(0x7fe0780ac370 msgr2=0x7fe0780ac750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:53.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.495+0000 7fe07ca00640 1 -- 172.21.15.28:0/3249862831 shutdown_connections 2024-02-21T23:03:53.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.495+0000 7fe07ca00640 1 -- 172.21.15.28:0/3249862831 wait complete. 2024-02-21T23:03:53.503 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:03:53.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.719+0000 7fd563000640 1 Processor -- start 2024-02-21T23:03:53.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.719+0000 7fd563000640 1 -- start start 2024-02-21T23:03:53.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.720+0000 7fd563000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fd55c0c4fe0 0x7fd55c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:53.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.720+0000 7fd563000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd55c06ed10 con 0x7fd55c1569d0 2024-02-21T23:03:53.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.720+0000 7fd563000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd55c159900 con 0x7fd55c159400 2024-02-21T23:03:53.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.720+0000 7fd563000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd55c159be0 con 0x7fd55c0c4fe0 2024-02-21T23:03:53.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.721+0000 7fd55be00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd55c1569d0 0x7fd55c0c3280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:47036/0 (socket says 172.21.15.28:47036) 2024-02-21T23:03:53.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.721+0000 7fd55be00640 1 -- 172.21.15.28:0/649689881 learned_addr learned my addr 172.21.15.28:0/649689881 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:03:53.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.721+0000 7fd55b400640 1 --2- 172.21.15.28:0/649689881 >> v2:172.21.15.81:3300/0 conn(0x7fd55c0c4fe0 0x7fd55c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:53.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.721+0000 7fd55aa00640 1 -- 172.21.15.28:0/649689881 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1163340145 0 0) 0x7fd55c06ed10 con 0x7fd55c1569d0 2024-02-21T23:03:53.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.721+0000 7fd55aa00640 1 -- 172.21.15.28:0/649689881 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd540003520 con 0x7fd55c1569d0 2024-02-21T23:03:53.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.721+0000 7fd55aa00640 1 -- 172.21.15.28:0/649689881 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3095120176 0 0) 0x7fd540003520 con 0x7fd55c1569d0 2024-02-21T23:03:53.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.722+0000 7fd55aa00640 1 -- 172.21.15.28:0/649689881 >> v2:172.21.15.81:3300/0 conn(0x7fd55c0c4fe0 msgr2=0x7fd55c06e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:53.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.722+0000 7fd55aa00640 1 --2- 172.21.15.28:0/649689881 >> v2:172.21.15.81:3300/0 conn(0x7fd55c0c4fe0 0x7fd55c06e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:53.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.722+0000 7fd55aa00640 1 -- 172.21.15.28:0/649689881 >> v1:172.21.15.81:6789/0 conn(0x7fd55c159400 legacy=0x7fd55c06e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:53.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.722+0000 7fd55aa00640 1 -- 172.21.15.28:0/649689881 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd55c159ec0 con 0x7fd55c1569d0 2024-02-21T23:03:53.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.722+0000 7fd55aa00640 1 -- 172.21.15.28:0/649689881 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fd54c001670 con 0x7fd55c1569d0 2024-02-21T23:03:53.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.722+0000 7fd55aa00640 1 -- 172.21.15.28:0/649689881 >> v1:172.21.15.28:6789/0 conn(0x7fd55c1569d0 legacy=0x7fd55c0c3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:53.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.722+0000 7fd55aa00640 1 --2- 172.21.15.28:0/649689881 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd5400039b0 0x7fd540003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:53.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.722+0000 7fd55aa00640 1 --2- 172.21.15.28:0/649689881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd540004800 0x7fd540004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:53.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.722+0000 7fd55aa00640 1 --2- 172.21.15.28:0/649689881 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5400051c0 0x7fd54000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:53.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.722+0000 7fd55aa00640 1 -- 172.21.15.28:0/649689881 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd55c159be0 con 0x7fd540004800 2024-02-21T23:03:53.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.722+0000 7fd55aa00640 1 -- 172.21.15.28:0/649689881 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd54000c0f0 con 0x7fd5400051c0 2024-02-21T23:03:53.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.722+0000 7fd55aa00640 1 -- 172.21.15.28:0/649689881 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd54000c2f0 con 0x7fd5400039b0 2024-02-21T23:03:53.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.722+0000 7fd55b400640 1 --2- 172.21.15.28:0/649689881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd540004800 0x7fd540004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:53.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.722+0000 7fd561000640 1 --2- 172.21.15.28:0/649689881 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5400051c0 0x7fd54000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:53.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.722+0000 7fd55b400640 1 -- 172.21.15.28:0/649689881 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd5400039b0 msgr2=0x7fd540003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:53.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.722+0000 7fd55b400640 1 --2- 172.21.15.28:0/649689881 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd5400039b0 0x7fd540003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:53.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.722+0000 7fd55b400640 1 -- 172.21.15.28:0/649689881 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5400051c0 msgr2=0x7fd54000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:53.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.722+0000 7fd55b400640 1 --2- 172.21.15.28:0/649689881 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5400051c0 0x7fd54000b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:53.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.722+0000 7fd561000640 1 -- 172.21.15.28:0/649689881 reap_dead start 2024-02-21T23:03:53.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.722+0000 7fd55b400640 1 -- 172.21.15.28:0/649689881 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd540003520 con 0x7fd540004800 2024-02-21T23:03:53.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.723+0000 7fd55b400640 1 --2- 172.21.15.28:0/649689881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd540004800 0x7fd540004c80 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fd548009ac0 tx=0x7fd540005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a66440cd66265a1f server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:53.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.724+0000 7fd55aa00640 1 -- 172.21.15.28:0/649689881 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd548039070 con 0x7fd540004800 2024-02-21T23:03:53.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.724+0000 7fd55aa00640 1 -- 172.21.15.28:0/649689881 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd54802c6b0 con 0x7fd540004800 2024-02-21T23:03:53.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.724+0000 7fd55aa00640 1 -- 172.21.15.28:0/649689881 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd548009690 con 0x7fd540004800 2024-02-21T23:03:53.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.724+0000 7fd563000640 1 -- 172.21.15.28:0/649689881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd540004800 msgr2=0x7fd540004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:53.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.724+0000 7fd563000640 1 --2- 172.21.15.28:0/649689881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd540004800 0x7fd540004c80 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fd548009ac0 tx=0x7fd540005830 comp rx=0 tx=0).stop 2024-02-21T23:03:53.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.724+0000 7fd563000640 1 -- 172.21.15.28:0/649689881 shutdown_connections 2024-02-21T23:03:53.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.724+0000 7fd563000640 1 --2- 172.21.15.28:0/649689881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd540004800 0x7fd540004c80 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:53.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.724+0000 7fd563000640 1 -- 172.21.15.28:0/649689881 >> 172.21.15.28:0/649689881 conn(0x7fd55c0c67f0 msgr2=0x7fd55c0c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:53.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.724+0000 7fd563000640 1 -- 172.21.15.28:0/649689881 shutdown_connections 2024-02-21T23:03:53.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.724+0000 7fd563000640 1 -- 172.21.15.28:0/649689881 wait complete. 2024-02-21T23:03:53.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.726+0000 7fd563000640 1 Processor -- start 2024-02-21T23:03:53.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.727+0000 7fd563000640 1 -- start start 2024-02-21T23:03:53.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.727+0000 7fd563000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd540004800 0x7fd55c1f27d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:53.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.727+0000 7fd563000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd55c15a5f0 0x7fd55c153370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:53.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.728+0000 7fd563000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd55c1f2d10 0x7fd55c1538b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:53.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.728+0000 7fd563000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd55c0c46f0 con 0x7fd55c1f2d10 2024-02-21T23:03:53.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.728+0000 7fd563000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd55c156db0 con 0x7fd55c15a5f0 2024-02-21T23:03:53.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.728+0000 7fd563000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd55c0c4870 con 0x7fd540004800 2024-02-21T23:03:53.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.728+0000 7fd561000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd55c1f2d10 0x7fd55c1538b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:53.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.728+0000 7fd561000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd55c1f2d10 0x7fd55c1538b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41954/0 (socket says 172.21.15.28:41954) 2024-02-21T23:03:53.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.728+0000 7fd561000640 1 -- 172.21.15.28:0/1273547634 learned_addr learned my addr 172.21.15.28:0/1273547634 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:53.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.728+0000 7fd55b400640 1 --2- 172.21.15.28:0/1273547634 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd55c15a5f0 0x7fd55c153370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:53.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.728+0000 7fd55be00640 1 --2- 172.21.15.28:0/1273547634 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd540004800 0x7fd55c1f27d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:53.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.728+0000 7fd561000640 1 -- 172.21.15.28:0/1273547634 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd540004800 msgr2=0x7fd55c1f27d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:53.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.728+0000 7fd561000640 1 --2- 172.21.15.28:0/1273547634 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd540004800 0x7fd55c1f27d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:53.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.728+0000 7fd561000640 1 -- 172.21.15.28:0/1273547634 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd55c15a5f0 msgr2=0x7fd55c153370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:53.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.728+0000 7fd561000640 1 --2- 172.21.15.28:0/1273547634 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd55c15a5f0 0x7fd55c153370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:53.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.728+0000 7fd561000640 1 -- 172.21.15.28:0/1273547634 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd55c0ac650 con 0x7fd55c1f2d10 2024-02-21T23:03:53.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.729+0000 7fd561000640 1 --2- 172.21.15.28:0/1273547634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd55c1f2d10 0x7fd55c1538b0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fd55c06f630 tx=0x7fd55400d9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:53.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.729+0000 7fd553e00640 1 -- 172.21.15.28:0/1273547634 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd554016070 con 0x7fd55c1f2d10 2024-02-21T23:03:53.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.729+0000 7fd553e00640 1 -- 172.21.15.28:0/1273547634 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd554012800 con 0x7fd55c1f2d10 2024-02-21T23:03:53.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.729+0000 7fd553e00640 1 -- 172.21.15.28:0/1273547634 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd554012bf0 con 0x7fd55c1f2d10 2024-02-21T23:03:53.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.729+0000 7fd563000640 1 -- 172.21.15.28:0/1273547634 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd55c1516e0 con 0x7fd55c1f2d10 2024-02-21T23:03:53.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.729+0000 7fd563000640 1 -- 172.21.15.28:0/1273547634 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd55c1519a0 con 0x7fd55c1f2d10 2024-02-21T23:03:53.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.730+0000 7fd563000640 1 -- 172.21.15.28:0/1273547634 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd55c0c4410 con 0x7fd55c1f2d10 2024-02-21T23:03:53.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.731+0000 7fd553e00640 1 -- 172.21.15.28:0/1273547634 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd554019070 con 0x7fd55c1f2d10 2024-02-21T23:03:53.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.732+0000 7fd553e00640 1 --2- 172.21.15.28:0/1273547634 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd53c06b4d0 0x7fd53c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:53.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.732+0000 7fd553e00640 1 -- 172.21.15.28:0/1273547634 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6735+0+0 (secure 0 0 0) 0x7fd55408a490 con 0x7fd55c1f2d10 2024-02-21T23:03:53.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.733+0000 7fd55be00640 1 --2- 172.21.15.28:0/1273547634 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd53c06b4d0 0x7fd53c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:53.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.734+0000 7fd55be00640 1 --2- 172.21.15.28:0/1273547634 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd53c06b4d0 0x7fd53c06d990 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd55c0ac1a0 tx=0x7fd54c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:53.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.736+0000 7fd553e00640 1 -- 172.21.15.28:0/1273547634 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd55c0c4410 con 0x7fd55c1f2d10 2024-02-21T23:03:53.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.939+0000 7fd563000640 1 -- 172.21.15.28:0/1273547634 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd55c151ba0 con 0x7fd55c1f2d10 2024-02-21T23:03:53.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.942+0000 7fd553e00640 1 -- 172.21.15.28:0/1273547634 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+18843 (secure 0 0 0) 0x7fd55c151ba0 con 0x7fd55c1f2d10 2024-02-21T23:03:53.932 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:03:53.932 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":26,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:03:23.860109+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:02:35.795158+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-21T23:02:36.840110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-02-21T23:02:40.441388+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":3,"to":2}]},{"pgid":"3.12","mappings":[{"from":0,"to":1}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:03:53.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.945+0000 7fd563000640 1 -- 172.21.15.28:0/1273547634 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd53c06b4d0 msgr2=0x7fd53c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:53.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.945+0000 7fd563000640 1 --2- 172.21.15.28:0/1273547634 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd53c06b4d0 0x7fd53c06d990 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd55c0ac1a0 tx=0x7fd54c009040 comp rx=0 tx=0).stop 2024-02-21T23:03:53.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.945+0000 7fd563000640 1 -- 172.21.15.28:0/1273547634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd55c1f2d10 msgr2=0x7fd55c1538b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:53.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.945+0000 7fd563000640 1 --2- 172.21.15.28:0/1273547634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd55c1f2d10 0x7fd55c1538b0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fd55c06f630 tx=0x7fd55400d9f0 comp rx=0 tx=0).stop 2024-02-21T23:03:53.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.945+0000 7fd563000640 1 -- 172.21.15.28:0/1273547634 shutdown_connections 2024-02-21T23:03:53.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.945+0000 7fd563000640 1 --2- 172.21.15.28:0/1273547634 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd53c06b4d0 0x7fd53c06d990 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:53.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.945+0000 7fd563000640 1 --2- 172.21.15.28:0/1273547634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd55c1f2d10 0x7fd55c1538b0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:53.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.945+0000 7fd563000640 1 --2- 172.21.15.28:0/1273547634 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd55c15a5f0 0x7fd55c153370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:53.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.945+0000 7fd563000640 1 --2- 172.21.15.28:0/1273547634 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd540004800 0x7fd55c1f27d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:53.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.945+0000 7fd563000640 1 -- 172.21.15.28:0/1273547634 >> 172.21.15.28:0/1273547634 conn(0x7fd55c0c67f0 msgr2=0x7fd55c154780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:53.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.945+0000 7fd563000640 1 -- 172.21.15.28:0/1273547634 shutdown_connections 2024-02-21T23:03:53.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:53.945+0000 7fd563000640 1 -- 172.21.15.28:0/1273547634 wait complete. 2024-02-21T23:03:53.954 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-02-21T23:03:53.955 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:03:54.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.180+0000 7f2b45a00640 1 Processor -- start 2024-02-21T23:03:54.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.180+0000 7f2b45a00640 1 -- start start 2024-02-21T23:03:54.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.182+0000 7f2b45a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f2b4006ee40 0x7f2b400a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:54.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.182+0000 7f2b45a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b400c2760 con 0x7f2b40156af0 2024-02-21T23:03:54.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.182+0000 7f2b45a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b40156310 con 0x7f2b40150450 2024-02-21T23:03:54.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.182+0000 7f2b45a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b400c22c0 con 0x7f2b4006ee40 2024-02-21T23:03:54.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.182+0000 7f2b45000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f2b40156af0 0x7f2b4006dff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:47052/0 (socket says 172.21.15.28:47052) 2024-02-21T23:03:54.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.182+0000 7f2b45000640 1 -- 172.21.15.28:0/1418563676 learned_addr learned my addr 172.21.15.28:0/1418563676 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:03:54.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.183+0000 7f2b3f400640 1 -- 172.21.15.28:0/1418563676 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 272935413 0 0) 0x7f2b400c2760 con 0x7f2b40156af0 2024-02-21T23:03:54.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.183+0000 7f2b3f400640 1 -- 172.21.15.28:0/1418563676 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b24003520 con 0x7f2b40156af0 2024-02-21T23:03:54.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.183+0000 7f2b37e00640 1 --2- 172.21.15.28:0/1418563676 >> v2:172.21.15.81:3300/0 conn(0x7f2b4006ee40 0x7f2b400a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:54.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.183+0000 7f2b3f400640 1 -- 172.21.15.28:0/1418563676 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 416364736 0 0) 0x7f2b24003520 con 0x7f2b40156af0 2024-02-21T23:03:54.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.183+0000 7f2b3f400640 1 -- 172.21.15.28:0/1418563676 >> v2:172.21.15.81:3300/0 conn(0x7f2b4006ee40 msgr2=0x7f2b400a5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:54.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.183+0000 7f2b3f400640 1 --2- 172.21.15.28:0/1418563676 >> v2:172.21.15.81:3300/0 conn(0x7f2b4006ee40 0x7f2b400a5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:54.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.183+0000 7f2b3f400640 1 -- 172.21.15.28:0/1418563676 >> v1:172.21.15.81:6789/0 conn(0x7f2b40150450 legacy=0x7f2b401536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:54.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.183+0000 7f2b3f400640 1 -- 172.21.15.28:0/1418563676 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b400c29d0 con 0x7f2b40156af0 2024-02-21T23:03:54.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.183+0000 7f2b3f400640 1 -- 172.21.15.28:0/1418563676 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f2b380019f0 con 0x7f2b40156af0 2024-02-21T23:03:54.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.183+0000 7f2b3f400640 1 -- 172.21.15.28:0/1418563676 >> v1:172.21.15.28:6789/0 conn(0x7f2b40156af0 legacy=0x7f2b4006dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:54.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.183+0000 7f2b3f400640 1 --2- 172.21.15.28:0/1418563676 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2b24003a00 0x7f2b24003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:54.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.183+0000 7f2b3f400640 1 --2- 172.21.15.28:0/1418563676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b24004800 0x7f2b24004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:54.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.183+0000 7f2b3f400640 1 --2- 172.21.15.28:0/1418563676 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2b24005210 0x7f2b2400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:54.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.183+0000 7f2b3f400640 1 -- 172.21.15.28:0/1418563676 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b2400c130 con 0x7f2b24004800 2024-02-21T23:03:54.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.183+0000 7f2b3f400640 1 -- 172.21.15.28:0/1418563676 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b2400c2e0 con 0x7f2b24003a00 2024-02-21T23:03:54.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.183+0000 7f2b3f400640 1 -- 172.21.15.28:0/1418563676 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b2400c4e0 con 0x7f2b24005210 2024-02-21T23:03:54.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.183+0000 7f2b37e00640 1 --2- 172.21.15.28:0/1418563676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b24004800 0x7f2b24004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:54.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.184+0000 7f2b3fe00640 1 --2- 172.21.15.28:0/1418563676 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2b24003a00 0x7f2b24003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:54.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.184+0000 7f2b45000640 1 --2- 172.21.15.28:0/1418563676 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2b24005210 0x7f2b2400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:54.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.184+0000 7f2b37e00640 1 -- 172.21.15.28:0/1418563676 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2b24005210 msgr2=0x7f2b2400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:54.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.184+0000 7f2b37e00640 1 --2- 172.21.15.28:0/1418563676 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2b24005210 0x7f2b2400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:54.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.184+0000 7f2b37e00640 1 -- 172.21.15.28:0/1418563676 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2b24003a00 msgr2=0x7f2b24003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:54.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.184+0000 7f2b37e00640 1 --2- 172.21.15.28:0/1418563676 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2b24003a00 0x7f2b24003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:54.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.184+0000 7f2b37e00640 1 -- 172.21.15.28:0/1418563676 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b24003520 con 0x7f2b24004800 2024-02-21T23:03:54.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.184+0000 7f2b45000640 1 --2- 172.21.15.28:0/1418563676 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2b24005210 0x7f2b2400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:03:54.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.184+0000 7f2b3fe00640 1 --2- 172.21.15.28:0/1418563676 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2b24003a00 0x7f2b24003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:03:54.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.184+0000 7f2b45000640 1 -- 172.21.15.28:0/1418563676 reap_dead start 2024-02-21T23:03:54.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.185+0000 7f2b37e00640 1 --2- 172.21.15.28:0/1418563676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b24004800 0x7f2b24004c80 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f2b2800ba30 tx=0x7f2b24005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfbe3a6dc4e9138 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:54.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.185+0000 7f2b3f400640 1 -- 172.21.15.28:0/1418563676 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b28036020 con 0x7f2b24004800 2024-02-21T23:03:54.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.185+0000 7f2b3f400640 1 -- 172.21.15.28:0/1418563676 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b28032640 con 0x7f2b24004800 2024-02-21T23:03:54.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.185+0000 7f2b3f400640 1 -- 172.21.15.28:0/1418563676 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b2800b600 con 0x7f2b24004800 2024-02-21T23:03:54.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.186+0000 7f2b45a00640 1 -- 172.21.15.28:0/1418563676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b24004800 msgr2=0x7f2b24004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:54.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.186+0000 7f2b45a00640 1 --2- 172.21.15.28:0/1418563676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b24004800 0x7f2b24004c80 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f2b2800ba30 tx=0x7f2b24005830 comp rx=0 tx=0).stop 2024-02-21T23:03:54.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.186+0000 7f2b45a00640 1 -- 172.21.15.28:0/1418563676 shutdown_connections 2024-02-21T23:03:54.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.186+0000 7f2b45a00640 1 --2- 172.21.15.28:0/1418563676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b24004800 0x7f2b24004c80 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:54.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.186+0000 7f2b45a00640 1 -- 172.21.15.28:0/1418563676 >> 172.21.15.28:0/1418563676 conn(0x7f2b4006b630 msgr2=0x7f2b400c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:54.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.186+0000 7f2b45a00640 1 -- 172.21.15.28:0/1418563676 shutdown_connections 2024-02-21T23:03:54.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.186+0000 7f2b45a00640 1 -- 172.21.15.28:0/1418563676 wait complete. 2024-02-21T23:03:54.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.188+0000 7f2b45a00640 1 Processor -- start 2024-02-21T23:03:54.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.188+0000 7f2b45a00640 1 -- start start 2024-02-21T23:03:54.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.189+0000 7f2b45a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2b24004800 0x7f2b400c5880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:54.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.189+0000 7f2b45a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2b4014e5b0 0x7f2b400c5dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:54.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.189+0000 7f2b45a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b401eac70 0x7f2b401eb0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:54.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.189+0000 7f2b45a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b40156f80 con 0x7f2b401eac70 2024-02-21T23:03:54.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.189+0000 7f2b45a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b40156660 con 0x7f2b4014e5b0 2024-02-21T23:03:54.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.189+0000 7f2b45a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b40157100 con 0x7f2b24004800 2024-02-21T23:03:54.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.189+0000 7f2b45000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b401eac70 0x7f2b401eb0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:54.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.189+0000 7f2b45000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b401eac70 0x7f2b401eb0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41982/0 (socket says 172.21.15.28:41982) 2024-02-21T23:03:54.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.189+0000 7f2b45000640 1 -- 172.21.15.28:0/4036573933 learned_addr learned my addr 172.21.15.28:0/4036573933 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:54.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.189+0000 7f2b3fe00640 1 --2- 172.21.15.28:0/4036573933 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2b24004800 0x7f2b400c5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:54.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.189+0000 7f2b37e00640 1 --2- 172.21.15.28:0/4036573933 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2b4014e5b0 0x7f2b400c5dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:54.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.190+0000 7f2b45000640 1 -- 172.21.15.28:0/4036573933 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2b24004800 msgr2=0x7f2b400c5880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:54.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.190+0000 7f2b45000640 1 --2- 172.21.15.28:0/4036573933 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2b24004800 0x7f2b400c5880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:54.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.190+0000 7f2b45000640 1 -- 172.21.15.28:0/4036573933 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2b4014e5b0 msgr2=0x7f2b400c5dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:54.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.190+0000 7f2b45000640 1 --2- 172.21.15.28:0/4036573933 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2b4014e5b0 0x7f2b400c5dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:54.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.190+0000 7f2b45000640 1 -- 172.21.15.28:0/4036573933 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b401eeb80 con 0x7f2b401eac70 2024-02-21T23:03:54.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.190+0000 7f2b37e00640 1 --2- 172.21.15.28:0/4036573933 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2b4014e5b0 0x7f2b400c5dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:03:54.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.190+0000 7f2b45000640 1 --2- 172.21.15.28:0/4036573933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b401eac70 0x7f2b401eb0f0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f2b400c64b0 tx=0x7f2b3800b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:54.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.190+0000 7f2b3cc00640 1 -- 172.21.15.28:0/4036573933 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b38004d60 con 0x7f2b401eac70 2024-02-21T23:03:54.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.190+0000 7f2b3cc00640 1 -- 172.21.15.28:0/4036573933 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b3800bb30 con 0x7f2b401eac70 2024-02-21T23:03:54.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.190+0000 7f2b45a00640 1 -- 172.21.15.28:0/4036573933 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b40072b70 con 0x7f2b401eac70 2024-02-21T23:03:54.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.190+0000 7f2b3cc00640 1 -- 172.21.15.28:0/4036573933 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b38007070 con 0x7f2b401eac70 2024-02-21T23:03:54.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.190+0000 7f2b45a00640 1 -- 172.21.15.28:0/4036573933 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b40072860 con 0x7f2b401eac70 2024-02-21T23:03:54.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.191+0000 7f2b45a00640 1 -- 172.21.15.28:0/4036573933 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b4004f290 con 0x7f2b401eac70 2024-02-21T23:03:54.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.192+0000 7f2b3cc00640 1 -- 172.21.15.28:0/4036573933 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2b3801b020 con 0x7f2b401eac70 2024-02-21T23:03:54.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.194+0000 7f2b3cc00640 1 --2- 172.21.15.28:0/4036573933 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2b1c06b4d0 0x7f2b1c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:54.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.194+0000 7f2b3cc00640 1 -- 172.21.15.28:0/4036573933 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f2b3808a720 con 0x7f2b401eac70 2024-02-21T23:03:54.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.194+0000 7f2b3fe00640 1 --2- 172.21.15.28:0/4036573933 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2b1c06b4d0 0x7f2b1c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:54.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.195+0000 7f2b3fe00640 1 --2- 172.21.15.28:0/4036573933 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2b1c06b4d0 0x7f2b1c06d990 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f2b30000c00 tx=0x7f2b30009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:54.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.198+0000 7f2b3cc00640 1 -- 172.21.15.28:0/4036573933 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f2b380591a0 con 0x7f2b401eac70 2024-02-21T23:03:54.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.441+0000 7f2b45a00640 1 -- 172.21.15.28:0/4036573933 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2b4006e8c0 con 0x7f2b401eac70 2024-02-21T23:03:54.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.442+0000 7f2b3cc00640 1 -- 172.21.15.28:0/4036573933 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4934 (secure 0 0 0) 0x7f2b38058aa0 con 0x7f2b401eac70 2024-02-21T23:03:54.432 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:03:54.432 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4230,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/2559793999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2559793999},{"type":"v1","addr":"172.21.15.28:6834","nonce":2559793999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4232,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3930208177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3930208177},{"type":"v1","addr":"172.21.15.28:6835","nonce":3930208177}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4238,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/2875984988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":2875984988},{"type":"v1","addr":"172.21.15.81:6836","nonce":2875984988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:02:37.812604+0000","modified":"2024-02-21T23:03:06.243463+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4267},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4267":{"gid":4267,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6837/2328151245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2328151245},{"type":"v1","addr":"172.21.15.81:6837","nonce":2328151245}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-02-21T23:03:54.432 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 9 2024-02-21T23:03:54.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.445+0000 7f2b45a00640 1 -- 172.21.15.28:0/4036573933 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2b1c06b4d0 msgr2=0x7f2b1c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:54.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.445+0000 7f2b45a00640 1 --2- 172.21.15.28:0/4036573933 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2b1c06b4d0 0x7f2b1c06d990 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f2b30000c00 tx=0x7f2b30009040 comp rx=0 tx=0).stop 2024-02-21T23:03:54.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.445+0000 7f2b45a00640 1 -- 172.21.15.28:0/4036573933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b401eac70 msgr2=0x7f2b401eb0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:54.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.445+0000 7f2b45a00640 1 --2- 172.21.15.28:0/4036573933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b401eac70 0x7f2b401eb0f0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f2b400c64b0 tx=0x7f2b3800b4a0 comp rx=0 tx=0).stop 2024-02-21T23:03:54.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.446+0000 7f2b45a00640 1 -- 172.21.15.28:0/4036573933 shutdown_connections 2024-02-21T23:03:54.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.446+0000 7f2b45a00640 1 --2- 172.21.15.28:0/4036573933 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2b1c06b4d0 0x7f2b1c06d990 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:54.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.446+0000 7f2b45a00640 1 --2- 172.21.15.28:0/4036573933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b401eac70 0x7f2b401eb0f0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:54.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.446+0000 7f2b45a00640 1 --2- 172.21.15.28:0/4036573933 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2b4014e5b0 0x7f2b400c5dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:54.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.446+0000 7f2b45a00640 1 --2- 172.21.15.28:0/4036573933 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2b24004800 0x7f2b400c5880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:54.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.446+0000 7f2b45a00640 1 -- 172.21.15.28:0/4036573933 >> 172.21.15.28:0/4036573933 conn(0x7f2b4006b630 msgr2=0x7f2b4014f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:54.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.446+0000 7f2b45a00640 1 -- 172.21.15.28:0/4036573933 shutdown_connections 2024-02-21T23:03:54.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.446+0000 7f2b45a00640 1 -- 172.21.15.28:0/4036573933 wait complete. 2024-02-21T23:03:54.456 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:03:54.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.684+0000 7f8bb1600640 1 Processor -- start 2024-02-21T23:03:54.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.684+0000 7f8bb1600640 1 -- start start 2024-02-21T23:03:54.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.685+0000 7f8bb1600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8bac0aff90 0x7f8bac0c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:54.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.685+0000 7f8bb1600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8bac06dbf0 0x7f8bac158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:54.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.685+0000 7f8bb1600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bac0c74c0 con 0x7f8bac0c5720 2024-02-21T23:03:54.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.685+0000 7f8bb1600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bac0c7270 con 0x7f8bac06dbf0 2024-02-21T23:03:54.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.685+0000 7f8bb1600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bac158630 con 0x7f8bac0aff90 2024-02-21T23:03:54.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.686+0000 7f8ba3400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8bac06dbf0 0x7f8bac158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:54.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.686+0000 7f8ba3400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8bac06dbf0 0x7f8bac158ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41986/0 (socket says 172.21.15.28:41986) 2024-02-21T23:03:54.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.686+0000 7f8ba3400640 1 -- 172.21.15.28:0/3420039877 learned_addr learned my addr 172.21.15.28:0/3420039877 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:54.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.686+0000 7f8babe00640 1 --2- 172.21.15.28:0/3420039877 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8bac0aff90 0x7f8bac0c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:54.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.686+0000 7f8ba3400640 1 -- 172.21.15.28:0/3420039877 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8bac0aff90 msgr2=0x7f8bac0c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:54.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.686+0000 7f8ba3400640 1 --2- 172.21.15.28:0/3420039877 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8bac0aff90 0x7f8bac0c51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:54.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.686+0000 7f8ba3400640 1 -- 172.21.15.28:0/3420039877 >> v1:172.21.15.81:6789/0 conn(0x7f8bac0c5720 legacy=0x7f8bac06c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:54.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.686+0000 7f8ba3400640 1 -- 172.21.15.28:0/3420039877 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bac0c77a0 con 0x7f8bac06dbf0 2024-02-21T23:03:54.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.687+0000 7f8ba3400640 1 --2- 172.21.15.28:0/3420039877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bac06dbf0 0x7f8bac158ac0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f8b94009870 tx=0x7f8b9402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41496319ecedfb6f server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:54.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.688+0000 7f8bab400640 1 -- 172.21.15.28:0/3420039877 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b94035020 con 0x7f8bac06dbf0 2024-02-21T23:03:54.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.688+0000 7f8bab400640 1 -- 172.21.15.28:0/3420039877 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b94031640 con 0x7f8bac06dbf0 2024-02-21T23:03:54.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.688+0000 7f8bab400640 1 -- 172.21.15.28:0/3420039877 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b9402d040 con 0x7f8bac06dbf0 2024-02-21T23:03:54.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.688+0000 7f8bb1600640 1 -- 172.21.15.28:0/3420039877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bac06dbf0 msgr2=0x7f8bac158ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:54.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.688+0000 7f8bb1600640 1 --2- 172.21.15.28:0/3420039877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bac06dbf0 0x7f8bac158ac0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f8b94009870 tx=0x7f8b9402b8f0 comp rx=0 tx=0).stop 2024-02-21T23:03:54.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.688+0000 7f8bb1600640 1 -- 172.21.15.28:0/3420039877 shutdown_connections 2024-02-21T23:03:54.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.688+0000 7f8bb1600640 1 --2- 172.21.15.28:0/3420039877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bac06dbf0 0x7f8bac158ac0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:54.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.688+0000 7f8bb1600640 1 --2- 172.21.15.28:0/3420039877 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8bac0aff90 0x7f8bac0c51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:54.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.688+0000 7f8bb1600640 1 -- 172.21.15.28:0/3420039877 >> 172.21.15.28:0/3420039877 conn(0x7f8bac0ac370 msgr2=0x7f8bac0ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:54.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.688+0000 7f8bb1600640 1 -- 172.21.15.28:0/3420039877 shutdown_connections 2024-02-21T23:03:54.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.688+0000 7f8bb1600640 1 -- 172.21.15.28:0/3420039877 wait complete. 2024-02-21T23:03:54.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.690+0000 7f8bb1600640 1 Processor -- start 2024-02-21T23:03:54.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.690+0000 7f8bb1600640 1 -- start start 2024-02-21T23:03:54.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.691+0000 7f8bb1600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8bac06dbf0 0x7f8bac0aec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:54.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.691+0000 7f8bb1600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bac0aff90 0x7f8bac1eb260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:54.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.691+0000 7f8bb1600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8bac0c5720 0x7f8bac1eb7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:54.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.691+0000 7f8bb1600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bac0c8760 con 0x7f8bac0aff90 2024-02-21T23:03:54.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.691+0000 7f8bb1600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bac0be0c0 con 0x7f8bac06dbf0 2024-02-21T23:03:54.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.691+0000 7f8bb1600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bac0c88e0 con 0x7f8bac0c5720 2024-02-21T23:03:54.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.691+0000 7f8ba3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bac0aff90 0x7f8bac1eb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:54.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.691+0000 7f8ba3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bac0aff90 0x7f8bac1eb260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41992/0 (socket says 172.21.15.28:41992) 2024-02-21T23:03:54.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.691+0000 7f8ba3400640 1 -- 172.21.15.28:0/533231809 learned_addr learned my addr 172.21.15.28:0/533231809 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:54.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.692+0000 7f8babe00640 1 --2- 172.21.15.28:0/533231809 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8bac06dbf0 0x7f8bac0aec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:54.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.692+0000 7f8bb0c00640 1 --2- 172.21.15.28:0/533231809 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8bac0c5720 0x7f8bac1eb7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:54.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.692+0000 7f8ba3400640 1 -- 172.21.15.28:0/533231809 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8bac0c5720 msgr2=0x7f8bac1eb7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:54.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.692+0000 7f8ba3400640 1 --2- 172.21.15.28:0/533231809 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8bac0c5720 0x7f8bac1eb7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:54.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.692+0000 7f8ba3400640 1 -- 172.21.15.28:0/533231809 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8bac06dbf0 msgr2=0x7f8bac0aec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:54.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.692+0000 7f8ba3400640 1 --2- 172.21.15.28:0/533231809 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8bac06dbf0 0x7f8bac0aec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:54.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.692+0000 7f8ba3400640 1 -- 172.21.15.28:0/533231809 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bac1540d0 con 0x7f8bac0aff90 2024-02-21T23:03:54.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.692+0000 7f8ba3400640 1 --2- 172.21.15.28:0/533231809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bac0aff90 0x7f8bac1eb260 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f8b94000c00 tx=0x7f8b940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:54.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.693+0000 7f8ba8c00640 1 -- 172.21.15.28:0/533231809 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b9403aa60 con 0x7f8bac0aff90 2024-02-21T23:03:54.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.693+0000 7f8ba8c00640 1 -- 172.21.15.28:0/533231809 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b9403ac00 con 0x7f8bac0aff90 2024-02-21T23:03:54.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.693+0000 7f8ba8c00640 1 -- 172.21.15.28:0/533231809 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b940443f0 con 0x7f8bac0aff90 2024-02-21T23:03:54.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.693+0000 7f8bb1600640 1 -- 172.21.15.28:0/533231809 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bac1eef20 con 0x7f8bac0aff90 2024-02-21T23:03:54.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.693+0000 7f8bb1600640 1 -- 172.21.15.28:0/533231809 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bac0c13a0 con 0x7f8bac0aff90 2024-02-21T23:03:54.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.694+0000 7f8bb1600640 1 -- 172.21.15.28:0/533231809 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bac04f290 con 0x7f8bac0aff90 2024-02-21T23:03:54.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.695+0000 7f8ba8c00640 1 -- 172.21.15.28:0/533231809 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8b94038020 con 0x7f8bac0aff90 2024-02-21T23:03:54.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.696+0000 7f8ba8c00640 1 --2- 172.21.15.28:0/533231809 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8b8806b4d0 0x7f8b8806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:54.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.696+0000 7f8ba8c00640 1 -- 172.21.15.28:0/533231809 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f8b940ac410 con 0x7f8bac0aff90 2024-02-21T23:03:54.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.697+0000 7f8babe00640 1 --2- 172.21.15.28:0/533231809 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8b8806b4d0 0x7f8b8806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:54.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.698+0000 7f8babe00640 1 --2- 172.21.15.28:0/533231809 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8b8806b4d0 0x7f8b8806d990 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c009a60 tx=0x7f8b9c003990 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:54.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.701+0000 7f8ba8c00640 1 -- 172.21.15.28:0/533231809 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f8b9403f070 con 0x7f8bac0aff90 2024-02-21T23:03:54.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.941+0000 7f8bb1600640 1 -- 172.21.15.28:0/533231809 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8bac06d9d0 con 0x7f8bac0aff90 2024-02-21T23:03:54.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.942+0000 7f8ba8c00640 1 -- 172.21.15.28:0/533231809 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4934 (secure 0 0 0) 0x7f8b9407b020 con 0x7f8bac0aff90 2024-02-21T23:03:54.932 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:03:54.932 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4230,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/2559793999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2559793999},{"type":"v1","addr":"172.21.15.28:6834","nonce":2559793999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4232,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3930208177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3930208177},{"type":"v1","addr":"172.21.15.28:6835","nonce":3930208177}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4238,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/2875984988","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":2875984988},{"type":"v1","addr":"172.21.15.81:6836","nonce":2875984988}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:02:37.812604+0000","modified":"2024-02-21T23:03:06.243463+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4267},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4267":{"gid":4267,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6837/2328151245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2328151245},{"type":"v1","addr":"172.21.15.81:6837","nonce":2328151245}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-02-21T23:03:54.932 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 9 2024-02-21T23:03:54.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.945+0000 7f8bb1600640 1 -- 172.21.15.28:0/533231809 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8b8806b4d0 msgr2=0x7f8b8806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:54.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.945+0000 7f8bb1600640 1 --2- 172.21.15.28:0/533231809 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8b8806b4d0 0x7f8b8806d990 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c009a60 tx=0x7f8b9c003990 comp rx=0 tx=0).stop 2024-02-21T23:03:54.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.945+0000 7f8bb1600640 1 -- 172.21.15.28:0/533231809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bac0aff90 msgr2=0x7f8bac1eb260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:54.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.945+0000 7f8bb1600640 1 --2- 172.21.15.28:0/533231809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bac0aff90 0x7f8bac1eb260 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f8b94000c00 tx=0x7f8b940046c0 comp rx=0 tx=0).stop 2024-02-21T23:03:54.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.945+0000 7f8bb1600640 1 -- 172.21.15.28:0/533231809 shutdown_connections 2024-02-21T23:03:54.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.945+0000 7f8bb1600640 1 --2- 172.21.15.28:0/533231809 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8b8806b4d0 0x7f8b8806d990 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:54.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.945+0000 7f8bb1600640 1 --2- 172.21.15.28:0/533231809 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8bac0c5720 0x7f8bac1eb7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:54.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.945+0000 7f8bb1600640 1 --2- 172.21.15.28:0/533231809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bac0aff90 0x7f8bac1eb260 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:54.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.945+0000 7f8bb1600640 1 --2- 172.21.15.28:0/533231809 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8bac06dbf0 0x7f8bac0aec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:54.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.945+0000 7f8bb1600640 1 -- 172.21.15.28:0/533231809 >> 172.21.15.28:0/533231809 conn(0x7f8bac0ac370 msgr2=0x7f8bac0ac750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:54.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.945+0000 7f8bb1600640 1 -- 172.21.15.28:0/533231809 shutdown_connections 2024-02-21T23:03:54.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:54.945+0000 7f8bb1600640 1 -- 172.21.15.28:0/533231809 wait complete. 2024-02-21T23:03:54.954 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:03:55.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.184+0000 7fe182000640 1 Processor -- start 2024-02-21T23:03:55.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.184+0000 7fe182000640 1 -- start start 2024-02-21T23:03:55.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.186+0000 7fe182000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe17c150450 0x7fe17c0a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:55.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.186+0000 7fe182000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fe17c06ee40 0x7fe17c1536f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:55.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.186+0000 7fe182000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe17c0c2760 con 0x7fe17c156af0 2024-02-21T23:03:55.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.186+0000 7fe182000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe17c0c22c0 con 0x7fe17c06ee40 2024-02-21T23:03:55.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.186+0000 7fe182000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe17c156660 con 0x7fe17c150450 2024-02-21T23:03:55.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.186+0000 7fe181600640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe17c156af0 0x7fe17c06dff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:40398/0 (socket says 172.21.15.28:40398) 2024-02-21T23:03:55.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.186+0000 7fe181600640 1 -- 172.21.15.28:0/4288217832 learned_addr learned my addr 172.21.15.28:0/4288217832 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:03:55.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.186+0000 7fe17be00640 1 --2- 172.21.15.28:0/4288217832 >> v2:172.21.15.81:3300/0 conn(0x7fe17c06ee40 0x7fe17c1536f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:55.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.186+0000 7fe180c00640 1 --2- 172.21.15.28:0/4288217832 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe17c150450 0x7fe17c0a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:55.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.187+0000 7fe17b400640 1 -- 172.21.15.28:0/4288217832 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1594799840 0 0) 0x7fe17c0c2760 con 0x7fe17c156af0 2024-02-21T23:03:55.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.187+0000 7fe17b400640 1 -- 172.21.15.28:0/4288217832 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe164003520 con 0x7fe17c156af0 2024-02-21T23:03:55.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.187+0000 7fe17b400640 1 -- 172.21.15.28:0/4288217832 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2192467173 0 0) 0x7fe164003520 con 0x7fe17c156af0 2024-02-21T23:03:55.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.187+0000 7fe17b400640 1 -- 172.21.15.28:0/4288217832 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe17c150450 msgr2=0x7fe17c0a5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:55.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.187+0000 7fe17b400640 1 --2- 172.21.15.28:0/4288217832 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe17c150450 0x7fe17c0a5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:55.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.187+0000 7fe17b400640 1 -- 172.21.15.28:0/4288217832 >> v2:172.21.15.81:3300/0 conn(0x7fe17c06ee40 msgr2=0x7fe17c1536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:55.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.187+0000 7fe17b400640 1 --2- 172.21.15.28:0/4288217832 >> v2:172.21.15.81:3300/0 conn(0x7fe17c06ee40 0x7fe17c1536f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:55.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.187+0000 7fe17b400640 1 -- 172.21.15.28:0/4288217832 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe17c153c30 con 0x7fe17c156af0 2024-02-21T23:03:55.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.187+0000 7fe17b400640 1 -- 172.21.15.28:0/4288217832 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fe1740019f0 con 0x7fe17c156af0 2024-02-21T23:03:55.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.187+0000 7fe17b400640 1 -- 172.21.15.28:0/4288217832 >> v1:172.21.15.28:6789/0 conn(0x7fe17c156af0 legacy=0x7fe17c06dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:55.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.187+0000 7fe17b400640 1 --2- 172.21.15.28:0/4288217832 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe17c150450 0x7fe1640039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:55.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.187+0000 7fe17b400640 1 --2- 172.21.15.28:0/4288217832 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe164003f30 0x7fe164004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:55.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.187+0000 7fe17b400640 1 --2- 172.21.15.28:0/4288217832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe164004d40 0x7fe1640051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:55.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.187+0000 7fe17b400640 1 -- 172.21.15.28:0/4288217832 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe17c156660 con 0x7fe164004d40 2024-02-21T23:03:55.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.187+0000 7fe17b400640 1 -- 172.21.15.28:0/4288217832 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe16400b9c0 con 0x7fe164003f30 2024-02-21T23:03:55.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.187+0000 7fe17b400640 1 -- 172.21.15.28:0/4288217832 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe16400bb40 con 0x7fe17c150450 2024-02-21T23:03:55.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.187+0000 7fe181600640 1 --2- 172.21.15.28:0/4288217832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe164004d40 0x7fe1640051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:55.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.188+0000 7fe17be00640 1 --2- 172.21.15.28:0/4288217832 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe164003f30 0x7fe164004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:55.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.188+0000 7fe180c00640 1 --2- 172.21.15.28:0/4288217832 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe17c150450 0x7fe1640039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:55.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.188+0000 7fe181600640 1 -- 172.21.15.28:0/4288217832 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe17c150450 msgr2=0x7fe1640039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:55.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.188+0000 7fe181600640 1 --2- 172.21.15.28:0/4288217832 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe17c150450 0x7fe1640039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:55.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.188+0000 7fe181600640 1 -- 172.21.15.28:0/4288217832 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe164003f30 msgr2=0x7fe164004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:55.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.188+0000 7fe181600640 1 --2- 172.21.15.28:0/4288217832 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe164003f30 0x7fe164004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:55.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.188+0000 7fe181600640 1 -- 172.21.15.28:0/4288217832 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe164003520 con 0x7fe164004d40 2024-02-21T23:03:55.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.188+0000 7fe17be00640 1 --2- 172.21.15.28:0/4288217832 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe164003f30 0x7fe164004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:03:55.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.189+0000 7fe181600640 1 --2- 172.21.15.28:0/4288217832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe164004d40 0x7fe1640051c0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fe17400a920 tx=0x7fe164005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfa5bf0836ad63bb server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:55.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.189+0000 7fe17b400640 1 -- 172.21.15.28:0/4288217832 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe17400a4c0 con 0x7fe164004d40 2024-02-21T23:03:55.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.189+0000 7fe17b400640 1 -- 172.21.15.28:0/4288217832 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe174034650 con 0x7fe164004d40 2024-02-21T23:03:55.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.189+0000 7fe17b400640 1 -- 172.21.15.28:0/4288217832 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe174034a80 con 0x7fe164004d40 2024-02-21T23:03:55.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.190+0000 7fe182000640 1 -- 172.21.15.28:0/4288217832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe164004d40 msgr2=0x7fe1640051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:55.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.190+0000 7fe182000640 1 --2- 172.21.15.28:0/4288217832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe164004d40 0x7fe1640051c0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fe17400a920 tx=0x7fe164005830 comp rx=0 tx=0).stop 2024-02-21T23:03:55.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.190+0000 7fe181600640 1 -- 172.21.15.28:0/4288217832 reap_dead start 2024-02-21T23:03:55.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.190+0000 7fe182000640 1 -- 172.21.15.28:0/4288217832 shutdown_connections 2024-02-21T23:03:55.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.190+0000 7fe182000640 1 -- 172.21.15.28:0/4288217832 >> 172.21.15.28:0/4288217832 conn(0x7fe17c06b630 msgr2=0x7fe17c0c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:55.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.190+0000 7fe182000640 1 -- 172.21.15.28:0/4288217832 shutdown_connections 2024-02-21T23:03:55.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.190+0000 7fe182000640 1 -- 172.21.15.28:0/4288217832 wait complete. 2024-02-21T23:03:55.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.192+0000 7fe182000640 1 Processor -- start 2024-02-21T23:03:55.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.192+0000 7fe182000640 1 -- start start 2024-02-21T23:03:55.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.193+0000 7fe182000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe17c0a6f20 0x7fe17c149770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:55.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.193+0000 7fe182000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe17c072e50 0x7fe17c149cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:55.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.193+0000 7fe182000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe17c1ed9a0 0x7fe17c1ede50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:55.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.193+0000 7fe182000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe17c0c2cb0 con 0x7fe17c072e50 2024-02-21T23:03:55.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.193+0000 7fe182000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe17c0c29d0 con 0x7fe17c0a6f20 2024-02-21T23:03:55.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.193+0000 7fe182000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe17c06e700 con 0x7fe17c1ed9a0 2024-02-21T23:03:55.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.193+0000 7fe17be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe17c072e50 0x7fe17c149cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:55.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.193+0000 7fe17be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe17c072e50 0x7fe17c149cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45444/0 (socket says 172.21.15.28:45444) 2024-02-21T23:03:55.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.193+0000 7fe17be00640 1 -- 172.21.15.28:0/695851073 learned_addr learned my addr 172.21.15.28:0/695851073 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:55.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.193+0000 7fe180c00640 1 --2- 172.21.15.28:0/695851073 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe17c0a6f20 0x7fe17c149770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:55.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.194+0000 7fe181600640 1 --2- 172.21.15.28:0/695851073 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe17c1ed9a0 0x7fe17c1ede50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:55.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.194+0000 7fe17be00640 1 -- 172.21.15.28:0/695851073 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe17c1ed9a0 msgr2=0x7fe17c1ede50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:55.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.194+0000 7fe17be00640 1 --2- 172.21.15.28:0/695851073 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe17c1ed9a0 0x7fe17c1ede50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:55.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.194+0000 7fe17be00640 1 -- 172.21.15.28:0/695851073 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe17c0a6f20 msgr2=0x7fe17c149770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:55.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.194+0000 7fe17be00640 1 --2- 172.21.15.28:0/695851073 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe17c0a6f20 0x7fe17c149770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:55.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.194+0000 7fe17be00640 1 -- 172.21.15.28:0/695851073 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe17c071d80 con 0x7fe17c072e50 2024-02-21T23:03:55.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.194+0000 7fe17be00640 1 --2- 172.21.15.28:0/695851073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe17c072e50 0x7fe17c149cb0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fe16c010a00 tx=0x7fe16c010ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:55.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.194+0000 7fe178c00640 1 -- 172.21.15.28:0/695851073 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe16c00f040 con 0x7fe17c072e50 2024-02-21T23:03:55.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.194+0000 7fe178c00640 1 -- 172.21.15.28:0/695851073 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe16c00ec20 con 0x7fe17c072e50 2024-02-21T23:03:55.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.194+0000 7fe178c00640 1 -- 172.21.15.28:0/695851073 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe16c01c400 con 0x7fe17c072e50 2024-02-21T23:03:55.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.194+0000 7fe182000640 1 -- 172.21.15.28:0/695851073 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe17c072060 con 0x7fe17c072e50 2024-02-21T23:03:55.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.195+0000 7fe182000640 1 -- 172.21.15.28:0/695851073 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe17c0c4bd0 con 0x7fe17c072e50 2024-02-21T23:03:55.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.196+0000 7fe182000640 1 -- 172.21.15.28:0/695851073 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe17c04f290 con 0x7fe17c072e50 2024-02-21T23:03:55.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.197+0000 7fe178c00640 1 -- 172.21.15.28:0/695851073 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe16c01c5a0 con 0x7fe17c072e50 2024-02-21T23:03:55.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.197+0000 7fe178c00640 1 --2- 172.21.15.28:0/695851073 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe16006b4d0 0x7fe16006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:55.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.198+0000 7fe178c00640 1 -- 172.21.15.28:0/695851073 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6735+0+0 (secure 0 0 0) 0x7fe16c08ae50 con 0x7fe17c072e50 2024-02-21T23:03:55.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.198+0000 7fe180c00640 1 --2- 172.21.15.28:0/695851073 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe16006b4d0 0x7fe16006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:55.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.199+0000 7fe180c00640 1 --2- 172.21.15.28:0/695851073 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe16006b4d0 0x7fe16006d990 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe170009a10 tx=0x7fe170008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:55.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.202+0000 7fe178c00640 1 -- 172.21.15.28:0/695851073 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe16c018030 con 0x7fe17c072e50 2024-02-21T23:03:55.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.403+0000 7fe182000640 1 -- 172.21.15.28:0/695851073 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe17c0c1f10 con 0x7fe17c072e50 2024-02-21T23:03:55.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.404+0000 7fe178c00640 1 -- 172.21.15.28:0/695851073 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+18843 (secure 0 0 0) 0x7fe16c059830 con 0x7fe17c072e50 2024-02-21T23:03:55.395 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:03:55.395 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":26,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:03:23.860109+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:02:35.795158+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-21T23:02:36.840110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-02-21T23:02:40.441388+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":3,"to":2}]},{"pgid":"3.12","mappings":[{"from":0,"to":1}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:03:55.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.408+0000 7fe182000640 1 -- 172.21.15.28:0/695851073 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe16006b4d0 msgr2=0x7fe16006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:55.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.408+0000 7fe182000640 1 --2- 172.21.15.28:0/695851073 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe16006b4d0 0x7fe16006d990 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe170009a10 tx=0x7fe170008040 comp rx=0 tx=0).stop 2024-02-21T23:03:55.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.408+0000 7fe182000640 1 -- 172.21.15.28:0/695851073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe17c072e50 msgr2=0x7fe17c149cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:55.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.408+0000 7fe182000640 1 --2- 172.21.15.28:0/695851073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe17c072e50 0x7fe17c149cb0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fe16c010a00 tx=0x7fe16c010ec0 comp rx=0 tx=0).stop 2024-02-21T23:03:55.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.408+0000 7fe182000640 1 -- 172.21.15.28:0/695851073 shutdown_connections 2024-02-21T23:03:55.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.408+0000 7fe182000640 1 --2- 172.21.15.28:0/695851073 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe16006b4d0 0x7fe16006d990 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:55.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.408+0000 7fe182000640 1 --2- 172.21.15.28:0/695851073 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe17c1ed9a0 0x7fe17c1ede50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:55.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.408+0000 7fe182000640 1 --2- 172.21.15.28:0/695851073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe17c072e50 0x7fe17c149cb0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:55.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.408+0000 7fe182000640 1 --2- 172.21.15.28:0/695851073 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe17c0a6f20 0x7fe17c149770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:55.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.408+0000 7fe182000640 1 -- 172.21.15.28:0/695851073 >> 172.21.15.28:0/695851073 conn(0x7fe17c06b630 msgr2=0x7fe17c14b060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:55.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.408+0000 7fe182000640 1 -- 172.21.15.28:0/695851073 shutdown_connections 2024-02-21T23:03:55.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.408+0000 7fe182000640 1 -- 172.21.15.28:0/695851073 wait complete. 2024-02-21T23:03:55.418 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-02-21T23:03:55.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.652+0000 7f086de00640 1 Processor -- start 2024-02-21T23:03:55.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.652+0000 7f086de00640 1 -- start start 2024-02-21T23:03:55.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.653+0000 7f086de00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f08680c2df0 0x7f0868156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:55.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.653+0000 7f086de00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f08680c5f20 0x7f08680c31d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:55.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.653+0000 7f086de00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08680c5c10 con 0x7f08680c3710 2024-02-21T23:03:55.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.653+0000 7f086de00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f086806e6a0 con 0x7f08680c5f20 2024-02-21T23:03:55.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.653+0000 7f086de00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f086806e8d0 con 0x7f08680c2df0 2024-02-21T23:03:55.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.654+0000 7f085fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f08680c5f20 0x7f08680c31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:55.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.654+0000 7f0867400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f08680c3710 0x7f0868153220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:40414/0 (socket says 172.21.15.28:40414) 2024-02-21T23:03:55.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.654+0000 7f0867400640 1 -- 172.21.15.28:0/180696477 learned_addr learned my addr 172.21.15.28:0/180696477 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:03:55.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.654+0000 7f0866a00640 1 --2- 172.21.15.28:0/180696477 >> v2:172.21.15.81:3300/0 conn(0x7f08680c2df0 0x7f0868156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:55.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.654+0000 7f085fe00640 1 -- 172.21.15.28:0/180696477 >> v2:172.21.15.81:3300/0 conn(0x7f08680c2df0 msgr2=0x7f0868156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:55.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.654+0000 7f085fe00640 1 --2- 172.21.15.28:0/180696477 >> v2:172.21.15.81:3300/0 conn(0x7f08680c2df0 0x7f0868156ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:55.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.654+0000 7f085fe00640 1 -- 172.21.15.28:0/180696477 >> v1:172.21.15.28:6789/0 conn(0x7f08680c3710 legacy=0x7f0868153220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:55.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.654+0000 7f085fe00640 1 -- 172.21.15.28:0/180696477 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f086806ebb0 con 0x7f08680c5f20 2024-02-21T23:03:55.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.655+0000 7f085fe00640 1 --2- 172.21.15.28:0/180696477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08680c5f20 0x7f08680c31d0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f086806f760 tx=0x7f085002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51f172163d2d971 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:55.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.656+0000 7f0866000640 1 -- 172.21.15.28:0/180696477 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0850035020 con 0x7f08680c5f20 2024-02-21T23:03:55.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.656+0000 7f0866000640 1 -- 172.21.15.28:0/180696477 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0850031830 con 0x7f08680c5f20 2024-02-21T23:03:55.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.656+0000 7f0866000640 1 -- 172.21.15.28:0/180696477 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0850031c00 con 0x7f08680c5f20 2024-02-21T23:03:55.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.657+0000 7f086de00640 1 -- 172.21.15.28:0/180696477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08680c5f20 msgr2=0x7f08680c31d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:55.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.657+0000 7f086de00640 1 --2- 172.21.15.28:0/180696477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08680c5f20 0x7f08680c31d0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f086806f760 tx=0x7f085002b8d0 comp rx=0 tx=0).stop 2024-02-21T23:03:55.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.657+0000 7f086de00640 1 -- 172.21.15.28:0/180696477 shutdown_connections 2024-02-21T23:03:55.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.657+0000 7f086de00640 1 --2- 172.21.15.28:0/180696477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08680c5f20 0x7f08680c31d0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:55.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.657+0000 7f086de00640 1 --2- 172.21.15.28:0/180696477 >> v2:172.21.15.81:3300/0 conn(0x7f08680c2df0 0x7f0868156ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:55.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.657+0000 7f086de00640 1 -- 172.21.15.28:0/180696477 >> 172.21.15.28:0/180696477 conn(0x7f086806b820 msgr2=0x7f08680c67f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:55.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.657+0000 7f086de00640 1 -- 172.21.15.28:0/180696477 shutdown_connections 2024-02-21T23:03:55.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.657+0000 7f086de00640 1 -- 172.21.15.28:0/180696477 wait complete. 2024-02-21T23:03:55.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.658+0000 7f086de00640 1 Processor -- start 2024-02-21T23:03:55.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.659+0000 7f086de00640 1 -- start start 2024-02-21T23:03:55.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.659+0000 7f086de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08680c2df0 0x7f08681eb8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:55.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.659+0000 7f086de00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f08680c3710 0x7f08680bfc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:55.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.659+0000 7f086de00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f08680c5f20 0x7f08681ee890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:55.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.660+0000 7f086de00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0868153930 con 0x7f08680c2df0 2024-02-21T23:03:55.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.660+0000 7f086de00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f086806f170 con 0x7f08680c3710 2024-02-21T23:03:55.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.660+0000 7f086de00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0868153ab0 con 0x7f08680c5f20 2024-02-21T23:03:55.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.660+0000 7f0866a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08680c2df0 0x7f08681eb8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:55.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.660+0000 7f0866a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08680c2df0 0x7f08681eb8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45462/0 (socket says 172.21.15.28:45462) 2024-02-21T23:03:55.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.660+0000 7f0866a00640 1 -- 172.21.15.28:0/2506197578 learned_addr learned my addr 172.21.15.28:0/2506197578 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:55.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.660+0000 7f0867400640 1 --2- 172.21.15.28:0/2506197578 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f08680c5f20 0x7f08681ee890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:55.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.660+0000 7f085fe00640 1 --2- 172.21.15.28:0/2506197578 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f08680c3710 0x7f08680bfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:55.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.660+0000 7f0866a00640 1 -- 172.21.15.28:0/2506197578 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f08680c5f20 msgr2=0x7f08681ee890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:55.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.660+0000 7f0866a00640 1 --2- 172.21.15.28:0/2506197578 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f08680c5f20 0x7f08681ee890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:55.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.660+0000 7f0866a00640 1 -- 172.21.15.28:0/2506197578 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f08680c3710 msgr2=0x7f08680bfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:55.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.660+0000 7f0866a00640 1 --2- 172.21.15.28:0/2506197578 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f08680c3710 0x7f08680bfc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:55.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.660+0000 7f0866a00640 1 -- 172.21.15.28:0/2506197578 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08681eaf40 con 0x7f08680c2df0 2024-02-21T23:03:55.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.660+0000 7f0867400640 1 --2- 172.21.15.28:0/2506197578 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f08680c5f20 0x7f08681ee890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:03:55.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.660+0000 7f085fe00640 1 --2- 172.21.15.28:0/2506197578 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f08680c3710 0x7f08680bfc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:03:55.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.661+0000 7f0866a00640 1 --2- 172.21.15.28:0/2506197578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08680c2df0 0x7f08681eb8e0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f085800d800 tx=0x7f085800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:55.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.661+0000 7f085f400640 1 -- 172.21.15.28:0/2506197578 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f085800b660 con 0x7f08680c2df0 2024-02-21T23:03:55.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.661+0000 7f085f400640 1 -- 172.21.15.28:0/2506197578 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0858019070 con 0x7f08680c2df0 2024-02-21T23:03:55.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.661+0000 7f086de00640 1 -- 172.21.15.28:0/2506197578 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f086814fe50 con 0x7f08680c2df0 2024-02-21T23:03:55.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.661+0000 7f085f400640 1 -- 172.21.15.28:0/2506197578 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f085800ba20 con 0x7f08680c2df0 2024-02-21T23:03:55.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.661+0000 7f086de00640 1 -- 172.21.15.28:0/2506197578 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f086814eff0 con 0x7f08680c2df0 2024-02-21T23:03:55.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.663+0000 7f086de00640 1 -- 172.21.15.28:0/2506197578 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f086814f230 con 0x7f08680c2df0 2024-02-21T23:03:55.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.663+0000 7f085f400640 1 -- 172.21.15.28:0/2506197578 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f085800bbc0 con 0x7f08680c2df0 2024-02-21T23:03:55.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.664+0000 7f085f400640 1 --2- 172.21.15.28:0/2506197578 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f084406b500 0x7f084406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:55.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.664+0000 7f085f400640 1 -- 172.21.15.28:0/2506197578 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f085808aa10 con 0x7f08680c2df0 2024-02-21T23:03:55.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.665+0000 7f085fe00640 1 --2- 172.21.15.28:0/2506197578 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f084406b500 0x7f084406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:55.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.666+0000 7f085fe00640 1 --2- 172.21.15.28:0/2506197578 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f084406b500 0x7f084406d9c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f0850036000 tx=0x7f0850008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:55.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.669+0000 7f085f400640 1 -- 172.21.15.28:0/2506197578 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f086814f230 con 0x7f08680c2df0 2024-02-21T23:03:55.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.909+0000 7f086de00640 1 -- 172.21.15.28:0/2506197578 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f086806f400 con 0x7f08680c2df0 2024-02-21T23:03:55.900 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-02-21T23:03:55.910+0000 7f5ddc400640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-21T23:03:55.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.914+0000 7f085f400640 1 -- 172.21.15.28:0/2506197578 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v10) v1 ==== 161+0+0 (secure 0 0 0) 0x7f0858059480 con 0x7f08680c2df0 2024-02-21T23:03:55.904 INFO:teuthology.orchestra.run.smithi028.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-02-21T23:03:55.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.918+0000 7f086de00640 1 -- 172.21.15.28:0/2506197578 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f084406b500 msgr2=0x7f084406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:55.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.918+0000 7f086de00640 1 --2- 172.21.15.28:0/2506197578 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f084406b500 0x7f084406d9c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f0850036000 tx=0x7f0850008030 comp rx=0 tx=0).stop 2024-02-21T23:03:55.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.918+0000 7f086de00640 1 -- 172.21.15.28:0/2506197578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08680c2df0 msgr2=0x7f08681eb8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:55.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.918+0000 7f086de00640 1 --2- 172.21.15.28:0/2506197578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08680c2df0 0x7f08681eb8e0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f085800d800 tx=0x7f085800dcc0 comp rx=0 tx=0).stop 2024-02-21T23:03:55.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.918+0000 7f086de00640 1 -- 172.21.15.28:0/2506197578 shutdown_connections 2024-02-21T23:03:55.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.918+0000 7f086de00640 1 --2- 172.21.15.28:0/2506197578 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f084406b500 0x7f084406d9c0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:55.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.918+0000 7f086de00640 1 --2- 172.21.15.28:0/2506197578 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f08680c5f20 0x7f08681ee890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:55.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.918+0000 7f086de00640 1 --2- 172.21.15.28:0/2506197578 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f08680c3710 0x7f08680bfc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:55.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.918+0000 7f086de00640 1 --2- 172.21.15.28:0/2506197578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08680c2df0 0x7f08681eb8e0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:55.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.918+0000 7f086de00640 1 -- 172.21.15.28:0/2506197578 >> 172.21.15.28:0/2506197578 conn(0x7f086806b820 msgr2=0x7f08680c4b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:55.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.918+0000 7f086de00640 1 -- 172.21.15.28:0/2506197578 shutdown_connections 2024-02-21T23:03:55.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:55.918+0000 7f086de00640 1 -- 172.21.15.28:0/2506197578 wait complete. 2024-02-21T23:03:55.926 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-02-21T23:03:55.934 INFO:tasks.ceph.mds.b.smithi081.stderr: -4610> 2024-02-21T23:02:41.541+0000 7fc410c00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-21T23:03:56.007 INFO:tasks.ceph.mds.b.smithi081.stdout:starting mds.b at 2024-02-21T23:03:56.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:56.136+0000 7fb39be00640 1 Processor -- start 2024-02-21T23:03:56.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:56.136+0000 7fb39be00640 1 -- start start 2024-02-21T23:03:56.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:56.138+0000 7fb39be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb39c06f400 0x7fb39c06e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:56.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:56.138+0000 7fb39be00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb39c06db10 con 0x7fb39c0c2c60 2024-02-21T23:03:56.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:56.138+0000 7fb39be00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb39c1539a0 con 0x7fb39c1500f0 2024-02-21T23:03:56.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:56.138+0000 7fb39be00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb39c0c1080 con 0x7fb39c06f400 2024-02-21T23:03:56.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:56.138+0000 7fb39a000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb39c0c2c60 0x7fb39c06dff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:40430/0 (socket says 172.21.15.28:40430) 2024-02-21T23:03:56.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:56.138+0000 7fb39a000640 1 -- 172.21.15.28:0/349185777 learned_addr learned my addr 172.21.15.28:0/349185777 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:03:56.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:56.139+0000 7fb39b400640 1 --2- 172.21.15.28:0/349185777 >> v2:172.21.15.28:3300/0 conn(0x7fb39c06f400 0x7fb39c06e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:56.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:56.139+0000 7fb399600640 1 -- 172.21.15.28:0/349185777 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2117739590 0 0) 0x7fb39c06db10 con 0x7fb39c0c2c60 2024-02-21T23:03:56.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:56.139+0000 7fb399600640 1 -- 172.21.15.28:0/349185777 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb380003520 con 0x7fb39c0c2c60 2024-02-21T23:03:56.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:56.139+0000 7fb39b400640 1 -- 172.21.15.28:0/349185777 >> v1:172.21.15.81:6789/0 conn(0x7fb39c1500f0 legacy=0x7fb39c06ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:56.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:56.139+0000 7fb39b400640 1 -- 172.21.15.28:0/349185777 >> v1:172.21.15.28:6789/0 conn(0x7fb39c0c2c60 legacy=0x7fb39c06dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:56.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:56.139+0000 7fb39b400640 1 -- 172.21.15.28:0/349185777 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb39c0c12d0 con 0x7fb39c06f400 2024-02-21T23:03:56.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:56.140+0000 7fb39b400640 1 --2- 172.21.15.28:0/349185777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb39c06f400 0x7fb39c06e700 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fb394009d90 tx=0x7fb39402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36e809d19c19c6ad server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:56.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:56.141+0000 7fb399600640 1 -- 172.21.15.28:0/349185777 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb394039070 con 0x7fb39c06f400 2024-02-21T23:03:56.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:56.141+0000 7fb399600640 1 -- 172.21.15.28:0/349185777 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb39402cd00 con 0x7fb39c06f400 2024-02-21T23:03:56.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:56.141+0000 7fb399600640 1 -- 172.21.15.28:0/349185777 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb394034640 con 0x7fb39c06f400 2024-02-21T23:03:56.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:56.141+0000 7fb39be00640 1 -- 172.21.15.28:0/349185777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb39c06f400 msgr2=0x7fb39c06e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:56.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:56.141+0000 7fb39be00640 1 --2- 172.21.15.28:0/349185777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb39c06f400 0x7fb39c06e700 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fb394009d90 tx=0x7fb39402bf50 comp rx=0 tx=0).stop 2024-02-21T23:03:56.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:56.141+0000 7fb39be00640 1 -- 172.21.15.28:0/349185777 shutdown_connections 2024-02-21T23:03:56.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:56.141+0000 7fb39be00640 1 --2- 172.21.15.28:0/349185777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb39c06f400 0x7fb39c06e700 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:56.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:56.141+0000 7fb39be00640 1 -- 172.21.15.28:0/349185777 >> 172.21.15.28:0/349185777 conn(0x7fb39c0a8530 msgr2=0x7fb39c0a8910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:56.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:56.142+0000 7fb39be00640 1 -- 172.21.15.28:0/349185777 shutdown_connections 2024-02-21T23:03:56.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:56.142+0000 7fb39be00640 1 -- 172.21.15.28:0/349185777 wait complete. 2024-02-21T23:03:56.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:56.143+0000 7fb39be00640 1 Processor -- start 2024-02-21T23:03:56.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:56.143+0000 7fb39be00640 1 -- start start 2024-02-21T23:03:56.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:56.144+0000 7fb39be00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb39c06f400 0x7fb39c0729f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:56.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:56.144+0000 7fb39be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb39c0c2c60 0x7fb39c1eb710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:56.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:56.144+0000 7fb39be00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb39c1500f0 0x7fb39c0bfe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:56.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:56.144+0000 7fb39be00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb39c06b580 con 0x7fb39c0c2c60 2024-02-21T23:03:56.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:56.144+0000 7fb39be00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb39c1504d0 con 0x7fb39c1500f0 2024-02-21T23:03:56.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:56.144+0000 7fb39be00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb39c06b700 con 0x7fb39c06f400 2024-02-21T23:03:56.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:56.145+0000 7fb39a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb39c0c2c60 0x7fb39c1eb710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:56.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:56.145+0000 7fb39a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb39c0c2c60 0x7fb39c1eb710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45484/0 (socket says 172.21.15.28:45484) 2024-02-21T23:03:56.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:56.145+0000 7fb39a000640 1 -- 172.21.15.28:0/3823385689 learned_addr learned my addr 172.21.15.28:0/3823385689 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:56.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:56.145+0000 7fb39aa00640 1 --2- 172.21.15.28:0/3823385689 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb39c06f400 0x7fb39c0729f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:56.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:56.145+0000 7fb39a000640 1 -- 172.21.15.28:0/3823385689 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb39c06f400 msgr2=0x7fb39c0729f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:56.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:56.145+0000 7fb39a000640 1 --2- 172.21.15.28:0/3823385689 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb39c06f400 0x7fb39c0729f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:56.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:56.145+0000 7fb39a000640 1 -- 172.21.15.28:0/3823385689 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb39c1500f0 msgr2=0x7fb39c0bfe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:56.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:56.145+0000 7fb39a000640 1 --2- 172.21.15.28:0/3823385689 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb39c1500f0 0x7fb39c0bfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:56.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:56.145+0000 7fb39a000640 1 -- 172.21.15.28:0/3823385689 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb39c073040 con 0x7fb39c0c2c60 2024-02-21T23:03:56.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:56.145+0000 7fb39aa00640 1 --2- 172.21.15.28:0/3823385689 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb39c06f400 0x7fb39c0729f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:03:56.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:56.145+0000 7fb39a000640 1 --2- 172.21.15.28:0/3823385689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb39c0c2c60 0x7fb39c1eb710 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fb38400c500 tx=0x7fb38400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:56.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:56.146+0000 7fb392a00640 1 -- 172.21.15.28:0/3823385689 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb384017070 con 0x7fb39c0c2c60 2024-02-21T23:03:56.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:56.146+0000 7fb392a00640 1 -- 172.21.15.28:0/3823385689 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb384005ce0 con 0x7fb39c0c2c60 2024-02-21T23:03:56.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:56.146+0000 7fb392a00640 1 -- 172.21.15.28:0/3823385689 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3840072f0 con 0x7fb39c0c2c60 2024-02-21T23:03:56.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:56.146+0000 7fb39be00640 1 -- 172.21.15.28:0/3823385689 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb39c071440 con 0x7fb39c0c2c60 2024-02-21T23:03:56.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:56.146+0000 7fb39be00640 1 -- 172.21.15.28:0/3823385689 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb39c0c0360 con 0x7fb39c0c2c60 2024-02-21T23:03:56.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:56.147+0000 7fb39be00640 1 -- 172.21.15.28:0/3823385689 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb39c04f290 con 0x7fb39c0c2c60 2024-02-21T23:03:56.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:56.148+0000 7fb392a00640 1 -- 172.21.15.28:0/3823385689 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb38401a070 con 0x7fb39c0c2c60 2024-02-21T23:03:56.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:56.149+0000 7fb392a00640 1 --2- 172.21.15.28:0/3823385689 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb37806b4d0 0x7fb37806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:56.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:56.149+0000 7fb392a00640 1 -- 172.21.15.28:0/3823385689 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6821+0+0 (secure 0 0 0) 0x7fb3840899f0 con 0x7fb39c0c2c60 2024-02-21T23:03:56.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:56.150+0000 7fb39aa00640 1 --2- 172.21.15.28:0/3823385689 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb37806b4d0 0x7fb37806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:56.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:56.151+0000 7fb39aa00640 1 --2- 172.21.15.28:0/3823385689 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb37806b4d0 0x7fb37806d990 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fb38c000f60 tx=0x7fb38c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:56.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:56.153+0000 7fb392a00640 1 -- 172.21.15.28:0/3823385689 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb384057db0 con 0x7fb39c0c2c60 2024-02-21T23:03:56.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:56.400+0000 7fb39be00640 1 -- 172.21.15.28:0/3823385689 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fb39c06f1e0 con 0x7fb39c0c2c60 2024-02-21T23:03:56.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:56.918+0000 7fb392a00640 1 -- 172.21.15.28:0/3823385689 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v11) v1 ==== 104+0+0 (secure 0 0 0) 0x7fb38405be50 con 0x7fb39c0c2c60 2024-02-21T23:03:56.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:56.921+0000 7fb39be00640 1 -- 172.21.15.28:0/3823385689 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb37806b4d0 msgr2=0x7fb37806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:56.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:56.921+0000 7fb39be00640 1 --2- 172.21.15.28:0/3823385689 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb37806b4d0 0x7fb37806d990 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fb38c000f60 tx=0x7fb38c009040 comp rx=0 tx=0).stop 2024-02-21T23:03:56.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:56.921+0000 7fb39be00640 1 -- 172.21.15.28:0/3823385689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb39c0c2c60 msgr2=0x7fb39c1eb710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:56.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:56.921+0000 7fb39be00640 1 --2- 172.21.15.28:0/3823385689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb39c0c2c60 0x7fb39c1eb710 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fb38400c500 tx=0x7fb38400c9c0 comp rx=0 tx=0).stop 2024-02-21T23:03:56.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:56.921+0000 7fb39be00640 1 -- 172.21.15.28:0/3823385689 shutdown_connections 2024-02-21T23:03:56.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:56.921+0000 7fb39be00640 1 --2- 172.21.15.28:0/3823385689 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb37806b4d0 0x7fb37806d990 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:56.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:56.921+0000 7fb39be00640 1 --2- 172.21.15.28:0/3823385689 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb39c1500f0 0x7fb39c0bfe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:56.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:56.921+0000 7fb39be00640 1 --2- 172.21.15.28:0/3823385689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb39c0c2c60 0x7fb39c1eb710 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:56.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:56.921+0000 7fb39be00640 1 --2- 172.21.15.28:0/3823385689 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb39c06f400 0x7fb39c0729f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:56.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:56.921+0000 7fb39be00640 1 -- 172.21.15.28:0/3823385689 >> 172.21.15.28:0/3823385689 conn(0x7fb39c0a8530 msgr2=0x7fb39c14b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:56.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:56.922+0000 7fb39be00640 1 -- 172.21.15.28:0/3823385689 shutdown_connections 2024-02-21T23:03:56.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:56.922+0000 7fb39be00640 1 -- 172.21.15.28:0/3823385689 wait complete. 2024-02-21T23:03:56.934 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-02-21T23:03:57.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.169+0000 7f8652400640 1 Processor -- start 2024-02-21T23:03:57.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.169+0000 7f8652400640 1 -- start start 2024-02-21T23:03:57.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.171+0000 7f8652400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f864c1501c0 0x7f864c0a96c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:57.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.171+0000 7f8652400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f864c152fd0 con 0x7f864c0a9c00 2024-02-21T23:03:57.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.171+0000 7f8652400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f864c1532b0 con 0x7f864c0c1010 2024-02-21T23:03:57.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.171+0000 7f8652400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f864c153590 con 0x7f864c1501c0 2024-02-21T23:03:57.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.171+0000 7f864be00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f864c0a9c00 0x7f864c06e330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:40436/0 (socket says 172.21.15.28:40436) 2024-02-21T23:03:57.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.171+0000 7f864be00640 1 -- 172.21.15.28:0/1568810409 learned_addr learned my addr 172.21.15.28:0/1568810409 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:03:57.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.172+0000 7f8651000640 1 --2- 172.21.15.28:0/1568810409 >> v2:172.21.15.81:3300/0 conn(0x7f864c1501c0 0x7f864c0a96c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:57.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.172+0000 7f864b400640 1 -- 172.21.15.28:0/1568810409 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1042642925 0 0) 0x7f864c152fd0 con 0x7f864c0a9c00 2024-02-21T23:03:57.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.172+0000 7f864b400640 1 -- 172.21.15.28:0/1568810409 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8634007930 con 0x7f864c0a9c00 2024-02-21T23:03:57.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.172+0000 7f864b400640 1 -- 172.21.15.28:0/1568810409 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4021877310 0 0) 0x7f8634007930 con 0x7f864c0a9c00 2024-02-21T23:03:57.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.172+0000 7f864b400640 1 -- 172.21.15.28:0/1568810409 >> v2:172.21.15.81:3300/0 conn(0x7f864c1501c0 msgr2=0x7f864c0a96c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:57.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.172+0000 7f864b400640 1 --2- 172.21.15.28:0/1568810409 >> v2:172.21.15.81:3300/0 conn(0x7f864c1501c0 0x7f864c0a96c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:57.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.172+0000 7f864b400640 1 -- 172.21.15.28:0/1568810409 >> v1:172.21.15.81:6789/0 conn(0x7f864c0c1010 legacy=0x7f864c06ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:57.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.172+0000 7f864b400640 1 -- 172.21.15.28:0/1568810409 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f864c153870 con 0x7f864c0a9c00 2024-02-21T23:03:57.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.172+0000 7f864b400640 1 -- 172.21.15.28:0/1568810409 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f8638001670 con 0x7f864c0a9c00 2024-02-21T23:03:57.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.172+0000 7f864b400640 1 -- 172.21.15.28:0/1568810409 >> v1:172.21.15.28:6789/0 conn(0x7f864c0a9c00 legacy=0x7f864c06e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:57.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.172+0000 7f864b400640 1 --2- 172.21.15.28:0/1568810409 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8634007e10 0x7f8634008230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:57.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.172+0000 7f864b400640 1 --2- 172.21.15.28:0/1568810409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8634008c10 0x7f8634009090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:57.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.172+0000 7f864b400640 1 --2- 172.21.15.28:0/1568810409 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f86340096f0 0x7f86340145c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:57.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.172+0000 7f864b400640 1 -- 172.21.15.28:0/1568810409 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f864c153590 con 0x7f8634008c10 2024-02-21T23:03:57.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.172+0000 7f864b400640 1 -- 172.21.15.28:0/1568810409 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8634014b00 con 0x7f8634007e10 2024-02-21T23:03:57.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.172+0000 7f864b400640 1 -- 172.21.15.28:0/1568810409 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8634014c80 con 0x7f86340096f0 2024-02-21T23:03:57.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.173+0000 7f864be00640 1 --2- 172.21.15.28:0/1568810409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8634008c10 0x7f8634009090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:57.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.173+0000 7f864be00640 1 -- 172.21.15.28:0/1568810409 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f86340096f0 msgr2=0x7f86340145c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:57.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.173+0000 7f864be00640 1 --2- 172.21.15.28:0/1568810409 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f86340096f0 0x7f86340145c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:57.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.173+0000 7f864be00640 1 -- 172.21.15.28:0/1568810409 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8634007e10 msgr2=0x7f8634008230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:57.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.173+0000 7f864be00640 1 --2- 172.21.15.28:0/1568810409 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8634007e10 0x7f8634008230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:57.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.173+0000 7f864be00640 1 -- 172.21.15.28:0/1568810409 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8634007930 con 0x7f8634008c10 2024-02-21T23:03:57.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.173+0000 7f8651a00640 1 -- 172.21.15.28:0/1568810409 reap_dead start 2024-02-21T23:03:57.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.174+0000 7f864be00640 1 --2- 172.21.15.28:0/1568810409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8634008c10 0x7f8634009090 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f863800c110 tx=0x7f8634009c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47403bc1f726dbc3 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:57.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.174+0000 7f864b400640 1 -- 172.21.15.28:0/1568810409 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8638007c20 con 0x7f8634008c10 2024-02-21T23:03:57.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.174+0000 7f864b400640 1 -- 172.21.15.28:0/1568810409 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8638030070 con 0x7f8634008c10 2024-02-21T23:03:57.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.174+0000 7f864b400640 1 -- 172.21.15.28:0/1568810409 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8638005d70 con 0x7f8634008c10 2024-02-21T23:03:57.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.175+0000 7f8652400640 1 -- 172.21.15.28:0/1568810409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8634008c10 msgr2=0x7f8634009090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:57.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.175+0000 7f8652400640 1 --2- 172.21.15.28:0/1568810409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8634008c10 0x7f8634009090 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f863800c110 tx=0x7f8634009c40 comp rx=0 tx=0).stop 2024-02-21T23:03:57.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.175+0000 7f8652400640 1 -- 172.21.15.28:0/1568810409 shutdown_connections 2024-02-21T23:03:57.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.175+0000 7f8652400640 1 --2- 172.21.15.28:0/1568810409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8634008c10 0x7f8634009090 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:57.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.175+0000 7f8652400640 1 -- 172.21.15.28:0/1568810409 >> 172.21.15.28:0/1568810409 conn(0x7f864c0c6610 msgr2=0x7f864c0c69f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:57.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.175+0000 7f8652400640 1 -- 172.21.15.28:0/1568810409 shutdown_connections 2024-02-21T23:03:57.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.175+0000 7f8652400640 1 -- 172.21.15.28:0/1568810409 wait complete. 2024-02-21T23:03:57.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.178+0000 7f8652400640 1 Processor -- start 2024-02-21T23:03:57.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.178+0000 7f8652400640 1 -- start start 2024-02-21T23:03:57.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.179+0000 7f8652400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8634008c10 0x7f864c1e9690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:57.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.179+0000 7f8652400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f864c13a450 0x7f864c14d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:57.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.179+0000 7f8652400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f864c14df20 0x7f864c1e3530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:57.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.179+0000 7f8652400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f864c0c7340 con 0x7f864c14df20 2024-02-21T23:03:57.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.179+0000 7f8652400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f864c0c0c20 con 0x7f8634008c10 2024-02-21T23:03:57.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.179+0000 7f8652400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f864c1505a0 con 0x7f864c13a450 2024-02-21T23:03:57.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.180+0000 7f8651a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f864c14df20 0x7f864c1e3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:57.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.180+0000 7f8651a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f864c14df20 0x7f864c1e3530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45498/0 (socket says 172.21.15.28:45498) 2024-02-21T23:03:57.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.180+0000 7f8651a00640 1 -- 172.21.15.28:0/2622693918 learned_addr learned my addr 172.21.15.28:0/2622693918 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:57.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.180+0000 7f8651000640 1 --2- 172.21.15.28:0/2622693918 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8634008c10 0x7f864c1e9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:57.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.180+0000 7f864be00640 1 --2- 172.21.15.28:0/2622693918 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f864c13a450 0x7f864c14d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:57.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.180+0000 7f8651a00640 1 -- 172.21.15.28:0/2622693918 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f864c13a450 msgr2=0x7f864c14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:57.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.180+0000 7f8651a00640 1 --2- 172.21.15.28:0/2622693918 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f864c13a450 0x7f864c14d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:57.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.180+0000 7f8651a00640 1 -- 172.21.15.28:0/2622693918 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8634008c10 msgr2=0x7f864c1e9690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:57.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.180+0000 7f8651a00640 1 --2- 172.21.15.28:0/2622693918 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8634008c10 0x7f864c1e9690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:57.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.180+0000 7f8651a00640 1 -- 172.21.15.28:0/2622693918 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f864c1e9d90 con 0x7f864c14df20 2024-02-21T23:03:57.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.180+0000 7f864be00640 1 --2- 172.21.15.28:0/2622693918 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f864c13a450 0x7f864c14d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:03:57.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.180+0000 7f8651000640 1 --2- 172.21.15.28:0/2622693918 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8634008c10 0x7f864c1e9690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:03:57.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.181+0000 7f8651a00640 1 --2- 172.21.15.28:0/2622693918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f864c14df20 0x7f864c1e3530 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f864c0c30e0 tx=0x7f864400cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:57.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.181+0000 7f8648c00640 1 -- 172.21.15.28:0/2622693918 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8644017070 con 0x7f864c14df20 2024-02-21T23:03:57.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.181+0000 7f8648c00640 1 -- 172.21.15.28:0/2622693918 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8644005d40 con 0x7f864c14df20 2024-02-21T23:03:57.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.181+0000 7f8652400640 1 -- 172.21.15.28:0/2622693918 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f864c1ea070 con 0x7f864c14df20 2024-02-21T23:03:57.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.181+0000 7f8648c00640 1 -- 172.21.15.28:0/2622693918 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8644007310 con 0x7f864c14df20 2024-02-21T23:03:57.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.181+0000 7f8652400640 1 -- 172.21.15.28:0/2622693918 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f864c1e3a70 con 0x7f864c14df20 2024-02-21T23:03:57.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.182+0000 7f8652400640 1 -- 172.21.15.28:0/2622693918 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f864c1ea590 con 0x7f864c14df20 2024-02-21T23:03:57.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.183+0000 7f8648c00640 1 -- 172.21.15.28:0/2622693918 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f864401a070 con 0x7f864c14df20 2024-02-21T23:03:57.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.184+0000 7f8648c00640 1 --2- 172.21.15.28:0/2622693918 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f862c06b4d0 0x7f862c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:57.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.184+0000 7f8648c00640 1 -- 172.21.15.28:0/2622693918 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f86440898c0 con 0x7f864c14df20 2024-02-21T23:03:57.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.185+0000 7f8651000640 1 --2- 172.21.15.28:0/2622693918 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f862c06b4d0 0x7f862c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:57.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.186+0000 7f8651000640 1 --2- 172.21.15.28:0/2622693918 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f862c06b4d0 0x7f862c06d990 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f864c050b90 tx=0x7f8640008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:57.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.188+0000 7f8648c00640 1 -- 172.21.15.28:0/2622693918 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f864c1ea590 con 0x7f864c14df20 2024-02-21T23:03:57.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:57.392+0000 7f8652400640 1 -- 172.21.15.28:0/2622693918 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f864c04f290 con 0x7f864c14df20 2024-02-21T23:03:58.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.285+0000 7f8648c00640 1 -- 172.21.15.28:0/2622693918 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v28) v1 ==== 181+0+0 (secure 0 0 0) 0x7f86440582e0 con 0x7f864c14df20 2024-02-21T23:03:58.275 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' removed 2024-02-21T23:03:58.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.289+0000 7f8652400640 1 -- 172.21.15.28:0/2622693918 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f862c06b4d0 msgr2=0x7f862c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:58.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.289+0000 7f8652400640 1 --2- 172.21.15.28:0/2622693918 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f862c06b4d0 0x7f862c06d990 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f864c050b90 tx=0x7f8640008040 comp rx=0 tx=0).stop 2024-02-21T23:03:58.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.289+0000 7f8652400640 1 -- 172.21.15.28:0/2622693918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f864c14df20 msgr2=0x7f864c1e3530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:58.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.289+0000 7f8652400640 1 --2- 172.21.15.28:0/2622693918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f864c14df20 0x7f864c1e3530 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f864c0c30e0 tx=0x7f864400cb90 comp rx=0 tx=0).stop 2024-02-21T23:03:58.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.289+0000 7f8652400640 1 -- 172.21.15.28:0/2622693918 shutdown_connections 2024-02-21T23:03:58.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.289+0000 7f8652400640 1 --2- 172.21.15.28:0/2622693918 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f862c06b4d0 0x7f862c06d990 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:58.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.289+0000 7f8652400640 1 --2- 172.21.15.28:0/2622693918 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f864c13a450 0x7f864c14d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:58.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.289+0000 7f8652400640 1 --2- 172.21.15.28:0/2622693918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f864c14df20 0x7f864c1e3530 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:58.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.289+0000 7f8652400640 1 --2- 172.21.15.28:0/2622693918 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8634008c10 0x7f864c1e9690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:58.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.289+0000 7f8652400640 1 -- 172.21.15.28:0/2622693918 >> 172.21.15.28:0/2622693918 conn(0x7f864c0c6610 msgr2=0x7f864c0c1750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:58.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.289+0000 7f8652400640 1 -- 172.21.15.28:0/2622693918 shutdown_connections 2024-02-21T23:03:58.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.289+0000 7f8652400640 1 -- 172.21.15.28:0/2622693918 wait complete. 2024-02-21T23:03:58.305 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-02-21T23:03:58.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.526+0000 7fc66d600640 1 Processor -- start 2024-02-21T23:03:58.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.526+0000 7fc66d600640 1 -- start start 2024-02-21T23:03:58.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.528+0000 7fc66d600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc6680ca540 0x7fc668158460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:58.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.528+0000 7fc66d600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fc6680c6290 0x7fc6681522c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:58.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.528+0000 7fc66d600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6680caf40 con 0x7fc66806cfa0 2024-02-21T23:03:58.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.528+0000 7fc66d600640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6680c8ae0 con 0x7fc6680c6290 2024-02-21T23:03:58.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.528+0000 7fc66d600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6680c87b0 con 0x7fc6680ca540 2024-02-21T23:03:58.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.529+0000 7fc667400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc6680ca540 0x7fc668158460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:58.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.529+0000 7fc66cc00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fc6680c6290 0x7fc6681522c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:58.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.529+0000 7fc667400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc6680ca540 0x7fc668158460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3301/0 says I am v2:172.21.15.28:46188/0 (socket says 172.21.15.28:46188) 2024-02-21T23:03:58.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.529+0000 7fc667400640 1 -- 172.21.15.28:0/1083882469 learned_addr learned my addr 172.21.15.28:0/1083882469 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:58.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.530+0000 7fc667400640 1 -- 172.21.15.28:0/1083882469 >> v2:172.21.15.81:3300/0 conn(0x7fc6680c6290 msgr2=0x7fc6681522c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:03:58.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.530+0000 7fc667400640 1 --2- 172.21.15.28:0/1083882469 >> v2:172.21.15.81:3300/0 conn(0x7fc6680c6290 0x7fc6681522c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:58.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.530+0000 7fc667400640 1 -- 172.21.15.28:0/1083882469 >> v1:172.21.15.81:6789/0 conn(0x7fc66806cfa0 legacy=0x7fc6680c8f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:58.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.530+0000 7fc667400640 1 -- 172.21.15.28:0/1083882469 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6680c7f80 con 0x7fc6680ca540 2024-02-21T23:03:58.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.530+0000 7fc66cc00640 1 --2- 172.21.15.28:0/1083882469 >> v2:172.21.15.81:3300/0 conn(0x7fc6680c6290 0x7fc6681522c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-21T23:03:58.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.531+0000 7fc667400640 1 --2- 172.21.15.28:0/1083882469 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc6680ca540 0x7fc668158460 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc650009870 tx=0x7fc65002b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7db53f0815a75b26 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:58.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.532+0000 7fc666a00640 1 -- 172.21.15.28:0/1083882469 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6500317c0 con 0x7fc6680ca540 2024-02-21T23:03:58.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.532+0000 7fc666a00640 1 -- 172.21.15.28:0/1083882469 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc650037070 con 0x7fc6680ca540 2024-02-21T23:03:58.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.532+0000 7fc666a00640 1 -- 172.21.15.28:0/1083882469 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc650031b80 con 0x7fc6680ca540 2024-02-21T23:03:58.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.532+0000 7fc66d600640 1 -- 172.21.15.28:0/1083882469 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc6680ca540 msgr2=0x7fc668158460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:58.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.532+0000 7fc66d600640 1 --2- 172.21.15.28:0/1083882469 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc6680ca540 0x7fc668158460 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc650009870 tx=0x7fc65002b860 comp rx=0 tx=0).stop 2024-02-21T23:03:58.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.533+0000 7fc66d600640 1 -- 172.21.15.28:0/1083882469 shutdown_connections 2024-02-21T23:03:58.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.533+0000 7fc66d600640 1 --2- 172.21.15.28:0/1083882469 >> v2:172.21.15.81:3300/0 conn(0x7fc6680c6290 0x7fc6681522c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:58.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.533+0000 7fc66d600640 1 --2- 172.21.15.28:0/1083882469 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc6680ca540 0x7fc668158460 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:58.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.533+0000 7fc66d600640 1 -- 172.21.15.28:0/1083882469 >> 172.21.15.28:0/1083882469 conn(0x7fc6680725a0 msgr2=0x7fc6680729b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:58.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.533+0000 7fc66d600640 1 -- 172.21.15.28:0/1083882469 shutdown_connections 2024-02-21T23:03:58.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.533+0000 7fc66d600640 1 -- 172.21.15.28:0/1083882469 wait complete. 2024-02-21T23:03:58.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.534+0000 7fc66d600640 1 Processor -- start 2024-02-21T23:03:58.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.535+0000 7fc66d600640 1 -- start start 2024-02-21T23:03:58.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.535+0000 7fc66d600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc66806cfa0 0x7fc6681536d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:58.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.535+0000 7fc66d600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc6680c6290 0x7fc6681edb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:58.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.535+0000 7fc66d600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6680ca540 0x7fc6681ee0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:58.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.535+0000 7fc66d600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6680c6c90 con 0x7fc6680ca540 2024-02-21T23:03:58.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.535+0000 7fc66d600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6680c69b0 con 0x7fc66806cfa0 2024-02-21T23:03:58.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.535+0000 7fc66d600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc66806d380 con 0x7fc6680c6290 2024-02-21T23:03:58.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.536+0000 7fc66cc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6680ca540 0x7fc6681ee0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:58.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.536+0000 7fc66cc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6680ca540 0x7fc6681ee0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45514/0 (socket says 172.21.15.28:45514) 2024-02-21T23:03:58.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.536+0000 7fc66cc00640 1 -- 172.21.15.28:0/3600602481 learned_addr learned my addr 172.21.15.28:0/3600602481 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:58.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.536+0000 7fc667400640 1 --2- 172.21.15.28:0/3600602481 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc6680c6290 0x7fc6681edb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:58.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.536+0000 7fc667e00640 1 --2- 172.21.15.28:0/3600602481 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc66806cfa0 0x7fc6681536d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:58.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.536+0000 7fc66cc00640 1 -- 172.21.15.28:0/3600602481 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc6680c6290 msgr2=0x7fc6681edb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:58.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.536+0000 7fc66cc00640 1 --2- 172.21.15.28:0/3600602481 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc6680c6290 0x7fc6681edb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:58.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.536+0000 7fc66cc00640 1 -- 172.21.15.28:0/3600602481 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc66806cfa0 msgr2=0x7fc6681536d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:58.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.536+0000 7fc66cc00640 1 --2- 172.21.15.28:0/3600602481 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc66806cfa0 0x7fc6681536d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:58.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.536+0000 7fc66cc00640 1 -- 172.21.15.28:0/3600602481 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc668155480 con 0x7fc6680ca540 2024-02-21T23:03:58.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.537+0000 7fc66cc00640 1 --2- 172.21.15.28:0/3600602481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6680ca540 0x7fc6681ee0b0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fc66000e880 tx=0x7fc66000ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:58.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.537+0000 7fc65fe00640 1 -- 172.21.15.28:0/3600602481 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc660017070 con 0x7fc6680ca540 2024-02-21T23:03:58.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.537+0000 7fc65fe00640 1 -- 172.21.15.28:0/3600602481 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc660004020 con 0x7fc6680ca540 2024-02-21T23:03:58.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.537+0000 7fc65fe00640 1 -- 172.21.15.28:0/3600602481 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6600043f0 con 0x7fc6680ca540 2024-02-21T23:03:58.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.537+0000 7fc66d600640 1 -- 172.21.15.28:0/3600602481 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6680ab9c0 con 0x7fc6680ca540 2024-02-21T23:03:58.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.537+0000 7fc66d600640 1 -- 172.21.15.28:0/3600602481 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc668153c10 con 0x7fc6680ca540 2024-02-21T23:03:58.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.539+0000 7fc66d600640 1 -- 172.21.15.28:0/3600602481 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc66804f290 con 0x7fc6680ca540 2024-02-21T23:03:58.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.539+0000 7fc65fe00640 1 -- 172.21.15.28:0/3600602481 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc66000c940 con 0x7fc6680ca540 2024-02-21T23:03:58.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.540+0000 7fc65fe00640 1 --2- 172.21.15.28:0/3600602481 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc64406b4d0 0x7fc64406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:58.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.540+0000 7fc65fe00640 1 -- 172.21.15.28:0/3600602481 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6383+0+0 (secure 0 0 0) 0x7fc66008a6e0 con 0x7fc6680ca540 2024-02-21T23:03:58.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.541+0000 7fc667e00640 1 --2- 172.21.15.28:0/3600602481 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc64406b4d0 0x7fc64406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:58.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.542+0000 7fc667e00640 1 --2- 172.21.15.28:0/3600602481 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc64406b4d0 0x7fc64406d990 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fc6680c9d30 tx=0x7fc658009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:58.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.545+0000 7fc65fe00640 1 -- 172.21.15.28:0/3600602481 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc660059220 con 0x7fc6680ca540 2024-02-21T23:03:58.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:58.745+0000 7fc66d600640 1 -- 172.21.15.28:0/3600602481 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc668155250 con 0x7fc6680ca540 2024-02-21T23:03:59.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:59.289+0000 7fc65fe00640 1 -- 172.21.15.28:0/3600602481 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v29) v1 ==== 169+0+0 (secure 0 0 0) 0x7fc660058b20 con 0x7fc6680ca540 2024-02-21T23:03:59.279 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' removed 2024-02-21T23:03:59.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:59.293+0000 7fc66d600640 1 -- 172.21.15.28:0/3600602481 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc64406b4d0 msgr2=0x7fc64406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:59.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:59.293+0000 7fc66d600640 1 --2- 172.21.15.28:0/3600602481 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc64406b4d0 0x7fc64406d990 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fc6680c9d30 tx=0x7fc658009040 comp rx=0 tx=0).stop 2024-02-21T23:03:59.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:59.293+0000 7fc66d600640 1 -- 172.21.15.28:0/3600602481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6680ca540 msgr2=0x7fc6681ee0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:59.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:59.293+0000 7fc66d600640 1 --2- 172.21.15.28:0/3600602481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6680ca540 0x7fc6681ee0b0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fc66000e880 tx=0x7fc66000ed40 comp rx=0 tx=0).stop 2024-02-21T23:03:59.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:59.295+0000 7fc66d600640 1 -- 172.21.15.28:0/3600602481 shutdown_connections 2024-02-21T23:03:59.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:59.295+0000 7fc66d600640 1 --2- 172.21.15.28:0/3600602481 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc64406b4d0 0x7fc64406d990 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:59.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:59.295+0000 7fc66d600640 1 --2- 172.21.15.28:0/3600602481 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc6680c6290 0x7fc6681edb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:59.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:59.295+0000 7fc66d600640 1 --2- 172.21.15.28:0/3600602481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6680ca540 0x7fc6681ee0b0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:59.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:59.295+0000 7fc66d600640 1 --2- 172.21.15.28:0/3600602481 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc66806cfa0 0x7fc6681536d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:59.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:59.295+0000 7fc66d600640 1 -- 172.21.15.28:0/3600602481 >> 172.21.15.28:0/3600602481 conn(0x7fc6680725a0 msgr2=0x7fc668157090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:59.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:59.298+0000 7fc66d600640 1 -- 172.21.15.28:0/3600602481 shutdown_connections 2024-02-21T23:03:59.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:59.298+0000 7fc66d600640 1 -- 172.21.15.28:0/3600602481 wait complete. 2024-02-21T23:03:59.315 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-02-21T23:03:59.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:59.525+0000 7f0ecca00640 1 Processor -- start 2024-02-21T23:03:59.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:59.525+0000 7f0ecca00640 1 -- start start 2024-02-21T23:03:59.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:59.526+0000 7f0ecca00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0ec81526f0 0x7f0ec80c1340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:59.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:59.526+0000 7f0ecca00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ec8152ad0 con 0x7f0ec80c46f0 2024-02-21T23:03:59.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:59.526+0000 7f0ecca00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ec8155240 con 0x7f0ec80c4dc0 2024-02-21T23:03:59.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:59.526+0000 7f0ecca00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ec8155520 con 0x7f0ec81526f0 2024-02-21T23:03:59.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:59.527+0000 7f0ec5400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f0ec80c46f0 0x7f0ec8158820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:40450/0 (socket says 172.21.15.28:40450) 2024-02-21T23:03:59.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:59.527+0000 7f0ec5400640 1 -- 172.21.15.28:0/77396579 learned_addr learned my addr 172.21.15.28:0/77396579 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:03:59.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:59.527+0000 7f0ec5e00640 1 --2- 172.21.15.28:0/77396579 >> v2:172.21.15.81:3300/0 conn(0x7f0ec81526f0 0x7f0ec80c1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:59.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:59.527+0000 7f0ec4a00640 1 -- 172.21.15.28:0/77396579 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4127379383 0 0) 0x7f0ec8152ad0 con 0x7f0ec80c46f0 2024-02-21T23:03:59.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:59.527+0000 7f0ec4a00640 1 -- 172.21.15.28:0/77396579 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0eac003520 con 0x7f0ec80c46f0 2024-02-21T23:03:59.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:59.528+0000 7f0ec5e00640 1 -- 172.21.15.28:0/77396579 >> v1:172.21.15.81:6789/0 conn(0x7f0ec80c4dc0 legacy=0x7f0ec8158f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:59.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:59.528+0000 7f0ec5e00640 1 -- 172.21.15.28:0/77396579 >> v1:172.21.15.28:6789/0 conn(0x7f0ec80c46f0 legacy=0x7f0ec8158820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:59.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:59.528+0000 7f0ec5e00640 1 -- 172.21.15.28:0/77396579 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ec8155800 con 0x7f0ec81526f0 2024-02-21T23:03:59.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:59.529+0000 7f0ec5e00640 1 --2- 172.21.15.28:0/77396579 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ec81526f0 0x7f0ec80c1340 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f0eb8009dc0 tx=0x7f0eb802be90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20c84df479cb0974 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:59.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:59.530+0000 7f0ec4a00640 1 -- 172.21.15.28:0/77396579 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eb8039070 con 0x7f0ec81526f0 2024-02-21T23:03:59.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:59.530+0000 7f0ec4a00640 1 -- 172.21.15.28:0/77396579 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0eb802cc40 con 0x7f0ec81526f0 2024-02-21T23:03:59.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:59.530+0000 7f0ec4a00640 1 -- 172.21.15.28:0/77396579 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eb8034440 con 0x7f0ec81526f0 2024-02-21T23:03:59.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:59.530+0000 7f0ecca00640 1 -- 172.21.15.28:0/77396579 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ec81526f0 msgr2=0x7f0ec80c1340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:59.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:59.530+0000 7f0ecca00640 1 --2- 172.21.15.28:0/77396579 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ec81526f0 0x7f0ec80c1340 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f0eb8009dc0 tx=0x7f0eb802be90 comp rx=0 tx=0).stop 2024-02-21T23:03:59.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:59.530+0000 7f0ecca00640 1 -- 172.21.15.28:0/77396579 shutdown_connections 2024-02-21T23:03:59.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:59.530+0000 7f0ecca00640 1 --2- 172.21.15.28:0/77396579 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ec81526f0 0x7f0ec80c1340 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:59.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:59.530+0000 7f0ecca00640 1 -- 172.21.15.28:0/77396579 >> 172.21.15.28:0/77396579 conn(0x7f0ec806b830 msgr2=0x7f0ec80c67f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:03:59.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:59.531+0000 7f0ecca00640 1 -- 172.21.15.28:0/77396579 shutdown_connections 2024-02-21T23:03:59.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:59.531+0000 7f0ecca00640 1 -- 172.21.15.28:0/77396579 wait complete. 2024-02-21T23:03:59.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:59.532+0000 7f0ecca00640 1 Processor -- start 2024-02-21T23:03:59.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:59.532+0000 7f0ecca00640 1 -- start start 2024-02-21T23:03:59.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:59.533+0000 7f0ecca00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ec80c46f0 0x7f0ec81ede60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:59.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:59.533+0000 7f0ecca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ec80c4dc0 0x7f0ec814b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:59.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:59.533+0000 7f0ecca00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0ec81526f0 0x7f0ec814d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:59.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:59.533+0000 7f0ecca00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec80c3c30 con 0x7f0ec80c4dc0 2024-02-21T23:03:59.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:59.533+0000 7f0ecca00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec8155dc0 con 0x7f0ec80c46f0 2024-02-21T23:03:59.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:59.533+0000 7f0ecca00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec80c1880 con 0x7f0ec81526f0 2024-02-21T23:03:59.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:59.534+0000 7f0ec5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ec80c4dc0 0x7f0ec814b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:59.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:59.534+0000 7f0ec5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ec80c4dc0 0x7f0ec814b380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45518/0 (socket says 172.21.15.28:45518) 2024-02-21T23:03:59.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:59.534+0000 7f0ec5400640 1 -- 172.21.15.28:0/3126811615 learned_addr learned my addr 172.21.15.28:0/3126811615 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:03:59.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:59.534+0000 7f0ec5e00640 1 --2- 172.21.15.28:0/3126811615 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ec80c46f0 0x7f0ec81ede60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:59.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:59.534+0000 7f0ec5400640 1 -- 172.21.15.28:0/3126811615 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0ec81526f0 msgr2=0x7f0ec814d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:59.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:59.534+0000 7f0ec5400640 1 --2- 172.21.15.28:0/3126811615 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0ec81526f0 0x7f0ec814d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:59.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:59.534+0000 7f0ec5400640 1 -- 172.21.15.28:0/3126811615 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ec80c46f0 msgr2=0x7f0ec81ede60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:03:59.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:59.534+0000 7f0ec5400640 1 --2- 172.21.15.28:0/3126811615 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ec80c46f0 0x7f0ec81ede60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:03:59.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:59.534+0000 7f0ec5400640 1 -- 172.21.15.28:0/3126811615 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ec81493d0 con 0x7f0ec80c4dc0 2024-02-21T23:03:59.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:59.534+0000 7f0ec5400640 1 --2- 172.21.15.28:0/3126811615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ec80c4dc0 0x7f0ec814b380 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f0eb000c700 tx=0x7f0eb000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:59.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:59.535+0000 7f0ebe000640 1 -- 172.21.15.28:0/3126811615 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eb0017070 con 0x7f0ec80c4dc0 2024-02-21T23:03:59.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:59.535+0000 7f0ebe000640 1 -- 172.21.15.28:0/3126811615 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0eb0007270 con 0x7f0ec80c4dc0 2024-02-21T23:03:59.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:59.535+0000 7f0ecca00640 1 -- 172.21.15.28:0/3126811615 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ec8148ef0 con 0x7f0ec80c4dc0 2024-02-21T23:03:59.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:59.535+0000 7f0ebe000640 1 -- 172.21.15.28:0/3126811615 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eb00057a0 con 0x7f0ec80c4dc0 2024-02-21T23:03:59.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:59.535+0000 7f0ecca00640 1 -- 172.21.15.28:0/3126811615 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ec814b8c0 con 0x7f0ec80c4dc0 2024-02-21T23:03:59.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:59.536+0000 7f0ecca00640 1 -- 172.21.15.28:0/3126811615 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ec814c990 con 0x7f0ec80c4dc0 2024-02-21T23:03:59.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:59.537+0000 7f0ebe000640 1 -- 172.21.15.28:0/3126811615 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0eb001a070 con 0x7f0ec80c4dc0 2024-02-21T23:03:59.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:59.538+0000 7f0ebe000640 1 --2- 172.21.15.28:0/3126811615 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0ea406b4d0 0x7f0ea406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:03:59.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:59.538+0000 7f0ebe000640 1 -- 172.21.15.28:0/3126811615 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f0eb0089760 con 0x7f0ec80c4dc0 2024-02-21T23:03:59.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:59.539+0000 7f0ec5e00640 1 --2- 172.21.15.28:0/3126811615 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0ea406b4d0 0x7f0ea406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:03:59.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:59.540+0000 7f0ec5e00640 1 --2- 172.21.15.28:0/3126811615 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0ea406b4d0 0x7f0ea406d990 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f0eb8036000 tx=0x7f0eb8030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:03:59.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:59.543+0000 7f0ebe000640 1 -- 172.21.15.28:0/3126811615 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f0eb00584c0 con 0x7f0ec80c4dc0 2024-02-21T23:03:59.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:03:59.747+0000 7f0ecca00640 1 -- 172.21.15.28:0/3126811615 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f0ec8067f30 con 0x7f0ec80c4dc0 2024-02-21T23:04:00.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.307+0000 7f0ebe000640 1 -- 172.21.15.28:0/3126811615 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v30) v1 ==== 178+0+0 (secure 0 0 0) 0x7f0eb0007db0 con 0x7f0ec80c4dc0 2024-02-21T23:04:00.296 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data_ec' removed 2024-02-21T23:04:00.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.311+0000 7f0ecca00640 1 -- 172.21.15.28:0/3126811615 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0ea406b4d0 msgr2=0x7f0ea406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:00.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.311+0000 7f0ecca00640 1 --2- 172.21.15.28:0/3126811615 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0ea406b4d0 0x7f0ea406d990 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f0eb8036000 tx=0x7f0eb8030000 comp rx=0 tx=0).stop 2024-02-21T23:04:00.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.311+0000 7f0ecca00640 1 -- 172.21.15.28:0/3126811615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ec80c4dc0 msgr2=0x7f0ec814b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:00.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.311+0000 7f0ecca00640 1 --2- 172.21.15.28:0/3126811615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ec80c4dc0 0x7f0ec814b380 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f0eb000c700 tx=0x7f0eb000cbc0 comp rx=0 tx=0).stop 2024-02-21T23:04:00.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.312+0000 7f0ecca00640 1 -- 172.21.15.28:0/3126811615 shutdown_connections 2024-02-21T23:04:00.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.312+0000 7f0ecca00640 1 --2- 172.21.15.28:0/3126811615 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0ea406b4d0 0x7f0ea406d990 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:00.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.312+0000 7f0ecca00640 1 --2- 172.21.15.28:0/3126811615 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0ec81526f0 0x7f0ec814d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:00.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.312+0000 7f0ecca00640 1 --2- 172.21.15.28:0/3126811615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ec80c4dc0 0x7f0ec814b380 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:00.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.312+0000 7f0ecca00640 1 --2- 172.21.15.28:0/3126811615 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ec80c46f0 0x7f0ec81ede60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:00.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.312+0000 7f0ecca00640 1 -- 172.21.15.28:0/3126811615 >> 172.21.15.28:0/3126811615 conn(0x7f0ec806b830 msgr2=0x7f0ec8152fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:00.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.313+0000 7f0ecca00640 1 -- 172.21.15.28:0/3126811615 shutdown_connections 2024-02-21T23:04:00.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.314+0000 7f0ecca00640 1 -- 172.21.15.28:0/3126811615 wait complete. 2024-02-21T23:04:00.327 INFO:tasks.ceph.mds.a:Restarting daemon 2024-02-21T23:04:00.327 INFO:tasks.ceph.mds.a:Stopping old one... 2024-02-21T23:04:00.328 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-02-21T23:04:00.328 INFO:teuthology.orchestra.run:waiting for 300 2024-02-21T23:04:00.328 INFO:tasks.ceph.mds.c:Restarting daemon 2024-02-21T23:04:00.328 INFO:tasks.ceph.mds.c:Stopping old one... 2024-02-21T23:04:00.328 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-02-21T23:04:00.328 INFO:teuthology.orchestra.run:waiting for 300 2024-02-21T23:04:00.328 INFO:tasks.ceph.mds.b:Restarting daemon 2024-02-21T23:04:00.329 INFO:tasks.ceph.mds.b:Stopping old one... 2024-02-21T23:04:00.329 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-02-21T23:04:00.329 INFO:teuthology.orchestra.run:waiting for 300 2024-02-21T23:04:00.329 INFO:tasks.ceph.mds.d:Restarting daemon 2024-02-21T23:04:00.329 INFO:tasks.ceph.mds.d:Stopping old one... 2024-02-21T23:04:00.329 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-02-21T23:04:00.329 INFO:teuthology.orchestra.run:waiting for 300 2024-02-21T23:04:00.348 INFO:tasks.ceph.mds.c:Stopped 2024-02-21T23:04:00.348 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-02-21T23:04:00.349 INFO:tasks.ceph.mds.a:Stopped 2024-02-21T23:04:00.349 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-02-21T23:04:00.350 INFO:tasks.ceph.mds.b:Stopped 2024-02-21T23:04:00.350 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-02-21T23:04:00.351 INFO:tasks.ceph.mds.d:Stopped 2024-02-21T23:04:00.351 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-02-21T23:04:00.396 INFO:tasks.ceph.mds.c:Started 2024-02-21T23:04:00.396 DEBUG:teuthology.parallel:result is None 2024-02-21T23:04:00.398 INFO:tasks.ceph.mds.b:Started 2024-02-21T23:04:00.398 DEBUG:teuthology.parallel:result is None 2024-02-21T23:04:00.401 INFO:tasks.ceph.mds.a:Started 2024-02-21T23:04:00.401 DEBUG:teuthology.parallel:result is None 2024-02-21T23:04:00.403 INFO:tasks.ceph.mds.d:Started 2024-02-21T23:04:00.404 DEBUG:teuthology.parallel:result is None 2024-02-21T23:04:00.404 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-02-21T23:04:00.553 INFO:tasks.ceph.mds.d.smithi081.stdout:starting mds.d at 2024-02-21T23:04:00.554 INFO:tasks.ceph.mds.b.smithi081.stdout:starting mds.b at 2024-02-21T23:04:00.554 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-02-21T23:04:00.555 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-02-21T23:04:00.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.601+0000 7fa92de00640 1 Processor -- start 2024-02-21T23:04:00.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.601+0000 7fa92de00640 1 -- start start 2024-02-21T23:04:00.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.602+0000 7fa92de00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fa928152740 0x7fa9280c6a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:00.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.602+0000 7fa92de00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa9280c5260 0x7fa9280c56f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:00.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.602+0000 7fa92de00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9280c6640 con 0x7fa9280bf880 2024-02-21T23:04:00.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.602+0000 7fa92de00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa928155fc0 con 0x7fa9280c5260 2024-02-21T23:04:00.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.602+0000 7fa92de00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa928155200 con 0x7fa928152740 2024-02-21T23:04:00.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.602+0000 7fa91fe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa9280bf880 0x7fa9280c72f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:40472/0 (socket says 172.21.15.28:40472) 2024-02-21T23:04:00.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.602+0000 7fa91fe00640 1 -- 172.21.15.28:0/1517361684 learned_addr learned my addr 172.21.15.28:0/1517361684 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:04:00.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.602+0000 7fa92d400640 1 --2- 172.21.15.28:0/1517361684 >> v2:172.21.15.28:3300/0 conn(0x7fa9280c5260 0x7fa9280c56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:00.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.603+0000 7fa92ca00640 1 --2- 172.21.15.28:0/1517361684 >> v2:172.21.15.81:3300/0 conn(0x7fa928152740 0x7fa9280c6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:00.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.603+0000 7fa927e00640 1 -- 172.21.15.28:0/1517361684 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 663427928 0 0) 0x7fa9280c6640 con 0x7fa9280bf880 2024-02-21T23:04:00.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.603+0000 7fa927e00640 1 -- 172.21.15.28:0/1517361684 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa90c003520 con 0x7fa9280bf880 2024-02-21T23:04:00.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.603+0000 7fa92d400640 1 -- 172.21.15.28:0/1517361684 >> v2:172.21.15.81:3300/0 conn(0x7fa928152740 msgr2=0x7fa9280c6a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:00.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.603+0000 7fa92d400640 1 --2- 172.21.15.28:0/1517361684 >> v2:172.21.15.81:3300/0 conn(0x7fa928152740 0x7fa9280c6a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:00.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.603+0000 7fa92d400640 1 -- 172.21.15.28:0/1517361684 >> v1:172.21.15.28:6789/0 conn(0x7fa9280bf880 legacy=0x7fa9280c72f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:00.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.603+0000 7fa92d400640 1 -- 172.21.15.28:0/1517361684 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9281554e0 con 0x7fa9280c5260 2024-02-21T23:04:00.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.603+0000 7fa92ca00640 1 --2- 172.21.15.28:0/1517361684 >> v2:172.21.15.81:3300/0 conn(0x7fa928152740 0x7fa9280c6a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-21T23:04:00.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.604+0000 7fa92d400640 1 --2- 172.21.15.28:0/1517361684 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9280c5260 0x7fa9280c56f0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fa9280731c0 tx=0x7fa92002be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e07f6f741b077b0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:00.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.604+0000 7fa927e00640 1 -- 172.21.15.28:0/1517361684 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa920039070 con 0x7fa9280c5260 2024-02-21T23:04:00.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.604+0000 7fa927e00640 1 -- 172.21.15.28:0/1517361684 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa92002cbe0 con 0x7fa9280c5260 2024-02-21T23:04:00.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.604+0000 7fa927e00640 1 -- 172.21.15.28:0/1517361684 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa920034440 con 0x7fa9280c5260 2024-02-21T23:04:00.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.605+0000 7fa92de00640 1 -- 172.21.15.28:0/1517361684 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9280c5260 msgr2=0x7fa9280c56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:00.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.605+0000 7fa92de00640 1 --2- 172.21.15.28:0/1517361684 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9280c5260 0x7fa9280c56f0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fa9280731c0 tx=0x7fa92002be30 comp rx=0 tx=0).stop 2024-02-21T23:04:00.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.605+0000 7fa92de00640 1 -- 172.21.15.28:0/1517361684 shutdown_connections 2024-02-21T23:04:00.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.605+0000 7fa92de00640 1 --2- 172.21.15.28:0/1517361684 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9280c5260 0x7fa9280c56f0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:00.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.605+0000 7fa92de00640 1 --2- 172.21.15.28:0/1517361684 >> v2:172.21.15.81:3300/0 conn(0x7fa928152740 0x7fa9280c6a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:00.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.605+0000 7fa92de00640 1 -- 172.21.15.28:0/1517361684 >> 172.21.15.28:0/1517361684 conn(0x7fa928067080 msgr2=0x7fa9280725a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:00.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.605+0000 7fa92de00640 1 -- 172.21.15.28:0/1517361684 shutdown_connections 2024-02-21T23:04:00.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.605+0000 7fa92de00640 1 -- 172.21.15.28:0/1517361684 wait complete. 2024-02-21T23:04:00.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.607+0000 7fa92de00640 1 Processor -- start 2024-02-21T23:04:00.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.607+0000 7fa92de00640 1 -- start start 2024-02-21T23:04:00.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.608+0000 7fa92de00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa9280bf880 0x7fa9281442d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:00.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.608+0000 7fa92de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9280c5260 0x7fa9281548b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:00.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.608+0000 7fa92de00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa928152740 0x7fa9281432e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:00.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.608+0000 7fa92de00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9280c7a00 con 0x7fa9280c5260 2024-02-21T23:04:00.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.608+0000 7fa92de00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa928155aa0 con 0x7fa9280bf880 2024-02-21T23:04:00.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.608+0000 7fa92de00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9280c7b80 con 0x7fa928152740 2024-02-21T23:04:00.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.608+0000 7fa91fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9280c5260 0x7fa9281548b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:00.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.608+0000 7fa91fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9280c5260 0x7fa9281548b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45556/0 (socket says 172.21.15.28:45556) 2024-02-21T23:04:00.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.608+0000 7fa91fe00640 1 -- 172.21.15.28:0/3894589707 learned_addr learned my addr 172.21.15.28:0/3894589707 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:00.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.609+0000 7fa92ca00640 1 --2- 172.21.15.28:0/3894589707 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa9280bf880 0x7fa9281442d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:00.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.609+0000 7fa92d400640 1 --2- 172.21.15.28:0/3894589707 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa928152740 0x7fa9281432e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:00.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.609+0000 7fa91fe00640 1 -- 172.21.15.28:0/3894589707 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa928152740 msgr2=0x7fa9281432e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:00.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.609+0000 7fa91fe00640 1 --2- 172.21.15.28:0/3894589707 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa928152740 0x7fa9281432e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:00.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.609+0000 7fa91fe00640 1 -- 172.21.15.28:0/3894589707 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa9280bf880 msgr2=0x7fa9281442d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:00.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.609+0000 7fa91fe00640 1 --2- 172.21.15.28:0/3894589707 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa9280bf880 0x7fa9281442d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:00.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.609+0000 7fa91fe00640 1 -- 172.21.15.28:0/3894589707 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa928143f80 con 0x7fa9280c5260 2024-02-21T23:04:00.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.609+0000 7fa91fe00640 1 --2- 172.21.15.28:0/3894589707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9280c5260 0x7fa9281548b0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fa91000c460 tx=0x7fa91000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:00.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.609+0000 7fa925600640 1 -- 172.21.15.28:0/3894589707 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa910017070 con 0x7fa9280c5260 2024-02-21T23:04:00.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.609+0000 7fa925600640 1 -- 172.21.15.28:0/3894589707 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa910005ce0 con 0x7fa9280c5260 2024-02-21T23:04:00.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.609+0000 7fa92de00640 1 -- 172.21.15.28:0/3894589707 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9281eeb60 con 0x7fa9280c5260 2024-02-21T23:04:00.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.610+0000 7fa925600640 1 -- 172.21.15.28:0/3894589707 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa910007290 con 0x7fa9280c5260 2024-02-21T23:04:00.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.610+0000 7fa92de00640 1 -- 172.21.15.28:0/3894589707 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9280cabc0 con 0x7fa9280c5260 2024-02-21T23:04:00.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.611+0000 7fa92de00640 1 -- 172.21.15.28:0/3894589707 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa92804f290 con 0x7fa9280c5260 2024-02-21T23:04:00.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.613+0000 7fa925600640 1 -- 172.21.15.28:0/3894589707 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa91001a070 con 0x7fa9280c5260 2024-02-21T23:04:00.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.613+0000 7fa925600640 1 --2- 172.21.15.28:0/3894589707 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa90406b520 0x7fa90406d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:00.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.614+0000 7fa925600640 1 -- 172.21.15.28:0/3894589707 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5335+0+0 (secure 0 0 0) 0x7fa910089800 con 0x7fa9280c5260 2024-02-21T23:04:00.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.614+0000 7fa92ca00640 1 --2- 172.21.15.28:0/3894589707 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa90406b520 0x7fa90406d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:00.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.615+0000 7fa92ca00640 1 --2- 172.21.15.28:0/3894589707 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa90406b520 0x7fa90406d9e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa9281551d0 tx=0x7fa918009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:00.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.617+0000 7fa925600640 1 -- 172.21.15.28:0/3894589707 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fa910021020 con 0x7fa9280c5260 2024-02-21T23:04:00.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:00.829+0000 7fa92de00640 1 -- 172.21.15.28:0/3894589707 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fa9280c6830 con 0x7fa9280c5260 2024-02-21T23:04:01.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.345+0000 7fa925600640 1 -- 172.21.15.28:0/3894589707 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v32) v1 ==== 97+0+0 (secure 0 0 0) 0x7fa910058210 con 0x7fa9280c5260 2024-02-21T23:04:01.334 INFO:teuthology.orchestra.run.smithi028.stderr: removed all blocklist entries 2024-02-21T23:04:01.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.349+0000 7fa92de00640 1 -- 172.21.15.28:0/3894589707 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa90406b520 msgr2=0x7fa90406d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:01.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.349+0000 7fa92de00640 1 --2- 172.21.15.28:0/3894589707 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa90406b520 0x7fa90406d9e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa9281551d0 tx=0x7fa918009190 comp rx=0 tx=0).stop 2024-02-21T23:04:01.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.349+0000 7fa92de00640 1 -- 172.21.15.28:0/3894589707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9280c5260 msgr2=0x7fa9281548b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:01.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.349+0000 7fa92de00640 1 --2- 172.21.15.28:0/3894589707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9280c5260 0x7fa9281548b0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fa91000c460 tx=0x7fa91000c920 comp rx=0 tx=0).stop 2024-02-21T23:04:01.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.349+0000 7fa92de00640 1 -- 172.21.15.28:0/3894589707 shutdown_connections 2024-02-21T23:04:01.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.349+0000 7fa92de00640 1 --2- 172.21.15.28:0/3894589707 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa90406b520 0x7fa90406d9e0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:01.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.349+0000 7fa92de00640 1 --2- 172.21.15.28:0/3894589707 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa928152740 0x7fa9281432e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:01.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.349+0000 7fa92de00640 1 --2- 172.21.15.28:0/3894589707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9280c5260 0x7fa9281548b0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:01.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.349+0000 7fa92de00640 1 --2- 172.21.15.28:0/3894589707 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa9280bf880 0x7fa9281442d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:01.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.349+0000 7fa92de00640 1 -- 172.21.15.28:0/3894589707 >> 172.21.15.28:0/3894589707 conn(0x7fa928067080 msgr2=0x7fa9280c5640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:01.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.350+0000 7fa92de00640 1 -- 172.21.15.28:0/3894589707 shutdown_connections 2024-02-21T23:04:01.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.350+0000 7fa92de00640 1 -- 172.21.15.28:0/3894589707 wait complete. 2024-02-21T23:04:01.363 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-02-21T23:04:01.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.589+0000 7fd806200640 1 Processor -- start 2024-02-21T23:04:01.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.589+0000 7fd806200640 1 -- start start 2024-02-21T23:04:01.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.590+0000 7fd806200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd800152570 0x7fd8000707d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:01.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.590+0000 7fd806200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd8000c6450 0x7fd8001558b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:01.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.590+0000 7fd806200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8000c7670 con 0x7fd800070d10 2024-02-21T23:04:01.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.590+0000 7fd806200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd800155df0 con 0x7fd8000c6450 2024-02-21T23:04:01.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.590+0000 7fd806200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8000c42b0 con 0x7fd800152570 2024-02-21T23:04:01.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.591+0000 7fd805800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd8000c6450 0x7fd8001558b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:01.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.591+0000 7fd805800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd8000c6450 0x7fd8001558b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45558/0 (socket says 172.21.15.28:45558) 2024-02-21T23:04:01.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.591+0000 7fd7ffe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd800070d10 0x7fd8001551a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:40488/0 (socket says 172.21.15.28:40488) 2024-02-21T23:04:01.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.591+0000 7fd805800640 1 -- 172.21.15.28:0/111319725 learned_addr learned my addr 172.21.15.28:0/111319725 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:01.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.591+0000 7fd804e00640 1 --2- 172.21.15.28:0/111319725 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd800152570 0x7fd8000707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:01.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.591+0000 7fd805800640 1 -- 172.21.15.28:0/111319725 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd800152570 msgr2=0x7fd8000707d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:01.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.591+0000 7fd805800640 1 --2- 172.21.15.28:0/111319725 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd800152570 0x7fd8000707d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:01.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.591+0000 7fd805800640 1 -- 172.21.15.28:0/111319725 >> v1:172.21.15.28:6789/0 conn(0x7fd800070d10 legacy=0x7fd8001551a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:01.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.591+0000 7fd805800640 1 -- 172.21.15.28:0/111319725 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8000c7020 con 0x7fd8000c6450 2024-02-21T23:04:01.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.591+0000 7fd7ff400640 1 -- 172.21.15.28:0/111319725 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3128796796 0 0) 0x7fd7ec001110 con 0x7fd800070d10 2024-02-21T23:04:01.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.592+0000 7fd805800640 1 --2- 172.21.15.28:0/111319725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8000c6450 0x7fd8001558b0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fd7f8009e50 tx=0x7fd7f802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5100d516574975b1 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:01.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.593+0000 7fd7ff400640 1 -- 172.21.15.28:0/111319725 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7f802ca70 con 0x7fd8000c6450 2024-02-21T23:04:01.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.593+0000 7fd7ff400640 1 -- 172.21.15.28:0/111319725 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7f802cc10 con 0x7fd8000c6450 2024-02-21T23:04:01.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.593+0000 7fd7ff400640 1 -- 172.21.15.28:0/111319725 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7f80343f0 con 0x7fd8000c6450 2024-02-21T23:04:01.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.593+0000 7fd806200640 1 -- 172.21.15.28:0/111319725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8000c6450 msgr2=0x7fd8001558b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:01.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.593+0000 7fd806200640 1 --2- 172.21.15.28:0/111319725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8000c6450 0x7fd8001558b0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fd7f8009e50 tx=0x7fd7f802bed0 comp rx=0 tx=0).stop 2024-02-21T23:04:01.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.593+0000 7fd806200640 1 -- 172.21.15.28:0/111319725 shutdown_connections 2024-02-21T23:04:01.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.593+0000 7fd806200640 1 --2- 172.21.15.28:0/111319725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8000c6450 0x7fd8001558b0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:01.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.593+0000 7fd806200640 1 --2- 172.21.15.28:0/111319725 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd800152570 0x7fd8000707d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:01.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.593+0000 7fd806200640 1 -- 172.21.15.28:0/111319725 >> 172.21.15.28:0/111319725 conn(0x7fd8000aa280 msgr2=0x7fd8000aa660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:01.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.593+0000 7fd806200640 1 -- 172.21.15.28:0/111319725 shutdown_connections 2024-02-21T23:04:01.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.594+0000 7fd806200640 1 -- 172.21.15.28:0/111319725 wait complete. 2024-02-21T23:04:01.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.595+0000 7fd806200640 1 Processor -- start 2024-02-21T23:04:01.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.595+0000 7fd806200640 1 -- start start 2024-02-21T23:04:01.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.596+0000 7fd806200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd800070d10 0x7fd80006d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:01.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.596+0000 7fd806200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8000c6450 0x7fd8001542c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:01.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.596+0000 7fd806200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd800152570 0x7fd800154800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:01.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.596+0000 7fd806200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd800071980 con 0x7fd8000c6450 2024-02-21T23:04:01.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.596+0000 7fd806200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8000c3f30 con 0x7fd800152570 2024-02-21T23:04:01.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.596+0000 7fd806200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd800071b00 con 0x7fd800070d10 2024-02-21T23:04:01.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.596+0000 7fd7ffe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8000c6450 0x7fd8001542c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:01.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.596+0000 7fd7ffe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8000c6450 0x7fd8001542c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45570/0 (socket says 172.21.15.28:45570) 2024-02-21T23:04:01.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.596+0000 7fd7ffe00640 1 -- 172.21.15.28:0/2499836004 learned_addr learned my addr 172.21.15.28:0/2499836004 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:01.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.597+0000 7fd804e00640 1 --2- 172.21.15.28:0/2499836004 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd800070d10 0x7fd80006d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:01.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.597+0000 7fd805800640 1 --2- 172.21.15.28:0/2499836004 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd800152570 0x7fd800154800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:01.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.597+0000 7fd7ffe00640 1 -- 172.21.15.28:0/2499836004 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd800070d10 msgr2=0x7fd80006d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:01.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.597+0000 7fd7ffe00640 1 --2- 172.21.15.28:0/2499836004 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd800070d10 0x7fd80006d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:01.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.597+0000 7fd7ffe00640 1 -- 172.21.15.28:0/2499836004 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd800152570 msgr2=0x7fd800154800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:01.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.597+0000 7fd7ffe00640 1 --2- 172.21.15.28:0/2499836004 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd800152570 0x7fd800154800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:01.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.597+0000 7fd7ffe00640 1 -- 172.21.15.28:0/2499836004 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd80014dae0 con 0x7fd8000c6450 2024-02-21T23:04:01.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.597+0000 7fd805800640 1 --2- 172.21.15.28:0/2499836004 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd800152570 0x7fd800154800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:04:01.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.597+0000 7fd7ffe00640 1 --2- 172.21.15.28:0/2499836004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8000c6450 0x7fd8001542c0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec00c430 tx=0x7fd7ec00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:01.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.598+0000 7fd7fcc00640 1 -- 172.21.15.28:0/2499836004 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7ec0074e0 con 0x7fd8000c6450 2024-02-21T23:04:01.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.598+0000 7fd7fcc00640 1 -- 172.21.15.28:0/2499836004 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7ec0047e0 con 0x7fd8000c6450 2024-02-21T23:04:01.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.598+0000 7fd7fcc00640 1 -- 172.21.15.28:0/2499836004 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7ec004bd0 con 0x7fd8000c6450 2024-02-21T23:04:01.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.598+0000 7fd806200640 1 -- 172.21.15.28:0/2499836004 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8001e8280 con 0x7fd8000c6450 2024-02-21T23:04:01.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.598+0000 7fd806200640 1 -- 172.21.15.28:0/2499836004 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8001575f0 con 0x7fd8000c6450 2024-02-21T23:04:01.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.599+0000 7fd806200640 1 -- 172.21.15.28:0/2499836004 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8000c7850 con 0x7fd8000c6450 2024-02-21T23:04:01.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.600+0000 7fd7fcc00640 1 -- 172.21.15.28:0/2499836004 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd7ec011070 con 0x7fd8000c6450 2024-02-21T23:04:01.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.601+0000 7fd7fcc00640 1 --2- 172.21.15.28:0/2499836004 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd7e006b4d0 0x7fd7e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:01.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.601+0000 7fd7fcc00640 1 -- 172.21.15.28:0/2499836004 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fd7ec089cc0 con 0x7fd8000c6450 2024-02-21T23:04:01.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.601+0000 7fd804e00640 1 --2- 172.21.15.28:0/2499836004 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd7e006b4d0 0x7fd7e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:01.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.603+0000 7fd804e00640 1 --2- 172.21.15.28:0/2499836004 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd7e006b4d0 0x7fd7e006d990 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fd7f0006fd0 tx=0x7fd7f0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:01.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.605+0000 7fd7fcc00640 1 -- 172.21.15.28:0/2499836004 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd8000c7850 con 0x7fd8000c6450 2024-02-21T23:04:01.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.876+0000 7fd806200640 1 -- 172.21.15.28:0/2499836004 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fd80004f290 con 0x7fd8000c6450 2024-02-21T23:04:01.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.878+0000 7fd7fcc00640 1 -- 172.21.15.28:0/2499836004 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v4) v1 ==== 80+0+6065 (secure 0 0 0) 0x7fd8000c7670 con 0x7fd8000c6450 2024-02-21T23:04:01.866 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:04:01.866 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-21T23:04:01.866 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_dump": [ 2024-02-21T23:04:01.866 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:04:01.867 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.a", 2024-02-21T23:04:01.867 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDSgNZl4FY1KRAAJSK5Ibb0ibKiKXYpTLgtmQ==", 2024-02-21T23:04:01.867 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:04:01.867 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:04:01.867 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-02-21T23:04:01.867 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-02-21T23:04:01.867 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:04:01.867 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:04:01.867 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:04:01.867 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:04:01.867 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.b", 2024-02-21T23:04:01.867 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDTgNZlwRcAERAAgsYmmkuobKelBEJVCRpXUg==", 2024-02-21T23:04:01.867 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:04:01.868 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:04:01.868 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-02-21T23:04:01.868 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-02-21T23:04:01.868 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:04:01.868 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:04:01.868 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:04:01.868 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:04:01.868 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.c", 2024-02-21T23:04:01.868 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDSgNZloFpQOxAAK2WBArXRVNTgqH9WWDEPDA==", 2024-02-21T23:04:01.868 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:04:01.868 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:04:01.869 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-02-21T23:04:01.869 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-02-21T23:04:01.869 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:04:01.869 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:04:01.869 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:04:01.869 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:04:01.869 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.d", 2024-02-21T23:04:01.869 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDTgNZlblOmIRAAxe2HfSKGSkkzJDN4aVdzgQ==", 2024-02-21T23:04:01.869 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:04:01.869 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:04:01.869 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-02-21T23:04:01.869 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-02-21T23:04:01.869 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:04:01.869 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:04:01.870 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:04:01.870 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:04:01.870 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.0", 2024-02-21T23:04:01.870 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDVgNZlxNJbLxAAXot7Q3chrLEBsFNO8kT6QQ==", 2024-02-21T23:04:01.870 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:04:01.870 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:04:01.870 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:04:01.870 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:04:01.870 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:04:01.870 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:04:01.870 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:04:01.870 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.1", 2024-02-21T23:04:01.870 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDYgNZlZZfWORAAr4XkOxLYLmumVbu362N2HQ==", 2024-02-21T23:04:01.870 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:04:01.871 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:04:01.871 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:04:01.871 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:04:01.871 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:04:01.871 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:04:01.871 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:04:01.871 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.2", 2024-02-21T23:04:01.871 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDcgNZl9mcDGBAAIZGOF8IUMenwo386x95mwg==", 2024-02-21T23:04:01.871 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:04:01.871 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:04:01.871 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:04:01.871 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:04:01.871 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:04:01.872 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:04:01.872 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:04:01.872 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.3", 2024-02-21T23:04:01.872 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDfgNZl8ALSMBAAag9cmGH0/4xALWSUmrZowg==", 2024-02-21T23:04:01.872 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:04:01.872 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:04:01.872 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:04:01.872 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:04:01.872 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:04:01.872 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:04:01.872 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:04:01.872 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.4", 2024-02-21T23:04:01.873 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDngNZlBgeKABAAeW99RVfCBvB1lp7w7mQBlQ==", 2024-02-21T23:04:01.873 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:04:01.873 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:04:01.873 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:04:01.873 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:04:01.873 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:04:01.873 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:04:01.873 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:04:01.873 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.5", 2024-02-21T23:04:01.873 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDqgNZljQG3ChAAY/IZEIcvcGNvlFGIWib3gg==", 2024-02-21T23:04:01.873 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:04:01.873 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:04:01.874 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:04:01.874 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:04:01.874 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:04:01.874 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:04:01.874 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:04:01.874 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.6", 2024-02-21T23:04:01.874 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDtgNZlKnakIxAAkW5xlkshzvEHJdmpgPxR+g==", 2024-02-21T23:04:01.874 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:04:01.874 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:04:01.874 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:04:01.874 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:04:01.874 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:04:01.874 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:04:01.874 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:04:01.875 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.7", 2024-02-21T23:04:01.875 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDxgNZlTWxYARAASX1/gmqO+8mQPP8uA4xjiQ==", 2024-02-21T23:04:01.875 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:04:01.875 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:04:01.875 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:04:01.875 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:04:01.875 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:04:01.875 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:04:01.875 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:04:01.875 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.0", 2024-02-21T23:04:01.875 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDTgNZlILvYLxAAoREWxKKT7YcXnhkgk8YvoQ==", 2024-02-21T23:04:01.875 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:04:01.875 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:04:01.876 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-02-21T23:04:01.876 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-02-21T23:04:01.876 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-02-21T23:04:01.876 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:04:01.876 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:04:01.876 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:04:01.876 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.1", 2024-02-21T23:04:01.876 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDTgNZlm7ImOxAA8iRcXlzFPZPaXQfsKhuO/g==", 2024-02-21T23:04:01.876 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:04:01.876 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:04:01.876 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-02-21T23:04:01.876 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-02-21T23:04:01.876 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-02-21T23:04:01.876 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:04:01.877 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:04:01.877 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:04:01.877 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.admin", 2024-02-21T23:04:01.877 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDRgNZlBYd1GhAAbK5qBCIkf0Ej5nEyPnTxIQ==", 2024-02-21T23:04:01.877 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:04:01.877 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-02-21T23:04:01.877 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-02-21T23:04:01.877 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-02-21T23:04:01.877 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:04:01.877 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:04:01.877 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:04:01.877 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:04:01.877 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.bootstrap-mds", 2024-02-21T23:04:01.877 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQD5gNZlb5s9JBAADcvPfXkiP+V5dZ8+YeeMuw==", 2024-02-21T23:04:01.878 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:04:01.878 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile bootstrap-mds" 2024-02-21T23:04:01.878 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:04:01.878 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:04:01.878 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:04:01.878 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.bootstrap-mgr", 2024-02-21T23:04:01.878 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQD5gNZlibA9JBAAA8oa5WyxDUN6W1AUrHoQAw==", 2024-02-21T23:04:01.878 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:04:01.878 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile bootstrap-mgr" 2024-02-21T23:04:01.878 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:04:01.878 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:04:01.878 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:04:01.878 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.bootstrap-osd", 2024-02-21T23:04:01.878 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQD5gNZlp8E9JBAAbjXzyKgiCndMfBPa4nXm9A==", 2024-02-21T23:04:01.879 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:04:01.879 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile bootstrap-osd" 2024-02-21T23:04:01.879 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:04:01.879 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:04:01.879 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:04:01.879 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.bootstrap-rbd", 2024-02-21T23:04:01.879 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQD5gNZlAtU9JBAA2ACVFN3ctWcUfKjuMdeI8w==", 2024-02-21T23:04:01.879 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:04:01.879 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile bootstrap-rbd" 2024-02-21T23:04:01.879 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:04:01.879 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:04:01.879 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:04:01.879 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-02-21T23:04:01.879 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQD5gNZlieY9JBAAoShh//cexncNwZPyjGFtjg==", 2024-02-21T23:04:01.880 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:04:01.880 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-02-21T23:04:01.880 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:04:01.880 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:04:01.880 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:04:01.880 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.bootstrap-rgw", 2024-02-21T23:04:01.880 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQD5gNZl9/c9JBAA9Y91rlKPZS6TeseL/AMpyw==", 2024-02-21T23:04:01.880 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:04:01.880 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile bootstrap-rgw" 2024-02-21T23:04:01.880 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:04:01.880 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:04:01.880 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:04:01.880 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.x", 2024-02-21T23:04:01.880 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDSgNZlJYKeHBAAYPskmLRr7Jw44TdKzM/JMg==", 2024-02-21T23:04:01.881 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:04:01.881 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-02-21T23:04:01.881 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-02-21T23:04:01.881 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:04:01.881 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:04:01.881 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:04:01.881 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:04:01.881 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.y", 2024-02-21T23:04:01.881 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDSgNZlSRkOEBAAOhRj9PsZDpjoV6WT4drrZA==", 2024-02-21T23:04:01.881 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:04:01.881 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-02-21T23:04:01.881 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-02-21T23:04:01.881 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:04:01.881 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:04:01.881 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:04:01.882 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-21T23:04:01.882 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-21T23:04:01.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.880+0000 7fd806200640 1 -- 172.21.15.28:0/2499836004 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd7e006b4d0 msgr2=0x7fd7e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:01.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.880+0000 7fd806200640 1 --2- 172.21.15.28:0/2499836004 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd7e006b4d0 0x7fd7e006d990 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fd7f0006fd0 tx=0x7fd7f0008040 comp rx=0 tx=0).stop 2024-02-21T23:04:01.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.880+0000 7fd806200640 1 -- 172.21.15.28:0/2499836004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8000c6450 msgr2=0x7fd8001542c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:01.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.880+0000 7fd806200640 1 --2- 172.21.15.28:0/2499836004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8000c6450 0x7fd8001542c0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec00c430 tx=0x7fd7ec00c8f0 comp rx=0 tx=0).stop 2024-02-21T23:04:01.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.880+0000 7fd806200640 1 -- 172.21.15.28:0/2499836004 shutdown_connections 2024-02-21T23:04:01.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.880+0000 7fd806200640 1 --2- 172.21.15.28:0/2499836004 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd7e006b4d0 0x7fd7e006d990 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:01.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.880+0000 7fd806200640 1 --2- 172.21.15.28:0/2499836004 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd800152570 0x7fd800154800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:01.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.880+0000 7fd806200640 1 --2- 172.21.15.28:0/2499836004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8000c6450 0x7fd8001542c0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:01.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.880+0000 7fd806200640 1 --2- 172.21.15.28:0/2499836004 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd800070d10 0x7fd80006d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:01.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.880+0000 7fd806200640 1 -- 172.21.15.28:0/2499836004 >> 172.21.15.28:0/2499836004 conn(0x7fd8000aa280 msgr2=0x7fd8000c8e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:01.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.880+0000 7fd806200640 1 -- 172.21.15.28:0/2499836004 shutdown_connections 2024-02-21T23:04:01.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:01.881+0000 7fd806200640 1 -- 172.21.15.28:0/2499836004 wait complete. 2024-02-21T23:04:01.887 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-02-21T23:04:02.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.120+0000 7f54e3c00640 1 Processor -- start 2024-02-21T23:04:02.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.121+0000 7f54e3c00640 1 -- start start 2024-02-21T23:04:02.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.122+0000 7f54e3c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f54dc072430 0x7f54dc0c3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:02.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.122+0000 7f54e3c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54dc0c6560 con 0x7f54dc0c3280 2024-02-21T23:04:02.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.122+0000 7f54e3c00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54dc0c5cc0 con 0x7f54dc155860 2024-02-21T23:04:02.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.122+0000 7f54e3c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54dc0c5fa0 con 0x7f54dc072430 2024-02-21T23:04:02.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.122+0000 7f54e1200640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f54dc0c3280 0x7f54dc1526e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:40500/0 (socket says 172.21.15.28:40500) 2024-02-21T23:04:02.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.122+0000 7f54e1200640 1 -- 172.21.15.28:0/172867323 learned_addr learned my addr 172.21.15.28:0/172867323 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:04:02.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.123+0000 7f54dbe00640 1 --2- 172.21.15.28:0/172867323 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f54dc072430 0x7f54dc0c3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:02.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.123+0000 7f54db400640 1 -- 172.21.15.28:0/172867323 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1656173632 0 0) 0x7f54dc0c6560 con 0x7f54dc0c3280 2024-02-21T23:04:02.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.123+0000 7f54db400640 1 -- 172.21.15.28:0/172867323 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54c4003520 con 0x7f54dc0c3280 2024-02-21T23:04:02.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.123+0000 7f54db400640 1 -- 172.21.15.28:0/172867323 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2570178179 0 0) 0x7f54c4003520 con 0x7f54dc0c3280 2024-02-21T23:04:02.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.123+0000 7f54db400640 1 -- 172.21.15.28:0/172867323 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f54dc072430 msgr2=0x7f54dc0c3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:02.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.123+0000 7f54db400640 1 --2- 172.21.15.28:0/172867323 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f54dc072430 0x7f54dc0c3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:02.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.123+0000 7f54db400640 1 -- 172.21.15.28:0/172867323 >> v1:172.21.15.81:6789/0 conn(0x7f54dc155860 legacy=0x7f54dc15c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:02.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.123+0000 7f54db400640 1 -- 172.21.15.28:0/172867323 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54dc0c6280 con 0x7f54dc0c3280 2024-02-21T23:04:02.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.123+0000 7f54db400640 1 -- 172.21.15.28:0/172867323 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f54d0001670 con 0x7f54dc0c3280 2024-02-21T23:04:02.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.123+0000 7f54db400640 1 -- 172.21.15.28:0/172867323 >> v1:172.21.15.28:6789/0 conn(0x7f54dc0c3280 legacy=0x7f54dc1526e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:02.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.123+0000 7f54db400640 1 --2- 172.21.15.28:0/172867323 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54c40039e0 0x7f54c4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:02.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.123+0000 7f54db400640 1 --2- 172.21.15.28:0/172867323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54c4004800 0x7f54c4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:02.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.124+0000 7f54db400640 1 --2- 172.21.15.28:0/172867323 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f54dc072430 0x7f54c4005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:02.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.124+0000 7f54db400640 1 -- 172.21.15.28:0/172867323 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54dc0c5fa0 con 0x7f54c4004800 2024-02-21T23:04:02.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.124+0000 7f54db400640 1 -- 172.21.15.28:0/172867323 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54c400baf0 con 0x7f54c40039e0 2024-02-21T23:04:02.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.124+0000 7f54db400640 1 -- 172.21.15.28:0/172867323 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54c400bcf0 con 0x7f54dc072430 2024-02-21T23:04:02.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.124+0000 7f54dbe00640 1 --2- 172.21.15.28:0/172867323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54c4004800 0x7f54c4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:02.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.124+0000 7f54e1200640 1 --2- 172.21.15.28:0/172867323 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54c40039e0 0x7f54c4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:02.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.124+0000 7f54e1c00640 1 --2- 172.21.15.28:0/172867323 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f54dc072430 0x7f54c4005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:02.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.124+0000 7f54dbe00640 1 -- 172.21.15.28:0/172867323 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f54dc072430 msgr2=0x7f54c4005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:02.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.124+0000 7f54dbe00640 1 --2- 172.21.15.28:0/172867323 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f54dc072430 0x7f54c4005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:02.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.124+0000 7f54dbe00640 1 -- 172.21.15.28:0/172867323 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54c40039e0 msgr2=0x7f54c4003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:02.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.124+0000 7f54dbe00640 1 --2- 172.21.15.28:0/172867323 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54c40039e0 0x7f54c4003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:02.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.124+0000 7f54dbe00640 1 -- 172.21.15.28:0/172867323 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54c4003520 con 0x7f54c4004800 2024-02-21T23:04:02.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.125+0000 7f54dbe00640 1 --2- 172.21.15.28:0/172867323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54c4004800 0x7f54c4004c80 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f54c8009650 tx=0x7f54c4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8acd34e6a165980c server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:02.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.126+0000 7f54db400640 1 -- 172.21.15.28:0/172867323 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54c803b070 con 0x7f54c4004800 2024-02-21T23:04:02.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.126+0000 7f54db400640 1 -- 172.21.15.28:0/172867323 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54c802e890 con 0x7f54c4004800 2024-02-21T23:04:02.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.126+0000 7f54db400640 1 -- 172.21.15.28:0/172867323 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54c8009690 con 0x7f54c4004800 2024-02-21T23:04:02.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.126+0000 7f54e3c00640 1 -- 172.21.15.28:0/172867323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54c4004800 msgr2=0x7f54c4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:02.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.126+0000 7f54e3c00640 1 --2- 172.21.15.28:0/172867323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54c4004800 0x7f54c4004c80 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f54c8009650 tx=0x7f54c4005830 comp rx=0 tx=0).stop 2024-02-21T23:04:02.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.126+0000 7f54e1c00640 1 -- 172.21.15.28:0/172867323 reap_dead start 2024-02-21T23:04:02.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.126+0000 7f54e3c00640 1 -- 172.21.15.28:0/172867323 shutdown_connections 2024-02-21T23:04:02.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.126+0000 7f54e3c00640 1 -- 172.21.15.28:0/172867323 >> 172.21.15.28:0/172867323 conn(0x7f54dc0c67f0 msgr2=0x7f54dc0c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:02.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.126+0000 7f54e3c00640 1 -- 172.21.15.28:0/172867323 shutdown_connections 2024-02-21T23:04:02.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.126+0000 7f54e3c00640 1 -- 172.21.15.28:0/172867323 wait complete. 2024-02-21T23:04:02.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.128+0000 7f54e3c00640 1 Processor -- start 2024-02-21T23:04:02.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.128+0000 7f54e3c00640 1 -- start start 2024-02-21T23:04:02.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.129+0000 7f54e3c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54c4004800 0x7f54dc1eb370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:02.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.129+0000 7f54e3c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54dc0a6950 0x7f54dc1ddd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:02.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.129+0000 7f54e3c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f54dc1ee540 0x7f54dc1ee9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:02.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.129+0000 7f54e3c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54dc0c5aa0 con 0x7f54dc0a6950 2024-02-21T23:04:02.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.129+0000 7f54e3c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54dc0c5920 con 0x7f54c4004800 2024-02-21T23:04:02.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.129+0000 7f54e3c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54dc0733e0 con 0x7f54dc1ee540 2024-02-21T23:04:02.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.129+0000 7f54dbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54dc0a6950 0x7f54dc1ddd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:02.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.129+0000 7f54dbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54dc0a6950 0x7f54dc1ddd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45590/0 (socket says 172.21.15.28:45590) 2024-02-21T23:04:02.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.129+0000 7f54dbe00640 1 -- 172.21.15.28:0/1533160246 learned_addr learned my addr 172.21.15.28:0/1533160246 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:02.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.130+0000 7f54e1200640 1 --2- 172.21.15.28:0/1533160246 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54c4004800 0x7f54dc1eb370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:02.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.130+0000 7f54e1c00640 1 --2- 172.21.15.28:0/1533160246 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f54dc1ee540 0x7f54dc1ee9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:02.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.130+0000 7f54dbe00640 1 -- 172.21.15.28:0/1533160246 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f54dc1ee540 msgr2=0x7f54dc1ee9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:02.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.130+0000 7f54dbe00640 1 --2- 172.21.15.28:0/1533160246 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f54dc1ee540 0x7f54dc1ee9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:02.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.130+0000 7f54dbe00640 1 -- 172.21.15.28:0/1533160246 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54c4004800 msgr2=0x7f54dc1eb370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:02.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.130+0000 7f54dbe00640 1 --2- 172.21.15.28:0/1533160246 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54c4004800 0x7f54dc1eb370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:02.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.130+0000 7f54dbe00640 1 -- 172.21.15.28:0/1533160246 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54dc0c1a00 con 0x7f54dc0a6950 2024-02-21T23:04:02.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.130+0000 7f54dbe00640 1 --2- 172.21.15.28:0/1533160246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54dc0a6950 0x7f54dc1ddd70 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f54c8000c00 tx=0x7f54c8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:02.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.130+0000 7f54d8c00640 1 -- 172.21.15.28:0/1533160246 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54c803b040 con 0x7f54dc0a6950 2024-02-21T23:04:02.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.131+0000 7f54d8c00640 1 -- 172.21.15.28:0/1533160246 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54c8044070 con 0x7f54dc0a6950 2024-02-21T23:04:02.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.131+0000 7f54e3c00640 1 -- 172.21.15.28:0/1533160246 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54dc0c1ce0 con 0x7f54dc0a6950 2024-02-21T23:04:02.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.131+0000 7f54d8c00640 1 -- 172.21.15.28:0/1533160246 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54c8003610 con 0x7f54dc0a6950 2024-02-21T23:04:02.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.131+0000 7f54e3c00640 1 -- 172.21.15.28:0/1533160246 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54dc0c22a0 con 0x7f54dc0a6950 2024-02-21T23:04:02.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.132+0000 7f54e3c00640 1 -- 172.21.15.28:0/1533160246 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54dc067f30 con 0x7f54dc0a6950 2024-02-21T23:04:02.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.133+0000 7f54d8c00640 1 -- 172.21.15.28:0/1533160246 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f54c80037b0 con 0x7f54dc0a6950 2024-02-21T23:04:02.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.134+0000 7f54d8c00640 1 --2- 172.21.15.28:0/1533160246 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f54bc06b500 0x7f54bc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:02.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.134+0000 7f54d8c00640 1 -- 172.21.15.28:0/1533160246 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f54c80ab570 con 0x7f54dc0a6950 2024-02-21T23:04:02.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.134+0000 7f54e1200640 1 --2- 172.21.15.28:0/1533160246 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f54bc06b500 0x7f54bc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:02.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.135+0000 7f54e1200640 1 --2- 172.21.15.28:0/1533160246 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f54bc06b500 0x7f54bc06d9c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f54d0001eb0 tx=0x7f54d0000fe0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:02.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.138+0000 7f54d8c00640 1 -- 172.21.15.28:0/1533160246 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f54c804e020 con 0x7f54dc0a6950 2024-02-21T23:04:02.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.398+0000 7f54e3c00640 1 -- 172.21.15.28:0/1533160246 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7f54dc06f2d0 con 0x7f54dc0a6950 2024-02-21T23:04:02.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.403+0000 7f54d8c00640 1 -- 172.21.15.28:0/1533160246 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v5) v1 ==== 90+0+0 (secure 0 0 0) 0x7f54c807a5a0 con 0x7f54dc0a6950 2024-02-21T23:04:02.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.405+0000 7f54e3c00640 1 -- 172.21.15.28:0/1533160246 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f54bc06b500 msgr2=0x7f54bc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:02.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.405+0000 7f54e3c00640 1 --2- 172.21.15.28:0/1533160246 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f54bc06b500 0x7f54bc06d9c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f54d0001eb0 tx=0x7f54d0000fe0 comp rx=0 tx=0).stop 2024-02-21T23:04:02.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.405+0000 7f54e3c00640 1 -- 172.21.15.28:0/1533160246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54dc0a6950 msgr2=0x7f54dc1ddd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:02.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.405+0000 7f54e3c00640 1 --2- 172.21.15.28:0/1533160246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54dc0a6950 0x7f54dc1ddd70 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f54c8000c00 tx=0x7f54c8004730 comp rx=0 tx=0).stop 2024-02-21T23:04:02.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.405+0000 7f54e3c00640 1 -- 172.21.15.28:0/1533160246 shutdown_connections 2024-02-21T23:04:02.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.405+0000 7f54e3c00640 1 --2- 172.21.15.28:0/1533160246 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f54bc06b500 0x7f54bc06d9c0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:02.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.405+0000 7f54e3c00640 1 --2- 172.21.15.28:0/1533160246 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f54dc1ee540 0x7f54dc1ee9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:02.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.405+0000 7f54e3c00640 1 --2- 172.21.15.28:0/1533160246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54dc0a6950 0x7f54dc1ddd70 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:02.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.405+0000 7f54e3c00640 1 --2- 172.21.15.28:0/1533160246 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54c4004800 0x7f54dc1eb370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:02.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.405+0000 7f54e3c00640 1 -- 172.21.15.28:0/1533160246 >> 172.21.15.28:0/1533160246 conn(0x7f54dc0c67f0 msgr2=0x7f54dc14c850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:02.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.406+0000 7f54e3c00640 1 -- 172.21.15.28:0/1533160246 shutdown_connections 2024-02-21T23:04:02.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.406+0000 7f54e3c00640 1 -- 172.21.15.28:0/1533160246 wait complete. 2024-02-21T23:04:02.411 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-02-21T23:04:02.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.633+0000 7f9432800640 1 Processor -- start 2024-02-21T23:04:02.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.633+0000 7f9432800640 1 -- start start 2024-02-21T23:04:02.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.634+0000 7f9432800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f942c153a60 0x7f942c0c44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:02.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.634+0000 7f9432800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f942c0c4a00 0x7f942c06f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:02.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.634+0000 7f9432800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f942c06f300 con 0x7f942c0c7890 2024-02-21T23:04:02.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.634+0000 7f9432800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f942c0c4030 con 0x7f942c0c4a00 2024-02-21T23:04:02.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.634+0000 7f9432800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f942c0722b0 con 0x7f942c153a60 2024-02-21T23:04:02.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.635+0000 7f9431400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f942c0c7890 0x7f942c13be10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:40510/0 (socket says 172.21.15.28:40510) 2024-02-21T23:04:02.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.635+0000 7f9431400640 1 -- 172.21.15.28:0/228995630 learned_addr learned my addr 172.21.15.28:0/228995630 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:04:02.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.635+0000 7f9431e00640 1 --2- 172.21.15.28:0/228995630 >> v2:172.21.15.81:3300/0 conn(0x7f942c0c4a00 0x7f942c06f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:02.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.635+0000 7f9430a00640 1 --2- 172.21.15.28:0/228995630 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f942c153a60 0x7f942c0c44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:02.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.635+0000 7f942be00640 1 -- 172.21.15.28:0/228995630 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2583291118 0 0) 0x7f942c06f300 con 0x7f942c0c7890 2024-02-21T23:04:02.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.635+0000 7f942be00640 1 -- 172.21.15.28:0/228995630 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9410003520 con 0x7f942c0c7890 2024-02-21T23:04:02.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.636+0000 7f942be00640 1 -- 172.21.15.28:0/228995630 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 220148778 0 0) 0x7f9410003520 con 0x7f942c0c7890 2024-02-21T23:04:02.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.636+0000 7f942be00640 1 -- 172.21.15.28:0/228995630 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f942c153a60 msgr2=0x7f942c0c44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:02.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.636+0000 7f942be00640 1 --2- 172.21.15.28:0/228995630 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f942c153a60 0x7f942c0c44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:02.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.636+0000 7f942be00640 1 -- 172.21.15.28:0/228995630 >> v2:172.21.15.81:3300/0 conn(0x7f942c0c4a00 msgr2=0x7f942c06f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:02.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.636+0000 7f942be00640 1 --2- 172.21.15.28:0/228995630 >> v2:172.21.15.81:3300/0 conn(0x7f942c0c4a00 0x7f942c06f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:02.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.636+0000 7f942be00640 1 -- 172.21.15.28:0/228995630 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f942c072590 con 0x7f942c0c7890 2024-02-21T23:04:02.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.636+0000 7f942be00640 1 -- 172.21.15.28:0/228995630 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f941c001670 con 0x7f942c0c7890 2024-02-21T23:04:02.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.636+0000 7f942be00640 1 -- 172.21.15.28:0/228995630 >> v1:172.21.15.28:6789/0 conn(0x7f942c0c7890 legacy=0x7f942c13be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:02.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.636+0000 7f942be00640 1 --2- 172.21.15.28:0/228995630 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f942c153a60 0x7f94100039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:02.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.636+0000 7f942be00640 1 --2- 172.21.15.28:0/228995630 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9410003f30 0x7f9410004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:02.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.636+0000 7f942be00640 1 --2- 172.21.15.28:0/228995630 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9410004d40 0x7f94100051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:02.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.636+0000 7f942be00640 1 -- 172.21.15.28:0/228995630 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f942c0722b0 con 0x7f9410003f30 2024-02-21T23:04:02.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.636+0000 7f942be00640 1 -- 172.21.15.28:0/228995630 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f941000b9c0 con 0x7f9410004d40 2024-02-21T23:04:02.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.636+0000 7f942be00640 1 -- 172.21.15.28:0/228995630 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f941000bb40 con 0x7f942c153a60 2024-02-21T23:04:02.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.636+0000 7f9430a00640 1 --2- 172.21.15.28:0/228995630 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9410003f30 0x7f9410004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:02.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.636+0000 7f9431400640 1 --2- 172.21.15.28:0/228995630 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f942c153a60 0x7f94100039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:02.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.637+0000 7f9431e00640 1 --2- 172.21.15.28:0/228995630 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9410004d40 0x7f94100051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:02.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.637+0000 7f9430a00640 1 -- 172.21.15.28:0/228995630 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f942c153a60 msgr2=0x7f94100039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:02.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.637+0000 7f9430a00640 1 --2- 172.21.15.28:0/228995630 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f942c153a60 0x7f94100039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:02.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.637+0000 7f9430a00640 1 -- 172.21.15.28:0/228995630 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9410004d40 msgr2=0x7f94100051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:02.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.637+0000 7f9430a00640 1 --2- 172.21.15.28:0/228995630 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9410004d40 0x7f94100051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:02.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.637+0000 7f9430a00640 1 -- 172.21.15.28:0/228995630 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9410003520 con 0x7f9410003f30 2024-02-21T23:04:02.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.638+0000 7f9430a00640 1 --2- 172.21.15.28:0/228995630 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9410003f30 0x7f9410004800 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f9418009ac0 tx=0x7f9410005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=866d6685571806e0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:02.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.638+0000 7f942be00640 1 -- 172.21.15.28:0/228995630 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9418039070 con 0x7f9410003f30 2024-02-21T23:04:02.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.638+0000 7f942be00640 1 -- 172.21.15.28:0/228995630 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f941802c6b0 con 0x7f9410003f30 2024-02-21T23:04:02.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.638+0000 7f942be00640 1 -- 172.21.15.28:0/228995630 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9418009690 con 0x7f9410003f30 2024-02-21T23:04:02.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.639+0000 7f9432800640 1 -- 172.21.15.28:0/228995630 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9410003f30 msgr2=0x7f9410004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:02.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.639+0000 7f9432800640 1 --2- 172.21.15.28:0/228995630 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9410003f30 0x7f9410004800 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f9418009ac0 tx=0x7f9410005830 comp rx=0 tx=0).stop 2024-02-21T23:04:02.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.639+0000 7f9431e00640 1 -- 172.21.15.28:0/228995630 reap_dead start 2024-02-21T23:04:02.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.639+0000 7f9432800640 1 -- 172.21.15.28:0/228995630 shutdown_connections 2024-02-21T23:04:02.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.639+0000 7f9432800640 1 -- 172.21.15.28:0/228995630 >> 172.21.15.28:0/228995630 conn(0x7f942c0a88d0 msgr2=0x7f942c06bf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:02.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.639+0000 7f9432800640 1 -- 172.21.15.28:0/228995630 shutdown_connections 2024-02-21T23:04:02.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.639+0000 7f9432800640 1 -- 172.21.15.28:0/228995630 wait complete. 2024-02-21T23:04:02.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.641+0000 7f9432800640 1 Processor -- start 2024-02-21T23:04:02.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.641+0000 7f9432800640 1 -- start start 2024-02-21T23:04:02.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.642+0000 7f9432800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9410003f30 0x7f942c13a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:02.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.642+0000 7f9432800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f942c156080 0x7f942c1e88b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:02.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.642+0000 7f9432800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f942c1564d0 0x7f942c1e8df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:02.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.642+0000 7f9432800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f942c153ec0 con 0x7f942c156080 2024-02-21T23:04:02.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.642+0000 7f9432800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f942c072b50 con 0x7f942c1564d0 2024-02-21T23:04:02.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.642+0000 7f9432800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f942c06cd40 con 0x7f9410003f30 2024-02-21T23:04:02.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.642+0000 7f9430a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f942c156080 0x7f942c1e88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:02.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.642+0000 7f9430a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f942c156080 0x7f942c1e88b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45606/0 (socket says 172.21.15.28:45606) 2024-02-21T23:04:02.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.642+0000 7f9430a00640 1 -- 172.21.15.28:0/2644306991 learned_addr learned my addr 172.21.15.28:0/2644306991 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:02.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.643+0000 7f9431e00640 1 --2- 172.21.15.28:0/2644306991 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f942c1564d0 0x7f942c1e8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:02.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.643+0000 7f9431400640 1 --2- 172.21.15.28:0/2644306991 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9410003f30 0x7f942c13a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:02.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.643+0000 7f9430a00640 1 -- 172.21.15.28:0/2644306991 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9410003f30 msgr2=0x7f942c13a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:02.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.643+0000 7f9430a00640 1 --2- 172.21.15.28:0/2644306991 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9410003f30 0x7f942c13a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:02.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.643+0000 7f9430a00640 1 -- 172.21.15.28:0/2644306991 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f942c1564d0 msgr2=0x7f942c1e8df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:02.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.643+0000 7f9430a00640 1 --2- 172.21.15.28:0/2644306991 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f942c1564d0 0x7f942c1e8df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:02.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.643+0000 7f9430a00640 1 -- 172.21.15.28:0/2644306991 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f942c0ac690 con 0x7f942c156080 2024-02-21T23:04:02.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.643+0000 7f9431400640 1 --2- 172.21.15.28:0/2644306991 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9410003f30 0x7f942c13a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:04:02.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.643+0000 7f9430a00640 1 --2- 172.21.15.28:0/2644306991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f942c156080 0x7f942c1e88b0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f9418009ac0 tx=0x7f9418004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:02.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.643+0000 7f9429600640 1 -- 172.21.15.28:0/2644306991 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9418039050 con 0x7f942c156080 2024-02-21T23:04:02.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.643+0000 7f9429600640 1 -- 172.21.15.28:0/2644306991 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9418045070 con 0x7f942c156080 2024-02-21T23:04:02.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.643+0000 7f9429600640 1 -- 172.21.15.28:0/2644306991 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94180044e0 con 0x7f942c156080 2024-02-21T23:04:02.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.644+0000 7f9432800640 1 -- 172.21.15.28:0/2644306991 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f942c0ac970 con 0x7f942c156080 2024-02-21T23:04:02.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.644+0000 7f9432800640 1 -- 172.21.15.28:0/2644306991 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f942c0acf30 con 0x7f942c156080 2024-02-21T23:04:02.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.645+0000 7f9432800640 1 -- 172.21.15.28:0/2644306991 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f942c04f290 con 0x7f942c156080 2024-02-21T23:04:02.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.646+0000 7f9429600640 1 -- 172.21.15.28:0/2644306991 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9418003850 con 0x7f942c156080 2024-02-21T23:04:02.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.647+0000 7f9429600640 1 --2- 172.21.15.28:0/2644306991 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f940c06b500 0x7f940c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:02.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.647+0000 7f9429600640 1 -- 172.21.15.28:0/2644306991 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f94180abaa0 con 0x7f942c156080 2024-02-21T23:04:02.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.648+0000 7f9431400640 1 --2- 172.21.15.28:0/2644306991 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f940c06b500 0x7f940c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:02.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.649+0000 7f9431400640 1 --2- 172.21.15.28:0/2644306991 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f940c06b500 0x7f940c06d9c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f941c001860 tx=0x7f941c001db0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:02.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.652+0000 7f9429600640 1 -- 172.21.15.28:0/2644306991 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f941804e020 con 0x7f942c156080 2024-02-21T23:04:02.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.933+0000 7f9432800640 1 -- 172.21.15.28:0/2644306991 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7f942c0c1cc0 con 0x7f942c156080 2024-02-21T23:04:02.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.938+0000 7f9429600640 1 -- 172.21.15.28:0/2644306991 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v6) v1 ==== 90+0+0 (secure 0 0 0) 0x7f941807aad0 con 0x7f942c156080 2024-02-21T23:04:02.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.941+0000 7f9432800640 1 -- 172.21.15.28:0/2644306991 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f940c06b500 msgr2=0x7f940c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:02.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.941+0000 7f9432800640 1 --2- 172.21.15.28:0/2644306991 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f940c06b500 0x7f940c06d9c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f941c001860 tx=0x7f941c001db0 comp rx=0 tx=0).stop 2024-02-21T23:04:02.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.941+0000 7f9432800640 1 -- 172.21.15.28:0/2644306991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f942c156080 msgr2=0x7f942c1e88b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:02.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.941+0000 7f9432800640 1 --2- 172.21.15.28:0/2644306991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f942c156080 0x7f942c1e88b0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f9418009ac0 tx=0x7f9418004730 comp rx=0 tx=0).stop 2024-02-21T23:04:02.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.941+0000 7f9432800640 1 -- 172.21.15.28:0/2644306991 shutdown_connections 2024-02-21T23:04:02.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.941+0000 7f9432800640 1 --2- 172.21.15.28:0/2644306991 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f940c06b500 0x7f940c06d9c0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:02.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.941+0000 7f9432800640 1 --2- 172.21.15.28:0/2644306991 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f942c1564d0 0x7f942c1e8df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:02.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.941+0000 7f9432800640 1 --2- 172.21.15.28:0/2644306991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f942c156080 0x7f942c1e88b0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:02.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.941+0000 7f9432800640 1 --2- 172.21.15.28:0/2644306991 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9410003f30 0x7f942c13a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:02.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.941+0000 7f9432800640 1 -- 172.21.15.28:0/2644306991 >> 172.21.15.28:0/2644306991 conn(0x7f942c0a88d0 msgr2=0x7f942c0c2ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:02.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.942+0000 7f9432800640 1 -- 172.21.15.28:0/2644306991 shutdown_connections 2024-02-21T23:04:02.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:02.942+0000 7f9432800640 1 -- 172.21.15.28:0/2644306991 wait complete. 2024-02-21T23:04:02.948 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-02-21T23:04:03.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.182+0000 7fb945400640 1 Processor -- start 2024-02-21T23:04:03.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.182+0000 7fb945400640 1 -- start start 2024-02-21T23:04:03.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.183+0000 7fb945400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fb9400c54c0 0x7fb9400a7f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:03.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.183+0000 7fb945400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb940158940 0x7fb940158db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:03.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.183+0000 7fb945400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9400c6820 con 0x7fb9400c6ef0 2024-02-21T23:04:03.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.184+0000 7fb945400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9400c3de0 con 0x7fb940158940 2024-02-21T23:04:03.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.184+0000 7fb945400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9400c7fe0 con 0x7fb9400c54c0 2024-02-21T23:04:03.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.184+0000 7fb93de00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb9400c6ef0 0x7fb940152560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:40520/0 (socket says 172.21.15.28:40520) 2024-02-21T23:04:03.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.184+0000 7fb93de00640 1 -- 172.21.15.28:0/3901749278 learned_addr learned my addr 172.21.15.28:0/3901749278 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:04:03.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.184+0000 7fb93f200640 1 --2- 172.21.15.28:0/3901749278 >> v2:172.21.15.28:3300/0 conn(0x7fb940158940 0x7fb940158db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:03.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.184+0000 7fb93e800640 1 --2- 172.21.15.28:0/3901749278 >> v2:172.21.15.81:3300/0 conn(0x7fb9400c54c0 0x7fb9400a7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:03.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.185+0000 7fb93d400640 1 -- 172.21.15.28:0/3901749278 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2487791179 0 0) 0x7fb9400c6820 con 0x7fb9400c6ef0 2024-02-21T23:04:03.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.185+0000 7fb93d400640 1 -- 172.21.15.28:0/3901749278 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb924003520 con 0x7fb9400c6ef0 2024-02-21T23:04:03.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.185+0000 7fb93f200640 1 -- 172.21.15.28:0/3901749278 >> v2:172.21.15.81:3300/0 conn(0x7fb9400c54c0 msgr2=0x7fb9400a7f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:03.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.185+0000 7fb93f200640 1 --2- 172.21.15.28:0/3901749278 >> v2:172.21.15.81:3300/0 conn(0x7fb9400c54c0 0x7fb9400a7f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:03.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.185+0000 7fb93f200640 1 -- 172.21.15.28:0/3901749278 >> v1:172.21.15.28:6789/0 conn(0x7fb9400c6ef0 legacy=0x7fb940152560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:03.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.185+0000 7fb93d400640 1 -- 172.21.15.28:0/3901749278 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 660011715 0 0) 0x7fb924003520 con 0x7fb9400c6ef0 2024-02-21T23:04:03.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.185+0000 7fb93f200640 1 -- 172.21.15.28:0/3901749278 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9400c6b00 con 0x7fb940158940 2024-02-21T23:04:03.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.186+0000 7fb93f200640 1 --2- 172.21.15.28:0/3901749278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb940158940 0x7fb940158db0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fb94006be10 tx=0x7fb93802b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a1e31e800448181 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:03.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.186+0000 7fb93d400640 1 -- 172.21.15.28:0/3901749278 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb938037070 con 0x7fb940158940 2024-02-21T23:04:03.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.186+0000 7fb93d400640 1 -- 172.21.15.28:0/3901749278 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb938030510 con 0x7fb940158940 2024-02-21T23:04:03.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.186+0000 7fb93d400640 1 -- 172.21.15.28:0/3901749278 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9300019f0 con 0x7fb940158940 2024-02-21T23:04:03.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.187+0000 7fb945400640 1 -- 172.21.15.28:0/3901749278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb940158940 msgr2=0x7fb940158db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:03.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.187+0000 7fb945400640 1 --2- 172.21.15.28:0/3901749278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb940158940 0x7fb940158db0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fb94006be10 tx=0x7fb93802b710 comp rx=0 tx=0).stop 2024-02-21T23:04:03.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.187+0000 7fb945400640 1 -- 172.21.15.28:0/3901749278 shutdown_connections 2024-02-21T23:04:03.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.187+0000 7fb945400640 1 --2- 172.21.15.28:0/3901749278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb940158940 0x7fb940158db0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:03.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.187+0000 7fb945400640 1 --2- 172.21.15.28:0/3901749278 >> v2:172.21.15.81:3300/0 conn(0x7fb9400c54c0 0x7fb9400a7f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:03.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.187+0000 7fb945400640 1 -- 172.21.15.28:0/3901749278 >> 172.21.15.28:0/3901749278 conn(0x7fb9400c8810 msgr2=0x7fb9400c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:03.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.187+0000 7fb945400640 1 -- 172.21.15.28:0/3901749278 shutdown_connections 2024-02-21T23:04:03.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.187+0000 7fb945400640 1 -- 172.21.15.28:0/3901749278 wait complete. 2024-02-21T23:04:03.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.189+0000 7fb945400640 1 Processor -- start 2024-02-21T23:04:03.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.189+0000 7fb945400640 1 -- start start 2024-02-21T23:04:03.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.190+0000 7fb945400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb9400c54c0 0x7fb94014ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:03.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.190+0000 7fb945400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9400c6ef0 0x7fb9400ac2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:03.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.190+0000 7fb945400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb940158940 0x7fb940157600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:03.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.190+0000 7fb945400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb940152c70 con 0x7fb940158940 2024-02-21T23:04:03.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.190+0000 7fb945400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9400c85a0 con 0x7fb9400c6ef0 2024-02-21T23:04:03.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.190+0000 7fb945400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb940152df0 con 0x7fb9400c54c0 2024-02-21T23:04:03.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.190+0000 7fb93f200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb940158940 0x7fb940157600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:03.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.190+0000 7fb93f200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb940158940 0x7fb940157600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45618/0 (socket says 172.21.15.28:45618) 2024-02-21T23:04:03.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.190+0000 7fb93f200640 1 -- 172.21.15.28:0/2706182463 learned_addr learned my addr 172.21.15.28:0/2706182463 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:03.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.190+0000 7fb93e800640 1 --2- 172.21.15.28:0/2706182463 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb9400c54c0 0x7fb94014ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:03.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.190+0000 7fb93de00640 1 --2- 172.21.15.28:0/2706182463 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9400c6ef0 0x7fb9400ac2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:03.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.191+0000 7fb93f200640 1 -- 172.21.15.28:0/2706182463 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb9400c54c0 msgr2=0x7fb94014ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:03.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.191+0000 7fb93f200640 1 --2- 172.21.15.28:0/2706182463 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb9400c54c0 0x7fb94014ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:03.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.191+0000 7fb93f200640 1 -- 172.21.15.28:0/2706182463 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9400c6ef0 msgr2=0x7fb9400ac2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:03.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.191+0000 7fb93f200640 1 --2- 172.21.15.28:0/2706182463 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9400c6ef0 0x7fb9400ac2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:03.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.191+0000 7fb93f200640 1 -- 172.21.15.28:0/2706182463 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9400ac810 con 0x7fb940158940 2024-02-21T23:04:03.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.191+0000 7fb93f200640 1 --2- 172.21.15.28:0/2706182463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb940158940 0x7fb940157600 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fb93802b750 tx=0x7fb9380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:03.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.191+0000 7fb936a00640 1 -- 172.21.15.28:0/2706182463 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb938041020 con 0x7fb940158940 2024-02-21T23:04:03.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.192+0000 7fb936a00640 1 -- 172.21.15.28:0/2706182463 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb93803c890 con 0x7fb940158940 2024-02-21T23:04:03.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.192+0000 7fb936a00640 1 -- 172.21.15.28:0/2706182463 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb938037040 con 0x7fb940158940 2024-02-21T23:04:03.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.192+0000 7fb945400640 1 -- 172.21.15.28:0/2706182463 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9401edbd0 con 0x7fb940158940 2024-02-21T23:04:03.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.192+0000 7fb945400640 1 -- 172.21.15.28:0/2706182463 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9400ace50 con 0x7fb940158940 2024-02-21T23:04:03.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.193+0000 7fb945400640 1 -- 172.21.15.28:0/2706182463 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9400c3590 con 0x7fb940158940 2024-02-21T23:04:03.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.194+0000 7fb936a00640 1 -- 172.21.15.28:0/2706182463 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb938044020 con 0x7fb940158940 2024-02-21T23:04:03.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.195+0000 7fb936a00640 1 --2- 172.21.15.28:0/2706182463 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb91c06b4d0 0x7fb91c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:03.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.195+0000 7fb936a00640 1 -- 172.21.15.28:0/2706182463 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fb9380ab900 con 0x7fb940158940 2024-02-21T23:04:03.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.195+0000 7fb93e800640 1 --2- 172.21.15.28:0/2706182463 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb91c06b4d0 0x7fb91c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:03.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.196+0000 7fb93e800640 1 --2- 172.21.15.28:0/2706182463 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb91c06b4d0 0x7fb91c06d990 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fb928009ad0 tx=0x7fb928008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:03.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.199+0000 7fb936a00640 1 -- 172.21.15.28:0/2706182463 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb93804b030 con 0x7fb940158940 2024-02-21T23:04:03.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.473+0000 7fb945400640 1 -- 172.21.15.28:0/2706182463 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7fb940157b40 con 0x7fb940158940 2024-02-21T23:04:03.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.477+0000 7fb936a00640 1 -- 172.21.15.28:0/2706182463 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v7) v1 ==== 90+0+0 (secure 0 0 0) 0x7fb93807a9d0 con 0x7fb940158940 2024-02-21T23:04:03.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.480+0000 7fb945400640 1 -- 172.21.15.28:0/2706182463 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb91c06b4d0 msgr2=0x7fb91c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:03.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.480+0000 7fb945400640 1 --2- 172.21.15.28:0/2706182463 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb91c06b4d0 0x7fb91c06d990 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fb928009ad0 tx=0x7fb928008040 comp rx=0 tx=0).stop 2024-02-21T23:04:03.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.480+0000 7fb945400640 1 -- 172.21.15.28:0/2706182463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb940158940 msgr2=0x7fb940157600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:03.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.480+0000 7fb945400640 1 --2- 172.21.15.28:0/2706182463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb940158940 0x7fb940157600 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fb93802b750 tx=0x7fb9380046c0 comp rx=0 tx=0).stop 2024-02-21T23:04:03.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.480+0000 7fb945400640 1 -- 172.21.15.28:0/2706182463 shutdown_connections 2024-02-21T23:04:03.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.480+0000 7fb945400640 1 --2- 172.21.15.28:0/2706182463 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb91c06b4d0 0x7fb91c06d990 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:03.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.480+0000 7fb945400640 1 --2- 172.21.15.28:0/2706182463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb940158940 0x7fb940157600 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:03.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.480+0000 7fb945400640 1 --2- 172.21.15.28:0/2706182463 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9400c6ef0 0x7fb9400ac2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:03.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.480+0000 7fb945400640 1 --2- 172.21.15.28:0/2706182463 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb9400c54c0 0x7fb94014ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:03.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.480+0000 7fb945400640 1 -- 172.21.15.28:0/2706182463 >> 172.21.15.28:0/2706182463 conn(0x7fb9400c8810 msgr2=0x7fb940158d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:03.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.480+0000 7fb945400640 1 -- 172.21.15.28:0/2706182463 shutdown_connections 2024-02-21T23:04:03.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.480+0000 7fb945400640 1 -- 172.21.15.28:0/2706182463 wait complete. 2024-02-21T23:04:03.487 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-02-21T23:04:03.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.692+0000 7f974f200640 1 Processor -- start 2024-02-21T23:04:03.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.692+0000 7f974f200640 1 -- start start 2024-02-21T23:04:03.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.693+0000 7f974f200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9750152590 0x7f9750071590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:03.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.693+0000 7f974f200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9750158e50 0x7f9750154e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:03.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.693+0000 7f974f200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9750072ef0 con 0x7f97500a8c20 2024-02-21T23:04:03.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.693+0000 7f974f200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9750071100 con 0x7f9750152590 2024-02-21T23:04:03.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.693+0000 7f974f200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9750073190 con 0x7f9750158e50 2024-02-21T23:04:03.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.694+0000 7f974de00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9750152590 0x7f9750071590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:03.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.694+0000 7f974de00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9750152590 0x7f9750071590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45626/0 (socket says 172.21.15.28:45626) 2024-02-21T23:04:03.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.694+0000 7f974de00640 1 -- 172.21.15.28:0/1501592037 learned_addr learned my addr 172.21.15.28:0/1501592037 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:03.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.694+0000 7f974de00640 1 -- 172.21.15.28:0/1501592037 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9750158e50 msgr2=0x7f9750154e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:03.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.694+0000 7f974de00640 1 --2- 172.21.15.28:0/1501592037 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9750158e50 0x7f9750154e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:03.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.694+0000 7f974de00640 1 -- 172.21.15.28:0/1501592037 >> v1:172.21.15.81:6789/0 conn(0x7f97500a8c20 legacy=0x7f975006e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:03.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.694+0000 7f974de00640 1 -- 172.21.15.28:0/1501592037 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9750073470 con 0x7f9750152590 2024-02-21T23:04:03.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.695+0000 7f974de00640 1 --2- 172.21.15.28:0/1501592037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9750152590 0x7f9750071590 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f9750071280 tx=0x7f974002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2513508799cb46ff server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:03.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.696+0000 7f974d400640 1 -- 172.21.15.28:0/1501592037 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f974002e070 con 0x7f9750152590 2024-02-21T23:04:03.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.696+0000 7f974d400640 1 -- 172.21.15.28:0/1501592037 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9740037070 con 0x7f9750152590 2024-02-21T23:04:03.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.696+0000 7f974d400640 1 -- 172.21.15.28:0/1501592037 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9740031a00 con 0x7f9750152590 2024-02-21T23:04:03.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.696+0000 7f974f200640 1 -- 172.21.15.28:0/1501592037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9750152590 msgr2=0x7f9750071590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:03.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.696+0000 7f974f200640 1 --2- 172.21.15.28:0/1501592037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9750152590 0x7f9750071590 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f9750071280 tx=0x7f974002b880 comp rx=0 tx=0).stop 2024-02-21T23:04:03.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.696+0000 7f974f200640 1 -- 172.21.15.28:0/1501592037 shutdown_connections 2024-02-21T23:04:03.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.696+0000 7f974f200640 1 --2- 172.21.15.28:0/1501592037 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9750158e50 0x7f9750154e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:03.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.696+0000 7f974f200640 1 --2- 172.21.15.28:0/1501592037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9750152590 0x7f9750071590 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:03.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.696+0000 7f974f200640 1 -- 172.21.15.28:0/1501592037 >> 172.21.15.28:0/1501592037 conn(0x7f975006b550 msgr2=0x7f97500c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:03.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.697+0000 7f974f200640 1 -- 172.21.15.28:0/1501592037 shutdown_connections 2024-02-21T23:04:03.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.697+0000 7f974f200640 1 -- 172.21.15.28:0/1501592037 wait complete. 2024-02-21T23:04:03.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.698+0000 7f974f200640 1 Processor -- start 2024-02-21T23:04:03.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.698+0000 7f974f200640 1 -- start start 2024-02-21T23:04:03.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.699+0000 7f974f200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f97500a8c20 0x7f9750144de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:03.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.699+0000 7f974f200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9750152590 0x7f97500a93a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:03.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.699+0000 7f974f200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9750158e50 0x7f9750146420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:03.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.699+0000 7f974f200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f975006e8a0 con 0x7f9750152590 2024-02-21T23:04:03.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.699+0000 7f974f200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f975006e720 con 0x7f9750158e50 2024-02-21T23:04:03.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.699+0000 7f974f200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97500c0f00 con 0x7f97500a8c20 2024-02-21T23:04:03.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.699+0000 7f9747e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9750152590 0x7f97500a93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:03.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.700+0000 7f9747e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9750152590 0x7f97500a93a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45634/0 (socket says 172.21.15.28:45634) 2024-02-21T23:04:03.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.700+0000 7f9747e00640 1 -- 172.21.15.28:0/835906563 learned_addr learned my addr 172.21.15.28:0/835906563 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:03.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.700+0000 7f974e800640 1 --2- 172.21.15.28:0/835906563 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9750158e50 0x7f9750146420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:03.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.700+0000 7f9747e00640 1 -- 172.21.15.28:0/835906563 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f97500a8c20 msgr2=0x7f9750144de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:03.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.700+0000 7f9747e00640 1 --2- 172.21.15.28:0/835906563 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f97500a8c20 0x7f9750144de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:03.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.700+0000 7f9747e00640 1 -- 172.21.15.28:0/835906563 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9750158e50 msgr2=0x7f9750146420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:03.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.700+0000 7f9747e00640 1 --2- 172.21.15.28:0/835906563 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9750158e50 0x7f9750146420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:03.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.700+0000 7f9747e00640 1 -- 172.21.15.28:0/835906563 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9750153c80 con 0x7f9750152590 2024-02-21T23:04:03.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.700+0000 7f9747e00640 1 --2- 172.21.15.28:0/835906563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9750152590 0x7f97500a93a0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f973800c440 tx=0x7f973800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:03.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.701+0000 7f9746000640 1 -- 172.21.15.28:0/835906563 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9738017070 con 0x7f9750152590 2024-02-21T23:04:03.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.701+0000 7f9746000640 1 -- 172.21.15.28:0/835906563 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97380035f0 con 0x7f9750152590 2024-02-21T23:04:03.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.701+0000 7f9746000640 1 -- 172.21.15.28:0/835906563 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97380039e0 con 0x7f9750152590 2024-02-21T23:04:03.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.701+0000 7f974f200640 1 -- 172.21.15.28:0/835906563 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97500ab280 con 0x7f9750152590 2024-02-21T23:04:03.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.701+0000 7f974f200640 1 -- 172.21.15.28:0/835906563 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97500ab7d0 con 0x7f9750152590 2024-02-21T23:04:03.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.702+0000 7f974f200640 1 -- 172.21.15.28:0/835906563 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f975004f290 con 0x7f9750152590 2024-02-21T23:04:03.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.703+0000 7f9746000640 1 -- 172.21.15.28:0/835906563 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9738003b80 con 0x7f9750152590 2024-02-21T23:04:03.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.704+0000 7f9746000640 1 --2- 172.21.15.28:0/835906563 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f972c06b4d0 0x7f972c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:03.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.704+0000 7f9746000640 1 -- 172.21.15.28:0/835906563 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f9738089370 con 0x7f9750152590 2024-02-21T23:04:03.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.704+0000 7f974de00640 1 --2- 172.21.15.28:0/835906563 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f972c06b4d0 0x7f972c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:03.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.706+0000 7f974de00640 1 --2- 172.21.15.28:0/835906563 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f972c06b4d0 0x7f972c06d990 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f9740036000 tx=0x7f9740030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:03.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.709+0000 7f9746000640 1 -- 172.21.15.28:0/835906563 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f97380583a0 con 0x7f9750152590 2024-02-21T23:04:03.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.984+0000 7f974f200640 1 -- 172.21.15.28:0/835906563 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7f97500a7b30 con 0x7f9750152590 2024-02-21T23:04:03.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.988+0000 7f9746000640 1 -- 172.21.15.28:0/835906563 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v8) v1 ==== 90+0+0 (secure 0 0 0) 0x7f97380010c0 con 0x7f9750152590 2024-02-21T23:04:03.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.990+0000 7f974f200640 1 -- 172.21.15.28:0/835906563 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f972c06b4d0 msgr2=0x7f972c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:03.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.990+0000 7f974f200640 1 --2- 172.21.15.28:0/835906563 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f972c06b4d0 0x7f972c06d990 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f9740036000 tx=0x7f9740030000 comp rx=0 tx=0).stop 2024-02-21T23:04:03.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.990+0000 7f974f200640 1 -- 172.21.15.28:0/835906563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9750152590 msgr2=0x7f97500a93a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:03.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.990+0000 7f974f200640 1 --2- 172.21.15.28:0/835906563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9750152590 0x7f97500a93a0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f973800c440 tx=0x7f973800c900 comp rx=0 tx=0).stop 2024-02-21T23:04:03.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.991+0000 7f974f200640 1 -- 172.21.15.28:0/835906563 shutdown_connections 2024-02-21T23:04:03.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.991+0000 7f974f200640 1 --2- 172.21.15.28:0/835906563 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f972c06b4d0 0x7f972c06d990 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:03.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.991+0000 7f974f200640 1 --2- 172.21.15.28:0/835906563 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9750158e50 0x7f9750146420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:03.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.991+0000 7f974f200640 1 --2- 172.21.15.28:0/835906563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9750152590 0x7f97500a93a0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:03.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.991+0000 7f974f200640 1 --2- 172.21.15.28:0/835906563 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f97500a8c20 0x7f9750144de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:03.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.991+0000 7f974f200640 1 -- 172.21.15.28:0/835906563 >> 172.21.15.28:0/835906563 conn(0x7f975006b550 msgr2=0x7f9750150a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:03.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.991+0000 7f974f200640 1 -- 172.21.15.28:0/835906563 shutdown_connections 2024-02-21T23:04:03.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:03.991+0000 7f974f200640 1 -- 172.21.15.28:0/835906563 wait complete. 2024-02-21T23:04:03.999 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-02-21T23:04:04.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.234+0000 7fb604200640 1 Processor -- start 2024-02-21T23:04:04.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.234+0000 7fb604200640 1 -- start start 2024-02-21T23:04:04.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.236+0000 7fb604200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fb5fc152890 0x7fb5fc0c3c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:04.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.236+0000 7fb604200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb5fc070f80 0x7fb5fc0c1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:04.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.236+0000 7fb604200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5fc159110 con 0x7fb5fc0c2330 2024-02-21T23:04:04.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.236+0000 7fb604200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5fc0c5100 con 0x7fb5fc152890 2024-02-21T23:04:04.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.236+0000 7fb604200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5fc1593f0 con 0x7fb5fc070f80 2024-02-21T23:04:04.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.236+0000 7fb602200640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb5fc0c2330 0x7fb5fc158650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:40528/0 (socket says 172.21.15.28:40528) 2024-02-21T23:04:04.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.236+0000 7fb602200640 1 -- 172.21.15.28:0/4134071510 learned_addr learned my addr 172.21.15.28:0/4134071510 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:04:04.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.237+0000 7fb600e00640 1 --2- 172.21.15.28:0/4134071510 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb5fc070f80 0x7fb5fc0c1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:04.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.237+0000 7fb5fbe00640 1 -- 172.21.15.28:0/4134071510 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 979335662 0 0) 0x7fb5fc159110 con 0x7fb5fc0c2330 2024-02-21T23:04:04.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.237+0000 7fb601800640 1 --2- 172.21.15.28:0/4134071510 >> v2:172.21.15.81:3300/0 conn(0x7fb5fc152890 0x7fb5fc0c3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:04.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.237+0000 7fb5fbe00640 1 -- 172.21.15.28:0/4134071510 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5e4003520 con 0x7fb5fc0c2330 2024-02-21T23:04:04.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.237+0000 7fb5fbe00640 1 -- 172.21.15.28:0/4134071510 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2405118127 0 0) 0x7fb5e4003520 con 0x7fb5fc0c2330 2024-02-21T23:04:04.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.237+0000 7fb5fbe00640 1 -- 172.21.15.28:0/4134071510 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb5fc070f80 msgr2=0x7fb5fc0c1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:04.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.237+0000 7fb5fbe00640 1 --2- 172.21.15.28:0/4134071510 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb5fc070f80 0x7fb5fc0c1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:04.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.237+0000 7fb5fbe00640 1 -- 172.21.15.28:0/4134071510 >> v2:172.21.15.81:3300/0 conn(0x7fb5fc152890 msgr2=0x7fb5fc0c3c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:04.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.237+0000 7fb5fbe00640 1 --2- 172.21.15.28:0/4134071510 >> v2:172.21.15.81:3300/0 conn(0x7fb5fc152890 0x7fb5fc0c3c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:04.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.237+0000 7fb5fbe00640 1 -- 172.21.15.28:0/4134071510 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5fc070780 con 0x7fb5fc0c2330 2024-02-21T23:04:04.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.237+0000 7fb5fbe00640 1 -- 172.21.15.28:0/4134071510 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fb5f40019f0 con 0x7fb5fc0c2330 2024-02-21T23:04:04.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.237+0000 7fb600e00640 1 --2- 172.21.15.28:0/4134071510 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb5fc070f80 0x7fb5fc0c1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:04:04.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.237+0000 7fb5fbe00640 1 -- 172.21.15.28:0/4134071510 >> v1:172.21.15.28:6789/0 conn(0x7fb5fc0c2330 legacy=0x7fb5fc158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:04.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.237+0000 7fb5fbe00640 1 --2- 172.21.15.28:0/4134071510 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5e40039b0 0x7fb5e4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:04.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.237+0000 7fb5fbe00640 1 --2- 172.21.15.28:0/4134071510 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb5fc070f80 0x7fb5e4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:04.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.237+0000 7fb5fbe00640 1 --2- 172.21.15.28:0/4134071510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5e4004d40 0x7fb5e40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:04.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.237+0000 7fb5fbe00640 1 -- 172.21.15.28:0/4134071510 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5fc0c5100 con 0x7fb5e4004d40 2024-02-21T23:04:04.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.237+0000 7fb5fbe00640 1 -- 172.21.15.28:0/4134071510 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5fc1593f0 con 0x7fb5e40039b0 2024-02-21T23:04:04.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.237+0000 7fb5fbe00640 1 -- 172.21.15.28:0/4134071510 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e400ba70 con 0x7fb5fc070f80 2024-02-21T23:04:04.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.238+0000 7fb602200640 1 --2- 172.21.15.28:0/4134071510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5e4004d40 0x7fb5e40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:04.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.238+0000 7fb600e00640 1 --2- 172.21.15.28:0/4134071510 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb5fc070f80 0x7fb5e4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:04.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.238+0000 7fb601800640 1 --2- 172.21.15.28:0/4134071510 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5e40039b0 0x7fb5e4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:04.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.238+0000 7fb602200640 1 -- 172.21.15.28:0/4134071510 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb5fc070f80 msgr2=0x7fb5e4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:04.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.238+0000 7fb602200640 1 --2- 172.21.15.28:0/4134071510 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb5fc070f80 0x7fb5e4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:04.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.238+0000 7fb602200640 1 -- 172.21.15.28:0/4134071510 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5e40039b0 msgr2=0x7fb5e4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:04.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.238+0000 7fb602200640 1 --2- 172.21.15.28:0/4134071510 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5e40039b0 0x7fb5e4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:04.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.238+0000 7fb602200640 1 -- 172.21.15.28:0/4134071510 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5e4003520 con 0x7fb5e4004d40 2024-02-21T23:04:04.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.239+0000 7fb602200640 1 --2- 172.21.15.28:0/4134071510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5e4004d40 0x7fb5e40051c0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fb5f400a920 tx=0x7fb5e4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95462c220ee08c7d server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:04.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.240+0000 7fb5fbe00640 1 -- 172.21.15.28:0/4134071510 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5f400a4c0 con 0x7fb5e4004d40 2024-02-21T23:04:04.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.240+0000 7fb5fbe00640 1 -- 172.21.15.28:0/4134071510 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5f4034650 con 0x7fb5e4004d40 2024-02-21T23:04:04.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.240+0000 7fb5fbe00640 1 -- 172.21.15.28:0/4134071510 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5f4034a40 con 0x7fb5e4004d40 2024-02-21T23:04:04.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.240+0000 7fb604200640 1 -- 172.21.15.28:0/4134071510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5e4004d40 msgr2=0x7fb5e40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:04.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.240+0000 7fb604200640 1 --2- 172.21.15.28:0/4134071510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5e4004d40 0x7fb5e40051c0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fb5f400a920 tx=0x7fb5e4005830 comp rx=0 tx=0).stop 2024-02-21T23:04:04.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.240+0000 7fb602200640 1 -- 172.21.15.28:0/4134071510 reap_dead start 2024-02-21T23:04:04.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.240+0000 7fb604200640 1 -- 172.21.15.28:0/4134071510 shutdown_connections 2024-02-21T23:04:04.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.240+0000 7fb604200640 1 -- 172.21.15.28:0/4134071510 >> 172.21.15.28:0/4134071510 conn(0x7fb5fc0aa550 msgr2=0x7fb5fc0aa930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:04.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.241+0000 7fb604200640 1 -- 172.21.15.28:0/4134071510 shutdown_connections 2024-02-21T23:04:04.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.241+0000 7fb604200640 1 -- 172.21.15.28:0/4134071510 wait complete. 2024-02-21T23:04:04.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.242+0000 7fb604200640 1 Processor -- start 2024-02-21T23:04:04.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.242+0000 7fb604200640 1 -- start start 2024-02-21T23:04:04.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.243+0000 7fb604200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5e4004d40 0x7fb5fc1514a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:04.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.243+0000 7fb604200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb5fc1519e0 0x7fb5fc1eb600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:04.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.243+0000 7fb604200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5fc148df0 0x7fb5fc1ebb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:04.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.243+0000 7fb604200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5fc0a80f0 con 0x7fb5fc148df0 2024-02-21T23:04:04.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.243+0000 7fb604200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5fc0c41d0 con 0x7fb5e4004d40 2024-02-21T23:04:04.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.243+0000 7fb604200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5fc158d60 con 0x7fb5fc1519e0 2024-02-21T23:04:04.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.244+0000 7fb602200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5fc148df0 0x7fb5fc1ebb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:04.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.244+0000 7fb602200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5fc148df0 0x7fb5fc1ebb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45666/0 (socket says 172.21.15.28:45666) 2024-02-21T23:04:04.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.244+0000 7fb602200640 1 -- 172.21.15.28:0/2054664464 learned_addr learned my addr 172.21.15.28:0/2054664464 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:04.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.244+0000 7fb600e00640 1 --2- 172.21.15.28:0/2054664464 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb5fc1519e0 0x7fb5fc1eb600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:04.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.244+0000 7fb601800640 1 --2- 172.21.15.28:0/2054664464 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5e4004d40 0x7fb5fc1514a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:04.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.244+0000 7fb602200640 1 -- 172.21.15.28:0/2054664464 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb5fc1519e0 msgr2=0x7fb5fc1eb600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:04.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.244+0000 7fb602200640 1 --2- 172.21.15.28:0/2054664464 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb5fc1519e0 0x7fb5fc1eb600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:04.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.244+0000 7fb602200640 1 -- 172.21.15.28:0/2054664464 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5e4004d40 msgr2=0x7fb5fc1514a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:04.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.244+0000 7fb602200640 1 --2- 172.21.15.28:0/2054664464 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5e4004d40 0x7fb5fc1514a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:04.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.244+0000 7fb602200640 1 -- 172.21.15.28:0/2054664464 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5fc148940 con 0x7fb5fc148df0 2024-02-21T23:04:04.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.245+0000 7fb602200640 1 --2- 172.21.15.28:0/2054664464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5fc148df0 0x7fb5fc1ebb40 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fb5f402ccb0 tx=0x7fb5fc070950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:04.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.245+0000 7fb5f9600640 1 -- 172.21.15.28:0/2054664464 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5f4045070 con 0x7fb5fc148df0 2024-02-21T23:04:04.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.245+0000 7fb5f9600640 1 -- 172.21.15.28:0/2054664464 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5f403a040 con 0x7fb5fc148df0 2024-02-21T23:04:04.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.245+0000 7fb5f9600640 1 -- 172.21.15.28:0/2054664464 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5f4002000 con 0x7fb5fc148df0 2024-02-21T23:04:04.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.245+0000 7fb604200640 1 -- 172.21.15.28:0/2054664464 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5fc1ef080 con 0x7fb5fc148df0 2024-02-21T23:04:04.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.245+0000 7fb604200640 1 -- 172.21.15.28:0/2054664464 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5fc1e22d0 con 0x7fb5fc148df0 2024-02-21T23:04:04.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.246+0000 7fb604200640 1 -- 172.21.15.28:0/2054664464 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5fc04f370 con 0x7fb5fc148df0 2024-02-21T23:04:04.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.247+0000 7fb5f9600640 1 -- 172.21.15.28:0/2054664464 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb5f4048070 con 0x7fb5fc148df0 2024-02-21T23:04:04.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.248+0000 7fb5f9600640 1 --2- 172.21.15.28:0/2054664464 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb5dc06b4d0 0x7fb5dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:04.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.248+0000 7fb5f9600640 1 -- 172.21.15.28:0/2054664464 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fb5f40abb20 con 0x7fb5fc148df0 2024-02-21T23:04:04.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.249+0000 7fb601800640 1 --2- 172.21.15.28:0/2054664464 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb5dc06b4d0 0x7fb5dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:04.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.250+0000 7fb601800640 1 --2- 172.21.15.28:0/2054664464 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb5dc06b4d0 0x7fb5dc06d990 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb5f0006fd0 tx=0x7fb5f0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:04.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.253+0000 7fb5f9600640 1 -- 172.21.15.28:0/2054664464 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb5f407a500 con 0x7fb5fc148df0 2024-02-21T23:04:04.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.516+0000 7fb604200640 1 -- 172.21.15.28:0/2054664464 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7fb5fc0c94e0 con 0x7fb5fc148df0 2024-02-21T23:04:04.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.520+0000 7fb5f9600640 1 -- 172.21.15.28:0/2054664464 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v9) v1 ==== 97+0+0 (secure 0 0 0) 0x7fb5f407e5a0 con 0x7fb5fc148df0 2024-02-21T23:04:04.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.523+0000 7fb604200640 1 -- 172.21.15.28:0/2054664464 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb5dc06b4d0 msgr2=0x7fb5dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:04.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.523+0000 7fb604200640 1 --2- 172.21.15.28:0/2054664464 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb5dc06b4d0 0x7fb5dc06d990 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb5f0006fd0 tx=0x7fb5f0008040 comp rx=0 tx=0).stop 2024-02-21T23:04:04.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.523+0000 7fb604200640 1 -- 172.21.15.28:0/2054664464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5fc148df0 msgr2=0x7fb5fc1ebb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:04.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.523+0000 7fb604200640 1 --2- 172.21.15.28:0/2054664464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5fc148df0 0x7fb5fc1ebb40 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fb5f402ccb0 tx=0x7fb5fc070950 comp rx=0 tx=0).stop 2024-02-21T23:04:04.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.523+0000 7fb604200640 1 -- 172.21.15.28:0/2054664464 shutdown_connections 2024-02-21T23:04:04.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.523+0000 7fb604200640 1 --2- 172.21.15.28:0/2054664464 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb5dc06b4d0 0x7fb5dc06d990 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:04.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.523+0000 7fb604200640 1 --2- 172.21.15.28:0/2054664464 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb5fc1519e0 0x7fb5fc1eb600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:04.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.523+0000 7fb604200640 1 --2- 172.21.15.28:0/2054664464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5fc148df0 0x7fb5fc1ebb40 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:04.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.523+0000 7fb604200640 1 --2- 172.21.15.28:0/2054664464 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5e4004d40 0x7fb5fc1514a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:04.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.523+0000 7fb604200640 1 -- 172.21.15.28:0/2054664464 >> 172.21.15.28:0/2054664464 conn(0x7fb5fc0aa550 msgr2=0x7fb5fc0c8510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:04.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.523+0000 7fb604200640 1 -- 172.21.15.28:0/2054664464 shutdown_connections 2024-02-21T23:04:04.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.523+0000 7fb604200640 1 -- 172.21.15.28:0/2054664464 wait complete. 2024-02-21T23:04:04.531 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-02-21T23:04:04.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.755+0000 7f8e18a00640 1 Processor -- start 2024-02-21T23:04:04.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.756+0000 7f8e18a00640 1 -- start start 2024-02-21T23:04:04.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.757+0000 7f8e18a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f8e14072430 0x7f8e140c3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:04.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.757+0000 7f8e18a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8e14155860 0x7f8e14155d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:04.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.757+0000 7f8e18a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e140c6560 con 0x7f8e140c3280 2024-02-21T23:04:04.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.757+0000 7f8e18a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e140c5cc0 con 0x7f8e14072430 2024-02-21T23:04:04.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.757+0000 7f8e18a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e140c5fa0 con 0x7f8e14155860 2024-02-21T23:04:04.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.757+0000 7f8e11e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f8e140c3280 0x7f8e141526e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:40538/0 (socket says 172.21.15.28:40538) 2024-02-21T23:04:04.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.757+0000 7f8e11e00640 1 -- 172.21.15.28:0/2025515505 learned_addr learned my addr 172.21.15.28:0/2025515505 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:04:04.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.758+0000 7f8e0be00640 1 --2- 172.21.15.28:0/2025515505 >> v2:172.21.15.81:3300/0 conn(0x7f8e14072430 0x7f8e140c3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:04.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.758+0000 7f8e12800640 1 --2- 172.21.15.28:0/2025515505 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8e14155860 0x7f8e14155d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:04.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.758+0000 7f8e11400640 1 -- 172.21.15.28:0/2025515505 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3237770837 0 0) 0x7f8e140c6560 con 0x7f8e140c3280 2024-02-21T23:04:04.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.758+0000 7f8e11400640 1 -- 172.21.15.28:0/2025515505 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8df4003520 con 0x7f8e140c3280 2024-02-21T23:04:04.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.758+0000 7f8e11400640 1 -- 172.21.15.28:0/2025515505 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3202483867 0 0) 0x7f8df4003520 con 0x7f8e140c3280 2024-02-21T23:04:04.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.758+0000 7f8e11400640 1 -- 172.21.15.28:0/2025515505 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8e14155860 msgr2=0x7f8e14155d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:04.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.758+0000 7f8e11400640 1 --2- 172.21.15.28:0/2025515505 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8e14155860 0x7f8e14155d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:04.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.758+0000 7f8e11400640 1 -- 172.21.15.28:0/2025515505 >> v2:172.21.15.81:3300/0 conn(0x7f8e14072430 msgr2=0x7f8e140c3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:04.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.758+0000 7f8e11400640 1 --2- 172.21.15.28:0/2025515505 >> v2:172.21.15.81:3300/0 conn(0x7f8e14072430 0x7f8e140c3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:04.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.758+0000 7f8e11400640 1 -- 172.21.15.28:0/2025515505 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e140c6280 con 0x7f8e140c3280 2024-02-21T23:04:04.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.758+0000 7f8e11400640 1 -- 172.21.15.28:0/2025515505 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f8e04001670 con 0x7f8e140c3280 2024-02-21T23:04:04.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.758+0000 7f8e11400640 1 -- 172.21.15.28:0/2025515505 >> v1:172.21.15.28:6789/0 conn(0x7f8e140c3280 legacy=0x7f8e141526e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:04.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.758+0000 7f8e11400640 1 --2- 172.21.15.28:0/2025515505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8df40039b0 0x7f8df4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:04.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.759+0000 7f8e11400640 1 --2- 172.21.15.28:0/2025515505 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8e14155860 0x7f8df4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:04.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.759+0000 7f8e11400640 1 --2- 172.21.15.28:0/2025515505 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8df4004d40 0x7f8df40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:04.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.759+0000 7f8e11400640 1 -- 172.21.15.28:0/2025515505 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e140c5cc0 con 0x7f8df40039b0 2024-02-21T23:04:04.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.759+0000 7f8e11400640 1 -- 172.21.15.28:0/2025515505 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e140c5fa0 con 0x7f8df4004d40 2024-02-21T23:04:04.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.759+0000 7f8e11400640 1 -- 172.21.15.28:0/2025515505 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8df400dbd0 con 0x7f8e14155860 2024-02-21T23:04:04.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.759+0000 7f8e11e00640 1 --2- 172.21.15.28:0/2025515505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8df40039b0 0x7f8df4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:04.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.759+0000 7f8e11e00640 1 -- 172.21.15.28:0/2025515505 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8e14155860 msgr2=0x7f8df4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:04.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.759+0000 7f8e11e00640 1 --2- 172.21.15.28:0/2025515505 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8e14155860 0x7f8df4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:04.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.759+0000 7f8e11e00640 1 -- 172.21.15.28:0/2025515505 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8df4004d40 msgr2=0x7f8df40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:04.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.759+0000 7f8e11e00640 1 --2- 172.21.15.28:0/2025515505 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8df4004d40 0x7f8df40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:04.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.759+0000 7f8e11e00640 1 -- 172.21.15.28:0/2025515505 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8df4003520 con 0x7f8df40039b0 2024-02-21T23:04:04.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.760+0000 7f8e11e00640 1 --2- 172.21.15.28:0/2025515505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8df40039b0 0x7f8df4003dd0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f8e0400c240 tx=0x7f8df4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=721d98be21b91e38 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:04.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.761+0000 7f8e11400640 1 -- 172.21.15.28:0/2025515505 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e04007d30 con 0x7f8df40039b0 2024-02-21T23:04:04.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.761+0000 7f8e11400640 1 -- 172.21.15.28:0/2025515505 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e04030070 con 0x7f8df40039b0 2024-02-21T23:04:04.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.761+0000 7f8e11400640 1 -- 172.21.15.28:0/2025515505 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e0402cd60 con 0x7f8df40039b0 2024-02-21T23:04:04.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.761+0000 7f8e18a00640 1 -- 172.21.15.28:0/2025515505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8df40039b0 msgr2=0x7f8df4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:04.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.761+0000 7f8e18a00640 1 --2- 172.21.15.28:0/2025515505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8df40039b0 0x7f8df4003dd0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f8e0400c240 tx=0x7f8df4005830 comp rx=0 tx=0).stop 2024-02-21T23:04:04.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.761+0000 7f8e12800640 1 -- 172.21.15.28:0/2025515505 reap_dead start 2024-02-21T23:04:04.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.761+0000 7f8e18a00640 1 -- 172.21.15.28:0/2025515505 shutdown_connections 2024-02-21T23:04:04.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.761+0000 7f8e18a00640 1 -- 172.21.15.28:0/2025515505 >> 172.21.15.28:0/2025515505 conn(0x7f8e140c67f0 msgr2=0x7f8e140c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:04.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.761+0000 7f8e18a00640 1 -- 172.21.15.28:0/2025515505 shutdown_connections 2024-02-21T23:04:04.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.762+0000 7f8e18a00640 1 -- 172.21.15.28:0/2025515505 wait complete. 2024-02-21T23:04:04.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.763+0000 7f8e18a00640 1 Processor -- start 2024-02-21T23:04:04.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.763+0000 7f8e18a00640 1 -- start start 2024-02-21T23:04:04.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.764+0000 7f8e18a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8df40039b0 0x7f8e140bfe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:04.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.764+0000 7f8e18a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e140a5d20 0x7f8e14149840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:04.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.764+0000 7f8e18a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8e14153050 0x7f8e14149d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:04.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.764+0000 7f8e18a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e140c5aa0 con 0x7f8e140a5d20 2024-02-21T23:04:04.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.764+0000 7f8e18a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e140c5920 con 0x7f8df40039b0 2024-02-21T23:04:04.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.764+0000 7f8e18a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e140733e0 con 0x7f8e14153050 2024-02-21T23:04:04.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.764+0000 7f8e0be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e140a5d20 0x7f8e14149840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:04.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.765+0000 7f8e0be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e140a5d20 0x7f8e14149840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45680/0 (socket says 172.21.15.28:45680) 2024-02-21T23:04:04.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.765+0000 7f8e0be00640 1 -- 172.21.15.28:0/721690056 learned_addr learned my addr 172.21.15.28:0/721690056 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:04.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.765+0000 7f8e11e00640 1 --2- 172.21.15.28:0/721690056 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8df40039b0 0x7f8e140bfe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:04.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.765+0000 7f8e0be00640 1 -- 172.21.15.28:0/721690056 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8e14153050 msgr2=0x7f8e14149d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:04.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.765+0000 7f8e0be00640 1 --2- 172.21.15.28:0/721690056 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8e14153050 0x7f8e14149d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:04.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.765+0000 7f8e0be00640 1 -- 172.21.15.28:0/721690056 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8df40039b0 msgr2=0x7f8e140bfe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:04.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.765+0000 7f8e0be00640 1 --2- 172.21.15.28:0/721690056 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8df40039b0 0x7f8e140bfe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:04.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.765+0000 7f8e0be00640 1 -- 172.21.15.28:0/721690056 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e140a6730 con 0x7f8e140a5d20 2024-02-21T23:04:04.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.765+0000 7f8e0be00640 1 --2- 172.21.15.28:0/721690056 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e140a5d20 0x7f8e14149840 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc00e7c0 tx=0x7f8dfc00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:04.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.765+0000 7f8e0a000640 1 -- 172.21.15.28:0/721690056 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dfc017070 con 0x7f8e140a5d20 2024-02-21T23:04:04.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.766+0000 7f8e0a000640 1 -- 172.21.15.28:0/721690056 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dfc00c840 con 0x7f8e140a5d20 2024-02-21T23:04:04.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.766+0000 7f8e0a000640 1 -- 172.21.15.28:0/721690056 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dfc00cc30 con 0x7f8e140a5d20 2024-02-21T23:04:04.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.766+0000 7f8e18a00640 1 -- 172.21.15.28:0/721690056 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e141ef2d0 con 0x7f8e140a5d20 2024-02-21T23:04:04.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.766+0000 7f8e18a00640 1 -- 172.21.15.28:0/721690056 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e140c03d0 con 0x7f8e140a5d20 2024-02-21T23:04:04.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.767+0000 7f8e18a00640 1 -- 172.21.15.28:0/721690056 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e140c2b30 con 0x7f8e140a5d20 2024-02-21T23:04:04.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.768+0000 7f8e0a000640 1 -- 172.21.15.28:0/721690056 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8dfc01a070 con 0x7f8e140a5d20 2024-02-21T23:04:04.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.769+0000 7f8e0a000640 1 --2- 172.21.15.28:0/721690056 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8df006b500 0x7f8df006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:04.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.769+0000 7f8e0a000640 1 -- 172.21.15.28:0/721690056 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f8dfc08a110 con 0x7f8e140a5d20 2024-02-21T23:04:04.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.769+0000 7f8e11e00640 1 --2- 172.21.15.28:0/721690056 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8df006b500 0x7f8df006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:04.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.770+0000 7f8e11e00640 1 --2- 172.21.15.28:0/721690056 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8df006b500 0x7f8df006d9c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f8e04034000 tx=0x7f8e04007000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:04.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:04.773+0000 7f8e0a000640 1 -- 172.21.15.28:0/721690056 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f8e140c2b30 con 0x7f8e140a5d20 2024-02-21T23:04:05.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.050+0000 7f8e18a00640 1 -- 172.21.15.28:0/721690056 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7f8e140c7000 con 0x7f8e140a5d20 2024-02-21T23:04:05.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.054+0000 7f8e0a000640 1 -- 172.21.15.28:0/721690056 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v10) v1 ==== 90+0+0 (secure 0 0 0) 0x7f8e140c7000 con 0x7f8e140a5d20 2024-02-21T23:04:05.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.057+0000 7f8e18a00640 1 -- 172.21.15.28:0/721690056 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8df006b500 msgr2=0x7f8df006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:05.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.057+0000 7f8e18a00640 1 --2- 172.21.15.28:0/721690056 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8df006b500 0x7f8df006d9c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f8e04034000 tx=0x7f8e04007000 comp rx=0 tx=0).stop 2024-02-21T23:04:05.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.057+0000 7f8e18a00640 1 -- 172.21.15.28:0/721690056 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e140a5d20 msgr2=0x7f8e14149840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:05.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.057+0000 7f8e18a00640 1 --2- 172.21.15.28:0/721690056 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e140a5d20 0x7f8e14149840 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc00e7c0 tx=0x7f8dfc00ec80 comp rx=0 tx=0).stop 2024-02-21T23:04:05.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.057+0000 7f8e18a00640 1 -- 172.21.15.28:0/721690056 shutdown_connections 2024-02-21T23:04:05.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.057+0000 7f8e18a00640 1 --2- 172.21.15.28:0/721690056 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8df006b500 0x7f8df006d9c0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:05.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.057+0000 7f8e18a00640 1 --2- 172.21.15.28:0/721690056 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8e14153050 0x7f8e14149d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:05.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.057+0000 7f8e18a00640 1 --2- 172.21.15.28:0/721690056 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e140a5d20 0x7f8e14149840 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:05.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.057+0000 7f8e18a00640 1 --2- 172.21.15.28:0/721690056 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8df40039b0 0x7f8e140bfe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:05.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.057+0000 7f8e18a00640 1 -- 172.21.15.28:0/721690056 >> 172.21.15.28:0/721690056 conn(0x7f8e140c67f0 msgr2=0x7f8e1414d120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:05.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.057+0000 7f8e18a00640 1 -- 172.21.15.28:0/721690056 shutdown_connections 2024-02-21T23:04:05.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.057+0000 7f8e18a00640 1 -- 172.21.15.28:0/721690056 wait complete. 2024-02-21T23:04:05.065 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:04:05.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.299+0000 7fb2a7200640 1 Processor -- start 2024-02-21T23:04:05.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.299+0000 7fb2a7200640 1 -- start start 2024-02-21T23:04:05.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.300+0000 7fb2a7200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fb2a00a6160 0x7fb2a0155180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:05.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.300+0000 7fb2a7200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2a0155760 con 0x7fb2a01521c0 2024-02-21T23:04:05.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.300+0000 7fb2a7200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2a0155a40 con 0x7fb2a014f0b0 2024-02-21T23:04:05.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.300+0000 7fb2a7200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2a0151d30 con 0x7fb2a00a6160 2024-02-21T23:04:05.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.301+0000 7fb2a6800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb2a01521c0 0x7fb2a015c510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:56638/0 (socket says 172.21.15.28:56638) 2024-02-21T23:04:05.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.301+0000 7fb2a6800640 1 -- 172.21.15.28:0/1226828501 learned_addr learned my addr 172.21.15.28:0/1226828501 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:04:05.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.301+0000 7fb2a5400640 1 --2- 172.21.15.28:0/1226828501 >> v2:172.21.15.81:3300/0 conn(0x7fb2a00a6160 0x7fb2a0155180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:05.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.301+0000 7fb2a4a00640 1 -- 172.21.15.28:0/1226828501 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 723504390 0 0) 0x7fb2a0155760 con 0x7fb2a01521c0 2024-02-21T23:04:05.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.301+0000 7fb2a4a00640 1 -- 172.21.15.28:0/1226828501 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb288003520 con 0x7fb2a01521c0 2024-02-21T23:04:05.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.302+0000 7fb2a4a00640 1 -- 172.21.15.28:0/1226828501 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 36889612 0 0) 0x7fb288003520 con 0x7fb2a01521c0 2024-02-21T23:04:05.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.302+0000 7fb2a4a00640 1 -- 172.21.15.28:0/1226828501 >> v2:172.21.15.81:3300/0 conn(0x7fb2a00a6160 msgr2=0x7fb2a0155180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:05.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.302+0000 7fb2a4a00640 1 --2- 172.21.15.28:0/1226828501 >> v2:172.21.15.81:3300/0 conn(0x7fb2a00a6160 0x7fb2a0155180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:05.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.302+0000 7fb2a4a00640 1 -- 172.21.15.28:0/1226828501 >> v1:172.21.15.81:6789/0 conn(0x7fb2a014f0b0 legacy=0x7fb2a00c2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:05.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.302+0000 7fb2a4a00640 1 -- 172.21.15.28:0/1226828501 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2a0155d20 con 0x7fb2a01521c0 2024-02-21T23:04:05.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.302+0000 7fb2a4a00640 1 -- 172.21.15.28:0/1226828501 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fb29c0019f0 con 0x7fb2a01521c0 2024-02-21T23:04:05.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.302+0000 7fb2a4a00640 1 -- 172.21.15.28:0/1226828501 >> v1:172.21.15.28:6789/0 conn(0x7fb2a01521c0 legacy=0x7fb2a015c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:05.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.302+0000 7fb2a4a00640 1 --2- 172.21.15.28:0/1226828501 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb288003a00 0x7fb288003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:05.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.302+0000 7fb2a4a00640 1 --2- 172.21.15.28:0/1226828501 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb288004800 0x7fb288004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:05.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.302+0000 7fb2a4a00640 1 --2- 172.21.15.28:0/1226828501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2880051c0 0x7fb28800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:05.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.302+0000 7fb2a4a00640 1 -- 172.21.15.28:0/1226828501 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2a0151d30 con 0x7fb2880051c0 2024-02-21T23:04:05.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.302+0000 7fb2a4a00640 1 -- 172.21.15.28:0/1226828501 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb28800c0c0 con 0x7fb288004800 2024-02-21T23:04:05.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.302+0000 7fb2a4a00640 1 -- 172.21.15.28:0/1226828501 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb28800c2c0 con 0x7fb288003a00 2024-02-21T23:04:05.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.302+0000 7fb2a6800640 1 --2- 172.21.15.28:0/1226828501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2880051c0 0x7fb28800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:05.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.303+0000 7fb2a5400640 1 --2- 172.21.15.28:0/1226828501 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb288004800 0x7fb288004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:05.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.303+0000 7fb2a5e00640 1 --2- 172.21.15.28:0/1226828501 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb288003a00 0x7fb288003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:05.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.303+0000 7fb2a6800640 1 -- 172.21.15.28:0/1226828501 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb288003a00 msgr2=0x7fb288003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:05.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.303+0000 7fb2a6800640 1 --2- 172.21.15.28:0/1226828501 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb288003a00 0x7fb288003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:05.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.303+0000 7fb2a6800640 1 -- 172.21.15.28:0/1226828501 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb288004800 msgr2=0x7fb288004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:05.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.303+0000 7fb2a6800640 1 --2- 172.21.15.28:0/1226828501 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb288004800 0x7fb288004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:05.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.303+0000 7fb2a6800640 1 -- 172.21.15.28:0/1226828501 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb288003520 con 0x7fb2880051c0 2024-02-21T23:04:05.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.304+0000 7fb2a6800640 1 -- 172.21.15.28:0/1226828501 reap_dead start 2024-02-21T23:04:05.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.304+0000 7fb2a6800640 1 --2- 172.21.15.28:0/1226828501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2880051c0 0x7fb28800b9c0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fb29c00a770 tx=0x7fb288005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=845e47ba81f77ebb server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:05.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.305+0000 7fb2a4a00640 1 -- 172.21.15.28:0/1226828501 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb29c00a390 con 0x7fb2880051c0 2024-02-21T23:04:05.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.305+0000 7fb2a4a00640 1 -- 172.21.15.28:0/1226828501 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb29c034650 con 0x7fb2880051c0 2024-02-21T23:04:05.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.305+0000 7fb2a4a00640 1 -- 172.21.15.28:0/1226828501 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb29c034a40 con 0x7fb2880051c0 2024-02-21T23:04:05.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.305+0000 7fb2a7200640 1 -- 172.21.15.28:0/1226828501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2880051c0 msgr2=0x7fb28800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:05.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.305+0000 7fb2a7200640 1 --2- 172.21.15.28:0/1226828501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2880051c0 0x7fb28800b9c0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fb29c00a770 tx=0x7fb288005830 comp rx=0 tx=0).stop 2024-02-21T23:04:05.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.306+0000 7fb2a7200640 1 -- 172.21.15.28:0/1226828501 shutdown_connections 2024-02-21T23:04:05.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.306+0000 7fb2a7200640 1 --2- 172.21.15.28:0/1226828501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2880051c0 0x7fb28800b9c0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:05.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.306+0000 7fb2a7200640 1 -- 172.21.15.28:0/1226828501 >> 172.21.15.28:0/1226828501 conn(0x7fb2a006b630 msgr2=0x7fb2a00c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:05.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.306+0000 7fb2a7200640 1 -- 172.21.15.28:0/1226828501 shutdown_connections 2024-02-21T23:04:05.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.306+0000 7fb2a7200640 1 -- 172.21.15.28:0/1226828501 wait complete. 2024-02-21T23:04:05.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.308+0000 7fb2a7200640 1 Processor -- start 2024-02-21T23:04:05.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.308+0000 7fb2a7200640 1 -- start start 2024-02-21T23:04:05.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.309+0000 7fb2a7200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2880051c0 0x7fb2a01ec2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:05.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.309+0000 7fb2a7200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb2a01ec820 0x7fb2a00c2610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:05.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.309+0000 7fb2a7200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2a00c2b50 0x7fb2a01449b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:05.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.309+0000 7fb2a7200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2a00a7f70 con 0x7fb2880051c0 2024-02-21T23:04:05.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.309+0000 7fb2a7200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2a00a68d0 con 0x7fb2a00c2b50 2024-02-21T23:04:05.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.309+0000 7fb2a7200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2a00a80f0 con 0x7fb2a01ec820 2024-02-21T23:04:05.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.309+0000 7fb2a5e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2880051c0 0x7fb2a01ec2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:05.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.309+0000 7fb2a5e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2880051c0 0x7fb2a01ec2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56200/0 (socket says 172.21.15.28:56200) 2024-02-21T23:04:05.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.309+0000 7fb2a5e00640 1 -- 172.21.15.28:0/44129346 learned_addr learned my addr 172.21.15.28:0/44129346 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:05.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.310+0000 7fb2a5400640 1 --2- 172.21.15.28:0/44129346 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb2a01ec820 0x7fb2a00c2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:05.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.310+0000 7fb2a6800640 1 --2- 172.21.15.28:0/44129346 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2a00c2b50 0x7fb2a01449b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:05.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.310+0000 7fb2a5e00640 1 -- 172.21.15.28:0/44129346 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb2a01ec820 msgr2=0x7fb2a00c2610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:05.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.310+0000 7fb2a5e00640 1 --2- 172.21.15.28:0/44129346 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb2a01ec820 0x7fb2a00c2610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:05.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.310+0000 7fb2a5e00640 1 -- 172.21.15.28:0/44129346 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2a00c2b50 msgr2=0x7fb2a01449b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:05.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.310+0000 7fb2a5e00640 1 --2- 172.21.15.28:0/44129346 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2a00c2b50 0x7fb2a01449b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:05.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.310+0000 7fb2a5e00640 1 -- 172.21.15.28:0/44129346 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2a00c1e50 con 0x7fb2880051c0 2024-02-21T23:04:05.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.310+0000 7fb2a5e00640 1 --2- 172.21.15.28:0/44129346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2880051c0 0x7fb2a01ec2e0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fb29400e4e0 tx=0x7fb29400e9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:05.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.310+0000 7fb29a000640 1 -- 172.21.15.28:0/44129346 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2940052e0 con 0x7fb2880051c0 2024-02-21T23:04:05.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.311+0000 7fb29a000640 1 -- 172.21.15.28:0/44129346 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb294018070 con 0x7fb2880051c0 2024-02-21T23:04:05.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.311+0000 7fb29a000640 1 -- 172.21.15.28:0/44129346 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb294003830 con 0x7fb2880051c0 2024-02-21T23:04:05.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.311+0000 7fb2a7200640 1 -- 172.21.15.28:0/44129346 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2a00a9100 con 0x7fb2880051c0 2024-02-21T23:04:05.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.311+0000 7fb2a7200640 1 -- 172.21.15.28:0/44129346 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2a00a9a60 con 0x7fb2880051c0 2024-02-21T23:04:05.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.312+0000 7fb2a7200640 1 -- 172.21.15.28:0/44129346 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2a0073540 con 0x7fb2880051c0 2024-02-21T23:04:05.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.313+0000 7fb29a000640 1 -- 172.21.15.28:0/44129346 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb2940039d0 con 0x7fb2880051c0 2024-02-21T23:04:05.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.314+0000 7fb29a000640 1 --2- 172.21.15.28:0/44129346 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb28406b4d0 0x7fb28406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:05.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.314+0000 7fb29a000640 1 -- 172.21.15.28:0/44129346 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fb294089c30 con 0x7fb2880051c0 2024-02-21T23:04:05.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.315+0000 7fb2a5400640 1 --2- 172.21.15.28:0/44129346 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb28406b4d0 0x7fb28406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:05.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.316+0000 7fb2a5400640 1 --2- 172.21.15.28:0/44129346 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb28406b4d0 0x7fb28406d990 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fb2a0151680 tx=0x7fb290008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:05.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.319+0000 7fb29a000640 1 -- 172.21.15.28:0/44129346 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb2a0073540 con 0x7fb2880051c0 2024-02-21T23:04:05.312 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-21T23:04:05.318 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-21T23:04:05.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.531+0000 7fb2a7200640 1 -- 172.21.15.28:0/44129346 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb2a004f290 con 0x7fb2880051c0 2024-02-21T23:04:05.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.532+0000 7fb29a000640 1 -- 172.21.15.28:0/44129346 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+13104 (secure 0 0 0) 0x7fb294058c60 con 0x7fb2880051c0 2024-02-21T23:04:05.522 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:04:05.522 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":34,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:04:01.360152+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:04:05.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.535+0000 7fb2a7200640 1 -- 172.21.15.28:0/44129346 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb28406b4d0 msgr2=0x7fb28406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:05.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.535+0000 7fb2a7200640 1 --2- 172.21.15.28:0/44129346 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb28406b4d0 0x7fb28406d990 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fb2a0151680 tx=0x7fb290008040 comp rx=0 tx=0).stop 2024-02-21T23:04:05.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.535+0000 7fb2a7200640 1 -- 172.21.15.28:0/44129346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2880051c0 msgr2=0x7fb2a01ec2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:05.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.535+0000 7fb2a7200640 1 --2- 172.21.15.28:0/44129346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2880051c0 0x7fb2a01ec2e0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fb29400e4e0 tx=0x7fb29400e9a0 comp rx=0 tx=0).stop 2024-02-21T23:04:05.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.535+0000 7fb2a7200640 1 -- 172.21.15.28:0/44129346 shutdown_connections 2024-02-21T23:04:05.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.535+0000 7fb2a7200640 1 --2- 172.21.15.28:0/44129346 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb28406b4d0 0x7fb28406d990 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:05.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.535+0000 7fb2a7200640 1 --2- 172.21.15.28:0/44129346 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb2a01ec820 0x7fb2a00c2610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:05.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.535+0000 7fb2a7200640 1 --2- 172.21.15.28:0/44129346 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2a00c2b50 0x7fb2a01449b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:05.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.535+0000 7fb2a7200640 1 --2- 172.21.15.28:0/44129346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2880051c0 0x7fb2a01ec2e0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:05.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.535+0000 7fb2a7200640 1 -- 172.21.15.28:0/44129346 >> 172.21.15.28:0/44129346 conn(0x7fb2a006b630 msgr2=0x7fb2a01505a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:05.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.536+0000 7fb2a7200640 1 -- 172.21.15.28:0/44129346 shutdown_connections 2024-02-21T23:04:05.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.536+0000 7fb2a7200640 1 -- 172.21.15.28:0/44129346 wait complete. 2024-02-21T23:04:05.542 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-21T23:02:30.728329+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-21T23:04:05.543 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-21T23:04:05.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.777+0000 7f7337800640 1 Processor -- start 2024-02-21T23:04:05.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.777+0000 7f7337800640 1 -- start start 2024-02-21T23:04:05.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.778+0000 7f7337800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f73300c5cc0 0x7f73300c6c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:05.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.779+0000 7f7337800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73300c3b70 con 0x7f73300c3030 2024-02-21T23:04:05.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.779+0000 7f7337800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7330159580 con 0x7f73300c7590 2024-02-21T23:04:05.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.779+0000 7f7337800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73300c6230 con 0x7f73300c5cc0 2024-02-21T23:04:05.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.779+0000 7f7335800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f73300c3030 0x7f73300c3460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:56644/0 (socket says 172.21.15.28:56644) 2024-02-21T23:04:05.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.779+0000 7f7335800640 1 -- 172.21.15.28:0/3364494541 learned_addr learned my addr 172.21.15.28:0/3364494541 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:04:05.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.779+0000 7f732fe00640 1 --2- 172.21.15.28:0/3364494541 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f73300c5cc0 0x7f73300c6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:05.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.780+0000 7f732f400640 1 -- 172.21.15.28:0/3364494541 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4269325432 0 0) 0x7f73300c3b70 con 0x7f73300c3030 2024-02-21T23:04:05.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.780+0000 7f732f400640 1 -- 172.21.15.28:0/3364494541 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7314003520 con 0x7f73300c3030 2024-02-21T23:04:05.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.780+0000 7f732f400640 1 -- 172.21.15.28:0/3364494541 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2434401076 0 0) 0x7f7330159580 con 0x7f73300c7590 2024-02-21T23:04:05.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.780+0000 7f732f400640 1 -- 172.21.15.28:0/3364494541 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73300c3b70 con 0x7f73300c7590 2024-02-21T23:04:05.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.780+0000 7f732f400640 1 -- 172.21.15.28:0/3364494541 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1126935186 0 0) 0x7f7314003520 con 0x7f73300c3030 2024-02-21T23:04:05.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.780+0000 7f732f400640 1 -- 172.21.15.28:0/3364494541 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f73300c5cc0 msgr2=0x7f73300c6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:05.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.780+0000 7f732f400640 1 --2- 172.21.15.28:0/3364494541 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f73300c5cc0 0x7f73300c6c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:05.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.780+0000 7f732f400640 1 -- 172.21.15.28:0/3364494541 >> v1:172.21.15.81:6789/0 conn(0x7f73300c7590 legacy=0x7f73301524e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:05.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.780+0000 7f732f400640 1 -- 172.21.15.28:0/3364494541 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73301587c0 con 0x7f73300c3030 2024-02-21T23:04:05.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.780+0000 7f732f400640 1 -- 172.21.15.28:0/3364494541 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f7328001670 con 0x7f73300c3030 2024-02-21T23:04:05.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.780+0000 7f732f400640 1 -- 172.21.15.28:0/3364494541 >> v1:172.21.15.28:6789/0 conn(0x7f73300c3030 legacy=0x7f73300c3460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:05.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.780+0000 7f732f400640 1 --2- 172.21.15.28:0/3364494541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7314003b70 0x7f7314003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:05.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.780+0000 7f732f400640 1 --2- 172.21.15.28:0/3364494541 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7314004800 0x7f7314004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:05.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.781+0000 7f732f400640 1 --2- 172.21.15.28:0/3364494541 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f73140051c0 0x7f731400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:05.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.781+0000 7f732f400640 1 -- 172.21.15.28:0/3364494541 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73300c6230 con 0x7f7314003b70 2024-02-21T23:04:05.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.781+0000 7f732f400640 1 -- 172.21.15.28:0/3364494541 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f731400c090 con 0x7f7314004800 2024-02-21T23:04:05.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.781+0000 7f732f400640 1 -- 172.21.15.28:0/3364494541 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f731400c290 con 0x7f73140051c0 2024-02-21T23:04:05.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.781+0000 7f732fe00640 1 --2- 172.21.15.28:0/3364494541 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f73300c5cc0 0x7f73300c6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-21T23:04:05.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.781+0000 7f7334e00640 1 --2- 172.21.15.28:0/3364494541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7314003b70 0x7f7314003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:05.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.781+0000 7f7335800640 1 --2- 172.21.15.28:0/3364494541 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f73140051c0 0x7f731400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:05.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.781+0000 7f7334e00640 1 -- 172.21.15.28:0/3364494541 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f73140051c0 msgr2=0x7f731400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:05.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.781+0000 7f732fe00640 1 --2- 172.21.15.28:0/3364494541 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7314004800 0x7f7314004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:05.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.781+0000 7f7334e00640 1 --2- 172.21.15.28:0/3364494541 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f73140051c0 0x7f731400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:05.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.781+0000 7f7334e00640 1 -- 172.21.15.28:0/3364494541 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7314004800 msgr2=0x7f7314004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:05.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.781+0000 7f7334e00640 1 --2- 172.21.15.28:0/3364494541 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7314004800 0x7f7314004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:05.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.781+0000 7f7334e00640 1 -- 172.21.15.28:0/3364494541 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7314003520 con 0x7f7314003b70 2024-02-21T23:04:05.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.781+0000 7f7335800640 1 --2- 172.21.15.28:0/3364494541 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f73140051c0 0x7f731400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:04:05.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.781+0000 7f732fe00640 1 --2- 172.21.15.28:0/3364494541 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7314004800 0x7f7314004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:04:05.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.783+0000 7f7334e00640 1 --2- 172.21.15.28:0/3364494541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7314003b70 0x7f7314003f70 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f73180018c0 tx=0x7f7314005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=206b0775a3c1ed04 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:05.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.783+0000 7f732f400640 1 -- 172.21.15.28:0/3364494541 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7318037070 con 0x7f7314003b70 2024-02-21T23:04:05.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.783+0000 7f732f400640 1 -- 172.21.15.28:0/3364494541 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73180050d0 con 0x7f7314003b70 2024-02-21T23:04:05.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.783+0000 7f732f400640 1 -- 172.21.15.28:0/3364494541 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73180014e0 con 0x7f7314003b70 2024-02-21T23:04:05.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.783+0000 7f7337800640 1 -- 172.21.15.28:0/3364494541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7314003b70 msgr2=0x7f7314003f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:05.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.783+0000 7f7337800640 1 --2- 172.21.15.28:0/3364494541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7314003b70 0x7f7314003f70 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f73180018c0 tx=0x7f7314005830 comp rx=0 tx=0).stop 2024-02-21T23:04:05.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.784+0000 7f7335800640 1 -- 172.21.15.28:0/3364494541 reap_dead start 2024-02-21T23:04:05.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.784+0000 7f7337800640 1 -- 172.21.15.28:0/3364494541 shutdown_connections 2024-02-21T23:04:05.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.784+0000 7f7337800640 1 -- 172.21.15.28:0/3364494541 >> 172.21.15.28:0/3364494541 conn(0x7f733008afe0 msgr2=0x7f73300725a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:05.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.784+0000 7f7337800640 1 -- 172.21.15.28:0/3364494541 shutdown_connections 2024-02-21T23:04:05.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.784+0000 7f7337800640 1 -- 172.21.15.28:0/3364494541 wait complete. 2024-02-21T23:04:05.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.785+0000 7f7337800640 1 Processor -- start 2024-02-21T23:04:05.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.786+0000 7f7337800640 1 -- start start 2024-02-21T23:04:05.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.787+0000 7f7337800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7314003b70 0x7f73301df460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:05.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.787+0000 7f7337800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73301e0880 0x7f733014a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:05.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.787+0000 7f7337800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f733014aa30 0x7f73300c0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:05.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.787+0000 7f7337800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7330159060 con 0x7f733014aa30 2024-02-21T23:04:05.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.787+0000 7f7337800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7330158d80 con 0x7f73301e0880 2024-02-21T23:04:05.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.787+0000 7f7337800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73300c6720 con 0x7f7314003b70 2024-02-21T23:04:05.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.787+0000 7f7335800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f733014aa30 0x7f73300c0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:05.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.787+0000 7f7335800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f733014aa30 0x7f73300c0ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56212/0 (socket says 172.21.15.28:56212) 2024-02-21T23:04:05.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.787+0000 7f7335800640 1 -- 172.21.15.28:0/781849161 learned_addr learned my addr 172.21.15.28:0/781849161 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:05.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.787+0000 7f732fe00640 1 --2- 172.21.15.28:0/781849161 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73301e0880 0x7f733014a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:05.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.787+0000 7f7334e00640 1 --2- 172.21.15.28:0/781849161 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7314003b70 0x7f73301df460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:05.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.787+0000 7f7335800640 1 -- 172.21.15.28:0/781849161 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7314003b70 msgr2=0x7f73301df460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:05.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.787+0000 7f7335800640 1 --2- 172.21.15.28:0/781849161 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7314003b70 0x7f73301df460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:05.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.787+0000 7f7335800640 1 -- 172.21.15.28:0/781849161 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73301e0880 msgr2=0x7f733014a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:05.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.787+0000 7f7335800640 1 --2- 172.21.15.28:0/781849161 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73301e0880 0x7f733014a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:05.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.787+0000 7f7335800640 1 -- 172.21.15.28:0/781849161 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f733014b290 con 0x7f733014aa30 2024-02-21T23:04:05.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.787+0000 7f732fe00640 1 --2- 172.21.15.28:0/781849161 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73301e0880 0x7f733014a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:04:05.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.787+0000 7f7334e00640 1 --2- 172.21.15.28:0/781849161 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7314003b70 0x7f73301df460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:04:05.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.788+0000 7f7335800640 1 --2- 172.21.15.28:0/781849161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f733014aa30 0x7f73300c0ed0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f73300c8390 tx=0x7f732800d890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:05.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.788+0000 7f732cc00640 1 -- 172.21.15.28:0/781849161 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73280074e0 con 0x7f733014aa30 2024-02-21T23:04:05.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.788+0000 7f732cc00640 1 -- 172.21.15.28:0/781849161 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73280047e0 con 0x7f733014aa30 2024-02-21T23:04:05.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.788+0000 7f732cc00640 1 -- 172.21.15.28:0/781849161 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7328004bb0 con 0x7f733014aa30 2024-02-21T23:04:05.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.788+0000 7f7337800640 1 -- 172.21.15.28:0/781849161 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73300c04c0 con 0x7f733014aa30 2024-02-21T23:04:05.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.788+0000 7f7337800640 1 -- 172.21.15.28:0/781849161 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7330158440 con 0x7f733014aa30 2024-02-21T23:04:05.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.789+0000 7f7337800640 1 -- 172.21.15.28:0/781849161 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7330067f30 con 0x7f733014aa30 2024-02-21T23:04:05.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.790+0000 7f732cc00640 1 -- 172.21.15.28:0/781849161 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f73280133d0 con 0x7f733014aa30 2024-02-21T23:04:05.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.792+0000 7f732cc00640 1 --2- 172.21.15.28:0/781849161 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f730c06b4d0 0x7f730c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:05.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.792+0000 7f732cc00640 1 -- 172.21.15.28:0/781849161 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f7328089810 con 0x7f733014aa30 2024-02-21T23:04:05.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.792+0000 7f7334e00640 1 --2- 172.21.15.28:0/781849161 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f730c06b4d0 0x7f730c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:05.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.793+0000 7f7334e00640 1 --2- 172.21.15.28:0/781849161 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f730c06b4d0 0x7f730c06d990 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f73180018c0 tx=0x7f7318009030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:05.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.796+0000 7f732cc00640 1 -- 172.21.15.28:0/781849161 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7328058840 con 0x7f733014aa30 2024-02-21T23:04:05.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.994+0000 7f7337800640 1 -- 172.21.15.28:0/781849161 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f733014b460 con 0x7f733014aa30 2024-02-21T23:04:05.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.995+0000 7f732cc00640 1 -- 172.21.15.28:0/781849161 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v34) v1 ==== 93+0+10 (secure 0 0 0) 0x7f732805e070 con 0x7f733014aa30 2024-02-21T23:04:05.984 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-02-21T23:04:05.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.998+0000 7f7337800640 1 -- 172.21.15.28:0/781849161 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f730c06b4d0 msgr2=0x7f730c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:05.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.998+0000 7f7337800640 1 --2- 172.21.15.28:0/781849161 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f730c06b4d0 0x7f730c06d990 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f73180018c0 tx=0x7f7318009030 comp rx=0 tx=0).stop 2024-02-21T23:04:05.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.998+0000 7f7337800640 1 -- 172.21.15.28:0/781849161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f733014aa30 msgr2=0x7f73300c0ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:05.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.998+0000 7f7337800640 1 --2- 172.21.15.28:0/781849161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f733014aa30 0x7f73300c0ed0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f73300c8390 tx=0x7f732800d890 comp rx=0 tx=0).stop 2024-02-21T23:04:05.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.998+0000 7f7337800640 1 -- 172.21.15.28:0/781849161 shutdown_connections 2024-02-21T23:04:05.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.998+0000 7f7337800640 1 --2- 172.21.15.28:0/781849161 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f730c06b4d0 0x7f730c06d990 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:05.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.998+0000 7f7337800640 1 --2- 172.21.15.28:0/781849161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f733014aa30 0x7f73300c0ed0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:05.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.998+0000 7f7337800640 1 --2- 172.21.15.28:0/781849161 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73301e0880 0x7f733014a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:05.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.998+0000 7f7337800640 1 --2- 172.21.15.28:0/781849161 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7314003b70 0x7f73301df460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:05.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.998+0000 7f7337800640 1 -- 172.21.15.28:0/781849161 >> 172.21.15.28:0/781849161 conn(0x7f733008afe0 msgr2=0x7f7330151760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:05.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.998+0000 7f7337800640 1 -- 172.21.15.28:0/781849161 shutdown_connections 2024-02-21T23:04:05.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:05.999+0000 7f7337800640 1 -- 172.21.15.28:0/781849161 wait complete. 2024-02-21T23:04:06.008 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-02-21T23:04:06.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.231+0000 7fa5d5200640 1 Processor -- start 2024-02-21T23:04:06.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.231+0000 7fa5d5200640 1 -- start start 2024-02-21T23:04:06.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.232+0000 7fa5d5200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fa5d00b0130 0x7fa5d00c8740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:06.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.232+0000 7fa5d5200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa5d0158c20 0x7fa5d00c5670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:06.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.232+0000 7fa5d5200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5d01550b0 con 0x7fa5d00c51c0 2024-02-21T23:04:06.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.232+0000 7fa5d5200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5d0152a40 con 0x7fa5d0158c20 2024-02-21T23:04:06.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.232+0000 7fa5d5200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5d0155350 con 0x7fa5d00b0130 2024-02-21T23:04:06.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.233+0000 7fa5cfe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa5d0158c20 0x7fa5d00c5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:06.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.233+0000 7fa5cfe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa5d0158c20 0x7fa5d00c5670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56224/0 (socket says 172.21.15.28:56224) 2024-02-21T23:04:06.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.233+0000 7fa5cfe00640 1 -- 172.21.15.28:0/2412073144 learned_addr learned my addr 172.21.15.28:0/2412073144 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:06.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.233+0000 7fa5cf400640 1 --2- 172.21.15.28:0/2412073144 >> v2:172.21.15.81:3300/0 conn(0x7fa5d00b0130 0x7fa5d00c8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:06.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.233+0000 7fa5cfe00640 1 -- 172.21.15.28:0/2412073144 >> v2:172.21.15.81:3300/0 conn(0x7fa5d00b0130 msgr2=0x7fa5d00c8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:06.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.233+0000 7fa5cfe00640 1 --2- 172.21.15.28:0/2412073144 >> v2:172.21.15.81:3300/0 conn(0x7fa5d00b0130 0x7fa5d00c8740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:06.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.233+0000 7fa5cfe00640 1 -- 172.21.15.28:0/2412073144 >> v1:172.21.15.81:6789/0 conn(0x7fa5d00c51c0 legacy=0x7fa5d0158510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:06.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.233+0000 7fa5cfe00640 1 -- 172.21.15.28:0/2412073144 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5d0155630 con 0x7fa5d0158c20 2024-02-21T23:04:06.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.234+0000 7fa5cfe00640 1 --2- 172.21.15.28:0/2412073144 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5d0158c20 0x7fa5d00c5670 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fa5d00c7f50 tx=0x7fa5c802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a5db2db519dcbb6 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:06.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.235+0000 7fa5ce000640 1 -- 172.21.15.28:0/2412073144 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5c802c9f0 con 0x7fa5d0158c20 2024-02-21T23:04:06.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.235+0000 7fa5ce000640 1 -- 172.21.15.28:0/2412073144 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5c802cb90 con 0x7fa5d0158c20 2024-02-21T23:04:06.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.235+0000 7fa5ce000640 1 -- 172.21.15.28:0/2412073144 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5c80343f0 con 0x7fa5d0158c20 2024-02-21T23:04:06.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.235+0000 7fa5d5200640 1 -- 172.21.15.28:0/2412073144 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5d0158c20 msgr2=0x7fa5d00c5670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:06.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.235+0000 7fa5d5200640 1 --2- 172.21.15.28:0/2412073144 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5d0158c20 0x7fa5d00c5670 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fa5d00c7f50 tx=0x7fa5c802be50 comp rx=0 tx=0).stop 2024-02-21T23:04:06.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.236+0000 7fa5d5200640 1 -- 172.21.15.28:0/2412073144 shutdown_connections 2024-02-21T23:04:06.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.236+0000 7fa5d5200640 1 --2- 172.21.15.28:0/2412073144 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5d0158c20 0x7fa5d00c5670 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:06.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.236+0000 7fa5d5200640 1 --2- 172.21.15.28:0/2412073144 >> v2:172.21.15.81:3300/0 conn(0x7fa5d00b0130 0x7fa5d00c8740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:06.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.236+0000 7fa5d5200640 1 -- 172.21.15.28:0/2412073144 >> 172.21.15.28:0/2412073144 conn(0x7fa5d00ac370 msgr2=0x7fa5d00ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:06.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.236+0000 7fa5d5200640 1 -- 172.21.15.28:0/2412073144 shutdown_connections 2024-02-21T23:04:06.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.236+0000 7fa5d5200640 1 -- 172.21.15.28:0/2412073144 wait complete. 2024-02-21T23:04:06.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.237+0000 7fa5d5200640 1 Processor -- start 2024-02-21T23:04:06.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.237+0000 7fa5d5200640 1 -- start start 2024-02-21T23:04:06.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.238+0000 7fa5d5200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5d00b0130 0x7fa5d0151ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:06.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.238+0000 7fa5d5200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa5d00c51c0 0x7fa5d00be300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:06.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.238+0000 7fa5d5200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa5d0158c20 0x7fa5d00beef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:06.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.238+0000 7fa5d5200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d00c8dd0 con 0x7fa5d00b0130 2024-02-21T23:04:06.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.238+0000 7fa5d5200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d0155e30 con 0x7fa5d0158c20 2024-02-21T23:04:06.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.238+0000 7fa5d5200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d00be060 con 0x7fa5d00c51c0 2024-02-21T23:04:06.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.239+0000 7fa5cf400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5d00b0130 0x7fa5d0151ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:06.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.239+0000 7fa5cf400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5d00b0130 0x7fa5d0151ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56240/0 (socket says 172.21.15.28:56240) 2024-02-21T23:04:06.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.239+0000 7fa5cf400640 1 -- 172.21.15.28:0/1709454856 learned_addr learned my addr 172.21.15.28:0/1709454856 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:06.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.239+0000 7fa5cf400640 1 -- 172.21.15.28:0/1709454856 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa5d00c51c0 msgr2=0x7fa5d00be300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:06.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.239+0000 7fa5cea00640 1 --2- 172.21.15.28:0/1709454856 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa5d00c51c0 0x7fa5d00be300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:06.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.239+0000 7fa5cf400640 1 --2- 172.21.15.28:0/1709454856 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa5d00c51c0 0x7fa5d00be300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:06.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.239+0000 7fa5cf400640 1 -- 172.21.15.28:0/1709454856 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa5d0158c20 msgr2=0x7fa5d00beef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:06.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.239+0000 7fa5cf400640 1 --2- 172.21.15.28:0/1709454856 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa5d0158c20 0x7fa5d00beef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:06.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.239+0000 7fa5cf400640 1 -- 172.21.15.28:0/1709454856 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5d00c39b0 con 0x7fa5d00b0130 2024-02-21T23:04:06.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.239+0000 7fa5cf400640 1 --2- 172.21.15.28:0/1709454856 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5d00b0130 0x7fa5d0151ca0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fa5c000d800 tx=0x7fa5c000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:06.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.240+0000 7fa5c7400640 1 -- 172.21.15.28:0/1709454856 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5c000b6c0 con 0x7fa5d00b0130 2024-02-21T23:04:06.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.240+0000 7fa5c7400640 1 -- 172.21.15.28:0/1709454856 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5c0019070 con 0x7fa5d00b0130 2024-02-21T23:04:06.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.240+0000 7fa5c7400640 1 -- 172.21.15.28:0/1709454856 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5c000ba80 con 0x7fa5d00b0130 2024-02-21T23:04:06.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.240+0000 7fa5d5200640 1 -- 172.21.15.28:0/1709454856 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5d01eb3c0 con 0x7fa5d00b0130 2024-02-21T23:04:06.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.240+0000 7fa5d5200640 1 -- 172.21.15.28:0/1709454856 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5d00c36d0 con 0x7fa5d00b0130 2024-02-21T23:04:06.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.241+0000 7fa5d5200640 1 -- 172.21.15.28:0/1709454856 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5d004f290 con 0x7fa5d00b0130 2024-02-21T23:04:06.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.242+0000 7fa5c7400640 1 -- 172.21.15.28:0/1709454856 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa5c000bc20 con 0x7fa5d00b0130 2024-02-21T23:04:06.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.243+0000 7fa5c7400640 1 --2- 172.21.15.28:0/1709454856 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa5b006b4d0 0x7fa5b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:06.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.243+0000 7fa5c7400640 1 -- 172.21.15.28:0/1709454856 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fa5c008a5c0 con 0x7fa5d00b0130 2024-02-21T23:04:06.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.244+0000 7fa5cea00640 1 --2- 172.21.15.28:0/1709454856 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa5b006b4d0 0x7fa5b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:06.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.245+0000 7fa5cea00640 1 --2- 172.21.15.28:0/1709454856 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa5b006b4d0 0x7fa5b006d990 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa5d00c7c50 tx=0x7fa5bc009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:06.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.247+0000 7fa5c7400640 1 -- 172.21.15.28:0/1709454856 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fa5c0059560 con 0x7fa5d00b0130 2024-02-21T23:04:06.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.448+0000 7fa5d5200640 1 -- 172.21.15.28:0/1709454856 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fa5d00c0e60 con 0x7fa5d00b0130 2024-02-21T23:04:06.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.449+0000 7fa5c7400640 1 -- 172.21.15.28:0/1709454856 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v34) v1 ==== 84+0+65 (secure 0 0 0) 0x7fa5c0058e60 con 0x7fa5d00b0130 2024-02-21T23:04:06.438 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:04:06.438 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-02-21T23:04:06.438 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:04:06.438 INFO:teuthology.orchestra.run.smithi028.stdout: "poolnum": 1, 2024-02-21T23:04:06.438 INFO:teuthology.orchestra.run.smithi028.stdout: "poolname": ".mgr" 2024-02-21T23:04:06.438 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:04:06.438 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-02-21T23:04:06.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.452+0000 7fa5d5200640 1 -- 172.21.15.28:0/1709454856 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa5b006b4d0 msgr2=0x7fa5b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:06.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.452+0000 7fa5d5200640 1 --2- 172.21.15.28:0/1709454856 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa5b006b4d0 0x7fa5b006d990 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa5d00c7c50 tx=0x7fa5bc009040 comp rx=0 tx=0).stop 2024-02-21T23:04:06.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.452+0000 7fa5d5200640 1 -- 172.21.15.28:0/1709454856 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5d00b0130 msgr2=0x7fa5d0151ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:06.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.452+0000 7fa5d5200640 1 --2- 172.21.15.28:0/1709454856 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5d00b0130 0x7fa5d0151ca0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fa5c000d800 tx=0x7fa5c000dcc0 comp rx=0 tx=0).stop 2024-02-21T23:04:06.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.452+0000 7fa5d5200640 1 -- 172.21.15.28:0/1709454856 shutdown_connections 2024-02-21T23:04:06.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.452+0000 7fa5d5200640 1 --2- 172.21.15.28:0/1709454856 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa5b006b4d0 0x7fa5b006d990 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:06.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.452+0000 7fa5d5200640 1 --2- 172.21.15.28:0/1709454856 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa5d00c51c0 0x7fa5d00be300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:06.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.452+0000 7fa5d5200640 1 --2- 172.21.15.28:0/1709454856 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa5d0158c20 0x7fa5d00beef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:06.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.452+0000 7fa5d5200640 1 --2- 172.21.15.28:0/1709454856 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5d00b0130 0x7fa5d0151ca0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:06.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.452+0000 7fa5d5200640 1 -- 172.21.15.28:0/1709454856 >> 172.21.15.28:0/1709454856 conn(0x7fa5d00ac370 msgr2=0x7fa5d00ab550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:06.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.452+0000 7fa5d5200640 1 -- 172.21.15.28:0/1709454856 shutdown_connections 2024-02-21T23:04:06.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.452+0000 7fa5d5200640 1 -- 172.21.15.28:0/1709454856 wait complete. 2024-02-21T23:04:06.460 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-02-21T23:04:06.460 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-02-21T23:04:06.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.701+0000 7f1f5fe00640 1 Processor -- start 2024-02-21T23:04:06.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.701+0000 7f1f5fe00640 1 -- start start 2024-02-21T23:04:06.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.703+0000 7f1f5fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1f600c1010 0x7f1f6006e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:06.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.703+0000 7f1f5fe00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1f6006e870 0x7f1f6015d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:06.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.703+0000 7f1f5fe00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f60152fd0 con 0x7f1f601501c0 2024-02-21T23:04:06.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.703+0000 7f1f5fe00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f601532b0 con 0x7f1f600c1010 2024-02-21T23:04:06.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.703+0000 7f1f5fe00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f60153590 con 0x7f1f6006e870 2024-02-21T23:04:06.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.703+0000 7f1f5e000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1f600c1010 0x7f1f6006e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:06.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.703+0000 7f1f5e000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1f600c1010 0x7f1f6006e330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56246/0 (socket says 172.21.15.28:56246) 2024-02-21T23:04:06.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.703+0000 7f1f5e000640 1 -- 172.21.15.28:0/3853714720 learned_addr learned my addr 172.21.15.28:0/3853714720 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:06.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.703+0000 7f1f5f400640 1 --2- 172.21.15.28:0/3853714720 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1f6006e870 0x7f1f6015d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:06.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.704+0000 7f1f5e000640 1 -- 172.21.15.28:0/3853714720 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1f6006e870 msgr2=0x7f1f6015d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:06.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.704+0000 7f1f5e000640 1 --2- 172.21.15.28:0/3853714720 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1f6006e870 0x7f1f6015d5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:06.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.704+0000 7f1f5e000640 1 -- 172.21.15.28:0/3853714720 >> v1:172.21.15.81:6789/0 conn(0x7f1f601501c0 legacy=0x7f1f600a96c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:06.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.704+0000 7f1f5e000640 1 -- 172.21.15.28:0/3853714720 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f60153870 con 0x7f1f600c1010 2024-02-21T23:04:06.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.705+0000 7f1f5e000640 1 --2- 172.21.15.28:0/3853714720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f600c1010 0x7f1f6006e330 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f1f4c009870 tx=0x7f1f4c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f87fc2d935b85d4 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:06.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.705+0000 7f1f5d600640 1 -- 172.21.15.28:0/3853714720 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f4c0317c0 con 0x7f1f600c1010 2024-02-21T23:04:06.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.705+0000 7f1f5d600640 1 -- 172.21.15.28:0/3853714720 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f4c037070 con 0x7f1f600c1010 2024-02-21T23:04:06.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.705+0000 7f1f5d600640 1 -- 172.21.15.28:0/3853714720 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f4c031b80 con 0x7f1f600c1010 2024-02-21T23:04:06.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.706+0000 7f1f5fe00640 1 -- 172.21.15.28:0/3853714720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f600c1010 msgr2=0x7f1f6006e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:06.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.706+0000 7f1f5fe00640 1 --2- 172.21.15.28:0/3853714720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f600c1010 0x7f1f6006e330 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f1f4c009870 tx=0x7f1f4c02b860 comp rx=0 tx=0).stop 2024-02-21T23:04:06.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.706+0000 7f1f5fe00640 1 -- 172.21.15.28:0/3853714720 shutdown_connections 2024-02-21T23:04:06.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.706+0000 7f1f5fe00640 1 --2- 172.21.15.28:0/3853714720 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1f6006e870 0x7f1f6015d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:06.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.706+0000 7f1f5fe00640 1 --2- 172.21.15.28:0/3853714720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f600c1010 0x7f1f6006e330 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:06.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.706+0000 7f1f5fe00640 1 -- 172.21.15.28:0/3853714720 >> 172.21.15.28:0/3853714720 conn(0x7f1f600c6610 msgr2=0x7f1f600c69f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:06.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.706+0000 7f1f5fe00640 1 -- 172.21.15.28:0/3853714720 shutdown_connections 2024-02-21T23:04:06.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.706+0000 7f1f5fe00640 1 -- 172.21.15.28:0/3853714720 wait complete. 2024-02-21T23:04:06.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.708+0000 7f1f5fe00640 1 Processor -- start 2024-02-21T23:04:06.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.708+0000 7f1f5fe00640 1 -- start start 2024-02-21T23:04:06.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.709+0000 7f1f5fe00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1f6006e870 0x7f1f600bfcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:06.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.709+0000 7f1f5fe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1f600c1010 0x7f1f6013ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:06.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.709+0000 7f1f5fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f601501c0 0x7f1f6013b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:06.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.709+0000 7f1f5fe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f600c7340 con 0x7f1f601501c0 2024-02-21T23:04:06.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.709+0000 7f1f5fe00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f600c0c20 con 0x7f1f600c1010 2024-02-21T23:04:06.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.709+0000 7f1f5fe00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f601505a0 con 0x7f1f6006e870 2024-02-21T23:04:06.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.709+0000 7f1f5f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f601501c0 0x7f1f6013b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:06.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.709+0000 7f1f5f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f601501c0 0x7f1f6013b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56256/0 (socket says 172.21.15.28:56256) 2024-02-21T23:04:06.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.709+0000 7f1f5f400640 1 -- 172.21.15.28:0/3406077424 learned_addr learned my addr 172.21.15.28:0/3406077424 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:06.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.709+0000 7f1f5e000640 1 --2- 172.21.15.28:0/3406077424 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1f600c1010 0x7f1f6013ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:06.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.709+0000 7f1f5ea00640 1 --2- 172.21.15.28:0/3406077424 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1f6006e870 0x7f1f600bfcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:06.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.710+0000 7f1f5f400640 1 -- 172.21.15.28:0/3406077424 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1f6006e870 msgr2=0x7f1f600bfcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:06.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.710+0000 7f1f5f400640 1 --2- 172.21.15.28:0/3406077424 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1f6006e870 0x7f1f600bfcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:06.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.710+0000 7f1f5f400640 1 -- 172.21.15.28:0/3406077424 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1f600c1010 msgr2=0x7f1f6013ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:06.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.710+0000 7f1f5f400640 1 --2- 172.21.15.28:0/3406077424 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1f600c1010 0x7f1f6013ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:06.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.710+0000 7f1f5f400640 1 -- 172.21.15.28:0/3406077424 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f60150d40 con 0x7f1f601501c0 2024-02-21T23:04:06.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.710+0000 7f1f5f400640 1 --2- 172.21.15.28:0/3406077424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f601501c0 0x7f1f6013b2b0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f1f58007e90 tx=0x7f1f5800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:06.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.710+0000 7f1f56a00640 1 -- 172.21.15.28:0/3406077424 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f58014420 con 0x7f1f601501c0 2024-02-21T23:04:06.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.710+0000 7f1f56a00640 1 -- 172.21.15.28:0/3406077424 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f58019070 con 0x7f1f601501c0 2024-02-21T23:04:06.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.710+0000 7f1f56a00640 1 -- 172.21.15.28:0/3406077424 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f58014800 con 0x7f1f601501c0 2024-02-21T23:04:06.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.710+0000 7f1f5fe00640 1 -- 172.21.15.28:0/3406077424 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f60151020 con 0x7f1f601501c0 2024-02-21T23:04:06.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.711+0000 7f1f5fe00640 1 -- 172.21.15.28:0/3406077424 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f601515e0 con 0x7f1f601501c0 2024-02-21T23:04:06.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.712+0000 7f1f5fe00640 1 -- 172.21.15.28:0/3406077424 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f600c0a10 con 0x7f1f601501c0 2024-02-21T23:04:06.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.713+0000 7f1f56a00640 1 -- 172.21.15.28:0/3406077424 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1f580149a0 con 0x7f1f601501c0 2024-02-21T23:04:06.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.714+0000 7f1f56a00640 1 --2- 172.21.15.28:0/3406077424 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1f4006b4d0 0x7f1f4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:06.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.714+0000 7f1f56a00640 1 -- 172.21.15.28:0/3406077424 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f1f5808a880 con 0x7f1f601501c0 2024-02-21T23:04:06.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.715+0000 7f1f5ea00640 1 --2- 172.21.15.28:0/3406077424 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1f4006b4d0 0x7f1f4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:06.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.716+0000 7f1f5ea00640 1 --2- 172.21.15.28:0/3406077424 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1f4006b4d0 0x7f1f4006d990 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f1f6006bf90 tx=0x7f1f50009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:06.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.719+0000 7f1f56a00640 1 -- 172.21.15.28:0/3406077424 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1f600c0a10 con 0x7f1f601501c0 2024-02-21T23:04:06.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:06.921+0000 7f1f5fe00640 1 -- 172.21.15.28:0/3406077424 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f1f600aa4d0 con 0x7f1f601501c0 2024-02-21T23:04:07.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.542+0000 7f1f56a00640 1 -- 172.21.15.28:0/3406077424 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v35) v1 ==== 138+0+0 (secure 0 0 0) 0x7f1f600aa4d0 con 0x7f1f601501c0 2024-02-21T23:04:07.531 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' created 2024-02-21T23:04:07.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.546+0000 7f1f5fe00640 1 -- 172.21.15.28:0/3406077424 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1f4006b4d0 msgr2=0x7f1f4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:07.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.546+0000 7f1f5fe00640 1 --2- 172.21.15.28:0/3406077424 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1f4006b4d0 0x7f1f4006d990 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f1f6006bf90 tx=0x7f1f50009040 comp rx=0 tx=0).stop 2024-02-21T23:04:07.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.546+0000 7f1f5fe00640 1 -- 172.21.15.28:0/3406077424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f601501c0 msgr2=0x7f1f6013b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:07.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.546+0000 7f1f5fe00640 1 --2- 172.21.15.28:0/3406077424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f601501c0 0x7f1f6013b2b0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f1f58007e90 tx=0x7f1f5800e4b0 comp rx=0 tx=0).stop 2024-02-21T23:04:07.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.546+0000 7f1f5fe00640 1 -- 172.21.15.28:0/3406077424 shutdown_connections 2024-02-21T23:04:07.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.546+0000 7f1f5fe00640 1 --2- 172.21.15.28:0/3406077424 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1f4006b4d0 0x7f1f4006d990 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:07.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.546+0000 7f1f5fe00640 1 --2- 172.21.15.28:0/3406077424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f601501c0 0x7f1f6013b2b0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:07.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.546+0000 7f1f5fe00640 1 --2- 172.21.15.28:0/3406077424 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1f600c1010 0x7f1f6013ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:07.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.546+0000 7f1f5fe00640 1 --2- 172.21.15.28:0/3406077424 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1f6006e870 0x7f1f600bfcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:07.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.546+0000 7f1f5fe00640 1 -- 172.21.15.28:0/3406077424 >> 172.21.15.28:0/3406077424 conn(0x7f1f600c6610 msgr2=0x7f1f60145010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:07.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.547+0000 7f1f5fe00640 1 -- 172.21.15.28:0/3406077424 shutdown_connections 2024-02-21T23:04:07.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.547+0000 7f1f5fe00640 1 -- 172.21.15.28:0/3406077424 wait complete. 2024-02-21T23:04:07.563 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-02-21T23:04:07.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.795+0000 7efd5b200640 1 Processor -- start 2024-02-21T23:04:07.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.796+0000 7efd5b200640 1 -- start start 2024-02-21T23:04:07.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.797+0000 7efd5b200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7efd5c0c35c0 0x7efd5c156bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:07.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.797+0000 7efd5b200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efd5c0a96c0 0x7efd5c14ff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:07.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.798+0000 7efd5b200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd5c1508e0 con 0x7efd5c150470 2024-02-21T23:04:07.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.798+0000 7efd5b200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd5c06ea40 con 0x7efd5c0c35c0 2024-02-21T23:04:07.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.798+0000 7efd5b200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd5c06f1b0 con 0x7efd5c0a96c0 2024-02-21T23:04:07.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.798+0000 7efd5a800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7efd5c150470 0x7efd5c152c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:56646/0 (socket says 172.21.15.28:56646) 2024-02-21T23:04:07.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.798+0000 7efd5a800640 1 -- 172.21.15.28:0/2251870595 learned_addr learned my addr 172.21.15.28:0/2251870595 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:04:07.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.798+0000 7efd59400640 1 --2- 172.21.15.28:0/2251870595 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efd5c0a96c0 0x7efd5c14ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:07.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.798+0000 7efd59e00640 1 --2- 172.21.15.28:0/2251870595 >> v2:172.21.15.81:3300/0 conn(0x7efd5c0c35c0 0x7efd5c156bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:07.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.799+0000 7efd58a00640 1 -- 172.21.15.28:0/2251870595 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2103280829 0 0) 0x7efd5c1508e0 con 0x7efd5c150470 2024-02-21T23:04:07.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.799+0000 7efd58a00640 1 -- 172.21.15.28:0/2251870595 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd40003520 con 0x7efd5c150470 2024-02-21T23:04:07.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.799+0000 7efd58a00640 1 -- 172.21.15.28:0/2251870595 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3625906249 0 0) 0x7efd40003520 con 0x7efd5c150470 2024-02-21T23:04:07.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.799+0000 7efd58a00640 1 -- 172.21.15.28:0/2251870595 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efd5c0a96c0 msgr2=0x7efd5c14ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:07.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.799+0000 7efd58a00640 1 --2- 172.21.15.28:0/2251870595 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efd5c0a96c0 0x7efd5c14ff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:07.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.799+0000 7efd58a00640 1 -- 172.21.15.28:0/2251870595 >> v2:172.21.15.81:3300/0 conn(0x7efd5c0c35c0 msgr2=0x7efd5c156bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:07.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.799+0000 7efd58a00640 1 --2- 172.21.15.28:0/2251870595 >> v2:172.21.15.81:3300/0 conn(0x7efd5c0c35c0 0x7efd5c156bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:07.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.799+0000 7efd58a00640 1 -- 172.21.15.28:0/2251870595 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd5c0c2ae0 con 0x7efd5c150470 2024-02-21T23:04:07.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.799+0000 7efd58a00640 1 -- 172.21.15.28:0/2251870595 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7efd54001670 con 0x7efd5c150470 2024-02-21T23:04:07.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.799+0000 7efd58a00640 1 -- 172.21.15.28:0/2251870595 >> v1:172.21.15.28:6789/0 conn(0x7efd5c150470 legacy=0x7efd5c152c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:07.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.799+0000 7efd58a00640 1 --2- 172.21.15.28:0/2251870595 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efd5c0a96c0 0x7efd400039d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:07.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.799+0000 7efd58a00640 1 --2- 172.21.15.28:0/2251870595 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd40003fa0 0x7efd40004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:07.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.799+0000 7efd58a00640 1 --2- 172.21.15.28:0/2251870595 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd40004d40 0x7efd400051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:07.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.800+0000 7efd58a00640 1 -- 172.21.15.28:0/2251870595 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd5c06ea40 con 0x7efd40003fa0 2024-02-21T23:04:07.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.800+0000 7efd58a00640 1 -- 172.21.15.28:0/2251870595 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd5c06f1b0 con 0x7efd40004d40 2024-02-21T23:04:07.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.800+0000 7efd58a00640 1 -- 172.21.15.28:0/2251870595 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd4000ba70 con 0x7efd5c0a96c0 2024-02-21T23:04:07.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.800+0000 7efd59400640 1 --2- 172.21.15.28:0/2251870595 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd40003fa0 0x7efd40004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:07.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.800+0000 7efd59e00640 1 --2- 172.21.15.28:0/2251870595 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efd5c0a96c0 0x7efd400039d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:07.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.800+0000 7efd5a800640 1 --2- 172.21.15.28:0/2251870595 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd40004d40 0x7efd400051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:07.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.800+0000 7efd59400640 1 -- 172.21.15.28:0/2251870595 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efd5c0a96c0 msgr2=0x7efd400039d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:07.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.800+0000 7efd59400640 1 --2- 172.21.15.28:0/2251870595 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efd5c0a96c0 0x7efd400039d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:07.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.800+0000 7efd59400640 1 -- 172.21.15.28:0/2251870595 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd40004d40 msgr2=0x7efd400051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:07.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.800+0000 7efd59400640 1 --2- 172.21.15.28:0/2251870595 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd40004d40 0x7efd400051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:07.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.800+0000 7efd59400640 1 -- 172.21.15.28:0/2251870595 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd40003520 con 0x7efd40003fa0 2024-02-21T23:04:07.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.802+0000 7efd59400640 1 --2- 172.21.15.28:0/2251870595 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd40003fa0 0x7efd40004800 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7efd48009ac0 tx=0x7efd40005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab48599e88c8c033 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:07.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.802+0000 7efd58a00640 1 -- 172.21.15.28:0/2251870595 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd48039070 con 0x7efd40003fa0 2024-02-21T23:04:07.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.802+0000 7efd58a00640 1 -- 172.21.15.28:0/2251870595 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd4802c6b0 con 0x7efd40003fa0 2024-02-21T23:04:07.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.802+0000 7efd58a00640 1 -- 172.21.15.28:0/2251870595 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd48009690 con 0x7efd40003fa0 2024-02-21T23:04:07.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.802+0000 7efd5b200640 1 -- 172.21.15.28:0/2251870595 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd40003fa0 msgr2=0x7efd40004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:07.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.802+0000 7efd5b200640 1 --2- 172.21.15.28:0/2251870595 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd40003fa0 0x7efd40004800 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7efd48009ac0 tx=0x7efd40005830 comp rx=0 tx=0).stop 2024-02-21T23:04:07.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.803+0000 7efd5a800640 1 -- 172.21.15.28:0/2251870595 reap_dead start 2024-02-21T23:04:07.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.803+0000 7efd5b200640 1 -- 172.21.15.28:0/2251870595 shutdown_connections 2024-02-21T23:04:07.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.803+0000 7efd5b200640 1 -- 172.21.15.28:0/2251870595 >> 172.21.15.28:0/2251870595 conn(0x7efd5c0a8690 msgr2=0x7efd5c0a8a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:07.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.803+0000 7efd5b200640 1 -- 172.21.15.28:0/2251870595 shutdown_connections 2024-02-21T23:04:07.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.803+0000 7efd5b200640 1 -- 172.21.15.28:0/2251870595 wait complete. 2024-02-21T23:04:07.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.805+0000 7efd5b200640 1 Processor -- start 2024-02-21T23:04:07.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.805+0000 7efd5b200640 1 -- start start 2024-02-21T23:04:07.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.806+0000 7efd5b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd40003fa0 0x7efd5c14bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:07.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.806+0000 7efd5b200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efd5c145d70 0x7efd5c153350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:07.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.806+0000 7efd5b200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd5c153890 0x7efd5c154fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:07.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.806+0000 7efd5b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd5c06e7b0 con 0x7efd40003fa0 2024-02-21T23:04:07.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.806+0000 7efd5b200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd5c06ec90 con 0x7efd5c153890 2024-02-21T23:04:07.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.806+0000 7efd5b200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd5c0c39a0 con 0x7efd5c145d70 2024-02-21T23:04:07.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.806+0000 7efd59e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd40003fa0 0x7efd5c14bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:07.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.806+0000 7efd59e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd40003fa0 0x7efd5c14bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56264/0 (socket says 172.21.15.28:56264) 2024-02-21T23:04:07.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.806+0000 7efd59e00640 1 -- 172.21.15.28:0/1976636213 learned_addr learned my addr 172.21.15.28:0/1976636213 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:07.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.806+0000 7efd59e00640 1 -- 172.21.15.28:0/1976636213 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efd5c145d70 msgr2=0x7efd5c153350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:07.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.806+0000 7efd59e00640 1 --2- 172.21.15.28:0/1976636213 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efd5c145d70 0x7efd5c153350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:07.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.806+0000 7efd59e00640 1 -- 172.21.15.28:0/1976636213 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd5c153890 msgr2=0x7efd5c154fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:07.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.806+0000 7efd5a800640 1 --2- 172.21.15.28:0/1976636213 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd5c153890 0x7efd5c154fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:07.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.806+0000 7efd59e00640 1 --2- 172.21.15.28:0/1976636213 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd5c153890 0x7efd5c154fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:07.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.806+0000 7efd59e00640 1 -- 172.21.15.28:0/1976636213 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd5c14d1a0 con 0x7efd40003fa0 2024-02-21T23:04:07.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.807+0000 7efd59e00640 1 --2- 172.21.15.28:0/1976636213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd40003fa0 0x7efd5c14bda0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7efd4c007f60 tx=0x7efd4c007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:07.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.807+0000 7efd52000640 1 -- 172.21.15.28:0/1976636213 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd4c01a020 con 0x7efd40003fa0 2024-02-21T23:04:07.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.807+0000 7efd52000640 1 -- 172.21.15.28:0/1976636213 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd4c016470 con 0x7efd40003fa0 2024-02-21T23:04:07.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.807+0000 7efd52000640 1 -- 172.21.15.28:0/1976636213 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd4c016860 con 0x7efd40003fa0 2024-02-21T23:04:07.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.807+0000 7efd5b200640 1 -- 172.21.15.28:0/1976636213 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd5c1461f0 con 0x7efd40003fa0 2024-02-21T23:04:07.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.807+0000 7efd5b200640 1 -- 172.21.15.28:0/1976636213 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd5c14d440 con 0x7efd40003fa0 2024-02-21T23:04:07.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.808+0000 7efd5b200640 1 -- 172.21.15.28:0/1976636213 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd5c04f290 con 0x7efd40003fa0 2024-02-21T23:04:07.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.810+0000 7efd52000640 1 -- 172.21.15.28:0/1976636213 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efd4c005d10 con 0x7efd40003fa0 2024-02-21T23:04:07.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.811+0000 7efd52000640 1 --2- 172.21.15.28:0/1976636213 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7efd3c06b500 0x7efd3c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:07.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.811+0000 7efd52000640 1 -- 172.21.15.28:0/1976636213 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5619+0+0 (secure 0 0 0) 0x7efd4c08b860 con 0x7efd40003fa0 2024-02-21T23:04:07.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.811+0000 7efd59400640 1 --2- 172.21.15.28:0/1976636213 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7efd3c06b500 0x7efd3c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:07.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.813+0000 7efd59400640 1 --2- 172.21.15.28:0/1976636213 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7efd3c06b500 0x7efd3c06d9c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7efd48036000 tx=0x7efd4802e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:07.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:07.815+0000 7efd52000640 1 -- 172.21.15.28:0/1976636213 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7efd4c024030 con 0x7efd40003fa0 2024-02-21T23:04:08.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.017+0000 7efd5b200640 1 -- 172.21.15.28:0/1976636213 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7efd5c152b70 con 0x7efd40003fa0 2024-02-21T23:04:08.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.549+0000 7efd52000640 1 -- 172.21.15.28:0/1976636213 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v36) v1 ==== 170+0+0 (secure 0 0 0) 0x7efd4c05a7a0 con 0x7efd40003fa0 2024-02-21T23:04:08.538 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' created 2024-02-21T23:04:08.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.553+0000 7efd5b200640 1 -- 172.21.15.28:0/1976636213 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7efd3c06b500 msgr2=0x7efd3c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:08.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.553+0000 7efd5b200640 1 --2- 172.21.15.28:0/1976636213 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7efd3c06b500 0x7efd3c06d9c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7efd48036000 tx=0x7efd4802e000 comp rx=0 tx=0).stop 2024-02-21T23:04:08.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.553+0000 7efd5b200640 1 -- 172.21.15.28:0/1976636213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd40003fa0 msgr2=0x7efd5c14bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:08.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.553+0000 7efd5b200640 1 --2- 172.21.15.28:0/1976636213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd40003fa0 0x7efd5c14bda0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7efd4c007f60 tx=0x7efd4c007f90 comp rx=0 tx=0).stop 2024-02-21T23:04:08.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.554+0000 7efd5b200640 1 -- 172.21.15.28:0/1976636213 shutdown_connections 2024-02-21T23:04:08.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.554+0000 7efd5b200640 1 --2- 172.21.15.28:0/1976636213 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7efd3c06b500 0x7efd3c06d9c0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:08.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.554+0000 7efd5b200640 1 --2- 172.21.15.28:0/1976636213 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efd5c145d70 0x7efd5c153350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:08.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.554+0000 7efd5b200640 1 --2- 172.21.15.28:0/1976636213 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd5c153890 0x7efd5c154fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:08.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.554+0000 7efd5b200640 1 --2- 172.21.15.28:0/1976636213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd40003fa0 0x7efd5c14bda0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:08.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.554+0000 7efd5b200640 1 -- 172.21.15.28:0/1976636213 >> 172.21.15.28:0/1976636213 conn(0x7efd5c0a8690 msgr2=0x7efd5c0a8a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:08.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.554+0000 7efd5b200640 1 -- 172.21.15.28:0/1976636213 shutdown_connections 2024-02-21T23:04:08.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.554+0000 7efd5b200640 1 -- 172.21.15.28:0/1976636213 wait complete. 2024-02-21T23:04:08.576 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-02-21T23:04:08.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.778+0000 7f3125000640 1 Processor -- start 2024-02-21T23:04:08.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.778+0000 7f3125000640 1 -- start start 2024-02-21T23:04:08.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.780+0000 7f3125000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f312013e0c0 0x7f31200c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:08.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.780+0000 7f3125000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f312006dbc0 0x7f312006c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:08.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.780+0000 7f3125000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f31200c76f0 0x7f312006c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:08.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.780+0000 7f3125000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31200c89a0 con 0x7f312013e0c0 2024-02-21T23:04:08.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.780+0000 7f3125000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3120158550 con 0x7f312006dbc0 2024-02-21T23:04:08.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.780+0000 7f3125000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31201587d0 con 0x7f31200c76f0 2024-02-21T23:04:08.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.780+0000 7f311f400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f312013e0c0 0x7f31200c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:08.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.780+0000 7f311f400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f312013e0c0 0x7f31200c71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56268/0 (socket says 172.21.15.28:56268) 2024-02-21T23:04:08.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.780+0000 7f311f400640 1 -- 172.21.15.28:0/2592670822 learned_addr learned my addr 172.21.15.28:0/2592670822 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:08.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.780+0000 7f311fe00640 1 --2- 172.21.15.28:0/2592670822 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f31200c76f0 0x7f312006c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:08.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.781+0000 7f311ea00640 1 --2- 172.21.15.28:0/2592670822 >> v2:172.21.15.81:3300/0 conn(0x7f312006dbc0 0x7f312006c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:08.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.781+0000 7f311f400640 1 -- 172.21.15.28:0/2592670822 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f31200c76f0 msgr2=0x7f312006c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:08.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.781+0000 7f311f400640 1 --2- 172.21.15.28:0/2592670822 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f31200c76f0 0x7f312006c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:08.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.781+0000 7f311f400640 1 -- 172.21.15.28:0/2592670822 >> v2:172.21.15.81:3300/0 conn(0x7f312006dbc0 msgr2=0x7f312006c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:08.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.781+0000 7f311f400640 1 --2- 172.21.15.28:0/2592670822 >> v2:172.21.15.81:3300/0 conn(0x7f312006dbc0 0x7f312006c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:08.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.781+0000 7f311f400640 1 -- 172.21.15.28:0/2592670822 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3120158ab0 con 0x7f312013e0c0 2024-02-21T23:04:08.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.782+0000 7f311f400640 1 --2- 172.21.15.28:0/2592670822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f312013e0c0 0x7f31200c71b0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f31200c7100 tx=0x7f311002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8a8f361b563f8e0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:08.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.783+0000 7f311e000640 1 -- 172.21.15.28:0/2592670822 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f311002e070 con 0x7f312013e0c0 2024-02-21T23:04:08.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.783+0000 7f311e000640 1 -- 172.21.15.28:0/2592670822 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3110037070 con 0x7f312013e0c0 2024-02-21T23:04:08.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.783+0000 7f311e000640 1 -- 172.21.15.28:0/2592670822 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3110031a00 con 0x7f312013e0c0 2024-02-21T23:04:08.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.783+0000 7f3125000640 1 -- 172.21.15.28:0/2592670822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f312013e0c0 msgr2=0x7f31200c71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:08.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.783+0000 7f3125000640 1 --2- 172.21.15.28:0/2592670822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f312013e0c0 0x7f31200c71b0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f31200c7100 tx=0x7f311002b880 comp rx=0 tx=0).stop 2024-02-21T23:04:08.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.783+0000 7f3125000640 1 -- 172.21.15.28:0/2592670822 shutdown_connections 2024-02-21T23:04:08.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.783+0000 7f3125000640 1 --2- 172.21.15.28:0/2592670822 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f31200c76f0 0x7f312006c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:08.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.783+0000 7f3125000640 1 --2- 172.21.15.28:0/2592670822 >> v2:172.21.15.81:3300/0 conn(0x7f312006dbc0 0x7f312006c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:08.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.783+0000 7f3125000640 1 --2- 172.21.15.28:0/2592670822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f312013e0c0 0x7f31200c71b0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:08.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.783+0000 7f3125000640 1 -- 172.21.15.28:0/2592670822 >> 172.21.15.28:0/2592670822 conn(0x7f3120073110 msgr2=0x7f31200ac490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:08.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.784+0000 7f3125000640 1 -- 172.21.15.28:0/2592670822 shutdown_connections 2024-02-21T23:04:08.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.784+0000 7f3125000640 1 -- 172.21.15.28:0/2592670822 wait complete. 2024-02-21T23:04:08.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.785+0000 7f3125000640 1 Processor -- start 2024-02-21T23:04:08.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.785+0000 7f3125000640 1 -- start start 2024-02-21T23:04:08.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.786+0000 7f3125000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f312006dbc0 0x7f31200ad8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:08.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.786+0000 7f3125000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f31200c76f0 0x7f3120133a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:08.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.786+0000 7f3125000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f312013e0c0 0x7f3120133fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:08.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.786+0000 7f3125000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f312006cd00 con 0x7f312006dbc0 2024-02-21T23:04:08.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.786+0000 7f3125000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3120159070 con 0x7f312013e0c0 2024-02-21T23:04:08.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.786+0000 7f3125000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f312006ce80 con 0x7f31200c76f0 2024-02-21T23:04:08.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.787+0000 7f311f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f312006dbc0 0x7f31200ad8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:08.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.787+0000 7f311f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f312006dbc0 0x7f31200ad8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56270/0 (socket says 172.21.15.28:56270) 2024-02-21T23:04:08.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.787+0000 7f311f400640 1 -- 172.21.15.28:0/4074499337 learned_addr learned my addr 172.21.15.28:0/4074499337 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:08.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.787+0000 7f311fe00640 1 --2- 172.21.15.28:0/4074499337 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f312013e0c0 0x7f3120133fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:08.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.787+0000 7f311ea00640 1 --2- 172.21.15.28:0/4074499337 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f31200c76f0 0x7f3120133a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:08.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.787+0000 7f311f400640 1 -- 172.21.15.28:0/4074499337 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f31200c76f0 msgr2=0x7f3120133a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:08.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.787+0000 7f311f400640 1 --2- 172.21.15.28:0/4074499337 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f31200c76f0 0x7f3120133a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:08.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.787+0000 7f311f400640 1 -- 172.21.15.28:0/4074499337 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f312013e0c0 msgr2=0x7f3120133fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:08.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.787+0000 7f311f400640 1 --2- 172.21.15.28:0/4074499337 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f312013e0c0 0x7f3120133fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:08.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.787+0000 7f311f400640 1 -- 172.21.15.28:0/4074499337 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3120139ec0 con 0x7f312006dbc0 2024-02-21T23:04:08.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.787+0000 7f311ea00640 1 --2- 172.21.15.28:0/4074499337 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f31200c76f0 0x7f3120133a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:04:08.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.788+0000 7f311f400640 1 --2- 172.21.15.28:0/4074499337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f312006dbc0 0x7f31200ad8f0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f3110000c00 tx=0x7f31100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:08.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.788+0000 7f3117400640 1 -- 172.21.15.28:0/4074499337 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3110032670 con 0x7f312006dbc0 2024-02-21T23:04:08.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.788+0000 7f3117400640 1 -- 172.21.15.28:0/4074499337 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3110032810 con 0x7f312006dbc0 2024-02-21T23:04:08.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.788+0000 7f3125000640 1 -- 172.21.15.28:0/4074499337 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f312014d8f0 con 0x7f312006dbc0 2024-02-21T23:04:08.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.788+0000 7f3117400640 1 -- 172.21.15.28:0/4074499337 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3110032c00 con 0x7f312006dbc0 2024-02-21T23:04:08.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.788+0000 7f3125000640 1 -- 172.21.15.28:0/4074499337 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3120139900 con 0x7f312006dbc0 2024-02-21T23:04:08.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.789+0000 7f3125000640 1 -- 172.21.15.28:0/4074499337 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f312004f290 con 0x7f312006dbc0 2024-02-21T23:04:08.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.790+0000 7f3117400640 1 -- 172.21.15.28:0/4074499337 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f311004c060 con 0x7f312006dbc0 2024-02-21T23:04:08.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.791+0000 7f3117400640 1 --2- 172.21.15.28:0/4074499337 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f310006b500 0x7f310006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:08.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.791+0000 7f3117400640 1 -- 172.21.15.28:0/4074499337 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 6001+0+0 (secure 0 0 0) 0x7f31100abc70 con 0x7f312006dbc0 2024-02-21T23:04:08.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.792+0000 7f311ea00640 1 --2- 172.21.15.28:0/4074499337 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f310006b500 0x7f310006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:08.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.793+0000 7f311ea00640 1 --2- 172.21.15.28:0/4074499337 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f310006b500 0x7f310006d9c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f312013c1a0 tx=0x7f310c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:08.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:08.797+0000 7f3117400640 1 -- 172.21.15.28:0/4074499337 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3110040070 con 0x7f312006dbc0 2024-02-21T23:04:09.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.034+0000 7f3125000640 1 -- 172.21.15.28:0/4074499337 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f312006d9a0 con 0x7f312006dbc0 2024-02-21T23:04:09.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.577+0000 7f3117400640 1 -- 172.21.15.28:0/4074499337 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 5 and data pool 6 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7f311007a9e0 con 0x7f312006dbc0 2024-02-21T23:04:09.567 INFO:teuthology.orchestra.run.smithi028.stderr:new fs with metadata pool 5 and data pool 6 2024-02-21T23:04:09.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.581+0000 7f3125000640 1 -- 172.21.15.28:0/4074499337 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f310006b500 msgr2=0x7f310006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:09.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.581+0000 7f3125000640 1 --2- 172.21.15.28:0/4074499337 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f310006b500 0x7f310006d9c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f312013c1a0 tx=0x7f310c008040 comp rx=0 tx=0).stop 2024-02-21T23:04:09.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.581+0000 7f3125000640 1 -- 172.21.15.28:0/4074499337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f312006dbc0 msgr2=0x7f31200ad8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:09.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.581+0000 7f3125000640 1 --2- 172.21.15.28:0/4074499337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f312006dbc0 0x7f31200ad8f0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f3110000c00 tx=0x7f31100046c0 comp rx=0 tx=0).stop 2024-02-21T23:04:09.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.581+0000 7f3125000640 1 -- 172.21.15.28:0/4074499337 shutdown_connections 2024-02-21T23:04:09.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.581+0000 7f3125000640 1 --2- 172.21.15.28:0/4074499337 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f310006b500 0x7f310006d9c0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:09.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.581+0000 7f3125000640 1 --2- 172.21.15.28:0/4074499337 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f31200c76f0 0x7f3120133a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:09.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.581+0000 7f3125000640 1 --2- 172.21.15.28:0/4074499337 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f312013e0c0 0x7f3120133fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:09.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.581+0000 7f3125000640 1 --2- 172.21.15.28:0/4074499337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f312006dbc0 0x7f31200ad8f0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:09.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.581+0000 7f3125000640 1 -- 172.21.15.28:0/4074499337 >> 172.21.15.28:0/4074499337 conn(0x7f3120073110 msgr2=0x7f31200c5ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:09.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.581+0000 7f3125000640 1 -- 172.21.15.28:0/4074499337 shutdown_connections 2024-02-21T23:04:09.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.581+0000 7f3125000640 1 -- 172.21.15.28:0/4074499337 wait complete. 2024-02-21T23:04:09.610 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:04:09.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.807+0000 7f79cf000640 1 Processor -- start 2024-02-21T23:04:09.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.808+0000 7f79cf000640 1 -- start start 2024-02-21T23:04:09.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.809+0000 7f79cf000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f79c806dbf0 0x7f79c8158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:09.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.809+0000 7f79cf000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f79c8155960 0x7f79c8159000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:09.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.809+0000 7f79cf000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79c80c74c0 con 0x7f79c80aff90 2024-02-21T23:04:09.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.809+0000 7f79cf000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79c80c7270 con 0x7f79c806dbf0 2024-02-21T23:04:09.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.809+0000 7f79cf000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79c8158630 con 0x7f79c8155960 2024-02-21T23:04:09.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.809+0000 7f79cd200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f79c806dbf0 0x7f79c8158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:09.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.809+0000 7f79cd200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f79c806dbf0 0x7f79c8158ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56280/0 (socket says 172.21.15.28:56280) 2024-02-21T23:04:09.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.809+0000 7f79cd200640 1 -- 172.21.15.28:0/3112015475 learned_addr learned my addr 172.21.15.28:0/3112015475 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:09.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.810+0000 7f79ce600640 1 --2- 172.21.15.28:0/3112015475 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f79c8155960 0x7f79c8159000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:09.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.810+0000 7f79cd200640 1 -- 172.21.15.28:0/3112015475 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f79c8155960 msgr2=0x7f79c8159000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:09.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.810+0000 7f79cd200640 1 --2- 172.21.15.28:0/3112015475 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f79c8155960 0x7f79c8159000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:09.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.810+0000 7f79cd200640 1 -- 172.21.15.28:0/3112015475 >> v1:172.21.15.81:6789/0 conn(0x7f79c80aff90 legacy=0x7f79c80c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:09.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.810+0000 7f79cd200640 1 -- 172.21.15.28:0/3112015475 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79c80c77a0 con 0x7f79c806dbf0 2024-02-21T23:04:09.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.811+0000 7f79cd200640 1 --2- 172.21.15.28:0/3112015475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79c806dbf0 0x7f79c8158ac0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f79b4009870 tx=0x7f79b402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dc9e8aa6e6ce0fe server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:09.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.811+0000 7f79c7e00640 1 -- 172.21.15.28:0/3112015475 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79b4035020 con 0x7f79c806dbf0 2024-02-21T23:04:09.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.811+0000 7f79c7e00640 1 -- 172.21.15.28:0/3112015475 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79b4031640 con 0x7f79c806dbf0 2024-02-21T23:04:09.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.811+0000 7f79c7e00640 1 -- 172.21.15.28:0/3112015475 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79b402d040 con 0x7f79c806dbf0 2024-02-21T23:04:09.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.812+0000 7f79cf000640 1 -- 172.21.15.28:0/3112015475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79c806dbf0 msgr2=0x7f79c8158ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:09.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.812+0000 7f79cf000640 1 --2- 172.21.15.28:0/3112015475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79c806dbf0 0x7f79c8158ac0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f79b4009870 tx=0x7f79b402b8f0 comp rx=0 tx=0).stop 2024-02-21T23:04:09.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.812+0000 7f79cf000640 1 -- 172.21.15.28:0/3112015475 shutdown_connections 2024-02-21T23:04:09.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.812+0000 7f79cf000640 1 --2- 172.21.15.28:0/3112015475 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f79c8155960 0x7f79c8159000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:09.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.812+0000 7f79cf000640 1 --2- 172.21.15.28:0/3112015475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79c806dbf0 0x7f79c8158ac0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:09.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.812+0000 7f79cf000640 1 -- 172.21.15.28:0/3112015475 >> 172.21.15.28:0/3112015475 conn(0x7f79c80ac370 msgr2=0x7f79c80ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:09.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.812+0000 7f79cf000640 1 -- 172.21.15.28:0/3112015475 shutdown_connections 2024-02-21T23:04:09.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.812+0000 7f79cf000640 1 -- 172.21.15.28:0/3112015475 wait complete. 2024-02-21T23:04:09.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.814+0000 7f79cf000640 1 Processor -- start 2024-02-21T23:04:09.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.814+0000 7f79cf000640 1 -- start start 2024-02-21T23:04:09.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.815+0000 7f79cf000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f79c806dbf0 0x7f79c80ab710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:09.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.815+0000 7f79cf000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79c80aff90 0x7f79c81ebe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:09.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.815+0000 7f79cf000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f79c8155960 0x7f79c81ec340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:09.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.815+0000 7f79cf000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79c80c58f0 con 0x7f79c80aff90 2024-02-21T23:04:09.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.815+0000 7f79cf000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79c80be0c0 con 0x7f79c806dbf0 2024-02-21T23:04:09.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.815+0000 7f79cf000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79c80c5a70 con 0x7f79c8155960 2024-02-21T23:04:09.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.815+0000 7f79cd200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79c80aff90 0x7f79c81ebe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:09.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.815+0000 7f79cd200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79c80aff90 0x7f79c81ebe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56294/0 (socket says 172.21.15.28:56294) 2024-02-21T23:04:09.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.815+0000 7f79cd200640 1 -- 172.21.15.28:0/326618792 learned_addr learned my addr 172.21.15.28:0/326618792 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:09.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.815+0000 7f79cdc00640 1 --2- 172.21.15.28:0/326618792 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f79c806dbf0 0x7f79c80ab710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:09.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.815+0000 7f79ce600640 1 --2- 172.21.15.28:0/326618792 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f79c8155960 0x7f79c81ec340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:09.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.816+0000 7f79cd200640 1 -- 172.21.15.28:0/326618792 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f79c8155960 msgr2=0x7f79c81ec340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:09.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.816+0000 7f79cd200640 1 --2- 172.21.15.28:0/326618792 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f79c8155960 0x7f79c81ec340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:09.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.816+0000 7f79cd200640 1 -- 172.21.15.28:0/326618792 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f79c806dbf0 msgr2=0x7f79c80ab710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:09.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.816+0000 7f79cd200640 1 --2- 172.21.15.28:0/326618792 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f79c806dbf0 0x7f79c80ab710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:09.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.816+0000 7f79cd200640 1 -- 172.21.15.28:0/326618792 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79c81deef0 con 0x7f79c80aff90 2024-02-21T23:04:09.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.816+0000 7f79cd200640 1 --2- 172.21.15.28:0/326618792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79c80aff90 0x7f79c81ebe00 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f79b402b930 tx=0x7f79b40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:09.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.816+0000 7f79c5600640 1 -- 172.21.15.28:0/326618792 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79b4040070 con 0x7f79c80aff90 2024-02-21T23:04:09.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.816+0000 7f79c5600640 1 -- 172.21.15.28:0/326618792 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79b40399f0 con 0x7f79c80aff90 2024-02-21T23:04:09.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.816+0000 7f79c5600640 1 -- 172.21.15.28:0/326618792 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79b4035040 con 0x7f79c80aff90 2024-02-21T23:04:09.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.816+0000 7f79cf000640 1 -- 172.21.15.28:0/326618792 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79c81df1d0 con 0x7f79c80aff90 2024-02-21T23:04:09.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.817+0000 7f79cf000640 1 -- 172.21.15.28:0/326618792 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79c81ee430 con 0x7f79c80aff90 2024-02-21T23:04:09.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.818+0000 7f79cf000640 1 -- 172.21.15.28:0/326618792 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79c804f290 con 0x7f79c80aff90 2024-02-21T23:04:09.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.819+0000 7f79c5600640 1 -- 172.21.15.28:0/326618792 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f79b4038020 con 0x7f79c80aff90 2024-02-21T23:04:09.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.819+0000 7f79c5600640 1 --2- 172.21.15.28:0/326618792 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f79a806b4d0 0x7f79a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:09.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.820+0000 7f79c5600640 1 -- 172.21.15.28:0/326618792 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f79b40abd00 con 0x7f79c80aff90 2024-02-21T23:04:09.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.820+0000 7f79cdc00640 1 --2- 172.21.15.28:0/326618792 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f79a806b4d0 0x7f79a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:09.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.821+0000 7f79cdc00640 1 --2- 172.21.15.28:0/326618792 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f79a806b4d0 0x7f79a806d990 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f79c80c18e0 tx=0x7f79bc009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:09.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:09.824+0000 7f79c5600640 1 -- 172.21.15.28:0/326618792 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f79b407aa80 con 0x7f79c80aff90 2024-02-21T23:04:10.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.021+0000 7f79cf000640 1 -- 172.21.15.28:0/326618792 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f79c80cb2a0 con 0x7f79c80aff90 2024-02-21T23:04:10.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.024+0000 7f79c5600640 1 -- 172.21.15.28:0/326618792 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) v1 ==== 74+0+16895 (secure 0 0 0) 0x7f79b407a380 con 0x7f79c80aff90 2024-02-21T23:04:10.013 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:04:10.013 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":37,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:04:09.035957+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:04:06.977614+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-02-21T23:04:08.074258+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:04:10.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.026+0000 7f79cf000640 1 -- 172.21.15.28:0/326618792 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f79a806b4d0 msgr2=0x7f79a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:10.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.026+0000 7f79cf000640 1 --2- 172.21.15.28:0/326618792 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f79a806b4d0 0x7f79a806d990 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f79c80c18e0 tx=0x7f79bc009040 comp rx=0 tx=0).stop 2024-02-21T23:04:10.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.026+0000 7f79cf000640 1 -- 172.21.15.28:0/326618792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79c80aff90 msgr2=0x7f79c81ebe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:10.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.026+0000 7f79cf000640 1 --2- 172.21.15.28:0/326618792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79c80aff90 0x7f79c81ebe00 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f79b402b930 tx=0x7f79b40046c0 comp rx=0 tx=0).stop 2024-02-21T23:04:10.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.026+0000 7f79cf000640 1 -- 172.21.15.28:0/326618792 shutdown_connections 2024-02-21T23:04:10.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.026+0000 7f79cf000640 1 --2- 172.21.15.28:0/326618792 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f79a806b4d0 0x7f79a806d990 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:10.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.026+0000 7f79cf000640 1 --2- 172.21.15.28:0/326618792 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f79c8155960 0x7f79c81ec340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:10.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.026+0000 7f79cf000640 1 --2- 172.21.15.28:0/326618792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79c80aff90 0x7f79c81ebe00 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:10.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.026+0000 7f79cf000640 1 --2- 172.21.15.28:0/326618792 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f79c806dbf0 0x7f79c80ab710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:10.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.027+0000 7f79cf000640 1 -- 172.21.15.28:0/326618792 >> 172.21.15.28:0/326618792 conn(0x7f79c80ac370 msgr2=0x7f79c8156c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:10.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.027+0000 7f79cf000640 1 -- 172.21.15.28:0/326618792 shutdown_connections 2024-02-21T23:04:10.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.027+0000 7f79cf000640 1 -- 172.21.15.28:0/326618792 wait complete. 2024-02-21T23:04:10.032 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:04:10.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.263+0000 7f26e5400640 1 Processor -- start 2024-02-21T23:04:10.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.263+0000 7f26e5400640 1 -- start start 2024-02-21T23:04:10.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.265+0000 7f26e5400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f26e006ee40 0x7f26e00a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:10.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.265+0000 7f26e5400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f26e0156af0 0x7f26e006dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:10.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.265+0000 7f26e5400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26e00c2760 con 0x7f26e0150450 2024-02-21T23:04:10.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.265+0000 7f26e5400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26e00c22c0 con 0x7f26e0156af0 2024-02-21T23:04:10.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.265+0000 7f26e5400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26e0156660 con 0x7f26e006ee40 2024-02-21T23:04:10.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.265+0000 7f26e4a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f26e0156af0 0x7f26e006dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:10.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.265+0000 7f26e4a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f26e0156af0 0x7f26e006dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56308/0 (socket says 172.21.15.28:56308) 2024-02-21T23:04:10.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.265+0000 7f26e4a00640 1 -- 172.21.15.28:0/27649055 learned_addr learned my addr 172.21.15.28:0/27649055 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:10.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.266+0000 7f26d7400640 1 --2- 172.21.15.28:0/27649055 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f26e006ee40 0x7f26e00a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:10.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.266+0000 7f26e4a00640 1 -- 172.21.15.28:0/27649055 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f26e006ee40 msgr2=0x7f26e00a5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:10.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.266+0000 7f26e4a00640 1 --2- 172.21.15.28:0/27649055 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f26e006ee40 0x7f26e00a5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:10.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.266+0000 7f26e4a00640 1 -- 172.21.15.28:0/27649055 >> v1:172.21.15.81:6789/0 conn(0x7f26e0150450 legacy=0x7f26e01536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:10.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.266+0000 7f26e4a00640 1 -- 172.21.15.28:0/27649055 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e0156310 con 0x7f26e0156af0 2024-02-21T23:04:10.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.267+0000 7f26e4a00640 1 --2- 172.21.15.28:0/27649055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26e0156af0 0x7f26e006dff0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f26d8009870 tx=0x7f26d802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e34543662399212 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:10.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.268+0000 7f26df400640 1 -- 172.21.15.28:0/27649055 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d8034070 con 0x7f26e0156af0 2024-02-21T23:04:10.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.268+0000 7f26df400640 1 -- 172.21.15.28:0/27649055 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26d8030840 con 0x7f26e0156af0 2024-02-21T23:04:10.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.268+0000 7f26df400640 1 -- 172.21.15.28:0/27649055 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d8030c10 con 0x7f26e0156af0 2024-02-21T23:04:10.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.269+0000 7f26e5400640 1 -- 172.21.15.28:0/27649055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26e0156af0 msgr2=0x7f26e006dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:10.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.269+0000 7f26e5400640 1 --2- 172.21.15.28:0/27649055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26e0156af0 0x7f26e006dff0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f26d8009870 tx=0x7f26d802b980 comp rx=0 tx=0).stop 2024-02-21T23:04:10.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.269+0000 7f26e5400640 1 -- 172.21.15.28:0/27649055 shutdown_connections 2024-02-21T23:04:10.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.269+0000 7f26e5400640 1 --2- 172.21.15.28:0/27649055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26e0156af0 0x7f26e006dff0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:10.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.269+0000 7f26e5400640 1 --2- 172.21.15.28:0/27649055 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f26e006ee40 0x7f26e00a5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:10.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.269+0000 7f26e5400640 1 -- 172.21.15.28:0/27649055 >> 172.21.15.28:0/27649055 conn(0x7f26e006b630 msgr2=0x7f26e00c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:10.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.269+0000 7f26e5400640 1 -- 172.21.15.28:0/27649055 shutdown_connections 2024-02-21T23:04:10.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.269+0000 7f26e5400640 1 -- 172.21.15.28:0/27649055 wait complete. 2024-02-21T23:04:10.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.271+0000 7f26e5400640 1 Processor -- start 2024-02-21T23:04:10.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.271+0000 7f26e5400640 1 -- start start 2024-02-21T23:04:10.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.272+0000 7f26e5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26e006ee40 0x7f26e01551f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:10.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.272+0000 7f26e5400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f26e0150450 0x7f26e01eb420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:10.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.272+0000 7f26e5400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f26e0156af0 0x7f26e014ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:10.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.272+0000 7f26e5400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e0156f80 con 0x7f26e006ee40 2024-02-21T23:04:10.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.273+0000 7f26e5400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e00c2cb0 con 0x7f26e0156af0 2024-02-21T23:04:10.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.273+0000 7f26e5400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26e0157100 con 0x7f26e0150450 2024-02-21T23:04:10.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.273+0000 7f26dfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26e006ee40 0x7f26e01551f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:10.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.273+0000 7f26dfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26e006ee40 0x7f26e01551f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56324/0 (socket says 172.21.15.28:56324) 2024-02-21T23:04:10.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.273+0000 7f26dfe00640 1 -- 172.21.15.28:0/690920121 learned_addr learned my addr 172.21.15.28:0/690920121 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:10.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.273+0000 7f26d7400640 1 --2- 172.21.15.28:0/690920121 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f26e0150450 0x7f26e01eb420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:10.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.273+0000 7f26e4a00640 1 --2- 172.21.15.28:0/690920121 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f26e0156af0 0x7f26e014ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:10.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.273+0000 7f26dfe00640 1 -- 172.21.15.28:0/690920121 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f26e0150450 msgr2=0x7f26e01eb420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:10.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.273+0000 7f26dfe00640 1 --2- 172.21.15.28:0/690920121 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f26e0150450 0x7f26e01eb420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:10.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.273+0000 7f26dfe00640 1 -- 172.21.15.28:0/690920121 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f26e0156af0 msgr2=0x7f26e014ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:10.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.273+0000 7f26dfe00640 1 --2- 172.21.15.28:0/690920121 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f26e0156af0 0x7f26e014ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:10.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.273+0000 7f26dfe00640 1 -- 172.21.15.28:0/690920121 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e01557d0 con 0x7f26e006ee40 2024-02-21T23:04:10.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.273+0000 7f26e4a00640 1 --2- 172.21.15.28:0/690920121 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f26e0156af0 0x7f26e014ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:04:10.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.274+0000 7f26dfe00640 1 --2- 172.21.15.28:0/690920121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26e006ee40 0x7f26e01551f0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f26d000c790 tx=0x7f26d000cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:10.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.274+0000 7f26dcc00640 1 -- 172.21.15.28:0/690920121 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d0016020 con 0x7f26e006ee40 2024-02-21T23:04:10.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.274+0000 7f26dcc00640 1 -- 172.21.15.28:0/690920121 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26d0006d50 con 0x7f26e006ee40 2024-02-21T23:04:10.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.274+0000 7f26e5400640 1 -- 172.21.15.28:0/690920121 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26e0155ab0 con 0x7f26e006ee40 2024-02-21T23:04:10.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.274+0000 7f26dcc00640 1 -- 172.21.15.28:0/690920121 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d0003bb0 con 0x7f26e006ee40 2024-02-21T23:04:10.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.274+0000 7f26e5400640 1 -- 172.21.15.28:0/690920121 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26e00c5960 con 0x7f26e006ee40 2024-02-21T23:04:10.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.276+0000 7f26e5400640 1 -- 172.21.15.28:0/690920121 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26e004f290 con 0x7f26e006ee40 2024-02-21T23:04:10.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.276+0000 7f26dcc00640 1 -- 172.21.15.28:0/690920121 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f26d000a850 con 0x7f26e006ee40 2024-02-21T23:04:10.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.277+0000 7f26dcc00640 1 --2- 172.21.15.28:0/690920121 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f26bc06b4d0 0x7f26bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:10.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.277+0000 7f26dcc00640 1 -- 172.21.15.28:0/690920121 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f26d008a0e0 con 0x7f26e006ee40 2024-02-21T23:04:10.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.278+0000 7f26d7400640 1 --2- 172.21.15.28:0/690920121 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f26bc06b4d0 0x7f26bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:10.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.279+0000 7f26d7400640 1 --2- 172.21.15.28:0/690920121 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f26bc06b4d0 0x7f26bc06d990 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f26e014d8f0 tx=0x7f26c8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:10.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.282+0000 7f26dcc00640 1 -- 172.21.15.28:0/690920121 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f26d0020020 con 0x7f26e006ee40 2024-02-21T23:04:10.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.494+0000 7f26e5400640 1 -- 172.21.15.28:0/690920121 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f26e00c1f10 con 0x7f26e006ee40 2024-02-21T23:04:10.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.495+0000 7f26dcc00640 1 -- 172.21.15.28:0/690920121 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) v1 ==== 74+0+16895 (secure 0 0 0) 0x7f26d0019310 con 0x7f26e006ee40 2024-02-21T23:04:10.485 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:04:10.485 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":37,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:04:09.035957+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:04:06.977614+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-02-21T23:04:08.074258+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:04:10.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.498+0000 7f26e5400640 1 -- 172.21.15.28:0/690920121 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f26bc06b4d0 msgr2=0x7f26bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:10.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.498+0000 7f26e5400640 1 --2- 172.21.15.28:0/690920121 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f26bc06b4d0 0x7f26bc06d990 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f26e014d8f0 tx=0x7f26c8008040 comp rx=0 tx=0).stop 2024-02-21T23:04:10.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.498+0000 7f26e5400640 1 -- 172.21.15.28:0/690920121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26e006ee40 msgr2=0x7f26e01551f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:10.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.498+0000 7f26e5400640 1 --2- 172.21.15.28:0/690920121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26e006ee40 0x7f26e01551f0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f26d000c790 tx=0x7f26d000cc50 comp rx=0 tx=0).stop 2024-02-21T23:04:10.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.498+0000 7f26e5400640 1 -- 172.21.15.28:0/690920121 shutdown_connections 2024-02-21T23:04:10.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.498+0000 7f26e5400640 1 --2- 172.21.15.28:0/690920121 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f26bc06b4d0 0x7f26bc06d990 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:10.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.498+0000 7f26e5400640 1 --2- 172.21.15.28:0/690920121 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f26e0150450 0x7f26e01eb420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:10.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.498+0000 7f26e5400640 1 --2- 172.21.15.28:0/690920121 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f26e0156af0 0x7f26e014ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:10.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.498+0000 7f26e5400640 1 --2- 172.21.15.28:0/690920121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26e006ee40 0x7f26e01551f0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:10.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.498+0000 7f26e5400640 1 -- 172.21.15.28:0/690920121 >> 172.21.15.28:0/690920121 conn(0x7f26e006b630 msgr2=0x7f26e00a6f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:10.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.498+0000 7f26e5400640 1 -- 172.21.15.28:0/690920121 shutdown_connections 2024-02-21T23:04:10.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.499+0000 7f26e5400640 1 -- 172.21.15.28:0/690920121 wait complete. 2024-02-21T23:04:10.506 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-02-21T23:04:10.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.732+0000 7f7853e00640 1 Processor -- start 2024-02-21T23:04:10.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.732+0000 7f7853e00640 1 -- start start 2024-02-21T23:04:10.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.734+0000 7f7853e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f785413e0c0 0x7f78540c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:10.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.734+0000 7f7853e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f785406dbc0 0x7f785406c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:10.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.734+0000 7f7853e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7854158550 con 0x7f78540c76f0 2024-02-21T23:04:10.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.734+0000 7f7853e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78540c89a0 con 0x7f785406dbc0 2024-02-21T23:04:10.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.734+0000 7f7853e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78541587d0 con 0x7f785413e0c0 2024-02-21T23:04:10.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.734+0000 7f7852000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f785406dbc0 0x7f785406c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:10.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.735+0000 7f7852000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f785406dbc0 0x7f785406c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56330/0 (socket says 172.21.15.28:56330) 2024-02-21T23:04:10.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.735+0000 7f7852000640 1 -- 172.21.15.28:0/3602564578 learned_addr learned my addr 172.21.15.28:0/3602564578 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:10.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.735+0000 7f7852a00640 1 --2- 172.21.15.28:0/3602564578 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f785413e0c0 0x7f78540c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:10.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.735+0000 7f7852000640 1 -- 172.21.15.28:0/3602564578 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f785413e0c0 msgr2=0x7f78540c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:10.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.735+0000 7f7852000640 1 --2- 172.21.15.28:0/3602564578 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f785413e0c0 0x7f78540c71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:10.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.735+0000 7f7852000640 1 -- 172.21.15.28:0/3602564578 >> v1:172.21.15.81:6789/0 conn(0x7f78540c76f0 legacy=0x7f785406c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:10.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.735+0000 7f7852000640 1 -- 172.21.15.28:0/3602564578 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7854158ab0 con 0x7f785406dbc0 2024-02-21T23:04:10.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.736+0000 7f7852000640 1 --2- 172.21.15.28:0/3602564578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f785406dbc0 0x7f785406c280 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f783c009870 tx=0x7f783c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a23af0730e170dd7 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:10.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.736+0000 7f7851600640 1 -- 172.21.15.28:0/3602564578 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f783c0317c0 con 0x7f785406dbc0 2024-02-21T23:04:10.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.737+0000 7f7851600640 1 -- 172.21.15.28:0/3602564578 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f783c037070 con 0x7f785406dbc0 2024-02-21T23:04:10.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.737+0000 7f7851600640 1 -- 172.21.15.28:0/3602564578 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f783c031b80 con 0x7f785406dbc0 2024-02-21T23:04:10.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.737+0000 7f7853e00640 1 -- 172.21.15.28:0/3602564578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f785406dbc0 msgr2=0x7f785406c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:10.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.737+0000 7f7853e00640 1 --2- 172.21.15.28:0/3602564578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f785406dbc0 0x7f785406c280 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f783c009870 tx=0x7f783c02b860 comp rx=0 tx=0).stop 2024-02-21T23:04:10.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.737+0000 7f7853e00640 1 -- 172.21.15.28:0/3602564578 shutdown_connections 2024-02-21T23:04:10.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.737+0000 7f7853e00640 1 --2- 172.21.15.28:0/3602564578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f785406dbc0 0x7f785406c280 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:10.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.737+0000 7f7853e00640 1 --2- 172.21.15.28:0/3602564578 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f785413e0c0 0x7f78540c71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:10.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.737+0000 7f7853e00640 1 -- 172.21.15.28:0/3602564578 >> 172.21.15.28:0/3602564578 conn(0x7f7854073110 msgr2=0x7f78540ac490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:10.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.737+0000 7f7853e00640 1 -- 172.21.15.28:0/3602564578 shutdown_connections 2024-02-21T23:04:10.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.737+0000 7f7853e00640 1 -- 172.21.15.28:0/3602564578 wait complete. 2024-02-21T23:04:10.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.739+0000 7f7853e00640 1 Processor -- start 2024-02-21T23:04:10.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.739+0000 7f7853e00640 1 -- start start 2024-02-21T23:04:10.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.740+0000 7f7853e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f785406dbc0 0x7f78540b0040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:10.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.740+0000 7f7853e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78540c76f0 0x7f7854153830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:10.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.740+0000 7f7853e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f785413e0c0 0x7f7854157620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:10.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.740+0000 7f7853e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f785406ced0 con 0x7f78540c76f0 2024-02-21T23:04:10.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.740+0000 7f7853e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7854159070 con 0x7f785413e0c0 2024-02-21T23:04:10.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.740+0000 7f7853e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f785413e520 con 0x7f785406dbc0 2024-02-21T23:04:10.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.740+0000 7f7852000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78540c76f0 0x7f7854153830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:10.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.741+0000 7f7852000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78540c76f0 0x7f7854153830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56332/0 (socket says 172.21.15.28:56332) 2024-02-21T23:04:10.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.741+0000 7f7852000640 1 -- 172.21.15.28:0/1285094496 learned_addr learned my addr 172.21.15.28:0/1285094496 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:10.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.741+0000 7f7852a00640 1 --2- 172.21.15.28:0/1285094496 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f785406dbc0 0x7f78540b0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:10.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.741+0000 7f7853400640 1 --2- 172.21.15.28:0/1285094496 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f785413e0c0 0x7f7854157620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:10.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.741+0000 7f7852000640 1 -- 172.21.15.28:0/1285094496 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f785406dbc0 msgr2=0x7f78540b0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:10.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.741+0000 7f7852000640 1 --2- 172.21.15.28:0/1285094496 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f785406dbc0 0x7f78540b0040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:10.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.741+0000 7f7852000640 1 -- 172.21.15.28:0/1285094496 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f785413e0c0 msgr2=0x7f7854157620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:10.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.741+0000 7f7852000640 1 --2- 172.21.15.28:0/1285094496 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f785413e0c0 0x7f7854157620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:10.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.741+0000 7f7852000640 1 -- 172.21.15.28:0/1285094496 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7854153ea0 con 0x7f78540c76f0 2024-02-21T23:04:10.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.741+0000 7f7852000640 1 --2- 172.21.15.28:0/1285094496 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78540c76f0 0x7f7854153830 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f783c02b8a0 tx=0x7f783c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:10.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.742+0000 7f784aa00640 1 -- 172.21.15.28:0/1285094496 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f783c032650 con 0x7f78540c76f0 2024-02-21T23:04:10.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.742+0000 7f784aa00640 1 -- 172.21.15.28:0/1285094496 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f783c032810 con 0x7f78540c76f0 2024-02-21T23:04:10.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.742+0000 7f784aa00640 1 -- 172.21.15.28:0/1285094496 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f783c032c00 con 0x7f78540c76f0 2024-02-21T23:04:10.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.742+0000 7f7853e00640 1 -- 172.21.15.28:0/1285094496 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7854153090 con 0x7f78540c76f0 2024-02-21T23:04:10.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.742+0000 7f7853e00640 1 -- 172.21.15.28:0/1285094496 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7854158250 con 0x7f78540c76f0 2024-02-21T23:04:10.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.743+0000 7f7853e00640 1 -- 172.21.15.28:0/1285094496 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78541eac30 con 0x7f78540c76f0 2024-02-21T23:04:10.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.744+0000 7f784aa00640 1 -- 172.21.15.28:0/1285094496 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f783c04c060 con 0x7f78540c76f0 2024-02-21T23:04:10.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.745+0000 7f784aa00640 1 --2- 172.21.15.28:0/1285094496 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f783006b4d0 0x7f783006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:10.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.745+0000 7f784aa00640 1 -- 172.21.15.28:0/1285094496 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f783c0abe30 con 0x7f78540c76f0 2024-02-21T23:04:10.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.746+0000 7f7852a00640 1 --2- 172.21.15.28:0/1285094496 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f783006b4d0 0x7f783006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:10.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.747+0000 7f7852a00640 1 --2- 172.21.15.28:0/1285094496 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f783006b4d0 0x7f783006d990 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f7844006fd0 tx=0x7f7844008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:10.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.754+0000 7f784aa00640 1 -- 172.21.15.28:0/1285094496 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f783c040070 con 0x7f78540c76f0 2024-02-21T23:04:10.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:10.996+0000 7f7853e00640 1 -- 172.21.15.28:0/1285094496 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f785404f290 con 0x7f78540c76f0 2024-02-21T23:04:11.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:11.625+0000 7f784aa00640 1 -- 172.21.15.28:0/1285094496 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) v1 ==== 118+0+0 (secure 0 0 0) 0x7f7854158550 con 0x7f78540c76f0 2024-02-21T23:04:11.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:11.627+0000 7f7853e00640 1 -- 172.21.15.28:0/1285094496 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f783006b4d0 msgr2=0x7f783006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:11.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:11.627+0000 7f7853e00640 1 --2- 172.21.15.28:0/1285094496 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f783006b4d0 0x7f783006d990 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f7844006fd0 tx=0x7f7844008040 comp rx=0 tx=0).stop 2024-02-21T23:04:11.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:11.627+0000 7f7853e00640 1 -- 172.21.15.28:0/1285094496 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78540c76f0 msgr2=0x7f7854153830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:11.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:11.627+0000 7f7853e00640 1 --2- 172.21.15.28:0/1285094496 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78540c76f0 0x7f7854153830 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f783c02b8a0 tx=0x7f783c0046c0 comp rx=0 tx=0).stop 2024-02-21T23:04:11.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:11.628+0000 7f7853e00640 1 -- 172.21.15.28:0/1285094496 shutdown_connections 2024-02-21T23:04:11.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:11.628+0000 7f7853e00640 1 --2- 172.21.15.28:0/1285094496 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f783006b4d0 0x7f783006d990 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:11.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:11.628+0000 7f7853e00640 1 --2- 172.21.15.28:0/1285094496 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f785413e0c0 0x7f7854157620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:11.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:11.628+0000 7f7853e00640 1 --2- 172.21.15.28:0/1285094496 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78540c76f0 0x7f7854153830 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:11.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:11.628+0000 7f7853e00640 1 --2- 172.21.15.28:0/1285094496 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f785406dbc0 0x7f78540b0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:11.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:11.628+0000 7f7853e00640 1 -- 172.21.15.28:0/1285094496 >> 172.21.15.28:0/1285094496 conn(0x7f7854073110 msgr2=0x7f78540c5020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:11.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:11.628+0000 7f7853e00640 1 -- 172.21.15.28:0/1285094496 shutdown_connections 2024-02-21T23:04:11.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:11.628+0000 7f7853e00640 1 -- 172.21.15.28:0/1285094496 wait complete. 2024-02-21T23:04:11.634 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-02-21T23:04:11.634 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-02-21T23:04:11.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:11.864+0000 7fb4e1c00640 1 Processor -- start 2024-02-21T23:04:11.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:11.865+0000 7fb4e1c00640 1 -- start start 2024-02-21T23:04:11.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:11.868+0000 7fb4e1c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb4dc06dbc0 0x7fb4dc06c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:11.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:11.868+0000 7fb4e1c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4dc158550 con 0x7fb4dc13e0c0 2024-02-21T23:04:11.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:11.868+0000 7fb4e1c00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4dc1587d0 con 0x7fb4dc13e520 2024-02-21T23:04:11.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:11.868+0000 7fb4e1c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4dc0c89a0 con 0x7fb4dc06dbc0 2024-02-21T23:04:11.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:11.868+0000 7fb4dbe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb4dc13e0c0 0x7fb4dc0c71b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:56648/0 (socket says 172.21.15.28:56648) 2024-02-21T23:04:11.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:11.868+0000 7fb4dbe00640 1 -- 172.21.15.28:0/3277170687 learned_addr learned my addr 172.21.15.28:0/3277170687 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:04:11.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:11.868+0000 7fb4db400640 1 --2- 172.21.15.28:0/3277170687 >> v2:172.21.15.28:3300/0 conn(0x7fb4dc06dbc0 0x7fb4dc06c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:11.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:11.868+0000 7fb4db400640 1 -- 172.21.15.28:0/3277170687 >> v1:172.21.15.81:6789/0 conn(0x7fb4dc13e520 legacy=0x7fb4dc06c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:11.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:11.868+0000 7fb4db400640 1 -- 172.21.15.28:0/3277170687 >> v1:172.21.15.28:6789/0 conn(0x7fb4dc13e0c0 legacy=0x7fb4dc0c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:11.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:11.869+0000 7fb4daa00640 1 -- 172.21.15.28:0/3277170687 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3379431090 0 0) 0x7fb4d4001490 con 0x7fb4dc13e0c0 2024-02-21T23:04:11.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:11.869+0000 7fb4db400640 1 -- 172.21.15.28:0/3277170687 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4dc158ab0 con 0x7fb4dc06dbc0 2024-02-21T23:04:11.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:11.870+0000 7fb4db400640 1 --2- 172.21.15.28:0/3277170687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4dc06dbc0 0x7fb4dc06c280 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fb4c40097a0 tx=0x7fb4c402b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e448de79c6ec37eb server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:11.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:11.870+0000 7fb4daa00640 1 -- 172.21.15.28:0/3277170687 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c4038070 con 0x7fb4dc06dbc0 2024-02-21T23:04:11.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:11.870+0000 7fb4daa00640 1 -- 172.21.15.28:0/3277170687 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4c4031700 con 0x7fb4dc06dbc0 2024-02-21T23:04:11.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:11.870+0000 7fb4daa00640 1 -- 172.21.15.28:0/3277170687 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c4031ad0 con 0x7fb4dc06dbc0 2024-02-21T23:04:11.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:11.871+0000 7fb4e1c00640 1 -- 172.21.15.28:0/3277170687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4dc06dbc0 msgr2=0x7fb4dc06c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:11.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:11.871+0000 7fb4e1c00640 1 --2- 172.21.15.28:0/3277170687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4dc06dbc0 0x7fb4dc06c280 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fb4c40097a0 tx=0x7fb4c402b7a0 comp rx=0 tx=0).stop 2024-02-21T23:04:11.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:11.871+0000 7fb4e1c00640 1 -- 172.21.15.28:0/3277170687 shutdown_connections 2024-02-21T23:04:11.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:11.871+0000 7fb4e1c00640 1 --2- 172.21.15.28:0/3277170687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4dc06dbc0 0x7fb4dc06c280 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:11.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:11.871+0000 7fb4e1c00640 1 -- 172.21.15.28:0/3277170687 >> 172.21.15.28:0/3277170687 conn(0x7fb4dc073110 msgr2=0x7fb4dc0ac490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:11.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:11.871+0000 7fb4e1c00640 1 -- 172.21.15.28:0/3277170687 shutdown_connections 2024-02-21T23:04:11.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:11.871+0000 7fb4e1c00640 1 -- 172.21.15.28:0/3277170687 wait complete. 2024-02-21T23:04:11.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:11.873+0000 7fb4e1c00640 1 Processor -- start 2024-02-21T23:04:11.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:11.873+0000 7fb4e1c00640 1 -- start start 2024-02-21T23:04:11.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:11.874+0000 7fb4e1c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4dc06dbc0 0x7fb4dc1578a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:11.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:11.874+0000 7fb4e1c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb4dc13e0c0 0x7fb4dc14f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:11.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:11.874+0000 7fb4e1c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb4dc13e520 0x7fb4dc14fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:11.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:11.874+0000 7fb4e1c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4dc06ced0 con 0x7fb4dc06dbc0 2024-02-21T23:04:11.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:11.874+0000 7fb4e1c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4dc159070 con 0x7fb4dc13e520 2024-02-21T23:04:11.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:11.874+0000 7fb4e1c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4dc0c78c0 con 0x7fb4dc13e0c0 2024-02-21T23:04:11.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:11.874+0000 7fb4dbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4dc06dbc0 0x7fb4dc1578a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:11.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:11.874+0000 7fb4dbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4dc06dbc0 0x7fb4dc1578a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56344/0 (socket says 172.21.15.28:56344) 2024-02-21T23:04:11.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:11.874+0000 7fb4dbe00640 1 -- 172.21.15.28:0/2531867782 learned_addr learned my addr 172.21.15.28:0/2531867782 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:11.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:11.874+0000 7fb4db400640 1 --2- 172.21.15.28:0/2531867782 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb4dc13e0c0 0x7fb4dc14f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:11.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:11.874+0000 7fb4e1200640 1 --2- 172.21.15.28:0/2531867782 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb4dc13e520 0x7fb4dc14fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:11.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:11.875+0000 7fb4dbe00640 1 -- 172.21.15.28:0/2531867782 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb4dc13e0c0 msgr2=0x7fb4dc14f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:11.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:11.875+0000 7fb4dbe00640 1 --2- 172.21.15.28:0/2531867782 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb4dc13e0c0 0x7fb4dc14f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:11.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:11.875+0000 7fb4dbe00640 1 -- 172.21.15.28:0/2531867782 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb4dc13e520 msgr2=0x7fb4dc14fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:11.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:11.875+0000 7fb4dbe00640 1 --2- 172.21.15.28:0/2531867782 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb4dc13e520 0x7fb4dc14fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:11.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:11.875+0000 7fb4dbe00640 1 -- 172.21.15.28:0/2531867782 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4dc1e0cc0 con 0x7fb4dc06dbc0 2024-02-21T23:04:11.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:11.875+0000 7fb4db400640 1 --2- 172.21.15.28:0/2531867782 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb4dc13e0c0 0x7fb4dc14f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:04:11.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:11.875+0000 7fb4dbe00640 1 --2- 172.21.15.28:0/2531867782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4dc06dbc0 0x7fb4dc1578a0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fb4d400c7c0 tx=0x7fb4d400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:11.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:11.875+0000 7fb4d3e00640 1 -- 172.21.15.28:0/2531867782 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4d4018070 con 0x7fb4dc06dbc0 2024-02-21T23:04:11.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:11.875+0000 7fb4d3e00640 1 -- 172.21.15.28:0/2531867782 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4d400a640 con 0x7fb4dc06dbc0 2024-02-21T23:04:11.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:11.875+0000 7fb4e1c00640 1 -- 172.21.15.28:0/2531867782 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4dc1ee090 con 0x7fb4dc06dbc0 2024-02-21T23:04:11.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:11.875+0000 7fb4d3e00640 1 -- 172.21.15.28:0/2531867782 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4d400aa30 con 0x7fb4dc06dbc0 2024-02-21T23:04:11.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:11.876+0000 7fb4e1c00640 1 -- 172.21.15.28:0/2531867782 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4dc1ee650 con 0x7fb4dc06dbc0 2024-02-21T23:04:11.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:11.877+0000 7fb4e1c00640 1 -- 172.21.15.28:0/2531867782 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4dc04f290 con 0x7fb4dc06dbc0 2024-02-21T23:04:11.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:11.878+0000 7fb4d3e00640 1 -- 172.21.15.28:0/2531867782 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb4d401b070 con 0x7fb4dc06dbc0 2024-02-21T23:04:11.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:11.879+0000 7fb4d3e00640 1 --2- 172.21.15.28:0/2531867782 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb4b806b4d0 0x7fb4b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:11.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:11.879+0000 7fb4d3e00640 1 -- 172.21.15.28:0/2531867782 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb4d40899b0 con 0x7fb4dc06dbc0 2024-02-21T23:04:11.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:11.879+0000 7fb4db400640 1 --2- 172.21.15.28:0/2531867782 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb4b806b4d0 0x7fb4b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:11.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:11.880+0000 7fb4db400640 1 --2- 172.21.15.28:0/2531867782 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb4b806b4d0 0x7fb4b806d990 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb4c4035000 tx=0x7fb4c4008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:11.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:11.883+0000 7fb4d3e00640 1 -- 172.21.15.28:0/2531867782 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb4dc158550 con 0x7fb4dc06dbc0 2024-02-21T23:04:12.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.122+0000 7fb4e1c00640 1 -- 172.21.15.28:0/2531867782 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fb4dc06d9a0 con 0x7fb4dc06dbc0 2024-02-21T23:04:12.346 INFO:tasks.ceph.mds.a.smithi028.stderr:2024-02-21T23:04:12.357+0000 7f238f600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-21T23:04:12.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.628+0000 7fb4d3e00640 1 -- 172.21.15.28:0/2531867782 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v18) v1 ==== 122+0+0 (secure 0 0 0) 0x7fb4d4058690 con 0x7fb4dc06dbc0 2024-02-21T23:04:12.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.631+0000 7fb4e1c00640 1 -- 172.21.15.28:0/2531867782 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb4b806b4d0 msgr2=0x7fb4b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:12.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.631+0000 7fb4e1c00640 1 --2- 172.21.15.28:0/2531867782 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb4b806b4d0 0x7fb4b806d990 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb4c4035000 tx=0x7fb4c4008030 comp rx=0 tx=0).stop 2024-02-21T23:04:12.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.631+0000 7fb4e1c00640 1 -- 172.21.15.28:0/2531867782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4dc06dbc0 msgr2=0x7fb4dc1578a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:12.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.631+0000 7fb4e1c00640 1 --2- 172.21.15.28:0/2531867782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4dc06dbc0 0x7fb4dc1578a0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fb4d400c7c0 tx=0x7fb4d400cc80 comp rx=0 tx=0).stop 2024-02-21T23:04:12.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.632+0000 7fb4e1c00640 1 -- 172.21.15.28:0/2531867782 shutdown_connections 2024-02-21T23:04:12.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.632+0000 7fb4e1c00640 1 --2- 172.21.15.28:0/2531867782 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb4b806b4d0 0x7fb4b806d990 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:12.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.632+0000 7fb4e1c00640 1 --2- 172.21.15.28:0/2531867782 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb4dc13e0c0 0x7fb4dc14f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:12.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.632+0000 7fb4e1c00640 1 --2- 172.21.15.28:0/2531867782 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb4dc13e520 0x7fb4dc14fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:12.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.632+0000 7fb4e1c00640 1 --2- 172.21.15.28:0/2531867782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4dc06dbc0 0x7fb4dc1578a0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:12.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.632+0000 7fb4e1c00640 1 -- 172.21.15.28:0/2531867782 >> 172.21.15.28:0/2531867782 conn(0x7fb4dc073110 msgr2=0x7fb4dc0c5020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:12.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.632+0000 7fb4e1c00640 1 -- 172.21.15.28:0/2531867782 shutdown_connections 2024-02-21T23:04:12.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.632+0000 7fb4e1c00640 1 -- 172.21.15.28:0/2531867782 wait complete. 2024-02-21T23:04:12.650 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-02-21T23:04:12.650 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:04:12.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.876+0000 7fde95400640 1 Processor -- start 2024-02-21T23:04:12.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.876+0000 7fde95400640 1 -- start start 2024-02-21T23:04:12.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.877+0000 7fde95400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fde900c11f0 0x7fde90153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:12.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.877+0000 7fde95400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fde9015b6d0 0x7fde9015dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:12.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.877+0000 7fde95400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde900c1a30 con 0x7fde900c15d0 2024-02-21T23:04:12.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.877+0000 7fde95400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde900a5ec0 con 0x7fde9015b6d0 2024-02-21T23:04:12.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.877+0000 7fde95400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde90152e70 con 0x7fde900c11f0 2024-02-21T23:04:12.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.878+0000 7fde87e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fde900c15d0 0x7fde9006dce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:56658/0 (socket says 172.21.15.28:56658) 2024-02-21T23:04:12.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.878+0000 7fde87e00640 1 -- 172.21.15.28:0/3295392712 learned_addr learned my addr 172.21.15.28:0/3295392712 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:04:12.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.878+0000 7fde8f200640 1 --2- 172.21.15.28:0/3295392712 >> v2:172.21.15.81:3300/0 conn(0x7fde9015b6d0 0x7fde9015dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:12.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.878+0000 7fde8e800640 1 --2- 172.21.15.28:0/3295392712 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fde900c11f0 0x7fde90153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:12.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.878+0000 7fde8de00640 1 -- 172.21.15.28:0/3295392712 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2441466285 0 0) 0x7fde900c1a30 con 0x7fde900c15d0 2024-02-21T23:04:12.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.878+0000 7fde8de00640 1 -- 172.21.15.28:0/3295392712 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde74003520 con 0x7fde900c15d0 2024-02-21T23:04:12.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.879+0000 7fde8f200640 1 -- 172.21.15.28:0/3295392712 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fde900c11f0 msgr2=0x7fde90153950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:12.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.879+0000 7fde8f200640 1 --2- 172.21.15.28:0/3295392712 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fde900c11f0 0x7fde90153950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:12.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.879+0000 7fde8f200640 1 -- 172.21.15.28:0/3295392712 >> v1:172.21.15.28:6789/0 conn(0x7fde900c15d0 legacy=0x7fde9006dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:12.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.879+0000 7fde8de00640 1 -- 172.21.15.28:0/3295392712 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1852439822 0 0) 0x7fde74003520 con 0x7fde900c15d0 2024-02-21T23:04:12.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.879+0000 7fde8f200640 1 -- 172.21.15.28:0/3295392712 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde90153150 con 0x7fde9015b6d0 2024-02-21T23:04:12.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.880+0000 7fde8f200640 1 --2- 172.21.15.28:0/3295392712 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde9015b6d0 0x7fde9015dac0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fde88009d90 tx=0x7fde8802bc50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a93dc52b87c2fbdd server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:12.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.881+0000 7fde8de00640 1 -- 172.21.15.28:0/3295392712 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde88039070 con 0x7fde9015b6d0 2024-02-21T23:04:12.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.881+0000 7fde8de00640 1 -- 172.21.15.28:0/3295392712 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde8802c780 con 0x7fde9015b6d0 2024-02-21T23:04:12.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.881+0000 7fde8de00640 1 -- 172.21.15.28:0/3295392712 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde78001670 con 0x7fde9015b6d0 2024-02-21T23:04:12.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.881+0000 7fde95400640 1 -- 172.21.15.28:0/3295392712 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde9015b6d0 msgr2=0x7fde9015dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:12.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.881+0000 7fde95400640 1 --2- 172.21.15.28:0/3295392712 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde9015b6d0 0x7fde9015dac0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fde88009d90 tx=0x7fde8802bc50 comp rx=0 tx=0).stop 2024-02-21T23:04:12.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.882+0000 7fde95400640 1 -- 172.21.15.28:0/3295392712 shutdown_connections 2024-02-21T23:04:12.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.882+0000 7fde95400640 1 --2- 172.21.15.28:0/3295392712 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde9015b6d0 0x7fde9015dac0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:12.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.882+0000 7fde95400640 1 --2- 172.21.15.28:0/3295392712 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fde900c11f0 0x7fde90153950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:12.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.882+0000 7fde95400640 1 -- 172.21.15.28:0/3295392712 >> 172.21.15.28:0/3295392712 conn(0x7fde900c67f0 msgr2=0x7fde900c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:12.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.882+0000 7fde95400640 1 -- 172.21.15.28:0/3295392712 shutdown_connections 2024-02-21T23:04:12.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.882+0000 7fde95400640 1 -- 172.21.15.28:0/3295392712 wait complete. 2024-02-21T23:04:12.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.884+0000 7fde95400640 1 Processor -- start 2024-02-21T23:04:12.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.884+0000 7fde95400640 1 -- start start 2024-02-21T23:04:12.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.886+0000 7fde95400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fde900c11f0 0x7fde900bfe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:12.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.886+0000 7fde95400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde900c15d0 0x7fde90072b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:12.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.886+0000 7fde95400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde9015b6d0 0x7fde9013f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:12.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.886+0000 7fde95400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde900c4c00 con 0x7fde9015b6d0 2024-02-21T23:04:12.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.886+0000 7fde95400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde9014d1c0 con 0x7fde900c15d0 2024-02-21T23:04:12.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.886+0000 7fde95400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde900a6110 con 0x7fde900c11f0 2024-02-21T23:04:12.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.886+0000 7fde8f200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde9015b6d0 0x7fde9013f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:12.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.886+0000 7fde8f200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde9015b6d0 0x7fde9013f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56352/0 (socket says 172.21.15.28:56352) 2024-02-21T23:04:12.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.886+0000 7fde8f200640 1 -- 172.21.15.28:0/1381064728 learned_addr learned my addr 172.21.15.28:0/1381064728 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:12.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.886+0000 7fde87e00640 1 --2- 172.21.15.28:0/1381064728 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde900c15d0 0x7fde90072b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:12.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.886+0000 7fde8e800640 1 --2- 172.21.15.28:0/1381064728 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fde900c11f0 0x7fde900bfe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:12.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.887+0000 7fde8f200640 1 -- 172.21.15.28:0/1381064728 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fde900c11f0 msgr2=0x7fde900bfe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:12.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.887+0000 7fde8f200640 1 --2- 172.21.15.28:0/1381064728 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fde900c11f0 0x7fde900bfe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:12.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.887+0000 7fde8f200640 1 -- 172.21.15.28:0/1381064728 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde900c15d0 msgr2=0x7fde90072b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:12.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.887+0000 7fde8f200640 1 --2- 172.21.15.28:0/1381064728 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde900c15d0 0x7fde90072b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:12.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.887+0000 7fde8f200640 1 -- 172.21.15.28:0/1381064728 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde90149ac0 con 0x7fde9015b6d0 2024-02-21T23:04:12.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.887+0000 7fde87e00640 1 --2- 172.21.15.28:0/1381064728 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde900c15d0 0x7fde90072b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:04:12.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.887+0000 7fde8f200640 1 --2- 172.21.15.28:0/1381064728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde9015b6d0 0x7fde9013f710 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fde88009d90 tx=0x7fde88004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:12.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.887+0000 7fde86a00640 1 -- 172.21.15.28:0/1381064728 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde88039050 con 0x7fde9015b6d0 2024-02-21T23:04:12.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.887+0000 7fde86a00640 1 -- 172.21.15.28:0/1381064728 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde8803a040 con 0x7fde9015b6d0 2024-02-21T23:04:12.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.887+0000 7fde95400640 1 -- 172.21.15.28:0/1381064728 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde901eb520 con 0x7fde9015b6d0 2024-02-21T23:04:12.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.887+0000 7fde86a00640 1 -- 172.21.15.28:0/1381064728 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde88004450 con 0x7fde9015b6d0 2024-02-21T23:04:12.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.888+0000 7fde95400640 1 -- 172.21.15.28:0/1381064728 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde901ebae0 con 0x7fde9015b6d0 2024-02-21T23:04:12.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.889+0000 7fde95400640 1 -- 172.21.15.28:0/1381064728 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde900c0d90 con 0x7fde9015b6d0 2024-02-21T23:04:12.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.890+0000 7fde86a00640 1 -- 172.21.15.28:0/1381064728 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fde88047020 con 0x7fde9015b6d0 2024-02-21T23:04:12.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.891+0000 7fde86a00640 1 --2- 172.21.15.28:0/1381064728 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fde6c06b4d0 0x7fde6c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:12.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.891+0000 7fde86a00640 1 -- 172.21.15.28:0/1381064728 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fde880ac190 con 0x7fde9015b6d0 2024-02-21T23:04:12.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.892+0000 7fde8e800640 1 --2- 172.21.15.28:0/1381064728 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fde6c06b4d0 0x7fde6c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:12.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.893+0000 7fde8e800640 1 --2- 172.21.15.28:0/1381064728 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fde6c06b4d0 0x7fde6c06d990 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fde900728f0 tx=0x7fde80008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:12.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:12.901+0000 7fde86a00640 1 -- 172.21.15.28:0/1381064728 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fde900c0d90 con 0x7fde9015b6d0 2024-02-21T23:04:13.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.138+0000 7fde95400640 1 -- 172.21.15.28:0/1381064728 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde9014a7b0 con 0x7fde9015b6d0 2024-02-21T23:04:13.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.139+0000 7fde86a00640 1 -- 172.21.15.28:0/1381064728 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+4938 (secure 0 0 0) 0x7fde9014a7b0 con 0x7fde9015b6d0 2024-02-21T23:04:13.127 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:04:13.128 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":18,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4778,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/2827878900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2827878900},{"type":"v1","addr":"172.21.15.28:6834","nonce":2827878900}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":4978,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/3612644344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":3612644344},{"type":"v1","addr":"172.21.15.81:6836","nonce":3612644344}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4981,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3623937361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":3623937361},{"type":"v1","addr":"172.21.15.81:6837","nonce":3623937361}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:04:09.036302+0000","modified":"2024-02-21T23:04:12.123968+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4987},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4987":{"gid":4987,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6835/1213702315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":1213702315},{"type":"v1","addr":"172.21.15.28:6835","nonce":1213702315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-02-21T23:04:13.128 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 18 2024-02-21T23:04:13.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.141+0000 7fde95400640 1 -- 172.21.15.28:0/1381064728 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fde6c06b4d0 msgr2=0x7fde6c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:13.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.141+0000 7fde95400640 1 --2- 172.21.15.28:0/1381064728 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fde6c06b4d0 0x7fde6c06d990 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fde900728f0 tx=0x7fde80008040 comp rx=0 tx=0).stop 2024-02-21T23:04:13.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.141+0000 7fde95400640 1 -- 172.21.15.28:0/1381064728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde9015b6d0 msgr2=0x7fde9013f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:13.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.141+0000 7fde95400640 1 --2- 172.21.15.28:0/1381064728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde9015b6d0 0x7fde9013f710 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fde88009d90 tx=0x7fde88004a40 comp rx=0 tx=0).stop 2024-02-21T23:04:13.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.142+0000 7fde95400640 1 -- 172.21.15.28:0/1381064728 shutdown_connections 2024-02-21T23:04:13.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.142+0000 7fde95400640 1 --2- 172.21.15.28:0/1381064728 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fde6c06b4d0 0x7fde6c06d990 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:13.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.142+0000 7fde95400640 1 --2- 172.21.15.28:0/1381064728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde9015b6d0 0x7fde9013f710 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:13.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.142+0000 7fde95400640 1 --2- 172.21.15.28:0/1381064728 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde900c15d0 0x7fde90072b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:13.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.142+0000 7fde95400640 1 --2- 172.21.15.28:0/1381064728 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fde900c11f0 0x7fde900bfe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:13.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.142+0000 7fde95400640 1 -- 172.21.15.28:0/1381064728 >> 172.21.15.28:0/1381064728 conn(0x7fde900c67f0 msgr2=0x7fde900c1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:13.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.142+0000 7fde95400640 1 -- 172.21.15.28:0/1381064728 shutdown_connections 2024-02-21T23:04:13.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.142+0000 7fde95400640 1 -- 172.21.15.28:0/1381064728 wait complete. 2024-02-21T23:04:13.152 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:04:13.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.364+0000 7f2c9ca00640 1 Processor -- start 2024-02-21T23:04:13.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.365+0000 7f2c9ca00640 1 -- start start 2024-02-21T23:04:13.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.366+0000 7f2c9ca00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2c9806e3d0 0x7f2c9814b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:13.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.366+0000 7f2c9ca00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c980a8b40 con 0x7f2c9806dff0 2024-02-21T23:04:13.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.366+0000 7f2c9ca00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c9806e7d0 con 0x7f2c980c0fe0 2024-02-21T23:04:13.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.366+0000 7f2c9ca00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c980a8e00 con 0x7f2c9806e3d0 2024-02-21T23:04:13.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.366+0000 7f2c97e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2c9806e3d0 0x7f2c9814b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:13.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.366+0000 7f2c97400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f2c9806dff0 0x7f2c98071350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:56664/0 (socket says 172.21.15.28:56664) 2024-02-21T23:04:13.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.366+0000 7f2c97400640 1 -- 172.21.15.28:0/3969834613 learned_addr learned my addr 172.21.15.28:0/3969834613 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:04:13.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.367+0000 7f2c97e00640 1 -- 172.21.15.28:0/3969834613 >> v1:172.21.15.81:6789/0 conn(0x7f2c980c0fe0 legacy=0x7f2c98158580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:13.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.367+0000 7f2c97e00640 1 -- 172.21.15.28:0/3969834613 >> v1:172.21.15.28:6789/0 conn(0x7f2c9806dff0 legacy=0x7f2c98071350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:13.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.367+0000 7f2c97e00640 1 -- 172.21.15.28:0/3969834613 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c980a90e0 con 0x7f2c9806e3d0 2024-02-21T23:04:13.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.368+0000 7f2c97e00640 1 --2- 172.21.15.28:0/3969834613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c9806e3d0 0x7f2c9814b2b0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f2c90009d90 tx=0x7f2c9002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=946060493523d57d server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:13.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.368+0000 7f2c96a00640 1 -- 172.21.15.28:0/3969834613 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c90039070 con 0x7f2c9806e3d0 2024-02-21T23:04:13.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.368+0000 7f2c96a00640 1 -- 172.21.15.28:0/3969834613 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c9002cc60 con 0x7f2c9806e3d0 2024-02-21T23:04:13.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.368+0000 7f2c96a00640 1 -- 172.21.15.28:0/3969834613 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c90034660 con 0x7f2c9806e3d0 2024-02-21T23:04:13.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.369+0000 7f2c9ca00640 1 -- 172.21.15.28:0/3969834613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c9806e3d0 msgr2=0x7f2c9814b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:13.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.369+0000 7f2c9ca00640 1 --2- 172.21.15.28:0/3969834613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c9806e3d0 0x7f2c9814b2b0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f2c90009d90 tx=0x7f2c9002beb0 comp rx=0 tx=0).stop 2024-02-21T23:04:13.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.369+0000 7f2c9ca00640 1 -- 172.21.15.28:0/3969834613 shutdown_connections 2024-02-21T23:04:13.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.369+0000 7f2c9ca00640 1 --2- 172.21.15.28:0/3969834613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c9806e3d0 0x7f2c9814b2b0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:13.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.369+0000 7f2c9ca00640 1 -- 172.21.15.28:0/3969834613 >> 172.21.15.28:0/3969834613 conn(0x7f2c9806b630 msgr2=0x7f2c980c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:13.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.369+0000 7f2c9ca00640 1 -- 172.21.15.28:0/3969834613 shutdown_connections 2024-02-21T23:04:13.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.369+0000 7f2c9ca00640 1 -- 172.21.15.28:0/3969834613 wait complete. 2024-02-21T23:04:13.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.370+0000 7f2c9ca00640 1 Processor -- start 2024-02-21T23:04:13.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.370+0000 7f2c9ca00640 1 -- start start 2024-02-21T23:04:13.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.371+0000 7f2c9ca00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2c9806dff0 0x7f2c981ee500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:13.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.371+0000 7f2c9ca00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2c9806e3d0 0x7f2c98154b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:13.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.371+0000 7f2c9ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c980c0fe0 0x7f2c98155070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:13.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.371+0000 7f2c9ca00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c98073520 con 0x7f2c980c0fe0 2024-02-21T23:04:13.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.371+0000 7f2c9ca00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c980731b0 con 0x7f2c9806e3d0 2024-02-21T23:04:13.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.371+0000 7f2c9ca00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c9814b7f0 con 0x7f2c9806dff0 2024-02-21T23:04:13.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.372+0000 7f2c97e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c980c0fe0 0x7f2c98155070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:13.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.372+0000 7f2c97e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c980c0fe0 0x7f2c98155070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56370/0 (socket says 172.21.15.28:56370) 2024-02-21T23:04:13.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.372+0000 7f2c97e00640 1 -- 172.21.15.28:0/46916641 learned_addr learned my addr 172.21.15.28:0/46916641 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:13.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.372+0000 7f2c8fe00640 1 --2- 172.21.15.28:0/46916641 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2c9806e3d0 0x7f2c98154b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:13.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.372+0000 7f2c97400640 1 --2- 172.21.15.28:0/46916641 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2c9806dff0 0x7f2c981ee500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:13.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.372+0000 7f2c97e00640 1 -- 172.21.15.28:0/46916641 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2c9806dff0 msgr2=0x7f2c981ee500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:13.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.372+0000 7f2c97e00640 1 --2- 172.21.15.28:0/46916641 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2c9806dff0 0x7f2c981ee500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:13.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.372+0000 7f2c97e00640 1 -- 172.21.15.28:0/46916641 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2c9806e3d0 msgr2=0x7f2c98154b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:13.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.372+0000 7f2c97e00640 1 --2- 172.21.15.28:0/46916641 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2c9806e3d0 0x7f2c98154b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:13.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.372+0000 7f2c97e00640 1 -- 172.21.15.28:0/46916641 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c981525a0 con 0x7f2c980c0fe0 2024-02-21T23:04:13.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.372+0000 7f2c97400640 1 --2- 172.21.15.28:0/46916641 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2c9806dff0 0x7f2c981ee500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:04:13.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.373+0000 7f2c97e00640 1 --2- 172.21.15.28:0/46916641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c980c0fe0 0x7f2c98155070 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f2c90009d90 tx=0x7f2c90004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:13.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.373+0000 7f2c8f400640 1 -- 172.21.15.28:0/46916641 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c90039050 con 0x7f2c980c0fe0 2024-02-21T23:04:13.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.373+0000 7f2c8f400640 1 -- 172.21.15.28:0/46916641 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c9003a040 con 0x7f2c980c0fe0 2024-02-21T23:04:13.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.373+0000 7f2c8f400640 1 -- 172.21.15.28:0/46916641 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c9003e610 con 0x7f2c980c0fe0 2024-02-21T23:04:13.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.373+0000 7f2c9ca00640 1 -- 172.21.15.28:0/46916641 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c981eee40 con 0x7f2c980c0fe0 2024-02-21T23:04:13.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.373+0000 7f2c9ca00640 1 -- 172.21.15.28:0/46916641 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c981ef120 con 0x7f2c980c0fe0 2024-02-21T23:04:13.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.374+0000 7f2c9ca00640 1 -- 172.21.15.28:0/46916641 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c9804f290 con 0x7f2c980c0fe0 2024-02-21T23:04:13.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.375+0000 7f2c8f400640 1 -- 172.21.15.28:0/46916641 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2c90047020 con 0x7f2c980c0fe0 2024-02-21T23:04:13.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.376+0000 7f2c8f400640 1 --2- 172.21.15.28:0/46916641 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2c7406b4d0 0x7f2c7406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:13.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.376+0000 7f2c8f400640 1 -- 172.21.15.28:0/46916641 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f2c900ac830 con 0x7f2c980c0fe0 2024-02-21T23:04:13.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.377+0000 7f2c97400640 1 --2- 172.21.15.28:0/46916641 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2c7406b4d0 0x7f2c7406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:13.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.378+0000 7f2c97400640 1 --2- 172.21.15.28:0/46916641 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2c7406b4d0 0x7f2c7406d990 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f2c88000f60 tx=0x7f2c88009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:13.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.380+0000 7f2c8f400640 1 -- 172.21.15.28:0/46916641 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f2c9004e020 con 0x7f2c980c0fe0 2024-02-21T23:04:13.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.579+0000 7f2c9ca00640 1 -- 172.21.15.28:0/46916641 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2c9806ecb0 con 0x7f2c980c0fe0 2024-02-21T23:04:13.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.582+0000 7f2c8f400640 1 -- 172.21.15.28:0/46916641 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) v1 ==== 74+0+16882 (secure 0 0 0) 0x7f2c9007b540 con 0x7f2c980c0fe0 2024-02-21T23:04:13.571 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:04:13.571 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":38,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:04:10.594881+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:04:06.977614+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-02-21T23:04:08.074258+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:04:13.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.584+0000 7f2c9ca00640 1 -- 172.21.15.28:0/46916641 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2c7406b4d0 msgr2=0x7f2c7406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:13.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.584+0000 7f2c9ca00640 1 --2- 172.21.15.28:0/46916641 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2c7406b4d0 0x7f2c7406d990 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f2c88000f60 tx=0x7f2c88009040 comp rx=0 tx=0).stop 2024-02-21T23:04:13.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.584+0000 7f2c9ca00640 1 -- 172.21.15.28:0/46916641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c980c0fe0 msgr2=0x7f2c98155070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:13.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.584+0000 7f2c9ca00640 1 --2- 172.21.15.28:0/46916641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c980c0fe0 0x7f2c98155070 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f2c90009d90 tx=0x7f2c90004a40 comp rx=0 tx=0).stop 2024-02-21T23:04:13.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.584+0000 7f2c9ca00640 1 -- 172.21.15.28:0/46916641 shutdown_connections 2024-02-21T23:04:13.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.584+0000 7f2c9ca00640 1 --2- 172.21.15.28:0/46916641 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2c7406b4d0 0x7f2c7406d990 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:13.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.584+0000 7f2c9ca00640 1 --2- 172.21.15.28:0/46916641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c980c0fe0 0x7f2c98155070 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:13.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.584+0000 7f2c9ca00640 1 --2- 172.21.15.28:0/46916641 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2c9806e3d0 0x7f2c98154b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:13.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.584+0000 7f2c9ca00640 1 --2- 172.21.15.28:0/46916641 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2c9806dff0 0x7f2c981ee500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:13.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.584+0000 7f2c9ca00640 1 -- 172.21.15.28:0/46916641 >> 172.21.15.28:0/46916641 conn(0x7f2c9806b630 msgr2=0x7f2c980c50d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:13.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.584+0000 7f2c9ca00640 1 -- 172.21.15.28:0/46916641 shutdown_connections 2024-02-21T23:04:13.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.585+0000 7f2c9ca00640 1 -- 172.21.15.28:0/46916641 wait complete. 2024-02-21T23:04:13.592 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-02-21T23:04:13.592 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-21T23:04:13.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.804+0000 7f764aa00640 1 Processor -- start 2024-02-21T23:04:13.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.804+0000 7f764aa00640 1 -- start start 2024-02-21T23:04:13.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.805+0000 7f764aa00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f76440a9930 0x7f76440a6f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:13.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.805+0000 7f764aa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7644152bb0 0x7f764414fea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:13.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.805+0000 7f764aa00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7644156c50 con 0x7f76440c1a40 2024-02-21T23:04:13.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.805+0000 7f764aa00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7644072a40 con 0x7f7644152bb0 2024-02-21T23:04:13.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.805+0000 7f764aa00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7644156ef0 con 0x7f76440a9930 2024-02-21T23:04:13.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.806+0000 7f764a000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7644152bb0 0x7f764414fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:13.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.806+0000 7f764a000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7644152bb0 0x7f764414fea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56386/0 (socket says 172.21.15.28:56386) 2024-02-21T23:04:13.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.806+0000 7f764a000640 1 -- 172.21.15.28:0/1853802757 learned_addr learned my addr 172.21.15.28:0/1853802757 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:13.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.806+0000 7f7649600640 1 --2- 172.21.15.28:0/1853802757 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f76440a9930 0x7f76440a6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:13.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.806+0000 7f764a000640 1 -- 172.21.15.28:0/1853802757 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f76440a9930 msgr2=0x7f76440a6f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:13.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.806+0000 7f764a000640 1 --2- 172.21.15.28:0/1853802757 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f76440a9930 0x7f76440a6f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:13.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.807+0000 7f764a000640 1 -- 172.21.15.28:0/1853802757 >> v1:172.21.15.81:6789/0 conn(0x7f76440c1a40 legacy=0x7f76440c2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:13.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.807+0000 7f764a000640 1 -- 172.21.15.28:0/1853802757 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76440c2830 con 0x7f7644152bb0 2024-02-21T23:04:13.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.807+0000 7f7649600640 1 --2- 172.21.15.28:0/1853802757 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f76440a9930 0x7f76440a6f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:04:13.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.808+0000 7f764a000640 1 --2- 172.21.15.28:0/1853802757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7644152bb0 0x7f764414fea0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f763c009e50 tx=0x7f763c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a8fc19a1da21e30 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:13.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.808+0000 7f7643e00640 1 -- 172.21.15.28:0/1853802757 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f763c02ca70 con 0x7f7644152bb0 2024-02-21T23:04:13.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.808+0000 7f7643e00640 1 -- 172.21.15.28:0/1853802757 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f763c02cc10 con 0x7f7644152bb0 2024-02-21T23:04:13.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.808+0000 7f7643e00640 1 -- 172.21.15.28:0/1853802757 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f763c0343f0 con 0x7f7644152bb0 2024-02-21T23:04:13.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.809+0000 7f764aa00640 1 -- 172.21.15.28:0/1853802757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7644152bb0 msgr2=0x7f764414fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:13.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.809+0000 7f764aa00640 1 --2- 172.21.15.28:0/1853802757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7644152bb0 0x7f764414fea0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f763c009e50 tx=0x7f763c02bed0 comp rx=0 tx=0).stop 2024-02-21T23:04:13.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.809+0000 7f764aa00640 1 -- 172.21.15.28:0/1853802757 shutdown_connections 2024-02-21T23:04:13.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.809+0000 7f764aa00640 1 --2- 172.21.15.28:0/1853802757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7644152bb0 0x7f764414fea0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:13.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.809+0000 7f764aa00640 1 --2- 172.21.15.28:0/1853802757 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f76440a9930 0x7f76440a6f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:13.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.809+0000 7f764aa00640 1 -- 172.21.15.28:0/1853802757 >> 172.21.15.28:0/1853802757 conn(0x7f76440c7720 msgr2=0x7f76440c7b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:13.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.809+0000 7f764aa00640 1 -- 172.21.15.28:0/1853802757 shutdown_connections 2024-02-21T23:04:13.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.809+0000 7f764aa00640 1 -- 172.21.15.28:0/1853802757 wait complete. 2024-02-21T23:04:13.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.811+0000 7f764aa00640 1 Processor -- start 2024-02-21T23:04:13.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.811+0000 7f764aa00640 1 -- start start 2024-02-21T23:04:13.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.812+0000 7f764aa00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f76440a9930 0x7f76440c0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:13.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.812+0000 7f764aa00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76440c1a40 0x7f7644154730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:13.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.812+0000 7f764aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7644152bb0 0x7f7644149940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:13.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.812+0000 7f764aa00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76440a9e90 con 0x7f7644152bb0 2024-02-21T23:04:13.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.812+0000 7f764aa00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76440a9d10 con 0x7f76440c1a40 2024-02-21T23:04:13.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.812+0000 7f764aa00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76441503e0 con 0x7f76440a9930 2024-02-21T23:04:13.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.812+0000 7f764a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7644152bb0 0x7f7644149940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:13.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.812+0000 7f764a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7644152bb0 0x7f7644149940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56400/0 (socket says 172.21.15.28:56400) 2024-02-21T23:04:13.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.812+0000 7f764a000640 1 -- 172.21.15.28:0/4247325896 learned_addr learned my addr 172.21.15.28:0/4247325896 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:13.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.812+0000 7f7649600640 1 --2- 172.21.15.28:0/4247325896 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f76440a9930 0x7f76440c0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:13.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.812+0000 7f7648c00640 1 --2- 172.21.15.28:0/4247325896 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76440c1a40 0x7f7644154730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:13.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.813+0000 7f764a000640 1 -- 172.21.15.28:0/4247325896 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f76440a9930 msgr2=0x7f76440c0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:13.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.813+0000 7f764a000640 1 --2- 172.21.15.28:0/4247325896 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f76440a9930 0x7f76440c0dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:13.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.813+0000 7f764a000640 1 -- 172.21.15.28:0/4247325896 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76440c1a40 msgr2=0x7f7644154730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:13.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.813+0000 7f764a000640 1 --2- 172.21.15.28:0/4247325896 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76440c1a40 0x7f7644154730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:13.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.813+0000 7f764a000640 1 -- 172.21.15.28:0/4247325896 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76440c69a0 con 0x7f7644152bb0 2024-02-21T23:04:13.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.813+0000 7f764a000640 1 --2- 172.21.15.28:0/4247325896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7644152bb0 0x7f7644149940 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f763c036040 tx=0x7f763c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:13.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.813+0000 7f7641600640 1 -- 172.21.15.28:0/4247325896 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f763c043020 con 0x7f7644152bb0 2024-02-21T23:04:13.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.813+0000 7f7641600640 1 -- 172.21.15.28:0/4247325896 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f763c03e450 con 0x7f7644152bb0 2024-02-21T23:04:13.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.813+0000 7f7641600640 1 -- 172.21.15.28:0/4247325896 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f763c009510 con 0x7f7644152bb0 2024-02-21T23:04:13.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.813+0000 7f764aa00640 1 -- 172.21.15.28:0/4247325896 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76440c6c80 con 0x7f7644152bb0 2024-02-21T23:04:13.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.813+0000 7f764aa00640 1 -- 172.21.15.28:0/4247325896 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76440c7240 con 0x7f7644152bb0 2024-02-21T23:04:13.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.814+0000 7f764aa00640 1 -- 172.21.15.28:0/4247325896 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f764414f900 con 0x7f7644152bb0 2024-02-21T23:04:13.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.816+0000 7f7641600640 1 -- 172.21.15.28:0/4247325896 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f763c046020 con 0x7f7644152bb0 2024-02-21T23:04:13.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.817+0000 7f7641600640 1 --2- 172.21.15.28:0/4247325896 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f762406b4d0 0x7f762406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:13.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.817+0000 7f7641600640 1 -- 172.21.15.28:0/4247325896 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f763c0acae0 con 0x7f7644152bb0 2024-02-21T23:04:13.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.818+0000 7f7649600640 1 --2- 172.21.15.28:0/4247325896 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f762406b4d0 0x7f762406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:13.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.819+0000 7f7649600640 1 --2- 172.21.15.28:0/4247325896 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f762406b4d0 0x7f762406d990 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f7638006fd0 tx=0x7f7638008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:13.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:13.823+0000 7f7641600640 1 -- 172.21.15.28:0/4247325896 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f764414f900 con 0x7f7644152bb0 2024-02-21T23:04:14.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.014+0000 7f764aa00640 1 -- 172.21.15.28:0/4247325896 --> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f76440a7500 con 0x7f762406b4d0 2024-02-21T23:04:14.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.035+0000 7f7641600640 1 -- 172.21.15.28:0/4247325896 <== mgr.4101 v2:172.21.15.81:6832/1848538577 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251984 (secure 0 0 0) 0x7f7638010870 con 0x7f762406b4d0 2024-02-21T23:04:14.027 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:04:14.029 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-21T23:04:14.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.040+0000 7f764aa00640 1 -- 172.21.15.28:0/4247325896 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f762406b4d0 msgr2=0x7f762406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:14.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.040+0000 7f764aa00640 1 --2- 172.21.15.28:0/4247325896 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f762406b4d0 0x7f762406d990 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f7638006fd0 tx=0x7f7638008040 comp rx=0 tx=0).stop 2024-02-21T23:04:14.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.040+0000 7f764aa00640 1 -- 172.21.15.28:0/4247325896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7644152bb0 msgr2=0x7f7644149940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:14.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.040+0000 7f764aa00640 1 --2- 172.21.15.28:0/4247325896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7644152bb0 0x7f7644149940 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f763c036040 tx=0x7f763c004a40 comp rx=0 tx=0).stop 2024-02-21T23:04:14.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.041+0000 7f764aa00640 1 -- 172.21.15.28:0/4247325896 shutdown_connections 2024-02-21T23:04:14.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.041+0000 7f764aa00640 1 --2- 172.21.15.28:0/4247325896 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f762406b4d0 0x7f762406d990 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:14.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.041+0000 7f764aa00640 1 --2- 172.21.15.28:0/4247325896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7644152bb0 0x7f7644149940 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:14.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.041+0000 7f764aa00640 1 --2- 172.21.15.28:0/4247325896 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76440c1a40 0x7f7644154730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:14.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.041+0000 7f764aa00640 1 --2- 172.21.15.28:0/4247325896 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f76440a9930 0x7f76440c0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:14.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.041+0000 7f764aa00640 1 -- 172.21.15.28:0/4247325896 >> 172.21.15.28:0/4247325896 conn(0x7f76440c7720 msgr2=0x7f76440c41b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:14.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.041+0000 7f764aa00640 1 -- 172.21.15.28:0/4247325896 shutdown_connections 2024-02-21T23:04:14.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.041+0000 7f764aa00640 1 -- 172.21.15.28:0/4247325896 wait complete. 2024-02-21T23:04:14.048 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":82,"stamp":"2024-02-21T23:04:12.694083+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":95,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":232208,"kb_used_data":3532,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":754742512,"statfs":{"total":773094113280,"available":772856332288,"internally_reserved":0,"allocated":3616768,"data_stored":2659000,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":234016305},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.149879"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-21T23:04:10.604997+0000","last_change":"2024-02-21T23:04:09.604886+0000","last_active":"2024-02-21T23:04:10.604997+0000","last_peered":"2024-02-21T23:04:10.604997+0000","last_clean":"2024-02-21T23:04:10.604997+0000","last_became_active":"2024-02-21T23:04:09.604561+0000","last_became_peered":"2024-02-21T23:04:09.604561+0000","last_unstale":"2024-02-21T23:04:10.604997+0000","last_undegraded":"2024-02-21T23:04:10.604997+0000","last_fullsized":"2024-02-21T23:04:10.604997+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:59:36.867318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-21T23:04:10.604730+0000","last_change":"2024-02-21T23:04:09.604955+0000","last_active":"2024-02-21T23:04:10.604730+0000","last_peered":"2024-02-21T23:04:10.604730+0000","last_clean":"2024-02-21T23:04:10.604730+0000","last_became_active":"2024-02-21T23:04:09.604618+0000","last_became_peered":"2024-02-21T23:04:09.604618+0000","last_unstale":"2024-02-21T23:04:10.604730+0000","last_undegraded":"2024-02-21T23:04:10.604730+0000","last_fullsized":"2024-02-21T23:04:10.604730+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:11:19.186546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.628842+0000","last_change":"2024-02-21T23:04:09.626315+0000","last_active":"2024-02-21T23:04:09.628842+0000","last_peered":"2024-02-21T23:04:09.628842+0000","last_clean":"2024-02-21T23:04:09.628842+0000","last_became_active":"2024-02-21T23:04:09.619401+0000","last_became_peered":"2024-02-21T23:04:09.619401+0000","last_unstale":"2024-02-21T23:04:09.628842+0000","last_undegraded":"2024-02-21T23:04:09.628842+0000","last_fullsized":"2024-02-21T23:04:09.628842+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:52:26.223286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-21T23:04:10.602740+0000","last_change":"2024-02-21T23:04:09.609287+0000","last_active":"2024-02-21T23:04:10.602740+0000","last_peered":"2024-02-21T23:04:10.602740+0000","last_clean":"2024-02-21T23:04:10.602740+0000","last_became_active":"2024-02-21T23:04:09.608998+0000","last_became_peered":"2024-02-21T23:04:09.608998+0000","last_unstale":"2024-02-21T23:04:10.602740+0000","last_undegraded":"2024-02-21T23:04:10.602740+0000","last_fullsized":"2024-02-21T23:04:10.602740+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:35:59.346373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-21T23:04:10.602252+0000","last_change":"2024-02-21T23:04:09.602470+0000","last_active":"2024-02-21T23:04:10.602252+0000","last_peered":"2024-02-21T23:04:10.602252+0000","last_clean":"2024-02-21T23:04:10.602252+0000","last_became_active":"2024-02-21T23:04:09.602202+0000","last_became_peered":"2024-02-21T23:04:09.602202+0000","last_unstale":"2024-02-21T23:04:10.602252+0000","last_undegraded":"2024-02-21T23:04:10.602252+0000","last_fullsized":"2024-02-21T23:04:10.602252+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:31:37.302983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.614726+0000","last_change":"2024-02-21T23:04:09.609901+0000","last_active":"2024-02-21T23:04:09.614726+0000","last_peered":"2024-02-21T23:04:09.614726+0000","last_clean":"2024-02-21T23:04:09.614726+0000","last_became_active":"2024-02-21T23:04:09.609546+0000","last_became_peered":"2024-02-21T23:04:09.609546+0000","last_unstale":"2024-02-21T23:04:09.614726+0000","last_undegraded":"2024-02-21T23:04:09.614726+0000","last_fullsized":"2024-02-21T23:04:09.614726+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:12:08.720118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.625920+0000","last_change":"2024-02-21T23:04:09.617619+0000","last_active":"2024-02-21T23:04:09.625920+0000","last_peered":"2024-02-21T23:04:09.625920+0000","last_clean":"2024-02-21T23:04:09.625920+0000","last_became_active":"2024-02-21T23:04:09.617298+0000","last_became_peered":"2024-02-21T23:04:09.617298+0000","last_unstale":"2024-02-21T23:04:09.625920+0000","last_undegraded":"2024-02-21T23:04:09.625920+0000","last_fullsized":"2024-02-21T23:04:09.625920+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:20:21.010019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-21T23:04:10.604019+0000","last_change":"2024-02-21T23:04:09.615381+0000","last_active":"2024-02-21T23:04:10.604019+0000","last_peered":"2024-02-21T23:04:10.604019+0000","last_clean":"2024-02-21T23:04:10.604019+0000","last_became_active":"2024-02-21T23:04:09.615032+0000","last_became_peered":"2024-02-21T23:04:09.615032+0000","last_unstale":"2024-02-21T23:04:10.604019+0000","last_undegraded":"2024-02-21T23:04:10.604019+0000","last_fullsized":"2024-02-21T23:04:10.604019+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:45:51.879737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.628818+0000","last_change":"2024-02-21T23:04:09.625511+0000","last_active":"2024-02-21T23:04:09.628818+0000","last_peered":"2024-02-21T23:04:09.628818+0000","last_clean":"2024-02-21T23:04:09.628818+0000","last_became_active":"2024-02-21T23:04:09.621287+0000","last_became_peered":"2024-02-21T23:04:09.621287+0000","last_unstale":"2024-02-21T23:04:09.628818+0000","last_undegraded":"2024-02-21T23:04:09.628818+0000","last_fullsized":"2024-02-21T23:04:09.628818+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:42:21.080709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.628274+0000","last_change":"2024-02-21T23:04:09.621922+0000","last_active":"2024-02-21T23:04:09.628274+0000","last_peered":"2024-02-21T23:04:09.628274+0000","last_clean":"2024-02-21T23:04:09.628274+0000","last_became_active":"2024-02-21T23:04:09.621640+0000","last_became_peered":"2024-02-21T23:04:09.621640+0000","last_unstale":"2024-02-21T23:04:09.628274+0000","last_undegraded":"2024-02-21T23:04:09.628274+0000","last_fullsized":"2024-02-21T23:04:09.628274+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:48:20.132275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.628495+0000","last_change":"2024-02-21T23:04:09.622325+0000","last_active":"2024-02-21T23:04:09.628495+0000","last_peered":"2024-02-21T23:04:09.628495+0000","last_clean":"2024-02-21T23:04:09.628495+0000","last_became_active":"2024-02-21T23:04:09.619954+0000","last_became_peered":"2024-02-21T23:04:09.619954+0000","last_unstale":"2024-02-21T23:04:09.628495+0000","last_undegraded":"2024-02-21T23:04:09.628495+0000","last_fullsized":"2024-02-21T23:04:09.628495+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:02:35.839059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.601874+0000","last_change":"2024-02-21T23:04:09.599332+0000","last_active":"2024-02-21T23:04:09.601874+0000","last_peered":"2024-02-21T23:04:09.601874+0000","last_clean":"2024-02-21T23:04:09.601874+0000","last_became_active":"2024-02-21T23:04:09.599077+0000","last_became_peered":"2024-02-21T23:04:09.599077+0000","last_unstale":"2024-02-21T23:04:09.601874+0000","last_undegraded":"2024-02-21T23:04:09.601874+0000","last_fullsized":"2024-02-21T23:04:09.601874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:39:02.297782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.623989+0000","last_change":"2024-02-21T23:04:09.619500+0000","last_active":"2024-02-21T23:04:09.623989+0000","last_peered":"2024-02-21T23:04:09.623989+0000","last_clean":"2024-02-21T23:04:09.623989+0000","last_became_active":"2024-02-21T23:04:09.618627+0000","last_became_peered":"2024-02-21T23:04:09.618627+0000","last_unstale":"2024-02-21T23:04:09.623989+0000","last_undegraded":"2024-02-21T23:04:09.623989+0000","last_fullsized":"2024-02-21T23:04:09.623989+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:21:27.477217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.635155+0000","last_change":"2024-02-21T23:04:09.628339+0000","last_active":"2024-02-21T23:04:09.635155+0000","last_peered":"2024-02-21T23:04:09.635155+0000","last_clean":"2024-02-21T23:04:09.635155+0000","last_became_active":"2024-02-21T23:04:09.628001+0000","last_became_peered":"2024-02-21T23:04:09.628001+0000","last_unstale":"2024-02-21T23:04:09.635155+0000","last_undegraded":"2024-02-21T23:04:09.635155+0000","last_fullsized":"2024-02-21T23:04:09.635155+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:12:08.720118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.626076+0000","last_change":"2024-02-21T23:04:09.619253+0000","last_active":"2024-02-21T23:04:09.626076+0000","last_peered":"2024-02-21T23:04:09.626076+0000","last_clean":"2024-02-21T23:04:09.626076+0000","last_became_active":"2024-02-21T23:04:09.618395+0000","last_became_peered":"2024-02-21T23:04:09.618395+0000","last_unstale":"2024-02-21T23:04:09.626076+0000","last_undegraded":"2024-02-21T23:04:09.626076+0000","last_fullsized":"2024-02-21T23:04:09.626076+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:17:53.131987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.631707+0000","last_change":"2024-02-21T23:04:09.625455+0000","last_active":"2024-02-21T23:04:09.631707+0000","last_peered":"2024-02-21T23:04:09.631707+0000","last_clean":"2024-02-21T23:04:09.631707+0000","last_became_active":"2024-02-21T23:04:09.625173+0000","last_became_peered":"2024-02-21T23:04:09.625173+0000","last_unstale":"2024-02-21T23:04:09.631707+0000","last_undegraded":"2024-02-21T23:04:09.631707+0000","last_fullsized":"2024-02-21T23:04:09.631707+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:41:02.807550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.635555+0000","last_change":"2024-02-21T23:04:09.630271+0000","last_active":"2024-02-21T23:04:09.635555+0000","last_peered":"2024-02-21T23:04:09.635555+0000","last_clean":"2024-02-21T23:04:09.635555+0000","last_became_active":"2024-02-21T23:04:09.629967+0000","last_became_peered":"2024-02-21T23:04:09.629967+0000","last_unstale":"2024-02-21T23:04:09.635555+0000","last_undegraded":"2024-02-21T23:04:09.635555+0000","last_fullsized":"2024-02-21T23:04:09.635555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:42:58.911438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-21T23:04:10.606737+0000","last_change":"2024-02-21T23:04:09.618559+0000","last_active":"2024-02-21T23:04:10.606737+0000","last_peered":"2024-02-21T23:04:10.606737+0000","last_clean":"2024-02-21T23:04:10.606737+0000","last_became_active":"2024-02-21T23:04:09.618278+0000","last_became_peered":"2024-02-21T23:04:09.618278+0000","last_unstale":"2024-02-21T23:04:10.606737+0000","last_undegraded":"2024-02-21T23:04:10.606737+0000","last_fullsized":"2024-02-21T23:04:10.606737+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:42:58.911438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-21T23:04:10.602699+0000","last_change":"2024-02-21T23:04:09.597652+0000","last_active":"2024-02-21T23:04:10.602699+0000","last_peered":"2024-02-21T23:04:10.602699+0000","last_clean":"2024-02-21T23:04:10.602699+0000","last_became_active":"2024-02-21T23:04:09.597388+0000","last_became_peered":"2024-02-21T23:04:09.597388+0000","last_unstale":"2024-02-21T23:04:10.602699+0000","last_undegraded":"2024-02-21T23:04:10.602699+0000","last_fullsized":"2024-02-21T23:04:10.602699+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:28:32.248822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-21T23:04:10.606330+0000","last_change":"2024-02-21T23:04:09.619916+0000","last_active":"2024-02-21T23:04:10.606330+0000","last_peered":"2024-02-21T23:04:10.606330+0000","last_clean":"2024-02-21T23:04:10.606330+0000","last_became_active":"2024-02-21T23:04:09.617098+0000","last_became_peered":"2024-02-21T23:04:09.617098+0000","last_unstale":"2024-02-21T23:04:10.606330+0000","last_undegraded":"2024-02-21T23:04:10.606330+0000","last_fullsized":"2024-02-21T23:04:10.606330+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:52:26.223286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.624011+0000","last_change":"2024-02-21T23:04:09.621231+0000","last_active":"2024-02-21T23:04:09.624011+0000","last_peered":"2024-02-21T23:04:09.624011+0000","last_clean":"2024-02-21T23:04:09.624011+0000","last_became_active":"2024-02-21T23:04:09.620885+0000","last_became_peered":"2024-02-21T23:04:09.620885+0000","last_unstale":"2024-02-21T23:04:09.624011+0000","last_undegraded":"2024-02-21T23:04:09.624011+0000","last_fullsized":"2024-02-21T23:04:09.624011+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:57:41.332804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.614227+0000","last_change":"2024-02-21T23:04:09.607445+0000","last_active":"2024-02-21T23:04:09.614227+0000","last_peered":"2024-02-21T23:04:09.614227+0000","last_clean":"2024-02-21T23:04:09.614227+0000","last_became_active":"2024-02-21T23:04:09.606419+0000","last_became_peered":"2024-02-21T23:04:09.606419+0000","last_unstale":"2024-02-21T23:04:09.614227+0000","last_undegraded":"2024-02-21T23:04:09.614227+0000","last_fullsized":"2024-02-21T23:04:09.614227+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:38:20.998379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.633803+0000","last_change":"2024-02-21T23:04:09.629852+0000","last_active":"2024-02-21T23:04:09.633803+0000","last_peered":"2024-02-21T23:04:09.633803+0000","last_clean":"2024-02-21T23:04:09.633803+0000","last_became_active":"2024-02-21T23:04:09.629479+0000","last_became_peered":"2024-02-21T23:04:09.629479+0000","last_unstale":"2024-02-21T23:04:09.633803+0000","last_undegraded":"2024-02-21T23:04:09.633803+0000","last_fullsized":"2024-02-21T23:04:09.633803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:48:43.036434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-21T23:04:10.602399+0000","last_change":"2024-02-21T23:04:09.621204+0000","last_active":"2024-02-21T23:04:10.602399+0000","last_peered":"2024-02-21T23:04:10.602399+0000","last_clean":"2024-02-21T23:04:10.602399+0000","last_became_active":"2024-02-21T23:04:09.620125+0000","last_became_peered":"2024-02-21T23:04:09.620125+0000","last_unstale":"2024-02-21T23:04:10.602399+0000","last_undegraded":"2024-02-21T23:04:10.602399+0000","last_fullsized":"2024-02-21T23:04:10.602399+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:47:11.522234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-21T23:04:10.606543+0000","last_change":"2024-02-21T23:04:09.618492+0000","last_active":"2024-02-21T23:04:10.606543+0000","last_peered":"2024-02-21T23:04:10.606543+0000","last_clean":"2024-02-21T23:04:10.606543+0000","last_became_active":"2024-02-21T23:04:09.617606+0000","last_became_peered":"2024-02-21T23:04:09.617606+0000","last_unstale":"2024-02-21T23:04:10.606543+0000","last_undegraded":"2024-02-21T23:04:10.606543+0000","last_fullsized":"2024-02-21T23:04:10.606543+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:48:12.283971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.627664+0000","last_change":"2024-02-21T23:04:09.621098+0000","last_active":"2024-02-21T23:04:09.627664+0000","last_peered":"2024-02-21T23:04:09.627664+0000","last_clean":"2024-02-21T23:04:09.627664+0000","last_became_active":"2024-02-21T23:04:09.620752+0000","last_became_peered":"2024-02-21T23:04:09.620752+0000","last_unstale":"2024-02-21T23:04:09.627664+0000","last_undegraded":"2024-02-21T23:04:09.627664+0000","last_fullsized":"2024-02-21T23:04:09.627664+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:41:02.807550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-21T23:04:10.606501+0000","last_change":"2024-02-21T23:04:09.625567+0000","last_active":"2024-02-21T23:04:10.606501+0000","last_peered":"2024-02-21T23:04:10.606501+0000","last_clean":"2024-02-21T23:04:10.606501+0000","last_became_active":"2024-02-21T23:04:09.625194+0000","last_became_peered":"2024-02-21T23:04:09.625194+0000","last_unstale":"2024-02-21T23:04:10.606501+0000","last_undegraded":"2024-02-21T23:04:10.606501+0000","last_fullsized":"2024-02-21T23:04:10.606501+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:48:43.036434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.625563+0000","last_change":"2024-02-21T23:04:09.620496+0000","last_active":"2024-02-21T23:04:09.625563+0000","last_peered":"2024-02-21T23:04:09.625563+0000","last_clean":"2024-02-21T23:04:09.625563+0000","last_became_active":"2024-02-21T23:04:09.620193+0000","last_became_peered":"2024-02-21T23:04:09.620193+0000","last_unstale":"2024-02-21T23:04:09.625563+0000","last_undegraded":"2024-02-21T23:04:09.625563+0000","last_fullsized":"2024-02-21T23:04:09.625563+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:59:38.223730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-21T23:04:10.605907+0000","last_change":"2024-02-21T23:04:09.623456+0000","last_active":"2024-02-21T23:04:10.605907+0000","last_peered":"2024-02-21T23:04:10.605907+0000","last_clean":"2024-02-21T23:04:10.605907+0000","last_became_active":"2024-02-21T23:04:09.623126+0000","last_became_peered":"2024-02-21T23:04:09.623126+0000","last_unstale":"2024-02-21T23:04:10.605907+0000","last_undegraded":"2024-02-21T23:04:10.605907+0000","last_fullsized":"2024-02-21T23:04:10.605907+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:42:21.080709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-21T23:04:10.605874+0000","last_change":"2024-02-21T23:04:09.626410+0000","last_active":"2024-02-21T23:04:10.605874+0000","last_peered":"2024-02-21T23:04:10.605874+0000","last_clean":"2024-02-21T23:04:10.605874+0000","last_became_active":"2024-02-21T23:04:09.625585+0000","last_became_peered":"2024-02-21T23:04:09.625585+0000","last_unstale":"2024-02-21T23:04:10.605874+0000","last_undegraded":"2024-02-21T23:04:10.605874+0000","last_fullsized":"2024-02-21T23:04:10.605874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:41:02.807550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.633783+0000","last_change":"2024-02-21T23:04:09.629688+0000","last_active":"2024-02-21T23:04:09.633783+0000","last_peered":"2024-02-21T23:04:09.633783+0000","last_clean":"2024-02-21T23:04:09.633783+0000","last_became_active":"2024-02-21T23:04:09.624392+0000","last_became_peered":"2024-02-21T23:04:09.624392+0000","last_unstale":"2024-02-21T23:04:09.633783+0000","last_undegraded":"2024-02-21T23:04:09.633783+0000","last_fullsized":"2024-02-21T23:04:09.633783+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:52:26.223286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.605803+0000","last_change":"2024-02-21T23:04:09.603310+0000","last_active":"2024-02-21T23:04:09.605803+0000","last_peered":"2024-02-21T23:04:09.605803+0000","last_clean":"2024-02-21T23:04:09.605803+0000","last_became_active":"2024-02-21T23:04:09.603009+0000","last_became_peered":"2024-02-21T23:04:09.603009+0000","last_unstale":"2024-02-21T23:04:09.605803+0000","last_undegraded":"2024-02-21T23:04:09.605803+0000","last_fullsized":"2024-02-21T23:04:09.605803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:03:19.900047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.633758+0000","last_change":"2024-02-21T23:04:09.629888+0000","last_active":"2024-02-21T23:04:09.633758+0000","last_peered":"2024-02-21T23:04:09.633758+0000","last_clean":"2024-02-21T23:04:09.633758+0000","last_became_active":"2024-02-21T23:04:09.629432+0000","last_became_peered":"2024-02-21T23:04:09.629432+0000","last_unstale":"2024-02-21T23:04:09.633758+0000","last_undegraded":"2024-02-21T23:04:09.633758+0000","last_fullsized":"2024-02-21T23:04:09.633758+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:48:12.283971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"37'4","reported_seq":18,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-21T23:04:10.606444+0000","last_change":"2024-02-21T23:04:08.587883+0000","last_active":"2024-02-21T23:04:10.606444+0000","last_peered":"2024-02-21T23:04:10.606444+0000","last_clean":"2024-02-21T23:04:10.606444+0000","last_became_active":"2024-02-21T23:04:08.587362+0000","last_became_peered":"2024-02-21T23:04:08.587362+0000","last_unstale":"2024-02-21T23:04:10.606444+0000","last_undegraded":"2024-02-21T23:04:10.606444+0000","last_fullsized":"2024-02-21T23:04:10.606444+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_clean_scrub_stamp":"2024-02-21T23:04:07.527107+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:12:56.945740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.628609+0000","last_change":"2024-02-21T23:04:09.621971+0000","last_active":"2024-02-21T23:04:09.628609+0000","last_peered":"2024-02-21T23:04:09.628609+0000","last_clean":"2024-02-21T23:04:09.628609+0000","last_became_active":"2024-02-21T23:04:09.621550+0000","last_became_peered":"2024-02-21T23:04:09.621550+0000","last_unstale":"2024-02-21T23:04:09.628609+0000","last_undegraded":"2024-02-21T23:04:09.628609+0000","last_fullsized":"2024-02-21T23:04:09.628609+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:42:21.080709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"37'1","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.631605+0000","last_change":"2024-02-21T23:04:08.590209+0000","last_active":"2024-02-21T23:04:09.631605+0000","last_peered":"2024-02-21T23:04:09.631605+0000","last_clean":"2024-02-21T23:04:09.631605+0000","last_became_active":"2024-02-21T23:04:08.589854+0000","last_became_peered":"2024-02-21T23:04:08.589854+0000","last_unstale":"2024-02-21T23:04:09.631605+0000","last_undegraded":"2024-02-21T23:04:09.631605+0000","last_fullsized":"2024-02-21T23:04:09.631605+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_clean_scrub_stamp":"2024-02-21T23:04:07.527107+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:35:58.330966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.640553+0000","last_change":"2024-02-21T23:04:09.636467+0000","last_active":"2024-02-21T23:04:09.640553+0000","last_peered":"2024-02-21T23:04:09.640553+0000","last_clean":"2024-02-21T23:04:09.640553+0000","last_became_active":"2024-02-21T23:04:09.636180+0000","last_became_peered":"2024-02-21T23:04:09.636180+0000","last_unstale":"2024-02-21T23:04:09.640553+0000","last_undegraded":"2024-02-21T23:04:09.640553+0000","last_fullsized":"2024-02-21T23:04:09.640553+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:41:02.807550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"37'2","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.637784+0000","last_change":"2024-02-21T23:04:08.590700+0000","last_active":"2024-02-21T23:04:09.637784+0000","last_peered":"2024-02-21T23:04:09.637784+0000","last_clean":"2024-02-21T23:04:09.637784+0000","last_became_active":"2024-02-21T23:04:08.590305+0000","last_became_peered":"2024-02-21T23:04:08.590305+0000","last_unstale":"2024-02-21T23:04:09.637784+0000","last_undegraded":"2024-02-21T23:04:09.637784+0000","last_fullsized":"2024-02-21T23:04:09.637784+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_clean_scrub_stamp":"2024-02-21T23:04:07.527107+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:39:01.282374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-21T23:04:10.604884+0000","last_change":"2024-02-21T23:04:09.634662+0000","last_active":"2024-02-21T23:04:10.604884+0000","last_peered":"2024-02-21T23:04:10.604884+0000","last_clean":"2024-02-21T23:04:10.604884+0000","last_became_active":"2024-02-21T23:04:09.634326+0000","last_became_peered":"2024-02-21T23:04:09.634326+0000","last_unstale":"2024-02-21T23:04:10.604884+0000","last_undegraded":"2024-02-21T23:04:10.604884+0000","last_fullsized":"2024-02-21T23:04:10.604884+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:54:44.049051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"37'2","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.643221+0000","last_change":"2024-02-21T23:04:08.588381+0000","last_active":"2024-02-21T23:04:09.643221+0000","last_peered":"2024-02-21T23:04:09.643221+0000","last_clean":"2024-02-21T23:04:09.643221+0000","last_became_active":"2024-02-21T23:04:08.588053+0000","last_became_peered":"2024-02-21T23:04:08.588053+0000","last_unstale":"2024-02-21T23:04:09.643221+0000","last_undegraded":"2024-02-21T23:04:09.643221+0000","last_fullsized":"2024-02-21T23:04:09.643221+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_clean_scrub_stamp":"2024-02-21T23:04:07.527107+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:35:58.330966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.603188+0000","last_change":"2024-02-21T23:04:09.596646+0000","last_active":"2024-02-21T23:04:09.603188+0000","last_peered":"2024-02-21T23:04:09.603188+0000","last_clean":"2024-02-21T23:04:09.603188+0000","last_became_active":"2024-02-21T23:04:09.596300+0000","last_became_peered":"2024-02-21T23:04:09.596300+0000","last_unstale":"2024-02-21T23:04:09.603188+0000","last_undegraded":"2024-02-21T23:04:09.603188+0000","last_fullsized":"2024-02-21T23:04:09.603188+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:12:57.961148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"37'1","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.632253+0000","last_change":"2024-02-21T23:04:08.591545+0000","last_active":"2024-02-21T23:04:09.632253+0000","last_peered":"2024-02-21T23:04:09.632253+0000","last_clean":"2024-02-21T23:04:09.632253+0000","last_became_active":"2024-02-21T23:04:08.591190+0000","last_became_peered":"2024-02-21T23:04:08.591190+0000","last_unstale":"2024-02-21T23:04:09.632253+0000","last_undegraded":"2024-02-21T23:04:09.632253+0000","last_fullsized":"2024-02-21T23:04:09.632253+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_clean_scrub_stamp":"2024-02-21T23:04:07.527107+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:42:57.896031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.626098+0000","last_change":"2024-02-21T23:04:09.620887+0000","last_active":"2024-02-21T23:04:09.626098+0000","last_peered":"2024-02-21T23:04:09.626098+0000","last_clean":"2024-02-21T23:04:09.626098+0000","last_became_active":"2024-02-21T23:04:09.620572+0000","last_became_peered":"2024-02-21T23:04:09.620572+0000","last_unstale":"2024-02-21T23:04:09.626098+0000","last_undegraded":"2024-02-21T23:04:09.626098+0000","last_fullsized":"2024-02-21T23:04:09.626098+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:28:58.719324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"37'1","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.638555+0000","last_change":"2024-02-21T23:04:08.599221+0000","last_active":"2024-02-21T23:04:09.638555+0000","last_peered":"2024-02-21T23:04:09.638555+0000","last_clean":"2024-02-21T23:04:09.638555+0000","last_became_active":"2024-02-21T23:04:08.598877+0000","last_became_peered":"2024-02-21T23:04:08.598877+0000","last_unstale":"2024-02-21T23:04:09.638555+0000","last_undegraded":"2024-02-21T23:04:09.638555+0000","last_fullsized":"2024-02-21T23:04:09.638555+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_clean_scrub_stamp":"2024-02-21T23:04:07.527107+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:43:30.887358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.633331+0000","last_change":"2024-02-21T23:04:09.623815+0000","last_active":"2024-02-21T23:04:09.633331+0000","last_peered":"2024-02-21T23:04:09.633331+0000","last_clean":"2024-02-21T23:04:09.633331+0000","last_became_active":"2024-02-21T23:04:09.623444+0000","last_became_peered":"2024-02-21T23:04:09.623444+0000","last_unstale":"2024-02-21T23:04:09.633331+0000","last_undegraded":"2024-02-21T23:04:09.633331+0000","last_fullsized":"2024-02-21T23:04:09.633331+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:03:19.900047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.595582+0000","last_change":"2024-02-21T23:04:08.590739+0000","last_active":"2024-02-21T23:04:09.595582+0000","last_peered":"2024-02-21T23:04:09.595582+0000","last_clean":"2024-02-21T23:04:09.595582+0000","last_became_active":"2024-02-21T23:04:08.590119+0000","last_became_peered":"2024-02-21T23:04:08.590119+0000","last_unstale":"2024-02-21T23:04:09.595582+0000","last_undegraded":"2024-02-21T23:04:09.595582+0000","last_fullsized":"2024-02-21T23:04:09.595582+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_clean_scrub_stamp":"2024-02-21T23:04:07.527107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:12:56.945740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-21T23:04:10.604076+0000","last_change":"2024-02-21T23:04:09.610542+0000","last_active":"2024-02-21T23:04:10.604076+0000","last_peered":"2024-02-21T23:04:10.604076+0000","last_clean":"2024-02-21T23:04:10.604076+0000","last_became_active":"2024-02-21T23:04:09.610322+0000","last_became_peered":"2024-02-21T23:04:09.610322+0000","last_unstale":"2024-02-21T23:04:10.604076+0000","last_undegraded":"2024-02-21T23:04:10.604076+0000","last_fullsized":"2024-02-21T23:04:10.604076+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:17:20.153344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.590276+0000","last_change":"2024-02-21T23:04:08.583183+0000","last_active":"2024-02-21T23:04:09.590276+0000","last_peered":"2024-02-21T23:04:09.590276+0000","last_clean":"2024-02-21T23:04:09.590276+0000","last_became_active":"2024-02-21T23:04:08.582207+0000","last_became_peered":"2024-02-21T23:04:08.582207+0000","last_unstale":"2024-02-21T23:04:09.590276+0000","last_undegraded":"2024-02-21T23:04:09.590276+0000","last_fullsized":"2024-02-21T23:04:09.590276+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_clean_scrub_stamp":"2024-02-21T23:04:07.527107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:31:36.287576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-21T23:04:10.602912+0000","last_change":"2024-02-21T23:04:09.610792+0000","last_active":"2024-02-21T23:04:10.602912+0000","last_peered":"2024-02-21T23:04:10.602912+0000","last_clean":"2024-02-21T23:04:10.602912+0000","last_became_active":"2024-02-21T23:04:09.609106+0000","last_became_peered":"2024-02-21T23:04:09.609106+0000","last_unstale":"2024-02-21T23:04:10.602912+0000","last_undegraded":"2024-02-21T23:04:10.602912+0000","last_fullsized":"2024-02-21T23:04:10.602912+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:12:57.961148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-21T23:04:10.605075+0000","last_change":"2024-02-21T23:04:08.572324+0000","last_active":"2024-02-21T23:04:10.605075+0000","last_peered":"2024-02-21T23:04:10.605075+0000","last_clean":"2024-02-21T23:04:10.605075+0000","last_became_active":"2024-02-21T23:04:08.572018+0000","last_became_peered":"2024-02-21T23:04:08.572018+0000","last_unstale":"2024-02-21T23:04:10.605075+0000","last_undegraded":"2024-02-21T23:04:10.605075+0000","last_fullsized":"2024-02-21T23:04:10.605075+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_clean_scrub_stamp":"2024-02-21T23:04:07.527107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:59:37.208322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.641079+0000","last_change":"2024-02-21T23:04:09.635684+0000","last_active":"2024-02-21T23:04:09.641079+0000","last_peered":"2024-02-21T23:04:09.641079+0000","last_clean":"2024-02-21T23:04:09.641079+0000","last_became_active":"2024-02-21T23:04:09.635166+0000","last_became_peered":"2024-02-21T23:04:09.635166+0000","last_unstale":"2024-02-21T23:04:09.641079+0000","last_undegraded":"2024-02-21T23:04:09.641079+0000","last_fullsized":"2024-02-21T23:04:09.641079+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:20:21.010019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.612434+0000","last_change":"2024-02-21T23:04:08.591105+0000","last_active":"2024-02-21T23:04:09.612434+0000","last_peered":"2024-02-21T23:04:09.612434+0000","last_clean":"2024-02-21T23:04:09.612434+0000","last_became_active":"2024-02-21T23:04:08.590825+0000","last_became_peered":"2024-02-21T23:04:08.590825+0000","last_unstale":"2024-02-21T23:04:09.612434+0000","last_undegraded":"2024-02-21T23:04:09.612434+0000","last_fullsized":"2024-02-21T23:04:09.612434+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_clean_scrub_stamp":"2024-02-21T23:04:07.527107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:48:42.021027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.628598+0000","last_change":"2024-02-21T23:04:09.623029+0000","last_active":"2024-02-21T23:04:09.628598+0000","last_peered":"2024-02-21T23:04:09.628598+0000","last_clean":"2024-02-21T23:04:09.628598+0000","last_became_active":"2024-02-21T23:04:09.617550+0000","last_became_peered":"2024-02-21T23:04:09.617550+0000","last_unstale":"2024-02-21T23:04:09.628598+0000","last_undegraded":"2024-02-21T23:04:09.628598+0000","last_fullsized":"2024-02-21T23:04:09.628598+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:41:02.807550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"37'3","reported_seq":15,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.639317+0000","last_change":"2024-02-21T23:04:08.592296+0000","last_active":"2024-02-21T23:04:09.639317+0000","last_peered":"2024-02-21T23:04:09.639317+0000","last_clean":"2024-02-21T23:04:09.639317+0000","last_became_active":"2024-02-21T23:04:08.589032+0000","last_became_peered":"2024-02-21T23:04:08.589032+0000","last_unstale":"2024-02-21T23:04:09.639317+0000","last_undegraded":"2024-02-21T23:04:09.639317+0000","last_fullsized":"2024-02-21T23:04:09.639317+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_clean_scrub_stamp":"2024-02-21T23:04:07.527107+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:48:11.268564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.614476+0000","last_change":"2024-02-21T23:04:09.609319+0000","last_active":"2024-02-21T23:04:09.614476+0000","last_peered":"2024-02-21T23:04:09.614476+0000","last_clean":"2024-02-21T23:04:09.614476+0000","last_became_active":"2024-02-21T23:04:09.608169+0000","last_became_peered":"2024-02-21T23:04:09.608169+0000","last_unstale":"2024-02-21T23:04:09.614476+0000","last_undegraded":"2024-02-21T23:04:09.614476+0000","last_fullsized":"2024-02-21T23:04:09.614476+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:43:31.902765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"37'6","reported_seq":20,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-21T23:04:10.604795+0000","last_change":"2024-02-21T23:04:08.588332+0000","last_active":"2024-02-21T23:04:10.604795+0000","last_peered":"2024-02-21T23:04:10.604795+0000","last_clean":"2024-02-21T23:04:10.604795+0000","last_became_active":"2024-02-21T23:04:08.587826+0000","last_became_peered":"2024-02-21T23:04:08.587826+0000","last_unstale":"2024-02-21T23:04:10.604795+0000","last_undegraded":"2024-02-21T23:04:10.604795+0000","last_fullsized":"2024-02-21T23:04:10.604795+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_clean_scrub_stamp":"2024-02-21T23:04:07.527107+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:21:26.461810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.628530+0000","last_change":"2024-02-21T23:04:09.620053+0000","last_active":"2024-02-21T23:04:09.628530+0000","last_peered":"2024-02-21T23:04:09.628530+0000","last_clean":"2024-02-21T23:04:09.628530+0000","last_became_active":"2024-02-21T23:04:09.616123+0000","last_became_peered":"2024-02-21T23:04:09.616123+0000","last_unstale":"2024-02-21T23:04:09.628530+0000","last_undegraded":"2024-02-21T23:04:09.628530+0000","last_fullsized":"2024-02-21T23:04:09.628530+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:12:08.720118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"37'2","reported_seq":16,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-21T23:04:10.604463+0000","last_change":"2024-02-21T23:04:08.578453+0000","last_active":"2024-02-21T23:04:10.604463+0000","last_peered":"2024-02-21T23:04:10.604463+0000","last_clean":"2024-02-21T23:04:10.604463+0000","last_became_active":"2024-02-21T23:04:08.575240+0000","last_became_peered":"2024-02-21T23:04:08.575240+0000","last_unstale":"2024-02-21T23:04:10.604463+0000","last_undegraded":"2024-02-21T23:04:10.604463+0000","last_fullsized":"2024-02-21T23:04:10.604463+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_clean_scrub_stamp":"2024-02-21T23:04:07.527107+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:25:42.570905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-21T23:04:10.603962+0000","last_change":"2024-02-21T23:04:09.603858+0000","last_active":"2024-02-21T23:04:10.603962+0000","last_peered":"2024-02-21T23:04:10.603962+0000","last_clean":"2024-02-21T23:04:10.603962+0000","last_became_active":"2024-02-21T23:04:09.603561+0000","last_became_peered":"2024-02-21T23:04:09.603561+0000","last_unstale":"2024-02-21T23:04:10.603962+0000","last_undegraded":"2024-02-21T23:04:10.603962+0000","last_fullsized":"2024-02-21T23:04:10.603962+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:28:25.253674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.596420+0000","last_change":"2024-02-21T23:04:08.592119+0000","last_active":"2024-02-21T23:04:09.596420+0000","last_peered":"2024-02-21T23:04:09.596420+0000","last_clean":"2024-02-21T23:04:09.596420+0000","last_became_active":"2024-02-21T23:04:08.591851+0000","last_became_peered":"2024-02-21T23:04:08.591851+0000","last_unstale":"2024-02-21T23:04:09.596420+0000","last_undegraded":"2024-02-21T23:04:09.596420+0000","last_fullsized":"2024-02-21T23:04:09.596420+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_clean_scrub_stamp":"2024-02-21T23:04:07.527107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:43:30.887358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.628301+0000","last_change":"2024-02-21T23:04:09.625606+0000","last_active":"2024-02-21T23:04:09.628301+0000","last_peered":"2024-02-21T23:04:09.628301+0000","last_clean":"2024-02-21T23:04:09.628301+0000","last_became_active":"2024-02-21T23:04:09.625280+0000","last_became_peered":"2024-02-21T23:04:09.625280+0000","last_unstale":"2024-02-21T23:04:09.628301+0000","last_undegraded":"2024-02-21T23:04:09.628301+0000","last_fullsized":"2024-02-21T23:04:09.628301+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:42:21.080709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-21T23:04:10.605176+0000","last_change":"2024-02-21T23:04:08.593155+0000","last_active":"2024-02-21T23:04:10.605176+0000","last_peered":"2024-02-21T23:04:10.605176+0000","last_clean":"2024-02-21T23:04:10.605176+0000","last_became_active":"2024-02-21T23:04:08.592850+0000","last_became_peered":"2024-02-21T23:04:08.592850+0000","last_unstale":"2024-02-21T23:04:10.605176+0000","last_undegraded":"2024-02-21T23:04:10.605176+0000","last_fullsized":"2024-02-21T23:04:10.605176+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_clean_scrub_stamp":"2024-02-21T23:04:07.527107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:47:10.506827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-21T23:04:10.604342+0000","last_change":"2024-02-21T23:04:09.604835+0000","last_active":"2024-02-21T23:04:10.604342+0000","last_peered":"2024-02-21T23:04:10.604342+0000","last_clean":"2024-02-21T23:04:10.604342+0000","last_became_active":"2024-02-21T23:04:09.604489+0000","last_became_peered":"2024-02-21T23:04:09.604489+0000","last_unstale":"2024-02-21T23:04:10.604342+0000","last_undegraded":"2024-02-21T23:04:10.604342+0000","last_fullsized":"2024-02-21T23:04:10.604342+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:03:19.008942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"37'8","reported_seq":20,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.641113+0000","last_change":"2024-02-21T23:04:08.599153+0000","last_active":"2024-02-21T23:04:09.641113+0000","last_peered":"2024-02-21T23:04:09.641113+0000","last_clean":"2024-02-21T23:04:09.641113+0000","last_became_active":"2024-02-21T23:04:08.598820+0000","last_became_peered":"2024-02-21T23:04:08.598820+0000","last_unstale":"2024-02-21T23:04:09.641113+0000","last_undegraded":"2024-02-21T23:04:09.641113+0000","last_fullsized":"2024-02-21T23:04:09.641113+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_clean_scrub_stamp":"2024-02-21T23:04:07.527107+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:39:01.282374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-21T23:04:10.604510+0000","last_change":"2024-02-21T23:04:09.633198+0000","last_active":"2024-02-21T23:04:10.604510+0000","last_peered":"2024-02-21T23:04:10.604510+0000","last_clean":"2024-02-21T23:04:10.604510+0000","last_became_active":"2024-02-21T23:04:09.632938+0000","last_became_peered":"2024-02-21T23:04:09.632938+0000","last_unstale":"2024-02-21T23:04:10.604510+0000","last_undegraded":"2024-02-21T23:04:10.604510+0000","last_fullsized":"2024-02-21T23:04:10.604510+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:16:31.510876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"37'1","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.640845+0000","last_change":"2024-02-21T23:04:08.599197+0000","last_active":"2024-02-21T23:04:09.640845+0000","last_peered":"2024-02-21T23:04:09.640845+0000","last_clean":"2024-02-21T23:04:09.640845+0000","last_became_active":"2024-02-21T23:04:08.598898+0000","last_became_peered":"2024-02-21T23:04:08.598898+0000","last_unstale":"2024-02-21T23:04:09.640845+0000","last_undegraded":"2024-02-21T23:04:09.640845+0000","last_fullsized":"2024-02-21T23:04:09.640845+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_clean_scrub_stamp":"2024-02-21T23:04:07.527107+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:02:29.779590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.632341+0000","last_change":"2024-02-21T23:04:09.624622+0000","last_active":"2024-02-21T23:04:09.632341+0000","last_peered":"2024-02-21T23:04:09.632341+0000","last_clean":"2024-02-21T23:04:09.632341+0000","last_became_active":"2024-02-21T23:04:09.620423+0000","last_became_peered":"2024-02-21T23:04:09.620423+0000","last_unstale":"2024-02-21T23:04:09.632341+0000","last_undegraded":"2024-02-21T23:04:09.632341+0000","last_fullsized":"2024-02-21T23:04:09.632341+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:27:10.937642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"37'2","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.643378+0000","last_change":"2024-02-21T23:04:08.595297+0000","last_active":"2024-02-21T23:04:09.643378+0000","last_peered":"2024-02-21T23:04:09.643378+0000","last_clean":"2024-02-21T23:04:09.643378+0000","last_became_active":"2024-02-21T23:04:08.594876+0000","last_became_peered":"2024-02-21T23:04:08.594876+0000","last_unstale":"2024-02-21T23:04:09.643378+0000","last_undegraded":"2024-02-21T23:04:09.643378+0000","last_fullsized":"2024-02-21T23:04:09.643378+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_clean_scrub_stamp":"2024-02-21T23:04:07.527107+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:28:31.233414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.611162+0000","last_change":"2024-02-21T23:04:09.602680+0000","last_active":"2024-02-21T23:04:09.611162+0000","last_peered":"2024-02-21T23:04:09.611162+0000","last_clean":"2024-02-21T23:04:09.611162+0000","last_became_active":"2024-02-21T23:04:09.601812+0000","last_became_peered":"2024-02-21T23:04:09.601812+0000","last_unstale":"2024-02-21T23:04:09.611162+0000","last_undegraded":"2024-02-21T23:04:09.611162+0000","last_fullsized":"2024-02-21T23:04:09.611162+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:12:57.961148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"37'1","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-21T23:04:10.602842+0000","last_change":"2024-02-21T23:04:08.588371+0000","last_active":"2024-02-21T23:04:10.602842+0000","last_peered":"2024-02-21T23:04:10.602842+0000","last_clean":"2024-02-21T23:04:10.602842+0000","last_became_active":"2024-02-21T23:04:08.588026+0000","last_became_peered":"2024-02-21T23:04:08.588026+0000","last_unstale":"2024-02-21T23:04:10.602842+0000","last_undegraded":"2024-02-21T23:04:10.602842+0000","last_fullsized":"2024-02-21T23:04:10.602842+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_clean_scrub_stamp":"2024-02-21T23:04:07.527107+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:45:52.618830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-21T23:04:10.603124+0000","last_change":"2024-02-21T23:04:09.606412+0000","last_active":"2024-02-21T23:04:10.603124+0000","last_peered":"2024-02-21T23:04:10.603124+0000","last_clean":"2024-02-21T23:04:10.603124+0000","last_became_active":"2024-02-21T23:04:09.606111+0000","last_became_peered":"2024-02-21T23:04:09.606111+0000","last_unstale":"2024-02-21T23:04:10.603124+0000","last_undegraded":"2024-02-21T23:04:10.603124+0000","last_fullsized":"2024-02-21T23:04:10.603124+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:38:20.998379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.595916+0000","last_change":"2024-02-21T23:04:08.593221+0000","last_active":"2024-02-21T23:04:09.595916+0000","last_peered":"2024-02-21T23:04:09.595916+0000","last_clean":"2024-02-21T23:04:09.595916+0000","last_became_active":"2024-02-21T23:04:08.592964+0000","last_became_peered":"2024-02-21T23:04:08.592964+0000","last_unstale":"2024-02-21T23:04:09.595916+0000","last_undegraded":"2024-02-21T23:04:09.595916+0000","last_fullsized":"2024-02-21T23:04:09.595916+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_clean_scrub_stamp":"2024-02-21T23:04:07.527107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:38:19.982972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.631685+0000","last_change":"2024-02-21T23:04:09.624820+0000","last_active":"2024-02-21T23:04:09.631685+0000","last_peered":"2024-02-21T23:04:09.631685+0000","last_clean":"2024-02-21T23:04:09.631685+0000","last_became_active":"2024-02-21T23:04:09.624549+0000","last_became_peered":"2024-02-21T23:04:09.624549+0000","last_unstale":"2024-02-21T23:04:09.631685+0000","last_undegraded":"2024-02-21T23:04:09.631685+0000","last_fullsized":"2024-02-21T23:04:09.631685+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:12:08.720118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"37'2","reported_seq":16,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-21T23:04:10.602452+0000","last_change":"2024-02-21T23:04:08.573752+0000","last_active":"2024-02-21T23:04:10.602452+0000","last_peered":"2024-02-21T23:04:10.602452+0000","last_clean":"2024-02-21T23:04:10.602452+0000","last_became_active":"2024-02-21T23:04:08.573438+0000","last_became_peered":"2024-02-21T23:04:08.573438+0000","last_unstale":"2024-02-21T23:04:10.602452+0000","last_undegraded":"2024-02-21T23:04:10.602452+0000","last_fullsized":"2024-02-21T23:04:10.602452+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_clean_scrub_stamp":"2024-02-21T23:04:07.527107+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:19:29.864117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.627897+0000","last_change":"2024-02-21T23:04:09.618068+0000","last_active":"2024-02-21T23:04:09.627897+0000","last_peered":"2024-02-21T23:04:09.627897+0000","last_clean":"2024-02-21T23:04:09.627897+0000","last_became_active":"2024-02-21T23:04:09.617807+0000","last_became_peered":"2024-02-21T23:04:09.617807+0000","last_unstale":"2024-02-21T23:04:09.627897+0000","last_undegraded":"2024-02-21T23:04:09.627897+0000","last_fullsized":"2024-02-21T23:04:09.627897+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:03:19.900047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-21T23:04:10.602283+0000","last_change":"2024-02-21T23:04:08.591987+0000","last_active":"2024-02-21T23:04:10.602283+0000","last_peered":"2024-02-21T23:04:10.602283+0000","last_clean":"2024-02-21T23:04:10.602283+0000","last_became_active":"2024-02-21T23:04:08.591713+0000","last_became_peered":"2024-02-21T23:04:08.591713+0000","last_unstale":"2024-02-21T23:04:10.602283+0000","last_undegraded":"2024-02-21T23:04:10.602283+0000","last_fullsized":"2024-02-21T23:04:10.602283+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_clean_scrub_stamp":"2024-02-21T23:04:07.527107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:09:30.103400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.642970+0000","last_change":"2024-02-21T23:04:09.637391+0000","last_active":"2024-02-21T23:04:09.642970+0000","last_peered":"2024-02-21T23:04:09.642970+0000","last_clean":"2024-02-21T23:04:09.642970+0000","last_became_active":"2024-02-21T23:04:09.637047+0000","last_became_peered":"2024-02-21T23:04:09.637047+0000","last_unstale":"2024-02-21T23:04:09.642970+0000","last_undegraded":"2024-02-21T23:04:09.642970+0000","last_fullsized":"2024-02-21T23:04:09.642970+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:29:05.689326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.608808+0000","last_change":"2024-02-21T23:04:08.591002+0000","last_active":"2024-02-21T23:04:09.608808+0000","last_peered":"2024-02-21T23:04:09.608808+0000","last_clean":"2024-02-21T23:04:09.608808+0000","last_became_active":"2024-02-21T23:04:08.589857+0000","last_became_peered":"2024-02-21T23:04:08.589857+0000","last_unstale":"2024-02-21T23:04:09.608808+0000","last_undegraded":"2024-02-21T23:04:09.608808+0000","last_fullsized":"2024-02-21T23:04:09.608808+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_clean_scrub_stamp":"2024-02-21T23:04:07.527107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:38:19.982972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-21T23:04:10.604359+0000","last_change":"2024-02-21T23:04:09.610189+0000","last_active":"2024-02-21T23:04:10.604359+0000","last_peered":"2024-02-21T23:04:10.604359+0000","last_clean":"2024-02-21T23:04:10.604359+0000","last_became_active":"2024-02-21T23:04:09.609881+0000","last_became_peered":"2024-02-21T23:04:09.609881+0000","last_unstale":"2024-02-21T23:04:10.604359+0000","last_undegraded":"2024-02-21T23:04:10.604359+0000","last_fullsized":"2024-02-21T23:04:10.604359+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:25:15.239988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-21T23:04:10.602723+0000","last_change":"2024-02-21T23:04:08.568505+0000","last_active":"2024-02-21T23:04:10.602723+0000","last_peered":"2024-02-21T23:04:10.602723+0000","last_clean":"2024-02-21T23:04:10.602723+0000","last_became_active":"2024-02-21T23:04:08.568196+0000","last_became_peered":"2024-02-21T23:04:08.568196+0000","last_unstale":"2024-02-21T23:04:10.602723+0000","last_undegraded":"2024-02-21T23:04:10.602723+0000","last_fullsized":"2024-02-21T23:04:10.602723+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_clean_scrub_stamp":"2024-02-21T23:04:07.527107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:54:50.559993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.629114+0000","last_change":"2024-02-21T23:04:09.627968+0000","last_active":"2024-02-21T23:04:09.629114+0000","last_peered":"2024-02-21T23:04:09.629114+0000","last_clean":"2024-02-21T23:04:09.629114+0000","last_became_active":"2024-02-21T23:04:09.625013+0000","last_became_peered":"2024-02-21T23:04:09.625013+0000","last_unstale":"2024-02-21T23:04:09.629114+0000","last_undegraded":"2024-02-21T23:04:09.629114+0000","last_fullsized":"2024-02-21T23:04:09.629114+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:52:26.223286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"37'2","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.637143+0000","last_change":"2024-02-21T23:04:08.593913+0000","last_active":"2024-02-21T23:04:09.637143+0000","last_peered":"2024-02-21T23:04:09.637143+0000","last_clean":"2024-02-21T23:04:09.637143+0000","last_became_active":"2024-02-21T23:04:08.593492+0000","last_became_peered":"2024-02-21T23:04:08.593492+0000","last_unstale":"2024-02-21T23:04:09.637143+0000","last_undegraded":"2024-02-21T23:04:09.637143+0000","last_fullsized":"2024-02-21T23:04:09.637143+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_clean_scrub_stamp":"2024-02-21T23:04:07.527107+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:30:43.428924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.641045+0000","last_change":"2024-02-21T23:04:09.635457+0000","last_active":"2024-02-21T23:04:09.641045+0000","last_peered":"2024-02-21T23:04:09.641045+0000","last_clean":"2024-02-21T23:04:09.641045+0000","last_became_active":"2024-02-21T23:04:09.634764+0000","last_became_peered":"2024-02-21T23:04:09.634764+0000","last_unstale":"2024-02-21T23:04:09.641045+0000","last_undegraded":"2024-02-21T23:04:09.641045+0000","last_fullsized":"2024-02-21T23:04:09.641045+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:30:44.444331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-21T23:04:10.602674+0000","last_change":"2024-02-21T23:04:08.592520+0000","last_active":"2024-02-21T23:04:10.602674+0000","last_peered":"2024-02-21T23:04:10.602674+0000","last_clean":"2024-02-21T23:04:10.602674+0000","last_became_active":"2024-02-21T23:04:08.592184+0000","last_became_peered":"2024-02-21T23:04:08.592184+0000","last_unstale":"2024-02-21T23:04:10.602674+0000","last_undegraded":"2024-02-21T23:04:10.602674+0000","last_fullsized":"2024-02-21T23:04:10.602674+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_clean_scrub_stamp":"2024-02-21T23:04:07.527107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:11:38.769235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-21T23:04:10.604765+0000","last_change":"2024-02-21T23:04:09.611514+0000","last_active":"2024-02-21T23:04:10.604765+0000","last_peered":"2024-02-21T23:04:10.604765+0000","last_clean":"2024-02-21T23:04:10.604765+0000","last_became_active":"2024-02-21T23:04:09.611294+0000","last_became_peered":"2024-02-21T23:04:09.611294+0000","last_unstale":"2024-02-21T23:04:10.604765+0000","last_undegraded":"2024-02-21T23:04:10.604765+0000","last_fullsized":"2024-02-21T23:04:10.604765+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:02:09.505024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.592262+0000","last_change":"2024-02-21T23:04:08.583914+0000","last_active":"2024-02-21T23:04:09.592262+0000","last_peered":"2024-02-21T23:04:09.592262+0000","last_clean":"2024-02-21T23:04:09.592262+0000","last_became_active":"2024-02-21T23:04:08.583535+0000","last_became_peered":"2024-02-21T23:04:08.583535+0000","last_unstale":"2024-02-21T23:04:09.592262+0000","last_undegraded":"2024-02-21T23:04:09.592262+0000","last_fullsized":"2024-02-21T23:04:09.592262+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_clean_scrub_stamp":"2024-02-21T23:04:07.527107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:28:31.233414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.614044+0000","last_change":"2024-02-21T23:04:09.605152+0000","last_active":"2024-02-21T23:04:09.614044+0000","last_peered":"2024-02-21T23:04:09.614044+0000","last_clean":"2024-02-21T23:04:09.614044+0000","last_became_active":"2024-02-21T23:04:09.604840+0000","last_became_peered":"2024-02-21T23:04:09.604840+0000","last_unstale":"2024-02-21T23:04:09.614044+0000","last_undegraded":"2024-02-21T23:04:09.614044+0000","last_fullsized":"2024-02-21T23:04:09.614044+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:43:31.902765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.598821+0000","last_change":"2024-02-21T23:04:08.584097+0000","last_active":"2024-02-21T23:04:09.598821+0000","last_peered":"2024-02-21T23:04:09.598821+0000","last_clean":"2024-02-21T23:04:09.598821+0000","last_became_active":"2024-02-21T23:04:08.583773+0000","last_became_peered":"2024-02-21T23:04:08.583773+0000","last_unstale":"2024-02-21T23:04:09.598821+0000","last_undegraded":"2024-02-21T23:04:09.598821+0000","last_fullsized":"2024-02-21T23:04:09.598821+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_clean_scrub_stamp":"2024-02-21T23:04:07.527107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:38:19.982972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.629967+0000","last_change":"2024-02-21T23:04:09.628185+0000","last_active":"2024-02-21T23:04:09.629967+0000","last_peered":"2024-02-21T23:04:09.629967+0000","last_clean":"2024-02-21T23:04:09.629967+0000","last_became_active":"2024-02-21T23:04:09.627879+0000","last_became_peered":"2024-02-21T23:04:09.627879+0000","last_unstale":"2024-02-21T23:04:09.629967+0000","last_undegraded":"2024-02-21T23:04:09.629967+0000","last_fullsized":"2024-02-21T23:04:09.629967+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:48:43.036434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-21T23:04:10.605660+0000","last_change":"2024-02-21T23:04:08.581319+0000","last_active":"2024-02-21T23:04:10.605660+0000","last_peered":"2024-02-21T23:04:10.605660+0000","last_clean":"2024-02-21T23:04:10.605660+0000","last_became_active":"2024-02-21T23:04:08.580400+0000","last_became_peered":"2024-02-21T23:04:08.580400+0000","last_unstale":"2024-02-21T23:04:10.605660+0000","last_undegraded":"2024-02-21T23:04:10.605660+0000","last_fullsized":"2024-02-21T23:04:10.605660+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_clean_scrub_stamp":"2024-02-21T23:04:07.527107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:39:01.282374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.614369+0000","last_change":"2024-02-21T23:04:09.609299+0000","last_active":"2024-02-21T23:04:09.614369+0000","last_peered":"2024-02-21T23:04:09.614369+0000","last_clean":"2024-02-21T23:04:09.614369+0000","last_became_active":"2024-02-21T23:04:09.608982+0000","last_became_peered":"2024-02-21T23:04:09.608982+0000","last_unstale":"2024-02-21T23:04:09.614369+0000","last_undegraded":"2024-02-21T23:04:09.614369+0000","last_fullsized":"2024-02-21T23:04:09.614369+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:39:02.297782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.590753+0000","last_change":"2024-02-21T23:04:08.585533+0000","last_active":"2024-02-21T23:04:09.590753+0000","last_peered":"2024-02-21T23:04:09.590753+0000","last_clean":"2024-02-21T23:04:09.590753+0000","last_became_active":"2024-02-21T23:04:08.585240+0000","last_became_peered":"2024-02-21T23:04:08.585240+0000","last_unstale":"2024-02-21T23:04:09.590753+0000","last_undegraded":"2024-02-21T23:04:09.590753+0000","last_fullsized":"2024-02-21T23:04:09.590753+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_clean_scrub_stamp":"2024-02-21T23:04:07.527107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:47:10.506827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.638668+0000","last_change":"2024-02-21T23:04:09.633061+0000","last_active":"2024-02-21T23:04:09.638668+0000","last_peered":"2024-02-21T23:04:09.638668+0000","last_clean":"2024-02-21T23:04:09.638668+0000","last_became_active":"2024-02-21T23:04:09.632750+0000","last_became_peered":"2024-02-21T23:04:09.632750+0000","last_unstale":"2024-02-21T23:04:09.638668+0000","last_undegraded":"2024-02-21T23:04:09.638668+0000","last_fullsized":"2024-02-21T23:04:09.638668+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:02:30.794997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-21T23:04:10.603700+0000","last_change":"2024-02-21T23:04:08.575057+0000","last_active":"2024-02-21T23:04:10.603700+0000","last_peered":"2024-02-21T23:04:10.603700+0000","last_clean":"2024-02-21T23:04:10.603700+0000","last_became_active":"2024-02-21T23:04:08.574768+0000","last_became_peered":"2024-02-21T23:04:08.574768+0000","last_unstale":"2024-02-21T23:04:10.603700+0000","last_undegraded":"2024-02-21T23:04:10.603700+0000","last_fullsized":"2024-02-21T23:04:10.603700+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_clean_scrub_stamp":"2024-02-21T23:04:07.527107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:17:52.116580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.624574+0000","last_change":"2024-02-21T23:04:09.620488+0000","last_active":"2024-02-21T23:04:09.624574+0000","last_peered":"2024-02-21T23:04:09.624574+0000","last_clean":"2024-02-21T23:04:09.624574+0000","last_became_active":"2024-02-21T23:04:09.620101+0000","last_became_peered":"2024-02-21T23:04:09.620101+0000","last_unstale":"2024-02-21T23:04:09.624574+0000","last_undegraded":"2024-02-21T23:04:09.624574+0000","last_fullsized":"2024-02-21T23:04:09.624574+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:25:43.586312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.613104+0000","last_change":"2024-02-21T23:04:08.590185+0000","last_active":"2024-02-21T23:04:09.613104+0000","last_peered":"2024-02-21T23:04:09.613104+0000","last_clean":"2024-02-21T23:04:09.613104+0000","last_became_active":"2024-02-21T23:04:08.589877+0000","last_became_peered":"2024-02-21T23:04:08.589877+0000","last_unstale":"2024-02-21T23:04:09.613104+0000","last_undegraded":"2024-02-21T23:04:09.613104+0000","last_fullsized":"2024-02-21T23:04:09.613104+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_clean_scrub_stamp":"2024-02-21T23:04:07.527107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:03:18.884640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":201,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.611859+0000","last_change":"2024-02-21T23:02:32.289335+0000","last_active":"2024-02-21T23:04:09.611859+0000","last_peered":"2024-02-21T23:04:09.611859+0000","last_clean":"2024-02-21T23:04:09.611859+0000","last_became_active":"2024-02-21T23:02:32.286913+0000","last_became_peered":"2024-02-21T23:02:32.286913+0000","last_unstale":"2024-02-21T23:04:09.611859+0000","last_undegraded":"2024-02-21T23:04:09.611859+0000","last_fullsized":"2024-02-21T23:04:09.611859+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_clean_scrub_stamp":"2024-02-21T23:02:31.172569+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:36:23.987374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574870,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27516,"kb_used_data":496,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344324,"statfs":{"total":96636764160,"available":96608587776,"internally_reserved":0,"allocated":507904,"data_stored":775394,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574871,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27444,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344396,"statfs":{"total":96636764160,"available":96608661504,"internally_reserved":0,"allocated":430080,"data_stored":183926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574870,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27452,"kb_used_data":424,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344388,"statfs":{"total":96636764160,"available":96608653312,"internally_reserved":0,"allocated":434176,"data_stored":183456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574871,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27452,"kb_used_data":424,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344388,"statfs":{"total":96636764160,"available":96608653312,"internally_reserved":0,"allocated":434176,"data_stored":183948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574871,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31620,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340220,"statfs":{"total":96636764160,"available":96604385280,"internally_reserved":0,"allocated":442368,"data_stored":184990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574872,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31612,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340228,"statfs":{"total":96636764160,"available":96604393472,"internally_reserved":0,"allocated":442368,"data_stored":190106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574870,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27448,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344392,"statfs":{"total":96636764160,"available":96608657408,"internally_reserved":0,"allocated":430080,"data_stored":183434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574870,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31664,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340176,"statfs":{"total":96636764160,"available":96604340224,"internally_reserved":0,"allocated":495616,"data_stored":773746,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-21T23:04:14.052 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-21T23:04:14.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.316+0000 7fc0ea600640 1 Processor -- start 2024-02-21T23:04:14.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.316+0000 7fc0ea600640 1 -- start start 2024-02-21T23:04:14.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.318+0000 7fc0ea600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc0e4159010 0x7fc0e40cb390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:14.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.318+0000 7fc0ea600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0e4155120 con 0x7fc0e41525c0 2024-02-21T23:04:14.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.318+0000 7fc0ea600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0e4155390 con 0x7fc0e40c3280 2024-02-21T23:04:14.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.318+0000 7fc0ea600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e4155670 con 0x7fc0e4159010 2024-02-21T23:04:14.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.318+0000 7fc0e3400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fc0e41525c0 0x7fc0e40c6a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:56680/0 (socket says 172.21.15.28:56680) 2024-02-21T23:04:14.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.318+0000 7fc0e3400640 1 -- 172.21.15.28:0/3064105311 learned_addr learned my addr 172.21.15.28:0/3064105311 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:04:14.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.319+0000 7fc0daa00640 1 --2- 172.21.15.28:0/3064105311 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc0e4159010 0x7fc0e40cb390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:14.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.319+0000 7fc0e2a00640 1 -- 172.21.15.28:0/3064105311 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1372169679 0 0) 0x7fc0e4155120 con 0x7fc0e41525c0 2024-02-21T23:04:14.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.319+0000 7fc0e2a00640 1 -- 172.21.15.28:0/3064105311 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0c8003520 con 0x7fc0e41525c0 2024-02-21T23:04:14.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.319+0000 7fc0e2a00640 1 -- 172.21.15.28:0/3064105311 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2063090080 0 0) 0x7fc0c8003520 con 0x7fc0e41525c0 2024-02-21T23:04:14.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.319+0000 7fc0e2a00640 1 -- 172.21.15.28:0/3064105311 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc0e4159010 msgr2=0x7fc0e40cb390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:14.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.319+0000 7fc0e2a00640 1 --2- 172.21.15.28:0/3064105311 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc0e4159010 0x7fc0e40cb390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:14.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.319+0000 7fc0e2a00640 1 -- 172.21.15.28:0/3064105311 >> v1:172.21.15.81:6789/0 conn(0x7fc0e40c3280 legacy=0x7fc0e40c36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:14.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.319+0000 7fc0e2a00640 1 -- 172.21.15.28:0/3064105311 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0e4155950 con 0x7fc0e41525c0 2024-02-21T23:04:14.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.319+0000 7fc0e2a00640 1 -- 172.21.15.28:0/3064105311 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fc0d40019f0 con 0x7fc0e41525c0 2024-02-21T23:04:14.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.319+0000 7fc0e2a00640 1 -- 172.21.15.28:0/3064105311 >> v1:172.21.15.28:6789/0 conn(0x7fc0e41525c0 legacy=0x7fc0e40c6a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:14.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.319+0000 7fc0e2a00640 1 --2- 172.21.15.28:0/3064105311 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc0c80039e0 0x7fc0c8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:14.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.319+0000 7fc0e2a00640 1 --2- 172.21.15.28:0/3064105311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0c8004390 0x7fc0c8004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:14.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.319+0000 7fc0e2a00640 1 --2- 172.21.15.28:0/3064105311 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0c8005060 0x7fc0c800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:14.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.319+0000 7fc0e2a00640 1 -- 172.21.15.28:0/3064105311 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e4155670 con 0x7fc0c8004390 2024-02-21T23:04:14.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.320+0000 7fc0e2a00640 1 -- 172.21.15.28:0/3064105311 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0c800bf00 con 0x7fc0c8005060 2024-02-21T23:04:14.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.320+0000 7fc0e2a00640 1 -- 172.21.15.28:0/3064105311 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0c800c0d0 con 0x7fc0c80039e0 2024-02-21T23:04:14.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.320+0000 7fc0daa00640 1 --2- 172.21.15.28:0/3064105311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0c8004390 0x7fc0c8004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:14.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.320+0000 7fc0e3e00640 1 --2- 172.21.15.28:0/3064105311 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0c8005060 0x7fc0c800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:14.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.320+0000 7fc0e3400640 1 --2- 172.21.15.28:0/3064105311 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc0c80039e0 0x7fc0c8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:14.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.320+0000 7fc0daa00640 1 -- 172.21.15.28:0/3064105311 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc0c80039e0 msgr2=0x7fc0c8003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:14.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.320+0000 7fc0daa00640 1 --2- 172.21.15.28:0/3064105311 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc0c80039e0 0x7fc0c8003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:14.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.320+0000 7fc0daa00640 1 -- 172.21.15.28:0/3064105311 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0c8005060 msgr2=0x7fc0c800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:14.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.320+0000 7fc0daa00640 1 --2- 172.21.15.28:0/3064105311 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0c8005060 0x7fc0c800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:14.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.320+0000 7fc0daa00640 1 -- 172.21.15.28:0/3064105311 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0c8003520 con 0x7fc0c8004390 2024-02-21T23:04:14.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.322+0000 7fc0daa00640 1 --2- 172.21.15.28:0/3064105311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0c8004390 0x7fc0c8004860 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fc0cc009ac0 tx=0x7fc0c8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28d51105554d27a0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:14.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.322+0000 7fc0e2a00640 1 -- 172.21.15.28:0/3064105311 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0cc039070 con 0x7fc0c8004390 2024-02-21T23:04:14.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.322+0000 7fc0e2a00640 1 -- 172.21.15.28:0/3064105311 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0cc02c6b0 con 0x7fc0c8004390 2024-02-21T23:04:14.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.323+0000 7fc0e2a00640 1 -- 172.21.15.28:0/3064105311 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0cc009690 con 0x7fc0c8004390 2024-02-21T23:04:14.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.323+0000 7fc0ea600640 1 -- 172.21.15.28:0/3064105311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0c8004390 msgr2=0x7fc0c8004860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:14.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.323+0000 7fc0ea600640 1 --2- 172.21.15.28:0/3064105311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0c8004390 0x7fc0c8004860 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fc0cc009ac0 tx=0x7fc0c8005830 comp rx=0 tx=0).stop 2024-02-21T23:04:14.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.323+0000 7fc0e3e00640 1 -- 172.21.15.28:0/3064105311 reap_dead start 2024-02-21T23:04:14.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.323+0000 7fc0ea600640 1 -- 172.21.15.28:0/3064105311 shutdown_connections 2024-02-21T23:04:14.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.323+0000 7fc0ea600640 1 -- 172.21.15.28:0/3064105311 >> 172.21.15.28:0/3064105311 conn(0x7fc0e40729c0 msgr2=0x7fc0e4072dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:14.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.323+0000 7fc0ea600640 1 -- 172.21.15.28:0/3064105311 shutdown_connections 2024-02-21T23:04:14.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.323+0000 7fc0ea600640 1 -- 172.21.15.28:0/3064105311 wait complete. 2024-02-21T23:04:14.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.325+0000 7fc0ea600640 1 Processor -- start 2024-02-21T23:04:14.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.325+0000 7fc0ea600640 1 -- start start 2024-02-21T23:04:14.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.326+0000 7fc0ea600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0c8004390 0x7fc0e41ee850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:14.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.326+0000 7fc0ea600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0e41492e0 0x7fc0e4157fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:14.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.326+0000 7fc0ea600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc0e40c1520 0x7fc0e41ebb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:14.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.326+0000 7fc0ea600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e41594a0 con 0x7fc0c8004390 2024-02-21T23:04:14.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.326+0000 7fc0ea600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e40caff0 con 0x7fc0e41492e0 2024-02-21T23:04:14.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.326+0000 7fc0ea600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e40c7110 con 0x7fc0e40c1520 2024-02-21T23:04:14.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.326+0000 7fc0e3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0c8004390 0x7fc0e41ee850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:14.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.326+0000 7fc0e3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0c8004390 0x7fc0e41ee850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56426/0 (socket says 172.21.15.28:56426) 2024-02-21T23:04:14.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.326+0000 7fc0e3400640 1 -- 172.21.15.28:0/384813996 learned_addr learned my addr 172.21.15.28:0/384813996 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:14.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.327+0000 7fc0e3400640 1 -- 172.21.15.28:0/384813996 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc0e40c1520 msgr2=0x7fc0e41ebb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:14.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.327+0000 7fc0e3400640 1 --2- 172.21.15.28:0/384813996 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc0e40c1520 0x7fc0e41ebb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:14.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.327+0000 7fc0e3400640 1 -- 172.21.15.28:0/384813996 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0e41492e0 msgr2=0x7fc0e4157fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:14.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.327+0000 7fc0e3400640 1 --2- 172.21.15.28:0/384813996 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0e41492e0 0x7fc0e4157fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:14.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.327+0000 7fc0e3400640 1 -- 172.21.15.28:0/384813996 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0e41eb640 con 0x7fc0c8004390 2024-02-21T23:04:14.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.327+0000 7fc0e3400640 1 --2- 172.21.15.28:0/384813996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0c8004390 0x7fc0e41ee850 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fc0d4007ac0 tx=0x7fc0d400e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:14.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.327+0000 7fc0dbe00640 1 -- 172.21.15.28:0/384813996 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0d400ee50 con 0x7fc0c8004390 2024-02-21T23:04:14.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.327+0000 7fc0dbe00640 1 -- 172.21.15.28:0/384813996 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0d4018070 con 0x7fc0c8004390 2024-02-21T23:04:14.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.327+0000 7fc0dbe00640 1 -- 172.21.15.28:0/384813996 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0d4013660 con 0x7fc0c8004390 2024-02-21T23:04:14.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.327+0000 7fc0ea600640 1 -- 172.21.15.28:0/384813996 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0e41eb360 con 0x7fc0c8004390 2024-02-21T23:04:14.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.327+0000 7fc0ea600640 1 -- 172.21.15.28:0/384813996 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0e41ee050 con 0x7fc0c8004390 2024-02-21T23:04:14.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.329+0000 7fc0ea600640 1 -- 172.21.15.28:0/384813996 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0e4067f30 con 0x7fc0c8004390 2024-02-21T23:04:14.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.329+0000 7fc0dbe00640 1 -- 172.21.15.28:0/384813996 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc0d4013800 con 0x7fc0c8004390 2024-02-21T23:04:14.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.330+0000 7fc0dbe00640 1 --2- 172.21.15.28:0/384813996 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc0c006b4d0 0x7fc0c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:14.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.330+0000 7fc0dbe00640 1 -- 172.21.15.28:0/384813996 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc0d408aaf0 con 0x7fc0c8004390 2024-02-21T23:04:14.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.331+0000 7fc0daa00640 1 --2- 172.21.15.28:0/384813996 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc0c006b4d0 0x7fc0c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:14.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.332+0000 7fc0daa00640 1 --2- 172.21.15.28:0/384813996 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc0c006b4d0 0x7fc0c006d990 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fc0cc036000 tx=0x7fc0cc02e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:14.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.335+0000 7fc0dbe00640 1 -- 172.21.15.28:0/384813996 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc0d40597d0 con 0x7fc0c8004390 2024-02-21T23:04:14.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.531+0000 7fc0ea600640 1 -- 172.21.15.28:0/384813996 --> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc0e41eb110 con 0x7fc0c006b4d0 2024-02-21T23:04:14.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.550+0000 7fc0dbe00640 1 -- 172.21.15.28:0/384813996 <== mgr.4101 v2:172.21.15.81:6832/1848538577 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251984 (secure 0 0 0) 0x7fc0e41eb110 con 0x7fc0c006b4d0 2024-02-21T23:04:14.542 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:04:14.543 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-21T23:04:14.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.555+0000 7fc0ea600640 1 -- 172.21.15.28:0/384813996 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc0c006b4d0 msgr2=0x7fc0c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:14.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.555+0000 7fc0ea600640 1 --2- 172.21.15.28:0/384813996 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc0c006b4d0 0x7fc0c006d990 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fc0cc036000 tx=0x7fc0cc02e000 comp rx=0 tx=0).stop 2024-02-21T23:04:14.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.555+0000 7fc0ea600640 1 -- 172.21.15.28:0/384813996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0c8004390 msgr2=0x7fc0e41ee850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:14.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.555+0000 7fc0ea600640 1 --2- 172.21.15.28:0/384813996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0c8004390 0x7fc0e41ee850 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fc0d4007ac0 tx=0x7fc0d400e530 comp rx=0 tx=0).stop 2024-02-21T23:04:14.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.555+0000 7fc0ea600640 1 -- 172.21.15.28:0/384813996 shutdown_connections 2024-02-21T23:04:14.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.555+0000 7fc0ea600640 1 --2- 172.21.15.28:0/384813996 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc0c006b4d0 0x7fc0c006d990 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:14.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.555+0000 7fc0ea600640 1 --2- 172.21.15.28:0/384813996 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc0e40c1520 0x7fc0e41ebb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:14.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.555+0000 7fc0ea600640 1 --2- 172.21.15.28:0/384813996 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0e41492e0 0x7fc0e4157fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:14.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.556+0000 7fc0ea600640 1 --2- 172.21.15.28:0/384813996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0c8004390 0x7fc0e41ee850 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:14.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.556+0000 7fc0ea600640 1 -- 172.21.15.28:0/384813996 >> 172.21.15.28:0/384813996 conn(0x7fc0e40729c0 msgr2=0x7fc0e40c5240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:14.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.556+0000 7fc0ea600640 1 -- 172.21.15.28:0/384813996 shutdown_connections 2024-02-21T23:04:14.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.556+0000 7fc0ea600640 1 -- 172.21.15.28:0/384813996 wait complete. 2024-02-21T23:04:14.564 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":82,"stamp":"2024-02-21T23:04:12.694083+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":95,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":232208,"kb_used_data":3532,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":754742512,"statfs":{"total":773094113280,"available":772856332288,"internally_reserved":0,"allocated":3616768,"data_stored":2659000,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":234016305},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.149879"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-21T23:04:10.604997+0000","last_change":"2024-02-21T23:04:09.604886+0000","last_active":"2024-02-21T23:04:10.604997+0000","last_peered":"2024-02-21T23:04:10.604997+0000","last_clean":"2024-02-21T23:04:10.604997+0000","last_became_active":"2024-02-21T23:04:09.604561+0000","last_became_peered":"2024-02-21T23:04:09.604561+0000","last_unstale":"2024-02-21T23:04:10.604997+0000","last_undegraded":"2024-02-21T23:04:10.604997+0000","last_fullsized":"2024-02-21T23:04:10.604997+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:59:36.867318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-21T23:04:10.604730+0000","last_change":"2024-02-21T23:04:09.604955+0000","last_active":"2024-02-21T23:04:10.604730+0000","last_peered":"2024-02-21T23:04:10.604730+0000","last_clean":"2024-02-21T23:04:10.604730+0000","last_became_active":"2024-02-21T23:04:09.604618+0000","last_became_peered":"2024-02-21T23:04:09.604618+0000","last_unstale":"2024-02-21T23:04:10.604730+0000","last_undegraded":"2024-02-21T23:04:10.604730+0000","last_fullsized":"2024-02-21T23:04:10.604730+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:11:19.186546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.628842+0000","last_change":"2024-02-21T23:04:09.626315+0000","last_active":"2024-02-21T23:04:09.628842+0000","last_peered":"2024-02-21T23:04:09.628842+0000","last_clean":"2024-02-21T23:04:09.628842+0000","last_became_active":"2024-02-21T23:04:09.619401+0000","last_became_peered":"2024-02-21T23:04:09.619401+0000","last_unstale":"2024-02-21T23:04:09.628842+0000","last_undegraded":"2024-02-21T23:04:09.628842+0000","last_fullsized":"2024-02-21T23:04:09.628842+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:52:26.223286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-21T23:04:10.602740+0000","last_change":"2024-02-21T23:04:09.609287+0000","last_active":"2024-02-21T23:04:10.602740+0000","last_peered":"2024-02-21T23:04:10.602740+0000","last_clean":"2024-02-21T23:04:10.602740+0000","last_became_active":"2024-02-21T23:04:09.608998+0000","last_became_peered":"2024-02-21T23:04:09.608998+0000","last_unstale":"2024-02-21T23:04:10.602740+0000","last_undegraded":"2024-02-21T23:04:10.602740+0000","last_fullsized":"2024-02-21T23:04:10.602740+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:35:59.346373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-21T23:04:10.602252+0000","last_change":"2024-02-21T23:04:09.602470+0000","last_active":"2024-02-21T23:04:10.602252+0000","last_peered":"2024-02-21T23:04:10.602252+0000","last_clean":"2024-02-21T23:04:10.602252+0000","last_became_active":"2024-02-21T23:04:09.602202+0000","last_became_peered":"2024-02-21T23:04:09.602202+0000","last_unstale":"2024-02-21T23:04:10.602252+0000","last_undegraded":"2024-02-21T23:04:10.602252+0000","last_fullsized":"2024-02-21T23:04:10.602252+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:31:37.302983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.614726+0000","last_change":"2024-02-21T23:04:09.609901+0000","last_active":"2024-02-21T23:04:09.614726+0000","last_peered":"2024-02-21T23:04:09.614726+0000","last_clean":"2024-02-21T23:04:09.614726+0000","last_became_active":"2024-02-21T23:04:09.609546+0000","last_became_peered":"2024-02-21T23:04:09.609546+0000","last_unstale":"2024-02-21T23:04:09.614726+0000","last_undegraded":"2024-02-21T23:04:09.614726+0000","last_fullsized":"2024-02-21T23:04:09.614726+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:12:08.720118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.625920+0000","last_change":"2024-02-21T23:04:09.617619+0000","last_active":"2024-02-21T23:04:09.625920+0000","last_peered":"2024-02-21T23:04:09.625920+0000","last_clean":"2024-02-21T23:04:09.625920+0000","last_became_active":"2024-02-21T23:04:09.617298+0000","last_became_peered":"2024-02-21T23:04:09.617298+0000","last_unstale":"2024-02-21T23:04:09.625920+0000","last_undegraded":"2024-02-21T23:04:09.625920+0000","last_fullsized":"2024-02-21T23:04:09.625920+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:20:21.010019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-21T23:04:10.604019+0000","last_change":"2024-02-21T23:04:09.615381+0000","last_active":"2024-02-21T23:04:10.604019+0000","last_peered":"2024-02-21T23:04:10.604019+0000","last_clean":"2024-02-21T23:04:10.604019+0000","last_became_active":"2024-02-21T23:04:09.615032+0000","last_became_peered":"2024-02-21T23:04:09.615032+0000","last_unstale":"2024-02-21T23:04:10.604019+0000","last_undegraded":"2024-02-21T23:04:10.604019+0000","last_fullsized":"2024-02-21T23:04:10.604019+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:45:51.879737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.628818+0000","last_change":"2024-02-21T23:04:09.625511+0000","last_active":"2024-02-21T23:04:09.628818+0000","last_peered":"2024-02-21T23:04:09.628818+0000","last_clean":"2024-02-21T23:04:09.628818+0000","last_became_active":"2024-02-21T23:04:09.621287+0000","last_became_peered":"2024-02-21T23:04:09.621287+0000","last_unstale":"2024-02-21T23:04:09.628818+0000","last_undegraded":"2024-02-21T23:04:09.628818+0000","last_fullsized":"2024-02-21T23:04:09.628818+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:42:21.080709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.628274+0000","last_change":"2024-02-21T23:04:09.621922+0000","last_active":"2024-02-21T23:04:09.628274+0000","last_peered":"2024-02-21T23:04:09.628274+0000","last_clean":"2024-02-21T23:04:09.628274+0000","last_became_active":"2024-02-21T23:04:09.621640+0000","last_became_peered":"2024-02-21T23:04:09.621640+0000","last_unstale":"2024-02-21T23:04:09.628274+0000","last_undegraded":"2024-02-21T23:04:09.628274+0000","last_fullsized":"2024-02-21T23:04:09.628274+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:48:20.132275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.628495+0000","last_change":"2024-02-21T23:04:09.622325+0000","last_active":"2024-02-21T23:04:09.628495+0000","last_peered":"2024-02-21T23:04:09.628495+0000","last_clean":"2024-02-21T23:04:09.628495+0000","last_became_active":"2024-02-21T23:04:09.619954+0000","last_became_peered":"2024-02-21T23:04:09.619954+0000","last_unstale":"2024-02-21T23:04:09.628495+0000","last_undegraded":"2024-02-21T23:04:09.628495+0000","last_fullsized":"2024-02-21T23:04:09.628495+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:02:35.839059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.601874+0000","last_change":"2024-02-21T23:04:09.599332+0000","last_active":"2024-02-21T23:04:09.601874+0000","last_peered":"2024-02-21T23:04:09.601874+0000","last_clean":"2024-02-21T23:04:09.601874+0000","last_became_active":"2024-02-21T23:04:09.599077+0000","last_became_peered":"2024-02-21T23:04:09.599077+0000","last_unstale":"2024-02-21T23:04:09.601874+0000","last_undegraded":"2024-02-21T23:04:09.601874+0000","last_fullsized":"2024-02-21T23:04:09.601874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:39:02.297782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.623989+0000","last_change":"2024-02-21T23:04:09.619500+0000","last_active":"2024-02-21T23:04:09.623989+0000","last_peered":"2024-02-21T23:04:09.623989+0000","last_clean":"2024-02-21T23:04:09.623989+0000","last_became_active":"2024-02-21T23:04:09.618627+0000","last_became_peered":"2024-02-21T23:04:09.618627+0000","last_unstale":"2024-02-21T23:04:09.623989+0000","last_undegraded":"2024-02-21T23:04:09.623989+0000","last_fullsized":"2024-02-21T23:04:09.623989+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:21:27.477217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.635155+0000","last_change":"2024-02-21T23:04:09.628339+0000","last_active":"2024-02-21T23:04:09.635155+0000","last_peered":"2024-02-21T23:04:09.635155+0000","last_clean":"2024-02-21T23:04:09.635155+0000","last_became_active":"2024-02-21T23:04:09.628001+0000","last_became_peered":"2024-02-21T23:04:09.628001+0000","last_unstale":"2024-02-21T23:04:09.635155+0000","last_undegraded":"2024-02-21T23:04:09.635155+0000","last_fullsized":"2024-02-21T23:04:09.635155+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:12:08.720118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.626076+0000","last_change":"2024-02-21T23:04:09.619253+0000","last_active":"2024-02-21T23:04:09.626076+0000","last_peered":"2024-02-21T23:04:09.626076+0000","last_clean":"2024-02-21T23:04:09.626076+0000","last_became_active":"2024-02-21T23:04:09.618395+0000","last_became_peered":"2024-02-21T23:04:09.618395+0000","last_unstale":"2024-02-21T23:04:09.626076+0000","last_undegraded":"2024-02-21T23:04:09.626076+0000","last_fullsized":"2024-02-21T23:04:09.626076+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:17:53.131987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.631707+0000","last_change":"2024-02-21T23:04:09.625455+0000","last_active":"2024-02-21T23:04:09.631707+0000","last_peered":"2024-02-21T23:04:09.631707+0000","last_clean":"2024-02-21T23:04:09.631707+0000","last_became_active":"2024-02-21T23:04:09.625173+0000","last_became_peered":"2024-02-21T23:04:09.625173+0000","last_unstale":"2024-02-21T23:04:09.631707+0000","last_undegraded":"2024-02-21T23:04:09.631707+0000","last_fullsized":"2024-02-21T23:04:09.631707+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:41:02.807550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.635555+0000","last_change":"2024-02-21T23:04:09.630271+0000","last_active":"2024-02-21T23:04:09.635555+0000","last_peered":"2024-02-21T23:04:09.635555+0000","last_clean":"2024-02-21T23:04:09.635555+0000","last_became_active":"2024-02-21T23:04:09.629967+0000","last_became_peered":"2024-02-21T23:04:09.629967+0000","last_unstale":"2024-02-21T23:04:09.635555+0000","last_undegraded":"2024-02-21T23:04:09.635555+0000","last_fullsized":"2024-02-21T23:04:09.635555+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:42:58.911438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-21T23:04:10.606737+0000","last_change":"2024-02-21T23:04:09.618559+0000","last_active":"2024-02-21T23:04:10.606737+0000","last_peered":"2024-02-21T23:04:10.606737+0000","last_clean":"2024-02-21T23:04:10.606737+0000","last_became_active":"2024-02-21T23:04:09.618278+0000","last_became_peered":"2024-02-21T23:04:09.618278+0000","last_unstale":"2024-02-21T23:04:10.606737+0000","last_undegraded":"2024-02-21T23:04:10.606737+0000","last_fullsized":"2024-02-21T23:04:10.606737+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:42:58.911438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-21T23:04:10.602699+0000","last_change":"2024-02-21T23:04:09.597652+0000","last_active":"2024-02-21T23:04:10.602699+0000","last_peered":"2024-02-21T23:04:10.602699+0000","last_clean":"2024-02-21T23:04:10.602699+0000","last_became_active":"2024-02-21T23:04:09.597388+0000","last_became_peered":"2024-02-21T23:04:09.597388+0000","last_unstale":"2024-02-21T23:04:10.602699+0000","last_undegraded":"2024-02-21T23:04:10.602699+0000","last_fullsized":"2024-02-21T23:04:10.602699+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:28:32.248822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-21T23:04:10.606330+0000","last_change":"2024-02-21T23:04:09.619916+0000","last_active":"2024-02-21T23:04:10.606330+0000","last_peered":"2024-02-21T23:04:10.606330+0000","last_clean":"2024-02-21T23:04:10.606330+0000","last_became_active":"2024-02-21T23:04:09.617098+0000","last_became_peered":"2024-02-21T23:04:09.617098+0000","last_unstale":"2024-02-21T23:04:10.606330+0000","last_undegraded":"2024-02-21T23:04:10.606330+0000","last_fullsized":"2024-02-21T23:04:10.606330+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:52:26.223286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.624011+0000","last_change":"2024-02-21T23:04:09.621231+0000","last_active":"2024-02-21T23:04:09.624011+0000","last_peered":"2024-02-21T23:04:09.624011+0000","last_clean":"2024-02-21T23:04:09.624011+0000","last_became_active":"2024-02-21T23:04:09.620885+0000","last_became_peered":"2024-02-21T23:04:09.620885+0000","last_unstale":"2024-02-21T23:04:09.624011+0000","last_undegraded":"2024-02-21T23:04:09.624011+0000","last_fullsized":"2024-02-21T23:04:09.624011+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:57:41.332804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.614227+0000","last_change":"2024-02-21T23:04:09.607445+0000","last_active":"2024-02-21T23:04:09.614227+0000","last_peered":"2024-02-21T23:04:09.614227+0000","last_clean":"2024-02-21T23:04:09.614227+0000","last_became_active":"2024-02-21T23:04:09.606419+0000","last_became_peered":"2024-02-21T23:04:09.606419+0000","last_unstale":"2024-02-21T23:04:09.614227+0000","last_undegraded":"2024-02-21T23:04:09.614227+0000","last_fullsized":"2024-02-21T23:04:09.614227+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:38:20.998379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.633803+0000","last_change":"2024-02-21T23:04:09.629852+0000","last_active":"2024-02-21T23:04:09.633803+0000","last_peered":"2024-02-21T23:04:09.633803+0000","last_clean":"2024-02-21T23:04:09.633803+0000","last_became_active":"2024-02-21T23:04:09.629479+0000","last_became_peered":"2024-02-21T23:04:09.629479+0000","last_unstale":"2024-02-21T23:04:09.633803+0000","last_undegraded":"2024-02-21T23:04:09.633803+0000","last_fullsized":"2024-02-21T23:04:09.633803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:48:43.036434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-21T23:04:10.602399+0000","last_change":"2024-02-21T23:04:09.621204+0000","last_active":"2024-02-21T23:04:10.602399+0000","last_peered":"2024-02-21T23:04:10.602399+0000","last_clean":"2024-02-21T23:04:10.602399+0000","last_became_active":"2024-02-21T23:04:09.620125+0000","last_became_peered":"2024-02-21T23:04:09.620125+0000","last_unstale":"2024-02-21T23:04:10.602399+0000","last_undegraded":"2024-02-21T23:04:10.602399+0000","last_fullsized":"2024-02-21T23:04:10.602399+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:47:11.522234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-21T23:04:10.606543+0000","last_change":"2024-02-21T23:04:09.618492+0000","last_active":"2024-02-21T23:04:10.606543+0000","last_peered":"2024-02-21T23:04:10.606543+0000","last_clean":"2024-02-21T23:04:10.606543+0000","last_became_active":"2024-02-21T23:04:09.617606+0000","last_became_peered":"2024-02-21T23:04:09.617606+0000","last_unstale":"2024-02-21T23:04:10.606543+0000","last_undegraded":"2024-02-21T23:04:10.606543+0000","last_fullsized":"2024-02-21T23:04:10.606543+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:48:12.283971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.627664+0000","last_change":"2024-02-21T23:04:09.621098+0000","last_active":"2024-02-21T23:04:09.627664+0000","last_peered":"2024-02-21T23:04:09.627664+0000","last_clean":"2024-02-21T23:04:09.627664+0000","last_became_active":"2024-02-21T23:04:09.620752+0000","last_became_peered":"2024-02-21T23:04:09.620752+0000","last_unstale":"2024-02-21T23:04:09.627664+0000","last_undegraded":"2024-02-21T23:04:09.627664+0000","last_fullsized":"2024-02-21T23:04:09.627664+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:41:02.807550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-21T23:04:10.606501+0000","last_change":"2024-02-21T23:04:09.625567+0000","last_active":"2024-02-21T23:04:10.606501+0000","last_peered":"2024-02-21T23:04:10.606501+0000","last_clean":"2024-02-21T23:04:10.606501+0000","last_became_active":"2024-02-21T23:04:09.625194+0000","last_became_peered":"2024-02-21T23:04:09.625194+0000","last_unstale":"2024-02-21T23:04:10.606501+0000","last_undegraded":"2024-02-21T23:04:10.606501+0000","last_fullsized":"2024-02-21T23:04:10.606501+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:48:43.036434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.625563+0000","last_change":"2024-02-21T23:04:09.620496+0000","last_active":"2024-02-21T23:04:09.625563+0000","last_peered":"2024-02-21T23:04:09.625563+0000","last_clean":"2024-02-21T23:04:09.625563+0000","last_became_active":"2024-02-21T23:04:09.620193+0000","last_became_peered":"2024-02-21T23:04:09.620193+0000","last_unstale":"2024-02-21T23:04:09.625563+0000","last_undegraded":"2024-02-21T23:04:09.625563+0000","last_fullsized":"2024-02-21T23:04:09.625563+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:59:38.223730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-21T23:04:10.605907+0000","last_change":"2024-02-21T23:04:09.623456+0000","last_active":"2024-02-21T23:04:10.605907+0000","last_peered":"2024-02-21T23:04:10.605907+0000","last_clean":"2024-02-21T23:04:10.605907+0000","last_became_active":"2024-02-21T23:04:09.623126+0000","last_became_peered":"2024-02-21T23:04:09.623126+0000","last_unstale":"2024-02-21T23:04:10.605907+0000","last_undegraded":"2024-02-21T23:04:10.605907+0000","last_fullsized":"2024-02-21T23:04:10.605907+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:42:21.080709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-21T23:04:10.605874+0000","last_change":"2024-02-21T23:04:09.626410+0000","last_active":"2024-02-21T23:04:10.605874+0000","last_peered":"2024-02-21T23:04:10.605874+0000","last_clean":"2024-02-21T23:04:10.605874+0000","last_became_active":"2024-02-21T23:04:09.625585+0000","last_became_peered":"2024-02-21T23:04:09.625585+0000","last_unstale":"2024-02-21T23:04:10.605874+0000","last_undegraded":"2024-02-21T23:04:10.605874+0000","last_fullsized":"2024-02-21T23:04:10.605874+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:41:02.807550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.633783+0000","last_change":"2024-02-21T23:04:09.629688+0000","last_active":"2024-02-21T23:04:09.633783+0000","last_peered":"2024-02-21T23:04:09.633783+0000","last_clean":"2024-02-21T23:04:09.633783+0000","last_became_active":"2024-02-21T23:04:09.624392+0000","last_became_peered":"2024-02-21T23:04:09.624392+0000","last_unstale":"2024-02-21T23:04:09.633783+0000","last_undegraded":"2024-02-21T23:04:09.633783+0000","last_fullsized":"2024-02-21T23:04:09.633783+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:52:26.223286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.605803+0000","last_change":"2024-02-21T23:04:09.603310+0000","last_active":"2024-02-21T23:04:09.605803+0000","last_peered":"2024-02-21T23:04:09.605803+0000","last_clean":"2024-02-21T23:04:09.605803+0000","last_became_active":"2024-02-21T23:04:09.603009+0000","last_became_peered":"2024-02-21T23:04:09.603009+0000","last_unstale":"2024-02-21T23:04:09.605803+0000","last_undegraded":"2024-02-21T23:04:09.605803+0000","last_fullsized":"2024-02-21T23:04:09.605803+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:03:19.900047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.633758+0000","last_change":"2024-02-21T23:04:09.629888+0000","last_active":"2024-02-21T23:04:09.633758+0000","last_peered":"2024-02-21T23:04:09.633758+0000","last_clean":"2024-02-21T23:04:09.633758+0000","last_became_active":"2024-02-21T23:04:09.629432+0000","last_became_peered":"2024-02-21T23:04:09.629432+0000","last_unstale":"2024-02-21T23:04:09.633758+0000","last_undegraded":"2024-02-21T23:04:09.633758+0000","last_fullsized":"2024-02-21T23:04:09.633758+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:48:12.283971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"37'4","reported_seq":18,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-21T23:04:10.606444+0000","last_change":"2024-02-21T23:04:08.587883+0000","last_active":"2024-02-21T23:04:10.606444+0000","last_peered":"2024-02-21T23:04:10.606444+0000","last_clean":"2024-02-21T23:04:10.606444+0000","last_became_active":"2024-02-21T23:04:08.587362+0000","last_became_peered":"2024-02-21T23:04:08.587362+0000","last_unstale":"2024-02-21T23:04:10.606444+0000","last_undegraded":"2024-02-21T23:04:10.606444+0000","last_fullsized":"2024-02-21T23:04:10.606444+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_clean_scrub_stamp":"2024-02-21T23:04:07.527107+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:12:56.945740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.628609+0000","last_change":"2024-02-21T23:04:09.621971+0000","last_active":"2024-02-21T23:04:09.628609+0000","last_peered":"2024-02-21T23:04:09.628609+0000","last_clean":"2024-02-21T23:04:09.628609+0000","last_became_active":"2024-02-21T23:04:09.621550+0000","last_became_peered":"2024-02-21T23:04:09.621550+0000","last_unstale":"2024-02-21T23:04:09.628609+0000","last_undegraded":"2024-02-21T23:04:09.628609+0000","last_fullsized":"2024-02-21T23:04:09.628609+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:42:21.080709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"37'1","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.631605+0000","last_change":"2024-02-21T23:04:08.590209+0000","last_active":"2024-02-21T23:04:09.631605+0000","last_peered":"2024-02-21T23:04:09.631605+0000","last_clean":"2024-02-21T23:04:09.631605+0000","last_became_active":"2024-02-21T23:04:08.589854+0000","last_became_peered":"2024-02-21T23:04:08.589854+0000","last_unstale":"2024-02-21T23:04:09.631605+0000","last_undegraded":"2024-02-21T23:04:09.631605+0000","last_fullsized":"2024-02-21T23:04:09.631605+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_clean_scrub_stamp":"2024-02-21T23:04:07.527107+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:35:58.330966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.640553+0000","last_change":"2024-02-21T23:04:09.636467+0000","last_active":"2024-02-21T23:04:09.640553+0000","last_peered":"2024-02-21T23:04:09.640553+0000","last_clean":"2024-02-21T23:04:09.640553+0000","last_became_active":"2024-02-21T23:04:09.636180+0000","last_became_peered":"2024-02-21T23:04:09.636180+0000","last_unstale":"2024-02-21T23:04:09.640553+0000","last_undegraded":"2024-02-21T23:04:09.640553+0000","last_fullsized":"2024-02-21T23:04:09.640553+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:41:02.807550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"37'2","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.637784+0000","last_change":"2024-02-21T23:04:08.590700+0000","last_active":"2024-02-21T23:04:09.637784+0000","last_peered":"2024-02-21T23:04:09.637784+0000","last_clean":"2024-02-21T23:04:09.637784+0000","last_became_active":"2024-02-21T23:04:08.590305+0000","last_became_peered":"2024-02-21T23:04:08.590305+0000","last_unstale":"2024-02-21T23:04:09.637784+0000","last_undegraded":"2024-02-21T23:04:09.637784+0000","last_fullsized":"2024-02-21T23:04:09.637784+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_clean_scrub_stamp":"2024-02-21T23:04:07.527107+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:39:01.282374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-21T23:04:10.604884+0000","last_change":"2024-02-21T23:04:09.634662+0000","last_active":"2024-02-21T23:04:10.604884+0000","last_peered":"2024-02-21T23:04:10.604884+0000","last_clean":"2024-02-21T23:04:10.604884+0000","last_became_active":"2024-02-21T23:04:09.634326+0000","last_became_peered":"2024-02-21T23:04:09.634326+0000","last_unstale":"2024-02-21T23:04:10.604884+0000","last_undegraded":"2024-02-21T23:04:10.604884+0000","last_fullsized":"2024-02-21T23:04:10.604884+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:54:44.049051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"37'2","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.643221+0000","last_change":"2024-02-21T23:04:08.588381+0000","last_active":"2024-02-21T23:04:09.643221+0000","last_peered":"2024-02-21T23:04:09.643221+0000","last_clean":"2024-02-21T23:04:09.643221+0000","last_became_active":"2024-02-21T23:04:08.588053+0000","last_became_peered":"2024-02-21T23:04:08.588053+0000","last_unstale":"2024-02-21T23:04:09.643221+0000","last_undegraded":"2024-02-21T23:04:09.643221+0000","last_fullsized":"2024-02-21T23:04:09.643221+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_clean_scrub_stamp":"2024-02-21T23:04:07.527107+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:35:58.330966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.603188+0000","last_change":"2024-02-21T23:04:09.596646+0000","last_active":"2024-02-21T23:04:09.603188+0000","last_peered":"2024-02-21T23:04:09.603188+0000","last_clean":"2024-02-21T23:04:09.603188+0000","last_became_active":"2024-02-21T23:04:09.596300+0000","last_became_peered":"2024-02-21T23:04:09.596300+0000","last_unstale":"2024-02-21T23:04:09.603188+0000","last_undegraded":"2024-02-21T23:04:09.603188+0000","last_fullsized":"2024-02-21T23:04:09.603188+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:12:57.961148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"37'1","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.632253+0000","last_change":"2024-02-21T23:04:08.591545+0000","last_active":"2024-02-21T23:04:09.632253+0000","last_peered":"2024-02-21T23:04:09.632253+0000","last_clean":"2024-02-21T23:04:09.632253+0000","last_became_active":"2024-02-21T23:04:08.591190+0000","last_became_peered":"2024-02-21T23:04:08.591190+0000","last_unstale":"2024-02-21T23:04:09.632253+0000","last_undegraded":"2024-02-21T23:04:09.632253+0000","last_fullsized":"2024-02-21T23:04:09.632253+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_clean_scrub_stamp":"2024-02-21T23:04:07.527107+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:42:57.896031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.626098+0000","last_change":"2024-02-21T23:04:09.620887+0000","last_active":"2024-02-21T23:04:09.626098+0000","last_peered":"2024-02-21T23:04:09.626098+0000","last_clean":"2024-02-21T23:04:09.626098+0000","last_became_active":"2024-02-21T23:04:09.620572+0000","last_became_peered":"2024-02-21T23:04:09.620572+0000","last_unstale":"2024-02-21T23:04:09.626098+0000","last_undegraded":"2024-02-21T23:04:09.626098+0000","last_fullsized":"2024-02-21T23:04:09.626098+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:28:58.719324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"37'1","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.638555+0000","last_change":"2024-02-21T23:04:08.599221+0000","last_active":"2024-02-21T23:04:09.638555+0000","last_peered":"2024-02-21T23:04:09.638555+0000","last_clean":"2024-02-21T23:04:09.638555+0000","last_became_active":"2024-02-21T23:04:08.598877+0000","last_became_peered":"2024-02-21T23:04:08.598877+0000","last_unstale":"2024-02-21T23:04:09.638555+0000","last_undegraded":"2024-02-21T23:04:09.638555+0000","last_fullsized":"2024-02-21T23:04:09.638555+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_clean_scrub_stamp":"2024-02-21T23:04:07.527107+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:43:30.887358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.633331+0000","last_change":"2024-02-21T23:04:09.623815+0000","last_active":"2024-02-21T23:04:09.633331+0000","last_peered":"2024-02-21T23:04:09.633331+0000","last_clean":"2024-02-21T23:04:09.633331+0000","last_became_active":"2024-02-21T23:04:09.623444+0000","last_became_peered":"2024-02-21T23:04:09.623444+0000","last_unstale":"2024-02-21T23:04:09.633331+0000","last_undegraded":"2024-02-21T23:04:09.633331+0000","last_fullsized":"2024-02-21T23:04:09.633331+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:03:19.900047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.595582+0000","last_change":"2024-02-21T23:04:08.590739+0000","last_active":"2024-02-21T23:04:09.595582+0000","last_peered":"2024-02-21T23:04:09.595582+0000","last_clean":"2024-02-21T23:04:09.595582+0000","last_became_active":"2024-02-21T23:04:08.590119+0000","last_became_peered":"2024-02-21T23:04:08.590119+0000","last_unstale":"2024-02-21T23:04:09.595582+0000","last_undegraded":"2024-02-21T23:04:09.595582+0000","last_fullsized":"2024-02-21T23:04:09.595582+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_clean_scrub_stamp":"2024-02-21T23:04:07.527107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:12:56.945740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-21T23:04:10.604076+0000","last_change":"2024-02-21T23:04:09.610542+0000","last_active":"2024-02-21T23:04:10.604076+0000","last_peered":"2024-02-21T23:04:10.604076+0000","last_clean":"2024-02-21T23:04:10.604076+0000","last_became_active":"2024-02-21T23:04:09.610322+0000","last_became_peered":"2024-02-21T23:04:09.610322+0000","last_unstale":"2024-02-21T23:04:10.604076+0000","last_undegraded":"2024-02-21T23:04:10.604076+0000","last_fullsized":"2024-02-21T23:04:10.604076+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:17:20.153344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.590276+0000","last_change":"2024-02-21T23:04:08.583183+0000","last_active":"2024-02-21T23:04:09.590276+0000","last_peered":"2024-02-21T23:04:09.590276+0000","last_clean":"2024-02-21T23:04:09.590276+0000","last_became_active":"2024-02-21T23:04:08.582207+0000","last_became_peered":"2024-02-21T23:04:08.582207+0000","last_unstale":"2024-02-21T23:04:09.590276+0000","last_undegraded":"2024-02-21T23:04:09.590276+0000","last_fullsized":"2024-02-21T23:04:09.590276+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_clean_scrub_stamp":"2024-02-21T23:04:07.527107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:31:36.287576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-21T23:04:10.602912+0000","last_change":"2024-02-21T23:04:09.610792+0000","last_active":"2024-02-21T23:04:10.602912+0000","last_peered":"2024-02-21T23:04:10.602912+0000","last_clean":"2024-02-21T23:04:10.602912+0000","last_became_active":"2024-02-21T23:04:09.609106+0000","last_became_peered":"2024-02-21T23:04:09.609106+0000","last_unstale":"2024-02-21T23:04:10.602912+0000","last_undegraded":"2024-02-21T23:04:10.602912+0000","last_fullsized":"2024-02-21T23:04:10.602912+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:12:57.961148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-21T23:04:10.605075+0000","last_change":"2024-02-21T23:04:08.572324+0000","last_active":"2024-02-21T23:04:10.605075+0000","last_peered":"2024-02-21T23:04:10.605075+0000","last_clean":"2024-02-21T23:04:10.605075+0000","last_became_active":"2024-02-21T23:04:08.572018+0000","last_became_peered":"2024-02-21T23:04:08.572018+0000","last_unstale":"2024-02-21T23:04:10.605075+0000","last_undegraded":"2024-02-21T23:04:10.605075+0000","last_fullsized":"2024-02-21T23:04:10.605075+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_clean_scrub_stamp":"2024-02-21T23:04:07.527107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:59:37.208322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.641079+0000","last_change":"2024-02-21T23:04:09.635684+0000","last_active":"2024-02-21T23:04:09.641079+0000","last_peered":"2024-02-21T23:04:09.641079+0000","last_clean":"2024-02-21T23:04:09.641079+0000","last_became_active":"2024-02-21T23:04:09.635166+0000","last_became_peered":"2024-02-21T23:04:09.635166+0000","last_unstale":"2024-02-21T23:04:09.641079+0000","last_undegraded":"2024-02-21T23:04:09.641079+0000","last_fullsized":"2024-02-21T23:04:09.641079+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:20:21.010019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.612434+0000","last_change":"2024-02-21T23:04:08.591105+0000","last_active":"2024-02-21T23:04:09.612434+0000","last_peered":"2024-02-21T23:04:09.612434+0000","last_clean":"2024-02-21T23:04:09.612434+0000","last_became_active":"2024-02-21T23:04:08.590825+0000","last_became_peered":"2024-02-21T23:04:08.590825+0000","last_unstale":"2024-02-21T23:04:09.612434+0000","last_undegraded":"2024-02-21T23:04:09.612434+0000","last_fullsized":"2024-02-21T23:04:09.612434+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_clean_scrub_stamp":"2024-02-21T23:04:07.527107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:48:42.021027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.628598+0000","last_change":"2024-02-21T23:04:09.623029+0000","last_active":"2024-02-21T23:04:09.628598+0000","last_peered":"2024-02-21T23:04:09.628598+0000","last_clean":"2024-02-21T23:04:09.628598+0000","last_became_active":"2024-02-21T23:04:09.617550+0000","last_became_peered":"2024-02-21T23:04:09.617550+0000","last_unstale":"2024-02-21T23:04:09.628598+0000","last_undegraded":"2024-02-21T23:04:09.628598+0000","last_fullsized":"2024-02-21T23:04:09.628598+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:41:02.807550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"37'3","reported_seq":15,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.639317+0000","last_change":"2024-02-21T23:04:08.592296+0000","last_active":"2024-02-21T23:04:09.639317+0000","last_peered":"2024-02-21T23:04:09.639317+0000","last_clean":"2024-02-21T23:04:09.639317+0000","last_became_active":"2024-02-21T23:04:08.589032+0000","last_became_peered":"2024-02-21T23:04:08.589032+0000","last_unstale":"2024-02-21T23:04:09.639317+0000","last_undegraded":"2024-02-21T23:04:09.639317+0000","last_fullsized":"2024-02-21T23:04:09.639317+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_clean_scrub_stamp":"2024-02-21T23:04:07.527107+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:48:11.268564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.614476+0000","last_change":"2024-02-21T23:04:09.609319+0000","last_active":"2024-02-21T23:04:09.614476+0000","last_peered":"2024-02-21T23:04:09.614476+0000","last_clean":"2024-02-21T23:04:09.614476+0000","last_became_active":"2024-02-21T23:04:09.608169+0000","last_became_peered":"2024-02-21T23:04:09.608169+0000","last_unstale":"2024-02-21T23:04:09.614476+0000","last_undegraded":"2024-02-21T23:04:09.614476+0000","last_fullsized":"2024-02-21T23:04:09.614476+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:43:31.902765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"37'6","reported_seq":20,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-21T23:04:10.604795+0000","last_change":"2024-02-21T23:04:08.588332+0000","last_active":"2024-02-21T23:04:10.604795+0000","last_peered":"2024-02-21T23:04:10.604795+0000","last_clean":"2024-02-21T23:04:10.604795+0000","last_became_active":"2024-02-21T23:04:08.587826+0000","last_became_peered":"2024-02-21T23:04:08.587826+0000","last_unstale":"2024-02-21T23:04:10.604795+0000","last_undegraded":"2024-02-21T23:04:10.604795+0000","last_fullsized":"2024-02-21T23:04:10.604795+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_clean_scrub_stamp":"2024-02-21T23:04:07.527107+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:21:26.461810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.628530+0000","last_change":"2024-02-21T23:04:09.620053+0000","last_active":"2024-02-21T23:04:09.628530+0000","last_peered":"2024-02-21T23:04:09.628530+0000","last_clean":"2024-02-21T23:04:09.628530+0000","last_became_active":"2024-02-21T23:04:09.616123+0000","last_became_peered":"2024-02-21T23:04:09.616123+0000","last_unstale":"2024-02-21T23:04:09.628530+0000","last_undegraded":"2024-02-21T23:04:09.628530+0000","last_fullsized":"2024-02-21T23:04:09.628530+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:12:08.720118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"37'2","reported_seq":16,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-21T23:04:10.604463+0000","last_change":"2024-02-21T23:04:08.578453+0000","last_active":"2024-02-21T23:04:10.604463+0000","last_peered":"2024-02-21T23:04:10.604463+0000","last_clean":"2024-02-21T23:04:10.604463+0000","last_became_active":"2024-02-21T23:04:08.575240+0000","last_became_peered":"2024-02-21T23:04:08.575240+0000","last_unstale":"2024-02-21T23:04:10.604463+0000","last_undegraded":"2024-02-21T23:04:10.604463+0000","last_fullsized":"2024-02-21T23:04:10.604463+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_clean_scrub_stamp":"2024-02-21T23:04:07.527107+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:25:42.570905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-21T23:04:10.603962+0000","last_change":"2024-02-21T23:04:09.603858+0000","last_active":"2024-02-21T23:04:10.603962+0000","last_peered":"2024-02-21T23:04:10.603962+0000","last_clean":"2024-02-21T23:04:10.603962+0000","last_became_active":"2024-02-21T23:04:09.603561+0000","last_became_peered":"2024-02-21T23:04:09.603561+0000","last_unstale":"2024-02-21T23:04:10.603962+0000","last_undegraded":"2024-02-21T23:04:10.603962+0000","last_fullsized":"2024-02-21T23:04:10.603962+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:28:25.253674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.596420+0000","last_change":"2024-02-21T23:04:08.592119+0000","last_active":"2024-02-21T23:04:09.596420+0000","last_peered":"2024-02-21T23:04:09.596420+0000","last_clean":"2024-02-21T23:04:09.596420+0000","last_became_active":"2024-02-21T23:04:08.591851+0000","last_became_peered":"2024-02-21T23:04:08.591851+0000","last_unstale":"2024-02-21T23:04:09.596420+0000","last_undegraded":"2024-02-21T23:04:09.596420+0000","last_fullsized":"2024-02-21T23:04:09.596420+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_clean_scrub_stamp":"2024-02-21T23:04:07.527107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:43:30.887358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.628301+0000","last_change":"2024-02-21T23:04:09.625606+0000","last_active":"2024-02-21T23:04:09.628301+0000","last_peered":"2024-02-21T23:04:09.628301+0000","last_clean":"2024-02-21T23:04:09.628301+0000","last_became_active":"2024-02-21T23:04:09.625280+0000","last_became_peered":"2024-02-21T23:04:09.625280+0000","last_unstale":"2024-02-21T23:04:09.628301+0000","last_undegraded":"2024-02-21T23:04:09.628301+0000","last_fullsized":"2024-02-21T23:04:09.628301+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:42:21.080709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-21T23:04:10.605176+0000","last_change":"2024-02-21T23:04:08.593155+0000","last_active":"2024-02-21T23:04:10.605176+0000","last_peered":"2024-02-21T23:04:10.605176+0000","last_clean":"2024-02-21T23:04:10.605176+0000","last_became_active":"2024-02-21T23:04:08.592850+0000","last_became_peered":"2024-02-21T23:04:08.592850+0000","last_unstale":"2024-02-21T23:04:10.605176+0000","last_undegraded":"2024-02-21T23:04:10.605176+0000","last_fullsized":"2024-02-21T23:04:10.605176+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_clean_scrub_stamp":"2024-02-21T23:04:07.527107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:47:10.506827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-21T23:04:10.604342+0000","last_change":"2024-02-21T23:04:09.604835+0000","last_active":"2024-02-21T23:04:10.604342+0000","last_peered":"2024-02-21T23:04:10.604342+0000","last_clean":"2024-02-21T23:04:10.604342+0000","last_became_active":"2024-02-21T23:04:09.604489+0000","last_became_peered":"2024-02-21T23:04:09.604489+0000","last_unstale":"2024-02-21T23:04:10.604342+0000","last_undegraded":"2024-02-21T23:04:10.604342+0000","last_fullsized":"2024-02-21T23:04:10.604342+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:03:19.008942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"37'8","reported_seq":20,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.641113+0000","last_change":"2024-02-21T23:04:08.599153+0000","last_active":"2024-02-21T23:04:09.641113+0000","last_peered":"2024-02-21T23:04:09.641113+0000","last_clean":"2024-02-21T23:04:09.641113+0000","last_became_active":"2024-02-21T23:04:08.598820+0000","last_became_peered":"2024-02-21T23:04:08.598820+0000","last_unstale":"2024-02-21T23:04:09.641113+0000","last_undegraded":"2024-02-21T23:04:09.641113+0000","last_fullsized":"2024-02-21T23:04:09.641113+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_clean_scrub_stamp":"2024-02-21T23:04:07.527107+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:39:01.282374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-21T23:04:10.604510+0000","last_change":"2024-02-21T23:04:09.633198+0000","last_active":"2024-02-21T23:04:10.604510+0000","last_peered":"2024-02-21T23:04:10.604510+0000","last_clean":"2024-02-21T23:04:10.604510+0000","last_became_active":"2024-02-21T23:04:09.632938+0000","last_became_peered":"2024-02-21T23:04:09.632938+0000","last_unstale":"2024-02-21T23:04:10.604510+0000","last_undegraded":"2024-02-21T23:04:10.604510+0000","last_fullsized":"2024-02-21T23:04:10.604510+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:16:31.510876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"37'1","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.640845+0000","last_change":"2024-02-21T23:04:08.599197+0000","last_active":"2024-02-21T23:04:09.640845+0000","last_peered":"2024-02-21T23:04:09.640845+0000","last_clean":"2024-02-21T23:04:09.640845+0000","last_became_active":"2024-02-21T23:04:08.598898+0000","last_became_peered":"2024-02-21T23:04:08.598898+0000","last_unstale":"2024-02-21T23:04:09.640845+0000","last_undegraded":"2024-02-21T23:04:09.640845+0000","last_fullsized":"2024-02-21T23:04:09.640845+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_clean_scrub_stamp":"2024-02-21T23:04:07.527107+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:02:29.779590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.632341+0000","last_change":"2024-02-21T23:04:09.624622+0000","last_active":"2024-02-21T23:04:09.632341+0000","last_peered":"2024-02-21T23:04:09.632341+0000","last_clean":"2024-02-21T23:04:09.632341+0000","last_became_active":"2024-02-21T23:04:09.620423+0000","last_became_peered":"2024-02-21T23:04:09.620423+0000","last_unstale":"2024-02-21T23:04:09.632341+0000","last_undegraded":"2024-02-21T23:04:09.632341+0000","last_fullsized":"2024-02-21T23:04:09.632341+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:27:10.937642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"37'2","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.643378+0000","last_change":"2024-02-21T23:04:08.595297+0000","last_active":"2024-02-21T23:04:09.643378+0000","last_peered":"2024-02-21T23:04:09.643378+0000","last_clean":"2024-02-21T23:04:09.643378+0000","last_became_active":"2024-02-21T23:04:08.594876+0000","last_became_peered":"2024-02-21T23:04:08.594876+0000","last_unstale":"2024-02-21T23:04:09.643378+0000","last_undegraded":"2024-02-21T23:04:09.643378+0000","last_fullsized":"2024-02-21T23:04:09.643378+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_clean_scrub_stamp":"2024-02-21T23:04:07.527107+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:28:31.233414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.611162+0000","last_change":"2024-02-21T23:04:09.602680+0000","last_active":"2024-02-21T23:04:09.611162+0000","last_peered":"2024-02-21T23:04:09.611162+0000","last_clean":"2024-02-21T23:04:09.611162+0000","last_became_active":"2024-02-21T23:04:09.601812+0000","last_became_peered":"2024-02-21T23:04:09.601812+0000","last_unstale":"2024-02-21T23:04:09.611162+0000","last_undegraded":"2024-02-21T23:04:09.611162+0000","last_fullsized":"2024-02-21T23:04:09.611162+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:12:57.961148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"37'1","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-21T23:04:10.602842+0000","last_change":"2024-02-21T23:04:08.588371+0000","last_active":"2024-02-21T23:04:10.602842+0000","last_peered":"2024-02-21T23:04:10.602842+0000","last_clean":"2024-02-21T23:04:10.602842+0000","last_became_active":"2024-02-21T23:04:08.588026+0000","last_became_peered":"2024-02-21T23:04:08.588026+0000","last_unstale":"2024-02-21T23:04:10.602842+0000","last_undegraded":"2024-02-21T23:04:10.602842+0000","last_fullsized":"2024-02-21T23:04:10.602842+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_clean_scrub_stamp":"2024-02-21T23:04:07.527107+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:45:52.618830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-21T23:04:10.603124+0000","last_change":"2024-02-21T23:04:09.606412+0000","last_active":"2024-02-21T23:04:10.603124+0000","last_peered":"2024-02-21T23:04:10.603124+0000","last_clean":"2024-02-21T23:04:10.603124+0000","last_became_active":"2024-02-21T23:04:09.606111+0000","last_became_peered":"2024-02-21T23:04:09.606111+0000","last_unstale":"2024-02-21T23:04:10.603124+0000","last_undegraded":"2024-02-21T23:04:10.603124+0000","last_fullsized":"2024-02-21T23:04:10.603124+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:38:20.998379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.595916+0000","last_change":"2024-02-21T23:04:08.593221+0000","last_active":"2024-02-21T23:04:09.595916+0000","last_peered":"2024-02-21T23:04:09.595916+0000","last_clean":"2024-02-21T23:04:09.595916+0000","last_became_active":"2024-02-21T23:04:08.592964+0000","last_became_peered":"2024-02-21T23:04:08.592964+0000","last_unstale":"2024-02-21T23:04:09.595916+0000","last_undegraded":"2024-02-21T23:04:09.595916+0000","last_fullsized":"2024-02-21T23:04:09.595916+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_clean_scrub_stamp":"2024-02-21T23:04:07.527107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:38:19.982972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.631685+0000","last_change":"2024-02-21T23:04:09.624820+0000","last_active":"2024-02-21T23:04:09.631685+0000","last_peered":"2024-02-21T23:04:09.631685+0000","last_clean":"2024-02-21T23:04:09.631685+0000","last_became_active":"2024-02-21T23:04:09.624549+0000","last_became_peered":"2024-02-21T23:04:09.624549+0000","last_unstale":"2024-02-21T23:04:09.631685+0000","last_undegraded":"2024-02-21T23:04:09.631685+0000","last_fullsized":"2024-02-21T23:04:09.631685+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:12:08.720118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"37'2","reported_seq":16,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-21T23:04:10.602452+0000","last_change":"2024-02-21T23:04:08.573752+0000","last_active":"2024-02-21T23:04:10.602452+0000","last_peered":"2024-02-21T23:04:10.602452+0000","last_clean":"2024-02-21T23:04:10.602452+0000","last_became_active":"2024-02-21T23:04:08.573438+0000","last_became_peered":"2024-02-21T23:04:08.573438+0000","last_unstale":"2024-02-21T23:04:10.602452+0000","last_undegraded":"2024-02-21T23:04:10.602452+0000","last_fullsized":"2024-02-21T23:04:10.602452+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_clean_scrub_stamp":"2024-02-21T23:04:07.527107+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:19:29.864117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.627897+0000","last_change":"2024-02-21T23:04:09.618068+0000","last_active":"2024-02-21T23:04:09.627897+0000","last_peered":"2024-02-21T23:04:09.627897+0000","last_clean":"2024-02-21T23:04:09.627897+0000","last_became_active":"2024-02-21T23:04:09.617807+0000","last_became_peered":"2024-02-21T23:04:09.617807+0000","last_unstale":"2024-02-21T23:04:09.627897+0000","last_undegraded":"2024-02-21T23:04:09.627897+0000","last_fullsized":"2024-02-21T23:04:09.627897+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:03:19.900047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-21T23:04:10.602283+0000","last_change":"2024-02-21T23:04:08.591987+0000","last_active":"2024-02-21T23:04:10.602283+0000","last_peered":"2024-02-21T23:04:10.602283+0000","last_clean":"2024-02-21T23:04:10.602283+0000","last_became_active":"2024-02-21T23:04:08.591713+0000","last_became_peered":"2024-02-21T23:04:08.591713+0000","last_unstale":"2024-02-21T23:04:10.602283+0000","last_undegraded":"2024-02-21T23:04:10.602283+0000","last_fullsized":"2024-02-21T23:04:10.602283+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_clean_scrub_stamp":"2024-02-21T23:04:07.527107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:09:30.103400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.642970+0000","last_change":"2024-02-21T23:04:09.637391+0000","last_active":"2024-02-21T23:04:09.642970+0000","last_peered":"2024-02-21T23:04:09.642970+0000","last_clean":"2024-02-21T23:04:09.642970+0000","last_became_active":"2024-02-21T23:04:09.637047+0000","last_became_peered":"2024-02-21T23:04:09.637047+0000","last_unstale":"2024-02-21T23:04:09.642970+0000","last_undegraded":"2024-02-21T23:04:09.642970+0000","last_fullsized":"2024-02-21T23:04:09.642970+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:29:05.689326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.608808+0000","last_change":"2024-02-21T23:04:08.591002+0000","last_active":"2024-02-21T23:04:09.608808+0000","last_peered":"2024-02-21T23:04:09.608808+0000","last_clean":"2024-02-21T23:04:09.608808+0000","last_became_active":"2024-02-21T23:04:08.589857+0000","last_became_peered":"2024-02-21T23:04:08.589857+0000","last_unstale":"2024-02-21T23:04:09.608808+0000","last_undegraded":"2024-02-21T23:04:09.608808+0000","last_fullsized":"2024-02-21T23:04:09.608808+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_clean_scrub_stamp":"2024-02-21T23:04:07.527107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:38:19.982972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-21T23:04:10.604359+0000","last_change":"2024-02-21T23:04:09.610189+0000","last_active":"2024-02-21T23:04:10.604359+0000","last_peered":"2024-02-21T23:04:10.604359+0000","last_clean":"2024-02-21T23:04:10.604359+0000","last_became_active":"2024-02-21T23:04:09.609881+0000","last_became_peered":"2024-02-21T23:04:09.609881+0000","last_unstale":"2024-02-21T23:04:10.604359+0000","last_undegraded":"2024-02-21T23:04:10.604359+0000","last_fullsized":"2024-02-21T23:04:10.604359+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:25:15.239988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-21T23:04:10.602723+0000","last_change":"2024-02-21T23:04:08.568505+0000","last_active":"2024-02-21T23:04:10.602723+0000","last_peered":"2024-02-21T23:04:10.602723+0000","last_clean":"2024-02-21T23:04:10.602723+0000","last_became_active":"2024-02-21T23:04:08.568196+0000","last_became_peered":"2024-02-21T23:04:08.568196+0000","last_unstale":"2024-02-21T23:04:10.602723+0000","last_undegraded":"2024-02-21T23:04:10.602723+0000","last_fullsized":"2024-02-21T23:04:10.602723+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_clean_scrub_stamp":"2024-02-21T23:04:07.527107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:54:50.559993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.629114+0000","last_change":"2024-02-21T23:04:09.627968+0000","last_active":"2024-02-21T23:04:09.629114+0000","last_peered":"2024-02-21T23:04:09.629114+0000","last_clean":"2024-02-21T23:04:09.629114+0000","last_became_active":"2024-02-21T23:04:09.625013+0000","last_became_peered":"2024-02-21T23:04:09.625013+0000","last_unstale":"2024-02-21T23:04:09.629114+0000","last_undegraded":"2024-02-21T23:04:09.629114+0000","last_fullsized":"2024-02-21T23:04:09.629114+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:52:26.223286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"37'2","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.637143+0000","last_change":"2024-02-21T23:04:08.593913+0000","last_active":"2024-02-21T23:04:09.637143+0000","last_peered":"2024-02-21T23:04:09.637143+0000","last_clean":"2024-02-21T23:04:09.637143+0000","last_became_active":"2024-02-21T23:04:08.593492+0000","last_became_peered":"2024-02-21T23:04:08.593492+0000","last_unstale":"2024-02-21T23:04:09.637143+0000","last_undegraded":"2024-02-21T23:04:09.637143+0000","last_fullsized":"2024-02-21T23:04:09.637143+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_clean_scrub_stamp":"2024-02-21T23:04:07.527107+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:30:43.428924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.641045+0000","last_change":"2024-02-21T23:04:09.635457+0000","last_active":"2024-02-21T23:04:09.641045+0000","last_peered":"2024-02-21T23:04:09.641045+0000","last_clean":"2024-02-21T23:04:09.641045+0000","last_became_active":"2024-02-21T23:04:09.634764+0000","last_became_peered":"2024-02-21T23:04:09.634764+0000","last_unstale":"2024-02-21T23:04:09.641045+0000","last_undegraded":"2024-02-21T23:04:09.641045+0000","last_fullsized":"2024-02-21T23:04:09.641045+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:30:44.444331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-21T23:04:10.602674+0000","last_change":"2024-02-21T23:04:08.592520+0000","last_active":"2024-02-21T23:04:10.602674+0000","last_peered":"2024-02-21T23:04:10.602674+0000","last_clean":"2024-02-21T23:04:10.602674+0000","last_became_active":"2024-02-21T23:04:08.592184+0000","last_became_peered":"2024-02-21T23:04:08.592184+0000","last_unstale":"2024-02-21T23:04:10.602674+0000","last_undegraded":"2024-02-21T23:04:10.602674+0000","last_fullsized":"2024-02-21T23:04:10.602674+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_clean_scrub_stamp":"2024-02-21T23:04:07.527107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:11:38.769235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-21T23:04:10.604765+0000","last_change":"2024-02-21T23:04:09.611514+0000","last_active":"2024-02-21T23:04:10.604765+0000","last_peered":"2024-02-21T23:04:10.604765+0000","last_clean":"2024-02-21T23:04:10.604765+0000","last_became_active":"2024-02-21T23:04:09.611294+0000","last_became_peered":"2024-02-21T23:04:09.611294+0000","last_unstale":"2024-02-21T23:04:10.604765+0000","last_undegraded":"2024-02-21T23:04:10.604765+0000","last_fullsized":"2024-02-21T23:04:10.604765+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:02:09.505024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.592262+0000","last_change":"2024-02-21T23:04:08.583914+0000","last_active":"2024-02-21T23:04:09.592262+0000","last_peered":"2024-02-21T23:04:09.592262+0000","last_clean":"2024-02-21T23:04:09.592262+0000","last_became_active":"2024-02-21T23:04:08.583535+0000","last_became_peered":"2024-02-21T23:04:08.583535+0000","last_unstale":"2024-02-21T23:04:09.592262+0000","last_undegraded":"2024-02-21T23:04:09.592262+0000","last_fullsized":"2024-02-21T23:04:09.592262+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_clean_scrub_stamp":"2024-02-21T23:04:07.527107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:28:31.233414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.614044+0000","last_change":"2024-02-21T23:04:09.605152+0000","last_active":"2024-02-21T23:04:09.614044+0000","last_peered":"2024-02-21T23:04:09.614044+0000","last_clean":"2024-02-21T23:04:09.614044+0000","last_became_active":"2024-02-21T23:04:09.604840+0000","last_became_peered":"2024-02-21T23:04:09.604840+0000","last_unstale":"2024-02-21T23:04:09.614044+0000","last_undegraded":"2024-02-21T23:04:09.614044+0000","last_fullsized":"2024-02-21T23:04:09.614044+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:43:31.902765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.598821+0000","last_change":"2024-02-21T23:04:08.584097+0000","last_active":"2024-02-21T23:04:09.598821+0000","last_peered":"2024-02-21T23:04:09.598821+0000","last_clean":"2024-02-21T23:04:09.598821+0000","last_became_active":"2024-02-21T23:04:08.583773+0000","last_became_peered":"2024-02-21T23:04:08.583773+0000","last_unstale":"2024-02-21T23:04:09.598821+0000","last_undegraded":"2024-02-21T23:04:09.598821+0000","last_fullsized":"2024-02-21T23:04:09.598821+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_clean_scrub_stamp":"2024-02-21T23:04:07.527107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:38:19.982972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.629967+0000","last_change":"2024-02-21T23:04:09.628185+0000","last_active":"2024-02-21T23:04:09.629967+0000","last_peered":"2024-02-21T23:04:09.629967+0000","last_clean":"2024-02-21T23:04:09.629967+0000","last_became_active":"2024-02-21T23:04:09.627879+0000","last_became_peered":"2024-02-21T23:04:09.627879+0000","last_unstale":"2024-02-21T23:04:09.629967+0000","last_undegraded":"2024-02-21T23:04:09.629967+0000","last_fullsized":"2024-02-21T23:04:09.629967+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:48:43.036434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-21T23:04:10.605660+0000","last_change":"2024-02-21T23:04:08.581319+0000","last_active":"2024-02-21T23:04:10.605660+0000","last_peered":"2024-02-21T23:04:10.605660+0000","last_clean":"2024-02-21T23:04:10.605660+0000","last_became_active":"2024-02-21T23:04:08.580400+0000","last_became_peered":"2024-02-21T23:04:08.580400+0000","last_unstale":"2024-02-21T23:04:10.605660+0000","last_undegraded":"2024-02-21T23:04:10.605660+0000","last_fullsized":"2024-02-21T23:04:10.605660+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_clean_scrub_stamp":"2024-02-21T23:04:07.527107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:39:01.282374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.614369+0000","last_change":"2024-02-21T23:04:09.609299+0000","last_active":"2024-02-21T23:04:09.614369+0000","last_peered":"2024-02-21T23:04:09.614369+0000","last_clean":"2024-02-21T23:04:09.614369+0000","last_became_active":"2024-02-21T23:04:09.608982+0000","last_became_peered":"2024-02-21T23:04:09.608982+0000","last_unstale":"2024-02-21T23:04:09.614369+0000","last_undegraded":"2024-02-21T23:04:09.614369+0000","last_fullsized":"2024-02-21T23:04:09.614369+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:39:02.297782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.590753+0000","last_change":"2024-02-21T23:04:08.585533+0000","last_active":"2024-02-21T23:04:09.590753+0000","last_peered":"2024-02-21T23:04:09.590753+0000","last_clean":"2024-02-21T23:04:09.590753+0000","last_became_active":"2024-02-21T23:04:08.585240+0000","last_became_peered":"2024-02-21T23:04:08.585240+0000","last_unstale":"2024-02-21T23:04:09.590753+0000","last_undegraded":"2024-02-21T23:04:09.590753+0000","last_fullsized":"2024-02-21T23:04:09.590753+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_clean_scrub_stamp":"2024-02-21T23:04:07.527107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:47:10.506827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.638668+0000","last_change":"2024-02-21T23:04:09.633061+0000","last_active":"2024-02-21T23:04:09.638668+0000","last_peered":"2024-02-21T23:04:09.638668+0000","last_clean":"2024-02-21T23:04:09.638668+0000","last_became_active":"2024-02-21T23:04:09.632750+0000","last_became_peered":"2024-02-21T23:04:09.632750+0000","last_unstale":"2024-02-21T23:04:09.638668+0000","last_undegraded":"2024-02-21T23:04:09.638668+0000","last_fullsized":"2024-02-21T23:04:09.638668+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:02:30.794997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-02-21T23:04:10.603700+0000","last_change":"2024-02-21T23:04:08.575057+0000","last_active":"2024-02-21T23:04:10.603700+0000","last_peered":"2024-02-21T23:04:10.603700+0000","last_clean":"2024-02-21T23:04:10.603700+0000","last_became_active":"2024-02-21T23:04:08.574768+0000","last_became_peered":"2024-02-21T23:04:08.574768+0000","last_unstale":"2024-02-21T23:04:10.603700+0000","last_undegraded":"2024-02-21T23:04:10.603700+0000","last_fullsized":"2024-02-21T23:04:10.603700+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_clean_scrub_stamp":"2024-02-21T23:04:07.527107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:17:52.116580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.624574+0000","last_change":"2024-02-21T23:04:09.620488+0000","last_active":"2024-02-21T23:04:09.624574+0000","last_peered":"2024-02-21T23:04:09.624574+0000","last_clean":"2024-02-21T23:04:09.624574+0000","last_became_active":"2024-02-21T23:04:09.620101+0000","last_became_peered":"2024-02-21T23:04:09.620101+0000","last_unstale":"2024-02-21T23:04:09.624574+0000","last_undegraded":"2024-02-21T23:04:09.624574+0000","last_fullsized":"2024-02-21T23:04:09.624574+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:08.542514+0000","last_clean_scrub_stamp":"2024-02-21T23:04:08.542514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:25:43.586312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.613104+0000","last_change":"2024-02-21T23:04:08.590185+0000","last_active":"2024-02-21T23:04:09.613104+0000","last_peered":"2024-02-21T23:04:09.613104+0000","last_clean":"2024-02-21T23:04:09.613104+0000","last_became_active":"2024-02-21T23:04:08.589877+0000","last_became_peered":"2024-02-21T23:04:08.589877+0000","last_unstale":"2024-02-21T23:04:09.613104+0000","last_undegraded":"2024-02-21T23:04:09.613104+0000","last_fullsized":"2024-02-21T23:04:09.613104+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:07.527107+0000","last_clean_scrub_stamp":"2024-02-21T23:04:07.527107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:03:18.884640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":201,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-02-21T23:04:09.611859+0000","last_change":"2024-02-21T23:02:32.289335+0000","last_active":"2024-02-21T23:04:09.611859+0000","last_peered":"2024-02-21T23:04:09.611859+0000","last_clean":"2024-02-21T23:04:09.611859+0000","last_became_active":"2024-02-21T23:02:32.286913+0000","last_became_peered":"2024-02-21T23:02:32.286913+0000","last_unstale":"2024-02-21T23:04:09.611859+0000","last_undegraded":"2024-02-21T23:04:09.611859+0000","last_fullsized":"2024-02-21T23:04:09.611859+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_clean_scrub_stamp":"2024-02-21T23:02:31.172569+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:36:23.987374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574870,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27516,"kb_used_data":496,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344324,"statfs":{"total":96636764160,"available":96608587776,"internally_reserved":0,"allocated":507904,"data_stored":775394,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574871,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27444,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344396,"statfs":{"total":96636764160,"available":96608661504,"internally_reserved":0,"allocated":430080,"data_stored":183926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574870,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27452,"kb_used_data":424,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344388,"statfs":{"total":96636764160,"available":96608653312,"internally_reserved":0,"allocated":434176,"data_stored":183456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574871,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27452,"kb_used_data":424,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344388,"statfs":{"total":96636764160,"available":96608653312,"internally_reserved":0,"allocated":434176,"data_stored":183948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574871,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31620,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340220,"statfs":{"total":96636764160,"available":96604385280,"internally_reserved":0,"allocated":442368,"data_stored":184990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574872,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31612,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340228,"statfs":{"total":96636764160,"available":96604393472,"internally_reserved":0,"allocated":442368,"data_stored":190106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574870,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27448,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344392,"statfs":{"total":96636764160,"available":96608657408,"internally_reserved":0,"allocated":430080,"data_stored":183434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574870,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31664,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340176,"statfs":{"total":96636764160,"available":96604340224,"internally_reserved":0,"allocated":495616,"data_stored":773746,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-21T23:04:14.567 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-02-21T23:04:14.568 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-02-21T23:04:14.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.852+0000 7f8599e00640 1 Processor -- start 2024-02-21T23:04:14.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.852+0000 7f8599e00640 1 -- start start 2024-02-21T23:04:14.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.854+0000 7f8599e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f85940c6c60 0x7f85940c3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:14.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.854+0000 7f8599e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8594159580 con 0x7f85940c5cc0 2024-02-21T23:04:14.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.854+0000 7f8599e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85941587c0 con 0x7f85940c7590 2024-02-21T23:04:14.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.854+0000 7f8599e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85940c6230 con 0x7f85940c6c60 2024-02-21T23:04:14.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.854+0000 7f8592000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f85940c5cc0 0x7f85940c3030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:56682/0 (socket says 172.21.15.28:56682) 2024-02-21T23:04:14.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.854+0000 7f8592000640 1 -- 172.21.15.28:0/2295561740 learned_addr learned my addr 172.21.15.28:0/2295561740 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:04:14.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.854+0000 7f8593400640 1 --2- 172.21.15.28:0/2295561740 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f85940c6c60 0x7f85940c3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:14.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.855+0000 7f8591600640 1 -- 172.21.15.28:0/2295561740 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 449047618 0 0) 0x7f8594159580 con 0x7f85940c5cc0 2024-02-21T23:04:14.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.855+0000 7f8591600640 1 -- 172.21.15.28:0/2295561740 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8578003520 con 0x7f85940c5cc0 2024-02-21T23:04:14.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.855+0000 7f8591600640 1 -- 172.21.15.28:0/2295561740 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1137750903 0 0) 0x7f8578003520 con 0x7f85940c5cc0 2024-02-21T23:04:14.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.855+0000 7f8591600640 1 -- 172.21.15.28:0/2295561740 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f85940c6c60 msgr2=0x7f85940c3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:14.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.855+0000 7f8591600640 1 --2- 172.21.15.28:0/2295561740 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f85940c6c60 0x7f85940c3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:14.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.855+0000 7f8591600640 1 -- 172.21.15.28:0/2295561740 >> v1:172.21.15.81:6789/0 conn(0x7f85940c7590 legacy=0x7f85941524e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:14.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.855+0000 7f8591600640 1 -- 172.21.15.28:0/2295561740 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8594158aa0 con 0x7f85940c5cc0 2024-02-21T23:04:14.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.855+0000 7f8591600640 1 -- 172.21.15.28:0/2295561740 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f8580001670 con 0x7f85940c5cc0 2024-02-21T23:04:14.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.855+0000 7f8591600640 1 -- 172.21.15.28:0/2295561740 >> v1:172.21.15.28:6789/0 conn(0x7f85940c5cc0 legacy=0x7f85940c3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:14.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.855+0000 7f8591600640 1 --2- 172.21.15.28:0/2295561740 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f85940c6c60 0x7f85780039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:14.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.855+0000 7f8591600640 1 --2- 172.21.15.28:0/2295561740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8578003f60 0x7f8578004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:14.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.855+0000 7f8591600640 1 --2- 172.21.15.28:0/2295561740 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8578004d40 0x7f85780051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:14.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.855+0000 7f8591600640 1 -- 172.21.15.28:0/2295561740 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85940c6230 con 0x7f8578003f60 2024-02-21T23:04:14.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.855+0000 7f8591600640 1 -- 172.21.15.28:0/2295561740 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f857800bb50 con 0x7f8578004d40 2024-02-21T23:04:14.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.855+0000 7f8591600640 1 -- 172.21.15.28:0/2295561740 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f857800bd50 con 0x7f85940c6c60 2024-02-21T23:04:14.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.856+0000 7f8592000640 1 --2- 172.21.15.28:0/2295561740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8578003f60 0x7f8578004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:14.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.856+0000 7f8592a00640 1 --2- 172.21.15.28:0/2295561740 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f85940c6c60 0x7f85780039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:14.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.856+0000 7f8593400640 1 --2- 172.21.15.28:0/2295561740 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8578004d40 0x7f85780051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:14.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.856+0000 7f8592000640 1 -- 172.21.15.28:0/2295561740 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f85940c6c60 msgr2=0x7f85780039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:14.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.856+0000 7f8592000640 1 --2- 172.21.15.28:0/2295561740 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f85940c6c60 0x7f85780039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:14.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.856+0000 7f8592000640 1 -- 172.21.15.28:0/2295561740 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8578004d40 msgr2=0x7f85780051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:14.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.856+0000 7f8592000640 1 --2- 172.21.15.28:0/2295561740 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8578004d40 0x7f85780051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:14.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.856+0000 7f8592000640 1 -- 172.21.15.28:0/2295561740 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8578003520 con 0x7f8578003f60 2024-02-21T23:04:14.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.856+0000 7f8592a00640 1 --2- 172.21.15.28:0/2295561740 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f85940c6c60 0x7f85780039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:04:14.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.857+0000 7f8592000640 1 --2- 172.21.15.28:0/2295561740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8578003f60 0x7f8578004800 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f858000c240 tx=0x7f8578005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d348fe306d2fb4bc server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:14.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.858+0000 7f8591600640 1 -- 172.21.15.28:0/2295561740 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8580007d10 con 0x7f8578003f60 2024-02-21T23:04:14.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.858+0000 7f8591600640 1 -- 172.21.15.28:0/2295561740 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8580030070 con 0x7f8578003f60 2024-02-21T23:04:14.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.858+0000 7f8591600640 1 -- 172.21.15.28:0/2295561740 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f858002cd60 con 0x7f8578003f60 2024-02-21T23:04:14.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.858+0000 7f8599e00640 1 -- 172.21.15.28:0/2295561740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8578003f60 msgr2=0x7f8578004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:14.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.858+0000 7f8599e00640 1 --2- 172.21.15.28:0/2295561740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8578003f60 0x7f8578004800 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f858000c240 tx=0x7f8578005830 comp rx=0 tx=0).stop 2024-02-21T23:04:14.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.858+0000 7f8593400640 1 -- 172.21.15.28:0/2295561740 reap_dead start 2024-02-21T23:04:14.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.858+0000 7f8599e00640 1 -- 172.21.15.28:0/2295561740 shutdown_connections 2024-02-21T23:04:14.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.858+0000 7f8599e00640 1 -- 172.21.15.28:0/2295561740 >> 172.21.15.28:0/2295561740 conn(0x7f859408afe0 msgr2=0x7f85940725a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:14.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.858+0000 7f8599e00640 1 -- 172.21.15.28:0/2295561740 shutdown_connections 2024-02-21T23:04:14.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.858+0000 7f8599e00640 1 -- 172.21.15.28:0/2295561740 wait complete. 2024-02-21T23:04:14.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.861+0000 7f8599e00640 1 Processor -- start 2024-02-21T23:04:14.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.861+0000 7f8599e00640 1 -- start start 2024-02-21T23:04:14.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.863+0000 7f8599e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8578003f60 0x7f85941448b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:14.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.863+0000 7f8599e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85941441b0 0x7f85941eae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:14.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.863+0000 7f8599e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f85941eb380 0x7f85941465a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:14.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.863+0000 7f8599e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85940c6720 con 0x7f85941441b0 2024-02-21T23:04:14.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.863+0000 7f8599e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8594159060 con 0x7f85941eb380 2024-02-21T23:04:14.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.863+0000 7f8599e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85940c68a0 con 0x7f8578003f60 2024-02-21T23:04:14.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.863+0000 7f8592000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85941441b0 0x7f85941eae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:14.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.863+0000 7f8592000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85941441b0 0x7f85941eae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56442/0 (socket says 172.21.15.28:56442) 2024-02-21T23:04:14.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.863+0000 7f8592000640 1 -- 172.21.15.28:0/4089829197 learned_addr learned my addr 172.21.15.28:0/4089829197 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:14.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.863+0000 7f8592a00640 1 --2- 172.21.15.28:0/4089829197 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8578003f60 0x7f85941448b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:14.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.864+0000 7f8593400640 1 --2- 172.21.15.28:0/4089829197 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f85941eb380 0x7f85941465a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:14.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.864+0000 7f8592000640 1 -- 172.21.15.28:0/4089829197 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8578003f60 msgr2=0x7f85941448b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:14.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.864+0000 7f8592000640 1 --2- 172.21.15.28:0/4089829197 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8578003f60 0x7f85941448b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:14.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.864+0000 7f8592000640 1 -- 172.21.15.28:0/4089829197 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f85941eb380 msgr2=0x7f85941465a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:14.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.864+0000 7f8592000640 1 --2- 172.21.15.28:0/4089829197 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f85941eb380 0x7f85941465a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:14.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.864+0000 7f8592000640 1 -- 172.21.15.28:0/4089829197 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85940c9a20 con 0x7f85941441b0 2024-02-21T23:04:14.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.864+0000 7f8592000640 1 --2- 172.21.15.28:0/4089829197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85941441b0 0x7f85941eae40 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f858002c570 tx=0x7f85941560f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:14.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.865+0000 7f858b400640 1 -- 172.21.15.28:0/4089829197 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8580045070 con 0x7f85941441b0 2024-02-21T23:04:14.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.865+0000 7f858b400640 1 -- 172.21.15.28:0/4089829197 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8580004bf0 con 0x7f85941441b0 2024-02-21T23:04:14.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.865+0000 7f858b400640 1 -- 172.21.15.28:0/4089829197 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8580007d10 con 0x7f85941441b0 2024-02-21T23:04:14.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.865+0000 7f8599e00640 1 -- 172.21.15.28:0/4089829197 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85941ebd40 con 0x7f85941441b0 2024-02-21T23:04:14.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.865+0000 7f8599e00640 1 -- 172.21.15.28:0/4089829197 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85940c9cc0 con 0x7f85941441b0 2024-02-21T23:04:14.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.866+0000 7f8599e00640 1 -- 172.21.15.28:0/4089829197 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8594067f30 con 0x7f85941441b0 2024-02-21T23:04:14.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.867+0000 7f858b400640 1 -- 172.21.15.28:0/4089829197 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8580007070 con 0x7f85941441b0 2024-02-21T23:04:14.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.868+0000 7f858b400640 1 --2- 172.21.15.28:0/4089829197 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f857406b500 0x7f857406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:14.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.868+0000 7f858b400640 1 -- 172.21.15.28:0/4089829197 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f85800ab470 con 0x7f85941441b0 2024-02-21T23:04:14.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.869+0000 7f8592a00640 1 --2- 172.21.15.28:0/4089829197 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f857406b500 0x7f857406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:14.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.870+0000 7f8592a00640 1 --2- 172.21.15.28:0/4089829197 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f857406b500 0x7f857406d9c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f85840054d0 tx=0x7f8584009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:14.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:14.872+0000 7f858b400640 1 -- 172.21.15.28:0/4089829197 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f858007a1d0 con 0x7f85941441b0 2024-02-21T23:04:15.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.281+0000 7f8599e00640 1 -- 172.21.15.28:0/4089829197 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f859414e980 con 0x7f85941441b0 2024-02-21T23:04:15.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.285+0000 7f858b400640 1 -- 172.21.15.28:0/4089829197 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v11) v1 ==== 190+0+147 (secure 0 0 0) 0x7f8580079ad0 con 0x7f85941441b0 2024-02-21T23:04:15.274 INFO:teuthology.orchestra.run.smithi028.stdout:[client.0] 2024-02-21T23:04:15.274 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQDTgNZlILvYLxAAoREWxKKT7YcXnhkgk8YvoQ== 2024-02-21T23:04:15.274 INFO:teuthology.orchestra.run.smithi028.stdout: caps mds = "allow" 2024-02-21T23:04:15.274 INFO:teuthology.orchestra.run.smithi028.stdout: caps mon = "allow r" 2024-02-21T23:04:15.274 INFO:teuthology.orchestra.run.smithi028.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-02-21T23:04:15.274 INFO:teuthology.orchestra.run.smithi028.stderr:updated caps for client.0 2024-02-21T23:04:15.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.288+0000 7f8599e00640 1 -- 172.21.15.28:0/4089829197 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f857406b500 msgr2=0x7f857406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:15.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.288+0000 7f8599e00640 1 --2- 172.21.15.28:0/4089829197 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f857406b500 0x7f857406d9c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f85840054d0 tx=0x7f8584009040 comp rx=0 tx=0).stop 2024-02-21T23:04:15.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.288+0000 7f8599e00640 1 -- 172.21.15.28:0/4089829197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85941441b0 msgr2=0x7f85941eae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:15.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.288+0000 7f8599e00640 1 --2- 172.21.15.28:0/4089829197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85941441b0 0x7f85941eae40 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f858002c570 tx=0x7f85941560f0 comp rx=0 tx=0).stop 2024-02-21T23:04:15.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.288+0000 7f8599e00640 1 -- 172.21.15.28:0/4089829197 shutdown_connections 2024-02-21T23:04:15.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.288+0000 7f8599e00640 1 --2- 172.21.15.28:0/4089829197 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f857406b500 0x7f857406d9c0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:15.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.288+0000 7f8599e00640 1 --2- 172.21.15.28:0/4089829197 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f85941eb380 0x7f85941465a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:15.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.288+0000 7f8599e00640 1 --2- 172.21.15.28:0/4089829197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85941441b0 0x7f85941eae40 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:15.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.288+0000 7f8599e00640 1 --2- 172.21.15.28:0/4089829197 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8578003f60 0x7f85941448b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:15.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.288+0000 7f8599e00640 1 -- 172.21.15.28:0/4089829197 >> 172.21.15.28:0/4089829197 conn(0x7f859408afe0 msgr2=0x7f859414ff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:15.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.289+0000 7f8599e00640 1 -- 172.21.15.28:0/4089829197 shutdown_connections 2024-02-21T23:04:15.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.289+0000 7f8599e00640 1 -- 172.21.15.28:0/4089829197 wait complete. 2024-02-21T23:04:15.296 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:04:15.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.520+0000 7f7560a00640 1 Processor -- start 2024-02-21T23:04:15.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.520+0000 7f7560a00640 1 -- start start 2024-02-21T23:04:15.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.521+0000 7f7560a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f755c0aff90 0x7f755c0c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:15.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.521+0000 7f7560a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f755c06dbf0 0x7f755c158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:15.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.521+0000 7f7560a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f755c0c74c0 con 0x7f755c0c5720 2024-02-21T23:04:15.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.521+0000 7f7560a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f755c0c7270 con 0x7f755c0aff90 2024-02-21T23:04:15.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.521+0000 7f7560a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f755c158630 con 0x7f755c06dbf0 2024-02-21T23:04:15.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.522+0000 7f755be00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f755c0c5720 0x7f755c06c270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60410/0 (socket says 172.21.15.28:60410) 2024-02-21T23:04:15.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.522+0000 7f755be00640 1 -- 172.21.15.28:0/3760094067 learned_addr learned my addr 172.21.15.28:0/3760094067 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:04:15.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.522+0000 7f755aa00640 1 --2- 172.21.15.28:0/3760094067 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f755c06dbf0 0x7f755c158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:15.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.522+0000 7f755b400640 1 --2- 172.21.15.28:0/3760094067 >> v2:172.21.15.81:3300/0 conn(0x7f755c0aff90 0x7f755c0c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:15.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.522+0000 7f755a000640 1 -- 172.21.15.28:0/3760094067 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1245859773 0 0) 0x7f755c0c74c0 con 0x7f755c0c5720 2024-02-21T23:04:15.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.522+0000 7f755a000640 1 -- 172.21.15.28:0/3760094067 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7540003520 con 0x7f755c0c5720 2024-02-21T23:04:15.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.523+0000 7f755a000640 1 -- 172.21.15.28:0/3760094067 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 778430562 0 0) 0x7f7540003520 con 0x7f755c0c5720 2024-02-21T23:04:15.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.523+0000 7f755a000640 1 -- 172.21.15.28:0/3760094067 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f755c06dbf0 msgr2=0x7f755c158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:15.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.523+0000 7f755a000640 1 --2- 172.21.15.28:0/3760094067 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f755c06dbf0 0x7f755c158ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:15.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.523+0000 7f755a000640 1 -- 172.21.15.28:0/3760094067 >> v2:172.21.15.81:3300/0 conn(0x7f755c0aff90 msgr2=0x7f755c0c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:15.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.523+0000 7f755a000640 1 --2- 172.21.15.28:0/3760094067 >> v2:172.21.15.81:3300/0 conn(0x7f755c0aff90 0x7f755c0c51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:15.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.523+0000 7f755a000640 1 -- 172.21.15.28:0/3760094067 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f755c0c77a0 con 0x7f755c0c5720 2024-02-21T23:04:15.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.523+0000 7f755a000640 1 -- 172.21.15.28:0/3760094067 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f75540019f0 con 0x7f755c0c5720 2024-02-21T23:04:15.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.523+0000 7f755a000640 1 -- 172.21.15.28:0/3760094067 >> v1:172.21.15.28:6789/0 conn(0x7f755c0c5720 legacy=0x7f755c06c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:15.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.523+0000 7f755a000640 1 --2- 172.21.15.28:0/3760094067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75400039b0 0x7f7540003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:15.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.523+0000 7f755a000640 1 --2- 172.21.15.28:0/3760094067 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f755c06dbf0 0x7f7540004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:15.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.523+0000 7f755a000640 1 --2- 172.21.15.28:0/3760094067 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7540004d40 0x7f75400051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:15.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.523+0000 7f755a000640 1 -- 172.21.15.28:0/3760094067 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f755c0c7270 con 0x7f75400039b0 2024-02-21T23:04:15.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.523+0000 7f755a000640 1 -- 172.21.15.28:0/3760094067 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f755c158630 con 0x7f7540004d40 2024-02-21T23:04:15.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.523+0000 7f755a000640 1 -- 172.21.15.28:0/3760094067 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f754000b9c0 con 0x7f755c06dbf0 2024-02-21T23:04:15.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.523+0000 7f755b400640 1 --2- 172.21.15.28:0/3760094067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75400039b0 0x7f7540003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:15.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.524+0000 7f755aa00640 1 --2- 172.21.15.28:0/3760094067 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f755c06dbf0 0x7f7540004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:15.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.524+0000 7f755be00640 1 --2- 172.21.15.28:0/3760094067 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7540004d40 0x7f75400051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:15.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.524+0000 7f755b400640 1 -- 172.21.15.28:0/3760094067 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f755c06dbf0 msgr2=0x7f7540004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:15.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.524+0000 7f755b400640 1 --2- 172.21.15.28:0/3760094067 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f755c06dbf0 0x7f7540004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:15.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.524+0000 7f755b400640 1 -- 172.21.15.28:0/3760094067 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7540004d40 msgr2=0x7f75400051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:15.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.524+0000 7f755b400640 1 --2- 172.21.15.28:0/3760094067 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7540004d40 0x7f75400051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:15.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.524+0000 7f755b400640 1 -- 172.21.15.28:0/3760094067 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7540003520 con 0x7f75400039b0 2024-02-21T23:04:15.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.526+0000 7f755b400640 1 --2- 172.21.15.28:0/3760094067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75400039b0 0x7f7540003dd0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f754c009ac0 tx=0x7f7540005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba72441d5de59577 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:15.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.526+0000 7f755a000640 1 -- 172.21.15.28:0/3760094067 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f754c039070 con 0x7f75400039b0 2024-02-21T23:04:15.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.526+0000 7f755a000640 1 -- 172.21.15.28:0/3760094067 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f754c02c710 con 0x7f75400039b0 2024-02-21T23:04:15.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.526+0000 7f755a000640 1 -- 172.21.15.28:0/3760094067 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f754c009690 con 0x7f75400039b0 2024-02-21T23:04:15.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.527+0000 7f7560a00640 1 -- 172.21.15.28:0/3760094067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75400039b0 msgr2=0x7f7540003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:15.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.527+0000 7f7560a00640 1 --2- 172.21.15.28:0/3760094067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75400039b0 0x7f7540003dd0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f754c009ac0 tx=0x7f7540005830 comp rx=0 tx=0).stop 2024-02-21T23:04:15.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.527+0000 7f755be00640 1 -- 172.21.15.28:0/3760094067 reap_dead start 2024-02-21T23:04:15.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.527+0000 7f7560a00640 1 -- 172.21.15.28:0/3760094067 shutdown_connections 2024-02-21T23:04:15.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.527+0000 7f7560a00640 1 -- 172.21.15.28:0/3760094067 >> 172.21.15.28:0/3760094067 conn(0x7f755c0ac370 msgr2=0x7f755c0ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:15.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.527+0000 7f7560a00640 1 -- 172.21.15.28:0/3760094067 shutdown_connections 2024-02-21T23:04:15.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.527+0000 7f7560a00640 1 -- 172.21.15.28:0/3760094067 wait complete. 2024-02-21T23:04:15.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.529+0000 7f7560a00640 1 Processor -- start 2024-02-21T23:04:15.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.529+0000 7f7560a00640 1 -- start start 2024-02-21T23:04:15.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.530+0000 7f7560a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75400039b0 0x7f755c1eada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:15.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.530+0000 7f7560a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f755c154600 0x7f755c0ae740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:15.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.530+0000 7f7560a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f755c1eb910 0x7f755c1ebd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:15.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.530+0000 7f7560a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f755c0c8760 con 0x7f75400039b0 2024-02-21T23:04:15.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.530+0000 7f7560a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f755c0be0c0 con 0x7f755c154600 2024-02-21T23:04:15.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.530+0000 7f7560a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f755c0c88e0 con 0x7f755c1eb910 2024-02-21T23:04:15.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.530+0000 7f755b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75400039b0 0x7f755c1eada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:15.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.530+0000 7f755b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75400039b0 0x7f755c1eada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52524/0 (socket says 172.21.15.28:52524) 2024-02-21T23:04:15.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.530+0000 7f755b400640 1 -- 172.21.15.28:0/4105782557 learned_addr learned my addr 172.21.15.28:0/4105782557 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:15.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.531+0000 7f755aa00640 1 --2- 172.21.15.28:0/4105782557 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f755c154600 0x7f755c0ae740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:15.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.531+0000 7f755be00640 1 --2- 172.21.15.28:0/4105782557 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f755c1eb910 0x7f755c1ebd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:15.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.531+0000 7f755b400640 1 -- 172.21.15.28:0/4105782557 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f755c1eb910 msgr2=0x7f755c1ebd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:15.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.531+0000 7f755b400640 1 --2- 172.21.15.28:0/4105782557 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f755c1eb910 0x7f755c1ebd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:15.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.531+0000 7f755b400640 1 -- 172.21.15.28:0/4105782557 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f755c154600 msgr2=0x7f755c0ae740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:15.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.531+0000 7f755b400640 1 --2- 172.21.15.28:0/4105782557 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f755c154600 0x7f755c0ae740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:15.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.531+0000 7f755b400640 1 -- 172.21.15.28:0/4105782557 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f755c153c50 con 0x7f75400039b0 2024-02-21T23:04:15.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.531+0000 7f755aa00640 1 --2- 172.21.15.28:0/4105782557 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f755c154600 0x7f755c0ae740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:04:15.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.531+0000 7f755b400640 1 --2- 172.21.15.28:0/4105782557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75400039b0 0x7f755c1eada0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f754c02c380 tx=0x7f754c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:15.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.531+0000 7f7553400640 1 -- 172.21.15.28:0/4105782557 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f754c039050 con 0x7f75400039b0 2024-02-21T23:04:15.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.531+0000 7f7553400640 1 -- 172.21.15.28:0/4105782557 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f754c045070 con 0x7f75400039b0 2024-02-21T23:04:15.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.532+0000 7f7560a00640 1 -- 172.21.15.28:0/4105782557 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f755c152bb0 con 0x7f75400039b0 2024-02-21T23:04:15.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.532+0000 7f7553400640 1 -- 172.21.15.28:0/4105782557 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f754c003a50 con 0x7f75400039b0 2024-02-21T23:04:15.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.532+0000 7f7560a00640 1 -- 172.21.15.28:0/4105782557 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f755c153170 con 0x7f75400039b0 2024-02-21T23:04:15.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.533+0000 7f7560a00640 1 -- 172.21.15.28:0/4105782557 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f755c04f290 con 0x7f75400039b0 2024-02-21T23:04:15.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.534+0000 7f7553400640 1 -- 172.21.15.28:0/4105782557 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f754c003bf0 con 0x7f75400039b0 2024-02-21T23:04:15.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.535+0000 7f7553400640 1 --2- 172.21.15.28:0/4105782557 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f753c06b500 0x7f753c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:15.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.535+0000 7f7553400640 1 -- 172.21.15.28:0/4105782557 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f754c0abb00 con 0x7f75400039b0 2024-02-21T23:04:15.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.535+0000 7f755aa00640 1 --2- 172.21.15.28:0/4105782557 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f753c06b500 0x7f753c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:15.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.537+0000 7f755aa00640 1 --2- 172.21.15.28:0/4105782557 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f753c06b500 0x7f753c06d9c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f755c0aef70 tx=0x7f7548008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:15.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.539+0000 7f7553400640 1 -- 172.21.15.28:0/4105782557 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f754c07a750 con 0x7f75400039b0 2024-02-21T23:04:15.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.803+0000 7f7560a00640 1 -- 172.21.15.28:0/4105782557 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f755c06d9d0 con 0x7f75400039b0 2024-02-21T23:04:15.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.804+0000 7f7553400640 1 -- 172.21.15.28:0/4105782557 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+4938 (secure 0 0 0) 0x7f754c07a050 con 0x7f75400039b0 2024-02-21T23:04:15.793 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:04:15.793 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":18,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4778,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/2827878900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2827878900},{"type":"v1","addr":"172.21.15.28:6834","nonce":2827878900}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":4978,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/3612644344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":3612644344},{"type":"v1","addr":"172.21.15.81:6836","nonce":3612644344}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4981,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3623937361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":3623937361},{"type":"v1","addr":"172.21.15.81:6837","nonce":3623937361}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:04:09.036302+0000","modified":"2024-02-21T23:04:12.123968+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4987},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4987":{"gid":4987,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6835/1213702315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":1213702315},{"type":"v1","addr":"172.21.15.28:6835","nonce":1213702315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-02-21T23:04:15.794 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 18 2024-02-21T23:04:15.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.807+0000 7f7560a00640 1 -- 172.21.15.28:0/4105782557 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f753c06b500 msgr2=0x7f753c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:15.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.807+0000 7f7560a00640 1 --2- 172.21.15.28:0/4105782557 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f753c06b500 0x7f753c06d9c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f755c0aef70 tx=0x7f7548008040 comp rx=0 tx=0).stop 2024-02-21T23:04:15.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.807+0000 7f7560a00640 1 -- 172.21.15.28:0/4105782557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75400039b0 msgr2=0x7f755c1eada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:15.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.807+0000 7f7560a00640 1 --2- 172.21.15.28:0/4105782557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75400039b0 0x7f755c1eada0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f754c02c380 tx=0x7f754c004730 comp rx=0 tx=0).stop 2024-02-21T23:04:15.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.807+0000 7f7560a00640 1 -- 172.21.15.28:0/4105782557 shutdown_connections 2024-02-21T23:04:15.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.807+0000 7f7560a00640 1 --2- 172.21.15.28:0/4105782557 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f753c06b500 0x7f753c06d9c0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:15.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.807+0000 7f7560a00640 1 --2- 172.21.15.28:0/4105782557 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f755c1eb910 0x7f755c1ebd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:15.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.807+0000 7f7560a00640 1 --2- 172.21.15.28:0/4105782557 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f755c154600 0x7f755c0ae740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:15.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.807+0000 7f7560a00640 1 --2- 172.21.15.28:0/4105782557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75400039b0 0x7f755c1eada0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:15.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.807+0000 7f7560a00640 1 -- 172.21.15.28:0/4105782557 >> 172.21.15.28:0/4105782557 conn(0x7f755c0ac370 msgr2=0x7f755c0ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:15.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.807+0000 7f7560a00640 1 -- 172.21.15.28:0/4105782557 shutdown_connections 2024-02-21T23:04:15.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:15.807+0000 7f7560a00640 1 -- 172.21.15.28:0/4105782557 wait complete. 2024-02-21T23:04:16.217 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 18, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:04:09.036302+0000', 'modified': '2024-02-21T23:04:12.123968+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4987}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4987': {'gid': 4987, 'name': 'a', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.28:6835/1213702315', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6833', 'nonce': 1213702315}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 1213702315}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:04:16.218 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-21T23:04:16.218 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-02-21T23:04:16.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.456+0000 7f2706600640 1 Processor -- start 2024-02-21T23:04:16.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.456+0000 7f2706600640 1 -- start start 2024-02-21T23:04:16.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.458+0000 7f2706600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f27000a5a70 0x7f27000c1530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:16.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.458+0000 7f2706600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f27000a54b0 0x7f2700159750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:16.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.458+0000 7f2706600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27000c5200 con 0x7f27000c1150 2024-02-21T23:04:16.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.458+0000 7f2706600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27000c1a70 con 0x7f27000a54b0 2024-02-21T23:04:16.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.458+0000 7f2706600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27000c6250 con 0x7f27000a5a70 2024-02-21T23:04:16.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.458+0000 7f2705c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f27000a54b0 0x7f2700159750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:16.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.458+0000 7f2705c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f27000a54b0 0x7f2700159750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52526/0 (socket says 172.21.15.28:52526) 2024-02-21T23:04:16.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.458+0000 7f2705c00640 1 -- 172.21.15.28:0/1466500157 learned_addr learned my addr 172.21.15.28:0/1466500157 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:16.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.458+0000 7f26ffe00640 1 --2- 172.21.15.28:0/1466500157 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f27000a5a70 0x7f27000c1530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:16.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.459+0000 7f2705c00640 1 -- 172.21.15.28:0/1466500157 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f27000a5a70 msgr2=0x7f27000c1530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:16.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.459+0000 7f2705c00640 1 --2- 172.21.15.28:0/1466500157 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f27000a5a70 0x7f27000c1530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:16.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.459+0000 7f2705c00640 1 -- 172.21.15.28:0/1466500157 >> v1:172.21.15.81:6789/0 conn(0x7f27000c1150 legacy=0x7f2700156930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:16.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.459+0000 7f2705c00640 1 -- 172.21.15.28:0/1466500157 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27000c4f20 con 0x7f27000a54b0 2024-02-21T23:04:16.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.460+0000 7f2705c00640 1 --2- 172.21.15.28:0/1466500157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27000a54b0 0x7f2700159750 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f26f8009e50 tx=0x7f26f802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c36e092acc82bebc server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:16.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.460+0000 7f26ff400640 1 -- 172.21.15.28:0/1466500157 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f802ca80 con 0x7f27000a54b0 2024-02-21T23:04:16.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.460+0000 7f26ff400640 1 -- 172.21.15.28:0/1466500157 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26f802cc20 con 0x7f27000a54b0 2024-02-21T23:04:16.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.460+0000 7f26ff400640 1 -- 172.21.15.28:0/1466500157 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f80343f0 con 0x7f27000a54b0 2024-02-21T23:04:16.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.461+0000 7f2706600640 1 -- 172.21.15.28:0/1466500157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27000a54b0 msgr2=0x7f2700159750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:16.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.461+0000 7f2706600640 1 --2- 172.21.15.28:0/1466500157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27000a54b0 0x7f2700159750 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f26f8009e50 tx=0x7f26f802bee0 comp rx=0 tx=0).stop 2024-02-21T23:04:16.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.461+0000 7f2706600640 1 -- 172.21.15.28:0/1466500157 shutdown_connections 2024-02-21T23:04:16.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.461+0000 7f2706600640 1 --2- 172.21.15.28:0/1466500157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27000a54b0 0x7f2700159750 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:16.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.461+0000 7f2706600640 1 --2- 172.21.15.28:0/1466500157 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f27000a5a70 0x7f27000c1530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:16.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.461+0000 7f2706600640 1 -- 172.21.15.28:0/1466500157 >> 172.21.15.28:0/1466500157 conn(0x7f27000a8320 msgr2=0x7f27000a8730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:16.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.461+0000 7f2706600640 1 -- 172.21.15.28:0/1466500157 shutdown_connections 2024-02-21T23:04:16.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.461+0000 7f2706600640 1 -- 172.21.15.28:0/1466500157 wait complete. 2024-02-21T23:04:16.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.463+0000 7f2706600640 1 Processor -- start 2024-02-21T23:04:16.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.463+0000 7f2706600640 1 -- start start 2024-02-21T23:04:16.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.464+0000 7f2706600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f27000a54b0 0x7f27000a71a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:16.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.464+0000 7f2706600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27000a5a70 0x7f270015c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:16.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.464+0000 7f2706600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f27000c1150 0x7f27001eec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:16.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.464+0000 7f2706600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2700157040 con 0x7f27000a5a70 2024-02-21T23:04:16.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.464+0000 7f2706600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27000c1cb0 con 0x7f27000c1150 2024-02-21T23:04:16.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.464+0000 7f2706600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27000a4ab0 con 0x7f27000a54b0 2024-02-21T23:04:16.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.464+0000 7f26ffe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27000a5a70 0x7f270015c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:16.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.464+0000 7f26ffe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27000a5a70 0x7f270015c0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52536/0 (socket says 172.21.15.28:52536) 2024-02-21T23:04:16.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.464+0000 7f26ffe00640 1 -- 172.21.15.28:0/2780931156 learned_addr learned my addr 172.21.15.28:0/2780931156 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:16.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.464+0000 7f2705200640 1 --2- 172.21.15.28:0/2780931156 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f27000a54b0 0x7f27000a71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:16.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.465+0000 7f26ffe00640 1 -- 172.21.15.28:0/2780931156 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f27000a54b0 msgr2=0x7f27000a71a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:16.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.465+0000 7f26ffe00640 1 --2- 172.21.15.28:0/2780931156 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f27000a54b0 0x7f27000a71a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:16.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.465+0000 7f2705c00640 1 --2- 172.21.15.28:0/2780931156 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f27000c1150 0x7f27001eec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:16.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.465+0000 7f26ffe00640 1 -- 172.21.15.28:0/2780931156 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f27000c1150 msgr2=0x7f27001eec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:16.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.465+0000 7f26ffe00640 1 --2- 172.21.15.28:0/2780931156 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f27000c1150 0x7f27001eec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:16.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.465+0000 7f26ffe00640 1 -- 172.21.15.28:0/2780931156 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f270015b0d0 con 0x7f27000a5a70 2024-02-21T23:04:16.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.465+0000 7f2705200640 1 --2- 172.21.15.28:0/2780931156 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f27000a54b0 0x7f27000a71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:04:16.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.465+0000 7f2705c00640 1 --2- 172.21.15.28:0/2780931156 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f27000c1150 0x7f27001eec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:04:16.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.465+0000 7f26ffe00640 1 --2- 172.21.15.28:0/2780931156 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27000a5a70 0x7f270015c0d0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f26ec00d800 tx=0x7f26ec00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:16.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.465+0000 7f26fcc00640 1 -- 172.21.15.28:0/2780931156 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26ec00b6c0 con 0x7f27000a5a70 2024-02-21T23:04:16.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.465+0000 7f26fcc00640 1 -- 172.21.15.28:0/2780931156 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26ec019070 con 0x7f27000a5a70 2024-02-21T23:04:16.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.465+0000 7f26fcc00640 1 -- 172.21.15.28:0/2780931156 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26ec00baa0 con 0x7f27000a5a70 2024-02-21T23:04:16.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.465+0000 7f2706600640 1 -- 172.21.15.28:0/2780931156 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f270015b3b0 con 0x7f27000a5a70 2024-02-21T23:04:16.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.465+0000 7f2706600640 1 -- 172.21.15.28:0/2780931156 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27000a4740 con 0x7f27000a5a70 2024-02-21T23:04:16.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.468+0000 7f26fcc00640 1 -- 172.21.15.28:0/2780931156 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f26ec00bc40 con 0x7f27000a5a70 2024-02-21T23:04:16.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.469+0000 7f26fcc00640 1 --2- 172.21.15.28:0/2780931156 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f26e006b500 0x7f26e006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:16.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.469+0000 7f26fcc00640 1 -- 172.21.15.28:0/2780931156 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f26ec08a850 con 0x7f27000a5a70 2024-02-21T23:04:16.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.469+0000 7f270aa6a740 1 Processor -- start 2024-02-21T23:04:16.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.469+0000 7f270aa6a740 1 -- start start 2024-02-21T23:04:16.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.469+0000 7f2705200640 1 --2- 172.21.15.28:0/2780931156 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f26e006b500 0x7f26e006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:16.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.470+0000 7f270aa6a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559462aff020 0x559462aff400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:16.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.470+0000 7f270aa6a740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x559462b00330 0x559462b3b170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:16.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.470+0000 7f270aa6a740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559462b3b6b0 0x559462b39720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:16.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.470+0000 7f270aa6a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x559462a63470 con 0x559462aff020 2024-02-21T23:04:16.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.470+0000 7f270aa6a740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x559462a0caf0 con 0x559462b3b6b0 2024-02-21T23:04:16.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.470+0000 7f270aa6a740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x559462a5f510 con 0x559462b00330 2024-02-21T23:04:16.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.470+0000 7f2705200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559462aff020 0x559462aff400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:16.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.470+0000 7f2705200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559462aff020 0x559462aff400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52542/0 (socket says 172.21.15.28:52542) 2024-02-21T23:04:16.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.470+0000 7f2705200640 1 -- 172.21.15.28:0/1171131013 learned_addr learned my addr 172.21.15.28:0/1171131013 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:16.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.471+0000 7f2705200640 1 --2- 172.21.15.28:0/2780931156 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f26e006b500 0x7f26e006d9c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f26f4000f60 tx=0x7f26f4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:16.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.471+0000 7f26ffe00640 1 --2- 172.21.15.28:0/1171131013 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x559462b00330 0x559462b3b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:16.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.471+0000 7f2705c00640 1 --2- 172.21.15.28:0/1171131013 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559462b3b6b0 0x559462b39720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:16.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.471+0000 7f2705200640 1 -- 172.21.15.28:0/1171131013 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x559462b00330 msgr2=0x559462b3b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:16.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.471+0000 7f2705200640 1 --2- 172.21.15.28:0/1171131013 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x559462b00330 0x559462b3b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:16.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.471+0000 7f2705200640 1 -- 172.21.15.28:0/1171131013 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559462b3b6b0 msgr2=0x559462b39720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:16.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.471+0000 7f2705200640 1 --2- 172.21.15.28:0/1171131013 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559462b3b6b0 0x559462b39720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:16.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.471+0000 7f2705200640 1 -- 172.21.15.28:0/1171131013 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559462a13d90 con 0x559462aff020 2024-02-21T23:04:16.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.471+0000 7f26ffe00640 1 --2- 172.21.15.28:0/1171131013 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x559462b00330 0x559462b3b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:04:16.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.471+0000 7f2705c00640 1 --2- 172.21.15.28:0/1171131013 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559462b3b6b0 0x559462b39720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:04:16.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.471+0000 7f2705200640 1 --2- 172.21.15.28:0/1171131013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559462aff020 0x559462aff400 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f26f400daf0 tx=0x7f26f4010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=174dc30d874aa3f5 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:16.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.472+0000 7f26f1600640 1 -- 172.21.15.28:0/1171131013 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f4024070 con 0x559462aff020 2024-02-21T23:04:16.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.472+0000 7f26f1600640 1 -- 172.21.15.28:0/1171131013 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26f4014db0 con 0x559462aff020 2024-02-21T23:04:16.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.472+0000 7f26f1600640 1 -- 172.21.15.28:0/1171131013 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f401f600 con 0x559462aff020 2024-02-21T23:04:16.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.472+0000 7f270aa6a740 1 -- 172.21.15.28:0/1171131013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559462aff020 msgr2=0x559462aff400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:16.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.472+0000 7f270aa6a740 1 --2- 172.21.15.28:0/1171131013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559462aff020 0x559462aff400 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f26f400daf0 tx=0x7f26f4010000 comp rx=0 tx=0).stop 2024-02-21T23:04:16.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.472+0000 7f270aa6a740 1 -- 172.21.15.28:0/1171131013 shutdown_connections 2024-02-21T23:04:16.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.472+0000 7f270aa6a740 1 --2- 172.21.15.28:0/1171131013 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x559462b00330 0x559462b3b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:16.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.472+0000 7f270aa6a740 1 --2- 172.21.15.28:0/1171131013 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559462b3b6b0 0x559462b39720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:16.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.472+0000 7f270aa6a740 1 --2- 172.21.15.28:0/1171131013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559462aff020 0x559462aff400 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:16.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.472+0000 7f270aa6a740 1 -- 172.21.15.28:0/1171131013 >> 172.21.15.28:0/1171131013 conn(0x559462afb970 msgr2=0x559462b20380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:16.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.472+0000 7f270aa6a740 1 -- 172.21.15.28:0/1171131013 shutdown_connections 2024-02-21T23:04:16.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.473+0000 7f270aa6a740 1 -- 172.21.15.28:0/1171131013 wait complete. 2024-02-21T23:04:16.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.473+0000 7f270aa6a740 1 -- start start 2024-02-21T23:04:16.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.473+0000 7f270aa6a740 1 Processor -- start 2024-02-21T23:04:16.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.474+0000 7f270aa6a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559462b00330 0x559462b4eb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:16.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.474+0000 7f270aa6a740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559462b3b6b0 0x559462b1ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:16.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.474+0000 7f270aa6a740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x559462b221e0 0x559462b0cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:16.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.474+0000 7f270aa6a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x559462a02040 con 0x559462b00330 2024-02-21T23:04:16.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.474+0000 7f270aa6a740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x559462b20a90 con 0x559462b3b6b0 2024-02-21T23:04:16.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.474+0000 7f270aa6a740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x559462b1bad0 con 0x559462b221e0 2024-02-21T23:04:16.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.475+0000 7f270aa6a740 10 client.0 _ll_register_callbacks cb 0x559462a1f920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:04:16.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.475+0000 7f2705200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559462b00330 0x559462b4eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:16.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.475+0000 7f2705200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559462b00330 0x559462b4eb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52550/0 (socket says 172.21.15.28:52550) 2024-02-21T23:04:16.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.475+0000 7f2705200640 1 -- 172.21.15.28:0/3118611579 learned_addr learned my addr 172.21.15.28:0/3118611579 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:16.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.475+0000 7f2705c00640 1 --2- 172.21.15.28:0/3118611579 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x559462b221e0 0x559462b0cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:16.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.475+0000 7f26ffe00640 1 --2- 172.21.15.28:0/3118611579 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559462b3b6b0 0x559462b1ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:16.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.475+0000 7f2705200640 1 -- 172.21.15.28:0/3118611579 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x559462b221e0 msgr2=0x559462b0cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:16.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.475+0000 7f2705200640 1 --2- 172.21.15.28:0/3118611579 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x559462b221e0 0x559462b0cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:16.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.475+0000 7f2705200640 1 -- 172.21.15.28:0/3118611579 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559462b3b6b0 msgr2=0x559462b1ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:16.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.475+0000 7f2705200640 1 --2- 172.21.15.28:0/3118611579 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559462b3b6b0 0x559462b1ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:16.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.475+0000 7f2705200640 1 -- 172.21.15.28:0/3118611579 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559462a21aa0 con 0x559462b00330 2024-02-21T23:04:16.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.476+0000 7f2705200640 1 --2- 172.21.15.28:0/3118611579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559462b00330 0x559462b4eb10 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f26f4009010 tx=0x7f26f4000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da5af7db2b24686b server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:16.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.476+0000 7f26e6000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:04:16.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.476+0000 7f26e6000640 1 -- 172.21.15.28:0/3118611579 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f402d060 con 0x559462b00330 2024-02-21T23:04:16.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.476+0000 7f270aa6a740 1 -- 172.21.15.28:0/3118611579 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f26f4014db0 con 0x559462b00330 2024-02-21T23:04:16.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.476+0000 7f26e6000640 1 -- 172.21.15.28:0/3118611579 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(38..38 src has 1..38) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f26f4007470 con 0x559462b00330 2024-02-21T23:04:16.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.476+0000 7f26e6000640 1 -- 172.21.15.28:0/3118611579 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=18) v2 ==== 24+0+0 (secure 0 0 0) 0x7f26f4033050 con 0x559462b00330 2024-02-21T23:04:16.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.476+0000 7f270aa6a740 10 client.5394 fetch_fsmap learned FSMap version 18 2024-02-21T23:04:16.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.476+0000 7f270aa6a740 1 -- 172.21.15.28:0/3118611579 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=18,monmap=2+}) v3 -- 0x5594627782d0 con 0x559462b00330 2024-02-21T23:04:16.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.477+0000 7f26e6000640 1 -- 172.21.15.28:0/3118611579 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26f4014db0 con 0x559462b00330 2024-02-21T23:04:16.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.477+0000 7f26e6000640 1 -- 172.21.15.28:0/3118611579 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 18) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f26f4036020 con 0x559462b00330 2024-02-21T23:04:16.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.477+0000 7f270aa6a740 10 client.5394 fetch_fsmap finished waiting for FSMap version 18 2024-02-21T23:04:16.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.477+0000 7f270aa6a740 10 client.5394 resolve_mds: resolved name 'a' to daemon mds.a 2024-02-21T23:04:16.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.477+0000 7f270aa6a740 20 client.5394 populate_metadata read hostname 'smithi028' 2024-02-21T23:04:16.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.477+0000 7f270aa6a740 1 --2- 172.21.15.28:0/3118611579 >> [v2:172.21.15.28:6833/1213702315,v1:172.21.15.28:6835/1213702315] conn(0x559462b38230 0x559462b38610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:16.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.477+0000 7f270aa6a740 4 client.5394 mds_command: new command op to 4987 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-21T23:04:16.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.477+0000 7f270aa6a740 1 -- 172.21.15.28:0/3118611579 --> [v2:172.21.15.28:6833/1213702315,v1:172.21.15.28:6835/1213702315] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559462a709b0 con 0x559462b38230 2024-02-21T23:04:16.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.477+0000 7f26ffe00640 1 --2- 172.21.15.28:0/3118611579 >> [v2:172.21.15.28:6833/1213702315,v1:172.21.15.28:6835/1213702315] conn(0x559462b38230 0x559462b38610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:16.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.478+0000 7f26ffe00640 1 --2- 172.21.15.28:0/3118611579 >> [v2:172.21.15.28:6833/1213702315,v1:172.21.15.28:6835/1213702315] conn(0x559462b38230 0x559462b38610 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7381b0c2c0f2a018 server_cookie=58baa657a9147f0d in_seq=0 out_seq=0 2024-02-21T23:04:16.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.478+0000 7f26e6000640 10 client.5394 ms_handle_connect on v2:172.21.15.28:6833/1213702315 2024-02-21T23:04:16.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.478+0000 7f26e6000640 1 -- 172.21.15.28:0/3118611579 <== mds.0 v2:172.21.15.28:6833/1213702315 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x559462a709b0 con 0x559462b38230 2024-02-21T23:04:16.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.478+0000 7f26e6000640 10 client.5394 handle_command_reply: tid=0 2024-02-21T23:04:16.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.479+0000 7f270aa6a740 1 client.5394 shutdown 2024-02-21T23:04:16.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.479+0000 7f270aa6a740 1 -- 172.21.15.28:0/3118611579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559462b00330 msgr2=0x559462b4eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:16.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.479+0000 7f270aa6a740 1 --2- 172.21.15.28:0/3118611579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559462b00330 0x559462b4eb10 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f26f4009010 tx=0x7f26f4000ed0 comp rx=0 tx=0).stop 2024-02-21T23:04:16.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.479+0000 7f270aa6a740 1 -- 172.21.15.28:0/3118611579 shutdown_connections 2024-02-21T23:04:16.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.479+0000 7f270aa6a740 1 --2- 172.21.15.28:0/3118611579 >> [v2:172.21.15.28:6833/1213702315,v1:172.21.15.28:6835/1213702315] conn(0x559462b38230 0x559462b38610 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:16.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.479+0000 7f270aa6a740 1 --2- 172.21.15.28:0/3118611579 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x559462b221e0 0x559462b0cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:16.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.479+0000 7f270aa6a740 1 --2- 172.21.15.28:0/3118611579 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559462b3b6b0 0x559462b1ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:16.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.479+0000 7f26e6000640 0 client.5394 ms_handle_reset on v2:172.21.15.28:6833/1213702315 2024-02-21T23:04:16.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.479+0000 7f270aa6a740 1 --2- 172.21.15.28:0/3118611579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559462b00330 0x559462b4eb10 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:16.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.479+0000 7f270aa6a740 1 -- 172.21.15.28:0/3118611579 >> 172.21.15.28:0/3118611579 conn(0x559462aff020 msgr2=0x559462b39ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:16.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.479+0000 7f270aa6a740 1 -- 172.21.15.28:0/3118611579 shutdown_connections 2024-02-21T23:04:16.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.479+0000 7f270aa6a740 1 -- 172.21.15.28:0/3118611579 wait complete. 2024-02-21T23:04:16.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.479+0000 7f270aa6a740 20 client.5394 trim_cache size 0 max 16384 2024-02-21T23:04:16.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.498+0000 7f270aa6a740 1 Processor -- start 2024-02-21T23:04:16.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.498+0000 7f270aa6a740 1 -- start start 2024-02-21T23:04:16.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.498+0000 7f270aa6a740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559462b00330 0x559462b201e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:16.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.498+0000 7f270aa6a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559462b221e0 0x559462b3cd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:16.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.498+0000 7f270aa6a740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x559462b3b6b0 0x559462b3d260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:16.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.499+0000 7f270aa6a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27001faf70 con 0x559462b221e0 2024-02-21T23:04:16.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.499+0000 7f270aa6a740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27001f2de0 con 0x559462b00330 2024-02-21T23:04:16.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.499+0000 7f270aa6a740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27001fa970 con 0x559462b3b6b0 2024-02-21T23:04:16.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.499+0000 7f26ffe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559462b221e0 0x559462b3cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:16.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.499+0000 7f26ffe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559462b221e0 0x559462b3cd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52552/0 (socket says 172.21.15.28:52552) 2024-02-21T23:04:16.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.499+0000 7f26ffe00640 1 -- 172.21.15.28:0/2948455988 learned_addr learned my addr 172.21.15.28:0/2948455988 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:16.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.499+0000 7f26ffe00640 1 -- 172.21.15.28:0/2948455988 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x559462b3b6b0 msgr2=0x559462b3d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:16.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.499+0000 7f2705c00640 1 --2- 172.21.15.28:0/2948455988 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x559462b3b6b0 0x559462b3d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:16.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.499+0000 7f2705200640 1 --2- 172.21.15.28:0/2948455988 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559462b00330 0x559462b201e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:16.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.499+0000 7f26ffe00640 1 --2- 172.21.15.28:0/2948455988 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x559462b3b6b0 0x559462b3d260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:16.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.499+0000 7f26ffe00640 1 -- 172.21.15.28:0/2948455988 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559462b00330 msgr2=0x559462b201e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:16.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.499+0000 7f26ffe00640 1 --2- 172.21.15.28:0/2948455988 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559462b00330 0x559462b201e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:16.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.499+0000 7f26ffe00640 1 -- 172.21.15.28:0/2948455988 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559462b38980 con 0x559462b221e0 2024-02-21T23:04:16.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.500+0000 7f26ffe00640 1 --2- 172.21.15.28:0/2948455988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559462b221e0 0x559462b3cd20 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f26ec006fd0 tx=0x7f26ec088ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5c5d772e87339f3 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:16.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.500+0000 7f26f0c00640 1 -- 172.21.15.28:0/2948455988 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26ec05cf80 con 0x559462b221e0 2024-02-21T23:04:16.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.500+0000 7f26f0c00640 1 -- 172.21.15.28:0/2948455988 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26ec087070 con 0x559462b221e0 2024-02-21T23:04:16.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.500+0000 7f26f0c00640 1 -- 172.21.15.28:0/2948455988 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26ec061540 con 0x559462b221e0 2024-02-21T23:04:16.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.500+0000 7f270aa6a740 1 -- 172.21.15.28:0/2948455988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559462b221e0 msgr2=0x559462b3cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:16.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.500+0000 7f270aa6a740 1 --2- 172.21.15.28:0/2948455988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559462b221e0 0x559462b3cd20 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f26ec006fd0 tx=0x7f26ec088ab0 comp rx=0 tx=0).stop 2024-02-21T23:04:16.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.501+0000 7f270aa6a740 1 -- 172.21.15.28:0/2948455988 shutdown_connections 2024-02-21T23:04:16.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.501+0000 7f270aa6a740 1 --2- 172.21.15.28:0/2948455988 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x559462b3b6b0 0x559462b3d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:16.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.501+0000 7f270aa6a740 1 --2- 172.21.15.28:0/2948455988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559462b221e0 0x559462b3cd20 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:16.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.501+0000 7f270aa6a740 1 --2- 172.21.15.28:0/2948455988 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559462b00330 0x559462b201e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:16.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.501+0000 7f270aa6a740 1 -- 172.21.15.28:0/2948455988 >> 172.21.15.28:0/2948455988 conn(0x559462aff020 msgr2=0x559462b0cac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:16.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.501+0000 7f270aa6a740 1 -- 172.21.15.28:0/2948455988 shutdown_connections 2024-02-21T23:04:16.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.501+0000 7f270aa6a740 1 -- 172.21.15.28:0/2948455988 wait complete. 2024-02-21T23:04:16.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.501+0000 7f270aa6a740 1 -- start start 2024-02-21T23:04:16.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.502+0000 7f270aa6a740 1 Processor -- start 2024-02-21T23:04:16.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.502+0000 7f270aa6a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559462b00330 0x559462b55ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:16.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.502+0000 7f270aa6a740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x559462b221e0 0x559462b47cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:16.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.502+0000 7f270aa6a740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559462b3b6b0 0x559462b3e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:16.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.502+0000 7f270aa6a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x559462b1fd70 con 0x559462b00330 2024-02-21T23:04:16.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.502+0000 7f270aa6a740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x559462b21530 con 0x559462b3b6b0 2024-02-21T23:04:16.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.502+0000 7f270aa6a740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x559462b0d320 con 0x559462b221e0 2024-02-21T23:04:16.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.503+0000 7f270aa6a740 10 client.0 _ll_register_callbacks cb 0x7f26f400d960 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:04:16.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.503+0000 7f2705200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559462b00330 0x559462b55ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:16.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.503+0000 7f2705200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559462b00330 0x559462b55ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52558/0 (socket says 172.21.15.28:52558) 2024-02-21T23:04:16.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.503+0000 7f2705200640 1 -- 172.21.15.28:0/206137097 learned_addr learned my addr 172.21.15.28:0/206137097 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:16.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.503+0000 7f26ffe00640 1 --2- 172.21.15.28:0/206137097 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x559462b221e0 0x559462b47cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:16.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.503+0000 7f2705c00640 1 --2- 172.21.15.28:0/206137097 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559462b3b6b0 0x559462b3e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:16.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.503+0000 7f2705200640 1 -- 172.21.15.28:0/206137097 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x559462b221e0 msgr2=0x559462b47cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:16.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.503+0000 7f2705200640 1 --2- 172.21.15.28:0/206137097 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x559462b221e0 0x559462b47cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:16.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.503+0000 7f2705200640 1 -- 172.21.15.28:0/206137097 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559462b3b6b0 msgr2=0x559462b3e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:16.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.503+0000 7f2705200640 1 --2- 172.21.15.28:0/206137097 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559462b3b6b0 0x559462b3e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:16.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.503+0000 7f2705200640 1 -- 172.21.15.28:0/206137097 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559462aff770 con 0x559462b00330 2024-02-21T23:04:16.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.504+0000 7f2705200640 1 --2- 172.21.15.28:0/206137097 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559462b00330 0x559462b55ee0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f26f400d520 tx=0x7f26f4012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db396c1240ea2635 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:16.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.504+0000 7f26e6000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:04:16.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.504+0000 7f26e6000640 1 -- 172.21.15.28:0/206137097 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f4024040 con 0x559462b00330 2024-02-21T23:04:16.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.504+0000 7f26e6000640 1 -- 172.21.15.28:0/206137097 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(38..38 src has 1..38) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f26f4028c90 con 0x559462b00330 2024-02-21T23:04:16.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.504+0000 7f270aa6a740 1 -- 172.21.15.28:0/206137097 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f26ec087070 con 0x559462b00330 2024-02-21T23:04:16.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.504+0000 7f26e6000640 1 -- 172.21.15.28:0/206137097 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=18) v2 ==== 24+0+0 (secure 0 0 0) 0x7f26f4045050 con 0x559462b00330 2024-02-21T23:04:16.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.504+0000 7f270aa6a740 10 client.5400 fetch_fsmap learned FSMap version 18 2024-02-21T23:04:16.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.504+0000 7f270aa6a740 1 -- 172.21.15.28:0/206137097 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=18,monmap=2+}) v3 -- 0x5594628426b0 con 0x559462b00330 2024-02-21T23:04:16.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.505+0000 7f26e6000640 1 -- 172.21.15.28:0/206137097 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26ec087070 con 0x559462b00330 2024-02-21T23:04:16.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.505+0000 7f26e6000640 1 -- 172.21.15.28:0/206137097 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 18) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f26f4048020 con 0x559462b00330 2024-02-21T23:04:16.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.505+0000 7f270aa6a740 10 client.5400 fetch_fsmap finished waiting for FSMap version 18 2024-02-21T23:04:16.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.505+0000 7f270aa6a740 10 client.5400 resolve_mds: resolved name 'a' to daemon mds.a 2024-02-21T23:04:16.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.505+0000 7f270aa6a740 20 client.5400 populate_metadata read hostname 'smithi028' 2024-02-21T23:04:16.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.505+0000 7f270aa6a740 1 --2- 172.21.15.28:0/206137097 >> [v2:172.21.15.28:6833/1213702315,v1:172.21.15.28:6835/1213702315] conn(0x559462b3ee50 0x559462ba72f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:16.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.505+0000 7f270aa6a740 4 client.5400 mds_command: new command op to 4987 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-21T23:04:16.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.505+0000 7f270aa6a740 1 -- 172.21.15.28:0/206137097 --> [v2:172.21.15.28:6833/1213702315,v1:172.21.15.28:6835/1213702315] -- command(tid 0: {"prefix": "status"}) v1 -- 0x559462a6ad00 con 0x559462b3ee50 2024-02-21T23:04:16.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.505+0000 7f26ffe00640 1 --2- 172.21.15.28:0/206137097 >> [v2:172.21.15.28:6833/1213702315,v1:172.21.15.28:6835/1213702315] conn(0x559462b3ee50 0x559462ba72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:16.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.506+0000 7f26ffe00640 1 --2- 172.21.15.28:0/206137097 >> [v2:172.21.15.28:6833/1213702315,v1:172.21.15.28:6835/1213702315] conn(0x559462b3ee50 0x559462ba72f0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd549bca06313c61 server_cookie=b5cc9b8c40c0623 in_seq=0 out_seq=0 2024-02-21T23:04:16.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.506+0000 7f26e6000640 10 client.5400 ms_handle_connect on v2:172.21.15.28:6833/1213702315 2024-02-21T23:04:16.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.506+0000 7f26e6000640 1 -- 172.21.15.28:0/206137097 <== mds.0 v2:172.21.15.28:6833/1213702315 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x559462a6ad00 con 0x559462b3ee50 2024-02-21T23:04:16.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.506+0000 7f26e6000640 10 client.5400 handle_command_reply: tid=0 2024-02-21T23:04:16.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.506+0000 7f270aa6a740 1 client.5400 shutdown 2024-02-21T23:04:16.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.506+0000 7f270aa6a740 1 -- 172.21.15.28:0/206137097 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559462b00330 msgr2=0x559462b55ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:16.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.506+0000 7f270aa6a740 1 --2- 172.21.15.28:0/206137097 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559462b00330 0x559462b55ee0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f26f400d520 tx=0x7f26f4012000 comp rx=0 tx=0).stop 2024-02-21T23:04:16.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.506+0000 7f270aa6a740 1 -- 172.21.15.28:0/206137097 shutdown_connections 2024-02-21T23:04:16.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.506+0000 7f270aa6a740 1 --2- 172.21.15.28:0/206137097 >> [v2:172.21.15.28:6833/1213702315,v1:172.21.15.28:6835/1213702315] conn(0x559462b3ee50 0x559462ba72f0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:16.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.506+0000 7f270aa6a740 1 --2- 172.21.15.28:0/206137097 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x559462b221e0 0x559462b47cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:16.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.506+0000 7f270aa6a740 1 --2- 172.21.15.28:0/206137097 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x559462b3b6b0 0x559462b3e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:16.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.506+0000 7f270aa6a740 1 --2- 172.21.15.28:0/206137097 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559462b00330 0x559462b55ee0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:16.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.506+0000 7f26e6000640 0 client.5400 ms_handle_reset on v2:172.21.15.28:6833/1213702315 2024-02-21T23:04:16.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.506+0000 7f270aa6a740 1 -- 172.21.15.28:0/206137097 >> 172.21.15.28:0/206137097 conn(0x559462aff020 msgr2=0x559462b48760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:16.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.506+0000 7f270aa6a740 1 -- 172.21.15.28:0/206137097 shutdown_connections 2024-02-21T23:04:16.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.506+0000 7f270aa6a740 1 -- 172.21.15.28:0/206137097 wait complete. 2024-02-21T23:04:16.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.507+0000 7f270aa6a740 20 client.5400 trim_cache size 0 max 16384 2024-02-21T23:04:16.500 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-21T23:04:16.500 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac", 2024-02-21T23:04:16.500 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-02-21T23:04:16.500 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4987, 2024-02-21T23:04:16.500 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-02-21T23:04:16.500 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-02-21T23:04:16.501 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-02-21T23:04:16.501 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 6.9250018549999997, 2024-02-21T23:04:16.501 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 18, 2024-02-21T23:04:16.501 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 38, 2024-02-21T23:04:16.501 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 38, 2024-02-21T23:04:16.501 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 15.931671857 2024-02-21T23:04:16.501 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-21T23:04:16.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.507+0000 7f2706600640 1 -- 172.21.15.28:0/2780931156 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f26e006b500 msgr2=0x7f26e006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:16.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.507+0000 7f2706600640 1 --2- 172.21.15.28:0/2780931156 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f26e006b500 0x7f26e006d9c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f26f4000f60 tx=0x7f26f4009040 comp rx=0 tx=0).stop 2024-02-21T23:04:16.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.507+0000 7f2706600640 1 -- 172.21.15.28:0/2780931156 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27000a5a70 msgr2=0x7f270015c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:16.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.507+0000 7f2706600640 1 --2- 172.21.15.28:0/2780931156 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27000a5a70 0x7f270015c0d0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f26ec00d800 tx=0x7f26ec00dcc0 comp rx=0 tx=0).stop 2024-02-21T23:04:16.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.507+0000 7f2706600640 1 -- 172.21.15.28:0/2780931156 shutdown_connections 2024-02-21T23:04:16.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.507+0000 7f2706600640 1 --2- 172.21.15.28:0/2780931156 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f26e006b500 0x7f26e006d9c0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:16.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.507+0000 7f2706600640 1 --2- 172.21.15.28:0/2780931156 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f27000c1150 0x7f27001eec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:16.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.507+0000 7f2706600640 1 --2- 172.21.15.28:0/2780931156 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27000a5a70 0x7f270015c0d0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:16.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.507+0000 7f2706600640 1 --2- 172.21.15.28:0/2780931156 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f27000a54b0 0x7f27000a71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:16.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.507+0000 7f2706600640 1 -- 172.21.15.28:0/2780931156 >> 172.21.15.28:0/2780931156 conn(0x7f27000a8320 msgr2=0x7f27000c40d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:16.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.507+0000 7f2706600640 1 -- 172.21.15.28:0/2780931156 shutdown_connections 2024-02-21T23:04:16.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.507+0000 7f2706600640 1 -- 172.21.15.28:0/2780931156 wait complete. 2024-02-21T23:04:16.513 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:04:16.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.734+0000 7fd21ee00640 1 Processor -- start 2024-02-21T23:04:16.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.734+0000 7fd21ee00640 1 -- start start 2024-02-21T23:04:16.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.737+0000 7fd21ee00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fd2180c87d0 0x7fd218152bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:16.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.737+0000 7fd21ee00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd218153540 con 0x7fd218071730 2024-02-21T23:04:16.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.737+0000 7fd21ee00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd218153820 con 0x7fd218150440 2024-02-21T23:04:16.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.737+0000 7fd21ee00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd218150890 con 0x7fd2180c87d0 2024-02-21T23:04:16.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.737+0000 7fd21d000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd218071730 0x7fd21814fd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60422/0 (socket says 172.21.15.28:60422) 2024-02-21T23:04:16.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.737+0000 7fd21d000640 1 -- 172.21.15.28:0/3692160948 learned_addr learned my addr 172.21.15.28:0/3692160948 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:04:16.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.738+0000 7fd21da00640 1 --2- 172.21.15.28:0/3692160948 >> v2:172.21.15.81:3300/0 conn(0x7fd2180c87d0 0x7fd218152bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:16.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.738+0000 7fd217e00640 1 -- 172.21.15.28:0/3692160948 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4042603526 0 0) 0x7fd218153540 con 0x7fd218071730 2024-02-21T23:04:16.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.738+0000 7fd217e00640 1 -- 172.21.15.28:0/3692160948 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd200003520 con 0x7fd218071730 2024-02-21T23:04:16.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.738+0000 7fd217e00640 1 -- 172.21.15.28:0/3692160948 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2081527491 0 0) 0x7fd200003520 con 0x7fd218071730 2024-02-21T23:04:16.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.738+0000 7fd217e00640 1 -- 172.21.15.28:0/3692160948 >> v2:172.21.15.81:3300/0 conn(0x7fd2180c87d0 msgr2=0x7fd218152bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:16.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.738+0000 7fd217e00640 1 --2- 172.21.15.28:0/3692160948 >> v2:172.21.15.81:3300/0 conn(0x7fd2180c87d0 0x7fd218152bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:16.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.738+0000 7fd217e00640 1 -- 172.21.15.28:0/3692160948 >> v1:172.21.15.81:6789/0 conn(0x7fd218150440 legacy=0x7fd2180c6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:16.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.738+0000 7fd217e00640 1 -- 172.21.15.28:0/3692160948 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd218153b00 con 0x7fd218071730 2024-02-21T23:04:16.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.739+0000 7fd217e00640 1 -- 172.21.15.28:0/3692160948 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fd204001670 con 0x7fd218071730 2024-02-21T23:04:16.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.739+0000 7fd217e00640 1 -- 172.21.15.28:0/3692160948 >> v1:172.21.15.28:6789/0 conn(0x7fd218071730 legacy=0x7fd21814fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:16.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.739+0000 7fd217e00640 1 --2- 172.21.15.28:0/3692160948 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2000039b0 0x7fd200003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:16.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.739+0000 7fd217e00640 1 --2- 172.21.15.28:0/3692160948 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd200004800 0x7fd200004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:16.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.739+0000 7fd217e00640 1 --2- 172.21.15.28:0/3692160948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd200005210 0x7fd20000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:16.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.739+0000 7fd217e00640 1 -- 172.21.15.28:0/3692160948 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd218150890 con 0x7fd200005210 2024-02-21T23:04:16.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.739+0000 7fd217e00640 1 -- 172.21.15.28:0/3692160948 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd20000c140 con 0x7fd2000039b0 2024-02-21T23:04:16.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.739+0000 7fd217e00640 1 -- 172.21.15.28:0/3692160948 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd20000c340 con 0x7fd200004800 2024-02-21T23:04:16.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.739+0000 7fd21da00640 1 --2- 172.21.15.28:0/3692160948 >> v2:172.21.15.81:3300/0 conn(0x7fd2180c87d0 0x7fd218152bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-21T23:04:16.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.739+0000 7fd21e400640 1 --2- 172.21.15.28:0/3692160948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd200005210 0x7fd20000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:16.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.739+0000 7fd21da00640 1 --2- 172.21.15.28:0/3692160948 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2000039b0 0x7fd200003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:16.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.739+0000 7fd21e400640 1 -- 172.21.15.28:0/3692160948 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd200004800 msgr2=0x7fd200004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:16.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.739+0000 7fd21e400640 1 --2- 172.21.15.28:0/3692160948 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd200004800 0x7fd200004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:16.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.739+0000 7fd21e400640 1 -- 172.21.15.28:0/3692160948 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2000039b0 msgr2=0x7fd200003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:16.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.739+0000 7fd21e400640 1 --2- 172.21.15.28:0/3692160948 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2000039b0 0x7fd200003df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:16.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.739+0000 7fd21e400640 1 -- 172.21.15.28:0/3692160948 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd200003520 con 0x7fd200005210 2024-02-21T23:04:16.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.740+0000 7fd21e400640 1 -- 172.21.15.28:0/3692160948 reap_dead start 2024-02-21T23:04:16.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.741+0000 7fd21e400640 1 --2- 172.21.15.28:0/3692160948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd200005210 0x7fd20000b9c0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fd210001e50 tx=0x7fd200005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d078b2000615fbe server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:16.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.741+0000 7fd217e00640 1 -- 172.21.15.28:0/3692160948 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd210001a70 con 0x7fd200005210 2024-02-21T23:04:16.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.741+0000 7fd217e00640 1 -- 172.21.15.28:0/3692160948 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd210003b90 con 0x7fd200005210 2024-02-21T23:04:16.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.741+0000 7fd217e00640 1 -- 172.21.15.28:0/3692160948 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2100050d0 con 0x7fd200005210 2024-02-21T23:04:16.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.741+0000 7fd21ee00640 1 -- 172.21.15.28:0/3692160948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd200005210 msgr2=0x7fd20000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:16.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.741+0000 7fd21ee00640 1 --2- 172.21.15.28:0/3692160948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd200005210 0x7fd20000b9c0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fd210001e50 tx=0x7fd200005830 comp rx=0 tx=0).stop 2024-02-21T23:04:16.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.742+0000 7fd21ee00640 1 -- 172.21.15.28:0/3692160948 shutdown_connections 2024-02-21T23:04:16.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.742+0000 7fd21ee00640 1 --2- 172.21.15.28:0/3692160948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd200005210 0x7fd20000b9c0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:16.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.742+0000 7fd21ee00640 1 -- 172.21.15.28:0/3692160948 >> 172.21.15.28:0/3692160948 conn(0x7fd21806ae70 msgr2=0x7fd21806b280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:16.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.742+0000 7fd21ee00640 1 -- 172.21.15.28:0/3692160948 shutdown_connections 2024-02-21T23:04:16.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.742+0000 7fd21ee00640 1 -- 172.21.15.28:0/3692160948 wait complete. 2024-02-21T23:04:16.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.743+0000 7fd21ee00640 1 Processor -- start 2024-02-21T23:04:16.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.743+0000 7fd21ee00640 1 -- start start 2024-02-21T23:04:16.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.744+0000 7fd21ee00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd200005210 0x7fd218148b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:16.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.744+0000 7fd21ee00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd2180ad6a0 0x7fd2181555d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:16.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.744+0000 7fd21ee00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd218155b10 0x7fd2180ac9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:16.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.744+0000 7fd21ee00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2180c8bb0 con 0x7fd218155b10 2024-02-21T23:04:16.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.744+0000 7fd21ee00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2180c6850 con 0x7fd200005210 2024-02-21T23:04:16.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.744+0000 7fd21ee00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2180c8d30 con 0x7fd2180ad6a0 2024-02-21T23:04:16.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.745+0000 7fd21e400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd218155b10 0x7fd2180ac9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:16.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.745+0000 7fd21e400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd218155b10 0x7fd2180ac9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52564/0 (socket says 172.21.15.28:52564) 2024-02-21T23:04:16.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.745+0000 7fd21e400640 1 -- 172.21.15.28:0/329437103 learned_addr learned my addr 172.21.15.28:0/329437103 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:16.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.745+0000 7fd21d000640 1 --2- 172.21.15.28:0/329437103 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd2180ad6a0 0x7fd2181555d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:16.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.745+0000 7fd21da00640 1 --2- 172.21.15.28:0/329437103 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd200005210 0x7fd218148b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:16.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.745+0000 7fd21e400640 1 -- 172.21.15.28:0/329437103 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd2180ad6a0 msgr2=0x7fd2181555d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:16.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.745+0000 7fd21e400640 1 --2- 172.21.15.28:0/329437103 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd2180ad6a0 0x7fd2181555d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:16.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.745+0000 7fd21e400640 1 -- 172.21.15.28:0/329437103 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd200005210 msgr2=0x7fd218148b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:16.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.745+0000 7fd21e400640 1 --2- 172.21.15.28:0/329437103 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd200005210 0x7fd218148b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:16.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.745+0000 7fd21e400640 1 -- 172.21.15.28:0/329437103 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2181e97c0 con 0x7fd218155b10 2024-02-21T23:04:16.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.746+0000 7fd21e400640 1 --2- 172.21.15.28:0/329437103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd218155b10 0x7fd2180ac9f0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fd21002c060 tx=0x7fd2100014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:16.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.746+0000 7fd215600640 1 -- 172.21.15.28:0/329437103 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd210006d50 con 0x7fd218155b10 2024-02-21T23:04:16.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.746+0000 7fd215600640 1 -- 172.21.15.28:0/329437103 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd210007030 con 0x7fd218155b10 2024-02-21T23:04:16.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.746+0000 7fd215600640 1 -- 172.21.15.28:0/329437103 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd210007420 con 0x7fd218155b10 2024-02-21T23:04:16.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.746+0000 7fd21ee00640 1 -- 172.21.15.28:0/329437103 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2181e9aa0 con 0x7fd218155b10 2024-02-21T23:04:16.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.746+0000 7fd21ee00640 1 -- 172.21.15.28:0/329437103 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2181ea060 con 0x7fd218155b10 2024-02-21T23:04:16.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.747+0000 7fd21ee00640 1 -- 172.21.15.28:0/329437103 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2181ea260 con 0x7fd218155b10 2024-02-21T23:04:16.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.749+0000 7fd215600640 1 -- 172.21.15.28:0/329437103 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd2100075c0 con 0x7fd218155b10 2024-02-21T23:04:16.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.750+0000 7fd215600640 1 --2- 172.21.15.28:0/329437103 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd1f806b500 0x7fd1f806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:16.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.750+0000 7fd215600640 1 -- 172.21.15.28:0/329437103 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd2100abff0 con 0x7fd218155b10 2024-02-21T23:04:16.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.751+0000 7fd21da00640 1 --2- 172.21.15.28:0/329437103 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd1f806b500 0x7fd1f806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:16.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.752+0000 7fd21da00640 1 --2- 172.21.15.28:0/329437103 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd1f806b500 0x7fd1f806d9c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fd20c009ba0 tx=0x7fd20c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:16.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.755+0000 7fd215600640 1 -- 172.21.15.28:0/329437103 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd2181ea260 con 0x7fd218155b10 2024-02-21T23:04:16.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.996+0000 7fd21ee00640 1 -- 172.21.15.28:0/329437103 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd21804f290 con 0x7fd218155b10 2024-02-21T23:04:16.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:16.997+0000 7fd215600640 1 -- 172.21.15.28:0/329437103 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+4938 (secure 0 0 0) 0x7fd210033030 con 0x7fd218155b10 2024-02-21T23:04:16.986 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:04:16.986 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":18,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4778,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/2827878900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2827878900},{"type":"v1","addr":"172.21.15.28:6834","nonce":2827878900}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":4978,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/3612644344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":3612644344},{"type":"v1","addr":"172.21.15.81:6836","nonce":3612644344}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4981,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3623937361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":3623937361},{"type":"v1","addr":"172.21.15.81:6837","nonce":3623937361}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:04:09.036302+0000","modified":"2024-02-21T23:04:12.123968+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4987},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4987":{"gid":4987,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6835/1213702315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":1213702315},{"type":"v1","addr":"172.21.15.28:6835","nonce":1213702315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-02-21T23:04:16.987 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 18 2024-02-21T23:04:16.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.000+0000 7fd21ee00640 1 -- 172.21.15.28:0/329437103 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd1f806b500 msgr2=0x7fd1f806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:16.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.000+0000 7fd21ee00640 1 --2- 172.21.15.28:0/329437103 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd1f806b500 0x7fd1f806d9c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fd20c009ba0 tx=0x7fd20c008040 comp rx=0 tx=0).stop 2024-02-21T23:04:16.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.000+0000 7fd21ee00640 1 -- 172.21.15.28:0/329437103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd218155b10 msgr2=0x7fd2180ac9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:16.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.000+0000 7fd21ee00640 1 --2- 172.21.15.28:0/329437103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd218155b10 0x7fd2180ac9f0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fd21002c060 tx=0x7fd2100014a0 comp rx=0 tx=0).stop 2024-02-21T23:04:16.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.000+0000 7fd21ee00640 1 -- 172.21.15.28:0/329437103 shutdown_connections 2024-02-21T23:04:16.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.000+0000 7fd21ee00640 1 --2- 172.21.15.28:0/329437103 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd1f806b500 0x7fd1f806d9c0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:16.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.000+0000 7fd21ee00640 1 --2- 172.21.15.28:0/329437103 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd2180ad6a0 0x7fd2181555d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:16.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.000+0000 7fd21ee00640 1 --2- 172.21.15.28:0/329437103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd218155b10 0x7fd2180ac9f0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:16.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.000+0000 7fd21ee00640 1 --2- 172.21.15.28:0/329437103 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd200005210 0x7fd218148b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:16.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.000+0000 7fd21ee00640 1 -- 172.21.15.28:0/329437103 >> 172.21.15.28:0/329437103 conn(0x7fd21806ae70 msgr2=0x7fd21814dc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:16.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.000+0000 7fd21ee00640 1 -- 172.21.15.28:0/329437103 shutdown_connections 2024-02-21T23:04:16.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.001+0000 7fd21ee00640 1 -- 172.21.15.28:0/329437103 wait complete. 2024-02-21T23:04:17.009 INFO:tasks.cephfs.test_failover:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 2, "epoch": 18, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-02-21T23:04:09.036302+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 18, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4987": { "addr": "172.21.15.28:6835/1213702315", "addrs": { "addrvec": [ { "addr": "172.21.15.28:6833", "nonce": 1213702315, "type": "v2" }, { "addr": "172.21.15.28:6835", "nonce": 1213702315, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4987, "incarnation": 14, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 4 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 5, "modified": "2024-02-21T23:04:12.123968+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4987 } } } ], "standbys": [ { "addr": "172.21.15.28:6834/2827878900", "addrs": { "addrvec": [ { "addr": "172.21.15.28:6832", "nonce": 2827878900, "type": "v2" }, { "addr": "172.21.15.28:6834", "nonce": 2827878900, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 13, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4778, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.81:6836/3612644344", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6834", "nonce": 3612644344, "type": "v2" }, { "addr": "172.21.15.81:6836", "nonce": 3612644344, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 12, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4978, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.81:6837/3623937361", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6835", "nonce": 3623937361, "type": "v2" }, { "addr": "172.21.15.81:6837", "nonce": 3623937361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 13, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4981, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-02-21T23:04:17.009 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-02-21T23:04:17.009 INFO:tasks.cephfs.test_failover:current = [{'gid': 4987, 'name': 'a', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.28:6835/1213702315', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6833', 'nonce': 1213702315}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 1213702315}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 4981, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.81:6837/3623937361', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6835', 'nonce': 3623937361}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3623937361}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 13}, {'gid': 4778, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.28:6834/2827878900', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 2827878900}, {'type': 'v1', 'addr': '172.21.15.28:6834', 'nonce': 2827878900}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 13}, {'gid': 4978, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.81:6836/3612644344', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6834', 'nonce': 3612644344}, {'type': 'v1', 'addr': '172.21.15.81:6836', 'nonce': 3612644344}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 12}] 2024-02-21T23:04:17.010 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.c mds_join_fs cephfs 2024-02-21T23:04:17.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.229+0000 7f7909a00640 1 Processor -- start 2024-02-21T23:04:17.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.230+0000 7f7909a00640 1 -- start start 2024-02-21T23:04:17.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.231+0000 7f7909a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f7904152650 0x7f79040c8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:17.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.231+0000 7f7909a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f79041550e0 0x7f7904155530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:17.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.231+0000 7f7909a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7904158920 con 0x7f790406daa0 2024-02-21T23:04:17.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.231+0000 7f7909a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79041586d0 con 0x7f7904152650 2024-02-21T23:04:17.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.231+0000 7f7909a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7904158c00 con 0x7f79041550e0 2024-02-21T23:04:17.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.231+0000 7f7903400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f790406daa0 0x7f79040c7160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60430/0 (socket says 172.21.15.28:60430) 2024-02-21T23:04:17.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.231+0000 7f7903400640 1 -- 172.21.15.28:0/1409710434 learned_addr learned my addr 172.21.15.28:0/1409710434 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:04:17.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.232+0000 7f7903e00640 1 --2- 172.21.15.28:0/1409710434 >> v2:172.21.15.81:3300/0 conn(0x7f7904152650 0x7f79040c8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:17.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.232+0000 7f7909000640 1 --2- 172.21.15.28:0/1409710434 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f79041550e0 0x7f7904155530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:17.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.232+0000 7f7902a00640 1 -- 172.21.15.28:0/1409710434 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1145975550 0 0) 0x7f7904158920 con 0x7f790406daa0 2024-02-21T23:04:17.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.232+0000 7f7902a00640 1 -- 172.21.15.28:0/1409710434 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78e8003520 con 0x7f790406daa0 2024-02-21T23:04:17.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.233+0000 7f7903e00640 1 -- 172.21.15.28:0/1409710434 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f79041550e0 msgr2=0x7f7904155530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:17.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.233+0000 7f7903e00640 1 --2- 172.21.15.28:0/1409710434 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f79041550e0 0x7f7904155530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:17.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.233+0000 7f7902a00640 1 -- 172.21.15.28:0/1409710434 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3360202747 0 0) 0x7f78e8003520 con 0x7f790406daa0 2024-02-21T23:04:17.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.233+0000 7f7903e00640 1 -- 172.21.15.28:0/1409710434 >> v1:172.21.15.28:6789/0 conn(0x7f790406daa0 legacy=0x7f79040c7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:17.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.233+0000 7f7903e00640 1 -- 172.21.15.28:0/1409710434 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7904158ee0 con 0x7f7904152650 2024-02-21T23:04:17.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.235+0000 7f7903e00640 1 --2- 172.21.15.28:0/1409710434 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7904152650 0x7f79040c8ce0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f78f40097a0 tx=0x7f78f402b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a33d0be4ecbe122e server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:17.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.235+0000 7f7902a00640 1 -- 172.21.15.28:0/1409710434 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f4037070 con 0x7f7904152650 2024-02-21T23:04:17.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.235+0000 7f7902a00640 1 -- 172.21.15.28:0/1409710434 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78f4030510 con 0x7f7904152650 2024-02-21T23:04:17.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.235+0000 7f7902a00640 1 -- 172.21.15.28:0/1409710434 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f0001670 con 0x7f7904152650 2024-02-21T23:04:17.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.236+0000 7f7909a00640 1 -- 172.21.15.28:0/1409710434 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7904152650 msgr2=0x7f79040c8ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:17.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.236+0000 7f7909a00640 1 --2- 172.21.15.28:0/1409710434 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7904152650 0x7f79040c8ce0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f78f40097a0 tx=0x7f78f402b710 comp rx=0 tx=0).stop 2024-02-21T23:04:17.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.236+0000 7f7909a00640 1 -- 172.21.15.28:0/1409710434 shutdown_connections 2024-02-21T23:04:17.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.236+0000 7f7909a00640 1 --2- 172.21.15.28:0/1409710434 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f79041550e0 0x7f7904155530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:17.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.236+0000 7f7909a00640 1 --2- 172.21.15.28:0/1409710434 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7904152650 0x7f79040c8ce0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:17.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.236+0000 7f7909a00640 1 -- 172.21.15.28:0/1409710434 >> 172.21.15.28:0/1409710434 conn(0x7f7904072ce0 msgr2=0x7f79040ac340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:17.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.236+0000 7f7909a00640 1 -- 172.21.15.28:0/1409710434 shutdown_connections 2024-02-21T23:04:17.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.236+0000 7f7909a00640 1 -- 172.21.15.28:0/1409710434 wait complete. 2024-02-21T23:04:17.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.238+0000 7f7909a00640 1 Processor -- start 2024-02-21T23:04:17.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.238+0000 7f7909a00640 1 -- start start 2024-02-21T23:04:17.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.239+0000 7f7909a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f790406daa0 0x7f7904157680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:17.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.239+0000 7f7909a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7904152650 0x7f79041484f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:17.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.239+0000 7f7909a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79041550e0 0x7f7904148a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:17.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.239+0000 7f7909a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7904152a30 con 0x7f79041550e0 2024-02-21T23:04:17.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.239+0000 7f7909a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79040bdfb0 con 0x7f7904152650 2024-02-21T23:04:17.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.239+0000 7f7909a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79040c7870 con 0x7f790406daa0 2024-02-21T23:04:17.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.239+0000 7f7909000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79041550e0 0x7f7904148a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:17.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.239+0000 7f7909000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79041550e0 0x7f7904148a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52578/0 (socket says 172.21.15.28:52578) 2024-02-21T23:04:17.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.239+0000 7f7909000640 1 -- 172.21.15.28:0/752027932 learned_addr learned my addr 172.21.15.28:0/752027932 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:17.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.239+0000 7f7903e00640 1 --2- 172.21.15.28:0/752027932 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f790406daa0 0x7f7904157680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:17.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.240+0000 7f7909000640 1 -- 172.21.15.28:0/752027932 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f790406daa0 msgr2=0x7f7904157680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:17.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.240+0000 7f7903400640 1 --2- 172.21.15.28:0/752027932 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7904152650 0x7f79041484f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:17.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.240+0000 7f7909000640 1 --2- 172.21.15.28:0/752027932 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f790406daa0 0x7f7904157680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:17.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.240+0000 7f7909000640 1 -- 172.21.15.28:0/752027932 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7904152650 msgr2=0x7f79041484f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:17.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.240+0000 7f7909000640 1 --2- 172.21.15.28:0/752027932 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7904152650 0x7f79041484f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:17.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.240+0000 7f7909000640 1 -- 172.21.15.28:0/752027932 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79040abbd0 con 0x7f79041550e0 2024-02-21T23:04:17.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.240+0000 7f7909000640 1 --2- 172.21.15.28:0/752027932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79041550e0 0x7f7904148a30 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f78fc009d30 tx=0x7f78fc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:17.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.240+0000 7f78fbe00640 1 -- 172.21.15.28:0/752027932 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78fc018020 con 0x7f79041550e0 2024-02-21T23:04:17.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.240+0000 7f78fbe00640 1 -- 172.21.15.28:0/752027932 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78fc014480 con 0x7f79041550e0 2024-02-21T23:04:17.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.240+0000 7f78fbe00640 1 -- 172.21.15.28:0/752027932 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78fc014870 con 0x7f79041550e0 2024-02-21T23:04:17.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.240+0000 7f7909a00640 1 -- 172.21.15.28:0/752027932 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79040abeb0 con 0x7f79041550e0 2024-02-21T23:04:17.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.241+0000 7f7909a00640 1 -- 172.21.15.28:0/752027932 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79040c4340 con 0x7f79041550e0 2024-02-21T23:04:17.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.242+0000 7f7909a00640 1 -- 172.21.15.28:0/752027932 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f790404f290 con 0x7f79041550e0 2024-02-21T23:04:17.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.243+0000 7f78fbe00640 1 -- 172.21.15.28:0/752027932 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f78fc005d10 con 0x7f79041550e0 2024-02-21T23:04:17.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.244+0000 7f78fbe00640 1 --2- 172.21.15.28:0/752027932 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f78e406b500 0x7f78e406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:17.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.244+0000 7f78fbe00640 1 -- 172.21.15.28:0/752027932 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f78fc08ad60 con 0x7f79041550e0 2024-02-21T23:04:17.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.245+0000 7f7903e00640 1 --2- 172.21.15.28:0/752027932 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f78e406b500 0x7f78e406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:17.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.246+0000 7f7903e00640 1 --2- 172.21.15.28:0/752027932 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f78e406b500 0x7f78e406d9c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f78f4034000 tx=0x7f78f4008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:17.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.249+0000 7f78fbe00640 1 -- 172.21.15.28:0/752027932 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f78fc022070 con 0x7f79041550e0 2024-02-21T23:04:17.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.449+0000 7f7909a00640 1 -- 172.21.15.28:0/752027932 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) v1 -- 0x7f79040c4510 con 0x7f79041550e0 2024-02-21T23:04:17.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.455+0000 7f78fbe00640 1 -- 172.21.15.28:0/752027932 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v12)=0 v12) v1 ==== 116+0+0 (secure 0 0 0) 0x7f78fc010040 con 0x7f79041550e0 2024-02-21T23:04:17.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.458+0000 7f7909a00640 1 -- 172.21.15.28:0/752027932 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f78e406b500 msgr2=0x7f78e406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:17.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.458+0000 7f7909a00640 1 --2- 172.21.15.28:0/752027932 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f78e406b500 0x7f78e406d9c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f78f4034000 tx=0x7f78f4008030 comp rx=0 tx=0).stop 2024-02-21T23:04:17.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.458+0000 7f7909a00640 1 -- 172.21.15.28:0/752027932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79041550e0 msgr2=0x7f7904148a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:17.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.458+0000 7f7909a00640 1 --2- 172.21.15.28:0/752027932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79041550e0 0x7f7904148a30 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f78fc009d30 tx=0x7f78fc00c490 comp rx=0 tx=0).stop 2024-02-21T23:04:17.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.458+0000 7f7909a00640 1 -- 172.21.15.28:0/752027932 shutdown_connections 2024-02-21T23:04:17.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.458+0000 7f7909a00640 1 --2- 172.21.15.28:0/752027932 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f78e406b500 0x7f78e406d9c0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:17.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.458+0000 7f7909a00640 1 --2- 172.21.15.28:0/752027932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79041550e0 0x7f7904148a30 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:17.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.458+0000 7f7909a00640 1 --2- 172.21.15.28:0/752027932 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7904152650 0x7f79041484f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:17.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.458+0000 7f7909a00640 1 --2- 172.21.15.28:0/752027932 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f790406daa0 0x7f7904157680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:17.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.458+0000 7f7909a00640 1 -- 172.21.15.28:0/752027932 >> 172.21.15.28:0/752027932 conn(0x7f7904072ce0 msgr2=0x7f7904154950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:17.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.458+0000 7f7909a00640 1 -- 172.21.15.28:0/752027932 shutdown_connections 2024-02-21T23:04:17.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.458+0000 7f7909a00640 1 -- 172.21.15.28:0/752027932 wait complete. 2024-02-21T23:04:17.466 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:04:17.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.679+0000 7f3c7b400640 1 Processor -- start 2024-02-21T23:04:17.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.680+0000 7f3c7b400640 1 -- start start 2024-02-21T23:04:17.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.681+0000 7f3c7b400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3c741525c0 0x7f3c740cb390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:17.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.681+0000 7f3c7b400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f3c740c3280 0x7f3c740c3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:17.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.681+0000 7f3c7b400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c74155120 con 0x7f3c74159010 2024-02-21T23:04:17.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.681+0000 7f3c7b400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c74155390 con 0x7f3c741525c0 2024-02-21T23:04:17.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.681+0000 7f3c7b400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c74155670 con 0x7f3c740c3280 2024-02-21T23:04:17.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.682+0000 7f3c78a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3c741525c0 0x7f3c740cb390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:17.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.682+0000 7f3c78a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3c741525c0 0x7f3c740cb390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52580/0 (socket says 172.21.15.28:52580) 2024-02-21T23:04:17.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.682+0000 7f3c78a00640 1 -- 172.21.15.28:0/143306420 learned_addr learned my addr 172.21.15.28:0/143306420 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:17.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.682+0000 7f3c78a00640 1 -- 172.21.15.28:0/143306420 >> v2:172.21.15.81:3300/0 conn(0x7f3c740c3280 msgr2=0x7f3c740c3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:17.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.682+0000 7f3c78a00640 1 --2- 172.21.15.28:0/143306420 >> v2:172.21.15.81:3300/0 conn(0x7f3c740c3280 0x7f3c740c3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:17.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.682+0000 7f3c78a00640 1 -- 172.21.15.28:0/143306420 >> v1:172.21.15.81:6789/0 conn(0x7f3c74159010 legacy=0x7f3c740c6a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:17.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.682+0000 7f3c78a00640 1 -- 172.21.15.28:0/143306420 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c74155950 con 0x7f3c741525c0 2024-02-21T23:04:17.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.683+0000 7f3c78a00640 1 --2- 172.21.15.28:0/143306420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c741525c0 0x7f3c740cb390 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f3c7408c1e0 tx=0x7f3c6802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e19c9df65c426d2 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:17.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.683+0000 7f3c73400640 1 -- 172.21.15.28:0/143306420 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c68035020 con 0x7f3c741525c0 2024-02-21T23:04:17.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.683+0000 7f3c73400640 1 -- 172.21.15.28:0/143306420 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c68031870 con 0x7f3c741525c0 2024-02-21T23:04:17.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.683+0000 7f3c73400640 1 -- 172.21.15.28:0/143306420 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c68031c40 con 0x7f3c741525c0 2024-02-21T23:04:17.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.684+0000 7f3c7b400640 1 -- 172.21.15.28:0/143306420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c741525c0 msgr2=0x7f3c740cb390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:17.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.684+0000 7f3c7b400640 1 --2- 172.21.15.28:0/143306420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c741525c0 0x7f3c740cb390 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f3c7408c1e0 tx=0x7f3c6802b910 comp rx=0 tx=0).stop 2024-02-21T23:04:17.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.684+0000 7f3c7b400640 1 -- 172.21.15.28:0/143306420 shutdown_connections 2024-02-21T23:04:17.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.684+0000 7f3c7b400640 1 --2- 172.21.15.28:0/143306420 >> v2:172.21.15.81:3300/0 conn(0x7f3c740c3280 0x7f3c740c3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:17.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.684+0000 7f3c7b400640 1 --2- 172.21.15.28:0/143306420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c741525c0 0x7f3c740cb390 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:17.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.684+0000 7f3c7b400640 1 -- 172.21.15.28:0/143306420 >> 172.21.15.28:0/143306420 conn(0x7f3c740729c0 msgr2=0x7f3c74072dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:17.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.684+0000 7f3c7b400640 1 -- 172.21.15.28:0/143306420 shutdown_connections 2024-02-21T23:04:17.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.684+0000 7f3c7b400640 1 -- 172.21.15.28:0/143306420 wait complete. 2024-02-21T23:04:17.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.686+0000 7f3c7b400640 1 Processor -- start 2024-02-21T23:04:17.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.686+0000 7f3c7b400640 1 -- start start 2024-02-21T23:04:17.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.687+0000 7f3c7b400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3c740c3280 0x7f3c741eeb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:17.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.687+0000 7f3c7b400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c741525c0 0x7f3c74149770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:17.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.687+0000 7f3c7b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c7414a670 0x7f3c74156dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:17.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.687+0000 7f3c7b400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c741594a0 con 0x7f3c7414a670 2024-02-21T23:04:17.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.687+0000 7f3c7b400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c740caff0 con 0x7f3c741525c0 2024-02-21T23:04:17.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.687+0000 7f3c7b400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c740c3bd0 con 0x7f3c740c3280 2024-02-21T23:04:17.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.687+0000 7f3c79400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c7414a670 0x7f3c74156dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:17.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.687+0000 7f3c79400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c7414a670 0x7f3c74156dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52584/0 (socket says 172.21.15.28:52584) 2024-02-21T23:04:17.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.687+0000 7f3c79400640 1 -- 172.21.15.28:0/60315760 learned_addr learned my addr 172.21.15.28:0/60315760 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:17.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.688+0000 7f3c78a00640 1 --2- 172.21.15.28:0/60315760 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3c740c3280 0x7f3c741eeb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:17.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.688+0000 7f3c73e00640 1 --2- 172.21.15.28:0/60315760 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c741525c0 0x7f3c74149770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:17.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.688+0000 7f3c79400640 1 -- 172.21.15.28:0/60315760 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3c740c3280 msgr2=0x7f3c741eeb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:17.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.688+0000 7f3c79400640 1 --2- 172.21.15.28:0/60315760 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3c740c3280 0x7f3c741eeb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:17.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.688+0000 7f3c79400640 1 -- 172.21.15.28:0/60315760 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c741525c0 msgr2=0x7f3c74149770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:17.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.688+0000 7f3c79400640 1 --2- 172.21.15.28:0/60315760 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c741525c0 0x7f3c74149770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:17.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.688+0000 7f3c79400640 1 -- 172.21.15.28:0/60315760 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c74156010 con 0x7f3c7414a670 2024-02-21T23:04:17.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.688+0000 7f3c79400640 1 --2- 172.21.15.28:0/60315760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c7414a670 0x7f3c74156dd0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f3c6c00ba10 tx=0x7f3c6c00bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:17.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.688+0000 7f3c70c00640 1 -- 172.21.15.28:0/60315760 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c6c019070 con 0x7f3c7414a670 2024-02-21T23:04:17.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.689+0000 7f3c70c00640 1 -- 172.21.15.28:0/60315760 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c6c00cae0 con 0x7f3c7414a670 2024-02-21T23:04:17.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.689+0000 7f3c70c00640 1 -- 172.21.15.28:0/60315760 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c6c014440 con 0x7f3c7414a670 2024-02-21T23:04:17.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.689+0000 7f3c7b400640 1 -- 172.21.15.28:0/60315760 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c741562f0 con 0x7f3c7414a670 2024-02-21T23:04:17.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.689+0000 7f3c7b400640 1 -- 172.21.15.28:0/60315760 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c741568b0 con 0x7f3c7414a670 2024-02-21T23:04:17.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.690+0000 7f3c7b400640 1 -- 172.21.15.28:0/60315760 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c74067f30 con 0x7f3c7414a670 2024-02-21T23:04:17.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.691+0000 7f3c70c00640 1 -- 172.21.15.28:0/60315760 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3c6c01c070 con 0x7f3c7414a670 2024-02-21T23:04:17.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.692+0000 7f3c70c00640 1 --2- 172.21.15.28:0/60315760 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3c5406b4d0 0x7f3c5406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:17.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.692+0000 7f3c70c00640 1 -- 172.21.15.28:0/60315760 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f3c6c089fd0 con 0x7f3c7414a670 2024-02-21T23:04:17.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.692+0000 7f3c78a00640 1 --2- 172.21.15.28:0/60315760 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3c5406b4d0 0x7f3c5406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:17.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.694+0000 7f3c78a00640 1 --2- 172.21.15.28:0/60315760 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3c5406b4d0 0x7f3c5406d990 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f3c68036000 tx=0x7f3c6803c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:17.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.701+0000 7f3c70c00640 1 -- 172.21.15.28:0/60315760 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3c6c058d50 con 0x7f3c7414a670 2024-02-21T23:04:17.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.941+0000 7f3c7b400640 1 -- 172.21.15.28:0/60315760 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c740b0130 con 0x7f3c7414a670 2024-02-21T23:04:17.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.943+0000 7f3c70c00640 1 -- 172.21.15.28:0/60315760 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+4938 (secure 0 0 0) 0x7f3c6c058650 con 0x7f3c7414a670 2024-02-21T23:04:17.931 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:04:17.932 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":18,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4778,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/2827878900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2827878900},{"type":"v1","addr":"172.21.15.28:6834","nonce":2827878900}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":4978,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/3612644344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":3612644344},{"type":"v1","addr":"172.21.15.81:6836","nonce":3612644344}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4981,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3623937361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":3623937361},{"type":"v1","addr":"172.21.15.81:6837","nonce":3623937361}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:04:09.036302+0000","modified":"2024-02-21T23:04:12.123968+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4987},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4987":{"gid":4987,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6835/1213702315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":1213702315},{"type":"v1","addr":"172.21.15.28:6835","nonce":1213702315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-02-21T23:04:17.932 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 18 2024-02-21T23:04:17.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.945+0000 7f3c7b400640 1 -- 172.21.15.28:0/60315760 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3c5406b4d0 msgr2=0x7f3c5406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:17.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.945+0000 7f3c7b400640 1 --2- 172.21.15.28:0/60315760 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3c5406b4d0 0x7f3c5406d990 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f3c68036000 tx=0x7f3c6803c040 comp rx=0 tx=0).stop 2024-02-21T23:04:17.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.945+0000 7f3c7b400640 1 -- 172.21.15.28:0/60315760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c7414a670 msgr2=0x7f3c74156dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:17.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.945+0000 7f3c7b400640 1 --2- 172.21.15.28:0/60315760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c7414a670 0x7f3c74156dd0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f3c6c00ba10 tx=0x7f3c6c00bed0 comp rx=0 tx=0).stop 2024-02-21T23:04:17.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.946+0000 7f3c7b400640 1 -- 172.21.15.28:0/60315760 shutdown_connections 2024-02-21T23:04:17.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.946+0000 7f3c7b400640 1 --2- 172.21.15.28:0/60315760 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3c5406b4d0 0x7f3c5406d990 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:17.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.946+0000 7f3c7b400640 1 --2- 172.21.15.28:0/60315760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c7414a670 0x7f3c74156dd0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:17.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.946+0000 7f3c7b400640 1 --2- 172.21.15.28:0/60315760 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c741525c0 0x7f3c74149770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:17.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.946+0000 7f3c7b400640 1 --2- 172.21.15.28:0/60315760 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3c740c3280 0x7f3c741eeb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:17.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.946+0000 7f3c7b400640 1 -- 172.21.15.28:0/60315760 >> 172.21.15.28:0/60315760 conn(0x7f3c740729c0 msgr2=0x7f3c740c7680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:17.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.946+0000 7f3c7b400640 1 -- 172.21.15.28:0/60315760 shutdown_connections 2024-02-21T23:04:17.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:17.946+0000 7f3c7b400640 1 -- 172.21.15.28:0/60315760 wait complete. 2024-02-21T23:04:18.355 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 18, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:04:09.036302+0000', 'modified': '2024-02-21T23:04:12.123968+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4987}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4987': {'gid': 4987, 'name': 'a', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.28:6835/1213702315', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6833', 'nonce': 1213702315}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 1213702315}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:04:18.356 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-21T23:04:18.356 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-02-21T23:04:18.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.589+0000 7fd4e3e00640 1 Processor -- start 2024-02-21T23:04:18.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.589+0000 7fd4e3e00640 1 -- start start 2024-02-21T23:04:18.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.591+0000 7fd4e3e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fd4e41527a0 0x7fd4e4158f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:18.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.591+0000 7fd4e3e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4e40c2070 con 0x7fd4e40bf9a0 2024-02-21T23:04:18.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.591+0000 7fd4e3e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4e40c2350 con 0x7fd4e406d410 2024-02-21T23:04:18.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.591+0000 7fd4e3e00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4e406e790 con 0x7fd4e41527a0 2024-02-21T23:04:18.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.591+0000 7fd4dbe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd4e40bf9a0 0x7fd4e40bfdf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60438/0 (socket says 172.21.15.28:60438) 2024-02-21T23:04:18.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.591+0000 7fd4dbe00640 1 -- 172.21.15.28:0/3624256790 learned_addr learned my addr 172.21.15.28:0/3624256790 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:04:18.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.591+0000 7fd4e2a00640 1 --2- 172.21.15.28:0/3624256790 >> v2:172.21.15.81:3300/0 conn(0x7fd4e41527a0 0x7fd4e4158f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:18.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.592+0000 7fd4e2000640 1 -- 172.21.15.28:0/3624256790 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3730746409 0 0) 0x7fd4e40c2070 con 0x7fd4e40bf9a0 2024-02-21T23:04:18.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.592+0000 7fd4e2000640 1 -- 172.21.15.28:0/3624256790 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4c8003520 con 0x7fd4e40bf9a0 2024-02-21T23:04:18.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.592+0000 7fd4e2000640 1 -- 172.21.15.28:0/3624256790 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2583893420 0 0) 0x7fd4c8003520 con 0x7fd4e40bf9a0 2024-02-21T23:04:18.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.592+0000 7fd4e2000640 1 -- 172.21.15.28:0/3624256790 >> v2:172.21.15.81:3300/0 conn(0x7fd4e41527a0 msgr2=0x7fd4e4158f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:18.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.592+0000 7fd4e2000640 1 --2- 172.21.15.28:0/3624256790 >> v2:172.21.15.81:3300/0 conn(0x7fd4e41527a0 0x7fd4e4158f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:18.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.592+0000 7fd4e2000640 1 -- 172.21.15.28:0/3624256790 >> v1:172.21.15.81:6789/0 conn(0x7fd4e406d410 legacy=0x7fd4e406d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:18.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.592+0000 7fd4e2000640 1 -- 172.21.15.28:0/3624256790 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4e40c2630 con 0x7fd4e40bf9a0 2024-02-21T23:04:18.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.592+0000 7fd4e2000640 1 -- 172.21.15.28:0/3624256790 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fd4cc001670 con 0x7fd4e40bf9a0 2024-02-21T23:04:18.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.592+0000 7fd4e2000640 1 -- 172.21.15.28:0/3624256790 >> v1:172.21.15.28:6789/0 conn(0x7fd4e40bf9a0 legacy=0x7fd4e40bfdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:18.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.592+0000 7fd4e2000640 1 --2- 172.21.15.28:0/3624256790 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd4c8003a00 0x7fd4c8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:18.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.592+0000 7fd4e2000640 1 --2- 172.21.15.28:0/3624256790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4c8004800 0x7fd4c8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:18.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.592+0000 7fd4e2000640 1 --2- 172.21.15.28:0/3624256790 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd4c80051c0 0x7fd4c800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:18.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.592+0000 7fd4e2000640 1 -- 172.21.15.28:0/3624256790 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e406e790 con 0x7fd4c8004800 2024-02-21T23:04:18.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.592+0000 7fd4e2000640 1 -- 172.21.15.28:0/3624256790 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c800c0a0 con 0x7fd4c80051c0 2024-02-21T23:04:18.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.592+0000 7fd4e2000640 1 -- 172.21.15.28:0/3624256790 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c800c2a0 con 0x7fd4c8003a00 2024-02-21T23:04:18.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.593+0000 7fd4dbe00640 1 --2- 172.21.15.28:0/3624256790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4c8004800 0x7fd4c8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:18.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.593+0000 7fd4e2a00640 1 --2- 172.21.15.28:0/3624256790 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd4c8003a00 0x7fd4c8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:18.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.593+0000 7fd4e3400640 1 --2- 172.21.15.28:0/3624256790 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd4c80051c0 0x7fd4c800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:18.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.593+0000 7fd4dbe00640 1 -- 172.21.15.28:0/3624256790 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd4c8003a00 msgr2=0x7fd4c8003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:18.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.593+0000 7fd4dbe00640 1 --2- 172.21.15.28:0/3624256790 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd4c8003a00 0x7fd4c8003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:18.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.593+0000 7fd4dbe00640 1 -- 172.21.15.28:0/3624256790 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd4c80051c0 msgr2=0x7fd4c800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:18.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.593+0000 7fd4dbe00640 1 --2- 172.21.15.28:0/3624256790 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd4c80051c0 0x7fd4c800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:18.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.593+0000 7fd4dbe00640 1 -- 172.21.15.28:0/3624256790 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4c8003520 con 0x7fd4c8004800 2024-02-21T23:04:18.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.593+0000 7fd4e3400640 1 --2- 172.21.15.28:0/3624256790 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd4c80051c0 0x7fd4c800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:04:18.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.593+0000 7fd4e3400640 1 -- 172.21.15.28:0/3624256790 reap_dead start 2024-02-21T23:04:18.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.594+0000 7fd4dbe00640 1 --2- 172.21.15.28:0/3624256790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4c8004800 0x7fd4c8004c80 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fd4cc00c0e0 tx=0x7fd4c8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=526c7885d508c3e1 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:18.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.595+0000 7fd4e2000640 1 -- 172.21.15.28:0/3624256790 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4cc007c20 con 0x7fd4c8004800 2024-02-21T23:04:18.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.595+0000 7fd4e2000640 1 -- 172.21.15.28:0/3624256790 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4cc030070 con 0x7fd4c8004800 2024-02-21T23:04:18.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.595+0000 7fd4e2000640 1 -- 172.21.15.28:0/3624256790 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4cc005d70 con 0x7fd4c8004800 2024-02-21T23:04:18.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.595+0000 7fd4e3e00640 1 -- 172.21.15.28:0/3624256790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4c8004800 msgr2=0x7fd4c8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:18.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.595+0000 7fd4e3e00640 1 --2- 172.21.15.28:0/3624256790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4c8004800 0x7fd4c8004c80 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fd4cc00c0e0 tx=0x7fd4c8005830 comp rx=0 tx=0).stop 2024-02-21T23:04:18.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.596+0000 7fd4e3e00640 1 -- 172.21.15.28:0/3624256790 shutdown_connections 2024-02-21T23:04:18.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.596+0000 7fd4e3e00640 1 --2- 172.21.15.28:0/3624256790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4c8004800 0x7fd4c8004c80 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:18.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.596+0000 7fd4e3e00640 1 -- 172.21.15.28:0/3624256790 >> 172.21.15.28:0/3624256790 conn(0x7fd4e40a8320 msgr2=0x7fd4e40a8730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:18.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.596+0000 7fd4e3e00640 1 -- 172.21.15.28:0/3624256790 shutdown_connections 2024-02-21T23:04:18.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.596+0000 7fd4e3e00640 1 -- 172.21.15.28:0/3624256790 wait complete. 2024-02-21T23:04:18.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.597+0000 7fd4e3e00640 1 Processor -- start 2024-02-21T23:04:18.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.598+0000 7fd4e3e00640 1 -- start start 2024-02-21T23:04:18.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.599+0000 7fd4e3e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd4c8004800 0x7fd4e40a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:18.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.599+0000 7fd4e3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4e4150140 0x7fd4e40aad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:18.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.599+0000 7fd4e3e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd4e41de5d0 0x7fd4e40ab280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:18.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.599+0000 7fd4e3e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e4158f70 con 0x7fd4e4150140 2024-02-21T23:04:18.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.599+0000 7fd4e3e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e40c29a0 con 0x7fd4e41de5d0 2024-02-21T23:04:18.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.599+0000 7fd4e3e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e41590f0 con 0x7fd4c8004800 2024-02-21T23:04:18.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.599+0000 7fd4dbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4e4150140 0x7fd4e40aad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:18.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.599+0000 7fd4dbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4e4150140 0x7fd4e40aad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52598/0 (socket says 172.21.15.28:52598) 2024-02-21T23:04:18.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.599+0000 7fd4dbe00640 1 -- 172.21.15.28:0/411043501 learned_addr learned my addr 172.21.15.28:0/411043501 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:18.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.599+0000 7fd4e2a00640 1 --2- 172.21.15.28:0/411043501 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd4c8004800 0x7fd4e40a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:18.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.599+0000 7fd4e3400640 1 --2- 172.21.15.28:0/411043501 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd4e41de5d0 0x7fd4e40ab280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:18.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.600+0000 7fd4dbe00640 1 -- 172.21.15.28:0/411043501 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd4c8004800 msgr2=0x7fd4e40a6260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:18.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.600+0000 7fd4dbe00640 1 --2- 172.21.15.28:0/411043501 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd4c8004800 0x7fd4e40a6260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:18.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.600+0000 7fd4dbe00640 1 -- 172.21.15.28:0/411043501 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd4e41de5d0 msgr2=0x7fd4e40ab280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:18.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.600+0000 7fd4dbe00640 1 --2- 172.21.15.28:0/411043501 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd4e41de5d0 0x7fd4e40ab280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:18.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.600+0000 7fd4dbe00640 1 -- 172.21.15.28:0/411043501 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4e414f970 con 0x7fd4e4150140 2024-02-21T23:04:18.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.600+0000 7fd4dbe00640 1 --2- 172.21.15.28:0/411043501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4e4150140 0x7fd4e40aad40 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fd4cc00c200 tx=0x7fd4e40c2f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:18.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.600+0000 7fd4db400640 1 -- 172.21.15.28:0/411043501 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4cc004bf0 con 0x7fd4e4150140 2024-02-21T23:04:18.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.600+0000 7fd4db400640 1 -- 172.21.15.28:0/411043501 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4cc000ed0 con 0x7fd4e4150140 2024-02-21T23:04:18.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.600+0000 7fd4db400640 1 -- 172.21.15.28:0/411043501 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4cc007070 con 0x7fd4e4150140 2024-02-21T23:04:18.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.600+0000 7fd4e3e00640 1 -- 172.21.15.28:0/411043501 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4e40aa2a0 con 0x7fd4e4150140 2024-02-21T23:04:18.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.600+0000 7fd4e3e00640 1 -- 172.21.15.28:0/411043501 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4e40aa540 con 0x7fd4e4150140 2024-02-21T23:04:18.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.603+0000 7fd4db400640 1 -- 172.21.15.28:0/411043501 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd4cc048020 con 0x7fd4e4150140 2024-02-21T23:04:18.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.605+0000 7fd4db400640 1 --2- 172.21.15.28:0/411043501 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd4c006b4d0 0x7fd4c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:18.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.605+0000 7fd4ecc24740 1 Processor -- start 2024-02-21T23:04:18.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.605+0000 7fd4db400640 1 -- 172.21.15.28:0/411043501 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd4cc0ab7e0 con 0x7fd4e4150140 2024-02-21T23:04:18.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.605+0000 7fd4ecc24740 1 -- start start 2024-02-21T23:04:18.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.605+0000 7fd4e2a00640 1 --2- 172.21.15.28:0/411043501 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd4c006b4d0 0x7fd4c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:18.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.606+0000 7fd4ecc24740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556345341020 0x556345341400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:18.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.606+0000 7fd4ecc24740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556345342330 0x55634537d170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:18.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.606+0000 7fd4ecc24740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55634537d6b0 0x55634537b720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:18.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.606+0000 7fd4ecc24740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563452a5470 con 0x556345341020 2024-02-21T23:04:18.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.606+0000 7fd4ecc24740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55634524eaf0 con 0x556345342330 2024-02-21T23:04:18.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.606+0000 7fd4ecc24740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x5563452a1510 con 0x55634537d6b0 2024-02-21T23:04:18.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.606+0000 7fd4e2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556345341020 0x556345341400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:18.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.606+0000 7fd4e2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556345341020 0x556345341400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52600/0 (socket says 172.21.15.28:52600) 2024-02-21T23:04:18.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.606+0000 7fd4e2a00640 1 -- 172.21.15.28:0/89259867 learned_addr learned my addr 172.21.15.28:0/89259867 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:18.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.607+0000 7fd4dbe00640 1 --2- 172.21.15.28:0/89259867 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556345342330 0x55634537d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:18.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.607+0000 7fd4e2a00640 1 -- 172.21.15.28:0/89259867 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55634537d6b0 msgr2=0x55634537b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:18.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.607+0000 7fd4e2a00640 1 --2- 172.21.15.28:0/89259867 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55634537d6b0 0x55634537b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:18.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.607+0000 7fd4e2a00640 1 -- 172.21.15.28:0/89259867 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556345342330 msgr2=0x55634537d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:18.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.607+0000 7fd4e2a00640 1 --2- 172.21.15.28:0/89259867 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556345342330 0x55634537d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:18.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.607+0000 7fd4e2a00640 1 -- 172.21.15.28:0/89259867 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556345255d90 con 0x556345341020 2024-02-21T23:04:18.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.607+0000 7fd4dbe00640 1 --2- 172.21.15.28:0/89259867 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556345342330 0x55634537d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:04:18.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.607+0000 7fd4e2a00640 1 --2- 172.21.15.28:0/411043501 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd4c006b4d0 0x7fd4c006d990 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd4e4050b90 tx=0x7fd4d4009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:18.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.607+0000 7fd4e2a00640 1 --2- 172.21.15.28:0/89259867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556345341020 0x556345341400 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fd4d4013a50 tx=0x7fd4d4020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e74d2bab59aadfd2 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:18.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.607+0000 7fd4d3400640 1 -- 172.21.15.28:0/89259867 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d4018cd0 con 0x556345341020 2024-02-21T23:04:18.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.607+0000 7fd4d3400640 1 -- 172.21.15.28:0/89259867 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4d4004020 con 0x556345341020 2024-02-21T23:04:18.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.607+0000 7fd4d3400640 1 -- 172.21.15.28:0/89259867 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d40043d0 con 0x556345341020 2024-02-21T23:04:18.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.608+0000 7fd4ecc24740 1 -- 172.21.15.28:0/89259867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556345341020 msgr2=0x556345341400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:18.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.608+0000 7fd4ecc24740 1 --2- 172.21.15.28:0/89259867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556345341020 0x556345341400 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fd4d4013a50 tx=0x7fd4d4020000 comp rx=0 tx=0).stop 2024-02-21T23:04:18.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.608+0000 7fd4ecc24740 1 -- 172.21.15.28:0/89259867 shutdown_connections 2024-02-21T23:04:18.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.608+0000 7fd4ecc24740 1 --2- 172.21.15.28:0/89259867 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55634537d6b0 0x55634537b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:18.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.608+0000 7fd4ecc24740 1 --2- 172.21.15.28:0/89259867 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556345342330 0x55634537d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:18.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.608+0000 7fd4ecc24740 1 --2- 172.21.15.28:0/89259867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556345341020 0x556345341400 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:18.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.608+0000 7fd4ecc24740 1 -- 172.21.15.28:0/89259867 >> 172.21.15.28:0/89259867 conn(0x55634533d970 msgr2=0x556345362380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:18.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.608+0000 7fd4ecc24740 1 -- 172.21.15.28:0/89259867 shutdown_connections 2024-02-21T23:04:18.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.609+0000 7fd4ecc24740 1 -- 172.21.15.28:0/89259867 wait complete. 2024-02-21T23:04:18.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.609+0000 7fd4ecc24740 1 -- start start 2024-02-21T23:04:18.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.610+0000 7fd4ecc24740 1 Processor -- start 2024-02-21T23:04:18.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.611+0000 7fd4ecc24740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556345342330 0x556345374a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:18.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.611+0000 7fd4ecc24740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55634537d6b0 0x556345376230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:18.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.611+0000 7fd4ecc24740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556345362170 0x55634535bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:18.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.611+0000 7fd4ecc24740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x556345244040 con 0x556345342330 2024-02-21T23:04:18.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.611+0000 7fd4ecc24740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x556345362a90 con 0x556345362170 2024-02-21T23:04:18.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.611+0000 7fd4ecc24740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x556345375290 con 0x55634537d6b0 2024-02-21T23:04:18.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.611+0000 7fd4ecc24740 10 client.0 _ll_register_callbacks cb 0x556345261920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:04:18.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.611+0000 7fd4e2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556345342330 0x556345374a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:18.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.611+0000 7fd4e2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556345342330 0x556345374a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52616/0 (socket says 172.21.15.28:52616) 2024-02-21T23:04:18.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.611+0000 7fd4e2a00640 1 -- 172.21.15.28:0/589234220 learned_addr learned my addr 172.21.15.28:0/589234220 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:18.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.611+0000 7fd4dbe00640 1 --2- 172.21.15.28:0/589234220 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55634537d6b0 0x556345376230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:18.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.611+0000 7fd4e3400640 1 --2- 172.21.15.28:0/589234220 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556345362170 0x55634535bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:18.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.612+0000 7fd4e2a00640 1 -- 172.21.15.28:0/589234220 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55634537d6b0 msgr2=0x556345376230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:18.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.612+0000 7fd4e2a00640 1 --2- 172.21.15.28:0/589234220 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55634537d6b0 0x556345376230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:18.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.612+0000 7fd4e2a00640 1 -- 172.21.15.28:0/589234220 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556345362170 msgr2=0x55634535bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:18.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.612+0000 7fd4e2a00640 1 --2- 172.21.15.28:0/589234220 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556345362170 0x55634535bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:18.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.612+0000 7fd4e2a00640 1 -- 172.21.15.28:0/589234220 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556345263aa0 con 0x556345342330 2024-02-21T23:04:18.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.612+0000 7fd4e3400640 1 --2- 172.21.15.28:0/589234220 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556345362170 0x55634535bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:04:18.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.612+0000 7fd4e2a00640 1 --2- 172.21.15.28:0/589234220 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556345342330 0x556345374a00 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fd4d4008010 tx=0x7fd4d40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7246b366b63f0c97 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:18.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.612+0000 7fd4c7e00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:04:18.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.612+0000 7fd4c7e00640 1 -- 172.21.15.28:0/589234220 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d4032070 con 0x556345342330 2024-02-21T23:04:18.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.612+0000 7fd4c7e00640 1 -- 172.21.15.28:0/589234220 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(38..38 src has 1..38) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd4d4003610 con 0x556345342330 2024-02-21T23:04:18.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.612+0000 7fd4ecc24740 1 -- 172.21.15.28:0/589234220 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd4d4004020 con 0x556345342330 2024-02-21T23:04:18.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.613+0000 7fd4c7e00640 1 -- 172.21.15.28:0/589234220 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=18) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd4d4003850 con 0x556345342330 2024-02-21T23:04:18.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.613+0000 7fd4ecc24740 10 client.5430 fetch_fsmap learned FSMap version 18 2024-02-21T23:04:18.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.613+0000 7fd4ecc24740 1 -- 172.21.15.28:0/589234220 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=18,monmap=2+}) v3 -- 0x556344fba2d0 con 0x556345342330 2024-02-21T23:04:18.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.613+0000 7fd4c7e00640 1 -- 172.21.15.28:0/589234220 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4d4004020 con 0x556345342330 2024-02-21T23:04:18.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.613+0000 7fd4c7e00640 1 -- 172.21.15.28:0/589234220 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 18) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fd4d4040080 con 0x556345342330 2024-02-21T23:04:18.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.613+0000 7fd4ecc24740 10 client.5430 fetch_fsmap finished waiting for FSMap version 18 2024-02-21T23:04:18.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.613+0000 7fd4ecc24740 10 client.5430 resolve_mds: resolved name 'a' to daemon mds.a 2024-02-21T23:04:18.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.613+0000 7fd4ecc24740 20 client.5430 populate_metadata read hostname 'smithi028' 2024-02-21T23:04:18.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.613+0000 7fd4ecc24740 1 --2- 172.21.15.28:0/589234220 >> [v2:172.21.15.28:6833/1213702315,v1:172.21.15.28:6835/1213702315] conn(0x556345390ab0 0x556345390eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:18.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.613+0000 7fd4ecc24740 4 client.5430 mds_command: new command op to 4987 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-21T23:04:18.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.613+0000 7fd4ecc24740 1 -- 172.21.15.28:0/589234220 --> [v2:172.21.15.28:6833/1213702315,v1:172.21.15.28:6835/1213702315] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5563452b29b0 con 0x556345390ab0 2024-02-21T23:04:18.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.613+0000 7fd4dbe00640 1 --2- 172.21.15.28:0/589234220 >> [v2:172.21.15.28:6833/1213702315,v1:172.21.15.28:6835/1213702315] conn(0x556345390ab0 0x556345390eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:18.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.614+0000 7fd4dbe00640 1 --2- 172.21.15.28:0/589234220 >> [v2:172.21.15.28:6833/1213702315,v1:172.21.15.28:6835/1213702315] conn(0x556345390ab0 0x556345390eb0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1178bc69d5510837 server_cookie=d5042f15375107b5 in_seq=0 out_seq=0 2024-02-21T23:04:18.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.614+0000 7fd4c7e00640 10 client.5430 ms_handle_connect on v2:172.21.15.28:6833/1213702315 2024-02-21T23:04:18.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.615+0000 7fd4c7e00640 1 -- 172.21.15.28:0/589234220 <== mds.0 v2:172.21.15.28:6833/1213702315 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5563452b29b0 con 0x556345390ab0 2024-02-21T23:04:18.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.615+0000 7fd4c7e00640 10 client.5430 handle_command_reply: tid=0 2024-02-21T23:04:18.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.615+0000 7fd4ecc24740 1 client.5430 shutdown 2024-02-21T23:04:18.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.615+0000 7fd4ecc24740 1 -- 172.21.15.28:0/589234220 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556345342330 msgr2=0x556345374a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:18.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.615+0000 7fd4ecc24740 1 --2- 172.21.15.28:0/589234220 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556345342330 0x556345374a00 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fd4d4008010 tx=0x7fd4d40046c0 comp rx=0 tx=0).stop 2024-02-21T23:04:18.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.615+0000 7fd4ecc24740 1 -- 172.21.15.28:0/589234220 shutdown_connections 2024-02-21T23:04:18.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.615+0000 7fd4ecc24740 1 --2- 172.21.15.28:0/589234220 >> [v2:172.21.15.28:6833/1213702315,v1:172.21.15.28:6835/1213702315] conn(0x556345390ab0 0x556345390eb0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:18.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.615+0000 7fd4ecc24740 1 --2- 172.21.15.28:0/589234220 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55634537d6b0 0x556345376230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:18.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.615+0000 7fd4ecc24740 1 --2- 172.21.15.28:0/589234220 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556345362170 0x55634535bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:18.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.615+0000 7fd4ecc24740 1 --2- 172.21.15.28:0/589234220 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556345342330 0x556345374a00 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:18.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.615+0000 7fd4ecc24740 1 -- 172.21.15.28:0/589234220 >> 172.21.15.28:0/589234220 conn(0x556345341020 msgr2=0x55634537bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:18.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.615+0000 7fd4c7e00640 0 client.5430 ms_handle_reset on v2:172.21.15.28:6833/1213702315 2024-02-21T23:04:18.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.615+0000 7fd4ecc24740 1 -- 172.21.15.28:0/589234220 shutdown_connections 2024-02-21T23:04:18.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.615+0000 7fd4ecc24740 1 -- 172.21.15.28:0/589234220 wait complete. 2024-02-21T23:04:18.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.616+0000 7fd4ecc24740 20 client.5430 trim_cache size 0 max 16384 2024-02-21T23:04:18.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.635+0000 7fd4ecc24740 1 Processor -- start 2024-02-21T23:04:18.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.635+0000 7fd4ecc24740 1 -- start start 2024-02-21T23:04:18.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.636+0000 7fd4ecc24740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556345342330 0x5563453736e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:18.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.636+0000 7fd4ecc24740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556345362170 0x55634533a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:18.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.637+0000 7fd4ecc24740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55634537d6b0 0x55634533ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:18.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.637+0000 7fd4ecc24740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e41f73f0 con 0x556345362170 2024-02-21T23:04:18.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.637+0000 7fd4ecc24740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e41f6820 con 0x556345342330 2024-02-21T23:04:18.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.637+0000 7fd4ecc24740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e41f6e20 con 0x55634537d6b0 2024-02-21T23:04:18.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.637+0000 7fd4dbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556345362170 0x55634533a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:18.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.637+0000 7fd4dbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556345362170 0x55634533a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52628/0 (socket says 172.21.15.28:52628) 2024-02-21T23:04:18.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.637+0000 7fd4dbe00640 1 -- 172.21.15.28:0/2829007479 learned_addr learned my addr 172.21.15.28:0/2829007479 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:18.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.637+0000 7fd4dbe00640 1 -- 172.21.15.28:0/2829007479 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55634537d6b0 msgr2=0x55634533ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:18.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.637+0000 7fd4dbe00640 1 --2- 172.21.15.28:0/2829007479 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55634537d6b0 0x55634533ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:18.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.637+0000 7fd4dbe00640 1 -- 172.21.15.28:0/2829007479 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556345342330 msgr2=0x5563453736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:18.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.637+0000 7fd4dbe00640 1 --2- 172.21.15.28:0/2829007479 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556345342330 0x5563453736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:18.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.637+0000 7fd4dbe00640 1 -- 172.21.15.28:0/2829007479 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556345341770 con 0x556345362170 2024-02-21T23:04:18.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.638+0000 7fd4dbe00640 1 --2- 172.21.15.28:0/2829007479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556345362170 0x55634533a5f0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fd4cc001860 tx=0x7fd4cc0ab040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b28b921b881f12c8 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:18.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.638+0000 7fd4d2a00640 1 -- 172.21.15.28:0/2829007479 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4cc0a8070 con 0x556345362170 2024-02-21T23:04:18.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.638+0000 7fd4d2a00640 1 -- 172.21.15.28:0/2829007479 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4cc030040 con 0x556345362170 2024-02-21T23:04:18.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.638+0000 7fd4d2a00640 1 -- 172.21.15.28:0/2829007479 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4cc0800b0 con 0x556345362170 2024-02-21T23:04:18.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.638+0000 7fd4ecc24740 1 -- 172.21.15.28:0/2829007479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556345362170 msgr2=0x55634533a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:18.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.638+0000 7fd4ecc24740 1 --2- 172.21.15.28:0/2829007479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556345362170 0x55634533a5f0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fd4cc001860 tx=0x7fd4cc0ab040 comp rx=0 tx=0).stop 2024-02-21T23:04:18.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.638+0000 7fd4ecc24740 1 -- 172.21.15.28:0/2829007479 shutdown_connections 2024-02-21T23:04:18.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.638+0000 7fd4ecc24740 1 --2- 172.21.15.28:0/2829007479 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55634537d6b0 0x55634533ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:18.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.638+0000 7fd4ecc24740 1 --2- 172.21.15.28:0/2829007479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556345362170 0x55634533a5f0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:18.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.638+0000 7fd4ecc24740 1 --2- 172.21.15.28:0/2829007479 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556345342330 0x5563453736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:18.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.638+0000 7fd4ecc24740 1 -- 172.21.15.28:0/2829007479 >> 172.21.15.28:0/2829007479 conn(0x556345341020 msgr2=0x556345375420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:18.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.639+0000 7fd4ecc24740 1 -- 172.21.15.28:0/2829007479 shutdown_connections 2024-02-21T23:04:18.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.639+0000 7fd4ecc24740 1 -- 172.21.15.28:0/2829007479 wait complete. 2024-02-21T23:04:18.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.639+0000 7fd4ecc24740 1 -- start start 2024-02-21T23:04:18.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.639+0000 7fd4ecc24740 1 Processor -- start 2024-02-21T23:04:18.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.640+0000 7fd4ecc24740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x556345342330 0x556345362970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:18.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.640+0000 7fd4ecc24740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556345362170 0x5563453e9870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:18.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.640+0000 7fd4ecc24740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55634537d6b0 0x5563453e9db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:18.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.640+0000 7fd4ecc24740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55634535ca10 con 0x55634537d6b0 2024-02-21T23:04:18.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.640+0000 7fd4ecc24740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55634538d750 con 0x556345362170 2024-02-21T23:04:18.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.640+0000 7fd4ecc24740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x556345374090 con 0x556345342330 2024-02-21T23:04:18.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.640+0000 7fd4ecc24740 10 client.0 _ll_register_callbacks cb 0x7fd4d40042a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:04:18.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.641+0000 7fd4e3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55634537d6b0 0x5563453e9db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:18.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.641+0000 7fd4e3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55634537d6b0 0x5563453e9db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52640/0 (socket says 172.21.15.28:52640) 2024-02-21T23:04:18.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.641+0000 7fd4e3400640 1 -- 172.21.15.28:0/2326753843 learned_addr learned my addr 172.21.15.28:0/2326753843 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:18.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.641+0000 7fd4e2a00640 1 --2- 172.21.15.28:0/2326753843 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x556345342330 0x556345362970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:18.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.641+0000 7fd4dbe00640 1 --2- 172.21.15.28:0/2326753843 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556345362170 0x5563453e9870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:18.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.641+0000 7fd4e3400640 1 -- 172.21.15.28:0/2326753843 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x556345342330 msgr2=0x556345362970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:18.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.641+0000 7fd4e3400640 1 --2- 172.21.15.28:0/2326753843 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x556345342330 0x556345362970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:18.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.641+0000 7fd4e3400640 1 -- 172.21.15.28:0/2326753843 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556345362170 msgr2=0x5563453e9870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:18.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.641+0000 7fd4e3400640 1 --2- 172.21.15.28:0/2326753843 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556345362170 0x5563453e9870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:18.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.641+0000 7fd4e3400640 1 -- 172.21.15.28:0/2326753843 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5563450846b0 con 0x55634537d6b0 2024-02-21T23:04:18.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.641+0000 7fd4dbe00640 1 --2- 172.21.15.28:0/2326753843 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556345362170 0x5563453e9870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:04:18.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.642+0000 7fd4e3400640 1 --2- 172.21.15.28:0/2326753843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55634537d6b0 0x5563453e9db0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fd4e406e370 tx=0x7fd4dc0021f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23ebdeee95432496 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:18.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.642+0000 7fd4c7e00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:04:18.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.642+0000 7fd4c7e00640 1 -- 172.21.15.28:0/2326753843 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4dc018060 con 0x55634537d6b0 2024-02-21T23:04:18.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.642+0000 7fd4c7e00640 1 -- 172.21.15.28:0/2326753843 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(38..38 src has 1..38) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd4dc003b70 con 0x55634537d6b0 2024-02-21T23:04:18.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.642+0000 7fd4ecc24740 1 -- 172.21.15.28:0/2326753843 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd4cc030040 con 0x55634537d6b0 2024-02-21T23:04:18.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.642+0000 7fd4c7e00640 1 -- 172.21.15.28:0/2326753843 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=18) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd4dc01e050 con 0x55634537d6b0 2024-02-21T23:04:18.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.642+0000 7fd4ecc24740 10 client.5436 fetch_fsmap learned FSMap version 18 2024-02-21T23:04:18.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.642+0000 7fd4ecc24740 1 -- 172.21.15.28:0/2326753843 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=18,monmap=2+}) v3 -- 0x556345363120 con 0x55634537d6b0 2024-02-21T23:04:18.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.642+0000 7fd4c7e00640 1 -- 172.21.15.28:0/2326753843 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4cc030040 con 0x55634537d6b0 2024-02-21T23:04:18.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.642+0000 7fd4c7e00640 1 -- 172.21.15.28:0/2326753843 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 18) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fd4dc021020 con 0x55634537d6b0 2024-02-21T23:04:18.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.642+0000 7fd4ecc24740 10 client.5436 fetch_fsmap finished waiting for FSMap version 18 2024-02-21T23:04:18.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.642+0000 7fd4ecc24740 10 client.5436 resolve_mds: resolved name 'a' to daemon mds.a 2024-02-21T23:04:18.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.642+0000 7fd4ecc24740 20 client.5436 populate_metadata read hostname 'smithi028' 2024-02-21T23:04:18.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.642+0000 7fd4ecc24740 1 --2- 172.21.15.28:0/2326753843 >> [v2:172.21.15.28:6833/1213702315,v1:172.21.15.28:6835/1213702315] conn(0x55634535aad0 0x55634535aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:18.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.643+0000 7fd4ecc24740 4 client.5436 mds_command: new command op to 4987 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-21T23:04:18.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.643+0000 7fd4ecc24740 1 -- 172.21.15.28:0/2326753843 --> [v2:172.21.15.28:6833/1213702315,v1:172.21.15.28:6835/1213702315] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5563452acd00 con 0x55634535aad0 2024-02-21T23:04:18.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.643+0000 7fd4e2a00640 1 --2- 172.21.15.28:0/2326753843 >> [v2:172.21.15.28:6833/1213702315,v1:172.21.15.28:6835/1213702315] conn(0x55634535aad0 0x55634535aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:18.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.643+0000 7fd4e2a00640 1 --2- 172.21.15.28:0/2326753843 >> [v2:172.21.15.28:6833/1213702315,v1:172.21.15.28:6835/1213702315] conn(0x55634535aad0 0x55634535aeb0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d4e0a1eefbcf86ef server_cookie=9d180a68b6492a09 in_seq=0 out_seq=0 2024-02-21T23:04:18.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.643+0000 7fd4c7e00640 10 client.5436 ms_handle_connect on v2:172.21.15.28:6833/1213702315 2024-02-21T23:04:18.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.644+0000 7fd4c7e00640 1 -- 172.21.15.28:0/2326753843 <== mds.0 v2:172.21.15.28:6833/1213702315 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x5563452acd00 con 0x55634535aad0 2024-02-21T23:04:18.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.644+0000 7fd4c7e00640 10 client.5436 handle_command_reply: tid=0 2024-02-21T23:04:18.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.644+0000 7fd4ecc24740 1 client.5436 shutdown 2024-02-21T23:04:18.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.644+0000 7fd4ecc24740 1 -- 172.21.15.28:0/2326753843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55634537d6b0 msgr2=0x5563453e9db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:18.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.644+0000 7fd4ecc24740 1 --2- 172.21.15.28:0/2326753843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55634537d6b0 0x5563453e9db0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fd4e406e370 tx=0x7fd4dc0021f0 comp rx=0 tx=0).stop 2024-02-21T23:04:18.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.644+0000 7fd4ecc24740 1 -- 172.21.15.28:0/2326753843 shutdown_connections 2024-02-21T23:04:18.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.644+0000 7fd4ecc24740 1 --2- 172.21.15.28:0/2326753843 >> [v2:172.21.15.28:6833/1213702315,v1:172.21.15.28:6835/1213702315] conn(0x55634535aad0 0x55634535aeb0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:18.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.644+0000 7fd4ecc24740 1 --2- 172.21.15.28:0/2326753843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55634537d6b0 0x5563453e9db0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:18.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.644+0000 7fd4ecc24740 1 --2- 172.21.15.28:0/2326753843 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556345362170 0x5563453e9870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:18.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.644+0000 7fd4ecc24740 1 --2- 172.21.15.28:0/2326753843 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x556345342330 0x556345362970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:18.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.644+0000 7fd4c7e00640 0 client.5436 ms_handle_reset on v2:172.21.15.28:6833/1213702315 2024-02-21T23:04:18.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.644+0000 7fd4ecc24740 1 -- 172.21.15.28:0/2326753843 >> 172.21.15.28:0/2326753843 conn(0x556345341020 msgr2=0x55634535bcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:18.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.644+0000 7fd4ecc24740 1 -- 172.21.15.28:0/2326753843 shutdown_connections 2024-02-21T23:04:18.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.644+0000 7fd4ecc24740 1 -- 172.21.15.28:0/2326753843 wait complete. 2024-02-21T23:04:18.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.644+0000 7fd4ecc24740 20 client.5436 trim_cache size 0 max 16384 2024-02-21T23:04:18.637 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-21T23:04:18.637 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac", 2024-02-21T23:04:18.637 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-02-21T23:04:18.637 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4987, 2024-02-21T23:04:18.637 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-02-21T23:04:18.637 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-02-21T23:04:18.637 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-02-21T23:04:18.637 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 9.0627791290000008, 2024-02-21T23:04:18.637 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 18, 2024-02-21T23:04:18.637 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 38, 2024-02-21T23:04:18.637 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 38, 2024-02-21T23:04:18.637 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 18.069445565999999 2024-02-21T23:04:18.638 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-21T23:04:18.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.645+0000 7fd4e3e00640 1 -- 172.21.15.28:0/411043501 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd4c006b4d0 msgr2=0x7fd4c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:18.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.645+0000 7fd4e3e00640 1 --2- 172.21.15.28:0/411043501 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd4c006b4d0 0x7fd4c006d990 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd4e4050b90 tx=0x7fd4d4009190 comp rx=0 tx=0).stop 2024-02-21T23:04:18.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.645+0000 7fd4e3e00640 1 -- 172.21.15.28:0/411043501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4e4150140 msgr2=0x7fd4e40aad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:18.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.645+0000 7fd4e3e00640 1 --2- 172.21.15.28:0/411043501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4e4150140 0x7fd4e40aad40 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fd4cc00c200 tx=0x7fd4e40c2f70 comp rx=0 tx=0).stop 2024-02-21T23:04:18.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.645+0000 7fd4e3e00640 1 -- 172.21.15.28:0/411043501 shutdown_connections 2024-02-21T23:04:18.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.645+0000 7fd4e3e00640 1 --2- 172.21.15.28:0/411043501 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd4c006b4d0 0x7fd4c006d990 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:18.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.645+0000 7fd4e3e00640 1 --2- 172.21.15.28:0/411043501 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd4e41de5d0 0x7fd4e40ab280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:18.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.645+0000 7fd4e3e00640 1 --2- 172.21.15.28:0/411043501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4e4150140 0x7fd4e40aad40 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:18.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.645+0000 7fd4e3e00640 1 --2- 172.21.15.28:0/411043501 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd4c8004800 0x7fd4e40a6260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:18.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.645+0000 7fd4e3e00640 1 -- 172.21.15.28:0/411043501 >> 172.21.15.28:0/411043501 conn(0x7fd4e40a8320 msgr2=0x7fd4e4157920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:18.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.645+0000 7fd4e3e00640 1 -- 172.21.15.28:0/411043501 shutdown_connections 2024-02-21T23:04:18.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:18.645+0000 7fd4e3e00640 1 -- 172.21.15.28:0/411043501 wait complete. 2024-02-21T23:04:18.652 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 2, "epoch": 18, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-02-21T23:04:09.036302+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 18, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4987": { "addr": "172.21.15.28:6835/1213702315", "addrs": { "addrvec": [ { "addr": "172.21.15.28:6833", "nonce": 1213702315, "type": "v2" }, { "addr": "172.21.15.28:6835", "nonce": 1213702315, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4987, "incarnation": 14, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 4 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 5, "modified": "2024-02-21T23:04:12.123968+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4987 } } } ], "standbys": [ { "addr": "172.21.15.28:6834/2827878900", "addrs": { "addrvec": [ { "addr": "172.21.15.28:6832", "nonce": 2827878900, "type": "v2" }, { "addr": "172.21.15.28:6834", "nonce": 2827878900, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 13, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4778, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.81:6836/3612644344", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6834", "nonce": 3612644344, "type": "v2" }, { "addr": "172.21.15.81:6836", "nonce": 3612644344, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 12, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4978, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.81:6837/3623937361", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6835", "nonce": 3623937361, "type": "v2" }, { "addr": "172.21.15.81:6837", "nonce": 3623937361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 13, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4981, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-02-21T23:04:18.653 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': 2, 'name': 'c', 'state': 'up:active'}, {'join_fscid': -1, 'name': 'd'}] 2024-02-21T23:04:18.654 INFO:tasks.cephfs.test_failover:current = [{'gid': 4987, 'name': 'a', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.28:6835/1213702315', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6833', 'nonce': 1213702315}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 1213702315}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 4981, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.81:6837/3623937361', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6835', 'nonce': 3623937361}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3623937361}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 13}, {'gid': 4778, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.28:6834/2827878900', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 2827878900}, {'type': 'v1', 'addr': '172.21.15.28:6834', 'nonce': 2827878900}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 13}, {'gid': 4978, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.81:6836/3612644344', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6834', 'nonce': 3612644344}, {'type': 'v1', 'addr': '172.21.15.81:6836', 'nonce': 3612644344}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 12}] 2024-02-21T23:04:18.654 DEBUG:tasks.cephfs.test_failover:2 != -1 2024-02-21T23:04:18.654 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-02-21T23:04:20.694 INFO:tasks.ceph.mds.a.smithi028.stderr: -378> 2024-02-21T23:04:12.357+0000 7f238f600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-21T23:04:20.745 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-02-21T23:04:23.655 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:04:23.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:23.886+0000 7f915d400640 1 Processor -- start 2024-02-21T23:04:23.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:23.886+0000 7f915d400640 1 -- start start 2024-02-21T23:04:23.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:23.888+0000 7f915d400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f91580c6740 0x7f9158154fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:23.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:23.888+0000 7f915d400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f91581554f0 0x7f91580c8a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:23.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:23.888+0000 7f915d400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9158155a00 con 0x7f915814ef60 2024-02-21T23:04:23.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:23.888+0000 7f915d400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9158155ce0 con 0x7f91580c6740 2024-02-21T23:04:23.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:23.888+0000 7f915d400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91581586d0 con 0x7f91581554f0 2024-02-21T23:04:23.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:23.888+0000 7f9157400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f91580c6740 0x7f9158154fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:23.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:23.888+0000 7f9157400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f91580c6740 0x7f9158154fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52670/0 (socket says 172.21.15.28:52670) 2024-02-21T23:04:23.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:23.888+0000 7f9157400640 1 -- 172.21.15.28:0/1574055906 learned_addr learned my addr 172.21.15.28:0/1574055906 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:23.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:23.888+0000 7f9157e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f915814ef60 0x7f9158158ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60440/0 (socket says 172.21.15.28:60440) 2024-02-21T23:04:23.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:23.889+0000 7f915ca00640 1 --2- 172.21.15.28:0/1574055906 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f91581554f0 0x7f91580c8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:23.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:23.889+0000 7f9157400640 1 -- 172.21.15.28:0/1574055906 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f91581554f0 msgr2=0x7f91580c8a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:23.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:23.889+0000 7f9157400640 1 --2- 172.21.15.28:0/1574055906 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f91581554f0 0x7f91580c8a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:23.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:23.889+0000 7f9157400640 1 -- 172.21.15.28:0/1574055906 >> v1:172.21.15.28:6789/0 conn(0x7f915814ef60 legacy=0x7f9158158ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:23.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:23.889+0000 7f9157400640 1 -- 172.21.15.28:0/1574055906 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91581589b0 con 0x7f91580c6740 2024-02-21T23:04:23.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:23.890+0000 7f9157400640 1 --2- 172.21.15.28:0/1574055906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91580c6740 0x7f9158154fb0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f9144009870 tx=0x7f914402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dac808b59a4b1784 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:23.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:23.890+0000 7f9156a00640 1 -- 172.21.15.28:0/1574055906 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9144034070 con 0x7f91580c6740 2024-02-21T23:04:23.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:23.890+0000 7f9156a00640 1 -- 172.21.15.28:0/1574055906 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9144030840 con 0x7f91580c6740 2024-02-21T23:04:23.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:23.890+0000 7f9156a00640 1 -- 172.21.15.28:0/1574055906 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9144030c10 con 0x7f91580c6740 2024-02-21T23:04:23.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:23.891+0000 7f915d400640 1 -- 172.21.15.28:0/1574055906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91580c6740 msgr2=0x7f9158154fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:23.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:23.891+0000 7f915d400640 1 --2- 172.21.15.28:0/1574055906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91580c6740 0x7f9158154fb0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f9144009870 tx=0x7f914402b980 comp rx=0 tx=0).stop 2024-02-21T23:04:23.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:23.891+0000 7f915d400640 1 -- 172.21.15.28:0/1574055906 shutdown_connections 2024-02-21T23:04:23.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:23.891+0000 7f915d400640 1 --2- 172.21.15.28:0/1574055906 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f91581554f0 0x7f91580c8a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:23.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:23.891+0000 7f915d400640 1 --2- 172.21.15.28:0/1574055906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91580c6740 0x7f9158154fb0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:23.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:23.891+0000 7f915d400640 1 -- 172.21.15.28:0/1574055906 >> 172.21.15.28:0/1574055906 conn(0x7f915806b260 msgr2=0x7f915806b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:23.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:23.891+0000 7f915d400640 1 -- 172.21.15.28:0/1574055906 shutdown_connections 2024-02-21T23:04:23.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:23.891+0000 7f915d400640 1 -- 172.21.15.28:0/1574055906 wait complete. 2024-02-21T23:04:23.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:23.894+0000 7f915d400640 1 Processor -- start 2024-02-21T23:04:23.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:23.894+0000 7f915d400640 1 -- start start 2024-02-21T23:04:23.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:23.895+0000 7f915d400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f91580c6740 0x7f915814d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:23.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:23.896+0000 7f915d400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f915814ef60 0x7f91581ea900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:23.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:23.896+0000 7f915d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91581554f0 0x7f91581eae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:23.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:23.896+0000 7f915d400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91580c2b80 con 0x7f91581554f0 2024-02-21T23:04:23.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:23.896+0000 7f915d400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91580c2a00 con 0x7f91580c6740 2024-02-21T23:04:23.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:23.896+0000 7f915d400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91580c6ba0 con 0x7f915814ef60 2024-02-21T23:04:23.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:23.896+0000 7f915ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91581554f0 0x7f91581eae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:23.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:23.896+0000 7f915ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91581554f0 0x7f91581eae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52678/0 (socket says 172.21.15.28:52678) 2024-02-21T23:04:23.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:23.896+0000 7f915ca00640 1 -- 172.21.15.28:0/3210304008 learned_addr learned my addr 172.21.15.28:0/3210304008 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:23.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:23.896+0000 7f9157e00640 1 --2- 172.21.15.28:0/3210304008 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f91580c6740 0x7f915814d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:23.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:23.896+0000 7f9157400640 1 --2- 172.21.15.28:0/3210304008 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f915814ef60 0x7f91581ea900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:23.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:23.896+0000 7f915ca00640 1 -- 172.21.15.28:0/3210304008 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f915814ef60 msgr2=0x7f91581ea900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:23.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:23.896+0000 7f915ca00640 1 --2- 172.21.15.28:0/3210304008 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f915814ef60 0x7f91581ea900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:23.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:23.896+0000 7f915ca00640 1 -- 172.21.15.28:0/3210304008 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f91580c6740 msgr2=0x7f915814d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:23.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:23.896+0000 7f915ca00640 1 --2- 172.21.15.28:0/3210304008 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f91580c6740 0x7f915814d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:23.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:23.896+0000 7f915ca00640 1 -- 172.21.15.28:0/3210304008 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f915814e380 con 0x7f91581554f0 2024-02-21T23:04:23.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:23.897+0000 7f9157e00640 1 --2- 172.21.15.28:0/3210304008 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f91580c6740 0x7f915814d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:04:23.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:23.897+0000 7f915ca00640 1 --2- 172.21.15.28:0/3210304008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91581554f0 0x7f91581eae40 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f9150007e90 tx=0x7f915000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:23.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:23.897+0000 7f914fe00640 1 -- 172.21.15.28:0/3210304008 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9150014420 con 0x7f91581554f0 2024-02-21T23:04:23.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:23.897+0000 7f914fe00640 1 -- 172.21.15.28:0/3210304008 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9150019070 con 0x7f91581554f0 2024-02-21T23:04:23.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:23.897+0000 7f914fe00640 1 -- 172.21.15.28:0/3210304008 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9150014800 con 0x7f91581554f0 2024-02-21T23:04:23.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:23.897+0000 7f915d400640 1 -- 172.21.15.28:0/3210304008 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9158153a10 con 0x7f91581554f0 2024-02-21T23:04:23.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:23.897+0000 7f915d400640 1 -- 172.21.15.28:0/3210304008 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f915814e660 con 0x7f91581554f0 2024-02-21T23:04:23.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:23.899+0000 7f915d400640 1 -- 172.21.15.28:0/3210304008 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91580c7ef0 con 0x7f91581554f0 2024-02-21T23:04:23.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:23.900+0000 7f914fe00640 1 -- 172.21.15.28:0/3210304008 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f91500149a0 con 0x7f91581554f0 2024-02-21T23:04:23.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:23.901+0000 7f914fe00640 1 --2- 172.21.15.28:0/3210304008 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f913806b4d0 0x7f913806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:23.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:23.901+0000 7f914fe00640 1 -- 172.21.15.28:0/3210304008 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6216+0+0 (secure 0 0 0) 0x7f915008ac40 con 0x7f91581554f0 2024-02-21T23:04:23.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:23.901+0000 7f9157e00640 1 --2- 172.21.15.28:0/3210304008 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f913806b4d0 0x7f913806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:23.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:23.902+0000 7f9157e00640 1 --2- 172.21.15.28:0/3210304008 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f913806b4d0 0x7f913806d990 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f9148007fd0 tx=0x7f9148007d60 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:23.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:23.906+0000 7f914fe00640 1 -- 172.21.15.28:0/3210304008 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f91580c7ef0 con 0x7f91581554f0 2024-02-21T23:04:24.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.149+0000 7f915d400640 1 -- 172.21.15.28:0/3210304008 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f91580c6520 con 0x7f91581554f0 2024-02-21T23:04:24.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.150+0000 7f914fe00640 1 -- 172.21.15.28:0/3210304008 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+4932 (secure 0 0 0) 0x7f91500598b0 con 0x7f91581554f0 2024-02-21T23:04:24.140 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:04:24.140 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":23,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4978,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/3612644344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":3612644344},{"type":"v1","addr":"172.21.15.81:6836","nonce":3612644344}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4981,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3623937361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":3623937361},{"type":"v1","addr":"172.21.15.81:6837","nonce":3623937361}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5442,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/33413357","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":33413357},{"type":"v1","addr":"172.21.15.28:6835","nonce":33413357}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:04:09.036302+0000","modified":"2024-02-21T23:04:22.712438+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4778},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4778":{"gid":4778,"name":"c","rank":0,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.28:6834/2827878900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2827878900},{"type":"v1","addr":"172.21.15.28:6834","nonce":2827878900}]},"join_fscid":2,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-02-21T23:04:24.140 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 23 2024-02-21T23:04:24.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.153+0000 7f915d400640 1 -- 172.21.15.28:0/3210304008 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f913806b4d0 msgr2=0x7f913806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:24.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.153+0000 7f915d400640 1 --2- 172.21.15.28:0/3210304008 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f913806b4d0 0x7f913806d990 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f9148007fd0 tx=0x7f9148007d60 comp rx=0 tx=0).stop 2024-02-21T23:04:24.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.153+0000 7f915d400640 1 -- 172.21.15.28:0/3210304008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91581554f0 msgr2=0x7f91581eae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:24.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.153+0000 7f915d400640 1 --2- 172.21.15.28:0/3210304008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91581554f0 0x7f91581eae40 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f9150007e90 tx=0x7f915000e4b0 comp rx=0 tx=0).stop 2024-02-21T23:04:24.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.153+0000 7f915d400640 1 -- 172.21.15.28:0/3210304008 shutdown_connections 2024-02-21T23:04:24.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.153+0000 7f915d400640 1 --2- 172.21.15.28:0/3210304008 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f913806b4d0 0x7f913806d990 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:24.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.153+0000 7f915d400640 1 --2- 172.21.15.28:0/3210304008 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f915814ef60 0x7f91581ea900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:24.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.153+0000 7f915d400640 1 --2- 172.21.15.28:0/3210304008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91581554f0 0x7f91581eae40 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:24.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.153+0000 7f915d400640 1 --2- 172.21.15.28:0/3210304008 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f91580c6740 0x7f915814d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:24.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.153+0000 7f915d400640 1 -- 172.21.15.28:0/3210304008 >> 172.21.15.28:0/3210304008 conn(0x7f915806b260 msgr2=0x7f915806b640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:24.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.154+0000 7f915d400640 1 -- 172.21.15.28:0/3210304008 shutdown_connections 2024-02-21T23:04:24.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.154+0000 7f915d400640 1 -- 172.21.15.28:0/3210304008 wait complete. 2024-02-21T23:04:24.565 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:04:09.036302+0000', 'modified': '2024-02-21T23:04:22.712438+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 39, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4778}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4778': {'gid': 4778, 'name': 'c', 'rank': 0, 'incarnation': 20, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.28:6834/2827878900', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 2827878900}, {'type': 'v1', 'addr': '172.21.15.28:6834', 'nonce': 2827878900}]}, 'join_fscid': 2, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:04:24.565 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-21T23:04:24.566 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-02-21T23:04:24.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.816+0000 7fef02c00640 1 Processor -- start 2024-02-21T23:04:24.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.817+0000 7fef02c00640 1 -- start start 2024-02-21T23:04:24.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.819+0000 7fef02c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7feefc14eb90 0x7feefc0c46f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:24.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.819+0000 7fef02c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feefc0c11f0 0x7feefc0c2a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:24.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.819+0000 7fef02c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feefc0c1cb0 con 0x7feefc06d110 2024-02-21T23:04:24.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.819+0000 7fef02c00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feefc14ef70 con 0x7feefc14eb90 2024-02-21T23:04:24.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.819+0000 7fef02c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feefc0c1f90 con 0x7feefc0c11f0 2024-02-21T23:04:24.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.819+0000 7fef00c00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7feefc06d110 0x7feefc072350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60444/0 (socket says 172.21.15.28:60444) 2024-02-21T23:04:24.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.819+0000 7fef00c00640 1 -- 172.21.15.28:0/2773669339 learned_addr learned my addr 172.21.15.28:0/2773669339 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:04:24.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.820+0000 7feefbe00640 1 --2- 172.21.15.28:0/2773669339 >> v2:172.21.15.81:3300/0 conn(0x7feefc14eb90 0x7feefc0c46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:24.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.820+0000 7feefb400640 1 --2- 172.21.15.28:0/2773669339 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feefc0c11f0 0x7feefc0c2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:24.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.820+0000 7feefaa00640 1 -- 172.21.15.28:0/2773669339 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2726867608 0 0) 0x7feefc0c1cb0 con 0x7feefc06d110 2024-02-21T23:04:24.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.820+0000 7feefaa00640 1 -- 172.21.15.28:0/2773669339 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feee0003520 con 0x7feefc06d110 2024-02-21T23:04:24.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.820+0000 7feefaa00640 1 -- 172.21.15.28:0/2773669339 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3053912496 0 0) 0x7feee0003520 con 0x7feefc06d110 2024-02-21T23:04:24.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.820+0000 7feefaa00640 1 -- 172.21.15.28:0/2773669339 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feefc0c11f0 msgr2=0x7feefc0c2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:24.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.820+0000 7feefaa00640 1 --2- 172.21.15.28:0/2773669339 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feefc0c11f0 0x7feefc0c2a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:24.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.820+0000 7feefaa00640 1 -- 172.21.15.28:0/2773669339 >> v2:172.21.15.81:3300/0 conn(0x7feefc14eb90 msgr2=0x7feefc0c46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:24.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.820+0000 7feefaa00640 1 --2- 172.21.15.28:0/2773669339 >> v2:172.21.15.81:3300/0 conn(0x7feefc14eb90 0x7feefc0c46f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:24.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.820+0000 7feefaa00640 1 -- 172.21.15.28:0/2773669339 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feefc0c2270 con 0x7feefc06d110 2024-02-21T23:04:24.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.820+0000 7feefaa00640 1 -- 172.21.15.28:0/2773669339 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7feef40019f0 con 0x7feefc06d110 2024-02-21T23:04:24.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.820+0000 7feefbe00640 1 --2- 172.21.15.28:0/2773669339 >> v2:172.21.15.81:3300/0 conn(0x7feefc14eb90 0x7feefc0c46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-21T23:04:24.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.821+0000 7feefaa00640 1 -- 172.21.15.28:0/2773669339 >> v1:172.21.15.28:6789/0 conn(0x7feefc06d110 legacy=0x7feefc072350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:24.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.821+0000 7feefaa00640 1 --2- 172.21.15.28:0/2773669339 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feee00039b0 0x7feee0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:24.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.821+0000 7feefaa00640 1 --2- 172.21.15.28:0/2773669339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feee0004800 0x7feee0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:24.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.821+0000 7feefaa00640 1 --2- 172.21.15.28:0/2773669339 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feefc0c11f0 0x7feee00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:24.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.821+0000 7feefaa00640 1 -- 172.21.15.28:0/2773669339 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feefc14ef70 con 0x7feee0004800 2024-02-21T23:04:24.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.821+0000 7feefaa00640 1 -- 172.21.15.28:0/2773669339 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feefc0c1f90 con 0x7feee00039b0 2024-02-21T23:04:24.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.821+0000 7feefaa00640 1 -- 172.21.15.28:0/2773669339 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feee000dc20 con 0x7feefc0c11f0 2024-02-21T23:04:24.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.821+0000 7feefb400640 1 --2- 172.21.15.28:0/2773669339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feee0004800 0x7feee0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:24.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.821+0000 7feefbe00640 1 --2- 172.21.15.28:0/2773669339 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feee00039b0 0x7feee0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:24.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.821+0000 7fef00c00640 1 --2- 172.21.15.28:0/2773669339 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feefc0c11f0 0x7feee00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:24.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.821+0000 7feefb400640 1 -- 172.21.15.28:0/2773669339 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feefc0c11f0 msgr2=0x7feee00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:24.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.821+0000 7feefb400640 1 --2- 172.21.15.28:0/2773669339 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feefc0c11f0 0x7feee00051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:24.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.821+0000 7feefb400640 1 -- 172.21.15.28:0/2773669339 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feee00039b0 msgr2=0x7feee0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:24.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.821+0000 7feefb400640 1 --2- 172.21.15.28:0/2773669339 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feee00039b0 0x7feee0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:24.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.821+0000 7feefb400640 1 -- 172.21.15.28:0/2773669339 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feee0003520 con 0x7feee0004800 2024-02-21T23:04:24.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.823+0000 7feefb400640 1 --2- 172.21.15.28:0/2773669339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feee0004800 0x7feee0004c80 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7feee800aa70 tx=0x7feee0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6837f676c378269 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:24.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.823+0000 7feefaa00640 1 -- 172.21.15.28:0/2773669339 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feee8039070 con 0x7feee0004800 2024-02-21T23:04:24.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.823+0000 7feefaa00640 1 -- 172.21.15.28:0/2773669339 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feee802cbb0 con 0x7feee0004800 2024-02-21T23:04:24.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.823+0000 7feefaa00640 1 -- 172.21.15.28:0/2773669339 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feee800b040 con 0x7feee0004800 2024-02-21T23:04:24.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.823+0000 7fef02c00640 1 -- 172.21.15.28:0/2773669339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feee0004800 msgr2=0x7feee0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:24.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.823+0000 7fef02c00640 1 --2- 172.21.15.28:0/2773669339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feee0004800 0x7feee0004c80 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7feee800aa70 tx=0x7feee0005830 comp rx=0 tx=0).stop 2024-02-21T23:04:24.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.823+0000 7fef00c00640 1 -- 172.21.15.28:0/2773669339 reap_dead start 2024-02-21T23:04:24.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.824+0000 7fef02c00640 1 -- 172.21.15.28:0/2773669339 shutdown_connections 2024-02-21T23:04:24.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.824+0000 7fef02c00640 1 -- 172.21.15.28:0/2773669339 >> 172.21.15.28:0/2773669339 conn(0x7feefc06b820 msgr2=0x7feefc0c67f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:24.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.824+0000 7fef02c00640 1 -- 172.21.15.28:0/2773669339 shutdown_connections 2024-02-21T23:04:24.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.824+0000 7fef02c00640 1 -- 172.21.15.28:0/2773669339 wait complete. 2024-02-21T23:04:24.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.825+0000 7fef02c00640 1 Processor -- start 2024-02-21T23:04:24.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.826+0000 7fef02c00640 1 -- start start 2024-02-21T23:04:24.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.826+0000 7fef02c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feee0004800 0x7feefc14cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:24.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.826+0000 7fef02c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feefc14b250 0x7feefc1e3e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:24.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.827+0000 7fef02c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feefc1e7510 0x7feefc1e4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:24.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.827+0000 7fef02c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feefc0c40b0 con 0x7feefc14b250 2024-02-21T23:04:24.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.827+0000 7fef02c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feefc0c3c30 con 0x7feee0004800 2024-02-21T23:04:24.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.827+0000 7fef02c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feefc151500 con 0x7feefc1e7510 2024-02-21T23:04:24.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.827+0000 7feefb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feefc14b250 0x7feefc1e3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:24.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.827+0000 7feefb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feefc14b250 0x7feefc1e3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52708/0 (socket says 172.21.15.28:52708) 2024-02-21T23:04:24.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.827+0000 7feefb400640 1 -- 172.21.15.28:0/2150762038 learned_addr learned my addr 172.21.15.28:0/2150762038 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:24.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.827+0000 7feefbe00640 1 --2- 172.21.15.28:0/2150762038 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feee0004800 0x7feefc14cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:24.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.827+0000 7fef00c00640 1 --2- 172.21.15.28:0/2150762038 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feefc1e7510 0x7feefc1e4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:24.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.827+0000 7feefb400640 1 -- 172.21.15.28:0/2150762038 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feefc1e7510 msgr2=0x7feefc1e4360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:24.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.827+0000 7feefb400640 1 --2- 172.21.15.28:0/2150762038 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feefc1e7510 0x7feefc1e4360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:24.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.827+0000 7feefb400640 1 -- 172.21.15.28:0/2150762038 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feee0004800 msgr2=0x7feefc14cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:24.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.827+0000 7feefb400640 1 --2- 172.21.15.28:0/2150762038 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feee0004800 0x7feefc14cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:24.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.827+0000 7feefb400640 1 -- 172.21.15.28:0/2150762038 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feefc153420 con 0x7feefc14b250 2024-02-21T23:04:24.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.827+0000 7fef00c00640 1 --2- 172.21.15.28:0/2150762038 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feefc1e7510 0x7feefc1e4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:04:24.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.828+0000 7feefb400640 1 --2- 172.21.15.28:0/2150762038 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feefc14b250 0x7feefc1e3e20 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7feee8037040 tx=0x7feee8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:24.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.828+0000 7feef3e00640 1 -- 172.21.15.28:0/2150762038 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feee8039050 con 0x7feefc14b250 2024-02-21T23:04:24.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.828+0000 7feef3e00640 1 -- 172.21.15.28:0/2150762038 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feee8045070 con 0x7feefc14b250 2024-02-21T23:04:24.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.828+0000 7feef3e00640 1 -- 172.21.15.28:0/2150762038 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feee800b040 con 0x7feefc14b250 2024-02-21T23:04:24.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.828+0000 7fef02c00640 1 -- 172.21.15.28:0/2150762038 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feefc152e60 con 0x7feefc14b250 2024-02-21T23:04:24.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.828+0000 7fef02c00640 1 -- 172.21.15.28:0/2150762038 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feefc1ea100 con 0x7feefc14b250 2024-02-21T23:04:24.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.832+0000 7feef3e00640 1 -- 172.21.15.28:0/2150762038 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feee8003a60 con 0x7feefc14b250 2024-02-21T23:04:24.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.832+0000 7fef05bfc740 1 Processor -- start 2024-02-21T23:04:24.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.832+0000 7fef05bfc740 1 -- start start 2024-02-21T23:04:24.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.833+0000 7feef3e00640 1 --2- 172.21.15.28:0/2150762038 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7feedc06b500 0x7feedc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:24.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.833+0000 7feef3e00640 1 -- 172.21.15.28:0/2150762038 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6216+0+0 (secure 0 0 0) 0x7feee80ac010 con 0x7feefc14b250 2024-02-21T23:04:24.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.833+0000 7fef05bfc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630ebddffb0 0x5630ebd9c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:24.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.833+0000 7fef05bfc740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5630ebd9cb80 0x5630ebdeb150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:24.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.833+0000 7fef05bfc740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630ebde0390 0x5630ebdeb690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:24.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.833+0000 7fef05bfc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630ebd06470 con 0x5630ebddffb0 2024-02-21T23:04:24.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.833+0000 7fef05bfc740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630ebcafaf0 con 0x5630ebde0390 2024-02-21T23:04:24.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.833+0000 7fef05bfc740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x5630ebd02510 con 0x5630ebd9cb80 2024-02-21T23:04:24.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.833+0000 7feefbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630ebddffb0 0x5630ebd9c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:24.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.833+0000 7feefbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630ebddffb0 0x5630ebd9c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52724/0 (socket says 172.21.15.28:52724) 2024-02-21T23:04:24.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.833+0000 7feefbe00640 1 -- 172.21.15.28:0/1943595747 learned_addr learned my addr 172.21.15.28:0/1943595747 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:24.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.834+0000 7fef00c00640 1 --2- 172.21.15.28:0/2150762038 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7feedc06b500 0x7feedc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:24.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.834+0000 7feefb400640 1 --2- 172.21.15.28:0/1943595747 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5630ebd9cb80 0x5630ebdeb150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:24.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.834+0000 7fef00c00640 1 --2- 172.21.15.28:0/1943595747 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630ebde0390 0x5630ebdeb690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:24.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.834+0000 7feefbe00640 1 -- 172.21.15.28:0/1943595747 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5630ebd9cb80 msgr2=0x5630ebdeb150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:24.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.834+0000 7feefbe00640 1 --2- 172.21.15.28:0/1943595747 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5630ebd9cb80 0x5630ebdeb150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:24.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.834+0000 7feefbe00640 1 -- 172.21.15.28:0/1943595747 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630ebde0390 msgr2=0x5630ebdeb690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:24.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.834+0000 7feefbe00640 1 --2- 172.21.15.28:0/1943595747 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630ebde0390 0x5630ebdeb690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:24.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.834+0000 7feefbe00640 1 -- 172.21.15.28:0/1943595747 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5630ebcb6d90 con 0x5630ebddffb0 2024-02-21T23:04:24.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.834+0000 7fef00c00640 1 --2- 172.21.15.28:0/1943595747 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630ebde0390 0x5630ebdeb690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:04:24.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.834+0000 7feefbe00640 1 --2- 172.21.15.28:0/1943595747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630ebddffb0 0x5630ebd9c640 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7feeec009d50 tx=0x7feeec009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84a2307cd5b40d14 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:24.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.835+0000 7feee7e00640 1 -- 172.21.15.28:0/1943595747 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeec01a030 con 0x5630ebddffb0 2024-02-21T23:04:24.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.835+0000 7feee7e00640 1 -- 172.21.15.28:0/1943595747 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feeec013d50 con 0x5630ebddffb0 2024-02-21T23:04:24.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.835+0000 7feee7e00640 1 -- 172.21.15.28:0/1943595747 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeec016660 con 0x5630ebddffb0 2024-02-21T23:04:24.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.835+0000 7fef00c00640 1 --2- 172.21.15.28:0/2150762038 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7feedc06b500 0x7feedc06d9c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7feefc06eb10 tx=0x7feef40017b0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:24.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.835+0000 7fef05bfc740 1 -- 172.21.15.28:0/1943595747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630ebddffb0 msgr2=0x5630ebd9c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:24.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.835+0000 7fef05bfc740 1 --2- 172.21.15.28:0/1943595747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630ebddffb0 0x5630ebd9c640 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7feeec009d50 tx=0x7feeec009190 comp rx=0 tx=0).stop 2024-02-21T23:04:24.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.836+0000 7fef05bfc740 1 -- 172.21.15.28:0/1943595747 shutdown_connections 2024-02-21T23:04:24.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.836+0000 7fef05bfc740 1 --2- 172.21.15.28:0/1943595747 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5630ebd9cb80 0x5630ebdeb150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:24.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.836+0000 7fef05bfc740 1 --2- 172.21.15.28:0/1943595747 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630ebde0390 0x5630ebdeb690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:24.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.836+0000 7fef05bfc740 1 --2- 172.21.15.28:0/1943595747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630ebddffb0 0x5630ebd9c640 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:24.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.836+0000 7fef05bfc740 1 -- 172.21.15.28:0/1943595747 >> 172.21.15.28:0/1943595747 conn(0x5630ebd9e790 msgr2=0x5630ebdc31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:24.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.836+0000 7fef05bfc740 1 -- 172.21.15.28:0/1943595747 shutdown_connections 2024-02-21T23:04:24.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.836+0000 7fef05bfc740 1 -- 172.21.15.28:0/1943595747 wait complete. 2024-02-21T23:04:24.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.837+0000 7fef05bfc740 1 -- start start 2024-02-21T23:04:24.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.837+0000 7fef05bfc740 1 Processor -- start 2024-02-21T23:04:24.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.839+0000 7fef05bfc740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630ebddffb0 0x5630ebdc6620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:24.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.839+0000 7fef05bfc740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5630ebde0390 0x5630ebdc5020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:24.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.839+0000 7fef05bfc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630ebdc5560 0x5630ebdc1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:24.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.839+0000 7fef05bfc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630ebde0a70 con 0x5630ebdc5560 2024-02-21T23:04:24.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.839+0000 7fef05bfc740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630ebca5040 con 0x5630ebddffb0 2024-02-21T23:04:24.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.839+0000 7fef05bfc740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x5630ebda2330 con 0x5630ebde0390 2024-02-21T23:04:24.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.839+0000 7fef05bfc740 10 client.0 _ll_register_callbacks cb 0x5630ebcc2920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:04:24.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.839+0000 7fef00c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630ebdc5560 0x5630ebdc1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:24.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.839+0000 7fef00c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630ebdc5560 0x5630ebdc1260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52738/0 (socket says 172.21.15.28:52738) 2024-02-21T23:04:24.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.839+0000 7fef00c00640 1 -- 172.21.15.28:0/679657234 learned_addr learned my addr 172.21.15.28:0/679657234 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:24.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.839+0000 7feefb400640 1 --2- 172.21.15.28:0/679657234 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5630ebde0390 0x5630ebdc5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:24.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.839+0000 7feefbe00640 1 --2- 172.21.15.28:0/679657234 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630ebddffb0 0x5630ebdc6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:24.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.840+0000 7fef00c00640 1 -- 172.21.15.28:0/679657234 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5630ebde0390 msgr2=0x5630ebdc5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:24.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.840+0000 7fef00c00640 1 --2- 172.21.15.28:0/679657234 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5630ebde0390 0x5630ebdc5020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:24.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.840+0000 7fef00c00640 1 -- 172.21.15.28:0/679657234 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630ebddffb0 msgr2=0x5630ebdc6620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:24.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.840+0000 7fef00c00640 1 --2- 172.21.15.28:0/679657234 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630ebddffb0 0x5630ebdc6620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:24.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.840+0000 7fef00c00640 1 -- 172.21.15.28:0/679657234 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5630ebcc4aa0 con 0x5630ebdc5560 2024-02-21T23:04:24.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.840+0000 7feefbe00640 1 --2- 172.21.15.28:0/679657234 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630ebddffb0 0x5630ebdc6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:04:24.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.840+0000 7fef00c00640 1 --2- 172.21.15.28:0/679657234 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630ebdc5560 0x5630ebdc1260 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7feefc06f8f0 tx=0x7feefc0c2440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d15cc5357dbec935 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:24.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.840+0000 7feee4c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:04:24.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.840+0000 7feee4c00640 1 -- 172.21.15.28:0/679657234 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feef4026060 con 0x5630ebdc5560 2024-02-21T23:04:24.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.840+0000 7feee4c00640 1 -- 172.21.15.28:0/679657234 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(41..41 src has 1..41) v4 ==== 6216+0+0 (secure 0 0 0) 0x7feef4015cb0 con 0x5630ebdc5560 2024-02-21T23:04:24.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.840+0000 7fef05bfc740 1 -- 172.21.15.28:0/679657234 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7feeec013d50 con 0x5630ebdc5560 2024-02-21T23:04:24.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.841+0000 7feee4c00640 1 -- 172.21.15.28:0/679657234 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7feef4015eb0 con 0x5630ebdc5560 2024-02-21T23:04:24.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.841+0000 7fef05bfc740 10 client.5463 fetch_fsmap learned FSMap version 23 2024-02-21T23:04:24.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.841+0000 7fef05bfc740 1 -- 172.21.15.28:0/679657234 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=2+}) v3 -- 0x5630eba1b2d0 con 0x5630ebdc5560 2024-02-21T23:04:24.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.841+0000 7feee4c00640 1 -- 172.21.15.28:0/679657234 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feeec013d50 con 0x5630ebdc5560 2024-02-21T23:04:24.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.841+0000 7feee4c00640 1 -- 172.21.15.28:0/679657234 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 23) v1 ==== 3294+0+0 (secure 0 0 0) 0x7feef4016050 con 0x5630ebdc5560 2024-02-21T23:04:24.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.841+0000 7fef05bfc740 10 client.5463 fetch_fsmap finished waiting for FSMap version 23 2024-02-21T23:04:24.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.841+0000 7fef05bfc740 10 client.5463 resolve_mds: resolved name 'c' to daemon mds.c 2024-02-21T23:04:24.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.841+0000 7fef05bfc740 20 client.5463 populate_metadata read hostname 'smithi028' 2024-02-21T23:04:24.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.841+0000 7fef05bfc740 1 --2- 172.21.15.28:0/679657234 >> [v2:172.21.15.28:6832/2827878900,v1:172.21.15.28:6834/2827878900] conn(0x5630ebdf58f0 0x5630ebdf5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:24.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.841+0000 7fef05bfc740 4 client.5463 mds_command: new command op to 4778 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-21T23:04:24.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.841+0000 7fef05bfc740 1 -- 172.21.15.28:0/679657234 --> [v2:172.21.15.28:6832/2827878900,v1:172.21.15.28:6834/2827878900] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5630ebd139b0 con 0x5630ebdf58f0 2024-02-21T23:04:24.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.842+0000 7feefbe00640 1 --2- 172.21.15.28:0/679657234 >> [v2:172.21.15.28:6832/2827878900,v1:172.21.15.28:6834/2827878900] conn(0x5630ebdf58f0 0x5630ebdf5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:24.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.842+0000 7feefbe00640 1 --2- 172.21.15.28:0/679657234 >> [v2:172.21.15.28:6832/2827878900,v1:172.21.15.28:6834/2827878900] conn(0x5630ebdf58f0 0x5630ebdf5cd0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a31cefd0f53b5d7 server_cookie=2a9524e8a10053af in_seq=0 out_seq=0 2024-02-21T23:04:24.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.842+0000 7feee4c00640 10 client.5463 ms_handle_connect on v2:172.21.15.28:6832/2827878900 2024-02-21T23:04:24.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.844+0000 7feee4c00640 1 -- 172.21.15.28:0/679657234 <== mds.0 v2:172.21.15.28:6832/2827878900 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5630ebd139b0 con 0x5630ebdf58f0 2024-02-21T23:04:24.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.844+0000 7feee4c00640 10 client.5463 handle_command_reply: tid=0 2024-02-21T23:04:24.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.844+0000 7fef05bfc740 1 client.5463 shutdown 2024-02-21T23:04:24.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.844+0000 7fef05bfc740 1 -- 172.21.15.28:0/679657234 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630ebdc5560 msgr2=0x5630ebdc1260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:24.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.844+0000 7fef05bfc740 1 --2- 172.21.15.28:0/679657234 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630ebdc5560 0x5630ebdc1260 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7feefc06f8f0 tx=0x7feefc0c2440 comp rx=0 tx=0).stop 2024-02-21T23:04:24.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.844+0000 7fef05bfc740 1 -- 172.21.15.28:0/679657234 shutdown_connections 2024-02-21T23:04:24.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.844+0000 7fef05bfc740 1 --2- 172.21.15.28:0/679657234 >> [v2:172.21.15.28:6832/2827878900,v1:172.21.15.28:6834/2827878900] conn(0x5630ebdf58f0 0x5630ebdf5cd0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:24.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.844+0000 7fef05bfc740 1 --2- 172.21.15.28:0/679657234 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5630ebde0390 0x5630ebdc5020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:24.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.844+0000 7fef05bfc740 1 --2- 172.21.15.28:0/679657234 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630ebdc5560 0x5630ebdc1260 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:24.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.844+0000 7fef05bfc740 1 --2- 172.21.15.28:0/679657234 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630ebddffb0 0x5630ebdc6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:24.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.844+0000 7fef05bfc740 1 -- 172.21.15.28:0/679657234 >> 172.21.15.28:0/679657234 conn(0x5630ebd9cb80 msgr2=0x5630ebd9b5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:24.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.844+0000 7feee4c00640 0 client.5463 ms_handle_reset on v2:172.21.15.28:6832/2827878900 2024-02-21T23:04:24.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.844+0000 7fef05bfc740 1 -- 172.21.15.28:0/679657234 shutdown_connections 2024-02-21T23:04:24.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.844+0000 7fef05bfc740 1 -- 172.21.15.28:0/679657234 wait complete. 2024-02-21T23:04:24.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.845+0000 7fef05bfc740 20 client.5463 trim_cache size 0 max 16384 2024-02-21T23:04:24.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.863+0000 7fef05bfc740 1 Processor -- start 2024-02-21T23:04:24.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.863+0000 7fef05bfc740 1 -- start start 2024-02-21T23:04:24.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.864+0000 7fef05bfc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630ebdc5560 0x5630ebda1720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:24.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.864+0000 7fef05bfc740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630ebddffb0 0x5630ebdef4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:24.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.864+0000 7fef05bfc740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5630ebde0390 0x5630ebdc4710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:24.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.864+0000 7fef05bfc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feefc1f3610 con 0x5630ebdc5560 2024-02-21T23:04:24.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.864+0000 7fef05bfc740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feefc150710 con 0x5630ebddffb0 2024-02-21T23:04:24.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.864+0000 7fef05bfc740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feefc150d10 con 0x5630ebde0390 2024-02-21T23:04:24.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.864+0000 7feefbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630ebdc5560 0x5630ebda1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:24.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.864+0000 7feefbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630ebdc5560 0x5630ebda1720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52752/0 (socket says 172.21.15.28:52752) 2024-02-21T23:04:24.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.864+0000 7feefbe00640 1 -- 172.21.15.28:0/2124973378 learned_addr learned my addr 172.21.15.28:0/2124973378 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:24.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.865+0000 7feefbe00640 1 -- 172.21.15.28:0/2124973378 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5630ebde0390 msgr2=0x5630ebdc4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:24.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.865+0000 7feefbe00640 1 --2- 172.21.15.28:0/2124973378 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5630ebde0390 0x5630ebdc4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:24.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.865+0000 7feefbe00640 1 -- 172.21.15.28:0/2124973378 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630ebddffb0 msgr2=0x5630ebdef4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:24.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.865+0000 7feefbe00640 1 --2- 172.21.15.28:0/2124973378 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630ebddffb0 0x5630ebdef4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:24.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.865+0000 7feefbe00640 1 -- 172.21.15.28:0/2124973378 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5630ebd9e530 con 0x5630ebdc5560 2024-02-21T23:04:24.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.865+0000 7feefbe00640 1 --2- 172.21.15.28:0/2124973378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630ebdc5560 0x5630ebda1720 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7feeec008010 tx=0x7feeec018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cd247362f3b4189 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:24.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.865+0000 7feee7400640 1 -- 172.21.15.28:0/2124973378 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeec0042c0 con 0x5630ebdc5560 2024-02-21T23:04:24.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.865+0000 7feee7400640 1 -- 172.21.15.28:0/2124973378 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feeec031070 con 0x5630ebdc5560 2024-02-21T23:04:24.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.865+0000 7feee7400640 1 -- 172.21.15.28:0/2124973378 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeec02b680 con 0x5630ebdc5560 2024-02-21T23:04:24.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.866+0000 7fef05bfc740 1 -- 172.21.15.28:0/2124973378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630ebdc5560 msgr2=0x5630ebda1720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:24.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.866+0000 7fef05bfc740 1 --2- 172.21.15.28:0/2124973378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630ebdc5560 0x5630ebda1720 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7feeec008010 tx=0x7feeec018000 comp rx=0 tx=0).stop 2024-02-21T23:04:24.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.866+0000 7fef05bfc740 1 -- 172.21.15.28:0/2124973378 shutdown_connections 2024-02-21T23:04:24.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.866+0000 7fef05bfc740 1 --2- 172.21.15.28:0/2124973378 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5630ebde0390 0x5630ebdc4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:24.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.866+0000 7fef05bfc740 1 --2- 172.21.15.28:0/2124973378 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630ebddffb0 0x5630ebdef4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:24.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.866+0000 7fef05bfc740 1 --2- 172.21.15.28:0/2124973378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630ebdc5560 0x5630ebda1720 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:24.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.866+0000 7fef05bfc740 1 -- 172.21.15.28:0/2124973378 >> 172.21.15.28:0/2124973378 conn(0x5630ebd9cb80 msgr2=0x5630ebddc490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:24.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.866+0000 7fef05bfc740 1 -- 172.21.15.28:0/2124973378 shutdown_connections 2024-02-21T23:04:24.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.866+0000 7fef05bfc740 1 -- 172.21.15.28:0/2124973378 wait complete. 2024-02-21T23:04:24.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.867+0000 7fef05bfc740 1 -- start start 2024-02-21T23:04:24.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.867+0000 7fef05bfc740 1 Processor -- start 2024-02-21T23:04:24.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.868+0000 7fef05bfc740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5630ebddffb0 0x5630ebdc3340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:24.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.868+0000 7fef05bfc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630ebde0390 0x5630ebdbde70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:24.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.868+0000 7fef05bfc740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630ebdbe3b0 0x5630ebe4ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:24.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.868+0000 7fef05bfc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630ebdf2e50 con 0x5630ebde0390 2024-02-21T23:04:24.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.868+0000 7fef05bfc740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630ebdd7ad0 con 0x5630ebdbe3b0 2024-02-21T23:04:24.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.868+0000 7fef05bfc740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x5630ebdd6d10 con 0x5630ebddffb0 2024-02-21T23:04:24.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.868+0000 7fef05bfc740 10 client.0 _ll_register_callbacks cb 0x7feef4010e40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:04:24.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.868+0000 7feefb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630ebde0390 0x5630ebdbde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:24.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.868+0000 7feefb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630ebde0390 0x5630ebdbde70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52766/0 (socket says 172.21.15.28:52766) 2024-02-21T23:04:24.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.868+0000 7feefb400640 1 -- 172.21.15.28:0/871719880 learned_addr learned my addr 172.21.15.28:0/871719880 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:24.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.869+0000 7feefbe00640 1 --2- 172.21.15.28:0/871719880 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5630ebddffb0 0x5630ebdc3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:24.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.869+0000 7fef00c00640 1 --2- 172.21.15.28:0/871719880 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630ebdbe3b0 0x5630ebe4ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:24.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.869+0000 7feefb400640 1 -- 172.21.15.28:0/871719880 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5630ebddffb0 msgr2=0x5630ebdc3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:24.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.869+0000 7feefb400640 1 --2- 172.21.15.28:0/871719880 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5630ebddffb0 0x5630ebdc3340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:24.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.869+0000 7feefb400640 1 -- 172.21.15.28:0/871719880 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630ebdbe3b0 msgr2=0x5630ebe4ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:24.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.869+0000 7feefb400640 1 --2- 172.21.15.28:0/871719880 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630ebdbe3b0 0x5630ebe4ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:24.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.869+0000 7feefb400640 1 -- 172.21.15.28:0/871719880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5630ebae56b0 con 0x5630ebde0390 2024-02-21T23:04:24.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.869+0000 7feefbe00640 1 --2- 172.21.15.28:0/871719880 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5630ebddffb0 0x5630ebdc3340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:04:24.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.869+0000 7fef00c00640 1 --2- 172.21.15.28:0/871719880 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630ebdbe3b0 0x5630ebe4ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:04:24.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.869+0000 7feefb400640 1 --2- 172.21.15.28:0/871719880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630ebde0390 0x5630ebdbde70 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7feee80aa3d0 tx=0x7feee807a290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9f64a5949401650 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:24.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.869+0000 7feee4c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:04:24.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.869+0000 7feee4c00640 1 -- 172.21.15.28:0/871719880 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feee80a8060 con 0x5630ebde0390 2024-02-21T23:04:24.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.869+0000 7feee4c00640 1 -- 172.21.15.28:0/871719880 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(41..41 src has 1..41) v4 ==== 6216+0+0 (secure 0 0 0) 0x7feee80828a0 con 0x5630ebde0390 2024-02-21T23:04:24.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.869+0000 7fef05bfc740 1 -- 172.21.15.28:0/871719880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7feeec031070 con 0x5630ebde0390 2024-02-21T23:04:24.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.869+0000 7feee4c00640 1 -- 172.21.15.28:0/871719880 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7feee807a620 con 0x5630ebde0390 2024-02-21T23:04:24.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.869+0000 7fef05bfc740 10 client.5469 fetch_fsmap learned FSMap version 23 2024-02-21T23:04:24.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.869+0000 7fef05bfc740 1 -- 172.21.15.28:0/871719880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=2+}) v3 -- 0x5630ebd9be80 con 0x5630ebde0390 2024-02-21T23:04:24.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.870+0000 7feee4c00640 1 -- 172.21.15.28:0/871719880 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feeec031070 con 0x5630ebde0390 2024-02-21T23:04:24.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.870+0000 7feee4c00640 1 -- 172.21.15.28:0/871719880 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 23) v1 ==== 3294+0+0 (secure 0 0 0) 0x7feee802cbb0 con 0x5630ebde0390 2024-02-21T23:04:24.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.870+0000 7fef05bfc740 10 client.5469 fetch_fsmap finished waiting for FSMap version 23 2024-02-21T23:04:24.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.870+0000 7fef05bfc740 10 client.5469 resolve_mds: resolved name 'c' to daemon mds.c 2024-02-21T23:04:24.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.870+0000 7fef05bfc740 20 client.5469 populate_metadata read hostname 'smithi028' 2024-02-21T23:04:24.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.870+0000 7fef05bfc740 1 --2- 172.21.15.28:0/871719880 >> [v2:172.21.15.28:6832/2827878900,v1:172.21.15.28:6834/2827878900] conn(0x5630ebddab20 0x5630ebddaf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:24.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.870+0000 7fef05bfc740 4 client.5469 mds_command: new command op to 4778 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-21T23:04:24.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.870+0000 7fef05bfc740 1 -- 172.21.15.28:0/871719880 --> [v2:172.21.15.28:6832/2827878900,v1:172.21.15.28:6834/2827878900] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5630ebd0dd00 con 0x5630ebddab20 2024-02-21T23:04:24.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.870+0000 7feefbe00640 1 --2- 172.21.15.28:0/871719880 >> [v2:172.21.15.28:6832/2827878900,v1:172.21.15.28:6834/2827878900] conn(0x5630ebddab20 0x5630ebddaf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:24.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.871+0000 7feefbe00640 1 --2- 172.21.15.28:0/871719880 >> [v2:172.21.15.28:6832/2827878900,v1:172.21.15.28:6834/2827878900] conn(0x5630ebddab20 0x5630ebddaf00 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e287de8b0246a01 server_cookie=6050a2551a62169c in_seq=0 out_seq=0 2024-02-21T23:04:24.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.871+0000 7feee4c00640 10 client.5469 ms_handle_connect on v2:172.21.15.28:6832/2827878900 2024-02-21T23:04:24.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.871+0000 7feee4c00640 1 -- 172.21.15.28:0/871719880 <== mds.0 v2:172.21.15.28:6832/2827878900 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x5630ebd0dd00 con 0x5630ebddab20 2024-02-21T23:04:24.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.871+0000 7feee4c00640 10 client.5469 handle_command_reply: tid=0 2024-02-21T23:04:24.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.871+0000 7fef05bfc740 1 client.5469 shutdown 2024-02-21T23:04:24.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.871+0000 7fef05bfc740 1 -- 172.21.15.28:0/871719880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630ebde0390 msgr2=0x5630ebdbde70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:24.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.871+0000 7fef05bfc740 1 --2- 172.21.15.28:0/871719880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630ebde0390 0x5630ebdbde70 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7feee80aa3d0 tx=0x7feee807a290 comp rx=0 tx=0).stop 2024-02-21T23:04:24.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.871+0000 7fef05bfc740 1 -- 172.21.15.28:0/871719880 shutdown_connections 2024-02-21T23:04:24.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.871+0000 7fef05bfc740 1 --2- 172.21.15.28:0/871719880 >> [v2:172.21.15.28:6832/2827878900,v1:172.21.15.28:6834/2827878900] conn(0x5630ebddab20 0x5630ebddaf00 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:24.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.871+0000 7fef05bfc740 1 --2- 172.21.15.28:0/871719880 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5630ebdbe3b0 0x5630ebe4ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:24.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.871+0000 7fef05bfc740 1 --2- 172.21.15.28:0/871719880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630ebde0390 0x5630ebdbde70 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:24.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.871+0000 7fef05bfc740 1 --2- 172.21.15.28:0/871719880 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5630ebddffb0 0x5630ebdc3340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:24.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.871+0000 7feee4c00640 0 client.5469 ms_handle_reset on v2:172.21.15.28:6832/2827878900 2024-02-21T23:04:24.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.871+0000 7fef05bfc740 1 -- 172.21.15.28:0/871719880 >> 172.21.15.28:0/871719880 conn(0x5630ebd9cb80 msgr2=0x5630ebdef4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:24.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.872+0000 7fef05bfc740 1 -- 172.21.15.28:0/871719880 shutdown_connections 2024-02-21T23:04:24.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.872+0000 7fef05bfc740 1 -- 172.21.15.28:0/871719880 wait complete. 2024-02-21T23:04:24.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.872+0000 7fef05bfc740 20 client.5469 trim_cache size 0 max 16384 2024-02-21T23:04:24.865 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-21T23:04:24.865 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac", 2024-02-21T23:04:24.865 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-02-21T23:04:24.865 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4778, 2024-02-21T23:04:24.866 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-02-21T23:04:24.866 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-02-21T23:04:24.866 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-02-21T23:04:24.866 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 4.1788022079999996, 2024-02-21T23:04:24.866 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 23, 2024-02-21T23:04:24.866 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 41, 2024-02-21T23:04:24.866 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 39, 2024-02-21T23:04:24.866 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 24.300574596000001 2024-02-21T23:04:24.866 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-21T23:04:24.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.872+0000 7fef02c00640 1 -- 172.21.15.28:0/2150762038 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7feedc06b500 msgr2=0x7feedc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:24.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.872+0000 7fef02c00640 1 --2- 172.21.15.28:0/2150762038 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7feedc06b500 0x7feedc06d9c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7feefc06eb10 tx=0x7feef40017b0 comp rx=0 tx=0).stop 2024-02-21T23:04:24.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.872+0000 7fef02c00640 1 -- 172.21.15.28:0/2150762038 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feefc14b250 msgr2=0x7feefc1e3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:24.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.872+0000 7fef02c00640 1 --2- 172.21.15.28:0/2150762038 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feefc14b250 0x7feefc1e3e20 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7feee8037040 tx=0x7feee8004730 comp rx=0 tx=0).stop 2024-02-21T23:04:24.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.872+0000 7fef02c00640 1 -- 172.21.15.28:0/2150762038 shutdown_connections 2024-02-21T23:04:24.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.872+0000 7fef02c00640 1 --2- 172.21.15.28:0/2150762038 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7feedc06b500 0x7feedc06d9c0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:24.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.872+0000 7fef02c00640 1 --2- 172.21.15.28:0/2150762038 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feefc1e7510 0x7feefc1e4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:24.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.872+0000 7fef02c00640 1 --2- 172.21.15.28:0/2150762038 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feefc14b250 0x7feefc1e3e20 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:24.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.872+0000 7fef02c00640 1 --2- 172.21.15.28:0/2150762038 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feee0004800 0x7feefc14cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:24.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.872+0000 7fef02c00640 1 -- 172.21.15.28:0/2150762038 >> 172.21.15.28:0/2150762038 conn(0x7feefc06b820 msgr2=0x7feefc0a7d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:24.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.873+0000 7fef02c00640 1 -- 172.21.15.28:0/2150762038 shutdown_connections 2024-02-21T23:04:24.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:24.873+0000 7fef02c00640 1 -- 172.21.15.28:0/2150762038 wait complete. 2024-02-21T23:04:24.880 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 2, "epoch": 23, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-02-21T23:04:09.036302+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 23, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4778": { "addr": "172.21.15.28:6834/2827878900", "addrs": { "addrvec": [ { "addr": "172.21.15.28:6832", "nonce": 2827878900, "type": "v2" }, { "addr": "172.21.15.28:6834", "nonce": 2827878900, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4778, "incarnation": 20, "join_fscid": 2, "name": "c", "rank": 0, "state": "up:active", "state_seq": 9 } }, "last_failure": 0, "last_failure_osd_epoch": 39, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 5, "modified": "2024-02-21T23:04:22.712438+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4778 } } } ], "standbys": [ { "addr": "172.21.15.81:6836/3612644344", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6834", "nonce": 3612644344, "type": "v2" }, { "addr": "172.21.15.81:6836", "nonce": 3612644344, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 12, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4978, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.81:6837/3623937361", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6835", "nonce": 3623937361, "type": "v2" }, { "addr": "172.21.15.81:6837", "nonce": 3623937361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 13, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4981, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.28:6835/33413357", "addrs": { "addrvec": [ { "addr": "172.21.15.28:6833", "nonce": 33413357, "type": "v2" }, { "addr": "172.21.15.28:6835", "nonce": 33413357, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 21, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5442, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-02-21T23:04:24.881 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': 2, 'name': 'c', 'state': 'up:active'}, {'join_fscid': -1, 'name': 'd'}] 2024-02-21T23:04:24.882 INFO:tasks.cephfs.test_failover:current = [{'gid': 5442, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.28:6835/33413357', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6833', 'nonce': 33413357}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 33413357}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 21}, {'gid': 4981, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.81:6837/3623937361', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6835', 'nonce': 3623937361}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3623937361}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 13}, {'gid': 4778, 'name': 'c', 'rank': 0, 'incarnation': 20, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.28:6834/2827878900', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 2827878900}, {'type': 'v1', 'addr': '172.21.15.28:6834', 'nonce': 2827878900}]}, 'join_fscid': 2, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 4978, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.81:6836/3612644344', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6834', 'nonce': 3612644344}, {'type': 'v1', 'addr': '172.21.15.81:6836', 'nonce': 3612644344}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 12}] 2024-02-21T23:04:24.882 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-02-21T23:04:24.882 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.c mds_join_fs 2024-02-21T23:04:25.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.112+0000 7fdf64600640 1 Processor -- start 2024-02-21T23:04:25.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.113+0000 7fdf64600640 1 -- start start 2024-02-21T23:04:25.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.114+0000 7fdf64600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fdf5c150700 0x7fdf5c156500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:25.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.114+0000 7fdf64600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf5c152fb0 con 0x7fdf5c06b0e0 2024-02-21T23:04:25.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.114+0000 7fdf64600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf5c153290 con 0x7fdf5c0c5f50 2024-02-21T23:04:25.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.114+0000 7fdf64600640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf5c153bc0 con 0x7fdf5c150700 2024-02-21T23:04:25.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.115+0000 7fdf61c00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fdf5c06b0e0 0x7fdf5c0c5840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39896/0 (socket says 172.21.15.28:39896) 2024-02-21T23:04:25.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.115+0000 7fdf61c00640 1 -- 172.21.15.28:0/1376912671 learned_addr learned my addr 172.21.15.28:0/1376912671 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:04:25.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.115+0000 7fdf62600640 1 --2- 172.21.15.28:0/1376912671 >> v2:172.21.15.81:3300/0 conn(0x7fdf5c150700 0x7fdf5c156500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:25.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.115+0000 7fdf5be00640 1 -- 172.21.15.28:0/1376912671 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3344293886 0 0) 0x7fdf5c152fb0 con 0x7fdf5c06b0e0 2024-02-21T23:04:25.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.115+0000 7fdf5be00640 1 -- 172.21.15.28:0/1376912671 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf40003520 con 0x7fdf5c06b0e0 2024-02-21T23:04:25.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.116+0000 7fdf5be00640 1 -- 172.21.15.28:0/1376912671 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3077481636 0 0) 0x7fdf40003520 con 0x7fdf5c06b0e0 2024-02-21T23:04:25.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.116+0000 7fdf5be00640 1 -- 172.21.15.28:0/1376912671 >> v2:172.21.15.81:3300/0 conn(0x7fdf5c150700 msgr2=0x7fdf5c156500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:25.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.116+0000 7fdf5be00640 1 --2- 172.21.15.28:0/1376912671 >> v2:172.21.15.81:3300/0 conn(0x7fdf5c150700 0x7fdf5c156500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:25.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.116+0000 7fdf5be00640 1 -- 172.21.15.28:0/1376912671 >> v1:172.21.15.81:6789/0 conn(0x7fdf5c0c5f50 legacy=0x7fdf5c14fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:25.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.116+0000 7fdf5be00640 1 -- 172.21.15.28:0/1376912671 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf5c153570 con 0x7fdf5c06b0e0 2024-02-21T23:04:25.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.116+0000 7fdf5be00640 1 -- 172.21.15.28:0/1376912671 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fdf4c001670 con 0x7fdf5c06b0e0 2024-02-21T23:04:25.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.116+0000 7fdf5be00640 1 -- 172.21.15.28:0/1376912671 >> v1:172.21.15.28:6789/0 conn(0x7fdf5c06b0e0 legacy=0x7fdf5c0c5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:25.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.116+0000 7fdf5be00640 1 --2- 172.21.15.28:0/1376912671 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdf400039b0 0x7fdf40003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:25.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.116+0000 7fdf5be00640 1 --2- 172.21.15.28:0/1376912671 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf40004800 0x7fdf40004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:25.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.116+0000 7fdf5be00640 1 --2- 172.21.15.28:0/1376912671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf400051c0 0x7fdf4000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:25.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.116+0000 7fdf5be00640 1 -- 172.21.15.28:0/1376912671 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf5c153bc0 con 0x7fdf400051c0 2024-02-21T23:04:25.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.116+0000 7fdf5be00640 1 -- 172.21.15.28:0/1376912671 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf4000c160 con 0x7fdf40004800 2024-02-21T23:04:25.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.116+0000 7fdf5be00640 1 -- 172.21.15.28:0/1376912671 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf4000c360 con 0x7fdf400039b0 2024-02-21T23:04:25.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.116+0000 7fdf62600640 1 --2- 172.21.15.28:0/1376912671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf400051c0 0x7fdf4000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:25.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.117+0000 7fdf61200640 1 --2- 172.21.15.28:0/1376912671 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf40004800 0x7fdf40004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:25.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.117+0000 7fdf61c00640 1 --2- 172.21.15.28:0/1376912671 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdf400039b0 0x7fdf40003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:25.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.117+0000 7fdf62600640 1 -- 172.21.15.28:0/1376912671 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdf400039b0 msgr2=0x7fdf40003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:25.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.117+0000 7fdf62600640 1 --2- 172.21.15.28:0/1376912671 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdf400039b0 0x7fdf40003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:25.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.117+0000 7fdf62600640 1 -- 172.21.15.28:0/1376912671 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf40004800 msgr2=0x7fdf40004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:25.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.117+0000 7fdf62600640 1 --2- 172.21.15.28:0/1376912671 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf40004800 0x7fdf40004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:25.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.117+0000 7fdf62600640 1 -- 172.21.15.28:0/1376912671 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf40003520 con 0x7fdf400051c0 2024-02-21T23:04:25.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.118+0000 7fdf62600640 1 -- 172.21.15.28:0/1376912671 reap_dead start 2024-02-21T23:04:25.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.118+0000 7fdf62600640 1 --2- 172.21.15.28:0/1376912671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf400051c0 0x7fdf4000b9c0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fdf54009fc0 tx=0x7fdf40005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d60c2a1a1919907 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:25.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.118+0000 7fdf5be00640 1 -- 172.21.15.28:0/1376912671 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf54009c10 con 0x7fdf400051c0 2024-02-21T23:04:25.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.118+0000 7fdf5be00640 1 -- 172.21.15.28:0/1376912671 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf5402cc60 con 0x7fdf400051c0 2024-02-21T23:04:25.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.118+0000 7fdf5be00640 1 -- 172.21.15.28:0/1376912671 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf54003bd0 con 0x7fdf400051c0 2024-02-21T23:04:25.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.119+0000 7fdf64600640 1 -- 172.21.15.28:0/1376912671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf400051c0 msgr2=0x7fdf4000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:25.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.119+0000 7fdf64600640 1 --2- 172.21.15.28:0/1376912671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf400051c0 0x7fdf4000b9c0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fdf54009fc0 tx=0x7fdf40005830 comp rx=0 tx=0).stop 2024-02-21T23:04:25.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.119+0000 7fdf64600640 1 -- 172.21.15.28:0/1376912671 shutdown_connections 2024-02-21T23:04:25.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.119+0000 7fdf64600640 1 --2- 172.21.15.28:0/1376912671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf400051c0 0x7fdf4000b9c0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:25.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.119+0000 7fdf64600640 1 -- 172.21.15.28:0/1376912671 >> 172.21.15.28:0/1376912671 conn(0x7fdf5c0a85a0 msgr2=0x7fdf5c0a8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:25.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.119+0000 7fdf64600640 1 -- 172.21.15.28:0/1376912671 shutdown_connections 2024-02-21T23:04:25.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.119+0000 7fdf64600640 1 -- 172.21.15.28:0/1376912671 wait complete. 2024-02-21T23:04:25.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.122+0000 7fdf64600640 1 Processor -- start 2024-02-21T23:04:25.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.122+0000 7fdf64600640 1 -- start start 2024-02-21T23:04:25.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.124+0000 7fdf64600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf400051c0 0x7fdf5c0bfe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:25.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.124+0000 7fdf64600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf5c152a30 0x7fdf5c14da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:25.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.124+0000 7fdf64600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdf5c1eada0 0x7fdf5c1eb220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:25.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.124+0000 7fdf64600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf5c06b640 con 0x7fdf5c152a30 2024-02-21T23:04:25.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.124+0000 7fdf64600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf5c06b4c0 con 0x7fdf400051c0 2024-02-21T23:04:25.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.124+0000 7fdf64600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf5c0c5840 con 0x7fdf5c1eada0 2024-02-21T23:04:25.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.124+0000 7fdf61200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf5c152a30 0x7fdf5c14da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:25.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.124+0000 7fdf61200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf5c152a30 0x7fdf5c14da80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48028/0 (socket says 172.21.15.28:48028) 2024-02-21T23:04:25.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.124+0000 7fdf61200640 1 -- 172.21.15.28:0/1347228534 learned_addr learned my addr 172.21.15.28:0/1347228534 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:25.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.124+0000 7fdf61c00640 1 --2- 172.21.15.28:0/1347228534 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf400051c0 0x7fdf5c0bfe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:25.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.124+0000 7fdf62600640 1 --2- 172.21.15.28:0/1347228534 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdf5c1eada0 0x7fdf5c1eb220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:25.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.124+0000 7fdf61200640 1 -- 172.21.15.28:0/1347228534 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdf5c1eada0 msgr2=0x7fdf5c1eb220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:25.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.124+0000 7fdf61200640 1 --2- 172.21.15.28:0/1347228534 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdf5c1eada0 0x7fdf5c1eb220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:25.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.124+0000 7fdf61200640 1 -- 172.21.15.28:0/1347228534 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf400051c0 msgr2=0x7fdf5c0bfe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:25.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.125+0000 7fdf61200640 1 --2- 172.21.15.28:0/1347228534 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf400051c0 0x7fdf5c0bfe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:25.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.125+0000 7fdf61200640 1 -- 172.21.15.28:0/1347228534 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf5c14f7e0 con 0x7fdf5c152a30 2024-02-21T23:04:25.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.125+0000 7fdf61200640 1 --2- 172.21.15.28:0/1347228534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf5c152a30 0x7fdf5c14da80 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fdf4800e4d0 tx=0x7fdf4800e990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:25.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.125+0000 7fdf59600640 1 -- 172.21.15.28:0/1347228534 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf480052e0 con 0x7fdf5c152a30 2024-02-21T23:04:25.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.125+0000 7fdf59600640 1 -- 172.21.15.28:0/1347228534 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf48018070 con 0x7fdf5c152a30 2024-02-21T23:04:25.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.125+0000 7fdf59600640 1 -- 172.21.15.28:0/1347228534 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf48003810 con 0x7fdf5c152a30 2024-02-21T23:04:25.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.125+0000 7fdf64600640 1 -- 172.21.15.28:0/1347228534 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf5c0a74b0 con 0x7fdf5c152a30 2024-02-21T23:04:25.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.125+0000 7fdf64600640 1 -- 172.21.15.28:0/1347228534 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf5c1ee740 con 0x7fdf5c152a30 2024-02-21T23:04:25.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.127+0000 7fdf64600640 1 -- 172.21.15.28:0/1347228534 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf5c067f30 con 0x7fdf5c152a30 2024-02-21T23:04:25.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.127+0000 7fdf59600640 1 -- 172.21.15.28:0/1347228534 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdf480039b0 con 0x7fdf5c152a30 2024-02-21T23:04:25.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.128+0000 7fdf59600640 1 --2- 172.21.15.28:0/1347228534 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdf3c06b500 0x7fdf3c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:25.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.128+0000 7fdf59600640 1 -- 172.21.15.28:0/1347228534 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6216+0+0 (secure 0 0 0) 0x7fdf48089ff0 con 0x7fdf5c152a30 2024-02-21T23:04:25.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.129+0000 7fdf61c00640 1 --2- 172.21.15.28:0/1347228534 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdf3c06b500 0x7fdf3c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:25.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.130+0000 7fdf61c00640 1 --2- 172.21.15.28:0/1347228534 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdf3c06b500 0x7fdf3c06d9c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c000fb0 tx=0x7fdf4c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:25.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.133+0000 7fdf59600640 1 -- 172.21.15.28:0/1347228534 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fdf5c0a74b0 con 0x7fdf5c152a30 2024-02-21T23:04:25.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.334+0000 7fdf64600640 1 -- 172.21.15.28:0/1347228534 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.c", "name": "mds_join_fs"} v 0) v1 -- 0x7fdf5c14cce0 con 0x7fdf5c152a30 2024-02-21T23:04:25.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.340+0000 7fdf59600640 1 -- 172.21.15.28:0/1347228534 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.c", "name": "mds_join_fs"}]=0 v13) v1 ==== 96+0+0 (secure 0 0 0) 0x7fdf48058c60 con 0x7fdf5c152a30 2024-02-21T23:04:25.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.343+0000 7fdf64600640 1 -- 172.21.15.28:0/1347228534 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdf3c06b500 msgr2=0x7fdf3c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:25.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.343+0000 7fdf64600640 1 --2- 172.21.15.28:0/1347228534 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdf3c06b500 0x7fdf3c06d9c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c000fb0 tx=0x7fdf4c009040 comp rx=0 tx=0).stop 2024-02-21T23:04:25.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.343+0000 7fdf64600640 1 -- 172.21.15.28:0/1347228534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf5c152a30 msgr2=0x7fdf5c14da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:25.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.343+0000 7fdf64600640 1 --2- 172.21.15.28:0/1347228534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf5c152a30 0x7fdf5c14da80 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fdf4800e4d0 tx=0x7fdf4800e990 comp rx=0 tx=0).stop 2024-02-21T23:04:25.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.343+0000 7fdf64600640 1 -- 172.21.15.28:0/1347228534 shutdown_connections 2024-02-21T23:04:25.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.343+0000 7fdf64600640 1 --2- 172.21.15.28:0/1347228534 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdf3c06b500 0x7fdf3c06d9c0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:25.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.343+0000 7fdf64600640 1 --2- 172.21.15.28:0/1347228534 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdf5c1eada0 0x7fdf5c1eb220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:25.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.343+0000 7fdf64600640 1 --2- 172.21.15.28:0/1347228534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf5c152a30 0x7fdf5c14da80 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:25.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.343+0000 7fdf64600640 1 --2- 172.21.15.28:0/1347228534 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf400051c0 0x7fdf5c0bfe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:25.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.343+0000 7fdf64600640 1 -- 172.21.15.28:0/1347228534 >> 172.21.15.28:0/1347228534 conn(0x7fdf5c0a85a0 msgr2=0x7fdf5c14c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:25.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.344+0000 7fdf64600640 1 -- 172.21.15.28:0/1347228534 shutdown_connections 2024-02-21T23:04:25.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.344+0000 7fdf64600640 1 -- 172.21.15.28:0/1347228534 wait complete. 2024-02-21T23:04:25.354 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:04:25.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.576+0000 7f3622a00640 1 Processor -- start 2024-02-21T23:04:25.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.576+0000 7f3622a00640 1 -- start start 2024-02-21T23:04:25.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.579+0000 7f3622a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f361c073360 0x7f361c1583a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:25.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.579+0000 7f3622a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f361c072770 0x7f361c1588e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:25.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.579+0000 7f3622a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f361c155ba0 con 0x7f361c070750 2024-02-21T23:04:25.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.579+0000 7f3622a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f361c0c3b60 con 0x7f361c073360 2024-02-21T23:04:25.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.579+0000 7f3622a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f361c0c3e40 con 0x7f361c072770 2024-02-21T23:04:25.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.580+0000 7f3620c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f361c073360 0x7f361c1583a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:25.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.580+0000 7f3620c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f361c073360 0x7f361c1583a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48034/0 (socket says 172.21.15.28:48034) 2024-02-21T23:04:25.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.580+0000 7f3620c00640 1 -- 172.21.15.28:0/810762147 learned_addr learned my addr 172.21.15.28:0/810762147 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:25.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.580+0000 7f3620c00640 1 -- 172.21.15.28:0/810762147 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f361c072770 msgr2=0x7f361c1588e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:25.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.580+0000 7f3620c00640 1 --2- 172.21.15.28:0/810762147 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f361c072770 0x7f361c1588e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:25.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.580+0000 7f3620c00640 1 -- 172.21.15.28:0/810762147 >> v1:172.21.15.81:6789/0 conn(0x7f361c070750 legacy=0x7f361c06ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:25.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.580+0000 7f3620c00640 1 -- 172.21.15.28:0/810762147 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f361c0a9cd0 con 0x7f361c073360 2024-02-21T23:04:25.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.581+0000 7f3620c00640 1 --2- 172.21.15.28:0/810762147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f361c073360 0x7f361c1583a0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f3608009870 tx=0x7f360802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cb3ec783e0517f6 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:25.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.582+0000 7f361be00640 1 -- 172.21.15.28:0/810762147 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3608035020 con 0x7f361c073360 2024-02-21T23:04:25.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.582+0000 7f361be00640 1 -- 172.21.15.28:0/810762147 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3608031640 con 0x7f361c073360 2024-02-21T23:04:25.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.582+0000 7f361be00640 1 -- 172.21.15.28:0/810762147 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f360802d040 con 0x7f361c073360 2024-02-21T23:04:25.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.582+0000 7f3622a00640 1 -- 172.21.15.28:0/810762147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f361c073360 msgr2=0x7f361c1583a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:25.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.582+0000 7f3622a00640 1 --2- 172.21.15.28:0/810762147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f361c073360 0x7f361c1583a0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f3608009870 tx=0x7f360802b8f0 comp rx=0 tx=0).stop 2024-02-21T23:04:25.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.583+0000 7f3622a00640 1 -- 172.21.15.28:0/810762147 shutdown_connections 2024-02-21T23:04:25.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.583+0000 7f3622a00640 1 --2- 172.21.15.28:0/810762147 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f361c072770 0x7f361c1588e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:25.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.583+0000 7f3622a00640 1 --2- 172.21.15.28:0/810762147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f361c073360 0x7f361c1583a0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:25.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.583+0000 7f3622a00640 1 -- 172.21.15.28:0/810762147 >> 172.21.15.28:0/810762147 conn(0x7f361c0c7720 msgr2=0x7f361c0c7b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:25.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.583+0000 7f3622a00640 1 -- 172.21.15.28:0/810762147 shutdown_connections 2024-02-21T23:04:25.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.583+0000 7f3622a00640 1 -- 172.21.15.28:0/810762147 wait complete. 2024-02-21T23:04:25.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.585+0000 7f3622a00640 1 Processor -- start 2024-02-21T23:04:25.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.586+0000 7f3622a00640 1 -- start start 2024-02-21T23:04:25.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.587+0000 7f3622a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f361c070750 0x7f361c1ee3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:25.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.587+0000 7f3622a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f361c072770 0x7f361c0a8b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:25.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.587+0000 7f3622a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f361c073360 0x7f361c0a90a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:25.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.587+0000 7f3622a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f361c155830 con 0x7f361c072770 2024-02-21T23:04:25.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.587+0000 7f3622a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f361c070b30 con 0x7f361c070750 2024-02-21T23:04:25.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.587+0000 7f3622a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f361c06b580 con 0x7f361c073360 2024-02-21T23:04:25.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.588+0000 7f3620c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f361c072770 0x7f361c0a8b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:25.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.588+0000 7f3620c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f361c072770 0x7f361c0a8b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48042/0 (socket says 172.21.15.28:48042) 2024-02-21T23:04:25.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.588+0000 7f3620c00640 1 -- 172.21.15.28:0/1088825734 learned_addr learned my addr 172.21.15.28:0/1088825734 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:25.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.588+0000 7f3622000640 1 --2- 172.21.15.28:0/1088825734 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f361c073360 0x7f361c0a90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:25.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.588+0000 7f3621600640 1 --2- 172.21.15.28:0/1088825734 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f361c070750 0x7f361c1ee3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:25.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.588+0000 7f3620c00640 1 -- 172.21.15.28:0/1088825734 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f361c073360 msgr2=0x7f361c0a90a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:25.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.588+0000 7f3620c00640 1 --2- 172.21.15.28:0/1088825734 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f361c073360 0x7f361c0a90a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:25.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.588+0000 7f3620c00640 1 -- 172.21.15.28:0/1088825734 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f361c070750 msgr2=0x7f361c1ee3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:25.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.588+0000 7f3620c00640 1 --2- 172.21.15.28:0/1088825734 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f361c070750 0x7f361c1ee3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:25.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.588+0000 7f3620c00640 1 -- 172.21.15.28:0/1088825734 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f361c1eec00 con 0x7f361c072770 2024-02-21T23:04:25.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.588+0000 7f3620c00640 1 --2- 172.21.15.28:0/1088825734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f361c072770 0x7f361c0a8b60 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f3608000c00 tx=0x7f36080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:25.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.589+0000 7f3619600640 1 -- 172.21.15.28:0/1088825734 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f360803ac60 con 0x7f361c072770 2024-02-21T23:04:25.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.589+0000 7f3619600640 1 -- 172.21.15.28:0/1088825734 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f360803ae20 con 0x7f361c072770 2024-02-21T23:04:25.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.589+0000 7f3619600640 1 -- 172.21.15.28:0/1088825734 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3608044630 con 0x7f361c072770 2024-02-21T23:04:25.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.589+0000 7f3622a00640 1 -- 172.21.15.28:0/1088825734 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f361c1eeee0 con 0x7f361c072770 2024-02-21T23:04:25.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.589+0000 7f3622a00640 1 -- 172.21.15.28:0/1088825734 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f361c1ee900 con 0x7f361c072770 2024-02-21T23:04:25.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.590+0000 7f3622a00640 1 -- 172.21.15.28:0/1088825734 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f361c04f290 con 0x7f361c072770 2024-02-21T23:04:25.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.591+0000 7f3619600640 1 -- 172.21.15.28:0/1088825734 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3608038020 con 0x7f361c072770 2024-02-21T23:04:25.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.592+0000 7f3619600640 1 --2- 172.21.15.28:0/1088825734 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f35fc06b500 0x7f35fc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:25.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.592+0000 7f3619600640 1 -- 172.21.15.28:0/1088825734 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6216+0+0 (secure 0 0 0) 0x7f36080ac3f0 con 0x7f361c072770 2024-02-21T23:04:25.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.593+0000 7f3621600640 1 --2- 172.21.15.28:0/1088825734 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f35fc06b500 0x7f35fc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:25.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.594+0000 7f3621600640 1 --2- 172.21.15.28:0/1088825734 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f35fc06b500 0x7f35fc06d9c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f361c0c6ef0 tx=0x7f3610009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:25.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.597+0000 7f3619600640 1 -- 172.21.15.28:0/1088825734 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f360803f070 con 0x7f361c072770 2024-02-21T23:04:25.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.873+0000 7f3622a00640 1 -- 172.21.15.28:0/1088825734 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f361c0a7170 con 0x7f361c072770 2024-02-21T23:04:25.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.874+0000 7f3619600640 1 -- 172.21.15.28:0/1088825734 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+4932 (secure 0 0 0) 0x7f361c0a7170 con 0x7f361c072770 2024-02-21T23:04:25.863 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:04:25.864 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":23,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4978,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/3612644344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":3612644344},{"type":"v1","addr":"172.21.15.81:6836","nonce":3612644344}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4981,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3623937361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":3623937361},{"type":"v1","addr":"172.21.15.81:6837","nonce":3623937361}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5442,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/33413357","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":33413357},{"type":"v1","addr":"172.21.15.28:6835","nonce":33413357}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:04:09.036302+0000","modified":"2024-02-21T23:04:22.712438+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4778},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4778":{"gid":4778,"name":"c","rank":0,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.28:6834/2827878900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2827878900},{"type":"v1","addr":"172.21.15.28:6834","nonce":2827878900}]},"join_fscid":2,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-02-21T23:04:25.864 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 23 2024-02-21T23:04:25.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.878+0000 7f3622a00640 1 -- 172.21.15.28:0/1088825734 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f35fc06b500 msgr2=0x7f35fc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:25.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.878+0000 7f3622a00640 1 --2- 172.21.15.28:0/1088825734 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f35fc06b500 0x7f35fc06d9c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f361c0c6ef0 tx=0x7f3610009040 comp rx=0 tx=0).stop 2024-02-21T23:04:25.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.878+0000 7f3622a00640 1 -- 172.21.15.28:0/1088825734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f361c072770 msgr2=0x7f361c0a8b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:25.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.878+0000 7f3622a00640 1 --2- 172.21.15.28:0/1088825734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f361c072770 0x7f361c0a8b60 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f3608000c00 tx=0x7f36080046c0 comp rx=0 tx=0).stop 2024-02-21T23:04:25.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.878+0000 7f3622a00640 1 -- 172.21.15.28:0/1088825734 shutdown_connections 2024-02-21T23:04:25.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.878+0000 7f3622a00640 1 --2- 172.21.15.28:0/1088825734 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f35fc06b500 0x7f35fc06d9c0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:25.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.878+0000 7f3622a00640 1 --2- 172.21.15.28:0/1088825734 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f361c073360 0x7f361c0a90a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:25.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.878+0000 7f3622a00640 1 --2- 172.21.15.28:0/1088825734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f361c072770 0x7f361c0a8b60 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:25.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.878+0000 7f3622a00640 1 --2- 172.21.15.28:0/1088825734 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f361c070750 0x7f361c1ee3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:25.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.878+0000 7f3622a00640 1 -- 172.21.15.28:0/1088825734 >> 172.21.15.28:0/1088825734 conn(0x7f361c0c7720 msgr2=0x7f361c0c7b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:25.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.878+0000 7f3622a00640 1 -- 172.21.15.28:0/1088825734 shutdown_connections 2024-02-21T23:04:25.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:25.878+0000 7f3622a00640 1 -- 172.21.15.28:0/1088825734 wait complete. 2024-02-21T23:04:26.288 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:04:09.036302+0000', 'modified': '2024-02-21T23:04:22.712438+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 39, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4778}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4778': {'gid': 4778, 'name': 'c', 'rank': 0, 'incarnation': 20, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.28:6834/2827878900', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 2827878900}, {'type': 'v1', 'addr': '172.21.15.28:6834', 'nonce': 2827878900}]}, 'join_fscid': 2, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:04:26.288 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-21T23:04:26.288 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-02-21T23:04:26.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.538+0000 7face1800640 1 Processor -- start 2024-02-21T23:04:26.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.539+0000 7face1800640 1 -- start start 2024-02-21T23:04:26.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.540+0000 7face1800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7facdc150450 0x7facdc06f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:26.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.540+0000 7face1800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7facdc0aaec0 0x7facdc06f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:26.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.540+0000 7face1800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facdc1536a0 con 0x7facdc0c3f30 2024-02-21T23:04:26.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.540+0000 7face1800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facdc0c3a20 con 0x7facdc0aaec0 2024-02-21T23:04:26.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.540+0000 7face1800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facdc0c3230 con 0x7facdc150450 2024-02-21T23:04:26.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.541+0000 7facdb400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7facdc0aaec0 0x7facdc06f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:26.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.541+0000 7facdb400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7facdc0aaec0 0x7facdc06f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48058/0 (socket says 172.21.15.28:48058) 2024-02-21T23:04:26.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.541+0000 7facdb400640 1 -- 172.21.15.28:0/2755616223 learned_addr learned my addr 172.21.15.28:0/2755616223 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:26.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.541+0000 7facdbe00640 1 --2- 172.21.15.28:0/2755616223 >> v2:172.21.15.81:3300/0 conn(0x7facdc150450 0x7facdc06f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:26.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.541+0000 7facdb400640 1 -- 172.21.15.28:0/2755616223 >> v2:172.21.15.81:3300/0 conn(0x7facdc150450 msgr2=0x7facdc06f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:26.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.541+0000 7facdb400640 1 --2- 172.21.15.28:0/2755616223 >> v2:172.21.15.81:3300/0 conn(0x7facdc150450 0x7facdc06f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:26.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.541+0000 7facdb400640 1 -- 172.21.15.28:0/2755616223 >> v1:172.21.15.28:6789/0 conn(0x7facdc0c3f30 legacy=0x7facdc0c43a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:26.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.541+0000 7facdb400640 1 -- 172.21.15.28:0/2755616223 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facdc153960 con 0x7facdc0aaec0 2024-02-21T23:04:26.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.542+0000 7facdb400640 1 --2- 172.21.15.28:0/2755616223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facdc0aaec0 0x7facdc06f670 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7facdc0c2ef0 tx=0x7facc802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecce3cd17cc08e19 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:26.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.543+0000 7facdaa00640 1 -- 172.21.15.28:0/2755616223 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc8034070 con 0x7facdc0aaec0 2024-02-21T23:04:26.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.543+0000 7facdaa00640 1 -- 172.21.15.28:0/2755616223 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facc8030860 con 0x7facdc0aaec0 2024-02-21T23:04:26.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.543+0000 7facdaa00640 1 -- 172.21.15.28:0/2755616223 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc8030c30 con 0x7facdc0aaec0 2024-02-21T23:04:26.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.543+0000 7face1800640 1 -- 172.21.15.28:0/2755616223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facdc0aaec0 msgr2=0x7facdc06f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:26.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.543+0000 7face1800640 1 --2- 172.21.15.28:0/2755616223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facdc0aaec0 0x7facdc06f670 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7facdc0c2ef0 tx=0x7facc802b9a0 comp rx=0 tx=0).stop 2024-02-21T23:04:26.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.544+0000 7face1800640 1 -- 172.21.15.28:0/2755616223 shutdown_connections 2024-02-21T23:04:26.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.544+0000 7face1800640 1 --2- 172.21.15.28:0/2755616223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facdc0aaec0 0x7facdc06f670 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:26.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.544+0000 7face1800640 1 --2- 172.21.15.28:0/2755616223 >> v2:172.21.15.81:3300/0 conn(0x7facdc150450 0x7facdc06f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:26.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.544+0000 7face1800640 1 -- 172.21.15.28:0/2755616223 >> 172.21.15.28:0/2755616223 conn(0x7facdc0aa3d0 msgr2=0x7facdc0aa7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:26.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.544+0000 7face1800640 1 -- 172.21.15.28:0/2755616223 shutdown_connections 2024-02-21T23:04:26.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.544+0000 7face1800640 1 -- 172.21.15.28:0/2755616223 wait complete. 2024-02-21T23:04:26.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.545+0000 7face1800640 1 Processor -- start 2024-02-21T23:04:26.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.546+0000 7face1800640 1 -- start start 2024-02-21T23:04:26.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.546+0000 7face1800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7facdc0aaec0 0x7facdc14a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:26.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.546+0000 7face1800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facdc0c3f30 0x7facdc144c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:26.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.546+0000 7face1800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7facdc150450 0x7facdc1451b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:26.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.547+0000 7face1800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facdc0c4ab0 con 0x7facdc0c3f30 2024-02-21T23:04:26.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.547+0000 7face1800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facdc06fbb0 con 0x7facdc150450 2024-02-21T23:04:26.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.547+0000 7face1800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facdc0c4c30 con 0x7facdc0aaec0 2024-02-21T23:04:26.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.547+0000 7facdb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facdc0c3f30 0x7facdc144c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:26.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.547+0000 7facdb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facdc0c3f30 0x7facdc144c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48072/0 (socket says 172.21.15.28:48072) 2024-02-21T23:04:26.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.547+0000 7facdb400640 1 -- 172.21.15.28:0/3204353978 learned_addr learned my addr 172.21.15.28:0/3204353978 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:26.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.547+0000 7facdb400640 1 -- 172.21.15.28:0/3204353978 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7facdc0aaec0 msgr2=0x7facdc14a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:26.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.547+0000 7facdb400640 1 --2- 172.21.15.28:0/3204353978 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7facdc0aaec0 0x7facdc14a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:26.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.547+0000 7facdb400640 1 -- 172.21.15.28:0/3204353978 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7facdc150450 msgr2=0x7facdc1451b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:26.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.547+0000 7face0e00640 1 --2- 172.21.15.28:0/3204353978 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7facdc150450 0x7facdc1451b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:26.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.547+0000 7facdb400640 1 --2- 172.21.15.28:0/3204353978 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7facdc150450 0x7facdc1451b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:26.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.547+0000 7facdb400640 1 -- 172.21.15.28:0/3204353978 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facdc14c840 con 0x7facdc0c3f30 2024-02-21T23:04:26.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.547+0000 7face0e00640 1 --2- 172.21.15.28:0/3204353978 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7facdc150450 0x7facdc1451b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:04:26.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.548+0000 7facdb400640 1 --2- 172.21.15.28:0/3204353978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facdc0c3f30 0x7facdc144c70 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7facc802b9e0 tx=0x7facc80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:26.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.548+0000 7facd3e00640 1 -- 172.21.15.28:0/3204353978 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc8034040 con 0x7facdc0c3f30 2024-02-21T23:04:26.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.548+0000 7facd3e00640 1 -- 172.21.15.28:0/3204353978 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facc8039d90 con 0x7facdc0c3f30 2024-02-21T23:04:26.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.548+0000 7face1800640 1 -- 172.21.15.28:0/3204353978 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facdc14cb20 con 0x7facdc0c3f30 2024-02-21T23:04:26.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.548+0000 7facd3e00640 1 -- 172.21.15.28:0/3204353978 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc8043610 con 0x7facdc0c3f30 2024-02-21T23:04:26.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.548+0000 7face1800640 1 -- 172.21.15.28:0/3204353978 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facdc14d0e0 con 0x7facdc0c3f30 2024-02-21T23:04:26.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.551+0000 7facd3e00640 1 -- 172.21.15.28:0/3204353978 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7facc804a060 con 0x7facdc0c3f30 2024-02-21T23:04:26.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.552+0000 7facd3e00640 1 --2- 172.21.15.28:0/3204353978 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7facbc06b4d0 0x7facbc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:26.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.552+0000 7facd3e00640 1 -- 172.21.15.28:0/3204353978 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6216+0+0 (secure 0 0 0) 0x7facc80ac5f0 con 0x7facdc0c3f30 2024-02-21T23:04:26.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.553+0000 7facdbe00640 1 --2- 172.21.15.28:0/3204353978 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7facbc06b4d0 0x7facbc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:26.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.554+0000 7facdbe00640 1 --2- 172.21.15.28:0/3204353978 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7facbc06b4d0 0x7facbc06d990 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7faccc0098f0 tx=0x7faccc008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:26.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.554+0000 7face5c04740 1 Processor -- start 2024-02-21T23:04:26.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.554+0000 7face5c04740 1 -- start start 2024-02-21T23:04:26.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.556+0000 7face5c04740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55d37e38e190 0x55d37e34a820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:26.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.556+0000 7face5c04740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d37e34ad60 0x55d37e399330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:26.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.556+0000 7face5c04740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d37e38e570 0x55d37e399870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:26.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.556+0000 7face5c04740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d37e2b4470 con 0x55d37e38e570 2024-02-21T23:04:26.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.556+0000 7face5c04740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d37e25daf0 con 0x55d37e34ad60 2024-02-21T23:04:26.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.556+0000 7face5c04740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d37e2b0510 con 0x55d37e38e190 2024-02-21T23:04:26.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.556+0000 7face0e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d37e38e570 0x55d37e399870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:26.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.556+0000 7face0e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d37e38e570 0x55d37e399870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48080/0 (socket says 172.21.15.28:48080) 2024-02-21T23:04:26.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.556+0000 7face0e00640 1 -- 172.21.15.28:0/3553573520 learned_addr learned my addr 172.21.15.28:0/3553573520 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:26.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.556+0000 7facdbe00640 1 --2- 172.21.15.28:0/3553573520 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55d37e38e190 0x55d37e34a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:26.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.557+0000 7facdb400640 1 --2- 172.21.15.28:0/3553573520 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d37e34ad60 0x55d37e399330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:26.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.557+0000 7face0e00640 1 -- 172.21.15.28:0/3553573520 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55d37e38e190 msgr2=0x55d37e34a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:26.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.557+0000 7face0e00640 1 --2- 172.21.15.28:0/3553573520 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55d37e38e190 0x55d37e34a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:26.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.557+0000 7face0e00640 1 -- 172.21.15.28:0/3553573520 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d37e34ad60 msgr2=0x55d37e399330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:26.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.557+0000 7face0e00640 1 --2- 172.21.15.28:0/3553573520 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d37e34ad60 0x55d37e399330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:26.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.557+0000 7face0e00640 1 -- 172.21.15.28:0/3553573520 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d37e264d90 con 0x55d37e38e570 2024-02-21T23:04:26.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.557+0000 7face0e00640 1 --2- 172.21.15.28:0/3553573520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d37e38e570 0x55d37e399870 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7facd4001da0 tx=0x7facd400d150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e55ed225d2f5acc0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:26.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.557+0000 7facd0c00640 1 -- 172.21.15.28:0/3553573520 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd4018070 con 0x55d37e38e570 2024-02-21T23:04:26.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.558+0000 7facd0c00640 1 -- 172.21.15.28:0/3553573520 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facd4003da0 con 0x55d37e38e570 2024-02-21T23:04:26.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.558+0000 7facd0c00640 1 -- 172.21.15.28:0/3553573520 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd400dba0 con 0x55d37e38e570 2024-02-21T23:04:26.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.558+0000 7face5c04740 1 -- 172.21.15.28:0/3553573520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d37e38e570 msgr2=0x55d37e399870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:26.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.558+0000 7face5c04740 1 --2- 172.21.15.28:0/3553573520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d37e38e570 0x55d37e399870 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7facd4001da0 tx=0x7facd400d150 comp rx=0 tx=0).stop 2024-02-21T23:04:26.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.559+0000 7face5c04740 1 -- 172.21.15.28:0/3553573520 shutdown_connections 2024-02-21T23:04:26.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.559+0000 7face5c04740 1 --2- 172.21.15.28:0/3553573520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d37e38e570 0x55d37e399870 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:26.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.559+0000 7face5c04740 1 --2- 172.21.15.28:0/3553573520 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d37e34ad60 0x55d37e399330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:26.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.559+0000 7face5c04740 1 --2- 172.21.15.28:0/3553573520 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55d37e38e190 0x55d37e34a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:26.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.559+0000 7face5c04740 1 -- 172.21.15.28:0/3553573520 >> 172.21.15.28:0/3553573520 conn(0x55d37e34c970 msgr2=0x55d37e371380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:26.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.559+0000 7face5c04740 1 -- 172.21.15.28:0/3553573520 shutdown_connections 2024-02-21T23:04:26.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.559+0000 7face5c04740 1 -- 172.21.15.28:0/3553573520 wait complete. 2024-02-21T23:04:26.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.559+0000 7face5c04740 1 -- start start 2024-02-21T23:04:26.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.560+0000 7face5c04740 1 Processor -- start 2024-02-21T23:04:26.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.560+0000 7face5c04740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d37e38e190 0x55d37e3a4090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:26.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.560+0000 7face5c04740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d37e38e570 0x55d37e36ff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:26.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.560+0000 7face5c04740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55d37e371ad0 0x55d37e36cc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:26.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.561+0000 7face5c04740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d37e38ec50 con 0x55d37e38e190 2024-02-21T23:04:26.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.561+0000 7face5c04740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d37e253040 con 0x55d37e38e570 2024-02-21T23:04:26.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.561+0000 7face5c04740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d37e350510 con 0x55d37e371ad0 2024-02-21T23:04:26.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.561+0000 7face5c04740 10 client.0 _ll_register_callbacks cb 0x55d37e270920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:04:26.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.561+0000 7facdbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d37e38e190 0x55d37e3a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:26.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.561+0000 7facdbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d37e38e190 0x55d37e3a4090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48092/0 (socket says 172.21.15.28:48092) 2024-02-21T23:04:26.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.561+0000 7facdbe00640 1 -- 172.21.15.28:0/406429587 learned_addr learned my addr 172.21.15.28:0/406429587 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:26.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.561+0000 7facdb400640 1 --2- 172.21.15.28:0/406429587 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d37e38e570 0x55d37e36ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:26.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.561+0000 7face0e00640 1 --2- 172.21.15.28:0/406429587 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55d37e371ad0 0x55d37e36cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:26.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.561+0000 7facdbe00640 1 -- 172.21.15.28:0/406429587 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55d37e371ad0 msgr2=0x55d37e36cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:26.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.561+0000 7facdbe00640 1 --2- 172.21.15.28:0/406429587 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55d37e371ad0 0x55d37e36cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:26.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.561+0000 7facdbe00640 1 -- 172.21.15.28:0/406429587 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d37e38e570 msgr2=0x55d37e36ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:26.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.561+0000 7facdbe00640 1 --2- 172.21.15.28:0/406429587 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d37e38e570 0x55d37e36ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:26.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.561+0000 7facdbe00640 1 -- 172.21.15.28:0/406429587 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d37e272aa0 con 0x55d37e38e190 2024-02-21T23:04:26.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.562+0000 7facdbe00640 1 --2- 172.21.15.28:0/406429587 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d37e38e190 0x55d37e3a4090 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7faccc00dee0 tx=0x7faccc006e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31b736bf1ffa469 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:26.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.562+0000 7facc5600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:04:26.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.562+0000 7facc5600640 1 -- 172.21.15.28:0/406429587 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faccc014e40 con 0x55d37e38e190 2024-02-21T23:04:26.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.562+0000 7face5c04740 1 -- 172.21.15.28:0/406429587 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7facd4003da0 con 0x55d37e38e190 2024-02-21T23:04:26.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.562+0000 7facc5600640 1 -- 172.21.15.28:0/406429587 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(41..41 src has 1..41) v4 ==== 6216+0+0 (secure 0 0 0) 0x7faccc00f040 con 0x55d37e38e190 2024-02-21T23:04:26.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.563+0000 7facc5600640 1 -- 172.21.15.28:0/406429587 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7faccc005990 con 0x55d37e38e190 2024-02-21T23:04:26.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.563+0000 7face5c04740 10 client.5496 fetch_fsmap learned FSMap version 23 2024-02-21T23:04:26.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.563+0000 7face5c04740 1 -- 172.21.15.28:0/406429587 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=2+}) v3 -- 0x55d37dfc92d0 con 0x55d37e38e190 2024-02-21T23:04:26.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.563+0000 7facc5600640 1 -- 172.21.15.28:0/406429587 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facd4003da0 con 0x55d37e38e190 2024-02-21T23:04:26.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.563+0000 7facc5600640 1 -- 172.21.15.28:0/406429587 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 23) v1 ==== 3294+0+0 (secure 0 0 0) 0x7faccc013070 con 0x55d37e38e190 2024-02-21T23:04:26.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.563+0000 7face5c04740 10 client.5496 fetch_fsmap finished waiting for FSMap version 23 2024-02-21T23:04:26.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.563+0000 7face5c04740 10 client.5496 resolve_mds: resolved name 'c' to daemon mds.c 2024-02-21T23:04:26.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.563+0000 7face5c04740 20 client.5496 populate_metadata read hostname 'smithi028' 2024-02-21T23:04:26.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.563+0000 7face5c04740 1 --2- 172.21.15.28:0/406429587 >> [v2:172.21.15.28:6832/2827878900,v1:172.21.15.28:6834/2827878900] conn(0x55d37e3a0720 0x55d37e3a0b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:26.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.563+0000 7face5c04740 4 client.5496 mds_command: new command op to 4778 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-21T23:04:26.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.563+0000 7face5c04740 1 -- 172.21.15.28:0/406429587 --> [v2:172.21.15.28:6832/2827878900,v1:172.21.15.28:6834/2827878900] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d37e2c19b0 con 0x55d37e3a0720 2024-02-21T23:04:26.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.563+0000 7facdb400640 1 --2- 172.21.15.28:0/406429587 >> [v2:172.21.15.28:6832/2827878900,v1:172.21.15.28:6834/2827878900] conn(0x55d37e3a0720 0x55d37e3a0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:26.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.564+0000 7facdb400640 1 --2- 172.21.15.28:0/406429587 >> [v2:172.21.15.28:6832/2827878900,v1:172.21.15.28:6834/2827878900] conn(0x55d37e3a0720 0x55d37e3a0b00 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c5d269ad5ae96e69 server_cookie=f6bf953ddcc3ad29 in_seq=0 out_seq=0 2024-02-21T23:04:26.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.564+0000 7facc5600640 10 client.5496 ms_handle_connect on v2:172.21.15.28:6832/2827878900 2024-02-21T23:04:26.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.565+0000 7facc5600640 1 -- 172.21.15.28:0/406429587 <== mds.0 v2:172.21.15.28:6832/2827878900 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55d37e2c19b0 con 0x55d37e3a0720 2024-02-21T23:04:26.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.565+0000 7facc5600640 10 client.5496 handle_command_reply: tid=0 2024-02-21T23:04:26.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.565+0000 7face5c04740 1 client.5496 shutdown 2024-02-21T23:04:26.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.565+0000 7face5c04740 1 -- 172.21.15.28:0/406429587 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d37e38e190 msgr2=0x55d37e3a4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:26.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.565+0000 7face5c04740 1 --2- 172.21.15.28:0/406429587 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d37e38e190 0x55d37e3a4090 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7faccc00dee0 tx=0x7faccc006e10 comp rx=0 tx=0).stop 2024-02-21T23:04:26.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.565+0000 7face5c04740 1 -- 172.21.15.28:0/406429587 shutdown_connections 2024-02-21T23:04:26.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.565+0000 7face5c04740 1 --2- 172.21.15.28:0/406429587 >> [v2:172.21.15.28:6832/2827878900,v1:172.21.15.28:6834/2827878900] conn(0x55d37e3a0720 0x55d37e3a0b00 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:26.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.565+0000 7face5c04740 1 --2- 172.21.15.28:0/406429587 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55d37e371ad0 0x55d37e36cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:26.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.565+0000 7facc5600640 0 client.5496 ms_handle_reset on v2:172.21.15.28:6832/2827878900 2024-02-21T23:04:26.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.565+0000 7face5c04740 1 --2- 172.21.15.28:0/406429587 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d37e38e570 0x55d37e36ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:26.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.565+0000 7face5c04740 1 --2- 172.21.15.28:0/406429587 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d37e38e190 0x55d37e3a4090 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:26.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.565+0000 7face5c04740 1 -- 172.21.15.28:0/406429587 >> 172.21.15.28:0/406429587 conn(0x55d37e34ad60 msgr2=0x55d37e3495f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:26.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.565+0000 7face5c04740 1 -- 172.21.15.28:0/406429587 shutdown_connections 2024-02-21T23:04:26.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.565+0000 7face5c04740 1 -- 172.21.15.28:0/406429587 wait complete. 2024-02-21T23:04:26.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.566+0000 7face5c04740 20 client.5496 trim_cache size 0 max 16384 2024-02-21T23:04:26.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.584+0000 7face5c04740 1 Processor -- start 2024-02-21T23:04:26.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.584+0000 7face5c04740 1 -- start start 2024-02-21T23:04:26.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.585+0000 7face5c04740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55d37e371ad0 0x55d37e3a3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:26.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.585+0000 7face5c04740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d37e38e190 0x55d37e36ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:26.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.585+0000 7face5c04740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d37e38e570 0x55d37e36ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:26.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.585+0000 7face5c04740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facdc06da20 con 0x55d37e38e190 2024-02-21T23:04:26.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.585+0000 7face5c04740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facdc1514c0 con 0x55d37e38e570 2024-02-21T23:04:26.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.585+0000 7face5c04740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facdc06d420 con 0x55d37e371ad0 2024-02-21T23:04:26.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.585+0000 7facdb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d37e38e190 0x55d37e36ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:26.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.585+0000 7facdb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d37e38e190 0x55d37e36ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48094/0 (socket says 172.21.15.28:48094) 2024-02-21T23:04:26.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.585+0000 7facdb400640 1 -- 172.21.15.28:0/2602449756 learned_addr learned my addr 172.21.15.28:0/2602449756 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:26.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.586+0000 7facdb400640 1 -- 172.21.15.28:0/2602449756 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55d37e371ad0 msgr2=0x55d37e3a3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:26.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.586+0000 7facdb400640 1 --2- 172.21.15.28:0/2602449756 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55d37e371ad0 0x55d37e3a3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:26.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.586+0000 7facdb400640 1 -- 172.21.15.28:0/2602449756 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d37e38e570 msgr2=0x55d37e36ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:26.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.586+0000 7facdb400640 1 --2- 172.21.15.28:0/2602449756 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d37e38e570 0x55d37e36ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:26.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.586+0000 7facdb400640 1 -- 172.21.15.28:0/2602449756 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d37e370e70 con 0x55d37e38e190 2024-02-21T23:04:26.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.586+0000 7facdb400640 1 --2- 172.21.15.28:0/2602449756 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d37e38e190 0x55d37e36ea50 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7facc807aa60 tx=0x7facc80ac040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76dfad0852352023 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:26.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.586+0000 7facc7e00640 1 -- 172.21.15.28:0/2602449756 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc8092030 con 0x55d37e38e190 2024-02-21T23:04:26.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.586+0000 7facc7e00640 1 -- 172.21.15.28:0/2602449756 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facc807ec20 con 0x55d37e38e190 2024-02-21T23:04:26.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.586+0000 7facc7e00640 1 -- 172.21.15.28:0/2602449756 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc80831e0 con 0x55d37e38e190 2024-02-21T23:04:26.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.587+0000 7face5c04740 1 -- 172.21.15.28:0/2602449756 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d37e38e190 msgr2=0x55d37e36ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:26.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.587+0000 7face5c04740 1 --2- 172.21.15.28:0/2602449756 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d37e38e190 0x55d37e36ea50 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7facc807aa60 tx=0x7facc80ac040 comp rx=0 tx=0).stop 2024-02-21T23:04:26.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.587+0000 7face5c04740 1 -- 172.21.15.28:0/2602449756 shutdown_connections 2024-02-21T23:04:26.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.587+0000 7face5c04740 1 --2- 172.21.15.28:0/2602449756 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d37e38e570 0x55d37e36ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:26.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.587+0000 7face5c04740 1 --2- 172.21.15.28:0/2602449756 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d37e38e190 0x55d37e36ea50 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:26.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.587+0000 7face5c04740 1 --2- 172.21.15.28:0/2602449756 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55d37e371ad0 0x55d37e3a3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:26.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.587+0000 7face5c04740 1 -- 172.21.15.28:0/2602449756 >> 172.21.15.28:0/2602449756 conn(0x55d37e34ad60 msgr2=0x55d37e39dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:26.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.587+0000 7face5c04740 1 -- 172.21.15.28:0/2602449756 shutdown_connections 2024-02-21T23:04:26.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.587+0000 7face5c04740 1 -- 172.21.15.28:0/2602449756 wait complete. 2024-02-21T23:04:26.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.588+0000 7face5c04740 1 -- start start 2024-02-21T23:04:26.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.588+0000 7face5c04740 1 Processor -- start 2024-02-21T23:04:26.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.589+0000 7face5c04740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d37e371ad0 0x55d37e36bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:26.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.589+0000 7face5c04740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d37e38e190 0x55d37e403290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:26.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.589+0000 7face5c04740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55d37e38e570 0x55d37e3fcf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:26.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.589+0000 7face5c04740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d37e3757d0 con 0x55d37e371ad0 2024-02-21T23:04:26.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.589+0000 7face5c04740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d37e36d8c0 con 0x55d37e38e190 2024-02-21T23:04:26.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.589+0000 7face5c04740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d37e371520 con 0x55d37e38e570 2024-02-21T23:04:26.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.589+0000 7face5c04740 10 client.0 _ll_register_callbacks cb 0x7faccc00db00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:04:26.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.589+0000 7facdbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d37e371ad0 0x55d37e36bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:26.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.589+0000 7facdbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d37e371ad0 0x55d37e36bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48104/0 (socket says 172.21.15.28:48104) 2024-02-21T23:04:26.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.589+0000 7facdbe00640 1 -- 172.21.15.28:0/3914245042 learned_addr learned my addr 172.21.15.28:0/3914245042 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:26.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.589+0000 7facdb400640 1 --2- 172.21.15.28:0/3914245042 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d37e38e190 0x55d37e403290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:26.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.589+0000 7face0e00640 1 --2- 172.21.15.28:0/3914245042 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55d37e38e570 0x55d37e3fcf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:26.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.590+0000 7facdbe00640 1 -- 172.21.15.28:0/3914245042 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55d37e38e570 msgr2=0x55d37e3fcf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:26.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.590+0000 7facdbe00640 1 --2- 172.21.15.28:0/3914245042 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55d37e38e570 0x55d37e3fcf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:26.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.590+0000 7facdbe00640 1 -- 172.21.15.28:0/3914245042 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d37e38e190 msgr2=0x55d37e403290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:26.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.590+0000 7facdbe00640 1 --2- 172.21.15.28:0/3914245042 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d37e38e190 0x55d37e403290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:26.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.590+0000 7facdbe00640 1 -- 172.21.15.28:0/3914245042 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d37e0936b0 con 0x55d37e371ad0 2024-02-21T23:04:26.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.590+0000 7facdb400640 1 --2- 172.21.15.28:0/3914245042 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d37e38e190 0x55d37e403290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:04:26.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.590+0000 7facdbe00640 1 --2- 172.21.15.28:0/3914245042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d37e371ad0 0x55d37e36bf30 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7faccc00d810 tx=0x7faccc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9603520a8ff37bf4 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:26.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.590+0000 7facc5600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:04:26.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.590+0000 7facc5600640 1 -- 172.21.15.28:0/3914245042 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faccc035020 con 0x55d37e371ad0 2024-02-21T23:04:26.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.590+0000 7facc5600640 1 -- 172.21.15.28:0/3914245042 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(41..41 src has 1..41) v4 ==== 6216+0+0 (secure 0 0 0) 0x7faccc012800 con 0x55d37e371ad0 2024-02-21T23:04:26.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.590+0000 7face5c04740 1 -- 172.21.15.28:0/3914245042 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7facc807ec20 con 0x55d37e371ad0 2024-02-21T23:04:26.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.591+0000 7facc5600640 1 -- 172.21.15.28:0/3914245042 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7faccc03f050 con 0x55d37e371ad0 2024-02-21T23:04:26.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.591+0000 7face5c04740 10 client.5502 fetch_fsmap learned FSMap version 23 2024-02-21T23:04:26.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.591+0000 7face5c04740 1 -- 172.21.15.28:0/3914245042 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=2+}) v3 -- 0x55d37e371100 con 0x55d37e371ad0 2024-02-21T23:04:26.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.591+0000 7facc5600640 1 -- 172.21.15.28:0/3914245042 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facc807ec20 con 0x55d37e371ad0 2024-02-21T23:04:26.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.591+0000 7facc5600640 1 -- 172.21.15.28:0/3914245042 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 23) v1 ==== 3294+0+0 (secure 0 0 0) 0x7faccc042020 con 0x55d37e371ad0 2024-02-21T23:04:26.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.591+0000 7face5c04740 10 client.5502 fetch_fsmap finished waiting for FSMap version 23 2024-02-21T23:04:26.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.591+0000 7face5c04740 10 client.5502 resolve_mds: resolved name 'c' to daemon mds.c 2024-02-21T23:04:26.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.591+0000 7face5c04740 20 client.5502 populate_metadata read hostname 'smithi028' 2024-02-21T23:04:26.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.591+0000 7face5c04740 1 --2- 172.21.15.28:0/3914245042 >> [v2:172.21.15.28:6832/2827878900,v1:172.21.15.28:6834/2827878900] conn(0x55d37e381fd0 0x55d37e3823b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:26.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.591+0000 7face5c04740 4 client.5502 mds_command: new command op to 4778 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-21T23:04:26.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.591+0000 7face5c04740 1 -- 172.21.15.28:0/3914245042 --> [v2:172.21.15.28:6832/2827878900,v1:172.21.15.28:6834/2827878900] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55d37e2bbd00 con 0x55d37e381fd0 2024-02-21T23:04:26.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.591+0000 7facdb400640 1 --2- 172.21.15.28:0/3914245042 >> [v2:172.21.15.28:6832/2827878900,v1:172.21.15.28:6834/2827878900] conn(0x55d37e381fd0 0x55d37e3823b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:26.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.592+0000 7facdb400640 1 --2- 172.21.15.28:0/3914245042 >> [v2:172.21.15.28:6832/2827878900,v1:172.21.15.28:6834/2827878900] conn(0x55d37e381fd0 0x55d37e3823b0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb8bc90b5e4c1f8 server_cookie=5e612343fd216510 in_seq=0 out_seq=0 2024-02-21T23:04:26.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.592+0000 7facc5600640 10 client.5502 ms_handle_connect on v2:172.21.15.28:6832/2827878900 2024-02-21T23:04:26.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.592+0000 7facc5600640 1 -- 172.21.15.28:0/3914245042 <== mds.0 v2:172.21.15.28:6832/2827878900 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55d37e2bbd00 con 0x55d37e381fd0 2024-02-21T23:04:26.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.592+0000 7facc5600640 10 client.5502 handle_command_reply: tid=0 2024-02-21T23:04:26.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.592+0000 7face5c04740 1 client.5502 shutdown 2024-02-21T23:04:26.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.592+0000 7face5c04740 1 -- 172.21.15.28:0/3914245042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d37e371ad0 msgr2=0x55d37e36bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:26.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.592+0000 7face5c04740 1 --2- 172.21.15.28:0/3914245042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d37e371ad0 0x55d37e36bf30 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7faccc00d810 tx=0x7faccc0046c0 comp rx=0 tx=0).stop 2024-02-21T23:04:26.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.593+0000 7face5c04740 1 -- 172.21.15.28:0/3914245042 shutdown_connections 2024-02-21T23:04:26.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.593+0000 7face5c04740 1 --2- 172.21.15.28:0/3914245042 >> [v2:172.21.15.28:6832/2827878900,v1:172.21.15.28:6834/2827878900] conn(0x55d37e381fd0 0x55d37e3823b0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:26.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.593+0000 7face5c04740 1 --2- 172.21.15.28:0/3914245042 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55d37e38e570 0x55d37e3fcf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:26.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.593+0000 7facc5600640 0 client.5502 ms_handle_reset on v2:172.21.15.28:6832/2827878900 2024-02-21T23:04:26.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.593+0000 7face5c04740 1 --2- 172.21.15.28:0/3914245042 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55d37e38e190 0x55d37e403290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:26.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.593+0000 7face5c04740 1 --2- 172.21.15.28:0/3914245042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d37e371ad0 0x55d37e36bf30 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:26.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.593+0000 7face5c04740 1 -- 172.21.15.28:0/3914245042 >> 172.21.15.28:0/3914245042 conn(0x55d37e34ad60 msgr2=0x55d37e345d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:26.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.593+0000 7face5c04740 1 -- 172.21.15.28:0/3914245042 shutdown_connections 2024-02-21T23:04:26.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.593+0000 7face5c04740 1 -- 172.21.15.28:0/3914245042 wait complete. 2024-02-21T23:04:26.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.593+0000 7face5c04740 20 client.5502 trim_cache size 0 max 16384 2024-02-21T23:04:26.587 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-21T23:04:26.587 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac", 2024-02-21T23:04:26.587 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-02-21T23:04:26.587 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4778, 2024-02-21T23:04:26.587 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-02-21T23:04:26.587 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-02-21T23:04:26.587 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-02-21T23:04:26.587 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 5.9000014199999997, 2024-02-21T23:04:26.588 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 23, 2024-02-21T23:04:26.588 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 41, 2024-02-21T23:04:26.588 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 39, 2024-02-21T23:04:26.588 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 26.021773370999998 2024-02-21T23:04:26.588 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-21T23:04:26.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.594+0000 7face1800640 1 -- 172.21.15.28:0/3204353978 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7facbc06b4d0 msgr2=0x7facbc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:26.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.594+0000 7face1800640 1 --2- 172.21.15.28:0/3204353978 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7facbc06b4d0 0x7facbc06d990 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7faccc0098f0 tx=0x7faccc008040 comp rx=0 tx=0).stop 2024-02-21T23:04:26.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.594+0000 7face1800640 1 -- 172.21.15.28:0/3204353978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facdc0c3f30 msgr2=0x7facdc144c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:26.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.594+0000 7face1800640 1 --2- 172.21.15.28:0/3204353978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facdc0c3f30 0x7facdc144c70 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7facc802b9e0 tx=0x7facc80046c0 comp rx=0 tx=0).stop 2024-02-21T23:04:26.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.594+0000 7face1800640 1 -- 172.21.15.28:0/3204353978 shutdown_connections 2024-02-21T23:04:26.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.594+0000 7face1800640 1 --2- 172.21.15.28:0/3204353978 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7facbc06b4d0 0x7facbc06d990 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:26.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.594+0000 7face1800640 1 --2- 172.21.15.28:0/3204353978 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7facdc150450 0x7facdc1451b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:26.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.594+0000 7face1800640 1 --2- 172.21.15.28:0/3204353978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facdc0c3f30 0x7facdc144c70 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:26.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.594+0000 7face1800640 1 --2- 172.21.15.28:0/3204353978 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7facdc0aaec0 0x7facdc14a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:26.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.594+0000 7face1800640 1 -- 172.21.15.28:0/3204353978 >> 172.21.15.28:0/3204353978 conn(0x7facdc0aa3d0 msgr2=0x7facdc0c7970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:26.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.594+0000 7face1800640 1 -- 172.21.15.28:0/3204353978 shutdown_connections 2024-02-21T23:04:26.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:26.594+0000 7face1800640 1 -- 172.21.15.28:0/3204353978 wait complete. 2024-02-21T23:04:26.600 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 2, "epoch": 23, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-02-21T23:04:09.036302+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 23, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4778": { "addr": "172.21.15.28:6834/2827878900", "addrs": { "addrvec": [ { "addr": "172.21.15.28:6832", "nonce": 2827878900, "type": "v2" }, { "addr": "172.21.15.28:6834", "nonce": 2827878900, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4778, "incarnation": 20, "join_fscid": 2, "name": "c", "rank": 0, "state": "up:active", "state_seq": 9 } }, "last_failure": 0, "last_failure_osd_epoch": 39, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 5, "modified": "2024-02-21T23:04:22.712438+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4778 } } } ], "standbys": [ { "addr": "172.21.15.81:6836/3612644344", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6834", "nonce": 3612644344, "type": "v2" }, { "addr": "172.21.15.81:6836", "nonce": 3612644344, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 12, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4978, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.81:6837/3623937361", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6835", "nonce": 3623937361, "type": "v2" }, { "addr": "172.21.15.81:6837", "nonce": 3623937361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 13, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4981, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.28:6835/33413357", "addrs": { "addrvec": [ { "addr": "172.21.15.28:6833", "nonce": 33413357, "type": "v2" }, { "addr": "172.21.15.28:6835", "nonce": 33413357, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 21, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5442, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-02-21T23:04:26.601 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c', 'state': 'up:active'}, {'join_fscid': -1, 'name': 'd'}] 2024-02-21T23:04:26.601 INFO:tasks.cephfs.test_failover:current = [{'gid': 5442, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.28:6835/33413357', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6833', 'nonce': 33413357}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 33413357}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 21}, {'gid': 4981, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.81:6837/3623937361', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6835', 'nonce': 3623937361}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3623937361}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 13}, {'gid': 4778, 'name': 'c', 'rank': 0, 'incarnation': 20, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.28:6834/2827878900', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 2827878900}, {'type': 'v1', 'addr': '172.21.15.28:6834', 'nonce': 2827878900}]}, 'join_fscid': 2, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 4978, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.81:6836/3612644344', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6834', 'nonce': 3612644344}, {'type': 'v1', 'addr': '172.21.15.81:6836', 'nonce': 3612644344}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 12}] 2024-02-21T23:04:26.601 DEBUG:tasks.cephfs.test_failover:-1 != 2 2024-02-21T23:04:26.601 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-02-21T23:04:27.345 INFO:tasks.ceph.mds.c.smithi028.stderr:2024-02-21T23:04:27.355+0000 7fc419000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-21T23:04:31.602 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:04:31.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:31.895+0000 7feb81200640 1 Processor -- start 2024-02-21T23:04:31.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:31.895+0000 7feb81200640 1 -- start start 2024-02-21T23:04:31.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:31.897+0000 7feb81200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7feb7c0aff90 0x7feb7c0c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:31.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:31.897+0000 7feb81200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feb7c0c5720 0x7feb7c15f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:31.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:31.897+0000 7feb81200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb7c0c74c0 con 0x7feb7c06dbf0 2024-02-21T23:04:31.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:31.897+0000 7feb81200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb7c0c7270 con 0x7feb7c0aff90 2024-02-21T23:04:31.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:31.897+0000 7feb81200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb7c158630 con 0x7feb7c0c5720 2024-02-21T23:04:31.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:31.897+0000 7feb7b400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7feb7c0aff90 0x7feb7c0c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:31.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:31.897+0000 7feb73e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7feb7c06dbf0 0x7feb7c158ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39918/0 (socket says 172.21.15.28:39918) 2024-02-21T23:04:31.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:31.897+0000 7feb73e00640 1 -- 172.21.15.28:0/2809183617 learned_addr learned my addr 172.21.15.28:0/2809183617 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:04:31.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:31.898+0000 7feb7be00640 1 --2- 172.21.15.28:0/2809183617 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feb7c0c5720 0x7feb7c15f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:31.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:31.898+0000 7feb7aa00640 1 -- 172.21.15.28:0/2809183617 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3636583458 0 0) 0x7feb7c0c74c0 con 0x7feb7c06dbf0 2024-02-21T23:04:31.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:31.898+0000 7feb7aa00640 1 -- 172.21.15.28:0/2809183617 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb60003520 con 0x7feb7c06dbf0 2024-02-21T23:04:31.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:31.898+0000 7feb7b400640 1 -- 172.21.15.28:0/2809183617 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feb7c0c5720 msgr2=0x7feb7c15f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:31.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:31.898+0000 7feb7b400640 1 --2- 172.21.15.28:0/2809183617 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feb7c0c5720 0x7feb7c15f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:31.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:31.898+0000 7feb7b400640 1 -- 172.21.15.28:0/2809183617 >> v1:172.21.15.28:6789/0 conn(0x7feb7c06dbf0 legacy=0x7feb7c158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:31.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:31.898+0000 7feb7b400640 1 -- 172.21.15.28:0/2809183617 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb7c0c77a0 con 0x7feb7c0aff90 2024-02-21T23:04:31.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:31.899+0000 7feb7b400640 1 --2- 172.21.15.28:0/2809183617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb7c0aff90 0x7feb7c0c51e0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7feb6c0097a0 tx=0x7feb6c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=598867b7ae8285a0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:31.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:31.900+0000 7feb7aa00640 1 -- 172.21.15.28:0/2809183617 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb6c035020 con 0x7feb7c0aff90 2024-02-21T23:04:31.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:31.900+0000 7feb7aa00640 1 -- 172.21.15.28:0/2809183617 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb6c031830 con 0x7feb7c0aff90 2024-02-21T23:04:31.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:31.900+0000 7feb7aa00640 1 -- 172.21.15.28:0/2809183617 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb6c031be0 con 0x7feb7c0aff90 2024-02-21T23:04:31.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:31.900+0000 7feb81200640 1 -- 172.21.15.28:0/2809183617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb7c0aff90 msgr2=0x7feb7c0c51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:31.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:31.900+0000 7feb81200640 1 --2- 172.21.15.28:0/2809183617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb7c0aff90 0x7feb7c0c51e0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7feb6c0097a0 tx=0x7feb6c02b8d0 comp rx=0 tx=0).stop 2024-02-21T23:04:31.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:31.900+0000 7feb81200640 1 -- 172.21.15.28:0/2809183617 shutdown_connections 2024-02-21T23:04:31.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:31.900+0000 7feb81200640 1 --2- 172.21.15.28:0/2809183617 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feb7c0c5720 0x7feb7c15f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:31.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:31.900+0000 7feb81200640 1 --2- 172.21.15.28:0/2809183617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb7c0aff90 0x7feb7c0c51e0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:31.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:31.900+0000 7feb81200640 1 -- 172.21.15.28:0/2809183617 >> 172.21.15.28:0/2809183617 conn(0x7feb7c0ac370 msgr2=0x7feb7c0ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:31.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:31.901+0000 7feb81200640 1 -- 172.21.15.28:0/2809183617 shutdown_connections 2024-02-21T23:04:31.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:31.901+0000 7feb81200640 1 -- 172.21.15.28:0/2809183617 wait complete. 2024-02-21T23:04:31.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:31.902+0000 7feb81200640 1 Processor -- start 2024-02-21T23:04:31.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:31.902+0000 7feb81200640 1 -- start start 2024-02-21T23:04:31.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:31.903+0000 7feb81200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb7c06dbf0 0x7feb7c0ae740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:31.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:31.903+0000 7feb81200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feb7c0aff90 0x7feb7c14e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:31.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:31.903+0000 7feb81200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feb7c0c5720 0x7feb7c14edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:31.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:31.903+0000 7feb81200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb7c0c8760 con 0x7feb7c06dbf0 2024-02-21T23:04:31.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:31.903+0000 7feb81200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb7c0be0c0 con 0x7feb7c0c5720 2024-02-21T23:04:31.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:31.903+0000 7feb81200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb7c0c88e0 con 0x7feb7c0aff90 2024-02-21T23:04:31.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:31.903+0000 7feb7b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb7c06dbf0 0x7feb7c0ae740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:31.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:31.903+0000 7feb7b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb7c06dbf0 0x7feb7c0ae740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48126/0 (socket says 172.21.15.28:48126) 2024-02-21T23:04:31.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:31.903+0000 7feb7b400640 1 -- 172.21.15.28:0/1635261246 learned_addr learned my addr 172.21.15.28:0/1635261246 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:31.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:31.904+0000 7feb7b400640 1 -- 172.21.15.28:0/1635261246 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feb7c0aff90 msgr2=0x7feb7c14e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:31.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:31.904+0000 7feb7b400640 1 --2- 172.21.15.28:0/1635261246 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feb7c0aff90 0x7feb7c14e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:31.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:31.904+0000 7feb7b400640 1 -- 172.21.15.28:0/1635261246 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feb7c0c5720 msgr2=0x7feb7c14edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:31.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:31.904+0000 7feb7b400640 1 --2- 172.21.15.28:0/1635261246 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feb7c0c5720 0x7feb7c14edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:31.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:31.904+0000 7feb7b400640 1 -- 172.21.15.28:0/1635261246 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb7c14f330 con 0x7feb7c06dbf0 2024-02-21T23:04:31.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:31.904+0000 7feb7b400640 1 --2- 172.21.15.28:0/1635261246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb7c06dbf0 0x7feb7c0ae740 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7feb6c0098c0 tx=0x7feb6c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:31.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:31.904+0000 7feb73400640 1 -- 172.21.15.28:0/1635261246 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb6c041070 con 0x7feb7c06dbf0 2024-02-21T23:04:31.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:31.904+0000 7feb73400640 1 -- 172.21.15.28:0/1635261246 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb6c03b900 con 0x7feb7c06dbf0 2024-02-21T23:04:31.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:31.904+0000 7feb73400640 1 -- 172.21.15.28:0/1635261246 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb6c035040 con 0x7feb7c06dbf0 2024-02-21T23:04:31.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:31.904+0000 7feb81200640 1 -- 172.21.15.28:0/1635261246 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb7c154a40 con 0x7feb7c06dbf0 2024-02-21T23:04:31.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:31.905+0000 7feb81200640 1 -- 172.21.15.28:0/1635261246 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb7c1ee500 con 0x7feb7c06dbf0 2024-02-21T23:04:31.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:31.906+0000 7feb81200640 1 -- 172.21.15.28:0/1635261246 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb7c04f290 con 0x7feb7c06dbf0 2024-02-21T23:04:31.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:31.907+0000 7feb73400640 1 -- 172.21.15.28:0/1635261246 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feb6c04c060 con 0x7feb7c06dbf0 2024-02-21T23:04:31.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:31.908+0000 7feb73400640 1 --2- 172.21.15.28:0/1635261246 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7feb5806b4d0 0x7feb5806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:31.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:31.908+0000 7feb73400640 1 -- 172.21.15.28:0/1635261246 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6216+0+0 (secure 0 0 0) 0x7feb6c0abfc0 con 0x7feb7c06dbf0 2024-02-21T23:04:31.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:31.909+0000 7feb73e00640 1 --2- 172.21.15.28:0/1635261246 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7feb5806b4d0 0x7feb5806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:31.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:31.910+0000 7feb73e00640 1 --2- 172.21.15.28:0/1635261246 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7feb5806b4d0 0x7feb5806d990 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7feb64000f60 tx=0x7feb640078f0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:31.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:31.912+0000 7feb73400640 1 -- 172.21.15.28:0/1635261246 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7feb6c02f2e0 con 0x7feb7c06dbf0 2024-02-21T23:04:32.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.155+0000 7feb81200640 1 -- 172.21.15.28:0/1635261246 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feb7c06d9d0 con 0x7feb7c06dbf0 2024-02-21T23:04:32.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.156+0000 7feb73400640 1 -- 172.21.15.28:0/1635261246 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4934 (secure 0 0 0) 0x7feb6c031800 con 0x7feb7c06dbf0 2024-02-21T23:04:32.146 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:04:32.146 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":24,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4978,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/3612644344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":3612644344},{"type":"v1","addr":"172.21.15.81:6836","nonce":3612644344}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4981,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3623937361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":3623937361},{"type":"v1","addr":"172.21.15.81:6837","nonce":3623937361}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5442,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/33413357","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":33413357},{"type":"v1","addr":"172.21.15.28:6835","nonce":33413357}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:04:09.036302+0000","modified":"2024-02-21T23:04:26.710368+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4778},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4778":{"gid":4778,"name":"c","rank":0,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.28:6834/2827878900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2827878900},{"type":"v1","addr":"172.21.15.28:6834","nonce":2827878900}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-02-21T23:04:32.146 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 24 2024-02-21T23:04:32.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.159+0000 7feb81200640 1 -- 172.21.15.28:0/1635261246 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7feb5806b4d0 msgr2=0x7feb5806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:32.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.159+0000 7feb81200640 1 --2- 172.21.15.28:0/1635261246 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7feb5806b4d0 0x7feb5806d990 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7feb64000f60 tx=0x7feb640078f0 comp rx=0 tx=0).stop 2024-02-21T23:04:32.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.159+0000 7feb81200640 1 -- 172.21.15.28:0/1635261246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb7c06dbf0 msgr2=0x7feb7c0ae740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:32.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.159+0000 7feb81200640 1 --2- 172.21.15.28:0/1635261246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb7c06dbf0 0x7feb7c0ae740 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7feb6c0098c0 tx=0x7feb6c0046c0 comp rx=0 tx=0).stop 2024-02-21T23:04:32.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.159+0000 7feb81200640 1 -- 172.21.15.28:0/1635261246 shutdown_connections 2024-02-21T23:04:32.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.159+0000 7feb81200640 1 --2- 172.21.15.28:0/1635261246 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7feb5806b4d0 0x7feb5806d990 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:32.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.159+0000 7feb81200640 1 --2- 172.21.15.28:0/1635261246 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feb7c0aff90 0x7feb7c14e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:32.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.159+0000 7feb81200640 1 --2- 172.21.15.28:0/1635261246 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feb7c0c5720 0x7feb7c14edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:32.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.159+0000 7feb81200640 1 --2- 172.21.15.28:0/1635261246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb7c06dbf0 0x7feb7c0ae740 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:32.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.159+0000 7feb81200640 1 -- 172.21.15.28:0/1635261246 >> 172.21.15.28:0/1635261246 conn(0x7feb7c0ac370 msgr2=0x7feb7c1573e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:32.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.159+0000 7feb81200640 1 -- 172.21.15.28:0/1635261246 shutdown_connections 2024-02-21T23:04:32.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.159+0000 7feb81200640 1 -- 172.21.15.28:0/1635261246 wait complete. 2024-02-21T23:04:32.569 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:04:09.036302+0000', 'modified': '2024-02-21T23:04:26.710368+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 39, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4778}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4778': {'gid': 4778, 'name': 'c', 'rank': 0, 'incarnation': 20, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.28:6834/2827878900', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 2827878900}, {'type': 'v1', 'addr': '172.21.15.28:6834', 'nonce': 2827878900}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:04:32.569 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-21T23:04:32.569 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-02-21T23:04:32.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.803+0000 7f4ad9e00640 1 Processor -- start 2024-02-21T23:04:32.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.804+0000 7f4ad9e00640 1 -- start start 2024-02-21T23:04:32.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.806+0000 7f4ad9e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4ad41524c0 0x7f4ad4155a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:32.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.806+0000 7f4ad9e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4ad40c85c0 0x7f4ad40c70d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:32.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.806+0000 7f4ad9e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ad406df50 con 0x7f4ad40c5040 2024-02-21T23:04:32.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.806+0000 7f4ad9e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ad40c5520 con 0x7f4ad40c85c0 2024-02-21T23:04:32.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.806+0000 7f4ad9e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad40c5800 con 0x7f4ad41524c0 2024-02-21T23:04:32.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.806+0000 7f4ad9400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4ad40c85c0 0x7f4ad40c70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:32.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.807+0000 7f4ad9400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4ad40c85c0 0x7f4ad40c70d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48128/0 (socket says 172.21.15.28:48128) 2024-02-21T23:04:32.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.807+0000 7f4ad9400640 1 -- 172.21.15.28:0/3419487838 learned_addr learned my addr 172.21.15.28:0/3419487838 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:32.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.807+0000 7f4ad8a00640 1 --2- 172.21.15.28:0/3419487838 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4ad41524c0 0x7f4ad4155a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:32.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.807+0000 7f4ad9400640 1 -- 172.21.15.28:0/3419487838 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4ad41524c0 msgr2=0x7f4ad4155a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:32.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.807+0000 7f4ad9400640 1 --2- 172.21.15.28:0/3419487838 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4ad41524c0 0x7f4ad4155a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:32.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.807+0000 7f4ad9400640 1 -- 172.21.15.28:0/3419487838 >> v1:172.21.15.28:6789/0 conn(0x7f4ad40c5040 legacy=0x7f4ad406ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:32.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.807+0000 7f4ad9400640 1 -- 172.21.15.28:0/3419487838 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ad406c670 con 0x7f4ad40c85c0 2024-02-21T23:04:32.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.808+0000 7f4ad9400640 1 --2- 172.21.15.28:0/3419487838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ad40c85c0 0x7f4ad40c70d0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f4acc009e50 tx=0x7f4acc02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5943d15101b85693 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:32.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.809+0000 7f4ad3400640 1 -- 172.21.15.28:0/3419487838 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4acc02c9f0 con 0x7f4ad40c85c0 2024-02-21T23:04:32.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.809+0000 7f4ad3400640 1 -- 172.21.15.28:0/3419487838 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4acc02cb90 con 0x7f4ad40c85c0 2024-02-21T23:04:32.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.809+0000 7f4ad3400640 1 -- 172.21.15.28:0/3419487838 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4acc0343f0 con 0x7f4ad40c85c0 2024-02-21T23:04:32.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.810+0000 7f4ad9e00640 1 -- 172.21.15.28:0/3419487838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ad40c85c0 msgr2=0x7f4ad40c70d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:32.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.810+0000 7f4ad9e00640 1 --2- 172.21.15.28:0/3419487838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ad40c85c0 0x7f4ad40c70d0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f4acc009e50 tx=0x7f4acc02be50 comp rx=0 tx=0).stop 2024-02-21T23:04:32.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.810+0000 7f4ad9e00640 1 -- 172.21.15.28:0/3419487838 shutdown_connections 2024-02-21T23:04:32.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.810+0000 7f4ad9e00640 1 --2- 172.21.15.28:0/3419487838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ad40c85c0 0x7f4ad40c70d0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:32.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.810+0000 7f4ad9e00640 1 --2- 172.21.15.28:0/3419487838 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4ad41524c0 0x7f4ad4155a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:32.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.810+0000 7f4ad9e00640 1 -- 172.21.15.28:0/3419487838 >> 172.21.15.28:0/3419487838 conn(0x7f4ad40ac2b0 msgr2=0x7f4ad40ac6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:32.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.810+0000 7f4ad9e00640 1 -- 172.21.15.28:0/3419487838 shutdown_connections 2024-02-21T23:04:32.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.810+0000 7f4ad9e00640 1 -- 172.21.15.28:0/3419487838 wait complete. 2024-02-21T23:04:32.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.812+0000 7f4ad9e00640 1 Processor -- start 2024-02-21T23:04:32.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.812+0000 7f4ad9e00640 1 -- start start 2024-02-21T23:04:32.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.813+0000 7f4ad9e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4ad40c5040 0x7f4ad414b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:32.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.813+0000 7f4ad9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ad40c85c0 0x7f4ad41eac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:32.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.813+0000 7f4ad9e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ad41524c0 0x7f4ad41eb180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:32.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.813+0000 7f4ad9e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad40c8a70 con 0x7f4ad40c85c0 2024-02-21T23:04:32.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.813+0000 7f4ad9e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad41528a0 con 0x7f4ad41524c0 2024-02-21T23:04:32.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.813+0000 7f4ad9e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad40c7610 con 0x7f4ad40c5040 2024-02-21T23:04:32.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.813+0000 7f4ad3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ad40c85c0 0x7f4ad41eac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:32.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.813+0000 7f4ad3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ad40c85c0 0x7f4ad41eac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48142/0 (socket says 172.21.15.28:48142) 2024-02-21T23:04:32.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.813+0000 7f4ad3e00640 1 -- 172.21.15.28:0/2758980332 learned_addr learned my addr 172.21.15.28:0/2758980332 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:32.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.813+0000 7f4ad8a00640 1 --2- 172.21.15.28:0/2758980332 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4ad40c5040 0x7f4ad414b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:32.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.813+0000 7f4ad9400640 1 --2- 172.21.15.28:0/2758980332 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ad41524c0 0x7f4ad41eb180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:32.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.814+0000 7f4ad3e00640 1 -- 172.21.15.28:0/2758980332 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4ad40c5040 msgr2=0x7f4ad414b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:32.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.814+0000 7f4ad3e00640 1 --2- 172.21.15.28:0/2758980332 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4ad40c5040 0x7f4ad414b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:32.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.814+0000 7f4ad3e00640 1 -- 172.21.15.28:0/2758980332 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ad41524c0 msgr2=0x7f4ad41eb180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:32.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.814+0000 7f4ad3e00640 1 --2- 172.21.15.28:0/2758980332 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ad41524c0 0x7f4ad41eb180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:32.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.814+0000 7f4ad3e00640 1 -- 172.21.15.28:0/2758980332 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ad41551d0 con 0x7f4ad40c85c0 2024-02-21T23:04:32.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.814+0000 7f4ad9400640 1 --2- 172.21.15.28:0/2758980332 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ad41524c0 0x7f4ad41eb180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:04:32.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.814+0000 7f4ad3e00640 1 --2- 172.21.15.28:0/2758980332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ad40c85c0 0x7f4ad41eac40 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f4ac000c400 tx=0x7f4ac000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:32.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.814+0000 7f4ad0c00640 1 -- 172.21.15.28:0/2758980332 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ac00052d0 con 0x7f4ad40c85c0 2024-02-21T23:04:32.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.814+0000 7f4ad0c00640 1 -- 172.21.15.28:0/2758980332 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ac0005490 con 0x7f4ad40c85c0 2024-02-21T23:04:32.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.814+0000 7f4ad9e00640 1 -- 172.21.15.28:0/2758980332 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ad41ee170 con 0x7f4ad40c85c0 2024-02-21T23:04:32.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.814+0000 7f4ad0c00640 1 -- 172.21.15.28:0/2758980332 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ac0007260 con 0x7f4ad40c85c0 2024-02-21T23:04:32.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.814+0000 7f4ad9e00640 1 -- 172.21.15.28:0/2758980332 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ad41ee730 con 0x7f4ad40c85c0 2024-02-21T23:04:32.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.817+0000 7f4ad0c00640 1 -- 172.21.15.28:0/2758980332 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4ac0007400 con 0x7f4ad40c85c0 2024-02-21T23:04:32.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.817+0000 7f4ad0c00640 1 --2- 172.21.15.28:0/2758980332 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4ab406b4d0 0x7f4ab406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:32.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.818+0000 7f4ad0c00640 1 -- 172.21.15.28:0/2758980332 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6216+0+0 (secure 0 0 0) 0x7f4ac0089990 con 0x7f4ad40c85c0 2024-02-21T23:04:32.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.818+0000 7f4ade1f6740 1 Processor -- start 2024-02-21T23:04:32.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.818+0000 7f4ad8a00640 1 --2- 172.21.15.28:0/2758980332 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4ab406b4d0 0x7f4ab406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:32.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.818+0000 7f4ade1f6740 1 -- start start 2024-02-21T23:04:32.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.819+0000 7f4ade1f6740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f199a25020 0x55f199a25400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:32.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.819+0000 7f4ade1f6740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55f199a26330 0x55f199a61170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:32.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.819+0000 7f4ade1f6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f199a616b0 0x55f199a5f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:32.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.819+0000 7f4ade1f6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f199989470 con 0x55f199a616b0 2024-02-21T23:04:32.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.819+0000 7f4ade1f6740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f199932af0 con 0x55f199a25020 2024-02-21T23:04:32.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.819+0000 7f4ade1f6740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f199985510 con 0x55f199a26330 2024-02-21T23:04:32.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.819+0000 7f4ad9400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f199a616b0 0x55f199a5f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:32.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.820+0000 7f4ad9400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f199a616b0 0x55f199a5f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48156/0 (socket says 172.21.15.28:48156) 2024-02-21T23:04:32.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.820+0000 7f4ad9400640 1 -- 172.21.15.28:0/2838518599 learned_addr learned my addr 172.21.15.28:0/2838518599 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:32.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.820+0000 7f4ad8a00640 1 --2- 172.21.15.28:0/2758980332 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4ab406b4d0 0x7f4ab406d990 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f4ac8006fd0 tx=0x7f4ac8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:32.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.820+0000 7f4ad3e00640 1 --2- 172.21.15.28:0/2838518599 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55f199a26330 0x55f199a61170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:32.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.820+0000 7f4ad8a00640 1 --2- 172.21.15.28:0/2838518599 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f199a25020 0x55f199a25400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:32.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.820+0000 7f4ad9400640 1 -- 172.21.15.28:0/2838518599 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55f199a26330 msgr2=0x55f199a61170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:32.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.820+0000 7f4ad9400640 1 --2- 172.21.15.28:0/2838518599 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55f199a26330 0x55f199a61170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:32.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.820+0000 7f4ad9400640 1 -- 172.21.15.28:0/2838518599 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f199a25020 msgr2=0x55f199a25400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:32.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.820+0000 7f4ad9400640 1 --2- 172.21.15.28:0/2838518599 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f199a25020 0x55f199a25400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:32.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.820+0000 7f4ad9400640 1 -- 172.21.15.28:0/2838518599 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f199939d90 con 0x55f199a616b0 2024-02-21T23:04:32.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.820+0000 7f4ad9400640 1 --2- 172.21.15.28:0/2838518599 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f199a616b0 0x55f199a5f720 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f4acc009f70 tx=0x7f4acc02e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cb9d45ec5661ab5 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:32.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.821+0000 7f4ac5200640 1 -- 172.21.15.28:0/2838518599 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4acc02c9f0 con 0x55f199a616b0 2024-02-21T23:04:32.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.821+0000 7f4ac5200640 1 -- 172.21.15.28:0/2838518599 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4acc02cb90 con 0x55f199a616b0 2024-02-21T23:04:32.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.821+0000 7f4ac5200640 1 -- 172.21.15.28:0/2838518599 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4acc03e430 con 0x55f199a616b0 2024-02-21T23:04:32.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.821+0000 7f4ade1f6740 1 -- 172.21.15.28:0/2838518599 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f199a616b0 msgr2=0x55f199a5f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:32.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.821+0000 7f4ade1f6740 1 --2- 172.21.15.28:0/2838518599 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f199a616b0 0x55f199a5f720 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f4acc009f70 tx=0x7f4acc02e000 comp rx=0 tx=0).stop 2024-02-21T23:04:32.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.821+0000 7f4ade1f6740 1 -- 172.21.15.28:0/2838518599 shutdown_connections 2024-02-21T23:04:32.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.821+0000 7f4ade1f6740 1 --2- 172.21.15.28:0/2838518599 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55f199a26330 0x55f199a61170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:32.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.821+0000 7f4ade1f6740 1 --2- 172.21.15.28:0/2838518599 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f199a616b0 0x55f199a5f720 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:32.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.821+0000 7f4ade1f6740 1 --2- 172.21.15.28:0/2838518599 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f199a25020 0x55f199a25400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:32.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.821+0000 7f4ade1f6740 1 -- 172.21.15.28:0/2838518599 >> 172.21.15.28:0/2838518599 conn(0x55f199a21970 msgr2=0x55f199a46380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:32.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.821+0000 7f4ade1f6740 1 -- 172.21.15.28:0/2838518599 shutdown_connections 2024-02-21T23:04:32.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.822+0000 7f4ade1f6740 1 -- 172.21.15.28:0/2838518599 wait complete. 2024-02-21T23:04:32.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.822+0000 7f4ade1f6740 1 -- start start 2024-02-21T23:04:32.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.822+0000 7f4ade1f6740 1 Processor -- start 2024-02-21T23:04:32.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.823+0000 7f4ade1f6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f199a26330 0x55f199a59c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:32.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.823+0000 7f4ade1f6740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f199a616b0 0x55f199a58c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:32.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.823+0000 7f4ade1f6740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55f199a413d0 0x55f199a5e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:32.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.823+0000 7f4ade1f6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f199928040 con 0x55f199a26330 2024-02-21T23:04:32.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.823+0000 7f4ade1f6740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f199a46a90 con 0x55f199a616b0 2024-02-21T23:04:32.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.823+0000 7f4ade1f6740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f199a72550 con 0x55f199a413d0 2024-02-21T23:04:32.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.823+0000 7f4ade1f6740 10 client.0 _ll_register_callbacks cb 0x55f199945920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:04:32.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.824+0000 7f4ad8a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f199a26330 0x55f199a59c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:32.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.824+0000 7f4ad8a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f199a26330 0x55f199a59c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48164/0 (socket says 172.21.15.28:48164) 2024-02-21T23:04:32.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.824+0000 7f4ad8a00640 1 -- 172.21.15.28:0/472252035 learned_addr learned my addr 172.21.15.28:0/472252035 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:32.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.824+0000 7f4ad9400640 1 --2- 172.21.15.28:0/472252035 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55f199a413d0 0x55f199a5e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:32.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.824+0000 7f4ad3e00640 1 --2- 172.21.15.28:0/472252035 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f199a616b0 0x55f199a58c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:32.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.824+0000 7f4ad8a00640 1 -- 172.21.15.28:0/472252035 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55f199a413d0 msgr2=0x55f199a5e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:32.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.824+0000 7f4ad8a00640 1 --2- 172.21.15.28:0/472252035 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55f199a413d0 0x55f199a5e2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:32.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.824+0000 7f4ad8a00640 1 -- 172.21.15.28:0/472252035 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f199a616b0 msgr2=0x55f199a58c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:32.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.824+0000 7f4ad8a00640 1 --2- 172.21.15.28:0/472252035 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f199a616b0 0x55f199a58c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:32.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.824+0000 7f4ad8a00640 1 -- 172.21.15.28:0/472252035 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f199947aa0 con 0x55f199a26330 2024-02-21T23:04:32.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.824+0000 7f4ad3e00640 1 --2- 172.21.15.28:0/472252035 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f199a616b0 0x55f199a58c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:04:32.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.825+0000 7f4ad8a00640 1 --2- 172.21.15.28:0/472252035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f199a26330 0x55f199a59c70 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f4ac800b2a0 tx=0x7f4ac8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3ff6f6a1ddd2ea5 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:32.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.825+0000 7f4ab9600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:04:32.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.825+0000 7f4ab9600640 1 -- 172.21.15.28:0/472252035 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ac8025020 con 0x55f199a26330 2024-02-21T23:04:32.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.825+0000 7f4ab9600640 1 -- 172.21.15.28:0/472252035 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(41..41 src has 1..41) v4 ==== 6216+0+0 (secure 0 0 0) 0x7f4ac8021830 con 0x55f199a26330 2024-02-21T23:04:32.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.825+0000 7f4ade1f6740 1 -- 172.21.15.28:0/472252035 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4acc02cb90 con 0x55f199a26330 2024-02-21T23:04:32.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.825+0000 7f4ab9600640 1 -- 172.21.15.28:0/472252035 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4ac802f050 con 0x55f199a26330 2024-02-21T23:04:32.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.825+0000 7f4ade1f6740 10 client.5526 fetch_fsmap learned FSMap version 24 2024-02-21T23:04:32.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.825+0000 7f4ade1f6740 1 -- 172.21.15.28:0/472252035 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=2+}) v3 -- 0x55f19969e2d0 con 0x55f199a26330 2024-02-21T23:04:32.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.825+0000 7f4ab9600640 1 -- 172.21.15.28:0/472252035 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4acc02cb90 con 0x55f199a26330 2024-02-21T23:04:32.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.826+0000 7f4ab9600640 1 -- 172.21.15.28:0/472252035 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 24) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f4ac8032020 con 0x55f199a26330 2024-02-21T23:04:32.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.826+0000 7f4ade1f6740 10 client.5526 fetch_fsmap finished waiting for FSMap version 24 2024-02-21T23:04:32.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.826+0000 7f4ade1f6740 10 client.5526 resolve_mds: resolved name 'c' to daemon mds.c 2024-02-21T23:04:32.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.826+0000 7f4ade1f6740 20 client.5526 populate_metadata read hostname 'smithi028' 2024-02-21T23:04:32.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.826+0000 7f4ade1f6740 1 --2- 172.21.15.28:0/472252035 >> [v2:172.21.15.28:6832/2827878900,v1:172.21.15.28:6834/2827878900] conn(0x55f199a40440 0x55f199a40820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:32.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.826+0000 7f4ade1f6740 4 client.5526 mds_command: new command op to 4778 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-21T23:04:32.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.826+0000 7f4ade1f6740 1 -- 172.21.15.28:0/472252035 --> [v2:172.21.15.28:6832/2827878900,v1:172.21.15.28:6834/2827878900] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f1999969b0 con 0x55f199a40440 2024-02-21T23:04:32.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.826+0000 7f4ad3e00640 1 --2- 172.21.15.28:0/472252035 >> [v2:172.21.15.28:6832/2827878900,v1:172.21.15.28:6834/2827878900] conn(0x55f199a40440 0x55f199a40820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:32.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.826+0000 7f4ab9600640 10 client.5526 ms_handle_connect on v2:172.21.15.28:6832/2827878900 2024-02-21T23:04:32.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.826+0000 7f4ad3e00640 1 --2- 172.21.15.28:0/472252035 >> [v2:172.21.15.28:6832/2827878900,v1:172.21.15.28:6834/2827878900] conn(0x55f199a40440 0x55f199a40820 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=37d582e865b58d47 server_cookie=264371d792c4b22 in_seq=0 out_seq=0 2024-02-21T23:04:32.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.827+0000 7f4ab9600640 1 -- 172.21.15.28:0/472252035 <== mds.0 v2:172.21.15.28:6832/2827878900 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f1999969b0 con 0x55f199a40440 2024-02-21T23:04:32.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.827+0000 7f4ab9600640 10 client.5526 handle_command_reply: tid=0 2024-02-21T23:04:32.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.827+0000 7f4ade1f6740 1 client.5526 shutdown 2024-02-21T23:04:32.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.827+0000 7f4ade1f6740 1 -- 172.21.15.28:0/472252035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f199a26330 msgr2=0x55f199a59c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:32.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.827+0000 7f4ade1f6740 1 --2- 172.21.15.28:0/472252035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f199a26330 0x55f199a59c70 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f4ac800b2a0 tx=0x7f4ac8004730 comp rx=0 tx=0).stop 2024-02-21T23:04:32.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.828+0000 7f4ade1f6740 1 -- 172.21.15.28:0/472252035 shutdown_connections 2024-02-21T23:04:32.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.828+0000 7f4ade1f6740 1 --2- 172.21.15.28:0/472252035 >> [v2:172.21.15.28:6832/2827878900,v1:172.21.15.28:6834/2827878900] conn(0x55f199a40440 0x55f199a40820 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:32.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.828+0000 7f4ade1f6740 1 --2- 172.21.15.28:0/472252035 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55f199a413d0 0x55f199a5e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:32.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.828+0000 7f4ade1f6740 1 --2- 172.21.15.28:0/472252035 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f199a616b0 0x55f199a58c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:32.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.828+0000 7f4ade1f6740 1 --2- 172.21.15.28:0/472252035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f199a26330 0x55f199a59c70 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:32.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.828+0000 7f4ab9600640 0 client.5526 ms_handle_reset on v2:172.21.15.28:6832/2827878900 2024-02-21T23:04:32.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.828+0000 7f4ade1f6740 1 -- 172.21.15.28:0/472252035 >> 172.21.15.28:0/472252035 conn(0x55f199a25020 msgr2=0x55f199a1e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:32.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.828+0000 7f4ade1f6740 1 -- 172.21.15.28:0/472252035 shutdown_connections 2024-02-21T23:04:32.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.828+0000 7f4ade1f6740 1 -- 172.21.15.28:0/472252035 wait complete. 2024-02-21T23:04:32.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.828+0000 7f4ade1f6740 20 client.5526 trim_cache size 0 max 16384 2024-02-21T23:04:32.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.846+0000 7f4ade1f6740 1 Processor -- start 2024-02-21T23:04:32.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.846+0000 7f4ade1f6740 1 -- start start 2024-02-21T23:04:32.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.847+0000 7f4ade1f6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f199a26330 0x55f199a1ad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:32.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.847+0000 7f4ade1f6740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f199a413d0 0x55f199a5f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:32.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.847+0000 7f4ade1f6740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55f199a616b0 0x55f199a5fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:32.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.847+0000 7f4ade1f6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad41f6c10 con 0x55f199a26330 2024-02-21T23:04:32.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.847+0000 7f4ade1f6740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad41549a0 con 0x55f199a413d0 2024-02-21T23:04:32.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.847+0000 7f4ade1f6740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad41f6670 con 0x55f199a616b0 2024-02-21T23:04:32.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.847+0000 7f4ad8a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f199a26330 0x55f199a1ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:32.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.847+0000 7f4ad8a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f199a26330 0x55f199a1ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48176/0 (socket says 172.21.15.28:48176) 2024-02-21T23:04:32.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.847+0000 7f4ad8a00640 1 -- 172.21.15.28:0/1830990997 learned_addr learned my addr 172.21.15.28:0/1830990997 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:32.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.848+0000 7f4ad8a00640 1 -- 172.21.15.28:0/1830990997 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55f199a616b0 msgr2=0x55f199a5fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:32.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.848+0000 7f4ad8a00640 1 --2- 172.21.15.28:0/1830990997 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55f199a616b0 0x55f199a5fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:32.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.848+0000 7f4ad8a00640 1 -- 172.21.15.28:0/1830990997 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f199a413d0 msgr2=0x55f199a5f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:32.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.848+0000 7f4ad8a00640 1 --2- 172.21.15.28:0/1830990997 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f199a413d0 0x55f199a5f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:32.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.848+0000 7f4ad8a00640 1 -- 172.21.15.28:0/1830990997 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f1997686b0 con 0x55f199a26330 2024-02-21T23:04:32.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.848+0000 7f4ad8a00640 1 --2- 172.21.15.28:0/1830990997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f199a26330 0x55f199a1ad10 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f4ac800fc60 tx=0x7f4ac8020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0e093c58c8b8a36 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:32.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.848+0000 7f4abbe00640 1 -- 172.21.15.28:0/1830990997 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ac8025050 con 0x55f199a26330 2024-02-21T23:04:32.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.848+0000 7f4abbe00640 1 -- 172.21.15.28:0/1830990997 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ac801c040 con 0x55f199a26330 2024-02-21T23:04:32.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.848+0000 7f4abbe00640 1 -- 172.21.15.28:0/1830990997 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ac801b070 con 0x55f199a26330 2024-02-21T23:04:32.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.849+0000 7f4ade1f6740 1 -- 172.21.15.28:0/1830990997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f199a26330 msgr2=0x55f199a1ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:32.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.849+0000 7f4ade1f6740 1 --2- 172.21.15.28:0/1830990997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f199a26330 0x55f199a1ad10 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f4ac800fc60 tx=0x7f4ac8020000 comp rx=0 tx=0).stop 2024-02-21T23:04:32.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.849+0000 7f4ade1f6740 1 -- 172.21.15.28:0/1830990997 shutdown_connections 2024-02-21T23:04:32.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.849+0000 7f4ade1f6740 1 --2- 172.21.15.28:0/1830990997 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55f199a616b0 0x55f199a5fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:32.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.849+0000 7f4ade1f6740 1 --2- 172.21.15.28:0/1830990997 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f199a413d0 0x55f199a5f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:32.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.849+0000 7f4ade1f6740 1 --2- 172.21.15.28:0/1830990997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f199a26330 0x55f199a1ad10 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:32.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.849+0000 7f4ade1f6740 1 -- 172.21.15.28:0/1830990997 >> 172.21.15.28:0/1830990997 conn(0x55f199a25020 msgr2=0x55f199a7c150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:32.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.849+0000 7f4ade1f6740 1 -- 172.21.15.28:0/1830990997 shutdown_connections 2024-02-21T23:04:32.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.849+0000 7f4ade1f6740 1 -- 172.21.15.28:0/1830990997 wait complete. 2024-02-21T23:04:32.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.850+0000 7f4ade1f6740 1 -- start start 2024-02-21T23:04:32.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.850+0000 7f4ade1f6740 1 Processor -- start 2024-02-21T23:04:32.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.851+0000 7f4ade1f6740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55f199a26330 0x55f199a5fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:32.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.851+0000 7f4ade1f6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f199a413d0 0x55f199ad3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:32.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.851+0000 7f4ade1f6740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f199a616b0 0x55f199acd940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:32.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.851+0000 7f4ade1f6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f199a58420 con 0x55f199a413d0 2024-02-21T23:04:32.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.851+0000 7f4ade1f6740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f199a402c0 con 0x55f199a616b0 2024-02-21T23:04:32.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.851+0000 7f4ade1f6740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f199a60290 con 0x55f199a26330 2024-02-21T23:04:32.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.851+0000 7f4ade1f6740 10 client.0 _ll_register_callbacks cb 0x7f4ac800fcd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:04:32.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.851+0000 7f4ad3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f199a413d0 0x55f199ad3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:32.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.851+0000 7f4ad3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f199a413d0 0x55f199ad3cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48180/0 (socket says 172.21.15.28:48180) 2024-02-21T23:04:32.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.851+0000 7f4ad3e00640 1 -- 172.21.15.28:0/3435848663 learned_addr learned my addr 172.21.15.28:0/3435848663 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:32.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.851+0000 7f4ad8a00640 1 --2- 172.21.15.28:0/3435848663 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55f199a26330 0x55f199a5fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:32.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.851+0000 7f4ad9400640 1 --2- 172.21.15.28:0/3435848663 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f199a616b0 0x55f199acd940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:32.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.851+0000 7f4ad3e00640 1 -- 172.21.15.28:0/3435848663 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55f199a26330 msgr2=0x55f199a5fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:32.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.851+0000 7f4ad3e00640 1 --2- 172.21.15.28:0/3435848663 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55f199a26330 0x55f199a5fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:32.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.851+0000 7f4ad3e00640 1 -- 172.21.15.28:0/3435848663 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f199a616b0 msgr2=0x55f199acd940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:32.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.852+0000 7f4ad3e00640 1 --2- 172.21.15.28:0/3435848663 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f199a616b0 0x55f199acd940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:32.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.852+0000 7f4ad3e00640 1 -- 172.21.15.28:0/3435848663 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f199a259d0 con 0x55f199a413d0 2024-02-21T23:04:32.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.852+0000 7f4ad8a00640 1 --2- 172.21.15.28:0/3435848663 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55f199a26330 0x55f199a5fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:04:32.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.852+0000 7f4ad3e00640 1 --2- 172.21.15.28:0/3435848663 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f199a413d0 0x55f199ad3cf0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f4ac0057c00 tx=0x7f4ac005c110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd8a115970cda693 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:32.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.852+0000 7f4ab9600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:04:32.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.852+0000 7f4ab9600640 1 -- 172.21.15.28:0/3435848663 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ac0085020 con 0x55f199a413d0 2024-02-21T23:04:32.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.852+0000 7f4ab9600640 1 -- 172.21.15.28:0/3435848663 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(41..41 src has 1..41) v4 ==== 6216+0+0 (secure 0 0 0) 0x7f4ac0060180 con 0x55f199a413d0 2024-02-21T23:04:32.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.852+0000 7f4ade1f6740 1 -- 172.21.15.28:0/3435848663 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4ac801c040 con 0x55f199a413d0 2024-02-21T23:04:32.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.852+0000 7f4ab9600640 1 -- 172.21.15.28:0/3435848663 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4ac0087df0 con 0x55f199a413d0 2024-02-21T23:04:32.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.852+0000 7f4ade1f6740 10 client.5532 fetch_fsmap learned FSMap version 24 2024-02-21T23:04:32.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.852+0000 7f4ade1f6740 1 -- 172.21.15.28:0/3435848663 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=2+}) v3 -- 0x55f199a723f0 con 0x55f199a413d0 2024-02-21T23:04:32.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.853+0000 7f4ab9600640 1 -- 172.21.15.28:0/3435848663 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ac801c040 con 0x55f199a413d0 2024-02-21T23:04:32.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.853+0000 7f4ab9600640 1 -- 172.21.15.28:0/3435848663 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 24) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f4ac0057350 con 0x55f199a413d0 2024-02-21T23:04:32.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.853+0000 7f4ade1f6740 10 client.5532 fetch_fsmap finished waiting for FSMap version 24 2024-02-21T23:04:32.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.853+0000 7f4ade1f6740 10 client.5532 resolve_mds: resolved name 'c' to daemon mds.c 2024-02-21T23:04:32.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.853+0000 7f4ade1f6740 20 client.5532 populate_metadata read hostname 'smithi028' 2024-02-21T23:04:32.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.853+0000 7f4ade1f6740 1 --2- 172.21.15.28:0/3435848663 >> [v2:172.21.15.28:6832/2827878900,v1:172.21.15.28:6834/2827878900] conn(0x55f199a42a50 0x55f199a42e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:32.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.853+0000 7f4ade1f6740 4 client.5532 mds_command: new command op to 4778 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-21T23:04:32.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.853+0000 7f4ade1f6740 1 -- 172.21.15.28:0/3435848663 --> [v2:172.21.15.28:6832/2827878900,v1:172.21.15.28:6834/2827878900] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55f199990d00 con 0x55f199a42a50 2024-02-21T23:04:32.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.853+0000 7f4ad8a00640 1 --2- 172.21.15.28:0/3435848663 >> [v2:172.21.15.28:6832/2827878900,v1:172.21.15.28:6834/2827878900] conn(0x55f199a42a50 0x55f199a42e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:32.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.854+0000 7f4ad8a00640 1 --2- 172.21.15.28:0/3435848663 >> [v2:172.21.15.28:6832/2827878900,v1:172.21.15.28:6834/2827878900] conn(0x55f199a42a50 0x55f199a42e30 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c59668dd154aa8e server_cookie=c43b783fa7bc9aa1 in_seq=0 out_seq=0 2024-02-21T23:04:32.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.854+0000 7f4ab9600640 10 client.5532 ms_handle_connect on v2:172.21.15.28:6832/2827878900 2024-02-21T23:04:32.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.854+0000 7f4ab9600640 1 -- 172.21.15.28:0/3435848663 <== mds.0 v2:172.21.15.28:6832/2827878900 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x55f199990d00 con 0x55f199a42a50 2024-02-21T23:04:32.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.854+0000 7f4ab9600640 10 client.5532 handle_command_reply: tid=0 2024-02-21T23:04:32.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.854+0000 7f4ade1f6740 1 client.5532 shutdown 2024-02-21T23:04:32.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.854+0000 7f4ade1f6740 1 -- 172.21.15.28:0/3435848663 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f199a413d0 msgr2=0x55f199ad3cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:32.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.854+0000 7f4ade1f6740 1 --2- 172.21.15.28:0/3435848663 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f199a413d0 0x55f199ad3cf0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f4ac0057c00 tx=0x7f4ac005c110 comp rx=0 tx=0).stop 2024-02-21T23:04:32.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.854+0000 7f4ade1f6740 1 -- 172.21.15.28:0/3435848663 shutdown_connections 2024-02-21T23:04:32.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.854+0000 7f4ade1f6740 1 --2- 172.21.15.28:0/3435848663 >> [v2:172.21.15.28:6832/2827878900,v1:172.21.15.28:6834/2827878900] conn(0x55f199a42a50 0x55f199a42e30 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:32.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.854+0000 7f4ade1f6740 1 --2- 172.21.15.28:0/3435848663 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f199a616b0 0x55f199acd940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:32.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.854+0000 7f4ade1f6740 1 --2- 172.21.15.28:0/3435848663 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f199a413d0 0x55f199ad3cf0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:32.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.854+0000 7f4ade1f6740 1 --2- 172.21.15.28:0/3435848663 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55f199a26330 0x55f199a5fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:32.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.854+0000 7f4ade1f6740 1 -- 172.21.15.28:0/3435848663 >> 172.21.15.28:0/3435848663 conn(0x55f199a25020 msgr2=0x55f199a59c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:32.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.854+0000 7f4ab9600640 0 client.5532 ms_handle_reset on v2:172.21.15.28:6832/2827878900 2024-02-21T23:04:32.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.855+0000 7f4ade1f6740 1 -- 172.21.15.28:0/3435848663 shutdown_connections 2024-02-21T23:04:32.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.855+0000 7f4ade1f6740 1 -- 172.21.15.28:0/3435848663 wait complete. 2024-02-21T23:04:32.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.855+0000 7f4ade1f6740 20 client.5532 trim_cache size 0 max 16384 2024-02-21T23:04:32.850 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-21T23:04:32.850 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac", 2024-02-21T23:04:32.850 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-02-21T23:04:32.850 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4778, 2024-02-21T23:04:32.850 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-02-21T23:04:32.850 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-02-21T23:04:32.850 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-02-21T23:04:32.850 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 12.161774016000001, 2024-02-21T23:04:32.850 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 24, 2024-02-21T23:04:32.851 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 41, 2024-02-21T23:04:32.851 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 41, 2024-02-21T23:04:32.851 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 32.283547358 2024-02-21T23:04:32.851 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-21T23:04:32.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.855+0000 7f4ad9e00640 1 -- 172.21.15.28:0/2758980332 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4ab406b4d0 msgr2=0x7f4ab406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:32.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.855+0000 7f4ad9e00640 1 --2- 172.21.15.28:0/2758980332 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4ab406b4d0 0x7f4ab406d990 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f4ac8006fd0 tx=0x7f4ac8008040 comp rx=0 tx=0).stop 2024-02-21T23:04:32.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.855+0000 7f4ad9e00640 1 -- 172.21.15.28:0/2758980332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ad40c85c0 msgr2=0x7f4ad41eac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:32.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.855+0000 7f4ad9e00640 1 --2- 172.21.15.28:0/2758980332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ad40c85c0 0x7f4ad41eac40 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f4ac000c400 tx=0x7f4ac000c8c0 comp rx=0 tx=0).stop 2024-02-21T23:04:32.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.855+0000 7f4ad9e00640 1 -- 172.21.15.28:0/2758980332 shutdown_connections 2024-02-21T23:04:32.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.855+0000 7f4ad9e00640 1 --2- 172.21.15.28:0/2758980332 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4ab406b4d0 0x7f4ab406d990 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:32.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.855+0000 7f4ad9e00640 1 --2- 172.21.15.28:0/2758980332 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ad41524c0 0x7f4ad41eb180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:32.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.855+0000 7f4ad9e00640 1 --2- 172.21.15.28:0/2758980332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ad40c85c0 0x7f4ad41eac40 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:32.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.855+0000 7f4ad9e00640 1 --2- 172.21.15.28:0/2758980332 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4ad40c5040 0x7f4ad414b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:32.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.856+0000 7f4ad9e00640 1 -- 172.21.15.28:0/2758980332 >> 172.21.15.28:0/2758980332 conn(0x7f4ad40ac2b0 msgr2=0x7f4ad40c7cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:32.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.856+0000 7f4ad9e00640 1 -- 172.21.15.28:0/2758980332 shutdown_connections 2024-02-21T23:04:32.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:32.856+0000 7f4ad9e00640 1 -- 172.21.15.28:0/2758980332 wait complete. 2024-02-21T23:04:32.862 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 2, "epoch": 24, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-02-21T23:04:09.036302+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 24, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4778": { "addr": "172.21.15.28:6834/2827878900", "addrs": { "addrvec": [ { "addr": "172.21.15.28:6832", "nonce": 2827878900, "type": "v2" }, { "addr": "172.21.15.28:6834", "nonce": 2827878900, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4778, "incarnation": 20, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 10 } }, "last_failure": 0, "last_failure_osd_epoch": 39, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 5, "modified": "2024-02-21T23:04:26.710368+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4778 } } } ], "standbys": [ { "addr": "172.21.15.81:6836/3612644344", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6834", "nonce": 3612644344, "type": "v2" }, { "addr": "172.21.15.81:6836", "nonce": 3612644344, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 12, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4978, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.81:6837/3623937361", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6835", "nonce": 3623937361, "type": "v2" }, { "addr": "172.21.15.81:6837", "nonce": 3623937361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 13, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4981, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.28:6835/33413357", "addrs": { "addrvec": [ { "addr": "172.21.15.28:6833", "nonce": 33413357, "type": "v2" }, { "addr": "172.21.15.28:6835", "nonce": 33413357, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 21, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5442, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-02-21T23:04:32.863 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c', 'state': 'up:active'}, {'join_fscid': -1, 'name': 'd'}] 2024-02-21T23:04:32.863 INFO:tasks.cephfs.test_failover:current = [{'gid': 5442, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.28:6835/33413357', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6833', 'nonce': 33413357}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 33413357}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 21}, {'gid': 4981, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.81:6837/3623937361', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6835', 'nonce': 3623937361}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3623937361}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 13}, {'gid': 4778, 'name': 'c', 'rank': 0, 'incarnation': 20, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.28:6834/2827878900', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 2827878900}, {'type': 'v1', 'addr': '172.21.15.28:6834', 'nonce': 2827878900}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 4978, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.81:6836/3612644344', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6834', 'nonce': 3612644344}, {'type': 'v1', 'addr': '172.21.15.81:6836', 'nonce': 3612644344}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 12}] 2024-02-21T23:04:32.863 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-02-21T23:04:32.863 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-02-21T23:04:32.864 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-21T23:04:32.976 DEBUG:teuthology.orchestra.run.smithi081:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-21T23:04:33.092 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-02-21T23:04:33.092 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-21T23:04:33.092 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-21T23:04:33.148 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054608,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-02-21T23:04:33.148 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-02-21T23:04:33.148 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-21T23:04:33.206 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053833,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498812k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249404k,nr_inodes=812351,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498812k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-02-21T23:04:33.206 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:04:33.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.442+0000 7fe504a00640 1 Processor -- start 2024-02-21T23:04:33.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.443+0000 7fe504a00640 1 -- start start 2024-02-21T23:04:33.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.444+0000 7fe504a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe50006dbf0 0x7fe500158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:33.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.444+0000 7fe504a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5000c74c0 con 0x7fe500155960 2024-02-21T23:04:33.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.444+0000 7fe504a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5000c77a0 con 0x7fe5000aff90 2024-02-21T23:04:33.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.444+0000 7fe504a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5000c7270 con 0x7fe50006dbf0 2024-02-21T23:04:33.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.444+0000 7fe4f7e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe50006dbf0 0x7fe500158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:33.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.445+0000 7fe4f7e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe50006dbf0 0x7fe500158ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48194/0 (socket says 172.21.15.28:48194) 2024-02-21T23:04:33.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.445+0000 7fe4f7e00640 1 -- 172.21.15.28:0/38562236 learned_addr learned my addr 172.21.15.28:0/38562236 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:33.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.445+0000 7fe4ff200640 1 --1- 172.21.15.28:0/38562236 >> v1:172.21.15.28:6789/0 conn(0x7fe500155960 0x7fe50015f890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39950/0 (socket says 172.21.15.28:39950) 2024-02-21T23:04:33.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.445+0000 7fe4f7e00640 1 -- 172.21.15.28:0/38562236 >> v1:172.21.15.81:6789/0 conn(0x7fe5000aff90 legacy=0x7fe5000c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:33.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.445+0000 7fe4f7e00640 1 -- 172.21.15.28:0/38562236 >> v1:172.21.15.28:6789/0 conn(0x7fe500155960 legacy=0x7fe50015f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:33.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.445+0000 7fe4f7e00640 1 -- 172.21.15.28:0/38562236 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5000c7a80 con 0x7fe50006dbf0 2024-02-21T23:04:33.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.446+0000 7fe4f7e00640 1 --2- 172.21.15.28:0/38562236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe50006dbf0 0x7fe500158ac0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fe4e8009870 tx=0x7fe4e802ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c5e06bf90a3fb51 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:33.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.447+0000 7fe4fde00640 1 -- 172.21.15.28:0/38562236 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4e802f850 con 0x7fe50006dbf0 2024-02-21T23:04:33.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.447+0000 7fe4fde00640 1 -- 172.21.15.28:0/38562236 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4e802f9f0 con 0x7fe50006dbf0 2024-02-21T23:04:33.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.447+0000 7fe4fde00640 1 -- 172.21.15.28:0/38562236 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4e802fdc0 con 0x7fe50006dbf0 2024-02-21T23:04:33.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.447+0000 7fe504a00640 1 -- 172.21.15.28:0/38562236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe50006dbf0 msgr2=0x7fe500158ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:33.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.447+0000 7fe504a00640 1 --2- 172.21.15.28:0/38562236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe50006dbf0 0x7fe500158ac0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fe4e8009870 tx=0x7fe4e802ba30 comp rx=0 tx=0).stop 2024-02-21T23:04:33.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.447+0000 7fe504a00640 1 -- 172.21.15.28:0/38562236 shutdown_connections 2024-02-21T23:04:33.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.447+0000 7fe504a00640 1 --2- 172.21.15.28:0/38562236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe50006dbf0 0x7fe500158ac0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:33.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.447+0000 7fe504a00640 1 -- 172.21.15.28:0/38562236 >> 172.21.15.28:0/38562236 conn(0x7fe5000ac370 msgr2=0x7fe5000ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:33.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.447+0000 7fe504a00640 1 -- 172.21.15.28:0/38562236 shutdown_connections 2024-02-21T23:04:33.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.448+0000 7fe504a00640 1 -- 172.21.15.28:0/38562236 wait complete. 2024-02-21T23:04:33.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.449+0000 7fe504a00640 1 Processor -- start 2024-02-21T23:04:33.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.449+0000 7fe504a00640 1 -- start start 2024-02-21T23:04:33.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.450+0000 7fe504a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe50006dbf0 0x7fe5000c12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:33.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.450+0000 7fe504a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe5000aff90 0x7fe5001eae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:33.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.450+0000 7fe504a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe500155960 0x7fe5001542a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:33.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.450+0000 7fe504a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5000c58f0 con 0x7fe50006dbf0 2024-02-21T23:04:33.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.450+0000 7fe504a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe500158630 con 0x7fe5000aff90 2024-02-21T23:04:33.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.450+0000 7fe504a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5000c5a70 con 0x7fe500155960 2024-02-21T23:04:33.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.451+0000 7fe4fe800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe50006dbf0 0x7fe5000c12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:33.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.451+0000 7fe4fe800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe50006dbf0 0x7fe5000c12d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48206/0 (socket says 172.21.15.28:48206) 2024-02-21T23:04:33.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.451+0000 7fe4fe800640 1 -- 172.21.15.28:0/1601062182 learned_addr learned my addr 172.21.15.28:0/1601062182 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:33.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.451+0000 7fe4f7e00640 1 --2- 172.21.15.28:0/1601062182 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe5000aff90 0x7fe5001eae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:33.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.451+0000 7fe4ff200640 1 --2- 172.21.15.28:0/1601062182 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe500155960 0x7fe5001542a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:33.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.451+0000 7fe4fe800640 1 -- 172.21.15.28:0/1601062182 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe500155960 msgr2=0x7fe5001542a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:33.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.451+0000 7fe4fe800640 1 --2- 172.21.15.28:0/1601062182 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe500155960 0x7fe5001542a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:33.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.451+0000 7fe4fe800640 1 -- 172.21.15.28:0/1601062182 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe5000aff90 msgr2=0x7fe5001eae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:33.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.451+0000 7fe4fe800640 1 --2- 172.21.15.28:0/1601062182 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe5000aff90 0x7fe5001eae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:33.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.451+0000 7fe4fe800640 1 -- 172.21.15.28:0/1601062182 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5000c4240 con 0x7fe50006dbf0 2024-02-21T23:04:33.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.451+0000 7fe4f7e00640 1 --2- 172.21.15.28:0/1601062182 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe5000aff90 0x7fe5001eae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:04:33.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.452+0000 7fe4fe800640 1 --2- 172.21.15.28:0/1601062182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe50006dbf0 0x7fe5000c12d0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fe4f000c480 tx=0x7fe4f000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:33.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.452+0000 7fe4f6a00640 1 -- 172.21.15.28:0/1601062182 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4f00052d0 con 0x7fe50006dbf0 2024-02-21T23:04:33.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.452+0000 7fe4f6a00640 1 -- 172.21.15.28:0/1601062182 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4f0005470 con 0x7fe50006dbf0 2024-02-21T23:04:33.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.452+0000 7fe504a00640 1 -- 172.21.15.28:0/1601062182 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5000c4520 con 0x7fe50006dbf0 2024-02-21T23:04:33.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.452+0000 7fe4f6a00640 1 -- 172.21.15.28:0/1601062182 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4f0012870 con 0x7fe50006dbf0 2024-02-21T23:04:33.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.452+0000 7fe504a00640 1 -- 172.21.15.28:0/1601062182 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5001547e0 con 0x7fe50006dbf0 2024-02-21T23:04:33.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.453+0000 7fe504a00640 1 -- 172.21.15.28:0/1601062182 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe50004f290 con 0x7fe50006dbf0 2024-02-21T23:04:33.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.454+0000 7fe4f6a00640 1 -- 172.21.15.28:0/1601062182 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe4f0012a10 con 0x7fe50006dbf0 2024-02-21T23:04:33.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.456+0000 7fe4f6a00640 1 --2- 172.21.15.28:0/1601062182 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe4dc06b500 0x7fe4dc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:33.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.456+0000 7fe4f6a00640 1 -- 172.21.15.28:0/1601062182 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6216+0+0 (secure 0 0 0) 0x7fe4f008a0f0 con 0x7fe50006dbf0 2024-02-21T23:04:33.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.456+0000 7fe4f7e00640 1 --2- 172.21.15.28:0/1601062182 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe4dc06b500 0x7fe4dc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:33.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.457+0000 7fe4f7e00640 1 --2- 172.21.15.28:0/1601062182 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe4dc06b500 0x7fe4dc06d9c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe4e8036000 tx=0x7fe4e8039040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:33.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.461+0000 7fe4f6a00640 1 -- 172.21.15.28:0/1601062182 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe4f0058d60 con 0x7fe50006dbf0 2024-02-21T23:04:33.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.728+0000 7fe504a00640 1 -- 172.21.15.28:0/1601062182 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe50006d9d0 con 0x7fe50006dbf0 2024-02-21T23:04:33.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.729+0000 7fe4f6a00640 1 -- 172.21.15.28:0/1601062182 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4934 (secure 0 0 0) 0x7fe4f0058660 con 0x7fe50006dbf0 2024-02-21T23:04:33.719 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:04:33.719 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":24,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4978,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/3612644344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":3612644344},{"type":"v1","addr":"172.21.15.81:6836","nonce":3612644344}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4981,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3623937361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":3623937361},{"type":"v1","addr":"172.21.15.81:6837","nonce":3623937361}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5442,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/33413357","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":33413357},{"type":"v1","addr":"172.21.15.28:6835","nonce":33413357}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:04:09.036302+0000","modified":"2024-02-21T23:04:26.710368+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4778},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4778":{"gid":4778,"name":"c","rank":0,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.28:6834/2827878900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2827878900},{"type":"v1","addr":"172.21.15.28:6834","nonce":2827878900}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-02-21T23:04:33.719 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 24 2024-02-21T23:04:33.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.732+0000 7fe504a00640 1 -- 172.21.15.28:0/1601062182 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe4dc06b500 msgr2=0x7fe4dc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:33.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.732+0000 7fe504a00640 1 --2- 172.21.15.28:0/1601062182 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe4dc06b500 0x7fe4dc06d9c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe4e8036000 tx=0x7fe4e8039040 comp rx=0 tx=0).stop 2024-02-21T23:04:33.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.732+0000 7fe504a00640 1 -- 172.21.15.28:0/1601062182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe50006dbf0 msgr2=0x7fe5000c12d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:33.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.732+0000 7fe504a00640 1 --2- 172.21.15.28:0/1601062182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe50006dbf0 0x7fe5000c12d0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fe4f000c480 tx=0x7fe4f000c940 comp rx=0 tx=0).stop 2024-02-21T23:04:33.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.732+0000 7fe504a00640 1 -- 172.21.15.28:0/1601062182 shutdown_connections 2024-02-21T23:04:33.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.732+0000 7fe504a00640 1 --2- 172.21.15.28:0/1601062182 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe4dc06b500 0x7fe4dc06d9c0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:33.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.732+0000 7fe504a00640 1 --2- 172.21.15.28:0/1601062182 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe500155960 0x7fe5001542a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:33.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.733+0000 7fe504a00640 1 --2- 172.21.15.28:0/1601062182 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe5000aff90 0x7fe5001eae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:33.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.733+0000 7fe504a00640 1 --2- 172.21.15.28:0/1601062182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe50006dbf0 0x7fe5000c12d0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:33.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.733+0000 7fe504a00640 1 -- 172.21.15.28:0/1601062182 >> 172.21.15.28:0/1601062182 conn(0x7fe5000ac370 msgr2=0x7fe500156d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:33.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.733+0000 7fe504a00640 1 -- 172.21.15.28:0/1601062182 shutdown_connections 2024-02-21T23:04:33.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.733+0000 7fe504a00640 1 -- 172.21.15.28:0/1601062182 wait complete. 2024-02-21T23:04:33.742 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:04:33.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.957+0000 7fe1baa00640 1 Processor -- start 2024-02-21T23:04:33.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.957+0000 7fe1baa00640 1 -- start start 2024-02-21T23:04:33.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.958+0000 7fe1baa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe1b4155960 0x7fe1b41608a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:33.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.959+0000 7fe1baa00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1b40c74c0 con 0x7fe1b40aff90 2024-02-21T23:04:33.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.959+0000 7fe1baa00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1b40c77a0 con 0x7fe1b406dbf0 2024-02-21T23:04:33.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.959+0000 7fe1baa00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1b40c7270 con 0x7fe1b4155960 2024-02-21T23:04:33.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.959+0000 7fe1ba000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe1b4155960 0x7fe1b41608a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:33.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.959+0000 7fe1b9600640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe1b40aff90 0x7fe1b40c51e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39952/0 (socket says 172.21.15.28:39952) 2024-02-21T23:04:33.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.959+0000 7fe1b9600640 1 -- 172.21.15.28:0/1502906788 learned_addr learned my addr 172.21.15.28:0/1502906788 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:04:33.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.959+0000 7fe1ba000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe1b4155960 0x7fe1b41608a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48222/0 (socket says 172.21.15.28:48222) 2024-02-21T23:04:33.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.959+0000 7fe1ba000640 1 -- 172.21.15.28:0/1502906788 >> v1:172.21.15.81:6789/0 conn(0x7fe1b406dbf0 legacy=0x7fe1b4158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:33.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.960+0000 7fe1ba000640 1 -- 172.21.15.28:0/1502906788 >> v1:172.21.15.28:6789/0 conn(0x7fe1b40aff90 legacy=0x7fe1b40c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:33.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.960+0000 7fe1ba000640 1 -- 172.21.15.28:0/1502906788 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1b40c7a80 con 0x7fe1b4155960 2024-02-21T23:04:33.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.960+0000 7fe1b3e00640 1 -- 172.21.15.28:0/1502906788 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 527985926 0 0) 0x7fe1b40c74c0 con 0x7fe1b40aff90 2024-02-21T23:04:33.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.961+0000 7fe1ba000640 1 --2- 172.21.15.28:0/1502906788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1b4155960 0x7fe1b41608a0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fe1ac009d90 tx=0x7fe1ac02bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfbaf0de354e9fbd server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:33.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.961+0000 7fe1b3e00640 1 -- 172.21.15.28:0/1502906788 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1ac039070 con 0x7fe1b4155960 2024-02-21T23:04:33.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.961+0000 7fe1b3e00640 1 -- 172.21.15.28:0/1502906788 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1ac02cd00 con 0x7fe1b4155960 2024-02-21T23:04:33.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.961+0000 7fe1b3e00640 1 -- 172.21.15.28:0/1502906788 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1ac034660 con 0x7fe1b4155960 2024-02-21T23:04:33.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.962+0000 7fe1baa00640 1 -- 172.21.15.28:0/1502906788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1b4155960 msgr2=0x7fe1b41608a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:33.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.962+0000 7fe1baa00640 1 --2- 172.21.15.28:0/1502906788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1b4155960 0x7fe1b41608a0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fe1ac009d90 tx=0x7fe1ac02bf50 comp rx=0 tx=0).stop 2024-02-21T23:04:33.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.962+0000 7fe1baa00640 1 -- 172.21.15.28:0/1502906788 shutdown_connections 2024-02-21T23:04:33.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.962+0000 7fe1baa00640 1 --2- 172.21.15.28:0/1502906788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1b4155960 0x7fe1b41608a0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:33.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.962+0000 7fe1baa00640 1 -- 172.21.15.28:0/1502906788 >> 172.21.15.28:0/1502906788 conn(0x7fe1b40ac370 msgr2=0x7fe1b40ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:33.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.962+0000 7fe1baa00640 1 -- 172.21.15.28:0/1502906788 shutdown_connections 2024-02-21T23:04:33.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.962+0000 7fe1baa00640 1 -- 172.21.15.28:0/1502906788 wait complete. 2024-02-21T23:04:33.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.964+0000 7fe1baa00640 1 Processor -- start 2024-02-21T23:04:33.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.964+0000 7fe1baa00640 1 -- start start 2024-02-21T23:04:33.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.965+0000 7fe1baa00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe1b406dbf0 0x7fe1b4150e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:33.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.965+0000 7fe1baa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1b40aff90 0x7fe1b414e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:33.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.965+0000 7fe1baa00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe1b4155960 0x7fe1b414eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:33.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.965+0000 7fe1baa00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b40c58f0 con 0x7fe1b40aff90 2024-02-21T23:04:33.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.965+0000 7fe1baa00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b4158630 con 0x7fe1b406dbf0 2024-02-21T23:04:33.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.965+0000 7fe1baa00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b40c5a70 con 0x7fe1b4155960 2024-02-21T23:04:33.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.965+0000 7fe1b8c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1b40aff90 0x7fe1b414e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:33.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.965+0000 7fe1b8c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1b40aff90 0x7fe1b414e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48224/0 (socket says 172.21.15.28:48224) 2024-02-21T23:04:33.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.965+0000 7fe1b8c00640 1 -- 172.21.15.28:0/1093126160 learned_addr learned my addr 172.21.15.28:0/1093126160 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:33.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.965+0000 7fe1b9600640 1 --2- 172.21.15.28:0/1093126160 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe1b406dbf0 0x7fe1b4150e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:33.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.965+0000 7fe1b8c00640 1 -- 172.21.15.28:0/1093126160 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe1b4155960 msgr2=0x7fe1b414eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:33.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.965+0000 7fe1b8c00640 1 --2- 172.21.15.28:0/1093126160 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe1b4155960 0x7fe1b414eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:33.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.965+0000 7fe1b8c00640 1 -- 172.21.15.28:0/1093126160 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe1b406dbf0 msgr2=0x7fe1b4150e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:33.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.965+0000 7fe1b8c00640 1 --2- 172.21.15.28:0/1093126160 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe1b406dbf0 0x7fe1b4150e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:33.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.965+0000 7fe1b8c00640 1 -- 172.21.15.28:0/1093126160 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1b41531d0 con 0x7fe1b40aff90 2024-02-21T23:04:33.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.965+0000 7fe1b9600640 1 --2- 172.21.15.28:0/1093126160 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe1b406dbf0 0x7fe1b4150e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:04:33.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.966+0000 7fe1b8c00640 1 --2- 172.21.15.28:0/1093126160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1b40aff90 0x7fe1b414e660 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fe1a000c400 tx=0x7fe1a000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:33.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.966+0000 7fe1b1600640 1 -- 172.21.15.28:0/1093126160 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1a0017070 con 0x7fe1b40aff90 2024-02-21T23:04:33.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.966+0000 7fe1b1600640 1 -- 172.21.15.28:0/1093126160 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1a00052d0 con 0x7fe1b40aff90 2024-02-21T23:04:33.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.966+0000 7fe1baa00640 1 -- 172.21.15.28:0/1093126160 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1b41534b0 con 0x7fe1b40aff90 2024-02-21T23:04:33.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.966+0000 7fe1b1600640 1 -- 172.21.15.28:0/1093126160 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1a0003830 con 0x7fe1b40aff90 2024-02-21T23:04:33.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.966+0000 7fe1baa00640 1 -- 172.21.15.28:0/1093126160 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1b4153a70 con 0x7fe1b40aff90 2024-02-21T23:04:33.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.967+0000 7fe1baa00640 1 -- 172.21.15.28:0/1093126160 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1b404f290 con 0x7fe1b40aff90 2024-02-21T23:04:33.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.968+0000 7fe1b1600640 1 -- 172.21.15.28:0/1093126160 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe1a001a070 con 0x7fe1b40aff90 2024-02-21T23:04:33.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.969+0000 7fe1b1600640 1 --2- 172.21.15.28:0/1093126160 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe19406b4d0 0x7fe19406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:33.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.969+0000 7fe1b1600640 1 -- 172.21.15.28:0/1093126160 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6216+0+0 (secure 0 0 0) 0x7fe1a0089810 con 0x7fe1b40aff90 2024-02-21T23:04:33.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.970+0000 7fe1b9600640 1 --2- 172.21.15.28:0/1093126160 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe19406b4d0 0x7fe19406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:33.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.971+0000 7fe1b9600640 1 --2- 172.21.15.28:0/1093126160 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe19406b4d0 0x7fe19406d990 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fe1a8000f60 tx=0x7fe1a8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:33.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:33.974+0000 7fe1b1600640 1 -- 172.21.15.28:0/1093126160 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe1a0058480 con 0x7fe1b40aff90 2024-02-21T23:04:34.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.175+0000 7fe1baa00640 1 -- 172.21.15.28:0/1093126160 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe1b406d9d0 con 0x7fe1b40aff90 2024-02-21T23:04:34.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.178+0000 7fe1b1600640 1 -- 172.21.15.28:0/1093126160 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+17004 (secure 0 0 0) 0x7fe1a0057d80 con 0x7fe1b40aff90 2024-02-21T23:04:34.168 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:04:34.168 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":41,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:04:24.717066+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:04:06.977614+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-02-21T23:04:08.074258+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.36","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6835/1213702315":"2024-02-22T23:04:20.686099+0000","172.21.15.28:6833/1213702315":"2024-02-22T23:04:20.686099+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:04:34.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.181+0000 7fe1baa00640 1 -- 172.21.15.28:0/1093126160 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe19406b4d0 msgr2=0x7fe19406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:34.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.181+0000 7fe1baa00640 1 --2- 172.21.15.28:0/1093126160 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe19406b4d0 0x7fe19406d990 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fe1a8000f60 tx=0x7fe1a8009040 comp rx=0 tx=0).stop 2024-02-21T23:04:34.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.181+0000 7fe1baa00640 1 -- 172.21.15.28:0/1093126160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1b40aff90 msgr2=0x7fe1b414e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:34.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.181+0000 7fe1baa00640 1 --2- 172.21.15.28:0/1093126160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1b40aff90 0x7fe1b414e660 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fe1a000c400 tx=0x7fe1a000c8c0 comp rx=0 tx=0).stop 2024-02-21T23:04:34.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.181+0000 7fe1baa00640 1 -- 172.21.15.28:0/1093126160 shutdown_connections 2024-02-21T23:04:34.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.181+0000 7fe1baa00640 1 --2- 172.21.15.28:0/1093126160 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe19406b4d0 0x7fe19406d990 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:34.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.181+0000 7fe1baa00640 1 --2- 172.21.15.28:0/1093126160 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe1b4155960 0x7fe1b414eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:34.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.181+0000 7fe1baa00640 1 --2- 172.21.15.28:0/1093126160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1b40aff90 0x7fe1b414e660 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:34.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.181+0000 7fe1baa00640 1 --2- 172.21.15.28:0/1093126160 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe1b406dbf0 0x7fe1b4150e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:34.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.181+0000 7fe1baa00640 1 -- 172.21.15.28:0/1093126160 >> 172.21.15.28:0/1093126160 conn(0x7fe1b40ac370 msgr2=0x7fe1b40c51e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:34.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.181+0000 7fe1baa00640 1 -- 172.21.15.28:0/1093126160 shutdown_connections 2024-02-21T23:04:34.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.181+0000 7fe1baa00640 1 -- 172.21.15.28:0/1093126160 wait complete. 2024-02-21T23:04:34.190 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-21T23:02:30.728329+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-21T23:04:06.977614+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '37', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-02-21T23:04:08.074258+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-21T23:04:34.191 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-21T23:04:34.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.413+0000 7f9641e00640 1 Processor -- start 2024-02-21T23:04:34.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.413+0000 7f9641e00640 1 -- start start 2024-02-21T23:04:34.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.415+0000 7f9641e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f963c13e0c0 0x7f963c0c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:34.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.415+0000 7f9641e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f963c06dbc0 0x7f963c06c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:34.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.415+0000 7f9641e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f963c0c76f0 0x7f963c06c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:34.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.415+0000 7f9641e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f963c0c89a0 con 0x7f963c13e0c0 2024-02-21T23:04:34.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.415+0000 7f9641e00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f963c158550 con 0x7f963c06dbc0 2024-02-21T23:04:34.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.415+0000 7f9641e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f963c1587d0 con 0x7f963c0c76f0 2024-02-21T23:04:34.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.415+0000 7f9640a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f963c13e0c0 0x7f963c0c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:34.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.415+0000 7f9640a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f963c13e0c0 0x7f963c0c71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48238/0 (socket says 172.21.15.28:48238) 2024-02-21T23:04:34.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.415+0000 7f9640a00640 1 -- 172.21.15.28:0/3918778436 learned_addr learned my addr 172.21.15.28:0/3918778436 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:34.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.416+0000 7f963be00640 1 --2- 172.21.15.28:0/3918778436 >> v2:172.21.15.81:3300/0 conn(0x7f963c06dbc0 0x7f963c06c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:34.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.416+0000 7f9640a00640 1 -- 172.21.15.28:0/3918778436 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f963c0c76f0 msgr2=0x7f963c06c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:34.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.416+0000 7f9641400640 1 --2- 172.21.15.28:0/3918778436 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f963c0c76f0 0x7f963c06c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:34.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.416+0000 7f9640a00640 1 --2- 172.21.15.28:0/3918778436 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f963c0c76f0 0x7f963c06c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:34.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.416+0000 7f9640a00640 1 -- 172.21.15.28:0/3918778436 >> v2:172.21.15.81:3300/0 conn(0x7f963c06dbc0 msgr2=0x7f963c06c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:34.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.416+0000 7f9640a00640 1 --2- 172.21.15.28:0/3918778436 >> v2:172.21.15.81:3300/0 conn(0x7f963c06dbc0 0x7f963c06c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:34.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.416+0000 7f9640a00640 1 -- 172.21.15.28:0/3918778436 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f963c158ab0 con 0x7f963c13e0c0 2024-02-21T23:04:34.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.416+0000 7f963be00640 1 --2- 172.21.15.28:0/3918778436 >> v2:172.21.15.81:3300/0 conn(0x7f963c06dbc0 0x7f963c06c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:04:34.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.416+0000 7f9641400640 1 --2- 172.21.15.28:0/3918778436 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f963c0c76f0 0x7f963c06c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:04:34.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.417+0000 7f9640a00640 1 --2- 172.21.15.28:0/3918778436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f963c13e0c0 0x7f963c0c71b0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f963c0c7100 tx=0x7f962c02b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16ebedb2dad5a7e8 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:34.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.417+0000 7f963b400640 1 -- 172.21.15.28:0/3918778436 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f962c0317f0 con 0x7f963c13e0c0 2024-02-21T23:04:34.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.417+0000 7f963b400640 1 -- 172.21.15.28:0/3918778436 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f962c037070 con 0x7f963c13e0c0 2024-02-21T23:04:34.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.417+0000 7f963b400640 1 -- 172.21.15.28:0/3918778436 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f962c031bb0 con 0x7f963c13e0c0 2024-02-21T23:04:34.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.418+0000 7f9641e00640 1 -- 172.21.15.28:0/3918778436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f963c13e0c0 msgr2=0x7f963c0c71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:34.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.418+0000 7f9641e00640 1 --2- 172.21.15.28:0/3918778436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f963c13e0c0 0x7f963c0c71b0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f963c0c7100 tx=0x7f962c02b890 comp rx=0 tx=0).stop 2024-02-21T23:04:34.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.418+0000 7f9641e00640 1 -- 172.21.15.28:0/3918778436 shutdown_connections 2024-02-21T23:04:34.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.418+0000 7f9641e00640 1 --2- 172.21.15.28:0/3918778436 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f963c0c76f0 0x7f963c06c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:34.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.418+0000 7f9641e00640 1 --2- 172.21.15.28:0/3918778436 >> v2:172.21.15.81:3300/0 conn(0x7f963c06dbc0 0x7f963c06c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:34.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.418+0000 7f9641e00640 1 --2- 172.21.15.28:0/3918778436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f963c13e0c0 0x7f963c0c71b0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:34.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.418+0000 7f9641e00640 1 -- 172.21.15.28:0/3918778436 >> 172.21.15.28:0/3918778436 conn(0x7f963c073110 msgr2=0x7f963c0ac490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:34.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.418+0000 7f9641e00640 1 -- 172.21.15.28:0/3918778436 shutdown_connections 2024-02-21T23:04:34.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.418+0000 7f9641e00640 1 -- 172.21.15.28:0/3918778436 wait complete. 2024-02-21T23:04:34.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.421+0000 7f9641e00640 1 Processor -- start 2024-02-21T23:04:34.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.421+0000 7f9641e00640 1 -- start start 2024-02-21T23:04:34.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.422+0000 7f9641e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f963c06dbc0 0x7f963c1ebfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:34.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.422+0000 7f9641e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f963c0c76f0 0x7f963c1ec500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:34.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.422+0000 7f9641e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f963c13e0c0 0x7f963c138540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:34.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.422+0000 7f9641e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f963c06cd00 con 0x7f963c06dbc0 2024-02-21T23:04:34.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.422+0000 7f9641e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f963c159070 con 0x7f963c13e0c0 2024-02-21T23:04:34.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.422+0000 7f9641e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f963c06ce80 con 0x7f963c0c76f0 2024-02-21T23:04:34.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.422+0000 7f9640a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f963c06dbc0 0x7f963c1ebfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:34.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.422+0000 7f9640a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f963c06dbc0 0x7f963c1ebfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48250/0 (socket says 172.21.15.28:48250) 2024-02-21T23:04:34.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.422+0000 7f9640a00640 1 -- 172.21.15.28:0/1406694653 learned_addr learned my addr 172.21.15.28:0/1406694653 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:34.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.422+0000 7f963be00640 1 --2- 172.21.15.28:0/1406694653 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f963c0c76f0 0x7f963c1ec500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:34.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.422+0000 7f9641400640 1 --2- 172.21.15.28:0/1406694653 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f963c13e0c0 0x7f963c138540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:34.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.422+0000 7f9640a00640 1 -- 172.21.15.28:0/1406694653 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f963c0c76f0 msgr2=0x7f963c1ec500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:34.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.422+0000 7f9640a00640 1 --2- 172.21.15.28:0/1406694653 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f963c0c76f0 0x7f963c1ec500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:34.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.422+0000 7f9640a00640 1 -- 172.21.15.28:0/1406694653 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f963c13e0c0 msgr2=0x7f963c138540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:34.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.422+0000 7f9640a00640 1 --2- 172.21.15.28:0/1406694653 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f963c13e0c0 0x7f963c138540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:34.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.422+0000 7f9640a00640 1 -- 172.21.15.28:0/1406694653 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f963c134100 con 0x7f963c06dbc0 2024-02-21T23:04:34.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.422+0000 7f963be00640 1 --2- 172.21.15.28:0/1406694653 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f963c0c76f0 0x7f963c1ec500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:04:34.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.422+0000 7f9640a00640 1 --2- 172.21.15.28:0/1406694653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f963c06dbc0 0x7f963c1ebfc0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f962c000c00 tx=0x7f962c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:34.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.422+0000 7f9638c00640 1 -- 172.21.15.28:0/1406694653 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f962c032890 con 0x7f963c06dbc0 2024-02-21T23:04:34.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.422+0000 7f9638c00640 1 -- 172.21.15.28:0/1406694653 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f962c02e070 con 0x7f963c06dbc0 2024-02-21T23:04:34.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.422+0000 7f9638c00640 1 -- 172.21.15.28:0/1406694653 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f962c032c70 con 0x7f963c06dbc0 2024-02-21T23:04:34.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.422+0000 7f9641e00640 1 -- 172.21.15.28:0/1406694653 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f963c1ecc50 con 0x7f963c06dbc0 2024-02-21T23:04:34.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.423+0000 7f9641e00640 1 -- 172.21.15.28:0/1406694653 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f963c1343a0 con 0x7f963c06dbc0 2024-02-21T23:04:34.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.424+0000 7f9641e00640 1 -- 172.21.15.28:0/1406694653 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f963c04f290 con 0x7f963c06dbc0 2024-02-21T23:04:34.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.425+0000 7f9638c00640 1 -- 172.21.15.28:0/1406694653 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f962c04c060 con 0x7f963c06dbc0 2024-02-21T23:04:34.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.427+0000 7f9638c00640 1 --2- 172.21.15.28:0/1406694653 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f962006b4d0 0x7f962006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:34.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.427+0000 7f9638c00640 1 -- 172.21.15.28:0/1406694653 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6216+0+0 (secure 0 0 0) 0x7f962c0ac000 con 0x7f963c06dbc0 2024-02-21T23:04:34.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.427+0000 7f963be00640 1 --2- 172.21.15.28:0/1406694653 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f962006b4d0 0x7f962006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:34.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.428+0000 7f963be00640 1 --2- 172.21.15.28:0/1406694653 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f962006b4d0 0x7f962006d990 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f963c139f30 tx=0x7f9630008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:34.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.432+0000 7f9638c00640 1 -- 172.21.15.28:0/1406694653 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f962c040070 con 0x7f963c06dbc0 2024-02-21T23:04:34.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.633+0000 7f9641e00640 1 -- 172.21.15.28:0/1406694653 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f963c06d9a0 con 0x7f963c06dbc0 2024-02-21T23:04:34.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.634+0000 7f9638c00640 1 -- 172.21.15.28:0/1406694653 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v41) v1 ==== 93+0+10 (secure 0 0 0) 0x7f962c07ad10 con 0x7f963c06dbc0 2024-02-21T23:04:34.622 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-02-21T23:04:34.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.637+0000 7f9641e00640 1 -- 172.21.15.28:0/1406694653 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f962006b4d0 msgr2=0x7f962006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:34.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.637+0000 7f9641e00640 1 --2- 172.21.15.28:0/1406694653 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f962006b4d0 0x7f962006d990 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f963c139f30 tx=0x7f9630008040 comp rx=0 tx=0).stop 2024-02-21T23:04:34.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.637+0000 7f9641e00640 1 -- 172.21.15.28:0/1406694653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f963c06dbc0 msgr2=0x7f963c1ebfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:34.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.637+0000 7f9641e00640 1 --2- 172.21.15.28:0/1406694653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f963c06dbc0 0x7f963c1ebfc0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f962c000c00 tx=0x7f962c0046c0 comp rx=0 tx=0).stop 2024-02-21T23:04:34.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.637+0000 7f9641e00640 1 -- 172.21.15.28:0/1406694653 shutdown_connections 2024-02-21T23:04:34.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.637+0000 7f9641e00640 1 --2- 172.21.15.28:0/1406694653 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f962006b4d0 0x7f962006d990 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:34.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.637+0000 7f9641e00640 1 --2- 172.21.15.28:0/1406694653 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f963c0c76f0 0x7f963c1ec500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:34.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.637+0000 7f9641e00640 1 --2- 172.21.15.28:0/1406694653 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f963c13e0c0 0x7f963c138540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:34.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.637+0000 7f9641e00640 1 --2- 172.21.15.28:0/1406694653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f963c06dbc0 0x7f963c1ebfc0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:34.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.637+0000 7f9641e00640 1 -- 172.21.15.28:0/1406694653 >> 172.21.15.28:0/1406694653 conn(0x7f963c073110 msgr2=0x7f963c0c5ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:34.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.638+0000 7f9641e00640 1 -- 172.21.15.28:0/1406694653 shutdown_connections 2024-02-21T23:04:34.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.638+0000 7f9641e00640 1 -- 172.21.15.28:0/1406694653 wait complete. 2024-02-21T23:04:34.645 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-21T23:04:34.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.877+0000 7f9fe4000640 1 Processor -- start 2024-02-21T23:04:34.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.877+0000 7f9fe4000640 1 -- start start 2024-02-21T23:04:34.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.878+0000 7f9fe4000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f9fdc06f180 0x7f9fdc0ac500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:34.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.878+0000 7f9fe4000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9fdc072ec0 0x7f9fdc15fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:34.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.879+0000 7f9fe4000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fdc06f560 con 0x7f9fdc155b40 2024-02-21T23:04:34.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.879+0000 7f9fe4000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fdc0c2170 con 0x7f9fdc06f180 2024-02-21T23:04:34.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.879+0000 7f9fe4000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fdc072690 con 0x7f9fdc072ec0 2024-02-21T23:04:34.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.879+0000 7f9fe0c00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f9fdc155b40 0x7f9fdc06be80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39960/0 (socket says 172.21.15.28:39960) 2024-02-21T23:04:34.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.879+0000 7f9fe0c00640 1 -- 172.21.15.28:0/1744299934 learned_addr learned my addr 172.21.15.28:0/1744299934 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:04:34.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.879+0000 7f9fe1600640 1 --2- 172.21.15.28:0/1744299934 >> v2:172.21.15.81:3300/0 conn(0x7f9fdc06f180 0x7f9fdc0ac500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:34.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.879+0000 7f9fe2000640 1 --2- 172.21.15.28:0/1744299934 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9fdc072ec0 0x7f9fdc15fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:34.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.880+0000 7f9fdbe00640 1 -- 172.21.15.28:0/1744299934 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2538947706 0 0) 0x7f9fdc06f560 con 0x7f9fdc155b40 2024-02-21T23:04:34.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.880+0000 7f9fdbe00640 1 -- 172.21.15.28:0/1744299934 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fc0003520 con 0x7f9fdc155b40 2024-02-21T23:04:34.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.880+0000 7f9fdbe00640 1 -- 172.21.15.28:0/1744299934 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1387623257 0 0) 0x7f9fc0003520 con 0x7f9fdc155b40 2024-02-21T23:04:34.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.880+0000 7f9fdbe00640 1 -- 172.21.15.28:0/1744299934 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9fdc072ec0 msgr2=0x7f9fdc15fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:34.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.880+0000 7f9fdbe00640 1 --2- 172.21.15.28:0/1744299934 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9fdc072ec0 0x7f9fdc15fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:34.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.880+0000 7f9fdbe00640 1 -- 172.21.15.28:0/1744299934 >> v2:172.21.15.81:3300/0 conn(0x7f9fdc06f180 msgr2=0x7f9fdc0ac500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:34.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.880+0000 7f9fdbe00640 1 --2- 172.21.15.28:0/1744299934 >> v2:172.21.15.81:3300/0 conn(0x7f9fdc06f180 0x7f9fdc0ac500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:34.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.880+0000 7f9fdbe00640 1 -- 172.21.15.28:0/1744299934 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fdc072970 con 0x7f9fdc155b40 2024-02-21T23:04:34.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.880+0000 7f9fdbe00640 1 -- 172.21.15.28:0/1744299934 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f9fc8001670 con 0x7f9fdc155b40 2024-02-21T23:04:34.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.880+0000 7f9fdbe00640 1 -- 172.21.15.28:0/1744299934 >> v1:172.21.15.28:6789/0 conn(0x7f9fdc155b40 legacy=0x7f9fdc06be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:34.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.880+0000 7f9fdbe00640 1 --2- 172.21.15.28:0/1744299934 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9fc00039b0 0x7f9fc0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:34.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.880+0000 7f9fdbe00640 1 --2- 172.21.15.28:0/1744299934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9fc0004800 0x7f9fc0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:34.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.880+0000 7f9fdbe00640 1 --2- 172.21.15.28:0/1744299934 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9fdc072ec0 0x7f9fc00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:34.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.880+0000 7f9fdbe00640 1 -- 172.21.15.28:0/1744299934 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fdc0c2170 con 0x7f9fc0004800 2024-02-21T23:04:34.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.880+0000 7f9fdbe00640 1 -- 172.21.15.28:0/1744299934 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fdc072690 con 0x7f9fc00039b0 2024-02-21T23:04:34.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.880+0000 7f9fdbe00640 1 -- 172.21.15.28:0/1744299934 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fc000b9c0 con 0x7f9fdc072ec0 2024-02-21T23:04:34.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.881+0000 7f9fe0c00640 1 --2- 172.21.15.28:0/1744299934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9fc0004800 0x7f9fc0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:34.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.881+0000 7f9fe2000640 1 --2- 172.21.15.28:0/1744299934 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9fdc072ec0 0x7f9fc00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:34.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.881+0000 7f9fe0c00640 1 -- 172.21.15.28:0/1744299934 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9fdc072ec0 msgr2=0x7f9fc00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:34.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.881+0000 7f9fe1600640 1 --2- 172.21.15.28:0/1744299934 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9fc00039b0 0x7f9fc0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:34.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.881+0000 7f9fe0c00640 1 --2- 172.21.15.28:0/1744299934 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9fdc072ec0 0x7f9fc00051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:34.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.881+0000 7f9fe0c00640 1 -- 172.21.15.28:0/1744299934 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9fc00039b0 msgr2=0x7f9fc0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:34.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.881+0000 7f9fe0c00640 1 --2- 172.21.15.28:0/1744299934 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9fc00039b0 0x7f9fc0003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:34.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.881+0000 7f9fe0c00640 1 -- 172.21.15.28:0/1744299934 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fc0003520 con 0x7f9fc0004800 2024-02-21T23:04:34.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.882+0000 7f9fe0c00640 1 --2- 172.21.15.28:0/1744299934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9fc0004800 0x7f9fc0004c80 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc155f50 tx=0x7f9fc0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4eb14d88f9eedbeb server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:34.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.882+0000 7f9fdbe00640 1 -- 172.21.15.28:0/1744299934 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fc8007d60 con 0x7f9fc0004800 2024-02-21T23:04:34.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.883+0000 7f9fdbe00640 1 -- 172.21.15.28:0/1744299934 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fc8030070 con 0x7f9fc0004800 2024-02-21T23:04:34.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.883+0000 7f9fdbe00640 1 -- 172.21.15.28:0/1744299934 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fc8005d10 con 0x7f9fc0004800 2024-02-21T23:04:34.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.883+0000 7f9fe4000640 1 -- 172.21.15.28:0/1744299934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9fc0004800 msgr2=0x7f9fc0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:34.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.883+0000 7f9fe4000640 1 --2- 172.21.15.28:0/1744299934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9fc0004800 0x7f9fc0004c80 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc155f50 tx=0x7f9fc0005830 comp rx=0 tx=0).stop 2024-02-21T23:04:34.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.883+0000 7f9fe2000640 1 -- 172.21.15.28:0/1744299934 reap_dead start 2024-02-21T23:04:34.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.883+0000 7f9fe4000640 1 -- 172.21.15.28:0/1744299934 shutdown_connections 2024-02-21T23:04:34.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.883+0000 7f9fe4000640 1 -- 172.21.15.28:0/1744299934 >> 172.21.15.28:0/1744299934 conn(0x7f9fdc071b00 msgr2=0x7f9fdc071ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:34.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.883+0000 7f9fe4000640 1 -- 172.21.15.28:0/1744299934 shutdown_connections 2024-02-21T23:04:34.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.883+0000 7f9fe4000640 1 -- 172.21.15.28:0/1744299934 wait complete. 2024-02-21T23:04:34.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.885+0000 7f9fe4000640 1 Processor -- start 2024-02-21T23:04:34.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.885+0000 7f9fe4000640 1 -- start start 2024-02-21T23:04:34.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.886+0000 7f9fe4000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9fc0004800 0x7f9fdc152750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:34.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.886+0000 7f9fe4000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9fdc1edb60 0x7f9fdc1edfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:34.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.886+0000 7f9fe4000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9fdc1ea380 0x7f9fdc1ea7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:34.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.886+0000 7f9fe4000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fdc06c590 con 0x7f9fdc1ea380 2024-02-21T23:04:34.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.886+0000 7f9fe4000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fdc156020 con 0x7f9fdc1edb60 2024-02-21T23:04:34.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.886+0000 7f9fe4000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fdc06c710 con 0x7f9fc0004800 2024-02-21T23:04:34.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.886+0000 7f9fe2000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9fdc1ea380 0x7f9fdc1ea7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:34.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.887+0000 7f9fe2000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9fdc1ea380 0x7f9fdc1ea7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48256/0 (socket says 172.21.15.28:48256) 2024-02-21T23:04:34.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.887+0000 7f9fe2000640 1 -- 172.21.15.28:0/2566177446 learned_addr learned my addr 172.21.15.28:0/2566177446 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:34.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.887+0000 7f9fe1600640 1 --2- 172.21.15.28:0/2566177446 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9fc0004800 0x7f9fdc152750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:34.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.887+0000 7f9fe0c00640 1 --2- 172.21.15.28:0/2566177446 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9fdc1edb60 0x7f9fdc1edfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:34.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.887+0000 7f9fe2000640 1 -- 172.21.15.28:0/2566177446 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9fc0004800 msgr2=0x7f9fdc152750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:34.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.887+0000 7f9fe2000640 1 --2- 172.21.15.28:0/2566177446 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9fc0004800 0x7f9fdc152750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:34.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.887+0000 7f9fe2000640 1 -- 172.21.15.28:0/2566177446 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9fdc1edb60 msgr2=0x7f9fdc1edfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:34.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.887+0000 7f9fe2000640 1 --2- 172.21.15.28:0/2566177446 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9fdc1edb60 0x7f9fdc1edfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:34.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.887+0000 7f9fe2000640 1 -- 172.21.15.28:0/2566177446 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fdc1500f0 con 0x7f9fdc1ea380 2024-02-21T23:04:34.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.887+0000 7f9fe0c00640 1 --2- 172.21.15.28:0/2566177446 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9fdc1edb60 0x7f9fdc1edfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:04:34.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.887+0000 7f9fe2000640 1 --2- 172.21.15.28:0/2566177446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9fdc1ea380 0x7f9fdc1ea7e0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc071730 tx=0x7f9fd4010c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:34.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.888+0000 7f9fd9600640 1 -- 172.21.15.28:0/2566177446 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fd400f040 con 0x7f9fdc1ea380 2024-02-21T23:04:34.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.888+0000 7f9fd9600640 1 -- 172.21.15.28:0/2566177446 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fd400e8e0 con 0x7f9fdc1ea380 2024-02-21T23:04:34.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.888+0000 7f9fd9600640 1 -- 172.21.15.28:0/2566177446 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fd400ecd0 con 0x7f9fdc1ea380 2024-02-21T23:04:34.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.888+0000 7f9fe4000640 1 -- 172.21.15.28:0/2566177446 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fdc1503d0 con 0x7f9fdc1ea380 2024-02-21T23:04:34.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.888+0000 7f9fe4000640 1 -- 172.21.15.28:0/2566177446 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fdc158f10 con 0x7f9fdc1ea380 2024-02-21T23:04:34.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.889+0000 7f9fe4000640 1 -- 172.21.15.28:0/2566177446 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fdc068640 con 0x7f9fdc1ea380 2024-02-21T23:04:34.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.891+0000 7f9fd9600640 1 -- 172.21.15.28:0/2566177446 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9fd40253f0 con 0x7f9fdc1ea380 2024-02-21T23:04:34.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.892+0000 7f9fd9600640 1 --2- 172.21.15.28:0/2566177446 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9fbc06b4d0 0x7f9fbc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:34.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.892+0000 7f9fd9600640 1 -- 172.21.15.28:0/2566177446 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6216+0+0 (secure 0 0 0) 0x7f9fd408a6a0 con 0x7f9fdc1ea380 2024-02-21T23:04:34.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.893+0000 7f9fe1600640 1 --2- 172.21.15.28:0/2566177446 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9fbc06b4d0 0x7f9fbc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:34.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.894+0000 7f9fe1600640 1 --2- 172.21.15.28:0/2566177446 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9fbc06b4d0 0x7f9fbc06d990 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f9fcc0074a0 tx=0x7f9fcc00b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:34.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:34.897+0000 7f9fd9600640 1 -- 172.21.15.28:0/2566177446 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f9fd4018030 con 0x7f9fdc1ea380 2024-02-21T23:04:35.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.095+0000 7f9fe4000640 1 -- 172.21.15.28:0/2566177446 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9fdc06f9a0 con 0x7f9fdc1ea380 2024-02-21T23:04:35.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.095+0000 7f9fd9600640 1 -- 172.21.15.28:0/2566177446 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v41) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9fd4058cc0 con 0x7f9fdc1ea380 2024-02-21T23:04:35.084 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-02-21T23:04:35.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.099+0000 7f9fe4000640 1 -- 172.21.15.28:0/2566177446 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9fbc06b4d0 msgr2=0x7f9fbc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:35.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.099+0000 7f9fe4000640 1 --2- 172.21.15.28:0/2566177446 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9fbc06b4d0 0x7f9fbc06d990 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f9fcc0074a0 tx=0x7f9fcc00b040 comp rx=0 tx=0).stop 2024-02-21T23:04:35.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.099+0000 7f9fe4000640 1 -- 172.21.15.28:0/2566177446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9fdc1ea380 msgr2=0x7f9fdc1ea7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:35.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.099+0000 7f9fe4000640 1 --2- 172.21.15.28:0/2566177446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9fdc1ea380 0x7f9fdc1ea7e0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc071730 tx=0x7f9fd4010c20 comp rx=0 tx=0).stop 2024-02-21T23:04:35.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.099+0000 7f9fe4000640 1 -- 172.21.15.28:0/2566177446 shutdown_connections 2024-02-21T23:04:35.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.099+0000 7f9fe4000640 1 --2- 172.21.15.28:0/2566177446 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9fbc06b4d0 0x7f9fbc06d990 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:35.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.099+0000 7f9fe4000640 1 --2- 172.21.15.28:0/2566177446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9fdc1ea380 0x7f9fdc1ea7e0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:35.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.099+0000 7f9fe4000640 1 --2- 172.21.15.28:0/2566177446 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9fdc1edb60 0x7f9fdc1edfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:35.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.099+0000 7f9fe4000640 1 --2- 172.21.15.28:0/2566177446 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9fc0004800 0x7f9fdc152750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:35.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.099+0000 7f9fe4000640 1 -- 172.21.15.28:0/2566177446 >> 172.21.15.28:0/2566177446 conn(0x7f9fdc071b00 msgr2=0x7f9fdc071ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:35.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.099+0000 7f9fe4000640 1 -- 172.21.15.28:0/2566177446 shutdown_connections 2024-02-21T23:04:35.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.099+0000 7f9fe4000640 1 -- 172.21.15.28:0/2566177446 wait complete. 2024-02-21T23:04:35.109 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-21T23:04:35.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.343+0000 7fbbed000640 1 Processor -- start 2024-02-21T23:04:35.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.344+0000 7fbbed000640 1 -- start start 2024-02-21T23:04:35.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.346+0000 7fbbed000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fbbe80ab9f0 0x7fbbe80abe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:35.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.347+0000 7fbbed000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbe80c2dd0 con 0x7fbbe80ac3c0 2024-02-21T23:04:35.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.347+0000 7fbbed000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbe80c30b0 con 0x7fbbe806c1f0 2024-02-21T23:04:35.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.347+0000 7fbbed000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbe80c3390 con 0x7fbbe80ab9f0 2024-02-21T23:04:35.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.347+0000 7fbbe7e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fbbe80ac3c0 0x7fbbe8155a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39880/0 (socket says 172.21.15.28:39880) 2024-02-21T23:04:35.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.347+0000 7fbbe7e00640 1 -- 172.21.15.28:0/1404461954 learned_addr learned my addr 172.21.15.28:0/1404461954 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:04:35.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.347+0000 7fbbe6a00640 1 --2- 172.21.15.28:0/1404461954 >> v2:172.21.15.81:3300/0 conn(0x7fbbe80ab9f0 0x7fbbe80abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:35.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.348+0000 7fbbe6000640 1 -- 172.21.15.28:0/1404461954 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2970822289 0 0) 0x7fbbe80c2dd0 con 0x7fbbe80ac3c0 2024-02-21T23:04:35.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.348+0000 7fbbe6000640 1 -- 172.21.15.28:0/1404461954 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbcc005730 con 0x7fbbe80ac3c0 2024-02-21T23:04:35.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.348+0000 7fbbe6000640 1 -- 172.21.15.28:0/1404461954 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2708031091 0 0) 0x7fbbcc005730 con 0x7fbbe80ac3c0 2024-02-21T23:04:35.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.348+0000 7fbbe6000640 1 -- 172.21.15.28:0/1404461954 >> v2:172.21.15.81:3300/0 conn(0x7fbbe80ab9f0 msgr2=0x7fbbe80abe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:35.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.348+0000 7fbbe6000640 1 --2- 172.21.15.28:0/1404461954 >> v2:172.21.15.81:3300/0 conn(0x7fbbe80ab9f0 0x7fbbe80abe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:35.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.348+0000 7fbbe6000640 1 -- 172.21.15.28:0/1404461954 >> v1:172.21.15.81:6789/0 conn(0x7fbbe806c1f0 legacy=0x7fbbe80c61c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:35.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.348+0000 7fbbe6000640 1 -- 172.21.15.28:0/1404461954 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbe80c3670 con 0x7fbbe80ac3c0 2024-02-21T23:04:35.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.348+0000 7fbbe6000640 1 -- 172.21.15.28:0/1404461954 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fbbe0001670 con 0x7fbbe80ac3c0 2024-02-21T23:04:35.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.348+0000 7fbbe6000640 1 -- 172.21.15.28:0/1404461954 >> v1:172.21.15.28:6789/0 conn(0x7fbbe80ac3c0 legacy=0x7fbbe8155a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:35.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.348+0000 7fbbe6000640 1 --2- 172.21.15.28:0/1404461954 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbbcc005c10 0x7fbbcc006030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:35.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.348+0000 7fbbe6000640 1 --2- 172.21.15.28:0/1404461954 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbcc006a10 0x7fbbcc006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:35.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.348+0000 7fbbe6000640 1 --2- 172.21.15.28:0/1404461954 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbbcc0073d0 0x7fbbcc00dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:35.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.348+0000 7fbbe6000640 1 -- 172.21.15.28:0/1404461954 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbe80c3390 con 0x7fbbcc006a10 2024-02-21T23:04:35.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.348+0000 7fbbe6000640 1 -- 172.21.15.28:0/1404461954 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbcc00e2d0 con 0x7fbbcc005c10 2024-02-21T23:04:35.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.348+0000 7fbbe6000640 1 -- 172.21.15.28:0/1404461954 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbcc00e4d0 con 0x7fbbcc0073d0 2024-02-21T23:04:35.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.349+0000 7fbbe6a00640 1 --2- 172.21.15.28:0/1404461954 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbcc006a10 0x7fbbcc006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:35.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.349+0000 7fbbe7e00640 1 --2- 172.21.15.28:0/1404461954 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbbcc0073d0 0x7fbbcc00dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:35.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.349+0000 7fbbe7400640 1 --2- 172.21.15.28:0/1404461954 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbbcc005c10 0x7fbbcc006030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:35.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.349+0000 7fbbe6a00640 1 -- 172.21.15.28:0/1404461954 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbbcc0073d0 msgr2=0x7fbbcc00dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:35.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.349+0000 7fbbe6a00640 1 --2- 172.21.15.28:0/1404461954 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbbcc0073d0 0x7fbbcc00dbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:35.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.349+0000 7fbbe6a00640 1 -- 172.21.15.28:0/1404461954 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbbcc005c10 msgr2=0x7fbbcc006030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:35.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.349+0000 7fbbe6a00640 1 --2- 172.21.15.28:0/1404461954 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbbcc005c10 0x7fbbcc006030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:35.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.349+0000 7fbbe6a00640 1 -- 172.21.15.28:0/1404461954 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbcc005730 con 0x7fbbcc006a10 2024-02-21T23:04:35.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.349+0000 7fbbe7e00640 1 -- 172.21.15.28:0/1404461954 reap_dead start 2024-02-21T23:04:35.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.350+0000 7fbbe6a00640 1 --2- 172.21.15.28:0/1404461954 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbcc006a10 0x7fbbcc006e90 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fbbd0009ac0 tx=0x7fbbcc007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc5c57f2d3b6275b server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:35.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.350+0000 7fbbe6000640 1 -- 172.21.15.28:0/1404461954 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbd0039070 con 0x7fbbcc006a10 2024-02-21T23:04:35.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.350+0000 7fbbe6000640 1 -- 172.21.15.28:0/1404461954 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbd002c6b0 con 0x7fbbcc006a10 2024-02-21T23:04:35.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.350+0000 7fbbe6000640 1 -- 172.21.15.28:0/1404461954 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbd0009690 con 0x7fbbcc006a10 2024-02-21T23:04:35.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.351+0000 7fbbed000640 1 -- 172.21.15.28:0/1404461954 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbcc006a10 msgr2=0x7fbbcc006e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:35.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.351+0000 7fbbed000640 1 --2- 172.21.15.28:0/1404461954 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbcc006a10 0x7fbbcc006e90 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fbbd0009ac0 tx=0x7fbbcc007a40 comp rx=0 tx=0).stop 2024-02-21T23:04:35.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.351+0000 7fbbed000640 1 -- 172.21.15.28:0/1404461954 shutdown_connections 2024-02-21T23:04:35.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.351+0000 7fbbed000640 1 --2- 172.21.15.28:0/1404461954 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbcc006a10 0x7fbbcc006e90 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:35.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.351+0000 7fbbed000640 1 -- 172.21.15.28:0/1404461954 >> 172.21.15.28:0/1404461954 conn(0x7fbbe80a9700 msgr2=0x7fbbe80a9ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:35.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.351+0000 7fbbed000640 1 -- 172.21.15.28:0/1404461954 shutdown_connections 2024-02-21T23:04:35.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.351+0000 7fbbed000640 1 -- 172.21.15.28:0/1404461954 wait complete. 2024-02-21T23:04:35.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.354+0000 7fbbed000640 1 Processor -- start 2024-02-21T23:04:35.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.354+0000 7fbbed000640 1 -- start start 2024-02-21T23:04:35.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.356+0000 7fbbed000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbbcc006a10 0x7fbbe80bee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:35.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.356+0000 7fbbed000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbe8151300 0x7fbbe806ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:35.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.356+0000 7fbbed000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbbe81e6ca0 0x7fbbe80bd9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:35.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.356+0000 7fbbed000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbe806c750 con 0x7fbbe8151300 2024-02-21T23:04:35.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.356+0000 7fbbed000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbe806c5d0 con 0x7fbbe81e6ca0 2024-02-21T23:04:35.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.356+0000 7fbbed000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbe80abe80 con 0x7fbbcc006a10 2024-02-21T23:04:35.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.356+0000 7fbbe6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbe8151300 0x7fbbe806ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:35.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.356+0000 7fbbe6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbe8151300 0x7fbbe806ae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52362/0 (socket says 172.21.15.28:52362) 2024-02-21T23:04:35.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.356+0000 7fbbe6a00640 1 -- 172.21.15.28:0/1517804325 learned_addr learned my addr 172.21.15.28:0/1517804325 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:35.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.356+0000 7fbbe7400640 1 --2- 172.21.15.28:0/1517804325 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbbcc006a10 0x7fbbe80bee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:35.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.356+0000 7fbbe7e00640 1 --2- 172.21.15.28:0/1517804325 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbbe81e6ca0 0x7fbbe80bd9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:35.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.356+0000 7fbbe6a00640 1 -- 172.21.15.28:0/1517804325 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbbcc006a10 msgr2=0x7fbbe80bee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:35.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.356+0000 7fbbe6a00640 1 --2- 172.21.15.28:0/1517804325 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbbcc006a10 0x7fbbe80bee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:35.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.356+0000 7fbbe6a00640 1 -- 172.21.15.28:0/1517804325 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbbe81e6ca0 msgr2=0x7fbbe80bd9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:35.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.356+0000 7fbbe6a00640 1 --2- 172.21.15.28:0/1517804325 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbbe81e6ca0 0x7fbbe80bd9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:35.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.356+0000 7fbbe6a00640 1 -- 172.21.15.28:0/1517804325 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbe80bf780 con 0x7fbbe8151300 2024-02-21T23:04:35.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.356+0000 7fbbe7e00640 1 --2- 172.21.15.28:0/1517804325 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbbe81e6ca0 0x7fbbe80bd9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:04:35.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.357+0000 7fbbe6a00640 1 --2- 172.21.15.28:0/1517804325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbe8151300 0x7fbbe806ae70 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fbbd0000c00 tx=0x7fbbd0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:35.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.357+0000 7fbbdf400640 1 -- 172.21.15.28:0/1517804325 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbd0042020 con 0x7fbbe8151300 2024-02-21T23:04:35.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.357+0000 7fbbdf400640 1 -- 172.21.15.28:0/1517804325 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbd003a040 con 0x7fbbe8151300 2024-02-21T23:04:35.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.357+0000 7fbbed000640 1 -- 172.21.15.28:0/1517804325 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbe81e61c0 con 0x7fbbe8151300 2024-02-21T23:04:35.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.357+0000 7fbbdf400640 1 -- 172.21.15.28:0/1517804325 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbd0039070 con 0x7fbbe8151300 2024-02-21T23:04:35.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.357+0000 7fbbed000640 1 -- 172.21.15.28:0/1517804325 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbe81e6780 con 0x7fbbe8151300 2024-02-21T23:04:35.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.358+0000 7fbbed000640 1 -- 172.21.15.28:0/1517804325 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbe804f290 con 0x7fbbe8151300 2024-02-21T23:04:35.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.359+0000 7fbbdf400640 1 -- 172.21.15.28:0/1517804325 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbbd0045020 con 0x7fbbe8151300 2024-02-21T23:04:35.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.360+0000 7fbbdf400640 1 --2- 172.21.15.28:0/1517804325 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbbc406b4d0 0x7fbbc406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:35.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.361+0000 7fbbdf400640 1 -- 172.21.15.28:0/1517804325 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6216+0+0 (secure 0 0 0) 0x7fbbd00ab8d0 con 0x7fbbe8151300 2024-02-21T23:04:35.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.361+0000 7fbbe7400640 1 --2- 172.21.15.28:0/1517804325 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbbc406b4d0 0x7fbbc406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:35.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.362+0000 7fbbe7400640 1 --2- 172.21.15.28:0/1517804325 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbbc406b4d0 0x7fbbc406d990 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fbbd8007fd0 tx=0x7fbbd8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:35.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.370+0000 7fbbdf400640 1 -- 172.21.15.28:0/1517804325 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fbbd004c070 con 0x7fbbe8151300 2024-02-21T23:04:35.453 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-21T23:04:35.460 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-21T23:04:35.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.577+0000 7fbbed000640 1 -- 172.21.15.28:0/1517804325 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fbbe80c5fa0 con 0x7fbbe8151300 2024-02-21T23:04:35.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.578+0000 7fbbdf400640 1 -- 172.21.15.28:0/1517804325 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v41) v1 ==== 100+0+11 (secure 0 0 0) 0x7fbbd007ee20 con 0x7fbbe8151300 2024-02-21T23:04:35.566 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-02-21T23:04:35.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.580+0000 7fbbed000640 1 -- 172.21.15.28:0/1517804325 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbbc406b4d0 msgr2=0x7fbbc406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:35.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.580+0000 7fbbed000640 1 --2- 172.21.15.28:0/1517804325 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbbc406b4d0 0x7fbbc406d990 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fbbd8007fd0 tx=0x7fbbd8009040 comp rx=0 tx=0).stop 2024-02-21T23:04:35.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.580+0000 7fbbed000640 1 -- 172.21.15.28:0/1517804325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbe8151300 msgr2=0x7fbbe806ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:35.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.580+0000 7fbbed000640 1 --2- 172.21.15.28:0/1517804325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbe8151300 0x7fbbe806ae70 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fbbd0000c00 tx=0x7fbbd0004730 comp rx=0 tx=0).stop 2024-02-21T23:04:35.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.580+0000 7fbbed000640 1 -- 172.21.15.28:0/1517804325 shutdown_connections 2024-02-21T23:04:35.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.580+0000 7fbbed000640 1 --2- 172.21.15.28:0/1517804325 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbbc406b4d0 0x7fbbc406d990 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:35.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.580+0000 7fbbed000640 1 --2- 172.21.15.28:0/1517804325 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbbe81e6ca0 0x7fbbe80bd9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:35.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.580+0000 7fbbed000640 1 --2- 172.21.15.28:0/1517804325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbe8151300 0x7fbbe806ae70 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:35.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.580+0000 7fbbed000640 1 --2- 172.21.15.28:0/1517804325 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbbcc006a10 0x7fbbe80bee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:35.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.580+0000 7fbbed000640 1 -- 172.21.15.28:0/1517804325 >> 172.21.15.28:0/1517804325 conn(0x7fbbe80a9700 msgr2=0x7fbbe80a7670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:35.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.581+0000 7fbbed000640 1 -- 172.21.15.28:0/1517804325 shutdown_connections 2024-02-21T23:04:35.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.581+0000 7fbbed000640 1 -- 172.21.15.28:0/1517804325 wait complete. 2024-02-21T23:04:35.589 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:04:35.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.864+0000 7ff17a800640 1 Processor -- start 2024-02-21T23:04:35.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.864+0000 7ff17a800640 1 -- start start 2024-02-21T23:04:35.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.867+0000 7ff17a800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff1741569b0 0x7ff174159db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:35.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.867+0000 7ff17a800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff174071070 0x7ff174071500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:35.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.867+0000 7ff17a800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1740c1390 con 0x7ff1740c0f00 2024-02-21T23:04:35.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.867+0000 7ff17a800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1740c1fa0 con 0x7ff174071070 2024-02-21T23:04:35.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.867+0000 7ff17a800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1740c1670 con 0x7ff1741569b0 2024-02-21T23:04:35.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.867+0000 7ff179e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff174071070 0x7ff174071500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:35.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.867+0000 7ff179e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff174071070 0x7ff174071500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52366/0 (socket says 172.21.15.28:52366) 2024-02-21T23:04:35.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.867+0000 7ff179e00640 1 -- 172.21.15.28:0/3675517950 learned_addr learned my addr 172.21.15.28:0/3675517950 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:35.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.867+0000 7ff179400640 1 --2- 172.21.15.28:0/3675517950 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff1741569b0 0x7ff174159db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:35.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.868+0000 7ff179e00640 1 -- 172.21.15.28:0/3675517950 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff1741569b0 msgr2=0x7ff174159db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:35.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.868+0000 7ff179e00640 1 --2- 172.21.15.28:0/3675517950 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff1741569b0 0x7ff174159db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:35.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.868+0000 7ff179e00640 1 -- 172.21.15.28:0/3675517950 >> v1:172.21.15.81:6789/0 conn(0x7ff1740c0f00 legacy=0x7ff17406dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:35.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.868+0000 7ff179e00640 1 -- 172.21.15.28:0/3675517950 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1740c1950 con 0x7ff174071070 2024-02-21T23:04:35.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.869+0000 7ff179e00640 1 --2- 172.21.15.28:0/3675517950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff174071070 0x7ff174071500 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7ff16c009e50 tx=0x7ff16c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a99ba39e2bb9e0d6 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:35.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.869+0000 7ff173e00640 1 -- 172.21.15.28:0/3675517950 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff16c02ca70 con 0x7ff174071070 2024-02-21T23:04:35.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.869+0000 7ff173e00640 1 -- 172.21.15.28:0/3675517950 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff16c02cc10 con 0x7ff174071070 2024-02-21T23:04:35.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.869+0000 7ff173e00640 1 -- 172.21.15.28:0/3675517950 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff16c0343f0 con 0x7ff174071070 2024-02-21T23:04:35.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.870+0000 7ff17a800640 1 -- 172.21.15.28:0/3675517950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff174071070 msgr2=0x7ff174071500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:35.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.870+0000 7ff17a800640 1 --2- 172.21.15.28:0/3675517950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff174071070 0x7ff174071500 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7ff16c009e50 tx=0x7ff16c02bed0 comp rx=0 tx=0).stop 2024-02-21T23:04:35.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.870+0000 7ff17a800640 1 -- 172.21.15.28:0/3675517950 shutdown_connections 2024-02-21T23:04:35.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.870+0000 7ff17a800640 1 --2- 172.21.15.28:0/3675517950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff174071070 0x7ff174071500 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:35.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.870+0000 7ff17a800640 1 --2- 172.21.15.28:0/3675517950 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff1741569b0 0x7ff174159db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:35.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.870+0000 7ff17a800640 1 -- 172.21.15.28:0/3675517950 >> 172.21.15.28:0/3675517950 conn(0x7ff17406b630 msgr2=0x7ff1740c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:35.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.870+0000 7ff17a800640 1 -- 172.21.15.28:0/3675517950 shutdown_connections 2024-02-21T23:04:35.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.870+0000 7ff17a800640 1 -- 172.21.15.28:0/3675517950 wait complete. 2024-02-21T23:04:35.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.873+0000 7ff17a800640 1 Processor -- start 2024-02-21T23:04:35.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.873+0000 7ff17a800640 1 -- start start 2024-02-21T23:04:35.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.875+0000 7ff17a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff174071070 0x7ff17415cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:35.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.875+0000 7ff17a800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff1741569b0 0x7ff174155580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:35.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.875+0000 7ff17a800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff1740a6e10 0x7ff174155ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:35.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.875+0000 7ff17a800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff17406e700 con 0x7ff174071070 2024-02-21T23:04:35.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.875+0000 7ff17a800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1740c2ba0 con 0x7ff1741569b0 2024-02-21T23:04:35.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.875+0000 7ff17a800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff17406e880 con 0x7ff1740a6e10 2024-02-21T23:04:35.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.875+0000 7ff179400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff174071070 0x7ff17415cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:35.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.875+0000 7ff179400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff174071070 0x7ff17415cad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52378/0 (socket says 172.21.15.28:52378) 2024-02-21T23:04:35.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.875+0000 7ff179400640 1 -- 172.21.15.28:0/3910474821 learned_addr learned my addr 172.21.15.28:0/3910474821 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:35.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.875+0000 7ff178a00640 1 --2- 172.21.15.28:0/3910474821 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff1741569b0 0x7ff174155580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:35.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.876+0000 7ff179e00640 1 --2- 172.21.15.28:0/3910474821 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff1740a6e10 0x7ff174155ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:35.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.876+0000 7ff179400640 1 -- 172.21.15.28:0/3910474821 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff1740a6e10 msgr2=0x7ff174155ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:35.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.876+0000 7ff179400640 1 --2- 172.21.15.28:0/3910474821 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff1740a6e10 0x7ff174155ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:35.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.876+0000 7ff179400640 1 -- 172.21.15.28:0/3910474821 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff1741569b0 msgr2=0x7ff174155580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:35.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.876+0000 7ff179400640 1 --2- 172.21.15.28:0/3910474821 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff1741569b0 0x7ff174155580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:35.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.876+0000 7ff179400640 1 -- 172.21.15.28:0/3910474821 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1741f3540 con 0x7ff174071070 2024-02-21T23:04:35.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.876+0000 7ff179400640 1 --2- 172.21.15.28:0/3910474821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff174071070 0x7ff17415cad0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7ff16800d5f0 tx=0x7ff16800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:35.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.876+0000 7ff171600640 1 -- 172.21.15.28:0/3910474821 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1680125f0 con 0x7ff174071070 2024-02-21T23:04:35.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.876+0000 7ff171600640 1 -- 172.21.15.28:0/3910474821 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff168019070 con 0x7ff174071070 2024-02-21T23:04:35.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.877+0000 7ff17a800640 1 -- 172.21.15.28:0/3910474821 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1741e4350 con 0x7ff174071070 2024-02-21T23:04:35.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.877+0000 7ff171600640 1 -- 172.21.15.28:0/3910474821 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1680129d0 con 0x7ff174071070 2024-02-21T23:04:35.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.877+0000 7ff17a800640 1 -- 172.21.15.28:0/3910474821 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff17415d010 con 0x7ff174071070 2024-02-21T23:04:35.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.878+0000 7ff17a800640 1 -- 172.21.15.28:0/3910474821 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff17404f290 con 0x7ff174071070 2024-02-21T23:04:35.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.879+0000 7ff171600640 1 -- 172.21.15.28:0/3910474821 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff168012b70 con 0x7ff174071070 2024-02-21T23:04:35.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.880+0000 7ff171600640 1 --2- 172.21.15.28:0/3910474821 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff15406b500 0x7ff15406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:35.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.880+0000 7ff171600640 1 -- 172.21.15.28:0/3910474821 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6216+0+0 (secure 0 0 0) 0x7ff16808a740 con 0x7ff174071070 2024-02-21T23:04:35.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.880+0000 7ff178a00640 1 --2- 172.21.15.28:0/3910474821 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff15406b500 0x7ff15406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:35.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.881+0000 7ff178a00640 1 --2- 172.21.15.28:0/3910474821 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff15406b500 0x7ff15406d9c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff174072870 tx=0x7ff160009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:35.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:35.884+0000 7ff171600640 1 -- 172.21.15.28:0/3910474821 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ff168059450 con 0x7ff174071070 2024-02-21T23:04:36.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.122+0000 7ff17a800640 1 -- 172.21.15.28:0/3910474821 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff174073540 con 0x7ff174071070 2024-02-21T23:04:36.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.123+0000 7ff171600640 1 -- 172.21.15.28:0/3910474821 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4934 (secure 0 0 0) 0x7ff168058d50 con 0x7ff174071070 2024-02-21T23:04:36.112 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:04:36.112 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":24,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4978,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/3612644344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":3612644344},{"type":"v1","addr":"172.21.15.81:6836","nonce":3612644344}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4981,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3623937361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":3623937361},{"type":"v1","addr":"172.21.15.81:6837","nonce":3623937361}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5442,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/33413357","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":33413357},{"type":"v1","addr":"172.21.15.28:6835","nonce":33413357}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:04:09.036302+0000","modified":"2024-02-21T23:04:26.710368+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4778},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4778":{"gid":4778,"name":"c","rank":0,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.28:6834/2827878900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2827878900},{"type":"v1","addr":"172.21.15.28:6834","nonce":2827878900}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-02-21T23:04:36.112 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 24 2024-02-21T23:04:36.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.125+0000 7ff17a800640 1 -- 172.21.15.28:0/3910474821 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff15406b500 msgr2=0x7ff15406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:36.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.126+0000 7ff17a800640 1 --2- 172.21.15.28:0/3910474821 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff15406b500 0x7ff15406d9c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff174072870 tx=0x7ff160009040 comp rx=0 tx=0).stop 2024-02-21T23:04:36.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.126+0000 7ff17a800640 1 -- 172.21.15.28:0/3910474821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff174071070 msgr2=0x7ff17415cad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:36.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.126+0000 7ff17a800640 1 --2- 172.21.15.28:0/3910474821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff174071070 0x7ff17415cad0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7ff16800d5f0 tx=0x7ff16800dab0 comp rx=0 tx=0).stop 2024-02-21T23:04:36.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.126+0000 7ff17a800640 1 -- 172.21.15.28:0/3910474821 shutdown_connections 2024-02-21T23:04:36.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.126+0000 7ff17a800640 1 --2- 172.21.15.28:0/3910474821 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff15406b500 0x7ff15406d9c0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:36.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.126+0000 7ff17a800640 1 --2- 172.21.15.28:0/3910474821 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff1740a6e10 0x7ff174155ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:36.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.126+0000 7ff17a800640 1 --2- 172.21.15.28:0/3910474821 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff1741569b0 0x7ff174155580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:36.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.126+0000 7ff17a800640 1 --2- 172.21.15.28:0/3910474821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff174071070 0x7ff17415cad0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:36.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.126+0000 7ff17a800640 1 -- 172.21.15.28:0/3910474821 >> 172.21.15.28:0/3910474821 conn(0x7ff17406b630 msgr2=0x7ff17406dff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:36.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.126+0000 7ff17a800640 1 -- 172.21.15.28:0/3910474821 shutdown_connections 2024-02-21T23:04:36.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.126+0000 7ff17a800640 1 -- 172.21.15.28:0/3910474821 wait complete. 2024-02-21T23:04:36.132 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:04:36.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.357+0000 7fb4a2400640 1 Processor -- start 2024-02-21T23:04:36.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.357+0000 7fb4a2400640 1 -- start start 2024-02-21T23:04:36.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.359+0000 7fb4a2400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fb49c0701e0 0x7fb49c151de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:36.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.359+0000 7fb4a2400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb49c0c5910 con 0x7fb49c06fe00 2024-02-21T23:04:36.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.359+0000 7fb4a2400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb49c0c5bf0 con 0x7fb49c0a6d30 2024-02-21T23:04:36.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.359+0000 7fb4a2400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb49c0c5ed0 con 0x7fb49c0701e0 2024-02-21T23:04:36.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.359+0000 7fb49b400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb49c06fe00 0x7fb49c0c6e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39892/0 (socket says 172.21.15.28:39892) 2024-02-21T23:04:36.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.359+0000 7fb49b400640 1 -- 172.21.15.28:0/2017216546 learned_addr learned my addr 172.21.15.28:0/2017216546 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:04:36.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.359+0000 7fb49be00640 1 --2- 172.21.15.28:0/2017216546 >> v2:172.21.15.81:3300/0 conn(0x7fb49c0701e0 0x7fb49c151de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:36.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.359+0000 7fb49aa00640 1 -- 172.21.15.28:0/2017216546 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2837377370 0 0) 0x7fb49c0c5910 con 0x7fb49c06fe00 2024-02-21T23:04:36.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.359+0000 7fb49aa00640 1 -- 172.21.15.28:0/2017216546 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb480003520 con 0x7fb49c06fe00 2024-02-21T23:04:36.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.360+0000 7fb49aa00640 1 -- 172.21.15.28:0/2017216546 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3633533603 0 0) 0x7fb480003520 con 0x7fb49c06fe00 2024-02-21T23:04:36.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.360+0000 7fb49aa00640 1 -- 172.21.15.28:0/2017216546 >> v2:172.21.15.81:3300/0 conn(0x7fb49c0701e0 msgr2=0x7fb49c151de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:36.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.360+0000 7fb49aa00640 1 --2- 172.21.15.28:0/2017216546 >> v2:172.21.15.81:3300/0 conn(0x7fb49c0701e0 0x7fb49c151de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:36.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.360+0000 7fb49aa00640 1 -- 172.21.15.28:0/2017216546 >> v1:172.21.15.81:6789/0 conn(0x7fb49c0a6d30 legacy=0x7fb49c0c26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:36.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.360+0000 7fb49aa00640 1 -- 172.21.15.28:0/2017216546 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb49c0c61b0 con 0x7fb49c06fe00 2024-02-21T23:04:36.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.360+0000 7fb49aa00640 1 -- 172.21.15.28:0/2017216546 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fb48c001670 con 0x7fb49c06fe00 2024-02-21T23:04:36.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.360+0000 7fb49aa00640 1 -- 172.21.15.28:0/2017216546 >> v1:172.21.15.28:6789/0 conn(0x7fb49c06fe00 legacy=0x7fb49c0c6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:36.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.360+0000 7fb49aa00640 1 --2- 172.21.15.28:0/2017216546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4800039b0 0x7fb480003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:36.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.360+0000 7fb49aa00640 1 --2- 172.21.15.28:0/2017216546 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb480004800 0x7fb480004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:36.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.360+0000 7fb49aa00640 1 --2- 172.21.15.28:0/2017216546 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb4800051c0 0x7fb48000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:36.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.360+0000 7fb49aa00640 1 -- 172.21.15.28:0/2017216546 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb49c0c5ed0 con 0x7fb4800039b0 2024-02-21T23:04:36.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.360+0000 7fb49aa00640 1 -- 172.21.15.28:0/2017216546 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb48000c060 con 0x7fb4800051c0 2024-02-21T23:04:36.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.360+0000 7fb49aa00640 1 -- 172.21.15.28:0/2017216546 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb48000e470 con 0x7fb480004800 2024-02-21T23:04:36.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.360+0000 7fb49b400640 1 --2- 172.21.15.28:0/2017216546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4800039b0 0x7fb480003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:36.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.361+0000 7fb492a00640 1 --2- 172.21.15.28:0/2017216546 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb480004800 0x7fb480004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:36.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.361+0000 7fb49be00640 1 --2- 172.21.15.28:0/2017216546 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb4800051c0 0x7fb48000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:36.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.361+0000 7fb49b400640 1 -- 172.21.15.28:0/2017216546 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb480004800 msgr2=0x7fb480004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:36.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.361+0000 7fb49b400640 1 --2- 172.21.15.28:0/2017216546 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb480004800 0x7fb480004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:36.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.361+0000 7fb49b400640 1 -- 172.21.15.28:0/2017216546 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb4800051c0 msgr2=0x7fb48000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:36.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.361+0000 7fb49b400640 1 --2- 172.21.15.28:0/2017216546 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb4800051c0 0x7fb48000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:36.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.361+0000 7fb49b400640 1 -- 172.21.15.28:0/2017216546 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb480003520 con 0x7fb4800039b0 2024-02-21T23:04:36.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.361+0000 7fb49be00640 1 -- 172.21.15.28:0/2017216546 reap_dead start 2024-02-21T23:04:36.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.362+0000 7fb49b400640 1 --2- 172.21.15.28:0/2017216546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4800039b0 0x7fb480003dd0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fb48c00c240 tx=0x7fb480005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ecd01714effe08e server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:36.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.362+0000 7fb49aa00640 1 -- 172.21.15.28:0/2017216546 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb48c007cf0 con 0x7fb4800039b0 2024-02-21T23:04:36.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.362+0000 7fb49aa00640 1 -- 172.21.15.28:0/2017216546 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb48c030070 con 0x7fb4800039b0 2024-02-21T23:04:36.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.362+0000 7fb49aa00640 1 -- 172.21.15.28:0/2017216546 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb48c005d30 con 0x7fb4800039b0 2024-02-21T23:04:36.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.363+0000 7fb4a2400640 1 -- 172.21.15.28:0/2017216546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4800039b0 msgr2=0x7fb480003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:36.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.363+0000 7fb4a2400640 1 --2- 172.21.15.28:0/2017216546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4800039b0 0x7fb480003dd0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fb48c00c240 tx=0x7fb480005830 comp rx=0 tx=0).stop 2024-02-21T23:04:36.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.363+0000 7fb4a2400640 1 -- 172.21.15.28:0/2017216546 shutdown_connections 2024-02-21T23:04:36.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.363+0000 7fb4a2400640 1 --2- 172.21.15.28:0/2017216546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4800039b0 0x7fb480003dd0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:36.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.363+0000 7fb4a2400640 1 -- 172.21.15.28:0/2017216546 >> 172.21.15.28:0/2017216546 conn(0x7fb49c06c5b0 msgr2=0x7fb49c0c7800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:36.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.363+0000 7fb4a2400640 1 -- 172.21.15.28:0/2017216546 shutdown_connections 2024-02-21T23:04:36.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.363+0000 7fb4a2400640 1 -- 172.21.15.28:0/2017216546 wait complete. 2024-02-21T23:04:36.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.365+0000 7fb4a2400640 1 Processor -- start 2024-02-21T23:04:36.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.365+0000 7fb4a2400640 1 -- start start 2024-02-21T23:04:36.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.366+0000 7fb4a2400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb4800039b0 0x7fb49c0a97a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:36.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.366+0000 7fb4a2400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb49c151390 0x7fb49c151810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:36.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.366+0000 7fb4a2400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb49c1e9c20 0x7fb49c1ea0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:36.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.366+0000 7fb4a2400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb49c0c71a0 con 0x7fb49c1e9c20 2024-02-21T23:04:36.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.366+0000 7fb4a2400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb49c0c7020 con 0x7fb4800039b0 2024-02-21T23:04:36.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.366+0000 7fb4a2400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb49c0c7320 con 0x7fb49c151390 2024-02-21T23:04:36.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.367+0000 7fb49be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb49c1e9c20 0x7fb49c1ea0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:36.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.367+0000 7fb49be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb49c1e9c20 0x7fb49c1ea0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52396/0 (socket says 172.21.15.28:52396) 2024-02-21T23:04:36.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.367+0000 7fb49be00640 1 -- 172.21.15.28:0/3433842581 learned_addr learned my addr 172.21.15.28:0/3433842581 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:36.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.367+0000 7fb49b400640 1 --2- 172.21.15.28:0/3433842581 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb4800039b0 0x7fb49c0a97a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:36.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.367+0000 7fb492a00640 1 --2- 172.21.15.28:0/3433842581 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb49c151390 0x7fb49c151810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:36.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.367+0000 7fb49be00640 1 -- 172.21.15.28:0/3433842581 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb49c151390 msgr2=0x7fb49c151810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:36.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.367+0000 7fb49be00640 1 --2- 172.21.15.28:0/3433842581 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb49c151390 0x7fb49c151810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:36.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.367+0000 7fb49be00640 1 -- 172.21.15.28:0/3433842581 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb4800039b0 msgr2=0x7fb49c0a97a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:36.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.367+0000 7fb49be00640 1 --2- 172.21.15.28:0/3433842581 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb4800039b0 0x7fb49c0a97a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:36.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.367+0000 7fb49be00640 1 -- 172.21.15.28:0/3433842581 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb49c152570 con 0x7fb49c1e9c20 2024-02-21T23:04:36.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.368+0000 7fb49be00640 1 --2- 172.21.15.28:0/3433842581 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb49c1e9c20 0x7fb49c1ea0d0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fb49c06e660 tx=0x7fb4940115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:36.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.368+0000 7fb493e00640 1 -- 172.21.15.28:0/3433842581 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb494011e10 con 0x7fb49c1e9c20 2024-02-21T23:04:36.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.368+0000 7fb493e00640 1 -- 172.21.15.28:0/3433842581 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb49401b070 con 0x7fb49c1e9c20 2024-02-21T23:04:36.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.368+0000 7fb493e00640 1 -- 172.21.15.28:0/3433842581 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb494003990 con 0x7fb49c1e9c20 2024-02-21T23:04:36.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.368+0000 7fb4a2400640 1 -- 172.21.15.28:0/3433842581 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb49c1e74a0 con 0x7fb49c1e9c20 2024-02-21T23:04:36.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.368+0000 7fb4a2400640 1 -- 172.21.15.28:0/3433842581 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb49c0a9ce0 con 0x7fb49c1e9c20 2024-02-21T23:04:36.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.369+0000 7fb4a2400640 1 -- 172.21.15.28:0/3433842581 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb49c068640 con 0x7fb49c1e9c20 2024-02-21T23:04:36.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.371+0000 7fb493e00640 1 -- 172.21.15.28:0/3433842581 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb494005d10 con 0x7fb49c1e9c20 2024-02-21T23:04:36.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.372+0000 7fb493e00640 1 --2- 172.21.15.28:0/3433842581 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb47806b500 0x7fb47806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:36.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.372+0000 7fb493e00640 1 -- 172.21.15.28:0/3433842581 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6216+0+0 (secure 0 0 0) 0x7fb49408a660 con 0x7fb49c1e9c20 2024-02-21T23:04:36.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.373+0000 7fb49b400640 1 --2- 172.21.15.28:0/3433842581 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb47806b500 0x7fb47806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:36.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.374+0000 7fb49b400640 1 --2- 172.21.15.28:0/3433842581 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb47806b500 0x7fb47806d9c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fb48c034000 tx=0x7fb48c007000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:36.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.377+0000 7fb493e00640 1 -- 172.21.15.28:0/3433842581 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb494058c80 con 0x7fb49c1e9c20 2024-02-21T23:04:36.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.577+0000 7fb4a2400640 1 -- 172.21.15.28:0/3433842581 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb49c0c1ee0 con 0x7fb49c1e9c20 2024-02-21T23:04:36.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.579+0000 7fb493e00640 1 -- 172.21.15.28:0/3433842581 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+17004 (secure 0 0 0) 0x7fb49405cd20 con 0x7fb49c1e9c20 2024-02-21T23:04:36.568 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:04:36.568 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":41,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:04:24.717066+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:04:06.977614+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-02-21T23:04:08.074258+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.36","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6835/1213702315":"2024-02-22T23:04:20.686099+0000","172.21.15.28:6833/1213702315":"2024-02-22T23:04:20.686099+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:04:36.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.582+0000 7fb4a2400640 1 -- 172.21.15.28:0/3433842581 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb47806b500 msgr2=0x7fb47806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:36.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.582+0000 7fb4a2400640 1 --2- 172.21.15.28:0/3433842581 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb47806b500 0x7fb47806d9c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fb48c034000 tx=0x7fb48c007000 comp rx=0 tx=0).stop 2024-02-21T23:04:36.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.582+0000 7fb4a2400640 1 -- 172.21.15.28:0/3433842581 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb49c1e9c20 msgr2=0x7fb49c1ea0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:36.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.582+0000 7fb4a2400640 1 --2- 172.21.15.28:0/3433842581 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb49c1e9c20 0x7fb49c1ea0d0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fb49c06e660 tx=0x7fb4940115a0 comp rx=0 tx=0).stop 2024-02-21T23:04:36.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.582+0000 7fb4a2400640 1 -- 172.21.15.28:0/3433842581 shutdown_connections 2024-02-21T23:04:36.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.582+0000 7fb4a2400640 1 --2- 172.21.15.28:0/3433842581 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb47806b500 0x7fb47806d9c0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:36.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.582+0000 7fb4a2400640 1 --2- 172.21.15.28:0/3433842581 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb49c151390 0x7fb49c151810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:36.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.582+0000 7fb4a2400640 1 --2- 172.21.15.28:0/3433842581 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb49c1e9c20 0x7fb49c1ea0d0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:36.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.582+0000 7fb4a2400640 1 --2- 172.21.15.28:0/3433842581 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb4800039b0 0x7fb49c0a97a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:36.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.582+0000 7fb4a2400640 1 -- 172.21.15.28:0/3433842581 >> 172.21.15.28:0/3433842581 conn(0x7fb49c06c5b0 msgr2=0x7fb49c0c4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:36.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.582+0000 7fb4a2400640 1 -- 172.21.15.28:0/3433842581 shutdown_connections 2024-02-21T23:04:36.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.582+0000 7fb4a2400640 1 -- 172.21.15.28:0/3433842581 wait complete. 2024-02-21T23:04:36.592 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-02-21T23:04:36.592 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:04:36.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.814+0000 7f63da400640 1 Processor -- start 2024-02-21T23:04:36.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.814+0000 7f63da400640 1 -- start start 2024-02-21T23:04:36.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.815+0000 7f63da400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f63d4152500 0x7f63d41558c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:36.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.815+0000 7f63da400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f63d40ca050 0x7f63d40bf6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:36.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.815+0000 7f63da400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63d41528e0 con 0x7f63d40bfc00 2024-02-21T23:04:36.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.816+0000 7f63da400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63d40c86e0 con 0x7f63d4152500 2024-02-21T23:04:36.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.816+0000 7f63da400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63d40c8980 con 0x7f63d40ca050 2024-02-21T23:04:36.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.816+0000 7f63d9a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f63d40bfc00 0x7f63d406bbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39900/0 (socket says 172.21.15.28:39900) 2024-02-21T23:04:36.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.816+0000 7f63d9a00640 1 -- 172.21.15.28:0/2126694531 learned_addr learned my addr 172.21.15.28:0/2126694531 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:04:36.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.816+0000 7f63cbe00640 1 --2- 172.21.15.28:0/2126694531 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f63d40ca050 0x7f63d40bf6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:36.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.816+0000 7f63d9000640 1 --2- 172.21.15.28:0/2126694531 >> v2:172.21.15.81:3300/0 conn(0x7f63d4152500 0x7f63d41558c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:36.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.817+0000 7f63d3e00640 1 -- 172.21.15.28:0/2126694531 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 72079112 0 0) 0x7f63d41528e0 con 0x7f63d40bfc00 2024-02-21T23:04:36.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.817+0000 7f63d3e00640 1 -- 172.21.15.28:0/2126694531 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63b4003520 con 0x7f63d40bfc00 2024-02-21T23:04:36.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.817+0000 7f63d3e00640 1 -- 172.21.15.28:0/2126694531 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 735592606 0 0) 0x7f63b4003520 con 0x7f63d40bfc00 2024-02-21T23:04:36.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.817+0000 7f63d3e00640 1 -- 172.21.15.28:0/2126694531 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f63d40ca050 msgr2=0x7f63d40bf6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:36.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.817+0000 7f63d3e00640 1 --2- 172.21.15.28:0/2126694531 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f63d40ca050 0x7f63d40bf6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:36.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.817+0000 7f63d3e00640 1 -- 172.21.15.28:0/2126694531 >> v2:172.21.15.81:3300/0 conn(0x7f63d4152500 msgr2=0x7f63d41558c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:36.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.817+0000 7f63d3e00640 1 --2- 172.21.15.28:0/2126694531 >> v2:172.21.15.81:3300/0 conn(0x7f63d4152500 0x7f63d41558c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:36.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.817+0000 7f63d3e00640 1 -- 172.21.15.28:0/2126694531 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63d40c8c60 con 0x7f63d40bfc00 2024-02-21T23:04:36.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.817+0000 7f63d3e00640 1 -- 172.21.15.28:0/2126694531 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f63cc0019f0 con 0x7f63d40bfc00 2024-02-21T23:04:36.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.817+0000 7f63d3e00640 1 -- 172.21.15.28:0/2126694531 >> v1:172.21.15.28:6789/0 conn(0x7f63d40bfc00 legacy=0x7f63d406bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:36.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.817+0000 7f63d3e00640 1 --2- 172.21.15.28:0/2126694531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63b40039b0 0x7f63b4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:36.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.817+0000 7f63d3e00640 1 --2- 172.21.15.28:0/2126694531 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f63d40ca050 0x7f63b4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:36.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.817+0000 7f63d3e00640 1 --2- 172.21.15.28:0/2126694531 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63b4004d40 0x7f63b40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:36.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.817+0000 7f63d3e00640 1 -- 172.21.15.28:0/2126694531 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63d40c86e0 con 0x7f63b40039b0 2024-02-21T23:04:36.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.817+0000 7f63d3e00640 1 -- 172.21.15.28:0/2126694531 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63d40c8980 con 0x7f63b4004d40 2024-02-21T23:04:36.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.817+0000 7f63d3e00640 1 -- 172.21.15.28:0/2126694531 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63b400b9f0 con 0x7f63d40ca050 2024-02-21T23:04:36.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.817+0000 7f63d9000640 1 --2- 172.21.15.28:0/2126694531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63b40039b0 0x7f63b4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:36.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.818+0000 7f63d9a00640 1 --2- 172.21.15.28:0/2126694531 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63b4004d40 0x7f63b40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:36.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.818+0000 7f63cbe00640 1 --2- 172.21.15.28:0/2126694531 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f63d40ca050 0x7f63b4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:36.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.818+0000 7f63d9000640 1 -- 172.21.15.28:0/2126694531 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f63d40ca050 msgr2=0x7f63b4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:36.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.818+0000 7f63d9000640 1 --2- 172.21.15.28:0/2126694531 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f63d40ca050 0x7f63b4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:36.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.818+0000 7f63d9000640 1 -- 172.21.15.28:0/2126694531 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63b4004d40 msgr2=0x7f63b40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:36.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.818+0000 7f63d9000640 1 --2- 172.21.15.28:0/2126694531 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63b4004d40 0x7f63b40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:36.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.818+0000 7f63d9000640 1 -- 172.21.15.28:0/2126694531 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63b4003520 con 0x7f63b40039b0 2024-02-21T23:04:36.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.819+0000 7f63d9000640 1 --2- 172.21.15.28:0/2126694531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63b40039b0 0x7f63b4003dd0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f63c4009ac0 tx=0x7f63b4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50cc89b9859b1599 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:36.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.819+0000 7f63d3e00640 1 -- 172.21.15.28:0/2126694531 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c4039070 con 0x7f63b40039b0 2024-02-21T23:04:36.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.819+0000 7f63d3e00640 1 -- 172.21.15.28:0/2126694531 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63c402c710 con 0x7f63b40039b0 2024-02-21T23:04:36.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.819+0000 7f63d3e00640 1 -- 172.21.15.28:0/2126694531 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c4009690 con 0x7f63b40039b0 2024-02-21T23:04:36.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.820+0000 7f63da400640 1 -- 172.21.15.28:0/2126694531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63b40039b0 msgr2=0x7f63b4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:36.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.820+0000 7f63da400640 1 --2- 172.21.15.28:0/2126694531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63b40039b0 0x7f63b4003dd0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f63c4009ac0 tx=0x7f63b4005830 comp rx=0 tx=0).stop 2024-02-21T23:04:36.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.820+0000 7f63d9a00640 1 -- 172.21.15.28:0/2126694531 reap_dead start 2024-02-21T23:04:36.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.820+0000 7f63da400640 1 -- 172.21.15.28:0/2126694531 shutdown_connections 2024-02-21T23:04:36.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.820+0000 7f63da400640 1 -- 172.21.15.28:0/2126694531 >> 172.21.15.28:0/2126694531 conn(0x7f63d40ac450 msgr2=0x7f63d40ac860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:36.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.820+0000 7f63da400640 1 -- 172.21.15.28:0/2126694531 shutdown_connections 2024-02-21T23:04:36.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.820+0000 7f63da400640 1 -- 172.21.15.28:0/2126694531 wait complete. 2024-02-21T23:04:36.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.822+0000 7f63da400640 1 Processor -- start 2024-02-21T23:04:36.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.822+0000 7f63da400640 1 -- start start 2024-02-21T23:04:36.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.823+0000 7f63da400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f63b40039b0 0x7f63d41f2490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:36.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.823+0000 7f63da400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63d414c7a0 0x7f63d414eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:36.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.823+0000 7f63da400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63d41f29d0 0x7f63d414f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:36.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.823+0000 7f63da400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63d40bf6c0 con 0x7f63d41f29d0 2024-02-21T23:04:36.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.823+0000 7f63da400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63d40c9220 con 0x7f63d414c7a0 2024-02-21T23:04:36.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.823+0000 7f63da400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63d40bf840 con 0x7f63b40039b0 2024-02-21T23:04:36.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.823+0000 7f63d9a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63d41f29d0 0x7f63d414f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:36.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.823+0000 7f63d9a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63d41f29d0 0x7f63d414f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52422/0 (socket says 172.21.15.28:52422) 2024-02-21T23:04:36.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.823+0000 7f63d9a00640 1 -- 172.21.15.28:0/1563788082 learned_addr learned my addr 172.21.15.28:0/1563788082 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:36.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.824+0000 7f63cbe00640 1 --2- 172.21.15.28:0/1563788082 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63d414c7a0 0x7f63d414eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:36.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.824+0000 7f63d9000640 1 --2- 172.21.15.28:0/1563788082 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f63b40039b0 0x7f63d41f2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:36.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.824+0000 7f63d9a00640 1 -- 172.21.15.28:0/1563788082 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f63b40039b0 msgr2=0x7f63d41f2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:36.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.824+0000 7f63d9a00640 1 --2- 172.21.15.28:0/1563788082 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f63b40039b0 0x7f63d41f2490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:36.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.824+0000 7f63d9a00640 1 -- 172.21.15.28:0/1563788082 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63d414c7a0 msgr2=0x7f63d414eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:36.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.824+0000 7f63d9a00640 1 --2- 172.21.15.28:0/1563788082 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63d414c7a0 0x7f63d414eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:36.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.824+0000 7f63d9a00640 1 -- 172.21.15.28:0/1563788082 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63d40c33b0 con 0x7f63d41f29d0 2024-02-21T23:04:36.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.824+0000 7f63cbe00640 1 --2- 172.21.15.28:0/1563788082 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63d414c7a0 0x7f63d414eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:04:36.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.824+0000 7f63d9a00640 1 --2- 172.21.15.28:0/1563788082 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63d41f29d0 0x7f63d414f430 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f63d408c9d0 tx=0x7f63cc007bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:36.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.824+0000 7f63d1600640 1 -- 172.21.15.28:0/1563788082 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63cc00eb00 con 0x7f63d41f29d0 2024-02-21T23:04:36.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.824+0000 7f63d1600640 1 -- 172.21.15.28:0/1563788082 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63cc018070 con 0x7f63d41f29d0 2024-02-21T23:04:36.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.824+0000 7f63d1600640 1 -- 172.21.15.28:0/1563788082 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63cc013420 con 0x7f63d41f29d0 2024-02-21T23:04:36.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.824+0000 7f63da400640 1 -- 172.21.15.28:0/1563788082 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63d40c3690 con 0x7f63d41f29d0 2024-02-21T23:04:36.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.825+0000 7f63da400640 1 -- 172.21.15.28:0/1563788082 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63d40c3c50 con 0x7f63d41f29d0 2024-02-21T23:04:36.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.826+0000 7f63da400640 1 -- 172.21.15.28:0/1563788082 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63d404f290 con 0x7f63d41f29d0 2024-02-21T23:04:36.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.827+0000 7f63d1600640 1 -- 172.21.15.28:0/1563788082 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f63cc0135c0 con 0x7f63d41f29d0 2024-02-21T23:04:36.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.828+0000 7f63d1600640 1 --2- 172.21.15.28:0/1563788082 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f63b006b4d0 0x7f63b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:36.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.828+0000 7f63d1600640 1 -- 172.21.15.28:0/1563788082 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6216+0+0 (secure 0 0 0) 0x7f63cc08a750 con 0x7f63d41f29d0 2024-02-21T23:04:36.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.828+0000 7f63d9000640 1 --2- 172.21.15.28:0/1563788082 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f63b006b4d0 0x7f63b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:36.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.829+0000 7f63d9000640 1 --2- 172.21.15.28:0/1563788082 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f63b006b4d0 0x7f63b006d990 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f63c4036000 tx=0x7f63c4030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:36.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:36.837+0000 7f63d1600640 1 -- 172.21.15.28:0/1563788082 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f63cc0593d0 con 0x7f63d41f29d0 2024-02-21T23:04:37.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.123+0000 7f63da400640 1 -- 172.21.15.28:0/1563788082 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f63d40c4170 con 0x7f63d41f29d0 2024-02-21T23:04:37.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.124+0000 7f63d1600640 1 -- 172.21.15.28:0/1563788082 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4934 (secure 0 0 0) 0x7f63cc05f0d0 con 0x7f63d41f29d0 2024-02-21T23:04:37.112 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:04:37.113 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":24,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4978,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/3612644344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":3612644344},{"type":"v1","addr":"172.21.15.81:6836","nonce":3612644344}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4981,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3623937361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":3623937361},{"type":"v1","addr":"172.21.15.81:6837","nonce":3623937361}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5442,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/33413357","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":33413357},{"type":"v1","addr":"172.21.15.28:6835","nonce":33413357}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:04:09.036302+0000","modified":"2024-02-21T23:04:26.710368+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4778},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4778":{"gid":4778,"name":"c","rank":0,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.28:6834/2827878900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2827878900},{"type":"v1","addr":"172.21.15.28:6834","nonce":2827878900}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-02-21T23:04:37.113 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 24 2024-02-21T23:04:37.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.127+0000 7f63da400640 1 -- 172.21.15.28:0/1563788082 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f63b006b4d0 msgr2=0x7f63b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:37.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.127+0000 7f63da400640 1 --2- 172.21.15.28:0/1563788082 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f63b006b4d0 0x7f63b006d990 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f63c4036000 tx=0x7f63c4030000 comp rx=0 tx=0).stop 2024-02-21T23:04:37.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.127+0000 7f63da400640 1 -- 172.21.15.28:0/1563788082 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63d41f29d0 msgr2=0x7f63d414f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:37.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.127+0000 7f63da400640 1 --2- 172.21.15.28:0/1563788082 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63d41f29d0 0x7f63d414f430 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f63d408c9d0 tx=0x7f63cc007bc0 comp rx=0 tx=0).stop 2024-02-21T23:04:37.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.127+0000 7f63da400640 1 -- 172.21.15.28:0/1563788082 shutdown_connections 2024-02-21T23:04:37.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.127+0000 7f63da400640 1 --2- 172.21.15.28:0/1563788082 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f63b006b4d0 0x7f63b006d990 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:37.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.127+0000 7f63da400640 1 --2- 172.21.15.28:0/1563788082 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63d41f29d0 0x7f63d414f430 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:37.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.127+0000 7f63da400640 1 --2- 172.21.15.28:0/1563788082 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63d414c7a0 0x7f63d414eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:37.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.127+0000 7f63da400640 1 --2- 172.21.15.28:0/1563788082 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f63b40039b0 0x7f63d41f2490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:37.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.127+0000 7f63da400640 1 -- 172.21.15.28:0/1563788082 >> 172.21.15.28:0/1563788082 conn(0x7f63d40ac450 msgr2=0x7f63d40c7390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:37.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.128+0000 7f63da400640 1 -- 172.21.15.28:0/1563788082 shutdown_connections 2024-02-21T23:04:37.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.128+0000 7f63da400640 1 -- 172.21.15.28:0/1563788082 wait complete. 2024-02-21T23:04:37.135 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:04:37.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.347+0000 7f1263200640 1 Processor -- start 2024-02-21T23:04:37.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.347+0000 7f1263200640 1 -- start start 2024-02-21T23:04:37.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.348+0000 7f1263200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f126406dbf0 0x7f1264158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:37.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.348+0000 7f1263200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12640c74c0 con 0x7f12640aff90 2024-02-21T23:04:37.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.348+0000 7f1263200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12640c77a0 con 0x7f1264155960 2024-02-21T23:04:37.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.348+0000 7f1263200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12640c7270 con 0x7f126406dbf0 2024-02-21T23:04:37.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.349+0000 7f1261400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f126406dbf0 0x7f1264158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:37.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.349+0000 7f1261e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f12640aff90 0x7f12640c51e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39912/0 (socket says 172.21.15.28:39912) 2024-02-21T23:04:37.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.349+0000 7f1261e00640 1 -- 172.21.15.28:0/2172525756 learned_addr learned my addr 172.21.15.28:0/2172525756 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:04:37.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.349+0000 7f1261400640 1 -- 172.21.15.28:0/2172525756 >> v1:172.21.15.81:6789/0 conn(0x7f1264155960 legacy=0x7f126415f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:37.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.349+0000 7f1261400640 1 -- 172.21.15.28:0/2172525756 >> v1:172.21.15.28:6789/0 conn(0x7f12640aff90 legacy=0x7f12640c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:37.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.349+0000 7f1261400640 1 -- 172.21.15.28:0/2172525756 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12640c7a80 con 0x7f126406dbf0 2024-02-21T23:04:37.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.349+0000 7f1260a00640 1 -- 172.21.15.28:0/2172525756 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3131980530 0 0) 0x7f12640c74c0 con 0x7f12640aff90 2024-02-21T23:04:37.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.351+0000 7f1261400640 1 --2- 172.21.15.28:0/2172525756 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f126406dbf0 0x7f1264158ac0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f12500097a0 tx=0x7f125002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a03136097ce2eb1b server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:37.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.351+0000 7f1260a00640 1 -- 172.21.15.28:0/2172525756 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1250035020 con 0x7f126406dbf0 2024-02-21T23:04:37.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.351+0000 7f1260a00640 1 -- 172.21.15.28:0/2172525756 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1250031830 con 0x7f126406dbf0 2024-02-21T23:04:37.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.351+0000 7f1260a00640 1 -- 172.21.15.28:0/2172525756 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1250031c00 con 0x7f126406dbf0 2024-02-21T23:04:37.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.352+0000 7f1263200640 1 -- 172.21.15.28:0/2172525756 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f126406dbf0 msgr2=0x7f1264158ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:37.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.352+0000 7f1263200640 1 --2- 172.21.15.28:0/2172525756 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f126406dbf0 0x7f1264158ac0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f12500097a0 tx=0x7f125002b8d0 comp rx=0 tx=0).stop 2024-02-21T23:04:37.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.352+0000 7f1263200640 1 -- 172.21.15.28:0/2172525756 shutdown_connections 2024-02-21T23:04:37.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.352+0000 7f1263200640 1 --2- 172.21.15.28:0/2172525756 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f126406dbf0 0x7f1264158ac0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:37.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.352+0000 7f1263200640 1 -- 172.21.15.28:0/2172525756 >> 172.21.15.28:0/2172525756 conn(0x7f12640ac370 msgr2=0x7f12640ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:37.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.352+0000 7f1263200640 1 -- 172.21.15.28:0/2172525756 shutdown_connections 2024-02-21T23:04:37.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.352+0000 7f1263200640 1 -- 172.21.15.28:0/2172525756 wait complete. 2024-02-21T23:04:37.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.355+0000 7f1263200640 1 Processor -- start 2024-02-21T23:04:37.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.355+0000 7f1263200640 1 -- start start 2024-02-21T23:04:37.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.356+0000 7f1263200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f126406dbf0 0x7f12640c1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:37.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.356+0000 7f1263200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f12640aff90 0x7f12640c0230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:37.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.357+0000 7f1263200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1264155960 0x7f12640bf540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:37.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.357+0000 7f1263200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12640c58f0 con 0x7f126406dbf0 2024-02-21T23:04:37.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.357+0000 7f1263200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1264158630 con 0x7f1264155960 2024-02-21T23:04:37.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.357+0000 7f1263200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12640c5a70 con 0x7f12640aff90 2024-02-21T23:04:37.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.357+0000 7f1261e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f126406dbf0 0x7f12640c1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:37.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.357+0000 7f1261e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f126406dbf0 0x7f12640c1930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52434/0 (socket says 172.21.15.28:52434) 2024-02-21T23:04:37.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.357+0000 7f1261e00640 1 -- 172.21.15.28:0/1229589396 learned_addr learned my addr 172.21.15.28:0/1229589396 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:37.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.357+0000 7f1262800640 1 --2- 172.21.15.28:0/1229589396 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1264155960 0x7f12640bf540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:37.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.357+0000 7f1261400640 1 --2- 172.21.15.28:0/1229589396 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f12640aff90 0x7f12640c0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:37.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.357+0000 7f1261e00640 1 -- 172.21.15.28:0/1229589396 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f12640aff90 msgr2=0x7f12640c0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:37.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.357+0000 7f1261e00640 1 --2- 172.21.15.28:0/1229589396 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f12640aff90 0x7f12640c0230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:37.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.357+0000 7f1261e00640 1 -- 172.21.15.28:0/1229589396 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1264155960 msgr2=0x7f12640bf540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:37.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.357+0000 7f1261e00640 1 --2- 172.21.15.28:0/1229589396 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1264155960 0x7f12640bf540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:37.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.357+0000 7f1261e00640 1 -- 172.21.15.28:0/1229589396 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1264155430 con 0x7f126406dbf0 2024-02-21T23:04:37.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.357+0000 7f1262800640 1 --2- 172.21.15.28:0/1229589396 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1264155960 0x7f12640bf540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:04:37.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.358+0000 7f1261e00640 1 --2- 172.21.15.28:0/1229589396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f126406dbf0 0x7f12640c1930 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f125400c500 tx=0x7f125400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:37.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.358+0000 7f125a000640 1 -- 172.21.15.28:0/1229589396 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1254018070 con 0x7f126406dbf0 2024-02-21T23:04:37.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.358+0000 7f125a000640 1 -- 172.21.15.28:0/1229589396 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1254007580 con 0x7f126406dbf0 2024-02-21T23:04:37.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.358+0000 7f125a000640 1 -- 172.21.15.28:0/1229589396 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1254004a20 con 0x7f126406dbf0 2024-02-21T23:04:37.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.358+0000 7f1263200640 1 -- 172.21.15.28:0/1229589396 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12640be4a0 con 0x7f126406dbf0 2024-02-21T23:04:37.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.358+0000 7f1263200640 1 -- 172.21.15.28:0/1229589396 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12640bea60 con 0x7f126406dbf0 2024-02-21T23:04:37.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.360+0000 7f1263200640 1 -- 172.21.15.28:0/1229589396 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f126404f290 con 0x7f126406dbf0 2024-02-21T23:04:37.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.361+0000 7f125a000640 1 -- 172.21.15.28:0/1229589396 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1254004bc0 con 0x7f126406dbf0 2024-02-21T23:04:37.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.362+0000 7f125a000640 1 --2- 172.21.15.28:0/1229589396 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f124406b4d0 0x7f124406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:37.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.362+0000 7f125a000640 1 -- 172.21.15.28:0/1229589396 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6216+0+0 (secure 0 0 0) 0x7f12540896d0 con 0x7f126406dbf0 2024-02-21T23:04:37.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.362+0000 7f1261400640 1 --2- 172.21.15.28:0/1229589396 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f124406b4d0 0x7f124406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:37.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.364+0000 7f1261400640 1 --2- 172.21.15.28:0/1229589396 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f124406b4d0 0x7f124406d990 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f1250036000 tx=0x7f1250008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:37.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.366+0000 7f125a000640 1 -- 172.21.15.28:0/1229589396 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1254058340 con 0x7f126406dbf0 2024-02-21T23:04:37.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.604+0000 7f1263200640 1 -- 172.21.15.28:0/1229589396 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f12640af490 con 0x7f126406dbf0 2024-02-21T23:04:37.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.605+0000 7f125a000640 1 -- 172.21.15.28:0/1229589396 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4934 (secure 0 0 0) 0x7f1254057c40 con 0x7f126406dbf0 2024-02-21T23:04:37.593 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:04:37.594 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":24,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4978,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/3612644344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":3612644344},{"type":"v1","addr":"172.21.15.81:6836","nonce":3612644344}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4981,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3623937361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":3623937361},{"type":"v1","addr":"172.21.15.81:6837","nonce":3623937361}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5442,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/33413357","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":33413357},{"type":"v1","addr":"172.21.15.28:6835","nonce":33413357}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:04:09.036302+0000","modified":"2024-02-21T23:04:26.710368+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4778},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4778":{"gid":4778,"name":"c","rank":0,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.28:6834/2827878900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2827878900},{"type":"v1","addr":"172.21.15.28:6834","nonce":2827878900}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-02-21T23:04:37.594 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 24 2024-02-21T23:04:37.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.607+0000 7f1263200640 1 -- 172.21.15.28:0/1229589396 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f124406b4d0 msgr2=0x7f124406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:37.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.607+0000 7f1263200640 1 --2- 172.21.15.28:0/1229589396 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f124406b4d0 0x7f124406d990 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f1250036000 tx=0x7f1250008030 comp rx=0 tx=0).stop 2024-02-21T23:04:37.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.607+0000 7f1263200640 1 -- 172.21.15.28:0/1229589396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f126406dbf0 msgr2=0x7f12640c1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:37.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.607+0000 7f1263200640 1 --2- 172.21.15.28:0/1229589396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f126406dbf0 0x7f12640c1930 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f125400c500 tx=0x7f125400c9c0 comp rx=0 tx=0).stop 2024-02-21T23:04:37.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.607+0000 7f1263200640 1 -- 172.21.15.28:0/1229589396 shutdown_connections 2024-02-21T23:04:37.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.607+0000 7f1263200640 1 --2- 172.21.15.28:0/1229589396 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f124406b4d0 0x7f124406d990 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:37.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.607+0000 7f1263200640 1 --2- 172.21.15.28:0/1229589396 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f12640aff90 0x7f12640c0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:37.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.607+0000 7f1263200640 1 --2- 172.21.15.28:0/1229589396 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1264155960 0x7f12640bf540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:37.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.607+0000 7f1263200640 1 --2- 172.21.15.28:0/1229589396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f126406dbf0 0x7f12640c1930 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:37.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.608+0000 7f1263200640 1 -- 172.21.15.28:0/1229589396 >> 172.21.15.28:0/1229589396 conn(0x7f12640ac370 msgr2=0x7f1264157220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:37.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.608+0000 7f1263200640 1 -- 172.21.15.28:0/1229589396 shutdown_connections 2024-02-21T23:04:37.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.608+0000 7f1263200640 1 -- 172.21.15.28:0/1229589396 wait complete. 2024-02-21T23:04:37.614 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:04:37.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.851+0000 7f6a7c800640 1 Processor -- start 2024-02-21T23:04:37.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.851+0000 7f6a7c800640 1 -- start start 2024-02-21T23:04:37.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.852+0000 7f6a7c800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6a740c11f0 0x7f6a74153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:37.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.852+0000 7f6a7c800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f6a740c15d0 0x7f6a7406dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:37.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.852+0000 7f6a7c800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6a74156350 0x7f6a74156780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:37.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.852+0000 7f6a7c800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a740c1a60 con 0x7f6a740c11f0 2024-02-21T23:04:37.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.852+0000 7f6a7c800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a740a5ec0 con 0x7f6a740c15d0 2024-02-21T23:04:37.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.852+0000 7f6a7c800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a74152e70 con 0x7f6a74156350 2024-02-21T23:04:37.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.852+0000 7f6a79e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6a740c11f0 0x7f6a74153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:37.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.853+0000 7f6a79e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6a740c11f0 0x7f6a74153950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52440/0 (socket says 172.21.15.28:52440) 2024-02-21T23:04:37.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.853+0000 7f6a79e00640 1 -- 172.21.15.28:0/1462899946 learned_addr learned my addr 172.21.15.28:0/1462899946 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:37.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.853+0000 7f6a7a800640 1 --2- 172.21.15.28:0/1462899946 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6a74156350 0x7f6a74156780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:37.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.853+0000 7f6a79400640 1 --2- 172.21.15.28:0/1462899946 >> v2:172.21.15.81:3300/0 conn(0x7f6a740c15d0 0x7f6a7406dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:37.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.853+0000 7f6a79e00640 1 -- 172.21.15.28:0/1462899946 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6a74156350 msgr2=0x7f6a74156780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:37.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.853+0000 7f6a79e00640 1 --2- 172.21.15.28:0/1462899946 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6a74156350 0x7f6a74156780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:37.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.853+0000 7f6a79e00640 1 -- 172.21.15.28:0/1462899946 >> v2:172.21.15.81:3300/0 conn(0x7f6a740c15d0 msgr2=0x7f6a7406dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:37.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.853+0000 7f6a79e00640 1 --2- 172.21.15.28:0/1462899946 >> v2:172.21.15.81:3300/0 conn(0x7f6a740c15d0 0x7f6a7406dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:37.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.853+0000 7f6a79e00640 1 -- 172.21.15.28:0/1462899946 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a74153150 con 0x7f6a740c11f0 2024-02-21T23:04:37.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.854+0000 7f6a79e00640 1 --2- 172.21.15.28:0/1462899946 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a740c11f0 0x7f6a74153950 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f6a740c24a0 tx=0x7f6a6c02b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfb5fadd89fcf003 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:37.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.855+0000 7f6a78a00640 1 -- 172.21.15.28:0/1462899946 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a6c038070 con 0x7f6a740c11f0 2024-02-21T23:04:37.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.855+0000 7f6a78a00640 1 -- 172.21.15.28:0/1462899946 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a6c02f050 con 0x7f6a740c11f0 2024-02-21T23:04:37.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.855+0000 7f6a78a00640 1 -- 172.21.15.28:0/1462899946 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a6c031970 con 0x7f6a740c11f0 2024-02-21T23:04:37.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.855+0000 7f6a7c800640 1 -- 172.21.15.28:0/1462899946 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a740c11f0 msgr2=0x7f6a74153950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:37.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.855+0000 7f6a7c800640 1 --2- 172.21.15.28:0/1462899946 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a740c11f0 0x7f6a74153950 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f6a740c24a0 tx=0x7f6a6c02b7f0 comp rx=0 tx=0).stop 2024-02-21T23:04:37.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.855+0000 7f6a7c800640 1 -- 172.21.15.28:0/1462899946 shutdown_connections 2024-02-21T23:04:37.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.855+0000 7f6a7c800640 1 --2- 172.21.15.28:0/1462899946 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6a74156350 0x7f6a74156780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:37.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.855+0000 7f6a7c800640 1 --2- 172.21.15.28:0/1462899946 >> v2:172.21.15.81:3300/0 conn(0x7f6a740c15d0 0x7f6a7406dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:37.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.856+0000 7f6a7c800640 1 --2- 172.21.15.28:0/1462899946 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a740c11f0 0x7f6a74153950 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:37.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.856+0000 7f6a7c800640 1 -- 172.21.15.28:0/1462899946 >> 172.21.15.28:0/1462899946 conn(0x7f6a740c67f0 msgr2=0x7f6a740c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:37.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.856+0000 7f6a7c800640 1 -- 172.21.15.28:0/1462899946 shutdown_connections 2024-02-21T23:04:37.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.856+0000 7f6a7c800640 1 -- 172.21.15.28:0/1462899946 wait complete. 2024-02-21T23:04:37.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.857+0000 7f6a7c800640 1 Processor -- start 2024-02-21T23:04:37.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.857+0000 7f6a7c800640 1 -- start start 2024-02-21T23:04:37.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.858+0000 7f6a7c800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6a740c11f0 0x7f6a74071f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:37.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.858+0000 7f6a7c800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a740c15d0 0x7f6a74145a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:37.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.858+0000 7f6a7c800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6a74156350 0x7f6a74145f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:37.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.858+0000 7f6a7c800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a740c4c00 con 0x7f6a740c15d0 2024-02-21T23:04:37.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.858+0000 7f6a7c800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a7414d1c0 con 0x7f6a74156350 2024-02-21T23:04:37.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.858+0000 7f6a7c800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a740a6110 con 0x7f6a740c11f0 2024-02-21T23:04:37.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.858+0000 7f6a79400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a740c15d0 0x7f6a74145a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:37.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.859+0000 7f6a79400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a740c15d0 0x7f6a74145a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52456/0 (socket says 172.21.15.28:52456) 2024-02-21T23:04:37.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.859+0000 7f6a79400640 1 -- 172.21.15.28:0/1079743565 learned_addr learned my addr 172.21.15.28:0/1079743565 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:37.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.859+0000 7f6a7a800640 1 --2- 172.21.15.28:0/1079743565 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6a74156350 0x7f6a74145f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:37.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.859+0000 7f6a79e00640 1 --2- 172.21.15.28:0/1079743565 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6a740c11f0 0x7f6a74071f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:37.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.859+0000 7f6a79400640 1 -- 172.21.15.28:0/1079743565 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6a740c11f0 msgr2=0x7f6a74071f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:37.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.859+0000 7f6a79400640 1 --2- 172.21.15.28:0/1079743565 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6a740c11f0 0x7f6a74071f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:37.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.859+0000 7f6a79400640 1 -- 172.21.15.28:0/1079743565 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6a74156350 msgr2=0x7f6a74145f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:37.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.859+0000 7f6a79400640 1 --2- 172.21.15.28:0/1079743565 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6a74156350 0x7f6a74145f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:37.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.859+0000 7f6a79400640 1 -- 172.21.15.28:0/1079743565 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a74149bd0 con 0x7f6a740c15d0 2024-02-21T23:04:37.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.859+0000 7f6a79e00640 1 --2- 172.21.15.28:0/1079743565 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6a740c11f0 0x7f6a74071f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:04:37.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.859+0000 7f6a7a800640 1 --2- 172.21.15.28:0/1079743565 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6a74156350 0x7f6a74145f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:04:37.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.859+0000 7f6a79400640 1 --2- 172.21.15.28:0/1079743565 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a740c15d0 0x7f6a74145a00 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f6a6400e870 tx=0x7f6a6400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:37.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.860+0000 7f6a6a000640 1 -- 172.21.15.28:0/1079743565 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a64017070 con 0x7f6a740c15d0 2024-02-21T23:04:37.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.860+0000 7f6a6a000640 1 -- 172.21.15.28:0/1079743565 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a6400ee60 con 0x7f6a740c15d0 2024-02-21T23:04:37.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.860+0000 7f6a7c800640 1 -- 172.21.15.28:0/1079743565 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a74146480 con 0x7f6a740c15d0 2024-02-21T23:04:37.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.860+0000 7f6a6a000640 1 -- 172.21.15.28:0/1079743565 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a6400cb80 con 0x7f6a740c15d0 2024-02-21T23:04:37.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.860+0000 7f6a7c800640 1 -- 172.21.15.28:0/1079743565 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a7414e430 con 0x7f6a740c15d0 2024-02-21T23:04:37.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.861+0000 7f6a7c800640 1 -- 172.21.15.28:0/1079743565 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a74067f30 con 0x7f6a740c15d0 2024-02-21T23:04:37.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.863+0000 7f6a6a000640 1 -- 172.21.15.28:0/1079743565 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6a6401a070 con 0x7f6a740c15d0 2024-02-21T23:04:37.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.864+0000 7f6a6a000640 1 --2- 172.21.15.28:0/1079743565 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6a5806b4d0 0x7f6a5806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:37.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.864+0000 7f6a6a000640 1 -- 172.21.15.28:0/1079743565 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6216+0+0 (secure 0 0 0) 0x7f6a6408a300 con 0x7f6a740c15d0 2024-02-21T23:04:37.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.865+0000 7f6a79e00640 1 --2- 172.21.15.28:0/1079743565 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6a5806b4d0 0x7f6a5806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:37.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.866+0000 7f6a79e00640 1 --2- 172.21.15.28:0/1079743565 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6a5806b4d0 0x7f6a5806d990 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f6a6c035000 tx=0x7f6a6c008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:37.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:37.870+0000 7f6a6a000640 1 -- 172.21.15.28:0/1079743565 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f6a64058f70 con 0x7f6a740c15d0 2024-02-21T23:04:38.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.061+0000 7f6a7c800640 1 -- 172.21.15.28:0/1079743565 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6a740c0960 con 0x7f6a740c15d0 2024-02-21T23:04:38.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.063+0000 7f6a6a000640 1 -- 172.21.15.28:0/1079743565 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+17004 (secure 0 0 0) 0x7f6a64058870 con 0x7f6a740c15d0 2024-02-21T23:04:38.052 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:04:38.052 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":41,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:04:24.717066+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:04:06.977614+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-02-21T23:04:08.074258+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.36","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6835/1213702315":"2024-02-22T23:04:20.686099+0000","172.21.15.28:6833/1213702315":"2024-02-22T23:04:20.686099+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:04:38.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.066+0000 7f6a7c800640 1 -- 172.21.15.28:0/1079743565 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6a5806b4d0 msgr2=0x7f6a5806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:38.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.066+0000 7f6a7c800640 1 --2- 172.21.15.28:0/1079743565 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6a5806b4d0 0x7f6a5806d990 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f6a6c035000 tx=0x7f6a6c008030 comp rx=0 tx=0).stop 2024-02-21T23:04:38.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.066+0000 7f6a7c800640 1 -- 172.21.15.28:0/1079743565 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a740c15d0 msgr2=0x7f6a74145a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:38.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.066+0000 7f6a7c800640 1 --2- 172.21.15.28:0/1079743565 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a740c15d0 0x7f6a74145a00 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f6a6400e870 tx=0x7f6a6400ed30 comp rx=0 tx=0).stop 2024-02-21T23:04:38.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.066+0000 7f6a7c800640 1 -- 172.21.15.28:0/1079743565 shutdown_connections 2024-02-21T23:04:38.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.066+0000 7f6a7c800640 1 --2- 172.21.15.28:0/1079743565 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6a5806b4d0 0x7f6a5806d990 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:38.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.066+0000 7f6a7c800640 1 --2- 172.21.15.28:0/1079743565 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6a74156350 0x7f6a74145f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:38.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.066+0000 7f6a7c800640 1 --2- 172.21.15.28:0/1079743565 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a740c15d0 0x7f6a74145a00 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:38.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.066+0000 7f6a7c800640 1 --2- 172.21.15.28:0/1079743565 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6a740c11f0 0x7f6a74071f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:38.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.066+0000 7f6a7c800640 1 -- 172.21.15.28:0/1079743565 >> 172.21.15.28:0/1079743565 conn(0x7f6a740c67f0 msgr2=0x7f6a740c5370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:38.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.066+0000 7f6a7c800640 1 -- 172.21.15.28:0/1079743565 shutdown_connections 2024-02-21T23:04:38.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.066+0000 7f6a7c800640 1 -- 172.21.15.28:0/1079743565 wait complete. 2024-02-21T23:04:38.074 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-02-21T23:04:38.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.292+0000 7fa8eb400640 1 Processor -- start 2024-02-21T23:04:38.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.292+0000 7fa8eb400640 1 -- start start 2024-02-21T23:04:38.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.294+0000 7fa8eb400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa8e413e480 0x7fa8e4155160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:38.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.294+0000 7fa8eb400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa8e40bfb60 0x7fa8e40bffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:38.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.294+0000 7fa8eb400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8e4073380 con 0x7fa8e40c5320 2024-02-21T23:04:38.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.294+0000 7fa8eb400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8e40c5760 con 0x7fa8e413e480 2024-02-21T23:04:38.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.294+0000 7fa8eb400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e40c5a00 con 0x7fa8e40bfb60 2024-02-21T23:04:38.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.294+0000 7fa8e8a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa8e413e480 0x7fa8e4155160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:38.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.294+0000 7fa8e3e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa8e40c5320 0x7fa8e40ca850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39916/0 (socket says 172.21.15.28:39916) 2024-02-21T23:04:38.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.294+0000 7fa8e3e00640 1 -- 172.21.15.28:0/1440137417 learned_addr learned my addr 172.21.15.28:0/1440137417 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:04:38.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.294+0000 7fa8e8a00640 1 -- 172.21.15.28:0/1440137417 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa8e40bfb60 msgr2=0x7fa8e40bffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:38.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.294+0000 7fa8e8a00640 1 --2- 172.21.15.28:0/1440137417 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa8e40bfb60 0x7fa8e40bffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:38.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.294+0000 7fa8e8a00640 1 -- 172.21.15.28:0/1440137417 >> v1:172.21.15.28:6789/0 conn(0x7fa8e40c5320 legacy=0x7fa8e40ca850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:38.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.295+0000 7fa8e8a00640 1 -- 172.21.15.28:0/1440137417 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8e40c5ce0 con 0x7fa8e413e480 2024-02-21T23:04:38.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.296+0000 7fa8e8a00640 1 --2- 172.21.15.28:0/1440137417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8e413e480 0x7fa8e4155160 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fa8d80097a0 tx=0x7fa8d802b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e98e6a8f63d85aa server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:38.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.296+0000 7fa8e3400640 1 -- 172.21.15.28:0/1440137417 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8d8031780 con 0x7fa8e413e480 2024-02-21T23:04:38.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.296+0000 7fa8e3400640 1 -- 172.21.15.28:0/1440137417 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8d8037070 con 0x7fa8e413e480 2024-02-21T23:04:38.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.296+0000 7fa8e3400640 1 -- 172.21.15.28:0/1440137417 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8d8031b40 con 0x7fa8e413e480 2024-02-21T23:04:38.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.296+0000 7fa8eb400640 1 -- 172.21.15.28:0/1440137417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8e413e480 msgr2=0x7fa8e4155160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:38.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.296+0000 7fa8eb400640 1 --2- 172.21.15.28:0/1440137417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8e413e480 0x7fa8e4155160 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fa8d80097a0 tx=0x7fa8d802b820 comp rx=0 tx=0).stop 2024-02-21T23:04:38.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.297+0000 7fa8eb400640 1 -- 172.21.15.28:0/1440137417 shutdown_connections 2024-02-21T23:04:38.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.297+0000 7fa8eb400640 1 --2- 172.21.15.28:0/1440137417 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa8e40bfb60 0x7fa8e40bffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:38.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.297+0000 7fa8eb400640 1 --2- 172.21.15.28:0/1440137417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8e413e480 0x7fa8e4155160 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:38.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.297+0000 7fa8eb400640 1 -- 172.21.15.28:0/1440137417 >> 172.21.15.28:0/1440137417 conn(0x7fa8e408b1c0 msgr2=0x7fa8e40725a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:38.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.297+0000 7fa8eb400640 1 -- 172.21.15.28:0/1440137417 shutdown_connections 2024-02-21T23:04:38.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.297+0000 7fa8eb400640 1 -- 172.21.15.28:0/1440137417 wait complete. 2024-02-21T23:04:38.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.298+0000 7fa8eb400640 1 Processor -- start 2024-02-21T23:04:38.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.298+0000 7fa8eb400640 1 -- start start 2024-02-21T23:04:38.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.299+0000 7fa8eb400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa8e40bfb60 0x7fa8e4158c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:38.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.299+0000 7fa8eb400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa8e40c5320 0x7fa8e41ec8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:38.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.299+0000 7fa8eb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8e413e480 0x7fa8e41eec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:38.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.299+0000 7fa8eb400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e40c6580 con 0x7fa8e413e480 2024-02-21T23:04:38.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.299+0000 7fa8eb400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e40c62a0 con 0x7fa8e40bfb60 2024-02-21T23:04:38.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.299+0000 7fa8eb400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e413e860 con 0x7fa8e40c5320 2024-02-21T23:04:38.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.300+0000 7fa8e9400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8e413e480 0x7fa8e41eec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:38.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.300+0000 7fa8e9400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8e413e480 0x7fa8e41eec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52462/0 (socket says 172.21.15.28:52462) 2024-02-21T23:04:38.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.300+0000 7fa8e9400640 1 -- 172.21.15.28:0/801477684 learned_addr learned my addr 172.21.15.28:0/801477684 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:38.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.300+0000 7fa8e9400640 1 -- 172.21.15.28:0/801477684 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa8e40c5320 msgr2=0x7fa8e41ec8a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-02-21T23:04:38.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.300+0000 7fa8e9400640 1 --2- 172.21.15.28:0/801477684 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa8e40c5320 0x7fa8e41ec8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:38.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.300+0000 7fa8e9400640 1 -- 172.21.15.28:0/801477684 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa8e40bfb60 msgr2=0x7fa8e4158c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:38.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.300+0000 7fa8e9400640 1 --2- 172.21.15.28:0/801477684 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa8e40bfb60 0x7fa8e4158c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:38.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.300+0000 7fa8e9400640 1 -- 172.21.15.28:0/801477684 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8e41536e0 con 0x7fa8e413e480 2024-02-21T23:04:38.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.301+0000 7fa8e9400640 1 --2- 172.21.15.28:0/801477684 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8e413e480 0x7fa8e41eec10 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fa8dc00b970 tx=0x7fa8dc00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:38.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.301+0000 7fa8e0c00640 1 -- 172.21.15.28:0/801477684 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8dc019070 con 0x7fa8e413e480 2024-02-21T23:04:38.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.301+0000 7fa8e0c00640 1 -- 172.21.15.28:0/801477684 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8dc00cb60 con 0x7fa8e413e480 2024-02-21T23:04:38.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.301+0000 7fa8eb400640 1 -- 172.21.15.28:0/801477684 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8e4158150 con 0x7fa8e413e480 2024-02-21T23:04:38.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.301+0000 7fa8e0c00640 1 -- 172.21.15.28:0/801477684 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8dc014440 con 0x7fa8e413e480 2024-02-21T23:04:38.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.301+0000 7fa8eb400640 1 -- 172.21.15.28:0/801477684 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8e40c4e90 con 0x7fa8e413e480 2024-02-21T23:04:38.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.302+0000 7fa8eb400640 1 -- 172.21.15.28:0/801477684 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8e406b2a0 con 0x7fa8e413e480 2024-02-21T23:04:38.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.303+0000 7fa8e0c00640 1 -- 172.21.15.28:0/801477684 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa8dc01c070 con 0x7fa8e413e480 2024-02-21T23:04:38.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.304+0000 7fa8e0c00640 1 --2- 172.21.15.28:0/801477684 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa8c406b500 0x7fa8c406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:38.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.304+0000 7fa8e0c00640 1 -- 172.21.15.28:0/801477684 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6216+0+0 (secure 0 0 0) 0x7fa8dc08a160 con 0x7fa8e413e480 2024-02-21T23:04:38.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.305+0000 7fa8e8a00640 1 --2- 172.21.15.28:0/801477684 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa8c406b500 0x7fa8c406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:38.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.306+0000 7fa8e8a00640 1 --2- 172.21.15.28:0/801477684 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa8c406b500 0x7fa8c406d9c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fa8d8036000 tx=0x7fa8d8030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:38.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.309+0000 7fa8e0c00640 1 -- 172.21.15.28:0/801477684 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fa8dc058dd0 con 0x7fa8e413e480 2024-02-21T23:04:38.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.544+0000 7fa8eb400640 1 -- 172.21.15.28:0/801477684 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fa8e413bf20 con 0x7fa8e413e480 2024-02-21T23:04:38.533 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-02-21T23:04:38.545+0000 7f5ddc400640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-21T23:04:38.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.548+0000 7fa8e0c00640 1 -- 172.21.15.28:0/801477684 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v25) v1 ==== 161+0+0 (secure 0 0 0) 0x7fa8dc0586d0 con 0x7fa8e413e480 2024-02-21T23:04:38.537 INFO:teuthology.orchestra.run.smithi028.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-02-21T23:04:38.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.552+0000 7fa8eb400640 1 -- 172.21.15.28:0/801477684 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa8c406b500 msgr2=0x7fa8c406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:38.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.552+0000 7fa8eb400640 1 --2- 172.21.15.28:0/801477684 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa8c406b500 0x7fa8c406d9c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fa8d8036000 tx=0x7fa8d8030000 comp rx=0 tx=0).stop 2024-02-21T23:04:38.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.552+0000 7fa8eb400640 1 -- 172.21.15.28:0/801477684 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8e413e480 msgr2=0x7fa8e41eec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:38.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.552+0000 7fa8eb400640 1 --2- 172.21.15.28:0/801477684 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8e413e480 0x7fa8e41eec10 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fa8dc00b970 tx=0x7fa8dc00be30 comp rx=0 tx=0).stop 2024-02-21T23:04:38.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.552+0000 7fa8eb400640 1 -- 172.21.15.28:0/801477684 shutdown_connections 2024-02-21T23:04:38.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.552+0000 7fa8eb400640 1 --2- 172.21.15.28:0/801477684 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa8c406b500 0x7fa8c406d9c0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:38.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.552+0000 7fa8eb400640 1 --2- 172.21.15.28:0/801477684 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa8e40c5320 0x7fa8e41ec8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:38.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.552+0000 7fa8eb400640 1 --2- 172.21.15.28:0/801477684 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8e413e480 0x7fa8e41eec10 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:38.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.552+0000 7fa8eb400640 1 --2- 172.21.15.28:0/801477684 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa8e40bfb60 0x7fa8e4158c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:38.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.552+0000 7fa8eb400640 1 -- 172.21.15.28:0/801477684 >> 172.21.15.28:0/801477684 conn(0x7fa8e408b1c0 msgr2=0x7fa8e4137e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:38.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.553+0000 7fa8eb400640 1 -- 172.21.15.28:0/801477684 shutdown_connections 2024-02-21T23:04:38.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.553+0000 7fa8eb400640 1 -- 172.21.15.28:0/801477684 wait complete. 2024-02-21T23:04:38.560 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-02-21T23:04:38.561 INFO:tasks.ceph.mds.c.smithi028.stderr: -367> 2024-02-21T23:04:27.355+0000 7fc419000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-21T23:04:38.608 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-02-21T23:04:38.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.763+0000 7f1d6e000640 1 Processor -- start 2024-02-21T23:04:38.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.763+0000 7f1d6e000640 1 -- start start 2024-02-21T23:04:38.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.765+0000 7f1d6e000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1d680c6220 0x7f1d681586b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:38.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.765+0000 7f1d6e000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1d68158bf0 0x7f1d6815f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:38.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.765+0000 7f1d6e000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d680c20d0 con 0x7f1d680c4960 2024-02-21T23:04:38.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.765+0000 7f1d6e000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d680a62e0 con 0x7f1d68158bf0 2024-02-21T23:04:38.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.765+0000 7f1d6e000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d680c1960 con 0x7f1d680c6220 2024-02-21T23:04:38.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.765+0000 7f1d67e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1d68158bf0 0x7f1d6815f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:38.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.765+0000 7f1d67400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f1d680c4960 0x7f1d681524f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39930/0 (socket says 172.21.15.28:39930) 2024-02-21T23:04:38.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.765+0000 7f1d67400640 1 -- 172.21.15.28:0/2197342596 learned_addr learned my addr 172.21.15.28:0/2197342596 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:04:38.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.765+0000 7f1d66a00640 1 --2- 172.21.15.28:0/2197342596 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1d680c6220 0x7f1d681586b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:38.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.766+0000 7f1d67e00640 1 -- 172.21.15.28:0/2197342596 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1d680c6220 msgr2=0x7f1d681586b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:38.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.766+0000 7f1d67e00640 1 --2- 172.21.15.28:0/2197342596 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1d680c6220 0x7f1d681586b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:38.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.766+0000 7f1d67e00640 1 -- 172.21.15.28:0/2197342596 >> v1:172.21.15.28:6789/0 conn(0x7f1d680c4960 legacy=0x7f1d681524f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:38.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.766+0000 7f1d66000640 1 -- 172.21.15.28:0/2197342596 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2892168918 0 0) 0x7f1d680c20d0 con 0x7f1d680c4960 2024-02-21T23:04:38.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.766+0000 7f1d67e00640 1 -- 172.21.15.28:0/2197342596 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d680c1290 con 0x7f1d68158bf0 2024-02-21T23:04:38.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.767+0000 7f1d67e00640 1 --2- 172.21.15.28:0/2197342596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d68158bf0 0x7f1d6815f9e0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f1d60009d90 tx=0x7f1d6002be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5112105fadd5ad6 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:38.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.767+0000 7f1d66000640 1 -- 172.21.15.28:0/2197342596 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d60039070 con 0x7f1d68158bf0 2024-02-21T23:04:38.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.767+0000 7f1d66000640 1 -- 172.21.15.28:0/2197342596 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d6002cbc0 con 0x7f1d68158bf0 2024-02-21T23:04:38.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.767+0000 7f1d66000640 1 -- 172.21.15.28:0/2197342596 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d60034440 con 0x7f1d68158bf0 2024-02-21T23:04:38.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.768+0000 7f1d6e000640 1 -- 172.21.15.28:0/2197342596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d68158bf0 msgr2=0x7f1d6815f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:38.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.768+0000 7f1d6e000640 1 --2- 172.21.15.28:0/2197342596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d68158bf0 0x7f1d6815f9e0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f1d60009d90 tx=0x7f1d6002be10 comp rx=0 tx=0).stop 2024-02-21T23:04:38.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.768+0000 7f1d6e000640 1 -- 172.21.15.28:0/2197342596 shutdown_connections 2024-02-21T23:04:38.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.768+0000 7f1d6e000640 1 --2- 172.21.15.28:0/2197342596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d68158bf0 0x7f1d6815f9e0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:38.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.768+0000 7f1d6e000640 1 --2- 172.21.15.28:0/2197342596 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1d680c6220 0x7f1d681586b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:38.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.768+0000 7f1d6e000640 1 -- 172.21.15.28:0/2197342596 >> 172.21.15.28:0/2197342596 conn(0x7f1d680a85a0 msgr2=0x7f1d680a8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:38.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.768+0000 7f1d6e000640 1 -- 172.21.15.28:0/2197342596 shutdown_connections 2024-02-21T23:04:38.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.768+0000 7f1d6e000640 1 -- 172.21.15.28:0/2197342596 wait complete. 2024-02-21T23:04:38.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.770+0000 7f1d6e000640 1 Processor -- start 2024-02-21T23:04:38.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.770+0000 7f1d6e000640 1 -- start start 2024-02-21T23:04:38.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.771+0000 7f1d6e000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d680c4960 0x7f1d6814ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:38.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.771+0000 7f1d6e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d680c6220 0x7f1d681deac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:38.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.771+0000 7f1d6e000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1d68158bf0 0x7f1d680aa810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:38.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.771+0000 7f1d6e000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d680a5f20 con 0x7f1d680c6220 2024-02-21T23:04:38.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.771+0000 7f1d6e000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d680c1bb0 con 0x7f1d680c4960 2024-02-21T23:04:38.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.771+0000 7f1d6e000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d680a60a0 con 0x7f1d68158bf0 2024-02-21T23:04:38.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.771+0000 7f1d66a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d680c6220 0x7f1d681deac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:38.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.771+0000 7f1d66a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d680c6220 0x7f1d681deac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52494/0 (socket says 172.21.15.28:52494) 2024-02-21T23:04:38.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.771+0000 7f1d66a00640 1 -- 172.21.15.28:0/3628858244 learned_addr learned my addr 172.21.15.28:0/3628858244 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:38.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.771+0000 7f1d66a00640 1 -- 172.21.15.28:0/3628858244 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1d68158bf0 msgr2=0x7f1d680aa810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:38.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.771+0000 7f1d67e00640 1 --2- 172.21.15.28:0/3628858244 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1d68158bf0 0x7f1d680aa810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:38.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.771+0000 7f1d66a00640 1 --2- 172.21.15.28:0/3628858244 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1d68158bf0 0x7f1d680aa810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:38.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.771+0000 7f1d66a00640 1 -- 172.21.15.28:0/3628858244 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d680c4960 msgr2=0x7f1d6814ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:38.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.771+0000 7f1d66a00640 1 --2- 172.21.15.28:0/3628858244 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d680c4960 0x7f1d6814ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:38.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.771+0000 7f1d66a00640 1 -- 172.21.15.28:0/3628858244 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d6814e730 con 0x7f1d680c6220 2024-02-21T23:04:38.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.772+0000 7f1d66a00640 1 --2- 172.21.15.28:0/3628858244 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d680c6220 0x7f1d681deac0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f1d5400d800 tx=0x7f1d5400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:38.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.772+0000 7f1d5f400640 1 -- 172.21.15.28:0/3628858244 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d5400b6c0 con 0x7f1d680c6220 2024-02-21T23:04:38.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.772+0000 7f1d5f400640 1 -- 172.21.15.28:0/3628858244 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d54019070 con 0x7f1d680c6220 2024-02-21T23:04:38.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.772+0000 7f1d5f400640 1 -- 172.21.15.28:0/3628858244 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d5400ba80 con 0x7f1d680c6220 2024-02-21T23:04:38.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.772+0000 7f1d6e000640 1 -- 172.21.15.28:0/3628858244 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d681eb980 con 0x7f1d680c6220 2024-02-21T23:04:38.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.772+0000 7f1d6e000640 1 -- 172.21.15.28:0/3628858244 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d680aa010 con 0x7f1d680c6220 2024-02-21T23:04:38.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.773+0000 7f1d6e000640 1 -- 172.21.15.28:0/3628858244 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d680a9b20 con 0x7f1d680c6220 2024-02-21T23:04:38.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.774+0000 7f1d5f400640 1 -- 172.21.15.28:0/3628858244 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1d5400bc20 con 0x7f1d680c6220 2024-02-21T23:04:38.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.776+0000 7f1d5f400640 1 --2- 172.21.15.28:0/3628858244 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1d4806b500 0x7f1d4806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:38.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.776+0000 7f1d5f400640 1 -- 172.21.15.28:0/3628858244 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6302+0+0 (secure 0 0 0) 0x7f1d5408a880 con 0x7f1d680c6220 2024-02-21T23:04:38.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.776+0000 7f1d67400640 1 --2- 172.21.15.28:0/3628858244 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1d4806b500 0x7f1d4806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:38.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.777+0000 7f1d67400640 1 --2- 172.21.15.28:0/3628858244 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1d4806b500 0x7f1d4806d9c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f1d58007fd0 tx=0x7f1d58007920 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:38.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:38.781+0000 7f1d5f400640 1 -- 172.21.15.28:0/3628858244 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1d5401c360 con 0x7f1d680c6220 2024-02-21T23:04:39.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.023+0000 7f1d6e000640 1 -- 172.21.15.28:0/3628858244 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f1d68067f30 con 0x7f1d680c6220 2024-02-21T23:04:39.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.552+0000 7f1d5f400640 1 -- 172.21.15.28:0/3628858244 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v26) v1 ==== 104+0+0 (secure 0 0 0) 0x7f1d54059540 con 0x7f1d680c6220 2024-02-21T23:04:39.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.556+0000 7f1d6e000640 1 -- 172.21.15.28:0/3628858244 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1d4806b500 msgr2=0x7f1d4806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:39.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.556+0000 7f1d6e000640 1 --2- 172.21.15.28:0/3628858244 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1d4806b500 0x7f1d4806d9c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f1d58007fd0 tx=0x7f1d58007920 comp rx=0 tx=0).stop 2024-02-21T23:04:39.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.556+0000 7f1d6e000640 1 -- 172.21.15.28:0/3628858244 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d680c6220 msgr2=0x7f1d681deac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:39.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.556+0000 7f1d6e000640 1 --2- 172.21.15.28:0/3628858244 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d680c6220 0x7f1d681deac0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f1d5400d800 tx=0x7f1d5400dcc0 comp rx=0 tx=0).stop 2024-02-21T23:04:39.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.556+0000 7f1d6e000640 1 -- 172.21.15.28:0/3628858244 shutdown_connections 2024-02-21T23:04:39.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.556+0000 7f1d6e000640 1 --2- 172.21.15.28:0/3628858244 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1d4806b500 0x7f1d4806d9c0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:39.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.556+0000 7f1d6e000640 1 --2- 172.21.15.28:0/3628858244 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1d68158bf0 0x7f1d680aa810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:39.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.556+0000 7f1d6e000640 1 --2- 172.21.15.28:0/3628858244 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d680c6220 0x7f1d681deac0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:39.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.556+0000 7f1d6e000640 1 --2- 172.21.15.28:0/3628858244 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d680c4960 0x7f1d6814ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:39.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.556+0000 7f1d6e000640 1 -- 172.21.15.28:0/3628858244 >> 172.21.15.28:0/3628858244 conn(0x7f1d680a85a0 msgr2=0x7f1d681522c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:39.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.556+0000 7f1d6e000640 1 -- 172.21.15.28:0/3628858244 shutdown_connections 2024-02-21T23:04:39.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.557+0000 7f1d6e000640 1 -- 172.21.15.28:0/3628858244 wait complete. 2024-02-21T23:04:39.568 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-02-21T23:04:39.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.806+0000 7feb56e00640 1 Processor -- start 2024-02-21T23:04:39.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.806+0000 7feb56e00640 1 -- start start 2024-02-21T23:04:39.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.808+0000 7feb56e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7feb5006cfa0 0x7feb500ca540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:39.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.808+0000 7feb56e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feb50158460 0x7feb500c6290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:39.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.808+0000 7feb56e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb500caf40 con 0x7feb500c8f70 2024-02-21T23:04:39.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.808+0000 7feb56e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb500c8ae0 con 0x7feb5006cfa0 2024-02-21T23:04:39.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.808+0000 7feb56e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb500c87b0 con 0x7feb50158460 2024-02-21T23:04:39.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.808+0000 7feb56400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7feb500c8f70 0x7feb501522c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39946/0 (socket says 172.21.15.28:39946) 2024-02-21T23:04:39.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.808+0000 7feb56400640 1 -- 172.21.15.28:0/1398443572 learned_addr learned my addr 172.21.15.28:0/1398443572 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:04:39.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.808+0000 7feb55a00640 1 --2- 172.21.15.28:0/1398443572 >> v2:172.21.15.28:3300/0 conn(0x7feb5006cfa0 0x7feb500ca540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:39.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.809+0000 7feb55000640 1 --2- 172.21.15.28:0/1398443572 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feb50158460 0x7feb500c6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:39.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.809+0000 7feb4fe00640 1 -- 172.21.15.28:0/1398443572 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2122459722 0 0) 0x7feb500caf40 con 0x7feb500c8f70 2024-02-21T23:04:39.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.809+0000 7feb4fe00640 1 -- 172.21.15.28:0/1398443572 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb38003520 con 0x7feb500c8f70 2024-02-21T23:04:39.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.809+0000 7feb55a00640 1 -- 172.21.15.28:0/1398443572 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feb50158460 msgr2=0x7feb500c6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:39.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.809+0000 7feb55a00640 1 --2- 172.21.15.28:0/1398443572 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feb50158460 0x7feb500c6290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:39.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.809+0000 7feb55a00640 1 -- 172.21.15.28:0/1398443572 >> v1:172.21.15.28:6789/0 conn(0x7feb500c8f70 legacy=0x7feb501522c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:39.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.809+0000 7feb55a00640 1 -- 172.21.15.28:0/1398443572 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb501588f0 con 0x7feb5006cfa0 2024-02-21T23:04:39.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.811+0000 7feb55a00640 1 --2- 172.21.15.28:0/1398443572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb5006cfa0 0x7feb500ca540 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7feb440097a0 tx=0x7feb4402b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61ee36c7b4e598dd server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:39.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.811+0000 7feb4fe00640 1 -- 172.21.15.28:0/1398443572 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb44039070 con 0x7feb5006cfa0 2024-02-21T23:04:39.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.811+0000 7feb4fe00640 1 -- 172.21.15.28:0/1398443572 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb440316f0 con 0x7feb5006cfa0 2024-02-21T23:04:39.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.811+0000 7feb4fe00640 1 -- 172.21.15.28:0/1398443572 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb44031ac0 con 0x7feb5006cfa0 2024-02-21T23:04:39.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.812+0000 7feb56e00640 1 -- 172.21.15.28:0/1398443572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb5006cfa0 msgr2=0x7feb500ca540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:39.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.812+0000 7feb56e00640 1 --2- 172.21.15.28:0/1398443572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb5006cfa0 0x7feb500ca540 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7feb440097a0 tx=0x7feb4402b790 comp rx=0 tx=0).stop 2024-02-21T23:04:39.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.812+0000 7feb56e00640 1 -- 172.21.15.28:0/1398443572 shutdown_connections 2024-02-21T23:04:39.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.812+0000 7feb56e00640 1 --2- 172.21.15.28:0/1398443572 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feb50158460 0x7feb500c6290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:39.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.812+0000 7feb56e00640 1 --2- 172.21.15.28:0/1398443572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb5006cfa0 0x7feb500ca540 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:39.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.812+0000 7feb56e00640 1 -- 172.21.15.28:0/1398443572 >> 172.21.15.28:0/1398443572 conn(0x7feb500725a0 msgr2=0x7feb500729b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:39.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.812+0000 7feb56e00640 1 -- 172.21.15.28:0/1398443572 shutdown_connections 2024-02-21T23:04:39.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.812+0000 7feb56e00640 1 -- 172.21.15.28:0/1398443572 wait complete. 2024-02-21T23:04:39.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.814+0000 7feb56e00640 1 Processor -- start 2024-02-21T23:04:39.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.814+0000 7feb56e00640 1 -- start start 2024-02-21T23:04:39.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.815+0000 7feb56e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feb5006cfa0 0x7feb50156c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:39.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.815+0000 7feb56e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feb500c8f70 0x7feb501552e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:39.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.815+0000 7feb56e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb50158460 0x7feb50149a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:39.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.815+0000 7feb56e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb500c6c90 con 0x7feb50158460 2024-02-21T23:04:39.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.815+0000 7feb56e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb500c69b0 con 0x7feb500c8f70 2024-02-21T23:04:39.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.815+0000 7feb56e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb501529d0 con 0x7feb5006cfa0 2024-02-21T23:04:39.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.815+0000 7feb56400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb50158460 0x7feb50149a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:39.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.815+0000 7feb56400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb50158460 0x7feb50149a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52528/0 (socket says 172.21.15.28:52528) 2024-02-21T23:04:39.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.815+0000 7feb56400640 1 -- 172.21.15.28:0/2655535181 learned_addr learned my addr 172.21.15.28:0/2655535181 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:39.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.816+0000 7feb55000640 1 --2- 172.21.15.28:0/2655535181 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feb500c8f70 0x7feb501552e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:39.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.816+0000 7feb55a00640 1 --2- 172.21.15.28:0/2655535181 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feb5006cfa0 0x7feb50156c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:39.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.816+0000 7feb56400640 1 -- 172.21.15.28:0/2655535181 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feb5006cfa0 msgr2=0x7feb50156c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:39.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.816+0000 7feb56400640 1 --2- 172.21.15.28:0/2655535181 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feb5006cfa0 0x7feb50156c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:39.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.816+0000 7feb56400640 1 -- 172.21.15.28:0/2655535181 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feb500c8f70 msgr2=0x7feb501552e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:39.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.816+0000 7feb56400640 1 --2- 172.21.15.28:0/2655535181 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feb500c8f70 0x7feb501552e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:39.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.816+0000 7feb56400640 1 -- 172.21.15.28:0/2655535181 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb500ac4b0 con 0x7feb50158460 2024-02-21T23:04:39.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.816+0000 7feb56400640 1 --2- 172.21.15.28:0/2655535181 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb50158460 0x7feb50149a80 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7feb4800cb70 tx=0x7feb4800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:39.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.817+0000 7feb4d600640 1 -- 172.21.15.28:0/2655535181 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb48017070 con 0x7feb50158460 2024-02-21T23:04:39.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.817+0000 7feb56e00640 1 -- 172.21.15.28:0/2655535181 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb500ac790 con 0x7feb50158460 2024-02-21T23:04:39.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.817+0000 7feb4d600640 1 -- 172.21.15.28:0/2655535181 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb4800ace0 con 0x7feb50158460 2024-02-21T23:04:39.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.817+0000 7feb4d600640 1 -- 172.21.15.28:0/2655535181 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb48012620 con 0x7feb50158460 2024-02-21T23:04:39.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.817+0000 7feb56e00640 1 -- 172.21.15.28:0/2655535181 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb500acd50 con 0x7feb50158460 2024-02-21T23:04:39.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.818+0000 7feb56e00640 1 -- 172.21.15.28:0/2655535181 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb5004f290 con 0x7feb50158460 2024-02-21T23:04:39.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.819+0000 7feb4d600640 1 -- 172.21.15.28:0/2655535181 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feb4801a070 con 0x7feb50158460 2024-02-21T23:04:39.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.821+0000 7feb4d600640 1 --2- 172.21.15.28:0/2655535181 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7feb3406b4d0 0x7feb3406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:39.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.821+0000 7feb4d600640 1 -- 172.21.15.28:0/2655535181 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6302+0+0 (secure 0 0 0) 0x7feb4808a2d0 con 0x7feb50158460 2024-02-21T23:04:39.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.821+0000 7feb55a00640 1 --2- 172.21.15.28:0/2655535181 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7feb3406b4d0 0x7feb3406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:39.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.822+0000 7feb55a00640 1 --2- 172.21.15.28:0/2655535181 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7feb3406b4d0 0x7feb3406d990 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7feb44036000 tx=0x7feb44008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:39.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:39.825+0000 7feb4d600640 1 -- 172.21.15.28:0/2655535181 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7feb48058ef0 con 0x7feb50158460 2024-02-21T23:04:40.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:40.024+0000 7feb56e00640 1 -- 172.21.15.28:0/2655535181 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7feb500cad60 con 0x7feb50158460 2024-02-21T23:04:40.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:40.812+0000 7feb4d600640 1 -- 172.21.15.28:0/2655535181 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v43) v1 ==== 181+0+0 (secure 0 0 0) 0x7feb480587f0 con 0x7feb50158460 2024-02-21T23:04:40.802 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' removed 2024-02-21T23:04:40.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:40.816+0000 7feb56e00640 1 -- 172.21.15.28:0/2655535181 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7feb3406b4d0 msgr2=0x7feb3406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:40.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:40.816+0000 7feb56e00640 1 --2- 172.21.15.28:0/2655535181 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7feb3406b4d0 0x7feb3406d990 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7feb44036000 tx=0x7feb44008030 comp rx=0 tx=0).stop 2024-02-21T23:04:40.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:40.816+0000 7feb56e00640 1 -- 172.21.15.28:0/2655535181 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb50158460 msgr2=0x7feb50149a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:40.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:40.816+0000 7feb56e00640 1 --2- 172.21.15.28:0/2655535181 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb50158460 0x7feb50149a80 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7feb4800cb70 tx=0x7feb4800a430 comp rx=0 tx=0).stop 2024-02-21T23:04:40.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:40.816+0000 7feb56e00640 1 -- 172.21.15.28:0/2655535181 shutdown_connections 2024-02-21T23:04:40.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:40.816+0000 7feb56e00640 1 --2- 172.21.15.28:0/2655535181 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7feb3406b4d0 0x7feb3406d990 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:40.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:40.816+0000 7feb56e00640 1 --2- 172.21.15.28:0/2655535181 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb50158460 0x7feb50149a80 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:40.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:40.816+0000 7feb56e00640 1 --2- 172.21.15.28:0/2655535181 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feb500c8f70 0x7feb501552e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:40.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:40.816+0000 7feb56e00640 1 --2- 172.21.15.28:0/2655535181 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feb5006cfa0 0x7feb50156c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:40.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:40.817+0000 7feb56e00640 1 -- 172.21.15.28:0/2655535181 >> 172.21.15.28:0/2655535181 conn(0x7feb500725a0 msgr2=0x7feb50072980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:40.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:40.819+0000 7feb56e00640 1 -- 172.21.15.28:0/2655535181 shutdown_connections 2024-02-21T23:04:40.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:40.820+0000 7feb56e00640 1 -- 172.21.15.28:0/2655535181 wait complete. 2024-02-21T23:04:40.833 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-02-21T23:04:41.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.054+0000 7fe605000640 1 Processor -- start 2024-02-21T23:04:41.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.054+0000 7fe605000640 1 -- start start 2024-02-21T23:04:41.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.056+0000 7fe605000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fe6001523f0 0x7fe6000c66e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:41.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.056+0000 7fe605000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6000c3a40 con 0x7fe600158f30 2024-02-21T23:04:41.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.056+0000 7fe605000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6001593c0 con 0x7fe6000c2f00 2024-02-21T23:04:41.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.056+0000 7fe605000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6000c6d70 con 0x7fe6001523f0 2024-02-21T23:04:41.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.056+0000 7fe5fea00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe600158f30 0x7fe6000c71a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39958/0 (socket says 172.21.15.28:39958) 2024-02-21T23:04:41.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.056+0000 7fe5fea00640 1 -- 172.21.15.28:0/49515021 learned_addr learned my addr 172.21.15.28:0/49515021 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:04:41.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.057+0000 7fe5fe000640 1 -- 172.21.15.28:0/49515021 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1917602216 0 0) 0x7fe6000c3a40 con 0x7fe600158f30 2024-02-21T23:04:41.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.057+0000 7fe5fe000640 1 -- 172.21.15.28:0/49515021 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5e4003520 con 0x7fe600158f30 2024-02-21T23:04:41.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.057+0000 7fe5ff400640 1 --2- 172.21.15.28:0/49515021 >> v2:172.21.15.81:3300/0 conn(0x7fe6001523f0 0x7fe6000c66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:41.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.057+0000 7fe5fe000640 1 -- 172.21.15.28:0/49515021 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 291813929 0 0) 0x7fe5e4003520 con 0x7fe600158f30 2024-02-21T23:04:41.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.057+0000 7fe5fe000640 1 -- 172.21.15.28:0/49515021 >> v2:172.21.15.81:3300/0 conn(0x7fe6001523f0 msgr2=0x7fe6000c66e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:41.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.057+0000 7fe5fe000640 1 --2- 172.21.15.28:0/49515021 >> v2:172.21.15.81:3300/0 conn(0x7fe6001523f0 0x7fe6000c66e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:41.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.057+0000 7fe5fe000640 1 -- 172.21.15.28:0/49515021 >> v1:172.21.15.81:6789/0 conn(0x7fe6000c2f00 legacy=0x7fe6000c3330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:41.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.057+0000 7fe5fe000640 1 -- 172.21.15.28:0/49515021 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe600155e20 con 0x7fe600158f30 2024-02-21T23:04:41.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.057+0000 7fe5fe000640 1 -- 172.21.15.28:0/49515021 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fe5e8001670 con 0x7fe600158f30 2024-02-21T23:04:41.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.057+0000 7fe5fe000640 1 -- 172.21.15.28:0/49515021 >> v1:172.21.15.28:6789/0 conn(0x7fe600158f30 legacy=0x7fe6000c71a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:41.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.057+0000 7fe5fe000640 1 --2- 172.21.15.28:0/49515021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5e40039b0 0x7fe5e4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:41.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.057+0000 7fe5fe000640 1 --2- 172.21.15.28:0/49515021 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe5e4004800 0x7fe5e4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:41.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.058+0000 7fe5fe000640 1 --2- 172.21.15.28:0/49515021 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe5e40052e0 0x7fe5e400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:41.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.058+0000 7fe5fe000640 1 -- 172.21.15.28:0/49515021 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6000c6d70 con 0x7fe5e40039b0 2024-02-21T23:04:41.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.058+0000 7fe5fe000640 1 -- 172.21.15.28:0/49515021 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e400c1b0 con 0x7fe5e40052e0 2024-02-21T23:04:41.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.058+0000 7fe5fe000640 1 -- 172.21.15.28:0/49515021 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e400c3b0 con 0x7fe5e4004800 2024-02-21T23:04:41.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.058+0000 7fe5ff400640 1 --2- 172.21.15.28:0/49515021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5e40039b0 0x7fe5e4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:41.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.058+0000 7fe5fea00640 1 --2- 172.21.15.28:0/49515021 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe5e4004800 0x7fe5e4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:41.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.058+0000 7fe5ffe00640 1 --2- 172.21.15.28:0/49515021 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe5e40052e0 0x7fe5e400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:41.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.058+0000 7fe5ff400640 1 -- 172.21.15.28:0/49515021 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe5e4004800 msgr2=0x7fe5e4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:41.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.058+0000 7fe5ff400640 1 --2- 172.21.15.28:0/49515021 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe5e4004800 0x7fe5e4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:41.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.058+0000 7fe5ff400640 1 -- 172.21.15.28:0/49515021 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe5e40052e0 msgr2=0x7fe5e400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:41.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.058+0000 7fe5ff400640 1 --2- 172.21.15.28:0/49515021 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe5e40052e0 0x7fe5e400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:41.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.058+0000 7fe5ff400640 1 -- 172.21.15.28:0/49515021 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5e4003520 con 0x7fe5e40039b0 2024-02-21T23:04:41.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.058+0000 7fe5fea00640 1 --2- 172.21.15.28:0/49515021 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe5e4004800 0x7fe5e4004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:04:41.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.058+0000 7fe5ffe00640 1 --2- 172.21.15.28:0/49515021 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe5e40052e0 0x7fe5e400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:04:41.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.058+0000 7fe5ffe00640 1 -- 172.21.15.28:0/49515021 reap_dead start 2024-02-21T23:04:41.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.059+0000 7fe5ff400640 1 --2- 172.21.15.28:0/49515021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5e40039b0 0x7fe5e4003dd0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fe5f000ba00 tx=0x7fe5e4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=518ab7120dc87b71 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:41.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.060+0000 7fe5fe000640 1 -- 172.21.15.28:0/49515021 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f0036020 con 0x7fe5e40039b0 2024-02-21T23:04:41.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.060+0000 7fe5fe000640 1 -- 172.21.15.28:0/49515021 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5f0032620 con 0x7fe5e40039b0 2024-02-21T23:04:41.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.060+0000 7fe5fe000640 1 -- 172.21.15.28:0/49515021 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f000b600 con 0x7fe5e40039b0 2024-02-21T23:04:41.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.060+0000 7fe605000640 1 -- 172.21.15.28:0/49515021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5e40039b0 msgr2=0x7fe5e4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:41.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.060+0000 7fe605000640 1 --2- 172.21.15.28:0/49515021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5e40039b0 0x7fe5e4003dd0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fe5f000ba00 tx=0x7fe5e4005830 comp rx=0 tx=0).stop 2024-02-21T23:04:41.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.061+0000 7fe605000640 1 -- 172.21.15.28:0/49515021 shutdown_connections 2024-02-21T23:04:41.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.061+0000 7fe605000640 1 --2- 172.21.15.28:0/49515021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5e40039b0 0x7fe5e4003dd0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:41.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.061+0000 7fe605000640 1 -- 172.21.15.28:0/49515021 >> 172.21.15.28:0/49515021 conn(0x7fe600072c90 msgr2=0x7fe6000ca790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:41.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.061+0000 7fe605000640 1 -- 172.21.15.28:0/49515021 shutdown_connections 2024-02-21T23:04:41.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.061+0000 7fe605000640 1 -- 172.21.15.28:0/49515021 wait complete. 2024-02-21T23:04:41.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.062+0000 7fe605000640 1 Processor -- start 2024-02-21T23:04:41.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.063+0000 7fe605000640 1 -- start start 2024-02-21T23:04:41.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.063+0000 7fe605000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5e40039b0 0x7fe60014a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:41.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.063+0000 7fe605000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe6001eaa40 0x7fe6001ee530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:41.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.063+0000 7fe605000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe600148aa0 0x7fe6001eea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:41.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.063+0000 7fe605000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe600155620 con 0x7fe5e40039b0 2024-02-21T23:04:41.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.064+0000 7fe605000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe600155340 con 0x7fe600148aa0 2024-02-21T23:04:41.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.064+0000 7fe605000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe600155900 con 0x7fe6001eaa40 2024-02-21T23:04:41.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.064+0000 7fe5ff400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5e40039b0 0x7fe60014a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:41.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.064+0000 7fe5ff400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5e40039b0 0x7fe60014a630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52544/0 (socket says 172.21.15.28:52544) 2024-02-21T23:04:41.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.064+0000 7fe5ff400640 1 -- 172.21.15.28:0/1327514748 learned_addr learned my addr 172.21.15.28:0/1327514748 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:41.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.064+0000 7fe5fea00640 1 --2- 172.21.15.28:0/1327514748 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe6001eaa40 0x7fe6001ee530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:41.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.064+0000 7fe5ffe00640 1 --2- 172.21.15.28:0/1327514748 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe600148aa0 0x7fe6001eea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:41.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.064+0000 7fe5ff400640 1 -- 172.21.15.28:0/1327514748 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe6001eaa40 msgr2=0x7fe6001ee530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:41.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.064+0000 7fe5ff400640 1 --2- 172.21.15.28:0/1327514748 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe6001eaa40 0x7fe6001ee530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:41.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.064+0000 7fe5ff400640 1 -- 172.21.15.28:0/1327514748 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe600148aa0 msgr2=0x7fe6001eea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:41.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.064+0000 7fe5ff400640 1 --2- 172.21.15.28:0/1327514748 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe600148aa0 0x7fe6001eea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:41.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.064+0000 7fe5ff400640 1 -- 172.21.15.28:0/1327514748 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6001481a0 con 0x7fe5e40039b0 2024-02-21T23:04:41.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.065+0000 7fe5ff400640 1 --2- 172.21.15.28:0/1327514748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5e40039b0 0x7fe60014a630 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fe5f0035040 tx=0x7fe5f0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:41.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.065+0000 7fe5f7400640 1 -- 172.21.15.28:0/1327514748 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f0041070 con 0x7fe5e40039b0 2024-02-21T23:04:41.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.065+0000 7fe5f7400640 1 -- 172.21.15.28:0/1327514748 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5f003b620 con 0x7fe5e40039b0 2024-02-21T23:04:41.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.065+0000 7fe5f7400640 1 -- 172.21.15.28:0/1327514748 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f0036040 con 0x7fe5e40039b0 2024-02-21T23:04:41.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.065+0000 7fe605000640 1 -- 172.21.15.28:0/1327514748 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6001ee050 con 0x7fe5e40039b0 2024-02-21T23:04:41.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.065+0000 7fe605000640 1 -- 172.21.15.28:0/1327514748 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe600151730 con 0x7fe5e40039b0 2024-02-21T23:04:41.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.066+0000 7fe605000640 1 -- 172.21.15.28:0/1327514748 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe60004f290 con 0x7fe5e40039b0 2024-02-21T23:04:41.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.067+0000 7fe5f7400640 1 -- 172.21.15.28:0/1327514748 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe5f004c060 con 0x7fe5e40039b0 2024-02-21T23:04:41.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.069+0000 7fe5f7400640 1 --2- 172.21.15.28:0/1327514748 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe5dc06b4d0 0x7fe5dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:41.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.069+0000 7fe5f7400640 1 -- 172.21.15.28:0/1327514748 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5864+0+0 (secure 0 0 0) 0x7fe5f00abd60 con 0x7fe5e40039b0 2024-02-21T23:04:41.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.069+0000 7fe5fea00640 1 --2- 172.21.15.28:0/1327514748 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe5dc06b4d0 0x7fe5dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:41.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.070+0000 7fe5fea00640 1 --2- 172.21.15.28:0/1327514748 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe5dc06b4d0 0x7fe5dc06d990 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fe6001eae20 tx=0x7fe5e8001bc0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:41.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.073+0000 7fe5f7400640 1 -- 172.21.15.28:0/1327514748 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe5f007ab30 con 0x7fe5e40039b0 2024-02-21T23:04:41.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.274+0000 7fe605000640 1 -- 172.21.15.28:0/1327514748 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe60014b9d0 con 0x7fe5e40039b0 2024-02-21T23:04:41.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.816+0000 7fe5f7400640 1 -- 172.21.15.28:0/1327514748 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v44) v1 ==== 169+0+0 (secure 0 0 0) 0x7fe5f007a430 con 0x7fe5e40039b0 2024-02-21T23:04:41.805 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' removed 2024-02-21T23:04:41.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.820+0000 7fe605000640 1 -- 172.21.15.28:0/1327514748 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe5dc06b4d0 msgr2=0x7fe5dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:41.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.820+0000 7fe605000640 1 --2- 172.21.15.28:0/1327514748 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe5dc06b4d0 0x7fe5dc06d990 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fe6001eae20 tx=0x7fe5e8001bc0 comp rx=0 tx=0).stop 2024-02-21T23:04:41.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.820+0000 7fe605000640 1 -- 172.21.15.28:0/1327514748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5e40039b0 msgr2=0x7fe60014a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:41.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.820+0000 7fe605000640 1 --2- 172.21.15.28:0/1327514748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5e40039b0 0x7fe60014a630 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fe5f0035040 tx=0x7fe5f0004730 comp rx=0 tx=0).stop 2024-02-21T23:04:41.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.821+0000 7fe605000640 1 -- 172.21.15.28:0/1327514748 shutdown_connections 2024-02-21T23:04:41.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.821+0000 7fe605000640 1 --2- 172.21.15.28:0/1327514748 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe5dc06b4d0 0x7fe5dc06d990 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:41.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.821+0000 7fe605000640 1 --2- 172.21.15.28:0/1327514748 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe6001eaa40 0x7fe6001ee530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:41.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.821+0000 7fe605000640 1 --2- 172.21.15.28:0/1327514748 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe600148aa0 0x7fe6001eea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:41.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.821+0000 7fe605000640 1 --2- 172.21.15.28:0/1327514748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5e40039b0 0x7fe60014a630 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:41.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.821+0000 7fe605000640 1 -- 172.21.15.28:0/1327514748 >> 172.21.15.28:0/1327514748 conn(0x7fe600072c90 msgr2=0x7fe6000c4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:41.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.822+0000 7fe605000640 1 -- 172.21.15.28:0/1327514748 shutdown_connections 2024-02-21T23:04:41.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:41.824+0000 7fe605000640 1 -- 172.21.15.28:0/1327514748 wait complete. 2024-02-21T23:04:41.839 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.c mds_join_fs 2024-02-21T23:04:42.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.051+0000 7fc49ca00640 1 Processor -- start 2024-02-21T23:04:42.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.051+0000 7fc49ca00640 1 -- start start 2024-02-21T23:04:42.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.052+0000 7fc49ca00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc498152360 0x7fc4980c5800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:42.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.052+0000 7fc49ca00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc4980c48a0 0x7fc49806ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:42.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.052+0000 7fc49ca00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fc49806b3b0 0x7fc498154e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:42.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.052+0000 7fc49ca00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4980c1ee0 con 0x7fc498152360 2024-02-21T23:04:42.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.052+0000 7fc49ca00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4980c21c0 con 0x7fc49806b3b0 2024-02-21T23:04:42.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.052+0000 7fc49ca00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4980c24a0 con 0x7fc4980c48a0 2024-02-21T23:04:42.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.052+0000 7fc497400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc498152360 0x7fc4980c5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:42.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.053+0000 7fc497400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc498152360 0x7fc4980c5800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52546/0 (socket says 172.21.15.28:52546) 2024-02-21T23:04:42.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.053+0000 7fc497400640 1 -- 172.21.15.28:0/3831188711 learned_addr learned my addr 172.21.15.28:0/3831188711 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:42.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.053+0000 7fc497400640 1 -- 172.21.15.28:0/3831188711 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc4980c48a0 msgr2=0x7fc49806ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:42.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.053+0000 7fc496a00640 1 --2- 172.21.15.28:0/3831188711 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc4980c48a0 0x7fc49806ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:42.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.053+0000 7fc497e00640 1 --2- 172.21.15.28:0/3831188711 >> v2:172.21.15.81:3300/0 conn(0x7fc49806b3b0 0x7fc498154e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:42.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.053+0000 7fc497400640 1 --2- 172.21.15.28:0/3831188711 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc4980c48a0 0x7fc49806ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:42.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.053+0000 7fc497400640 1 -- 172.21.15.28:0/3831188711 >> v2:172.21.15.81:3300/0 conn(0x7fc49806b3b0 msgr2=0x7fc498154e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:42.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.053+0000 7fc497400640 1 --2- 172.21.15.28:0/3831188711 >> v2:172.21.15.81:3300/0 conn(0x7fc49806b3b0 0x7fc498154e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:42.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.053+0000 7fc497400640 1 -- 172.21.15.28:0/3831188711 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4980c2780 con 0x7fc498152360 2024-02-21T23:04:42.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.053+0000 7fc496a00640 1 --2- 172.21.15.28:0/3831188711 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc4980c48a0 0x7fc49806ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:04:42.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.053+0000 7fc497e00640 1 --2- 172.21.15.28:0/3831188711 >> v2:172.21.15.81:3300/0 conn(0x7fc49806b3b0 0x7fc498154e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:04:42.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.054+0000 7fc497400640 1 --2- 172.21.15.28:0/3831188711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc498152360 0x7fc4980c5800 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fc4980a5d30 tx=0x7fc48802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=198b2cb19a1d2f8 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:42.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.055+0000 7fc496000640 1 -- 172.21.15.28:0/3831188711 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc48802e070 con 0x7fc498152360 2024-02-21T23:04:42.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.055+0000 7fc496000640 1 -- 172.21.15.28:0/3831188711 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc488037070 con 0x7fc498152360 2024-02-21T23:04:42.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.055+0000 7fc496000640 1 -- 172.21.15.28:0/3831188711 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc488031a00 con 0x7fc498152360 2024-02-21T23:04:42.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.055+0000 7fc49ca00640 1 -- 172.21.15.28:0/3831188711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc498152360 msgr2=0x7fc4980c5800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:42.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.055+0000 7fc49ca00640 1 --2- 172.21.15.28:0/3831188711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc498152360 0x7fc4980c5800 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fc4980a5d30 tx=0x7fc48802b880 comp rx=0 tx=0).stop 2024-02-21T23:04:42.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.056+0000 7fc49ca00640 1 -- 172.21.15.28:0/3831188711 shutdown_connections 2024-02-21T23:04:42.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.056+0000 7fc49ca00640 1 --2- 172.21.15.28:0/3831188711 >> v2:172.21.15.81:3300/0 conn(0x7fc49806b3b0 0x7fc498154e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:42.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.056+0000 7fc49ca00640 1 --2- 172.21.15.28:0/3831188711 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc4980c48a0 0x7fc49806ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:42.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.056+0000 7fc49ca00640 1 --2- 172.21.15.28:0/3831188711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc498152360 0x7fc4980c5800 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:42.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.056+0000 7fc49ca00640 1 -- 172.21.15.28:0/3831188711 >> 172.21.15.28:0/3831188711 conn(0x7fc4980a8530 msgr2=0x7fc4980a8910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:42.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.056+0000 7fc49ca00640 1 -- 172.21.15.28:0/3831188711 shutdown_connections 2024-02-21T23:04:42.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.056+0000 7fc49ca00640 1 -- 172.21.15.28:0/3831188711 wait complete. 2024-02-21T23:04:42.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.057+0000 7fc49ca00640 1 Processor -- start 2024-02-21T23:04:42.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.058+0000 7fc49ca00640 1 -- start start 2024-02-21T23:04:42.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.058+0000 7fc49ca00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc49806b3b0 0x7fc4981eaab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:42.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.058+0000 7fc49ca00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc4980c48a0 0x7fc498072e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:42.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.058+0000 7fc49ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc498152360 0x7fc498147ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:42.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.058+0000 7fc49ca00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc498152740 con 0x7fc498152360 2024-02-21T23:04:42.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.058+0000 7fc49ca00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4980c1b70 con 0x7fc4980c48a0 2024-02-21T23:04:42.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.058+0000 7fc49ca00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4981528c0 con 0x7fc49806b3b0 2024-02-21T23:04:42.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.059+0000 7fc497e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc498152360 0x7fc498147ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:42.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.059+0000 7fc497e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc498152360 0x7fc498147ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52566/0 (socket says 172.21.15.28:52566) 2024-02-21T23:04:42.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.059+0000 7fc497e00640 1 -- 172.21.15.28:0/2262310669 learned_addr learned my addr 172.21.15.28:0/2262310669 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:42.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.059+0000 7fc496a00640 1 --2- 172.21.15.28:0/2262310669 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc4980c48a0 0x7fc498072e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:42.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.059+0000 7fc497400640 1 --2- 172.21.15.28:0/2262310669 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc49806b3b0 0x7fc4981eaab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:42.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.059+0000 7fc497e00640 1 -- 172.21.15.28:0/2262310669 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc49806b3b0 msgr2=0x7fc4981eaab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:42.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.059+0000 7fc497e00640 1 --2- 172.21.15.28:0/2262310669 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc49806b3b0 0x7fc4981eaab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:42.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.059+0000 7fc497e00640 1 -- 172.21.15.28:0/2262310669 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc4980c48a0 msgr2=0x7fc498072e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:42.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.059+0000 7fc497e00640 1 --2- 172.21.15.28:0/2262310669 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc4980c48a0 0x7fc498072e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:42.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.059+0000 7fc497e00640 1 -- 172.21.15.28:0/2262310669 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4981eaff0 con 0x7fc498152360 2024-02-21T23:04:42.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.060+0000 7fc497e00640 1 --2- 172.21.15.28:0/2262310669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc498152360 0x7fc498147ae0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fc49000c7f0 tx=0x7fc49000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:42.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.060+0000 7fc48f400640 1 -- 172.21.15.28:0/2262310669 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc490015070 con 0x7fc498152360 2024-02-21T23:04:42.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.060+0000 7fc48f400640 1 -- 172.21.15.28:0/2262310669 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc490007820 con 0x7fc498152360 2024-02-21T23:04:42.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.060+0000 7fc48f400640 1 -- 172.21.15.28:0/2262310669 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc490007c50 con 0x7fc498152360 2024-02-21T23:04:42.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.060+0000 7fc49ca00640 1 -- 172.21.15.28:0/2262310669 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc49814ceb0 con 0x7fc498152360 2024-02-21T23:04:42.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.060+0000 7fc49ca00640 1 -- 172.21.15.28:0/2262310669 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4980a9d90 con 0x7fc498152360 2024-02-21T23:04:42.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.061+0000 7fc49ca00640 1 -- 172.21.15.28:0/2262310669 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc49804f290 con 0x7fc498152360 2024-02-21T23:04:42.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.063+0000 7fc48f400640 1 -- 172.21.15.28:0/2262310669 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc490018070 con 0x7fc498152360 2024-02-21T23:04:42.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.064+0000 7fc48f400640 1 --2- 172.21.15.28:0/2262310669 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc47806b4d0 0x7fc47806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:42.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.064+0000 7fc48f400640 1 -- 172.21.15.28:0/2262310669 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5421+0+0 (secure 0 0 0) 0x7fc49008a2b0 con 0x7fc498152360 2024-02-21T23:04:42.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.065+0000 7fc497400640 1 --2- 172.21.15.28:0/2262310669 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc47806b4d0 0x7fc47806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:42.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.066+0000 7fc497400640 1 --2- 172.21.15.28:0/2262310669 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc47806b4d0 0x7fc47806d990 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fc488036000 tx=0x7fc488008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:42.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.068+0000 7fc48f400640 1 -- 172.21.15.28:0/2262310669 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc4900591b0 con 0x7fc498152360 2024-02-21T23:04:42.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.280+0000 7fc49ca00640 1 -- 172.21.15.28:0/2262310669 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.c", "name": "mds_join_fs"} v 0) v1 -- 0x7fc49806e890 con 0x7fc498152360 2024-02-21T23:04:42.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.281+0000 7fc48f400640 1 -- 172.21.15.28:0/2262310669 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.c", "name": "mds_join_fs"}]=0 v13) v1 ==== 96+0+0 (secure 0 0 0) 0x7fc490058ab0 con 0x7fc498152360 2024-02-21T23:04:42.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.284+0000 7fc49ca00640 1 -- 172.21.15.28:0/2262310669 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc47806b4d0 msgr2=0x7fc47806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:42.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.284+0000 7fc49ca00640 1 --2- 172.21.15.28:0/2262310669 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc47806b4d0 0x7fc47806d990 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fc488036000 tx=0x7fc488008030 comp rx=0 tx=0).stop 2024-02-21T23:04:42.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.284+0000 7fc49ca00640 1 -- 172.21.15.28:0/2262310669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc498152360 msgr2=0x7fc498147ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:42.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.284+0000 7fc49ca00640 1 --2- 172.21.15.28:0/2262310669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc498152360 0x7fc498147ae0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fc49000c7f0 tx=0x7fc49000ccb0 comp rx=0 tx=0).stop 2024-02-21T23:04:42.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.285+0000 7fc49ca00640 1 -- 172.21.15.28:0/2262310669 shutdown_connections 2024-02-21T23:04:42.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.285+0000 7fc49ca00640 1 --2- 172.21.15.28:0/2262310669 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc47806b4d0 0x7fc47806d990 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:42.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.285+0000 7fc49ca00640 1 --2- 172.21.15.28:0/2262310669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc498152360 0x7fc498147ae0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:42.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.285+0000 7fc49ca00640 1 --2- 172.21.15.28:0/2262310669 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc4980c48a0 0x7fc498072e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:42.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.285+0000 7fc49ca00640 1 --2- 172.21.15.28:0/2262310669 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc49806b3b0 0x7fc4981eaab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:42.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.285+0000 7fc49ca00640 1 -- 172.21.15.28:0/2262310669 >> 172.21.15.28:0/2262310669 conn(0x7fc4980a8530 msgr2=0x7fc4980c3a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:42.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.285+0000 7fc49ca00640 1 -- 172.21.15.28:0/2262310669 shutdown_connections 2024-02-21T23:04:42.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.285+0000 7fc49ca00640 1 -- 172.21.15.28:0/2262310669 wait complete. 2024-02-21T23:04:42.293 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-02-21T23:04:42.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.529+0000 7f432a000640 1 Processor -- start 2024-02-21T23:04:42.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.529+0000 7f432a000640 1 -- start start 2024-02-21T23:04:42.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.530+0000 7f432a000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f43240c1080 0x7f4324159c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:42.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.530+0000 7f432a000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f432415a1a0 con 0x7f432415d170 2024-02-21T23:04:42.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.530+0000 7f432a000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43240c1590 con 0x7f4324153470 2024-02-21T23:04:42.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.530+0000 7f432a000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f432415cce0 con 0x7f43240c1080 2024-02-21T23:04:42.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.531+0000 7f4323e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f43240c1080 0x7f4324159c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:42.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.531+0000 7f4323e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f43240c1080 0x7f4324159c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52568/0 (socket says 172.21.15.28:52568) 2024-02-21T23:04:42.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.531+0000 7f4323e00640 1 -- 172.21.15.28:0/1052481833 learned_addr learned my addr 172.21.15.28:0/1052481833 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:42.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.531+0000 7f4329600640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f432415d170 0x7f432406ae70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39970/0 (socket says 172.21.15.28:39970) 2024-02-21T23:04:42.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.531+0000 7f4323e00640 1 -- 172.21.15.28:0/1052481833 >> v1:172.21.15.81:6789/0 conn(0x7f4324153470 legacy=0x7f43240c4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:42.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.531+0000 7f4323e00640 1 -- 172.21.15.28:0/1052481833 >> v1:172.21.15.28:6789/0 conn(0x7f432415d170 legacy=0x7f432406ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:42.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.532+0000 7f4323400640 1 -- 172.21.15.28:0/1052481833 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3131719333 0 0) 0x7f431c001490 con 0x7f432415d170 2024-02-21T23:04:42.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.532+0000 7f4323e00640 1 -- 172.21.15.28:0/1052481833 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43240c1870 con 0x7f43240c1080 2024-02-21T23:04:42.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.533+0000 7f4323e00640 1 --2- 172.21.15.28:0/1052481833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43240c1080 0x7f4324159c60 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f4310009870 tx=0x7f431002ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd08c0b09f251542 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:42.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.533+0000 7f4323400640 1 -- 172.21.15.28:0/1052481833 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f431002f840 con 0x7f43240c1080 2024-02-21T23:04:42.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.533+0000 7f4323400640 1 -- 172.21.15.28:0/1052481833 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f431002f9e0 con 0x7f43240c1080 2024-02-21T23:04:42.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.533+0000 7f4323400640 1 -- 172.21.15.28:0/1052481833 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f431002fdb0 con 0x7f43240c1080 2024-02-21T23:04:42.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.534+0000 7f432a000640 1 -- 172.21.15.28:0/1052481833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43240c1080 msgr2=0x7f4324159c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:42.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.534+0000 7f432a000640 1 --2- 172.21.15.28:0/1052481833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43240c1080 0x7f4324159c60 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f4310009870 tx=0x7f431002ba20 comp rx=0 tx=0).stop 2024-02-21T23:04:42.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.534+0000 7f432a000640 1 -- 172.21.15.28:0/1052481833 shutdown_connections 2024-02-21T23:04:42.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.534+0000 7f432a000640 1 --2- 172.21.15.28:0/1052481833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43240c1080 0x7f4324159c60 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:42.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.534+0000 7f432a000640 1 -- 172.21.15.28:0/1052481833 >> 172.21.15.28:0/1052481833 conn(0x7f43240a8530 msgr2=0x7f43240a8910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:42.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.534+0000 7f432a000640 1 -- 172.21.15.28:0/1052481833 shutdown_connections 2024-02-21T23:04:42.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.534+0000 7f432a000640 1 -- 172.21.15.28:0/1052481833 wait complete. 2024-02-21T23:04:42.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.536+0000 7f432a000640 1 Processor -- start 2024-02-21T23:04:42.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.536+0000 7f432a000640 1 -- start start 2024-02-21T23:04:42.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.537+0000 7f432a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43240c1080 0x7f43240710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:42.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.537+0000 7f432a000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4324153470 0x7f43241f2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:42.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.537+0000 7f432a000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f432415d170 0x7f43240a9890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:42.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.537+0000 7f432a000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f432406b580 con 0x7f43240c1080 2024-02-21T23:04:42.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.537+0000 7f432a000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f432415d600 con 0x7f4324153470 2024-02-21T23:04:42.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.537+0000 7f432a000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f432406b700 con 0x7f432415d170 2024-02-21T23:04:42.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.537+0000 7f4328c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43240c1080 0x7f43240710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:42.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.537+0000 7f4328c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43240c1080 0x7f43240710f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52576/0 (socket says 172.21.15.28:52576) 2024-02-21T23:04:42.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.537+0000 7f4328c00640 1 -- 172.21.15.28:0/1659529920 learned_addr learned my addr 172.21.15.28:0/1659529920 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:42.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.537+0000 7f4323e00640 1 --2- 172.21.15.28:0/1659529920 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4324153470 0x7f43241f2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:42.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.538+0000 7f4329600640 1 --2- 172.21.15.28:0/1659529920 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f432415d170 0x7f43240a9890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:42.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.538+0000 7f4328c00640 1 -- 172.21.15.28:0/1659529920 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f432415d170 msgr2=0x7f43240a9890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:42.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.538+0000 7f4328c00640 1 --2- 172.21.15.28:0/1659529920 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f432415d170 0x7f43240a9890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:42.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.538+0000 7f4328c00640 1 -- 172.21.15.28:0/1659529920 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4324153470 msgr2=0x7f43241f2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:42.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.538+0000 7f4328c00640 1 --2- 172.21.15.28:0/1659529920 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4324153470 0x7f43241f2c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:42.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.538+0000 7f4328c00640 1 -- 172.21.15.28:0/1659529920 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43241f31d0 con 0x7f43240c1080 2024-02-21T23:04:42.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.538+0000 7f4323e00640 1 --2- 172.21.15.28:0/1659529920 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4324153470 0x7f43241f2c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:04:42.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.538+0000 7f4329600640 1 --2- 172.21.15.28:0/1659529920 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f432415d170 0x7f43240a9890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:04:42.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.538+0000 7f4328c00640 1 --2- 172.21.15.28:0/1659529920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43240c1080 0x7f43240710f0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f431400c450 tx=0x7f431400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:42.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.538+0000 7f4320c00640 1 -- 172.21.15.28:0/1659529920 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43140052d0 con 0x7f43240c1080 2024-02-21T23:04:42.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.538+0000 7f4320c00640 1 -- 172.21.15.28:0/1659529920 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4314005470 con 0x7f43240c1080 2024-02-21T23:04:42.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.538+0000 7f4320c00640 1 -- 172.21.15.28:0/1659529920 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4314012870 con 0x7f43240c1080 2024-02-21T23:04:42.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.538+0000 7f432a000640 1 -- 172.21.15.28:0/1659529920 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f432414c700 con 0x7f43240c1080 2024-02-21T23:04:42.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.539+0000 7f432a000640 1 -- 172.21.15.28:0/1659529920 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f432414cab0 con 0x7f43240c1080 2024-02-21T23:04:42.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.540+0000 7f432a000640 1 -- 172.21.15.28:0/1659529920 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4324158b90 con 0x7f43240c1080 2024-02-21T23:04:42.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.541+0000 7f4320c00640 1 -- 172.21.15.28:0/1659529920 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4314012a10 con 0x7f43240c1080 2024-02-21T23:04:42.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.541+0000 7f4320c00640 1 --2- 172.21.15.28:0/1659529920 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f430406b4d0 0x7f430406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:42.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.542+0000 7f4320c00640 1 -- 172.21.15.28:0/1659529920 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f4314089d30 con 0x7f43240c1080 2024-02-21T23:04:42.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.542+0000 7f4323e00640 1 --2- 172.21.15.28:0/1659529920 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f430406b4d0 0x7f430406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:42.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.543+0000 7f4323e00640 1 --2- 172.21.15.28:0/1659529920 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f430406b4d0 0x7f430406d990 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f4310036000 tx=0x7f4310039040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:42.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.547+0000 7f4320c00640 1 -- 172.21.15.28:0/1659529920 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f43241f31d0 con 0x7f43240c1080 2024-02-21T23:04:42.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.757+0000 7f432a000640 1 -- 172.21.15.28:0/1659529920 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f432404f290 con 0x7f43240c1080 2024-02-21T23:04:42.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.764+0000 7f4320c00640 1 -- 172.21.15.28:0/1659529920 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v14) v1 ==== 104+0+0 (secure 0 0 0) 0x7f4314058cc0 con 0x7f43240c1080 2024-02-21T23:04:42.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.767+0000 7f432a000640 1 -- 172.21.15.28:0/1659529920 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f430406b4d0 msgr2=0x7f430406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:42.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.767+0000 7f432a000640 1 --2- 172.21.15.28:0/1659529920 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f430406b4d0 0x7f430406d990 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f4310036000 tx=0x7f4310039040 comp rx=0 tx=0).stop 2024-02-21T23:04:42.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.767+0000 7f432a000640 1 -- 172.21.15.28:0/1659529920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43240c1080 msgr2=0x7f43240710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:42.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.767+0000 7f432a000640 1 --2- 172.21.15.28:0/1659529920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43240c1080 0x7f43240710f0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f431400c450 tx=0x7f431400c910 comp rx=0 tx=0).stop 2024-02-21T23:04:42.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.767+0000 7f432a000640 1 -- 172.21.15.28:0/1659529920 shutdown_connections 2024-02-21T23:04:42.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.767+0000 7f432a000640 1 --2- 172.21.15.28:0/1659529920 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f430406b4d0 0x7f430406d990 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:42.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.767+0000 7f432a000640 1 --2- 172.21.15.28:0/1659529920 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f432415d170 0x7f43240a9890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:42.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.767+0000 7f432a000640 1 --2- 172.21.15.28:0/1659529920 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4324153470 0x7f43241f2c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:42.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.767+0000 7f432a000640 1 --2- 172.21.15.28:0/1659529920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43240c1080 0x7f43240710f0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:42.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.767+0000 7f432a000640 1 -- 172.21.15.28:0/1659529920 >> 172.21.15.28:0/1659529920 conn(0x7f43240a8530 msgr2=0x7f43241507c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:42.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.767+0000 7f432a000640 1 -- 172.21.15.28:0/1659529920 shutdown_connections 2024-02-21T23:04:42.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:42.767+0000 7f432a000640 1 -- 172.21.15.28:0/1659529920 wait complete. 2024-02-21T23:04:42.776 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_drop' 2024-02-21T23:04:42.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.007+0000 7f33c7200640 1 Processor -- start 2024-02-21T23:04:42.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.007+0000 7f33c7200640 1 -- start start 2024-02-21T23:04:42.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.008+0000 7f33c7200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f33c806dff0 0x7f33c806e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:42.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.008+0000 7f33c7200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f33c80c0fe0 0x7f33c8071350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:42.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.009+0000 7f33c7200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33c80a8b40 con 0x7f33c8158580 2024-02-21T23:04:42.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.009+0000 7f33c7200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33c80a8e00 con 0x7f33c806dff0 2024-02-21T23:04:42.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.009+0000 7f33c7200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33c80a90e0 con 0x7f33c80c0fe0 2024-02-21T23:04:42.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.009+0000 7f33c5e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f33c806dff0 0x7f33c806e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:42.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.009+0000 7f33c6800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f33c8158580 0x7f33c814b2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39978/0 (socket says 172.21.15.28:39978) 2024-02-21T23:04:42.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.009+0000 7f33c6800640 1 -- 172.21.15.28:0/802536005 learned_addr learned my addr 172.21.15.28:0/802536005 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:04:42.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.009+0000 7f33bfe00640 1 --2- 172.21.15.28:0/802536005 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f33c80c0fe0 0x7f33c8071350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:42.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.009+0000 7f33c5e00640 1 -- 172.21.15.28:0/802536005 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f33c80c0fe0 msgr2=0x7f33c8071350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:42.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.009+0000 7f33c5e00640 1 --2- 172.21.15.28:0/802536005 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f33c80c0fe0 0x7f33c8071350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:42.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.009+0000 7f33c5e00640 1 -- 172.21.15.28:0/802536005 >> v1:172.21.15.28:6789/0 conn(0x7f33c8158580 legacy=0x7f33c814b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:42.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.009+0000 7f33c5e00640 1 -- 172.21.15.28:0/802536005 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33c8072ed0 con 0x7f33c806dff0 2024-02-21T23:04:42.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.011+0000 7f33c5e00640 1 --2- 172.21.15.28:0/802536005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33c806dff0 0x7f33c806e3d0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f33b80097a0 tx=0x7f33b802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d90044f1dda992 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:43.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.011+0000 7f33c5400640 1 -- 172.21.15.28:0/802536005 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b8035020 con 0x7f33c806dff0 2024-02-21T23:04:43.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.011+0000 7f33c5400640 1 -- 172.21.15.28:0/802536005 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33b8031830 con 0x7f33c806dff0 2024-02-21T23:04:43.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.011+0000 7f33c5400640 1 -- 172.21.15.28:0/802536005 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b8031c00 con 0x7f33c806dff0 2024-02-21T23:04:43.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.011+0000 7f33c7200640 1 -- 172.21.15.28:0/802536005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33c806dff0 msgr2=0x7f33c806e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:43.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.011+0000 7f33c7200640 1 --2- 172.21.15.28:0/802536005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33c806dff0 0x7f33c806e3d0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f33b80097a0 tx=0x7f33b802b8d0 comp rx=0 tx=0).stop 2024-02-21T23:04:43.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.012+0000 7f33c7200640 1 -- 172.21.15.28:0/802536005 shutdown_connections 2024-02-21T23:04:43.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.012+0000 7f33c7200640 1 --2- 172.21.15.28:0/802536005 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f33c80c0fe0 0x7f33c8071350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:43.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.012+0000 7f33c7200640 1 --2- 172.21.15.28:0/802536005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33c806dff0 0x7f33c806e3d0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:43.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.012+0000 7f33c7200640 1 -- 172.21.15.28:0/802536005 >> 172.21.15.28:0/802536005 conn(0x7f33c806b630 msgr2=0x7f33c80c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:43.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.012+0000 7f33c7200640 1 -- 172.21.15.28:0/802536005 shutdown_connections 2024-02-21T23:04:43.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.012+0000 7f33c7200640 1 -- 172.21.15.28:0/802536005 wait complete. 2024-02-21T23:04:43.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.013+0000 7f33c7200640 1 Processor -- start 2024-02-21T23:04:43.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.013+0000 7f33c7200640 1 -- start start 2024-02-21T23:04:43.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.014+0000 7f33c7200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33c806dff0 0x7f33c814edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:43.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.014+0000 7f33c7200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f33c80c0fe0 0x7f33c81eb010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:43.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.014+0000 7f33c7200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33c8158580 0x7f33c80ab570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:43.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.014+0000 7f33c7200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33c8071890 con 0x7f33c8158580 2024-02-21T23:04:43.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.014+0000 7f33c7200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33c8073520 con 0x7f33c806dff0 2024-02-21T23:04:43.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.014+0000 7f33c7200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33c8158a10 con 0x7f33c80c0fe0 2024-02-21T23:04:43.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.015+0000 7f33c6800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33c8158580 0x7f33c80ab570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:43.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.015+0000 7f33c6800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33c8158580 0x7f33c80ab570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52602/0 (socket says 172.21.15.28:52602) 2024-02-21T23:04:43.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.015+0000 7f33c6800640 1 -- 172.21.15.28:0/113097850 learned_addr learned my addr 172.21.15.28:0/113097850 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:43.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.015+0000 7f33c5e00640 1 --2- 172.21.15.28:0/113097850 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33c806dff0 0x7f33c814edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:43.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.015+0000 7f33bfe00640 1 --2- 172.21.15.28:0/113097850 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f33c80c0fe0 0x7f33c81eb010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:43.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.015+0000 7f33c6800640 1 -- 172.21.15.28:0/113097850 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f33c80c0fe0 msgr2=0x7f33c81eb010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:43.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.015+0000 7f33c6800640 1 --2- 172.21.15.28:0/113097850 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f33c80c0fe0 0x7f33c81eb010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:43.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.015+0000 7f33c6800640 1 -- 172.21.15.28:0/113097850 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33c806dff0 msgr2=0x7f33c814edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:43.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.015+0000 7f33c6800640 1 --2- 172.21.15.28:0/113097850 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33c806dff0 0x7f33c814edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:43.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.015+0000 7f33c6800640 1 -- 172.21.15.28:0/113097850 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33c81eb550 con 0x7f33c8158580 2024-02-21T23:04:43.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.015+0000 7f33c5e00640 1 --2- 172.21.15.28:0/113097850 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33c806dff0 0x7f33c814edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:04:43.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.015+0000 7f33bfe00640 1 --2- 172.21.15.28:0/113097850 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f33c80c0fe0 0x7f33c81eb010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:04:43.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.015+0000 7f33c6800640 1 --2- 172.21.15.28:0/113097850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33c8158580 0x7f33c80ab570 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f33c000cad0 tx=0x7f33c000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:43.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.016+0000 7f33be000640 1 -- 172.21.15.28:0/113097850 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33c0017070 con 0x7f33c8158580 2024-02-21T23:04:43.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.016+0000 7f33be000640 1 -- 172.21.15.28:0/113097850 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33c000ac30 con 0x7f33c8158580 2024-02-21T23:04:43.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.016+0000 7f33be000640 1 -- 172.21.15.28:0/113097850 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33c0012620 con 0x7f33c8158580 2024-02-21T23:04:43.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.016+0000 7f33c7200640 1 -- 172.21.15.28:0/113097850 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33c81547d0 con 0x7f33c8158580 2024-02-21T23:04:43.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.016+0000 7f33c7200640 1 -- 172.21.15.28:0/113097850 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33c809a5c0 con 0x7f33c8158580 2024-02-21T23:04:43.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.017+0000 7f33c7200640 1 -- 172.21.15.28:0/113097850 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33c804f290 con 0x7f33c8158580 2024-02-21T23:04:43.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.018+0000 7f33be000640 1 -- 172.21.15.28:0/113097850 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f33c001a070 con 0x7f33c8158580 2024-02-21T23:04:43.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.019+0000 7f33be000640 1 --2- 172.21.15.28:0/113097850 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f33a406b500 0x7f33a406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:43.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.019+0000 7f33be000640 1 -- 172.21.15.28:0/113097850 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f33c0089e30 con 0x7f33c8158580 2024-02-21T23:04:43.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.020+0000 7f33c5e00640 1 --2- 172.21.15.28:0/113097850 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f33a406b500 0x7f33a406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:43.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.021+0000 7f33c5e00640 1 --2- 172.21.15.28:0/113097850 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f33a406b500 0x7f33a406d9c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f33b8036000 tx=0x7f33b8008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:43.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.023+0000 7f33be000640 1 -- 172.21.15.28:0/113097850 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f33c81eb550 con 0x7f33c8158580 2024-02-21T23:04:43.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.299+0000 7f33c7200640 1 -- 172.21.15.28:0/113097850 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_drop"]} v 0) v1 -- 0x7f33c806e7e0 con 0x7f33c8158580 2024-02-21T23:04:43.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.844+0000 7f33be000640 1 -- 172.21.15.28:0/113097850 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_drop"]}]=0 v137) v1 ==== 143+0+0 (secure 0 0 0) 0x7f33c000ae20 con 0x7f33c8158580 2024-02-21T23:04:43.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.849+0000 7f33c7200640 1 -- 172.21.15.28:0/113097850 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f33a406b500 msgr2=0x7f33a406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:43.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.849+0000 7f33c7200640 1 --2- 172.21.15.28:0/113097850 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f33a406b500 0x7f33a406d9c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f33b8036000 tx=0x7f33b8008030 comp rx=0 tx=0).stop 2024-02-21T23:04:43.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.849+0000 7f33c7200640 1 -- 172.21.15.28:0/113097850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33c8158580 msgr2=0x7f33c80ab570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:43.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.849+0000 7f33c7200640 1 --2- 172.21.15.28:0/113097850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33c8158580 0x7f33c80ab570 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f33c000cad0 tx=0x7f33c000cf90 comp rx=0 tx=0).stop 2024-02-21T23:04:43.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.849+0000 7f33c7200640 1 -- 172.21.15.28:0/113097850 shutdown_connections 2024-02-21T23:04:43.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.849+0000 7f33c7200640 1 --2- 172.21.15.28:0/113097850 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f33a406b500 0x7f33a406d9c0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:43.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.849+0000 7f33c7200640 1 --2- 172.21.15.28:0/113097850 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f33c80c0fe0 0x7f33c81eb010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:43.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.849+0000 7f33c7200640 1 --2- 172.21.15.28:0/113097850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33c8158580 0x7f33c80ab570 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:43.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.849+0000 7f33c7200640 1 --2- 172.21.15.28:0/113097850 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f33c806dff0 0x7f33c814edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:43.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.849+0000 7f33c7200640 1 -- 172.21.15.28:0/113097850 >> 172.21.15.28:0/113097850 conn(0x7f33c806b630 msgr2=0x7f33c80c1af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:43.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.849+0000 7f33c7200640 1 -- 172.21.15.28:0/113097850 shutdown_connections 2024-02-21T23:04:43.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:43.849+0000 7f33c7200640 1 -- 172.21.15.28:0/113097850 wait complete. 2024-02-21T23:04:43.860 INFO:tasks.cephfs_test_runner:That unsetting mds_join_fs will not cause failover if no high-affinity standby exists. ... ok 2024-02-21T23:04:43.860 INFO:tasks.cephfs_test_runner:Starting test: test_join_fs_last_resort (tasks.cephfs.test_failover.TestClusterAffinity) That a standby with mds_join_fs set to another fs is still used if necessary. 2024-02-21T23:04:43.860 INFO:tasks.cephfs_test_runner:test_join_fs_last_resort (tasks.cephfs.test_failover.TestClusterAffinity) 2024-02-21T23:04:43.861 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:04:44.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.090+0000 7f3f35c00640 1 Processor -- start 2024-02-21T23:04:44.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.090+0000 7f3f35c00640 1 -- start start 2024-02-21T23:04:44.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.092+0000 7f3f35c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3f30150450 0x7f3f300a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:44.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.092+0000 7f3f35c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3f30156af0 0x7f3f3006dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:44.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.092+0000 7f3f35c00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f300c2760 con 0x7f3f3006ee40 2024-02-21T23:04:44.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.092+0000 7f3f35c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f300c22c0 con 0x7f3f30156af0 2024-02-21T23:04:44.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.092+0000 7f3f35c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f30156660 con 0x7f3f30150450 2024-02-21T23:04:44.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.092+0000 7f3f35200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3f30156af0 0x7f3f3006dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:44.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.092+0000 7f3f35200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3f30156af0 0x7f3f3006dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52610/0 (socket says 172.21.15.28:52610) 2024-02-21T23:04:44.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.092+0000 7f3f35200640 1 -- 172.21.15.28:0/1275348439 learned_addr learned my addr 172.21.15.28:0/1275348439 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:44.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.092+0000 7f3f2fe00640 1 --2- 172.21.15.28:0/1275348439 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3f30150450 0x7f3f300a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:44.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.093+0000 7f3f35200640 1 -- 172.21.15.28:0/1275348439 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3f30150450 msgr2=0x7f3f300a5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:44.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.093+0000 7f3f35200640 1 --2- 172.21.15.28:0/1275348439 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3f30150450 0x7f3f300a5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:44.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.093+0000 7f3f35200640 1 -- 172.21.15.28:0/1275348439 >> v1:172.21.15.81:6789/0 conn(0x7f3f3006ee40 legacy=0x7f3f301536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:44.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.093+0000 7f3f35200640 1 -- 172.21.15.28:0/1275348439 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f30156310 con 0x7f3f30156af0 2024-02-21T23:04:44.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.094+0000 7f3f35200640 1 --2- 172.21.15.28:0/1275348439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f30156af0 0x7f3f3006dff0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f3f28009e50 tx=0x7f3f2802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bc6c7dfbfbfd215 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:44.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.094+0000 7f3f2ea00640 1 -- 172.21.15.28:0/1275348439 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f2802ca70 con 0x7f3f30156af0 2024-02-21T23:04:44.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.094+0000 7f3f2ea00640 1 -- 172.21.15.28:0/1275348439 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f2802cc10 con 0x7f3f30156af0 2024-02-21T23:04:44.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.094+0000 7f3f2ea00640 1 -- 172.21.15.28:0/1275348439 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f280343f0 con 0x7f3f30156af0 2024-02-21T23:04:44.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.095+0000 7f3f35c00640 1 -- 172.21.15.28:0/1275348439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f30156af0 msgr2=0x7f3f3006dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:44.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.095+0000 7f3f35c00640 1 --2- 172.21.15.28:0/1275348439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f30156af0 0x7f3f3006dff0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f3f28009e50 tx=0x7f3f2802bed0 comp rx=0 tx=0).stop 2024-02-21T23:04:44.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.095+0000 7f3f35c00640 1 -- 172.21.15.28:0/1275348439 shutdown_connections 2024-02-21T23:04:44.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.095+0000 7f3f35c00640 1 --2- 172.21.15.28:0/1275348439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f30156af0 0x7f3f3006dff0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:44.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.095+0000 7f3f35c00640 1 --2- 172.21.15.28:0/1275348439 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3f30150450 0x7f3f300a5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:44.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.095+0000 7f3f35c00640 1 -- 172.21.15.28:0/1275348439 >> 172.21.15.28:0/1275348439 conn(0x7f3f3006b630 msgr2=0x7f3f300c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:44.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.095+0000 7f3f35c00640 1 -- 172.21.15.28:0/1275348439 shutdown_connections 2024-02-21T23:04:44.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.095+0000 7f3f35c00640 1 -- 172.21.15.28:0/1275348439 wait complete. 2024-02-21T23:04:44.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.097+0000 7f3f35c00640 1 Processor -- start 2024-02-21T23:04:44.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.097+0000 7f3f35c00640 1 -- start start 2024-02-21T23:04:44.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.098+0000 7f3f35c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f3006ee40 0x7f3f300a96e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:44.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.098+0000 7f3f35c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f30150450 0x7f3f30152d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:44.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.098+0000 7f3f35c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3f30156af0 0x7f3f301ebd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:44.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.098+0000 7f3f35c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f30156f80 con 0x7f3f3006ee40 2024-02-21T23:04:44.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.098+0000 7f3f35c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f300c2cb0 con 0x7f3f30150450 2024-02-21T23:04:44.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.098+0000 7f3f35c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f30157100 con 0x7f3f30156af0 2024-02-21T23:04:44.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.098+0000 7f3f2fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f3006ee40 0x7f3f300a96e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:44.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.098+0000 7f3f2fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f3006ee40 0x7f3f300a96e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52614/0 (socket says 172.21.15.28:52614) 2024-02-21T23:04:44.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.098+0000 7f3f2fe00640 1 -- 172.21.15.28:0/1776930734 learned_addr learned my addr 172.21.15.28:0/1776930734 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:44.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.098+0000 7f3f2fe00640 1 -- 172.21.15.28:0/1776930734 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3f30156af0 msgr2=0x7f3f301ebd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:44.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.098+0000 7f3f2fe00640 1 --2- 172.21.15.28:0/1776930734 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3f30156af0 0x7f3f301ebd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:44.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.098+0000 7f3f2fe00640 1 -- 172.21.15.28:0/1776930734 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f30150450 msgr2=0x7f3f30152d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:44.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.098+0000 7f3f2fe00640 1 --2- 172.21.15.28:0/1776930734 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f30150450 0x7f3f30152d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:44.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.098+0000 7f3f2fe00640 1 -- 172.21.15.28:0/1776930734 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f301499f0 con 0x7f3f3006ee40 2024-02-21T23:04:44.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.099+0000 7f3f2fe00640 1 --2- 172.21.15.28:0/1776930734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f3006ee40 0x7f3f300a96e0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f3f2000d800 tx=0x7f3f2000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:44.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.099+0000 7f3f27e00640 1 -- 172.21.15.28:0/1776930734 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f2000b6c0 con 0x7f3f3006ee40 2024-02-21T23:04:44.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.099+0000 7f3f27e00640 1 -- 172.21.15.28:0/1776930734 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f20019070 con 0x7f3f3006ee40 2024-02-21T23:04:44.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.099+0000 7f3f27e00640 1 -- 172.21.15.28:0/1776930734 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f2000ba80 con 0x7f3f3006ee40 2024-02-21T23:04:44.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.099+0000 7f3f35c00640 1 -- 172.21.15.28:0/1776930734 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f30149530 con 0x7f3f3006ee40 2024-02-21T23:04:44.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.099+0000 7f3f35c00640 1 -- 172.21.15.28:0/1776930734 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f300c0880 con 0x7f3f3006ee40 2024-02-21T23:04:44.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.100+0000 7f3f35c00640 1 -- 172.21.15.28:0/1776930734 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f3004f290 con 0x7f3f3006ee40 2024-02-21T23:04:44.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.102+0000 7f3f27e00640 1 -- 172.21.15.28:0/1776930734 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3f2000bc20 con 0x7f3f3006ee40 2024-02-21T23:04:44.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.103+0000 7f3f27e00640 1 --2- 172.21.15.28:0/1776930734 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3f1006b500 0x7f3f1006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:44.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.103+0000 7f3f27e00640 1 -- 172.21.15.28:0/1776930734 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f3f2008a580 con 0x7f3f3006ee40 2024-02-21T23:04:44.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.104+0000 7f3f2f400640 1 --2- 172.21.15.28:0/1776930734 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3f1006b500 0x7f3f1006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:44.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.105+0000 7f3f2f400640 1 --2- 172.21.15.28:0/1776930734 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3f1006b500 0x7f3f1006d9c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f3f3014b300 tx=0x7f3f1c0079b0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:44.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.107+0000 7f3f27e00640 1 -- 172.21.15.28:0/1776930734 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3f20059510 con 0x7f3f3006ee40 2024-02-21T23:04:44.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.305+0000 7f3f35c00640 1 -- 172.21.15.28:0/1776930734 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3f300c1f10 con 0x7f3f3006ee40 2024-02-21T23:04:44.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.307+0000 7f3f27e00640 1 -- 172.21.15.28:0/1776930734 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) v1 ==== 74+0+13363 (secure 0 0 0) 0x7f3f20058e10 con 0x7f3f3006ee40 2024-02-21T23:04:44.296 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:04:44.296 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":44,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:04:41.813025+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6834/2827878900":"2024-02-22T23:04:38.545674+0000","172.21.15.28:6832/2827878900":"2024-02-22T23:04:38.545674+0000","172.21.15.28:6835/1213702315":"2024-02-22T23:04:20.686099+0000","172.21.15.28:6833/1213702315":"2024-02-22T23:04:20.686099+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:04:44.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.310+0000 7f3f35c00640 1 -- 172.21.15.28:0/1776930734 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3f1006b500 msgr2=0x7f3f1006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:44.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.310+0000 7f3f35c00640 1 --2- 172.21.15.28:0/1776930734 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3f1006b500 0x7f3f1006d9c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f3f3014b300 tx=0x7f3f1c0079b0 comp rx=0 tx=0).stop 2024-02-21T23:04:44.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.310+0000 7f3f35c00640 1 -- 172.21.15.28:0/1776930734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f3006ee40 msgr2=0x7f3f300a96e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:44.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.310+0000 7f3f35c00640 1 --2- 172.21.15.28:0/1776930734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f3006ee40 0x7f3f300a96e0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f3f2000d800 tx=0x7f3f2000dcc0 comp rx=0 tx=0).stop 2024-02-21T23:04:44.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.310+0000 7f3f35c00640 1 -- 172.21.15.28:0/1776930734 shutdown_connections 2024-02-21T23:04:44.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.310+0000 7f3f35c00640 1 --2- 172.21.15.28:0/1776930734 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3f1006b500 0x7f3f1006d9c0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:44.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.310+0000 7f3f35c00640 1 --2- 172.21.15.28:0/1776930734 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3f30156af0 0x7f3f301ebd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:44.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.310+0000 7f3f35c00640 1 --2- 172.21.15.28:0/1776930734 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f30150450 0x7f3f30152d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:44.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.310+0000 7f3f35c00640 1 --2- 172.21.15.28:0/1776930734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f3006ee40 0x7f3f300a96e0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:44.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.310+0000 7f3f35c00640 1 -- 172.21.15.28:0/1776930734 >> 172.21.15.28:0/1776930734 conn(0x7f3f3006b630 msgr2=0x7f3f300a70c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:44.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.310+0000 7f3f35c00640 1 -- 172.21.15.28:0/1776930734 shutdown_connections 2024-02-21T23:04:44.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.310+0000 7f3f35c00640 1 -- 172.21.15.28:0/1776930734 wait complete. 2024-02-21T23:04:44.319 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-21T23:02:30.728329+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-21T23:04:44.319 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-21T23:04:44.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.555+0000 7f3292000640 1 Processor -- start 2024-02-21T23:04:44.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.555+0000 7f3292000640 1 -- start start 2024-02-21T23:04:44.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.557+0000 7f3292000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f328c150700 0x7f328c156500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:44.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.557+0000 7f3292000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f328c152fb0 con 0x7f328c0c5f50 2024-02-21T23:04:44.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.557+0000 7f3292000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f328c153290 con 0x7f328c06b0e0 2024-02-21T23:04:44.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.557+0000 7f3292000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f328c153bc0 con 0x7f328c150700 2024-02-21T23:04:44.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.557+0000 7f328aa00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f328c0c5f50 0x7f328c14fff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39982/0 (socket says 172.21.15.28:39982) 2024-02-21T23:04:44.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.557+0000 7f328aa00640 1 -- 172.21.15.28:0/2222027132 learned_addr learned my addr 172.21.15.28:0/2222027132 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:04:44.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.558+0000 7f328a000640 1 -- 172.21.15.28:0/2222027132 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 732314780 0 0) 0x7f328c152fb0 con 0x7f328c0c5f50 2024-02-21T23:04:44.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.558+0000 7f328be00640 1 --2- 172.21.15.28:0/2222027132 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f328c150700 0x7f328c156500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:44.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.558+0000 7f328a000640 1 -- 172.21.15.28:0/2222027132 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3270003520 con 0x7f328c0c5f50 2024-02-21T23:04:44.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.558+0000 7f328a000640 1 -- 172.21.15.28:0/2222027132 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4083872696 0 0) 0x7f3270003520 con 0x7f328c0c5f50 2024-02-21T23:04:44.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.558+0000 7f328a000640 1 -- 172.21.15.28:0/2222027132 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f328c150700 msgr2=0x7f328c156500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:44.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.558+0000 7f328a000640 1 --2- 172.21.15.28:0/2222027132 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f328c150700 0x7f328c156500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:44.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.558+0000 7f328a000640 1 -- 172.21.15.28:0/2222027132 >> v1:172.21.15.81:6789/0 conn(0x7f328c06b0e0 legacy=0x7f328c0c5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:44.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.558+0000 7f328a000640 1 -- 172.21.15.28:0/2222027132 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f328c153570 con 0x7f328c0c5f50 2024-02-21T23:04:44.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.558+0000 7f328a000640 1 -- 172.21.15.28:0/2222027132 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f3274001670 con 0x7f328c0c5f50 2024-02-21T23:04:44.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.558+0000 7f328a000640 1 -- 172.21.15.28:0/2222027132 >> v1:172.21.15.28:6789/0 conn(0x7f328c0c5f50 legacy=0x7f328c14fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:44.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.559+0000 7f328a000640 1 --2- 172.21.15.28:0/2222027132 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f32700039e0 0x7f3270003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:44.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.559+0000 7f328a000640 1 --2- 172.21.15.28:0/2222027132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3270004800 0x7f3270004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:44.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.559+0000 7f328a000640 1 --2- 172.21.15.28:0/2222027132 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f328c150700 0x7f3270005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:44.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.559+0000 7f328a000640 1 -- 172.21.15.28:0/2222027132 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f328c153bc0 con 0x7f3270004800 2024-02-21T23:04:44.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.559+0000 7f328a000640 1 -- 172.21.15.28:0/2222027132 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f327000bb80 con 0x7f32700039e0 2024-02-21T23:04:44.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.559+0000 7f328a000640 1 -- 172.21.15.28:0/2222027132 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f327000bd80 con 0x7f328c150700 2024-02-21T23:04:44.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.559+0000 7f328aa00640 1 --2- 172.21.15.28:0/2222027132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3270004800 0x7f3270004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:44.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.559+0000 7f328aa00640 1 -- 172.21.15.28:0/2222027132 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f328c150700 msgr2=0x7f3270005220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:44.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.559+0000 7f328aa00640 1 --2- 172.21.15.28:0/2222027132 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f328c150700 0x7f3270005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:44.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.559+0000 7f328b400640 1 --2- 172.21.15.28:0/2222027132 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f32700039e0 0x7f3270003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:44.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.559+0000 7f328aa00640 1 -- 172.21.15.28:0/2222027132 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f32700039e0 msgr2=0x7f3270003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:44.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.559+0000 7f328aa00640 1 --2- 172.21.15.28:0/2222027132 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f32700039e0 0x7f3270003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:44.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.559+0000 7f328aa00640 1 -- 172.21.15.28:0/2222027132 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3270003520 con 0x7f3270004800 2024-02-21T23:04:44.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.559+0000 7f328b400640 1 --2- 172.21.15.28:0/2222027132 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f32700039e0 0x7f3270003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:04:44.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.560+0000 7f328aa00640 1 --2- 172.21.15.28:0/2222027132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3270004800 0x7f3270004c80 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f32740050b0 tx=0x7f3270005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a72213e3c009e2e3 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:44.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.561+0000 7f328a000640 1 -- 172.21.15.28:0/2222027132 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3274004c80 con 0x7f3270004800 2024-02-21T23:04:44.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.561+0000 7f328a000640 1 -- 172.21.15.28:0/2222027132 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3274005b90 con 0x7f3270004800 2024-02-21T23:04:44.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.561+0000 7f328a000640 1 -- 172.21.15.28:0/2222027132 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32740080d0 con 0x7f3270004800 2024-02-21T23:04:44.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.561+0000 7f3292000640 1 -- 172.21.15.28:0/2222027132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3270004800 msgr2=0x7f3270004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:44.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.561+0000 7f3292000640 1 --2- 172.21.15.28:0/2222027132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3270004800 0x7f3270004c80 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f32740050b0 tx=0x7f3270005830 comp rx=0 tx=0).stop 2024-02-21T23:04:44.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.561+0000 7f328be00640 1 -- 172.21.15.28:0/2222027132 reap_dead start 2024-02-21T23:04:44.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.561+0000 7f3292000640 1 -- 172.21.15.28:0/2222027132 shutdown_connections 2024-02-21T23:04:44.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.561+0000 7f3292000640 1 -- 172.21.15.28:0/2222027132 >> 172.21.15.28:0/2222027132 conn(0x7f328c0a85a0 msgr2=0x7f328c0a8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:44.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.562+0000 7f3292000640 1 -- 172.21.15.28:0/2222027132 shutdown_connections 2024-02-21T23:04:44.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.562+0000 7f3292000640 1 -- 172.21.15.28:0/2222027132 wait complete. 2024-02-21T23:04:44.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.563+0000 7f3292000640 1 Processor -- start 2024-02-21T23:04:44.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.563+0000 7f3292000640 1 -- start start 2024-02-21T23:04:44.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.564+0000 7f3292000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3270004800 0x7f328c0c1360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:44.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.564+0000 7f3292000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f328c0a9f20 0x7f328c0c3570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:44.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.564+0000 7f3292000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f328c0c3ab0 0x7f328c1edfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:44.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.564+0000 7f3292000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f328c06b640 con 0x7f3270004800 2024-02-21T23:04:44.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.564+0000 7f3292000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f328c06b4c0 con 0x7f328c0c3ab0 2024-02-21T23:04:44.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.564+0000 7f3292000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f328c156500 con 0x7f328c0a9f20 2024-02-21T23:04:44.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.564+0000 7f328b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3270004800 0x7f328c0c1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:44.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.564+0000 7f328b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3270004800 0x7f328c0c1360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52620/0 (socket says 172.21.15.28:52620) 2024-02-21T23:04:44.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.564+0000 7f328b400640 1 -- 172.21.15.28:0/3675908696 learned_addr learned my addr 172.21.15.28:0/3675908696 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:44.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.565+0000 7f328be00640 1 --2- 172.21.15.28:0/3675908696 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f328c0c3ab0 0x7f328c1edfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:44.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.565+0000 7f328b400640 1 -- 172.21.15.28:0/3675908696 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f328c0a9f20 msgr2=0x7f328c0c3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:44.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.565+0000 7f328aa00640 1 --2- 172.21.15.28:0/3675908696 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f328c0a9f20 0x7f328c0c3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:44.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.565+0000 7f328b400640 1 --2- 172.21.15.28:0/3675908696 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f328c0a9f20 0x7f328c0c3570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:44.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.565+0000 7f328b400640 1 -- 172.21.15.28:0/3675908696 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f328c0c3ab0 msgr2=0x7f328c1edfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:44.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.565+0000 7f328b400640 1 --2- 172.21.15.28:0/3675908696 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f328c0c3ab0 0x7f328c1edfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:44.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.565+0000 7f328b400640 1 -- 172.21.15.28:0/3675908696 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f328c0c3090 con 0x7f3270004800 2024-02-21T23:04:44.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.565+0000 7f328be00640 1 --2- 172.21.15.28:0/3675908696 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f328c0c3ab0 0x7f328c1edfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:04:44.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.565+0000 7f328b400640 1 --2- 172.21.15.28:0/3675908696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3270004800 0x7f328c0c1360 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f327c00d7a0 tx=0x7f327c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:44.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.565+0000 7f3283400640 1 -- 172.21.15.28:0/3675908696 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f327c005070 con 0x7f3270004800 2024-02-21T23:04:44.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.565+0000 7f3283400640 1 -- 172.21.15.28:0/3675908696 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f327c005210 con 0x7f3270004800 2024-02-21T23:04:44.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.565+0000 7f3283400640 1 -- 172.21.15.28:0/3675908696 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f327c007030 con 0x7f3270004800 2024-02-21T23:04:44.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.565+0000 7f3292000640 1 -- 172.21.15.28:0/3675908696 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f328c0a9940 con 0x7f3270004800 2024-02-21T23:04:44.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.566+0000 7f3292000640 1 -- 172.21.15.28:0/3675908696 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f328c1eb950 con 0x7f3270004800 2024-02-21T23:04:44.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.567+0000 7f3292000640 1 -- 172.21.15.28:0/3675908696 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f328c067f30 con 0x7f3270004800 2024-02-21T23:04:44.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.568+0000 7f3283400640 1 -- 172.21.15.28:0/3675908696 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f327c01a020 con 0x7f3270004800 2024-02-21T23:04:44.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.568+0000 7f3283400640 1 --2- 172.21.15.28:0/3675908696 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f326806b550 0x7f326806da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:44.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.569+0000 7f3283400640 1 -- 172.21.15.28:0/3675908696 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f327c089830 con 0x7f3270004800 2024-02-21T23:04:44.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.569+0000 7f328aa00640 1 --2- 172.21.15.28:0/3675908696 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f326806b550 0x7f326806da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:44.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.571+0000 7f328aa00640 1 --2- 172.21.15.28:0/3675908696 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f326806b550 0x7f326806da10 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f3274005080 tx=0x7f327400a070 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:44.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.573+0000 7f3283400640 1 -- 172.21.15.28:0/3675908696 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f327c058840 con 0x7f3270004800 2024-02-21T23:04:44.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.773+0000 7f3292000640 1 -- 172.21.15.28:0/3675908696 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f328c14a350 con 0x7f3270004800 2024-02-21T23:04:44.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.774+0000 7f3283400640 1 -- 172.21.15.28:0/3675908696 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v44) v1 ==== 93+0+10 (secure 0 0 0) 0x7f327c05e030 con 0x7f3270004800 2024-02-21T23:04:44.763 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-02-21T23:04:44.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.777+0000 7f3292000640 1 -- 172.21.15.28:0/3675908696 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f326806b550 msgr2=0x7f326806da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:44.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.777+0000 7f3292000640 1 --2- 172.21.15.28:0/3675908696 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f326806b550 0x7f326806da10 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f3274005080 tx=0x7f327400a070 comp rx=0 tx=0).stop 2024-02-21T23:04:44.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.777+0000 7f3292000640 1 -- 172.21.15.28:0/3675908696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3270004800 msgr2=0x7f328c0c1360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:44.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.777+0000 7f3292000640 1 --2- 172.21.15.28:0/3675908696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3270004800 0x7f328c0c1360 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f327c00d7a0 tx=0x7f327c00dc60 comp rx=0 tx=0).stop 2024-02-21T23:04:44.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.778+0000 7f3292000640 1 -- 172.21.15.28:0/3675908696 shutdown_connections 2024-02-21T23:04:44.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.778+0000 7f3292000640 1 --2- 172.21.15.28:0/3675908696 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f326806b550 0x7f326806da10 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:44.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.778+0000 7f3292000640 1 --2- 172.21.15.28:0/3675908696 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f328c0a9f20 0x7f328c0c3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:44.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.778+0000 7f3292000640 1 --2- 172.21.15.28:0/3675908696 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f328c0c3ab0 0x7f328c1edfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:44.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.778+0000 7f3292000640 1 --2- 172.21.15.28:0/3675908696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3270004800 0x7f328c0c1360 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:44.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.778+0000 7f3292000640 1 -- 172.21.15.28:0/3675908696 >> 172.21.15.28:0/3675908696 conn(0x7f328c0a85a0 msgr2=0x7f328c14be50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:44.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.778+0000 7f3292000640 1 -- 172.21.15.28:0/3675908696 shutdown_connections 2024-02-21T23:04:44.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:44.778+0000 7f3292000640 1 -- 172.21.15.28:0/3675908696 wait complete. 2024-02-21T23:04:44.787 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_last_resort' 2024-02-21T23:04:45.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.019+0000 7f7f27a00640 1 Processor -- start 2024-02-21T23:04:45.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.019+0000 7f7f27a00640 1 -- start start 2024-02-21T23:04:45.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.021+0000 7f7f27a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7f200c4fe0 0x7f7f2006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:45.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.021+0000 7f7f27a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7f20159400 0x7f7f2006e600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:45.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.021+0000 7f7f27a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f20159900 con 0x7f7f201569d0 2024-02-21T23:04:45.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.021+0000 7f7f27a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f20159be0 con 0x7f7f200c4fe0 2024-02-21T23:04:45.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.021+0000 7f7f27a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f20159ec0 con 0x7f7f20159400 2024-02-21T23:04:45.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.021+0000 7f7f17e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7f200c4fe0 0x7f7f2006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:45.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.021+0000 7f7f25000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f7f201569d0 0x7f7f200c3280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:57082/0 (socket says 172.21.15.28:57082) 2024-02-21T23:04:45.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.021+0000 7f7f25000640 1 -- 172.21.15.28:0/2524743119 learned_addr learned my addr 172.21.15.28:0/2524743119 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:04:45.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.022+0000 7f7f25a00640 1 --2- 172.21.15.28:0/2524743119 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7f20159400 0x7f7f2006e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:45.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.022+0000 7f7f17e00640 1 -- 172.21.15.28:0/2524743119 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7f20159400 msgr2=0x7f7f2006e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:45.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.022+0000 7f7f17e00640 1 --2- 172.21.15.28:0/2524743119 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7f20159400 0x7f7f2006e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:45.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.022+0000 7f7f17e00640 1 -- 172.21.15.28:0/2524743119 >> v1:172.21.15.28:6789/0 conn(0x7f7f201569d0 legacy=0x7f7f200c3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:45.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.022+0000 7f7f17e00640 1 -- 172.21.15.28:0/2524743119 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f2015a1a0 con 0x7f7f200c4fe0 2024-02-21T23:04:45.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.023+0000 7f7f17e00640 1 --2- 172.21.15.28:0/2524743119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f200c4fe0 0x7f7f2006e0c0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f7f080097a0 tx=0x7f7f0802b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e20cd8c56aa731f3 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:45.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.023+0000 7f7f1fe00640 1 -- 172.21.15.28:0/2524743119 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f08031780 con 0x7f7f200c4fe0 2024-02-21T23:04:45.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.024+0000 7f7f1fe00640 1 -- 172.21.15.28:0/2524743119 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f08037070 con 0x7f7f200c4fe0 2024-02-21T23:04:45.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.024+0000 7f7f1fe00640 1 -- 172.21.15.28:0/2524743119 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f08031b40 con 0x7f7f200c4fe0 2024-02-21T23:04:45.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.024+0000 7f7f27a00640 1 -- 172.21.15.28:0/2524743119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f200c4fe0 msgr2=0x7f7f2006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:45.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.024+0000 7f7f27a00640 1 --2- 172.21.15.28:0/2524743119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f200c4fe0 0x7f7f2006e0c0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f7f080097a0 tx=0x7f7f0802b820 comp rx=0 tx=0).stop 2024-02-21T23:04:45.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.024+0000 7f7f27a00640 1 -- 172.21.15.28:0/2524743119 shutdown_connections 2024-02-21T23:04:45.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.024+0000 7f7f27a00640 1 --2- 172.21.15.28:0/2524743119 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7f20159400 0x7f7f2006e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:45.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.024+0000 7f7f27a00640 1 --2- 172.21.15.28:0/2524743119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f200c4fe0 0x7f7f2006e0c0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:45.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.024+0000 7f7f27a00640 1 -- 172.21.15.28:0/2524743119 >> 172.21.15.28:0/2524743119 conn(0x7f7f200c67f0 msgr2=0x7f7f200c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:45.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.024+0000 7f7f27a00640 1 -- 172.21.15.28:0/2524743119 shutdown_connections 2024-02-21T23:04:45.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.024+0000 7f7f27a00640 1 -- 172.21.15.28:0/2524743119 wait complete. 2024-02-21T23:04:45.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.026+0000 7f7f27a00640 1 Processor -- start 2024-02-21T23:04:45.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.026+0000 7f7f27a00640 1 -- start start 2024-02-21T23:04:45.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.027+0000 7f7f27a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7f200c4fe0 0x7f7f200aa220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:45.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.027+0000 7f7f27a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f201569d0 0x7f7f2014ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:45.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.027+0000 7f7f27a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f20159400 0x7f7f20152fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:45.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.027+0000 7f7f27a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f200c3b10 con 0x7f7f20159400 2024-02-21T23:04:45.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.027+0000 7f7f27a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f200c3990 con 0x7f7f201569d0 2024-02-21T23:04:45.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.027+0000 7f7f27a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2006eb40 con 0x7f7f200c4fe0 2024-02-21T23:04:45.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.027+0000 7f7f25a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f20159400 0x7f7f20152fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:45.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.027+0000 7f7f25a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f20159400 0x7f7f20152fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36576/0 (socket says 172.21.15.28:36576) 2024-02-21T23:04:45.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.028+0000 7f7f25a00640 1 -- 172.21.15.28:0/1245604027 learned_addr learned my addr 172.21.15.28:0/1245604027 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:45.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.028+0000 7f7f25000640 1 --2- 172.21.15.28:0/1245604027 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7f200c4fe0 0x7f7f200aa220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:45.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.028+0000 7f7f25a00640 1 -- 172.21.15.28:0/1245604027 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7f200c4fe0 msgr2=0x7f7f200aa220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:45.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.028+0000 7f7f25a00640 1 --2- 172.21.15.28:0/1245604027 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7f200c4fe0 0x7f7f200aa220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:45.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.028+0000 7f7f25a00640 1 -- 172.21.15.28:0/1245604027 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f201569d0 msgr2=0x7f7f2014ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:45.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.028+0000 7f7f17e00640 1 --2- 172.21.15.28:0/1245604027 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f201569d0 0x7f7f2014ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:45.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.028+0000 7f7f25a00640 1 --2- 172.21.15.28:0/1245604027 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f201569d0 0x7f7f2014ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:45.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.028+0000 7f7f25a00640 1 -- 172.21.15.28:0/1245604027 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f20151360 con 0x7f7f20159400 2024-02-21T23:04:45.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.028+0000 7f7f17e00640 1 --2- 172.21.15.28:0/1245604027 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f201569d0 0x7f7f2014ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:04:45.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.029+0000 7f7f25a00640 1 --2- 172.21.15.28:0/1245604027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f20159400 0x7f7f20152fb0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f7f1800d800 tx=0x7f7f1800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:45.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.029+0000 7f7f1d600640 1 -- 172.21.15.28:0/1245604027 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f1800b660 con 0x7f7f20159400 2024-02-21T23:04:45.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.029+0000 7f7f1d600640 1 -- 172.21.15.28:0/1245604027 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f18019070 con 0x7f7f20159400 2024-02-21T23:04:45.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.029+0000 7f7f27a00640 1 -- 172.21.15.28:0/1245604027 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f20150da0 con 0x7f7f20159400 2024-02-21T23:04:45.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.029+0000 7f7f1d600640 1 -- 172.21.15.28:0/1245604027 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f1800ba60 con 0x7f7f20159400 2024-02-21T23:04:45.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.029+0000 7f7f27a00640 1 -- 172.21.15.28:0/1245604027 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f200aa760 con 0x7f7f20159400 2024-02-21T23:04:45.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.030+0000 7f7f27a00640 1 -- 172.21.15.28:0/1245604027 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f20067f30 con 0x7f7f20159400 2024-02-21T23:04:45.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.031+0000 7f7f1d600640 1 -- 172.21.15.28:0/1245604027 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7f1800bc00 con 0x7f7f20159400 2024-02-21T23:04:45.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.032+0000 7f7f1d600640 1 --2- 172.21.15.28:0/1245604027 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7efc06b4d0 0x7f7efc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:45.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.032+0000 7f7f1d600640 1 -- 172.21.15.28:0/1245604027 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f7f1808a510 con 0x7f7f20159400 2024-02-21T23:04:45.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.033+0000 7f7f25000640 1 --2- 172.21.15.28:0/1245604027 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7efc06b4d0 0x7f7efc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:45.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.034+0000 7f7f25000640 1 --2- 172.21.15.28:0/1245604027 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7efc06b4d0 0x7f7efc06d990 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f7f100012e0 tx=0x7f7f10009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:45.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.037+0000 7f7f1d600640 1 -- 172.21.15.28:0/1245604027 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7f180594a0 con 0x7f7f20159400 2024-02-21T23:04:45.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.310+0000 7f7f27a00640 1 -- 172.21.15.28:0/1245604027 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_last_resort"]} v 0) v1 -- 0x7f7f20158980 con 0x7f7f20159400 2024-02-21T23:04:45.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.858+0000 7f7f1d600640 1 -- 172.21.15.28:0/1245604027 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_last_resort"]}]=0 v139) v1 ==== 153+0+0 (secure 0 0 0) 0x7f7f18058da0 con 0x7f7f20159400 2024-02-21T23:04:45.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.861+0000 7f7f27a00640 1 -- 172.21.15.28:0/1245604027 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7efc06b4d0 msgr2=0x7f7efc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:45.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.861+0000 7f7f27a00640 1 --2- 172.21.15.28:0/1245604027 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7efc06b4d0 0x7f7efc06d990 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f7f100012e0 tx=0x7f7f10009040 comp rx=0 tx=0).stop 2024-02-21T23:04:45.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.861+0000 7f7f27a00640 1 -- 172.21.15.28:0/1245604027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f20159400 msgr2=0x7f7f20152fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:45.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.861+0000 7f7f27a00640 1 --2- 172.21.15.28:0/1245604027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f20159400 0x7f7f20152fb0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f7f1800d800 tx=0x7f7f1800dcc0 comp rx=0 tx=0).stop 2024-02-21T23:04:45.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.862+0000 7f7f27a00640 1 -- 172.21.15.28:0/1245604027 shutdown_connections 2024-02-21T23:04:45.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.862+0000 7f7f27a00640 1 --2- 172.21.15.28:0/1245604027 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7efc06b4d0 0x7f7efc06d990 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:45.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.862+0000 7f7f27a00640 1 --2- 172.21.15.28:0/1245604027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f20159400 0x7f7f20152fb0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:45.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.862+0000 7f7f27a00640 1 --2- 172.21.15.28:0/1245604027 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f201569d0 0x7f7f2014ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:45.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.862+0000 7f7f27a00640 1 --2- 172.21.15.28:0/1245604027 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7f200c4fe0 0x7f7f200aa220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:45.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.862+0000 7f7f27a00640 1 -- 172.21.15.28:0/1245604027 >> 172.21.15.28:0/1245604027 conn(0x7f7f200c67f0 msgr2=0x7f7f200c1320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:45.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.862+0000 7f7f27a00640 1 -- 172.21.15.28:0/1245604027 shutdown_connections 2024-02-21T23:04:45.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:45.862+0000 7f7f27a00640 1 -- 172.21.15.28:0/1245604027 wait complete. 2024-02-21T23:04:45.873 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-02-21T23:04:46.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.106+0000 7f9e01e00640 1 Processor -- start 2024-02-21T23:04:46.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.106+0000 7f9e01e00640 1 -- start start 2024-02-21T23:04:46.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.108+0000 7f9e01e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9dfc13e0c0 0x7f9dfc0c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:46.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.108+0000 7f9e01e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9dfc0c76f0 0x7f9dfc06c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:46.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.108+0000 7f9e01e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dfc158550 con 0x7f9dfc06dbc0 2024-02-21T23:04:46.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.108+0000 7f9e01e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9dfc0c89a0 con 0x7f9dfc0c76f0 2024-02-21T23:04:46.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.108+0000 7f9e01e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dfc1587d0 con 0x7f9dfc13e0c0 2024-02-21T23:04:46.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.109+0000 7f9dfbe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f9dfc06dbc0 0x7f9dfc06c280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:57092/0 (socket says 172.21.15.28:57092) 2024-02-21T23:04:46.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.109+0000 7f9dfbe00640 1 -- 172.21.15.28:0/4149437315 learned_addr learned my addr 172.21.15.28:0/4149437315 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:04:46.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.109+0000 7f9e01400640 1 --2- 172.21.15.28:0/4149437315 >> v2:172.21.15.28:3300/0 conn(0x7f9dfc0c76f0 0x7f9dfc06c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:46.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.109+0000 7f9e00a00640 1 --2- 172.21.15.28:0/4149437315 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9dfc13e0c0 0x7f9dfc0c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:46.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.109+0000 7f9e01400640 1 -- 172.21.15.28:0/4149437315 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9dfc13e0c0 msgr2=0x7f9dfc0c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:46.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.109+0000 7f9e01400640 1 --2- 172.21.15.28:0/4149437315 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9dfc13e0c0 0x7f9dfc0c71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:46.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.109+0000 7f9e01400640 1 -- 172.21.15.28:0/4149437315 >> v1:172.21.15.28:6789/0 conn(0x7f9dfc06dbc0 legacy=0x7f9dfc06c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:46.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.109+0000 7f9e01400640 1 -- 172.21.15.28:0/4149437315 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dfc158ab0 con 0x7f9dfc0c76f0 2024-02-21T23:04:46.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.109+0000 7f9dfb400640 1 -- 172.21.15.28:0/4149437315 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4028325653 0 0) 0x7f9de8001110 con 0x7f9dfc06dbc0 2024-02-21T23:04:46.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.110+0000 7f9e01400640 1 --2- 172.21.15.28:0/4149437315 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9dfc0c76f0 0x7f9dfc06c990 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f9df4009d90 tx=0x7f9df402be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e1420df7ea43f9c server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:46.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.111+0000 7f9dfb400640 1 -- 172.21.15.28:0/4149437315 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df4039070 con 0x7f9dfc0c76f0 2024-02-21T23:04:46.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.111+0000 7f9dfb400640 1 -- 172.21.15.28:0/4149437315 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9df402cbc0 con 0x7f9dfc0c76f0 2024-02-21T23:04:46.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.111+0000 7f9dfb400640 1 -- 172.21.15.28:0/4149437315 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df4034440 con 0x7f9dfc0c76f0 2024-02-21T23:04:46.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.111+0000 7f9e01e00640 1 -- 172.21.15.28:0/4149437315 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9dfc0c76f0 msgr2=0x7f9dfc06c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:46.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.111+0000 7f9e01e00640 1 --2- 172.21.15.28:0/4149437315 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9dfc0c76f0 0x7f9dfc06c990 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f9df4009d90 tx=0x7f9df402be10 comp rx=0 tx=0).stop 2024-02-21T23:04:46.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.111+0000 7f9e01e00640 1 -- 172.21.15.28:0/4149437315 shutdown_connections 2024-02-21T23:04:46.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.111+0000 7f9e01e00640 1 --2- 172.21.15.28:0/4149437315 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9dfc0c76f0 0x7f9dfc06c990 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:46.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.111+0000 7f9e01e00640 1 --2- 172.21.15.28:0/4149437315 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9dfc13e0c0 0x7f9dfc0c71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:46.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.111+0000 7f9e01e00640 1 -- 172.21.15.28:0/4149437315 >> 172.21.15.28:0/4149437315 conn(0x7f9dfc073110 msgr2=0x7f9dfc0ac490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:46.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.112+0000 7f9e01e00640 1 -- 172.21.15.28:0/4149437315 shutdown_connections 2024-02-21T23:04:46.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.112+0000 7f9e01e00640 1 -- 172.21.15.28:0/4149437315 wait complete. 2024-02-21T23:04:46.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.113+0000 7f9e01e00640 1 Processor -- start 2024-02-21T23:04:46.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.114+0000 7f9e01e00640 1 -- start start 2024-02-21T23:04:46.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.114+0000 7f9e01e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9dfc06dbc0 0x7f9dfc1ee9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:46.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.114+0000 7f9e01e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9dfc0c76f0 0x7f9dfc0ac070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:46.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.115+0000 7f9e01e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9dfc13e0c0 0x7f9dfc154490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:46.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.115+0000 7f9e01e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dfc06ced0 con 0x7f9dfc06dbc0 2024-02-21T23:04:46.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.115+0000 7f9e01e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dfc159070 con 0x7f9dfc0c76f0 2024-02-21T23:04:46.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.115+0000 7f9e01e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dfc13e520 con 0x7f9dfc13e0c0 2024-02-21T23:04:46.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.115+0000 7f9e00a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9dfc06dbc0 0x7f9dfc1ee9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:46.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.115+0000 7f9e00a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9dfc06dbc0 0x7f9dfc1ee9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36594/0 (socket says 172.21.15.28:36594) 2024-02-21T23:04:46.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.115+0000 7f9e00a00640 1 -- 172.21.15.28:0/609511702 learned_addr learned my addr 172.21.15.28:0/609511702 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:46.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.115+0000 7f9dfbe00640 1 --2- 172.21.15.28:0/609511702 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9dfc0c76f0 0x7f9dfc0ac070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:46.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.115+0000 7f9e00a00640 1 -- 172.21.15.28:0/609511702 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9dfc13e0c0 msgr2=0x7f9dfc154490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:46.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.115+0000 7f9e00a00640 1 --2- 172.21.15.28:0/609511702 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9dfc13e0c0 0x7f9dfc154490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:46.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.115+0000 7f9e00a00640 1 -- 172.21.15.28:0/609511702 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9dfc0c76f0 msgr2=0x7f9dfc0ac070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:46.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.115+0000 7f9e00a00640 1 --2- 172.21.15.28:0/609511702 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9dfc0c76f0 0x7f9dfc0ac070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:46.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.115+0000 7f9e00a00640 1 -- 172.21.15.28:0/609511702 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dfc1ef010 con 0x7f9dfc06dbc0 2024-02-21T23:04:46.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.115+0000 7f9dfbe00640 1 --2- 172.21.15.28:0/609511702 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9dfc0c76f0 0x7f9dfc0ac070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:04:46.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.116+0000 7f9e00a00640 1 --2- 172.21.15.28:0/609511702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9dfc06dbc0 0x7f9dfc1ee9e0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f9dec00d620 tx=0x7f9dec00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:46.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.116+0000 7f9df8c00640 1 -- 172.21.15.28:0/609511702 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dec00b620 con 0x7f9dfc06dbc0 2024-02-21T23:04:46.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.116+0000 7f9df8c00640 1 -- 172.21.15.28:0/609511702 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dec019070 con 0x7f9dfc06dbc0 2024-02-21T23:04:46.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.116+0000 7f9df8c00640 1 -- 172.21.15.28:0/609511702 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dec00ba00 con 0x7f9dfc06dbc0 2024-02-21T23:04:46.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.116+0000 7f9e01e00640 1 -- 172.21.15.28:0/609511702 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dfc1ef2f0 con 0x7f9dfc06dbc0 2024-02-21T23:04:46.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.116+0000 7f9e01e00640 1 -- 172.21.15.28:0/609511702 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dfc1ef8b0 con 0x7f9dfc06dbc0 2024-02-21T23:04:46.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.117+0000 7f9e01e00640 1 -- 172.21.15.28:0/609511702 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dfc04f290 con 0x7f9dfc06dbc0 2024-02-21T23:04:46.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.118+0000 7f9df8c00640 1 -- 172.21.15.28:0/609511702 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9dec00bba0 con 0x7f9dfc06dbc0 2024-02-21T23:04:46.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.119+0000 7f9df8c00640 1 --2- 172.21.15.28:0/609511702 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9ddc06b4d0 0x7f9ddc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:46.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.119+0000 7f9df8c00640 1 -- 172.21.15.28:0/609511702 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f9dec08a530 con 0x7f9dfc06dbc0 2024-02-21T23:04:46.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.120+0000 7f9dfbe00640 1 --2- 172.21.15.28:0/609511702 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9ddc06b4d0 0x7f9ddc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:46.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.121+0000 7f9dfbe00640 1 --2- 172.21.15.28:0/609511702 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9ddc06b4d0 0x7f9ddc06d990 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f9de8000f60 tx=0x7f9de8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:46.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.123+0000 7f9df8c00640 1 -- 172.21.15.28:0/609511702 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f9dec0594c0 con 0x7f9dfc06dbc0 2024-02-21T23:04:46.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.332+0000 7f9e01e00640 1 -- 172.21.15.28:0/609511702 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f9dfc06d9a0 con 0x7f9dfc06dbc0 2024-02-21T23:04:46.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.338+0000 7f9df8c00640 1 -- 172.21.15.28:0/609511702 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v15)=0 v15) v1 ==== 122+0+0 (secure 0 0 0) 0x7f9dec058dc0 con 0x7f9dfc06dbc0 2024-02-21T23:04:46.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.341+0000 7f9e01e00640 1 -- 172.21.15.28:0/609511702 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9ddc06b4d0 msgr2=0x7f9ddc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:46.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.341+0000 7f9e01e00640 1 --2- 172.21.15.28:0/609511702 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9ddc06b4d0 0x7f9ddc06d990 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f9de8000f60 tx=0x7f9de8009040 comp rx=0 tx=0).stop 2024-02-21T23:04:46.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.341+0000 7f9e01e00640 1 -- 172.21.15.28:0/609511702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9dfc06dbc0 msgr2=0x7f9dfc1ee9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:46.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.341+0000 7f9e01e00640 1 --2- 172.21.15.28:0/609511702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9dfc06dbc0 0x7f9dfc1ee9e0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f9dec00d620 tx=0x7f9dec00dae0 comp rx=0 tx=0).stop 2024-02-21T23:04:46.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.341+0000 7f9e01e00640 1 -- 172.21.15.28:0/609511702 shutdown_connections 2024-02-21T23:04:46.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.341+0000 7f9e01e00640 1 --2- 172.21.15.28:0/609511702 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9ddc06b4d0 0x7f9ddc06d990 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:46.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.341+0000 7f9e01e00640 1 --2- 172.21.15.28:0/609511702 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9dfc13e0c0 0x7f9dfc154490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:46.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.341+0000 7f9e01e00640 1 --2- 172.21.15.28:0/609511702 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9dfc0c76f0 0x7f9dfc0ac070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:46.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.341+0000 7f9e01e00640 1 --2- 172.21.15.28:0/609511702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9dfc06dbc0 0x7f9dfc1ee9e0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:46.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.341+0000 7f9e01e00640 1 -- 172.21.15.28:0/609511702 >> 172.21.15.28:0/609511702 conn(0x7f9dfc073110 msgr2=0x7f9dfc13be80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:46.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.342+0000 7f9e01e00640 1 -- 172.21.15.28:0/609511702 shutdown_connections 2024-02-21T23:04:46.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.342+0000 7f9e01e00640 1 -- 172.21.15.28:0/609511702 wait complete. 2024-02-21T23:04:46.351 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-02-21T23:04:46.351 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-21T23:04:46.463 DEBUG:teuthology.orchestra.run.smithi081:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-21T23:04:46.588 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-02-21T23:04:46.588 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-21T23:04:46.588 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-21T23:04:46.648 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054608,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-02-21T23:04:46.648 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-02-21T23:04:46.648 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-21T23:04:46.705 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053833,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498812k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249404k,nr_inodes=812351,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498812k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-02-21T23:04:46.705 INFO:tasks.cephfs.cephfs_test_case:[, ] 2024-02-21T23:04:46.705 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:04:46.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.955+0000 7f35a6000640 1 Processor -- start 2024-02-21T23:04:46.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.955+0000 7f35a6000640 1 -- start start 2024-02-21T23:04:46.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.956+0000 7f35a6000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f35a014f0b0 0x7f35a0155180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:46.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.956+0000 7f35a6000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f35a00a6160 0x7f35a00c2010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:46.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.956+0000 7f35a6000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a0155760 con 0x7f35a00c2550 2024-02-21T23:04:46.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.956+0000 7f35a6000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35a0151d30 con 0x7f35a014f0b0 2024-02-21T23:04:46.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.956+0000 7f35a6000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35a0155a40 con 0x7f35a00a6160 2024-02-21T23:04:46.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.957+0000 7f35a5600640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f35a00c2550 0x7f35a01521c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:57096/0 (socket says 172.21.15.28:57096) 2024-02-21T23:04:46.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.957+0000 7f35a5600640 1 -- 172.21.15.28:0/3606254340 learned_addr learned my addr 172.21.15.28:0/3606254340 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:04:46.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.957+0000 7f35a4c00640 1 --2- 172.21.15.28:0/3606254340 >> v2:172.21.15.28:3300/0 conn(0x7f35a014f0b0 0x7f35a0155180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:46.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.957+0000 7f3597e00640 1 --2- 172.21.15.28:0/3606254340 >> v2:172.21.15.81:3300/0 conn(0x7f35a00a6160 0x7f35a00c2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:46.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.958+0000 7f359fe00640 1 -- 172.21.15.28:0/3606254340 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 114787991 0 0) 0x7f35a0155760 con 0x7f35a00c2550 2024-02-21T23:04:46.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.958+0000 7f359fe00640 1 -- 172.21.15.28:0/3606254340 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3584005730 con 0x7f35a00c2550 2024-02-21T23:04:46.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.958+0000 7f35a4c00640 1 -- 172.21.15.28:0/3606254340 >> v2:172.21.15.81:3300/0 conn(0x7f35a00a6160 msgr2=0x7f35a00c2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:46.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.958+0000 7f35a4c00640 1 --2- 172.21.15.28:0/3606254340 >> v2:172.21.15.81:3300/0 conn(0x7f35a00a6160 0x7f35a00c2010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:46.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.958+0000 7f35a4c00640 1 -- 172.21.15.28:0/3606254340 >> v1:172.21.15.28:6789/0 conn(0x7f35a00c2550 legacy=0x7f35a01521c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:46.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.958+0000 7f35a4c00640 1 -- 172.21.15.28:0/3606254340 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35a0155d20 con 0x7f35a014f0b0 2024-02-21T23:04:46.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.959+0000 7f35a4c00640 1 --2- 172.21.15.28:0/3606254340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35a014f0b0 0x7f35a0155180 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f35a0151eb0 tx=0x7f359002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8420693fc73c3ded server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:46.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.960+0000 7f359fe00640 1 -- 172.21.15.28:0/3606254340 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3590039070 con 0x7f35a014f0b0 2024-02-21T23:04:46.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.960+0000 7f359fe00640 1 -- 172.21.15.28:0/3606254340 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f359002cc60 con 0x7f35a014f0b0 2024-02-21T23:04:46.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.960+0000 7f359fe00640 1 -- 172.21.15.28:0/3606254340 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3590034640 con 0x7f35a014f0b0 2024-02-21T23:04:46.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.960+0000 7f35a6000640 1 -- 172.21.15.28:0/3606254340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35a014f0b0 msgr2=0x7f35a0155180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:46.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.960+0000 7f35a6000640 1 --2- 172.21.15.28:0/3606254340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35a014f0b0 0x7f35a0155180 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f35a0151eb0 tx=0x7f359002beb0 comp rx=0 tx=0).stop 2024-02-21T23:04:46.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.960+0000 7f35a6000640 1 -- 172.21.15.28:0/3606254340 shutdown_connections 2024-02-21T23:04:46.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.960+0000 7f35a6000640 1 --2- 172.21.15.28:0/3606254340 >> v2:172.21.15.81:3300/0 conn(0x7f35a00a6160 0x7f35a00c2010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:46.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.960+0000 7f35a6000640 1 --2- 172.21.15.28:0/3606254340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35a014f0b0 0x7f35a0155180 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:46.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.960+0000 7f35a6000640 1 -- 172.21.15.28:0/3606254340 >> 172.21.15.28:0/3606254340 conn(0x7f35a006b630 msgr2=0x7f35a00c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:46.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.960+0000 7f35a6000640 1 -- 172.21.15.28:0/3606254340 shutdown_connections 2024-02-21T23:04:46.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.961+0000 7f35a6000640 1 -- 172.21.15.28:0/3606254340 wait complete. 2024-02-21T23:04:46.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.962+0000 7f35a6000640 1 Processor -- start 2024-02-21T23:04:46.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.962+0000 7f35a6000640 1 -- start start 2024-02-21T23:04:46.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.963+0000 7f35a6000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f35a00a6160 0x7f35a0152f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:46.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.963+0000 7f35a6000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f35a00c2550 0x7f35a00a95f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:46.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.963+0000 7f35a6000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35a014f0b0 0x7f35a0149c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:46.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.963+0000 7f35a6000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a00a7f70 con 0x7f35a014f0b0 2024-02-21T23:04:46.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.963+0000 7f35a6000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a00a68d0 con 0x7f35a00c2550 2024-02-21T23:04:46.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.963+0000 7f35a6000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35a00a80f0 con 0x7f35a00a6160 2024-02-21T23:04:46.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.963+0000 7f35a5600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35a014f0b0 0x7f35a0149c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:46.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.963+0000 7f35a5600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35a014f0b0 0x7f35a0149c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36602/0 (socket says 172.21.15.28:36602) 2024-02-21T23:04:46.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.963+0000 7f35a5600640 1 -- 172.21.15.28:0/437441674 learned_addr learned my addr 172.21.15.28:0/437441674 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:46.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.964+0000 7f35a4c00640 1 --2- 172.21.15.28:0/437441674 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f35a00a6160 0x7f35a0152f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:46.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.964+0000 7f3597e00640 1 --2- 172.21.15.28:0/437441674 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f35a00c2550 0x7f35a00a95f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:46.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.964+0000 7f35a5600640 1 -- 172.21.15.28:0/437441674 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f35a00a6160 msgr2=0x7f35a0152f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:46.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.964+0000 7f35a5600640 1 --2- 172.21.15.28:0/437441674 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f35a00a6160 0x7f35a0152f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:46.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.964+0000 7f35a5600640 1 -- 172.21.15.28:0/437441674 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f35a00c2550 msgr2=0x7f35a00a95f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:46.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.964+0000 7f35a5600640 1 --2- 172.21.15.28:0/437441674 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f35a00c2550 0x7f35a00a95f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:46.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.964+0000 7f35a5600640 1 -- 172.21.15.28:0/437441674 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35a0153760 con 0x7f35a014f0b0 2024-02-21T23:04:46.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.964+0000 7f35a4c00640 1 --2- 172.21.15.28:0/437441674 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f35a00a6160 0x7f35a0152f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:04:46.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.964+0000 7f35a5600640 1 --2- 172.21.15.28:0/437441674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35a014f0b0 0x7f35a0149c10 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f359800c430 tx=0x7f359800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:46.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.964+0000 7f359d600640 1 -- 172.21.15.28:0/437441674 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3598017070 con 0x7f35a014f0b0 2024-02-21T23:04:46.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.965+0000 7f359d600640 1 -- 172.21.15.28:0/437441674 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3598005ce0 con 0x7f35a014f0b0 2024-02-21T23:04:46.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.965+0000 7f359d600640 1 -- 172.21.15.28:0/437441674 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3598007310 con 0x7f35a014f0b0 2024-02-21T23:04:46.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.965+0000 7f35a6000640 1 -- 172.21.15.28:0/437441674 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35a01eb8e0 con 0x7f35a014f0b0 2024-02-21T23:04:46.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.965+0000 7f35a6000640 1 -- 172.21.15.28:0/437441674 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35a01eae00 con 0x7f35a014f0b0 2024-02-21T23:04:46.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.966+0000 7f35a6000640 1 -- 172.21.15.28:0/437441674 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35a0067f30 con 0x7f35a014f0b0 2024-02-21T23:04:46.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.967+0000 7f359d600640 1 -- 172.21.15.28:0/437441674 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f359801a070 con 0x7f35a014f0b0 2024-02-21T23:04:46.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.968+0000 7f359d600640 1 --2- 172.21.15.28:0/437441674 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f357c06b4d0 0x7f357c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:46.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.968+0000 7f359d600640 1 -- 172.21.15.28:0/437441674 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f3598089610 con 0x7f35a014f0b0 2024-02-21T23:04:46.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.968+0000 7f35a4c00640 1 --2- 172.21.15.28:0/437441674 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f357c06b4d0 0x7f357c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:46.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.970+0000 7f35a4c00640 1 --2- 172.21.15.28:0/437441674 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f357c06b4d0 0x7f357c06d990 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f3590036000 tx=0x7f3590030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:46.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:46.972+0000 7f359d600640 1 -- 172.21.15.28:0/437441674 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f35a0067f30 con 0x7f35a014f0b0 2024-02-21T23:04:47.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.217+0000 7f35a6000640 1 -- 172.21.15.28:0/437441674 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f35a00c0be0 con 0x7f35a014f0b0 2024-02-21T23:04:47.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.218+0000 7f359d600640 1 -- 172.21.15.28:0/437441674 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+3693 (secure 0 0 0) 0x7f3598058510 con 0x7f35a014f0b0 2024-02-21T23:04:47.207 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:04:47.207 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":26,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4978,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/3612644344","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":3612644344},{"type":"v1","addr":"172.21.15.81:6836","nonce":3612644344}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4981,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3623937361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":3623937361},{"type":"v1","addr":"172.21.15.81:6837","nonce":3623937361}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5442,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/33413357","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":33413357},{"type":"v1","addr":"172.21.15.28:6835","nonce":33413357}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":5616,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/1624848000","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1624848000},{"type":"v1","addr":"172.21.15.28:6834","nonce":1624848000}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":26}],"filesystems":[]} 2024-02-21T23:04:47.207 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 26 2024-02-21T23:04:47.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.221+0000 7f35a6000640 1 -- 172.21.15.28:0/437441674 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f357c06b4d0 msgr2=0x7f357c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:47.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.221+0000 7f35a6000640 1 --2- 172.21.15.28:0/437441674 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f357c06b4d0 0x7f357c06d990 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f3590036000 tx=0x7f3590030000 comp rx=0 tx=0).stop 2024-02-21T23:04:47.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.221+0000 7f35a6000640 1 -- 172.21.15.28:0/437441674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35a014f0b0 msgr2=0x7f35a0149c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:47.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.221+0000 7f35a6000640 1 --2- 172.21.15.28:0/437441674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35a014f0b0 0x7f35a0149c10 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f359800c430 tx=0x7f359800c8f0 comp rx=0 tx=0).stop 2024-02-21T23:04:47.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.221+0000 7f35a6000640 1 -- 172.21.15.28:0/437441674 shutdown_connections 2024-02-21T23:04:47.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.221+0000 7f35a6000640 1 --2- 172.21.15.28:0/437441674 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f357c06b4d0 0x7f357c06d990 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:47.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.221+0000 7f35a6000640 1 --2- 172.21.15.28:0/437441674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35a014f0b0 0x7f35a0149c10 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:47.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.221+0000 7f35a6000640 1 --2- 172.21.15.28:0/437441674 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f35a00c2550 0x7f35a00a95f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:47.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.221+0000 7f35a6000640 1 --2- 172.21.15.28:0/437441674 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f35a00a6160 0x7f35a0152f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:47.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.221+0000 7f35a6000640 1 -- 172.21.15.28:0/437441674 >> 172.21.15.28:0/437441674 conn(0x7f35a006b630 msgr2=0x7f35a014bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:47.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.221+0000 7f35a6000640 1 -- 172.21.15.28:0/437441674 shutdown_connections 2024-02-21T23:04:47.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.221+0000 7f35a6000640 1 -- 172.21.15.28:0/437441674 wait complete. 2024-02-21T23:04:47.231 INFO:tasks.ceph.mds.a:Restarting daemon 2024-02-21T23:04:47.231 INFO:tasks.ceph.mds.a:Stopping old one... 2024-02-21T23:04:47.231 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-02-21T23:04:47.231 INFO:teuthology.orchestra.run:waiting for 300 2024-02-21T23:04:47.231 INFO:tasks.ceph.mds.c:Restarting daemon 2024-02-21T23:04:47.231 INFO:tasks.ceph.mds.c:Stopping old one... 2024-02-21T23:04:47.232 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-02-21T23:04:47.232 INFO:teuthology.orchestra.run:waiting for 300 2024-02-21T23:04:47.232 INFO:tasks.ceph.mds.b:Restarting daemon 2024-02-21T23:04:47.232 INFO:tasks.ceph.mds.b:Stopping old one... 2024-02-21T23:04:47.232 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-02-21T23:04:47.232 INFO:teuthology.orchestra.run:waiting for 300 2024-02-21T23:04:47.232 INFO:tasks.ceph.mds.d:Restarting daemon 2024-02-21T23:04:47.233 INFO:tasks.ceph.mds.d:Stopping old one... 2024-02-21T23:04:47.233 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-02-21T23:04:47.233 INFO:teuthology.orchestra.run:waiting for 300 2024-02-21T23:04:47.249 INFO:tasks.ceph.mds.a:Stopped 2024-02-21T23:04:47.249 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-02-21T23:04:47.252 INFO:tasks.ceph.mds.c:Stopped 2024-02-21T23:04:47.252 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-02-21T23:04:47.253 INFO:tasks.ceph.mds.b:Stopped 2024-02-21T23:04:47.253 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-02-21T23:04:47.254 INFO:tasks.ceph.mds.d:Stopped 2024-02-21T23:04:47.255 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-02-21T23:04:47.299 INFO:tasks.ceph.mds.a:Started 2024-02-21T23:04:47.299 DEBUG:teuthology.parallel:result is None 2024-02-21T23:04:47.301 INFO:tasks.ceph.mds.b:Started 2024-02-21T23:04:47.302 DEBUG:teuthology.parallel:result is None 2024-02-21T23:04:47.307 INFO:tasks.ceph.mds.c:Started 2024-02-21T23:04:47.308 DEBUG:teuthology.parallel:result is None 2024-02-21T23:04:47.308 INFO:tasks.ceph.mds.d:Started 2024-02-21T23:04:47.308 DEBUG:teuthology.parallel:result is None 2024-02-21T23:04:47.309 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-02-21T23:04:47.460 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-02-21T23:04:47.461 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-02-21T23:04:47.464 INFO:tasks.ceph.mds.b.smithi081.stdout:starting mds.b at 2024-02-21T23:04:47.468 INFO:tasks.ceph.mds.d.smithi081.stdout:starting mds.d at 2024-02-21T23:04:47.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.513+0000 7f98c1a00640 1 Processor -- start 2024-02-21T23:04:47.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.513+0000 7f98c1a00640 1 -- start start 2024-02-21T23:04:47.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.514+0000 7f98c1a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f98bc06ed50 0x7f98bc0c3280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:47.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.514+0000 7f98c1a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f98bc0c1c00 0x7f98bc1587c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:47.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.514+0000 7f98c1a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f98bc0c37c0 0x7f98bc158d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:47.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.514+0000 7f98c1a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98bc0c5660 con 0x7f98bc0c37c0 2024-02-21T23:04:47.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.514+0000 7f98c1a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98bc0c5af0 con 0x7f98bc06ed50 2024-02-21T23:04:47.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.514+0000 7f98c1a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98bc0c4d30 con 0x7f98bc0c1c00 2024-02-21T23:04:47.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.515+0000 7f98bb400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f98bc0c37c0 0x7f98bc158d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:47.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.515+0000 7f98bb400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f98bc0c37c0 0x7f98bc158d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36622/0 (socket says 172.21.15.28:36622) 2024-02-21T23:04:47.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.515+0000 7f98bb400640 1 -- 172.21.15.28:0/2610374404 learned_addr learned my addr 172.21.15.28:0/2610374404 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:47.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.515+0000 7f98baa00640 1 --2- 172.21.15.28:0/2610374404 >> v2:172.21.15.81:3300/0 conn(0x7f98bc06ed50 0x7f98bc0c3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:47.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.515+0000 7f98ba000640 1 --2- 172.21.15.28:0/2610374404 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f98bc0c1c00 0x7f98bc1587c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:47.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.515+0000 7f98bb400640 1 -- 172.21.15.28:0/2610374404 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f98bc0c1c00 msgr2=0x7f98bc1587c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:47.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.515+0000 7f98bb400640 1 --2- 172.21.15.28:0/2610374404 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f98bc0c1c00 0x7f98bc1587c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:47.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.515+0000 7f98bb400640 1 -- 172.21.15.28:0/2610374404 >> v2:172.21.15.81:3300/0 conn(0x7f98bc06ed50 msgr2=0x7f98bc0c3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:47.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.515+0000 7f98bb400640 1 --2- 172.21.15.28:0/2610374404 >> v2:172.21.15.81:3300/0 conn(0x7f98bc06ed50 0x7f98bc0c3280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:47.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.515+0000 7f98bb400640 1 -- 172.21.15.28:0/2610374404 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98bc0c5010 con 0x7f98bc0c37c0 2024-02-21T23:04:47.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.515+0000 7f98baa00640 1 --2- 172.21.15.28:0/2610374404 >> v2:172.21.15.81:3300/0 conn(0x7f98bc06ed50 0x7f98bc0c3280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-21T23:04:47.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.516+0000 7f98bb400640 1 --2- 172.21.15.28:0/2610374404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98bc0c37c0 0x7f98bc158d00 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f98bc06f970 tx=0x7f98b402bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b664eb2f5d881283 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:47.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.517+0000 7f98b9600640 1 -- 172.21.15.28:0/2610374404 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b402c960 con 0x7f98bc0c37c0 2024-02-21T23:04:47.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.517+0000 7f98b9600640 1 -- 172.21.15.28:0/2610374404 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98b402cb00 con 0x7f98bc0c37c0 2024-02-21T23:04:47.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.517+0000 7f98b9600640 1 -- 172.21.15.28:0/2610374404 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b40343f0 con 0x7f98bc0c37c0 2024-02-21T23:04:47.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.517+0000 7f98c1a00640 1 -- 172.21.15.28:0/2610374404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98bc0c37c0 msgr2=0x7f98bc158d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:47.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.517+0000 7f98c1a00640 1 --2- 172.21.15.28:0/2610374404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98bc0c37c0 0x7f98bc158d00 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f98bc06f970 tx=0x7f98b402bdc0 comp rx=0 tx=0).stop 2024-02-21T23:04:47.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.517+0000 7f98c1a00640 1 -- 172.21.15.28:0/2610374404 shutdown_connections 2024-02-21T23:04:47.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.517+0000 7f98c1a00640 1 --2- 172.21.15.28:0/2610374404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98bc0c37c0 0x7f98bc158d00 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:47.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.517+0000 7f98c1a00640 1 --2- 172.21.15.28:0/2610374404 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f98bc0c1c00 0x7f98bc1587c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:47.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.517+0000 7f98c1a00640 1 --2- 172.21.15.28:0/2610374404 >> v2:172.21.15.81:3300/0 conn(0x7f98bc06ed50 0x7f98bc0c3280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:47.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.517+0000 7f98c1a00640 1 -- 172.21.15.28:0/2610374404 >> 172.21.15.28:0/2610374404 conn(0x7f98bc06b820 msgr2=0x7f98bc0c67f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:47.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.517+0000 7f98c1a00640 1 -- 172.21.15.28:0/2610374404 shutdown_connections 2024-02-21T23:04:47.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.517+0000 7f98c1a00640 1 -- 172.21.15.28:0/2610374404 wait complete. 2024-02-21T23:04:47.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.519+0000 7f98c1a00640 1 Processor -- start 2024-02-21T23:04:47.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.519+0000 7f98c1a00640 1 -- start start 2024-02-21T23:04:47.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.520+0000 7f98c1a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f98bc06ed50 0x7f98bc0ab900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:47.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.520+0000 7f98c1a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98bc0c1c00 0x7f98bc14fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:47.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.520+0000 7f98c1a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f98bc0c37c0 0x7f98bc14e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:47.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.520+0000 7f98c1a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98bc0c6070 con 0x7f98bc0c1c00 2024-02-21T23:04:47.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.520+0000 7f98c1a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98bc06f130 con 0x7f98bc0c37c0 2024-02-21T23:04:47.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.520+0000 7f98c1a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98bc159240 con 0x7f98bc06ed50 2024-02-21T23:04:47.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.521+0000 7f98ba000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98bc0c1c00 0x7f98bc14fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:47.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.521+0000 7f98ba000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98bc0c1c00 0x7f98bc14fe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36624/0 (socket says 172.21.15.28:36624) 2024-02-21T23:04:47.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.521+0000 7f98ba000640 1 -- 172.21.15.28:0/536888105 learned_addr learned my addr 172.21.15.28:0/536888105 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:47.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.521+0000 7f98bb400640 1 --2- 172.21.15.28:0/536888105 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f98bc0c37c0 0x7f98bc14e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:47.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.521+0000 7f98baa00640 1 --2- 172.21.15.28:0/536888105 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f98bc06ed50 0x7f98bc0ab900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:47.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.521+0000 7f98ba000640 1 -- 172.21.15.28:0/536888105 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f98bc06ed50 msgr2=0x7f98bc0ab900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:47.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.521+0000 7f98ba000640 1 --2- 172.21.15.28:0/536888105 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f98bc06ed50 0x7f98bc0ab900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:47.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.521+0000 7f98ba000640 1 -- 172.21.15.28:0/536888105 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f98bc0c37c0 msgr2=0x7f98bc14e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:47.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.521+0000 7f98ba000640 1 --2- 172.21.15.28:0/536888105 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f98bc0c37c0 0x7f98bc14e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:47.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.521+0000 7f98ba000640 1 -- 172.21.15.28:0/536888105 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98bc1eb020 con 0x7f98bc0c1c00 2024-02-21T23:04:47.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.521+0000 7f98bb400640 1 --2- 172.21.15.28:0/536888105 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f98bc0c37c0 0x7f98bc14e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:04:47.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.521+0000 7f98baa00640 1 --2- 172.21.15.28:0/536888105 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f98bc06ed50 0x7f98bc0ab900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:04:47.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.521+0000 7f98ba000640 1 --2- 172.21.15.28:0/536888105 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98bc0c1c00 0x7f98bc14fe30 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f98a800e870 tx=0x7f98a800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:47.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.522+0000 7f98b2a00640 1 -- 172.21.15.28:0/536888105 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98a8017070 con 0x7f98bc0c1c00 2024-02-21T23:04:47.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.522+0000 7f98b2a00640 1 -- 172.21.15.28:0/536888105 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98a800c940 con 0x7f98bc0c1c00 2024-02-21T23:04:47.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.522+0000 7f98b2a00640 1 -- 172.21.15.28:0/536888105 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98a800cd30 con 0x7f98bc0c1c00 2024-02-21T23:04:47.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.522+0000 7f98c1a00640 1 -- 172.21.15.28:0/536888105 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98bc1eb300 con 0x7f98bc0c1c00 2024-02-21T23:04:47.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.522+0000 7f98c1a00640 1 -- 172.21.15.28:0/536888105 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98bc1eb8c0 con 0x7f98bc0c1c00 2024-02-21T23:04:47.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.523+0000 7f98c1a00640 1 -- 172.21.15.28:0/536888105 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98bc04f370 con 0x7f98bc0c1c00 2024-02-21T23:04:47.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.524+0000 7f98b2a00640 1 -- 172.21.15.28:0/536888105 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f98a801a070 con 0x7f98bc0c1c00 2024-02-21T23:04:47.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.525+0000 7f98b2a00640 1 --2- 172.21.15.28:0/536888105 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f989c06b4d0 0x7f989c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:47.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.525+0000 7f98b2a00640 1 -- 172.21.15.28:0/536888105 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f98a808a3a0 con 0x7f98bc0c1c00 2024-02-21T23:04:47.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.526+0000 7f98baa00640 1 --2- 172.21.15.28:0/536888105 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f989c06b4d0 0x7f989c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:47.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.527+0000 7f98baa00640 1 --2- 172.21.15.28:0/536888105 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f989c06b4d0 0x7f989c06d990 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f98ac0099a0 tx=0x7f98ac008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:47.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.530+0000 7f98b2a00640 1 -- 172.21.15.28:0/536888105 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f98a80592a0 con 0x7f98bc0c1c00 2024-02-21T23:04:47.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:47.748+0000 7f98c1a00640 1 -- 172.21.15.28:0/536888105 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f98bc155dc0 con 0x7f98bc0c1c00 2024-02-21T23:04:48.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:48.897+0000 7f98b2a00640 1 -- 172.21.15.28:0/536888105 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v49) v1 ==== 97+0+0 (secure 0 0 0) 0x7f98a8058ba0 con 0x7f98bc0c1c00 2024-02-21T23:04:48.886 INFO:teuthology.orchestra.run.smithi028.stderr: removed all blocklist entries 2024-02-21T23:04:48.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:48.901+0000 7f98c1a00640 1 -- 172.21.15.28:0/536888105 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f989c06b4d0 msgr2=0x7f989c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:48.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:48.901+0000 7f98c1a00640 1 --2- 172.21.15.28:0/536888105 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f989c06b4d0 0x7f989c06d990 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f98ac0099a0 tx=0x7f98ac008040 comp rx=0 tx=0).stop 2024-02-21T23:04:48.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:48.901+0000 7f98c1a00640 1 -- 172.21.15.28:0/536888105 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98bc0c1c00 msgr2=0x7f98bc14fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:48.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:48.901+0000 7f98c1a00640 1 --2- 172.21.15.28:0/536888105 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98bc0c1c00 0x7f98bc14fe30 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f98a800e870 tx=0x7f98a800ed30 comp rx=0 tx=0).stop 2024-02-21T23:04:48.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:48.901+0000 7f98c1a00640 1 -- 172.21.15.28:0/536888105 shutdown_connections 2024-02-21T23:04:48.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:48.901+0000 7f98c1a00640 1 --2- 172.21.15.28:0/536888105 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f989c06b4d0 0x7f989c06d990 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:48.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:48.901+0000 7f98c1a00640 1 --2- 172.21.15.28:0/536888105 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f98bc0c37c0 0x7f98bc14e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:48.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:48.901+0000 7f98c1a00640 1 --2- 172.21.15.28:0/536888105 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98bc0c1c00 0x7f98bc14fe30 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:48.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:48.901+0000 7f98c1a00640 1 --2- 172.21.15.28:0/536888105 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f98bc06ed50 0x7f98bc0ab900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:48.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:48.901+0000 7f98c1a00640 1 -- 172.21.15.28:0/536888105 >> 172.21.15.28:0/536888105 conn(0x7f98bc06b820 msgr2=0x7f98bc155280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:48.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:48.901+0000 7f98c1a00640 1 -- 172.21.15.28:0/536888105 shutdown_connections 2024-02-21T23:04:48.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:48.901+0000 7f98c1a00640 1 -- 172.21.15.28:0/536888105 wait complete. 2024-02-21T23:04:48.911 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-02-21T23:04:49.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.137+0000 7f1822000640 1 Processor -- start 2024-02-21T23:04:49.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.137+0000 7f1822000640 1 -- start start 2024-02-21T23:04:49.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.139+0000 7f1822000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f181c0c0980 0x7f181c06dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:49.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.139+0000 7f1822000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f181c06e750 con 0x7f181c0c3490 2024-02-21T23:04:49.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.139+0000 7f1822000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f181c06ea30 con 0x7f181c0a5f40 2024-02-21T23:04:49.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.139+0000 7f1822000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f181c06eff0 con 0x7f181c0c0980 2024-02-21T23:04:49.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.139+0000 7f181b400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f181c0c3490 0x7f181c152140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:57114/0 (socket says 172.21.15.28:57114) 2024-02-21T23:04:49.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.139+0000 7f181b400640 1 -- 172.21.15.28:0/3622079349 learned_addr learned my addr 172.21.15.28:0/3622079349 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:04:49.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.139+0000 7f181be00640 1 --2- 172.21.15.28:0/3622079349 >> v2:172.21.15.28:3300/0 conn(0x7f181c0c0980 0x7f181c06dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:49.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.140+0000 7f181aa00640 1 -- 172.21.15.28:0/3622079349 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1788010189 0 0) 0x7f181c06e750 con 0x7f181c0c3490 2024-02-21T23:04:49.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.140+0000 7f181aa00640 1 -- 172.21.15.28:0/3622079349 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1800003520 con 0x7f181c0c3490 2024-02-21T23:04:49.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.140+0000 7f181be00640 1 -- 172.21.15.28:0/3622079349 >> v1:172.21.15.81:6789/0 conn(0x7f181c0a5f40 legacy=0x7f181c0c0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:49.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.140+0000 7f181be00640 1 -- 172.21.15.28:0/3622079349 >> v1:172.21.15.28:6789/0 conn(0x7f181c0c3490 legacy=0x7f181c152140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:49.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.140+0000 7f181be00640 1 -- 172.21.15.28:0/3622079349 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f181c06ed10 con 0x7f181c0c0980 2024-02-21T23:04:49.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.141+0000 7f181be00640 1 --2- 172.21.15.28:0/3622079349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f181c0c0980 0x7f181c06dfb0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f18140097a0 tx=0x7f181402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6fe00625fa14cde server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:49.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.141+0000 7f181aa00640 1 -- 172.21.15.28:0/3622079349 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1814035020 con 0x7f181c0c0980 2024-02-21T23:04:49.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.141+0000 7f181aa00640 1 -- 172.21.15.28:0/3622079349 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1814031830 con 0x7f181c0c0980 2024-02-21T23:04:49.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.141+0000 7f181aa00640 1 -- 172.21.15.28:0/3622079349 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1814031c00 con 0x7f181c0c0980 2024-02-21T23:04:49.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.142+0000 7f1822000640 1 -- 172.21.15.28:0/3622079349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f181c0c0980 msgr2=0x7f181c06dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:49.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.142+0000 7f1822000640 1 --2- 172.21.15.28:0/3622079349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f181c0c0980 0x7f181c06dfb0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f18140097a0 tx=0x7f181402b8d0 comp rx=0 tx=0).stop 2024-02-21T23:04:49.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.142+0000 7f1822000640 1 -- 172.21.15.28:0/3622079349 shutdown_connections 2024-02-21T23:04:49.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.142+0000 7f1822000640 1 --2- 172.21.15.28:0/3622079349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f181c0c0980 0x7f181c06dfb0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:49.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.142+0000 7f1822000640 1 -- 172.21.15.28:0/3622079349 >> 172.21.15.28:0/3622079349 conn(0x7f181c0c67f0 msgr2=0x7f181c0c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:49.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.142+0000 7f1822000640 1 -- 172.21.15.28:0/3622079349 shutdown_connections 2024-02-21T23:04:49.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.142+0000 7f1822000640 1 -- 172.21.15.28:0/3622079349 wait complete. 2024-02-21T23:04:49.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.144+0000 7f1822000640 1 Processor -- start 2024-02-21T23:04:49.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.144+0000 7f1822000640 1 -- start start 2024-02-21T23:04:49.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.145+0000 7f1822000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f181c0a5f40 0x7f181c071c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:49.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.145+0000 7f1822000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f181c0c0980 0x7f181c149e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:49.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.145+0000 7f1822000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f181c0c3490 0x7f181c1e7940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:49.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.145+0000 7f1822000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f181c06e4f0 con 0x7f181c0c0980 2024-02-21T23:04:49.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.145+0000 7f1822000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f181c0bfde0 con 0x7f181c0c3490 2024-02-21T23:04:49.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.145+0000 7f1822000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f181c152850 con 0x7f181c0a5f40 2024-02-21T23:04:49.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.145+0000 7f1812a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f181c0c0980 0x7f181c149e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:49.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.145+0000 7f1812a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f181c0c0980 0x7f181c149e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36640/0 (socket says 172.21.15.28:36640) 2024-02-21T23:04:49.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.145+0000 7f1812a00640 1 -- 172.21.15.28:0/1275516648 learned_addr learned my addr 172.21.15.28:0/1275516648 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:49.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.145+0000 7f181b400640 1 --2- 172.21.15.28:0/1275516648 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f181c0a5f40 0x7f181c071c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:49.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.145+0000 7f181be00640 1 --2- 172.21.15.28:0/1275516648 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f181c0c3490 0x7f181c1e7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:49.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.145+0000 7f1812a00640 1 -- 172.21.15.28:0/1275516648 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f181c0a5f40 msgr2=0x7f181c071c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:49.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.145+0000 7f1812a00640 1 --2- 172.21.15.28:0/1275516648 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f181c0a5f40 0x7f181c071c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:49.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.146+0000 7f1812a00640 1 -- 172.21.15.28:0/1275516648 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f181c0c3490 msgr2=0x7f181c1e7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:49.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.146+0000 7f1812a00640 1 --2- 172.21.15.28:0/1275516648 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f181c0c3490 0x7f181c1e7940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:49.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.146+0000 7f1812a00640 1 -- 172.21.15.28:0/1275516648 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f181c14a940 con 0x7f181c0c0980 2024-02-21T23:04:49.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.146+0000 7f181be00640 1 --2- 172.21.15.28:0/1275516648 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f181c0c3490 0x7f181c1e7940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:04:49.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.146+0000 7f1812a00640 1 --2- 172.21.15.28:0/1275516648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f181c0c0980 0x7f181c149e80 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f180400c430 tx=0x7f180400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:49.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.146+0000 7f1813e00640 1 -- 172.21.15.28:0/1275516648 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18040052d0 con 0x7f181c0c0980 2024-02-21T23:04:49.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.146+0000 7f1813e00640 1 -- 172.21.15.28:0/1275516648 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1804005490 con 0x7f181c0c0980 2024-02-21T23:04:49.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.146+0000 7f1813e00640 1 -- 172.21.15.28:0/1275516648 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1804007260 con 0x7f181c0c0980 2024-02-21T23:04:49.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.146+0000 7f1822000640 1 -- 172.21.15.28:0/1275516648 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f181c0721d0 con 0x7f181c0c0980 2024-02-21T23:04:49.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.146+0000 7f1822000640 1 -- 172.21.15.28:0/1275516648 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f181c073240 con 0x7f181c0c0980 2024-02-21T23:04:49.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.147+0000 7f1822000640 1 -- 172.21.15.28:0/1275516648 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f181c067f30 con 0x7f181c0c0980 2024-02-21T23:04:49.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.148+0000 7f1813e00640 1 -- 172.21.15.28:0/1275516648 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1804007400 con 0x7f181c0c0980 2024-02-21T23:04:49.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.149+0000 7f1813e00640 1 --2- 172.21.15.28:0/1275516648 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f17f806b500 0x7f17f806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:49.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.149+0000 7f1813e00640 1 -- 172.21.15.28:0/1275516648 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f18040895d0 con 0x7f181c0c0980 2024-02-21T23:04:49.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.150+0000 7f181b400640 1 --2- 172.21.15.28:0/1275516648 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f17f806b500 0x7f17f806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:49.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.151+0000 7f181b400640 1 --2- 172.21.15.28:0/1275516648 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f17f806b500 0x7f17f806d9c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f180c001560 tx=0x7f180c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:49.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.154+0000 7f1813e00640 1 -- 172.21.15.28:0/1275516648 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f181c14a940 con 0x7f181c0c0980 2024-02-21T23:04:49.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.425+0000 7f1822000640 1 -- 172.21.15.28:0/1275516648 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f181c06dd00 con 0x7f181c0c0980 2024-02-21T23:04:49.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.426+0000 7f1813e00640 1 -- 172.21.15.28:0/1275516648 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v11) v1 ==== 80+0+4724 (secure 0 0 0) 0x7f1804058570 con 0x7f181c0c0980 2024-02-21T23:04:49.415 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:04:49.415 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-21T23:04:49.415 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_dump": [ 2024-02-21T23:04:49.415 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:04:49.415 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.a", 2024-02-21T23:04:49.415 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDSgNZl4FY1KRAAJSK5Ibb0ibKiKXYpTLgtmQ==", 2024-02-21T23:04:49.415 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:04:49.415 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:04:49.415 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-02-21T23:04:49.416 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-02-21T23:04:49.416 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:04:49.416 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:04:49.416 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:04:49.416 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:04:49.416 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.b", 2024-02-21T23:04:49.416 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDTgNZlwRcAERAAgsYmmkuobKelBEJVCRpXUg==", 2024-02-21T23:04:49.416 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:04:49.416 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:04:49.416 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-02-21T23:04:49.417 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-02-21T23:04:49.417 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:04:49.417 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:04:49.417 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:04:49.417 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:04:49.417 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.c", 2024-02-21T23:04:49.417 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDSgNZloFpQOxAAK2WBArXRVNTgqH9WWDEPDA==", 2024-02-21T23:04:49.417 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:04:49.417 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:04:49.417 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-02-21T23:04:49.417 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-02-21T23:04:49.417 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:04:49.418 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:04:49.418 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:04:49.418 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:04:49.418 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.d", 2024-02-21T23:04:49.418 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDTgNZlblOmIRAAxe2HfSKGSkkzJDN4aVdzgQ==", 2024-02-21T23:04:49.418 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:04:49.418 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:04:49.418 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-02-21T23:04:49.418 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-02-21T23:04:49.418 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:04:49.418 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:04:49.418 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:04:49.418 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:04:49.418 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.0", 2024-02-21T23:04:49.419 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDVgNZlxNJbLxAAXot7Q3chrLEBsFNO8kT6QQ==", 2024-02-21T23:04:49.419 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:04:49.419 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:04:49.419 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:04:49.419 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:04:49.419 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:04:49.419 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:04:49.419 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:04:49.419 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.1", 2024-02-21T23:04:49.419 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDYgNZlZZfWORAAr4XkOxLYLmumVbu362N2HQ==", 2024-02-21T23:04:49.419 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:04:49.419 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:04:49.419 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:04:49.419 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:04:49.420 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:04:49.420 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:04:49.420 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:04:49.420 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.2", 2024-02-21T23:04:49.420 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDcgNZl9mcDGBAAIZGOF8IUMenwo386x95mwg==", 2024-02-21T23:04:49.420 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:04:49.420 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:04:49.420 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:04:49.420 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:04:49.420 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:04:49.420 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:04:49.420 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:04:49.420 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.3", 2024-02-21T23:04:49.420 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDfgNZl8ALSMBAAag9cmGH0/4xALWSUmrZowg==", 2024-02-21T23:04:49.421 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:04:49.421 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:04:49.421 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:04:49.421 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:04:49.421 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:04:49.421 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:04:49.421 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:04:49.421 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.4", 2024-02-21T23:04:49.421 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDngNZlBgeKABAAeW99RVfCBvB1lp7w7mQBlQ==", 2024-02-21T23:04:49.421 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:04:49.421 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:04:49.421 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:04:49.421 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:04:49.421 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:04:49.421 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:04:49.422 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:04:49.422 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.5", 2024-02-21T23:04:49.422 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDqgNZljQG3ChAAY/IZEIcvcGNvlFGIWib3gg==", 2024-02-21T23:04:49.422 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:04:49.422 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:04:49.422 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:04:49.422 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:04:49.422 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:04:49.422 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:04:49.422 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:04:49.422 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.6", 2024-02-21T23:04:49.422 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDtgNZlKnakIxAAkW5xlkshzvEHJdmpgPxR+g==", 2024-02-21T23:04:49.422 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:04:49.422 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:04:49.423 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:04:49.423 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:04:49.423 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:04:49.423 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:04:49.423 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:04:49.423 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.7", 2024-02-21T23:04:49.423 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDxgNZlTWxYARAASX1/gmqO+8mQPP8uA4xjiQ==", 2024-02-21T23:04:49.423 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:04:49.423 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:04:49.423 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:04:49.423 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:04:49.423 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:04:49.423 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:04:49.423 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:04:49.423 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.0", 2024-02-21T23:04:49.424 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDTgNZlILvYLxAAoREWxKKT7YcXnhkgk8YvoQ==", 2024-02-21T23:04:49.424 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:04:49.424 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:04:49.424 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow r", 2024-02-21T23:04:49.424 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-02-21T23:04:49.424 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:04:49.424 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:04:49.424 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:04:49.424 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.1", 2024-02-21T23:04:49.424 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDTgNZlm7ImOxAA8iRcXlzFPZPaXQfsKhuO/g==", 2024-02-21T23:04:49.424 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:04:49.424 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:04:49.424 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-02-21T23:04:49.424 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-02-21T23:04:49.425 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-02-21T23:04:49.425 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:04:49.425 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:04:49.425 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:04:49.425 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.admin", 2024-02-21T23:04:49.425 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDRgNZlBYd1GhAAbK5qBCIkf0Ej5nEyPnTxIQ==", 2024-02-21T23:04:49.425 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:04:49.425 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-02-21T23:04:49.425 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-02-21T23:04:49.425 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-02-21T23:04:49.425 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:04:49.425 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:04:49.425 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:04:49.425 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:04:49.425 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.x", 2024-02-21T23:04:49.425 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDSgNZlJYKeHBAAYPskmLRr7Jw44TdKzM/JMg==", 2024-02-21T23:04:49.426 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:04:49.426 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-02-21T23:04:49.426 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-02-21T23:04:49.426 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:04:49.426 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:04:49.426 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:04:49.426 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:04:49.426 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.y", 2024-02-21T23:04:49.426 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDSgNZlSRkOEBAAOhRj9PsZDpjoV6WT4drrZA==", 2024-02-21T23:04:49.426 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:04:49.426 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-02-21T23:04:49.426 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-02-21T23:04:49.426 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:04:49.426 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:04:49.426 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:04:49.426 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-21T23:04:49.427 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-21T23:04:49.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.429+0000 7f1822000640 1 -- 172.21.15.28:0/1275516648 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f17f806b500 msgr2=0x7f17f806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:49.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.429+0000 7f1822000640 1 --2- 172.21.15.28:0/1275516648 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f17f806b500 0x7f17f806d9c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f180c001560 tx=0x7f180c009040 comp rx=0 tx=0).stop 2024-02-21T23:04:49.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.429+0000 7f1822000640 1 -- 172.21.15.28:0/1275516648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f181c0c0980 msgr2=0x7f181c149e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:49.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.429+0000 7f1822000640 1 --2- 172.21.15.28:0/1275516648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f181c0c0980 0x7f181c149e80 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f180400c430 tx=0x7f180400c8f0 comp rx=0 tx=0).stop 2024-02-21T23:04:49.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.429+0000 7f1822000640 1 -- 172.21.15.28:0/1275516648 shutdown_connections 2024-02-21T23:04:49.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.429+0000 7f1822000640 1 --2- 172.21.15.28:0/1275516648 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f17f806b500 0x7f17f806d9c0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:49.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.429+0000 7f1822000640 1 --2- 172.21.15.28:0/1275516648 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f181c0c3490 0x7f181c1e7940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:49.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.429+0000 7f1822000640 1 --2- 172.21.15.28:0/1275516648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f181c0c0980 0x7f181c149e80 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:49.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.429+0000 7f1822000640 1 --2- 172.21.15.28:0/1275516648 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f181c0a5f40 0x7f181c071c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:49.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.429+0000 7f1822000640 1 -- 172.21.15.28:0/1275516648 >> 172.21.15.28:0/1275516648 conn(0x7f181c0c67f0 msgr2=0x7f181c0c43e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:49.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.429+0000 7f1822000640 1 -- 172.21.15.28:0/1275516648 shutdown_connections 2024-02-21T23:04:49.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.429+0000 7f1822000640 1 -- 172.21.15.28:0/1275516648 wait complete. 2024-02-21T23:04:49.437 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:04:49.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.671+0000 7fa521c00640 1 Processor -- start 2024-02-21T23:04:49.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.671+0000 7fa521c00640 1 -- start start 2024-02-21T23:04:49.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.673+0000 7fa521c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa51c1569b0 0x7fa51c159db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:49.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.673+0000 7fa521c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fa51c0c0f00 0x7fa51c06dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:49.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.673+0000 7fa521c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa51c0c1390 con 0x7fa51c06e530 2024-02-21T23:04:49.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.673+0000 7fa521c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa51c0c1fa0 con 0x7fa51c1569b0 2024-02-21T23:04:49.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.673+0000 7fa521c00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa51c0c1670 con 0x7fa51c0c0f00 2024-02-21T23:04:49.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.673+0000 7fa51be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa51c1569b0 0x7fa51c159db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:49.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.673+0000 7fa521200640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa51c06e530 0x7fa51c15c890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:57122/0 (socket says 172.21.15.28:57122) 2024-02-21T23:04:49.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.673+0000 7fa521200640 1 -- 172.21.15.28:0/2955735913 learned_addr learned my addr 172.21.15.28:0/2955735913 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:04:49.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.674+0000 7fa51b400640 1 --2- 172.21.15.28:0/2955735913 >> v2:172.21.15.81:3300/0 conn(0x7fa51c0c0f00 0x7fa51c06dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:49.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.674+0000 7fa51aa00640 1 -- 172.21.15.28:0/2955735913 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 105073459 0 0) 0x7fa51c0c1390 con 0x7fa51c06e530 2024-02-21T23:04:49.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.674+0000 7fa51aa00640 1 -- 172.21.15.28:0/2955735913 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa500003520 con 0x7fa51c06e530 2024-02-21T23:04:49.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.674+0000 7fa51be00640 1 -- 172.21.15.28:0/2955735913 >> v2:172.21.15.81:3300/0 conn(0x7fa51c0c0f00 msgr2=0x7fa51c06dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:49.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.674+0000 7fa51be00640 1 --2- 172.21.15.28:0/2955735913 >> v2:172.21.15.81:3300/0 conn(0x7fa51c0c0f00 0x7fa51c06dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:49.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.674+0000 7fa51be00640 1 -- 172.21.15.28:0/2955735913 >> v1:172.21.15.28:6789/0 conn(0x7fa51c06e530 legacy=0x7fa51c15c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:49.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.674+0000 7fa51be00640 1 -- 172.21.15.28:0/2955735913 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa51c0c1950 con 0x7fa51c1569b0 2024-02-21T23:04:49.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.674+0000 7fa51b400640 1 --2- 172.21.15.28:0/2955735913 >> v2:172.21.15.81:3300/0 conn(0x7fa51c0c0f00 0x7fa51c06dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:04:49.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.676+0000 7fa51be00640 1 --2- 172.21.15.28:0/2955735913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa51c1569b0 0x7fa51c159db0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fa51c06f3f0 tx=0x7fa50c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afa2e42a9fda7522 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:49.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.676+0000 7fa51aa00640 1 -- 172.21.15.28:0/2955735913 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa50c0315a0 con 0x7fa51c1569b0 2024-02-21T23:04:49.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.676+0000 7fa51aa00640 1 -- 172.21.15.28:0/2955735913 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa50c037070 con 0x7fa51c1569b0 2024-02-21T23:04:49.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.676+0000 7fa51aa00640 1 -- 172.21.15.28:0/2955735913 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa50c031960 con 0x7fa51c1569b0 2024-02-21T23:04:49.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.677+0000 7fa521c00640 1 -- 172.21.15.28:0/2955735913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa51c1569b0 msgr2=0x7fa51c159db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:49.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.677+0000 7fa521c00640 1 --2- 172.21.15.28:0/2955735913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa51c1569b0 0x7fa51c159db0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fa51c06f3f0 tx=0x7fa50c02b840 comp rx=0 tx=0).stop 2024-02-21T23:04:49.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.677+0000 7fa521c00640 1 -- 172.21.15.28:0/2955735913 shutdown_connections 2024-02-21T23:04:49.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.677+0000 7fa521c00640 1 --2- 172.21.15.28:0/2955735913 >> v2:172.21.15.81:3300/0 conn(0x7fa51c0c0f00 0x7fa51c06dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:49.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.677+0000 7fa521c00640 1 --2- 172.21.15.28:0/2955735913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa51c1569b0 0x7fa51c159db0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:49.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.677+0000 7fa521c00640 1 -- 172.21.15.28:0/2955735913 >> 172.21.15.28:0/2955735913 conn(0x7fa51c06b630 msgr2=0x7fa51c0c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:49.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.677+0000 7fa521c00640 1 -- 172.21.15.28:0/2955735913 shutdown_connections 2024-02-21T23:04:49.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.677+0000 7fa521c00640 1 -- 172.21.15.28:0/2955735913 wait complete. 2024-02-21T23:04:49.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.679+0000 7fa521c00640 1 Processor -- start 2024-02-21T23:04:49.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.679+0000 7fa521c00640 1 -- start start 2024-02-21T23:04:49.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.680+0000 7fa521c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa51c06e530 0x7fa51c157b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:49.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.680+0000 7fa521c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa51c1569b0 0x7fa51c152bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:49.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.680+0000 7fa521c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa51c0a49e0 0x7fa51c153100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:49.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.680+0000 7fa521c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa51c15cfa0 con 0x7fa51c1569b0 2024-02-21T23:04:49.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.680+0000 7fa521c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa51c0c2ba0 con 0x7fa51c0a49e0 2024-02-21T23:04:49.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.680+0000 7fa521c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa51c15d120 con 0x7fa51c06e530 2024-02-21T23:04:49.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.680+0000 7fa51b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa51c1569b0 0x7fa51c152bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:49.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.680+0000 7fa51b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa51c1569b0 0x7fa51c152bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36666/0 (socket says 172.21.15.28:36666) 2024-02-21T23:04:49.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.680+0000 7fa51b400640 1 -- 172.21.15.28:0/3958616406 learned_addr learned my addr 172.21.15.28:0/3958616406 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:49.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.680+0000 7fa51be00640 1 --2- 172.21.15.28:0/3958616406 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa51c06e530 0x7fa51c157b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:49.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.680+0000 7fa51b400640 1 -- 172.21.15.28:0/3958616406 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa51c06e530 msgr2=0x7fa51c157b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:49.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.680+0000 7fa51b400640 1 --2- 172.21.15.28:0/3958616406 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa51c06e530 0x7fa51c157b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:49.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.680+0000 7fa51b400640 1 -- 172.21.15.28:0/3958616406 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa51c0a49e0 msgr2=0x7fa51c153100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:49.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.680+0000 7fa51b400640 1 --2- 172.21.15.28:0/3958616406 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa51c0a49e0 0x7fa51c153100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:49.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.680+0000 7fa51b400640 1 -- 172.21.15.28:0/3958616406 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa51c0a5360 con 0x7fa51c1569b0 2024-02-21T23:04:49.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.681+0000 7fa51b400640 1 --2- 172.21.15.28:0/3958616406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa51c1569b0 0x7fa51c152bc0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fa50800e810 tx=0x7fa50800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:49.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.681+0000 7fa513e00640 1 -- 172.21.15.28:0/3958616406 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa50800f040 con 0x7fa51c1569b0 2024-02-21T23:04:49.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.681+0000 7fa513e00640 1 -- 172.21.15.28:0/3958616406 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa508004030 con 0x7fa51c1569b0 2024-02-21T23:04:49.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.681+0000 7fa513e00640 1 -- 172.21.15.28:0/3958616406 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa508004400 con 0x7fa51c1569b0 2024-02-21T23:04:49.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.681+0000 7fa521c00640 1 -- 172.21.15.28:0/3958616406 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa51c153640 con 0x7fa51c1569b0 2024-02-21T23:04:49.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.681+0000 7fa521c00640 1 -- 172.21.15.28:0/3958616406 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa51c158070 con 0x7fa51c1569b0 2024-02-21T23:04:49.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.682+0000 7fa521c00640 1 -- 172.21.15.28:0/3958616406 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa51c04f290 con 0x7fa51c1569b0 2024-02-21T23:04:49.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.683+0000 7fa513e00640 1 -- 172.21.15.28:0/3958616406 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa50800c6d0 con 0x7fa51c1569b0 2024-02-21T23:04:49.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.684+0000 7fa513e00640 1 --2- 172.21.15.28:0/3958616406 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa4fc06b4d0 0x7fa4fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:49.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.684+0000 7fa513e00640 1 -- 172.21.15.28:0/3958616406 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fa508089d90 con 0x7fa51c1569b0 2024-02-21T23:04:49.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.685+0000 7fa51be00640 1 --2- 172.21.15.28:0/3958616406 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa4fc06b4d0 0x7fa4fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:49.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.686+0000 7fa51be00640 1 --2- 172.21.15.28:0/3958616406 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa4fc06b4d0 0x7fa4fc06d990 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fa50c036000 tx=0x7fa50c008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:49.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.689+0000 7fa513e00640 1 -- 172.21.15.28:0/3958616406 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fa508017020 con 0x7fa51c1569b0 2024-02-21T23:04:49.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.890+0000 7fa521c00640 1 -- 172.21.15.28:0/3958616406 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa51c073540 con 0x7fa51c1569b0 2024-02-21T23:04:49.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.893+0000 7fa513e00640 1 -- 172.21.15.28:0/3958616406 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v49) v1 ==== 74+0+13104 (secure 0 0 0) 0x7fa508058e60 con 0x7fa51c1569b0 2024-02-21T23:04:49.881 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:04:49.882 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":49,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:04:48.893831+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:04:49.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.895+0000 7fa521c00640 1 -- 172.21.15.28:0/3958616406 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa4fc06b4d0 msgr2=0x7fa4fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:49.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.895+0000 7fa521c00640 1 --2- 172.21.15.28:0/3958616406 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa4fc06b4d0 0x7fa4fc06d990 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fa50c036000 tx=0x7fa50c008030 comp rx=0 tx=0).stop 2024-02-21T23:04:49.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.895+0000 7fa521c00640 1 -- 172.21.15.28:0/3958616406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa51c1569b0 msgr2=0x7fa51c152bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:49.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.895+0000 7fa521c00640 1 --2- 172.21.15.28:0/3958616406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa51c1569b0 0x7fa51c152bc0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fa50800e810 tx=0x7fa50800ecd0 comp rx=0 tx=0).stop 2024-02-21T23:04:49.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.895+0000 7fa521c00640 1 -- 172.21.15.28:0/3958616406 shutdown_connections 2024-02-21T23:04:49.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.895+0000 7fa521c00640 1 --2- 172.21.15.28:0/3958616406 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa4fc06b4d0 0x7fa4fc06d990 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:49.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.895+0000 7fa521c00640 1 --2- 172.21.15.28:0/3958616406 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa51c0a49e0 0x7fa51c153100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:49.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.895+0000 7fa521c00640 1 --2- 172.21.15.28:0/3958616406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa51c1569b0 0x7fa51c152bc0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:49.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.895+0000 7fa521c00640 1 --2- 172.21.15.28:0/3958616406 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa51c06e530 0x7fa51c157b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:49.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.895+0000 7fa521c00640 1 -- 172.21.15.28:0/3958616406 >> 172.21.15.28:0/3958616406 conn(0x7fa51c06b630 msgr2=0x7fa51c0c64e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:49.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.896+0000 7fa521c00640 1 -- 172.21.15.28:0/3958616406 shutdown_connections 2024-02-21T23:04:49.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:49.896+0000 7fa521c00640 1 -- 172.21.15.28:0/3958616406 wait complete. 2024-02-21T23:04:49.905 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-21T23:02:30.728329+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-21T23:04:49.905 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-21T23:04:50.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.146+0000 7f5eaa000640 1 Processor -- start 2024-02-21T23:04:50.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.147+0000 7f5eaa000640 1 -- start start 2024-02-21T23:04:50.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.149+0000 7f5eaa000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5ea413e0c0 0x7f5ea40c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:50.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.149+0000 7f5eaa000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ea4158550 con 0x7f5ea406dbc0 2024-02-21T23:04:50.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.149+0000 7f5eaa000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ea41587d0 con 0x7f5ea40c76f0 2024-02-21T23:04:50.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.149+0000 7f5eaa000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea40c89a0 con 0x7f5ea413e0c0 2024-02-21T23:04:50.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.149+0000 7f5ea3e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f5ea406dbc0 0x7f5ea406c280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:57128/0 (socket says 172.21.15.28:57128) 2024-02-21T23:04:50.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.149+0000 7f5ea3e00640 1 -- 172.21.15.28:0/901141070 learned_addr learned my addr 172.21.15.28:0/901141070 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:04:50.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.149+0000 7f5ea8c00640 1 --2- 172.21.15.28:0/901141070 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5ea413e0c0 0x7f5ea40c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:50.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.150+0000 7f5ea3400640 1 -- 172.21.15.28:0/901141070 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4151813986 0 0) 0x7f5e90001110 con 0x7f5ea406dbc0 2024-02-21T23:04:50.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.150+0000 7f5ea3400640 1 -- 172.21.15.28:0/901141070 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e88003520 con 0x7f5ea406dbc0 2024-02-21T23:04:50.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.150+0000 7f5ea3400640 1 -- 172.21.15.28:0/901141070 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3521171053 0 0) 0x7f5e88003520 con 0x7f5ea406dbc0 2024-02-21T23:04:50.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.150+0000 7f5ea3400640 1 -- 172.21.15.28:0/901141070 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5ea413e0c0 msgr2=0x7f5ea40c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:50.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.150+0000 7f5ea3400640 1 --2- 172.21.15.28:0/901141070 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5ea413e0c0 0x7f5ea40c71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:50.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.150+0000 7f5ea3400640 1 -- 172.21.15.28:0/901141070 >> v1:172.21.15.81:6789/0 conn(0x7f5ea40c76f0 legacy=0x7f5ea406c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:50.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.150+0000 7f5ea3400640 1 -- 172.21.15.28:0/901141070 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ea4158ab0 con 0x7f5ea406dbc0 2024-02-21T23:04:50.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.150+0000 7f5ea3400640 1 -- 172.21.15.28:0/901141070 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f5e90001840 con 0x7f5ea406dbc0 2024-02-21T23:04:50.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.150+0000 7f5ea3400640 1 -- 172.21.15.28:0/901141070 >> v1:172.21.15.28:6789/0 conn(0x7f5ea406dbc0 legacy=0x7f5ea406c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:50.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.150+0000 7f5ea3400640 1 --2- 172.21.15.28:0/901141070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e880039e0 0x7f5e88003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:50.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.150+0000 7f5ea3400640 1 --2- 172.21.15.28:0/901141070 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5ea413e0c0 0x7f5e88004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:50.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.150+0000 7f5ea3400640 1 --2- 172.21.15.28:0/901141070 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e88004d40 0x7f5e880051a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:50.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.150+0000 7f5ea3400640 1 -- 172.21.15.28:0/901141070 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e8800bba0 con 0x7f5e880039e0 2024-02-21T23:04:50.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.150+0000 7f5ea3400640 1 -- 172.21.15.28:0/901141070 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e8800bd50 con 0x7f5e88004d40 2024-02-21T23:04:50.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.150+0000 7f5ea3400640 1 -- 172.21.15.28:0/901141070 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e8800bf50 con 0x7f5ea413e0c0 2024-02-21T23:04:50.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.151+0000 7f5ea8c00640 1 --2- 172.21.15.28:0/901141070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e880039e0 0x7f5e88003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:50.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.151+0000 7f5ea8c00640 1 -- 172.21.15.28:0/901141070 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5ea413e0c0 msgr2=0x7f5e88004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:50.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.151+0000 7f5ea8c00640 1 --2- 172.21.15.28:0/901141070 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5ea413e0c0 0x7f5e88004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:50.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.151+0000 7f5ea8c00640 1 -- 172.21.15.28:0/901141070 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e88004d40 msgr2=0x7f5e880051a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:50.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.151+0000 7f5ea8c00640 1 --2- 172.21.15.28:0/901141070 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e88004d40 0x7f5e880051a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:50.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.151+0000 7f5ea8c00640 1 -- 172.21.15.28:0/901141070 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e88003520 con 0x7f5e880039e0 2024-02-21T23:04:50.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.152+0000 7f5ea8c00640 1 --2- 172.21.15.28:0/901141070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e880039e0 0x7f5e88003e00 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f5e94009ac0 tx=0x7f5e88005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10ac55e701c65c87 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:50.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.152+0000 7f5ea3400640 1 -- 172.21.15.28:0/901141070 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e94039070 con 0x7f5e880039e0 2024-02-21T23:04:50.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.152+0000 7f5ea3400640 1 -- 172.21.15.28:0/901141070 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e9402c710 con 0x7f5e880039e0 2024-02-21T23:04:50.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.152+0000 7f5ea3400640 1 -- 172.21.15.28:0/901141070 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e94009690 con 0x7f5e880039e0 2024-02-21T23:04:50.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.153+0000 7f5eaa000640 1 -- 172.21.15.28:0/901141070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e880039e0 msgr2=0x7f5e88003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:50.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.153+0000 7f5eaa000640 1 --2- 172.21.15.28:0/901141070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e880039e0 0x7f5e88003e00 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f5e94009ac0 tx=0x7f5e88005830 comp rx=0 tx=0).stop 2024-02-21T23:04:50.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.153+0000 7f5ea9600640 1 -- 172.21.15.28:0/901141070 reap_dead start 2024-02-21T23:04:50.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.153+0000 7f5eaa000640 1 -- 172.21.15.28:0/901141070 shutdown_connections 2024-02-21T23:04:50.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.153+0000 7f5eaa000640 1 -- 172.21.15.28:0/901141070 >> 172.21.15.28:0/901141070 conn(0x7f5ea4073110 msgr2=0x7f5ea40ac490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:50.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.153+0000 7f5eaa000640 1 -- 172.21.15.28:0/901141070 shutdown_connections 2024-02-21T23:04:50.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.153+0000 7f5eaa000640 1 -- 172.21.15.28:0/901141070 wait complete. 2024-02-21T23:04:50.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.155+0000 7f5eaa000640 1 Processor -- start 2024-02-21T23:04:50.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.155+0000 7f5eaa000640 1 -- start start 2024-02-21T23:04:50.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.156+0000 7f5eaa000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5e880039e0 0x7f5ea4137380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:50.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.156+0000 7f5eaa000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ea4151f80 0x7f5ea41eb520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:50.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.156+0000 7f5eaa000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5ea40af4a0 0x7f5ea40c9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:50.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.156+0000 7f5eaa000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea413e520 con 0x7f5ea4151f80 2024-02-21T23:04:50.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.156+0000 7f5eaa000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea4159070 con 0x7f5ea40af4a0 2024-02-21T23:04:50.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.156+0000 7f5eaa000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea413e6a0 con 0x7f5e880039e0 2024-02-21T23:04:50.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.156+0000 7f5ea3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ea4151f80 0x7f5ea41eb520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:50.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.156+0000 7f5ea3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ea4151f80 0x7f5ea41eb520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36678/0 (socket says 172.21.15.28:36678) 2024-02-21T23:04:50.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.156+0000 7f5ea3e00640 1 -- 172.21.15.28:0/4261624979 learned_addr learned my addr 172.21.15.28:0/4261624979 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:50.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.156+0000 7f5ea8c00640 1 --2- 172.21.15.28:0/4261624979 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5e880039e0 0x7f5ea4137380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:50.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.157+0000 7f5ea9600640 1 --2- 172.21.15.28:0/4261624979 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5ea40af4a0 0x7f5ea40c9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:50.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.157+0000 7f5ea3e00640 1 -- 172.21.15.28:0/4261624979 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5e880039e0 msgr2=0x7f5ea4137380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:50.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.157+0000 7f5ea3e00640 1 --2- 172.21.15.28:0/4261624979 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5e880039e0 0x7f5ea4137380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:50.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.157+0000 7f5ea3e00640 1 -- 172.21.15.28:0/4261624979 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5ea40af4a0 msgr2=0x7f5ea40c9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:50.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.157+0000 7f5ea3e00640 1 --2- 172.21.15.28:0/4261624979 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5ea40af4a0 0x7f5ea40c9ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:50.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.157+0000 7f5ea3e00640 1 -- 172.21.15.28:0/4261624979 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ea41378c0 con 0x7f5ea4151f80 2024-02-21T23:04:50.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.157+0000 7f5ea3e00640 1 --2- 172.21.15.28:0/4261624979 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ea4151f80 0x7f5ea41eb520 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f5e9000c700 tx=0x7f5e9000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:50.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.157+0000 7f5ea0c00640 1 -- 172.21.15.28:0/4261624979 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e90018070 con 0x7f5ea4151f80 2024-02-21T23:04:50.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.157+0000 7f5ea0c00640 1 -- 172.21.15.28:0/4261624979 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e900072f0 con 0x7f5ea4151f80 2024-02-21T23:04:50.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.157+0000 7f5ea0c00640 1 -- 172.21.15.28:0/4261624979 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e900047e0 con 0x7f5ea4151f80 2024-02-21T23:04:50.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.157+0000 7f5eaa000640 1 -- 172.21.15.28:0/4261624979 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ea41eba60 con 0x7f5ea4151f80 2024-02-21T23:04:50.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.158+0000 7f5eaa000640 1 -- 172.21.15.28:0/4261624979 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ea41ead20 con 0x7f5ea4151f80 2024-02-21T23:04:50.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.159+0000 7f5eaa000640 1 -- 172.21.15.28:0/4261624979 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ea40cb610 con 0x7f5ea4151f80 2024-02-21T23:04:50.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.160+0000 7f5ea0c00640 1 -- 172.21.15.28:0/4261624979 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5e90004980 con 0x7f5ea4151f80 2024-02-21T23:04:50.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.161+0000 7f5ea0c00640 1 --2- 172.21.15.28:0/4261624979 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5e8406b500 0x7f5e8406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:50.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.161+0000 7f5ea0c00640 1 -- 172.21.15.28:0/4261624979 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f5e90089430 con 0x7f5ea4151f80 2024-02-21T23:04:50.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.161+0000 7f5ea8c00640 1 --2- 172.21.15.28:0/4261624979 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5e8406b500 0x7f5e8406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:50.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.163+0000 7f5ea8c00640 1 --2- 172.21.15.28:0/4261624979 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5e8406b500 0x7f5e8406d9c0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f5e94036000 tx=0x7f5e94030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:50.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.165+0000 7f5ea0c00640 1 -- 172.21.15.28:0/4261624979 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f5ea40cb610 con 0x7f5ea4151f80 2024-02-21T23:04:50.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.365+0000 7f5eaa000640 1 -- 172.21.15.28:0/4261624979 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5ea41eab10 con 0x7f5ea4151f80 2024-02-21T23:04:50.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.366+0000 7f5ea0c00640 1 -- 172.21.15.28:0/4261624979 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v49) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5ea4158550 con 0x7f5ea4151f80 2024-02-21T23:04:50.355 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-02-21T23:04:50.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.369+0000 7f5eaa000640 1 -- 172.21.15.28:0/4261624979 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5e8406b500 msgr2=0x7f5e8406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:50.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.369+0000 7f5eaa000640 1 --2- 172.21.15.28:0/4261624979 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5e8406b500 0x7f5e8406d9c0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f5e94036000 tx=0x7f5e94030000 comp rx=0 tx=0).stop 2024-02-21T23:04:50.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.369+0000 7f5eaa000640 1 -- 172.21.15.28:0/4261624979 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ea4151f80 msgr2=0x7f5ea41eb520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:50.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.369+0000 7f5eaa000640 1 --2- 172.21.15.28:0/4261624979 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ea4151f80 0x7f5ea41eb520 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f5e9000c700 tx=0x7f5e9000cbc0 comp rx=0 tx=0).stop 2024-02-21T23:04:50.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.369+0000 7f5eaa000640 1 -- 172.21.15.28:0/4261624979 shutdown_connections 2024-02-21T23:04:50.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.369+0000 7f5eaa000640 1 --2- 172.21.15.28:0/4261624979 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5e8406b500 0x7f5e8406d9c0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:50.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.369+0000 7f5eaa000640 1 --2- 172.21.15.28:0/4261624979 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5ea40af4a0 0x7f5ea40c9ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:50.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.369+0000 7f5eaa000640 1 --2- 172.21.15.28:0/4261624979 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ea4151f80 0x7f5ea41eb520 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:50.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.369+0000 7f5eaa000640 1 --2- 172.21.15.28:0/4261624979 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5e880039e0 0x7f5ea4137380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:50.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.369+0000 7f5eaa000640 1 -- 172.21.15.28:0/4261624979 >> 172.21.15.28:0/4261624979 conn(0x7f5ea4073110 msgr2=0x7f5ea4138e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:50.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.369+0000 7f5eaa000640 1 -- 172.21.15.28:0/4261624979 shutdown_connections 2024-02-21T23:04:50.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.369+0000 7f5eaa000640 1 -- 172.21.15.28:0/4261624979 wait complete. 2024-02-21T23:04:50.378 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-02-21T23:04:50.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.612+0000 7ff0a9c00640 1 Processor -- start 2024-02-21T23:04:50.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.613+0000 7ff0a9c00640 1 -- start start 2024-02-21T23:04:50.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.614+0000 7ff0a9c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff0a4152ab0 0x7ff0a40c7fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:50.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.614+0000 7ff0a9c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7ff0a40c6710 0x7ff0a40c5210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:50.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.614+0000 7ff0a9c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff0a40c5750 0x7ff0a4158850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:50.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.614+0000 7ff0a9c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0a40c3ce0 con 0x7ff0a40c5750 2024-02-21T23:04:50.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.614+0000 7ff0a9c00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0a40a8180 con 0x7ff0a40c6710 2024-02-21T23:04:50.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.614+0000 7ff0a9c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0a40c33b0 con 0x7ff0a4152ab0 2024-02-21T23:04:50.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.614+0000 7ff0a3400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff0a40c5750 0x7ff0a4158850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:50.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.615+0000 7ff0a3400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff0a40c5750 0x7ff0a4158850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36680/0 (socket says 172.21.15.28:36680) 2024-02-21T23:04:50.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.615+0000 7ff0a3400640 1 -- 172.21.15.28:0/578829388 learned_addr learned my addr 172.21.15.28:0/578829388 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:50.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.615+0000 7ff0a2a00640 1 --2- 172.21.15.28:0/578829388 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff0a4152ab0 0x7ff0a40c7fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:50.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.615+0000 7ff0a2000640 1 --2- 172.21.15.28:0/578829388 >> v2:172.21.15.81:3300/0 conn(0x7ff0a40c6710 0x7ff0a40c5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:50.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.615+0000 7ff0a3400640 1 -- 172.21.15.28:0/578829388 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff0a4152ab0 msgr2=0x7ff0a40c7fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:50.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.615+0000 7ff0a3400640 1 --2- 172.21.15.28:0/578829388 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff0a4152ab0 0x7ff0a40c7fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:50.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.615+0000 7ff0a3400640 1 -- 172.21.15.28:0/578829388 >> v2:172.21.15.81:3300/0 conn(0x7ff0a40c6710 msgr2=0x7ff0a40c5210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:50.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.615+0000 7ff0a3400640 1 --2- 172.21.15.28:0/578829388 >> v2:172.21.15.81:3300/0 conn(0x7ff0a40c6710 0x7ff0a40c5210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:50.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.615+0000 7ff0a3400640 1 -- 172.21.15.28:0/578829388 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0a40c3690 con 0x7ff0a40c5750 2024-02-21T23:04:50.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.615+0000 7ff0a2a00640 1 --2- 172.21.15.28:0/578829388 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff0a4152ab0 0x7ff0a40c7fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:04:50.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.616+0000 7ff0a3400640 1 --2- 172.21.15.28:0/578829388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0a40c5750 0x7ff0a4158850 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7ff0a40c63d0 tx=0x7ff09c02b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2431aa74e0958b99 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:50.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.616+0000 7ff0a1600640 1 -- 172.21.15.28:0/578829388 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff09c035020 con 0x7ff0a40c5750 2024-02-21T23:04:50.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.617+0000 7ff0a1600640 1 -- 172.21.15.28:0/578829388 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff09c02d040 con 0x7ff0a40c5750 2024-02-21T23:04:50.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.617+0000 7ff0a1600640 1 -- 172.21.15.28:0/578829388 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff09c031a80 con 0x7ff0a40c5750 2024-02-21T23:04:50.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.617+0000 7ff0a9c00640 1 -- 172.21.15.28:0/578829388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0a40c5750 msgr2=0x7ff0a4158850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:50.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.617+0000 7ff0a9c00640 1 --2- 172.21.15.28:0/578829388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0a40c5750 0x7ff0a4158850 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7ff0a40c63d0 tx=0x7ff09c02b900 comp rx=0 tx=0).stop 2024-02-21T23:04:50.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.617+0000 7ff0a9c00640 1 -- 172.21.15.28:0/578829388 shutdown_connections 2024-02-21T23:04:50.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.617+0000 7ff0a9c00640 1 --2- 172.21.15.28:0/578829388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0a40c5750 0x7ff0a4158850 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:50.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.617+0000 7ff0a9c00640 1 --2- 172.21.15.28:0/578829388 >> v2:172.21.15.81:3300/0 conn(0x7ff0a40c6710 0x7ff0a40c5210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:50.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.617+0000 7ff0a9c00640 1 --2- 172.21.15.28:0/578829388 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff0a4152ab0 0x7ff0a40c7fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:50.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.617+0000 7ff0a9c00640 1 -- 172.21.15.28:0/578829388 >> 172.21.15.28:0/578829388 conn(0x7ff0a40c8810 msgr2=0x7ff0a40c8bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:50.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.617+0000 7ff0a9c00640 1 -- 172.21.15.28:0/578829388 shutdown_connections 2024-02-21T23:04:50.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.617+0000 7ff0a9c00640 1 -- 172.21.15.28:0/578829388 wait complete. 2024-02-21T23:04:50.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.619+0000 7ff0a9c00640 1 Processor -- start 2024-02-21T23:04:50.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.619+0000 7ff0a9c00640 1 -- start start 2024-02-21T23:04:50.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.620+0000 7ff0a9c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0a40c5750 0x7ff0a41eb4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:50.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.620+0000 7ff0a9c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff0a40c6710 0x7ff0a41ee990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:50.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.620+0000 7ff0a9c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0a4152ab0 0x7ff0a4150db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:50.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.620+0000 7ff0a9c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0a4158f10 con 0x7ff0a4152ab0 2024-02-21T23:04:50.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.620+0000 7ff0a9c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0a4158d90 con 0x7ff0a40c5750 2024-02-21T23:04:50.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.620+0000 7ff0a9c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0a4159090 con 0x7ff0a40c6710 2024-02-21T23:04:50.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.620+0000 7ff0a3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0a4152ab0 0x7ff0a4150db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:50.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.620+0000 7ff0a3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0a4152ab0 0x7ff0a4150db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36692/0 (socket says 172.21.15.28:36692) 2024-02-21T23:04:50.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.620+0000 7ff0a3400640 1 -- 172.21.15.28:0/3612200580 learned_addr learned my addr 172.21.15.28:0/3612200580 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:50.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.621+0000 7ff0a2a00640 1 --2- 172.21.15.28:0/3612200580 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0a40c5750 0x7ff0a41eb4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:50.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.621+0000 7ff0a2000640 1 --2- 172.21.15.28:0/3612200580 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff0a40c6710 0x7ff0a41ee990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:50.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.621+0000 7ff0a3400640 1 -- 172.21.15.28:0/3612200580 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff0a40c6710 msgr2=0x7ff0a41ee990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:50.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.621+0000 7ff0a3400640 1 --2- 172.21.15.28:0/3612200580 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff0a40c6710 0x7ff0a41ee990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:50.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.621+0000 7ff0a3400640 1 -- 172.21.15.28:0/3612200580 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0a40c5750 msgr2=0x7ff0a41eb4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:50.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.621+0000 7ff0a3400640 1 --2- 172.21.15.28:0/3612200580 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0a40c5750 0x7ff0a41eb4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:50.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.621+0000 7ff0a3400640 1 -- 172.21.15.28:0/3612200580 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0a41eeed0 con 0x7ff0a4152ab0 2024-02-21T23:04:50.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.621+0000 7ff0a3400640 1 --2- 172.21.15.28:0/3612200580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0a4152ab0 0x7ff0a4150db0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7ff09c02b940 tx=0x7ff09c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:50.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.622+0000 7ff09aa00640 1 -- 172.21.15.28:0/3612200580 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff09c040070 con 0x7ff0a4152ab0 2024-02-21T23:04:50.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.622+0000 7ff09aa00640 1 -- 172.21.15.28:0/3612200580 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff09c039c10 con 0x7ff0a4152ab0 2024-02-21T23:04:50.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.622+0000 7ff09aa00640 1 -- 172.21.15.28:0/3612200580 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff09c035040 con 0x7ff0a4152ab0 2024-02-21T23:04:50.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.622+0000 7ff0a9c00640 1 -- 172.21.15.28:0/3612200580 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0a41eaa40 con 0x7ff0a4152ab0 2024-02-21T23:04:50.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.622+0000 7ff0a9c00640 1 -- 172.21.15.28:0/3612200580 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0a40712a0 con 0x7ff0a4152ab0 2024-02-21T23:04:50.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.623+0000 7ff0a9c00640 1 -- 172.21.15.28:0/3612200580 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0a4067f30 con 0x7ff0a4152ab0 2024-02-21T23:04:50.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.624+0000 7ff09aa00640 1 -- 172.21.15.28:0/3612200580 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff09c038020 con 0x7ff0a4152ab0 2024-02-21T23:04:50.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.625+0000 7ff09aa00640 1 --2- 172.21.15.28:0/3612200580 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff08006b4d0 0x7ff08006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:50.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.625+0000 7ff09aa00640 1 -- 172.21.15.28:0/3612200580 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5249+0+0 (secure 0 0 0) 0x7ff09c0abdb0 con 0x7ff0a4152ab0 2024-02-21T23:04:50.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.625+0000 7ff0a2a00640 1 --2- 172.21.15.28:0/3612200580 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff08006b4d0 0x7ff08006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:50.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.627+0000 7ff0a2a00640 1 --2- 172.21.15.28:0/3612200580 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff08006b4d0 0x7ff08006d990 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff094007510 tx=0x7ff0940074a0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:50.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.629+0000 7ff09aa00640 1 -- 172.21.15.28:0/3612200580 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ff0a41eaa40 con 0x7ff0a4152ab0 2024-02-21T23:04:50.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.832+0000 7ff0a9c00640 1 -- 172.21.15.28:0/3612200580 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7ff0a40c6190 con 0x7ff0a4152ab0 2024-02-21T23:04:50.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.833+0000 7ff09aa00640 1 -- 172.21.15.28:0/3612200580 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v49) v1 ==== 84+0+65 (secure 0 0 0) 0x7ff09c07ae80 con 0x7ff0a4152ab0 2024-02-21T23:04:50.821 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:04:50.821 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-02-21T23:04:50.821 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:04:50.822 INFO:teuthology.orchestra.run.smithi028.stdout: "poolnum": 1, 2024-02-21T23:04:50.822 INFO:teuthology.orchestra.run.smithi028.stdout: "poolname": ".mgr" 2024-02-21T23:04:50.822 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:04:50.822 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-02-21T23:04:50.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.835+0000 7ff0a9c00640 1 -- 172.21.15.28:0/3612200580 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff08006b4d0 msgr2=0x7ff08006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:50.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.835+0000 7ff0a9c00640 1 --2- 172.21.15.28:0/3612200580 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff08006b4d0 0x7ff08006d990 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff094007510 tx=0x7ff0940074a0 comp rx=0 tx=0).stop 2024-02-21T23:04:50.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.835+0000 7ff0a9c00640 1 -- 172.21.15.28:0/3612200580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0a4152ab0 msgr2=0x7ff0a4150db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:50.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.835+0000 7ff0a9c00640 1 --2- 172.21.15.28:0/3612200580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0a4152ab0 0x7ff0a4150db0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7ff09c02b940 tx=0x7ff09c0046c0 comp rx=0 tx=0).stop 2024-02-21T23:04:50.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.836+0000 7ff0a9c00640 1 -- 172.21.15.28:0/3612200580 shutdown_connections 2024-02-21T23:04:50.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.836+0000 7ff0a9c00640 1 --2- 172.21.15.28:0/3612200580 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff08006b4d0 0x7ff08006d990 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:50.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.836+0000 7ff0a9c00640 1 --2- 172.21.15.28:0/3612200580 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff0a40c6710 0x7ff0a41ee990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:50.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.836+0000 7ff0a9c00640 1 --2- 172.21.15.28:0/3612200580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0a4152ab0 0x7ff0a4150db0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:50.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.836+0000 7ff0a9c00640 1 --2- 172.21.15.28:0/3612200580 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0a40c5750 0x7ff0a41eb4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:50.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.836+0000 7ff0a9c00640 1 -- 172.21.15.28:0/3612200580 >> 172.21.15.28:0/3612200580 conn(0x7ff0a40c8810 msgr2=0x7ff0a406c930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:50.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.836+0000 7ff0a9c00640 1 -- 172.21.15.28:0/3612200580 shutdown_connections 2024-02-21T23:04:50.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:50.836+0000 7ff0a9c00640 1 -- 172.21.15.28:0/3612200580 wait complete. 2024-02-21T23:04:50.843 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-02-21T23:04:50.843 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-02-21T23:04:51.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:51.084+0000 7efe3b200640 1 Processor -- start 2024-02-21T23:04:51.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:51.084+0000 7efe3b200640 1 -- start start 2024-02-21T23:04:51.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:51.086+0000 7efe3b200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7efe3c06ea40 0x7efe3c15e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:51.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:51.086+0000 7efe3b200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe3c152fd0 con 0x7efe3c0c1010 2024-02-21T23:04:51.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:51.086+0000 7efe3b200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe3c1532b0 con 0x7efe3c1501c0 2024-02-21T23:04:51.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:51.086+0000 7efe3b200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe3c153590 con 0x7efe3c06ea40 2024-02-21T23:04:51.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:51.086+0000 7efe3a800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7efe3c06ea40 0x7efe3c15e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:51.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:51.086+0000 7efe3a800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7efe3c06ea40 0x7efe3c15e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36704/0 (socket says 172.21.15.28:36704) 2024-02-21T23:04:51.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:51.086+0000 7efe3a800640 1 -- 172.21.15.28:0/2931541559 learned_addr learned my addr 172.21.15.28:0/2931541559 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:51.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:51.087+0000 7efe3a800640 1 -- 172.21.15.28:0/2931541559 >> v1:172.21.15.81:6789/0 conn(0x7efe3c1501c0 legacy=0x7efe3c0a96c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:51.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:51.087+0000 7efe3a800640 1 -- 172.21.15.28:0/2931541559 >> v1:172.21.15.28:6789/0 conn(0x7efe3c0c1010 legacy=0x7efe3c06e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:51.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:51.087+0000 7efe38a00640 1 -- 172.21.15.28:0/2931541559 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 109034881 0 0) 0x7efe3c152fd0 con 0x7efe3c0c1010 2024-02-21T23:04:51.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:51.087+0000 7efe3a800640 1 -- 172.21.15.28:0/2931541559 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe3c153870 con 0x7efe3c06ea40 2024-02-21T23:04:51.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:51.088+0000 7efe3a800640 1 --2- 172.21.15.28:0/2931541559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe3c06ea40 0x7efe3c15e5c0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7efe34009e50 tx=0x7efe3402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3839b7f2c9d14237 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:51.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:51.088+0000 7efe38a00640 1 -- 172.21.15.28:0/2931541559 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe3402cb20 con 0x7efe3c06ea40 2024-02-21T23:04:51.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:51.088+0000 7efe38a00640 1 -- 172.21.15.28:0/2931541559 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe3402ccc0 con 0x7efe3c06ea40 2024-02-21T23:04:51.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:51.088+0000 7efe38a00640 1 -- 172.21.15.28:0/2931541559 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe34034610 con 0x7efe3c06ea40 2024-02-21T23:04:51.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:51.089+0000 7efe3b200640 1 -- 172.21.15.28:0/2931541559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe3c06ea40 msgr2=0x7efe3c15e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:51.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:51.089+0000 7efe3b200640 1 --2- 172.21.15.28:0/2931541559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe3c06ea40 0x7efe3c15e5c0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7efe34009e50 tx=0x7efe3402bf80 comp rx=0 tx=0).stop 2024-02-21T23:04:51.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:51.089+0000 7efe3b200640 1 -- 172.21.15.28:0/2931541559 shutdown_connections 2024-02-21T23:04:51.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:51.089+0000 7efe3b200640 1 --2- 172.21.15.28:0/2931541559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe3c06ea40 0x7efe3c15e5c0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:51.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:51.089+0000 7efe3b200640 1 -- 172.21.15.28:0/2931541559 >> 172.21.15.28:0/2931541559 conn(0x7efe3c0c6610 msgr2=0x7efe3c0c69f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:51.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:51.089+0000 7efe3b200640 1 -- 172.21.15.28:0/2931541559 shutdown_connections 2024-02-21T23:04:51.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:51.089+0000 7efe3b200640 1 -- 172.21.15.28:0/2931541559 wait complete. 2024-02-21T23:04:51.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:51.091+0000 7efe3b200640 1 Processor -- start 2024-02-21T23:04:51.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:51.091+0000 7efe3b200640 1 -- start start 2024-02-21T23:04:51.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:51.092+0000 7efe3b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe3c06ea40 0x7efe3c1ee350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:51.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:51.092+0000 7efe3b200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efe3c0c1010 0x7efe3c0bfcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:51.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:51.092+0000 7efe3b200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efe3c1501c0 0x7efe3c1e7570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:51.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:51.092+0000 7efe3b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe3c0c7340 con 0x7efe3c06ea40 2024-02-21T23:04:51.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:51.092+0000 7efe3b200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe3c0c0c20 con 0x7efe3c0c1010 2024-02-21T23:04:51.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:51.092+0000 7efe3b200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe3c1505a0 con 0x7efe3c1501c0 2024-02-21T23:04:51.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:51.092+0000 7efe39e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe3c06ea40 0x7efe3c1ee350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:51.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:51.092+0000 7efe39e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe3c06ea40 0x7efe3c1ee350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36716/0 (socket says 172.21.15.28:36716) 2024-02-21T23:04:51.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:51.092+0000 7efe39e00640 1 -- 172.21.15.28:0/1904719369 learned_addr learned my addr 172.21.15.28:0/1904719369 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:51.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:51.092+0000 7efe39400640 1 --2- 172.21.15.28:0/1904719369 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efe3c0c1010 0x7efe3c0bfcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:51.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:51.092+0000 7efe3a800640 1 --2- 172.21.15.28:0/1904719369 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efe3c1501c0 0x7efe3c1e7570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:51.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:51.092+0000 7efe39e00640 1 -- 172.21.15.28:0/1904719369 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efe3c1501c0 msgr2=0x7efe3c1e7570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:51.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:51.092+0000 7efe39e00640 1 --2- 172.21.15.28:0/1904719369 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efe3c1501c0 0x7efe3c1e7570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:51.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:51.092+0000 7efe39e00640 1 -- 172.21.15.28:0/1904719369 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efe3c0c1010 msgr2=0x7efe3c0bfcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:51.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:51.092+0000 7efe39e00640 1 --2- 172.21.15.28:0/1904719369 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efe3c0c1010 0x7efe3c0bfcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:51.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:51.092+0000 7efe39e00640 1 -- 172.21.15.28:0/1904719369 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe3c0c5500 con 0x7efe3c06ea40 2024-02-21T23:04:51.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:51.092+0000 7efe39400640 1 --2- 172.21.15.28:0/1904719369 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efe3c0c1010 0x7efe3c0bfcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:04:51.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:51.093+0000 7efe39e00640 1 --2- 172.21.15.28:0/1904719369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe3c06ea40 0x7efe3c1ee350 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7efe2400c480 tx=0x7efe2400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:51.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:51.093+0000 7efe32000640 1 -- 172.21.15.28:0/1904719369 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe240052d0 con 0x7efe3c06ea40 2024-02-21T23:04:51.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:51.093+0000 7efe32000640 1 -- 172.21.15.28:0/1904719369 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe24005470 con 0x7efe3c06ea40 2024-02-21T23:04:51.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:51.093+0000 7efe3b200640 1 -- 172.21.15.28:0/1904719369 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe3c0c57e0 con 0x7efe3c06ea40 2024-02-21T23:04:51.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:51.093+0000 7efe32000640 1 -- 172.21.15.28:0/1904719369 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe24012870 con 0x7efe3c06ea40 2024-02-21T23:04:51.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:51.093+0000 7efe3b200640 1 -- 172.21.15.28:0/1904719369 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe3c0c5da0 con 0x7efe3c06ea40 2024-02-21T23:04:51.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:51.094+0000 7efe3b200640 1 -- 172.21.15.28:0/1904719369 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe3c04f290 con 0x7efe3c06ea40 2024-02-21T23:04:51.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:51.095+0000 7efe32000640 1 -- 172.21.15.28:0/1904719369 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efe24012a10 con 0x7efe3c06ea40 2024-02-21T23:04:51.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:51.096+0000 7efe32000640 1 --2- 172.21.15.28:0/1904719369 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7efe1806b4d0 0x7efe1806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:51.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:51.096+0000 7efe32000640 1 -- 172.21.15.28:0/1904719369 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5249+0+0 (secure 0 0 0) 0x7efe24089d10 con 0x7efe3c06ea40 2024-02-21T23:04:51.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:51.097+0000 7efe39400640 1 --2- 172.21.15.28:0/1904719369 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7efe1806b4d0 0x7efe1806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:51.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:51.098+0000 7efe39400640 1 --2- 172.21.15.28:0/1904719369 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7efe1806b4d0 0x7efe1806d990 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7efe3c0c04c0 tx=0x7efe2c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:51.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:51.101+0000 7efe32000640 1 -- 172.21.15.28:0/1904719369 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7efe24058d40 con 0x7efe3c06ea40 2024-02-21T23:04:51.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:51.301+0000 7efe3b200640 1 -- 172.21.15.28:0/1904719369 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7efe3c06b9c0 con 0x7efe3c06ea40 2024-02-21T23:04:51.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:51.928+0000 7efe32000640 1 -- 172.21.15.28:0/1904719369 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v50) v1 ==== 138+0+0 (secure 0 0 0) 0x7efe24058640 con 0x7efe3c06ea40 2024-02-21T23:04:51.917 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' created 2024-02-21T23:04:51.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:51.932+0000 7efe3b200640 1 -- 172.21.15.28:0/1904719369 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7efe1806b4d0 msgr2=0x7efe1806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:51.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:51.932+0000 7efe3b200640 1 --2- 172.21.15.28:0/1904719369 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7efe1806b4d0 0x7efe1806d990 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7efe3c0c04c0 tx=0x7efe2c009040 comp rx=0 tx=0).stop 2024-02-21T23:04:51.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:51.932+0000 7efe3b200640 1 -- 172.21.15.28:0/1904719369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe3c06ea40 msgr2=0x7efe3c1ee350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:51.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:51.932+0000 7efe3b200640 1 --2- 172.21.15.28:0/1904719369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe3c06ea40 0x7efe3c1ee350 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7efe2400c480 tx=0x7efe2400c940 comp rx=0 tx=0).stop 2024-02-21T23:04:51.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:51.932+0000 7efe3b200640 1 -- 172.21.15.28:0/1904719369 shutdown_connections 2024-02-21T23:04:51.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:51.932+0000 7efe3b200640 1 --2- 172.21.15.28:0/1904719369 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7efe1806b4d0 0x7efe1806d990 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:51.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:51.932+0000 7efe3b200640 1 --2- 172.21.15.28:0/1904719369 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efe3c1501c0 0x7efe3c1e7570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:51.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:51.932+0000 7efe3b200640 1 --2- 172.21.15.28:0/1904719369 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efe3c0c1010 0x7efe3c0bfcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:51.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:51.932+0000 7efe3b200640 1 --2- 172.21.15.28:0/1904719369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe3c06ea40 0x7efe3c1ee350 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:51.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:51.932+0000 7efe3b200640 1 -- 172.21.15.28:0/1904719369 >> 172.21.15.28:0/1904719369 conn(0x7efe3c0c6610 msgr2=0x7efe3c142cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:51.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:51.932+0000 7efe3b200640 1 -- 172.21.15.28:0/1904719369 shutdown_connections 2024-02-21T23:04:51.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:51.933+0000 7efe3b200640 1 -- 172.21.15.28:0/1904719369 wait complete. 2024-02-21T23:04:51.951 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-02-21T23:04:52.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.169+0000 7f4418a00640 1 Processor -- start 2024-02-21T23:04:52.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.169+0000 7f4418a00640 1 -- start start 2024-02-21T23:04:52.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.171+0000 7f4418a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f44140c35c0 0x7f4414156bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:52.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.171+0000 7f4418a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f4414150640 0x7f4414152c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:52.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.171+0000 7f4418a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f441406f1b0 con 0x7f44140a96c0 2024-02-21T23:04:52.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.171+0000 7f4418a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f441406ea40 con 0x7f44140c35c0 2024-02-21T23:04:52.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.171+0000 7f4418a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44140c2ae0 con 0x7f4414150640 2024-02-21T23:04:52.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.171+0000 7f4412800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f44140c35c0 0x7f4414156bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:52.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.171+0000 7f4412800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f44140c35c0 0x7f4414156bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36726/0 (socket says 172.21.15.28:36726) 2024-02-21T23:04:52.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.171+0000 7f4412800640 1 -- 172.21.15.28:0/3301741694 learned_addr learned my addr 172.21.15.28:0/3301741694 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:52.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.171+0000 7f4413200640 1 --2- 172.21.15.28:0/3301741694 >> v2:172.21.15.81:3300/0 conn(0x7f4414150640 0x7f4414152c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:52.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.172+0000 7f4412800640 1 -- 172.21.15.28:0/3301741694 >> v2:172.21.15.81:3300/0 conn(0x7f4414150640 msgr2=0x7f4414152c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:52.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.172+0000 7f4412800640 1 --2- 172.21.15.28:0/3301741694 >> v2:172.21.15.81:3300/0 conn(0x7f4414150640 0x7f4414152c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:52.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.172+0000 7f4412800640 1 -- 172.21.15.28:0/3301741694 >> v1:172.21.15.81:6789/0 conn(0x7f44140a96c0 legacy=0x7f441414ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:52.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.172+0000 7f4412800640 1 -- 172.21.15.28:0/3301741694 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44140c2dc0 con 0x7f44140c35c0 2024-02-21T23:04:52.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.172+0000 7f4413200640 1 --2- 172.21.15.28:0/3301741694 >> v2:172.21.15.81:3300/0 conn(0x7f4414150640 0x7f4414152c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:04:52.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.173+0000 7f4412800640 1 --2- 172.21.15.28:0/3301741694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44140c35c0 0x7f4414156bf0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f44140ac2b0 tx=0x7f440402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a8ff004d56e4d58 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:52.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.173+0000 7f4411400640 1 -- 172.21.15.28:0/3301741694 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f440402e070 con 0x7f44140c35c0 2024-02-21T23:04:52.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.173+0000 7f4411400640 1 -- 172.21.15.28:0/3301741694 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4404037070 con 0x7f44140c35c0 2024-02-21T23:04:52.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.173+0000 7f4411400640 1 -- 172.21.15.28:0/3301741694 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4404031a00 con 0x7f44140c35c0 2024-02-21T23:04:52.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.174+0000 7f4418a00640 1 -- 172.21.15.28:0/3301741694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44140c35c0 msgr2=0x7f4414156bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:52.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.174+0000 7f4418a00640 1 --2- 172.21.15.28:0/3301741694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44140c35c0 0x7f4414156bf0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f44140ac2b0 tx=0x7f440402b880 comp rx=0 tx=0).stop 2024-02-21T23:04:52.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.174+0000 7f4418a00640 1 -- 172.21.15.28:0/3301741694 shutdown_connections 2024-02-21T23:04:52.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.174+0000 7f4418a00640 1 --2- 172.21.15.28:0/3301741694 >> v2:172.21.15.81:3300/0 conn(0x7f4414150640 0x7f4414152c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:52.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.174+0000 7f4418a00640 1 --2- 172.21.15.28:0/3301741694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44140c35c0 0x7f4414156bf0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:52.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.174+0000 7f4418a00640 1 -- 172.21.15.28:0/3301741694 >> 172.21.15.28:0/3301741694 conn(0x7f44140a8690 msgr2=0x7f44140a8a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:52.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.174+0000 7f4418a00640 1 -- 172.21.15.28:0/3301741694 shutdown_connections 2024-02-21T23:04:52.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.174+0000 7f4418a00640 1 -- 172.21.15.28:0/3301741694 wait complete. 2024-02-21T23:04:52.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.176+0000 7f4418a00640 1 Processor -- start 2024-02-21T23:04:52.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.176+0000 7f4418a00640 1 -- start start 2024-02-21T23:04:52.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.177+0000 7f4418a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44140a96c0 0x7f4414155700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:52.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.177+0000 7f4418a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44140c35c0 0x7f44141ea610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:52.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.177+0000 7f4418a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4414150640 0x7f44141eab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:52.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.177+0000 7f4418a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44140c39a0 con 0x7f44140c35c0 2024-02-21T23:04:52.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.177+0000 7f4418a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f441406e7b0 con 0x7f44140a96c0 2024-02-21T23:04:52.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.177+0000 7f4418a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44140c3b20 con 0x7f4414150640 2024-02-21T23:04:52.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.177+0000 7f4411e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44140c35c0 0x7f44141ea610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:52.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.177+0000 7f4411e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44140c35c0 0x7f44141ea610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36728/0 (socket says 172.21.15.28:36728) 2024-02-21T23:04:52.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.177+0000 7f4411e00640 1 -- 172.21.15.28:0/2363340487 learned_addr learned my addr 172.21.15.28:0/2363340487 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:52.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.177+0000 7f4412800640 1 --2- 172.21.15.28:0/2363340487 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44140a96c0 0x7f4414155700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:52.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.177+0000 7f4413200640 1 --2- 172.21.15.28:0/2363340487 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4414150640 0x7f44141eab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:52.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.178+0000 7f4411e00640 1 -- 172.21.15.28:0/2363340487 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4414150640 msgr2=0x7f44141eab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:52.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.178+0000 7f4411e00640 1 --2- 172.21.15.28:0/2363340487 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4414150640 0x7f44141eab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:52.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.178+0000 7f4411e00640 1 -- 172.21.15.28:0/2363340487 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44140a96c0 msgr2=0x7f4414155700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:52.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.178+0000 7f4411e00640 1 --2- 172.21.15.28:0/2363340487 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44140a96c0 0x7f4414155700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:52.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.178+0000 7f4411e00640 1 -- 172.21.15.28:0/2363340487 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44140c5e20 con 0x7f44140c35c0 2024-02-21T23:04:52.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.178+0000 7f4412800640 1 --2- 172.21.15.28:0/2363340487 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44140a96c0 0x7f4414155700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:04:52.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.178+0000 7f4411e00640 1 --2- 172.21.15.28:0/2363340487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44140c35c0 0x7f44141ea610 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f440000b420 tx=0x7f440000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:52.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.178+0000 7f440aa00640 1 -- 172.21.15.28:0/2363340487 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4400011830 con 0x7f44140c35c0 2024-02-21T23:04:52.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.178+0000 7f440aa00640 1 -- 172.21.15.28:0/2363340487 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4400016070 con 0x7f44140c35c0 2024-02-21T23:04:52.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.178+0000 7f440aa00640 1 -- 172.21.15.28:0/2363340487 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4400011c10 con 0x7f44140c35c0 2024-02-21T23:04:52.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.178+0000 7f4418a00640 1 -- 172.21.15.28:0/2363340487 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f441414a690 con 0x7f44140c35c0 2024-02-21T23:04:52.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.179+0000 7f4418a00640 1 -- 172.21.15.28:0/2363340487 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44140c41f0 con 0x7f44140c35c0 2024-02-21T23:04:52.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.180+0000 7f4418a00640 1 -- 172.21.15.28:0/2363340487 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44140ac9b0 con 0x7f44140c35c0 2024-02-21T23:04:52.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.181+0000 7f440aa00640 1 -- 172.21.15.28:0/2363340487 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4400004020 con 0x7f44140c35c0 2024-02-21T23:04:52.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.182+0000 7f440aa00640 1 --2- 172.21.15.28:0/2363340487 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f43f406b4d0 0x7f43f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:52.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.182+0000 7f440aa00640 1 -- 172.21.15.28:0/2363340487 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5619+0+0 (secure 0 0 0) 0x7f440008a690 con 0x7f44140c35c0 2024-02-21T23:04:52.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.183+0000 7f4412800640 1 --2- 172.21.15.28:0/2363340487 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f43f406b4d0 0x7f43f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:52.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.184+0000 7f4412800640 1 --2- 172.21.15.28:0/2363340487 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f43f406b4d0 0x7f43f406d990 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f4404036000 tx=0x7f440403d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:52.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.186+0000 7f440aa00640 1 -- 172.21.15.28:0/2363340487 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f44140ac9b0 con 0x7f44140c35c0 2024-02-21T23:04:52.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.386+0000 7f4418a00640 1 -- 172.21.15.28:0/2363340487 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f441404f290 con 0x7f44140c35c0 2024-02-21T23:04:52.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.939+0000 7f440aa00640 1 -- 172.21.15.28:0/2363340487 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v51) v1 ==== 170+0+0 (secure 0 0 0) 0x7f4400020020 con 0x7f44140c35c0 2024-02-21T23:04:52.928 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' created 2024-02-21T23:04:52.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.943+0000 7f4418a00640 1 -- 172.21.15.28:0/2363340487 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f43f406b4d0 msgr2=0x7f43f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:52.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.943+0000 7f4418a00640 1 --2- 172.21.15.28:0/2363340487 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f43f406b4d0 0x7f43f406d990 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f4404036000 tx=0x7f440403d040 comp rx=0 tx=0).stop 2024-02-21T23:04:52.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.943+0000 7f4418a00640 1 -- 172.21.15.28:0/2363340487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44140c35c0 msgr2=0x7f44141ea610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:52.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.943+0000 7f4418a00640 1 --2- 172.21.15.28:0/2363340487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44140c35c0 0x7f44141ea610 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f440000b420 tx=0x7f440000b8e0 comp rx=0 tx=0).stop 2024-02-21T23:04:52.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.943+0000 7f4418a00640 1 -- 172.21.15.28:0/2363340487 shutdown_connections 2024-02-21T23:04:52.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.943+0000 7f4418a00640 1 --2- 172.21.15.28:0/2363340487 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f43f406b4d0 0x7f43f406d990 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:52.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.943+0000 7f4418a00640 1 --2- 172.21.15.28:0/2363340487 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4414150640 0x7f44141eab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:52.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.943+0000 7f4418a00640 1 --2- 172.21.15.28:0/2363340487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44140c35c0 0x7f44141ea610 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:52.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.943+0000 7f4418a00640 1 --2- 172.21.15.28:0/2363340487 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44140a96c0 0x7f4414155700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:52.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.943+0000 7f4418a00640 1 -- 172.21.15.28:0/2363340487 >> 172.21.15.28:0/2363340487 conn(0x7f44140a8690 msgr2=0x7f441414f200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:52.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.944+0000 7f4418a00640 1 -- 172.21.15.28:0/2363340487 shutdown_connections 2024-02-21T23:04:52.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:52.945+0000 7f4418a00640 1 -- 172.21.15.28:0/2363340487 wait complete. 2024-02-21T23:04:52.972 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-02-21T23:04:53.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.168+0000 7f4462200640 1 Processor -- start 2024-02-21T23:04:53.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.168+0000 7f4462200640 1 -- start start 2024-02-21T23:04:53.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.169+0000 7f4462200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f445c0c5a70 0x7f445c155a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:53.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.169+0000 7f4462200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f445c0c2b30 0x7f445c06d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:53.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.170+0000 7f4462200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f445c071970 con 0x7f445c0c22e0 2024-02-21T23:04:53.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.170+0000 7f4462200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f445c072a10 con 0x7f445c0c2b30 2024-02-21T23:04:53.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.170+0000 7f4462200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f445c072580 con 0x7f445c0c5a70 2024-02-21T23:04:53.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.170+0000 7f445be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f445c0c2b30 0x7f445c06d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:53.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.170+0000 7f445be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f445c0c2b30 0x7f445c06d830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36734/0 (socket says 172.21.15.28:36734) 2024-02-21T23:04:53.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.170+0000 7f445be00640 1 -- 172.21.15.28:0/2497213002 learned_addr learned my addr 172.21.15.28:0/2497213002 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:53.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.170+0000 7f445aa00640 1 --1- 172.21.15.28:0/2497213002 >> v1:172.21.15.28:6789/0 conn(0x7f445c0c22e0 0x7f445c06d120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:57142/0 (socket says 172.21.15.28:57142) 2024-02-21T23:04:53.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.170+0000 7f445b400640 1 --2- 172.21.15.28:0/2497213002 >> v2:172.21.15.81:3300/0 conn(0x7f445c0c5a70 0x7f445c155a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:53.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.170+0000 7f445be00640 1 -- 172.21.15.28:0/2497213002 >> v2:172.21.15.81:3300/0 conn(0x7f445c0c5a70 msgr2=0x7f445c155a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:53.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.170+0000 7f445be00640 1 --2- 172.21.15.28:0/2497213002 >> v2:172.21.15.81:3300/0 conn(0x7f445c0c5a70 0x7f445c155a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:53.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.170+0000 7f445be00640 1 -- 172.21.15.28:0/2497213002 >> v1:172.21.15.28:6789/0 conn(0x7f445c0c22e0 legacy=0x7f445c06d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:53.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.170+0000 7f445be00640 1 -- 172.21.15.28:0/2497213002 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f445c071c50 con 0x7f445c0c2b30 2024-02-21T23:04:53.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.170+0000 7f445a000640 1 -- 172.21.15.28:0/2497213002 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1836851892 0 0) 0x7f445c071970 con 0x7f445c0c22e0 2024-02-21T23:04:53.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.170+0000 7f445b400640 1 --2- 172.21.15.28:0/2497213002 >> v2:172.21.15.81:3300/0 conn(0x7f445c0c5a70 0x7f445c155a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:04:53.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.172+0000 7f445be00640 1 --2- 172.21.15.28:0/2497213002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f445c0c2b30 0x7f445c06d830 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f445c0c7810 tx=0x7f445402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2919b9e80bbcb43 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:53.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.172+0000 7f445a000640 1 -- 172.21.15.28:0/2497213002 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f445402ca80 con 0x7f445c0c2b30 2024-02-21T23:04:53.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.172+0000 7f445a000640 1 -- 172.21.15.28:0/2497213002 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f445402cc20 con 0x7f445c0c2b30 2024-02-21T23:04:53.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.172+0000 7f445a000640 1 -- 172.21.15.28:0/2497213002 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44540343f0 con 0x7f445c0c2b30 2024-02-21T23:04:53.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.172+0000 7f4462200640 1 -- 172.21.15.28:0/2497213002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f445c0c2b30 msgr2=0x7f445c06d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:53.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.172+0000 7f4462200640 1 --2- 172.21.15.28:0/2497213002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f445c0c2b30 0x7f445c06d830 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f445c0c7810 tx=0x7f445402bee0 comp rx=0 tx=0).stop 2024-02-21T23:04:53.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.173+0000 7f4462200640 1 -- 172.21.15.28:0/2497213002 shutdown_connections 2024-02-21T23:04:53.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.173+0000 7f4462200640 1 --2- 172.21.15.28:0/2497213002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f445c0c2b30 0x7f445c06d830 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:53.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.173+0000 7f4462200640 1 --2- 172.21.15.28:0/2497213002 >> v2:172.21.15.81:3300/0 conn(0x7f445c0c5a70 0x7f445c155a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:53.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.173+0000 7f4462200640 1 -- 172.21.15.28:0/2497213002 >> 172.21.15.28:0/2497213002 conn(0x7f445c0a85a0 msgr2=0x7f445c0a8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:53.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.173+0000 7f4462200640 1 -- 172.21.15.28:0/2497213002 shutdown_connections 2024-02-21T23:04:53.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.173+0000 7f4462200640 1 -- 172.21.15.28:0/2497213002 wait complete. 2024-02-21T23:04:53.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.174+0000 7f4462200640 1 Processor -- start 2024-02-21T23:04:53.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.175+0000 7f4462200640 1 -- start start 2024-02-21T23:04:53.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.175+0000 7f4462200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f445c0c22e0 0x7f445c150220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:53.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.175+0000 7f4462200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f445c0c2b30 0x7f445c1480c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:53.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.176+0000 7f4462200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f445c0c5a70 0x7f445c148600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:53.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.176+0000 7f4462200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f445c0a6160 con 0x7f445c0c5a70 2024-02-21T23:04:53.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.176+0000 7f4462200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f445c072210 con 0x7f445c0c22e0 2024-02-21T23:04:53.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.176+0000 7f4462200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f445c0a62e0 con 0x7f445c0c2b30 2024-02-21T23:04:53.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.176+0000 7f445be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f445c0c5a70 0x7f445c148600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:53.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.176+0000 7f445be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f445c0c5a70 0x7f445c148600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36746/0 (socket says 172.21.15.28:36746) 2024-02-21T23:04:53.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.176+0000 7f445be00640 1 -- 172.21.15.28:0/2380885886 learned_addr learned my addr 172.21.15.28:0/2380885886 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:53.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.176+0000 7f445aa00640 1 --2- 172.21.15.28:0/2380885886 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f445c0c2b30 0x7f445c1480c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:53.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.176+0000 7f445b400640 1 --2- 172.21.15.28:0/2380885886 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f445c0c22e0 0x7f445c150220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:53.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.176+0000 7f445be00640 1 -- 172.21.15.28:0/2380885886 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f445c0c2b30 msgr2=0x7f445c1480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:53.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.176+0000 7f445be00640 1 --2- 172.21.15.28:0/2380885886 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f445c0c2b30 0x7f445c1480c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:53.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.176+0000 7f445be00640 1 -- 172.21.15.28:0/2380885886 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f445c0c22e0 msgr2=0x7f445c150220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:53.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.176+0000 7f445be00640 1 --2- 172.21.15.28:0/2380885886 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f445c0c22e0 0x7f445c150220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:53.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.176+0000 7f445be00640 1 -- 172.21.15.28:0/2380885886 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f445c1ec840 con 0x7f445c0c5a70 2024-02-21T23:04:53.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.176+0000 7f445b400640 1 --2- 172.21.15.28:0/2380885886 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f445c0c22e0 0x7f445c150220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:04:53.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.177+0000 7f445be00640 1 --2- 172.21.15.28:0/2380885886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f445c0c5a70 0x7f445c148600 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f4454036040 tx=0x7f4454004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:53.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.177+0000 7f4453400640 1 -- 172.21.15.28:0/2380885886 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4454043020 con 0x7f445c0c5a70 2024-02-21T23:04:53.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.177+0000 7f4453400640 1 -- 172.21.15.28:0/2380885886 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f445403e450 con 0x7f445c0c5a70 2024-02-21T23:04:53.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.177+0000 7f4462200640 1 -- 172.21.15.28:0/2380885886 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f445c151440 con 0x7f445c0c5a70 2024-02-21T23:04:53.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.177+0000 7f4453400640 1 -- 172.21.15.28:0/2380885886 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4454009510 con 0x7f445c0c5a70 2024-02-21T23:04:53.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.177+0000 7f4462200640 1 -- 172.21.15.28:0/2380885886 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f445c151a00 con 0x7f445c0c5a70 2024-02-21T23:04:53.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.178+0000 7f4462200640 1 -- 172.21.15.28:0/2380885886 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f445c067f30 con 0x7f445c0c5a70 2024-02-21T23:04:53.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.179+0000 7f4453400640 1 -- 172.21.15.28:0/2380885886 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4454046020 con 0x7f445c0c5a70 2024-02-21T23:04:53.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.180+0000 7f4453400640 1 --2- 172.21.15.28:0/2380885886 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f443c06b4d0 0x7f443c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:53.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.180+0000 7f4453400640 1 -- 172.21.15.28:0/2380885886 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6001+0+0 (secure 0 0 0) 0x7f44540acc60 con 0x7f445c0c5a70 2024-02-21T23:04:53.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.181+0000 7f445b400640 1 --2- 172.21.15.28:0/2380885886 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f443c06b4d0 0x7f443c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:53.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.182+0000 7f445b400640 1 --2- 172.21.15.28:0/2380885886 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f443c06b4d0 0x7f443c06d990 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f444c006fd0 tx=0x7f444c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:53.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.186+0000 7f4453400640 1 -- 172.21.15.28:0/2380885886 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f445404d030 con 0x7f445c0c5a70 2024-02-21T23:04:53.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.429+0000 7f4462200640 1 -- 172.21.15.28:0/2380885886 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f445c0a5b10 con 0x7f445c0c5a70 2024-02-21T23:04:53.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.944+0000 7f4453400640 1 -- 172.21.15.28:0/2380885886 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 7 and data pool 8 v28) v1 ==== 172+0+0 (secure 0 0 0) 0x7f445407b9a0 con 0x7f445c0c5a70 2024-02-21T23:04:53.933 INFO:teuthology.orchestra.run.smithi028.stderr:new fs with metadata pool 7 and data pool 8 2024-02-21T23:04:53.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.950+0000 7f4462200640 1 -- 172.21.15.28:0/2380885886 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f443c06b4d0 msgr2=0x7f443c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:53.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.950+0000 7f4462200640 1 --2- 172.21.15.28:0/2380885886 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f443c06b4d0 0x7f443c06d990 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f444c006fd0 tx=0x7f444c008040 comp rx=0 tx=0).stop 2024-02-21T23:04:53.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.950+0000 7f4462200640 1 -- 172.21.15.28:0/2380885886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f445c0c5a70 msgr2=0x7f445c148600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:53.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.950+0000 7f4462200640 1 --2- 172.21.15.28:0/2380885886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f445c0c5a70 0x7f445c148600 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f4454036040 tx=0x7f4454004a40 comp rx=0 tx=0).stop 2024-02-21T23:04:53.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.951+0000 7f4462200640 1 -- 172.21.15.28:0/2380885886 shutdown_connections 2024-02-21T23:04:53.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.951+0000 7f4462200640 1 --2- 172.21.15.28:0/2380885886 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f443c06b4d0 0x7f443c06d990 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:53.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.951+0000 7f4462200640 1 --2- 172.21.15.28:0/2380885886 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f445c0c2b30 0x7f445c1480c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:53.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.951+0000 7f4462200640 1 --2- 172.21.15.28:0/2380885886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f445c0c5a70 0x7f445c148600 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:53.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.951+0000 7f4462200640 1 --2- 172.21.15.28:0/2380885886 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f445c0c22e0 0x7f445c150220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:53.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.951+0000 7f4462200640 1 -- 172.21.15.28:0/2380885886 >> 172.21.15.28:0/2380885886 conn(0x7f445c0a85a0 msgr2=0x7f445c15d0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:53.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.953+0000 7f4462200640 1 -- 172.21.15.28:0/2380885886 shutdown_connections 2024-02-21T23:04:53.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:53.953+0000 7f4462200640 1 -- 172.21.15.28:0/2380885886 wait complete. 2024-02-21T23:04:53.984 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:04:54.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.171+0000 7fe0c5800640 1 Processor -- start 2024-02-21T23:04:54.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.172+0000 7fe0c5800640 1 -- start start 2024-02-21T23:04:54.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.173+0000 7fe0c5800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe0c00c11f0 0x7fe0c0153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:54.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.173+0000 7fe0c5800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0c00c1a30 con 0x7fe0c015b6d0 2024-02-21T23:04:54.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.173+0000 7fe0c5800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0c0152e70 con 0x7fe0c00c15d0 2024-02-21T23:04:54.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.173+0000 7fe0c5800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c00a5ec0 con 0x7fe0c00c11f0 2024-02-21T23:04:54.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.173+0000 7fe0bf400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe0c015b6d0 0x7fe0c015dac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:57152/0 (socket says 172.21.15.28:57152) 2024-02-21T23:04:54.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.173+0000 7fe0bf400640 1 -- 172.21.15.28:0/2138524861 learned_addr learned my addr 172.21.15.28:0/2138524861 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:04:54.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.174+0000 7fe0bea00640 1 --2- 172.21.15.28:0/2138524861 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe0c00c11f0 0x7fe0c0153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:54.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.174+0000 7fe0be000640 1 -- 172.21.15.28:0/2138524861 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2121947575 0 0) 0x7fe0c00c1a30 con 0x7fe0c015b6d0 2024-02-21T23:04:54.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.174+0000 7fe0be000640 1 -- 172.21.15.28:0/2138524861 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0a0003520 con 0x7fe0c015b6d0 2024-02-21T23:04:54.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.174+0000 7fe0be000640 1 -- 172.21.15.28:0/2138524861 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 574527040 0 0) 0x7fe0a0003520 con 0x7fe0c015b6d0 2024-02-21T23:04:54.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.174+0000 7fe0be000640 1 -- 172.21.15.28:0/2138524861 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe0c00c11f0 msgr2=0x7fe0c0153950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:54.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.174+0000 7fe0be000640 1 --2- 172.21.15.28:0/2138524861 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe0c00c11f0 0x7fe0c0153950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:54.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.174+0000 7fe0be000640 1 -- 172.21.15.28:0/2138524861 >> v1:172.21.15.81:6789/0 conn(0x7fe0c00c15d0 legacy=0x7fe0c006dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:54.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.174+0000 7fe0be000640 1 -- 172.21.15.28:0/2138524861 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0c0153150 con 0x7fe0c015b6d0 2024-02-21T23:04:54.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.174+0000 7fe0be000640 1 -- 172.21.15.28:0/2138524861 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fe0b80019f0 con 0x7fe0c015b6d0 2024-02-21T23:04:54.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.174+0000 7fe0be000640 1 -- 172.21.15.28:0/2138524861 >> v1:172.21.15.28:6789/0 conn(0x7fe0c015b6d0 legacy=0x7fe0c015dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:54.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.174+0000 7fe0be000640 1 --2- 172.21.15.28:0/2138524861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0a00039e0 0x7fe0a0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:54.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.174+0000 7fe0be000640 1 --2- 172.21.15.28:0/2138524861 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe0a0004800 0x7fe0a0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:54.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.174+0000 7fe0be000640 1 --2- 172.21.15.28:0/2138524861 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe0c00c11f0 0x7fe0a000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:54.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.174+0000 7fe0be000640 1 -- 172.21.15.28:0/2138524861 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c00a5ec0 con 0x7fe0a00039e0 2024-02-21T23:04:54.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.174+0000 7fe0be000640 1 -- 172.21.15.28:0/2138524861 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0a0005520 con 0x7fe0a0004800 2024-02-21T23:04:54.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.174+0000 7fe0be000640 1 -- 172.21.15.28:0/2138524861 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0a000bf00 con 0x7fe0c00c11f0 2024-02-21T23:04:54.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.174+0000 7fe0bea00640 1 --2- 172.21.15.28:0/2138524861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0a00039e0 0x7fe0a0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:54.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.175+0000 7fe0bf400640 1 --2- 172.21.15.28:0/2138524861 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe0c00c11f0 0x7fe0a000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:54.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.175+0000 7fe0bea00640 1 -- 172.21.15.28:0/2138524861 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe0c00c11f0 msgr2=0x7fe0a000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:54.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.175+0000 7fe0bea00640 1 --2- 172.21.15.28:0/2138524861 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe0c00c11f0 0x7fe0a000b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:54.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.175+0000 7fe0bea00640 1 -- 172.21.15.28:0/2138524861 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe0a0004800 msgr2=0x7fe0a0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:54.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.175+0000 7fe0bea00640 1 --2- 172.21.15.28:0/2138524861 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe0a0004800 0x7fe0a0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:54.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.175+0000 7fe0bea00640 1 -- 172.21.15.28:0/2138524861 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0a0003520 con 0x7fe0a00039e0 2024-02-21T23:04:54.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.176+0000 7fe0bea00640 1 --2- 172.21.15.28:0/2138524861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0a00039e0 0x7fe0a0003e20 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fe0b0009ac0 tx=0x7fe0a0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d754bb389a51c41 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:54.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.176+0000 7fe0be000640 1 -- 172.21.15.28:0/2138524861 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0b0039070 con 0x7fe0a00039e0 2024-02-21T23:04:54.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.176+0000 7fe0be000640 1 -- 172.21.15.28:0/2138524861 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0b002c6d0 con 0x7fe0a00039e0 2024-02-21T23:04:54.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.176+0000 7fe0be000640 1 -- 172.21.15.28:0/2138524861 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0b0009690 con 0x7fe0a00039e0 2024-02-21T23:04:54.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.177+0000 7fe0c5800640 1 -- 172.21.15.28:0/2138524861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0a00039e0 msgr2=0x7fe0a0003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:54.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.177+0000 7fe0c5800640 1 --2- 172.21.15.28:0/2138524861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0a00039e0 0x7fe0a0003e20 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fe0b0009ac0 tx=0x7fe0a0005830 comp rx=0 tx=0).stop 2024-02-21T23:04:54.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.177+0000 7fe0bf400640 1 -- 172.21.15.28:0/2138524861 reap_dead start 2024-02-21T23:04:54.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.177+0000 7fe0c5800640 1 -- 172.21.15.28:0/2138524861 shutdown_connections 2024-02-21T23:04:54.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.177+0000 7fe0c5800640 1 -- 172.21.15.28:0/2138524861 >> 172.21.15.28:0/2138524861 conn(0x7fe0c00c67f0 msgr2=0x7fe0c00c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:54.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.177+0000 7fe0c5800640 1 -- 172.21.15.28:0/2138524861 shutdown_connections 2024-02-21T23:04:54.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.177+0000 7fe0c5800640 1 -- 172.21.15.28:0/2138524861 wait complete. 2024-02-21T23:04:54.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.179+0000 7fe0c5800640 1 Processor -- start 2024-02-21T23:04:54.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.179+0000 7fe0c5800640 1 -- start start 2024-02-21T23:04:54.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.180+0000 7fe0c5800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0a00039e0 0x7fe0c01eed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:54.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.180+0000 7fe0c5800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe0c014c250 0x7fe0c00aac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:54.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.180+0000 7fe0c5800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe0c0071380 0x7fe0c0071800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:54.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.180+0000 7fe0c5800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c00c4c00 con 0x7fe0a00039e0 2024-02-21T23:04:54.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.180+0000 7fe0c5800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c014d1c0 con 0x7fe0c0071380 2024-02-21T23:04:54.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.180+0000 7fe0c5800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c00a6110 con 0x7fe0c014c250 2024-02-21T23:04:54.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.180+0000 7fe0bea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0a00039e0 0x7fe0c01eed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:54.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.180+0000 7fe0bea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0a00039e0 0x7fe0c01eed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36756/0 (socket says 172.21.15.28:36756) 2024-02-21T23:04:54.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.180+0000 7fe0bea00640 1 -- 172.21.15.28:0/2376798413 learned_addr learned my addr 172.21.15.28:0/2376798413 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:54.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.180+0000 7fe0bea00640 1 -- 172.21.15.28:0/2376798413 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe0c014c250 msgr2=0x7fe0c00aac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:54.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.180+0000 7fe0b7e00640 1 --2- 172.21.15.28:0/2376798413 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe0c014c250 0x7fe0c00aac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:54.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.181+0000 7fe0bea00640 1 --2- 172.21.15.28:0/2376798413 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe0c014c250 0x7fe0c00aac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:54.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.181+0000 7fe0bea00640 1 -- 172.21.15.28:0/2376798413 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe0c0071380 msgr2=0x7fe0c0071800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:54.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.181+0000 7fe0bf400640 1 --2- 172.21.15.28:0/2376798413 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe0c0071380 0x7fe0c0071800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:54.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.181+0000 7fe0bea00640 1 --2- 172.21.15.28:0/2376798413 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe0c0071380 0x7fe0c0071800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:54.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.181+0000 7fe0bea00640 1 -- 172.21.15.28:0/2376798413 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0c00c0090 con 0x7fe0a00039e0 2024-02-21T23:04:54.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.181+0000 7fe0b7e00640 1 --2- 172.21.15.28:0/2376798413 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe0c014c250 0x7fe0c00aac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:04:54.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.181+0000 7fe0bf400640 1 --2- 172.21.15.28:0/2376798413 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe0c0071380 0x7fe0c0071800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:04:54.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.181+0000 7fe0bea00640 1 --2- 172.21.15.28:0/2376798413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0a00039e0 0x7fe0c01eed70 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fe0b0009ac0 tx=0x7fe0b0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:54.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.181+0000 7fe0b6a00640 1 -- 172.21.15.28:0/2376798413 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0b0042020 con 0x7fe0a00039e0 2024-02-21T23:04:54.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.181+0000 7fe0b6a00640 1 -- 172.21.15.28:0/2376798413 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0b003a040 con 0x7fe0a00039e0 2024-02-21T23:04:54.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.181+0000 7fe0b6a00640 1 -- 172.21.15.28:0/2376798413 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0b0039070 con 0x7fe0a00039e0 2024-02-21T23:04:54.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.181+0000 7fe0c5800640 1 -- 172.21.15.28:0/2376798413 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0c00ac330 con 0x7fe0a00039e0 2024-02-21T23:04:54.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.182+0000 7fe0c5800640 1 -- 172.21.15.28:0/2376798413 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0c01ee570 con 0x7fe0a00039e0 2024-02-21T23:04:54.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.183+0000 7fe0c5800640 1 -- 172.21.15.28:0/2376798413 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0c0067f30 con 0x7fe0a00039e0 2024-02-21T23:04:54.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.183+0000 7fe0b6a00640 1 -- 172.21.15.28:0/2376798413 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe0b0045020 con 0x7fe0a00039e0 2024-02-21T23:04:54.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.184+0000 7fe0b6a00640 1 --2- 172.21.15.28:0/2376798413 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe09c06b4d0 0x7fe09c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:54.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.184+0000 7fe0b6a00640 1 -- 172.21.15.28:0/2376798413 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe0b00ab770 con 0x7fe0a00039e0 2024-02-21T23:04:54.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.185+0000 7fe0b7e00640 1 --2- 172.21.15.28:0/2376798413 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe09c06b4d0 0x7fe09c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:54.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.186+0000 7fe0b7e00640 1 --2- 172.21.15.28:0/2376798413 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe09c06b4d0 0x7fe09c06d990 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe0c00c4930 tx=0x7fe0a8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:54.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.189+0000 7fe0b6a00640 1 -- 172.21.15.28:0/2376798413 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe0b004c070 con 0x7fe0a00039e0 2024-02-21T23:04:54.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.387+0000 7fe0c5800640 1 -- 172.21.15.28:0/2376798413 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe0c00c0960 con 0x7fe0a00039e0 2024-02-21T23:04:54.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.390+0000 7fe0b6a00640 1 -- 172.21.15.28:0/2376798413 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v52) v1 ==== 74+0+16895 (secure 0 0 0) 0x7fe0c00c0090 con 0x7fe0a00039e0 2024-02-21T23:04:54.379 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:04:54.379 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":52,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:04:53.431332+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:04:51.354130+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-02-21T23:04:52.441690+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:04:54.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.393+0000 7fe0c5800640 1 -- 172.21.15.28:0/2376798413 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe09c06b4d0 msgr2=0x7fe09c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:54.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.393+0000 7fe0c5800640 1 --2- 172.21.15.28:0/2376798413 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe09c06b4d0 0x7fe09c06d990 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe0c00c4930 tx=0x7fe0a8009040 comp rx=0 tx=0).stop 2024-02-21T23:04:54.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.393+0000 7fe0c5800640 1 -- 172.21.15.28:0/2376798413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0a00039e0 msgr2=0x7fe0c01eed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:54.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.393+0000 7fe0c5800640 1 --2- 172.21.15.28:0/2376798413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0a00039e0 0x7fe0c01eed70 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fe0b0009ac0 tx=0x7fe0b0004730 comp rx=0 tx=0).stop 2024-02-21T23:04:54.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.394+0000 7fe0c5800640 1 -- 172.21.15.28:0/2376798413 shutdown_connections 2024-02-21T23:04:54.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.394+0000 7fe0c5800640 1 --2- 172.21.15.28:0/2376798413 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe09c06b4d0 0x7fe09c06d990 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:54.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.394+0000 7fe0c5800640 1 --2- 172.21.15.28:0/2376798413 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe0c014c250 0x7fe0c00aac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:54.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.394+0000 7fe0c5800640 1 --2- 172.21.15.28:0/2376798413 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe0c0071380 0x7fe0c0071800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:54.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.394+0000 7fe0c5800640 1 --2- 172.21.15.28:0/2376798413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0a00039e0 0x7fe0c01eed70 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:54.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.394+0000 7fe0c5800640 1 -- 172.21.15.28:0/2376798413 >> 172.21.15.28:0/2376798413 conn(0x7fe0c00c67f0 msgr2=0x7fe0c00c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:54.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.394+0000 7fe0c5800640 1 -- 172.21.15.28:0/2376798413 shutdown_connections 2024-02-21T23:04:54.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.394+0000 7fe0c5800640 1 -- 172.21.15.28:0/2376798413 wait complete. 2024-02-21T23:04:54.399 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:04:54.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.621+0000 7f598be00640 1 Processor -- start 2024-02-21T23:04:54.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.622+0000 7f598be00640 1 -- start start 2024-02-21T23:04:54.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.623+0000 7f598be00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f598c0aff90 0x7f598c0c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:54.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.623+0000 7f598be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f598c0c5720 0x7f598c15f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:54.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.623+0000 7f598be00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f598c0c74c0 con 0x7f598c06dbf0 2024-02-21T23:04:54.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.623+0000 7f598be00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f598c0c7270 con 0x7f598c0c5720 2024-02-21T23:04:54.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.623+0000 7f598be00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f598c158630 con 0x7f598c0aff90 2024-02-21T23:04:54.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.623+0000 7f598a000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f598c06dbf0 0x7f598c158ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:57154/0 (socket says 172.21.15.28:57154) 2024-02-21T23:04:54.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.623+0000 7f598a000640 1 -- 172.21.15.28:0/957182894 learned_addr learned my addr 172.21.15.28:0/957182894 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:04:54.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.623+0000 7f598b400640 1 --2- 172.21.15.28:0/957182894 >> v2:172.21.15.28:3300/0 conn(0x7f598c0c5720 0x7f598c15f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:54.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.624+0000 7f598aa00640 1 --2- 172.21.15.28:0/957182894 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f598c0aff90 0x7f598c0c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:54.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.624+0000 7f5989600640 1 -- 172.21.15.28:0/957182894 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2554255078 0 0) 0x7f598c0c74c0 con 0x7f598c06dbf0 2024-02-21T23:04:54.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.624+0000 7f5989600640 1 -- 172.21.15.28:0/957182894 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5970003520 con 0x7f598c06dbf0 2024-02-21T23:04:54.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.624+0000 7f598b400640 1 -- 172.21.15.28:0/957182894 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f598c0aff90 msgr2=0x7f598c0c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:54.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.624+0000 7f598b400640 1 --2- 172.21.15.28:0/957182894 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f598c0aff90 0x7f598c0c51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:54.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.624+0000 7f598b400640 1 -- 172.21.15.28:0/957182894 >> v1:172.21.15.28:6789/0 conn(0x7f598c06dbf0 legacy=0x7f598c158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:54.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.624+0000 7f598b400640 1 -- 172.21.15.28:0/957182894 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f598c0c77a0 con 0x7f598c0c5720 2024-02-21T23:04:54.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.625+0000 7f598b400640 1 --2- 172.21.15.28:0/957182894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f598c0c5720 0x7f598c15f890 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f59840097a0 tx=0x7f598402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f635f14ec62a383 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:54.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.625+0000 7f5989600640 1 -- 172.21.15.28:0/957182894 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5984035020 con 0x7f598c0c5720 2024-02-21T23:04:54.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.625+0000 7f5989600640 1 -- 172.21.15.28:0/957182894 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5984031830 con 0x7f598c0c5720 2024-02-21T23:04:54.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.625+0000 7f5989600640 1 -- 172.21.15.28:0/957182894 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5984031c00 con 0x7f598c0c5720 2024-02-21T23:04:54.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.626+0000 7f598be00640 1 -- 172.21.15.28:0/957182894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f598c0c5720 msgr2=0x7f598c15f890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:54.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.626+0000 7f598be00640 1 --2- 172.21.15.28:0/957182894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f598c0c5720 0x7f598c15f890 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f59840097a0 tx=0x7f598402b8d0 comp rx=0 tx=0).stop 2024-02-21T23:04:54.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.626+0000 7f598be00640 1 -- 172.21.15.28:0/957182894 shutdown_connections 2024-02-21T23:04:54.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.626+0000 7f598be00640 1 --2- 172.21.15.28:0/957182894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f598c0c5720 0x7f598c15f890 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:54.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.626+0000 7f598be00640 1 --2- 172.21.15.28:0/957182894 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f598c0aff90 0x7f598c0c51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:54.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.626+0000 7f598be00640 1 -- 172.21.15.28:0/957182894 >> 172.21.15.28:0/957182894 conn(0x7f598c0ac370 msgr2=0x7f598c0ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:54.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.626+0000 7f598be00640 1 -- 172.21.15.28:0/957182894 shutdown_connections 2024-02-21T23:04:54.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.626+0000 7f598be00640 1 -- 172.21.15.28:0/957182894 wait complete. 2024-02-21T23:04:54.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.628+0000 7f598be00640 1 Processor -- start 2024-02-21T23:04:54.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.628+0000 7f598be00640 1 -- start start 2024-02-21T23:04:54.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.629+0000 7f598be00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f598c06dbf0 0x7f598c0c1360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:54.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.629+0000 7f598be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f598c0aff90 0x7f598c0c2340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:54.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.629+0000 7f598be00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f598c0c5720 0x7f598c0c2880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:54.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.629+0000 7f598be00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f598c0c8760 con 0x7f598c0aff90 2024-02-21T23:04:54.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.629+0000 7f598be00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f598c0be0c0 con 0x7f598c06dbf0 2024-02-21T23:04:54.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.629+0000 7f598be00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f598c0c88e0 con 0x7f598c0c5720 2024-02-21T23:04:54.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.629+0000 7f598a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f598c0aff90 0x7f598c0c2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:54.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.629+0000 7f598a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f598c0aff90 0x7f598c0c2340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36786/0 (socket says 172.21.15.28:36786) 2024-02-21T23:04:54.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.629+0000 7f598a000640 1 -- 172.21.15.28:0/1570255264 learned_addr learned my addr 172.21.15.28:0/1570255264 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:54.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.629+0000 7f598b400640 1 --2- 172.21.15.28:0/1570255264 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f598c0c5720 0x7f598c0c2880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:54.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.629+0000 7f598aa00640 1 --2- 172.21.15.28:0/1570255264 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f598c06dbf0 0x7f598c0c1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:54.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.630+0000 7f598a000640 1 -- 172.21.15.28:0/1570255264 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f598c0c5720 msgr2=0x7f598c0c2880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:54.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.630+0000 7f598a000640 1 --2- 172.21.15.28:0/1570255264 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f598c0c5720 0x7f598c0c2880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:54.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.630+0000 7f598a000640 1 -- 172.21.15.28:0/1570255264 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f598c06dbf0 msgr2=0x7f598c0c1360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:54.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.630+0000 7f598a000640 1 --2- 172.21.15.28:0/1570255264 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f598c06dbf0 0x7f598c0c1360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:54.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.630+0000 7f598a000640 1 -- 172.21.15.28:0/1570255264 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f598c0c3950 con 0x7f598c0aff90 2024-02-21T23:04:54.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.630+0000 7f598a000640 1 --2- 172.21.15.28:0/1570255264 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f598c0aff90 0x7f598c0c2340 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f597400c430 tx=0x7f597400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:54.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.630+0000 7f5982a00640 1 -- 172.21.15.28:0/1570255264 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5974017070 con 0x7f598c0aff90 2024-02-21T23:04:54.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.630+0000 7f5982a00640 1 -- 172.21.15.28:0/1570255264 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5974005ce0 con 0x7f598c0aff90 2024-02-21T23:04:54.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.630+0000 7f598be00640 1 -- 172.21.15.28:0/1570255264 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f598c0bd970 con 0x7f598c0aff90 2024-02-21T23:04:54.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.630+0000 7f5982a00640 1 -- 172.21.15.28:0/1570255264 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59740072f0 con 0x7f598c0aff90 2024-02-21T23:04:54.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.631+0000 7f598be00640 1 -- 172.21.15.28:0/1570255264 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f598c0c42f0 con 0x7f598c0aff90 2024-02-21T23:04:54.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.632+0000 7f598be00640 1 -- 172.21.15.28:0/1570255264 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f598c04f290 con 0x7f598c0aff90 2024-02-21T23:04:54.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.633+0000 7f5982a00640 1 -- 172.21.15.28:0/1570255264 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f597401a070 con 0x7f598c0aff90 2024-02-21T23:04:54.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.633+0000 7f5982a00640 1 --2- 172.21.15.28:0/1570255264 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f596806b520 0x7f596806d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:54.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.634+0000 7f5982a00640 1 -- 172.21.15.28:0/1570255264 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f59740897e0 con 0x7f598c0aff90 2024-02-21T23:04:54.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.634+0000 7f598aa00640 1 --2- 172.21.15.28:0/1570255264 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f596806b520 0x7f596806d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:54.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.635+0000 7f598aa00640 1 --2- 172.21.15.28:0/1570255264 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f596806b520 0x7f596806d9e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f597c009a60 tx=0x7f597c003990 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:54.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.638+0000 7f5982a00640 1 -- 172.21.15.28:0/1570255264 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f5974058540 con 0x7f598c0aff90 2024-02-21T23:04:54.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.843+0000 7f598be00640 1 -- 172.21.15.28:0/1570255264 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f598c06d9d0 con 0x7f598c0aff90 2024-02-21T23:04:54.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.845+0000 7f5982a00640 1 -- 172.21.15.28:0/1570255264 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v52) v1 ==== 74+0+16895 (secure 0 0 0) 0x7f5974057e40 con 0x7f598c0aff90 2024-02-21T23:04:54.835 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:04:54.835 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":52,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:04:53.431332+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:04:51.354130+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-02-21T23:04:52.441690+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:04:54.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.848+0000 7f598be00640 1 -- 172.21.15.28:0/1570255264 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f596806b520 msgr2=0x7f596806d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:54.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.848+0000 7f598be00640 1 --2- 172.21.15.28:0/1570255264 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f596806b520 0x7f596806d9e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f597c009a60 tx=0x7f597c003990 comp rx=0 tx=0).stop 2024-02-21T23:04:54.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.848+0000 7f598be00640 1 -- 172.21.15.28:0/1570255264 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f598c0aff90 msgr2=0x7f598c0c2340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:54.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.848+0000 7f598be00640 1 --2- 172.21.15.28:0/1570255264 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f598c0aff90 0x7f598c0c2340 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f597400c430 tx=0x7f597400c8f0 comp rx=0 tx=0).stop 2024-02-21T23:04:54.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.849+0000 7f598be00640 1 -- 172.21.15.28:0/1570255264 shutdown_connections 2024-02-21T23:04:54.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.849+0000 7f598be00640 1 --2- 172.21.15.28:0/1570255264 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f596806b520 0x7f596806d9e0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:54.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.849+0000 7f598be00640 1 --2- 172.21.15.28:0/1570255264 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f598c0c5720 0x7f598c0c2880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:54.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.849+0000 7f598be00640 1 --2- 172.21.15.28:0/1570255264 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f598c0aff90 0x7f598c0c2340 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:54.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.849+0000 7f598be00640 1 --2- 172.21.15.28:0/1570255264 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f598c06dbf0 0x7f598c0c1360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:54.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.849+0000 7f598be00640 1 -- 172.21.15.28:0/1570255264 >> 172.21.15.28:0/1570255264 conn(0x7f598c0ac370 msgr2=0x7f598c0c8ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:54.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.849+0000 7f598be00640 1 -- 172.21.15.28:0/1570255264 shutdown_connections 2024-02-21T23:04:54.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:54.849+0000 7f598be00640 1 -- 172.21.15.28:0/1570255264 wait complete. 2024-02-21T23:04:54.855 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-02-21T23:04:55.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.073+0000 7f1ac1200640 1 Processor -- start 2024-02-21T23:04:55.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.073+0000 7f1ac1200640 1 -- start start 2024-02-21T23:04:55.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.074+0000 7f1ac1200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f1abc06be80 0x7f1abc06c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:55.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.075+0000 7f1ac1200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1abc0a84b0 0x7f1abc06eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:55.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.075+0000 7f1ac1200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1abc072830 0x7f1abc158c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:55.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.075+0000 7f1ac1200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1abc155840 con 0x7f1abc0a84b0 2024-02-21T23:04:55.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.075+0000 7f1ac1200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1abc155d70 con 0x7f1abc06be80 2024-02-21T23:04:55.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.075+0000 7f1ac1200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1abc155a90 con 0x7f1abc072830 2024-02-21T23:04:55.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.075+0000 7f1ab3e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1abc0a84b0 0x7f1abc06eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:55.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.075+0000 7f1ab3e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1abc0a84b0 0x7f1abc06eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45488/0 (socket says 172.21.15.28:45488) 2024-02-21T23:04:55.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.075+0000 7f1ab3e00640 1 -- 172.21.15.28:0/3800685887 learned_addr learned my addr 172.21.15.28:0/3800685887 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:55.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.075+0000 7f1abb400640 1 --2- 172.21.15.28:0/3800685887 >> v2:172.21.15.81:3300/0 conn(0x7f1abc06be80 0x7f1abc06c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:55.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.075+0000 7f1abbe00640 1 --2- 172.21.15.28:0/3800685887 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1abc072830 0x7f1abc158c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:55.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.076+0000 7f1ab3e00640 1 -- 172.21.15.28:0/3800685887 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1abc072830 msgr2=0x7f1abc158c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:55.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.076+0000 7f1ab3e00640 1 --2- 172.21.15.28:0/3800685887 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1abc072830 0x7f1abc158c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:55.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.076+0000 7f1ab3e00640 1 -- 172.21.15.28:0/3800685887 >> v2:172.21.15.81:3300/0 conn(0x7f1abc06be80 msgr2=0x7f1abc06c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:55.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.076+0000 7f1ab3e00640 1 --2- 172.21.15.28:0/3800685887 >> v2:172.21.15.81:3300/0 conn(0x7f1abc06be80 0x7f1abc06c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:55.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.076+0000 7f1ab3e00640 1 -- 172.21.15.28:0/3800685887 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1abc158480 con 0x7f1abc0a84b0 2024-02-21T23:04:55.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.076+0000 7f1abbe00640 1 --2- 172.21.15.28:0/3800685887 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1abc072830 0x7f1abc158c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:04:55.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.076+0000 7f1abb400640 1 --2- 172.21.15.28:0/3800685887 >> v2:172.21.15.81:3300/0 conn(0x7f1abc06be80 0x7f1abc06c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:04:55.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.077+0000 7f1ab3e00640 1 --2- 172.21.15.28:0/3800685887 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1abc0a84b0 0x7f1abc06eb80 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f1abc0c8030 tx=0x7f1aa402b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dbb0ca88f8ffa5 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:55.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.077+0000 7f1abaa00640 1 -- 172.21.15.28:0/3800685887 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aa4038070 con 0x7f1abc0a84b0 2024-02-21T23:04:55.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.077+0000 7f1abaa00640 1 -- 172.21.15.28:0/3800685887 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1aa402f050 con 0x7f1abc0a84b0 2024-02-21T23:04:55.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.077+0000 7f1abaa00640 1 -- 172.21.15.28:0/3800685887 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aa4031970 con 0x7f1abc0a84b0 2024-02-21T23:04:55.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.078+0000 7f1ac1200640 1 -- 172.21.15.28:0/3800685887 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1abc0a84b0 msgr2=0x7f1abc06eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:55.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.078+0000 7f1ac1200640 1 --2- 172.21.15.28:0/3800685887 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1abc0a84b0 0x7f1abc06eb80 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f1abc0c8030 tx=0x7f1aa402b7f0 comp rx=0 tx=0).stop 2024-02-21T23:04:55.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.078+0000 7f1ac1200640 1 -- 172.21.15.28:0/3800685887 shutdown_connections 2024-02-21T23:04:55.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.078+0000 7f1ac1200640 1 --2- 172.21.15.28:0/3800685887 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1abc072830 0x7f1abc158c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:55.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.078+0000 7f1ac1200640 1 --2- 172.21.15.28:0/3800685887 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1abc0a84b0 0x7f1abc06eb80 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:55.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.078+0000 7f1ac1200640 1 --2- 172.21.15.28:0/3800685887 >> v2:172.21.15.81:3300/0 conn(0x7f1abc06be80 0x7f1abc06c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:55.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.078+0000 7f1ac1200640 1 -- 172.21.15.28:0/3800685887 >> 172.21.15.28:0/3800685887 conn(0x7f1abc071b20 msgr2=0x7f1abc071f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:55.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.078+0000 7f1ac1200640 1 -- 172.21.15.28:0/3800685887 shutdown_connections 2024-02-21T23:04:55.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.078+0000 7f1ac1200640 1 -- 172.21.15.28:0/3800685887 wait complete. 2024-02-21T23:04:55.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.079+0000 7f1ac1200640 1 Processor -- start 2024-02-21T23:04:55.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.080+0000 7f1ac1200640 1 -- start start 2024-02-21T23:04:55.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.080+0000 7f1ac1200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1abc06be80 0x7f1abc153d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:55.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.080+0000 7f1ac1200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1abc072830 0x7f1abc154240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:55.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.080+0000 7f1ac1200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1abc0a84b0 0x7f1abc0c2920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:55.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.080+0000 7f1ac1200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1abc1591c0 con 0x7f1abc072830 2024-02-21T23:04:55.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.080+0000 7f1ac1200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1abc13e280 con 0x7f1abc0a84b0 2024-02-21T23:04:55.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.081+0000 7f1ac1200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1abc13b150 con 0x7f1abc06be80 2024-02-21T23:04:55.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.081+0000 7f1ab3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1abc072830 0x7f1abc154240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:55.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.081+0000 7f1ab3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1abc072830 0x7f1abc154240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45502/0 (socket says 172.21.15.28:45502) 2024-02-21T23:04:55.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.081+0000 7f1ab3e00640 1 -- 172.21.15.28:0/2981218422 learned_addr learned my addr 172.21.15.28:0/2981218422 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:55.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.081+0000 7f1abbe00640 1 --2- 172.21.15.28:0/2981218422 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1abc0a84b0 0x7f1abc0c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:55.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.081+0000 7f1ab3e00640 1 -- 172.21.15.28:0/2981218422 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1abc06be80 msgr2=0x7f1abc153d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:55.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.081+0000 7f1ab3e00640 1 --2- 172.21.15.28:0/2981218422 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1abc06be80 0x7f1abc153d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:55.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.081+0000 7f1ab3e00640 1 -- 172.21.15.28:0/2981218422 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1abc0a84b0 msgr2=0x7f1abc0c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:55.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.081+0000 7f1ab3e00640 1 --2- 172.21.15.28:0/2981218422 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1abc0a84b0 0x7f1abc0c2920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:55.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.081+0000 7f1ab3e00640 1 -- 172.21.15.28:0/2981218422 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1abc154780 con 0x7f1abc072830 2024-02-21T23:04:55.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.081+0000 7f1abbe00640 1 --2- 172.21.15.28:0/2981218422 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1abc0a84b0 0x7f1abc0c2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:04:55.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.082+0000 7f1ab3e00640 1 --2- 172.21.15.28:0/2981218422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1abc072830 0x7f1abc154240 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f1aa40090b0 tx=0x7f1aa40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:55.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.082+0000 7f1ab3400640 1 -- 172.21.15.28:0/2981218422 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aa4042070 con 0x7f1abc072830 2024-02-21T23:04:55.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.082+0000 7f1ab3400640 1 -- 172.21.15.28:0/2981218422 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1aa403d890 con 0x7f1abc072830 2024-02-21T23:04:55.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.082+0000 7f1ab3400640 1 -- 172.21.15.28:0/2981218422 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aa4038040 con 0x7f1abc072830 2024-02-21T23:04:55.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.082+0000 7f1ac1200640 1 -- 172.21.15.28:0/2981218422 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1abc156550 con 0x7f1abc072830 2024-02-21T23:04:55.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.082+0000 7f1ac1200640 1 -- 172.21.15.28:0/2981218422 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1abc157ea0 con 0x7f1abc072830 2024-02-21T23:04:55.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.084+0000 7f1ac1200640 1 -- 172.21.15.28:0/2981218422 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1abc04f290 con 0x7f1abc072830 2024-02-21T23:04:55.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.084+0000 7f1ab3400640 1 -- 172.21.15.28:0/2981218422 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1aa4045070 con 0x7f1abc072830 2024-02-21T23:04:55.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.086+0000 7f1ab3400640 1 --2- 172.21.15.28:0/2981218422 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1a9806b500 0x7f1a9806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:55.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.086+0000 7f1ab3400640 1 -- 172.21.15.28:0/2981218422 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1aa40ac2c0 con 0x7f1abc072830 2024-02-21T23:04:55.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.086+0000 7f1abb400640 1 --2- 172.21.15.28:0/2981218422 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1a9806b500 0x7f1a9806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:55.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.087+0000 7f1abb400640 1 --2- 172.21.15.28:0/2981218422 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1a9806b500 0x7f1a9806d9c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f1aac006e80 tx=0x7f1aac008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:55.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.089+0000 7f1ab3400640 1 -- 172.21.15.28:0/2981218422 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1aa407b0c0 con 0x7f1abc072830 2024-02-21T23:04:55.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.328+0000 7f1ac1200640 1 -- 172.21.15.28:0/2981218422 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f1abc067200 con 0x7f1abc072830 2024-02-21T23:04:55.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.985+0000 7f1ab3400640 1 -- 172.21.15.28:0/2981218422 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v31) v1 ==== 118+0+0 (secure 0 0 0) 0x7f1aa407a9c0 con 0x7f1abc072830 2024-02-21T23:04:55.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.989+0000 7f1ac1200640 1 -- 172.21.15.28:0/2981218422 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1a9806b500 msgr2=0x7f1a9806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:55.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.989+0000 7f1ac1200640 1 --2- 172.21.15.28:0/2981218422 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1a9806b500 0x7f1a9806d9c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f1aac006e80 tx=0x7f1aac008040 comp rx=0 tx=0).stop 2024-02-21T23:04:55.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.989+0000 7f1ac1200640 1 -- 172.21.15.28:0/2981218422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1abc072830 msgr2=0x7f1abc154240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:55.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.989+0000 7f1ac1200640 1 --2- 172.21.15.28:0/2981218422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1abc072830 0x7f1abc154240 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f1aa40090b0 tx=0x7f1aa40046c0 comp rx=0 tx=0).stop 2024-02-21T23:04:55.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.989+0000 7f1ac1200640 1 -- 172.21.15.28:0/2981218422 shutdown_connections 2024-02-21T23:04:55.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.989+0000 7f1ac1200640 1 --2- 172.21.15.28:0/2981218422 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1a9806b500 0x7f1a9806d9c0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:55.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.989+0000 7f1ac1200640 1 --2- 172.21.15.28:0/2981218422 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1abc0a84b0 0x7f1abc0c2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:55.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.989+0000 7f1ac1200640 1 --2- 172.21.15.28:0/2981218422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1abc072830 0x7f1abc154240 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:55.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.989+0000 7f1ac1200640 1 --2- 172.21.15.28:0/2981218422 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1abc06be80 0x7f1abc153d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:55.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.989+0000 7f1ac1200640 1 -- 172.21.15.28:0/2981218422 >> 172.21.15.28:0/2981218422 conn(0x7f1abc071b20 msgr2=0x7f1abc139e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:55.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.989+0000 7f1ac1200640 1 -- 172.21.15.28:0/2981218422 shutdown_connections 2024-02-21T23:04:55.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:55.989+0000 7f1ac1200640 1 -- 172.21.15.28:0/2981218422 wait complete. 2024-02-21T23:04:56.000 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-02-21T23:04:56.000 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-02-21T23:04:56.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.226+0000 7ff907200640 1 Processor -- start 2024-02-21T23:04:56.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.226+0000 7ff907200640 1 -- start start 2024-02-21T23:04:56.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.228+0000 7ff907200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7ff90806dbc0 0x7ff90806c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:56.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.228+0000 7ff907200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff908158550 con 0x7ff90813e520 2024-02-21T23:04:56.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.228+0000 7ff907200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9081587d0 con 0x7ff90813e0c0 2024-02-21T23:04:56.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.228+0000 7ff907200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9080c89a0 con 0x7ff90806dbc0 2024-02-21T23:04:56.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.229+0000 7ff906800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff90813e520 0x7ff90806c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34440/0 (socket says 172.21.15.28:34440) 2024-02-21T23:04:56.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.229+0000 7ff906800640 1 -- 172.21.15.28:0/4038854106 learned_addr learned my addr 172.21.15.28:0/4038854106 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:04:56.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.229+0000 7ff8fd400640 1 --2- 172.21.15.28:0/4038854106 >> v2:172.21.15.81:3300/0 conn(0x7ff90806dbc0 0x7ff90806c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:56.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.229+0000 7ff905400640 1 -- 172.21.15.28:0/4038854106 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2209542982 0 0) 0x7ff900001490 con 0x7ff90813e520 2024-02-21T23:04:56.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.229+0000 7ff905400640 1 -- 172.21.15.28:0/4038854106 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8ec003520 con 0x7ff90813e520 2024-02-21T23:04:56.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.230+0000 7ff905400640 1 -- 172.21.15.28:0/4038854106 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2722262568 0 0) 0x7ff9081587d0 con 0x7ff90813e0c0 2024-02-21T23:04:56.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.230+0000 7ff905400640 1 -- 172.21.15.28:0/4038854106 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff900001490 con 0x7ff90813e0c0 2024-02-21T23:04:56.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.230+0000 7ff905400640 1 -- 172.21.15.28:0/4038854106 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 726312614 0 0) 0x7ff8ec003520 con 0x7ff90813e520 2024-02-21T23:04:56.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.230+0000 7ff905400640 1 -- 172.21.15.28:0/4038854106 >> v2:172.21.15.81:3300/0 conn(0x7ff90806dbc0 msgr2=0x7ff90806c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:56.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.230+0000 7ff905400640 1 --2- 172.21.15.28:0/4038854106 >> v2:172.21.15.81:3300/0 conn(0x7ff90806dbc0 0x7ff90806c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:56.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.230+0000 7ff905400640 1 -- 172.21.15.28:0/4038854106 >> v1:172.21.15.81:6789/0 conn(0x7ff90813e0c0 legacy=0x7ff9080c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:56.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.230+0000 7ff905400640 1 -- 172.21.15.28:0/4038854106 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff908158ab0 con 0x7ff90813e520 2024-02-21T23:04:56.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.230+0000 7ff905400640 1 -- 172.21.15.28:0/4038854106 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7ff900001bc0 con 0x7ff90813e520 2024-02-21T23:04:56.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.230+0000 7ff905400640 1 -- 172.21.15.28:0/4038854106 >> v1:172.21.15.28:6789/0 conn(0x7ff90813e520 legacy=0x7ff90806c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:56.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.230+0000 7ff905400640 1 --2- 172.21.15.28:0/4038854106 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff8ec003b40 0x7ff8ec003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:56.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.230+0000 7ff905400640 1 --2- 172.21.15.28:0/4038854106 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff8ec004800 0x7ff8ec004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:56.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.230+0000 7ff905400640 1 --2- 172.21.15.28:0/4038854106 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8ec005230 0x7ff8ec00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:56.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.230+0000 7ff905400640 1 -- 172.21.15.28:0/4038854106 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8ec00c160 con 0x7ff8ec005230 2024-02-21T23:04:56.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.230+0000 7ff905400640 1 -- 172.21.15.28:0/4038854106 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8ec00c310 con 0x7ff8ec004800 2024-02-21T23:04:56.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.230+0000 7ff905400640 1 -- 172.21.15.28:0/4038854106 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8ec00c510 con 0x7ff8ec003b40 2024-02-21T23:04:56.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.230+0000 7ff906800640 1 --2- 172.21.15.28:0/4038854106 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8ec005230 0x7ff8ec00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:56.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.231+0000 7ff8fd400640 1 --2- 172.21.15.28:0/4038854106 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff8ec004800 0x7ff8ec004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:56.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.231+0000 7ff905e00640 1 --2- 172.21.15.28:0/4038854106 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff8ec003b40 0x7ff8ec003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:56.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.231+0000 7ff906800640 1 -- 172.21.15.28:0/4038854106 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff8ec003b40 msgr2=0x7ff8ec003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:56.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.231+0000 7ff906800640 1 --2- 172.21.15.28:0/4038854106 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff8ec003b40 0x7ff8ec003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:56.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.231+0000 7ff906800640 1 -- 172.21.15.28:0/4038854106 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff8ec004800 msgr2=0x7ff8ec004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:56.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.231+0000 7ff906800640 1 --2- 172.21.15.28:0/4038854106 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff8ec004800 0x7ff8ec004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:56.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.231+0000 7ff906800640 1 -- 172.21.15.28:0/4038854106 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8ec003520 con 0x7ff8ec005230 2024-02-21T23:04:56.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.232+0000 7ff906800640 1 -- 172.21.15.28:0/4038854106 reap_dead start 2024-02-21T23:04:56.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.232+0000 7ff906800640 1 --2- 172.21.15.28:0/4038854106 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8ec005230 0x7ff8ec00b9c0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7ff90000a9d0 tx=0x7ff8ec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4270ad415d3ba289 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:56.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.232+0000 7ff905400640 1 -- 172.21.15.28:0/4038854106 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff90000a620 con 0x7ff8ec005230 2024-02-21T23:04:56.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.232+0000 7ff905400640 1 -- 172.21.15.28:0/4038854106 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff900007290 con 0x7ff8ec005230 2024-02-21T23:04:56.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.232+0000 7ff905400640 1 -- 172.21.15.28:0/4038854106 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff900007680 con 0x7ff8ec005230 2024-02-21T23:04:56.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.233+0000 7ff907200640 1 -- 172.21.15.28:0/4038854106 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8ec005230 msgr2=0x7ff8ec00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:56.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.233+0000 7ff907200640 1 --2- 172.21.15.28:0/4038854106 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8ec005230 0x7ff8ec00b9c0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7ff90000a9d0 tx=0x7ff8ec005830 comp rx=0 tx=0).stop 2024-02-21T23:04:56.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.233+0000 7ff907200640 1 -- 172.21.15.28:0/4038854106 shutdown_connections 2024-02-21T23:04:56.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.233+0000 7ff907200640 1 --2- 172.21.15.28:0/4038854106 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8ec005230 0x7ff8ec00b9c0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:56.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.233+0000 7ff907200640 1 -- 172.21.15.28:0/4038854106 >> 172.21.15.28:0/4038854106 conn(0x7ff908073110 msgr2=0x7ff9080ac490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:56.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.233+0000 7ff907200640 1 -- 172.21.15.28:0/4038854106 shutdown_connections 2024-02-21T23:04:56.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.233+0000 7ff907200640 1 -- 172.21.15.28:0/4038854106 wait complete. 2024-02-21T23:04:56.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.236+0000 7ff907200640 1 Processor -- start 2024-02-21T23:04:56.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.236+0000 7ff907200640 1 -- start start 2024-02-21T23:04:56.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.238+0000 7ff907200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff8ec005230 0x7ff908153760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:56.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.238+0000 7ff907200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff9081eddf0 0x7ff908154ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:56.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.238+0000 7ff907200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9081ee270 0x7ff908155220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:56.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.238+0000 7ff907200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9080c71b0 con 0x7ff9081ee270 2024-02-21T23:04:56.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.238+0000 7ff907200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff908159070 con 0x7ff9081eddf0 2024-02-21T23:04:56.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.238+0000 7ff907200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9080c7330 con 0x7ff8ec005230 2024-02-21T23:04:56.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.238+0000 7ff906800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9081ee270 0x7ff908155220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:56.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.238+0000 7ff906800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9081ee270 0x7ff908155220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45530/0 (socket says 172.21.15.28:45530) 2024-02-21T23:04:56.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.239+0000 7ff906800640 1 -- 172.21.15.28:0/1040216116 learned_addr learned my addr 172.21.15.28:0/1040216116 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:56.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.239+0000 7ff8fd400640 1 --2- 172.21.15.28:0/1040216116 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff9081eddf0 0x7ff908154ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:56.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.239+0000 7ff906800640 1 -- 172.21.15.28:0/1040216116 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff8ec005230 msgr2=0x7ff908153760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:56.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.239+0000 7ff906800640 1 --2- 172.21.15.28:0/1040216116 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff8ec005230 0x7ff908153760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:56.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.239+0000 7ff906800640 1 -- 172.21.15.28:0/1040216116 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff9081eddf0 msgr2=0x7ff908154ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:56.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.239+0000 7ff906800640 1 --2- 172.21.15.28:0/1040216116 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff9081eddf0 0x7ff908154ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:56.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.239+0000 7ff906800640 1 -- 172.21.15.28:0/1040216116 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9080ada70 con 0x7ff9081ee270 2024-02-21T23:04:56.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.239+0000 7ff906800640 1 --2- 172.21.15.28:0/1040216116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9081ee270 0x7ff908155220 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7ff90000a9d0 tx=0x7ff900001f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:56.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.240+0000 7ff8fea00640 1 -- 172.21.15.28:0/1040216116 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff900045070 con 0x7ff9081ee270 2024-02-21T23:04:56.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.240+0000 7ff8fea00640 1 -- 172.21.15.28:0/1040216116 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff90000a600 con 0x7ff9081ee270 2024-02-21T23:04:56.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.240+0000 7ff8fea00640 1 -- 172.21.15.28:0/1040216116 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff900007440 con 0x7ff9081ee270 2024-02-21T23:04:56.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.240+0000 7ff907200640 1 -- 172.21.15.28:0/1040216116 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9081ee730 con 0x7ff9081ee270 2024-02-21T23:04:56.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.240+0000 7ff907200640 1 -- 172.21.15.28:0/1040216116 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9080ac160 con 0x7ff9081ee270 2024-02-21T23:04:56.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.242+0000 7ff907200640 1 -- 172.21.15.28:0/1040216116 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff90804f290 con 0x7ff9081ee270 2024-02-21T23:04:56.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.244+0000 7ff8fea00640 1 -- 172.21.15.28:0/1040216116 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff900048070 con 0x7ff9081ee270 2024-02-21T23:04:56.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.244+0000 7ff8fea00640 1 --2- 172.21.15.28:0/1040216116 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff8e406b4d0 0x7ff8e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:56.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.245+0000 7ff8fea00640 1 -- 172.21.15.28:0/1040216116 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff9000abc60 con 0x7ff9081ee270 2024-02-21T23:04:56.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.245+0000 7ff905e00640 1 --2- 172.21.15.28:0/1040216116 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff8e406b4d0 0x7ff8e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:56.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.246+0000 7ff905e00640 1 --2- 172.21.15.28:0/1040216116 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff8e406b4d0 0x7ff8e406d990 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7ff8f8007fd0 tx=0x7ff8f8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:56.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.249+0000 7ff8fea00640 1 -- 172.21.15.28:0/1040216116 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ff908158550 con 0x7ff9081ee270 2024-02-21T23:04:56.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.491+0000 7ff907200640 1 -- 172.21.15.28:0/1040216116 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7ff90806d9a0 con 0x7ff9081ee270 2024-02-21T23:04:56.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:56.996+0000 7ff8fea00640 1 -- 172.21.15.28:0/1040216116 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v32) v1 ==== 122+0+0 (secure 0 0 0) 0x7ff90007a410 con 0x7ff9081ee270 2024-02-21T23:04:56.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.002+0000 7ff907200640 1 -- 172.21.15.28:0/1040216116 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff8e406b4d0 msgr2=0x7ff8e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:56.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.002+0000 7ff907200640 1 --2- 172.21.15.28:0/1040216116 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff8e406b4d0 0x7ff8e406d990 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7ff8f8007fd0 tx=0x7ff8f8009040 comp rx=0 tx=0).stop 2024-02-21T23:04:56.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.002+0000 7ff907200640 1 -- 172.21.15.28:0/1040216116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9081ee270 msgr2=0x7ff908155220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:56.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.002+0000 7ff907200640 1 --2- 172.21.15.28:0/1040216116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9081ee270 0x7ff908155220 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7ff90000a9d0 tx=0x7ff900001f10 comp rx=0 tx=0).stop 2024-02-21T23:04:56.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.002+0000 7ff907200640 1 -- 172.21.15.28:0/1040216116 shutdown_connections 2024-02-21T23:04:56.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.002+0000 7ff907200640 1 --2- 172.21.15.28:0/1040216116 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff8e406b4d0 0x7ff8e406d990 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:56.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.002+0000 7ff907200640 1 --2- 172.21.15.28:0/1040216116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9081ee270 0x7ff908155220 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:56.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.002+0000 7ff907200640 1 --2- 172.21.15.28:0/1040216116 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff9081eddf0 0x7ff908154ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:56.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.002+0000 7ff907200640 1 --2- 172.21.15.28:0/1040216116 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff8ec005230 0x7ff908153760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:56.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.002+0000 7ff907200640 1 -- 172.21.15.28:0/1040216116 >> 172.21.15.28:0/1040216116 conn(0x7ff908073110 msgr2=0x7ff908150dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:56.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.002+0000 7ff907200640 1 -- 172.21.15.28:0/1040216116 shutdown_connections 2024-02-21T23:04:56.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.002+0000 7ff907200640 1 -- 172.21.15.28:0/1040216116 wait complete. 2024-02-21T23:04:57.011 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-02-21T23:04:57.011 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:04:57.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.231+0000 7f40e0c00640 1 Processor -- start 2024-02-21T23:04:57.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.232+0000 7f40e0c00640 1 -- start start 2024-02-21T23:04:57.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.233+0000 7f40e0c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f40dc0a9930 0x7f40dc0a6f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:57.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.233+0000 7f40e0c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f40dc152bb0 0x7f40dc14fea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:57.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.233+0000 7f40e0c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40dc156c50 con 0x7f40dc0c1a40 2024-02-21T23:04:57.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.233+0000 7f40e0c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40dc072a40 con 0x7f40dc0a9930 2024-02-21T23:04:57.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.233+0000 7f40e0c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40dc156ef0 con 0x7f40dc152bb0 2024-02-21T23:04:57.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.233+0000 7f40db400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f40dc0a9930 0x7f40dc0a6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:57.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.233+0000 7f40db400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f40dc0a9930 0x7f40dc0a6f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45540/0 (socket says 172.21.15.28:45540) 2024-02-21T23:04:57.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.233+0000 7f40db400640 1 -- 172.21.15.28:0/2200510490 learned_addr learned my addr 172.21.15.28:0/2200510490 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:57.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.233+0000 7f40dbe00640 1 --2- 172.21.15.28:0/2200510490 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f40dc152bb0 0x7f40dc14fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:57.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.234+0000 7f40db400640 1 -- 172.21.15.28:0/2200510490 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f40dc152bb0 msgr2=0x7f40dc14fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:57.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.234+0000 7f40db400640 1 --2- 172.21.15.28:0/2200510490 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f40dc152bb0 0x7f40dc14fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:57.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.234+0000 7f40db400640 1 -- 172.21.15.28:0/2200510490 >> v1:172.21.15.28:6789/0 conn(0x7f40dc0c1a40 legacy=0x7f40dc0c2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:57.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.234+0000 7f40db400640 1 -- 172.21.15.28:0/2200510490 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40dc0c2830 con 0x7f40dc0a9930 2024-02-21T23:04:57.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.234+0000 7f40da000640 1 -- 172.21.15.28:0/2200510490 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 427242511 0 0) 0x7f40dc156c50 con 0x7f40dc0c1a40 2024-02-21T23:04:57.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.234+0000 7f40dbe00640 1 --2- 172.21.15.28:0/2200510490 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f40dc152bb0 0x7f40dc14fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:04:57.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.235+0000 7f40db400640 1 --2- 172.21.15.28:0/2200510490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40dc0a9930 0x7f40dc0a6f50 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f40cc009e50 tx=0x7f40cc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dc253fdcb280412 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:57.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.236+0000 7f40da000640 1 -- 172.21.15.28:0/2200510490 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40cc02ca70 con 0x7f40dc0a9930 2024-02-21T23:04:57.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.236+0000 7f40da000640 1 -- 172.21.15.28:0/2200510490 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40cc02cc10 con 0x7f40dc0a9930 2024-02-21T23:04:57.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.236+0000 7f40da000640 1 -- 172.21.15.28:0/2200510490 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40cc0343f0 con 0x7f40dc0a9930 2024-02-21T23:04:57.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.236+0000 7f40e0c00640 1 -- 172.21.15.28:0/2200510490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40dc0a9930 msgr2=0x7f40dc0a6f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:57.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.236+0000 7f40e0c00640 1 --2- 172.21.15.28:0/2200510490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40dc0a9930 0x7f40dc0a6f50 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f40cc009e50 tx=0x7f40cc02bed0 comp rx=0 tx=0).stop 2024-02-21T23:04:57.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.236+0000 7f40e0c00640 1 -- 172.21.15.28:0/2200510490 shutdown_connections 2024-02-21T23:04:57.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.236+0000 7f40e0c00640 1 --2- 172.21.15.28:0/2200510490 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f40dc152bb0 0x7f40dc14fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:57.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.236+0000 7f40e0c00640 1 --2- 172.21.15.28:0/2200510490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40dc0a9930 0x7f40dc0a6f50 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:57.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.236+0000 7f40e0c00640 1 -- 172.21.15.28:0/2200510490 >> 172.21.15.28:0/2200510490 conn(0x7f40dc0c7720 msgr2=0x7f40dc0c7b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:57.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.236+0000 7f40e0c00640 1 -- 172.21.15.28:0/2200510490 shutdown_connections 2024-02-21T23:04:57.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.237+0000 7f40e0c00640 1 -- 172.21.15.28:0/2200510490 wait complete. 2024-02-21T23:04:57.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.238+0000 7f40e0c00640 1 Processor -- start 2024-02-21T23:04:57.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.238+0000 7f40e0c00640 1 -- start start 2024-02-21T23:04:57.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.239+0000 7f40e0c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f40dc0a9930 0x7f40dc151b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:57.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.239+0000 7f40e0c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40dc0c1a40 0x7f40dc1ea450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:57.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.239+0000 7f40e0c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f40dc152bb0 0x7f40dc1ea990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:57.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.239+0000 7f40e0c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40dc0a9e90 con 0x7f40dc0c1a40 2024-02-21T23:04:57.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.239+0000 7f40e0c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40dc0a9d10 con 0x7f40dc0a9930 2024-02-21T23:04:57.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.239+0000 7f40e0c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40dc1503e0 con 0x7f40dc152bb0 2024-02-21T23:04:57.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.239+0000 7f40daa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40dc0c1a40 0x7f40dc1ea450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:57.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.239+0000 7f40daa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40dc0c1a40 0x7f40dc1ea450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45550/0 (socket says 172.21.15.28:45550) 2024-02-21T23:04:57.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.239+0000 7f40daa00640 1 -- 172.21.15.28:0/2649881480 learned_addr learned my addr 172.21.15.28:0/2649881480 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:57.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.240+0000 7f40db400640 1 --2- 172.21.15.28:0/2649881480 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f40dc0a9930 0x7f40dc151b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:57.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.240+0000 7f40dbe00640 1 --2- 172.21.15.28:0/2649881480 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f40dc152bb0 0x7f40dc1ea990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:57.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.240+0000 7f40daa00640 1 -- 172.21.15.28:0/2649881480 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f40dc152bb0 msgr2=0x7f40dc1ea990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:57.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.240+0000 7f40daa00640 1 --2- 172.21.15.28:0/2649881480 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f40dc152bb0 0x7f40dc1ea990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:57.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.240+0000 7f40daa00640 1 -- 172.21.15.28:0/2649881480 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f40dc0a9930 msgr2=0x7f40dc151b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:57.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.240+0000 7f40daa00640 1 --2- 172.21.15.28:0/2649881480 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f40dc0a9930 0x7f40dc151b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:57.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.240+0000 7f40daa00640 1 -- 172.21.15.28:0/2649881480 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40dc156010 con 0x7f40dc0c1a40 2024-02-21T23:04:57.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.240+0000 7f40dbe00640 1 --2- 172.21.15.28:0/2649881480 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f40dc152bb0 0x7f40dc1ea990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:04:57.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.240+0000 7f40daa00640 1 --2- 172.21.15.28:0/2649881480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40dc0c1a40 0x7f40dc1ea450 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f40c400c520 tx=0x7f40c400c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:57.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.240+0000 7f40d3400640 1 -- 172.21.15.28:0/2649881480 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40c4017070 con 0x7f40dc0c1a40 2024-02-21T23:04:57.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.241+0000 7f40d3400640 1 -- 172.21.15.28:0/2649881480 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40c40052d0 con 0x7f40dc0c1a40 2024-02-21T23:04:57.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.241+0000 7f40d3400640 1 -- 172.21.15.28:0/2649881480 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40c4012870 con 0x7f40dc0c1a40 2024-02-21T23:04:57.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.241+0000 7f40e0c00640 1 -- 172.21.15.28:0/2649881480 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40dc0a77f0 con 0x7f40dc0c1a40 2024-02-21T23:04:57.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.241+0000 7f40e0c00640 1 -- 172.21.15.28:0/2649881480 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40dc0a7ab0 con 0x7f40dc0c1a40 2024-02-21T23:04:57.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.242+0000 7f40e0c00640 1 -- 172.21.15.28:0/2649881480 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40dc04f290 con 0x7f40dc0c1a40 2024-02-21T23:04:57.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.243+0000 7f40d3400640 1 -- 172.21.15.28:0/2649881480 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f40c401a070 con 0x7f40dc0c1a40 2024-02-21T23:04:57.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.244+0000 7f40d3400640 1 --2- 172.21.15.28:0/2649881480 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f40b806b4d0 0x7f40b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:57.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.244+0000 7f40d3400640 1 -- 172.21.15.28:0/2649881480 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f40c40896b0 con 0x7f40dc0c1a40 2024-02-21T23:04:57.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.244+0000 7f40db400640 1 --2- 172.21.15.28:0/2649881480 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f40b806b4d0 0x7f40b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:57.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.245+0000 7f40db400640 1 --2- 172.21.15.28:0/2649881480 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f40b806b4d0 0x7f40b806d990 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f40cc035000 tx=0x7f40cc030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:57.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.248+0000 7f40d3400640 1 -- 172.21.15.28:0/2649881480 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f40c40583b0 con 0x7f40dc0c1a40 2024-02-21T23:04:57.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.487+0000 7f40e0c00640 1 -- 172.21.15.28:0/2649881480 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f40dc153820 con 0x7f40dc0c1a40 2024-02-21T23:04:57.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.488+0000 7f40d3400640 1 -- 172.21.15.28:0/2649881480 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+4935 (secure 0 0 0) 0x7f40c4057cb0 con 0x7f40dc0c1a40 2024-02-21T23:04:57.477 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:04:57.477 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":32,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5006,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3468780363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3468780363},{"type":"v1","addr":"172.21.15.28:6835","nonce":3468780363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27},{"gid":5221,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/2152160500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2152160500},{"type":"v1","addr":"172.21.15.81:6835","nonce":2152160500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27},{"gid":5224,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/442767245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":442767245},{"type":"v1","addr":"172.21.15.81:6837","nonce":442767245}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:04:53.432188+0000","modified":"2024-02-21T23:04:56.493528+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5715},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5715":{"gid":5715,"name":"a","rank":0,"incarnation":28,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6834/2803823391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2803823391},{"type":"v1","addr":"172.21.15.28:6834","nonce":2803823391}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-02-21T23:04:57.478 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 32 2024-02-21T23:04:57.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.490+0000 7f40e0c00640 1 -- 172.21.15.28:0/2649881480 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f40b806b4d0 msgr2=0x7f40b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:57.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.490+0000 7f40e0c00640 1 --2- 172.21.15.28:0/2649881480 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f40b806b4d0 0x7f40b806d990 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f40cc035000 tx=0x7f40cc030000 comp rx=0 tx=0).stop 2024-02-21T23:04:57.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.490+0000 7f40e0c00640 1 -- 172.21.15.28:0/2649881480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40dc0c1a40 msgr2=0x7f40dc1ea450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:57.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.490+0000 7f40e0c00640 1 --2- 172.21.15.28:0/2649881480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40dc0c1a40 0x7f40dc1ea450 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f40c400c520 tx=0x7f40c400c9e0 comp rx=0 tx=0).stop 2024-02-21T23:04:57.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.490+0000 7f40e0c00640 1 -- 172.21.15.28:0/2649881480 shutdown_connections 2024-02-21T23:04:57.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.490+0000 7f40e0c00640 1 --2- 172.21.15.28:0/2649881480 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f40b806b4d0 0x7f40b806d990 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:57.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.490+0000 7f40e0c00640 1 --2- 172.21.15.28:0/2649881480 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f40dc152bb0 0x7f40dc1ea990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:57.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.490+0000 7f40e0c00640 1 --2- 172.21.15.28:0/2649881480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40dc0c1a40 0x7f40dc1ea450 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:57.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.490+0000 7f40e0c00640 1 --2- 172.21.15.28:0/2649881480 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f40dc0a9930 0x7f40dc151b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:57.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.490+0000 7f40e0c00640 1 -- 172.21.15.28:0/2649881480 >> 172.21.15.28:0/2649881480 conn(0x7f40dc0c7720 msgr2=0x7f40dc070270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:57.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.491+0000 7f40e0c00640 1 -- 172.21.15.28:0/2649881480 shutdown_connections 2024-02-21T23:04:57.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.491+0000 7f40e0c00640 1 -- 172.21.15.28:0/2649881480 wait complete. 2024-02-21T23:04:57.499 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:04:57.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.717+0000 7fe269c00640 1 Processor -- start 2024-02-21T23:04:57.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.717+0000 7fe269c00640 1 -- start start 2024-02-21T23:04:57.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.719+0000 7fe269c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe2641558c0 0x7fe26406c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:57.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.719+0000 7fe269c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2641528e0 con 0x7fe2640ca050 2024-02-21T23:04:57.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.719+0000 7fe269c00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2640c86e0 con 0x7fe264152500 2024-02-21T23:04:57.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.719+0000 7fe269c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe26406c840 con 0x7fe2641558c0 2024-02-21T23:04:57.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.719+0000 7fe263400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe2640ca050 0x7fe26406bbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34468/0 (socket says 172.21.15.28:34468) 2024-02-21T23:04:57.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.719+0000 7fe263400640 1 -- 172.21.15.28:0/4117973531 learned_addr learned my addr 172.21.15.28:0/4117973531 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:04:57.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.719+0000 7fe269200640 1 --2- 172.21.15.28:0/4117973531 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe2641558c0 0x7fe26406c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:57.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.720+0000 7fe262a00640 1 -- 172.21.15.28:0/4117973531 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3306612054 0 0) 0x7fe2641528e0 con 0x7fe2640ca050 2024-02-21T23:04:57.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.720+0000 7fe262a00640 1 -- 172.21.15.28:0/4117973531 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe248003520 con 0x7fe2640ca050 2024-02-21T23:04:57.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.720+0000 7fe262a00640 1 -- 172.21.15.28:0/4117973531 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3242530855 0 0) 0x7fe2640c86e0 con 0x7fe264152500 2024-02-21T23:04:57.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.720+0000 7fe262a00640 1 -- 172.21.15.28:0/4117973531 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2641528e0 con 0x7fe264152500 2024-02-21T23:04:57.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.720+0000 7fe269200640 1 -- 172.21.15.28:0/4117973531 >> v1:172.21.15.81:6789/0 conn(0x7fe264152500 legacy=0x7fe2640bf6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:57.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.720+0000 7fe269200640 1 -- 172.21.15.28:0/4117973531 >> v1:172.21.15.28:6789/0 conn(0x7fe2640ca050 legacy=0x7fe26406bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:57.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.720+0000 7fe269200640 1 -- 172.21.15.28:0/4117973531 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2640c8980 con 0x7fe2641558c0 2024-02-21T23:04:57.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.722+0000 7fe269200640 1 --2- 172.21.15.28:0/4117973531 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe2641558c0 0x7fe26406c300 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe25c009dc0 tx=0x7fe25c02be80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4cab5fa9dbb2442 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:57.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.722+0000 7fe262a00640 1 -- 172.21.15.28:0/4117973531 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe25c039070 con 0x7fe2641558c0 2024-02-21T23:04:57.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.722+0000 7fe262a00640 1 -- 172.21.15.28:0/4117973531 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe25c02cc30 con 0x7fe2641558c0 2024-02-21T23:04:57.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.722+0000 7fe262a00640 1 -- 172.21.15.28:0/4117973531 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe25c034440 con 0x7fe2641558c0 2024-02-21T23:04:57.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.723+0000 7fe269c00640 1 -- 172.21.15.28:0/4117973531 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe2641558c0 msgr2=0x7fe26406c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:57.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.723+0000 7fe269c00640 1 --2- 172.21.15.28:0/4117973531 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe2641558c0 0x7fe26406c300 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe25c009dc0 tx=0x7fe25c02be80 comp rx=0 tx=0).stop 2024-02-21T23:04:57.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.723+0000 7fe269c00640 1 -- 172.21.15.28:0/4117973531 shutdown_connections 2024-02-21T23:04:57.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.723+0000 7fe269c00640 1 --2- 172.21.15.28:0/4117973531 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe2641558c0 0x7fe26406c300 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:57.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.723+0000 7fe269c00640 1 -- 172.21.15.28:0/4117973531 >> 172.21.15.28:0/4117973531 conn(0x7fe2640ac450 msgr2=0x7fe2640ac860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:57.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.723+0000 7fe269c00640 1 -- 172.21.15.28:0/4117973531 shutdown_connections 2024-02-21T23:04:57.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.723+0000 7fe269c00640 1 -- 172.21.15.28:0/4117973531 wait complete. 2024-02-21T23:04:57.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.726+0000 7fe269c00640 1 Processor -- start 2024-02-21T23:04:57.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.726+0000 7fe269c00640 1 -- start start 2024-02-21T23:04:57.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.727+0000 7fe269c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe2640ca050 0x7fe2641e1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:57.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.727+0000 7fe269c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe264152500 0x7fe2641f1990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:57.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.727+0000 7fe269c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2641558c0 0x7fe2641f1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:57.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.727+0000 7fe269c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2640c9220 con 0x7fe264152500 2024-02-21T23:04:57.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.727+0000 7fe269c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2640c8f40 con 0x7fe2641558c0 2024-02-21T23:04:57.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.728+0000 7fe269c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe264155d10 con 0x7fe2640ca050 2024-02-21T23:04:57.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.728+0000 7fe263400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe264152500 0x7fe2641f1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:57.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.728+0000 7fe263400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe264152500 0x7fe2641f1990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45560/0 (socket says 172.21.15.28:45560) 2024-02-21T23:04:57.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.728+0000 7fe263400640 1 -- 172.21.15.28:0/3563646818 learned_addr learned my addr 172.21.15.28:0/3563646818 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:57.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.728+0000 7fe263e00640 1 --2- 172.21.15.28:0/3563646818 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe2640ca050 0x7fe2641e1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:57.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.728+0000 7fe269200640 1 --2- 172.21.15.28:0/3563646818 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2641558c0 0x7fe2641f1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:57.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.728+0000 7fe263400640 1 -- 172.21.15.28:0/3563646818 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe2640ca050 msgr2=0x7fe2641e1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:57.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.728+0000 7fe263400640 1 --2- 172.21.15.28:0/3563646818 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe2640ca050 0x7fe2641e1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:57.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.728+0000 7fe263400640 1 -- 172.21.15.28:0/3563646818 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2641558c0 msgr2=0x7fe2641f1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:57.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.728+0000 7fe263400640 1 --2- 172.21.15.28:0/3563646818 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2641558c0 0x7fe2641f1ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:57.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.728+0000 7fe263400640 1 -- 172.21.15.28:0/3563646818 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2641f03e0 con 0x7fe264152500 2024-02-21T23:04:57.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.728+0000 7fe269200640 1 --2- 172.21.15.28:0/3563646818 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2641558c0 0x7fe2641f1ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:04:57.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.729+0000 7fe263400640 1 --2- 172.21.15.28:0/3563646818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe264152500 0x7fe2641f1990 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fe25000c460 tx=0x7fe25000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:57.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.729+0000 7fe25be00640 1 -- 172.21.15.28:0/3563646818 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe250017070 con 0x7fe264152500 2024-02-21T23:04:57.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.729+0000 7fe25be00640 1 -- 172.21.15.28:0/3563646818 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe250005da0 con 0x7fe264152500 2024-02-21T23:04:57.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.729+0000 7fe25be00640 1 -- 172.21.15.28:0/3563646818 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2500072b0 con 0x7fe264152500 2024-02-21T23:04:57.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.729+0000 7fe269c00640 1 -- 172.21.15.28:0/3563646818 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2641efe90 con 0x7fe264152500 2024-02-21T23:04:57.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.729+0000 7fe269c00640 1 -- 172.21.15.28:0/3563646818 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2640c71e0 con 0x7fe264152500 2024-02-21T23:04:57.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.731+0000 7fe269c00640 1 -- 172.21.15.28:0/3563646818 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe26404f290 con 0x7fe264152500 2024-02-21T23:04:57.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.731+0000 7fe25be00640 1 -- 172.21.15.28:0/3563646818 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe25001a070 con 0x7fe264152500 2024-02-21T23:04:57.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.732+0000 7fe25be00640 1 --2- 172.21.15.28:0/3563646818 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe24406b4d0 0x7fe24406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:57.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.732+0000 7fe25be00640 1 -- 172.21.15.28:0/3563646818 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe250089600 con 0x7fe264152500 2024-02-21T23:04:57.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.733+0000 7fe263e00640 1 --2- 172.21.15.28:0/3563646818 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe24406b4d0 0x7fe24406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:57.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.734+0000 7fe263e00640 1 --2- 172.21.15.28:0/3563646818 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe24406b4d0 0x7fe24406d990 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fe2540011e0 tx=0x7fe254009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:57.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:57.737+0000 7fe25be00640 1 -- 172.21.15.28:0/3563646818 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe2641f03e0 con 0x7fe264152500 2024-02-21T23:04:58.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.067+0000 7fe269c00640 1 -- 172.21.15.28:0/3563646818 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe264150cb0 con 0x7fe264152500 2024-02-21T23:04:58.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.069+0000 7fe25be00640 1 -- 172.21.15.28:0/3563646818 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v53) v1 ==== 74+0+16882 (secure 0 0 0) 0x7fe250058380 con 0x7fe264152500 2024-02-21T23:04:58.059 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:04:58.059 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":53,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:04:54.959306+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:04:51.354130+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-02-21T23:04:52.441690+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:04:58.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.072+0000 7fe269c00640 1 -- 172.21.15.28:0/3563646818 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe24406b4d0 msgr2=0x7fe24406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:58.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.072+0000 7fe269c00640 1 --2- 172.21.15.28:0/3563646818 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe24406b4d0 0x7fe24406d990 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fe2540011e0 tx=0x7fe254009040 comp rx=0 tx=0).stop 2024-02-21T23:04:58.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.072+0000 7fe269c00640 1 -- 172.21.15.28:0/3563646818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe264152500 msgr2=0x7fe2641f1990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:58.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.072+0000 7fe269c00640 1 --2- 172.21.15.28:0/3563646818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe264152500 0x7fe2641f1990 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fe25000c460 tx=0x7fe25000c920 comp rx=0 tx=0).stop 2024-02-21T23:04:58.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.072+0000 7fe269c00640 1 -- 172.21.15.28:0/3563646818 shutdown_connections 2024-02-21T23:04:58.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.072+0000 7fe269c00640 1 --2- 172.21.15.28:0/3563646818 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe24406b4d0 0x7fe24406d990 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:58.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.072+0000 7fe269c00640 1 --2- 172.21.15.28:0/3563646818 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2641558c0 0x7fe2641f1ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:58.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.072+0000 7fe269c00640 1 --2- 172.21.15.28:0/3563646818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe264152500 0x7fe2641f1990 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:58.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.072+0000 7fe269c00640 1 --2- 172.21.15.28:0/3563646818 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe2640ca050 0x7fe2641e1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:58.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.072+0000 7fe269c00640 1 -- 172.21.15.28:0/3563646818 >> 172.21.15.28:0/3563646818 conn(0x7fe2640ac450 msgr2=0x7fe2641514b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:58.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.072+0000 7fe269c00640 1 -- 172.21.15.28:0/3563646818 shutdown_connections 2024-02-21T23:04:58.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.072+0000 7fe269c00640 1 -- 172.21.15.28:0/3563646818 wait complete. 2024-02-21T23:04:58.080 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-02-21T23:04:58.080 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-21T23:04:58.347 INFO:tasks.ceph.mds.a.smithi028.stderr:2024-02-21T23:04:58.358+0000 7fabf3e00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-21T23:04:58.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.372+0000 7f806cc00640 1 Processor -- start 2024-02-21T23:04:58.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.372+0000 7f806cc00640 1 -- start start 2024-02-21T23:04:58.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.374+0000 7f806cc00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f8068150450 0x7f80680a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:58.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.374+0000 7f806cc00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f806806ee40 0x7f80681536f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:58.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.374+0000 7f806cc00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8068156af0 0x7f806806dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:58.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.374+0000 7f806cc00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80680c22c0 con 0x7f806806ee40 2024-02-21T23:04:58.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.374+0000 7f806cc00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8068156660 con 0x7f8068150450 2024-02-21T23:04:58.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.374+0000 7f806cc00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80680c2760 con 0x7f8068156af0 2024-02-21T23:04:58.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.374+0000 7f805fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f806806ee40 0x7f80681536f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:58.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.374+0000 7f805fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f806806ee40 0x7f80681536f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45566/0 (socket says 172.21.15.28:45566) 2024-02-21T23:04:58.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.374+0000 7f805fe00640 1 -- 172.21.15.28:0/3722324794 learned_addr learned my addr 172.21.15.28:0/3722324794 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:58.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.375+0000 7f8067400640 1 --2- 172.21.15.28:0/3722324794 >> v2:172.21.15.81:3300/0 conn(0x7f8068150450 0x7f80680a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:58.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.375+0000 7f8067e00640 1 --2- 172.21.15.28:0/3722324794 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8068156af0 0x7f806806dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:58.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.375+0000 7f805fe00640 1 -- 172.21.15.28:0/3722324794 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8068156af0 msgr2=0x7f806806dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:58.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.375+0000 7f805fe00640 1 --2- 172.21.15.28:0/3722324794 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8068156af0 0x7f806806dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:58.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.375+0000 7f805fe00640 1 -- 172.21.15.28:0/3722324794 >> v2:172.21.15.81:3300/0 conn(0x7f8068150450 msgr2=0x7f80680a5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:58.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.375+0000 7f805fe00640 1 --2- 172.21.15.28:0/3722324794 >> v2:172.21.15.81:3300/0 conn(0x7f8068150450 0x7f80680a5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:58.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.375+0000 7f805fe00640 1 -- 172.21.15.28:0/3722324794 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8068153c30 con 0x7f806806ee40 2024-02-21T23:04:58.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.376+0000 7f805fe00640 1 --2- 172.21.15.28:0/3722324794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f806806ee40 0x7f80681536f0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f806806f600 tx=0x7f805002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6e8909bc13c17d8 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:58.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.376+0000 7f8066a00640 1 -- 172.21.15.28:0/3722324794 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8050035020 con 0x7f806806ee40 2024-02-21T23:04:58.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.376+0000 7f8066a00640 1 -- 172.21.15.28:0/3722324794 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8050031870 con 0x7f806806ee40 2024-02-21T23:04:58.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.376+0000 7f8066a00640 1 -- 172.21.15.28:0/3722324794 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8050031c40 con 0x7f806806ee40 2024-02-21T23:04:58.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.377+0000 7f806cc00640 1 -- 172.21.15.28:0/3722324794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f806806ee40 msgr2=0x7f80681536f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:58.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.377+0000 7f806cc00640 1 --2- 172.21.15.28:0/3722324794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f806806ee40 0x7f80681536f0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f806806f600 tx=0x7f805002b910 comp rx=0 tx=0).stop 2024-02-21T23:04:58.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.377+0000 7f806cc00640 1 -- 172.21.15.28:0/3722324794 shutdown_connections 2024-02-21T23:04:58.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.377+0000 7f806cc00640 1 --2- 172.21.15.28:0/3722324794 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8068156af0 0x7f806806dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:58.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.377+0000 7f806cc00640 1 --2- 172.21.15.28:0/3722324794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f806806ee40 0x7f80681536f0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:58.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.377+0000 7f806cc00640 1 --2- 172.21.15.28:0/3722324794 >> v2:172.21.15.81:3300/0 conn(0x7f8068150450 0x7f80680a5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:58.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.377+0000 7f806cc00640 1 -- 172.21.15.28:0/3722324794 >> 172.21.15.28:0/3722324794 conn(0x7f806806b630 msgr2=0x7f80680c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:58.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.377+0000 7f806cc00640 1 -- 172.21.15.28:0/3722324794 shutdown_connections 2024-02-21T23:04:58.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.377+0000 7f806cc00640 1 -- 172.21.15.28:0/3722324794 wait complete. 2024-02-21T23:04:58.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.379+0000 7f806cc00640 1 Processor -- start 2024-02-21T23:04:58.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.379+0000 7f806cc00640 1 -- start start 2024-02-21T23:04:58.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.380+0000 7f806cc00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f806806ee40 0x7f80681557c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:58.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.380+0000 7f806cc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8068150450 0x7f80681eab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:58.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.380+0000 7f806cc00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8068156af0 0x7f80681eb070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:58.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.380+0000 7f806cc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80680c2cb0 con 0x7f8068150450 2024-02-21T23:04:58.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.380+0000 7f806cc00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80680c29d0 con 0x7f806806ee40 2024-02-21T23:04:58.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.380+0000 7f806cc00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8068156f80 con 0x7f8068156af0 2024-02-21T23:04:58.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.381+0000 7f805fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8068150450 0x7f80681eab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:58.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.381+0000 7f805fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8068150450 0x7f80681eab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45584/0 (socket says 172.21.15.28:45584) 2024-02-21T23:04:58.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.381+0000 7f805fe00640 1 -- 172.21.15.28:0/515585286 learned_addr learned my addr 172.21.15.28:0/515585286 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:58.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.381+0000 7f8067400640 1 --2- 172.21.15.28:0/515585286 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f806806ee40 0x7f80681557c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:58.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.381+0000 7f8067e00640 1 --2- 172.21.15.28:0/515585286 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8068156af0 0x7f80681eb070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:58.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.381+0000 7f805fe00640 1 -- 172.21.15.28:0/515585286 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8068156af0 msgr2=0x7f80681eb070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:58.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.381+0000 7f805fe00640 1 --2- 172.21.15.28:0/515585286 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8068156af0 0x7f80681eb070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:58.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.381+0000 7f805fe00640 1 -- 172.21.15.28:0/515585286 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f806806ee40 msgr2=0x7f80681557c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:58.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.381+0000 7f805fe00640 1 --2- 172.21.15.28:0/515585286 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f806806ee40 0x7f80681557c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:58.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.381+0000 7f805fe00640 1 -- 172.21.15.28:0/515585286 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80681497e0 con 0x7f8068150450 2024-02-21T23:04:58.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.381+0000 7f805fe00640 1 --2- 172.21.15.28:0/515585286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8068150450 0x7f80681eab30 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f80500097c0 tx=0x7f80500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:58.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.382+0000 7f805f400640 1 -- 172.21.15.28:0/515585286 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8050040070 con 0x7f8068150450 2024-02-21T23:04:58.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.382+0000 7f805f400640 1 -- 172.21.15.28:0/515585286 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8050030d00 con 0x7f8068150450 2024-02-21T23:04:58.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.382+0000 7f806cc00640 1 -- 172.21.15.28:0/515585286 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80680720c0 con 0x7f8068150450 2024-02-21T23:04:58.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.382+0000 7f805f400640 1 -- 172.21.15.28:0/515585286 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8050035040 con 0x7f8068150450 2024-02-21T23:04:58.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.382+0000 7f806cc00640 1 -- 172.21.15.28:0/515585286 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8068072680 con 0x7f8068150450 2024-02-21T23:04:58.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.383+0000 7f806cc00640 1 -- 172.21.15.28:0/515585286 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f806804f290 con 0x7f8068150450 2024-02-21T23:04:58.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.384+0000 7f805f400640 1 -- 172.21.15.28:0/515585286 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f805004b060 con 0x7f8068150450 2024-02-21T23:04:58.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.385+0000 7f805f400640 1 --2- 172.21.15.28:0/515585286 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f804406b4d0 0x7f804406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:58.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.385+0000 7f805f400640 1 -- 172.21.15.28:0/515585286 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f80500ac1a0 con 0x7f8068150450 2024-02-21T23:04:58.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.385+0000 7f8067400640 1 --2- 172.21.15.28:0/515585286 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f804406b4d0 0x7f804406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:58.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.387+0000 7f8067400640 1 --2- 172.21.15.28:0/515585286 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f804406b4d0 0x7f804406d990 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f8058006fd0 tx=0x7f8058008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:58.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.389+0000 7f805f400640 1 -- 172.21.15.28:0/515585286 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f805007adf0 con 0x7f8068150450 2024-02-21T23:04:58.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.575+0000 7f806cc00640 1 -- 172.21.15.28:0/515585286 --> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f80680c1f10 con 0x7f804406b4d0 2024-02-21T23:04:58.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.595+0000 7f805f400640 1 -- 172.21.15.28:0/515585286 <== mgr.4101 v2:172.21.15.81:6832/1848538577 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252006 (secure 0 0 0) 0x7f80680c1f10 con 0x7f804406b4d0 2024-02-21T23:04:58.589 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:04:58.591 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-21T23:04:58.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.601+0000 7f806cc00640 1 -- 172.21.15.28:0/515585286 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f804406b4d0 msgr2=0x7f804406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:58.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.601+0000 7f806cc00640 1 --2- 172.21.15.28:0/515585286 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f804406b4d0 0x7f804406d990 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f8058006fd0 tx=0x7f8058008040 comp rx=0 tx=0).stop 2024-02-21T23:04:58.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.601+0000 7f806cc00640 1 -- 172.21.15.28:0/515585286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8068150450 msgr2=0x7f80681eab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:58.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.601+0000 7f806cc00640 1 --2- 172.21.15.28:0/515585286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8068150450 0x7f80681eab30 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f80500097c0 tx=0x7f80500046c0 comp rx=0 tx=0).stop 2024-02-21T23:04:58.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.602+0000 7f806cc00640 1 -- 172.21.15.28:0/515585286 shutdown_connections 2024-02-21T23:04:58.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.602+0000 7f806cc00640 1 --2- 172.21.15.28:0/515585286 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f804406b4d0 0x7f804406d990 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:58.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.602+0000 7f806cc00640 1 --2- 172.21.15.28:0/515585286 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8068156af0 0x7f80681eb070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:58.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.602+0000 7f806cc00640 1 --2- 172.21.15.28:0/515585286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8068150450 0x7f80681eab30 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:58.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.602+0000 7f806cc00640 1 --2- 172.21.15.28:0/515585286 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f806806ee40 0x7f80681557c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:58.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.602+0000 7f806cc00640 1 -- 172.21.15.28:0/515585286 >> 172.21.15.28:0/515585286 conn(0x7f806806b630 msgr2=0x7f806806dff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:58.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.602+0000 7f806cc00640 1 -- 172.21.15.28:0/515585286 shutdown_connections 2024-02-21T23:04:58.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.602+0000 7f806cc00640 1 -- 172.21.15.28:0/515585286 wait complete. 2024-02-21T23:04:58.611 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":119,"stamp":"2024-02-21T23:04:56.711376+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":135,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237892,"kb_used_data":5048,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754736828,"statfs":{"total":773094113280,"available":772850511872,"internally_reserved":0,"allocated":5169152,"data_stored":3412156,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":46,"apply_latency_ms":46,"commit_latency_ns":46000000,"apply_latency_ns":46000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.781034"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-02-21T23:04:53.989051+0000","last_change":"2024-02-21T23:04:53.986963+0000","last_active":"2024-02-21T23:04:53.989051+0000","last_peered":"2024-02-21T23:04:53.989051+0000","last_clean":"2024-02-21T23:04:53.989051+0000","last_became_active":"2024-02-21T23:04:53.986659+0000","last_became_peered":"2024-02-21T23:04:53.986659+0000","last_unstale":"2024-02-21T23:04:53.989051+0000","last_undegraded":"2024-02-21T23:04:53.989051+0000","last_fullsized":"2024-02-21T23:04:53.989051+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:45:50.638257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-02-21T23:04:53.995829+0000","last_change":"2024-02-21T23:04:53.992488+0000","last_active":"2024-02-21T23:04:53.995829+0000","last_peered":"2024-02-21T23:04:53.995829+0000","last_clean":"2024-02-21T23:04:53.995829+0000","last_became_active":"2024-02-21T23:04:53.992187+0000","last_became_peered":"2024-02-21T23:04:53.992187+0000","last_unstale":"2024-02-21T23:04:53.995829+0000","last_undegraded":"2024-02-21T23:04:53.995829+0000","last_fullsized":"2024-02-21T23:04:53.995829+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:48:17.190667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.979425+0000","last_change":"2024-02-21T23:04:54.002212+0000","last_active":"2024-02-21T23:04:54.979425+0000","last_peered":"2024-02-21T23:04:54.979425+0000","last_clean":"2024-02-21T23:04:54.979425+0000","last_became_active":"2024-02-21T23:04:54.001921+0000","last_became_peered":"2024-02-21T23:04:54.001921+0000","last_unstale":"2024-02-21T23:04:54.979425+0000","last_undegraded":"2024-02-21T23:04:54.979425+0000","last_fullsized":"2024-02-21T23:04:54.979425+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:43:19.009080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-02-21T23:04:53.998407+0000","last_change":"2024-02-21T23:04:53.994828+0000","last_active":"2024-02-21T23:04:53.998407+0000","last_peered":"2024-02-21T23:04:53.998407+0000","last_clean":"2024-02-21T23:04:53.998407+0000","last_became_active":"2024-02-21T23:04:53.993942+0000","last_became_peered":"2024-02-21T23:04:53.993942+0000","last_unstale":"2024-02-21T23:04:53.998407+0000","last_undegraded":"2024-02-21T23:04:53.998407+0000","last_fullsized":"2024-02-21T23:04:53.998407+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:39:16.498706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:55.510609+0000","last_change":"2024-02-21T23:04:53.984836+0000","last_active":"2024-02-21T23:04:55.510609+0000","last_peered":"2024-02-21T23:04:55.510609+0000","last_clean":"2024-02-21T23:04:55.510609+0000","last_became_active":"2024-02-21T23:04:53.984493+0000","last_became_peered":"2024-02-21T23:04:53.984493+0000","last_unstale":"2024-02-21T23:04:55.510609+0000","last_undegraded":"2024-02-21T23:04:55.510609+0000","last_fullsized":"2024-02-21T23:04:55.510609+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:45:50.638257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-02-21T23:04:53.987322+0000","last_change":"2024-02-21T23:04:53.984765+0000","last_active":"2024-02-21T23:04:53.987322+0000","last_peered":"2024-02-21T23:04:53.987322+0000","last_clean":"2024-02-21T23:04:53.987322+0000","last_became_active":"2024-02-21T23:04:53.984432+0000","last_became_peered":"2024-02-21T23:04:53.984432+0000","last_unstale":"2024-02-21T23:04:53.987322+0000","last_undegraded":"2024-02-21T23:04:53.987322+0000","last_fullsized":"2024-02-21T23:04:53.987322+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:39:32.177358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.977895+0000","last_change":"2024-02-21T23:04:54.003595+0000","last_active":"2024-02-21T23:04:54.977895+0000","last_peered":"2024-02-21T23:04:54.977895+0000","last_clean":"2024-02-21T23:04:54.977895+0000","last_became_active":"2024-02-21T23:04:54.003085+0000","last_became_peered":"2024-02-21T23:04:54.003085+0000","last_unstale":"2024-02-21T23:04:54.977895+0000","last_undegraded":"2024-02-21T23:04:54.977895+0000","last_fullsized":"2024-02-21T23:04:54.977895+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:20:56.263919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.981592+0000","last_change":"2024-02-21T23:04:53.991959+0000","last_active":"2024-02-21T23:04:54.981592+0000","last_peered":"2024-02-21T23:04:54.981592+0000","last_clean":"2024-02-21T23:04:54.981592+0000","last_became_active":"2024-02-21T23:04:53.988135+0000","last_became_peered":"2024-02-21T23:04:53.988135+0000","last_unstale":"2024-02-21T23:04:54.981592+0000","last_undegraded":"2024-02-21T23:04:54.981592+0000","last_fullsized":"2024-02-21T23:04:54.981592+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:28:35.805544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.982863+0000","last_change":"2024-02-21T23:04:53.982813+0000","last_active":"2024-02-21T23:04:54.982863+0000","last_peered":"2024-02-21T23:04:54.982863+0000","last_clean":"2024-02-21T23:04:54.982863+0000","last_became_active":"2024-02-21T23:04:53.982515+0000","last_became_peered":"2024-02-21T23:04:53.982515+0000","last_unstale":"2024-02-21T23:04:54.982863+0000","last_undegraded":"2024-02-21T23:04:54.982863+0000","last_fullsized":"2024-02-21T23:04:54.982863+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:03:03.310447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-02-21T23:04:53.986722+0000","last_change":"2024-02-21T23:04:53.982276+0000","last_active":"2024-02-21T23:04:53.986722+0000","last_peered":"2024-02-21T23:04:53.986722+0000","last_clean":"2024-02-21T23:04:53.986722+0000","last_became_active":"2024-02-21T23:04:53.981735+0000","last_became_peered":"2024-02-21T23:04:53.981735+0000","last_unstale":"2024-02-21T23:04:53.986722+0000","last_undegraded":"2024-02-21T23:04:53.986722+0000","last_fullsized":"2024-02-21T23:04:53.986722+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:47:28.385794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-02-21T23:04:53.994969+0000","last_change":"2024-02-21T23:04:53.992336+0000","last_active":"2024-02-21T23:04:53.994969+0000","last_peered":"2024-02-21T23:04:53.994969+0000","last_clean":"2024-02-21T23:04:53.994969+0000","last_became_active":"2024-02-21T23:04:53.992068+0000","last_became_peered":"2024-02-21T23:04:53.992068+0000","last_unstale":"2024-02-21T23:04:53.994969+0000","last_undegraded":"2024-02-21T23:04:53.994969+0000","last_fullsized":"2024-02-21T23:04:53.994969+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:49:04.519126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.982368+0000","last_change":"2024-02-21T23:04:53.991879+0000","last_active":"2024-02-21T23:04:54.982368+0000","last_peered":"2024-02-21T23:04:54.982368+0000","last_clean":"2024-02-21T23:04:54.982368+0000","last_became_active":"2024-02-21T23:04:53.985749+0000","last_became_peered":"2024-02-21T23:04:53.985749+0000","last_unstale":"2024-02-21T23:04:54.982368+0000","last_undegraded":"2024-02-21T23:04:54.982368+0000","last_fullsized":"2024-02-21T23:04:54.982368+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:21:11.752620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-02-21T23:04:53.986913+0000","last_change":"2024-02-21T23:04:53.982139+0000","last_active":"2024-02-21T23:04:53.986913+0000","last_peered":"2024-02-21T23:04:53.986913+0000","last_clean":"2024-02-21T23:04:53.986913+0000","last_became_active":"2024-02-21T23:04:53.981800+0000","last_became_peered":"2024-02-21T23:04:53.981800+0000","last_unstale":"2024-02-21T23:04:53.986913+0000","last_undegraded":"2024-02-21T23:04:53.986913+0000","last_fullsized":"2024-02-21T23:04:53.986913+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:00:21.254168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.978586+0000","last_change":"2024-02-21T23:04:54.003361+0000","last_active":"2024-02-21T23:04:54.978586+0000","last_peered":"2024-02-21T23:04:54.978586+0000","last_clean":"2024-02-21T23:04:54.978586+0000","last_became_active":"2024-02-21T23:04:54.002829+0000","last_became_peered":"2024-02-21T23:04:54.002829+0000","last_unstale":"2024-02-21T23:04:54.978586+0000","last_undegraded":"2024-02-21T23:04:54.978586+0000","last_fullsized":"2024-02-21T23:04:54.978586+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:20:06.730366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-02-21T23:04:53.990505+0000","last_change":"2024-02-21T23:04:53.981560+0000","last_active":"2024-02-21T23:04:53.990505+0000","last_peered":"2024-02-21T23:04:53.990505+0000","last_clean":"2024-02-21T23:04:53.990505+0000","last_became_active":"2024-02-21T23:04:53.981260+0000","last_became_peered":"2024-02-21T23:04:53.981260+0000","last_unstale":"2024-02-21T23:04:53.990505+0000","last_undegraded":"2024-02-21T23:04:53.990505+0000","last_fullsized":"2024-02-21T23:04:53.990505+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:55:24.090577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-02-21T23:04:53.982444+0000","last_change":"2024-02-21T23:04:53.981072+0000","last_active":"2024-02-21T23:04:53.982444+0000","last_peered":"2024-02-21T23:04:53.982444+0000","last_clean":"2024-02-21T23:04:53.982444+0000","last_became_active":"2024-02-21T23:04:53.979351+0000","last_became_peered":"2024-02-21T23:04:53.979351+0000","last_unstale":"2024-02-21T23:04:53.982444+0000","last_undegraded":"2024-02-21T23:04:53.982444+0000","last_fullsized":"2024-02-21T23:04:53.982444+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:55:28.435901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.980429+0000","last_change":"2024-02-21T23:04:53.990161+0000","last_active":"2024-02-21T23:04:54.980429+0000","last_peered":"2024-02-21T23:04:54.980429+0000","last_clean":"2024-02-21T23:04:54.980429+0000","last_became_active":"2024-02-21T23:04:53.989849+0000","last_became_peered":"2024-02-21T23:04:53.989849+0000","last_unstale":"2024-02-21T23:04:54.980429+0000","last_undegraded":"2024-02-21T23:04:54.980429+0000","last_fullsized":"2024-02-21T23:04:54.980429+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:57:04.392889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-02-21T23:04:53.995850+0000","last_change":"2024-02-21T23:04:53.992512+0000","last_active":"2024-02-21T23:04:53.995850+0000","last_peered":"2024-02-21T23:04:53.995850+0000","last_clean":"2024-02-21T23:04:53.995850+0000","last_became_active":"2024-02-21T23:04:53.992255+0000","last_became_peered":"2024-02-21T23:04:53.992255+0000","last_unstale":"2024-02-21T23:04:53.995850+0000","last_undegraded":"2024-02-21T23:04:53.995850+0000","last_fullsized":"2024-02-21T23:04:53.995850+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:10:31.924664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.979230+0000","last_change":"2024-02-21T23:04:53.985272+0000","last_active":"2024-02-21T23:04:54.979230+0000","last_peered":"2024-02-21T23:04:54.979230+0000","last_clean":"2024-02-21T23:04:54.979230+0000","last_became_active":"2024-02-21T23:04:53.984967+0000","last_became_peered":"2024-02-21T23:04:53.984967+0000","last_unstale":"2024-02-21T23:04:54.979230+0000","last_undegraded":"2024-02-21T23:04:54.979230+0000","last_fullsized":"2024-02-21T23:04:54.979230+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:10:20.341856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.980991+0000","last_change":"2024-02-21T23:04:53.985751+0000","last_active":"2024-02-21T23:04:54.980991+0000","last_peered":"2024-02-21T23:04:54.980991+0000","last_clean":"2024-02-21T23:04:54.980991+0000","last_became_active":"2024-02-21T23:04:53.983212+0000","last_became_peered":"2024-02-21T23:04:53.983212+0000","last_unstale":"2024-02-21T23:04:54.980991+0000","last_undegraded":"2024-02-21T23:04:54.980991+0000","last_fullsized":"2024-02-21T23:04:54.980991+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:16:59.134374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.980617+0000","last_change":"2024-02-21T23:04:53.984599+0000","last_active":"2024-02-21T23:04:54.980617+0000","last_peered":"2024-02-21T23:04:54.980617+0000","last_clean":"2024-02-21T23:04:54.980617+0000","last_became_active":"2024-02-21T23:04:53.983405+0000","last_became_peered":"2024-02-21T23:04:53.983405+0000","last_unstale":"2024-02-21T23:04:54.980617+0000","last_undegraded":"2024-02-21T23:04:54.980617+0000","last_fullsized":"2024-02-21T23:04:54.980617+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:59:35.227750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.982096+0000","last_change":"2024-02-21T23:04:53.988824+0000","last_active":"2024-02-21T23:04:54.982096+0000","last_peered":"2024-02-21T23:04:54.982096+0000","last_clean":"2024-02-21T23:04:54.982096+0000","last_became_active":"2024-02-21T23:04:53.988190+0000","last_became_peered":"2024-02-21T23:04:53.988190+0000","last_unstale":"2024-02-21T23:04:54.982096+0000","last_undegraded":"2024-02-21T23:04:54.982096+0000","last_fullsized":"2024-02-21T23:04:54.982096+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:59:41.759091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-02-21T23:04:53.994679+0000","last_change":"2024-02-21T23:04:53.989836+0000","last_active":"2024-02-21T23:04:53.994679+0000","last_peered":"2024-02-21T23:04:53.994679+0000","last_clean":"2024-02-21T23:04:53.994679+0000","last_became_active":"2024-02-21T23:04:53.989511+0000","last_became_peered":"2024-02-21T23:04:53.989511+0000","last_unstale":"2024-02-21T23:04:53.994679+0000","last_undegraded":"2024-02-21T23:04:53.994679+0000","last_fullsized":"2024-02-21T23:04:53.994679+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:58:25.719654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.977619+0000","last_change":"2024-02-21T23:04:53.980319+0000","last_active":"2024-02-21T23:04:54.977619+0000","last_peered":"2024-02-21T23:04:54.977619+0000","last_clean":"2024-02-21T23:04:54.977619+0000","last_became_active":"2024-02-21T23:04:53.980056+0000","last_became_peered":"2024-02-21T23:04:53.980056+0000","last_unstale":"2024-02-21T23:04:54.977619+0000","last_undegraded":"2024-02-21T23:04:54.977619+0000","last_fullsized":"2024-02-21T23:04:54.977619+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:27:08.923372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.980833+0000","last_change":"2024-02-21T23:04:53.994043+0000","last_active":"2024-02-21T23:04:54.980833+0000","last_peered":"2024-02-21T23:04:54.980833+0000","last_clean":"2024-02-21T23:04:54.980833+0000","last_became_active":"2024-02-21T23:04:53.993693+0000","last_became_peered":"2024-02-21T23:04:53.993693+0000","last_unstale":"2024-02-21T23:04:54.980833+0000","last_undegraded":"2024-02-21T23:04:54.980833+0000","last_fullsized":"2024-02-21T23:04:54.980833+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:33:53.507158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.982154+0000","last_change":"2024-02-21T23:04:53.994446+0000","last_active":"2024-02-21T23:04:54.982154+0000","last_peered":"2024-02-21T23:04:54.982154+0000","last_clean":"2024-02-21T23:04:54.982154+0000","last_became_active":"2024-02-21T23:04:53.994181+0000","last_became_peered":"2024-02-21T23:04:53.994181+0000","last_unstale":"2024-02-21T23:04:54.982154+0000","last_undegraded":"2024-02-21T23:04:54.982154+0000","last_fullsized":"2024-02-21T23:04:54.982154+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:18:47.785124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-02-21T23:04:53.977305+0000","last_change":"2024-02-21T23:04:53.976184+0000","last_active":"2024-02-21T23:04:53.977305+0000","last_peered":"2024-02-21T23:04:53.977305+0000","last_clean":"2024-02-21T23:04:53.977305+0000","last_became_active":"2024-02-21T23:04:53.975098+0000","last_became_peered":"2024-02-21T23:04:53.975098+0000","last_unstale":"2024-02-21T23:04:53.977305+0000","last_undegraded":"2024-02-21T23:04:53.977305+0000","last_fullsized":"2024-02-21T23:04:53.977305+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:12:03.573396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.981130+0000","last_change":"2024-02-21T23:04:53.995281+0000","last_active":"2024-02-21T23:04:54.981130+0000","last_peered":"2024-02-21T23:04:54.981130+0000","last_clean":"2024-02-21T23:04:54.981130+0000","last_became_active":"2024-02-21T23:04:53.994940+0000","last_became_peered":"2024-02-21T23:04:53.994940+0000","last_unstale":"2024-02-21T23:04:54.981130+0000","last_undegraded":"2024-02-21T23:04:54.981130+0000","last_fullsized":"2024-02-21T23:04:54.981130+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:55:04.908945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.978313+0000","last_change":"2024-02-21T23:04:53.972067+0000","last_active":"2024-02-21T23:04:54.978313+0000","last_peered":"2024-02-21T23:04:54.978313+0000","last_clean":"2024-02-21T23:04:54.978313+0000","last_became_active":"2024-02-21T23:04:53.971855+0000","last_became_peered":"2024-02-21T23:04:53.971855+0000","last_unstale":"2024-02-21T23:04:54.978313+0000","last_undegraded":"2024-02-21T23:04:54.978313+0000","last_fullsized":"2024-02-21T23:04:54.978313+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:40:20.534202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-02-21T23:04:53.990960+0000","last_change":"2024-02-21T23:04:53.983537+0000","last_active":"2024-02-21T23:04:53.990960+0000","last_peered":"2024-02-21T23:04:53.990960+0000","last_clean":"2024-02-21T23:04:53.990960+0000","last_became_active":"2024-02-21T23:04:53.983264+0000","last_became_peered":"2024-02-21T23:04:53.983264+0000","last_unstale":"2024-02-21T23:04:53.990960+0000","last_undegraded":"2024-02-21T23:04:53.990960+0000","last_fullsized":"2024-02-21T23:04:53.990960+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:16:59.134374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.978083+0000","last_change":"2024-02-21T23:04:54.002341+0000","last_active":"2024-02-21T23:04:54.978083+0000","last_peered":"2024-02-21T23:04:54.978083+0000","last_clean":"2024-02-21T23:04:54.978083+0000","last_became_active":"2024-02-21T23:04:54.002066+0000","last_became_peered":"2024-02-21T23:04:54.002066+0000","last_unstale":"2024-02-21T23:04:54.978083+0000","last_undegraded":"2024-02-21T23:04:54.978083+0000","last_fullsized":"2024-02-21T23:04:54.978083+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:51:12.681873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:55.495664+0000","last_change":"2024-02-21T23:04:53.996395+0000","last_active":"2024-02-21T23:04:55.495664+0000","last_peered":"2024-02-21T23:04:55.495664+0000","last_clean":"2024-02-21T23:04:55.495664+0000","last_became_active":"2024-02-21T23:04:53.996219+0000","last_became_peered":"2024-02-21T23:04:53.996219+0000","last_unstale":"2024-02-21T23:04:55.495664+0000","last_undegraded":"2024-02-21T23:04:55.495664+0000","last_fullsized":"2024-02-21T23:04:55.495664+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:27:17.374378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-02-21T23:04:53.995648+0000","last_change":"2024-02-21T23:04:53.991752+0000","last_active":"2024-02-21T23:04:53.995648+0000","last_peered":"2024-02-21T23:04:53.995648+0000","last_clean":"2024-02-21T23:04:53.995648+0000","last_became_active":"2024-02-21T23:04:53.991458+0000","last_became_peered":"2024-02-21T23:04:53.991458+0000","last_unstale":"2024-02-21T23:04:53.995648+0000","last_undegraded":"2024-02-21T23:04:53.995648+0000","last_fullsized":"2024-02-21T23:04:53.995648+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:55:04.908945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"52'4","reported_seq":18,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.980851+0000","last_change":"2024-02-21T23:04:52.982273+0000","last_active":"2024-02-21T23:04:54.980851+0000","last_peered":"2024-02-21T23:04:54.980851+0000","last_clean":"2024-02-21T23:04:54.980851+0000","last_became_active":"2024-02-21T23:04:52.981799+0000","last_became_peered":"2024-02-21T23:04:52.981799+0000","last_unstale":"2024-02-21T23:04:54.980851+0000","last_undegraded":"2024-02-21T23:04:54.980851+0000","last_fullsized":"2024-02-21T23:04:54.980851+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:18:03.518976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:55.495283+0000","last_change":"2024-02-21T23:04:53.991059+0000","last_active":"2024-02-21T23:04:55.495283+0000","last_peered":"2024-02-21T23:04:55.495283+0000","last_clean":"2024-02-21T23:04:55.495283+0000","last_became_active":"2024-02-21T23:04:53.990310+0000","last_became_peered":"2024-02-21T23:04:53.990310+0000","last_unstale":"2024-02-21T23:04:55.495283+0000","last_undegraded":"2024-02-21T23:04:55.495283+0000","last_fullsized":"2024-02-21T23:04:55.495283+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:12:19.379347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"52'1","reported_seq":13,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-02-21T23:04:53.995137+0000","last_change":"2024-02-21T23:04:52.971612+0000","last_active":"2024-02-21T23:04:53.995137+0000","last_peered":"2024-02-21T23:04:53.995137+0000","last_clean":"2024-02-21T23:04:53.995137+0000","last_became_active":"2024-02-21T23:04:52.971220+0000","last_became_peered":"2024-02-21T23:04:52.971220+0000","last_unstale":"2024-02-21T23:04:53.995137+0000","last_undegraded":"2024-02-21T23:04:53.995137+0000","last_fullsized":"2024-02-21T23:04:53.995137+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:18:03.518976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.980148+0000","last_change":"2024-02-21T23:04:53.988422+0000","last_active":"2024-02-21T23:04:54.980148+0000","last_peered":"2024-02-21T23:04:54.980148+0000","last_clean":"2024-02-21T23:04:54.980148+0000","last_became_active":"2024-02-21T23:04:53.987586+0000","last_became_peered":"2024-02-21T23:04:53.987586+0000","last_unstale":"2024-02-21T23:04:54.980148+0000","last_undegraded":"2024-02-21T23:04:54.980148+0000","last_fullsized":"2024-02-21T23:04:54.980148+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:06:08.079155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"52'2","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-02-21T23:04:53.979782+0000","last_change":"2024-02-21T23:04:52.983315+0000","last_active":"2024-02-21T23:04:53.979782+0000","last_peered":"2024-02-21T23:04:53.979782+0000","last_clean":"2024-02-21T23:04:53.979782+0000","last_became_active":"2024-02-21T23:04:52.982925+0000","last_became_peered":"2024-02-21T23:04:52.982925+0000","last_unstale":"2024-02-21T23:04:53.979782+0000","last_undegraded":"2024-02-21T23:04:53.979782+0000","last_fullsized":"2024-02-21T23:04:53.979782+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:04:02.374574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.983140+0000","last_change":"2024-02-21T23:04:53.997493+0000","last_active":"2024-02-21T23:04:54.983140+0000","last_peered":"2024-02-21T23:04:54.983140+0000","last_clean":"2024-02-21T23:04:54.983140+0000","last_became_active":"2024-02-21T23:04:53.997216+0000","last_became_peered":"2024-02-21T23:04:53.997216+0000","last_unstale":"2024-02-21T23:04:54.983140+0000","last_undegraded":"2024-02-21T23:04:54.983140+0000","last_fullsized":"2024-02-21T23:04:54.983140+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:52:32.070580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"52'2","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:55.495553+0000","last_change":"2024-02-21T23:04:52.972332+0000","last_active":"2024-02-21T23:04:55.495553+0000","last_peered":"2024-02-21T23:04:55.495553+0000","last_clean":"2024-02-21T23:04:55.495553+0000","last_became_active":"2024-02-21T23:04:52.972014+0000","last_became_peered":"2024-02-21T23:04:52.972014+0000","last_unstale":"2024-02-21T23:04:55.495553+0000","last_undegraded":"2024-02-21T23:04:55.495553+0000","last_fullsized":"2024-02-21T23:04:55.495553+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:46:43.465994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:55.510996+0000","last_change":"2024-02-21T23:04:53.988566+0000","last_active":"2024-02-21T23:04:55.510996+0000","last_peered":"2024-02-21T23:04:55.510996+0000","last_clean":"2024-02-21T23:04:55.510996+0000","last_became_active":"2024-02-21T23:04:53.988271+0000","last_became_peered":"2024-02-21T23:04:53.988271+0000","last_unstale":"2024-02-21T23:04:55.510996+0000","last_undegraded":"2024-02-21T23:04:55.510996+0000","last_fullsized":"2024-02-21T23:04:55.510996+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:59:35.227750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"52'1","reported_seq":15,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.978450+0000","last_change":"2024-02-21T23:04:52.972962+0000","last_active":"2024-02-21T23:04:54.978450+0000","last_peered":"2024-02-21T23:04:54.978450+0000","last_clean":"2024-02-21T23:04:54.978450+0000","last_became_active":"2024-02-21T23:04:52.972642+0000","last_became_peered":"2024-02-21T23:04:52.972642+0000","last_unstale":"2024-02-21T23:04:54.978450+0000","last_undegraded":"2024-02-21T23:04:54.978450+0000","last_fullsized":"2024-02-21T23:04:54.978450+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:02:52.870656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-02-21T23:04:53.990210+0000","last_change":"2024-02-21T23:04:53.982723+0000","last_active":"2024-02-21T23:04:53.990210+0000","last_peered":"2024-02-21T23:04:53.990210+0000","last_clean":"2024-02-21T23:04:53.990210+0000","last_became_active":"2024-02-21T23:04:53.982447+0000","last_became_peered":"2024-02-21T23:04:53.982447+0000","last_unstale":"2024-02-21T23:04:53.990210+0000","last_undegraded":"2024-02-21T23:04:53.990210+0000","last_fullsized":"2024-02-21T23:04:53.990210+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:23:27.013837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"52'1","reported_seq":15,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:55.510227+0000","last_change":"2024-02-21T23:04:52.977866+0000","last_active":"2024-02-21T23:04:55.510227+0000","last_peered":"2024-02-21T23:04:55.510227+0000","last_clean":"2024-02-21T23:04:55.510227+0000","last_became_active":"2024-02-21T23:04:52.977477+0000","last_became_peered":"2024-02-21T23:04:52.977477+0000","last_unstale":"2024-02-21T23:04:55.510227+0000","last_undegraded":"2024-02-21T23:04:55.510227+0000","last_fullsized":"2024-02-21T23:04:55.510227+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:25:58.605620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.982755+0000","last_change":"2024-02-21T23:04:53.983494+0000","last_active":"2024-02-21T23:04:54.982755+0000","last_peered":"2024-02-21T23:04:54.982755+0000","last_clean":"2024-02-21T23:04:54.982755+0000","last_became_active":"2024-02-21T23:04:53.983213+0000","last_became_peered":"2024-02-21T23:04:53.983213+0000","last_unstale":"2024-02-21T23:04:54.982755+0000","last_undegraded":"2024-02-21T23:04:54.982755+0000","last_fullsized":"2024-02-21T23:04:54.982755+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:53:11.354446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":14,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.979914+0000","last_change":"2024-02-21T23:04:52.976990+0000","last_active":"2024-02-21T23:04:54.979914+0000","last_peered":"2024-02-21T23:04:54.979914+0000","last_clean":"2024-02-21T23:04:54.979914+0000","last_became_active":"2024-02-21T23:04:52.975637+0000","last_became_peered":"2024-02-21T23:04:52.975637+0000","last_unstale":"2024-02-21T23:04:54.979914+0000","last_undegraded":"2024-02-21T23:04:54.979914+0000","last_fullsized":"2024-02-21T23:04:54.979914+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:04:02.374574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-02-21T23:04:53.990450+0000","last_change":"2024-02-21T23:04:53.981493+0000","last_active":"2024-02-21T23:04:53.990450+0000","last_peered":"2024-02-21T23:04:53.990450+0000","last_clean":"2024-02-21T23:04:53.990450+0000","last_became_active":"2024-02-21T23:04:53.981109+0000","last_became_peered":"2024-02-21T23:04:53.981109+0000","last_unstale":"2024-02-21T23:04:53.990450+0000","last_undegraded":"2024-02-21T23:04:53.990450+0000","last_fullsized":"2024-02-21T23:04:53.990450+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:59:35.227750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":14,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.977743+0000","last_change":"2024-02-21T23:04:52.971808+0000","last_active":"2024-02-21T23:04:54.977743+0000","last_peered":"2024-02-21T23:04:54.977743+0000","last_clean":"2024-02-21T23:04:54.977743+0000","last_became_active":"2024-02-21T23:04:52.971529+0000","last_became_peered":"2024-02-21T23:04:52.971529+0000","last_unstale":"2024-02-21T23:04:54.977743+0000","last_undegraded":"2024-02-21T23:04:54.977743+0000","last_fullsized":"2024-02-21T23:04:54.977743+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:00:20.232950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:55.510298+0000","last_change":"2024-02-21T23:04:53.994186+0000","last_active":"2024-02-21T23:04:55.510298+0000","last_peered":"2024-02-21T23:04:55.510298+0000","last_clean":"2024-02-21T23:04:55.510298+0000","last_became_active":"2024-02-21T23:04:53.993902+0000","last_became_peered":"2024-02-21T23:04:53.993902+0000","last_unstale":"2024-02-21T23:04:55.510298+0000","last_undegraded":"2024-02-21T23:04:55.510298+0000","last_fullsized":"2024-02-21T23:04:55.510298+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:39:11.623072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-02-21T23:04:53.975623+0000","last_change":"2024-02-21T23:04:52.976339+0000","last_active":"2024-02-21T23:04:53.975623+0000","last_peered":"2024-02-21T23:04:53.975623+0000","last_clean":"2024-02-21T23:04:53.975623+0000","last_became_active":"2024-02-21T23:04:52.975958+0000","last_became_peered":"2024-02-21T23:04:52.975958+0000","last_unstale":"2024-02-21T23:04:53.975623+0000","last_undegraded":"2024-02-21T23:04:53.975623+0000","last_fullsized":"2024-02-21T23:04:53.975623+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:06:07.057936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:55.495463+0000","last_change":"2024-02-21T23:04:53.980747+0000","last_active":"2024-02-21T23:04:55.495463+0000","last_peered":"2024-02-21T23:04:55.495463+0000","last_clean":"2024-02-21T23:04:55.495463+0000","last_became_active":"2024-02-21T23:04:53.980456+0000","last_became_peered":"2024-02-21T23:04:53.980456+0000","last_unstale":"2024-02-21T23:04:55.495463+0000","last_undegraded":"2024-02-21T23:04:55.495463+0000","last_fullsized":"2024-02-21T23:04:55.495463+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:40:20.534202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":14,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:55.510657+0000","last_change":"2024-02-21T23:04:52.974873+0000","last_active":"2024-02-21T23:04:55.510657+0000","last_peered":"2024-02-21T23:04:55.510657+0000","last_clean":"2024-02-21T23:04:55.510657+0000","last_became_active":"2024-02-21T23:04:52.973750+0000","last_became_peered":"2024-02-21T23:04:52.973750+0000","last_unstale":"2024-02-21T23:04:55.510657+0000","last_undegraded":"2024-02-21T23:04:55.510657+0000","last_fullsized":"2024-02-21T23:04:55.510657+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:04:02.374574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.979078+0000","last_change":"2024-02-21T23:04:53.983830+0000","last_active":"2024-02-21T23:04:54.979078+0000","last_peered":"2024-02-21T23:04:54.979078+0000","last_clean":"2024-02-21T23:04:54.979078+0000","last_became_active":"2024-02-21T23:04:53.974390+0000","last_became_peered":"2024-02-21T23:04:53.974390+0000","last_unstale":"2024-02-21T23:04:54.979078+0000","last_undegraded":"2024-02-21T23:04:54.979078+0000","last_fullsized":"2024-02-21T23:04:54.979078+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:27:17.374378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"52'3","reported_seq":15,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-02-21T23:04:53.983841+0000","last_change":"2024-02-21T23:04:52.976896+0000","last_active":"2024-02-21T23:04:53.983841+0000","last_peered":"2024-02-21T23:04:53.983841+0000","last_clean":"2024-02-21T23:04:53.983841+0000","last_became_active":"2024-02-21T23:04:52.976592+0000","last_became_peered":"2024-02-21T23:04:52.976592+0000","last_unstale":"2024-02-21T23:04:53.983841+0000","last_undegraded":"2024-02-21T23:04:53.983841+0000","last_fullsized":"2024-02-21T23:04:53.983841+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:39:31.156140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:55.510904+0000","last_change":"2024-02-21T23:04:53.997903+0000","last_active":"2024-02-21T23:04:55.510904+0000","last_peered":"2024-02-21T23:04:55.510904+0000","last_clean":"2024-02-21T23:04:55.510904+0000","last_became_active":"2024-02-21T23:04:53.997538+0000","last_became_peered":"2024-02-21T23:04:53.997538+0000","last_unstale":"2024-02-21T23:04:55.510904+0000","last_undegraded":"2024-02-21T23:04:55.510904+0000","last_fullsized":"2024-02-21T23:04:55.510904+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:55:24.090577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"52'6","reported_seq":20,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.982539+0000","last_change":"2024-02-21T23:04:52.975159+0000","last_active":"2024-02-21T23:04:54.982539+0000","last_peered":"2024-02-21T23:04:54.982539+0000","last_clean":"2024-02-21T23:04:54.982539+0000","last_became_active":"2024-02-21T23:04:52.974845+0000","last_became_peered":"2024-02-21T23:04:52.974845+0000","last_unstale":"2024-02-21T23:04:54.982539+0000","last_undegraded":"2024-02-21T23:04:54.982539+0000","last_fullsized":"2024-02-21T23:04:54.982539+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:20:05.709148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:55.495151+0000","last_change":"2024-02-21T23:04:53.987920+0000","last_active":"2024-02-21T23:04:55.495151+0000","last_peered":"2024-02-21T23:04:55.495151+0000","last_clean":"2024-02-21T23:04:55.495151+0000","last_became_active":"2024-02-21T23:04:53.986231+0000","last_became_peered":"2024-02-21T23:04:53.986231+0000","last_unstale":"2024-02-21T23:04:55.495151+0000","last_undegraded":"2024-02-21T23:04:55.495151+0000","last_fullsized":"2024-02-21T23:04:55.495151+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:20:06.730366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"52'2","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.978579+0000","last_change":"2024-02-21T23:04:52.977012+0000","last_active":"2024-02-21T23:04:54.978579+0000","last_peered":"2024-02-21T23:04:54.978579+0000","last_clean":"2024-02-21T23:04:54.978579+0000","last_became_active":"2024-02-21T23:04:52.973324+0000","last_became_peered":"2024-02-21T23:04:52.973324+0000","last_unstale":"2024-02-21T23:04:54.978579+0000","last_undegraded":"2024-02-21T23:04:54.978579+0000","last_fullsized":"2024-02-21T23:04:54.978579+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:55:27.414683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.979416+0000","last_change":"2024-02-21T23:04:53.969380+0000","last_active":"2024-02-21T23:04:54.979416+0000","last_peered":"2024-02-21T23:04:54.979416+0000","last_clean":"2024-02-21T23:04:54.979416+0000","last_became_active":"2024-02-21T23:04:53.966766+0000","last_became_peered":"2024-02-21T23:04:53.966766+0000","last_unstale":"2024-02-21T23:04:54.979416+0000","last_undegraded":"2024-02-21T23:04:54.979416+0000","last_fullsized":"2024-02-21T23:04:54.979416+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:22:45.699161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":14,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.980224+0000","last_change":"2024-02-21T23:04:52.981590+0000","last_active":"2024-02-21T23:04:54.980224+0000","last_peered":"2024-02-21T23:04:54.980224+0000","last_clean":"2024-02-21T23:04:54.980224+0000","last_became_active":"2024-02-21T23:04:52.981228+0000","last_became_peered":"2024-02-21T23:04:52.981228+0000","last_unstale":"2024-02-21T23:04:54.980224+0000","last_undegraded":"2024-02-21T23:04:54.980224+0000","last_fullsized":"2024-02-21T23:04:54.980224+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:25:58.605620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.982607+0000","last_change":"2024-02-21T23:04:53.983659+0000","last_active":"2024-02-21T23:04:54.982607+0000","last_peered":"2024-02-21T23:04:54.982607+0000","last_clean":"2024-02-21T23:04:54.982607+0000","last_became_active":"2024-02-21T23:04:53.983290+0000","last_became_peered":"2024-02-21T23:04:53.983290+0000","last_unstale":"2024-02-21T23:04:54.982607+0000","last_undegraded":"2024-02-21T23:04:54.982607+0000","last_fullsized":"2024-02-21T23:04:54.982607+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:05:20.754391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-02-21T23:04:53.975277+0000","last_change":"2024-02-21T23:04:52.982825+0000","last_active":"2024-02-21T23:04:53.975277+0000","last_peered":"2024-02-21T23:04:53.975277+0000","last_clean":"2024-02-21T23:04:53.975277+0000","last_became_active":"2024-02-21T23:04:52.982523+0000","last_became_peered":"2024-02-21T23:04:52.982523+0000","last_unstale":"2024-02-21T23:04:53.975277+0000","last_undegraded":"2024-02-21T23:04:53.975277+0000","last_fullsized":"2024-02-21T23:04:53.975277+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:45:49.617039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.980183+0000","last_change":"2024-02-21T23:04:53.988578+0000","last_active":"2024-02-21T23:04:54.980183+0000","last_peered":"2024-02-21T23:04:54.980183+0000","last_clean":"2024-02-21T23:04:54.980183+0000","last_became_active":"2024-02-21T23:04:53.988310+0000","last_became_peered":"2024-02-21T23:04:53.988310+0000","last_unstale":"2024-02-21T23:04:54.980183+0000","last_undegraded":"2024-02-21T23:04:54.980183+0000","last_fullsized":"2024-02-21T23:04:54.980183+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:17:58.257074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"52'8","reported_seq":22,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.981720+0000","last_change":"2024-02-21T23:04:52.978259+0000","last_active":"2024-02-21T23:04:54.981720+0000","last_peered":"2024-02-21T23:04:54.981720+0000","last_clean":"2024-02-21T23:04:54.981720+0000","last_became_active":"2024-02-21T23:04:52.975275+0000","last_became_peered":"2024-02-21T23:04:52.975275+0000","last_unstale":"2024-02-21T23:04:54.981720+0000","last_undegraded":"2024-02-21T23:04:54.981720+0000","last_fullsized":"2024-02-21T23:04:54.981720+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:20:55.242700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.981819+0000","last_change":"2024-02-21T23:04:53.988231+0000","last_active":"2024-02-21T23:04:54.981819+0000","last_peered":"2024-02-21T23:04:54.981819+0000","last_clean":"2024-02-21T23:04:54.981819+0000","last_became_active":"2024-02-21T23:04:53.987814+0000","last_became_peered":"2024-02-21T23:04:53.987814+0000","last_unstale":"2024-02-21T23:04:54.981819+0000","last_undegraded":"2024-02-21T23:04:54.981819+0000","last_fullsized":"2024-02-21T23:04:54.981819+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:37:47.491317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"52'1","reported_seq":15,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.980694+0000","last_change":"2024-02-21T23:04:52.981262+0000","last_active":"2024-02-21T23:04:54.980694+0000","last_peered":"2024-02-21T23:04:54.980694+0000","last_clean":"2024-02-21T23:04:54.980694+0000","last_became_active":"2024-02-21T23:04:52.980981+0000","last_became_peered":"2024-02-21T23:04:52.980981+0000","last_unstale":"2024-02-21T23:04:54.980694+0000","last_undegraded":"2024-02-21T23:04:54.980694+0000","last_fullsized":"2024-02-21T23:04:54.980694+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:45:49.617039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-02-21T23:04:53.972934+0000","last_change":"2024-02-21T23:04:53.967738+0000","last_active":"2024-02-21T23:04:53.972934+0000","last_peered":"2024-02-21T23:04:53.972934+0000","last_clean":"2024-02-21T23:04:53.972934+0000","last_became_active":"2024-02-21T23:04:53.967430+0000","last_became_peered":"2024-02-21T23:04:53.967430+0000","last_unstale":"2024-02-21T23:04:53.972934+0000","last_undegraded":"2024-02-21T23:04:53.972934+0000","last_fullsized":"2024-02-21T23:04:53.972934+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:17:15.897726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"52'2","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.001287+0000","last_change":"2024-02-21T23:04:52.966722+0000","last_active":"2024-02-21T23:04:54.001287+0000","last_peered":"2024-02-21T23:04:54.001287+0000","last_clean":"2024-02-21T23:04:54.001287+0000","last_became_active":"2024-02-21T23:04:52.966322+0000","last_became_peered":"2024-02-21T23:04:52.966322+0000","last_unstale":"2024-02-21T23:04:54.001287+0000","last_undegraded":"2024-02-21T23:04:54.001287+0000","last_fullsized":"2024-02-21T23:04:54.001287+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:29:42.084956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:55.510656+0000","last_change":"2024-02-21T23:04:53.989868+0000","last_active":"2024-02-21T23:04:55.510656+0000","last_peered":"2024-02-21T23:04:55.510656+0000","last_clean":"2024-02-21T23:04:55.510656+0000","last_became_active":"2024-02-21T23:04:53.989585+0000","last_became_peered":"2024-02-21T23:04:53.989585+0000","last_unstale":"2024-02-21T23:04:55.510656+0000","last_undegraded":"2024-02-21T23:04:55.510656+0000","last_fullsized":"2024-02-21T23:04:55.510656+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:06:08.079155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"52'1","reported_seq":15,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.980899+0000","last_change":"2024-02-21T23:04:52.976203+0000","last_active":"2024-02-21T23:04:54.980899+0000","last_peered":"2024-02-21T23:04:54.980899+0000","last_clean":"2024-02-21T23:04:54.980899+0000","last_became_active":"2024-02-21T23:04:52.975787+0000","last_became_peered":"2024-02-21T23:04:52.975787+0000","last_unstale":"2024-02-21T23:04:54.980899+0000","last_undegraded":"2024-02-21T23:04:54.980899+0000","last_fullsized":"2024-02-21T23:04:54.980899+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:02:52.870656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-02-21T23:04:53.989033+0000","last_change":"2024-02-21T23:04:53.986757+0000","last_active":"2024-02-21T23:04:53.989033+0000","last_peered":"2024-02-21T23:04:53.989033+0000","last_clean":"2024-02-21T23:04:53.989033+0000","last_became_active":"2024-02-21T23:04:53.982468+0000","last_became_peered":"2024-02-21T23:04:53.982468+0000","last_unstale":"2024-02-21T23:04:53.989033+0000","last_undegraded":"2024-02-21T23:04:53.989033+0000","last_fullsized":"2024-02-21T23:04:53.989033+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:39:11.623072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":14,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.981260+0000","last_change":"2024-02-21T23:04:52.978566+0000","last_active":"2024-02-21T23:04:54.981260+0000","last_peered":"2024-02-21T23:04:54.981260+0000","last_clean":"2024-02-21T23:04:54.981260+0000","last_became_active":"2024-02-21T23:04:52.978262+0000","last_became_peered":"2024-02-21T23:04:52.978262+0000","last_unstale":"2024-02-21T23:04:54.981260+0000","last_undegraded":"2024-02-21T23:04:54.981260+0000","last_fullsized":"2024-02-21T23:04:54.981260+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:12:02.552178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:55.495478+0000","last_change":"2024-02-21T23:04:53.982199+0000","last_active":"2024-02-21T23:04:55.495478+0000","last_peered":"2024-02-21T23:04:55.495478+0000","last_clean":"2024-02-21T23:04:55.495478+0000","last_became_active":"2024-02-21T23:04:53.981777+0000","last_became_peered":"2024-02-21T23:04:53.981777+0000","last_unstale":"2024-02-21T23:04:55.495478+0000","last_undegraded":"2024-02-21T23:04:55.495478+0000","last_fullsized":"2024-02-21T23:04:55.495478+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:52:43.502544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"52'2","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-02-21T23:04:53.988401+0000","last_change":"2024-02-21T23:04:52.961423+0000","last_active":"2024-02-21T23:04:53.988401+0000","last_peered":"2024-02-21T23:04:53.988401+0000","last_clean":"2024-02-21T23:04:53.988401+0000","last_became_active":"2024-02-21T23:04:52.961090+0000","last_became_peered":"2024-02-21T23:04:52.961090+0000","last_unstale":"2024-02-21T23:04:53.988401+0000","last_undegraded":"2024-02-21T23:04:53.988401+0000","last_fullsized":"2024-02-21T23:04:53.988401+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:25:58.605620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.980739+0000","last_change":"2024-02-21T23:04:53.983031+0000","last_active":"2024-02-21T23:04:54.980739+0000","last_peered":"2024-02-21T23:04:54.980739+0000","last_clean":"2024-02-21T23:04:54.980739+0000","last_became_active":"2024-02-21T23:04:53.981770+0000","last_became_peered":"2024-02-21T23:04:53.981770+0000","last_unstale":"2024-02-21T23:04:54.980739+0000","last_undegraded":"2024-02-21T23:04:54.980739+0000","last_fullsized":"2024-02-21T23:04:54.980739+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:23:27.013837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":14,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.981729+0000","last_change":"2024-02-21T23:04:52.975453+0000","last_active":"2024-02-21T23:04:54.981729+0000","last_peered":"2024-02-21T23:04:54.981729+0000","last_clean":"2024-02-21T23:04:54.981729+0000","last_became_active":"2024-02-21T23:04:52.975121+0000","last_became_peered":"2024-02-21T23:04:52.975121+0000","last_unstale":"2024-02-21T23:04:54.981729+0000","last_undegraded":"2024-02-21T23:04:54.981729+0000","last_fullsized":"2024-02-21T23:04:54.981729+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:51:11.660655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.979723+0000","last_change":"2024-02-21T23:04:53.978362+0000","last_active":"2024-02-21T23:04:54.979723+0000","last_peered":"2024-02-21T23:04:54.979723+0000","last_clean":"2024-02-21T23:04:54.979723+0000","last_became_active":"2024-02-21T23:04:53.978070+0000","last_became_peered":"2024-02-21T23:04:53.978070+0000","last_unstale":"2024-02-21T23:04:54.979723+0000","last_undegraded":"2024-02-21T23:04:54.979723+0000","last_fullsized":"2024-02-21T23:04:54.979723+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:52:55.798286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":14,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.978816+0000","last_change":"2024-02-21T23:04:52.979855+0000","last_active":"2024-02-21T23:04:54.978816+0000","last_peered":"2024-02-21T23:04:54.978816+0000","last_clean":"2024-02-21T23:04:54.978816+0000","last_became_active":"2024-02-21T23:04:52.977225+0000","last_became_peered":"2024-02-21T23:04:52.977225+0000","last_unstale":"2024-02-21T23:04:54.978816+0000","last_undegraded":"2024-02-21T23:04:54.978816+0000","last_fullsized":"2024-02-21T23:04:54.978816+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:39:10.601854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.000684+0000","last_change":"2024-02-21T23:04:53.996234+0000","last_active":"2024-02-21T23:04:54.000684+0000","last_peered":"2024-02-21T23:04:54.000684+0000","last_clean":"2024-02-21T23:04:54.000684+0000","last_became_active":"2024-02-21T23:04:53.995859+0000","last_became_peered":"2024-02-21T23:04:53.995859+0000","last_unstale":"2024-02-21T23:04:54.000684+0000","last_undegraded":"2024-02-21T23:04:54.000684+0000","last_fullsized":"2024-02-21T23:04:54.000684+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:06:08.079155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-02-21T23:04:53.980209+0000","last_change":"2024-02-21T23:04:52.972729+0000","last_active":"2024-02-21T23:04:53.980209+0000","last_peered":"2024-02-21T23:04:53.980209+0000","last_clean":"2024-02-21T23:04:53.980209+0000","last_became_active":"2024-02-21T23:04:52.972434+0000","last_became_peered":"2024-02-21T23:04:52.972434+0000","last_unstale":"2024-02-21T23:04:53.980209+0000","last_undegraded":"2024-02-21T23:04:53.980209+0000","last_fullsized":"2024-02-21T23:04:53.980209+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:00:21.589361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:55.495836+0000","last_change":"2024-02-21T23:04:53.996387+0000","last_active":"2024-02-21T23:04:55.495836+0000","last_peered":"2024-02-21T23:04:55.495836+0000","last_clean":"2024-02-21T23:04:55.495836+0000","last_became_active":"2024-02-21T23:04:53.996123+0000","last_became_peered":"2024-02-21T23:04:53.996123+0000","last_unstale":"2024-02-21T23:04:55.495836+0000","last_undegraded":"2024-02-21T23:04:55.495836+0000","last_fullsized":"2024-02-21T23:04:55.495836+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:43:19.009080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"52'2","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.981127+0000","last_change":"2024-02-21T23:04:52.981486+0000","last_active":"2024-02-21T23:04:54.981127+0000","last_peered":"2024-02-21T23:04:54.981127+0000","last_clean":"2024-02-21T23:04:54.981127+0000","last_became_active":"2024-02-21T23:04:52.981135+0000","last_became_peered":"2024-02-21T23:04:52.981135+0000","last_unstale":"2024-02-21T23:04:54.981127+0000","last_undegraded":"2024-02-21T23:04:54.981127+0000","last_fullsized":"2024-02-21T23:04:54.981127+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:17:14.876508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-02-21T23:04:53.978346+0000","last_change":"2024-02-21T23:04:53.977118+0000","last_active":"2024-02-21T23:04:53.978346+0000","last_peered":"2024-02-21T23:04:53.978346+0000","last_clean":"2024-02-21T23:04:53.978346+0000","last_became_active":"2024-02-21T23:04:53.976042+0000","last_became_peered":"2024-02-21T23:04:53.976042+0000","last_unstale":"2024-02-21T23:04:53.978346+0000","last_undegraded":"2024-02-21T23:04:53.978346+0000","last_fullsized":"2024-02-21T23:04:53.978346+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:02:53.891874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-02-21T23:04:53.966474+0000","last_change":"2024-02-21T23:04:52.974536+0000","last_active":"2024-02-21T23:04:53.966474+0000","last_peered":"2024-02-21T23:04:53.966474+0000","last_clean":"2024-02-21T23:04:53.966474+0000","last_became_active":"2024-02-21T23:04:52.974221+0000","last_became_peered":"2024-02-21T23:04:52.974221+0000","last_unstale":"2024-02-21T23:04:53.966474+0000","last_undegraded":"2024-02-21T23:04:53.966474+0000","last_fullsized":"2024-02-21T23:04:53.966474+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:39:10.601854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.983327+0000","last_change":"2024-02-21T23:04:53.979161+0000","last_active":"2024-02-21T23:04:54.983327+0000","last_peered":"2024-02-21T23:04:54.983327+0000","last_clean":"2024-02-21T23:04:54.983327+0000","last_became_active":"2024-02-21T23:04:53.978843+0000","last_became_peered":"2024-02-21T23:04:53.978843+0000","last_unstale":"2024-02-21T23:04:54.983327+0000","last_undegraded":"2024-02-21T23:04:54.983327+0000","last_fullsized":"2024-02-21T23:04:54.983327+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:34:45.807165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":14,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.978706+0000","last_change":"2024-02-21T23:04:52.969804+0000","last_active":"2024-02-21T23:04:54.978706+0000","last_peered":"2024-02-21T23:04:54.978706+0000","last_clean":"2024-02-21T23:04:54.978706+0000","last_became_active":"2024-02-21T23:04:52.969353+0000","last_became_peered":"2024-02-21T23:04:52.969353+0000","last_unstale":"2024-02-21T23:04:54.978706+0000","last_undegraded":"2024-02-21T23:04:54.978706+0000","last_fullsized":"2024-02-21T23:04:54.978706+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:47:27.364576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.978582+0000","last_change":"2024-02-21T23:04:54.000144+0000","last_active":"2024-02-21T23:04:54.978582+0000","last_peered":"2024-02-21T23:04:54.978582+0000","last_clean":"2024-02-21T23:04:54.978582+0000","last_became_active":"2024-02-21T23:04:53.999687+0000","last_became_peered":"2024-02-21T23:04:53.999687+0000","last_unstale":"2024-02-21T23:04:54.978582+0000","last_undegraded":"2024-02-21T23:04:54.978582+0000","last_fullsized":"2024-02-21T23:04:54.978582+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:12:19.379347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":14,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.983433+0000","last_change":"2024-02-21T23:04:52.975668+0000","last_active":"2024-02-21T23:04:54.983433+0000","last_peered":"2024-02-21T23:04:54.983433+0000","last_clean":"2024-02-21T23:04:54.983433+0000","last_became_active":"2024-02-21T23:04:52.975348+0000","last_became_peered":"2024-02-21T23:04:52.975348+0000","last_unstale":"2024-02-21T23:04:54.983433+0000","last_undegraded":"2024-02-21T23:04:54.983433+0000","last_fullsized":"2024-02-21T23:04:54.983433+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:12:18.358129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:55.495731+0000","last_change":"2024-02-21T23:04:53.988119+0000","last_active":"2024-02-21T23:04:55.495731+0000","last_peered":"2024-02-21T23:04:55.495731+0000","last_clean":"2024-02-21T23:04:55.495731+0000","last_became_active":"2024-02-21T23:04:53.986237+0000","last_became_peered":"2024-02-21T23:04:53.986237+0000","last_unstale":"2024-02-21T23:04:55.495731+0000","last_undegraded":"2024-02-21T23:04:55.495731+0000","last_fullsized":"2024-02-21T23:04:55.495731+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:10:20.341856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":14,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.981876+0000","last_change":"2024-02-21T23:04:52.976653+0000","last_active":"2024-02-21T23:04:54.981876+0000","last_peered":"2024-02-21T23:04:54.981876+0000","last_clean":"2024-02-21T23:04:54.981876+0000","last_became_active":"2024-02-21T23:04:52.976249+0000","last_became_peered":"2024-02-21T23:04:52.976249+0000","last_unstale":"2024-02-21T23:04:54.981876+0000","last_undegraded":"2024-02-21T23:04:54.981876+0000","last_fullsized":"2024-02-21T23:04:54.981876+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:40:19.512984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.978897+0000","last_change":"2024-02-21T23:04:53.993076+0000","last_active":"2024-02-21T23:04:54.978897+0000","last_peered":"2024-02-21T23:04:54.978897+0000","last_clean":"2024-02-21T23:04:54.978897+0000","last_became_active":"2024-02-21T23:04:53.992734+0000","last_became_peered":"2024-02-21T23:04:53.992734+0000","last_unstale":"2024-02-21T23:04:54.978897+0000","last_undegraded":"2024-02-21T23:04:54.978897+0000","last_fullsized":"2024-02-21T23:04:54.978897+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:52:43.502544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":14,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:55.510160+0000","last_change":"2024-02-21T23:04:52.971065+0000","last_active":"2024-02-21T23:04:55.510160+0000","last_peered":"2024-02-21T23:04:55.510160+0000","last_clean":"2024-02-21T23:04:55.510160+0000","last_became_active":"2024-02-21T23:04:52.970410+0000","last_became_peered":"2024-02-21T23:04:52.970410+0000","last_unstale":"2024-02-21T23:04:55.510160+0000","last_undegraded":"2024-02-21T23:04:55.510160+0000","last_fullsized":"2024-02-21T23:04:55.510160+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:18:03.518976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.983009+0000","last_change":"2024-02-21T23:04:53.995393+0000","last_active":"2024-02-21T23:04:54.983009+0000","last_peered":"2024-02-21T23:04:54.983009+0000","last_clean":"2024-02-21T23:04:54.983009+0000","last_became_active":"2024-02-21T23:04:53.995131+0000","last_became_peered":"2024-02-21T23:04:53.995131+0000","last_unstale":"2024-02-21T23:04:54.983009+0000","last_undegraded":"2024-02-21T23:04:54.983009+0000","last_fullsized":"2024-02-21T23:04:54.983009+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:34:02.144778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":14,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:55.495907+0000","last_change":"2024-02-21T23:04:52.968036+0000","last_active":"2024-02-21T23:04:55.495907+0000","last_peered":"2024-02-21T23:04:55.495907+0000","last_clean":"2024-02-21T23:04:55.495907+0000","last_became_active":"2024-02-21T23:04:52.967626+0000","last_became_peered":"2024-02-21T23:04:52.967626+0000","last_unstale":"2024-02-21T23:04:55.495907+0000","last_undegraded":"2024-02-21T23:04:55.495907+0000","last_fullsized":"2024-02-21T23:04:55.495907+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:33:33.298554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.980186+0000","last_change":"2024-02-21T23:04:53.985061+0000","last_active":"2024-02-21T23:04:54.980186+0000","last_peered":"2024-02-21T23:04:54.980186+0000","last_clean":"2024-02-21T23:04:54.980186+0000","last_became_active":"2024-02-21T23:04:53.984756+0000","last_became_peered":"2024-02-21T23:04:53.984756+0000","last_unstale":"2024-02-21T23:04:54.980186+0000","last_undegraded":"2024-02-21T23:04:54.980186+0000","last_fullsized":"2024-02-21T23:04:54.980186+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:55:24.090577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":14,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.979720+0000","last_change":"2024-02-21T23:04:52.979253+0000","last_active":"2024-02-21T23:04:54.979720+0000","last_peered":"2024-02-21T23:04:54.979720+0000","last_clean":"2024-02-21T23:04:54.979720+0000","last_became_active":"2024-02-21T23:04:52.978773+0000","last_became_peered":"2024-02-21T23:04:52.978773+0000","last_unstale":"2024-02-21T23:04:54.979720+0000","last_undegraded":"2024-02-21T23:04:54.979720+0000","last_fullsized":"2024-02-21T23:04:54.979720+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:39:31.156140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":229,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-02-21T23:04:53.963786+0000","last_change":"2024-02-21T23:02:32.289335+0000","last_active":"2024-02-21T23:04:53.963786+0000","last_peered":"2024-02-21T23:04:53.963786+0000","last_clean":"2024-02-21T23:04:53.963786+0000","last_became_active":"2024-02-21T23:02:32.286913+0000","last_became_peered":"2024-02-21T23:02:32.286913+0000","last_unstale":"2024-02-21T23:04:53.963786+0000","last_undegraded":"2024-02-21T23:04:53.963786+0000","last_fullsized":"2024-02-21T23:04:53.963786+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_clean_scrub_stamp":"2024-02-21T23:02:31.172569+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:36:23.987374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574879,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27704,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344136,"statfs":{"total":96636764160,"available":96608395264,"internally_reserved":0,"allocated":692224,"data_stored":867144,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574880,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27648,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344192,"statfs":{"total":96636764160,"available":96608452608,"internally_reserved":0,"allocated":638976,"data_stored":283996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574879,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27628,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344212,"statfs":{"total":96636764160,"available":96608473088,"internally_reserved":0,"allocated":618496,"data_stored":276652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574880,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27632,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344208,"statfs":{"total":96636764160,"available":96608468992,"internally_reserved":0,"allocated":622592,"data_stored":277234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574880,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31820,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340020,"statfs":{"total":96636764160,"available":96604180480,"internally_reserved":0,"allocated":647168,"data_stored":284978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574881,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31800,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340040,"statfs":{"total":96636764160,"available":96604200960,"internally_reserved":0,"allocated":630784,"data_stored":278264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574879,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31796,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340044,"statfs":{"total":96636764160,"available":96604205056,"internally_reserved":0,"allocated":626688,"data_stored":276812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574879,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31864,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339976,"statfs":{"total":96636764160,"available":96604135424,"internally_reserved":0,"allocated":692224,"data_stored":867076,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-21T23:04:58.615 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-21T23:04:58.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.879+0000 7f2c9b200640 1 Processor -- start 2024-02-21T23:04:58.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.879+0000 7f2c9b200640 1 -- start start 2024-02-21T23:04:58.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.880+0000 7f2c9b200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2c9c0aff90 0x7f2c9c0c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:58.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.880+0000 7f2c9b200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2c9c0c5720 0x7f2c9c15f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:58.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.880+0000 7f2c9b200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c9c0c74c0 con 0x7f2c9c06dbf0 2024-02-21T23:04:58.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.880+0000 7f2c9b200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c9c0c7270 con 0x7f2c9c0aff90 2024-02-21T23:04:58.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.880+0000 7f2c9b200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c9c158630 con 0x7f2c9c0c5720 2024-02-21T23:04:58.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.880+0000 7f2c99e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2c9c0aff90 0x7f2c9c0c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:58.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.881+0000 7f2c99e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2c9c0aff90 0x7f2c9c0c51e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45596/0 (socket says 172.21.15.28:45596) 2024-02-21T23:04:58.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.881+0000 7f2c99e00640 1 -- 172.21.15.28:0/2237647544 learned_addr learned my addr 172.21.15.28:0/2237647544 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:58.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.881+0000 7f2c9a800640 1 --2- 172.21.15.28:0/2237647544 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2c9c0c5720 0x7f2c9c15f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:58.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.881+0000 7f2c99e00640 1 -- 172.21.15.28:0/2237647544 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2c9c0c5720 msgr2=0x7f2c9c15f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:58.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.881+0000 7f2c99e00640 1 --2- 172.21.15.28:0/2237647544 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2c9c0c5720 0x7f2c9c15f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:58.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.881+0000 7f2c99e00640 1 -- 172.21.15.28:0/2237647544 >> v1:172.21.15.81:6789/0 conn(0x7f2c9c06dbf0 legacy=0x7f2c9c158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:58.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.881+0000 7f2c99e00640 1 -- 172.21.15.28:0/2237647544 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c9c0c77a0 con 0x7f2c9c0aff90 2024-02-21T23:04:58.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.881+0000 7f2c9a800640 1 --2- 172.21.15.28:0/2237647544 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2c9c0c5720 0x7f2c9c15f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:04:58.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.882+0000 7f2c99e00640 1 --2- 172.21.15.28:0/2237647544 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c9c0aff90 0x7f2c9c0c51e0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c009e50 tx=0x7f2c8c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d2db1ee11444317 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:58.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.882+0000 7f2c98a00640 1 -- 172.21.15.28:0/2237647544 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c8c02ca70 con 0x7f2c9c0aff90 2024-02-21T23:04:58.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.883+0000 7f2c98a00640 1 -- 172.21.15.28:0/2237647544 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c8c02cc10 con 0x7f2c9c0aff90 2024-02-21T23:04:58.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.883+0000 7f2c98a00640 1 -- 172.21.15.28:0/2237647544 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c8c0343f0 con 0x7f2c9c0aff90 2024-02-21T23:04:58.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.883+0000 7f2c9b200640 1 -- 172.21.15.28:0/2237647544 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c9c0aff90 msgr2=0x7f2c9c0c51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:58.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.883+0000 7f2c9b200640 1 --2- 172.21.15.28:0/2237647544 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c9c0aff90 0x7f2c9c0c51e0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c009e50 tx=0x7f2c8c02bed0 comp rx=0 tx=0).stop 2024-02-21T23:04:58.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.883+0000 7f2c9b200640 1 -- 172.21.15.28:0/2237647544 shutdown_connections 2024-02-21T23:04:58.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.883+0000 7f2c9b200640 1 --2- 172.21.15.28:0/2237647544 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2c9c0c5720 0x7f2c9c15f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:58.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.883+0000 7f2c9b200640 1 --2- 172.21.15.28:0/2237647544 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c9c0aff90 0x7f2c9c0c51e0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:58.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.883+0000 7f2c9b200640 1 -- 172.21.15.28:0/2237647544 >> 172.21.15.28:0/2237647544 conn(0x7f2c9c0ac370 msgr2=0x7f2c9c0ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:58.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.883+0000 7f2c9b200640 1 -- 172.21.15.28:0/2237647544 shutdown_connections 2024-02-21T23:04:58.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.883+0000 7f2c9b200640 1 -- 172.21.15.28:0/2237647544 wait complete. 2024-02-21T23:04:58.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.885+0000 7f2c9b200640 1 Processor -- start 2024-02-21T23:04:58.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.885+0000 7f2c9b200640 1 -- start start 2024-02-21T23:04:58.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.886+0000 7f2c9b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c9c06dbf0 0x7f2c9c0c2600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:58.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.886+0000 7f2c9b200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2c9c0aff90 0x7f2c9c0abb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:58.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.886+0000 7f2c9b200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2c9c0c5720 0x7f2c9c0ade00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:58.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.886+0000 7f2c9b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c9c0c8760 con 0x7f2c9c06dbf0 2024-02-21T23:04:58.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.886+0000 7f2c9b200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c9c0be0c0 con 0x7f2c9c0c5720 2024-02-21T23:04:58.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.886+0000 7f2c9b200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c9c0c88e0 con 0x7f2c9c0aff90 2024-02-21T23:04:58.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.886+0000 7f2c99e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c9c06dbf0 0x7f2c9c0c2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:58.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.886+0000 7f2c99e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c9c06dbf0 0x7f2c9c0c2600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45598/0 (socket says 172.21.15.28:45598) 2024-02-21T23:04:58.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.886+0000 7f2c99e00640 1 -- 172.21.15.28:0/1856650537 learned_addr learned my addr 172.21.15.28:0/1856650537 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:58.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.887+0000 7f2c9a800640 1 --2- 172.21.15.28:0/1856650537 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2c9c0c5720 0x7f2c9c0ade00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:58.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.887+0000 7f2c99400640 1 --2- 172.21.15.28:0/1856650537 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2c9c0aff90 0x7f2c9c0abb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:58.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.887+0000 7f2c99e00640 1 -- 172.21.15.28:0/1856650537 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2c9c0aff90 msgr2=0x7f2c9c0abb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:58.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.887+0000 7f2c99e00640 1 --2- 172.21.15.28:0/1856650537 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2c9c0aff90 0x7f2c9c0abb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:58.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.887+0000 7f2c99e00640 1 -- 172.21.15.28:0/1856650537 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2c9c0c5720 msgr2=0x7f2c9c0ade00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:58.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.887+0000 7f2c99e00640 1 --2- 172.21.15.28:0/1856650537 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2c9c0c5720 0x7f2c9c0ade00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:58.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.887+0000 7f2c99e00640 1 -- 172.21.15.28:0/1856650537 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c9c1eb3c0 con 0x7f2c9c06dbf0 2024-02-21T23:04:58.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.887+0000 7f2c99e00640 1 --2- 172.21.15.28:0/1856650537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c9c06dbf0 0x7f2c9c0c2600 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c009e50 tx=0x7f2c8c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:58.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.888+0000 7f2c92000640 1 -- 172.21.15.28:0/1856650537 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c8c043020 con 0x7f2c9c06dbf0 2024-02-21T23:04:58.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.888+0000 7f2c92000640 1 -- 172.21.15.28:0/1856650537 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c8c03e450 con 0x7f2c9c06dbf0 2024-02-21T23:04:58.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.888+0000 7f2c92000640 1 -- 172.21.15.28:0/1856650537 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c8c03e840 con 0x7f2c9c06dbf0 2024-02-21T23:04:58.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.888+0000 7f2c9b200640 1 -- 172.21.15.28:0/1856650537 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c9c1eb6a0 con 0x7f2c9c06dbf0 2024-02-21T23:04:58.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.888+0000 7f2c9b200640 1 -- 172.21.15.28:0/1856650537 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c9c0ac0d0 con 0x7f2c9c06dbf0 2024-02-21T23:04:58.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.889+0000 7f2c9b200640 1 -- 172.21.15.28:0/1856650537 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c9c04f290 con 0x7f2c9c06dbf0 2024-02-21T23:04:58.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.890+0000 7f2c92000640 1 -- 172.21.15.28:0/1856650537 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2c8c046020 con 0x7f2c9c06dbf0 2024-02-21T23:04:58.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.891+0000 7f2c92000640 1 --2- 172.21.15.28:0/1856650537 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2c7806b4d0 0x7f2c7806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:58.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.891+0000 7f2c92000640 1 -- 172.21.15.28:0/1856650537 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f2c8c0acd10 con 0x7f2c9c06dbf0 2024-02-21T23:04:58.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.892+0000 7f2c99400640 1 --2- 172.21.15.28:0/1856650537 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2c7806b4d0 0x7f2c7806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:58.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.893+0000 7f2c99400640 1 --2- 172.21.15.28:0/1856650537 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2c7806b4d0 0x7f2c7806d990 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f2c9c0c18a0 tx=0x7f2c84009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:58.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:58.896+0000 7f2c92000640 1 -- 172.21.15.28:0/1856650537 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f2c8c0b2020 con 0x7f2c9c06dbf0 2024-02-21T23:04:59.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.085+0000 7f2c9b200640 1 -- 172.21.15.28:0/1856650537 --> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2c9c0af120 con 0x7f2c7806b4d0 2024-02-21T23:04:59.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.105+0000 7f2c92000640 1 -- 172.21.15.28:0/1856650537 <== mgr.4101 v2:172.21.15.81:6832/1848538577 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252006 (secure 0 0 0) 0x7f2c840053f0 con 0x7f2c7806b4d0 2024-02-21T23:04:59.098 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:04:59.099 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-21T23:04:59.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.110+0000 7f2c9b200640 1 -- 172.21.15.28:0/1856650537 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2c7806b4d0 msgr2=0x7f2c7806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:59.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.110+0000 7f2c9b200640 1 --2- 172.21.15.28:0/1856650537 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2c7806b4d0 0x7f2c7806d990 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f2c9c0c18a0 tx=0x7f2c84009040 comp rx=0 tx=0).stop 2024-02-21T23:04:59.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.110+0000 7f2c9b200640 1 -- 172.21.15.28:0/1856650537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c9c06dbf0 msgr2=0x7f2c9c0c2600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:59.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.110+0000 7f2c9b200640 1 --2- 172.21.15.28:0/1856650537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c9c06dbf0 0x7f2c9c0c2600 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c009e50 tx=0x7f2c8c004a40 comp rx=0 tx=0).stop 2024-02-21T23:04:59.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.110+0000 7f2c9b200640 1 -- 172.21.15.28:0/1856650537 shutdown_connections 2024-02-21T23:04:59.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.110+0000 7f2c9b200640 1 --2- 172.21.15.28:0/1856650537 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2c7806b4d0 0x7f2c7806d990 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:59.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.110+0000 7f2c9b200640 1 --2- 172.21.15.28:0/1856650537 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2c9c0aff90 0x7f2c9c0abb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:59.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.110+0000 7f2c9b200640 1 --2- 172.21.15.28:0/1856650537 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2c9c0c5720 0x7f2c9c0ade00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:59.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.110+0000 7f2c9b200640 1 --2- 172.21.15.28:0/1856650537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c9c06dbf0 0x7f2c9c0c2600 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:59.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.110+0000 7f2c9b200640 1 -- 172.21.15.28:0/1856650537 >> 172.21.15.28:0/1856650537 conn(0x7f2c9c0ac370 msgr2=0x7f2c9c0ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:59.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.111+0000 7f2c9b200640 1 -- 172.21.15.28:0/1856650537 shutdown_connections 2024-02-21T23:04:59.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.111+0000 7f2c9b200640 1 -- 172.21.15.28:0/1856650537 wait complete. 2024-02-21T23:04:59.118 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":120,"stamp":"2024-02-21T23:04:58.712262+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":135,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237892,"kb_used_data":5048,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754736828,"statfs":{"total":773094113280,"available":772850511872,"internally_reserved":0,"allocated":5169152,"data_stored":3412156,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":46,"apply_latency_ms":46,"commit_latency_ns":46000000,"apply_latency_ns":46000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002232"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-02-21T23:04:53.989051+0000","last_change":"2024-02-21T23:04:53.986963+0000","last_active":"2024-02-21T23:04:53.989051+0000","last_peered":"2024-02-21T23:04:53.989051+0000","last_clean":"2024-02-21T23:04:53.989051+0000","last_became_active":"2024-02-21T23:04:53.986659+0000","last_became_peered":"2024-02-21T23:04:53.986659+0000","last_unstale":"2024-02-21T23:04:53.989051+0000","last_undegraded":"2024-02-21T23:04:53.989051+0000","last_fullsized":"2024-02-21T23:04:53.989051+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:45:50.638257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-02-21T23:04:53.995829+0000","last_change":"2024-02-21T23:04:53.992488+0000","last_active":"2024-02-21T23:04:53.995829+0000","last_peered":"2024-02-21T23:04:53.995829+0000","last_clean":"2024-02-21T23:04:53.995829+0000","last_became_active":"2024-02-21T23:04:53.992187+0000","last_became_peered":"2024-02-21T23:04:53.992187+0000","last_unstale":"2024-02-21T23:04:53.995829+0000","last_undegraded":"2024-02-21T23:04:53.995829+0000","last_fullsized":"2024-02-21T23:04:53.995829+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:48:17.190667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.979425+0000","last_change":"2024-02-21T23:04:54.002212+0000","last_active":"2024-02-21T23:04:54.979425+0000","last_peered":"2024-02-21T23:04:54.979425+0000","last_clean":"2024-02-21T23:04:54.979425+0000","last_became_active":"2024-02-21T23:04:54.001921+0000","last_became_peered":"2024-02-21T23:04:54.001921+0000","last_unstale":"2024-02-21T23:04:54.979425+0000","last_undegraded":"2024-02-21T23:04:54.979425+0000","last_fullsized":"2024-02-21T23:04:54.979425+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:43:19.009080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-02-21T23:04:53.998407+0000","last_change":"2024-02-21T23:04:53.994828+0000","last_active":"2024-02-21T23:04:53.998407+0000","last_peered":"2024-02-21T23:04:53.998407+0000","last_clean":"2024-02-21T23:04:53.998407+0000","last_became_active":"2024-02-21T23:04:53.993942+0000","last_became_peered":"2024-02-21T23:04:53.993942+0000","last_unstale":"2024-02-21T23:04:53.998407+0000","last_undegraded":"2024-02-21T23:04:53.998407+0000","last_fullsized":"2024-02-21T23:04:53.998407+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:39:16.498706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:55.510609+0000","last_change":"2024-02-21T23:04:53.984836+0000","last_active":"2024-02-21T23:04:55.510609+0000","last_peered":"2024-02-21T23:04:55.510609+0000","last_clean":"2024-02-21T23:04:55.510609+0000","last_became_active":"2024-02-21T23:04:53.984493+0000","last_became_peered":"2024-02-21T23:04:53.984493+0000","last_unstale":"2024-02-21T23:04:55.510609+0000","last_undegraded":"2024-02-21T23:04:55.510609+0000","last_fullsized":"2024-02-21T23:04:55.510609+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:45:50.638257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-02-21T23:04:53.987322+0000","last_change":"2024-02-21T23:04:53.984765+0000","last_active":"2024-02-21T23:04:53.987322+0000","last_peered":"2024-02-21T23:04:53.987322+0000","last_clean":"2024-02-21T23:04:53.987322+0000","last_became_active":"2024-02-21T23:04:53.984432+0000","last_became_peered":"2024-02-21T23:04:53.984432+0000","last_unstale":"2024-02-21T23:04:53.987322+0000","last_undegraded":"2024-02-21T23:04:53.987322+0000","last_fullsized":"2024-02-21T23:04:53.987322+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:39:32.177358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.977895+0000","last_change":"2024-02-21T23:04:54.003595+0000","last_active":"2024-02-21T23:04:54.977895+0000","last_peered":"2024-02-21T23:04:54.977895+0000","last_clean":"2024-02-21T23:04:54.977895+0000","last_became_active":"2024-02-21T23:04:54.003085+0000","last_became_peered":"2024-02-21T23:04:54.003085+0000","last_unstale":"2024-02-21T23:04:54.977895+0000","last_undegraded":"2024-02-21T23:04:54.977895+0000","last_fullsized":"2024-02-21T23:04:54.977895+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:20:56.263919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.981592+0000","last_change":"2024-02-21T23:04:53.991959+0000","last_active":"2024-02-21T23:04:54.981592+0000","last_peered":"2024-02-21T23:04:54.981592+0000","last_clean":"2024-02-21T23:04:54.981592+0000","last_became_active":"2024-02-21T23:04:53.988135+0000","last_became_peered":"2024-02-21T23:04:53.988135+0000","last_unstale":"2024-02-21T23:04:54.981592+0000","last_undegraded":"2024-02-21T23:04:54.981592+0000","last_fullsized":"2024-02-21T23:04:54.981592+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:28:35.805544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.982863+0000","last_change":"2024-02-21T23:04:53.982813+0000","last_active":"2024-02-21T23:04:54.982863+0000","last_peered":"2024-02-21T23:04:54.982863+0000","last_clean":"2024-02-21T23:04:54.982863+0000","last_became_active":"2024-02-21T23:04:53.982515+0000","last_became_peered":"2024-02-21T23:04:53.982515+0000","last_unstale":"2024-02-21T23:04:54.982863+0000","last_undegraded":"2024-02-21T23:04:54.982863+0000","last_fullsized":"2024-02-21T23:04:54.982863+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:03:03.310447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-02-21T23:04:53.986722+0000","last_change":"2024-02-21T23:04:53.982276+0000","last_active":"2024-02-21T23:04:53.986722+0000","last_peered":"2024-02-21T23:04:53.986722+0000","last_clean":"2024-02-21T23:04:53.986722+0000","last_became_active":"2024-02-21T23:04:53.981735+0000","last_became_peered":"2024-02-21T23:04:53.981735+0000","last_unstale":"2024-02-21T23:04:53.986722+0000","last_undegraded":"2024-02-21T23:04:53.986722+0000","last_fullsized":"2024-02-21T23:04:53.986722+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:47:28.385794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-02-21T23:04:53.994969+0000","last_change":"2024-02-21T23:04:53.992336+0000","last_active":"2024-02-21T23:04:53.994969+0000","last_peered":"2024-02-21T23:04:53.994969+0000","last_clean":"2024-02-21T23:04:53.994969+0000","last_became_active":"2024-02-21T23:04:53.992068+0000","last_became_peered":"2024-02-21T23:04:53.992068+0000","last_unstale":"2024-02-21T23:04:53.994969+0000","last_undegraded":"2024-02-21T23:04:53.994969+0000","last_fullsized":"2024-02-21T23:04:53.994969+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:49:04.519126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.982368+0000","last_change":"2024-02-21T23:04:53.991879+0000","last_active":"2024-02-21T23:04:54.982368+0000","last_peered":"2024-02-21T23:04:54.982368+0000","last_clean":"2024-02-21T23:04:54.982368+0000","last_became_active":"2024-02-21T23:04:53.985749+0000","last_became_peered":"2024-02-21T23:04:53.985749+0000","last_unstale":"2024-02-21T23:04:54.982368+0000","last_undegraded":"2024-02-21T23:04:54.982368+0000","last_fullsized":"2024-02-21T23:04:54.982368+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:21:11.752620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-02-21T23:04:53.986913+0000","last_change":"2024-02-21T23:04:53.982139+0000","last_active":"2024-02-21T23:04:53.986913+0000","last_peered":"2024-02-21T23:04:53.986913+0000","last_clean":"2024-02-21T23:04:53.986913+0000","last_became_active":"2024-02-21T23:04:53.981800+0000","last_became_peered":"2024-02-21T23:04:53.981800+0000","last_unstale":"2024-02-21T23:04:53.986913+0000","last_undegraded":"2024-02-21T23:04:53.986913+0000","last_fullsized":"2024-02-21T23:04:53.986913+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:00:21.254168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.978586+0000","last_change":"2024-02-21T23:04:54.003361+0000","last_active":"2024-02-21T23:04:54.978586+0000","last_peered":"2024-02-21T23:04:54.978586+0000","last_clean":"2024-02-21T23:04:54.978586+0000","last_became_active":"2024-02-21T23:04:54.002829+0000","last_became_peered":"2024-02-21T23:04:54.002829+0000","last_unstale":"2024-02-21T23:04:54.978586+0000","last_undegraded":"2024-02-21T23:04:54.978586+0000","last_fullsized":"2024-02-21T23:04:54.978586+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:20:06.730366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-02-21T23:04:53.990505+0000","last_change":"2024-02-21T23:04:53.981560+0000","last_active":"2024-02-21T23:04:53.990505+0000","last_peered":"2024-02-21T23:04:53.990505+0000","last_clean":"2024-02-21T23:04:53.990505+0000","last_became_active":"2024-02-21T23:04:53.981260+0000","last_became_peered":"2024-02-21T23:04:53.981260+0000","last_unstale":"2024-02-21T23:04:53.990505+0000","last_undegraded":"2024-02-21T23:04:53.990505+0000","last_fullsized":"2024-02-21T23:04:53.990505+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:55:24.090577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-02-21T23:04:53.982444+0000","last_change":"2024-02-21T23:04:53.981072+0000","last_active":"2024-02-21T23:04:53.982444+0000","last_peered":"2024-02-21T23:04:53.982444+0000","last_clean":"2024-02-21T23:04:53.982444+0000","last_became_active":"2024-02-21T23:04:53.979351+0000","last_became_peered":"2024-02-21T23:04:53.979351+0000","last_unstale":"2024-02-21T23:04:53.982444+0000","last_undegraded":"2024-02-21T23:04:53.982444+0000","last_fullsized":"2024-02-21T23:04:53.982444+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:55:28.435901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.980429+0000","last_change":"2024-02-21T23:04:53.990161+0000","last_active":"2024-02-21T23:04:54.980429+0000","last_peered":"2024-02-21T23:04:54.980429+0000","last_clean":"2024-02-21T23:04:54.980429+0000","last_became_active":"2024-02-21T23:04:53.989849+0000","last_became_peered":"2024-02-21T23:04:53.989849+0000","last_unstale":"2024-02-21T23:04:54.980429+0000","last_undegraded":"2024-02-21T23:04:54.980429+0000","last_fullsized":"2024-02-21T23:04:54.980429+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:57:04.392889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-02-21T23:04:53.995850+0000","last_change":"2024-02-21T23:04:53.992512+0000","last_active":"2024-02-21T23:04:53.995850+0000","last_peered":"2024-02-21T23:04:53.995850+0000","last_clean":"2024-02-21T23:04:53.995850+0000","last_became_active":"2024-02-21T23:04:53.992255+0000","last_became_peered":"2024-02-21T23:04:53.992255+0000","last_unstale":"2024-02-21T23:04:53.995850+0000","last_undegraded":"2024-02-21T23:04:53.995850+0000","last_fullsized":"2024-02-21T23:04:53.995850+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:10:31.924664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.979230+0000","last_change":"2024-02-21T23:04:53.985272+0000","last_active":"2024-02-21T23:04:54.979230+0000","last_peered":"2024-02-21T23:04:54.979230+0000","last_clean":"2024-02-21T23:04:54.979230+0000","last_became_active":"2024-02-21T23:04:53.984967+0000","last_became_peered":"2024-02-21T23:04:53.984967+0000","last_unstale":"2024-02-21T23:04:54.979230+0000","last_undegraded":"2024-02-21T23:04:54.979230+0000","last_fullsized":"2024-02-21T23:04:54.979230+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:10:20.341856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.980991+0000","last_change":"2024-02-21T23:04:53.985751+0000","last_active":"2024-02-21T23:04:54.980991+0000","last_peered":"2024-02-21T23:04:54.980991+0000","last_clean":"2024-02-21T23:04:54.980991+0000","last_became_active":"2024-02-21T23:04:53.983212+0000","last_became_peered":"2024-02-21T23:04:53.983212+0000","last_unstale":"2024-02-21T23:04:54.980991+0000","last_undegraded":"2024-02-21T23:04:54.980991+0000","last_fullsized":"2024-02-21T23:04:54.980991+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:16:59.134374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.980617+0000","last_change":"2024-02-21T23:04:53.984599+0000","last_active":"2024-02-21T23:04:54.980617+0000","last_peered":"2024-02-21T23:04:54.980617+0000","last_clean":"2024-02-21T23:04:54.980617+0000","last_became_active":"2024-02-21T23:04:53.983405+0000","last_became_peered":"2024-02-21T23:04:53.983405+0000","last_unstale":"2024-02-21T23:04:54.980617+0000","last_undegraded":"2024-02-21T23:04:54.980617+0000","last_fullsized":"2024-02-21T23:04:54.980617+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:59:35.227750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.982096+0000","last_change":"2024-02-21T23:04:53.988824+0000","last_active":"2024-02-21T23:04:54.982096+0000","last_peered":"2024-02-21T23:04:54.982096+0000","last_clean":"2024-02-21T23:04:54.982096+0000","last_became_active":"2024-02-21T23:04:53.988190+0000","last_became_peered":"2024-02-21T23:04:53.988190+0000","last_unstale":"2024-02-21T23:04:54.982096+0000","last_undegraded":"2024-02-21T23:04:54.982096+0000","last_fullsized":"2024-02-21T23:04:54.982096+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:59:41.759091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-02-21T23:04:53.994679+0000","last_change":"2024-02-21T23:04:53.989836+0000","last_active":"2024-02-21T23:04:53.994679+0000","last_peered":"2024-02-21T23:04:53.994679+0000","last_clean":"2024-02-21T23:04:53.994679+0000","last_became_active":"2024-02-21T23:04:53.989511+0000","last_became_peered":"2024-02-21T23:04:53.989511+0000","last_unstale":"2024-02-21T23:04:53.994679+0000","last_undegraded":"2024-02-21T23:04:53.994679+0000","last_fullsized":"2024-02-21T23:04:53.994679+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:58:25.719654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.977619+0000","last_change":"2024-02-21T23:04:53.980319+0000","last_active":"2024-02-21T23:04:54.977619+0000","last_peered":"2024-02-21T23:04:54.977619+0000","last_clean":"2024-02-21T23:04:54.977619+0000","last_became_active":"2024-02-21T23:04:53.980056+0000","last_became_peered":"2024-02-21T23:04:53.980056+0000","last_unstale":"2024-02-21T23:04:54.977619+0000","last_undegraded":"2024-02-21T23:04:54.977619+0000","last_fullsized":"2024-02-21T23:04:54.977619+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:27:08.923372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.980833+0000","last_change":"2024-02-21T23:04:53.994043+0000","last_active":"2024-02-21T23:04:54.980833+0000","last_peered":"2024-02-21T23:04:54.980833+0000","last_clean":"2024-02-21T23:04:54.980833+0000","last_became_active":"2024-02-21T23:04:53.993693+0000","last_became_peered":"2024-02-21T23:04:53.993693+0000","last_unstale":"2024-02-21T23:04:54.980833+0000","last_undegraded":"2024-02-21T23:04:54.980833+0000","last_fullsized":"2024-02-21T23:04:54.980833+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:33:53.507158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.982154+0000","last_change":"2024-02-21T23:04:53.994446+0000","last_active":"2024-02-21T23:04:54.982154+0000","last_peered":"2024-02-21T23:04:54.982154+0000","last_clean":"2024-02-21T23:04:54.982154+0000","last_became_active":"2024-02-21T23:04:53.994181+0000","last_became_peered":"2024-02-21T23:04:53.994181+0000","last_unstale":"2024-02-21T23:04:54.982154+0000","last_undegraded":"2024-02-21T23:04:54.982154+0000","last_fullsized":"2024-02-21T23:04:54.982154+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:18:47.785124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-02-21T23:04:53.977305+0000","last_change":"2024-02-21T23:04:53.976184+0000","last_active":"2024-02-21T23:04:53.977305+0000","last_peered":"2024-02-21T23:04:53.977305+0000","last_clean":"2024-02-21T23:04:53.977305+0000","last_became_active":"2024-02-21T23:04:53.975098+0000","last_became_peered":"2024-02-21T23:04:53.975098+0000","last_unstale":"2024-02-21T23:04:53.977305+0000","last_undegraded":"2024-02-21T23:04:53.977305+0000","last_fullsized":"2024-02-21T23:04:53.977305+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:12:03.573396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.981130+0000","last_change":"2024-02-21T23:04:53.995281+0000","last_active":"2024-02-21T23:04:54.981130+0000","last_peered":"2024-02-21T23:04:54.981130+0000","last_clean":"2024-02-21T23:04:54.981130+0000","last_became_active":"2024-02-21T23:04:53.994940+0000","last_became_peered":"2024-02-21T23:04:53.994940+0000","last_unstale":"2024-02-21T23:04:54.981130+0000","last_undegraded":"2024-02-21T23:04:54.981130+0000","last_fullsized":"2024-02-21T23:04:54.981130+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:55:04.908945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.978313+0000","last_change":"2024-02-21T23:04:53.972067+0000","last_active":"2024-02-21T23:04:54.978313+0000","last_peered":"2024-02-21T23:04:54.978313+0000","last_clean":"2024-02-21T23:04:54.978313+0000","last_became_active":"2024-02-21T23:04:53.971855+0000","last_became_peered":"2024-02-21T23:04:53.971855+0000","last_unstale":"2024-02-21T23:04:54.978313+0000","last_undegraded":"2024-02-21T23:04:54.978313+0000","last_fullsized":"2024-02-21T23:04:54.978313+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:40:20.534202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-02-21T23:04:53.990960+0000","last_change":"2024-02-21T23:04:53.983537+0000","last_active":"2024-02-21T23:04:53.990960+0000","last_peered":"2024-02-21T23:04:53.990960+0000","last_clean":"2024-02-21T23:04:53.990960+0000","last_became_active":"2024-02-21T23:04:53.983264+0000","last_became_peered":"2024-02-21T23:04:53.983264+0000","last_unstale":"2024-02-21T23:04:53.990960+0000","last_undegraded":"2024-02-21T23:04:53.990960+0000","last_fullsized":"2024-02-21T23:04:53.990960+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:16:59.134374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.978083+0000","last_change":"2024-02-21T23:04:54.002341+0000","last_active":"2024-02-21T23:04:54.978083+0000","last_peered":"2024-02-21T23:04:54.978083+0000","last_clean":"2024-02-21T23:04:54.978083+0000","last_became_active":"2024-02-21T23:04:54.002066+0000","last_became_peered":"2024-02-21T23:04:54.002066+0000","last_unstale":"2024-02-21T23:04:54.978083+0000","last_undegraded":"2024-02-21T23:04:54.978083+0000","last_fullsized":"2024-02-21T23:04:54.978083+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:51:12.681873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:55.495664+0000","last_change":"2024-02-21T23:04:53.996395+0000","last_active":"2024-02-21T23:04:55.495664+0000","last_peered":"2024-02-21T23:04:55.495664+0000","last_clean":"2024-02-21T23:04:55.495664+0000","last_became_active":"2024-02-21T23:04:53.996219+0000","last_became_peered":"2024-02-21T23:04:53.996219+0000","last_unstale":"2024-02-21T23:04:55.495664+0000","last_undegraded":"2024-02-21T23:04:55.495664+0000","last_fullsized":"2024-02-21T23:04:55.495664+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:27:17.374378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-02-21T23:04:53.995648+0000","last_change":"2024-02-21T23:04:53.991752+0000","last_active":"2024-02-21T23:04:53.995648+0000","last_peered":"2024-02-21T23:04:53.995648+0000","last_clean":"2024-02-21T23:04:53.995648+0000","last_became_active":"2024-02-21T23:04:53.991458+0000","last_became_peered":"2024-02-21T23:04:53.991458+0000","last_unstale":"2024-02-21T23:04:53.995648+0000","last_undegraded":"2024-02-21T23:04:53.995648+0000","last_fullsized":"2024-02-21T23:04:53.995648+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:55:04.908945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"52'4","reported_seq":18,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.980851+0000","last_change":"2024-02-21T23:04:52.982273+0000","last_active":"2024-02-21T23:04:54.980851+0000","last_peered":"2024-02-21T23:04:54.980851+0000","last_clean":"2024-02-21T23:04:54.980851+0000","last_became_active":"2024-02-21T23:04:52.981799+0000","last_became_peered":"2024-02-21T23:04:52.981799+0000","last_unstale":"2024-02-21T23:04:54.980851+0000","last_undegraded":"2024-02-21T23:04:54.980851+0000","last_fullsized":"2024-02-21T23:04:54.980851+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:18:03.518976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:55.495283+0000","last_change":"2024-02-21T23:04:53.991059+0000","last_active":"2024-02-21T23:04:55.495283+0000","last_peered":"2024-02-21T23:04:55.495283+0000","last_clean":"2024-02-21T23:04:55.495283+0000","last_became_active":"2024-02-21T23:04:53.990310+0000","last_became_peered":"2024-02-21T23:04:53.990310+0000","last_unstale":"2024-02-21T23:04:55.495283+0000","last_undegraded":"2024-02-21T23:04:55.495283+0000","last_fullsized":"2024-02-21T23:04:55.495283+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:12:19.379347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"52'1","reported_seq":13,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-02-21T23:04:53.995137+0000","last_change":"2024-02-21T23:04:52.971612+0000","last_active":"2024-02-21T23:04:53.995137+0000","last_peered":"2024-02-21T23:04:53.995137+0000","last_clean":"2024-02-21T23:04:53.995137+0000","last_became_active":"2024-02-21T23:04:52.971220+0000","last_became_peered":"2024-02-21T23:04:52.971220+0000","last_unstale":"2024-02-21T23:04:53.995137+0000","last_undegraded":"2024-02-21T23:04:53.995137+0000","last_fullsized":"2024-02-21T23:04:53.995137+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:18:03.518976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.980148+0000","last_change":"2024-02-21T23:04:53.988422+0000","last_active":"2024-02-21T23:04:54.980148+0000","last_peered":"2024-02-21T23:04:54.980148+0000","last_clean":"2024-02-21T23:04:54.980148+0000","last_became_active":"2024-02-21T23:04:53.987586+0000","last_became_peered":"2024-02-21T23:04:53.987586+0000","last_unstale":"2024-02-21T23:04:54.980148+0000","last_undegraded":"2024-02-21T23:04:54.980148+0000","last_fullsized":"2024-02-21T23:04:54.980148+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:06:08.079155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"52'2","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-02-21T23:04:53.979782+0000","last_change":"2024-02-21T23:04:52.983315+0000","last_active":"2024-02-21T23:04:53.979782+0000","last_peered":"2024-02-21T23:04:53.979782+0000","last_clean":"2024-02-21T23:04:53.979782+0000","last_became_active":"2024-02-21T23:04:52.982925+0000","last_became_peered":"2024-02-21T23:04:52.982925+0000","last_unstale":"2024-02-21T23:04:53.979782+0000","last_undegraded":"2024-02-21T23:04:53.979782+0000","last_fullsized":"2024-02-21T23:04:53.979782+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:04:02.374574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.983140+0000","last_change":"2024-02-21T23:04:53.997493+0000","last_active":"2024-02-21T23:04:54.983140+0000","last_peered":"2024-02-21T23:04:54.983140+0000","last_clean":"2024-02-21T23:04:54.983140+0000","last_became_active":"2024-02-21T23:04:53.997216+0000","last_became_peered":"2024-02-21T23:04:53.997216+0000","last_unstale":"2024-02-21T23:04:54.983140+0000","last_undegraded":"2024-02-21T23:04:54.983140+0000","last_fullsized":"2024-02-21T23:04:54.983140+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:52:32.070580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"52'2","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:55.495553+0000","last_change":"2024-02-21T23:04:52.972332+0000","last_active":"2024-02-21T23:04:55.495553+0000","last_peered":"2024-02-21T23:04:55.495553+0000","last_clean":"2024-02-21T23:04:55.495553+0000","last_became_active":"2024-02-21T23:04:52.972014+0000","last_became_peered":"2024-02-21T23:04:52.972014+0000","last_unstale":"2024-02-21T23:04:55.495553+0000","last_undegraded":"2024-02-21T23:04:55.495553+0000","last_fullsized":"2024-02-21T23:04:55.495553+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:46:43.465994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:55.510996+0000","last_change":"2024-02-21T23:04:53.988566+0000","last_active":"2024-02-21T23:04:55.510996+0000","last_peered":"2024-02-21T23:04:55.510996+0000","last_clean":"2024-02-21T23:04:55.510996+0000","last_became_active":"2024-02-21T23:04:53.988271+0000","last_became_peered":"2024-02-21T23:04:53.988271+0000","last_unstale":"2024-02-21T23:04:55.510996+0000","last_undegraded":"2024-02-21T23:04:55.510996+0000","last_fullsized":"2024-02-21T23:04:55.510996+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:59:35.227750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"52'1","reported_seq":15,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.978450+0000","last_change":"2024-02-21T23:04:52.972962+0000","last_active":"2024-02-21T23:04:54.978450+0000","last_peered":"2024-02-21T23:04:54.978450+0000","last_clean":"2024-02-21T23:04:54.978450+0000","last_became_active":"2024-02-21T23:04:52.972642+0000","last_became_peered":"2024-02-21T23:04:52.972642+0000","last_unstale":"2024-02-21T23:04:54.978450+0000","last_undegraded":"2024-02-21T23:04:54.978450+0000","last_fullsized":"2024-02-21T23:04:54.978450+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:02:52.870656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-02-21T23:04:53.990210+0000","last_change":"2024-02-21T23:04:53.982723+0000","last_active":"2024-02-21T23:04:53.990210+0000","last_peered":"2024-02-21T23:04:53.990210+0000","last_clean":"2024-02-21T23:04:53.990210+0000","last_became_active":"2024-02-21T23:04:53.982447+0000","last_became_peered":"2024-02-21T23:04:53.982447+0000","last_unstale":"2024-02-21T23:04:53.990210+0000","last_undegraded":"2024-02-21T23:04:53.990210+0000","last_fullsized":"2024-02-21T23:04:53.990210+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:23:27.013837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"52'1","reported_seq":15,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:55.510227+0000","last_change":"2024-02-21T23:04:52.977866+0000","last_active":"2024-02-21T23:04:55.510227+0000","last_peered":"2024-02-21T23:04:55.510227+0000","last_clean":"2024-02-21T23:04:55.510227+0000","last_became_active":"2024-02-21T23:04:52.977477+0000","last_became_peered":"2024-02-21T23:04:52.977477+0000","last_unstale":"2024-02-21T23:04:55.510227+0000","last_undegraded":"2024-02-21T23:04:55.510227+0000","last_fullsized":"2024-02-21T23:04:55.510227+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:25:58.605620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.982755+0000","last_change":"2024-02-21T23:04:53.983494+0000","last_active":"2024-02-21T23:04:54.982755+0000","last_peered":"2024-02-21T23:04:54.982755+0000","last_clean":"2024-02-21T23:04:54.982755+0000","last_became_active":"2024-02-21T23:04:53.983213+0000","last_became_peered":"2024-02-21T23:04:53.983213+0000","last_unstale":"2024-02-21T23:04:54.982755+0000","last_undegraded":"2024-02-21T23:04:54.982755+0000","last_fullsized":"2024-02-21T23:04:54.982755+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:53:11.354446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":14,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.979914+0000","last_change":"2024-02-21T23:04:52.976990+0000","last_active":"2024-02-21T23:04:54.979914+0000","last_peered":"2024-02-21T23:04:54.979914+0000","last_clean":"2024-02-21T23:04:54.979914+0000","last_became_active":"2024-02-21T23:04:52.975637+0000","last_became_peered":"2024-02-21T23:04:52.975637+0000","last_unstale":"2024-02-21T23:04:54.979914+0000","last_undegraded":"2024-02-21T23:04:54.979914+0000","last_fullsized":"2024-02-21T23:04:54.979914+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:04:02.374574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-02-21T23:04:53.990450+0000","last_change":"2024-02-21T23:04:53.981493+0000","last_active":"2024-02-21T23:04:53.990450+0000","last_peered":"2024-02-21T23:04:53.990450+0000","last_clean":"2024-02-21T23:04:53.990450+0000","last_became_active":"2024-02-21T23:04:53.981109+0000","last_became_peered":"2024-02-21T23:04:53.981109+0000","last_unstale":"2024-02-21T23:04:53.990450+0000","last_undegraded":"2024-02-21T23:04:53.990450+0000","last_fullsized":"2024-02-21T23:04:53.990450+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:59:35.227750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":14,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.977743+0000","last_change":"2024-02-21T23:04:52.971808+0000","last_active":"2024-02-21T23:04:54.977743+0000","last_peered":"2024-02-21T23:04:54.977743+0000","last_clean":"2024-02-21T23:04:54.977743+0000","last_became_active":"2024-02-21T23:04:52.971529+0000","last_became_peered":"2024-02-21T23:04:52.971529+0000","last_unstale":"2024-02-21T23:04:54.977743+0000","last_undegraded":"2024-02-21T23:04:54.977743+0000","last_fullsized":"2024-02-21T23:04:54.977743+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:00:20.232950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:55.510298+0000","last_change":"2024-02-21T23:04:53.994186+0000","last_active":"2024-02-21T23:04:55.510298+0000","last_peered":"2024-02-21T23:04:55.510298+0000","last_clean":"2024-02-21T23:04:55.510298+0000","last_became_active":"2024-02-21T23:04:53.993902+0000","last_became_peered":"2024-02-21T23:04:53.993902+0000","last_unstale":"2024-02-21T23:04:55.510298+0000","last_undegraded":"2024-02-21T23:04:55.510298+0000","last_fullsized":"2024-02-21T23:04:55.510298+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:39:11.623072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-02-21T23:04:53.975623+0000","last_change":"2024-02-21T23:04:52.976339+0000","last_active":"2024-02-21T23:04:53.975623+0000","last_peered":"2024-02-21T23:04:53.975623+0000","last_clean":"2024-02-21T23:04:53.975623+0000","last_became_active":"2024-02-21T23:04:52.975958+0000","last_became_peered":"2024-02-21T23:04:52.975958+0000","last_unstale":"2024-02-21T23:04:53.975623+0000","last_undegraded":"2024-02-21T23:04:53.975623+0000","last_fullsized":"2024-02-21T23:04:53.975623+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:06:07.057936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:55.495463+0000","last_change":"2024-02-21T23:04:53.980747+0000","last_active":"2024-02-21T23:04:55.495463+0000","last_peered":"2024-02-21T23:04:55.495463+0000","last_clean":"2024-02-21T23:04:55.495463+0000","last_became_active":"2024-02-21T23:04:53.980456+0000","last_became_peered":"2024-02-21T23:04:53.980456+0000","last_unstale":"2024-02-21T23:04:55.495463+0000","last_undegraded":"2024-02-21T23:04:55.495463+0000","last_fullsized":"2024-02-21T23:04:55.495463+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:40:20.534202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":14,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:55.510657+0000","last_change":"2024-02-21T23:04:52.974873+0000","last_active":"2024-02-21T23:04:55.510657+0000","last_peered":"2024-02-21T23:04:55.510657+0000","last_clean":"2024-02-21T23:04:55.510657+0000","last_became_active":"2024-02-21T23:04:52.973750+0000","last_became_peered":"2024-02-21T23:04:52.973750+0000","last_unstale":"2024-02-21T23:04:55.510657+0000","last_undegraded":"2024-02-21T23:04:55.510657+0000","last_fullsized":"2024-02-21T23:04:55.510657+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:04:02.374574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.979078+0000","last_change":"2024-02-21T23:04:53.983830+0000","last_active":"2024-02-21T23:04:54.979078+0000","last_peered":"2024-02-21T23:04:54.979078+0000","last_clean":"2024-02-21T23:04:54.979078+0000","last_became_active":"2024-02-21T23:04:53.974390+0000","last_became_peered":"2024-02-21T23:04:53.974390+0000","last_unstale":"2024-02-21T23:04:54.979078+0000","last_undegraded":"2024-02-21T23:04:54.979078+0000","last_fullsized":"2024-02-21T23:04:54.979078+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:27:17.374378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"52'3","reported_seq":15,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-02-21T23:04:53.983841+0000","last_change":"2024-02-21T23:04:52.976896+0000","last_active":"2024-02-21T23:04:53.983841+0000","last_peered":"2024-02-21T23:04:53.983841+0000","last_clean":"2024-02-21T23:04:53.983841+0000","last_became_active":"2024-02-21T23:04:52.976592+0000","last_became_peered":"2024-02-21T23:04:52.976592+0000","last_unstale":"2024-02-21T23:04:53.983841+0000","last_undegraded":"2024-02-21T23:04:53.983841+0000","last_fullsized":"2024-02-21T23:04:53.983841+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:39:31.156140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:55.510904+0000","last_change":"2024-02-21T23:04:53.997903+0000","last_active":"2024-02-21T23:04:55.510904+0000","last_peered":"2024-02-21T23:04:55.510904+0000","last_clean":"2024-02-21T23:04:55.510904+0000","last_became_active":"2024-02-21T23:04:53.997538+0000","last_became_peered":"2024-02-21T23:04:53.997538+0000","last_unstale":"2024-02-21T23:04:55.510904+0000","last_undegraded":"2024-02-21T23:04:55.510904+0000","last_fullsized":"2024-02-21T23:04:55.510904+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:55:24.090577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"52'6","reported_seq":20,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.982539+0000","last_change":"2024-02-21T23:04:52.975159+0000","last_active":"2024-02-21T23:04:54.982539+0000","last_peered":"2024-02-21T23:04:54.982539+0000","last_clean":"2024-02-21T23:04:54.982539+0000","last_became_active":"2024-02-21T23:04:52.974845+0000","last_became_peered":"2024-02-21T23:04:52.974845+0000","last_unstale":"2024-02-21T23:04:54.982539+0000","last_undegraded":"2024-02-21T23:04:54.982539+0000","last_fullsized":"2024-02-21T23:04:54.982539+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:20:05.709148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:55.495151+0000","last_change":"2024-02-21T23:04:53.987920+0000","last_active":"2024-02-21T23:04:55.495151+0000","last_peered":"2024-02-21T23:04:55.495151+0000","last_clean":"2024-02-21T23:04:55.495151+0000","last_became_active":"2024-02-21T23:04:53.986231+0000","last_became_peered":"2024-02-21T23:04:53.986231+0000","last_unstale":"2024-02-21T23:04:55.495151+0000","last_undegraded":"2024-02-21T23:04:55.495151+0000","last_fullsized":"2024-02-21T23:04:55.495151+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:20:06.730366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"52'2","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.978579+0000","last_change":"2024-02-21T23:04:52.977012+0000","last_active":"2024-02-21T23:04:54.978579+0000","last_peered":"2024-02-21T23:04:54.978579+0000","last_clean":"2024-02-21T23:04:54.978579+0000","last_became_active":"2024-02-21T23:04:52.973324+0000","last_became_peered":"2024-02-21T23:04:52.973324+0000","last_unstale":"2024-02-21T23:04:54.978579+0000","last_undegraded":"2024-02-21T23:04:54.978579+0000","last_fullsized":"2024-02-21T23:04:54.978579+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:55:27.414683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.979416+0000","last_change":"2024-02-21T23:04:53.969380+0000","last_active":"2024-02-21T23:04:54.979416+0000","last_peered":"2024-02-21T23:04:54.979416+0000","last_clean":"2024-02-21T23:04:54.979416+0000","last_became_active":"2024-02-21T23:04:53.966766+0000","last_became_peered":"2024-02-21T23:04:53.966766+0000","last_unstale":"2024-02-21T23:04:54.979416+0000","last_undegraded":"2024-02-21T23:04:54.979416+0000","last_fullsized":"2024-02-21T23:04:54.979416+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:22:45.699161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":14,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.980224+0000","last_change":"2024-02-21T23:04:52.981590+0000","last_active":"2024-02-21T23:04:54.980224+0000","last_peered":"2024-02-21T23:04:54.980224+0000","last_clean":"2024-02-21T23:04:54.980224+0000","last_became_active":"2024-02-21T23:04:52.981228+0000","last_became_peered":"2024-02-21T23:04:52.981228+0000","last_unstale":"2024-02-21T23:04:54.980224+0000","last_undegraded":"2024-02-21T23:04:54.980224+0000","last_fullsized":"2024-02-21T23:04:54.980224+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:25:58.605620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.982607+0000","last_change":"2024-02-21T23:04:53.983659+0000","last_active":"2024-02-21T23:04:54.982607+0000","last_peered":"2024-02-21T23:04:54.982607+0000","last_clean":"2024-02-21T23:04:54.982607+0000","last_became_active":"2024-02-21T23:04:53.983290+0000","last_became_peered":"2024-02-21T23:04:53.983290+0000","last_unstale":"2024-02-21T23:04:54.982607+0000","last_undegraded":"2024-02-21T23:04:54.982607+0000","last_fullsized":"2024-02-21T23:04:54.982607+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:05:20.754391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-02-21T23:04:53.975277+0000","last_change":"2024-02-21T23:04:52.982825+0000","last_active":"2024-02-21T23:04:53.975277+0000","last_peered":"2024-02-21T23:04:53.975277+0000","last_clean":"2024-02-21T23:04:53.975277+0000","last_became_active":"2024-02-21T23:04:52.982523+0000","last_became_peered":"2024-02-21T23:04:52.982523+0000","last_unstale":"2024-02-21T23:04:53.975277+0000","last_undegraded":"2024-02-21T23:04:53.975277+0000","last_fullsized":"2024-02-21T23:04:53.975277+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:45:49.617039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.980183+0000","last_change":"2024-02-21T23:04:53.988578+0000","last_active":"2024-02-21T23:04:54.980183+0000","last_peered":"2024-02-21T23:04:54.980183+0000","last_clean":"2024-02-21T23:04:54.980183+0000","last_became_active":"2024-02-21T23:04:53.988310+0000","last_became_peered":"2024-02-21T23:04:53.988310+0000","last_unstale":"2024-02-21T23:04:54.980183+0000","last_undegraded":"2024-02-21T23:04:54.980183+0000","last_fullsized":"2024-02-21T23:04:54.980183+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:17:58.257074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"52'8","reported_seq":22,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.981720+0000","last_change":"2024-02-21T23:04:52.978259+0000","last_active":"2024-02-21T23:04:54.981720+0000","last_peered":"2024-02-21T23:04:54.981720+0000","last_clean":"2024-02-21T23:04:54.981720+0000","last_became_active":"2024-02-21T23:04:52.975275+0000","last_became_peered":"2024-02-21T23:04:52.975275+0000","last_unstale":"2024-02-21T23:04:54.981720+0000","last_undegraded":"2024-02-21T23:04:54.981720+0000","last_fullsized":"2024-02-21T23:04:54.981720+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:20:55.242700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.981819+0000","last_change":"2024-02-21T23:04:53.988231+0000","last_active":"2024-02-21T23:04:54.981819+0000","last_peered":"2024-02-21T23:04:54.981819+0000","last_clean":"2024-02-21T23:04:54.981819+0000","last_became_active":"2024-02-21T23:04:53.987814+0000","last_became_peered":"2024-02-21T23:04:53.987814+0000","last_unstale":"2024-02-21T23:04:54.981819+0000","last_undegraded":"2024-02-21T23:04:54.981819+0000","last_fullsized":"2024-02-21T23:04:54.981819+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:37:47.491317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"52'1","reported_seq":15,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.980694+0000","last_change":"2024-02-21T23:04:52.981262+0000","last_active":"2024-02-21T23:04:54.980694+0000","last_peered":"2024-02-21T23:04:54.980694+0000","last_clean":"2024-02-21T23:04:54.980694+0000","last_became_active":"2024-02-21T23:04:52.980981+0000","last_became_peered":"2024-02-21T23:04:52.980981+0000","last_unstale":"2024-02-21T23:04:54.980694+0000","last_undegraded":"2024-02-21T23:04:54.980694+0000","last_fullsized":"2024-02-21T23:04:54.980694+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:45:49.617039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-02-21T23:04:53.972934+0000","last_change":"2024-02-21T23:04:53.967738+0000","last_active":"2024-02-21T23:04:53.972934+0000","last_peered":"2024-02-21T23:04:53.972934+0000","last_clean":"2024-02-21T23:04:53.972934+0000","last_became_active":"2024-02-21T23:04:53.967430+0000","last_became_peered":"2024-02-21T23:04:53.967430+0000","last_unstale":"2024-02-21T23:04:53.972934+0000","last_undegraded":"2024-02-21T23:04:53.972934+0000","last_fullsized":"2024-02-21T23:04:53.972934+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:17:15.897726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"52'2","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.001287+0000","last_change":"2024-02-21T23:04:52.966722+0000","last_active":"2024-02-21T23:04:54.001287+0000","last_peered":"2024-02-21T23:04:54.001287+0000","last_clean":"2024-02-21T23:04:54.001287+0000","last_became_active":"2024-02-21T23:04:52.966322+0000","last_became_peered":"2024-02-21T23:04:52.966322+0000","last_unstale":"2024-02-21T23:04:54.001287+0000","last_undegraded":"2024-02-21T23:04:54.001287+0000","last_fullsized":"2024-02-21T23:04:54.001287+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:29:42.084956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:55.510656+0000","last_change":"2024-02-21T23:04:53.989868+0000","last_active":"2024-02-21T23:04:55.510656+0000","last_peered":"2024-02-21T23:04:55.510656+0000","last_clean":"2024-02-21T23:04:55.510656+0000","last_became_active":"2024-02-21T23:04:53.989585+0000","last_became_peered":"2024-02-21T23:04:53.989585+0000","last_unstale":"2024-02-21T23:04:55.510656+0000","last_undegraded":"2024-02-21T23:04:55.510656+0000","last_fullsized":"2024-02-21T23:04:55.510656+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:06:08.079155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"52'1","reported_seq":15,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.980899+0000","last_change":"2024-02-21T23:04:52.976203+0000","last_active":"2024-02-21T23:04:54.980899+0000","last_peered":"2024-02-21T23:04:54.980899+0000","last_clean":"2024-02-21T23:04:54.980899+0000","last_became_active":"2024-02-21T23:04:52.975787+0000","last_became_peered":"2024-02-21T23:04:52.975787+0000","last_unstale":"2024-02-21T23:04:54.980899+0000","last_undegraded":"2024-02-21T23:04:54.980899+0000","last_fullsized":"2024-02-21T23:04:54.980899+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:02:52.870656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-02-21T23:04:53.989033+0000","last_change":"2024-02-21T23:04:53.986757+0000","last_active":"2024-02-21T23:04:53.989033+0000","last_peered":"2024-02-21T23:04:53.989033+0000","last_clean":"2024-02-21T23:04:53.989033+0000","last_became_active":"2024-02-21T23:04:53.982468+0000","last_became_peered":"2024-02-21T23:04:53.982468+0000","last_unstale":"2024-02-21T23:04:53.989033+0000","last_undegraded":"2024-02-21T23:04:53.989033+0000","last_fullsized":"2024-02-21T23:04:53.989033+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:39:11.623072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":14,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.981260+0000","last_change":"2024-02-21T23:04:52.978566+0000","last_active":"2024-02-21T23:04:54.981260+0000","last_peered":"2024-02-21T23:04:54.981260+0000","last_clean":"2024-02-21T23:04:54.981260+0000","last_became_active":"2024-02-21T23:04:52.978262+0000","last_became_peered":"2024-02-21T23:04:52.978262+0000","last_unstale":"2024-02-21T23:04:54.981260+0000","last_undegraded":"2024-02-21T23:04:54.981260+0000","last_fullsized":"2024-02-21T23:04:54.981260+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:12:02.552178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:55.495478+0000","last_change":"2024-02-21T23:04:53.982199+0000","last_active":"2024-02-21T23:04:55.495478+0000","last_peered":"2024-02-21T23:04:55.495478+0000","last_clean":"2024-02-21T23:04:55.495478+0000","last_became_active":"2024-02-21T23:04:53.981777+0000","last_became_peered":"2024-02-21T23:04:53.981777+0000","last_unstale":"2024-02-21T23:04:55.495478+0000","last_undegraded":"2024-02-21T23:04:55.495478+0000","last_fullsized":"2024-02-21T23:04:55.495478+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:52:43.502544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"52'2","reported_seq":14,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-02-21T23:04:53.988401+0000","last_change":"2024-02-21T23:04:52.961423+0000","last_active":"2024-02-21T23:04:53.988401+0000","last_peered":"2024-02-21T23:04:53.988401+0000","last_clean":"2024-02-21T23:04:53.988401+0000","last_became_active":"2024-02-21T23:04:52.961090+0000","last_became_peered":"2024-02-21T23:04:52.961090+0000","last_unstale":"2024-02-21T23:04:53.988401+0000","last_undegraded":"2024-02-21T23:04:53.988401+0000","last_fullsized":"2024-02-21T23:04:53.988401+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:25:58.605620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.980739+0000","last_change":"2024-02-21T23:04:53.983031+0000","last_active":"2024-02-21T23:04:54.980739+0000","last_peered":"2024-02-21T23:04:54.980739+0000","last_clean":"2024-02-21T23:04:54.980739+0000","last_became_active":"2024-02-21T23:04:53.981770+0000","last_became_peered":"2024-02-21T23:04:53.981770+0000","last_unstale":"2024-02-21T23:04:54.980739+0000","last_undegraded":"2024-02-21T23:04:54.980739+0000","last_fullsized":"2024-02-21T23:04:54.980739+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:23:27.013837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":14,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.981729+0000","last_change":"2024-02-21T23:04:52.975453+0000","last_active":"2024-02-21T23:04:54.981729+0000","last_peered":"2024-02-21T23:04:54.981729+0000","last_clean":"2024-02-21T23:04:54.981729+0000","last_became_active":"2024-02-21T23:04:52.975121+0000","last_became_peered":"2024-02-21T23:04:52.975121+0000","last_unstale":"2024-02-21T23:04:54.981729+0000","last_undegraded":"2024-02-21T23:04:54.981729+0000","last_fullsized":"2024-02-21T23:04:54.981729+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:51:11.660655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.979723+0000","last_change":"2024-02-21T23:04:53.978362+0000","last_active":"2024-02-21T23:04:54.979723+0000","last_peered":"2024-02-21T23:04:54.979723+0000","last_clean":"2024-02-21T23:04:54.979723+0000","last_became_active":"2024-02-21T23:04:53.978070+0000","last_became_peered":"2024-02-21T23:04:53.978070+0000","last_unstale":"2024-02-21T23:04:54.979723+0000","last_undegraded":"2024-02-21T23:04:54.979723+0000","last_fullsized":"2024-02-21T23:04:54.979723+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:52:55.798286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":14,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.978816+0000","last_change":"2024-02-21T23:04:52.979855+0000","last_active":"2024-02-21T23:04:54.978816+0000","last_peered":"2024-02-21T23:04:54.978816+0000","last_clean":"2024-02-21T23:04:54.978816+0000","last_became_active":"2024-02-21T23:04:52.977225+0000","last_became_peered":"2024-02-21T23:04:52.977225+0000","last_unstale":"2024-02-21T23:04:54.978816+0000","last_undegraded":"2024-02-21T23:04:54.978816+0000","last_fullsized":"2024-02-21T23:04:54.978816+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:39:10.601854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.000684+0000","last_change":"2024-02-21T23:04:53.996234+0000","last_active":"2024-02-21T23:04:54.000684+0000","last_peered":"2024-02-21T23:04:54.000684+0000","last_clean":"2024-02-21T23:04:54.000684+0000","last_became_active":"2024-02-21T23:04:53.995859+0000","last_became_peered":"2024-02-21T23:04:53.995859+0000","last_unstale":"2024-02-21T23:04:54.000684+0000","last_undegraded":"2024-02-21T23:04:54.000684+0000","last_fullsized":"2024-02-21T23:04:54.000684+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:06:08.079155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-02-21T23:04:53.980209+0000","last_change":"2024-02-21T23:04:52.972729+0000","last_active":"2024-02-21T23:04:53.980209+0000","last_peered":"2024-02-21T23:04:53.980209+0000","last_clean":"2024-02-21T23:04:53.980209+0000","last_became_active":"2024-02-21T23:04:52.972434+0000","last_became_peered":"2024-02-21T23:04:52.972434+0000","last_unstale":"2024-02-21T23:04:53.980209+0000","last_undegraded":"2024-02-21T23:04:53.980209+0000","last_fullsized":"2024-02-21T23:04:53.980209+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:00:21.589361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:55.495836+0000","last_change":"2024-02-21T23:04:53.996387+0000","last_active":"2024-02-21T23:04:55.495836+0000","last_peered":"2024-02-21T23:04:55.495836+0000","last_clean":"2024-02-21T23:04:55.495836+0000","last_became_active":"2024-02-21T23:04:53.996123+0000","last_became_peered":"2024-02-21T23:04:53.996123+0000","last_unstale":"2024-02-21T23:04:55.495836+0000","last_undegraded":"2024-02-21T23:04:55.495836+0000","last_fullsized":"2024-02-21T23:04:55.495836+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:43:19.009080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"52'2","reported_seq":16,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.981127+0000","last_change":"2024-02-21T23:04:52.981486+0000","last_active":"2024-02-21T23:04:54.981127+0000","last_peered":"2024-02-21T23:04:54.981127+0000","last_clean":"2024-02-21T23:04:54.981127+0000","last_became_active":"2024-02-21T23:04:52.981135+0000","last_became_peered":"2024-02-21T23:04:52.981135+0000","last_unstale":"2024-02-21T23:04:54.981127+0000","last_undegraded":"2024-02-21T23:04:54.981127+0000","last_fullsized":"2024-02-21T23:04:54.981127+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:17:14.876508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-02-21T23:04:53.978346+0000","last_change":"2024-02-21T23:04:53.977118+0000","last_active":"2024-02-21T23:04:53.978346+0000","last_peered":"2024-02-21T23:04:53.978346+0000","last_clean":"2024-02-21T23:04:53.978346+0000","last_became_active":"2024-02-21T23:04:53.976042+0000","last_became_peered":"2024-02-21T23:04:53.976042+0000","last_unstale":"2024-02-21T23:04:53.978346+0000","last_undegraded":"2024-02-21T23:04:53.978346+0000","last_fullsized":"2024-02-21T23:04:53.978346+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:02:53.891874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":12,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-02-21T23:04:53.966474+0000","last_change":"2024-02-21T23:04:52.974536+0000","last_active":"2024-02-21T23:04:53.966474+0000","last_peered":"2024-02-21T23:04:53.966474+0000","last_clean":"2024-02-21T23:04:53.966474+0000","last_became_active":"2024-02-21T23:04:52.974221+0000","last_became_peered":"2024-02-21T23:04:52.974221+0000","last_unstale":"2024-02-21T23:04:53.966474+0000","last_undegraded":"2024-02-21T23:04:53.966474+0000","last_fullsized":"2024-02-21T23:04:53.966474+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:39:10.601854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.983327+0000","last_change":"2024-02-21T23:04:53.979161+0000","last_active":"2024-02-21T23:04:54.983327+0000","last_peered":"2024-02-21T23:04:54.983327+0000","last_clean":"2024-02-21T23:04:54.983327+0000","last_became_active":"2024-02-21T23:04:53.978843+0000","last_became_peered":"2024-02-21T23:04:53.978843+0000","last_unstale":"2024-02-21T23:04:54.983327+0000","last_undegraded":"2024-02-21T23:04:54.983327+0000","last_fullsized":"2024-02-21T23:04:54.983327+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:34:45.807165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":14,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.978706+0000","last_change":"2024-02-21T23:04:52.969804+0000","last_active":"2024-02-21T23:04:54.978706+0000","last_peered":"2024-02-21T23:04:54.978706+0000","last_clean":"2024-02-21T23:04:54.978706+0000","last_became_active":"2024-02-21T23:04:52.969353+0000","last_became_peered":"2024-02-21T23:04:52.969353+0000","last_unstale":"2024-02-21T23:04:54.978706+0000","last_undegraded":"2024-02-21T23:04:54.978706+0000","last_fullsized":"2024-02-21T23:04:54.978706+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:47:27.364576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.978582+0000","last_change":"2024-02-21T23:04:54.000144+0000","last_active":"2024-02-21T23:04:54.978582+0000","last_peered":"2024-02-21T23:04:54.978582+0000","last_clean":"2024-02-21T23:04:54.978582+0000","last_became_active":"2024-02-21T23:04:53.999687+0000","last_became_peered":"2024-02-21T23:04:53.999687+0000","last_unstale":"2024-02-21T23:04:54.978582+0000","last_undegraded":"2024-02-21T23:04:54.978582+0000","last_fullsized":"2024-02-21T23:04:54.978582+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:12:19.379347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":14,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.983433+0000","last_change":"2024-02-21T23:04:52.975668+0000","last_active":"2024-02-21T23:04:54.983433+0000","last_peered":"2024-02-21T23:04:54.983433+0000","last_clean":"2024-02-21T23:04:54.983433+0000","last_became_active":"2024-02-21T23:04:52.975348+0000","last_became_peered":"2024-02-21T23:04:52.975348+0000","last_unstale":"2024-02-21T23:04:54.983433+0000","last_undegraded":"2024-02-21T23:04:54.983433+0000","last_fullsized":"2024-02-21T23:04:54.983433+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:12:18.358129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:55.495731+0000","last_change":"2024-02-21T23:04:53.988119+0000","last_active":"2024-02-21T23:04:55.495731+0000","last_peered":"2024-02-21T23:04:55.495731+0000","last_clean":"2024-02-21T23:04:55.495731+0000","last_became_active":"2024-02-21T23:04:53.986237+0000","last_became_peered":"2024-02-21T23:04:53.986237+0000","last_unstale":"2024-02-21T23:04:55.495731+0000","last_undegraded":"2024-02-21T23:04:55.495731+0000","last_fullsized":"2024-02-21T23:04:55.495731+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:10:20.341856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":14,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.981876+0000","last_change":"2024-02-21T23:04:52.976653+0000","last_active":"2024-02-21T23:04:54.981876+0000","last_peered":"2024-02-21T23:04:54.981876+0000","last_clean":"2024-02-21T23:04:54.981876+0000","last_became_active":"2024-02-21T23:04:52.976249+0000","last_became_peered":"2024-02-21T23:04:52.976249+0000","last_unstale":"2024-02-21T23:04:54.981876+0000","last_undegraded":"2024-02-21T23:04:54.981876+0000","last_fullsized":"2024-02-21T23:04:54.981876+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:40:19.512984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.978897+0000","last_change":"2024-02-21T23:04:53.993076+0000","last_active":"2024-02-21T23:04:54.978897+0000","last_peered":"2024-02-21T23:04:54.978897+0000","last_clean":"2024-02-21T23:04:54.978897+0000","last_became_active":"2024-02-21T23:04:53.992734+0000","last_became_peered":"2024-02-21T23:04:53.992734+0000","last_unstale":"2024-02-21T23:04:54.978897+0000","last_undegraded":"2024-02-21T23:04:54.978897+0000","last_fullsized":"2024-02-21T23:04:54.978897+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:52:43.502544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":14,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:55.510160+0000","last_change":"2024-02-21T23:04:52.971065+0000","last_active":"2024-02-21T23:04:55.510160+0000","last_peered":"2024-02-21T23:04:55.510160+0000","last_clean":"2024-02-21T23:04:55.510160+0000","last_became_active":"2024-02-21T23:04:52.970410+0000","last_became_peered":"2024-02-21T23:04:52.970410+0000","last_unstale":"2024-02-21T23:04:55.510160+0000","last_undegraded":"2024-02-21T23:04:55.510160+0000","last_fullsized":"2024-02-21T23:04:55.510160+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:18:03.518976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.983009+0000","last_change":"2024-02-21T23:04:53.995393+0000","last_active":"2024-02-21T23:04:54.983009+0000","last_peered":"2024-02-21T23:04:54.983009+0000","last_clean":"2024-02-21T23:04:54.983009+0000","last_became_active":"2024-02-21T23:04:53.995131+0000","last_became_peered":"2024-02-21T23:04:53.995131+0000","last_unstale":"2024-02-21T23:04:54.983009+0000","last_undegraded":"2024-02-21T23:04:54.983009+0000","last_fullsized":"2024-02-21T23:04:54.983009+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:34:02.144778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":14,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:55.495907+0000","last_change":"2024-02-21T23:04:52.968036+0000","last_active":"2024-02-21T23:04:55.495907+0000","last_peered":"2024-02-21T23:04:55.495907+0000","last_clean":"2024-02-21T23:04:55.495907+0000","last_became_active":"2024-02-21T23:04:52.967626+0000","last_became_peered":"2024-02-21T23:04:52.967626+0000","last_unstale":"2024-02-21T23:04:55.495907+0000","last_undegraded":"2024-02-21T23:04:55.495907+0000","last_fullsized":"2024-02-21T23:04:55.495907+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:33:33.298554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":12,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.980186+0000","last_change":"2024-02-21T23:04:53.985061+0000","last_active":"2024-02-21T23:04:54.980186+0000","last_peered":"2024-02-21T23:04:54.980186+0000","last_clean":"2024-02-21T23:04:54.980186+0000","last_became_active":"2024-02-21T23:04:53.984756+0000","last_became_peered":"2024-02-21T23:04:53.984756+0000","last_unstale":"2024-02-21T23:04:54.980186+0000","last_undegraded":"2024-02-21T23:04:54.980186+0000","last_fullsized":"2024-02-21T23:04:54.980186+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:55:24.090577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":14,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-02-21T23:04:54.979720+0000","last_change":"2024-02-21T23:04:52.979253+0000","last_active":"2024-02-21T23:04:54.979720+0000","last_peered":"2024-02-21T23:04:54.979720+0000","last_clean":"2024-02-21T23:04:54.979720+0000","last_became_active":"2024-02-21T23:04:52.978773+0000","last_became_peered":"2024-02-21T23:04:52.978773+0000","last_unstale":"2024-02-21T23:04:54.979720+0000","last_undegraded":"2024-02-21T23:04:54.979720+0000","last_fullsized":"2024-02-21T23:04:54.979720+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:39:31.156140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":229,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-02-21T23:04:53.963786+0000","last_change":"2024-02-21T23:02:32.289335+0000","last_active":"2024-02-21T23:04:53.963786+0000","last_peered":"2024-02-21T23:04:53.963786+0000","last_clean":"2024-02-21T23:04:53.963786+0000","last_became_active":"2024-02-21T23:02:32.286913+0000","last_became_peered":"2024-02-21T23:02:32.286913+0000","last_unstale":"2024-02-21T23:04:53.963786+0000","last_undegraded":"2024-02-21T23:04:53.963786+0000","last_fullsized":"2024-02-21T23:04:53.963786+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_clean_scrub_stamp":"2024-02-21T23:02:31.172569+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:36:23.987374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574879,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27704,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344136,"statfs":{"total":96636764160,"available":96608395264,"internally_reserved":0,"allocated":692224,"data_stored":867144,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574880,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27648,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344192,"statfs":{"total":96636764160,"available":96608452608,"internally_reserved":0,"allocated":638976,"data_stored":283996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574879,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27628,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344212,"statfs":{"total":96636764160,"available":96608473088,"internally_reserved":0,"allocated":618496,"data_stored":276652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574880,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27632,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344208,"statfs":{"total":96636764160,"available":96608468992,"internally_reserved":0,"allocated":622592,"data_stored":277234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574880,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31820,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340020,"statfs":{"total":96636764160,"available":96604180480,"internally_reserved":0,"allocated":647168,"data_stored":284978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574881,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31800,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340040,"statfs":{"total":96636764160,"available":96604200960,"internally_reserved":0,"allocated":630784,"data_stored":278264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574879,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31796,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340044,"statfs":{"total":96636764160,"available":96604205056,"internally_reserved":0,"allocated":626688,"data_stored":276812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574879,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31864,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339976,"statfs":{"total":96636764160,"available":96604135424,"internally_reserved":0,"allocated":692224,"data_stored":867076,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-21T23:04:59.122 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-02-21T23:04:59.122 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-02-21T23:04:59.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.395+0000 7fcf5f800640 1 Processor -- start 2024-02-21T23:04:59.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.396+0000 7fcf5f800640 1 -- start start 2024-02-21T23:04:59.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.398+0000 7fcf5f800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcf580c6c60 0x7fcf580c3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:59.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.398+0000 7fcf5f800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf58159580 con 0x7fcf580c5cc0 2024-02-21T23:04:59.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.398+0000 7fcf5f800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf581587c0 con 0x7fcf580c7590 2024-02-21T23:04:59.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.398+0000 7fcf5f800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf580c6230 con 0x7fcf580c6c60 2024-02-21T23:04:59.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.398+0000 7fcf57e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fcf580c5cc0 0x7fcf580c3030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34484/0 (socket says 172.21.15.28:34484) 2024-02-21T23:04:59.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.398+0000 7fcf57e00640 1 -- 172.21.15.28:0/3307271824 learned_addr learned my addr 172.21.15.28:0/3307271824 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:04:59.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.398+0000 7fcf5d800640 1 --2- 172.21.15.28:0/3307271824 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcf580c6c60 0x7fcf580c3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:59.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.398+0000 7fcf57400640 1 -- 172.21.15.28:0/3307271824 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1454968485 0 0) 0x7fcf58159580 con 0x7fcf580c5cc0 2024-02-21T23:04:59.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.398+0000 7fcf57400640 1 -- 172.21.15.28:0/3307271824 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf40003520 con 0x7fcf580c5cc0 2024-02-21T23:04:59.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.399+0000 7fcf57400640 1 -- 172.21.15.28:0/3307271824 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1684286551 0 0) 0x7fcf40003520 con 0x7fcf580c5cc0 2024-02-21T23:04:59.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.399+0000 7fcf57400640 1 -- 172.21.15.28:0/3307271824 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcf580c6c60 msgr2=0x7fcf580c3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:59.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.399+0000 7fcf57400640 1 --2- 172.21.15.28:0/3307271824 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcf580c6c60 0x7fcf580c3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:59.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.399+0000 7fcf57400640 1 -- 172.21.15.28:0/3307271824 >> v1:172.21.15.81:6789/0 conn(0x7fcf580c7590 legacy=0x7fcf581524e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:59.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.399+0000 7fcf57400640 1 -- 172.21.15.28:0/3307271824 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf58158aa0 con 0x7fcf580c5cc0 2024-02-21T23:04:59.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.399+0000 7fcf57400640 1 -- 172.21.15.28:0/3307271824 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fcf44001670 con 0x7fcf580c5cc0 2024-02-21T23:04:59.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.399+0000 7fcf57400640 1 -- 172.21.15.28:0/3307271824 >> v1:172.21.15.28:6789/0 conn(0x7fcf580c5cc0 legacy=0x7fcf580c3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:59.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.399+0000 7fcf57400640 1 --2- 172.21.15.28:0/3307271824 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf400039e0 0x7fcf40003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:59.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.399+0000 7fcf57400640 1 --2- 172.21.15.28:0/3307271824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf40004800 0x7fcf40004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:59.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.399+0000 7fcf57400640 1 --2- 172.21.15.28:0/3307271824 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcf580c6c60 0x7fcf400051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:59.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.399+0000 7fcf57400640 1 -- 172.21.15.28:0/3307271824 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf580c6230 con 0x7fcf40004800 2024-02-21T23:04:59.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.399+0000 7fcf57400640 1 -- 172.21.15.28:0/3307271824 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf4000bac0 con 0x7fcf400039e0 2024-02-21T23:04:59.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.399+0000 7fcf57400640 1 -- 172.21.15.28:0/3307271824 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf4000bcc0 con 0x7fcf580c6c60 2024-02-21T23:04:59.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.399+0000 7fcf57e00640 1 --2- 172.21.15.28:0/3307271824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf40004800 0x7fcf40004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:59.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.400+0000 7fcf5ce00640 1 --2- 172.21.15.28:0/3307271824 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf400039e0 0x7fcf40003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:59.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.400+0000 7fcf5d800640 1 --2- 172.21.15.28:0/3307271824 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcf580c6c60 0x7fcf400051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:59.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.400+0000 7fcf57e00640 1 -- 172.21.15.28:0/3307271824 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcf580c6c60 msgr2=0x7fcf400051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:59.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.400+0000 7fcf57e00640 1 --2- 172.21.15.28:0/3307271824 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcf580c6c60 0x7fcf400051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:59.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.400+0000 7fcf57e00640 1 -- 172.21.15.28:0/3307271824 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf400039e0 msgr2=0x7fcf40003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:59.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.400+0000 7fcf57e00640 1 --2- 172.21.15.28:0/3307271824 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf400039e0 0x7fcf40003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:59.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.400+0000 7fcf57e00640 1 -- 172.21.15.28:0/3307271824 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf40003520 con 0x7fcf40004800 2024-02-21T23:04:59.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.401+0000 7fcf57e00640 1 --2- 172.21.15.28:0/3307271824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf40004800 0x7fcf40004c80 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fcf4400c190 tx=0x7fcf40005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e2afe121871fdfb server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:59.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.401+0000 7fcf57400640 1 -- 172.21.15.28:0/3307271824 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf44007c60 con 0x7fcf40004800 2024-02-21T23:04:59.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.401+0000 7fcf57400640 1 -- 172.21.15.28:0/3307271824 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf44030070 con 0x7fcf40004800 2024-02-21T23:04:59.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.402+0000 7fcf57400640 1 -- 172.21.15.28:0/3307271824 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf44005d10 con 0x7fcf40004800 2024-02-21T23:04:59.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.402+0000 7fcf5f800640 1 -- 172.21.15.28:0/3307271824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf40004800 msgr2=0x7fcf40004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:59.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.402+0000 7fcf5f800640 1 --2- 172.21.15.28:0/3307271824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf40004800 0x7fcf40004c80 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fcf4400c190 tx=0x7fcf40005830 comp rx=0 tx=0).stop 2024-02-21T23:04:59.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.402+0000 7fcf5d800640 1 -- 172.21.15.28:0/3307271824 reap_dead start 2024-02-21T23:04:59.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.402+0000 7fcf5f800640 1 -- 172.21.15.28:0/3307271824 shutdown_connections 2024-02-21T23:04:59.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.402+0000 7fcf5f800640 1 -- 172.21.15.28:0/3307271824 >> 172.21.15.28:0/3307271824 conn(0x7fcf5808afe0 msgr2=0x7fcf580725a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:59.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.402+0000 7fcf5f800640 1 -- 172.21.15.28:0/3307271824 shutdown_connections 2024-02-21T23:04:59.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.402+0000 7fcf5f800640 1 -- 172.21.15.28:0/3307271824 wait complete. 2024-02-21T23:04:59.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.404+0000 7fcf5f800640 1 Processor -- start 2024-02-21T23:04:59.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.404+0000 7fcf5f800640 1 -- start start 2024-02-21T23:04:59.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.405+0000 7fcf5f800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf40004800 0x7fcf580c0510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:59.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.405+0000 7fcf5f800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcf580c8730 0x7fcf58147590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:59.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.405+0000 7fcf5f800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf580c8f40 0x7fcf580c9c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:59.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.405+0000 7fcf5f800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf580c6720 con 0x7fcf40004800 2024-02-21T23:04:59.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.405+0000 7fcf5f800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf58159060 con 0x7fcf580c8f40 2024-02-21T23:04:59.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.405+0000 7fcf5f800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf580c68a0 con 0x7fcf580c8730 2024-02-21T23:04:59.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.405+0000 7fcf5ce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf40004800 0x7fcf580c0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:59.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.405+0000 7fcf5ce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf40004800 0x7fcf580c0510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45610/0 (socket says 172.21.15.28:45610) 2024-02-21T23:04:59.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.405+0000 7fcf5ce00640 1 -- 172.21.15.28:0/4017593957 learned_addr learned my addr 172.21.15.28:0/4017593957 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:59.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.406+0000 7fcf5d800640 1 --2- 172.21.15.28:0/4017593957 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf580c8f40 0x7fcf580c9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:59.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.406+0000 7fcf5ce00640 1 -- 172.21.15.28:0/4017593957 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcf580c8730 msgr2=0x7fcf58147590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:59.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.406+0000 7fcf5ce00640 1 --2- 172.21.15.28:0/4017593957 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcf580c8730 0x7fcf58147590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:59.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.406+0000 7fcf5ce00640 1 -- 172.21.15.28:0/4017593957 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf580c8f40 msgr2=0x7fcf580c9c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:59.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.406+0000 7fcf5ce00640 1 --2- 172.21.15.28:0/4017593957 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf580c8f40 0x7fcf580c9c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:59.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.406+0000 7fcf5ce00640 1 -- 172.21.15.28:0/4017593957 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf580c0a50 con 0x7fcf40004800 2024-02-21T23:04:59.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.406+0000 7fcf5d800640 1 --2- 172.21.15.28:0/4017593957 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf580c8f40 0x7fcf580c9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:04:59.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.406+0000 7fcf5ce00640 1 --2- 172.21.15.28:0/4017593957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf40004800 0x7fcf580c0510 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fcf4c00e620 tx=0x7fcf4c00eae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:59.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.406+0000 7fcf54c00640 1 -- 172.21.15.28:0/4017593957 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf4c0052e0 con 0x7fcf40004800 2024-02-21T23:04:59.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.407+0000 7fcf54c00640 1 -- 172.21.15.28:0/4017593957 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf4c018070 con 0x7fcf40004800 2024-02-21T23:04:59.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.407+0000 7fcf54c00640 1 -- 172.21.15.28:0/4017593957 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf4c003830 con 0x7fcf40004800 2024-02-21T23:04:59.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.407+0000 7fcf5f800640 1 -- 172.21.15.28:0/4017593957 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf580bffe0 con 0x7fcf40004800 2024-02-21T23:04:59.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.407+0000 7fcf5f800640 1 -- 172.21.15.28:0/4017593957 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf580bfa10 con 0x7fcf40004800 2024-02-21T23:04:59.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.408+0000 7fcf5f800640 1 -- 172.21.15.28:0/4017593957 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf581ef1d0 con 0x7fcf40004800 2024-02-21T23:04:59.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.409+0000 7fcf54c00640 1 -- 172.21.15.28:0/4017593957 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcf4c0039d0 con 0x7fcf40004800 2024-02-21T23:04:59.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.410+0000 7fcf54c00640 1 --2- 172.21.15.28:0/4017593957 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcf3806b500 0x7fcf3806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:59.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.410+0000 7fcf54c00640 1 -- 172.21.15.28:0/4017593957 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fcf4c08a110 con 0x7fcf40004800 2024-02-21T23:04:59.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.411+0000 7fcf57e00640 1 --2- 172.21.15.28:0/4017593957 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcf3806b500 0x7fcf3806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:59.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.412+0000 7fcf57e00640 1 --2- 172.21.15.28:0/4017593957 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcf3806b500 0x7fcf3806d9c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fcf44034000 tx=0x7fcf44039040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:59.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.414+0000 7fcf54c00640 1 -- 172.21.15.28:0/4017593957 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fcf581ef1d0 con 0x7fcf40004800 2024-02-21T23:04:59.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.694+0000 7fcf5f800640 1 -- 172.21.15.28:0/4017593957 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fcf58150330 con 0x7fcf40004800 2024-02-21T23:04:59.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.699+0000 7fcf54c00640 1 -- 172.21.15.28:0/4017593957 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v12) v1 ==== 190+0+147 (secure 0 0 0) 0x7fcf4c058f50 con 0x7fcf40004800 2024-02-21T23:04:59.689 INFO:teuthology.orchestra.run.smithi028.stdout:[client.0] 2024-02-21T23:04:59.689 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQDTgNZlILvYLxAAoREWxKKT7YcXnhkgk8YvoQ== 2024-02-21T23:04:59.689 INFO:teuthology.orchestra.run.smithi028.stdout: caps mds = "allow" 2024-02-21T23:04:59.689 INFO:teuthology.orchestra.run.smithi028.stdout: caps mon = "allow r" 2024-02-21T23:04:59.689 INFO:teuthology.orchestra.run.smithi028.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-02-21T23:04:59.689 INFO:teuthology.orchestra.run.smithi028.stderr:updated caps for client.0 2024-02-21T23:04:59.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.702+0000 7fcf5f800640 1 -- 172.21.15.28:0/4017593957 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcf3806b500 msgr2=0x7fcf3806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:59.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.702+0000 7fcf5f800640 1 --2- 172.21.15.28:0/4017593957 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcf3806b500 0x7fcf3806d9c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fcf44034000 tx=0x7fcf44039040 comp rx=0 tx=0).stop 2024-02-21T23:04:59.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.702+0000 7fcf5f800640 1 -- 172.21.15.28:0/4017593957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf40004800 msgr2=0x7fcf580c0510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:59.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.702+0000 7fcf5f800640 1 --2- 172.21.15.28:0/4017593957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf40004800 0x7fcf580c0510 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fcf4c00e620 tx=0x7fcf4c00eae0 comp rx=0 tx=0).stop 2024-02-21T23:04:59.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.702+0000 7fcf5f800640 1 -- 172.21.15.28:0/4017593957 shutdown_connections 2024-02-21T23:04:59.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.702+0000 7fcf5f800640 1 --2- 172.21.15.28:0/4017593957 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcf3806b500 0x7fcf3806d9c0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:59.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.702+0000 7fcf5f800640 1 --2- 172.21.15.28:0/4017593957 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcf580c8730 0x7fcf58147590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:59.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.702+0000 7fcf5f800640 1 --2- 172.21.15.28:0/4017593957 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf580c8f40 0x7fcf580c9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:59.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.702+0000 7fcf5f800640 1 --2- 172.21.15.28:0/4017593957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf40004800 0x7fcf580c0510 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:59.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.702+0000 7fcf5f800640 1 -- 172.21.15.28:0/4017593957 >> 172.21.15.28:0/4017593957 conn(0x7fcf5808afe0 msgr2=0x7fcf580725a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:59.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.702+0000 7fcf5f800640 1 -- 172.21.15.28:0/4017593957 shutdown_connections 2024-02-21T23:04:59.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.702+0000 7fcf5f800640 1 -- 172.21.15.28:0/4017593957 wait complete. 2024-02-21T23:04:59.711 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:04:59.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.934+0000 7fb7d9600640 1 Processor -- start 2024-02-21T23:04:59.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.934+0000 7fb7d9600640 1 -- start start 2024-02-21T23:04:59.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.935+0000 7fb7d9600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb7d40aff90 0x7fb7d40c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:59.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.935+0000 7fb7d9600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fb7d406dbf0 0x7fb7d4158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:59.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.935+0000 7fb7d9600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7d40c74c0 con 0x7fb7d40c5720 2024-02-21T23:04:59.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.935+0000 7fb7d9600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7d40c7270 con 0x7fb7d40aff90 2024-02-21T23:04:59.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.935+0000 7fb7d9600640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7d4158630 con 0x7fb7d406dbf0 2024-02-21T23:04:59.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.935+0000 7fb7d3e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb7d40aff90 0x7fb7d40c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:59.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.936+0000 7fb7d3e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb7d40aff90 0x7fb7d40c51e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45620/0 (socket says 172.21.15.28:45620) 2024-02-21T23:04:59.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.936+0000 7fb7d3e00640 1 -- 172.21.15.28:0/3354821706 learned_addr learned my addr 172.21.15.28:0/3354821706 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:59.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.936+0000 7fb7d3400640 1 --2- 172.21.15.28:0/3354821706 >> v2:172.21.15.81:3300/0 conn(0x7fb7d406dbf0 0x7fb7d4158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:59.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.936+0000 7fb7d3e00640 1 -- 172.21.15.28:0/3354821706 >> v2:172.21.15.81:3300/0 conn(0x7fb7d406dbf0 msgr2=0x7fb7d4158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:59.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.936+0000 7fb7d3e00640 1 --2- 172.21.15.28:0/3354821706 >> v2:172.21.15.81:3300/0 conn(0x7fb7d406dbf0 0x7fb7d4158ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:59.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.936+0000 7fb7d3e00640 1 -- 172.21.15.28:0/3354821706 >> v1:172.21.15.81:6789/0 conn(0x7fb7d40c5720 legacy=0x7fb7d406c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:04:59.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.936+0000 7fb7d3e00640 1 -- 172.21.15.28:0/3354821706 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7d40c77a0 con 0x7fb7d40aff90 2024-02-21T23:04:59.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.937+0000 7fb7d3e00640 1 --2- 172.21.15.28:0/3354821706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7d40aff90 0x7fb7d40c51e0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fb7d40c73f0 tx=0x7fb7c402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab6c09a4562cb414 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:59.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.937+0000 7fb7d2a00640 1 -- 172.21.15.28:0/3354821706 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c4035020 con 0x7fb7d40aff90 2024-02-21T23:04:59.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.938+0000 7fb7d2a00640 1 -- 172.21.15.28:0/3354821706 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7c4031870 con 0x7fb7d40aff90 2024-02-21T23:04:59.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.938+0000 7fb7d2a00640 1 -- 172.21.15.28:0/3354821706 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c4031c40 con 0x7fb7d40aff90 2024-02-21T23:04:59.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.938+0000 7fb7d9600640 1 -- 172.21.15.28:0/3354821706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7d40aff90 msgr2=0x7fb7d40c51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:59.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.938+0000 7fb7d9600640 1 --2- 172.21.15.28:0/3354821706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7d40aff90 0x7fb7d40c51e0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fb7d40c73f0 tx=0x7fb7c402b910 comp rx=0 tx=0).stop 2024-02-21T23:04:59.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.938+0000 7fb7d9600640 1 -- 172.21.15.28:0/3354821706 shutdown_connections 2024-02-21T23:04:59.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.938+0000 7fb7d9600640 1 --2- 172.21.15.28:0/3354821706 >> v2:172.21.15.81:3300/0 conn(0x7fb7d406dbf0 0x7fb7d4158ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:59.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.938+0000 7fb7d9600640 1 --2- 172.21.15.28:0/3354821706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7d40aff90 0x7fb7d40c51e0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:59.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.938+0000 7fb7d9600640 1 -- 172.21.15.28:0/3354821706 >> 172.21.15.28:0/3354821706 conn(0x7fb7d40ac370 msgr2=0x7fb7d40ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:04:59.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.938+0000 7fb7d9600640 1 -- 172.21.15.28:0/3354821706 shutdown_connections 2024-02-21T23:04:59.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.938+0000 7fb7d9600640 1 -- 172.21.15.28:0/3354821706 wait complete. 2024-02-21T23:04:59.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.940+0000 7fb7d9600640 1 Processor -- start 2024-02-21T23:04:59.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.940+0000 7fb7d9600640 1 -- start start 2024-02-21T23:04:59.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.941+0000 7fb7d9600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb7d406dbf0 0x7fb7d41eb4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:59.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.941+0000 7fb7d9600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7d40aff90 0x7fb7d41515e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:59.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.941+0000 7fb7d9600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7d40c5720 0x7fb7d40c1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:59.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.941+0000 7fb7d9600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d40c8760 con 0x7fb7d40aff90 2024-02-21T23:04:59.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.941+0000 7fb7d9600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d40be0c0 con 0x7fb7d40c5720 2024-02-21T23:04:59.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.941+0000 7fb7d9600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d40c88e0 con 0x7fb7d406dbf0 2024-02-21T23:04:59.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.941+0000 7fb7d3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7d40aff90 0x7fb7d41515e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:59.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.941+0000 7fb7d3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7d40aff90 0x7fb7d41515e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45622/0 (socket says 172.21.15.28:45622) 2024-02-21T23:04:59.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.941+0000 7fb7d3400640 1 -- 172.21.15.28:0/2151257147 learned_addr learned my addr 172.21.15.28:0/2151257147 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:04:59.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.941+0000 7fb7d3e00640 1 --2- 172.21.15.28:0/2151257147 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb7d406dbf0 0x7fb7d41eb4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:59.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.941+0000 7fb7d3400640 1 -- 172.21.15.28:0/2151257147 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb7d406dbf0 msgr2=0x7fb7d41eb4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:59.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.942+0000 7fb7d3400640 1 --2- 172.21.15.28:0/2151257147 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb7d406dbf0 0x7fb7d41eb4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:59.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.941+0000 7fb7d8c00640 1 --2- 172.21.15.28:0/2151257147 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7d40c5720 0x7fb7d40c1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:59.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.942+0000 7fb7d3400640 1 -- 172.21.15.28:0/2151257147 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7d40c5720 msgr2=0x7fb7d40c1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:04:59.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.942+0000 7fb7d3400640 1 --2- 172.21.15.28:0/2151257147 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7d40c5720 0x7fb7d40c1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:04:59.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.942+0000 7fb7d3400640 1 -- 172.21.15.28:0/2151257147 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7d4154a00 con 0x7fb7d40aff90 2024-02-21T23:04:59.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.942+0000 7fb7d8c00640 1 --2- 172.21.15.28:0/2151257147 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7d40c5720 0x7fb7d40c1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:04:59.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.942+0000 7fb7d3400640 1 --2- 172.21.15.28:0/2151257147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7d40aff90 0x7fb7d41515e0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc00c7f0 tx=0x7fb7bc00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:59.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.942+0000 7fb7cbe00640 1 -- 172.21.15.28:0/2151257147 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7bc015070 con 0x7fb7d40aff90 2024-02-21T23:04:59.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.942+0000 7fb7cbe00640 1 -- 172.21.15.28:0/2151257147 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7bc004220 con 0x7fb7d40aff90 2024-02-21T23:04:59.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.942+0000 7fb7cbe00640 1 -- 172.21.15.28:0/2151257147 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7bc003610 con 0x7fb7d40aff90 2024-02-21T23:04:59.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.942+0000 7fb7d9600640 1 -- 172.21.15.28:0/2151257147 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7d4152af0 con 0x7fb7d40aff90 2024-02-21T23:04:59.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.942+0000 7fb7d9600640 1 -- 172.21.15.28:0/2151257147 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7d41530b0 con 0x7fb7d40aff90 2024-02-21T23:04:59.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.943+0000 7fb7d9600640 1 -- 172.21.15.28:0/2151257147 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7d404f290 con 0x7fb7d40aff90 2024-02-21T23:04:59.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.944+0000 7fb7cbe00640 1 -- 172.21.15.28:0/2151257147 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb7bc0039c0 con 0x7fb7d40aff90 2024-02-21T23:04:59.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.945+0000 7fb7cbe00640 1 --2- 172.21.15.28:0/2151257147 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb7b006b500 0x7fb7b006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:04:59.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.945+0000 7fb7cbe00640 1 -- 172.21.15.28:0/2151257147 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb7bc08a2a0 con 0x7fb7d40aff90 2024-02-21T23:04:59.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.946+0000 7fb7d3e00640 1 --2- 172.21.15.28:0/2151257147 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb7b006b500 0x7fb7b006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:04:59.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.947+0000 7fb7d3e00640 1 --2- 172.21.15.28:0/2151257147 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb7b006b500 0x7fb7b006d9c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fb7c4036000 tx=0x7fb7c4008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:04:59.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:04:59.949+0000 7fb7cbe00640 1 -- 172.21.15.28:0/2151257147 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb7bc0590e0 con 0x7fb7d40aff90 2024-02-21T23:05:00.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.190+0000 7fb7d9600640 1 -- 172.21.15.28:0/2151257147 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb7d4153280 con 0x7fb7d40aff90 2024-02-21T23:05:00.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.191+0000 7fb7cbe00640 1 -- 172.21.15.28:0/2151257147 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+4935 (secure 0 0 0) 0x7fb7bc08a0c0 con 0x7fb7d40aff90 2024-02-21T23:05:00.180 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:05:00.180 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":32,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5006,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3468780363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3468780363},{"type":"v1","addr":"172.21.15.28:6835","nonce":3468780363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27},{"gid":5221,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/2152160500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2152160500},{"type":"v1","addr":"172.21.15.81:6835","nonce":2152160500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27},{"gid":5224,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/442767245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":442767245},{"type":"v1","addr":"172.21.15.81:6837","nonce":442767245}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:04:53.432188+0000","modified":"2024-02-21T23:04:56.493528+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5715},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5715":{"gid":5715,"name":"a","rank":0,"incarnation":28,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6834/2803823391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2803823391},{"type":"v1","addr":"172.21.15.28:6834","nonce":2803823391}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-02-21T23:05:00.181 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 32 2024-02-21T23:05:00.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.193+0000 7fb7d9600640 1 -- 172.21.15.28:0/2151257147 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb7b006b500 msgr2=0x7fb7b006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:00.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.193+0000 7fb7d9600640 1 --2- 172.21.15.28:0/2151257147 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb7b006b500 0x7fb7b006d9c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fb7c4036000 tx=0x7fb7c4008030 comp rx=0 tx=0).stop 2024-02-21T23:05:00.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.193+0000 7fb7d9600640 1 -- 172.21.15.28:0/2151257147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7d40aff90 msgr2=0x7fb7d41515e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:00.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.193+0000 7fb7d9600640 1 --2- 172.21.15.28:0/2151257147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7d40aff90 0x7fb7d41515e0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc00c7f0 tx=0x7fb7bc00ccb0 comp rx=0 tx=0).stop 2024-02-21T23:05:00.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.193+0000 7fb7d9600640 1 -- 172.21.15.28:0/2151257147 shutdown_connections 2024-02-21T23:05:00.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.193+0000 7fb7d9600640 1 --2- 172.21.15.28:0/2151257147 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb7b006b500 0x7fb7b006d9c0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:00.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.193+0000 7fb7d9600640 1 --2- 172.21.15.28:0/2151257147 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7d40c5720 0x7fb7d40c1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:00.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.193+0000 7fb7d9600640 1 --2- 172.21.15.28:0/2151257147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7d40aff90 0x7fb7d41515e0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:00.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.193+0000 7fb7d9600640 1 --2- 172.21.15.28:0/2151257147 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb7d406dbf0 0x7fb7d41eb4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:00.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.193+0000 7fb7d9600640 1 -- 172.21.15.28:0/2151257147 >> 172.21.15.28:0/2151257147 conn(0x7fb7d40ac370 msgr2=0x7fb7d40ac750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:00.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.193+0000 7fb7d9600640 1 -- 172.21.15.28:0/2151257147 shutdown_connections 2024-02-21T23:05:00.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.194+0000 7fb7d9600640 1 -- 172.21.15.28:0/2151257147 wait complete. 2024-02-21T23:05:00.602 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 32, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:04:53.432188+0000', 'modified': '2024-02-21T23:04:56.493528+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5715}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5715': {'gid': 5715, 'name': 'a', 'rank': 0, 'incarnation': 28, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6834/2803823391', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 2803823391}, {'type': 'v1', 'addr': '172.21.15.28:6834', 'nonce': 2803823391}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [8], 'metadata_pool': 7, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:05:00.602 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-21T23:05:00.603 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-02-21T23:05:00.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.838+0000 7f9752e00640 1 Processor -- start 2024-02-21T23:05:00.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.838+0000 7f9752e00640 1 -- start start 2024-02-21T23:05:00.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.841+0000 7f9752e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f974c1527a0 0x7f974c158f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:00.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.841+0000 7f9752e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f974c06d410 0x7f974c06d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:00.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.841+0000 7f9752e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f974c0c2070 con 0x7f974c0bf9a0 2024-02-21T23:05:00.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.841+0000 7f9752e00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f974c06e790 con 0x7f974c1527a0 2024-02-21T23:05:00.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.841+0000 7f9752e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f974c0c29a0 con 0x7f974c06d410 2024-02-21T23:05:00.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.842+0000 7f9752400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f974c06d410 0x7f974c06d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:00.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.842+0000 7f9751a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f974c1527a0 0x7f974c158f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:00.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.842+0000 7f9751000640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f974c0bf9a0 0x7f974c0bfdf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.28:50262/0 (socket says 172.21.15.28:50262) 2024-02-21T23:05:00.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.842+0000 7f9751000640 1 -- 172.21.15.28:0/3003639707 learned_addr learned my addr 172.21.15.28:0/3003639707 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:05:00.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.843+0000 7f9751a00640 1 -- 172.21.15.28:0/3003639707 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f974c06d410 msgr2=0x7f974c06d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:00.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.843+0000 7f9751a00640 1 --2- 172.21.15.28:0/3003639707 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f974c06d410 0x7f974c06d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:00.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.843+0000 7f9751a00640 1 -- 172.21.15.28:0/3003639707 >> v1:172.21.15.81:6789/0 conn(0x7f974c0bf9a0 legacy=0x7f974c0bfdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:00.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.843+0000 7f9751a00640 1 -- 172.21.15.28:0/3003639707 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f974c0c2350 con 0x7f974c1527a0 2024-02-21T23:05:00.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.844+0000 7f9751a00640 1 --2- 172.21.15.28:0/3003639707 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f974c1527a0 0x7f974c158f70 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f973c0097a0 tx=0x7f973c02b940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df68f2e948a729a6 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:00.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.844+0000 7f974be00640 1 -- 172.21.15.28:0/3003639707 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f973c034070 con 0x7f974c1527a0 2024-02-21T23:05:00.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.844+0000 7f974be00640 1 -- 172.21.15.28:0/3003639707 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f973c030800 con 0x7f974c1527a0 2024-02-21T23:05:00.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.844+0000 7f974be00640 1 -- 172.21.15.28:0/3003639707 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f973c030bd0 con 0x7f974c1527a0 2024-02-21T23:05:00.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.845+0000 7f9752e00640 1 -- 172.21.15.28:0/3003639707 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f974c1527a0 msgr2=0x7f974c158f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:00.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.845+0000 7f9752e00640 1 --2- 172.21.15.28:0/3003639707 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f974c1527a0 0x7f974c158f70 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f973c0097a0 tx=0x7f973c02b940 comp rx=0 tx=0).stop 2024-02-21T23:05:00.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.846+0000 7f9752e00640 1 -- 172.21.15.28:0/3003639707 shutdown_connections 2024-02-21T23:05:00.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.846+0000 7f9752e00640 1 --2- 172.21.15.28:0/3003639707 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f974c06d410 0x7f974c06d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:00.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.846+0000 7f9752e00640 1 --2- 172.21.15.28:0/3003639707 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f974c1527a0 0x7f974c158f70 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:00.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.846+0000 7f9752e00640 1 -- 172.21.15.28:0/3003639707 >> 172.21.15.28:0/3003639707 conn(0x7f974c0a8320 msgr2=0x7f974c0a8730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:00.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.846+0000 7f9752e00640 1 -- 172.21.15.28:0/3003639707 shutdown_connections 2024-02-21T23:05:00.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.846+0000 7f9752e00640 1 -- 172.21.15.28:0/3003639707 wait complete. 2024-02-21T23:05:00.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.849+0000 7f9752e00640 1 Processor -- start 2024-02-21T23:05:00.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.849+0000 7f9752e00640 1 -- start start 2024-02-21T23:05:00.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.851+0000 7f9752e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f974c06d410 0x7f974c0c30a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:00.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.851+0000 7f9752e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f974c0bf9a0 0x7f974c14cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:00.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.851+0000 7f9752e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f974c1527a0 0x7f974c14d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:00.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.852+0000 7f9752e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f974c0c1d20 con 0x7f974c06d410 2024-02-21T23:05:00.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.852+0000 7f9752e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f974c0c0500 con 0x7f974c1527a0 2024-02-21T23:05:00.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.852+0000 7f9752e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f974c06dda0 con 0x7f974c0bf9a0 2024-02-21T23:05:00.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.852+0000 7f9751a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f974c06d410 0x7f974c0c30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:00.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.852+0000 7f9751a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f974c06d410 0x7f974c0c30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45638/0 (socket says 172.21.15.28:45638) 2024-02-21T23:05:00.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.852+0000 7f9751a00640 1 -- 172.21.15.28:0/641483944 learned_addr learned my addr 172.21.15.28:0/641483944 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:00.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.852+0000 7f9751000640 1 --2- 172.21.15.28:0/641483944 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f974c0bf9a0 0x7f974c14cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:00.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.852+0000 7f9752400640 1 --2- 172.21.15.28:0/641483944 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f974c1527a0 0x7f974c14d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:00.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.852+0000 7f9751a00640 1 -- 172.21.15.28:0/641483944 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f974c0bf9a0 msgr2=0x7f974c14cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:00.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.852+0000 7f9751a00640 1 --2- 172.21.15.28:0/641483944 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f974c0bf9a0 0x7f974c14cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:00.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.852+0000 7f9751a00640 1 -- 172.21.15.28:0/641483944 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f974c1527a0 msgr2=0x7f974c14d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:00.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.852+0000 7f9751a00640 1 --2- 172.21.15.28:0/641483944 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f974c1527a0 0x7f974c14d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:00.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.852+0000 7f9751a00640 1 -- 172.21.15.28:0/641483944 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f974c152e40 con 0x7f974c06d410 2024-02-21T23:05:00.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.853+0000 7f9751a00640 1 --2- 172.21.15.28:0/641483944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f974c06d410 0x7f974c0c30a0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f973c009890 tx=0x7f973c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:00.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.853+0000 7f9749600640 1 -- 172.21.15.28:0/641483944 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f973c034040 con 0x7f974c06d410 2024-02-21T23:05:00.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.853+0000 7f9749600640 1 -- 172.21.15.28:0/641483944 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f973c03add0 con 0x7f974c06d410 2024-02-21T23:05:00.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.853+0000 7f9749600640 1 -- 172.21.15.28:0/641483944 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f973c006e10 con 0x7f974c06d410 2024-02-21T23:05:00.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.853+0000 7f9752e00640 1 -- 172.21.15.28:0/641483944 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f974c1eed00 con 0x7f974c06d410 2024-02-21T23:05:00.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.853+0000 7f9752e00640 1 -- 172.21.15.28:0/641483944 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f974c0a9ac0 con 0x7f974c06d410 2024-02-21T23:05:00.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.856+0000 7f9749600640 1 -- 172.21.15.28:0/641483944 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f973c04b060 con 0x7f974c06d410 2024-02-21T23:05:00.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.857+0000 7f97573c7740 1 Processor -- start 2024-02-21T23:05:00.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.857+0000 7f9749600640 1 --2- 172.21.15.28:0/641483944 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f972c06b4d0 0x7f972c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:00.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.857+0000 7f9749600640 1 -- 172.21.15.28:0/641483944 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f973c0ac520 con 0x7f974c06d410 2024-02-21T23:05:00.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.858+0000 7f97573c7740 1 -- start start 2024-02-21T23:05:00.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.858+0000 7f9751000640 1 --2- 172.21.15.28:0/641483944 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f972c06b4d0 0x7f972c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:00.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.858+0000 7f97573c7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f72f5be020 0x55f72f5be400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:00.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.858+0000 7f97573c7740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f72f5bf330 0x55f72f5fa170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:00.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.858+0000 7f97573c7740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55f72f5fa6b0 0x55f72f5f8720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:00.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.859+0000 7f97573c7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f72f522470 con 0x55f72f5be020 2024-02-21T23:05:00.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.859+0000 7f97573c7740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f72f4cbaf0 con 0x55f72f5bf330 2024-02-21T23:05:00.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.859+0000 7f97573c7740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f72f51e510 con 0x55f72f5fa6b0 2024-02-21T23:05:00.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.859+0000 7f9751a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f72f5be020 0x55f72f5be400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:00.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.859+0000 7f9751a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f72f5be020 0x55f72f5be400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45640/0 (socket says 172.21.15.28:45640) 2024-02-21T23:05:00.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.859+0000 7f9751a00640 1 -- 172.21.15.28:0/1601806423 learned_addr learned my addr 172.21.15.28:0/1601806423 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:00.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.859+0000 7f9751a00640 1 -- 172.21.15.28:0/1601806423 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55f72f5fa6b0 msgr2=0x55f72f5f8720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:00.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.859+0000 7f9751a00640 1 --2- 172.21.15.28:0/1601806423 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55f72f5fa6b0 0x55f72f5f8720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:00.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.859+0000 7f9751000640 1 --2- 172.21.15.28:0/1601806423 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f72f5bf330 0x55f72f5fa170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:00.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.859+0000 7f9751a00640 1 -- 172.21.15.28:0/1601806423 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f72f5bf330 msgr2=0x55f72f5fa170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:00.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.859+0000 7f9751a00640 1 --2- 172.21.15.28:0/1601806423 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f72f5bf330 0x55f72f5fa170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:00.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.859+0000 7f9751a00640 1 -- 172.21.15.28:0/1601806423 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f72f4d2d90 con 0x55f72f5be020 2024-02-21T23:05:00.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.859+0000 7f9751000640 1 --2- 172.21.15.28:0/641483944 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f972c06b4d0 0x7f972c06d990 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f974c0c3950 tx=0x7f9738009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:00.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.860+0000 7f9751a00640 1 --2- 172.21.15.28:0/1601806423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f72f5be020 0x55f72f5be400 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f973c077080 tx=0x7f973c076f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c37d4db1a11f3ad server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:00.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.860+0000 7f9742000640 1 -- 172.21.15.28:0/1601806423 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f973c07b320 con 0x55f72f5be020 2024-02-21T23:05:00.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.860+0000 7f9742000640 1 -- 172.21.15.28:0/1601806423 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f973c07ac20 con 0x55f72f5be020 2024-02-21T23:05:00.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.860+0000 7f9742000640 1 -- 172.21.15.28:0/1601806423 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f973c07ed70 con 0x55f72f5be020 2024-02-21T23:05:00.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.860+0000 7f97573c7740 1 -- 172.21.15.28:0/1601806423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f72f5be020 msgr2=0x55f72f5be400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:00.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.861+0000 7f97573c7740 1 --2- 172.21.15.28:0/1601806423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f72f5be020 0x55f72f5be400 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f973c077080 tx=0x7f973c076f40 comp rx=0 tx=0).stop 2024-02-21T23:05:00.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.861+0000 7f97573c7740 1 -- 172.21.15.28:0/1601806423 shutdown_connections 2024-02-21T23:05:00.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.861+0000 7f97573c7740 1 --2- 172.21.15.28:0/1601806423 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55f72f5fa6b0 0x55f72f5f8720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:00.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.861+0000 7f97573c7740 1 --2- 172.21.15.28:0/1601806423 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f72f5bf330 0x55f72f5fa170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:00.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.861+0000 7f97573c7740 1 --2- 172.21.15.28:0/1601806423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f72f5be020 0x55f72f5be400 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:00.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.861+0000 7f97573c7740 1 -- 172.21.15.28:0/1601806423 >> 172.21.15.28:0/1601806423 conn(0x55f72f5ba970 msgr2=0x55f72f5df380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:00.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.861+0000 7f97573c7740 1 -- 172.21.15.28:0/1601806423 shutdown_connections 2024-02-21T23:05:00.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.861+0000 7f97573c7740 1 -- 172.21.15.28:0/1601806423 wait complete. 2024-02-21T23:05:00.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.861+0000 7f97573c7740 1 -- start start 2024-02-21T23:05:00.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.862+0000 7f97573c7740 1 Processor -- start 2024-02-21T23:05:00.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.863+0000 7f97573c7740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55f72f5bf330 0x55f72f5d96e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:00.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.863+0000 7f97573c7740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f72f5fa6b0 0x55f72f5f8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:00.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.863+0000 7f97573c7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f72f5f90c0 0x55f72f60a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:00.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.863+0000 7f97573c7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f72f4cbaf0 con 0x55f72f5f90c0 2024-02-21T23:05:00.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.863+0000 7f97573c7740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f72f4c1040 con 0x55f72f5fa6b0 2024-02-21T23:05:00.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.863+0000 7f97573c7740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f72f522470 con 0x55f72f5bf330 2024-02-21T23:05:00.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.863+0000 7f97573c7740 10 client.0 _ll_register_callbacks cb 0x55f72f4de920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:05:00.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.863+0000 7f9752400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f72f5f90c0 0x55f72f60a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:00.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.863+0000 7f9752400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f72f5f90c0 0x55f72f60a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45642/0 (socket says 172.21.15.28:45642) 2024-02-21T23:05:00.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.863+0000 7f9752400640 1 -- 172.21.15.28:0/2989377199 learned_addr learned my addr 172.21.15.28:0/2989377199 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:00.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.863+0000 7f9751a00640 1 --2- 172.21.15.28:0/2989377199 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55f72f5bf330 0x55f72f5d96e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:00.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.863+0000 7f9751000640 1 --2- 172.21.15.28:0/2989377199 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f72f5fa6b0 0x55f72f5f8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:00.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.864+0000 7f9752400640 1 -- 172.21.15.28:0/2989377199 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55f72f5bf330 msgr2=0x55f72f5d96e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:00.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.864+0000 7f9752400640 1 --2- 172.21.15.28:0/2989377199 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55f72f5bf330 0x55f72f5d96e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:00.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.864+0000 7f9752400640 1 -- 172.21.15.28:0/2989377199 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f72f5fa6b0 msgr2=0x55f72f5f8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:00.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.864+0000 7f9752400640 1 --2- 172.21.15.28:0/2989377199 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f72f5fa6b0 0x55f72f5f8b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:00.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.864+0000 7f9752400640 1 -- 172.21.15.28:0/2989377199 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f72f4e0aa0 con 0x55f72f5f90c0 2024-02-21T23:05:00.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.864+0000 7f9752400640 1 --2- 172.21.15.28:0/2989377199 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f72f5f90c0 0x55f72f60a1b0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f97440104a0 tx=0x7f9744010960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3f73e996a84e43 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:00.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.864+0000 7f9736a00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:05:00.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.864+0000 7f9736a00640 1 -- 172.21.15.28:0/2989377199 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9744018070 con 0x55f72f5f90c0 2024-02-21T23:05:00.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.864+0000 7f97573c7740 1 -- 172.21.15.28:0/2989377199 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f973c07ac20 con 0x55f72f5f90c0 2024-02-21T23:05:00.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.865+0000 7f9736a00640 1 -- 172.21.15.28:0/2989377199 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(53..53 src has 1..53) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f9744014800 con 0x55f72f5f90c0 2024-02-21T23:05:00.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.865+0000 7f9736a00640 1 -- 172.21.15.28:0/2989377199 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9744023050 con 0x55f72f5f90c0 2024-02-21T23:05:00.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.865+0000 7f97573c7740 10 client.5859 fetch_fsmap learned FSMap version 32 2024-02-21T23:05:00.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.865+0000 7f97573c7740 1 -- 172.21.15.28:0/2989377199 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=2+}) v3 -- 0x55f72f2372d0 con 0x55f72f5f90c0 2024-02-21T23:05:00.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.865+0000 7f9736a00640 1 -- 172.21.15.28:0/2989377199 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f973c07ac20 con 0x55f72f5f90c0 2024-02-21T23:05:00.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.865+0000 7f9736a00640 1 -- 172.21.15.28:0/2989377199 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f9744026020 con 0x55f72f5f90c0 2024-02-21T23:05:00.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.865+0000 7f97573c7740 10 client.5859 fetch_fsmap finished waiting for FSMap version 32 2024-02-21T23:05:00.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.865+0000 7f97573c7740 10 client.5859 resolve_mds: resolved name 'a' to daemon mds.a 2024-02-21T23:05:00.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.865+0000 7f97573c7740 20 client.5859 populate_metadata read hostname 'smithi028' 2024-02-21T23:05:00.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.865+0000 7f97573c7740 1 --2- 172.21.15.28:0/2989377199 >> [v2:172.21.15.28:6832/2803823391,v1:172.21.15.28:6834/2803823391] conn(0x55f72f60dbd0 0x55f72f60dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:00.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.865+0000 7f97573c7740 4 client.5859 mds_command: new command op to 5715 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-21T23:05:00.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.865+0000 7f97573c7740 1 -- 172.21.15.28:0/2989377199 --> [v2:172.21.15.28:6832/2803823391,v1:172.21.15.28:6834/2803823391] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f72f52f9b0 con 0x55f72f60dbd0 2024-02-21T23:05:00.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.866+0000 7f9751a00640 1 --2- 172.21.15.28:0/2989377199 >> [v2:172.21.15.28:6832/2803823391,v1:172.21.15.28:6834/2803823391] conn(0x55f72f60dbd0 0x55f72f60dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:00.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.866+0000 7f9751a00640 1 --2- 172.21.15.28:0/2989377199 >> [v2:172.21.15.28:6832/2803823391,v1:172.21.15.28:6834/2803823391] conn(0x55f72f60dbd0 0x55f72f60dfb0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ce7facb706780c server_cookie=7efb2fecbe23ec01 in_seq=0 out_seq=0 2024-02-21T23:05:00.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.866+0000 7f9736a00640 10 client.5859 ms_handle_connect on v2:172.21.15.28:6832/2803823391 2024-02-21T23:05:00.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.867+0000 7f9736a00640 1 -- 172.21.15.28:0/2989377199 <== mds.0 v2:172.21.15.28:6832/2803823391 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f72f52f9b0 con 0x55f72f60dbd0 2024-02-21T23:05:00.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.867+0000 7f9736a00640 10 client.5859 handle_command_reply: tid=0 2024-02-21T23:05:00.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.867+0000 7f97573c7740 1 client.5859 shutdown 2024-02-21T23:05:00.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.868+0000 7f97573c7740 1 -- 172.21.15.28:0/2989377199 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f72f5f90c0 msgr2=0x55f72f60a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:00.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.868+0000 7f97573c7740 1 --2- 172.21.15.28:0/2989377199 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f72f5f90c0 0x55f72f60a1b0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f97440104a0 tx=0x7f9744010960 comp rx=0 tx=0).stop 2024-02-21T23:05:00.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.868+0000 7f97573c7740 1 -- 172.21.15.28:0/2989377199 shutdown_connections 2024-02-21T23:05:00.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.868+0000 7f97573c7740 1 --2- 172.21.15.28:0/2989377199 >> [v2:172.21.15.28:6832/2803823391,v1:172.21.15.28:6834/2803823391] conn(0x55f72f60dbd0 0x55f72f60dfb0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:00.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.868+0000 7f97573c7740 1 --2- 172.21.15.28:0/2989377199 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f72f5f90c0 0x55f72f60a1b0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:00.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.868+0000 7f97573c7740 1 --2- 172.21.15.28:0/2989377199 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f72f5fa6b0 0x55f72f5f8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:00.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.868+0000 7f97573c7740 1 --2- 172.21.15.28:0/2989377199 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55f72f5bf330 0x55f72f5d96e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:00.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.868+0000 7f9736a00640 0 client.5859 ms_handle_reset on v2:172.21.15.28:6832/2803823391 2024-02-21T23:05:00.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.868+0000 7f97573c7740 1 -- 172.21.15.28:0/2989377199 >> 172.21.15.28:0/2989377199 conn(0x55f72f5be020 msgr2=0x55f72f5fd790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:00.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.868+0000 7f97573c7740 1 -- 172.21.15.28:0/2989377199 shutdown_connections 2024-02-21T23:05:00.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.868+0000 7f97573c7740 1 -- 172.21.15.28:0/2989377199 wait complete. 2024-02-21T23:05:00.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.868+0000 7f97573c7740 20 client.5859 trim_cache size 0 max 16384 2024-02-21T23:05:00.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.887+0000 7f97573c7740 1 Processor -- start 2024-02-21T23:05:00.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.887+0000 7f97573c7740 1 -- start start 2024-02-21T23:05:00.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.888+0000 7f97573c7740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f72f5bf330 0x55f72f5df500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:00.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.888+0000 7f97573c7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f72f5f90c0 0x55f72f5f8860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:00.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.888+0000 7f97573c7740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55f72f5fa6b0 0x55f72f5dda60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:00.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.888+0000 7f97573c7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f974c1f6ce0 con 0x55f72f5f90c0 2024-02-21T23:05:00.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.888+0000 7f97573c7740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f974c072ba0 con 0x55f72f5bf330 2024-02-21T23:05:00.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.888+0000 7f97573c7740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f974c1f66e0 con 0x55f72f5fa6b0 2024-02-21T23:05:00.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.888+0000 7f9751000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f72f5f90c0 0x55f72f5f8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:00.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.888+0000 7f9751000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f72f5f90c0 0x55f72f5f8860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45644/0 (socket says 172.21.15.28:45644) 2024-02-21T23:05:00.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.888+0000 7f9751000640 1 -- 172.21.15.28:0/695019567 learned_addr learned my addr 172.21.15.28:0/695019567 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:00.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.888+0000 7f9751a00640 1 --2- 172.21.15.28:0/695019567 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f72f5bf330 0x55f72f5df500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:00.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.888+0000 7f9751000640 1 -- 172.21.15.28:0/695019567 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55f72f5fa6b0 msgr2=0x55f72f5dda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:00.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.888+0000 7f9752400640 1 --2- 172.21.15.28:0/695019567 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55f72f5fa6b0 0x55f72f5dda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:00.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.889+0000 7f9751000640 1 --2- 172.21.15.28:0/695019567 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55f72f5fa6b0 0x55f72f5dda60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:00.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.889+0000 7f9751000640 1 -- 172.21.15.28:0/695019567 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f72f5bf330 msgr2=0x55f72f5df500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:00.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.889+0000 7f9751000640 1 --2- 172.21.15.28:0/695019567 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f72f5bf330 0x55f72f5df500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:00.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.889+0000 7f9751000640 1 -- 172.21.15.28:0/695019567 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f72f3016b0 con 0x55f72f5f90c0 2024-02-21T23:05:00.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.889+0000 7f9752400640 1 --2- 172.21.15.28:0/695019567 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55f72f5fa6b0 0x55f72f5dda60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:05:00.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.889+0000 7f9751a00640 1 --2- 172.21.15.28:0/695019567 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f72f5bf330 0x55f72f5df500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:05:00.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.889+0000 7f9751000640 1 --2- 172.21.15.28:0/695019567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f72f5f90c0 0x55f72f5f8860 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f97380012e0 tx=0x7f9738014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d387badb99ede75 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:00.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.889+0000 7f9741600640 1 -- 172.21.15.28:0/695019567 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97380135f0 con 0x55f72f5f90c0 2024-02-21T23:05:00.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.889+0000 7f9741600640 1 -- 172.21.15.28:0/695019567 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9738027070 con 0x55f72f5f90c0 2024-02-21T23:05:00.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.889+0000 7f9741600640 1 -- 172.21.15.28:0/695019567 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9738013990 con 0x55f72f5f90c0 2024-02-21T23:05:00.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.890+0000 7f97573c7740 1 -- 172.21.15.28:0/695019567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f72f5f90c0 msgr2=0x55f72f5f8860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:00.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.890+0000 7f97573c7740 1 --2- 172.21.15.28:0/695019567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f72f5f90c0 0x55f72f5f8860 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f97380012e0 tx=0x7f9738014000 comp rx=0 tx=0).stop 2024-02-21T23:05:00.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.890+0000 7f97573c7740 1 -- 172.21.15.28:0/695019567 shutdown_connections 2024-02-21T23:05:00.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.890+0000 7f97573c7740 1 --2- 172.21.15.28:0/695019567 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55f72f5fa6b0 0x55f72f5dda60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:00.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.890+0000 7f97573c7740 1 --2- 172.21.15.28:0/695019567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f72f5f90c0 0x55f72f5f8860 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:00.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.890+0000 7f97573c7740 1 --2- 172.21.15.28:0/695019567 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f72f5bf330 0x55f72f5df500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:00.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.890+0000 7f97573c7740 1 -- 172.21.15.28:0/695019567 >> 172.21.15.28:0/695019567 conn(0x55f72f5be020 msgr2=0x55f72f5f0930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:00.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.890+0000 7f97573c7740 1 -- 172.21.15.28:0/695019567 shutdown_connections 2024-02-21T23:05:00.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.890+0000 7f97573c7740 1 -- 172.21.15.28:0/695019567 wait complete. 2024-02-21T23:05:00.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.891+0000 7f97573c7740 1 -- start start 2024-02-21T23:05:00.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.891+0000 7f97573c7740 1 Processor -- start 2024-02-21T23:05:00.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.892+0000 7f97573c7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f72f5bf330 0x55f72f5d26f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:00.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.892+0000 7f97573c7740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f72f5fa6b0 0x55f72f5fbf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:00.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.892+0000 7f97573c7740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55f72f606920 0x55f72f666520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:00.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.892+0000 7f97573c7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f72f60ac90 con 0x55f72f5bf330 2024-02-21T23:05:00.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.892+0000 7f97573c7740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f72f5f6660 con 0x55f72f5fa6b0 2024-02-21T23:05:00.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.892+0000 7f97573c7740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f72f5da740 con 0x55f72f606920 2024-02-21T23:05:00.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.892+0000 7f97573c7740 10 client.0 _ll_register_callbacks cb 0x7f9744009e60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:05:00.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.892+0000 7f9751a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f72f5bf330 0x55f72f5d26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:00.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.892+0000 7f9751a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f72f5bf330 0x55f72f5d26f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45658/0 (socket says 172.21.15.28:45658) 2024-02-21T23:05:00.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.892+0000 7f9751a00640 1 -- 172.21.15.28:0/3410357627 learned_addr learned my addr 172.21.15.28:0/3410357627 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:00.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.892+0000 7f9751a00640 1 -- 172.21.15.28:0/3410357627 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55f72f606920 msgr2=0x55f72f666520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:00.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.892+0000 7f9751a00640 1 --2- 172.21.15.28:0/3410357627 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55f72f606920 0x55f72f666520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:00.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.892+0000 7f9751a00640 1 -- 172.21.15.28:0/3410357627 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f72f5fa6b0 msgr2=0x55f72f5fbf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:00.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.892+0000 7f9751000640 1 --2- 172.21.15.28:0/3410357627 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f72f5fa6b0 0x55f72f5fbf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:00.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.892+0000 7f9751a00640 1 --2- 172.21.15.28:0/3410357627 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f72f5fa6b0 0x55f72f5fbf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:00.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.892+0000 7f9751a00640 1 -- 172.21.15.28:0/3410357627 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f72f5f35f0 con 0x55f72f5bf330 2024-02-21T23:05:00.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.893+0000 7f9751a00640 1 --2- 172.21.15.28:0/3410357627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f72f5bf330 0x55f72f5d26f0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f973c077080 tx=0x7f973c0ac000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=191b428db4f04274 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:00.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.893+0000 7f9736a00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:05:00.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.893+0000 7f9736a00640 1 -- 172.21.15.28:0/3410357627 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f973c088060 con 0x55f72f5bf330 2024-02-21T23:05:00.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.893+0000 7f9736a00640 1 -- 172.21.15.28:0/3410357627 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(53..53 src has 1..53) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f973c077290 con 0x55f72f5bf330 2024-02-21T23:05:00.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.893+0000 7f97573c7740 1 -- 172.21.15.28:0/3410357627 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9738027070 con 0x55f72f5bf330 2024-02-21T23:05:00.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.893+0000 7f9736a00640 1 -- 172.21.15.28:0/3410357627 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f973c08e050 con 0x55f72f5bf330 2024-02-21T23:05:00.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.893+0000 7f97573c7740 10 client.5865 fetch_fsmap learned FSMap version 32 2024-02-21T23:05:00.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.893+0000 7f97573c7740 1 -- 172.21.15.28:0/3410357627 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=2+}) v3 -- 0x55f72f66c790 con 0x55f72f5bf330 2024-02-21T23:05:00.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.894+0000 7f9736a00640 1 -- 172.21.15.28:0/3410357627 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9738027070 con 0x55f72f5bf330 2024-02-21T23:05:00.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.894+0000 7f9736a00640 1 -- 172.21.15.28:0/3410357627 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f973c0ac0e0 con 0x55f72f5bf330 2024-02-21T23:05:00.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.894+0000 7f97573c7740 10 client.5865 fetch_fsmap finished waiting for FSMap version 32 2024-02-21T23:05:00.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.894+0000 7f97573c7740 10 client.5865 resolve_mds: resolved name 'a' to daemon mds.a 2024-02-21T23:05:00.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.894+0000 7f97573c7740 20 client.5865 populate_metadata read hostname 'smithi028' 2024-02-21T23:05:00.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.894+0000 7f97573c7740 1 --2- 172.21.15.28:0/3410357627 >> [v2:172.21.15.28:6832/2803823391,v1:172.21.15.28:6834/2803823391] conn(0x55f72f5f5180 0x55f72f5f5560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:00.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.894+0000 7f97573c7740 4 client.5865 mds_command: new command op to 5715 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-21T23:05:00.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.894+0000 7f97573c7740 1 -- 172.21.15.28:0/3410357627 --> [v2:172.21.15.28:6832/2803823391,v1:172.21.15.28:6834/2803823391] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55f72f529d00 con 0x55f72f5f5180 2024-02-21T23:05:00.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.894+0000 7f9751000640 1 --2- 172.21.15.28:0/3410357627 >> [v2:172.21.15.28:6832/2803823391,v1:172.21.15.28:6834/2803823391] conn(0x55f72f5f5180 0x55f72f5f5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:00.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.895+0000 7f9751000640 1 --2- 172.21.15.28:0/3410357627 >> [v2:172.21.15.28:6832/2803823391,v1:172.21.15.28:6834/2803823391] conn(0x55f72f5f5180 0x55f72f5f5560 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9cc86d8a61fff5e6 server_cookie=5b47a68c8cf16e57 in_seq=0 out_seq=0 2024-02-21T23:05:00.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.895+0000 7f9736a00640 10 client.5865 ms_handle_connect on v2:172.21.15.28:6832/2803823391 2024-02-21T23:05:00.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.895+0000 7f9736a00640 1 -- 172.21.15.28:0/3410357627 <== mds.0 v2:172.21.15.28:6832/2803823391 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55f72f529d00 con 0x55f72f5f5180 2024-02-21T23:05:00.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.895+0000 7f9736a00640 10 client.5865 handle_command_reply: tid=0 2024-02-21T23:05:00.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.895+0000 7f97573c7740 1 client.5865 shutdown 2024-02-21T23:05:00.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.895+0000 7f97573c7740 1 -- 172.21.15.28:0/3410357627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f72f5bf330 msgr2=0x55f72f5d26f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:00.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.895+0000 7f97573c7740 1 --2- 172.21.15.28:0/3410357627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f72f5bf330 0x55f72f5d26f0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f973c077080 tx=0x7f973c0ac000 comp rx=0 tx=0).stop 2024-02-21T23:05:00.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.895+0000 7f97573c7740 1 -- 172.21.15.28:0/3410357627 shutdown_connections 2024-02-21T23:05:00.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.895+0000 7f97573c7740 1 --2- 172.21.15.28:0/3410357627 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55f72f606920 0x55f72f666520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:00.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.895+0000 7f97573c7740 1 --2- 172.21.15.28:0/3410357627 >> [v2:172.21.15.28:6832/2803823391,v1:172.21.15.28:6834/2803823391] conn(0x55f72f5f5180 0x55f72f5f5560 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:00.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.895+0000 7f97573c7740 1 --2- 172.21.15.28:0/3410357627 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f72f5fa6b0 0x55f72f5fbf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:00.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.895+0000 7f97573c7740 1 --2- 172.21.15.28:0/3410357627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f72f5bf330 0x55f72f5d26f0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:00.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.895+0000 7f97573c7740 1 -- 172.21.15.28:0/3410357627 >> 172.21.15.28:0/3410357627 conn(0x55f72f5be020 msgr2=0x55f72f5b3d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:00.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.895+0000 7f9736a00640 0 client.5865 ms_handle_reset on v2:172.21.15.28:6832/2803823391 2024-02-21T23:05:00.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.896+0000 7f97573c7740 1 -- 172.21.15.28:0/3410357627 shutdown_connections 2024-02-21T23:05:00.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.896+0000 7f97573c7740 1 -- 172.21.15.28:0/3410357627 wait complete. 2024-02-21T23:05:00.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.896+0000 7f97573c7740 20 client.5865 trim_cache size 0 max 16384 2024-02-21T23:05:00.903 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-21T23:05:00.903 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac", 2024-02-21T23:05:00.903 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-02-21T23:05:00.903 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 5715, 2024-02-21T23:05:00.903 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-02-21T23:05:00.903 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-02-21T23:05:00.903 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-02-21T23:05:00.903 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 6.9505862980000002, 2024-02-21T23:05:00.904 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 32, 2024-02-21T23:05:00.904 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 53, 2024-02-21T23:05:00.904 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 53, 2024-02-21T23:05:00.904 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 13.419159086000001 2024-02-21T23:05:00.904 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-21T23:05:00.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.896+0000 7f9752e00640 1 -- 172.21.15.28:0/641483944 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f972c06b4d0 msgr2=0x7f972c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:00.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.896+0000 7f9752e00640 1 --2- 172.21.15.28:0/641483944 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f972c06b4d0 0x7f972c06d990 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f974c0c3950 tx=0x7f9738009040 comp rx=0 tx=0).stop 2024-02-21T23:05:00.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.896+0000 7f9752e00640 1 -- 172.21.15.28:0/641483944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f974c06d410 msgr2=0x7f974c0c30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:00.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.896+0000 7f9752e00640 1 --2- 172.21.15.28:0/641483944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f974c06d410 0x7f974c0c30a0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f973c009890 tx=0x7f973c0046c0 comp rx=0 tx=0).stop 2024-02-21T23:05:00.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.896+0000 7f9752e00640 1 -- 172.21.15.28:0/641483944 shutdown_connections 2024-02-21T23:05:00.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.896+0000 7f9752e00640 1 --2- 172.21.15.28:0/641483944 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f972c06b4d0 0x7f972c06d990 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:00.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.896+0000 7f9752e00640 1 --2- 172.21.15.28:0/641483944 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f974c0bf9a0 0x7f974c14cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:00.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.896+0000 7f9752e00640 1 --2- 172.21.15.28:0/641483944 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f974c1527a0 0x7f974c14d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:00.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.896+0000 7f9752e00640 1 --2- 172.21.15.28:0/641483944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f974c06d410 0x7f974c0c30a0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:00.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.896+0000 7f9752e00640 1 -- 172.21.15.28:0/641483944 >> 172.21.15.28:0/641483944 conn(0x7f974c0a8320 msgr2=0x7f974c150f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:00.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.897+0000 7f9752e00640 1 -- 172.21.15.28:0/641483944 shutdown_connections 2024-02-21T23:05:00.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:00.897+0000 7f9752e00640 1 -- 172.21.15.28:0/641483944 wait complete. 2024-02-21T23:05:00.906 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:05:01.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.126+0000 7ff942000640 1 Processor -- start 2024-02-21T23:05:01.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.126+0000 7ff942000640 1 -- start start 2024-02-21T23:05:01.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.127+0000 7ff942000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff93c0aff90 0x7ff93c0c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:01.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.127+0000 7ff942000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7ff93c06dbf0 0x7ff93c158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:01.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.127+0000 7ff942000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff93c0c74c0 con 0x7ff93c0c5720 2024-02-21T23:05:01.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.127+0000 7ff942000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff93c0c7270 con 0x7ff93c0aff90 2024-02-21T23:05:01.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.127+0000 7ff942000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff93c158630 con 0x7ff93c06dbf0 2024-02-21T23:05:01.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.128+0000 7ff940c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff93c0aff90 0x7ff93c0c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:01.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.128+0000 7ff941600640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff93c0c5720 0x7ff93c06c270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34492/0 (socket says 172.21.15.28:34492) 2024-02-21T23:05:01.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.128+0000 7ff941600640 1 -- 172.21.15.28:0/3977776378 learned_addr learned my addr 172.21.15.28:0/3977776378 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:05:01.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.128+0000 7ff93be00640 1 --2- 172.21.15.28:0/3977776378 >> v2:172.21.15.81:3300/0 conn(0x7ff93c06dbf0 0x7ff93c158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:01.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.129+0000 7ff940c00640 1 -- 172.21.15.28:0/3977776378 >> v2:172.21.15.81:3300/0 conn(0x7ff93c06dbf0 msgr2=0x7ff93c158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:01.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.129+0000 7ff940c00640 1 --2- 172.21.15.28:0/3977776378 >> v2:172.21.15.81:3300/0 conn(0x7ff93c06dbf0 0x7ff93c158ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:01.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.129+0000 7ff940c00640 1 -- 172.21.15.28:0/3977776378 >> v1:172.21.15.28:6789/0 conn(0x7ff93c0c5720 legacy=0x7ff93c06c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:01.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.129+0000 7ff940c00640 1 -- 172.21.15.28:0/3977776378 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff93c0c77a0 con 0x7ff93c0aff90 2024-02-21T23:05:01.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.129+0000 7ff93be00640 1 --2- 172.21.15.28:0/3977776378 >> v2:172.21.15.81:3300/0 conn(0x7ff93c06dbf0 0x7ff93c158ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:05:01.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.130+0000 7ff940c00640 1 --2- 172.21.15.28:0/3977776378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff93c0aff90 0x7ff93c0c51e0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7ff93c0c73f0 tx=0x7ff93002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1560cb52360654ee server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:01.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.130+0000 7ff93b400640 1 -- 172.21.15.28:0/3977776378 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff930035020 con 0x7ff93c0aff90 2024-02-21T23:05:01.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.130+0000 7ff93b400640 1 -- 172.21.15.28:0/3977776378 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff930031830 con 0x7ff93c0aff90 2024-02-21T23:05:01.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.130+0000 7ff93b400640 1 -- 172.21.15.28:0/3977776378 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff930031c00 con 0x7ff93c0aff90 2024-02-21T23:05:01.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.131+0000 7ff942000640 1 -- 172.21.15.28:0/3977776378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff93c0aff90 msgr2=0x7ff93c0c51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:01.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.131+0000 7ff942000640 1 --2- 172.21.15.28:0/3977776378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff93c0aff90 0x7ff93c0c51e0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7ff93c0c73f0 tx=0x7ff93002b8d0 comp rx=0 tx=0).stop 2024-02-21T23:05:01.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.131+0000 7ff942000640 1 -- 172.21.15.28:0/3977776378 shutdown_connections 2024-02-21T23:05:01.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.131+0000 7ff942000640 1 --2- 172.21.15.28:0/3977776378 >> v2:172.21.15.81:3300/0 conn(0x7ff93c06dbf0 0x7ff93c158ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:01.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.131+0000 7ff942000640 1 --2- 172.21.15.28:0/3977776378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff93c0aff90 0x7ff93c0c51e0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:01.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.131+0000 7ff942000640 1 -- 172.21.15.28:0/3977776378 >> 172.21.15.28:0/3977776378 conn(0x7ff93c0ac370 msgr2=0x7ff93c0ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:01.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.131+0000 7ff942000640 1 -- 172.21.15.28:0/3977776378 shutdown_connections 2024-02-21T23:05:01.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.131+0000 7ff942000640 1 -- 172.21.15.28:0/3977776378 wait complete. 2024-02-21T23:05:01.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.132+0000 7ff942000640 1 Processor -- start 2024-02-21T23:05:01.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.133+0000 7ff942000640 1 -- start start 2024-02-21T23:05:01.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.133+0000 7ff942000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff93c06dbf0 0x7ff93c0c12f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:01.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.133+0000 7ff942000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff93c0aff90 0x7ff93c0c43a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:01.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.133+0000 7ff942000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff93c0c5720 0x7ff93c0be240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:01.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.133+0000 7ff942000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff93c0c8760 con 0x7ff93c0aff90 2024-02-21T23:05:01.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.133+0000 7ff942000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff93c0be0c0 con 0x7ff93c0c5720 2024-02-21T23:05:01.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.133+0000 7ff942000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff93c0c88e0 con 0x7ff93c06dbf0 2024-02-21T23:05:01.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.134+0000 7ff93be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff93c0aff90 0x7ff93c0c43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:01.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.134+0000 7ff93be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff93c0aff90 0x7ff93c0c43a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45672/0 (socket says 172.21.15.28:45672) 2024-02-21T23:05:01.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.134+0000 7ff93be00640 1 -- 172.21.15.28:0/921796673 learned_addr learned my addr 172.21.15.28:0/921796673 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:01.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.134+0000 7ff940c00640 1 --2- 172.21.15.28:0/921796673 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff93c06dbf0 0x7ff93c0c12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:01.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.134+0000 7ff941600640 1 --2- 172.21.15.28:0/921796673 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff93c0c5720 0x7ff93c0be240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:01.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.134+0000 7ff93be00640 1 -- 172.21.15.28:0/921796673 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff93c06dbf0 msgr2=0x7ff93c0c12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:01.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.134+0000 7ff93be00640 1 --2- 172.21.15.28:0/921796673 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff93c06dbf0 0x7ff93c0c12f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:01.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.134+0000 7ff93be00640 1 -- 172.21.15.28:0/921796673 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff93c0c5720 msgr2=0x7ff93c0be240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:01.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.134+0000 7ff93be00640 1 --2- 172.21.15.28:0/921796673 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff93c0c5720 0x7ff93c0be240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:01.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.134+0000 7ff93be00640 1 -- 172.21.15.28:0/921796673 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff93c154690 con 0x7ff93c0aff90 2024-02-21T23:05:01.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.134+0000 7ff941600640 1 --2- 172.21.15.28:0/921796673 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff93c0c5720 0x7ff93c0be240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:05:01.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.135+0000 7ff93be00640 1 --2- 172.21.15.28:0/921796673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff93c0aff90 0x7ff93c0c43a0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7ff92800e810 tx=0x7ff92800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:01.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.135+0000 7ff938c00640 1 -- 172.21.15.28:0/921796673 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff92800f040 con 0x7ff93c0aff90 2024-02-21T23:05:01.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.135+0000 7ff938c00640 1 -- 172.21.15.28:0/921796673 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff928004030 con 0x7ff93c0aff90 2024-02-21T23:05:01.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.135+0000 7ff942000640 1 -- 172.21.15.28:0/921796673 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff93c154970 con 0x7ff93c0aff90 2024-02-21T23:05:01.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.135+0000 7ff938c00640 1 -- 172.21.15.28:0/921796673 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff928004400 con 0x7ff93c0aff90 2024-02-21T23:05:01.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.135+0000 7ff942000640 1 -- 172.21.15.28:0/921796673 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff93c0c48e0 con 0x7ff93c0aff90 2024-02-21T23:05:01.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.136+0000 7ff942000640 1 -- 172.21.15.28:0/921796673 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff93c04f290 con 0x7ff93c0aff90 2024-02-21T23:05:01.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.137+0000 7ff938c00640 1 -- 172.21.15.28:0/921796673 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff92800c6d0 con 0x7ff93c0aff90 2024-02-21T23:05:01.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.139+0000 7ff938c00640 1 --2- 172.21.15.28:0/921796673 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff91c06b4d0 0x7ff91c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:01.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.139+0000 7ff938c00640 1 -- 172.21.15.28:0/921796673 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff92808a080 con 0x7ff93c0aff90 2024-02-21T23:05:01.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.139+0000 7ff940c00640 1 --2- 172.21.15.28:0/921796673 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff91c06b4d0 0x7ff91c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:01.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.141+0000 7ff940c00640 1 --2- 172.21.15.28:0/921796673 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff91c06b4d0 0x7ff91c06d990 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7ff930036000 tx=0x7ff930008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:01.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.143+0000 7ff938c00640 1 -- 172.21.15.28:0/921796673 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ff928017020 con 0x7ff93c0aff90 2024-02-21T23:05:01.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.383+0000 7ff942000640 1 -- 172.21.15.28:0/921796673 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff93c06d9d0 con 0x7ff93c0aff90 2024-02-21T23:05:01.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.384+0000 7ff938c00640 1 -- 172.21.15.28:0/921796673 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+4935 (secure 0 0 0) 0x7ff928058d60 con 0x7ff93c0aff90 2024-02-21T23:05:01.373 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:05:01.373 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":32,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5006,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3468780363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3468780363},{"type":"v1","addr":"172.21.15.28:6835","nonce":3468780363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27},{"gid":5221,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/2152160500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2152160500},{"type":"v1","addr":"172.21.15.81:6835","nonce":2152160500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27},{"gid":5224,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/442767245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":442767245},{"type":"v1","addr":"172.21.15.81:6837","nonce":442767245}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:04:53.432188+0000","modified":"2024-02-21T23:04:56.493528+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5715},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5715":{"gid":5715,"name":"a","rank":0,"incarnation":28,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6834/2803823391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2803823391},{"type":"v1","addr":"172.21.15.28:6834","nonce":2803823391}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-02-21T23:05:01.373 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 32 2024-02-21T23:05:01.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.386+0000 7ff942000640 1 -- 172.21.15.28:0/921796673 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff91c06b4d0 msgr2=0x7ff91c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:01.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.386+0000 7ff942000640 1 --2- 172.21.15.28:0/921796673 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff91c06b4d0 0x7ff91c06d990 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7ff930036000 tx=0x7ff930008030 comp rx=0 tx=0).stop 2024-02-21T23:05:01.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.386+0000 7ff942000640 1 -- 172.21.15.28:0/921796673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff93c0aff90 msgr2=0x7ff93c0c43a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:01.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.386+0000 7ff942000640 1 --2- 172.21.15.28:0/921796673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff93c0aff90 0x7ff93c0c43a0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7ff92800e810 tx=0x7ff92800ecd0 comp rx=0 tx=0).stop 2024-02-21T23:05:01.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.386+0000 7ff942000640 1 -- 172.21.15.28:0/921796673 shutdown_connections 2024-02-21T23:05:01.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.386+0000 7ff942000640 1 --2- 172.21.15.28:0/921796673 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff91c06b4d0 0x7ff91c06d990 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:01.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.386+0000 7ff942000640 1 --2- 172.21.15.28:0/921796673 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff93c0c5720 0x7ff93c0be240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:01.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.386+0000 7ff942000640 1 --2- 172.21.15.28:0/921796673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff93c0aff90 0x7ff93c0c43a0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:01.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.386+0000 7ff942000640 1 --2- 172.21.15.28:0/921796673 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff93c06dbf0 0x7ff93c0c12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:01.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.386+0000 7ff942000640 1 -- 172.21.15.28:0/921796673 >> 172.21.15.28:0/921796673 conn(0x7ff93c0ac370 msgr2=0x7ff93c0ac750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:01.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.386+0000 7ff942000640 1 -- 172.21.15.28:0/921796673 shutdown_connections 2024-02-21T23:05:01.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.386+0000 7ff942000640 1 -- 172.21.15.28:0/921796673 wait complete. 2024-02-21T23:05:01.397 INFO:tasks.cephfs.test_failover:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 3, "epoch": 32, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 3, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-02-21T23:04:53.432188+0000", "damaged": [], "data_pools": [ 8 ], "enabled": true, "epoch": 32, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5715": { "addr": "172.21.15.28:6834/2803823391", "addrs": { "addrvec": [ { "addr": "172.21.15.28:6832", "nonce": 2803823391, "type": "v2" }, { "addr": "172.21.15.28:6834", "nonce": 2803823391, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5715, "incarnation": 28, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 7, "modified": "2024-02-21T23:04:56.493528+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5715 } } } ], "standbys": [ { "addr": "172.21.15.28:6835/3468780363", "addrs": { "addrvec": [ { "addr": "172.21.15.28:6833", "nonce": 3468780363, "type": "v2" }, { "addr": "172.21.15.28:6835", "nonce": 3468780363, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 27, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5006, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.81:6835/2152160500", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6834", "nonce": 2152160500, "type": "v2" }, { "addr": "172.21.15.81:6835", "nonce": 2152160500, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 27, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5221, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.81:6837/442767245", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6836", "nonce": 442767245, "type": "v2" }, { "addr": "172.21.15.81:6837", "nonce": 442767245, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 27, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5224, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-02-21T23:05:01.397 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-02-21T23:05:01.397 INFO:tasks.cephfs.test_failover:current = [{'gid': 5715, 'name': 'a', 'rank': 0, 'incarnation': 28, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6834/2803823391', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 2803823391}, {'type': 'v1', 'addr': '172.21.15.28:6834', 'nonce': 2803823391}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 5221, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.81:6835/2152160500', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6834', 'nonce': 2152160500}, {'type': 'v1', 'addr': '172.21.15.81:6835', 'nonce': 2152160500}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 27}, {'gid': 5006, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.28:6835/3468780363', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6833', 'nonce': 3468780363}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3468780363}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 27}, {'gid': 5224, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.81:6837/442767245', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 442767245}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 442767245}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 27}] 2024-02-21T23:05:01.397 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.c mds_join_fs cephfs2 2024-02-21T23:05:01.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.615+0000 7f518da00640 1 Processor -- start 2024-02-21T23:05:01.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.615+0000 7f518da00640 1 -- start start 2024-02-21T23:05:01.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.617+0000 7f518da00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f518813e0c0 0x7f51880c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:01.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.617+0000 7f518da00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f51880c76f0 0x7f518806c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:01.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.617+0000 7f518da00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5188158550 con 0x7f518806dbc0 2024-02-21T23:05:01.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.617+0000 7f518da00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51880c89a0 con 0x7f518813e0c0 2024-02-21T23:05:01.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.617+0000 7f518da00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51881587d0 con 0x7f51880c76f0 2024-02-21T23:05:01.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.618+0000 7f5187e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f518813e0c0 0x7f51880c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:01.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.618+0000 7f5187e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f518813e0c0 0x7f51880c71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45674/0 (socket says 172.21.15.28:45674) 2024-02-21T23:05:01.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.618+0000 7f5187e00640 1 -- 172.21.15.28:0/3200778623 learned_addr learned my addr 172.21.15.28:0/3200778623 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:01.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.618+0000 7f5187400640 1 --1- 172.21.15.28:0/3200778623 >> v1:172.21.15.28:6789/0 conn(0x7f518806dbc0 0x7f518806c280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34502/0 (socket says 172.21.15.28:34502) 2024-02-21T23:05:01.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.618+0000 7f518d000640 1 --2- 172.21.15.28:0/3200778623 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f51880c76f0 0x7f518806c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:01.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.618+0000 7f5187e00640 1 -- 172.21.15.28:0/3200778623 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f51880c76f0 msgr2=0x7f518806c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:01.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.618+0000 7f5187e00640 1 --2- 172.21.15.28:0/3200778623 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f51880c76f0 0x7f518806c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:01.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.618+0000 7f5187e00640 1 -- 172.21.15.28:0/3200778623 >> v1:172.21.15.28:6789/0 conn(0x7f518806dbc0 legacy=0x7f518806c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:01.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.618+0000 7f5187e00640 1 -- 172.21.15.28:0/3200778623 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5188158ab0 con 0x7f518813e0c0 2024-02-21T23:05:01.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.619+0000 7f5187e00640 1 --2- 172.21.15.28:0/3200778623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f518813e0c0 0x7f51880c71b0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f5178009870 tx=0x7f517802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30ea4a7607d520f server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:01.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.620+0000 7f5186a00640 1 -- 172.21.15.28:0/3200778623 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5178035020 con 0x7f518813e0c0 2024-02-21T23:05:01.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.620+0000 7f5186a00640 1 -- 172.21.15.28:0/3200778623 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5178031640 con 0x7f518813e0c0 2024-02-21T23:05:01.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.620+0000 7f5186a00640 1 -- 172.21.15.28:0/3200778623 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f517802d040 con 0x7f518813e0c0 2024-02-21T23:05:01.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.620+0000 7f518da00640 1 -- 172.21.15.28:0/3200778623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f518813e0c0 msgr2=0x7f51880c71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:01.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.620+0000 7f518da00640 1 --2- 172.21.15.28:0/3200778623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f518813e0c0 0x7f51880c71b0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f5178009870 tx=0x7f517802b8f0 comp rx=0 tx=0).stop 2024-02-21T23:05:01.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.620+0000 7f518da00640 1 -- 172.21.15.28:0/3200778623 shutdown_connections 2024-02-21T23:05:01.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.620+0000 7f518da00640 1 --2- 172.21.15.28:0/3200778623 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f51880c76f0 0x7f518806c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:01.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.620+0000 7f518da00640 1 --2- 172.21.15.28:0/3200778623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f518813e0c0 0x7f51880c71b0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:01.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.620+0000 7f518da00640 1 -- 172.21.15.28:0/3200778623 >> 172.21.15.28:0/3200778623 conn(0x7f5188073110 msgr2=0x7f51880ac490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:01.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.620+0000 7f518da00640 1 -- 172.21.15.28:0/3200778623 shutdown_connections 2024-02-21T23:05:01.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.620+0000 7f518da00640 1 -- 172.21.15.28:0/3200778623 wait complete. 2024-02-21T23:05:01.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.622+0000 7f518da00640 1 Processor -- start 2024-02-21T23:05:01.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.622+0000 7f518da00640 1 -- start start 2024-02-21T23:05:01.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.623+0000 7f518da00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f518806dbc0 0x7f5188151570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:01.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.623+0000 7f518da00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51880c76f0 0x7f51881554c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:01.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.623+0000 7f518da00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f518813e0c0 0x7f5188155a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:01.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.623+0000 7f518da00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f518806ced0 con 0x7f51880c76f0 2024-02-21T23:05:01.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.623+0000 7f518da00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5188159070 con 0x7f518806dbc0 2024-02-21T23:05:01.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.623+0000 7f518da00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f518813e520 con 0x7f518813e0c0 2024-02-21T23:05:01.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.623+0000 7f5187400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51880c76f0 0x7f51881554c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:01.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.623+0000 7f5187400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51880c76f0 0x7f51881554c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45688/0 (socket says 172.21.15.28:45688) 2024-02-21T23:05:01.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.623+0000 7f5187400640 1 -- 172.21.15.28:0/2471402726 learned_addr learned my addr 172.21.15.28:0/2471402726 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:01.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.623+0000 7f5187e00640 1 --2- 172.21.15.28:0/2471402726 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f518806dbc0 0x7f5188151570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:01.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.623+0000 7f518d000640 1 --2- 172.21.15.28:0/2471402726 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f518813e0c0 0x7f5188155a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:01.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.623+0000 7f5187400640 1 -- 172.21.15.28:0/2471402726 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f518813e0c0 msgr2=0x7f5188155a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:01.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.623+0000 7f5187400640 1 --2- 172.21.15.28:0/2471402726 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f518813e0c0 0x7f5188155a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:01.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.623+0000 7f5187400640 1 -- 172.21.15.28:0/2471402726 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f518806dbc0 msgr2=0x7f5188151570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:01.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.623+0000 7f5187400640 1 --2- 172.21.15.28:0/2471402726 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f518806dbc0 0x7f5188151570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:01.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.623+0000 7f5187400640 1 -- 172.21.15.28:0/2471402726 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5188150040 con 0x7f51880c76f0 2024-02-21T23:05:01.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.624+0000 7f5187400640 1 --2- 172.21.15.28:0/2471402726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51880c76f0 0x7f51881554c0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f517400c430 tx=0x7f517400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:01.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.624+0000 7f517fe00640 1 -- 172.21.15.28:0/2471402726 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51740074e0 con 0x7f51880c76f0 2024-02-21T23:05:01.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.624+0000 7f517fe00640 1 -- 172.21.15.28:0/2471402726 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5174007680 con 0x7f51880c76f0 2024-02-21T23:05:01.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.624+0000 7f517fe00640 1 -- 172.21.15.28:0/2471402726 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5174004a00 con 0x7f51880c76f0 2024-02-21T23:05:01.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.624+0000 7f518da00640 1 -- 172.21.15.28:0/2471402726 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5188150320 con 0x7f51880c76f0 2024-02-21T23:05:01.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.624+0000 7f518da00640 1 -- 172.21.15.28:0/2471402726 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51881508e0 con 0x7f51880c76f0 2024-02-21T23:05:01.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.625+0000 7f518da00640 1 -- 172.21.15.28:0/2471402726 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f518813b8e0 con 0x7f51880c76f0 2024-02-21T23:05:01.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.626+0000 7f517fe00640 1 -- 172.21.15.28:0/2471402726 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f517401a020 con 0x7f51880c76f0 2024-02-21T23:05:01.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.627+0000 7f517fe00640 1 --2- 172.21.15.28:0/2471402726 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f516806b4d0 0x7f516806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:01.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.627+0000 7f517fe00640 1 -- 172.21.15.28:0/2471402726 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5174089af0 con 0x7f51880c76f0 2024-02-21T23:05:01.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.628+0000 7f5187e00640 1 --2- 172.21.15.28:0/2471402726 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f516806b4d0 0x7f516806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:01.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.629+0000 7f5187e00640 1 --2- 172.21.15.28:0/2471402726 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f516806b4d0 0x7f516806d990 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f5178036000 tx=0x7f5178008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:01.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.632+0000 7f517fe00640 1 -- 172.21.15.28:0/2471402726 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f518813b8e0 con 0x7f51880c76f0 2024-02-21T23:05:01.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.832+0000 7f518da00640 1 -- 172.21.15.28:0/2471402726 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) v1 -- 0x7f518804f290 con 0x7f51880c76f0 2024-02-21T23:05:01.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.837+0000 7f517fe00640 1 -- 172.21.15.28:0/2471402726 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v16)=0 v16) v1 ==== 117+0+0 (secure 0 0 0) 0x7f5188158550 con 0x7f51880c76f0 2024-02-21T23:05:01.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.840+0000 7f518da00640 1 -- 172.21.15.28:0/2471402726 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f516806b4d0 msgr2=0x7f516806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:01.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.840+0000 7f518da00640 1 --2- 172.21.15.28:0/2471402726 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f516806b4d0 0x7f516806d990 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f5178036000 tx=0x7f5178008030 comp rx=0 tx=0).stop 2024-02-21T23:05:01.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.840+0000 7f518da00640 1 -- 172.21.15.28:0/2471402726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51880c76f0 msgr2=0x7f51881554c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:01.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.840+0000 7f518da00640 1 --2- 172.21.15.28:0/2471402726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51880c76f0 0x7f51881554c0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f517400c430 tx=0x7f517400c8f0 comp rx=0 tx=0).stop 2024-02-21T23:05:01.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.840+0000 7f518da00640 1 -- 172.21.15.28:0/2471402726 shutdown_connections 2024-02-21T23:05:01.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.840+0000 7f518da00640 1 --2- 172.21.15.28:0/2471402726 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f516806b4d0 0x7f516806d990 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:01.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.840+0000 7f518da00640 1 --2- 172.21.15.28:0/2471402726 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f518813e0c0 0x7f5188155a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:01.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.840+0000 7f518da00640 1 --2- 172.21.15.28:0/2471402726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51880c76f0 0x7f51881554c0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:01.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.840+0000 7f518da00640 1 --2- 172.21.15.28:0/2471402726 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f518806dbc0 0x7f5188151570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:01.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.840+0000 7f518da00640 1 -- 172.21.15.28:0/2471402726 >> 172.21.15.28:0/2471402726 conn(0x7f5188073110 msgr2=0x7f518813bb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:01.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.840+0000 7f518da00640 1 -- 172.21.15.28:0/2471402726 shutdown_connections 2024-02-21T23:05:01.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:01.840+0000 7f518da00640 1 -- 172.21.15.28:0/2471402726 wait complete. 2024-02-21T23:05:01.849 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.b mds_join_fs cephfs2 2024-02-21T23:05:02.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.080+0000 7f1d11e00640 1 Processor -- start 2024-02-21T23:05:02.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.081+0000 7f1d11e00640 1 -- start start 2024-02-21T23:05:02.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.083+0000 7f1d11e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1d0c14eb00 0x7f1d0c06d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:02.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.083+0000 7f1d11e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f1d0c0c17d0 0x7f1d0c0c1d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:02.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.083+0000 7f1d11e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d0c06dd70 con 0x7f1d0c06f250 2024-02-21T23:05:02.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.083+0000 7f1d11e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d0c152180 con 0x7f1d0c14eb00 2024-02-21T23:05:02.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.083+0000 7f1d11e00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d0c0a60f0 con 0x7f1d0c0c17d0 2024-02-21T23:05:02.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.084+0000 7f1d0aa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1d0c14eb00 0x7f1d0c06d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:02.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.084+0000 7f1d0aa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1d0c14eb00 0x7f1d0c06d120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45690/0 (socket says 172.21.15.28:45690) 2024-02-21T23:05:02.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.084+0000 7f1d0aa00640 1 -- 172.21.15.28:0/1395258013 learned_addr learned my addr 172.21.15.28:0/1395258013 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:02.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.084+0000 7f1d0b400640 1 --2- 172.21.15.28:0/1395258013 >> v2:172.21.15.81:3300/0 conn(0x7f1d0c0c17d0 0x7f1d0c0c1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:02.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.084+0000 7f1d0aa00640 1 -- 172.21.15.28:0/1395258013 >> v2:172.21.15.81:3300/0 conn(0x7f1d0c0c17d0 msgr2=0x7f1d0c0c1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:02.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.084+0000 7f1d0aa00640 1 --2- 172.21.15.28:0/1395258013 >> v2:172.21.15.81:3300/0 conn(0x7f1d0c0c17d0 0x7f1d0c0c1d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:02.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.084+0000 7f1d0aa00640 1 -- 172.21.15.28:0/1395258013 >> v1:172.21.15.28:6789/0 conn(0x7f1d0c06f250 legacy=0x7f1d0c06d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:02.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.084+0000 7f1d0aa00640 1 -- 172.21.15.28:0/1395258013 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d0c0c1390 con 0x7f1d0c14eb00 2024-02-21T23:05:02.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.084+0000 7f1d0b400640 1 --2- 172.21.15.28:0/1395258013 >> v2:172.21.15.81:3300/0 conn(0x7f1d0c0c17d0 0x7f1d0c0c1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:05:02.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.084+0000 7f1d09600640 1 -- 172.21.15.28:0/1395258013 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2223429651 0 0) 0x7f1d0c06dd70 con 0x7f1d0c06f250 2024-02-21T23:05:02.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.085+0000 7f1d0aa00640 1 --2- 172.21.15.28:0/1395258013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d0c14eb00 0x7f1d0c06d120 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f1d0c0c0d10 tx=0x7f1cfc02deb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29c59e95b34705dc server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:02.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.086+0000 7f1d09600640 1 -- 172.21.15.28:0/1395258013 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cfc034070 con 0x7f1d0c14eb00 2024-02-21T23:05:02.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.086+0000 7f1d09600640 1 -- 172.21.15.28:0/1395258013 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cfc02ec60 con 0x7f1d0c14eb00 2024-02-21T23:05:02.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.086+0000 7f1d09600640 1 -- 172.21.15.28:0/1395258013 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cfc038620 con 0x7f1d0c14eb00 2024-02-21T23:05:02.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.086+0000 7f1d11e00640 1 -- 172.21.15.28:0/1395258013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d0c14eb00 msgr2=0x7f1d0c06d120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:02.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.086+0000 7f1d11e00640 1 --2- 172.21.15.28:0/1395258013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d0c14eb00 0x7f1d0c06d120 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f1d0c0c0d10 tx=0x7f1cfc02deb0 comp rx=0 tx=0).stop 2024-02-21T23:05:02.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.086+0000 7f1d11e00640 1 -- 172.21.15.28:0/1395258013 shutdown_connections 2024-02-21T23:05:02.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.086+0000 7f1d11e00640 1 --2- 172.21.15.28:0/1395258013 >> v2:172.21.15.81:3300/0 conn(0x7f1d0c0c17d0 0x7f1d0c0c1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:02.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.086+0000 7f1d11e00640 1 --2- 172.21.15.28:0/1395258013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d0c14eb00 0x7f1d0c06d120 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:02.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.086+0000 7f1d11e00640 1 -- 172.21.15.28:0/1395258013 >> 172.21.15.28:0/1395258013 conn(0x7f1d0c0a85a0 msgr2=0x7f1d0c0a8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:02.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.087+0000 7f1d11e00640 1 -- 172.21.15.28:0/1395258013 shutdown_connections 2024-02-21T23:05:02.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.087+0000 7f1d11e00640 1 -- 172.21.15.28:0/1395258013 wait complete. 2024-02-21T23:05:02.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.088+0000 7f1d11e00640 1 Processor -- start 2024-02-21T23:05:02.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.088+0000 7f1d11e00640 1 -- start start 2024-02-21T23:05:02.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.089+0000 7f1d11e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d0c06f250 0x7f1d0c1de490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:02.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.089+0000 7f1d11e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1d0c0c17d0 0x7f1d0c146d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:02.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.089+0000 7f1d11e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d0c14eb00 0x7f1d0c1472a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:02.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.089+0000 7f1d11e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d0c06aff0 con 0x7f1d0c14eb00 2024-02-21T23:05:02.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.089+0000 7f1d11e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d0c06ae70 con 0x7f1d0c06f250 2024-02-21T23:05:02.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.089+0000 7f1d11e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d0c06b170 con 0x7f1d0c0c17d0 2024-02-21T23:05:02.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.090+0000 7f1d0b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d0c14eb00 0x7f1d0c1472a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:02.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.090+0000 7f1d0b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d0c14eb00 0x7f1d0c1472a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45700/0 (socket says 172.21.15.28:45700) 2024-02-21T23:05:02.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.090+0000 7f1d0b400640 1 -- 172.21.15.28:0/153961542 learned_addr learned my addr 172.21.15.28:0/153961542 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:02.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.090+0000 7f1d0aa00640 1 --2- 172.21.15.28:0/153961542 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d0c06f250 0x7f1d0c1de490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:02.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.090+0000 7f1d0a000640 1 --2- 172.21.15.28:0/153961542 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1d0c0c17d0 0x7f1d0c146d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:02.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.090+0000 7f1d0b400640 1 -- 172.21.15.28:0/153961542 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1d0c0c17d0 msgr2=0x7f1d0c146d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:02.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.090+0000 7f1d0b400640 1 --2- 172.21.15.28:0/153961542 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1d0c0c17d0 0x7f1d0c146d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:02.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.090+0000 7f1d0b400640 1 -- 172.21.15.28:0/153961542 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d0c06f250 msgr2=0x7f1d0c1de490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:02.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.090+0000 7f1d0b400640 1 --2- 172.21.15.28:0/153961542 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d0c06f250 0x7f1d0c1de490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:02.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.090+0000 7f1d0b400640 1 -- 172.21.15.28:0/153961542 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d0c0ab910 con 0x7f1d0c14eb00 2024-02-21T23:05:02.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.091+0000 7f1d0b400640 1 --2- 172.21.15.28:0/153961542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d0c14eb00 0x7f1d0c1472a0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f1d0400cef0 tx=0x7f1d0400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:02.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.091+0000 7f1d03400640 1 -- 172.21.15.28:0/153961542 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d0401a070 con 0x7f1d0c14eb00 2024-02-21T23:05:02.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.091+0000 7f1d03400640 1 -- 172.21.15.28:0/153961542 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d0400d860 con 0x7f1d0c14eb00 2024-02-21T23:05:02.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.091+0000 7f1d03400640 1 -- 172.21.15.28:0/153961542 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d0400dc50 con 0x7f1d0c14eb00 2024-02-21T23:05:02.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.091+0000 7f1d11e00640 1 -- 172.21.15.28:0/153961542 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d0c0aa2f0 con 0x7f1d0c14eb00 2024-02-21T23:05:02.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.091+0000 7f1d11e00640 1 -- 172.21.15.28:0/153961542 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d0c0ab4f0 con 0x7f1d0c14eb00 2024-02-21T23:05:02.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.092+0000 7f1d11e00640 1 -- 172.21.15.28:0/153961542 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d0c067f30 con 0x7f1d0c14eb00 2024-02-21T23:05:02.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.093+0000 7f1d03400640 1 -- 172.21.15.28:0/153961542 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1d0401d070 con 0x7f1d0c14eb00 2024-02-21T23:05:02.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.095+0000 7f1d03400640 1 --2- 172.21.15.28:0/153961542 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1cec06b500 0x7f1cec06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:02.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.095+0000 7f1d03400640 1 -- 172.21.15.28:0/153961542 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1d0408ac30 con 0x7f1d0c14eb00 2024-02-21T23:05:02.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.095+0000 7f1d0aa00640 1 --2- 172.21.15.28:0/153961542 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1cec06b500 0x7f1cec06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:02.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.096+0000 7f1d0aa00640 1 --2- 172.21.15.28:0/153961542 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1cec06b500 0x7f1cec06d9c0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f1cfc039000 tx=0x7f1cfc034000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:02.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.099+0000 7f1d03400640 1 -- 172.21.15.28:0/153961542 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1d04059910 con 0x7f1d0c14eb00 2024-02-21T23:05:02.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.351+0000 7f1d11e00640 1 -- 172.21.15.28:0/153961542 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) v1 -- 0x7f1d0c06bcc0 con 0x7f1d0c14eb00 2024-02-21T23:05:02.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.357+0000 7f1d03400640 1 -- 172.21.15.28:0/153961542 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v17)=0 v17) v1 ==== 117+0+0 (secure 0 0 0) 0x7f1d04059210 con 0x7f1d0c14eb00 2024-02-21T23:05:02.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.360+0000 7f1d11e00640 1 -- 172.21.15.28:0/153961542 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1cec06b500 msgr2=0x7f1cec06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:02.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.360+0000 7f1d11e00640 1 --2- 172.21.15.28:0/153961542 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1cec06b500 0x7f1cec06d9c0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f1cfc039000 tx=0x7f1cfc034000 comp rx=0 tx=0).stop 2024-02-21T23:05:02.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.360+0000 7f1d11e00640 1 -- 172.21.15.28:0/153961542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d0c14eb00 msgr2=0x7f1d0c1472a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:02.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.360+0000 7f1d11e00640 1 --2- 172.21.15.28:0/153961542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d0c14eb00 0x7f1d0c1472a0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f1d0400cef0 tx=0x7f1d0400a710 comp rx=0 tx=0).stop 2024-02-21T23:05:02.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.360+0000 7f1d11e00640 1 -- 172.21.15.28:0/153961542 shutdown_connections 2024-02-21T23:05:02.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.360+0000 7f1d11e00640 1 --2- 172.21.15.28:0/153961542 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1cec06b500 0x7f1cec06d9c0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:02.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.360+0000 7f1d11e00640 1 --2- 172.21.15.28:0/153961542 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1d0c0c17d0 0x7f1d0c146d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:02.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.360+0000 7f1d11e00640 1 --2- 172.21.15.28:0/153961542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d0c14eb00 0x7f1d0c1472a0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:02.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.360+0000 7f1d11e00640 1 --2- 172.21.15.28:0/153961542 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d0c06f250 0x7f1d0c1de490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:02.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.360+0000 7f1d11e00640 1 -- 172.21.15.28:0/153961542 >> 172.21.15.28:0/153961542 conn(0x7f1d0c0a85a0 msgr2=0x7f1d0c0c57c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:02.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.360+0000 7f1d11e00640 1 -- 172.21.15.28:0/153961542 shutdown_connections 2024-02-21T23:05:02.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.360+0000 7f1d11e00640 1 -- 172.21.15.28:0/153961542 wait complete. 2024-02-21T23:05:02.370 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.d mds_join_fs cephfs2 2024-02-21T23:05:02.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.596+0000 7f5029600640 1 Processor -- start 2024-02-21T23:05:02.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.597+0000 7f5029600640 1 -- start start 2024-02-21T23:05:02.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.600+0000 7f5029600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f50240c4030 0x7f50240c5d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:02.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.600+0000 7f5029600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f50240c4d30 0x7f502406cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:02.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.600+0000 7f5029600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50241594d0 con 0x7f50240c62a0 2024-02-21T23:05:02.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.600+0000 7f5029600640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5024155500 con 0x7f50240c4d30 2024-02-21T23:05:02.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.600+0000 7f5029600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50241557e0 con 0x7f50240c4030 2024-02-21T23:05:02.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.601+0000 7f5022a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f50240c4030 0x7f50240c5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:02.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.601+0000 7f5022a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f50240c4030 0x7f50240c5d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3301/0 says I am v2:172.21.15.28:34546/0 (socket says 172.21.15.28:34546) 2024-02-21T23:05:02.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.601+0000 7f5022a00640 1 -- 172.21.15.28:0/2123789969 learned_addr learned my addr 172.21.15.28:0/2123789969 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:02.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.601+0000 7f5022000640 1 --2- 172.21.15.28:0/2123789969 >> v2:172.21.15.81:3300/0 conn(0x7f50240c4d30 0x7f502406cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:02.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.601+0000 7f5022a00640 1 -- 172.21.15.28:0/2123789969 >> v2:172.21.15.81:3300/0 conn(0x7f50240c4d30 msgr2=0x7f502406cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:02.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.601+0000 7f5022a00640 1 --2- 172.21.15.28:0/2123789969 >> v2:172.21.15.81:3300/0 conn(0x7f50240c4d30 0x7f502406cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:02.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.601+0000 7f5022a00640 1 -- 172.21.15.28:0/2123789969 >> v1:172.21.15.81:6789/0 conn(0x7f50240c62a0 legacy=0x7f50240c6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:02.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.601+0000 7f5022a00640 1 -- 172.21.15.28:0/2123789969 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5024155ac0 con 0x7f50240c4030 2024-02-21T23:05:02.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.603+0000 7f5022a00640 1 --2- 172.21.15.28:0/2123789969 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f50240c4030 0x7f50240c5d60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5014009870 tx=0x7f501402b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9752f92eb3817b5a server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:02.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.603+0000 7f5021600640 1 -- 172.21.15.28:0/2123789969 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50140317c0 con 0x7f50240c4030 2024-02-21T23:05:02.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.603+0000 7f5021600640 1 -- 172.21.15.28:0/2123789969 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5014037070 con 0x7f50240c4030 2024-02-21T23:05:02.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.603+0000 7f5021600640 1 -- 172.21.15.28:0/2123789969 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5014031b80 con 0x7f50240c4030 2024-02-21T23:05:02.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.604+0000 7f5029600640 1 -- 172.21.15.28:0/2123789969 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f50240c4030 msgr2=0x7f50240c5d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:02.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.604+0000 7f5029600640 1 --2- 172.21.15.28:0/2123789969 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f50240c4030 0x7f50240c5d60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5014009870 tx=0x7f501402b860 comp rx=0 tx=0).stop 2024-02-21T23:05:02.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.604+0000 7f5029600640 1 -- 172.21.15.28:0/2123789969 shutdown_connections 2024-02-21T23:05:02.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.604+0000 7f5029600640 1 --2- 172.21.15.28:0/2123789969 >> v2:172.21.15.81:3300/0 conn(0x7f50240c4d30 0x7f502406cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:02.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.604+0000 7f5029600640 1 --2- 172.21.15.28:0/2123789969 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f50240c4030 0x7f50240c5d60 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:02.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.604+0000 7f5029600640 1 -- 172.21.15.28:0/2123789969 >> 172.21.15.28:0/2123789969 conn(0x7f50240725a0 msgr2=0x7f50240729b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:02.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.604+0000 7f5029600640 1 -- 172.21.15.28:0/2123789969 shutdown_connections 2024-02-21T23:05:02.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.604+0000 7f5029600640 1 -- 172.21.15.28:0/2123789969 wait complete. 2024-02-21T23:05:02.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.606+0000 7f5029600640 1 Processor -- start 2024-02-21T23:05:02.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.606+0000 7f5029600640 1 -- start start 2024-02-21T23:05:02.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.607+0000 7f5029600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f50240c4030 0x7f50240c90c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:02.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.607+0000 7f5029600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50240c4d30 0x7f5024150ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:02.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.607+0000 7f5029600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f50240c62a0 0x7f5024150ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:02.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.607+0000 7f5029600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50241592b0 con 0x7f50240c4d30 2024-02-21T23:05:02.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.607+0000 7f5029600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5024159130 con 0x7f50240c62a0 2024-02-21T23:05:02.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.607+0000 7f5029600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5024155fe0 con 0x7f50240c4030 2024-02-21T23:05:02.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.607+0000 7f5022000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50240c4d30 0x7f5024150ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:02.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.608+0000 7f5022000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50240c4d30 0x7f5024150ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45716/0 (socket says 172.21.15.28:45716) 2024-02-21T23:05:02.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.608+0000 7f5022000640 1 -- 172.21.15.28:0/3246109179 learned_addr learned my addr 172.21.15.28:0/3246109179 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:02.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.608+0000 7f5022a00640 1 --2- 172.21.15.28:0/3246109179 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f50240c4030 0x7f50240c90c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:02.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.608+0000 7f5023400640 1 --2- 172.21.15.28:0/3246109179 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f50240c62a0 0x7f5024150ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:02.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.608+0000 7f5022000640 1 -- 172.21.15.28:0/3246109179 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f50240c4030 msgr2=0x7f50240c90c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:02.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.608+0000 7f5022000640 1 --2- 172.21.15.28:0/3246109179 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f50240c4030 0x7f50240c90c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:02.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.608+0000 7f5022000640 1 -- 172.21.15.28:0/3246109179 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f50240c62a0 msgr2=0x7f5024150ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:02.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.608+0000 7f5022000640 1 --2- 172.21.15.28:0/3246109179 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f50240c62a0 0x7f5024150ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:02.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.608+0000 7f5022000640 1 -- 172.21.15.28:0/3246109179 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f502414d410 con 0x7f50240c4d30 2024-02-21T23:05:02.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.608+0000 7f5022000640 1 --2- 172.21.15.28:0/3246109179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50240c4d30 0x7f5024150ab0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f501000e980 tx=0x7f501000ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:02.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.609+0000 7f501aa00640 1 -- 172.21.15.28:0/3246109179 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5010017070 con 0x7f50240c4d30 2024-02-21T23:05:02.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.609+0000 7f501aa00640 1 -- 172.21.15.28:0/3246109179 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f501000ca00 con 0x7f50240c4d30 2024-02-21T23:05:02.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.609+0000 7f501aa00640 1 -- 172.21.15.28:0/3246109179 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f501000cdd0 con 0x7f50240c4d30 2024-02-21T23:05:02.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.609+0000 7f5029600640 1 -- 172.21.15.28:0/3246109179 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5024150560 con 0x7f50240c4d30 2024-02-21T23:05:02.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.609+0000 7f5029600640 1 -- 172.21.15.28:0/3246109179 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50241eeca0 con 0x7f50240c4d30 2024-02-21T23:05:02.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.610+0000 7f5029600640 1 -- 172.21.15.28:0/3246109179 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5024067f30 con 0x7f50240c4d30 2024-02-21T23:05:02.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.611+0000 7f501aa00640 1 -- 172.21.15.28:0/3246109179 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f501001a070 con 0x7f50240c4d30 2024-02-21T23:05:02.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.612+0000 7f501aa00640 1 --2- 172.21.15.28:0/3246109179 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f500406b4d0 0x7f500406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:02.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.612+0000 7f501aa00640 1 -- 172.21.15.28:0/3246109179 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f501008a2e0 con 0x7f50240c4d30 2024-02-21T23:05:02.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.612+0000 7f5022a00640 1 --2- 172.21.15.28:0/3246109179 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f500406b4d0 0x7f500406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:02.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.614+0000 7f5022a00640 1 --2- 172.21.15.28:0/3246109179 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f500406b4d0 0x7f500406d990 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f5014036000 tx=0x7f501403d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:02.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.617+0000 7f501aa00640 1 -- 172.21.15.28:0/3246109179 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f5010058970 con 0x7f50240c4d30 2024-02-21T23:05:02.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.822+0000 7f5029600640 1 -- 172.21.15.28:0/3246109179 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) v1 -- 0x7f50240c6080 con 0x7f50240c4d30 2024-02-21T23:05:02.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.827+0000 7f501aa00640 1 -- 172.21.15.28:0/3246109179 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v18)=0 v18) v1 ==== 117+0+0 (secure 0 0 0) 0x7f501005ca10 con 0x7f50240c4d30 2024-02-21T23:05:02.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.831+0000 7f5029600640 1 -- 172.21.15.28:0/3246109179 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f500406b4d0 msgr2=0x7f500406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:02.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.831+0000 7f5029600640 1 --2- 172.21.15.28:0/3246109179 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f500406b4d0 0x7f500406d990 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f5014036000 tx=0x7f501403d040 comp rx=0 tx=0).stop 2024-02-21T23:05:02.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.831+0000 7f5029600640 1 -- 172.21.15.28:0/3246109179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50240c4d30 msgr2=0x7f5024150ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:02.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.831+0000 7f5029600640 1 --2- 172.21.15.28:0/3246109179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50240c4d30 0x7f5024150ab0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f501000e980 tx=0x7f501000ee40 comp rx=0 tx=0).stop 2024-02-21T23:05:02.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.831+0000 7f5029600640 1 -- 172.21.15.28:0/3246109179 shutdown_connections 2024-02-21T23:05:02.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.831+0000 7f5029600640 1 --2- 172.21.15.28:0/3246109179 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f500406b4d0 0x7f500406d990 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:02.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.831+0000 7f5029600640 1 --2- 172.21.15.28:0/3246109179 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f50240c62a0 0x7f5024150ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:02.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.831+0000 7f5029600640 1 --2- 172.21.15.28:0/3246109179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50240c4d30 0x7f5024150ab0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:02.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.831+0000 7f5029600640 1 --2- 172.21.15.28:0/3246109179 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f50240c4030 0x7f50240c90c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:02.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.831+0000 7f5029600640 1 -- 172.21.15.28:0/3246109179 >> 172.21.15.28:0/3246109179 conn(0x7f50240725a0 msgr2=0x7f50240c0260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:02.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.831+0000 7f5029600640 1 -- 172.21.15.28:0/3246109179 shutdown_connections 2024-02-21T23:05:02.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:02.831+0000 7f5029600640 1 -- 172.21.15.28:0/3246109179 wait complete. 2024-02-21T23:05:02.838 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:05:03.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.056+0000 7ff674e00640 1 Processor -- start 2024-02-21T23:05:03.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.056+0000 7ff674e00640 1 -- start start 2024-02-21T23:05:03.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.058+0000 7ff674e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff670155960 0x7ff6701608a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:03.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.058+0000 7ff674e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6700c74c0 con 0x7ff67006dbf0 2024-02-21T23:05:03.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.058+0000 7ff674e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6700c77a0 con 0x7ff6700aff90 2024-02-21T23:05:03.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.058+0000 7ff674e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6700c7270 con 0x7ff670155960 2024-02-21T23:05:03.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.058+0000 7ff66ea00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff67006dbf0 0x7ff670158ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34510/0 (socket says 172.21.15.28:34510) 2024-02-21T23:05:03.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.058+0000 7ff66ea00640 1 -- 172.21.15.28:0/3212724273 learned_addr learned my addr 172.21.15.28:0/3212724273 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:05:03.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.059+0000 7ff66fe00640 1 --2- 172.21.15.28:0/3212724273 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff670155960 0x7ff6701608a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:03.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.059+0000 7ff66e000640 1 -- 172.21.15.28:0/3212724273 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 995718942 0 0) 0x7ff6700c74c0 con 0x7ff67006dbf0 2024-02-21T23:05:03.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.059+0000 7ff66e000640 1 -- 172.21.15.28:0/3212724273 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff654003520 con 0x7ff67006dbf0 2024-02-21T23:05:03.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.059+0000 7ff66fe00640 1 -- 172.21.15.28:0/3212724273 >> v1:172.21.15.81:6789/0 conn(0x7ff6700aff90 legacy=0x7ff6700c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:03.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.059+0000 7ff66fe00640 1 -- 172.21.15.28:0/3212724273 >> v1:172.21.15.28:6789/0 conn(0x7ff67006dbf0 legacy=0x7ff670158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:03.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.059+0000 7ff66e000640 1 -- 172.21.15.28:0/3212724273 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4081767753 0 0) 0x7ff654003520 con 0x7ff67006dbf0 2024-02-21T23:05:03.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.059+0000 7ff66fe00640 1 -- 172.21.15.28:0/3212724273 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6700c7a80 con 0x7ff670155960 2024-02-21T23:05:03.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.061+0000 7ff66fe00640 1 --2- 172.21.15.28:0/3212724273 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff670155960 0x7ff6701608a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff668009dc0 tx=0x7ff66802bde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c78cc9a2b9ea36e server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:03.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.061+0000 7ff66e000640 1 -- 172.21.15.28:0/3212724273 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff668039070 con 0x7ff670155960 2024-02-21T23:05:03.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.061+0000 7ff66e000640 1 -- 172.21.15.28:0/3212724273 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff66802c910 con 0x7ff670155960 2024-02-21T23:05:03.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.061+0000 7ff66e000640 1 -- 172.21.15.28:0/3212724273 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff65c001670 con 0x7ff670155960 2024-02-21T23:05:03.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.062+0000 7ff674e00640 1 -- 172.21.15.28:0/3212724273 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff670155960 msgr2=0x7ff6701608a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:03.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.062+0000 7ff674e00640 1 --2- 172.21.15.28:0/3212724273 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff670155960 0x7ff6701608a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff668009dc0 tx=0x7ff66802bde0 comp rx=0 tx=0).stop 2024-02-21T23:05:03.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.062+0000 7ff674e00640 1 -- 172.21.15.28:0/3212724273 shutdown_connections 2024-02-21T23:05:03.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.062+0000 7ff674e00640 1 --2- 172.21.15.28:0/3212724273 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff670155960 0x7ff6701608a0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:03.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.062+0000 7ff674e00640 1 -- 172.21.15.28:0/3212724273 >> 172.21.15.28:0/3212724273 conn(0x7ff6700ac370 msgr2=0x7ff6700ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:03.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.062+0000 7ff674e00640 1 -- 172.21.15.28:0/3212724273 shutdown_connections 2024-02-21T23:05:03.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.062+0000 7ff674e00640 1 -- 172.21.15.28:0/3212724273 wait complete. 2024-02-21T23:05:03.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.064+0000 7ff674e00640 1 Processor -- start 2024-02-21T23:05:03.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.064+0000 7ff674e00640 1 -- start start 2024-02-21T23:05:03.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.065+0000 7ff674e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff67006dbf0 0x7ff6701eb4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:03.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.065+0000 7ff674e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6700aff90 0x7ff6700ae740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:03.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.065+0000 7ff674e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff670155960 0x7ff6701534f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:03.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.065+0000 7ff674e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6700c58f0 con 0x7ff6700aff90 2024-02-21T23:05:03.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.065+0000 7ff674e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff670158630 con 0x7ff670155960 2024-02-21T23:05:03.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.065+0000 7ff674e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6700c5a70 con 0x7ff67006dbf0 2024-02-21T23:05:03.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.065+0000 7ff66ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6700aff90 0x7ff6700ae740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:03.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.065+0000 7ff66ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6700aff90 0x7ff6700ae740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45722/0 (socket says 172.21.15.28:45722) 2024-02-21T23:05:03.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.065+0000 7ff66ea00640 1 -- 172.21.15.28:0/814167556 learned_addr learned my addr 172.21.15.28:0/814167556 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:03.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.065+0000 7ff66fe00640 1 --2- 172.21.15.28:0/814167556 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff670155960 0x7ff6701534f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:03.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.066+0000 7ff66f400640 1 --2- 172.21.15.28:0/814167556 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff67006dbf0 0x7ff6701eb4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:03.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.066+0000 7ff66ea00640 1 -- 172.21.15.28:0/814167556 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff67006dbf0 msgr2=0x7ff6701eb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:03.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.066+0000 7ff66ea00640 1 --2- 172.21.15.28:0/814167556 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff67006dbf0 0x7ff6701eb4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:03.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.066+0000 7ff66ea00640 1 -- 172.21.15.28:0/814167556 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff670155960 msgr2=0x7ff6701534f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:03.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.066+0000 7ff66ea00640 1 --2- 172.21.15.28:0/814167556 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff670155960 0x7ff6701534f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:03.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.066+0000 7ff66ea00640 1 -- 172.21.15.28:0/814167556 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff670154ae0 con 0x7ff6700aff90 2024-02-21T23:05:03.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.066+0000 7ff66fe00640 1 --2- 172.21.15.28:0/814167556 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff670155960 0x7ff6701534f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:05:03.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.066+0000 7ff66f400640 1 --2- 172.21.15.28:0/814167556 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff67006dbf0 0x7ff6701eb4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:05:03.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.066+0000 7ff66ea00640 1 --2- 172.21.15.28:0/814167556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6700aff90 0x7ff6700ae740 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7ff65c00c730 tx=0x7ff65c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:03.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.066+0000 7ff667400640 1 -- 172.21.15.28:0/814167556 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff65c017070 con 0x7ff6700aff90 2024-02-21T23:05:03.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.066+0000 7ff667400640 1 -- 172.21.15.28:0/814167556 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff65c005ad0 con 0x7ff6700aff90 2024-02-21T23:05:03.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.066+0000 7ff667400640 1 -- 172.21.15.28:0/814167556 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff65c007070 con 0x7ff6700aff90 2024-02-21T23:05:03.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.066+0000 7ff674e00640 1 -- 172.21.15.28:0/814167556 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff670154dc0 con 0x7ff6700aff90 2024-02-21T23:05:03.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.067+0000 7ff674e00640 1 -- 172.21.15.28:0/814167556 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff670152fd0 con 0x7ff6700aff90 2024-02-21T23:05:03.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.068+0000 7ff674e00640 1 -- 172.21.15.28:0/814167556 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6701531d0 con 0x7ff6700aff90 2024-02-21T23:05:03.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.069+0000 7ff667400640 1 -- 172.21.15.28:0/814167556 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff65c01a070 con 0x7ff6700aff90 2024-02-21T23:05:03.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.070+0000 7ff667400640 1 --2- 172.21.15.28:0/814167556 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff65006b4d0 0x7ff65006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:03.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.071+0000 7ff667400640 1 -- 172.21.15.28:0/814167556 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff65c089850 con 0x7ff6700aff90 2024-02-21T23:05:03.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.071+0000 7ff66f400640 1 --2- 172.21.15.28:0/814167556 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff65006b4d0 0x7ff65006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:03.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.072+0000 7ff66f400640 1 --2- 172.21.15.28:0/814167556 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff65006b4d0 0x7ff65006d990 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7ff6700bddd0 tx=0x7ff660009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:03.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.075+0000 7ff667400640 1 -- 172.21.15.28:0/814167556 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ff65c057e60 con 0x7ff6700aff90 2024-02-21T23:05:03.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.276+0000 7ff674e00640 1 -- 172.21.15.28:0/814167556 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff67004f290 con 0x7ff6700aff90 2024-02-21T23:05:03.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.278+0000 7ff667400640 1 -- 172.21.15.28:0/814167556 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v53) v1 ==== 74+0+16882 (secure 0 0 0) 0x7ff65c05ce20 con 0x7ff6700aff90 2024-02-21T23:05:03.268 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:05:03.268 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":53,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:04:54.959306+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:04:51.354130+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-02-21T23:04:52.441690+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:05:03.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.281+0000 7ff674e00640 1 -- 172.21.15.28:0/814167556 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff65006b4d0 msgr2=0x7ff65006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:03.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.281+0000 7ff674e00640 1 --2- 172.21.15.28:0/814167556 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff65006b4d0 0x7ff65006d990 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7ff6700bddd0 tx=0x7ff660009040 comp rx=0 tx=0).stop 2024-02-21T23:05:03.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.281+0000 7ff674e00640 1 -- 172.21.15.28:0/814167556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6700aff90 msgr2=0x7ff6700ae740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:03.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.281+0000 7ff674e00640 1 --2- 172.21.15.28:0/814167556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6700aff90 0x7ff6700ae740 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7ff65c00c730 tx=0x7ff65c00cbf0 comp rx=0 tx=0).stop 2024-02-21T23:05:03.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.281+0000 7ff674e00640 1 -- 172.21.15.28:0/814167556 shutdown_connections 2024-02-21T23:05:03.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.281+0000 7ff674e00640 1 --2- 172.21.15.28:0/814167556 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff65006b4d0 0x7ff65006d990 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:03.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.281+0000 7ff674e00640 1 --2- 172.21.15.28:0/814167556 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff670155960 0x7ff6701534f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:03.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.281+0000 7ff674e00640 1 --2- 172.21.15.28:0/814167556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6700aff90 0x7ff6700ae740 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:03.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.281+0000 7ff674e00640 1 --2- 172.21.15.28:0/814167556 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff67006dbf0 0x7ff6701eb4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:03.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.281+0000 7ff674e00640 1 -- 172.21.15.28:0/814167556 >> 172.21.15.28:0/814167556 conn(0x7ff6700ac370 msgr2=0x7ff6700c50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:03.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.281+0000 7ff674e00640 1 -- 172.21.15.28:0/814167556 shutdown_connections 2024-02-21T23:05:03.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.281+0000 7ff674e00640 1 -- 172.21.15.28:0/814167556 wait complete. 2024-02-21T23:05:03.296 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-21T23:02:30.728329+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-21T23:04:51.354130+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '52', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2024-02-21T23:04:52.441690+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '53', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-21T23:05:03.296 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-21T23:05:03.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.533+0000 7f76d7e00640 1 Processor -- start 2024-02-21T23:05:03.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.533+0000 7f76d7e00640 1 -- start start 2024-02-21T23:05:03.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.535+0000 7f76d7e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f76d8152650 0x7f76d80c8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:03.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.535+0000 7f76d7e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f76d81550e0 0x7f76d8155530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:03.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.535+0000 7f76d7e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76d8158920 con 0x7f76d806daa0 2024-02-21T23:05:03.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.535+0000 7f76d7e00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76d81586d0 con 0x7f76d8152650 2024-02-21T23:05:03.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.535+0000 7f76d7e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76d8158c00 con 0x7f76d81550e0 2024-02-21T23:05:03.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.535+0000 7f76d6000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f76d806daa0 0x7f76d80c7160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34514/0 (socket says 172.21.15.28:34514) 2024-02-21T23:05:03.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.535+0000 7f76d6000640 1 -- 172.21.15.28:0/75466978 learned_addr learned my addr 172.21.15.28:0/75466978 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:05:03.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.535+0000 7f76d7400640 1 --2- 172.21.15.28:0/75466978 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f76d81550e0 0x7f76d8155530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:03.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.536+0000 7f76d6a00640 1 --2- 172.21.15.28:0/75466978 >> v2:172.21.15.81:3300/0 conn(0x7f76d8152650 0x7f76d80c8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:03.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.536+0000 7f76d5600640 1 -- 172.21.15.28:0/75466978 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4121924136 0 0) 0x7f76d8158920 con 0x7f76d806daa0 2024-02-21T23:05:03.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.536+0000 7f76d5600640 1 -- 172.21.15.28:0/75466978 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76bc003520 con 0x7f76d806daa0 2024-02-21T23:05:03.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.536+0000 7f76d5600640 1 -- 172.21.15.28:0/75466978 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3953093111 0 0) 0x7f76bc003520 con 0x7f76d806daa0 2024-02-21T23:05:03.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.536+0000 7f76d5600640 1 -- 172.21.15.28:0/75466978 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f76d81550e0 msgr2=0x7f76d8155530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:03.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.536+0000 7f76d5600640 1 --2- 172.21.15.28:0/75466978 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f76d81550e0 0x7f76d8155530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:03.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.536+0000 7f76d5600640 1 -- 172.21.15.28:0/75466978 >> v2:172.21.15.81:3300/0 conn(0x7f76d8152650 msgr2=0x7f76d80c8ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:03.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.536+0000 7f76d5600640 1 --2- 172.21.15.28:0/75466978 >> v2:172.21.15.81:3300/0 conn(0x7f76d8152650 0x7f76d80c8ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:03.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.536+0000 7f76d5600640 1 -- 172.21.15.28:0/75466978 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76d8158ee0 con 0x7f76d806daa0 2024-02-21T23:05:03.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.536+0000 7f76d5600640 1 -- 172.21.15.28:0/75466978 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f76c4001670 con 0x7f76d806daa0 2024-02-21T23:05:03.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.536+0000 7f76d5600640 1 -- 172.21.15.28:0/75466978 >> v1:172.21.15.28:6789/0 conn(0x7f76d806daa0 legacy=0x7f76d80c7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:03.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.536+0000 7f76d5600640 1 --2- 172.21.15.28:0/75466978 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76bc0039b0 0x7f76bc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:03.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.536+0000 7f76d5600640 1 --2- 172.21.15.28:0/75466978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76bc004800 0x7f76bc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:03.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.536+0000 7f76d5600640 1 --2- 172.21.15.28:0/75466978 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f76d81550e0 0x7f76bc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:03.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.536+0000 7f76d5600640 1 -- 172.21.15.28:0/75466978 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76d81586d0 con 0x7f76bc004800 2024-02-21T23:05:03.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.536+0000 7f76d5600640 1 -- 172.21.15.28:0/75466978 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76d8158c00 con 0x7f76bc0039b0 2024-02-21T23:05:03.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.536+0000 7f76d5600640 1 -- 172.21.15.28:0/75466978 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76bc00ba10 con 0x7f76d81550e0 2024-02-21T23:05:03.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.537+0000 7f76d6000640 1 --2- 172.21.15.28:0/75466978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76bc004800 0x7f76bc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:03.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.537+0000 7f76d6a00640 1 --2- 172.21.15.28:0/75466978 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76bc0039b0 0x7f76bc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:03.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.537+0000 7f76d7400640 1 --2- 172.21.15.28:0/75466978 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f76d81550e0 0x7f76bc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:03.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.537+0000 7f76d6000640 1 -- 172.21.15.28:0/75466978 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f76d81550e0 msgr2=0x7f76bc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:03.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.537+0000 7f76d6000640 1 --2- 172.21.15.28:0/75466978 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f76d81550e0 0x7f76bc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:03.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.537+0000 7f76d6000640 1 -- 172.21.15.28:0/75466978 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76bc0039b0 msgr2=0x7f76bc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:03.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.537+0000 7f76d6000640 1 --2- 172.21.15.28:0/75466978 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76bc0039b0 0x7f76bc003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:03.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.537+0000 7f76d6000640 1 -- 172.21.15.28:0/75466978 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76bc003520 con 0x7f76bc004800 2024-02-21T23:05:03.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.539+0000 7f76d6000640 1 --2- 172.21.15.28:0/75466978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76bc004800 0x7f76bc004c80 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f76c400c1c0 tx=0x7f76bc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba141bd1940f0b1e server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:03.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.539+0000 7f76d5600640 1 -- 172.21.15.28:0/75466978 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c4007c60 con 0x7f76bc004800 2024-02-21T23:05:03.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.539+0000 7f76d5600640 1 -- 172.21.15.28:0/75466978 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76c4030070 con 0x7f76bc004800 2024-02-21T23:05:03.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.539+0000 7f76d5600640 1 -- 172.21.15.28:0/75466978 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c4005d30 con 0x7f76bc004800 2024-02-21T23:05:03.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.539+0000 7f76d7e00640 1 -- 172.21.15.28:0/75466978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76bc004800 msgr2=0x7f76bc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:03.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.539+0000 7f76d7e00640 1 --2- 172.21.15.28:0/75466978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76bc004800 0x7f76bc004c80 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f76c400c1c0 tx=0x7f76bc005830 comp rx=0 tx=0).stop 2024-02-21T23:05:03.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.540+0000 7f76d7400640 1 -- 172.21.15.28:0/75466978 reap_dead start 2024-02-21T23:05:03.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.540+0000 7f76d7e00640 1 -- 172.21.15.28:0/75466978 shutdown_connections 2024-02-21T23:05:03.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.540+0000 7f76d7e00640 1 -- 172.21.15.28:0/75466978 >> 172.21.15.28:0/75466978 conn(0x7f76d8072ce0 msgr2=0x7f76d80ac340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:03.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.540+0000 7f76d7e00640 1 -- 172.21.15.28:0/75466978 shutdown_connections 2024-02-21T23:05:03.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.540+0000 7f76d7e00640 1 -- 172.21.15.28:0/75466978 wait complete. 2024-02-21T23:05:03.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.541+0000 7f76d7e00640 1 Processor -- start 2024-02-21T23:05:03.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.542+0000 7f76d7e00640 1 -- start start 2024-02-21T23:05:03.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.542+0000 7f76d7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76bc004800 0x7f76d8156940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:03.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.542+0000 7f76d7e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f76d814d490 0x7f76d81ee840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:03.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.542+0000 7f76d7e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76d80ab6a0 0x7f76d8157560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:03.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.543+0000 7f76d7e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76d8152a30 con 0x7f76bc004800 2024-02-21T23:05:03.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.543+0000 7f76d7e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76d80bdfb0 con 0x7f76d80ab6a0 2024-02-21T23:05:03.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.543+0000 7f76d7e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76d80c7870 con 0x7f76d814d490 2024-02-21T23:05:03.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.543+0000 7f76d6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76bc004800 0x7f76d8156940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:03.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.543+0000 7f76d6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76bc004800 0x7f76d8156940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45732/0 (socket says 172.21.15.28:45732) 2024-02-21T23:05:03.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.543+0000 7f76d6a00640 1 -- 172.21.15.28:0/1011830227 learned_addr learned my addr 172.21.15.28:0/1011830227 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:03.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.543+0000 7f76d6a00640 1 -- 172.21.15.28:0/1011830227 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f76d814d490 msgr2=0x7f76d81ee840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:03.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.543+0000 7f76d6a00640 1 --2- 172.21.15.28:0/1011830227 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f76d814d490 0x7f76d81ee840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:03.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.543+0000 7f76d6a00640 1 -- 172.21.15.28:0/1011830227 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76d80ab6a0 msgr2=0x7f76d8157560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:03.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.543+0000 7f76d6a00640 1 --2- 172.21.15.28:0/1011830227 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76d80ab6a0 0x7f76d8157560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:03.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.543+0000 7f76d6a00640 1 -- 172.21.15.28:0/1011830227 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76d81491b0 con 0x7f76bc004800 2024-02-21T23:05:03.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.544+0000 7f76d6a00640 1 --2- 172.21.15.28:0/1011830227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76bc004800 0x7f76d8156940 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f76c80118e0 tx=0x7f76c8011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:03.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.544+0000 7f76cea00640 1 -- 172.21.15.28:0/1011830227 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c800fa70 con 0x7f76bc004800 2024-02-21T23:05:03.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.544+0000 7f76cea00640 1 -- 172.21.15.28:0/1011830227 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76c800fc10 con 0x7f76bc004800 2024-02-21T23:05:03.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.544+0000 7f76cea00640 1 -- 172.21.15.28:0/1011830227 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c8014440 con 0x7f76bc004800 2024-02-21T23:05:03.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.544+0000 7f76d7e00640 1 -- 172.21.15.28:0/1011830227 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76d8149490 con 0x7f76bc004800 2024-02-21T23:05:03.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.544+0000 7f76d7e00640 1 -- 172.21.15.28:0/1011830227 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76d814cc00 con 0x7f76bc004800 2024-02-21T23:05:03.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.545+0000 7f76d7e00640 1 -- 172.21.15.28:0/1011830227 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76d804f290 con 0x7f76bc004800 2024-02-21T23:05:03.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.546+0000 7f76cea00640 1 -- 172.21.15.28:0/1011830227 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f76c8012040 con 0x7f76bc004800 2024-02-21T23:05:03.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.548+0000 7f76cea00640 1 --2- 172.21.15.28:0/1011830227 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f76b806b4d0 0x7f76b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:03.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.548+0000 7f76cea00640 1 -- 172.21.15.28:0/1011830227 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f76c808aa50 con 0x7f76bc004800 2024-02-21T23:05:03.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.548+0000 7f76d6000640 1 --2- 172.21.15.28:0/1011830227 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f76b806b4d0 0x7f76b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:03.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.549+0000 7f76d6000640 1 --2- 172.21.15.28:0/1011830227 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f76b806b4d0 0x7f76b806d990 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f76c4034000 tx=0x7f76c4039040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:03.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.552+0000 7f76cea00640 1 -- 172.21.15.28:0/1011830227 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f76c8019030 con 0x7f76bc004800 2024-02-21T23:05:03.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.751+0000 7f76d7e00640 1 -- 172.21.15.28:0/1011830227 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f76d8152bb0 con 0x7f76bc004800 2024-02-21T23:05:03.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.752+0000 7f76cea00640 1 -- 172.21.15.28:0/1011830227 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v53) v1 ==== 93+0+10 (secure 0 0 0) 0x7f76c8059730 con 0x7f76bc004800 2024-02-21T23:05:03.741 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-02-21T23:05:03.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.754+0000 7f76d7e00640 1 -- 172.21.15.28:0/1011830227 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f76b806b4d0 msgr2=0x7f76b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:03.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.754+0000 7f76d7e00640 1 --2- 172.21.15.28:0/1011830227 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f76b806b4d0 0x7f76b806d990 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f76c4034000 tx=0x7f76c4039040 comp rx=0 tx=0).stop 2024-02-21T23:05:03.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.754+0000 7f76d7e00640 1 -- 172.21.15.28:0/1011830227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76bc004800 msgr2=0x7f76d8156940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:03.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.754+0000 7f76d7e00640 1 --2- 172.21.15.28:0/1011830227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76bc004800 0x7f76d8156940 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f76c80118e0 tx=0x7f76c8011da0 comp rx=0 tx=0).stop 2024-02-21T23:05:03.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.754+0000 7f76d7e00640 1 -- 172.21.15.28:0/1011830227 shutdown_connections 2024-02-21T23:05:03.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.754+0000 7f76d7e00640 1 --2- 172.21.15.28:0/1011830227 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f76b806b4d0 0x7f76b806d990 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:03.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.754+0000 7f76d7e00640 1 --2- 172.21.15.28:0/1011830227 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f76d814d490 0x7f76d81ee840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:03.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.754+0000 7f76d7e00640 1 --2- 172.21.15.28:0/1011830227 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f76d80ab6a0 0x7f76d8157560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:03.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.755+0000 7f76d7e00640 1 --2- 172.21.15.28:0/1011830227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76bc004800 0x7f76d8156940 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:03.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.755+0000 7f76d7e00640 1 -- 172.21.15.28:0/1011830227 >> 172.21.15.28:0/1011830227 conn(0x7f76d8072ce0 msgr2=0x7f76d80c9330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:03.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.755+0000 7f76d7e00640 1 -- 172.21.15.28:0/1011830227 shutdown_connections 2024-02-21T23:05:03.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.755+0000 7f76d7e00640 1 -- 172.21.15.28:0/1011830227 wait complete. 2024-02-21T23:05:03.765 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-21T23:05:03.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.998+0000 7f2a13c00640 1 Processor -- start 2024-02-21T23:05:03.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:03.998+0000 7f2a13c00640 1 -- start start 2024-02-21T23:05:03.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.001+0000 7f2a13c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2a0c06f9d0 0x7f2a0c0c04e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:03.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.001+0000 7f2a13c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f2a0c0c0a20 0x7f2a0c15f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:03.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.001+0000 7f2a13c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a0c0a8550 con 0x7f2a0c152630 2024-02-21T23:05:03.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.002+0000 7f2a13c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a0c152a10 con 0x7f2a0c06f9d0 2024-02-21T23:05:03.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.002+0000 7f2a13c00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a0c155ec0 con 0x7f2a0c0c0a20 2024-02-21T23:05:03.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.002+0000 7f2a11200640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f2a0c152630 0x7f2a0c066f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34524/0 (socket says 172.21.15.28:34524) 2024-02-21T23:05:03.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.002+0000 7f2a11200640 1 -- 172.21.15.28:0/1097512353 learned_addr learned my addr 172.21.15.28:0/1097512353 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:05:03.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.002+0000 7f2a0be00640 1 --2- 172.21.15.28:0/1097512353 >> v2:172.21.15.28:3300/0 conn(0x7f2a0c06f9d0 0x7f2a0c0c04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:03.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.002+0000 7f2a11c00640 1 --2- 172.21.15.28:0/1097512353 >> v2:172.21.15.81:3300/0 conn(0x7f2a0c0c0a20 0x7f2a0c15f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:03.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.002+0000 7f2a0be00640 1 -- 172.21.15.28:0/1097512353 >> v2:172.21.15.81:3300/0 conn(0x7f2a0c0c0a20 msgr2=0x7f2a0c15f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:03.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.002+0000 7f2a0be00640 1 --2- 172.21.15.28:0/1097512353 >> v2:172.21.15.81:3300/0 conn(0x7f2a0c0c0a20 0x7f2a0c15f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:03.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.002+0000 7f2a0be00640 1 -- 172.21.15.28:0/1097512353 >> v1:172.21.15.28:6789/0 conn(0x7f2a0c152630 legacy=0x7f2a0c066f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:03.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.002+0000 7f2a0be00640 1 -- 172.21.15.28:0/1097512353 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a0c06c2a0 con 0x7f2a0c06f9d0 2024-02-21T23:05:03.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.002+0000 7f2a0b400640 1 -- 172.21.15.28:0/1097512353 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3824577853 0 0) 0x7f2a0c0a8550 con 0x7f2a0c152630 2024-02-21T23:05:03.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.004+0000 7f2a0be00640 1 --2- 172.21.15.28:0/1097512353 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a0c06f9d0 0x7f2a0c0c04e0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f2a0c0c7040 tx=0x7f29f802b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f2adfbf92e45f89 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:03.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.004+0000 7f2a0b400640 1 -- 172.21.15.28:0/1097512353 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f8038070 con 0x7f2a0c06f9d0 2024-02-21T23:05:03.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.004+0000 7f2a0b400640 1 -- 172.21.15.28:0/1097512353 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29f8031710 con 0x7f2a0c06f9d0 2024-02-21T23:05:03.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.004+0000 7f2a0b400640 1 -- 172.21.15.28:0/1097512353 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f8031ae0 con 0x7f2a0c06f9d0 2024-02-21T23:05:03.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.004+0000 7f2a13c00640 1 -- 172.21.15.28:0/1097512353 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a0c06f9d0 msgr2=0x7f2a0c0c04e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:03.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.004+0000 7f2a13c00640 1 --2- 172.21.15.28:0/1097512353 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a0c06f9d0 0x7f2a0c0c04e0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f2a0c0c7040 tx=0x7f29f802b7b0 comp rx=0 tx=0).stop 2024-02-21T23:05:03.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.005+0000 7f2a13c00640 1 -- 172.21.15.28:0/1097512353 shutdown_connections 2024-02-21T23:05:03.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.005+0000 7f2a13c00640 1 --2- 172.21.15.28:0/1097512353 >> v2:172.21.15.81:3300/0 conn(0x7f2a0c0c0a20 0x7f2a0c15f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:03.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.005+0000 7f2a13c00640 1 --2- 172.21.15.28:0/1097512353 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a0c06f9d0 0x7f2a0c0c04e0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:03.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.005+0000 7f2a13c00640 1 -- 172.21.15.28:0/1097512353 >> 172.21.15.28:0/1097512353 conn(0x7f2a0c06ea40 msgr2=0x7f2a0c06ee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:03.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.005+0000 7f2a13c00640 1 -- 172.21.15.28:0/1097512353 shutdown_connections 2024-02-21T23:05:03.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.005+0000 7f2a13c00640 1 -- 172.21.15.28:0/1097512353 wait complete. 2024-02-21T23:05:03.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.006+0000 7f2a13c00640 1 Processor -- start 2024-02-21T23:05:03.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.007+0000 7f2a13c00640 1 -- start start 2024-02-21T23:05:03.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.007+0000 7f2a13c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2a0c06f9d0 0x7f2a0c146c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:03.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.007+0000 7f2a13c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a0c0c0a20 0x7f2a0c147170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:03.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.007+0000 7f2a13c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2a0c152630 0x7f2a0c1edff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:03.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.007+0000 7f2a13c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0c073110 con 0x7f2a0c0c0a20 2024-02-21T23:05:03.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.007+0000 7f2a13c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0c155980 con 0x7f2a0c152630 2024-02-21T23:05:03.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.007+0000 7f2a13c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0c0a8a30 con 0x7f2a0c06f9d0 2024-02-21T23:05:03.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.008+0000 7f2a0be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a0c0c0a20 0x7f2a0c147170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:03.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.008+0000 7f2a0be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a0c0c0a20 0x7f2a0c147170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45754/0 (socket says 172.21.15.28:45754) 2024-02-21T23:05:03.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.008+0000 7f2a0be00640 1 -- 172.21.15.28:0/1123390644 learned_addr learned my addr 172.21.15.28:0/1123390644 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:03.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.008+0000 7f2a11200640 1 --2- 172.21.15.28:0/1123390644 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2a0c06f9d0 0x7f2a0c146c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:03.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.008+0000 7f2a11c00640 1 --2- 172.21.15.28:0/1123390644 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2a0c152630 0x7f2a0c1edff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:03.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.008+0000 7f2a0be00640 1 -- 172.21.15.28:0/1123390644 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2a0c06f9d0 msgr2=0x7f2a0c146c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:03.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.008+0000 7f2a0be00640 1 --2- 172.21.15.28:0/1123390644 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2a0c06f9d0 0x7f2a0c146c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:03.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.008+0000 7f2a0be00640 1 -- 172.21.15.28:0/1123390644 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2a0c152630 msgr2=0x7f2a0c1edff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:03.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.008+0000 7f2a0be00640 1 --2- 172.21.15.28:0/1123390644 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2a0c152630 0x7f2a0c1edff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:03.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.008+0000 7f2a0be00640 1 -- 172.21.15.28:0/1123390644 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a0c1476b0 con 0x7f2a0c0c0a20 2024-02-21T23:05:03.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.008+0000 7f2a11200640 1 --2- 172.21.15.28:0/1123390644 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2a0c06f9d0 0x7f2a0c146c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:05:03.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.008+0000 7f2a11c00640 1 --2- 172.21.15.28:0/1123390644 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2a0c152630 0x7f2a0c1edff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:05:03.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.009+0000 7f2a0be00640 1 --2- 172.21.15.28:0/1123390644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a0c0c0a20 0x7f2a0c147170 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f29f8000c00 tx=0x7f29f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:04.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.009+0000 7f2a08c00640 1 -- 172.21.15.28:0/1123390644 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f8042070 con 0x7f2a0c0c0a20 2024-02-21T23:05:04.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.009+0000 7f2a08c00640 1 -- 172.21.15.28:0/1123390644 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29f8033a40 con 0x7f2a0c0c0a20 2024-02-21T23:05:04.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.009+0000 7f2a13c00640 1 -- 172.21.15.28:0/1123390644 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a0c0c21c0 con 0x7f2a0c0c0a20 2024-02-21T23:05:04.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.009+0000 7f2a08c00640 1 -- 172.21.15.28:0/1123390644 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f8038040 con 0x7f2a0c0c0a20 2024-02-21T23:05:04.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.009+0000 7f2a13c00640 1 -- 172.21.15.28:0/1123390644 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a0c0c3050 con 0x7f2a0c0c0a20 2024-02-21T23:05:04.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.010+0000 7f2a13c00640 1 -- 172.21.15.28:0/1123390644 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a0c0c2e40 con 0x7f2a0c0c0a20 2024-02-21T23:05:04.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.012+0000 7f2a08c00640 1 -- 172.21.15.28:0/1123390644 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f29f803b070 con 0x7f2a0c0c0a20 2024-02-21T23:05:04.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.013+0000 7f2a08c00640 1 --2- 172.21.15.28:0/1123390644 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f29ec06b4d0 0x7f29ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:04.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.013+0000 7f2a08c00640 1 -- 172.21.15.28:0/1123390644 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f29f80ab990 con 0x7f2a0c0c0a20 2024-02-21T23:05:04.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.014+0000 7f2a11200640 1 --2- 172.21.15.28:0/1123390644 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f29ec06b4d0 0x7f29ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:04.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.015+0000 7f2a11200640 1 --2- 172.21.15.28:0/1123390644 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f29ec06b4d0 0x7f29ec06d990 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f2a00000f60 tx=0x7f2a00009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:04.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.018+0000 7f2a08c00640 1 -- 172.21.15.28:0/1123390644 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f29f804b070 con 0x7f2a0c0c0a20 2024-02-21T23:05:04.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.216+0000 7f2a13c00640 1 -- 172.21.15.28:0/1123390644 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2a0c067f30 con 0x7f2a0c0c0a20 2024-02-21T23:05:04.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.217+0000 7f2a08c00640 1 -- 172.21.15.28:0/1123390644 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v53) v1 ==== 104+0+11 (secure 0 0 0) 0x7f29f8045360 con 0x7f2a0c0c0a20 2024-02-21T23:05:04.206 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-02-21T23:05:04.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.219+0000 7f2a13c00640 1 -- 172.21.15.28:0/1123390644 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f29ec06b4d0 msgr2=0x7f29ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:04.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.219+0000 7f2a13c00640 1 --2- 172.21.15.28:0/1123390644 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f29ec06b4d0 0x7f29ec06d990 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f2a00000f60 tx=0x7f2a00009040 comp rx=0 tx=0).stop 2024-02-21T23:05:04.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.219+0000 7f2a13c00640 1 -- 172.21.15.28:0/1123390644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a0c0c0a20 msgr2=0x7f2a0c147170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:04.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.219+0000 7f2a13c00640 1 --2- 172.21.15.28:0/1123390644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a0c0c0a20 0x7f2a0c147170 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f29f8000c00 tx=0x7f29f80046c0 comp rx=0 tx=0).stop 2024-02-21T23:05:04.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.219+0000 7f2a13c00640 1 -- 172.21.15.28:0/1123390644 shutdown_connections 2024-02-21T23:05:04.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.219+0000 7f2a13c00640 1 --2- 172.21.15.28:0/1123390644 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f29ec06b4d0 0x7f29ec06d990 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:04.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.219+0000 7f2a13c00640 1 --2- 172.21.15.28:0/1123390644 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2a0c152630 0x7f2a0c1edff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:04.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.219+0000 7f2a13c00640 1 --2- 172.21.15.28:0/1123390644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a0c0c0a20 0x7f2a0c147170 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:04.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.219+0000 7f2a13c00640 1 --2- 172.21.15.28:0/1123390644 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2a0c06f9d0 0x7f2a0c146c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:04.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.219+0000 7f2a13c00640 1 -- 172.21.15.28:0/1123390644 >> 172.21.15.28:0/1123390644 conn(0x7f2a0c06ea40 msgr2=0x7f2a0c14bdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:04.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.219+0000 7f2a13c00640 1 -- 172.21.15.28:0/1123390644 shutdown_connections 2024-02-21T23:05:04.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.219+0000 7f2a13c00640 1 -- 172.21.15.28:0/1123390644 wait complete. 2024-02-21T23:05:04.228 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-21T23:05:04.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.464+0000 7f0dab200640 1 Processor -- start 2024-02-21T23:05:04.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.465+0000 7f0dab200640 1 -- start start 2024-02-21T23:05:04.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.467+0000 7f0dab200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0dac152650 0x7f0dac0c8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:04.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.467+0000 7f0dab200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0dac1550e0 0x7f0dac155550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:04.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.468+0000 7f0dab200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dac158920 con 0x7f0dac06daa0 2024-02-21T23:05:04.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.468+0000 7f0dab200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0dac1586d0 con 0x7f0dac1550e0 2024-02-21T23:05:04.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.468+0000 7f0dab200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dac158c00 con 0x7f0dac152650 2024-02-21T23:05:04.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.468+0000 7f0daa800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0dac1550e0 0x7f0dac155550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:04.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.468+0000 7f0da9400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f0dac06daa0 0x7f0dac0c7160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34528/0 (socket says 172.21.15.28:34528) 2024-02-21T23:05:04.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.468+0000 7f0da9400640 1 -- 172.21.15.28:0/3149602555 learned_addr learned my addr 172.21.15.28:0/3149602555 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:05:04.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.468+0000 7f0da9e00640 1 --2- 172.21.15.28:0/3149602555 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0dac152650 0x7f0dac0c8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:04.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.469+0000 7f0daa800640 1 -- 172.21.15.28:0/3149602555 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0dac152650 msgr2=0x7f0dac0c8ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:04.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.469+0000 7f0daa800640 1 --2- 172.21.15.28:0/3149602555 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0dac152650 0x7f0dac0c8ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:04.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.469+0000 7f0daa800640 1 -- 172.21.15.28:0/3149602555 >> v1:172.21.15.28:6789/0 conn(0x7f0dac06daa0 legacy=0x7f0dac0c7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:04.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.469+0000 7f0daa800640 1 -- 172.21.15.28:0/3149602555 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dac158ee0 con 0x7f0dac1550e0 2024-02-21T23:05:04.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.469+0000 7f0da9e00640 1 --2- 172.21.15.28:0/3149602555 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0dac152650 0x7f0dac0c8ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:05:04.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.470+0000 7f0daa800640 1 --2- 172.21.15.28:0/3149602555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0dac1550e0 0x7f0dac155550 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f0da4009d90 tx=0x7f0da402be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ed54f02d1192a36 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:04.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.470+0000 7f0da8a00640 1 -- 172.21.15.28:0/3149602555 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0da4039070 con 0x7f0dac1550e0 2024-02-21T23:05:04.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.470+0000 7f0da8a00640 1 -- 172.21.15.28:0/3149602555 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0da402cbd0 con 0x7f0dac1550e0 2024-02-21T23:05:04.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.470+0000 7f0da8a00640 1 -- 172.21.15.28:0/3149602555 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0da4034440 con 0x7f0dac1550e0 2024-02-21T23:05:04.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.471+0000 7f0dab200640 1 -- 172.21.15.28:0/3149602555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0dac1550e0 msgr2=0x7f0dac155550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:04.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.471+0000 7f0dab200640 1 --2- 172.21.15.28:0/3149602555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0dac1550e0 0x7f0dac155550 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f0da4009d90 tx=0x7f0da402be20 comp rx=0 tx=0).stop 2024-02-21T23:05:04.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.471+0000 7f0dab200640 1 -- 172.21.15.28:0/3149602555 shutdown_connections 2024-02-21T23:05:04.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.471+0000 7f0dab200640 1 --2- 172.21.15.28:0/3149602555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0dac1550e0 0x7f0dac155550 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:04.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.471+0000 7f0dab200640 1 --2- 172.21.15.28:0/3149602555 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0dac152650 0x7f0dac0c8ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:04.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.471+0000 7f0dab200640 1 -- 172.21.15.28:0/3149602555 >> 172.21.15.28:0/3149602555 conn(0x7f0dac072ce0 msgr2=0x7f0dac0ac340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:04.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.471+0000 7f0dab200640 1 -- 172.21.15.28:0/3149602555 shutdown_connections 2024-02-21T23:05:04.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.471+0000 7f0dab200640 1 -- 172.21.15.28:0/3149602555 wait complete. 2024-02-21T23:05:04.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.474+0000 7f0dab200640 1 Processor -- start 2024-02-21T23:05:04.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.474+0000 7f0dab200640 1 -- start start 2024-02-21T23:05:04.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.476+0000 7f0dab200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0dac06daa0 0x7f0dac151750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:04.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.476+0000 7f0dab200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0dac152650 0x7f0dac157b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:04.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.476+0000 7f0dab200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0dac1550e0 0x7f0dac1580c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:04.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.476+0000 7f0dab200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dac152a30 con 0x7f0dac1550e0 2024-02-21T23:05:04.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.476+0000 7f0dab200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dac0bdfb0 con 0x7f0dac152650 2024-02-21T23:05:04.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.476+0000 7f0dab200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dac0c7870 con 0x7f0dac06daa0 2024-02-21T23:05:04.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.476+0000 7f0daa800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0dac1550e0 0x7f0dac1580c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:04.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.476+0000 7f0daa800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0dac1550e0 0x7f0dac1580c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45760/0 (socket says 172.21.15.28:45760) 2024-02-21T23:05:04.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.476+0000 7f0daa800640 1 -- 172.21.15.28:0/1551493996 learned_addr learned my addr 172.21.15.28:0/1551493996 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:04.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.476+0000 7f0daa800640 1 -- 172.21.15.28:0/1551493996 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0dac06daa0 msgr2=0x7f0dac151750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:04.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.476+0000 7f0daa800640 1 --2- 172.21.15.28:0/1551493996 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0dac06daa0 0x7f0dac151750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:04.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.476+0000 7f0daa800640 1 -- 172.21.15.28:0/1551493996 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0dac152650 msgr2=0x7f0dac157b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:04.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.476+0000 7f0daa800640 1 --2- 172.21.15.28:0/1551493996 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0dac152650 0x7f0dac157b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:04.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.476+0000 7f0daa800640 1 -- 172.21.15.28:0/1551493996 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dac14eac0 con 0x7f0dac1550e0 2024-02-21T23:05:04.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.477+0000 7f0daa800640 1 --2- 172.21.15.28:0/1551493996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0dac1550e0 0x7f0dac1580c0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f0da4009d90 tx=0x7f0da4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:04.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.477+0000 7f0da2000640 1 -- 172.21.15.28:0/1551493996 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0da4039050 con 0x7f0dac1550e0 2024-02-21T23:05:04.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.477+0000 7f0da2000640 1 -- 172.21.15.28:0/1551493996 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0da4045070 con 0x7f0dac1550e0 2024-02-21T23:05:04.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.477+0000 7f0dab200640 1 -- 172.21.15.28:0/1551493996 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dac14ed60 con 0x7f0dac1550e0 2024-02-21T23:05:04.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.477+0000 7f0da2000640 1 -- 172.21.15.28:0/1551493996 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0da403e830 con 0x7f0dac1550e0 2024-02-21T23:05:04.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.477+0000 7f0dab200640 1 -- 172.21.15.28:0/1551493996 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dac14f320 con 0x7f0dac1550e0 2024-02-21T23:05:04.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.478+0000 7f0dab200640 1 -- 172.21.15.28:0/1551493996 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dac04f290 con 0x7f0dac1550e0 2024-02-21T23:05:04.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.480+0000 7f0da2000640 1 -- 172.21.15.28:0/1551493996 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0da403e9d0 con 0x7f0dac1550e0 2024-02-21T23:05:04.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.480+0000 7f0da2000640 1 --2- 172.21.15.28:0/1551493996 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0d8c06b500 0x7f0d8c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:04.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.481+0000 7f0da2000640 1 -- 172.21.15.28:0/1551493996 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f0da40ad0e0 con 0x7f0dac1550e0 2024-02-21T23:05:04.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.481+0000 7f0da9e00640 1 --2- 172.21.15.28:0/1551493996 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0d8c06b500 0x7f0d8c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:04.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.482+0000 7f0da9e00640 1 --2- 172.21.15.28:0/1551493996 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0d8c06b500 0x7f0d8c06d9c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f0d9c006fd0 tx=0x7f0d9c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:04.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.490+0000 7f0da2000640 1 -- 172.21.15.28:0/1551493996 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f0da4032e20 con 0x7f0dac1550e0 2024-02-21T23:05:04.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.691+0000 7f0dab200640 1 -- 172.21.15.28:0/1551493996 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0dac156080 con 0x7f0dac1550e0 2024-02-21T23:05:04.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.692+0000 7f0da2000640 1 -- 172.21.15.28:0/1551493996 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v53) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0da407bdd0 con 0x7f0dac1550e0 2024-02-21T23:05:04.681 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-02-21T23:05:04.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.695+0000 7f0dab200640 1 -- 172.21.15.28:0/1551493996 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0d8c06b500 msgr2=0x7f0d8c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:04.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.695+0000 7f0dab200640 1 --2- 172.21.15.28:0/1551493996 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0d8c06b500 0x7f0d8c06d9c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f0d9c006fd0 tx=0x7f0d9c008040 comp rx=0 tx=0).stop 2024-02-21T23:05:04.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.695+0000 7f0dab200640 1 -- 172.21.15.28:0/1551493996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0dac1550e0 msgr2=0x7f0dac1580c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:04.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.695+0000 7f0dab200640 1 --2- 172.21.15.28:0/1551493996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0dac1550e0 0x7f0dac1580c0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f0da4009d90 tx=0x7f0da4004a40 comp rx=0 tx=0).stop 2024-02-21T23:05:04.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.695+0000 7f0dab200640 1 -- 172.21.15.28:0/1551493996 shutdown_connections 2024-02-21T23:05:04.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.695+0000 7f0dab200640 1 --2- 172.21.15.28:0/1551493996 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0d8c06b500 0x7f0d8c06d9c0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:04.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.695+0000 7f0dab200640 1 --2- 172.21.15.28:0/1551493996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0dac1550e0 0x7f0dac1580c0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:04.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.695+0000 7f0dab200640 1 --2- 172.21.15.28:0/1551493996 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0dac152650 0x7f0dac157b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:04.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.695+0000 7f0dab200640 1 --2- 172.21.15.28:0/1551493996 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0dac06daa0 0x7f0dac151750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:04.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.695+0000 7f0dab200640 1 -- 172.21.15.28:0/1551493996 >> 172.21.15.28:0/1551493996 conn(0x7f0dac072ce0 msgr2=0x7f0dac152f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:04.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.695+0000 7f0dab200640 1 -- 172.21.15.28:0/1551493996 shutdown_connections 2024-02-21T23:05:04.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.695+0000 7f0dab200640 1 -- 172.21.15.28:0/1551493996 wait complete. 2024-02-21T23:05:04.707 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-02-21T23:05:04.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.931+0000 7fb7e9800640 1 Processor -- start 2024-02-21T23:05:04.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.931+0000 7fb7e9800640 1 -- start start 2024-02-21T23:05:04.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.933+0000 7fb7e9800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb7e40b0130 0x7fb7e40c8740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:04.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.933+0000 7fb7e9800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb7e40c51c0 0x7fb7e40c5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:04.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.933+0000 7fb7e9800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7e41550b0 con 0x7fb7e4158510 2024-02-21T23:05:04.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.933+0000 7fb7e9800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7e4152a40 con 0x7fb7e40c51c0 2024-02-21T23:05:04.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.933+0000 7fb7e9800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e4155350 con 0x7fb7e40b0130 2024-02-21T23:05:04.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.933+0000 7fb7db400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb7e40c51c0 0x7fb7e40c5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:04.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.933+0000 7fb7db400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb7e40c51c0 0x7fb7e40c5640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36040/0 (socket says 172.21.15.28:36040) 2024-02-21T23:05:04.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.933+0000 7fb7db400640 1 -- 172.21.15.28:0/2454707131 learned_addr learned my addr 172.21.15.28:0/2454707131 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:04.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.933+0000 7fb7e3e00640 1 --2- 172.21.15.28:0/2454707131 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb7e40b0130 0x7fb7e40c8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:04.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.934+0000 7fb7db400640 1 -- 172.21.15.28:0/2454707131 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb7e40b0130 msgr2=0x7fb7e40c8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:04.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.934+0000 7fb7db400640 1 --2- 172.21.15.28:0/2454707131 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb7e40b0130 0x7fb7e40c8740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:04.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.934+0000 7fb7db400640 1 -- 172.21.15.28:0/2454707131 >> v1:172.21.15.81:6789/0 conn(0x7fb7e4158510 legacy=0x7fb7e4158960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:04.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.934+0000 7fb7db400640 1 -- 172.21.15.28:0/2454707131 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7e4155630 con 0x7fb7e40c51c0 2024-02-21T23:05:04.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.935+0000 7fb7db400640 1 --2- 172.21.15.28:0/2454707131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7e40c51c0 0x7fb7e40c5640 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc009870 tx=0x7fb7cc02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=195af5565fd68a2a server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:04.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.935+0000 7fb7e3400640 1 -- 172.21.15.28:0/2454707131 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7cc0317c0 con 0x7fb7e40c51c0 2024-02-21T23:05:04.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.935+0000 7fb7e3400640 1 -- 172.21.15.28:0/2454707131 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7cc037070 con 0x7fb7e40c51c0 2024-02-21T23:05:04.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.935+0000 7fb7e3400640 1 -- 172.21.15.28:0/2454707131 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7cc031b80 con 0x7fb7e40c51c0 2024-02-21T23:05:04.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.936+0000 7fb7e9800640 1 -- 172.21.15.28:0/2454707131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7e40c51c0 msgr2=0x7fb7e40c5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:04.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.936+0000 7fb7e9800640 1 --2- 172.21.15.28:0/2454707131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7e40c51c0 0x7fb7e40c5640 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc009870 tx=0x7fb7cc02b860 comp rx=0 tx=0).stop 2024-02-21T23:05:04.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.936+0000 7fb7e9800640 1 -- 172.21.15.28:0/2454707131 shutdown_connections 2024-02-21T23:05:04.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.936+0000 7fb7e9800640 1 --2- 172.21.15.28:0/2454707131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7e40c51c0 0x7fb7e40c5640 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:04.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.936+0000 7fb7e9800640 1 --2- 172.21.15.28:0/2454707131 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb7e40b0130 0x7fb7e40c8740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:04.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.936+0000 7fb7e9800640 1 -- 172.21.15.28:0/2454707131 >> 172.21.15.28:0/2454707131 conn(0x7fb7e40ac370 msgr2=0x7fb7e40ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:04.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.936+0000 7fb7e9800640 1 -- 172.21.15.28:0/2454707131 shutdown_connections 2024-02-21T23:05:04.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.936+0000 7fb7e9800640 1 -- 172.21.15.28:0/2454707131 wait complete. 2024-02-21T23:05:04.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.938+0000 7fb7e9800640 1 Processor -- start 2024-02-21T23:05:04.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.938+0000 7fb7e9800640 1 -- start start 2024-02-21T23:05:04.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.939+0000 7fb7e9800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb7e40b0130 0x7fb7e40c3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:04.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.939+0000 7fb7e9800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7e40c51c0 0x7fb7e40c4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:04.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.939+0000 7fb7e9800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7e4158510 0x7fb7e40ad920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:04.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.939+0000 7fb7e9800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e40c8dd0 con 0x7fb7e4158510 2024-02-21T23:05:04.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.939+0000 7fb7e9800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e4155e30 con 0x7fb7e40c51c0 2024-02-21T23:05:04.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.939+0000 7fb7e9800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e40be060 con 0x7fb7e40b0130 2024-02-21T23:05:04.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.940+0000 7fb7e8e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7e4158510 0x7fb7e40ad920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:04.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.940+0000 7fb7e8e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7e4158510 0x7fb7e40ad920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36054/0 (socket says 172.21.15.28:36054) 2024-02-21T23:05:04.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.940+0000 7fb7e8e00640 1 -- 172.21.15.28:0/75647540 learned_addr learned my addr 172.21.15.28:0/75647540 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:04.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.940+0000 7fb7e3e00640 1 --2- 172.21.15.28:0/75647540 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb7e40b0130 0x7fb7e40c3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:04.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.940+0000 7fb7db400640 1 --2- 172.21.15.28:0/75647540 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7e40c51c0 0x7fb7e40c4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:04.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.940+0000 7fb7e8e00640 1 -- 172.21.15.28:0/75647540 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb7e40b0130 msgr2=0x7fb7e40c3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:04.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.940+0000 7fb7e8e00640 1 --2- 172.21.15.28:0/75647540 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb7e40b0130 0x7fb7e40c3ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:04.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.940+0000 7fb7e8e00640 1 -- 172.21.15.28:0/75647540 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7e40c51c0 msgr2=0x7fb7e40c4430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:04.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.940+0000 7fb7e8e00640 1 --2- 172.21.15.28:0/75647540 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7e40c51c0 0x7fb7e40c4430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:04.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.940+0000 7fb7e8e00640 1 -- 172.21.15.28:0/75647540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7e40c4970 con 0x7fb7e4158510 2024-02-21T23:05:04.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.940+0000 7fb7e3e00640 1 --2- 172.21.15.28:0/75647540 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb7e40b0130 0x7fb7e40c3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:05:04.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.941+0000 7fb7e8e00640 1 --2- 172.21.15.28:0/75647540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7e4158510 0x7fb7e40ad920 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc00b9a0 tx=0x7fb7dc00be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:04.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.941+0000 7fb7e0c00640 1 -- 172.21.15.28:0/75647540 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7dc019070 con 0x7fb7e4158510 2024-02-21T23:05:04.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.941+0000 7fb7e0c00640 1 -- 172.21.15.28:0/75647540 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7dc00cba0 con 0x7fb7e4158510 2024-02-21T23:05:04.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.941+0000 7fb7e0c00640 1 -- 172.21.15.28:0/75647540 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7dc014440 con 0x7fb7e4158510 2024-02-21T23:05:04.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.941+0000 7fb7e9800640 1 -- 172.21.15.28:0/75647540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7e4153a30 con 0x7fb7e4158510 2024-02-21T23:05:04.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.941+0000 7fb7e9800640 1 -- 172.21.15.28:0/75647540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7e4153ff0 con 0x7fb7e4158510 2024-02-21T23:05:04.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.942+0000 7fb7e9800640 1 -- 172.21.15.28:0/75647540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7e4154320 con 0x7fb7e4158510 2024-02-21T23:05:04.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.944+0000 7fb7e0c00640 1 -- 172.21.15.28:0/75647540 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb7dc01c070 con 0x7fb7e4158510 2024-02-21T23:05:04.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.945+0000 7fb7e0c00640 1 --2- 172.21.15.28:0/75647540 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb7c006b500 0x7fb7c006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:04.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.945+0000 7fb7e0c00640 1 -- 172.21.15.28:0/75647540 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb7dc089f10 con 0x7fb7e4158510 2024-02-21T23:05:04.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.945+0000 7fb7e3e00640 1 --2- 172.21.15.28:0/75647540 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb7c006b500 0x7fb7c006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:04.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.946+0000 7fb7e3e00640 1 --2- 172.21.15.28:0/75647540 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb7c006b500 0x7fb7c006d9c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fb7d4006fd0 tx=0x7fb7d4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:04.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:04.949+0000 7fb7e0c00640 1 -- 172.21.15.28:0/75647540 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb7dc058bf0 con 0x7fb7e4158510 2024-02-21T23:05:05.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.151+0000 7fb7e9800640 1 -- 172.21.15.28:0/75647540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fb7e404f290 con 0x7fb7e4158510 2024-02-21T23:05:05.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.152+0000 7fb7e0c00640 1 -- 172.21.15.28:0/75647540 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v53) v1 ==== 84+0+207 (secure 0 0 0) 0x7fb7dc0584f0 con 0x7fb7e4158510 2024-02-21T23:05:05.142 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:05:05.142 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-02-21T23:05:05.142 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:05:05.142 INFO:teuthology.orchestra.run.smithi028.stdout: "poolnum": 1, 2024-02-21T23:05:05.142 INFO:teuthology.orchestra.run.smithi028.stdout: "poolname": ".mgr" 2024-02-21T23:05:05.142 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:05:05.142 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:05:05.142 INFO:teuthology.orchestra.run.smithi028.stdout: "poolnum": 7, 2024-02-21T23:05:05.142 INFO:teuthology.orchestra.run.smithi028.stdout: "poolname": "cephfs_metadata" 2024-02-21T23:05:05.142 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:05:05.142 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:05:05.143 INFO:teuthology.orchestra.run.smithi028.stdout: "poolnum": 8, 2024-02-21T23:05:05.143 INFO:teuthology.orchestra.run.smithi028.stdout: "poolname": "cephfs_data" 2024-02-21T23:05:05.143 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:05:05.143 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-02-21T23:05:05.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.155+0000 7fb7e9800640 1 -- 172.21.15.28:0/75647540 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb7c006b500 msgr2=0x7fb7c006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:05.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.155+0000 7fb7e9800640 1 --2- 172.21.15.28:0/75647540 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb7c006b500 0x7fb7c006d9c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fb7d4006fd0 tx=0x7fb7d4008040 comp rx=0 tx=0).stop 2024-02-21T23:05:05.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.155+0000 7fb7e9800640 1 -- 172.21.15.28:0/75647540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7e4158510 msgr2=0x7fb7e40ad920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:05.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.155+0000 7fb7e9800640 1 --2- 172.21.15.28:0/75647540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7e4158510 0x7fb7e40ad920 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc00b9a0 tx=0x7fb7dc00be60 comp rx=0 tx=0).stop 2024-02-21T23:05:05.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.155+0000 7fb7e9800640 1 -- 172.21.15.28:0/75647540 shutdown_connections 2024-02-21T23:05:05.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.155+0000 7fb7e9800640 1 --2- 172.21.15.28:0/75647540 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb7c006b500 0x7fb7c006d9c0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:05.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.155+0000 7fb7e9800640 1 --2- 172.21.15.28:0/75647540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7e4158510 0x7fb7e40ad920 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:05.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.155+0000 7fb7e9800640 1 --2- 172.21.15.28:0/75647540 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7e40c51c0 0x7fb7e40c4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:05.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.155+0000 7fb7e9800640 1 --2- 172.21.15.28:0/75647540 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb7e40b0130 0x7fb7e40c3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:05.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.155+0000 7fb7e9800640 1 -- 172.21.15.28:0/75647540 >> 172.21.15.28:0/75647540 conn(0x7fb7e40ac370 msgr2=0x7fb7e40ac750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:05.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.155+0000 7fb7e9800640 1 -- 172.21.15.28:0/75647540 shutdown_connections 2024-02-21T23:05:05.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.155+0000 7fb7e9800640 1 -- 172.21.15.28:0/75647540 wait complete. 2024-02-21T23:05:05.163 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs2' 2024-02-21T23:05:05.163 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs2_metadata --pg_num_min 64 2024-02-21T23:05:05.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.391+0000 7f3197800640 1 Processor -- start 2024-02-21T23:05:05.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.392+0000 7f3197800640 1 -- start start 2024-02-21T23:05:05.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.393+0000 7f3197800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f319006c0f0 0x7f319013a890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:05.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.393+0000 7f3197800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f319006fa00 0x7f319013add0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:05.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.393+0000 7f3197800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3190154f80 con 0x7f31900c1b70 2024-02-21T23:05:05.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.393+0000 7f3197800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3190155240 con 0x7f319006fa00 2024-02-21T23:05:05.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.393+0000 7f3197800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3190155520 con 0x7f319006c0f0 2024-02-21T23:05:05.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.394+0000 7f3194e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f319006c0f0 0x7f319013a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:05.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.394+0000 7f3194e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f319006c0f0 0x7f319013a890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3301/0 says I am v2:172.21.15.28:38324/0 (socket says 172.21.15.28:38324) 2024-02-21T23:05:05.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.394+0000 7f3194e00640 1 -- 172.21.15.28:0/781515108 learned_addr learned my addr 172.21.15.28:0/781515108 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:05.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.394+0000 7f318fe00640 1 --2- 172.21.15.28:0/781515108 >> v2:172.21.15.81:3300/0 conn(0x7f319006fa00 0x7f319013add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:05.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.395+0000 7f3194e00640 1 -- 172.21.15.28:0/781515108 >> v2:172.21.15.81:3300/0 conn(0x7f319006fa00 msgr2=0x7f319013add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:05.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.395+0000 7f3194e00640 1 --2- 172.21.15.28:0/781515108 >> v2:172.21.15.81:3300/0 conn(0x7f319006fa00 0x7f319013add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:05.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.395+0000 7f3194e00640 1 -- 172.21.15.28:0/781515108 >> v1:172.21.15.81:6789/0 conn(0x7f31900c1b70 legacy=0x7f3190158570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:05.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.395+0000 7f3194e00640 1 -- 172.21.15.28:0/781515108 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3190155800 con 0x7f319006c0f0 2024-02-21T23:05:05.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.395+0000 7f318fe00640 1 --2- 172.21.15.28:0/781515108 >> v2:172.21.15.81:3300/0 conn(0x7f319006fa00 0x7f319013add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-21T23:05:05.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.397+0000 7f3194e00640 1 --2- 172.21.15.28:0/781515108 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f319006c0f0 0x7f319013a890 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f31840098a0 tx=0x7f318402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a2ad6ea345c2182c server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:05.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.398+0000 7f318f400640 1 -- 172.21.15.28:0/781515108 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3184034070 con 0x7f319006c0f0 2024-02-21T23:05:05.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.398+0000 7f318f400640 1 -- 172.21.15.28:0/781515108 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3184030840 con 0x7f319006c0f0 2024-02-21T23:05:05.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.398+0000 7f318f400640 1 -- 172.21.15.28:0/781515108 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3184030c10 con 0x7f319006c0f0 2024-02-21T23:05:05.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.399+0000 7f3197800640 1 -- 172.21.15.28:0/781515108 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f319006c0f0 msgr2=0x7f319013a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:05.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.399+0000 7f3197800640 1 --2- 172.21.15.28:0/781515108 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f319006c0f0 0x7f319013a890 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f31840098a0 tx=0x7f318402b980 comp rx=0 tx=0).stop 2024-02-21T23:05:05.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.399+0000 7f3197800640 1 -- 172.21.15.28:0/781515108 shutdown_connections 2024-02-21T23:05:05.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.399+0000 7f3197800640 1 --2- 172.21.15.28:0/781515108 >> v2:172.21.15.81:3300/0 conn(0x7f319006fa00 0x7f319013add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:05.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.399+0000 7f3197800640 1 --2- 172.21.15.28:0/781515108 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f319006c0f0 0x7f319013a890 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:05.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.399+0000 7f3197800640 1 -- 172.21.15.28:0/781515108 >> 172.21.15.28:0/781515108 conn(0x7f3190071b10 msgr2=0x7f3190071ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:05.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.399+0000 7f3197800640 1 -- 172.21.15.28:0/781515108 shutdown_connections 2024-02-21T23:05:05.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.399+0000 7f3197800640 1 -- 172.21.15.28:0/781515108 wait complete. 2024-02-21T23:05:05.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.401+0000 7f3197800640 1 Processor -- start 2024-02-21T23:05:05.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.401+0000 7f3197800640 1 -- start start 2024-02-21T23:05:05.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.402+0000 7f3197800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f319006c0f0 0x7f319012dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:05.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.402+0000 7f3197800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3190158140 0x7f31901ee5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:05.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.402+0000 7f3197800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31901397b0 0x7f3190151990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:05.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.402+0000 7f3197800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f319006c650 con 0x7f31901397b0 2024-02-21T23:05:05.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.402+0000 7f3197800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f319006c4d0 con 0x7f3190158140 2024-02-21T23:05:05.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.402+0000 7f3197800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3190072de0 con 0x7f319006c0f0 2024-02-21T23:05:05.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.402+0000 7f3195800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31901397b0 0x7f3190151990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:05.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.402+0000 7f3195800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31901397b0 0x7f3190151990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36068/0 (socket says 172.21.15.28:36068) 2024-02-21T23:05:05.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.402+0000 7f3195800640 1 -- 172.21.15.28:0/3059247155 learned_addr learned my addr 172.21.15.28:0/3059247155 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:05.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.402+0000 7f3194e00640 1 --2- 172.21.15.28:0/3059247155 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f319006c0f0 0x7f319012dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:05.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.402+0000 7f318fe00640 1 --2- 172.21.15.28:0/3059247155 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3190158140 0x7f31901ee5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:05.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.403+0000 7f3195800640 1 -- 172.21.15.28:0/3059247155 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f319006c0f0 msgr2=0x7f319012dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:05.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.403+0000 7f3195800640 1 --2- 172.21.15.28:0/3059247155 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f319006c0f0 0x7f319012dc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:05.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.403+0000 7f3195800640 1 -- 172.21.15.28:0/3059247155 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3190158140 msgr2=0x7f31901ee5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:05.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.403+0000 7f3195800640 1 --2- 172.21.15.28:0/3059247155 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3190158140 0x7f31901ee5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:05.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.403+0000 7f3195800640 1 -- 172.21.15.28:0/3059247155 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f319012e1c0 con 0x7f31901397b0 2024-02-21T23:05:05.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.403+0000 7f3195800640 1 --2- 172.21.15.28:0/3059247155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31901397b0 0x7f3190151990 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f318800c680 tx=0x7f318800cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:05.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.403+0000 7f318cc00640 1 -- 172.21.15.28:0/3059247155 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3188003b70 con 0x7f31901397b0 2024-02-21T23:05:05.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.403+0000 7f318cc00640 1 -- 172.21.15.28:0/3059247155 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3188003d10 con 0x7f31901397b0 2024-02-21T23:05:05.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.403+0000 7f318cc00640 1 -- 172.21.15.28:0/3059247155 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3188005310 con 0x7f31901397b0 2024-02-21T23:05:05.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.403+0000 7f3197800640 1 -- 172.21.15.28:0/3059247155 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3190157960 con 0x7f31901397b0 2024-02-21T23:05:05.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.403+0000 7f3197800640 1 -- 172.21.15.28:0/3059247155 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f319012d760 con 0x7f31901397b0 2024-02-21T23:05:05.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.405+0000 7f3197800640 1 -- 172.21.15.28:0/3059247155 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3190068640 con 0x7f31901397b0 2024-02-21T23:05:05.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.406+0000 7f318cc00640 1 -- 172.21.15.28:0/3059247155 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f31880054b0 con 0x7f31901397b0 2024-02-21T23:05:05.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.407+0000 7f318cc00640 1 --2- 172.21.15.28:0/3059247155 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f317406b4d0 0x7f317406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:05.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.407+0000 7f318cc00640 1 -- 172.21.15.28:0/3059247155 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f318808a240 con 0x7f31901397b0 2024-02-21T23:05:05.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.408+0000 7f3194e00640 1 --2- 172.21.15.28:0/3059247155 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f317406b4d0 0x7f317406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:05.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.409+0000 7f3194e00640 1 --2- 172.21.15.28:0/3059247155 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f317406b4d0 0x7f317406d990 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f3184036000 tx=0x7f318403b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:05.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.412+0000 7f318cc00640 1 -- 172.21.15.28:0/3059247155 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3188058f20 con 0x7f31901397b0 2024-02-21T23:05:05.599 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-21T23:05:05.607 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-21T23:05:05.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:05.644+0000 7f3197800640 1 -- 172.21.15.28:0/3059247155 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs2_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f3190072bc0 con 0x7f31901397b0 2024-02-21T23:05:06.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.392+0000 7f318cc00640 1 -- 172.21.15.28:0/3059247155 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs2_metadata", "pg_num_min": 64}]=0 pool 'cephfs2_metadata' created v54) v1 ==== 140+0+0 (secure 0 0 0) 0x7f3188058820 con 0x7f31901397b0 2024-02-21T23:05:06.382 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs2_metadata' created 2024-02-21T23:05:06.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.397+0000 7f3197800640 1 -- 172.21.15.28:0/3059247155 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f317406b4d0 msgr2=0x7f317406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:06.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.397+0000 7f3197800640 1 --2- 172.21.15.28:0/3059247155 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f317406b4d0 0x7f317406d990 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f3184036000 tx=0x7f318403b040 comp rx=0 tx=0).stop 2024-02-21T23:05:06.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.397+0000 7f3197800640 1 -- 172.21.15.28:0/3059247155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31901397b0 msgr2=0x7f3190151990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:06.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.397+0000 7f3197800640 1 --2- 172.21.15.28:0/3059247155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31901397b0 0x7f3190151990 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f318800c680 tx=0x7f318800cb40 comp rx=0 tx=0).stop 2024-02-21T23:05:06.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.398+0000 7f3197800640 1 -- 172.21.15.28:0/3059247155 shutdown_connections 2024-02-21T23:05:06.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.398+0000 7f3197800640 1 --2- 172.21.15.28:0/3059247155 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f317406b4d0 0x7f317406d990 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:06.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.398+0000 7f3197800640 1 --2- 172.21.15.28:0/3059247155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31901397b0 0x7f3190151990 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:06.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.398+0000 7f3197800640 1 --2- 172.21.15.28:0/3059247155 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3190158140 0x7f31901ee5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:06.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.398+0000 7f3197800640 1 --2- 172.21.15.28:0/3059247155 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f319006c0f0 0x7f319012dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:06.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.398+0000 7f3197800640 1 -- 172.21.15.28:0/3059247155 >> 172.21.15.28:0/3059247155 conn(0x7f3190071b10 msgr2=0x7f3190152b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:06.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.399+0000 7f3197800640 1 -- 172.21.15.28:0/3059247155 shutdown_connections 2024-02-21T23:05:06.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.399+0000 7f3197800640 1 -- 172.21.15.28:0/3059247155 wait complete. 2024-02-21T23:05:06.416 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs2_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-02-21T23:05:06.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.632+0000 7f6f1e000640 1 Processor -- start 2024-02-21T23:05:06.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.632+0000 7f6f1e000640 1 -- start start 2024-02-21T23:05:06.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.634+0000 7f6f1e000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6f181505c0 0x7f6f18153890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:06.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.634+0000 7f6f1e000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f6f18072920 0x7f6f1806be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:06.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.634+0000 7f6f1e000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f18072180 con 0x7f6f1806c3c0 2024-02-21T23:05:06.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.634+0000 7f6f1e000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f18153dd0 con 0x7f6f181505c0 2024-02-21T23:05:06.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.634+0000 7f6f1e000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f180ac4d0 con 0x7f6f18072920 2024-02-21T23:05:06.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.635+0000 7f6f1cc00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6f181505c0 0x7f6f18153890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:06.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.635+0000 7f6f1cc00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6f181505c0 0x7f6f18153890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36084/0 (socket says 172.21.15.28:36084) 2024-02-21T23:05:06.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.635+0000 7f6f1cc00640 1 -- 172.21.15.28:0/862836405 learned_addr learned my addr 172.21.15.28:0/862836405 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:06.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.635+0000 7f6f17e00640 1 --2- 172.21.15.28:0/862836405 >> v2:172.21.15.81:3300/0 conn(0x7f6f18072920 0x7f6f1806be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:06.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.635+0000 7f6f1cc00640 1 -- 172.21.15.28:0/862836405 >> v2:172.21.15.81:3300/0 conn(0x7f6f18072920 msgr2=0x7f6f1806be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:06.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.635+0000 7f6f1cc00640 1 --2- 172.21.15.28:0/862836405 >> v2:172.21.15.81:3300/0 conn(0x7f6f18072920 0x7f6f1806be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:06.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.635+0000 7f6f1cc00640 1 -- 172.21.15.28:0/862836405 >> v1:172.21.15.28:6789/0 conn(0x7f6f1806c3c0 legacy=0x7f6f180aaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:06.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.635+0000 7f6f1cc00640 1 -- 172.21.15.28:0/862836405 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f18072460 con 0x7f6f181505c0 2024-02-21T23:05:06.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.636+0000 7f6f1cc00640 1 --2- 172.21.15.28:0/862836405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f181505c0 0x7f6f18153890 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f6f180c4940 tx=0x7f6f0c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94a00b0f5bee84fa server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:06.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.637+0000 7f6f17400640 1 -- 172.21.15.28:0/862836405 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f0c02e070 con 0x7f6f181505c0 2024-02-21T23:05:06.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.637+0000 7f6f17400640 1 -- 172.21.15.28:0/862836405 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f0c037070 con 0x7f6f181505c0 2024-02-21T23:05:06.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.637+0000 7f6f17400640 1 -- 172.21.15.28:0/862836405 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f0c031a00 con 0x7f6f181505c0 2024-02-21T23:05:06.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.637+0000 7f6f1e000640 1 -- 172.21.15.28:0/862836405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f181505c0 msgr2=0x7f6f18153890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:06.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.637+0000 7f6f1e000640 1 --2- 172.21.15.28:0/862836405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f181505c0 0x7f6f18153890 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f6f180c4940 tx=0x7f6f0c02b880 comp rx=0 tx=0).stop 2024-02-21T23:05:06.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.637+0000 7f6f1e000640 1 -- 172.21.15.28:0/862836405 shutdown_connections 2024-02-21T23:05:06.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.637+0000 7f6f1e000640 1 --2- 172.21.15.28:0/862836405 >> v2:172.21.15.81:3300/0 conn(0x7f6f18072920 0x7f6f1806be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:06.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.637+0000 7f6f1e000640 1 --2- 172.21.15.28:0/862836405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f181505c0 0x7f6f18153890 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:06.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.637+0000 7f6f1e000640 1 -- 172.21.15.28:0/862836405 >> 172.21.15.28:0/862836405 conn(0x7f6f180c4000 msgr2=0x7f6f180719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:06.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.638+0000 7f6f1e000640 1 -- 172.21.15.28:0/862836405 shutdown_connections 2024-02-21T23:05:06.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.638+0000 7f6f1e000640 1 -- 172.21.15.28:0/862836405 wait complete. 2024-02-21T23:05:06.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.640+0000 7f6f1e000640 1 Processor -- start 2024-02-21T23:05:06.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.641+0000 7f6f1e000640 1 -- start start 2024-02-21T23:05:06.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.642+0000 7f6f1e000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f1806c3c0 0x7f6f181533f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:06.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.642+0000 7f6f1e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f18072920 0x7f6f181eba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:06.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.642+0000 7f6f1e000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6f181505c0 0x7f6f181ebf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:06.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.642+0000 7f6f1e000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f180ab130 con 0x7f6f18072920 2024-02-21T23:05:06.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.642+0000 7f6f1e000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f181509a0 con 0x7f6f1806c3c0 2024-02-21T23:05:06.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.642+0000 7f6f1e000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f180ab2b0 con 0x7f6f181505c0 2024-02-21T23:05:06.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.642+0000 7f6f17e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f18072920 0x7f6f181eba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:06.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.643+0000 7f6f17e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f18072920 0x7f6f181eba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36094/0 (socket says 172.21.15.28:36094) 2024-02-21T23:05:06.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.643+0000 7f6f17e00640 1 -- 172.21.15.28:0/2674854010 learned_addr learned my addr 172.21.15.28:0/2674854010 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:06.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.643+0000 7f6f1d600640 1 --2- 172.21.15.28:0/2674854010 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6f181505c0 0x7f6f181ebf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:06.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.643+0000 7f6f1cc00640 1 --2- 172.21.15.28:0/2674854010 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f1806c3c0 0x7f6f181533f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:06.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.643+0000 7f6f17e00640 1 -- 172.21.15.28:0/2674854010 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6f181505c0 msgr2=0x7f6f181ebf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:06.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.643+0000 7f6f17e00640 1 --2- 172.21.15.28:0/2674854010 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6f181505c0 0x7f6f181ebf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:06.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.643+0000 7f6f17e00640 1 -- 172.21.15.28:0/2674854010 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f1806c3c0 msgr2=0x7f6f181533f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:06.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.643+0000 7f6f17e00640 1 --2- 172.21.15.28:0/2674854010 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f1806c3c0 0x7f6f181533f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:06.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.643+0000 7f6f17e00640 1 -- 172.21.15.28:0/2674854010 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f180bbef0 con 0x7f6f18072920 2024-02-21T23:05:06.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.643+0000 7f6f17e00640 1 --2- 172.21.15.28:0/2674854010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f18072920 0x7f6f181eba10 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f6f0400d800 tx=0x7f6f0400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:06.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.644+0000 7f6f14c00640 1 -- 172.21.15.28:0/2674854010 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f0400b6c0 con 0x7f6f18072920 2024-02-21T23:05:06.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.644+0000 7f6f14c00640 1 -- 172.21.15.28:0/2674854010 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f04019070 con 0x7f6f18072920 2024-02-21T23:05:06.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.644+0000 7f6f1e000640 1 -- 172.21.15.28:0/2674854010 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f180c1450 con 0x7f6f18072920 2024-02-21T23:05:06.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.644+0000 7f6f14c00640 1 -- 172.21.15.28:0/2674854010 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f0400ba80 con 0x7f6f18072920 2024-02-21T23:05:06.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.644+0000 7f6f1e000640 1 -- 172.21.15.28:0/2674854010 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f1814c260 con 0x7f6f18072920 2024-02-21T23:05:06.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.645+0000 7f6f1e000640 1 -- 172.21.15.28:0/2674854010 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f18072e10 con 0x7f6f18072920 2024-02-21T23:05:06.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.646+0000 7f6f14c00640 1 -- 172.21.15.28:0/2674854010 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6f0400bc20 con 0x7f6f18072920 2024-02-21T23:05:06.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.647+0000 7f6f14c00640 1 --2- 172.21.15.28:0/2674854010 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6ef806b4d0 0x7f6ef806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:06.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.647+0000 7f6f14c00640 1 -- 172.21.15.28:0/2674854010 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6472+0+0 (secure 0 0 0) 0x7f6f0408b350 con 0x7f6f18072920 2024-02-21T23:05:06.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.648+0000 7f6f1cc00640 1 --2- 172.21.15.28:0/2674854010 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6ef806b4d0 0x7f6ef806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:06.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.649+0000 7f6f1cc00640 1 --2- 172.21.15.28:0/2674854010 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6ef806b4d0 0x7f6ef806d990 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c036000 tx=0x7f6f0c03d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:06.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.651+0000 7f6f14c00640 1 -- 172.21.15.28:0/2674854010 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f6f18072e10 con 0x7f6f18072920 2024-02-21T23:05:06.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:06.850+0000 7f6f1e000640 1 -- 172.21.15.28:0/2674854010 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs2_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f6f1804f290 con 0x7f6f18072920 2024-02-21T23:05:07.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.398+0000 7f6f14c00640 1 -- 172.21.15.28:0/2674854010 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs2_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs2_data' created v55) v1 ==== 172+0+0 (secure 0 0 0) 0x7f6f0405a040 con 0x7f6f18072920 2024-02-21T23:05:07.388 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs2_data' created 2024-02-21T23:05:07.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.402+0000 7f6f1e000640 1 -- 172.21.15.28:0/2674854010 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6ef806b4d0 msgr2=0x7f6ef806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:07.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.402+0000 7f6f1e000640 1 --2- 172.21.15.28:0/2674854010 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6ef806b4d0 0x7f6ef806d990 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c036000 tx=0x7f6f0c03d040 comp rx=0 tx=0).stop 2024-02-21T23:05:07.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.402+0000 7f6f1e000640 1 -- 172.21.15.28:0/2674854010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f18072920 msgr2=0x7f6f181eba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:07.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.402+0000 7f6f1e000640 1 --2- 172.21.15.28:0/2674854010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f18072920 0x7f6f181eba10 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f6f0400d800 tx=0x7f6f0400dcc0 comp rx=0 tx=0).stop 2024-02-21T23:05:07.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.402+0000 7f6f1e000640 1 -- 172.21.15.28:0/2674854010 shutdown_connections 2024-02-21T23:05:07.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.402+0000 7f6f1e000640 1 --2- 172.21.15.28:0/2674854010 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6ef806b4d0 0x7f6ef806d990 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:07.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.402+0000 7f6f1e000640 1 --2- 172.21.15.28:0/2674854010 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6f181505c0 0x7f6f181ebf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:07.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.402+0000 7f6f1e000640 1 --2- 172.21.15.28:0/2674854010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f18072920 0x7f6f181eba10 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:07.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.402+0000 7f6f1e000640 1 --2- 172.21.15.28:0/2674854010 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f1806c3c0 0x7f6f181533f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:07.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.402+0000 7f6f1e000640 1 -- 172.21.15.28:0/2674854010 >> 172.21.15.28:0/2674854010 conn(0x7f6f180c4000 msgr2=0x7f6f1806ae70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:07.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.403+0000 7f6f1e000640 1 -- 172.21.15.28:0/2674854010 shutdown_connections 2024-02-21T23:05:07.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.403+0000 7f6f1e000640 1 -- 172.21.15.28:0/2674854010 wait complete. 2024-02-21T23:05:07.431 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs2 cephfs2_metadata cephfs2_data 2024-02-21T23:05:07.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.628+0000 7f272e200640 1 Processor -- start 2024-02-21T23:05:07.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.628+0000 7f272e200640 1 -- start start 2024-02-21T23:05:07.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.630+0000 7f272e200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f27281525a0 0x7f27280c86a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:07.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.630+0000 7f272e200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2728158cf0 0x7f27280c7120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:07.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.630+0000 7f272e200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27280c5120 con 0x7f272806dd20 2024-02-21T23:05:07.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.630+0000 7f272e200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2728158530 con 0x7f27281525a0 2024-02-21T23:05:07.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.630+0000 7f272e200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27280c53a0 con 0x7f2728158cf0 2024-02-21T23:05:07.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.630+0000 7f272ce00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f27281525a0 0x7f27280c86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:07.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.630+0000 7f272ce00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f27281525a0 0x7f27280c86a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36104/0 (socket says 172.21.15.28:36104) 2024-02-21T23:05:07.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.630+0000 7f272ce00640 1 -- 172.21.15.28:0/2374974339 learned_addr learned my addr 172.21.15.28:0/2374974339 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:07.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.630+0000 7f272d800640 1 --2- 172.21.15.28:0/2374974339 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2728158cf0 0x7f27280c7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:07.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.631+0000 7f272ce00640 1 -- 172.21.15.28:0/2374974339 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2728158cf0 msgr2=0x7f27280c7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:07.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.631+0000 7f272ce00640 1 --2- 172.21.15.28:0/2374974339 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2728158cf0 0x7f27280c7120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:07.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.631+0000 7f272ce00640 1 -- 172.21.15.28:0/2374974339 >> v1:172.21.15.28:6789/0 conn(0x7f272806dd20 legacy=0x7f2728155920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:07.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.631+0000 7f272ce00640 1 -- 172.21.15.28:0/2374974339 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27280c5680 con 0x7f27281525a0 2024-02-21T23:05:07.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.632+0000 7f272ce00640 1 --2- 172.21.15.28:0/2374974339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27281525a0 0x7f27280c86a0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f271c009870 tx=0x7f271c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4c1968cd002725c server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:07.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.632+0000 7f2727400640 1 -- 172.21.15.28:0/2374974339 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f271c035020 con 0x7f27281525a0 2024-02-21T23:05:07.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.632+0000 7f2727400640 1 -- 172.21.15.28:0/2374974339 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f271c031640 con 0x7f27281525a0 2024-02-21T23:05:07.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.632+0000 7f2727400640 1 -- 172.21.15.28:0/2374974339 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f271c02d040 con 0x7f27281525a0 2024-02-21T23:05:07.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.633+0000 7f272e200640 1 -- 172.21.15.28:0/2374974339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27281525a0 msgr2=0x7f27280c86a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:07.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.633+0000 7f272e200640 1 --2- 172.21.15.28:0/2374974339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27281525a0 0x7f27280c86a0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f271c009870 tx=0x7f271c02b8f0 comp rx=0 tx=0).stop 2024-02-21T23:05:07.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.633+0000 7f272e200640 1 -- 172.21.15.28:0/2374974339 shutdown_connections 2024-02-21T23:05:07.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.633+0000 7f272e200640 1 --2- 172.21.15.28:0/2374974339 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2728158cf0 0x7f27280c7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:07.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.633+0000 7f272e200640 1 --2- 172.21.15.28:0/2374974339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27281525a0 0x7f27280c86a0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:07.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.633+0000 7f272e200640 1 -- 172.21.15.28:0/2374974339 >> 172.21.15.28:0/2374974339 conn(0x7f27280ac4d0 msgr2=0x7f27280ac8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:07.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.633+0000 7f272e200640 1 -- 172.21.15.28:0/2374974339 shutdown_connections 2024-02-21T23:05:07.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.633+0000 7f272e200640 1 -- 172.21.15.28:0/2374974339 wait complete. 2024-02-21T23:05:07.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.634+0000 7f272e200640 1 Processor -- start 2024-02-21T23:05:07.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.635+0000 7f272e200640 1 -- start start 2024-02-21T23:05:07.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.635+0000 7f272e200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f272806dd20 0x7f27280c1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:07.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.635+0000 7f272e200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f27281525a0 0x7f27281ebaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:07.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.635+0000 7f272e200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2728158cf0 0x7f27281ec030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:07.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.636+0000 7f272e200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27280c5ae0 con 0x7f2728158cf0 2024-02-21T23:05:07.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.636+0000 7f272e200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27280c5960 con 0x7f272806dd20 2024-02-21T23:05:07.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.636+0000 7f272e200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2728159120 con 0x7f27281525a0 2024-02-21T23:05:07.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.636+0000 7f272d800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2728158cf0 0x7f27281ec030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:07.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.636+0000 7f272d800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2728158cf0 0x7f27281ec030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36118/0 (socket says 172.21.15.28:36118) 2024-02-21T23:05:07.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.636+0000 7f272d800640 1 -- 172.21.15.28:0/2772296669 learned_addr learned my addr 172.21.15.28:0/2772296669 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:07.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.636+0000 7f2727e00640 1 --2- 172.21.15.28:0/2772296669 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f27281525a0 0x7f27281ebaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:07.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.636+0000 7f272ce00640 1 --2- 172.21.15.28:0/2772296669 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f272806dd20 0x7f27280c1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:07.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.636+0000 7f272d800640 1 -- 172.21.15.28:0/2772296669 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f27281525a0 msgr2=0x7f27281ebaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:07.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.636+0000 7f272d800640 1 --2- 172.21.15.28:0/2772296669 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f27281525a0 0x7f27281ebaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:07.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.636+0000 7f272d800640 1 -- 172.21.15.28:0/2772296669 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f272806dd20 msgr2=0x7f27280c1ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:07.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.636+0000 7f272d800640 1 --2- 172.21.15.28:0/2772296669 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f272806dd20 0x7f27280c1ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:07.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.636+0000 7f272d800640 1 -- 172.21.15.28:0/2772296669 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f272814a4d0 con 0x7f2728158cf0 2024-02-21T23:05:07.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.636+0000 7f2727e00640 1 --2- 172.21.15.28:0/2772296669 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f27281525a0 0x7f27281ebaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:05:07.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.637+0000 7f272d800640 1 --2- 172.21.15.28:0/2772296669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2728158cf0 0x7f27281ec030 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f2720007e90 tx=0x7f272000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:07.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.637+0000 7f2724c00640 1 -- 172.21.15.28:0/2772296669 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2720014420 con 0x7f2728158cf0 2024-02-21T23:05:07.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.637+0000 7f2724c00640 1 -- 172.21.15.28:0/2772296669 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2720019070 con 0x7f2728158cf0 2024-02-21T23:05:07.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.637+0000 7f272e200640 1 -- 172.21.15.28:0/2772296669 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f272814a7b0 con 0x7f2728158cf0 2024-02-21T23:05:07.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.637+0000 7f2724c00640 1 -- 172.21.15.28:0/2772296669 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2720014800 con 0x7f2728158cf0 2024-02-21T23:05:07.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.637+0000 7f272e200640 1 -- 172.21.15.28:0/2772296669 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27280c3d80 con 0x7f2728158cf0 2024-02-21T23:05:07.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.639+0000 7f272e200640 1 -- 172.21.15.28:0/2772296669 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f272804f290 con 0x7f2728158cf0 2024-02-21T23:05:07.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.639+0000 7f2724c00640 1 -- 172.21.15.28:0/2772296669 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f27200149a0 con 0x7f2728158cf0 2024-02-21T23:05:07.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.640+0000 7f2724c00640 1 --2- 172.21.15.28:0/2772296669 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f270806b4d0 0x7f270806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:07.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.640+0000 7f2724c00640 1 -- 172.21.15.28:0/2772296669 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6855+0+0 (secure 0 0 0) 0x7f272008af50 con 0x7f2728158cf0 2024-02-21T23:05:07.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.641+0000 7f272ce00640 1 --2- 172.21.15.28:0/2772296669 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f270806b4d0 0x7f270806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:07.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.642+0000 7f272ce00640 1 --2- 172.21.15.28:0/2772296669 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f270806b4d0 0x7f270806d990 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f271c036000 tx=0x7f271c008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:07.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:07.645+0000 7f2724c00640 1 -- 172.21.15.28:0/2772296669 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f27200598b0 con 0x7f2728158cf0 2024-02-21T23:05:08.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.014+0000 7f272e200640 1 -- 172.21.15.28:0/2772296669 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs2", "metadata": "cephfs2_metadata", "data": "cephfs2_data"} v 0) v1 -- 0x7f272806da70 con 0x7f2728158cf0 2024-02-21T23:05:08.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.434+0000 7f2724c00640 1 -- 172.21.15.28:0/2772296669 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs2", "metadata": "cephfs2_metadata", "data": "cephfs2_data"}]=0 new fs with metadata pool 9 and data pool 10 v33) v1 ==== 176+0+0 (secure 0 0 0) 0x7f27200591b0 con 0x7f2728158cf0 2024-02-21T23:05:08.427 INFO:teuthology.orchestra.run.smithi028.stderr:new fs with metadata pool 9 and data pool 10 2024-02-21T23:05:08.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.444+0000 7f272e200640 1 -- 172.21.15.28:0/2772296669 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f270806b4d0 msgr2=0x7f270806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:08.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.444+0000 7f272e200640 1 --2- 172.21.15.28:0/2772296669 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f270806b4d0 0x7f270806d990 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f271c036000 tx=0x7f271c008030 comp rx=0 tx=0).stop 2024-02-21T23:05:08.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.444+0000 7f272e200640 1 -- 172.21.15.28:0/2772296669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2728158cf0 msgr2=0x7f27281ec030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:08.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.444+0000 7f272e200640 1 --2- 172.21.15.28:0/2772296669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2728158cf0 0x7f27281ec030 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f2720007e90 tx=0x7f272000e4b0 comp rx=0 tx=0).stop 2024-02-21T23:05:08.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.445+0000 7f272e200640 1 -- 172.21.15.28:0/2772296669 shutdown_connections 2024-02-21T23:05:08.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.445+0000 7f272e200640 1 --2- 172.21.15.28:0/2772296669 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f270806b4d0 0x7f270806d990 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:08.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.445+0000 7f272e200640 1 --2- 172.21.15.28:0/2772296669 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f27281525a0 0x7f27281ebaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:08.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.445+0000 7f272e200640 1 --2- 172.21.15.28:0/2772296669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2728158cf0 0x7f27281ec030 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:08.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.445+0000 7f272e200640 1 --2- 172.21.15.28:0/2772296669 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f272806dd20 0x7f27280c1ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:08.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.445+0000 7f272e200640 1 -- 172.21.15.28:0/2772296669 >> 172.21.15.28:0/2772296669 conn(0x7f27280ac4d0 msgr2=0x7f272814ab20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:08.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.445+0000 7f272e200640 1 -- 172.21.15.28:0/2772296669 shutdown_connections 2024-02-21T23:05:08.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.445+0000 7f272e200640 1 -- 172.21.15.28:0/2772296669 wait complete. 2024-02-21T23:05:08.471 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:05:08.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.663+0000 7fa1c9000640 1 Processor -- start 2024-02-21T23:05:08.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.663+0000 7fa1c9000640 1 -- start start 2024-02-21T23:05:08.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.665+0000 7fa1c9000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fa1c40c5510 0x7fa1c40a7e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:08.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.665+0000 7fa1c9000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1c40c8040 con 0x7fa1c40c7320 2024-02-21T23:05:08.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.665+0000 7fa1c9000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1c40c85c0 con 0x7fa1c4155040 2024-02-21T23:05:08.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.665+0000 7fa1c9000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1c40a8380 con 0x7fa1c40c5510 2024-02-21T23:05:08.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.665+0000 7fa1c1400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa1c40c7320 0x7fa1c41521d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34436/0 (socket says 172.21.15.28:34436) 2024-02-21T23:05:08.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.665+0000 7fa1c1400640 1 -- 172.21.15.28:0/1554599734 learned_addr learned my addr 172.21.15.28:0/1554599734 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:05:08.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.665+0000 7fa1c1e00640 1 --2- 172.21.15.28:0/1554599734 >> v2:172.21.15.81:3300/0 conn(0x7fa1c40c5510 0x7fa1c40a7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:08.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.666+0000 7fa1c0a00640 1 -- 172.21.15.28:0/1554599734 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1431346186 0 0) 0x7fa1c40c8040 con 0x7fa1c40c7320 2024-02-21T23:05:08.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.666+0000 7fa1c0a00640 1 -- 172.21.15.28:0/1554599734 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1a8003520 con 0x7fa1c40c7320 2024-02-21T23:05:08.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.666+0000 7fa1c0a00640 1 -- 172.21.15.28:0/1554599734 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1527164510 0 0) 0x7fa1a8003520 con 0x7fa1c40c7320 2024-02-21T23:05:08.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.666+0000 7fa1c0a00640 1 -- 172.21.15.28:0/1554599734 >> v2:172.21.15.81:3300/0 conn(0x7fa1c40c5510 msgr2=0x7fa1c40a7e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:08.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.666+0000 7fa1c0a00640 1 --2- 172.21.15.28:0/1554599734 >> v2:172.21.15.81:3300/0 conn(0x7fa1c40c5510 0x7fa1c40a7e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:08.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.666+0000 7fa1c0a00640 1 -- 172.21.15.28:0/1554599734 >> v1:172.21.15.81:6789/0 conn(0x7fa1c4155040 legacy=0x7fa1c4155440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:08.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.666+0000 7fa1c0a00640 1 -- 172.21.15.28:0/1554599734 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1c40c82e0 con 0x7fa1c40c7320 2024-02-21T23:05:08.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.666+0000 7fa1c0a00640 1 -- 172.21.15.28:0/1554599734 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fa1b0001690 con 0x7fa1c40c7320 2024-02-21T23:05:08.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.666+0000 7fa1c0a00640 1 -- 172.21.15.28:0/1554599734 >> v1:172.21.15.28:6789/0 conn(0x7fa1c40c7320 legacy=0x7fa1c41521d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:08.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.666+0000 7fa1c0a00640 1 --2- 172.21.15.28:0/1554599734 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1a80039b0 0x7fa1a8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:08.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.666+0000 7fa1c0a00640 1 --2- 172.21.15.28:0/1554599734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1a80043b0 0x7fa1a8004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:08.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.666+0000 7fa1c0a00640 1 --2- 172.21.15.28:0/1554599734 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa1a8005080 0x7fa1a800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:08.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.666+0000 7fa1c0a00640 1 -- 172.21.15.28:0/1554599734 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c40a8380 con 0x7fa1a80043b0 2024-02-21T23:05:08.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.666+0000 7fa1c0a00640 1 -- 172.21.15.28:0/1554599734 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a800bfe0 con 0x7fa1a80039b0 2024-02-21T23:05:08.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.666+0000 7fa1c0a00640 1 -- 172.21.15.28:0/1554599734 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a800c1e0 con 0x7fa1a8005080 2024-02-21T23:05:08.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.667+0000 7fa1c1400640 1 --2- 172.21.15.28:0/1554599734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1a80043b0 0x7fa1a8004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:08.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.667+0000 7fa1c1e00640 1 --2- 172.21.15.28:0/1554599734 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1a80039b0 0x7fa1a8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:08.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.667+0000 7fa1c1400640 1 -- 172.21.15.28:0/1554599734 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa1a8005080 msgr2=0x7fa1a800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:08.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.667+0000 7fa1c1400640 1 --2- 172.21.15.28:0/1554599734 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa1a8005080 0x7fa1a800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:08.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.667+0000 7fa1c1400640 1 -- 172.21.15.28:0/1554599734 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1a80039b0 msgr2=0x7fa1a8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:08.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.667+0000 7fa1c1400640 1 --2- 172.21.15.28:0/1554599734 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1a80039b0 0x7fa1a8003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:08.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.667+0000 7fa1c1400640 1 -- 172.21.15.28:0/1554599734 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1a8003520 con 0x7fa1a80043b0 2024-02-21T23:05:08.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.667+0000 7fa1c2800640 1 -- 172.21.15.28:0/1554599734 reap_dead start 2024-02-21T23:05:08.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.668+0000 7fa1c1400640 1 --2- 172.21.15.28:0/1554599734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1a80043b0 0x7fa1a8004880 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fa1c40c8210 tx=0x7fa1a8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92b5713941658927 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:08.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.669+0000 7fa1c0a00640 1 -- 172.21.15.28:0/1554599734 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1b0007d50 con 0x7fa1a80043b0 2024-02-21T23:05:08.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.669+0000 7fa1c0a00640 1 -- 172.21.15.28:0/1554599734 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1b0030070 con 0x7fa1a80043b0 2024-02-21T23:05:08.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.669+0000 7fa1c0a00640 1 -- 172.21.15.28:0/1554599734 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1b0005d30 con 0x7fa1a80043b0 2024-02-21T23:05:08.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.669+0000 7fa1c9000640 1 -- 172.21.15.28:0/1554599734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1a80043b0 msgr2=0x7fa1a8004880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:08.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.669+0000 7fa1c9000640 1 --2- 172.21.15.28:0/1554599734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1a80043b0 0x7fa1a8004880 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fa1c40c8210 tx=0x7fa1a8005830 comp rx=0 tx=0).stop 2024-02-21T23:05:08.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.669+0000 7fa1c9000640 1 -- 172.21.15.28:0/1554599734 shutdown_connections 2024-02-21T23:05:08.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.669+0000 7fa1c9000640 1 --2- 172.21.15.28:0/1554599734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1a80043b0 0x7fa1a8004880 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:08.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.669+0000 7fa1c9000640 1 -- 172.21.15.28:0/1554599734 >> 172.21.15.28:0/1554599734 conn(0x7fa1c40c8810 msgr2=0x7fa1c40c8bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:08.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.669+0000 7fa1c9000640 1 -- 172.21.15.28:0/1554599734 shutdown_connections 2024-02-21T23:05:08.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.669+0000 7fa1c9000640 1 -- 172.21.15.28:0/1554599734 wait complete. 2024-02-21T23:05:08.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.672+0000 7fa1c9000640 1 Processor -- start 2024-02-21T23:05:08.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.672+0000 7fa1c9000640 1 -- start start 2024-02-21T23:05:08.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.674+0000 7fa1c9000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa1a80043b0 0x7fa1c40a9160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:08.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.674+0000 7fa1c9000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1c40a8530 0x7fa1c40a96a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:08.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.674+0000 7fa1c9000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1c41590e0 0x7fa1c41ee1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:08.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.674+0000 7fa1c9000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c40c58f0 con 0x7fa1c41590e0 2024-02-21T23:05:08.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.674+0000 7fa1c9000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c40c6cb0 con 0x7fa1c40a8530 2024-02-21T23:05:08.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.674+0000 7fa1c9000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c40c5a70 con 0x7fa1a80043b0 2024-02-21T23:05:08.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.674+0000 7fa1c2800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1c41590e0 0x7fa1c41ee1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:08.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.674+0000 7fa1c2800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1c41590e0 0x7fa1c41ee1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36124/0 (socket says 172.21.15.28:36124) 2024-02-21T23:05:08.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.674+0000 7fa1c2800640 1 -- 172.21.15.28:0/4223179948 learned_addr learned my addr 172.21.15.28:0/4223179948 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:08.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.674+0000 7fa1c1e00640 1 --2- 172.21.15.28:0/4223179948 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa1a80043b0 0x7fa1c40a9160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:08.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.675+0000 7fa1c1400640 1 --2- 172.21.15.28:0/4223179948 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1c40a8530 0x7fa1c40a96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:08.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.675+0000 7fa1c2800640 1 -- 172.21.15.28:0/4223179948 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa1a80043b0 msgr2=0x7fa1c40a9160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:08.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.675+0000 7fa1c2800640 1 --2- 172.21.15.28:0/4223179948 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa1a80043b0 0x7fa1c40a9160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:08.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.675+0000 7fa1c2800640 1 -- 172.21.15.28:0/4223179948 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1c40a8530 msgr2=0x7fa1c40a96a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:08.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.675+0000 7fa1c2800640 1 --2- 172.21.15.28:0/4223179948 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1c40a8530 0x7fa1c40a96a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:08.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.675+0000 7fa1c2800640 1 -- 172.21.15.28:0/4223179948 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1c40a8d20 con 0x7fa1c41590e0 2024-02-21T23:05:08.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.675+0000 7fa1c1e00640 1 --2- 172.21.15.28:0/4223179948 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa1a80043b0 0x7fa1c40a9160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:05:08.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.675+0000 7fa1c1400640 1 --2- 172.21.15.28:0/4223179948 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1c40a8530 0x7fa1c40a96a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:05:08.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.675+0000 7fa1c2800640 1 --2- 172.21.15.28:0/4223179948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1c41590e0 0x7fa1c41ee1e0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fa1c406be30 tx=0x7fa1bc00cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:08.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.675+0000 7fa1ba000640 1 -- 172.21.15.28:0/4223179948 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1bc017070 con 0x7fa1c41590e0 2024-02-21T23:05:08.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.675+0000 7fa1ba000640 1 -- 172.21.15.28:0/4223179948 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1bc005d80 con 0x7fa1c41590e0 2024-02-21T23:05:08.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.675+0000 7fa1ba000640 1 -- 172.21.15.28:0/4223179948 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1bc00a820 con 0x7fa1c41590e0 2024-02-21T23:05:08.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.675+0000 7fa1c9000640 1 -- 172.21.15.28:0/4223179948 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1c41eb890 con 0x7fa1c41590e0 2024-02-21T23:05:08.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.675+0000 7fa1c9000640 1 -- 172.21.15.28:0/4223179948 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1c40ab170 con 0x7fa1c41590e0 2024-02-21T23:05:08.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.677+0000 7fa1c9000640 1 -- 172.21.15.28:0/4223179948 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1c41eba60 con 0x7fa1c41590e0 2024-02-21T23:05:08.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.678+0000 7fa1ba000640 1 -- 172.21.15.28:0/4223179948 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa1bc00a9c0 con 0x7fa1c41590e0 2024-02-21T23:05:08.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.680+0000 7fa1ba000640 1 --2- 172.21.15.28:0/4223179948 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa1a406b4d0 0x7fa1a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:08.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.680+0000 7fa1ba000640 1 -- 172.21.15.28:0/4223179948 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6957+0+0 (secure 0 0 0) 0x7fa1bc089830 con 0x7fa1c41590e0 2024-02-21T23:05:08.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.680+0000 7fa1c1e00640 1 --2- 172.21.15.28:0/4223179948 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa1a406b4d0 0x7fa1a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:08.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.681+0000 7fa1c1e00640 1 --2- 172.21.15.28:0/4223179948 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa1a406b4d0 0x7fa1a406d990 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fa1c4070e80 tx=0x7fa1b4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:08.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.684+0000 7fa1ba000640 1 -- 172.21.15.28:0/4223179948 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fa1bc057b80 con 0x7fa1c41590e0 2024-02-21T23:05:08.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.885+0000 7fa1c9000640 1 -- 172.21.15.28:0/4223179948 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa1c41eb6a0 con 0x7fa1c41590e0 2024-02-21T23:05:08.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.890+0000 7fa1ba000640 1 -- 172.21.15.28:0/4223179948 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) v1 ==== 74+0+20619 (secure 0 0 0) 0x7fa1bc05bc20 con 0x7fa1c41590e0 2024-02-21T23:05:08.880 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:05:08.880 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":56,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:05:08.016717+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:04:51.354130+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-02-21T23:04:52.441690+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs2_metadata","create_time":"2024-02-21T23:05:05.701276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs2_data","create_time":"2024-02-21T23:05:06.904887+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:05:08.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.892+0000 7fa1c9000640 1 -- 172.21.15.28:0/4223179948 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa1a406b4d0 msgr2=0x7fa1a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:08.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.892+0000 7fa1c9000640 1 --2- 172.21.15.28:0/4223179948 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa1a406b4d0 0x7fa1a406d990 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fa1c4070e80 tx=0x7fa1b4008040 comp rx=0 tx=0).stop 2024-02-21T23:05:08.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.892+0000 7fa1c9000640 1 -- 172.21.15.28:0/4223179948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1c41590e0 msgr2=0x7fa1c41ee1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:08.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.892+0000 7fa1c9000640 1 --2- 172.21.15.28:0/4223179948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1c41590e0 0x7fa1c41ee1e0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fa1c406be30 tx=0x7fa1bc00cb10 comp rx=0 tx=0).stop 2024-02-21T23:05:08.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.892+0000 7fa1c9000640 1 -- 172.21.15.28:0/4223179948 shutdown_connections 2024-02-21T23:05:08.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.892+0000 7fa1c9000640 1 --2- 172.21.15.28:0/4223179948 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa1a406b4d0 0x7fa1a406d990 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:08.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.892+0000 7fa1c9000640 1 --2- 172.21.15.28:0/4223179948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1c41590e0 0x7fa1c41ee1e0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:08.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.892+0000 7fa1c9000640 1 --2- 172.21.15.28:0/4223179948 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1c40a8530 0x7fa1c40a96a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:08.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.892+0000 7fa1c9000640 1 --2- 172.21.15.28:0/4223179948 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa1a80043b0 0x7fa1c40a9160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:08.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.892+0000 7fa1c9000640 1 -- 172.21.15.28:0/4223179948 >> 172.21.15.28:0/4223179948 conn(0x7fa1c40c8810 msgr2=0x7fa1c40c1c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:08.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.893+0000 7fa1c9000640 1 -- 172.21.15.28:0/4223179948 shutdown_connections 2024-02-21T23:05:08.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:08.893+0000 7fa1c9000640 1 -- 172.21.15.28:0/4223179948 wait complete. 2024-02-21T23:05:08.901 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:05:09.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.119+0000 7f60ab200640 1 Processor -- start 2024-02-21T23:05:09.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.119+0000 7f60ab200640 1 -- start start 2024-02-21T23:05:09.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.121+0000 7f60ab200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f60ac06f130 0x7f60ac0a9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:09.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.121+0000 7f60ab200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f60ac06bad0 0x7f60ac06f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:09.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.121+0000 7f60ab200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60ac06b6d0 con 0x7f60ac0c4d20 2024-02-21T23:05:09.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.121+0000 7f60ab200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60ac06fc80 con 0x7f60ac06f130 2024-02-21T23:05:09.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.121+0000 7f60ab200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60ac06ff60 con 0x7f60ac06bad0 2024-02-21T23:05:09.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.121+0000 7f60a9e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f60ac06f130 0x7f60ac0a9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:09.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.121+0000 7f60a9e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f60ac06f130 0x7f60ac0a9eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36130/0 (socket says 172.21.15.28:36130) 2024-02-21T23:05:09.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.121+0000 7f60a9e00640 1 -- 172.21.15.28:0/1927817475 learned_addr learned my addr 172.21.15.28:0/1927817475 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:09.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.122+0000 7f60a9400640 1 --2- 172.21.15.28:0/1927817475 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f60ac06bad0 0x7f60ac06f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:09.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.122+0000 7f60a9e00640 1 -- 172.21.15.28:0/1927817475 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f60ac06bad0 msgr2=0x7f60ac06f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:09.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.122+0000 7f60a9e00640 1 --2- 172.21.15.28:0/1927817475 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f60ac06bad0 0x7f60ac06f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:09.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.122+0000 7f60a9e00640 1 -- 172.21.15.28:0/1927817475 >> v1:172.21.15.81:6789/0 conn(0x7f60ac0c4d20 legacy=0x7f60ac158580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:09.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.122+0000 7f60a9e00640 1 -- 172.21.15.28:0/1927817475 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60ac070240 con 0x7f60ac06f130 2024-02-21T23:05:09.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.123+0000 7f60a9e00640 1 --2- 172.21.15.28:0/1927817475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60ac06f130 0x7f60ac0a9eb0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f609c009870 tx=0x7f609c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cc9df5b7fa7b850 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:09.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.123+0000 7f60a8a00640 1 -- 172.21.15.28:0/1927817475 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f609c035020 con 0x7f60ac06f130 2024-02-21T23:05:09.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.123+0000 7f60a8a00640 1 -- 172.21.15.28:0/1927817475 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f609c031640 con 0x7f60ac06f130 2024-02-21T23:05:09.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.123+0000 7f60a8a00640 1 -- 172.21.15.28:0/1927817475 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f609c02d040 con 0x7f60ac06f130 2024-02-21T23:05:09.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.124+0000 7f60ab200640 1 -- 172.21.15.28:0/1927817475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60ac06f130 msgr2=0x7f60ac0a9eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:09.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.124+0000 7f60ab200640 1 --2- 172.21.15.28:0/1927817475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60ac06f130 0x7f60ac0a9eb0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f609c009870 tx=0x7f609c02b8f0 comp rx=0 tx=0).stop 2024-02-21T23:05:09.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.124+0000 7f60ab200640 1 -- 172.21.15.28:0/1927817475 shutdown_connections 2024-02-21T23:05:09.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.124+0000 7f60ab200640 1 --2- 172.21.15.28:0/1927817475 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f60ac06bad0 0x7f60ac06f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:09.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.124+0000 7f60ab200640 1 --2- 172.21.15.28:0/1927817475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60ac06f130 0x7f60ac0a9eb0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:09.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.124+0000 7f60ab200640 1 -- 172.21.15.28:0/1927817475 >> 172.21.15.28:0/1927817475 conn(0x7f60ac067050 msgr2=0x7f60ac0a84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:09.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.124+0000 7f60ab200640 1 -- 172.21.15.28:0/1927817475 shutdown_connections 2024-02-21T23:05:09.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.124+0000 7f60ab200640 1 -- 172.21.15.28:0/1927817475 wait complete. 2024-02-21T23:05:09.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.125+0000 7f60ab200640 1 Processor -- start 2024-02-21T23:05:09.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.126+0000 7f60ab200640 1 -- start start 2024-02-21T23:05:09.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.126+0000 7f60ab200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f60ac06bad0 0x7f60ac1eb910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:09.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.127+0000 7f60ab200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f60ac06f130 0x7f60ac0c69c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:09.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.127+0000 7f60ab200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60ac0c4d20 0x7f60ac1dedb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:09.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.127+0000 7f60ab200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60ac0a9990 con 0x7f60ac0c4d20 2024-02-21T23:05:09.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.127+0000 7f60ab200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60ac0a96b0 con 0x7f60ac06f130 2024-02-21T23:05:09.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.127+0000 7f60ab200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60ac158c90 con 0x7f60ac06bad0 2024-02-21T23:05:09.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.127+0000 7f60aa800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60ac0c4d20 0x7f60ac1dedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:09.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.127+0000 7f60aa800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60ac0c4d20 0x7f60ac1dedb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36136/0 (socket says 172.21.15.28:36136) 2024-02-21T23:05:09.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.127+0000 7f60aa800640 1 -- 172.21.15.28:0/1138625874 learned_addr learned my addr 172.21.15.28:0/1138625874 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:09.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.127+0000 7f60a9e00640 1 --2- 172.21.15.28:0/1138625874 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f60ac06bad0 0x7f60ac1eb910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:09.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.127+0000 7f60a9400640 1 --2- 172.21.15.28:0/1138625874 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f60ac06f130 0x7f60ac0c69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:09.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.127+0000 7f60aa800640 1 -- 172.21.15.28:0/1138625874 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f60ac06bad0 msgr2=0x7f60ac1eb910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:09.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.127+0000 7f60aa800640 1 --2- 172.21.15.28:0/1138625874 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f60ac06bad0 0x7f60ac1eb910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:09.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.127+0000 7f60aa800640 1 -- 172.21.15.28:0/1138625874 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f60ac06f130 msgr2=0x7f60ac0c69c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:09.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.127+0000 7f60aa800640 1 --2- 172.21.15.28:0/1138625874 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f60ac06f130 0x7f60ac0c69c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:09.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.127+0000 7f60aa800640 1 -- 172.21.15.28:0/1138625874 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60ac1de5f0 con 0x7f60ac0c4d20 2024-02-21T23:05:09.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.128+0000 7f60aa800640 1 --2- 172.21.15.28:0/1138625874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60ac0c4d20 0x7f60ac1dedb0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f60a400c760 tx=0x7f60a400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:09.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.128+0000 7f60a2000640 1 -- 172.21.15.28:0/1138625874 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a4003b70 con 0x7f60ac0c4d20 2024-02-21T23:05:09.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.128+0000 7f60a2000640 1 -- 172.21.15.28:0/1138625874 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60a4003d10 con 0x7f60ac0c4d20 2024-02-21T23:05:09.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.128+0000 7f60a2000640 1 -- 172.21.15.28:0/1138625874 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a400aa30 con 0x7f60ac0c4d20 2024-02-21T23:05:09.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.128+0000 7f60ab200640 1 -- 172.21.15.28:0/1138625874 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60ac153530 con 0x7f60ac0c4d20 2024-02-21T23:05:09.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.128+0000 7f60ab200640 1 -- 172.21.15.28:0/1138625874 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60ac0c6f00 con 0x7f60ac0c4d20 2024-02-21T23:05:09.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.129+0000 7f60ab200640 1 -- 172.21.15.28:0/1138625874 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60ac04f290 con 0x7f60ac0c4d20 2024-02-21T23:05:09.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.130+0000 7f60a2000640 1 -- 172.21.15.28:0/1138625874 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f60a400abd0 con 0x7f60ac0c4d20 2024-02-21T23:05:09.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.131+0000 7f60a2000640 1 --2- 172.21.15.28:0/1138625874 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f608c06b500 0x7f608c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:09.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.131+0000 7f60a2000640 1 -- 172.21.15.28:0/1138625874 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f60a408a570 con 0x7f60ac0c4d20 2024-02-21T23:05:09.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.132+0000 7f60a9e00640 1 --2- 172.21.15.28:0/1138625874 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f608c06b500 0x7f608c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:09.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.133+0000 7f60a9e00640 1 --2- 172.21.15.28:0/1138625874 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f608c06b500 0x7f608c06d9c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f609c036000 tx=0x7f609c03c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:09.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.136+0000 7f60a2000640 1 -- 172.21.15.28:0/1138625874 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f60a4058f00 con 0x7f60ac0c4d20 2024-02-21T23:05:09.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.328+0000 7f60ab200640 1 -- 172.21.15.28:0/1138625874 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f60ac0c48c0 con 0x7f60ac0c4d20 2024-02-21T23:05:09.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.332+0000 7f60a2000640 1 -- 172.21.15.28:0/1138625874 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) v1 ==== 74+0+20619 (secure 0 0 0) 0x7f60a4058800 con 0x7f60ac0c4d20 2024-02-21T23:05:09.321 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:05:09.321 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":56,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:05:08.016717+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:04:51.354130+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-02-21T23:04:52.441690+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs2_metadata","create_time":"2024-02-21T23:05:05.701276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs2_data","create_time":"2024-02-21T23:05:06.904887+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:05:09.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.334+0000 7f60ab200640 1 -- 172.21.15.28:0/1138625874 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f608c06b500 msgr2=0x7f608c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:09.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.334+0000 7f60ab200640 1 --2- 172.21.15.28:0/1138625874 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f608c06b500 0x7f608c06d9c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f609c036000 tx=0x7f609c03c040 comp rx=0 tx=0).stop 2024-02-21T23:05:09.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.334+0000 7f60ab200640 1 -- 172.21.15.28:0/1138625874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60ac0c4d20 msgr2=0x7f60ac1dedb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:09.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.334+0000 7f60ab200640 1 --2- 172.21.15.28:0/1138625874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60ac0c4d20 0x7f60ac1dedb0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f60a400c760 tx=0x7f60a400cc20 comp rx=0 tx=0).stop 2024-02-21T23:05:09.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.334+0000 7f60ab200640 1 -- 172.21.15.28:0/1138625874 shutdown_connections 2024-02-21T23:05:09.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.334+0000 7f60ab200640 1 --2- 172.21.15.28:0/1138625874 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f608c06b500 0x7f608c06d9c0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:09.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.334+0000 7f60ab200640 1 --2- 172.21.15.28:0/1138625874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60ac0c4d20 0x7f60ac1dedb0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:09.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.334+0000 7f60ab200640 1 --2- 172.21.15.28:0/1138625874 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f60ac06f130 0x7f60ac0c69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:09.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.334+0000 7f60ab200640 1 --2- 172.21.15.28:0/1138625874 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f60ac06bad0 0x7f60ac1eb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:09.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.334+0000 7f60ab200640 1 -- 172.21.15.28:0/1138625874 >> 172.21.15.28:0/1138625874 conn(0x7f60ac067050 msgr2=0x7f60ac0a84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:09.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.334+0000 7f60ab200640 1 -- 172.21.15.28:0/1138625874 shutdown_connections 2024-02-21T23:05:09.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.334+0000 7f60ab200640 1 -- 172.21.15.28:0/1138625874 wait complete. 2024-02-21T23:05:09.341 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs2 standby_count_wanted 0 2024-02-21T23:05:09.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.561+0000 7f3af3e00640 1 Processor -- start 2024-02-21T23:05:09.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.561+0000 7f3af3e00640 1 -- start start 2024-02-21T23:05:09.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.562+0000 7f3af3e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3af4152560 0x7f3af4154f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:09.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.562+0000 7f3af3e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3af4158480 0x7f3af406ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:09.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.562+0000 7f3af3e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3af40a6270 con 0x7f3af406b3b0 2024-02-21T23:05:09.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.562+0000 7f3af3e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3af41554a0 con 0x7f3af4152560 2024-02-21T23:05:09.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.562+0000 7f3af3e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3af41556d0 con 0x7f3af4158480 2024-02-21T23:05:09.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.563+0000 7f3af2a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3af4152560 0x7f3af4154f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:09.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.563+0000 7f3af3400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f3af406b3b0 0x7f3af406e9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34444/0 (socket says 172.21.15.28:34444) 2024-02-21T23:05:09.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.563+0000 7f3af3400640 1 -- 172.21.15.28:0/1293259767 learned_addr learned my addr 172.21.15.28:0/1293259767 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:05:09.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.563+0000 7f3af2000640 1 --2- 172.21.15.28:0/1293259767 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3af4158480 0x7f3af406ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:09.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.563+0000 7f3af2a00640 1 -- 172.21.15.28:0/1293259767 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3af4158480 msgr2=0x7f3af406ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:09.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.563+0000 7f3af2a00640 1 --2- 172.21.15.28:0/1293259767 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3af4158480 0x7f3af406ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:09.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.563+0000 7f3af2a00640 1 -- 172.21.15.28:0/1293259767 >> v1:172.21.15.28:6789/0 conn(0x7f3af406b3b0 legacy=0x7f3af406e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:09.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.563+0000 7f3af2a00640 1 -- 172.21.15.28:0/1293259767 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3af41559b0 con 0x7f3af4152560 2024-02-21T23:05:09.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.564+0000 7f3af2a00640 1 --2- 172.21.15.28:0/1293259767 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3af4152560 0x7f3af4154f60 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f3ae40097a0 tx=0x7f3ae402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f70a469f2b1fd825 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:09.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.565+0000 7f3af1600640 1 -- 172.21.15.28:0/1293259767 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae4035020 con 0x7f3af4152560 2024-02-21T23:05:09.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.565+0000 7f3af1600640 1 -- 172.21.15.28:0/1293259767 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ae4031810 con 0x7f3af4152560 2024-02-21T23:05:09.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.565+0000 7f3af1600640 1 -- 172.21.15.28:0/1293259767 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae4031be0 con 0x7f3af4152560 2024-02-21T23:05:09.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.565+0000 7f3af3e00640 1 -- 172.21.15.28:0/1293259767 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3af4152560 msgr2=0x7f3af4154f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:09.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.565+0000 7f3af3e00640 1 --2- 172.21.15.28:0/1293259767 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3af4152560 0x7f3af4154f60 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f3ae40097a0 tx=0x7f3ae402b8b0 comp rx=0 tx=0).stop 2024-02-21T23:05:09.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.565+0000 7f3af3e00640 1 -- 172.21.15.28:0/1293259767 shutdown_connections 2024-02-21T23:05:09.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.565+0000 7f3af3e00640 1 --2- 172.21.15.28:0/1293259767 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3af4158480 0x7f3af406ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:09.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.565+0000 7f3af3e00640 1 --2- 172.21.15.28:0/1293259767 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3af4152560 0x7f3af4154f60 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:09.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.565+0000 7f3af3e00640 1 -- 172.21.15.28:0/1293259767 >> 172.21.15.28:0/1293259767 conn(0x7f3af40a8530 msgr2=0x7f3af40a8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:09.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.566+0000 7f3af3e00640 1 -- 172.21.15.28:0/1293259767 shutdown_connections 2024-02-21T23:05:09.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.566+0000 7f3af3e00640 1 -- 172.21.15.28:0/1293259767 wait complete. 2024-02-21T23:05:09.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.567+0000 7f3af3e00640 1 Processor -- start 2024-02-21T23:05:09.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.567+0000 7f3af3e00640 1 -- start start 2024-02-21T23:05:09.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.568+0000 7f3af3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3af406b3b0 0x7f3af41542f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:09.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.568+0000 7f3af3e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3af4152560 0x7f3af4156ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:09.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.568+0000 7f3af3e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3af4158480 0x7f3af4071990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:09.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.568+0000 7f3af3e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3af4152940 con 0x7f3af406b3b0 2024-02-21T23:05:09.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.568+0000 7f3af3e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3af406e4b0 con 0x7f3af4158480 2024-02-21T23:05:09.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.568+0000 7f3af3e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3af40c1bc0 con 0x7f3af4152560 2024-02-21T23:05:09.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.568+0000 7f3af2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3af406b3b0 0x7f3af41542f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:09.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.568+0000 7f3af2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3af406b3b0 0x7f3af41542f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36154/0 (socket says 172.21.15.28:36154) 2024-02-21T23:05:09.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.568+0000 7f3af2a00640 1 -- 172.21.15.28:0/1851462330 learned_addr learned my addr 172.21.15.28:0/1851462330 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:09.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.569+0000 7f3af2000640 1 --2- 172.21.15.28:0/1851462330 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3af4152560 0x7f3af4156ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:09.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.569+0000 7f3af3400640 1 --2- 172.21.15.28:0/1851462330 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3af4158480 0x7f3af4071990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:09.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.569+0000 7f3af2a00640 1 -- 172.21.15.28:0/1851462330 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3af4152560 msgr2=0x7f3af4156ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:09.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.569+0000 7f3af2a00640 1 --2- 172.21.15.28:0/1851462330 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3af4152560 0x7f3af4156ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:09.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.569+0000 7f3af2a00640 1 -- 172.21.15.28:0/1851462330 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3af4158480 msgr2=0x7f3af4071990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:09.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.569+0000 7f3af2a00640 1 --2- 172.21.15.28:0/1851462330 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3af4158480 0x7f3af4071990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:09.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.569+0000 7f3af2a00640 1 -- 172.21.15.28:0/1851462330 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3af4151210 con 0x7f3af406b3b0 2024-02-21T23:05:09.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.569+0000 7f3af3400640 1 --2- 172.21.15.28:0/1851462330 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3af4158480 0x7f3af4071990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:05:09.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.569+0000 7f3af2a00640 1 --2- 172.21.15.28:0/1851462330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3af406b3b0 0x7f3af41542f0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f3ae402bec0 tx=0x7f3ae40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:09.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.570+0000 7f3aeaa00640 1 -- 172.21.15.28:0/1851462330 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae403bd70 con 0x7f3af406b3b0 2024-02-21T23:05:09.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.570+0000 7f3aeaa00640 1 -- 172.21.15.28:0/1851462330 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ae40453f0 con 0x7f3af406b3b0 2024-02-21T23:05:09.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.570+0000 7f3aeaa00640 1 -- 172.21.15.28:0/1851462330 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae40457e0 con 0x7f3af406b3b0 2024-02-21T23:05:09.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.570+0000 7f3af3e00640 1 -- 172.21.15.28:0/1851462330 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3af4153950 con 0x7f3af406b3b0 2024-02-21T23:05:09.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.570+0000 7f3af3e00640 1 -- 172.21.15.28:0/1851462330 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3af4153500 con 0x7f3af406b3b0 2024-02-21T23:05:09.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.571+0000 7f3af3e00640 1 -- 172.21.15.28:0/1851462330 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3af404f290 con 0x7f3af406b3b0 2024-02-21T23:05:09.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.572+0000 7f3aeaa00640 1 -- 172.21.15.28:0/1851462330 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3ae4038020 con 0x7f3af406b3b0 2024-02-21T23:05:09.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.573+0000 7f3aeaa00640 1 --2- 172.21.15.28:0/1851462330 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3ad006b550 0x7f3ad006da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:09.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.573+0000 7f3aeaa00640 1 -- 172.21.15.28:0/1851462330 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f3ae40ac410 con 0x7f3af406b3b0 2024-02-21T23:05:09.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.573+0000 7f3af2000640 1 --2- 172.21.15.28:0/1851462330 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3ad006b550 0x7f3ad006da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:09.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.574+0000 7f3af2000640 1 --2- 172.21.15.28:0/1851462330 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3ad006b550 0x7f3ad006da10 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f3af414a790 tx=0x7f3adc008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:09.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.577+0000 7f3aeaa00640 1 -- 172.21.15.28:0/1851462330 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3ae4040070 con 0x7f3af406b3b0 2024-02-21T23:05:09.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:09.817+0000 7f3af3e00640 1 -- 172.21.15.28:0/1851462330 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs2", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f3af4153b20 con 0x7f3af406b3b0 2024-02-21T23:05:10.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.458+0000 7f3aeaa00640 1 -- 172.21.15.28:0/1851462330 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs2", "var": "standby_count_wanted", "val": "0"}]=0 v36) v1 ==== 119+0+0 (secure 0 0 0) 0x7f3ae407a8d0 con 0x7f3af406b3b0 2024-02-21T23:05:10.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.462+0000 7f3af3e00640 1 -- 172.21.15.28:0/1851462330 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3ad006b550 msgr2=0x7f3ad006da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:10.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.462+0000 7f3af3e00640 1 --2- 172.21.15.28:0/1851462330 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3ad006b550 0x7f3ad006da10 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f3af414a790 tx=0x7f3adc008040 comp rx=0 tx=0).stop 2024-02-21T23:05:10.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.462+0000 7f3af3e00640 1 -- 172.21.15.28:0/1851462330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3af406b3b0 msgr2=0x7f3af41542f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:10.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.462+0000 7f3af3e00640 1 --2- 172.21.15.28:0/1851462330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3af406b3b0 0x7f3af41542f0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f3ae402bec0 tx=0x7f3ae40046c0 comp rx=0 tx=0).stop 2024-02-21T23:05:10.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.462+0000 7f3af3e00640 1 -- 172.21.15.28:0/1851462330 shutdown_connections 2024-02-21T23:05:10.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.462+0000 7f3af3e00640 1 --2- 172.21.15.28:0/1851462330 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3ad006b550 0x7f3ad006da10 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:10.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.462+0000 7f3af3e00640 1 --2- 172.21.15.28:0/1851462330 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3af4152560 0x7f3af4156ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:10.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.462+0000 7f3af3e00640 1 --2- 172.21.15.28:0/1851462330 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3af4158480 0x7f3af4071990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:10.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.462+0000 7f3af3e00640 1 --2- 172.21.15.28:0/1851462330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3af406b3b0 0x7f3af41542f0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:10.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.463+0000 7f3af3e00640 1 -- 172.21.15.28:0/1851462330 >> 172.21.15.28:0/1851462330 conn(0x7f3af40a8530 msgr2=0x7f3af40a8910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:10.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.463+0000 7f3af3e00640 1 -- 172.21.15.28:0/1851462330 shutdown_connections 2024-02-21T23:05:10.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.463+0000 7f3af3e00640 1 -- 172.21.15.28:0/1851462330 wait complete. 2024-02-21T23:05:10.480 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-02-21T23:05:10.480 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs2 allow_standby_replay false 2024-02-21T23:05:10.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.712+0000 7eff61e00640 1 Processor -- start 2024-02-21T23:05:10.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.712+0000 7eff61e00640 1 -- start start 2024-02-21T23:05:10.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.714+0000 7eff61e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7eff5c13e0c0 0x7eff5c0c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:10.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.714+0000 7eff61e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7eff5c0c76f0 0x7eff5c06c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:10.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.714+0000 7eff61e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff5c158550 con 0x7eff5c06dbc0 2024-02-21T23:05:10.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.714+0000 7eff61e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff5c0c89a0 con 0x7eff5c13e0c0 2024-02-21T23:05:10.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.714+0000 7eff61e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff5c1587d0 con 0x7eff5c0c76f0 2024-02-21T23:05:10.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.714+0000 7eff60a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7eff5c13e0c0 0x7eff5c0c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:10.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.715+0000 7eff60a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7eff5c13e0c0 0x7eff5c0c71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36158/0 (socket says 172.21.15.28:36158) 2024-02-21T23:05:10.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.715+0000 7eff60a00640 1 -- 172.21.15.28:0/385774115 learned_addr learned my addr 172.21.15.28:0/385774115 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:10.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.715+0000 7eff61400640 1 --2- 172.21.15.28:0/385774115 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7eff5c0c76f0 0x7eff5c06c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:10.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.715+0000 7eff60a00640 1 -- 172.21.15.28:0/385774115 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7eff5c0c76f0 msgr2=0x7eff5c06c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:10.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.715+0000 7eff60a00640 1 --2- 172.21.15.28:0/385774115 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7eff5c0c76f0 0x7eff5c06c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:10.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.715+0000 7eff60a00640 1 -- 172.21.15.28:0/385774115 >> v1:172.21.15.81:6789/0 conn(0x7eff5c06dbc0 legacy=0x7eff5c06c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:10.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.715+0000 7eff60a00640 1 -- 172.21.15.28:0/385774115 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff5c158ab0 con 0x7eff5c13e0c0 2024-02-21T23:05:10.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.717+0000 7eff60a00640 1 --2- 172.21.15.28:0/385774115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff5c13e0c0 0x7eff5c0c71b0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7eff50009870 tx=0x7eff5002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=175038fdca5accfd server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:10.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.717+0000 7eff5b400640 1 -- 172.21.15.28:0/385774115 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff500317c0 con 0x7eff5c13e0c0 2024-02-21T23:05:10.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.717+0000 7eff5b400640 1 -- 172.21.15.28:0/385774115 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff50037070 con 0x7eff5c13e0c0 2024-02-21T23:05:10.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.717+0000 7eff5b400640 1 -- 172.21.15.28:0/385774115 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff50031b80 con 0x7eff5c13e0c0 2024-02-21T23:05:10.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.718+0000 7eff61e00640 1 -- 172.21.15.28:0/385774115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff5c13e0c0 msgr2=0x7eff5c0c71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:10.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.718+0000 7eff61e00640 1 --2- 172.21.15.28:0/385774115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff5c13e0c0 0x7eff5c0c71b0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7eff50009870 tx=0x7eff5002b860 comp rx=0 tx=0).stop 2024-02-21T23:05:10.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.718+0000 7eff61e00640 1 -- 172.21.15.28:0/385774115 shutdown_connections 2024-02-21T23:05:10.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.718+0000 7eff61e00640 1 --2- 172.21.15.28:0/385774115 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7eff5c0c76f0 0x7eff5c06c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:10.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.718+0000 7eff61e00640 1 --2- 172.21.15.28:0/385774115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff5c13e0c0 0x7eff5c0c71b0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:10.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.718+0000 7eff61e00640 1 -- 172.21.15.28:0/385774115 >> 172.21.15.28:0/385774115 conn(0x7eff5c073110 msgr2=0x7eff5c0ac490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:10.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.718+0000 7eff61e00640 1 -- 172.21.15.28:0/385774115 shutdown_connections 2024-02-21T23:05:10.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.718+0000 7eff61e00640 1 -- 172.21.15.28:0/385774115 wait complete. 2024-02-21T23:05:10.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.720+0000 7eff61e00640 1 Processor -- start 2024-02-21T23:05:10.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.720+0000 7eff61e00640 1 -- start start 2024-02-21T23:05:10.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.721+0000 7eff61e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7eff5c06dbc0 0x7eff5c157630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:10.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.721+0000 7eff61e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff5c0c76f0 0x7eff5c1e01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:10.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.721+0000 7eff61e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7eff5c13e0c0 0x7eff5c1e0700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:10.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.721+0000 7eff61e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff5c06ced0 con 0x7eff5c0c76f0 2024-02-21T23:05:10.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.721+0000 7eff61e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff5c159070 con 0x7eff5c13e0c0 2024-02-21T23:05:10.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.721+0000 7eff61e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff5c13e520 con 0x7eff5c06dbc0 2024-02-21T23:05:10.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.721+0000 7eff5be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff5c0c76f0 0x7eff5c1e01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:10.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.722+0000 7eff5be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff5c0c76f0 0x7eff5c1e01c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36160/0 (socket says 172.21.15.28:36160) 2024-02-21T23:05:10.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.722+0000 7eff5be00640 1 -- 172.21.15.28:0/680904894 learned_addr learned my addr 172.21.15.28:0/680904894 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:10.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.722+0000 7eff60a00640 1 --2- 172.21.15.28:0/680904894 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7eff5c06dbc0 0x7eff5c157630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:10.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.722+0000 7eff61400640 1 --2- 172.21.15.28:0/680904894 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7eff5c13e0c0 0x7eff5c1e0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:10.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.722+0000 7eff5be00640 1 -- 172.21.15.28:0/680904894 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7eff5c06dbc0 msgr2=0x7eff5c157630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:10.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.722+0000 7eff5be00640 1 --2- 172.21.15.28:0/680904894 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7eff5c06dbc0 0x7eff5c157630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:10.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.722+0000 7eff5be00640 1 -- 172.21.15.28:0/680904894 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7eff5c13e0c0 msgr2=0x7eff5c1e0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:10.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.722+0000 7eff5be00640 1 --2- 172.21.15.28:0/680904894 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7eff5c13e0c0 0x7eff5c1e0700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:10.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.722+0000 7eff5be00640 1 -- 172.21.15.28:0/680904894 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff5c1e0d40 con 0x7eff5c0c76f0 2024-02-21T23:05:10.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.722+0000 7eff61400640 1 --2- 172.21.15.28:0/680904894 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7eff5c13e0c0 0x7eff5c1e0700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:05:10.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.722+0000 7eff5be00640 1 --2- 172.21.15.28:0/680904894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff5c0c76f0 0x7eff5c1e01c0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7eff4800c430 tx=0x7eff4800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:10.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.723+0000 7eff58c00640 1 -- 172.21.15.28:0/680904894 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff48011790 con 0x7eff5c0c76f0 2024-02-21T23:05:10.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.723+0000 7eff58c00640 1 -- 172.21.15.28:0/680904894 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff48017070 con 0x7eff5c0c76f0 2024-02-21T23:05:10.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.723+0000 7eff58c00640 1 -- 172.21.15.28:0/680904894 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff48011bb0 con 0x7eff5c0c76f0 2024-02-21T23:05:10.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.723+0000 7eff61e00640 1 -- 172.21.15.28:0/680904894 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff5c1de690 con 0x7eff5c0c76f0 2024-02-21T23:05:10.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.723+0000 7eff61e00640 1 -- 172.21.15.28:0/680904894 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff5c1dec50 con 0x7eff5c0c76f0 2024-02-21T23:05:10.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.724+0000 7eff61e00640 1 -- 172.21.15.28:0/680904894 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff5c04f290 con 0x7eff5c0c76f0 2024-02-21T23:05:10.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.726+0000 7eff58c00640 1 -- 172.21.15.28:0/680904894 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7eff4801c460 con 0x7eff5c0c76f0 2024-02-21T23:05:10.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.727+0000 7eff58c00640 1 --2- 172.21.15.28:0/680904894 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7eff3c06b4d0 0x7eff3c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:10.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.727+0000 7eff58c00640 1 -- 172.21.15.28:0/680904894 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6957+0+0 (secure 0 0 0) 0x7eff4808a5d0 con 0x7eff5c0c76f0 2024-02-21T23:05:10.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.728+0000 7eff60a00640 1 --2- 172.21.15.28:0/680904894 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7eff3c06b4d0 0x7eff3c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:10.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.729+0000 7eff60a00640 1 --2- 172.21.15.28:0/680904894 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7eff3c06b4d0 0x7eff3c06d990 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7eff50036000 tx=0x7eff5003d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:10.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.732+0000 7eff58c00640 1 -- 172.21.15.28:0/680904894 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7eff48058f60 con 0x7eff5c0c76f0 2024-02-21T23:05:10.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:10.973+0000 7eff61e00640 1 -- 172.21.15.28:0/680904894 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs2", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7eff5c13a2f0 con 0x7eff5c0c76f0 2024-02-21T23:05:11.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.470+0000 7eff58c00640 1 -- 172.21.15.28:0/680904894 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs2", "var": "allow_standby_replay", "val": "false"}]=0 v37) v1 ==== 123+0+0 (secure 0 0 0) 0x7eff48058860 con 0x7eff5c0c76f0 2024-02-21T23:05:11.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.475+0000 7eff61e00640 1 -- 172.21.15.28:0/680904894 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7eff3c06b4d0 msgr2=0x7eff3c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:11.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.475+0000 7eff61e00640 1 --2- 172.21.15.28:0/680904894 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7eff3c06b4d0 0x7eff3c06d990 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7eff50036000 tx=0x7eff5003d040 comp rx=0 tx=0).stop 2024-02-21T23:05:11.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.475+0000 7eff61e00640 1 -- 172.21.15.28:0/680904894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff5c0c76f0 msgr2=0x7eff5c1e01c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:11.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.475+0000 7eff61e00640 1 --2- 172.21.15.28:0/680904894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff5c0c76f0 0x7eff5c1e01c0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7eff4800c430 tx=0x7eff4800c8f0 comp rx=0 tx=0).stop 2024-02-21T23:05:11.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.475+0000 7eff61e00640 1 -- 172.21.15.28:0/680904894 shutdown_connections 2024-02-21T23:05:11.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.475+0000 7eff61e00640 1 --2- 172.21.15.28:0/680904894 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7eff3c06b4d0 0x7eff3c06d990 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:11.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.475+0000 7eff61e00640 1 --2- 172.21.15.28:0/680904894 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7eff5c13e0c0 0x7eff5c1e0700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:11.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.475+0000 7eff61e00640 1 --2- 172.21.15.28:0/680904894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff5c0c76f0 0x7eff5c1e01c0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:11.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.475+0000 7eff61e00640 1 --2- 172.21.15.28:0/680904894 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7eff5c06dbc0 0x7eff5c157630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:11.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.475+0000 7eff61e00640 1 -- 172.21.15.28:0/680904894 >> 172.21.15.28:0/680904894 conn(0x7eff5c073110 msgr2=0x7eff5c0c8fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:11.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.476+0000 7eff61e00640 1 -- 172.21.15.28:0/680904894 shutdown_connections 2024-02-21T23:05:11.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.476+0000 7eff61e00640 1 -- 172.21.15.28:0/680904894 wait complete. 2024-02-21T23:05:11.485 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs2' 2024-02-21T23:05:11.485 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:05:11.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.709+0000 7f48a6c00640 1 Processor -- start 2024-02-21T23:05:11.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.709+0000 7f48a6c00640 1 -- start start 2024-02-21T23:05:11.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.711+0000 7f48a6c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f48a00c0fe0 0x7f48a006e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:11.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.711+0000 7f48a6c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f48a0158580 0x7f48a014b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:11.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.711+0000 7f48a6c00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48a00a8b40 con 0x7f48a006dff0 2024-02-21T23:05:11.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.711+0000 7f48a6c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48a00a8e00 con 0x7f48a0158580 2024-02-21T23:05:11.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.711+0000 7f48a6c00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48a00a90e0 con 0x7f48a00c0fe0 2024-02-21T23:05:11.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.711+0000 7f48a6200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f48a0158580 0x7f48a014b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:11.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.711+0000 7f48a6200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f48a0158580 0x7f48a014b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36174/0 (socket says 172.21.15.28:36174) 2024-02-21T23:05:11.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.711+0000 7f48a6200640 1 -- 172.21.15.28:0/412410801 learned_addr learned my addr 172.21.15.28:0/412410801 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:11.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.712+0000 7f48a4e00640 1 --2- 172.21.15.28:0/412410801 >> v2:172.21.15.81:3300/0 conn(0x7f48a00c0fe0 0x7f48a006e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:11.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.712+0000 7f48a6200640 1 -- 172.21.15.28:0/412410801 >> v2:172.21.15.81:3300/0 conn(0x7f48a00c0fe0 msgr2=0x7f48a006e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:11.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.712+0000 7f48a6200640 1 --2- 172.21.15.28:0/412410801 >> v2:172.21.15.81:3300/0 conn(0x7f48a00c0fe0 0x7f48a006e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:11.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.712+0000 7f48a6200640 1 -- 172.21.15.28:0/412410801 >> v1:172.21.15.81:6789/0 conn(0x7f48a006dff0 legacy=0x7f48a0071350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:11.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.712+0000 7f48a6200640 1 -- 172.21.15.28:0/412410801 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48a0072ed0 con 0x7f48a0158580 2024-02-21T23:05:11.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.713+0000 7f48a6200640 1 --2- 172.21.15.28:0/412410801 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48a0158580 0x7f48a014b2b0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f48a006ccd0 tx=0x7f489802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c4e2fdfd11a277f server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:11.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.714+0000 7f489fe00640 1 -- 172.21.15.28:0/412410801 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f489802ca80 con 0x7f48a0158580 2024-02-21T23:05:11.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.714+0000 7f489fe00640 1 -- 172.21.15.28:0/412410801 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f489802cc20 con 0x7f48a0158580 2024-02-21T23:05:11.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.714+0000 7f489fe00640 1 -- 172.21.15.28:0/412410801 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48980343f0 con 0x7f48a0158580 2024-02-21T23:05:11.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.714+0000 7f48a6c00640 1 -- 172.21.15.28:0/412410801 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48a0158580 msgr2=0x7f48a014b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:11.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.714+0000 7f48a6c00640 1 --2- 172.21.15.28:0/412410801 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48a0158580 0x7f48a014b2b0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f48a006ccd0 tx=0x7f489802bee0 comp rx=0 tx=0).stop 2024-02-21T23:05:11.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.714+0000 7f48a6c00640 1 -- 172.21.15.28:0/412410801 shutdown_connections 2024-02-21T23:05:11.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.714+0000 7f48a6c00640 1 --2- 172.21.15.28:0/412410801 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48a0158580 0x7f48a014b2b0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:11.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.714+0000 7f48a6c00640 1 --2- 172.21.15.28:0/412410801 >> v2:172.21.15.81:3300/0 conn(0x7f48a00c0fe0 0x7f48a006e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:11.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.714+0000 7f48a6c00640 1 -- 172.21.15.28:0/412410801 >> 172.21.15.28:0/412410801 conn(0x7f48a006b630 msgr2=0x7f48a00c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:11.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.714+0000 7f48a6c00640 1 -- 172.21.15.28:0/412410801 shutdown_connections 2024-02-21T23:05:11.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.714+0000 7f48a6c00640 1 -- 172.21.15.28:0/412410801 wait complete. 2024-02-21T23:05:11.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.716+0000 7f48a6c00640 1 Processor -- start 2024-02-21T23:05:11.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.716+0000 7f48a6c00640 1 -- start start 2024-02-21T23:05:11.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.717+0000 7f48a6c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f48a006dff0 0x7f48a00ac290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:11.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.717+0000 7f48a6c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48a00c0fe0 0x7f48a01ede70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:11.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.717+0000 7f48a6c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f48a0158580 0x7f48a00ab060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:11.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.717+0000 7f48a6c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48a0158980 con 0x7f48a00c0fe0 2024-02-21T23:05:11.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.717+0000 7f48a6c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48a0073520 con 0x7f48a0158580 2024-02-21T23:05:11.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.717+0000 7f48a6c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48a0158b00 con 0x7f48a006dff0 2024-02-21T23:05:11.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.717+0000 7f48a4e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48a00c0fe0 0x7f48a01ede70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:11.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.718+0000 7f48a4e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48a00c0fe0 0x7f48a01ede70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36180/0 (socket says 172.21.15.28:36180) 2024-02-21T23:05:11.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.718+0000 7f48a4e00640 1 -- 172.21.15.28:0/3379035709 learned_addr learned my addr 172.21.15.28:0/3379035709 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:11.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.718+0000 7f48a6200640 1 --2- 172.21.15.28:0/3379035709 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f48a0158580 0x7f48a00ab060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:11.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.718+0000 7f48a5800640 1 --2- 172.21.15.28:0/3379035709 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f48a006dff0 0x7f48a00ac290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:11.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.718+0000 7f48a4e00640 1 -- 172.21.15.28:0/3379035709 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f48a006dff0 msgr2=0x7f48a00ac290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:11.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.718+0000 7f48a4e00640 1 --2- 172.21.15.28:0/3379035709 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f48a006dff0 0x7f48a00ac290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:11.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.718+0000 7f48a4e00640 1 -- 172.21.15.28:0/3379035709 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f48a0158580 msgr2=0x7f48a00ab060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:11.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.718+0000 7f48a4e00640 1 --2- 172.21.15.28:0/3379035709 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f48a0158580 0x7f48a00ab060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:11.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.718+0000 7f48a4e00640 1 -- 172.21.15.28:0/3379035709 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48a00ab890 con 0x7f48a00c0fe0 2024-02-21T23:05:11.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.718+0000 7f48a4e00640 1 --2- 172.21.15.28:0/3379035709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48a00c0fe0 0x7f48a01ede70 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f489000d830 tx=0x7f489000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:11.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.719+0000 7f489d600640 1 -- 172.21.15.28:0/3379035709 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f489000b6c0 con 0x7f48a00c0fe0 2024-02-21T23:05:11.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.719+0000 7f489d600640 1 -- 172.21.15.28:0/3379035709 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4890019070 con 0x7f48a00c0fe0 2024-02-21T23:05:11.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.719+0000 7f489d600640 1 -- 172.21.15.28:0/3379035709 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f489000ba80 con 0x7f48a00c0fe0 2024-02-21T23:05:11.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.719+0000 7f48a6c00640 1 -- 172.21.15.28:0/3379035709 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48a0154020 con 0x7f48a00c0fe0 2024-02-21T23:05:11.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.719+0000 7f48a6c00640 1 -- 172.21.15.28:0/3379035709 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48a00ab5a0 con 0x7f48a00c0fe0 2024-02-21T23:05:11.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.720+0000 7f48a6c00640 1 -- 172.21.15.28:0/3379035709 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48a004f290 con 0x7f48a00c0fe0 2024-02-21T23:05:11.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.721+0000 7f489d600640 1 -- 172.21.15.28:0/3379035709 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f489001e460 con 0x7f48a00c0fe0 2024-02-21T23:05:11.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.722+0000 7f489d600640 1 --2- 172.21.15.28:0/3379035709 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f488406b500 0x7f488406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:11.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.722+0000 7f489d600640 1 -- 172.21.15.28:0/3379035709 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f489008ad90 con 0x7f48a00c0fe0 2024-02-21T23:05:11.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.722+0000 7f48a5800640 1 --2- 172.21.15.28:0/3379035709 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f488406b500 0x7f488406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:11.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.723+0000 7f48a5800640 1 --2- 172.21.15.28:0/3379035709 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f488406b500 0x7f488406d9c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f4894006fd0 tx=0x7f4894008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:11.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.726+0000 7f489d600640 1 -- 172.21.15.28:0/3379035709 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4890059720 con 0x7f48a00c0fe0 2024-02-21T23:05:11.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.995+0000 7f48a6c00640 1 -- 172.21.15.28:0/3379035709 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f48a0152890 con 0x7f48a00c0fe0 2024-02-21T23:05:11.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.996+0000 7f489d600640 1 -- 172.21.15.28:0/3379035709 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+6177 (secure 0 0 0) 0x7f4890059020 con 0x7f48a00c0fe0 2024-02-21T23:05:11.985 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:05:11.985 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":37,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5006,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3468780363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3468780363},{"type":"v1","addr":"172.21.15.28:6835","nonce":3468780363}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27},{"gid":5221,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/2152160500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2152160500},{"type":"v1","addr":"172.21.15.81:6835","nonce":2152160500}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:04:53.432188+0000","modified":"2024-02-21T23:04:56.493528+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5715},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5715":{"gid":5715,"name":"a","rank":0,"incarnation":28,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6834/2803823391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2803823391},{"type":"v1","addr":"172.21.15.28:6834","nonce":2803823391}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3},{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:05:08.017586+0000","modified":"2024-02-21T23:05:10.974691+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5224},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5224":{"gid":5224,"name":"d","rank":0,"incarnation":33,"state":"up:active","state_seq":7,"addr":"172.21.15.81:6837/442767245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":442767245},{"type":"v1","addr":"172.21.15.81:6837","nonce":442767245}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-02-21T23:05:11.985 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 37 2024-02-21T23:05:11.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.999+0000 7f48a6c00640 1 -- 172.21.15.28:0/3379035709 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f488406b500 msgr2=0x7f488406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:11.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.999+0000 7f48a6c00640 1 --2- 172.21.15.28:0/3379035709 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f488406b500 0x7f488406d9c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f4894006fd0 tx=0x7f4894008040 comp rx=0 tx=0).stop 2024-02-21T23:05:11.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.999+0000 7f48a6c00640 1 -- 172.21.15.28:0/3379035709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48a00c0fe0 msgr2=0x7f48a01ede70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:11.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.999+0000 7f48a6c00640 1 --2- 172.21.15.28:0/3379035709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48a00c0fe0 0x7f48a01ede70 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f489000d830 tx=0x7f489000dcf0 comp rx=0 tx=0).stop 2024-02-21T23:05:11.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.999+0000 7f48a6c00640 1 -- 172.21.15.28:0/3379035709 shutdown_connections 2024-02-21T23:05:11.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.999+0000 7f48a6c00640 1 --2- 172.21.15.28:0/3379035709 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f488406b500 0x7f488406d9c0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:11.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.999+0000 7f48a6c00640 1 --2- 172.21.15.28:0/3379035709 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f48a0158580 0x7f48a00ab060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:11.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.999+0000 7f48a6c00640 1 --2- 172.21.15.28:0/3379035709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48a00c0fe0 0x7f48a01ede70 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:11.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.999+0000 7f48a6c00640 1 --2- 172.21.15.28:0/3379035709 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f48a006dff0 0x7f48a00ac290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:11.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.999+0000 7f48a6c00640 1 -- 172.21.15.28:0/3379035709 >> 172.21.15.28:0/3379035709 conn(0x7f48a006b630 msgr2=0x7f48a0157a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:11.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.999+0000 7f48a6c00640 1 -- 172.21.15.28:0/3379035709 shutdown_connections 2024-02-21T23:05:11.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:11.999+0000 7f48a6c00640 1 -- 172.21.15.28:0/3379035709 wait complete. 2024-02-21T23:05:12.015 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:05:12.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.244+0000 7fee2b000640 1 Processor -- start 2024-02-21T23:05:12.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.245+0000 7fee2b000640 1 -- start start 2024-02-21T23:05:12.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.246+0000 7fee2b000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fee24156c90 0x7fee240c8e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:12.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.246+0000 7fee2b000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fee241540b0 0x7fee2415d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:12.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.246+0000 7fee2b000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee240c8000 con 0x7fee240c8a80 2024-02-21T23:05:12.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.246+0000 7fee2b000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee24157110 con 0x7fee241540b0 2024-02-21T23:05:12.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.246+0000 7fee2b000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee240c82e0 con 0x7fee24156c90 2024-02-21T23:05:12.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.247+0000 7fee2a600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fee241540b0 0x7fee2415d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:12.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.247+0000 7fee2a600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fee241540b0 0x7fee2415d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36182/0 (socket says 172.21.15.28:36182) 2024-02-21T23:05:12.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.247+0000 7fee2a600640 1 -- 172.21.15.28:0/4292090146 learned_addr learned my addr 172.21.15.28:0/4292090146 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:12.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.247+0000 7fee29200640 1 --2- 172.21.15.28:0/4292090146 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fee24156c90 0x7fee240c8e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:12.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.247+0000 7fee2a600640 1 -- 172.21.15.28:0/4292090146 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fee24156c90 msgr2=0x7fee240c8e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:12.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.247+0000 7fee2a600640 1 --2- 172.21.15.28:0/4292090146 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fee24156c90 0x7fee240c8e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:12.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.247+0000 7fee2a600640 1 -- 172.21.15.28:0/4292090146 >> v1:172.21.15.81:6789/0 conn(0x7fee240c8a80 legacy=0x7fee24153640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:12.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.247+0000 7fee2a600640 1 -- 172.21.15.28:0/4292090146 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee241561b0 con 0x7fee241540b0 2024-02-21T23:05:12.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.249+0000 7fee2a600640 1 --2- 172.21.15.28:0/4292090146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee241540b0 0x7fee2415d540 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fee1c009870 tx=0x7fee1c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9647f80444ba647f server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:12.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.249+0000 7fee23e00640 1 -- 172.21.15.28:0/4292090146 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee1c035020 con 0x7fee241540b0 2024-02-21T23:05:12.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.249+0000 7fee23e00640 1 -- 172.21.15.28:0/4292090146 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee1c031640 con 0x7fee241540b0 2024-02-21T23:05:12.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.249+0000 7fee23e00640 1 -- 172.21.15.28:0/4292090146 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee1c02d040 con 0x7fee241540b0 2024-02-21T23:05:12.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.250+0000 7fee2b000640 1 -- 172.21.15.28:0/4292090146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee241540b0 msgr2=0x7fee2415d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:12.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.250+0000 7fee2b000640 1 --2- 172.21.15.28:0/4292090146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee241540b0 0x7fee2415d540 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fee1c009870 tx=0x7fee1c02b8f0 comp rx=0 tx=0).stop 2024-02-21T23:05:12.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.250+0000 7fee2b000640 1 -- 172.21.15.28:0/4292090146 shutdown_connections 2024-02-21T23:05:12.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.250+0000 7fee2b000640 1 --2- 172.21.15.28:0/4292090146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee241540b0 0x7fee2415d540 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:12.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.250+0000 7fee2b000640 1 --2- 172.21.15.28:0/4292090146 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fee24156c90 0x7fee240c8e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:12.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.250+0000 7fee2b000640 1 -- 172.21.15.28:0/4292090146 >> 172.21.15.28:0/4292090146 conn(0x7fee2406b260 msgr2=0x7fee2406b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:12.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.250+0000 7fee2b000640 1 -- 172.21.15.28:0/4292090146 shutdown_connections 2024-02-21T23:05:12.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.250+0000 7fee2b000640 1 -- 172.21.15.28:0/4292090146 wait complete. 2024-02-21T23:05:12.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.253+0000 7fee2b000640 1 Processor -- start 2024-02-21T23:05:12.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.253+0000 7fee2b000640 1 -- start start 2024-02-21T23:05:12.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.254+0000 7fee2b000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee240c8a80 0x7fee24151620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:12.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.254+0000 7fee2b000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee241540b0 0x7fee240ab440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:12.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.254+0000 7fee2b000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fee24156c90 0x7fee241437f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:12.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.254+0000 7fee2b000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee240c7c90 con 0x7fee240c8a80 2024-02-21T23:05:12.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.254+0000 7fee2b000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee24156770 con 0x7fee241540b0 2024-02-21T23:05:12.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.254+0000 7fee2b000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee240c2c30 con 0x7fee24156c90 2024-02-21T23:05:12.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.255+0000 7fee29c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee240c8a80 0x7fee24151620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:12.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.255+0000 7fee29c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee240c8a80 0x7fee24151620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36198/0 (socket says 172.21.15.28:36198) 2024-02-21T23:05:12.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.255+0000 7fee29c00640 1 -- 172.21.15.28:0/2844078751 learned_addr learned my addr 172.21.15.28:0/2844078751 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:12.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.255+0000 7fee2a600640 1 --2- 172.21.15.28:0/2844078751 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fee24156c90 0x7fee241437f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:12.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.255+0000 7fee29c00640 1 -- 172.21.15.28:0/2844078751 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fee24156c90 msgr2=0x7fee241437f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:12.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.255+0000 7fee29c00640 1 --2- 172.21.15.28:0/2844078751 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fee24156c90 0x7fee241437f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:12.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.255+0000 7fee29c00640 1 -- 172.21.15.28:0/2844078751 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee241540b0 msgr2=0x7fee240ab440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:12.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.255+0000 7fee29c00640 1 --2- 172.21.15.28:0/2844078751 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee241540b0 0x7fee240ab440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:12.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.255+0000 7fee29c00640 1 -- 172.21.15.28:0/2844078751 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee240ad270 con 0x7fee240c8a80 2024-02-21T23:05:12.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.255+0000 7fee2a600640 1 --2- 172.21.15.28:0/2844078751 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fee24156c90 0x7fee241437f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:05:12.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.256+0000 7fee29c00640 1 --2- 172.21.15.28:0/2844078751 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee240c8a80 0x7fee24151620 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fee1400c730 tx=0x7fee1400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:12.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.256+0000 7fee21600640 1 -- 172.21.15.28:0/2844078751 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee14016020 con 0x7fee240c8a80 2024-02-21T23:05:12.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.256+0000 7fee21600640 1 -- 172.21.15.28:0/2844078751 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee14006d50 con 0x7fee240c8a80 2024-02-21T23:05:12.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.256+0000 7fee21600640 1 -- 172.21.15.28:0/2844078751 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee14003bb0 con 0x7fee240c8a80 2024-02-21T23:05:12.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.256+0000 7fee2b000640 1 -- 172.21.15.28:0/2844078751 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee2414a9b0 con 0x7fee240c8a80 2024-02-21T23:05:12.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.256+0000 7fee2b000640 1 -- 172.21.15.28:0/2844078751 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee24146710 con 0x7fee240c8a80 2024-02-21T23:05:12.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.257+0000 7fee2b000640 1 -- 172.21.15.28:0/2844078751 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee2404f290 con 0x7fee240c8a80 2024-02-21T23:05:12.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.258+0000 7fee21600640 1 -- 172.21.15.28:0/2844078751 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fee1400a850 con 0x7fee240c8a80 2024-02-21T23:05:12.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.259+0000 7fee21600640 1 --2- 172.21.15.28:0/2844078751 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fee0406b4d0 0x7fee0406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:12.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.259+0000 7fee21600640 1 -- 172.21.15.28:0/2844078751 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6957+0+0 (secure 0 0 0) 0x7fee1408a3a0 con 0x7fee240c8a80 2024-02-21T23:05:12.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.260+0000 7fee29200640 1 --2- 172.21.15.28:0/2844078751 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fee0406b4d0 0x7fee0406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:12.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.261+0000 7fee29200640 1 --2- 172.21.15.28:0/2844078751 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fee0406b4d0 0x7fee0406d990 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fee10006a90 tx=0x7fee10006a20 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:12.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.264+0000 7fee21600640 1 -- 172.21.15.28:0/2844078751 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fee14020020 con 0x7fee240c8a80 2024-02-21T23:05:12.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.556+0000 7fee2b000640 1 -- 172.21.15.28:0/2844078751 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fee24150570 con 0x7fee240c8a80 2024-02-21T23:05:12.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.559+0000 7fee21600640 1 -- 172.21.15.28:0/2844078751 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) v1 ==== 74+0+20606 (secure 0 0 0) 0x7fee14019310 con 0x7fee240c8a80 2024-02-21T23:05:12.548 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:05:12.548 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":57,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:05:09.435529+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:04:51.354130+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-02-21T23:04:52.441690+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs2_metadata","create_time":"2024-02-21T23:05:05.701276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs2_data","create_time":"2024-02-21T23:05:06.904887+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:05:12.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.562+0000 7fee2b000640 1 -- 172.21.15.28:0/2844078751 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fee0406b4d0 msgr2=0x7fee0406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:12.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.562+0000 7fee2b000640 1 --2- 172.21.15.28:0/2844078751 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fee0406b4d0 0x7fee0406d990 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fee10006a90 tx=0x7fee10006a20 comp rx=0 tx=0).stop 2024-02-21T23:05:12.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.562+0000 7fee2b000640 1 -- 172.21.15.28:0/2844078751 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee240c8a80 msgr2=0x7fee24151620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:12.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.562+0000 7fee2b000640 1 --2- 172.21.15.28:0/2844078751 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee240c8a80 0x7fee24151620 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fee1400c730 tx=0x7fee1400cbf0 comp rx=0 tx=0).stop 2024-02-21T23:05:12.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.562+0000 7fee2b000640 1 -- 172.21.15.28:0/2844078751 shutdown_connections 2024-02-21T23:05:12.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.562+0000 7fee2b000640 1 --2- 172.21.15.28:0/2844078751 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fee0406b4d0 0x7fee0406d990 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:12.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.562+0000 7fee2b000640 1 --2- 172.21.15.28:0/2844078751 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fee24156c90 0x7fee241437f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:12.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.562+0000 7fee2b000640 1 --2- 172.21.15.28:0/2844078751 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee241540b0 0x7fee240ab440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:12.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.562+0000 7fee2b000640 1 --2- 172.21.15.28:0/2844078751 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee240c8a80 0x7fee24151620 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:12.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.562+0000 7fee2b000640 1 -- 172.21.15.28:0/2844078751 >> 172.21.15.28:0/2844078751 conn(0x7fee2406b260 msgr2=0x7fee24155960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:12.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.562+0000 7fee2b000640 1 -- 172.21.15.28:0/2844078751 shutdown_connections 2024-02-21T23:05:12.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.563+0000 7fee2b000640 1 -- 172.21.15.28:0/2844078751 wait complete. 2024-02-21T23:05:12.573 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-02-21T23:05:12.573 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-21T23:05:12.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.801+0000 7fa962c00640 1 Processor -- start 2024-02-21T23:05:12.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.801+0000 7fa962c00640 1 -- start start 2024-02-21T23:05:12.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.802+0000 7fa962c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa95c06dbf0 0x7fa95c158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:12.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.802+0000 7fa962c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa95c0c74c0 con 0x7fa95c0aff90 2024-02-21T23:05:12.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.802+0000 7fa962c00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa95c0c77a0 con 0x7fa95c155960 2024-02-21T23:05:12.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.802+0000 7fa962c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa95c0c7270 con 0x7fa95c06dbf0 2024-02-21T23:05:12.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.803+0000 7fa961800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa95c0aff90 0x7fa95c0c51e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34446/0 (socket says 172.21.15.28:34446) 2024-02-21T23:05:12.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.803+0000 7fa961800640 1 -- 172.21.15.28:0/327393898 learned_addr learned my addr 172.21.15.28:0/327393898 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:05:12.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.803+0000 7fa960e00640 1 --2- 172.21.15.28:0/327393898 >> v2:172.21.15.28:3300/0 conn(0x7fa95c06dbf0 0x7fa95c158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:12.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.803+0000 7fa960e00640 1 -- 172.21.15.28:0/327393898 >> v1:172.21.15.81:6789/0 conn(0x7fa95c155960 legacy=0x7fa95c15f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:12.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.803+0000 7fa95be00640 1 -- 172.21.15.28:0/327393898 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1855898719 0 0) 0x7fa95c0c74c0 con 0x7fa95c0aff90 2024-02-21T23:05:12.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.803+0000 7fa960e00640 1 -- 172.21.15.28:0/327393898 >> v1:172.21.15.28:6789/0 conn(0x7fa95c0aff90 legacy=0x7fa95c0c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:12.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.803+0000 7fa960e00640 1 -- 172.21.15.28:0/327393898 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa95c0c7a80 con 0x7fa95c06dbf0 2024-02-21T23:05:12.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.805+0000 7fa960e00640 1 --2- 172.21.15.28:0/327393898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa95c06dbf0 0x7fa95c158ac0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fa9480097a0 tx=0x7fa94802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97e81bb5af488bf9 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:12.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.805+0000 7fa95be00640 1 -- 172.21.15.28:0/327393898 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa948035020 con 0x7fa95c06dbf0 2024-02-21T23:05:12.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.805+0000 7fa95be00640 1 -- 172.21.15.28:0/327393898 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa948031830 con 0x7fa95c06dbf0 2024-02-21T23:05:12.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.805+0000 7fa95be00640 1 -- 172.21.15.28:0/327393898 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa948031c00 con 0x7fa95c06dbf0 2024-02-21T23:05:12.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.805+0000 7fa962c00640 1 -- 172.21.15.28:0/327393898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa95c06dbf0 msgr2=0x7fa95c158ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:12.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.805+0000 7fa962c00640 1 --2- 172.21.15.28:0/327393898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa95c06dbf0 0x7fa95c158ac0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fa9480097a0 tx=0x7fa94802b8d0 comp rx=0 tx=0).stop 2024-02-21T23:05:12.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.806+0000 7fa962c00640 1 -- 172.21.15.28:0/327393898 shutdown_connections 2024-02-21T23:05:12.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.806+0000 7fa962c00640 1 --2- 172.21.15.28:0/327393898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa95c06dbf0 0x7fa95c158ac0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:12.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.806+0000 7fa962c00640 1 -- 172.21.15.28:0/327393898 >> 172.21.15.28:0/327393898 conn(0x7fa95c0ac370 msgr2=0x7fa95c0ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:12.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.806+0000 7fa962c00640 1 -- 172.21.15.28:0/327393898 shutdown_connections 2024-02-21T23:05:12.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.806+0000 7fa962c00640 1 -- 172.21.15.28:0/327393898 wait complete. 2024-02-21T23:05:12.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.808+0000 7fa962c00640 1 Processor -- start 2024-02-21T23:05:12.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.809+0000 7fa962c00640 1 -- start start 2024-02-21T23:05:12.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.810+0000 7fa962c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa95c06dbf0 0x7fa95c0c1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:12.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.810+0000 7fa962c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa95c0aff90 0x7fa95c0c0230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:12.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.810+0000 7fa962c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa95c155960 0x7fa95c0bf540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:12.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.810+0000 7fa962c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa95c0c58f0 con 0x7fa95c155960 2024-02-21T23:05:12.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.810+0000 7fa962c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa95c158630 con 0x7fa95c0aff90 2024-02-21T23:05:12.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.810+0000 7fa962c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa95c0c5a70 con 0x7fa95c06dbf0 2024-02-21T23:05:12.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.810+0000 7fa962200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa95c155960 0x7fa95c0bf540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:12.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.811+0000 7fa962200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa95c155960 0x7fa95c0bf540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36228/0 (socket says 172.21.15.28:36228) 2024-02-21T23:05:12.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.811+0000 7fa962200640 1 -- 172.21.15.28:0/487316919 learned_addr learned my addr 172.21.15.28:0/487316919 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:12.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.811+0000 7fa960e00640 1 --2- 172.21.15.28:0/487316919 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa95c0aff90 0x7fa95c0c0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:12.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.811+0000 7fa961800640 1 --2- 172.21.15.28:0/487316919 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa95c06dbf0 0x7fa95c0c1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:12.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.811+0000 7fa962200640 1 -- 172.21.15.28:0/487316919 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa95c06dbf0 msgr2=0x7fa95c0c1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:12.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.811+0000 7fa962200640 1 --2- 172.21.15.28:0/487316919 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa95c06dbf0 0x7fa95c0c1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:12.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.811+0000 7fa962200640 1 -- 172.21.15.28:0/487316919 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa95c0aff90 msgr2=0x7fa95c0c0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:12.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.811+0000 7fa962200640 1 --2- 172.21.15.28:0/487316919 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa95c0aff90 0x7fa95c0c0230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:12.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.811+0000 7fa962200640 1 -- 172.21.15.28:0/487316919 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa95c155430 con 0x7fa95c155960 2024-02-21T23:05:12.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.811+0000 7fa961800640 1 --2- 172.21.15.28:0/487316919 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa95c06dbf0 0x7fa95c0c1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:05:12.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.811+0000 7fa962200640 1 --2- 172.21.15.28:0/487316919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa95c155960 0x7fa95c0bf540 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fa95400b9a0 tx=0x7fa95400be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:12.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.812+0000 7fa959600640 1 -- 172.21.15.28:0/487316919 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa954019070 con 0x7fa95c155960 2024-02-21T23:05:12.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.812+0000 7fa959600640 1 -- 172.21.15.28:0/487316919 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa95400cba0 con 0x7fa95c155960 2024-02-21T23:05:12.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.812+0000 7fa959600640 1 -- 172.21.15.28:0/487316919 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa954014440 con 0x7fa95c155960 2024-02-21T23:05:12.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.812+0000 7fa962c00640 1 -- 172.21.15.28:0/487316919 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa95c0be4a0 con 0x7fa95c155960 2024-02-21T23:05:12.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.812+0000 7fa962c00640 1 -- 172.21.15.28:0/487316919 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa95c0bea60 con 0x7fa95c155960 2024-02-21T23:05:12.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.813+0000 7fa962c00640 1 -- 172.21.15.28:0/487316919 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa95c04f290 con 0x7fa95c155960 2024-02-21T23:05:12.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.814+0000 7fa959600640 1 -- 172.21.15.28:0/487316919 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa95401c070 con 0x7fa95c155960 2024-02-21T23:05:12.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.815+0000 7fa959600640 1 --2- 172.21.15.28:0/487316919 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa93c06b4d0 0x7fa93c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:12.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.815+0000 7fa959600640 1 -- 172.21.15.28:0/487316919 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6957+0+0 (secure 0 0 0) 0x7fa95408a280 con 0x7fa95c155960 2024-02-21T23:05:12.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.816+0000 7fa961800640 1 --2- 172.21.15.28:0/487316919 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa93c06b4d0 0x7fa93c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:12.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.817+0000 7fa961800640 1 --2- 172.21.15.28:0/487316919 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa93c06b4d0 0x7fa93c06d990 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fa950000f60 tx=0x7fa950009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:12.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:12.820+0000 7fa959600640 1 -- 172.21.15.28:0/487316919 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fa954058c10 con 0x7fa95c155960 2024-02-21T23:05:13.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.099+0000 7fa962c00640 1 -- 172.21.15.28:0/487316919 --> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa95c0af490 con 0x7fa93c06b4d0 2024-02-21T23:05:13.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.140+0000 7fa959600640 1 -- 172.21.15.28:0/487316919 <== mgr.4101 v2:172.21.15.81:6832/1848538577 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+490559 (secure 0 0 0) 0x7fa95c0af490 con 0x7fa93c06b4d0 2024-02-21T23:05:13.136 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:05:13.139 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-21T23:05:13.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.149+0000 7fa962c00640 1 -- 172.21.15.28:0/487316919 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa93c06b4d0 msgr2=0x7fa93c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:13.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.149+0000 7fa962c00640 1 --2- 172.21.15.28:0/487316919 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa93c06b4d0 0x7fa93c06d990 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fa950000f60 tx=0x7fa950009040 comp rx=0 tx=0).stop 2024-02-21T23:05:13.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.149+0000 7fa962c00640 1 -- 172.21.15.28:0/487316919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa95c155960 msgr2=0x7fa95c0bf540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:13.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.149+0000 7fa962c00640 1 --2- 172.21.15.28:0/487316919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa95c155960 0x7fa95c0bf540 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fa95400b9a0 tx=0x7fa95400be60 comp rx=0 tx=0).stop 2024-02-21T23:05:13.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.149+0000 7fa962c00640 1 -- 172.21.15.28:0/487316919 shutdown_connections 2024-02-21T23:05:13.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.149+0000 7fa962c00640 1 --2- 172.21.15.28:0/487316919 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa93c06b4d0 0x7fa93c06d990 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:13.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.149+0000 7fa962c00640 1 --2- 172.21.15.28:0/487316919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa95c155960 0x7fa95c0bf540 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:13.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.149+0000 7fa962c00640 1 --2- 172.21.15.28:0/487316919 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa95c0aff90 0x7fa95c0c0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:13.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.149+0000 7fa962c00640 1 --2- 172.21.15.28:0/487316919 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa95c06dbf0 0x7fa95c0c1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:13.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.149+0000 7fa962c00640 1 -- 172.21.15.28:0/487316919 >> 172.21.15.28:0/487316919 conn(0x7fa95c0ac370 msgr2=0x7fa95c157220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:13.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.149+0000 7fa962c00640 1 -- 172.21.15.28:0/487316919 shutdown_connections 2024-02-21T23:05:13.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.149+0000 7fa962c00640 1 -- 172.21.15.28:0/487316919 wait complete. 2024-02-21T23:05:13.172 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":131,"stamp":"2024-02-21T23:05:12.717191+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595188,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":249,"num_write_kb":2444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":195,"ondisk_log_size":195,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":343,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":238380,"kb_used_data":5544,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754736340,"statfs":{"total":773094113280,"available":772850012160,"internally_reserved":0,"allocated":5677056,"data_stored":3683238,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001912"},"pg_stats":[{"pgid":"10.3d","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.011982+0000","last_change":"2024-02-21T23:05:08.470952+0000","last_active":"2024-02-21T23:05:10.011982+0000","last_peered":"2024-02-21T23:05:10.011982+0000","last_clean":"2024-02-21T23:05:10.011982+0000","last_became_active":"2024-02-21T23:05:08.470663+0000","last_became_peered":"2024-02-21T23:05:08.470663+0000","last_unstale":"2024-02-21T23:05:10.011982+0000","last_undegraded":"2024-02-21T23:05:10.011982+0000","last_fullsized":"2024-02-21T23:05:10.011982+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:15:50.294164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3f","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.443191+0000","last_change":"2024-02-21T23:04:53.986963+0000","last_active":"2024-02-21T23:05:09.443191+0000","last_peered":"2024-02-21T23:05:09.443191+0000","last_clean":"2024-02-21T23:05:09.443191+0000","last_became_active":"2024-02-21T23:04:53.986659+0000","last_became_peered":"2024-02-21T23:04:53.986659+0000","last_unstale":"2024-02-21T23:05:09.443191+0000","last_undegraded":"2024-02-21T23:05:09.443191+0000","last_fullsized":"2024-02-21T23:05:09.443191+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:33:34.319772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.448323+0000","last_change":"2024-02-21T23:05:08.451977+0000","last_active":"2024-02-21T23:05:09.448323+0000","last_peered":"2024-02-21T23:05:09.448323+0000","last_clean":"2024-02-21T23:05:09.448323+0000","last_became_active":"2024-02-21T23:05:08.451635+0000","last_became_peered":"2024-02-21T23:05:08.451635+0000","last_unstale":"2024-02-21T23:05:09.448323+0000","last_undegraded":"2024-02-21T23:05:09.448323+0000","last_fullsized":"2024-02-21T23:05:09.448323+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:43:33.473694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.009309+0000","last_change":"2024-02-21T23:04:53.992488+0000","last_active":"2024-02-21T23:05:10.009309+0000","last_peered":"2024-02-21T23:05:10.009309+0000","last_clean":"2024-02-21T23:05:10.009309+0000","last_became_active":"2024-02-21T23:04:53.992187+0000","last_became_peered":"2024-02-21T23:04:53.992187+0000","last_unstale":"2024-02-21T23:05:10.009309+0000","last_undegraded":"2024-02-21T23:05:10.009309+0000","last_fullsized":"2024-02-21T23:05:10.009309+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:27:08.923372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3f","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.443089+0000","last_change":"2024-02-21T23:05:08.460212+0000","last_active":"2024-02-21T23:05:09.443089+0000","last_peered":"2024-02-21T23:05:09.443089+0000","last_clean":"2024-02-21T23:05:09.443089+0000","last_became_active":"2024-02-21T23:05:08.459526+0000","last_became_peered":"2024-02-21T23:05:08.459526+0000","last_unstale":"2024-02-21T23:05:09.443089+0000","last_undegraded":"2024-02-21T23:05:09.443089+0000","last_fullsized":"2024-02-21T23:05:09.443089+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:21:26.217233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.051410+0000","last_change":"2024-02-21T23:04:54.002212+0000","last_active":"2024-02-21T23:05:10.051410+0000","last_peered":"2024-02-21T23:05:10.051410+0000","last_clean":"2024-02-21T23:05:10.051410+0000","last_became_active":"2024-02-21T23:04:54.001921+0000","last_became_peered":"2024-02-21T23:04:54.001921+0000","last_unstale":"2024-02-21T23:05:10.051410+0000","last_undegraded":"2024-02-21T23:05:10.051410+0000","last_fullsized":"2024-02-21T23:05:10.051410+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:43:19.009080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.442413+0000","last_change":"2024-02-21T23:05:08.450709+0000","last_active":"2024-02-21T23:05:09.442413+0000","last_peered":"2024-02-21T23:05:09.442413+0000","last_clean":"2024-02-21T23:05:09.442413+0000","last_became_active":"2024-02-21T23:05:08.447867+0000","last_became_peered":"2024-02-21T23:05:08.447867+0000","last_unstale":"2024-02-21T23:05:09.442413+0000","last_undegraded":"2024-02-21T23:05:09.442413+0000","last_fullsized":"2024-02-21T23:05:09.442413+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:59:56.223705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.008531+0000","last_change":"2024-02-21T23:04:53.994828+0000","last_active":"2024-02-21T23:05:10.008531+0000","last_peered":"2024-02-21T23:05:10.008531+0000","last_clean":"2024-02-21T23:05:10.008531+0000","last_became_active":"2024-02-21T23:04:53.993942+0000","last_became_peered":"2024-02-21T23:04:53.993942+0000","last_unstale":"2024-02-21T23:05:10.008531+0000","last_undegraded":"2024-02-21T23:05:10.008531+0000","last_fullsized":"2024-02-21T23:05:10.008531+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:56:39.552779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.469413+0000","last_change":"2024-02-21T23:05:08.467524+0000","last_active":"2024-02-21T23:05:09.469413+0000","last_peered":"2024-02-21T23:05:09.469413+0000","last_clean":"2024-02-21T23:05:09.469413+0000","last_became_active":"2024-02-21T23:05:08.467082+0000","last_became_peered":"2024-02-21T23:05:08.467082+0000","last_unstale":"2024-02-21T23:05:09.469413+0000","last_undegraded":"2024-02-21T23:05:09.469413+0000","last_fullsized":"2024-02-21T23:05:09.469413+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:12:18.038010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.446665+0000","last_change":"2024-02-21T23:04:53.984836+0000","last_active":"2024-02-21T23:05:09.446665+0000","last_peered":"2024-02-21T23:05:09.446665+0000","last_clean":"2024-02-21T23:05:09.446665+0000","last_became_active":"2024-02-21T23:04:53.984493+0000","last_became_peered":"2024-02-21T23:04:53.984493+0000","last_unstale":"2024-02-21T23:05:09.446665+0000","last_undegraded":"2024-02-21T23:05:09.446665+0000","last_fullsized":"2024-02-21T23:05:09.446665+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:45:50.638257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.465139+0000","last_change":"2024-02-21T23:05:08.468540+0000","last_active":"2024-02-21T23:05:09.465139+0000","last_peered":"2024-02-21T23:05:09.465139+0000","last_clean":"2024-02-21T23:05:09.465139+0000","last_became_active":"2024-02-21T23:05:08.468230+0000","last_became_peered":"2024-02-21T23:05:08.468230+0000","last_unstale":"2024-02-21T23:05:09.465139+0000","last_undegraded":"2024-02-21T23:05:09.465139+0000","last_fullsized":"2024-02-21T23:05:09.465139+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:33:22.889577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.442333+0000","last_change":"2024-02-21T23:04:53.984765+0000","last_active":"2024-02-21T23:05:09.442333+0000","last_peered":"2024-02-21T23:05:09.442333+0000","last_clean":"2024-02-21T23:05:09.442333+0000","last_became_active":"2024-02-21T23:04:53.984432+0000","last_became_peered":"2024-02-21T23:04:53.984432+0000","last_unstale":"2024-02-21T23:05:09.442333+0000","last_undegraded":"2024-02-21T23:05:09.442333+0000","last_fullsized":"2024-02-21T23:05:09.442333+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:55:04.908945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.114786+0000","last_change":"2024-02-21T23:05:08.467626+0000","last_active":"2024-02-21T23:05:10.114786+0000","last_peered":"2024-02-21T23:05:10.114786+0000","last_clean":"2024-02-21T23:05:10.114786+0000","last_became_active":"2024-02-21T23:05:08.467361+0000","last_became_peered":"2024-02-21T23:05:08.467361+0000","last_unstale":"2024-02-21T23:05:10.114786+0000","last_undegraded":"2024-02-21T23:05:10.114786+0000","last_fullsized":"2024-02-21T23:05:10.114786+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:57:18.857503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.050580+0000","last_change":"2024-02-21T23:04:54.003595+0000","last_active":"2024-02-21T23:05:10.050580+0000","last_peered":"2024-02-21T23:05:10.050580+0000","last_clean":"2024-02-21T23:05:10.050580+0000","last_became_active":"2024-02-21T23:04:54.003085+0000","last_became_peered":"2024-02-21T23:04:54.003085+0000","last_unstale":"2024-02-21T23:05:10.050580+0000","last_undegraded":"2024-02-21T23:05:10.050580+0000","last_fullsized":"2024-02-21T23:05:10.050580+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:20:56.263919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.053192+0000","last_change":"2024-02-21T23:05:08.467952+0000","last_active":"2024-02-21T23:05:10.053192+0000","last_peered":"2024-02-21T23:05:10.053192+0000","last_clean":"2024-02-21T23:05:10.053192+0000","last_became_active":"2024-02-21T23:05:08.467169+0000","last_became_peered":"2024-02-21T23:05:08.467169+0000","last_unstale":"2024-02-21T23:05:10.053192+0000","last_undegraded":"2024-02-21T23:05:10.053192+0000","last_fullsized":"2024-02-21T23:05:10.053192+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:38:14.739157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.443780+0000","last_change":"2024-02-21T23:04:53.991959+0000","last_active":"2024-02-21T23:05:09.443780+0000","last_peered":"2024-02-21T23:05:09.443780+0000","last_clean":"2024-02-21T23:05:09.443780+0000","last_became_active":"2024-02-21T23:04:53.988135+0000","last_became_peered":"2024-02-21T23:04:53.988135+0000","last_unstale":"2024-02-21T23:05:09.443780+0000","last_undegraded":"2024-02-21T23:05:09.443780+0000","last_fullsized":"2024-02-21T23:05:09.443780+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:28:35.805544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.448327+0000","last_change":"2024-02-21T23:05:08.455271+0000","last_active":"2024-02-21T23:05:09.448327+0000","last_peered":"2024-02-21T23:05:09.448327+0000","last_clean":"2024-02-21T23:05:09.448327+0000","last_became_active":"2024-02-21T23:05:08.454744+0000","last_became_peered":"2024-02-21T23:05:08.454744+0000","last_unstale":"2024-02-21T23:05:09.448327+0000","last_undegraded":"2024-02-21T23:05:09.448327+0000","last_fullsized":"2024-02-21T23:05:09.448327+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:23:00.163775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.445324+0000","last_change":"2024-02-21T23:04:53.982813+0000","last_active":"2024-02-21T23:05:09.445324+0000","last_peered":"2024-02-21T23:05:09.445324+0000","last_clean":"2024-02-21T23:05:09.445324+0000","last_became_active":"2024-02-21T23:04:53.982515+0000","last_became_peered":"2024-02-21T23:04:53.982515+0000","last_unstale":"2024-02-21T23:05:09.445324+0000","last_undegraded":"2024-02-21T23:05:09.445324+0000","last_fullsized":"2024-02-21T23:05:09.445324+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:03:03.310447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.465411+0000","last_change":"2024-02-21T23:05:08.466034+0000","last_active":"2024-02-21T23:05:09.465411+0000","last_peered":"2024-02-21T23:05:09.465411+0000","last_clean":"2024-02-21T23:05:09.465411+0000","last_became_active":"2024-02-21T23:05:08.464909+0000","last_became_peered":"2024-02-21T23:05:08.464909+0000","last_unstale":"2024-02-21T23:05:09.465411+0000","last_undegraded":"2024-02-21T23:05:09.465411+0000","last_fullsized":"2024-02-21T23:05:09.465411+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:38:50.302069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.442462+0000","last_change":"2024-02-21T23:04:53.982276+0000","last_active":"2024-02-21T23:05:09.442462+0000","last_peered":"2024-02-21T23:05:09.442462+0000","last_clean":"2024-02-21T23:05:09.442462+0000","last_became_active":"2024-02-21T23:04:53.981735+0000","last_became_peered":"2024-02-21T23:04:53.981735+0000","last_unstale":"2024-02-21T23:05:09.442462+0000","last_undegraded":"2024-02-21T23:05:09.442462+0000","last_fullsized":"2024-02-21T23:05:09.442462+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:23:27.013837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.012316+0000","last_change":"2024-02-21T23:05:08.473567+0000","last_active":"2024-02-21T23:05:10.012316+0000","last_peered":"2024-02-21T23:05:10.012316+0000","last_clean":"2024-02-21T23:05:10.012316+0000","last_became_active":"2024-02-21T23:05:08.473270+0000","last_became_peered":"2024-02-21T23:05:08.473270+0000","last_unstale":"2024-02-21T23:05:10.012316+0000","last_undegraded":"2024-02-21T23:05:10.012316+0000","last_fullsized":"2024-02-21T23:05:10.012316+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:03:07.620481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.470208+0000","last_change":"2024-02-21T23:04:53.992336+0000","last_active":"2024-02-21T23:05:09.470208+0000","last_peered":"2024-02-21T23:05:09.470208+0000","last_clean":"2024-02-21T23:05:09.470208+0000","last_became_active":"2024-02-21T23:04:53.992068+0000","last_became_peered":"2024-02-21T23:04:53.992068+0000","last_unstale":"2024-02-21T23:05:09.470208+0000","last_undegraded":"2024-02-21T23:05:09.470208+0000","last_fullsized":"2024-02-21T23:05:09.470208+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:29:50.076176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.011973+0000","last_change":"2024-02-21T23:05:08.474602+0000","last_active":"2024-02-21T23:05:10.011973+0000","last_peered":"2024-02-21T23:05:10.011973+0000","last_clean":"2024-02-21T23:05:10.011973+0000","last_became_active":"2024-02-21T23:05:08.470591+0000","last_became_peered":"2024-02-21T23:05:08.470591+0000","last_unstale":"2024-02-21T23:05:10.011973+0000","last_undegraded":"2024-02-21T23:05:10.011973+0000","last_fullsized":"2024-02-21T23:05:10.011973+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:15:17.047282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.445260+0000","last_change":"2024-02-21T23:04:53.991879+0000","last_active":"2024-02-21T23:05:09.445260+0000","last_peered":"2024-02-21T23:05:09.445260+0000","last_clean":"2024-02-21T23:05:09.445260+0000","last_became_active":"2024-02-21T23:04:53.985749+0000","last_became_peered":"2024-02-21T23:04:53.985749+0000","last_unstale":"2024-02-21T23:05:09.445260+0000","last_undegraded":"2024-02-21T23:05:09.445260+0000","last_fullsized":"2024-02-21T23:05:09.445260+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:21:11.752620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.010126+0000","last_change":"2024-02-21T23:05:08.478582+0000","last_active":"2024-02-21T23:05:10.010126+0000","last_peered":"2024-02-21T23:05:10.010126+0000","last_clean":"2024-02-21T23:05:10.010126+0000","last_became_active":"2024-02-21T23:05:08.477087+0000","last_became_peered":"2024-02-21T23:05:08.477087+0000","last_unstale":"2024-02-21T23:05:10.010126+0000","last_undegraded":"2024-02-21T23:05:10.010126+0000","last_fullsized":"2024-02-21T23:05:10.010126+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:39:36.687065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.441750+0000","last_change":"2024-02-21T23:04:53.982139+0000","last_active":"2024-02-21T23:05:09.441750+0000","last_peered":"2024-02-21T23:05:09.441750+0000","last_clean":"2024-02-21T23:05:09.441750+0000","last_became_active":"2024-02-21T23:04:53.981800+0000","last_became_peered":"2024-02-21T23:04:53.981800+0000","last_unstale":"2024-02-21T23:05:09.441750+0000","last_undegraded":"2024-02-21T23:05:09.441750+0000","last_fullsized":"2024-02-21T23:05:09.441750+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:16:59.134374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.469302+0000","last_change":"2024-02-21T23:05:08.457170+0000","last_active":"2024-02-21T23:05:09.469302+0000","last_peered":"2024-02-21T23:05:09.469302+0000","last_clean":"2024-02-21T23:05:09.469302+0000","last_became_active":"2024-02-21T23:05:08.456713+0000","last_became_peered":"2024-02-21T23:05:08.456713+0000","last_unstale":"2024-02-21T23:05:09.469302+0000","last_undegraded":"2024-02-21T23:05:09.469302+0000","last_fullsized":"2024-02-21T23:05:09.469302+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:17:30.362340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.050342+0000","last_change":"2024-02-21T23:04:54.003361+0000","last_active":"2024-02-21T23:05:10.050342+0000","last_peered":"2024-02-21T23:05:10.050342+0000","last_clean":"2024-02-21T23:05:10.050342+0000","last_became_active":"2024-02-21T23:04:54.002829+0000","last_became_peered":"2024-02-21T23:04:54.002829+0000","last_unstale":"2024-02-21T23:05:10.050342+0000","last_undegraded":"2024-02-21T23:05:10.050342+0000","last_fullsized":"2024-02-21T23:05:10.050342+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:20:06.730366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.465066+0000","last_change":"2024-02-21T23:05:08.472180+0000","last_active":"2024-02-21T23:05:09.465066+0000","last_peered":"2024-02-21T23:05:09.465066+0000","last_clean":"2024-02-21T23:05:09.465066+0000","last_became_active":"2024-02-21T23:05:08.471879+0000","last_became_peered":"2024-02-21T23:05:08.471879+0000","last_unstale":"2024-02-21T23:05:09.465066+0000","last_undegraded":"2024-02-21T23:05:09.465066+0000","last_fullsized":"2024-02-21T23:05:09.465066+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:15:50.294164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.010612+0000","last_change":"2024-02-21T23:04:53.981560+0000","last_active":"2024-02-21T23:05:10.010612+0000","last_peered":"2024-02-21T23:05:10.010612+0000","last_clean":"2024-02-21T23:05:10.010612+0000","last_became_active":"2024-02-21T23:04:53.981260+0000","last_became_peered":"2024-02-21T23:04:53.981260+0000","last_unstale":"2024-02-21T23:05:10.010612+0000","last_undegraded":"2024-02-21T23:05:10.010612+0000","last_fullsized":"2024-02-21T23:05:10.010612+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:04:13.645498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.010185+0000","last_change":"2024-02-21T23:05:08.469030+0000","last_active":"2024-02-21T23:05:10.010185+0000","last_peered":"2024-02-21T23:05:10.010185+0000","last_clean":"2024-02-21T23:05:10.010185+0000","last_became_active":"2024-02-21T23:05:08.468718+0000","last_became_peered":"2024-02-21T23:05:08.468718+0000","last_unstale":"2024-02-21T23:05:10.010185+0000","last_undegraded":"2024-02-21T23:05:10.010185+0000","last_fullsized":"2024-02-21T23:05:10.010185+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:39:49.442464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.441458+0000","last_change":"2024-02-21T23:04:53.981072+0000","last_active":"2024-02-21T23:05:09.441458+0000","last_peered":"2024-02-21T23:05:09.441458+0000","last_clean":"2024-02-21T23:05:09.441458+0000","last_became_active":"2024-02-21T23:04:53.979351+0000","last_became_peered":"2024-02-21T23:04:53.979351+0000","last_unstale":"2024-02-21T23:05:09.441458+0000","last_undegraded":"2024-02-21T23:05:09.441458+0000","last_fullsized":"2024-02-21T23:05:09.441458+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:55:24.090577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.115460+0000","last_change":"2024-02-21T23:05:08.452430+0000","last_active":"2024-02-21T23:05:10.115460+0000","last_peered":"2024-02-21T23:05:10.115460+0000","last_clean":"2024-02-21T23:05:10.115460+0000","last_became_active":"2024-02-21T23:05:08.451516+0000","last_became_peered":"2024-02-21T23:05:08.451516+0000","last_unstale":"2024-02-21T23:05:10.115460+0000","last_undegraded":"2024-02-21T23:05:10.115460+0000","last_fullsized":"2024-02-21T23:05:10.115460+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:38:01.955930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.051802+0000","last_change":"2024-02-21T23:04:53.990161+0000","last_active":"2024-02-21T23:05:10.051802+0000","last_peered":"2024-02-21T23:05:10.051802+0000","last_clean":"2024-02-21T23:05:10.051802+0000","last_became_active":"2024-02-21T23:04:53.989849+0000","last_became_peered":"2024-02-21T23:04:53.989849+0000","last_unstale":"2024-02-21T23:05:10.051802+0000","last_undegraded":"2024-02-21T23:05:10.051802+0000","last_fullsized":"2024-02-21T23:05:10.051802+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:57:04.392889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.444824+0000","last_change":"2024-02-21T23:05:08.457917+0000","last_active":"2024-02-21T23:05:09.444824+0000","last_peered":"2024-02-21T23:05:09.444824+0000","last_clean":"2024-02-21T23:05:09.444824+0000","last_became_active":"2024-02-21T23:05:08.457573+0000","last_became_peered":"2024-02-21T23:05:08.457573+0000","last_unstale":"2024-02-21T23:05:09.444824+0000","last_undegraded":"2024-02-21T23:05:09.444824+0000","last_fullsized":"2024-02-21T23:05:09.444824+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:26:18.251950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.008867+0000","last_change":"2024-02-21T23:04:53.992512+0000","last_active":"2024-02-21T23:05:10.008867+0000","last_peered":"2024-02-21T23:05:10.008867+0000","last_clean":"2024-02-21T23:05:10.008867+0000","last_became_active":"2024-02-21T23:04:53.992255+0000","last_became_peered":"2024-02-21T23:04:53.992255+0000","last_unstale":"2024-02-21T23:05:10.008867+0000","last_undegraded":"2024-02-21T23:05:10.008867+0000","last_fullsized":"2024-02-21T23:05:10.008867+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:20:56.263919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.443058+0000","last_change":"2024-02-21T23:05:08.457071+0000","last_active":"2024-02-21T23:05:09.443058+0000","last_peered":"2024-02-21T23:05:09.443058+0000","last_clean":"2024-02-21T23:05:09.443058+0000","last_became_active":"2024-02-21T23:05:08.456811+0000","last_became_peered":"2024-02-21T23:05:08.456811+0000","last_unstale":"2024-02-21T23:05:09.443058+0000","last_undegraded":"2024-02-21T23:05:09.443058+0000","last_fullsized":"2024-02-21T23:05:09.443058+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:19:02.249738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.051837+0000","last_change":"2024-02-21T23:04:53.985272+0000","last_active":"2024-02-21T23:05:10.051837+0000","last_peered":"2024-02-21T23:05:10.051837+0000","last_clean":"2024-02-21T23:05:10.051837+0000","last_became_active":"2024-02-21T23:04:53.984967+0000","last_became_peered":"2024-02-21T23:04:53.984967+0000","last_unstale":"2024-02-21T23:05:10.051837+0000","last_undegraded":"2024-02-21T23:05:10.051837+0000","last_fullsized":"2024-02-21T23:05:10.051837+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:10:20.341856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.448291+0000","last_change":"2024-02-21T23:05:08.450209+0000","last_active":"2024-02-21T23:05:09.448291+0000","last_peered":"2024-02-21T23:05:09.448291+0000","last_clean":"2024-02-21T23:05:09.448291+0000","last_became_active":"2024-02-21T23:05:08.449874+0000","last_became_peered":"2024-02-21T23:05:08.449874+0000","last_unstale":"2024-02-21T23:05:09.448291+0000","last_undegraded":"2024-02-21T23:05:09.448291+0000","last_fullsized":"2024-02-21T23:05:09.448291+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:10:34.806470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.113487+0000","last_change":"2024-02-21T23:04:53.985751+0000","last_active":"2024-02-21T23:05:10.113487+0000","last_peered":"2024-02-21T23:05:10.113487+0000","last_clean":"2024-02-21T23:05:10.113487+0000","last_became_active":"2024-02-21T23:04:53.983212+0000","last_became_peered":"2024-02-21T23:04:53.983212+0000","last_unstale":"2024-02-21T23:05:10.113487+0000","last_undegraded":"2024-02-21T23:05:10.113487+0000","last_fullsized":"2024-02-21T23:05:10.113487+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:16:59.134374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.443981+0000","last_change":"2024-02-21T23:05:08.465137+0000","last_active":"2024-02-21T23:05:09.443981+0000","last_peered":"2024-02-21T23:05:09.443981+0000","last_clean":"2024-02-21T23:05:09.443981+0000","last_became_active":"2024-02-21T23:05:08.464713+0000","last_became_peered":"2024-02-21T23:05:08.464713+0000","last_unstale":"2024-02-21T23:05:09.443981+0000","last_undegraded":"2024-02-21T23:05:09.443981+0000","last_fullsized":"2024-02-21T23:05:09.443981+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:33:31.480704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.115069+0000","last_change":"2024-02-21T23:04:53.984599+0000","last_active":"2024-02-21T23:05:10.115069+0000","last_peered":"2024-02-21T23:05:10.115069+0000","last_clean":"2024-02-21T23:05:10.115069+0000","last_became_active":"2024-02-21T23:04:53.983405+0000","last_became_peered":"2024-02-21T23:04:53.983405+0000","last_unstale":"2024-02-21T23:05:10.115069+0000","last_undegraded":"2024-02-21T23:05:10.115069+0000","last_fullsized":"2024-02-21T23:05:10.115069+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:59:35.227750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.009702+0000","last_change":"2024-02-21T23:05:08.470094+0000","last_active":"2024-02-21T23:05:10.009702+0000","last_peered":"2024-02-21T23:05:10.009702+0000","last_clean":"2024-02-21T23:05:10.009702+0000","last_became_active":"2024-02-21T23:05:08.469832+0000","last_became_peered":"2024-02-21T23:05:08.469832+0000","last_unstale":"2024-02-21T23:05:10.009702+0000","last_undegraded":"2024-02-21T23:05:10.009702+0000","last_fullsized":"2024-02-21T23:05:10.009702+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:44:33.158628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.444550+0000","last_change":"2024-02-21T23:04:53.988824+0000","last_active":"2024-02-21T23:05:09.444550+0000","last_peered":"2024-02-21T23:05:09.444550+0000","last_clean":"2024-02-21T23:05:09.444550+0000","last_became_active":"2024-02-21T23:04:53.988190+0000","last_became_peered":"2024-02-21T23:04:53.988190+0000","last_unstale":"2024-02-21T23:05:09.444550+0000","last_undegraded":"2024-02-21T23:05:09.444550+0000","last_fullsized":"2024-02-21T23:05:09.444550+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:59:41.759091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.465354+0000","last_change":"2024-02-21T23:05:08.462083+0000","last_active":"2024-02-21T23:05:09.465354+0000","last_peered":"2024-02-21T23:05:09.465354+0000","last_clean":"2024-02-21T23:05:09.465354+0000","last_became_active":"2024-02-21T23:05:08.461812+0000","last_became_peered":"2024-02-21T23:05:08.461812+0000","last_unstale":"2024-02-21T23:05:09.465354+0000","last_undegraded":"2024-02-21T23:05:09.465354+0000","last_fullsized":"2024-02-21T23:05:09.465354+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:38:14.739157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.469558+0000","last_change":"2024-02-21T23:04:53.989836+0000","last_active":"2024-02-21T23:05:09.469558+0000","last_peered":"2024-02-21T23:05:09.469558+0000","last_clean":"2024-02-21T23:05:09.469558+0000","last_became_active":"2024-02-21T23:04:53.989511+0000","last_became_peered":"2024-02-21T23:04:53.989511+0000","last_unstale":"2024-02-21T23:05:09.469558+0000","last_undegraded":"2024-02-21T23:05:09.469558+0000","last_fullsized":"2024-02-21T23:05:09.469558+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:27:55.324492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.444355+0000","last_change":"2024-02-21T23:05:08.471477+0000","last_active":"2024-02-21T23:05:09.444355+0000","last_peered":"2024-02-21T23:05:09.444355+0000","last_clean":"2024-02-21T23:05:09.444355+0000","last_became_active":"2024-02-21T23:05:08.470891+0000","last_became_peered":"2024-02-21T23:05:08.470891+0000","last_unstale":"2024-02-21T23:05:09.444355+0000","last_undegraded":"2024-02-21T23:05:09.444355+0000","last_fullsized":"2024-02-21T23:05:09.444355+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:58:12.890553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.051035+0000","last_change":"2024-02-21T23:04:53.980319+0000","last_active":"2024-02-21T23:05:10.051035+0000","last_peered":"2024-02-21T23:05:10.051035+0000","last_clean":"2024-02-21T23:05:10.051035+0000","last_became_active":"2024-02-21T23:04:53.980056+0000","last_became_peered":"2024-02-21T23:04:53.980056+0000","last_unstale":"2024-02-21T23:05:10.051035+0000","last_undegraded":"2024-02-21T23:05:10.051035+0000","last_fullsized":"2024-02-21T23:05:10.051035+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:27:08.923372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.447758+0000","last_change":"2024-02-21T23:05:08.450459+0000","last_active":"2024-02-21T23:05:09.447758+0000","last_peered":"2024-02-21T23:05:09.447758+0000","last_clean":"2024-02-21T23:05:09.447758+0000","last_became_active":"2024-02-21T23:05:08.450099+0000","last_became_peered":"2024-02-21T23:05:08.450099+0000","last_unstale":"2024-02-21T23:05:09.447758+0000","last_undegraded":"2024-02-21T23:05:09.447758+0000","last_fullsized":"2024-02-21T23:05:09.447758+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:27:31.838991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.050787+0000","last_change":"2024-02-21T23:04:53.994043+0000","last_active":"2024-02-21T23:05:10.050787+0000","last_peered":"2024-02-21T23:05:10.050787+0000","last_clean":"2024-02-21T23:05:10.050787+0000","last_became_active":"2024-02-21T23:04:53.993693+0000","last_became_peered":"2024-02-21T23:04:53.993693+0000","last_unstale":"2024-02-21T23:05:10.050787+0000","last_undegraded":"2024-02-21T23:05:10.050787+0000","last_fullsized":"2024-02-21T23:05:10.050787+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:33:53.507158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.442036+0000","last_change":"2024-02-21T23:05:08.453713+0000","last_active":"2024-02-21T23:05:09.442036+0000","last_peered":"2024-02-21T23:05:09.442036+0000","last_clean":"2024-02-21T23:05:09.442036+0000","last_became_active":"2024-02-21T23:05:08.453119+0000","last_became_peered":"2024-02-21T23:05:08.453119+0000","last_unstale":"2024-02-21T23:05:09.442036+0000","last_undegraded":"2024-02-21T23:05:09.442036+0000","last_fullsized":"2024-02-21T23:05:09.442036+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:47:50.018715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.445216+0000","last_change":"2024-02-21T23:04:53.994446+0000","last_active":"2024-02-21T23:05:09.445216+0000","last_peered":"2024-02-21T23:05:09.445216+0000","last_clean":"2024-02-21T23:05:09.445216+0000","last_became_active":"2024-02-21T23:04:53.994181+0000","last_became_peered":"2024-02-21T23:04:53.994181+0000","last_unstale":"2024-02-21T23:05:09.445216+0000","last_undegraded":"2024-02-21T23:05:09.445216+0000","last_fullsized":"2024-02-21T23:05:09.445216+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:18:47.785124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.465659+0000","last_change":"2024-02-21T23:05:08.460708+0000","last_active":"2024-02-21T23:05:09.465659+0000","last_peered":"2024-02-21T23:05:09.465659+0000","last_clean":"2024-02-21T23:05:09.465659+0000","last_became_active":"2024-02-21T23:05:08.460363+0000","last_became_peered":"2024-02-21T23:05:08.460363+0000","last_unstale":"2024-02-21T23:05:09.465659+0000","last_undegraded":"2024-02-21T23:05:09.465659+0000","last_fullsized":"2024-02-21T23:05:09.465659+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:56:07.508974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.442442+0000","last_change":"2024-02-21T23:04:53.976184+0000","last_active":"2024-02-21T23:05:09.442442+0000","last_peered":"2024-02-21T23:05:09.442442+0000","last_clean":"2024-02-21T23:05:09.442442+0000","last_became_active":"2024-02-21T23:04:53.975098+0000","last_became_peered":"2024-02-21T23:04:53.975098+0000","last_unstale":"2024-02-21T23:05:09.442442+0000","last_undegraded":"2024-02-21T23:05:09.442442+0000","last_fullsized":"2024-02-21T23:05:09.442442+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:39:16.498706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.465310+0000","last_change":"2024-02-21T23:05:08.459982+0000","last_active":"2024-02-21T23:05:09.465310+0000","last_peered":"2024-02-21T23:05:09.465310+0000","last_clean":"2024-02-21T23:05:09.465310+0000","last_became_active":"2024-02-21T23:05:08.459705+0000","last_became_peered":"2024-02-21T23:05:08.459705+0000","last_unstale":"2024-02-21T23:05:09.465310+0000","last_undegraded":"2024-02-21T23:05:09.465310+0000","last_fullsized":"2024-02-21T23:05:09.465310+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:03:07.620481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.113482+0000","last_change":"2024-02-21T23:04:53.995281+0000","last_active":"2024-02-21T23:05:10.113482+0000","last_peered":"2024-02-21T23:05:10.113482+0000","last_clean":"2024-02-21T23:05:10.113482+0000","last_became_active":"2024-02-21T23:04:53.994940+0000","last_became_peered":"2024-02-21T23:04:53.994940+0000","last_unstale":"2024-02-21T23:05:10.113482+0000","last_undegraded":"2024-02-21T23:05:10.113482+0000","last_fullsized":"2024-02-21T23:05:10.113482+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:55:04.908945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.445670+0000","last_change":"2024-02-21T23:05:08.446116+0000","last_active":"2024-02-21T23:05:09.445670+0000","last_peered":"2024-02-21T23:05:09.445670+0000","last_clean":"2024-02-21T23:05:09.445670+0000","last_became_active":"2024-02-21T23:05:08.445654+0000","last_became_peered":"2024-02-21T23:05:08.445654+0000","last_unstale":"2024-02-21T23:05:09.445670+0000","last_undegraded":"2024-02-21T23:05:09.445670+0000","last_fullsized":"2024-02-21T23:05:09.445670+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:52:57.967157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.052812+0000","last_change":"2024-02-21T23:04:53.972067+0000","last_active":"2024-02-21T23:05:10.052812+0000","last_peered":"2024-02-21T23:05:10.052812+0000","last_clean":"2024-02-21T23:05:10.052812+0000","last_became_active":"2024-02-21T23:04:53.971855+0000","last_became_peered":"2024-02-21T23:04:53.971855+0000","last_unstale":"2024-02-21T23:05:10.052812+0000","last_undegraded":"2024-02-21T23:05:10.052812+0000","last_fullsized":"2024-02-21T23:05:10.052812+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:40:20.534202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.051308+0000","last_change":"2024-02-21T23:05:08.447052+0000","last_active":"2024-02-21T23:05:10.051308+0000","last_peered":"2024-02-21T23:05:10.051308+0000","last_clean":"2024-02-21T23:05:10.051308+0000","last_became_active":"2024-02-21T23:05:08.446685+0000","last_became_peered":"2024-02-21T23:05:08.446685+0000","last_unstale":"2024-02-21T23:05:10.051308+0000","last_undegraded":"2024-02-21T23:05:10.051308+0000","last_fullsized":"2024-02-21T23:05:10.051308+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:33:22.889577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.008506+0000","last_change":"2024-02-21T23:04:53.983537+0000","last_active":"2024-02-21T23:05:10.008506+0000","last_peered":"2024-02-21T23:05:10.008506+0000","last_clean":"2024-02-21T23:05:10.008506+0000","last_became_active":"2024-02-21T23:04:53.983264+0000","last_became_peered":"2024-02-21T23:04:53.983264+0000","last_unstale":"2024-02-21T23:05:10.008506+0000","last_undegraded":"2024-02-21T23:05:10.008506+0000","last_fullsized":"2024-02-21T23:05:10.008506+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:36:35.426082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.011463+0000","last_change":"2024-02-21T23:05:08.475581+0000","last_active":"2024-02-21T23:05:10.011463+0000","last_peered":"2024-02-21T23:05:10.011463+0000","last_clean":"2024-02-21T23:05:10.011463+0000","last_became_active":"2024-02-21T23:05:08.475278+0000","last_became_peered":"2024-02-21T23:05:08.475278+0000","last_unstale":"2024-02-21T23:05:10.011463+0000","last_undegraded":"2024-02-21T23:05:10.011463+0000","last_fullsized":"2024-02-21T23:05:10.011463+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:00:36.493162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.050307+0000","last_change":"2024-02-21T23:04:54.002341+0000","last_active":"2024-02-21T23:05:10.050307+0000","last_peered":"2024-02-21T23:05:10.050307+0000","last_clean":"2024-02-21T23:05:10.050307+0000","last_became_active":"2024-02-21T23:04:54.002066+0000","last_became_peered":"2024-02-21T23:04:54.002066+0000","last_unstale":"2024-02-21T23:05:10.050307+0000","last_undegraded":"2024-02-21T23:05:10.050307+0000","last_fullsized":"2024-02-21T23:05:10.050307+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:51:12.681873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.465004+0000","last_change":"2024-02-21T23:05:08.478547+0000","last_active":"2024-02-21T23:05:09.465004+0000","last_peered":"2024-02-21T23:05:09.465004+0000","last_clean":"2024-02-21T23:05:09.465004+0000","last_became_active":"2024-02-21T23:05:08.478086+0000","last_became_peered":"2024-02-21T23:05:08.478086+0000","last_unstale":"2024-02-21T23:05:09.465004+0000","last_undegraded":"2024-02-21T23:05:09.465004+0000","last_fullsized":"2024-02-21T23:05:09.465004+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:15:17.047282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.465167+0000","last_change":"2024-02-21T23:04:53.996395+0000","last_active":"2024-02-21T23:05:09.465167+0000","last_peered":"2024-02-21T23:05:09.465167+0000","last_clean":"2024-02-21T23:05:09.465167+0000","last_became_active":"2024-02-21T23:04:53.996219+0000","last_became_peered":"2024-02-21T23:04:53.996219+0000","last_unstale":"2024-02-21T23:05:09.465167+0000","last_undegraded":"2024-02-21T23:05:09.465167+0000","last_fullsized":"2024-02-21T23:05:09.465167+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:27:17.374378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.114311+0000","last_change":"2024-02-21T23:05:08.456218+0000","last_active":"2024-02-21T23:05:10.114311+0000","last_peered":"2024-02-21T23:05:10.114311+0000","last_clean":"2024-02-21T23:05:10.114311+0000","last_became_active":"2024-02-21T23:05:08.455146+0000","last_became_peered":"2024-02-21T23:05:08.455146+0000","last_unstale":"2024-02-21T23:05:10.114311+0000","last_undegraded":"2024-02-21T23:05:10.114311+0000","last_fullsized":"2024-02-21T23:05:10.114311+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:53:10.262900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.009658+0000","last_change":"2024-02-21T23:05:07.437412+0000","last_active":"2024-02-21T23:05:10.009658+0000","last_peered":"2024-02-21T23:05:10.009658+0000","last_clean":"2024-02-21T23:05:10.009658+0000","last_became_active":"2024-02-21T23:05:07.437123+0000","last_became_peered":"2024-02-21T23:05:07.437123+0000","last_unstale":"2024-02-21T23:05:10.009658+0000","last_undegraded":"2024-02-21T23:05:10.009658+0000","last_fullsized":"2024-02-21T23:05:10.009658+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_clean_scrub_stamp":"2024-02-21T23:05:06.380814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:59:55.210541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.008430+0000","last_change":"2024-02-21T23:04:53.991752+0000","last_active":"2024-02-21T23:05:10.008430+0000","last_peered":"2024-02-21T23:05:10.008430+0000","last_clean":"2024-02-21T23:05:10.008430+0000","last_became_active":"2024-02-21T23:04:53.991458+0000","last_became_peered":"2024-02-21T23:04:53.991458+0000","last_unstale":"2024-02-21T23:05:10.008430+0000","last_undegraded":"2024-02-21T23:05:10.008430+0000","last_fullsized":"2024-02-21T23:05:10.008430+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:33:34.319772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"52'4","reported_seq":26,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.115478+0000","last_change":"2024-02-21T23:04:52.982273+0000","last_active":"2024-02-21T23:05:10.115478+0000","last_peered":"2024-02-21T23:05:10.115478+0000","last_clean":"2024-02-21T23:05:10.115478+0000","last_became_active":"2024-02-21T23:04:52.981799+0000","last_became_peered":"2024-02-21T23:04:52.981799+0000","last_unstale":"2024-02-21T23:05:10.115478+0000","last_undegraded":"2024-02-21T23:05:10.115478+0000","last_fullsized":"2024-02-21T23:05:10.115478+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:18:03.518976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.443242+0000","last_change":"2024-02-21T23:05:08.474827+0000","last_active":"2024-02-21T23:05:09.443242+0000","last_peered":"2024-02-21T23:05:09.443242+0000","last_clean":"2024-02-21T23:05:09.443242+0000","last_became_active":"2024-02-21T23:05:08.474160+0000","last_became_peered":"2024-02-21T23:05:08.474160+0000","last_unstale":"2024-02-21T23:05:09.443242+0000","last_undegraded":"2024-02-21T23:05:09.443242+0000","last_fullsized":"2024-02-21T23:05:09.443242+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:05:35.219005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"56'8","reported_seq":22,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.445359+0000","last_change":"2024-02-21T23:05:07.439772+0000","last_active":"2024-02-21T23:05:09.445359+0000","last_peered":"2024-02-21T23:05:09.445359+0000","last_clean":"2024-02-21T23:05:09.445359+0000","last_became_active":"2024-02-21T23:05:07.439495+0000","last_became_peered":"2024-02-21T23:05:07.439495+0000","last_unstale":"2024-02-21T23:05:09.445359+0000","last_undegraded":"2024-02-21T23:05:09.445359+0000","last_fullsized":"2024-02-21T23:05:09.445359+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_clean_scrub_stamp":"2024-02-21T23:05:06.380814+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:46:57.938662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.465114+0000","last_change":"2024-02-21T23:04:53.991059+0000","last_active":"2024-02-21T23:05:09.465114+0000","last_peered":"2024-02-21T23:05:09.465114+0000","last_clean":"2024-02-21T23:05:09.465114+0000","last_became_active":"2024-02-21T23:04:53.990310+0000","last_became_peered":"2024-02-21T23:04:53.990310+0000","last_unstale":"2024-02-21T23:05:09.465114+0000","last_undegraded":"2024-02-21T23:05:09.465114+0000","last_fullsized":"2024-02-21T23:05:09.465114+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:12:19.379347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"52'1","reported_seq":23,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.442553+0000","last_change":"2024-02-21T23:04:52.971612+0000","last_active":"2024-02-21T23:05:09.442553+0000","last_peered":"2024-02-21T23:05:09.442553+0000","last_clean":"2024-02-21T23:05:09.442553+0000","last_became_active":"2024-02-21T23:04:52.971220+0000","last_became_peered":"2024-02-21T23:04:52.971220+0000","last_unstale":"2024-02-21T23:05:09.442553+0000","last_undegraded":"2024-02-21T23:05:09.442553+0000","last_fullsized":"2024-02-21T23:05:09.442553+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:18:03.518976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.445536+0000","last_change":"2024-02-21T23:05:08.468473+0000","last_active":"2024-02-21T23:05:09.445536+0000","last_peered":"2024-02-21T23:05:09.445536+0000","last_clean":"2024-02-21T23:05:09.445536+0000","last_became_active":"2024-02-21T23:05:08.467235+0000","last_became_peered":"2024-02-21T23:05:08.467235+0000","last_unstale":"2024-02-21T23:05:09.445536+0000","last_undegraded":"2024-02-21T23:05:09.445536+0000","last_fullsized":"2024-02-21T23:05:09.445536+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:20:21.194980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"56'2","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.011277+0000","last_change":"2024-02-21T23:05:07.447016+0000","last_active":"2024-02-21T23:05:10.011277+0000","last_peered":"2024-02-21T23:05:10.011277+0000","last_clean":"2024-02-21T23:05:10.011277+0000","last_became_active":"2024-02-21T23:05:07.446706+0000","last_became_peered":"2024-02-21T23:05:07.446706+0000","last_unstale":"2024-02-21T23:05:10.011277+0000","last_undegraded":"2024-02-21T23:05:10.011277+0000","last_fullsized":"2024-02-21T23:05:10.011277+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_clean_scrub_stamp":"2024-02-21T23:05:06.380814+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:28:49.256993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.114423+0000","last_change":"2024-02-21T23:04:53.988422+0000","last_active":"2024-02-21T23:05:10.114423+0000","last_peered":"2024-02-21T23:05:10.114423+0000","last_clean":"2024-02-21T23:05:10.114423+0000","last_became_active":"2024-02-21T23:04:53.987586+0000","last_became_peered":"2024-02-21T23:04:53.987586+0000","last_unstale":"2024-02-21T23:05:10.114423+0000","last_undegraded":"2024-02-21T23:05:10.114423+0000","last_fullsized":"2024-02-21T23:05:10.114423+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:06:08.079155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"52'2","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.442543+0000","last_change":"2024-02-21T23:04:52.983315+0000","last_active":"2024-02-21T23:05:09.442543+0000","last_peered":"2024-02-21T23:05:09.442543+0000","last_clean":"2024-02-21T23:05:09.442543+0000","last_became_active":"2024-02-21T23:04:52.982925+0000","last_became_peered":"2024-02-21T23:04:52.982925+0000","last_unstale":"2024-02-21T23:05:09.442543+0000","last_undegraded":"2024-02-21T23:05:09.442543+0000","last_fullsized":"2024-02-21T23:05:09.442543+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:04:02.374574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.465485+0000","last_change":"2024-02-21T23:05:08.461733+0000","last_active":"2024-02-21T23:05:09.465485+0000","last_peered":"2024-02-21T23:05:09.465485+0000","last_clean":"2024-02-21T23:05:09.465485+0000","last_became_active":"2024-02-21T23:05:08.461410+0000","last_became_peered":"2024-02-21T23:05:08.461410+0000","last_unstale":"2024-02-21T23:05:09.465485+0000","last_undegraded":"2024-02-21T23:05:09.465485+0000","last_fullsized":"2024-02-21T23:05:09.465485+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:25:56.439594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.465471+0000","last_change":"2024-02-21T23:05:07.442679+0000","last_active":"2024-02-21T23:05:09.465471+0000","last_peered":"2024-02-21T23:05:09.465471+0000","last_clean":"2024-02-21T23:05:09.465471+0000","last_became_active":"2024-02-21T23:05:07.442407+0000","last_became_peered":"2024-02-21T23:05:07.442407+0000","last_unstale":"2024-02-21T23:05:09.465471+0000","last_undegraded":"2024-02-21T23:05:09.465471+0000","last_fullsized":"2024-02-21T23:05:09.465471+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_clean_scrub_stamp":"2024-02-21T23:05:06.380814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:21:25.204069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.444794+0000","last_change":"2024-02-21T23:04:53.997493+0000","last_active":"2024-02-21T23:05:09.444794+0000","last_peered":"2024-02-21T23:05:09.444794+0000","last_clean":"2024-02-21T23:05:09.444794+0000","last_became_active":"2024-02-21T23:04:53.997216+0000","last_became_peered":"2024-02-21T23:04:53.997216+0000","last_unstale":"2024-02-21T23:05:09.444794+0000","last_undegraded":"2024-02-21T23:05:09.444794+0000","last_fullsized":"2024-02-21T23:05:09.444794+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:52:32.070580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"52'2","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.465914+0000","last_change":"2024-02-21T23:04:52.972332+0000","last_active":"2024-02-21T23:05:09.465914+0000","last_peered":"2024-02-21T23:05:09.465914+0000","last_clean":"2024-02-21T23:05:09.465914+0000","last_became_active":"2024-02-21T23:04:52.972014+0000","last_became_peered":"2024-02-21T23:04:52.972014+0000","last_unstale":"2024-02-21T23:05:09.465914+0000","last_undegraded":"2024-02-21T23:05:09.465914+0000","last_fullsized":"2024-02-21T23:05:09.465914+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:46:43.465994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.115753+0000","last_change":"2024-02-21T23:05:08.462349+0000","last_active":"2024-02-21T23:05:10.115753+0000","last_peered":"2024-02-21T23:05:10.115753+0000","last_clean":"2024-02-21T23:05:10.115753+0000","last_became_active":"2024-02-21T23:05:08.459568+0000","last_became_peered":"2024-02-21T23:05:08.459568+0000","last_unstale":"2024-02-21T23:05:10.115753+0000","last_undegraded":"2024-02-21T23:05:10.115753+0000","last_fullsized":"2024-02-21T23:05:10.115753+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:19:02.249738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.15","version":"56'3","reported_seq":17,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.012202+0000","last_change":"2024-02-21T23:05:07.440258+0000","last_active":"2024-02-21T23:05:10.012202+0000","last_peered":"2024-02-21T23:05:10.012202+0000","last_clean":"2024-02-21T23:05:10.012202+0000","last_became_active":"2024-02-21T23:05:07.439978+0000","last_became_peered":"2024-02-21T23:05:07.439978+0000","last_unstale":"2024-02-21T23:05:10.012202+0000","last_undegraded":"2024-02-21T23:05:10.012202+0000","last_fullsized":"2024-02-21T23:05:10.012202+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_clean_scrub_stamp":"2024-02-21T23:05:06.380814+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:47:49.005551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.447761+0000","last_change":"2024-02-21T23:04:53.988566+0000","last_active":"2024-02-21T23:05:09.447761+0000","last_peered":"2024-02-21T23:05:09.447761+0000","last_clean":"2024-02-21T23:05:09.447761+0000","last_became_active":"2024-02-21T23:04:53.988271+0000","last_became_peered":"2024-02-21T23:04:53.988271+0000","last_unstale":"2024-02-21T23:05:09.447761+0000","last_undegraded":"2024-02-21T23:05:09.447761+0000","last_fullsized":"2024-02-21T23:05:09.447761+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:59:35.227750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"52'1","reported_seq":23,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.053549+0000","last_change":"2024-02-21T23:04:52.972962+0000","last_active":"2024-02-21T23:05:10.053549+0000","last_peered":"2024-02-21T23:05:10.053549+0000","last_clean":"2024-02-21T23:05:10.053549+0000","last_became_active":"2024-02-21T23:04:52.972642+0000","last_became_peered":"2024-02-21T23:04:52.972642+0000","last_unstale":"2024-02-21T23:05:10.053549+0000","last_undegraded":"2024-02-21T23:05:10.053549+0000","last_fullsized":"2024-02-21T23:05:10.053549+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:02:52.870656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.448579+0000","last_change":"2024-02-21T23:05:08.450521+0000","last_active":"2024-02-21T23:05:09.448579+0000","last_peered":"2024-02-21T23:05:09.448579+0000","last_clean":"2024-02-21T23:05:09.448579+0000","last_became_active":"2024-02-21T23:05:08.450019+0000","last_became_peered":"2024-02-21T23:05:08.450019+0000","last_unstale":"2024-02-21T23:05:09.448579+0000","last_undegraded":"2024-02-21T23:05:09.448579+0000","last_fullsized":"2024-02-21T23:05:09.448579+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:18:12.721687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"56'6","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.442142+0000","last_change":"2024-02-21T23:05:07.436244+0000","last_active":"2024-02-21T23:05:09.442142+0000","last_peered":"2024-02-21T23:05:09.442142+0000","last_clean":"2024-02-21T23:05:09.442142+0000","last_became_active":"2024-02-21T23:05:07.436009+0000","last_became_peered":"2024-02-21T23:05:07.436009+0000","last_unstale":"2024-02-21T23:05:09.442142+0000","last_undegraded":"2024-02-21T23:05:09.442142+0000","last_fullsized":"2024-02-21T23:05:09.442142+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_clean_scrub_stamp":"2024-02-21T23:05:06.380814+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:51:26.133323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.009114+0000","last_change":"2024-02-21T23:04:53.982723+0000","last_active":"2024-02-21T23:05:10.009114+0000","last_peered":"2024-02-21T23:05:10.009114+0000","last_clean":"2024-02-21T23:05:10.009114+0000","last_became_active":"2024-02-21T23:04:53.982447+0000","last_became_peered":"2024-02-21T23:04:53.982447+0000","last_unstale":"2024-02-21T23:05:10.009114+0000","last_undegraded":"2024-02-21T23:05:10.009114+0000","last_fullsized":"2024-02-21T23:05:10.009114+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:48:09.768408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"52'1","reported_seq":23,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.446005+0000","last_change":"2024-02-21T23:04:52.977866+0000","last_active":"2024-02-21T23:05:09.446005+0000","last_peered":"2024-02-21T23:05:09.446005+0000","last_clean":"2024-02-21T23:05:09.446005+0000","last_became_active":"2024-02-21T23:04:52.977477+0000","last_became_peered":"2024-02-21T23:04:52.977477+0000","last_unstale":"2024-02-21T23:05:09.446005+0000","last_undegraded":"2024-02-21T23:05:09.446005+0000","last_fullsized":"2024-02-21T23:05:09.446005+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:25:58.605620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.442114+0000","last_change":"2024-02-21T23:05:08.445605+0000","last_active":"2024-02-21T23:05:09.442114+0000","last_peered":"2024-02-21T23:05:09.442114+0000","last_clean":"2024-02-21T23:05:09.442114+0000","last_became_active":"2024-02-21T23:05:08.445269+0000","last_became_peered":"2024-02-21T23:05:08.445269+0000","last_unstale":"2024-02-21T23:05:09.442114+0000","last_undegraded":"2024-02-21T23:05:09.442114+0000","last_fullsized":"2024-02-21T23:05:09.442114+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:38:01.955930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.012753+0000","last_change":"2024-02-21T23:05:07.442036+0000","last_active":"2024-02-21T23:05:10.012753+0000","last_peered":"2024-02-21T23:05:10.012753+0000","last_clean":"2024-02-21T23:05:10.012753+0000","last_became_active":"2024-02-21T23:05:07.441674+0000","last_became_peered":"2024-02-21T23:05:07.441674+0000","last_unstale":"2024-02-21T23:05:10.012753+0000","last_undegraded":"2024-02-21T23:05:10.012753+0000","last_fullsized":"2024-02-21T23:05:10.012753+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_clean_scrub_stamp":"2024-02-21T23:05:06.380814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:34:06.958608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.445337+0000","last_change":"2024-02-21T23:04:53.983494+0000","last_active":"2024-02-21T23:05:09.445337+0000","last_peered":"2024-02-21T23:05:09.445337+0000","last_clean":"2024-02-21T23:05:09.445337+0000","last_became_active":"2024-02-21T23:04:53.983213+0000","last_became_peered":"2024-02-21T23:04:53.983213+0000","last_unstale":"2024-02-21T23:05:09.445337+0000","last_undegraded":"2024-02-21T23:05:09.445337+0000","last_fullsized":"2024-02-21T23:05:09.445337+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:53:11.354446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":22,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.114211+0000","last_change":"2024-02-21T23:04:52.976990+0000","last_active":"2024-02-21T23:05:10.114211+0000","last_peered":"2024-02-21T23:05:10.114211+0000","last_clean":"2024-02-21T23:05:10.114211+0000","last_became_active":"2024-02-21T23:04:52.975637+0000","last_became_peered":"2024-02-21T23:04:52.975637+0000","last_unstale":"2024-02-21T23:05:10.114211+0000","last_undegraded":"2024-02-21T23:05:10.114211+0000","last_fullsized":"2024-02-21T23:05:10.114211+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:04:02.374574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.445717+0000","last_change":"2024-02-21T23:05:08.455454+0000","last_active":"2024-02-21T23:05:09.445717+0000","last_peered":"2024-02-21T23:05:09.445717+0000","last_clean":"2024-02-21T23:05:09.445717+0000","last_became_active":"2024-02-21T23:05:08.455046+0000","last_became_peered":"2024-02-21T23:05:08.455046+0000","last_unstale":"2024-02-21T23:05:09.445717+0000","last_undegraded":"2024-02-21T23:05:09.445717+0000","last_fullsized":"2024-02-21T23:05:09.445717+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:56:23.271636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.447921+0000","last_change":"2024-02-21T23:05:07.441101+0000","last_active":"2024-02-21T23:05:09.447921+0000","last_peered":"2024-02-21T23:05:09.447921+0000","last_clean":"2024-02-21T23:05:09.447921+0000","last_became_active":"2024-02-21T23:05:07.440782+0000","last_became_peered":"2024-02-21T23:05:07.440782+0000","last_unstale":"2024-02-21T23:05:09.447921+0000","last_undegraded":"2024-02-21T23:05:09.447921+0000","last_fullsized":"2024-02-21T23:05:09.447921+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_clean_scrub_stamp":"2024-02-21T23:05:06.380814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:39:29.950155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.012440+0000","last_change":"2024-02-21T23:04:53.981493+0000","last_active":"2024-02-21T23:05:10.012440+0000","last_peered":"2024-02-21T23:05:10.012440+0000","last_clean":"2024-02-21T23:05:10.012440+0000","last_became_active":"2024-02-21T23:04:53.981109+0000","last_became_peered":"2024-02-21T23:04:53.981109+0000","last_unstale":"2024-02-21T23:05:10.012440+0000","last_undegraded":"2024-02-21T23:05:10.012440+0000","last_fullsized":"2024-02-21T23:05:10.012440+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:09:29.344651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":22,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.052501+0000","last_change":"2024-02-21T23:04:52.971808+0000","last_active":"2024-02-21T23:05:10.052501+0000","last_peered":"2024-02-21T23:05:10.052501+0000","last_clean":"2024-02-21T23:05:10.052501+0000","last_became_active":"2024-02-21T23:04:52.971529+0000","last_became_peered":"2024-02-21T23:04:52.971529+0000","last_unstale":"2024-02-21T23:05:10.052501+0000","last_undegraded":"2024-02-21T23:05:10.052501+0000","last_fullsized":"2024-02-21T23:05:10.052501+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:00:20.232950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.114772+0000","last_change":"2024-02-21T23:05:08.454743+0000","last_active":"2024-02-21T23:05:10.114772+0000","last_peered":"2024-02-21T23:05:10.114772+0000","last_clean":"2024-02-21T23:05:10.114772+0000","last_became_active":"2024-02-21T23:05:08.451327+0000","last_became_peered":"2024-02-21T23:05:08.451327+0000","last_unstale":"2024-02-21T23:05:10.114772+0000","last_undegraded":"2024-02-21T23:05:10.114772+0000","last_fullsized":"2024-02-21T23:05:10.114772+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:34:07.971772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.444192+0000","last_change":"2024-02-21T23:05:07.444788+0000","last_active":"2024-02-21T23:05:09.444192+0000","last_peered":"2024-02-21T23:05:09.444192+0000","last_clean":"2024-02-21T23:05:09.444192+0000","last_became_active":"2024-02-21T23:05:07.444463+0000","last_became_peered":"2024-02-21T23:05:07.444463+0000","last_unstale":"2024-02-21T23:05:09.444192+0000","last_undegraded":"2024-02-21T23:05:09.444192+0000","last_fullsized":"2024-02-21T23:05:09.444192+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_clean_scrub_stamp":"2024-02-21T23:05:06.380814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:56:06.495810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.445696+0000","last_change":"2024-02-21T23:04:53.994186+0000","last_active":"2024-02-21T23:05:09.445696+0000","last_peered":"2024-02-21T23:05:09.445696+0000","last_clean":"2024-02-21T23:05:09.445696+0000","last_became_active":"2024-02-21T23:04:53.993902+0000","last_became_peered":"2024-02-21T23:04:53.993902+0000","last_unstale":"2024-02-21T23:05:09.445696+0000","last_undegraded":"2024-02-21T23:05:09.445696+0000","last_fullsized":"2024-02-21T23:05:09.445696+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:39:11.623072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":22,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.012823+0000","last_change":"2024-02-21T23:04:52.976339+0000","last_active":"2024-02-21T23:05:10.012823+0000","last_peered":"2024-02-21T23:05:10.012823+0000","last_clean":"2024-02-21T23:05:10.012823+0000","last_became_active":"2024-02-21T23:04:52.975958+0000","last_became_peered":"2024-02-21T23:04:52.975958+0000","last_unstale":"2024-02-21T23:05:10.012823+0000","last_undegraded":"2024-02-21T23:05:10.012823+0000","last_fullsized":"2024-02-21T23:05:10.012823+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:06:07.057936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.445687+0000","last_change":"2024-02-21T23:05:08.463501+0000","last_active":"2024-02-21T23:05:09.445687+0000","last_peered":"2024-02-21T23:05:09.445687+0000","last_clean":"2024-02-21T23:05:09.445687+0000","last_became_active":"2024-02-21T23:05:08.463203+0000","last_became_peered":"2024-02-21T23:05:08.463203+0000","last_unstale":"2024-02-21T23:05:09.445687+0000","last_undegraded":"2024-02-21T23:05:09.445687+0000","last_fullsized":"2024-02-21T23:05:09.445687+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:21:49.905786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.113972+0000","last_change":"2024-02-21T23:05:07.439970+0000","last_active":"2024-02-21T23:05:10.113972+0000","last_peered":"2024-02-21T23:05:10.113972+0000","last_clean":"2024-02-21T23:05:10.113972+0000","last_became_active":"2024-02-21T23:05:07.439647+0000","last_became_peered":"2024-02-21T23:05:07.439647+0000","last_unstale":"2024-02-21T23:05:10.113972+0000","last_undegraded":"2024-02-21T23:05:10.113972+0000","last_fullsized":"2024-02-21T23:05:10.113972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_clean_scrub_stamp":"2024-02-21T23:05:06.380814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:36:48.877531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.464986+0000","last_change":"2024-02-21T23:04:53.980747+0000","last_active":"2024-02-21T23:05:09.464986+0000","last_peered":"2024-02-21T23:05:09.464986+0000","last_clean":"2024-02-21T23:05:09.464986+0000","last_became_active":"2024-02-21T23:04:53.980456+0000","last_became_peered":"2024-02-21T23:04:53.980456+0000","last_unstale":"2024-02-21T23:05:09.464986+0000","last_undegraded":"2024-02-21T23:05:09.464986+0000","last_fullsized":"2024-02-21T23:05:09.464986+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:40:20.534202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":22,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.448970+0000","last_change":"2024-02-21T23:04:52.974873+0000","last_active":"2024-02-21T23:05:09.448970+0000","last_peered":"2024-02-21T23:05:09.448970+0000","last_clean":"2024-02-21T23:05:09.448970+0000","last_became_active":"2024-02-21T23:04:52.973750+0000","last_became_peered":"2024-02-21T23:04:52.973750+0000","last_unstale":"2024-02-21T23:05:09.448970+0000","last_undegraded":"2024-02-21T23:05:09.448970+0000","last_fullsized":"2024-02-21T23:05:09.448970+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:04:02.374574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.469108+0000","last_change":"2024-02-21T23:05:08.444476+0000","last_active":"2024-02-21T23:05:09.469108+0000","last_peered":"2024-02-21T23:05:09.469108+0000","last_clean":"2024-02-21T23:05:09.469108+0000","last_became_active":"2024-02-21T23:05:08.444162+0000","last_became_peered":"2024-02-21T23:05:08.444162+0000","last_unstale":"2024-02-21T23:05:09.469108+0000","last_undegraded":"2024-02-21T23:05:09.469108+0000","last_fullsized":"2024-02-21T23:05:09.469108+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:26:14.091452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"56'1","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.444726+0000","last_change":"2024-02-21T23:05:07.428466+0000","last_active":"2024-02-21T23:05:09.444726+0000","last_peered":"2024-02-21T23:05:09.444726+0000","last_clean":"2024-02-21T23:05:09.444726+0000","last_became_active":"2024-02-21T23:05:07.428147+0000","last_became_peered":"2024-02-21T23:05:07.428147+0000","last_unstale":"2024-02-21T23:05:09.444726+0000","last_undegraded":"2024-02-21T23:05:09.444726+0000","last_fullsized":"2024-02-21T23:05:09.444726+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_clean_scrub_stamp":"2024-02-21T23:05:06.380814+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:26:52.157806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.053580+0000","last_change":"2024-02-21T23:04:53.983830+0000","last_active":"2024-02-21T23:05:10.053580+0000","last_peered":"2024-02-21T23:05:10.053580+0000","last_clean":"2024-02-21T23:05:10.053580+0000","last_became_active":"2024-02-21T23:04:53.974390+0000","last_became_peered":"2024-02-21T23:04:53.974390+0000","last_unstale":"2024-02-21T23:05:10.053580+0000","last_undegraded":"2024-02-21T23:05:10.053580+0000","last_fullsized":"2024-02-21T23:05:10.053580+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:27:17.374378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"52'3","reported_seq":25,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.012002+0000","last_change":"2024-02-21T23:04:52.976896+0000","last_active":"2024-02-21T23:05:10.012002+0000","last_peered":"2024-02-21T23:05:10.012002+0000","last_clean":"2024-02-21T23:05:10.012002+0000","last_became_active":"2024-02-21T23:04:52.976592+0000","last_became_peered":"2024-02-21T23:04:52.976592+0000","last_unstale":"2024-02-21T23:05:10.012002+0000","last_undegraded":"2024-02-21T23:05:10.012002+0000","last_fullsized":"2024-02-21T23:05:10.012002+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:39:31.156140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.441915+0000","last_change":"2024-02-21T23:05:08.457742+0000","last_active":"2024-02-21T23:05:09.441915+0000","last_peered":"2024-02-21T23:05:09.441915+0000","last_clean":"2024-02-21T23:05:09.441915+0000","last_became_active":"2024-02-21T23:05:08.457403+0000","last_became_peered":"2024-02-21T23:05:08.457403+0000","last_unstale":"2024-02-21T23:05:09.441915+0000","last_undegraded":"2024-02-21T23:05:09.441915+0000","last_fullsized":"2024-02-21T23:05:09.441915+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:19:27.871458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"56'1","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.444273+0000","last_change":"2024-02-21T23:05:07.433349+0000","last_active":"2024-02-21T23:05:09.444273+0000","last_peered":"2024-02-21T23:05:09.444273+0000","last_clean":"2024-02-21T23:05:09.444273+0000","last_became_active":"2024-02-21T23:05:07.431938+0000","last_became_peered":"2024-02-21T23:05:07.431938+0000","last_unstale":"2024-02-21T23:05:09.444273+0000","last_undegraded":"2024-02-21T23:05:09.444273+0000","last_fullsized":"2024-02-21T23:05:09.444273+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_clean_scrub_stamp":"2024-02-21T23:05:06.380814+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:38:13.725993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.446904+0000","last_change":"2024-02-21T23:04:53.997903+0000","last_active":"2024-02-21T23:05:09.446904+0000","last_peered":"2024-02-21T23:05:09.446904+0000","last_clean":"2024-02-21T23:05:09.446904+0000","last_became_active":"2024-02-21T23:04:53.997538+0000","last_became_peered":"2024-02-21T23:04:53.997538+0000","last_unstale":"2024-02-21T23:05:09.446904+0000","last_undegraded":"2024-02-21T23:05:09.446904+0000","last_fullsized":"2024-02-21T23:05:09.446904+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:55:24.090577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"52'6","reported_seq":28,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.445445+0000","last_change":"2024-02-21T23:04:52.975159+0000","last_active":"2024-02-21T23:05:09.445445+0000","last_peered":"2024-02-21T23:05:09.445445+0000","last_clean":"2024-02-21T23:05:09.445445+0000","last_became_active":"2024-02-21T23:04:52.974845+0000","last_became_peered":"2024-02-21T23:04:52.974845+0000","last_unstale":"2024-02-21T23:05:09.445445+0000","last_undegraded":"2024-02-21T23:05:09.445445+0000","last_fullsized":"2024-02-21T23:05:09.445445+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:20:05.709148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.054750+0000","last_change":"2024-02-21T23:05:08.445427+0000","last_active":"2024-02-21T23:05:10.054750+0000","last_peered":"2024-02-21T23:05:10.054750+0000","last_clean":"2024-02-21T23:05:10.054750+0000","last_became_active":"2024-02-21T23:05:08.445137+0000","last_became_peered":"2024-02-21T23:05:08.445137+0000","last_unstale":"2024-02-21T23:05:10.054750+0000","last_undegraded":"2024-02-21T23:05:10.054750+0000","last_fullsized":"2024-02-21T23:05:10.054750+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:41:33.700197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"56'2","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.053236+0000","last_change":"2024-02-21T23:05:07.435343+0000","last_active":"2024-02-21T23:05:10.053236+0000","last_peered":"2024-02-21T23:05:10.053236+0000","last_clean":"2024-02-21T23:05:10.053236+0000","last_became_active":"2024-02-21T23:05:07.434893+0000","last_became_peered":"2024-02-21T23:05:07.434893+0000","last_unstale":"2024-02-21T23:05:10.053236+0000","last_undegraded":"2024-02-21T23:05:10.053236+0000","last_fullsized":"2024-02-21T23:05:10.053236+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_clean_scrub_stamp":"2024-02-21T23:05:06.380814+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:44:32.145464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.465320+0000","last_change":"2024-02-21T23:04:53.987920+0000","last_active":"2024-02-21T23:05:09.465320+0000","last_peered":"2024-02-21T23:05:09.465320+0000","last_clean":"2024-02-21T23:05:09.465320+0000","last_became_active":"2024-02-21T23:04:53.986231+0000","last_became_peered":"2024-02-21T23:04:53.986231+0000","last_unstale":"2024-02-21T23:05:09.465320+0000","last_undegraded":"2024-02-21T23:05:09.465320+0000","last_fullsized":"2024-02-21T23:05:09.465320+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:20:06.730366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"52'2","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.053912+0000","last_change":"2024-02-21T23:04:52.977012+0000","last_active":"2024-02-21T23:05:10.053912+0000","last_peered":"2024-02-21T23:05:10.053912+0000","last_clean":"2024-02-21T23:05:10.053912+0000","last_became_active":"2024-02-21T23:04:52.973324+0000","last_became_peered":"2024-02-21T23:04:52.973324+0000","last_unstale":"2024-02-21T23:05:10.053912+0000","last_undegraded":"2024-02-21T23:05:10.053912+0000","last_fullsized":"2024-02-21T23:05:10.053912+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:55:27.414683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.012269+0000","last_change":"2024-02-21T23:05:08.468833+0000","last_active":"2024-02-21T23:05:10.012269+0000","last_peered":"2024-02-21T23:05:10.012269+0000","last_clean":"2024-02-21T23:05:10.012269+0000","last_became_active":"2024-02-21T23:05:08.468556+0000","last_became_peered":"2024-02-21T23:05:08.468556+0000","last_unstale":"2024-02-21T23:05:10.012269+0000","last_undegraded":"2024-02-21T23:05:10.012269+0000","last_fullsized":"2024-02-21T23:05:10.012269+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:33:22.889577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"56'2","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.053446+0000","last_change":"2024-02-21T23:05:07.445016+0000","last_active":"2024-02-21T23:05:10.053446+0000","last_peered":"2024-02-21T23:05:10.053446+0000","last_clean":"2024-02-21T23:05:10.053446+0000","last_became_active":"2024-02-21T23:05:07.441365+0000","last_became_peered":"2024-02-21T23:05:07.441365+0000","last_unstale":"2024-02-21T23:05:10.053446+0000","last_undegraded":"2024-02-21T23:05:10.053446+0000","last_fullsized":"2024-02-21T23:05:10.053446+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_clean_scrub_stamp":"2024-02-21T23:05:06.380814+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:52:45.522029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.050492+0000","last_change":"2024-02-21T23:04:53.969380+0000","last_active":"2024-02-21T23:05:10.050492+0000","last_peered":"2024-02-21T23:05:10.050492+0000","last_clean":"2024-02-21T23:05:10.050492+0000","last_became_active":"2024-02-21T23:04:53.966766+0000","last_became_peered":"2024-02-21T23:04:53.966766+0000","last_unstale":"2024-02-21T23:05:10.050492+0000","last_undegraded":"2024-02-21T23:05:10.050492+0000","last_fullsized":"2024-02-21T23:05:10.050492+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:22:45.699161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":22,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.115026+0000","last_change":"2024-02-21T23:04:52.981590+0000","last_active":"2024-02-21T23:05:10.115026+0000","last_peered":"2024-02-21T23:05:10.115026+0000","last_clean":"2024-02-21T23:05:10.115026+0000","last_became_active":"2024-02-21T23:04:52.981228+0000","last_became_peered":"2024-02-21T23:04:52.981228+0000","last_unstale":"2024-02-21T23:05:10.115026+0000","last_undegraded":"2024-02-21T23:05:10.115026+0000","last_fullsized":"2024-02-21T23:05:10.115026+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:25:58.605620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.115044+0000","last_change":"2024-02-21T23:05:08.451167+0000","last_active":"2024-02-21T23:05:10.115044+0000","last_peered":"2024-02-21T23:05:10.115044+0000","last_clean":"2024-02-21T23:05:10.115044+0000","last_became_active":"2024-02-21T23:05:08.450830+0000","last_became_peered":"2024-02-21T23:05:08.450830+0000","last_unstale":"2024-02-21T23:05:10.115044+0000","last_undegraded":"2024-02-21T23:05:10.115044+0000","last_fullsized":"2024-02-21T23:05:10.115044+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:47:50.018715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"56'4","reported_seq":18,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.443181+0000","last_change":"2024-02-21T23:05:07.442657+0000","last_active":"2024-02-21T23:05:09.443181+0000","last_peered":"2024-02-21T23:05:09.443181+0000","last_clean":"2024-02-21T23:05:09.443181+0000","last_became_active":"2024-02-21T23:05:07.442095+0000","last_became_peered":"2024-02-21T23:05:07.442095+0000","last_unstale":"2024-02-21T23:05:09.443181+0000","last_undegraded":"2024-02-21T23:05:09.443181+0000","last_fullsized":"2024-02-21T23:05:09.443181+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_clean_scrub_stamp":"2024-02-21T23:05:06.380814+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:20:20.181815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.445225+0000","last_change":"2024-02-21T23:04:53.983659+0000","last_active":"2024-02-21T23:05:09.445225+0000","last_peered":"2024-02-21T23:05:09.445225+0000","last_clean":"2024-02-21T23:05:09.445225+0000","last_became_active":"2024-02-21T23:04:53.983290+0000","last_became_peered":"2024-02-21T23:04:53.983290+0000","last_unstale":"2024-02-21T23:05:09.445225+0000","last_undegraded":"2024-02-21T23:05:09.445225+0000","last_fullsized":"2024-02-21T23:05:09.445225+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:05:20.754391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":22,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.010892+0000","last_change":"2024-02-21T23:04:52.982825+0000","last_active":"2024-02-21T23:05:10.010892+0000","last_peered":"2024-02-21T23:05:10.010892+0000","last_clean":"2024-02-21T23:05:10.010892+0000","last_became_active":"2024-02-21T23:04:52.982523+0000","last_became_peered":"2024-02-21T23:04:52.982523+0000","last_unstale":"2024-02-21T23:05:10.010892+0000","last_undegraded":"2024-02-21T23:05:10.010892+0000","last_fullsized":"2024-02-21T23:05:10.010892+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:45:49.617039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.448614+0000","last_change":"2024-02-21T23:05:08.455052+0000","last_active":"2024-02-21T23:05:09.448614+0000","last_peered":"2024-02-21T23:05:09.448614+0000","last_clean":"2024-02-21T23:05:09.448614+0000","last_became_active":"2024-02-21T23:05:08.454407+0000","last_became_peered":"2024-02-21T23:05:08.454407+0000","last_unstale":"2024-02-21T23:05:09.448614+0000","last_undegraded":"2024-02-21T23:05:09.448614+0000","last_fullsized":"2024-02-21T23:05:09.448614+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:57:18.857503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"56'1","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.054937+0000","last_change":"2024-02-21T23:05:07.436290+0000","last_active":"2024-02-21T23:05:10.054937+0000","last_peered":"2024-02-21T23:05:10.054937+0000","last_clean":"2024-02-21T23:05:10.054937+0000","last_became_active":"2024-02-21T23:05:07.433578+0000","last_became_peered":"2024-02-21T23:05:07.433578+0000","last_unstale":"2024-02-21T23:05:10.054937+0000","last_undegraded":"2024-02-21T23:05:10.054937+0000","last_fullsized":"2024-02-21T23:05:10.054937+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_clean_scrub_stamp":"2024-02-21T23:05:06.380814+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:17:32.435292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.051104+0000","last_change":"2024-02-21T23:04:53.988578+0000","last_active":"2024-02-21T23:05:10.051104+0000","last_peered":"2024-02-21T23:05:10.051104+0000","last_clean":"2024-02-21T23:05:10.051104+0000","last_became_active":"2024-02-21T23:04:53.988310+0000","last_became_peered":"2024-02-21T23:04:53.988310+0000","last_unstale":"2024-02-21T23:05:10.051104+0000","last_undegraded":"2024-02-21T23:05:10.051104+0000","last_fullsized":"2024-02-21T23:05:10.051104+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:17:58.257074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"52'8","reported_seq":30,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.443890+0000","last_change":"2024-02-21T23:04:52.978259+0000","last_active":"2024-02-21T23:05:09.443890+0000","last_peered":"2024-02-21T23:05:09.443890+0000","last_clean":"2024-02-21T23:05:09.443890+0000","last_became_active":"2024-02-21T23:04:52.975275+0000","last_became_peered":"2024-02-21T23:04:52.975275+0000","last_unstale":"2024-02-21T23:05:09.443890+0000","last_undegraded":"2024-02-21T23:05:09.443890+0000","last_fullsized":"2024-02-21T23:05:09.443890+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:20:55.242700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.442078+0000","last_change":"2024-02-21T23:05:08.459936+0000","last_active":"2024-02-21T23:05:09.442078+0000","last_peered":"2024-02-21T23:05:09.442078+0000","last_clean":"2024-02-21T23:05:09.442078+0000","last_became_active":"2024-02-21T23:05:08.459609+0000","last_became_peered":"2024-02-21T23:05:08.459609+0000","last_unstale":"2024-02-21T23:05:09.442078+0000","last_undegraded":"2024-02-21T23:05:09.442078+0000","last_fullsized":"2024-02-21T23:05:09.442078+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:28:50.270158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.444167+0000","last_change":"2024-02-21T23:05:07.444533+0000","last_active":"2024-02-21T23:05:09.444167+0000","last_peered":"2024-02-21T23:05:09.444167+0000","last_clean":"2024-02-21T23:05:09.444167+0000","last_became_active":"2024-02-21T23:05:07.444202+0000","last_became_peered":"2024-02-21T23:05:07.444202+0000","last_unstale":"2024-02-21T23:05:09.444167+0000","last_undegraded":"2024-02-21T23:05:09.444167+0000","last_fullsized":"2024-02-21T23:05:09.444167+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_clean_scrub_stamp":"2024-02-21T23:05:06.380814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:54:38.014722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.443822+0000","last_change":"2024-02-21T23:04:53.988231+0000","last_active":"2024-02-21T23:05:09.443822+0000","last_peered":"2024-02-21T23:05:09.443822+0000","last_clean":"2024-02-21T23:05:09.443822+0000","last_became_active":"2024-02-21T23:04:53.987814+0000","last_became_peered":"2024-02-21T23:04:53.987814+0000","last_unstale":"2024-02-21T23:05:09.443822+0000","last_undegraded":"2024-02-21T23:05:09.443822+0000","last_fullsized":"2024-02-21T23:05:09.443822+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:37:47.491317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"52'1","reported_seq":23,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.054030+0000","last_change":"2024-02-21T23:04:52.981262+0000","last_active":"2024-02-21T23:05:10.054030+0000","last_peered":"2024-02-21T23:05:10.054030+0000","last_clean":"2024-02-21T23:05:10.054030+0000","last_became_active":"2024-02-21T23:04:52.980981+0000","last_became_peered":"2024-02-21T23:04:52.980981+0000","last_unstale":"2024-02-21T23:05:10.054030+0000","last_undegraded":"2024-02-21T23:05:10.054030+0000","last_fullsized":"2024-02-21T23:05:10.054030+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:45:49.617039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.469741+0000","last_change":"2024-02-21T23:05:08.454574+0000","last_active":"2024-02-21T23:05:09.469741+0000","last_peered":"2024-02-21T23:05:09.469741+0000","last_clean":"2024-02-21T23:05:09.469741+0000","last_became_active":"2024-02-21T23:05:08.454208+0000","last_became_peered":"2024-02-21T23:05:08.454208+0000","last_unstale":"2024-02-21T23:05:09.469741+0000","last_undegraded":"2024-02-21T23:05:09.469741+0000","last_fullsized":"2024-02-21T23:05:09.469741+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:03:08.356487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.441747+0000","last_change":"2024-02-21T23:05:07.441184+0000","last_active":"2024-02-21T23:05:09.441747+0000","last_peered":"2024-02-21T23:05:09.441747+0000","last_clean":"2024-02-21T23:05:09.441747+0000","last_became_active":"2024-02-21T23:05:07.440719+0000","last_became_peered":"2024-02-21T23:05:07.440719+0000","last_unstale":"2024-02-21T23:05:09.441747+0000","last_undegraded":"2024-02-21T23:05:09.441747+0000","last_fullsized":"2024-02-21T23:05:09.441747+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_clean_scrub_stamp":"2024-02-21T23:05:06.380814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:52:56.953993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.441843+0000","last_change":"2024-02-21T23:04:53.967738+0000","last_active":"2024-02-21T23:05:09.441843+0000","last_peered":"2024-02-21T23:05:09.441843+0000","last_clean":"2024-02-21T23:05:09.441843+0000","last_became_active":"2024-02-21T23:04:53.967430+0000","last_became_peered":"2024-02-21T23:04:53.967430+0000","last_unstale":"2024-02-21T23:05:09.441843+0000","last_undegraded":"2024-02-21T23:05:09.441843+0000","last_fullsized":"2024-02-21T23:05:09.441843+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:59:35.227750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"52'2","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.470666+0000","last_change":"2024-02-21T23:04:52.966722+0000","last_active":"2024-02-21T23:05:09.470666+0000","last_peered":"2024-02-21T23:05:09.470666+0000","last_clean":"2024-02-21T23:05:09.470666+0000","last_became_active":"2024-02-21T23:04:52.966322+0000","last_became_peered":"2024-02-21T23:04:52.966322+0000","last_unstale":"2024-02-21T23:05:09.470666+0000","last_undegraded":"2024-02-21T23:05:09.470666+0000","last_fullsized":"2024-02-21T23:05:09.470666+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:29:42.084956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.113890+0000","last_change":"2024-02-21T23:05:08.454805+0000","last_active":"2024-02-21T23:05:10.113890+0000","last_peered":"2024-02-21T23:05:10.113890+0000","last_clean":"2024-02-21T23:05:10.113890+0000","last_became_active":"2024-02-21T23:05:08.454156+0000","last_became_peered":"2024-02-21T23:05:08.454156+0000","last_unstale":"2024-02-21T23:05:10.113890+0000","last_undegraded":"2024-02-21T23:05:10.113890+0000","last_fullsized":"2024-02-21T23:05:10.113890+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:23:00.163775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.011501+0000","last_change":"2024-02-21T23:05:07.438658+0000","last_active":"2024-02-21T23:05:10.011501+0000","last_peered":"2024-02-21T23:05:10.011501+0000","last_clean":"2024-02-21T23:05:10.011501+0000","last_became_active":"2024-02-21T23:05:07.438362+0000","last_became_peered":"2024-02-21T23:05:07.438362+0000","last_unstale":"2024-02-21T23:05:10.011501+0000","last_undegraded":"2024-02-21T23:05:10.011501+0000","last_fullsized":"2024-02-21T23:05:10.011501+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_clean_scrub_stamp":"2024-02-21T23:05:06.380814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:19:26.858293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.445906+0000","last_change":"2024-02-21T23:04:53.989868+0000","last_active":"2024-02-21T23:05:09.445906+0000","last_peered":"2024-02-21T23:05:09.445906+0000","last_clean":"2024-02-21T23:05:09.445906+0000","last_became_active":"2024-02-21T23:04:53.989585+0000","last_became_peered":"2024-02-21T23:04:53.989585+0000","last_unstale":"2024-02-21T23:05:09.445906+0000","last_undegraded":"2024-02-21T23:05:09.445906+0000","last_fullsized":"2024-02-21T23:05:09.445906+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:06:08.079155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"52'1","reported_seq":23,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.115775+0000","last_change":"2024-02-21T23:04:52.976203+0000","last_active":"2024-02-21T23:05:10.115775+0000","last_peered":"2024-02-21T23:05:10.115775+0000","last_clean":"2024-02-21T23:05:10.115775+0000","last_became_active":"2024-02-21T23:04:52.975787+0000","last_became_peered":"2024-02-21T23:04:52.975787+0000","last_unstale":"2024-02-21T23:05:10.115775+0000","last_undegraded":"2024-02-21T23:05:10.115775+0000","last_fullsized":"2024-02-21T23:05:10.115775+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:02:52.870656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.445589+0000","last_change":"2024-02-21T23:05:08.479352+0000","last_active":"2024-02-21T23:05:09.445589+0000","last_peered":"2024-02-21T23:05:09.445589+0000","last_clean":"2024-02-21T23:05:09.445589+0000","last_became_active":"2024-02-21T23:05:08.479057+0000","last_became_peered":"2024-02-21T23:05:08.479057+0000","last_unstale":"2024-02-21T23:05:09.445589+0000","last_undegraded":"2024-02-21T23:05:09.445589+0000","last_fullsized":"2024-02-21T23:05:09.445589+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:12:33.843961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.444387+0000","last_change":"2024-02-21T23:05:07.438855+0000","last_active":"2024-02-21T23:05:09.444387+0000","last_peered":"2024-02-21T23:05:09.444387+0000","last_clean":"2024-02-21T23:05:09.444387+0000","last_became_active":"2024-02-21T23:05:07.438539+0000","last_became_peered":"2024-02-21T23:05:07.438539+0000","last_unstale":"2024-02-21T23:05:09.444387+0000","last_undegraded":"2024-02-21T23:05:09.444387+0000","last_fullsized":"2024-02-21T23:05:09.444387+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_clean_scrub_stamp":"2024-02-21T23:05:06.380814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:38:49.288904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.443113+0000","last_change":"2024-02-21T23:04:53.986757+0000","last_active":"2024-02-21T23:05:09.443113+0000","last_peered":"2024-02-21T23:05:09.443113+0000","last_clean":"2024-02-21T23:05:09.443113+0000","last_became_active":"2024-02-21T23:04:53.982468+0000","last_became_peered":"2024-02-21T23:04:53.982468+0000","last_unstale":"2024-02-21T23:05:09.443113+0000","last_undegraded":"2024-02-21T23:05:09.443113+0000","last_fullsized":"2024-02-21T23:05:09.443113+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:48:17.190667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":22,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.114996+0000","last_change":"2024-02-21T23:04:52.978566+0000","last_active":"2024-02-21T23:05:10.114996+0000","last_peered":"2024-02-21T23:05:10.114996+0000","last_clean":"2024-02-21T23:05:10.114996+0000","last_became_active":"2024-02-21T23:04:52.978262+0000","last_became_peered":"2024-02-21T23:04:52.978262+0000","last_unstale":"2024-02-21T23:05:10.114996+0000","last_undegraded":"2024-02-21T23:05:10.114996+0000","last_fullsized":"2024-02-21T23:05:10.114996+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:12:02.552178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.465501+0000","last_change":"2024-02-21T23:05:08.459858+0000","last_active":"2024-02-21T23:05:09.465501+0000","last_peered":"2024-02-21T23:05:09.465501+0000","last_clean":"2024-02-21T23:05:09.465501+0000","last_became_active":"2024-02-21T23:05:08.459574+0000","last_became_peered":"2024-02-21T23:05:08.459574+0000","last_unstale":"2024-02-21T23:05:09.465501+0000","last_undegraded":"2024-02-21T23:05:09.465501+0000","last_fullsized":"2024-02-21T23:05:09.465501+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:41:33.700197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.053114+0000","last_change":"2024-02-21T23:05:07.435108+0000","last_active":"2024-02-21T23:05:10.053114+0000","last_peered":"2024-02-21T23:05:10.053114+0000","last_clean":"2024-02-21T23:05:10.053114+0000","last_became_active":"2024-02-21T23:05:07.434442+0000","last_became_peered":"2024-02-21T23:05:07.434442+0000","last_unstale":"2024-02-21T23:05:10.053114+0000","last_undegraded":"2024-02-21T23:05:10.053114+0000","last_fullsized":"2024-02-21T23:05:10.053114+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_clean_scrub_stamp":"2024-02-21T23:05:06.380814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:39:35.673901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.465775+0000","last_change":"2024-02-21T23:04:53.982199+0000","last_active":"2024-02-21T23:05:09.465775+0000","last_peered":"2024-02-21T23:05:09.465775+0000","last_clean":"2024-02-21T23:05:09.465775+0000","last_became_active":"2024-02-21T23:04:53.981777+0000","last_became_peered":"2024-02-21T23:04:53.981777+0000","last_unstale":"2024-02-21T23:05:09.465775+0000","last_undegraded":"2024-02-21T23:05:09.465775+0000","last_fullsized":"2024-02-21T23:05:09.465775+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:52:43.502544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"52'2","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.443226+0000","last_change":"2024-02-21T23:04:52.961423+0000","last_active":"2024-02-21T23:05:09.443226+0000","last_peered":"2024-02-21T23:05:09.443226+0000","last_clean":"2024-02-21T23:05:09.443226+0000","last_became_active":"2024-02-21T23:04:52.961090+0000","last_became_peered":"2024-02-21T23:04:52.961090+0000","last_unstale":"2024-02-21T23:05:09.443226+0000","last_undegraded":"2024-02-21T23:05:09.443226+0000","last_fullsized":"2024-02-21T23:05:09.443226+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:25:58.605620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.115907+0000","last_change":"2024-02-21T23:05:08.455489+0000","last_active":"2024-02-21T23:05:10.115907+0000","last_peered":"2024-02-21T23:05:10.115907+0000","last_clean":"2024-02-21T23:05:10.115907+0000","last_became_active":"2024-02-21T23:05:08.455172+0000","last_became_peered":"2024-02-21T23:05:08.455172+0000","last_unstale":"2024-02-21T23:05:10.115907+0000","last_undegraded":"2024-02-21T23:05:10.115907+0000","last_fullsized":"2024-02-21T23:05:10.115907+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:21:26.217233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.442233+0000","last_change":"2024-02-21T23:05:07.441094+0000","last_active":"2024-02-21T23:05:09.442233+0000","last_peered":"2024-02-21T23:05:09.442233+0000","last_clean":"2024-02-21T23:05:09.442233+0000","last_became_active":"2024-02-21T23:05:07.440751+0000","last_became_peered":"2024-02-21T23:05:07.440751+0000","last_unstale":"2024-02-21T23:05:09.442233+0000","last_undegraded":"2024-02-21T23:05:09.442233+0000","last_fullsized":"2024-02-21T23:05:09.442233+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_clean_scrub_stamp":"2024-02-21T23:05:06.380814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:40:33.985651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.113414+0000","last_change":"2024-02-21T23:04:53.983031+0000","last_active":"2024-02-21T23:05:10.113414+0000","last_peered":"2024-02-21T23:05:10.113414+0000","last_clean":"2024-02-21T23:05:10.113414+0000","last_became_active":"2024-02-21T23:04:53.981770+0000","last_became_peered":"2024-02-21T23:04:53.981770+0000","last_unstale":"2024-02-21T23:05:10.113414+0000","last_undegraded":"2024-02-21T23:05:10.113414+0000","last_fullsized":"2024-02-21T23:05:10.113414+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:23:27.013837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":22,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.444552+0000","last_change":"2024-02-21T23:04:52.975453+0000","last_active":"2024-02-21T23:05:09.444552+0000","last_peered":"2024-02-21T23:05:09.444552+0000","last_clean":"2024-02-21T23:05:09.444552+0000","last_became_active":"2024-02-21T23:04:52.975121+0000","last_became_peered":"2024-02-21T23:04:52.975121+0000","last_unstale":"2024-02-21T23:05:09.444552+0000","last_undegraded":"2024-02-21T23:05:09.444552+0000","last_fullsized":"2024-02-21T23:05:09.444552+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:51:11.660655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.054303+0000","last_change":"2024-02-21T23:05:08.461579+0000","last_active":"2024-02-21T23:05:10.054303+0000","last_peered":"2024-02-21T23:05:10.054303+0000","last_clean":"2024-02-21T23:05:10.054303+0000","last_became_active":"2024-02-21T23:05:08.461173+0000","last_became_peered":"2024-02-21T23:05:08.461173+0000","last_unstale":"2024-02-21T23:05:10.054303+0000","last_undegraded":"2024-02-21T23:05:10.054303+0000","last_fullsized":"2024-02-21T23:05:10.054303+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:38:50.302069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"56'2","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.465885+0000","last_change":"2024-02-21T23:05:07.435315+0000","last_active":"2024-02-21T23:05:09.465885+0000","last_peered":"2024-02-21T23:05:09.465885+0000","last_clean":"2024-02-21T23:05:09.465885+0000","last_became_active":"2024-02-21T23:05:07.435015+0000","last_became_peered":"2024-02-21T23:05:07.435015+0000","last_unstale":"2024-02-21T23:05:09.465885+0000","last_undegraded":"2024-02-21T23:05:09.465885+0000","last_fullsized":"2024-02-21T23:05:09.465885+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_clean_scrub_stamp":"2024-02-21T23:05:06.380814+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:19:01.236574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.055136+0000","last_change":"2024-02-21T23:04:53.978362+0000","last_active":"2024-02-21T23:05:10.055136+0000","last_peered":"2024-02-21T23:05:10.055136+0000","last_clean":"2024-02-21T23:05:10.055136+0000","last_became_active":"2024-02-21T23:04:53.978070+0000","last_became_peered":"2024-02-21T23:04:53.978070+0000","last_unstale":"2024-02-21T23:05:10.055136+0000","last_undegraded":"2024-02-21T23:05:10.055136+0000","last_fullsized":"2024-02-21T23:05:10.055136+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:52:55.798286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":22,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.053064+0000","last_change":"2024-02-21T23:04:52.979855+0000","last_active":"2024-02-21T23:05:10.053064+0000","last_peered":"2024-02-21T23:05:10.053064+0000","last_clean":"2024-02-21T23:05:10.053064+0000","last_became_active":"2024-02-21T23:04:52.977225+0000","last_became_peered":"2024-02-21T23:04:52.977225+0000","last_unstale":"2024-02-21T23:05:10.053064+0000","last_undegraded":"2024-02-21T23:05:10.053064+0000","last_fullsized":"2024-02-21T23:05:10.053064+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:39:10.601854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.470182+0000","last_change":"2024-02-21T23:05:08.444548+0000","last_active":"2024-02-21T23:05:09.470182+0000","last_peered":"2024-02-21T23:05:09.470182+0000","last_clean":"2024-02-21T23:05:09.470182+0000","last_became_active":"2024-02-21T23:05:08.444185+0000","last_became_peered":"2024-02-21T23:05:08.444185+0000","last_unstale":"2024-02-21T23:05:09.470182+0000","last_undegraded":"2024-02-21T23:05:09.470182+0000","last_fullsized":"2024-02-21T23:05:09.470182+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:55:42.900515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"57'1","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.468181+0000","last_change":"2024-02-21T23:05:07.421231+0000","last_active":"2024-02-21T23:05:09.468181+0000","last_peered":"2024-02-21T23:05:09.468181+0000","last_clean":"2024-02-21T23:05:09.468181+0000","last_became_active":"2024-02-21T23:05:07.420844+0000","last_became_peered":"2024-02-21T23:05:07.420844+0000","last_unstale":"2024-02-21T23:05:09.468181+0000","last_undegraded":"2024-02-21T23:05:09.468181+0000","last_fullsized":"2024-02-21T23:05:09.468181+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_clean_scrub_stamp":"2024-02-21T23:05:06.380814+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:38:00.942766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.443238+0000","last_change":"2024-02-21T23:04:53.996234+0000","last_active":"2024-02-21T23:05:09.443238+0000","last_peered":"2024-02-21T23:05:09.443238+0000","last_clean":"2024-02-21T23:05:09.443238+0000","last_became_active":"2024-02-21T23:04:53.995859+0000","last_became_peered":"2024-02-21T23:04:53.995859+0000","last_unstale":"2024-02-21T23:05:09.443238+0000","last_undegraded":"2024-02-21T23:05:09.443238+0000","last_fullsized":"2024-02-21T23:05:09.443238+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:46:44.487212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":22,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.469351+0000","last_change":"2024-02-21T23:04:52.972729+0000","last_active":"2024-02-21T23:05:09.469351+0000","last_peered":"2024-02-21T23:05:09.469351+0000","last_clean":"2024-02-21T23:05:09.469351+0000","last_became_active":"2024-02-21T23:04:52.972434+0000","last_became_peered":"2024-02-21T23:04:52.972434+0000","last_unstale":"2024-02-21T23:05:09.469351+0000","last_undegraded":"2024-02-21T23:05:09.469351+0000","last_fullsized":"2024-02-21T23:05:09.469351+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:00:21.589361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.114854+0000","last_change":"2024-02-21T23:05:08.455465+0000","last_active":"2024-02-21T23:05:10.114854+0000","last_peered":"2024-02-21T23:05:10.114854+0000","last_clean":"2024-02-21T23:05:10.114854+0000","last_became_active":"2024-02-21T23:05:08.455119+0000","last_became_peered":"2024-02-21T23:05:08.455119+0000","last_unstale":"2024-02-21T23:05:10.114854+0000","last_undegraded":"2024-02-21T23:05:10.114854+0000","last_fullsized":"2024-02-21T23:05:10.114854+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:19:27.871458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.052433+0000","last_change":"2024-02-21T23:05:07.434362+0000","last_active":"2024-02-21T23:05:10.052433+0000","last_peered":"2024-02-21T23:05:10.052433+0000","last_clean":"2024-02-21T23:05:10.052433+0000","last_became_active":"2024-02-21T23:05:07.433274+0000","last_became_peered":"2024-02-21T23:05:07.433274+0000","last_unstale":"2024-02-21T23:05:10.052433+0000","last_undegraded":"2024-02-21T23:05:10.052433+0000","last_fullsized":"2024-02-21T23:05:10.052433+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_clean_scrub_stamp":"2024-02-21T23:05:06.380814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:34:15.596228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.466065+0000","last_change":"2024-02-21T23:04:53.996387+0000","last_active":"2024-02-21T23:05:09.466065+0000","last_peered":"2024-02-21T23:05:09.466065+0000","last_clean":"2024-02-21T23:05:09.466065+0000","last_became_active":"2024-02-21T23:04:53.996123+0000","last_became_peered":"2024-02-21T23:04:53.996123+0000","last_unstale":"2024-02-21T23:05:09.466065+0000","last_undegraded":"2024-02-21T23:05:09.466065+0000","last_fullsized":"2024-02-21T23:05:09.466065+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:43:19.009080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"52'2","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.115816+0000","last_change":"2024-02-21T23:04:52.981486+0000","last_active":"2024-02-21T23:05:10.115816+0000","last_peered":"2024-02-21T23:05:10.115816+0000","last_clean":"2024-02-21T23:05:10.115816+0000","last_became_active":"2024-02-21T23:04:52.981135+0000","last_became_peered":"2024-02-21T23:04:52.981135+0000","last_unstale":"2024-02-21T23:05:10.115816+0000","last_undegraded":"2024-02-21T23:05:10.115816+0000","last_fullsized":"2024-02-21T23:05:10.115816+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:17:14.876508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.052275+0000","last_change":"2024-02-21T23:05:08.447608+0000","last_active":"2024-02-21T23:05:10.052275+0000","last_peered":"2024-02-21T23:05:10.052275+0000","last_clean":"2024-02-21T23:05:10.052275+0000","last_became_active":"2024-02-21T23:05:08.445997+0000","last_became_peered":"2024-02-21T23:05:08.445997+0000","last_unstale":"2024-02-21T23:05:10.052275+0000","last_undegraded":"2024-02-21T23:05:10.052275+0000","last_fullsized":"2024-02-21T23:05:10.052275+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:03:07.620481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"56'2","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.053967+0000","last_change":"2024-02-21T23:05:07.443603+0000","last_active":"2024-02-21T23:05:10.053967+0000","last_peered":"2024-02-21T23:05:10.053967+0000","last_clean":"2024-02-21T23:05:10.053967+0000","last_became_active":"2024-02-21T23:05:07.443390+0000","last_became_peered":"2024-02-21T23:05:07.443390+0000","last_unstale":"2024-02-21T23:05:10.053967+0000","last_undegraded":"2024-02-21T23:05:10.053967+0000","last_fullsized":"2024-02-21T23:05:10.053967+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_clean_scrub_stamp":"2024-02-21T23:05:06.380814+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:34:59.258614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.442463+0000","last_change":"2024-02-21T23:04:53.977118+0000","last_active":"2024-02-21T23:05:09.442463+0000","last_peered":"2024-02-21T23:05:09.442463+0000","last_clean":"2024-02-21T23:05:09.442463+0000","last_became_active":"2024-02-21T23:04:53.976042+0000","last_became_peered":"2024-02-21T23:04:53.976042+0000","last_unstale":"2024-02-21T23:05:09.442463+0000","last_undegraded":"2024-02-21T23:05:09.442463+0000","last_fullsized":"2024-02-21T23:05:09.442463+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:10:31.924664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":22,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.012176+0000","last_change":"2024-02-21T23:04:52.974536+0000","last_active":"2024-02-21T23:05:10.012176+0000","last_peered":"2024-02-21T23:05:10.012176+0000","last_clean":"2024-02-21T23:05:10.012176+0000","last_became_active":"2024-02-21T23:04:52.974221+0000","last_became_peered":"2024-02-21T23:04:52.974221+0000","last_unstale":"2024-02-21T23:05:10.012176+0000","last_undegraded":"2024-02-21T23:05:10.012176+0000","last_fullsized":"2024-02-21T23:05:10.012176+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:39:10.601854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.464834+0000","last_change":"2024-02-21T23:05:08.460693+0000","last_active":"2024-02-21T23:05:09.464834+0000","last_peered":"2024-02-21T23:05:09.464834+0000","last_clean":"2024-02-21T23:05:09.464834+0000","last_became_active":"2024-02-21T23:05:08.460424+0000","last_became_peered":"2024-02-21T23:05:08.460424+0000","last_unstale":"2024-02-21T23:05:09.464834+0000","last_undegraded":"2024-02-21T23:05:09.464834+0000","last_fullsized":"2024-02-21T23:05:09.464834+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:00:36.493162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.465910+0000","last_change":"2024-02-21T23:05:07.443614+0000","last_active":"2024-02-21T23:05:09.465910+0000","last_peered":"2024-02-21T23:05:09.465910+0000","last_clean":"2024-02-21T23:05:09.465910+0000","last_became_active":"2024-02-21T23:05:07.443088+0000","last_became_peered":"2024-02-21T23:05:07.443088+0000","last_unstale":"2024-02-21T23:05:09.465910+0000","last_undegraded":"2024-02-21T23:05:09.465910+0000","last_fullsized":"2024-02-21T23:05:09.465910+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_clean_scrub_stamp":"2024-02-21T23:05:06.380814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:28:49.256993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.444962+0000","last_change":"2024-02-21T23:04:53.979161+0000","last_active":"2024-02-21T23:05:09.444962+0000","last_peered":"2024-02-21T23:05:09.444962+0000","last_clean":"2024-02-21T23:05:09.444962+0000","last_became_active":"2024-02-21T23:04:53.978843+0000","last_became_peered":"2024-02-21T23:04:53.978843+0000","last_unstale":"2024-02-21T23:05:09.444962+0000","last_undegraded":"2024-02-21T23:05:09.444962+0000","last_fullsized":"2024-02-21T23:05:09.444962+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:34:45.807165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":22,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.053950+0000","last_change":"2024-02-21T23:04:52.969804+0000","last_active":"2024-02-21T23:05:10.053950+0000","last_peered":"2024-02-21T23:05:10.053950+0000","last_clean":"2024-02-21T23:05:10.053950+0000","last_became_active":"2024-02-21T23:04:52.969353+0000","last_became_peered":"2024-02-21T23:04:52.969353+0000","last_unstale":"2024-02-21T23:05:10.053950+0000","last_undegraded":"2024-02-21T23:05:10.053950+0000","last_fullsized":"2024-02-21T23:05:10.053950+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:47:27.364576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.448509+0000","last_change":"2024-02-21T23:05:08.466401+0000","last_active":"2024-02-21T23:05:09.448509+0000","last_peered":"2024-02-21T23:05:09.448509+0000","last_clean":"2024-02-21T23:05:09.448509+0000","last_became_active":"2024-02-21T23:05:08.466067+0000","last_became_peered":"2024-02-21T23:05:08.466067+0000","last_unstale":"2024-02-21T23:05:09.448509+0000","last_undegraded":"2024-02-21T23:05:09.448509+0000","last_fullsized":"2024-02-21T23:05:09.448509+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:53:10.262900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"56'1","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.011616+0000","last_change":"2024-02-21T23:05:07.438319+0000","last_active":"2024-02-21T23:05:10.011616+0000","last_peered":"2024-02-21T23:05:10.011616+0000","last_clean":"2024-02-21T23:05:10.011616+0000","last_became_active":"2024-02-21T23:05:07.437958+0000","last_became_peered":"2024-02-21T23:05:07.437958+0000","last_unstale":"2024-02-21T23:05:10.011616+0000","last_undegraded":"2024-02-21T23:05:10.011616+0000","last_fullsized":"2024-02-21T23:05:10.011616+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_clean_scrub_stamp":"2024-02-21T23:05:06.380814+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:57:17.844338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.050414+0000","last_change":"2024-02-21T23:04:54.000144+0000","last_active":"2024-02-21T23:05:10.050414+0000","last_peered":"2024-02-21T23:05:10.050414+0000","last_clean":"2024-02-21T23:05:10.050414+0000","last_became_active":"2024-02-21T23:04:53.999687+0000","last_became_peered":"2024-02-21T23:04:53.999687+0000","last_unstale":"2024-02-21T23:05:10.050414+0000","last_undegraded":"2024-02-21T23:05:10.050414+0000","last_fullsized":"2024-02-21T23:05:10.050414+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:12:19.379347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":22,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.445245+0000","last_change":"2024-02-21T23:04:52.975668+0000","last_active":"2024-02-21T23:05:09.445245+0000","last_peered":"2024-02-21T23:05:09.445245+0000","last_clean":"2024-02-21T23:05:09.445245+0000","last_became_active":"2024-02-21T23:04:52.975348+0000","last_became_peered":"2024-02-21T23:04:52.975348+0000","last_unstale":"2024-02-21T23:05:09.445245+0000","last_undegraded":"2024-02-21T23:05:09.445245+0000","last_fullsized":"2024-02-21T23:05:09.445245+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:12:18.358129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.115717+0000","last_change":"2024-02-21T23:05:08.460822+0000","last_active":"2024-02-21T23:05:10.115717+0000","last_peered":"2024-02-21T23:05:10.115717+0000","last_clean":"2024-02-21T23:05:10.115717+0000","last_became_active":"2024-02-21T23:05:08.460519+0000","last_became_peered":"2024-02-21T23:05:08.460519+0000","last_unstale":"2024-02-21T23:05:10.115717+0000","last_undegraded":"2024-02-21T23:05:10.115717+0000","last_fullsized":"2024-02-21T23:05:10.115717+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:59:56.223705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.445004+0000","last_change":"2024-02-21T23:05:07.439510+0000","last_active":"2024-02-21T23:05:09.445004+0000","last_peered":"2024-02-21T23:05:09.445004+0000","last_clean":"2024-02-21T23:05:09.445004+0000","last_became_active":"2024-02-21T23:05:07.439198+0000","last_became_peered":"2024-02-21T23:05:07.439198+0000","last_unstale":"2024-02-21T23:05:09.445004+0000","last_undegraded":"2024-02-21T23:05:09.445004+0000","last_fullsized":"2024-02-21T23:05:09.445004+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_clean_scrub_stamp":"2024-02-21T23:05:06.380814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:48:17.959315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.465728+0000","last_change":"2024-02-21T23:04:53.988119+0000","last_active":"2024-02-21T23:05:09.465728+0000","last_peered":"2024-02-21T23:05:09.465728+0000","last_clean":"2024-02-21T23:05:09.465728+0000","last_became_active":"2024-02-21T23:04:53.986237+0000","last_became_peered":"2024-02-21T23:04:53.986237+0000","last_unstale":"2024-02-21T23:05:09.465728+0000","last_undegraded":"2024-02-21T23:05:09.465728+0000","last_fullsized":"2024-02-21T23:05:09.465728+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:10:20.341856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":22,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.444670+0000","last_change":"2024-02-21T23:04:52.976653+0000","last_active":"2024-02-21T23:05:09.444670+0000","last_peered":"2024-02-21T23:05:09.444670+0000","last_clean":"2024-02-21T23:05:09.444670+0000","last_became_active":"2024-02-21T23:04:52.976249+0000","last_became_peered":"2024-02-21T23:04:52.976249+0000","last_unstale":"2024-02-21T23:05:09.444670+0000","last_undegraded":"2024-02-21T23:05:09.444670+0000","last_fullsized":"2024-02-21T23:05:09.444670+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:40:19.512984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.011757+0000","last_change":"2024-02-21T23:05:08.474817+0000","last_active":"2024-02-21T23:05:10.011757+0000","last_peered":"2024-02-21T23:05:10.011757+0000","last_clean":"2024-02-21T23:05:10.011757+0000","last_became_active":"2024-02-21T23:05:08.474309+0000","last_became_peered":"2024-02-21T23:05:08.474309+0000","last_unstale":"2024-02-21T23:05:10.011757+0000","last_undegraded":"2024-02-21T23:05:10.011757+0000","last_fullsized":"2024-02-21T23:05:10.011757+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:25:56.439594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"56'1","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.466178+0000","last_change":"2024-02-21T23:05:07.440280+0000","last_active":"2024-02-21T23:05:09.466178+0000","last_peered":"2024-02-21T23:05:09.466178+0000","last_clean":"2024-02-21T23:05:09.466178+0000","last_became_active":"2024-02-21T23:05:07.440008+0000","last_became_peered":"2024-02-21T23:05:07.440008+0000","last_unstale":"2024-02-21T23:05:09.466178+0000","last_undegraded":"2024-02-21T23:05:09.466178+0000","last_fullsized":"2024-02-21T23:05:09.466178+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_clean_scrub_stamp":"2024-02-21T23:05:06.380814+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:59:55.210541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.051585+0000","last_change":"2024-02-21T23:04:53.993076+0000","last_active":"2024-02-21T23:05:10.051585+0000","last_peered":"2024-02-21T23:05:10.051585+0000","last_clean":"2024-02-21T23:05:10.051585+0000","last_became_active":"2024-02-21T23:04:53.992734+0000","last_became_peered":"2024-02-21T23:04:53.992734+0000","last_unstale":"2024-02-21T23:05:10.051585+0000","last_undegraded":"2024-02-21T23:05:10.051585+0000","last_fullsized":"2024-02-21T23:05:10.051585+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:52:43.502544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":22,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.446077+0000","last_change":"2024-02-21T23:04:52.971065+0000","last_active":"2024-02-21T23:05:09.446077+0000","last_peered":"2024-02-21T23:05:09.446077+0000","last_clean":"2024-02-21T23:05:09.446077+0000","last_became_active":"2024-02-21T23:04:52.970410+0000","last_became_peered":"2024-02-21T23:04:52.970410+0000","last_unstale":"2024-02-21T23:05:09.446077+0000","last_undegraded":"2024-02-21T23:05:09.446077+0000","last_fullsized":"2024-02-21T23:05:09.446077+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:18:03.518976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.115438+0000","last_change":"2024-02-21T23:05:08.471258+0000","last_active":"2024-02-21T23:05:10.115438+0000","last_peered":"2024-02-21T23:05:10.115438+0000","last_clean":"2024-02-21T23:05:10.115438+0000","last_became_active":"2024-02-21T23:05:08.470955+0000","last_became_peered":"2024-02-21T23:05:08.470955+0000","last_unstale":"2024-02-21T23:05:10.115438+0000","last_undegraded":"2024-02-21T23:05:10.115438+0000","last_fullsized":"2024-02-21T23:05:10.115438+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:28:50.270158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"56'2","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.012215+0000","last_change":"2024-02-21T23:05:07.441965+0000","last_active":"2024-02-21T23:05:10.012215+0000","last_peered":"2024-02-21T23:05:10.012215+0000","last_clean":"2024-02-21T23:05:10.012215+0000","last_became_active":"2024-02-21T23:05:07.441586+0000","last_became_peered":"2024-02-21T23:05:07.441586+0000","last_unstale":"2024-02-21T23:05:10.012215+0000","last_undegraded":"2024-02-21T23:05:10.012215+0000","last_fullsized":"2024-02-21T23:05:10.012215+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_clean_scrub_stamp":"2024-02-21T23:05:06.380814+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:38:00.942766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.445429+0000","last_change":"2024-02-21T23:04:53.995393+0000","last_active":"2024-02-21T23:05:09.445429+0000","last_peered":"2024-02-21T23:05:09.445429+0000","last_clean":"2024-02-21T23:05:09.445429+0000","last_became_active":"2024-02-21T23:04:53.995131+0000","last_became_peered":"2024-02-21T23:04:53.995131+0000","last_unstale":"2024-02-21T23:05:09.445429+0000","last_undegraded":"2024-02-21T23:05:09.445429+0000","last_fullsized":"2024-02-21T23:05:09.445429+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:34:02.144778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":22,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.465868+0000","last_change":"2024-02-21T23:04:52.968036+0000","last_active":"2024-02-21T23:05:09.465868+0000","last_peered":"2024-02-21T23:05:09.465868+0000","last_clean":"2024-02-21T23:05:09.465868+0000","last_became_active":"2024-02-21T23:04:52.967626+0000","last_became_peered":"2024-02-21T23:04:52.967626+0000","last_unstale":"2024-02-21T23:05:09.465868+0000","last_undegraded":"2024-02-21T23:05:09.465868+0000","last_fullsized":"2024-02-21T23:05:09.465868+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:33:33.298554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.114693+0000","last_change":"2024-02-21T23:05:08.452963+0000","last_active":"2024-02-21T23:05:10.114693+0000","last_peered":"2024-02-21T23:05:10.114693+0000","last_clean":"2024-02-21T23:05:10.114693+0000","last_became_active":"2024-02-21T23:05:08.452680+0000","last_became_peered":"2024-02-21T23:05:08.452680+0000","last_unstale":"2024-02-21T23:05:10.114693+0000","last_undegraded":"2024-02-21T23:05:10.114693+0000","last_fullsized":"2024-02-21T23:05:10.114693+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:18:12.721687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.442927+0000","last_change":"2024-02-21T23:05:07.427069+0000","last_active":"2024-02-21T23:05:09.442927+0000","last_peered":"2024-02-21T23:05:09.442927+0000","last_clean":"2024-02-21T23:05:09.442927+0000","last_became_active":"2024-02-21T23:05:07.426609+0000","last_became_peered":"2024-02-21T23:05:07.426609+0000","last_unstale":"2024-02-21T23:05:09.442927+0000","last_undegraded":"2024-02-21T23:05:09.442927+0000","last_fullsized":"2024-02-21T23:05:09.442927+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_clean_scrub_stamp":"2024-02-21T23:05:06.380814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:41:32.687033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.113368+0000","last_change":"2024-02-21T23:04:53.985061+0000","last_active":"2024-02-21T23:05:10.113368+0000","last_peered":"2024-02-21T23:05:10.113368+0000","last_clean":"2024-02-21T23:05:10.113368+0000","last_became_active":"2024-02-21T23:04:53.984756+0000","last_became_peered":"2024-02-21T23:04:53.984756+0000","last_unstale":"2024-02-21T23:05:10.113368+0000","last_undegraded":"2024-02-21T23:05:10.113368+0000","last_fullsized":"2024-02-21T23:05:10.113368+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:55:24.090577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"53'1","reported_seq":23,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.055151+0000","last_change":"2024-02-21T23:04:52.979253+0000","last_active":"2024-02-21T23:05:10.055151+0000","last_peered":"2024-02-21T23:05:10.055151+0000","last_clean":"2024-02-21T23:05:10.055151+0000","last_became_active":"2024-02-21T23:04:52.978773+0000","last_became_peered":"2024-02-21T23:04:52.978773+0000","last_unstale":"2024-02-21T23:05:10.055151+0000","last_undegraded":"2024-02-21T23:05:10.055151+0000","last_fullsized":"2024-02-21T23:05:10.055151+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:39:31.156140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":239,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.010467+0000","last_change":"2024-02-21T23:02:32.289335+0000","last_active":"2024-02-21T23:05:10.010467+0000","last_peered":"2024-02-21T23:05:10.010467+0000","last_clean":"2024-02-21T23:05:10.010467+0000","last_became_active":"2024-02-21T23:02:32.286913+0000","last_became_peered":"2024-02-21T23:02:32.286913+0000","last_unstale":"2024-02-21T23:05:10.010467+0000","last_undegraded":"2024-02-21T23:05:10.010467+0000","last_fullsized":"2024-02-21T23:05:10.010467+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_clean_scrub_stamp":"2024-02-21T23:02:31.172569+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:36:23.987374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574882,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27764,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344076,"statfs":{"total":96636764160,"available":96608333824,"internally_reserved":0,"allocated":757760,"data_stored":897214,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574883,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27700,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344140,"statfs":{"total":96636764160,"available":96608399360,"internally_reserved":0,"allocated":692224,"data_stored":314181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574882,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27692,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344148,"statfs":{"total":96636764160,"available":96608407552,"internally_reserved":0,"allocated":684032,"data_stored":313517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574883,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27704,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344136,"statfs":{"total":96636764160,"available":96608395264,"internally_reserved":0,"allocated":696320,"data_stored":315689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574883,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31864,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339976,"statfs":{"total":96636764160,"available":96604135424,"internally_reserved":0,"allocated":696320,"data_stored":315129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574884,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31860,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339980,"statfs":{"total":96636764160,"available":96604139520,"internally_reserved":0,"allocated":692224,"data_stored":315107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574882,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31888,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339952,"statfs":{"total":96636764160,"available":96604110848,"internally_reserved":0,"allocated":716800,"data_stored":316031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574882,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31908,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339932,"statfs":{"total":96636764160,"available":96604090368,"internally_reserved":0,"allocated":741376,"data_stored":896370,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-21T23:05:13.181 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-21T23:05:13.348 INFO:tasks.ceph.mds.d.smithi081.stderr:2024-02-21T23:05:13.359+0000 7f7bfcc00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-21T23:05:13.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.439+0000 7f59de200640 1 Processor -- start 2024-02-21T23:05:13.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.439+0000 7f59de200640 1 -- start start 2024-02-21T23:05:13.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.440+0000 7f59de200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f59d80c8a80 0x7f59d8156c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:13.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.440+0000 7f59de200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f59d81540b0 0x7f59d815d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:13.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.440+0000 7f59de200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59d80c8000 con 0x7f59d80c8e60 2024-02-21T23:05:13.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.440+0000 7f59de200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59d80c9310 con 0x7f59d80c8a80 2024-02-21T23:05:13.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.440+0000 7f59de200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59d80c82e0 con 0x7f59d81540b0 2024-02-21T23:05:13.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.441+0000 7f59d7e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f59d80c8e60 0x7f59d8153640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34454/0 (socket says 172.21.15.28:34454) 2024-02-21T23:05:13.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.441+0000 7f59d7e00640 1 -- 172.21.15.28:0/2012777950 learned_addr learned my addr 172.21.15.28:0/2012777950 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:05:13.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.441+0000 7f59dce00640 1 --2- 172.21.15.28:0/2012777950 >> v2:172.21.15.81:3300/0 conn(0x7f59d80c8a80 0x7f59d8156c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:13.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.441+0000 7f59dd800640 1 --2- 172.21.15.28:0/2012777950 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f59d81540b0 0x7f59d815d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:13.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.441+0000 7f59d7400640 1 -- 172.21.15.28:0/2012777950 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3350369889 0 0) 0x7f59d80c8000 con 0x7f59d80c8e60 2024-02-21T23:05:13.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.441+0000 7f59d7400640 1 -- 172.21.15.28:0/2012777950 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59c0003520 con 0x7f59d80c8e60 2024-02-21T23:05:13.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.442+0000 7f59d7400640 1 -- 172.21.15.28:0/2012777950 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1278407476 0 0) 0x7f59c0003520 con 0x7f59d80c8e60 2024-02-21T23:05:13.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.442+0000 7f59d7400640 1 -- 172.21.15.28:0/2012777950 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f59d81540b0 msgr2=0x7f59d815d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:13.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.442+0000 7f59d7400640 1 --2- 172.21.15.28:0/2012777950 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f59d81540b0 0x7f59d815d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:13.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.442+0000 7f59d7400640 1 -- 172.21.15.28:0/2012777950 >> v2:172.21.15.81:3300/0 conn(0x7f59d80c8a80 msgr2=0x7f59d8156c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:13.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.442+0000 7f59d7400640 1 --2- 172.21.15.28:0/2012777950 >> v2:172.21.15.81:3300/0 conn(0x7f59d80c8a80 0x7f59d8156c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:13.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.442+0000 7f59d7400640 1 -- 172.21.15.28:0/2012777950 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59d81561b0 con 0x7f59d80c8e60 2024-02-21T23:05:13.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.442+0000 7f59d7400640 1 -- 172.21.15.28:0/2012777950 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f59c4001670 con 0x7f59d80c8e60 2024-02-21T23:05:13.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.442+0000 7f59d7400640 1 -- 172.21.15.28:0/2012777950 >> v1:172.21.15.28:6789/0 conn(0x7f59d80c8e60 legacy=0x7f59d8153640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:13.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.442+0000 7f59d7400640 1 --2- 172.21.15.28:0/2012777950 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f59d81540b0 0x7f59c00039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:13.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.442+0000 7f59d7400640 1 --2- 172.21.15.28:0/2012777950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59c0003f30 0x7f59c0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:13.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.442+0000 7f59d7400640 1 --2- 172.21.15.28:0/2012777950 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f59c0004d40 0x7f59c00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:13.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.442+0000 7f59d7400640 1 -- 172.21.15.28:0/2012777950 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d80c9310 con 0x7f59c0003f30 2024-02-21T23:05:13.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.442+0000 7f59d7400640 1 -- 172.21.15.28:0/2012777950 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d80c82e0 con 0x7f59c0004d40 2024-02-21T23:05:13.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.442+0000 7f59d7400640 1 -- 172.21.15.28:0/2012777950 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59c000ba10 con 0x7f59d81540b0 2024-02-21T23:05:13.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.442+0000 7f59d7e00640 1 --2- 172.21.15.28:0/2012777950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59c0003f30 0x7f59c0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:13.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.443+0000 7f59dce00640 1 --2- 172.21.15.28:0/2012777950 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f59d81540b0 0x7f59c00039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:13.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.443+0000 7f59dd800640 1 --2- 172.21.15.28:0/2012777950 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f59c0004d40 0x7f59c00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:13.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.443+0000 7f59d7e00640 1 -- 172.21.15.28:0/2012777950 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f59d81540b0 msgr2=0x7f59c00039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:13.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.443+0000 7f59d7e00640 1 --2- 172.21.15.28:0/2012777950 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f59d81540b0 0x7f59c00039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:13.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.443+0000 7f59d7e00640 1 -- 172.21.15.28:0/2012777950 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f59c0004d40 msgr2=0x7f59c00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:13.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.443+0000 7f59d7e00640 1 --2- 172.21.15.28:0/2012777950 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f59c0004d40 0x7f59c00051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:13.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.443+0000 7f59d7e00640 1 -- 172.21.15.28:0/2012777950 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59c0003520 con 0x7f59c0003f30 2024-02-21T23:05:13.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.443+0000 7f59dce00640 1 --2- 172.21.15.28:0/2012777950 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f59d81540b0 0x7f59c00039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:05:13.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.444+0000 7f59d7e00640 1 --2- 172.21.15.28:0/2012777950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59c0003f30 0x7f59c0004800 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f59c400c270 tx=0x7f59c0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b58f8d7bf0e82145 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:13.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.444+0000 7f59d7400640 1 -- 172.21.15.28:0/2012777950 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59c4007d30 con 0x7f59c0003f30 2024-02-21T23:05:13.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.444+0000 7f59d7400640 1 -- 172.21.15.28:0/2012777950 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59c4030070 con 0x7f59c0003f30 2024-02-21T23:05:13.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.444+0000 7f59d7400640 1 -- 172.21.15.28:0/2012777950 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59c402cda0 con 0x7f59c0003f30 2024-02-21T23:05:13.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.445+0000 7f59de200640 1 -- 172.21.15.28:0/2012777950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59c0003f30 msgr2=0x7f59c0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:13.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.445+0000 7f59de200640 1 --2- 172.21.15.28:0/2012777950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59c0003f30 0x7f59c0004800 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f59c400c270 tx=0x7f59c0005830 comp rx=0 tx=0).stop 2024-02-21T23:05:13.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.445+0000 7f59dd800640 1 -- 172.21.15.28:0/2012777950 reap_dead start 2024-02-21T23:05:13.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.445+0000 7f59de200640 1 -- 172.21.15.28:0/2012777950 shutdown_connections 2024-02-21T23:05:13.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.445+0000 7f59de200640 1 -- 172.21.15.28:0/2012777950 >> 172.21.15.28:0/2012777950 conn(0x7f59d806b260 msgr2=0x7f59d806b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:13.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.445+0000 7f59de200640 1 -- 172.21.15.28:0/2012777950 shutdown_connections 2024-02-21T23:05:13.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.445+0000 7f59de200640 1 -- 172.21.15.28:0/2012777950 wait complete. 2024-02-21T23:05:13.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.447+0000 7f59de200640 1 Processor -- start 2024-02-21T23:05:13.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.447+0000 7f59de200640 1 -- start start 2024-02-21T23:05:13.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.448+0000 7f59de200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f59c0003f30 0x7f59d81475e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:13.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.448+0000 7f59de200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59d81dc220 0x7f59d8149250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:13.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.449+0000 7f59de200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f59d8149790 0x7f59d806c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:13.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.449+0000 7f59de200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d80c7c90 con 0x7f59d81dc220 2024-02-21T23:05:13.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.449+0000 7f59de200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d8156770 con 0x7f59d8149790 2024-02-21T23:05:13.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.449+0000 7f59de200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59d80c2c30 con 0x7f59c0003f30 2024-02-21T23:05:13.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.449+0000 7f59d7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59d81dc220 0x7f59d8149250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:13.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.449+0000 7f59d7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59d81dc220 0x7f59d8149250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36250/0 (socket says 172.21.15.28:36250) 2024-02-21T23:05:13.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.449+0000 7f59d7e00640 1 -- 172.21.15.28:0/1944756548 learned_addr learned my addr 172.21.15.28:0/1944756548 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:13.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.449+0000 7f59dce00640 1 --2- 172.21.15.28:0/1944756548 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f59c0003f30 0x7f59d81475e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:13.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.449+0000 7f59d7e00640 1 -- 172.21.15.28:0/1944756548 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f59c0003f30 msgr2=0x7f59d81475e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:13.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.449+0000 7f59d7e00640 1 --2- 172.21.15.28:0/1944756548 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f59c0003f30 0x7f59d81475e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:13.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.449+0000 7f59d7e00640 1 -- 172.21.15.28:0/1944756548 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f59d8149790 msgr2=0x7f59d806c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:13.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.449+0000 7f59d7e00640 1 --2- 172.21.15.28:0/1944756548 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f59d8149790 0x7f59d806c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:13.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.449+0000 7f59d7e00640 1 -- 172.21.15.28:0/1944756548 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59d8147b20 con 0x7f59d81dc220 2024-02-21T23:05:13.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.449+0000 7f59dce00640 1 --2- 172.21.15.28:0/1944756548 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f59c0003f30 0x7f59d81475e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:05:13.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.450+0000 7f59d7e00640 1 --2- 172.21.15.28:0/1944756548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59d81dc220 0x7f59d8149250 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f59c400c390 tx=0x7f59d8156380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:13.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.450+0000 7f59d4c00640 1 -- 172.21.15.28:0/1944756548 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59c4045070 con 0x7f59d81dc220 2024-02-21T23:05:13.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.450+0000 7f59d4c00640 1 -- 172.21.15.28:0/1944756548 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59c4004bf0 con 0x7f59d81dc220 2024-02-21T23:05:13.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.450+0000 7f59d4c00640 1 -- 172.21.15.28:0/1944756548 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59c4001c70 con 0x7f59d81dc220 2024-02-21T23:05:13.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.450+0000 7f59de200640 1 -- 172.21.15.28:0/1944756548 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59d806db30 con 0x7f59d81dc220 2024-02-21T23:05:13.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.450+0000 7f59de200640 1 -- 172.21.15.28:0/1944756548 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59d8150350 con 0x7f59d81dc220 2024-02-21T23:05:13.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.451+0000 7f59de200640 1 -- 172.21.15.28:0/1944756548 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59d8068640 con 0x7f59d81dc220 2024-02-21T23:05:13.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.452+0000 7f59d4c00640 1 -- 172.21.15.28:0/1944756548 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f59c4007070 con 0x7f59d81dc220 2024-02-21T23:05:13.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.454+0000 7f59d4c00640 1 --2- 172.21.15.28:0/1944756548 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f59b806b500 0x7f59b806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:13.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.454+0000 7f59d4c00640 1 -- 172.21.15.28:0/1944756548 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f59c40ab5c0 con 0x7f59d81dc220 2024-02-21T23:05:13.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.455+0000 7f59dce00640 1 --2- 172.21.15.28:0/1944756548 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f59b806b500 0x7f59b806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:13.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.456+0000 7f59dce00640 1 --2- 172.21.15.28:0/1944756548 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f59b806b500 0x7f59b806d9c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f59cc009aa0 tx=0x7f59cc008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:13.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.458+0000 7f59d4c00640 1 -- 172.21.15.28:0/1944756548 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f59d8068640 con 0x7f59d81dc220 2024-02-21T23:05:13.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.656+0000 7f59de200640 1 -- 172.21.15.28:0/1944756548 --> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f59d814a6f0 con 0x7f59b806b500 2024-02-21T23:05:13.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.716+0000 7f59d4c00640 1 -- 172.21.15.28:0/1944756548 <== mgr.4101 v2:172.21.15.81:6832/1848538577 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+490559 (secure 0 0 0) 0x7f59d814a6f0 con 0x7f59b806b500 2024-02-21T23:05:13.712 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:05:13.715 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-21T23:05:13.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.724+0000 7f59de200640 1 -- 172.21.15.28:0/1944756548 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f59b806b500 msgr2=0x7f59b806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:13.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.724+0000 7f59de200640 1 --2- 172.21.15.28:0/1944756548 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f59b806b500 0x7f59b806d9c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f59cc009aa0 tx=0x7f59cc008040 comp rx=0 tx=0).stop 2024-02-21T23:05:13.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.724+0000 7f59de200640 1 -- 172.21.15.28:0/1944756548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59d81dc220 msgr2=0x7f59d8149250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:13.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.724+0000 7f59de200640 1 --2- 172.21.15.28:0/1944756548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59d81dc220 0x7f59d8149250 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f59c400c390 tx=0x7f59d8156380 comp rx=0 tx=0).stop 2024-02-21T23:05:13.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.724+0000 7f59de200640 1 -- 172.21.15.28:0/1944756548 shutdown_connections 2024-02-21T23:05:13.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.724+0000 7f59de200640 1 --2- 172.21.15.28:0/1944756548 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f59b806b500 0x7f59b806d9c0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:13.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.724+0000 7f59de200640 1 --2- 172.21.15.28:0/1944756548 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f59d8149790 0x7f59d806c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:13.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.724+0000 7f59de200640 1 --2- 172.21.15.28:0/1944756548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59d81dc220 0x7f59d8149250 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:13.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.724+0000 7f59de200640 1 --2- 172.21.15.28:0/1944756548 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f59c0003f30 0x7f59d81475e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:13.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.724+0000 7f59de200640 1 -- 172.21.15.28:0/1944756548 >> 172.21.15.28:0/1944756548 conn(0x7f59d806b260 msgr2=0x7f59d81df2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:13.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.724+0000 7f59de200640 1 -- 172.21.15.28:0/1944756548 shutdown_connections 2024-02-21T23:05:13.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:13.724+0000 7f59de200640 1 -- 172.21.15.28:0/1944756548 wait complete. 2024-02-21T23:05:13.735 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":131,"stamp":"2024-02-21T23:05:12.717191+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595188,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":249,"num_write_kb":2444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":195,"ondisk_log_size":195,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":343,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":238380,"kb_used_data":5544,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754736340,"statfs":{"total":773094113280,"available":772850012160,"internally_reserved":0,"allocated":5677056,"data_stored":3683238,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001912"},"pg_stats":[{"pgid":"10.3d","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.011982+0000","last_change":"2024-02-21T23:05:08.470952+0000","last_active":"2024-02-21T23:05:10.011982+0000","last_peered":"2024-02-21T23:05:10.011982+0000","last_clean":"2024-02-21T23:05:10.011982+0000","last_became_active":"2024-02-21T23:05:08.470663+0000","last_became_peered":"2024-02-21T23:05:08.470663+0000","last_unstale":"2024-02-21T23:05:10.011982+0000","last_undegraded":"2024-02-21T23:05:10.011982+0000","last_fullsized":"2024-02-21T23:05:10.011982+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:15:50.294164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3f","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.443191+0000","last_change":"2024-02-21T23:04:53.986963+0000","last_active":"2024-02-21T23:05:09.443191+0000","last_peered":"2024-02-21T23:05:09.443191+0000","last_clean":"2024-02-21T23:05:09.443191+0000","last_became_active":"2024-02-21T23:04:53.986659+0000","last_became_peered":"2024-02-21T23:04:53.986659+0000","last_unstale":"2024-02-21T23:05:09.443191+0000","last_undegraded":"2024-02-21T23:05:09.443191+0000","last_fullsized":"2024-02-21T23:05:09.443191+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:33:34.319772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.448323+0000","last_change":"2024-02-21T23:05:08.451977+0000","last_active":"2024-02-21T23:05:09.448323+0000","last_peered":"2024-02-21T23:05:09.448323+0000","last_clean":"2024-02-21T23:05:09.448323+0000","last_became_active":"2024-02-21T23:05:08.451635+0000","last_became_peered":"2024-02-21T23:05:08.451635+0000","last_unstale":"2024-02-21T23:05:09.448323+0000","last_undegraded":"2024-02-21T23:05:09.448323+0000","last_fullsized":"2024-02-21T23:05:09.448323+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:43:33.473694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.009309+0000","last_change":"2024-02-21T23:04:53.992488+0000","last_active":"2024-02-21T23:05:10.009309+0000","last_peered":"2024-02-21T23:05:10.009309+0000","last_clean":"2024-02-21T23:05:10.009309+0000","last_became_active":"2024-02-21T23:04:53.992187+0000","last_became_peered":"2024-02-21T23:04:53.992187+0000","last_unstale":"2024-02-21T23:05:10.009309+0000","last_undegraded":"2024-02-21T23:05:10.009309+0000","last_fullsized":"2024-02-21T23:05:10.009309+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:27:08.923372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3f","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.443089+0000","last_change":"2024-02-21T23:05:08.460212+0000","last_active":"2024-02-21T23:05:09.443089+0000","last_peered":"2024-02-21T23:05:09.443089+0000","last_clean":"2024-02-21T23:05:09.443089+0000","last_became_active":"2024-02-21T23:05:08.459526+0000","last_became_peered":"2024-02-21T23:05:08.459526+0000","last_unstale":"2024-02-21T23:05:09.443089+0000","last_undegraded":"2024-02-21T23:05:09.443089+0000","last_fullsized":"2024-02-21T23:05:09.443089+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:21:26.217233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.051410+0000","last_change":"2024-02-21T23:04:54.002212+0000","last_active":"2024-02-21T23:05:10.051410+0000","last_peered":"2024-02-21T23:05:10.051410+0000","last_clean":"2024-02-21T23:05:10.051410+0000","last_became_active":"2024-02-21T23:04:54.001921+0000","last_became_peered":"2024-02-21T23:04:54.001921+0000","last_unstale":"2024-02-21T23:05:10.051410+0000","last_undegraded":"2024-02-21T23:05:10.051410+0000","last_fullsized":"2024-02-21T23:05:10.051410+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:43:19.009080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.442413+0000","last_change":"2024-02-21T23:05:08.450709+0000","last_active":"2024-02-21T23:05:09.442413+0000","last_peered":"2024-02-21T23:05:09.442413+0000","last_clean":"2024-02-21T23:05:09.442413+0000","last_became_active":"2024-02-21T23:05:08.447867+0000","last_became_peered":"2024-02-21T23:05:08.447867+0000","last_unstale":"2024-02-21T23:05:09.442413+0000","last_undegraded":"2024-02-21T23:05:09.442413+0000","last_fullsized":"2024-02-21T23:05:09.442413+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:59:56.223705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.008531+0000","last_change":"2024-02-21T23:04:53.994828+0000","last_active":"2024-02-21T23:05:10.008531+0000","last_peered":"2024-02-21T23:05:10.008531+0000","last_clean":"2024-02-21T23:05:10.008531+0000","last_became_active":"2024-02-21T23:04:53.993942+0000","last_became_peered":"2024-02-21T23:04:53.993942+0000","last_unstale":"2024-02-21T23:05:10.008531+0000","last_undegraded":"2024-02-21T23:05:10.008531+0000","last_fullsized":"2024-02-21T23:05:10.008531+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:56:39.552779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.469413+0000","last_change":"2024-02-21T23:05:08.467524+0000","last_active":"2024-02-21T23:05:09.469413+0000","last_peered":"2024-02-21T23:05:09.469413+0000","last_clean":"2024-02-21T23:05:09.469413+0000","last_became_active":"2024-02-21T23:05:08.467082+0000","last_became_peered":"2024-02-21T23:05:08.467082+0000","last_unstale":"2024-02-21T23:05:09.469413+0000","last_undegraded":"2024-02-21T23:05:09.469413+0000","last_fullsized":"2024-02-21T23:05:09.469413+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:12:18.038010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.446665+0000","last_change":"2024-02-21T23:04:53.984836+0000","last_active":"2024-02-21T23:05:09.446665+0000","last_peered":"2024-02-21T23:05:09.446665+0000","last_clean":"2024-02-21T23:05:09.446665+0000","last_became_active":"2024-02-21T23:04:53.984493+0000","last_became_peered":"2024-02-21T23:04:53.984493+0000","last_unstale":"2024-02-21T23:05:09.446665+0000","last_undegraded":"2024-02-21T23:05:09.446665+0000","last_fullsized":"2024-02-21T23:05:09.446665+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:45:50.638257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.465139+0000","last_change":"2024-02-21T23:05:08.468540+0000","last_active":"2024-02-21T23:05:09.465139+0000","last_peered":"2024-02-21T23:05:09.465139+0000","last_clean":"2024-02-21T23:05:09.465139+0000","last_became_active":"2024-02-21T23:05:08.468230+0000","last_became_peered":"2024-02-21T23:05:08.468230+0000","last_unstale":"2024-02-21T23:05:09.465139+0000","last_undegraded":"2024-02-21T23:05:09.465139+0000","last_fullsized":"2024-02-21T23:05:09.465139+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:33:22.889577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.442333+0000","last_change":"2024-02-21T23:04:53.984765+0000","last_active":"2024-02-21T23:05:09.442333+0000","last_peered":"2024-02-21T23:05:09.442333+0000","last_clean":"2024-02-21T23:05:09.442333+0000","last_became_active":"2024-02-21T23:04:53.984432+0000","last_became_peered":"2024-02-21T23:04:53.984432+0000","last_unstale":"2024-02-21T23:05:09.442333+0000","last_undegraded":"2024-02-21T23:05:09.442333+0000","last_fullsized":"2024-02-21T23:05:09.442333+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:55:04.908945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.114786+0000","last_change":"2024-02-21T23:05:08.467626+0000","last_active":"2024-02-21T23:05:10.114786+0000","last_peered":"2024-02-21T23:05:10.114786+0000","last_clean":"2024-02-21T23:05:10.114786+0000","last_became_active":"2024-02-21T23:05:08.467361+0000","last_became_peered":"2024-02-21T23:05:08.467361+0000","last_unstale":"2024-02-21T23:05:10.114786+0000","last_undegraded":"2024-02-21T23:05:10.114786+0000","last_fullsized":"2024-02-21T23:05:10.114786+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:57:18.857503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.050580+0000","last_change":"2024-02-21T23:04:54.003595+0000","last_active":"2024-02-21T23:05:10.050580+0000","last_peered":"2024-02-21T23:05:10.050580+0000","last_clean":"2024-02-21T23:05:10.050580+0000","last_became_active":"2024-02-21T23:04:54.003085+0000","last_became_peered":"2024-02-21T23:04:54.003085+0000","last_unstale":"2024-02-21T23:05:10.050580+0000","last_undegraded":"2024-02-21T23:05:10.050580+0000","last_fullsized":"2024-02-21T23:05:10.050580+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:20:56.263919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.053192+0000","last_change":"2024-02-21T23:05:08.467952+0000","last_active":"2024-02-21T23:05:10.053192+0000","last_peered":"2024-02-21T23:05:10.053192+0000","last_clean":"2024-02-21T23:05:10.053192+0000","last_became_active":"2024-02-21T23:05:08.467169+0000","last_became_peered":"2024-02-21T23:05:08.467169+0000","last_unstale":"2024-02-21T23:05:10.053192+0000","last_undegraded":"2024-02-21T23:05:10.053192+0000","last_fullsized":"2024-02-21T23:05:10.053192+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:38:14.739157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.443780+0000","last_change":"2024-02-21T23:04:53.991959+0000","last_active":"2024-02-21T23:05:09.443780+0000","last_peered":"2024-02-21T23:05:09.443780+0000","last_clean":"2024-02-21T23:05:09.443780+0000","last_became_active":"2024-02-21T23:04:53.988135+0000","last_became_peered":"2024-02-21T23:04:53.988135+0000","last_unstale":"2024-02-21T23:05:09.443780+0000","last_undegraded":"2024-02-21T23:05:09.443780+0000","last_fullsized":"2024-02-21T23:05:09.443780+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:28:35.805544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.448327+0000","last_change":"2024-02-21T23:05:08.455271+0000","last_active":"2024-02-21T23:05:09.448327+0000","last_peered":"2024-02-21T23:05:09.448327+0000","last_clean":"2024-02-21T23:05:09.448327+0000","last_became_active":"2024-02-21T23:05:08.454744+0000","last_became_peered":"2024-02-21T23:05:08.454744+0000","last_unstale":"2024-02-21T23:05:09.448327+0000","last_undegraded":"2024-02-21T23:05:09.448327+0000","last_fullsized":"2024-02-21T23:05:09.448327+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:23:00.163775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.445324+0000","last_change":"2024-02-21T23:04:53.982813+0000","last_active":"2024-02-21T23:05:09.445324+0000","last_peered":"2024-02-21T23:05:09.445324+0000","last_clean":"2024-02-21T23:05:09.445324+0000","last_became_active":"2024-02-21T23:04:53.982515+0000","last_became_peered":"2024-02-21T23:04:53.982515+0000","last_unstale":"2024-02-21T23:05:09.445324+0000","last_undegraded":"2024-02-21T23:05:09.445324+0000","last_fullsized":"2024-02-21T23:05:09.445324+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:03:03.310447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.465411+0000","last_change":"2024-02-21T23:05:08.466034+0000","last_active":"2024-02-21T23:05:09.465411+0000","last_peered":"2024-02-21T23:05:09.465411+0000","last_clean":"2024-02-21T23:05:09.465411+0000","last_became_active":"2024-02-21T23:05:08.464909+0000","last_became_peered":"2024-02-21T23:05:08.464909+0000","last_unstale":"2024-02-21T23:05:09.465411+0000","last_undegraded":"2024-02-21T23:05:09.465411+0000","last_fullsized":"2024-02-21T23:05:09.465411+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:38:50.302069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.442462+0000","last_change":"2024-02-21T23:04:53.982276+0000","last_active":"2024-02-21T23:05:09.442462+0000","last_peered":"2024-02-21T23:05:09.442462+0000","last_clean":"2024-02-21T23:05:09.442462+0000","last_became_active":"2024-02-21T23:04:53.981735+0000","last_became_peered":"2024-02-21T23:04:53.981735+0000","last_unstale":"2024-02-21T23:05:09.442462+0000","last_undegraded":"2024-02-21T23:05:09.442462+0000","last_fullsized":"2024-02-21T23:05:09.442462+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:23:27.013837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.012316+0000","last_change":"2024-02-21T23:05:08.473567+0000","last_active":"2024-02-21T23:05:10.012316+0000","last_peered":"2024-02-21T23:05:10.012316+0000","last_clean":"2024-02-21T23:05:10.012316+0000","last_became_active":"2024-02-21T23:05:08.473270+0000","last_became_peered":"2024-02-21T23:05:08.473270+0000","last_unstale":"2024-02-21T23:05:10.012316+0000","last_undegraded":"2024-02-21T23:05:10.012316+0000","last_fullsized":"2024-02-21T23:05:10.012316+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:03:07.620481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.470208+0000","last_change":"2024-02-21T23:04:53.992336+0000","last_active":"2024-02-21T23:05:09.470208+0000","last_peered":"2024-02-21T23:05:09.470208+0000","last_clean":"2024-02-21T23:05:09.470208+0000","last_became_active":"2024-02-21T23:04:53.992068+0000","last_became_peered":"2024-02-21T23:04:53.992068+0000","last_unstale":"2024-02-21T23:05:09.470208+0000","last_undegraded":"2024-02-21T23:05:09.470208+0000","last_fullsized":"2024-02-21T23:05:09.470208+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:29:50.076176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.011973+0000","last_change":"2024-02-21T23:05:08.474602+0000","last_active":"2024-02-21T23:05:10.011973+0000","last_peered":"2024-02-21T23:05:10.011973+0000","last_clean":"2024-02-21T23:05:10.011973+0000","last_became_active":"2024-02-21T23:05:08.470591+0000","last_became_peered":"2024-02-21T23:05:08.470591+0000","last_unstale":"2024-02-21T23:05:10.011973+0000","last_undegraded":"2024-02-21T23:05:10.011973+0000","last_fullsized":"2024-02-21T23:05:10.011973+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:15:17.047282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.445260+0000","last_change":"2024-02-21T23:04:53.991879+0000","last_active":"2024-02-21T23:05:09.445260+0000","last_peered":"2024-02-21T23:05:09.445260+0000","last_clean":"2024-02-21T23:05:09.445260+0000","last_became_active":"2024-02-21T23:04:53.985749+0000","last_became_peered":"2024-02-21T23:04:53.985749+0000","last_unstale":"2024-02-21T23:05:09.445260+0000","last_undegraded":"2024-02-21T23:05:09.445260+0000","last_fullsized":"2024-02-21T23:05:09.445260+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:21:11.752620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.010126+0000","last_change":"2024-02-21T23:05:08.478582+0000","last_active":"2024-02-21T23:05:10.010126+0000","last_peered":"2024-02-21T23:05:10.010126+0000","last_clean":"2024-02-21T23:05:10.010126+0000","last_became_active":"2024-02-21T23:05:08.477087+0000","last_became_peered":"2024-02-21T23:05:08.477087+0000","last_unstale":"2024-02-21T23:05:10.010126+0000","last_undegraded":"2024-02-21T23:05:10.010126+0000","last_fullsized":"2024-02-21T23:05:10.010126+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:39:36.687065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.441750+0000","last_change":"2024-02-21T23:04:53.982139+0000","last_active":"2024-02-21T23:05:09.441750+0000","last_peered":"2024-02-21T23:05:09.441750+0000","last_clean":"2024-02-21T23:05:09.441750+0000","last_became_active":"2024-02-21T23:04:53.981800+0000","last_became_peered":"2024-02-21T23:04:53.981800+0000","last_unstale":"2024-02-21T23:05:09.441750+0000","last_undegraded":"2024-02-21T23:05:09.441750+0000","last_fullsized":"2024-02-21T23:05:09.441750+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:16:59.134374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.469302+0000","last_change":"2024-02-21T23:05:08.457170+0000","last_active":"2024-02-21T23:05:09.469302+0000","last_peered":"2024-02-21T23:05:09.469302+0000","last_clean":"2024-02-21T23:05:09.469302+0000","last_became_active":"2024-02-21T23:05:08.456713+0000","last_became_peered":"2024-02-21T23:05:08.456713+0000","last_unstale":"2024-02-21T23:05:09.469302+0000","last_undegraded":"2024-02-21T23:05:09.469302+0000","last_fullsized":"2024-02-21T23:05:09.469302+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:17:30.362340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.050342+0000","last_change":"2024-02-21T23:04:54.003361+0000","last_active":"2024-02-21T23:05:10.050342+0000","last_peered":"2024-02-21T23:05:10.050342+0000","last_clean":"2024-02-21T23:05:10.050342+0000","last_became_active":"2024-02-21T23:04:54.002829+0000","last_became_peered":"2024-02-21T23:04:54.002829+0000","last_unstale":"2024-02-21T23:05:10.050342+0000","last_undegraded":"2024-02-21T23:05:10.050342+0000","last_fullsized":"2024-02-21T23:05:10.050342+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:20:06.730366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.465066+0000","last_change":"2024-02-21T23:05:08.472180+0000","last_active":"2024-02-21T23:05:09.465066+0000","last_peered":"2024-02-21T23:05:09.465066+0000","last_clean":"2024-02-21T23:05:09.465066+0000","last_became_active":"2024-02-21T23:05:08.471879+0000","last_became_peered":"2024-02-21T23:05:08.471879+0000","last_unstale":"2024-02-21T23:05:09.465066+0000","last_undegraded":"2024-02-21T23:05:09.465066+0000","last_fullsized":"2024-02-21T23:05:09.465066+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:15:50.294164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.010612+0000","last_change":"2024-02-21T23:04:53.981560+0000","last_active":"2024-02-21T23:05:10.010612+0000","last_peered":"2024-02-21T23:05:10.010612+0000","last_clean":"2024-02-21T23:05:10.010612+0000","last_became_active":"2024-02-21T23:04:53.981260+0000","last_became_peered":"2024-02-21T23:04:53.981260+0000","last_unstale":"2024-02-21T23:05:10.010612+0000","last_undegraded":"2024-02-21T23:05:10.010612+0000","last_fullsized":"2024-02-21T23:05:10.010612+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:04:13.645498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.010185+0000","last_change":"2024-02-21T23:05:08.469030+0000","last_active":"2024-02-21T23:05:10.010185+0000","last_peered":"2024-02-21T23:05:10.010185+0000","last_clean":"2024-02-21T23:05:10.010185+0000","last_became_active":"2024-02-21T23:05:08.468718+0000","last_became_peered":"2024-02-21T23:05:08.468718+0000","last_unstale":"2024-02-21T23:05:10.010185+0000","last_undegraded":"2024-02-21T23:05:10.010185+0000","last_fullsized":"2024-02-21T23:05:10.010185+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:39:49.442464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.441458+0000","last_change":"2024-02-21T23:04:53.981072+0000","last_active":"2024-02-21T23:05:09.441458+0000","last_peered":"2024-02-21T23:05:09.441458+0000","last_clean":"2024-02-21T23:05:09.441458+0000","last_became_active":"2024-02-21T23:04:53.979351+0000","last_became_peered":"2024-02-21T23:04:53.979351+0000","last_unstale":"2024-02-21T23:05:09.441458+0000","last_undegraded":"2024-02-21T23:05:09.441458+0000","last_fullsized":"2024-02-21T23:05:09.441458+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:55:24.090577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.115460+0000","last_change":"2024-02-21T23:05:08.452430+0000","last_active":"2024-02-21T23:05:10.115460+0000","last_peered":"2024-02-21T23:05:10.115460+0000","last_clean":"2024-02-21T23:05:10.115460+0000","last_became_active":"2024-02-21T23:05:08.451516+0000","last_became_peered":"2024-02-21T23:05:08.451516+0000","last_unstale":"2024-02-21T23:05:10.115460+0000","last_undegraded":"2024-02-21T23:05:10.115460+0000","last_fullsized":"2024-02-21T23:05:10.115460+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:38:01.955930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.051802+0000","last_change":"2024-02-21T23:04:53.990161+0000","last_active":"2024-02-21T23:05:10.051802+0000","last_peered":"2024-02-21T23:05:10.051802+0000","last_clean":"2024-02-21T23:05:10.051802+0000","last_became_active":"2024-02-21T23:04:53.989849+0000","last_became_peered":"2024-02-21T23:04:53.989849+0000","last_unstale":"2024-02-21T23:05:10.051802+0000","last_undegraded":"2024-02-21T23:05:10.051802+0000","last_fullsized":"2024-02-21T23:05:10.051802+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:57:04.392889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.444824+0000","last_change":"2024-02-21T23:05:08.457917+0000","last_active":"2024-02-21T23:05:09.444824+0000","last_peered":"2024-02-21T23:05:09.444824+0000","last_clean":"2024-02-21T23:05:09.444824+0000","last_became_active":"2024-02-21T23:05:08.457573+0000","last_became_peered":"2024-02-21T23:05:08.457573+0000","last_unstale":"2024-02-21T23:05:09.444824+0000","last_undegraded":"2024-02-21T23:05:09.444824+0000","last_fullsized":"2024-02-21T23:05:09.444824+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:26:18.251950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.008867+0000","last_change":"2024-02-21T23:04:53.992512+0000","last_active":"2024-02-21T23:05:10.008867+0000","last_peered":"2024-02-21T23:05:10.008867+0000","last_clean":"2024-02-21T23:05:10.008867+0000","last_became_active":"2024-02-21T23:04:53.992255+0000","last_became_peered":"2024-02-21T23:04:53.992255+0000","last_unstale":"2024-02-21T23:05:10.008867+0000","last_undegraded":"2024-02-21T23:05:10.008867+0000","last_fullsized":"2024-02-21T23:05:10.008867+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:20:56.263919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.443058+0000","last_change":"2024-02-21T23:05:08.457071+0000","last_active":"2024-02-21T23:05:09.443058+0000","last_peered":"2024-02-21T23:05:09.443058+0000","last_clean":"2024-02-21T23:05:09.443058+0000","last_became_active":"2024-02-21T23:05:08.456811+0000","last_became_peered":"2024-02-21T23:05:08.456811+0000","last_unstale":"2024-02-21T23:05:09.443058+0000","last_undegraded":"2024-02-21T23:05:09.443058+0000","last_fullsized":"2024-02-21T23:05:09.443058+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:19:02.249738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.051837+0000","last_change":"2024-02-21T23:04:53.985272+0000","last_active":"2024-02-21T23:05:10.051837+0000","last_peered":"2024-02-21T23:05:10.051837+0000","last_clean":"2024-02-21T23:05:10.051837+0000","last_became_active":"2024-02-21T23:04:53.984967+0000","last_became_peered":"2024-02-21T23:04:53.984967+0000","last_unstale":"2024-02-21T23:05:10.051837+0000","last_undegraded":"2024-02-21T23:05:10.051837+0000","last_fullsized":"2024-02-21T23:05:10.051837+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:10:20.341856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.448291+0000","last_change":"2024-02-21T23:05:08.450209+0000","last_active":"2024-02-21T23:05:09.448291+0000","last_peered":"2024-02-21T23:05:09.448291+0000","last_clean":"2024-02-21T23:05:09.448291+0000","last_became_active":"2024-02-21T23:05:08.449874+0000","last_became_peered":"2024-02-21T23:05:08.449874+0000","last_unstale":"2024-02-21T23:05:09.448291+0000","last_undegraded":"2024-02-21T23:05:09.448291+0000","last_fullsized":"2024-02-21T23:05:09.448291+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:10:34.806470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.113487+0000","last_change":"2024-02-21T23:04:53.985751+0000","last_active":"2024-02-21T23:05:10.113487+0000","last_peered":"2024-02-21T23:05:10.113487+0000","last_clean":"2024-02-21T23:05:10.113487+0000","last_became_active":"2024-02-21T23:04:53.983212+0000","last_became_peered":"2024-02-21T23:04:53.983212+0000","last_unstale":"2024-02-21T23:05:10.113487+0000","last_undegraded":"2024-02-21T23:05:10.113487+0000","last_fullsized":"2024-02-21T23:05:10.113487+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:16:59.134374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.443981+0000","last_change":"2024-02-21T23:05:08.465137+0000","last_active":"2024-02-21T23:05:09.443981+0000","last_peered":"2024-02-21T23:05:09.443981+0000","last_clean":"2024-02-21T23:05:09.443981+0000","last_became_active":"2024-02-21T23:05:08.464713+0000","last_became_peered":"2024-02-21T23:05:08.464713+0000","last_unstale":"2024-02-21T23:05:09.443981+0000","last_undegraded":"2024-02-21T23:05:09.443981+0000","last_fullsized":"2024-02-21T23:05:09.443981+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:33:31.480704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.115069+0000","last_change":"2024-02-21T23:04:53.984599+0000","last_active":"2024-02-21T23:05:10.115069+0000","last_peered":"2024-02-21T23:05:10.115069+0000","last_clean":"2024-02-21T23:05:10.115069+0000","last_became_active":"2024-02-21T23:04:53.983405+0000","last_became_peered":"2024-02-21T23:04:53.983405+0000","last_unstale":"2024-02-21T23:05:10.115069+0000","last_undegraded":"2024-02-21T23:05:10.115069+0000","last_fullsized":"2024-02-21T23:05:10.115069+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:59:35.227750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.009702+0000","last_change":"2024-02-21T23:05:08.470094+0000","last_active":"2024-02-21T23:05:10.009702+0000","last_peered":"2024-02-21T23:05:10.009702+0000","last_clean":"2024-02-21T23:05:10.009702+0000","last_became_active":"2024-02-21T23:05:08.469832+0000","last_became_peered":"2024-02-21T23:05:08.469832+0000","last_unstale":"2024-02-21T23:05:10.009702+0000","last_undegraded":"2024-02-21T23:05:10.009702+0000","last_fullsized":"2024-02-21T23:05:10.009702+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:44:33.158628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.444550+0000","last_change":"2024-02-21T23:04:53.988824+0000","last_active":"2024-02-21T23:05:09.444550+0000","last_peered":"2024-02-21T23:05:09.444550+0000","last_clean":"2024-02-21T23:05:09.444550+0000","last_became_active":"2024-02-21T23:04:53.988190+0000","last_became_peered":"2024-02-21T23:04:53.988190+0000","last_unstale":"2024-02-21T23:05:09.444550+0000","last_undegraded":"2024-02-21T23:05:09.444550+0000","last_fullsized":"2024-02-21T23:05:09.444550+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:59:41.759091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.465354+0000","last_change":"2024-02-21T23:05:08.462083+0000","last_active":"2024-02-21T23:05:09.465354+0000","last_peered":"2024-02-21T23:05:09.465354+0000","last_clean":"2024-02-21T23:05:09.465354+0000","last_became_active":"2024-02-21T23:05:08.461812+0000","last_became_peered":"2024-02-21T23:05:08.461812+0000","last_unstale":"2024-02-21T23:05:09.465354+0000","last_undegraded":"2024-02-21T23:05:09.465354+0000","last_fullsized":"2024-02-21T23:05:09.465354+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:38:14.739157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.469558+0000","last_change":"2024-02-21T23:04:53.989836+0000","last_active":"2024-02-21T23:05:09.469558+0000","last_peered":"2024-02-21T23:05:09.469558+0000","last_clean":"2024-02-21T23:05:09.469558+0000","last_became_active":"2024-02-21T23:04:53.989511+0000","last_became_peered":"2024-02-21T23:04:53.989511+0000","last_unstale":"2024-02-21T23:05:09.469558+0000","last_undegraded":"2024-02-21T23:05:09.469558+0000","last_fullsized":"2024-02-21T23:05:09.469558+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:27:55.324492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.444355+0000","last_change":"2024-02-21T23:05:08.471477+0000","last_active":"2024-02-21T23:05:09.444355+0000","last_peered":"2024-02-21T23:05:09.444355+0000","last_clean":"2024-02-21T23:05:09.444355+0000","last_became_active":"2024-02-21T23:05:08.470891+0000","last_became_peered":"2024-02-21T23:05:08.470891+0000","last_unstale":"2024-02-21T23:05:09.444355+0000","last_undegraded":"2024-02-21T23:05:09.444355+0000","last_fullsized":"2024-02-21T23:05:09.444355+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:58:12.890553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.051035+0000","last_change":"2024-02-21T23:04:53.980319+0000","last_active":"2024-02-21T23:05:10.051035+0000","last_peered":"2024-02-21T23:05:10.051035+0000","last_clean":"2024-02-21T23:05:10.051035+0000","last_became_active":"2024-02-21T23:04:53.980056+0000","last_became_peered":"2024-02-21T23:04:53.980056+0000","last_unstale":"2024-02-21T23:05:10.051035+0000","last_undegraded":"2024-02-21T23:05:10.051035+0000","last_fullsized":"2024-02-21T23:05:10.051035+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:27:08.923372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.447758+0000","last_change":"2024-02-21T23:05:08.450459+0000","last_active":"2024-02-21T23:05:09.447758+0000","last_peered":"2024-02-21T23:05:09.447758+0000","last_clean":"2024-02-21T23:05:09.447758+0000","last_became_active":"2024-02-21T23:05:08.450099+0000","last_became_peered":"2024-02-21T23:05:08.450099+0000","last_unstale":"2024-02-21T23:05:09.447758+0000","last_undegraded":"2024-02-21T23:05:09.447758+0000","last_fullsized":"2024-02-21T23:05:09.447758+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:27:31.838991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.050787+0000","last_change":"2024-02-21T23:04:53.994043+0000","last_active":"2024-02-21T23:05:10.050787+0000","last_peered":"2024-02-21T23:05:10.050787+0000","last_clean":"2024-02-21T23:05:10.050787+0000","last_became_active":"2024-02-21T23:04:53.993693+0000","last_became_peered":"2024-02-21T23:04:53.993693+0000","last_unstale":"2024-02-21T23:05:10.050787+0000","last_undegraded":"2024-02-21T23:05:10.050787+0000","last_fullsized":"2024-02-21T23:05:10.050787+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:33:53.507158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.442036+0000","last_change":"2024-02-21T23:05:08.453713+0000","last_active":"2024-02-21T23:05:09.442036+0000","last_peered":"2024-02-21T23:05:09.442036+0000","last_clean":"2024-02-21T23:05:09.442036+0000","last_became_active":"2024-02-21T23:05:08.453119+0000","last_became_peered":"2024-02-21T23:05:08.453119+0000","last_unstale":"2024-02-21T23:05:09.442036+0000","last_undegraded":"2024-02-21T23:05:09.442036+0000","last_fullsized":"2024-02-21T23:05:09.442036+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:47:50.018715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.445216+0000","last_change":"2024-02-21T23:04:53.994446+0000","last_active":"2024-02-21T23:05:09.445216+0000","last_peered":"2024-02-21T23:05:09.445216+0000","last_clean":"2024-02-21T23:05:09.445216+0000","last_became_active":"2024-02-21T23:04:53.994181+0000","last_became_peered":"2024-02-21T23:04:53.994181+0000","last_unstale":"2024-02-21T23:05:09.445216+0000","last_undegraded":"2024-02-21T23:05:09.445216+0000","last_fullsized":"2024-02-21T23:05:09.445216+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:18:47.785124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.465659+0000","last_change":"2024-02-21T23:05:08.460708+0000","last_active":"2024-02-21T23:05:09.465659+0000","last_peered":"2024-02-21T23:05:09.465659+0000","last_clean":"2024-02-21T23:05:09.465659+0000","last_became_active":"2024-02-21T23:05:08.460363+0000","last_became_peered":"2024-02-21T23:05:08.460363+0000","last_unstale":"2024-02-21T23:05:09.465659+0000","last_undegraded":"2024-02-21T23:05:09.465659+0000","last_fullsized":"2024-02-21T23:05:09.465659+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:56:07.508974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.442442+0000","last_change":"2024-02-21T23:04:53.976184+0000","last_active":"2024-02-21T23:05:09.442442+0000","last_peered":"2024-02-21T23:05:09.442442+0000","last_clean":"2024-02-21T23:05:09.442442+0000","last_became_active":"2024-02-21T23:04:53.975098+0000","last_became_peered":"2024-02-21T23:04:53.975098+0000","last_unstale":"2024-02-21T23:05:09.442442+0000","last_undegraded":"2024-02-21T23:05:09.442442+0000","last_fullsized":"2024-02-21T23:05:09.442442+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:39:16.498706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.465310+0000","last_change":"2024-02-21T23:05:08.459982+0000","last_active":"2024-02-21T23:05:09.465310+0000","last_peered":"2024-02-21T23:05:09.465310+0000","last_clean":"2024-02-21T23:05:09.465310+0000","last_became_active":"2024-02-21T23:05:08.459705+0000","last_became_peered":"2024-02-21T23:05:08.459705+0000","last_unstale":"2024-02-21T23:05:09.465310+0000","last_undegraded":"2024-02-21T23:05:09.465310+0000","last_fullsized":"2024-02-21T23:05:09.465310+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:03:07.620481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.113482+0000","last_change":"2024-02-21T23:04:53.995281+0000","last_active":"2024-02-21T23:05:10.113482+0000","last_peered":"2024-02-21T23:05:10.113482+0000","last_clean":"2024-02-21T23:05:10.113482+0000","last_became_active":"2024-02-21T23:04:53.994940+0000","last_became_peered":"2024-02-21T23:04:53.994940+0000","last_unstale":"2024-02-21T23:05:10.113482+0000","last_undegraded":"2024-02-21T23:05:10.113482+0000","last_fullsized":"2024-02-21T23:05:10.113482+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:55:04.908945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.445670+0000","last_change":"2024-02-21T23:05:08.446116+0000","last_active":"2024-02-21T23:05:09.445670+0000","last_peered":"2024-02-21T23:05:09.445670+0000","last_clean":"2024-02-21T23:05:09.445670+0000","last_became_active":"2024-02-21T23:05:08.445654+0000","last_became_peered":"2024-02-21T23:05:08.445654+0000","last_unstale":"2024-02-21T23:05:09.445670+0000","last_undegraded":"2024-02-21T23:05:09.445670+0000","last_fullsized":"2024-02-21T23:05:09.445670+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:52:57.967157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.052812+0000","last_change":"2024-02-21T23:04:53.972067+0000","last_active":"2024-02-21T23:05:10.052812+0000","last_peered":"2024-02-21T23:05:10.052812+0000","last_clean":"2024-02-21T23:05:10.052812+0000","last_became_active":"2024-02-21T23:04:53.971855+0000","last_became_peered":"2024-02-21T23:04:53.971855+0000","last_unstale":"2024-02-21T23:05:10.052812+0000","last_undegraded":"2024-02-21T23:05:10.052812+0000","last_fullsized":"2024-02-21T23:05:10.052812+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:40:20.534202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.051308+0000","last_change":"2024-02-21T23:05:08.447052+0000","last_active":"2024-02-21T23:05:10.051308+0000","last_peered":"2024-02-21T23:05:10.051308+0000","last_clean":"2024-02-21T23:05:10.051308+0000","last_became_active":"2024-02-21T23:05:08.446685+0000","last_became_peered":"2024-02-21T23:05:08.446685+0000","last_unstale":"2024-02-21T23:05:10.051308+0000","last_undegraded":"2024-02-21T23:05:10.051308+0000","last_fullsized":"2024-02-21T23:05:10.051308+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:33:22.889577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.008506+0000","last_change":"2024-02-21T23:04:53.983537+0000","last_active":"2024-02-21T23:05:10.008506+0000","last_peered":"2024-02-21T23:05:10.008506+0000","last_clean":"2024-02-21T23:05:10.008506+0000","last_became_active":"2024-02-21T23:04:53.983264+0000","last_became_peered":"2024-02-21T23:04:53.983264+0000","last_unstale":"2024-02-21T23:05:10.008506+0000","last_undegraded":"2024-02-21T23:05:10.008506+0000","last_fullsized":"2024-02-21T23:05:10.008506+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:36:35.426082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.011463+0000","last_change":"2024-02-21T23:05:08.475581+0000","last_active":"2024-02-21T23:05:10.011463+0000","last_peered":"2024-02-21T23:05:10.011463+0000","last_clean":"2024-02-21T23:05:10.011463+0000","last_became_active":"2024-02-21T23:05:08.475278+0000","last_became_peered":"2024-02-21T23:05:08.475278+0000","last_unstale":"2024-02-21T23:05:10.011463+0000","last_undegraded":"2024-02-21T23:05:10.011463+0000","last_fullsized":"2024-02-21T23:05:10.011463+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:00:36.493162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.050307+0000","last_change":"2024-02-21T23:04:54.002341+0000","last_active":"2024-02-21T23:05:10.050307+0000","last_peered":"2024-02-21T23:05:10.050307+0000","last_clean":"2024-02-21T23:05:10.050307+0000","last_became_active":"2024-02-21T23:04:54.002066+0000","last_became_peered":"2024-02-21T23:04:54.002066+0000","last_unstale":"2024-02-21T23:05:10.050307+0000","last_undegraded":"2024-02-21T23:05:10.050307+0000","last_fullsized":"2024-02-21T23:05:10.050307+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:51:12.681873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.465004+0000","last_change":"2024-02-21T23:05:08.478547+0000","last_active":"2024-02-21T23:05:09.465004+0000","last_peered":"2024-02-21T23:05:09.465004+0000","last_clean":"2024-02-21T23:05:09.465004+0000","last_became_active":"2024-02-21T23:05:08.478086+0000","last_became_peered":"2024-02-21T23:05:08.478086+0000","last_unstale":"2024-02-21T23:05:09.465004+0000","last_undegraded":"2024-02-21T23:05:09.465004+0000","last_fullsized":"2024-02-21T23:05:09.465004+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:15:17.047282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.465167+0000","last_change":"2024-02-21T23:04:53.996395+0000","last_active":"2024-02-21T23:05:09.465167+0000","last_peered":"2024-02-21T23:05:09.465167+0000","last_clean":"2024-02-21T23:05:09.465167+0000","last_became_active":"2024-02-21T23:04:53.996219+0000","last_became_peered":"2024-02-21T23:04:53.996219+0000","last_unstale":"2024-02-21T23:05:09.465167+0000","last_undegraded":"2024-02-21T23:05:09.465167+0000","last_fullsized":"2024-02-21T23:05:09.465167+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:27:17.374378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.114311+0000","last_change":"2024-02-21T23:05:08.456218+0000","last_active":"2024-02-21T23:05:10.114311+0000","last_peered":"2024-02-21T23:05:10.114311+0000","last_clean":"2024-02-21T23:05:10.114311+0000","last_became_active":"2024-02-21T23:05:08.455146+0000","last_became_peered":"2024-02-21T23:05:08.455146+0000","last_unstale":"2024-02-21T23:05:10.114311+0000","last_undegraded":"2024-02-21T23:05:10.114311+0000","last_fullsized":"2024-02-21T23:05:10.114311+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:53:10.262900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.009658+0000","last_change":"2024-02-21T23:05:07.437412+0000","last_active":"2024-02-21T23:05:10.009658+0000","last_peered":"2024-02-21T23:05:10.009658+0000","last_clean":"2024-02-21T23:05:10.009658+0000","last_became_active":"2024-02-21T23:05:07.437123+0000","last_became_peered":"2024-02-21T23:05:07.437123+0000","last_unstale":"2024-02-21T23:05:10.009658+0000","last_undegraded":"2024-02-21T23:05:10.009658+0000","last_fullsized":"2024-02-21T23:05:10.009658+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_clean_scrub_stamp":"2024-02-21T23:05:06.380814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:59:55.210541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.008430+0000","last_change":"2024-02-21T23:04:53.991752+0000","last_active":"2024-02-21T23:05:10.008430+0000","last_peered":"2024-02-21T23:05:10.008430+0000","last_clean":"2024-02-21T23:05:10.008430+0000","last_became_active":"2024-02-21T23:04:53.991458+0000","last_became_peered":"2024-02-21T23:04:53.991458+0000","last_unstale":"2024-02-21T23:05:10.008430+0000","last_undegraded":"2024-02-21T23:05:10.008430+0000","last_fullsized":"2024-02-21T23:05:10.008430+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:33:34.319772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"52'4","reported_seq":26,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.115478+0000","last_change":"2024-02-21T23:04:52.982273+0000","last_active":"2024-02-21T23:05:10.115478+0000","last_peered":"2024-02-21T23:05:10.115478+0000","last_clean":"2024-02-21T23:05:10.115478+0000","last_became_active":"2024-02-21T23:04:52.981799+0000","last_became_peered":"2024-02-21T23:04:52.981799+0000","last_unstale":"2024-02-21T23:05:10.115478+0000","last_undegraded":"2024-02-21T23:05:10.115478+0000","last_fullsized":"2024-02-21T23:05:10.115478+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:18:03.518976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.443242+0000","last_change":"2024-02-21T23:05:08.474827+0000","last_active":"2024-02-21T23:05:09.443242+0000","last_peered":"2024-02-21T23:05:09.443242+0000","last_clean":"2024-02-21T23:05:09.443242+0000","last_became_active":"2024-02-21T23:05:08.474160+0000","last_became_peered":"2024-02-21T23:05:08.474160+0000","last_unstale":"2024-02-21T23:05:09.443242+0000","last_undegraded":"2024-02-21T23:05:09.443242+0000","last_fullsized":"2024-02-21T23:05:09.443242+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:05:35.219005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"56'8","reported_seq":22,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.445359+0000","last_change":"2024-02-21T23:05:07.439772+0000","last_active":"2024-02-21T23:05:09.445359+0000","last_peered":"2024-02-21T23:05:09.445359+0000","last_clean":"2024-02-21T23:05:09.445359+0000","last_became_active":"2024-02-21T23:05:07.439495+0000","last_became_peered":"2024-02-21T23:05:07.439495+0000","last_unstale":"2024-02-21T23:05:09.445359+0000","last_undegraded":"2024-02-21T23:05:09.445359+0000","last_fullsized":"2024-02-21T23:05:09.445359+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_clean_scrub_stamp":"2024-02-21T23:05:06.380814+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:46:57.938662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.465114+0000","last_change":"2024-02-21T23:04:53.991059+0000","last_active":"2024-02-21T23:05:09.465114+0000","last_peered":"2024-02-21T23:05:09.465114+0000","last_clean":"2024-02-21T23:05:09.465114+0000","last_became_active":"2024-02-21T23:04:53.990310+0000","last_became_peered":"2024-02-21T23:04:53.990310+0000","last_unstale":"2024-02-21T23:05:09.465114+0000","last_undegraded":"2024-02-21T23:05:09.465114+0000","last_fullsized":"2024-02-21T23:05:09.465114+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:12:19.379347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"52'1","reported_seq":23,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.442553+0000","last_change":"2024-02-21T23:04:52.971612+0000","last_active":"2024-02-21T23:05:09.442553+0000","last_peered":"2024-02-21T23:05:09.442553+0000","last_clean":"2024-02-21T23:05:09.442553+0000","last_became_active":"2024-02-21T23:04:52.971220+0000","last_became_peered":"2024-02-21T23:04:52.971220+0000","last_unstale":"2024-02-21T23:05:09.442553+0000","last_undegraded":"2024-02-21T23:05:09.442553+0000","last_fullsized":"2024-02-21T23:05:09.442553+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:18:03.518976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.445536+0000","last_change":"2024-02-21T23:05:08.468473+0000","last_active":"2024-02-21T23:05:09.445536+0000","last_peered":"2024-02-21T23:05:09.445536+0000","last_clean":"2024-02-21T23:05:09.445536+0000","last_became_active":"2024-02-21T23:05:08.467235+0000","last_became_peered":"2024-02-21T23:05:08.467235+0000","last_unstale":"2024-02-21T23:05:09.445536+0000","last_undegraded":"2024-02-21T23:05:09.445536+0000","last_fullsized":"2024-02-21T23:05:09.445536+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:20:21.194980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"56'2","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.011277+0000","last_change":"2024-02-21T23:05:07.447016+0000","last_active":"2024-02-21T23:05:10.011277+0000","last_peered":"2024-02-21T23:05:10.011277+0000","last_clean":"2024-02-21T23:05:10.011277+0000","last_became_active":"2024-02-21T23:05:07.446706+0000","last_became_peered":"2024-02-21T23:05:07.446706+0000","last_unstale":"2024-02-21T23:05:10.011277+0000","last_undegraded":"2024-02-21T23:05:10.011277+0000","last_fullsized":"2024-02-21T23:05:10.011277+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_clean_scrub_stamp":"2024-02-21T23:05:06.380814+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:28:49.256993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.114423+0000","last_change":"2024-02-21T23:04:53.988422+0000","last_active":"2024-02-21T23:05:10.114423+0000","last_peered":"2024-02-21T23:05:10.114423+0000","last_clean":"2024-02-21T23:05:10.114423+0000","last_became_active":"2024-02-21T23:04:53.987586+0000","last_became_peered":"2024-02-21T23:04:53.987586+0000","last_unstale":"2024-02-21T23:05:10.114423+0000","last_undegraded":"2024-02-21T23:05:10.114423+0000","last_fullsized":"2024-02-21T23:05:10.114423+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:06:08.079155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"52'2","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.442543+0000","last_change":"2024-02-21T23:04:52.983315+0000","last_active":"2024-02-21T23:05:09.442543+0000","last_peered":"2024-02-21T23:05:09.442543+0000","last_clean":"2024-02-21T23:05:09.442543+0000","last_became_active":"2024-02-21T23:04:52.982925+0000","last_became_peered":"2024-02-21T23:04:52.982925+0000","last_unstale":"2024-02-21T23:05:09.442543+0000","last_undegraded":"2024-02-21T23:05:09.442543+0000","last_fullsized":"2024-02-21T23:05:09.442543+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:04:02.374574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.465485+0000","last_change":"2024-02-21T23:05:08.461733+0000","last_active":"2024-02-21T23:05:09.465485+0000","last_peered":"2024-02-21T23:05:09.465485+0000","last_clean":"2024-02-21T23:05:09.465485+0000","last_became_active":"2024-02-21T23:05:08.461410+0000","last_became_peered":"2024-02-21T23:05:08.461410+0000","last_unstale":"2024-02-21T23:05:09.465485+0000","last_undegraded":"2024-02-21T23:05:09.465485+0000","last_fullsized":"2024-02-21T23:05:09.465485+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:25:56.439594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.465471+0000","last_change":"2024-02-21T23:05:07.442679+0000","last_active":"2024-02-21T23:05:09.465471+0000","last_peered":"2024-02-21T23:05:09.465471+0000","last_clean":"2024-02-21T23:05:09.465471+0000","last_became_active":"2024-02-21T23:05:07.442407+0000","last_became_peered":"2024-02-21T23:05:07.442407+0000","last_unstale":"2024-02-21T23:05:09.465471+0000","last_undegraded":"2024-02-21T23:05:09.465471+0000","last_fullsized":"2024-02-21T23:05:09.465471+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_clean_scrub_stamp":"2024-02-21T23:05:06.380814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:21:25.204069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.444794+0000","last_change":"2024-02-21T23:04:53.997493+0000","last_active":"2024-02-21T23:05:09.444794+0000","last_peered":"2024-02-21T23:05:09.444794+0000","last_clean":"2024-02-21T23:05:09.444794+0000","last_became_active":"2024-02-21T23:04:53.997216+0000","last_became_peered":"2024-02-21T23:04:53.997216+0000","last_unstale":"2024-02-21T23:05:09.444794+0000","last_undegraded":"2024-02-21T23:05:09.444794+0000","last_fullsized":"2024-02-21T23:05:09.444794+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:52:32.070580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"52'2","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.465914+0000","last_change":"2024-02-21T23:04:52.972332+0000","last_active":"2024-02-21T23:05:09.465914+0000","last_peered":"2024-02-21T23:05:09.465914+0000","last_clean":"2024-02-21T23:05:09.465914+0000","last_became_active":"2024-02-21T23:04:52.972014+0000","last_became_peered":"2024-02-21T23:04:52.972014+0000","last_unstale":"2024-02-21T23:05:09.465914+0000","last_undegraded":"2024-02-21T23:05:09.465914+0000","last_fullsized":"2024-02-21T23:05:09.465914+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:46:43.465994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.115753+0000","last_change":"2024-02-21T23:05:08.462349+0000","last_active":"2024-02-21T23:05:10.115753+0000","last_peered":"2024-02-21T23:05:10.115753+0000","last_clean":"2024-02-21T23:05:10.115753+0000","last_became_active":"2024-02-21T23:05:08.459568+0000","last_became_peered":"2024-02-21T23:05:08.459568+0000","last_unstale":"2024-02-21T23:05:10.115753+0000","last_undegraded":"2024-02-21T23:05:10.115753+0000","last_fullsized":"2024-02-21T23:05:10.115753+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:19:02.249738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.15","version":"56'3","reported_seq":17,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.012202+0000","last_change":"2024-02-21T23:05:07.440258+0000","last_active":"2024-02-21T23:05:10.012202+0000","last_peered":"2024-02-21T23:05:10.012202+0000","last_clean":"2024-02-21T23:05:10.012202+0000","last_became_active":"2024-02-21T23:05:07.439978+0000","last_became_peered":"2024-02-21T23:05:07.439978+0000","last_unstale":"2024-02-21T23:05:10.012202+0000","last_undegraded":"2024-02-21T23:05:10.012202+0000","last_fullsized":"2024-02-21T23:05:10.012202+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_clean_scrub_stamp":"2024-02-21T23:05:06.380814+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:47:49.005551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.447761+0000","last_change":"2024-02-21T23:04:53.988566+0000","last_active":"2024-02-21T23:05:09.447761+0000","last_peered":"2024-02-21T23:05:09.447761+0000","last_clean":"2024-02-21T23:05:09.447761+0000","last_became_active":"2024-02-21T23:04:53.988271+0000","last_became_peered":"2024-02-21T23:04:53.988271+0000","last_unstale":"2024-02-21T23:05:09.447761+0000","last_undegraded":"2024-02-21T23:05:09.447761+0000","last_fullsized":"2024-02-21T23:05:09.447761+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:59:35.227750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"52'1","reported_seq":23,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.053549+0000","last_change":"2024-02-21T23:04:52.972962+0000","last_active":"2024-02-21T23:05:10.053549+0000","last_peered":"2024-02-21T23:05:10.053549+0000","last_clean":"2024-02-21T23:05:10.053549+0000","last_became_active":"2024-02-21T23:04:52.972642+0000","last_became_peered":"2024-02-21T23:04:52.972642+0000","last_unstale":"2024-02-21T23:05:10.053549+0000","last_undegraded":"2024-02-21T23:05:10.053549+0000","last_fullsized":"2024-02-21T23:05:10.053549+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:02:52.870656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.448579+0000","last_change":"2024-02-21T23:05:08.450521+0000","last_active":"2024-02-21T23:05:09.448579+0000","last_peered":"2024-02-21T23:05:09.448579+0000","last_clean":"2024-02-21T23:05:09.448579+0000","last_became_active":"2024-02-21T23:05:08.450019+0000","last_became_peered":"2024-02-21T23:05:08.450019+0000","last_unstale":"2024-02-21T23:05:09.448579+0000","last_undegraded":"2024-02-21T23:05:09.448579+0000","last_fullsized":"2024-02-21T23:05:09.448579+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:18:12.721687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"56'6","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.442142+0000","last_change":"2024-02-21T23:05:07.436244+0000","last_active":"2024-02-21T23:05:09.442142+0000","last_peered":"2024-02-21T23:05:09.442142+0000","last_clean":"2024-02-21T23:05:09.442142+0000","last_became_active":"2024-02-21T23:05:07.436009+0000","last_became_peered":"2024-02-21T23:05:07.436009+0000","last_unstale":"2024-02-21T23:05:09.442142+0000","last_undegraded":"2024-02-21T23:05:09.442142+0000","last_fullsized":"2024-02-21T23:05:09.442142+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_clean_scrub_stamp":"2024-02-21T23:05:06.380814+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:51:26.133323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.009114+0000","last_change":"2024-02-21T23:04:53.982723+0000","last_active":"2024-02-21T23:05:10.009114+0000","last_peered":"2024-02-21T23:05:10.009114+0000","last_clean":"2024-02-21T23:05:10.009114+0000","last_became_active":"2024-02-21T23:04:53.982447+0000","last_became_peered":"2024-02-21T23:04:53.982447+0000","last_unstale":"2024-02-21T23:05:10.009114+0000","last_undegraded":"2024-02-21T23:05:10.009114+0000","last_fullsized":"2024-02-21T23:05:10.009114+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:48:09.768408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"52'1","reported_seq":23,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.446005+0000","last_change":"2024-02-21T23:04:52.977866+0000","last_active":"2024-02-21T23:05:09.446005+0000","last_peered":"2024-02-21T23:05:09.446005+0000","last_clean":"2024-02-21T23:05:09.446005+0000","last_became_active":"2024-02-21T23:04:52.977477+0000","last_became_peered":"2024-02-21T23:04:52.977477+0000","last_unstale":"2024-02-21T23:05:09.446005+0000","last_undegraded":"2024-02-21T23:05:09.446005+0000","last_fullsized":"2024-02-21T23:05:09.446005+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:25:58.605620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.442114+0000","last_change":"2024-02-21T23:05:08.445605+0000","last_active":"2024-02-21T23:05:09.442114+0000","last_peered":"2024-02-21T23:05:09.442114+0000","last_clean":"2024-02-21T23:05:09.442114+0000","last_became_active":"2024-02-21T23:05:08.445269+0000","last_became_peered":"2024-02-21T23:05:08.445269+0000","last_unstale":"2024-02-21T23:05:09.442114+0000","last_undegraded":"2024-02-21T23:05:09.442114+0000","last_fullsized":"2024-02-21T23:05:09.442114+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:38:01.955930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.012753+0000","last_change":"2024-02-21T23:05:07.442036+0000","last_active":"2024-02-21T23:05:10.012753+0000","last_peered":"2024-02-21T23:05:10.012753+0000","last_clean":"2024-02-21T23:05:10.012753+0000","last_became_active":"2024-02-21T23:05:07.441674+0000","last_became_peered":"2024-02-21T23:05:07.441674+0000","last_unstale":"2024-02-21T23:05:10.012753+0000","last_undegraded":"2024-02-21T23:05:10.012753+0000","last_fullsized":"2024-02-21T23:05:10.012753+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_clean_scrub_stamp":"2024-02-21T23:05:06.380814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:34:06.958608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.445337+0000","last_change":"2024-02-21T23:04:53.983494+0000","last_active":"2024-02-21T23:05:09.445337+0000","last_peered":"2024-02-21T23:05:09.445337+0000","last_clean":"2024-02-21T23:05:09.445337+0000","last_became_active":"2024-02-21T23:04:53.983213+0000","last_became_peered":"2024-02-21T23:04:53.983213+0000","last_unstale":"2024-02-21T23:05:09.445337+0000","last_undegraded":"2024-02-21T23:05:09.445337+0000","last_fullsized":"2024-02-21T23:05:09.445337+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:53:11.354446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":22,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.114211+0000","last_change":"2024-02-21T23:04:52.976990+0000","last_active":"2024-02-21T23:05:10.114211+0000","last_peered":"2024-02-21T23:05:10.114211+0000","last_clean":"2024-02-21T23:05:10.114211+0000","last_became_active":"2024-02-21T23:04:52.975637+0000","last_became_peered":"2024-02-21T23:04:52.975637+0000","last_unstale":"2024-02-21T23:05:10.114211+0000","last_undegraded":"2024-02-21T23:05:10.114211+0000","last_fullsized":"2024-02-21T23:05:10.114211+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:04:02.374574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.445717+0000","last_change":"2024-02-21T23:05:08.455454+0000","last_active":"2024-02-21T23:05:09.445717+0000","last_peered":"2024-02-21T23:05:09.445717+0000","last_clean":"2024-02-21T23:05:09.445717+0000","last_became_active":"2024-02-21T23:05:08.455046+0000","last_became_peered":"2024-02-21T23:05:08.455046+0000","last_unstale":"2024-02-21T23:05:09.445717+0000","last_undegraded":"2024-02-21T23:05:09.445717+0000","last_fullsized":"2024-02-21T23:05:09.445717+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:56:23.271636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.447921+0000","last_change":"2024-02-21T23:05:07.441101+0000","last_active":"2024-02-21T23:05:09.447921+0000","last_peered":"2024-02-21T23:05:09.447921+0000","last_clean":"2024-02-21T23:05:09.447921+0000","last_became_active":"2024-02-21T23:05:07.440782+0000","last_became_peered":"2024-02-21T23:05:07.440782+0000","last_unstale":"2024-02-21T23:05:09.447921+0000","last_undegraded":"2024-02-21T23:05:09.447921+0000","last_fullsized":"2024-02-21T23:05:09.447921+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_clean_scrub_stamp":"2024-02-21T23:05:06.380814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:39:29.950155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.012440+0000","last_change":"2024-02-21T23:04:53.981493+0000","last_active":"2024-02-21T23:05:10.012440+0000","last_peered":"2024-02-21T23:05:10.012440+0000","last_clean":"2024-02-21T23:05:10.012440+0000","last_became_active":"2024-02-21T23:04:53.981109+0000","last_became_peered":"2024-02-21T23:04:53.981109+0000","last_unstale":"2024-02-21T23:05:10.012440+0000","last_undegraded":"2024-02-21T23:05:10.012440+0000","last_fullsized":"2024-02-21T23:05:10.012440+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:09:29.344651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":22,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.052501+0000","last_change":"2024-02-21T23:04:52.971808+0000","last_active":"2024-02-21T23:05:10.052501+0000","last_peered":"2024-02-21T23:05:10.052501+0000","last_clean":"2024-02-21T23:05:10.052501+0000","last_became_active":"2024-02-21T23:04:52.971529+0000","last_became_peered":"2024-02-21T23:04:52.971529+0000","last_unstale":"2024-02-21T23:05:10.052501+0000","last_undegraded":"2024-02-21T23:05:10.052501+0000","last_fullsized":"2024-02-21T23:05:10.052501+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:00:20.232950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.114772+0000","last_change":"2024-02-21T23:05:08.454743+0000","last_active":"2024-02-21T23:05:10.114772+0000","last_peered":"2024-02-21T23:05:10.114772+0000","last_clean":"2024-02-21T23:05:10.114772+0000","last_became_active":"2024-02-21T23:05:08.451327+0000","last_became_peered":"2024-02-21T23:05:08.451327+0000","last_unstale":"2024-02-21T23:05:10.114772+0000","last_undegraded":"2024-02-21T23:05:10.114772+0000","last_fullsized":"2024-02-21T23:05:10.114772+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:34:07.971772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.444192+0000","last_change":"2024-02-21T23:05:07.444788+0000","last_active":"2024-02-21T23:05:09.444192+0000","last_peered":"2024-02-21T23:05:09.444192+0000","last_clean":"2024-02-21T23:05:09.444192+0000","last_became_active":"2024-02-21T23:05:07.444463+0000","last_became_peered":"2024-02-21T23:05:07.444463+0000","last_unstale":"2024-02-21T23:05:09.444192+0000","last_undegraded":"2024-02-21T23:05:09.444192+0000","last_fullsized":"2024-02-21T23:05:09.444192+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_clean_scrub_stamp":"2024-02-21T23:05:06.380814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:56:06.495810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.445696+0000","last_change":"2024-02-21T23:04:53.994186+0000","last_active":"2024-02-21T23:05:09.445696+0000","last_peered":"2024-02-21T23:05:09.445696+0000","last_clean":"2024-02-21T23:05:09.445696+0000","last_became_active":"2024-02-21T23:04:53.993902+0000","last_became_peered":"2024-02-21T23:04:53.993902+0000","last_unstale":"2024-02-21T23:05:09.445696+0000","last_undegraded":"2024-02-21T23:05:09.445696+0000","last_fullsized":"2024-02-21T23:05:09.445696+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:39:11.623072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":22,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.012823+0000","last_change":"2024-02-21T23:04:52.976339+0000","last_active":"2024-02-21T23:05:10.012823+0000","last_peered":"2024-02-21T23:05:10.012823+0000","last_clean":"2024-02-21T23:05:10.012823+0000","last_became_active":"2024-02-21T23:04:52.975958+0000","last_became_peered":"2024-02-21T23:04:52.975958+0000","last_unstale":"2024-02-21T23:05:10.012823+0000","last_undegraded":"2024-02-21T23:05:10.012823+0000","last_fullsized":"2024-02-21T23:05:10.012823+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:06:07.057936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.445687+0000","last_change":"2024-02-21T23:05:08.463501+0000","last_active":"2024-02-21T23:05:09.445687+0000","last_peered":"2024-02-21T23:05:09.445687+0000","last_clean":"2024-02-21T23:05:09.445687+0000","last_became_active":"2024-02-21T23:05:08.463203+0000","last_became_peered":"2024-02-21T23:05:08.463203+0000","last_unstale":"2024-02-21T23:05:09.445687+0000","last_undegraded":"2024-02-21T23:05:09.445687+0000","last_fullsized":"2024-02-21T23:05:09.445687+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:21:49.905786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.113972+0000","last_change":"2024-02-21T23:05:07.439970+0000","last_active":"2024-02-21T23:05:10.113972+0000","last_peered":"2024-02-21T23:05:10.113972+0000","last_clean":"2024-02-21T23:05:10.113972+0000","last_became_active":"2024-02-21T23:05:07.439647+0000","last_became_peered":"2024-02-21T23:05:07.439647+0000","last_unstale":"2024-02-21T23:05:10.113972+0000","last_undegraded":"2024-02-21T23:05:10.113972+0000","last_fullsized":"2024-02-21T23:05:10.113972+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_clean_scrub_stamp":"2024-02-21T23:05:06.380814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:36:48.877531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.464986+0000","last_change":"2024-02-21T23:04:53.980747+0000","last_active":"2024-02-21T23:05:09.464986+0000","last_peered":"2024-02-21T23:05:09.464986+0000","last_clean":"2024-02-21T23:05:09.464986+0000","last_became_active":"2024-02-21T23:04:53.980456+0000","last_became_peered":"2024-02-21T23:04:53.980456+0000","last_unstale":"2024-02-21T23:05:09.464986+0000","last_undegraded":"2024-02-21T23:05:09.464986+0000","last_fullsized":"2024-02-21T23:05:09.464986+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:40:20.534202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":22,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.448970+0000","last_change":"2024-02-21T23:04:52.974873+0000","last_active":"2024-02-21T23:05:09.448970+0000","last_peered":"2024-02-21T23:05:09.448970+0000","last_clean":"2024-02-21T23:05:09.448970+0000","last_became_active":"2024-02-21T23:04:52.973750+0000","last_became_peered":"2024-02-21T23:04:52.973750+0000","last_unstale":"2024-02-21T23:05:09.448970+0000","last_undegraded":"2024-02-21T23:05:09.448970+0000","last_fullsized":"2024-02-21T23:05:09.448970+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:04:02.374574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.469108+0000","last_change":"2024-02-21T23:05:08.444476+0000","last_active":"2024-02-21T23:05:09.469108+0000","last_peered":"2024-02-21T23:05:09.469108+0000","last_clean":"2024-02-21T23:05:09.469108+0000","last_became_active":"2024-02-21T23:05:08.444162+0000","last_became_peered":"2024-02-21T23:05:08.444162+0000","last_unstale":"2024-02-21T23:05:09.469108+0000","last_undegraded":"2024-02-21T23:05:09.469108+0000","last_fullsized":"2024-02-21T23:05:09.469108+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:26:14.091452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"56'1","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.444726+0000","last_change":"2024-02-21T23:05:07.428466+0000","last_active":"2024-02-21T23:05:09.444726+0000","last_peered":"2024-02-21T23:05:09.444726+0000","last_clean":"2024-02-21T23:05:09.444726+0000","last_became_active":"2024-02-21T23:05:07.428147+0000","last_became_peered":"2024-02-21T23:05:07.428147+0000","last_unstale":"2024-02-21T23:05:09.444726+0000","last_undegraded":"2024-02-21T23:05:09.444726+0000","last_fullsized":"2024-02-21T23:05:09.444726+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_clean_scrub_stamp":"2024-02-21T23:05:06.380814+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:26:52.157806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.053580+0000","last_change":"2024-02-21T23:04:53.983830+0000","last_active":"2024-02-21T23:05:10.053580+0000","last_peered":"2024-02-21T23:05:10.053580+0000","last_clean":"2024-02-21T23:05:10.053580+0000","last_became_active":"2024-02-21T23:04:53.974390+0000","last_became_peered":"2024-02-21T23:04:53.974390+0000","last_unstale":"2024-02-21T23:05:10.053580+0000","last_undegraded":"2024-02-21T23:05:10.053580+0000","last_fullsized":"2024-02-21T23:05:10.053580+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:27:17.374378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"52'3","reported_seq":25,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.012002+0000","last_change":"2024-02-21T23:04:52.976896+0000","last_active":"2024-02-21T23:05:10.012002+0000","last_peered":"2024-02-21T23:05:10.012002+0000","last_clean":"2024-02-21T23:05:10.012002+0000","last_became_active":"2024-02-21T23:04:52.976592+0000","last_became_peered":"2024-02-21T23:04:52.976592+0000","last_unstale":"2024-02-21T23:05:10.012002+0000","last_undegraded":"2024-02-21T23:05:10.012002+0000","last_fullsized":"2024-02-21T23:05:10.012002+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:39:31.156140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.441915+0000","last_change":"2024-02-21T23:05:08.457742+0000","last_active":"2024-02-21T23:05:09.441915+0000","last_peered":"2024-02-21T23:05:09.441915+0000","last_clean":"2024-02-21T23:05:09.441915+0000","last_became_active":"2024-02-21T23:05:08.457403+0000","last_became_peered":"2024-02-21T23:05:08.457403+0000","last_unstale":"2024-02-21T23:05:09.441915+0000","last_undegraded":"2024-02-21T23:05:09.441915+0000","last_fullsized":"2024-02-21T23:05:09.441915+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:19:27.871458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"56'1","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.444273+0000","last_change":"2024-02-21T23:05:07.433349+0000","last_active":"2024-02-21T23:05:09.444273+0000","last_peered":"2024-02-21T23:05:09.444273+0000","last_clean":"2024-02-21T23:05:09.444273+0000","last_became_active":"2024-02-21T23:05:07.431938+0000","last_became_peered":"2024-02-21T23:05:07.431938+0000","last_unstale":"2024-02-21T23:05:09.444273+0000","last_undegraded":"2024-02-21T23:05:09.444273+0000","last_fullsized":"2024-02-21T23:05:09.444273+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_clean_scrub_stamp":"2024-02-21T23:05:06.380814+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:38:13.725993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.446904+0000","last_change":"2024-02-21T23:04:53.997903+0000","last_active":"2024-02-21T23:05:09.446904+0000","last_peered":"2024-02-21T23:05:09.446904+0000","last_clean":"2024-02-21T23:05:09.446904+0000","last_became_active":"2024-02-21T23:04:53.997538+0000","last_became_peered":"2024-02-21T23:04:53.997538+0000","last_unstale":"2024-02-21T23:05:09.446904+0000","last_undegraded":"2024-02-21T23:05:09.446904+0000","last_fullsized":"2024-02-21T23:05:09.446904+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:55:24.090577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"52'6","reported_seq":28,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.445445+0000","last_change":"2024-02-21T23:04:52.975159+0000","last_active":"2024-02-21T23:05:09.445445+0000","last_peered":"2024-02-21T23:05:09.445445+0000","last_clean":"2024-02-21T23:05:09.445445+0000","last_became_active":"2024-02-21T23:04:52.974845+0000","last_became_peered":"2024-02-21T23:04:52.974845+0000","last_unstale":"2024-02-21T23:05:09.445445+0000","last_undegraded":"2024-02-21T23:05:09.445445+0000","last_fullsized":"2024-02-21T23:05:09.445445+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:20:05.709148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.054750+0000","last_change":"2024-02-21T23:05:08.445427+0000","last_active":"2024-02-21T23:05:10.054750+0000","last_peered":"2024-02-21T23:05:10.054750+0000","last_clean":"2024-02-21T23:05:10.054750+0000","last_became_active":"2024-02-21T23:05:08.445137+0000","last_became_peered":"2024-02-21T23:05:08.445137+0000","last_unstale":"2024-02-21T23:05:10.054750+0000","last_undegraded":"2024-02-21T23:05:10.054750+0000","last_fullsized":"2024-02-21T23:05:10.054750+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:41:33.700197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"56'2","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.053236+0000","last_change":"2024-02-21T23:05:07.435343+0000","last_active":"2024-02-21T23:05:10.053236+0000","last_peered":"2024-02-21T23:05:10.053236+0000","last_clean":"2024-02-21T23:05:10.053236+0000","last_became_active":"2024-02-21T23:05:07.434893+0000","last_became_peered":"2024-02-21T23:05:07.434893+0000","last_unstale":"2024-02-21T23:05:10.053236+0000","last_undegraded":"2024-02-21T23:05:10.053236+0000","last_fullsized":"2024-02-21T23:05:10.053236+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_clean_scrub_stamp":"2024-02-21T23:05:06.380814+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:44:32.145464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.465320+0000","last_change":"2024-02-21T23:04:53.987920+0000","last_active":"2024-02-21T23:05:09.465320+0000","last_peered":"2024-02-21T23:05:09.465320+0000","last_clean":"2024-02-21T23:05:09.465320+0000","last_became_active":"2024-02-21T23:04:53.986231+0000","last_became_peered":"2024-02-21T23:04:53.986231+0000","last_unstale":"2024-02-21T23:05:09.465320+0000","last_undegraded":"2024-02-21T23:05:09.465320+0000","last_fullsized":"2024-02-21T23:05:09.465320+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:20:06.730366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"52'2","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.053912+0000","last_change":"2024-02-21T23:04:52.977012+0000","last_active":"2024-02-21T23:05:10.053912+0000","last_peered":"2024-02-21T23:05:10.053912+0000","last_clean":"2024-02-21T23:05:10.053912+0000","last_became_active":"2024-02-21T23:04:52.973324+0000","last_became_peered":"2024-02-21T23:04:52.973324+0000","last_unstale":"2024-02-21T23:05:10.053912+0000","last_undegraded":"2024-02-21T23:05:10.053912+0000","last_fullsized":"2024-02-21T23:05:10.053912+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:55:27.414683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.012269+0000","last_change":"2024-02-21T23:05:08.468833+0000","last_active":"2024-02-21T23:05:10.012269+0000","last_peered":"2024-02-21T23:05:10.012269+0000","last_clean":"2024-02-21T23:05:10.012269+0000","last_became_active":"2024-02-21T23:05:08.468556+0000","last_became_peered":"2024-02-21T23:05:08.468556+0000","last_unstale":"2024-02-21T23:05:10.012269+0000","last_undegraded":"2024-02-21T23:05:10.012269+0000","last_fullsized":"2024-02-21T23:05:10.012269+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:33:22.889577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"56'2","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.053446+0000","last_change":"2024-02-21T23:05:07.445016+0000","last_active":"2024-02-21T23:05:10.053446+0000","last_peered":"2024-02-21T23:05:10.053446+0000","last_clean":"2024-02-21T23:05:10.053446+0000","last_became_active":"2024-02-21T23:05:07.441365+0000","last_became_peered":"2024-02-21T23:05:07.441365+0000","last_unstale":"2024-02-21T23:05:10.053446+0000","last_undegraded":"2024-02-21T23:05:10.053446+0000","last_fullsized":"2024-02-21T23:05:10.053446+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_clean_scrub_stamp":"2024-02-21T23:05:06.380814+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:52:45.522029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.050492+0000","last_change":"2024-02-21T23:04:53.969380+0000","last_active":"2024-02-21T23:05:10.050492+0000","last_peered":"2024-02-21T23:05:10.050492+0000","last_clean":"2024-02-21T23:05:10.050492+0000","last_became_active":"2024-02-21T23:04:53.966766+0000","last_became_peered":"2024-02-21T23:04:53.966766+0000","last_unstale":"2024-02-21T23:05:10.050492+0000","last_undegraded":"2024-02-21T23:05:10.050492+0000","last_fullsized":"2024-02-21T23:05:10.050492+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:22:45.699161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":22,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.115026+0000","last_change":"2024-02-21T23:04:52.981590+0000","last_active":"2024-02-21T23:05:10.115026+0000","last_peered":"2024-02-21T23:05:10.115026+0000","last_clean":"2024-02-21T23:05:10.115026+0000","last_became_active":"2024-02-21T23:04:52.981228+0000","last_became_peered":"2024-02-21T23:04:52.981228+0000","last_unstale":"2024-02-21T23:05:10.115026+0000","last_undegraded":"2024-02-21T23:05:10.115026+0000","last_fullsized":"2024-02-21T23:05:10.115026+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:25:58.605620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.115044+0000","last_change":"2024-02-21T23:05:08.451167+0000","last_active":"2024-02-21T23:05:10.115044+0000","last_peered":"2024-02-21T23:05:10.115044+0000","last_clean":"2024-02-21T23:05:10.115044+0000","last_became_active":"2024-02-21T23:05:08.450830+0000","last_became_peered":"2024-02-21T23:05:08.450830+0000","last_unstale":"2024-02-21T23:05:10.115044+0000","last_undegraded":"2024-02-21T23:05:10.115044+0000","last_fullsized":"2024-02-21T23:05:10.115044+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:47:50.018715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"56'4","reported_seq":18,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.443181+0000","last_change":"2024-02-21T23:05:07.442657+0000","last_active":"2024-02-21T23:05:09.443181+0000","last_peered":"2024-02-21T23:05:09.443181+0000","last_clean":"2024-02-21T23:05:09.443181+0000","last_became_active":"2024-02-21T23:05:07.442095+0000","last_became_peered":"2024-02-21T23:05:07.442095+0000","last_unstale":"2024-02-21T23:05:09.443181+0000","last_undegraded":"2024-02-21T23:05:09.443181+0000","last_fullsized":"2024-02-21T23:05:09.443181+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_clean_scrub_stamp":"2024-02-21T23:05:06.380814+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:20:20.181815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.445225+0000","last_change":"2024-02-21T23:04:53.983659+0000","last_active":"2024-02-21T23:05:09.445225+0000","last_peered":"2024-02-21T23:05:09.445225+0000","last_clean":"2024-02-21T23:05:09.445225+0000","last_became_active":"2024-02-21T23:04:53.983290+0000","last_became_peered":"2024-02-21T23:04:53.983290+0000","last_unstale":"2024-02-21T23:05:09.445225+0000","last_undegraded":"2024-02-21T23:05:09.445225+0000","last_fullsized":"2024-02-21T23:05:09.445225+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:05:20.754391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":22,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.010892+0000","last_change":"2024-02-21T23:04:52.982825+0000","last_active":"2024-02-21T23:05:10.010892+0000","last_peered":"2024-02-21T23:05:10.010892+0000","last_clean":"2024-02-21T23:05:10.010892+0000","last_became_active":"2024-02-21T23:04:52.982523+0000","last_became_peered":"2024-02-21T23:04:52.982523+0000","last_unstale":"2024-02-21T23:05:10.010892+0000","last_undegraded":"2024-02-21T23:05:10.010892+0000","last_fullsized":"2024-02-21T23:05:10.010892+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:45:49.617039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.448614+0000","last_change":"2024-02-21T23:05:08.455052+0000","last_active":"2024-02-21T23:05:09.448614+0000","last_peered":"2024-02-21T23:05:09.448614+0000","last_clean":"2024-02-21T23:05:09.448614+0000","last_became_active":"2024-02-21T23:05:08.454407+0000","last_became_peered":"2024-02-21T23:05:08.454407+0000","last_unstale":"2024-02-21T23:05:09.448614+0000","last_undegraded":"2024-02-21T23:05:09.448614+0000","last_fullsized":"2024-02-21T23:05:09.448614+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:57:18.857503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"56'1","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.054937+0000","last_change":"2024-02-21T23:05:07.436290+0000","last_active":"2024-02-21T23:05:10.054937+0000","last_peered":"2024-02-21T23:05:10.054937+0000","last_clean":"2024-02-21T23:05:10.054937+0000","last_became_active":"2024-02-21T23:05:07.433578+0000","last_became_peered":"2024-02-21T23:05:07.433578+0000","last_unstale":"2024-02-21T23:05:10.054937+0000","last_undegraded":"2024-02-21T23:05:10.054937+0000","last_fullsized":"2024-02-21T23:05:10.054937+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_clean_scrub_stamp":"2024-02-21T23:05:06.380814+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:17:32.435292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.051104+0000","last_change":"2024-02-21T23:04:53.988578+0000","last_active":"2024-02-21T23:05:10.051104+0000","last_peered":"2024-02-21T23:05:10.051104+0000","last_clean":"2024-02-21T23:05:10.051104+0000","last_became_active":"2024-02-21T23:04:53.988310+0000","last_became_peered":"2024-02-21T23:04:53.988310+0000","last_unstale":"2024-02-21T23:05:10.051104+0000","last_undegraded":"2024-02-21T23:05:10.051104+0000","last_fullsized":"2024-02-21T23:05:10.051104+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:17:58.257074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"52'8","reported_seq":30,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.443890+0000","last_change":"2024-02-21T23:04:52.978259+0000","last_active":"2024-02-21T23:05:09.443890+0000","last_peered":"2024-02-21T23:05:09.443890+0000","last_clean":"2024-02-21T23:05:09.443890+0000","last_became_active":"2024-02-21T23:04:52.975275+0000","last_became_peered":"2024-02-21T23:04:52.975275+0000","last_unstale":"2024-02-21T23:05:09.443890+0000","last_undegraded":"2024-02-21T23:05:09.443890+0000","last_fullsized":"2024-02-21T23:05:09.443890+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:20:55.242700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.442078+0000","last_change":"2024-02-21T23:05:08.459936+0000","last_active":"2024-02-21T23:05:09.442078+0000","last_peered":"2024-02-21T23:05:09.442078+0000","last_clean":"2024-02-21T23:05:09.442078+0000","last_became_active":"2024-02-21T23:05:08.459609+0000","last_became_peered":"2024-02-21T23:05:08.459609+0000","last_unstale":"2024-02-21T23:05:09.442078+0000","last_undegraded":"2024-02-21T23:05:09.442078+0000","last_fullsized":"2024-02-21T23:05:09.442078+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:28:50.270158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.444167+0000","last_change":"2024-02-21T23:05:07.444533+0000","last_active":"2024-02-21T23:05:09.444167+0000","last_peered":"2024-02-21T23:05:09.444167+0000","last_clean":"2024-02-21T23:05:09.444167+0000","last_became_active":"2024-02-21T23:05:07.444202+0000","last_became_peered":"2024-02-21T23:05:07.444202+0000","last_unstale":"2024-02-21T23:05:09.444167+0000","last_undegraded":"2024-02-21T23:05:09.444167+0000","last_fullsized":"2024-02-21T23:05:09.444167+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_clean_scrub_stamp":"2024-02-21T23:05:06.380814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:54:38.014722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.443822+0000","last_change":"2024-02-21T23:04:53.988231+0000","last_active":"2024-02-21T23:05:09.443822+0000","last_peered":"2024-02-21T23:05:09.443822+0000","last_clean":"2024-02-21T23:05:09.443822+0000","last_became_active":"2024-02-21T23:04:53.987814+0000","last_became_peered":"2024-02-21T23:04:53.987814+0000","last_unstale":"2024-02-21T23:05:09.443822+0000","last_undegraded":"2024-02-21T23:05:09.443822+0000","last_fullsized":"2024-02-21T23:05:09.443822+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:37:47.491317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"52'1","reported_seq":23,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.054030+0000","last_change":"2024-02-21T23:04:52.981262+0000","last_active":"2024-02-21T23:05:10.054030+0000","last_peered":"2024-02-21T23:05:10.054030+0000","last_clean":"2024-02-21T23:05:10.054030+0000","last_became_active":"2024-02-21T23:04:52.980981+0000","last_became_peered":"2024-02-21T23:04:52.980981+0000","last_unstale":"2024-02-21T23:05:10.054030+0000","last_undegraded":"2024-02-21T23:05:10.054030+0000","last_fullsized":"2024-02-21T23:05:10.054030+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:45:49.617039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.469741+0000","last_change":"2024-02-21T23:05:08.454574+0000","last_active":"2024-02-21T23:05:09.469741+0000","last_peered":"2024-02-21T23:05:09.469741+0000","last_clean":"2024-02-21T23:05:09.469741+0000","last_became_active":"2024-02-21T23:05:08.454208+0000","last_became_peered":"2024-02-21T23:05:08.454208+0000","last_unstale":"2024-02-21T23:05:09.469741+0000","last_undegraded":"2024-02-21T23:05:09.469741+0000","last_fullsized":"2024-02-21T23:05:09.469741+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:03:08.356487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.441747+0000","last_change":"2024-02-21T23:05:07.441184+0000","last_active":"2024-02-21T23:05:09.441747+0000","last_peered":"2024-02-21T23:05:09.441747+0000","last_clean":"2024-02-21T23:05:09.441747+0000","last_became_active":"2024-02-21T23:05:07.440719+0000","last_became_peered":"2024-02-21T23:05:07.440719+0000","last_unstale":"2024-02-21T23:05:09.441747+0000","last_undegraded":"2024-02-21T23:05:09.441747+0000","last_fullsized":"2024-02-21T23:05:09.441747+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_clean_scrub_stamp":"2024-02-21T23:05:06.380814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:52:56.953993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.441843+0000","last_change":"2024-02-21T23:04:53.967738+0000","last_active":"2024-02-21T23:05:09.441843+0000","last_peered":"2024-02-21T23:05:09.441843+0000","last_clean":"2024-02-21T23:05:09.441843+0000","last_became_active":"2024-02-21T23:04:53.967430+0000","last_became_peered":"2024-02-21T23:04:53.967430+0000","last_unstale":"2024-02-21T23:05:09.441843+0000","last_undegraded":"2024-02-21T23:05:09.441843+0000","last_fullsized":"2024-02-21T23:05:09.441843+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:59:35.227750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"52'2","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.470666+0000","last_change":"2024-02-21T23:04:52.966722+0000","last_active":"2024-02-21T23:05:09.470666+0000","last_peered":"2024-02-21T23:05:09.470666+0000","last_clean":"2024-02-21T23:05:09.470666+0000","last_became_active":"2024-02-21T23:04:52.966322+0000","last_became_peered":"2024-02-21T23:04:52.966322+0000","last_unstale":"2024-02-21T23:05:09.470666+0000","last_undegraded":"2024-02-21T23:05:09.470666+0000","last_fullsized":"2024-02-21T23:05:09.470666+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:29:42.084956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.113890+0000","last_change":"2024-02-21T23:05:08.454805+0000","last_active":"2024-02-21T23:05:10.113890+0000","last_peered":"2024-02-21T23:05:10.113890+0000","last_clean":"2024-02-21T23:05:10.113890+0000","last_became_active":"2024-02-21T23:05:08.454156+0000","last_became_peered":"2024-02-21T23:05:08.454156+0000","last_unstale":"2024-02-21T23:05:10.113890+0000","last_undegraded":"2024-02-21T23:05:10.113890+0000","last_fullsized":"2024-02-21T23:05:10.113890+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:23:00.163775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.011501+0000","last_change":"2024-02-21T23:05:07.438658+0000","last_active":"2024-02-21T23:05:10.011501+0000","last_peered":"2024-02-21T23:05:10.011501+0000","last_clean":"2024-02-21T23:05:10.011501+0000","last_became_active":"2024-02-21T23:05:07.438362+0000","last_became_peered":"2024-02-21T23:05:07.438362+0000","last_unstale":"2024-02-21T23:05:10.011501+0000","last_undegraded":"2024-02-21T23:05:10.011501+0000","last_fullsized":"2024-02-21T23:05:10.011501+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_clean_scrub_stamp":"2024-02-21T23:05:06.380814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:19:26.858293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.445906+0000","last_change":"2024-02-21T23:04:53.989868+0000","last_active":"2024-02-21T23:05:09.445906+0000","last_peered":"2024-02-21T23:05:09.445906+0000","last_clean":"2024-02-21T23:05:09.445906+0000","last_became_active":"2024-02-21T23:04:53.989585+0000","last_became_peered":"2024-02-21T23:04:53.989585+0000","last_unstale":"2024-02-21T23:05:09.445906+0000","last_undegraded":"2024-02-21T23:05:09.445906+0000","last_fullsized":"2024-02-21T23:05:09.445906+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:06:08.079155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"52'1","reported_seq":23,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.115775+0000","last_change":"2024-02-21T23:04:52.976203+0000","last_active":"2024-02-21T23:05:10.115775+0000","last_peered":"2024-02-21T23:05:10.115775+0000","last_clean":"2024-02-21T23:05:10.115775+0000","last_became_active":"2024-02-21T23:04:52.975787+0000","last_became_peered":"2024-02-21T23:04:52.975787+0000","last_unstale":"2024-02-21T23:05:10.115775+0000","last_undegraded":"2024-02-21T23:05:10.115775+0000","last_fullsized":"2024-02-21T23:05:10.115775+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:02:52.870656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.445589+0000","last_change":"2024-02-21T23:05:08.479352+0000","last_active":"2024-02-21T23:05:09.445589+0000","last_peered":"2024-02-21T23:05:09.445589+0000","last_clean":"2024-02-21T23:05:09.445589+0000","last_became_active":"2024-02-21T23:05:08.479057+0000","last_became_peered":"2024-02-21T23:05:08.479057+0000","last_unstale":"2024-02-21T23:05:09.445589+0000","last_undegraded":"2024-02-21T23:05:09.445589+0000","last_fullsized":"2024-02-21T23:05:09.445589+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:12:33.843961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.444387+0000","last_change":"2024-02-21T23:05:07.438855+0000","last_active":"2024-02-21T23:05:09.444387+0000","last_peered":"2024-02-21T23:05:09.444387+0000","last_clean":"2024-02-21T23:05:09.444387+0000","last_became_active":"2024-02-21T23:05:07.438539+0000","last_became_peered":"2024-02-21T23:05:07.438539+0000","last_unstale":"2024-02-21T23:05:09.444387+0000","last_undegraded":"2024-02-21T23:05:09.444387+0000","last_fullsized":"2024-02-21T23:05:09.444387+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_clean_scrub_stamp":"2024-02-21T23:05:06.380814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:38:49.288904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.443113+0000","last_change":"2024-02-21T23:04:53.986757+0000","last_active":"2024-02-21T23:05:09.443113+0000","last_peered":"2024-02-21T23:05:09.443113+0000","last_clean":"2024-02-21T23:05:09.443113+0000","last_became_active":"2024-02-21T23:04:53.982468+0000","last_became_peered":"2024-02-21T23:04:53.982468+0000","last_unstale":"2024-02-21T23:05:09.443113+0000","last_undegraded":"2024-02-21T23:05:09.443113+0000","last_fullsized":"2024-02-21T23:05:09.443113+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:48:17.190667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":22,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.114996+0000","last_change":"2024-02-21T23:04:52.978566+0000","last_active":"2024-02-21T23:05:10.114996+0000","last_peered":"2024-02-21T23:05:10.114996+0000","last_clean":"2024-02-21T23:05:10.114996+0000","last_became_active":"2024-02-21T23:04:52.978262+0000","last_became_peered":"2024-02-21T23:04:52.978262+0000","last_unstale":"2024-02-21T23:05:10.114996+0000","last_undegraded":"2024-02-21T23:05:10.114996+0000","last_fullsized":"2024-02-21T23:05:10.114996+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:12:02.552178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.465501+0000","last_change":"2024-02-21T23:05:08.459858+0000","last_active":"2024-02-21T23:05:09.465501+0000","last_peered":"2024-02-21T23:05:09.465501+0000","last_clean":"2024-02-21T23:05:09.465501+0000","last_became_active":"2024-02-21T23:05:08.459574+0000","last_became_peered":"2024-02-21T23:05:08.459574+0000","last_unstale":"2024-02-21T23:05:09.465501+0000","last_undegraded":"2024-02-21T23:05:09.465501+0000","last_fullsized":"2024-02-21T23:05:09.465501+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:41:33.700197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.053114+0000","last_change":"2024-02-21T23:05:07.435108+0000","last_active":"2024-02-21T23:05:10.053114+0000","last_peered":"2024-02-21T23:05:10.053114+0000","last_clean":"2024-02-21T23:05:10.053114+0000","last_became_active":"2024-02-21T23:05:07.434442+0000","last_became_peered":"2024-02-21T23:05:07.434442+0000","last_unstale":"2024-02-21T23:05:10.053114+0000","last_undegraded":"2024-02-21T23:05:10.053114+0000","last_fullsized":"2024-02-21T23:05:10.053114+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_clean_scrub_stamp":"2024-02-21T23:05:06.380814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:39:35.673901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.465775+0000","last_change":"2024-02-21T23:04:53.982199+0000","last_active":"2024-02-21T23:05:09.465775+0000","last_peered":"2024-02-21T23:05:09.465775+0000","last_clean":"2024-02-21T23:05:09.465775+0000","last_became_active":"2024-02-21T23:04:53.981777+0000","last_became_peered":"2024-02-21T23:04:53.981777+0000","last_unstale":"2024-02-21T23:05:09.465775+0000","last_undegraded":"2024-02-21T23:05:09.465775+0000","last_fullsized":"2024-02-21T23:05:09.465775+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:52:43.502544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"52'2","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.443226+0000","last_change":"2024-02-21T23:04:52.961423+0000","last_active":"2024-02-21T23:05:09.443226+0000","last_peered":"2024-02-21T23:05:09.443226+0000","last_clean":"2024-02-21T23:05:09.443226+0000","last_became_active":"2024-02-21T23:04:52.961090+0000","last_became_peered":"2024-02-21T23:04:52.961090+0000","last_unstale":"2024-02-21T23:05:09.443226+0000","last_undegraded":"2024-02-21T23:05:09.443226+0000","last_fullsized":"2024-02-21T23:05:09.443226+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:25:58.605620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.115907+0000","last_change":"2024-02-21T23:05:08.455489+0000","last_active":"2024-02-21T23:05:10.115907+0000","last_peered":"2024-02-21T23:05:10.115907+0000","last_clean":"2024-02-21T23:05:10.115907+0000","last_became_active":"2024-02-21T23:05:08.455172+0000","last_became_peered":"2024-02-21T23:05:08.455172+0000","last_unstale":"2024-02-21T23:05:10.115907+0000","last_undegraded":"2024-02-21T23:05:10.115907+0000","last_fullsized":"2024-02-21T23:05:10.115907+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:21:26.217233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.442233+0000","last_change":"2024-02-21T23:05:07.441094+0000","last_active":"2024-02-21T23:05:09.442233+0000","last_peered":"2024-02-21T23:05:09.442233+0000","last_clean":"2024-02-21T23:05:09.442233+0000","last_became_active":"2024-02-21T23:05:07.440751+0000","last_became_peered":"2024-02-21T23:05:07.440751+0000","last_unstale":"2024-02-21T23:05:09.442233+0000","last_undegraded":"2024-02-21T23:05:09.442233+0000","last_fullsized":"2024-02-21T23:05:09.442233+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_clean_scrub_stamp":"2024-02-21T23:05:06.380814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:40:33.985651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.113414+0000","last_change":"2024-02-21T23:04:53.983031+0000","last_active":"2024-02-21T23:05:10.113414+0000","last_peered":"2024-02-21T23:05:10.113414+0000","last_clean":"2024-02-21T23:05:10.113414+0000","last_became_active":"2024-02-21T23:04:53.981770+0000","last_became_peered":"2024-02-21T23:04:53.981770+0000","last_unstale":"2024-02-21T23:05:10.113414+0000","last_undegraded":"2024-02-21T23:05:10.113414+0000","last_fullsized":"2024-02-21T23:05:10.113414+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:23:27.013837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":22,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.444552+0000","last_change":"2024-02-21T23:04:52.975453+0000","last_active":"2024-02-21T23:05:09.444552+0000","last_peered":"2024-02-21T23:05:09.444552+0000","last_clean":"2024-02-21T23:05:09.444552+0000","last_became_active":"2024-02-21T23:04:52.975121+0000","last_became_peered":"2024-02-21T23:04:52.975121+0000","last_unstale":"2024-02-21T23:05:09.444552+0000","last_undegraded":"2024-02-21T23:05:09.444552+0000","last_fullsized":"2024-02-21T23:05:09.444552+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:51:11.660655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.054303+0000","last_change":"2024-02-21T23:05:08.461579+0000","last_active":"2024-02-21T23:05:10.054303+0000","last_peered":"2024-02-21T23:05:10.054303+0000","last_clean":"2024-02-21T23:05:10.054303+0000","last_became_active":"2024-02-21T23:05:08.461173+0000","last_became_peered":"2024-02-21T23:05:08.461173+0000","last_unstale":"2024-02-21T23:05:10.054303+0000","last_undegraded":"2024-02-21T23:05:10.054303+0000","last_fullsized":"2024-02-21T23:05:10.054303+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:38:50.302069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"56'2","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.465885+0000","last_change":"2024-02-21T23:05:07.435315+0000","last_active":"2024-02-21T23:05:09.465885+0000","last_peered":"2024-02-21T23:05:09.465885+0000","last_clean":"2024-02-21T23:05:09.465885+0000","last_became_active":"2024-02-21T23:05:07.435015+0000","last_became_peered":"2024-02-21T23:05:07.435015+0000","last_unstale":"2024-02-21T23:05:09.465885+0000","last_undegraded":"2024-02-21T23:05:09.465885+0000","last_fullsized":"2024-02-21T23:05:09.465885+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_clean_scrub_stamp":"2024-02-21T23:05:06.380814+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:19:01.236574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.055136+0000","last_change":"2024-02-21T23:04:53.978362+0000","last_active":"2024-02-21T23:05:10.055136+0000","last_peered":"2024-02-21T23:05:10.055136+0000","last_clean":"2024-02-21T23:05:10.055136+0000","last_became_active":"2024-02-21T23:04:53.978070+0000","last_became_peered":"2024-02-21T23:04:53.978070+0000","last_unstale":"2024-02-21T23:05:10.055136+0000","last_undegraded":"2024-02-21T23:05:10.055136+0000","last_fullsized":"2024-02-21T23:05:10.055136+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:52:55.798286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":22,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.053064+0000","last_change":"2024-02-21T23:04:52.979855+0000","last_active":"2024-02-21T23:05:10.053064+0000","last_peered":"2024-02-21T23:05:10.053064+0000","last_clean":"2024-02-21T23:05:10.053064+0000","last_became_active":"2024-02-21T23:04:52.977225+0000","last_became_peered":"2024-02-21T23:04:52.977225+0000","last_unstale":"2024-02-21T23:05:10.053064+0000","last_undegraded":"2024-02-21T23:05:10.053064+0000","last_fullsized":"2024-02-21T23:05:10.053064+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:39:10.601854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.470182+0000","last_change":"2024-02-21T23:05:08.444548+0000","last_active":"2024-02-21T23:05:09.470182+0000","last_peered":"2024-02-21T23:05:09.470182+0000","last_clean":"2024-02-21T23:05:09.470182+0000","last_became_active":"2024-02-21T23:05:08.444185+0000","last_became_peered":"2024-02-21T23:05:08.444185+0000","last_unstale":"2024-02-21T23:05:09.470182+0000","last_undegraded":"2024-02-21T23:05:09.470182+0000","last_fullsized":"2024-02-21T23:05:09.470182+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:55:42.900515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"57'1","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.468181+0000","last_change":"2024-02-21T23:05:07.421231+0000","last_active":"2024-02-21T23:05:09.468181+0000","last_peered":"2024-02-21T23:05:09.468181+0000","last_clean":"2024-02-21T23:05:09.468181+0000","last_became_active":"2024-02-21T23:05:07.420844+0000","last_became_peered":"2024-02-21T23:05:07.420844+0000","last_unstale":"2024-02-21T23:05:09.468181+0000","last_undegraded":"2024-02-21T23:05:09.468181+0000","last_fullsized":"2024-02-21T23:05:09.468181+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_clean_scrub_stamp":"2024-02-21T23:05:06.380814+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:38:00.942766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.443238+0000","last_change":"2024-02-21T23:04:53.996234+0000","last_active":"2024-02-21T23:05:09.443238+0000","last_peered":"2024-02-21T23:05:09.443238+0000","last_clean":"2024-02-21T23:05:09.443238+0000","last_became_active":"2024-02-21T23:04:53.995859+0000","last_became_peered":"2024-02-21T23:04:53.995859+0000","last_unstale":"2024-02-21T23:05:09.443238+0000","last_undegraded":"2024-02-21T23:05:09.443238+0000","last_fullsized":"2024-02-21T23:05:09.443238+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:46:44.487212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":22,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.469351+0000","last_change":"2024-02-21T23:04:52.972729+0000","last_active":"2024-02-21T23:05:09.469351+0000","last_peered":"2024-02-21T23:05:09.469351+0000","last_clean":"2024-02-21T23:05:09.469351+0000","last_became_active":"2024-02-21T23:04:52.972434+0000","last_became_peered":"2024-02-21T23:04:52.972434+0000","last_unstale":"2024-02-21T23:05:09.469351+0000","last_undegraded":"2024-02-21T23:05:09.469351+0000","last_fullsized":"2024-02-21T23:05:09.469351+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:00:21.589361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.114854+0000","last_change":"2024-02-21T23:05:08.455465+0000","last_active":"2024-02-21T23:05:10.114854+0000","last_peered":"2024-02-21T23:05:10.114854+0000","last_clean":"2024-02-21T23:05:10.114854+0000","last_became_active":"2024-02-21T23:05:08.455119+0000","last_became_peered":"2024-02-21T23:05:08.455119+0000","last_unstale":"2024-02-21T23:05:10.114854+0000","last_undegraded":"2024-02-21T23:05:10.114854+0000","last_fullsized":"2024-02-21T23:05:10.114854+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:19:27.871458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.052433+0000","last_change":"2024-02-21T23:05:07.434362+0000","last_active":"2024-02-21T23:05:10.052433+0000","last_peered":"2024-02-21T23:05:10.052433+0000","last_clean":"2024-02-21T23:05:10.052433+0000","last_became_active":"2024-02-21T23:05:07.433274+0000","last_became_peered":"2024-02-21T23:05:07.433274+0000","last_unstale":"2024-02-21T23:05:10.052433+0000","last_undegraded":"2024-02-21T23:05:10.052433+0000","last_fullsized":"2024-02-21T23:05:10.052433+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_clean_scrub_stamp":"2024-02-21T23:05:06.380814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:34:15.596228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.466065+0000","last_change":"2024-02-21T23:04:53.996387+0000","last_active":"2024-02-21T23:05:09.466065+0000","last_peered":"2024-02-21T23:05:09.466065+0000","last_clean":"2024-02-21T23:05:09.466065+0000","last_became_active":"2024-02-21T23:04:53.996123+0000","last_became_peered":"2024-02-21T23:04:53.996123+0000","last_unstale":"2024-02-21T23:05:09.466065+0000","last_undegraded":"2024-02-21T23:05:09.466065+0000","last_fullsized":"2024-02-21T23:05:09.466065+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:43:19.009080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"52'2","reported_seq":24,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.115816+0000","last_change":"2024-02-21T23:04:52.981486+0000","last_active":"2024-02-21T23:05:10.115816+0000","last_peered":"2024-02-21T23:05:10.115816+0000","last_clean":"2024-02-21T23:05:10.115816+0000","last_became_active":"2024-02-21T23:04:52.981135+0000","last_became_peered":"2024-02-21T23:04:52.981135+0000","last_unstale":"2024-02-21T23:05:10.115816+0000","last_undegraded":"2024-02-21T23:05:10.115816+0000","last_fullsized":"2024-02-21T23:05:10.115816+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:17:14.876508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.052275+0000","last_change":"2024-02-21T23:05:08.447608+0000","last_active":"2024-02-21T23:05:10.052275+0000","last_peered":"2024-02-21T23:05:10.052275+0000","last_clean":"2024-02-21T23:05:10.052275+0000","last_became_active":"2024-02-21T23:05:08.445997+0000","last_became_peered":"2024-02-21T23:05:08.445997+0000","last_unstale":"2024-02-21T23:05:10.052275+0000","last_undegraded":"2024-02-21T23:05:10.052275+0000","last_fullsized":"2024-02-21T23:05:10.052275+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:03:07.620481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"56'2","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.053967+0000","last_change":"2024-02-21T23:05:07.443603+0000","last_active":"2024-02-21T23:05:10.053967+0000","last_peered":"2024-02-21T23:05:10.053967+0000","last_clean":"2024-02-21T23:05:10.053967+0000","last_became_active":"2024-02-21T23:05:07.443390+0000","last_became_peered":"2024-02-21T23:05:07.443390+0000","last_unstale":"2024-02-21T23:05:10.053967+0000","last_undegraded":"2024-02-21T23:05:10.053967+0000","last_fullsized":"2024-02-21T23:05:10.053967+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_clean_scrub_stamp":"2024-02-21T23:05:06.380814+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:34:59.258614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.442463+0000","last_change":"2024-02-21T23:04:53.977118+0000","last_active":"2024-02-21T23:05:09.442463+0000","last_peered":"2024-02-21T23:05:09.442463+0000","last_clean":"2024-02-21T23:05:09.442463+0000","last_became_active":"2024-02-21T23:04:53.976042+0000","last_became_peered":"2024-02-21T23:04:53.976042+0000","last_unstale":"2024-02-21T23:05:09.442463+0000","last_undegraded":"2024-02-21T23:05:09.442463+0000","last_fullsized":"2024-02-21T23:05:09.442463+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:10:31.924664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":22,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.012176+0000","last_change":"2024-02-21T23:04:52.974536+0000","last_active":"2024-02-21T23:05:10.012176+0000","last_peered":"2024-02-21T23:05:10.012176+0000","last_clean":"2024-02-21T23:05:10.012176+0000","last_became_active":"2024-02-21T23:04:52.974221+0000","last_became_peered":"2024-02-21T23:04:52.974221+0000","last_unstale":"2024-02-21T23:05:10.012176+0000","last_undegraded":"2024-02-21T23:05:10.012176+0000","last_fullsized":"2024-02-21T23:05:10.012176+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:39:10.601854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.464834+0000","last_change":"2024-02-21T23:05:08.460693+0000","last_active":"2024-02-21T23:05:09.464834+0000","last_peered":"2024-02-21T23:05:09.464834+0000","last_clean":"2024-02-21T23:05:09.464834+0000","last_became_active":"2024-02-21T23:05:08.460424+0000","last_became_peered":"2024-02-21T23:05:08.460424+0000","last_unstale":"2024-02-21T23:05:09.464834+0000","last_undegraded":"2024-02-21T23:05:09.464834+0000","last_fullsized":"2024-02-21T23:05:09.464834+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:00:36.493162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.465910+0000","last_change":"2024-02-21T23:05:07.443614+0000","last_active":"2024-02-21T23:05:09.465910+0000","last_peered":"2024-02-21T23:05:09.465910+0000","last_clean":"2024-02-21T23:05:09.465910+0000","last_became_active":"2024-02-21T23:05:07.443088+0000","last_became_peered":"2024-02-21T23:05:07.443088+0000","last_unstale":"2024-02-21T23:05:09.465910+0000","last_undegraded":"2024-02-21T23:05:09.465910+0000","last_fullsized":"2024-02-21T23:05:09.465910+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_clean_scrub_stamp":"2024-02-21T23:05:06.380814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:28:49.256993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.444962+0000","last_change":"2024-02-21T23:04:53.979161+0000","last_active":"2024-02-21T23:05:09.444962+0000","last_peered":"2024-02-21T23:05:09.444962+0000","last_clean":"2024-02-21T23:05:09.444962+0000","last_became_active":"2024-02-21T23:04:53.978843+0000","last_became_peered":"2024-02-21T23:04:53.978843+0000","last_unstale":"2024-02-21T23:05:09.444962+0000","last_undegraded":"2024-02-21T23:05:09.444962+0000","last_fullsized":"2024-02-21T23:05:09.444962+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:34:45.807165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":22,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.053950+0000","last_change":"2024-02-21T23:04:52.969804+0000","last_active":"2024-02-21T23:05:10.053950+0000","last_peered":"2024-02-21T23:05:10.053950+0000","last_clean":"2024-02-21T23:05:10.053950+0000","last_became_active":"2024-02-21T23:04:52.969353+0000","last_became_peered":"2024-02-21T23:04:52.969353+0000","last_unstale":"2024-02-21T23:05:10.053950+0000","last_undegraded":"2024-02-21T23:05:10.053950+0000","last_fullsized":"2024-02-21T23:05:10.053950+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:47:27.364576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.448509+0000","last_change":"2024-02-21T23:05:08.466401+0000","last_active":"2024-02-21T23:05:09.448509+0000","last_peered":"2024-02-21T23:05:09.448509+0000","last_clean":"2024-02-21T23:05:09.448509+0000","last_became_active":"2024-02-21T23:05:08.466067+0000","last_became_peered":"2024-02-21T23:05:08.466067+0000","last_unstale":"2024-02-21T23:05:09.448509+0000","last_undegraded":"2024-02-21T23:05:09.448509+0000","last_fullsized":"2024-02-21T23:05:09.448509+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:53:10.262900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"56'1","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.011616+0000","last_change":"2024-02-21T23:05:07.438319+0000","last_active":"2024-02-21T23:05:10.011616+0000","last_peered":"2024-02-21T23:05:10.011616+0000","last_clean":"2024-02-21T23:05:10.011616+0000","last_became_active":"2024-02-21T23:05:07.437958+0000","last_became_peered":"2024-02-21T23:05:07.437958+0000","last_unstale":"2024-02-21T23:05:10.011616+0000","last_undegraded":"2024-02-21T23:05:10.011616+0000","last_fullsized":"2024-02-21T23:05:10.011616+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_clean_scrub_stamp":"2024-02-21T23:05:06.380814+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:57:17.844338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.050414+0000","last_change":"2024-02-21T23:04:54.000144+0000","last_active":"2024-02-21T23:05:10.050414+0000","last_peered":"2024-02-21T23:05:10.050414+0000","last_clean":"2024-02-21T23:05:10.050414+0000","last_became_active":"2024-02-21T23:04:53.999687+0000","last_became_peered":"2024-02-21T23:04:53.999687+0000","last_unstale":"2024-02-21T23:05:10.050414+0000","last_undegraded":"2024-02-21T23:05:10.050414+0000","last_fullsized":"2024-02-21T23:05:10.050414+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:12:19.379347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":22,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.445245+0000","last_change":"2024-02-21T23:04:52.975668+0000","last_active":"2024-02-21T23:05:09.445245+0000","last_peered":"2024-02-21T23:05:09.445245+0000","last_clean":"2024-02-21T23:05:09.445245+0000","last_became_active":"2024-02-21T23:04:52.975348+0000","last_became_peered":"2024-02-21T23:04:52.975348+0000","last_unstale":"2024-02-21T23:05:09.445245+0000","last_undegraded":"2024-02-21T23:05:09.445245+0000","last_fullsized":"2024-02-21T23:05:09.445245+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:12:18.358129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.115717+0000","last_change":"2024-02-21T23:05:08.460822+0000","last_active":"2024-02-21T23:05:10.115717+0000","last_peered":"2024-02-21T23:05:10.115717+0000","last_clean":"2024-02-21T23:05:10.115717+0000","last_became_active":"2024-02-21T23:05:08.460519+0000","last_became_peered":"2024-02-21T23:05:08.460519+0000","last_unstale":"2024-02-21T23:05:10.115717+0000","last_undegraded":"2024-02-21T23:05:10.115717+0000","last_fullsized":"2024-02-21T23:05:10.115717+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:59:56.223705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.445004+0000","last_change":"2024-02-21T23:05:07.439510+0000","last_active":"2024-02-21T23:05:09.445004+0000","last_peered":"2024-02-21T23:05:09.445004+0000","last_clean":"2024-02-21T23:05:09.445004+0000","last_became_active":"2024-02-21T23:05:07.439198+0000","last_became_peered":"2024-02-21T23:05:07.439198+0000","last_unstale":"2024-02-21T23:05:09.445004+0000","last_undegraded":"2024-02-21T23:05:09.445004+0000","last_fullsized":"2024-02-21T23:05:09.445004+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_clean_scrub_stamp":"2024-02-21T23:05:06.380814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:48:17.959315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.465728+0000","last_change":"2024-02-21T23:04:53.988119+0000","last_active":"2024-02-21T23:05:09.465728+0000","last_peered":"2024-02-21T23:05:09.465728+0000","last_clean":"2024-02-21T23:05:09.465728+0000","last_became_active":"2024-02-21T23:04:53.986237+0000","last_became_peered":"2024-02-21T23:04:53.986237+0000","last_unstale":"2024-02-21T23:05:09.465728+0000","last_undegraded":"2024-02-21T23:05:09.465728+0000","last_fullsized":"2024-02-21T23:05:09.465728+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:10:20.341856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":22,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.444670+0000","last_change":"2024-02-21T23:04:52.976653+0000","last_active":"2024-02-21T23:05:09.444670+0000","last_peered":"2024-02-21T23:05:09.444670+0000","last_clean":"2024-02-21T23:05:09.444670+0000","last_became_active":"2024-02-21T23:04:52.976249+0000","last_became_peered":"2024-02-21T23:04:52.976249+0000","last_unstale":"2024-02-21T23:05:09.444670+0000","last_undegraded":"2024-02-21T23:05:09.444670+0000","last_fullsized":"2024-02-21T23:05:09.444670+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:40:19.512984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.011757+0000","last_change":"2024-02-21T23:05:08.474817+0000","last_active":"2024-02-21T23:05:10.011757+0000","last_peered":"2024-02-21T23:05:10.011757+0000","last_clean":"2024-02-21T23:05:10.011757+0000","last_became_active":"2024-02-21T23:05:08.474309+0000","last_became_peered":"2024-02-21T23:05:08.474309+0000","last_unstale":"2024-02-21T23:05:10.011757+0000","last_undegraded":"2024-02-21T23:05:10.011757+0000","last_fullsized":"2024-02-21T23:05:10.011757+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:25:56.439594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"56'1","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.466178+0000","last_change":"2024-02-21T23:05:07.440280+0000","last_active":"2024-02-21T23:05:09.466178+0000","last_peered":"2024-02-21T23:05:09.466178+0000","last_clean":"2024-02-21T23:05:09.466178+0000","last_became_active":"2024-02-21T23:05:07.440008+0000","last_became_peered":"2024-02-21T23:05:07.440008+0000","last_unstale":"2024-02-21T23:05:09.466178+0000","last_undegraded":"2024-02-21T23:05:09.466178+0000","last_fullsized":"2024-02-21T23:05:09.466178+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_clean_scrub_stamp":"2024-02-21T23:05:06.380814+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:59:55.210541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.051585+0000","last_change":"2024-02-21T23:04:53.993076+0000","last_active":"2024-02-21T23:05:10.051585+0000","last_peered":"2024-02-21T23:05:10.051585+0000","last_clean":"2024-02-21T23:05:10.051585+0000","last_became_active":"2024-02-21T23:04:53.992734+0000","last_became_peered":"2024-02-21T23:04:53.992734+0000","last_unstale":"2024-02-21T23:05:10.051585+0000","last_undegraded":"2024-02-21T23:05:10.051585+0000","last_fullsized":"2024-02-21T23:05:10.051585+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:52:43.502544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":22,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.446077+0000","last_change":"2024-02-21T23:04:52.971065+0000","last_active":"2024-02-21T23:05:09.446077+0000","last_peered":"2024-02-21T23:05:09.446077+0000","last_clean":"2024-02-21T23:05:09.446077+0000","last_became_active":"2024-02-21T23:04:52.970410+0000","last_became_peered":"2024-02-21T23:04:52.970410+0000","last_unstale":"2024-02-21T23:05:09.446077+0000","last_undegraded":"2024-02-21T23:05:09.446077+0000","last_fullsized":"2024-02-21T23:05:09.446077+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:18:03.518976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.115438+0000","last_change":"2024-02-21T23:05:08.471258+0000","last_active":"2024-02-21T23:05:10.115438+0000","last_peered":"2024-02-21T23:05:10.115438+0000","last_clean":"2024-02-21T23:05:10.115438+0000","last_became_active":"2024-02-21T23:05:08.470955+0000","last_became_peered":"2024-02-21T23:05:08.470955+0000","last_unstale":"2024-02-21T23:05:10.115438+0000","last_undegraded":"2024-02-21T23:05:10.115438+0000","last_fullsized":"2024-02-21T23:05:10.115438+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:28:50.270158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"56'2","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.012215+0000","last_change":"2024-02-21T23:05:07.441965+0000","last_active":"2024-02-21T23:05:10.012215+0000","last_peered":"2024-02-21T23:05:10.012215+0000","last_clean":"2024-02-21T23:05:10.012215+0000","last_became_active":"2024-02-21T23:05:07.441586+0000","last_became_peered":"2024-02-21T23:05:07.441586+0000","last_unstale":"2024-02-21T23:05:10.012215+0000","last_undegraded":"2024-02-21T23:05:10.012215+0000","last_fullsized":"2024-02-21T23:05:10.012215+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_clean_scrub_stamp":"2024-02-21T23:05:06.380814+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:38:00.942766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.445429+0000","last_change":"2024-02-21T23:04:53.995393+0000","last_active":"2024-02-21T23:05:09.445429+0000","last_peered":"2024-02-21T23:05:09.445429+0000","last_clean":"2024-02-21T23:05:09.445429+0000","last_became_active":"2024-02-21T23:04:53.995131+0000","last_became_peered":"2024-02-21T23:04:53.995131+0000","last_unstale":"2024-02-21T23:05:09.445429+0000","last_undegraded":"2024-02-21T23:05:09.445429+0000","last_fullsized":"2024-02-21T23:05:09.445429+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:34:02.144778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":22,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.465868+0000","last_change":"2024-02-21T23:04:52.968036+0000","last_active":"2024-02-21T23:05:09.465868+0000","last_peered":"2024-02-21T23:05:09.465868+0000","last_clean":"2024-02-21T23:05:09.465868+0000","last_became_active":"2024-02-21T23:04:52.967626+0000","last_became_peered":"2024-02-21T23:04:52.967626+0000","last_unstale":"2024-02-21T23:05:09.465868+0000","last_undegraded":"2024-02-21T23:05:09.465868+0000","last_fullsized":"2024-02-21T23:05:09.465868+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:33:33.298554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.114693+0000","last_change":"2024-02-21T23:05:08.452963+0000","last_active":"2024-02-21T23:05:10.114693+0000","last_peered":"2024-02-21T23:05:10.114693+0000","last_clean":"2024-02-21T23:05:10.114693+0000","last_became_active":"2024-02-21T23:05:08.452680+0000","last_became_peered":"2024-02-21T23:05:08.452680+0000","last_unstale":"2024-02-21T23:05:10.114693+0000","last_undegraded":"2024-02-21T23:05:10.114693+0000","last_fullsized":"2024-02-21T23:05:10.114693+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:07.393978+0000","last_clean_scrub_stamp":"2024-02-21T23:05:07.393978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:18:12.721687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:09.442927+0000","last_change":"2024-02-21T23:05:07.427069+0000","last_active":"2024-02-21T23:05:09.442927+0000","last_peered":"2024-02-21T23:05:09.442927+0000","last_clean":"2024-02-21T23:05:09.442927+0000","last_became_active":"2024-02-21T23:05:07.426609+0000","last_became_peered":"2024-02-21T23:05:07.426609+0000","last_unstale":"2024-02-21T23:05:09.442927+0000","last_undegraded":"2024-02-21T23:05:09.442927+0000","last_fullsized":"2024-02-21T23:05:09.442927+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:06.380814+0000","last_clean_scrub_stamp":"2024-02-21T23:05:06.380814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:41:32.687033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.113368+0000","last_change":"2024-02-21T23:04:53.985061+0000","last_active":"2024-02-21T23:05:10.113368+0000","last_peered":"2024-02-21T23:05:10.113368+0000","last_clean":"2024-02-21T23:05:10.113368+0000","last_became_active":"2024-02-21T23:04:53.984756+0000","last_became_peered":"2024-02-21T23:04:53.984756+0000","last_unstale":"2024-02-21T23:05:10.113368+0000","last_undegraded":"2024-02-21T23:05:10.113368+0000","last_fullsized":"2024-02-21T23:05:10.113368+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:52.929364+0000","last_clean_scrub_stamp":"2024-02-21T23:04:52.929364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:55:24.090577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"53'1","reported_seq":23,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.055151+0000","last_change":"2024-02-21T23:04:52.979253+0000","last_active":"2024-02-21T23:05:10.055151+0000","last_peered":"2024-02-21T23:05:10.055151+0000","last_clean":"2024-02-21T23:05:10.055151+0000","last_became_active":"2024-02-21T23:04:52.978773+0000","last_became_peered":"2024-02-21T23:04:52.978773+0000","last_unstale":"2024-02-21T23:05:10.055151+0000","last_undegraded":"2024-02-21T23:05:10.055151+0000","last_fullsized":"2024-02-21T23:05:10.055151+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":50,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:04:51.908146+0000","last_clean_scrub_stamp":"2024-02-21T23:04:51.908146+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:39:31.156140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":239,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-02-21T23:05:10.010467+0000","last_change":"2024-02-21T23:02:32.289335+0000","last_active":"2024-02-21T23:05:10.010467+0000","last_peered":"2024-02-21T23:05:10.010467+0000","last_clean":"2024-02-21T23:05:10.010467+0000","last_became_active":"2024-02-21T23:02:32.286913+0000","last_became_peered":"2024-02-21T23:02:32.286913+0000","last_unstale":"2024-02-21T23:05:10.010467+0000","last_undegraded":"2024-02-21T23:05:10.010467+0000","last_fullsized":"2024-02-21T23:05:10.010467+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_clean_scrub_stamp":"2024-02-21T23:02:31.172569+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:36:23.987374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574882,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27764,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344076,"statfs":{"total":96636764160,"available":96608333824,"internally_reserved":0,"allocated":757760,"data_stored":897214,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574883,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27700,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344140,"statfs":{"total":96636764160,"available":96608399360,"internally_reserved":0,"allocated":692224,"data_stored":314181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574882,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27692,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344148,"statfs":{"total":96636764160,"available":96608407552,"internally_reserved":0,"allocated":684032,"data_stored":313517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574883,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27704,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344136,"statfs":{"total":96636764160,"available":96608395264,"internally_reserved":0,"allocated":696320,"data_stored":315689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574883,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31864,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339976,"statfs":{"total":96636764160,"available":96604135424,"internally_reserved":0,"allocated":696320,"data_stored":315129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574884,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31860,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339980,"statfs":{"total":96636764160,"available":96604139520,"internally_reserved":0,"allocated":692224,"data_stored":315107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574882,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31888,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339952,"statfs":{"total":96636764160,"available":96604110848,"internally_reserved":0,"allocated":716800,"data_stored":316031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574882,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31908,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339932,"statfs":{"total":96636764160,"available":96604090368,"internally_reserved":0,"allocated":741376,"data_stored":896370,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-21T23:05:13.742 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-02-21T23:05:13.743 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail 3:0 2024-02-21T23:05:13.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.008+0000 7f8673e00640 1 Processor -- start 2024-02-21T23:05:13.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.009+0000 7f8673e00640 1 -- start start 2024-02-21T23:05:13.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.010+0000 7f8673e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f86741558a0 0x7f867406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:13.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.010+0000 7f8673e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8674158540 con 0x7f86740c6720 2024-02-21T23:05:13.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.010+0000 7f8673e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8674155cd0 con 0x7f867414ba70 2024-02-21T23:05:13.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.010+0000 7f8673e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86740c31c0 con 0x7f86741558a0 2024-02-21T23:05:13.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.010+0000 7f8673400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f86741558a0 0x7f867406d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:13.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.011+0000 7f8673400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f86741558a0 0x7f867406d2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36266/0 (socket says 172.21.15.28:36266) 2024-02-21T23:05:13.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.011+0000 7f8673400640 1 -- 172.21.15.28:0/1247782715 learned_addr learned my addr 172.21.15.28:0/1247782715 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:14.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.011+0000 7f8673400640 1 -- 172.21.15.28:0/1247782715 >> v1:172.21.15.81:6789/0 conn(0x7f867414ba70 legacy=0x7f86740c8b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:14.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.011+0000 7f8673400640 1 -- 172.21.15.28:0/1247782715 >> v1:172.21.15.28:6789/0 conn(0x7f86740c6720 legacy=0x7f8674158a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:14.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.011+0000 7f8673400640 1 -- 172.21.15.28:0/1247782715 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86740c34a0 con 0x7f86741558a0 2024-02-21T23:05:14.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.012+0000 7f8673400640 1 --2- 172.21.15.28:0/1247782715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86741558a0 0x7f867406d2c0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f866c009e50 tx=0x7f866c02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bc1595db69fafc4 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:14.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.012+0000 7f8671600640 1 -- 172.21.15.28:0/1247782715 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f866c02c9f0 con 0x7f86741558a0 2024-02-21T23:05:14.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.013+0000 7f8671600640 1 -- 172.21.15.28:0/1247782715 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f866c02cb90 con 0x7f86741558a0 2024-02-21T23:05:14.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.013+0000 7f8671600640 1 -- 172.21.15.28:0/1247782715 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f866c0343f0 con 0x7f86741558a0 2024-02-21T23:05:14.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.013+0000 7f8673e00640 1 -- 172.21.15.28:0/1247782715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86741558a0 msgr2=0x7f867406d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:14.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.013+0000 7f8673e00640 1 --2- 172.21.15.28:0/1247782715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86741558a0 0x7f867406d2c0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f866c009e50 tx=0x7f866c02be50 comp rx=0 tx=0).stop 2024-02-21T23:05:14.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.013+0000 7f8673e00640 1 -- 172.21.15.28:0/1247782715 shutdown_connections 2024-02-21T23:05:14.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.013+0000 7f8673e00640 1 --2- 172.21.15.28:0/1247782715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86741558a0 0x7f867406d2c0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:14.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.013+0000 7f8673e00640 1 -- 172.21.15.28:0/1247782715 >> 172.21.15.28:0/1247782715 conn(0x7f867406b260 msgr2=0x7f867406b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:14.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.013+0000 7f8673e00640 1 -- 172.21.15.28:0/1247782715 shutdown_connections 2024-02-21T23:05:14.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.013+0000 7f8673e00640 1 -- 172.21.15.28:0/1247782715 wait complete. 2024-02-21T23:05:14.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.015+0000 7f8673e00640 1 Processor -- start 2024-02-21T23:05:14.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.015+0000 7f8673e00640 1 -- start start 2024-02-21T23:05:14.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.016+0000 7f8673e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86740c6720 0x7f8674154590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:14.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.016+0000 7f8673e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f867414ba70 0x7f86741df5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:14.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.016+0000 7f8673e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86741558a0 0x7f86741dfb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:14.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.016+0000 7f8673e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86740c3a60 con 0x7f86740c6720 2024-02-21T23:05:14.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.016+0000 7f8673e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86740c2d20 con 0x7f86741558a0 2024-02-21T23:05:14.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.016+0000 7f8673e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86740c9230 con 0x7f867414ba70 2024-02-21T23:05:14.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.016+0000 7f8672a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86740c6720 0x7f8674154590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:14.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.016+0000 7f8672a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86740c6720 0x7f8674154590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36280/0 (socket says 172.21.15.28:36280) 2024-02-21T23:05:14.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.016+0000 7f8672a00640 1 -- 172.21.15.28:0/779331214 learned_addr learned my addr 172.21.15.28:0/779331214 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:14.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.017+0000 7f8673400640 1 --2- 172.21.15.28:0/779331214 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86741558a0 0x7f86741dfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:14.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.017+0000 7f8672000640 1 --2- 172.21.15.28:0/779331214 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f867414ba70 0x7f86741df5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:14.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.017+0000 7f8672a00640 1 -- 172.21.15.28:0/779331214 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f867414ba70 msgr2=0x7f86741df5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:14.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.017+0000 7f8672a00640 1 --2- 172.21.15.28:0/779331214 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f867414ba70 0x7f86741df5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:14.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.017+0000 7f8672a00640 1 -- 172.21.15.28:0/779331214 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86741558a0 msgr2=0x7f86741dfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:14.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.017+0000 7f8672a00640 1 --2- 172.21.15.28:0/779331214 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86741558a0 0x7f86741dfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:14.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.017+0000 7f8672a00640 1 -- 172.21.15.28:0/779331214 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8674156460 con 0x7f86740c6720 2024-02-21T23:05:14.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.017+0000 7f8672a00640 1 --2- 172.21.15.28:0/779331214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86740c6720 0x7f8674154590 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f866400c460 tx=0x7f866400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:14.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.018+0000 7f866aa00640 1 -- 172.21.15.28:0/779331214 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8664016020 con 0x7f86740c6720 2024-02-21T23:05:14.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.018+0000 7f866aa00640 1 -- 172.21.15.28:0/779331214 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8664012870 con 0x7f86740c6720 2024-02-21T23:05:14.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.018+0000 7f866aa00640 1 -- 172.21.15.28:0/779331214 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8664012c60 con 0x7f86740c6720 2024-02-21T23:05:14.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.018+0000 7f8673e00640 1 -- 172.21.15.28:0/779331214 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8674156740 con 0x7f86740c6720 2024-02-21T23:05:14.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.018+0000 7f8673e00640 1 -- 172.21.15.28:0/779331214 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8674156d00 con 0x7f86740c6720 2024-02-21T23:05:14.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.019+0000 7f8673e00640 1 -- 172.21.15.28:0/779331214 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86740c6280 con 0x7f86740c6720 2024-02-21T23:05:14.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.020+0000 7f866aa00640 1 -- 172.21.15.28:0/779331214 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8664019020 con 0x7f86740c6720 2024-02-21T23:05:14.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.021+0000 7f866aa00640 1 --2- 172.21.15.28:0/779331214 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f865406b4d0 0x7f865406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:14.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.022+0000 7f866aa00640 1 -- 172.21.15.28:0/779331214 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f866408a390 con 0x7f86740c6720 2024-02-21T23:05:14.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.022+0000 7f8672000640 1 --2- 172.21.15.28:0/779331214 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f865406b4d0 0x7f865406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:14.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.023+0000 7f8672000640 1 --2- 172.21.15.28:0/779331214 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f865406b4d0 0x7f865406d990 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f8674152630 tx=0x7f8660009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:14.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.025+0000 7f866aa00640 1 -- 172.21.15.28:0/779331214 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f86740c6280 con 0x7f86740c6720 2024-02-21T23:05:14.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.384+0000 7f8673e00640 1 -- 172.21.15.28:0/779331214 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "3:0"} v 0) v1 -- 0x7f867404f290 con 0x7f86740c6720 2024-02-21T23:05:14.373 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-02-21T23:05:14.385+0000 7f5ddc400640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-21T23:05:14.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.388+0000 7f866aa00640 1 -- 172.21.15.28:0/779331214 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "3:0"}]=0 failed mds gid 5715 v39) v1 ==== 97+0+0 (secure 0 0 0) 0x7f8664020030 con 0x7f86740c6720 2024-02-21T23:05:14.376 INFO:teuthology.orchestra.run.smithi028.stderr:failed mds gid 5715 2024-02-21T23:05:14.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.391+0000 7f8673e00640 1 -- 172.21.15.28:0/779331214 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f865406b4d0 msgr2=0x7f865406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:14.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.391+0000 7f8673e00640 1 --2- 172.21.15.28:0/779331214 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f865406b4d0 0x7f865406d990 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f8674152630 tx=0x7f8660009040 comp rx=0 tx=0).stop 2024-02-21T23:05:14.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.391+0000 7f8673e00640 1 -- 172.21.15.28:0/779331214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86740c6720 msgr2=0x7f8674154590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:14.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.391+0000 7f8673e00640 1 --2- 172.21.15.28:0/779331214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86740c6720 0x7f8674154590 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f866400c460 tx=0x7f866400c920 comp rx=0 tx=0).stop 2024-02-21T23:05:14.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.392+0000 7f8673e00640 1 -- 172.21.15.28:0/779331214 shutdown_connections 2024-02-21T23:05:14.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.392+0000 7f8673e00640 1 --2- 172.21.15.28:0/779331214 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f865406b4d0 0x7f865406d990 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:14.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.392+0000 7f8673e00640 1 --2- 172.21.15.28:0/779331214 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f867414ba70 0x7f86741df5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:14.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.392+0000 7f8673e00640 1 --2- 172.21.15.28:0/779331214 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86741558a0 0x7f86741dfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:14.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.392+0000 7f8673e00640 1 --2- 172.21.15.28:0/779331214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86740c6720 0x7f8674154590 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:14.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.392+0000 7f8673e00640 1 -- 172.21.15.28:0/779331214 >> 172.21.15.28:0/779331214 conn(0x7f867406b260 msgr2=0x7f86740700b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:14.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.394+0000 7f8673e00640 1 -- 172.21.15.28:0/779331214 shutdown_connections 2024-02-21T23:05:14.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.394+0000 7f8673e00640 1 -- 172.21.15.28:0/779331214 wait complete. 2024-02-21T23:05:14.390 INFO:tasks.ceph.mds.a.smithi028.stderr: -550> 2024-02-21T23:04:58.358+0000 7fabf3e00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-21T23:05:14.409 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:05:14.441 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-02-21T23:05:14.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.606+0000 7fe9e4a00640 1 Processor -- start 2024-02-21T23:05:14.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.606+0000 7fe9e4a00640 1 -- start start 2024-02-21T23:05:14.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.607+0000 7fe9e4a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fe9e00c11f0 0x7fe9e0153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:14.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.607+0000 7fe9e4a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe9e00c15d0 0x7fe9e006dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:14.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.607+0000 7fe9e4a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9e00c1a30 con 0x7fe9e0156350 2024-02-21T23:05:14.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.608+0000 7fe9e4a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9e00a5ec0 con 0x7fe9e00c11f0 2024-02-21T23:05:14.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.608+0000 7fe9e4a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e0152e70 con 0x7fe9e00c15d0 2024-02-21T23:05:14.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.608+0000 7fe9de800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe9e0156350 0x7fe9e01567a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34468/0 (socket says 172.21.15.28:34468) 2024-02-21T23:05:14.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.608+0000 7fe9de800640 1 -- 172.21.15.28:0/4199883551 learned_addr learned my addr 172.21.15.28:0/4199883551 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:05:14.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.608+0000 7fe9dde00640 1 --2- 172.21.15.28:0/4199883551 >> v2:172.21.15.81:3300/0 conn(0x7fe9e00c11f0 0x7fe9e0153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:14.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.608+0000 7fe9dd400640 1 --2- 172.21.15.28:0/4199883551 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe9e00c15d0 0x7fe9e006dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:14.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.608+0000 7fe9dca00640 1 -- 172.21.15.28:0/4199883551 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 803604021 0 0) 0x7fe9e00c1a30 con 0x7fe9e0156350 2024-02-21T23:05:14.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.608+0000 7fe9dca00640 1 -- 172.21.15.28:0/4199883551 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9c4003520 con 0x7fe9e0156350 2024-02-21T23:05:14.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.609+0000 7fe9dca00640 1 -- 172.21.15.28:0/4199883551 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2172129637 0 0) 0x7fe9c4003520 con 0x7fe9e0156350 2024-02-21T23:05:14.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.609+0000 7fe9dca00640 1 -- 172.21.15.28:0/4199883551 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe9e00c15d0 msgr2=0x7fe9e006dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:14.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.609+0000 7fe9dca00640 1 --2- 172.21.15.28:0/4199883551 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe9e00c15d0 0x7fe9e006dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:14.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.609+0000 7fe9dca00640 1 -- 172.21.15.28:0/4199883551 >> v2:172.21.15.81:3300/0 conn(0x7fe9e00c11f0 msgr2=0x7fe9e0153950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:14.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.609+0000 7fe9dca00640 1 --2- 172.21.15.28:0/4199883551 >> v2:172.21.15.81:3300/0 conn(0x7fe9e00c11f0 0x7fe9e0153950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:14.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.609+0000 7fe9dca00640 1 -- 172.21.15.28:0/4199883551 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9e0153150 con 0x7fe9e0156350 2024-02-21T23:05:14.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.609+0000 7fe9dca00640 1 -- 172.21.15.28:0/4199883551 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fe9d80019f0 con 0x7fe9e0156350 2024-02-21T23:05:14.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.609+0000 7fe9dca00640 1 -- 172.21.15.28:0/4199883551 >> v1:172.21.15.28:6789/0 conn(0x7fe9e0156350 legacy=0x7fe9e01567a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:14.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.609+0000 7fe9dca00640 1 --2- 172.21.15.28:0/4199883551 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9c40039b0 0x7fe9c4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:14.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.609+0000 7fe9dca00640 1 --2- 172.21.15.28:0/4199883551 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe9e00c15d0 0x7fe9c4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:14.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.609+0000 7fe9dca00640 1 --2- 172.21.15.28:0/4199883551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9c4004d40 0x7fe9c40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:14.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.609+0000 7fe9dca00640 1 -- 172.21.15.28:0/4199883551 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e00a5ec0 con 0x7fe9c4004d40 2024-02-21T23:05:14.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.609+0000 7fe9dca00640 1 -- 172.21.15.28:0/4199883551 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e0152e70 con 0x7fe9c40039b0 2024-02-21T23:05:14.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.609+0000 7fe9dca00640 1 -- 172.21.15.28:0/4199883551 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c400ba70 con 0x7fe9e00c15d0 2024-02-21T23:05:14.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.609+0000 7fe9de800640 1 --2- 172.21.15.28:0/4199883551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9c4004d40 0x7fe9c40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:14.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.610+0000 7fe9dde00640 1 --2- 172.21.15.28:0/4199883551 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9c40039b0 0x7fe9c4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:14.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.610+0000 7fe9dd400640 1 --2- 172.21.15.28:0/4199883551 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe9e00c15d0 0x7fe9c4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:14.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.610+0000 7fe9de800640 1 -- 172.21.15.28:0/4199883551 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe9e00c15d0 msgr2=0x7fe9c4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:14.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.610+0000 7fe9de800640 1 --2- 172.21.15.28:0/4199883551 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe9e00c15d0 0x7fe9c4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:14.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.610+0000 7fe9de800640 1 -- 172.21.15.28:0/4199883551 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9c40039b0 msgr2=0x7fe9c4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:14.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.610+0000 7fe9de800640 1 --2- 172.21.15.28:0/4199883551 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9c40039b0 0x7fe9c4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:14.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.610+0000 7fe9de800640 1 -- 172.21.15.28:0/4199883551 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c4003520 con 0x7fe9c4004d40 2024-02-21T23:05:14.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.610+0000 7fe9dd400640 1 --2- 172.21.15.28:0/4199883551 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe9e00c15d0 0x7fe9c4004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:05:14.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.610+0000 7fe9dde00640 1 --2- 172.21.15.28:0/4199883551 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9c40039b0 0x7fe9c4003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:05:14.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.611+0000 7fe9de800640 1 --2- 172.21.15.28:0/4199883551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9c4004d40 0x7fe9c40051c0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fe9d800a8a0 tx=0x7fe9c4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8563900881805d9 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:14.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.611+0000 7fe9dca00640 1 -- 172.21.15.28:0/4199883551 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d800a4c0 con 0x7fe9c4004d40 2024-02-21T23:05:14.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.611+0000 7fe9dca00640 1 -- 172.21.15.28:0/4199883551 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9d8034650 con 0x7fe9c4004d40 2024-02-21T23:05:14.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.611+0000 7fe9dca00640 1 -- 172.21.15.28:0/4199883551 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d8034a40 con 0x7fe9c4004d40 2024-02-21T23:05:14.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.612+0000 7fe9e4a00640 1 -- 172.21.15.28:0/4199883551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9c4004d40 msgr2=0x7fe9c40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:14.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.612+0000 7fe9e4a00640 1 --2- 172.21.15.28:0/4199883551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9c4004d40 0x7fe9c40051c0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fe9d800a8a0 tx=0x7fe9c4005830 comp rx=0 tx=0).stop 2024-02-21T23:05:14.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.612+0000 7fe9de800640 1 -- 172.21.15.28:0/4199883551 reap_dead start 2024-02-21T23:05:14.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.612+0000 7fe9e4a00640 1 -- 172.21.15.28:0/4199883551 shutdown_connections 2024-02-21T23:05:14.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.612+0000 7fe9e4a00640 1 -- 172.21.15.28:0/4199883551 >> 172.21.15.28:0/4199883551 conn(0x7fe9e00c67f0 msgr2=0x7fe9e00c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:14.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.612+0000 7fe9e4a00640 1 -- 172.21.15.28:0/4199883551 shutdown_connections 2024-02-21T23:05:14.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.612+0000 7fe9e4a00640 1 -- 172.21.15.28:0/4199883551 wait complete. 2024-02-21T23:05:14.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.614+0000 7fe9e4a00640 1 Processor -- start 2024-02-21T23:05:14.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.614+0000 7fe9e4a00640 1 -- start start 2024-02-21T23:05:14.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.615+0000 7fe9e4a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9c4004d40 0x7fe9e00a63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:14.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.615+0000 7fe9e4a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe9e01529c0 0x7fe9e0155560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:14.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.615+0000 7fe9e4a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9e0155aa0 0x7fe9e0151b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:14.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.615+0000 7fe9e4a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e00c4c00 con 0x7fe9c4004d40 2024-02-21T23:05:14.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.615+0000 7fe9e4a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e014d1c0 con 0x7fe9e0155aa0 2024-02-21T23:05:14.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.615+0000 7fe9e4a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e00a6110 con 0x7fe9e01529c0 2024-02-21T23:05:14.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.615+0000 7fe9dde00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9c4004d40 0x7fe9e00a63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:14.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.615+0000 7fe9dde00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9c4004d40 0x7fe9e00a63d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36318/0 (socket says 172.21.15.28:36318) 2024-02-21T23:05:14.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.615+0000 7fe9dde00640 1 -- 172.21.15.28:0/957215092 learned_addr learned my addr 172.21.15.28:0/957215092 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:14.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.615+0000 7fe9dd400640 1 --2- 172.21.15.28:0/957215092 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe9e01529c0 0x7fe9e0155560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:14.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.615+0000 7fe9de800640 1 --2- 172.21.15.28:0/957215092 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9e0155aa0 0x7fe9e0151b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:14.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.616+0000 7fe9dde00640 1 -- 172.21.15.28:0/957215092 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe9e01529c0 msgr2=0x7fe9e0155560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:14.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.616+0000 7fe9dde00640 1 --2- 172.21.15.28:0/957215092 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe9e01529c0 0x7fe9e0155560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:14.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.616+0000 7fe9dde00640 1 -- 172.21.15.28:0/957215092 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9e0155aa0 msgr2=0x7fe9e0151b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:14.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.616+0000 7fe9dde00640 1 --2- 172.21.15.28:0/957215092 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9e0155aa0 0x7fe9e0151b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:14.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.616+0000 7fe9dde00640 1 -- 172.21.15.28:0/957215092 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9e01dd180 con 0x7fe9c4004d40 2024-02-21T23:05:14.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.616+0000 7fe9dde00640 1 --2- 172.21.15.28:0/957215092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9c4004d40 0x7fe9e00a63d0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fe9d0011600 tx=0x7fe9d0011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:14.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.616+0000 7fe9d6000640 1 -- 172.21.15.28:0/957215092 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d000f920 con 0x7fe9c4004d40 2024-02-21T23:05:14.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.616+0000 7fe9d6000640 1 -- 172.21.15.28:0/957215092 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9d000fac0 con 0x7fe9c4004d40 2024-02-21T23:05:14.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.616+0000 7fe9d6000640 1 -- 172.21.15.28:0/957215092 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d000fe90 con 0x7fe9c4004d40 2024-02-21T23:05:14.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.616+0000 7fe9e4a00640 1 -- 172.21.15.28:0/957215092 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9e01e9190 con 0x7fe9c4004d40 2024-02-21T23:05:14.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.616+0000 7fe9e4a00640 1 -- 172.21.15.28:0/957215092 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9e0154d60 con 0x7fe9c4004d40 2024-02-21T23:05:14.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.618+0000 7fe9e4a00640 1 -- 172.21.15.28:0/957215092 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9e0067f30 con 0x7fe9c4004d40 2024-02-21T23:05:14.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.619+0000 7fe9d6000640 1 -- 172.21.15.28:0/957215092 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe9d0015460 con 0x7fe9c4004d40 2024-02-21T23:05:14.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.620+0000 7fe9d6000640 1 --2- 172.21.15.28:0/957215092 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe9c006b670 0x7fe9c006db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:14.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.620+0000 7fe9d6000640 1 -- 172.21.15.28:0/957215092 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7043+0+0 (secure 0 0 0) 0x7fe9d008b130 con 0x7fe9c4004d40 2024-02-21T23:05:14.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.621+0000 7fe9dd400640 1 --2- 172.21.15.28:0/957215092 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe9c006b670 0x7fe9c006db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:14.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.622+0000 7fe9dd400640 1 --2- 172.21.15.28:0/957215092 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe9c006b670 0x7fe9c006db30 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fe9e014b960 tx=0x7fe9cc008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:14.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.625+0000 7fe9d6000640 1 -- 172.21.15.28:0/957215092 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe9e01dd180 con 0x7fe9c4004d40 2024-02-21T23:05:14.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.861+0000 7fe9e4a00640 1 -- 172.21.15.28:0/957215092 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe9e00c6450 con 0x7fe9c4004d40 2024-02-21T23:05:14.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.862+0000 7fe9d6000640 1 -- 172.21.15.28:0/957215092 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+5385 (secure 0 0 0) 0x7fe9d0019030 con 0x7fe9c4004d40 2024-02-21T23:05:14.851 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:05:14.851 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":40,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5221,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":7,"addr":"172.21.15.81:6835/2152160500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2152160500},{"type":"v1","addr":"172.21.15.81:6835","nonce":2152160500}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:04:53.432188+0000","modified":"2024-02-21T23:05:14.394868+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5006},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5006":{"gid":5006,"name":"c","rank":0,"incarnation":40,"state":"up:replay","state_seq":7,"addr":"172.21.15.28:6835/3468780363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3468780363},{"type":"v1","addr":"172.21.15.28:6835","nonce":3468780363}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3},{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:05:08.017586+0000","modified":"2024-02-21T23:05:10.974691+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5224},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5224":{"gid":5224,"name":"d","rank":0,"incarnation":33,"state":"up:active","state_seq":7,"addr":"172.21.15.81:6837/442767245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":442767245},{"type":"v1","addr":"172.21.15.81:6837","nonce":442767245}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-02-21T23:05:14.851 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 40 2024-02-21T23:05:14.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.865+0000 7fe9e4a00640 1 -- 172.21.15.28:0/957215092 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe9c006b670 msgr2=0x7fe9c006db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:14.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.865+0000 7fe9e4a00640 1 --2- 172.21.15.28:0/957215092 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe9c006b670 0x7fe9c006db30 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fe9e014b960 tx=0x7fe9cc008040 comp rx=0 tx=0).stop 2024-02-21T23:05:14.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.865+0000 7fe9e4a00640 1 -- 172.21.15.28:0/957215092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9c4004d40 msgr2=0x7fe9e00a63d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:14.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.865+0000 7fe9e4a00640 1 --2- 172.21.15.28:0/957215092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9c4004d40 0x7fe9e00a63d0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fe9d0011600 tx=0x7fe9d0011ac0 comp rx=0 tx=0).stop 2024-02-21T23:05:14.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.865+0000 7fe9e4a00640 1 -- 172.21.15.28:0/957215092 shutdown_connections 2024-02-21T23:05:14.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.865+0000 7fe9e4a00640 1 --2- 172.21.15.28:0/957215092 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe9c006b670 0x7fe9c006db30 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:14.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.865+0000 7fe9e4a00640 1 --2- 172.21.15.28:0/957215092 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe9e01529c0 0x7fe9e0155560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:14.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.865+0000 7fe9e4a00640 1 --2- 172.21.15.28:0/957215092 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9e0155aa0 0x7fe9e0151b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:14.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.865+0000 7fe9e4a00640 1 --2- 172.21.15.28:0/957215092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9c4004d40 0x7fe9e00a63d0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:14.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.865+0000 7fe9e4a00640 1 -- 172.21.15.28:0/957215092 >> 172.21.15.28:0/957215092 conn(0x7fe9e00c67f0 msgr2=0x7fe9e00c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:14.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.865+0000 7fe9e4a00640 1 -- 172.21.15.28:0/957215092 shutdown_connections 2024-02-21T23:05:14.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:14.865+0000 7fe9e4a00640 1 -- 172.21.15.28:0/957215092 wait complete. 2024-02-21T23:05:14.872 INFO:tasks.cephfs.test_failover:Waiting for former active to reclaim its spot 2024-02-21T23:05:14.872 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok --format=json config get mds_beacon_grace 2024-02-21T23:05:15.136 INFO:teuthology.orchestra.run.smithi028.stdout:{"mds_beacon_grace":"15.000000"} 2024-02-21T23:05:15.137 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_beacon_grace": "15.000000" } 2024-02-21T23:05:15.137 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:05:15.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.374+0000 7f17d6800640 1 Processor -- start 2024-02-21T23:05:15.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.374+0000 7f17d6800640 1 -- start start 2024-02-21T23:05:15.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.376+0000 7f17d6800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f17d00aff90 0x7f17d00c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:15.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.376+0000 7f17d6800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f17d00c5720 0x7f17d015f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:15.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.376+0000 7f17d6800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17d00c74c0 con 0x7f17d006dbf0 2024-02-21T23:05:15.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.376+0000 7f17d6800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17d00c7270 con 0x7f17d00aff90 2024-02-21T23:05:15.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.376+0000 7f17d6800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17d0158630 con 0x7f17d00c5720 2024-02-21T23:05:15.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.376+0000 7f17d5400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f17d00aff90 0x7f17d00c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:15.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.376+0000 7f17d5400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f17d00aff90 0x7f17d00c51e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41370/0 (socket says 172.21.15.28:41370) 2024-02-21T23:05:15.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.376+0000 7f17d5400640 1 -- 172.21.15.28:0/1283666880 learned_addr learned my addr 172.21.15.28:0/1283666880 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:15.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.377+0000 7f17d5400640 1 -- 172.21.15.28:0/1283666880 >> v2:172.21.15.81:3300/0 conn(0x7f17d00c5720 msgr2=0x7f17d015f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:15.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.377+0000 7f17d5400640 1 --2- 172.21.15.28:0/1283666880 >> v2:172.21.15.81:3300/0 conn(0x7f17d00c5720 0x7f17d015f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:15.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.377+0000 7f17d5400640 1 -- 172.21.15.28:0/1283666880 >> v1:172.21.15.81:6789/0 conn(0x7f17d006dbf0 legacy=0x7f17d0158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:15.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.377+0000 7f17d5400640 1 -- 172.21.15.28:0/1283666880 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17d00c77a0 con 0x7f17d00aff90 2024-02-21T23:05:15.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.378+0000 7f17d5400640 1 --2- 172.21.15.28:0/1283666880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17d00aff90 0x7f17d00c51e0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f17d00c73f0 tx=0x7f17c402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c22dda0717d5f2b1 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:15.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.378+0000 7f17cfe00640 1 -- 172.21.15.28:0/1283666880 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c4035020 con 0x7f17d00aff90 2024-02-21T23:05:15.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.378+0000 7f17cfe00640 1 -- 172.21.15.28:0/1283666880 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17c4031870 con 0x7f17d00aff90 2024-02-21T23:05:15.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.378+0000 7f17cfe00640 1 -- 172.21.15.28:0/1283666880 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c4031c40 con 0x7f17d00aff90 2024-02-21T23:05:15.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.379+0000 7f17d6800640 1 -- 172.21.15.28:0/1283666880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17d00aff90 msgr2=0x7f17d00c51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:15.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.379+0000 7f17d6800640 1 --2- 172.21.15.28:0/1283666880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17d00aff90 0x7f17d00c51e0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f17d00c73f0 tx=0x7f17c402b910 comp rx=0 tx=0).stop 2024-02-21T23:05:15.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.379+0000 7f17d6800640 1 -- 172.21.15.28:0/1283666880 shutdown_connections 2024-02-21T23:05:15.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.379+0000 7f17d6800640 1 --2- 172.21.15.28:0/1283666880 >> v2:172.21.15.81:3300/0 conn(0x7f17d00c5720 0x7f17d015f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:15.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.379+0000 7f17d6800640 1 --2- 172.21.15.28:0/1283666880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17d00aff90 0x7f17d00c51e0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:15.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.379+0000 7f17d6800640 1 -- 172.21.15.28:0/1283666880 >> 172.21.15.28:0/1283666880 conn(0x7f17d00ac370 msgr2=0x7f17d00ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:15.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.379+0000 7f17d6800640 1 -- 172.21.15.28:0/1283666880 shutdown_connections 2024-02-21T23:05:15.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.379+0000 7f17d6800640 1 -- 172.21.15.28:0/1283666880 wait complete. 2024-02-21T23:05:15.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.381+0000 7f17d6800640 1 Processor -- start 2024-02-21T23:05:15.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.382+0000 7f17d6800640 1 -- start start 2024-02-21T23:05:15.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.383+0000 7f17d6800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17d006dbf0 0x7f17d01eba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:15.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.383+0000 7f17d6800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f17d00aff90 0x7f17d01eed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:15.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.383+0000 7f17d6800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f17d00c5720 0x7f17d00ab470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:15.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.383+0000 7f17d6800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17d00c8760 con 0x7f17d006dbf0 2024-02-21T23:05:15.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.383+0000 7f17d6800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17d00be0c0 con 0x7f17d00c5720 2024-02-21T23:05:15.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.383+0000 7f17d6800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17d00c88e0 con 0x7f17d00aff90 2024-02-21T23:05:15.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.383+0000 7f17d5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17d006dbf0 0x7f17d01eba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:15.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.383+0000 7f17d5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17d006dbf0 0x7f17d01eba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41386/0 (socket says 172.21.15.28:41386) 2024-02-21T23:05:15.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.383+0000 7f17d5400640 1 -- 172.21.15.28:0/1927237378 learned_addr learned my addr 172.21.15.28:0/1927237378 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:15.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.384+0000 7f17d5e00640 1 --2- 172.21.15.28:0/1927237378 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f17d00c5720 0x7f17d00ab470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:15.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.384+0000 7f17d4a00640 1 --2- 172.21.15.28:0/1927237378 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f17d00aff90 0x7f17d01eed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:15.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.384+0000 7f17d5400640 1 -- 172.21.15.28:0/1927237378 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f17d00aff90 msgr2=0x7f17d01eed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:15.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.384+0000 7f17d5400640 1 --2- 172.21.15.28:0/1927237378 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f17d00aff90 0x7f17d01eed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:15.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.384+0000 7f17d5400640 1 -- 172.21.15.28:0/1927237378 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f17d00c5720 msgr2=0x7f17d00ab470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:15.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.384+0000 7f17d5400640 1 --2- 172.21.15.28:0/1927237378 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f17d00c5720 0x7f17d00ab470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:15.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.384+0000 7f17d5400640 1 -- 172.21.15.28:0/1927237378 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17d01ef280 con 0x7f17d006dbf0 2024-02-21T23:05:15.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.384+0000 7f17d5400640 1 --2- 172.21.15.28:0/1927237378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17d006dbf0 0x7f17d01eba00 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f17c4000c00 tx=0x7f17c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:15.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.384+0000 7f17cd600640 1 -- 172.21.15.28:0/1927237378 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c40443f0 con 0x7f17d006dbf0 2024-02-21T23:05:15.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.385+0000 7f17cd600640 1 -- 172.21.15.28:0/1927237378 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17c4044590 con 0x7f17d006dbf0 2024-02-21T23:05:15.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.385+0000 7f17cd600640 1 -- 172.21.15.28:0/1927237378 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c4044980 con 0x7f17d006dbf0 2024-02-21T23:05:15.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.385+0000 7f17d6800640 1 -- 172.21.15.28:0/1927237378 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17d01548d0 con 0x7f17d006dbf0 2024-02-21T23:05:15.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.385+0000 7f17d6800640 1 -- 172.21.15.28:0/1927237378 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17d01eb200 con 0x7f17d006dbf0 2024-02-21T23:05:15.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.386+0000 7f17d6800640 1 -- 172.21.15.28:0/1927237378 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17d004f290 con 0x7f17d006dbf0 2024-02-21T23:05:15.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.387+0000 7f17cd600640 1 -- 172.21.15.28:0/1927237378 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f17c4038020 con 0x7f17d006dbf0 2024-02-21T23:05:15.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.388+0000 7f17cd600640 1 --2- 172.21.15.28:0/1927237378 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f17b006b4d0 0x7f17b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:15.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.388+0000 7f17cd600640 1 -- 172.21.15.28:0/1927237378 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 7043+0+0 (secure 0 0 0) 0x7f17c40acae0 con 0x7f17d006dbf0 2024-02-21T23:05:15.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.388+0000 7f17d4a00640 1 --2- 172.21.15.28:0/1927237378 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f17b006b4d0 0x7f17b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:15.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.390+0000 7f17d4a00640 1 --2- 172.21.15.28:0/1927237378 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f17b006b4d0 0x7f17b006d990 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f17d0152c40 tx=0x7f17bc009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:15.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.393+0000 7f17cd600640 1 -- 172.21.15.28:0/1927237378 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f17c403f070 con 0x7f17d006dbf0 2024-02-21T23:05:15.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.647+0000 7f17d6800640 1 -- 172.21.15.28:0/1927237378 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f17d01524f0 con 0x7f17d006dbf0 2024-02-21T23:05:15.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.648+0000 7f17cd600640 1 -- 172.21.15.28:0/1927237378 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+6179 (secure 0 0 0) 0x7f17c407b410 con 0x7f17d006dbf0 2024-02-21T23:05:15.636 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:05:15.637 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":41,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5221,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":7,"addr":"172.21.15.81:6835/2152160500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2152160500},{"type":"v1","addr":"172.21.15.81:6835","nonce":2152160500}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":6021,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3835332438","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3835332438},{"type":"v1","addr":"172.21.15.28:6834","nonce":3835332438}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:04:53.432188+0000","modified":"2024-02-21T23:05:14.424939+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5006},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5006":{"gid":5006,"name":"c","rank":0,"incarnation":40,"state":"up:reconnect","state_seq":8,"addr":"172.21.15.28:6835/3468780363","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3468780363},{"type":"v1","addr":"172.21.15.28:6835","nonce":3468780363}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3},{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:05:08.017586+0000","modified":"2024-02-21T23:05:10.974691+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5224},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5224":{"gid":5224,"name":"d","rank":0,"incarnation":33,"state":"up:active","state_seq":7,"addr":"172.21.15.81:6837/442767245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":442767245},{"type":"v1","addr":"172.21.15.81:6837","nonce":442767245}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-02-21T23:05:15.637 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 41 2024-02-21T23:05:15.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.651+0000 7f17d6800640 1 -- 172.21.15.28:0/1927237378 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f17b006b4d0 msgr2=0x7f17b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:15.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.651+0000 7f17d6800640 1 --2- 172.21.15.28:0/1927237378 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f17b006b4d0 0x7f17b006d990 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f17d0152c40 tx=0x7f17bc009040 comp rx=0 tx=0).stop 2024-02-21T23:05:15.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.651+0000 7f17d6800640 1 -- 172.21.15.28:0/1927237378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17d006dbf0 msgr2=0x7f17d01eba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:15.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.651+0000 7f17d6800640 1 --2- 172.21.15.28:0/1927237378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17d006dbf0 0x7f17d01eba00 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f17c4000c00 tx=0x7f17c40046c0 comp rx=0 tx=0).stop 2024-02-21T23:05:15.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.651+0000 7f17d6800640 1 -- 172.21.15.28:0/1927237378 shutdown_connections 2024-02-21T23:05:15.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.651+0000 7f17d6800640 1 --2- 172.21.15.28:0/1927237378 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f17b006b4d0 0x7f17b006d990 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:15.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.651+0000 7f17d6800640 1 --2- 172.21.15.28:0/1927237378 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f17d00aff90 0x7f17d01eed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:15.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.651+0000 7f17d6800640 1 --2- 172.21.15.28:0/1927237378 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f17d00c5720 0x7f17d00ab470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:15.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.651+0000 7f17d6800640 1 --2- 172.21.15.28:0/1927237378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17d006dbf0 0x7f17d01eba00 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:15.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.651+0000 7f17d6800640 1 -- 172.21.15.28:0/1927237378 >> 172.21.15.28:0/1927237378 conn(0x7f17d00ac370 msgr2=0x7f17d00c9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:15.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.651+0000 7f17d6800640 1 -- 172.21.15.28:0/1927237378 shutdown_connections 2024-02-21T23:05:15.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:15.651+0000 7f17d6800640 1 -- 172.21.15.28:0/1927237378 wait complete. 2024-02-21T23:05:15.661 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=45.0 retry_count=0)... 2024-02-21T23:05:17.469 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-02-21T23:05:20.662 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:05:20.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:20.903+0000 7f933be00640 1 Processor -- start 2024-02-21T23:05:20.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:20.903+0000 7f933be00640 1 -- start start 2024-02-21T23:05:20.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:20.904+0000 7f933be00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f9334153950 0x7f93340c15d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:20.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:20.904+0000 7f933be00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f933415b6d0 0x7f933415dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:20.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:20.904+0000 7f933be00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9334152e70 con 0x7f93340c11f0 2024-02-21T23:05:20.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:20.904+0000 7f933be00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93340a5ec0 con 0x7f9334153950 2024-02-21T23:05:20.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:20.904+0000 7f933be00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9334153150 con 0x7f933415b6d0 2024-02-21T23:05:20.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:20.905+0000 7f9339e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f933415b6d0 0x7f933415dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:20.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:20.905+0000 7f9338a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f9334153950 0x7f93340c15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:20.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:20.905+0000 7f9339400640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f93340c11f0 0x7f933406dce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.28:41480/0 (socket says 172.21.15.28:41480) 2024-02-21T23:05:20.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:20.905+0000 7f9339400640 1 -- 172.21.15.28:0/1840105829 learned_addr learned my addr 172.21.15.28:0/1840105829 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:05:20.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:20.906+0000 7f9338a00640 1 -- 172.21.15.28:0/1840105829 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f933415b6d0 msgr2=0x7f933415dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:20.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:20.906+0000 7f9338a00640 1 --2- 172.21.15.28:0/1840105829 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f933415b6d0 0x7f933415dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:20.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:20.906+0000 7f9338a00640 1 -- 172.21.15.28:0/1840105829 >> v1:172.21.15.81:6789/0 conn(0x7f93340c11f0 legacy=0x7f933406dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:20.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:20.906+0000 7f9338a00640 1 -- 172.21.15.28:0/1840105829 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9334153430 con 0x7f9334153950 2024-02-21T23:05:20.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:20.908+0000 7f9338a00640 1 --2- 172.21.15.28:0/1840105829 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9334153950 0x7f93340c15d0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f93200097a0 tx=0x7f932002b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e061a9f2d646597 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:20.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:20.908+0000 7f9333e00640 1 -- 172.21.15.28:0/1840105829 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9320031780 con 0x7f9334153950 2024-02-21T23:05:20.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:20.908+0000 7f9333e00640 1 -- 172.21.15.28:0/1840105829 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9320037070 con 0x7f9334153950 2024-02-21T23:05:20.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:20.908+0000 7f9333e00640 1 -- 172.21.15.28:0/1840105829 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9320031b40 con 0x7f9334153950 2024-02-21T23:05:20.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:20.909+0000 7f933be00640 1 -- 172.21.15.28:0/1840105829 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9334153950 msgr2=0x7f93340c15d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:20.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:20.909+0000 7f933be00640 1 --2- 172.21.15.28:0/1840105829 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9334153950 0x7f93340c15d0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f93200097a0 tx=0x7f932002b820 comp rx=0 tx=0).stop 2024-02-21T23:05:20.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:20.909+0000 7f933be00640 1 -- 172.21.15.28:0/1840105829 shutdown_connections 2024-02-21T23:05:20.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:20.909+0000 7f933be00640 1 --2- 172.21.15.28:0/1840105829 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f933415b6d0 0x7f933415dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:20.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:20.909+0000 7f933be00640 1 --2- 172.21.15.28:0/1840105829 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9334153950 0x7f93340c15d0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:20.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:20.909+0000 7f933be00640 1 -- 172.21.15.28:0/1840105829 >> 172.21.15.28:0/1840105829 conn(0x7f93340c67f0 msgr2=0x7f93340c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:20.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:20.909+0000 7f933be00640 1 -- 172.21.15.28:0/1840105829 shutdown_connections 2024-02-21T23:05:20.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:20.909+0000 7f933be00640 1 -- 172.21.15.28:0/1840105829 wait complete. 2024-02-21T23:05:20.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:20.911+0000 7f933be00640 1 Processor -- start 2024-02-21T23:05:20.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:20.911+0000 7f933be00640 1 -- start start 2024-02-21T23:05:20.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:20.912+0000 7f933be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93340c11f0 0x7f93341eed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:20.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:20.912+0000 7f933be00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9334153950 0x7f93340a7550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:20.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:20.912+0000 7f933be00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f933415b6d0 0x7f93341eada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:20.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:20.912+0000 7f933be00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93340a6110 con 0x7f93340c11f0 2024-02-21T23:05:20.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:20.912+0000 7f933be00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93340c4c00 con 0x7f933415b6d0 2024-02-21T23:05:20.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:20.912+0000 7f933be00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93340c0d90 con 0x7f9334153950 2024-02-21T23:05:20.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:20.912+0000 7f9339400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93340c11f0 0x7f93341eed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:20.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:20.912+0000 7f9339400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93340c11f0 0x7f93341eed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41398/0 (socket says 172.21.15.28:41398) 2024-02-21T23:05:20.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:20.912+0000 7f9339400640 1 -- 172.21.15.28:0/3136236421 learned_addr learned my addr 172.21.15.28:0/3136236421 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:20.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:20.913+0000 7f9339e00640 1 --2- 172.21.15.28:0/3136236421 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f933415b6d0 0x7f93341eada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:20.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:20.913+0000 7f9338a00640 1 --2- 172.21.15.28:0/3136236421 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9334153950 0x7f93340a7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:20.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:20.913+0000 7f9339400640 1 -- 172.21.15.28:0/3136236421 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9334153950 msgr2=0x7f93340a7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:20.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:20.913+0000 7f9339400640 1 --2- 172.21.15.28:0/3136236421 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9334153950 0x7f93340a7550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:20.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:20.913+0000 7f9339400640 1 -- 172.21.15.28:0/3136236421 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f933415b6d0 msgr2=0x7f93341eada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:20.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:20.913+0000 7f9339400640 1 --2- 172.21.15.28:0/3136236421 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f933415b6d0 0x7f93341eada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:20.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:20.913+0000 7f9339400640 1 -- 172.21.15.28:0/3136236421 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93340a9030 con 0x7f93340c11f0 2024-02-21T23:05:20.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:20.913+0000 7f9339400640 1 --2- 172.21.15.28:0/3136236421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93340c11f0 0x7f93341eed20 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f932800c3f0 tx=0x7f932800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:20.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:20.913+0000 7f9331600640 1 -- 172.21.15.28:0/3136236421 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9328017070 con 0x7f93340c11f0 2024-02-21T23:05:20.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:20.913+0000 7f9331600640 1 -- 172.21.15.28:0/3136236421 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9328005ce0 con 0x7f93340c11f0 2024-02-21T23:05:20.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:20.913+0000 7f9331600640 1 -- 172.21.15.28:0/3136236421 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93280072f0 con 0x7f93340c11f0 2024-02-21T23:05:20.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:20.914+0000 7f933be00640 1 -- 172.21.15.28:0/3136236421 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93340a92d0 con 0x7f93340c11f0 2024-02-21T23:05:20.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:20.914+0000 7f933be00640 1 -- 172.21.15.28:0/3136236421 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93340a7a90 con 0x7f93340c11f0 2024-02-21T23:05:20.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:20.915+0000 7f933be00640 1 -- 172.21.15.28:0/3136236421 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93340c46f0 con 0x7f93340c11f0 2024-02-21T23:05:20.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:20.916+0000 7f9331600640 1 -- 172.21.15.28:0/3136236421 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f932801a070 con 0x7f93340c11f0 2024-02-21T23:05:20.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:20.917+0000 7f9331600640 1 --2- 172.21.15.28:0/3136236421 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f931406b4d0 0x7f931406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:20.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:20.917+0000 7f9331600640 1 -- 172.21.15.28:0/3136236421 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7129+0+0 (secure 0 0 0) 0x7f9328089b10 con 0x7f93340c11f0 2024-02-21T23:05:20.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:20.918+0000 7f9338a00640 1 --2- 172.21.15.28:0/3136236421 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f931406b4d0 0x7f931406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:20.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:20.919+0000 7f9338a00640 1 --2- 172.21.15.28:0/3136236421 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f931406b4d0 0x7f931406d990 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f9320036000 tx=0x7f932003d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:20.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:20.922+0000 7f9331600640 1 -- 172.21.15.28:0/3136236421 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f93340c46f0 con 0x7f93340c11f0 2024-02-21T23:05:21.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.169+0000 7f933be00640 1 -- 172.21.15.28:0/3136236421 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f93341ee310 con 0x7f93340c11f0 2024-02-21T23:05:21.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.170+0000 7f9331600640 1 -- 172.21.15.28:0/3136236421 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+6176 (secure 0 0 0) 0x7f9328058410 con 0x7f93340c11f0 2024-02-21T23:05:21.159 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:05:21.159 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":47,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5221,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":7,"addr":"172.21.15.81:6835/2152160500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2152160500},{"type":"v1","addr":"172.21.15.81:6835","nonce":2152160500}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":6039,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3807390545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3807390545},{"type":"v1","addr":"172.21.15.28:6835","nonce":3807390545}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:04:53.432188+0000","modified":"2024-02-21T23:05:19.431400+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6021},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6021":{"gid":6021,"name":"a","rank":0,"incarnation":44,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6834/3835332438","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3835332438},{"type":"v1","addr":"172.21.15.28:6834","nonce":3835332438}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3},{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:05:08.017586+0000","modified":"2024-02-21T23:05:10.974691+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5224},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5224":{"gid":5224,"name":"d","rank":0,"incarnation":33,"state":"up:active","state_seq":7,"addr":"172.21.15.81:6837/442767245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":442767245},{"type":"v1","addr":"172.21.15.81:6837","nonce":442767245}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-02-21T23:05:21.159 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 47 2024-02-21T23:05:21.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.172+0000 7f933be00640 1 -- 172.21.15.28:0/3136236421 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f931406b4d0 msgr2=0x7f931406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:21.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.173+0000 7f933be00640 1 --2- 172.21.15.28:0/3136236421 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f931406b4d0 0x7f931406d990 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f9320036000 tx=0x7f932003d040 comp rx=0 tx=0).stop 2024-02-21T23:05:21.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.173+0000 7f933be00640 1 -- 172.21.15.28:0/3136236421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93340c11f0 msgr2=0x7f93341eed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:21.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.173+0000 7f933be00640 1 --2- 172.21.15.28:0/3136236421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93340c11f0 0x7f93341eed20 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f932800c3f0 tx=0x7f932800c8b0 comp rx=0 tx=0).stop 2024-02-21T23:05:21.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.173+0000 7f933be00640 1 -- 172.21.15.28:0/3136236421 shutdown_connections 2024-02-21T23:05:21.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.173+0000 7f933be00640 1 --2- 172.21.15.28:0/3136236421 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f931406b4d0 0x7f931406d990 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:21.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.173+0000 7f933be00640 1 --2- 172.21.15.28:0/3136236421 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9334153950 0x7f93340a7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:21.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.173+0000 7f933be00640 1 --2- 172.21.15.28:0/3136236421 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f933415b6d0 0x7f93341eada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:21.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.173+0000 7f933be00640 1 --2- 172.21.15.28:0/3136236421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93340c11f0 0x7f93341eed20 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:21.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.173+0000 7f933be00640 1 -- 172.21.15.28:0/3136236421 >> 172.21.15.28:0/3136236421 conn(0x7f93340c67f0 msgr2=0x7f93340c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:21.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.173+0000 7f933be00640 1 -- 172.21.15.28:0/3136236421 shutdown_connections 2024-02-21T23:05:21.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.173+0000 7f933be00640 1 -- 172.21.15.28:0/3136236421 wait complete. 2024-02-21T23:05:21.180 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-02-21T23:05:21.181 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-02-21T23:05:21.181 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-21T23:05:21.295 DEBUG:teuthology.orchestra.run.smithi081:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-21T23:05:21.410 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-02-21T23:05:21.410 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-21T23:05:21.410 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-21T23:05:21.473 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054608,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-02-21T23:05:21.473 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-02-21T23:05:21.474 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-21T23:05:21.530 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053833,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498812k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249404k,nr_inodes=812351,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498812k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-02-21T23:05:21.531 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:05:21.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.790+0000 7f25c7200640 1 Processor -- start 2024-02-21T23:05:21.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.790+0000 7f25c7200640 1 -- start start 2024-02-21T23:05:21.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.793+0000 7f25c7200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f25c013e620 0x7f25c0159070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:21.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.793+0000 7f25c7200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f25c00c6460 0x7f25c00ca8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:21.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.793+0000 7f25c7200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f25c00cadf0 0x7f25c00bf990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:21.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.793+0000 7f25c7200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25c00c71d0 con 0x7f25c00cadf0 2024-02-21T23:05:21.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.793+0000 7f25c7200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25c01551b0 con 0x7f25c00c6460 2024-02-21T23:05:21.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.793+0000 7f25c7200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25c01595b0 con 0x7f25c013e620 2024-02-21T23:05:21.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.793+0000 7f25c5200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f25c00cadf0 0x7f25c00bf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:21.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.794+0000 7f25c5200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f25c00cadf0 0x7f25c00bf990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41410/0 (socket says 172.21.15.28:41410) 2024-02-21T23:05:21.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.794+0000 7f25c5200640 1 -- 172.21.15.28:0/1044274775 learned_addr learned my addr 172.21.15.28:0/1044274775 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:21.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.794+0000 7f25bfe00640 1 --2- 172.21.15.28:0/1044274775 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f25c013e620 0x7f25c0159070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:21.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.794+0000 7f25bf400640 1 --2- 172.21.15.28:0/1044274775 >> v2:172.21.15.81:3300/0 conn(0x7f25c00c6460 0x7f25c00ca8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:21.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.794+0000 7f25c5200640 1 -- 172.21.15.28:0/1044274775 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f25c013e620 msgr2=0x7f25c0159070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:21.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.794+0000 7f25c5200640 1 --2- 172.21.15.28:0/1044274775 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f25c013e620 0x7f25c0159070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:21.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.794+0000 7f25c5200640 1 -- 172.21.15.28:0/1044274775 >> v2:172.21.15.81:3300/0 conn(0x7f25c00c6460 msgr2=0x7f25c00ca8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:21.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.794+0000 7f25c5200640 1 --2- 172.21.15.28:0/1044274775 >> v2:172.21.15.81:3300/0 conn(0x7f25c00c6460 0x7f25c00ca8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:21.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.794+0000 7f25c5200640 1 -- 172.21.15.28:0/1044274775 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25c0155420 con 0x7f25c00cadf0 2024-02-21T23:05:21.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.794+0000 7f25bf400640 1 --2- 172.21.15.28:0/1044274775 >> v2:172.21.15.81:3300/0 conn(0x7f25c00c6460 0x7f25c00ca8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:05:21.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.795+0000 7f25c5200640 1 --2- 172.21.15.28:0/1044274775 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25c00cadf0 0x7f25c00bf990 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f25c008c3d0 tx=0x7f25b802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fa6e3e70cdec022 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:21.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.796+0000 7f25bea00640 1 -- 172.21.15.28:0/1044274775 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b802c9e0 con 0x7f25c00cadf0 2024-02-21T23:05:21.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.796+0000 7f25bea00640 1 -- 172.21.15.28:0/1044274775 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25b802cb80 con 0x7f25c00cadf0 2024-02-21T23:05:21.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.796+0000 7f25bea00640 1 -- 172.21.15.28:0/1044274775 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b80343f0 con 0x7f25c00cadf0 2024-02-21T23:05:21.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.796+0000 7f25c7200640 1 -- 172.21.15.28:0/1044274775 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25c00cadf0 msgr2=0x7f25c00bf990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:21.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.796+0000 7f25c7200640 1 --2- 172.21.15.28:0/1044274775 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25c00cadf0 0x7f25c00bf990 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f25c008c3d0 tx=0x7f25b802be40 comp rx=0 tx=0).stop 2024-02-21T23:05:21.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.796+0000 7f25c7200640 1 -- 172.21.15.28:0/1044274775 shutdown_connections 2024-02-21T23:05:21.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.796+0000 7f25c7200640 1 --2- 172.21.15.28:0/1044274775 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25c00cadf0 0x7f25c00bf990 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:21.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.796+0000 7f25c7200640 1 --2- 172.21.15.28:0/1044274775 >> v2:172.21.15.81:3300/0 conn(0x7f25c00c6460 0x7f25c00ca8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:21.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.796+0000 7f25c7200640 1 --2- 172.21.15.28:0/1044274775 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f25c013e620 0x7f25c0159070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:21.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.796+0000 7f25c7200640 1 -- 172.21.15.28:0/1044274775 >> 172.21.15.28:0/1044274775 conn(0x7f25c008b1c0 msgr2=0x7f25c00725a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:21.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.796+0000 7f25c7200640 1 -- 172.21.15.28:0/1044274775 shutdown_connections 2024-02-21T23:05:21.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.797+0000 7f25c7200640 1 -- 172.21.15.28:0/1044274775 wait complete. 2024-02-21T23:05:21.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.798+0000 7f25c7200640 1 Processor -- start 2024-02-21T23:05:21.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.798+0000 7f25c7200640 1 -- start start 2024-02-21T23:05:21.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.799+0000 7f25c7200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f25c00c6460 0x7f25c00c4d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:21.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.799+0000 7f25c7200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f25c00cadf0 0x7f25c00ca1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:21.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.799+0000 7f25c7200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25c013e620 0x7f25c01eb080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:21.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.799+0000 7f25c7200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25c0155cc0 con 0x7f25c013e620 2024-02-21T23:05:21.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.799+0000 7f25c7200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25c01559e0 con 0x7f25c00c6460 2024-02-21T23:05:21.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.799+0000 7f25c7200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25c013ea00 con 0x7f25c00cadf0 2024-02-21T23:05:21.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.799+0000 7f25c5200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25c013e620 0x7f25c01eb080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:21.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.800+0000 7f25c5200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25c013e620 0x7f25c01eb080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41422/0 (socket says 172.21.15.28:41422) 2024-02-21T23:05:21.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.800+0000 7f25c5200640 1 -- 172.21.15.28:0/1955133810 learned_addr learned my addr 172.21.15.28:0/1955133810 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:21.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.800+0000 7f25bfe00640 1 --2- 172.21.15.28:0/1955133810 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f25c00c6460 0x7f25c00c4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:21.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.800+0000 7f25bf400640 1 --2- 172.21.15.28:0/1955133810 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f25c00cadf0 0x7f25c00ca1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:21.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.800+0000 7f25c5200640 1 -- 172.21.15.28:0/1955133810 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f25c00cadf0 msgr2=0x7f25c00ca1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:21.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.800+0000 7f25c5200640 1 --2- 172.21.15.28:0/1955133810 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f25c00cadf0 0x7f25c00ca1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:21.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.800+0000 7f25c5200640 1 -- 172.21.15.28:0/1955133810 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f25c00c6460 msgr2=0x7f25c00c4d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:21.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.800+0000 7f25c5200640 1 --2- 172.21.15.28:0/1955133810 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f25c00c6460 0x7f25c00c4d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:21.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.800+0000 7f25c5200640 1 -- 172.21.15.28:0/1955133810 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25c0156970 con 0x7f25c013e620 2024-02-21T23:05:21.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.800+0000 7f25bf400640 1 --2- 172.21.15.28:0/1955133810 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f25c00cadf0 0x7f25c00ca1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:05:21.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.801+0000 7f25c5200640 1 --2- 172.21.15.28:0/1955133810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25c013e620 0x7f25c01eb080 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f25b8036040 tx=0x7f25b8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:21.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.801+0000 7f25b7e00640 1 -- 172.21.15.28:0/1955133810 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b8043020 con 0x7f25c013e620 2024-02-21T23:05:21.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.801+0000 7f25b7e00640 1 -- 172.21.15.28:0/1955133810 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25b803e450 con 0x7f25c013e620 2024-02-21T23:05:21.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.801+0000 7f25b7e00640 1 -- 172.21.15.28:0/1955133810 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b8009510 con 0x7f25c013e620 2024-02-21T23:05:21.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.801+0000 7f25c7200640 1 -- 172.21.15.28:0/1955133810 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25c01eecb0 con 0x7f25c013e620 2024-02-21T23:05:21.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.801+0000 7f25c7200640 1 -- 172.21.15.28:0/1955133810 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25c01eab60 con 0x7f25c013e620 2024-02-21T23:05:21.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.802+0000 7f25c7200640 1 -- 172.21.15.28:0/1955133810 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25c00c2850 con 0x7f25c013e620 2024-02-21T23:05:21.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.803+0000 7f25b7e00640 1 -- 172.21.15.28:0/1955133810 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f25b8046020 con 0x7f25c013e620 2024-02-21T23:05:21.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.804+0000 7f25b7e00640 1 --2- 172.21.15.28:0/1955133810 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f25a006b4d0 0x7f25a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:21.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.804+0000 7f25b7e00640 1 -- 172.21.15.28:0/1955133810 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7129+0+0 (secure 0 0 0) 0x7f25b80ad160 con 0x7f25c013e620 2024-02-21T23:05:21.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.805+0000 7f25bfe00640 1 --2- 172.21.15.28:0/1955133810 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f25a006b4d0 0x7f25a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:21.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.806+0000 7f25bfe00640 1 --2- 172.21.15.28:0/1955133810 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f25a006b4d0 0x7f25a006d990 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f25c01df0c0 tx=0x7f25b0009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:21.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:21.809+0000 7f25b7e00640 1 -- 172.21.15.28:0/1955133810 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f25c00c2850 con 0x7f25c013e620 2024-02-21T23:05:22.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.047+0000 7f25c7200640 1 -- 172.21.15.28:0/1955133810 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f25c01ead60 con 0x7f25c013e620 2024-02-21T23:05:22.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.048+0000 7f25b7e00640 1 -- 172.21.15.28:0/1955133810 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+6176 (secure 0 0 0) 0x7f25c01ead60 con 0x7f25c013e620 2024-02-21T23:05:22.037 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:05:22.037 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":47,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5221,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":7,"addr":"172.21.15.81:6835/2152160500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2152160500},{"type":"v1","addr":"172.21.15.81:6835","nonce":2152160500}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":6039,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3807390545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3807390545},{"type":"v1","addr":"172.21.15.28:6835","nonce":3807390545}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:04:53.432188+0000","modified":"2024-02-21T23:05:19.431400+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6021},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6021":{"gid":6021,"name":"a","rank":0,"incarnation":44,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6834/3835332438","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3835332438},{"type":"v1","addr":"172.21.15.28:6834","nonce":3835332438}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3},{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:05:08.017586+0000","modified":"2024-02-21T23:05:10.974691+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5224},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5224":{"gid":5224,"name":"d","rank":0,"incarnation":33,"state":"up:active","state_seq":7,"addr":"172.21.15.81:6837/442767245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":442767245},{"type":"v1","addr":"172.21.15.81:6837","nonce":442767245}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-02-21T23:05:22.038 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 47 2024-02-21T23:05:22.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.051+0000 7f25c7200640 1 -- 172.21.15.28:0/1955133810 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f25a006b4d0 msgr2=0x7f25a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:22.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.051+0000 7f25c7200640 1 --2- 172.21.15.28:0/1955133810 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f25a006b4d0 0x7f25a006d990 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f25c01df0c0 tx=0x7f25b0009190 comp rx=0 tx=0).stop 2024-02-21T23:05:22.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.051+0000 7f25c7200640 1 -- 172.21.15.28:0/1955133810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25c013e620 msgr2=0x7f25c01eb080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:22.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.051+0000 7f25c7200640 1 --2- 172.21.15.28:0/1955133810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25c013e620 0x7f25c01eb080 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f25b8036040 tx=0x7f25b8004a40 comp rx=0 tx=0).stop 2024-02-21T23:05:22.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.051+0000 7f25c7200640 1 -- 172.21.15.28:0/1955133810 shutdown_connections 2024-02-21T23:05:22.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.051+0000 7f25c7200640 1 --2- 172.21.15.28:0/1955133810 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f25a006b4d0 0x7f25a006d990 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:22.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.051+0000 7f25c7200640 1 --2- 172.21.15.28:0/1955133810 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f25c00cadf0 0x7f25c00ca1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:22.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.051+0000 7f25c7200640 1 --2- 172.21.15.28:0/1955133810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25c013e620 0x7f25c01eb080 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:22.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.051+0000 7f25c7200640 1 --2- 172.21.15.28:0/1955133810 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f25c00c6460 0x7f25c00c4d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:22.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.051+0000 7f25c7200640 1 -- 172.21.15.28:0/1955133810 >> 172.21.15.28:0/1955133810 conn(0x7f25c008b1c0 msgr2=0x7f25c013d5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:22.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.051+0000 7f25c7200640 1 -- 172.21.15.28:0/1955133810 shutdown_connections 2024-02-21T23:05:22.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.051+0000 7f25c7200640 1 -- 172.21.15.28:0/1955133810 wait complete. 2024-02-21T23:05:22.058 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:05:22.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.281+0000 7f2974200640 1 Processor -- start 2024-02-21T23:05:22.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.281+0000 7f2974200640 1 -- start start 2024-02-21T23:05:22.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.283+0000 7f2974200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f296c155ca0 0x7f296c06cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:22.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.283+0000 7f2974200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f296c0c5e60 con 0x7f296c0c4ce0 2024-02-21T23:05:22.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.283+0000 7f2974200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f296c0c6140 con 0x7f296c0af970 2024-02-21T23:05:22.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.283+0000 7f2974200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f296c1561a0 con 0x7f296c155ca0 2024-02-21T23:05:22.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.283+0000 7f2972200640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f296c0c4ce0 0x7f296c06d3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39258/0 (socket says 172.21.15.28:39258) 2024-02-21T23:05:22.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.283+0000 7f2972200640 1 -- 172.21.15.28:0/3997780275 learned_addr learned my addr 172.21.15.28:0/3997780275 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:05:22.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.284+0000 7f2970e00640 1 --2- 172.21.15.28:0/3997780275 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f296c155ca0 0x7f296c06cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:22.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.284+0000 7f296be00640 1 -- 172.21.15.28:0/3997780275 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3672211403 0 0) 0x7f296c0c5e60 con 0x7f296c0c4ce0 2024-02-21T23:05:22.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.284+0000 7f296be00640 1 -- 172.21.15.28:0/3997780275 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2954003520 con 0x7f296c0c4ce0 2024-02-21T23:05:22.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.284+0000 7f296be00640 1 -- 172.21.15.28:0/3997780275 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3989746108 0 0) 0x7f296c0c6140 con 0x7f296c0af970 2024-02-21T23:05:22.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.284+0000 7f296be00640 1 -- 172.21.15.28:0/3997780275 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f296c0c5e60 con 0x7f296c0af970 2024-02-21T23:05:22.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.284+0000 7f296be00640 1 -- 172.21.15.28:0/3997780275 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3748368497 0 0) 0x7f2954003520 con 0x7f296c0c4ce0 2024-02-21T23:05:22.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.284+0000 7f296be00640 1 -- 172.21.15.28:0/3997780275 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f296c155ca0 msgr2=0x7f296c06cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:22.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.284+0000 7f296be00640 1 --2- 172.21.15.28:0/3997780275 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f296c155ca0 0x7f296c06cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:22.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.284+0000 7f296be00640 1 -- 172.21.15.28:0/3997780275 >> v1:172.21.15.81:6789/0 conn(0x7f296c0af970 legacy=0x7f296c0c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:22.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.284+0000 7f296be00640 1 -- 172.21.15.28:0/3997780275 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f296c0c6420 con 0x7f296c0c4ce0 2024-02-21T23:05:22.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.284+0000 7f296be00640 1 -- 172.21.15.28:0/3997780275 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f29640019f0 con 0x7f296c0c4ce0 2024-02-21T23:05:22.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.284+0000 7f296be00640 1 -- 172.21.15.28:0/3997780275 >> v1:172.21.15.28:6789/0 conn(0x7f296c0c4ce0 legacy=0x7f296c06d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:22.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.284+0000 7f296be00640 1 --2- 172.21.15.28:0/3997780275 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f296c155ca0 0x7f2954003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:22.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.284+0000 7f296be00640 1 --2- 172.21.15.28:0/3997780275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29540040d0 0x7f2954004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:22.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.285+0000 7f296be00640 1 --2- 172.21.15.28:0/3997780275 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2954004d40 0x7f29540051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:22.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.285+0000 7f296be00640 1 -- 172.21.15.28:0/3997780275 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f296c1561a0 con 0x7f29540040d0 2024-02-21T23:05:22.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.285+0000 7f296be00640 1 -- 172.21.15.28:0/3997780275 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f295400bb50 con 0x7f2954004d40 2024-02-21T23:05:22.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.285+0000 7f296be00640 1 -- 172.21.15.28:0/3997780275 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f295400bd20 con 0x7f296c155ca0 2024-02-21T23:05:22.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.285+0000 7f2970e00640 1 --2- 172.21.15.28:0/3997780275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29540040d0 0x7f2954004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:22.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.285+0000 7f2972200640 1 --2- 172.21.15.28:0/3997780275 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2954004d40 0x7f29540051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:22.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.285+0000 7f2971800640 1 --2- 172.21.15.28:0/3997780275 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f296c155ca0 0x7f2954003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:22.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.285+0000 7f2970e00640 1 -- 172.21.15.28:0/3997780275 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f296c155ca0 msgr2=0x7f2954003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:22.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.285+0000 7f2970e00640 1 --2- 172.21.15.28:0/3997780275 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f296c155ca0 0x7f2954003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:22.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.285+0000 7f2970e00640 1 -- 172.21.15.28:0/3997780275 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2954004d40 msgr2=0x7f29540051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:22.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.285+0000 7f2970e00640 1 --2- 172.21.15.28:0/3997780275 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2954004d40 0x7f29540051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:22.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.285+0000 7f2970e00640 1 -- 172.21.15.28:0/3997780275 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2954003520 con 0x7f29540040d0 2024-02-21T23:05:22.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.287+0000 7f2970e00640 1 --2- 172.21.15.28:0/3997780275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29540040d0 0x7f2954004800 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f295c009ac0 tx=0x7f2954005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dba32b5e7c5f5699 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:22.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.287+0000 7f296be00640 1 -- 172.21.15.28:0/3997780275 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f295c039070 con 0x7f29540040d0 2024-02-21T23:05:22.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.287+0000 7f296be00640 1 -- 172.21.15.28:0/3997780275 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f295c02c6b0 con 0x7f29540040d0 2024-02-21T23:05:22.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.287+0000 7f296be00640 1 -- 172.21.15.28:0/3997780275 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f295c009690 con 0x7f29540040d0 2024-02-21T23:05:22.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.287+0000 7f2974200640 1 -- 172.21.15.28:0/3997780275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29540040d0 msgr2=0x7f2954004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:22.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.287+0000 7f2974200640 1 --2- 172.21.15.28:0/3997780275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29540040d0 0x7f2954004800 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f295c009ac0 tx=0x7f2954005830 comp rx=0 tx=0).stop 2024-02-21T23:05:22.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.287+0000 7f2972200640 1 -- 172.21.15.28:0/3997780275 reap_dead start 2024-02-21T23:05:22.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.287+0000 7f2974200640 1 -- 172.21.15.28:0/3997780275 shutdown_connections 2024-02-21T23:05:22.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.288+0000 7f2974200640 1 -- 172.21.15.28:0/3997780275 >> 172.21.15.28:0/3997780275 conn(0x7f296c0725a0 msgr2=0x7f296c0729b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:22.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.288+0000 7f2974200640 1 -- 172.21.15.28:0/3997780275 shutdown_connections 2024-02-21T23:05:22.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.288+0000 7f2974200640 1 -- 172.21.15.28:0/3997780275 wait complete. 2024-02-21T23:05:22.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.289+0000 7f2974200640 1 Processor -- start 2024-02-21T23:05:22.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.289+0000 7f2974200640 1 -- start start 2024-02-21T23:05:22.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.290+0000 7f2974200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29540040d0 0x7f296c156950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:22.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.290+0000 7f2974200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f296c158d80 0x7f296c157940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:22.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.290+0000 7f2974200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f296c154480 0x7f296c1ebe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:22.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.290+0000 7f2974200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f296c0c7a60 con 0x7f296c158d80 2024-02-21T23:05:22.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.290+0000 7f2974200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f296c06daf0 con 0x7f29540040d0 2024-02-21T23:05:22.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.290+0000 7f2974200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f296c0c7be0 con 0x7f296c154480 2024-02-21T23:05:22.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.291+0000 7f2970e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f296c158d80 0x7f296c157940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:22.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.291+0000 7f2970e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f296c158d80 0x7f296c157940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41438/0 (socket says 172.21.15.28:41438) 2024-02-21T23:05:22.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.291+0000 7f2970e00640 1 -- 172.21.15.28:0/3627228040 learned_addr learned my addr 172.21.15.28:0/3627228040 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:22.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.291+0000 7f2971800640 1 --2- 172.21.15.28:0/3627228040 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29540040d0 0x7f296c156950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:22.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.291+0000 7f2972200640 1 --2- 172.21.15.28:0/3627228040 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f296c154480 0x7f296c1ebe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:22.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.291+0000 7f2970e00640 1 -- 172.21.15.28:0/3627228040 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f296c154480 msgr2=0x7f296c1ebe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:22.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.291+0000 7f2970e00640 1 --2- 172.21.15.28:0/3627228040 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f296c154480 0x7f296c1ebe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:22.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.291+0000 7f2970e00640 1 -- 172.21.15.28:0/3627228040 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29540040d0 msgr2=0x7f296c156950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:22.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.291+0000 7f2970e00640 1 --2- 172.21.15.28:0/3627228040 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29540040d0 0x7f296c156950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:22.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.291+0000 7f2970e00640 1 -- 172.21.15.28:0/3627228040 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f296c157e80 con 0x7f296c158d80 2024-02-21T23:05:22.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.291+0000 7f2972200640 1 --2- 172.21.15.28:0/3627228040 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f296c154480 0x7f296c1ebe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:05:22.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.291+0000 7f2971800640 1 --2- 172.21.15.28:0/3627228040 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29540040d0 0x7f296c156950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:05:22.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.292+0000 7f2970e00640 1 --2- 172.21.15.28:0/3627228040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f296c158d80 0x7f296c157940 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f295c000c00 tx=0x7f295c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:22.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.292+0000 7f2969600640 1 -- 172.21.15.28:0/3627228040 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f295c042020 con 0x7f296c158d80 2024-02-21T23:05:22.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.292+0000 7f2969600640 1 -- 172.21.15.28:0/3627228040 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f295c03a040 con 0x7f296c158d80 2024-02-21T23:05:22.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.292+0000 7f2969600640 1 -- 172.21.15.28:0/3627228040 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f295c039070 con 0x7f296c158d80 2024-02-21T23:05:22.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.292+0000 7f2974200640 1 -- 172.21.15.28:0/3627228040 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f296c14e200 con 0x7f296c158d80 2024-02-21T23:05:22.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.292+0000 7f2974200640 1 -- 172.21.15.28:0/3627228040 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f296c155150 con 0x7f296c158d80 2024-02-21T23:05:22.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.293+0000 7f2974200640 1 -- 172.21.15.28:0/3627228040 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f296c155350 con 0x7f296c158d80 2024-02-21T23:05:22.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.294+0000 7f2969600640 1 -- 172.21.15.28:0/3627228040 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f295c045020 con 0x7f296c158d80 2024-02-21T23:05:22.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.296+0000 7f2969600640 1 --2- 172.21.15.28:0/3627228040 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f295006b4d0 0x7f295006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:22.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.296+0000 7f2969600640 1 -- 172.21.15.28:0/3627228040 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7129+0+0 (secure 0 0 0) 0x7f295c0abad0 con 0x7f296c158d80 2024-02-21T23:05:22.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.296+0000 7f2971800640 1 --2- 172.21.15.28:0/3627228040 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f295006b4d0 0x7f295006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:22.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.297+0000 7f2971800640 1 --2- 172.21.15.28:0/3627228040 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f295006b4d0 0x7f295006d990 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f2960007ae0 tx=0x7f2960009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:22.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.300+0000 7f2969600640 1 -- 172.21.15.28:0/3627228040 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f295c04c070 con 0x7f296c158d80 2024-02-21T23:05:22.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.504+0000 7f2974200640 1 -- 172.21.15.28:0/3627228040 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f296c068640 con 0x7f296c158d80 2024-02-21T23:05:22.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.505+0000 7f2969600640 1 -- 172.21.15.28:0/3627228040 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v59) v1 ==== 74+0+20865 (secure 0 0 0) 0x7f295c07a430 con 0x7f296c158d80 2024-02-21T23:05:22.494 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:05:22.495 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":59,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:05:17.415270+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:04:51.354130+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-02-21T23:04:52.441690+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs2_metadata","create_time":"2024-02-21T23:05:05.701276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs2_data","create_time":"2024-02-21T23:05:06.904887+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6835/3468780363":"2024-02-22T23:05:17.415180+0000","172.21.15.28:6833/3468780363":"2024-02-22T23:05:17.415180+0000","172.21.15.28:6834/2803823391":"2024-02-22T23:05:14.385354+0000","172.21.15.28:6832/2803823391":"2024-02-22T23:05:14.385354+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:05:22.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.508+0000 7f2974200640 1 -- 172.21.15.28:0/3627228040 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f295006b4d0 msgr2=0x7f295006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:22.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.508+0000 7f2974200640 1 --2- 172.21.15.28:0/3627228040 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f295006b4d0 0x7f295006d990 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f2960007ae0 tx=0x7f2960009040 comp rx=0 tx=0).stop 2024-02-21T23:05:22.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.508+0000 7f2974200640 1 -- 172.21.15.28:0/3627228040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f296c158d80 msgr2=0x7f296c157940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:22.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.508+0000 7f2974200640 1 --2- 172.21.15.28:0/3627228040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f296c158d80 0x7f296c157940 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f295c000c00 tx=0x7f295c004730 comp rx=0 tx=0).stop 2024-02-21T23:05:22.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.508+0000 7f2974200640 1 -- 172.21.15.28:0/3627228040 shutdown_connections 2024-02-21T23:05:22.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.508+0000 7f2974200640 1 --2- 172.21.15.28:0/3627228040 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f295006b4d0 0x7f295006d990 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:22.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.508+0000 7f2974200640 1 --2- 172.21.15.28:0/3627228040 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f296c154480 0x7f296c1ebe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:22.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.508+0000 7f2974200640 1 --2- 172.21.15.28:0/3627228040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f296c158d80 0x7f296c157940 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:22.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.508+0000 7f2974200640 1 --2- 172.21.15.28:0/3627228040 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29540040d0 0x7f296c156950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:22.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.508+0000 7f2974200640 1 -- 172.21.15.28:0/3627228040 >> 172.21.15.28:0/3627228040 conn(0x7f296c0725a0 msgr2=0x7f296c0c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:22.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.508+0000 7f2974200640 1 -- 172.21.15.28:0/3627228040 shutdown_connections 2024-02-21T23:05:22.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.509+0000 7f2974200640 1 -- 172.21.15.28:0/3627228040 wait complete. 2024-02-21T23:05:22.516 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-21T23:02:30.728329+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-21T23:04:51.354130+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '52', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2024-02-21T23:04:52.441690+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '53', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs2_metadata', 'create_time': '2024-02-21T23:05:05.701276+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '56', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs2'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs2_data', 'create_time': '2024-02-21T23:05:06.904887+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '57', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs2'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-21T23:05:22.517 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-21T23:05:22.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.747+0000 7f8284400640 1 Processor -- start 2024-02-21T23:05:22.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.747+0000 7f8284400640 1 -- start start 2024-02-21T23:05:22.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.749+0000 7f8284400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f827c0c6220 0x7f827c1586b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:22.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.749+0000 7f8284400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f827c0c20d0 con 0x7f827c158bf0 2024-02-21T23:05:22.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.749+0000 7f8284400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f827c0c1290 con 0x7f827c0c4960 2024-02-21T23:05:22.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.749+0000 7f8284400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f827c0a62e0 con 0x7f827c0c6220 2024-02-21T23:05:22.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.750+0000 7f8282400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f827c158bf0 0x7f827c15f9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39264/0 (socket says 172.21.15.28:39264) 2024-02-21T23:05:22.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.750+0000 7f8281a00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f827c0c4960 0x7f827c1524f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.28:41486/0 (socket says 172.21.15.28:41486) 2024-02-21T23:05:22.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.750+0000 7f8282400640 1 -- 172.21.15.28:0/382856027 learned_addr learned my addr 172.21.15.28:0/382856027 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:05:22.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.750+0000 7f8281000640 1 --2- 172.21.15.28:0/382856027 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f827c0c6220 0x7f827c1586b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:22.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.750+0000 7f827be00640 1 -- 172.21.15.28:0/382856027 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4201488974 0 0) 0x7f827c0c20d0 con 0x7f827c158bf0 2024-02-21T23:05:22.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.750+0000 7f827be00640 1 -- 172.21.15.28:0/382856027 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8264003520 con 0x7f827c158bf0 2024-02-21T23:05:22.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.750+0000 7f827be00640 1 -- 172.21.15.28:0/382856027 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3157392961 0 0) 0x7f827c0c1290 con 0x7f827c0c4960 2024-02-21T23:05:22.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.750+0000 7f827be00640 1 -- 172.21.15.28:0/382856027 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f827c0c20d0 con 0x7f827c0c4960 2024-02-21T23:05:22.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.750+0000 7f8281000640 1 -- 172.21.15.28:0/382856027 >> v1:172.21.15.81:6789/0 conn(0x7f827c0c4960 legacy=0x7f827c1524f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:22.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.751+0000 7f8281000640 1 -- 172.21.15.28:0/382856027 >> v1:172.21.15.28:6789/0 conn(0x7f827c158bf0 legacy=0x7f827c15f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:22.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.751+0000 7f8281000640 1 -- 172.21.15.28:0/382856027 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f827c0c1570 con 0x7f827c0c6220 2024-02-21T23:05:22.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.752+0000 7f8281000640 1 --2- 172.21.15.28:0/382856027 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f827c0c6220 0x7f827c1586b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f82680097d0 tx=0x7f826802b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=338266e41e59220a server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:22.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.752+0000 7f827be00640 1 -- 172.21.15.28:0/382856027 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8268035020 con 0x7f827c0c6220 2024-02-21T23:05:22.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.753+0000 7f827be00640 1 -- 172.21.15.28:0/382856027 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8268031810 con 0x7f827c0c6220 2024-02-21T23:05:22.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.753+0000 7f827be00640 1 -- 172.21.15.28:0/382856027 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8268031bc0 con 0x7f827c0c6220 2024-02-21T23:05:22.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.753+0000 7f8284400640 1 -- 172.21.15.28:0/382856027 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f827c0c6220 msgr2=0x7f827c1586b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:22.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.753+0000 7f8284400640 1 --2- 172.21.15.28:0/382856027 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f827c0c6220 0x7f827c1586b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f82680097d0 tx=0x7f826802b8b0 comp rx=0 tx=0).stop 2024-02-21T23:05:22.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.753+0000 7f8284400640 1 -- 172.21.15.28:0/382856027 shutdown_connections 2024-02-21T23:05:22.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.753+0000 7f8284400640 1 --2- 172.21.15.28:0/382856027 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f827c0c6220 0x7f827c1586b0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:22.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.753+0000 7f8284400640 1 -- 172.21.15.28:0/382856027 >> 172.21.15.28:0/382856027 conn(0x7f827c0a85a0 msgr2=0x7f827c0a8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:22.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.753+0000 7f8284400640 1 -- 172.21.15.28:0/382856027 shutdown_connections 2024-02-21T23:05:22.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.753+0000 7f8284400640 1 -- 172.21.15.28:0/382856027 wait complete. 2024-02-21T23:05:22.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.755+0000 7f8284400640 1 Processor -- start 2024-02-21T23:05:22.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.755+0000 7f8284400640 1 -- start start 2024-02-21T23:05:22.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.756+0000 7f8284400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f827c0c4960 0x7f827c1ee120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:22.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.756+0000 7f8284400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f827c0c6220 0x7f827c0ab070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:22.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.756+0000 7f8284400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f827c158bf0 0x7f827c072c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:22.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.756+0000 7f8284400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f827c0a5f20 con 0x7f827c0c4960 2024-02-21T23:05:22.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.756+0000 7f8284400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f827c0c1960 con 0x7f827c158bf0 2024-02-21T23:05:22.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.756+0000 7f8284400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f827c0a60a0 con 0x7f827c0c6220 2024-02-21T23:05:22.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.757+0000 7f8281a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f827c0c4960 0x7f827c1ee120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:22.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.757+0000 7f8281a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f827c0c4960 0x7f827c1ee120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41454/0 (socket says 172.21.15.28:41454) 2024-02-21T23:05:22.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.757+0000 7f8281a00640 1 -- 172.21.15.28:0/2788768729 learned_addr learned my addr 172.21.15.28:0/2788768729 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:22.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.757+0000 7f8282400640 1 --2- 172.21.15.28:0/2788768729 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f827c158bf0 0x7f827c072c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:22.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.757+0000 7f8281000640 1 --2- 172.21.15.28:0/2788768729 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f827c0c6220 0x7f827c0ab070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:22.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.757+0000 7f8281a00640 1 -- 172.21.15.28:0/2788768729 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f827c0c6220 msgr2=0x7f827c0ab070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:22.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.757+0000 7f8281a00640 1 --2- 172.21.15.28:0/2788768729 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f827c0c6220 0x7f827c0ab070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:22.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.757+0000 7f8281a00640 1 -- 172.21.15.28:0/2788768729 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f827c158bf0 msgr2=0x7f827c072c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:22.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.757+0000 7f8281a00640 1 --2- 172.21.15.28:0/2788768729 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f827c158bf0 0x7f827c072c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:22.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.757+0000 7f8281a00640 1 -- 172.21.15.28:0/2788768729 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f827c0ab5b0 con 0x7f827c0c4960 2024-02-21T23:05:22.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.757+0000 7f8282400640 1 --2- 172.21.15.28:0/2788768729 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f827c158bf0 0x7f827c072c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:05:22.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.758+0000 7f8281a00640 1 --2- 172.21.15.28:0/2788768729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f827c0c4960 0x7f827c1ee120 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f827000c500 tx=0x7f827000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:22.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.758+0000 7f8279600640 1 -- 172.21.15.28:0/2788768729 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8270018070 con 0x7f827c0c4960 2024-02-21T23:05:22.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.758+0000 7f8279600640 1 -- 172.21.15.28:0/2788768729 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8270007500 con 0x7f827c0c4960 2024-02-21T23:05:22.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.758+0000 7f8279600640 1 -- 172.21.15.28:0/2788768729 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8270004a20 con 0x7f827c0c4960 2024-02-21T23:05:22.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.758+0000 7f8284400640 1 -- 172.21.15.28:0/2788768729 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f827c158200 con 0x7f827c0c4960 2024-02-21T23:05:22.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.758+0000 7f8284400640 1 -- 172.21.15.28:0/2788768729 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f827c071310 con 0x7f827c0c4960 2024-02-21T23:05:22.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.759+0000 7f8284400640 1 -- 172.21.15.28:0/2788768729 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f827c0abaa0 con 0x7f827c0c4960 2024-02-21T23:05:22.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.760+0000 7f8279600640 1 -- 172.21.15.28:0/2788768729 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f827001b070 con 0x7f827c0c4960 2024-02-21T23:05:22.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.762+0000 7f8279600640 1 --2- 172.21.15.28:0/2788768729 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f825c06b4d0 0x7f825c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:22.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.762+0000 7f8279600640 1 -- 172.21.15.28:0/2788768729 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7129+0+0 (secure 0 0 0) 0x7f8270089a50 con 0x7f827c0c4960 2024-02-21T23:05:22.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.762+0000 7f8281000640 1 --2- 172.21.15.28:0/2788768729 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f825c06b4d0 0x7f825c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:22.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.764+0000 7f8281000640 1 --2- 172.21.15.28:0/2788768729 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f825c06b4d0 0x7f825c06d990 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f8268036000 tx=0x7f8268008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:22.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.767+0000 7f8279600640 1 -- 172.21.15.28:0/2788768729 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f827c0abaa0 con 0x7f827c0c4960 2024-02-21T23:05:22.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.965+0000 7f8284400640 1 -- 172.21.15.28:0/2788768729 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f827c0c43e0 con 0x7f827c0c4960 2024-02-21T23:05:22.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.965+0000 7f8279600640 1 -- 172.21.15.28:0/2788768729 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v59) v1 ==== 93+0+10 (secure 0 0 0) 0x7f827c0c43e0 con 0x7f827c0c4960 2024-02-21T23:05:22.954 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-02-21T23:05:22.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.968+0000 7f8284400640 1 -- 172.21.15.28:0/2788768729 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f825c06b4d0 msgr2=0x7f825c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:22.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.968+0000 7f8284400640 1 --2- 172.21.15.28:0/2788768729 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f825c06b4d0 0x7f825c06d990 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f8268036000 tx=0x7f8268008030 comp rx=0 tx=0).stop 2024-02-21T23:05:22.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.968+0000 7f8284400640 1 -- 172.21.15.28:0/2788768729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f827c0c4960 msgr2=0x7f827c1ee120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:22.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.968+0000 7f8284400640 1 --2- 172.21.15.28:0/2788768729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f827c0c4960 0x7f827c1ee120 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f827000c500 tx=0x7f827000c9c0 comp rx=0 tx=0).stop 2024-02-21T23:05:22.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.968+0000 7f8284400640 1 -- 172.21.15.28:0/2788768729 shutdown_connections 2024-02-21T23:05:22.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.968+0000 7f8284400640 1 --2- 172.21.15.28:0/2788768729 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f825c06b4d0 0x7f825c06d990 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:22.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.968+0000 7f8284400640 1 --2- 172.21.15.28:0/2788768729 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f827c0c6220 0x7f827c0ab070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:22.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.968+0000 7f8284400640 1 --2- 172.21.15.28:0/2788768729 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f827c158bf0 0x7f827c072c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:22.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.968+0000 7f8284400640 1 --2- 172.21.15.28:0/2788768729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f827c0c4960 0x7f827c1ee120 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:22.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.968+0000 7f8284400640 1 -- 172.21.15.28:0/2788768729 >> 172.21.15.28:0/2788768729 conn(0x7f827c0a85a0 msgr2=0x7f827c0a8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:22.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.968+0000 7f8284400640 1 -- 172.21.15.28:0/2788768729 shutdown_connections 2024-02-21T23:05:22.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:22.968+0000 7f8284400640 1 -- 172.21.15.28:0/2788768729 wait complete. 2024-02-21T23:05:22.976 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-21T23:05:23.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.204+0000 7f7fe6600640 1 Processor -- start 2024-02-21T23:05:23.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.204+0000 7f7fe6600640 1 -- start start 2024-02-21T23:05:23.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.206+0000 7f7fe6600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f7fe0072cb0 0x7f7fe014fea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:23.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.206+0000 7f7fe6600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fe00c2f60 con 0x7f7fe006c0f0 2024-02-21T23:05:23.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.206+0000 7f7fe6600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fe00c3200 con 0x7f7fe01503e0 2024-02-21T23:05:23.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.206+0000 7f7fe6600640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fe00a8bc0 con 0x7f7fe0072cb0 2024-02-21T23:05:23.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.206+0000 7f7fe5200640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f7fe006c0f0 0x7f7fe00a84b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39266/0 (socket says 172.21.15.28:39266) 2024-02-21T23:05:23.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.206+0000 7f7fe5200640 1 -- 172.21.15.28:0/969050879 learned_addr learned my addr 172.21.15.28:0/969050879 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:05:23.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.207+0000 7f7fdfe00640 1 --2- 172.21.15.28:0/969050879 >> v2:172.21.15.81:3300/0 conn(0x7f7fe0072cb0 0x7f7fe014fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:23.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.207+0000 7f7fdf400640 1 -- 172.21.15.28:0/969050879 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 577273348 0 0) 0x7f7fe00c2f60 con 0x7f7fe006c0f0 2024-02-21T23:05:23.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.207+0000 7f7fdf400640 1 -- 172.21.15.28:0/969050879 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fc8003520 con 0x7f7fe006c0f0 2024-02-21T23:05:23.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.207+0000 7f7fdfe00640 1 -- 172.21.15.28:0/969050879 >> v1:172.21.15.81:6789/0 conn(0x7f7fe01503e0 legacy=0x7f7fe00c1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:23.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.208+0000 7f7fdf400640 1 -- 172.21.15.28:0/969050879 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 562233091 0 0) 0x7f7fc8003520 con 0x7f7fe006c0f0 2024-02-21T23:05:23.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.208+0000 7f7fdfe00640 1 -- 172.21.15.28:0/969050879 >> v1:172.21.15.28:6789/0 conn(0x7f7fe006c0f0 legacy=0x7f7fe00a84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:23.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.208+0000 7f7fdfe00640 1 -- 172.21.15.28:0/969050879 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fe00c34e0 con 0x7f7fe0072cb0 2024-02-21T23:05:23.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.209+0000 7f7fdfe00640 1 --2- 172.21.15.28:0/969050879 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7fe0072cb0 0x7f7fe014fea0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7fcc0097a0 tx=0x7f7fcc02b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=219fc362f92a87c2 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:23.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.209+0000 7f7fdf400640 1 -- 172.21.15.28:0/969050879 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fcc037070 con 0x7f7fe0072cb0 2024-02-21T23:05:23.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.209+0000 7f7fdf400640 1 -- 172.21.15.28:0/969050879 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fcc030510 con 0x7f7fe0072cb0 2024-02-21T23:05:23.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.209+0000 7f7fdf400640 1 -- 172.21.15.28:0/969050879 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fd4001670 con 0x7f7fe0072cb0 2024-02-21T23:05:23.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.210+0000 7f7fe6600640 1 -- 172.21.15.28:0/969050879 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7fe0072cb0 msgr2=0x7f7fe014fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:23.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.210+0000 7f7fe6600640 1 --2- 172.21.15.28:0/969050879 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7fe0072cb0 0x7f7fe014fea0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7fcc0097a0 tx=0x7f7fcc02b790 comp rx=0 tx=0).stop 2024-02-21T23:05:23.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.210+0000 7f7fe6600640 1 -- 172.21.15.28:0/969050879 shutdown_connections 2024-02-21T23:05:23.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.210+0000 7f7fe6600640 1 --2- 172.21.15.28:0/969050879 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7fe0072cb0 0x7f7fe014fea0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:23.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.210+0000 7f7fe6600640 1 -- 172.21.15.28:0/969050879 >> 172.21.15.28:0/969050879 conn(0x7f7fe0071b20 msgr2=0x7f7fe0071f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:23.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.210+0000 7f7fe6600640 1 -- 172.21.15.28:0/969050879 shutdown_connections 2024-02-21T23:05:23.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.210+0000 7f7fe6600640 1 -- 172.21.15.28:0/969050879 wait complete. 2024-02-21T23:05:23.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.212+0000 7f7fe6600640 1 Processor -- start 2024-02-21T23:05:23.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.212+0000 7f7fe6600640 1 -- start start 2024-02-21T23:05:23.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.213+0000 7f7fe6600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7fe006c0f0 0x7f7fe01dd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:23.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.213+0000 7f7fe6600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7fe0072cb0 0x7f7fe01e8de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:23.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.213+0000 7f7fe6600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fe01503e0 0x7f7fe00c62b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:23.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.213+0000 7f7fe6600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe00a8fa0 con 0x7f7fe01503e0 2024-02-21T23:05:23.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.213+0000 7f7fe6600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe00a8e20 con 0x7f7fe006c0f0 2024-02-21T23:05:23.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.213+0000 7f7fe6600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe0150830 con 0x7f7fe0072cb0 2024-02-21T23:05:23.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.213+0000 7f7fe5c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fe01503e0 0x7f7fe00c62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:23.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.214+0000 7f7fdfe00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7fe0072cb0 0x7f7fe01e8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:23.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.214+0000 7f7fe5c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fe01503e0 0x7f7fe00c62b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41462/0 (socket says 172.21.15.28:41462) 2024-02-21T23:05:23.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.214+0000 7f7fe5c00640 1 -- 172.21.15.28:0/2397758320 learned_addr learned my addr 172.21.15.28:0/2397758320 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:23.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.214+0000 7f7fe5200640 1 --2- 172.21.15.28:0/2397758320 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7fe006c0f0 0x7f7fe01dd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:23.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.214+0000 7f7fe5c00640 1 -- 172.21.15.28:0/2397758320 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7fe0072cb0 msgr2=0x7f7fe01e8de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:23.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.214+0000 7f7fe5c00640 1 --2- 172.21.15.28:0/2397758320 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7fe0072cb0 0x7f7fe01e8de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:23.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.214+0000 7f7fe5c00640 1 -- 172.21.15.28:0/2397758320 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7fe006c0f0 msgr2=0x7f7fe01dd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:23.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.214+0000 7f7fe5c00640 1 --2- 172.21.15.28:0/2397758320 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7fe006c0f0 0x7f7fe01dd7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:23.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.214+0000 7f7fe5c00640 1 -- 172.21.15.28:0/2397758320 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fe01ddce0 con 0x7f7fe01503e0 2024-02-21T23:05:23.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.214+0000 7f7fe5c00640 1 --2- 172.21.15.28:0/2397758320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fe01503e0 0x7f7fe00c62b0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f7fd800c7a0 tx=0x7f7fd800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:23.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.215+0000 7f7fdcc00640 1 -- 172.21.15.28:0/2397758320 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fd8003b70 con 0x7f7fe01503e0 2024-02-21T23:05:23.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.215+0000 7f7fdcc00640 1 -- 172.21.15.28:0/2397758320 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fd8003d30 con 0x7f7fe01503e0 2024-02-21T23:05:23.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.215+0000 7f7fe6600640 1 -- 172.21.15.28:0/2397758320 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fe01e9320 con 0x7f7fe01503e0 2024-02-21T23:05:23.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.215+0000 7f7fdcc00640 1 -- 172.21.15.28:0/2397758320 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fd800aa90 con 0x7f7fe01503e0 2024-02-21T23:05:23.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.215+0000 7f7fe6600640 1 -- 172.21.15.28:0/2397758320 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fe01db490 con 0x7f7fe01503e0 2024-02-21T23:05:23.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.216+0000 7f7fe6600640 1 -- 172.21.15.28:0/2397758320 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fe004f290 con 0x7f7fe01503e0 2024-02-21T23:05:23.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.217+0000 7f7fdcc00640 1 -- 172.21.15.28:0/2397758320 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7fd800ac30 con 0x7f7fe01503e0 2024-02-21T23:05:23.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.218+0000 7f7fdcc00640 1 --2- 172.21.15.28:0/2397758320 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7fc006b4d0 0x7f7fc006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:23.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.219+0000 7f7fdcc00640 1 -- 172.21.15.28:0/2397758320 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7129+0+0 (secure 0 0 0) 0x7f7fd808a5b0 con 0x7f7fe01503e0 2024-02-21T23:05:23.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.219+0000 7f7fe5200640 1 --2- 172.21.15.28:0/2397758320 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7fc006b4d0 0x7f7fc006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:23.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.220+0000 7f7fe5200640 1 --2- 172.21.15.28:0/2397758320 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7fc006b4d0 0x7f7fc006d990 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f7fd4005880 tx=0x7f7fd4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:23.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.223+0000 7f7fdcc00640 1 -- 172.21.15.28:0/2397758320 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7fd8058f30 con 0x7f7fe01503e0 2024-02-21T23:05:23.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.428+0000 7f7fe6600640 1 -- 172.21.15.28:0/2397758320 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7fe014b330 con 0x7f7fe01503e0 2024-02-21T23:05:23.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.429+0000 7f7fdcc00640 1 -- 172.21.15.28:0/2397758320 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v59) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7fd8058830 con 0x7f7fe01503e0 2024-02-21T23:05:23.418 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-02-21T23:05:23.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.432+0000 7f7fe6600640 1 -- 172.21.15.28:0/2397758320 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7fc006b4d0 msgr2=0x7f7fc006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:23.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.432+0000 7f7fe6600640 1 --2- 172.21.15.28:0/2397758320 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7fc006b4d0 0x7f7fc006d990 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f7fd4005880 tx=0x7f7fd4009040 comp rx=0 tx=0).stop 2024-02-21T23:05:23.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.432+0000 7f7fe6600640 1 -- 172.21.15.28:0/2397758320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fe01503e0 msgr2=0x7f7fe00c62b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:23.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.432+0000 7f7fe6600640 1 --2- 172.21.15.28:0/2397758320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fe01503e0 0x7f7fe00c62b0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f7fd800c7a0 tx=0x7f7fd800cc60 comp rx=0 tx=0).stop 2024-02-21T23:05:23.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.432+0000 7f7fe6600640 1 -- 172.21.15.28:0/2397758320 shutdown_connections 2024-02-21T23:05:23.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.432+0000 7f7fe6600640 1 --2- 172.21.15.28:0/2397758320 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7fc006b4d0 0x7f7fc006d990 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:23.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.432+0000 7f7fe6600640 1 --2- 172.21.15.28:0/2397758320 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7fe0072cb0 0x7f7fe01e8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:23.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.432+0000 7f7fe6600640 1 --2- 172.21.15.28:0/2397758320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fe01503e0 0x7f7fe00c62b0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:23.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.432+0000 7f7fe6600640 1 --2- 172.21.15.28:0/2397758320 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7fe006c0f0 0x7f7fe01dd7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:23.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.432+0000 7f7fe6600640 1 -- 172.21.15.28:0/2397758320 >> 172.21.15.28:0/2397758320 conn(0x7f7fe0071b20 msgr2=0x7f7fe0071f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:23.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.432+0000 7f7fe6600640 1 -- 172.21.15.28:0/2397758320 shutdown_connections 2024-02-21T23:05:23.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.432+0000 7f7fe6600640 1 -- 172.21.15.28:0/2397758320 wait complete. 2024-02-21T23:05:23.441 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-21T23:05:23.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.671+0000 7f845d400640 1 Processor -- start 2024-02-21T23:05:23.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.671+0000 7f845d400640 1 -- start start 2024-02-21T23:05:23.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.673+0000 7f845d400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f845806daa0 0x7f84580c8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:23.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.673+0000 7f845d400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f84581550e0 0x7f84581554f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:23.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.673+0000 7f845d400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8458158920 con 0x7f8458152650 2024-02-21T23:05:23.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.673+0000 7f845d400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84581586d0 con 0x7f845806daa0 2024-02-21T23:05:23.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.673+0000 7f845d400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8458158c00 con 0x7f84581550e0 2024-02-21T23:05:23.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.673+0000 7f844fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f845806daa0 0x7f84580c8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:23.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.673+0000 7f8457e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f8458152650 0x7f84580c7160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39282/0 (socket says 172.21.15.28:39282) 2024-02-21T23:05:23.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.673+0000 7f8457e00640 1 -- 172.21.15.28:0/1221042216 learned_addr learned my addr 172.21.15.28:0/1221042216 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:05:23.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.673+0000 7f845ca00640 1 --2- 172.21.15.28:0/1221042216 >> v2:172.21.15.81:3300/0 conn(0x7f84581550e0 0x7f84581554f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:23.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.674+0000 7f844fe00640 1 -- 172.21.15.28:0/1221042216 >> v2:172.21.15.81:3300/0 conn(0x7f84581550e0 msgr2=0x7f84581554f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:23.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.674+0000 7f844fe00640 1 --2- 172.21.15.28:0/1221042216 >> v2:172.21.15.81:3300/0 conn(0x7f84581550e0 0x7f84581554f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:23.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.674+0000 7f844fe00640 1 -- 172.21.15.28:0/1221042216 >> v1:172.21.15.28:6789/0 conn(0x7f8458152650 legacy=0x7f84580c7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:23.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.674+0000 7f844fe00640 1 -- 172.21.15.28:0/1221042216 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8458158ee0 con 0x7f845806daa0 2024-02-21T23:05:23.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.674+0000 7f8457400640 1 -- 172.21.15.28:0/1221042216 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2342632073 0 0) 0x7f8458158920 con 0x7f8458152650 2024-02-21T23:05:23.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.675+0000 7f844fe00640 1 --2- 172.21.15.28:0/1221042216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f845806daa0 0x7f84580c8ce0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f84580aa260 tx=0x7f844002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b4e15d619c0da3b server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:23.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.675+0000 7f8457400640 1 -- 172.21.15.28:0/1221042216 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84400315a0 con 0x7f845806daa0 2024-02-21T23:05:23.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.675+0000 7f8457400640 1 -- 172.21.15.28:0/1221042216 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8440037070 con 0x7f845806daa0 2024-02-21T23:05:23.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.675+0000 7f8457400640 1 -- 172.21.15.28:0/1221042216 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8440031960 con 0x7f845806daa0 2024-02-21T23:05:23.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.676+0000 7f845d400640 1 -- 172.21.15.28:0/1221042216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f845806daa0 msgr2=0x7f84580c8ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:23.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.676+0000 7f845d400640 1 --2- 172.21.15.28:0/1221042216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f845806daa0 0x7f84580c8ce0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f84580aa260 tx=0x7f844002b840 comp rx=0 tx=0).stop 2024-02-21T23:05:23.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.676+0000 7f845d400640 1 -- 172.21.15.28:0/1221042216 shutdown_connections 2024-02-21T23:05:23.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.676+0000 7f845d400640 1 --2- 172.21.15.28:0/1221042216 >> v2:172.21.15.81:3300/0 conn(0x7f84581550e0 0x7f84581554f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:23.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.676+0000 7f845d400640 1 --2- 172.21.15.28:0/1221042216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f845806daa0 0x7f84580c8ce0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:23.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.676+0000 7f845d400640 1 -- 172.21.15.28:0/1221042216 >> 172.21.15.28:0/1221042216 conn(0x7f8458072ce0 msgr2=0x7f84580ac340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:23.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.676+0000 7f845d400640 1 -- 172.21.15.28:0/1221042216 shutdown_connections 2024-02-21T23:05:23.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.676+0000 7f845d400640 1 -- 172.21.15.28:0/1221042216 wait complete. 2024-02-21T23:05:23.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.678+0000 7f845d400640 1 Processor -- start 2024-02-21T23:05:23.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.678+0000 7f845d400640 1 -- start start 2024-02-21T23:05:23.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.679+0000 7f845d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f845806daa0 0x7f845814dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:23.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.679+0000 7f845d400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8458152650 0x7f84581eaa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:23.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.679+0000 7f845d400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84581550e0 0x7f84581eafc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:23.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.679+0000 7f845d400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8458152a30 con 0x7f845806daa0 2024-02-21T23:05:23.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.679+0000 7f845d400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84580bdfb0 con 0x7f84581550e0 2024-02-21T23:05:23.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.679+0000 7f845d400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84580c7870 con 0x7f8458152650 2024-02-21T23:05:23.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.679+0000 7f8457e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f845806daa0 0x7f845814dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:23.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.679+0000 7f8457e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f845806daa0 0x7f845814dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41480/0 (socket says 172.21.15.28:41480) 2024-02-21T23:05:23.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.679+0000 7f8457e00640 1 -- 172.21.15.28:0/4102999894 learned_addr learned my addr 172.21.15.28:0/4102999894 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:23.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.680+0000 7f844fe00640 1 --2- 172.21.15.28:0/4102999894 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8458152650 0x7f84581eaa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:23.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.680+0000 7f8457e00640 1 -- 172.21.15.28:0/4102999894 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8458152650 msgr2=0x7f84581eaa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:23.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.680+0000 7f8457e00640 1 --2- 172.21.15.28:0/4102999894 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8458152650 0x7f84581eaa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:23.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.680+0000 7f8457e00640 1 -- 172.21.15.28:0/4102999894 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84581550e0 msgr2=0x7f84581eafc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:23.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.680+0000 7f8457e00640 1 --2- 172.21.15.28:0/4102999894 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84581550e0 0x7f84581eafc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:23.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.680+0000 7f8457e00640 1 -- 172.21.15.28:0/4102999894 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84581eee70 con 0x7f845806daa0 2024-02-21T23:05:23.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.680+0000 7f8457e00640 1 --2- 172.21.15.28:0/4102999894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f845806daa0 0x7f845814dcb0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f844800c890 tx=0x7f844800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:23.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.680+0000 7f8454c00640 1 -- 172.21.15.28:0/4102999894 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8448018070 con 0x7f845806daa0 2024-02-21T23:05:23.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.680+0000 7f8454c00640 1 -- 172.21.15.28:0/4102999894 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f844800a640 con 0x7f845806daa0 2024-02-21T23:05:23.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.680+0000 7f845d400640 1 -- 172.21.15.28:0/4102999894 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84580ac2d0 con 0x7f845806daa0 2024-02-21T23:05:23.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.680+0000 7f8454c00640 1 -- 172.21.15.28:0/4102999894 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f844800aa30 con 0x7f845806daa0 2024-02-21T23:05:23.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.681+0000 7f845d400640 1 -- 172.21.15.28:0/4102999894 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84580ab430 con 0x7f845806daa0 2024-02-21T23:05:23.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.682+0000 7f845d400640 1 -- 172.21.15.28:0/4102999894 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f845804f290 con 0x7f845806daa0 2024-02-21T23:05:23.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.683+0000 7f8454c00640 1 -- 172.21.15.28:0/4102999894 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f844801b070 con 0x7f845806daa0 2024-02-21T23:05:23.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.684+0000 7f8454c00640 1 --2- 172.21.15.28:0/4102999894 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f843406b4d0 0x7f843406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:23.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.684+0000 7f8454c00640 1 -- 172.21.15.28:0/4102999894 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7129+0+0 (secure 0 0 0) 0x7f8448089e90 con 0x7f845806daa0 2024-02-21T23:05:23.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.684+0000 7f844fe00640 1 --2- 172.21.15.28:0/4102999894 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f843406b4d0 0x7f843406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:23.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.685+0000 7f844fe00640 1 --2- 172.21.15.28:0/4102999894 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f843406b4d0 0x7f843406d990 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f8440036000 tx=0x7f844003d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:23.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.693+0000 7f8454c00640 1 -- 172.21.15.28:0/4102999894 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f8448058770 con 0x7f845806daa0 2024-02-21T23:05:23.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.895+0000 7f845d400640 1 -- 172.21.15.28:0/4102999894 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f84580ad190 con 0x7f845806daa0 2024-02-21T23:05:23.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.896+0000 7f8454c00640 1 -- 172.21.15.28:0/4102999894 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v59) v1 ==== 100+0+11 (secure 0 0 0) 0x7f844808b020 con 0x7f845806daa0 2024-02-21T23:05:23.884 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-02-21T23:05:23.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.898+0000 7f845d400640 1 -- 172.21.15.28:0/4102999894 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f843406b4d0 msgr2=0x7f843406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:23.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.898+0000 7f845d400640 1 --2- 172.21.15.28:0/4102999894 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f843406b4d0 0x7f843406d990 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f8440036000 tx=0x7f844003d040 comp rx=0 tx=0).stop 2024-02-21T23:05:23.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.898+0000 7f845d400640 1 -- 172.21.15.28:0/4102999894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f845806daa0 msgr2=0x7f845814dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:23.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.898+0000 7f845d400640 1 --2- 172.21.15.28:0/4102999894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f845806daa0 0x7f845814dcb0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f844800c890 tx=0x7f844800cd50 comp rx=0 tx=0).stop 2024-02-21T23:05:23.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.898+0000 7f845d400640 1 -- 172.21.15.28:0/4102999894 shutdown_connections 2024-02-21T23:05:23.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.898+0000 7f845d400640 1 --2- 172.21.15.28:0/4102999894 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f843406b4d0 0x7f843406d990 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:23.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.898+0000 7f845d400640 1 --2- 172.21.15.28:0/4102999894 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8458152650 0x7f84581eaa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:23.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.898+0000 7f845d400640 1 --2- 172.21.15.28:0/4102999894 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84581550e0 0x7f84581eafc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:23.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.898+0000 7f845d400640 1 --2- 172.21.15.28:0/4102999894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f845806daa0 0x7f845814dcb0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:23.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.898+0000 7f845d400640 1 -- 172.21.15.28:0/4102999894 >> 172.21.15.28:0/4102999894 conn(0x7f8458072ce0 msgr2=0x7f845806c640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:23.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.899+0000 7f845d400640 1 -- 172.21.15.28:0/4102999894 shutdown_connections 2024-02-21T23:05:23.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:23.899+0000 7f845d400640 1 -- 172.21.15.28:0/4102999894 wait complete. 2024-02-21T23:05:23.908 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs2_metadata pg_num 2024-02-21T23:05:24.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.143+0000 7fa1e8a00640 1 Processor -- start 2024-02-21T23:05:24.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.143+0000 7fa1e8a00640 1 -- start start 2024-02-21T23:05:24.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.145+0000 7fa1e8a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa1e413e520 0x7fa1e406c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:24.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.145+0000 7fa1e8a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1e4158550 con 0x7fa1e406dbc0 2024-02-21T23:05:24.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.145+0000 7fa1e8a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1e41587d0 con 0x7fa1e413e0c0 2024-02-21T23:05:24.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.145+0000 7fa1e8a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1e40c89a0 con 0x7fa1e413e520 2024-02-21T23:05:24.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.145+0000 7fa1e1e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa1e406dbc0 0x7fa1e406c280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39290/0 (socket says 172.21.15.28:39290) 2024-02-21T23:05:24.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.145+0000 7fa1e1e00640 1 -- 172.21.15.28:0/3405925485 learned_addr learned my addr 172.21.15.28:0/3405925485 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:05:24.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.145+0000 7fa1e3200640 1 --2- 172.21.15.28:0/3405925485 >> v2:172.21.15.28:3300/0 conn(0x7fa1e413e520 0x7fa1e406c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:24.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.146+0000 7fa1e1400640 1 -- 172.21.15.28:0/3405925485 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3143988721 0 0) 0x7fa1d0001110 con 0x7fa1e406dbc0 2024-02-21T23:05:24.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.146+0000 7fa1e1400640 1 -- 172.21.15.28:0/3405925485 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1c8003520 con 0x7fa1e406dbc0 2024-02-21T23:05:24.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.146+0000 7fa1e3200640 1 -- 172.21.15.28:0/3405925485 >> v1:172.21.15.81:6789/0 conn(0x7fa1e413e0c0 legacy=0x7fa1e40c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:24.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.146+0000 7fa1e3200640 1 -- 172.21.15.28:0/3405925485 >> v1:172.21.15.28:6789/0 conn(0x7fa1e406dbc0 legacy=0x7fa1e406c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:24.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.146+0000 7fa1e3200640 1 -- 172.21.15.28:0/3405925485 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1e4158ab0 con 0x7fa1e413e520 2024-02-21T23:05:24.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.147+0000 7fa1e3200640 1 --2- 172.21.15.28:0/3405925485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1e413e520 0x7fa1e406c990 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc009d90 tx=0x7fa1dc02be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=479793af426824c server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:24.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.148+0000 7fa1e1400640 1 -- 172.21.15.28:0/3405925485 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1dc039070 con 0x7fa1e413e520 2024-02-21T23:05:24.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.148+0000 7fa1e1400640 1 -- 172.21.15.28:0/3405925485 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1dc02cbd0 con 0x7fa1e413e520 2024-02-21T23:05:24.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.148+0000 7fa1e1400640 1 -- 172.21.15.28:0/3405925485 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1dc034440 con 0x7fa1e413e520 2024-02-21T23:05:24.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.148+0000 7fa1e8a00640 1 -- 172.21.15.28:0/3405925485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1e413e520 msgr2=0x7fa1e406c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:24.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.148+0000 7fa1e8a00640 1 --2- 172.21.15.28:0/3405925485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1e413e520 0x7fa1e406c990 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc009d90 tx=0x7fa1dc02be20 comp rx=0 tx=0).stop 2024-02-21T23:05:24.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.148+0000 7fa1e8a00640 1 -- 172.21.15.28:0/3405925485 shutdown_connections 2024-02-21T23:05:24.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.148+0000 7fa1e8a00640 1 --2- 172.21.15.28:0/3405925485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1e413e520 0x7fa1e406c990 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:24.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.148+0000 7fa1e8a00640 1 -- 172.21.15.28:0/3405925485 >> 172.21.15.28:0/3405925485 conn(0x7fa1e4073110 msgr2=0x7fa1e40ac490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:24.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.148+0000 7fa1e8a00640 1 -- 172.21.15.28:0/3405925485 shutdown_connections 2024-02-21T23:05:24.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.149+0000 7fa1e8a00640 1 -- 172.21.15.28:0/3405925485 wait complete. 2024-02-21T23:05:24.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.150+0000 7fa1e8a00640 1 Processor -- start 2024-02-21T23:05:24.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.150+0000 7fa1e8a00640 1 -- start start 2024-02-21T23:05:24.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.151+0000 7fa1e8a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1e406dbc0 0x7fa1e4154cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:24.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.151+0000 7fa1e8a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1e413e0c0 0x7fa1e41532b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:24.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.151+0000 7fa1e8a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa1e413e520 0x7fa1e4139720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:24.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.151+0000 7fa1e8a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e406ced0 con 0x7fa1e413e0c0 2024-02-21T23:05:24.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.151+0000 7fa1e8a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e4159070 con 0x7fa1e406dbc0 2024-02-21T23:05:24.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.151+0000 7fa1e8a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e40c78c0 con 0x7fa1e413e520 2024-02-21T23:05:24.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.152+0000 7fa1e1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1e413e0c0 0x7fa1e41532b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:24.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.152+0000 7fa1e1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1e413e0c0 0x7fa1e41532b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41500/0 (socket says 172.21.15.28:41500) 2024-02-21T23:05:24.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.152+0000 7fa1e1e00640 1 -- 172.21.15.28:0/2478284750 learned_addr learned my addr 172.21.15.28:0/2478284750 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:24.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.152+0000 7fa1e2800640 1 --2- 172.21.15.28:0/2478284750 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1e406dbc0 0x7fa1e4154cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:24.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.152+0000 7fa1e3200640 1 --2- 172.21.15.28:0/2478284750 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa1e413e520 0x7fa1e4139720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:24.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.152+0000 7fa1e1e00640 1 -- 172.21.15.28:0/2478284750 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa1e413e520 msgr2=0x7fa1e4139720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:24.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.152+0000 7fa1e1e00640 1 --2- 172.21.15.28:0/2478284750 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa1e413e520 0x7fa1e4139720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:24.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.152+0000 7fa1e1e00640 1 -- 172.21.15.28:0/2478284750 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1e406dbc0 msgr2=0x7fa1e4154cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:24.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.152+0000 7fa1e1e00640 1 --2- 172.21.15.28:0/2478284750 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1e406dbc0 0x7fa1e4154cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:24.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.152+0000 7fa1e1e00640 1 -- 172.21.15.28:0/2478284750 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1e4154730 con 0x7fa1e413e0c0 2024-02-21T23:05:24.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.153+0000 7fa1e1e00640 1 --2- 172.21.15.28:0/2478284750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1e413e0c0 0x7fa1e41532b0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fa1d000c430 tx=0x7fa1d000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:24.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.153+0000 7fa1daa00640 1 -- 172.21.15.28:0/2478284750 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1d0017070 con 0x7fa1e413e0c0 2024-02-21T23:05:24.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.153+0000 7fa1daa00640 1 -- 172.21.15.28:0/2478284750 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1d0005ce0 con 0x7fa1e413e0c0 2024-02-21T23:05:24.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.153+0000 7fa1daa00640 1 -- 172.21.15.28:0/2478284750 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1d00072d0 con 0x7fa1e413e0c0 2024-02-21T23:05:24.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.153+0000 7fa1e8a00640 1 -- 172.21.15.28:0/2478284750 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1e4153bb0 con 0x7fa1e413e0c0 2024-02-21T23:05:24.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.153+0000 7fa1e8a00640 1 -- 172.21.15.28:0/2478284750 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1e4154170 con 0x7fa1e413e0c0 2024-02-21T23:05:24.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.154+0000 7fa1e8a00640 1 -- 172.21.15.28:0/2478284750 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1e404f290 con 0x7fa1e413e0c0 2024-02-21T23:05:24.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.155+0000 7fa1daa00640 1 -- 172.21.15.28:0/2478284750 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa1d001a070 con 0x7fa1e413e0c0 2024-02-21T23:05:24.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.156+0000 7fa1daa00640 1 --2- 172.21.15.28:0/2478284750 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa1c406b4d0 0x7fa1c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:24.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.156+0000 7fa1daa00640 1 -- 172.21.15.28:0/2478284750 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 7129+0+0 (secure 0 0 0) 0x7fa1d0089a50 con 0x7fa1e413e0c0 2024-02-21T23:05:24.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.157+0000 7fa1e2800640 1 --2- 172.21.15.28:0/2478284750 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa1c406b4d0 0x7fa1c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:24.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.158+0000 7fa1e2800640 1 --2- 172.21.15.28:0/2478284750 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa1c406b4d0 0x7fa1c406d990 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fa1d4000f60 tx=0x7fa1d4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:24.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.161+0000 7fa1daa00640 1 -- 172.21.15.28:0/2478284750 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fa1e4158550 con 0x7fa1e413e0c0 2024-02-21T23:05:24.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.361+0000 7fa1e8a00640 1 -- 172.21.15.28:0/2478284750 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs2_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa1e413c020 con 0x7fa1e413e0c0 2024-02-21T23:05:24.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.362+0000 7fa1daa00640 1 -- 172.21.15.28:0/2478284750 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs2_metadata", "var": "pg_num"}]=0 v59) v1 ==== 105+0+11 (secure 0 0 0) 0x7fa1d0057cf0 con 0x7fa1e413e0c0 2024-02-21T23:05:24.351 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-02-21T23:05:24.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.364+0000 7fa1e8a00640 1 -- 172.21.15.28:0/2478284750 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa1c406b4d0 msgr2=0x7fa1c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:24.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.364+0000 7fa1e8a00640 1 --2- 172.21.15.28:0/2478284750 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa1c406b4d0 0x7fa1c406d990 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fa1d4000f60 tx=0x7fa1d4009040 comp rx=0 tx=0).stop 2024-02-21T23:05:24.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.364+0000 7fa1e8a00640 1 -- 172.21.15.28:0/2478284750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1e413e0c0 msgr2=0x7fa1e41532b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:24.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.364+0000 7fa1e8a00640 1 --2- 172.21.15.28:0/2478284750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1e413e0c0 0x7fa1e41532b0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fa1d000c430 tx=0x7fa1d000c8f0 comp rx=0 tx=0).stop 2024-02-21T23:05:24.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.365+0000 7fa1e8a00640 1 -- 172.21.15.28:0/2478284750 shutdown_connections 2024-02-21T23:05:24.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.365+0000 7fa1e8a00640 1 --2- 172.21.15.28:0/2478284750 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa1c406b4d0 0x7fa1c406d990 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:24.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.365+0000 7fa1e8a00640 1 --2- 172.21.15.28:0/2478284750 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa1e413e520 0x7fa1e4139720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:24.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.365+0000 7fa1e8a00640 1 --2- 172.21.15.28:0/2478284750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1e413e0c0 0x7fa1e41532b0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:24.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.365+0000 7fa1e8a00640 1 --2- 172.21.15.28:0/2478284750 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1e406dbc0 0x7fa1e4154cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:24.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.365+0000 7fa1e8a00640 1 -- 172.21.15.28:0/2478284750 >> 172.21.15.28:0/2478284750 conn(0x7fa1e4073110 msgr2=0x7fa1e4157870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:24.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.365+0000 7fa1e8a00640 1 -- 172.21.15.28:0/2478284750 shutdown_connections 2024-02-21T23:05:24.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.365+0000 7fa1e8a00640 1 -- 172.21.15.28:0/2478284750 wait complete. 2024-02-21T23:05:24.375 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs2_data pg_num 2024-02-21T23:05:24.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.602+0000 7ff037e00640 1 Processor -- start 2024-02-21T23:05:24.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.602+0000 7ff037e00640 1 -- start start 2024-02-21T23:05:24.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.604+0000 7ff037e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff03813e0c0 0x7ff0380c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:24.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.604+0000 7ff037e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff03806dbc0 0x7ff03806c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:24.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.604+0000 7ff037e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff038158550 con 0x7ff0380c76f0 2024-02-21T23:05:24.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.604+0000 7ff037e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0380c89a0 con 0x7ff03806dbc0 2024-02-21T23:05:24.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.604+0000 7ff037e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0381587d0 con 0x7ff03813e0c0 2024-02-21T23:05:24.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.604+0000 7ff036000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff03806dbc0 0x7ff03806c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:24.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.604+0000 7ff037400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff0380c76f0 0x7ff03806c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39292/0 (socket says 172.21.15.28:39292) 2024-02-21T23:05:24.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.604+0000 7ff037400640 1 -- 172.21.15.28:0/3256633110 learned_addr learned my addr 172.21.15.28:0/3256633110 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:05:24.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.604+0000 7ff036a00640 1 --2- 172.21.15.28:0/3256633110 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff03813e0c0 0x7ff0380c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:24.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.605+0000 7ff036000640 1 -- 172.21.15.28:0/3256633110 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff03813e0c0 msgr2=0x7ff0380c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:24.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.605+0000 7ff036000640 1 --2- 172.21.15.28:0/3256633110 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff03813e0c0 0x7ff0380c71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:24.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.605+0000 7ff036000640 1 -- 172.21.15.28:0/3256633110 >> v1:172.21.15.28:6789/0 conn(0x7ff0380c76f0 legacy=0x7ff03806c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:24.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.605+0000 7ff036000640 1 -- 172.21.15.28:0/3256633110 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff038158ab0 con 0x7ff03806dbc0 2024-02-21T23:05:24.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.605+0000 7ff035600640 1 -- 172.21.15.28:0/3256633110 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1691615329 0 0) 0x7ff030001490 con 0x7ff0380c76f0 2024-02-21T23:05:24.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.606+0000 7ff036000640 1 --2- 172.21.15.28:0/3256633110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff03806dbc0 0x7ff03806c280 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7ff0240097a0 tx=0x7ff02402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e38b46f60b9620a6 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:24.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.606+0000 7ff035600640 1 -- 172.21.15.28:0/3256633110 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0240315a0 con 0x7ff03806dbc0 2024-02-21T23:05:24.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.606+0000 7ff035600640 1 -- 172.21.15.28:0/3256633110 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff024037070 con 0x7ff03806dbc0 2024-02-21T23:05:24.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.606+0000 7ff035600640 1 -- 172.21.15.28:0/3256633110 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff024031960 con 0x7ff03806dbc0 2024-02-21T23:05:24.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.607+0000 7ff037e00640 1 -- 172.21.15.28:0/3256633110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff03806dbc0 msgr2=0x7ff03806c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:24.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.607+0000 7ff037e00640 1 --2- 172.21.15.28:0/3256633110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff03806dbc0 0x7ff03806c280 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7ff0240097a0 tx=0x7ff02402b840 comp rx=0 tx=0).stop 2024-02-21T23:05:24.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.607+0000 7ff037e00640 1 -- 172.21.15.28:0/3256633110 shutdown_connections 2024-02-21T23:05:24.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.607+0000 7ff037e00640 1 --2- 172.21.15.28:0/3256633110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff03806dbc0 0x7ff03806c280 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:24.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.607+0000 7ff037e00640 1 --2- 172.21.15.28:0/3256633110 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff03813e0c0 0x7ff0380c71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:24.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.607+0000 7ff037e00640 1 -- 172.21.15.28:0/3256633110 >> 172.21.15.28:0/3256633110 conn(0x7ff038073110 msgr2=0x7ff0380ac490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:24.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.607+0000 7ff037e00640 1 -- 172.21.15.28:0/3256633110 shutdown_connections 2024-02-21T23:05:24.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.607+0000 7ff037e00640 1 -- 172.21.15.28:0/3256633110 wait complete. 2024-02-21T23:05:24.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.609+0000 7ff037e00640 1 Processor -- start 2024-02-21T23:05:24.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.609+0000 7ff037e00640 1 -- start start 2024-02-21T23:05:24.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.611+0000 7ff037e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff03806dbc0 0x7ff038134bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:24.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.611+0000 7ff037e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0380c76f0 0x7ff038152310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:24.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.611+0000 7ff037e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff03813e0c0 0x7ff038137030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:24.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.611+0000 7ff037e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff03806ced0 con 0x7ff0380c76f0 2024-02-21T23:05:24.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.611+0000 7ff037e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff038159070 con 0x7ff03806dbc0 2024-02-21T23:05:24.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.611+0000 7ff037e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff03813e520 con 0x7ff03813e0c0 2024-02-21T23:05:24.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.611+0000 7ff036000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0380c76f0 0x7ff038152310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:24.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.611+0000 7ff036000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0380c76f0 0x7ff038152310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41512/0 (socket says 172.21.15.28:41512) 2024-02-21T23:05:24.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.611+0000 7ff036000640 1 -- 172.21.15.28:0/39692483 learned_addr learned my addr 172.21.15.28:0/39692483 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:24.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.611+0000 7ff036a00640 1 --2- 172.21.15.28:0/39692483 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff03806dbc0 0x7ff038134bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:24.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.612+0000 7ff037400640 1 --2- 172.21.15.28:0/39692483 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff03813e0c0 0x7ff038137030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:24.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.612+0000 7ff036000640 1 -- 172.21.15.28:0/39692483 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff03813e0c0 msgr2=0x7ff038137030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:24.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.612+0000 7ff036000640 1 --2- 172.21.15.28:0/39692483 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff03813e0c0 0x7ff038137030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:24.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.612+0000 7ff036000640 1 -- 172.21.15.28:0/39692483 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff03806dbc0 msgr2=0x7ff038134bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:24.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.612+0000 7ff036000640 1 --2- 172.21.15.28:0/39692483 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff03806dbc0 0x7ff038134bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:24.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.612+0000 7ff036000640 1 -- 172.21.15.28:0/39692483 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff038135190 con 0x7ff0380c76f0 2024-02-21T23:05:24.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.612+0000 7ff037400640 1 --2- 172.21.15.28:0/39692483 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff03813e0c0 0x7ff038137030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:05:24.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.612+0000 7ff036000640 1 --2- 172.21.15.28:0/39692483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0380c76f0 0x7ff038152310 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7ff0240098c0 tx=0x7ff0240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:24.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.612+0000 7ff02ea00640 1 -- 172.21.15.28:0/39692483 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff024032630 con 0x7ff0380c76f0 2024-02-21T23:05:24.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.612+0000 7ff02ea00640 1 -- 172.21.15.28:0/39692483 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0240327d0 con 0x7ff0380c76f0 2024-02-21T23:05:24.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.612+0000 7ff037e00640 1 -- 172.21.15.28:0/39692483 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff038135470 con 0x7ff0380c76f0 2024-02-21T23:05:24.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.612+0000 7ff02ea00640 1 -- 172.21.15.28:0/39692483 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff024032bc0 con 0x7ff0380c76f0 2024-02-21T23:05:24.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.613+0000 7ff037e00640 1 -- 172.21.15.28:0/39692483 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff038135a30 con 0x7ff0380c76f0 2024-02-21T23:05:24.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.614+0000 7ff037e00640 1 -- 172.21.15.28:0/39692483 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff03804f290 con 0x7ff0380c76f0 2024-02-21T23:05:24.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.615+0000 7ff02ea00640 1 -- 172.21.15.28:0/39692483 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff02403b3f0 con 0x7ff0380c76f0 2024-02-21T23:05:24.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.615+0000 7ff02ea00640 1 --2- 172.21.15.28:0/39692483 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff01806b4d0 0x7ff01806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:24.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.616+0000 7ff02ea00640 1 -- 172.21.15.28:0/39692483 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7332+0+0 (secure 0 0 0) 0x7ff02402f030 con 0x7ff0380c76f0 2024-02-21T23:05:24.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.616+0000 7ff036a00640 1 --2- 172.21.15.28:0/39692483 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff01806b4d0 0x7ff01806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:24.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.617+0000 7ff036a00640 1 --2- 172.21.15.28:0/39692483 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff01806b4d0 0x7ff01806d990 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7ff028009aa0 tx=0x7ff028009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:24.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.620+0000 7ff02ea00640 1 -- 172.21.15.28:0/39692483 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ff024040070 con 0x7ff0380c76f0 2024-02-21T23:05:24.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.820+0000 7ff037e00640 1 -- 172.21.15.28:0/39692483 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs2_data", "var": "pg_num"} v 0) v1 -- 0x7ff038137b70 con 0x7ff0380c76f0 2024-02-21T23:05:24.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.820+0000 7ff02ea00640 1 -- 172.21.15.28:0/39692483 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs2_data", "var": "pg_num"}]=0 v60) v1 ==== 101+0+11 (secure 0 0 0) 0x7ff02407b010 con 0x7ff0380c76f0 2024-02-21T23:05:24.809 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-02-21T23:05:24.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.822+0000 7ff037e00640 1 -- 172.21.15.28:0/39692483 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff01806b4d0 msgr2=0x7ff01806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:24.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.822+0000 7ff037e00640 1 --2- 172.21.15.28:0/39692483 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff01806b4d0 0x7ff01806d990 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7ff028009aa0 tx=0x7ff028009190 comp rx=0 tx=0).stop 2024-02-21T23:05:24.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.822+0000 7ff037e00640 1 -- 172.21.15.28:0/39692483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0380c76f0 msgr2=0x7ff038152310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:24.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.822+0000 7ff037e00640 1 --2- 172.21.15.28:0/39692483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0380c76f0 0x7ff038152310 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7ff0240098c0 tx=0x7ff0240046c0 comp rx=0 tx=0).stop 2024-02-21T23:05:24.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.823+0000 7ff037e00640 1 -- 172.21.15.28:0/39692483 shutdown_connections 2024-02-21T23:05:24.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.823+0000 7ff037e00640 1 --2- 172.21.15.28:0/39692483 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff01806b4d0 0x7ff01806d990 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:24.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.823+0000 7ff037e00640 1 --2- 172.21.15.28:0/39692483 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff03813e0c0 0x7ff038137030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:24.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.823+0000 7ff037e00640 1 --2- 172.21.15.28:0/39692483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0380c76f0 0x7ff038152310 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:24.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.823+0000 7ff037e00640 1 --2- 172.21.15.28:0/39692483 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff03806dbc0 0x7ff038134bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:24.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.823+0000 7ff037e00640 1 -- 172.21.15.28:0/39692483 >> 172.21.15.28:0/39692483 conn(0x7ff038073110 msgr2=0x7ff03813ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:24.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.823+0000 7ff037e00640 1 -- 172.21.15.28:0/39692483 shutdown_connections 2024-02-21T23:05:24.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:24.823+0000 7ff037e00640 1 -- 172.21.15.28:0/39692483 wait complete. 2024-02-21T23:05:24.829 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:05:25.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.080+0000 7f832fc00640 1 Processor -- start 2024-02-21T23:05:25.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.080+0000 7f832fc00640 1 -- start start 2024-02-21T23:05:25.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.081+0000 7f832fc00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f83281525c0 0x7f83280cb390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:25.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.081+0000 7f832fc00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f83280c3280 0x7f83280c3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:25.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.082+0000 7f832fc00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8328155120 con 0x7f8328159010 2024-02-21T23:05:25.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.082+0000 7f832fc00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8328155390 con 0x7f83281525c0 2024-02-21T23:05:25.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.082+0000 7f832fc00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8328155670 con 0x7f83280c3280 2024-02-21T23:05:25.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.082+0000 7f8327e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f8328159010 0x7f83280c6a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:44020/0 (socket says 172.21.15.28:44020) 2024-02-21T23:05:25.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.082+0000 7f832d200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f83281525c0 0x7f83280cb390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:25.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.082+0000 7f8327e00640 1 -- 172.21.15.28:0/3131409997 learned_addr learned my addr 172.21.15.28:0/3131409997 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:05:25.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.082+0000 7f832dc00640 1 --2- 172.21.15.28:0/3131409997 >> v2:172.21.15.81:3300/0 conn(0x7f83280c3280 0x7f83280c3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:25.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.082+0000 7f832d200640 1 -- 172.21.15.28:0/3131409997 >> v2:172.21.15.81:3300/0 conn(0x7f83280c3280 msgr2=0x7f83280c3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:25.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.082+0000 7f832d200640 1 --2- 172.21.15.28:0/3131409997 >> v2:172.21.15.81:3300/0 conn(0x7f83280c3280 0x7f83280c3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:25.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.082+0000 7f832d200640 1 -- 172.21.15.28:0/3131409997 >> v1:172.21.15.28:6789/0 conn(0x7f8328159010 legacy=0x7f83280c6a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:25.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.082+0000 7f832d200640 1 -- 172.21.15.28:0/3131409997 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8328155950 con 0x7f83281525c0 2024-02-21T23:05:25.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.083+0000 7f832dc00640 1 --2- 172.21.15.28:0/3131409997 >> v2:172.21.15.81:3300/0 conn(0x7f83280c3280 0x7f83280c3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:05:25.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.084+0000 7f832d200640 1 --2- 172.21.15.28:0/3131409997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83281525c0 0x7f83280cb390 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f832808c1e0 tx=0x7f831c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58c7b7de5cd3637b server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:25.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.084+0000 7f8327400640 1 -- 172.21.15.28:0/3131409997 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f831c035020 con 0x7f83281525c0 2024-02-21T23:05:25.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.084+0000 7f8327400640 1 -- 172.21.15.28:0/3131409997 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f831c031830 con 0x7f83281525c0 2024-02-21T23:05:25.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.084+0000 7f8327400640 1 -- 172.21.15.28:0/3131409997 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f831c031c00 con 0x7f83281525c0 2024-02-21T23:05:25.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.085+0000 7f832fc00640 1 -- 172.21.15.28:0/3131409997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83281525c0 msgr2=0x7f83280cb390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:25.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.085+0000 7f832fc00640 1 --2- 172.21.15.28:0/3131409997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83281525c0 0x7f83280cb390 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f832808c1e0 tx=0x7f831c02b8d0 comp rx=0 tx=0).stop 2024-02-21T23:05:25.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.085+0000 7f832fc00640 1 -- 172.21.15.28:0/3131409997 shutdown_connections 2024-02-21T23:05:25.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.085+0000 7f832fc00640 1 --2- 172.21.15.28:0/3131409997 >> v2:172.21.15.81:3300/0 conn(0x7f83280c3280 0x7f83280c3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:25.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.085+0000 7f832fc00640 1 --2- 172.21.15.28:0/3131409997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83281525c0 0x7f83280cb390 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:25.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.085+0000 7f832fc00640 1 -- 172.21.15.28:0/3131409997 >> 172.21.15.28:0/3131409997 conn(0x7f83280729c0 msgr2=0x7f8328072dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:25.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.085+0000 7f832fc00640 1 -- 172.21.15.28:0/3131409997 shutdown_connections 2024-02-21T23:05:25.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.085+0000 7f832fc00640 1 -- 172.21.15.28:0/3131409997 wait complete. 2024-02-21T23:05:25.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.086+0000 7f832fc00640 1 Processor -- start 2024-02-21T23:05:25.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.087+0000 7f832fc00640 1 -- start start 2024-02-21T23:05:25.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.087+0000 7f832fc00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f83280c3280 0x7f83280c0700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:25.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.087+0000 7f832fc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83281525c0 0x7f83281eed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:25.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.087+0000 7f832fc00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f83281e22d0 0x7f83281e1580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:25.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.087+0000 7f832fc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83281594a0 con 0x7f83281525c0 2024-02-21T23:05:25.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.087+0000 7f832fc00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83280caff0 con 0x7f83281e22d0 2024-02-21T23:05:25.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.087+0000 7f832fc00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83280c3bd0 con 0x7f83280c3280 2024-02-21T23:05:25.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.088+0000 7f8327e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83281525c0 0x7f83281eed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:25.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.088+0000 7f8327e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83281525c0 0x7f83281eed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40184/0 (socket says 172.21.15.28:40184) 2024-02-21T23:05:25.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.088+0000 7f8327e00640 1 -- 172.21.15.28:0/802843631 learned_addr learned my addr 172.21.15.28:0/802843631 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:25.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.088+0000 7f832d200640 1 --2- 172.21.15.28:0/802843631 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f83280c3280 0x7f83280c0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:25.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.088+0000 7f832dc00640 1 --2- 172.21.15.28:0/802843631 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f83281e22d0 0x7f83281e1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:25.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.088+0000 7f8327e00640 1 -- 172.21.15.28:0/802843631 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f83280c3280 msgr2=0x7f83280c0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:25.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.088+0000 7f8327e00640 1 --2- 172.21.15.28:0/802843631 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f83280c3280 0x7f83280c0700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:25.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.088+0000 7f8327e00640 1 -- 172.21.15.28:0/802843631 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f83281e22d0 msgr2=0x7f83281e1580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:25.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.088+0000 7f8327e00640 1 --2- 172.21.15.28:0/802843631 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f83281e22d0 0x7f83281e1580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:25.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.088+0000 7f8327e00640 1 -- 172.21.15.28:0/802843631 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83280c0ce0 con 0x7f83281525c0 2024-02-21T23:05:25.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.089+0000 7f8327e00640 1 --2- 172.21.15.28:0/802843631 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83281525c0 0x7f83281eed40 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f831400c3c0 tx=0x7f831400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:25.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.089+0000 7f8324c00640 1 -- 172.21.15.28:0/802843631 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8314017070 con 0x7f83281525c0 2024-02-21T23:05:25.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.089+0000 7f8324c00640 1 -- 172.21.15.28:0/802843631 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8314005ce0 con 0x7f83281525c0 2024-02-21T23:05:25.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.089+0000 7f8324c00640 1 -- 172.21.15.28:0/802843631 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83140072f0 con 0x7f83281525c0 2024-02-21T23:05:25.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.089+0000 7f832fc00640 1 -- 172.21.15.28:0/802843631 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83281ee260 con 0x7f83281525c0 2024-02-21T23:05:25.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.089+0000 7f832fc00640 1 -- 172.21.15.28:0/802843631 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83281ee820 con 0x7f83281525c0 2024-02-21T23:05:25.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.090+0000 7f832fc00640 1 -- 172.21.15.28:0/802843631 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83281eea20 con 0x7f83281525c0 2024-02-21T23:05:25.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.091+0000 7f8324c00640 1 -- 172.21.15.28:0/802843631 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f831401a070 con 0x7f83281525c0 2024-02-21T23:05:25.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.092+0000 7f8324c00640 1 --2- 172.21.15.28:0/802843631 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f830806b4d0 0x7f830806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:25.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.092+0000 7f8324c00640 1 -- 172.21.15.28:0/802843631 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 7332+0+0 (secure 0 0 0) 0x7f8314089bb0 con 0x7f83281525c0 2024-02-21T23:05:25.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.092+0000 7f832d200640 1 --2- 172.21.15.28:0/802843631 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f830806b4d0 0x7f830806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:25.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.094+0000 7f832d200640 1 --2- 172.21.15.28:0/802843631 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f830806b4d0 0x7f830806d990 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f831c036000 tx=0x7f831c008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:25.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.096+0000 7f8324c00640 1 -- 172.21.15.28:0/802843631 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f83281eea20 con 0x7f83281525c0 2024-02-21T23:05:25.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.335+0000 7f832fc00640 1 -- 172.21.15.28:0/802843631 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8328067f30 con 0x7f83281525c0 2024-02-21T23:05:25.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.336+0000 7f8324c00640 1 -- 172.21.15.28:0/802843631 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+6176 (secure 0 0 0) 0x7f8314058460 con 0x7f83281525c0 2024-02-21T23:05:25.325 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:05:25.325 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":47,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5221,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":7,"addr":"172.21.15.81:6835/2152160500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2152160500},{"type":"v1","addr":"172.21.15.81:6835","nonce":2152160500}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":6039,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3807390545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3807390545},{"type":"v1","addr":"172.21.15.28:6835","nonce":3807390545}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:04:53.432188+0000","modified":"2024-02-21T23:05:19.431400+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6021},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6021":{"gid":6021,"name":"a","rank":0,"incarnation":44,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6834/3835332438","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3835332438},{"type":"v1","addr":"172.21.15.28:6834","nonce":3835332438}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3},{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:05:08.017586+0000","modified":"2024-02-21T23:05:10.974691+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5224},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5224":{"gid":5224,"name":"d","rank":0,"incarnation":33,"state":"up:active","state_seq":7,"addr":"172.21.15.81:6837/442767245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":442767245},{"type":"v1","addr":"172.21.15.81:6837","nonce":442767245}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-02-21T23:05:25.325 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 47 2024-02-21T23:05:25.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.339+0000 7f832fc00640 1 -- 172.21.15.28:0/802843631 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f830806b4d0 msgr2=0x7f830806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:25.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.339+0000 7f832fc00640 1 --2- 172.21.15.28:0/802843631 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f830806b4d0 0x7f830806d990 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f831c036000 tx=0x7f831c008030 comp rx=0 tx=0).stop 2024-02-21T23:05:25.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.339+0000 7f832fc00640 1 -- 172.21.15.28:0/802843631 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83281525c0 msgr2=0x7f83281eed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:25.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.339+0000 7f832fc00640 1 --2- 172.21.15.28:0/802843631 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83281525c0 0x7f83281eed40 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f831400c3c0 tx=0x7f831400c880 comp rx=0 tx=0).stop 2024-02-21T23:05:25.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.339+0000 7f832fc00640 1 -- 172.21.15.28:0/802843631 shutdown_connections 2024-02-21T23:05:25.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.339+0000 7f832fc00640 1 --2- 172.21.15.28:0/802843631 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f830806b4d0 0x7f830806d990 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:25.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.339+0000 7f832fc00640 1 --2- 172.21.15.28:0/802843631 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f83281e22d0 0x7f83281e1580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:25.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.339+0000 7f832fc00640 1 --2- 172.21.15.28:0/802843631 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83281525c0 0x7f83281eed40 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:25.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.339+0000 7f832fc00640 1 --2- 172.21.15.28:0/802843631 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f83280c3280 0x7f83280c0700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:25.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.339+0000 7f832fc00640 1 -- 172.21.15.28:0/802843631 >> 172.21.15.28:0/802843631 conn(0x7f83280729c0 msgr2=0x7f8328072da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:25.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.340+0000 7f832fc00640 1 -- 172.21.15.28:0/802843631 shutdown_connections 2024-02-21T23:05:25.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.340+0000 7f832fc00640 1 -- 172.21.15.28:0/802843631 wait complete. 2024-02-21T23:05:25.347 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:05:25.402 INFO:tasks.ceph.mds.a.smithi028.stderr:2024-02-21T23:05:25.413+0000 7f0a2fe00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-21T23:05:25.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.581+0000 7f179ac00640 1 Processor -- start 2024-02-21T23:05:25.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.581+0000 7f179ac00640 1 -- start start 2024-02-21T23:05:25.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.582+0000 7f179ac00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f17940c1280 0x7f17940c2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:25.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.583+0000 7f179ac00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1794155c30 0x7f1794151fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:25.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.583+0000 7f179ac00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f179406e310 con 0x7f179406f5a0 2024-02-21T23:05:25.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.583+0000 7f179ac00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17940c3a40 con 0x7f1794155c30 2024-02-21T23:05:25.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.583+0000 7f179ac00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17941557a0 con 0x7f17940c1280 2024-02-21T23:05:25.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.583+0000 7f1798c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1794155c30 0x7f1794151fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:25.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.583+0000 7f1798c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1794155c30 0x7f1794151fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40186/0 (socket says 172.21.15.28:40186) 2024-02-21T23:05:25.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.583+0000 7f1798c00640 1 -- 172.21.15.28:0/3330115974 learned_addr learned my addr 172.21.15.28:0/3330115974 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:25.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.583+0000 7f1793e00640 1 --2- 172.21.15.28:0/3330115974 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f17940c1280 0x7f17940c2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:25.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.583+0000 7f1798c00640 1 -- 172.21.15.28:0/3330115974 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f17940c1280 msgr2=0x7f17940c2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:25.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.583+0000 7f1798c00640 1 --2- 172.21.15.28:0/3330115974 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f17940c1280 0x7f17940c2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:25.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.584+0000 7f1798c00640 1 -- 172.21.15.28:0/3330115974 >> v1:172.21.15.81:6789/0 conn(0x7f179406f5a0 legacy=0x7f179406dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:25.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.584+0000 7f1798c00640 1 -- 172.21.15.28:0/3330115974 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17940c3760 con 0x7f1794155c30 2024-02-21T23:05:25.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.585+0000 7f1798c00640 1 --2- 172.21.15.28:0/3330115974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1794155c30 0x7f1794151fe0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f17940c3bc0 tx=0x7f178c02bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=708608d49d450697 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:25.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.585+0000 7f1793400640 1 -- 172.21.15.28:0/3330115974 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f178c02caf0 con 0x7f1794155c30 2024-02-21T23:05:25.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.585+0000 7f1793400640 1 -- 172.21.15.28:0/3330115974 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f178c02cc90 con 0x7f1794155c30 2024-02-21T23:05:25.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.585+0000 7f1793400640 1 -- 172.21.15.28:0/3330115974 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f178c034610 con 0x7f1794155c30 2024-02-21T23:05:25.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.586+0000 7f179ac00640 1 -- 172.21.15.28:0/3330115974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1794155c30 msgr2=0x7f1794151fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:25.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.586+0000 7f179ac00640 1 --2- 172.21.15.28:0/3330115974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1794155c30 0x7f1794151fe0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f17940c3bc0 tx=0x7f178c02bf50 comp rx=0 tx=0).stop 2024-02-21T23:05:25.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.586+0000 7f179ac00640 1 -- 172.21.15.28:0/3330115974 shutdown_connections 2024-02-21T23:05:25.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.586+0000 7f179ac00640 1 --2- 172.21.15.28:0/3330115974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1794155c30 0x7f1794151fe0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:25.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.586+0000 7f179ac00640 1 --2- 172.21.15.28:0/3330115974 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f17940c1280 0x7f17940c2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:25.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.586+0000 7f179ac00640 1 -- 172.21.15.28:0/3330115974 >> 172.21.15.28:0/3330115974 conn(0x7f17940c3c20 msgr2=0x7f17940c4030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:25.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.586+0000 7f179ac00640 1 -- 172.21.15.28:0/3330115974 shutdown_connections 2024-02-21T23:05:25.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.586+0000 7f179ac00640 1 -- 172.21.15.28:0/3330115974 wait complete. 2024-02-21T23:05:25.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.589+0000 7f179ac00640 1 Processor -- start 2024-02-21T23:05:25.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.589+0000 7f179ac00640 1 -- start start 2024-02-21T23:05:25.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.590+0000 7f179ac00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f179406f5a0 0x7f1794151590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:25.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.590+0000 7f179ac00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17940c1280 0x7f17940c5b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:25.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.590+0000 7f179ac00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1794155c30 0x7f17941566e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:25.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.590+0000 7f179ac00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1794155430 con 0x7f17940c1280 2024-02-21T23:05:25.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.590+0000 7f179ac00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17940c3510 con 0x7f179406f5a0 2024-02-21T23:05:25.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.590+0000 7f179ac00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17940c1660 con 0x7f1794155c30 2024-02-21T23:05:25.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.591+0000 7f178be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17940c1280 0x7f17940c5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:25.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.591+0000 7f178be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17940c1280 0x7f17940c5b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40200/0 (socket says 172.21.15.28:40200) 2024-02-21T23:05:25.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.591+0000 7f178be00640 1 -- 172.21.15.28:0/460164788 learned_addr learned my addr 172.21.15.28:0/460164788 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:25.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.591+0000 7f1793e00640 1 --2- 172.21.15.28:0/460164788 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f179406f5a0 0x7f1794151590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:25.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.591+0000 7f1798c00640 1 --2- 172.21.15.28:0/460164788 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1794155c30 0x7f17941566e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:25.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.591+0000 7f178be00640 1 -- 172.21.15.28:0/460164788 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1794155c30 msgr2=0x7f17941566e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:25.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.591+0000 7f178be00640 1 --2- 172.21.15.28:0/460164788 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1794155c30 0x7f17941566e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:25.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.591+0000 7f178be00640 1 -- 172.21.15.28:0/460164788 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f179406f5a0 msgr2=0x7f1794151590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:25.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.591+0000 7f178be00640 1 --2- 172.21.15.28:0/460164788 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f179406f5a0 0x7f1794151590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:25.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.591+0000 7f178be00640 1 -- 172.21.15.28:0/460164788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17941eb9a0 con 0x7f17940c1280 2024-02-21T23:05:25.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.591+0000 7f1793e00640 1 --2- 172.21.15.28:0/460164788 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f179406f5a0 0x7f1794151590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:05:25.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.591+0000 7f1798c00640 1 --2- 172.21.15.28:0/460164788 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1794155c30 0x7f17941566e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:05:25.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.591+0000 7f178be00640 1 --2- 172.21.15.28:0/460164788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17940c1280 0x7f17940c5b30 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f177c00c430 tx=0x7f177c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:25.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.592+0000 7f1790c00640 1 -- 172.21.15.28:0/460164788 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f177c0052d0 con 0x7f17940c1280 2024-02-21T23:05:25.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.592+0000 7f1790c00640 1 -- 172.21.15.28:0/460164788 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f177c005490 con 0x7f17940c1280 2024-02-21T23:05:25.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.592+0000 7f1790c00640 1 -- 172.21.15.28:0/460164788 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f177c007260 con 0x7f17940c1280 2024-02-21T23:05:25.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.592+0000 7f179ac00640 1 -- 172.21.15.28:0/460164788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1794159070 con 0x7f17940c1280 2024-02-21T23:05:25.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.592+0000 7f179ac00640 1 -- 172.21.15.28:0/460164788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17940c6070 con 0x7f17940c1280 2024-02-21T23:05:25.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.593+0000 7f179ac00640 1 -- 172.21.15.28:0/460164788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1794074ab0 con 0x7f17940c1280 2024-02-21T23:05:25.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.594+0000 7f1790c00640 1 -- 172.21.15.28:0/460164788 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f177c007400 con 0x7f17940c1280 2024-02-21T23:05:25.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.595+0000 7f1790c00640 1 --2- 172.21.15.28:0/460164788 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f177006b4d0 0x7f177006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:25.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.595+0000 7f1790c00640 1 -- 172.21.15.28:0/460164788 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7332+0+0 (secure 0 0 0) 0x7f177c089ce0 con 0x7f17940c1280 2024-02-21T23:05:25.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.596+0000 7f1793e00640 1 --2- 172.21.15.28:0/460164788 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f177006b4d0 0x7f177006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:25.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.597+0000 7f1793e00640 1 --2- 172.21.15.28:0/460164788 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f177006b4d0 0x7f177006d990 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f1784006fd0 tx=0x7f1784008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:25.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.599+0000 7f1790c00640 1 -- 172.21.15.28:0/460164788 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f177c058570 con 0x7f17940c1280 2024-02-21T23:05:25.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.802+0000 7f179ac00640 1 -- 172.21.15.28:0/460164788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f179414c170 con 0x7f17940c1280 2024-02-21T23:05:25.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.805+0000 7f1790c00640 1 -- 172.21.15.28:0/460164788 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v61) v1 ==== 74+0+21144 (secure 0 0 0) 0x7f177c057e70 con 0x7f17940c1280 2024-02-21T23:05:25.794 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:05:25.794 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":61,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:05:25.472851+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:04:51.354130+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-02-21T23:04:52.441690+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs2_metadata","create_time":"2024-02-21T23:05:05.701276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs2_data","create_time":"2024-02-21T23:05:06.904887+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.25","mappings":[{"from":0,"to":6}]},{"pgid":"8.3d","mappings":[{"from":0,"to":6}]},{"pgid":"9.11","mappings":[{"from":7,"to":2}]},{"pgid":"10.19","mappings":[{"from":4,"to":5}]},{"pgid":"10.35","mappings":[{"from":4,"to":3}]},{"pgid":"10.36","mappings":[{"from":7,"to":3}]},{"pgid":"10.37","mappings":[{"from":7,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6835/3468780363":"2024-02-22T23:05:17.415180+0000","172.21.15.28:6833/3468780363":"2024-02-22T23:05:17.415180+0000","172.21.15.28:6834/2803823391":"2024-02-22T23:05:14.385354+0000","172.21.15.28:6832/2803823391":"2024-02-22T23:05:14.385354+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:05:25.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.807+0000 7f179ac00640 1 -- 172.21.15.28:0/460164788 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f177006b4d0 msgr2=0x7f177006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:25.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.807+0000 7f179ac00640 1 --2- 172.21.15.28:0/460164788 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f177006b4d0 0x7f177006d990 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f1784006fd0 tx=0x7f1784008040 comp rx=0 tx=0).stop 2024-02-21T23:05:25.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.807+0000 7f179ac00640 1 -- 172.21.15.28:0/460164788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17940c1280 msgr2=0x7f17940c5b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:25.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.807+0000 7f179ac00640 1 --2- 172.21.15.28:0/460164788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17940c1280 0x7f17940c5b30 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f177c00c430 tx=0x7f177c00c8f0 comp rx=0 tx=0).stop 2024-02-21T23:05:25.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.807+0000 7f179ac00640 1 -- 172.21.15.28:0/460164788 shutdown_connections 2024-02-21T23:05:25.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.807+0000 7f179ac00640 1 --2- 172.21.15.28:0/460164788 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f177006b4d0 0x7f177006d990 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:25.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.807+0000 7f179ac00640 1 --2- 172.21.15.28:0/460164788 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1794155c30 0x7f17941566e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:25.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.807+0000 7f179ac00640 1 --2- 172.21.15.28:0/460164788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17940c1280 0x7f17940c5b30 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:25.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.807+0000 7f179ac00640 1 --2- 172.21.15.28:0/460164788 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f179406f5a0 0x7f1794151590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:25.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.807+0000 7f179ac00640 1 -- 172.21.15.28:0/460164788 >> 172.21.15.28:0/460164788 conn(0x7f17940c3c20 msgr2=0x7f179406dc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:25.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.808+0000 7f179ac00640 1 -- 172.21.15.28:0/460164788 shutdown_connections 2024-02-21T23:05:25.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:25.808+0000 7f179ac00640 1 -- 172.21.15.28:0/460164788 wait complete. 2024-02-21T23:05:25.816 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-02-21T23:05:25.816 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:05:26.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.042+0000 7f69bca00640 1 Processor -- start 2024-02-21T23:05:26.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.042+0000 7f69bca00640 1 -- start start 2024-02-21T23:05:26.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.044+0000 7f69bca00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f69b8152500 0x7f69b81558c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:26.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.044+0000 7f69bca00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f69b806bbf0 0x7f69b806c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:26.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.044+0000 7f69bca00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69b81528e0 con 0x7f69b80ca050 2024-02-21T23:05:26.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.044+0000 7f69bca00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69b80c86e0 con 0x7f69b806bbf0 2024-02-21T23:05:26.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.044+0000 7f69bca00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69b80c8980 con 0x7f69b8152500 2024-02-21T23:05:26.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.044+0000 7f69b5e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f69b80ca050 0x7f69b80bf6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:44028/0 (socket says 172.21.15.28:44028) 2024-02-21T23:05:26.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.044+0000 7f69b5e00640 1 -- 172.21.15.28:0/1052997335 learned_addr learned my addr 172.21.15.28:0/1052997335 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:05:26.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.044+0000 7f69b7200640 1 --2- 172.21.15.28:0/1052997335 >> v2:172.21.15.28:3300/0 conn(0x7f69b806bbf0 0x7f69b806c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:26.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.044+0000 7f69b6800640 1 --2- 172.21.15.28:0/1052997335 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f69b8152500 0x7f69b81558c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:26.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.045+0000 7f69b5400640 1 -- 172.21.15.28:0/1052997335 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3819355902 0 0) 0x7f69b81528e0 con 0x7f69b80ca050 2024-02-21T23:05:26.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.045+0000 7f69b5400640 1 -- 172.21.15.28:0/1052997335 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f699c003520 con 0x7f69b80ca050 2024-02-21T23:05:26.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.045+0000 7f69b7200640 1 -- 172.21.15.28:0/1052997335 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f69b8152500 msgr2=0x7f69b81558c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:26.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.045+0000 7f69b7200640 1 --2- 172.21.15.28:0/1052997335 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f69b8152500 0x7f69b81558c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:26.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.045+0000 7f69b7200640 1 -- 172.21.15.28:0/1052997335 >> v1:172.21.15.28:6789/0 conn(0x7f69b80ca050 legacy=0x7f69b80bf6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:26.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.045+0000 7f69b7200640 1 -- 172.21.15.28:0/1052997335 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69b80c8c60 con 0x7f69b806bbf0 2024-02-21T23:05:26.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.047+0000 7f69b7200640 1 --2- 172.21.15.28:0/1052997335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69b806bbf0 0x7f69b806c040 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f69b0009d90 tx=0x7f69b002be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d3477024a110dd5 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:26.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.047+0000 7f69b5400640 1 -- 172.21.15.28:0/1052997335 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b0039070 con 0x7f69b806bbf0 2024-02-21T23:05:26.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.047+0000 7f69b5400640 1 -- 172.21.15.28:0/1052997335 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69b002cbc0 con 0x7f69b806bbf0 2024-02-21T23:05:26.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.047+0000 7f69b5400640 1 -- 172.21.15.28:0/1052997335 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b0034440 con 0x7f69b806bbf0 2024-02-21T23:05:26.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.048+0000 7f69bca00640 1 -- 172.21.15.28:0/1052997335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69b806bbf0 msgr2=0x7f69b806c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:26.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.048+0000 7f69bca00640 1 --2- 172.21.15.28:0/1052997335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69b806bbf0 0x7f69b806c040 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f69b0009d90 tx=0x7f69b002be10 comp rx=0 tx=0).stop 2024-02-21T23:05:26.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.048+0000 7f69bca00640 1 -- 172.21.15.28:0/1052997335 shutdown_connections 2024-02-21T23:05:26.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.048+0000 7f69bca00640 1 --2- 172.21.15.28:0/1052997335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69b806bbf0 0x7f69b806c040 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:26.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.048+0000 7f69bca00640 1 --2- 172.21.15.28:0/1052997335 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f69b8152500 0x7f69b81558c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:26.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.048+0000 7f69bca00640 1 -- 172.21.15.28:0/1052997335 >> 172.21.15.28:0/1052997335 conn(0x7f69b80ac450 msgr2=0x7f69b80ac860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:26.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.048+0000 7f69bca00640 1 -- 172.21.15.28:0/1052997335 shutdown_connections 2024-02-21T23:05:26.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.048+0000 7f69bca00640 1 -- 172.21.15.28:0/1052997335 wait complete. 2024-02-21T23:05:26.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.049+0000 7f69bca00640 1 Processor -- start 2024-02-21T23:05:26.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.050+0000 7f69bca00640 1 -- start start 2024-02-21T23:05:26.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.050+0000 7f69bca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69b806bbf0 0x7f69b81ede10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:26.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.050+0000 7f69bca00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f69b80ca050 0x7f69b81ee350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:26.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.050+0000 7f69bca00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f69b8152500 0x7f69b8147cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:26.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.050+0000 7f69bca00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69b80bfdd0 con 0x7f69b806bbf0 2024-02-21T23:05:26.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.050+0000 7f69bca00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69b80c9220 con 0x7f69b8152500 2024-02-21T23:05:26.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.050+0000 7f69bca00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69b80bff50 con 0x7f69b80ca050 2024-02-21T23:05:26.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.051+0000 7f69b6800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69b806bbf0 0x7f69b81ede10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:26.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.051+0000 7f69b6800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69b806bbf0 0x7f69b81ede10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40224/0 (socket says 172.21.15.28:40224) 2024-02-21T23:05:26.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.051+0000 7f69b6800640 1 -- 172.21.15.28:0/154566640 learned_addr learned my addr 172.21.15.28:0/154566640 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:26.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.051+0000 7f69b5e00640 1 --2- 172.21.15.28:0/154566640 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f69b80ca050 0x7f69b81ee350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:26.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.051+0000 7f69b7200640 1 --2- 172.21.15.28:0/154566640 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f69b8152500 0x7f69b8147cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:26.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.051+0000 7f69b6800640 1 -- 172.21.15.28:0/154566640 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f69b80ca050 msgr2=0x7f69b81ee350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:26.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.051+0000 7f69b6800640 1 --2- 172.21.15.28:0/154566640 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f69b80ca050 0x7f69b81ee350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:26.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.051+0000 7f69b6800640 1 -- 172.21.15.28:0/154566640 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f69b8152500 msgr2=0x7f69b8147cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:26.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.051+0000 7f69b6800640 1 --2- 172.21.15.28:0/154566640 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f69b8152500 0x7f69b8147cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:26.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.051+0000 7f69b6800640 1 -- 172.21.15.28:0/154566640 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69b80aead0 con 0x7f69b806bbf0 2024-02-21T23:05:26.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.052+0000 7f69b6800640 1 --2- 172.21.15.28:0/154566640 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69b806bbf0 0x7f69b81ede10 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f69a800e8a0 tx=0x7f69a800ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:26.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.052+0000 7f69aea00640 1 -- 172.21.15.28:0/154566640 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69a8017070 con 0x7f69b806bbf0 2024-02-21T23:05:26.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.052+0000 7f69aea00640 1 -- 172.21.15.28:0/154566640 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69a800c940 con 0x7f69b806bbf0 2024-02-21T23:05:26.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.052+0000 7f69aea00640 1 -- 172.21.15.28:0/154566640 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69a800cd30 con 0x7f69b806bbf0 2024-02-21T23:05:26.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.052+0000 7f69bca00640 1 -- 172.21.15.28:0/154566640 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69b81eea70 con 0x7f69b806bbf0 2024-02-21T23:05:26.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.052+0000 7f69bca00640 1 -- 172.21.15.28:0/154566640 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69b814cde0 con 0x7f69b806bbf0 2024-02-21T23:05:26.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.054+0000 7f69bca00640 1 -- 172.21.15.28:0/154566640 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69b804f290 con 0x7f69b806bbf0 2024-02-21T23:05:26.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.054+0000 7f69aea00640 1 -- 172.21.15.28:0/154566640 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f69a801a070 con 0x7f69b806bbf0 2024-02-21T23:05:26.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.055+0000 7f69aea00640 1 --2- 172.21.15.28:0/154566640 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f699806b4d0 0x7f699806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:26.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.055+0000 7f69aea00640 1 -- 172.21.15.28:0/154566640 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7332+0+0 (secure 0 0 0) 0x7f69a808a700 con 0x7f69b806bbf0 2024-02-21T23:05:26.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.056+0000 7f69b5e00640 1 --2- 172.21.15.28:0/154566640 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f699806b4d0 0x7f699806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:26.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.057+0000 7f69b5e00640 1 --2- 172.21.15.28:0/154566640 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f699806b4d0 0x7f699806d990 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f69b814c420 tx=0x7f69a4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:26.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.060+0000 7f69aea00640 1 -- 172.21.15.28:0/154566640 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f69a8058f10 con 0x7f69b806bbf0 2024-02-21T23:05:26.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.305+0000 7f69bca00640 1 -- 172.21.15.28:0/154566640 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f69b80c5160 con 0x7f69b806bbf0 2024-02-21T23:05:26.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.306+0000 7f69aea00640 1 -- 172.21.15.28:0/154566640 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+6176 (secure 0 0 0) 0x7f69a8058810 con 0x7f69b806bbf0 2024-02-21T23:05:26.295 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:05:26.295 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":47,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5221,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":7,"addr":"172.21.15.81:6835/2152160500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2152160500},{"type":"v1","addr":"172.21.15.81:6835","nonce":2152160500}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":6039,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3807390545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3807390545},{"type":"v1","addr":"172.21.15.28:6835","nonce":3807390545}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:04:53.432188+0000","modified":"2024-02-21T23:05:19.431400+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6021},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6021":{"gid":6021,"name":"a","rank":0,"incarnation":44,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6834/3835332438","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3835332438},{"type":"v1","addr":"172.21.15.28:6834","nonce":3835332438}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3},{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:05:08.017586+0000","modified":"2024-02-21T23:05:10.974691+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5224},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5224":{"gid":5224,"name":"d","rank":0,"incarnation":33,"state":"up:active","state_seq":7,"addr":"172.21.15.81:6837/442767245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":442767245},{"type":"v1","addr":"172.21.15.81:6837","nonce":442767245}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-02-21T23:05:26.295 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 47 2024-02-21T23:05:26.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.309+0000 7f69bca00640 1 -- 172.21.15.28:0/154566640 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f699806b4d0 msgr2=0x7f699806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:26.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.309+0000 7f69bca00640 1 --2- 172.21.15.28:0/154566640 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f699806b4d0 0x7f699806d990 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f69b814c420 tx=0x7f69a4009040 comp rx=0 tx=0).stop 2024-02-21T23:05:26.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.309+0000 7f69bca00640 1 -- 172.21.15.28:0/154566640 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69b806bbf0 msgr2=0x7f69b81ede10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:26.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.309+0000 7f69bca00640 1 --2- 172.21.15.28:0/154566640 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69b806bbf0 0x7f69b81ede10 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f69a800e8a0 tx=0x7f69a800ed60 comp rx=0 tx=0).stop 2024-02-21T23:05:26.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.309+0000 7f69bca00640 1 -- 172.21.15.28:0/154566640 shutdown_connections 2024-02-21T23:05:26.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.309+0000 7f69bca00640 1 --2- 172.21.15.28:0/154566640 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f699806b4d0 0x7f699806d990 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:26.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.309+0000 7f69bca00640 1 --2- 172.21.15.28:0/154566640 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f69b80ca050 0x7f69b81ee350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:26.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.309+0000 7f69bca00640 1 --2- 172.21.15.28:0/154566640 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f69b8152500 0x7f69b8147cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:26.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.309+0000 7f69bca00640 1 --2- 172.21.15.28:0/154566640 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69b806bbf0 0x7f69b81ede10 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:26.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.309+0000 7f69bca00640 1 -- 172.21.15.28:0/154566640 >> 172.21.15.28:0/154566640 conn(0x7f69b80ac450 msgr2=0x7f69b81534a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:26.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.309+0000 7f69bca00640 1 -- 172.21.15.28:0/154566640 shutdown_connections 2024-02-21T23:05:26.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.309+0000 7f69bca00640 1 -- 172.21.15.28:0/154566640 wait complete. 2024-02-21T23:05:26.317 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:05:26.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.533+0000 7fbf97200640 1 Processor -- start 2024-02-21T23:05:26.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.533+0000 7fbf97200640 1 -- start start 2024-02-21T23:05:26.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.534+0000 7fbf97200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbf900aff90 0x7fbf900c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:26.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.534+0000 7fbf97200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbf900c5720 0x7fbf9015f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:26.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.534+0000 7fbf97200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf900c74c0 con 0x7fbf9006dbf0 2024-02-21T23:05:26.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.534+0000 7fbf97200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf900c7270 con 0x7fbf900c5720 2024-02-21T23:05:26.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.534+0000 7fbf97200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf90158630 con 0x7fbf900aff90 2024-02-21T23:05:26.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.535+0000 7fbf95400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fbf9006dbf0 0x7fbf90158ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:44044/0 (socket says 172.21.15.28:44044) 2024-02-21T23:05:26.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.535+0000 7fbf95400640 1 -- 172.21.15.28:0/837178155 learned_addr learned my addr 172.21.15.28:0/837178155 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:05:26.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.535+0000 7fbf96800640 1 --2- 172.21.15.28:0/837178155 >> v2:172.21.15.28:3300/0 conn(0x7fbf900c5720 0x7fbf9015f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:26.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.535+0000 7fbf95e00640 1 --2- 172.21.15.28:0/837178155 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbf900aff90 0x7fbf900c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:26.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.535+0000 7fbf94a00640 1 -- 172.21.15.28:0/837178155 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4070404730 0 0) 0x7fbf900c74c0 con 0x7fbf9006dbf0 2024-02-21T23:05:26.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.535+0000 7fbf94a00640 1 -- 172.21.15.28:0/837178155 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf78003520 con 0x7fbf9006dbf0 2024-02-21T23:05:26.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.536+0000 7fbf96800640 1 -- 172.21.15.28:0/837178155 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbf900aff90 msgr2=0x7fbf900c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:26.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.536+0000 7fbf96800640 1 --2- 172.21.15.28:0/837178155 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbf900aff90 0x7fbf900c51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:26.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.536+0000 7fbf96800640 1 -- 172.21.15.28:0/837178155 >> v1:172.21.15.28:6789/0 conn(0x7fbf9006dbf0 legacy=0x7fbf90158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:26.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.536+0000 7fbf96800640 1 -- 172.21.15.28:0/837178155 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf900c77a0 con 0x7fbf900c5720 2024-02-21T23:05:26.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.537+0000 7fbf96800640 1 --2- 172.21.15.28:0/837178155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf900c5720 0x7fbf9015f890 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c009d90 tx=0x7fbf8c02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=893304ad44263948 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:26.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.537+0000 7fbf94a00640 1 -- 172.21.15.28:0/837178155 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf8c039070 con 0x7fbf900c5720 2024-02-21T23:05:26.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.537+0000 7fbf94a00640 1 -- 172.21.15.28:0/837178155 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf8c02cc50 con 0x7fbf900c5720 2024-02-21T23:05:26.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.537+0000 7fbf94a00640 1 -- 172.21.15.28:0/837178155 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf8c034440 con 0x7fbf900c5720 2024-02-21T23:05:26.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.538+0000 7fbf97200640 1 -- 172.21.15.28:0/837178155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf900c5720 msgr2=0x7fbf9015f890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:26.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.538+0000 7fbf97200640 1 --2- 172.21.15.28:0/837178155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf900c5720 0x7fbf9015f890 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c009d90 tx=0x7fbf8c02bea0 comp rx=0 tx=0).stop 2024-02-21T23:05:26.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.538+0000 7fbf97200640 1 -- 172.21.15.28:0/837178155 shutdown_connections 2024-02-21T23:05:26.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.538+0000 7fbf97200640 1 --2- 172.21.15.28:0/837178155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf900c5720 0x7fbf9015f890 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:26.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.538+0000 7fbf97200640 1 --2- 172.21.15.28:0/837178155 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbf900aff90 0x7fbf900c51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:26.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.538+0000 7fbf97200640 1 -- 172.21.15.28:0/837178155 >> 172.21.15.28:0/837178155 conn(0x7fbf900ac370 msgr2=0x7fbf900ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:26.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.538+0000 7fbf97200640 1 -- 172.21.15.28:0/837178155 shutdown_connections 2024-02-21T23:05:26.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.538+0000 7fbf97200640 1 -- 172.21.15.28:0/837178155 wait complete. 2024-02-21T23:05:26.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.540+0000 7fbf97200640 1 Processor -- start 2024-02-21T23:05:26.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.540+0000 7fbf97200640 1 -- start start 2024-02-21T23:05:26.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.541+0000 7fbf97200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf9006dbf0 0x7fbf901ee3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:26.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.541+0000 7fbf97200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbf900aff90 0x7fbf900c2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:26.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.541+0000 7fbf97200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf900c5720 0x7fbf900ae740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:26.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.541+0000 7fbf97200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf900c8760 con 0x7fbf900c5720 2024-02-21T23:05:26.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.541+0000 7fbf97200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf900be0c0 con 0x7fbf9006dbf0 2024-02-21T23:05:26.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.541+0000 7fbf97200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf900c88e0 con 0x7fbf900aff90 2024-02-21T23:05:26.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.541+0000 7fbf96800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf900c5720 0x7fbf900ae740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:26.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.541+0000 7fbf96800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf900c5720 0x7fbf900ae740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40228/0 (socket says 172.21.15.28:40228) 2024-02-21T23:05:26.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.541+0000 7fbf96800640 1 -- 172.21.15.28:0/2741483002 learned_addr learned my addr 172.21.15.28:0/2741483002 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:26.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.541+0000 7fbf95400640 1 --2- 172.21.15.28:0/2741483002 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbf900aff90 0x7fbf900c2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:26.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.541+0000 7fbf95e00640 1 --2- 172.21.15.28:0/2741483002 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf9006dbf0 0x7fbf901ee3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:26.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.542+0000 7fbf96800640 1 -- 172.21.15.28:0/2741483002 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbf900aff90 msgr2=0x7fbf900c2000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:26.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.542+0000 7fbf96800640 1 --2- 172.21.15.28:0/2741483002 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbf900aff90 0x7fbf900c2000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:26.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.542+0000 7fbf96800640 1 -- 172.21.15.28:0/2741483002 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf9006dbf0 msgr2=0x7fbf901ee3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:26.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.542+0000 7fbf96800640 1 --2- 172.21.15.28:0/2741483002 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf9006dbf0 0x7fbf901ee3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:26.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.542+0000 7fbf96800640 1 -- 172.21.15.28:0/2741483002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf900af220 con 0x7fbf900c5720 2024-02-21T23:05:26.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.542+0000 7fbf95e00640 1 --2- 172.21.15.28:0/2741483002 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf9006dbf0 0x7fbf901ee3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:05:26.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.542+0000 7fbf96800640 1 --2- 172.21.15.28:0/2741483002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf900c5720 0x7fbf900ae740 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c009eb0 tx=0x7fbf8c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:26.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.542+0000 7fbf8a000640 1 -- 172.21.15.28:0/2741483002 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf8c044020 con 0x7fbf900c5720 2024-02-21T23:05:26.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.542+0000 7fbf8a000640 1 -- 172.21.15.28:0/2741483002 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf8c02ca50 con 0x7fbf900c5720 2024-02-21T23:05:26.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.542+0000 7fbf8a000640 1 -- 172.21.15.28:0/2741483002 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf8c039040 con 0x7fbf900c5720 2024-02-21T23:05:26.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.542+0000 7fbf97200640 1 -- 172.21.15.28:0/2741483002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf901ee930 con 0x7fbf900c5720 2024-02-21T23:05:26.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.542+0000 7fbf97200640 1 -- 172.21.15.28:0/2741483002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf900c37e0 con 0x7fbf900c5720 2024-02-21T23:05:26.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.544+0000 7fbf97200640 1 -- 172.21.15.28:0/2741483002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf9004f290 con 0x7fbf900c5720 2024-02-21T23:05:26.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.545+0000 7fbf8a000640 1 -- 172.21.15.28:0/2741483002 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbf8c047020 con 0x7fbf900c5720 2024-02-21T23:05:26.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.546+0000 7fbf8a000640 1 --2- 172.21.15.28:0/2741483002 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbf7406b520 0x7fbf7406d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:26.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.546+0000 7fbf8a000640 1 -- 172.21.15.28:0/2741483002 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7332+0+0 (secure 0 0 0) 0x7fbf8c0acfd0 con 0x7fbf900c5720 2024-02-21T23:05:26.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.546+0000 7fbf95e00640 1 --2- 172.21.15.28:0/2741483002 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbf7406b520 0x7fbf7406d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:26.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.547+0000 7fbf95e00640 1 --2- 172.21.15.28:0/2741483002 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbf7406b520 0x7fbf7406d9e0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fbf84009ac0 tx=0x7fbf84008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:26.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.555+0000 7fbf8a000640 1 -- 172.21.15.28:0/2741483002 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fbf8c04e030 con 0x7fbf900c5720 2024-02-21T23:05:26.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.794+0000 7fbf97200640 1 -- 172.21.15.28:0/2741483002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf9006d9d0 con 0x7fbf900c5720 2024-02-21T23:05:26.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.795+0000 7fbf8a000640 1 -- 172.21.15.28:0/2741483002 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+6176 (secure 0 0 0) 0x7fbf8c07b990 con 0x7fbf900c5720 2024-02-21T23:05:26.784 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:05:26.784 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":47,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5221,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":7,"addr":"172.21.15.81:6835/2152160500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2152160500},{"type":"v1","addr":"172.21.15.81:6835","nonce":2152160500}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":6039,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3807390545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3807390545},{"type":"v1","addr":"172.21.15.28:6835","nonce":3807390545}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:04:53.432188+0000","modified":"2024-02-21T23:05:19.431400+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6021},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6021":{"gid":6021,"name":"a","rank":0,"incarnation":44,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6834/3835332438","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3835332438},{"type":"v1","addr":"172.21.15.28:6834","nonce":3835332438}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3},{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:05:08.017586+0000","modified":"2024-02-21T23:05:10.974691+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5224},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5224":{"gid":5224,"name":"d","rank":0,"incarnation":33,"state":"up:active","state_seq":7,"addr":"172.21.15.81:6837/442767245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":442767245},{"type":"v1","addr":"172.21.15.81:6837","nonce":442767245}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-02-21T23:05:26.784 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 47 2024-02-21T23:05:26.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.798+0000 7fbf97200640 1 -- 172.21.15.28:0/2741483002 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbf7406b520 msgr2=0x7fbf7406d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:26.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.798+0000 7fbf97200640 1 --2- 172.21.15.28:0/2741483002 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbf7406b520 0x7fbf7406d9e0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fbf84009ac0 tx=0x7fbf84008040 comp rx=0 tx=0).stop 2024-02-21T23:05:26.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.798+0000 7fbf97200640 1 -- 172.21.15.28:0/2741483002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf900c5720 msgr2=0x7fbf900ae740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:26.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.798+0000 7fbf97200640 1 --2- 172.21.15.28:0/2741483002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf900c5720 0x7fbf900ae740 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c009eb0 tx=0x7fbf8c004a40 comp rx=0 tx=0).stop 2024-02-21T23:05:26.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.798+0000 7fbf97200640 1 -- 172.21.15.28:0/2741483002 shutdown_connections 2024-02-21T23:05:26.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.798+0000 7fbf97200640 1 --2- 172.21.15.28:0/2741483002 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbf7406b520 0x7fbf7406d9e0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:26.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.798+0000 7fbf97200640 1 --2- 172.21.15.28:0/2741483002 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbf900aff90 0x7fbf900c2000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:26.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.798+0000 7fbf97200640 1 --2- 172.21.15.28:0/2741483002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf900c5720 0x7fbf900ae740 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:26.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.798+0000 7fbf97200640 1 --2- 172.21.15.28:0/2741483002 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf9006dbf0 0x7fbf901ee3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:26.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.798+0000 7fbf97200640 1 -- 172.21.15.28:0/2741483002 >> 172.21.15.28:0/2741483002 conn(0x7fbf900ac370 msgr2=0x7fbf900ac750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:26.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.799+0000 7fbf97200640 1 -- 172.21.15.28:0/2741483002 shutdown_connections 2024-02-21T23:05:26.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:26.799+0000 7fbf97200640 1 -- 172.21.15.28:0/2741483002 wait complete. 2024-02-21T23:05:26.805 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:05:27.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.023+0000 7f9ffa200640 1 Processor -- start 2024-02-21T23:05:27.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.023+0000 7f9ffa200640 1 -- start start 2024-02-21T23:05:27.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.025+0000 7f9ffa200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9ff40a9930 0x7f9ff40a6f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:27.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.025+0000 7f9ffa200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f9ff40c1a40 0x7f9ff4152bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:27.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.025+0000 7f9ffa200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9ff40c2120 0x7f9ff40c25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:27.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.025+0000 7f9ffa200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ff4156c50 con 0x7f9ff40c2120 2024-02-21T23:05:27.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.025+0000 7f9ffa200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ff4072a40 con 0x7f9ff40c1a40 2024-02-21T23:05:27.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.025+0000 7f9ffa200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff4156ef0 con 0x7f9ff40a9930 2024-02-21T23:05:27.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.025+0000 7f9ff9800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9ff40c2120 0x7f9ff40c25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:27.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.025+0000 7f9ff9800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9ff40c2120 0x7f9ff40c25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40240/0 (socket says 172.21.15.28:40240) 2024-02-21T23:05:27.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.025+0000 7f9ff9800640 1 -- 172.21.15.28:0/2245198751 learned_addr learned my addr 172.21.15.28:0/2245198751 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:27.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.026+0000 7f9ff3e00640 1 --2- 172.21.15.28:0/2245198751 >> v2:172.21.15.81:3300/0 conn(0x7f9ff40c1a40 0x7f9ff4152bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:27.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.026+0000 7f9ff8e00640 1 --2- 172.21.15.28:0/2245198751 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9ff40a9930 0x7f9ff40a6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:27.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.026+0000 7f9ff9800640 1 -- 172.21.15.28:0/2245198751 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9ff40a9930 msgr2=0x7f9ff40a6f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:27.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.026+0000 7f9ff9800640 1 --2- 172.21.15.28:0/2245198751 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9ff40a9930 0x7f9ff40a6f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:27.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.026+0000 7f9ff9800640 1 -- 172.21.15.28:0/2245198751 >> v2:172.21.15.81:3300/0 conn(0x7f9ff40c1a40 msgr2=0x7f9ff4152bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:27.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.026+0000 7f9ff9800640 1 --2- 172.21.15.28:0/2245198751 >> v2:172.21.15.81:3300/0 conn(0x7f9ff40c1a40 0x7f9ff4152bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:27.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.026+0000 7f9ff9800640 1 -- 172.21.15.28:0/2245198751 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ff4073450 con 0x7f9ff40c2120 2024-02-21T23:05:27.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.027+0000 7f9ff9800640 1 --2- 172.21.15.28:0/2245198751 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ff40c2120 0x7f9ff40c25b0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f9ff40aa590 tx=0x7f9fec02b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da88c55aa88a0df5 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:27.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.027+0000 7f9ff3400640 1 -- 172.21.15.28:0/2245198751 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fec038070 con 0x7f9ff40c2120 2024-02-21T23:05:27.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.027+0000 7f9ff3400640 1 -- 172.21.15.28:0/2245198751 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fec02f050 con 0x7f9ff40c2120 2024-02-21T23:05:27.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.027+0000 7f9ff3400640 1 -- 172.21.15.28:0/2245198751 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fec031970 con 0x7f9ff40c2120 2024-02-21T23:05:27.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.028+0000 7f9ffa200640 1 -- 172.21.15.28:0/2245198751 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ff40c2120 msgr2=0x7f9ff40c25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:27.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.028+0000 7f9ffa200640 1 --2- 172.21.15.28:0/2245198751 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ff40c2120 0x7f9ff40c25b0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f9ff40aa590 tx=0x7f9fec02b7f0 comp rx=0 tx=0).stop 2024-02-21T23:05:27.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.028+0000 7f9ffa200640 1 -- 172.21.15.28:0/2245198751 shutdown_connections 2024-02-21T23:05:27.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.028+0000 7f9ffa200640 1 --2- 172.21.15.28:0/2245198751 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ff40c2120 0x7f9ff40c25b0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:27.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.028+0000 7f9ffa200640 1 --2- 172.21.15.28:0/2245198751 >> v2:172.21.15.81:3300/0 conn(0x7f9ff40c1a40 0x7f9ff4152bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:27.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.028+0000 7f9ffa200640 1 --2- 172.21.15.28:0/2245198751 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9ff40a9930 0x7f9ff40a6f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:27.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.028+0000 7f9ffa200640 1 -- 172.21.15.28:0/2245198751 >> 172.21.15.28:0/2245198751 conn(0x7f9ff40c7720 msgr2=0x7f9ff40c7b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:27.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.028+0000 7f9ffa200640 1 -- 172.21.15.28:0/2245198751 shutdown_connections 2024-02-21T23:05:27.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.028+0000 7f9ffa200640 1 -- 172.21.15.28:0/2245198751 wait complete. 2024-02-21T23:05:27.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.030+0000 7f9ffa200640 1 Processor -- start 2024-02-21T23:05:27.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.030+0000 7f9ffa200640 1 -- start start 2024-02-21T23:05:27.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.031+0000 7f9ffa200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9ff40a9930 0x7f9ff41ebd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:27.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.031+0000 7f9ffa200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ff40c1a40 0x7f9ff41ec250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:27.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.031+0000 7f9ffa200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9ff40c2120 0x7f9ff40ac0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:27.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.031+0000 7f9ffa200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff414fea0 con 0x7f9ff40c1a40 2024-02-21T23:05:27.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.031+0000 7f9ffa200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff40a9f20 con 0x7f9ff40c2120 2024-02-21T23:05:27.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.031+0000 7f9ffa200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff4150020 con 0x7f9ff40a9930 2024-02-21T23:05:27.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.031+0000 7f9ff3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ff40c1a40 0x7f9ff41ec250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:27.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.031+0000 7f9ff3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ff40c1a40 0x7f9ff41ec250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40252/0 (socket says 172.21.15.28:40252) 2024-02-21T23:05:27.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.031+0000 7f9ff3e00640 1 -- 172.21.15.28:0/2426722104 learned_addr learned my addr 172.21.15.28:0/2426722104 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:27.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.031+0000 7f9ff8e00640 1 --2- 172.21.15.28:0/2426722104 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9ff40a9930 0x7f9ff41ebd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:27.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.031+0000 7f9ff9800640 1 --2- 172.21.15.28:0/2426722104 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9ff40c2120 0x7f9ff40ac0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:27.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.032+0000 7f9ff3e00640 1 -- 172.21.15.28:0/2426722104 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9ff40a9930 msgr2=0x7f9ff41ebd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:27.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.032+0000 7f9ff3e00640 1 --2- 172.21.15.28:0/2426722104 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9ff40a9930 0x7f9ff41ebd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:27.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.032+0000 7f9ff3e00640 1 -- 172.21.15.28:0/2426722104 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9ff40c2120 msgr2=0x7f9ff40ac0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:27.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.032+0000 7f9ff3e00640 1 --2- 172.21.15.28:0/2426722104 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9ff40c2120 0x7f9ff40ac0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:27.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.032+0000 7f9ff3e00640 1 -- 172.21.15.28:0/2426722104 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ff40aa9b0 con 0x7f9ff40c1a40 2024-02-21T23:05:27.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.032+0000 7f9ff3e00640 1 --2- 172.21.15.28:0/2426722104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ff40c1a40 0x7f9ff41ec250 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f9fe0007ef0 tx=0x7f9fe000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:27.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.032+0000 7f9ff0c00640 1 -- 172.21.15.28:0/2426722104 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fe0014620 con 0x7f9ff40c1a40 2024-02-21T23:05:27.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.032+0000 7f9ff0c00640 1 -- 172.21.15.28:0/2426722104 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fe0019070 con 0x7f9ff40c1a40 2024-02-21T23:05:27.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.032+0000 7f9ff0c00640 1 -- 172.21.15.28:0/2426722104 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fe0014a00 con 0x7f9ff40c1a40 2024-02-21T23:05:27.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.032+0000 7f9ffa200640 1 -- 172.21.15.28:0/2426722104 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ff40aac90 con 0x7f9ff40c1a40 2024-02-21T23:05:27.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.033+0000 7f9ffa200640 1 -- 172.21.15.28:0/2426722104 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ff414ccc0 con 0x7f9ff40c1a40 2024-02-21T23:05:27.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.034+0000 7f9ffa200640 1 -- 172.21.15.28:0/2426722104 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ff404f290 con 0x7f9ff40c1a40 2024-02-21T23:05:27.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.036+0000 7f9ff0c00640 1 -- 172.21.15.28:0/2426722104 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9fe0014ba0 con 0x7f9ff40c1a40 2024-02-21T23:05:27.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.037+0000 7f9ff0c00640 1 --2- 172.21.15.28:0/2426722104 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9fd406b500 0x7f9fd406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:27.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.037+0000 7f9ff0c00640 1 -- 172.21.15.28:0/2426722104 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7332+0+0 (secure 0 0 0) 0x7f9fe008b240 con 0x7f9ff40c1a40 2024-02-21T23:05:27.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.037+0000 7f9ff8e00640 1 --2- 172.21.15.28:0/2426722104 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9fd406b500 0x7f9fd406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:27.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.038+0000 7f9ff8e00640 1 --2- 172.21.15.28:0/2426722104 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9fd406b500 0x7f9fd406d9c0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f9fe4006fd0 tx=0x7f9fe4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:27.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.041+0000 7f9ff0c00640 1 -- 172.21.15.28:0/2426722104 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f9fe0059930 con 0x7f9ff40c1a40 2024-02-21T23:05:27.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.254+0000 7f9ffa200640 1 -- 172.21.15.28:0/2426722104 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9ff40c3500 con 0x7f9ff40c1a40 2024-02-21T23:05:27.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.258+0000 7f9ff0c00640 1 -- 172.21.15.28:0/2426722104 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v61) v1 ==== 74+0+21144 (secure 0 0 0) 0x7f9fe0059230 con 0x7f9ff40c1a40 2024-02-21T23:05:27.247 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:05:27.248 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":61,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:05:25.472851+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:04:51.354130+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-02-21T23:04:52.441690+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs2_metadata","create_time":"2024-02-21T23:05:05.701276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs2_data","create_time":"2024-02-21T23:05:06.904887+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.25","mappings":[{"from":0,"to":6}]},{"pgid":"8.3d","mappings":[{"from":0,"to":6}]},{"pgid":"9.11","mappings":[{"from":7,"to":2}]},{"pgid":"10.19","mappings":[{"from":4,"to":5}]},{"pgid":"10.35","mappings":[{"from":4,"to":3}]},{"pgid":"10.36","mappings":[{"from":7,"to":3}]},{"pgid":"10.37","mappings":[{"from":7,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6835/3468780363":"2024-02-22T23:05:17.415180+0000","172.21.15.28:6833/3468780363":"2024-02-22T23:05:17.415180+0000","172.21.15.28:6834/2803823391":"2024-02-22T23:05:14.385354+0000","172.21.15.28:6832/2803823391":"2024-02-22T23:05:14.385354+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:05:27.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.261+0000 7f9ffa200640 1 -- 172.21.15.28:0/2426722104 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9fd406b500 msgr2=0x7f9fd406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:27.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.261+0000 7f9ffa200640 1 --2- 172.21.15.28:0/2426722104 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9fd406b500 0x7f9fd406d9c0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f9fe4006fd0 tx=0x7f9fe4008040 comp rx=0 tx=0).stop 2024-02-21T23:05:27.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.261+0000 7f9ffa200640 1 -- 172.21.15.28:0/2426722104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ff40c1a40 msgr2=0x7f9ff41ec250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:27.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.261+0000 7f9ffa200640 1 --2- 172.21.15.28:0/2426722104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ff40c1a40 0x7f9ff41ec250 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f9fe0007ef0 tx=0x7f9fe000e4b0 comp rx=0 tx=0).stop 2024-02-21T23:05:27.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.262+0000 7f9ffa200640 1 -- 172.21.15.28:0/2426722104 shutdown_connections 2024-02-21T23:05:27.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.262+0000 7f9ffa200640 1 --2- 172.21.15.28:0/2426722104 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9fd406b500 0x7f9fd406d9c0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:27.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.262+0000 7f9ffa200640 1 --2- 172.21.15.28:0/2426722104 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9ff40c2120 0x7f9ff40ac0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:27.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.262+0000 7f9ffa200640 1 --2- 172.21.15.28:0/2426722104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ff40c1a40 0x7f9ff41ec250 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:27.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.262+0000 7f9ffa200640 1 --2- 172.21.15.28:0/2426722104 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9ff40a9930 0x7f9ff41ebd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:27.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.262+0000 7f9ffa200640 1 -- 172.21.15.28:0/2426722104 >> 172.21.15.28:0/2426722104 conn(0x7f9ff40c7720 msgr2=0x7f9ff4074360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:27.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.262+0000 7f9ffa200640 1 -- 172.21.15.28:0/2426722104 shutdown_connections 2024-02-21T23:05:27.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.262+0000 7f9ffa200640 1 -- 172.21.15.28:0/2426722104 wait complete. 2024-02-21T23:05:27.268 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-02-21T23:05:27.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.509+0000 7fcc12a00640 1 Processor -- start 2024-02-21T23:05:27.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.510+0000 7fcc12a00640 1 -- start start 2024-02-21T23:05:27.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.511+0000 7fcc12a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcc0c152550 0x7fcc0c0729c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:27.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.511+0000 7fcc12a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcc0c0ac500 0x7fcc0c155a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:27.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.511+0000 7fcc12a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc0c155f70 con 0x7fcc0c0c0550 2024-02-21T23:05:27.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.511+0000 7fcc12a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc0c072f00 con 0x7fcc0c152550 2024-02-21T23:05:27.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.511+0000 7fcc12a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0c06fb10 con 0x7fcc0c0ac500 2024-02-21T23:05:27.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.512+0000 7fcc10a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fcc0c0c0550 0x7fcc0c066f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:44046/0 (socket says 172.21.15.28:44046) 2024-02-21T23:05:27.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.512+0000 7fcc0be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcc0c152550 0x7fcc0c0729c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:27.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.512+0000 7fcc10a00640 1 -- 172.21.15.28:0/1498698832 learned_addr learned my addr 172.21.15.28:0/1498698832 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:05:27.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.512+0000 7fcc03e00640 1 --2- 172.21.15.28:0/1498698832 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcc0c0ac500 0x7fcc0c155a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:27.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.512+0000 7fcc0be00640 1 -- 172.21.15.28:0/1498698832 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcc0c0ac500 msgr2=0x7fcc0c155a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:27.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.512+0000 7fcc0be00640 1 --2- 172.21.15.28:0/1498698832 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcc0c0ac500 0x7fcc0c155a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:27.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.512+0000 7fcc0be00640 1 -- 172.21.15.28:0/1498698832 >> v1:172.21.15.28:6789/0 conn(0x7fcc0c0c0550 legacy=0x7fcc0c066f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:27.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.513+0000 7fcc0be00640 1 -- 172.21.15.28:0/1498698832 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc0c06c2e0 con 0x7fcc0c152550 2024-02-21T23:05:27.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.514+0000 7fcc0be00640 1 --2- 172.21.15.28:0/1498698832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc0c152550 0x7fcc0c0729c0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c06fc90 tx=0x7fcbfc02b650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fec2b88548f895 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:27.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.514+0000 7fcc0b400640 1 -- 172.21.15.28:0/1498698832 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbfc037070 con 0x7fcc0c152550 2024-02-21T23:05:27.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.514+0000 7fcc0b400640 1 -- 172.21.15.28:0/1498698832 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbfc02be50 con 0x7fcc0c152550 2024-02-21T23:05:27.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.514+0000 7fcc0b400640 1 -- 172.21.15.28:0/1498698832 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbfc030930 con 0x7fcc0c152550 2024-02-21T23:05:27.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.515+0000 7fcc12a00640 1 -- 172.21.15.28:0/1498698832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc0c152550 msgr2=0x7fcc0c0729c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:27.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.515+0000 7fcc12a00640 1 --2- 172.21.15.28:0/1498698832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc0c152550 0x7fcc0c0729c0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c06fc90 tx=0x7fcbfc02b650 comp rx=0 tx=0).stop 2024-02-21T23:05:27.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.515+0000 7fcc12a00640 1 -- 172.21.15.28:0/1498698832 shutdown_connections 2024-02-21T23:05:27.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.515+0000 7fcc12a00640 1 --2- 172.21.15.28:0/1498698832 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcc0c0ac500 0x7fcc0c155a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:27.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.515+0000 7fcc12a00640 1 --2- 172.21.15.28:0/1498698832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc0c152550 0x7fcc0c0729c0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:27.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.515+0000 7fcc12a00640 1 -- 172.21.15.28:0/1498698832 >> 172.21.15.28:0/1498698832 conn(0x7fcc0c071bb0 msgr2=0x7fcc0c071f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:27.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.515+0000 7fcc12a00640 1 -- 172.21.15.28:0/1498698832 shutdown_connections 2024-02-21T23:05:27.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.515+0000 7fcc12a00640 1 -- 172.21.15.28:0/1498698832 wait complete. 2024-02-21T23:05:27.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.517+0000 7fcc12a00640 1 Processor -- start 2024-02-21T23:05:27.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.517+0000 7fcc12a00640 1 -- start start 2024-02-21T23:05:27.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.518+0000 7fcc12a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcc0c0ac500 0x7fcc0c14d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:27.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.518+0000 7fcc12a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc0c0c0550 0x7fcc0c14c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:27.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.518+0000 7fcc12a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc0c152550 0x7fcc0c146870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:27.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.518+0000 7fcc12a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0c152930 con 0x7fcc0c152550 2024-02-21T23:05:27.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.518+0000 7fcc12a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0c06f7a0 con 0x7fcc0c0c0550 2024-02-21T23:05:27.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.518+0000 7fcc12a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0c0c0a00 con 0x7fcc0c0ac500 2024-02-21T23:05:27.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.518+0000 7fcc10a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc0c152550 0x7fcc0c146870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:27.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.518+0000 7fcc10a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc0c152550 0x7fcc0c146870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40272/0 (socket says 172.21.15.28:40272) 2024-02-21T23:05:27.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.518+0000 7fcc10a00640 1 -- 172.21.15.28:0/581166533 learned_addr learned my addr 172.21.15.28:0/581166533 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:27.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.518+0000 7fcc0be00640 1 --2- 172.21.15.28:0/581166533 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcc0c0ac500 0x7fcc0c14d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:27.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.518+0000 7fcc03e00640 1 --2- 172.21.15.28:0/581166533 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc0c0c0550 0x7fcc0c14c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:27.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.518+0000 7fcc10a00640 1 -- 172.21.15.28:0/581166533 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcc0c0ac500 msgr2=0x7fcc0c14d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:27.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.518+0000 7fcc10a00640 1 --2- 172.21.15.28:0/581166533 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcc0c0ac500 0x7fcc0c14d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:27.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.518+0000 7fcc10a00640 1 -- 172.21.15.28:0/581166533 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc0c0c0550 msgr2=0x7fcc0c14c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:27.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.518+0000 7fcc10a00640 1 --2- 172.21.15.28:0/581166533 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc0c0c0550 0x7fcc0c14c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:27.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.518+0000 7fcc10a00640 1 -- 172.21.15.28:0/581166533 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc0c1ebea0 con 0x7fcc0c152550 2024-02-21T23:05:27.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.519+0000 7fcc10a00640 1 --2- 172.21.15.28:0/581166533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc0c152550 0x7fcc0c146870 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fcc0400cb20 tx=0x7fcc040020f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:27.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.519+0000 7fcc08c00640 1 -- 172.21.15.28:0/581166533 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc04018070 con 0x7fcc0c152550 2024-02-21T23:05:27.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.519+0000 7fcc08c00640 1 -- 172.21.15.28:0/581166533 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc0400a760 con 0x7fcc0c152550 2024-02-21T23:05:27.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.519+0000 7fcc12a00640 1 -- 172.21.15.28:0/581166533 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc0c14c060 con 0x7fcc0c152550 2024-02-21T23:05:27.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.519+0000 7fcc08c00640 1 -- 172.21.15.28:0/581166533 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc0400ab30 con 0x7fcc0c152550 2024-02-21T23:05:27.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.519+0000 7fcc12a00640 1 -- 172.21.15.28:0/581166533 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc0c14da80 con 0x7fcc0c152550 2024-02-21T23:05:27.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.521+0000 7fcc12a00640 1 -- 172.21.15.28:0/581166533 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc0c04f370 con 0x7fcc0c152550 2024-02-21T23:05:27.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.521+0000 7fcc08c00640 1 -- 172.21.15.28:0/581166533 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcc0401b070 con 0x7fcc0c152550 2024-02-21T23:05:27.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.522+0000 7fcc08c00640 1 --2- 172.21.15.28:0/581166533 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcbe806b4d0 0x7fcbe806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:27.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.522+0000 7fcc08c00640 1 -- 172.21.15.28:0/581166533 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7332+0+0 (secure 0 0 0) 0x7fcc0408a030 con 0x7fcc0c152550 2024-02-21T23:05:27.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.523+0000 7fcc0be00640 1 --2- 172.21.15.28:0/581166533 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcbe806b4d0 0x7fcbe806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:27.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.524+0000 7fcc0be00640 1 --2- 172.21.15.28:0/581166533 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcbe806b4d0 0x7fcbe806d990 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc034000 tx=0x7fcbfc008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:27.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.527+0000 7fcc08c00640 1 -- 172.21.15.28:0/581166533 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fcc040588e0 con 0x7fcc0c152550 2024-02-21T23:05:27.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.770+0000 7fcc12a00640 1 -- 172.21.15.28:0/581166533 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fcc0c0726a0 con 0x7fcc0c152550 2024-02-21T23:05:27.760 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-02-21T23:05:27.772+0000 7f5ddc400640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-21T23:05:27.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.777+0000 7fcc08c00640 1 -- 172.21.15.28:0/581166533 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v48) v1 ==== 161+0+0 (secure 0 0 0) 0x7fcc040581e0 con 0x7fcc0c152550 2024-02-21T23:05:27.766 INFO:teuthology.orchestra.run.smithi028.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-02-21T23:05:27.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.782+0000 7fcc12a00640 1 -- 172.21.15.28:0/581166533 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcbe806b4d0 msgr2=0x7fcbe806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:27.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.782+0000 7fcc12a00640 1 --2- 172.21.15.28:0/581166533 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcbe806b4d0 0x7fcbe806d990 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc034000 tx=0x7fcbfc008030 comp rx=0 tx=0).stop 2024-02-21T23:05:27.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.782+0000 7fcc12a00640 1 -- 172.21.15.28:0/581166533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc0c152550 msgr2=0x7fcc0c146870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:27.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.782+0000 7fcc12a00640 1 --2- 172.21.15.28:0/581166533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc0c152550 0x7fcc0c146870 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fcc0400cb20 tx=0x7fcc040020f0 comp rx=0 tx=0).stop 2024-02-21T23:05:27.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.782+0000 7fcc12a00640 1 -- 172.21.15.28:0/581166533 shutdown_connections 2024-02-21T23:05:27.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.782+0000 7fcc12a00640 1 --2- 172.21.15.28:0/581166533 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcbe806b4d0 0x7fcbe806d990 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:27.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.782+0000 7fcc12a00640 1 --2- 172.21.15.28:0/581166533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc0c152550 0x7fcc0c146870 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:27.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.782+0000 7fcc12a00640 1 --2- 172.21.15.28:0/581166533 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc0c0c0550 0x7fcc0c14c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:27.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.782+0000 7fcc12a00640 1 --2- 172.21.15.28:0/581166533 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcc0c0ac500 0x7fcc0c14d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:27.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.782+0000 7fcc12a00640 1 -- 172.21.15.28:0/581166533 >> 172.21.15.28:0/581166533 conn(0x7fcc0c071bb0 msgr2=0x7fcc0c0c5a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:27.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.782+0000 7fcc12a00640 1 -- 172.21.15.28:0/581166533 shutdown_connections 2024-02-21T23:05:27.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.782+0000 7fcc12a00640 1 -- 172.21.15.28:0/581166533 wait complete. 2024-02-21T23:05:27.774 INFO:tasks.ceph.mds.a.smithi028.stderr: -103> 2024-02-21T23:05:25.413+0000 7f0a2fe00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-21T23:05:27.789 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-02-21T23:05:27.817 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-02-21T23:05:27.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.985+0000 7fe6d0e00640 1 Processor -- start 2024-02-21T23:05:27.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.985+0000 7fe6d0e00640 1 -- start start 2024-02-21T23:05:27.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.986+0000 7fe6d0e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe6cc1500f0 0x7fe6cc06ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:27.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.987+0000 7fe6d0e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe6cc0c2c60 0x7fe6cc06dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:27.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.987+0000 7fe6d0e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6cc06db10 con 0x7fe6cc06b3b0 2024-02-21T23:05:27.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.987+0000 7fe6d0e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6cc0c1080 con 0x7fe6cc1500f0 2024-02-21T23:05:27.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.987+0000 7fe6d0e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6cc1539a0 con 0x7fe6cc0c2c60 2024-02-21T23:05:27.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.987+0000 7fe6cbe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe6cc06b3b0 0x7fe6cc06e530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:44058/0 (socket says 172.21.15.28:44058) 2024-02-21T23:05:27.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.987+0000 7fe6cbe00640 1 -- 172.21.15.28:0/1376223964 learned_addr learned my addr 172.21.15.28:0/1376223964 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:05:27.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.987+0000 7fe6cb400640 1 --2- 172.21.15.28:0/1376223964 >> v2:172.21.15.28:3300/0 conn(0x7fe6cc1500f0 0x7fe6cc06ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:27.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.987+0000 7fe6caa00640 1 --2- 172.21.15.28:0/1376223964 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe6cc0c2c60 0x7fe6cc06dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:27.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.988+0000 7fe6ca000640 1 -- 172.21.15.28:0/1376223964 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1557872709 0 0) 0x7fe6cc06db10 con 0x7fe6cc06b3b0 2024-02-21T23:05:27.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.988+0000 7fe6ca000640 1 -- 172.21.15.28:0/1376223964 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6b0003520 con 0x7fe6cc06b3b0 2024-02-21T23:05:27.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.988+0000 7fe6cb400640 1 -- 172.21.15.28:0/1376223964 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe6cc0c2c60 msgr2=0x7fe6cc06dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:27.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.988+0000 7fe6cb400640 1 --2- 172.21.15.28:0/1376223964 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe6cc0c2c60 0x7fe6cc06dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:27.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.988+0000 7fe6cb400640 1 -- 172.21.15.28:0/1376223964 >> v1:172.21.15.28:6789/0 conn(0x7fe6cc06b3b0 legacy=0x7fe6cc06e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:27.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.988+0000 7fe6cb400640 1 -- 172.21.15.28:0/1376223964 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6cc0c12d0 con 0x7fe6cc1500f0 2024-02-21T23:05:27.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.989+0000 7fe6cb400640 1 --2- 172.21.15.28:0/1376223964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6cc1500f0 0x7fe6cc06ae70 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc0097a0 tx=0x7fe6bc02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ed38fa3aba9d946 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:27.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.989+0000 7fe6ca000640 1 -- 172.21.15.28:0/1376223964 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6bc035020 con 0x7fe6cc1500f0 2024-02-21T23:05:27.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.989+0000 7fe6ca000640 1 -- 172.21.15.28:0/1376223964 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6bc031810 con 0x7fe6cc1500f0 2024-02-21T23:05:27.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.989+0000 7fe6ca000640 1 -- 172.21.15.28:0/1376223964 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6bc031bc0 con 0x7fe6cc1500f0 2024-02-21T23:05:27.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.990+0000 7fe6d0e00640 1 -- 172.21.15.28:0/1376223964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6cc1500f0 msgr2=0x7fe6cc06ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:27.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.990+0000 7fe6d0e00640 1 --2- 172.21.15.28:0/1376223964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6cc1500f0 0x7fe6cc06ae70 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc0097a0 tx=0x7fe6bc02b8b0 comp rx=0 tx=0).stop 2024-02-21T23:05:27.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.990+0000 7fe6d0e00640 1 -- 172.21.15.28:0/1376223964 shutdown_connections 2024-02-21T23:05:27.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.990+0000 7fe6d0e00640 1 --2- 172.21.15.28:0/1376223964 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe6cc0c2c60 0x7fe6cc06dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:27.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.990+0000 7fe6d0e00640 1 --2- 172.21.15.28:0/1376223964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6cc1500f0 0x7fe6cc06ae70 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:27.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.990+0000 7fe6d0e00640 1 -- 172.21.15.28:0/1376223964 >> 172.21.15.28:0/1376223964 conn(0x7fe6cc0a8530 msgr2=0x7fe6cc0a8910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:27.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.990+0000 7fe6d0e00640 1 -- 172.21.15.28:0/1376223964 shutdown_connections 2024-02-21T23:05:27.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.990+0000 7fe6d0e00640 1 -- 172.21.15.28:0/1376223964 wait complete. 2024-02-21T23:05:27.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.992+0000 7fe6d0e00640 1 Processor -- start 2024-02-21T23:05:27.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.992+0000 7fe6d0e00640 1 -- start start 2024-02-21T23:05:27.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.993+0000 7fe6d0e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe6cc06b3b0 0x7fe6cc151c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:27.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.993+0000 7fe6d0e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6cc0c2c60 0x7fe6cc152170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:27.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.993+0000 7fe6d0e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe6cc1500f0 0x7fe6cc1ebef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:27.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.993+0000 7fe6d0e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6cc1535e0 con 0x7fe6cc0c2c60 2024-02-21T23:05:27.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.993+0000 7fe6d0e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6cc1504d0 con 0x7fe6cc06b3b0 2024-02-21T23:05:27.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.993+0000 7fe6d0e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6cc153760 con 0x7fe6cc1500f0 2024-02-21T23:05:27.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.993+0000 7fe6caa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6cc0c2c60 0x7fe6cc152170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:27.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.993+0000 7fe6caa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6cc0c2c60 0x7fe6cc152170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40316/0 (socket says 172.21.15.28:40316) 2024-02-21T23:05:27.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.993+0000 7fe6caa00640 1 -- 172.21.15.28:0/4145283202 learned_addr learned my addr 172.21.15.28:0/4145283202 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:27.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.993+0000 7fe6cb400640 1 --2- 172.21.15.28:0/4145283202 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe6cc06b3b0 0x7fe6cc151c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:27.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.993+0000 7fe6cbe00640 1 --2- 172.21.15.28:0/4145283202 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe6cc1500f0 0x7fe6cc1ebef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:27.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.993+0000 7fe6caa00640 1 -- 172.21.15.28:0/4145283202 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe6cc1500f0 msgr2=0x7fe6cc1ebef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:27.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.994+0000 7fe6caa00640 1 --2- 172.21.15.28:0/4145283202 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe6cc1500f0 0x7fe6cc1ebef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:27.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.994+0000 7fe6caa00640 1 -- 172.21.15.28:0/4145283202 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe6cc06b3b0 msgr2=0x7fe6cc151c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:27.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.994+0000 7fe6caa00640 1 --2- 172.21.15.28:0/4145283202 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe6cc06b3b0 0x7fe6cc151c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:27.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.994+0000 7fe6caa00640 1 -- 172.21.15.28:0/4145283202 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6cc0a9ab0 con 0x7fe6cc0c2c60 2024-02-21T23:05:27.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.994+0000 7fe6caa00640 1 --2- 172.21.15.28:0/4145283202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6cc0c2c60 0x7fe6cc152170 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fe6b800e9b0 tx=0x7fe6b800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:27.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.994+0000 7fe6c3400640 1 -- 172.21.15.28:0/4145283202 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b8017070 con 0x7fe6cc0c2c60 2024-02-21T23:05:27.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.994+0000 7fe6c3400640 1 -- 172.21.15.28:0/4145283202 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6b800c9a0 con 0x7fe6cc0c2c60 2024-02-21T23:05:27.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.994+0000 7fe6c3400640 1 -- 172.21.15.28:0/4145283202 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b800cd90 con 0x7fe6cc0c2c60 2024-02-21T23:05:27.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.994+0000 7fe6d0e00640 1 -- 172.21.15.28:0/4145283202 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6cc151430 con 0x7fe6cc0c2c60 2024-02-21T23:05:27.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.994+0000 7fe6d0e00640 1 -- 172.21.15.28:0/4145283202 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6cc153da0 con 0x7fe6cc0c2c60 2024-02-21T23:05:27.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.995+0000 7fe6d0e00640 1 -- 172.21.15.28:0/4145283202 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6cc0acb00 con 0x7fe6cc0c2c60 2024-02-21T23:05:27.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.997+0000 7fe6c3400640 1 -- 172.21.15.28:0/4145283202 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe6b801a070 con 0x7fe6cc0c2c60 2024-02-21T23:05:27.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.998+0000 7fe6c3400640 1 --2- 172.21.15.28:0/4145283202 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe6ac06b500 0x7fe6ac06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:27.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.998+0000 7fe6c3400640 1 -- 172.21.15.28:0/4145283202 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7418+0+0 (secure 0 0 0) 0x7fe6b808ace0 con 0x7fe6cc0c2c60 2024-02-21T23:05:27.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:27.999+0000 7fe6cb400640 1 --2- 172.21.15.28:0/4145283202 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe6ac06b500 0x7fe6ac06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:27.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:28.000+0000 7fe6cb400640 1 --2- 172.21.15.28:0/4145283202 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe6ac06b500 0x7fe6ac06d9c0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc036000 tx=0x7fe6bc03c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:27.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:28.003+0000 7fe6c3400640 1 -- 172.21.15.28:0/4145283202 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe6cc0acb00 con 0x7fe6cc0c2c60 2024-02-21T23:05:28.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:28.243+0000 7fe6d0e00640 1 -- 172.21.15.28:0/4145283202 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fe6cc0c07f0 con 0x7fe6cc0c2c60 2024-02-21T23:05:28.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:28.781+0000 7fe6c3400640 1 -- 172.21.15.28:0/4145283202 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v49) v1 ==== 104+0+0 (secure 0 0 0) 0x7fe6b8059410 con 0x7fe6cc0c2c60 2024-02-21T23:05:28.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:28.784+0000 7fe6d0e00640 1 -- 172.21.15.28:0/4145283202 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe6ac06b500 msgr2=0x7fe6ac06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:28.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:28.784+0000 7fe6d0e00640 1 --2- 172.21.15.28:0/4145283202 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe6ac06b500 0x7fe6ac06d9c0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc036000 tx=0x7fe6bc03c040 comp rx=0 tx=0).stop 2024-02-21T23:05:28.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:28.784+0000 7fe6d0e00640 1 -- 172.21.15.28:0/4145283202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6cc0c2c60 msgr2=0x7fe6cc152170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:28.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:28.784+0000 7fe6d0e00640 1 --2- 172.21.15.28:0/4145283202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6cc0c2c60 0x7fe6cc152170 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fe6b800e9b0 tx=0x7fe6b800ee70 comp rx=0 tx=0).stop 2024-02-21T23:05:28.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:28.785+0000 7fe6d0e00640 1 -- 172.21.15.28:0/4145283202 shutdown_connections 2024-02-21T23:05:28.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:28.785+0000 7fe6d0e00640 1 --2- 172.21.15.28:0/4145283202 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe6ac06b500 0x7fe6ac06d9c0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:28.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:28.785+0000 7fe6d0e00640 1 --2- 172.21.15.28:0/4145283202 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe6cc1500f0 0x7fe6cc1ebef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:28.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:28.785+0000 7fe6d0e00640 1 --2- 172.21.15.28:0/4145283202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6cc0c2c60 0x7fe6cc152170 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:28.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:28.785+0000 7fe6d0e00640 1 --2- 172.21.15.28:0/4145283202 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe6cc06b3b0 0x7fe6cc151c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:28.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:28.785+0000 7fe6d0e00640 1 -- 172.21.15.28:0/4145283202 >> 172.21.15.28:0/4145283202 conn(0x7fe6cc0a8530 msgr2=0x7fe6cc14a620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:28.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:28.785+0000 7fe6d0e00640 1 -- 172.21.15.28:0/4145283202 shutdown_connections 2024-02-21T23:05:28.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:28.785+0000 7fe6d0e00640 1 -- 172.21.15.28:0/4145283202 wait complete. 2024-02-21T23:05:28.793 INFO:tasks.ceph.mds.d.smithi081.stderr: -546> 2024-02-21T23:05:13.359+0000 7f7bfcc00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-21T23:05:28.799 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-02-21T23:05:28.840 INFO:tasks.ceph.mds.d.smithi081.stdout:starting mds.d at 2024-02-21T23:05:29.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.068+0000 7f1be5a00640 1 Processor -- start 2024-02-21T23:05:29.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.068+0000 7f1be5a00640 1 -- start start 2024-02-21T23:05:29.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.069+0000 7f1be5a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f1be01526f0 0x7f1be00c1340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:29.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.069+0000 7f1be5a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1be00c4dc0 0x7f1be0158f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:29.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.069+0000 7f1be5a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1be0152ad0 con 0x7f1be00c46f0 2024-02-21T23:05:29.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.069+0000 7f1be5a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1be0155240 con 0x7f1be00c4dc0 2024-02-21T23:05:29.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.069+0000 7f1be5a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1be0155520 con 0x7f1be01526f0 2024-02-21T23:05:29.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.069+0000 7f1bd7e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f1be00c46f0 0x7f1be0158820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:44060/0 (socket says 172.21.15.28:44060) 2024-02-21T23:05:29.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.070+0000 7f1bd7e00640 1 -- 172.21.15.28:0/2055251346 learned_addr learned my addr 172.21.15.28:0/2055251346 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:05:29.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.070+0000 7f1bdf400640 1 --2- 172.21.15.28:0/2055251346 >> v2:172.21.15.28:3300/0 conn(0x7f1be00c4dc0 0x7f1be0158f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:29.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.070+0000 7f1bdea00640 1 --2- 172.21.15.28:0/2055251346 >> v2:172.21.15.81:3300/0 conn(0x7f1be01526f0 0x7f1be00c1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:29.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.070+0000 7f1bde000640 1 -- 172.21.15.28:0/2055251346 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3121912481 0 0) 0x7f1be0152ad0 con 0x7f1be00c46f0 2024-02-21T23:05:29.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.070+0000 7f1bde000640 1 -- 172.21.15.28:0/2055251346 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bc4003520 con 0x7f1be00c46f0 2024-02-21T23:05:29.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.070+0000 7f1bdf400640 1 -- 172.21.15.28:0/2055251346 >> v2:172.21.15.81:3300/0 conn(0x7f1be01526f0 msgr2=0x7f1be00c1340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:29.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.070+0000 7f1bdf400640 1 --2- 172.21.15.28:0/2055251346 >> v2:172.21.15.81:3300/0 conn(0x7f1be01526f0 0x7f1be00c1340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:29.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.070+0000 7f1bdf400640 1 -- 172.21.15.28:0/2055251346 >> v1:172.21.15.28:6789/0 conn(0x7f1be00c46f0 legacy=0x7f1be0158820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:29.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.070+0000 7f1bdf400640 1 -- 172.21.15.28:0/2055251346 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1be0155800 con 0x7f1be00c4dc0 2024-02-21T23:05:29.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.071+0000 7f1bdf400640 1 --2- 172.21.15.28:0/2055251346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1be00c4dc0 0x7f1be0158f30 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f1bd8009d90 tx=0x7f1bd802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=135a4d0ecd8948d7 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:29.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.072+0000 7f1bde000640 1 -- 172.21.15.28:0/2055251346 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bd8039070 con 0x7f1be00c4dc0 2024-02-21T23:05:29.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.072+0000 7f1bde000640 1 -- 172.21.15.28:0/2055251346 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bd802cc60 con 0x7f1be00c4dc0 2024-02-21T23:05:29.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.072+0000 7f1bde000640 1 -- 172.21.15.28:0/2055251346 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bd8034660 con 0x7f1be00c4dc0 2024-02-21T23:05:29.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.072+0000 7f1be5a00640 1 -- 172.21.15.28:0/2055251346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1be00c4dc0 msgr2=0x7f1be0158f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:29.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.072+0000 7f1be5a00640 1 --2- 172.21.15.28:0/2055251346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1be00c4dc0 0x7f1be0158f30 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f1bd8009d90 tx=0x7f1bd802beb0 comp rx=0 tx=0).stop 2024-02-21T23:05:29.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.072+0000 7f1be5a00640 1 -- 172.21.15.28:0/2055251346 shutdown_connections 2024-02-21T23:05:29.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.072+0000 7f1be5a00640 1 --2- 172.21.15.28:0/2055251346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1be00c4dc0 0x7f1be0158f30 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:29.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.072+0000 7f1be5a00640 1 --2- 172.21.15.28:0/2055251346 >> v2:172.21.15.81:3300/0 conn(0x7f1be01526f0 0x7f1be00c1340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:29.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.072+0000 7f1be5a00640 1 -- 172.21.15.28:0/2055251346 >> 172.21.15.28:0/2055251346 conn(0x7f1be006b830 msgr2=0x7f1be00c67f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:29.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.072+0000 7f1be5a00640 1 -- 172.21.15.28:0/2055251346 shutdown_connections 2024-02-21T23:05:29.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.073+0000 7f1be5a00640 1 -- 172.21.15.28:0/2055251346 wait complete. 2024-02-21T23:05:29.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.074+0000 7f1be5a00640 1 Processor -- start 2024-02-21T23:05:29.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.074+0000 7f1be5a00640 1 -- start start 2024-02-21T23:05:29.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.075+0000 7f1be5a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1be00c46f0 0x7f1be01ead20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:29.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.075+0000 7f1be5a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1be00c4dc0 0x7f1be014ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:29.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.075+0000 7f1be5a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1be01526f0 0x7f1be01eeba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:29.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.075+0000 7f1be5a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1be0159470 con 0x7f1be00c46f0 2024-02-21T23:05:29.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.075+0000 7f1be5a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1be0155dc0 con 0x7f1be00c4dc0 2024-02-21T23:05:29.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.075+0000 7f1be5a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1be00c3c30 con 0x7f1be01526f0 2024-02-21T23:05:29.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.075+0000 7f1bdea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1be00c46f0 0x7f1be01ead20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:29.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.075+0000 7f1bdea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1be00c46f0 0x7f1be01ead20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40326/0 (socket says 172.21.15.28:40326) 2024-02-21T23:05:29.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.075+0000 7f1bdea00640 1 -- 172.21.15.28:0/2093571224 learned_addr learned my addr 172.21.15.28:0/2093571224 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:29.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.076+0000 7f1bd7e00640 1 --2- 172.21.15.28:0/2093571224 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1be00c4dc0 0x7f1be014ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:29.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.076+0000 7f1bdea00640 1 -- 172.21.15.28:0/2093571224 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1be01526f0 msgr2=0x7f1be01eeba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:29.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.076+0000 7f1bdea00640 1 --2- 172.21.15.28:0/2093571224 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1be01526f0 0x7f1be01eeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:29.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.076+0000 7f1bdea00640 1 -- 172.21.15.28:0/2093571224 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1be00c4dc0 msgr2=0x7f1be014ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:29.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.076+0000 7f1bdea00640 1 --2- 172.21.15.28:0/2093571224 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1be00c4dc0 0x7f1be014ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:29.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.076+0000 7f1bdea00640 1 -- 172.21.15.28:0/2093571224 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1be014f440 con 0x7f1be00c46f0 2024-02-21T23:05:29.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.076+0000 7f1bd7e00640 1 --2- 172.21.15.28:0/2093571224 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1be00c4dc0 0x7f1be014ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:05:29.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.076+0000 7f1bdea00640 1 --2- 172.21.15.28:0/2093571224 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1be00c46f0 0x7f1be01ead20 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f1bd000d5f0 tx=0x7f1bd000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:29.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.076+0000 7f1bd6a00640 1 -- 172.21.15.28:0/2093571224 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bd00125f0 con 0x7f1be00c46f0 2024-02-21T23:05:29.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.077+0000 7f1bd6a00640 1 -- 172.21.15.28:0/2093571224 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bd0019070 con 0x7f1be00c46f0 2024-02-21T23:05:29.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.077+0000 7f1bd6a00640 1 -- 172.21.15.28:0/2093571224 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bd00129d0 con 0x7f1be00c46f0 2024-02-21T23:05:29.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.077+0000 7f1be5a00640 1 -- 172.21.15.28:0/2093571224 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1be01eb260 con 0x7f1be00c46f0 2024-02-21T23:05:29.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.077+0000 7f1be5a00640 1 -- 172.21.15.28:0/2093571224 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1be014cec0 con 0x7f1be00c46f0 2024-02-21T23:05:29.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.078+0000 7f1be5a00640 1 -- 172.21.15.28:0/2093571224 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1be00c4110 con 0x7f1be00c46f0 2024-02-21T23:05:29.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.079+0000 7f1bd6a00640 1 -- 172.21.15.28:0/2093571224 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1bd0012b70 con 0x7f1be00c46f0 2024-02-21T23:05:29.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.080+0000 7f1bd6a00640 1 --2- 172.21.15.28:0/2093571224 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1bbc06b4d0 0x7f1bbc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:29.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.080+0000 7f1bd6a00640 1 -- 172.21.15.28:0/2093571224 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7504+0+0 (secure 0 0 0) 0x7f1bd008ace0 con 0x7f1be00c46f0 2024-02-21T23:05:29.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.080+0000 7f1bd7e00640 1 --2- 172.21.15.28:0/2093571224 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1bbc06b4d0 0x7f1bbc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:29.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.081+0000 7f1bd7e00640 1 --2- 172.21.15.28:0/2093571224 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1bbc06b4d0 0x7f1bbc06d990 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f1bc8007fd0 tx=0x7f1bc8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:29.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.084+0000 7f1bd6a00640 1 -- 172.21.15.28:0/2093571224 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1be00c4110 con 0x7f1be00c46f0 2024-02-21T23:05:29.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.286+0000 7f1be5a00640 1 -- 172.21.15.28:0/2093571224 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f1be0067f30 con 0x7f1be00c46f0 2024-02-21T23:05:29.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.793+0000 7f1bd6a00640 1 -- 172.21.15.28:0/2093571224 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v64) v1 ==== 181+0+0 (secure 0 0 0) 0x7f1bd0059470 con 0x7f1be00c46f0 2024-02-21T23:05:29.782 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' removed 2024-02-21T23:05:29.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.800+0000 7f1be5a00640 1 -- 172.21.15.28:0/2093571224 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1bbc06b4d0 msgr2=0x7f1bbc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:29.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.800+0000 7f1be5a00640 1 --2- 172.21.15.28:0/2093571224 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1bbc06b4d0 0x7f1bbc06d990 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f1bc8007fd0 tx=0x7f1bc8009040 comp rx=0 tx=0).stop 2024-02-21T23:05:29.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.800+0000 7f1be5a00640 1 -- 172.21.15.28:0/2093571224 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1be00c46f0 msgr2=0x7f1be01ead20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:29.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.800+0000 7f1be5a00640 1 --2- 172.21.15.28:0/2093571224 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1be00c46f0 0x7f1be01ead20 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f1bd000d5f0 tx=0x7f1bd000dab0 comp rx=0 tx=0).stop 2024-02-21T23:05:29.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.800+0000 7f1be5a00640 1 -- 172.21.15.28:0/2093571224 shutdown_connections 2024-02-21T23:05:29.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.800+0000 7f1be5a00640 1 --2- 172.21.15.28:0/2093571224 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1bbc06b4d0 0x7f1bbc06d990 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:29.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.800+0000 7f1be5a00640 1 --2- 172.21.15.28:0/2093571224 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1be01526f0 0x7f1be01eeba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:29.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.800+0000 7f1be5a00640 1 --2- 172.21.15.28:0/2093571224 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1be00c4dc0 0x7f1be014ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:29.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.800+0000 7f1be5a00640 1 --2- 172.21.15.28:0/2093571224 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1be00c46f0 0x7f1be01ead20 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:29.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.800+0000 7f1be5a00640 1 -- 172.21.15.28:0/2093571224 >> 172.21.15.28:0/2093571224 conn(0x7f1be006b830 msgr2=0x7f1be00c67f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:29.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.801+0000 7f1be5a00640 1 -- 172.21.15.28:0/2093571224 shutdown_connections 2024-02-21T23:05:29.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:29.801+0000 7f1be5a00640 1 -- 172.21.15.28:0/2093571224 wait complete. 2024-02-21T23:05:29.821 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-02-21T23:05:30.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.031+0000 7fee91200640 1 Processor -- start 2024-02-21T23:05:30.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.031+0000 7fee91200640 1 -- start start 2024-02-21T23:05:30.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.032+0000 7fee91200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fee8c070d60 0x7fee8c152f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:30.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.032+0000 7fee91200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fee8c0c7eb0 0x7fee8c0c4220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:30.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.032+0000 7fee91200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee8c0c8420 con 0x7fee8c0c4760 2024-02-21T23:05:30.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.032+0000 7fee91200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee8c0c9030 con 0x7fee8c0c7eb0 2024-02-21T23:05:30.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.032+0000 7fee91200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee8c0c8700 con 0x7fee8c070d60 2024-02-21T23:05:30.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.033+0000 7fee89400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fee8c0c7eb0 0x7fee8c0c4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:30.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.033+0000 7fee8a800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fee8c0c4760 0x7fee8c0c6830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:44076/0 (socket says 172.21.15.28:44076) 2024-02-21T23:05:30.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.033+0000 7fee8a800640 1 -- 172.21.15.28:0/3616316487 learned_addr learned my addr 172.21.15.28:0/3616316487 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:05:30.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.033+0000 7fee89e00640 1 --2- 172.21.15.28:0/3616316487 >> v2:172.21.15.81:3300/0 conn(0x7fee8c070d60 0x7fee8c152f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:30.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.033+0000 7fee89400640 1 -- 172.21.15.28:0/3616316487 >> v2:172.21.15.81:3300/0 conn(0x7fee8c070d60 msgr2=0x7fee8c152f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:30.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.033+0000 7fee89400640 1 --2- 172.21.15.28:0/3616316487 >> v2:172.21.15.81:3300/0 conn(0x7fee8c070d60 0x7fee8c152f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:30.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.033+0000 7fee89400640 1 -- 172.21.15.28:0/3616316487 >> v1:172.21.15.28:6789/0 conn(0x7fee8c0c4760 legacy=0x7fee8c0c6830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:30.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.033+0000 7fee89400640 1 -- 172.21.15.28:0/3616316487 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee8c0c89e0 con 0x7fee8c0c7eb0 2024-02-21T23:05:30.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.033+0000 7fee88a00640 1 -- 172.21.15.28:0/3616316487 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1669596291 0 0) 0x7fee8c0c8420 con 0x7fee8c0c4760 2024-02-21T23:05:30.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.035+0000 7fee89400640 1 --2- 172.21.15.28:0/3616316487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee8c0c7eb0 0x7fee8c0c4220 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fee8c0730f0 tx=0x7fee7802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39ecca73ea0ca03f server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:30.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.035+0000 7fee88a00640 1 -- 172.21.15.28:0/3616316487 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee78035020 con 0x7fee8c0c7eb0 2024-02-21T23:05:30.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.035+0000 7fee88a00640 1 -- 172.21.15.28:0/3616316487 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee78031830 con 0x7fee8c0c7eb0 2024-02-21T23:05:30.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.035+0000 7fee88a00640 1 -- 172.21.15.28:0/3616316487 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee78031c00 con 0x7fee8c0c7eb0 2024-02-21T23:05:30.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.035+0000 7fee91200640 1 -- 172.21.15.28:0/3616316487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee8c0c7eb0 msgr2=0x7fee8c0c4220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:30.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.035+0000 7fee91200640 1 --2- 172.21.15.28:0/3616316487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee8c0c7eb0 0x7fee8c0c4220 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fee8c0730f0 tx=0x7fee7802b8d0 comp rx=0 tx=0).stop 2024-02-21T23:05:30.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.035+0000 7fee91200640 1 -- 172.21.15.28:0/3616316487 shutdown_connections 2024-02-21T23:05:30.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.035+0000 7fee91200640 1 --2- 172.21.15.28:0/3616316487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee8c0c7eb0 0x7fee8c0c4220 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:30.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.036+0000 7fee91200640 1 --2- 172.21.15.28:0/3616316487 >> v2:172.21.15.81:3300/0 conn(0x7fee8c070d60 0x7fee8c152f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:30.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.036+0000 7fee91200640 1 -- 172.21.15.28:0/3616316487 >> 172.21.15.28:0/3616316487 conn(0x7fee8c0aa5c0 msgr2=0x7fee8c0aa9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:30.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.036+0000 7fee91200640 1 -- 172.21.15.28:0/3616316487 shutdown_connections 2024-02-21T23:05:30.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.036+0000 7fee91200640 1 -- 172.21.15.28:0/3616316487 wait complete. 2024-02-21T23:05:30.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.037+0000 7fee91200640 1 Processor -- start 2024-02-21T23:05:30.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.037+0000 7fee91200640 1 -- start start 2024-02-21T23:05:30.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.038+0000 7fee91200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fee8c070d60 0x7fee8c154f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:30.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.038+0000 7fee91200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee8c0c4760 0x7fee8c156320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:30.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.038+0000 7fee91200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee8c0c7eb0 0x7fee8c09a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:30.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.038+0000 7fee91200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee8c0c4d50 con 0x7fee8c0c4760 2024-02-21T23:05:30.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.038+0000 7fee91200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee8c0c4bd0 con 0x7fee8c0c7eb0 2024-02-21T23:05:30.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.038+0000 7fee91200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee8c06f410 con 0x7fee8c070d60 2024-02-21T23:05:30.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.039+0000 7fee89400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee8c0c4760 0x7fee8c156320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:30.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.039+0000 7fee89400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee8c0c4760 0x7fee8c156320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40344/0 (socket says 172.21.15.28:40344) 2024-02-21T23:05:30.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.039+0000 7fee89400640 1 -- 172.21.15.28:0/2211067281 learned_addr learned my addr 172.21.15.28:0/2211067281 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:30.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.039+0000 7fee89400640 1 -- 172.21.15.28:0/2211067281 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fee8c070d60 msgr2=0x7fee8c154f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:30.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.039+0000 7fee89400640 1 --2- 172.21.15.28:0/2211067281 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fee8c070d60 0x7fee8c154f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:30.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.039+0000 7fee89400640 1 -- 172.21.15.28:0/2211067281 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee8c0c7eb0 msgr2=0x7fee8c09a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:30.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.039+0000 7fee89400640 1 --2- 172.21.15.28:0/2211067281 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee8c0c7eb0 0x7fee8c09a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:30.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.039+0000 7fee89400640 1 -- 172.21.15.28:0/2211067281 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee8c156b30 con 0x7fee8c0c4760 2024-02-21T23:05:30.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.039+0000 7fee89400640 1 --2- 172.21.15.28:0/2211067281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee8c0c4760 0x7fee8c156320 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fee78009890 tx=0x7fee780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:30.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.040+0000 7fee82000640 1 -- 172.21.15.28:0/2211067281 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee78040070 con 0x7fee8c0c4760 2024-02-21T23:05:30.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.040+0000 7fee82000640 1 -- 172.21.15.28:0/2211067281 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee7803a8f0 con 0x7fee8c0c4760 2024-02-21T23:05:30.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.040+0000 7fee91200640 1 -- 172.21.15.28:0/2211067281 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee8c155650 con 0x7fee8c0c4760 2024-02-21T23:05:30.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.040+0000 7fee82000640 1 -- 172.21.15.28:0/2211067281 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee78035040 con 0x7fee8c0c4760 2024-02-21T23:05:30.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.040+0000 7fee91200640 1 -- 172.21.15.28:0/2211067281 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee8c099a50 con 0x7fee8c0c4760 2024-02-21T23:05:30.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.041+0000 7fee91200640 1 -- 172.21.15.28:0/2211067281 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee8c068640 con 0x7fee8c0c4760 2024-02-21T23:05:30.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.042+0000 7fee82000640 1 -- 172.21.15.28:0/2211067281 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fee78038020 con 0x7fee8c0c4760 2024-02-21T23:05:30.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.043+0000 7fee82000640 1 --2- 172.21.15.28:0/2211067281 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fee6c06b500 0x7fee6c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:30.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.043+0000 7fee82000640 1 -- 172.21.15.28:0/2211067281 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7066+0+0 (secure 0 0 0) 0x7fee780ac000 con 0x7fee8c0c4760 2024-02-21T23:05:30.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.044+0000 7fee89e00640 1 --2- 172.21.15.28:0/2211067281 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fee6c06b500 0x7fee6c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:30.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.045+0000 7fee89e00640 1 --2- 172.21.15.28:0/2211067281 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fee6c06b500 0x7fee6c06d9c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fee7c0070b0 tx=0x7fee7c007040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:30.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.048+0000 7fee82000640 1 -- 172.21.15.28:0/2211067281 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fee7807a930 con 0x7fee8c0c4760 2024-02-21T23:05:30.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.246+0000 7fee91200640 1 -- 172.21.15.28:0/2211067281 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fee8c0a8760 con 0x7fee8c0c4760 2024-02-21T23:05:30.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.816+0000 7fee82000640 1 -- 172.21.15.28:0/2211067281 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v65) v1 ==== 169+0+0 (secure 0 0 0) 0x7fee7807a230 con 0x7fee8c0c4760 2024-02-21T23:05:30.806 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' removed 2024-02-21T23:05:30.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.821+0000 7fee91200640 1 -- 172.21.15.28:0/2211067281 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fee6c06b500 msgr2=0x7fee6c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:30.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.821+0000 7fee91200640 1 --2- 172.21.15.28:0/2211067281 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fee6c06b500 0x7fee6c06d9c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fee7c0070b0 tx=0x7fee7c007040 comp rx=0 tx=0).stop 2024-02-21T23:05:30.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.821+0000 7fee91200640 1 -- 172.21.15.28:0/2211067281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee8c0c4760 msgr2=0x7fee8c156320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:30.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.821+0000 7fee91200640 1 --2- 172.21.15.28:0/2211067281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee8c0c4760 0x7fee8c156320 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fee78009890 tx=0x7fee780046c0 comp rx=0 tx=0).stop 2024-02-21T23:05:30.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.824+0000 7fee91200640 1 -- 172.21.15.28:0/2211067281 shutdown_connections 2024-02-21T23:05:30.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.824+0000 7fee91200640 1 --2- 172.21.15.28:0/2211067281 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fee6c06b500 0x7fee6c06d9c0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:30.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.824+0000 7fee91200640 1 --2- 172.21.15.28:0/2211067281 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee8c0c7eb0 0x7fee8c09a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:30.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.824+0000 7fee91200640 1 --2- 172.21.15.28:0/2211067281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee8c0c4760 0x7fee8c156320 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:30.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.824+0000 7fee91200640 1 --2- 172.21.15.28:0/2211067281 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fee8c070d60 0x7fee8c154f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:30.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.824+0000 7fee91200640 1 -- 172.21.15.28:0/2211067281 >> 172.21.15.28:0/2211067281 conn(0x7fee8c0aa5c0 msgr2=0x7fee8c0aa9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:30.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.826+0000 7fee91200640 1 -- 172.21.15.28:0/2211067281 shutdown_connections 2024-02-21T23:05:30.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:30.826+0000 7fee91200640 1 -- 172.21.15.28:0/2211067281 wait complete. 2024-02-21T23:05:30.842 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:05:31.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.045+0000 7fdd18c00640 1 Processor -- start 2024-02-21T23:05:31.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.045+0000 7fdd18c00640 1 -- start start 2024-02-21T23:05:31.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.046+0000 7fdd18c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdd14150450 0x7fdd140a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:31.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.047+0000 7fdd18c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fdd14156af0 0x7fdd1406dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:31.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.047+0000 7fdd18c00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd140c2760 con 0x7fdd1406ee40 2024-02-21T23:05:31.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.047+0000 7fdd18c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd140c22c0 con 0x7fdd14150450 2024-02-21T23:05:31.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.047+0000 7fdd18c00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd14156660 con 0x7fdd14156af0 2024-02-21T23:05:31.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.047+0000 7fdd13400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdd14150450 0x7fdd140a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:31.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.047+0000 7fdd13400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdd14150450 0x7fdd140a5fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40354/0 (socket says 172.21.15.28:40354) 2024-02-21T23:05:31.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.047+0000 7fdd13400640 1 -- 172.21.15.28:0/3405229964 learned_addr learned my addr 172.21.15.28:0/3405229964 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:31.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.047+0000 7fdd13400640 1 -- 172.21.15.28:0/3405229964 >> v2:172.21.15.81:3300/0 conn(0x7fdd14156af0 msgr2=0x7fdd1406dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:31.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.047+0000 7fdd13400640 1 --2- 172.21.15.28:0/3405229964 >> v2:172.21.15.81:3300/0 conn(0x7fdd14156af0 0x7fdd1406dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:31.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.047+0000 7fdd13400640 1 -- 172.21.15.28:0/3405229964 >> v1:172.21.15.81:6789/0 conn(0x7fdd1406ee40 legacy=0x7fdd141536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:31.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.047+0000 7fdd13400640 1 -- 172.21.15.28:0/3405229964 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd14156310 con 0x7fdd14150450 2024-02-21T23:05:31.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.048+0000 7fdd13400640 1 --2- 172.21.15.28:0/3405229964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd14150450 0x7fdd140a5fd0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fdd140734b0 tx=0x7fdd0402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11721bfdef4f64f8 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:31.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.049+0000 7fdd12000640 1 -- 172.21.15.28:0/3405229964 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd04035020 con 0x7fdd14150450 2024-02-21T23:05:31.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.049+0000 7fdd12000640 1 -- 172.21.15.28:0/3405229964 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd04031870 con 0x7fdd14150450 2024-02-21T23:05:31.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.049+0000 7fdd12000640 1 -- 172.21.15.28:0/3405229964 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd04031c40 con 0x7fdd14150450 2024-02-21T23:05:31.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.049+0000 7fdd18c00640 1 -- 172.21.15.28:0/3405229964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd14150450 msgr2=0x7fdd140a5fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:31.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.049+0000 7fdd18c00640 1 --2- 172.21.15.28:0/3405229964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd14150450 0x7fdd140a5fd0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fdd140734b0 tx=0x7fdd0402b910 comp rx=0 tx=0).stop 2024-02-21T23:05:31.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.050+0000 7fdd18c00640 1 -- 172.21.15.28:0/3405229964 shutdown_connections 2024-02-21T23:05:31.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.050+0000 7fdd18c00640 1 --2- 172.21.15.28:0/3405229964 >> v2:172.21.15.81:3300/0 conn(0x7fdd14156af0 0x7fdd1406dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:31.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.050+0000 7fdd18c00640 1 --2- 172.21.15.28:0/3405229964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd14150450 0x7fdd140a5fd0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:31.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.050+0000 7fdd18c00640 1 -- 172.21.15.28:0/3405229964 >> 172.21.15.28:0/3405229964 conn(0x7fdd1406b630 msgr2=0x7fdd140c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:31.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.050+0000 7fdd18c00640 1 -- 172.21.15.28:0/3405229964 shutdown_connections 2024-02-21T23:05:31.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.050+0000 7fdd18c00640 1 -- 172.21.15.28:0/3405229964 wait complete. 2024-02-21T23:05:31.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.051+0000 7fdd18c00640 1 Processor -- start 2024-02-21T23:05:31.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.052+0000 7fdd18c00640 1 -- start start 2024-02-21T23:05:31.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.052+0000 7fdd18c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd1406ee40 0x7fdd14154ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:31.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.052+0000 7fdd18c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdd14150450 0x7fdd141e8a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:31.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.053+0000 7fdd18c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd14156af0 0x7fdd141e8fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:31.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.053+0000 7fdd18c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd14156f80 con 0x7fdd1406ee40 2024-02-21T23:05:31.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.053+0000 7fdd18c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd140c2cb0 con 0x7fdd14156af0 2024-02-21T23:05:31.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.053+0000 7fdd18c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd14157100 con 0x7fdd14150450 2024-02-21T23:05:31.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.053+0000 7fdd13400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd1406ee40 0x7fdd14154ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:31.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.053+0000 7fdd13400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd1406ee40 0x7fdd14154ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40370/0 (socket says 172.21.15.28:40370) 2024-02-21T23:05:31.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.053+0000 7fdd13400640 1 -- 172.21.15.28:0/964512427 learned_addr learned my addr 172.21.15.28:0/964512427 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:31.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.053+0000 7fdd13e00640 1 --2- 172.21.15.28:0/964512427 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd14156af0 0x7fdd141e8fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:31.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.053+0000 7fdd12a00640 1 --2- 172.21.15.28:0/964512427 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdd14150450 0x7fdd141e8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:31.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.053+0000 7fdd13400640 1 -- 172.21.15.28:0/964512427 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdd14150450 msgr2=0x7fdd141e8a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:31.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.053+0000 7fdd13400640 1 --2- 172.21.15.28:0/964512427 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdd14150450 0x7fdd141e8a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:31.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.053+0000 7fdd13400640 1 -- 172.21.15.28:0/964512427 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd14156af0 msgr2=0x7fdd141e8fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:31.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.053+0000 7fdd13400640 1 --2- 172.21.15.28:0/964512427 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd14156af0 0x7fdd141e8fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:31.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.053+0000 7fdd13400640 1 -- 172.21.15.28:0/964512427 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd141ecce0 con 0x7fdd1406ee40 2024-02-21T23:05:31.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.054+0000 7fdd13400640 1 --2- 172.21.15.28:0/964512427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd1406ee40 0x7fdd14154ef0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fdd04009960 tx=0x7fdd040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:31.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.054+0000 7fdd0b400640 1 -- 172.21.15.28:0/964512427 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd040443f0 con 0x7fdd1406ee40 2024-02-21T23:05:31.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.054+0000 7fdd0b400640 1 -- 172.21.15.28:0/964512427 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd040445b0 con 0x7fdd1406ee40 2024-02-21T23:05:31.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.054+0000 7fdd0b400640 1 -- 172.21.15.28:0/964512427 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd040449a0 con 0x7fdd1406ee40 2024-02-21T23:05:31.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.054+0000 7fdd18c00640 1 -- 172.21.15.28:0/964512427 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd14148d30 con 0x7fdd1406ee40 2024-02-21T23:05:31.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.054+0000 7fdd18c00640 1 -- 172.21.15.28:0/964512427 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd14148fd0 con 0x7fdd1406ee40 2024-02-21T23:05:31.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.055+0000 7fdd18c00640 1 -- 172.21.15.28:0/964512427 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd1404f290 con 0x7fdd1406ee40 2024-02-21T23:05:31.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.056+0000 7fdd0b400640 1 -- 172.21.15.28:0/964512427 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdd04038020 con 0x7fdd1406ee40 2024-02-21T23:05:31.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.057+0000 7fdd0b400640 1 --2- 172.21.15.28:0/964512427 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdcf406b4d0 0x7fdcf406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:31.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.057+0000 7fdd0b400640 1 -- 172.21.15.28:0/964512427 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6594+0+0 (secure 0 0 0) 0x7fdd040ac7d0 con 0x7fdd1406ee40 2024-02-21T23:05:31.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.058+0000 7fdd12a00640 1 --2- 172.21.15.28:0/964512427 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdcf406b4d0 0x7fdcf406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:31.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.059+0000 7fdd12a00640 1 --2- 172.21.15.28:0/964512427 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdcf406b4d0 0x7fdcf406d990 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fdd140c0ac0 tx=0x7fdd00009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:31.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.062+0000 7fdd0b400640 1 -- 172.21.15.28:0/964512427 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fdd0403f070 con 0x7fdd1406ee40 2024-02-21T23:05:31.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.261+0000 7fdd18c00640 1 -- 172.21.15.28:0/964512427 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdd140c1f10 con 0x7fdd1406ee40 2024-02-21T23:05:31.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.264+0000 7fdd0b400640 1 -- 172.21.15.28:0/964512427 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v65) v1 ==== 74+0+17586 (secure 0 0 0) 0x7fdd0407b2c0 con 0x7fdd1406ee40 2024-02-21T23:05:31.254 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:05:31.254 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":65,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:05:30.807870+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs2_metadata","create_time":"2024-02-21T23:05:05.701276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs2_data","create_time":"2024-02-21T23:05:06.904887+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"9.11","mappings":[{"from":7,"to":2}]},{"pgid":"10.19","mappings":[{"from":4,"to":5}]},{"pgid":"10.35","mappings":[{"from":4,"to":3}]},{"pgid":"10.36","mappings":[{"from":7,"to":3}]},{"pgid":"10.37","mappings":[{"from":7,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.81:6836/442767245":"2024-02-22T23:05:28.786771+0000","172.21.15.28:6835/3468780363":"2024-02-22T23:05:17.415180+0000","172.21.15.28:6833/3468780363":"2024-02-22T23:05:17.415180+0000","172.21.15.81:6837/442767245":"2024-02-22T23:05:28.786771+0000","172.21.15.28:6834/3835332438":"2024-02-22T23:05:27.771834+0000","172.21.15.28:6832/3835332438":"2024-02-22T23:05:27.771834+0000","172.21.15.28:6834/2803823391":"2024-02-22T23:05:14.385354+0000","172.21.15.28:6832/2803823391":"2024-02-22T23:05:14.385354+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:05:31.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.266+0000 7fdd18c00640 1 -- 172.21.15.28:0/964512427 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdcf406b4d0 msgr2=0x7fdcf406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:31.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.266+0000 7fdd18c00640 1 --2- 172.21.15.28:0/964512427 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdcf406b4d0 0x7fdcf406d990 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fdd140c0ac0 tx=0x7fdd00009040 comp rx=0 tx=0).stop 2024-02-21T23:05:31.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.266+0000 7fdd18c00640 1 -- 172.21.15.28:0/964512427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd1406ee40 msgr2=0x7fdd14154ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:31.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.266+0000 7fdd18c00640 1 --2- 172.21.15.28:0/964512427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd1406ee40 0x7fdd14154ef0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fdd04009960 tx=0x7fdd040046c0 comp rx=0 tx=0).stop 2024-02-21T23:05:31.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.266+0000 7fdd18c00640 1 -- 172.21.15.28:0/964512427 shutdown_connections 2024-02-21T23:05:31.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.266+0000 7fdd18c00640 1 --2- 172.21.15.28:0/964512427 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdcf406b4d0 0x7fdcf406d990 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:31.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.266+0000 7fdd18c00640 1 --2- 172.21.15.28:0/964512427 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdd14150450 0x7fdd141e8a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:31.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.266+0000 7fdd18c00640 1 --2- 172.21.15.28:0/964512427 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdd14156af0 0x7fdd141e8fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:31.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.266+0000 7fdd18c00640 1 --2- 172.21.15.28:0/964512427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd1406ee40 0x7fdd14154ef0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:31.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.266+0000 7fdd18c00640 1 -- 172.21.15.28:0/964512427 >> 172.21.15.28:0/964512427 conn(0x7fdd1406b630 msgr2=0x7fdd1414f650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:31.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.267+0000 7fdd18c00640 1 -- 172.21.15.28:0/964512427 shutdown_connections 2024-02-21T23:05:31.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.267+0000 7fdd18c00640 1 -- 172.21.15.28:0/964512427 wait complete. 2024-02-21T23:05:31.275 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-21T23:02:30.728329+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs2_metadata', 'create_time': '2024-02-21T23:05:05.701276+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '56', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs2'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs2_data', 'create_time': '2024-02-21T23:05:06.904887+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '57', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs2'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-21T23:05:31.275 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-21T23:05:31.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.501+0000 7f325b000640 1 Processor -- start 2024-02-21T23:05:31.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.502+0000 7f325b000640 1 -- start start 2024-02-21T23:05:31.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.503+0000 7f325b000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f32541561a0 0x7f325406d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:31.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.503+0000 7f325b000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3254071970 con 0x7f32540c5a70 2024-02-21T23:05:31.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.503+0000 7f325b000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3254071c50 con 0x7f32540c22e0 2024-02-21T23:05:31.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.503+0000 7f325b000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3254072a10 con 0x7f32541561a0 2024-02-21T23:05:31.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.503+0000 7f3253e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f32540c5a70 0x7f3254155a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:44082/0 (socket says 172.21.15.28:44082) 2024-02-21T23:05:31.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.503+0000 7f3253e00640 1 -- 172.21.15.28:0/3370199438 learned_addr learned my addr 172.21.15.28:0/3370199438 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:05:31.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.504+0000 7f3252a00640 1 -- 172.21.15.28:0/3370199438 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 546045883 0 0) 0x7f3254071970 con 0x7f32540c5a70 2024-02-21T23:05:31.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.504+0000 7f3252a00640 1 -- 172.21.15.28:0/3370199438 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3234005730 con 0x7f32540c5a70 2024-02-21T23:05:31.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.504+0000 7f3259000640 1 --2- 172.21.15.28:0/3370199438 >> v2:172.21.15.81:3300/0 conn(0x7f32541561a0 0x7f325406d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:31.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.504+0000 7f3252a00640 1 -- 172.21.15.28:0/3370199438 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3384066328 0 0) 0x7f3234005730 con 0x7f32540c5a70 2024-02-21T23:05:31.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.504+0000 7f3252a00640 1 -- 172.21.15.28:0/3370199438 >> v2:172.21.15.81:3300/0 conn(0x7f32541561a0 msgr2=0x7f325406d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:31.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.504+0000 7f3252a00640 1 --2- 172.21.15.28:0/3370199438 >> v2:172.21.15.81:3300/0 conn(0x7f32541561a0 0x7f325406d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:31.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.504+0000 7f3252a00640 1 -- 172.21.15.28:0/3370199438 >> v1:172.21.15.81:6789/0 conn(0x7f32540c22e0 legacy=0x7f325406d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:31.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.504+0000 7f3252a00640 1 -- 172.21.15.28:0/3370199438 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3254071f30 con 0x7f32540c5a70 2024-02-21T23:05:31.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.504+0000 7f3252a00640 1 -- 172.21.15.28:0/3370199438 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f3244001670 con 0x7f32540c5a70 2024-02-21T23:05:31.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.504+0000 7f3252a00640 1 -- 172.21.15.28:0/3370199438 >> v1:172.21.15.28:6789/0 conn(0x7f32540c5a70 legacy=0x7f3254155a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:31.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.504+0000 7f3252a00640 1 --2- 172.21.15.28:0/3370199438 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3234005c10 0x7f3234006010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:31.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.504+0000 7f3252a00640 1 --2- 172.21.15.28:0/3370199438 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3234006a10 0x7f3234006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:31.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.504+0000 7f3252a00640 1 --2- 172.21.15.28:0/3370199438 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32340074a0 0x7f323400dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:31.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.504+0000 7f3252a00640 1 -- 172.21.15.28:0/3370199438 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3254072a10 con 0x7f32340074a0 2024-02-21T23:05:31.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.504+0000 7f3252a00640 1 -- 172.21.15.28:0/3370199438 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f323400e3f0 con 0x7f3234006a10 2024-02-21T23:05:31.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.504+0000 7f3252a00640 1 -- 172.21.15.28:0/3370199438 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f323400e5f0 con 0x7f3234005c10 2024-02-21T23:05:31.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.504+0000 7f3259000640 1 --2- 172.21.15.28:0/3370199438 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32340074a0 0x7f323400dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:31.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.505+0000 7f3253400640 1 --2- 172.21.15.28:0/3370199438 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3234006a10 0x7f3234006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:31.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.505+0000 7f3253e00640 1 --2- 172.21.15.28:0/3370199438 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3234005c10 0x7f3234006010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:31.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.505+0000 7f3259000640 1 -- 172.21.15.28:0/3370199438 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3234005c10 msgr2=0x7f3234006010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:31.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.505+0000 7f3259000640 1 --2- 172.21.15.28:0/3370199438 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3234005c10 0x7f3234006010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:31.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.505+0000 7f3259000640 1 -- 172.21.15.28:0/3370199438 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3234006a10 msgr2=0x7f3234006e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:31.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.505+0000 7f3259000640 1 --2- 172.21.15.28:0/3370199438 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3234006a10 0x7f3234006e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:31.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.505+0000 7f3259000640 1 -- 172.21.15.28:0/3370199438 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3234005730 con 0x7f32340074a0 2024-02-21T23:05:31.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.506+0000 7f3259000640 1 -- 172.21.15.28:0/3370199438 reap_dead start 2024-02-21T23:05:31.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.506+0000 7f3259000640 1 --2- 172.21.15.28:0/3370199438 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32340074a0 0x7f323400dbd0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f324c00ba30 tx=0x7f3234007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7105eb5d33285c0f server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:31.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.506+0000 7f3252a00640 1 -- 172.21.15.28:0/3370199438 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f324c036030 con 0x7f32340074a0 2024-02-21T23:05:31.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.506+0000 7f3252a00640 1 -- 172.21.15.28:0/3370199438 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f324c004020 con 0x7f32340074a0 2024-02-21T23:05:31.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.506+0000 7f3252a00640 1 -- 172.21.15.28:0/3370199438 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f324c00b600 con 0x7f32340074a0 2024-02-21T23:05:31.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.507+0000 7f325b000640 1 -- 172.21.15.28:0/3370199438 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32340074a0 msgr2=0x7f323400dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:31.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.507+0000 7f325b000640 1 --2- 172.21.15.28:0/3370199438 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32340074a0 0x7f323400dbd0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f324c00ba30 tx=0x7f3234007a40 comp rx=0 tx=0).stop 2024-02-21T23:05:31.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.507+0000 7f325b000640 1 -- 172.21.15.28:0/3370199438 shutdown_connections 2024-02-21T23:05:31.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.507+0000 7f325b000640 1 --2- 172.21.15.28:0/3370199438 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32340074a0 0x7f323400dbd0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:31.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.507+0000 7f325b000640 1 -- 172.21.15.28:0/3370199438 >> 172.21.15.28:0/3370199438 conn(0x7f32540a85a0 msgr2=0x7f32540a8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:31.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.507+0000 7f325b000640 1 -- 172.21.15.28:0/3370199438 shutdown_connections 2024-02-21T23:05:31.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.507+0000 7f325b000640 1 -- 172.21.15.28:0/3370199438 wait complete. 2024-02-21T23:05:31.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.509+0000 7f325b000640 1 Processor -- start 2024-02-21T23:05:31.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.509+0000 7f325b000640 1 -- start start 2024-02-21T23:05:31.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.510+0000 7f325b000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32340074a0 0x7f325406d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:31.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.510+0000 7f325b000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f32540bb810 0x7f325414ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:31.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.510+0000 7f325b000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f32540ab3a0 0x7f325414ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:31.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.510+0000 7f325b000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32540a6160 con 0x7f32340074a0 2024-02-21T23:05:31.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.510+0000 7f325b000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3254072580 con 0x7f32540bb810 2024-02-21T23:05:31.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.510+0000 7f325b000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32540a62e0 con 0x7f32540ab3a0 2024-02-21T23:05:31.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.510+0000 7f3253e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32340074a0 0x7f325406d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:31.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.510+0000 7f3253e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32340074a0 0x7f325406d370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40396/0 (socket says 172.21.15.28:40396) 2024-02-21T23:05:31.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.510+0000 7f3253e00640 1 -- 172.21.15.28:0/2457249679 learned_addr learned my addr 172.21.15.28:0/2457249679 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:31.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.510+0000 7f3259000640 1 --2- 172.21.15.28:0/2457249679 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f32540ab3a0 0x7f325414ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:31.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.510+0000 7f3253400640 1 --2- 172.21.15.28:0/2457249679 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f32540bb810 0x7f325414ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:31.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.511+0000 7f3253e00640 1 -- 172.21.15.28:0/2457249679 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f32540ab3a0 msgr2=0x7f325414ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:31.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.511+0000 7f3253e00640 1 --2- 172.21.15.28:0/2457249679 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f32540ab3a0 0x7f325414ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:31.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.511+0000 7f3253e00640 1 -- 172.21.15.28:0/2457249679 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f32540bb810 msgr2=0x7f325414ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:31.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.511+0000 7f3253e00640 1 --2- 172.21.15.28:0/2457249679 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f32540bb810 0x7f325414ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:31.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.511+0000 7f3253e00640 1 -- 172.21.15.28:0/2457249679 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32540bff50 con 0x7f32340074a0 2024-02-21T23:05:31.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.511+0000 7f3253400640 1 --2- 172.21.15.28:0/2457249679 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f32540bb810 0x7f325414ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:05:31.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.511+0000 7f3253e00640 1 --2- 172.21.15.28:0/2457249679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32340074a0 0x7f325406d370 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f324400d7d0 tx=0x7f324400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:31.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.511+0000 7f324be00640 1 -- 172.21.15.28:0/2457249679 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3244019070 con 0x7f32340074a0 2024-02-21T23:05:31.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.511+0000 7f324be00640 1 -- 172.21.15.28:0/2457249679 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32440074e0 con 0x7f32340074a0 2024-02-21T23:05:31.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.511+0000 7f324be00640 1 -- 172.21.15.28:0/2457249679 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3244004a00 con 0x7f32340074a0 2024-02-21T23:05:31.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.511+0000 7f325b000640 1 -- 172.21.15.28:0/2457249679 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32540aaf50 con 0x7f32340074a0 2024-02-21T23:05:31.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.511+0000 7f325b000640 1 -- 172.21.15.28:0/2457249679 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32540c01d0 con 0x7f32340074a0 2024-02-21T23:05:31.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.513+0000 7f325b000640 1 -- 172.21.15.28:0/2457249679 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3254067f30 con 0x7f32340074a0 2024-02-21T23:05:31.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.514+0000 7f324be00640 1 -- 172.21.15.28:0/2457249679 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f324401c070 con 0x7f32340074a0 2024-02-21T23:05:31.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.515+0000 7f324be00640 1 --2- 172.21.15.28:0/2457249679 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f323006b4d0 0x7f323006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:31.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.515+0000 7f324be00640 1 -- 172.21.15.28:0/2457249679 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6594+0+0 (secure 0 0 0) 0x7f3244089f10 con 0x7f32340074a0 2024-02-21T23:05:31.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.515+0000 7f3253400640 1 --2- 172.21.15.28:0/2457249679 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f323006b4d0 0x7f323006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:31.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.517+0000 7f3253400640 1 --2- 172.21.15.28:0/2457249679 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f323006b4d0 0x7f323006d990 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f323c007fd0 tx=0x7f323c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:31.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.519+0000 7f324be00640 1 -- 172.21.15.28:0/2457249679 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f32540bff50 con 0x7f32340074a0 2024-02-21T23:05:31.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.721+0000 7f325b000640 1 -- 172.21.15.28:0/2457249679 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f325406d8b0 con 0x7f32340074a0 2024-02-21T23:05:31.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.722+0000 7f324be00640 1 -- 172.21.15.28:0/2457249679 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v65) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3244058a10 con 0x7f32340074a0 2024-02-21T23:05:31.711 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-02-21T23:05:31.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.724+0000 7f325b000640 1 -- 172.21.15.28:0/2457249679 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f323006b4d0 msgr2=0x7f323006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:31.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.724+0000 7f325b000640 1 --2- 172.21.15.28:0/2457249679 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f323006b4d0 0x7f323006d990 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f323c007fd0 tx=0x7f323c009040 comp rx=0 tx=0).stop 2024-02-21T23:05:31.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.724+0000 7f325b000640 1 -- 172.21.15.28:0/2457249679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32340074a0 msgr2=0x7f325406d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:31.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.724+0000 7f325b000640 1 --2- 172.21.15.28:0/2457249679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32340074a0 0x7f325406d370 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f324400d7d0 tx=0x7f324400dc90 comp rx=0 tx=0).stop 2024-02-21T23:05:31.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.725+0000 7f325b000640 1 -- 172.21.15.28:0/2457249679 shutdown_connections 2024-02-21T23:05:31.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.725+0000 7f325b000640 1 --2- 172.21.15.28:0/2457249679 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f323006b4d0 0x7f323006d990 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:31.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.725+0000 7f325b000640 1 --2- 172.21.15.28:0/2457249679 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f32540ab3a0 0x7f325414ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:31.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.725+0000 7f325b000640 1 --2- 172.21.15.28:0/2457249679 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f32540bb810 0x7f325414ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:31.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.725+0000 7f325b000640 1 --2- 172.21.15.28:0/2457249679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32340074a0 0x7f325406d370 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:31.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.725+0000 7f325b000640 1 -- 172.21.15.28:0/2457249679 >> 172.21.15.28:0/2457249679 conn(0x7f32540a85a0 msgr2=0x7f32540a7040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:31.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.725+0000 7f325b000640 1 -- 172.21.15.28:0/2457249679 shutdown_connections 2024-02-21T23:05:31.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.725+0000 7f325b000640 1 -- 172.21.15.28:0/2457249679 wait complete. 2024-02-21T23:05:31.736 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs2_metadata pg_num 2024-02-21T23:05:31.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.967+0000 7faf52e00640 1 Processor -- start 2024-02-21T23:05:31.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.967+0000 7faf52e00640 1 -- start start 2024-02-21T23:05:31.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.969+0000 7faf52e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faf4c150240 0x7faf4c0c12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:31.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.969+0000 7faf52e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7faf4c0c4580 0x7faf4c06db10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:31.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.969+0000 7faf52e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf4c06f780 con 0x7faf4c0c2c10 2024-02-21T23:05:31.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.969+0000 7faf52e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf4c150620 con 0x7faf4c0c4580 2024-02-21T23:05:31.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.969+0000 7faf52e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf4c153a90 con 0x7faf4c150240 2024-02-21T23:05:31.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.969+0000 7faf51000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7faf4c0c4580 0x7faf4c06db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:31.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.969+0000 7faf51000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7faf4c0c4580 0x7faf4c06db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40398/0 (socket says 172.21.15.28:40398) 2024-02-21T23:05:31.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.969+0000 7faf51000640 1 -- 172.21.15.28:0/1431711506 learned_addr learned my addr 172.21.15.28:0/1431711506 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:31.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.970+0000 7faf51a00640 1 --2- 172.21.15.28:0/1431711506 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faf4c150240 0x7faf4c0c12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:31.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.970+0000 7faf51000640 1 -- 172.21.15.28:0/1431711506 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faf4c150240 msgr2=0x7faf4c0c12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:31.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.970+0000 7faf51000640 1 --2- 172.21.15.28:0/1431711506 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faf4c150240 0x7faf4c0c12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:31.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.970+0000 7faf51000640 1 -- 172.21.15.28:0/1431711506 >> v1:172.21.15.81:6789/0 conn(0x7faf4c0c2c10 legacy=0x7faf4c06ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:31.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.970+0000 7faf51000640 1 -- 172.21.15.28:0/1431711506 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf4c06f4a0 con 0x7faf4c0c4580 2024-02-21T23:05:31.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.971+0000 7faf51000640 1 --2- 172.21.15.28:0/1431711506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf4c0c4580 0x7faf4c06db10 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7faf38009870 tx=0x7faf3802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f027c99047a8fdbf server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:31.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.972+0000 7faf4be00640 1 -- 172.21.15.28:0/1431711506 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf380317c0 con 0x7faf4c0c4580 2024-02-21T23:05:31.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.972+0000 7faf4be00640 1 -- 172.21.15.28:0/1431711506 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf38037070 con 0x7faf4c0c4580 2024-02-21T23:05:31.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.972+0000 7faf4be00640 1 -- 172.21.15.28:0/1431711506 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf38031b80 con 0x7faf4c0c4580 2024-02-21T23:05:31.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.972+0000 7faf52e00640 1 -- 172.21.15.28:0/1431711506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf4c0c4580 msgr2=0x7faf4c06db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:31.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.972+0000 7faf52e00640 1 --2- 172.21.15.28:0/1431711506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf4c0c4580 0x7faf4c06db10 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7faf38009870 tx=0x7faf3802b860 comp rx=0 tx=0).stop 2024-02-21T23:05:31.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.973+0000 7faf52e00640 1 -- 172.21.15.28:0/1431711506 shutdown_connections 2024-02-21T23:05:31.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.973+0000 7faf52e00640 1 --2- 172.21.15.28:0/1431711506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf4c0c4580 0x7faf4c06db10 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:31.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.973+0000 7faf52e00640 1 --2- 172.21.15.28:0/1431711506 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faf4c150240 0x7faf4c0c12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:31.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.973+0000 7faf52e00640 1 -- 172.21.15.28:0/1431711506 >> 172.21.15.28:0/1431711506 conn(0x7faf4c0a8530 msgr2=0x7faf4c0a8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:31.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.973+0000 7faf52e00640 1 -- 172.21.15.28:0/1431711506 shutdown_connections 2024-02-21T23:05:31.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.973+0000 7faf52e00640 1 -- 172.21.15.28:0/1431711506 wait complete. 2024-02-21T23:05:31.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.974+0000 7faf52e00640 1 Processor -- start 2024-02-21T23:05:31.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.975+0000 7faf52e00640 1 -- start start 2024-02-21T23:05:31.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.975+0000 7faf52e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf4c0c2c10 0x7faf4c0aa300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:31.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.975+0000 7faf52e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faf4c0c4580 0x7faf4c14ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:31.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.975+0000 7faf52e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf4c150240 0x7faf4c1ebf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:31.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.975+0000 7faf52e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf4c0c4b90 con 0x7faf4c0c2c10 2024-02-21T23:05:31.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.975+0000 7faf52e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf4c0c4a10 con 0x7faf4c150240 2024-02-21T23:05:31.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.975+0000 7faf52e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf4c06b580 con 0x7faf4c0c4580 2024-02-21T23:05:31.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.976+0000 7faf51a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf4c0c2c10 0x7faf4c0aa300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:31.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.976+0000 7faf51a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf4c0c2c10 0x7faf4c0aa300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40406/0 (socket says 172.21.15.28:40406) 2024-02-21T23:05:31.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.976+0000 7faf51a00640 1 -- 172.21.15.28:0/2540116665 learned_addr learned my addr 172.21.15.28:0/2540116665 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:31.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.976+0000 7faf51000640 1 --2- 172.21.15.28:0/2540116665 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faf4c0c4580 0x7faf4c14ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:31.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.976+0000 7faf51a00640 1 -- 172.21.15.28:0/2540116665 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faf4c0c4580 msgr2=0x7faf4c14ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:31.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.976+0000 7faf51a00640 1 --2- 172.21.15.28:0/2540116665 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faf4c0c4580 0x7faf4c14ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:31.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.976+0000 7faf51a00640 1 -- 172.21.15.28:0/2540116665 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf4c150240 msgr2=0x7faf4c1ebf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:31.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.976+0000 7faf52400640 1 --2- 172.21.15.28:0/2540116665 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf4c150240 0x7faf4c1ebf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:31.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.976+0000 7faf51a00640 1 --2- 172.21.15.28:0/2540116665 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf4c150240 0x7faf4c1ebf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:31.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.976+0000 7faf51a00640 1 -- 172.21.15.28:0/2540116665 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf4c14b760 con 0x7faf4c0c2c10 2024-02-21T23:05:31.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.976+0000 7faf51000640 1 --2- 172.21.15.28:0/2540116665 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faf4c0c4580 0x7faf4c14ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:05:31.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.976+0000 7faf52400640 1 --2- 172.21.15.28:0/2540116665 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf4c150240 0x7faf4c1ebf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:05:31.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.977+0000 7faf51a00640 1 --2- 172.21.15.28:0/2540116665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf4c0c2c10 0x7faf4c0aa300 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7faf4000e9b0 tx=0x7faf4000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:31.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.977+0000 7faf49600640 1 -- 172.21.15.28:0/2540116665 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf40017070 con 0x7faf4c0c2c10 2024-02-21T23:05:31.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.977+0000 7faf49600640 1 -- 172.21.15.28:0/2540116665 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf4000c9a0 con 0x7faf4c0c2c10 2024-02-21T23:05:31.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.977+0000 7faf49600640 1 -- 172.21.15.28:0/2540116665 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf4000cd90 con 0x7faf4c0c2c10 2024-02-21T23:05:31.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.977+0000 7faf52e00640 1 -- 172.21.15.28:0/2540116665 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf4c154000 con 0x7faf4c0c2c10 2024-02-21T23:05:31.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.977+0000 7faf52e00640 1 -- 172.21.15.28:0/2540116665 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf4c1528c0 con 0x7faf4c0c2c10 2024-02-21T23:05:31.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.978+0000 7faf52e00640 1 -- 172.21.15.28:0/2540116665 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf4c04f290 con 0x7faf4c0c2c10 2024-02-21T23:05:31.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.979+0000 7faf49600640 1 -- 172.21.15.28:0/2540116665 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faf4001a070 con 0x7faf4c0c2c10 2024-02-21T23:05:31.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.981+0000 7faf49600640 1 --2- 172.21.15.28:0/2540116665 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7faf2c06b4d0 0x7faf2c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:31.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.981+0000 7faf49600640 1 -- 172.21.15.28:0/2540116665 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6594+0+0 (secure 0 0 0) 0x7faf4008a5d0 con 0x7faf4c0c2c10 2024-02-21T23:05:31.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.981+0000 7faf51000640 1 --2- 172.21.15.28:0/2540116665 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7faf2c06b4d0 0x7faf2c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:31.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.983+0000 7faf51000640 1 --2- 172.21.15.28:0/2540116665 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7faf2c06b4d0 0x7faf2c06d990 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7faf38036000 tx=0x7faf38008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:31.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:31.985+0000 7faf49600640 1 -- 172.21.15.28:0/2540116665 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7faf400590c0 con 0x7faf4c0c2c10 2024-02-21T23:05:32.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.184+0000 7faf52e00640 1 -- 172.21.15.28:0/2540116665 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs2_metadata", "var": "pg_num"} v 0) v1 -- 0x7faf4c06f0e0 con 0x7faf4c0c2c10 2024-02-21T23:05:32.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.185+0000 7faf49600640 1 -- 172.21.15.28:0/2540116665 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs2_metadata", "var": "pg_num"}]=0 v65) v1 ==== 105+0+11 (secure 0 0 0) 0x7faf400589c0 con 0x7faf4c0c2c10 2024-02-21T23:05:32.175 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-02-21T23:05:32.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.187+0000 7faf52e00640 1 -- 172.21.15.28:0/2540116665 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7faf2c06b4d0 msgr2=0x7faf2c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:32.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.187+0000 7faf52e00640 1 --2- 172.21.15.28:0/2540116665 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7faf2c06b4d0 0x7faf2c06d990 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7faf38036000 tx=0x7faf38008030 comp rx=0 tx=0).stop 2024-02-21T23:05:32.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.187+0000 7faf52e00640 1 -- 172.21.15.28:0/2540116665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf4c0c2c10 msgr2=0x7faf4c0aa300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:32.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.187+0000 7faf52e00640 1 --2- 172.21.15.28:0/2540116665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf4c0c2c10 0x7faf4c0aa300 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7faf4000e9b0 tx=0x7faf4000ee70 comp rx=0 tx=0).stop 2024-02-21T23:05:32.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.188+0000 7faf52e00640 1 -- 172.21.15.28:0/2540116665 shutdown_connections 2024-02-21T23:05:32.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.188+0000 7faf52e00640 1 --2- 172.21.15.28:0/2540116665 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7faf2c06b4d0 0x7faf2c06d990 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:32.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.188+0000 7faf52e00640 1 --2- 172.21.15.28:0/2540116665 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faf4c0c4580 0x7faf4c14ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:32.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.188+0000 7faf52e00640 1 --2- 172.21.15.28:0/2540116665 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf4c150240 0x7faf4c1ebf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:32.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.188+0000 7faf52e00640 1 --2- 172.21.15.28:0/2540116665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf4c0c2c10 0x7faf4c0aa300 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:32.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.188+0000 7faf52e00640 1 -- 172.21.15.28:0/2540116665 >> 172.21.15.28:0/2540116665 conn(0x7faf4c0a8530 msgr2=0x7faf4c14c5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:32.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.188+0000 7faf52e00640 1 -- 172.21.15.28:0/2540116665 shutdown_connections 2024-02-21T23:05:32.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.188+0000 7faf52e00640 1 -- 172.21.15.28:0/2540116665 wait complete. 2024-02-21T23:05:32.199 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs2_data pg_num 2024-02-21T23:05:32.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.438+0000 7fb39a000640 1 Processor -- start 2024-02-21T23:05:32.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.439+0000 7fb39a000640 1 -- start start 2024-02-21T23:05:32.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.440+0000 7fb39a000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb3941525a0 0x7fb3940c86a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:32.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.440+0000 7fb39a000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fb39406dd20 0x7fb394155920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:32.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.440+0000 7fb39a000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3940c5120 con 0x7fb394158cf0 2024-02-21T23:05:32.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.440+0000 7fb39a000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb394155e60 con 0x7fb3941525a0 2024-02-21T23:05:32.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.440+0000 7fb39a000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb394158530 con 0x7fb39406dd20 2024-02-21T23:05:32.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.441+0000 7fb398c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb3941525a0 0x7fb3940c86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:32.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.441+0000 7fb398c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb3941525a0 0x7fb3940c86a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40414/0 (socket says 172.21.15.28:40414) 2024-02-21T23:05:32.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.441+0000 7fb398c00640 1 -- 172.21.15.28:0/3134710793 learned_addr learned my addr 172.21.15.28:0/3134710793 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:32.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.441+0000 7fb393e00640 1 --2- 172.21.15.28:0/3134710793 >> v2:172.21.15.81:3300/0 conn(0x7fb39406dd20 0x7fb394155920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:32.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.441+0000 7fb398c00640 1 -- 172.21.15.28:0/3134710793 >> v2:172.21.15.81:3300/0 conn(0x7fb39406dd20 msgr2=0x7fb394155920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:32.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.441+0000 7fb398c00640 1 --2- 172.21.15.28:0/3134710793 >> v2:172.21.15.81:3300/0 conn(0x7fb39406dd20 0x7fb394155920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:32.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.441+0000 7fb398c00640 1 -- 172.21.15.28:0/3134710793 >> v1:172.21.15.28:6789/0 conn(0x7fb394158cf0 legacy=0x7fb3940c7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:32.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.441+0000 7fb398c00640 1 -- 172.21.15.28:0/3134710793 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3940c53a0 con 0x7fb3941525a0 2024-02-21T23:05:32.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.441+0000 7fb393400640 1 -- 172.21.15.28:0/3134710793 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4104762830 0 0) 0x7fb38c001490 con 0x7fb394158cf0 2024-02-21T23:05:32.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.443+0000 7fb398c00640 1 --2- 172.21.15.28:0/3134710793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3941525a0 0x7fb3940c86a0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fb39406d2e0 tx=0x7fb38802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20ba72b15e965f08 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:32.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.443+0000 7fb393400640 1 -- 172.21.15.28:0/3134710793 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb388035020 con 0x7fb3941525a0 2024-02-21T23:05:32.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.443+0000 7fb393400640 1 -- 172.21.15.28:0/3134710793 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb388031870 con 0x7fb3941525a0 2024-02-21T23:05:32.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.443+0000 7fb393400640 1 -- 172.21.15.28:0/3134710793 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb388031c40 con 0x7fb3941525a0 2024-02-21T23:05:32.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.444+0000 7fb39a000640 1 -- 172.21.15.28:0/3134710793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3941525a0 msgr2=0x7fb3940c86a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:32.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.444+0000 7fb39a000640 1 --2- 172.21.15.28:0/3134710793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3941525a0 0x7fb3940c86a0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fb39406d2e0 tx=0x7fb38802b910 comp rx=0 tx=0).stop 2024-02-21T23:05:32.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.444+0000 7fb39a000640 1 -- 172.21.15.28:0/3134710793 shutdown_connections 2024-02-21T23:05:32.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.444+0000 7fb39a000640 1 --2- 172.21.15.28:0/3134710793 >> v2:172.21.15.81:3300/0 conn(0x7fb39406dd20 0x7fb394155920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:32.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.444+0000 7fb39a000640 1 --2- 172.21.15.28:0/3134710793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3941525a0 0x7fb3940c86a0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:32.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.444+0000 7fb39a000640 1 -- 172.21.15.28:0/3134710793 >> 172.21.15.28:0/3134710793 conn(0x7fb3940ac4d0 msgr2=0x7fb3940ac8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:32.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.444+0000 7fb39a000640 1 -- 172.21.15.28:0/3134710793 shutdown_connections 2024-02-21T23:05:32.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.444+0000 7fb39a000640 1 -- 172.21.15.28:0/3134710793 wait complete. 2024-02-21T23:05:32.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.446+0000 7fb39a000640 1 Processor -- start 2024-02-21T23:05:32.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.446+0000 7fb39a000640 1 -- start start 2024-02-21T23:05:32.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.447+0000 7fb39a000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb39406dd20 0x7fb3941eb5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:32.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.447+0000 7fb39a000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb3941525a0 0x7fb3941470e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:32.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.447+0000 7fb39a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb394158cf0 0x7fb394156790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:32.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.447+0000 7fb39a000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3940c5960 con 0x7fb394158cf0 2024-02-21T23:05:32.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.447+0000 7fb39a000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3941587d0 con 0x7fb39406dd20 2024-02-21T23:05:32.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.447+0000 7fb39a000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3940c5ae0 con 0x7fb3941525a0 2024-02-21T23:05:32.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.448+0000 7fb399600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb394158cf0 0x7fb394156790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:32.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.448+0000 7fb399600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb394158cf0 0x7fb394156790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40428/0 (socket says 172.21.15.28:40428) 2024-02-21T23:05:32.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.448+0000 7fb399600640 1 -- 172.21.15.28:0/3314317998 learned_addr learned my addr 172.21.15.28:0/3314317998 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:32.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.448+0000 7fb393e00640 1 --2- 172.21.15.28:0/3314317998 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb3941525a0 0x7fb3941470e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:32.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.448+0000 7fb399600640 1 -- 172.21.15.28:0/3314317998 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb3941525a0 msgr2=0x7fb3941470e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:32.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.448+0000 7fb399600640 1 --2- 172.21.15.28:0/3314317998 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb3941525a0 0x7fb3941470e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:32.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.448+0000 7fb399600640 1 -- 172.21.15.28:0/3314317998 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb39406dd20 msgr2=0x7fb3941eb5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:32.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.448+0000 7fb398c00640 1 --2- 172.21.15.28:0/3314317998 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb39406dd20 0x7fb3941eb5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:32.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.448+0000 7fb399600640 1 --2- 172.21.15.28:0/3314317998 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb39406dd20 0x7fb3941eb5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:32.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.448+0000 7fb399600640 1 -- 172.21.15.28:0/3314317998 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb394157950 con 0x7fb394158cf0 2024-02-21T23:05:32.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.449+0000 7fb399600640 1 --2- 172.21.15.28:0/3314317998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb394158cf0 0x7fb394156790 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fb38c00c780 tx=0x7fb38c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:32.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.449+0000 7fb390c00640 1 -- 172.21.15.28:0/3314317998 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb38c003b90 con 0x7fb394158cf0 2024-02-21T23:05:32.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.449+0000 7fb390c00640 1 -- 172.21.15.28:0/3314317998 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb38c003d30 con 0x7fb394158cf0 2024-02-21T23:05:32.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.449+0000 7fb39a000640 1 -- 172.21.15.28:0/3314317998 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb394157c30 con 0x7fb394158cf0 2024-02-21T23:05:32.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.449+0000 7fb390c00640 1 -- 172.21.15.28:0/3314317998 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb38c00aa90 con 0x7fb394158cf0 2024-02-21T23:05:32.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.449+0000 7fb39a000640 1 -- 172.21.15.28:0/3314317998 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb394147620 con 0x7fb394158cf0 2024-02-21T23:05:32.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.451+0000 7fb39a000640 1 -- 172.21.15.28:0/3314317998 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb394067670 con 0x7fb394158cf0 2024-02-21T23:05:32.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.451+0000 7fb390c00640 1 -- 172.21.15.28:0/3314317998 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb38c00ac30 con 0x7fb394158cf0 2024-02-21T23:05:32.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.452+0000 7fb390c00640 1 --2- 172.21.15.28:0/3314317998 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb37406b4d0 0x7fb37406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:32.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.453+0000 7fb390c00640 1 -- 172.21.15.28:0/3314317998 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6594+0+0 (secure 0 0 0) 0x7fb38c08a530 con 0x7fb394158cf0 2024-02-21T23:05:32.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.453+0000 7fb398c00640 1 --2- 172.21.15.28:0/3314317998 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb37406b4d0 0x7fb37406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:32.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.455+0000 7fb398c00640 1 --2- 172.21.15.28:0/3314317998 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb37406b4d0 0x7fb37406d990 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fb388036000 tx=0x7fb388008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:32.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.458+0000 7fb390c00640 1 -- 172.21.15.28:0/3314317998 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb394067670 con 0x7fb394158cf0 2024-02-21T23:05:32.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.670+0000 7fb39a000640 1 -- 172.21.15.28:0/3314317998 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs2_data", "var": "pg_num"} v 0) v1 -- 0x7fb39404f290 con 0x7fb394158cf0 2024-02-21T23:05:32.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.671+0000 7fb390c00640 1 -- 172.21.15.28:0/3314317998 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs2_data", "var": "pg_num"}]=0 v65) v1 ==== 101+0+11 (secure 0 0 0) 0x7fb3940c5120 con 0x7fb394158cf0 2024-02-21T23:05:32.660 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-02-21T23:05:32.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.673+0000 7fb39a000640 1 -- 172.21.15.28:0/3314317998 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb37406b4d0 msgr2=0x7fb37406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:32.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.673+0000 7fb39a000640 1 --2- 172.21.15.28:0/3314317998 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb37406b4d0 0x7fb37406d990 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fb388036000 tx=0x7fb388008030 comp rx=0 tx=0).stop 2024-02-21T23:05:32.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.673+0000 7fb39a000640 1 -- 172.21.15.28:0/3314317998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb394158cf0 msgr2=0x7fb394156790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:32.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.673+0000 7fb39a000640 1 --2- 172.21.15.28:0/3314317998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb394158cf0 0x7fb394156790 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fb38c00c780 tx=0x7fb38c00cc40 comp rx=0 tx=0).stop 2024-02-21T23:05:32.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.674+0000 7fb39a000640 1 -- 172.21.15.28:0/3314317998 shutdown_connections 2024-02-21T23:05:32.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.674+0000 7fb39a000640 1 --2- 172.21.15.28:0/3314317998 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb37406b4d0 0x7fb37406d990 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:32.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.674+0000 7fb39a000640 1 --2- 172.21.15.28:0/3314317998 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb3941525a0 0x7fb3941470e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:32.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.674+0000 7fb39a000640 1 --2- 172.21.15.28:0/3314317998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb394158cf0 0x7fb394156790 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:32.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.674+0000 7fb39a000640 1 --2- 172.21.15.28:0/3314317998 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb39406dd20 0x7fb3941eb5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:32.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.674+0000 7fb39a000640 1 -- 172.21.15.28:0/3314317998 >> 172.21.15.28:0/3314317998 conn(0x7fb3940ac4d0 msgr2=0x7fb39414aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:32.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.674+0000 7fb39a000640 1 -- 172.21.15.28:0/3314317998 shutdown_connections 2024-02-21T23:05:32.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.674+0000 7fb39a000640 1 -- 172.21.15.28:0/3314317998 wait complete. 2024-02-21T23:05:32.682 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:05:32.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.918+0000 7f42e6a00640 1 Processor -- start 2024-02-21T23:05:32.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.918+0000 7f42e6a00640 1 -- start start 2024-02-21T23:05:32.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.919+0000 7f42e6a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f42e00c7180 0x7f42e00c75e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:32.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.919+0000 7f42e6a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f42e01587f0 0x7f42e0158c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:32.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.919+0000 7f42e6a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42e0152c80 con 0x7f42e01528a0 2024-02-21T23:05:32.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.920+0000 7f42e6a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42e0155300 con 0x7f42e01587f0 2024-02-21T23:05:32.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.920+0000 7f42e6a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42e01555e0 con 0x7f42e00c7180 2024-02-21T23:05:32.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.920+0000 7f42dfe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f42e01528a0 0x7f42e00c6650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:44104/0 (socket says 172.21.15.28:44104) 2024-02-21T23:05:32.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.920+0000 7f42dfe00640 1 -- 172.21.15.28:0/403039031 learned_addr learned my addr 172.21.15.28:0/403039031 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:05:32.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.920+0000 7f42e4a00640 1 --2- 172.21.15.28:0/403039031 >> v2:172.21.15.28:3300/0 conn(0x7f42e01587f0 0x7f42e0158c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:32.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.920+0000 7f42d7e00640 1 --2- 172.21.15.28:0/403039031 >> v2:172.21.15.81:3300/0 conn(0x7f42e00c7180 0x7f42e00c75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:32.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.920+0000 7f42df400640 1 -- 172.21.15.28:0/403039031 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 572414621 0 0) 0x7f42e0152c80 con 0x7f42e01528a0 2024-02-21T23:05:32.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.921+0000 7f42e4a00640 1 -- 172.21.15.28:0/403039031 >> v2:172.21.15.81:3300/0 conn(0x7f42e00c7180 msgr2=0x7f42e00c75e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:32.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.921+0000 7f42e4a00640 1 --2- 172.21.15.28:0/403039031 >> v2:172.21.15.81:3300/0 conn(0x7f42e00c7180 0x7f42e00c75e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:32.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.921+0000 7f42e4a00640 1 -- 172.21.15.28:0/403039031 >> v1:172.21.15.28:6789/0 conn(0x7f42e01528a0 legacy=0x7f42e00c6650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:32.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.921+0000 7f42e4a00640 1 -- 172.21.15.28:0/403039031 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42e01558c0 con 0x7f42e01587f0 2024-02-21T23:05:32.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.921+0000 7f42d7e00640 1 --2- 172.21.15.28:0/403039031 >> v2:172.21.15.81:3300/0 conn(0x7f42e00c7180 0x7f42e00c75e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:05:32.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.922+0000 7f42e4a00640 1 --2- 172.21.15.28:0/403039031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42e01587f0 0x7f42e0158c20 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f42e0067ac0 tx=0x7f42d802be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d0d747cc8800296 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:32.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.922+0000 7f42df400640 1 -- 172.21.15.28:0/403039031 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42d8039070 con 0x7f42e01587f0 2024-02-21T23:05:32.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.922+0000 7f42df400640 1 -- 172.21.15.28:0/403039031 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42d802cbe0 con 0x7f42e01587f0 2024-02-21T23:05:32.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.922+0000 7f42df400640 1 -- 172.21.15.28:0/403039031 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42d8034440 con 0x7f42e01587f0 2024-02-21T23:05:32.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.923+0000 7f42e6a00640 1 -- 172.21.15.28:0/403039031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42e01587f0 msgr2=0x7f42e0158c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:32.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.923+0000 7f42e6a00640 1 --2- 172.21.15.28:0/403039031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42e01587f0 0x7f42e0158c20 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f42e0067ac0 tx=0x7f42d802be30 comp rx=0 tx=0).stop 2024-02-21T23:05:32.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.923+0000 7f42e6a00640 1 -- 172.21.15.28:0/403039031 shutdown_connections 2024-02-21T23:05:32.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.923+0000 7f42e6a00640 1 --2- 172.21.15.28:0/403039031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42e01587f0 0x7f42e0158c20 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:32.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.923+0000 7f42e6a00640 1 --2- 172.21.15.28:0/403039031 >> v2:172.21.15.81:3300/0 conn(0x7f42e00c7180 0x7f42e00c75e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:32.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.923+0000 7f42e6a00640 1 -- 172.21.15.28:0/403039031 >> 172.21.15.28:0/403039031 conn(0x7f42e00aa4c0 msgr2=0x7f42e00aa8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:32.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.923+0000 7f42e6a00640 1 -- 172.21.15.28:0/403039031 shutdown_connections 2024-02-21T23:05:32.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.923+0000 7f42e6a00640 1 -- 172.21.15.28:0/403039031 wait complete. 2024-02-21T23:05:32.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.925+0000 7f42e6a00640 1 Processor -- start 2024-02-21T23:05:32.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.926+0000 7f42e6a00640 1 -- start start 2024-02-21T23:05:32.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.927+0000 7f42e6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42e00c7180 0x7f42e01580b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:32.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.927+0000 7f42e6a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f42e01528a0 0x7f42e006cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:32.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.927+0000 7f42e6a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f42e01587f0 0x7f42e01ee040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:32.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.927+0000 7f42e6a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42e00a8170 con 0x7f42e00c7180 2024-02-21T23:05:32.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.927+0000 7f42e6a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42e0155e80 con 0x7f42e01587f0 2024-02-21T23:05:32.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.927+0000 7f42e6a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42e00c6d60 con 0x7f42e01528a0 2024-02-21T23:05:32.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.927+0000 7f42dfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42e00c7180 0x7f42e01580b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:32.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.927+0000 7f42dfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42e00c7180 0x7f42e01580b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40454/0 (socket says 172.21.15.28:40454) 2024-02-21T23:05:32.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.927+0000 7f42dfe00640 1 -- 172.21.15.28:0/638778625 learned_addr learned my addr 172.21.15.28:0/638778625 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:32.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.928+0000 7f42d7e00640 1 --2- 172.21.15.28:0/638778625 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f42e01528a0 0x7f42e006cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:32.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.928+0000 7f42e4a00640 1 --2- 172.21.15.28:0/638778625 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f42e01587f0 0x7f42e01ee040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:32.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.928+0000 7f42dfe00640 1 -- 172.21.15.28:0/638778625 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f42e01528a0 msgr2=0x7f42e006cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:32.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.928+0000 7f42dfe00640 1 --2- 172.21.15.28:0/638778625 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f42e01528a0 0x7f42e006cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:32.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.928+0000 7f42dfe00640 1 -- 172.21.15.28:0/638778625 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f42e01587f0 msgr2=0x7f42e01ee040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:32.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.928+0000 7f42dfe00640 1 --2- 172.21.15.28:0/638778625 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f42e01587f0 0x7f42e01ee040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:32.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.928+0000 7f42dfe00640 1 -- 172.21.15.28:0/638778625 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42e01585f0 con 0x7f42e00c7180 2024-02-21T23:05:32.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.928+0000 7f42d7e00640 1 --2- 172.21.15.28:0/638778625 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f42e01528a0 0x7f42e006cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:05:32.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.928+0000 7f42e4a00640 1 --2- 172.21.15.28:0/638778625 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f42e01587f0 0x7f42e01ee040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:05:32.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.928+0000 7f42dfe00640 1 --2- 172.21.15.28:0/638778625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42e00c7180 0x7f42e01580b0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f42d000c430 tx=0x7f42d000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:32.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.928+0000 7f42dcc00640 1 -- 172.21.15.28:0/638778625 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42d00074e0 con 0x7f42e00c7180 2024-02-21T23:05:32.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.928+0000 7f42dcc00640 1 -- 172.21.15.28:0/638778625 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42d00047e0 con 0x7f42e00c7180 2024-02-21T23:05:32.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.928+0000 7f42dcc00640 1 -- 172.21.15.28:0/638778625 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42d0004bd0 con 0x7f42e00c7180 2024-02-21T23:05:32.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.928+0000 7f42e6a00640 1 -- 172.21.15.28:0/638778625 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42e00a8ea0 con 0x7f42e00c7180 2024-02-21T23:05:32.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.929+0000 7f42e6a00640 1 -- 172.21.15.28:0/638778625 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42e01567c0 con 0x7f42e00c7180 2024-02-21T23:05:32.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.930+0000 7f42e6a00640 1 -- 172.21.15.28:0/638778625 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42e00a9d30 con 0x7f42e00c7180 2024-02-21T23:05:32.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.931+0000 7f42dcc00640 1 -- 172.21.15.28:0/638778625 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f42d0011070 con 0x7f42e00c7180 2024-02-21T23:05:32.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.932+0000 7f42dcc00640 1 --2- 172.21.15.28:0/638778625 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f42bc06b4d0 0x7f42bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:32.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.932+0000 7f42dcc00640 1 -- 172.21.15.28:0/638778625 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6594+0+0 (secure 0 0 0) 0x7f42d008a200 con 0x7f42e00c7180 2024-02-21T23:05:32.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.933+0000 7f42d7e00640 1 --2- 172.21.15.28:0/638778625 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f42bc06b4d0 0x7f42bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:32.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.934+0000 7f42d7e00640 1 --2- 172.21.15.28:0/638778625 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f42bc06b4d0 0x7f42bc06d990 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f42e014f1d0 tx=0x7f42c8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:32.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:32.937+0000 7f42dcc00640 1 -- 172.21.15.28:0/638778625 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f42d0058cf0 con 0x7f42e00c7180 2024-02-21T23:05:33.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.173+0000 7f42e6a00640 1 -- 172.21.15.28:0/638778625 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f42e00c8f10 con 0x7f42e00c7180 2024-02-21T23:05:33.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.174+0000 7f42dcc00640 1 -- 172.21.15.28:0/638778625 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+4940 (secure 0 0 0) 0x7f42d00585f0 con 0x7f42e00c7180 2024-02-21T23:05:33.164 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:05:33.164 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":53,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5428,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/2146447666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":2146447666},{"type":"v1","addr":"172.21.15.81:6837","nonce":2146447666}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":6039,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3807390545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3807390545},{"type":"v1","addr":"172.21.15.28:6835","nonce":3807390545}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45},{"gid":6147,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/2305639796","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2305639796},{"type":"v1","addr":"172.21.15.28:6834","nonce":2305639796}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:05:08.017586+0000","modified":"2024-02-21T23:05:30.816087+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5221},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5221":{"gid":5221,"name":"b","rank":0,"incarnation":50,"state":"up:active","state_seq":14,"addr":"172.21.15.81:6835/2152160500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2152160500},{"type":"v1","addr":"172.21.15.81:6835","nonce":2152160500}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-02-21T23:05:33.165 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 53 2024-02-21T23:05:33.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.177+0000 7f42e6a00640 1 -- 172.21.15.28:0/638778625 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f42bc06b4d0 msgr2=0x7f42bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:33.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.177+0000 7f42e6a00640 1 --2- 172.21.15.28:0/638778625 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f42bc06b4d0 0x7f42bc06d990 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f42e014f1d0 tx=0x7f42c8008040 comp rx=0 tx=0).stop 2024-02-21T23:05:33.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.177+0000 7f42e6a00640 1 -- 172.21.15.28:0/638778625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42e00c7180 msgr2=0x7f42e01580b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:33.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.177+0000 7f42e6a00640 1 --2- 172.21.15.28:0/638778625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42e00c7180 0x7f42e01580b0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f42d000c430 tx=0x7f42d000c8f0 comp rx=0 tx=0).stop 2024-02-21T23:05:33.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.177+0000 7f42e6a00640 1 -- 172.21.15.28:0/638778625 shutdown_connections 2024-02-21T23:05:33.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.177+0000 7f42e6a00640 1 --2- 172.21.15.28:0/638778625 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f42bc06b4d0 0x7f42bc06d990 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:33.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.177+0000 7f42e6a00640 1 --2- 172.21.15.28:0/638778625 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f42e01528a0 0x7f42e006cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:33.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.177+0000 7f42e6a00640 1 --2- 172.21.15.28:0/638778625 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f42e01587f0 0x7f42e01ee040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:33.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.177+0000 7f42e6a00640 1 --2- 172.21.15.28:0/638778625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42e00c7180 0x7f42e01580b0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:33.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.177+0000 7f42e6a00640 1 -- 172.21.15.28:0/638778625 >> 172.21.15.28:0/638778625 conn(0x7f42e00aa4c0 msgr2=0x7f42e00aa8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:33.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.177+0000 7f42e6a00640 1 -- 172.21.15.28:0/638778625 shutdown_connections 2024-02-21T23:05:33.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.177+0000 7f42e6a00640 1 -- 172.21.15.28:0/638778625 wait complete. 2024-02-21T23:05:33.185 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:05:33.349 INFO:tasks.ceph.mds.b.smithi081.stderr:2024-02-21T23:05:33.360+0000 7f02a6200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-21T23:05:33.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.415+0000 7f619f200640 1 Processor -- start 2024-02-21T23:05:33.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.416+0000 7f619f200640 1 -- start start 2024-02-21T23:05:33.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.417+0000 7f619f200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6198150450 0x7f61980a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:33.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.417+0000 7f619f200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f619806ee40 0x7f61981536f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:33.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.417+0000 7f619f200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f6198156af0 0x7f619806dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:33.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.417+0000 7f619f200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61980c22c0 con 0x7f619806ee40 2024-02-21T23:05:33.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.417+0000 7f619f200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6198156660 con 0x7f6198156af0 2024-02-21T23:05:33.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.417+0000 7f619f200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61980c2760 con 0x7f6198150450 2024-02-21T23:05:33.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.418+0000 7f619d400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f619806ee40 0x7f61981536f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:33.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.418+0000 7f619d400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f619806ee40 0x7f61981536f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40456/0 (socket says 172.21.15.28:40456) 2024-02-21T23:05:33.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.418+0000 7f619d400640 1 -- 172.21.15.28:0/1706347619 learned_addr learned my addr 172.21.15.28:0/1706347619 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:33.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.418+0000 7f619e800640 1 --2- 172.21.15.28:0/1706347619 >> v2:172.21.15.81:3300/0 conn(0x7f6198156af0 0x7f619806dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:33.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.418+0000 7f619de00640 1 --2- 172.21.15.28:0/1706347619 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6198150450 0x7f61980a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:33.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.418+0000 7f619d400640 1 -- 172.21.15.28:0/1706347619 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6198150450 msgr2=0x7f61980a5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:33.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.418+0000 7f619d400640 1 --2- 172.21.15.28:0/1706347619 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6198150450 0x7f61980a5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:33.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.418+0000 7f619d400640 1 -- 172.21.15.28:0/1706347619 >> v2:172.21.15.81:3300/0 conn(0x7f6198156af0 msgr2=0x7f619806dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:33.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.418+0000 7f619d400640 1 --2- 172.21.15.28:0/1706347619 >> v2:172.21.15.81:3300/0 conn(0x7f6198156af0 0x7f619806dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:33.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.418+0000 7f619d400640 1 -- 172.21.15.28:0/1706347619 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6198153c30 con 0x7f619806ee40 2024-02-21T23:05:33.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.418+0000 7f619e800640 1 --2- 172.21.15.28:0/1706347619 >> v2:172.21.15.81:3300/0 conn(0x7f6198156af0 0x7f619806dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:05:33.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.419+0000 7f619d400640 1 --2- 172.21.15.28:0/1706347619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f619806ee40 0x7f61981536f0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f61980a87d0 tx=0x7f618802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=443d670025761298 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:33.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.420+0000 7f619ca00640 1 -- 172.21.15.28:0/1706347619 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6188035020 con 0x7f619806ee40 2024-02-21T23:05:33.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.420+0000 7f619ca00640 1 -- 172.21.15.28:0/1706347619 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6188031870 con 0x7f619806ee40 2024-02-21T23:05:33.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.420+0000 7f619ca00640 1 -- 172.21.15.28:0/1706347619 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6188031c40 con 0x7f619806ee40 2024-02-21T23:05:33.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.421+0000 7f619f200640 1 -- 172.21.15.28:0/1706347619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f619806ee40 msgr2=0x7f61981536f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:33.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.421+0000 7f619f200640 1 --2- 172.21.15.28:0/1706347619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f619806ee40 0x7f61981536f0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f61980a87d0 tx=0x7f618802b910 comp rx=0 tx=0).stop 2024-02-21T23:05:33.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.421+0000 7f619f200640 1 -- 172.21.15.28:0/1706347619 shutdown_connections 2024-02-21T23:05:33.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.421+0000 7f619f200640 1 --2- 172.21.15.28:0/1706347619 >> v2:172.21.15.81:3300/0 conn(0x7f6198156af0 0x7f619806dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:33.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.421+0000 7f619f200640 1 --2- 172.21.15.28:0/1706347619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f619806ee40 0x7f61981536f0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:33.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.421+0000 7f619f200640 1 --2- 172.21.15.28:0/1706347619 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6198150450 0x7f61980a5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:33.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.421+0000 7f619f200640 1 -- 172.21.15.28:0/1706347619 >> 172.21.15.28:0/1706347619 conn(0x7f619806b630 msgr2=0x7f61980c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:33.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.421+0000 7f619f200640 1 -- 172.21.15.28:0/1706347619 shutdown_connections 2024-02-21T23:05:33.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.421+0000 7f619f200640 1 -- 172.21.15.28:0/1706347619 wait complete. 2024-02-21T23:05:33.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.422+0000 7f619f200640 1 Processor -- start 2024-02-21T23:05:33.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.423+0000 7f619f200640 1 -- start start 2024-02-21T23:05:33.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.423+0000 7f619f200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f619806ee40 0x7f6198149af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:33.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.423+0000 7f619f200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6198150450 0x7f619814ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:33.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.423+0000 7f619f200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6198156af0 0x7f61980abe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:33.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.423+0000 7f619f200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61980c2cb0 con 0x7f6198150450 2024-02-21T23:05:33.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.423+0000 7f619f200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61980c29d0 con 0x7f6198156af0 2024-02-21T23:05:33.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.423+0000 7f619f200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6198156f80 con 0x7f619806ee40 2024-02-21T23:05:33.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.424+0000 7f619d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6198150450 0x7f619814ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:33.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.424+0000 7f619d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6198150450 0x7f619814ea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40470/0 (socket says 172.21.15.28:40470) 2024-02-21T23:05:33.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.424+0000 7f619d400640 1 -- 172.21.15.28:0/4123194719 learned_addr learned my addr 172.21.15.28:0/4123194719 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:33.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.424+0000 7f619d400640 1 -- 172.21.15.28:0/4123194719 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f619806ee40 msgr2=0x7f6198149af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:33.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.424+0000 7f619de00640 1 --2- 172.21.15.28:0/4123194719 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f619806ee40 0x7f6198149af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:33.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.424+0000 7f619d400640 1 --2- 172.21.15.28:0/4123194719 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f619806ee40 0x7f6198149af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:33.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.424+0000 7f619d400640 1 -- 172.21.15.28:0/4123194719 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6198156af0 msgr2=0x7f61980abe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:33.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.424+0000 7f619e800640 1 --2- 172.21.15.28:0/4123194719 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6198156af0 0x7f61980abe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:33.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.424+0000 7f619d400640 1 --2- 172.21.15.28:0/4123194719 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6198156af0 0x7f61980abe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:33.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.424+0000 7f619d400640 1 -- 172.21.15.28:0/4123194719 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61981eaba0 con 0x7f6198150450 2024-02-21T23:05:33.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.424+0000 7f619de00640 1 --2- 172.21.15.28:0/4123194719 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f619806ee40 0x7f6198149af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:05:33.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.424+0000 7f619d400640 1 --2- 172.21.15.28:0/4123194719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6198150450 0x7f619814ea10 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f61880097c0 tx=0x7f61880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:33.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.425+0000 7f6192000640 1 -- 172.21.15.28:0/4123194719 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6188040070 con 0x7f6198150450 2024-02-21T23:05:33.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.425+0000 7f6192000640 1 -- 172.21.15.28:0/4123194719 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6188030d00 con 0x7f6198150450 2024-02-21T23:05:33.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.425+0000 7f6192000640 1 -- 172.21.15.28:0/4123194719 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6188035040 con 0x7f6198150450 2024-02-21T23:05:33.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.425+0000 7f619f200640 1 -- 172.21.15.28:0/4123194719 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f619814a030 con 0x7f6198150450 2024-02-21T23:05:33.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.425+0000 7f619f200640 1 -- 172.21.15.28:0/4123194719 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61981eb140 con 0x7f6198150450 2024-02-21T23:05:33.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.426+0000 7f619f200640 1 -- 172.21.15.28:0/4123194719 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f619804f290 con 0x7f6198150450 2024-02-21T23:05:33.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.427+0000 7f6192000640 1 -- 172.21.15.28:0/4123194719 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f618804b060 con 0x7f6198150450 2024-02-21T23:05:33.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.428+0000 7f6192000640 1 --2- 172.21.15.28:0/4123194719 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f617c06b4d0 0x7f617c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:33.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.428+0000 7f6192000640 1 -- 172.21.15.28:0/4123194719 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6594+0+0 (secure 0 0 0) 0x7f61880ac300 con 0x7f6198150450 2024-02-21T23:05:33.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.429+0000 7f619de00640 1 --2- 172.21.15.28:0/4123194719 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f617c06b4d0 0x7f617c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:33.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.430+0000 7f619de00640 1 --2- 172.21.15.28:0/4123194719 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f617c06b4d0 0x7f617c06d990 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f618c006fd0 tx=0x7f618c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:33.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.432+0000 7f6192000640 1 -- 172.21.15.28:0/4123194719 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f618807adf0 con 0x7f6198150450 2024-02-21T23:05:33.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.635+0000 7f619f200640 1 -- 172.21.15.28:0/4123194719 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f61980a62f0 con 0x7f6198150450 2024-02-21T23:05:33.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.637+0000 7f6192000640 1 -- 172.21.15.28:0/4123194719 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v65) v1 ==== 74+0+17586 (secure 0 0 0) 0x7f618807a6f0 con 0x7f6198150450 2024-02-21T23:05:33.627 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:05:33.627 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":65,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:05:30.807870+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs2_metadata","create_time":"2024-02-21T23:05:05.701276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs2_data","create_time":"2024-02-21T23:05:06.904887+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"9.11","mappings":[{"from":7,"to":2}]},{"pgid":"10.19","mappings":[{"from":4,"to":5}]},{"pgid":"10.35","mappings":[{"from":4,"to":3}]},{"pgid":"10.36","mappings":[{"from":7,"to":3}]},{"pgid":"10.37","mappings":[{"from":7,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.81:6836/442767245":"2024-02-22T23:05:28.786771+0000","172.21.15.28:6835/3468780363":"2024-02-22T23:05:17.415180+0000","172.21.15.28:6833/3468780363":"2024-02-22T23:05:17.415180+0000","172.21.15.81:6837/442767245":"2024-02-22T23:05:28.786771+0000","172.21.15.28:6834/3835332438":"2024-02-22T23:05:27.771834+0000","172.21.15.28:6832/3835332438":"2024-02-22T23:05:27.771834+0000","172.21.15.28:6834/2803823391":"2024-02-22T23:05:14.385354+0000","172.21.15.28:6832/2803823391":"2024-02-22T23:05:14.385354+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:05:33.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.640+0000 7f619f200640 1 -- 172.21.15.28:0/4123194719 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f617c06b4d0 msgr2=0x7f617c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:33.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.640+0000 7f619f200640 1 --2- 172.21.15.28:0/4123194719 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f617c06b4d0 0x7f617c06d990 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f618c006fd0 tx=0x7f618c008040 comp rx=0 tx=0).stop 2024-02-21T23:05:33.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.640+0000 7f619f200640 1 -- 172.21.15.28:0/4123194719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6198150450 msgr2=0x7f619814ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:33.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.640+0000 7f619f200640 1 --2- 172.21.15.28:0/4123194719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6198150450 0x7f619814ea10 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f61880097c0 tx=0x7f61880046c0 comp rx=0 tx=0).stop 2024-02-21T23:05:33.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.640+0000 7f619f200640 1 -- 172.21.15.28:0/4123194719 shutdown_connections 2024-02-21T23:05:33.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.640+0000 7f619f200640 1 --2- 172.21.15.28:0/4123194719 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f617c06b4d0 0x7f617c06d990 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:33.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.640+0000 7f619f200640 1 --2- 172.21.15.28:0/4123194719 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6198156af0 0x7f61980abe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:33.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.640+0000 7f619f200640 1 --2- 172.21.15.28:0/4123194719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6198150450 0x7f619814ea10 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:33.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.640+0000 7f619f200640 1 --2- 172.21.15.28:0/4123194719 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f619806ee40 0x7f6198149af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:33.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.640+0000 7f619f200640 1 -- 172.21.15.28:0/4123194719 >> 172.21.15.28:0/4123194719 conn(0x7f619806b630 msgr2=0x7f61980c0500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:33.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.640+0000 7f619f200640 1 -- 172.21.15.28:0/4123194719 shutdown_connections 2024-02-21T23:05:33.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.641+0000 7f619f200640 1 -- 172.21.15.28:0/4123194719 wait complete. 2024-02-21T23:05:33.648 INFO:tasks.cephfs.filesystem:Destroying file system cephfs2 and related pools 2024-02-21T23:05:33.649 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:05:33.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.873+0000 7f400a400640 1 Processor -- start 2024-02-21T23:05:33.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.873+0000 7f400a400640 1 -- start start 2024-02-21T23:05:33.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.875+0000 7f400a400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f40040a5fd0 0x7f400406dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:33.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.875+0000 7f400a400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40040c2760 con 0x7f400406ee40 2024-02-21T23:05:33.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.875+0000 7f400a400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4004156310 con 0x7f4004150450 2024-02-21T23:05:33.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.875+0000 7f400a400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40040c22c0 con 0x7f40040a5fd0 2024-02-21T23:05:33.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.875+0000 7f4003e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f400406ee40 0x7f4004156af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:44120/0 (socket says 172.21.15.28:44120) 2024-02-21T23:05:33.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.875+0000 7f4003e00640 1 -- 172.21.15.28:0/3442619491 learned_addr learned my addr 172.21.15.28:0/3442619491 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:05:33.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.875+0000 7f4009a00640 1 --2- 172.21.15.28:0/3442619491 >> v2:172.21.15.28:3300/0 conn(0x7f40040a5fd0 0x7f400406dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:33.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.876+0000 7f4003400640 1 -- 172.21.15.28:0/3442619491 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4085291648 0 0) 0x7f40040c2760 con 0x7f400406ee40 2024-02-21T23:05:33.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.876+0000 7f4003400640 1 -- 172.21.15.28:0/3442619491 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fec003520 con 0x7f400406ee40 2024-02-21T23:05:33.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.876+0000 7f4009a00640 1 -- 172.21.15.28:0/3442619491 >> v1:172.21.15.81:6789/0 conn(0x7f4004150450 legacy=0x7f40041536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:33.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.876+0000 7f4009a00640 1 -- 172.21.15.28:0/3442619491 >> v1:172.21.15.28:6789/0 conn(0x7f400406ee40 legacy=0x7f4004156af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:33.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.876+0000 7f4009a00640 1 -- 172.21.15.28:0/3442619491 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40040c29d0 con 0x7f40040a5fd0 2024-02-21T23:05:33.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.877+0000 7f4009a00640 1 --2- 172.21.15.28:0/3442619491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40040a5fd0 0x7f400406dff0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc009d90 tx=0x7f3ffc02bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de3bed7767da3f7d server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:33.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.877+0000 7f4003400640 1 -- 172.21.15.28:0/3442619491 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ffc039070 con 0x7f40040a5fd0 2024-02-21T23:05:33.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.877+0000 7f4003400640 1 -- 172.21.15.28:0/3442619491 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ffc02ccf0 con 0x7f40040a5fd0 2024-02-21T23:05:33.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.877+0000 7f4003400640 1 -- 172.21.15.28:0/3442619491 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ffc034660 con 0x7f40040a5fd0 2024-02-21T23:05:33.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.878+0000 7f400a400640 1 -- 172.21.15.28:0/3442619491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40040a5fd0 msgr2=0x7f400406dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:33.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.878+0000 7f400a400640 1 --2- 172.21.15.28:0/3442619491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40040a5fd0 0x7f400406dff0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc009d90 tx=0x7f3ffc02bf40 comp rx=0 tx=0).stop 2024-02-21T23:05:33.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.878+0000 7f400a400640 1 -- 172.21.15.28:0/3442619491 shutdown_connections 2024-02-21T23:05:33.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.878+0000 7f400a400640 1 --2- 172.21.15.28:0/3442619491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40040a5fd0 0x7f400406dff0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:33.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.878+0000 7f400a400640 1 -- 172.21.15.28:0/3442619491 >> 172.21.15.28:0/3442619491 conn(0x7f400406b630 msgr2=0x7f40040c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:33.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.878+0000 7f400a400640 1 -- 172.21.15.28:0/3442619491 shutdown_connections 2024-02-21T23:05:33.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.878+0000 7f400a400640 1 -- 172.21.15.28:0/3442619491 wait complete. 2024-02-21T23:05:33.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.880+0000 7f400a400640 1 Processor -- start 2024-02-21T23:05:33.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.880+0000 7f400a400640 1 -- start start 2024-02-21T23:05:33.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.881+0000 7f400a400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f400406ee40 0x7f40041533a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:33.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.881+0000 7f400a400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40040a5fd0 0x7f4004145370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:33.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.881+0000 7f400a400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4004150450 0x7f40041458b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:33.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.881+0000 7f400a400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4004157200 con 0x7f40040a5fd0 2024-02-21T23:05:33.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.881+0000 7f400a400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4004156660 con 0x7f400406ee40 2024-02-21T23:05:33.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.881+0000 7f400a400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f400406e530 con 0x7f4004150450 2024-02-21T23:05:33.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.881+0000 7f4003e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40040a5fd0 0x7f4004145370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:33.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.881+0000 7f4003e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40040a5fd0 0x7f4004145370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40492/0 (socket says 172.21.15.28:40492) 2024-02-21T23:05:33.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.881+0000 7f4003e00640 1 -- 172.21.15.28:0/44958294 learned_addr learned my addr 172.21.15.28:0/44958294 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:33.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.881+0000 7f4009000640 1 --2- 172.21.15.28:0/44958294 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f400406ee40 0x7f40041533a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:33.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.881+0000 7f4009a00640 1 --2- 172.21.15.28:0/44958294 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4004150450 0x7f40041458b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:33.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.881+0000 7f4003e00640 1 -- 172.21.15.28:0/44958294 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4004150450 msgr2=0x7f40041458b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:33.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.881+0000 7f4003e00640 1 --2- 172.21.15.28:0/44958294 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4004150450 0x7f40041458b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:33.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.881+0000 7f4003e00640 1 -- 172.21.15.28:0/44958294 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f400406ee40 msgr2=0x7f40041533a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:33.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.881+0000 7f4003e00640 1 --2- 172.21.15.28:0/44958294 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f400406ee40 0x7f40041533a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:33.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.881+0000 7f4003e00640 1 -- 172.21.15.28:0/44958294 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40040c5cb0 con 0x7f40040a5fd0 2024-02-21T23:05:33.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.882+0000 7f4003e00640 1 --2- 172.21.15.28:0/44958294 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40040a5fd0 0x7f4004145370 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f3ff000c530 tx=0x7f3ff000c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:33.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.882+0000 7f4000c00640 1 -- 172.21.15.28:0/44958294 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ff0017070 con 0x7f40040a5fd0 2024-02-21T23:05:33.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.882+0000 7f4000c00640 1 -- 172.21.15.28:0/44958294 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ff0005ce0 con 0x7f40040a5fd0 2024-02-21T23:05:33.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.882+0000 7f400a400640 1 -- 172.21.15.28:0/44958294 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40040c56f0 con 0x7f40040a5fd0 2024-02-21T23:05:33.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.882+0000 7f4000c00640 1 -- 172.21.15.28:0/44958294 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ff00072b0 con 0x7f40040a5fd0 2024-02-21T23:05:33.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.882+0000 7f400a400640 1 -- 172.21.15.28:0/44958294 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40040c47f0 con 0x7f40040a5fd0 2024-02-21T23:05:33.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.883+0000 7f400a400640 1 -- 172.21.15.28:0/44958294 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4004153e20 con 0x7f40040a5fd0 2024-02-21T23:05:33.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.884+0000 7f4000c00640 1 -- 172.21.15.28:0/44958294 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3ff001a070 con 0x7f40040a5fd0 2024-02-21T23:05:33.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.885+0000 7f4000c00640 1 --2- 172.21.15.28:0/44958294 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3fe406b4d0 0x7f3fe406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:33.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.885+0000 7f4000c00640 1 -- 172.21.15.28:0/44958294 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6594+0+0 (secure 0 0 0) 0x7f3ff0089a60 con 0x7f40040a5fd0 2024-02-21T23:05:33.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.886+0000 7f4009000640 1 --2- 172.21.15.28:0/44958294 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3fe406b4d0 0x7f3fe406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:33.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.887+0000 7f4009000640 1 --2- 172.21.15.28:0/44958294 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3fe406b4d0 0x7f3fe406d990 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f3ff8000f60 tx=0x7f3ff8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:33.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:33.890+0000 7f4000c00640 1 -- 172.21.15.28:0/44958294 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3ff0058580 con 0x7f40040a5fd0 2024-02-21T23:05:34.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.131+0000 7f400a400640 1 -- 172.21.15.28:0/44958294 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f400404f290 con 0x7f40040a5fd0 2024-02-21T23:05:34.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.132+0000 7f4000c00640 1 -- 172.21.15.28:0/44958294 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+4940 (secure 0 0 0) 0x7f3ff0057e80 con 0x7f40040a5fd0 2024-02-21T23:05:34.121 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:05:34.121 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":53,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5428,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/2146447666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":2146447666},{"type":"v1","addr":"172.21.15.81:6837","nonce":2146447666}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":6039,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3807390545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3807390545},{"type":"v1","addr":"172.21.15.28:6835","nonce":3807390545}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45},{"gid":6147,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/2305639796","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2305639796},{"type":"v1","addr":"172.21.15.28:6834","nonce":2305639796}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:05:08.017586+0000","modified":"2024-02-21T23:05:30.816087+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5221},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5221":{"gid":5221,"name":"b","rank":0,"incarnation":50,"state":"up:active","state_seq":14,"addr":"172.21.15.81:6835/2152160500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2152160500},{"type":"v1","addr":"172.21.15.81:6835","nonce":2152160500}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-02-21T23:05:34.122 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 53 2024-02-21T23:05:34.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.134+0000 7f400a400640 1 -- 172.21.15.28:0/44958294 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3fe406b4d0 msgr2=0x7f3fe406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:34.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.134+0000 7f400a400640 1 --2- 172.21.15.28:0/44958294 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3fe406b4d0 0x7f3fe406d990 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f3ff8000f60 tx=0x7f3ff8009040 comp rx=0 tx=0).stop 2024-02-21T23:05:34.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.134+0000 7f400a400640 1 -- 172.21.15.28:0/44958294 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40040a5fd0 msgr2=0x7f4004145370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:34.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.134+0000 7f400a400640 1 --2- 172.21.15.28:0/44958294 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40040a5fd0 0x7f4004145370 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f3ff000c530 tx=0x7f3ff000c9f0 comp rx=0 tx=0).stop 2024-02-21T23:05:34.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.134+0000 7f400a400640 1 -- 172.21.15.28:0/44958294 shutdown_connections 2024-02-21T23:05:34.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.134+0000 7f400a400640 1 --2- 172.21.15.28:0/44958294 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3fe406b4d0 0x7f3fe406d990 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:34.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.134+0000 7f400a400640 1 --2- 172.21.15.28:0/44958294 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4004150450 0x7f40041458b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:34.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.134+0000 7f400a400640 1 --2- 172.21.15.28:0/44958294 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40040a5fd0 0x7f4004145370 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:34.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.134+0000 7f400a400640 1 --2- 172.21.15.28:0/44958294 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f400406ee40 0x7f40041533a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:34.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.134+0000 7f400a400640 1 -- 172.21.15.28:0/44958294 >> 172.21.15.28:0/44958294 conn(0x7f400406b630 msgr2=0x7f400414f650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:34.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.134+0000 7f400a400640 1 -- 172.21.15.28:0/44958294 shutdown_connections 2024-02-21T23:05:34.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.135+0000 7f400a400640 1 -- 172.21.15.28:0/44958294 wait complete. 2024-02-21T23:05:34.143 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:05:34.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.375+0000 7f8559200640 1 Processor -- start 2024-02-21T23:05:34.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.376+0000 7f8559200640 1 -- start start 2024-02-21T23:05:34.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.377+0000 7f8559200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f85540aff90 0x7f85540c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:34.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.377+0000 7f8559200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85540c74c0 con 0x7f855406dbf0 2024-02-21T23:05:34.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.377+0000 7f8559200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85540c77a0 con 0x7f85540c5720 2024-02-21T23:05:34.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.377+0000 7f8559200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85540c7270 con 0x7f85540aff90 2024-02-21T23:05:34.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.378+0000 7f854aa00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f855406dbf0 0x7f8554158ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:44130/0 (socket says 172.21.15.28:44130) 2024-02-21T23:05:34.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.378+0000 7f854aa00640 1 -- 172.21.15.28:0/3804627744 learned_addr learned my addr 172.21.15.28:0/3804627744 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:05:34.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.378+0000 7f8553400640 1 --2- 172.21.15.28:0/3804627744 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f85540aff90 0x7f85540c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:34.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.378+0000 7f8552a00640 1 -- 172.21.15.28:0/3804627744 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 928807039 0 0) 0x7f85540c74c0 con 0x7f855406dbf0 2024-02-21T23:05:34.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.378+0000 7f8552a00640 1 -- 172.21.15.28:0/3804627744 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8538003520 con 0x7f855406dbf0 2024-02-21T23:05:34.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.378+0000 7f8552a00640 1 -- 172.21.15.28:0/3804627744 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2061123516 0 0) 0x7f8538003520 con 0x7f855406dbf0 2024-02-21T23:05:34.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.378+0000 7f8552a00640 1 -- 172.21.15.28:0/3804627744 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f85540aff90 msgr2=0x7f85540c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:34.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.378+0000 7f8552a00640 1 --2- 172.21.15.28:0/3804627744 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f85540aff90 0x7f85540c51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:34.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.378+0000 7f8552a00640 1 -- 172.21.15.28:0/3804627744 >> v1:172.21.15.81:6789/0 conn(0x7f85540c5720 legacy=0x7f855415f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:34.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.378+0000 7f8552a00640 1 -- 172.21.15.28:0/3804627744 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85540c7a80 con 0x7f855406dbf0 2024-02-21T23:05:34.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.379+0000 7f8552a00640 1 -- 172.21.15.28:0/3804627744 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f853c001670 con 0x7f855406dbf0 2024-02-21T23:05:34.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.379+0000 7f8552a00640 1 -- 172.21.15.28:0/3804627744 >> v1:172.21.15.28:6789/0 conn(0x7f855406dbf0 legacy=0x7f8554158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:34.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.379+0000 7f8552a00640 1 --2- 172.21.15.28:0/3804627744 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f85380039e0 0x7f8538003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:34.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.379+0000 7f8552a00640 1 --2- 172.21.15.28:0/3804627744 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f85540aff90 0x7f8538004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:34.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.379+0000 7f8552a00640 1 --2- 172.21.15.28:0/3804627744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8538004d40 0x7f85380051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:34.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.379+0000 7f8552a00640 1 -- 172.21.15.28:0/3804627744 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85540c7270 con 0x7f8538004d40 2024-02-21T23:05:34.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.379+0000 7f8552a00640 1 -- 172.21.15.28:0/3804627744 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f853800ba70 con 0x7f85380039e0 2024-02-21T23:05:34.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.379+0000 7f8552a00640 1 -- 172.21.15.28:0/3804627744 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f853800bc70 con 0x7f85540aff90 2024-02-21T23:05:34.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.379+0000 7f8553e00640 1 --2- 172.21.15.28:0/3804627744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8538004d40 0x7f85380051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:34.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.379+0000 7f8553400640 1 --2- 172.21.15.28:0/3804627744 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f85380039e0 0x7f8538003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:34.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.379+0000 7f854aa00640 1 --2- 172.21.15.28:0/3804627744 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f85540aff90 0x7f8538004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:34.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.379+0000 7f8553e00640 1 -- 172.21.15.28:0/3804627744 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f85540aff90 msgr2=0x7f8538004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:34.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.379+0000 7f8553e00640 1 --2- 172.21.15.28:0/3804627744 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f85540aff90 0x7f8538004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:34.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.379+0000 7f8553e00640 1 -- 172.21.15.28:0/3804627744 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f85380039e0 msgr2=0x7f8538003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:34.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.379+0000 7f8553e00640 1 --2- 172.21.15.28:0/3804627744 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f85380039e0 0x7f8538003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:34.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.379+0000 7f8553e00640 1 -- 172.21.15.28:0/3804627744 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8538003520 con 0x7f8538004d40 2024-02-21T23:05:34.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.379+0000 7f8553400640 1 --2- 172.21.15.28:0/3804627744 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f85380039e0 0x7f8538003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:05:34.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.380+0000 7f8553e00640 1 --2- 172.21.15.28:0/3804627744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8538004d40 0x7f85380051c0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f854c001e20 tx=0x7f8538005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a40a95ead5efc3b server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:34.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.381+0000 7f8552a00640 1 -- 172.21.15.28:0/3804627744 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f854c001a70 con 0x7f8538004d40 2024-02-21T23:05:34.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.381+0000 7f8552a00640 1 -- 172.21.15.28:0/3804627744 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f854c003b70 con 0x7f8538004d40 2024-02-21T23:05:34.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.381+0000 7f8552a00640 1 -- 172.21.15.28:0/3804627744 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f854c0050d0 con 0x7f8538004d40 2024-02-21T23:05:34.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.381+0000 7f8559200640 1 -- 172.21.15.28:0/3804627744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8538004d40 msgr2=0x7f85380051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:34.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.381+0000 7f8559200640 1 --2- 172.21.15.28:0/3804627744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8538004d40 0x7f85380051c0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f854c001e20 tx=0x7f8538005830 comp rx=0 tx=0).stop 2024-02-21T23:05:34.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.381+0000 7f8553e00640 1 -- 172.21.15.28:0/3804627744 reap_dead start 2024-02-21T23:05:34.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.381+0000 7f8559200640 1 -- 172.21.15.28:0/3804627744 shutdown_connections 2024-02-21T23:05:34.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.381+0000 7f8559200640 1 -- 172.21.15.28:0/3804627744 >> 172.21.15.28:0/3804627744 conn(0x7f85540ac370 msgr2=0x7f85540ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:34.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.381+0000 7f8559200640 1 -- 172.21.15.28:0/3804627744 shutdown_connections 2024-02-21T23:05:34.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.382+0000 7f8559200640 1 -- 172.21.15.28:0/3804627744 wait complete. 2024-02-21T23:05:34.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.383+0000 7f8559200640 1 Processor -- start 2024-02-21T23:05:34.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.383+0000 7f8559200640 1 -- start start 2024-02-21T23:05:34.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.384+0000 7f8559200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8538004d40 0x7f85540ae740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:34.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.384+0000 7f8559200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f85541de570 0x7f855414fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:34.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.384+0000 7f8559200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f855414ff70 0x7f85541eaef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:34.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.384+0000 7f8559200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85540c8760 con 0x7f855414ff70 2024-02-21T23:05:34.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.384+0000 7f8559200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8554158630 con 0x7f85541de570 2024-02-21T23:05:34.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.384+0000 7f8559200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85540c88e0 con 0x7f8538004d40 2024-02-21T23:05:34.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.385+0000 7f8553e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f855414ff70 0x7f85541eaef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:34.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.385+0000 7f8553e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f855414ff70 0x7f85541eaef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40512/0 (socket says 172.21.15.28:40512) 2024-02-21T23:05:34.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.385+0000 7f8553e00640 1 -- 172.21.15.28:0/103176158 learned_addr learned my addr 172.21.15.28:0/103176158 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:34.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.385+0000 7f8553400640 1 --2- 172.21.15.28:0/103176158 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8538004d40 0x7f85540ae740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:34.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.385+0000 7f854aa00640 1 --2- 172.21.15.28:0/103176158 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f85541de570 0x7f855414fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:34.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.385+0000 7f8553e00640 1 -- 172.21.15.28:0/103176158 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8538004d40 msgr2=0x7f85540ae740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:34.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.385+0000 7f8553e00640 1 --2- 172.21.15.28:0/103176158 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8538004d40 0x7f85540ae740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:34.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.385+0000 7f8553e00640 1 -- 172.21.15.28:0/103176158 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f85541de570 msgr2=0x7f855414fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:34.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.385+0000 7f8553e00640 1 --2- 172.21.15.28:0/103176158 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f85541de570 0x7f855414fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:34.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.385+0000 7f8553e00640 1 -- 172.21.15.28:0/103176158 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85541549a0 con 0x7f855414ff70 2024-02-21T23:05:34.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.385+0000 7f8553e00640 1 --2- 172.21.15.28:0/103176158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f855414ff70 0x7f85541eaef0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f854c001f10 tx=0x7f854c0014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:34.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.386+0000 7f854be00640 1 -- 172.21.15.28:0/103176158 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f854c001a70 con 0x7f855414ff70 2024-02-21T23:05:34.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.386+0000 7f854be00640 1 -- 172.21.15.28:0/103176158 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f854c003970 con 0x7f855414ff70 2024-02-21T23:05:34.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.386+0000 7f854be00640 1 -- 172.21.15.28:0/103176158 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f854c007db0 con 0x7f855414ff70 2024-02-21T23:05:34.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.386+0000 7f8559200640 1 -- 172.21.15.28:0/103176158 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85541eee30 con 0x7f855414ff70 2024-02-21T23:05:34.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.386+0000 7f8559200640 1 -- 172.21.15.28:0/103176158 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8554154f20 con 0x7f855414ff70 2024-02-21T23:05:34.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.387+0000 7f8559200640 1 -- 172.21.15.28:0/103176158 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f855404f290 con 0x7f855414ff70 2024-02-21T23:05:34.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.388+0000 7f854be00640 1 -- 172.21.15.28:0/103176158 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f854c007030 con 0x7f855414ff70 2024-02-21T23:05:34.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.389+0000 7f854be00640 1 --2- 172.21.15.28:0/103176158 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f853006b4d0 0x7f853006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:34.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.389+0000 7f854be00640 1 -- 172.21.15.28:0/103176158 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6594+0+0 (secure 0 0 0) 0x7f854c0ab6b0 con 0x7f855414ff70 2024-02-21T23:05:34.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.390+0000 7f8553400640 1 --2- 172.21.15.28:0/103176158 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f853006b4d0 0x7f853006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:34.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.391+0000 7f8553400640 1 --2- 172.21.15.28:0/103176158 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f853006b4d0 0x7f853006d990 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f8544009b60 tx=0x7f8544008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:34.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.393+0000 7f854be00640 1 -- 172.21.15.28:0/103176158 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f854c033020 con 0x7f855414ff70 2024-02-21T23:05:34.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.632+0000 7f8559200640 1 -- 172.21.15.28:0/103176158 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f855406d9d0 con 0x7f855414ff70 2024-02-21T23:05:34.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.633+0000 7f854be00640 1 -- 172.21.15.28:0/103176158 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+4940 (secure 0 0 0) 0x7f854c07a240 con 0x7f855414ff70 2024-02-21T23:05:34.622 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:05:34.623 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":53,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5428,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/2146447666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":2146447666},{"type":"v1","addr":"172.21.15.81:6837","nonce":2146447666}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":6039,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3807390545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3807390545},{"type":"v1","addr":"172.21.15.28:6835","nonce":3807390545}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45},{"gid":6147,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/2305639796","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2305639796},{"type":"v1","addr":"172.21.15.28:6834","nonce":2305639796}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:05:08.017586+0000","modified":"2024-02-21T23:05:30.816087+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5221},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5221":{"gid":5221,"name":"b","rank":0,"incarnation":50,"state":"up:active","state_seq":14,"addr":"172.21.15.81:6835/2152160500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2152160500},{"type":"v1","addr":"172.21.15.81:6835","nonce":2152160500}]},"join_fscid":4,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-02-21T23:05:34.623 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 53 2024-02-21T23:05:34.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.635+0000 7f8559200640 1 -- 172.21.15.28:0/103176158 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f853006b4d0 msgr2=0x7f853006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:34.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.635+0000 7f8559200640 1 --2- 172.21.15.28:0/103176158 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f853006b4d0 0x7f853006d990 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f8544009b60 tx=0x7f8544008040 comp rx=0 tx=0).stop 2024-02-21T23:05:34.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.636+0000 7f8559200640 1 -- 172.21.15.28:0/103176158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f855414ff70 msgr2=0x7f85541eaef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:34.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.636+0000 7f8559200640 1 --2- 172.21.15.28:0/103176158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f855414ff70 0x7f85541eaef0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f854c001f10 tx=0x7f854c0014a0 comp rx=0 tx=0).stop 2024-02-21T23:05:34.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.636+0000 7f8559200640 1 -- 172.21.15.28:0/103176158 shutdown_connections 2024-02-21T23:05:34.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.636+0000 7f8559200640 1 --2- 172.21.15.28:0/103176158 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f853006b4d0 0x7f853006d990 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:34.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.636+0000 7f8559200640 1 --2- 172.21.15.28:0/103176158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f855414ff70 0x7f85541eaef0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:34.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.636+0000 7f8559200640 1 --2- 172.21.15.28:0/103176158 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f85541de570 0x7f855414fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:34.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.636+0000 7f8559200640 1 --2- 172.21.15.28:0/103176158 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8538004d40 0x7f85540ae740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:34.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.636+0000 7f8559200640 1 -- 172.21.15.28:0/103176158 >> 172.21.15.28:0/103176158 conn(0x7f85540ac370 msgr2=0x7f8554156300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:34.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.636+0000 7f8559200640 1 -- 172.21.15.28:0/103176158 shutdown_connections 2024-02-21T23:05:34.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.636+0000 7f8559200640 1 -- 172.21.15.28:0/103176158 wait complete. 2024-02-21T23:05:34.646 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:05:34.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.878+0000 7f01e6200640 1 Processor -- start 2024-02-21T23:05:34.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.878+0000 7f01e6200640 1 -- start start 2024-02-21T23:05:34.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.879+0000 7f01e6200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f01e0150450 0x7f01e00a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:34.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.880+0000 7f01e6200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f01e006ee40 0x7f01e01536f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:34.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.880+0000 7f01e6200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01e00c2760 con 0x7f01e0156af0 2024-02-21T23:05:34.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.880+0000 7f01e6200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01e00c22c0 con 0x7f01e0150450 2024-02-21T23:05:34.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.880+0000 7f01e6200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01e0156660 con 0x7f01e006ee40 2024-02-21T23:05:34.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.880+0000 7f01e5800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f01e0156af0 0x7f01e006dff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:44140/0 (socket says 172.21.15.28:44140) 2024-02-21T23:05:34.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.880+0000 7f01e5800640 1 -- 172.21.15.28:0/3180756218 learned_addr learned my addr 172.21.15.28:0/3180756218 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:05:34.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.880+0000 7f01e4e00640 1 --2- 172.21.15.28:0/3180756218 >> v2:172.21.15.28:3300/0 conn(0x7f01e0150450 0x7f01e00a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:34.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.880+0000 7f01d7e00640 1 --2- 172.21.15.28:0/3180756218 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f01e006ee40 0x7f01e01536f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:34.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.880+0000 7f01dfe00640 1 -- 172.21.15.28:0/3180756218 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4154938540 0 0) 0x7f01e00c2760 con 0x7f01e0156af0 2024-02-21T23:05:34.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.881+0000 7f01dfe00640 1 -- 172.21.15.28:0/3180756218 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01c4003520 con 0x7f01e0156af0 2024-02-21T23:05:34.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.881+0000 7f01e4e00640 1 -- 172.21.15.28:0/3180756218 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f01e006ee40 msgr2=0x7f01e01536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:34.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.881+0000 7f01e4e00640 1 --2- 172.21.15.28:0/3180756218 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f01e006ee40 0x7f01e01536f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:34.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.881+0000 7f01e4e00640 1 -- 172.21.15.28:0/3180756218 >> v1:172.21.15.28:6789/0 conn(0x7f01e0156af0 legacy=0x7f01e006dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:34.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.881+0000 7f01e4e00640 1 -- 172.21.15.28:0/3180756218 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01e0153c30 con 0x7f01e0150450 2024-02-21T23:05:34.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.882+0000 7f01e4e00640 1 --2- 172.21.15.28:0/3180756218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01e0150450 0x7f01e00a5fd0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f01d00097a0 tx=0x7f01d002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3023fb30541966 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:34.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.882+0000 7f01dfe00640 1 -- 172.21.15.28:0/3180756218 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01d0035020 con 0x7f01e0150450 2024-02-21T23:05:34.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.882+0000 7f01dfe00640 1 -- 172.21.15.28:0/3180756218 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01d0031830 con 0x7f01e0150450 2024-02-21T23:05:34.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.882+0000 7f01dfe00640 1 -- 172.21.15.28:0/3180756218 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01d0031c00 con 0x7f01e0150450 2024-02-21T23:05:34.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.883+0000 7f01e6200640 1 -- 172.21.15.28:0/3180756218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01e0150450 msgr2=0x7f01e00a5fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:34.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.883+0000 7f01e6200640 1 --2- 172.21.15.28:0/3180756218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01e0150450 0x7f01e00a5fd0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f01d00097a0 tx=0x7f01d002b8d0 comp rx=0 tx=0).stop 2024-02-21T23:05:34.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.883+0000 7f01e6200640 1 -- 172.21.15.28:0/3180756218 shutdown_connections 2024-02-21T23:05:34.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.883+0000 7f01e6200640 1 --2- 172.21.15.28:0/3180756218 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f01e006ee40 0x7f01e01536f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:34.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.883+0000 7f01e6200640 1 --2- 172.21.15.28:0/3180756218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01e0150450 0x7f01e00a5fd0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:34.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.883+0000 7f01e6200640 1 -- 172.21.15.28:0/3180756218 >> 172.21.15.28:0/3180756218 conn(0x7f01e006b630 msgr2=0x7f01e00c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:34.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.883+0000 7f01e6200640 1 -- 172.21.15.28:0/3180756218 shutdown_connections 2024-02-21T23:05:34.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.883+0000 7f01e6200640 1 -- 172.21.15.28:0/3180756218 wait complete. 2024-02-21T23:05:34.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.885+0000 7f01e6200640 1 Processor -- start 2024-02-21T23:05:34.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.885+0000 7f01e6200640 1 -- start start 2024-02-21T23:05:34.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.886+0000 7f01e6200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f01e006ee40 0x7f01e00bfba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:34.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.886+0000 7f01e6200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01e0150450 0x7f01e014c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:34.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.886+0000 7f01e6200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01e0156af0 0x7f01e014ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:34.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.886+0000 7f01e6200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01e00c2cb0 con 0x7f01e0156af0 2024-02-21T23:05:34.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.886+0000 7f01e6200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01e00c29d0 con 0x7f01e0150450 2024-02-21T23:05:34.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.886+0000 7f01e6200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01e006e700 con 0x7f01e006ee40 2024-02-21T23:05:34.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.886+0000 7f01e5800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01e0156af0 0x7f01e014ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:34.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.886+0000 7f01e5800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01e0156af0 0x7f01e014ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40536/0 (socket says 172.21.15.28:40536) 2024-02-21T23:05:34.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.886+0000 7f01e5800640 1 -- 172.21.15.28:0/1415040069 learned_addr learned my addr 172.21.15.28:0/1415040069 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:34.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.886+0000 7f01d7e00640 1 --2- 172.21.15.28:0/1415040069 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01e0150450 0x7f01e014c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:34.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.886+0000 7f01e4e00640 1 --2- 172.21.15.28:0/1415040069 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f01e006ee40 0x7f01e00bfba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:34.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.887+0000 7f01e5800640 1 -- 172.21.15.28:0/1415040069 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f01e006ee40 msgr2=0x7f01e00bfba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:34.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.887+0000 7f01e5800640 1 --2- 172.21.15.28:0/1415040069 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f01e006ee40 0x7f01e00bfba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:34.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.887+0000 7f01e5800640 1 -- 172.21.15.28:0/1415040069 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01e0150450 msgr2=0x7f01e014c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:34.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.887+0000 7f01e5800640 1 --2- 172.21.15.28:0/1415040069 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01e0150450 0x7f01e014c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:34.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.887+0000 7f01e5800640 1 -- 172.21.15.28:0/1415040069 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01e0152db0 con 0x7f01e0156af0 2024-02-21T23:05:34.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.887+0000 7f01e4e00640 1 --2- 172.21.15.28:0/1415040069 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f01e006ee40 0x7f01e00bfba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:05:34.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.887+0000 7f01d7e00640 1 --2- 172.21.15.28:0/1415040069 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01e0150450 0x7f01e014c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:05:34.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.887+0000 7f01e5800640 1 --2- 172.21.15.28:0/1415040069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01e0156af0 0x7f01e014ce80 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f01d800caa0 tx=0x7f01d800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:34.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.887+0000 7f01dd600640 1 -- 172.21.15.28:0/1415040069 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01d8017070 con 0x7f01e0156af0 2024-02-21T23:05:34.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.887+0000 7f01dd600640 1 -- 172.21.15.28:0/1415040069 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01d800ac00 con 0x7f01e0156af0 2024-02-21T23:05:34.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.887+0000 7f01dd600640 1 -- 172.21.15.28:0/1415040069 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01d8012400 con 0x7f01e0156af0 2024-02-21T23:05:34.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.887+0000 7f01e6200640 1 -- 172.21.15.28:0/1415040069 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01e0153090 con 0x7f01e0156af0 2024-02-21T23:05:34.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.887+0000 7f01e6200640 1 -- 172.21.15.28:0/1415040069 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01e00c00e0 con 0x7f01e0156af0 2024-02-21T23:05:34.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.888+0000 7f01e6200640 1 -- 172.21.15.28:0/1415040069 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01e004f290 con 0x7f01e0156af0 2024-02-21T23:05:34.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.890+0000 7f01dd600640 1 -- 172.21.15.28:0/1415040069 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f01d801a070 con 0x7f01e0156af0 2024-02-21T23:05:34.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.891+0000 7f01dd600640 1 --2- 172.21.15.28:0/1415040069 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f01bc06b4d0 0x7f01bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:34.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.891+0000 7f01dd600640 1 -- 172.21.15.28:0/1415040069 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6594+0+0 (secure 0 0 0) 0x7f01d808a390 con 0x7f01e0156af0 2024-02-21T23:05:34.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.891+0000 7f01e4e00640 1 --2- 172.21.15.28:0/1415040069 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f01bc06b4d0 0x7f01bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:34.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.893+0000 7f01e4e00640 1 --2- 172.21.15.28:0/1415040069 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f01bc06b4d0 0x7f01bc06d990 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f01d0036000 tx=0x7f01d0008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:34.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:34.895+0000 7f01dd600640 1 -- 172.21.15.28:0/1415040069 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f01d8058e80 con 0x7f01e0156af0 2024-02-21T23:05:35.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.097+0000 7f01e6200640 1 -- 172.21.15.28:0/1415040069 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f01e00c1f10 con 0x7f01e0156af0 2024-02-21T23:05:35.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.099+0000 7f01dd600640 1 -- 172.21.15.28:0/1415040069 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v65) v1 ==== 74+0+17586 (secure 0 0 0) 0x7f01d8058780 con 0x7f01e0156af0 2024-02-21T23:05:35.089 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:05:35.089 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":65,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:05:30.807870+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs2_metadata","create_time":"2024-02-21T23:05:05.701276+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs2_data","create_time":"2024-02-21T23:05:06.904887+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"9.11","mappings":[{"from":7,"to":2}]},{"pgid":"10.19","mappings":[{"from":4,"to":5}]},{"pgid":"10.35","mappings":[{"from":4,"to":3}]},{"pgid":"10.36","mappings":[{"from":7,"to":3}]},{"pgid":"10.37","mappings":[{"from":7,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.81:6836/442767245":"2024-02-22T23:05:28.786771+0000","172.21.15.28:6835/3468780363":"2024-02-22T23:05:17.415180+0000","172.21.15.28:6833/3468780363":"2024-02-22T23:05:17.415180+0000","172.21.15.81:6837/442767245":"2024-02-22T23:05:28.786771+0000","172.21.15.28:6834/3835332438":"2024-02-22T23:05:27.771834+0000","172.21.15.28:6832/3835332438":"2024-02-22T23:05:27.771834+0000","172.21.15.28:6834/2803823391":"2024-02-22T23:05:14.385354+0000","172.21.15.28:6832/2803823391":"2024-02-22T23:05:14.385354+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:05:35.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.102+0000 7f01e6200640 1 -- 172.21.15.28:0/1415040069 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f01bc06b4d0 msgr2=0x7f01bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:35.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.102+0000 7f01e6200640 1 --2- 172.21.15.28:0/1415040069 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f01bc06b4d0 0x7f01bc06d990 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f01d0036000 tx=0x7f01d0008030 comp rx=0 tx=0).stop 2024-02-21T23:05:35.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.102+0000 7f01e6200640 1 -- 172.21.15.28:0/1415040069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01e0156af0 msgr2=0x7f01e014ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:35.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.102+0000 7f01e6200640 1 --2- 172.21.15.28:0/1415040069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01e0156af0 0x7f01e014ce80 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f01d800caa0 tx=0x7f01d800cf60 comp rx=0 tx=0).stop 2024-02-21T23:05:35.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.102+0000 7f01e6200640 1 -- 172.21.15.28:0/1415040069 shutdown_connections 2024-02-21T23:05:35.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.102+0000 7f01e6200640 1 --2- 172.21.15.28:0/1415040069 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f01bc06b4d0 0x7f01bc06d990 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:35.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.102+0000 7f01e6200640 1 --2- 172.21.15.28:0/1415040069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01e0156af0 0x7f01e014ce80 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:35.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.102+0000 7f01e6200640 1 --2- 172.21.15.28:0/1415040069 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01e0150450 0x7f01e014c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:35.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.102+0000 7f01e6200640 1 --2- 172.21.15.28:0/1415040069 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f01e006ee40 0x7f01e00bfba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:35.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.102+0000 7f01e6200640 1 -- 172.21.15.28:0/1415040069 >> 172.21.15.28:0/1415040069 conn(0x7f01e006b630 msgr2=0x7f01e014f150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:35.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.102+0000 7f01e6200640 1 -- 172.21.15.28:0/1415040069 shutdown_connections 2024-02-21T23:05:35.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.102+0000 7f01e6200640 1 -- 172.21.15.28:0/1415040069 wait complete. 2024-02-21T23:05:35.112 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs2 2024-02-21T23:05:35.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.344+0000 7f6bb1600640 1 Processor -- start 2024-02-21T23:05:35.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.344+0000 7f6bb1600640 1 -- start start 2024-02-21T23:05:35.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.346+0000 7f6bb1600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6bac158f70 0x7f6bac0c00b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:35.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.346+0000 7f6bb1600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6bac06d410 0x7f6bac06d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:35.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.346+0000 7f6bb1600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bac0c2070 con 0x7f6bac1527a0 2024-02-21T23:05:35.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.346+0000 7f6bb1600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bac06e790 con 0x7f6bac06d410 2024-02-21T23:05:35.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.346+0000 7f6bb1600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bac0c29a0 con 0x7f6bac158f70 2024-02-21T23:05:35.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.346+0000 7f6bb0c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6bac06d410 0x7f6bac06d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:35.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.346+0000 7f6bb0c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6bac06d410 0x7f6bac06d860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50504/0 (socket says 172.21.15.28:50504) 2024-02-21T23:05:35.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.346+0000 7f6bb0c00640 1 -- 172.21.15.28:0/3488690961 learned_addr learned my addr 172.21.15.28:0/3488690961 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:35.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.346+0000 7f6bab400640 1 --2- 172.21.15.28:0/3488690961 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6bac158f70 0x7f6bac0c00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:35.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.347+0000 7f6bb0c00640 1 -- 172.21.15.28:0/3488690961 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6bac158f70 msgr2=0x7f6bac0c00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:35.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.347+0000 7f6bb0c00640 1 --2- 172.21.15.28:0/3488690961 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6bac158f70 0x7f6bac0c00b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:35.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.347+0000 7f6bb0c00640 1 -- 172.21.15.28:0/3488690961 >> v1:172.21.15.81:6789/0 conn(0x7f6bac1527a0 legacy=0x7f6bac0bf9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:35.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.347+0000 7f6bb0c00640 1 -- 172.21.15.28:0/3488690961 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bac0c2350 con 0x7f6bac06d410 2024-02-21T23:05:35.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.348+0000 7f6bb0c00640 1 --2- 172.21.15.28:0/3488690961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6bac06d410 0x7f6bac06d860 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f6ba4009e50 tx=0x7f6ba402bff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63c9fe9cb5af5d57 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:35.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.348+0000 7f6baaa00640 1 -- 172.21.15.28:0/3488690961 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba402cb90 con 0x7f6bac06d410 2024-02-21T23:05:35.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.348+0000 7f6baaa00640 1 -- 172.21.15.28:0/3488690961 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ba402cd30 con 0x7f6bac06d410 2024-02-21T23:05:35.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.348+0000 7f6baaa00640 1 -- 172.21.15.28:0/3488690961 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba4034610 con 0x7f6bac06d410 2024-02-21T23:05:35.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.349+0000 7f6bb1600640 1 -- 172.21.15.28:0/3488690961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6bac06d410 msgr2=0x7f6bac06d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:35.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.349+0000 7f6bb1600640 1 --2- 172.21.15.28:0/3488690961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6bac06d410 0x7f6bac06d860 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f6ba4009e50 tx=0x7f6ba402bff0 comp rx=0 tx=0).stop 2024-02-21T23:05:35.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.349+0000 7f6bb1600640 1 -- 172.21.15.28:0/3488690961 shutdown_connections 2024-02-21T23:05:35.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.349+0000 7f6bb1600640 1 --2- 172.21.15.28:0/3488690961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6bac06d410 0x7f6bac06d860 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:35.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.349+0000 7f6bb1600640 1 --2- 172.21.15.28:0/3488690961 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6bac158f70 0x7f6bac0c00b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:35.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.349+0000 7f6bb1600640 1 -- 172.21.15.28:0/3488690961 >> 172.21.15.28:0/3488690961 conn(0x7f6bac0a8320 msgr2=0x7f6bac0a8730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:35.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.349+0000 7f6bb1600640 1 -- 172.21.15.28:0/3488690961 shutdown_connections 2024-02-21T23:05:35.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.349+0000 7f6bb1600640 1 -- 172.21.15.28:0/3488690961 wait complete. 2024-02-21T23:05:35.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.351+0000 7f6bb1600640 1 Processor -- start 2024-02-21T23:05:35.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.351+0000 7f6bb1600640 1 -- start start 2024-02-21T23:05:35.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.352+0000 7f6bb1600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6bac06d410 0x7f6bac0a9910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:35.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.352+0000 7f6bb1600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6bac1527a0 0x7f6bac1ee1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:35.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.352+0000 7f6bb1600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6bac158f70 0x7f6bac1ee730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:35.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.352+0000 7f6bb1600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bac0c1d20 con 0x7f6bac158f70 2024-02-21T23:05:35.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.352+0000 7f6bb1600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bac159390 con 0x7f6bac1527a0 2024-02-21T23:05:35.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.352+0000 7f6bb1600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bac06dda0 con 0x7f6bac06d410 2024-02-21T23:05:35.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.352+0000 7f6bb0c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6bac158f70 0x7f6bac1ee730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:35.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.352+0000 7f6bb0c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6bac158f70 0x7f6bac1ee730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50520/0 (socket says 172.21.15.28:50520) 2024-02-21T23:05:35.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.352+0000 7f6bb0c00640 1 -- 172.21.15.28:0/2772885157 learned_addr learned my addr 172.21.15.28:0/2772885157 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:35.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.352+0000 7f6babe00640 1 --2- 172.21.15.28:0/2772885157 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6bac06d410 0x7f6bac0a9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:35.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.352+0000 7f6bab400640 1 --2- 172.21.15.28:0/2772885157 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6bac1527a0 0x7f6bac1ee1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:35.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.353+0000 7f6bb0c00640 1 -- 172.21.15.28:0/2772885157 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6bac06d410 msgr2=0x7f6bac0a9910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:35.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.353+0000 7f6bb0c00640 1 --2- 172.21.15.28:0/2772885157 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6bac06d410 0x7f6bac0a9910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:35.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.353+0000 7f6bb0c00640 1 -- 172.21.15.28:0/2772885157 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6bac1527a0 msgr2=0x7f6bac1ee1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:35.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.353+0000 7f6bb0c00640 1 --2- 172.21.15.28:0/2772885157 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6bac1527a0 0x7f6bac1ee1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:35.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.353+0000 7f6bb0c00640 1 -- 172.21.15.28:0/2772885157 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bac152fc0 con 0x7f6bac158f70 2024-02-21T23:05:35.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.353+0000 7f6babe00640 1 --2- 172.21.15.28:0/2772885157 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6bac06d410 0x7f6bac0a9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:05:35.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.353+0000 7f6bab400640 1 --2- 172.21.15.28:0/2772885157 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6bac1527a0 0x7f6bac1ee1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:05:35.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.353+0000 7f6bb0c00640 1 --2- 172.21.15.28:0/2772885157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6bac158f70 0x7f6bac1ee730 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f6ba4009f70 tx=0x7f6ba4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:35.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.354+0000 7f6ba3e00640 1 -- 172.21.15.28:0/2772885157 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba4043020 con 0x7f6bac158f70 2024-02-21T23:05:35.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.354+0000 7f6ba3e00640 1 -- 172.21.15.28:0/2772885157 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ba403e450 con 0x7f6bac158f70 2024-02-21T23:05:35.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.354+0000 7f6ba3e00640 1 -- 172.21.15.28:0/2772885157 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba4009510 con 0x7f6bac158f70 2024-02-21T23:05:35.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.354+0000 7f6bb1600640 1 -- 172.21.15.28:0/2772885157 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bac155400 con 0x7f6bac158f70 2024-02-21T23:05:35.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.354+0000 7f6bb1600640 1 -- 172.21.15.28:0/2772885157 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bac1eec70 con 0x7f6bac158f70 2024-02-21T23:05:35.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.355+0000 7f6bb1600640 1 -- 172.21.15.28:0/2772885157 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bac0c73f0 con 0x7f6bac158f70 2024-02-21T23:05:35.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.356+0000 7f6ba3e00640 1 -- 172.21.15.28:0/2772885157 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6ba4046020 con 0x7f6bac158f70 2024-02-21T23:05:35.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.357+0000 7f6ba3e00640 1 --2- 172.21.15.28:0/2772885157 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6b8c06b4d0 0x7f6b8c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:35.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.357+0000 7f6ba3e00640 1 -- 172.21.15.28:0/2772885157 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6594+0+0 (secure 0 0 0) 0x7f6ba40acbe0 con 0x7f6bac158f70 2024-02-21T23:05:35.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.358+0000 7f6babe00640 1 --2- 172.21.15.28:0/2772885157 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6b8c06b4d0 0x7f6b8c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:35.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.359+0000 7f6babe00640 1 --2- 172.21.15.28:0/2772885157 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6b8c06b4d0 0x7f6b8c06d990 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f6bac1574c0 tx=0x7f6b9c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:35.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.362+0000 7f6ba3e00640 1 -- 172.21.15.28:0/2772885157 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f6bac0c73f0 con 0x7f6bac158f70 2024-02-21T23:05:35.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.608+0000 7f6bb1600640 1 -- 172.21.15.28:0/2772885157 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs2"} v 0) v1 -- 0x7f6bac04f290 con 0x7f6bac158f70 2024-02-21T23:05:35.599 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-02-21T23:05:35.609+0000 7f5ddc400640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-21T23:05:35.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.613+0000 7f6ba3e00640 1 -- 172.21.15.28:0/2772885157 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs2"}]=0 cephfs2 marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v54) v1 ==== 163+0+0 (secure 0 0 0) 0x7f6ba404d030 con 0x7f6bac158f70 2024-02-21T23:05:35.602 INFO:teuthology.orchestra.run.smithi028.stderr:cephfs2 marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-02-21T23:05:35.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.616+0000 7f6bb1600640 1 -- 172.21.15.28:0/2772885157 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6b8c06b4d0 msgr2=0x7f6b8c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:35.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.616+0000 7f6bb1600640 1 --2- 172.21.15.28:0/2772885157 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6b8c06b4d0 0x7f6b8c06d990 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f6bac1574c0 tx=0x7f6b9c009040 comp rx=0 tx=0).stop 2024-02-21T23:05:35.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.616+0000 7f6bb1600640 1 -- 172.21.15.28:0/2772885157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6bac158f70 msgr2=0x7f6bac1ee730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:35.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.616+0000 7f6bb1600640 1 --2- 172.21.15.28:0/2772885157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6bac158f70 0x7f6bac1ee730 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f6ba4009f70 tx=0x7f6ba4004a40 comp rx=0 tx=0).stop 2024-02-21T23:05:35.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.617+0000 7f6bb1600640 1 -- 172.21.15.28:0/2772885157 shutdown_connections 2024-02-21T23:05:35.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.617+0000 7f6bb1600640 1 --2- 172.21.15.28:0/2772885157 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6b8c06b4d0 0x7f6b8c06d990 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:35.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.617+0000 7f6bb1600640 1 --2- 172.21.15.28:0/2772885157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6bac158f70 0x7f6bac1ee730 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:35.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.617+0000 7f6bb1600640 1 --2- 172.21.15.28:0/2772885157 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6bac1527a0 0x7f6bac1ee1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:35.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.617+0000 7f6bb1600640 1 --2- 172.21.15.28:0/2772885157 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6bac06d410 0x7f6bac0a9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:35.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.617+0000 7f6bb1600640 1 -- 172.21.15.28:0/2772885157 >> 172.21.15.28:0/2772885157 conn(0x7f6bac0a8320 msgr2=0x7f6bac157900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:35.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.617+0000 7f6bb1600640 1 -- 172.21.15.28:0/2772885157 shutdown_connections 2024-02-21T23:05:35.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.617+0000 7f6bb1600640 1 -- 172.21.15.28:0/2772885157 wait complete. 2024-02-21T23:05:35.629 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs2 --yes-i-really-mean-it 2024-02-21T23:05:35.637 INFO:tasks.ceph.mds.b.smithi081.stderr: -68> 2024-02-21T23:05:33.360+0000 7f02a6200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-21T23:05:35.705 INFO:tasks.ceph.mds.b.smithi081.stdout:starting mds.b at 2024-02-21T23:05:35.731 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-21T23:05:35.739 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-21T23:05:35.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.861+0000 7f8341800640 1 Processor -- start 2024-02-21T23:05:35.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.861+0000 7f8341800640 1 -- start start 2024-02-21T23:05:35.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.863+0000 7f8341800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f833c0c7590 0x7f833c0c6c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:35.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.863+0000 7f8341800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f833c0c5cc0 0x7f833c1524e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:35.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.863+0000 7f8341800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f833c159580 con 0x7f833c152a20 2024-02-21T23:05:35.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.863+0000 7f8341800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f833c0c6230 con 0x7f833c0c5cc0 2024-02-21T23:05:35.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.863+0000 7f8341800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833c1587c0 con 0x7f833c0c7590 2024-02-21T23:05:35.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.863+0000 7f8332000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f833c0c5cc0 0x7f833c1524e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:35.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.863+0000 7f833b400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f833c152a20 0x7f833c0c3030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:45348/0 (socket says 172.21.15.28:45348) 2024-02-21T23:05:35.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.863+0000 7f833b400640 1 -- 172.21.15.28:0/2825886871 learned_addr learned my addr 172.21.15.28:0/2825886871 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:05:35.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.863+0000 7f833aa00640 1 --2- 172.21.15.28:0/2825886871 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f833c0c7590 0x7f833c0c6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:35.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.863+0000 7f8332000640 1 -- 172.21.15.28:0/2825886871 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f833c0c7590 msgr2=0x7f833c0c6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:35.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.863+0000 7f8332000640 1 --2- 172.21.15.28:0/2825886871 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f833c0c7590 0x7f833c0c6c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:35.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.863+0000 7f8332000640 1 -- 172.21.15.28:0/2825886871 >> v1:172.21.15.28:6789/0 conn(0x7f833c152a20 legacy=0x7f833c0c3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:35.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.864+0000 7f8332000640 1 -- 172.21.15.28:0/2825886871 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f833c158aa0 con 0x7f833c0c5cc0 2024-02-21T23:05:35.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.865+0000 7f8332000640 1 --2- 172.21.15.28:0/2825886871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f833c0c5cc0 0x7f833c1524e0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f833c0c63b0 tx=0x7f832402b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cbc6e457ae63230 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:35.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.865+0000 7f833a000640 1 -- 172.21.15.28:0/2825886871 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8324031770 con 0x7f833c0c5cc0 2024-02-21T23:05:35.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.865+0000 7f833a000640 1 -- 172.21.15.28:0/2825886871 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8324037070 con 0x7f833c0c5cc0 2024-02-21T23:05:35.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.865+0000 7f833a000640 1 -- 172.21.15.28:0/2825886871 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8324031b30 con 0x7f833c0c5cc0 2024-02-21T23:05:35.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.866+0000 7f8341800640 1 -- 172.21.15.28:0/2825886871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f833c0c5cc0 msgr2=0x7f833c1524e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:35.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.866+0000 7f8341800640 1 --2- 172.21.15.28:0/2825886871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f833c0c5cc0 0x7f833c1524e0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f833c0c63b0 tx=0x7f832402b810 comp rx=0 tx=0).stop 2024-02-21T23:05:35.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.866+0000 7f8341800640 1 -- 172.21.15.28:0/2825886871 shutdown_connections 2024-02-21T23:05:35.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.866+0000 7f8341800640 1 --2- 172.21.15.28:0/2825886871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f833c0c5cc0 0x7f833c1524e0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:35.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.866+0000 7f8341800640 1 --2- 172.21.15.28:0/2825886871 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f833c0c7590 0x7f833c0c6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:35.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.866+0000 7f8341800640 1 -- 172.21.15.28:0/2825886871 >> 172.21.15.28:0/2825886871 conn(0x7f833c08afe0 msgr2=0x7f833c0725a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:35.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.866+0000 7f8341800640 1 -- 172.21.15.28:0/2825886871 shutdown_connections 2024-02-21T23:05:35.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.866+0000 7f8341800640 1 -- 172.21.15.28:0/2825886871 wait complete. 2024-02-21T23:05:35.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.867+0000 7f8341800640 1 Processor -- start 2024-02-21T23:05:35.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.868+0000 7f8341800640 1 -- start start 2024-02-21T23:05:35.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.868+0000 7f8341800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f833c0c5cc0 0x7f833c147fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:35.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.868+0000 7f8341800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f833c0c7590 0x7f833c145930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:35.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.868+0000 7f8341800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f833c152a20 0x7f833c0c88e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:35.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.869+0000 7f8341800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833c0c6720 con 0x7f833c0c5cc0 2024-02-21T23:05:35.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.869+0000 7f8341800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833c159060 con 0x7f833c0c7590 2024-02-21T23:05:35.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.869+0000 7f8341800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833c0c68a0 con 0x7f833c152a20 2024-02-21T23:05:35.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.869+0000 7f833aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f833c0c5cc0 0x7f833c147fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:35.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.869+0000 7f833aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f833c0c5cc0 0x7f833c147fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50536/0 (socket says 172.21.15.28:50536) 2024-02-21T23:05:35.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.869+0000 7f833aa00640 1 -- 172.21.15.28:0/771874447 learned_addr learned my addr 172.21.15.28:0/771874447 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:35.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.869+0000 7f8332000640 1 --2- 172.21.15.28:0/771874447 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f833c0c7590 0x7f833c145930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:35.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.869+0000 7f833b400640 1 --2- 172.21.15.28:0/771874447 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f833c152a20 0x7f833c0c88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:35.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.869+0000 7f833aa00640 1 -- 172.21.15.28:0/771874447 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f833c152a20 msgr2=0x7f833c0c88e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:35.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.869+0000 7f833aa00640 1 --2- 172.21.15.28:0/771874447 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f833c152a20 0x7f833c0c88e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:35.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.869+0000 7f833aa00640 1 -- 172.21.15.28:0/771874447 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f833c0c7590 msgr2=0x7f833c145930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:35.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.869+0000 7f833aa00640 1 --2- 172.21.15.28:0/771874447 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f833c0c7590 0x7f833c145930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:35.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.869+0000 7f833aa00640 1 -- 172.21.15.28:0/771874447 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f833c1edbc0 con 0x7f833c0c5cc0 2024-02-21T23:05:35.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.869+0000 7f833b400640 1 --2- 172.21.15.28:0/771874447 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f833c152a20 0x7f833c0c88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:05:35.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.870+0000 7f833aa00640 1 --2- 172.21.15.28:0/771874447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f833c0c5cc0 0x7f833c147fe0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f832c00d800 tx=0x7f832c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:35.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.870+0000 7f8333400640 1 -- 172.21.15.28:0/771874447 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f832c00b6c0 con 0x7f833c0c5cc0 2024-02-21T23:05:35.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.870+0000 7f8333400640 1 -- 172.21.15.28:0/771874447 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f832c019070 con 0x7f833c0c5cc0 2024-02-21T23:05:35.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.870+0000 7f8333400640 1 -- 172.21.15.28:0/771874447 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f832c00ba80 con 0x7f833c0c5cc0 2024-02-21T23:05:35.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.870+0000 7f8341800640 1 -- 172.21.15.28:0/771874447 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f833c0ca460 con 0x7f833c0c5cc0 2024-02-21T23:05:35.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.870+0000 7f8341800640 1 -- 172.21.15.28:0/771874447 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f833c1ee140 con 0x7f833c0c5cc0 2024-02-21T23:05:35.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.871+0000 7f8341800640 1 -- 172.21.15.28:0/771874447 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f833c067f30 con 0x7f833c0c5cc0 2024-02-21T23:05:35.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.872+0000 7f8333400640 1 -- 172.21.15.28:0/771874447 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f832c00bc20 con 0x7f833c0c5cc0 2024-02-21T23:05:35.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.873+0000 7f8333400640 1 --2- 172.21.15.28:0/771874447 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f831806b4d0 0x7f831806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:35.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.873+0000 7f8333400640 1 -- 172.21.15.28:0/771874447 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6680+0+0 (secure 0 0 0) 0x7f832c08aac0 con 0x7f833c0c5cc0 2024-02-21T23:05:35.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.874+0000 7f8332000640 1 --2- 172.21.15.28:0/771874447 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f831806b4d0 0x7f831806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:35.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.875+0000 7f8332000640 1 --2- 172.21.15.28:0/771874447 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f831806b4d0 0x7f831806d990 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f8324036000 tx=0x7f832403d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:35.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:35.878+0000 7f8333400640 1 -- 172.21.15.28:0/771874447 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f832c059560 con 0x7f833c0c5cc0 2024-02-21T23:05:36.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.117+0000 7f8341800640 1 -- 172.21.15.28:0/771874447 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs2", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f833c0c54f0 con 0x7f833c0c5cc0 2024-02-21T23:05:36.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.617+0000 7f8333400640 1 -- 172.21.15.28:0/771874447 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs2", "yes_i_really_mean_it": true}]=0 v55) v1 ==== 105+0+0 (secure 0 0 0) 0x7f832c058e60 con 0x7f833c0c5cc0 2024-02-21T23:05:36.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.620+0000 7f8341800640 1 -- 172.21.15.28:0/771874447 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f831806b4d0 msgr2=0x7f831806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:36.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.620+0000 7f8341800640 1 --2- 172.21.15.28:0/771874447 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f831806b4d0 0x7f831806d990 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f8324036000 tx=0x7f832403d040 comp rx=0 tx=0).stop 2024-02-21T23:05:36.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.620+0000 7f8341800640 1 -- 172.21.15.28:0/771874447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f833c0c5cc0 msgr2=0x7f833c147fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:36.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.620+0000 7f8341800640 1 --2- 172.21.15.28:0/771874447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f833c0c5cc0 0x7f833c147fe0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f832c00d800 tx=0x7f832c00dcc0 comp rx=0 tx=0).stop 2024-02-21T23:05:36.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.620+0000 7f8341800640 1 -- 172.21.15.28:0/771874447 shutdown_connections 2024-02-21T23:05:36.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.620+0000 7f8341800640 1 --2- 172.21.15.28:0/771874447 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f831806b4d0 0x7f831806d990 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:36.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.620+0000 7f8341800640 1 --2- 172.21.15.28:0/771874447 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f833c152a20 0x7f833c0c88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:36.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.620+0000 7f8341800640 1 --2- 172.21.15.28:0/771874447 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f833c0c7590 0x7f833c145930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:36.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.620+0000 7f8341800640 1 --2- 172.21.15.28:0/771874447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f833c0c5cc0 0x7f833c147fe0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:36.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.620+0000 7f8341800640 1 -- 172.21.15.28:0/771874447 >> 172.21.15.28:0/771874447 conn(0x7f833c08afe0 msgr2=0x7f833c14ff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:36.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.621+0000 7f8341800640 1 -- 172.21.15.28:0/771874447 shutdown_connections 2024-02-21T23:05:36.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.621+0000 7f8341800640 1 -- 172.21.15.28:0/771874447 wait complete. 2024-02-21T23:05:36.636 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs2_metadata cephfs2_metadata --yes-i-really-really-mean-it 2024-02-21T23:05:36.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.919+0000 7f4388600640 1 Processor -- start 2024-02-21T23:05:36.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.920+0000 7f4388600640 1 -- start start 2024-02-21T23:05:36.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.921+0000 7f4388600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f4380073150 0x7f4380156d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:36.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.921+0000 7f4388600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f43800c5770 0x7f43800c5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:36.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.921+0000 7f4388600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43801531b0 con 0x7f4380072660 2024-02-21T23:05:36.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.921+0000 7f4388600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4380153ae0 con 0x7f43800c5770 2024-02-21T23:05:36.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.921+0000 7f4388600640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4380153490 con 0x7f4380073150 2024-02-21T23:05:36.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.922+0000 7f4385c00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f4380072660 0x7f4380072a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:45350/0 (socket says 172.21.15.28:45350) 2024-02-21T23:05:36.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.922+0000 7f4385c00640 1 -- 172.21.15.28:0/742963685 learned_addr learned my addr 172.21.15.28:0/742963685 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:05:36.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.922+0000 7f4386600640 1 --2- 172.21.15.28:0/742963685 >> v2:172.21.15.28:3300/0 conn(0x7f43800c5770 0x7f43800c5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:36.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.922+0000 7f4385200640 1 --2- 172.21.15.28:0/742963685 >> v2:172.21.15.81:3300/0 conn(0x7f4380073150 0x7f4380156d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:36.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.922+0000 7f4386600640 1 -- 172.21.15.28:0/742963685 >> v2:172.21.15.81:3300/0 conn(0x7f4380073150 msgr2=0x7f4380156d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:36.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.922+0000 7f4386600640 1 --2- 172.21.15.28:0/742963685 >> v2:172.21.15.81:3300/0 conn(0x7f4380073150 0x7f4380156d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:36.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.922+0000 7f4386600640 1 -- 172.21.15.28:0/742963685 >> v1:172.21.15.28:6789/0 conn(0x7f4380072660 legacy=0x7f4380072a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:36.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.922+0000 7f4386600640 1 -- 172.21.15.28:0/742963685 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4380153770 con 0x7f43800c5770 2024-02-21T23:05:36.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.924+0000 7f4386600640 1 --2- 172.21.15.28:0/742963685 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43800c5770 0x7f43800c5b80 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f4380153c60 tx=0x7f437802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7433b5080c99fdf server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:36.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.924+0000 7f437fe00640 1 -- 172.21.15.28:0/742963685 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4378039070 con 0x7f43800c5770 2024-02-21T23:05:36.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.924+0000 7f437fe00640 1 -- 172.21.15.28:0/742963685 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f437802cc60 con 0x7f43800c5770 2024-02-21T23:05:36.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.924+0000 7f437fe00640 1 -- 172.21.15.28:0/742963685 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4378034660 con 0x7f43800c5770 2024-02-21T23:05:36.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.924+0000 7f4388600640 1 -- 172.21.15.28:0/742963685 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43800c5770 msgr2=0x7f43800c5b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:36.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.924+0000 7f4388600640 1 --2- 172.21.15.28:0/742963685 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43800c5770 0x7f43800c5b80 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f4380153c60 tx=0x7f437802beb0 comp rx=0 tx=0).stop 2024-02-21T23:05:36.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.925+0000 7f4388600640 1 -- 172.21.15.28:0/742963685 shutdown_connections 2024-02-21T23:05:36.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.925+0000 7f4388600640 1 --2- 172.21.15.28:0/742963685 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43800c5770 0x7f43800c5b80 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:36.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.925+0000 7f4388600640 1 --2- 172.21.15.28:0/742963685 >> v2:172.21.15.81:3300/0 conn(0x7f4380073150 0x7f4380156d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:36.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.925+0000 7f4388600640 1 -- 172.21.15.28:0/742963685 >> 172.21.15.28:0/742963685 conn(0x7f438006c5b0 msgr2=0x7f43800c7870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:36.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.925+0000 7f4388600640 1 -- 172.21.15.28:0/742963685 shutdown_connections 2024-02-21T23:05:36.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.925+0000 7f4388600640 1 -- 172.21.15.28:0/742963685 wait complete. 2024-02-21T23:05:36.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.926+0000 7f4388600640 1 Processor -- start 2024-02-21T23:05:36.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.927+0000 7f4388600640 1 -- start start 2024-02-21T23:05:36.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.927+0000 7f4388600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4380072660 0x7f4380152320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:36.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.928+0000 7f4388600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4380073150 0x7f43801dc690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:36.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.928+0000 7f4388600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43800c5770 0x7f43801dcbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:36.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.928+0000 7f4388600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43800c4c60 con 0x7f43800c5770 2024-02-21T23:05:36.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.928+0000 7f4388600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4380152e60 con 0x7f4380073150 2024-02-21T23:05:36.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.928+0000 7f4388600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43800c4de0 con 0x7f4380072660 2024-02-21T23:05:36.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.928+0000 7f4386600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43800c5770 0x7f43801dcbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:36.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.928+0000 7f4386600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43800c5770 0x7f43801dcbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50542/0 (socket says 172.21.15.28:50542) 2024-02-21T23:05:36.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.928+0000 7f4386600640 1 -- 172.21.15.28:0/2303012009 learned_addr learned my addr 172.21.15.28:0/2303012009 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:36.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.928+0000 7f4385200640 1 --2- 172.21.15.28:0/2303012009 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4380073150 0x7f43801dc690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:36.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.928+0000 7f4386600640 1 -- 172.21.15.28:0/2303012009 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4380072660 msgr2=0x7f4380152320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:36.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.928+0000 7f4385c00640 1 --2- 172.21.15.28:0/2303012009 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4380072660 0x7f4380152320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:36.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.928+0000 7f4386600640 1 --2- 172.21.15.28:0/2303012009 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4380072660 0x7f4380152320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:36.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.928+0000 7f4386600640 1 -- 172.21.15.28:0/2303012009 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4380073150 msgr2=0x7f43801dc690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:36.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.928+0000 7f4386600640 1 --2- 172.21.15.28:0/2303012009 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4380073150 0x7f43801dc690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:36.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.928+0000 7f4386600640 1 -- 172.21.15.28:0/2303012009 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43801dd110 con 0x7f43800c5770 2024-02-21T23:05:36.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.928+0000 7f4385200640 1 --2- 172.21.15.28:0/2303012009 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4380073150 0x7f43801dc690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:05:36.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.929+0000 7f4386600640 1 --2- 172.21.15.28:0/2303012009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43800c5770 0x7f43801dcbd0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f4378009e50 tx=0x7f4378004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:36.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.929+0000 7f437d600640 1 -- 172.21.15.28:0/2303012009 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4378039050 con 0x7f43800c5770 2024-02-21T23:05:36.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.929+0000 7f437d600640 1 -- 172.21.15.28:0/2303012009 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4378045070 con 0x7f43800c5770 2024-02-21T23:05:36.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.929+0000 7f437d600640 1 -- 172.21.15.28:0/2303012009 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f437803e830 con 0x7f43800c5770 2024-02-21T23:05:36.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.929+0000 7f4388600640 1 -- 172.21.15.28:0/2303012009 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43801ebc10 con 0x7f43800c5770 2024-02-21T23:05:36.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.929+0000 7f4388600640 1 -- 172.21.15.28:0/2303012009 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43801ee2b0 con 0x7f43800c5770 2024-02-21T23:05:36.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.930+0000 7f4388600640 1 -- 172.21.15.28:0/2303012009 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4380068640 con 0x7f43800c5770 2024-02-21T23:05:36.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.931+0000 7f437d600640 1 -- 172.21.15.28:0/2303012009 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f437803e9d0 con 0x7f43800c5770 2024-02-21T23:05:36.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.932+0000 7f437d600640 1 --2- 172.21.15.28:0/2303012009 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f436006b4d0 0x7f436006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:36.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.932+0000 7f437d600640 1 -- 172.21.15.28:0/2303012009 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6680+0+0 (secure 0 0 0) 0x7f43780ad310 con 0x7f43800c5770 2024-02-21T23:05:36.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.933+0000 7f4385c00640 1 --2- 172.21.15.28:0/2303012009 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f436006b4d0 0x7f436006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:36.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.934+0000 7f4385c00640 1 --2- 172.21.15.28:0/2303012009 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f436006b4d0 0x7f436006d990 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f4374000f60 tx=0x7f4374009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:36.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:36.937+0000 7f437d600640 1 -- 172.21.15.28:0/2303012009 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4378032e20 con 0x7f43800c5770 2024-02-21T23:05:37.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:37.136+0000 7f4388600640 1 -- 172.21.15.28:0/2303012009 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs2_metadata", "pool2": "cephfs2_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f4380074ab0 con 0x7f43800c5770 2024-02-21T23:05:37.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:37.857+0000 7f437d600640 1 -- 172.21.15.28:0/2303012009 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs2_metadata", "pool2": "cephfs2_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs2_metadata' removed v67) v1 ==== 184+0+0 (secure 0 0 0) 0x7f437807bdb0 con 0x7f43800c5770 2024-02-21T23:05:37.847 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs2_metadata' removed 2024-02-21T23:05:37.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:37.863+0000 7f4388600640 1 -- 172.21.15.28:0/2303012009 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f436006b4d0 msgr2=0x7f436006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:37.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:37.863+0000 7f4388600640 1 --2- 172.21.15.28:0/2303012009 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f436006b4d0 0x7f436006d990 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f4374000f60 tx=0x7f4374009040 comp rx=0 tx=0).stop 2024-02-21T23:05:37.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:37.864+0000 7f4388600640 1 -- 172.21.15.28:0/2303012009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43800c5770 msgr2=0x7f43801dcbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:37.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:37.864+0000 7f4388600640 1 --2- 172.21.15.28:0/2303012009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43800c5770 0x7f43801dcbd0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f4378009e50 tx=0x7f4378004a40 comp rx=0 tx=0).stop 2024-02-21T23:05:37.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:37.864+0000 7f4388600640 1 -- 172.21.15.28:0/2303012009 shutdown_connections 2024-02-21T23:05:37.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:37.864+0000 7f4388600640 1 --2- 172.21.15.28:0/2303012009 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f436006b4d0 0x7f436006d990 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:37.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:37.864+0000 7f4388600640 1 --2- 172.21.15.28:0/2303012009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43800c5770 0x7f43801dcbd0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:37.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:37.864+0000 7f4388600640 1 --2- 172.21.15.28:0/2303012009 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4380073150 0x7f43801dc690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:37.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:37.864+0000 7f4388600640 1 --2- 172.21.15.28:0/2303012009 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4380072660 0x7f4380152320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:37.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:37.864+0000 7f4388600640 1 -- 172.21.15.28:0/2303012009 >> 172.21.15.28:0/2303012009 conn(0x7f438006c5b0 msgr2=0x7f4380074360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:37.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:37.864+0000 7f4388600640 1 -- 172.21.15.28:0/2303012009 shutdown_connections 2024-02-21T23:05:37.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:37.865+0000 7f4388600640 1 -- 172.21.15.28:0/2303012009 wait complete. 2024-02-21T23:05:37.879 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs2_data cephfs2_data --yes-i-really-really-mean-it 2024-02-21T23:05:38.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.100+0000 7f7b5b400640 1 Processor -- start 2024-02-21T23:05:38.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.100+0000 7f7b5b400640 1 -- start start 2024-02-21T23:05:38.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.102+0000 7f7b5b400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f7b5406e780 0x7f7b5406eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:38.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.102+0000 7f7b5b400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7b540c5290 0x7f7b5415d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:38.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.102+0000 7f7b5b400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b54153cb0 con 0x7f7b540a5fc0 2024-02-21T23:05:38.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.102+0000 7f7b5b400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b540c5700 con 0x7f7b540c5290 2024-02-21T23:05:38.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.102+0000 7f7b5b400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b541543d0 con 0x7f7b5406e780 2024-02-21T23:05:38.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.102+0000 7f7b4be00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f7b540a5fc0 0x7f7b540c4b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:45352/0 (socket says 172.21.15.28:45352) 2024-02-21T23:05:38.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.102+0000 7f7b4be00640 1 -- 172.21.15.28:0/3371268313 learned_addr learned my addr 172.21.15.28:0/3371268313 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:05:38.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.102+0000 7f7b59400640 1 --2- 172.21.15.28:0/3371268313 >> v2:172.21.15.28:3300/0 conn(0x7f7b540c5290 0x7f7b5415d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:38.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.102+0000 7f7b58a00640 1 --2- 172.21.15.28:0/3371268313 >> v2:172.21.15.81:3300/0 conn(0x7f7b5406e780 0x7f7b5406eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:38.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.103+0000 7f7b53e00640 1 -- 172.21.15.28:0/3371268313 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3628359418 0 0) 0x7f7b54153cb0 con 0x7f7b540a5fc0 2024-02-21T23:05:38.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.103+0000 7f7b53e00640 1 -- 172.21.15.28:0/3371268313 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b38003520 con 0x7f7b540a5fc0 2024-02-21T23:05:38.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.103+0000 7f7b59400640 1 -- 172.21.15.28:0/3371268313 >> v2:172.21.15.81:3300/0 conn(0x7f7b5406e780 msgr2=0x7f7b5406eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:38.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.103+0000 7f7b59400640 1 --2- 172.21.15.28:0/3371268313 >> v2:172.21.15.81:3300/0 conn(0x7f7b5406e780 0x7f7b5406eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:38.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.103+0000 7f7b59400640 1 -- 172.21.15.28:0/3371268313 >> v1:172.21.15.28:6789/0 conn(0x7f7b540a5fc0 legacy=0x7f7b540c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:38.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.103+0000 7f7b59400640 1 -- 172.21.15.28:0/3371268313 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b540c5d20 con 0x7f7b540c5290 2024-02-21T23:05:38.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.104+0000 7f7b59400640 1 --2- 172.21.15.28:0/3371268313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b540c5290 0x7f7b5415d860 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f7b5406f810 tx=0x7f7b4c02be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33340f4470dd2f1f server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:38.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.104+0000 7f7b53e00640 1 -- 172.21.15.28:0/3371268313 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b4c039070 con 0x7f7b540c5290 2024-02-21T23:05:38.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.104+0000 7f7b53e00640 1 -- 172.21.15.28:0/3371268313 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b4c02cbd0 con 0x7f7b540c5290 2024-02-21T23:05:38.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.104+0000 7f7b53e00640 1 -- 172.21.15.28:0/3371268313 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b4c034440 con 0x7f7b540c5290 2024-02-21T23:05:38.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.105+0000 7f7b5b400640 1 -- 172.21.15.28:0/3371268313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b540c5290 msgr2=0x7f7b5415d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:38.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.105+0000 7f7b5b400640 1 --2- 172.21.15.28:0/3371268313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b540c5290 0x7f7b5415d860 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f7b5406f810 tx=0x7f7b4c02be20 comp rx=0 tx=0).stop 2024-02-21T23:05:38.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.105+0000 7f7b5b400640 1 -- 172.21.15.28:0/3371268313 shutdown_connections 2024-02-21T23:05:38.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.105+0000 7f7b5b400640 1 --2- 172.21.15.28:0/3371268313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b540c5290 0x7f7b5415d860 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:38.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.105+0000 7f7b5b400640 1 --2- 172.21.15.28:0/3371268313 >> v2:172.21.15.81:3300/0 conn(0x7f7b5406e780 0x7f7b5406eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:38.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.105+0000 7f7b5b400640 1 -- 172.21.15.28:0/3371268313 >> 172.21.15.28:0/3371268313 conn(0x7f7b5406b830 msgr2=0x7f7b540c67f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:38.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.105+0000 7f7b5b400640 1 -- 172.21.15.28:0/3371268313 shutdown_connections 2024-02-21T23:05:38.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.105+0000 7f7b5b400640 1 -- 172.21.15.28:0/3371268313 wait complete. 2024-02-21T23:05:38.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.107+0000 7f7b5b400640 1 Processor -- start 2024-02-21T23:05:38.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.107+0000 7f7b5b400640 1 -- start start 2024-02-21T23:05:38.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.108+0000 7f7b5b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b5406e780 0x7f7b54151ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:38.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.108+0000 7f7b5b400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7b540a5fc0 0x7f7b5414d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:38.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.108+0000 7f7b5b400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7b540c5290 0x7f7b5414de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:38.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.108+0000 7f7b5b400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b540c46f0 con 0x7f7b5406e780 2024-02-21T23:05:38.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.108+0000 7f7b5b400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b540c5ad0 con 0x7f7b540a5fc0 2024-02-21T23:05:38.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.108+0000 7f7b5b400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b54153960 con 0x7f7b540c5290 2024-02-21T23:05:38.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.108+0000 7f7b58a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b5406e780 0x7f7b54151ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:38.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.108+0000 7f7b58a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b5406e780 0x7f7b54151ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50562/0 (socket says 172.21.15.28:50562) 2024-02-21T23:05:38.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.108+0000 7f7b58a00640 1 -- 172.21.15.28:0/3780829594 learned_addr learned my addr 172.21.15.28:0/3780829594 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:38.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.108+0000 7f7b58a00640 1 -- 172.21.15.28:0/3780829594 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7b540c5290 msgr2=0x7f7b5414de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:38.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.108+0000 7f7b58a00640 1 --2- 172.21.15.28:0/3780829594 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7b540c5290 0x7f7b5414de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:38.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.108+0000 7f7b4be00640 1 --2- 172.21.15.28:0/3780829594 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7b540a5fc0 0x7f7b5414d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:38.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.108+0000 7f7b58a00640 1 -- 172.21.15.28:0/3780829594 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7b540a5fc0 msgr2=0x7f7b5414d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:38.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.108+0000 7f7b58a00640 1 --2- 172.21.15.28:0/3780829594 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7b540a5fc0 0x7f7b5414d910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:38.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.109+0000 7f7b58a00640 1 -- 172.21.15.28:0/3780829594 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b5414e390 con 0x7f7b5406e780 2024-02-21T23:05:38.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.109+0000 7f7b58a00640 1 --2- 172.21.15.28:0/3780829594 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b5406e780 0x7f7b54151ed0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f7b4400d5a0 tx=0x7f7b4400da60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:38.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.109+0000 7f7b51600640 1 -- 172.21.15.28:0/3780829594 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b440125f0 con 0x7f7b5406e780 2024-02-21T23:05:38.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.109+0000 7f7b51600640 1 -- 172.21.15.28:0/3780829594 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b44019070 con 0x7f7b5406e780 2024-02-21T23:05:38.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.109+0000 7f7b51600640 1 -- 172.21.15.28:0/3780829594 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b440129d0 con 0x7f7b5406e780 2024-02-21T23:05:38.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.109+0000 7f7b5b400640 1 -- 172.21.15.28:0/3780829594 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b540a9230 con 0x7f7b5406e780 2024-02-21T23:05:38.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.109+0000 7f7b5b400640 1 -- 172.21.15.28:0/3780829594 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b540bbc90 con 0x7f7b5406e780 2024-02-21T23:05:38.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.111+0000 7f7b5b400640 1 -- 172.21.15.28:0/3780829594 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b54067f30 con 0x7f7b5406e780 2024-02-21T23:05:38.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.112+0000 7f7b51600640 1 -- 172.21.15.28:0/3780829594 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7b44012b70 con 0x7f7b5406e780 2024-02-21T23:05:38.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.112+0000 7f7b51600640 1 --2- 172.21.15.28:0/3780829594 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7b3006b4d0 0x7f7b3006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:38.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.113+0000 7f7b51600640 1 -- 172.21.15.28:0/3780829594 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6211+0+0 (secure 0 0 0) 0x7f7b4408a6a0 con 0x7f7b5406e780 2024-02-21T23:05:38.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.113+0000 7f7b4be00640 1 --2- 172.21.15.28:0/3780829594 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7b3006b4d0 0x7f7b3006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:38.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.114+0000 7f7b4be00640 1 --2- 172.21.15.28:0/3780829594 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7b3006b4d0 0x7f7b3006d990 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c001360 tx=0x7f7b3c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:38.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.117+0000 7f7b51600640 1 -- 172.21.15.28:0/3780829594 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7b440593b0 con 0x7f7b5406e780 2024-02-21T23:05:38.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.318+0000 7f7b5b400640 1 -- 172.21.15.28:0/3780829594 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs2_data", "pool2": "cephfs2_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f7b540c58b0 con 0x7f7b5406e780 2024-02-21T23:05:38.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.860+0000 7f7b51600640 1 -- 172.21.15.28:0/3780829594 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs2_data", "pool2": "cephfs2_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs2_data' removed v68) v1 ==== 172+0+0 (secure 0 0 0) 0x7f7b44058cb0 con 0x7f7b5406e780 2024-02-21T23:05:38.850 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs2_data' removed 2024-02-21T23:05:38.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.865+0000 7f7b5b400640 1 -- 172.21.15.28:0/3780829594 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7b3006b4d0 msgr2=0x7f7b3006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:38.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.865+0000 7f7b5b400640 1 --2- 172.21.15.28:0/3780829594 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7b3006b4d0 0x7f7b3006d990 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c001360 tx=0x7f7b3c009040 comp rx=0 tx=0).stop 2024-02-21T23:05:38.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.865+0000 7f7b5b400640 1 -- 172.21.15.28:0/3780829594 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b5406e780 msgr2=0x7f7b54151ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:38.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.865+0000 7f7b5b400640 1 --2- 172.21.15.28:0/3780829594 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b5406e780 0x7f7b54151ed0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f7b4400d5a0 tx=0x7f7b4400da60 comp rx=0 tx=0).stop 2024-02-21T23:05:38.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.867+0000 7f7b5b400640 1 -- 172.21.15.28:0/3780829594 shutdown_connections 2024-02-21T23:05:38.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.867+0000 7f7b5b400640 1 --2- 172.21.15.28:0/3780829594 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7b3006b4d0 0x7f7b3006d990 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:38.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.867+0000 7f7b5b400640 1 --2- 172.21.15.28:0/3780829594 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7b540c5290 0x7f7b5414de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:38.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.867+0000 7f7b5b400640 1 --2- 172.21.15.28:0/3780829594 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7b540a5fc0 0x7f7b5414d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:38.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.867+0000 7f7b5b400640 1 --2- 172.21.15.28:0/3780829594 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b5406e780 0x7f7b54151ed0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:38.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.867+0000 7f7b5b400640 1 -- 172.21.15.28:0/3780829594 >> 172.21.15.28:0/3780829594 conn(0x7f7b5406b830 msgr2=0x7f7b541560f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:38.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.872+0000 7f7b5b400640 1 -- 172.21.15.28:0/3780829594 shutdown_connections 2024-02-21T23:05:38.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:38.872+0000 7f7b5b400640 1 -- 172.21.15.28:0/3780829594 wait complete. 2024-02-21T23:05:38.888 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.b mds_join_fs 2024-02-21T23:05:39.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.095+0000 7f5b24e00640 1 Processor -- start 2024-02-21T23:05:39.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.095+0000 7f5b24e00640 1 -- start start 2024-02-21T23:05:39.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.097+0000 7f5b24e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5b201500f0 0x7f5b2006ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:39.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.097+0000 7f5b24e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f5b200c2c60 0x7f5b2006dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:39.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.097+0000 7f5b24e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5b2006b3b0 0x7f5b2006e530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:39.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.097+0000 7f5b24e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b200c1080 con 0x7f5b201500f0 2024-02-21T23:05:39.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.097+0000 7f5b24e00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b2006db10 con 0x7f5b200c2c60 2024-02-21T23:05:39.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.097+0000 7f5b24e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b201539a0 con 0x7f5b2006b3b0 2024-02-21T23:05:39.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.097+0000 7f5b1f400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5b201500f0 0x7f5b2006ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:39.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.097+0000 7f5b1f400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5b201500f0 0x7f5b2006ae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50570/0 (socket says 172.21.15.28:50570) 2024-02-21T23:05:39.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.097+0000 7f5b1f400640 1 -- 172.21.15.28:0/3058303376 learned_addr learned my addr 172.21.15.28:0/3058303376 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:39.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.098+0000 7f5b16a00640 1 --2- 172.21.15.28:0/3058303376 >> v2:172.21.15.81:3300/0 conn(0x7f5b200c2c60 0x7f5b2006dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:39.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.098+0000 7f5b1f400640 1 -- 172.21.15.28:0/3058303376 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5b2006b3b0 msgr2=0x7f5b2006e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:39.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.098+0000 7f5b1fe00640 1 --2- 172.21.15.28:0/3058303376 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5b2006b3b0 0x7f5b2006e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:39.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.098+0000 7f5b1f400640 1 --2- 172.21.15.28:0/3058303376 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5b2006b3b0 0x7f5b2006e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:39.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.098+0000 7f5b1f400640 1 -- 172.21.15.28:0/3058303376 >> v2:172.21.15.81:3300/0 conn(0x7f5b200c2c60 msgr2=0x7f5b2006dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:39.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.098+0000 7f5b1f400640 1 --2- 172.21.15.28:0/3058303376 >> v2:172.21.15.81:3300/0 conn(0x7f5b200c2c60 0x7f5b2006dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:39.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.098+0000 7f5b1f400640 1 -- 172.21.15.28:0/3058303376 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b200c12d0 con 0x7f5b201500f0 2024-02-21T23:05:39.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.098+0000 7f5b16a00640 1 --2- 172.21.15.28:0/3058303376 >> v2:172.21.15.81:3300/0 conn(0x7f5b200c2c60 0x7f5b2006dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:05:39.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.098+0000 7f5b1fe00640 1 --2- 172.21.15.28:0/3058303376 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5b2006b3b0 0x7f5b2006e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:05:39.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.099+0000 7f5b1f400640 1 --2- 172.21.15.28:0/3058303376 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b201500f0 0x7f5b2006ae70 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f5b200a94a0 tx=0x7f5b1002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fedc835d82faa916 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:39.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.099+0000 7f5b1ea00640 1 -- 172.21.15.28:0/3058303376 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b10035020 con 0x7f5b201500f0 2024-02-21T23:05:39.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.100+0000 7f5b1ea00640 1 -- 172.21.15.28:0/3058303376 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b10031870 con 0x7f5b201500f0 2024-02-21T23:05:39.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.100+0000 7f5b1ea00640 1 -- 172.21.15.28:0/3058303376 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b10031c40 con 0x7f5b201500f0 2024-02-21T23:05:39.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.100+0000 7f5b24e00640 1 -- 172.21.15.28:0/3058303376 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b201500f0 msgr2=0x7f5b2006ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:39.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.100+0000 7f5b24e00640 1 --2- 172.21.15.28:0/3058303376 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b201500f0 0x7f5b2006ae70 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f5b200a94a0 tx=0x7f5b1002b910 comp rx=0 tx=0).stop 2024-02-21T23:05:39.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.100+0000 7f5b24e00640 1 -- 172.21.15.28:0/3058303376 shutdown_connections 2024-02-21T23:05:39.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.100+0000 7f5b24e00640 1 --2- 172.21.15.28:0/3058303376 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5b2006b3b0 0x7f5b2006e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:39.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.100+0000 7f5b24e00640 1 --2- 172.21.15.28:0/3058303376 >> v2:172.21.15.81:3300/0 conn(0x7f5b200c2c60 0x7f5b2006dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:39.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.100+0000 7f5b24e00640 1 --2- 172.21.15.28:0/3058303376 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b201500f0 0x7f5b2006ae70 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:39.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.100+0000 7f5b24e00640 1 -- 172.21.15.28:0/3058303376 >> 172.21.15.28:0/3058303376 conn(0x7f5b200a8530 msgr2=0x7f5b200a8910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:39.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.100+0000 7f5b24e00640 1 -- 172.21.15.28:0/3058303376 shutdown_connections 2024-02-21T23:05:39.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.100+0000 7f5b24e00640 1 -- 172.21.15.28:0/3058303376 wait complete. 2024-02-21T23:05:39.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.102+0000 7f5b24e00640 1 Processor -- start 2024-02-21T23:05:39.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.102+0000 7f5b24e00640 1 -- start start 2024-02-21T23:05:39.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.103+0000 7f5b24e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b2006b3b0 0x7f5b200c5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:39.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.103+0000 7f5b24e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b200c2c60 0x7f5b200c3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:39.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.103+0000 7f5b24e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5b201500f0 0x7f5b20152310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:39.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.103+0000 7f5b24e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b201535e0 con 0x7f5b200c2c60 2024-02-21T23:05:39.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.103+0000 7f5b24e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b201504d0 con 0x7f5b2006b3b0 2024-02-21T23:05:39.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.103+0000 7f5b24e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b20153760 con 0x7f5b201500f0 2024-02-21T23:05:39.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.104+0000 7f5b16a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b200c2c60 0x7f5b200c3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:39.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.104+0000 7f5b16a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b200c2c60 0x7f5b200c3ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50574/0 (socket says 172.21.15.28:50574) 2024-02-21T23:05:39.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.104+0000 7f5b16a00640 1 -- 172.21.15.28:0/3228528193 learned_addr learned my addr 172.21.15.28:0/3228528193 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:39.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.104+0000 7f5b1f400640 1 --2- 172.21.15.28:0/3228528193 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b2006b3b0 0x7f5b200c5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:39.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.104+0000 7f5b1fe00640 1 --2- 172.21.15.28:0/3228528193 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5b201500f0 0x7f5b20152310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:39.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.104+0000 7f5b16a00640 1 -- 172.21.15.28:0/3228528193 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5b201500f0 msgr2=0x7f5b20152310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:39.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.104+0000 7f5b16a00640 1 --2- 172.21.15.28:0/3228528193 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5b201500f0 0x7f5b20152310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:39.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.104+0000 7f5b16a00640 1 -- 172.21.15.28:0/3228528193 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b2006b3b0 msgr2=0x7f5b200c5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:39.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.104+0000 7f5b16a00640 1 --2- 172.21.15.28:0/3228528193 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b2006b3b0 0x7f5b200c5490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:39.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.104+0000 7f5b16a00640 1 -- 172.21.15.28:0/3228528193 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b20152850 con 0x7f5b200c2c60 2024-02-21T23:05:39.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.104+0000 7f5b16a00640 1 --2- 172.21.15.28:0/3228528193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b200c2c60 0x7f5b200c3ba0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f5b0800c7f0 tx=0x7f5b0800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:39.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.105+0000 7f5b17e00640 1 -- 172.21.15.28:0/3228528193 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b08015070 con 0x7f5b200c2c60 2024-02-21T23:05:39.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.105+0000 7f5b17e00640 1 -- 172.21.15.28:0/3228528193 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b080079d0 con 0x7f5b200c2c60 2024-02-21T23:05:39.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.105+0000 7f5b17e00640 1 -- 172.21.15.28:0/3228528193 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b08007da0 con 0x7f5b200c2c60 2024-02-21T23:05:39.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.105+0000 7f5b24e00640 1 -- 172.21.15.28:0/3228528193 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b20152da0 con 0x7f5b200c2c60 2024-02-21T23:05:39.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.105+0000 7f5b24e00640 1 -- 172.21.15.28:0/3228528193 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b20152ac0 con 0x7f5b200c2c60 2024-02-21T23:05:39.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.106+0000 7f5b24e00640 1 -- 172.21.15.28:0/3228528193 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b2004f290 con 0x7f5b200c2c60 2024-02-21T23:05:39.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.107+0000 7f5b17e00640 1 -- 172.21.15.28:0/3228528193 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5b08018070 con 0x7f5b200c2c60 2024-02-21T23:05:39.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.108+0000 7f5b17e00640 1 --2- 172.21.15.28:0/3228528193 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5afc06b4d0 0x7f5afc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:39.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.108+0000 7f5b17e00640 1 -- 172.21.15.28:0/3228528193 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5679+0+0 (secure 0 0 0) 0x7f5b0808a130 con 0x7f5b200c2c60 2024-02-21T23:05:39.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.109+0000 7f5b1f400640 1 --2- 172.21.15.28:0/3228528193 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5afc06b4d0 0x7f5afc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:39.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.110+0000 7f5b1f400640 1 --2- 172.21.15.28:0/3228528193 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5afc06b4d0 0x7f5afc06d990 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f5b10036000 tx=0x7f5b10008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:39.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.113+0000 7f5b17e00640 1 -- 172.21.15.28:0/3228528193 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f5b08058f30 con 0x7f5b200c2c60 2024-02-21T23:05:39.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.318+0000 7f5b24e00640 1 -- 172.21.15.28:0/3228528193 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.b", "name": "mds_join_fs"} v 0) v1 -- 0x7f5b2014bc30 con 0x7f5b200c2c60 2024-02-21T23:05:39.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.323+0000 7f5b17e00640 1 -- 172.21.15.28:0/3228528193 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.b", "name": "mds_join_fs"}]=0 v19) v1 ==== 96+0+0 (secure 0 0 0) 0x7f5b08058830 con 0x7f5b200c2c60 2024-02-21T23:05:39.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.326+0000 7f5b24e00640 1 -- 172.21.15.28:0/3228528193 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5afc06b4d0 msgr2=0x7f5afc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:39.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.326+0000 7f5b24e00640 1 --2- 172.21.15.28:0/3228528193 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5afc06b4d0 0x7f5afc06d990 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f5b10036000 tx=0x7f5b10008030 comp rx=0 tx=0).stop 2024-02-21T23:05:39.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.326+0000 7f5b24e00640 1 -- 172.21.15.28:0/3228528193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b200c2c60 msgr2=0x7f5b200c3ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:39.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.326+0000 7f5b24e00640 1 --2- 172.21.15.28:0/3228528193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b200c2c60 0x7f5b200c3ba0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f5b0800c7f0 tx=0x7f5b0800ccb0 comp rx=0 tx=0).stop 2024-02-21T23:05:39.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.327+0000 7f5b24e00640 1 -- 172.21.15.28:0/3228528193 shutdown_connections 2024-02-21T23:05:39.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.327+0000 7f5b24e00640 1 --2- 172.21.15.28:0/3228528193 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5afc06b4d0 0x7f5afc06d990 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:39.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.327+0000 7f5b24e00640 1 --2- 172.21.15.28:0/3228528193 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5b201500f0 0x7f5b20152310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:39.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.327+0000 7f5b24e00640 1 --2- 172.21.15.28:0/3228528193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b200c2c60 0x7f5b200c3ba0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:39.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.327+0000 7f5b24e00640 1 --2- 172.21.15.28:0/3228528193 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b2006b3b0 0x7f5b200c5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:39.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.327+0000 7f5b24e00640 1 -- 172.21.15.28:0/3228528193 >> 172.21.15.28:0/3228528193 conn(0x7f5b200a8530 msgr2=0x7f5b20149030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:39.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.327+0000 7f5b24e00640 1 -- 172.21.15.28:0/3228528193 shutdown_connections 2024-02-21T23:05:39.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.327+0000 7f5b24e00640 1 -- 172.21.15.28:0/3228528193 wait complete. 2024-02-21T23:05:39.336 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.d mds_join_fs 2024-02-21T23:05:39.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.571+0000 7fe942e00640 1 Processor -- start 2024-02-21T23:05:39.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.571+0000 7fe942e00640 1 -- start start 2024-02-21T23:05:39.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.573+0000 7fe942e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe93c06f400 0x7fe93c06e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:39.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.573+0000 7fe942e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe93c06db10 con 0x7fe93c0c2c60 2024-02-21T23:05:39.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.573+0000 7fe942e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe93c1539a0 con 0x7fe93c1500f0 2024-02-21T23:05:39.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.573+0000 7fe942e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe93c0c1080 con 0x7fe93c06f400 2024-02-21T23:05:39.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.573+0000 7fe942400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe93c06f400 0x7fe93c06e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:39.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.573+0000 7fe941000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe93c0c2c60 0x7fe93c06dff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:45366/0 (socket says 172.21.15.28:45366) 2024-02-21T23:05:39.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.573+0000 7fe941000640 1 -- 172.21.15.28:0/1484450396 learned_addr learned my addr 172.21.15.28:0/1484450396 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:05:39.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.574+0000 7fe942400640 1 -- 172.21.15.28:0/1484450396 >> v1:172.21.15.81:6789/0 conn(0x7fe93c1500f0 legacy=0x7fe93c06ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:39.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.574+0000 7fe942400640 1 -- 172.21.15.28:0/1484450396 >> v1:172.21.15.28:6789/0 conn(0x7fe93c0c2c60 legacy=0x7fe93c06dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:39.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.574+0000 7fe93be00640 1 -- 172.21.15.28:0/1484450396 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 566934330 0 0) 0x7fe93c06db10 con 0x7fe93c0c2c60 2024-02-21T23:05:39.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.574+0000 7fe942400640 1 -- 172.21.15.28:0/1484450396 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe93c0c12d0 con 0x7fe93c06f400 2024-02-21T23:05:39.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.575+0000 7fe942400640 1 --2- 172.21.15.28:0/1484450396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe93c06f400 0x7fe93c06e700 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fe93c0c0b80 tx=0x7fe93402be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bcec1fa3708549f server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:39.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.575+0000 7fe93be00640 1 -- 172.21.15.28:0/1484450396 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe934039070 con 0x7fe93c06f400 2024-02-21T23:05:39.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.575+0000 7fe93be00640 1 -- 172.21.15.28:0/1484450396 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe93402cc40 con 0x7fe93c06f400 2024-02-21T23:05:39.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.575+0000 7fe93be00640 1 -- 172.21.15.28:0/1484450396 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe934034440 con 0x7fe93c06f400 2024-02-21T23:05:39.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.576+0000 7fe942e00640 1 -- 172.21.15.28:0/1484450396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe93c06f400 msgr2=0x7fe93c06e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:39.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.576+0000 7fe942e00640 1 --2- 172.21.15.28:0/1484450396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe93c06f400 0x7fe93c06e700 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fe93c0c0b80 tx=0x7fe93402be90 comp rx=0 tx=0).stop 2024-02-21T23:05:39.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.576+0000 7fe942e00640 1 -- 172.21.15.28:0/1484450396 shutdown_connections 2024-02-21T23:05:39.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.576+0000 7fe942e00640 1 --2- 172.21.15.28:0/1484450396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe93c06f400 0x7fe93c06e700 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:39.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.576+0000 7fe942e00640 1 -- 172.21.15.28:0/1484450396 >> 172.21.15.28:0/1484450396 conn(0x7fe93c0a8530 msgr2=0x7fe93c0a8910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:39.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.576+0000 7fe942e00640 1 -- 172.21.15.28:0/1484450396 shutdown_connections 2024-02-21T23:05:39.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.576+0000 7fe942e00640 1 -- 172.21.15.28:0/1484450396 wait complete. 2024-02-21T23:05:39.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.578+0000 7fe942e00640 1 Processor -- start 2024-02-21T23:05:39.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.578+0000 7fe942e00640 1 -- start start 2024-02-21T23:05:39.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.579+0000 7fe942e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe93c06f400 0x7fe93c1ee5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:39.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.579+0000 7fe942e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe93c0c2c60 0x7fe93c0717d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:39.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.579+0000 7fe942e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe93c1500f0 0x7fe93c0bfe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:39.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.579+0000 7fe942e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe93c06b580 con 0x7fe93c0c2c60 2024-02-21T23:05:39.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.579+0000 7fe942e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe93c1504d0 con 0x7fe93c1500f0 2024-02-21T23:05:39.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.579+0000 7fe942e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe93c06b700 con 0x7fe93c06f400 2024-02-21T23:05:39.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.579+0000 7fe941000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe93c0c2c60 0x7fe93c0717d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:39.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.579+0000 7fe941000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe93c0c2c60 0x7fe93c0717d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50598/0 (socket says 172.21.15.28:50598) 2024-02-21T23:05:39.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.579+0000 7fe941000640 1 -- 172.21.15.28:0/372910873 learned_addr learned my addr 172.21.15.28:0/372910873 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:39.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.580+0000 7fe941a00640 1 --2- 172.21.15.28:0/372910873 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe93c06f400 0x7fe93c1ee5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:39.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.580+0000 7fe942400640 1 --2- 172.21.15.28:0/372910873 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe93c1500f0 0x7fe93c0bfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:39.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.580+0000 7fe941000640 1 -- 172.21.15.28:0/372910873 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe93c06f400 msgr2=0x7fe93c1ee5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:39.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.580+0000 7fe941000640 1 --2- 172.21.15.28:0/372910873 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe93c06f400 0x7fe93c1ee5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:39.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.580+0000 7fe941000640 1 -- 172.21.15.28:0/372910873 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe93c1500f0 msgr2=0x7fe93c0bfe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:39.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.580+0000 7fe941000640 1 --2- 172.21.15.28:0/372910873 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe93c1500f0 0x7fe93c0bfe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:39.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.580+0000 7fe941000640 1 -- 172.21.15.28:0/372910873 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe93c1debe0 con 0x7fe93c0c2c60 2024-02-21T23:05:39.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.580+0000 7fe941000640 1 --2- 172.21.15.28:0/372910873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe93c0c2c60 0x7fe93c0717d0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fe93000c4b0 tx=0x7fe93000c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:39.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.580+0000 7fe939600640 1 -- 172.21.15.28:0/372910873 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe930018070 con 0x7fe93c0c2c60 2024-02-21T23:05:39.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.581+0000 7fe939600640 1 -- 172.21.15.28:0/372910873 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe930007500 con 0x7fe93c0c2c60 2024-02-21T23:05:39.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.581+0000 7fe939600640 1 -- 172.21.15.28:0/372910873 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe930004a20 con 0x7fe93c0c2c60 2024-02-21T23:05:39.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.581+0000 7fe942e00640 1 -- 172.21.15.28:0/372910873 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe93c1deec0 con 0x7fe93c0c2c60 2024-02-21T23:05:39.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.581+0000 7fe942e00640 1 -- 172.21.15.28:0/372910873 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe93c0c0360 con 0x7fe93c0c2c60 2024-02-21T23:05:39.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.582+0000 7fe942e00640 1 -- 172.21.15.28:0/372910873 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe93c04f290 con 0x7fe93c0c2c60 2024-02-21T23:05:39.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.583+0000 7fe939600640 1 -- 172.21.15.28:0/372910873 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe930004bc0 con 0x7fe93c0c2c60 2024-02-21T23:05:39.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.585+0000 7fe939600640 1 --2- 172.21.15.28:0/372910873 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe92006b4d0 0x7fe92006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:39.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.585+0000 7fe939600640 1 -- 172.21.15.28:0/372910873 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5679+0+0 (secure 0 0 0) 0x7fe930089520 con 0x7fe93c0c2c60 2024-02-21T23:05:39.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.585+0000 7fe941a00640 1 --2- 172.21.15.28:0/372910873 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe92006b4d0 0x7fe92006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:39.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.587+0000 7fe941a00640 1 --2- 172.21.15.28:0/372910873 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe92006b4d0 0x7fe92006d990 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fe92c006fd0 tx=0x7fe92c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:39.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.589+0000 7fe939600640 1 -- 172.21.15.28:0/372910873 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe9300583e0 con 0x7fe93c0c2c60 2024-02-21T23:05:39.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.790+0000 7fe942e00640 1 -- 172.21.15.28:0/372910873 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.d", "name": "mds_join_fs"} v 0) v1 -- 0x7fe93c06f1e0 con 0x7fe93c0c2c60 2024-02-21T23:05:39.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.797+0000 7fe939600640 1 -- 172.21.15.28:0/372910873 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.d", "name": "mds_join_fs"}]=0 v20) v1 ==== 96+0+0 (secure 0 0 0) 0x7fe930057ce0 con 0x7fe93c0c2c60 2024-02-21T23:05:39.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.800+0000 7fe942e00640 1 -- 172.21.15.28:0/372910873 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe92006b4d0 msgr2=0x7fe92006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:39.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.800+0000 7fe942e00640 1 --2- 172.21.15.28:0/372910873 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe92006b4d0 0x7fe92006d990 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fe92c006fd0 tx=0x7fe92c008040 comp rx=0 tx=0).stop 2024-02-21T23:05:39.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.800+0000 7fe942e00640 1 -- 172.21.15.28:0/372910873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe93c0c2c60 msgr2=0x7fe93c0717d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:39.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.800+0000 7fe942e00640 1 --2- 172.21.15.28:0/372910873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe93c0c2c60 0x7fe93c0717d0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fe93000c4b0 tx=0x7fe93000c970 comp rx=0 tx=0).stop 2024-02-21T23:05:39.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.800+0000 7fe942e00640 1 -- 172.21.15.28:0/372910873 shutdown_connections 2024-02-21T23:05:39.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.800+0000 7fe942e00640 1 --2- 172.21.15.28:0/372910873 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe92006b4d0 0x7fe92006d990 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:39.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.800+0000 7fe942e00640 1 --2- 172.21.15.28:0/372910873 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe93c1500f0 0x7fe93c0bfe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:39.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.800+0000 7fe942e00640 1 --2- 172.21.15.28:0/372910873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe93c0c2c60 0x7fe93c0717d0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:39.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.800+0000 7fe942e00640 1 --2- 172.21.15.28:0/372910873 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe93c06f400 0x7fe93c1ee5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:39.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.800+0000 7fe942e00640 1 -- 172.21.15.28:0/372910873 >> 172.21.15.28:0/372910873 conn(0x7fe93c0a8530 msgr2=0x7fe93c14e2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:39.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.800+0000 7fe942e00640 1 -- 172.21.15.28:0/372910873 shutdown_connections 2024-02-21T23:05:39.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:39.801+0000 7fe942e00640 1 -- 172.21.15.28:0/372910873 wait complete. 2024-02-21T23:05:39.811 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.c mds_join_fs 2024-02-21T23:05:40.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.042+0000 7efccf000640 1 Processor -- start 2024-02-21T23:05:40.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.042+0000 7efccf000640 1 -- start start 2024-02-21T23:05:40.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.044+0000 7efccf000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7efcc80c6220 0x7efcc81586b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:40.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.044+0000 7efccf000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efcc8158bf0 0x7efcc815f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:40.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.044+0000 7efccf000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcc80c20d0 con 0x7efcc80c4960 2024-02-21T23:05:40.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.044+0000 7efccf000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcc80a62e0 con 0x7efcc80c6220 2024-02-21T23:05:40.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.044+0000 7efccf000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcc80c1960 con 0x7efcc8158bf0 2024-02-21T23:05:40.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.045+0000 7efcc7400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7efcc80c6220 0x7efcc81586b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:40.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.045+0000 7efccd000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efcc8158bf0 0x7efcc815f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:40.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.045+0000 7efcc7400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7efcc80c6220 0x7efcc81586b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.28:32934/0 (socket says 172.21.15.28:32934) 2024-02-21T23:05:40.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.045+0000 7efcc7400640 1 -- 172.21.15.28:0/3673539307 learned_addr learned my addr 172.21.15.28:0/3673539307 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:40.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.045+0000 7efcc7e00640 1 --1- 172.21.15.28:0/3673539307 >> v1:172.21.15.81:6789/0 conn(0x7efcc80c4960 0x7efcc81524f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.28:58380/0 (socket says 172.21.15.28:58380) 2024-02-21T23:05:40.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.046+0000 7efccd000640 1 -- 172.21.15.28:0/3673539307 >> v2:172.21.15.81:3300/0 conn(0x7efcc80c6220 msgr2=0x7efcc81586b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:40.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.046+0000 7efccd000640 1 --2- 172.21.15.28:0/3673539307 >> v2:172.21.15.81:3300/0 conn(0x7efcc80c6220 0x7efcc81586b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:40.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.046+0000 7efccd000640 1 -- 172.21.15.28:0/3673539307 >> v1:172.21.15.81:6789/0 conn(0x7efcc80c4960 legacy=0x7efcc81524f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:40.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.046+0000 7efccd000640 1 -- 172.21.15.28:0/3673539307 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcc80c1290 con 0x7efcc8158bf0 2024-02-21T23:05:40.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.047+0000 7efccd000640 1 --2- 172.21.15.28:0/3673539307 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efcc8158bf0 0x7efcc815f9e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7efcc0009d90 tx=0x7efcc002be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=786a68e94275a10 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:40.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.048+0000 7efcc6a00640 1 -- 172.21.15.28:0/3673539307 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcc0039070 con 0x7efcc8158bf0 2024-02-21T23:05:40.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.048+0000 7efcc6a00640 1 -- 172.21.15.28:0/3673539307 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcc002cbc0 con 0x7efcc8158bf0 2024-02-21T23:05:40.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.048+0000 7efcc6a00640 1 -- 172.21.15.28:0/3673539307 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcc0034440 con 0x7efcc8158bf0 2024-02-21T23:05:40.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.048+0000 7efccf000640 1 -- 172.21.15.28:0/3673539307 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efcc8158bf0 msgr2=0x7efcc815f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:40.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.048+0000 7efccf000640 1 --2- 172.21.15.28:0/3673539307 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efcc8158bf0 0x7efcc815f9e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7efcc0009d90 tx=0x7efcc002be10 comp rx=0 tx=0).stop 2024-02-21T23:05:40.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.048+0000 7efccf000640 1 -- 172.21.15.28:0/3673539307 shutdown_connections 2024-02-21T23:05:40.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.049+0000 7efccf000640 1 --2- 172.21.15.28:0/3673539307 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efcc8158bf0 0x7efcc815f9e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:40.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.049+0000 7efccf000640 1 --2- 172.21.15.28:0/3673539307 >> v2:172.21.15.81:3300/0 conn(0x7efcc80c6220 0x7efcc81586b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:40.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.049+0000 7efccf000640 1 -- 172.21.15.28:0/3673539307 >> 172.21.15.28:0/3673539307 conn(0x7efcc80a85a0 msgr2=0x7efcc80a8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:40.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.049+0000 7efccf000640 1 -- 172.21.15.28:0/3673539307 shutdown_connections 2024-02-21T23:05:40.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.049+0000 7efccf000640 1 -- 172.21.15.28:0/3673539307 wait complete. 2024-02-21T23:05:40.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.051+0000 7efccf000640 1 Processor -- start 2024-02-21T23:05:40.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.051+0000 7efccf000640 1 -- start start 2024-02-21T23:05:40.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.052+0000 7efccf000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efcc80c4960 0x7efcc8150fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:40.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.052+0000 7efccf000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efcc80c6220 0x7efcc81558d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:40.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.052+0000 7efccf000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcc8158bf0 0x7efcc814baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:40.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.052+0000 7efccf000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcc80a5f20 con 0x7efcc8158bf0 2024-02-21T23:05:40.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.052+0000 7efccf000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcc80c1bb0 con 0x7efcc80c6220 2024-02-21T23:05:40.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.052+0000 7efccf000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcc80a60a0 con 0x7efcc80c4960 2024-02-21T23:05:40.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.052+0000 7efccd000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcc8158bf0 0x7efcc814baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:40.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.052+0000 7efccd000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcc8158bf0 0x7efcc814baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50604/0 (socket says 172.21.15.28:50604) 2024-02-21T23:05:40.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.052+0000 7efccd000640 1 -- 172.21.15.28:0/2422838614 learned_addr learned my addr 172.21.15.28:0/2422838614 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:40.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.053+0000 7efcc7e00640 1 --2- 172.21.15.28:0/2422838614 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efcc80c4960 0x7efcc8150fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:40.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.053+0000 7efcc7400640 1 --2- 172.21.15.28:0/2422838614 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efcc80c6220 0x7efcc81558d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:40.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.053+0000 7efccd000640 1 -- 172.21.15.28:0/2422838614 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efcc80c4960 msgr2=0x7efcc8150fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:40.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.053+0000 7efccd000640 1 --2- 172.21.15.28:0/2422838614 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efcc80c4960 0x7efcc8150fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:40.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.053+0000 7efccd000640 1 -- 172.21.15.28:0/2422838614 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efcc80c6220 msgr2=0x7efcc81558d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:40.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.053+0000 7efccd000640 1 --2- 172.21.15.28:0/2422838614 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efcc80c6220 0x7efcc81558d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:40.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.053+0000 7efccd000640 1 -- 172.21.15.28:0/2422838614 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcc814e7f0 con 0x7efcc8158bf0 2024-02-21T23:05:40.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.053+0000 7efccd000640 1 --2- 172.21.15.28:0/2422838614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcc8158bf0 0x7efcc814baf0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7efcc0009d90 tx=0x7efcc0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:40.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.053+0000 7efcbfe00640 1 -- 172.21.15.28:0/2422838614 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcc0039050 con 0x7efcc8158bf0 2024-02-21T23:05:40.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.053+0000 7efcbfe00640 1 -- 172.21.15.28:0/2422838614 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcc0045070 con 0x7efcc8158bf0 2024-02-21T23:05:40.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.054+0000 7efccf000640 1 -- 172.21.15.28:0/2422838614 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcc81564b0 con 0x7efcc8158bf0 2024-02-21T23:05:40.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.054+0000 7efcbfe00640 1 -- 172.21.15.28:0/2422838614 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcc0009450 con 0x7efcc8158bf0 2024-02-21T23:05:40.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.054+0000 7efccf000640 1 -- 172.21.15.28:0/2422838614 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcc8156a70 con 0x7efcc8158bf0 2024-02-21T23:05:40.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.055+0000 7efccf000640 1 -- 172.21.15.28:0/2422838614 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcc8067f30 con 0x7efcc8158bf0 2024-02-21T23:05:40.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.056+0000 7efcbfe00640 1 -- 172.21.15.28:0/2422838614 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efcc003e850 con 0x7efcc8158bf0 2024-02-21T23:05:40.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.057+0000 7efcbfe00640 1 --2- 172.21.15.28:0/2422838614 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7efca806b4d0 0x7efca806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:40.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.057+0000 7efcbfe00640 1 -- 172.21.15.28:0/2422838614 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5679+0+0 (secure 0 0 0) 0x7efcc00acfa0 con 0x7efcc8158bf0 2024-02-21T23:05:40.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.057+0000 7efcc7e00640 1 --2- 172.21.15.28:0/2422838614 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7efca806b4d0 0x7efca806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:40.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.058+0000 7efcc7e00640 1 --2- 172.21.15.28:0/2422838614 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7efca806b4d0 0x7efca806d990 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7efcc8158ba0 tx=0x7efcb8007ce0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:40.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.062+0000 7efcbfe00640 1 -- 172.21.15.28:0/2422838614 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7efcc814e7f0 con 0x7efcc8158bf0 2024-02-21T23:05:40.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.264+0000 7efccf000640 1 -- 172.21.15.28:0/2422838614 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.c", "name": "mds_join_fs"} v 0) v1 -- 0x7efcc8073330 con 0x7efcc8158bf0 2024-02-21T23:05:40.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.271+0000 7efcbfe00640 1 -- 172.21.15.28:0/2422838614 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.c", "name": "mds_join_fs"}]=0 v21) v1 ==== 96+0+0 (secure 0 0 0) 0x7efcc007bf50 con 0x7efcc8158bf0 2024-02-21T23:05:40.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.274+0000 7efccf000640 1 -- 172.21.15.28:0/2422838614 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7efca806b4d0 msgr2=0x7efca806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:40.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.274+0000 7efccf000640 1 --2- 172.21.15.28:0/2422838614 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7efca806b4d0 0x7efca806d990 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7efcc8158ba0 tx=0x7efcb8007ce0 comp rx=0 tx=0).stop 2024-02-21T23:05:40.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.274+0000 7efccf000640 1 -- 172.21.15.28:0/2422838614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcc8158bf0 msgr2=0x7efcc814baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:40.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.274+0000 7efccf000640 1 --2- 172.21.15.28:0/2422838614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcc8158bf0 0x7efcc814baf0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7efcc0009d90 tx=0x7efcc0004a40 comp rx=0 tx=0).stop 2024-02-21T23:05:40.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.274+0000 7efccf000640 1 -- 172.21.15.28:0/2422838614 shutdown_connections 2024-02-21T23:05:40.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.274+0000 7efccf000640 1 --2- 172.21.15.28:0/2422838614 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7efca806b4d0 0x7efca806d990 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:40.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.274+0000 7efccf000640 1 --2- 172.21.15.28:0/2422838614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcc8158bf0 0x7efcc814baf0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:40.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.274+0000 7efccf000640 1 --2- 172.21.15.28:0/2422838614 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efcc80c6220 0x7efcc81558d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:40.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.274+0000 7efccf000640 1 --2- 172.21.15.28:0/2422838614 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efcc80c4960 0x7efcc8150fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:40.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.274+0000 7efccf000640 1 -- 172.21.15.28:0/2422838614 >> 172.21.15.28:0/2422838614 conn(0x7efcc80a85a0 msgr2=0x7efcc81522c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:40.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.274+0000 7efccf000640 1 -- 172.21.15.28:0/2422838614 shutdown_connections 2024-02-21T23:05:40.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.274+0000 7efccf000640 1 -- 172.21.15.28:0/2422838614 wait complete. 2024-02-21T23:05:40.284 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-02-21T23:05:40.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.513+0000 7f4925200640 1 Processor -- start 2024-02-21T23:05:40.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.513+0000 7f4925200640 1 -- start start 2024-02-21T23:05:40.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.514+0000 7f4925200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f49200a7180 0x7f4920151dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:40.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.514+0000 7f4925200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49200c2960 con 0x7f4920152300 2024-02-21T23:05:40.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.514+0000 7f4925200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49200c2680 con 0x7f492014f5f0 2024-02-21T23:05:40.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.514+0000 7f4925200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f492006e7f0 con 0x7f49200a7180 2024-02-21T23:05:40.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.515+0000 7f491e800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f4920152300 0x7f492015c680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:45372/0 (socket says 172.21.15.28:45372) 2024-02-21T23:05:40.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.515+0000 7f491e800640 1 -- 172.21.15.28:0/3775370677 learned_addr learned my addr 172.21.15.28:0/3775370677 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:05:40.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.515+0000 7f491d400640 1 --2- 172.21.15.28:0/3775370677 >> v2:172.21.15.81:3300/0 conn(0x7f49200a7180 0x7f4920151dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:40.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.515+0000 7f491ca00640 1 -- 172.21.15.28:0/3775370677 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2007221224 0 0) 0x7f49200c2960 con 0x7f4920152300 2024-02-21T23:05:40.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.515+0000 7f491ca00640 1 -- 172.21.15.28:0/3775370677 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4904003520 con 0x7f4920152300 2024-02-21T23:05:40.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.515+0000 7f491ca00640 1 -- 172.21.15.28:0/3775370677 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2472343787 0 0) 0x7f4904003520 con 0x7f4920152300 2024-02-21T23:05:40.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.516+0000 7f491ca00640 1 -- 172.21.15.28:0/3775370677 >> v2:172.21.15.81:3300/0 conn(0x7f49200a7180 msgr2=0x7f4920151dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:40.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.516+0000 7f491ca00640 1 --2- 172.21.15.28:0/3775370677 >> v2:172.21.15.81:3300/0 conn(0x7f49200a7180 0x7f4920151dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:40.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.516+0000 7f491ca00640 1 -- 172.21.15.28:0/3775370677 >> v1:172.21.15.81:6789/0 conn(0x7f492014f5f0 legacy=0x7f4920072860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:40.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.516+0000 7f491ca00640 1 -- 172.21.15.28:0/3775370677 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4920155440 con 0x7f4920152300 2024-02-21T23:05:40.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.516+0000 7f491ca00640 1 -- 172.21.15.28:0/3775370677 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f49180019f0 con 0x7f4920152300 2024-02-21T23:05:40.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.516+0000 7f491ca00640 1 -- 172.21.15.28:0/3775370677 >> v1:172.21.15.28:6789/0 conn(0x7f4920152300 legacy=0x7f492015c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:40.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.516+0000 7f491ca00640 1 --2- 172.21.15.28:0/3775370677 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4904003a00 0x7f4904003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:40.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.516+0000 7f491ca00640 1 --2- 172.21.15.28:0/3775370677 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4904004800 0x7f4904004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:40.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.516+0000 7f491ca00640 1 --2- 172.21.15.28:0/3775370677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4904005210 0x7f490400dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:40.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.516+0000 7f491ca00640 1 -- 172.21.15.28:0/3775370677 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f492006e7f0 con 0x7f4904005210 2024-02-21T23:05:40.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.516+0000 7f491ca00640 1 -- 172.21.15.28:0/3775370677 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f490400e350 con 0x7f4904004800 2024-02-21T23:05:40.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.516+0000 7f491ca00640 1 -- 172.21.15.28:0/3775370677 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f490400e550 con 0x7f4904003a00 2024-02-21T23:05:40.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.516+0000 7f491e800640 1 --2- 172.21.15.28:0/3775370677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4904005210 0x7f490400dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:40.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.516+0000 7f491d400640 1 --2- 172.21.15.28:0/3775370677 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4904004800 0x7f4904004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:40.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.516+0000 7f491de00640 1 --2- 172.21.15.28:0/3775370677 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4904003a00 0x7f4904003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:40.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.516+0000 7f491e800640 1 -- 172.21.15.28:0/3775370677 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4904003a00 msgr2=0x7f4904003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:40.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.516+0000 7f491e800640 1 --2- 172.21.15.28:0/3775370677 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4904003a00 0x7f4904003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:40.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.517+0000 7f491e800640 1 -- 172.21.15.28:0/3775370677 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4904004800 msgr2=0x7f4904004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:40.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.517+0000 7f491e800640 1 --2- 172.21.15.28:0/3775370677 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4904004800 0x7f4904004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:40.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.517+0000 7f491e800640 1 -- 172.21.15.28:0/3775370677 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4904003520 con 0x7f4904005210 2024-02-21T23:05:40.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.517+0000 7f491de00640 1 --2- 172.21.15.28:0/3775370677 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4904003a00 0x7f4904003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:05:40.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.517+0000 7f491e800640 1 -- 172.21.15.28:0/3775370677 reap_dead start 2024-02-21T23:05:40.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.518+0000 7f491e800640 1 --2- 172.21.15.28:0/3775370677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4904005210 0x7f490400dbd0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f491800a7a0 tx=0x7f4904005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15b528be7a379a7c server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:40.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.518+0000 7f491ca00640 1 -- 172.21.15.28:0/3775370677 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f491800a3f0 con 0x7f4904005210 2024-02-21T23:05:40.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.518+0000 7f491ca00640 1 -- 172.21.15.28:0/3775370677 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4918034650 con 0x7f4904005210 2024-02-21T23:05:40.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.518+0000 7f491ca00640 1 -- 172.21.15.28:0/3775370677 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4918034a40 con 0x7f4904005210 2024-02-21T23:05:40.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.518+0000 7f4925200640 1 -- 172.21.15.28:0/3775370677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4904005210 msgr2=0x7f490400dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:40.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.518+0000 7f4925200640 1 --2- 172.21.15.28:0/3775370677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4904005210 0x7f490400dbd0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f491800a7a0 tx=0x7f4904005830 comp rx=0 tx=0).stop 2024-02-21T23:05:40.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.519+0000 7f4925200640 1 -- 172.21.15.28:0/3775370677 shutdown_connections 2024-02-21T23:05:40.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.519+0000 7f4925200640 1 --2- 172.21.15.28:0/3775370677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4904005210 0x7f490400dbd0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:40.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.519+0000 7f4925200640 1 -- 172.21.15.28:0/3775370677 >> 172.21.15.28:0/3775370677 conn(0x7f49200a9870 msgr2=0x7f49200a9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:40.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.519+0000 7f4925200640 1 -- 172.21.15.28:0/3775370677 shutdown_connections 2024-02-21T23:05:40.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.519+0000 7f4925200640 1 -- 172.21.15.28:0/3775370677 wait complete. 2024-02-21T23:05:40.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.520+0000 7f4925200640 1 Processor -- start 2024-02-21T23:05:40.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.521+0000 7f4925200640 1 -- start start 2024-02-21T23:05:40.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.521+0000 7f4925200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4904005210 0x7f4920154d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:40.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.521+0000 7f4925200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f492006c080 0x7f492006c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:40.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.521+0000 7f4925200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f49201eaac0 0x7f49201eaf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:40.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.521+0000 7f4925200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49200c22e0 con 0x7f492006c080 2024-02-21T23:05:40.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.521+0000 7f4925200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4920155a00 con 0x7f4904005210 2024-02-21T23:05:40.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.521+0000 7f4925200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49200c2460 con 0x7f49201eaac0 2024-02-21T23:05:40.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.522+0000 7f491d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f492006c080 0x7f492006c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:40.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.522+0000 7f491d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f492006c080 0x7f492006c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50622/0 (socket says 172.21.15.28:50622) 2024-02-21T23:05:40.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.522+0000 7f491d400640 1 -- 172.21.15.28:0/2623776650 learned_addr learned my addr 172.21.15.28:0/2623776650 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:40.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.522+0000 7f491e800640 1 --2- 172.21.15.28:0/2623776650 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f49201eaac0 0x7f49201eaf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:40.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.522+0000 7f491de00640 1 --2- 172.21.15.28:0/2623776650 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4904005210 0x7f4920154d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:40.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.522+0000 7f491d400640 1 -- 172.21.15.28:0/2623776650 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f49201eaac0 msgr2=0x7f49201eaf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:40.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.522+0000 7f491d400640 1 --2- 172.21.15.28:0/2623776650 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f49201eaac0 0x7f49201eaf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:40.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.522+0000 7f491d400640 1 -- 172.21.15.28:0/2623776650 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4904005210 msgr2=0x7f4920154d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:40.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.522+0000 7f491d400640 1 --2- 172.21.15.28:0/2623776650 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4904005210 0x7f4920154d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:40.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.522+0000 7f491d400640 1 -- 172.21.15.28:0/2623776650 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49201eb750 con 0x7f492006c080 2024-02-21T23:05:40.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.522+0000 7f491de00640 1 --2- 172.21.15.28:0/2623776650 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4904005210 0x7f4920154d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:05:40.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.523+0000 7f491d400640 1 --2- 172.21.15.28:0/2623776650 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f492006c080 0x7f492006c500 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f490c0115c0 tx=0x7f490c011a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:40.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.523+0000 7f4916000640 1 -- 172.21.15.28:0/2623776650 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f490c004220 con 0x7f492006c080 2024-02-21T23:05:40.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.523+0000 7f4916000640 1 -- 172.21.15.28:0/2623776650 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f490c0043c0 con 0x7f492006c080 2024-02-21T23:05:40.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.523+0000 7f4916000640 1 -- 172.21.15.28:0/2623776650 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f490c015930 con 0x7f492006c080 2024-02-21T23:05:40.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.523+0000 7f4925200640 1 -- 172.21.15.28:0/2623776650 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49201eb490 con 0x7f492006c080 2024-02-21T23:05:40.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.523+0000 7f4925200640 1 -- 172.21.15.28:0/2623776650 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49201eec70 con 0x7f492006c080 2024-02-21T23:05:40.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.525+0000 7f4925200640 1 -- 172.21.15.28:0/2623776650 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4920068640 con 0x7f492006c080 2024-02-21T23:05:40.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.525+0000 7f4916000640 1 -- 172.21.15.28:0/2623776650 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f490c015ad0 con 0x7f492006c080 2024-02-21T23:05:40.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.526+0000 7f4916000640 1 --2- 172.21.15.28:0/2623776650 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f490006b520 0x7f490006d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:40.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.526+0000 7f4916000640 1 -- 172.21.15.28:0/2623776650 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5679+0+0 (secure 0 0 0) 0x7f490c08a340 con 0x7f492006c080 2024-02-21T23:05:40.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.527+0000 7f491de00640 1 --2- 172.21.15.28:0/2623776650 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f490006b520 0x7f490006d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:40.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.528+0000 7f491de00640 1 --2- 172.21.15.28:0/2623776650 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f490006b520 0x7f490006d9e0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f4910007fd0 tx=0x7f4910009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:40.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.531+0000 7f4916000640 1 -- 172.21.15.28:0/2623776650 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f490c019030 con 0x7f492006c080 2024-02-21T23:05:40.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.736+0000 7f4925200640 1 -- 172.21.15.28:0/2623776650 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f49200c62a0 con 0x7f492006c080 2024-02-21T23:05:40.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.741+0000 7f4916000640 1 -- 172.21.15.28:0/2623776650 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v22) v1 ==== 104+0+0 (secure 0 0 0) 0x7f490c059250 con 0x7f492006c080 2024-02-21T23:05:40.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.744+0000 7f4925200640 1 -- 172.21.15.28:0/2623776650 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f490006b520 msgr2=0x7f490006d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:40.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.744+0000 7f4925200640 1 --2- 172.21.15.28:0/2623776650 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f490006b520 0x7f490006d9e0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f4910007fd0 tx=0x7f4910009040 comp rx=0 tx=0).stop 2024-02-21T23:05:40.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.744+0000 7f4925200640 1 -- 172.21.15.28:0/2623776650 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f492006c080 msgr2=0x7f492006c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:40.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.744+0000 7f4925200640 1 --2- 172.21.15.28:0/2623776650 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f492006c080 0x7f492006c500 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f490c0115c0 tx=0x7f490c011a80 comp rx=0 tx=0).stop 2024-02-21T23:05:40.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.744+0000 7f4925200640 1 -- 172.21.15.28:0/2623776650 shutdown_connections 2024-02-21T23:05:40.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.744+0000 7f4925200640 1 --2- 172.21.15.28:0/2623776650 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f490006b520 0x7f490006d9e0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:40.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.744+0000 7f4925200640 1 --2- 172.21.15.28:0/2623776650 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f49201eaac0 0x7f49201eaf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:40.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.744+0000 7f4925200640 1 --2- 172.21.15.28:0/2623776650 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f492006c080 0x7f492006c500 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:40.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.744+0000 7f4925200640 1 --2- 172.21.15.28:0/2623776650 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4904005210 0x7f4920154d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:40.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.744+0000 7f4925200640 1 -- 172.21.15.28:0/2623776650 >> 172.21.15.28:0/2623776650 conn(0x7f49200a9870 msgr2=0x7f49200726a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:40.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.744+0000 7f4925200640 1 -- 172.21.15.28:0/2623776650 shutdown_connections 2024-02-21T23:05:40.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.744+0000 7f4925200640 1 -- 172.21.15.28:0/2623776650 wait complete. 2024-02-21T23:05:40.752 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_last_resort' 2024-02-21T23:05:40.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.984+0000 7f2caec00640 1 Processor -- start 2024-02-21T23:05:40.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.984+0000 7f2caec00640 1 -- start start 2024-02-21T23:05:40.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.985+0000 7f2caec00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2ca8070a60 0x7f2ca80c4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:40.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.985+0000 7f2caec00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ca8070f90 con 0x7f2ca81502f0 2024-02-21T23:05:40.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.985+0000 7f2caec00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ca8071270 con 0x7f2ca8156bc0 2024-02-21T23:05:40.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.985+0000 7f2caec00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca806ff30 con 0x7f2ca8070a60 2024-02-21T23:05:40.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.986+0000 7f2ca7e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f2ca81502f0 0x7f2ca8153850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:45388/0 (socket says 172.21.15.28:45388) 2024-02-21T23:05:40.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.986+0000 7f2ca7e00640 1 -- 172.21.15.28:0/1637443129 learned_addr learned my addr 172.21.15.28:0/1637443129 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:05:40.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.986+0000 7f2ca7400640 1 --2- 172.21.15.28:0/1637443129 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2ca8070a60 0x7f2ca80c4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:40.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.987+0000 7f2ca6a00640 1 -- 172.21.15.28:0/1637443129 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2297104439 0 0) 0x7f2ca8070f90 con 0x7f2ca81502f0 2024-02-21T23:05:40.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.987+0000 7f2ca6a00640 1 -- 172.21.15.28:0/1637443129 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c8c003520 con 0x7f2ca81502f0 2024-02-21T23:05:40.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.987+0000 7f2ca6a00640 1 -- 172.21.15.28:0/1637443129 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1731614018 0 0) 0x7f2c8c003520 con 0x7f2ca81502f0 2024-02-21T23:05:40.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.987+0000 7f2ca6a00640 1 -- 172.21.15.28:0/1637443129 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2ca8070a60 msgr2=0x7f2ca80c4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:40.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.987+0000 7f2ca6a00640 1 --2- 172.21.15.28:0/1637443129 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2ca8070a60 0x7f2ca80c4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:40.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.987+0000 7f2ca6a00640 1 -- 172.21.15.28:0/1637443129 >> v1:172.21.15.81:6789/0 conn(0x7f2ca8156bc0 legacy=0x7f2ca80c7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:40.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.987+0000 7f2ca6a00640 1 -- 172.21.15.28:0/1637443129 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ca8071550 con 0x7f2ca81502f0 2024-02-21T23:05:40.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.987+0000 7f2ca6a00640 1 -- 172.21.15.28:0/1637443129 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f2c98001670 con 0x7f2ca81502f0 2024-02-21T23:05:40.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.987+0000 7f2ca6a00640 1 -- 172.21.15.28:0/1637443129 >> v1:172.21.15.28:6789/0 conn(0x7f2ca81502f0 legacy=0x7f2ca8153850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:40.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.987+0000 7f2ca6a00640 1 --2- 172.21.15.28:0/1637443129 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2ca8070a60 0x7f2c8c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:40.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.987+0000 7f2ca6a00640 1 --2- 172.21.15.28:0/1637443129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c8c003f20 0x7f2c8c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:40.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.987+0000 7f2ca6a00640 1 --2- 172.21.15.28:0/1637443129 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2c8c004d40 0x7f2c8c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:40.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.987+0000 7f2ca6a00640 1 -- 172.21.15.28:0/1637443129 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c8c00ba70 con 0x7f2c8c003f20 2024-02-21T23:05:40.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.987+0000 7f2ca6a00640 1 -- 172.21.15.28:0/1637443129 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c8c00bbf0 con 0x7f2c8c004d40 2024-02-21T23:05:40.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.987+0000 7f2ca6a00640 1 -- 172.21.15.28:0/1637443129 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c8c00bdf0 con 0x7f2ca8070a60 2024-02-21T23:05:40.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.987+0000 7f2ca7400640 1 --2- 172.21.15.28:0/1637443129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c8c003f20 0x7f2c8c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:40.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.988+0000 7f2ca7e00640 1 --2- 172.21.15.28:0/1637443129 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2ca8070a60 0x7f2c8c0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:40.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.988+0000 7f2cacc00640 1 --2- 172.21.15.28:0/1637443129 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2c8c004d40 0x7f2c8c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:40.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.988+0000 7f2ca7400640 1 -- 172.21.15.28:0/1637443129 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2ca8070a60 msgr2=0x7f2c8c0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:40.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.988+0000 7f2ca7400640 1 --2- 172.21.15.28:0/1637443129 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2ca8070a60 0x7f2c8c0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:40.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.988+0000 7f2ca7400640 1 -- 172.21.15.28:0/1637443129 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2c8c004d40 msgr2=0x7f2c8c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:40.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.988+0000 7f2ca7400640 1 --2- 172.21.15.28:0/1637443129 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2c8c004d40 0x7f2c8c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:40.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.988+0000 7f2ca7400640 1 -- 172.21.15.28:0/1637443129 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c8c003520 con 0x7f2c8c003f20 2024-02-21T23:05:40.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.988+0000 7f2cacc00640 1 --2- 172.21.15.28:0/1637443129 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2c8c004d40 0x7f2c8c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:05:40.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.989+0000 7f2ca7400640 1 --2- 172.21.15.28:0/1637443129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c8c003f20 0x7f2c8c004800 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f2c94009a90 tx=0x7f2c8c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66c8dde3ff9d8195 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:40.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.989+0000 7f2ca6a00640 1 -- 172.21.15.28:0/1637443129 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c94039070 con 0x7f2c8c003f20 2024-02-21T23:05:40.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.989+0000 7f2ca6a00640 1 -- 172.21.15.28:0/1637443129 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c9402c680 con 0x7f2c8c003f20 2024-02-21T23:05:40.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.989+0000 7f2ca6a00640 1 -- 172.21.15.28:0/1637443129 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c94009690 con 0x7f2c8c003f20 2024-02-21T23:05:40.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.990+0000 7f2caec00640 1 -- 172.21.15.28:0/1637443129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c8c003f20 msgr2=0x7f2c8c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:40.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.990+0000 7f2caec00640 1 --2- 172.21.15.28:0/1637443129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c8c003f20 0x7f2c8c004800 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f2c94009a90 tx=0x7f2c8c005830 comp rx=0 tx=0).stop 2024-02-21T23:05:40.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.990+0000 7f2cacc00640 1 -- 172.21.15.28:0/1637443129 reap_dead start 2024-02-21T23:05:40.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.990+0000 7f2caec00640 1 -- 172.21.15.28:0/1637443129 shutdown_connections 2024-02-21T23:05:40.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.990+0000 7f2caec00640 1 -- 172.21.15.28:0/1637443129 >> 172.21.15.28:0/1637443129 conn(0x7f2ca80c8810 msgr2=0x7f2ca80c8bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:40.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.990+0000 7f2caec00640 1 -- 172.21.15.28:0/1637443129 shutdown_connections 2024-02-21T23:05:40.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.990+0000 7f2caec00640 1 -- 172.21.15.28:0/1637443129 wait complete. 2024-02-21T23:05:40.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.992+0000 7f2caec00640 1 Processor -- start 2024-02-21T23:05:40.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.992+0000 7f2caec00640 1 -- start start 2024-02-21T23:05:40.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.993+0000 7f2caec00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2c8c003f20 0x7f2ca81eae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:40.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.993+0000 7f2caec00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ca8155e10 0x7f2ca80c1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:40.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.993+0000 7f2caec00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ca814cbc0 0x7f2ca814d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:40.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.993+0000 7f2caec00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca80c7e70 con 0x7f2ca8155e10 2024-02-21T23:05:40.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.993+0000 7f2caec00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca81566a0 con 0x7f2ca814cbc0 2024-02-21T23:05:40.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.993+0000 7f2caec00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca80c7ff0 con 0x7f2c8c003f20 2024-02-21T23:05:40.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.993+0000 7f2ca7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ca8155e10 0x7f2ca80c1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:40.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.994+0000 7f2ca7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ca8155e10 0x7f2ca80c1eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50646/0 (socket says 172.21.15.28:50646) 2024-02-21T23:05:40.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.994+0000 7f2ca7400640 1 -- 172.21.15.28:0/1414260836 learned_addr learned my addr 172.21.15.28:0/1414260836 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:40.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.994+0000 7f2cacc00640 1 --2- 172.21.15.28:0/1414260836 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ca814cbc0 0x7f2ca814d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:40.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.994+0000 7f2ca7e00640 1 --2- 172.21.15.28:0/1414260836 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2c8c003f20 0x7f2ca81eae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:40.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.994+0000 7f2ca7400640 1 -- 172.21.15.28:0/1414260836 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2c8c003f20 msgr2=0x7f2ca81eae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:40.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.994+0000 7f2ca7400640 1 --2- 172.21.15.28:0/1414260836 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2c8c003f20 0x7f2ca81eae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:40.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.994+0000 7f2ca7400640 1 -- 172.21.15.28:0/1414260836 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ca814cbc0 msgr2=0x7f2ca814d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:40.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.994+0000 7f2ca7400640 1 --2- 172.21.15.28:0/1414260836 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ca814cbc0 0x7f2ca814d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:40.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.994+0000 7f2ca7400640 1 -- 172.21.15.28:0/1414260836 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ca806d840 con 0x7f2ca8155e10 2024-02-21T23:05:40.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.994+0000 7f2cacc00640 1 --2- 172.21.15.28:0/1414260836 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ca814cbc0 0x7f2ca814d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:05:40.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.994+0000 7f2ca7400640 1 --2- 172.21.15.28:0/1414260836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ca8155e10 0x7f2ca80c1eb0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f2c94009bb0 tx=0x7f2c94004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:40.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.995+0000 7f2c9fe00640 1 -- 172.21.15.28:0/1414260836 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c94042020 con 0x7f2ca8155e10 2024-02-21T23:05:40.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.995+0000 7f2c9fe00640 1 -- 172.21.15.28:0/1414260836 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c9403a040 con 0x7f2ca8155e10 2024-02-21T23:05:40.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.995+0000 7f2c9fe00640 1 -- 172.21.15.28:0/1414260836 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c94039070 con 0x7f2ca8155e10 2024-02-21T23:05:40.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.995+0000 7f2caec00640 1 -- 172.21.15.28:0/1414260836 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ca806db20 con 0x7f2ca8155e10 2024-02-21T23:05:40.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.995+0000 7f2caec00640 1 -- 172.21.15.28:0/1414260836 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ca81de790 con 0x7f2ca8155e10 2024-02-21T23:05:40.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.996+0000 7f2caec00640 1 -- 172.21.15.28:0/1414260836 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ca8067f30 con 0x7f2ca8155e10 2024-02-21T23:05:40.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.997+0000 7f2c9fe00640 1 -- 172.21.15.28:0/1414260836 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2c94045020 con 0x7f2ca8155e10 2024-02-21T23:05:40.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.998+0000 7f2c9fe00640 1 --2- 172.21.15.28:0/1414260836 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2c8806b4d0 0x7f2c8806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:40.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.998+0000 7f2c9fe00640 1 -- 172.21.15.28:0/1414260836 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5679+0+0 (secure 0 0 0) 0x7f2c940ab690 con 0x7f2ca8155e10 2024-02-21T23:05:40.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:40.999+0000 7f2ca7e00640 1 --2- 172.21.15.28:0/1414260836 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2c8806b4d0 0x7f2c8806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:40.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:41.000+0000 7f2ca7e00640 1 --2- 172.21.15.28:0/1414260836 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2c8806b4d0 0x7f2c8806d990 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f2c98001050 tx=0x7f2c98009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:40.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:41.003+0000 7f2c9fe00640 1 -- 172.21.15.28:0/1414260836 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f2c9404c070 con 0x7f2ca8155e10 2024-02-21T23:05:41.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:41.281+0000 7f2caec00640 1 -- 172.21.15.28:0/1414260836 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_last_resort"]} v 0) v1 -- 0x7f2ca814f7a0 con 0x7f2ca8155e10 2024-02-21T23:05:42.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.274+0000 7f2c9fe00640 1 -- 172.21.15.28:0/1414260836 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_last_resort"]}]=0 v193) v1 ==== 150+0+0 (secure 0 0 0) 0x7f2c9407a520 con 0x7f2ca8155e10 2024-02-21T23:05:42.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.278+0000 7f2caec00640 1 -- 172.21.15.28:0/1414260836 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2c8806b4d0 msgr2=0x7f2c8806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:42.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.278+0000 7f2caec00640 1 --2- 172.21.15.28:0/1414260836 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2c8806b4d0 0x7f2c8806d990 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f2c98001050 tx=0x7f2c98009040 comp rx=0 tx=0).stop 2024-02-21T23:05:42.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.278+0000 7f2caec00640 1 -- 172.21.15.28:0/1414260836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ca8155e10 msgr2=0x7f2ca80c1eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:42.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.278+0000 7f2caec00640 1 --2- 172.21.15.28:0/1414260836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ca8155e10 0x7f2ca80c1eb0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f2c94009bb0 tx=0x7f2c94004730 comp rx=0 tx=0).stop 2024-02-21T23:05:42.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.278+0000 7f2caec00640 1 -- 172.21.15.28:0/1414260836 shutdown_connections 2024-02-21T23:05:42.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.278+0000 7f2caec00640 1 --2- 172.21.15.28:0/1414260836 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2c8806b4d0 0x7f2c8806d990 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:42.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.278+0000 7f2caec00640 1 --2- 172.21.15.28:0/1414260836 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ca814cbc0 0x7f2ca814d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:42.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.278+0000 7f2caec00640 1 --2- 172.21.15.28:0/1414260836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ca8155e10 0x7f2ca80c1eb0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:42.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.278+0000 7f2caec00640 1 --2- 172.21.15.28:0/1414260836 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2c8c003f20 0x7f2ca81eae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:42.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.278+0000 7f2caec00640 1 -- 172.21.15.28:0/1414260836 >> 172.21.15.28:0/1414260836 conn(0x7f2ca80c8810 msgr2=0x7f2ca80a9b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:42.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.278+0000 7f2caec00640 1 -- 172.21.15.28:0/1414260836 shutdown_connections 2024-02-21T23:05:42.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.278+0000 7f2caec00640 1 -- 172.21.15.28:0/1414260836 wait complete. 2024-02-21T23:05:42.289 INFO:tasks.cephfs_test_runner:That a standby with mds_join_fs set to another fs is still used if necessary. ... ok 2024-02-21T23:05:42.289 INFO:tasks.cephfs_test_runner:Starting test: test_join_fs_last_resort_refused (tasks.cephfs.test_failover.TestClusterAffinity) That a standby with mds_join_fs set to another fs is not used if refuse_standby_for_another_fs is set. 2024-02-21T23:05:42.289 INFO:tasks.cephfs_test_runner:test_join_fs_last_resort_refused (tasks.cephfs.test_failover.TestClusterAffinity) 2024-02-21T23:05:42.290 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:05:42.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.519+0000 7ff626400640 1 Processor -- start 2024-02-21T23:05:42.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.520+0000 7ff626400640 1 -- start start 2024-02-21T23:05:42.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.521+0000 7ff626400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff6200ca050 0x7ff6201558c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:42.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.521+0000 7ff626400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6201528e0 con 0x7ff620152500 2024-02-21T23:05:42.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.521+0000 7ff626400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6200c86e0 con 0x7ff62006bbf0 2024-02-21T23:05:42.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.521+0000 7ff626400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6200c8980 con 0x7ff6200ca050 2024-02-21T23:05:42.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.522+0000 7ff61fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff6200ca050 0x7ff6201558c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:42.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.522+0000 7ff625000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff620152500 0x7ff6200bf6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:45404/0 (socket says 172.21.15.28:45404) 2024-02-21T23:05:42.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.522+0000 7ff625000640 1 -- 172.21.15.28:0/2035562692 learned_addr learned my addr 172.21.15.28:0/2035562692 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:05:42.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.522+0000 7ff61fe00640 1 -- 172.21.15.28:0/2035562692 >> v1:172.21.15.81:6789/0 conn(0x7ff62006bbf0 legacy=0x7ff62006bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:42.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.523+0000 7ff61fe00640 1 -- 172.21.15.28:0/2035562692 >> v1:172.21.15.28:6789/0 conn(0x7ff620152500 legacy=0x7ff6200bf6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:42.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.523+0000 7ff61fe00640 1 -- 172.21.15.28:0/2035562692 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6200c8c60 con 0x7ff6200ca050 2024-02-21T23:05:42.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.523+0000 7ff61f400640 1 -- 172.21.15.28:0/2035562692 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1095985275 0 0) 0x7ff6201528e0 con 0x7ff620152500 2024-02-21T23:05:42.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.524+0000 7ff61fe00640 1 --2- 172.21.15.28:0/2035562692 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6200ca050 0x7ff6201558c0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7ff610009d90 tx=0x7ff61002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48213bfedb4fe3a6 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:42.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.524+0000 7ff61f400640 1 -- 172.21.15.28:0/2035562692 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff610039070 con 0x7ff6200ca050 2024-02-21T23:05:42.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.524+0000 7ff61f400640 1 -- 172.21.15.28:0/2035562692 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff61002cc60 con 0x7ff6200ca050 2024-02-21T23:05:42.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.524+0000 7ff61f400640 1 -- 172.21.15.28:0/2035562692 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff610034660 con 0x7ff6200ca050 2024-02-21T23:05:42.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.525+0000 7ff626400640 1 -- 172.21.15.28:0/2035562692 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6200ca050 msgr2=0x7ff6201558c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:42.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.525+0000 7ff626400640 1 --2- 172.21.15.28:0/2035562692 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6200ca050 0x7ff6201558c0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7ff610009d90 tx=0x7ff61002beb0 comp rx=0 tx=0).stop 2024-02-21T23:05:42.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.525+0000 7ff626400640 1 -- 172.21.15.28:0/2035562692 shutdown_connections 2024-02-21T23:05:42.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.525+0000 7ff626400640 1 --2- 172.21.15.28:0/2035562692 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6200ca050 0x7ff6201558c0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:42.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.525+0000 7ff626400640 1 -- 172.21.15.28:0/2035562692 >> 172.21.15.28:0/2035562692 conn(0x7ff6200ac450 msgr2=0x7ff6200ac860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:42.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.525+0000 7ff626400640 1 -- 172.21.15.28:0/2035562692 shutdown_connections 2024-02-21T23:05:42.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.525+0000 7ff626400640 1 -- 172.21.15.28:0/2035562692 wait complete. 2024-02-21T23:05:42.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.527+0000 7ff626400640 1 Processor -- start 2024-02-21T23:05:42.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.527+0000 7ff626400640 1 -- start start 2024-02-21T23:05:42.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.528+0000 7ff626400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff62006bbf0 0x7ff620153fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:42.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.528+0000 7ff626400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6200ca050 0x7ff620147c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:42.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.528+0000 7ff626400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff620152500 0x7ff620146fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:42.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.528+0000 7ff626400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6200bfdd0 con 0x7ff6200ca050 2024-02-21T23:05:42.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.528+0000 7ff626400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6200c9220 con 0x7ff620152500 2024-02-21T23:05:42.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.528+0000 7ff626400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6200bff50 con 0x7ff62006bbf0 2024-02-21T23:05:42.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.528+0000 7ff61fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6200ca050 0x7ff620147c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:42.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.528+0000 7ff61fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6200ca050 0x7ff620147c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50656/0 (socket says 172.21.15.28:50656) 2024-02-21T23:05:42.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.528+0000 7ff61fe00640 1 -- 172.21.15.28:0/436405739 learned_addr learned my addr 172.21.15.28:0/436405739 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:42.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.528+0000 7ff61fe00640 1 -- 172.21.15.28:0/436405739 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff62006bbf0 msgr2=0x7ff620153fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:42.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.528+0000 7ff61fe00640 1 --2- 172.21.15.28:0/436405739 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff62006bbf0 0x7ff620153fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:42.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.528+0000 7ff61fe00640 1 -- 172.21.15.28:0/436405739 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff620152500 msgr2=0x7ff620146fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:42.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.528+0000 7ff61fe00640 1 --2- 172.21.15.28:0/436405739 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff620152500 0x7ff620146fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:42.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.528+0000 7ff61fe00640 1 -- 172.21.15.28:0/436405739 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6200c5f30 con 0x7ff6200ca050 2024-02-21T23:05:42.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.529+0000 7ff61fe00640 1 --2- 172.21.15.28:0/436405739 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6200ca050 0x7ff620147c90 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7ff610009d90 tx=0x7ff610004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:42.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.529+0000 7ff61cc00640 1 -- 172.21.15.28:0/436405739 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff610039050 con 0x7ff6200ca050 2024-02-21T23:05:42.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.529+0000 7ff61cc00640 1 -- 172.21.15.28:0/436405739 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff610045070 con 0x7ff6200ca050 2024-02-21T23:05:42.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.529+0000 7ff61cc00640 1 -- 172.21.15.28:0/436405739 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff61003ea50 con 0x7ff6200ca050 2024-02-21T23:05:42.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.529+0000 7ff626400640 1 -- 172.21.15.28:0/436405739 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6200c6ec0 con 0x7ff6200ca050 2024-02-21T23:05:42.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.529+0000 7ff626400640 1 -- 172.21.15.28:0/436405739 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6200c79e0 con 0x7ff6200ca050 2024-02-21T23:05:42.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.530+0000 7ff626400640 1 -- 172.21.15.28:0/436405739 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff62004f290 con 0x7ff6200ca050 2024-02-21T23:05:42.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.531+0000 7ff61cc00640 1 -- 172.21.15.28:0/436405739 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff61002cb20 con 0x7ff6200ca050 2024-02-21T23:05:42.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.532+0000 7ff61cc00640 1 --2- 172.21.15.28:0/436405739 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff5fc06b4d0 0x7ff5fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:42.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.532+0000 7ff61cc00640 1 -- 172.21.15.28:0/436405739 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5679+0+0 (secure 0 0 0) 0x7ff6100ad030 con 0x7ff6200ca050 2024-02-21T23:05:42.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.533+0000 7ff625000640 1 --2- 172.21.15.28:0/436405739 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff5fc06b4d0 0x7ff5fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:42.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.534+0000 7ff625000640 1 --2- 172.21.15.28:0/436405739 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff5fc06b4d0 0x7ff5fc06d990 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7ff608000c00 tx=0x7ff6080079c0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:42.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.537+0000 7ff61cc00640 1 -- 172.21.15.28:0/436405739 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ff61007bec0 con 0x7ff6200ca050 2024-02-21T23:05:42.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.736+0000 7ff626400640 1 -- 172.21.15.28:0/436405739 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff6200c6410 con 0x7ff6200ca050 2024-02-21T23:05:42.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.738+0000 7ff61cc00640 1 -- 172.21.15.28:0/436405739 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v68) v1 ==== 74+0+13752 (secure 0 0 0) 0x7ff61007b7c0 con 0x7ff6200ca050 2024-02-21T23:05:42.728 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:05:42.728 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":68,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:05:38.858237+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.81:6835/2152160500":"2024-02-22T23:05:35.610770+0000","172.21.15.81:6834/2152160500":"2024-02-22T23:05:35.610770+0000","172.21.15.81:6836/442767245":"2024-02-22T23:05:28.786771+0000","172.21.15.28:6835/3468780363":"2024-02-22T23:05:17.415180+0000","172.21.15.28:6833/3468780363":"2024-02-22T23:05:17.415180+0000","172.21.15.81:6837/442767245":"2024-02-22T23:05:28.786771+0000","172.21.15.28:6834/3835332438":"2024-02-22T23:05:27.771834+0000","172.21.15.28:6832/3835332438":"2024-02-22T23:05:27.771834+0000","172.21.15.28:6834/2803823391":"2024-02-22T23:05:14.385354+0000","172.21.15.28:6832/2803823391":"2024-02-22T23:05:14.385354+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:05:42.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.741+0000 7ff626400640 1 -- 172.21.15.28:0/436405739 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff5fc06b4d0 msgr2=0x7ff5fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:42.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.741+0000 7ff626400640 1 --2- 172.21.15.28:0/436405739 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff5fc06b4d0 0x7ff5fc06d990 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7ff608000c00 tx=0x7ff6080079c0 comp rx=0 tx=0).stop 2024-02-21T23:05:42.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.741+0000 7ff626400640 1 -- 172.21.15.28:0/436405739 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6200ca050 msgr2=0x7ff620147c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:42.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.741+0000 7ff626400640 1 --2- 172.21.15.28:0/436405739 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6200ca050 0x7ff620147c90 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7ff610009d90 tx=0x7ff610004a40 comp rx=0 tx=0).stop 2024-02-21T23:05:42.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.741+0000 7ff626400640 1 -- 172.21.15.28:0/436405739 shutdown_connections 2024-02-21T23:05:42.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.741+0000 7ff626400640 1 --2- 172.21.15.28:0/436405739 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff5fc06b4d0 0x7ff5fc06d990 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:42.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.741+0000 7ff626400640 1 --2- 172.21.15.28:0/436405739 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff620152500 0x7ff620146fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:42.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.741+0000 7ff626400640 1 --2- 172.21.15.28:0/436405739 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6200ca050 0x7ff620147c90 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:42.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.741+0000 7ff626400640 1 --2- 172.21.15.28:0/436405739 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff62006bbf0 0x7ff620153fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:42.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.741+0000 7ff626400640 1 -- 172.21.15.28:0/436405739 >> 172.21.15.28:0/436405739 conn(0x7ff6200ac450 msgr2=0x7ff620150730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:42.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.741+0000 7ff626400640 1 -- 172.21.15.28:0/436405739 shutdown_connections 2024-02-21T23:05:42.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.741+0000 7ff626400640 1 -- 172.21.15.28:0/436405739 wait complete. 2024-02-21T23:05:42.748 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-21T23:02:30.728329+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-21T23:05:42.748 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-21T23:05:42.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.982+0000 7f6b8b200640 1 Processor -- start 2024-02-21T23:05:42.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.983+0000 7f6b8b200640 1 -- start start 2024-02-21T23:05:42.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.984+0000 7f6b8b200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6b8c13e0c0 0x7f6b8c0c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:42.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.984+0000 7f6b8b200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f6b8c0c76f0 0x7f6b8c06c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:42.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.984+0000 7f6b8b200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b8c158550 con 0x7f6b8c06dbc0 2024-02-21T23:05:42.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.984+0000 7f6b8b200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b8c0c89a0 con 0x7f6b8c0c76f0 2024-02-21T23:05:42.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.984+0000 7f6b8b200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b8c1587d0 con 0x7f6b8c13e0c0 2024-02-21T23:05:42.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.985+0000 7f6b89400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f6b8c06dbc0 0x7f6b8c06c280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:45412/0 (socket says 172.21.15.28:45412) 2024-02-21T23:05:42.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.985+0000 7f6b89400640 1 -- 172.21.15.28:0/1006478443 learned_addr learned my addr 172.21.15.28:0/1006478443 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:05:42.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.985+0000 7f6b89e00640 1 --2- 172.21.15.28:0/1006478443 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6b8c13e0c0 0x7f6b8c0c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:42.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.985+0000 7f6b8a800640 1 --2- 172.21.15.28:0/1006478443 >> v2:172.21.15.81:3300/0 conn(0x7f6b8c0c76f0 0x7f6b8c06c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:42.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.985+0000 7f6b88a00640 1 -- 172.21.15.28:0/1006478443 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2507828420 0 0) 0x7f6b74001490 con 0x7f6b8c06dbc0 2024-02-21T23:05:42.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.985+0000 7f6b88a00640 1 -- 172.21.15.28:0/1006478443 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b6c003520 con 0x7f6b8c06dbc0 2024-02-21T23:05:42.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.985+0000 7f6b88a00640 1 -- 172.21.15.28:0/1006478443 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4191631939 0 0) 0x7f6b6c003520 con 0x7f6b8c06dbc0 2024-02-21T23:05:42.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.985+0000 7f6b88a00640 1 -- 172.21.15.28:0/1006478443 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6b8c13e0c0 msgr2=0x7f6b8c0c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:42.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.986+0000 7f6b88a00640 1 --2- 172.21.15.28:0/1006478443 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6b8c13e0c0 0x7f6b8c0c71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:42.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.986+0000 7f6b88a00640 1 -- 172.21.15.28:0/1006478443 >> v2:172.21.15.81:3300/0 conn(0x7f6b8c0c76f0 msgr2=0x7f6b8c06c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:42.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.986+0000 7f6b88a00640 1 --2- 172.21.15.28:0/1006478443 >> v2:172.21.15.81:3300/0 conn(0x7f6b8c0c76f0 0x7f6b8c06c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:42.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.986+0000 7f6b88a00640 1 -- 172.21.15.28:0/1006478443 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b8c158ab0 con 0x7f6b8c06dbc0 2024-02-21T23:05:42.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.986+0000 7f6b88a00640 1 -- 172.21.15.28:0/1006478443 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f6b74001bc0 con 0x7f6b8c06dbc0 2024-02-21T23:05:42.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.986+0000 7f6b88a00640 1 -- 172.21.15.28:0/1006478443 >> v1:172.21.15.28:6789/0 conn(0x7f6b8c06dbc0 legacy=0x7f6b8c06c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:42.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.986+0000 7f6b88a00640 1 --2- 172.21.15.28:0/1006478443 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b6c0039b0 0x7f6b6c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:42.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.986+0000 7f6b88a00640 1 --2- 172.21.15.28:0/1006478443 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6b8c13e0c0 0x7f6b6c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:42.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.986+0000 7f6b88a00640 1 --2- 172.21.15.28:0/1006478443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b6c004d40 0x7f6b6c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:42.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.986+0000 7f6b88a00640 1 -- 172.21.15.28:0/1006478443 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b8c1587d0 con 0x7f6b6c004d40 2024-02-21T23:05:42.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.986+0000 7f6b88a00640 1 -- 172.21.15.28:0/1006478443 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b6c00b9c0 con 0x7f6b6c0039b0 2024-02-21T23:05:42.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.986+0000 7f6b88a00640 1 -- 172.21.15.28:0/1006478443 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b6c00bb70 con 0x7f6b8c13e0c0 2024-02-21T23:05:42.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.986+0000 7f6b8a800640 1 --2- 172.21.15.28:0/1006478443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b6c004d40 0x7f6b6c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:42.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.986+0000 7f6b89e00640 1 --2- 172.21.15.28:0/1006478443 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b6c0039b0 0x7f6b6c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:42.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.986+0000 7f6b89400640 1 --2- 172.21.15.28:0/1006478443 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6b8c13e0c0 0x7f6b6c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:42.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.986+0000 7f6b8a800640 1 -- 172.21.15.28:0/1006478443 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6b8c13e0c0 msgr2=0x7f6b6c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:42.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.986+0000 7f6b8a800640 1 --2- 172.21.15.28:0/1006478443 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6b8c13e0c0 0x7f6b6c004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:42.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.986+0000 7f6b8a800640 1 -- 172.21.15.28:0/1006478443 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b6c0039b0 msgr2=0x7f6b6c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:42.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.986+0000 7f6b8a800640 1 --2- 172.21.15.28:0/1006478443 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b6c0039b0 0x7f6b6c003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:42.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.986+0000 7f6b8a800640 1 -- 172.21.15.28:0/1006478443 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b6c003520 con 0x7f6b6c004d40 2024-02-21T23:05:42.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.988+0000 7f6b8a800640 1 --2- 172.21.15.28:0/1006478443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b6c004d40 0x7f6b6c0051c0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f6b84009ac0 tx=0x7f6b6c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a96b3351d4424507 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:42.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.988+0000 7f6b88a00640 1 -- 172.21.15.28:0/1006478443 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b84039070 con 0x7f6b6c004d40 2024-02-21T23:05:42.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.988+0000 7f6b88a00640 1 -- 172.21.15.28:0/1006478443 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b8402c710 con 0x7f6b6c004d40 2024-02-21T23:05:42.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.988+0000 7f6b88a00640 1 -- 172.21.15.28:0/1006478443 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b84009690 con 0x7f6b6c004d40 2024-02-21T23:05:42.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.988+0000 7f6b8b200640 1 -- 172.21.15.28:0/1006478443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b6c004d40 msgr2=0x7f6b6c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:42.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.988+0000 7f6b8b200640 1 --2- 172.21.15.28:0/1006478443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b6c004d40 0x7f6b6c0051c0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f6b84009ac0 tx=0x7f6b6c005830 comp rx=0 tx=0).stop 2024-02-21T23:05:42.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.988+0000 7f6b8a800640 1 -- 172.21.15.28:0/1006478443 reap_dead start 2024-02-21T23:05:42.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.988+0000 7f6b8b200640 1 -- 172.21.15.28:0/1006478443 shutdown_connections 2024-02-21T23:05:42.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.988+0000 7f6b8b200640 1 -- 172.21.15.28:0/1006478443 >> 172.21.15.28:0/1006478443 conn(0x7f6b8c073110 msgr2=0x7f6b8c0ac490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:42.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.989+0000 7f6b8b200640 1 -- 172.21.15.28:0/1006478443 shutdown_connections 2024-02-21T23:05:42.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.989+0000 7f6b8b200640 1 -- 172.21.15.28:0/1006478443 wait complete. 2024-02-21T23:05:42.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.990+0000 7f6b8b200640 1 Processor -- start 2024-02-21T23:05:42.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.990+0000 7f6b8b200640 1 -- start start 2024-02-21T23:05:42.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.991+0000 7f6b8b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b6c004d40 0x7f6b8c1eb130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:42.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.991+0000 7f6b8b200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b8c1ebb60 0x7f6b8c1575f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:42.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.991+0000 7f6b8b200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6b8c1ee5c0 0x7f6b8c1e7940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:42.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.991+0000 7f6b8b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b8c06ced0 con 0x7f6b6c004d40 2024-02-21T23:05:42.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.991+0000 7f6b8b200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b8c159070 con 0x7f6b8c1ebb60 2024-02-21T23:05:42.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.991+0000 7f6b8b200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b8c13e520 con 0x7f6b8c1ee5c0 2024-02-21T23:05:42.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.992+0000 7f6b89e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b6c004d40 0x7f6b8c1eb130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:42.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.992+0000 7f6b89e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b6c004d40 0x7f6b8c1eb130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50674/0 (socket says 172.21.15.28:50674) 2024-02-21T23:05:42.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.992+0000 7f6b89e00640 1 -- 172.21.15.28:0/1919231577 learned_addr learned my addr 172.21.15.28:0/1919231577 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:42.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.992+0000 7f6b89e00640 1 -- 172.21.15.28:0/1919231577 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6b8c1ee5c0 msgr2=0x7f6b8c1e7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:42.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.992+0000 7f6b89e00640 1 --2- 172.21.15.28:0/1919231577 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6b8c1ee5c0 0x7f6b8c1e7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:42.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.992+0000 7f6b89e00640 1 -- 172.21.15.28:0/1919231577 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b8c1ebb60 msgr2=0x7f6b8c1575f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:42.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.992+0000 7f6b89e00640 1 --2- 172.21.15.28:0/1919231577 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b8c1ebb60 0x7f6b8c1575f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:42.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.992+0000 7f6b89e00640 1 -- 172.21.15.28:0/1919231577 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b8c0ac290 con 0x7f6b6c004d40 2024-02-21T23:05:42.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.992+0000 7f6b89e00640 1 --2- 172.21.15.28:0/1919231577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b6c004d40 0x7f6b8c1eb130 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c010860 tx=0x7f6b7c010d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:42.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.993+0000 7f6b82000640 1 -- 172.21.15.28:0/1919231577 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b7c00f040 con 0x7f6b6c004d40 2024-02-21T23:05:42.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.993+0000 7f6b82000640 1 -- 172.21.15.28:0/1919231577 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b7c00eae0 con 0x7f6b6c004d40 2024-02-21T23:05:42.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.993+0000 7f6b82000640 1 -- 172.21.15.28:0/1919231577 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b7c01c400 con 0x7f6b6c004d40 2024-02-21T23:05:42.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.993+0000 7f6b8b200640 1 -- 172.21.15.28:0/1919231577 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b8c0ac570 con 0x7f6b6c004d40 2024-02-21T23:05:42.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.993+0000 7f6b8b200640 1 -- 172.21.15.28:0/1919231577 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b8c1ee0c0 con 0x7f6b6c004d40 2024-02-21T23:05:42.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.995+0000 7f6b8b200640 1 -- 172.21.15.28:0/1919231577 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b8c0c9840 con 0x7f6b6c004d40 2024-02-21T23:05:42.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.996+0000 7f6b82000640 1 -- 172.21.15.28:0/1919231577 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6b7c01c5a0 con 0x7f6b6c004d40 2024-02-21T23:05:42.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.997+0000 7f6b82000640 1 --2- 172.21.15.28:0/1919231577 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6b6806b4d0 0x7f6b6806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:42.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.997+0000 7f6b82000640 1 -- 172.21.15.28:0/1919231577 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5679+0+0 (secure 0 0 0) 0x7f6b7c08a830 con 0x7f6b6c004d40 2024-02-21T23:05:42.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.997+0000 7f6b89400640 1 --2- 172.21.15.28:0/1919231577 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6b6806b4d0 0x7f6b6806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:42.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:42.997+0000 7f6b89400640 1 --2- 172.21.15.28:0/1919231577 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6b6806b4d0 0x7f6b6806d990 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f6b74007c40 tx=0x7f6b74009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:42.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:43.001+0000 7f6b82000640 1 -- 172.21.15.28:0/1919231577 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f6b8c0c9840 con 0x7f6b6c004d40 2024-02-21T23:05:43.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:43.200+0000 7f6b8b200640 1 -- 172.21.15.28:0/1919231577 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6b8c04f290 con 0x7f6b6c004d40 2024-02-21T23:05:43.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:43.201+0000 7f6b82000640 1 -- 172.21.15.28:0/1919231577 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v68) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6b7c018030 con 0x7f6b6c004d40 2024-02-21T23:05:43.190 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-02-21T23:05:43.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:43.203+0000 7f6b8b200640 1 -- 172.21.15.28:0/1919231577 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6b6806b4d0 msgr2=0x7f6b6806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:43.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:43.203+0000 7f6b8b200640 1 --2- 172.21.15.28:0/1919231577 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6b6806b4d0 0x7f6b6806d990 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f6b74007c40 tx=0x7f6b74009040 comp rx=0 tx=0).stop 2024-02-21T23:05:43.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:43.204+0000 7f6b8b200640 1 -- 172.21.15.28:0/1919231577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b6c004d40 msgr2=0x7f6b8c1eb130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:43.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:43.204+0000 7f6b8b200640 1 --2- 172.21.15.28:0/1919231577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b6c004d40 0x7f6b8c1eb130 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c010860 tx=0x7f6b7c010d20 comp rx=0 tx=0).stop 2024-02-21T23:05:43.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:43.204+0000 7f6b8b200640 1 -- 172.21.15.28:0/1919231577 shutdown_connections 2024-02-21T23:05:43.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:43.204+0000 7f6b8b200640 1 --2- 172.21.15.28:0/1919231577 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6b6806b4d0 0x7f6b6806d990 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:43.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:43.204+0000 7f6b8b200640 1 --2- 172.21.15.28:0/1919231577 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6b8c1ee5c0 0x7f6b8c1e7940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:43.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:43.204+0000 7f6b8b200640 1 --2- 172.21.15.28:0/1919231577 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b8c1ebb60 0x7f6b8c1575f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:43.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:43.204+0000 7f6b8b200640 1 --2- 172.21.15.28:0/1919231577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b6c004d40 0x7f6b8c1eb130 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:43.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:43.204+0000 7f6b8b200640 1 -- 172.21.15.28:0/1919231577 >> 172.21.15.28:0/1919231577 conn(0x7f6b8c073110 msgr2=0x7f6b8c0c9f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:43.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:43.204+0000 7f6b8b200640 1 -- 172.21.15.28:0/1919231577 shutdown_connections 2024-02-21T23:05:43.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:43.204+0000 7f6b8b200640 1 -- 172.21.15.28:0/1919231577 wait complete. 2024-02-21T23:05:43.215 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_last_resort_refused' 2024-02-21T23:05:43.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:43.442+0000 7ff77e600640 1 Processor -- start 2024-02-21T23:05:43.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:43.443+0000 7ff77e600640 1 -- start start 2024-02-21T23:05:43.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:43.444+0000 7ff77e600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff7780a5fd0 0x7ff77806dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:43.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:43.444+0000 7ff77e600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7780c2760 con 0x7ff77806ee40 2024-02-21T23:05:43.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:43.444+0000 7ff77e600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff778156310 con 0x7ff778150450 2024-02-21T23:05:43.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:43.444+0000 7ff77e600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7780c22c0 con 0x7ff7780a5fd0 2024-02-21T23:05:43.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:43.445+0000 7ff777e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff77806ee40 0x7ff778156af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:45416/0 (socket says 172.21.15.28:45416) 2024-02-21T23:05:43.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:43.445+0000 7ff777e00640 1 -- 172.21.15.28:0/1500399633 learned_addr learned my addr 172.21.15.28:0/1500399633 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:05:43.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:43.445+0000 7ff77dc00640 1 --2- 172.21.15.28:0/1500399633 >> v2:172.21.15.28:3300/0 conn(0x7ff7780a5fd0 0x7ff77806dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:43.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:43.445+0000 7ff777400640 1 -- 172.21.15.28:0/1500399633 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1153846687 0 0) 0x7ff7780c2760 con 0x7ff77806ee40 2024-02-21T23:05:43.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:43.445+0000 7ff77dc00640 1 -- 172.21.15.28:0/1500399633 >> v1:172.21.15.81:6789/0 conn(0x7ff778150450 legacy=0x7ff7781536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:43.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:43.445+0000 7ff77dc00640 1 -- 172.21.15.28:0/1500399633 >> v1:172.21.15.28:6789/0 conn(0x7ff77806ee40 legacy=0x7ff778156af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:43.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:43.445+0000 7ff77dc00640 1 -- 172.21.15.28:0/1500399633 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7780c29d0 con 0x7ff7780a5fd0 2024-02-21T23:05:43.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:43.447+0000 7ff77dc00640 1 --2- 172.21.15.28:0/1500399633 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7780a5fd0 0x7ff77806dff0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7ff770009d90 tx=0x7ff77002bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18be923b2beb12bc server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:43.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:43.447+0000 7ff777400640 1 -- 172.21.15.28:0/1500399633 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff770039070 con 0x7ff7780a5fd0 2024-02-21T23:05:43.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:43.447+0000 7ff777400640 1 -- 172.21.15.28:0/1500399633 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff77002cd00 con 0x7ff7780a5fd0 2024-02-21T23:05:43.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:43.447+0000 7ff777400640 1 -- 172.21.15.28:0/1500399633 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff770034660 con 0x7ff7780a5fd0 2024-02-21T23:05:43.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:43.447+0000 7ff77e600640 1 -- 172.21.15.28:0/1500399633 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7780a5fd0 msgr2=0x7ff77806dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:43.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:43.448+0000 7ff77e600640 1 --2- 172.21.15.28:0/1500399633 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7780a5fd0 0x7ff77806dff0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7ff770009d90 tx=0x7ff77002bf50 comp rx=0 tx=0).stop 2024-02-21T23:05:43.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:43.448+0000 7ff77e600640 1 -- 172.21.15.28:0/1500399633 shutdown_connections 2024-02-21T23:05:43.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:43.448+0000 7ff77e600640 1 --2- 172.21.15.28:0/1500399633 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7780a5fd0 0x7ff77806dff0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:43.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:43.448+0000 7ff77e600640 1 -- 172.21.15.28:0/1500399633 >> 172.21.15.28:0/1500399633 conn(0x7ff77806b630 msgr2=0x7ff7780c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:43.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:43.448+0000 7ff77e600640 1 -- 172.21.15.28:0/1500399633 shutdown_connections 2024-02-21T23:05:43.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:43.448+0000 7ff77e600640 1 -- 172.21.15.28:0/1500399633 wait complete. 2024-02-21T23:05:43.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:43.449+0000 7ff77e600640 1 Processor -- start 2024-02-21T23:05:43.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:43.449+0000 7ff77e600640 1 -- start start 2024-02-21T23:05:43.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:43.450+0000 7ff77e600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff77806ee40 0x7ff7780bfba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:43.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:43.450+0000 7ff77e600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7780a5fd0 0x7ff778146f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:43.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:43.450+0000 7ff77e600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff778150450 0x7ff7781ede00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:43.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:43.450+0000 7ff77e600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff778157200 con 0x7ff7780a5fd0 2024-02-21T23:05:43.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:43.450+0000 7ff77e600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff778156660 con 0x7ff778150450 2024-02-21T23:05:43.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:43.450+0000 7ff77e600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff77806e530 con 0x7ff77806ee40 2024-02-21T23:05:43.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:43.451+0000 7ff777e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7780a5fd0 0x7ff778146f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:43.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:43.451+0000 7ff777e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7780a5fd0 0x7ff778146f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50684/0 (socket says 172.21.15.28:50684) 2024-02-21T23:05:43.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:43.451+0000 7ff777e00640 1 -- 172.21.15.28:0/1723604876 learned_addr learned my addr 172.21.15.28:0/1723604876 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:43.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:43.451+0000 7ff777e00640 1 -- 172.21.15.28:0/1723604876 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff77806ee40 msgr2=0x7ff7780bfba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:43.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:43.451+0000 7ff777e00640 1 --2- 172.21.15.28:0/1723604876 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff77806ee40 0x7ff7780bfba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:43.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:43.451+0000 7ff777e00640 1 -- 172.21.15.28:0/1723604876 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff778150450 msgr2=0x7ff7781ede00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:43.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:43.451+0000 7ff777e00640 1 --2- 172.21.15.28:0/1723604876 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff778150450 0x7ff7781ede00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:43.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:43.451+0000 7ff777e00640 1 -- 172.21.15.28:0/1723604876 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff778146740 con 0x7ff7780a5fd0 2024-02-21T23:05:43.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:43.451+0000 7ff777e00640 1 --2- 172.21.15.28:0/1723604876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7780a5fd0 0x7ff778146f20 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7ff76400c4d0 tx=0x7ff76400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:43.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:43.452+0000 7ff774c00640 1 -- 172.21.15.28:0/1723604876 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff764018070 con 0x7ff7780a5fd0 2024-02-21T23:05:43.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:43.452+0000 7ff774c00640 1 -- 172.21.15.28:0/1723604876 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7640074e0 con 0x7ff7780a5fd0 2024-02-21T23:05:43.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:43.452+0000 7ff774c00640 1 -- 172.21.15.28:0/1723604876 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff764004a20 con 0x7ff7780a5fd0 2024-02-21T23:05:43.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:43.452+0000 7ff77e600640 1 -- 172.21.15.28:0/1723604876 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff778147e70 con 0x7ff7780a5fd0 2024-02-21T23:05:43.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:43.452+0000 7ff77e600640 1 -- 172.21.15.28:0/1723604876 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff778148430 con 0x7ff7780a5fd0 2024-02-21T23:05:43.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:43.453+0000 7ff77e600640 1 -- 172.21.15.28:0/1723604876 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff77814e6b0 con 0x7ff7780a5fd0 2024-02-21T23:05:43.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:43.454+0000 7ff774c00640 1 -- 172.21.15.28:0/1723604876 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff764004bc0 con 0x7ff7780a5fd0 2024-02-21T23:05:43.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:43.455+0000 7ff774c00640 1 --2- 172.21.15.28:0/1723604876 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff75806b4d0 0x7ff75806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:43.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:43.455+0000 7ff774c00640 1 -- 172.21.15.28:0/1723604876 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5679+0+0 (secure 0 0 0) 0x7ff764089400 con 0x7ff7780a5fd0 2024-02-21T23:05:43.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:43.455+0000 7ff77d200640 1 --2- 172.21.15.28:0/1723604876 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff75806b4d0 0x7ff75806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:43.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:43.456+0000 7ff77d200640 1 --2- 172.21.15.28:0/1723604876 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff75806b4d0 0x7ff75806d990 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7ff76c000f60 tx=0x7ff76c0079b0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:43.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:43.459+0000 7ff774c00640 1 -- 172.21.15.28:0/1723604876 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ff77814e6b0 con 0x7ff7780a5fd0 2024-02-21T23:05:43.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:43.724+0000 7ff77e600640 1 -- 172.21.15.28:0/1723604876 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_last_resort_refused"]} v 0) v1 -- 0x7ff77804f290 con 0x7ff7780a5fd0 2024-02-21T23:05:44.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.292+0000 7ff774c00640 1 -- 172.21.15.28:0/1723604876 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_last_resort_refused"]}]=0 v195) v1 ==== 161+0+0 (secure 0 0 0) 0x7ff764058290 con 0x7ff7780a5fd0 2024-02-21T23:05:44.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.295+0000 7ff77e600640 1 -- 172.21.15.28:0/1723604876 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff75806b4d0 msgr2=0x7ff75806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:44.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.295+0000 7ff77e600640 1 --2- 172.21.15.28:0/1723604876 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff75806b4d0 0x7ff75806d990 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7ff76c000f60 tx=0x7ff76c0079b0 comp rx=0 tx=0).stop 2024-02-21T23:05:44.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.295+0000 7ff77e600640 1 -- 172.21.15.28:0/1723604876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7780a5fd0 msgr2=0x7ff778146f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:44.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.295+0000 7ff77e600640 1 --2- 172.21.15.28:0/1723604876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7780a5fd0 0x7ff778146f20 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7ff76400c4d0 tx=0x7ff76400c990 comp rx=0 tx=0).stop 2024-02-21T23:05:44.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.295+0000 7ff77e600640 1 -- 172.21.15.28:0/1723604876 shutdown_connections 2024-02-21T23:05:44.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.295+0000 7ff77e600640 1 --2- 172.21.15.28:0/1723604876 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff75806b4d0 0x7ff75806d990 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:44.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.295+0000 7ff77e600640 1 --2- 172.21.15.28:0/1723604876 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff778150450 0x7ff7781ede00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:44.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.295+0000 7ff77e600640 1 --2- 172.21.15.28:0/1723604876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7780a5fd0 0x7ff778146f20 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:44.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.295+0000 7ff77e600640 1 --2- 172.21.15.28:0/1723604876 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff77806ee40 0x7ff7780bfba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:44.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.295+0000 7ff77e600640 1 -- 172.21.15.28:0/1723604876 >> 172.21.15.28:0/1723604876 conn(0x7ff77806b630 msgr2=0x7ff7780c3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:44.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.295+0000 7ff77e600640 1 -- 172.21.15.28:0/1723604876 shutdown_connections 2024-02-21T23:05:44.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.295+0000 7ff77e600640 1 -- 172.21.15.28:0/1723604876 wait complete. 2024-02-21T23:05:44.306 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-02-21T23:05:44.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.548+0000 7fa3ea800640 1 Processor -- start 2024-02-21T23:05:44.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.548+0000 7fa3ea800640 1 -- start start 2024-02-21T23:05:44.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.550+0000 7fa3ea800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fa3e4152560 0x7fa3e4154f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:44.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.550+0000 7fa3ea800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa3e406e9d0 0x7fa3e406ee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:44.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.550+0000 7fa3ea800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3e40a6270 con 0x7fa3e4158480 2024-02-21T23:05:44.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.550+0000 7fa3ea800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3e41554a0 con 0x7fa3e406e9d0 2024-02-21T23:05:44.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.550+0000 7fa3ea800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3e41556d0 con 0x7fa3e4152560 2024-02-21T23:05:44.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.550+0000 7fa3e9e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa3e406e9d0 0x7fa3e406ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:44.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.550+0000 7fa3e8a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa3e4158480 0x7fa3e406ae70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:45430/0 (socket says 172.21.15.28:45430) 2024-02-21T23:05:44.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.550+0000 7fa3e8a00640 1 -- 172.21.15.28:0/3506528800 learned_addr learned my addr 172.21.15.28:0/3506528800 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:05:44.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.551+0000 7fa3e9400640 1 --2- 172.21.15.28:0/3506528800 >> v2:172.21.15.81:3300/0 conn(0x7fa3e4152560 0x7fa3e4154f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:44.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.551+0000 7fa3e9e00640 1 -- 172.21.15.28:0/3506528800 >> v2:172.21.15.81:3300/0 conn(0x7fa3e4152560 msgr2=0x7fa3e4154f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:44.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.551+0000 7fa3e9e00640 1 --2- 172.21.15.28:0/3506528800 >> v2:172.21.15.81:3300/0 conn(0x7fa3e4152560 0x7fa3e4154f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:44.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.551+0000 7fa3e9e00640 1 -- 172.21.15.28:0/3506528800 >> v1:172.21.15.28:6789/0 conn(0x7fa3e4158480 legacy=0x7fa3e406ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:44.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.551+0000 7fa3e9e00640 1 -- 172.21.15.28:0/3506528800 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3e41559b0 con 0x7fa3e406e9d0 2024-02-21T23:05:44.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.552+0000 7fa3e9e00640 1 --2- 172.21.15.28:0/3506528800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3e406e9d0 0x7fa3e406ee00 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fa3e40a95b0 tx=0x7fa3dc02bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5631a0b6cf9778e9 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:44.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.552+0000 7fa3e3e00640 1 -- 172.21.15.28:0/3506528800 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3dc039070 con 0x7fa3e406e9d0 2024-02-21T23:05:44.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.552+0000 7fa3e3e00640 1 -- 172.21.15.28:0/3506528800 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3dc02cc70 con 0x7fa3e406e9d0 2024-02-21T23:05:44.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.552+0000 7fa3e3e00640 1 -- 172.21.15.28:0/3506528800 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3dc034660 con 0x7fa3e406e9d0 2024-02-21T23:05:44.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.553+0000 7fa3ea800640 1 -- 172.21.15.28:0/3506528800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3e406e9d0 msgr2=0x7fa3e406ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:44.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.553+0000 7fa3ea800640 1 --2- 172.21.15.28:0/3506528800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3e406e9d0 0x7fa3e406ee00 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fa3e40a95b0 tx=0x7fa3dc02bec0 comp rx=0 tx=0).stop 2024-02-21T23:05:44.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.553+0000 7fa3ea800640 1 -- 172.21.15.28:0/3506528800 shutdown_connections 2024-02-21T23:05:44.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.553+0000 7fa3ea800640 1 --2- 172.21.15.28:0/3506528800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3e406e9d0 0x7fa3e406ee00 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:44.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.553+0000 7fa3ea800640 1 --2- 172.21.15.28:0/3506528800 >> v2:172.21.15.81:3300/0 conn(0x7fa3e4152560 0x7fa3e4154f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:44.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.553+0000 7fa3ea800640 1 -- 172.21.15.28:0/3506528800 >> 172.21.15.28:0/3506528800 conn(0x7fa3e40a8530 msgr2=0x7fa3e40a8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:44.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.553+0000 7fa3ea800640 1 -- 172.21.15.28:0/3506528800 shutdown_connections 2024-02-21T23:05:44.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.553+0000 7fa3ea800640 1 -- 172.21.15.28:0/3506528800 wait complete. 2024-02-21T23:05:44.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.555+0000 7fa3ea800640 1 Processor -- start 2024-02-21T23:05:44.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.555+0000 7fa3ea800640 1 -- start start 2024-02-21T23:05:44.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.556+0000 7fa3ea800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa3e406e9d0 0x7fa3e4073150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:44.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.556+0000 7fa3ea800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3e4152560 0x7fa3e41e77a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:44.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.556+0000 7fa3ea800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa3e4158480 0x7fa3e414ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:44.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.556+0000 7fa3ea800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e406f340 con 0x7fa3e4152560 2024-02-21T23:05:44.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.556+0000 7fa3ea800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e406e4b0 con 0x7fa3e4158480 2024-02-21T23:05:44.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.556+0000 7fa3ea800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e4152940 con 0x7fa3e406e9d0 2024-02-21T23:05:44.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.556+0000 7fa3e8a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3e4152560 0x7fa3e41e77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:44.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.556+0000 7fa3e8a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3e4152560 0x7fa3e41e77a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50698/0 (socket says 172.21.15.28:50698) 2024-02-21T23:05:44.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.556+0000 7fa3e8a00640 1 -- 172.21.15.28:0/1380615867 learned_addr learned my addr 172.21.15.28:0/1380615867 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:44.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.556+0000 7fa3e9e00640 1 --2- 172.21.15.28:0/1380615867 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa3e4158480 0x7fa3e414ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:44.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.556+0000 7fa3e9400640 1 --2- 172.21.15.28:0/1380615867 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa3e406e9d0 0x7fa3e4073150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:44.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.556+0000 7fa3e8a00640 1 -- 172.21.15.28:0/1380615867 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa3e406e9d0 msgr2=0x7fa3e4073150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:44.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.557+0000 7fa3e8a00640 1 --2- 172.21.15.28:0/1380615867 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa3e406e9d0 0x7fa3e4073150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:44.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.557+0000 7fa3e8a00640 1 -- 172.21.15.28:0/1380615867 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa3e4158480 msgr2=0x7fa3e414ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:44.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.557+0000 7fa3e8a00640 1 --2- 172.21.15.28:0/1380615867 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa3e4158480 0x7fa3e414ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:44.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.557+0000 7fa3e8a00640 1 -- 172.21.15.28:0/1380615867 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3e41574c0 con 0x7fa3e4152560 2024-02-21T23:05:44.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.557+0000 7fa3e9e00640 1 --2- 172.21.15.28:0/1380615867 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa3e4158480 0x7fa3e414ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:05:44.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.557+0000 7fa3e8a00640 1 --2- 172.21.15.28:0/1380615867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3e4152560 0x7fa3e41e77a0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fa3d400c430 tx=0x7fa3d400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:44.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.557+0000 7fa3e1600640 1 -- 172.21.15.28:0/1380615867 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3d4017070 con 0x7fa3e4152560 2024-02-21T23:05:44.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.557+0000 7fa3e1600640 1 -- 172.21.15.28:0/1380615867 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3d4005ce0 con 0x7fa3e4152560 2024-02-21T23:05:44.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.557+0000 7fa3e1600640 1 -- 172.21.15.28:0/1380615867 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3d40072f0 con 0x7fa3e4152560 2024-02-21T23:05:44.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.557+0000 7fa3ea800640 1 -- 172.21.15.28:0/1380615867 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3e41eb250 con 0x7fa3e4152560 2024-02-21T23:05:44.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.557+0000 7fa3ea800640 1 -- 172.21.15.28:0/1380615867 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3e41eaed0 con 0x7fa3e4152560 2024-02-21T23:05:44.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.559+0000 7fa3ea800640 1 -- 172.21.15.28:0/1380615867 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3e404f290 con 0x7fa3e4152560 2024-02-21T23:05:44.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.560+0000 7fa3e1600640 1 -- 172.21.15.28:0/1380615867 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa3d401a070 con 0x7fa3e4152560 2024-02-21T23:05:44.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.561+0000 7fa3e1600640 1 --2- 172.21.15.28:0/1380615867 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa3c806b4d0 0x7fa3c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:44.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.561+0000 7fa3e1600640 1 -- 172.21.15.28:0/1380615867 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5679+0+0 (secure 0 0 0) 0x7fa3d4089610 con 0x7fa3e4152560 2024-02-21T23:05:44.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.562+0000 7fa3e9400640 1 --2- 172.21.15.28:0/1380615867 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa3c806b4d0 0x7fa3c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:44.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.563+0000 7fa3e9400640 1 --2- 172.21.15.28:0/1380615867 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa3c806b4d0 0x7fa3c806d990 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fa3d8006fd0 tx=0x7fa3d8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:44.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.565+0000 7fa3e1600640 1 -- 172.21.15.28:0/1380615867 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fa3d40584a0 con 0x7fa3e4152560 2024-02-21T23:05:44.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.766+0000 7fa3ea800640 1 -- 172.21.15.28:0/1380615867 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fa3e406e0f0 con 0x7fa3e4152560 2024-02-21T23:05:44.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.772+0000 7fa3e1600640 1 -- 172.21.15.28:0/1380615867 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v23)=0 v23) v1 ==== 122+0+0 (secure 0 0 0) 0x7fa3d4057da0 con 0x7fa3e4152560 2024-02-21T23:05:44.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.775+0000 7fa3ea800640 1 -- 172.21.15.28:0/1380615867 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa3c806b4d0 msgr2=0x7fa3c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:44.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.775+0000 7fa3ea800640 1 --2- 172.21.15.28:0/1380615867 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa3c806b4d0 0x7fa3c806d990 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fa3d8006fd0 tx=0x7fa3d8008040 comp rx=0 tx=0).stop 2024-02-21T23:05:44.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.775+0000 7fa3ea800640 1 -- 172.21.15.28:0/1380615867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3e4152560 msgr2=0x7fa3e41e77a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:44.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.775+0000 7fa3ea800640 1 --2- 172.21.15.28:0/1380615867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3e4152560 0x7fa3e41e77a0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fa3d400c430 tx=0x7fa3d400c8f0 comp rx=0 tx=0).stop 2024-02-21T23:05:44.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.776+0000 7fa3ea800640 1 -- 172.21.15.28:0/1380615867 shutdown_connections 2024-02-21T23:05:44.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.776+0000 7fa3ea800640 1 --2- 172.21.15.28:0/1380615867 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa3c806b4d0 0x7fa3c806d990 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:44.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.776+0000 7fa3ea800640 1 --2- 172.21.15.28:0/1380615867 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa3e4158480 0x7fa3e414ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:44.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.776+0000 7fa3ea800640 1 --2- 172.21.15.28:0/1380615867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3e4152560 0x7fa3e41e77a0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:44.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.776+0000 7fa3ea800640 1 --2- 172.21.15.28:0/1380615867 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa3e406e9d0 0x7fa3e4073150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:44.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.776+0000 7fa3ea800640 1 -- 172.21.15.28:0/1380615867 >> 172.21.15.28:0/1380615867 conn(0x7fa3e40a8530 msgr2=0x7fa3e406af70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:44.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.776+0000 7fa3ea800640 1 -- 172.21.15.28:0/1380615867 shutdown_connections 2024-02-21T23:05:44.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:44.776+0000 7fa3ea800640 1 -- 172.21.15.28:0/1380615867 wait complete. 2024-02-21T23:05:44.783 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-02-21T23:05:44.784 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-21T23:05:44.893 DEBUG:teuthology.orchestra.run.smithi081:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-21T23:05:45.006 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-02-21T23:05:45.006 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-21T23:05:45.006 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-21T23:05:45.066 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054608,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-02-21T23:05:45.066 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-02-21T23:05:45.066 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-21T23:05:45.123 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053833,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498812k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249404k,nr_inodes=812351,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498812k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-02-21T23:05:45.123 INFO:tasks.cephfs.cephfs_test_case:[, ] 2024-02-21T23:05:45.124 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:05:45.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.370+0000 7fa499800640 1 Processor -- start 2024-02-21T23:05:45.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.370+0000 7fa499800640 1 -- start start 2024-02-21T23:05:45.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.372+0000 7fa499800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fa49406dbf0 0x7fa494158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:45.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.372+0000 7fa499800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4940c74c0 con 0x7fa4940aff90 2024-02-21T23:05:45.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.372+0000 7fa499800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4940c77a0 con 0x7fa494155960 2024-02-21T23:05:45.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.372+0000 7fa499800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4940c7270 con 0x7fa49406dbf0 2024-02-21T23:05:45.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.372+0000 7fa493e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa4940aff90 0x7fa4940c51e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38220/0 (socket says 172.21.15.28:38220) 2024-02-21T23:05:45.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.372+0000 7fa493e00640 1 -- 172.21.15.28:0/3164230424 learned_addr learned my addr 172.21.15.28:0/3164230424 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:05:45.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.373+0000 7fa48b400640 1 --2- 172.21.15.28:0/3164230424 >> v2:172.21.15.81:3300/0 conn(0x7fa49406dbf0 0x7fa494158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:45.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.373+0000 7fa493400640 1 -- 172.21.15.28:0/3164230424 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1578680815 0 0) 0x7fa4940c74c0 con 0x7fa4940aff90 2024-02-21T23:05:45.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.373+0000 7fa493400640 1 -- 172.21.15.28:0/3164230424 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa478003520 con 0x7fa4940aff90 2024-02-21T23:05:45.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.373+0000 7fa48b400640 1 -- 172.21.15.28:0/3164230424 >> v1:172.21.15.81:6789/0 conn(0x7fa494155960 legacy=0x7fa49415f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:45.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.373+0000 7fa48b400640 1 -- 172.21.15.28:0/3164230424 >> v1:172.21.15.28:6789/0 conn(0x7fa4940aff90 legacy=0x7fa4940c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:45.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.373+0000 7fa48b400640 1 -- 172.21.15.28:0/3164230424 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4940c7a80 con 0x7fa49406dbf0 2024-02-21T23:05:45.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.375+0000 7fa48b400640 1 --2- 172.21.15.28:0/3164230424 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa49406dbf0 0x7fa494158ac0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa47c009d90 tx=0x7fa47c02bf40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=120c3795f7fa2786 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:45.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.376+0000 7fa493400640 1 -- 172.21.15.28:0/3164230424 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa47c039070 con 0x7fa49406dbf0 2024-02-21T23:05:45.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.376+0000 7fa493400640 1 -- 172.21.15.28:0/3164230424 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa47c02ccf0 con 0x7fa49406dbf0 2024-02-21T23:05:45.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.376+0000 7fa493400640 1 -- 172.21.15.28:0/3164230424 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa47c034660 con 0x7fa49406dbf0 2024-02-21T23:05:45.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.376+0000 7fa499800640 1 -- 172.21.15.28:0/3164230424 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa49406dbf0 msgr2=0x7fa494158ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:45.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.376+0000 7fa499800640 1 --2- 172.21.15.28:0/3164230424 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa49406dbf0 0x7fa494158ac0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa47c009d90 tx=0x7fa47c02bf40 comp rx=0 tx=0).stop 2024-02-21T23:05:45.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.376+0000 7fa499800640 1 -- 172.21.15.28:0/3164230424 shutdown_connections 2024-02-21T23:05:45.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.376+0000 7fa499800640 1 --2- 172.21.15.28:0/3164230424 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa49406dbf0 0x7fa494158ac0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:45.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.376+0000 7fa499800640 1 -- 172.21.15.28:0/3164230424 >> 172.21.15.28:0/3164230424 conn(0x7fa4940ac370 msgr2=0x7fa4940ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:45.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.376+0000 7fa499800640 1 -- 172.21.15.28:0/3164230424 shutdown_connections 2024-02-21T23:05:45.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.376+0000 7fa499800640 1 -- 172.21.15.28:0/3164230424 wait complete. 2024-02-21T23:05:45.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.378+0000 7fa499800640 1 Processor -- start 2024-02-21T23:05:45.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.378+0000 7fa499800640 1 -- start start 2024-02-21T23:05:45.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.379+0000 7fa499800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa49406dbf0 0x7fa4941ee350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:45.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.379+0000 7fa499800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa4940aff90 0x7fa49414bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:45.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.379+0000 7fa499800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa494155960 0x7fa49414c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:45.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.379+0000 7fa499800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4940c58f0 con 0x7fa494155960 2024-02-21T23:05:45.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.379+0000 7fa499800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa494158630 con 0x7fa49406dbf0 2024-02-21T23:05:45.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.379+0000 7fa499800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4940c5a70 con 0x7fa4940aff90 2024-02-21T23:05:45.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.379+0000 7fa498e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa494155960 0x7fa49414c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:45.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.380+0000 7fa498e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa494155960 0x7fa49414c1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47668/0 (socket says 172.21.15.28:47668) 2024-02-21T23:05:45.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.380+0000 7fa498e00640 1 -- 172.21.15.28:0/2010391406 learned_addr learned my addr 172.21.15.28:0/2010391406 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:45.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.380+0000 7fa48b400640 1 --2- 172.21.15.28:0/2010391406 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa4940aff90 0x7fa49414bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:45.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.380+0000 7fa493e00640 1 --2- 172.21.15.28:0/2010391406 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa49406dbf0 0x7fa4941ee350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:45.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.380+0000 7fa498e00640 1 -- 172.21.15.28:0/2010391406 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa4940aff90 msgr2=0x7fa49414bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:45.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.380+0000 7fa498e00640 1 --2- 172.21.15.28:0/2010391406 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa4940aff90 0x7fa49414bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:45.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.380+0000 7fa498e00640 1 -- 172.21.15.28:0/2010391406 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa49406dbf0 msgr2=0x7fa4941ee350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:45.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.380+0000 7fa498e00640 1 --2- 172.21.15.28:0/2010391406 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa49406dbf0 0x7fa4941ee350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:45.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.380+0000 7fa498e00640 1 -- 172.21.15.28:0/2010391406 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4941eedd0 con 0x7fa494155960 2024-02-21T23:05:45.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.380+0000 7fa48b400640 1 --2- 172.21.15.28:0/2010391406 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa4940aff90 0x7fa49414bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:05:45.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.380+0000 7fa493e00640 1 --2- 172.21.15.28:0/2010391406 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa49406dbf0 0x7fa4941ee350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:05:45.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.380+0000 7fa498e00640 1 --2- 172.21.15.28:0/2010391406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa494155960 0x7fa49414c1f0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fa48c00c390 tx=0x7fa48c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:45.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.381+0000 7fa490c00640 1 -- 172.21.15.28:0/2010391406 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa48c017070 con 0x7fa494155960 2024-02-21T23:05:45.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.381+0000 7fa490c00640 1 -- 172.21.15.28:0/2010391406 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa48c0052d0 con 0x7fa494155960 2024-02-21T23:05:45.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.381+0000 7fa499800640 1 -- 172.21.15.28:0/2010391406 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4941df020 con 0x7fa494155960 2024-02-21T23:05:45.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.381+0000 7fa490c00640 1 -- 172.21.15.28:0/2010391406 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa48c003810 con 0x7fa494155960 2024-02-21T23:05:45.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.381+0000 7fa499800640 1 -- 172.21.15.28:0/2010391406 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4941df5e0 con 0x7fa494155960 2024-02-21T23:05:45.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.382+0000 7fa499800640 1 -- 172.21.15.28:0/2010391406 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa49404f290 con 0x7fa494155960 2024-02-21T23:05:45.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.383+0000 7fa490c00640 1 -- 172.21.15.28:0/2010391406 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa48c01a070 con 0x7fa494155960 2024-02-21T23:05:45.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.384+0000 7fa490c00640 1 --2- 172.21.15.28:0/2010391406 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa47006b4d0 0x7fa47006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:45.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.384+0000 7fa490c00640 1 -- 172.21.15.28:0/2010391406 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5679+0+0 (secure 0 0 0) 0x7fa48c0895d0 con 0x7fa494155960 2024-02-21T23:05:45.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.385+0000 7fa493e00640 1 --2- 172.21.15.28:0/2010391406 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa47006b4d0 0x7fa47006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:45.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.386+0000 7fa493e00640 1 --2- 172.21.15.28:0/2010391406 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa47006b4d0 0x7fa47006d990 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fa4840011e0 tx=0x7fa484009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:45.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.389+0000 7fa490c00640 1 -- 172.21.15.28:0/2010391406 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fa48c0583d0 con 0x7fa494155960 2024-02-21T23:05:45.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.632+0000 7fa499800640 1 -- 172.21.15.28:0/2010391406 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa49406d9d0 con 0x7fa494155960 2024-02-21T23:05:45.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.633+0000 7fa490c00640 1 -- 172.21.15.28:0/2010391406 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+3699 (secure 0 0 0) 0x7fa48c05e0d0 con 0x7fa494155960 2024-02-21T23:05:45.622 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:05:45.622 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":55,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5258,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/3597525920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":3597525920},{"type":"v1","addr":"172.21.15.81:6835","nonce":3597525920}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":5428,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/2146447666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":2146447666},{"type":"v1","addr":"172.21.15.81:6837","nonce":2146447666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":6039,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3807390545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3807390545},{"type":"v1","addr":"172.21.15.28:6835","nonce":3807390545}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":55},{"gid":6147,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/2305639796","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2305639796},{"type":"v1","addr":"172.21.15.28:6834","nonce":2305639796}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[]} 2024-02-21T23:05:45.622 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 55 2024-02-21T23:05:45.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.636+0000 7fa499800640 1 -- 172.21.15.28:0/2010391406 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa47006b4d0 msgr2=0x7fa47006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:45.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.636+0000 7fa499800640 1 --2- 172.21.15.28:0/2010391406 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa47006b4d0 0x7fa47006d990 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fa4840011e0 tx=0x7fa484009040 comp rx=0 tx=0).stop 2024-02-21T23:05:45.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.636+0000 7fa499800640 1 -- 172.21.15.28:0/2010391406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa494155960 msgr2=0x7fa49414c1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:45.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.636+0000 7fa499800640 1 --2- 172.21.15.28:0/2010391406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa494155960 0x7fa49414c1f0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fa48c00c390 tx=0x7fa48c00c850 comp rx=0 tx=0).stop 2024-02-21T23:05:45.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.636+0000 7fa499800640 1 -- 172.21.15.28:0/2010391406 shutdown_connections 2024-02-21T23:05:45.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.636+0000 7fa499800640 1 --2- 172.21.15.28:0/2010391406 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa47006b4d0 0x7fa47006d990 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:45.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.636+0000 7fa499800640 1 --2- 172.21.15.28:0/2010391406 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa4940aff90 0x7fa49414bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:45.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.636+0000 7fa499800640 1 --2- 172.21.15.28:0/2010391406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa494155960 0x7fa49414c1f0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:45.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.636+0000 7fa499800640 1 --2- 172.21.15.28:0/2010391406 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa49406dbf0 0x7fa4941ee350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:45.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.636+0000 7fa499800640 1 -- 172.21.15.28:0/2010391406 >> 172.21.15.28:0/2010391406 conn(0x7fa4940ac370 msgr2=0x7fa4940ac750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:45.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.637+0000 7fa499800640 1 -- 172.21.15.28:0/2010391406 shutdown_connections 2024-02-21T23:05:45.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.637+0000 7fa499800640 1 -- 172.21.15.28:0/2010391406 wait complete. 2024-02-21T23:05:45.646 INFO:tasks.ceph.mds.a:Restarting daemon 2024-02-21T23:05:45.646 INFO:tasks.ceph.mds.a:Stopping old one... 2024-02-21T23:05:45.646 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-02-21T23:05:45.646 INFO:teuthology.orchestra.run:waiting for 300 2024-02-21T23:05:45.647 INFO:tasks.ceph.mds.c:Restarting daemon 2024-02-21T23:05:45.647 INFO:tasks.ceph.mds.c:Stopping old one... 2024-02-21T23:05:45.647 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-02-21T23:05:45.647 INFO:teuthology.orchestra.run:waiting for 300 2024-02-21T23:05:45.647 INFO:tasks.ceph.mds.b:Restarting daemon 2024-02-21T23:05:45.647 INFO:tasks.ceph.mds.b:Stopping old one... 2024-02-21T23:05:45.648 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-02-21T23:05:45.648 INFO:teuthology.orchestra.run:waiting for 300 2024-02-21T23:05:45.648 INFO:tasks.ceph.mds.d:Restarting daemon 2024-02-21T23:05:45.648 INFO:tasks.ceph.mds.d:Stopping old one... 2024-02-21T23:05:45.648 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-02-21T23:05:45.648 INFO:teuthology.orchestra.run:waiting for 300 2024-02-21T23:05:45.666 INFO:tasks.ceph.mds.a:Stopped 2024-02-21T23:05:45.667 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-02-21T23:05:45.668 INFO:tasks.ceph.mds.c:Stopped 2024-02-21T23:05:45.668 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-02-21T23:05:45.670 INFO:tasks.ceph.mds.d:Stopped 2024-02-21T23:05:45.670 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-02-21T23:05:45.673 INFO:tasks.ceph.mds.b:Stopped 2024-02-21T23:05:45.673 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-02-21T23:05:45.715 INFO:tasks.ceph.mds.a:Started 2024-02-21T23:05:45.715 DEBUG:teuthology.parallel:result is None 2024-02-21T23:05:45.720 INFO:tasks.ceph.mds.d:Started 2024-02-21T23:05:45.721 INFO:tasks.ceph.mds.c:Started 2024-02-21T23:05:45.721 DEBUG:teuthology.parallel:result is None 2024-02-21T23:05:45.721 DEBUG:teuthology.parallel:result is None 2024-02-21T23:05:45.727 INFO:tasks.ceph.mds.b:Started 2024-02-21T23:05:45.727 DEBUG:teuthology.parallel:result is None 2024-02-21T23:05:45.727 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-02-21T23:05:45.876 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-02-21T23:05:45.879 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-02-21T23:05:45.879 INFO:tasks.ceph.mds.d.smithi081.stdout:starting mds.d at 2024-02-21T23:05:45.886 INFO:tasks.ceph.mds.b.smithi081.stdout:starting mds.b at 2024-02-21T23:05:45.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.930+0000 7fc1b4400640 1 Processor -- start 2024-02-21T23:05:45.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.930+0000 7fc1b4400640 1 -- start start 2024-02-21T23:05:45.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.931+0000 7fc1b4400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fc1ac0c5f20 0x7fc1ac156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:45.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.931+0000 7fc1b4400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc1ac153220 0x7fc1ac153630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:45.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.931+0000 7fc1b4400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1ac0c5c10 con 0x7fc1ac0c2df0 2024-02-21T23:05:45.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.931+0000 7fc1b4400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1ac06e6a0 con 0x7fc1ac153220 2024-02-21T23:05:45.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.931+0000 7fc1b4400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1ac06e8d0 con 0x7fc1ac0c5f20 2024-02-21T23:05:45.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.931+0000 7fc1b2400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc1ac153220 0x7fc1ac153630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:45.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.931+0000 7fc1b2400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc1ac153220 0x7fc1ac153630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47710/0 (socket says 172.21.15.28:47710) 2024-02-21T23:05:45.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.931+0000 7fc1b2400640 1 -- 172.21.15.28:0/2488396722 learned_addr learned my addr 172.21.15.28:0/2488396722 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:45.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.931+0000 7fc1b1a00640 1 --1- 172.21.15.28:0/2488396722 >> v1:172.21.15.28:6789/0 conn(0x7fc1ac0c2df0 0x7fc1ac0c31d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38224/0 (socket says 172.21.15.28:38224) 2024-02-21T23:05:45.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.932+0000 7fc1b1000640 1 --2- 172.21.15.28:0/2488396722 >> v2:172.21.15.81:3300/0 conn(0x7fc1ac0c5f20 0x7fc1ac156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:45.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.932+0000 7fc1b2400640 1 -- 172.21.15.28:0/2488396722 >> v2:172.21.15.81:3300/0 conn(0x7fc1ac0c5f20 msgr2=0x7fc1ac156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:45.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.932+0000 7fc1b2400640 1 --2- 172.21.15.28:0/2488396722 >> v2:172.21.15.81:3300/0 conn(0x7fc1ac0c5f20 0x7fc1ac156ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:45.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.932+0000 7fc1b2400640 1 -- 172.21.15.28:0/2488396722 >> v1:172.21.15.28:6789/0 conn(0x7fc1ac0c2df0 legacy=0x7fc1ac0c31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:45.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.932+0000 7fc1b2400640 1 -- 172.21.15.28:0/2488396722 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1ac06ebb0 con 0x7fc1ac153220 2024-02-21T23:05:45.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.933+0000 7fc1b2400640 1 --2- 172.21.15.28:0/2488396722 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1ac153220 0x7fc1ac153630 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fc1ac06f8b0 tx=0x7fc1a402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=398079b3dd6b0850 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:45.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.933+0000 7fc1abe00640 1 -- 172.21.15.28:0/2488396722 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1a402cb20 con 0x7fc1ac153220 2024-02-21T23:05:45.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.933+0000 7fc1abe00640 1 -- 172.21.15.28:0/2488396722 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1a402ccc0 con 0x7fc1ac153220 2024-02-21T23:05:45.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.933+0000 7fc1abe00640 1 -- 172.21.15.28:0/2488396722 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1a4034610 con 0x7fc1ac153220 2024-02-21T23:05:45.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.934+0000 7fc1b4400640 1 -- 172.21.15.28:0/2488396722 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1ac153220 msgr2=0x7fc1ac153630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:45.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.934+0000 7fc1b4400640 1 --2- 172.21.15.28:0/2488396722 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1ac153220 0x7fc1ac153630 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fc1ac06f8b0 tx=0x7fc1a402bf80 comp rx=0 tx=0).stop 2024-02-21T23:05:45.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.934+0000 7fc1b4400640 1 -- 172.21.15.28:0/2488396722 shutdown_connections 2024-02-21T23:05:45.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.934+0000 7fc1b4400640 1 --2- 172.21.15.28:0/2488396722 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1ac153220 0x7fc1ac153630 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:45.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.934+0000 7fc1b4400640 1 --2- 172.21.15.28:0/2488396722 >> v2:172.21.15.81:3300/0 conn(0x7fc1ac0c5f20 0x7fc1ac156ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:45.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.934+0000 7fc1b4400640 1 -- 172.21.15.28:0/2488396722 >> 172.21.15.28:0/2488396722 conn(0x7fc1ac06b820 msgr2=0x7fc1ac0c67f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:45.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.934+0000 7fc1b4400640 1 -- 172.21.15.28:0/2488396722 shutdown_connections 2024-02-21T23:05:45.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.934+0000 7fc1b4400640 1 -- 172.21.15.28:0/2488396722 wait complete. 2024-02-21T23:05:45.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.936+0000 7fc1b4400640 1 Processor -- start 2024-02-21T23:05:45.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.936+0000 7fc1b4400640 1 -- start start 2024-02-21T23:05:45.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.937+0000 7fc1b4400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1ac0c2df0 0x7fc1ac155830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:45.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.937+0000 7fc1b4400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc1ac0c5f20 0x7fc1ac14fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:45.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.937+0000 7fc1b4400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc1ac153220 0x7fc1ac14ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:45.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.937+0000 7fc1b4400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1ac0c38e0 con 0x7fc1ac0c2df0 2024-02-21T23:05:45.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.937+0000 7fc1b4400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1ac06f170 con 0x7fc1ac153220 2024-02-21T23:05:45.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.937+0000 7fc1b4400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1ac0c3a60 con 0x7fc1ac0c5f20 2024-02-21T23:05:45.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.937+0000 7fc1b1a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1ac0c2df0 0x7fc1ac155830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:45.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.937+0000 7fc1b1a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1ac0c2df0 0x7fc1ac155830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47724/0 (socket says 172.21.15.28:47724) 2024-02-21T23:05:45.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.937+0000 7fc1b1a00640 1 -- 172.21.15.28:0/686811026 learned_addr learned my addr 172.21.15.28:0/686811026 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:45.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.937+0000 7fc1b1000640 1 --2- 172.21.15.28:0/686811026 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc1ac0c5f20 0x7fc1ac14fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:45.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.937+0000 7fc1b1a00640 1 -- 172.21.15.28:0/686811026 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc1ac0c5f20 msgr2=0x7fc1ac14fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:45.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.937+0000 7fc1b1a00640 1 --2- 172.21.15.28:0/686811026 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc1ac0c5f20 0x7fc1ac14fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:45.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.938+0000 7fc1b1a00640 1 -- 172.21.15.28:0/686811026 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc1ac153220 msgr2=0x7fc1ac14ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:45.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.938+0000 7fc1b1a00640 1 --2- 172.21.15.28:0/686811026 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc1ac153220 0x7fc1ac14ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:45.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.938+0000 7fc1b1a00640 1 -- 172.21.15.28:0/686811026 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1ac1ee300 con 0x7fc1ac0c2df0 2024-02-21T23:05:45.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.938+0000 7fc1b1000640 1 --2- 172.21.15.28:0/686811026 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc1ac0c5f20 0x7fc1ac14fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:05:45.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.938+0000 7fc1b1a00640 1 --2- 172.21.15.28:0/686811026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1ac0c2df0 0x7fc1ac155830 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fc19c00c3f0 tx=0x7fc19c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:45.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.938+0000 7fc1a9600640 1 -- 172.21.15.28:0/686811026 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc19c0074e0 con 0x7fc1ac0c2df0 2024-02-21T23:05:45.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.938+0000 7fc1a9600640 1 -- 172.21.15.28:0/686811026 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc19c007680 con 0x7fc1ac0c2df0 2024-02-21T23:05:45.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.938+0000 7fc1a9600640 1 -- 172.21.15.28:0/686811026 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc19c004a00 con 0x7fc1ac0c2df0 2024-02-21T23:05:45.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.938+0000 7fc1b4400640 1 -- 172.21.15.28:0/686811026 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1ac1ee5e0 con 0x7fc1ac0c2df0 2024-02-21T23:05:45.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.938+0000 7fc1b4400640 1 -- 172.21.15.28:0/686811026 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1ac1eeba0 con 0x7fc1ac0c2df0 2024-02-21T23:05:45.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.939+0000 7fc1b4400640 1 -- 172.21.15.28:0/686811026 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1ac06f400 con 0x7fc1ac0c2df0 2024-02-21T23:05:45.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.940+0000 7fc1a9600640 1 -- 172.21.15.28:0/686811026 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc19c01a020 con 0x7fc1ac0c2df0 2024-02-21T23:05:45.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.941+0000 7fc1a9600640 1 --2- 172.21.15.28:0/686811026 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc18c06b500 0x7fc18c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:45.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.941+0000 7fc1a9600640 1 -- 172.21.15.28:0/686811026 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5679+0+0 (secure 0 0 0) 0x7fc19c0899e0 con 0x7fc1ac0c2df0 2024-02-21T23:05:45.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.942+0000 7fc1b1000640 1 --2- 172.21.15.28:0/686811026 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc18c06b500 0x7fc18c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:45.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.943+0000 7fc1b1000640 1 --2- 172.21.15.28:0/686811026 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc18c06b500 0x7fc18c06d9c0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fc1ac0c33b0 tx=0x7fc198008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:45.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:45.946+0000 7fc1a9600640 1 -- 172.21.15.28:0/686811026 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc19c010300 con 0x7fc1ac0c2df0 2024-02-21T23:05:46.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.158+0000 7fc1b4400640 1 -- 172.21.15.28:0/686811026 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fc1ac04f370 con 0x7fc1ac0c2df0 2024-02-21T23:05:46.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.321+0000 7fc1a9600640 1 -- 172.21.15.28:0/686811026 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v70) v1 ==== 97+0+0 (secure 0 0 0) 0x7fc19c0587e0 con 0x7fc1ac0c2df0 2024-02-21T23:05:46.310 INFO:teuthology.orchestra.run.smithi028.stderr: removed all blocklist entries 2024-02-21T23:05:46.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.324+0000 7fc1b4400640 1 -- 172.21.15.28:0/686811026 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc18c06b500 msgr2=0x7fc18c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:46.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.324+0000 7fc1b4400640 1 --2- 172.21.15.28:0/686811026 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc18c06b500 0x7fc18c06d9c0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fc1ac0c33b0 tx=0x7fc198008040 comp rx=0 tx=0).stop 2024-02-21T23:05:46.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.324+0000 7fc1b4400640 1 -- 172.21.15.28:0/686811026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1ac0c2df0 msgr2=0x7fc1ac155830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:46.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.324+0000 7fc1b4400640 1 --2- 172.21.15.28:0/686811026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1ac0c2df0 0x7fc1ac155830 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fc19c00c3f0 tx=0x7fc19c00c8b0 comp rx=0 tx=0).stop 2024-02-21T23:05:46.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.324+0000 7fc1b4400640 1 -- 172.21.15.28:0/686811026 shutdown_connections 2024-02-21T23:05:46.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.324+0000 7fc1b4400640 1 --2- 172.21.15.28:0/686811026 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc18c06b500 0x7fc18c06d9c0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:46.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.324+0000 7fc1b4400640 1 --2- 172.21.15.28:0/686811026 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc1ac0c5f20 0x7fc1ac14fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:46.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.324+0000 7fc1b4400640 1 --2- 172.21.15.28:0/686811026 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc1ac153220 0x7fc1ac14ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:46.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.324+0000 7fc1b4400640 1 --2- 172.21.15.28:0/686811026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1ac0c2df0 0x7fc1ac155830 secure :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fc19c00c3f0 tx=0x7fc19c00c8b0 comp rx=0 tx=0).stop 2024-02-21T23:05:46.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.324+0000 7fc1b4400640 1 -- 172.21.15.28:0/686811026 >> 172.21.15.28:0/686811026 conn(0x7fc1ac06b820 msgr2=0x7fc1ac0c67b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:46.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.325+0000 7fc1b4400640 1 -- 172.21.15.28:0/686811026 shutdown_connections 2024-02-21T23:05:46.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.325+0000 7fc1b4400640 1 -- 172.21.15.28:0/686811026 wait complete. 2024-02-21T23:05:46.334 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-02-21T23:05:46.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.548+0000 7f55d3200640 1 Processor -- start 2024-02-21T23:05:46.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.549+0000 7f55d3200640 1 -- start start 2024-02-21T23:05:46.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.550+0000 7f55d3200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f55d40712e0 0x7f55d4072040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:46.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.550+0000 7f55d3200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f55d40a60f0 0x7f55d4152730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:46.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.550+0000 7f55d3200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f55d40a8a30 0x7f55d41551a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:46.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.550+0000 7f55d3200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55d40a85a0 con 0x7f55d40a8a30 2024-02-21T23:05:46.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.550+0000 7f55d3200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55d40a6590 con 0x7f55d40a60f0 2024-02-21T23:05:46.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.550+0000 7f55d3200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55d40a7f50 con 0x7f55d40712e0 2024-02-21T23:05:46.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.550+0000 7f55d2800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f55d40a8a30 0x7f55d41551a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:46.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.550+0000 7f55d2800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f55d40a8a30 0x7f55d41551a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47734/0 (socket says 172.21.15.28:47734) 2024-02-21T23:05:46.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.550+0000 7f55d2800640 1 -- 172.21.15.28:0/3119723718 learned_addr learned my addr 172.21.15.28:0/3119723718 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:46.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.551+0000 7f55d1400640 1 --2- 172.21.15.28:0/3119723718 >> v2:172.21.15.81:3300/0 conn(0x7f55d40a60f0 0x7f55d4152730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:46.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.551+0000 7f55d1e00640 1 --2- 172.21.15.28:0/3119723718 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f55d40712e0 0x7f55d4072040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:46.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.551+0000 7f55d2800640 1 -- 172.21.15.28:0/3119723718 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f55d40712e0 msgr2=0x7f55d4072040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:46.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.551+0000 7f55d2800640 1 --2- 172.21.15.28:0/3119723718 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f55d40712e0 0x7f55d4072040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:46.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.551+0000 7f55d2800640 1 -- 172.21.15.28:0/3119723718 >> v2:172.21.15.81:3300/0 conn(0x7f55d40a60f0 msgr2=0x7f55d4152730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:46.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.551+0000 7f55d2800640 1 --2- 172.21.15.28:0/3119723718 >> v2:172.21.15.81:3300/0 conn(0x7f55d40a60f0 0x7f55d4152730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:46.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.551+0000 7f55d2800640 1 -- 172.21.15.28:0/3119723718 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55d40a8230 con 0x7f55d40a8a30 2024-02-21T23:05:46.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.551+0000 7f55d1400640 1 --2- 172.21.15.28:0/3119723718 >> v2:172.21.15.81:3300/0 conn(0x7f55d40a60f0 0x7f55d4152730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:05:46.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.552+0000 7f55d2800640 1 --2- 172.21.15.28:0/3119723718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55d40a8a30 0x7f55d41551a0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f55d406f420 tx=0x7f55cc02dda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f182f365d26eacd server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:46.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.552+0000 7f55d0a00640 1 -- 172.21.15.28:0/3119723718 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55cc03b070 con 0x7f55d40a8a30 2024-02-21T23:05:46.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.552+0000 7f55d0a00640 1 -- 172.21.15.28:0/3119723718 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55cc02eb50 con 0x7f55d40a8a30 2024-02-21T23:05:46.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.552+0000 7f55d0a00640 1 -- 172.21.15.28:0/3119723718 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55cc036440 con 0x7f55d40a8a30 2024-02-21T23:05:46.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.553+0000 7f55d3200640 1 -- 172.21.15.28:0/3119723718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55d40a8a30 msgr2=0x7f55d41551a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:46.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.553+0000 7f55d3200640 1 --2- 172.21.15.28:0/3119723718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55d40a8a30 0x7f55d41551a0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f55d406f420 tx=0x7f55cc02dda0 comp rx=0 tx=0).stop 2024-02-21T23:05:46.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.553+0000 7f55d3200640 1 -- 172.21.15.28:0/3119723718 shutdown_connections 2024-02-21T23:05:46.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.553+0000 7f55d3200640 1 --2- 172.21.15.28:0/3119723718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55d40a8a30 0x7f55d41551a0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:46.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.553+0000 7f55d3200640 1 --2- 172.21.15.28:0/3119723718 >> v2:172.21.15.81:3300/0 conn(0x7f55d40a60f0 0x7f55d4152730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:46.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.553+0000 7f55d3200640 1 --2- 172.21.15.28:0/3119723718 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f55d40712e0 0x7f55d4072040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:46.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.553+0000 7f55d3200640 1 -- 172.21.15.28:0/3119723718 >> 172.21.15.28:0/3119723718 conn(0x7f55d406b550 msgr2=0x7f55d40c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:46.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.553+0000 7f55d3200640 1 -- 172.21.15.28:0/3119723718 shutdown_connections 2024-02-21T23:05:46.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.553+0000 7f55d3200640 1 -- 172.21.15.28:0/3119723718 wait complete. 2024-02-21T23:05:46.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.555+0000 7f55d3200640 1 Processor -- start 2024-02-21T23:05:46.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.555+0000 7f55d3200640 1 -- start start 2024-02-21T23:05:46.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.556+0000 7f55d3200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f55d40712e0 0x7f55d40a95b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:46.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.556+0000 7f55d3200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55d40a60f0 0x7f55d41e9e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:46.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.556+0000 7f55d3200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55d40a8a30 0x7f55d4072c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:46.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.556+0000 7f55d3200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55d40716c0 con 0x7f55d40a8a30 2024-02-21T23:05:46.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.556+0000 7f55d3200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55d40a6870 con 0x7f55d40a60f0 2024-02-21T23:05:46.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.556+0000 7f55d3200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55d4071840 con 0x7f55d40712e0 2024-02-21T23:05:46.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.556+0000 7f55d2800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55d40a8a30 0x7f55d4072c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:46.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.556+0000 7f55d2800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55d40a8a30 0x7f55d4072c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47738/0 (socket says 172.21.15.28:47738) 2024-02-21T23:05:46.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.556+0000 7f55d2800640 1 -- 172.21.15.28:0/1117555807 learned_addr learned my addr 172.21.15.28:0/1117555807 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:46.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.557+0000 7f55d1e00640 1 --2- 172.21.15.28:0/1117555807 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f55d40712e0 0x7f55d40a95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:46.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.557+0000 7f55d1400640 1 --2- 172.21.15.28:0/1117555807 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55d40a60f0 0x7f55d41e9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:46.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.557+0000 7f55d2800640 1 -- 172.21.15.28:0/1117555807 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f55d40712e0 msgr2=0x7f55d40a95b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:46.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.557+0000 7f55d2800640 1 --2- 172.21.15.28:0/1117555807 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f55d40712e0 0x7f55d40a95b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:46.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.557+0000 7f55d2800640 1 -- 172.21.15.28:0/1117555807 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55d40a60f0 msgr2=0x7f55d41e9e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:46.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.557+0000 7f55d2800640 1 --2- 172.21.15.28:0/1117555807 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55d40a60f0 0x7f55d41e9e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:46.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.557+0000 7f55d2800640 1 -- 172.21.15.28:0/1117555807 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55d4147c20 con 0x7f55d40a8a30 2024-02-21T23:05:46.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.557+0000 7f55d1e00640 1 --2- 172.21.15.28:0/1117555807 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f55d40712e0 0x7f55d40a95b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:05:46.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.557+0000 7f55d2800640 1 --2- 172.21.15.28:0/1117555807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55d40a8a30 0x7f55d4072c40 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f55cc0090b0 tx=0x7f55cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:46.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.557+0000 7f55ca000640 1 -- 172.21.15.28:0/1117555807 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55cc045020 con 0x7f55d40a8a30 2024-02-21T23:05:46.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.557+0000 7f55ca000640 1 -- 172.21.15.28:0/1117555807 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55cc041420 con 0x7f55d40a8a30 2024-02-21T23:05:46.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.557+0000 7f55ca000640 1 -- 172.21.15.28:0/1117555807 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55cc03b040 con 0x7f55d40a8a30 2024-02-21T23:05:46.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.557+0000 7f55d3200640 1 -- 172.21.15.28:0/1117555807 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55d4147f00 con 0x7f55d40a8a30 2024-02-21T23:05:46.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.558+0000 7f55d3200640 1 -- 172.21.15.28:0/1117555807 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55d40c5f30 con 0x7f55d40a8a30 2024-02-21T23:05:46.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.559+0000 7f55d3200640 1 -- 172.21.15.28:0/1117555807 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55d404f290 con 0x7f55d40a8a30 2024-02-21T23:05:46.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.560+0000 7f55ca000640 1 -- 172.21.15.28:0/1117555807 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f55cc048020 con 0x7f55d40a8a30 2024-02-21T23:05:46.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.560+0000 7f55ca000640 1 --2- 172.21.15.28:0/1117555807 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f55b006b4d0 0x7f55b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:46.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.561+0000 7f55ca000640 1 -- 172.21.15.28:0/1117555807 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f55cc0ae510 con 0x7f55d40a8a30 2024-02-21T23:05:46.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.561+0000 7f55d1e00640 1 --2- 172.21.15.28:0/1117555807 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f55b006b4d0 0x7f55b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:46.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.562+0000 7f55d1e00640 1 --2- 172.21.15.28:0/1117555807 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f55b006b4d0 0x7f55b006d990 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f55c4009f50 tx=0x7f55c400b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:46.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.565+0000 7f55ca000640 1 -- 172.21.15.28:0/1117555807 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f55cc04f030 con 0x7f55d40a8a30 2024-02-21T23:05:46.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.837+0000 7f55d3200640 1 -- 172.21.15.28:0/1117555807 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f55d40c1180 con 0x7f55d40a8a30 2024-02-21T23:05:46.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.838+0000 7f55ca000640 1 -- 172.21.15.28:0/1117555807 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v12) v1 ==== 80+0+4724 (secure 0 0 0) 0x7f55cc07d540 con 0x7f55d40a8a30 2024-02-21T23:05:46.826 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:05:46.826 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-21T23:05:46.827 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_dump": [ 2024-02-21T23:05:46.827 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:05:46.827 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.a", 2024-02-21T23:05:46.827 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDSgNZl4FY1KRAAJSK5Ibb0ibKiKXYpTLgtmQ==", 2024-02-21T23:05:46.827 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:05:46.827 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:05:46.827 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-02-21T23:05:46.827 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-02-21T23:05:46.827 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:05:46.827 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:05:46.827 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:05:46.827 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:05:46.828 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.b", 2024-02-21T23:05:46.828 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDTgNZlwRcAERAAgsYmmkuobKelBEJVCRpXUg==", 2024-02-21T23:05:46.828 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:05:46.828 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:05:46.828 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-02-21T23:05:46.828 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-02-21T23:05:46.828 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:05:46.828 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:05:46.828 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:05:46.828 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:05:46.828 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.c", 2024-02-21T23:05:46.828 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDSgNZloFpQOxAAK2WBArXRVNTgqH9WWDEPDA==", 2024-02-21T23:05:46.828 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:05:46.828 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:05:46.829 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-02-21T23:05:46.829 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-02-21T23:05:46.829 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:05:46.829 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:05:46.829 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:05:46.829 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:05:46.829 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.d", 2024-02-21T23:05:46.829 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDTgNZlblOmIRAAxe2HfSKGSkkzJDN4aVdzgQ==", 2024-02-21T23:05:46.829 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:05:46.829 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:05:46.829 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-02-21T23:05:46.829 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-02-21T23:05:46.829 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:05:46.829 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:05:46.830 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:05:46.830 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:05:46.830 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.0", 2024-02-21T23:05:46.830 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDVgNZlxNJbLxAAXot7Q3chrLEBsFNO8kT6QQ==", 2024-02-21T23:05:46.830 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:05:46.830 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:05:46.830 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:05:46.830 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:05:46.830 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:05:46.830 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:05:46.830 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:05:46.830 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.1", 2024-02-21T23:05:46.830 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDYgNZlZZfWORAAr4XkOxLYLmumVbu362N2HQ==", 2024-02-21T23:05:46.831 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:05:46.831 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:05:46.831 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:05:46.831 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:05:46.831 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:05:46.831 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:05:46.831 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:05:46.831 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.2", 2024-02-21T23:05:46.831 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDcgNZl9mcDGBAAIZGOF8IUMenwo386x95mwg==", 2024-02-21T23:05:46.831 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:05:46.831 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:05:46.831 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:05:46.831 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:05:46.831 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:05:46.832 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:05:46.832 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:05:46.832 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.3", 2024-02-21T23:05:46.832 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDfgNZl8ALSMBAAag9cmGH0/4xALWSUmrZowg==", 2024-02-21T23:05:46.832 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:05:46.832 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:05:46.832 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:05:46.832 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:05:46.832 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:05:46.832 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:05:46.832 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:05:46.832 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.4", 2024-02-21T23:05:46.832 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDngNZlBgeKABAAeW99RVfCBvB1lp7w7mQBlQ==", 2024-02-21T23:05:46.832 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:05:46.833 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:05:46.833 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:05:46.833 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:05:46.833 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:05:46.833 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:05:46.833 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:05:46.833 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.5", 2024-02-21T23:05:46.833 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDqgNZljQG3ChAAY/IZEIcvcGNvlFGIWib3gg==", 2024-02-21T23:05:46.833 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:05:46.833 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:05:46.833 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:05:46.833 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:05:46.833 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:05:46.833 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:05:46.833 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:05:46.834 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.6", 2024-02-21T23:05:46.834 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDtgNZlKnakIxAAkW5xlkshzvEHJdmpgPxR+g==", 2024-02-21T23:05:46.834 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:05:46.834 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:05:46.834 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:05:46.834 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:05:46.834 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:05:46.834 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:05:46.834 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:05:46.834 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.7", 2024-02-21T23:05:46.834 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDxgNZlTWxYARAASX1/gmqO+8mQPP8uA4xjiQ==", 2024-02-21T23:05:46.834 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:05:46.834 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:05:46.834 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:05:46.834 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:05:46.835 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:05:46.835 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:05:46.835 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:05:46.835 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.0", 2024-02-21T23:05:46.835 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDTgNZlILvYLxAAoREWxKKT7YcXnhkgk8YvoQ==", 2024-02-21T23:05:46.835 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:05:46.835 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:05:46.835 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow r", 2024-02-21T23:05:46.835 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-02-21T23:05:46.835 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:05:46.835 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:05:46.835 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:05:46.835 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.1", 2024-02-21T23:05:46.835 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDTgNZlm7ImOxAA8iRcXlzFPZPaXQfsKhuO/g==", 2024-02-21T23:05:46.836 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:05:46.836 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:05:46.836 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-02-21T23:05:46.836 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-02-21T23:05:46.836 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-02-21T23:05:46.836 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:05:46.836 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:05:46.836 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:05:46.836 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.admin", 2024-02-21T23:05:46.836 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDRgNZlBYd1GhAAbK5qBCIkf0Ej5nEyPnTxIQ==", 2024-02-21T23:05:46.836 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:05:46.836 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-02-21T23:05:46.836 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-02-21T23:05:46.836 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-02-21T23:05:46.837 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:05:46.837 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:05:46.837 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:05:46.837 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:05:46.837 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.x", 2024-02-21T23:05:46.837 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDSgNZlJYKeHBAAYPskmLRr7Jw44TdKzM/JMg==", 2024-02-21T23:05:46.837 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:05:46.837 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-02-21T23:05:46.837 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-02-21T23:05:46.837 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:05:46.837 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:05:46.837 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:05:46.837 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:05:46.837 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.y", 2024-02-21T23:05:46.838 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDSgNZlSRkOEBAAOhRj9PsZDpjoV6WT4drrZA==", 2024-02-21T23:05:46.838 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:05:46.838 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-02-21T23:05:46.838 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-02-21T23:05:46.838 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:05:46.838 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:05:46.838 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:05:46.838 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-21T23:05:46.838 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-21T23:05:46.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.840+0000 7f55d3200640 1 -- 172.21.15.28:0/1117555807 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f55b006b4d0 msgr2=0x7f55b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:46.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.840+0000 7f55d3200640 1 --2- 172.21.15.28:0/1117555807 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f55b006b4d0 0x7f55b006d990 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f55c4009f50 tx=0x7f55c400b040 comp rx=0 tx=0).stop 2024-02-21T23:05:46.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.840+0000 7f55d3200640 1 -- 172.21.15.28:0/1117555807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55d40a8a30 msgr2=0x7f55d4072c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:46.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.840+0000 7f55d3200640 1 --2- 172.21.15.28:0/1117555807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55d40a8a30 0x7f55d4072c40 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f55cc0090b0 tx=0x7f55cc0046c0 comp rx=0 tx=0).stop 2024-02-21T23:05:46.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.841+0000 7f55d3200640 1 -- 172.21.15.28:0/1117555807 shutdown_connections 2024-02-21T23:05:46.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.841+0000 7f55d3200640 1 --2- 172.21.15.28:0/1117555807 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f55b006b4d0 0x7f55b006d990 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:46.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.841+0000 7f55d3200640 1 --2- 172.21.15.28:0/1117555807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55d40a8a30 0x7f55d4072c40 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:46.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.841+0000 7f55d3200640 1 --2- 172.21.15.28:0/1117555807 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55d40a60f0 0x7f55d41e9e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:46.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.841+0000 7f55d3200640 1 --2- 172.21.15.28:0/1117555807 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f55d40712e0 0x7f55d40a95b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:46.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.841+0000 7f55d3200640 1 -- 172.21.15.28:0/1117555807 >> 172.21.15.28:0/1117555807 conn(0x7f55d406b550 msgr2=0x7f55d4149430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:46.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.841+0000 7f55d3200640 1 -- 172.21.15.28:0/1117555807 shutdown_connections 2024-02-21T23:05:46.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:46.841+0000 7f55d3200640 1 -- 172.21.15.28:0/1117555807 wait complete. 2024-02-21T23:05:46.847 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:05:47.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.077+0000 7f0ee3000640 1 Processor -- start 2024-02-21T23:05:47.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.077+0000 7f0ee3000640 1 -- start start 2024-02-21T23:05:47.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.079+0000 7f0ee3000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0edc156c90 0x7f0edc0c8e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:47.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.079+0000 7f0ee3000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0edc1540b0 0x7f0edc15d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:47.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.079+0000 7f0ee3000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0edc0c8000 con 0x7f0edc0c8a80 2024-02-21T23:05:47.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.079+0000 7f0ee3000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0edc0c82e0 con 0x7f0edc156c90 2024-02-21T23:05:47.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.079+0000 7f0ee3000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0edc1561b0 con 0x7f0edc1540b0 2024-02-21T23:05:47.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.080+0000 7f0ee1200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0edc156c90 0x7f0edc0c8e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:47.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.080+0000 7f0ee1200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0edc156c90 0x7f0edc0c8e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47748/0 (socket says 172.21.15.28:47748) 2024-02-21T23:05:47.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.080+0000 7f0ee1200640 1 -- 172.21.15.28:0/320045109 learned_addr learned my addr 172.21.15.28:0/320045109 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:47.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.080+0000 7f0ee2600640 1 --2- 172.21.15.28:0/320045109 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0edc1540b0 0x7f0edc15d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:47.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.080+0000 7f0ee1200640 1 -- 172.21.15.28:0/320045109 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0edc1540b0 msgr2=0x7f0edc15d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:47.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.080+0000 7f0ee1200640 1 --2- 172.21.15.28:0/320045109 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0edc1540b0 0x7f0edc15d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:47.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.080+0000 7f0ee1200640 1 -- 172.21.15.28:0/320045109 >> v1:172.21.15.81:6789/0 conn(0x7f0edc0c8a80 legacy=0x7f0edc153640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:47.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.080+0000 7f0ee1200640 1 -- 172.21.15.28:0/320045109 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0edc156490 con 0x7f0edc156c90 2024-02-21T23:05:47.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.080+0000 7f0ee2600640 1 --2- 172.21.15.28:0/320045109 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0edc1540b0 0x7f0edc15d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:05:47.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.081+0000 7f0ee1200640 1 --2- 172.21.15.28:0/320045109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0edc156c90 0x7f0edc0c8e60 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f0ec8009870 tx=0x7f0ec802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea66fff176c252e2 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:47.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.082+0000 7f0edbe00640 1 -- 172.21.15.28:0/320045109 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ec8035020 con 0x7f0edc156c90 2024-02-21T23:05:47.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.082+0000 7f0edbe00640 1 -- 172.21.15.28:0/320045109 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ec8031640 con 0x7f0edc156c90 2024-02-21T23:05:47.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.082+0000 7f0edbe00640 1 -- 172.21.15.28:0/320045109 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ec802d040 con 0x7f0edc156c90 2024-02-21T23:05:47.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.082+0000 7f0ee3000640 1 -- 172.21.15.28:0/320045109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0edc156c90 msgr2=0x7f0edc0c8e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:47.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.082+0000 7f0ee3000640 1 --2- 172.21.15.28:0/320045109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0edc156c90 0x7f0edc0c8e60 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f0ec8009870 tx=0x7f0ec802b8f0 comp rx=0 tx=0).stop 2024-02-21T23:05:47.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.083+0000 7f0ee3000640 1 -- 172.21.15.28:0/320045109 shutdown_connections 2024-02-21T23:05:47.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.083+0000 7f0ee3000640 1 --2- 172.21.15.28:0/320045109 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0edc1540b0 0x7f0edc15d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:47.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.083+0000 7f0ee3000640 1 --2- 172.21.15.28:0/320045109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0edc156c90 0x7f0edc0c8e60 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:47.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.083+0000 7f0ee3000640 1 -- 172.21.15.28:0/320045109 >> 172.21.15.28:0/320045109 conn(0x7f0edc06b260 msgr2=0x7f0edc06b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:47.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.083+0000 7f0ee3000640 1 -- 172.21.15.28:0/320045109 shutdown_connections 2024-02-21T23:05:47.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.083+0000 7f0ee3000640 1 -- 172.21.15.28:0/320045109 wait complete. 2024-02-21T23:05:47.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.084+0000 7f0ee3000640 1 Processor -- start 2024-02-21T23:05:47.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.084+0000 7f0ee3000640 1 -- start start 2024-02-21T23:05:47.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.085+0000 7f0ee3000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0edc0c8a80 0x7f0edc06bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:47.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.085+0000 7f0ee3000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0edc1540b0 0x7f0edc149c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:47.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.085+0000 7f0ee3000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0edc156c90 0x7f0edc0ac110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:47.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.085+0000 7f0ee3000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0edc0c2c30 con 0x7f0edc156c90 2024-02-21T23:05:47.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.085+0000 7f0ee3000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0edc0c7c90 con 0x7f0edc0c8a80 2024-02-21T23:05:47.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.085+0000 7f0ee3000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0edc14ced0 con 0x7f0edc1540b0 2024-02-21T23:05:47.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.086+0000 7f0ee2600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0edc156c90 0x7f0edc0ac110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:47.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.086+0000 7f0ee2600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0edc156c90 0x7f0edc0ac110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47762/0 (socket says 172.21.15.28:47762) 2024-02-21T23:05:47.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.086+0000 7f0ee2600640 1 -- 172.21.15.28:0/4085462298 learned_addr learned my addr 172.21.15.28:0/4085462298 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:47.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.086+0000 7f0ee1c00640 1 --2- 172.21.15.28:0/4085462298 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0edc0c8a80 0x7f0edc06bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:47.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.086+0000 7f0ee1200640 1 --2- 172.21.15.28:0/4085462298 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0edc1540b0 0x7f0edc149c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:47.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.086+0000 7f0ee2600640 1 -- 172.21.15.28:0/4085462298 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0edc1540b0 msgr2=0x7f0edc149c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:47.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.086+0000 7f0ee2600640 1 --2- 172.21.15.28:0/4085462298 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0edc1540b0 0x7f0edc149c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:47.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.086+0000 7f0ee2600640 1 -- 172.21.15.28:0/4085462298 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0edc0c8a80 msgr2=0x7f0edc06bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:47.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.086+0000 7f0ee2600640 1 --2- 172.21.15.28:0/4085462298 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0edc0c8a80 0x7f0edc06bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:47.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.086+0000 7f0ee2600640 1 -- 172.21.15.28:0/4085462298 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0edc0ab670 con 0x7f0edc156c90 2024-02-21T23:05:47.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.086+0000 7f0ee1200640 1 --2- 172.21.15.28:0/4085462298 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0edc1540b0 0x7f0edc149c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:05:47.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.087+0000 7f0ee2600640 1 --2- 172.21.15.28:0/4085462298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0edc156c90 0x7f0edc0ac110 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f0ed4007f20 tx=0x7f0ed400d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:47.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.087+0000 7f0ed9600640 1 -- 172.21.15.28:0/4085462298 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ed401a070 con 0x7f0edc156c90 2024-02-21T23:05:47.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.087+0000 7f0ed9600640 1 -- 172.21.15.28:0/4085462298 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ed4003b90 con 0x7f0edc156c90 2024-02-21T23:05:47.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.087+0000 7f0ed9600640 1 -- 172.21.15.28:0/4085462298 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ed4004030 con 0x7f0edc156c90 2024-02-21T23:05:47.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.087+0000 7f0ee3000640 1 -- 172.21.15.28:0/4085462298 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0edc1eb810 con 0x7f0edc156c90 2024-02-21T23:05:47.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.087+0000 7f0ee3000640 1 -- 172.21.15.28:0/4085462298 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0edc0abbf0 con 0x7f0edc156c90 2024-02-21T23:05:47.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.088+0000 7f0ee3000640 1 -- 172.21.15.28:0/4085462298 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0edc04f290 con 0x7f0edc156c90 2024-02-21T23:05:47.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.089+0000 7f0ed9600640 1 -- 172.21.15.28:0/4085462298 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0ed401d070 con 0x7f0edc156c90 2024-02-21T23:05:47.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.090+0000 7f0ed9600640 1 --2- 172.21.15.28:0/4085462298 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0ebc06b4d0 0x7f0ebc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:47.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.090+0000 7f0ed9600640 1 -- 172.21.15.28:0/4085462298 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f0ed408a490 con 0x7f0edc156c90 2024-02-21T23:05:47.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.091+0000 7f0ee1c00640 1 --2- 172.21.15.28:0/4085462298 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0ebc06b4d0 0x7f0ebc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:47.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.092+0000 7f0ee1c00640 1 --2- 172.21.15.28:0/4085462298 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0ebc06b4d0 0x7f0ebc06d990 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f0ed0000f60 tx=0x7f0ed0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:47.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.094+0000 7f0ed9600640 1 -- 172.21.15.28:0/4085462298 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f0ed4010070 con 0x7f0edc156c90 2024-02-21T23:05:47.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.297+0000 7f0ee3000640 1 -- 172.21.15.28:0/4085462298 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0edc0c7520 con 0x7f0edc156c90 2024-02-21T23:05:47.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.298+0000 7f0ed9600640 1 -- 172.21.15.28:0/4085462298 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) v1 ==== 74+0+13105 (secure 0 0 0) 0x7f0ed40593a0 con 0x7f0edc156c90 2024-02-21T23:05:47.286 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:05:47.287 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":72,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:05:46.333831+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:05:47.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.300+0000 7f0ee3000640 1 -- 172.21.15.28:0/4085462298 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0ebc06b4d0 msgr2=0x7f0ebc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:47.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.300+0000 7f0ee3000640 1 --2- 172.21.15.28:0/4085462298 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0ebc06b4d0 0x7f0ebc06d990 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f0ed0000f60 tx=0x7f0ed0009040 comp rx=0 tx=0).stop 2024-02-21T23:05:47.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.300+0000 7f0ee3000640 1 -- 172.21.15.28:0/4085462298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0edc156c90 msgr2=0x7f0edc0ac110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:47.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.300+0000 7f0ee3000640 1 --2- 172.21.15.28:0/4085462298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0edc156c90 0x7f0edc0ac110 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f0ed4007f20 tx=0x7f0ed400d560 comp rx=0 tx=0).stop 2024-02-21T23:05:47.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.300+0000 7f0ee3000640 1 -- 172.21.15.28:0/4085462298 shutdown_connections 2024-02-21T23:05:47.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.300+0000 7f0ee3000640 1 --2- 172.21.15.28:0/4085462298 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0ebc06b4d0 0x7f0ebc06d990 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:47.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.300+0000 7f0ee3000640 1 --2- 172.21.15.28:0/4085462298 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0edc1540b0 0x7f0edc149c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:47.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.300+0000 7f0ee3000640 1 --2- 172.21.15.28:0/4085462298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0edc156c90 0x7f0edc0ac110 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:47.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.300+0000 7f0ee3000640 1 --2- 172.21.15.28:0/4085462298 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0edc0c8a80 0x7f0edc06bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:47.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.300+0000 7f0ee3000640 1 -- 172.21.15.28:0/4085462298 >> 172.21.15.28:0/4085462298 conn(0x7f0edc06b260 msgr2=0x7f0edc0a9c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:47.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.301+0000 7f0ee3000640 1 -- 172.21.15.28:0/4085462298 shutdown_connections 2024-02-21T23:05:47.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.301+0000 7f0ee3000640 1 -- 172.21.15.28:0/4085462298 wait complete. 2024-02-21T23:05:47.314 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-21T23:02:30.728329+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-21T23:05:47.314 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-21T23:05:47.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.554+0000 7f0f39a00640 1 Processor -- start 2024-02-21T23:05:47.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.554+0000 7f0f39a00640 1 -- start start 2024-02-21T23:05:47.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.556+0000 7f0f39a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0f34150240 0x7f0f340c12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:47.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.556+0000 7f0f39a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f3406e220 con 0x7f0f340c4580 2024-02-21T23:05:47.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.556+0000 7f0f39a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f3406f780 con 0x7f0f340c2c10 2024-02-21T23:05:47.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.556+0000 7f0f39a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f34150620 con 0x7f0f34150240 2024-02-21T23:05:47.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.556+0000 7f0f33400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f0f340c4580 0x7f0f3406db10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38232/0 (socket says 172.21.15.28:38232) 2024-02-21T23:05:47.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.556+0000 7f0f33400640 1 -- 172.21.15.28:0/637633037 learned_addr learned my addr 172.21.15.28:0/637633037 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:05:47.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.557+0000 7f0f33e00640 1 --2- 172.21.15.28:0/637633037 >> v2:172.21.15.81:3300/0 conn(0x7f0f34150240 0x7f0f340c12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:47.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.557+0000 7f0f32a00640 1 -- 172.21.15.28:0/637633037 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 130506535 0 0) 0x7f0f3406e220 con 0x7f0f340c4580 2024-02-21T23:05:47.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.557+0000 7f0f32a00640 1 -- 172.21.15.28:0/637633037 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f18003520 con 0x7f0f340c4580 2024-02-21T23:05:47.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.558+0000 7f0f32a00640 1 -- 172.21.15.28:0/637633037 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2896863716 0 0) 0x7f0f18003520 con 0x7f0f340c4580 2024-02-21T23:05:47.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.558+0000 7f0f32a00640 1 -- 172.21.15.28:0/637633037 >> v2:172.21.15.81:3300/0 conn(0x7f0f34150240 msgr2=0x7f0f340c12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:47.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.558+0000 7f0f32a00640 1 --2- 172.21.15.28:0/637633037 >> v2:172.21.15.81:3300/0 conn(0x7f0f34150240 0x7f0f340c12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:47.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.558+0000 7f0f32a00640 1 -- 172.21.15.28:0/637633037 >> v1:172.21.15.81:6789/0 conn(0x7f0f340c2c10 legacy=0x7f0f3406ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:47.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.558+0000 7f0f32a00640 1 -- 172.21.15.28:0/637633037 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f3406f4a0 con 0x7f0f340c4580 2024-02-21T23:05:47.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.558+0000 7f0f32a00640 1 -- 172.21.15.28:0/637633037 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f0f20001670 con 0x7f0f340c4580 2024-02-21T23:05:47.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.558+0000 7f0f32a00640 1 -- 172.21.15.28:0/637633037 >> v1:172.21.15.28:6789/0 conn(0x7f0f340c4580 legacy=0x7f0f3406db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:47.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.558+0000 7f0f32a00640 1 --2- 172.21.15.28:0/637633037 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0f180039b0 0x7f0f18003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:47.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.558+0000 7f0f32a00640 1 --2- 172.21.15.28:0/637633037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f18004800 0x7f0f18004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:47.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.558+0000 7f0f32a00640 1 --2- 172.21.15.28:0/637633037 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f180051c0 0x7f0f1800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:47.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.558+0000 7f0f32a00640 1 -- 172.21.15.28:0/637633037 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1800c130 con 0x7f0f18004800 2024-02-21T23:05:47.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.558+0000 7f0f32a00640 1 -- 172.21.15.28:0/637633037 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1800c2e0 con 0x7f0f180051c0 2024-02-21T23:05:47.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.558+0000 7f0f32a00640 1 -- 172.21.15.28:0/637633037 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1800c4e0 con 0x7f0f180039b0 2024-02-21T23:05:47.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.558+0000 7f0f33400640 1 --2- 172.21.15.28:0/637633037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f18004800 0x7f0f18004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:47.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.558+0000 7f0f39000640 1 --2- 172.21.15.28:0/637633037 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f180051c0 0x7f0f1800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:47.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.558+0000 7f0f33e00640 1 --2- 172.21.15.28:0/637633037 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0f180039b0 0x7f0f18003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:47.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.559+0000 7f0f33400640 1 -- 172.21.15.28:0/637633037 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0f180039b0 msgr2=0x7f0f18003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:47.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.559+0000 7f0f33400640 1 --2- 172.21.15.28:0/637633037 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0f180039b0 0x7f0f18003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:47.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.559+0000 7f0f33400640 1 -- 172.21.15.28:0/637633037 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f180051c0 msgr2=0x7f0f1800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:47.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.559+0000 7f0f33400640 1 --2- 172.21.15.28:0/637633037 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f180051c0 0x7f0f1800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:47.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.559+0000 7f0f33400640 1 -- 172.21.15.28:0/637633037 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f18003520 con 0x7f0f18004800 2024-02-21T23:05:47.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.559+0000 7f0f39000640 1 -- 172.21.15.28:0/637633037 reap_dead start 2024-02-21T23:05:47.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.560+0000 7f0f33400640 1 --2- 172.21.15.28:0/637633037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f18004800 0x7f0f18004c80 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f0f20007cd0 tx=0x7f0f18005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d55494a2728589c9 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:47.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.560+0000 7f0f32a00640 1 -- 172.21.15.28:0/637633037 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f20007d10 con 0x7f0f18004800 2024-02-21T23:05:47.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.560+0000 7f0f32a00640 1 -- 172.21.15.28:0/637633037 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f20034070 con 0x7f0f18004800 2024-02-21T23:05:47.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.560+0000 7f0f32a00640 1 -- 172.21.15.28:0/637633037 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f20005af0 con 0x7f0f18004800 2024-02-21T23:05:47.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.561+0000 7f0f39a00640 1 -- 172.21.15.28:0/637633037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f18004800 msgr2=0x7f0f18004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:47.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.561+0000 7f0f39a00640 1 --2- 172.21.15.28:0/637633037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f18004800 0x7f0f18004c80 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f0f20007cd0 tx=0x7f0f18005830 comp rx=0 tx=0).stop 2024-02-21T23:05:47.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.561+0000 7f0f39a00640 1 -- 172.21.15.28:0/637633037 shutdown_connections 2024-02-21T23:05:47.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.561+0000 7f0f39a00640 1 --2- 172.21.15.28:0/637633037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f18004800 0x7f0f18004c80 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:47.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.561+0000 7f0f39a00640 1 -- 172.21.15.28:0/637633037 >> 172.21.15.28:0/637633037 conn(0x7f0f340a8530 msgr2=0x7f0f340a8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:47.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.561+0000 7f0f39a00640 1 -- 172.21.15.28:0/637633037 shutdown_connections 2024-02-21T23:05:47.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.561+0000 7f0f39a00640 1 -- 172.21.15.28:0/637633037 wait complete. 2024-02-21T23:05:47.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.563+0000 7f0f39a00640 1 Processor -- start 2024-02-21T23:05:47.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.563+0000 7f0f39a00640 1 -- start start 2024-02-21T23:05:47.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.563+0000 7f0f39a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f18004800 0x7f0f341e9950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:47.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.563+0000 7f0f39a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0f34149280 0x7f0f3414a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:47.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.564+0000 7f0f39a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f34155430 0x7f0f34147d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:47.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.564+0000 7f0f39a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f340c4a10 con 0x7f0f34155430 2024-02-21T23:05:47.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.564+0000 7f0f39a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f34153a90 con 0x7f0f18004800 2024-02-21T23:05:47.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.564+0000 7f0f39a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f340c4b90 con 0x7f0f34149280 2024-02-21T23:05:47.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.564+0000 7f0f39000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f34155430 0x7f0f34147d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:47.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.564+0000 7f0f39000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f34155430 0x7f0f34147d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47780/0 (socket says 172.21.15.28:47780) 2024-02-21T23:05:47.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.564+0000 7f0f39000640 1 -- 172.21.15.28:0/1736303502 learned_addr learned my addr 172.21.15.28:0/1736303502 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:47.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.564+0000 7f0f33e00640 1 --2- 172.21.15.28:0/1736303502 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f18004800 0x7f0f341e9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:47.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.564+0000 7f0f33400640 1 --2- 172.21.15.28:0/1736303502 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0f34149280 0x7f0f3414a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:47.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.564+0000 7f0f39000640 1 -- 172.21.15.28:0/1736303502 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0f34149280 msgr2=0x7f0f3414a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:47.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.564+0000 7f0f39000640 1 --2- 172.21.15.28:0/1736303502 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0f34149280 0x7f0f3414a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:47.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.564+0000 7f0f39000640 1 -- 172.21.15.28:0/1736303502 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f18004800 msgr2=0x7f0f341e9950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:47.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.564+0000 7f0f39000640 1 --2- 172.21.15.28:0/1736303502 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f18004800 0x7f0f341e9950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:47.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.564+0000 7f0f39000640 1 -- 172.21.15.28:0/1736303502 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f34148af0 con 0x7f0f34155430 2024-02-21T23:05:47.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.564+0000 7f0f33e00640 1 --2- 172.21.15.28:0/1736303502 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f18004800 0x7f0f341e9950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:05:47.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.565+0000 7f0f39000640 1 --2- 172.21.15.28:0/1736303502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f34155430 0x7f0f34147d90 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f0f3406bc80 tx=0x7f0f2c00e960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:47.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.565+0000 7f0f2be00640 1 -- 172.21.15.28:0/1736303502 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f2c019070 con 0x7f0f34155430 2024-02-21T23:05:47.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.565+0000 7f0f2be00640 1 -- 172.21.15.28:0/1736303502 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f2c006e20 con 0x7f0f34155430 2024-02-21T23:05:47.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.565+0000 7f0f2be00640 1 -- 172.21.15.28:0/1736303502 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f2c003b90 con 0x7f0f34155430 2024-02-21T23:05:47.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.565+0000 7f0f39a00640 1 -- 172.21.15.28:0/1736303502 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f34149760 con 0x7f0f34155430 2024-02-21T23:05:47.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.565+0000 7f0f39a00640 1 -- 172.21.15.28:0/1736303502 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f3414a570 con 0x7f0f34155430 2024-02-21T23:05:47.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.566+0000 7f0f39a00640 1 -- 172.21.15.28:0/1736303502 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f3404f290 con 0x7f0f34155430 2024-02-21T23:05:47.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.567+0000 7f0f2be00640 1 -- 172.21.15.28:0/1736303502 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0f2c005070 con 0x7f0f34155430 2024-02-21T23:05:47.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.568+0000 7f0f2be00640 1 --2- 172.21.15.28:0/1736303502 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0f1406b500 0x7f0f1406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:47.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.569+0000 7f0f2be00640 1 -- 172.21.15.28:0/1736303502 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f0f2c089500 con 0x7f0f34155430 2024-02-21T23:05:47.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.569+0000 7f0f33e00640 1 --2- 172.21.15.28:0/1736303502 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0f1406b500 0x7f0f1406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:47.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.570+0000 7f0f33e00640 1 --2- 172.21.15.28:0/1736303502 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0f1406b500 0x7f0f1406d9c0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f0f24009a80 tx=0x7f0f24008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:47.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.574+0000 7f0f2be00640 1 -- 172.21.15.28:0/1736303502 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f0f2c058530 con 0x7f0f34155430 2024-02-21T23:05:47.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.772+0000 7f0f39a00640 1 -- 172.21.15.28:0/1736303502 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0f3406f0e0 con 0x7f0f34155430 2024-02-21T23:05:47.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.772+0000 7f0f2be00640 1 -- 172.21.15.28:0/1736303502 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v72) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0f2c057e30 con 0x7f0f34155430 2024-02-21T23:05:47.761 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-02-21T23:05:47.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.775+0000 7f0f39a00640 1 -- 172.21.15.28:0/1736303502 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0f1406b500 msgr2=0x7f0f1406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:47.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.775+0000 7f0f39a00640 1 --2- 172.21.15.28:0/1736303502 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0f1406b500 0x7f0f1406d9c0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f0f24009a80 tx=0x7f0f24008040 comp rx=0 tx=0).stop 2024-02-21T23:05:47.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.775+0000 7f0f39a00640 1 -- 172.21.15.28:0/1736303502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f34155430 msgr2=0x7f0f34147d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:47.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.775+0000 7f0f39a00640 1 --2- 172.21.15.28:0/1736303502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f34155430 0x7f0f34147d90 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f0f3406bc80 tx=0x7f0f2c00e960 comp rx=0 tx=0).stop 2024-02-21T23:05:47.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.776+0000 7f0f39a00640 1 -- 172.21.15.28:0/1736303502 shutdown_connections 2024-02-21T23:05:47.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.776+0000 7f0f39a00640 1 --2- 172.21.15.28:0/1736303502 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0f1406b500 0x7f0f1406d9c0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:47.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.776+0000 7f0f39a00640 1 --2- 172.21.15.28:0/1736303502 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0f34149280 0x7f0f3414a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:47.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.776+0000 7f0f39a00640 1 --2- 172.21.15.28:0/1736303502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f34155430 0x7f0f34147d90 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:47.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.776+0000 7f0f39a00640 1 --2- 172.21.15.28:0/1736303502 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f18004800 0x7f0f341e9950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:47.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.776+0000 7f0f39a00640 1 -- 172.21.15.28:0/1736303502 >> 172.21.15.28:0/1736303502 conn(0x7f0f340a8530 msgr2=0x7f0f3406ae70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:47.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.776+0000 7f0f39a00640 1 -- 172.21.15.28:0/1736303502 shutdown_connections 2024-02-21T23:05:47.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:47.776+0000 7f0f39a00640 1 -- 172.21.15.28:0/1736303502 wait complete. 2024-02-21T23:05:47.782 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-02-21T23:05:48.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.013+0000 7f9f1be00640 1 Processor -- start 2024-02-21T23:05:48.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.013+0000 7f9f1be00640 1 -- start start 2024-02-21T23:05:48.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.015+0000 7f9f1be00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f9f1c0b0130 0x7f9f1c0c8740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:48.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.015+0000 7f9f1be00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f1c1550b0 con 0x7f9f1c158c20 2024-02-21T23:05:48.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.015+0000 7f9f1be00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f1c155350 con 0x7f9f1c0c51c0 2024-02-21T23:05:48.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.015+0000 7f9f1be00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f1c152a40 con 0x7f9f1c0b0130 2024-02-21T23:05:48.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.015+0000 7f9f1b400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f9f1c158c20 0x7f9f1c15f880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38238/0 (socket says 172.21.15.28:38238) 2024-02-21T23:05:48.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.015+0000 7f9f1b400640 1 -- 172.21.15.28:0/275548457 learned_addr learned my addr 172.21.15.28:0/275548457 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:05:48.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.016+0000 7f9f1aa00640 1 --2- 172.21.15.28:0/275548457 >> v2:172.21.15.81:3300/0 conn(0x7f9f1c0b0130 0x7f9f1c0c8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:48.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.016+0000 7f9f19600640 1 -- 172.21.15.28:0/275548457 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 571601151 0 0) 0x7f9f1c1550b0 con 0x7f9f1c158c20 2024-02-21T23:05:48.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.016+0000 7f9f19600640 1 -- 172.21.15.28:0/275548457 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f00003520 con 0x7f9f1c158c20 2024-02-21T23:05:48.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.016+0000 7f9f19600640 1 -- 172.21.15.28:0/275548457 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 163983741 0 0) 0x7f9f00003520 con 0x7f9f1c158c20 2024-02-21T23:05:48.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.016+0000 7f9f19600640 1 -- 172.21.15.28:0/275548457 >> v2:172.21.15.81:3300/0 conn(0x7f9f1c0b0130 msgr2=0x7f9f1c0c8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:48.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.016+0000 7f9f19600640 1 --2- 172.21.15.28:0/275548457 >> v2:172.21.15.81:3300/0 conn(0x7f9f1c0b0130 0x7f9f1c0c8740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:48.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.016+0000 7f9f19600640 1 -- 172.21.15.28:0/275548457 >> v1:172.21.15.81:6789/0 conn(0x7f9f1c0c51c0 legacy=0x7f9f1c158510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:48.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.016+0000 7f9f19600640 1 -- 172.21.15.28:0/275548457 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f1c155630 con 0x7f9f1c158c20 2024-02-21T23:05:48.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.016+0000 7f9f19600640 1 -- 172.21.15.28:0/275548457 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f9f14001a10 con 0x7f9f1c158c20 2024-02-21T23:05:48.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.016+0000 7f9f19600640 1 -- 172.21.15.28:0/275548457 >> v1:172.21.15.28:6789/0 conn(0x7f9f1c158c20 legacy=0x7f9f1c15f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:48.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.017+0000 7f9f19600640 1 --2- 172.21.15.28:0/275548457 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f00003a00 0x7f9f00003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:48.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.017+0000 7f9f19600640 1 --2- 172.21.15.28:0/275548457 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9f00004800 0x7f9f00004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:48.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.017+0000 7f9f19600640 1 --2- 172.21.15.28:0/275548457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f00005210 0x7f9f0000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:48.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.017+0000 7f9f19600640 1 -- 172.21.15.28:0/275548457 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f1c152a40 con 0x7f9f00005210 2024-02-21T23:05:48.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.017+0000 7f9f19600640 1 -- 172.21.15.28:0/275548457 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f0000c160 con 0x7f9f00003a00 2024-02-21T23:05:48.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.017+0000 7f9f19600640 1 -- 172.21.15.28:0/275548457 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f0000c360 con 0x7f9f00004800 2024-02-21T23:05:48.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.017+0000 7f9f1b400640 1 --2- 172.21.15.28:0/275548457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f00005210 0x7f9f0000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:48.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.017+0000 7f9f1aa00640 1 --2- 172.21.15.28:0/275548457 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f00003a00 0x7f9f00003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:48.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.017+0000 7f9f1a000640 1 --2- 172.21.15.28:0/275548457 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9f00004800 0x7f9f00004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:48.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.017+0000 7f9f1b400640 1 -- 172.21.15.28:0/275548457 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9f00004800 msgr2=0x7f9f00004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:48.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.017+0000 7f9f1b400640 1 --2- 172.21.15.28:0/275548457 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9f00004800 0x7f9f00004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:48.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.017+0000 7f9f1b400640 1 -- 172.21.15.28:0/275548457 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f00003a00 msgr2=0x7f9f00003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:48.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.017+0000 7f9f1b400640 1 --2- 172.21.15.28:0/275548457 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f00003a00 0x7f9f00003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:48.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.017+0000 7f9f1b400640 1 -- 172.21.15.28:0/275548457 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f00003520 con 0x7f9f00005210 2024-02-21T23:05:48.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.017+0000 7f9f1aa00640 1 --2- 172.21.15.28:0/275548457 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f00003a00 0x7f9f00003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:05:48.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.018+0000 7f9f1b400640 1 -- 172.21.15.28:0/275548457 reap_dead start 2024-02-21T23:05:48.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.019+0000 7f9f1b400640 1 --2- 172.21.15.28:0/275548457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f00005210 0x7f9f0000b9c0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f9f1c155280 tx=0x7f9f00005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fce4ea66be224c51 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:48.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.019+0000 7f9f19600640 1 -- 172.21.15.28:0/275548457 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f1400a3d0 con 0x7f9f00005210 2024-02-21T23:05:48.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.019+0000 7f9f19600640 1 -- 172.21.15.28:0/275548457 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f14034650 con 0x7f9f00005210 2024-02-21T23:05:48.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.019+0000 7f9f19600640 1 -- 172.21.15.28:0/275548457 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f14034a40 con 0x7f9f00005210 2024-02-21T23:05:48.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.019+0000 7f9f1be00640 1 -- 172.21.15.28:0/275548457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f00005210 msgr2=0x7f9f0000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:48.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.019+0000 7f9f1be00640 1 --2- 172.21.15.28:0/275548457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f00005210 0x7f9f0000b9c0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f9f1c155280 tx=0x7f9f00005830 comp rx=0 tx=0).stop 2024-02-21T23:05:48.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.020+0000 7f9f1be00640 1 -- 172.21.15.28:0/275548457 shutdown_connections 2024-02-21T23:05:48.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.020+0000 7f9f1be00640 1 --2- 172.21.15.28:0/275548457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f00005210 0x7f9f0000b9c0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:48.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.020+0000 7f9f1be00640 1 -- 172.21.15.28:0/275548457 >> 172.21.15.28:0/275548457 conn(0x7f9f1c0ac370 msgr2=0x7f9f1c0ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:48.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.020+0000 7f9f1be00640 1 -- 172.21.15.28:0/275548457 shutdown_connections 2024-02-21T23:05:48.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.020+0000 7f9f1be00640 1 -- 172.21.15.28:0/275548457 wait complete. 2024-02-21T23:05:48.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.021+0000 7f9f1be00640 1 Processor -- start 2024-02-21T23:05:48.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.022+0000 7f9f1be00640 1 -- start start 2024-02-21T23:05:48.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.022+0000 7f9f1be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f00005210 0x7f9f1c0c1910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:48.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.022+0000 7f9f1be00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9f1c1eae90 0x7f9f1c0adab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:48.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.022+0000 7f9f1be00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f1c0adff0 0x7f9f1c14ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:48.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.022+0000 7f9f1be00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f1c0c8dd0 con 0x7f9f00005210 2024-02-21T23:05:48.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.022+0000 7f9f1be00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f1c155e30 con 0x7f9f1c0adff0 2024-02-21T23:05:48.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.022+0000 7f9f1be00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f1c0be060 con 0x7f9f1c1eae90 2024-02-21T23:05:48.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.023+0000 7f9f1aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f00005210 0x7f9f1c0c1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:48.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.023+0000 7f9f1aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f00005210 0x7f9f1c0c1910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47788/0 (socket says 172.21.15.28:47788) 2024-02-21T23:05:48.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.023+0000 7f9f1aa00640 1 -- 172.21.15.28:0/552132027 learned_addr learned my addr 172.21.15.28:0/552132027 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:48.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.023+0000 7f9f1a000640 1 --2- 172.21.15.28:0/552132027 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9f1c1eae90 0x7f9f1c0adab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:48.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.023+0000 7f9f1aa00640 1 -- 172.21.15.28:0/552132027 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9f1c1eae90 msgr2=0x7f9f1c0adab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:48.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.023+0000 7f9f1aa00640 1 --2- 172.21.15.28:0/552132027 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9f1c1eae90 0x7f9f1c0adab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:48.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.023+0000 7f9f1aa00640 1 -- 172.21.15.28:0/552132027 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f1c0adff0 msgr2=0x7f9f1c14ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:48.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.023+0000 7f9f1b400640 1 --2- 172.21.15.28:0/552132027 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f1c0adff0 0x7f9f1c14ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:48.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.023+0000 7f9f1aa00640 1 --2- 172.21.15.28:0/552132027 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f1c0adff0 0x7f9f1c14ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:48.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.023+0000 7f9f1aa00640 1 -- 172.21.15.28:0/552132027 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f1c0ad320 con 0x7f9f00005210 2024-02-21T23:05:48.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.023+0000 7f9f1b400640 1 --2- 172.21.15.28:0/552132027 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f1c0adff0 0x7f9f1c14ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:05:48.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.024+0000 7f9f1aa00640 1 --2- 172.21.15.28:0/552132027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f00005210 0x7f9f1c0c1910 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c011540 tx=0x7f9f0c011a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:48.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.024+0000 7f9f12a00640 1 -- 172.21.15.28:0/552132027 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f0c015870 con 0x7f9f00005210 2024-02-21T23:05:48.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.024+0000 7f9f12a00640 1 -- 172.21.15.28:0/552132027 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f0c015a10 con 0x7f9f00005210 2024-02-21T23:05:48.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.024+0000 7f9f12a00640 1 -- 172.21.15.28:0/552132027 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f0c015dc0 con 0x7f9f00005210 2024-02-21T23:05:48.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.024+0000 7f9f1be00640 1 -- 172.21.15.28:0/552132027 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f1c0ae790 con 0x7f9f00005210 2024-02-21T23:05:48.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.024+0000 7f9f1be00640 1 -- 172.21.15.28:0/552132027 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f1c0aed50 con 0x7f9f00005210 2024-02-21T23:05:48.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.025+0000 7f9f1be00640 1 -- 172.21.15.28:0/552132027 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f1c04f290 con 0x7f9f00005210 2024-02-21T23:05:48.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.026+0000 7f9f12a00640 1 -- 172.21.15.28:0/552132027 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9f0c01d400 con 0x7f9f00005210 2024-02-21T23:05:48.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.027+0000 7f9f12a00640 1 --2- 172.21.15.28:0/552132027 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9efc06b500 0x7f9efc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:48.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.027+0000 7f9f12a00640 1 -- 172.21.15.28:0/552132027 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f9f0c08a8a0 con 0x7f9f00005210 2024-02-21T23:05:48.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.028+0000 7f9f1a000640 1 --2- 172.21.15.28:0/552132027 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9efc06b500 0x7f9efc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:48.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.029+0000 7f9f1a000640 1 --2- 172.21.15.28:0/552132027 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9efc06b500 0x7f9efc06d9c0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f9f1c0c9950 tx=0x7f9f08009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:48.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.032+0000 7f9f12a00640 1 -- 172.21.15.28:0/552132027 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f9f0c019030 con 0x7f9f00005210 2024-02-21T23:05:48.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.241+0000 7f9f1be00640 1 -- 172.21.15.28:0/552132027 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f9f1c154380 con 0x7f9f00005210 2024-02-21T23:05:48.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.242+0000 7f9f12a00640 1 -- 172.21.15.28:0/552132027 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v72) v1 ==== 84+0+65 (secure 0 0 0) 0x7f9f0c05f090 con 0x7f9f00005210 2024-02-21T23:05:48.231 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:05:48.231 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-02-21T23:05:48.231 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:05:48.231 INFO:teuthology.orchestra.run.smithi028.stdout: "poolnum": 1, 2024-02-21T23:05:48.231 INFO:teuthology.orchestra.run.smithi028.stdout: "poolname": ".mgr" 2024-02-21T23:05:48.231 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:05:48.231 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-02-21T23:05:48.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.245+0000 7f9f1be00640 1 -- 172.21.15.28:0/552132027 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9efc06b500 msgr2=0x7f9efc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:48.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.245+0000 7f9f1be00640 1 --2- 172.21.15.28:0/552132027 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9efc06b500 0x7f9efc06d9c0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f9f1c0c9950 tx=0x7f9f08009040 comp rx=0 tx=0).stop 2024-02-21T23:05:48.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.245+0000 7f9f1be00640 1 -- 172.21.15.28:0/552132027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f00005210 msgr2=0x7f9f1c0c1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:48.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.245+0000 7f9f1be00640 1 --2- 172.21.15.28:0/552132027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f00005210 0x7f9f1c0c1910 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c011540 tx=0x7f9f0c011a00 comp rx=0 tx=0).stop 2024-02-21T23:05:48.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.245+0000 7f9f1be00640 1 -- 172.21.15.28:0/552132027 shutdown_connections 2024-02-21T23:05:48.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.245+0000 7f9f1be00640 1 --2- 172.21.15.28:0/552132027 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9efc06b500 0x7f9efc06d9c0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:48.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.245+0000 7f9f1be00640 1 --2- 172.21.15.28:0/552132027 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9f1c1eae90 0x7f9f1c0adab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:48.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.245+0000 7f9f1be00640 1 --2- 172.21.15.28:0/552132027 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f1c0adff0 0x7f9f1c14ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:48.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.245+0000 7f9f1be00640 1 --2- 172.21.15.28:0/552132027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f00005210 0x7f9f1c0c1910 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:48.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.245+0000 7f9f1be00640 1 -- 172.21.15.28:0/552132027 >> 172.21.15.28:0/552132027 conn(0x7f9f1c0ac370 msgr2=0x7f9f1c0c8620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:48.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.245+0000 7f9f1be00640 1 -- 172.21.15.28:0/552132027 shutdown_connections 2024-02-21T23:05:48.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.245+0000 7f9f1be00640 1 -- 172.21.15.28:0/552132027 wait complete. 2024-02-21T23:05:48.254 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-02-21T23:05:48.255 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-02-21T23:05:48.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.497+0000 7fb231e00640 1 Processor -- start 2024-02-21T23:05:48.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.497+0000 7fb231e00640 1 -- start start 2024-02-21T23:05:48.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.498+0000 7fb231e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb22c0c49e0 0x7fb22c072590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:48.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.498+0000 7fb231e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb22c072ad0 0x7fb22c15c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:48.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.498+0000 7fb231e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb22c14f570 con 0x7fb22c14f190 2024-02-21T23:05:48.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.499+0000 7fb231e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb22c0c31d0 con 0x7fb22c072ad0 2024-02-21T23:05:48.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.499+0000 7fb231e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb22c0c34b0 con 0x7fb22c0c49e0 2024-02-21T23:05:48.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.499+0000 7fb231400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb22c072ad0 0x7fb22c15c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:48.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.499+0000 7fb230a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb22c14f190 0x7fb22c06f980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38250/0 (socket says 172.21.15.28:38250) 2024-02-21T23:05:48.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.499+0000 7fb230a00640 1 -- 172.21.15.28:0/4176568644 learned_addr learned my addr 172.21.15.28:0/4176568644 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:05:48.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.499+0000 7fb22be00640 1 --2- 172.21.15.28:0/4176568644 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb22c0c49e0 0x7fb22c072590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:48.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.499+0000 7fb231400640 1 -- 172.21.15.28:0/4176568644 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb22c0c49e0 msgr2=0x7fb22c072590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:48.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.499+0000 7fb231400640 1 --2- 172.21.15.28:0/4176568644 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb22c0c49e0 0x7fb22c072590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:48.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.499+0000 7fb231400640 1 -- 172.21.15.28:0/4176568644 >> v1:172.21.15.28:6789/0 conn(0x7fb22c14f190 legacy=0x7fb22c06f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:48.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.500+0000 7fb231400640 1 -- 172.21.15.28:0/4176568644 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb22c0c3790 con 0x7fb22c072ad0 2024-02-21T23:05:48.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.500+0000 7fb22b400640 1 -- 172.21.15.28:0/4176568644 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3433965106 0 0) 0x7fb21c001490 con 0x7fb22c14f190 2024-02-21T23:05:48.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.501+0000 7fb231400640 1 --2- 172.21.15.28:0/4176568644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb22c072ad0 0x7fb22c15c470 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fb2240097a0 tx=0x7fb22402b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bd3df59568ab0c0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:48.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.501+0000 7fb22b400640 1 -- 172.21.15.28:0/4176568644 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb224038070 con 0x7fb22c072ad0 2024-02-21T23:05:48.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.501+0000 7fb22b400640 1 -- 172.21.15.28:0/4176568644 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb224031700 con 0x7fb22c072ad0 2024-02-21T23:05:48.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.501+0000 7fb22b400640 1 -- 172.21.15.28:0/4176568644 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb224031ad0 con 0x7fb22c072ad0 2024-02-21T23:05:48.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.501+0000 7fb231e00640 1 -- 172.21.15.28:0/4176568644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb22c072ad0 msgr2=0x7fb22c15c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:48.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.501+0000 7fb231e00640 1 --2- 172.21.15.28:0/4176568644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb22c072ad0 0x7fb22c15c470 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fb2240097a0 tx=0x7fb22402b7a0 comp rx=0 tx=0).stop 2024-02-21T23:05:48.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.502+0000 7fb231e00640 1 -- 172.21.15.28:0/4176568644 shutdown_connections 2024-02-21T23:05:48.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.502+0000 7fb231e00640 1 --2- 172.21.15.28:0/4176568644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb22c072ad0 0x7fb22c15c470 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:48.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.502+0000 7fb231e00640 1 --2- 172.21.15.28:0/4176568644 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb22c0c49e0 0x7fb22c072590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:48.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.502+0000 7fb231e00640 1 -- 172.21.15.28:0/4176568644 >> 172.21.15.28:0/4176568644 conn(0x7fb22c066e40 msgr2=0x7fb22c0a84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:48.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.502+0000 7fb231e00640 1 -- 172.21.15.28:0/4176568644 shutdown_connections 2024-02-21T23:05:48.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.502+0000 7fb231e00640 1 -- 172.21.15.28:0/4176568644 wait complete. 2024-02-21T23:05:48.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.503+0000 7fb231e00640 1 Processor -- start 2024-02-21T23:05:48.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.503+0000 7fb231e00640 1 -- start start 2024-02-21T23:05:48.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.504+0000 7fb231e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb22c072ad0 0x7fb22c1495a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:48.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.504+0000 7fb231e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb22c0c49e0 0x7fb22c0c5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:48.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.504+0000 7fb231e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb22c14f190 0x7fb22c14b150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:48.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.504+0000 7fb231e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb22c070210 con 0x7fb22c0c49e0 2024-02-21T23:05:48.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.504+0000 7fb231e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb22c070090 con 0x7fb22c14f190 2024-02-21T23:05:48.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.504+0000 7fb231e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb22c0c4f30 con 0x7fb22c072ad0 2024-02-21T23:05:48.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.505+0000 7fb22be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb22c0c49e0 0x7fb22c0c5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:48.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.505+0000 7fb22be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb22c0c49e0 0x7fb22c0c5280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47808/0 (socket says 172.21.15.28:47808) 2024-02-21T23:05:48.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.505+0000 7fb22be00640 1 -- 172.21.15.28:0/3929053006 learned_addr learned my addr 172.21.15.28:0/3929053006 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:48.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.505+0000 7fb231400640 1 --2- 172.21.15.28:0/3929053006 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb22c14f190 0x7fb22c14b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:48.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.505+0000 7fb230a00640 1 --2- 172.21.15.28:0/3929053006 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb22c072ad0 0x7fb22c1495a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:48.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.505+0000 7fb22be00640 1 -- 172.21.15.28:0/3929053006 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb22c072ad0 msgr2=0x7fb22c1495a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:48.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.505+0000 7fb22be00640 1 --2- 172.21.15.28:0/3929053006 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb22c072ad0 0x7fb22c1495a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:48.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.505+0000 7fb22be00640 1 -- 172.21.15.28:0/3929053006 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb22c14f190 msgr2=0x7fb22c14b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:48.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.505+0000 7fb22be00640 1 --2- 172.21.15.28:0/3929053006 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb22c14f190 0x7fb22c14b150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:48.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.505+0000 7fb22be00640 1 -- 172.21.15.28:0/3929053006 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb22c1e97d0 con 0x7fb22c0c49e0 2024-02-21T23:05:48.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.505+0000 7fb230a00640 1 --2- 172.21.15.28:0/3929053006 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb22c072ad0 0x7fb22c1495a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:05:48.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.506+0000 7fb22be00640 1 --2- 172.21.15.28:0/3929053006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb22c0c49e0 0x7fb22c0c5280 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fb21400d620 tx=0x7fb21400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:48.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.506+0000 7fb228c00640 1 -- 172.21.15.28:0/3929053006 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb21400b620 con 0x7fb22c0c49e0 2024-02-21T23:05:48.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.506+0000 7fb228c00640 1 -- 172.21.15.28:0/3929053006 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb214019070 con 0x7fb22c0c49e0 2024-02-21T23:05:48.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.506+0000 7fb228c00640 1 -- 172.21.15.28:0/3929053006 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb21400ba00 con 0x7fb22c0c49e0 2024-02-21T23:05:48.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.506+0000 7fb231e00640 1 -- 172.21.15.28:0/3929053006 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb22c1e9a50 con 0x7fb22c0c49e0 2024-02-21T23:05:48.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.506+0000 7fb231e00640 1 -- 172.21.15.28:0/3929053006 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb22c148e10 con 0x7fb22c0c49e0 2024-02-21T23:05:48.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.507+0000 7fb231e00640 1 -- 172.21.15.28:0/3929053006 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb22c04f290 con 0x7fb22c0c49e0 2024-02-21T23:05:48.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.508+0000 7fb228c00640 1 -- 172.21.15.28:0/3929053006 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb21400bba0 con 0x7fb22c0c49e0 2024-02-21T23:05:48.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.509+0000 7fb228c00640 1 --2- 172.21.15.28:0/3929053006 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb20806b4d0 0x7fb20806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:48.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.509+0000 7fb228c00640 1 -- 172.21.15.28:0/3929053006 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fb21408a410 con 0x7fb22c0c49e0 2024-02-21T23:05:48.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.510+0000 7fb230a00640 1 --2- 172.21.15.28:0/3929053006 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb20806b4d0 0x7fb20806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:48.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.511+0000 7fb230a00640 1 --2- 172.21.15.28:0/3929053006 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb20806b4d0 0x7fb20806d990 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fb21c007fd0 tx=0x7fb21c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:48.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.514+0000 7fb228c00640 1 -- 172.21.15.28:0/3929053006 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb22c1e97d0 con 0x7fb22c0c49e0 2024-02-21T23:05:48.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:48.718+0000 7fb231e00640 1 -- 172.21.15.28:0/3929053006 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fb22c0c47c0 con 0x7fb22c0c49e0 2024-02-21T23:05:49.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.333+0000 7fb228c00640 1 -- 172.21.15.28:0/3929053006 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v73) v1 ==== 138+0+0 (secure 0 0 0) 0x7fb214059440 con 0x7fb22c0c49e0 2024-02-21T23:05:49.321 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' created 2024-02-21T23:05:49.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.337+0000 7fb231e00640 1 -- 172.21.15.28:0/3929053006 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb20806b4d0 msgr2=0x7fb20806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:49.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.337+0000 7fb231e00640 1 --2- 172.21.15.28:0/3929053006 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb20806b4d0 0x7fb20806d990 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fb21c007fd0 tx=0x7fb21c009040 comp rx=0 tx=0).stop 2024-02-21T23:05:49.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.337+0000 7fb231e00640 1 -- 172.21.15.28:0/3929053006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb22c0c49e0 msgr2=0x7fb22c0c5280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:49.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.337+0000 7fb231e00640 1 --2- 172.21.15.28:0/3929053006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb22c0c49e0 0x7fb22c0c5280 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fb21400d620 tx=0x7fb21400dae0 comp rx=0 tx=0).stop 2024-02-21T23:05:49.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.337+0000 7fb231e00640 1 -- 172.21.15.28:0/3929053006 shutdown_connections 2024-02-21T23:05:49.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.337+0000 7fb231e00640 1 --2- 172.21.15.28:0/3929053006 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb20806b4d0 0x7fb20806d990 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:49.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.337+0000 7fb231e00640 1 --2- 172.21.15.28:0/3929053006 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb22c14f190 0x7fb22c14b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:49.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.337+0000 7fb231e00640 1 --2- 172.21.15.28:0/3929053006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb22c0c49e0 0x7fb22c0c5280 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:49.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.337+0000 7fb231e00640 1 --2- 172.21.15.28:0/3929053006 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb22c072ad0 0x7fb22c1495a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:49.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.337+0000 7fb231e00640 1 -- 172.21.15.28:0/3929053006 >> 172.21.15.28:0/3929053006 conn(0x7fb22c066e40 msgr2=0x7fb22c0a84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:49.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.338+0000 7fb231e00640 1 -- 172.21.15.28:0/3929053006 shutdown_connections 2024-02-21T23:05:49.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.338+0000 7fb231e00640 1 -- 172.21.15.28:0/3929053006 wait complete. 2024-02-21T23:05:49.354 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-02-21T23:05:49.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.583+0000 7f0238a00640 1 Processor -- start 2024-02-21T23:05:49.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.584+0000 7f0238a00640 1 -- start start 2024-02-21T23:05:49.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.585+0000 7f0238a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f02340bde20 0x7f0234155940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:49.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.585+0000 7f0238a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0234154ea0 con 0x7f02340c5110 2024-02-21T23:05:49.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.585+0000 7f0238a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02340c5690 con 0x7f0234152370 2024-02-21T23:05:49.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.585+0000 7f0238a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02341554b0 con 0x7f02340bde20 2024-02-21T23:05:49.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.585+0000 7f0233200640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f02340c5110 0x7f02340cab00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38260/0 (socket says 172.21.15.28:38260) 2024-02-21T23:05:49.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.585+0000 7f0233200640 1 -- 172.21.15.28:0/3936824509 learned_addr learned my addr 172.21.15.28:0/3936824509 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:05:49.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.586+0000 7f0231e00640 1 --2- 172.21.15.28:0/3936824509 >> v2:172.21.15.81:3300/0 conn(0x7f02340bde20 0x7f0234155940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:49.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.586+0000 7f0231400640 1 -- 172.21.15.28:0/3936824509 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 511846699 0 0) 0x7f0234154ea0 con 0x7f02340c5110 2024-02-21T23:05:49.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.586+0000 7f0231400640 1 -- 172.21.15.28:0/3936824509 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0218003520 con 0x7f02340c5110 2024-02-21T23:05:49.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.586+0000 7f0231400640 1 -- 172.21.15.28:0/3936824509 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4204925281 0 0) 0x7f02340c5690 con 0x7f0234152370 2024-02-21T23:05:49.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.586+0000 7f0231400640 1 -- 172.21.15.28:0/3936824509 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0234154ea0 con 0x7f0234152370 2024-02-21T23:05:49.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.586+0000 7f0231400640 1 -- 172.21.15.28:0/3936824509 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 156704879 0 0) 0x7f0218003520 con 0x7f02340c5110 2024-02-21T23:05:49.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.586+0000 7f0231400640 1 -- 172.21.15.28:0/3936824509 >> v2:172.21.15.81:3300/0 conn(0x7f02340bde20 msgr2=0x7f0234155940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:49.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.586+0000 7f0231400640 1 --2- 172.21.15.28:0/3936824509 >> v2:172.21.15.81:3300/0 conn(0x7f02340bde20 0x7f0234155940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:49.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.586+0000 7f0231400640 1 -- 172.21.15.28:0/3936824509 >> v1:172.21.15.81:6789/0 conn(0x7f0234152370 legacy=0x7f02340c6f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:49.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.586+0000 7f0231400640 1 -- 172.21.15.28:0/3936824509 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0234155140 con 0x7f02340c5110 2024-02-21T23:05:49.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.586+0000 7f0231400640 1 -- 172.21.15.28:0/3936824509 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f022c001670 con 0x7f02340c5110 2024-02-21T23:05:49.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.586+0000 7f0231400640 1 -- 172.21.15.28:0/3936824509 >> v1:172.21.15.28:6789/0 conn(0x7f02340c5110 legacy=0x7f02340cab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:49.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.586+0000 7f0231400640 1 --2- 172.21.15.28:0/3936824509 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0218003b40 0x7f0218003f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:49.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.586+0000 7f0231400640 1 --2- 172.21.15.28:0/3936824509 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0218004800 0x7f0218004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:49.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.586+0000 7f0231400640 1 --2- 172.21.15.28:0/3936824509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f02180051c0 0x7f021800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:49.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.586+0000 7f0231400640 1 -- 172.21.15.28:0/3936824509 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02341554b0 con 0x7f02180051c0 2024-02-21T23:05:49.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.586+0000 7f0231400640 1 -- 172.21.15.28:0/3936824509 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f021800c160 con 0x7f0218004800 2024-02-21T23:05:49.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.586+0000 7f0231400640 1 -- 172.21.15.28:0/3936824509 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f021800c360 con 0x7f0218003b40 2024-02-21T23:05:49.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.587+0000 7f0233200640 1 --2- 172.21.15.28:0/3936824509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f02180051c0 0x7f021800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:49.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.587+0000 7f0231e00640 1 --2- 172.21.15.28:0/3936824509 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0218004800 0x7f0218004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:49.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.587+0000 7f0232800640 1 --2- 172.21.15.28:0/3936824509 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0218003b40 0x7f0218003f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:49.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.587+0000 7f0233200640 1 -- 172.21.15.28:0/3936824509 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0218003b40 msgr2=0x7f0218003f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:49.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.587+0000 7f0233200640 1 --2- 172.21.15.28:0/3936824509 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0218003b40 0x7f0218003f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:49.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.587+0000 7f0233200640 1 -- 172.21.15.28:0/3936824509 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0218004800 msgr2=0x7f0218004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:49.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.587+0000 7f0233200640 1 --2- 172.21.15.28:0/3936824509 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0218004800 0x7f0218004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:49.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.587+0000 7f0233200640 1 -- 172.21.15.28:0/3936824509 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0218003520 con 0x7f02180051c0 2024-02-21T23:05:49.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.587+0000 7f0231e00640 1 --2- 172.21.15.28:0/3936824509 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0218004800 0x7f0218004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:05:49.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.587+0000 7f0232800640 1 --2- 172.21.15.28:0/3936824509 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0218003b40 0x7f0218003f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:05:49.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.588+0000 7f0233200640 1 -- 172.21.15.28:0/3936824509 reap_dead start 2024-02-21T23:05:49.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.588+0000 7f0233200640 1 --2- 172.21.15.28:0/3936824509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f02180051c0 0x7f021800b9c0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f022c00c210 tx=0x7f0218005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cadfba1d20ec03f8 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:49.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.589+0000 7f0231400640 1 -- 172.21.15.28:0/3936824509 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f022c007ce0 con 0x7f02180051c0 2024-02-21T23:05:49.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.589+0000 7f0231400640 1 -- 172.21.15.28:0/3936824509 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f022c030070 con 0x7f02180051c0 2024-02-21T23:05:49.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.589+0000 7f0231400640 1 -- 172.21.15.28:0/3936824509 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f022c005d30 con 0x7f02180051c0 2024-02-21T23:05:49.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.589+0000 7f0238a00640 1 -- 172.21.15.28:0/3936824509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f02180051c0 msgr2=0x7f021800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:49.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.589+0000 7f0238a00640 1 --2- 172.21.15.28:0/3936824509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f02180051c0 0x7f021800b9c0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f022c00c210 tx=0x7f0218005830 comp rx=0 tx=0).stop 2024-02-21T23:05:49.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.589+0000 7f0238a00640 1 -- 172.21.15.28:0/3936824509 shutdown_connections 2024-02-21T23:05:49.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.589+0000 7f0238a00640 1 --2- 172.21.15.28:0/3936824509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f02180051c0 0x7f021800b9c0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:49.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.589+0000 7f0238a00640 1 -- 172.21.15.28:0/3936824509 >> 172.21.15.28:0/3936824509 conn(0x7f0234072780 msgr2=0x7f0234072b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:49.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.590+0000 7f0238a00640 1 -- 172.21.15.28:0/3936824509 shutdown_connections 2024-02-21T23:05:49.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.590+0000 7f0238a00640 1 -- 172.21.15.28:0/3936824509 wait complete. 2024-02-21T23:05:49.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.591+0000 7f0238a00640 1 Processor -- start 2024-02-21T23:05:49.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.591+0000 7f0238a00640 1 -- start start 2024-02-21T23:05:49.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.592+0000 7f0238a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f02180051c0 0x7f02340ab310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:49.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.592+0000 7f0238a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f02341eaa40 0x7f0234149540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:49.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.592+0000 7f0238a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0234158dc0 0x7f0234149a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:49.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.592+0000 7f0238a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02340c5950 con 0x7f0234158dc0 2024-02-21T23:05:49.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.592+0000 7f0238a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f023406dd00 con 0x7f02341eaa40 2024-02-21T23:05:49.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.592+0000 7f0238a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02340c5ad0 con 0x7f02180051c0 2024-02-21T23:05:49.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.593+0000 7f0233200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0234158dc0 0x7f0234149a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:49.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.593+0000 7f0233200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0234158dc0 0x7f0234149a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47832/0 (socket says 172.21.15.28:47832) 2024-02-21T23:05:49.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.593+0000 7f0233200640 1 -- 172.21.15.28:0/604371748 learned_addr learned my addr 172.21.15.28:0/604371748 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:49.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.593+0000 7f0232800640 1 --2- 172.21.15.28:0/604371748 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f02180051c0 0x7f02340ab310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:49.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.593+0000 7f0231e00640 1 --2- 172.21.15.28:0/604371748 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f02341eaa40 0x7f0234149540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:49.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.593+0000 7f0233200640 1 -- 172.21.15.28:0/604371748 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f02180051c0 msgr2=0x7f02340ab310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:49.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.593+0000 7f0233200640 1 --2- 172.21.15.28:0/604371748 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f02180051c0 0x7f02340ab310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:49.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.593+0000 7f0233200640 1 -- 172.21.15.28:0/604371748 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f02341eaa40 msgr2=0x7f0234149540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:49.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.593+0000 7f0233200640 1 --2- 172.21.15.28:0/604371748 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f02341eaa40 0x7f0234149540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:49.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.593+0000 7f0233200640 1 -- 172.21.15.28:0/604371748 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02341ee0e0 con 0x7f0234158dc0 2024-02-21T23:05:49.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.593+0000 7f0231e00640 1 --2- 172.21.15.28:0/604371748 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f02341eaa40 0x7f0234149540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:05:49.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.593+0000 7f0232800640 1 --2- 172.21.15.28:0/604371748 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f02180051c0 0x7f02340ab310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:05:49.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.593+0000 7f0233200640 1 --2- 172.21.15.28:0/604371748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0234158dc0 0x7f0234149a80 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f022c001eb0 tx=0x7f0234155420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:49.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.594+0000 7f022aa00640 1 -- 172.21.15.28:0/604371748 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f022c0049e0 con 0x7f0234158dc0 2024-02-21T23:05:49.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.594+0000 7f022aa00640 1 -- 172.21.15.28:0/604371748 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f022c004b80 con 0x7f0234158dc0 2024-02-21T23:05:49.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.594+0000 7f022aa00640 1 -- 172.21.15.28:0/604371748 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f022c007070 con 0x7f0234158dc0 2024-02-21T23:05:49.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.594+0000 7f0238a00640 1 -- 172.21.15.28:0/604371748 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02340be460 con 0x7f0234158dc0 2024-02-21T23:05:49.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.594+0000 7f0238a00640 1 -- 172.21.15.28:0/604371748 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02340bea20 con 0x7f0234158dc0 2024-02-21T23:05:49.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.595+0000 7f0238a00640 1 -- 172.21.15.28:0/604371748 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02341ee2b0 con 0x7f0234158dc0 2024-02-21T23:05:49.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.596+0000 7f022aa00640 1 -- 172.21.15.28:0/604371748 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f022c047060 con 0x7f0234158dc0 2024-02-21T23:05:49.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.597+0000 7f022aa00640 1 --2- 172.21.15.28:0/604371748 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f021006b500 0x7f021006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:49.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.597+0000 7f022aa00640 1 -- 172.21.15.28:0/604371748 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5619+0+0 (secure 0 0 0) 0x7f022c0aaea0 con 0x7f0234158dc0 2024-02-21T23:05:49.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.598+0000 7f0232800640 1 --2- 172.21.15.28:0/604371748 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f021006b500 0x7f021006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:49.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.599+0000 7f0232800640 1 --2- 172.21.15.28:0/604371748 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f021006b500 0x7f021006d9c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f021c007fd0 tx=0x7f021c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:49.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.602+0000 7f022aa00640 1 -- 172.21.15.28:0/604371748 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f022c079ce0 con 0x7f0234158dc0 2024-02-21T23:05:49.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:49.801+0000 7f0238a00640 1 -- 172.21.15.28:0/604371748 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f023404f290 con 0x7f0234158dc0 2024-02-21T23:05:50.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.339+0000 7f022aa00640 1 -- 172.21.15.28:0/604371748 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v74) v1 ==== 170+0+0 (secure 0 0 0) 0x7f022c0795e0 con 0x7f0234158dc0 2024-02-21T23:05:50.328 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' created 2024-02-21T23:05:50.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.344+0000 7f0238a00640 1 -- 172.21.15.28:0/604371748 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f021006b500 msgr2=0x7f021006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:50.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.344+0000 7f0238a00640 1 --2- 172.21.15.28:0/604371748 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f021006b500 0x7f021006d9c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f021c007fd0 tx=0x7f021c009040 comp rx=0 tx=0).stop 2024-02-21T23:05:50.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.344+0000 7f0238a00640 1 -- 172.21.15.28:0/604371748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0234158dc0 msgr2=0x7f0234149a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:50.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.344+0000 7f0238a00640 1 --2- 172.21.15.28:0/604371748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0234158dc0 0x7f0234149a80 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f022c001eb0 tx=0x7f0234155420 comp rx=0 tx=0).stop 2024-02-21T23:05:50.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.346+0000 7f0238a00640 1 -- 172.21.15.28:0/604371748 shutdown_connections 2024-02-21T23:05:50.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.346+0000 7f0238a00640 1 --2- 172.21.15.28:0/604371748 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f021006b500 0x7f021006d9c0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:50.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.346+0000 7f0238a00640 1 --2- 172.21.15.28:0/604371748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0234158dc0 0x7f0234149a80 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:50.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.346+0000 7f0238a00640 1 --2- 172.21.15.28:0/604371748 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f02341eaa40 0x7f0234149540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:50.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.346+0000 7f0238a00640 1 --2- 172.21.15.28:0/604371748 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f02180051c0 0x7f02340ab310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:50.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.346+0000 7f0238a00640 1 -- 172.21.15.28:0/604371748 >> 172.21.15.28:0/604371748 conn(0x7f0234072780 msgr2=0x7f02340c8570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:50.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.346+0000 7f0238a00640 1 -- 172.21.15.28:0/604371748 shutdown_connections 2024-02-21T23:05:50.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.347+0000 7f0238a00640 1 -- 172.21.15.28:0/604371748 wait complete. 2024-02-21T23:05:50.369 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-02-21T23:05:50.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.567+0000 7f7987000640 1 Processor -- start 2024-02-21T23:05:50.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.568+0000 7f7987000640 1 -- start start 2024-02-21T23:05:50.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.569+0000 7f7987000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f798006b0e0 0x7f79800c5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:50.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.569+0000 7f7987000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f7980150700 0x7f7980156500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:50.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.569+0000 7f7987000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79800c61e0 con 0x7f79800c5d80 2024-02-21T23:05:50.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.569+0000 7f7987000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7980153bc0 con 0x7f798006b0e0 2024-02-21T23:05:50.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.569+0000 7f7987000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7980152fb0 con 0x7f7980150700 2024-02-21T23:05:50.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.569+0000 7f797fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f798006b0e0 0x7f79800c5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:50.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.569+0000 7f797f400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f79800c5d80 0x7f798014fff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38272/0 (socket says 172.21.15.28:38272) 2024-02-21T23:05:50.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.569+0000 7f797f400640 1 -- 172.21.15.28:0/2073686622 learned_addr learned my addr 172.21.15.28:0/2073686622 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:05:50.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.570+0000 7f7985000640 1 --2- 172.21.15.28:0/2073686622 >> v2:172.21.15.81:3300/0 conn(0x7f7980150700 0x7f7980156500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:50.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.570+0000 7f797fe00640 1 -- 172.21.15.28:0/2073686622 >> v2:172.21.15.81:3300/0 conn(0x7f7980150700 msgr2=0x7f7980156500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:50.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.570+0000 7f797fe00640 1 --2- 172.21.15.28:0/2073686622 >> v2:172.21.15.81:3300/0 conn(0x7f7980150700 0x7f7980156500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:50.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.570+0000 7f797fe00640 1 -- 172.21.15.28:0/2073686622 >> v1:172.21.15.28:6789/0 conn(0x7f79800c5d80 legacy=0x7f798014fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:50.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.570+0000 7f797fe00640 1 -- 172.21.15.28:0/2073686622 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7980153290 con 0x7f798006b0e0 2024-02-21T23:05:50.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.571+0000 7f797fe00640 1 --2- 172.21.15.28:0/2073686622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f798006b0e0 0x7f79800c5840 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f79800a64b0 tx=0x7f797002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2a8245c2f3961a2 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:50.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.571+0000 7f797ea00640 1 -- 172.21.15.28:0/2073686622 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79700315a0 con 0x7f798006b0e0 2024-02-21T23:05:50.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.572+0000 7f797ea00640 1 -- 172.21.15.28:0/2073686622 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7970037070 con 0x7f798006b0e0 2024-02-21T23:05:50.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.572+0000 7f797ea00640 1 -- 172.21.15.28:0/2073686622 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7970031960 con 0x7f798006b0e0 2024-02-21T23:05:50.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.572+0000 7f7987000640 1 -- 172.21.15.28:0/2073686622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f798006b0e0 msgr2=0x7f79800c5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:50.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.572+0000 7f7987000640 1 --2- 172.21.15.28:0/2073686622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f798006b0e0 0x7f79800c5840 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f79800a64b0 tx=0x7f797002b840 comp rx=0 tx=0).stop 2024-02-21T23:05:50.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.572+0000 7f7987000640 1 -- 172.21.15.28:0/2073686622 shutdown_connections 2024-02-21T23:05:50.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.572+0000 7f7987000640 1 --2- 172.21.15.28:0/2073686622 >> v2:172.21.15.81:3300/0 conn(0x7f7980150700 0x7f7980156500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:50.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.572+0000 7f7987000640 1 --2- 172.21.15.28:0/2073686622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f798006b0e0 0x7f79800c5840 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:50.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.572+0000 7f7987000640 1 -- 172.21.15.28:0/2073686622 >> 172.21.15.28:0/2073686622 conn(0x7f79800a85a0 msgr2=0x7f79800a8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:50.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.572+0000 7f7987000640 1 -- 172.21.15.28:0/2073686622 shutdown_connections 2024-02-21T23:05:50.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.572+0000 7f7987000640 1 -- 172.21.15.28:0/2073686622 wait complete. 2024-02-21T23:05:50.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.574+0000 7f7987000640 1 Processor -- start 2024-02-21T23:05:50.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.574+0000 7f7987000640 1 -- start start 2024-02-21T23:05:50.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.575+0000 7f7987000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f798006b0e0 0x7f79800a5db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:50.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.575+0000 7f7987000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f79800c5d80 0x7f79800bfe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:50.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.575+0000 7f7987000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7980150700 0x7f79801eaac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:50.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.575+0000 7f7987000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f798006b4c0 con 0x7f7980150700 2024-02-21T23:05:50.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.575+0000 7f7987000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7980153850 con 0x7f798006b0e0 2024-02-21T23:05:50.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.575+0000 7f7987000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f798006b640 con 0x7f79800c5d80 2024-02-21T23:05:50.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.575+0000 7f7985000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7980150700 0x7f79801eaac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:50.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.576+0000 7f7985000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7980150700 0x7f79801eaac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47852/0 (socket says 172.21.15.28:47852) 2024-02-21T23:05:50.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.576+0000 7f7985000640 1 -- 172.21.15.28:0/1843835534 learned_addr learned my addr 172.21.15.28:0/1843835534 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:50.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.576+0000 7f797fe00640 1 --2- 172.21.15.28:0/1843835534 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f798006b0e0 0x7f79800a5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:50.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.576+0000 7f7985000640 1 -- 172.21.15.28:0/1843835534 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f79800c5d80 msgr2=0x7f79800bfe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:50.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.576+0000 7f7985000640 1 --2- 172.21.15.28:0/1843835534 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f79800c5d80 0x7f79800bfe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:50.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.576+0000 7f7985000640 1 -- 172.21.15.28:0/1843835534 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f798006b0e0 msgr2=0x7f79800a5db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:50.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.576+0000 7f7985000640 1 --2- 172.21.15.28:0/1843835534 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f798006b0e0 0x7f79800a5db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:50.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.576+0000 7f7985000640 1 -- 172.21.15.28:0/1843835534 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f798006f1e0 con 0x7f7980150700 2024-02-21T23:05:50.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.576+0000 7f797fe00640 1 --2- 172.21.15.28:0/1843835534 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f798006b0e0 0x7f79800a5db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:05:50.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.576+0000 7f7985000640 1 --2- 172.21.15.28:0/1843835534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7980150700 0x7f79801eaac0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f797800cc30 tx=0x7f797800a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:50.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.577+0000 7f7977e00640 1 -- 172.21.15.28:0/1843835534 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f797800ce40 con 0x7f7980150700 2024-02-21T23:05:50.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.577+0000 7f7977e00640 1 -- 172.21.15.28:0/1843835534 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7978003990 con 0x7f7980150700 2024-02-21T23:05:50.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.577+0000 7f7977e00640 1 -- 172.21.15.28:0/1843835534 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7978003d40 con 0x7f7980150700 2024-02-21T23:05:50.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.577+0000 7f7987000640 1 -- 172.21.15.28:0/1843835534 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f798014eb30 con 0x7f7980150700 2024-02-21T23:05:50.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.577+0000 7f7987000640 1 -- 172.21.15.28:0/1843835534 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f798014f0f0 con 0x7f7980150700 2024-02-21T23:05:50.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.578+0000 7f7987000640 1 -- 172.21.15.28:0/1843835534 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7980067f30 con 0x7f7980150700 2024-02-21T23:05:50.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.579+0000 7f7977e00640 1 -- 172.21.15.28:0/1843835534 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7978003ee0 con 0x7f7980150700 2024-02-21T23:05:50.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.580+0000 7f7977e00640 1 --2- 172.21.15.28:0/1843835534 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f796006b4d0 0x7f796006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:50.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.580+0000 7f7977e00640 1 -- 172.21.15.28:0/1843835534 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6001+0+0 (secure 0 0 0) 0x7f797808a870 con 0x7f7980150700 2024-02-21T23:05:50.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.581+0000 7f797fe00640 1 --2- 172.21.15.28:0/1843835534 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f796006b4d0 0x7f796006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:50.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.582+0000 7f797fe00640 1 --2- 172.21.15.28:0/1843835534 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f796006b4d0 0x7f796006d990 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f7970036000 tx=0x7f7970008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:50.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.585+0000 7f7977e00640 1 -- 172.21.15.28:0/1843835534 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7978014030 con 0x7f7980150700 2024-02-21T23:05:50.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:50.826+0000 7f7987000640 1 -- 172.21.15.28:0/1843835534 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f79800a8c30 con 0x7f7980150700 2024-02-21T23:05:51.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.403+0000 7f7977e00640 1 -- 172.21.15.28:0/1843835534 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 11 and data pool 12 v58) v1 ==== 174+0+0 (secure 0 0 0) 0x7f7978059520 con 0x7f7980150700 2024-02-21T23:05:51.392 INFO:teuthology.orchestra.run.smithi028.stderr:new fs with metadata pool 11 and data pool 12 2024-02-21T23:05:51.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.411+0000 7f7987000640 1 -- 172.21.15.28:0/1843835534 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f796006b4d0 msgr2=0x7f796006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:51.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.411+0000 7f7987000640 1 --2- 172.21.15.28:0/1843835534 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f796006b4d0 0x7f796006d990 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f7970036000 tx=0x7f7970008030 comp rx=0 tx=0).stop 2024-02-21T23:05:51.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.411+0000 7f7987000640 1 -- 172.21.15.28:0/1843835534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7980150700 msgr2=0x7f79801eaac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:51.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.411+0000 7f7987000640 1 --2- 172.21.15.28:0/1843835534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7980150700 0x7f79801eaac0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f797800cc30 tx=0x7f797800a5c0 comp rx=0 tx=0).stop 2024-02-21T23:05:51.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.412+0000 7f7987000640 1 -- 172.21.15.28:0/1843835534 shutdown_connections 2024-02-21T23:05:51.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.412+0000 7f7987000640 1 --2- 172.21.15.28:0/1843835534 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f796006b4d0 0x7f796006d990 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:51.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.412+0000 7f7987000640 1 --2- 172.21.15.28:0/1843835534 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f79800c5d80 0x7f79800bfe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:51.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.412+0000 7f7987000640 1 --2- 172.21.15.28:0/1843835534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7980150700 0x7f79801eaac0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:51.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.412+0000 7f7987000640 1 --2- 172.21.15.28:0/1843835534 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f798006b0e0 0x7f79800a5db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:51.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.412+0000 7f7987000640 1 -- 172.21.15.28:0/1843835534 >> 172.21.15.28:0/1843835534 conn(0x7f79800a85a0 msgr2=0x7f79800729d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:51.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.414+0000 7f7987000640 1 -- 172.21.15.28:0/1843835534 shutdown_connections 2024-02-21T23:05:51.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.414+0000 7f7987000640 1 -- 172.21.15.28:0/1843835534 wait complete. 2024-02-21T23:05:51.439 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:05:51.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.627+0000 7f8ddbe00640 1 Processor -- start 2024-02-21T23:05:51.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.627+0000 7f8ddbe00640 1 -- start start 2024-02-21T23:05:51.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.629+0000 7f8ddbe00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f8ddc150450 0x7f8ddc0a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:51.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.629+0000 7f8ddbe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8ddc156af0 0x7f8ddc06dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:51.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.629+0000 7f8ddbe00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ddc0c2760 con 0x7f8ddc06ee40 2024-02-21T23:05:51.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.629+0000 7f8ddbe00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ddc0c22c0 con 0x7f8ddc156af0 2024-02-21T23:05:51.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.629+0000 7f8ddbe00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ddc156660 con 0x7f8ddc150450 2024-02-21T23:05:51.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.629+0000 7f8ddb400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8ddc156af0 0x7f8ddc06dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:51.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.629+0000 7f8dd3e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f8ddc06ee40 0x7f8ddc1536f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38276/0 (socket says 172.21.15.28:38276) 2024-02-21T23:05:51.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.629+0000 7f8ddb400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8ddc156af0 0x7f8ddc06dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47864/0 (socket says 172.21.15.28:47864) 2024-02-21T23:05:51.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.629+0000 7f8dd3e00640 1 -- 172.21.15.28:0/1609877111 learned_addr learned my addr 172.21.15.28:0/1609877111 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:05:51.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.630+0000 7f8ddaa00640 1 --2- 172.21.15.28:0/1609877111 >> v2:172.21.15.81:3300/0 conn(0x7f8ddc150450 0x7f8ddc0a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:51.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.630+0000 7f8ddb400640 1 -- 172.21.15.28:0/1609877111 >> v2:172.21.15.81:3300/0 conn(0x7f8ddc150450 msgr2=0x7f8ddc0a5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:51.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.630+0000 7f8ddb400640 1 --2- 172.21.15.28:0/1609877111 >> v2:172.21.15.81:3300/0 conn(0x7f8ddc150450 0x7f8ddc0a5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:51.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.630+0000 7f8ddb400640 1 -- 172.21.15.28:0/1609877111 >> v1:172.21.15.28:6789/0 conn(0x7f8ddc06ee40 legacy=0x7f8ddc1536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:51.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.630+0000 7f8ddb400640 1 -- 172.21.15.28:0/1609877111 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ddc156310 con 0x7f8ddc156af0 2024-02-21T23:05:51.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.630+0000 7f8dda000640 1 -- 172.21.15.28:0/1609877111 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3653363589 0 0) 0x7f8ddc0c2760 con 0x7f8ddc06ee40 2024-02-21T23:05:51.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.631+0000 7f8ddb400640 1 --2- 172.21.15.28:0/1609877111 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ddc156af0 0x7f8ddc06dff0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc06f600 tx=0x7f8dd402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e286bcb36d85df3c server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:51.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.631+0000 7f8dda000640 1 -- 172.21.15.28:0/1609877111 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dd4039070 con 0x7f8ddc156af0 2024-02-21T23:05:51.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.631+0000 7f8dda000640 1 -- 172.21.15.28:0/1609877111 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dd402cc80 con 0x7f8ddc156af0 2024-02-21T23:05:51.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.631+0000 7f8dda000640 1 -- 172.21.15.28:0/1609877111 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dd4034660 con 0x7f8ddc156af0 2024-02-21T23:05:51.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.632+0000 7f8ddbe00640 1 -- 172.21.15.28:0/1609877111 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ddc156af0 msgr2=0x7f8ddc06dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:51.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.632+0000 7f8ddbe00640 1 --2- 172.21.15.28:0/1609877111 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ddc156af0 0x7f8ddc06dff0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc06f600 tx=0x7f8dd402bed0 comp rx=0 tx=0).stop 2024-02-21T23:05:51.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.632+0000 7f8ddbe00640 1 -- 172.21.15.28:0/1609877111 shutdown_connections 2024-02-21T23:05:51.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.632+0000 7f8ddbe00640 1 --2- 172.21.15.28:0/1609877111 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ddc156af0 0x7f8ddc06dff0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:51.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.632+0000 7f8ddbe00640 1 --2- 172.21.15.28:0/1609877111 >> v2:172.21.15.81:3300/0 conn(0x7f8ddc150450 0x7f8ddc0a5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:51.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.632+0000 7f8ddbe00640 1 -- 172.21.15.28:0/1609877111 >> 172.21.15.28:0/1609877111 conn(0x7f8ddc06b630 msgr2=0x7f8ddc0c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:51.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.632+0000 7f8ddbe00640 1 -- 172.21.15.28:0/1609877111 shutdown_connections 2024-02-21T23:05:51.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.632+0000 7f8ddbe00640 1 -- 172.21.15.28:0/1609877111 wait complete. 2024-02-21T23:05:51.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.634+0000 7f8ddbe00640 1 Processor -- start 2024-02-21T23:05:51.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.634+0000 7f8ddbe00640 1 -- start start 2024-02-21T23:05:51.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.635+0000 7f8ddbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ddc06ee40 0x7f8ddc1eb2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:51.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.635+0000 7f8ddbe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8ddc150450 0x7f8ddc0bfb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:51.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.635+0000 7f8ddbe00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8ddc156af0 0x7f8ddc0717a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:51.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.635+0000 7f8ddbe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ddc156f80 con 0x7f8ddc06ee40 2024-02-21T23:05:51.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.635+0000 7f8ddbe00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ddc0c2cb0 con 0x7f8ddc150450 2024-02-21T23:05:51.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.635+0000 7f8ddbe00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ddc157100 con 0x7f8ddc156af0 2024-02-21T23:05:51.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.635+0000 7f8ddaa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ddc06ee40 0x7f8ddc1eb2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:51.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.635+0000 7f8ddaa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ddc06ee40 0x7f8ddc1eb2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47866/0 (socket says 172.21.15.28:47866) 2024-02-21T23:05:51.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.635+0000 7f8ddaa00640 1 -- 172.21.15.28:0/3426333900 learned_addr learned my addr 172.21.15.28:0/3426333900 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:51.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.635+0000 7f8ddaa00640 1 -- 172.21.15.28:0/3426333900 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8ddc156af0 msgr2=0x7f8ddc0717a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:51.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.635+0000 7f8ddaa00640 1 --2- 172.21.15.28:0/3426333900 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8ddc156af0 0x7f8ddc0717a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:51.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.635+0000 7f8ddaa00640 1 -- 172.21.15.28:0/3426333900 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8ddc150450 msgr2=0x7f8ddc0bfb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:51.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.635+0000 7f8ddaa00640 1 --2- 172.21.15.28:0/3426333900 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8ddc150450 0x7f8ddc0bfb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:51.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.635+0000 7f8ddaa00640 1 -- 172.21.15.28:0/3426333900 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ddc0722d0 con 0x7f8ddc06ee40 2024-02-21T23:05:51.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.636+0000 7f8ddaa00640 1 --2- 172.21.15.28:0/3426333900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ddc06ee40 0x7f8ddc1eb2a0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc00d830 tx=0x7f8dcc00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:51.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.636+0000 7f8dd2a00640 1 -- 172.21.15.28:0/3426333900 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dcc00b6c0 con 0x7f8ddc06ee40 2024-02-21T23:05:51.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.636+0000 7f8dd2a00640 1 -- 172.21.15.28:0/3426333900 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dcc019070 con 0x7f8ddc06ee40 2024-02-21T23:05:51.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.636+0000 7f8dd2a00640 1 -- 172.21.15.28:0/3426333900 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dcc00ba60 con 0x7f8ddc06ee40 2024-02-21T23:05:51.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.636+0000 7f8ddbe00640 1 -- 172.21.15.28:0/3426333900 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ddc0725b0 con 0x7f8ddc06ee40 2024-02-21T23:05:51.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.636+0000 7f8ddbe00640 1 -- 172.21.15.28:0/3426333900 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ddc071280 con 0x7f8ddc06ee40 2024-02-21T23:05:51.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.637+0000 7f8ddbe00640 1 -- 172.21.15.28:0/3426333900 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ddc04f290 con 0x7f8ddc06ee40 2024-02-21T23:05:51.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.638+0000 7f8dd2a00640 1 -- 172.21.15.28:0/3426333900 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8dcc01e460 con 0x7f8ddc06ee40 2024-02-21T23:05:51.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.639+0000 7f8dd2a00640 1 --2- 172.21.15.28:0/3426333900 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8db806b4d0 0x7f8db806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:51.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.639+0000 7f8dd2a00640 1 -- 172.21.15.28:0/3426333900 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f8dcc08aa80 con 0x7f8ddc06ee40 2024-02-21T23:05:51.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.640+0000 7f8dd3e00640 1 --2- 172.21.15.28:0/3426333900 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8db806b4d0 0x7f8db806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:51.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.641+0000 7f8dd3e00640 1 --2- 172.21.15.28:0/3426333900 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8db806b4d0 0x7f8db806d990 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f8dc40011e0 tx=0x7f8dc4007940 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:51.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.644+0000 7f8dd2a00640 1 -- 172.21.15.28:0/3426333900 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f8dcc08ce20 con 0x7f8ddc06ee40 2024-02-21T23:05:51.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.837+0000 7f8ddbe00640 1 -- 172.21.15.28:0/3426333900 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8ddc0c1f10 con 0x7f8ddc06ee40 2024-02-21T23:05:51.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.840+0000 7f8dd2a00640 1 -- 172.21.15.28:0/3426333900 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v75) v1 ==== 74+0+16898 (secure 0 0 0) 0x7f8dcc059760 con 0x7f8ddc06ee40 2024-02-21T23:05:51.829 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:05:51.830 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":75,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:05:50.828171+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:05:48.772562+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-02-21T23:05:49.852659+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:05:51.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.842+0000 7f8ddbe00640 1 -- 172.21.15.28:0/3426333900 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8db806b4d0 msgr2=0x7f8db806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:51.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.842+0000 7f8ddbe00640 1 --2- 172.21.15.28:0/3426333900 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8db806b4d0 0x7f8db806d990 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f8dc40011e0 tx=0x7f8dc4007940 comp rx=0 tx=0).stop 2024-02-21T23:05:51.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.842+0000 7f8ddbe00640 1 -- 172.21.15.28:0/3426333900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ddc06ee40 msgr2=0x7f8ddc1eb2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:51.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.842+0000 7f8ddbe00640 1 --2- 172.21.15.28:0/3426333900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ddc06ee40 0x7f8ddc1eb2a0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc00d830 tx=0x7f8dcc00dcf0 comp rx=0 tx=0).stop 2024-02-21T23:05:51.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.842+0000 7f8ddbe00640 1 -- 172.21.15.28:0/3426333900 shutdown_connections 2024-02-21T23:05:51.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.842+0000 7f8ddbe00640 1 --2- 172.21.15.28:0/3426333900 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8db806b4d0 0x7f8db806d990 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:51.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.842+0000 7f8ddbe00640 1 --2- 172.21.15.28:0/3426333900 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8ddc156af0 0x7f8ddc0717a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:51.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.842+0000 7f8ddbe00640 1 --2- 172.21.15.28:0/3426333900 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8ddc150450 0x7f8ddc0bfb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:51.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.842+0000 7f8ddbe00640 1 --2- 172.21.15.28:0/3426333900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ddc06ee40 0x7f8ddc1eb2a0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:51.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.842+0000 7f8ddbe00640 1 -- 172.21.15.28:0/3426333900 >> 172.21.15.28:0/3426333900 conn(0x7f8ddc06b630 msgr2=0x7f8ddc14e380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:51.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.843+0000 7f8ddbe00640 1 -- 172.21.15.28:0/3426333900 shutdown_connections 2024-02-21T23:05:51.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:51.843+0000 7f8ddbe00640 1 -- 172.21.15.28:0/3426333900 wait complete. 2024-02-21T23:05:51.848 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:05:52.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.074+0000 7f0fe7200640 1 Processor -- start 2024-02-21T23:05:52.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.074+0000 7f0fe7200640 1 -- start start 2024-02-21T23:05:52.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.075+0000 7f0fe7200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fe0150450 0x7f0fe00a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:52.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.076+0000 7f0fe7200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0fe0156af0 0x7f0fe006dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:52.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.076+0000 7f0fe7200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fe00c2760 con 0x7f0fe006ee40 2024-02-21T23:05:52.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.076+0000 7f0fe7200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fe00c22c0 con 0x7f0fe0150450 2024-02-21T23:05:52.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.076+0000 7f0fe7200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe0156660 con 0x7f0fe0156af0 2024-02-21T23:05:52.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.076+0000 7f0fe5e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fe0150450 0x7f0fe00a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:52.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.076+0000 7f0fe5400640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f0fe006ee40 0x7f0fe01536f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.28:44616/0 (socket says 172.21.15.28:44616) 2024-02-21T23:05:52.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.076+0000 7f0fe5400640 1 -- 172.21.15.28:0/51563769 learned_addr learned my addr 172.21.15.28:0/51563769 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:05:52.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.076+0000 7f0fe6800640 1 --2- 172.21.15.28:0/51563769 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0fe0156af0 0x7f0fe006dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:52.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.077+0000 7f0fe4a00640 1 -- 172.21.15.28:0/51563769 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2651307824 0 0) 0x7f0fe00c2760 con 0x7f0fe006ee40 2024-02-21T23:05:52.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.077+0000 7f0fe4a00640 1 -- 172.21.15.28:0/51563769 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fc8003520 con 0x7f0fe006ee40 2024-02-21T23:05:52.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.077+0000 7f0fe5e00640 1 -- 172.21.15.28:0/51563769 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0fe0156af0 msgr2=0x7f0fe006dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:52.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.077+0000 7f0fe5e00640 1 --2- 172.21.15.28:0/51563769 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0fe0156af0 0x7f0fe006dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:52.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.077+0000 7f0fe5e00640 1 -- 172.21.15.28:0/51563769 >> v1:172.21.15.81:6789/0 conn(0x7f0fe006ee40 legacy=0x7f0fe01536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:52.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.077+0000 7f0fe5e00640 1 -- 172.21.15.28:0/51563769 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fe0156310 con 0x7f0fe0150450 2024-02-21T23:05:52.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.078+0000 7f0fe5e00640 1 --2- 172.21.15.28:0/51563769 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0fe0150450 0x7f0fe00a5fd0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0fd40097a0 tx=0x7f0fd402b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ddd349333d8bf76a server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:52.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.079+0000 7f0fe4a00640 1 -- 172.21.15.28:0/51563769 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fd4035020 con 0x7f0fe0150450 2024-02-21T23:05:52.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.079+0000 7f0fe4a00640 1 -- 172.21.15.28:0/51563769 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fd4031810 con 0x7f0fe0150450 2024-02-21T23:05:52.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.079+0000 7f0fe4a00640 1 -- 172.21.15.28:0/51563769 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fd4031bc0 con 0x7f0fe0150450 2024-02-21T23:05:52.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.079+0000 7f0fe7200640 1 -- 172.21.15.28:0/51563769 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0fe0150450 msgr2=0x7f0fe00a5fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:52.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.079+0000 7f0fe7200640 1 --2- 172.21.15.28:0/51563769 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0fe0150450 0x7f0fe00a5fd0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0fd40097a0 tx=0x7f0fd402b8b0 comp rx=0 tx=0).stop 2024-02-21T23:05:52.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.080+0000 7f0fe7200640 1 -- 172.21.15.28:0/51563769 shutdown_connections 2024-02-21T23:05:52.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.080+0000 7f0fe7200640 1 --2- 172.21.15.28:0/51563769 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0fe0156af0 0x7f0fe006dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:52.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.080+0000 7f0fe7200640 1 --2- 172.21.15.28:0/51563769 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0fe0150450 0x7f0fe00a5fd0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:52.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.080+0000 7f0fe7200640 1 -- 172.21.15.28:0/51563769 >> 172.21.15.28:0/51563769 conn(0x7f0fe006b630 msgr2=0x7f0fe00c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:52.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.080+0000 7f0fe7200640 1 -- 172.21.15.28:0/51563769 shutdown_connections 2024-02-21T23:05:52.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.080+0000 7f0fe7200640 1 -- 172.21.15.28:0/51563769 wait complete. 2024-02-21T23:05:52.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.081+0000 7f0fe7200640 1 Processor -- start 2024-02-21T23:05:52.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.081+0000 7f0fe7200640 1 -- start start 2024-02-21T23:05:52.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.082+0000 7f0fe7200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0fe006ee40 0x7f0fe00a6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:52.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.082+0000 7f0fe7200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0fe0150450 0x7f0fe0145710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:52.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.082+0000 7f0fe7200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0fe0156af0 0x7f0fe0145c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:52.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.082+0000 7f0fe7200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe0156f80 con 0x7f0fe0156af0 2024-02-21T23:05:52.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.082+0000 7f0fe7200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe00c2cb0 con 0x7f0fe006ee40 2024-02-21T23:05:52.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.082+0000 7f0fe7200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe0157100 con 0x7f0fe0150450 2024-02-21T23:05:52.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.083+0000 7f0fe6800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0fe0156af0 0x7f0fe0145c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:52.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.083+0000 7f0fe6800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0fe0156af0 0x7f0fe0145c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47868/0 (socket says 172.21.15.28:47868) 2024-02-21T23:05:52.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.083+0000 7f0fe6800640 1 -- 172.21.15.28:0/3817419813 learned_addr learned my addr 172.21.15.28:0/3817419813 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:52.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.083+0000 7f0fe5e00640 1 --2- 172.21.15.28:0/3817419813 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0fe006ee40 0x7f0fe00a6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:52.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.083+0000 7f0fe5400640 1 --2- 172.21.15.28:0/3817419813 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0fe0150450 0x7f0fe0145710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:52.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.083+0000 7f0fe6800640 1 -- 172.21.15.28:0/3817419813 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0fe0150450 msgr2=0x7f0fe0145710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:52.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.083+0000 7f0fe6800640 1 --2- 172.21.15.28:0/3817419813 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0fe0150450 0x7f0fe0145710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:52.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.083+0000 7f0fe6800640 1 -- 172.21.15.28:0/3817419813 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0fe006ee40 msgr2=0x7f0fe00a6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:52.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.083+0000 7f0fe6800640 1 --2- 172.21.15.28:0/3817419813 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0fe006ee40 0x7f0fe00a6ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:52.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.083+0000 7f0fe6800640 1 -- 172.21.15.28:0/3817419813 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fe01ee0c0 con 0x7f0fe0156af0 2024-02-21T23:05:52.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.083+0000 7f0fe6800640 1 --2- 172.21.15.28:0/3817419813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0fe0156af0 0x7f0fe0145c50 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f0fdc00efc0 tx=0x7f0fdc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:52.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.084+0000 7f0fda000640 1 -- 172.21.15.28:0/3817419813 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fdc018020 con 0x7f0fe0156af0 2024-02-21T23:05:52.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.084+0000 7f0fda000640 1 -- 172.21.15.28:0/3817419813 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fdc014480 con 0x7f0fe0156af0 2024-02-21T23:05:52.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.084+0000 7f0fe7200640 1 -- 172.21.15.28:0/3817419813 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fe01ee3a0 con 0x7f0fe0156af0 2024-02-21T23:05:52.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.084+0000 7f0fda000640 1 -- 172.21.15.28:0/3817419813 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fdc014870 con 0x7f0fe0156af0 2024-02-21T23:05:52.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.084+0000 7f0fe7200640 1 -- 172.21.15.28:0/3817419813 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fe00a7530 con 0x7f0fe0156af0 2024-02-21T23:05:52.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.085+0000 7f0fe7200640 1 -- 172.21.15.28:0/3817419813 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fe004f290 con 0x7f0fe0156af0 2024-02-21T23:05:52.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.086+0000 7f0fda000640 1 -- 172.21.15.28:0/3817419813 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0fdc005d10 con 0x7f0fe0156af0 2024-02-21T23:05:52.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.087+0000 7f0fda000640 1 --2- 172.21.15.28:0/3817419813 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0fc406b500 0x7f0fc406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:52.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.087+0000 7f0fda000640 1 -- 172.21.15.28:0/3817419813 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f0fdc08b130 con 0x7f0fe0156af0 2024-02-21T23:05:52.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.088+0000 7f0fe5e00640 1 --2- 172.21.15.28:0/3817419813 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0fc406b500 0x7f0fc406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:52.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.089+0000 7f0fe5e00640 1 --2- 172.21.15.28:0/3817419813 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0fc406b500 0x7f0fc406d9c0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f0fd4036000 tx=0x7f0fd4008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:52.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.092+0000 7f0fda000640 1 -- 172.21.15.28:0/3817419813 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f0fdc022030 con 0x7f0fe0156af0 2024-02-21T23:05:52.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.289+0000 7f0fe7200640 1 -- 172.21.15.28:0/3817419813 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0fe00c39a0 con 0x7f0fe0156af0 2024-02-21T23:05:52.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.291+0000 7f0fda000640 1 -- 172.21.15.28:0/3817419813 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v75) v1 ==== 74+0+16898 (secure 0 0 0) 0x7f0fdc082e20 con 0x7f0fe0156af0 2024-02-21T23:05:52.280 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:05:52.280 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":75,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:05:50.828171+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:05:48.772562+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-02-21T23:05:49.852659+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:05:52.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.294+0000 7f0fe7200640 1 -- 172.21.15.28:0/3817419813 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0fc406b500 msgr2=0x7f0fc406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:52.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.294+0000 7f0fe7200640 1 --2- 172.21.15.28:0/3817419813 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0fc406b500 0x7f0fc406d9c0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f0fd4036000 tx=0x7f0fd4008030 comp rx=0 tx=0).stop 2024-02-21T23:05:52.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.294+0000 7f0fe7200640 1 -- 172.21.15.28:0/3817419813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0fe0156af0 msgr2=0x7f0fe0145c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:52.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.294+0000 7f0fe7200640 1 --2- 172.21.15.28:0/3817419813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0fe0156af0 0x7f0fe0145c50 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f0fdc00efc0 tx=0x7f0fdc00c490 comp rx=0 tx=0).stop 2024-02-21T23:05:52.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.295+0000 7f0fe7200640 1 -- 172.21.15.28:0/3817419813 shutdown_connections 2024-02-21T23:05:52.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.295+0000 7f0fe7200640 1 --2- 172.21.15.28:0/3817419813 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0fc406b500 0x7f0fc406d9c0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:52.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.295+0000 7f0fe7200640 1 --2- 172.21.15.28:0/3817419813 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0fe0150450 0x7f0fe0145710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:52.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.295+0000 7f0fe7200640 1 --2- 172.21.15.28:0/3817419813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0fe0156af0 0x7f0fe0145c50 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:52.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.295+0000 7f0fe7200640 1 --2- 172.21.15.28:0/3817419813 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0fe006ee40 0x7f0fe00a6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:52.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.295+0000 7f0fe7200640 1 -- 172.21.15.28:0/3817419813 >> 172.21.15.28:0/3817419813 conn(0x7f0fe006b630 msgr2=0x7f0fe00c2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:52.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.295+0000 7f0fe7200640 1 -- 172.21.15.28:0/3817419813 shutdown_connections 2024-02-21T23:05:52.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.295+0000 7f0fe7200640 1 -- 172.21.15.28:0/3817419813 wait complete. 2024-02-21T23:05:52.301 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-02-21T23:05:52.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.525+0000 7f4da4a00640 1 Processor -- start 2024-02-21T23:05:52.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.525+0000 7f4da4a00640 1 -- start start 2024-02-21T23:05:52.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.527+0000 7f4da4a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4da0152680 0x7f4da00c0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:52.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.527+0000 7f4da4a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f4da00c1890 0x7f4da0158ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:52.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.527+0000 7f4da4a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4da00c2ff0 con 0x7f4da00c11e0 2024-02-21T23:05:52.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.527+0000 7f4da4a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4da00c32d0 con 0x7f4da0152680 2024-02-21T23:05:52.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.527+0000 7f4da4a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4da00c35b0 con 0x7f4da00c1890 2024-02-21T23:05:52.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.527+0000 7f4d9de00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4da0152680 0x7f4da00c0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:52.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.528+0000 7f4d9de00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4da0152680 0x7f4da00c0ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47882/0 (socket says 172.21.15.28:47882) 2024-02-21T23:05:52.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.528+0000 7f4d9de00640 1 -- 172.21.15.28:0/1634105272 learned_addr learned my addr 172.21.15.28:0/1634105272 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:52.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.528+0000 7f4d9d400640 1 --2- 172.21.15.28:0/1634105272 >> v2:172.21.15.81:3300/0 conn(0x7f4da00c1890 0x7f4da0158ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:52.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.528+0000 7f4d9de00640 1 -- 172.21.15.28:0/1634105272 >> v2:172.21.15.81:3300/0 conn(0x7f4da00c1890 msgr2=0x7f4da0158ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:52.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.528+0000 7f4d9de00640 1 --2- 172.21.15.28:0/1634105272 >> v2:172.21.15.81:3300/0 conn(0x7f4da00c1890 0x7f4da0158ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:52.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.528+0000 7f4d9de00640 1 -- 172.21.15.28:0/1634105272 >> v1:172.21.15.81:6789/0 conn(0x7f4da00c11e0 legacy=0x7f4da006ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:52.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.528+0000 7f4d9de00640 1 -- 172.21.15.28:0/1634105272 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4da00c3890 con 0x7f4da0152680 2024-02-21T23:05:52.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.530+0000 7f4d9de00640 1 --2- 172.21.15.28:0/1634105272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4da0152680 0x7f4da00c0ca0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f4da01562e0 tx=0x7f4d9002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5d3a9b85f0b4088 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:52.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.530+0000 7f4d9ca00640 1 -- 172.21.15.28:0/1634105272 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d90035020 con 0x7f4da0152680 2024-02-21T23:05:52.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.530+0000 7f4d9ca00640 1 -- 172.21.15.28:0/1634105272 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d90031870 con 0x7f4da0152680 2024-02-21T23:05:52.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.530+0000 7f4d9ca00640 1 -- 172.21.15.28:0/1634105272 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d90031c40 con 0x7f4da0152680 2024-02-21T23:05:52.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.531+0000 7f4da4a00640 1 -- 172.21.15.28:0/1634105272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4da0152680 msgr2=0x7f4da00c0ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:52.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.531+0000 7f4da4a00640 1 --2- 172.21.15.28:0/1634105272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4da0152680 0x7f4da00c0ca0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f4da01562e0 tx=0x7f4d9002b910 comp rx=0 tx=0).stop 2024-02-21T23:05:52.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.531+0000 7f4da4a00640 1 -- 172.21.15.28:0/1634105272 shutdown_connections 2024-02-21T23:05:52.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.531+0000 7f4da4a00640 1 --2- 172.21.15.28:0/1634105272 >> v2:172.21.15.81:3300/0 conn(0x7f4da00c1890 0x7f4da0158ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:52.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.531+0000 7f4da4a00640 1 --2- 172.21.15.28:0/1634105272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4da0152680 0x7f4da00c0ca0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:52.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.531+0000 7f4da4a00640 1 -- 172.21.15.28:0/1634105272 >> 172.21.15.28:0/1634105272 conn(0x7f4da00a9a10 msgr2=0x7f4da00a9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:52.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.531+0000 7f4da4a00640 1 -- 172.21.15.28:0/1634105272 shutdown_connections 2024-02-21T23:05:52.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.531+0000 7f4da4a00640 1 -- 172.21.15.28:0/1634105272 wait complete. 2024-02-21T23:05:52.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.534+0000 7f4da4a00640 1 Processor -- start 2024-02-21T23:05:52.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.534+0000 7f4da4a00640 1 -- start start 2024-02-21T23:05:52.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.535+0000 7f4da4a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4da00c11e0 0x7f4da014b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:52.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.535+0000 7f4da4a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4da00c1890 0x7f4da014d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:52.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.536+0000 7f4da4a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4da0152680 0x7f4da0150100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:52.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.536+0000 7f4da4a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4da0155330 con 0x7f4da0152680 2024-02-21T23:05:52.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.536+0000 7f4da4a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4da01551b0 con 0x7f4da00c11e0 2024-02-21T23:05:52.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.536+0000 7f4da4a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4da0152a60 con 0x7f4da00c1890 2024-02-21T23:05:52.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.536+0000 7f4d9e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4da0152680 0x7f4da0150100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:52.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.536+0000 7f4d9e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4da0152680 0x7f4da0150100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47892/0 (socket says 172.21.15.28:47892) 2024-02-21T23:05:52.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.536+0000 7f4d9e800640 1 -- 172.21.15.28:0/2707816154 learned_addr learned my addr 172.21.15.28:0/2707816154 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:52.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.536+0000 7f4d9de00640 1 --2- 172.21.15.28:0/2707816154 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4da00c11e0 0x7f4da014b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:52.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.536+0000 7f4d9d400640 1 --2- 172.21.15.28:0/2707816154 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4da00c1890 0x7f4da014d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:52.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.536+0000 7f4d9e800640 1 -- 172.21.15.28:0/2707816154 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4da00c1890 msgr2=0x7f4da014d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:52.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.536+0000 7f4d9e800640 1 --2- 172.21.15.28:0/2707816154 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4da00c1890 0x7f4da014d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:52.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.536+0000 7f4d9e800640 1 -- 172.21.15.28:0/2707816154 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4da00c11e0 msgr2=0x7f4da014b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:52.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.536+0000 7f4d9e800640 1 --2- 172.21.15.28:0/2707816154 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4da00c11e0 0x7f4da014b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:52.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.536+0000 7f4d9e800640 1 -- 172.21.15.28:0/2707816154 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4da014b460 con 0x7f4da0152680 2024-02-21T23:05:52.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.537+0000 7f4d9e800640 1 --2- 172.21.15.28:0/2707816154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4da0152680 0x7f4da0150100 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f4d9800b970 tx=0x7f4d9800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:52.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.537+0000 7f4d96000640 1 -- 172.21.15.28:0/2707816154 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d98019070 con 0x7f4da0152680 2024-02-21T23:05:52.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.537+0000 7f4d96000640 1 -- 172.21.15.28:0/2707816154 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d9800cb40 con 0x7f4da0152680 2024-02-21T23:05:52.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.537+0000 7f4da4a00640 1 -- 172.21.15.28:0/2707816154 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4da01eb8c0 con 0x7f4da0152680 2024-02-21T23:05:52.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.537+0000 7f4d96000640 1 -- 172.21.15.28:0/2707816154 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d98014440 con 0x7f4da0152680 2024-02-21T23:05:52.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.537+0000 7f4da4a00640 1 -- 172.21.15.28:0/2707816154 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4da014dc40 con 0x7f4da0152680 2024-02-21T23:05:52.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.538+0000 7f4da4a00640 1 -- 172.21.15.28:0/2707816154 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4da0068640 con 0x7f4da0152680 2024-02-21T23:05:52.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.539+0000 7f4d96000640 1 -- 172.21.15.28:0/2707816154 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4d9801c070 con 0x7f4da0152680 2024-02-21T23:05:52.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.541+0000 7f4d96000640 1 --2- 172.21.15.28:0/2707816154 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4d8006b4d0 0x7f4d8006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:52.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.541+0000 7f4d96000640 1 -- 172.21.15.28:0/2707816154 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4d98089f10 con 0x7f4da0152680 2024-02-21T23:05:52.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.541+0000 7f4d9de00640 1 --2- 172.21.15.28:0/2707816154 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4d8006b4d0 0x7f4d8006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:52.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.542+0000 7f4d9de00640 1 --2- 172.21.15.28:0/2707816154 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4d8006b4d0 0x7f4d8006d990 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f4d90036000 tx=0x7f4d9003c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:52.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.545+0000 7f4d96000640 1 -- 172.21.15.28:0/2707816154 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4d98058c90 con 0x7f4da0152680 2024-02-21T23:05:52.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:52.777+0000 7f4da4a00640 1 -- 172.21.15.28:0/2707816154 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f4da00704c0 con 0x7f4da0152680 2024-02-21T23:05:53.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.421+0000 7f4d96000640 1 -- 172.21.15.28:0/2707816154 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v61) v1 ==== 118+0+0 (secure 0 0 0) 0x7f4d98058590 con 0x7f4da0152680 2024-02-21T23:05:53.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.425+0000 7f4da4a00640 1 -- 172.21.15.28:0/2707816154 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4d8006b4d0 msgr2=0x7f4d8006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:53.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.425+0000 7f4da4a00640 1 --2- 172.21.15.28:0/2707816154 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4d8006b4d0 0x7f4d8006d990 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f4d90036000 tx=0x7f4d9003c040 comp rx=0 tx=0).stop 2024-02-21T23:05:53.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.425+0000 7f4da4a00640 1 -- 172.21.15.28:0/2707816154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4da0152680 msgr2=0x7f4da0150100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:53.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.425+0000 7f4da4a00640 1 --2- 172.21.15.28:0/2707816154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4da0152680 0x7f4da0150100 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f4d9800b970 tx=0x7f4d9800be30 comp rx=0 tx=0).stop 2024-02-21T23:05:53.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.425+0000 7f4da4a00640 1 -- 172.21.15.28:0/2707816154 shutdown_connections 2024-02-21T23:05:53.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.425+0000 7f4da4a00640 1 --2- 172.21.15.28:0/2707816154 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4d8006b4d0 0x7f4d8006d990 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:53.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.425+0000 7f4da4a00640 1 --2- 172.21.15.28:0/2707816154 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4da00c1890 0x7f4da014d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:53.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.425+0000 7f4da4a00640 1 --2- 172.21.15.28:0/2707816154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4da0152680 0x7f4da0150100 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:53.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.425+0000 7f4da4a00640 1 --2- 172.21.15.28:0/2707816154 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4da00c11e0 0x7f4da014b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:53.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.425+0000 7f4da4a00640 1 -- 172.21.15.28:0/2707816154 >> 172.21.15.28:0/2707816154 conn(0x7f4da00a9a10 msgr2=0x7f4da00a9df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:53.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.425+0000 7f4da4a00640 1 -- 172.21.15.28:0/2707816154 shutdown_connections 2024-02-21T23:05:53.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.425+0000 7f4da4a00640 1 -- 172.21.15.28:0/2707816154 wait complete. 2024-02-21T23:05:53.444 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-02-21T23:05:53.445 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-02-21T23:05:53.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.678+0000 7f74cae00640 1 Processor -- start 2024-02-21T23:05:53.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.678+0000 7f74cae00640 1 -- start start 2024-02-21T23:05:53.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.680+0000 7f74cae00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f74c413e0c0 0x7f74c40c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:53.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.680+0000 7f74cae00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f74c40c76f0 0x7f74c406c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:53.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.680+0000 7f74cae00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74c4158550 con 0x7f74c406dbc0 2024-02-21T23:05:53.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.680+0000 7f74cae00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74c40c89a0 con 0x7f74c413e0c0 2024-02-21T23:05:53.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.680+0000 7f74cae00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74c41587d0 con 0x7f74c40c76f0 2024-02-21T23:05:53.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.680+0000 7f74c9a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f74c413e0c0 0x7f74c40c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:53.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.680+0000 7f74c9000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f74c406dbc0 0x7f74c406c280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38280/0 (socket says 172.21.15.28:38280) 2024-02-21T23:05:53.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.681+0000 7f74c9a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f74c413e0c0 0x7f74c40c71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47898/0 (socket says 172.21.15.28:47898) 2024-02-21T23:05:53.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.681+0000 7f74c9000640 1 -- 172.21.15.28:0/3216843858 learned_addr learned my addr 172.21.15.28:0/3216843858 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:05:53.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.681+0000 7f74ca400640 1 --2- 172.21.15.28:0/3216843858 >> v2:172.21.15.81:3300/0 conn(0x7f74c40c76f0 0x7f74c406c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:53.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.681+0000 7f74c9a00640 1 -- 172.21.15.28:0/3216843858 >> v2:172.21.15.81:3300/0 conn(0x7f74c40c76f0 msgr2=0x7f74c406c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:53.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.681+0000 7f74c9a00640 1 --2- 172.21.15.28:0/3216843858 >> v2:172.21.15.81:3300/0 conn(0x7f74c40c76f0 0x7f74c406c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:53.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.681+0000 7f74c9a00640 1 -- 172.21.15.28:0/3216843858 >> v1:172.21.15.28:6789/0 conn(0x7f74c406dbc0 legacy=0x7f74c406c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:53.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.681+0000 7f74c9a00640 1 -- 172.21.15.28:0/3216843858 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74c4158ab0 con 0x7f74c413e0c0 2024-02-21T23:05:53.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.681+0000 7f74c3e00640 1 -- 172.21.15.28:0/3216843858 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 313655049 0 0) 0x7f74b0001110 con 0x7f74c406dbc0 2024-02-21T23:05:53.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.682+0000 7f74c9a00640 1 --2- 172.21.15.28:0/3216843858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74c413e0c0 0x7f74c40c71b0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f74c40c8260 tx=0x7f74b802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73e237ec4203eefa server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:53.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.683+0000 7f74c3e00640 1 -- 172.21.15.28:0/3216843858 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b80315a0 con 0x7f74c413e0c0 2024-02-21T23:05:53.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.683+0000 7f74c3e00640 1 -- 172.21.15.28:0/3216843858 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74b8037070 con 0x7f74c413e0c0 2024-02-21T23:05:53.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.683+0000 7f74c3e00640 1 -- 172.21.15.28:0/3216843858 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b8031960 con 0x7f74c413e0c0 2024-02-21T23:05:53.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.683+0000 7f74cae00640 1 -- 172.21.15.28:0/3216843858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74c413e0c0 msgr2=0x7f74c40c71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:53.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.683+0000 7f74cae00640 1 --2- 172.21.15.28:0/3216843858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74c413e0c0 0x7f74c40c71b0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f74c40c8260 tx=0x7f74b802b840 comp rx=0 tx=0).stop 2024-02-21T23:05:53.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.683+0000 7f74cae00640 1 -- 172.21.15.28:0/3216843858 shutdown_connections 2024-02-21T23:05:53.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.683+0000 7f74cae00640 1 --2- 172.21.15.28:0/3216843858 >> v2:172.21.15.81:3300/0 conn(0x7f74c40c76f0 0x7f74c406c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:53.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.683+0000 7f74cae00640 1 --2- 172.21.15.28:0/3216843858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74c413e0c0 0x7f74c40c71b0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:53.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.683+0000 7f74cae00640 1 -- 172.21.15.28:0/3216843858 >> 172.21.15.28:0/3216843858 conn(0x7f74c4073110 msgr2=0x7f74c40ac490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:53.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.684+0000 7f74cae00640 1 -- 172.21.15.28:0/3216843858 shutdown_connections 2024-02-21T23:05:53.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.684+0000 7f74cae00640 1 -- 172.21.15.28:0/3216843858 wait complete. 2024-02-21T23:05:53.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.685+0000 7f74cae00640 1 Processor -- start 2024-02-21T23:05:53.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.685+0000 7f74cae00640 1 -- start start 2024-02-21T23:05:53.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.686+0000 7f74cae00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74c406dbc0 0x7f74c4137df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:53.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.686+0000 7f74cae00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74c40c76f0 0x7f74c41578a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:53.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.686+0000 7f74cae00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f74c413e0c0 0x7f74c41ec750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:53.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.686+0000 7f74cae00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74c406ced0 con 0x7f74c406dbc0 2024-02-21T23:05:53.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.686+0000 7f74cae00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74c4159070 con 0x7f74c40c76f0 2024-02-21T23:05:53.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.686+0000 7f74cae00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74c413e520 con 0x7f74c413e0c0 2024-02-21T23:05:53.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.686+0000 7f74c9a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74c406dbc0 0x7f74c4137df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:53.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.686+0000 7f74c9a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74c406dbc0 0x7f74c4137df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47900/0 (socket says 172.21.15.28:47900) 2024-02-21T23:05:53.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.686+0000 7f74c9a00640 1 -- 172.21.15.28:0/947306324 learned_addr learned my addr 172.21.15.28:0/947306324 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:53.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.687+0000 7f74c9000640 1 --2- 172.21.15.28:0/947306324 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74c40c76f0 0x7f74c41578a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:53.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.687+0000 7f74c9a00640 1 -- 172.21.15.28:0/947306324 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f74c413e0c0 msgr2=0x7f74c41ec750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:53.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.687+0000 7f74c9a00640 1 --2- 172.21.15.28:0/947306324 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f74c413e0c0 0x7f74c41ec750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:53.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.687+0000 7f74c9a00640 1 -- 172.21.15.28:0/947306324 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74c40c76f0 msgr2=0x7f74c41578a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:53.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.687+0000 7f74c9a00640 1 --2- 172.21.15.28:0/947306324 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74c40c76f0 0x7f74c41578a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:53.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.687+0000 7f74c9a00640 1 -- 172.21.15.28:0/947306324 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74c41eeb80 con 0x7f74c406dbc0 2024-02-21T23:05:53.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.687+0000 7f74c9a00640 1 --2- 172.21.15.28:0/947306324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74c406dbc0 0x7f74c4137df0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f74b8009890 tx=0x7f74b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:53.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.688+0000 7f74c1600640 1 -- 172.21.15.28:0/947306324 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b8032840 con 0x7f74c406dbc0 2024-02-21T23:05:53.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.688+0000 7f74c1600640 1 -- 172.21.15.28:0/947306324 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74b80329e0 con 0x7f74c406dbc0 2024-02-21T23:05:53.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.688+0000 7f74c1600640 1 -- 172.21.15.28:0/947306324 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b8032dd0 con 0x7f74c406dbc0 2024-02-21T23:05:53.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.688+0000 7f74cae00640 1 -- 172.21.15.28:0/947306324 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74c41ebf50 con 0x7f74c406dbc0 2024-02-21T23:05:53.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.688+0000 7f74cae00640 1 -- 172.21.15.28:0/947306324 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74c40ab550 con 0x7f74c406dbc0 2024-02-21T23:05:53.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.689+0000 7f74cae00640 1 -- 172.21.15.28:0/947306324 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74c404f290 con 0x7f74c406dbc0 2024-02-21T23:05:53.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.690+0000 7f74c1600640 1 -- 172.21.15.28:0/947306324 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f74b803b3f0 con 0x7f74c406dbc0 2024-02-21T23:05:53.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.692+0000 7f74c1600640 1 --2- 172.21.15.28:0/947306324 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f74a406b500 0x7f74a406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:53.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.692+0000 7f74c1600640 1 -- 172.21.15.28:0/947306324 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f74b802f030 con 0x7f74c406dbc0 2024-02-21T23:05:53.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.692+0000 7f74c9000640 1 --2- 172.21.15.28:0/947306324 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f74a406b500 0x7f74a406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:53.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.693+0000 7f74c9000640 1 --2- 172.21.15.28:0/947306324 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f74a406b500 0x7f74a406d9c0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f74c4158290 tx=0x7f74b0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:53.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.695+0000 7f74c1600640 1 -- 172.21.15.28:0/947306324 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f74b8040070 con 0x7f74c406dbc0 2024-02-21T23:05:53.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:53.936+0000 7f74cae00640 1 -- 172.21.15.28:0/947306324 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f74c40ac700 con 0x7f74c406dbc0 2024-02-21T23:05:54.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.424+0000 7f74c1600640 1 -- 172.21.15.28:0/947306324 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v62) v1 ==== 122+0+0 (secure 0 0 0) 0x7f74b807b0b0 con 0x7f74c406dbc0 2024-02-21T23:05:54.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.428+0000 7f74cae00640 1 -- 172.21.15.28:0/947306324 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f74a406b500 msgr2=0x7f74a406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:54.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.428+0000 7f74cae00640 1 --2- 172.21.15.28:0/947306324 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f74a406b500 0x7f74a406d9c0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f74c4158290 tx=0x7f74b0009040 comp rx=0 tx=0).stop 2024-02-21T23:05:54.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.428+0000 7f74cae00640 1 -- 172.21.15.28:0/947306324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74c406dbc0 msgr2=0x7f74c4137df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:54.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.428+0000 7f74cae00640 1 --2- 172.21.15.28:0/947306324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74c406dbc0 0x7f74c4137df0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f74b8009890 tx=0x7f74b80046c0 comp rx=0 tx=0).stop 2024-02-21T23:05:54.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.428+0000 7f74cae00640 1 -- 172.21.15.28:0/947306324 shutdown_connections 2024-02-21T23:05:54.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.428+0000 7f74cae00640 1 --2- 172.21.15.28:0/947306324 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f74a406b500 0x7f74a406d9c0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:54.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.428+0000 7f74cae00640 1 --2- 172.21.15.28:0/947306324 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f74c413e0c0 0x7f74c41ec750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:54.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.428+0000 7f74cae00640 1 --2- 172.21.15.28:0/947306324 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74c40c76f0 0x7f74c41578a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:54.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.428+0000 7f74cae00640 1 --2- 172.21.15.28:0/947306324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74c406dbc0 0x7f74c4137df0 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:54.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.428+0000 7f74cae00640 1 -- 172.21.15.28:0/947306324 >> 172.21.15.28:0/947306324 conn(0x7f74c4073110 msgr2=0x7f74c40c8fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:54.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.428+0000 7f74cae00640 1 -- 172.21.15.28:0/947306324 shutdown_connections 2024-02-21T23:05:54.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.428+0000 7f74cae00640 1 -- 172.21.15.28:0/947306324 wait complete. 2024-02-21T23:05:54.439 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-02-21T23:05:54.440 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:05:54.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.677+0000 7f14ab200640 1 Processor -- start 2024-02-21T23:05:54.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.677+0000 7f14ab200640 1 -- start start 2024-02-21T23:05:54.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.679+0000 7f14ab200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f14ac0aff90 0x7f14ac0c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:54.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.679+0000 7f14ab200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14ac0c74c0 con 0x7f14ac06dbf0 2024-02-21T23:05:54.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.679+0000 7f14ab200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14ac0c77a0 con 0x7f14ac0c5720 2024-02-21T23:05:54.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.679+0000 7f14ab200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14ac0c7270 con 0x7f14ac0aff90 2024-02-21T23:05:54.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.679+0000 7f14a9400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f14ac06dbf0 0x7f14ac158ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38294/0 (socket says 172.21.15.28:38294) 2024-02-21T23:05:54.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.679+0000 7f14a9400640 1 -- 172.21.15.28:0/2078902971 learned_addr learned my addr 172.21.15.28:0/2078902971 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:05:54.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.679+0000 7f14a9e00640 1 --2- 172.21.15.28:0/2078902971 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f14ac0aff90 0x7f14ac0c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:54.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.680+0000 7f14a8a00640 1 -- 172.21.15.28:0/2078902971 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2603951302 0 0) 0x7f14ac0c74c0 con 0x7f14ac06dbf0 2024-02-21T23:05:54.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.680+0000 7f14a8a00640 1 -- 172.21.15.28:0/2078902971 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1490003520 con 0x7f14ac06dbf0 2024-02-21T23:05:54.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.680+0000 7f14a8a00640 1 -- 172.21.15.28:0/2078902971 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2376153083 0 0) 0x7f1490003520 con 0x7f14ac06dbf0 2024-02-21T23:05:54.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.680+0000 7f14a8a00640 1 -- 172.21.15.28:0/2078902971 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f14ac0aff90 msgr2=0x7f14ac0c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:54.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.680+0000 7f14a8a00640 1 --2- 172.21.15.28:0/2078902971 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f14ac0aff90 0x7f14ac0c51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:54.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.680+0000 7f14a8a00640 1 -- 172.21.15.28:0/2078902971 >> v1:172.21.15.81:6789/0 conn(0x7f14ac0c5720 legacy=0x7f14ac15f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:54.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.680+0000 7f14a8a00640 1 -- 172.21.15.28:0/2078902971 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14ac0c7a80 con 0x7f14ac06dbf0 2024-02-21T23:05:54.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.680+0000 7f14a8a00640 1 -- 172.21.15.28:0/2078902971 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f1498001670 con 0x7f14ac06dbf0 2024-02-21T23:05:54.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.680+0000 7f14a8a00640 1 -- 172.21.15.28:0/2078902971 >> v1:172.21.15.28:6789/0 conn(0x7f14ac06dbf0 legacy=0x7f14ac158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:54.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.680+0000 7f14a8a00640 1 --2- 172.21.15.28:0/2078902971 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f14ac0aff90 0x7f14900039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:54.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.680+0000 7f14a8a00640 1 --2- 172.21.15.28:0/2078902971 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1490003fb0 0x7f1490004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:54.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.680+0000 7f14a8a00640 1 --2- 172.21.15.28:0/2078902971 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1490004d40 0x7f14900051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:54.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.680+0000 7f14a8a00640 1 -- 172.21.15.28:0/2078902971 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14ac0c7270 con 0x7f1490004d40 2024-02-21T23:05:54.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.680+0000 7f14a8a00640 1 -- 172.21.15.28:0/2078902971 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f149000bb00 con 0x7f1490003fb0 2024-02-21T23:05:54.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.680+0000 7f14a8a00640 1 -- 172.21.15.28:0/2078902971 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f149000bd00 con 0x7f14ac0aff90 2024-02-21T23:05:54.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.681+0000 7f14aa800640 1 --2- 172.21.15.28:0/2078902971 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1490004d40 0x7f14900051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:54.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.681+0000 7f14a9e00640 1 --2- 172.21.15.28:0/2078902971 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f14ac0aff90 0x7f14900039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:54.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.681+0000 7f14a9400640 1 --2- 172.21.15.28:0/2078902971 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1490003fb0 0x7f1490004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:54.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.681+0000 7f14aa800640 1 -- 172.21.15.28:0/2078902971 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f14ac0aff90 msgr2=0x7f14900039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:54.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.681+0000 7f14aa800640 1 --2- 172.21.15.28:0/2078902971 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f14ac0aff90 0x7f14900039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:54.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.681+0000 7f14aa800640 1 -- 172.21.15.28:0/2078902971 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1490003fb0 msgr2=0x7f1490004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:54.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.681+0000 7f14aa800640 1 --2- 172.21.15.28:0/2078902971 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1490003fb0 0x7f1490004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:54.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.681+0000 7f14aa800640 1 -- 172.21.15.28:0/2078902971 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1490003520 con 0x7f1490004d40 2024-02-21T23:05:54.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.682+0000 7f14aa800640 1 --2- 172.21.15.28:0/2078902971 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1490004d40 0x7f14900051c0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f14a4001e50 tx=0x7f1490005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddf338311e1306f7 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:54.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.683+0000 7f14a8a00640 1 -- 172.21.15.28:0/2078902971 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a4001a70 con 0x7f1490004d40 2024-02-21T23:05:54.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.683+0000 7f14a8a00640 1 -- 172.21.15.28:0/2078902971 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14a4003b70 con 0x7f1490004d40 2024-02-21T23:05:54.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.683+0000 7f14a8a00640 1 -- 172.21.15.28:0/2078902971 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a40050d0 con 0x7f1490004d40 2024-02-21T23:05:54.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.683+0000 7f14ab200640 1 -- 172.21.15.28:0/2078902971 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1490004d40 msgr2=0x7f14900051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:54.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.683+0000 7f14ab200640 1 --2- 172.21.15.28:0/2078902971 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1490004d40 0x7f14900051c0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f14a4001e50 tx=0x7f1490005830 comp rx=0 tx=0).stop 2024-02-21T23:05:54.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.683+0000 7f14aa800640 1 -- 172.21.15.28:0/2078902971 reap_dead start 2024-02-21T23:05:54.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.683+0000 7f14ab200640 1 -- 172.21.15.28:0/2078902971 shutdown_connections 2024-02-21T23:05:54.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.683+0000 7f14ab200640 1 -- 172.21.15.28:0/2078902971 >> 172.21.15.28:0/2078902971 conn(0x7f14ac0ac370 msgr2=0x7f14ac0ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:54.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.684+0000 7f14ab200640 1 -- 172.21.15.28:0/2078902971 shutdown_connections 2024-02-21T23:05:54.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.684+0000 7f14ab200640 1 -- 172.21.15.28:0/2078902971 wait complete. 2024-02-21T23:05:54.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.686+0000 7f14ab200640 1 Processor -- start 2024-02-21T23:05:54.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.687+0000 7f14ab200640 1 -- start start 2024-02-21T23:05:54.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.688+0000 7f14ab200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14ac1eded0 0x7f14ac1ee2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:54.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.688+0000 7f14ab200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f14ac0c45a0 0x7f14ac0be3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:54.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.688+0000 7f14ab200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f14ac0be930 0x7f14ac0c1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:54.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.688+0000 7f14ab200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14ac0c8760 con 0x7f14ac1eded0 2024-02-21T23:05:54.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.688+0000 7f14ab200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14ac158630 con 0x7f14ac0c45a0 2024-02-21T23:05:54.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.688+0000 7f14ab200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14ac0c88e0 con 0x7f14ac0be930 2024-02-21T23:05:54.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.688+0000 7f14a9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14ac1eded0 0x7f14ac1ee2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:54.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.688+0000 7f14a9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14ac1eded0 0x7f14ac1ee2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47920/0 (socket says 172.21.15.28:47920) 2024-02-21T23:05:54.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.688+0000 7f14a9e00640 1 -- 172.21.15.28:0/1939195510 learned_addr learned my addr 172.21.15.28:0/1939195510 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:54.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.689+0000 7f14a9400640 1 --2- 172.21.15.28:0/1939195510 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f14ac0c45a0 0x7f14ac0be3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:54.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.689+0000 7f14aa800640 1 --2- 172.21.15.28:0/1939195510 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f14ac0be930 0x7f14ac0c1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:54.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.689+0000 7f14a9e00640 1 -- 172.21.15.28:0/1939195510 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f14ac0be930 msgr2=0x7f14ac0c1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:54.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.689+0000 7f14a9e00640 1 --2- 172.21.15.28:0/1939195510 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f14ac0be930 0x7f14ac0c1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:54.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.689+0000 7f14a9e00640 1 -- 172.21.15.28:0/1939195510 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f14ac0c45a0 msgr2=0x7f14ac0be3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:54.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.689+0000 7f14a9e00640 1 --2- 172.21.15.28:0/1939195510 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f14ac0c45a0 0x7f14ac0be3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:54.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.689+0000 7f14a9e00640 1 -- 172.21.15.28:0/1939195510 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14ac1555e0 con 0x7f14ac1eded0 2024-02-21T23:05:54.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.689+0000 7f14a9e00640 1 --2- 172.21.15.28:0/1939195510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14ac1eded0 0x7f14ac1ee2b0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f149c011630 tx=0x7f149c011af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:54.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.690+0000 7f14a2000640 1 -- 172.21.15.28:0/1939195510 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f149c00f920 con 0x7f14ac1eded0 2024-02-21T23:05:54.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.690+0000 7f14a2000640 1 -- 172.21.15.28:0/1939195510 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f149c00fac0 con 0x7f14ac1eded0 2024-02-21T23:05:54.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.690+0000 7f14ab200640 1 -- 172.21.15.28:0/1939195510 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14ac0c3dc0 con 0x7f14ac1eded0 2024-02-21T23:05:54.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.690+0000 7f14a2000640 1 -- 172.21.15.28:0/1939195510 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f149c00fe70 con 0x7f14ac1eded0 2024-02-21T23:05:54.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.690+0000 7f14ab200640 1 -- 172.21.15.28:0/1939195510 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14ac1ee8c0 con 0x7f14ac1eded0 2024-02-21T23:05:54.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.691+0000 7f14ab200640 1 -- 172.21.15.28:0/1939195510 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14ac04f290 con 0x7f14ac1eded0 2024-02-21T23:05:54.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.692+0000 7f14a2000640 1 -- 172.21.15.28:0/1939195510 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f149c015460 con 0x7f14ac1eded0 2024-02-21T23:05:54.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.693+0000 7f14a2000640 1 --2- 172.21.15.28:0/1939195510 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f148c06b4d0 0x7f148c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:54.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.693+0000 7f14a2000640 1 -- 172.21.15.28:0/1939195510 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f149c08ac80 con 0x7f14ac1eded0 2024-02-21T23:05:54.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.694+0000 7f14a9400640 1 --2- 172.21.15.28:0/1939195510 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f148c06b4d0 0x7f148c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:54.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.695+0000 7f14a9400640 1 --2- 172.21.15.28:0/1939195510 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f148c06b4d0 0x7f148c06d990 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f1498001eb0 tx=0x7f1498009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:54.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.698+0000 7f14a2000640 1 -- 172.21.15.28:0/1939195510 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f149c019030 con 0x7f14ac1eded0 2024-02-21T23:05:54.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.937+0000 7f14ab200640 1 -- 172.21.15.28:0/1939195510 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f14ac06d9d0 con 0x7f14ac1eded0 2024-02-21T23:05:54.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.939+0000 7f14a2000640 1 -- 172.21.15.28:0/1939195510 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+4934 (secure 0 0 0) 0x7f149c059a00 con 0x7f14ac1eded0 2024-02-21T23:05:54.928 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:05:54.928 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":62,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5288,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/2774951107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2774951107},{"type":"v1","addr":"172.21.15.81:6835","nonce":2774951107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":57},{"gid":5294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/4292940632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":4292940632},{"type":"v1","addr":"172.21.15.81:6837","nonce":4292940632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":57},{"gid":5539,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/624384999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":624384999},{"type":"v1","addr":"172.21.15.28:6833","nonce":624384999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:05:50.828988+0000","modified":"2024-02-21T23:05:53.937407+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6366},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6366":{"gid":6366,"name":"a","rank":0,"incarnation":58,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/595682298","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":595682298},{"type":"v1","addr":"172.21.15.28:6835","nonce":595682298}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-02-21T23:05:54.929 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 62 2024-02-21T23:05:54.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.942+0000 7f14ab200640 1 -- 172.21.15.28:0/1939195510 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f148c06b4d0 msgr2=0x7f148c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:54.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.942+0000 7f14ab200640 1 --2- 172.21.15.28:0/1939195510 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f148c06b4d0 0x7f148c06d990 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f1498001eb0 tx=0x7f1498009040 comp rx=0 tx=0).stop 2024-02-21T23:05:54.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.942+0000 7f14ab200640 1 -- 172.21.15.28:0/1939195510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14ac1eded0 msgr2=0x7f14ac1ee2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:54.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.942+0000 7f14ab200640 1 --2- 172.21.15.28:0/1939195510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14ac1eded0 0x7f14ac1ee2b0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f149c011630 tx=0x7f149c011af0 comp rx=0 tx=0).stop 2024-02-21T23:05:54.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.942+0000 7f14ab200640 1 -- 172.21.15.28:0/1939195510 shutdown_connections 2024-02-21T23:05:54.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.942+0000 7f14ab200640 1 --2- 172.21.15.28:0/1939195510 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f148c06b4d0 0x7f148c06d990 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:54.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.942+0000 7f14ab200640 1 --2- 172.21.15.28:0/1939195510 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f14ac0be930 0x7f14ac0c1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:54.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.942+0000 7f14ab200640 1 --2- 172.21.15.28:0/1939195510 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f14ac0c45a0 0x7f14ac0be3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:54.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.942+0000 7f14ab200640 1 --2- 172.21.15.28:0/1939195510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14ac1eded0 0x7f14ac1ee2b0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:54.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.942+0000 7f14ab200640 1 -- 172.21.15.28:0/1939195510 >> 172.21.15.28:0/1939195510 conn(0x7f14ac0ac370 msgr2=0x7f14ac0ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:54.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.942+0000 7f14ab200640 1 -- 172.21.15.28:0/1939195510 shutdown_connections 2024-02-21T23:05:54.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:54.942+0000 7f14ab200640 1 -- 172.21.15.28:0/1939195510 wait complete. 2024-02-21T23:05:54.949 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:05:55.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.165+0000 7faa2e200640 1 Processor -- start 2024-02-21T23:05:55.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.165+0000 7faa2e200640 1 -- start start 2024-02-21T23:05:55.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.167+0000 7faa2e200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7faa2813e620 0x7faa28159070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:55.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.167+0000 7faa2e200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faa280c6460 0x7faa280ca8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:55.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.167+0000 7faa2e200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa281551b0 con 0x7faa280cadf0 2024-02-21T23:05:55.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.167+0000 7faa2e200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa280c71d0 con 0x7faa2813e620 2024-02-21T23:05:55.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.167+0000 7faa2e200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa281595b0 con 0x7faa280c6460 2024-02-21T23:05:55.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.167+0000 7faa27400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7faa2813e620 0x7faa28159070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:55.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.167+0000 7faa27400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7faa2813e620 0x7faa28159070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54094/0 (socket says 172.21.15.28:54094) 2024-02-21T23:05:55.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.167+0000 7faa27400640 1 -- 172.21.15.28:0/4179909060 learned_addr learned my addr 172.21.15.28:0/4179909060 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:55.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.168+0000 7faa1ea00640 1 --2- 172.21.15.28:0/4179909060 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faa280c6460 0x7faa280ca8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:55.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.168+0000 7faa27400640 1 -- 172.21.15.28:0/4179909060 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faa280c6460 msgr2=0x7faa280ca8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:55.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.168+0000 7faa27400640 1 --2- 172.21.15.28:0/4179909060 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faa280c6460 0x7faa280ca8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:55.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.168+0000 7faa27400640 1 -- 172.21.15.28:0/4179909060 >> v1:172.21.15.81:6789/0 conn(0x7faa280cadf0 legacy=0x7faa280bf990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:55.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.168+0000 7faa27400640 1 -- 172.21.15.28:0/4179909060 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa28155420 con 0x7faa2813e620 2024-02-21T23:05:55.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.169+0000 7faa27400640 1 --2- 172.21.15.28:0/4179909060 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa2813e620 0x7faa28159070 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7faa18009e50 tx=0x7faa1802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd54949a8280d38e server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:55.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.170+0000 7faa26a00640 1 -- 172.21.15.28:0/4179909060 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa1802ca80 con 0x7faa2813e620 2024-02-21T23:05:55.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.170+0000 7faa26a00640 1 -- 172.21.15.28:0/4179909060 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa1802cc20 con 0x7faa2813e620 2024-02-21T23:05:55.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.170+0000 7faa26a00640 1 -- 172.21.15.28:0/4179909060 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa180343f0 con 0x7faa2813e620 2024-02-21T23:05:55.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.170+0000 7faa2e200640 1 -- 172.21.15.28:0/4179909060 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa2813e620 msgr2=0x7faa28159070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:55.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.170+0000 7faa2e200640 1 --2- 172.21.15.28:0/4179909060 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa2813e620 0x7faa28159070 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7faa18009e50 tx=0x7faa1802bee0 comp rx=0 tx=0).stop 2024-02-21T23:05:55.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.171+0000 7faa2e200640 1 -- 172.21.15.28:0/4179909060 shutdown_connections 2024-02-21T23:05:55.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.171+0000 7faa2e200640 1 --2- 172.21.15.28:0/4179909060 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faa280c6460 0x7faa280ca8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:55.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.171+0000 7faa2e200640 1 --2- 172.21.15.28:0/4179909060 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa2813e620 0x7faa28159070 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:55.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.171+0000 7faa2e200640 1 -- 172.21.15.28:0/4179909060 >> 172.21.15.28:0/4179909060 conn(0x7faa2808b1c0 msgr2=0x7faa280725a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:55.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.171+0000 7faa2e200640 1 -- 172.21.15.28:0/4179909060 shutdown_connections 2024-02-21T23:05:55.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.171+0000 7faa2e200640 1 -- 172.21.15.28:0/4179909060 wait complete. 2024-02-21T23:05:55.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.173+0000 7faa2e200640 1 Processor -- start 2024-02-21T23:05:55.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.173+0000 7faa2e200640 1 -- start start 2024-02-21T23:05:55.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.174+0000 7faa2e200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faa280c6460 0x7faa281ea950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:55.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.174+0000 7faa2e200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa280cadf0 0x7faa281eae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:55.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.174+0000 7faa2e200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faa2813e620 0x7faa280aefc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:55.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.174+0000 7faa2e200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa28155cc0 con 0x7faa280cadf0 2024-02-21T23:05:55.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.174+0000 7faa2e200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa281559e0 con 0x7faa280c6460 2024-02-21T23:05:55.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.174+0000 7faa2e200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa2813ea00 con 0x7faa2813e620 2024-02-21T23:05:55.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.174+0000 7faa1ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa280cadf0 0x7faa281eae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:55.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.174+0000 7faa1ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa280cadf0 0x7faa281eae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54104/0 (socket says 172.21.15.28:54104) 2024-02-21T23:05:55.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.174+0000 7faa1ea00640 1 -- 172.21.15.28:0/3732329205 learned_addr learned my addr 172.21.15.28:0/3732329205 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:55.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.174+0000 7faa27400640 1 --2- 172.21.15.28:0/3732329205 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faa280c6460 0x7faa281ea950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:55.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.174+0000 7faa27e00640 1 --2- 172.21.15.28:0/3732329205 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faa2813e620 0x7faa280aefc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:55.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.174+0000 7faa1ea00640 1 -- 172.21.15.28:0/3732329205 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faa2813e620 msgr2=0x7faa280aefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:55.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.174+0000 7faa1ea00640 1 --2- 172.21.15.28:0/3732329205 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faa2813e620 0x7faa280aefc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:55.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.174+0000 7faa1ea00640 1 -- 172.21.15.28:0/3732329205 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faa280c6460 msgr2=0x7faa281ea950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:55.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.174+0000 7faa1ea00640 1 --2- 172.21.15.28:0/3732329205 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faa280c6460 0x7faa281ea950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:55.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.174+0000 7faa1ea00640 1 -- 172.21.15.28:0/3732329205 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa281eb470 con 0x7faa280cadf0 2024-02-21T23:05:55.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.174+0000 7faa27e00640 1 --2- 172.21.15.28:0/3732329205 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faa2813e620 0x7faa280aefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:05:55.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.175+0000 7faa1ea00640 1 --2- 172.21.15.28:0/3732329205 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa280cadf0 0x7faa281eae90 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7faa1000d5d0 tx=0x7faa1000da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:55.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.175+0000 7faa1fe00640 1 -- 172.21.15.28:0/3732329205 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa100125f0 con 0x7faa280cadf0 2024-02-21T23:05:55.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.175+0000 7faa1fe00640 1 -- 172.21.15.28:0/3732329205 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa10019070 con 0x7faa280cadf0 2024-02-21T23:05:55.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.175+0000 7faa1fe00640 1 -- 172.21.15.28:0/3732329205 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa100129f0 con 0x7faa280cadf0 2024-02-21T23:05:55.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.175+0000 7faa2e200640 1 -- 172.21.15.28:0/3732329205 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa281eb750 con 0x7faa280cadf0 2024-02-21T23:05:55.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.175+0000 7faa2e200640 1 -- 172.21.15.28:0/3732329205 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa280c8c50 con 0x7faa280cadf0 2024-02-21T23:05:55.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.177+0000 7faa2e200640 1 -- 172.21.15.28:0/3732329205 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa280c8e50 con 0x7faa280cadf0 2024-02-21T23:05:55.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.177+0000 7faa1fe00640 1 -- 172.21.15.28:0/3732329205 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faa10012b90 con 0x7faa280cadf0 2024-02-21T23:05:55.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.178+0000 7faa1fe00640 1 --2- 172.21.15.28:0/3732329205 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7faa0406b4d0 0x7faa0406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:55.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.178+0000 7faa1fe00640 1 -- 172.21.15.28:0/3732329205 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6101+0+0 (secure 0 0 0) 0x7faa1008a7e0 con 0x7faa280cadf0 2024-02-21T23:05:55.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.179+0000 7faa27400640 1 --2- 172.21.15.28:0/3732329205 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7faa0406b4d0 0x7faa0406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:55.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.180+0000 7faa27400640 1 --2- 172.21.15.28:0/3732329205 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7faa0406b4d0 0x7faa0406d990 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7faa18035000 tx=0x7faa18030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:55.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.183+0000 7faa1fe00640 1 -- 172.21.15.28:0/3732329205 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7faa280c8e50 con 0x7faa280cadf0 2024-02-21T23:05:55.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.473+0000 7faa2e200640 1 -- 172.21.15.28:0/3732329205 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faa2813c050 con 0x7faa280cadf0 2024-02-21T23:05:55.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.475+0000 7faa1fe00640 1 -- 172.21.15.28:0/3732329205 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) v1 ==== 74+0+16885 (secure 0 0 0) 0x7faa100594c0 con 0x7faa280cadf0 2024-02-21T23:05:55.463 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:05:55.464 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":76,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:05:52.373861+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:05:48.772562+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-02-21T23:05:49.852659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:05:55.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.477+0000 7faa2e200640 1 -- 172.21.15.28:0/3732329205 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7faa0406b4d0 msgr2=0x7faa0406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:55.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.477+0000 7faa2e200640 1 --2- 172.21.15.28:0/3732329205 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7faa0406b4d0 0x7faa0406d990 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7faa18035000 tx=0x7faa18030000 comp rx=0 tx=0).stop 2024-02-21T23:05:55.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.477+0000 7faa2e200640 1 -- 172.21.15.28:0/3732329205 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa280cadf0 msgr2=0x7faa281eae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:55.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.477+0000 7faa2e200640 1 --2- 172.21.15.28:0/3732329205 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa280cadf0 0x7faa281eae90 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7faa1000d5d0 tx=0x7faa1000da90 comp rx=0 tx=0).stop 2024-02-21T23:05:55.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.478+0000 7faa2e200640 1 -- 172.21.15.28:0/3732329205 shutdown_connections 2024-02-21T23:05:55.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.478+0000 7faa2e200640 1 --2- 172.21.15.28:0/3732329205 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7faa0406b4d0 0x7faa0406d990 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:55.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.478+0000 7faa2e200640 1 --2- 172.21.15.28:0/3732329205 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faa2813e620 0x7faa280aefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:55.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.478+0000 7faa2e200640 1 --2- 172.21.15.28:0/3732329205 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa280cadf0 0x7faa281eae90 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:55.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.478+0000 7faa2e200640 1 --2- 172.21.15.28:0/3732329205 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faa280c6460 0x7faa281ea950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:55.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.478+0000 7faa2e200640 1 -- 172.21.15.28:0/3732329205 >> 172.21.15.28:0/3732329205 conn(0x7faa2808b1c0 msgr2=0x7faa2813c9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:55.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.478+0000 7faa2e200640 1 -- 172.21.15.28:0/3732329205 shutdown_connections 2024-02-21T23:05:55.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.478+0000 7faa2e200640 1 -- 172.21.15.28:0/3732329205 wait complete. 2024-02-21T23:05:55.484 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-02-21T23:05:55.484 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-21T23:05:55.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.713+0000 7fbca9600640 1 Processor -- start 2024-02-21T23:05:55.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.713+0000 7fbca9600640 1 -- start start 2024-02-21T23:05:55.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.714+0000 7fbca9600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbca40aff90 0x7fbca40c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:55.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.714+0000 7fbca9600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbca406dbf0 0x7fbca4158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:55.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.714+0000 7fbca9600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbca40c74c0 con 0x7fbca40c5720 2024-02-21T23:05:55.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.714+0000 7fbca9600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbca40c7270 con 0x7fbca40aff90 2024-02-21T23:05:55.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.714+0000 7fbca9600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbca4158630 con 0x7fbca406dbf0 2024-02-21T23:05:55.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.715+0000 7fbca3e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbca40aff90 0x7fbca40c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:55.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.715+0000 7fbca3e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbca40aff90 0x7fbca40c51e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54112/0 (socket says 172.21.15.28:54112) 2024-02-21T23:05:55.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.715+0000 7fbca3e00640 1 -- 172.21.15.28:0/3469491474 learned_addr learned my addr 172.21.15.28:0/3469491474 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:55.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.715+0000 7fbca3e00640 1 -- 172.21.15.28:0/3469491474 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbca406dbf0 msgr2=0x7fbca4158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:55.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.715+0000 7fbca3e00640 1 --2- 172.21.15.28:0/3469491474 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbca406dbf0 0x7fbca4158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:55.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.715+0000 7fbca3e00640 1 -- 172.21.15.28:0/3469491474 >> v1:172.21.15.81:6789/0 conn(0x7fbca40c5720 legacy=0x7fbca406c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:55.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.715+0000 7fbca3e00640 1 -- 172.21.15.28:0/3469491474 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbca40c77a0 con 0x7fbca40aff90 2024-02-21T23:05:55.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.716+0000 7fbca3e00640 1 --2- 172.21.15.28:0/3469491474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbca40aff90 0x7fbca40c51e0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fbc94009870 tx=0x7fbc9402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=173bc0bfcae850fc server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:55.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.717+0000 7fbca3400640 1 -- 172.21.15.28:0/3469491474 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc94035020 con 0x7fbca40aff90 2024-02-21T23:05:55.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.717+0000 7fbca3400640 1 -- 172.21.15.28:0/3469491474 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc94031640 con 0x7fbca40aff90 2024-02-21T23:05:55.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.717+0000 7fbca3400640 1 -- 172.21.15.28:0/3469491474 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc9402d040 con 0x7fbca40aff90 2024-02-21T23:05:55.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.717+0000 7fbca9600640 1 -- 172.21.15.28:0/3469491474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbca40aff90 msgr2=0x7fbca40c51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:55.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.717+0000 7fbca9600640 1 --2- 172.21.15.28:0/3469491474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbca40aff90 0x7fbca40c51e0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fbc94009870 tx=0x7fbc9402b8f0 comp rx=0 tx=0).stop 2024-02-21T23:05:55.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.717+0000 7fbca9600640 1 -- 172.21.15.28:0/3469491474 shutdown_connections 2024-02-21T23:05:55.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.717+0000 7fbca9600640 1 --2- 172.21.15.28:0/3469491474 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbca406dbf0 0x7fbca4158ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:55.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.717+0000 7fbca9600640 1 --2- 172.21.15.28:0/3469491474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbca40aff90 0x7fbca40c51e0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:55.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.717+0000 7fbca9600640 1 -- 172.21.15.28:0/3469491474 >> 172.21.15.28:0/3469491474 conn(0x7fbca40ac370 msgr2=0x7fbca40ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:55.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.717+0000 7fbca9600640 1 -- 172.21.15.28:0/3469491474 shutdown_connections 2024-02-21T23:05:55.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.717+0000 7fbca9600640 1 -- 172.21.15.28:0/3469491474 wait complete. 2024-02-21T23:05:55.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.720+0000 7fbca9600640 1 Processor -- start 2024-02-21T23:05:55.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.720+0000 7fbca9600640 1 -- start start 2024-02-21T23:05:55.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.722+0000 7fbca9600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbca406dbf0 0x7fbca414b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:55.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.722+0000 7fbca9600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbca40aff90 0x7fbca40ae4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:55.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.722+0000 7fbca9600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbca40c5720 0x7fbca41eae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:55.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.722+0000 7fbca9600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbca40c8760 con 0x7fbca40c5720 2024-02-21T23:05:55.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.722+0000 7fbca9600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbca40be0c0 con 0x7fbca406dbf0 2024-02-21T23:05:55.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.722+0000 7fbca9600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbca40c88e0 con 0x7fbca40aff90 2024-02-21T23:05:55.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.722+0000 7fbca8c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbca40c5720 0x7fbca41eae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:55.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.722+0000 7fbca8c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbca40c5720 0x7fbca41eae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54120/0 (socket says 172.21.15.28:54120) 2024-02-21T23:05:55.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.722+0000 7fbca8c00640 1 -- 172.21.15.28:0/2238256619 learned_addr learned my addr 172.21.15.28:0/2238256619 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:55.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.723+0000 7fbc9be00640 1 --2- 172.21.15.28:0/2238256619 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbca40aff90 0x7fbca40ae4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:55.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.723+0000 7fbca3e00640 1 --2- 172.21.15.28:0/2238256619 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbca406dbf0 0x7fbca414b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:55.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.723+0000 7fbca8c00640 1 -- 172.21.15.28:0/2238256619 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbca40aff90 msgr2=0x7fbca40ae4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:55.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.723+0000 7fbca8c00640 1 --2- 172.21.15.28:0/2238256619 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbca40aff90 0x7fbca40ae4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:55.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.723+0000 7fbca8c00640 1 -- 172.21.15.28:0/2238256619 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbca406dbf0 msgr2=0x7fbca414b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:55.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.723+0000 7fbca8c00640 1 --2- 172.21.15.28:0/2238256619 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbca406dbf0 0x7fbca414b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:55.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.723+0000 7fbca8c00640 1 -- 172.21.15.28:0/2238256619 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbca41eeda0 con 0x7fbca40c5720 2024-02-21T23:05:55.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.723+0000 7fbca8c00640 1 --2- 172.21.15.28:0/2238256619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbca40c5720 0x7fbca41eae80 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c00b990 tx=0x7fbc9c00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:55.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.723+0000 7fbca0c00640 1 -- 172.21.15.28:0/2238256619 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc9c019070 con 0x7fbca40c5720 2024-02-21T23:05:55.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.723+0000 7fbca0c00640 1 -- 172.21.15.28:0/2238256619 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc9c00cba0 con 0x7fbca40c5720 2024-02-21T23:05:55.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.723+0000 7fbca0c00640 1 -- 172.21.15.28:0/2238256619 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc9c014440 con 0x7fbca40c5720 2024-02-21T23:05:55.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.724+0000 7fbca9600640 1 -- 172.21.15.28:0/2238256619 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbca41ef080 con 0x7fbca40c5720 2024-02-21T23:05:55.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.724+0000 7fbca9600640 1 -- 172.21.15.28:0/2238256619 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbca40af0d0 con 0x7fbca40c5720 2024-02-21T23:05:55.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.725+0000 7fbca9600640 1 -- 172.21.15.28:0/2238256619 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbca404f290 con 0x7fbca40c5720 2024-02-21T23:05:55.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.726+0000 7fbca0c00640 1 -- 172.21.15.28:0/2238256619 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbc9c01c070 con 0x7fbca40c5720 2024-02-21T23:05:55.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.727+0000 7fbca0c00640 1 --2- 172.21.15.28:0/2238256619 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbc8006b500 0x7fbc8006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:55.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.727+0000 7fbca0c00640 1 -- 172.21.15.28:0/2238256619 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fbc9c08a140 con 0x7fbca40c5720 2024-02-21T23:05:55.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.728+0000 7fbca3e00640 1 --2- 172.21.15.28:0/2238256619 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbc8006b500 0x7fbc8006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:55.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.729+0000 7fbca3e00640 1 --2- 172.21.15.28:0/2238256619 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbc8006b500 0x7fbc8006d9c0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fbc94036000 tx=0x7fbc9403c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:55.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.732+0000 7fbca0c00640 1 -- 172.21.15.28:0/2238256619 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fbc9c058e20 con 0x7fbca40c5720 2024-02-21T23:05:55.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.952+0000 7fbca9600640 1 -- 172.21.15.28:0/2238256619 --> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbca406d9d0 con 0x7fbc8006b500 2024-02-21T23:05:55.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.971+0000 7fbca0c00640 1 -- 172.21.15.28:0/2238256619 <== mgr.4101 v2:172.21.15.81:6832/1848538577 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246827 (secure 0 0 0) 0x7fbca406d9d0 con 0x7fbc8006b500 2024-02-21T23:05:55.963 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:05:55.965 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-21T23:05:55.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.976+0000 7fbca9600640 1 -- 172.21.15.28:0/2238256619 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbc8006b500 msgr2=0x7fbc8006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:55.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.976+0000 7fbca9600640 1 --2- 172.21.15.28:0/2238256619 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbc8006b500 0x7fbc8006d9c0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fbc94036000 tx=0x7fbc9403c040 comp rx=0 tx=0).stop 2024-02-21T23:05:55.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.976+0000 7fbca9600640 1 -- 172.21.15.28:0/2238256619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbca40c5720 msgr2=0x7fbca41eae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:55.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.976+0000 7fbca9600640 1 --2- 172.21.15.28:0/2238256619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbca40c5720 0x7fbca41eae80 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c00b990 tx=0x7fbc9c00be50 comp rx=0 tx=0).stop 2024-02-21T23:05:55.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.977+0000 7fbca9600640 1 -- 172.21.15.28:0/2238256619 shutdown_connections 2024-02-21T23:05:55.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.977+0000 7fbca9600640 1 --2- 172.21.15.28:0/2238256619 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbc8006b500 0x7fbc8006d9c0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:55.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.977+0000 7fbca9600640 1 --2- 172.21.15.28:0/2238256619 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbca40aff90 0x7fbca40ae4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:55.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.977+0000 7fbca9600640 1 --2- 172.21.15.28:0/2238256619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbca40c5720 0x7fbca41eae80 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:55.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.977+0000 7fbca9600640 1 --2- 172.21.15.28:0/2238256619 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbca406dbf0 0x7fbca414b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:55.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.977+0000 7fbca9600640 1 -- 172.21.15.28:0/2238256619 >> 172.21.15.28:0/2238256619 conn(0x7fbca40ac370 msgr2=0x7fbca40ac750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:55.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.977+0000 7fbca9600640 1 -- 172.21.15.28:0/2238256619 shutdown_connections 2024-02-21T23:05:55.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:55.977+0000 7fbca9600640 1 -- 172.21.15.28:0/2238256619 wait complete. 2024-02-21T23:05:55.984 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":171,"stamp":"2024-02-21T23:05:54.734430+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":33,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":239968,"kb_used_data":7164,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754734752,"statfs":{"total":773094113280,"available":772848386048,"internally_reserved":0,"allocated":7335936,"data_stored":4598007,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.399420"},"pg_stats":[{"pgid":"12.3f","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.370547+0000","last_change":"2024-02-21T23:05:50.370547+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.370547+0000","last_undegraded":"2024-02-21T23:05:50.370547+0000","last_fullsized":"2024-02-21T23:05:50.370547+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.367674+0000","last_change":"2024-02-21T23:05:50.367674+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.367674+0000","last_undegraded":"2024-02-21T23:05:50.367674+0000","last_fullsized":"2024-02-21T23:05:50.367674+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.364429+0000","last_change":"2024-02-21T23:05:50.364429+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.364429+0000","last_undegraded":"2024-02-21T23:05:50.364429+0000","last_fullsized":"2024-02-21T23:05:50.364429+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.368568+0000","last_change":"2024-02-21T23:05:50.368568+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.368568+0000","last_undegraded":"2024-02-21T23:05:50.368568+0000","last_fullsized":"2024-02-21T23:05:50.368568+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.366342+0000","last_change":"2024-02-21T23:05:50.366342+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.366342+0000","last_undegraded":"2024-02-21T23:05:50.366342+0000","last_fullsized":"2024-02-21T23:05:50.366342+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.348142+0000","last_change":"2024-02-21T23:05:50.348142+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.348142+0000","last_undegraded":"2024-02-21T23:05:50.348142+0000","last_fullsized":"2024-02-21T23:05:50.348142+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.364864+0000","last_change":"2024-02-21T23:05:50.364864+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.364864+0000","last_undegraded":"2024-02-21T23:05:50.364864+0000","last_fullsized":"2024-02-21T23:05:50.364864+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.366485+0000","last_change":"2024-02-21T23:05:50.366485+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.366485+0000","last_undegraded":"2024-02-21T23:05:50.366485+0000","last_fullsized":"2024-02-21T23:05:50.366485+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.348832+0000","last_change":"2024-02-21T23:05:50.348832+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.348832+0000","last_undegraded":"2024-02-21T23:05:50.348832+0000","last_fullsized":"2024-02-21T23:05:50.348832+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.363423+0000","last_change":"2024-02-21T23:05:50.363423+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.363423+0000","last_undegraded":"2024-02-21T23:05:50.363423+0000","last_fullsized":"2024-02-21T23:05:50.363423+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.365264+0000","last_change":"2024-02-21T23:05:50.365264+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.365264+0000","last_undegraded":"2024-02-21T23:05:50.365264+0000","last_fullsized":"2024-02-21T23:05:50.365264+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.368080+0000","last_change":"2024-02-21T23:05:50.368080+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.368080+0000","last_undegraded":"2024-02-21T23:05:50.368080+0000","last_fullsized":"2024-02-21T23:05:50.368080+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.364931+0000","last_change":"2024-02-21T23:05:50.364931+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.364931+0000","last_undegraded":"2024-02-21T23:05:50.364931+0000","last_fullsized":"2024-02-21T23:05:50.364931+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.347638+0000","last_change":"2024-02-21T23:05:50.347638+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.347638+0000","last_undegraded":"2024-02-21T23:05:50.347638+0000","last_fullsized":"2024-02-21T23:05:50.347638+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.364788+0000","last_change":"2024-02-21T23:05:50.364788+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.364788+0000","last_undegraded":"2024-02-21T23:05:50.364788+0000","last_fullsized":"2024-02-21T23:05:50.364788+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.365889+0000","last_change":"2024-02-21T23:05:50.365889+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.365889+0000","last_undegraded":"2024-02-21T23:05:50.365889+0000","last_fullsized":"2024-02-21T23:05:50.365889+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.364734+0000","last_change":"2024-02-21T23:05:50.364734+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.364734+0000","last_undegraded":"2024-02-21T23:05:50.364734+0000","last_fullsized":"2024-02-21T23:05:50.364734+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.367567+0000","last_change":"2024-02-21T23:05:50.367567+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.367567+0000","last_undegraded":"2024-02-21T23:05:50.367567+0000","last_fullsized":"2024-02-21T23:05:50.367567+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.346931+0000","last_change":"2024-02-21T23:05:50.346931+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.346931+0000","last_undegraded":"2024-02-21T23:05:50.346931+0000","last_fullsized":"2024-02-21T23:05:50.346931+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.367396+0000","last_change":"2024-02-21T23:05:50.367396+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.367396+0000","last_undegraded":"2024-02-21T23:05:50.367396+0000","last_fullsized":"2024-02-21T23:05:50.367396+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.350181+0000","last_change":"2024-02-21T23:05:50.350181+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.350181+0000","last_undegraded":"2024-02-21T23:05:50.350181+0000","last_fullsized":"2024-02-21T23:05:50.350181+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.362886+0000","last_change":"2024-02-21T23:05:50.362886+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.362886+0000","last_undegraded":"2024-02-21T23:05:50.362886+0000","last_fullsized":"2024-02-21T23:05:50.362886+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.349267+0000","last_change":"2024-02-21T23:05:50.349267+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.349267+0000","last_undegraded":"2024-02-21T23:05:50.349267+0000","last_fullsized":"2024-02-21T23:05:50.349267+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.363272+0000","last_change":"2024-02-21T23:05:50.363272+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.363272+0000","last_undegraded":"2024-02-21T23:05:50.363272+0000","last_fullsized":"2024-02-21T23:05:50.363272+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.361928+0000","last_change":"2024-02-21T23:05:50.361928+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.361928+0000","last_undegraded":"2024-02-21T23:05:50.361928+0000","last_fullsized":"2024-02-21T23:05:50.361928+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.362770+0000","last_change":"2024-02-21T23:05:50.362770+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.362770+0000","last_undegraded":"2024-02-21T23:05:50.362770+0000","last_fullsized":"2024-02-21T23:05:50.362770+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.370781+0000","last_change":"2024-02-21T23:05:50.370781+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.370781+0000","last_undegraded":"2024-02-21T23:05:50.370781+0000","last_fullsized":"2024-02-21T23:05:50.370781+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.363453+0000","last_change":"2024-02-21T23:05:50.363453+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.363453+0000","last_undegraded":"2024-02-21T23:05:50.363453+0000","last_fullsized":"2024-02-21T23:05:50.363453+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.361847+0000","last_change":"2024-02-21T23:05:50.361847+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.361847+0000","last_undegraded":"2024-02-21T23:05:50.361847+0000","last_fullsized":"2024-02-21T23:05:50.361847+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.366516+0000","last_change":"2024-02-21T23:05:50.366516+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.366516+0000","last_undegraded":"2024-02-21T23:05:50.366516+0000","last_fullsized":"2024-02-21T23:05:50.366516+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.364518+0000","last_change":"2024-02-21T23:05:50.364518+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.364518+0000","last_undegraded":"2024-02-21T23:05:50.364518+0000","last_fullsized":"2024-02-21T23:05:50.364518+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.365266+0000","last_change":"2024-02-21T23:05:50.365266+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.365266+0000","last_undegraded":"2024-02-21T23:05:50.365266+0000","last_fullsized":"2024-02-21T23:05:50.365266+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.363578+0000","last_change":"2024-02-21T23:05:50.363578+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.363578+0000","last_undegraded":"2024-02-21T23:05:50.363578+0000","last_fullsized":"2024-02-21T23:05:50.363578+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-21T23:05:50.356881+0000","last_change":"2024-02-21T23:05:50.354109+0000","last_active":"2024-02-21T23:05:50.356881+0000","last_peered":"2024-02-21T23:05:50.356881+0000","last_clean":"2024-02-21T23:05:50.356881+0000","last_became_active":"2024-02-21T23:05:50.353809+0000","last_became_peered":"2024-02-21T23:05:50.353809+0000","last_unstale":"2024-02-21T23:05:50.356881+0000","last_undegraded":"2024-02-21T23:05:50.356881+0000","last_fullsized":"2024-02-21T23:05:50.356881+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:05:10.036678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.363179+0000","last_change":"2024-02-21T23:05:50.363179+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.363179+0000","last_undegraded":"2024-02-21T23:05:50.363179+0000","last_fullsized":"2024-02-21T23:05:50.363179+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-21T23:05:50.386939+0000","last_change":"2024-02-21T23:05:50.383438+0000","last_active":"2024-02-21T23:05:50.386939+0000","last_peered":"2024-02-21T23:05:50.386939+0000","last_clean":"2024-02-21T23:05:50.386939+0000","last_became_active":"2024-02-21T23:05:50.383134+0000","last_became_peered":"2024-02-21T23:05:50.383134+0000","last_unstale":"2024-02-21T23:05:50.386939+0000","last_undegraded":"2024-02-21T23:05:50.386939+0000","last_fullsized":"2024-02-21T23:05:50.386939+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:58:54.817119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.349716+0000","last_change":"2024-02-21T23:05:50.349716+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.349716+0000","last_undegraded":"2024-02-21T23:05:50.349716+0000","last_fullsized":"2024-02-21T23:05:50.349716+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-21T23:05:50.386971+0000","last_change":"2024-02-21T23:05:50.383231+0000","last_active":"2024-02-21T23:05:50.386971+0000","last_peered":"2024-02-21T23:05:50.386971+0000","last_clean":"2024-02-21T23:05:50.386971+0000","last_became_active":"2024-02-21T23:05:50.382757+0000","last_became_peered":"2024-02-21T23:05:50.382757+0000","last_unstale":"2024-02-21T23:05:50.386971+0000","last_undegraded":"2024-02-21T23:05:50.386971+0000","last_fullsized":"2024-02-21T23:05:50.386971+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:34:13.407270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.348408+0000","last_change":"2024-02-21T23:05:50.348408+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.348408+0000","last_undegraded":"2024-02-21T23:05:50.348408+0000","last_fullsized":"2024-02-21T23:05:50.348408+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-21T23:05:50.381772+0000","last_change":"2024-02-21T23:05:50.376050+0000","last_active":"2024-02-21T23:05:50.381772+0000","last_peered":"2024-02-21T23:05:50.381772+0000","last_clean":"2024-02-21T23:05:50.381772+0000","last_became_active":"2024-02-21T23:05:50.375731+0000","last_became_peered":"2024-02-21T23:05:50.375731+0000","last_unstale":"2024-02-21T23:05:50.381772+0000","last_undegraded":"2024-02-21T23:05:50.381772+0000","last_fullsized":"2024-02-21T23:05:50.381772+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:03:49.547047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.361723+0000","last_change":"2024-02-21T23:05:50.361723+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.361723+0000","last_undegraded":"2024-02-21T23:05:50.361723+0000","last_fullsized":"2024-02-21T23:05:50.361723+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-21T23:05:50.381028+0000","last_change":"2024-02-21T23:05:50.377574+0000","last_active":"2024-02-21T23:05:50.381028+0000","last_peered":"2024-02-21T23:05:50.381028+0000","last_clean":"2024-02-21T23:05:50.381028+0000","last_became_active":"2024-02-21T23:05:50.377250+0000","last_became_peered":"2024-02-21T23:05:50.377250+0000","last_unstale":"2024-02-21T23:05:50.381028+0000","last_undegraded":"2024-02-21T23:05:50.381028+0000","last_fullsized":"2024-02-21T23:05:50.381028+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:40:31.369030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.363653+0000","last_change":"2024-02-21T23:05:50.363653+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.363653+0000","last_undegraded":"2024-02-21T23:05:50.363653+0000","last_fullsized":"2024-02-21T23:05:50.363653+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-21T23:05:50.381197+0000","last_change":"2024-02-21T23:05:50.378668+0000","last_active":"2024-02-21T23:05:50.381197+0000","last_peered":"2024-02-21T23:05:50.381197+0000","last_clean":"2024-02-21T23:05:50.381197+0000","last_became_active":"2024-02-21T23:05:50.378369+0000","last_became_peered":"2024-02-21T23:05:50.378369+0000","last_unstale":"2024-02-21T23:05:50.381197+0000","last_undegraded":"2024-02-21T23:05:50.381197+0000","last_fullsized":"2024-02-21T23:05:50.381197+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:10:10.973567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.348744+0000","last_change":"2024-02-21T23:05:50.348744+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.348744+0000","last_undegraded":"2024-02-21T23:05:50.348744+0000","last_fullsized":"2024-02-21T23:05:50.348744+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-21T23:05:50.379722+0000","last_change":"2024-02-21T23:05:50.375534+0000","last_active":"2024-02-21T23:05:50.379722+0000","last_peered":"2024-02-21T23:05:50.379722+0000","last_clean":"2024-02-21T23:05:50.379722+0000","last_became_active":"2024-02-21T23:05:50.373762+0000","last_became_peered":"2024-02-21T23:05:50.373762+0000","last_unstale":"2024-02-21T23:05:50.379722+0000","last_undegraded":"2024-02-21T23:05:50.379722+0000","last_fullsized":"2024-02-21T23:05:50.379722+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:56:09.393950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.364589+0000","last_change":"2024-02-21T23:05:50.364589+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.364589+0000","last_undegraded":"2024-02-21T23:05:50.364589+0000","last_fullsized":"2024-02-21T23:05:50.364589+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-21T23:05:50.363824+0000","last_change":"2024-02-21T23:05:50.355567+0000","last_active":"2024-02-21T23:05:50.363824+0000","last_peered":"2024-02-21T23:05:50.363824+0000","last_clean":"2024-02-21T23:05:50.363824+0000","last_became_active":"2024-02-21T23:05:50.354835+0000","last_became_peered":"2024-02-21T23:05:50.354835+0000","last_unstale":"2024-02-21T23:05:50.363824+0000","last_undegraded":"2024-02-21T23:05:50.363824+0000","last_fullsized":"2024-02-21T23:05:50.363824+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:49:06.159588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.361014+0000","last_change":"2024-02-21T23:05:50.361014+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.361014+0000","last_undegraded":"2024-02-21T23:05:50.361014+0000","last_fullsized":"2024-02-21T23:05:50.361014+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-21T23:05:50.391190+0000","last_change":"2024-02-21T23:05:50.386639+0000","last_active":"2024-02-21T23:05:50.391190+0000","last_peered":"2024-02-21T23:05:50.391190+0000","last_clean":"2024-02-21T23:05:50.391190+0000","last_became_active":"2024-02-21T23:05:50.386158+0000","last_became_peered":"2024-02-21T23:05:50.386158+0000","last_unstale":"2024-02-21T23:05:50.391190+0000","last_undegraded":"2024-02-21T23:05:50.391190+0000","last_fullsized":"2024-02-21T23:05:50.391190+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:56:09.393950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.358895+0000","last_change":"2024-02-21T23:05:50.358895+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.358895+0000","last_undegraded":"2024-02-21T23:05:50.358895+0000","last_fullsized":"2024-02-21T23:05:50.358895+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-21T23:05:50.386510+0000","last_change":"2024-02-21T23:05:50.385146+0000","last_active":"2024-02-21T23:05:50.386510+0000","last_peered":"2024-02-21T23:05:50.386510+0000","last_clean":"2024-02-21T23:05:50.386510+0000","last_became_active":"2024-02-21T23:05:50.384418+0000","last_became_peered":"2024-02-21T23:05:50.384418+0000","last_unstale":"2024-02-21T23:05:50.386510+0000","last_undegraded":"2024-02-21T23:05:50.386510+0000","last_fullsized":"2024-02-21T23:05:50.386510+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:49:36.738197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.346234+0000","last_change":"2024-02-21T23:05:50.346234+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.346234+0000","last_undegraded":"2024-02-21T23:05:50.346234+0000","last_fullsized":"2024-02-21T23:05:50.346234+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-21T23:05:50.376035+0000","last_change":"2024-02-21T23:05:50.373189+0000","last_active":"2024-02-21T23:05:50.376035+0000","last_peered":"2024-02-21T23:05:50.376035+0000","last_clean":"2024-02-21T23:05:50.376035+0000","last_became_active":"2024-02-21T23:05:50.370680+0000","last_became_peered":"2024-02-21T23:05:50.370680+0000","last_unstale":"2024-02-21T23:05:50.376035+0000","last_undegraded":"2024-02-21T23:05:50.376035+0000","last_fullsized":"2024-02-21T23:05:50.376035+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:10:25.735830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.360832+0000","last_change":"2024-02-21T23:05:50.360832+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.360832+0000","last_undegraded":"2024-02-21T23:05:50.360832+0000","last_fullsized":"2024-02-21T23:05:50.360832+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-21T23:05:50.379685+0000","last_change":"2024-02-21T23:05:50.374208+0000","last_active":"2024-02-21T23:05:50.379685+0000","last_peered":"2024-02-21T23:05:50.379685+0000","last_clean":"2024-02-21T23:05:50.379685+0000","last_became_active":"2024-02-21T23:05:50.373869+0000","last_became_peered":"2024-02-21T23:05:50.373869+0000","last_unstale":"2024-02-21T23:05:50.379685+0000","last_undegraded":"2024-02-21T23:05:50.379685+0000","last_fullsized":"2024-02-21T23:05:50.379685+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:16:32.220730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.365281+0000","last_change":"2024-02-21T23:05:50.365281+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.365281+0000","last_undegraded":"2024-02-21T23:05:50.365281+0000","last_fullsized":"2024-02-21T23:05:50.365281+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-21T23:05:50.384116+0000","last_change":"2024-02-21T23:05:50.381643+0000","last_active":"2024-02-21T23:05:50.384116+0000","last_peered":"2024-02-21T23:05:50.384116+0000","last_clean":"2024-02-21T23:05:50.384116+0000","last_became_active":"2024-02-21T23:05:50.381345+0000","last_became_peered":"2024-02-21T23:05:50.381345+0000","last_unstale":"2024-02-21T23:05:50.384116+0000","last_undegraded":"2024-02-21T23:05:50.384116+0000","last_fullsized":"2024-02-21T23:05:50.384116+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:26:58.076002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.359127+0000","last_change":"2024-02-21T23:05:50.359127+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.359127+0000","last_undegraded":"2024-02-21T23:05:50.359127+0000","last_fullsized":"2024-02-21T23:05:50.359127+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-21T23:05:50.386822+0000","last_change":"2024-02-21T23:05:50.381962+0000","last_active":"2024-02-21T23:05:50.386822+0000","last_peered":"2024-02-21T23:05:50.386822+0000","last_clean":"2024-02-21T23:05:50.386822+0000","last_became_active":"2024-02-21T23:05:50.380838+0000","last_became_peered":"2024-02-21T23:05:50.380838+0000","last_unstale":"2024-02-21T23:05:50.386822+0000","last_undegraded":"2024-02-21T23:05:50.386822+0000","last_fullsized":"2024-02-21T23:05:50.386822+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:26:32.353509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.367278+0000","last_change":"2024-02-21T23:05:50.367278+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.367278+0000","last_undegraded":"2024-02-21T23:05:50.367278+0000","last_fullsized":"2024-02-21T23:05:50.367278+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-21T23:05:50.356314+0000","last_change":"2024-02-21T23:05:50.353116+0000","last_active":"2024-02-21T23:05:50.356314+0000","last_peered":"2024-02-21T23:05:50.356314+0000","last_clean":"2024-02-21T23:05:50.356314+0000","last_became_active":"2024-02-21T23:05:50.352768+0000","last_became_peered":"2024-02-21T23:05:50.352768+0000","last_unstale":"2024-02-21T23:05:50.356314+0000","last_undegraded":"2024-02-21T23:05:50.356314+0000","last_fullsized":"2024-02-21T23:05:50.356314+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:34:30.710951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.363438+0000","last_change":"2024-02-21T23:05:50.363438+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.363438+0000","last_undegraded":"2024-02-21T23:05:50.363438+0000","last_fullsized":"2024-02-21T23:05:50.363438+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-21T23:05:50.379708+0000","last_change":"2024-02-21T23:05:50.376063+0000","last_active":"2024-02-21T23:05:50.379708+0000","last_peered":"2024-02-21T23:05:50.379708+0000","last_clean":"2024-02-21T23:05:50.379708+0000","last_became_active":"2024-02-21T23:05:50.375772+0000","last_became_peered":"2024-02-21T23:05:50.375772+0000","last_unstale":"2024-02-21T23:05:50.379708+0000","last_undegraded":"2024-02-21T23:05:50.379708+0000","last_fullsized":"2024-02-21T23:05:50.379708+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:10:10.973567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.344761+0000","last_change":"2024-02-21T23:05:50.344761+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.344761+0000","last_undegraded":"2024-02-21T23:05:50.344761+0000","last_fullsized":"2024-02-21T23:05:50.344761+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-21T23:05:50.386991+0000","last_change":"2024-02-21T23:05:50.383778+0000","last_active":"2024-02-21T23:05:50.386991+0000","last_peered":"2024-02-21T23:05:50.386991+0000","last_clean":"2024-02-21T23:05:50.386991+0000","last_became_active":"2024-02-21T23:05:50.383436+0000","last_became_peered":"2024-02-21T23:05:50.383436+0000","last_unstale":"2024-02-21T23:05:50.386991+0000","last_undegraded":"2024-02-21T23:05:50.386991+0000","last_fullsized":"2024-02-21T23:05:50.386991+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:27:00.178516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.361666+0000","last_change":"2024-02-21T23:05:50.361666+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.361666+0000","last_undegraded":"2024-02-21T23:05:50.361666+0000","last_fullsized":"2024-02-21T23:05:50.361666+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-21T23:05:50.380951+0000","last_change":"2024-02-21T23:05:50.377181+0000","last_active":"2024-02-21T23:05:50.380951+0000","last_peered":"2024-02-21T23:05:50.380951+0000","last_clean":"2024-02-21T23:05:50.380951+0000","last_became_active":"2024-02-21T23:05:50.373524+0000","last_became_peered":"2024-02-21T23:05:50.373524+0000","last_unstale":"2024-02-21T23:05:50.380951+0000","last_undegraded":"2024-02-21T23:05:50.380951+0000","last_fullsized":"2024-02-21T23:05:50.380951+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:40:18.613631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.362003+0000","last_change":"2024-02-21T23:05:50.362003+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.362003+0000","last_undegraded":"2024-02-21T23:05:50.362003+0000","last_fullsized":"2024-02-21T23:05:50.362003+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-21T23:05:50.373359+0000","last_change":"2024-02-21T23:05:50.366307+0000","last_active":"2024-02-21T23:05:50.373359+0000","last_peered":"2024-02-21T23:05:50.373359+0000","last_clean":"2024-02-21T23:05:50.373359+0000","last_became_active":"2024-02-21T23:05:50.359426+0000","last_became_peered":"2024-02-21T23:05:50.359426+0000","last_unstale":"2024-02-21T23:05:50.373359+0000","last_undegraded":"2024-02-21T23:05:50.373359+0000","last_fullsized":"2024-02-21T23:05:50.373359+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:57:35.943958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.346778+0000","last_change":"2024-02-21T23:05:50.346778+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.346778+0000","last_undegraded":"2024-02-21T23:05:50.346778+0000","last_fullsized":"2024-02-21T23:05:50.346778+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-21T23:05:50.380935+0000","last_change":"2024-02-21T23:05:50.377924+0000","last_active":"2024-02-21T23:05:50.380935+0000","last_peered":"2024-02-21T23:05:50.380935+0000","last_clean":"2024-02-21T23:05:50.380935+0000","last_became_active":"2024-02-21T23:05:50.377440+0000","last_became_peered":"2024-02-21T23:05:50.377440+0000","last_unstale":"2024-02-21T23:05:50.380935+0000","last_undegraded":"2024-02-21T23:05:50.380935+0000","last_fullsized":"2024-02-21T23:05:50.380935+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:38:56.665723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.360027+0000","last_change":"2024-02-21T23:05:50.360027+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.360027+0000","last_undegraded":"2024-02-21T23:05:50.360027+0000","last_fullsized":"2024-02-21T23:05:50.360027+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-21T23:05:50.368340+0000","last_change":"2024-02-21T23:05:50.366877+0000","last_active":"2024-02-21T23:05:50.368340+0000","last_peered":"2024-02-21T23:05:50.368340+0000","last_clean":"2024-02-21T23:05:50.368340+0000","last_became_active":"2024-02-21T23:05:50.365566+0000","last_became_peered":"2024-02-21T23:05:50.365566+0000","last_unstale":"2024-02-21T23:05:50.368340+0000","last_undegraded":"2024-02-21T23:05:50.368340+0000","last_fullsized":"2024-02-21T23:05:50.368340+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:17:25.316787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.360318+0000","last_change":"2024-02-21T23:05:50.360318+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.360318+0000","last_undegraded":"2024-02-21T23:05:50.360318+0000","last_fullsized":"2024-02-21T23:05:50.360318+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-21T23:05:50.380580+0000","last_change":"2024-02-21T23:05:50.376888+0000","last_active":"2024-02-21T23:05:50.380580+0000","last_peered":"2024-02-21T23:05:50.380580+0000","last_clean":"2024-02-21T23:05:50.380580+0000","last_became_active":"2024-02-21T23:05:50.376610+0000","last_became_peered":"2024-02-21T23:05:50.376610+0000","last_unstale":"2024-02-21T23:05:50.380580+0000","last_undegraded":"2024-02-21T23:05:50.380580+0000","last_fullsized":"2024-02-21T23:05:50.380580+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:21:17.428155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.362319+0000","last_change":"2024-02-21T23:05:50.362319+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.362319+0000","last_undegraded":"2024-02-21T23:05:50.362319+0000","last_fullsized":"2024-02-21T23:05:50.362319+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-21T23:05:50.380361+0000","last_change":"2024-02-21T23:05:50.375808+0000","last_active":"2024-02-21T23:05:50.380361+0000","last_peered":"2024-02-21T23:05:50.380361+0000","last_clean":"2024-02-21T23:05:50.380361+0000","last_became_active":"2024-02-21T23:05:50.375467+0000","last_became_peered":"2024-02-21T23:05:50.375467+0000","last_unstale":"2024-02-21T23:05:50.380361+0000","last_undegraded":"2024-02-21T23:05:50.380361+0000","last_fullsized":"2024-02-21T23:05:50.380361+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:36:46.950114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.349887+0000","last_change":"2024-02-21T23:05:50.349887+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.349887+0000","last_undegraded":"2024-02-21T23:05:50.349887+0000","last_fullsized":"2024-02-21T23:05:50.349887+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-21T23:05:50.385611+0000","last_change":"2024-02-21T23:05:50.381948+0000","last_active":"2024-02-21T23:05:50.385611+0000","last_peered":"2024-02-21T23:05:50.385611+0000","last_clean":"2024-02-21T23:05:50.385611+0000","last_became_active":"2024-02-21T23:05:50.381452+0000","last_became_peered":"2024-02-21T23:05:50.381452+0000","last_unstale":"2024-02-21T23:05:50.385611+0000","last_undegraded":"2024-02-21T23:05:50.385611+0000","last_fullsized":"2024-02-21T23:05:50.385611+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:08:35.448428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.358620+0000","last_change":"2024-02-21T23:05:50.358620+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.358620+0000","last_undegraded":"2024-02-21T23:05:50.358620+0000","last_fullsized":"2024-02-21T23:05:50.358620+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-21T23:05:50.368851+0000","last_change":"2024-02-21T23:05:50.362370+0000","last_active":"2024-02-21T23:05:50.368851+0000","last_peered":"2024-02-21T23:05:50.368851+0000","last_clean":"2024-02-21T23:05:50.368851+0000","last_became_active":"2024-02-21T23:05:50.361848+0000","last_became_peered":"2024-02-21T23:05:50.361848+0000","last_unstale":"2024-02-21T23:05:50.368851+0000","last_undegraded":"2024-02-21T23:05:50.368851+0000","last_fullsized":"2024-02-21T23:05:50.368851+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:57:05.198202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.348307+0000","last_change":"2024-02-21T23:05:50.348307+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.348307+0000","last_undegraded":"2024-02-21T23:05:50.348307+0000","last_fullsized":"2024-02-21T23:05:50.348307+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-21T23:05:50.387049+0000","last_change":"2024-02-21T23:05:50.384680+0000","last_active":"2024-02-21T23:05:50.387049+0000","last_peered":"2024-02-21T23:05:50.387049+0000","last_clean":"2024-02-21T23:05:50.387049+0000","last_became_active":"2024-02-21T23:05:50.384208+0000","last_became_peered":"2024-02-21T23:05:50.384208+0000","last_unstale":"2024-02-21T23:05:50.387049+0000","last_undegraded":"2024-02-21T23:05:50.387049+0000","last_fullsized":"2024-02-21T23:05:50.387049+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:57:05.198202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.361295+0000","last_change":"2024-02-21T23:05:50.361295+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.361295+0000","last_undegraded":"2024-02-21T23:05:50.361295+0000","last_fullsized":"2024-02-21T23:05:50.361295+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-21T23:05:50.387015+0000","last_change":"2024-02-21T23:05:50.383920+0000","last_active":"2024-02-21T23:05:50.387015+0000","last_peered":"2024-02-21T23:05:50.387015+0000","last_clean":"2024-02-21T23:05:50.387015+0000","last_became_active":"2024-02-21T23:05:50.382488+0000","last_became_peered":"2024-02-21T23:05:50.382488+0000","last_unstale":"2024-02-21T23:05:50.387015+0000","last_undegraded":"2024-02-21T23:05:50.387015+0000","last_fullsized":"2024-02-21T23:05:50.387015+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:22:31.832352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.364404+0000","last_change":"2024-02-21T23:05:50.364404+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.364404+0000","last_undegraded":"2024-02-21T23:05:50.364404+0000","last_fullsized":"2024-02-21T23:05:50.364404+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-21T23:05:50.381691+0000","last_change":"2024-02-21T23:05:50.376027+0000","last_active":"2024-02-21T23:05:50.381691+0000","last_peered":"2024-02-21T23:05:50.381691+0000","last_clean":"2024-02-21T23:05:50.381691+0000","last_became_active":"2024-02-21T23:05:50.375170+0000","last_became_peered":"2024-02-21T23:05:50.375170+0000","last_unstale":"2024-02-21T23:05:50.381691+0000","last_undegraded":"2024-02-21T23:05:50.381691+0000","last_fullsized":"2024-02-21T23:05:50.381691+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:34:04.816143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.357977+0000","last_change":"2024-02-21T23:05:50.357977+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.357977+0000","last_undegraded":"2024-02-21T23:05:50.357977+0000","last_fullsized":"2024-02-21T23:05:50.357977+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-21T23:05:50.357289+0000","last_change":"2024-02-21T23:05:50.354805+0000","last_active":"2024-02-21T23:05:50.357289+0000","last_peered":"2024-02-21T23:05:50.357289+0000","last_clean":"2024-02-21T23:05:50.357289+0000","last_became_active":"2024-02-21T23:05:50.354525+0000","last_became_peered":"2024-02-21T23:05:50.354525+0000","last_unstale":"2024-02-21T23:05:50.357289+0000","last_undegraded":"2024-02-21T23:05:50.357289+0000","last_fullsized":"2024-02-21T23:05:50.357289+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:37:31.817261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.359043+0000","last_change":"2024-02-21T23:05:50.359043+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.359043+0000","last_undegraded":"2024-02-21T23:05:50.359043+0000","last_fullsized":"2024-02-21T23:05:50.359043+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-21T23:05:50.375794+0000","last_change":"2024-02-21T23:05:50.373554+0000","last_active":"2024-02-21T23:05:50.375794+0000","last_peered":"2024-02-21T23:05:50.375794+0000","last_clean":"2024-02-21T23:05:50.375794+0000","last_became_active":"2024-02-21T23:05:50.373022+0000","last_became_peered":"2024-02-21T23:05:50.373022+0000","last_unstale":"2024-02-21T23:05:50.375794+0000","last_undegraded":"2024-02-21T23:05:50.375794+0000","last_fullsized":"2024-02-21T23:05:50.375794+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:15:58.973847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.361311+0000","last_change":"2024-02-21T23:05:50.361311+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.361311+0000","last_undegraded":"2024-02-21T23:05:50.361311+0000","last_fullsized":"2024-02-21T23:05:50.361311+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-21T23:05:50.378410+0000","last_change":"2024-02-21T23:05:50.376356+0000","last_active":"2024-02-21T23:05:50.378410+0000","last_peered":"2024-02-21T23:05:50.378410+0000","last_clean":"2024-02-21T23:05:50.378410+0000","last_became_active":"2024-02-21T23:05:50.370467+0000","last_became_peered":"2024-02-21T23:05:50.370467+0000","last_unstale":"2024-02-21T23:05:50.378410+0000","last_undegraded":"2024-02-21T23:05:50.378410+0000","last_fullsized":"2024-02-21T23:05:50.378410+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:03:49.547047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.367118+0000","last_change":"2024-02-21T23:05:50.367118+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.367118+0000","last_undegraded":"2024-02-21T23:05:50.367118+0000","last_fullsized":"2024-02-21T23:05:50.367118+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-21T23:05:50.380143+0000","last_change":"2024-02-21T23:05:50.375581+0000","last_active":"2024-02-21T23:05:50.380143+0000","last_peered":"2024-02-21T23:05:50.380143+0000","last_clean":"2024-02-21T23:05:50.380143+0000","last_became_active":"2024-02-21T23:05:50.375282+0000","last_became_peered":"2024-02-21T23:05:50.375282+0000","last_unstale":"2024-02-21T23:05:50.380143+0000","last_undegraded":"2024-02-21T23:05:50.380143+0000","last_fullsized":"2024-02-21T23:05:50.380143+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:03:07.776245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":270,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-21T23:05:50.371109+0000","last_change":"2024-02-21T23:02:32.289335+0000","last_active":"2024-02-21T23:05:50.371109+0000","last_peered":"2024-02-21T23:05:50.371109+0000","last_clean":"2024-02-21T23:05:50.371109+0000","last_became_active":"2024-02-21T23:02:32.286913+0000","last_became_peered":"2024-02-21T23:02:32.286913+0000","last_unstale":"2024-02-21T23:05:50.371109+0000","last_undegraded":"2024-02-21T23:05:50.371109+0000","last_fullsized":"2024-02-21T23:05:50.371109+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_clean_scrub_stamp":"2024-02-21T23:02:31.172569+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:36:23.987374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574890,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27956,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343884,"statfs":{"total":96636764160,"available":96608137216,"internally_reserved":0,"allocated":958464,"data_stored":1013360,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574891,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27892,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343948,"statfs":{"total":96636764160,"available":96608202752,"internally_reserved":0,"allocated":892928,"data_stored":422992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574890,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27904,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343936,"statfs":{"total":96636764160,"available":96608190464,"internally_reserved":0,"allocated":905216,"data_stored":429659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574891,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27904,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343936,"statfs":{"total":96636764160,"available":96608190464,"internally_reserved":0,"allocated":905216,"data_stored":429659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574891,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32052,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339788,"statfs":{"total":96636764160,"available":96603942912,"internally_reserved":0,"allocated":892928,"data_stored":422992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574892,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32064,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339776,"statfs":{"total":96636764160,"available":96603930624,"internally_reserved":0,"allocated":905216,"data_stored":429659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574890,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32064,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339776,"statfs":{"total":96636764160,"available":96603930624,"internally_reserved":0,"allocated":905216,"data_stored":429659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574890,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32132,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339708,"statfs":{"total":96636764160,"available":96603860992,"internally_reserved":0,"allocated":970752,"data_stored":1020027,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-21T23:05:55.989 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-21T23:05:56.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.255+0000 7f8aa0a00640 1 Processor -- start 2024-02-21T23:05:56.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.256+0000 7f8aa0a00640 1 -- start start 2024-02-21T23:05:56.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.257+0000 7f8aa0a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8a9c155960 0x7f8a9c1608a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:56.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.257+0000 7f8aa0a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a9c0c74c0 con 0x7f8a9c06dbf0 2024-02-21T23:05:56.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.257+0000 7f8aa0a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a9c0c77a0 con 0x7f8a9c0aff90 2024-02-21T23:05:56.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.257+0000 7f8aa0a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a9c0c7270 con 0x7f8a9c155960 2024-02-21T23:05:56.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.257+0000 7f8a99e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f8a9c06dbf0 0x7f8a9c158ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43826/0 (socket says 172.21.15.28:43826) 2024-02-21T23:05:56.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.257+0000 7f8a99e00640 1 -- 172.21.15.28:0/3665937754 learned_addr learned my addr 172.21.15.28:0/3665937754 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:05:56.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.258+0000 7f8a9b200640 1 --2- 172.21.15.28:0/3665937754 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8a9c155960 0x7f8a9c1608a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:56.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.258+0000 7f8a99400640 1 -- 172.21.15.28:0/3665937754 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2932031356 0 0) 0x7f8a9c0c74c0 con 0x7f8a9c06dbf0 2024-02-21T23:05:56.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.258+0000 7f8a99400640 1 -- 172.21.15.28:0/3665937754 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a80003520 con 0x7f8a9c06dbf0 2024-02-21T23:05:56.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.258+0000 7f8a99400640 1 -- 172.21.15.28:0/3665937754 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 126600178 0 0) 0x7f8a80003520 con 0x7f8a9c06dbf0 2024-02-21T23:05:56.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.258+0000 7f8a99400640 1 -- 172.21.15.28:0/3665937754 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8a9c155960 msgr2=0x7f8a9c1608a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:56.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.258+0000 7f8a99400640 1 --2- 172.21.15.28:0/3665937754 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8a9c155960 0x7f8a9c1608a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:56.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.258+0000 7f8a99400640 1 -- 172.21.15.28:0/3665937754 >> v1:172.21.15.81:6789/0 conn(0x7f8a9c0aff90 legacy=0x7f8a9c0c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:56.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.258+0000 7f8a99400640 1 -- 172.21.15.28:0/3665937754 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a9c0c7a80 con 0x7f8a9c06dbf0 2024-02-21T23:05:56.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.258+0000 7f8a99400640 1 -- 172.21.15.28:0/3665937754 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f8a840019f0 con 0x7f8a9c06dbf0 2024-02-21T23:05:56.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.258+0000 7f8a99400640 1 -- 172.21.15.28:0/3665937754 >> v1:172.21.15.28:6789/0 conn(0x7f8a9c06dbf0 legacy=0x7f8a9c158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:56.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.259+0000 7f8a99400640 1 --2- 172.21.15.28:0/3665937754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a800039e0 0x7f8a80003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:56.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.259+0000 7f8a99400640 1 --2- 172.21.15.28:0/3665937754 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8a80004800 0x7f8a80004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:56.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.259+0000 7f8a99400640 1 --2- 172.21.15.28:0/3665937754 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8a9c155960 0x7f8a800051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:56.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.259+0000 7f8a99400640 1 -- 172.21.15.28:0/3665937754 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a9c0c7270 con 0x7f8a800039e0 2024-02-21T23:05:56.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.259+0000 7f8a99400640 1 -- 172.21.15.28:0/3665937754 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a8000baa0 con 0x7f8a80004800 2024-02-21T23:05:56.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.259+0000 7f8a99400640 1 -- 172.21.15.28:0/3665937754 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a8000bca0 con 0x7f8a9c155960 2024-02-21T23:05:56.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.259+0000 7f8a9a800640 1 --2- 172.21.15.28:0/3665937754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a800039e0 0x7f8a80003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:56.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.259+0000 7f8a99e00640 1 --2- 172.21.15.28:0/3665937754 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8a80004800 0x7f8a80004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:56.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.259+0000 7f8a9b200640 1 --2- 172.21.15.28:0/3665937754 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8a9c155960 0x7f8a800051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:56.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.259+0000 7f8a9a800640 1 -- 172.21.15.28:0/3665937754 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8a9c155960 msgr2=0x7f8a800051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:56.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.259+0000 7f8a9a800640 1 --2- 172.21.15.28:0/3665937754 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8a9c155960 0x7f8a800051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:56.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.259+0000 7f8a9a800640 1 -- 172.21.15.28:0/3665937754 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8a80004800 msgr2=0x7f8a80004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:56.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.259+0000 7f8a9a800640 1 --2- 172.21.15.28:0/3665937754 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8a80004800 0x7f8a80004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:56.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.259+0000 7f8a9a800640 1 -- 172.21.15.28:0/3665937754 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a80003520 con 0x7f8a800039e0 2024-02-21T23:05:56.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.261+0000 7f8a9a800640 1 --2- 172.21.15.28:0/3665937754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a800039e0 0x7f8a80003e00 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c001720 tx=0x7f8a80005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f6c7c35f8ece1ad server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:56.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.261+0000 7f8a99400640 1 -- 172.21.15.28:0/3665937754 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a8c001370 con 0x7f8a800039e0 2024-02-21T23:05:56.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.261+0000 7f8a99400640 1 -- 172.21.15.28:0/3665937754 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a8c0052d0 con 0x7f8a800039e0 2024-02-21T23:05:56.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.261+0000 7f8a99400640 1 -- 172.21.15.28:0/3665937754 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a8c003830 con 0x7f8a800039e0 2024-02-21T23:05:56.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.262+0000 7f8aa0a00640 1 -- 172.21.15.28:0/3665937754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a800039e0 msgr2=0x7f8a80003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:56.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.262+0000 7f8aa0a00640 1 --2- 172.21.15.28:0/3665937754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a800039e0 0x7f8a80003e00 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c001720 tx=0x7f8a80005830 comp rx=0 tx=0).stop 2024-02-21T23:05:56.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.262+0000 7f8a9b200640 1 -- 172.21.15.28:0/3665937754 reap_dead start 2024-02-21T23:05:56.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.262+0000 7f8aa0a00640 1 -- 172.21.15.28:0/3665937754 shutdown_connections 2024-02-21T23:05:56.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.262+0000 7f8aa0a00640 1 -- 172.21.15.28:0/3665937754 >> 172.21.15.28:0/3665937754 conn(0x7f8a9c0ac370 msgr2=0x7f8a9c0ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:56.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.262+0000 7f8aa0a00640 1 -- 172.21.15.28:0/3665937754 shutdown_connections 2024-02-21T23:05:56.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.262+0000 7f8aa0a00640 1 -- 172.21.15.28:0/3665937754 wait complete. 2024-02-21T23:05:56.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.263+0000 7f8aa0a00640 1 Processor -- start 2024-02-21T23:05:56.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.264+0000 7f8aa0a00640 1 -- start start 2024-02-21T23:05:56.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.265+0000 7f8aa0a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8a800039e0 0x7f8a9c1ee9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:56.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.265+0000 7f8aa0a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a9c1eb460 0x7f8a9c0be350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:56.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.265+0000 7f8aa0a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8a9c0aeb80 0x7f8a9c1518c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:56.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.265+0000 7f8aa0a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a9c0c58f0 con 0x7f8a9c1eb460 2024-02-21T23:05:56.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.265+0000 7f8aa0a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a9c158630 con 0x7f8a9c0aeb80 2024-02-21T23:05:56.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.265+0000 7f8aa0a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a9c0c5a70 con 0x7f8a800039e0 2024-02-21T23:05:56.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.265+0000 7f8a99e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a9c1eb460 0x7f8a9c0be350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:56.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.265+0000 7f8a99e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a9c1eb460 0x7f8a9c0be350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54126/0 (socket says 172.21.15.28:54126) 2024-02-21T23:05:56.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.265+0000 7f8a99e00640 1 -- 172.21.15.28:0/427943267 learned_addr learned my addr 172.21.15.28:0/427943267 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:56.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.265+0000 7f8a9a800640 1 --2- 172.21.15.28:0/427943267 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8a800039e0 0x7f8a9c1ee9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:56.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.265+0000 7f8a9b200640 1 --2- 172.21.15.28:0/427943267 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8a9c0aeb80 0x7f8a9c1518c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:56.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.265+0000 7f8a99e00640 1 -- 172.21.15.28:0/427943267 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8a800039e0 msgr2=0x7f8a9c1ee9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:56.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.265+0000 7f8a99e00640 1 --2- 172.21.15.28:0/427943267 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8a800039e0 0x7f8a9c1ee9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:56.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.265+0000 7f8a99e00640 1 -- 172.21.15.28:0/427943267 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8a9c0aeb80 msgr2=0x7f8a9c1518c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:56.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.265+0000 7f8a99e00640 1 --2- 172.21.15.28:0/427943267 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8a9c0aeb80 0x7f8a9c1518c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:56.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.265+0000 7f8a99e00640 1 -- 172.21.15.28:0/427943267 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a9c0be890 con 0x7f8a9c1eb460 2024-02-21T23:05:56.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.266+0000 7f8a99e00640 1 --2- 172.21.15.28:0/427943267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a9c1eb460 0x7f8a9c0be350 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f8a84001580 tx=0x7f8a8400e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:56.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.266+0000 7f8a92a00640 1 -- 172.21.15.28:0/427943267 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a8400ee50 con 0x7f8a9c1eb460 2024-02-21T23:05:56.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.266+0000 7f8a92a00640 1 -- 172.21.15.28:0/427943267 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a84018070 con 0x7f8a9c1eb460 2024-02-21T23:05:56.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.266+0000 7f8a92a00640 1 -- 172.21.15.28:0/427943267 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a84013640 con 0x7f8a9c1eb460 2024-02-21T23:05:56.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.266+0000 7f8aa0a00640 1 -- 172.21.15.28:0/427943267 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a9c0c3960 con 0x7f8a9c1eb460 2024-02-21T23:05:56.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.266+0000 7f8aa0a00640 1 -- 172.21.15.28:0/427943267 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a9c1eac60 con 0x7f8a9c1eb460 2024-02-21T23:05:56.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.267+0000 7f8aa0a00640 1 -- 172.21.15.28:0/427943267 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a9c0c3e80 con 0x7f8a9c1eb460 2024-02-21T23:05:56.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.268+0000 7f8a92a00640 1 -- 172.21.15.28:0/427943267 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8a840137e0 con 0x7f8a9c1eb460 2024-02-21T23:05:56.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.269+0000 7f8a92a00640 1 --2- 172.21.15.28:0/427943267 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8a7806b6a0 0x7f8a7806db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:56.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.269+0000 7f8a92a00640 1 -- 172.21.15.28:0/427943267 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f8a8408aa00 con 0x7f8a9c1eb460 2024-02-21T23:05:56.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.270+0000 7f8a9a800640 1 --2- 172.21.15.28:0/427943267 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8a7806b6a0 0x7f8a7806db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:56.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.271+0000 7f8a9a800640 1 --2- 172.21.15.28:0/427943267 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8a7806b6a0 0x7f8a7806db60 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c033000 tx=0x7f8a8c009030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:56.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.278+0000 7f8a92a00640 1 -- 172.21.15.28:0/427943267 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f8a9c0c3e80 con 0x7f8a9c1eb460 2024-02-21T23:05:56.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.481+0000 7f8aa0a00640 1 -- 172.21.15.28:0/427943267 --> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8a9c0c3710 con 0x7f8a7806b6a0 2024-02-21T23:05:56.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.499+0000 7f8a92a00640 1 -- 172.21.15.28:0/427943267 <== mgr.4101 v2:172.21.15.81:6832/1848538577 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246827 (secure 0 0 0) 0x7f8a9c0c3710 con 0x7f8a7806b6a0 2024-02-21T23:05:56.491 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:05:56.493 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-21T23:05:56.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.505+0000 7f8aa0a00640 1 -- 172.21.15.28:0/427943267 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8a7806b6a0 msgr2=0x7f8a7806db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:56.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.505+0000 7f8aa0a00640 1 --2- 172.21.15.28:0/427943267 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8a7806b6a0 0x7f8a7806db60 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c033000 tx=0x7f8a8c009030 comp rx=0 tx=0).stop 2024-02-21T23:05:56.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.505+0000 7f8aa0a00640 1 -- 172.21.15.28:0/427943267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a9c1eb460 msgr2=0x7f8a9c0be350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:56.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.505+0000 7f8aa0a00640 1 --2- 172.21.15.28:0/427943267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a9c1eb460 0x7f8a9c0be350 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f8a84001580 tx=0x7f8a8400e530 comp rx=0 tx=0).stop 2024-02-21T23:05:56.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.505+0000 7f8aa0a00640 1 -- 172.21.15.28:0/427943267 shutdown_connections 2024-02-21T23:05:56.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.505+0000 7f8aa0a00640 1 --2- 172.21.15.28:0/427943267 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8a7806b6a0 0x7f8a7806db60 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:56.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.505+0000 7f8aa0a00640 1 --2- 172.21.15.28:0/427943267 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8a9c0aeb80 0x7f8a9c1518c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:56.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.505+0000 7f8aa0a00640 1 --2- 172.21.15.28:0/427943267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a9c1eb460 0x7f8a9c0be350 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:56.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.505+0000 7f8aa0a00640 1 --2- 172.21.15.28:0/427943267 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8a800039e0 0x7f8a9c1ee9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:56.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.505+0000 7f8aa0a00640 1 -- 172.21.15.28:0/427943267 >> 172.21.15.28:0/427943267 conn(0x7f8a9c0ac370 msgr2=0x7f8a9c157e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:56.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.506+0000 7f8aa0a00640 1 -- 172.21.15.28:0/427943267 shutdown_connections 2024-02-21T23:05:56.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.506+0000 7f8aa0a00640 1 -- 172.21.15.28:0/427943267 wait complete. 2024-02-21T23:05:56.514 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":171,"stamp":"2024-02-21T23:05:54.734430+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":33,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":239968,"kb_used_data":7164,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754734752,"statfs":{"total":773094113280,"available":772848386048,"internally_reserved":0,"allocated":7335936,"data_stored":4598007,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.399420"},"pg_stats":[{"pgid":"12.3f","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.370547+0000","last_change":"2024-02-21T23:05:50.370547+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.370547+0000","last_undegraded":"2024-02-21T23:05:50.370547+0000","last_fullsized":"2024-02-21T23:05:50.370547+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.367674+0000","last_change":"2024-02-21T23:05:50.367674+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.367674+0000","last_undegraded":"2024-02-21T23:05:50.367674+0000","last_fullsized":"2024-02-21T23:05:50.367674+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.364429+0000","last_change":"2024-02-21T23:05:50.364429+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.364429+0000","last_undegraded":"2024-02-21T23:05:50.364429+0000","last_fullsized":"2024-02-21T23:05:50.364429+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.368568+0000","last_change":"2024-02-21T23:05:50.368568+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.368568+0000","last_undegraded":"2024-02-21T23:05:50.368568+0000","last_fullsized":"2024-02-21T23:05:50.368568+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.366342+0000","last_change":"2024-02-21T23:05:50.366342+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.366342+0000","last_undegraded":"2024-02-21T23:05:50.366342+0000","last_fullsized":"2024-02-21T23:05:50.366342+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.348142+0000","last_change":"2024-02-21T23:05:50.348142+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.348142+0000","last_undegraded":"2024-02-21T23:05:50.348142+0000","last_fullsized":"2024-02-21T23:05:50.348142+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.364864+0000","last_change":"2024-02-21T23:05:50.364864+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.364864+0000","last_undegraded":"2024-02-21T23:05:50.364864+0000","last_fullsized":"2024-02-21T23:05:50.364864+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.366485+0000","last_change":"2024-02-21T23:05:50.366485+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.366485+0000","last_undegraded":"2024-02-21T23:05:50.366485+0000","last_fullsized":"2024-02-21T23:05:50.366485+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.348832+0000","last_change":"2024-02-21T23:05:50.348832+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.348832+0000","last_undegraded":"2024-02-21T23:05:50.348832+0000","last_fullsized":"2024-02-21T23:05:50.348832+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.363423+0000","last_change":"2024-02-21T23:05:50.363423+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.363423+0000","last_undegraded":"2024-02-21T23:05:50.363423+0000","last_fullsized":"2024-02-21T23:05:50.363423+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.365264+0000","last_change":"2024-02-21T23:05:50.365264+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.365264+0000","last_undegraded":"2024-02-21T23:05:50.365264+0000","last_fullsized":"2024-02-21T23:05:50.365264+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.368080+0000","last_change":"2024-02-21T23:05:50.368080+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.368080+0000","last_undegraded":"2024-02-21T23:05:50.368080+0000","last_fullsized":"2024-02-21T23:05:50.368080+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.364931+0000","last_change":"2024-02-21T23:05:50.364931+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.364931+0000","last_undegraded":"2024-02-21T23:05:50.364931+0000","last_fullsized":"2024-02-21T23:05:50.364931+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.347638+0000","last_change":"2024-02-21T23:05:50.347638+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.347638+0000","last_undegraded":"2024-02-21T23:05:50.347638+0000","last_fullsized":"2024-02-21T23:05:50.347638+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.364788+0000","last_change":"2024-02-21T23:05:50.364788+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.364788+0000","last_undegraded":"2024-02-21T23:05:50.364788+0000","last_fullsized":"2024-02-21T23:05:50.364788+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.365889+0000","last_change":"2024-02-21T23:05:50.365889+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.365889+0000","last_undegraded":"2024-02-21T23:05:50.365889+0000","last_fullsized":"2024-02-21T23:05:50.365889+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.364734+0000","last_change":"2024-02-21T23:05:50.364734+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.364734+0000","last_undegraded":"2024-02-21T23:05:50.364734+0000","last_fullsized":"2024-02-21T23:05:50.364734+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.367567+0000","last_change":"2024-02-21T23:05:50.367567+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.367567+0000","last_undegraded":"2024-02-21T23:05:50.367567+0000","last_fullsized":"2024-02-21T23:05:50.367567+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.346931+0000","last_change":"2024-02-21T23:05:50.346931+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.346931+0000","last_undegraded":"2024-02-21T23:05:50.346931+0000","last_fullsized":"2024-02-21T23:05:50.346931+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.367396+0000","last_change":"2024-02-21T23:05:50.367396+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.367396+0000","last_undegraded":"2024-02-21T23:05:50.367396+0000","last_fullsized":"2024-02-21T23:05:50.367396+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.350181+0000","last_change":"2024-02-21T23:05:50.350181+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.350181+0000","last_undegraded":"2024-02-21T23:05:50.350181+0000","last_fullsized":"2024-02-21T23:05:50.350181+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.362886+0000","last_change":"2024-02-21T23:05:50.362886+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.362886+0000","last_undegraded":"2024-02-21T23:05:50.362886+0000","last_fullsized":"2024-02-21T23:05:50.362886+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.349267+0000","last_change":"2024-02-21T23:05:50.349267+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.349267+0000","last_undegraded":"2024-02-21T23:05:50.349267+0000","last_fullsized":"2024-02-21T23:05:50.349267+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.363272+0000","last_change":"2024-02-21T23:05:50.363272+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.363272+0000","last_undegraded":"2024-02-21T23:05:50.363272+0000","last_fullsized":"2024-02-21T23:05:50.363272+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.361928+0000","last_change":"2024-02-21T23:05:50.361928+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.361928+0000","last_undegraded":"2024-02-21T23:05:50.361928+0000","last_fullsized":"2024-02-21T23:05:50.361928+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.362770+0000","last_change":"2024-02-21T23:05:50.362770+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.362770+0000","last_undegraded":"2024-02-21T23:05:50.362770+0000","last_fullsized":"2024-02-21T23:05:50.362770+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.370781+0000","last_change":"2024-02-21T23:05:50.370781+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.370781+0000","last_undegraded":"2024-02-21T23:05:50.370781+0000","last_fullsized":"2024-02-21T23:05:50.370781+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.363453+0000","last_change":"2024-02-21T23:05:50.363453+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.363453+0000","last_undegraded":"2024-02-21T23:05:50.363453+0000","last_fullsized":"2024-02-21T23:05:50.363453+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.361847+0000","last_change":"2024-02-21T23:05:50.361847+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.361847+0000","last_undegraded":"2024-02-21T23:05:50.361847+0000","last_fullsized":"2024-02-21T23:05:50.361847+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.366516+0000","last_change":"2024-02-21T23:05:50.366516+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.366516+0000","last_undegraded":"2024-02-21T23:05:50.366516+0000","last_fullsized":"2024-02-21T23:05:50.366516+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.364518+0000","last_change":"2024-02-21T23:05:50.364518+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.364518+0000","last_undegraded":"2024-02-21T23:05:50.364518+0000","last_fullsized":"2024-02-21T23:05:50.364518+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.365266+0000","last_change":"2024-02-21T23:05:50.365266+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.365266+0000","last_undegraded":"2024-02-21T23:05:50.365266+0000","last_fullsized":"2024-02-21T23:05:50.365266+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.363578+0000","last_change":"2024-02-21T23:05:50.363578+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.363578+0000","last_undegraded":"2024-02-21T23:05:50.363578+0000","last_fullsized":"2024-02-21T23:05:50.363578+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-21T23:05:50.356881+0000","last_change":"2024-02-21T23:05:50.354109+0000","last_active":"2024-02-21T23:05:50.356881+0000","last_peered":"2024-02-21T23:05:50.356881+0000","last_clean":"2024-02-21T23:05:50.356881+0000","last_became_active":"2024-02-21T23:05:50.353809+0000","last_became_peered":"2024-02-21T23:05:50.353809+0000","last_unstale":"2024-02-21T23:05:50.356881+0000","last_undegraded":"2024-02-21T23:05:50.356881+0000","last_fullsized":"2024-02-21T23:05:50.356881+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:05:10.036678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.363179+0000","last_change":"2024-02-21T23:05:50.363179+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.363179+0000","last_undegraded":"2024-02-21T23:05:50.363179+0000","last_fullsized":"2024-02-21T23:05:50.363179+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-21T23:05:50.386939+0000","last_change":"2024-02-21T23:05:50.383438+0000","last_active":"2024-02-21T23:05:50.386939+0000","last_peered":"2024-02-21T23:05:50.386939+0000","last_clean":"2024-02-21T23:05:50.386939+0000","last_became_active":"2024-02-21T23:05:50.383134+0000","last_became_peered":"2024-02-21T23:05:50.383134+0000","last_unstale":"2024-02-21T23:05:50.386939+0000","last_undegraded":"2024-02-21T23:05:50.386939+0000","last_fullsized":"2024-02-21T23:05:50.386939+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:58:54.817119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.349716+0000","last_change":"2024-02-21T23:05:50.349716+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.349716+0000","last_undegraded":"2024-02-21T23:05:50.349716+0000","last_fullsized":"2024-02-21T23:05:50.349716+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-21T23:05:50.386971+0000","last_change":"2024-02-21T23:05:50.383231+0000","last_active":"2024-02-21T23:05:50.386971+0000","last_peered":"2024-02-21T23:05:50.386971+0000","last_clean":"2024-02-21T23:05:50.386971+0000","last_became_active":"2024-02-21T23:05:50.382757+0000","last_became_peered":"2024-02-21T23:05:50.382757+0000","last_unstale":"2024-02-21T23:05:50.386971+0000","last_undegraded":"2024-02-21T23:05:50.386971+0000","last_fullsized":"2024-02-21T23:05:50.386971+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:34:13.407270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.348408+0000","last_change":"2024-02-21T23:05:50.348408+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.348408+0000","last_undegraded":"2024-02-21T23:05:50.348408+0000","last_fullsized":"2024-02-21T23:05:50.348408+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-21T23:05:50.381772+0000","last_change":"2024-02-21T23:05:50.376050+0000","last_active":"2024-02-21T23:05:50.381772+0000","last_peered":"2024-02-21T23:05:50.381772+0000","last_clean":"2024-02-21T23:05:50.381772+0000","last_became_active":"2024-02-21T23:05:50.375731+0000","last_became_peered":"2024-02-21T23:05:50.375731+0000","last_unstale":"2024-02-21T23:05:50.381772+0000","last_undegraded":"2024-02-21T23:05:50.381772+0000","last_fullsized":"2024-02-21T23:05:50.381772+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:03:49.547047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.361723+0000","last_change":"2024-02-21T23:05:50.361723+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.361723+0000","last_undegraded":"2024-02-21T23:05:50.361723+0000","last_fullsized":"2024-02-21T23:05:50.361723+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-21T23:05:50.381028+0000","last_change":"2024-02-21T23:05:50.377574+0000","last_active":"2024-02-21T23:05:50.381028+0000","last_peered":"2024-02-21T23:05:50.381028+0000","last_clean":"2024-02-21T23:05:50.381028+0000","last_became_active":"2024-02-21T23:05:50.377250+0000","last_became_peered":"2024-02-21T23:05:50.377250+0000","last_unstale":"2024-02-21T23:05:50.381028+0000","last_undegraded":"2024-02-21T23:05:50.381028+0000","last_fullsized":"2024-02-21T23:05:50.381028+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:40:31.369030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.363653+0000","last_change":"2024-02-21T23:05:50.363653+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.363653+0000","last_undegraded":"2024-02-21T23:05:50.363653+0000","last_fullsized":"2024-02-21T23:05:50.363653+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-21T23:05:50.381197+0000","last_change":"2024-02-21T23:05:50.378668+0000","last_active":"2024-02-21T23:05:50.381197+0000","last_peered":"2024-02-21T23:05:50.381197+0000","last_clean":"2024-02-21T23:05:50.381197+0000","last_became_active":"2024-02-21T23:05:50.378369+0000","last_became_peered":"2024-02-21T23:05:50.378369+0000","last_unstale":"2024-02-21T23:05:50.381197+0000","last_undegraded":"2024-02-21T23:05:50.381197+0000","last_fullsized":"2024-02-21T23:05:50.381197+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:10:10.973567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.348744+0000","last_change":"2024-02-21T23:05:50.348744+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.348744+0000","last_undegraded":"2024-02-21T23:05:50.348744+0000","last_fullsized":"2024-02-21T23:05:50.348744+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-21T23:05:50.379722+0000","last_change":"2024-02-21T23:05:50.375534+0000","last_active":"2024-02-21T23:05:50.379722+0000","last_peered":"2024-02-21T23:05:50.379722+0000","last_clean":"2024-02-21T23:05:50.379722+0000","last_became_active":"2024-02-21T23:05:50.373762+0000","last_became_peered":"2024-02-21T23:05:50.373762+0000","last_unstale":"2024-02-21T23:05:50.379722+0000","last_undegraded":"2024-02-21T23:05:50.379722+0000","last_fullsized":"2024-02-21T23:05:50.379722+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:56:09.393950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.364589+0000","last_change":"2024-02-21T23:05:50.364589+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.364589+0000","last_undegraded":"2024-02-21T23:05:50.364589+0000","last_fullsized":"2024-02-21T23:05:50.364589+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-21T23:05:50.363824+0000","last_change":"2024-02-21T23:05:50.355567+0000","last_active":"2024-02-21T23:05:50.363824+0000","last_peered":"2024-02-21T23:05:50.363824+0000","last_clean":"2024-02-21T23:05:50.363824+0000","last_became_active":"2024-02-21T23:05:50.354835+0000","last_became_peered":"2024-02-21T23:05:50.354835+0000","last_unstale":"2024-02-21T23:05:50.363824+0000","last_undegraded":"2024-02-21T23:05:50.363824+0000","last_fullsized":"2024-02-21T23:05:50.363824+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:49:06.159588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.361014+0000","last_change":"2024-02-21T23:05:50.361014+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.361014+0000","last_undegraded":"2024-02-21T23:05:50.361014+0000","last_fullsized":"2024-02-21T23:05:50.361014+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-21T23:05:50.391190+0000","last_change":"2024-02-21T23:05:50.386639+0000","last_active":"2024-02-21T23:05:50.391190+0000","last_peered":"2024-02-21T23:05:50.391190+0000","last_clean":"2024-02-21T23:05:50.391190+0000","last_became_active":"2024-02-21T23:05:50.386158+0000","last_became_peered":"2024-02-21T23:05:50.386158+0000","last_unstale":"2024-02-21T23:05:50.391190+0000","last_undegraded":"2024-02-21T23:05:50.391190+0000","last_fullsized":"2024-02-21T23:05:50.391190+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:56:09.393950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.358895+0000","last_change":"2024-02-21T23:05:50.358895+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.358895+0000","last_undegraded":"2024-02-21T23:05:50.358895+0000","last_fullsized":"2024-02-21T23:05:50.358895+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-21T23:05:50.386510+0000","last_change":"2024-02-21T23:05:50.385146+0000","last_active":"2024-02-21T23:05:50.386510+0000","last_peered":"2024-02-21T23:05:50.386510+0000","last_clean":"2024-02-21T23:05:50.386510+0000","last_became_active":"2024-02-21T23:05:50.384418+0000","last_became_peered":"2024-02-21T23:05:50.384418+0000","last_unstale":"2024-02-21T23:05:50.386510+0000","last_undegraded":"2024-02-21T23:05:50.386510+0000","last_fullsized":"2024-02-21T23:05:50.386510+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:49:36.738197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.346234+0000","last_change":"2024-02-21T23:05:50.346234+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.346234+0000","last_undegraded":"2024-02-21T23:05:50.346234+0000","last_fullsized":"2024-02-21T23:05:50.346234+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-21T23:05:50.376035+0000","last_change":"2024-02-21T23:05:50.373189+0000","last_active":"2024-02-21T23:05:50.376035+0000","last_peered":"2024-02-21T23:05:50.376035+0000","last_clean":"2024-02-21T23:05:50.376035+0000","last_became_active":"2024-02-21T23:05:50.370680+0000","last_became_peered":"2024-02-21T23:05:50.370680+0000","last_unstale":"2024-02-21T23:05:50.376035+0000","last_undegraded":"2024-02-21T23:05:50.376035+0000","last_fullsized":"2024-02-21T23:05:50.376035+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:10:25.735830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.360832+0000","last_change":"2024-02-21T23:05:50.360832+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.360832+0000","last_undegraded":"2024-02-21T23:05:50.360832+0000","last_fullsized":"2024-02-21T23:05:50.360832+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-21T23:05:50.379685+0000","last_change":"2024-02-21T23:05:50.374208+0000","last_active":"2024-02-21T23:05:50.379685+0000","last_peered":"2024-02-21T23:05:50.379685+0000","last_clean":"2024-02-21T23:05:50.379685+0000","last_became_active":"2024-02-21T23:05:50.373869+0000","last_became_peered":"2024-02-21T23:05:50.373869+0000","last_unstale":"2024-02-21T23:05:50.379685+0000","last_undegraded":"2024-02-21T23:05:50.379685+0000","last_fullsized":"2024-02-21T23:05:50.379685+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:16:32.220730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.365281+0000","last_change":"2024-02-21T23:05:50.365281+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.365281+0000","last_undegraded":"2024-02-21T23:05:50.365281+0000","last_fullsized":"2024-02-21T23:05:50.365281+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-21T23:05:50.384116+0000","last_change":"2024-02-21T23:05:50.381643+0000","last_active":"2024-02-21T23:05:50.384116+0000","last_peered":"2024-02-21T23:05:50.384116+0000","last_clean":"2024-02-21T23:05:50.384116+0000","last_became_active":"2024-02-21T23:05:50.381345+0000","last_became_peered":"2024-02-21T23:05:50.381345+0000","last_unstale":"2024-02-21T23:05:50.384116+0000","last_undegraded":"2024-02-21T23:05:50.384116+0000","last_fullsized":"2024-02-21T23:05:50.384116+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:26:58.076002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.359127+0000","last_change":"2024-02-21T23:05:50.359127+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.359127+0000","last_undegraded":"2024-02-21T23:05:50.359127+0000","last_fullsized":"2024-02-21T23:05:50.359127+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-21T23:05:50.386822+0000","last_change":"2024-02-21T23:05:50.381962+0000","last_active":"2024-02-21T23:05:50.386822+0000","last_peered":"2024-02-21T23:05:50.386822+0000","last_clean":"2024-02-21T23:05:50.386822+0000","last_became_active":"2024-02-21T23:05:50.380838+0000","last_became_peered":"2024-02-21T23:05:50.380838+0000","last_unstale":"2024-02-21T23:05:50.386822+0000","last_undegraded":"2024-02-21T23:05:50.386822+0000","last_fullsized":"2024-02-21T23:05:50.386822+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:26:32.353509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.367278+0000","last_change":"2024-02-21T23:05:50.367278+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.367278+0000","last_undegraded":"2024-02-21T23:05:50.367278+0000","last_fullsized":"2024-02-21T23:05:50.367278+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-21T23:05:50.356314+0000","last_change":"2024-02-21T23:05:50.353116+0000","last_active":"2024-02-21T23:05:50.356314+0000","last_peered":"2024-02-21T23:05:50.356314+0000","last_clean":"2024-02-21T23:05:50.356314+0000","last_became_active":"2024-02-21T23:05:50.352768+0000","last_became_peered":"2024-02-21T23:05:50.352768+0000","last_unstale":"2024-02-21T23:05:50.356314+0000","last_undegraded":"2024-02-21T23:05:50.356314+0000","last_fullsized":"2024-02-21T23:05:50.356314+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:34:30.710951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.363438+0000","last_change":"2024-02-21T23:05:50.363438+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.363438+0000","last_undegraded":"2024-02-21T23:05:50.363438+0000","last_fullsized":"2024-02-21T23:05:50.363438+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-21T23:05:50.379708+0000","last_change":"2024-02-21T23:05:50.376063+0000","last_active":"2024-02-21T23:05:50.379708+0000","last_peered":"2024-02-21T23:05:50.379708+0000","last_clean":"2024-02-21T23:05:50.379708+0000","last_became_active":"2024-02-21T23:05:50.375772+0000","last_became_peered":"2024-02-21T23:05:50.375772+0000","last_unstale":"2024-02-21T23:05:50.379708+0000","last_undegraded":"2024-02-21T23:05:50.379708+0000","last_fullsized":"2024-02-21T23:05:50.379708+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:10:10.973567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.344761+0000","last_change":"2024-02-21T23:05:50.344761+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.344761+0000","last_undegraded":"2024-02-21T23:05:50.344761+0000","last_fullsized":"2024-02-21T23:05:50.344761+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-21T23:05:50.386991+0000","last_change":"2024-02-21T23:05:50.383778+0000","last_active":"2024-02-21T23:05:50.386991+0000","last_peered":"2024-02-21T23:05:50.386991+0000","last_clean":"2024-02-21T23:05:50.386991+0000","last_became_active":"2024-02-21T23:05:50.383436+0000","last_became_peered":"2024-02-21T23:05:50.383436+0000","last_unstale":"2024-02-21T23:05:50.386991+0000","last_undegraded":"2024-02-21T23:05:50.386991+0000","last_fullsized":"2024-02-21T23:05:50.386991+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:27:00.178516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.361666+0000","last_change":"2024-02-21T23:05:50.361666+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.361666+0000","last_undegraded":"2024-02-21T23:05:50.361666+0000","last_fullsized":"2024-02-21T23:05:50.361666+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-21T23:05:50.380951+0000","last_change":"2024-02-21T23:05:50.377181+0000","last_active":"2024-02-21T23:05:50.380951+0000","last_peered":"2024-02-21T23:05:50.380951+0000","last_clean":"2024-02-21T23:05:50.380951+0000","last_became_active":"2024-02-21T23:05:50.373524+0000","last_became_peered":"2024-02-21T23:05:50.373524+0000","last_unstale":"2024-02-21T23:05:50.380951+0000","last_undegraded":"2024-02-21T23:05:50.380951+0000","last_fullsized":"2024-02-21T23:05:50.380951+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:40:18.613631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.362003+0000","last_change":"2024-02-21T23:05:50.362003+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.362003+0000","last_undegraded":"2024-02-21T23:05:50.362003+0000","last_fullsized":"2024-02-21T23:05:50.362003+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-21T23:05:50.373359+0000","last_change":"2024-02-21T23:05:50.366307+0000","last_active":"2024-02-21T23:05:50.373359+0000","last_peered":"2024-02-21T23:05:50.373359+0000","last_clean":"2024-02-21T23:05:50.373359+0000","last_became_active":"2024-02-21T23:05:50.359426+0000","last_became_peered":"2024-02-21T23:05:50.359426+0000","last_unstale":"2024-02-21T23:05:50.373359+0000","last_undegraded":"2024-02-21T23:05:50.373359+0000","last_fullsized":"2024-02-21T23:05:50.373359+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:57:35.943958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.346778+0000","last_change":"2024-02-21T23:05:50.346778+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.346778+0000","last_undegraded":"2024-02-21T23:05:50.346778+0000","last_fullsized":"2024-02-21T23:05:50.346778+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-21T23:05:50.380935+0000","last_change":"2024-02-21T23:05:50.377924+0000","last_active":"2024-02-21T23:05:50.380935+0000","last_peered":"2024-02-21T23:05:50.380935+0000","last_clean":"2024-02-21T23:05:50.380935+0000","last_became_active":"2024-02-21T23:05:50.377440+0000","last_became_peered":"2024-02-21T23:05:50.377440+0000","last_unstale":"2024-02-21T23:05:50.380935+0000","last_undegraded":"2024-02-21T23:05:50.380935+0000","last_fullsized":"2024-02-21T23:05:50.380935+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:38:56.665723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.360027+0000","last_change":"2024-02-21T23:05:50.360027+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.360027+0000","last_undegraded":"2024-02-21T23:05:50.360027+0000","last_fullsized":"2024-02-21T23:05:50.360027+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-21T23:05:50.368340+0000","last_change":"2024-02-21T23:05:50.366877+0000","last_active":"2024-02-21T23:05:50.368340+0000","last_peered":"2024-02-21T23:05:50.368340+0000","last_clean":"2024-02-21T23:05:50.368340+0000","last_became_active":"2024-02-21T23:05:50.365566+0000","last_became_peered":"2024-02-21T23:05:50.365566+0000","last_unstale":"2024-02-21T23:05:50.368340+0000","last_undegraded":"2024-02-21T23:05:50.368340+0000","last_fullsized":"2024-02-21T23:05:50.368340+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:17:25.316787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.360318+0000","last_change":"2024-02-21T23:05:50.360318+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.360318+0000","last_undegraded":"2024-02-21T23:05:50.360318+0000","last_fullsized":"2024-02-21T23:05:50.360318+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-21T23:05:50.380580+0000","last_change":"2024-02-21T23:05:50.376888+0000","last_active":"2024-02-21T23:05:50.380580+0000","last_peered":"2024-02-21T23:05:50.380580+0000","last_clean":"2024-02-21T23:05:50.380580+0000","last_became_active":"2024-02-21T23:05:50.376610+0000","last_became_peered":"2024-02-21T23:05:50.376610+0000","last_unstale":"2024-02-21T23:05:50.380580+0000","last_undegraded":"2024-02-21T23:05:50.380580+0000","last_fullsized":"2024-02-21T23:05:50.380580+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:21:17.428155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.362319+0000","last_change":"2024-02-21T23:05:50.362319+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.362319+0000","last_undegraded":"2024-02-21T23:05:50.362319+0000","last_fullsized":"2024-02-21T23:05:50.362319+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-21T23:05:50.380361+0000","last_change":"2024-02-21T23:05:50.375808+0000","last_active":"2024-02-21T23:05:50.380361+0000","last_peered":"2024-02-21T23:05:50.380361+0000","last_clean":"2024-02-21T23:05:50.380361+0000","last_became_active":"2024-02-21T23:05:50.375467+0000","last_became_peered":"2024-02-21T23:05:50.375467+0000","last_unstale":"2024-02-21T23:05:50.380361+0000","last_undegraded":"2024-02-21T23:05:50.380361+0000","last_fullsized":"2024-02-21T23:05:50.380361+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:36:46.950114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.349887+0000","last_change":"2024-02-21T23:05:50.349887+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.349887+0000","last_undegraded":"2024-02-21T23:05:50.349887+0000","last_fullsized":"2024-02-21T23:05:50.349887+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-21T23:05:50.385611+0000","last_change":"2024-02-21T23:05:50.381948+0000","last_active":"2024-02-21T23:05:50.385611+0000","last_peered":"2024-02-21T23:05:50.385611+0000","last_clean":"2024-02-21T23:05:50.385611+0000","last_became_active":"2024-02-21T23:05:50.381452+0000","last_became_peered":"2024-02-21T23:05:50.381452+0000","last_unstale":"2024-02-21T23:05:50.385611+0000","last_undegraded":"2024-02-21T23:05:50.385611+0000","last_fullsized":"2024-02-21T23:05:50.385611+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:08:35.448428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.358620+0000","last_change":"2024-02-21T23:05:50.358620+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.358620+0000","last_undegraded":"2024-02-21T23:05:50.358620+0000","last_fullsized":"2024-02-21T23:05:50.358620+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-21T23:05:50.368851+0000","last_change":"2024-02-21T23:05:50.362370+0000","last_active":"2024-02-21T23:05:50.368851+0000","last_peered":"2024-02-21T23:05:50.368851+0000","last_clean":"2024-02-21T23:05:50.368851+0000","last_became_active":"2024-02-21T23:05:50.361848+0000","last_became_peered":"2024-02-21T23:05:50.361848+0000","last_unstale":"2024-02-21T23:05:50.368851+0000","last_undegraded":"2024-02-21T23:05:50.368851+0000","last_fullsized":"2024-02-21T23:05:50.368851+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:57:05.198202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.348307+0000","last_change":"2024-02-21T23:05:50.348307+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.348307+0000","last_undegraded":"2024-02-21T23:05:50.348307+0000","last_fullsized":"2024-02-21T23:05:50.348307+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-21T23:05:50.387049+0000","last_change":"2024-02-21T23:05:50.384680+0000","last_active":"2024-02-21T23:05:50.387049+0000","last_peered":"2024-02-21T23:05:50.387049+0000","last_clean":"2024-02-21T23:05:50.387049+0000","last_became_active":"2024-02-21T23:05:50.384208+0000","last_became_peered":"2024-02-21T23:05:50.384208+0000","last_unstale":"2024-02-21T23:05:50.387049+0000","last_undegraded":"2024-02-21T23:05:50.387049+0000","last_fullsized":"2024-02-21T23:05:50.387049+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:57:05.198202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.361295+0000","last_change":"2024-02-21T23:05:50.361295+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.361295+0000","last_undegraded":"2024-02-21T23:05:50.361295+0000","last_fullsized":"2024-02-21T23:05:50.361295+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-21T23:05:50.387015+0000","last_change":"2024-02-21T23:05:50.383920+0000","last_active":"2024-02-21T23:05:50.387015+0000","last_peered":"2024-02-21T23:05:50.387015+0000","last_clean":"2024-02-21T23:05:50.387015+0000","last_became_active":"2024-02-21T23:05:50.382488+0000","last_became_peered":"2024-02-21T23:05:50.382488+0000","last_unstale":"2024-02-21T23:05:50.387015+0000","last_undegraded":"2024-02-21T23:05:50.387015+0000","last_fullsized":"2024-02-21T23:05:50.387015+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:22:31.832352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.364404+0000","last_change":"2024-02-21T23:05:50.364404+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.364404+0000","last_undegraded":"2024-02-21T23:05:50.364404+0000","last_fullsized":"2024-02-21T23:05:50.364404+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-21T23:05:50.381691+0000","last_change":"2024-02-21T23:05:50.376027+0000","last_active":"2024-02-21T23:05:50.381691+0000","last_peered":"2024-02-21T23:05:50.381691+0000","last_clean":"2024-02-21T23:05:50.381691+0000","last_became_active":"2024-02-21T23:05:50.375170+0000","last_became_peered":"2024-02-21T23:05:50.375170+0000","last_unstale":"2024-02-21T23:05:50.381691+0000","last_undegraded":"2024-02-21T23:05:50.381691+0000","last_fullsized":"2024-02-21T23:05:50.381691+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:34:04.816143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.357977+0000","last_change":"2024-02-21T23:05:50.357977+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.357977+0000","last_undegraded":"2024-02-21T23:05:50.357977+0000","last_fullsized":"2024-02-21T23:05:50.357977+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-21T23:05:50.357289+0000","last_change":"2024-02-21T23:05:50.354805+0000","last_active":"2024-02-21T23:05:50.357289+0000","last_peered":"2024-02-21T23:05:50.357289+0000","last_clean":"2024-02-21T23:05:50.357289+0000","last_became_active":"2024-02-21T23:05:50.354525+0000","last_became_peered":"2024-02-21T23:05:50.354525+0000","last_unstale":"2024-02-21T23:05:50.357289+0000","last_undegraded":"2024-02-21T23:05:50.357289+0000","last_fullsized":"2024-02-21T23:05:50.357289+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:37:31.817261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.359043+0000","last_change":"2024-02-21T23:05:50.359043+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.359043+0000","last_undegraded":"2024-02-21T23:05:50.359043+0000","last_fullsized":"2024-02-21T23:05:50.359043+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-21T23:05:50.375794+0000","last_change":"2024-02-21T23:05:50.373554+0000","last_active":"2024-02-21T23:05:50.375794+0000","last_peered":"2024-02-21T23:05:50.375794+0000","last_clean":"2024-02-21T23:05:50.375794+0000","last_became_active":"2024-02-21T23:05:50.373022+0000","last_became_peered":"2024-02-21T23:05:50.373022+0000","last_unstale":"2024-02-21T23:05:50.375794+0000","last_undegraded":"2024-02-21T23:05:50.375794+0000","last_fullsized":"2024-02-21T23:05:50.375794+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:15:58.973847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.361311+0000","last_change":"2024-02-21T23:05:50.361311+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.361311+0000","last_undegraded":"2024-02-21T23:05:50.361311+0000","last_fullsized":"2024-02-21T23:05:50.361311+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-21T23:05:50.378410+0000","last_change":"2024-02-21T23:05:50.376356+0000","last_active":"2024-02-21T23:05:50.378410+0000","last_peered":"2024-02-21T23:05:50.378410+0000","last_clean":"2024-02-21T23:05:50.378410+0000","last_became_active":"2024-02-21T23:05:50.370467+0000","last_became_peered":"2024-02-21T23:05:50.370467+0000","last_unstale":"2024-02-21T23:05:50.378410+0000","last_undegraded":"2024-02-21T23:05:50.378410+0000","last_fullsized":"2024-02-21T23:05:50.378410+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:03:49.547047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":1,"reported_epoch":74,"state":"creating+peering","last_fresh":"2024-02-21T23:05:50.367118+0000","last_change":"2024-02-21T23:05:50.367118+0000","last_active":"2024-02-21T23:05:50.333365+0000","last_peered":"2024-02-21T23:05:50.333365+0000","last_clean":"2024-02-21T23:05:50.333365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:05:50.367118+0000","last_undegraded":"2024-02-21T23:05:50.367118+0000","last_fullsized":"2024-02-21T23:05:50.367118+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":9,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-21T23:05:50.380143+0000","last_change":"2024-02-21T23:05:50.375581+0000","last_active":"2024-02-21T23:05:50.380143+0000","last_peered":"2024-02-21T23:05:50.380143+0000","last_clean":"2024-02-21T23:05:50.380143+0000","last_became_active":"2024-02-21T23:05:50.375282+0000","last_became_peered":"2024-02-21T23:05:50.375282+0000","last_unstale":"2024-02-21T23:05:50.380143+0000","last_undegraded":"2024-02-21T23:05:50.380143+0000","last_fullsized":"2024-02-21T23:05:50.380143+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:03:07.776245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":270,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-02-21T23:05:50.371109+0000","last_change":"2024-02-21T23:02:32.289335+0000","last_active":"2024-02-21T23:05:50.371109+0000","last_peered":"2024-02-21T23:05:50.371109+0000","last_clean":"2024-02-21T23:05:50.371109+0000","last_became_active":"2024-02-21T23:02:32.286913+0000","last_became_peered":"2024-02-21T23:02:32.286913+0000","last_unstale":"2024-02-21T23:05:50.371109+0000","last_undegraded":"2024-02-21T23:05:50.371109+0000","last_fullsized":"2024-02-21T23:05:50.371109+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_clean_scrub_stamp":"2024-02-21T23:02:31.172569+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:36:23.987374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574890,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27956,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343884,"statfs":{"total":96636764160,"available":96608137216,"internally_reserved":0,"allocated":958464,"data_stored":1013360,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574891,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27892,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343948,"statfs":{"total":96636764160,"available":96608202752,"internally_reserved":0,"allocated":892928,"data_stored":422992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574890,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27904,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343936,"statfs":{"total":96636764160,"available":96608190464,"internally_reserved":0,"allocated":905216,"data_stored":429659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574891,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27904,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343936,"statfs":{"total":96636764160,"available":96608190464,"internally_reserved":0,"allocated":905216,"data_stored":429659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574891,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32052,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339788,"statfs":{"total":96636764160,"available":96603942912,"internally_reserved":0,"allocated":892928,"data_stored":422992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574892,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32064,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339776,"statfs":{"total":96636764160,"available":96603930624,"internally_reserved":0,"allocated":905216,"data_stored":429659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574890,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32064,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339776,"statfs":{"total":96636764160,"available":96603930624,"internally_reserved":0,"allocated":905216,"data_stored":429659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574890,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32132,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339708,"statfs":{"total":96636764160,"available":96603860992,"internally_reserved":0,"allocated":970752,"data_stored":1020027,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-21T23:05:56.519 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-21T23:05:56.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.789+0000 7f0ce0a00640 1 Processor -- start 2024-02-21T23:05:56.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.789+0000 7f0ce0a00640 1 -- start start 2024-02-21T23:05:56.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.791+0000 7f0ce0a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0cdc0aff90 0x7f0cdc0c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:56.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.791+0000 7f0ce0a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0cdc06dbf0 0x7f0cdc158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:56.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.791+0000 7f0ce0a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cdc0c74c0 con 0x7f0cdc0c5720 2024-02-21T23:05:56.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.791+0000 7f0ce0a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cdc0c7270 con 0x7f0cdc06dbf0 2024-02-21T23:05:56.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.791+0000 7f0ce0a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cdc158630 con 0x7f0cdc0aff90 2024-02-21T23:05:56.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.791+0000 7f0cd9e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0cdc06dbf0 0x7f0cdc158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:56.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.792+0000 7f0cd9e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0cdc06dbf0 0x7f0cdc158ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54134/0 (socket says 172.21.15.28:54134) 2024-02-21T23:05:56.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.792+0000 7f0cd9e00640 1 -- 172.21.15.28:0/3199095556 learned_addr learned my addr 172.21.15.28:0/3199095556 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:56.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.792+0000 7f0cda800640 1 --2- 172.21.15.28:0/3199095556 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0cdc0aff90 0x7f0cdc0c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:56.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.792+0000 7f0cd9e00640 1 -- 172.21.15.28:0/3199095556 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0cdc0aff90 msgr2=0x7f0cdc0c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:56.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.792+0000 7f0cd9e00640 1 --2- 172.21.15.28:0/3199095556 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0cdc0aff90 0x7f0cdc0c51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:56.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.792+0000 7f0cd9e00640 1 -- 172.21.15.28:0/3199095556 >> v1:172.21.15.81:6789/0 conn(0x7f0cdc0c5720 legacy=0x7f0cdc06c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:56.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.792+0000 7f0cd9e00640 1 -- 172.21.15.28:0/3199095556 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cdc0c77a0 con 0x7f0cdc06dbf0 2024-02-21T23:05:56.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.794+0000 7f0cd9e00640 1 --2- 172.21.15.28:0/3199095556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cdc06dbf0 0x7f0cdc158ac0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f0cc4009870 tx=0x7f0cc402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8403ea8c2a2a052 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:56.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.794+0000 7f0cd9400640 1 -- 172.21.15.28:0/3199095556 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cc4034070 con 0x7f0cdc06dbf0 2024-02-21T23:05:56.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.794+0000 7f0cd9400640 1 -- 172.21.15.28:0/3199095556 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cc4030840 con 0x7f0cdc06dbf0 2024-02-21T23:05:56.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.794+0000 7f0cd9400640 1 -- 172.21.15.28:0/3199095556 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cc4030c10 con 0x7f0cdc06dbf0 2024-02-21T23:05:56.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.795+0000 7f0ce0a00640 1 -- 172.21.15.28:0/3199095556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cdc06dbf0 msgr2=0x7f0cdc158ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:56.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.795+0000 7f0ce0a00640 1 --2- 172.21.15.28:0/3199095556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cdc06dbf0 0x7f0cdc158ac0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f0cc4009870 tx=0x7f0cc402b980 comp rx=0 tx=0).stop 2024-02-21T23:05:56.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.795+0000 7f0ce0a00640 1 -- 172.21.15.28:0/3199095556 shutdown_connections 2024-02-21T23:05:56.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.795+0000 7f0ce0a00640 1 --2- 172.21.15.28:0/3199095556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cdc06dbf0 0x7f0cdc158ac0 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:56.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.795+0000 7f0ce0a00640 1 --2- 172.21.15.28:0/3199095556 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0cdc0aff90 0x7f0cdc0c51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:56.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.795+0000 7f0ce0a00640 1 -- 172.21.15.28:0/3199095556 >> 172.21.15.28:0/3199095556 conn(0x7f0cdc0ac370 msgr2=0x7f0cdc0ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:56.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.795+0000 7f0ce0a00640 1 -- 172.21.15.28:0/3199095556 shutdown_connections 2024-02-21T23:05:56.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.795+0000 7f0ce0a00640 1 -- 172.21.15.28:0/3199095556 wait complete. 2024-02-21T23:05:56.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.798+0000 7f0ce0a00640 1 Processor -- start 2024-02-21T23:05:56.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.798+0000 7f0ce0a00640 1 -- start start 2024-02-21T23:05:56.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.799+0000 7f0ce0a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0cdc06dbf0 0x7f0cdc14e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:56.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.799+0000 7f0ce0a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cdc0aff90 0x7f0cdc0ae950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:56.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.799+0000 7f0ce0a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0cdc0c5720 0x7f0cdc14ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:56.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.799+0000 7f0ce0a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cdc0c8760 con 0x7f0cdc0aff90 2024-02-21T23:05:56.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.799+0000 7f0ce0a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cdc0be0c0 con 0x7f0cdc06dbf0 2024-02-21T23:05:56.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.799+0000 7f0ce0a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cdc0c88e0 con 0x7f0cdc0c5720 2024-02-21T23:05:56.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.800+0000 7f0cd9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cdc0aff90 0x7f0cdc0ae950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:56.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.800+0000 7f0cd9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cdc0aff90 0x7f0cdc0ae950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54138/0 (socket says 172.21.15.28:54138) 2024-02-21T23:05:56.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.800+0000 7f0cd9e00640 1 -- 172.21.15.28:0/2222305649 learned_addr learned my addr 172.21.15.28:0/2222305649 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:56.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.800+0000 7f0cd9e00640 1 -- 172.21.15.28:0/2222305649 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0cdc0c5720 msgr2=0x7f0cdc14ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:56.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.800+0000 7f0cda800640 1 --2- 172.21.15.28:0/2222305649 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0cdc06dbf0 0x7f0cdc14e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:56.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.800+0000 7f0cd9e00640 1 --2- 172.21.15.28:0/2222305649 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0cdc0c5720 0x7f0cdc14ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:56.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.800+0000 7f0cd9e00640 1 -- 172.21.15.28:0/2222305649 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0cdc06dbf0 msgr2=0x7f0cdc14e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:56.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.800+0000 7f0cd9e00640 1 --2- 172.21.15.28:0/2222305649 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0cdc06dbf0 0x7f0cdc14e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:56.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.800+0000 7f0cd9e00640 1 -- 172.21.15.28:0/2222305649 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cdc0c1cb0 con 0x7f0cdc0aff90 2024-02-21T23:05:56.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.801+0000 7f0cd9e00640 1 --2- 172.21.15.28:0/2222305649 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cdc0aff90 0x7f0cdc0ae950 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f0cc4000c00 tx=0x7f0cc4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:56.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.801+0000 7f0cd2a00640 1 -- 172.21.15.28:0/2222305649 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cc4048070 con 0x7f0cdc0aff90 2024-02-21T23:05:56.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.801+0000 7f0cd2a00640 1 -- 172.21.15.28:0/2222305649 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cc4030640 con 0x7f0cdc0aff90 2024-02-21T23:05:56.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.801+0000 7f0cd2a00640 1 -- 172.21.15.28:0/2222305649 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cc4034040 con 0x7f0cdc0aff90 2024-02-21T23:05:56.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.801+0000 7f0ce0a00640 1 -- 172.21.15.28:0/2222305649 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cdc154c10 con 0x7f0cdc0aff90 2024-02-21T23:05:56.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.801+0000 7f0ce0a00640 1 -- 172.21.15.28:0/2222305649 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cdc1eb440 con 0x7f0cdc0aff90 2024-02-21T23:05:56.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.802+0000 7f0ce0a00640 1 -- 172.21.15.28:0/2222305649 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cdc154de0 con 0x7f0cdc0aff90 2024-02-21T23:05:56.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.803+0000 7f0cd2a00640 1 -- 172.21.15.28:0/2222305649 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0cc4037070 con 0x7f0cdc0aff90 2024-02-21T23:05:56.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.805+0000 7f0cd2a00640 1 --2- 172.21.15.28:0/2222305649 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0cb806b500 0x7f0cb806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:56.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.805+0000 7f0cd2a00640 1 -- 172.21.15.28:0/2222305649 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f0cc40ac000 con 0x7f0cdc0aff90 2024-02-21T23:05:56.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.805+0000 7f0cda800640 1 --2- 172.21.15.28:0/2222305649 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0cb806b500 0x7f0cb806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:56.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.806+0000 7f0cda800640 1 --2- 172.21.15.28:0/2222305649 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0cb806b500 0x7f0cb806d9c0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc14fc30 tx=0x7f0ccc00b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:56.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:56.809+0000 7f0cd2a00640 1 -- 172.21.15.28:0/2222305649 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f0cc407ae40 con 0x7f0cdc0aff90 2024-02-21T23:05:56.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.009+0000 7f0ce0a00640 1 -- 172.21.15.28:0/2222305649 --> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0cdc04f290 con 0x7f0cb806b500 2024-02-21T23:05:57.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.028+0000 7f0cd2a00640 1 -- 172.21.15.28:0/2222305649 <== mgr.4101 v2:172.21.15.81:6832/1848538577 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252150 (secure 0 0 0) 0x7f0cdc04f290 con 0x7f0cb806b500 2024-02-21T23:05:57.018 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:05:57.022 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-21T23:05:57.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.035+0000 7f0ce0a00640 1 -- 172.21.15.28:0/2222305649 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0cb806b500 msgr2=0x7f0cb806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:57.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.035+0000 7f0ce0a00640 1 --2- 172.21.15.28:0/2222305649 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0cb806b500 0x7f0cb806d9c0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc14fc30 tx=0x7f0ccc00b040 comp rx=0 tx=0).stop 2024-02-21T23:05:57.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.035+0000 7f0ce0a00640 1 -- 172.21.15.28:0/2222305649 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cdc0aff90 msgr2=0x7f0cdc0ae950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:57.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.035+0000 7f0ce0a00640 1 --2- 172.21.15.28:0/2222305649 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cdc0aff90 0x7f0cdc0ae950 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f0cc4000c00 tx=0x7f0cc4004730 comp rx=0 tx=0).stop 2024-02-21T23:05:57.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.035+0000 7f0ce0a00640 1 -- 172.21.15.28:0/2222305649 shutdown_connections 2024-02-21T23:05:57.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.035+0000 7f0ce0a00640 1 --2- 172.21.15.28:0/2222305649 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0cb806b500 0x7f0cb806d9c0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:57.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.035+0000 7f0ce0a00640 1 --2- 172.21.15.28:0/2222305649 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0cdc0c5720 0x7f0cdc14ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:57.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.035+0000 7f0ce0a00640 1 --2- 172.21.15.28:0/2222305649 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cdc0aff90 0x7f0cdc0ae950 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:57.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.035+0000 7f0ce0a00640 1 --2- 172.21.15.28:0/2222305649 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0cdc06dbf0 0x7f0cdc14e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:57.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.035+0000 7f0ce0a00640 1 -- 172.21.15.28:0/2222305649 >> 172.21.15.28:0/2222305649 conn(0x7f0cdc0ac370 msgr2=0x7f0cdc0ac750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:57.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.036+0000 7f0ce0a00640 1 -- 172.21.15.28:0/2222305649 shutdown_connections 2024-02-21T23:05:57.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.036+0000 7f0ce0a00640 1 -- 172.21.15.28:0/2222305649 wait complete. 2024-02-21T23:05:57.052 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":172,"stamp":"2024-02-21T23:05:56.735092+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":252772,"kb_used_data":7456,"kb_used_omap":12,"kb_used_meta":245171,"kb_avail":754721948,"statfs":{"total":773094113280,"available":772835274752,"internally_reserved":0,"allocated":7634944,"data_stored":4733700,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":251055665},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":43,"apply_latency_ms":43,"commit_latency_ns":43000000,"apply_latency_ns":43000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.393211"},"pg_stats":[{"pgid":"12.3f","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.535895+0000","last_change":"2024-02-21T23:05:51.408831+0000","last_active":"2024-02-21T23:05:52.535895+0000","last_peered":"2024-02-21T23:05:52.535895+0000","last_clean":"2024-02-21T23:05:52.535895+0000","last_became_active":"2024-02-21T23:05:51.408353+0000","last_became_peered":"2024-02-21T23:05:51.408353+0000","last_unstale":"2024-02-21T23:05:52.535895+0000","last_undegraded":"2024-02-21T23:05:52.535895+0000","last_fullsized":"2024-02-21T23:05:52.535895+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:36:47.962935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.418527+0000","last_change":"2024-02-21T23:05:51.420557+0000","last_active":"2024-02-21T23:05:52.418527+0000","last_peered":"2024-02-21T23:05:52.418527+0000","last_clean":"2024-02-21T23:05:52.418527+0000","last_became_active":"2024-02-21T23:05:51.420078+0000","last_became_peered":"2024-02-21T23:05:51.420078+0000","last_unstale":"2024-02-21T23:05:52.418527+0000","last_undegraded":"2024-02-21T23:05:52.418527+0000","last_fullsized":"2024-02-21T23:05:52.418527+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:36:23.792395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.827683+0000","last_change":"2024-02-21T23:05:51.408934+0000","last_active":"2024-02-21T23:05:52.827683+0000","last_peered":"2024-02-21T23:05:52.827683+0000","last_clean":"2024-02-21T23:05:52.827683+0000","last_became_active":"2024-02-21T23:05:51.408597+0000","last_became_peered":"2024-02-21T23:05:51.408597+0000","last_unstale":"2024-02-21T23:05:52.827683+0000","last_undegraded":"2024-02-21T23:05:52.827683+0000","last_fullsized":"2024-02-21T23:05:52.827683+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:58:55.829941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.827692+0000","last_change":"2024-02-21T23:05:51.423479+0000","last_active":"2024-02-21T23:05:52.827692+0000","last_peered":"2024-02-21T23:05:52.827692+0000","last_clean":"2024-02-21T23:05:52.827692+0000","last_became_active":"2024-02-21T23:05:51.423108+0000","last_became_peered":"2024-02-21T23:05:51.423108+0000","last_unstale":"2024-02-21T23:05:52.827692+0000","last_undegraded":"2024-02-21T23:05:52.827692+0000","last_fullsized":"2024-02-21T23:05:52.827692+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:36:47.962935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.380595+0000","last_change":"2024-02-21T23:05:51.419436+0000","last_active":"2024-02-21T23:05:52.380595+0000","last_peered":"2024-02-21T23:05:52.380595+0000","last_clean":"2024-02-21T23:05:52.380595+0000","last_became_active":"2024-02-21T23:05:51.418562+0000","last_became_peered":"2024-02-21T23:05:51.418562+0000","last_unstale":"2024-02-21T23:05:52.380595+0000","last_undegraded":"2024-02-21T23:05:52.380595+0000","last_fullsized":"2024-02-21T23:05:52.380595+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:19:45.189126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.381201+0000","last_change":"2024-02-21T23:05:51.410014+0000","last_active":"2024-02-21T23:05:52.381201+0000","last_peered":"2024-02-21T23:05:52.381201+0000","last_clean":"2024-02-21T23:05:52.381201+0000","last_became_active":"2024-02-21T23:05:51.409637+0000","last_became_peered":"2024-02-21T23:05:51.409637+0000","last_unstale":"2024-02-21T23:05:52.381201+0000","last_undegraded":"2024-02-21T23:05:52.381201+0000","last_fullsized":"2024-02-21T23:05:52.381201+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:03:35.243949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.828504+0000","last_change":"2024-02-21T23:05:51.419505+0000","last_active":"2024-02-21T23:05:52.828504+0000","last_peered":"2024-02-21T23:05:52.828504+0000","last_clean":"2024-02-21T23:05:52.828504+0000","last_became_active":"2024-02-21T23:05:51.418040+0000","last_became_peered":"2024-02-21T23:05:51.418040+0000","last_unstale":"2024-02-21T23:05:52.828504+0000","last_undegraded":"2024-02-21T23:05:52.828504+0000","last_fullsized":"2024-02-21T23:05:52.828504+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:57:06.211024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.379325+0000","last_change":"2024-02-21T23:05:51.416524+0000","last_active":"2024-02-21T23:05:52.379325+0000","last_peered":"2024-02-21T23:05:52.379325+0000","last_clean":"2024-02-21T23:05:52.379325+0000","last_became_active":"2024-02-21T23:05:51.415391+0000","last_became_peered":"2024-02-21T23:05:51.415391+0000","last_unstale":"2024-02-21T23:05:52.379325+0000","last_undegraded":"2024-02-21T23:05:52.379325+0000","last_fullsized":"2024-02-21T23:05:52.379325+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:48:32.958103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.383972+0000","last_change":"2024-02-21T23:05:51.421068+0000","last_active":"2024-02-21T23:05:52.383972+0000","last_peered":"2024-02-21T23:05:52.383972+0000","last_clean":"2024-02-21T23:05:52.383972+0000","last_became_active":"2024-02-21T23:05:51.419746+0000","last_became_peered":"2024-02-21T23:05:51.419746+0000","last_unstale":"2024-02-21T23:05:52.383972+0000","last_undegraded":"2024-02-21T23:05:52.383972+0000","last_fullsized":"2024-02-21T23:05:52.383972+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:09:42.444733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.827198+0000","last_change":"2024-02-21T23:05:51.418295+0000","last_active":"2024-02-21T23:05:52.827198+0000","last_peered":"2024-02-21T23:05:52.827198+0000","last_clean":"2024-02-21T23:05:52.827198+0000","last_became_active":"2024-02-21T23:05:51.417780+0000","last_became_peered":"2024-02-21T23:05:51.417780+0000","last_unstale":"2024-02-21T23:05:52.827198+0000","last_undegraded":"2024-02-21T23:05:52.827198+0000","last_fullsized":"2024-02-21T23:05:52.827198+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:19:57.210565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.418290+0000","last_change":"2024-02-21T23:05:51.420518+0000","last_active":"2024-02-21T23:05:52.418290+0000","last_peered":"2024-02-21T23:05:52.418290+0000","last_clean":"2024-02-21T23:05:52.418290+0000","last_became_active":"2024-02-21T23:05:51.419564+0000","last_became_peered":"2024-02-21T23:05:51.419564+0000","last_unstale":"2024-02-21T23:05:52.418290+0000","last_undegraded":"2024-02-21T23:05:52.418290+0000","last_fullsized":"2024-02-21T23:05:52.418290+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:19:41.280587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.827207+0000","last_change":"2024-02-21T23:05:51.421691+0000","last_active":"2024-02-21T23:05:52.827207+0000","last_peered":"2024-02-21T23:05:52.827207+0000","last_clean":"2024-02-21T23:05:52.827207+0000","last_became_active":"2024-02-21T23:05:51.417621+0000","last_became_peered":"2024-02-21T23:05:51.417621+0000","last_unstale":"2024-02-21T23:05:52.827207+0000","last_undegraded":"2024-02-21T23:05:52.827207+0000","last_fullsized":"2024-02-21T23:05:52.827207+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:46:01.608149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.381774+0000","last_change":"2024-02-21T23:05:51.426871+0000","last_active":"2024-02-21T23:05:52.381774+0000","last_peered":"2024-02-21T23:05:52.381774+0000","last_clean":"2024-02-21T23:05:52.381774+0000","last_became_active":"2024-02-21T23:05:51.419412+0000","last_became_peered":"2024-02-21T23:05:51.419412+0000","last_unstale":"2024-02-21T23:05:52.381774+0000","last_undegraded":"2024-02-21T23:05:52.381774+0000","last_fullsized":"2024-02-21T23:05:52.381774+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:22:09.156621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.381402+0000","last_change":"2024-02-21T23:05:51.417593+0000","last_active":"2024-02-21T23:05:52.381402+0000","last_peered":"2024-02-21T23:05:52.381402+0000","last_clean":"2024-02-21T23:05:52.381402+0000","last_became_active":"2024-02-21T23:05:51.416945+0000","last_became_peered":"2024-02-21T23:05:51.416945+0000","last_unstale":"2024-02-21T23:05:52.381402+0000","last_undegraded":"2024-02-21T23:05:52.381402+0000","last_fullsized":"2024-02-21T23:05:52.381402+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:24:32.863072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.417444+0000","last_change":"2024-02-21T23:05:51.409645+0000","last_active":"2024-02-21T23:05:52.417444+0000","last_peered":"2024-02-21T23:05:52.417444+0000","last_clean":"2024-02-21T23:05:52.417444+0000","last_became_active":"2024-02-21T23:05:51.409356+0000","last_became_peered":"2024-02-21T23:05:51.409356+0000","last_unstale":"2024-02-21T23:05:52.417444+0000","last_undegraded":"2024-02-21T23:05:52.417444+0000","last_fullsized":"2024-02-21T23:05:52.417444+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:17:26.329609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.535086+0000","last_change":"2024-02-21T23:05:51.413656+0000","last_active":"2024-02-21T23:05:52.535086+0000","last_peered":"2024-02-21T23:05:52.535086+0000","last_clean":"2024-02-21T23:05:52.535086+0000","last_became_active":"2024-02-21T23:05:51.413350+0000","last_became_peered":"2024-02-21T23:05:51.413350+0000","last_unstale":"2024-02-21T23:05:52.535086+0000","last_undegraded":"2024-02-21T23:05:52.535086+0000","last_fullsized":"2024-02-21T23:05:52.535086+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:19:57.210565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.383335+0000","last_change":"2024-02-21T23:05:51.427044+0000","last_active":"2024-02-21T23:05:52.383335+0000","last_peered":"2024-02-21T23:05:52.383335+0000","last_clean":"2024-02-21T23:05:52.383335+0000","last_became_active":"2024-02-21T23:05:51.419477+0000","last_became_peered":"2024-02-21T23:05:51.419477+0000","last_unstale":"2024-02-21T23:05:52.383335+0000","last_undegraded":"2024-02-21T23:05:52.383335+0000","last_fullsized":"2024-02-21T23:05:52.383335+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:04:00.714448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.827738+0000","last_change":"2024-02-21T23:05:51.425126+0000","last_active":"2024-02-21T23:05:52.827738+0000","last_peered":"2024-02-21T23:05:52.827738+0000","last_clean":"2024-02-21T23:05:52.827738+0000","last_became_active":"2024-02-21T23:05:51.422511+0000","last_became_peered":"2024-02-21T23:05:51.422511+0000","last_unstale":"2024-02-21T23:05:52.827738+0000","last_undegraded":"2024-02-21T23:05:52.827738+0000","last_fullsized":"2024-02-21T23:05:52.827738+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:27:01.191337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.382064+0000","last_change":"2024-02-21T23:05:51.423258+0000","last_active":"2024-02-21T23:05:52.382064+0000","last_peered":"2024-02-21T23:05:52.382064+0000","last_clean":"2024-02-21T23:05:52.382064+0000","last_became_active":"2024-02-21T23:05:51.422987+0000","last_became_peered":"2024-02-21T23:05:51.422987+0000","last_unstale":"2024-02-21T23:05:52.382064+0000","last_undegraded":"2024-02-21T23:05:52.382064+0000","last_fullsized":"2024-02-21T23:05:52.382064+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:41:31.260842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.382602+0000","last_change":"2024-02-21T23:05:51.427468+0000","last_active":"2024-02-21T23:05:52.382602+0000","last_peered":"2024-02-21T23:05:52.382602+0000","last_clean":"2024-02-21T23:05:52.382602+0000","last_became_active":"2024-02-21T23:05:51.425476+0000","last_became_peered":"2024-02-21T23:05:51.425476+0000","last_unstale":"2024-02-21T23:05:52.382602+0000","last_undegraded":"2024-02-21T23:05:52.382602+0000","last_fullsized":"2024-02-21T23:05:52.382602+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:06:18.158392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.384633+0000","last_change":"2024-02-21T23:05:51.415772+0000","last_active":"2024-02-21T23:05:52.384633+0000","last_peered":"2024-02-21T23:05:52.384633+0000","last_clean":"2024-02-21T23:05:52.384633+0000","last_became_active":"2024-02-21T23:05:51.415494+0000","last_became_peered":"2024-02-21T23:05:51.415494+0000","last_unstale":"2024-02-21T23:05:52.384633+0000","last_undegraded":"2024-02-21T23:05:52.384633+0000","last_fullsized":"2024-02-21T23:05:52.384633+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:42:37.006685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.380314+0000","last_change":"2024-02-21T23:05:51.428557+0000","last_active":"2024-02-21T23:05:52.380314+0000","last_peered":"2024-02-21T23:05:52.380314+0000","last_clean":"2024-02-21T23:05:52.380314+0000","last_became_active":"2024-02-21T23:05:51.427612+0000","last_became_peered":"2024-02-21T23:05:51.427612+0000","last_unstale":"2024-02-21T23:05:52.380314+0000","last_undegraded":"2024-02-21T23:05:52.380314+0000","last_fullsized":"2024-02-21T23:05:52.380314+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:38:44.895318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.383489+0000","last_change":"2024-02-21T23:05:51.417302+0000","last_active":"2024-02-21T23:05:52.383489+0000","last_peered":"2024-02-21T23:05:52.383489+0000","last_clean":"2024-02-21T23:05:52.383489+0000","last_became_active":"2024-02-21T23:05:51.414092+0000","last_became_peered":"2024-02-21T23:05:51.414092+0000","last_unstale":"2024-02-21T23:05:52.383489+0000","last_undegraded":"2024-02-21T23:05:52.383489+0000","last_fullsized":"2024-02-21T23:05:52.383489+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:49:01.911867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.417508+0000","last_change":"2024-02-21T23:05:51.406411+0000","last_active":"2024-02-21T23:05:52.417508+0000","last_peered":"2024-02-21T23:05:52.417508+0000","last_clean":"2024-02-21T23:05:52.417508+0000","last_became_active":"2024-02-21T23:05:51.406104+0000","last_became_peered":"2024-02-21T23:05:51.406104+0000","last_unstale":"2024-02-21T23:05:52.417508+0000","last_undegraded":"2024-02-21T23:05:52.417508+0000","last_fullsized":"2024-02-21T23:05:52.417508+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:03:08.789066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.418700+0000","last_change":"2024-02-21T23:05:51.414112+0000","last_active":"2024-02-21T23:05:52.418700+0000","last_peered":"2024-02-21T23:05:52.418700+0000","last_clean":"2024-02-21T23:05:52.418700+0000","last_became_active":"2024-02-21T23:05:51.413796+0000","last_became_peered":"2024-02-21T23:05:51.413796+0000","last_unstale":"2024-02-21T23:05:52.418700+0000","last_undegraded":"2024-02-21T23:05:52.418700+0000","last_fullsized":"2024-02-21T23:05:52.418700+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:42:26.488510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.827519+0000","last_change":"2024-02-21T23:05:51.421208+0000","last_active":"2024-02-21T23:05:52.827519+0000","last_peered":"2024-02-21T23:05:52.827519+0000","last_clean":"2024-02-21T23:05:52.827519+0000","last_became_active":"2024-02-21T23:05:51.418171+0000","last_became_peered":"2024-02-21T23:05:51.418171+0000","last_unstale":"2024-02-21T23:05:52.827519+0000","last_undegraded":"2024-02-21T23:05:52.827519+0000","last_fullsized":"2024-02-21T23:05:52.827519+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:17:26.329609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.535471+0000","last_change":"2024-02-21T23:05:51.416857+0000","last_active":"2024-02-21T23:05:52.535471+0000","last_peered":"2024-02-21T23:05:52.535471+0000","last_clean":"2024-02-21T23:05:52.535471+0000","last_became_active":"2024-02-21T23:05:51.416577+0000","last_became_peered":"2024-02-21T23:05:51.416577+0000","last_unstale":"2024-02-21T23:05:52.535471+0000","last_undegraded":"2024-02-21T23:05:52.535471+0000","last_fullsized":"2024-02-21T23:05:52.535471+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:17:26.329609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.382662+0000","last_change":"2024-02-21T23:05:51.427489+0000","last_active":"2024-02-21T23:05:52.382662+0000","last_peered":"2024-02-21T23:05:52.382662+0000","last_clean":"2024-02-21T23:05:52.382662+0000","last_became_active":"2024-02-21T23:05:51.419458+0000","last_became_peered":"2024-02-21T23:05:51.419458+0000","last_unstale":"2024-02-21T23:05:52.382662+0000","last_undegraded":"2024-02-21T23:05:52.382662+0000","last_fullsized":"2024-02-21T23:05:52.382662+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:54:08.758447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.417641+0000","last_change":"2024-02-21T23:05:51.405741+0000","last_active":"2024-02-21T23:05:52.417641+0000","last_peered":"2024-02-21T23:05:52.417641+0000","last_clean":"2024-02-21T23:05:52.417641+0000","last_became_active":"2024-02-21T23:05:51.405301+0000","last_became_peered":"2024-02-21T23:05:51.405301+0000","last_unstale":"2024-02-21T23:05:52.417641+0000","last_undegraded":"2024-02-21T23:05:52.417641+0000","last_fullsized":"2024-02-21T23:05:52.417641+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:36:47.962935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.535038+0000","last_change":"2024-02-21T23:05:51.398663+0000","last_active":"2024-02-21T23:05:52.535038+0000","last_peered":"2024-02-21T23:05:52.535038+0000","last_clean":"2024-02-21T23:05:52.535038+0000","last_became_active":"2024-02-21T23:05:51.398146+0000","last_became_peered":"2024-02-21T23:05:51.398146+0000","last_unstale":"2024-02-21T23:05:52.535038+0000","last_undegraded":"2024-02-21T23:05:52.535038+0000","last_fullsized":"2024-02-21T23:05:52.535038+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:27:54.695721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.827090+0000","last_change":"2024-02-21T23:05:51.421330+0000","last_active":"2024-02-21T23:05:52.827090+0000","last_peered":"2024-02-21T23:05:52.827090+0000","last_clean":"2024-02-21T23:05:52.827090+0000","last_became_active":"2024-02-21T23:05:51.421012+0000","last_became_peered":"2024-02-21T23:05:51.421012+0000","last_unstale":"2024-02-21T23:05:52.827090+0000","last_undegraded":"2024-02-21T23:05:52.827090+0000","last_fullsized":"2024-02-21T23:05:52.827090+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:26:33.366331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.535369+0000","last_change":"2024-02-21T23:05:51.403173+0000","last_active":"2024-02-21T23:05:52.535369+0000","last_peered":"2024-02-21T23:05:52.535369+0000","last_clean":"2024-02-21T23:05:52.535369+0000","last_became_active":"2024-02-21T23:05:51.402610+0000","last_became_peered":"2024-02-21T23:05:51.402610+0000","last_unstale":"2024-02-21T23:05:52.535369+0000","last_undegraded":"2024-02-21T23:05:52.535369+0000","last_fullsized":"2024-02-21T23:05:52.535369+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:46:01.608149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.826863+0000","last_change":"2024-02-21T23:05:51.416534+0000","last_active":"2024-02-21T23:05:52.826863+0000","last_peered":"2024-02-21T23:05:52.826863+0000","last_clean":"2024-02-21T23:05:52.826863+0000","last_became_active":"2024-02-21T23:05:51.416244+0000","last_became_peered":"2024-02-21T23:05:51.416244+0000","last_unstale":"2024-02-21T23:05:52.826863+0000","last_undegraded":"2024-02-21T23:05:52.826863+0000","last_fullsized":"2024-02-21T23:05:52.826863+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:42:37.006685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"75'4","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.383405+0000","last_change":"2024-02-21T23:05:50.354109+0000","last_active":"2024-02-21T23:05:52.383405+0000","last_peered":"2024-02-21T23:05:52.383405+0000","last_clean":"2024-02-21T23:05:52.383405+0000","last_became_active":"2024-02-21T23:05:50.353809+0000","last_became_peered":"2024-02-21T23:05:50.353809+0000","last_unstale":"2024-02-21T23:05:52.383405+0000","last_undegraded":"2024-02-21T23:05:52.383405+0000","last_fullsized":"2024-02-21T23:05:52.383405+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:28:13.765557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.417929+0000","last_change":"2024-02-21T23:05:51.416268+0000","last_active":"2024-02-21T23:05:52.417929+0000","last_peered":"2024-02-21T23:05:52.417929+0000","last_clean":"2024-02-21T23:05:52.417929+0000","last_became_active":"2024-02-21T23:05:51.415980+0000","last_became_peered":"2024-02-21T23:05:51.415980+0000","last_unstale":"2024-02-21T23:05:52.417929+0000","last_undegraded":"2024-02-21T23:05:52.417929+0000","last_fullsized":"2024-02-21T23:05:52.417929+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:51:17.193881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"75'1","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.535409+0000","last_change":"2024-02-21T23:05:50.383438+0000","last_active":"2024-02-21T23:05:52.535409+0000","last_peered":"2024-02-21T23:05:52.535409+0000","last_clean":"2024-02-21T23:05:52.535409+0000","last_became_active":"2024-02-21T23:05:50.383134+0000","last_became_peered":"2024-02-21T23:05:50.383134+0000","last_unstale":"2024-02-21T23:05:52.535409+0000","last_undegraded":"2024-02-21T23:05:52.535409+0000","last_fullsized":"2024-02-21T23:05:52.535409+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:08:35.448428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.383786+0000","last_change":"2024-02-21T23:05:51.421571+0000","last_active":"2024-02-21T23:05:52.383786+0000","last_peered":"2024-02-21T23:05:52.383786+0000","last_clean":"2024-02-21T23:05:52.383786+0000","last_became_active":"2024-02-21T23:05:51.421110+0000","last_became_peered":"2024-02-21T23:05:51.421110+0000","last_unstale":"2024-02-21T23:05:52.383786+0000","last_undegraded":"2024-02-21T23:05:52.383786+0000","last_fullsized":"2024-02-21T23:05:52.383786+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:56:36.688152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"75'2","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.535656+0000","last_change":"2024-02-21T23:05:50.383231+0000","last_active":"2024-02-21T23:05:52.535656+0000","last_peered":"2024-02-21T23:05:52.535656+0000","last_clean":"2024-02-21T23:05:52.535656+0000","last_became_active":"2024-02-21T23:05:50.382757+0000","last_became_peered":"2024-02-21T23:05:50.382757+0000","last_unstale":"2024-02-21T23:05:52.535656+0000","last_undegraded":"2024-02-21T23:05:52.535656+0000","last_fullsized":"2024-02-21T23:05:52.535656+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:49:36.738197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.384761+0000","last_change":"2024-02-21T23:05:51.415892+0000","last_active":"2024-02-21T23:05:52.384761+0000","last_peered":"2024-02-21T23:05:52.384761+0000","last_clean":"2024-02-21T23:05:52.384761+0000","last_became_active":"2024-02-21T23:05:51.415520+0000","last_became_peered":"2024-02-21T23:05:51.415520+0000","last_unstale":"2024-02-21T23:05:52.384761+0000","last_undegraded":"2024-02-21T23:05:52.384761+0000","last_fullsized":"2024-02-21T23:05:52.384761+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:26:33.366331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"75'2","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.829069+0000","last_change":"2024-02-21T23:05:50.376050+0000","last_active":"2024-02-21T23:05:52.829069+0000","last_peered":"2024-02-21T23:05:52.829069+0000","last_clean":"2024-02-21T23:05:52.829069+0000","last_became_active":"2024-02-21T23:05:50.375731+0000","last_became_peered":"2024-02-21T23:05:50.375731+0000","last_unstale":"2024-02-21T23:05:52.829069+0000","last_undegraded":"2024-02-21T23:05:52.829069+0000","last_fullsized":"2024-02-21T23:05:52.829069+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:42:15.626763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.418119+0000","last_change":"2024-02-21T23:05:51.428435+0000","last_active":"2024-02-21T23:05:52.418119+0000","last_peered":"2024-02-21T23:05:52.418119+0000","last_clean":"2024-02-21T23:05:52.418119+0000","last_became_active":"2024-02-21T23:05:51.428133+0000","last_became_peered":"2024-02-21T23:05:51.428133+0000","last_unstale":"2024-02-21T23:05:52.418119+0000","last_undegraded":"2024-02-21T23:05:52.418119+0000","last_fullsized":"2024-02-21T23:05:52.418119+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:49:46.025250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"75'1","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.384804+0000","last_change":"2024-02-21T23:05:50.377574+0000","last_active":"2024-02-21T23:05:52.384804+0000","last_peered":"2024-02-21T23:05:52.384804+0000","last_clean":"2024-02-21T23:05:52.384804+0000","last_became_active":"2024-02-21T23:05:50.377250+0000","last_became_peered":"2024-02-21T23:05:50.377250+0000","last_unstale":"2024-02-21T23:05:52.384804+0000","last_undegraded":"2024-02-21T23:05:52.384804+0000","last_fullsized":"2024-02-21T23:05:52.384804+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:26:38.366160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.418441+0000","last_change":"2024-02-21T23:05:51.421508+0000","last_active":"2024-02-21T23:05:52.418441+0000","last_peered":"2024-02-21T23:05:52.418441+0000","last_clean":"2024-02-21T23:05:52.418441+0000","last_became_active":"2024-02-21T23:05:51.420041+0000","last_became_peered":"2024-02-21T23:05:51.420041+0000","last_unstale":"2024-02-21T23:05:52.418441+0000","last_undegraded":"2024-02-21T23:05:52.418441+0000","last_fullsized":"2024-02-21T23:05:52.418441+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:10:56.883222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"75'1","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.826394+0000","last_change":"2024-02-21T23:05:50.378668+0000","last_active":"2024-02-21T23:05:52.826394+0000","last_peered":"2024-02-21T23:05:52.826394+0000","last_clean":"2024-02-21T23:05:52.826394+0000","last_became_active":"2024-02-21T23:05:50.378369+0000","last_became_peered":"2024-02-21T23:05:50.378369+0000","last_unstale":"2024-02-21T23:05:52.826394+0000","last_undegraded":"2024-02-21T23:05:52.826394+0000","last_fullsized":"2024-02-21T23:05:52.826394+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:45:05.837402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.384312+0000","last_change":"2024-02-21T23:05:51.419084+0000","last_active":"2024-02-21T23:05:52.384312+0000","last_peered":"2024-02-21T23:05:52.384312+0000","last_clean":"2024-02-21T23:05:52.384312+0000","last_became_active":"2024-02-21T23:05:51.418752+0000","last_became_peered":"2024-02-21T23:05:51.418752+0000","last_unstale":"2024-02-21T23:05:52.384312+0000","last_undegraded":"2024-02-21T23:05:52.384312+0000","last_fullsized":"2024-02-21T23:05:52.384312+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:20:19.564331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.417555+0000","last_change":"2024-02-21T23:05:50.375534+0000","last_active":"2024-02-21T23:05:52.417555+0000","last_peered":"2024-02-21T23:05:52.417555+0000","last_clean":"2024-02-21T23:05:52.417555+0000","last_became_active":"2024-02-21T23:05:50.373762+0000","last_became_peered":"2024-02-21T23:05:50.373762+0000","last_unstale":"2024-02-21T23:05:52.417555+0000","last_undegraded":"2024-02-21T23:05:52.417555+0000","last_fullsized":"2024-02-21T23:05:52.417555+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:49:36.738197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.828380+0000","last_change":"2024-02-21T23:05:51.426711+0000","last_active":"2024-02-21T23:05:52.828380+0000","last_peered":"2024-02-21T23:05:52.828380+0000","last_clean":"2024-02-21T23:05:52.828380+0000","last_became_active":"2024-02-21T23:05:51.426415+0000","last_became_peered":"2024-02-21T23:05:51.426415+0000","last_unstale":"2024-02-21T23:05:52.828380+0000","last_undegraded":"2024-02-21T23:05:52.828380+0000","last_fullsized":"2024-02-21T23:05:52.828380+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:58:55.829941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.379409+0000","last_change":"2024-02-21T23:05:50.355567+0000","last_active":"2024-02-21T23:05:52.379409+0000","last_peered":"2024-02-21T23:05:52.379409+0000","last_clean":"2024-02-21T23:05:52.379409+0000","last_became_active":"2024-02-21T23:05:50.354835+0000","last_became_peered":"2024-02-21T23:05:50.354835+0000","last_unstale":"2024-02-21T23:05:52.379409+0000","last_undegraded":"2024-02-21T23:05:52.379409+0000","last_fullsized":"2024-02-21T23:05:52.379409+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:21:52.655098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.417589+0000","last_change":"2024-02-21T23:05:51.399923+0000","last_active":"2024-02-21T23:05:52.417589+0000","last_peered":"2024-02-21T23:05:52.417589+0000","last_clean":"2024-02-21T23:05:52.417589+0000","last_became_active":"2024-02-21T23:05:51.399642+0000","last_became_peered":"2024-02-21T23:05:51.399642+0000","last_unstale":"2024-02-21T23:05:52.417589+0000","last_undegraded":"2024-02-21T23:05:52.417589+0000","last_fullsized":"2024-02-21T23:05:52.417589+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:21:18.440977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.535856+0000","last_change":"2024-02-21T23:05:50.386639+0000","last_active":"2024-02-21T23:05:52.535856+0000","last_peered":"2024-02-21T23:05:52.535856+0000","last_clean":"2024-02-21T23:05:52.535856+0000","last_became_active":"2024-02-21T23:05:50.386158+0000","last_became_peered":"2024-02-21T23:05:50.386158+0000","last_unstale":"2024-02-21T23:05:52.535856+0000","last_undegraded":"2024-02-21T23:05:52.535856+0000","last_fullsized":"2024-02-21T23:05:52.535856+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:43:17.496894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.826815+0000","last_change":"2024-02-21T23:05:51.422844+0000","last_active":"2024-02-21T23:05:52.826815+0000","last_peered":"2024-02-21T23:05:52.826815+0000","last_clean":"2024-02-21T23:05:52.826815+0000","last_became_active":"2024-02-21T23:05:51.422488+0000","last_became_peered":"2024-02-21T23:05:51.422488+0000","last_unstale":"2024-02-21T23:05:52.826815+0000","last_undegraded":"2024-02-21T23:05:52.826815+0000","last_fullsized":"2024-02-21T23:05:52.826815+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:27:54.695721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.827451+0000","last_change":"2024-02-21T23:05:50.385146+0000","last_active":"2024-02-21T23:05:52.827451+0000","last_peered":"2024-02-21T23:05:52.827451+0000","last_clean":"2024-02-21T23:05:52.827451+0000","last_became_active":"2024-02-21T23:05:50.384418+0000","last_became_peered":"2024-02-21T23:05:50.384418+0000","last_unstale":"2024-02-21T23:05:52.827451+0000","last_undegraded":"2024-02-21T23:05:52.827451+0000","last_fullsized":"2024-02-21T23:05:52.827451+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:43:17.496894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.381706+0000","last_change":"2024-02-21T23:05:51.418004+0000","last_active":"2024-02-21T23:05:52.381706+0000","last_peered":"2024-02-21T23:05:52.381706+0000","last_clean":"2024-02-21T23:05:52.381706+0000","last_became_active":"2024-02-21T23:05:51.417463+0000","last_became_peered":"2024-02-21T23:05:51.417463+0000","last_unstale":"2024-02-21T23:05:52.381706+0000","last_undegraded":"2024-02-21T23:05:52.381706+0000","last_fullsized":"2024-02-21T23:05:52.381706+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:59:13.292494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"75'3","reported_seq":17,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.383080+0000","last_change":"2024-02-21T23:05:50.373189+0000","last_active":"2024-02-21T23:05:52.383080+0000","last_peered":"2024-02-21T23:05:52.383080+0000","last_clean":"2024-02-21T23:05:52.383080+0000","last_became_active":"2024-02-21T23:05:50.370680+0000","last_became_peered":"2024-02-21T23:05:50.370680+0000","last_unstale":"2024-02-21T23:05:52.383080+0000","last_undegraded":"2024-02-21T23:05:52.383080+0000","last_fullsized":"2024-02-21T23:05:52.383080+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:21:03.121546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.827306+0000","last_change":"2024-02-21T23:05:51.405242+0000","last_active":"2024-02-21T23:05:52.827306+0000","last_peered":"2024-02-21T23:05:52.827306+0000","last_clean":"2024-02-21T23:05:52.827306+0000","last_became_active":"2024-02-21T23:05:51.403777+0000","last_became_peered":"2024-02-21T23:05:51.403777+0000","last_unstale":"2024-02-21T23:05:52.827306+0000","last_undegraded":"2024-02-21T23:05:52.827306+0000","last_fullsized":"2024-02-21T23:05:52.827306+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:34:14.420091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"75'6","reported_seq":20,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.829576+0000","last_change":"2024-02-21T23:05:50.374208+0000","last_active":"2024-02-21T23:05:52.829576+0000","last_peered":"2024-02-21T23:05:52.829576+0000","last_clean":"2024-02-21T23:05:52.829576+0000","last_became_active":"2024-02-21T23:05:50.373869+0000","last_became_peered":"2024-02-21T23:05:50.373869+0000","last_unstale":"2024-02-21T23:05:52.829576+0000","last_undegraded":"2024-02-21T23:05:52.829576+0000","last_fullsized":"2024-02-21T23:05:52.829576+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:56:49.435540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.827623+0000","last_change":"2024-02-21T23:05:51.423479+0000","last_active":"2024-02-21T23:05:52.827623+0000","last_peered":"2024-02-21T23:05:52.827623+0000","last_clean":"2024-02-21T23:05:52.827623+0000","last_became_active":"2024-02-21T23:05:51.423190+0000","last_became_peered":"2024-02-21T23:05:51.423190+0000","last_unstale":"2024-02-21T23:05:52.827623+0000","last_undegraded":"2024-02-21T23:05:52.827623+0000","last_fullsized":"2024-02-21T23:05:52.827623+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:03:08.789066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"75'2","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.826505+0000","last_change":"2024-02-21T23:05:50.381643+0000","last_active":"2024-02-21T23:05:52.826505+0000","last_peered":"2024-02-21T23:05:52.826505+0000","last_clean":"2024-02-21T23:05:52.826505+0000","last_became_active":"2024-02-21T23:05:50.381345+0000","last_became_peered":"2024-02-21T23:05:50.381345+0000","last_unstale":"2024-02-21T23:05:52.826505+0000","last_undegraded":"2024-02-21T23:05:52.826505+0000","last_fullsized":"2024-02-21T23:05:52.826505+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:01:27.369089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.828068+0000","last_change":"2024-02-21T23:05:51.416753+0000","last_active":"2024-02-21T23:05:52.828068+0000","last_peered":"2024-02-21T23:05:52.828068+0000","last_clean":"2024-02-21T23:05:52.828068+0000","last_became_active":"2024-02-21T23:05:51.416427+0000","last_became_peered":"2024-02-21T23:05:51.416427+0000","last_unstale":"2024-02-21T23:05:52.828068+0000","last_undegraded":"2024-02-21T23:05:52.828068+0000","last_fullsized":"2024-02-21T23:05:52.828068+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:34:14.420091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.535386+0000","last_change":"2024-02-21T23:05:50.381962+0000","last_active":"2024-02-21T23:05:52.535386+0000","last_peered":"2024-02-21T23:05:52.535386+0000","last_clean":"2024-02-21T23:05:52.535386+0000","last_became_active":"2024-02-21T23:05:50.380838+0000","last_became_peered":"2024-02-21T23:05:50.380838+0000","last_unstale":"2024-02-21T23:05:52.535386+0000","last_undegraded":"2024-02-21T23:05:52.535386+0000","last_fullsized":"2024-02-21T23:05:52.535386+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:10:10.973567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.829697+0000","last_change":"2024-02-21T23:05:51.408507+0000","last_active":"2024-02-21T23:05:52.829697+0000","last_peered":"2024-02-21T23:05:52.829697+0000","last_clean":"2024-02-21T23:05:52.829697+0000","last_became_active":"2024-02-21T23:05:51.408198+0000","last_became_peered":"2024-02-21T23:05:51.408198+0000","last_unstale":"2024-02-21T23:05:52.829697+0000","last_undegraded":"2024-02-21T23:05:52.829697+0000","last_fullsized":"2024-02-21T23:05:52.829697+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:08:36.461250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.379927+0000","last_change":"2024-02-21T23:05:50.353116+0000","last_active":"2024-02-21T23:05:52.379927+0000","last_peered":"2024-02-21T23:05:52.379927+0000","last_clean":"2024-02-21T23:05:52.379927+0000","last_became_active":"2024-02-21T23:05:50.352768+0000","last_became_peered":"2024-02-21T23:05:50.352768+0000","last_unstale":"2024-02-21T23:05:52.379927+0000","last_undegraded":"2024-02-21T23:05:52.379927+0000","last_fullsized":"2024-02-21T23:05:52.379927+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:52:09.073053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.828739+0000","last_change":"2024-02-21T23:05:51.416424+0000","last_active":"2024-02-21T23:05:52.828739+0000","last_peered":"2024-02-21T23:05:52.828739+0000","last_clean":"2024-02-21T23:05:52.828739+0000","last_became_active":"2024-02-21T23:05:51.416070+0000","last_became_peered":"2024-02-21T23:05:51.416070+0000","last_unstale":"2024-02-21T23:05:52.828739+0000","last_undegraded":"2024-02-21T23:05:52.828739+0000","last_fullsized":"2024-02-21T23:05:52.828739+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:56:10.406772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"75'8","reported_seq":22,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.418474+0000","last_change":"2024-02-21T23:05:50.376063+0000","last_active":"2024-02-21T23:05:52.418474+0000","last_peered":"2024-02-21T23:05:52.418474+0000","last_clean":"2024-02-21T23:05:52.418474+0000","last_became_active":"2024-02-21T23:05:50.375772+0000","last_became_peered":"2024-02-21T23:05:50.375772+0000","last_unstale":"2024-02-21T23:05:52.418474+0000","last_undegraded":"2024-02-21T23:05:52.418474+0000","last_fullsized":"2024-02-21T23:05:52.418474+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:26:58.076002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.381136+0000","last_change":"2024-02-21T23:05:51.421565+0000","last_active":"2024-02-21T23:05:52.381136+0000","last_peered":"2024-02-21T23:05:52.381136+0000","last_clean":"2024-02-21T23:05:52.381136+0000","last_became_active":"2024-02-21T23:05:51.421252+0000","last_became_peered":"2024-02-21T23:05:51.421252+0000","last_unstale":"2024-02-21T23:05:52.381136+0000","last_undegraded":"2024-02-21T23:05:52.381136+0000","last_fullsized":"2024-02-21T23:05:52.381136+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:40:45.445418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"75'1","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.536140+0000","last_change":"2024-02-21T23:05:50.383778+0000","last_active":"2024-02-21T23:05:52.536140+0000","last_peered":"2024-02-21T23:05:52.536140+0000","last_clean":"2024-02-21T23:05:52.536140+0000","last_became_active":"2024-02-21T23:05:50.383436+0000","last_became_peered":"2024-02-21T23:05:50.383436+0000","last_unstale":"2024-02-21T23:05:52.536140+0000","last_undegraded":"2024-02-21T23:05:52.536140+0000","last_fullsized":"2024-02-21T23:05:52.536140+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:45:05.837402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.379870+0000","last_change":"2024-02-21T23:05:51.416676+0000","last_active":"2024-02-21T23:05:52.379870+0000","last_peered":"2024-02-21T23:05:52.379870+0000","last_clean":"2024-02-21T23:05:52.379870+0000","last_became_active":"2024-02-21T23:05:51.416297+0000","last_became_peered":"2024-02-21T23:05:51.416297+0000","last_unstale":"2024-02-21T23:05:52.379870+0000","last_undegraded":"2024-02-21T23:05:52.379870+0000","last_fullsized":"2024-02-21T23:05:52.379870+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:29:33.209545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"75'2","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.384553+0000","last_change":"2024-02-21T23:05:50.377181+0000","last_active":"2024-02-21T23:05:52.384553+0000","last_peered":"2024-02-21T23:05:52.384553+0000","last_clean":"2024-02-21T23:05:52.384553+0000","last_became_active":"2024-02-21T23:05:50.373524+0000","last_became_peered":"2024-02-21T23:05:50.373524+0000","last_unstale":"2024-02-21T23:05:52.384553+0000","last_undegraded":"2024-02-21T23:05:52.384553+0000","last_fullsized":"2024-02-21T23:05:52.384553+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:01:18.419728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.380361+0000","last_change":"2024-02-21T23:05:51.412971+0000","last_active":"2024-02-21T23:05:52.380361+0000","last_peered":"2024-02-21T23:05:52.380361+0000","last_clean":"2024-02-21T23:05:52.380361+0000","last_became_active":"2024-02-21T23:05:51.410966+0000","last_became_peered":"2024-02-21T23:05:51.410966+0000","last_unstale":"2024-02-21T23:05:52.380361+0000","last_undegraded":"2024-02-21T23:05:52.380361+0000","last_fullsized":"2024-02-21T23:05:52.380361+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:00:39.163092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"75'1","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.383103+0000","last_change":"2024-02-21T23:05:50.366307+0000","last_active":"2024-02-21T23:05:52.383103+0000","last_peered":"2024-02-21T23:05:52.383103+0000","last_clean":"2024-02-21T23:05:52.383103+0000","last_became_active":"2024-02-21T23:05:50.359426+0000","last_became_peered":"2024-02-21T23:05:50.359426+0000","last_unstale":"2024-02-21T23:05:52.383103+0000","last_undegraded":"2024-02-21T23:05:52.383103+0000","last_fullsized":"2024-02-21T23:05:52.383103+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:13:15.770526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.381306+0000","last_change":"2024-02-21T23:05:51.424067+0000","last_active":"2024-02-21T23:05:52.381306+0000","last_peered":"2024-02-21T23:05:52.381306+0000","last_clean":"2024-02-21T23:05:52.381306+0000","last_became_active":"2024-02-21T23:05:51.423644+0000","last_became_peered":"2024-02-21T23:05:51.423644+0000","last_unstale":"2024-02-21T23:05:52.381306+0000","last_undegraded":"2024-02-21T23:05:52.381306+0000","last_fullsized":"2024-02-21T23:05:52.381306+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:04:03.084512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.827402+0000","last_change":"2024-02-21T23:05:50.377924+0000","last_active":"2024-02-21T23:05:52.827402+0000","last_peered":"2024-02-21T23:05:52.827402+0000","last_clean":"2024-02-21T23:05:52.827402+0000","last_became_active":"2024-02-21T23:05:50.377440+0000","last_became_peered":"2024-02-21T23:05:50.377440+0000","last_unstale":"2024-02-21T23:05:52.827402+0000","last_undegraded":"2024-02-21T23:05:52.827402+0000","last_fullsized":"2024-02-21T23:05:52.827402+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:42:15.626763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.418107+0000","last_change":"2024-02-21T23:05:51.414619+0000","last_active":"2024-02-21T23:05:52.418107+0000","last_peered":"2024-02-21T23:05:52.418107+0000","last_clean":"2024-02-21T23:05:52.418107+0000","last_became_active":"2024-02-21T23:05:51.414059+0000","last_became_peered":"2024-02-21T23:05:51.414059+0000","last_unstale":"2024-02-21T23:05:52.418107+0000","last_undegraded":"2024-02-21T23:05:52.418107+0000","last_fullsized":"2024-02-21T23:05:52.418107+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:58:05.114297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"75'2","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.381450+0000","last_change":"2024-02-21T23:05:50.366877+0000","last_active":"2024-02-21T23:05:52.381450+0000","last_peered":"2024-02-21T23:05:52.381450+0000","last_clean":"2024-02-21T23:05:52.381450+0000","last_became_active":"2024-02-21T23:05:50.365566+0000","last_became_peered":"2024-02-21T23:05:50.365566+0000","last_unstale":"2024-02-21T23:05:52.381450+0000","last_undegraded":"2024-02-21T23:05:52.381450+0000","last_fullsized":"2024-02-21T23:05:52.381450+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:58:04.101475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.418441+0000","last_change":"2024-02-21T23:05:51.411005+0000","last_active":"2024-02-21T23:05:52.418441+0000","last_peered":"2024-02-21T23:05:52.418441+0000","last_clean":"2024-02-21T23:05:52.418441+0000","last_became_active":"2024-02-21T23:05:51.410503+0000","last_became_peered":"2024-02-21T23:05:51.410503+0000","last_unstale":"2024-02-21T23:05:52.418441+0000","last_undegraded":"2024-02-21T23:05:52.418441+0000","last_fullsized":"2024-02-21T23:05:52.418441+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:51:10.610852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.381097+0000","last_change":"2024-02-21T23:05:50.376888+0000","last_active":"2024-02-21T23:05:52.381097+0000","last_peered":"2024-02-21T23:05:52.381097+0000","last_clean":"2024-02-21T23:05:52.381097+0000","last_became_active":"2024-02-21T23:05:50.376610+0000","last_became_peered":"2024-02-21T23:05:50.376610+0000","last_unstale":"2024-02-21T23:05:52.381097+0000","last_undegraded":"2024-02-21T23:05:52.381097+0000","last_fullsized":"2024-02-21T23:05:52.381097+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:51:16.181060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.829202+0000","last_change":"2024-02-21T23:05:51.417167+0000","last_active":"2024-02-21T23:05:52.829202+0000","last_peered":"2024-02-21T23:05:52.829202+0000","last_clean":"2024-02-21T23:05:52.829202+0000","last_became_active":"2024-02-21T23:05:51.415957+0000","last_became_peered":"2024-02-21T23:05:51.415957+0000","last_unstale":"2024-02-21T23:05:52.829202+0000","last_undegraded":"2024-02-21T23:05:52.829202+0000","last_fullsized":"2024-02-21T23:05:52.829202+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:26:59.088824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.381605+0000","last_change":"2024-02-21T23:05:50.375808+0000","last_active":"2024-02-21T23:05:52.381605+0000","last_peered":"2024-02-21T23:05:52.381605+0000","last_clean":"2024-02-21T23:05:52.381605+0000","last_became_active":"2024-02-21T23:05:50.375467+0000","last_became_peered":"2024-02-21T23:05:50.375467+0000","last_unstale":"2024-02-21T23:05:52.381605+0000","last_undegraded":"2024-02-21T23:05:52.381605+0000","last_fullsized":"2024-02-21T23:05:52.381605+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:29:01.979463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.383305+0000","last_change":"2024-02-21T23:05:51.414453+0000","last_active":"2024-02-21T23:05:52.383305+0000","last_peered":"2024-02-21T23:05:52.383305+0000","last_clean":"2024-02-21T23:05:52.383305+0000","last_became_active":"2024-02-21T23:05:51.413499+0000","last_became_peered":"2024-02-21T23:05:51.413499+0000","last_unstale":"2024-02-21T23:05:52.383305+0000","last_undegraded":"2024-02-21T23:05:52.383305+0000","last_fullsized":"2024-02-21T23:05:52.383305+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:55:21.967274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"75'2","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.827140+0000","last_change":"2024-02-21T23:05:50.381948+0000","last_active":"2024-02-21T23:05:52.827140+0000","last_peered":"2024-02-21T23:05:52.827140+0000","last_clean":"2024-02-21T23:05:52.827140+0000","last_became_active":"2024-02-21T23:05:50.381452+0000","last_became_peered":"2024-02-21T23:05:50.381452+0000","last_unstale":"2024-02-21T23:05:52.827140+0000","last_undegraded":"2024-02-21T23:05:52.827140+0000","last_fullsized":"2024-02-21T23:05:52.827140+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:22:16.414686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.828536+0000","last_change":"2024-02-21T23:05:51.410619+0000","last_active":"2024-02-21T23:05:52.828536+0000","last_peered":"2024-02-21T23:05:52.828536+0000","last_clean":"2024-02-21T23:05:52.828536+0000","last_became_active":"2024-02-21T23:05:51.409937+0000","last_became_peered":"2024-02-21T23:05:51.409937+0000","last_unstale":"2024-02-21T23:05:52.828536+0000","last_undegraded":"2024-02-21T23:05:52.828536+0000","last_fullsized":"2024-02-21T23:05:52.828536+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:22:32.845173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"76'1","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.418529+0000","last_change":"2024-02-21T23:05:50.362370+0000","last_active":"2024-02-21T23:05:52.418529+0000","last_peered":"2024-02-21T23:05:52.418529+0000","last_clean":"2024-02-21T23:05:52.418529+0000","last_became_active":"2024-02-21T23:05:50.361848+0000","last_became_peered":"2024-02-21T23:05:50.361848+0000","last_unstale":"2024-02-21T23:05:52.418529+0000","last_undegraded":"2024-02-21T23:05:52.418529+0000","last_fullsized":"2024-02-21T23:05:52.418529+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:08:35.448428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.384236+0000","last_change":"2024-02-21T23:05:51.421357+0000","last_active":"2024-02-21T23:05:52.384236+0000","last_peered":"2024-02-21T23:05:52.384236+0000","last_clean":"2024-02-21T23:05:52.384236+0000","last_became_active":"2024-02-21T23:05:51.420734+0000","last_became_peered":"2024-02-21T23:05:51.420734+0000","last_unstale":"2024-02-21T23:05:52.384236+0000","last_undegraded":"2024-02-21T23:05:52.384236+0000","last_fullsized":"2024-02-21T23:05:52.384236+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:31:44.536584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.535981+0000","last_change":"2024-02-21T23:05:50.384680+0000","last_active":"2024-02-21T23:05:52.535981+0000","last_peered":"2024-02-21T23:05:52.535981+0000","last_clean":"2024-02-21T23:05:52.535981+0000","last_became_active":"2024-02-21T23:05:50.384208+0000","last_became_peered":"2024-02-21T23:05:50.384208+0000","last_unstale":"2024-02-21T23:05:52.535981+0000","last_undegraded":"2024-02-21T23:05:52.535981+0000","last_fullsized":"2024-02-21T23:05:52.535981+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:01:27.369089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.417801+0000","last_change":"2024-02-21T23:05:51.395859+0000","last_active":"2024-02-21T23:05:52.417801+0000","last_peered":"2024-02-21T23:05:52.417801+0000","last_clean":"2024-02-21T23:05:52.417801+0000","last_became_active":"2024-02-21T23:05:51.395012+0000","last_became_peered":"2024-02-21T23:05:51.395012+0000","last_unstale":"2024-02-21T23:05:52.417801+0000","last_undegraded":"2024-02-21T23:05:52.417801+0000","last_fullsized":"2024-02-21T23:05:52.417801+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:29:02.992284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.535249+0000","last_change":"2024-02-21T23:05:50.383920+0000","last_active":"2024-02-21T23:05:52.535249+0000","last_peered":"2024-02-21T23:05:52.535249+0000","last_clean":"2024-02-21T23:05:52.535249+0000","last_became_active":"2024-02-21T23:05:50.382488+0000","last_became_peered":"2024-02-21T23:05:50.382488+0000","last_unstale":"2024-02-21T23:05:52.535249+0000","last_undegraded":"2024-02-21T23:05:52.535249+0000","last_fullsized":"2024-02-21T23:05:52.535249+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:26:58.076002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.829134+0000","last_change":"2024-02-21T23:05:51.422495+0000","last_active":"2024-02-21T23:05:52.829134+0000","last_peered":"2024-02-21T23:05:52.829134+0000","last_clean":"2024-02-21T23:05:52.829134+0000","last_became_active":"2024-02-21T23:05:51.422196+0000","last_became_peered":"2024-02-21T23:05:51.422196+0000","last_unstale":"2024-02-21T23:05:52.829134+0000","last_undegraded":"2024-02-21T23:05:52.829134+0000","last_fullsized":"2024-02-21T23:05:52.829134+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:10:11.986389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.828484+0000","last_change":"2024-02-21T23:05:50.376027+0000","last_active":"2024-02-21T23:05:52.828484+0000","last_peered":"2024-02-21T23:05:52.828484+0000","last_clean":"2024-02-21T23:05:52.828484+0000","last_became_active":"2024-02-21T23:05:50.375170+0000","last_became_peered":"2024-02-21T23:05:50.375170+0000","last_unstale":"2024-02-21T23:05:52.828484+0000","last_undegraded":"2024-02-21T23:05:52.828484+0000","last_fullsized":"2024-02-21T23:05:52.828484+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:39:32.228635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.828497+0000","last_change":"2024-02-21T23:05:51.420500+0000","last_active":"2024-02-21T23:05:52.828497+0000","last_peered":"2024-02-21T23:05:52.828497+0000","last_clean":"2024-02-21T23:05:52.828497+0000","last_became_active":"2024-02-21T23:05:51.420103+0000","last_became_peered":"2024-02-21T23:05:51.420103+0000","last_unstale":"2024-02-21T23:05:52.828497+0000","last_undegraded":"2024-02-21T23:05:52.828497+0000","last_fullsized":"2024-02-21T23:05:52.828497+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:27:01.191337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.380442+0000","last_change":"2024-02-21T23:05:50.354805+0000","last_active":"2024-02-21T23:05:52.380442+0000","last_peered":"2024-02-21T23:05:52.380442+0000","last_clean":"2024-02-21T23:05:52.380442+0000","last_became_active":"2024-02-21T23:05:50.354525+0000","last_became_peered":"2024-02-21T23:05:50.354525+0000","last_unstale":"2024-02-21T23:05:52.380442+0000","last_undegraded":"2024-02-21T23:05:52.380442+0000","last_fullsized":"2024-02-21T23:05:52.380442+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:28:05.314552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.827781+0000","last_change":"2024-02-21T23:05:51.416548+0000","last_active":"2024-02-21T23:05:52.827781+0000","last_peered":"2024-02-21T23:05:52.827781+0000","last_clean":"2024-02-21T23:05:52.827781+0000","last_became_active":"2024-02-21T23:05:51.416222+0000","last_became_peered":"2024-02-21T23:05:51.416222+0000","last_unstale":"2024-02-21T23:05:52.827781+0000","last_undegraded":"2024-02-21T23:05:52.827781+0000","last_fullsized":"2024-02-21T23:05:52.827781+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:22:32.845173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.828804+0000","last_change":"2024-02-21T23:05:50.373554+0000","last_active":"2024-02-21T23:05:52.828804+0000","last_peered":"2024-02-21T23:05:52.828804+0000","last_clean":"2024-02-21T23:05:52.828804+0000","last_became_active":"2024-02-21T23:05:50.373022+0000","last_became_peered":"2024-02-21T23:05:50.373022+0000","last_unstale":"2024-02-21T23:05:52.828804+0000","last_undegraded":"2024-02-21T23:05:52.828804+0000","last_fullsized":"2024-02-21T23:05:52.828804+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:38:56.665723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.827906+0000","last_change":"2024-02-21T23:05:51.412683+0000","last_active":"2024-02-21T23:05:52.827906+0000","last_peered":"2024-02-21T23:05:52.827906+0000","last_clean":"2024-02-21T23:05:52.827906+0000","last_became_active":"2024-02-21T23:05:51.412389+0000","last_became_peered":"2024-02-21T23:05:51.412389+0000","last_unstale":"2024-02-21T23:05:52.827906+0000","last_undegraded":"2024-02-21T23:05:52.827906+0000","last_fullsized":"2024-02-21T23:05:52.827906+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:57:06.211024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.826797+0000","last_change":"2024-02-21T23:05:50.376356+0000","last_active":"2024-02-21T23:05:52.826797+0000","last_peered":"2024-02-21T23:05:52.826797+0000","last_clean":"2024-02-21T23:05:52.826797+0000","last_became_active":"2024-02-21T23:05:50.370467+0000","last_became_peered":"2024-02-21T23:05:50.370467+0000","last_unstale":"2024-02-21T23:05:52.826797+0000","last_undegraded":"2024-02-21T23:05:52.826797+0000","last_fullsized":"2024-02-21T23:05:52.826797+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:56:49.435540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.535891+0000","last_change":"2024-02-21T23:05:51.413008+0000","last_active":"2024-02-21T23:05:52.535891+0000","last_peered":"2024-02-21T23:05:52.535891+0000","last_clean":"2024-02-21T23:05:52.535891+0000","last_became_active":"2024-02-21T23:05:51.412714+0000","last_became_peered":"2024-02-21T23:05:51.412714+0000","last_unstale":"2024-02-21T23:05:52.535891+0000","last_undegraded":"2024-02-21T23:05:52.535891+0000","last_fullsized":"2024-02-21T23:05:52.535891+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:03:08.789066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.381834+0000","last_change":"2024-02-21T23:05:50.375581+0000","last_active":"2024-02-21T23:05:52.381834+0000","last_peered":"2024-02-21T23:05:52.381834+0000","last_clean":"2024-02-21T23:05:52.381834+0000","last_became_active":"2024-02-21T23:05:50.375282+0000","last_became_peered":"2024-02-21T23:05:50.375282+0000","last_unstale":"2024-02-21T23:05:52.381834+0000","last_undegraded":"2024-02-21T23:05:52.381834+0000","last_fullsized":"2024-02-21T23:05:52.381834+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:49:45.012428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":274,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.535353+0000","last_change":"2024-02-21T23:02:32.289335+0000","last_active":"2024-02-21T23:05:52.535353+0000","last_peered":"2024-02-21T23:05:52.535353+0000","last_clean":"2024-02-21T23:05:52.535353+0000","last_became_active":"2024-02-21T23:02:32.286913+0000","last_became_peered":"2024-02-21T23:02:32.286913+0000","last_unstale":"2024-02-21T23:05:52.535353+0000","last_undegraded":"2024-02-21T23:05:52.535353+0000","last_fullsized":"2024-02-21T23:05:52.535353+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_clean_scrub_stamp":"2024-02-21T23:02:31.172569+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:36:23.987374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574891,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32156,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339684,"statfs":{"total":96636764160,"available":96603836416,"internally_reserved":0,"allocated":999424,"data_stored":1033908,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574892,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32104,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339736,"statfs":{"total":96636764160,"available":96603889664,"internally_reserved":0,"allocated":942080,"data_stored":445130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574891,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27940,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343900,"statfs":{"total":96636764160,"available":96608153600,"internally_reserved":0,"allocated":933888,"data_stored":443552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574892,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32116,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339724,"statfs":{"total":96636764160,"available":96603877376,"internally_reserved":0,"allocated":950272,"data_stored":445234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574892,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32100,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339740,"statfs":{"total":96636764160,"available":96603893760,"internally_reserved":0,"allocated":933888,"data_stored":444100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574893,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32108,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339732,"statfs":{"total":96636764160,"available":96603885568,"internally_reserved":0,"allocated":942080,"data_stored":444260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574891,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32096,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339744,"statfs":{"total":96636764160,"available":96603897856,"internally_reserved":0,"allocated":937984,"data_stored":443630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574891,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32152,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339688,"statfs":{"total":96636764160,"available":96603840512,"internally_reserved":0,"allocated":995328,"data_stored":1033886,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-21T23:05:57.056 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-02-21T23:05:57.318 INFO:tasks.ceph.mds.a.smithi028.stderr:2024-02-21T23:05:57.330+0000 7ff6f2600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-21T23:05:57.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.332+0000 7f3217c00640 1 Processor -- start 2024-02-21T23:05:57.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.332+0000 7f3217c00640 1 -- start start 2024-02-21T23:05:57.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.333+0000 7f3217c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f3210073270 0x7f32100c0d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:57.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.334+0000 7f3217c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3210158d90 0x7f32100726f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:57.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.334+0000 7f3217c00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3210158600 con 0x7f32100c1b50 2024-02-21T23:05:57.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.334+0000 7f3217c00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32100c5910 con 0x7f3210073270 2024-02-21T23:05:57.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.334+0000 7f3217c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32100c5bf0 con 0x7f3210158d90 2024-02-21T23:05:57.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.334+0000 7f320fe00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f3210073270 0x7f32100c0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:57.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.334+0000 7f3215c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3210158d90 0x7f32100726f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:57.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.334+0000 7f320fe00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f3210073270 0x7f32100c0d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.28:51700/0 (socket says 172.21.15.28:51700) 2024-02-21T23:05:57.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.334+0000 7f320fe00640 1 -- 172.21.15.28:0/1524750305 learned_addr learned my addr 172.21.15.28:0/1524750305 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:57.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.335+0000 7f320fe00640 1 -- 172.21.15.28:0/1524750305 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3210158d90 msgr2=0x7f32100726f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:57.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.335+0000 7f320fe00640 1 --2- 172.21.15.28:0/1524750305 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3210158d90 0x7f32100726f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:57.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.335+0000 7f320fe00640 1 -- 172.21.15.28:0/1524750305 >> v1:172.21.15.81:6789/0 conn(0x7f32100c1b50 legacy=0x7f32101559f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:57.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.335+0000 7f320fe00640 1 -- 172.21.15.28:0/1524750305 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3210158870 con 0x7f3210073270 2024-02-21T23:05:57.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.337+0000 7f320fe00640 1 --2- 172.21.15.28:0/1524750305 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3210073270 0x7f32100c0d10 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f31fc009870 tx=0x7f31fc02b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c7d782f68c535c3b server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:57.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.337+0000 7f320f400640 1 -- 172.21.15.28:0/1524750305 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31fc0317c0 con 0x7f3210073270 2024-02-21T23:05:57.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.337+0000 7f320f400640 1 -- 172.21.15.28:0/1524750305 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31fc037070 con 0x7f3210073270 2024-02-21T23:05:57.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.337+0000 7f320f400640 1 -- 172.21.15.28:0/1524750305 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31fc031b80 con 0x7f3210073270 2024-02-21T23:05:57.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.338+0000 7f3217c00640 1 -- 172.21.15.28:0/1524750305 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3210073270 msgr2=0x7f32100c0d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:57.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.338+0000 7f3217c00640 1 --2- 172.21.15.28:0/1524750305 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3210073270 0x7f32100c0d10 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f31fc009870 tx=0x7f31fc02b860 comp rx=0 tx=0).stop 2024-02-21T23:05:57.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.338+0000 7f3217c00640 1 -- 172.21.15.28:0/1524750305 shutdown_connections 2024-02-21T23:05:57.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.338+0000 7f3217c00640 1 --2- 172.21.15.28:0/1524750305 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3210158d90 0x7f32100726f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:57.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.338+0000 7f3217c00640 1 --2- 172.21.15.28:0/1524750305 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3210073270 0x7f32100c0d10 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:57.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.338+0000 7f3217c00640 1 -- 172.21.15.28:0/1524750305 >> 172.21.15.28:0/1524750305 conn(0x7f321006c5b0 msgr2=0x7f32100c78e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:57.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.338+0000 7f3217c00640 1 -- 172.21.15.28:0/1524750305 shutdown_connections 2024-02-21T23:05:57.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.338+0000 7f3217c00640 1 -- 172.21.15.28:0/1524750305 wait complete. 2024-02-21T23:05:57.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.342+0000 7f3217c00640 1 Processor -- start 2024-02-21T23:05:57.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.343+0000 7f3217c00640 1 -- start start 2024-02-21T23:05:57.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.345+0000 7f3217c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3210073270 0x7f3210150190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:57.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.345+0000 7f3217c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f32100c1b50 0x7f32101510e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:57.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.345+0000 7f3217c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3210158d90 0x7f3210147870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:57.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.345+0000 7f3217c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32100c6050 con 0x7f3210158d90 2024-02-21T23:05:57.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.345+0000 7f3217c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32100c5ed0 con 0x7f32100c1b50 2024-02-21T23:05:57.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.345+0000 7f3217c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3210072c30 con 0x7f3210073270 2024-02-21T23:05:57.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.345+0000 7f3215c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3210158d90 0x7f3210147870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:57.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.345+0000 7f3215c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3210158d90 0x7f3210147870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54148/0 (socket says 172.21.15.28:54148) 2024-02-21T23:05:57.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.345+0000 7f3215c00640 1 -- 172.21.15.28:0/1928495451 learned_addr learned my addr 172.21.15.28:0/1928495451 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:57.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.345+0000 7f320fe00640 1 --2- 172.21.15.28:0/1928495451 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f32100c1b50 0x7f32101510e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:57.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.345+0000 7f3215200640 1 --2- 172.21.15.28:0/1928495451 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3210073270 0x7f3210150190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:57.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.346+0000 7f3215c00640 1 -- 172.21.15.28:0/1928495451 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3210073270 msgr2=0x7f3210150190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:57.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.346+0000 7f3215c00640 1 --2- 172.21.15.28:0/1928495451 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3210073270 0x7f3210150190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:57.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.346+0000 7f3215c00640 1 -- 172.21.15.28:0/1928495451 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f32100c1b50 msgr2=0x7f32101510e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:57.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.346+0000 7f3215c00640 1 --2- 172.21.15.28:0/1928495451 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f32100c1b50 0x7f32101510e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:57.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.346+0000 7f3215c00640 1 -- 172.21.15.28:0/1928495451 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32101ee5f0 con 0x7f3210158d90 2024-02-21T23:05:57.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.346+0000 7f3215200640 1 --2- 172.21.15.28:0/1928495451 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3210073270 0x7f3210150190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:05:57.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.346+0000 7f3215c00640 1 --2- 172.21.15.28:0/1928495451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3210158d90 0x7f3210147870 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f3208009d30 tx=0x7f320800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:57.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.347+0000 7f320cc00640 1 -- 172.21.15.28:0/1928495451 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3208018020 con 0x7f3210158d90 2024-02-21T23:05:57.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.347+0000 7f3217c00640 1 -- 172.21.15.28:0/1928495451 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32101ee890 con 0x7f3210158d90 2024-02-21T23:05:57.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.347+0000 7f320cc00640 1 -- 172.21.15.28:0/1928495451 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3208014480 con 0x7f3210158d90 2024-02-21T23:05:57.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.347+0000 7f320cc00640 1 -- 172.21.15.28:0/1928495451 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3208014870 con 0x7f3210158d90 2024-02-21T23:05:57.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.347+0000 7f3217c00640 1 -- 172.21.15.28:0/1928495451 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32101eee50 con 0x7f3210158d90 2024-02-21T23:05:57.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.348+0000 7f3217c00640 1 -- 172.21.15.28:0/1928495451 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3210068640 con 0x7f3210158d90 2024-02-21T23:05:57.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.349+0000 7f320cc00640 1 -- 172.21.15.28:0/1928495451 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3208005d10 con 0x7f3210158d90 2024-02-21T23:05:57.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.350+0000 7f320cc00640 1 --2- 172.21.15.28:0/1928495451 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f31f006b4d0 0x7f31f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:57.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.350+0000 7f320cc00640 1 -- 172.21.15.28:0/1928495451 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f320808ae60 con 0x7f3210158d90 2024-02-21T23:05:57.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.350+0000 7f3215200640 1 --2- 172.21.15.28:0/1928495451 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f31f006b4d0 0x7f31f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:57.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.351+0000 7f3215200640 1 --2- 172.21.15.28:0/1928495451 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f31f006b4d0 0x7f31f006d990 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f3204001700 tx=0x7f3204009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:57.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.354+0000 7f320cc00640 1 -- 172.21.15.28:0/1928495451 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3208022070 con 0x7f3210158d90 2024-02-21T23:05:57.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.673+0000 7f3217c00640 1 -- 172.21.15.28:0/1928495451 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f3210156500 con 0x7f3210158d90 2024-02-21T23:05:57.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.675+0000 7f320cc00640 1 -- 172.21.15.28:0/1928495451 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1432 (secure 0 0 0) 0x7f3208010040 con 0x7f3210158d90 2024-02-21T23:05:57.663 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:05:57.663 INFO:teuthology.orchestra.run.smithi028.stdout:{"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":220,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":76,"num_osds":8,"num_up_osds":8,"osd_up_since":1708556550,"num_in_osds":8,"osd_in_since":1708556541,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":24,"data_bytes":592778,"bytes_used":258838528,"bytes_avail":772835274752,"bytes_total":773094113280,"write_bytes_sec":2242,"read_op_per_sec":0,"write_op_per_sec":7},"fsmap":{"epoch":62,"id":5,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":5,"rank":0,"name":"a","status":"up:active","gid":6366}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-02-21T23:05:28.726167+0000","services":{"mds":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-02-21T23:05:57.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.678+0000 7f3217c00640 1 -- 172.21.15.28:0/1928495451 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f31f006b4d0 msgr2=0x7f31f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:57.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.678+0000 7f3217c00640 1 --2- 172.21.15.28:0/1928495451 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f31f006b4d0 0x7f31f006d990 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f3204001700 tx=0x7f3204009040 comp rx=0 tx=0).stop 2024-02-21T23:05:57.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.678+0000 7f3217c00640 1 -- 172.21.15.28:0/1928495451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3210158d90 msgr2=0x7f3210147870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:57.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.678+0000 7f3217c00640 1 --2- 172.21.15.28:0/1928495451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3210158d90 0x7f3210147870 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f3208009d30 tx=0x7f320800c490 comp rx=0 tx=0).stop 2024-02-21T23:05:57.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.678+0000 7f3217c00640 1 -- 172.21.15.28:0/1928495451 shutdown_connections 2024-02-21T23:05:57.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.678+0000 7f3217c00640 1 --2- 172.21.15.28:0/1928495451 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f31f006b4d0 0x7f31f006d990 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:57.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.678+0000 7f3217c00640 1 --2- 172.21.15.28:0/1928495451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3210158d90 0x7f3210147870 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:57.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.678+0000 7f3217c00640 1 --2- 172.21.15.28:0/1928495451 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f32100c1b50 0x7f32101510e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:57.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.678+0000 7f3217c00640 1 --2- 172.21.15.28:0/1928495451 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3210073270 0x7f3210150190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:57.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.678+0000 7f3217c00640 1 -- 172.21.15.28:0/1928495451 >> 172.21.15.28:0/1928495451 conn(0x7f321006c5b0 msgr2=0x7f32101546f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:57.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.678+0000 7f3217c00640 1 -- 172.21.15.28:0/1928495451 shutdown_connections 2024-02-21T23:05:57.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.678+0000 7f3217c00640 1 -- 172.21.15.28:0/1928495451 wait complete. 2024-02-21T23:05:57.687 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-02-21T23:05:57.687 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-21T23:05:57.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.916+0000 7fe09c800640 1 Processor -- start 2024-02-21T23:05:57.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.916+0000 7fe09c800640 1 -- start start 2024-02-21T23:05:57.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.918+0000 7fe09c800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe09413e6c0 0x7fe0940ca8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:57.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.918+0000 7fe09c800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe0940cade0 0x7fe094156ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:57.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.918+0000 7fe09c800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe09413eaa0 con 0x7fe0940c6f80 2024-02-21T23:05:57.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.918+0000 7fe09c800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe094155cd0 con 0x7fe09413e6c0 2024-02-21T23:05:57.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.918+0000 7fe09c800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0940c6780 con 0x7fe0940cade0 2024-02-21T23:05:57.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.918+0000 7fe099e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe09413e6c0 0x7fe0940ca8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:57.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.918+0000 7fe099400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe0940c6f80 0x7fe0941565d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43834/0 (socket says 172.21.15.28:43834) 2024-02-21T23:05:57.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.918+0000 7fe099e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe09413e6c0 0x7fe0940ca8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54160/0 (socket says 172.21.15.28:54160) 2024-02-21T23:05:57.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.918+0000 7fe099e00640 1 -- 172.21.15.28:0/2322885990 learned_addr learned my addr 172.21.15.28:0/2322885990 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:57.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.919+0000 7fe099e00640 1 -- 172.21.15.28:0/2322885990 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe0940cade0 msgr2=0x7fe094156ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:05:57.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.919+0000 7fe09a800640 1 --2- 172.21.15.28:0/2322885990 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe0940cade0 0x7fe094156ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:57.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.919+0000 7fe099e00640 1 --2- 172.21.15.28:0/2322885990 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe0940cade0 0x7fe094156ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:57.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.919+0000 7fe099e00640 1 -- 172.21.15.28:0/2322885990 >> v1:172.21.15.28:6789/0 conn(0x7fe0940c6f80 legacy=0x7fe0941565d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:57.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.919+0000 7fe099e00640 1 -- 172.21.15.28:0/2322885990 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0941550e0 con 0x7fe09413e6c0 2024-02-21T23:05:57.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.920+0000 7fe099e00640 1 --2- 172.21.15.28:0/2322885990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe09413e6c0 0x7fe0940ca8a0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fe08c009e50 tx=0x7fe08c02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c67b32646a8e89b server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:57.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.920+0000 7fe098a00640 1 -- 172.21.15.28:0/2322885990 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe08c02c9f0 con 0x7fe09413e6c0 2024-02-21T23:05:57.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.921+0000 7fe098a00640 1 -- 172.21.15.28:0/2322885990 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe08c02cb90 con 0x7fe09413e6c0 2024-02-21T23:05:57.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.921+0000 7fe098a00640 1 -- 172.21.15.28:0/2322885990 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe08c0343f0 con 0x7fe09413e6c0 2024-02-21T23:05:57.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.921+0000 7fe09c800640 1 -- 172.21.15.28:0/2322885990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe09413e6c0 msgr2=0x7fe0940ca8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:57.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.921+0000 7fe09c800640 1 --2- 172.21.15.28:0/2322885990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe09413e6c0 0x7fe0940ca8a0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fe08c009e50 tx=0x7fe08c02be50 comp rx=0 tx=0).stop 2024-02-21T23:05:57.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.921+0000 7fe09c800640 1 -- 172.21.15.28:0/2322885990 shutdown_connections 2024-02-21T23:05:57.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.921+0000 7fe09c800640 1 --2- 172.21.15.28:0/2322885990 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe0940cade0 0x7fe094156ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:57.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.921+0000 7fe09c800640 1 --2- 172.21.15.28:0/2322885990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe09413e6c0 0x7fe0940ca8a0 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:57.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.921+0000 7fe09c800640 1 -- 172.21.15.28:0/2322885990 >> 172.21.15.28:0/2322885990 conn(0x7fe09408b610 msgr2=0x7fe0940725a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:57.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.921+0000 7fe09c800640 1 -- 172.21.15.28:0/2322885990 shutdown_connections 2024-02-21T23:05:57.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.922+0000 7fe09c800640 1 -- 172.21.15.28:0/2322885990 wait complete. 2024-02-21T23:05:57.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.923+0000 7fe09c800640 1 Processor -- start 2024-02-21T23:05:57.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.923+0000 7fe09c800640 1 -- start start 2024-02-21T23:05:57.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.924+0000 7fe09c800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0940c6f80 0x7fe0941ee050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:57.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.924+0000 7fe09c800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe0940cade0 0x7fe0940c3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:57.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.924+0000 7fe09c800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe09413e6c0 0x7fe0940c4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:57.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.924+0000 7fe09c800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe094155960 con 0x7fe0940c6f80 2024-02-21T23:05:57.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.924+0000 7fe09c800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe094155680 con 0x7fe09413e6c0 2024-02-21T23:05:57.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.924+0000 7fe09c800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0940cb7a0 con 0x7fe0940cade0 2024-02-21T23:05:57.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.924+0000 7fe099e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0940c6f80 0x7fe0941ee050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:57.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.924+0000 7fe099e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0940c6f80 0x7fe0941ee050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54168/0 (socket says 172.21.15.28:54168) 2024-02-21T23:05:57.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.924+0000 7fe099e00640 1 -- 172.21.15.28:0/730628727 learned_addr learned my addr 172.21.15.28:0/730628727 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:05:57.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.925+0000 7fe099400640 1 --2- 172.21.15.28:0/730628727 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe0940cade0 0x7fe0940c3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:57.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.925+0000 7fe099e00640 1 -- 172.21.15.28:0/730628727 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe0940cade0 msgr2=0x7fe0940c3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:57.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.925+0000 7fe099e00640 1 --2- 172.21.15.28:0/730628727 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe0940cade0 0x7fe0940c3ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:57.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.925+0000 7fe099e00640 1 -- 172.21.15.28:0/730628727 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe09413e6c0 msgr2=0x7fe0940c4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:57.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.925+0000 7fe099e00640 1 --2- 172.21.15.28:0/730628727 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe09413e6c0 0x7fe0940c4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:57.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.925+0000 7fe099e00640 1 -- 172.21.15.28:0/730628727 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe09413ddb0 con 0x7fe0940c6f80 2024-02-21T23:05:57.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.925+0000 7fe099e00640 1 --2- 172.21.15.28:0/730628727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0940c6f80 0x7fe0941ee050 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fe08c036040 tx=0x7fe08c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:57.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.926+0000 7fe08a000640 1 -- 172.21.15.28:0/730628727 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe08c044020 con 0x7fe0940c6f80 2024-02-21T23:05:57.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.926+0000 7fe08a000640 1 -- 172.21.15.28:0/730628727 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe08c040470 con 0x7fe0940c6f80 2024-02-21T23:05:57.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.926+0000 7fe09c800640 1 -- 172.21.15.28:0/730628727 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0941eea40 con 0x7fe0940c6f80 2024-02-21T23:05:57.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.926+0000 7fe08a000640 1 -- 172.21.15.28:0/730628727 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe08c040860 con 0x7fe0940c6f80 2024-02-21T23:05:57.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.926+0000 7fe09c800640 1 -- 172.21.15.28:0/730628727 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe09413c0a0 con 0x7fe0940c6f80 2024-02-21T23:05:57.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.927+0000 7fe09c800640 1 -- 172.21.15.28:0/730628727 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe094158430 con 0x7fe0940c6f80 2024-02-21T23:05:57.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.928+0000 7fe08a000640 1 -- 172.21.15.28:0/730628727 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe08c047020 con 0x7fe0940c6f80 2024-02-21T23:05:57.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.930+0000 7fe08a000640 1 --2- 172.21.15.28:0/730628727 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe07806b4d0 0x7fe07806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:05:57.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.930+0000 7fe08a000640 1 -- 172.21.15.28:0/730628727 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe08c0acf30 con 0x7fe0940c6f80 2024-02-21T23:05:57.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.930+0000 7fe099400640 1 --2- 172.21.15.28:0/730628727 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe07806b4d0 0x7fe07806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:05:57.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.931+0000 7fe099400640 1 --2- 172.21.15.28:0/730628727 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe07806b4d0 0x7fe07806d990 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fe0940ca250 tx=0x7fe084009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:05:57.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:57.934+0000 7fe08a000640 1 -- 172.21.15.28:0/730628727 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe094158430 con 0x7fe0940c6f80 2024-02-21T23:05:58.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:58.130+0000 7fe09c800640 1 -- 172.21.15.28:0/730628727 --> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe094067f30 con 0x7fe07806b4d0 2024-02-21T23:05:58.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:58.149+0000 7fe08a000640 1 -- 172.21.15.28:0/730628727 <== mgr.4101 v2:172.21.15.81:6832/1848538577 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252150 (secure 0 0 0) 0x7fe094067f30 con 0x7fe07806b4d0 2024-02-21T23:05:58.141 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:05:58.142 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-21T23:05:58.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:58.154+0000 7fe09c800640 1 -- 172.21.15.28:0/730628727 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe07806b4d0 msgr2=0x7fe07806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:58.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:58.154+0000 7fe09c800640 1 --2- 172.21.15.28:0/730628727 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe07806b4d0 0x7fe07806d990 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fe0940ca250 tx=0x7fe084009040 comp rx=0 tx=0).stop 2024-02-21T23:05:58.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:58.154+0000 7fe09c800640 1 -- 172.21.15.28:0/730628727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0940c6f80 msgr2=0x7fe0941ee050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:05:58.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:58.154+0000 7fe09c800640 1 --2- 172.21.15.28:0/730628727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0940c6f80 0x7fe0941ee050 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fe08c036040 tx=0x7fe08c004a40 comp rx=0 tx=0).stop 2024-02-21T23:05:58.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:58.154+0000 7fe09c800640 1 -- 172.21.15.28:0/730628727 shutdown_connections 2024-02-21T23:05:58.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:58.154+0000 7fe09c800640 1 --2- 172.21.15.28:0/730628727 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe07806b4d0 0x7fe07806d990 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:58.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:58.154+0000 7fe09c800640 1 --2- 172.21.15.28:0/730628727 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe0940cade0 0x7fe0940c3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:58.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:58.154+0000 7fe09c800640 1 --2- 172.21.15.28:0/730628727 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe09413e6c0 0x7fe0940c4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:58.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:58.154+0000 7fe09c800640 1 --2- 172.21.15.28:0/730628727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0940c6f80 0x7fe0941ee050 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:05:58.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:58.154+0000 7fe09c800640 1 -- 172.21.15.28:0/730628727 >> 172.21.15.28:0/730628727 conn(0x7fe09408b610 msgr2=0x7fe094156d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:05:58.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:58.154+0000 7fe09c800640 1 -- 172.21.15.28:0/730628727 shutdown_connections 2024-02-21T23:05:58.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:05:58.154+0000 7fe09c800640 1 -- 172.21.15.28:0/730628727 wait complete. 2024-02-21T23:05:58.161 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":172,"stamp":"2024-02-21T23:05:56.735092+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":252772,"kb_used_data":7456,"kb_used_omap":12,"kb_used_meta":245171,"kb_avail":754721948,"statfs":{"total":773094113280,"available":772835274752,"internally_reserved":0,"allocated":7634944,"data_stored":4733700,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":251055665},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":43,"apply_latency_ms":43,"commit_latency_ns":43000000,"apply_latency_ns":43000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.393211"},"pg_stats":[{"pgid":"12.3f","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.535895+0000","last_change":"2024-02-21T23:05:51.408831+0000","last_active":"2024-02-21T23:05:52.535895+0000","last_peered":"2024-02-21T23:05:52.535895+0000","last_clean":"2024-02-21T23:05:52.535895+0000","last_became_active":"2024-02-21T23:05:51.408353+0000","last_became_peered":"2024-02-21T23:05:51.408353+0000","last_unstale":"2024-02-21T23:05:52.535895+0000","last_undegraded":"2024-02-21T23:05:52.535895+0000","last_fullsized":"2024-02-21T23:05:52.535895+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:36:47.962935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.418527+0000","last_change":"2024-02-21T23:05:51.420557+0000","last_active":"2024-02-21T23:05:52.418527+0000","last_peered":"2024-02-21T23:05:52.418527+0000","last_clean":"2024-02-21T23:05:52.418527+0000","last_became_active":"2024-02-21T23:05:51.420078+0000","last_became_peered":"2024-02-21T23:05:51.420078+0000","last_unstale":"2024-02-21T23:05:52.418527+0000","last_undegraded":"2024-02-21T23:05:52.418527+0000","last_fullsized":"2024-02-21T23:05:52.418527+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:36:23.792395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.827683+0000","last_change":"2024-02-21T23:05:51.408934+0000","last_active":"2024-02-21T23:05:52.827683+0000","last_peered":"2024-02-21T23:05:52.827683+0000","last_clean":"2024-02-21T23:05:52.827683+0000","last_became_active":"2024-02-21T23:05:51.408597+0000","last_became_peered":"2024-02-21T23:05:51.408597+0000","last_unstale":"2024-02-21T23:05:52.827683+0000","last_undegraded":"2024-02-21T23:05:52.827683+0000","last_fullsized":"2024-02-21T23:05:52.827683+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:58:55.829941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.827692+0000","last_change":"2024-02-21T23:05:51.423479+0000","last_active":"2024-02-21T23:05:52.827692+0000","last_peered":"2024-02-21T23:05:52.827692+0000","last_clean":"2024-02-21T23:05:52.827692+0000","last_became_active":"2024-02-21T23:05:51.423108+0000","last_became_peered":"2024-02-21T23:05:51.423108+0000","last_unstale":"2024-02-21T23:05:52.827692+0000","last_undegraded":"2024-02-21T23:05:52.827692+0000","last_fullsized":"2024-02-21T23:05:52.827692+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:36:47.962935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.380595+0000","last_change":"2024-02-21T23:05:51.419436+0000","last_active":"2024-02-21T23:05:52.380595+0000","last_peered":"2024-02-21T23:05:52.380595+0000","last_clean":"2024-02-21T23:05:52.380595+0000","last_became_active":"2024-02-21T23:05:51.418562+0000","last_became_peered":"2024-02-21T23:05:51.418562+0000","last_unstale":"2024-02-21T23:05:52.380595+0000","last_undegraded":"2024-02-21T23:05:52.380595+0000","last_fullsized":"2024-02-21T23:05:52.380595+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:19:45.189126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.381201+0000","last_change":"2024-02-21T23:05:51.410014+0000","last_active":"2024-02-21T23:05:52.381201+0000","last_peered":"2024-02-21T23:05:52.381201+0000","last_clean":"2024-02-21T23:05:52.381201+0000","last_became_active":"2024-02-21T23:05:51.409637+0000","last_became_peered":"2024-02-21T23:05:51.409637+0000","last_unstale":"2024-02-21T23:05:52.381201+0000","last_undegraded":"2024-02-21T23:05:52.381201+0000","last_fullsized":"2024-02-21T23:05:52.381201+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:03:35.243949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.828504+0000","last_change":"2024-02-21T23:05:51.419505+0000","last_active":"2024-02-21T23:05:52.828504+0000","last_peered":"2024-02-21T23:05:52.828504+0000","last_clean":"2024-02-21T23:05:52.828504+0000","last_became_active":"2024-02-21T23:05:51.418040+0000","last_became_peered":"2024-02-21T23:05:51.418040+0000","last_unstale":"2024-02-21T23:05:52.828504+0000","last_undegraded":"2024-02-21T23:05:52.828504+0000","last_fullsized":"2024-02-21T23:05:52.828504+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:57:06.211024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.379325+0000","last_change":"2024-02-21T23:05:51.416524+0000","last_active":"2024-02-21T23:05:52.379325+0000","last_peered":"2024-02-21T23:05:52.379325+0000","last_clean":"2024-02-21T23:05:52.379325+0000","last_became_active":"2024-02-21T23:05:51.415391+0000","last_became_peered":"2024-02-21T23:05:51.415391+0000","last_unstale":"2024-02-21T23:05:52.379325+0000","last_undegraded":"2024-02-21T23:05:52.379325+0000","last_fullsized":"2024-02-21T23:05:52.379325+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:48:32.958103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.383972+0000","last_change":"2024-02-21T23:05:51.421068+0000","last_active":"2024-02-21T23:05:52.383972+0000","last_peered":"2024-02-21T23:05:52.383972+0000","last_clean":"2024-02-21T23:05:52.383972+0000","last_became_active":"2024-02-21T23:05:51.419746+0000","last_became_peered":"2024-02-21T23:05:51.419746+0000","last_unstale":"2024-02-21T23:05:52.383972+0000","last_undegraded":"2024-02-21T23:05:52.383972+0000","last_fullsized":"2024-02-21T23:05:52.383972+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:09:42.444733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.827198+0000","last_change":"2024-02-21T23:05:51.418295+0000","last_active":"2024-02-21T23:05:52.827198+0000","last_peered":"2024-02-21T23:05:52.827198+0000","last_clean":"2024-02-21T23:05:52.827198+0000","last_became_active":"2024-02-21T23:05:51.417780+0000","last_became_peered":"2024-02-21T23:05:51.417780+0000","last_unstale":"2024-02-21T23:05:52.827198+0000","last_undegraded":"2024-02-21T23:05:52.827198+0000","last_fullsized":"2024-02-21T23:05:52.827198+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:19:57.210565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.418290+0000","last_change":"2024-02-21T23:05:51.420518+0000","last_active":"2024-02-21T23:05:52.418290+0000","last_peered":"2024-02-21T23:05:52.418290+0000","last_clean":"2024-02-21T23:05:52.418290+0000","last_became_active":"2024-02-21T23:05:51.419564+0000","last_became_peered":"2024-02-21T23:05:51.419564+0000","last_unstale":"2024-02-21T23:05:52.418290+0000","last_undegraded":"2024-02-21T23:05:52.418290+0000","last_fullsized":"2024-02-21T23:05:52.418290+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:19:41.280587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.827207+0000","last_change":"2024-02-21T23:05:51.421691+0000","last_active":"2024-02-21T23:05:52.827207+0000","last_peered":"2024-02-21T23:05:52.827207+0000","last_clean":"2024-02-21T23:05:52.827207+0000","last_became_active":"2024-02-21T23:05:51.417621+0000","last_became_peered":"2024-02-21T23:05:51.417621+0000","last_unstale":"2024-02-21T23:05:52.827207+0000","last_undegraded":"2024-02-21T23:05:52.827207+0000","last_fullsized":"2024-02-21T23:05:52.827207+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:46:01.608149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.381774+0000","last_change":"2024-02-21T23:05:51.426871+0000","last_active":"2024-02-21T23:05:52.381774+0000","last_peered":"2024-02-21T23:05:52.381774+0000","last_clean":"2024-02-21T23:05:52.381774+0000","last_became_active":"2024-02-21T23:05:51.419412+0000","last_became_peered":"2024-02-21T23:05:51.419412+0000","last_unstale":"2024-02-21T23:05:52.381774+0000","last_undegraded":"2024-02-21T23:05:52.381774+0000","last_fullsized":"2024-02-21T23:05:52.381774+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:22:09.156621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.381402+0000","last_change":"2024-02-21T23:05:51.417593+0000","last_active":"2024-02-21T23:05:52.381402+0000","last_peered":"2024-02-21T23:05:52.381402+0000","last_clean":"2024-02-21T23:05:52.381402+0000","last_became_active":"2024-02-21T23:05:51.416945+0000","last_became_peered":"2024-02-21T23:05:51.416945+0000","last_unstale":"2024-02-21T23:05:52.381402+0000","last_undegraded":"2024-02-21T23:05:52.381402+0000","last_fullsized":"2024-02-21T23:05:52.381402+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:24:32.863072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.417444+0000","last_change":"2024-02-21T23:05:51.409645+0000","last_active":"2024-02-21T23:05:52.417444+0000","last_peered":"2024-02-21T23:05:52.417444+0000","last_clean":"2024-02-21T23:05:52.417444+0000","last_became_active":"2024-02-21T23:05:51.409356+0000","last_became_peered":"2024-02-21T23:05:51.409356+0000","last_unstale":"2024-02-21T23:05:52.417444+0000","last_undegraded":"2024-02-21T23:05:52.417444+0000","last_fullsized":"2024-02-21T23:05:52.417444+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:17:26.329609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.535086+0000","last_change":"2024-02-21T23:05:51.413656+0000","last_active":"2024-02-21T23:05:52.535086+0000","last_peered":"2024-02-21T23:05:52.535086+0000","last_clean":"2024-02-21T23:05:52.535086+0000","last_became_active":"2024-02-21T23:05:51.413350+0000","last_became_peered":"2024-02-21T23:05:51.413350+0000","last_unstale":"2024-02-21T23:05:52.535086+0000","last_undegraded":"2024-02-21T23:05:52.535086+0000","last_fullsized":"2024-02-21T23:05:52.535086+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:19:57.210565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.383335+0000","last_change":"2024-02-21T23:05:51.427044+0000","last_active":"2024-02-21T23:05:52.383335+0000","last_peered":"2024-02-21T23:05:52.383335+0000","last_clean":"2024-02-21T23:05:52.383335+0000","last_became_active":"2024-02-21T23:05:51.419477+0000","last_became_peered":"2024-02-21T23:05:51.419477+0000","last_unstale":"2024-02-21T23:05:52.383335+0000","last_undegraded":"2024-02-21T23:05:52.383335+0000","last_fullsized":"2024-02-21T23:05:52.383335+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:04:00.714448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.827738+0000","last_change":"2024-02-21T23:05:51.425126+0000","last_active":"2024-02-21T23:05:52.827738+0000","last_peered":"2024-02-21T23:05:52.827738+0000","last_clean":"2024-02-21T23:05:52.827738+0000","last_became_active":"2024-02-21T23:05:51.422511+0000","last_became_peered":"2024-02-21T23:05:51.422511+0000","last_unstale":"2024-02-21T23:05:52.827738+0000","last_undegraded":"2024-02-21T23:05:52.827738+0000","last_fullsized":"2024-02-21T23:05:52.827738+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:27:01.191337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.382064+0000","last_change":"2024-02-21T23:05:51.423258+0000","last_active":"2024-02-21T23:05:52.382064+0000","last_peered":"2024-02-21T23:05:52.382064+0000","last_clean":"2024-02-21T23:05:52.382064+0000","last_became_active":"2024-02-21T23:05:51.422987+0000","last_became_peered":"2024-02-21T23:05:51.422987+0000","last_unstale":"2024-02-21T23:05:52.382064+0000","last_undegraded":"2024-02-21T23:05:52.382064+0000","last_fullsized":"2024-02-21T23:05:52.382064+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:41:31.260842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.382602+0000","last_change":"2024-02-21T23:05:51.427468+0000","last_active":"2024-02-21T23:05:52.382602+0000","last_peered":"2024-02-21T23:05:52.382602+0000","last_clean":"2024-02-21T23:05:52.382602+0000","last_became_active":"2024-02-21T23:05:51.425476+0000","last_became_peered":"2024-02-21T23:05:51.425476+0000","last_unstale":"2024-02-21T23:05:52.382602+0000","last_undegraded":"2024-02-21T23:05:52.382602+0000","last_fullsized":"2024-02-21T23:05:52.382602+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:06:18.158392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.384633+0000","last_change":"2024-02-21T23:05:51.415772+0000","last_active":"2024-02-21T23:05:52.384633+0000","last_peered":"2024-02-21T23:05:52.384633+0000","last_clean":"2024-02-21T23:05:52.384633+0000","last_became_active":"2024-02-21T23:05:51.415494+0000","last_became_peered":"2024-02-21T23:05:51.415494+0000","last_unstale":"2024-02-21T23:05:52.384633+0000","last_undegraded":"2024-02-21T23:05:52.384633+0000","last_fullsized":"2024-02-21T23:05:52.384633+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:42:37.006685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.380314+0000","last_change":"2024-02-21T23:05:51.428557+0000","last_active":"2024-02-21T23:05:52.380314+0000","last_peered":"2024-02-21T23:05:52.380314+0000","last_clean":"2024-02-21T23:05:52.380314+0000","last_became_active":"2024-02-21T23:05:51.427612+0000","last_became_peered":"2024-02-21T23:05:51.427612+0000","last_unstale":"2024-02-21T23:05:52.380314+0000","last_undegraded":"2024-02-21T23:05:52.380314+0000","last_fullsized":"2024-02-21T23:05:52.380314+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:38:44.895318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.383489+0000","last_change":"2024-02-21T23:05:51.417302+0000","last_active":"2024-02-21T23:05:52.383489+0000","last_peered":"2024-02-21T23:05:52.383489+0000","last_clean":"2024-02-21T23:05:52.383489+0000","last_became_active":"2024-02-21T23:05:51.414092+0000","last_became_peered":"2024-02-21T23:05:51.414092+0000","last_unstale":"2024-02-21T23:05:52.383489+0000","last_undegraded":"2024-02-21T23:05:52.383489+0000","last_fullsized":"2024-02-21T23:05:52.383489+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:49:01.911867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.417508+0000","last_change":"2024-02-21T23:05:51.406411+0000","last_active":"2024-02-21T23:05:52.417508+0000","last_peered":"2024-02-21T23:05:52.417508+0000","last_clean":"2024-02-21T23:05:52.417508+0000","last_became_active":"2024-02-21T23:05:51.406104+0000","last_became_peered":"2024-02-21T23:05:51.406104+0000","last_unstale":"2024-02-21T23:05:52.417508+0000","last_undegraded":"2024-02-21T23:05:52.417508+0000","last_fullsized":"2024-02-21T23:05:52.417508+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:03:08.789066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.418700+0000","last_change":"2024-02-21T23:05:51.414112+0000","last_active":"2024-02-21T23:05:52.418700+0000","last_peered":"2024-02-21T23:05:52.418700+0000","last_clean":"2024-02-21T23:05:52.418700+0000","last_became_active":"2024-02-21T23:05:51.413796+0000","last_became_peered":"2024-02-21T23:05:51.413796+0000","last_unstale":"2024-02-21T23:05:52.418700+0000","last_undegraded":"2024-02-21T23:05:52.418700+0000","last_fullsized":"2024-02-21T23:05:52.418700+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:42:26.488510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.827519+0000","last_change":"2024-02-21T23:05:51.421208+0000","last_active":"2024-02-21T23:05:52.827519+0000","last_peered":"2024-02-21T23:05:52.827519+0000","last_clean":"2024-02-21T23:05:52.827519+0000","last_became_active":"2024-02-21T23:05:51.418171+0000","last_became_peered":"2024-02-21T23:05:51.418171+0000","last_unstale":"2024-02-21T23:05:52.827519+0000","last_undegraded":"2024-02-21T23:05:52.827519+0000","last_fullsized":"2024-02-21T23:05:52.827519+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:17:26.329609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.535471+0000","last_change":"2024-02-21T23:05:51.416857+0000","last_active":"2024-02-21T23:05:52.535471+0000","last_peered":"2024-02-21T23:05:52.535471+0000","last_clean":"2024-02-21T23:05:52.535471+0000","last_became_active":"2024-02-21T23:05:51.416577+0000","last_became_peered":"2024-02-21T23:05:51.416577+0000","last_unstale":"2024-02-21T23:05:52.535471+0000","last_undegraded":"2024-02-21T23:05:52.535471+0000","last_fullsized":"2024-02-21T23:05:52.535471+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:17:26.329609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.382662+0000","last_change":"2024-02-21T23:05:51.427489+0000","last_active":"2024-02-21T23:05:52.382662+0000","last_peered":"2024-02-21T23:05:52.382662+0000","last_clean":"2024-02-21T23:05:52.382662+0000","last_became_active":"2024-02-21T23:05:51.419458+0000","last_became_peered":"2024-02-21T23:05:51.419458+0000","last_unstale":"2024-02-21T23:05:52.382662+0000","last_undegraded":"2024-02-21T23:05:52.382662+0000","last_fullsized":"2024-02-21T23:05:52.382662+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:54:08.758447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.417641+0000","last_change":"2024-02-21T23:05:51.405741+0000","last_active":"2024-02-21T23:05:52.417641+0000","last_peered":"2024-02-21T23:05:52.417641+0000","last_clean":"2024-02-21T23:05:52.417641+0000","last_became_active":"2024-02-21T23:05:51.405301+0000","last_became_peered":"2024-02-21T23:05:51.405301+0000","last_unstale":"2024-02-21T23:05:52.417641+0000","last_undegraded":"2024-02-21T23:05:52.417641+0000","last_fullsized":"2024-02-21T23:05:52.417641+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:36:47.962935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.535038+0000","last_change":"2024-02-21T23:05:51.398663+0000","last_active":"2024-02-21T23:05:52.535038+0000","last_peered":"2024-02-21T23:05:52.535038+0000","last_clean":"2024-02-21T23:05:52.535038+0000","last_became_active":"2024-02-21T23:05:51.398146+0000","last_became_peered":"2024-02-21T23:05:51.398146+0000","last_unstale":"2024-02-21T23:05:52.535038+0000","last_undegraded":"2024-02-21T23:05:52.535038+0000","last_fullsized":"2024-02-21T23:05:52.535038+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:27:54.695721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.827090+0000","last_change":"2024-02-21T23:05:51.421330+0000","last_active":"2024-02-21T23:05:52.827090+0000","last_peered":"2024-02-21T23:05:52.827090+0000","last_clean":"2024-02-21T23:05:52.827090+0000","last_became_active":"2024-02-21T23:05:51.421012+0000","last_became_peered":"2024-02-21T23:05:51.421012+0000","last_unstale":"2024-02-21T23:05:52.827090+0000","last_undegraded":"2024-02-21T23:05:52.827090+0000","last_fullsized":"2024-02-21T23:05:52.827090+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:26:33.366331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.535369+0000","last_change":"2024-02-21T23:05:51.403173+0000","last_active":"2024-02-21T23:05:52.535369+0000","last_peered":"2024-02-21T23:05:52.535369+0000","last_clean":"2024-02-21T23:05:52.535369+0000","last_became_active":"2024-02-21T23:05:51.402610+0000","last_became_peered":"2024-02-21T23:05:51.402610+0000","last_unstale":"2024-02-21T23:05:52.535369+0000","last_undegraded":"2024-02-21T23:05:52.535369+0000","last_fullsized":"2024-02-21T23:05:52.535369+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:46:01.608149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.826863+0000","last_change":"2024-02-21T23:05:51.416534+0000","last_active":"2024-02-21T23:05:52.826863+0000","last_peered":"2024-02-21T23:05:52.826863+0000","last_clean":"2024-02-21T23:05:52.826863+0000","last_became_active":"2024-02-21T23:05:51.416244+0000","last_became_peered":"2024-02-21T23:05:51.416244+0000","last_unstale":"2024-02-21T23:05:52.826863+0000","last_undegraded":"2024-02-21T23:05:52.826863+0000","last_fullsized":"2024-02-21T23:05:52.826863+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:42:37.006685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"75'4","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.383405+0000","last_change":"2024-02-21T23:05:50.354109+0000","last_active":"2024-02-21T23:05:52.383405+0000","last_peered":"2024-02-21T23:05:52.383405+0000","last_clean":"2024-02-21T23:05:52.383405+0000","last_became_active":"2024-02-21T23:05:50.353809+0000","last_became_peered":"2024-02-21T23:05:50.353809+0000","last_unstale":"2024-02-21T23:05:52.383405+0000","last_undegraded":"2024-02-21T23:05:52.383405+0000","last_fullsized":"2024-02-21T23:05:52.383405+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:28:13.765557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.417929+0000","last_change":"2024-02-21T23:05:51.416268+0000","last_active":"2024-02-21T23:05:52.417929+0000","last_peered":"2024-02-21T23:05:52.417929+0000","last_clean":"2024-02-21T23:05:52.417929+0000","last_became_active":"2024-02-21T23:05:51.415980+0000","last_became_peered":"2024-02-21T23:05:51.415980+0000","last_unstale":"2024-02-21T23:05:52.417929+0000","last_undegraded":"2024-02-21T23:05:52.417929+0000","last_fullsized":"2024-02-21T23:05:52.417929+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:51:17.193881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"75'1","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.535409+0000","last_change":"2024-02-21T23:05:50.383438+0000","last_active":"2024-02-21T23:05:52.535409+0000","last_peered":"2024-02-21T23:05:52.535409+0000","last_clean":"2024-02-21T23:05:52.535409+0000","last_became_active":"2024-02-21T23:05:50.383134+0000","last_became_peered":"2024-02-21T23:05:50.383134+0000","last_unstale":"2024-02-21T23:05:52.535409+0000","last_undegraded":"2024-02-21T23:05:52.535409+0000","last_fullsized":"2024-02-21T23:05:52.535409+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:08:35.448428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.383786+0000","last_change":"2024-02-21T23:05:51.421571+0000","last_active":"2024-02-21T23:05:52.383786+0000","last_peered":"2024-02-21T23:05:52.383786+0000","last_clean":"2024-02-21T23:05:52.383786+0000","last_became_active":"2024-02-21T23:05:51.421110+0000","last_became_peered":"2024-02-21T23:05:51.421110+0000","last_unstale":"2024-02-21T23:05:52.383786+0000","last_undegraded":"2024-02-21T23:05:52.383786+0000","last_fullsized":"2024-02-21T23:05:52.383786+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:56:36.688152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"75'2","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.535656+0000","last_change":"2024-02-21T23:05:50.383231+0000","last_active":"2024-02-21T23:05:52.535656+0000","last_peered":"2024-02-21T23:05:52.535656+0000","last_clean":"2024-02-21T23:05:52.535656+0000","last_became_active":"2024-02-21T23:05:50.382757+0000","last_became_peered":"2024-02-21T23:05:50.382757+0000","last_unstale":"2024-02-21T23:05:52.535656+0000","last_undegraded":"2024-02-21T23:05:52.535656+0000","last_fullsized":"2024-02-21T23:05:52.535656+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:49:36.738197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.384761+0000","last_change":"2024-02-21T23:05:51.415892+0000","last_active":"2024-02-21T23:05:52.384761+0000","last_peered":"2024-02-21T23:05:52.384761+0000","last_clean":"2024-02-21T23:05:52.384761+0000","last_became_active":"2024-02-21T23:05:51.415520+0000","last_became_peered":"2024-02-21T23:05:51.415520+0000","last_unstale":"2024-02-21T23:05:52.384761+0000","last_undegraded":"2024-02-21T23:05:52.384761+0000","last_fullsized":"2024-02-21T23:05:52.384761+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:26:33.366331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"75'2","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.829069+0000","last_change":"2024-02-21T23:05:50.376050+0000","last_active":"2024-02-21T23:05:52.829069+0000","last_peered":"2024-02-21T23:05:52.829069+0000","last_clean":"2024-02-21T23:05:52.829069+0000","last_became_active":"2024-02-21T23:05:50.375731+0000","last_became_peered":"2024-02-21T23:05:50.375731+0000","last_unstale":"2024-02-21T23:05:52.829069+0000","last_undegraded":"2024-02-21T23:05:52.829069+0000","last_fullsized":"2024-02-21T23:05:52.829069+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:42:15.626763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.418119+0000","last_change":"2024-02-21T23:05:51.428435+0000","last_active":"2024-02-21T23:05:52.418119+0000","last_peered":"2024-02-21T23:05:52.418119+0000","last_clean":"2024-02-21T23:05:52.418119+0000","last_became_active":"2024-02-21T23:05:51.428133+0000","last_became_peered":"2024-02-21T23:05:51.428133+0000","last_unstale":"2024-02-21T23:05:52.418119+0000","last_undegraded":"2024-02-21T23:05:52.418119+0000","last_fullsized":"2024-02-21T23:05:52.418119+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:49:46.025250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"75'1","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.384804+0000","last_change":"2024-02-21T23:05:50.377574+0000","last_active":"2024-02-21T23:05:52.384804+0000","last_peered":"2024-02-21T23:05:52.384804+0000","last_clean":"2024-02-21T23:05:52.384804+0000","last_became_active":"2024-02-21T23:05:50.377250+0000","last_became_peered":"2024-02-21T23:05:50.377250+0000","last_unstale":"2024-02-21T23:05:52.384804+0000","last_undegraded":"2024-02-21T23:05:52.384804+0000","last_fullsized":"2024-02-21T23:05:52.384804+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:26:38.366160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.418441+0000","last_change":"2024-02-21T23:05:51.421508+0000","last_active":"2024-02-21T23:05:52.418441+0000","last_peered":"2024-02-21T23:05:52.418441+0000","last_clean":"2024-02-21T23:05:52.418441+0000","last_became_active":"2024-02-21T23:05:51.420041+0000","last_became_peered":"2024-02-21T23:05:51.420041+0000","last_unstale":"2024-02-21T23:05:52.418441+0000","last_undegraded":"2024-02-21T23:05:52.418441+0000","last_fullsized":"2024-02-21T23:05:52.418441+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:10:56.883222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"75'1","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.826394+0000","last_change":"2024-02-21T23:05:50.378668+0000","last_active":"2024-02-21T23:05:52.826394+0000","last_peered":"2024-02-21T23:05:52.826394+0000","last_clean":"2024-02-21T23:05:52.826394+0000","last_became_active":"2024-02-21T23:05:50.378369+0000","last_became_peered":"2024-02-21T23:05:50.378369+0000","last_unstale":"2024-02-21T23:05:52.826394+0000","last_undegraded":"2024-02-21T23:05:52.826394+0000","last_fullsized":"2024-02-21T23:05:52.826394+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:45:05.837402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.384312+0000","last_change":"2024-02-21T23:05:51.419084+0000","last_active":"2024-02-21T23:05:52.384312+0000","last_peered":"2024-02-21T23:05:52.384312+0000","last_clean":"2024-02-21T23:05:52.384312+0000","last_became_active":"2024-02-21T23:05:51.418752+0000","last_became_peered":"2024-02-21T23:05:51.418752+0000","last_unstale":"2024-02-21T23:05:52.384312+0000","last_undegraded":"2024-02-21T23:05:52.384312+0000","last_fullsized":"2024-02-21T23:05:52.384312+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:20:19.564331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.417555+0000","last_change":"2024-02-21T23:05:50.375534+0000","last_active":"2024-02-21T23:05:52.417555+0000","last_peered":"2024-02-21T23:05:52.417555+0000","last_clean":"2024-02-21T23:05:52.417555+0000","last_became_active":"2024-02-21T23:05:50.373762+0000","last_became_peered":"2024-02-21T23:05:50.373762+0000","last_unstale":"2024-02-21T23:05:52.417555+0000","last_undegraded":"2024-02-21T23:05:52.417555+0000","last_fullsized":"2024-02-21T23:05:52.417555+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:49:36.738197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.828380+0000","last_change":"2024-02-21T23:05:51.426711+0000","last_active":"2024-02-21T23:05:52.828380+0000","last_peered":"2024-02-21T23:05:52.828380+0000","last_clean":"2024-02-21T23:05:52.828380+0000","last_became_active":"2024-02-21T23:05:51.426415+0000","last_became_peered":"2024-02-21T23:05:51.426415+0000","last_unstale":"2024-02-21T23:05:52.828380+0000","last_undegraded":"2024-02-21T23:05:52.828380+0000","last_fullsized":"2024-02-21T23:05:52.828380+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:58:55.829941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.379409+0000","last_change":"2024-02-21T23:05:50.355567+0000","last_active":"2024-02-21T23:05:52.379409+0000","last_peered":"2024-02-21T23:05:52.379409+0000","last_clean":"2024-02-21T23:05:52.379409+0000","last_became_active":"2024-02-21T23:05:50.354835+0000","last_became_peered":"2024-02-21T23:05:50.354835+0000","last_unstale":"2024-02-21T23:05:52.379409+0000","last_undegraded":"2024-02-21T23:05:52.379409+0000","last_fullsized":"2024-02-21T23:05:52.379409+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:21:52.655098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.417589+0000","last_change":"2024-02-21T23:05:51.399923+0000","last_active":"2024-02-21T23:05:52.417589+0000","last_peered":"2024-02-21T23:05:52.417589+0000","last_clean":"2024-02-21T23:05:52.417589+0000","last_became_active":"2024-02-21T23:05:51.399642+0000","last_became_peered":"2024-02-21T23:05:51.399642+0000","last_unstale":"2024-02-21T23:05:52.417589+0000","last_undegraded":"2024-02-21T23:05:52.417589+0000","last_fullsized":"2024-02-21T23:05:52.417589+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:21:18.440977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.535856+0000","last_change":"2024-02-21T23:05:50.386639+0000","last_active":"2024-02-21T23:05:52.535856+0000","last_peered":"2024-02-21T23:05:52.535856+0000","last_clean":"2024-02-21T23:05:52.535856+0000","last_became_active":"2024-02-21T23:05:50.386158+0000","last_became_peered":"2024-02-21T23:05:50.386158+0000","last_unstale":"2024-02-21T23:05:52.535856+0000","last_undegraded":"2024-02-21T23:05:52.535856+0000","last_fullsized":"2024-02-21T23:05:52.535856+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:43:17.496894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.826815+0000","last_change":"2024-02-21T23:05:51.422844+0000","last_active":"2024-02-21T23:05:52.826815+0000","last_peered":"2024-02-21T23:05:52.826815+0000","last_clean":"2024-02-21T23:05:52.826815+0000","last_became_active":"2024-02-21T23:05:51.422488+0000","last_became_peered":"2024-02-21T23:05:51.422488+0000","last_unstale":"2024-02-21T23:05:52.826815+0000","last_undegraded":"2024-02-21T23:05:52.826815+0000","last_fullsized":"2024-02-21T23:05:52.826815+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:27:54.695721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.827451+0000","last_change":"2024-02-21T23:05:50.385146+0000","last_active":"2024-02-21T23:05:52.827451+0000","last_peered":"2024-02-21T23:05:52.827451+0000","last_clean":"2024-02-21T23:05:52.827451+0000","last_became_active":"2024-02-21T23:05:50.384418+0000","last_became_peered":"2024-02-21T23:05:50.384418+0000","last_unstale":"2024-02-21T23:05:52.827451+0000","last_undegraded":"2024-02-21T23:05:52.827451+0000","last_fullsized":"2024-02-21T23:05:52.827451+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:43:17.496894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.381706+0000","last_change":"2024-02-21T23:05:51.418004+0000","last_active":"2024-02-21T23:05:52.381706+0000","last_peered":"2024-02-21T23:05:52.381706+0000","last_clean":"2024-02-21T23:05:52.381706+0000","last_became_active":"2024-02-21T23:05:51.417463+0000","last_became_peered":"2024-02-21T23:05:51.417463+0000","last_unstale":"2024-02-21T23:05:52.381706+0000","last_undegraded":"2024-02-21T23:05:52.381706+0000","last_fullsized":"2024-02-21T23:05:52.381706+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:59:13.292494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"75'3","reported_seq":17,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.383080+0000","last_change":"2024-02-21T23:05:50.373189+0000","last_active":"2024-02-21T23:05:52.383080+0000","last_peered":"2024-02-21T23:05:52.383080+0000","last_clean":"2024-02-21T23:05:52.383080+0000","last_became_active":"2024-02-21T23:05:50.370680+0000","last_became_peered":"2024-02-21T23:05:50.370680+0000","last_unstale":"2024-02-21T23:05:52.383080+0000","last_undegraded":"2024-02-21T23:05:52.383080+0000","last_fullsized":"2024-02-21T23:05:52.383080+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:21:03.121546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.827306+0000","last_change":"2024-02-21T23:05:51.405242+0000","last_active":"2024-02-21T23:05:52.827306+0000","last_peered":"2024-02-21T23:05:52.827306+0000","last_clean":"2024-02-21T23:05:52.827306+0000","last_became_active":"2024-02-21T23:05:51.403777+0000","last_became_peered":"2024-02-21T23:05:51.403777+0000","last_unstale":"2024-02-21T23:05:52.827306+0000","last_undegraded":"2024-02-21T23:05:52.827306+0000","last_fullsized":"2024-02-21T23:05:52.827306+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:34:14.420091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"75'6","reported_seq":20,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.829576+0000","last_change":"2024-02-21T23:05:50.374208+0000","last_active":"2024-02-21T23:05:52.829576+0000","last_peered":"2024-02-21T23:05:52.829576+0000","last_clean":"2024-02-21T23:05:52.829576+0000","last_became_active":"2024-02-21T23:05:50.373869+0000","last_became_peered":"2024-02-21T23:05:50.373869+0000","last_unstale":"2024-02-21T23:05:52.829576+0000","last_undegraded":"2024-02-21T23:05:52.829576+0000","last_fullsized":"2024-02-21T23:05:52.829576+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:56:49.435540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.827623+0000","last_change":"2024-02-21T23:05:51.423479+0000","last_active":"2024-02-21T23:05:52.827623+0000","last_peered":"2024-02-21T23:05:52.827623+0000","last_clean":"2024-02-21T23:05:52.827623+0000","last_became_active":"2024-02-21T23:05:51.423190+0000","last_became_peered":"2024-02-21T23:05:51.423190+0000","last_unstale":"2024-02-21T23:05:52.827623+0000","last_undegraded":"2024-02-21T23:05:52.827623+0000","last_fullsized":"2024-02-21T23:05:52.827623+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:03:08.789066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"75'2","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.826505+0000","last_change":"2024-02-21T23:05:50.381643+0000","last_active":"2024-02-21T23:05:52.826505+0000","last_peered":"2024-02-21T23:05:52.826505+0000","last_clean":"2024-02-21T23:05:52.826505+0000","last_became_active":"2024-02-21T23:05:50.381345+0000","last_became_peered":"2024-02-21T23:05:50.381345+0000","last_unstale":"2024-02-21T23:05:52.826505+0000","last_undegraded":"2024-02-21T23:05:52.826505+0000","last_fullsized":"2024-02-21T23:05:52.826505+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:01:27.369089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.828068+0000","last_change":"2024-02-21T23:05:51.416753+0000","last_active":"2024-02-21T23:05:52.828068+0000","last_peered":"2024-02-21T23:05:52.828068+0000","last_clean":"2024-02-21T23:05:52.828068+0000","last_became_active":"2024-02-21T23:05:51.416427+0000","last_became_peered":"2024-02-21T23:05:51.416427+0000","last_unstale":"2024-02-21T23:05:52.828068+0000","last_undegraded":"2024-02-21T23:05:52.828068+0000","last_fullsized":"2024-02-21T23:05:52.828068+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:34:14.420091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.535386+0000","last_change":"2024-02-21T23:05:50.381962+0000","last_active":"2024-02-21T23:05:52.535386+0000","last_peered":"2024-02-21T23:05:52.535386+0000","last_clean":"2024-02-21T23:05:52.535386+0000","last_became_active":"2024-02-21T23:05:50.380838+0000","last_became_peered":"2024-02-21T23:05:50.380838+0000","last_unstale":"2024-02-21T23:05:52.535386+0000","last_undegraded":"2024-02-21T23:05:52.535386+0000","last_fullsized":"2024-02-21T23:05:52.535386+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:10:10.973567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.829697+0000","last_change":"2024-02-21T23:05:51.408507+0000","last_active":"2024-02-21T23:05:52.829697+0000","last_peered":"2024-02-21T23:05:52.829697+0000","last_clean":"2024-02-21T23:05:52.829697+0000","last_became_active":"2024-02-21T23:05:51.408198+0000","last_became_peered":"2024-02-21T23:05:51.408198+0000","last_unstale":"2024-02-21T23:05:52.829697+0000","last_undegraded":"2024-02-21T23:05:52.829697+0000","last_fullsized":"2024-02-21T23:05:52.829697+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:08:36.461250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.379927+0000","last_change":"2024-02-21T23:05:50.353116+0000","last_active":"2024-02-21T23:05:52.379927+0000","last_peered":"2024-02-21T23:05:52.379927+0000","last_clean":"2024-02-21T23:05:52.379927+0000","last_became_active":"2024-02-21T23:05:50.352768+0000","last_became_peered":"2024-02-21T23:05:50.352768+0000","last_unstale":"2024-02-21T23:05:52.379927+0000","last_undegraded":"2024-02-21T23:05:52.379927+0000","last_fullsized":"2024-02-21T23:05:52.379927+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:52:09.073053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.828739+0000","last_change":"2024-02-21T23:05:51.416424+0000","last_active":"2024-02-21T23:05:52.828739+0000","last_peered":"2024-02-21T23:05:52.828739+0000","last_clean":"2024-02-21T23:05:52.828739+0000","last_became_active":"2024-02-21T23:05:51.416070+0000","last_became_peered":"2024-02-21T23:05:51.416070+0000","last_unstale":"2024-02-21T23:05:52.828739+0000","last_undegraded":"2024-02-21T23:05:52.828739+0000","last_fullsized":"2024-02-21T23:05:52.828739+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:56:10.406772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"75'8","reported_seq":22,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.418474+0000","last_change":"2024-02-21T23:05:50.376063+0000","last_active":"2024-02-21T23:05:52.418474+0000","last_peered":"2024-02-21T23:05:52.418474+0000","last_clean":"2024-02-21T23:05:52.418474+0000","last_became_active":"2024-02-21T23:05:50.375772+0000","last_became_peered":"2024-02-21T23:05:50.375772+0000","last_unstale":"2024-02-21T23:05:52.418474+0000","last_undegraded":"2024-02-21T23:05:52.418474+0000","last_fullsized":"2024-02-21T23:05:52.418474+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:26:58.076002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.381136+0000","last_change":"2024-02-21T23:05:51.421565+0000","last_active":"2024-02-21T23:05:52.381136+0000","last_peered":"2024-02-21T23:05:52.381136+0000","last_clean":"2024-02-21T23:05:52.381136+0000","last_became_active":"2024-02-21T23:05:51.421252+0000","last_became_peered":"2024-02-21T23:05:51.421252+0000","last_unstale":"2024-02-21T23:05:52.381136+0000","last_undegraded":"2024-02-21T23:05:52.381136+0000","last_fullsized":"2024-02-21T23:05:52.381136+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:40:45.445418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"75'1","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.536140+0000","last_change":"2024-02-21T23:05:50.383778+0000","last_active":"2024-02-21T23:05:52.536140+0000","last_peered":"2024-02-21T23:05:52.536140+0000","last_clean":"2024-02-21T23:05:52.536140+0000","last_became_active":"2024-02-21T23:05:50.383436+0000","last_became_peered":"2024-02-21T23:05:50.383436+0000","last_unstale":"2024-02-21T23:05:52.536140+0000","last_undegraded":"2024-02-21T23:05:52.536140+0000","last_fullsized":"2024-02-21T23:05:52.536140+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:45:05.837402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.379870+0000","last_change":"2024-02-21T23:05:51.416676+0000","last_active":"2024-02-21T23:05:52.379870+0000","last_peered":"2024-02-21T23:05:52.379870+0000","last_clean":"2024-02-21T23:05:52.379870+0000","last_became_active":"2024-02-21T23:05:51.416297+0000","last_became_peered":"2024-02-21T23:05:51.416297+0000","last_unstale":"2024-02-21T23:05:52.379870+0000","last_undegraded":"2024-02-21T23:05:52.379870+0000","last_fullsized":"2024-02-21T23:05:52.379870+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:29:33.209545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"75'2","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.384553+0000","last_change":"2024-02-21T23:05:50.377181+0000","last_active":"2024-02-21T23:05:52.384553+0000","last_peered":"2024-02-21T23:05:52.384553+0000","last_clean":"2024-02-21T23:05:52.384553+0000","last_became_active":"2024-02-21T23:05:50.373524+0000","last_became_peered":"2024-02-21T23:05:50.373524+0000","last_unstale":"2024-02-21T23:05:52.384553+0000","last_undegraded":"2024-02-21T23:05:52.384553+0000","last_fullsized":"2024-02-21T23:05:52.384553+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:01:18.419728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.380361+0000","last_change":"2024-02-21T23:05:51.412971+0000","last_active":"2024-02-21T23:05:52.380361+0000","last_peered":"2024-02-21T23:05:52.380361+0000","last_clean":"2024-02-21T23:05:52.380361+0000","last_became_active":"2024-02-21T23:05:51.410966+0000","last_became_peered":"2024-02-21T23:05:51.410966+0000","last_unstale":"2024-02-21T23:05:52.380361+0000","last_undegraded":"2024-02-21T23:05:52.380361+0000","last_fullsized":"2024-02-21T23:05:52.380361+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:00:39.163092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"75'1","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.383103+0000","last_change":"2024-02-21T23:05:50.366307+0000","last_active":"2024-02-21T23:05:52.383103+0000","last_peered":"2024-02-21T23:05:52.383103+0000","last_clean":"2024-02-21T23:05:52.383103+0000","last_became_active":"2024-02-21T23:05:50.359426+0000","last_became_peered":"2024-02-21T23:05:50.359426+0000","last_unstale":"2024-02-21T23:05:52.383103+0000","last_undegraded":"2024-02-21T23:05:52.383103+0000","last_fullsized":"2024-02-21T23:05:52.383103+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:13:15.770526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.381306+0000","last_change":"2024-02-21T23:05:51.424067+0000","last_active":"2024-02-21T23:05:52.381306+0000","last_peered":"2024-02-21T23:05:52.381306+0000","last_clean":"2024-02-21T23:05:52.381306+0000","last_became_active":"2024-02-21T23:05:51.423644+0000","last_became_peered":"2024-02-21T23:05:51.423644+0000","last_unstale":"2024-02-21T23:05:52.381306+0000","last_undegraded":"2024-02-21T23:05:52.381306+0000","last_fullsized":"2024-02-21T23:05:52.381306+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:04:03.084512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.827402+0000","last_change":"2024-02-21T23:05:50.377924+0000","last_active":"2024-02-21T23:05:52.827402+0000","last_peered":"2024-02-21T23:05:52.827402+0000","last_clean":"2024-02-21T23:05:52.827402+0000","last_became_active":"2024-02-21T23:05:50.377440+0000","last_became_peered":"2024-02-21T23:05:50.377440+0000","last_unstale":"2024-02-21T23:05:52.827402+0000","last_undegraded":"2024-02-21T23:05:52.827402+0000","last_fullsized":"2024-02-21T23:05:52.827402+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:42:15.626763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.418107+0000","last_change":"2024-02-21T23:05:51.414619+0000","last_active":"2024-02-21T23:05:52.418107+0000","last_peered":"2024-02-21T23:05:52.418107+0000","last_clean":"2024-02-21T23:05:52.418107+0000","last_became_active":"2024-02-21T23:05:51.414059+0000","last_became_peered":"2024-02-21T23:05:51.414059+0000","last_unstale":"2024-02-21T23:05:52.418107+0000","last_undegraded":"2024-02-21T23:05:52.418107+0000","last_fullsized":"2024-02-21T23:05:52.418107+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:58:05.114297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"75'2","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.381450+0000","last_change":"2024-02-21T23:05:50.366877+0000","last_active":"2024-02-21T23:05:52.381450+0000","last_peered":"2024-02-21T23:05:52.381450+0000","last_clean":"2024-02-21T23:05:52.381450+0000","last_became_active":"2024-02-21T23:05:50.365566+0000","last_became_peered":"2024-02-21T23:05:50.365566+0000","last_unstale":"2024-02-21T23:05:52.381450+0000","last_undegraded":"2024-02-21T23:05:52.381450+0000","last_fullsized":"2024-02-21T23:05:52.381450+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:58:04.101475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.418441+0000","last_change":"2024-02-21T23:05:51.411005+0000","last_active":"2024-02-21T23:05:52.418441+0000","last_peered":"2024-02-21T23:05:52.418441+0000","last_clean":"2024-02-21T23:05:52.418441+0000","last_became_active":"2024-02-21T23:05:51.410503+0000","last_became_peered":"2024-02-21T23:05:51.410503+0000","last_unstale":"2024-02-21T23:05:52.418441+0000","last_undegraded":"2024-02-21T23:05:52.418441+0000","last_fullsized":"2024-02-21T23:05:52.418441+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:51:10.610852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.381097+0000","last_change":"2024-02-21T23:05:50.376888+0000","last_active":"2024-02-21T23:05:52.381097+0000","last_peered":"2024-02-21T23:05:52.381097+0000","last_clean":"2024-02-21T23:05:52.381097+0000","last_became_active":"2024-02-21T23:05:50.376610+0000","last_became_peered":"2024-02-21T23:05:50.376610+0000","last_unstale":"2024-02-21T23:05:52.381097+0000","last_undegraded":"2024-02-21T23:05:52.381097+0000","last_fullsized":"2024-02-21T23:05:52.381097+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:51:16.181060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.829202+0000","last_change":"2024-02-21T23:05:51.417167+0000","last_active":"2024-02-21T23:05:52.829202+0000","last_peered":"2024-02-21T23:05:52.829202+0000","last_clean":"2024-02-21T23:05:52.829202+0000","last_became_active":"2024-02-21T23:05:51.415957+0000","last_became_peered":"2024-02-21T23:05:51.415957+0000","last_unstale":"2024-02-21T23:05:52.829202+0000","last_undegraded":"2024-02-21T23:05:52.829202+0000","last_fullsized":"2024-02-21T23:05:52.829202+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:26:59.088824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.381605+0000","last_change":"2024-02-21T23:05:50.375808+0000","last_active":"2024-02-21T23:05:52.381605+0000","last_peered":"2024-02-21T23:05:52.381605+0000","last_clean":"2024-02-21T23:05:52.381605+0000","last_became_active":"2024-02-21T23:05:50.375467+0000","last_became_peered":"2024-02-21T23:05:50.375467+0000","last_unstale":"2024-02-21T23:05:52.381605+0000","last_undegraded":"2024-02-21T23:05:52.381605+0000","last_fullsized":"2024-02-21T23:05:52.381605+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:29:01.979463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.383305+0000","last_change":"2024-02-21T23:05:51.414453+0000","last_active":"2024-02-21T23:05:52.383305+0000","last_peered":"2024-02-21T23:05:52.383305+0000","last_clean":"2024-02-21T23:05:52.383305+0000","last_became_active":"2024-02-21T23:05:51.413499+0000","last_became_peered":"2024-02-21T23:05:51.413499+0000","last_unstale":"2024-02-21T23:05:52.383305+0000","last_undegraded":"2024-02-21T23:05:52.383305+0000","last_fullsized":"2024-02-21T23:05:52.383305+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:55:21.967274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"75'2","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.827140+0000","last_change":"2024-02-21T23:05:50.381948+0000","last_active":"2024-02-21T23:05:52.827140+0000","last_peered":"2024-02-21T23:05:52.827140+0000","last_clean":"2024-02-21T23:05:52.827140+0000","last_became_active":"2024-02-21T23:05:50.381452+0000","last_became_peered":"2024-02-21T23:05:50.381452+0000","last_unstale":"2024-02-21T23:05:52.827140+0000","last_undegraded":"2024-02-21T23:05:52.827140+0000","last_fullsized":"2024-02-21T23:05:52.827140+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:22:16.414686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.828536+0000","last_change":"2024-02-21T23:05:51.410619+0000","last_active":"2024-02-21T23:05:52.828536+0000","last_peered":"2024-02-21T23:05:52.828536+0000","last_clean":"2024-02-21T23:05:52.828536+0000","last_became_active":"2024-02-21T23:05:51.409937+0000","last_became_peered":"2024-02-21T23:05:51.409937+0000","last_unstale":"2024-02-21T23:05:52.828536+0000","last_undegraded":"2024-02-21T23:05:52.828536+0000","last_fullsized":"2024-02-21T23:05:52.828536+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:22:32.845173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"76'1","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.418529+0000","last_change":"2024-02-21T23:05:50.362370+0000","last_active":"2024-02-21T23:05:52.418529+0000","last_peered":"2024-02-21T23:05:52.418529+0000","last_clean":"2024-02-21T23:05:52.418529+0000","last_became_active":"2024-02-21T23:05:50.361848+0000","last_became_peered":"2024-02-21T23:05:50.361848+0000","last_unstale":"2024-02-21T23:05:52.418529+0000","last_undegraded":"2024-02-21T23:05:52.418529+0000","last_fullsized":"2024-02-21T23:05:52.418529+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:08:35.448428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.384236+0000","last_change":"2024-02-21T23:05:51.421357+0000","last_active":"2024-02-21T23:05:52.384236+0000","last_peered":"2024-02-21T23:05:52.384236+0000","last_clean":"2024-02-21T23:05:52.384236+0000","last_became_active":"2024-02-21T23:05:51.420734+0000","last_became_peered":"2024-02-21T23:05:51.420734+0000","last_unstale":"2024-02-21T23:05:52.384236+0000","last_undegraded":"2024-02-21T23:05:52.384236+0000","last_fullsized":"2024-02-21T23:05:52.384236+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:31:44.536584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.535981+0000","last_change":"2024-02-21T23:05:50.384680+0000","last_active":"2024-02-21T23:05:52.535981+0000","last_peered":"2024-02-21T23:05:52.535981+0000","last_clean":"2024-02-21T23:05:52.535981+0000","last_became_active":"2024-02-21T23:05:50.384208+0000","last_became_peered":"2024-02-21T23:05:50.384208+0000","last_unstale":"2024-02-21T23:05:52.535981+0000","last_undegraded":"2024-02-21T23:05:52.535981+0000","last_fullsized":"2024-02-21T23:05:52.535981+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:01:27.369089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.417801+0000","last_change":"2024-02-21T23:05:51.395859+0000","last_active":"2024-02-21T23:05:52.417801+0000","last_peered":"2024-02-21T23:05:52.417801+0000","last_clean":"2024-02-21T23:05:52.417801+0000","last_became_active":"2024-02-21T23:05:51.395012+0000","last_became_peered":"2024-02-21T23:05:51.395012+0000","last_unstale":"2024-02-21T23:05:52.417801+0000","last_undegraded":"2024-02-21T23:05:52.417801+0000","last_fullsized":"2024-02-21T23:05:52.417801+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:29:02.992284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.535249+0000","last_change":"2024-02-21T23:05:50.383920+0000","last_active":"2024-02-21T23:05:52.535249+0000","last_peered":"2024-02-21T23:05:52.535249+0000","last_clean":"2024-02-21T23:05:52.535249+0000","last_became_active":"2024-02-21T23:05:50.382488+0000","last_became_peered":"2024-02-21T23:05:50.382488+0000","last_unstale":"2024-02-21T23:05:52.535249+0000","last_undegraded":"2024-02-21T23:05:52.535249+0000","last_fullsized":"2024-02-21T23:05:52.535249+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:26:58.076002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.829134+0000","last_change":"2024-02-21T23:05:51.422495+0000","last_active":"2024-02-21T23:05:52.829134+0000","last_peered":"2024-02-21T23:05:52.829134+0000","last_clean":"2024-02-21T23:05:52.829134+0000","last_became_active":"2024-02-21T23:05:51.422196+0000","last_became_peered":"2024-02-21T23:05:51.422196+0000","last_unstale":"2024-02-21T23:05:52.829134+0000","last_undegraded":"2024-02-21T23:05:52.829134+0000","last_fullsized":"2024-02-21T23:05:52.829134+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:10:11.986389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.828484+0000","last_change":"2024-02-21T23:05:50.376027+0000","last_active":"2024-02-21T23:05:52.828484+0000","last_peered":"2024-02-21T23:05:52.828484+0000","last_clean":"2024-02-21T23:05:52.828484+0000","last_became_active":"2024-02-21T23:05:50.375170+0000","last_became_peered":"2024-02-21T23:05:50.375170+0000","last_unstale":"2024-02-21T23:05:52.828484+0000","last_undegraded":"2024-02-21T23:05:52.828484+0000","last_fullsized":"2024-02-21T23:05:52.828484+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:39:32.228635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.828497+0000","last_change":"2024-02-21T23:05:51.420500+0000","last_active":"2024-02-21T23:05:52.828497+0000","last_peered":"2024-02-21T23:05:52.828497+0000","last_clean":"2024-02-21T23:05:52.828497+0000","last_became_active":"2024-02-21T23:05:51.420103+0000","last_became_peered":"2024-02-21T23:05:51.420103+0000","last_unstale":"2024-02-21T23:05:52.828497+0000","last_undegraded":"2024-02-21T23:05:52.828497+0000","last_fullsized":"2024-02-21T23:05:52.828497+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:27:01.191337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.380442+0000","last_change":"2024-02-21T23:05:50.354805+0000","last_active":"2024-02-21T23:05:52.380442+0000","last_peered":"2024-02-21T23:05:52.380442+0000","last_clean":"2024-02-21T23:05:52.380442+0000","last_became_active":"2024-02-21T23:05:50.354525+0000","last_became_peered":"2024-02-21T23:05:50.354525+0000","last_unstale":"2024-02-21T23:05:52.380442+0000","last_undegraded":"2024-02-21T23:05:52.380442+0000","last_fullsized":"2024-02-21T23:05:52.380442+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:28:05.314552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.827781+0000","last_change":"2024-02-21T23:05:51.416548+0000","last_active":"2024-02-21T23:05:52.827781+0000","last_peered":"2024-02-21T23:05:52.827781+0000","last_clean":"2024-02-21T23:05:52.827781+0000","last_became_active":"2024-02-21T23:05:51.416222+0000","last_became_peered":"2024-02-21T23:05:51.416222+0000","last_unstale":"2024-02-21T23:05:52.827781+0000","last_undegraded":"2024-02-21T23:05:52.827781+0000","last_fullsized":"2024-02-21T23:05:52.827781+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:22:32.845173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.828804+0000","last_change":"2024-02-21T23:05:50.373554+0000","last_active":"2024-02-21T23:05:52.828804+0000","last_peered":"2024-02-21T23:05:52.828804+0000","last_clean":"2024-02-21T23:05:52.828804+0000","last_became_active":"2024-02-21T23:05:50.373022+0000","last_became_peered":"2024-02-21T23:05:50.373022+0000","last_unstale":"2024-02-21T23:05:52.828804+0000","last_undegraded":"2024-02-21T23:05:52.828804+0000","last_fullsized":"2024-02-21T23:05:52.828804+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:38:56.665723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.827906+0000","last_change":"2024-02-21T23:05:51.412683+0000","last_active":"2024-02-21T23:05:52.827906+0000","last_peered":"2024-02-21T23:05:52.827906+0000","last_clean":"2024-02-21T23:05:52.827906+0000","last_became_active":"2024-02-21T23:05:51.412389+0000","last_became_peered":"2024-02-21T23:05:51.412389+0000","last_unstale":"2024-02-21T23:05:52.827906+0000","last_undegraded":"2024-02-21T23:05:52.827906+0000","last_fullsized":"2024-02-21T23:05:52.827906+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:57:06.211024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.826797+0000","last_change":"2024-02-21T23:05:50.376356+0000","last_active":"2024-02-21T23:05:52.826797+0000","last_peered":"2024-02-21T23:05:52.826797+0000","last_clean":"2024-02-21T23:05:52.826797+0000","last_became_active":"2024-02-21T23:05:50.370467+0000","last_became_peered":"2024-02-21T23:05:50.370467+0000","last_unstale":"2024-02-21T23:05:52.826797+0000","last_undegraded":"2024-02-21T23:05:52.826797+0000","last_fullsized":"2024-02-21T23:05:52.826797+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:56:49.435540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.535891+0000","last_change":"2024-02-21T23:05:51.413008+0000","last_active":"2024-02-21T23:05:52.535891+0000","last_peered":"2024-02-21T23:05:52.535891+0000","last_clean":"2024-02-21T23:05:52.535891+0000","last_became_active":"2024-02-21T23:05:51.412714+0000","last_became_peered":"2024-02-21T23:05:51.412714+0000","last_unstale":"2024-02-21T23:05:52.535891+0000","last_undegraded":"2024-02-21T23:05:52.535891+0000","last_fullsized":"2024-02-21T23:05:52.535891+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:03:08.789066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.381834+0000","last_change":"2024-02-21T23:05:50.375581+0000","last_active":"2024-02-21T23:05:52.381834+0000","last_peered":"2024-02-21T23:05:52.381834+0000","last_clean":"2024-02-21T23:05:52.381834+0000","last_became_active":"2024-02-21T23:05:50.375282+0000","last_became_peered":"2024-02-21T23:05:50.375282+0000","last_unstale":"2024-02-21T23:05:52.381834+0000","last_undegraded":"2024-02-21T23:05:52.381834+0000","last_fullsized":"2024-02-21T23:05:52.381834+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:49:45.012428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":274,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.535353+0000","last_change":"2024-02-21T23:02:32.289335+0000","last_active":"2024-02-21T23:05:52.535353+0000","last_peered":"2024-02-21T23:05:52.535353+0000","last_clean":"2024-02-21T23:05:52.535353+0000","last_became_active":"2024-02-21T23:02:32.286913+0000","last_became_peered":"2024-02-21T23:02:32.286913+0000","last_unstale":"2024-02-21T23:05:52.535353+0000","last_undegraded":"2024-02-21T23:05:52.535353+0000","last_fullsized":"2024-02-21T23:05:52.535353+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_clean_scrub_stamp":"2024-02-21T23:02:31.172569+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:36:23.987374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574891,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32156,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339684,"statfs":{"total":96636764160,"available":96603836416,"internally_reserved":0,"allocated":999424,"data_stored":1033908,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574892,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32104,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339736,"statfs":{"total":96636764160,"available":96603889664,"internally_reserved":0,"allocated":942080,"data_stored":445130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574891,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27940,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343900,"statfs":{"total":96636764160,"available":96608153600,"internally_reserved":0,"allocated":933888,"data_stored":443552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574892,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32116,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339724,"statfs":{"total":96636764160,"available":96603877376,"internally_reserved":0,"allocated":950272,"data_stored":445234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574892,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32100,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339740,"statfs":{"total":96636764160,"available":96603893760,"internally_reserved":0,"allocated":933888,"data_stored":444100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574893,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32108,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339732,"statfs":{"total":96636764160,"available":96603885568,"internally_reserved":0,"allocated":942080,"data_stored":444260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574891,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32096,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339744,"statfs":{"total":96636764160,"available":96603897856,"internally_reserved":0,"allocated":937984,"data_stored":443630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574891,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32152,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339688,"statfs":{"total":96636764160,"available":96603840512,"internally_reserved":0,"allocated":995328,"data_stored":1033886,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-21T23:06:01.166 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-21T23:06:01.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.403+0000 7f684bc00640 1 Processor -- start 2024-02-21T23:06:01.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.403+0000 7f684bc00640 1 -- start start 2024-02-21T23:06:01.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.405+0000 7f684bc00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f68440c11f0 0x7f6844153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:01.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.405+0000 7f684bc00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f68440c15d0 0x7f684406dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:01.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.405+0000 7f684bc00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6844152e70 con 0x7f6844156350 2024-02-21T23:06:01.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.405+0000 7f684bc00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68440c1a60 con 0x7f68440c15d0 2024-02-21T23:06:01.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.405+0000 7f684bc00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68440a5ec0 con 0x7f68440c11f0 2024-02-21T23:06:01.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.405+0000 7f6843e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f68440c15d0 0x7f684406dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:01.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.405+0000 7f6843e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f68440c15d0 0x7f684406dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54182/0 (socket says 172.21.15.28:54182) 2024-02-21T23:06:01.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.405+0000 7f6843e00640 1 -- 172.21.15.28:0/2257425879 learned_addr learned my addr 172.21.15.28:0/2257425879 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:01.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.405+0000 7f6849200640 1 --2- 172.21.15.28:0/2257425879 >> v2:172.21.15.81:3300/0 conn(0x7f68440c11f0 0x7f6844153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:01.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.406+0000 7f6843e00640 1 -- 172.21.15.28:0/2257425879 >> v2:172.21.15.81:3300/0 conn(0x7f68440c11f0 msgr2=0x7f6844153950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:01.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.406+0000 7f6843e00640 1 --2- 172.21.15.28:0/2257425879 >> v2:172.21.15.81:3300/0 conn(0x7f68440c11f0 0x7f6844153950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:01.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.406+0000 7f6843e00640 1 -- 172.21.15.28:0/2257425879 >> v1:172.21.15.81:6789/0 conn(0x7f6844156350 legacy=0x7f6844156760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:01.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.406+0000 7f6843e00640 1 -- 172.21.15.28:0/2257425879 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6844153150 con 0x7f68440c15d0 2024-02-21T23:06:01.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.407+0000 7f6843e00640 1 --2- 172.21.15.28:0/2257425879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68440c15d0 0x7f684406dce0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f68440c7590 tx=0x7f683002b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f216713ff67446f5 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:01.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.408+0000 7f6843400640 1 -- 172.21.15.28:0/2257425879 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6830038070 con 0x7f68440c15d0 2024-02-21T23:06:01.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.408+0000 7f6843400640 1 -- 172.21.15.28:0/2257425879 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f683002f050 con 0x7f68440c15d0 2024-02-21T23:06:01.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.408+0000 7f6843400640 1 -- 172.21.15.28:0/2257425879 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6830031970 con 0x7f68440c15d0 2024-02-21T23:06:01.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.408+0000 7f684bc00640 1 -- 172.21.15.28:0/2257425879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68440c15d0 msgr2=0x7f684406dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:01.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.408+0000 7f684bc00640 1 --2- 172.21.15.28:0/2257425879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68440c15d0 0x7f684406dce0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f68440c7590 tx=0x7f683002b7f0 comp rx=0 tx=0).stop 2024-02-21T23:06:01.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.409+0000 7f684bc00640 1 -- 172.21.15.28:0/2257425879 shutdown_connections 2024-02-21T23:06:01.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.409+0000 7f684bc00640 1 --2- 172.21.15.28:0/2257425879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68440c15d0 0x7f684406dce0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:01.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.409+0000 7f684bc00640 1 --2- 172.21.15.28:0/2257425879 >> v2:172.21.15.81:3300/0 conn(0x7f68440c11f0 0x7f6844153950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:01.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.409+0000 7f684bc00640 1 -- 172.21.15.28:0/2257425879 >> 172.21.15.28:0/2257425879 conn(0x7f68440c67f0 msgr2=0x7f68440c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:01.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.409+0000 7f684bc00640 1 -- 172.21.15.28:0/2257425879 shutdown_connections 2024-02-21T23:06:01.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.409+0000 7f684bc00640 1 -- 172.21.15.28:0/2257425879 wait complete. 2024-02-21T23:06:01.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.411+0000 7f684bc00640 1 Processor -- start 2024-02-21T23:06:01.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.412+0000 7f684bc00640 1 -- start start 2024-02-21T23:06:01.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.413+0000 7f684bc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68440c11f0 0x7f68440bfe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:01.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.413+0000 7f684bc00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f68440c15d0 0x7f6844143520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:01.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.413+0000 7f684bc00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6844156350 0x7f6844143a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:01.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.413+0000 7f684bc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68440c4c00 con 0x7f68440c11f0 2024-02-21T23:06:01.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.413+0000 7f684bc00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f684414d1c0 con 0x7f68440c15d0 2024-02-21T23:06:01.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.413+0000 7f684bc00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68440a6110 con 0x7f6844156350 2024-02-21T23:06:01.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.414+0000 7f6849200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68440c11f0 0x7f68440bfe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:01.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.414+0000 7f6849200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68440c11f0 0x7f68440bfe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54198/0 (socket says 172.21.15.28:54198) 2024-02-21T23:06:01.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.414+0000 7f6849200640 1 -- 172.21.15.28:0/2837991017 learned_addr learned my addr 172.21.15.28:0/2837991017 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:01.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.414+0000 7f6843e00640 1 --2- 172.21.15.28:0/2837991017 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f68440c15d0 0x7f6844143520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:01.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.414+0000 7f6849c00640 1 --2- 172.21.15.28:0/2837991017 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6844156350 0x7f6844143a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:01.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.414+0000 7f6849200640 1 -- 172.21.15.28:0/2837991017 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6844156350 msgr2=0x7f6844143a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:01.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.414+0000 7f6849200640 1 --2- 172.21.15.28:0/2837991017 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6844156350 0x7f6844143a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:01.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.414+0000 7f6849200640 1 -- 172.21.15.28:0/2837991017 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f68440c15d0 msgr2=0x7f6844143520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:01.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.414+0000 7f6849200640 1 --2- 172.21.15.28:0/2837991017 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f68440c15d0 0x7f6844143520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:01.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.414+0000 7f6849200640 1 -- 172.21.15.28:0/2837991017 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68441ef550 con 0x7f68440c11f0 2024-02-21T23:06:01.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.414+0000 7f6843e00640 1 --2- 172.21.15.28:0/2837991017 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f68440c15d0 0x7f6844143520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:06:01.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.415+0000 7f6849200640 1 --2- 172.21.15.28:0/2837991017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68440c11f0 0x7f68440bfe90 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f683800d620 tx=0x7f683800dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:01.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.415+0000 7f6840c00640 1 -- 172.21.15.28:0/2837991017 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f683800b620 con 0x7f68440c11f0 2024-02-21T23:06:01.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.415+0000 7f6840c00640 1 -- 172.21.15.28:0/2837991017 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6838019070 con 0x7f68440c11f0 2024-02-21T23:06:01.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.415+0000 7f684bc00640 1 -- 172.21.15.28:0/2837991017 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6844073350 con 0x7f68440c11f0 2024-02-21T23:06:01.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.415+0000 7f6840c00640 1 -- 172.21.15.28:0/2837991017 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f683800ba00 con 0x7f68440c11f0 2024-02-21T23:06:01.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.415+0000 7f684bc00640 1 -- 172.21.15.28:0/2837991017 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68441eea70 con 0x7f68440c11f0 2024-02-21T23:06:01.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.416+0000 7f684bc00640 1 -- 172.21.15.28:0/2837991017 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6844067f30 con 0x7f68440c11f0 2024-02-21T23:06:01.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.418+0000 7f6840c00640 1 -- 172.21.15.28:0/2837991017 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f683800bba0 con 0x7f68440c11f0 2024-02-21T23:06:01.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.419+0000 7f6840c00640 1 --2- 172.21.15.28:0/2837991017 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f682406b4d0 0x7f682406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:01.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.420+0000 7f6840c00640 1 -- 172.21.15.28:0/2837991017 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f683808a7e0 con 0x7f68440c11f0 2024-02-21T23:06:01.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.420+0000 7f6843e00640 1 --2- 172.21.15.28:0/2837991017 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f682406b4d0 0x7f682406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:01.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.422+0000 7f6843e00640 1 --2- 172.21.15.28:0/2837991017 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f682406b4d0 0x7f682406d990 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f6830035000 tx=0x7f6830008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:01.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.424+0000 7f6840c00640 1 -- 172.21.15.28:0/2837991017 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f68380594c0 con 0x7f68440c11f0 2024-02-21T23:06:01.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.621+0000 7f684bc00640 1 -- 172.21.15.28:0/2837991017 --> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f68440c0960 con 0x7f682406b4d0 2024-02-21T23:06:01.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.639+0000 7f6840c00640 1 -- 172.21.15.28:0/2837991017 <== mgr.4101 v2:172.21.15.81:6832/1848538577 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252150 (secure 0 0 0) 0x7f68440c0960 con 0x7f682406b4d0 2024-02-21T23:06:01.632 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:06:01.633 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-21T23:06:01.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.646+0000 7f684bc00640 1 -- 172.21.15.28:0/2837991017 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f682406b4d0 msgr2=0x7f682406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:01.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.646+0000 7f684bc00640 1 --2- 172.21.15.28:0/2837991017 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f682406b4d0 0x7f682406d990 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f6830035000 tx=0x7f6830008030 comp rx=0 tx=0).stop 2024-02-21T23:06:01.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.646+0000 7f684bc00640 1 -- 172.21.15.28:0/2837991017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68440c11f0 msgr2=0x7f68440bfe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:01.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.646+0000 7f684bc00640 1 --2- 172.21.15.28:0/2837991017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68440c11f0 0x7f68440bfe90 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f683800d620 tx=0x7f683800dae0 comp rx=0 tx=0).stop 2024-02-21T23:06:01.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.646+0000 7f684bc00640 1 -- 172.21.15.28:0/2837991017 shutdown_connections 2024-02-21T23:06:01.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.646+0000 7f684bc00640 1 --2- 172.21.15.28:0/2837991017 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f682406b4d0 0x7f682406d990 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:01.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.646+0000 7f684bc00640 1 --2- 172.21.15.28:0/2837991017 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6844156350 0x7f6844143a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:01.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.646+0000 7f684bc00640 1 --2- 172.21.15.28:0/2837991017 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f68440c15d0 0x7f6844143520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:01.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.646+0000 7f684bc00640 1 --2- 172.21.15.28:0/2837991017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68440c11f0 0x7f68440bfe90 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:01.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.646+0000 7f684bc00640 1 -- 172.21.15.28:0/2837991017 >> 172.21.15.28:0/2837991017 conn(0x7f68440c67f0 msgr2=0x7f68440c5ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:01.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.646+0000 7f684bc00640 1 -- 172.21.15.28:0/2837991017 shutdown_connections 2024-02-21T23:06:01.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.647+0000 7f684bc00640 1 -- 172.21.15.28:0/2837991017 wait complete. 2024-02-21T23:06:01.656 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":174,"stamp":"2024-02-21T23:06:00.736652+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":252772,"kb_used_data":7456,"kb_used_omap":12,"kb_used_meta":245171,"kb_avail":754721948,"statfs":{"total":773094113280,"available":772835274752,"internally_reserved":0,"allocated":7634944,"data_stored":4733700,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":251055665},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":43,"apply_latency_ms":43,"commit_latency_ns":43000000,"apply_latency_ns":43000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.361431"},"pg_stats":[{"pgid":"12.3f","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.535895+0000","last_change":"2024-02-21T23:05:51.408831+0000","last_active":"2024-02-21T23:05:52.535895+0000","last_peered":"2024-02-21T23:05:52.535895+0000","last_clean":"2024-02-21T23:05:52.535895+0000","last_became_active":"2024-02-21T23:05:51.408353+0000","last_became_peered":"2024-02-21T23:05:51.408353+0000","last_unstale":"2024-02-21T23:05:52.535895+0000","last_undegraded":"2024-02-21T23:05:52.535895+0000","last_fullsized":"2024-02-21T23:05:52.535895+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:36:47.962935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.418527+0000","last_change":"2024-02-21T23:05:51.420557+0000","last_active":"2024-02-21T23:05:52.418527+0000","last_peered":"2024-02-21T23:05:52.418527+0000","last_clean":"2024-02-21T23:05:52.418527+0000","last_became_active":"2024-02-21T23:05:51.420078+0000","last_became_peered":"2024-02-21T23:05:51.420078+0000","last_unstale":"2024-02-21T23:05:52.418527+0000","last_undegraded":"2024-02-21T23:05:52.418527+0000","last_fullsized":"2024-02-21T23:05:52.418527+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:36:23.792395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.827683+0000","last_change":"2024-02-21T23:05:51.408934+0000","last_active":"2024-02-21T23:05:52.827683+0000","last_peered":"2024-02-21T23:05:52.827683+0000","last_clean":"2024-02-21T23:05:52.827683+0000","last_became_active":"2024-02-21T23:05:51.408597+0000","last_became_peered":"2024-02-21T23:05:51.408597+0000","last_unstale":"2024-02-21T23:05:52.827683+0000","last_undegraded":"2024-02-21T23:05:52.827683+0000","last_fullsized":"2024-02-21T23:05:52.827683+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:58:55.829941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.827692+0000","last_change":"2024-02-21T23:05:51.423479+0000","last_active":"2024-02-21T23:05:52.827692+0000","last_peered":"2024-02-21T23:05:52.827692+0000","last_clean":"2024-02-21T23:05:52.827692+0000","last_became_active":"2024-02-21T23:05:51.423108+0000","last_became_peered":"2024-02-21T23:05:51.423108+0000","last_unstale":"2024-02-21T23:05:52.827692+0000","last_undegraded":"2024-02-21T23:05:52.827692+0000","last_fullsized":"2024-02-21T23:05:52.827692+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:36:47.962935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.380595+0000","last_change":"2024-02-21T23:05:51.419436+0000","last_active":"2024-02-21T23:05:52.380595+0000","last_peered":"2024-02-21T23:05:52.380595+0000","last_clean":"2024-02-21T23:05:52.380595+0000","last_became_active":"2024-02-21T23:05:51.418562+0000","last_became_peered":"2024-02-21T23:05:51.418562+0000","last_unstale":"2024-02-21T23:05:52.380595+0000","last_undegraded":"2024-02-21T23:05:52.380595+0000","last_fullsized":"2024-02-21T23:05:52.380595+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:19:45.189126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.381201+0000","last_change":"2024-02-21T23:05:51.410014+0000","last_active":"2024-02-21T23:05:52.381201+0000","last_peered":"2024-02-21T23:05:52.381201+0000","last_clean":"2024-02-21T23:05:52.381201+0000","last_became_active":"2024-02-21T23:05:51.409637+0000","last_became_peered":"2024-02-21T23:05:51.409637+0000","last_unstale":"2024-02-21T23:05:52.381201+0000","last_undegraded":"2024-02-21T23:05:52.381201+0000","last_fullsized":"2024-02-21T23:05:52.381201+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:03:35.243949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.828504+0000","last_change":"2024-02-21T23:05:51.419505+0000","last_active":"2024-02-21T23:05:52.828504+0000","last_peered":"2024-02-21T23:05:52.828504+0000","last_clean":"2024-02-21T23:05:52.828504+0000","last_became_active":"2024-02-21T23:05:51.418040+0000","last_became_peered":"2024-02-21T23:05:51.418040+0000","last_unstale":"2024-02-21T23:05:52.828504+0000","last_undegraded":"2024-02-21T23:05:52.828504+0000","last_fullsized":"2024-02-21T23:05:52.828504+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:57:06.211024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.379325+0000","last_change":"2024-02-21T23:05:51.416524+0000","last_active":"2024-02-21T23:05:52.379325+0000","last_peered":"2024-02-21T23:05:52.379325+0000","last_clean":"2024-02-21T23:05:52.379325+0000","last_became_active":"2024-02-21T23:05:51.415391+0000","last_became_peered":"2024-02-21T23:05:51.415391+0000","last_unstale":"2024-02-21T23:05:52.379325+0000","last_undegraded":"2024-02-21T23:05:52.379325+0000","last_fullsized":"2024-02-21T23:05:52.379325+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:48:32.958103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.383972+0000","last_change":"2024-02-21T23:05:51.421068+0000","last_active":"2024-02-21T23:05:52.383972+0000","last_peered":"2024-02-21T23:05:52.383972+0000","last_clean":"2024-02-21T23:05:52.383972+0000","last_became_active":"2024-02-21T23:05:51.419746+0000","last_became_peered":"2024-02-21T23:05:51.419746+0000","last_unstale":"2024-02-21T23:05:52.383972+0000","last_undegraded":"2024-02-21T23:05:52.383972+0000","last_fullsized":"2024-02-21T23:05:52.383972+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:09:42.444733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.827198+0000","last_change":"2024-02-21T23:05:51.418295+0000","last_active":"2024-02-21T23:05:52.827198+0000","last_peered":"2024-02-21T23:05:52.827198+0000","last_clean":"2024-02-21T23:05:52.827198+0000","last_became_active":"2024-02-21T23:05:51.417780+0000","last_became_peered":"2024-02-21T23:05:51.417780+0000","last_unstale":"2024-02-21T23:05:52.827198+0000","last_undegraded":"2024-02-21T23:05:52.827198+0000","last_fullsized":"2024-02-21T23:05:52.827198+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:19:57.210565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.418290+0000","last_change":"2024-02-21T23:05:51.420518+0000","last_active":"2024-02-21T23:05:52.418290+0000","last_peered":"2024-02-21T23:05:52.418290+0000","last_clean":"2024-02-21T23:05:52.418290+0000","last_became_active":"2024-02-21T23:05:51.419564+0000","last_became_peered":"2024-02-21T23:05:51.419564+0000","last_unstale":"2024-02-21T23:05:52.418290+0000","last_undegraded":"2024-02-21T23:05:52.418290+0000","last_fullsized":"2024-02-21T23:05:52.418290+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:19:41.280587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.827207+0000","last_change":"2024-02-21T23:05:51.421691+0000","last_active":"2024-02-21T23:05:52.827207+0000","last_peered":"2024-02-21T23:05:52.827207+0000","last_clean":"2024-02-21T23:05:52.827207+0000","last_became_active":"2024-02-21T23:05:51.417621+0000","last_became_peered":"2024-02-21T23:05:51.417621+0000","last_unstale":"2024-02-21T23:05:52.827207+0000","last_undegraded":"2024-02-21T23:05:52.827207+0000","last_fullsized":"2024-02-21T23:05:52.827207+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:46:01.608149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.381774+0000","last_change":"2024-02-21T23:05:51.426871+0000","last_active":"2024-02-21T23:05:52.381774+0000","last_peered":"2024-02-21T23:05:52.381774+0000","last_clean":"2024-02-21T23:05:52.381774+0000","last_became_active":"2024-02-21T23:05:51.419412+0000","last_became_peered":"2024-02-21T23:05:51.419412+0000","last_unstale":"2024-02-21T23:05:52.381774+0000","last_undegraded":"2024-02-21T23:05:52.381774+0000","last_fullsized":"2024-02-21T23:05:52.381774+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:22:09.156621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.381402+0000","last_change":"2024-02-21T23:05:51.417593+0000","last_active":"2024-02-21T23:05:52.381402+0000","last_peered":"2024-02-21T23:05:52.381402+0000","last_clean":"2024-02-21T23:05:52.381402+0000","last_became_active":"2024-02-21T23:05:51.416945+0000","last_became_peered":"2024-02-21T23:05:51.416945+0000","last_unstale":"2024-02-21T23:05:52.381402+0000","last_undegraded":"2024-02-21T23:05:52.381402+0000","last_fullsized":"2024-02-21T23:05:52.381402+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:24:32.863072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.417444+0000","last_change":"2024-02-21T23:05:51.409645+0000","last_active":"2024-02-21T23:05:52.417444+0000","last_peered":"2024-02-21T23:05:52.417444+0000","last_clean":"2024-02-21T23:05:52.417444+0000","last_became_active":"2024-02-21T23:05:51.409356+0000","last_became_peered":"2024-02-21T23:05:51.409356+0000","last_unstale":"2024-02-21T23:05:52.417444+0000","last_undegraded":"2024-02-21T23:05:52.417444+0000","last_fullsized":"2024-02-21T23:05:52.417444+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:17:26.329609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.535086+0000","last_change":"2024-02-21T23:05:51.413656+0000","last_active":"2024-02-21T23:05:52.535086+0000","last_peered":"2024-02-21T23:05:52.535086+0000","last_clean":"2024-02-21T23:05:52.535086+0000","last_became_active":"2024-02-21T23:05:51.413350+0000","last_became_peered":"2024-02-21T23:05:51.413350+0000","last_unstale":"2024-02-21T23:05:52.535086+0000","last_undegraded":"2024-02-21T23:05:52.535086+0000","last_fullsized":"2024-02-21T23:05:52.535086+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:19:57.210565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.383335+0000","last_change":"2024-02-21T23:05:51.427044+0000","last_active":"2024-02-21T23:05:52.383335+0000","last_peered":"2024-02-21T23:05:52.383335+0000","last_clean":"2024-02-21T23:05:52.383335+0000","last_became_active":"2024-02-21T23:05:51.419477+0000","last_became_peered":"2024-02-21T23:05:51.419477+0000","last_unstale":"2024-02-21T23:05:52.383335+0000","last_undegraded":"2024-02-21T23:05:52.383335+0000","last_fullsized":"2024-02-21T23:05:52.383335+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:04:00.714448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.827738+0000","last_change":"2024-02-21T23:05:51.425126+0000","last_active":"2024-02-21T23:05:52.827738+0000","last_peered":"2024-02-21T23:05:52.827738+0000","last_clean":"2024-02-21T23:05:52.827738+0000","last_became_active":"2024-02-21T23:05:51.422511+0000","last_became_peered":"2024-02-21T23:05:51.422511+0000","last_unstale":"2024-02-21T23:05:52.827738+0000","last_undegraded":"2024-02-21T23:05:52.827738+0000","last_fullsized":"2024-02-21T23:05:52.827738+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:27:01.191337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.382064+0000","last_change":"2024-02-21T23:05:51.423258+0000","last_active":"2024-02-21T23:05:52.382064+0000","last_peered":"2024-02-21T23:05:52.382064+0000","last_clean":"2024-02-21T23:05:52.382064+0000","last_became_active":"2024-02-21T23:05:51.422987+0000","last_became_peered":"2024-02-21T23:05:51.422987+0000","last_unstale":"2024-02-21T23:05:52.382064+0000","last_undegraded":"2024-02-21T23:05:52.382064+0000","last_fullsized":"2024-02-21T23:05:52.382064+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:41:31.260842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.382602+0000","last_change":"2024-02-21T23:05:51.427468+0000","last_active":"2024-02-21T23:05:52.382602+0000","last_peered":"2024-02-21T23:05:52.382602+0000","last_clean":"2024-02-21T23:05:52.382602+0000","last_became_active":"2024-02-21T23:05:51.425476+0000","last_became_peered":"2024-02-21T23:05:51.425476+0000","last_unstale":"2024-02-21T23:05:52.382602+0000","last_undegraded":"2024-02-21T23:05:52.382602+0000","last_fullsized":"2024-02-21T23:05:52.382602+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:06:18.158392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.384633+0000","last_change":"2024-02-21T23:05:51.415772+0000","last_active":"2024-02-21T23:05:52.384633+0000","last_peered":"2024-02-21T23:05:52.384633+0000","last_clean":"2024-02-21T23:05:52.384633+0000","last_became_active":"2024-02-21T23:05:51.415494+0000","last_became_peered":"2024-02-21T23:05:51.415494+0000","last_unstale":"2024-02-21T23:05:52.384633+0000","last_undegraded":"2024-02-21T23:05:52.384633+0000","last_fullsized":"2024-02-21T23:05:52.384633+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:42:37.006685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.380314+0000","last_change":"2024-02-21T23:05:51.428557+0000","last_active":"2024-02-21T23:05:52.380314+0000","last_peered":"2024-02-21T23:05:52.380314+0000","last_clean":"2024-02-21T23:05:52.380314+0000","last_became_active":"2024-02-21T23:05:51.427612+0000","last_became_peered":"2024-02-21T23:05:51.427612+0000","last_unstale":"2024-02-21T23:05:52.380314+0000","last_undegraded":"2024-02-21T23:05:52.380314+0000","last_fullsized":"2024-02-21T23:05:52.380314+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:38:44.895318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.383489+0000","last_change":"2024-02-21T23:05:51.417302+0000","last_active":"2024-02-21T23:05:52.383489+0000","last_peered":"2024-02-21T23:05:52.383489+0000","last_clean":"2024-02-21T23:05:52.383489+0000","last_became_active":"2024-02-21T23:05:51.414092+0000","last_became_peered":"2024-02-21T23:05:51.414092+0000","last_unstale":"2024-02-21T23:05:52.383489+0000","last_undegraded":"2024-02-21T23:05:52.383489+0000","last_fullsized":"2024-02-21T23:05:52.383489+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:49:01.911867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.417508+0000","last_change":"2024-02-21T23:05:51.406411+0000","last_active":"2024-02-21T23:05:52.417508+0000","last_peered":"2024-02-21T23:05:52.417508+0000","last_clean":"2024-02-21T23:05:52.417508+0000","last_became_active":"2024-02-21T23:05:51.406104+0000","last_became_peered":"2024-02-21T23:05:51.406104+0000","last_unstale":"2024-02-21T23:05:52.417508+0000","last_undegraded":"2024-02-21T23:05:52.417508+0000","last_fullsized":"2024-02-21T23:05:52.417508+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:03:08.789066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.418700+0000","last_change":"2024-02-21T23:05:51.414112+0000","last_active":"2024-02-21T23:05:52.418700+0000","last_peered":"2024-02-21T23:05:52.418700+0000","last_clean":"2024-02-21T23:05:52.418700+0000","last_became_active":"2024-02-21T23:05:51.413796+0000","last_became_peered":"2024-02-21T23:05:51.413796+0000","last_unstale":"2024-02-21T23:05:52.418700+0000","last_undegraded":"2024-02-21T23:05:52.418700+0000","last_fullsized":"2024-02-21T23:05:52.418700+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:42:26.488510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.827519+0000","last_change":"2024-02-21T23:05:51.421208+0000","last_active":"2024-02-21T23:05:52.827519+0000","last_peered":"2024-02-21T23:05:52.827519+0000","last_clean":"2024-02-21T23:05:52.827519+0000","last_became_active":"2024-02-21T23:05:51.418171+0000","last_became_peered":"2024-02-21T23:05:51.418171+0000","last_unstale":"2024-02-21T23:05:52.827519+0000","last_undegraded":"2024-02-21T23:05:52.827519+0000","last_fullsized":"2024-02-21T23:05:52.827519+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:17:26.329609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.535471+0000","last_change":"2024-02-21T23:05:51.416857+0000","last_active":"2024-02-21T23:05:52.535471+0000","last_peered":"2024-02-21T23:05:52.535471+0000","last_clean":"2024-02-21T23:05:52.535471+0000","last_became_active":"2024-02-21T23:05:51.416577+0000","last_became_peered":"2024-02-21T23:05:51.416577+0000","last_unstale":"2024-02-21T23:05:52.535471+0000","last_undegraded":"2024-02-21T23:05:52.535471+0000","last_fullsized":"2024-02-21T23:05:52.535471+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:17:26.329609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.382662+0000","last_change":"2024-02-21T23:05:51.427489+0000","last_active":"2024-02-21T23:05:52.382662+0000","last_peered":"2024-02-21T23:05:52.382662+0000","last_clean":"2024-02-21T23:05:52.382662+0000","last_became_active":"2024-02-21T23:05:51.419458+0000","last_became_peered":"2024-02-21T23:05:51.419458+0000","last_unstale":"2024-02-21T23:05:52.382662+0000","last_undegraded":"2024-02-21T23:05:52.382662+0000","last_fullsized":"2024-02-21T23:05:52.382662+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:54:08.758447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.417641+0000","last_change":"2024-02-21T23:05:51.405741+0000","last_active":"2024-02-21T23:05:52.417641+0000","last_peered":"2024-02-21T23:05:52.417641+0000","last_clean":"2024-02-21T23:05:52.417641+0000","last_became_active":"2024-02-21T23:05:51.405301+0000","last_became_peered":"2024-02-21T23:05:51.405301+0000","last_unstale":"2024-02-21T23:05:52.417641+0000","last_undegraded":"2024-02-21T23:05:52.417641+0000","last_fullsized":"2024-02-21T23:05:52.417641+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:36:47.962935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.535038+0000","last_change":"2024-02-21T23:05:51.398663+0000","last_active":"2024-02-21T23:05:52.535038+0000","last_peered":"2024-02-21T23:05:52.535038+0000","last_clean":"2024-02-21T23:05:52.535038+0000","last_became_active":"2024-02-21T23:05:51.398146+0000","last_became_peered":"2024-02-21T23:05:51.398146+0000","last_unstale":"2024-02-21T23:05:52.535038+0000","last_undegraded":"2024-02-21T23:05:52.535038+0000","last_fullsized":"2024-02-21T23:05:52.535038+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:27:54.695721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.827090+0000","last_change":"2024-02-21T23:05:51.421330+0000","last_active":"2024-02-21T23:05:52.827090+0000","last_peered":"2024-02-21T23:05:52.827090+0000","last_clean":"2024-02-21T23:05:52.827090+0000","last_became_active":"2024-02-21T23:05:51.421012+0000","last_became_peered":"2024-02-21T23:05:51.421012+0000","last_unstale":"2024-02-21T23:05:52.827090+0000","last_undegraded":"2024-02-21T23:05:52.827090+0000","last_fullsized":"2024-02-21T23:05:52.827090+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:26:33.366331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.535369+0000","last_change":"2024-02-21T23:05:51.403173+0000","last_active":"2024-02-21T23:05:52.535369+0000","last_peered":"2024-02-21T23:05:52.535369+0000","last_clean":"2024-02-21T23:05:52.535369+0000","last_became_active":"2024-02-21T23:05:51.402610+0000","last_became_peered":"2024-02-21T23:05:51.402610+0000","last_unstale":"2024-02-21T23:05:52.535369+0000","last_undegraded":"2024-02-21T23:05:52.535369+0000","last_fullsized":"2024-02-21T23:05:52.535369+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:46:01.608149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.826863+0000","last_change":"2024-02-21T23:05:51.416534+0000","last_active":"2024-02-21T23:05:52.826863+0000","last_peered":"2024-02-21T23:05:52.826863+0000","last_clean":"2024-02-21T23:05:52.826863+0000","last_became_active":"2024-02-21T23:05:51.416244+0000","last_became_peered":"2024-02-21T23:05:51.416244+0000","last_unstale":"2024-02-21T23:05:52.826863+0000","last_undegraded":"2024-02-21T23:05:52.826863+0000","last_fullsized":"2024-02-21T23:05:52.826863+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:42:37.006685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"75'4","reported_seq":18,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.383405+0000","last_change":"2024-02-21T23:05:50.354109+0000","last_active":"2024-02-21T23:05:52.383405+0000","last_peered":"2024-02-21T23:05:52.383405+0000","last_clean":"2024-02-21T23:05:52.383405+0000","last_became_active":"2024-02-21T23:05:50.353809+0000","last_became_peered":"2024-02-21T23:05:50.353809+0000","last_unstale":"2024-02-21T23:05:52.383405+0000","last_undegraded":"2024-02-21T23:05:52.383405+0000","last_fullsized":"2024-02-21T23:05:52.383405+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:28:13.765557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.417929+0000","last_change":"2024-02-21T23:05:51.416268+0000","last_active":"2024-02-21T23:05:52.417929+0000","last_peered":"2024-02-21T23:05:52.417929+0000","last_clean":"2024-02-21T23:05:52.417929+0000","last_became_active":"2024-02-21T23:05:51.415980+0000","last_became_peered":"2024-02-21T23:05:51.415980+0000","last_unstale":"2024-02-21T23:05:52.417929+0000","last_undegraded":"2024-02-21T23:05:52.417929+0000","last_fullsized":"2024-02-21T23:05:52.417929+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:51:17.193881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"75'1","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.535409+0000","last_change":"2024-02-21T23:05:50.383438+0000","last_active":"2024-02-21T23:05:52.535409+0000","last_peered":"2024-02-21T23:05:52.535409+0000","last_clean":"2024-02-21T23:05:52.535409+0000","last_became_active":"2024-02-21T23:05:50.383134+0000","last_became_peered":"2024-02-21T23:05:50.383134+0000","last_unstale":"2024-02-21T23:05:52.535409+0000","last_undegraded":"2024-02-21T23:05:52.535409+0000","last_fullsized":"2024-02-21T23:05:52.535409+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:08:35.448428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.383786+0000","last_change":"2024-02-21T23:05:51.421571+0000","last_active":"2024-02-21T23:05:52.383786+0000","last_peered":"2024-02-21T23:05:52.383786+0000","last_clean":"2024-02-21T23:05:52.383786+0000","last_became_active":"2024-02-21T23:05:51.421110+0000","last_became_peered":"2024-02-21T23:05:51.421110+0000","last_unstale":"2024-02-21T23:05:52.383786+0000","last_undegraded":"2024-02-21T23:05:52.383786+0000","last_fullsized":"2024-02-21T23:05:52.383786+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:56:36.688152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"75'2","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.535656+0000","last_change":"2024-02-21T23:05:50.383231+0000","last_active":"2024-02-21T23:05:52.535656+0000","last_peered":"2024-02-21T23:05:52.535656+0000","last_clean":"2024-02-21T23:05:52.535656+0000","last_became_active":"2024-02-21T23:05:50.382757+0000","last_became_peered":"2024-02-21T23:05:50.382757+0000","last_unstale":"2024-02-21T23:05:52.535656+0000","last_undegraded":"2024-02-21T23:05:52.535656+0000","last_fullsized":"2024-02-21T23:05:52.535656+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:49:36.738197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.384761+0000","last_change":"2024-02-21T23:05:51.415892+0000","last_active":"2024-02-21T23:05:52.384761+0000","last_peered":"2024-02-21T23:05:52.384761+0000","last_clean":"2024-02-21T23:05:52.384761+0000","last_became_active":"2024-02-21T23:05:51.415520+0000","last_became_peered":"2024-02-21T23:05:51.415520+0000","last_unstale":"2024-02-21T23:05:52.384761+0000","last_undegraded":"2024-02-21T23:05:52.384761+0000","last_fullsized":"2024-02-21T23:05:52.384761+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:26:33.366331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"75'2","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.829069+0000","last_change":"2024-02-21T23:05:50.376050+0000","last_active":"2024-02-21T23:05:52.829069+0000","last_peered":"2024-02-21T23:05:52.829069+0000","last_clean":"2024-02-21T23:05:52.829069+0000","last_became_active":"2024-02-21T23:05:50.375731+0000","last_became_peered":"2024-02-21T23:05:50.375731+0000","last_unstale":"2024-02-21T23:05:52.829069+0000","last_undegraded":"2024-02-21T23:05:52.829069+0000","last_fullsized":"2024-02-21T23:05:52.829069+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:42:15.626763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.418119+0000","last_change":"2024-02-21T23:05:51.428435+0000","last_active":"2024-02-21T23:05:52.418119+0000","last_peered":"2024-02-21T23:05:52.418119+0000","last_clean":"2024-02-21T23:05:52.418119+0000","last_became_active":"2024-02-21T23:05:51.428133+0000","last_became_peered":"2024-02-21T23:05:51.428133+0000","last_unstale":"2024-02-21T23:05:52.418119+0000","last_undegraded":"2024-02-21T23:05:52.418119+0000","last_fullsized":"2024-02-21T23:05:52.418119+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:49:46.025250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"75'1","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.384804+0000","last_change":"2024-02-21T23:05:50.377574+0000","last_active":"2024-02-21T23:05:52.384804+0000","last_peered":"2024-02-21T23:05:52.384804+0000","last_clean":"2024-02-21T23:05:52.384804+0000","last_became_active":"2024-02-21T23:05:50.377250+0000","last_became_peered":"2024-02-21T23:05:50.377250+0000","last_unstale":"2024-02-21T23:05:52.384804+0000","last_undegraded":"2024-02-21T23:05:52.384804+0000","last_fullsized":"2024-02-21T23:05:52.384804+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:26:38.366160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.418441+0000","last_change":"2024-02-21T23:05:51.421508+0000","last_active":"2024-02-21T23:05:52.418441+0000","last_peered":"2024-02-21T23:05:52.418441+0000","last_clean":"2024-02-21T23:05:52.418441+0000","last_became_active":"2024-02-21T23:05:51.420041+0000","last_became_peered":"2024-02-21T23:05:51.420041+0000","last_unstale":"2024-02-21T23:05:52.418441+0000","last_undegraded":"2024-02-21T23:05:52.418441+0000","last_fullsized":"2024-02-21T23:05:52.418441+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:10:56.883222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"75'1","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.826394+0000","last_change":"2024-02-21T23:05:50.378668+0000","last_active":"2024-02-21T23:05:52.826394+0000","last_peered":"2024-02-21T23:05:52.826394+0000","last_clean":"2024-02-21T23:05:52.826394+0000","last_became_active":"2024-02-21T23:05:50.378369+0000","last_became_peered":"2024-02-21T23:05:50.378369+0000","last_unstale":"2024-02-21T23:05:52.826394+0000","last_undegraded":"2024-02-21T23:05:52.826394+0000","last_fullsized":"2024-02-21T23:05:52.826394+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:45:05.837402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.384312+0000","last_change":"2024-02-21T23:05:51.419084+0000","last_active":"2024-02-21T23:05:52.384312+0000","last_peered":"2024-02-21T23:05:52.384312+0000","last_clean":"2024-02-21T23:05:52.384312+0000","last_became_active":"2024-02-21T23:05:51.418752+0000","last_became_peered":"2024-02-21T23:05:51.418752+0000","last_unstale":"2024-02-21T23:05:52.384312+0000","last_undegraded":"2024-02-21T23:05:52.384312+0000","last_fullsized":"2024-02-21T23:05:52.384312+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:20:19.564331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.417555+0000","last_change":"2024-02-21T23:05:50.375534+0000","last_active":"2024-02-21T23:05:52.417555+0000","last_peered":"2024-02-21T23:05:52.417555+0000","last_clean":"2024-02-21T23:05:52.417555+0000","last_became_active":"2024-02-21T23:05:50.373762+0000","last_became_peered":"2024-02-21T23:05:50.373762+0000","last_unstale":"2024-02-21T23:05:52.417555+0000","last_undegraded":"2024-02-21T23:05:52.417555+0000","last_fullsized":"2024-02-21T23:05:52.417555+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:49:36.738197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.828380+0000","last_change":"2024-02-21T23:05:51.426711+0000","last_active":"2024-02-21T23:05:52.828380+0000","last_peered":"2024-02-21T23:05:52.828380+0000","last_clean":"2024-02-21T23:05:52.828380+0000","last_became_active":"2024-02-21T23:05:51.426415+0000","last_became_peered":"2024-02-21T23:05:51.426415+0000","last_unstale":"2024-02-21T23:05:52.828380+0000","last_undegraded":"2024-02-21T23:05:52.828380+0000","last_fullsized":"2024-02-21T23:05:52.828380+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:58:55.829941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.379409+0000","last_change":"2024-02-21T23:05:50.355567+0000","last_active":"2024-02-21T23:05:52.379409+0000","last_peered":"2024-02-21T23:05:52.379409+0000","last_clean":"2024-02-21T23:05:52.379409+0000","last_became_active":"2024-02-21T23:05:50.354835+0000","last_became_peered":"2024-02-21T23:05:50.354835+0000","last_unstale":"2024-02-21T23:05:52.379409+0000","last_undegraded":"2024-02-21T23:05:52.379409+0000","last_fullsized":"2024-02-21T23:05:52.379409+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:21:52.655098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.417589+0000","last_change":"2024-02-21T23:05:51.399923+0000","last_active":"2024-02-21T23:05:52.417589+0000","last_peered":"2024-02-21T23:05:52.417589+0000","last_clean":"2024-02-21T23:05:52.417589+0000","last_became_active":"2024-02-21T23:05:51.399642+0000","last_became_peered":"2024-02-21T23:05:51.399642+0000","last_unstale":"2024-02-21T23:05:52.417589+0000","last_undegraded":"2024-02-21T23:05:52.417589+0000","last_fullsized":"2024-02-21T23:05:52.417589+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:21:18.440977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.535856+0000","last_change":"2024-02-21T23:05:50.386639+0000","last_active":"2024-02-21T23:05:52.535856+0000","last_peered":"2024-02-21T23:05:52.535856+0000","last_clean":"2024-02-21T23:05:52.535856+0000","last_became_active":"2024-02-21T23:05:50.386158+0000","last_became_peered":"2024-02-21T23:05:50.386158+0000","last_unstale":"2024-02-21T23:05:52.535856+0000","last_undegraded":"2024-02-21T23:05:52.535856+0000","last_fullsized":"2024-02-21T23:05:52.535856+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:43:17.496894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.826815+0000","last_change":"2024-02-21T23:05:51.422844+0000","last_active":"2024-02-21T23:05:52.826815+0000","last_peered":"2024-02-21T23:05:52.826815+0000","last_clean":"2024-02-21T23:05:52.826815+0000","last_became_active":"2024-02-21T23:05:51.422488+0000","last_became_peered":"2024-02-21T23:05:51.422488+0000","last_unstale":"2024-02-21T23:05:52.826815+0000","last_undegraded":"2024-02-21T23:05:52.826815+0000","last_fullsized":"2024-02-21T23:05:52.826815+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:27:54.695721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.827451+0000","last_change":"2024-02-21T23:05:50.385146+0000","last_active":"2024-02-21T23:05:52.827451+0000","last_peered":"2024-02-21T23:05:52.827451+0000","last_clean":"2024-02-21T23:05:52.827451+0000","last_became_active":"2024-02-21T23:05:50.384418+0000","last_became_peered":"2024-02-21T23:05:50.384418+0000","last_unstale":"2024-02-21T23:05:52.827451+0000","last_undegraded":"2024-02-21T23:05:52.827451+0000","last_fullsized":"2024-02-21T23:05:52.827451+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:43:17.496894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.381706+0000","last_change":"2024-02-21T23:05:51.418004+0000","last_active":"2024-02-21T23:05:52.381706+0000","last_peered":"2024-02-21T23:05:52.381706+0000","last_clean":"2024-02-21T23:05:52.381706+0000","last_became_active":"2024-02-21T23:05:51.417463+0000","last_became_peered":"2024-02-21T23:05:51.417463+0000","last_unstale":"2024-02-21T23:05:52.381706+0000","last_undegraded":"2024-02-21T23:05:52.381706+0000","last_fullsized":"2024-02-21T23:05:52.381706+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:59:13.292494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"75'3","reported_seq":17,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.383080+0000","last_change":"2024-02-21T23:05:50.373189+0000","last_active":"2024-02-21T23:05:52.383080+0000","last_peered":"2024-02-21T23:05:52.383080+0000","last_clean":"2024-02-21T23:05:52.383080+0000","last_became_active":"2024-02-21T23:05:50.370680+0000","last_became_peered":"2024-02-21T23:05:50.370680+0000","last_unstale":"2024-02-21T23:05:52.383080+0000","last_undegraded":"2024-02-21T23:05:52.383080+0000","last_fullsized":"2024-02-21T23:05:52.383080+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:21:03.121546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.827306+0000","last_change":"2024-02-21T23:05:51.405242+0000","last_active":"2024-02-21T23:05:52.827306+0000","last_peered":"2024-02-21T23:05:52.827306+0000","last_clean":"2024-02-21T23:05:52.827306+0000","last_became_active":"2024-02-21T23:05:51.403777+0000","last_became_peered":"2024-02-21T23:05:51.403777+0000","last_unstale":"2024-02-21T23:05:52.827306+0000","last_undegraded":"2024-02-21T23:05:52.827306+0000","last_fullsized":"2024-02-21T23:05:52.827306+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:34:14.420091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"75'6","reported_seq":20,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.829576+0000","last_change":"2024-02-21T23:05:50.374208+0000","last_active":"2024-02-21T23:05:52.829576+0000","last_peered":"2024-02-21T23:05:52.829576+0000","last_clean":"2024-02-21T23:05:52.829576+0000","last_became_active":"2024-02-21T23:05:50.373869+0000","last_became_peered":"2024-02-21T23:05:50.373869+0000","last_unstale":"2024-02-21T23:05:52.829576+0000","last_undegraded":"2024-02-21T23:05:52.829576+0000","last_fullsized":"2024-02-21T23:05:52.829576+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:56:49.435540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.827623+0000","last_change":"2024-02-21T23:05:51.423479+0000","last_active":"2024-02-21T23:05:52.827623+0000","last_peered":"2024-02-21T23:05:52.827623+0000","last_clean":"2024-02-21T23:05:52.827623+0000","last_became_active":"2024-02-21T23:05:51.423190+0000","last_became_peered":"2024-02-21T23:05:51.423190+0000","last_unstale":"2024-02-21T23:05:52.827623+0000","last_undegraded":"2024-02-21T23:05:52.827623+0000","last_fullsized":"2024-02-21T23:05:52.827623+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:03:08.789066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"75'2","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.826505+0000","last_change":"2024-02-21T23:05:50.381643+0000","last_active":"2024-02-21T23:05:52.826505+0000","last_peered":"2024-02-21T23:05:52.826505+0000","last_clean":"2024-02-21T23:05:52.826505+0000","last_became_active":"2024-02-21T23:05:50.381345+0000","last_became_peered":"2024-02-21T23:05:50.381345+0000","last_unstale":"2024-02-21T23:05:52.826505+0000","last_undegraded":"2024-02-21T23:05:52.826505+0000","last_fullsized":"2024-02-21T23:05:52.826505+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:01:27.369089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.828068+0000","last_change":"2024-02-21T23:05:51.416753+0000","last_active":"2024-02-21T23:05:52.828068+0000","last_peered":"2024-02-21T23:05:52.828068+0000","last_clean":"2024-02-21T23:05:52.828068+0000","last_became_active":"2024-02-21T23:05:51.416427+0000","last_became_peered":"2024-02-21T23:05:51.416427+0000","last_unstale":"2024-02-21T23:05:52.828068+0000","last_undegraded":"2024-02-21T23:05:52.828068+0000","last_fullsized":"2024-02-21T23:05:52.828068+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:34:14.420091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.535386+0000","last_change":"2024-02-21T23:05:50.381962+0000","last_active":"2024-02-21T23:05:52.535386+0000","last_peered":"2024-02-21T23:05:52.535386+0000","last_clean":"2024-02-21T23:05:52.535386+0000","last_became_active":"2024-02-21T23:05:50.380838+0000","last_became_peered":"2024-02-21T23:05:50.380838+0000","last_unstale":"2024-02-21T23:05:52.535386+0000","last_undegraded":"2024-02-21T23:05:52.535386+0000","last_fullsized":"2024-02-21T23:05:52.535386+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:10:10.973567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.829697+0000","last_change":"2024-02-21T23:05:51.408507+0000","last_active":"2024-02-21T23:05:52.829697+0000","last_peered":"2024-02-21T23:05:52.829697+0000","last_clean":"2024-02-21T23:05:52.829697+0000","last_became_active":"2024-02-21T23:05:51.408198+0000","last_became_peered":"2024-02-21T23:05:51.408198+0000","last_unstale":"2024-02-21T23:05:52.829697+0000","last_undegraded":"2024-02-21T23:05:52.829697+0000","last_fullsized":"2024-02-21T23:05:52.829697+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:08:36.461250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.379927+0000","last_change":"2024-02-21T23:05:50.353116+0000","last_active":"2024-02-21T23:05:52.379927+0000","last_peered":"2024-02-21T23:05:52.379927+0000","last_clean":"2024-02-21T23:05:52.379927+0000","last_became_active":"2024-02-21T23:05:50.352768+0000","last_became_peered":"2024-02-21T23:05:50.352768+0000","last_unstale":"2024-02-21T23:05:52.379927+0000","last_undegraded":"2024-02-21T23:05:52.379927+0000","last_fullsized":"2024-02-21T23:05:52.379927+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:52:09.073053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.828739+0000","last_change":"2024-02-21T23:05:51.416424+0000","last_active":"2024-02-21T23:05:52.828739+0000","last_peered":"2024-02-21T23:05:52.828739+0000","last_clean":"2024-02-21T23:05:52.828739+0000","last_became_active":"2024-02-21T23:05:51.416070+0000","last_became_peered":"2024-02-21T23:05:51.416070+0000","last_unstale":"2024-02-21T23:05:52.828739+0000","last_undegraded":"2024-02-21T23:05:52.828739+0000","last_fullsized":"2024-02-21T23:05:52.828739+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:56:10.406772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"75'8","reported_seq":22,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.418474+0000","last_change":"2024-02-21T23:05:50.376063+0000","last_active":"2024-02-21T23:05:52.418474+0000","last_peered":"2024-02-21T23:05:52.418474+0000","last_clean":"2024-02-21T23:05:52.418474+0000","last_became_active":"2024-02-21T23:05:50.375772+0000","last_became_peered":"2024-02-21T23:05:50.375772+0000","last_unstale":"2024-02-21T23:05:52.418474+0000","last_undegraded":"2024-02-21T23:05:52.418474+0000","last_fullsized":"2024-02-21T23:05:52.418474+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:26:58.076002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.381136+0000","last_change":"2024-02-21T23:05:51.421565+0000","last_active":"2024-02-21T23:05:52.381136+0000","last_peered":"2024-02-21T23:05:52.381136+0000","last_clean":"2024-02-21T23:05:52.381136+0000","last_became_active":"2024-02-21T23:05:51.421252+0000","last_became_peered":"2024-02-21T23:05:51.421252+0000","last_unstale":"2024-02-21T23:05:52.381136+0000","last_undegraded":"2024-02-21T23:05:52.381136+0000","last_fullsized":"2024-02-21T23:05:52.381136+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:40:45.445418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"75'1","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.536140+0000","last_change":"2024-02-21T23:05:50.383778+0000","last_active":"2024-02-21T23:05:52.536140+0000","last_peered":"2024-02-21T23:05:52.536140+0000","last_clean":"2024-02-21T23:05:52.536140+0000","last_became_active":"2024-02-21T23:05:50.383436+0000","last_became_peered":"2024-02-21T23:05:50.383436+0000","last_unstale":"2024-02-21T23:05:52.536140+0000","last_undegraded":"2024-02-21T23:05:52.536140+0000","last_fullsized":"2024-02-21T23:05:52.536140+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:45:05.837402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.379870+0000","last_change":"2024-02-21T23:05:51.416676+0000","last_active":"2024-02-21T23:05:52.379870+0000","last_peered":"2024-02-21T23:05:52.379870+0000","last_clean":"2024-02-21T23:05:52.379870+0000","last_became_active":"2024-02-21T23:05:51.416297+0000","last_became_peered":"2024-02-21T23:05:51.416297+0000","last_unstale":"2024-02-21T23:05:52.379870+0000","last_undegraded":"2024-02-21T23:05:52.379870+0000","last_fullsized":"2024-02-21T23:05:52.379870+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:29:33.209545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"75'2","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.384553+0000","last_change":"2024-02-21T23:05:50.377181+0000","last_active":"2024-02-21T23:05:52.384553+0000","last_peered":"2024-02-21T23:05:52.384553+0000","last_clean":"2024-02-21T23:05:52.384553+0000","last_became_active":"2024-02-21T23:05:50.373524+0000","last_became_peered":"2024-02-21T23:05:50.373524+0000","last_unstale":"2024-02-21T23:05:52.384553+0000","last_undegraded":"2024-02-21T23:05:52.384553+0000","last_fullsized":"2024-02-21T23:05:52.384553+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:01:18.419728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.380361+0000","last_change":"2024-02-21T23:05:51.412971+0000","last_active":"2024-02-21T23:05:52.380361+0000","last_peered":"2024-02-21T23:05:52.380361+0000","last_clean":"2024-02-21T23:05:52.380361+0000","last_became_active":"2024-02-21T23:05:51.410966+0000","last_became_peered":"2024-02-21T23:05:51.410966+0000","last_unstale":"2024-02-21T23:05:52.380361+0000","last_undegraded":"2024-02-21T23:05:52.380361+0000","last_fullsized":"2024-02-21T23:05:52.380361+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:00:39.163092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"75'1","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.383103+0000","last_change":"2024-02-21T23:05:50.366307+0000","last_active":"2024-02-21T23:05:52.383103+0000","last_peered":"2024-02-21T23:05:52.383103+0000","last_clean":"2024-02-21T23:05:52.383103+0000","last_became_active":"2024-02-21T23:05:50.359426+0000","last_became_peered":"2024-02-21T23:05:50.359426+0000","last_unstale":"2024-02-21T23:05:52.383103+0000","last_undegraded":"2024-02-21T23:05:52.383103+0000","last_fullsized":"2024-02-21T23:05:52.383103+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:13:15.770526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.381306+0000","last_change":"2024-02-21T23:05:51.424067+0000","last_active":"2024-02-21T23:05:52.381306+0000","last_peered":"2024-02-21T23:05:52.381306+0000","last_clean":"2024-02-21T23:05:52.381306+0000","last_became_active":"2024-02-21T23:05:51.423644+0000","last_became_peered":"2024-02-21T23:05:51.423644+0000","last_unstale":"2024-02-21T23:05:52.381306+0000","last_undegraded":"2024-02-21T23:05:52.381306+0000","last_fullsized":"2024-02-21T23:05:52.381306+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:04:03.084512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.827402+0000","last_change":"2024-02-21T23:05:50.377924+0000","last_active":"2024-02-21T23:05:52.827402+0000","last_peered":"2024-02-21T23:05:52.827402+0000","last_clean":"2024-02-21T23:05:52.827402+0000","last_became_active":"2024-02-21T23:05:50.377440+0000","last_became_peered":"2024-02-21T23:05:50.377440+0000","last_unstale":"2024-02-21T23:05:52.827402+0000","last_undegraded":"2024-02-21T23:05:52.827402+0000","last_fullsized":"2024-02-21T23:05:52.827402+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:42:15.626763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.418107+0000","last_change":"2024-02-21T23:05:51.414619+0000","last_active":"2024-02-21T23:05:52.418107+0000","last_peered":"2024-02-21T23:05:52.418107+0000","last_clean":"2024-02-21T23:05:52.418107+0000","last_became_active":"2024-02-21T23:05:51.414059+0000","last_became_peered":"2024-02-21T23:05:51.414059+0000","last_unstale":"2024-02-21T23:05:52.418107+0000","last_undegraded":"2024-02-21T23:05:52.418107+0000","last_fullsized":"2024-02-21T23:05:52.418107+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:58:05.114297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"75'2","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.381450+0000","last_change":"2024-02-21T23:05:50.366877+0000","last_active":"2024-02-21T23:05:52.381450+0000","last_peered":"2024-02-21T23:05:52.381450+0000","last_clean":"2024-02-21T23:05:52.381450+0000","last_became_active":"2024-02-21T23:05:50.365566+0000","last_became_peered":"2024-02-21T23:05:50.365566+0000","last_unstale":"2024-02-21T23:05:52.381450+0000","last_undegraded":"2024-02-21T23:05:52.381450+0000","last_fullsized":"2024-02-21T23:05:52.381450+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:58:04.101475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.418441+0000","last_change":"2024-02-21T23:05:51.411005+0000","last_active":"2024-02-21T23:05:52.418441+0000","last_peered":"2024-02-21T23:05:52.418441+0000","last_clean":"2024-02-21T23:05:52.418441+0000","last_became_active":"2024-02-21T23:05:51.410503+0000","last_became_peered":"2024-02-21T23:05:51.410503+0000","last_unstale":"2024-02-21T23:05:52.418441+0000","last_undegraded":"2024-02-21T23:05:52.418441+0000","last_fullsized":"2024-02-21T23:05:52.418441+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:51:10.610852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.381097+0000","last_change":"2024-02-21T23:05:50.376888+0000","last_active":"2024-02-21T23:05:52.381097+0000","last_peered":"2024-02-21T23:05:52.381097+0000","last_clean":"2024-02-21T23:05:52.381097+0000","last_became_active":"2024-02-21T23:05:50.376610+0000","last_became_peered":"2024-02-21T23:05:50.376610+0000","last_unstale":"2024-02-21T23:05:52.381097+0000","last_undegraded":"2024-02-21T23:05:52.381097+0000","last_fullsized":"2024-02-21T23:05:52.381097+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:51:16.181060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.829202+0000","last_change":"2024-02-21T23:05:51.417167+0000","last_active":"2024-02-21T23:05:52.829202+0000","last_peered":"2024-02-21T23:05:52.829202+0000","last_clean":"2024-02-21T23:05:52.829202+0000","last_became_active":"2024-02-21T23:05:51.415957+0000","last_became_peered":"2024-02-21T23:05:51.415957+0000","last_unstale":"2024-02-21T23:05:52.829202+0000","last_undegraded":"2024-02-21T23:05:52.829202+0000","last_fullsized":"2024-02-21T23:05:52.829202+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:26:59.088824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.381605+0000","last_change":"2024-02-21T23:05:50.375808+0000","last_active":"2024-02-21T23:05:52.381605+0000","last_peered":"2024-02-21T23:05:52.381605+0000","last_clean":"2024-02-21T23:05:52.381605+0000","last_became_active":"2024-02-21T23:05:50.375467+0000","last_became_peered":"2024-02-21T23:05:50.375467+0000","last_unstale":"2024-02-21T23:05:52.381605+0000","last_undegraded":"2024-02-21T23:05:52.381605+0000","last_fullsized":"2024-02-21T23:05:52.381605+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:29:01.979463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.383305+0000","last_change":"2024-02-21T23:05:51.414453+0000","last_active":"2024-02-21T23:05:52.383305+0000","last_peered":"2024-02-21T23:05:52.383305+0000","last_clean":"2024-02-21T23:05:52.383305+0000","last_became_active":"2024-02-21T23:05:51.413499+0000","last_became_peered":"2024-02-21T23:05:51.413499+0000","last_unstale":"2024-02-21T23:05:52.383305+0000","last_undegraded":"2024-02-21T23:05:52.383305+0000","last_fullsized":"2024-02-21T23:05:52.383305+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:55:21.967274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"75'2","reported_seq":16,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.827140+0000","last_change":"2024-02-21T23:05:50.381948+0000","last_active":"2024-02-21T23:05:52.827140+0000","last_peered":"2024-02-21T23:05:52.827140+0000","last_clean":"2024-02-21T23:05:52.827140+0000","last_became_active":"2024-02-21T23:05:50.381452+0000","last_became_peered":"2024-02-21T23:05:50.381452+0000","last_unstale":"2024-02-21T23:05:52.827140+0000","last_undegraded":"2024-02-21T23:05:52.827140+0000","last_fullsized":"2024-02-21T23:05:52.827140+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:22:16.414686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.828536+0000","last_change":"2024-02-21T23:05:51.410619+0000","last_active":"2024-02-21T23:05:52.828536+0000","last_peered":"2024-02-21T23:05:52.828536+0000","last_clean":"2024-02-21T23:05:52.828536+0000","last_became_active":"2024-02-21T23:05:51.409937+0000","last_became_peered":"2024-02-21T23:05:51.409937+0000","last_unstale":"2024-02-21T23:05:52.828536+0000","last_undegraded":"2024-02-21T23:05:52.828536+0000","last_fullsized":"2024-02-21T23:05:52.828536+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:22:32.845173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"76'1","reported_seq":15,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.418529+0000","last_change":"2024-02-21T23:05:50.362370+0000","last_active":"2024-02-21T23:05:52.418529+0000","last_peered":"2024-02-21T23:05:52.418529+0000","last_clean":"2024-02-21T23:05:52.418529+0000","last_became_active":"2024-02-21T23:05:50.361848+0000","last_became_peered":"2024-02-21T23:05:50.361848+0000","last_unstale":"2024-02-21T23:05:52.418529+0000","last_undegraded":"2024-02-21T23:05:52.418529+0000","last_fullsized":"2024-02-21T23:05:52.418529+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:08:35.448428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.384236+0000","last_change":"2024-02-21T23:05:51.421357+0000","last_active":"2024-02-21T23:05:52.384236+0000","last_peered":"2024-02-21T23:05:52.384236+0000","last_clean":"2024-02-21T23:05:52.384236+0000","last_became_active":"2024-02-21T23:05:51.420734+0000","last_became_peered":"2024-02-21T23:05:51.420734+0000","last_unstale":"2024-02-21T23:05:52.384236+0000","last_undegraded":"2024-02-21T23:05:52.384236+0000","last_fullsized":"2024-02-21T23:05:52.384236+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:31:44.536584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.535981+0000","last_change":"2024-02-21T23:05:50.384680+0000","last_active":"2024-02-21T23:05:52.535981+0000","last_peered":"2024-02-21T23:05:52.535981+0000","last_clean":"2024-02-21T23:05:52.535981+0000","last_became_active":"2024-02-21T23:05:50.384208+0000","last_became_peered":"2024-02-21T23:05:50.384208+0000","last_unstale":"2024-02-21T23:05:52.535981+0000","last_undegraded":"2024-02-21T23:05:52.535981+0000","last_fullsized":"2024-02-21T23:05:52.535981+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:01:27.369089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.417801+0000","last_change":"2024-02-21T23:05:51.395859+0000","last_active":"2024-02-21T23:05:52.417801+0000","last_peered":"2024-02-21T23:05:52.417801+0000","last_clean":"2024-02-21T23:05:52.417801+0000","last_became_active":"2024-02-21T23:05:51.395012+0000","last_became_peered":"2024-02-21T23:05:51.395012+0000","last_unstale":"2024-02-21T23:05:52.417801+0000","last_undegraded":"2024-02-21T23:05:52.417801+0000","last_fullsized":"2024-02-21T23:05:52.417801+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:29:02.992284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.535249+0000","last_change":"2024-02-21T23:05:50.383920+0000","last_active":"2024-02-21T23:05:52.535249+0000","last_peered":"2024-02-21T23:05:52.535249+0000","last_clean":"2024-02-21T23:05:52.535249+0000","last_became_active":"2024-02-21T23:05:50.382488+0000","last_became_peered":"2024-02-21T23:05:50.382488+0000","last_unstale":"2024-02-21T23:05:52.535249+0000","last_undegraded":"2024-02-21T23:05:52.535249+0000","last_fullsized":"2024-02-21T23:05:52.535249+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:26:58.076002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.829134+0000","last_change":"2024-02-21T23:05:51.422495+0000","last_active":"2024-02-21T23:05:52.829134+0000","last_peered":"2024-02-21T23:05:52.829134+0000","last_clean":"2024-02-21T23:05:52.829134+0000","last_became_active":"2024-02-21T23:05:51.422196+0000","last_became_peered":"2024-02-21T23:05:51.422196+0000","last_unstale":"2024-02-21T23:05:52.829134+0000","last_undegraded":"2024-02-21T23:05:52.829134+0000","last_fullsized":"2024-02-21T23:05:52.829134+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:10:11.986389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.828484+0000","last_change":"2024-02-21T23:05:50.376027+0000","last_active":"2024-02-21T23:05:52.828484+0000","last_peered":"2024-02-21T23:05:52.828484+0000","last_clean":"2024-02-21T23:05:52.828484+0000","last_became_active":"2024-02-21T23:05:50.375170+0000","last_became_peered":"2024-02-21T23:05:50.375170+0000","last_unstale":"2024-02-21T23:05:52.828484+0000","last_undegraded":"2024-02-21T23:05:52.828484+0000","last_fullsized":"2024-02-21T23:05:52.828484+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:39:32.228635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.828497+0000","last_change":"2024-02-21T23:05:51.420500+0000","last_active":"2024-02-21T23:05:52.828497+0000","last_peered":"2024-02-21T23:05:52.828497+0000","last_clean":"2024-02-21T23:05:52.828497+0000","last_became_active":"2024-02-21T23:05:51.420103+0000","last_became_peered":"2024-02-21T23:05:51.420103+0000","last_unstale":"2024-02-21T23:05:52.828497+0000","last_undegraded":"2024-02-21T23:05:52.828497+0000","last_fullsized":"2024-02-21T23:05:52.828497+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:27:01.191337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.380442+0000","last_change":"2024-02-21T23:05:50.354805+0000","last_active":"2024-02-21T23:05:52.380442+0000","last_peered":"2024-02-21T23:05:52.380442+0000","last_clean":"2024-02-21T23:05:52.380442+0000","last_became_active":"2024-02-21T23:05:50.354525+0000","last_became_peered":"2024-02-21T23:05:50.354525+0000","last_unstale":"2024-02-21T23:05:52.380442+0000","last_undegraded":"2024-02-21T23:05:52.380442+0000","last_fullsized":"2024-02-21T23:05:52.380442+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:28:05.314552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.827781+0000","last_change":"2024-02-21T23:05:51.416548+0000","last_active":"2024-02-21T23:05:52.827781+0000","last_peered":"2024-02-21T23:05:52.827781+0000","last_clean":"2024-02-21T23:05:52.827781+0000","last_became_active":"2024-02-21T23:05:51.416222+0000","last_became_peered":"2024-02-21T23:05:51.416222+0000","last_unstale":"2024-02-21T23:05:52.827781+0000","last_undegraded":"2024-02-21T23:05:52.827781+0000","last_fullsized":"2024-02-21T23:05:52.827781+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:22:32.845173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.828804+0000","last_change":"2024-02-21T23:05:50.373554+0000","last_active":"2024-02-21T23:05:52.828804+0000","last_peered":"2024-02-21T23:05:52.828804+0000","last_clean":"2024-02-21T23:05:52.828804+0000","last_became_active":"2024-02-21T23:05:50.373022+0000","last_became_peered":"2024-02-21T23:05:50.373022+0000","last_unstale":"2024-02-21T23:05:52.828804+0000","last_undegraded":"2024-02-21T23:05:52.828804+0000","last_fullsized":"2024-02-21T23:05:52.828804+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:38:56.665723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.827906+0000","last_change":"2024-02-21T23:05:51.412683+0000","last_active":"2024-02-21T23:05:52.827906+0000","last_peered":"2024-02-21T23:05:52.827906+0000","last_clean":"2024-02-21T23:05:52.827906+0000","last_became_active":"2024-02-21T23:05:51.412389+0000","last_became_peered":"2024-02-21T23:05:51.412389+0000","last_unstale":"2024-02-21T23:05:52.827906+0000","last_undegraded":"2024-02-21T23:05:52.827906+0000","last_fullsized":"2024-02-21T23:05:52.827906+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:57:06.211024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.826797+0000","last_change":"2024-02-21T23:05:50.376356+0000","last_active":"2024-02-21T23:05:52.826797+0000","last_peered":"2024-02-21T23:05:52.826797+0000","last_clean":"2024-02-21T23:05:52.826797+0000","last_became_active":"2024-02-21T23:05:50.370467+0000","last_became_peered":"2024-02-21T23:05:50.370467+0000","last_unstale":"2024-02-21T23:05:52.826797+0000","last_undegraded":"2024-02-21T23:05:52.826797+0000","last_fullsized":"2024-02-21T23:05:52.826797+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:56:49.435540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":12,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.535891+0000","last_change":"2024-02-21T23:05:51.413008+0000","last_active":"2024-02-21T23:05:52.535891+0000","last_peered":"2024-02-21T23:05:52.535891+0000","last_clean":"2024-02-21T23:05:52.535891+0000","last_became_active":"2024-02-21T23:05:51.412714+0000","last_became_peered":"2024-02-21T23:05:51.412714+0000","last_unstale":"2024-02-21T23:05:52.535891+0000","last_undegraded":"2024-02-21T23:05:52.535891+0000","last_fullsized":"2024-02-21T23:05:52.535891+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:03:08.789066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":14,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.381834+0000","last_change":"2024-02-21T23:05:50.375581+0000","last_active":"2024-02-21T23:05:52.381834+0000","last_peered":"2024-02-21T23:05:52.381834+0000","last_clean":"2024-02-21T23:05:52.381834+0000","last_became_active":"2024-02-21T23:05:50.375282+0000","last_became_peered":"2024-02-21T23:05:50.375282+0000","last_unstale":"2024-02-21T23:05:52.381834+0000","last_undegraded":"2024-02-21T23:05:52.381834+0000","last_fullsized":"2024-02-21T23:05:52.381834+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:49:45.012428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":274,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-02-21T23:05:52.535353+0000","last_change":"2024-02-21T23:02:32.289335+0000","last_active":"2024-02-21T23:05:52.535353+0000","last_peered":"2024-02-21T23:05:52.535353+0000","last_clean":"2024-02-21T23:05:52.535353+0000","last_became_active":"2024-02-21T23:02:32.286913+0000","last_became_peered":"2024-02-21T23:02:32.286913+0000","last_unstale":"2024-02-21T23:05:52.535353+0000","last_undegraded":"2024-02-21T23:05:52.535353+0000","last_fullsized":"2024-02-21T23:05:52.535353+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_clean_scrub_stamp":"2024-02-21T23:02:31.172569+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:36:23.987374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574891,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32156,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339684,"statfs":{"total":96636764160,"available":96603836416,"internally_reserved":0,"allocated":999424,"data_stored":1033908,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574892,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32104,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339736,"statfs":{"total":96636764160,"available":96603889664,"internally_reserved":0,"allocated":942080,"data_stored":445130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574891,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27940,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343900,"statfs":{"total":96636764160,"available":96608153600,"internally_reserved":0,"allocated":933888,"data_stored":443552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574892,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32116,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339724,"statfs":{"total":96636764160,"available":96603877376,"internally_reserved":0,"allocated":950272,"data_stored":445234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574892,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32100,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339740,"statfs":{"total":96636764160,"available":96603893760,"internally_reserved":0,"allocated":933888,"data_stored":444100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574893,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32108,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339732,"statfs":{"total":96636764160,"available":96603885568,"internally_reserved":0,"allocated":942080,"data_stored":444260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574891,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32096,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339744,"statfs":{"total":96636764160,"available":96603897856,"internally_reserved":0,"allocated":937984,"data_stored":443630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574891,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32152,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339688,"statfs":{"total":96636764160,"available":96603840512,"internally_reserved":0,"allocated":995328,"data_stored":1033886,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-21T23:06:01.660 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-02-21T23:06:01.660 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-02-21T23:06:01.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.928+0000 7fa2eac00640 1 Processor -- start 2024-02-21T23:06:01.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.928+0000 7fa2eac00640 1 -- start start 2024-02-21T23:06:01.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.929+0000 7fa2eac00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa2e40caaa0 0x7fa2e40cae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:01.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.929+0000 7fa2eac00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2e4159140 con 0x7fa2e41585d0 2024-02-21T23:06:01.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.929+0000 7fa2eac00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2e40c7310 con 0x7fa2e415d950 2024-02-21T23:06:01.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.929+0000 7fa2eac00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e40c8990 con 0x7fa2e40caaa0 2024-02-21T23:06:01.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.930+0000 7fa2e9800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa2e40caaa0 0x7fa2e40cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:01.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.930+0000 7fa2ea200640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fa2e415d950 0x7fa2e415fd40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.28:57244/0 (socket says 172.21.15.28:57244) 2024-02-21T23:06:01.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.930+0000 7fa2e8e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa2e41585d0 0x7fa2e4158a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43846/0 (socket says 172.21.15.28:43846) 2024-02-21T23:06:01.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.930+0000 7fa2ea200640 1 -- 172.21.15.28:0/4061886902 learned_addr learned my addr 172.21.15.28:0/4061886902 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:06:01.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.931+0000 7fa2e3e00640 1 -- 172.21.15.28:0/4061886902 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4154095785 0 0) 0x7fa2e40c7310 con 0x7fa2e415d950 2024-02-21T23:06:01.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.931+0000 7fa2e3e00640 1 -- 172.21.15.28:0/4061886902 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2cc003520 con 0x7fa2e415d950 2024-02-21T23:06:01.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.931+0000 7fa2e3e00640 1 -- 172.21.15.28:0/4061886902 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 137602095 0 0) 0x7fa2e4159140 con 0x7fa2e41585d0 2024-02-21T23:06:01.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.931+0000 7fa2e3e00640 1 -- 172.21.15.28:0/4061886902 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2e40c7310 con 0x7fa2e41585d0 2024-02-21T23:06:01.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.931+0000 7fa2e9800640 1 -- 172.21.15.28:0/4061886902 >> v1:172.21.15.81:6789/0 conn(0x7fa2e415d950 legacy=0x7fa2e415fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:01.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.931+0000 7fa2e9800640 1 -- 172.21.15.28:0/4061886902 >> v1:172.21.15.28:6789/0 conn(0x7fa2e41585d0 legacy=0x7fa2e4158a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:01.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.931+0000 7fa2e9800640 1 -- 172.21.15.28:0/4061886902 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2e40c75f0 con 0x7fa2e40caaa0 2024-02-21T23:06:01.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.932+0000 7fa2e9800640 1 --2- 172.21.15.28:0/4061886902 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa2e40caaa0 0x7fa2e40cae80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa2d80097d0 tx=0x7fa2d802b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73cbac8b322d01d9 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:01.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.933+0000 7fa2e3e00640 1 -- 172.21.15.28:0/4061886902 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2d8034070 con 0x7fa2e40caaa0 2024-02-21T23:06:01.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.933+0000 7fa2e3e00640 1 -- 172.21.15.28:0/4061886902 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2d8030800 con 0x7fa2e40caaa0 2024-02-21T23:06:01.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.933+0000 7fa2e3e00640 1 -- 172.21.15.28:0/4061886902 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2d8030bb0 con 0x7fa2e40caaa0 2024-02-21T23:06:01.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.934+0000 7fa2eac00640 1 -- 172.21.15.28:0/4061886902 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa2e40caaa0 msgr2=0x7fa2e40cae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:01.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.934+0000 7fa2eac00640 1 --2- 172.21.15.28:0/4061886902 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa2e40caaa0 0x7fa2e40cae80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa2d80097d0 tx=0x7fa2d802b940 comp rx=0 tx=0).stop 2024-02-21T23:06:01.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.934+0000 7fa2eac00640 1 -- 172.21.15.28:0/4061886902 shutdown_connections 2024-02-21T23:06:01.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.934+0000 7fa2eac00640 1 --2- 172.21.15.28:0/4061886902 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa2e40caaa0 0x7fa2e40cae80 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:01.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.934+0000 7fa2eac00640 1 -- 172.21.15.28:0/4061886902 >> 172.21.15.28:0/4061886902 conn(0x7fa2e4072780 msgr2=0x7fa2e4072b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:01.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.934+0000 7fa2eac00640 1 -- 172.21.15.28:0/4061886902 shutdown_connections 2024-02-21T23:06:01.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.934+0000 7fa2eac00640 1 -- 172.21.15.28:0/4061886902 wait complete. 2024-02-21T23:06:01.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.936+0000 7fa2eac00640 1 Processor -- start 2024-02-21T23:06:01.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.936+0000 7fa2eac00640 1 -- start start 2024-02-21T23:06:01.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.937+0000 7fa2eac00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa2e40caaa0 0x7fa2e40afec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:01.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.937+0000 7fa2eac00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa2e41585d0 0x7fa2e4157690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:01.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.937+0000 7fa2eac00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2e415d950 0x7fa2e40ae0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:01.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.937+0000 7fa2eac00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e406d920 con 0x7fa2e415d950 2024-02-21T23:06:01.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.937+0000 7fa2eac00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e40c7bb0 con 0x7fa2e40caaa0 2024-02-21T23:06:01.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.937+0000 7fa2eac00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e406dda0 con 0x7fa2e41585d0 2024-02-21T23:06:01.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.937+0000 7fa2ea200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2e415d950 0x7fa2e40ae0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:01.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.937+0000 7fa2ea200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2e415d950 0x7fa2e40ae0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54200/0 (socket says 172.21.15.28:54200) 2024-02-21T23:06:01.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.937+0000 7fa2ea200640 1 -- 172.21.15.28:0/559747590 learned_addr learned my addr 172.21.15.28:0/559747590 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:01.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.937+0000 7fa2e8e00640 1 --2- 172.21.15.28:0/559747590 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa2e41585d0 0x7fa2e4157690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:01.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.937+0000 7fa2e9800640 1 --2- 172.21.15.28:0/559747590 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa2e40caaa0 0x7fa2e40afec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:01.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.938+0000 7fa2ea200640 1 -- 172.21.15.28:0/559747590 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa2e41585d0 msgr2=0x7fa2e4157690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:01.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.938+0000 7fa2ea200640 1 --2- 172.21.15.28:0/559747590 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa2e41585d0 0x7fa2e4157690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:01.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.938+0000 7fa2ea200640 1 -- 172.21.15.28:0/559747590 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa2e40caaa0 msgr2=0x7fa2e40afec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:01.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.938+0000 7fa2ea200640 1 --2- 172.21.15.28:0/559747590 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa2e40caaa0 0x7fa2e40afec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:01.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.938+0000 7fa2ea200640 1 -- 172.21.15.28:0/559747590 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2e40ab940 con 0x7fa2e415d950 2024-02-21T23:06:01.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.938+0000 7fa2e8e00640 1 --2- 172.21.15.28:0/559747590 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa2e41585d0 0x7fa2e4157690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:06:01.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.938+0000 7fa2ea200640 1 --2- 172.21.15.28:0/559747590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2e415d950 0x7fa2e40ae0d0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc00cb40 tx=0x7fa2dc00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:01.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.938+0000 7fa2e1600640 1 -- 172.21.15.28:0/559747590 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2dc017070 con 0x7fa2e415d950 2024-02-21T23:06:01.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.938+0000 7fa2e1600640 1 -- 172.21.15.28:0/559747590 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2dc00aca0 con 0x7fa2e415d950 2024-02-21T23:06:01.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.938+0000 7fa2e1600640 1 -- 172.21.15.28:0/559747590 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2dc012600 con 0x7fa2e415d950 2024-02-21T23:06:01.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.938+0000 7fa2eac00640 1 -- 172.21.15.28:0/559747590 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2e40ad570 con 0x7fa2e415d950 2024-02-21T23:06:01.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.938+0000 7fa2eac00640 1 -- 172.21.15.28:0/559747590 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2e4157bd0 con 0x7fa2e415d950 2024-02-21T23:06:01.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.940+0000 7fa2eac00640 1 -- 172.21.15.28:0/559747590 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2e414ea10 con 0x7fa2e415d950 2024-02-21T23:06:01.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.941+0000 7fa2e1600640 1 -- 172.21.15.28:0/559747590 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa2dc01a070 con 0x7fa2e415d950 2024-02-21T23:06:01.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.942+0000 7fa2e1600640 1 --2- 172.21.15.28:0/559747590 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa2c806b4d0 0x7fa2c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:01.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.942+0000 7fa2e1600640 1 -- 172.21.15.28:0/559747590 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa2dc08a0f0 con 0x7fa2e415d950 2024-02-21T23:06:01.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.943+0000 7fa2e9800640 1 --2- 172.21.15.28:0/559747590 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa2c806b4d0 0x7fa2c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:01.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.944+0000 7fa2e9800640 1 --2- 172.21.15.28:0/559747590 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa2c806b4d0 0x7fa2c806d990 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fa2d8036000 tx=0x7fa2d8008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:01.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:01.946+0000 7fa2e1600640 1 -- 172.21.15.28:0/559747590 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fa2e414ea10 con 0x7fa2e415d950 2024-02-21T23:06:02.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.214+0000 7fa2eac00640 1 -- 172.21.15.28:0/559747590 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fa2e404f290 con 0x7fa2e415d950 2024-02-21T23:06:02.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.218+0000 7fa2e1600640 1 -- 172.21.15.28:0/559747590 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v13) v1 ==== 190+0+147 (secure 0 0 0) 0x7fa2dc0586f0 con 0x7fa2e415d950 2024-02-21T23:06:02.207 INFO:teuthology.orchestra.run.smithi028.stdout:[client.0] 2024-02-21T23:06:02.207 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQDTgNZlILvYLxAAoREWxKKT7YcXnhkgk8YvoQ== 2024-02-21T23:06:02.207 INFO:teuthology.orchestra.run.smithi028.stdout: caps mds = "allow" 2024-02-21T23:06:02.207 INFO:teuthology.orchestra.run.smithi028.stdout: caps mon = "allow r" 2024-02-21T23:06:02.207 INFO:teuthology.orchestra.run.smithi028.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-02-21T23:06:02.208 INFO:teuthology.orchestra.run.smithi028.stderr:updated caps for client.0 2024-02-21T23:06:02.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.221+0000 7fa2eac00640 1 -- 172.21.15.28:0/559747590 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa2c806b4d0 msgr2=0x7fa2c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:02.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.221+0000 7fa2eac00640 1 --2- 172.21.15.28:0/559747590 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa2c806b4d0 0x7fa2c806d990 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fa2d8036000 tx=0x7fa2d8008030 comp rx=0 tx=0).stop 2024-02-21T23:06:02.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.221+0000 7fa2eac00640 1 -- 172.21.15.28:0/559747590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2e415d950 msgr2=0x7fa2e40ae0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:02.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.221+0000 7fa2eac00640 1 --2- 172.21.15.28:0/559747590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2e415d950 0x7fa2e40ae0d0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc00cb40 tx=0x7fa2dc00a060 comp rx=0 tx=0).stop 2024-02-21T23:06:02.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.221+0000 7fa2eac00640 1 -- 172.21.15.28:0/559747590 shutdown_connections 2024-02-21T23:06:02.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.221+0000 7fa2eac00640 1 --2- 172.21.15.28:0/559747590 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa2c806b4d0 0x7fa2c806d990 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:02.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.221+0000 7fa2eac00640 1 --2- 172.21.15.28:0/559747590 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa2e41585d0 0x7fa2e4157690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:02.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.221+0000 7fa2eac00640 1 --2- 172.21.15.28:0/559747590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2e415d950 0x7fa2e40ae0d0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:02.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.221+0000 7fa2eac00640 1 --2- 172.21.15.28:0/559747590 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa2e40caaa0 0x7fa2e40afec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:02.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.221+0000 7fa2eac00640 1 -- 172.21.15.28:0/559747590 >> 172.21.15.28:0/559747590 conn(0x7fa2e4072780 msgr2=0x7fa2e414fd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:02.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.221+0000 7fa2eac00640 1 -- 172.21.15.28:0/559747590 shutdown_connections 2024-02-21T23:06:02.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.221+0000 7fa2eac00640 1 -- 172.21.15.28:0/559747590 wait complete. 2024-02-21T23:06:02.227 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:06:02.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.444+0000 7fd474c00640 1 Processor -- start 2024-02-21T23:06:02.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.445+0000 7fd474c00640 1 -- start start 2024-02-21T23:06:02.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.446+0000 7fd474c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fd4700c3510 0x7fd47015ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:02.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.446+0000 7fd474c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd4701592b0 0x7fd47015d230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:02.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.446+0000 7fd474c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd47015d770 con 0x7fd470156980 2024-02-21T23:06:02.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.446+0000 7fd474c00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd470156d60 con 0x7fd4700c3510 2024-02-21T23:06:02.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.446+0000 7fd474c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd47015c860 con 0x7fd4701592b0 2024-02-21T23:06:02.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.446+0000 7fd46f400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd470156980 0x7fd47006ae70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43856/0 (socket says 172.21.15.28:43856) 2024-02-21T23:06:02.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.446+0000 7fd46f400640 1 -- 172.21.15.28:0/1565479000 learned_addr learned my addr 172.21.15.28:0/1565479000 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:06:02.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.447+0000 7fd466a00640 1 --2- 172.21.15.28:0/1565479000 >> v2:172.21.15.81:3300/0 conn(0x7fd4700c3510 0x7fd47015ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:02.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.447+0000 7fd46fe00640 1 --2- 172.21.15.28:0/1565479000 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd4701592b0 0x7fd47015d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:02.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.447+0000 7fd46ea00640 1 -- 172.21.15.28:0/1565479000 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1729986292 0 0) 0x7fd47015d770 con 0x7fd470156980 2024-02-21T23:06:02.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.447+0000 7fd46ea00640 1 -- 172.21.15.28:0/1565479000 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd454003520 con 0x7fd470156980 2024-02-21T23:06:02.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.447+0000 7fd46ea00640 1 -- 172.21.15.28:0/1565479000 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3872401135 0 0) 0x7fd454003520 con 0x7fd470156980 2024-02-21T23:06:02.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.447+0000 7fd46ea00640 1 -- 172.21.15.28:0/1565479000 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd4701592b0 msgr2=0x7fd47015d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:02.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.447+0000 7fd46ea00640 1 --2- 172.21.15.28:0/1565479000 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd4701592b0 0x7fd47015d230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:02.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.447+0000 7fd46ea00640 1 -- 172.21.15.28:0/1565479000 >> v2:172.21.15.81:3300/0 conn(0x7fd4700c3510 msgr2=0x7fd47015ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:02.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.447+0000 7fd46ea00640 1 --2- 172.21.15.28:0/1565479000 >> v2:172.21.15.81:3300/0 conn(0x7fd4700c3510 0x7fd47015ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:02.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.447+0000 7fd46ea00640 1 -- 172.21.15.28:0/1565479000 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd47006f6d0 con 0x7fd470156980 2024-02-21T23:06:02.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.448+0000 7fd46ea00640 1 -- 172.21.15.28:0/1565479000 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fd460001690 con 0x7fd470156980 2024-02-21T23:06:02.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.448+0000 7fd46ea00640 1 -- 172.21.15.28:0/1565479000 >> v1:172.21.15.28:6789/0 conn(0x7fd470156980 legacy=0x7fd47006ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:02.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.448+0000 7fd46ea00640 1 --2- 172.21.15.28:0/1565479000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4540039b0 0x7fd454003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:02.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.448+0000 7fd46ea00640 1 --2- 172.21.15.28:0/1565479000 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd4701592b0 0x7fd454004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:02.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.448+0000 7fd46ea00640 1 --2- 172.21.15.28:0/1565479000 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd454004d40 0x7fd4540051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:02.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.448+0000 7fd46ea00640 1 -- 172.21.15.28:0/1565479000 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd470156d60 con 0x7fd4540039b0 2024-02-21T23:06:02.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.448+0000 7fd46ea00640 1 -- 172.21.15.28:0/1565479000 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd47015c860 con 0x7fd454004d40 2024-02-21T23:06:02.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.448+0000 7fd46ea00640 1 -- 172.21.15.28:0/1565479000 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd45400ba70 con 0x7fd4701592b0 2024-02-21T23:06:02.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.448+0000 7fd46f400640 1 --2- 172.21.15.28:0/1565479000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4540039b0 0x7fd454003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:02.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.448+0000 7fd46fe00640 1 --2- 172.21.15.28:0/1565479000 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd454004d40 0x7fd4540051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:02.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.448+0000 7fd466a00640 1 --2- 172.21.15.28:0/1565479000 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd4701592b0 0x7fd454004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:02.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.449+0000 7fd46f400640 1 -- 172.21.15.28:0/1565479000 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd4701592b0 msgr2=0x7fd454004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:02.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.449+0000 7fd46f400640 1 --2- 172.21.15.28:0/1565479000 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd4701592b0 0x7fd454004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:02.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.449+0000 7fd46f400640 1 -- 172.21.15.28:0/1565479000 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd454004d40 msgr2=0x7fd4540051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:02.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.449+0000 7fd46f400640 1 --2- 172.21.15.28:0/1565479000 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd454004d40 0x7fd4540051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:02.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.449+0000 7fd46f400640 1 -- 172.21.15.28:0/1565479000 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd454003520 con 0x7fd4540039b0 2024-02-21T23:06:02.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.449+0000 7fd46fe00640 1 --2- 172.21.15.28:0/1565479000 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd454004d40 0x7fd4540051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:06:02.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.450+0000 7fd46f400640 1 --2- 172.21.15.28:0/1565479000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4540039b0 0x7fd454003dd0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fd46000c210 tx=0x7fd454005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0626a08d8b90a50 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:02.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.451+0000 7fd46ea00640 1 -- 172.21.15.28:0/1565479000 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd460007d70 con 0x7fd4540039b0 2024-02-21T23:06:02.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.451+0000 7fd46ea00640 1 -- 172.21.15.28:0/1565479000 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd460030070 con 0x7fd4540039b0 2024-02-21T23:06:02.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.451+0000 7fd46ea00640 1 -- 172.21.15.28:0/1565479000 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd46002cd40 con 0x7fd4540039b0 2024-02-21T23:06:02.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.451+0000 7fd474c00640 1 -- 172.21.15.28:0/1565479000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4540039b0 msgr2=0x7fd454003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:02.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.451+0000 7fd474c00640 1 --2- 172.21.15.28:0/1565479000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4540039b0 0x7fd454003dd0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fd46000c210 tx=0x7fd454005830 comp rx=0 tx=0).stop 2024-02-21T23:06:02.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.451+0000 7fd46fe00640 1 -- 172.21.15.28:0/1565479000 reap_dead start 2024-02-21T23:06:02.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.451+0000 7fd474c00640 1 -- 172.21.15.28:0/1565479000 shutdown_connections 2024-02-21T23:06:02.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.451+0000 7fd474c00640 1 -- 172.21.15.28:0/1565479000 >> 172.21.15.28:0/1565479000 conn(0x7fd4700c7720 msgr2=0x7fd4700c7b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:02.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.451+0000 7fd474c00640 1 -- 172.21.15.28:0/1565479000 shutdown_connections 2024-02-21T23:06:02.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.452+0000 7fd474c00640 1 -- 172.21.15.28:0/1565479000 wait complete. 2024-02-21T23:06:02.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.453+0000 7fd474c00640 1 Processor -- start 2024-02-21T23:06:02.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.453+0000 7fd474c00640 1 -- start start 2024-02-21T23:06:02.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.454+0000 7fd474c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd4540039b0 0x7fd47014ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:02.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.454+0000 7fd474c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd470154bb0 0x7fd4701eed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:02.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.454+0000 7fd474c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd470152c60 0x7fd4701ef250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:02.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.454+0000 7fd474c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd470159de0 con 0x7fd470152c60 2024-02-21T23:06:02.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.454+0000 7fd474c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd470159b00 con 0x7fd4540039b0 2024-02-21T23:06:02.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.454+0000 7fd474c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd47015a0c0 con 0x7fd470154bb0 2024-02-21T23:06:02.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.455+0000 7fd46fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd470152c60 0x7fd4701ef250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:02.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.455+0000 7fd46fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd470152c60 0x7fd4701ef250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54204/0 (socket says 172.21.15.28:54204) 2024-02-21T23:06:02.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.455+0000 7fd46fe00640 1 -- 172.21.15.28:0/843808767 learned_addr learned my addr 172.21.15.28:0/843808767 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:02.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.455+0000 7fd466a00640 1 --2- 172.21.15.28:0/843808767 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd470154bb0 0x7fd4701eed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:02.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.455+0000 7fd46f400640 1 --2- 172.21.15.28:0/843808767 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd4540039b0 0x7fd47014ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:02.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.455+0000 7fd46fe00640 1 -- 172.21.15.28:0/843808767 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd470154bb0 msgr2=0x7fd4701eed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:02.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.455+0000 7fd46fe00640 1 --2- 172.21.15.28:0/843808767 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd470154bb0 0x7fd4701eed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:02.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.455+0000 7fd46fe00640 1 -- 172.21.15.28:0/843808767 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd4540039b0 msgr2=0x7fd47014ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:02.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.455+0000 7fd46fe00640 1 --2- 172.21.15.28:0/843808767 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd4540039b0 0x7fd47014ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:02.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.455+0000 7fd46fe00640 1 -- 172.21.15.28:0/843808767 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd470154560 con 0x7fd470152c60 2024-02-21T23:06:02.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.455+0000 7fd466a00640 1 --2- 172.21.15.28:0/843808767 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd470154bb0 0x7fd4701eed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:06:02.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.455+0000 7fd46f400640 1 --2- 172.21.15.28:0/843808767 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd4540039b0 0x7fd47014ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:06:02.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.456+0000 7fd46fe00640 1 --2- 172.21.15.28:0/843808767 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd470152c60 0x7fd4701ef250 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fd470159690 tx=0x7fd468010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:02.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.456+0000 7fd467e00640 1 -- 172.21.15.28:0/843808767 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd468010d80 con 0x7fd470152c60 2024-02-21T23:06:02.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.456+0000 7fd467e00640 1 -- 172.21.15.28:0/843808767 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd468003990 con 0x7fd470152c60 2024-02-21T23:06:02.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.456+0000 7fd467e00640 1 -- 172.21.15.28:0/843808767 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd468003dc0 con 0x7fd470152c60 2024-02-21T23:06:02.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.456+0000 7fd474c00640 1 -- 172.21.15.28:0/843808767 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd470154840 con 0x7fd470152c60 2024-02-21T23:06:02.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.456+0000 7fd474c00640 1 -- 172.21.15.28:0/843808767 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd47014f140 con 0x7fd470152c60 2024-02-21T23:06:02.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.457+0000 7fd474c00640 1 -- 172.21.15.28:0/843808767 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd47004f290 con 0x7fd470152c60 2024-02-21T23:06:02.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.458+0000 7fd467e00640 1 -- 172.21.15.28:0/843808767 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd468005d10 con 0x7fd470152c60 2024-02-21T23:06:02.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.460+0000 7fd467e00640 1 --2- 172.21.15.28:0/843808767 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd44c06b4d0 0x7fd44c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:02.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.460+0000 7fd467e00640 1 -- 172.21.15.28:0/843808767 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd46808ab70 con 0x7fd470152c60 2024-02-21T23:06:02.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.460+0000 7fd46f400640 1 --2- 172.21.15.28:0/843808767 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd44c06b4d0 0x7fd44c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:02.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.462+0000 7fd46f400640 1 --2- 172.21.15.28:0/843808767 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd44c06b4d0 0x7fd44c06d990 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fd460034000 tx=0x7fd460007000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:02.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.463+0000 7fd467e00640 1 -- 172.21.15.28:0/843808767 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd468025020 con 0x7fd470152c60 2024-02-21T23:06:02.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.704+0000 7fd474c00640 1 -- 172.21.15.28:0/843808767 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd47006f4b0 con 0x7fd470152c60 2024-02-21T23:06:02.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.706+0000 7fd467e00640 1 -- 172.21.15.28:0/843808767 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+4934 (secure 0 0 0) 0x7fd4680597c0 con 0x7fd470152c60 2024-02-21T23:06:02.694 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:06:02.694 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":62,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5288,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/2774951107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2774951107},{"type":"v1","addr":"172.21.15.81:6835","nonce":2774951107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":57},{"gid":5294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/4292940632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":4292940632},{"type":"v1","addr":"172.21.15.81:6837","nonce":4292940632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":57},{"gid":5539,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/624384999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":624384999},{"type":"v1","addr":"172.21.15.28:6833","nonce":624384999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:05:50.828988+0000","modified":"2024-02-21T23:05:53.937407+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6366},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6366":{"gid":6366,"name":"a","rank":0,"incarnation":58,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/595682298","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":595682298},{"type":"v1","addr":"172.21.15.28:6835","nonce":595682298}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-02-21T23:06:02.695 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 62 2024-02-21T23:06:02.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.708+0000 7fd474c00640 1 -- 172.21.15.28:0/843808767 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd44c06b4d0 msgr2=0x7fd44c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:02.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.708+0000 7fd474c00640 1 --2- 172.21.15.28:0/843808767 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd44c06b4d0 0x7fd44c06d990 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fd460034000 tx=0x7fd460007000 comp rx=0 tx=0).stop 2024-02-21T23:06:02.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.708+0000 7fd474c00640 1 -- 172.21.15.28:0/843808767 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd470152c60 msgr2=0x7fd4701ef250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:02.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.708+0000 7fd474c00640 1 --2- 172.21.15.28:0/843808767 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd470152c60 0x7fd4701ef250 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fd470159690 tx=0x7fd468010590 comp rx=0 tx=0).stop 2024-02-21T23:06:02.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.708+0000 7fd474c00640 1 -- 172.21.15.28:0/843808767 shutdown_connections 2024-02-21T23:06:02.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.708+0000 7fd474c00640 1 --2- 172.21.15.28:0/843808767 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd44c06b4d0 0x7fd44c06d990 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:02.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.708+0000 7fd474c00640 1 --2- 172.21.15.28:0/843808767 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd470154bb0 0x7fd4701eed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:02.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.708+0000 7fd474c00640 1 --2- 172.21.15.28:0/843808767 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd470152c60 0x7fd4701ef250 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:02.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.708+0000 7fd474c00640 1 --2- 172.21.15.28:0/843808767 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd4540039b0 0x7fd47014ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:02.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.708+0000 7fd474c00640 1 -- 172.21.15.28:0/843808767 >> 172.21.15.28:0/843808767 conn(0x7fd4700c7720 msgr2=0x7fd470070630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:02.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.709+0000 7fd474c00640 1 -- 172.21.15.28:0/843808767 shutdown_connections 2024-02-21T23:06:02.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:02.709+0000 7fd474c00640 1 -- 172.21.15.28:0/843808767 wait complete. 2024-02-21T23:06:03.119 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 62, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:05:50.828988+0000', 'modified': '2024-02-21T23:05:53.937407+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6366}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6366': {'gid': 6366, 'name': 'a', 'rank': 0, 'incarnation': 58, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/595682298', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 595682298}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 595682298}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [12], 'metadata_pool': 11, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:06:03.119 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-21T23:06:03.119 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-02-21T23:06:03.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.381+0000 7f6c1e200640 1 Processor -- start 2024-02-21T23:06:03.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.381+0000 7f6c1e200640 1 -- start start 2024-02-21T23:06:03.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.383+0000 7f6c1e200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6c180c2c70 0x7f6c180723f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:03.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.383+0000 7f6c1e200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c180acd50 con 0x7f6c181538d0 2024-02-21T23:06:03.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.383+0000 7f6c1e200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c18152df0 con 0x7f6c180ac0c0 2024-02-21T23:06:03.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.383+0000 7f6c1e200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c180ad030 con 0x7f6c180c2c70 2024-02-21T23:06:03.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.383+0000 7f6c17e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f6c181538d0 0x7f6c180c18d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43872/0 (socket says 172.21.15.28:43872) 2024-02-21T23:06:03.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.383+0000 7f6c17e00640 1 -- 172.21.15.28:0/3320715792 learned_addr learned my addr 172.21.15.28:0/3320715792 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:06:03.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.383+0000 7f6c1ce00640 1 --2- 172.21.15.28:0/3320715792 >> v2:172.21.15.28:3300/0 conn(0x7f6c180c2c70 0x7f6c180723f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:03.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.384+0000 7f6c17400640 1 -- 172.21.15.28:0/3320715792 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1892556712 0 0) 0x7f6c180acd50 con 0x7f6c181538d0 2024-02-21T23:06:03.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.384+0000 7f6c17400640 1 -- 172.21.15.28:0/3320715792 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c00003520 con 0x7f6c181538d0 2024-02-21T23:06:03.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.384+0000 7f6c1ce00640 1 -- 172.21.15.28:0/3320715792 >> v1:172.21.15.81:6789/0 conn(0x7f6c180ac0c0 legacy=0x7f6c1814c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:03.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.384+0000 7f6c1ce00640 1 -- 172.21.15.28:0/3320715792 >> v1:172.21.15.28:6789/0 conn(0x7f6c181538d0 legacy=0x7f6c180c18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:03.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.384+0000 7f6c1ce00640 1 -- 172.21.15.28:0/3320715792 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c181530d0 con 0x7f6c180c2c70 2024-02-21T23:06:03.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.385+0000 7f6c1ce00640 1 --2- 172.21.15.28:0/3320715792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c180c2c70 0x7f6c180723f0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c009d90 tx=0x7f6c0c02bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1709d4fcb10e0dc server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:03.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.386+0000 7f6c17400640 1 -- 172.21.15.28:0/3320715792 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c0c039070 con 0x7f6c180c2c70 2024-02-21T23:06:03.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.386+0000 7f6c17400640 1 -- 172.21.15.28:0/3320715792 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c0c02ccf0 con 0x7f6c180c2c70 2024-02-21T23:06:03.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.386+0000 7f6c17400640 1 -- 172.21.15.28:0/3320715792 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c0c034640 con 0x7f6c180c2c70 2024-02-21T23:06:03.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.386+0000 7f6c1e200640 1 -- 172.21.15.28:0/3320715792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c180c2c70 msgr2=0x7f6c180723f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:03.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.386+0000 7f6c1e200640 1 --2- 172.21.15.28:0/3320715792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c180c2c70 0x7f6c180723f0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c009d90 tx=0x7f6c0c02bf40 comp rx=0 tx=0).stop 2024-02-21T23:06:03.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.386+0000 7f6c1e200640 1 -- 172.21.15.28:0/3320715792 shutdown_connections 2024-02-21T23:06:03.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.386+0000 7f6c1e200640 1 --2- 172.21.15.28:0/3320715792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c180c2c70 0x7f6c180723f0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:03.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.386+0000 7f6c1e200640 1 -- 172.21.15.28:0/3320715792 >> 172.21.15.28:0/3320715792 conn(0x7f6c180718d0 msgr2=0x7f6c18071ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:03.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.387+0000 7f6c1e200640 1 -- 172.21.15.28:0/3320715792 shutdown_connections 2024-02-21T23:06:03.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.387+0000 7f6c1e200640 1 -- 172.21.15.28:0/3320715792 wait complete. 2024-02-21T23:06:03.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.388+0000 7f6c1e200640 1 Processor -- start 2024-02-21T23:06:03.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.388+0000 7f6c1e200640 1 -- start start 2024-02-21T23:06:03.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.389+0000 7f6c1e200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6c180ac0c0 0x7f6c180c0570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:03.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.389+0000 7f6c1e200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6c180c2c70 0x7f6c1814e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:03.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.389+0000 7f6c1e200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c181538d0 0x7f6c18149510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:03.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.389+0000 7f6c1e200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c180c1fe0 con 0x7f6c181538d0 2024-02-21T23:06:03.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.389+0000 7f6c1e200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c18153440 con 0x7f6c180ac0c0 2024-02-21T23:06:03.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.389+0000 7f6c1e200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c180c6bd0 con 0x7f6c180c2c70 2024-02-21T23:06:03.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.390+0000 7f6c1d800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c181538d0 0x7f6c18149510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:03.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.390+0000 7f6c1d800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c181538d0 0x7f6c18149510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54228/0 (socket says 172.21.15.28:54228) 2024-02-21T23:06:03.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.390+0000 7f6c1d800640 1 -- 172.21.15.28:0/4025165896 learned_addr learned my addr 172.21.15.28:0/4025165896 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:03.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.390+0000 7f6c1ce00640 1 --2- 172.21.15.28:0/4025165896 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6c180ac0c0 0x7f6c180c0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:03.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.390+0000 7f6c17e00640 1 --2- 172.21.15.28:0/4025165896 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6c180c2c70 0x7f6c1814e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:03.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.390+0000 7f6c1d800640 1 -- 172.21.15.28:0/4025165896 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6c180c2c70 msgr2=0x7f6c1814e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:03.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.390+0000 7f6c1d800640 1 --2- 172.21.15.28:0/4025165896 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6c180c2c70 0x7f6c1814e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:03.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.390+0000 7f6c1d800640 1 -- 172.21.15.28:0/4025165896 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6c180ac0c0 msgr2=0x7f6c180c0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:03.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.390+0000 7f6c1d800640 1 --2- 172.21.15.28:0/4025165896 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6c180ac0c0 0x7f6c180c0570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:03.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.390+0000 7f6c1d800640 1 -- 172.21.15.28:0/4025165896 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c181ee9f0 con 0x7f6c181538d0 2024-02-21T23:06:03.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.390+0000 7f6c1ce00640 1 --2- 172.21.15.28:0/4025165896 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6c180ac0c0 0x7f6c180c0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:06:03.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.391+0000 7f6c1d800640 1 --2- 172.21.15.28:0/4025165896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c181538d0 0x7f6c18149510 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f6c1000c3f0 tx=0x7f6c1000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:03.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.391+0000 7f6c14c00640 1 -- 172.21.15.28:0/4025165896 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c100052d0 con 0x7f6c181538d0 2024-02-21T23:06:03.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.391+0000 7f6c14c00640 1 -- 172.21.15.28:0/4025165896 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c10005490 con 0x7f6c181538d0 2024-02-21T23:06:03.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.391+0000 7f6c1e200640 1 -- 172.21.15.28:0/4025165896 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c180c6610 con 0x7f6c181538d0 2024-02-21T23:06:03.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.391+0000 7f6c14c00640 1 -- 172.21.15.28:0/4025165896 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c10007260 con 0x7f6c181538d0 2024-02-21T23:06:03.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.391+0000 7f6c1e200640 1 -- 172.21.15.28:0/4025165896 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c180aa100 con 0x7f6c181538d0 2024-02-21T23:06:03.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.393+0000 7f6c14c00640 1 -- 172.21.15.28:0/4025165896 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6c10007400 con 0x7f6c181538d0 2024-02-21T23:06:03.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.394+0000 7f6c14c00640 1 --2- 172.21.15.28:0/4025165896 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6bf806b4d0 0x7f6bf806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:03.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.394+0000 7f6c14c00640 1 -- 172.21.15.28:0/4025165896 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6c10089830 con 0x7f6c181538d0 2024-02-21T23:06:03.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.395+0000 7f6c227e0740 1 Processor -- start 2024-02-21T23:06:03.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.395+0000 7f6c1ce00640 1 --2- 172.21.15.28:0/4025165896 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6bf806b4d0 0x7f6bf806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:03.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.395+0000 7f6c227e0740 1 -- start start 2024-02-21T23:06:03.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.396+0000 7f6c227e0740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5589e9767020 0x5589e9767400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:03.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.396+0000 7f6c1ce00640 1 --2- 172.21.15.28:0/4025165896 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6bf806b4d0 0x7f6bf806d990 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c036000 tx=0x7f6c0c030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:03.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.396+0000 7f6c227e0740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589e9768330 0x5589e97a3170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:03.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.396+0000 7f6c227e0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589e97a36b0 0x5589e97a1720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:03.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.396+0000 7f6c227e0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589e96cb470 con 0x5589e97a36b0 2024-02-21T23:06:03.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.396+0000 7f6c227e0740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589e9674af0 con 0x5589e9768330 2024-02-21T23:06:03.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.396+0000 7f6c227e0740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x5589e96c7510 con 0x5589e9767020 2024-02-21T23:06:03.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.396+0000 7f6c1d800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589e97a36b0 0x5589e97a1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:03.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.396+0000 7f6c1d800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589e97a36b0 0x5589e97a1720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54244/0 (socket says 172.21.15.28:54244) 2024-02-21T23:06:03.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.396+0000 7f6c1d800640 1 -- 172.21.15.28:0/1901056813 learned_addr learned my addr 172.21.15.28:0/1901056813 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:03.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.397+0000 7f6c1ce00640 1 --2- 172.21.15.28:0/1901056813 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5589e9767020 0x5589e9767400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:03.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.397+0000 7f6c17e00640 1 --2- 172.21.15.28:0/1901056813 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589e9768330 0x5589e97a3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:03.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.397+0000 7f6c1d800640 1 -- 172.21.15.28:0/1901056813 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5589e9767020 msgr2=0x5589e9767400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:03.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.397+0000 7f6c1d800640 1 --2- 172.21.15.28:0/1901056813 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5589e9767020 0x5589e9767400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:03.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.397+0000 7f6c1d800640 1 -- 172.21.15.28:0/1901056813 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589e9768330 msgr2=0x5589e97a3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:03.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.397+0000 7f6c1d800640 1 --2- 172.21.15.28:0/1901056813 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589e9768330 0x5589e97a3170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:03.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.397+0000 7f6c1d800640 1 -- 172.21.15.28:0/1901056813 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5589e967bd90 con 0x5589e97a36b0 2024-02-21T23:06:03.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.397+0000 7f6c17e00640 1 --2- 172.21.15.28:0/1901056813 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589e9768330 0x5589e97a3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:06:03.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.397+0000 7f6c1d800640 1 --2- 172.21.15.28:0/1901056813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589e97a36b0 0x5589e97a1720 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f6c10087b00 tx=0x7f6c10000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbab6a0016ce0567 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:03.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.397+0000 7f6c09600640 1 -- 172.21.15.28:0/1901056813 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c10060520 con 0x5589e97a36b0 2024-02-21T23:06:03.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.397+0000 7f6c09600640 1 -- 172.21.15.28:0/1901056813 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c10056110 con 0x5589e97a36b0 2024-02-21T23:06:03.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.397+0000 7f6c09600640 1 -- 172.21.15.28:0/1901056813 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c10054480 con 0x5589e97a36b0 2024-02-21T23:06:03.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.398+0000 7f6c227e0740 1 -- 172.21.15.28:0/1901056813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589e97a36b0 msgr2=0x5589e97a1720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:03.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.398+0000 7f6c227e0740 1 --2- 172.21.15.28:0/1901056813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589e97a36b0 0x5589e97a1720 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f6c10087b00 tx=0x7f6c10000bc0 comp rx=0 tx=0).stop 2024-02-21T23:06:03.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.398+0000 7f6c227e0740 1 -- 172.21.15.28:0/1901056813 shutdown_connections 2024-02-21T23:06:03.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.398+0000 7f6c227e0740 1 --2- 172.21.15.28:0/1901056813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589e97a36b0 0x5589e97a1720 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:03.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.398+0000 7f6c227e0740 1 --2- 172.21.15.28:0/1901056813 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589e9768330 0x5589e97a3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:03.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.398+0000 7f6c227e0740 1 --2- 172.21.15.28:0/1901056813 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5589e9767020 0x5589e9767400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:03.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.398+0000 7f6c227e0740 1 -- 172.21.15.28:0/1901056813 >> 172.21.15.28:0/1901056813 conn(0x5589e9763970 msgr2=0x5589e9788380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:03.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.398+0000 7f6c227e0740 1 -- 172.21.15.28:0/1901056813 shutdown_connections 2024-02-21T23:06:03.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.398+0000 7f6c227e0740 1 -- 172.21.15.28:0/1901056813 wait complete. 2024-02-21T23:06:03.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.399+0000 7f6c227e0740 1 -- start start 2024-02-21T23:06:03.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.399+0000 7f6c227e0740 1 Processor -- start 2024-02-21T23:06:03.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.400+0000 7f6c227e0740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5589e9768330 0x5589e97b6b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:03.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.400+0000 7f6c227e0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589e97a36b0 0x5589e9789e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:03.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.400+0000 7f6c227e0740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589e9786fe0 0x5589e9787460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:03.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.400+0000 7f6c227e0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589e96cb470 con 0x5589e97a36b0 2024-02-21T23:06:03.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.400+0000 7f6c227e0740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589e966a040 con 0x5589e9786fe0 2024-02-21T23:06:03.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.400+0000 7f6c227e0740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x5589e97a8f80 con 0x5589e9768330 2024-02-21T23:06:03.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.400+0000 7f6c227e0740 10 client.0 _ll_register_callbacks cb 0x5589e9687920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:06:03.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.401+0000 7f6c17e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589e97a36b0 0x5589e9789e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:03.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.401+0000 7f6c17e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589e97a36b0 0x5589e9789e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54246/0 (socket says 172.21.15.28:54246) 2024-02-21T23:06:03.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.401+0000 7f6c17e00640 1 -- 172.21.15.28:0/877656768 learned_addr learned my addr 172.21.15.28:0/877656768 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:03.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.401+0000 7f6c1ce00640 1 --2- 172.21.15.28:0/877656768 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5589e9768330 0x5589e97b6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:03.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.401+0000 7f6c1d800640 1 --2- 172.21.15.28:0/877656768 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589e9786fe0 0x5589e9787460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:03.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.401+0000 7f6c17e00640 1 -- 172.21.15.28:0/877656768 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5589e9768330 msgr2=0x5589e97b6b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:03.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.401+0000 7f6c17e00640 1 --2- 172.21.15.28:0/877656768 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5589e9768330 0x5589e97b6b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:03.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.401+0000 7f6c17e00640 1 -- 172.21.15.28:0/877656768 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589e9786fe0 msgr2=0x5589e9787460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:03.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.401+0000 7f6c17e00640 1 --2- 172.21.15.28:0/877656768 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589e9786fe0 0x5589e9787460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:03.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.401+0000 7f6c17e00640 1 -- 172.21.15.28:0/877656768 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5589e9689aa0 con 0x5589e97a36b0 2024-02-21T23:06:03.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.402+0000 7f6c17e00640 1 --2- 172.21.15.28:0/877656768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589e97a36b0 0x5589e9789e90 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f6c04001e60 tx=0x7f6c0400f660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3d5aa6ef20393 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:03.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.402+0000 7f6bfe000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:06:03.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.402+0000 7f6bfe000640 1 -- 172.21.15.28:0/877656768 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c0400fe90 con 0x5589e97a36b0 2024-02-21T23:06:03.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.402+0000 7f6c227e0740 1 -- 172.21.15.28:0/877656768 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6c10056110 con 0x5589e97a36b0 2024-02-21T23:06:03.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.402+0000 7f6bfe000640 1 -- 172.21.15.28:0/877656768 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(76..76 src has 1..76) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6c04005ac0 con 0x5589e97a36b0 2024-02-21T23:06:03.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.402+0000 7f6bfe000640 1 -- 172.21.15.28:0/877656768 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6c04020050 con 0x5589e97a36b0 2024-02-21T23:06:03.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.402+0000 7f6c227e0740 10 client.6528 fetch_fsmap learned FSMap version 62 2024-02-21T23:06:03.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.402+0000 7f6c227e0740 1 -- 172.21.15.28:0/877656768 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=2+}) v3 -- 0x5589e93e02d0 con 0x5589e97a36b0 2024-02-21T23:06:03.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.402+0000 7f6bfe000640 1 -- 172.21.15.28:0/877656768 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c10056110 con 0x5589e97a36b0 2024-02-21T23:06:03.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.403+0000 7f6bfe000640 1 -- 172.21.15.28:0/877656768 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 62) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f6c04023020 con 0x5589e97a36b0 2024-02-21T23:06:03.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.403+0000 7f6c227e0740 10 client.6528 fetch_fsmap finished waiting for FSMap version 62 2024-02-21T23:06:03.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.403+0000 7f6c227e0740 10 client.6528 resolve_mds: resolved name 'a' to daemon mds.a 2024-02-21T23:06:03.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.403+0000 7f6c227e0740 20 client.6528 populate_metadata read hostname 'smithi028' 2024-02-21T23:06:03.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.403+0000 7f6c227e0740 1 --2- 172.21.15.28:0/877656768 >> [v2:172.21.15.28:6834/595682298,v1:172.21.15.28:6835/595682298] conn(0x5589e97b38e0 0x5589e97b3d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:03.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.403+0000 7f6c227e0740 4 client.6528 mds_command: new command op to 6366 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-21T23:06:03.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.403+0000 7f6c227e0740 1 -- 172.21.15.28:0/877656768 --> [v2:172.21.15.28:6834/595682298,v1:172.21.15.28:6835/595682298] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5589e96d89b0 con 0x5589e97b38e0 2024-02-21T23:06:03.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.403+0000 7f6c1ce00640 1 --2- 172.21.15.28:0/877656768 >> [v2:172.21.15.28:6834/595682298,v1:172.21.15.28:6835/595682298] conn(0x5589e97b38e0 0x5589e97b3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:03.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.404+0000 7f6c1ce00640 1 --2- 172.21.15.28:0/877656768 >> [v2:172.21.15.28:6834/595682298,v1:172.21.15.28:6835/595682298] conn(0x5589e97b38e0 0x5589e97b3d00 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c4456e76439ba9ec server_cookie=23d487899ccb324c in_seq=0 out_seq=0 2024-02-21T23:06:03.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.404+0000 7f6bfe000640 10 client.6528 ms_handle_connect on v2:172.21.15.28:6834/595682298 2024-02-21T23:06:03.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.405+0000 7f6bfe000640 1 -- 172.21.15.28:0/877656768 <== mds.0 v2:172.21.15.28:6834/595682298 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5589e96d89b0 con 0x5589e97b38e0 2024-02-21T23:06:03.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.405+0000 7f6bfe000640 10 client.6528 handle_command_reply: tid=0 2024-02-21T23:06:03.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.405+0000 7f6c227e0740 1 client.6528 shutdown 2024-02-21T23:06:03.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.405+0000 7f6c227e0740 1 -- 172.21.15.28:0/877656768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589e97a36b0 msgr2=0x5589e9789e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:03.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.405+0000 7f6c227e0740 1 --2- 172.21.15.28:0/877656768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589e97a36b0 0x5589e9789e90 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f6c04001e60 tx=0x7f6c0400f660 comp rx=0 tx=0).stop 2024-02-21T23:06:03.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.405+0000 7f6c227e0740 1 -- 172.21.15.28:0/877656768 shutdown_connections 2024-02-21T23:06:03.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.405+0000 7f6c227e0740 1 --2- 172.21.15.28:0/877656768 >> [v2:172.21.15.28:6834/595682298,v1:172.21.15.28:6835/595682298] conn(0x5589e97b38e0 0x5589e97b3d00 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:03.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.405+0000 7f6c227e0740 1 --2- 172.21.15.28:0/877656768 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589e9786fe0 0x5589e9787460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:03.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.405+0000 7f6c227e0740 1 --2- 172.21.15.28:0/877656768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589e97a36b0 0x5589e9789e90 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:03.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.405+0000 7f6bfe000640 0 client.6528 ms_handle_reset on v2:172.21.15.28:6834/595682298 2024-02-21T23:06:03.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.405+0000 7f6c227e0740 1 --2- 172.21.15.28:0/877656768 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5589e9768330 0x5589e97b6b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:03.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.405+0000 7f6c227e0740 1 -- 172.21.15.28:0/877656768 >> 172.21.15.28:0/877656768 conn(0x5589e9767020 msgr2=0x5589e97a6790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:03.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.405+0000 7f6c227e0740 1 -- 172.21.15.28:0/877656768 shutdown_connections 2024-02-21T23:06:03.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.405+0000 7f6c227e0740 1 -- 172.21.15.28:0/877656768 wait complete. 2024-02-21T23:06:03.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.405+0000 7f6c227e0740 20 client.6528 trim_cache size 0 max 16384 2024-02-21T23:06:03.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.443+0000 7f6c227e0740 1 Processor -- start 2024-02-21T23:06:03.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.443+0000 7f6c227e0740 1 -- start start 2024-02-21T23:06:03.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.445+0000 7f6c227e0740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589e9768330 0x5589e97b0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:03.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.445+0000 7f6c227e0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589e9786fe0 0x5589e9787670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:03.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.445+0000 7f6c227e0740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5589e97a36b0 0x5589e97af140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:03.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.445+0000 7f6c227e0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c181f6b30 con 0x5589e9786fe0 2024-02-21T23:06:03.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.445+0000 7f6c227e0740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c181eb4d0 con 0x5589e9768330 2024-02-21T23:06:03.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.445+0000 7f6c227e0740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c181f6530 con 0x5589e97a36b0 2024-02-21T23:06:03.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.446+0000 7f6c1ce00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589e9768330 0x5589e97b0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:03.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.446+0000 7f6c1d800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5589e97a36b0 0x5589e97af140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:03.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.446+0000 7f6c1ce00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589e9768330 0x5589e97b0b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.28:51788/0 (socket says 172.21.15.28:51788) 2024-02-21T23:06:03.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.446+0000 7f6c1ce00640 1 -- 172.21.15.28:0/764033344 learned_addr learned my addr 172.21.15.28:0/764033344 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:03.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.446+0000 7f6c17e00640 1 --2- 172.21.15.28:0/764033344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589e9786fe0 0x5589e9787670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:03.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.446+0000 7f6c1ce00640 1 -- 172.21.15.28:0/764033344 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5589e97a36b0 msgr2=0x5589e97af140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:03.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.446+0000 7f6c1ce00640 1 --2- 172.21.15.28:0/764033344 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5589e97a36b0 0x5589e97af140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:03.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.446+0000 7f6c1ce00640 1 -- 172.21.15.28:0/764033344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589e9786fe0 msgr2=0x5589e9787670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:03.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.446+0000 7f6c1ce00640 1 --2- 172.21.15.28:0/764033344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589e9786fe0 0x5589e9787670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:03.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.446+0000 7f6c1ce00640 1 -- 172.21.15.28:0/764033344 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5589e94aa6b0 con 0x5589e9768330 2024-02-21T23:06:03.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.446+0000 7f6c17e00640 1 --2- 172.21.15.28:0/764033344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589e9786fe0 0x5589e9787670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-21T23:06:03.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.447+0000 7f6c1ce00640 1 --2- 172.21.15.28:0/764033344 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589e9768330 0x5589e97b0b40 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c02c550 tx=0x7f6c0c02e000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c2ec0fc32456d76 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:03.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.447+0000 7f6c08c00640 1 -- 172.21.15.28:0/764033344 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c0c039050 con 0x5589e9768330 2024-02-21T23:06:03.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.447+0000 7f6c08c00640 1 -- 172.21.15.28:0/764033344 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c0c03a040 con 0x5589e9768330 2024-02-21T23:06:03.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.447+0000 7f6c08c00640 1 -- 172.21.15.28:0/764033344 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c0c053870 con 0x5589e9768330 2024-02-21T23:06:03.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.448+0000 7f6c227e0740 1 -- 172.21.15.28:0/764033344 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589e9768330 msgr2=0x5589e97b0b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:03.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.448+0000 7f6c227e0740 1 --2- 172.21.15.28:0/764033344 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589e9768330 0x5589e97b0b40 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c02c550 tx=0x7f6c0c02e000 comp rx=0 tx=0).stop 2024-02-21T23:06:03.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.448+0000 7f6c227e0740 1 -- 172.21.15.28:0/764033344 shutdown_connections 2024-02-21T23:06:03.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.448+0000 7f6c227e0740 1 --2- 172.21.15.28:0/764033344 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5589e97a36b0 0x5589e97af140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:03.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.448+0000 7f6c227e0740 1 --2- 172.21.15.28:0/764033344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589e9786fe0 0x5589e9787670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:03.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.448+0000 7f6c227e0740 1 --2- 172.21.15.28:0/764033344 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589e9768330 0x5589e97b0b40 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:03.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.448+0000 7f6c227e0740 1 -- 172.21.15.28:0/764033344 >> 172.21.15.28:0/764033344 conn(0x5589e9767020 msgr2=0x5589e9789d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:03.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.448+0000 7f6c227e0740 1 -- 172.21.15.28:0/764033344 shutdown_connections 2024-02-21T23:06:03.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.448+0000 7f6c227e0740 1 -- 172.21.15.28:0/764033344 wait complete. 2024-02-21T23:06:03.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.449+0000 7f6c227e0740 1 -- start start 2024-02-21T23:06:03.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.449+0000 7f6c227e0740 1 Processor -- start 2024-02-21T23:06:03.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.450+0000 7f6c227e0740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589e9768330 0x5589e97a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:03.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.450+0000 7f6c227e0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589e9786fe0 0x5589e98125e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:03.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.450+0000 7f6c227e0740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5589e97a36b0 0x5589e980f850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:03.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.450+0000 7f6c227e0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589e97837b0 con 0x5589e9786fe0 2024-02-21T23:06:03.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.450+0000 7f6c227e0740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589e9788230 con 0x5589e9768330 2024-02-21T23:06:03.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.450+0000 7f6c227e0740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x5589e9775500 con 0x5589e97a36b0 2024-02-21T23:06:03.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.450+0000 7f6c227e0740 10 client.0 _ll_register_callbacks cb 0x7f6c04001a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:06:03.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.450+0000 7f6c17e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589e9786fe0 0x5589e98125e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:03.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.450+0000 7f6c17e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589e9786fe0 0x5589e98125e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54262/0 (socket says 172.21.15.28:54262) 2024-02-21T23:06:03.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.450+0000 7f6c17e00640 1 -- 172.21.15.28:0/3279715895 learned_addr learned my addr 172.21.15.28:0/3279715895 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:03.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.451+0000 7f6c1ce00640 1 --2- 172.21.15.28:0/3279715895 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589e9768330 0x5589e97a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:03.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.451+0000 7f6c1d800640 1 --2- 172.21.15.28:0/3279715895 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5589e97a36b0 0x5589e980f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:03.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.451+0000 7f6c17e00640 1 -- 172.21.15.28:0/3279715895 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5589e97a36b0 msgr2=0x5589e980f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:03.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.451+0000 7f6c17e00640 1 --2- 172.21.15.28:0/3279715895 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5589e97a36b0 0x5589e980f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:03.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.451+0000 7f6c17e00640 1 -- 172.21.15.28:0/3279715895 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589e9768330 msgr2=0x5589e97a50c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:03.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.451+0000 7f6c17e00640 1 --2- 172.21.15.28:0/3279715895 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589e9768330 0x5589e97a50c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:03.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.451+0000 7f6c17e00640 1 -- 172.21.15.28:0/3279715895 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5589e9782e20 con 0x5589e9786fe0 2024-02-21T23:06:03.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.451+0000 7f6c1ce00640 1 --2- 172.21.15.28:0/3279715895 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589e9768330 0x5589e97a50c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:06:03.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.451+0000 7f6c1d800640 1 --2- 172.21.15.28:0/3279715895 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5589e97a36b0 0x5589e980f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:06:03.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.451+0000 7f6c17e00640 1 --2- 172.21.15.28:0/3279715895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589e9786fe0 0x5589e98125e0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f6c04000ea0 tx=0x7f6c04001150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fd7c0ba5bfef0ca server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:03.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.451+0000 7f6bfe000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:06:03.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.451+0000 7f6bfe000640 1 -- 172.21.15.28:0/3279715895 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c04005ac0 con 0x5589e9786fe0 2024-02-21T23:06:03.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.451+0000 7f6bfe000640 1 -- 172.21.15.28:0/3279715895 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(76..76 src has 1..76) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6c0402fc60 con 0x5589e9786fe0 2024-02-21T23:06:03.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.451+0000 7f6c227e0740 1 -- 172.21.15.28:0/3279715895 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6c0c03a040 con 0x5589e9786fe0 2024-02-21T23:06:03.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.452+0000 7f6bfe000640 1 -- 172.21.15.28:0/3279715895 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6c0402fe60 con 0x5589e9786fe0 2024-02-21T23:06:03.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.452+0000 7f6c227e0740 10 client.6534 fetch_fsmap learned FSMap version 62 2024-02-21T23:06:03.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.452+0000 7f6c227e0740 1 -- 172.21.15.28:0/3279715895 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=2+}) v3 -- 0x5589e97a2bb0 con 0x5589e9786fe0 2024-02-21T23:06:03.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.452+0000 7f6bfe000640 1 -- 172.21.15.28:0/3279715895 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c0c03a040 con 0x5589e9786fe0 2024-02-21T23:06:03.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.452+0000 7f6bfe000640 1 -- 172.21.15.28:0/3279715895 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 62) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f6c04023040 con 0x5589e9786fe0 2024-02-21T23:06:03.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.452+0000 7f6c227e0740 10 client.6534 fetch_fsmap finished waiting for FSMap version 62 2024-02-21T23:06:03.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.452+0000 7f6c227e0740 10 client.6534 resolve_mds: resolved name 'a' to daemon mds.a 2024-02-21T23:06:03.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.452+0000 7f6c227e0740 20 client.6534 populate_metadata read hostname 'smithi028' 2024-02-21T23:06:03.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.452+0000 7f6c227e0740 1 --2- 172.21.15.28:0/3279715895 >> [v2:172.21.15.28:6834/595682298,v1:172.21.15.28:6835/595682298] conn(0x5589e97a1670 0x5589e97a1a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:03.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.452+0000 7f6c227e0740 4 client.6534 mds_command: new command op to 6366 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-21T23:06:03.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.452+0000 7f6c227e0740 1 -- 172.21.15.28:0/3279715895 --> [v2:172.21.15.28:6834/595682298,v1:172.21.15.28:6835/595682298] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5589e96d2d00 con 0x5589e97a1670 2024-02-21T23:06:03.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.452+0000 7f6c1ce00640 1 --2- 172.21.15.28:0/3279715895 >> [v2:172.21.15.28:6834/595682298,v1:172.21.15.28:6835/595682298] conn(0x5589e97a1670 0x5589e97a1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:03.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.453+0000 7f6bfe000640 10 client.6534 ms_handle_connect on v2:172.21.15.28:6834/595682298 2024-02-21T23:06:03.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.453+0000 7f6c1ce00640 1 --2- 172.21.15.28:0/3279715895 >> [v2:172.21.15.28:6834/595682298,v1:172.21.15.28:6835/595682298] conn(0x5589e97a1670 0x5589e97a1a50 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=538909b1c5943fbd server_cookie=994c4245d5f76306 in_seq=0 out_seq=0 2024-02-21T23:06:03.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.453+0000 7f6bfe000640 1 -- 172.21.15.28:0/3279715895 <== mds.0 v2:172.21.15.28:6834/595682298 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x5589e96d2d00 con 0x5589e97a1670 2024-02-21T23:06:03.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.453+0000 7f6bfe000640 10 client.6534 handle_command_reply: tid=0 2024-02-21T23:06:03.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.453+0000 7f6c227e0740 1 client.6534 shutdown 2024-02-21T23:06:03.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.453+0000 7f6c227e0740 1 -- 172.21.15.28:0/3279715895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589e9786fe0 msgr2=0x5589e98125e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:03.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.453+0000 7f6c227e0740 1 --2- 172.21.15.28:0/3279715895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589e9786fe0 0x5589e98125e0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f6c04000ea0 tx=0x7f6c04001150 comp rx=0 tx=0).stop 2024-02-21T23:06:03.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.454+0000 7f6c227e0740 1 -- 172.21.15.28:0/3279715895 shutdown_connections 2024-02-21T23:06:03.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.454+0000 7f6c227e0740 1 --2- 172.21.15.28:0/3279715895 >> [v2:172.21.15.28:6834/595682298,v1:172.21.15.28:6835/595682298] conn(0x5589e97a1670 0x5589e97a1a50 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:03.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.454+0000 7f6c227e0740 1 --2- 172.21.15.28:0/3279715895 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5589e97a36b0 0x5589e980f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:03.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.454+0000 7f6c227e0740 1 --2- 172.21.15.28:0/3279715895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5589e9786fe0 0x5589e98125e0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:03.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.454+0000 7f6c227e0740 1 --2- 172.21.15.28:0/3279715895 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5589e9768330 0x5589e97a50c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:03.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.454+0000 7f6bfe000640 0 client.6534 ms_handle_reset on v2:172.21.15.28:6834/595682298 2024-02-21T23:06:03.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.454+0000 7f6c227e0740 1 -- 172.21.15.28:0/3279715895 >> 172.21.15.28:0/3279715895 conn(0x5589e9767020 msgr2=0x5589e9789d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:03.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.454+0000 7f6c227e0740 1 -- 172.21.15.28:0/3279715895 shutdown_connections 2024-02-21T23:06:03.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.454+0000 7f6c227e0740 1 -- 172.21.15.28:0/3279715895 wait complete. 2024-02-21T23:06:03.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.454+0000 7f6c227e0740 20 client.6534 trim_cache size 0 max 16384 2024-02-21T23:06:03.447 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-21T23:06:03.447 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac", 2024-02-21T23:06:03.447 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-02-21T23:06:03.447 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 6366, 2024-02-21T23:06:03.447 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-02-21T23:06:03.447 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-02-21T23:06:03.447 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-02-21T23:06:03.447 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 12.052058819000001, 2024-02-21T23:06:03.447 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 62, 2024-02-21T23:06:03.447 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 76, 2024-02-21T23:06:03.447 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 76, 2024-02-21T23:06:03.447 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 17.556801467 2024-02-21T23:06:03.447 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-21T23:06:03.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.455+0000 7f6c1e200640 1 -- 172.21.15.28:0/4025165896 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6bf806b4d0 msgr2=0x7f6bf806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:03.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.455+0000 7f6c1e200640 1 --2- 172.21.15.28:0/4025165896 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6bf806b4d0 0x7f6bf806d990 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c036000 tx=0x7f6c0c030000 comp rx=0 tx=0).stop 2024-02-21T23:06:03.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.455+0000 7f6c1e200640 1 -- 172.21.15.28:0/4025165896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c181538d0 msgr2=0x7f6c18149510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:03.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.455+0000 7f6c1e200640 1 --2- 172.21.15.28:0/4025165896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c181538d0 0x7f6c18149510 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f6c1000c3f0 tx=0x7f6c1000c8b0 comp rx=0 tx=0).stop 2024-02-21T23:06:03.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.455+0000 7f6c1e200640 1 -- 172.21.15.28:0/4025165896 shutdown_connections 2024-02-21T23:06:03.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.455+0000 7f6c1e200640 1 --2- 172.21.15.28:0/4025165896 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6bf806b4d0 0x7f6bf806d990 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:03.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.455+0000 7f6c1e200640 1 --2- 172.21.15.28:0/4025165896 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6c180c2c70 0x7f6c1814e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:03.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.455+0000 7f6c1e200640 1 --2- 172.21.15.28:0/4025165896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c181538d0 0x7f6c18149510 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:03.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.455+0000 7f6c1e200640 1 --2- 172.21.15.28:0/4025165896 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6c180ac0c0 0x7f6c180c0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:03.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.455+0000 7f6c1e200640 1 -- 172.21.15.28:0/4025165896 >> 172.21.15.28:0/4025165896 conn(0x7f6c180718d0 msgr2=0x7f6c1814f3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:03.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.455+0000 7f6c1e200640 1 -- 172.21.15.28:0/4025165896 shutdown_connections 2024-02-21T23:06:03.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.455+0000 7f6c1e200640 1 -- 172.21.15.28:0/4025165896 wait complete. 2024-02-21T23:06:03.462 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:06:03.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.675+0000 7f74a2c00640 1 Processor -- start 2024-02-21T23:06:03.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.676+0000 7f74a2c00640 1 -- start start 2024-02-21T23:06:03.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.677+0000 7f74a2c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f749c0c7800 0x7f749c15ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:03.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.677+0000 7f74a2c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f749c159230 con 0x7f749c156900 2024-02-21T23:06:03.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.677+0000 7f74a2c00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f749c1594f0 con 0x7f749c06ddb0 2024-02-21T23:06:03.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.677+0000 7f74a2c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f749c156ce0 con 0x7f749c0c7800 2024-02-21T23:06:03.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.678+0000 7f74a1800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f749c156900 0x7f749c0c8a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43876/0 (socket says 172.21.15.28:43876) 2024-02-21T23:06:03.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.678+0000 7f74a1800640 1 -- 172.21.15.28:0/1517386474 learned_addr learned my addr 172.21.15.28:0/1517386474 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:06:03.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.678+0000 7f74a2200640 1 --2- 172.21.15.28:0/1517386474 >> v2:172.21.15.28:3300/0 conn(0x7f749c0c7800 0x7f749c15ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:03.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.678+0000 7f749be00640 1 -- 172.21.15.28:0/1517386474 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3187160356 0 0) 0x7f749c159230 con 0x7f749c156900 2024-02-21T23:06:03.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.678+0000 7f749be00640 1 -- 172.21.15.28:0/1517386474 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7484003520 con 0x7f749c156900 2024-02-21T23:06:03.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.678+0000 7f74a2200640 1 -- 172.21.15.28:0/1517386474 >> v1:172.21.15.81:6789/0 conn(0x7f749c06ddb0 legacy=0x7f749c15a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:03.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.678+0000 7f74a2200640 1 -- 172.21.15.28:0/1517386474 >> v1:172.21.15.28:6789/0 conn(0x7f749c156900 legacy=0x7f749c0c8a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:03.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.678+0000 7f74a2200640 1 -- 172.21.15.28:0/1517386474 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f749c1597d0 con 0x7f749c0c7800 2024-02-21T23:06:03.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.679+0000 7f74a2200640 1 --2- 172.21.15.28:0/1517386474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f749c0c7800 0x7f749c15ae30 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f7494009d90 tx=0x7f749402be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d1eb303207ae909 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:03.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.680+0000 7f749be00640 1 -- 172.21.15.28:0/1517386474 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7494039070 con 0x7f749c0c7800 2024-02-21T23:06:03.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.680+0000 7f749be00640 1 -- 172.21.15.28:0/1517386474 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f749402cbd0 con 0x7f749c0c7800 2024-02-21T23:06:03.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.680+0000 7f749be00640 1 -- 172.21.15.28:0/1517386474 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7494034440 con 0x7f749c0c7800 2024-02-21T23:06:03.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.680+0000 7f74a2c00640 1 -- 172.21.15.28:0/1517386474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f749c0c7800 msgr2=0x7f749c15ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:03.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.680+0000 7f74a2c00640 1 --2- 172.21.15.28:0/1517386474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f749c0c7800 0x7f749c15ae30 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f7494009d90 tx=0x7f749402be20 comp rx=0 tx=0).stop 2024-02-21T23:06:03.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.680+0000 7f74a2c00640 1 -- 172.21.15.28:0/1517386474 shutdown_connections 2024-02-21T23:06:03.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.680+0000 7f74a2c00640 1 --2- 172.21.15.28:0/1517386474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f749c0c7800 0x7f749c15ae30 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:03.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.680+0000 7f74a2c00640 1 -- 172.21.15.28:0/1517386474 >> 172.21.15.28:0/1517386474 conn(0x7f749c06b260 msgr2=0x7f749c06b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:03.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.680+0000 7f74a2c00640 1 -- 172.21.15.28:0/1517386474 shutdown_connections 2024-02-21T23:06:03.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.681+0000 7f74a2c00640 1 -- 172.21.15.28:0/1517386474 wait complete. 2024-02-21T23:06:03.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.682+0000 7f74a2c00640 1 Processor -- start 2024-02-21T23:06:03.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.682+0000 7f74a2c00640 1 -- start start 2024-02-21T23:06:03.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.683+0000 7f74a2c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f749c06ddb0 0x7f749c1f21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:03.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.683+0000 7f74a2c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f749c0c7800 0x7f749c1f2730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:03.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.683+0000 7f74a2c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f749c156900 0x7f749c0ab450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:03.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.683+0000 7f74a2c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f749c0c9120 con 0x7f749c0c7800 2024-02-21T23:06:03.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.683+0000 7f74a2c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f749c159e20 con 0x7f749c156900 2024-02-21T23:06:03.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.683+0000 7f74a2c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f749c0c92a0 con 0x7f749c06ddb0 2024-02-21T23:06:03.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.683+0000 7f74a0e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f749c0c7800 0x7f749c1f2730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:03.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.683+0000 7f74a0e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f749c0c7800 0x7f749c1f2730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54276/0 (socket says 172.21.15.28:54276) 2024-02-21T23:06:03.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.683+0000 7f74a0e00640 1 -- 172.21.15.28:0/1676556624 learned_addr learned my addr 172.21.15.28:0/1676556624 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:03.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.684+0000 7f74a2200640 1 --2- 172.21.15.28:0/1676556624 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f749c156900 0x7f749c0ab450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:03.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.684+0000 7f74a1800640 1 --2- 172.21.15.28:0/1676556624 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f749c06ddb0 0x7f749c1f21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:03.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.684+0000 7f74a0e00640 1 -- 172.21.15.28:0/1676556624 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f749c06ddb0 msgr2=0x7f749c1f21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:03.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.684+0000 7f74a0e00640 1 --2- 172.21.15.28:0/1676556624 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f749c06ddb0 0x7f749c1f21f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:03.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.684+0000 7f74a0e00640 1 -- 172.21.15.28:0/1676556624 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f749c156900 msgr2=0x7f749c0ab450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:03.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.684+0000 7f74a0e00640 1 --2- 172.21.15.28:0/1676556624 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f749c156900 0x7f749c0ab450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:03.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.684+0000 7f74a0e00640 1 -- 172.21.15.28:0/1676556624 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f749c1e2d80 con 0x7f749c0c7800 2024-02-21T23:06:03.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.684+0000 7f74a2200640 1 --2- 172.21.15.28:0/1676556624 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f749c156900 0x7f749c0ab450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:06:03.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.684+0000 7f74a1800640 1 --2- 172.21.15.28:0/1676556624 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f749c06ddb0 0x7f749c1f21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:06:03.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.684+0000 7f74a0e00640 1 --2- 172.21.15.28:0/1676556624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f749c0c7800 0x7f749c1f2730 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f748800c400 tx=0x7f748800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:03.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.685+0000 7f7499600640 1 -- 172.21.15.28:0/1676556624 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74880052d0 con 0x7f749c0c7800 2024-02-21T23:06:03.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.685+0000 7f7499600640 1 -- 172.21.15.28:0/1676556624 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7488005490 con 0x7f749c0c7800 2024-02-21T23:06:03.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.685+0000 7f7499600640 1 -- 172.21.15.28:0/1676556624 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7488007260 con 0x7f749c0c7800 2024-02-21T23:06:03.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.685+0000 7f74a2c00640 1 -- 172.21.15.28:0/1676556624 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f749c1f2c70 con 0x7f749c0c7800 2024-02-21T23:06:03.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.685+0000 7f74a2c00640 1 -- 172.21.15.28:0/1676556624 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f749c152e10 con 0x7f749c0c7800 2024-02-21T23:06:03.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.686+0000 7f74a2c00640 1 -- 172.21.15.28:0/1676556624 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f749c04f290 con 0x7f749c0c7800 2024-02-21T23:06:03.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.687+0000 7f7499600640 1 -- 172.21.15.28:0/1676556624 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7488007400 con 0x7f749c0c7800 2024-02-21T23:06:03.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.688+0000 7f7499600640 1 --2- 172.21.15.28:0/1676556624 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f747c06b4d0 0x7f747c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:03.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.688+0000 7f7499600640 1 -- 172.21.15.28:0/1676556624 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f74880898f0 con 0x7f749c0c7800 2024-02-21T23:06:03.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.689+0000 7f74a1800640 1 --2- 172.21.15.28:0/1676556624 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f747c06b4d0 0x7f747c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:03.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.690+0000 7f74a1800640 1 --2- 172.21.15.28:0/1676556624 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f747c06b4d0 0x7f747c06d990 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f74900011e0 tx=0x7f7490009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:03.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.693+0000 7f7499600640 1 -- 172.21.15.28:0/1676556624 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f74880585d0 con 0x7f749c0c7800 2024-02-21T23:06:03.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.935+0000 7f74a2c00640 1 -- 172.21.15.28:0/1676556624 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f749c0c7570 con 0x7f749c0c7800 2024-02-21T23:06:03.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.936+0000 7f7499600640 1 -- 172.21.15.28:0/1676556624 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+4934 (secure 0 0 0) 0x7f7488057ed0 con 0x7f749c0c7800 2024-02-21T23:06:03.925 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:06:03.925 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":62,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5288,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/2774951107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2774951107},{"type":"v1","addr":"172.21.15.81:6835","nonce":2774951107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":57},{"gid":5294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/4292940632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":4292940632},{"type":"v1","addr":"172.21.15.81:6837","nonce":4292940632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":57},{"gid":5539,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/624384999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":624384999},{"type":"v1","addr":"172.21.15.28:6833","nonce":624384999}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:05:50.828988+0000","modified":"2024-02-21T23:05:53.937407+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6366},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6366":{"gid":6366,"name":"a","rank":0,"incarnation":58,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/595682298","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":595682298},{"type":"v1","addr":"172.21.15.28:6835","nonce":595682298}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-02-21T23:06:03.925 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 62 2024-02-21T23:06:03.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.939+0000 7f74a2c00640 1 -- 172.21.15.28:0/1676556624 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f747c06b4d0 msgr2=0x7f747c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:03.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.939+0000 7f74a2c00640 1 --2- 172.21.15.28:0/1676556624 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f747c06b4d0 0x7f747c06d990 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f74900011e0 tx=0x7f7490009040 comp rx=0 tx=0).stop 2024-02-21T23:06:03.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.939+0000 7f74a2c00640 1 -- 172.21.15.28:0/1676556624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f749c0c7800 msgr2=0x7f749c1f2730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:03.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.939+0000 7f74a2c00640 1 --2- 172.21.15.28:0/1676556624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f749c0c7800 0x7f749c1f2730 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f748800c400 tx=0x7f748800c8c0 comp rx=0 tx=0).stop 2024-02-21T23:06:03.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.939+0000 7f74a2c00640 1 -- 172.21.15.28:0/1676556624 shutdown_connections 2024-02-21T23:06:03.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.939+0000 7f74a2c00640 1 --2- 172.21.15.28:0/1676556624 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f747c06b4d0 0x7f747c06d990 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:03.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.939+0000 7f74a2c00640 1 --2- 172.21.15.28:0/1676556624 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f749c156900 0x7f749c0ab450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:03.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.939+0000 7f74a2c00640 1 --2- 172.21.15.28:0/1676556624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f749c0c7800 0x7f749c1f2730 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:03.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.939+0000 7f74a2c00640 1 --2- 172.21.15.28:0/1676556624 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f749c06ddb0 0x7f749c1f21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:03.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.939+0000 7f74a2c00640 1 -- 172.21.15.28:0/1676556624 >> 172.21.15.28:0/1676556624 conn(0x7f749c06b260 msgr2=0x7f749c0a8bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:03.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.939+0000 7f74a2c00640 1 -- 172.21.15.28:0/1676556624 shutdown_connections 2024-02-21T23:06:03.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:03.939+0000 7f74a2c00640 1 -- 172.21.15.28:0/1676556624 wait complete. 2024-02-21T23:06:03.948 INFO:tasks.cephfs.test_failover:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 5, "epoch": 62, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 5, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-02-21T23:05:50.828988+0000", "damaged": [], "data_pools": [ 12 ], "enabled": true, "epoch": 62, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6366": { "addr": "172.21.15.28:6835/595682298", "addrs": { "addrvec": [ { "addr": "172.21.15.28:6834", "nonce": 595682298, "type": "v2" }, { "addr": "172.21.15.28:6835", "nonce": 595682298, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6366, "incarnation": 58, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 11, "modified": "2024-02-21T23:05:53.937407+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6366 } } } ], "standbys": [ { "addr": "172.21.15.81:6835/2774951107", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6834", "nonce": 2774951107, "type": "v2" }, { "addr": "172.21.15.81:6835", "nonce": 2774951107, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 57, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5288, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.81:6837/4292940632", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6836", "nonce": 4292940632, "type": "v2" }, { "addr": "172.21.15.81:6837", "nonce": 4292940632, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 57, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5294, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.28:6833/624384999", "addrs": { "addrvec": [ { "addr": "172.21.15.28:6832", "nonce": 624384999, "type": "v2" }, { "addr": "172.21.15.28:6833", "nonce": 624384999, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 56, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5539, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-02-21T23:06:03.949 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-02-21T23:06:03.949 INFO:tasks.cephfs.test_failover:current = [{'gid': 6366, 'name': 'a', 'rank': 0, 'incarnation': 58, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/595682298', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 595682298}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 595682298}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 5294, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.81:6837/4292940632', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 4292940632}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 4292940632}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 57}, {'gid': 5539, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.28:6833/624384999', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 624384999}, {'type': 'v1', 'addr': '172.21.15.28:6833', 'nonce': 624384999}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 56}, {'gid': 5288, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.81:6835/2774951107', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6834', 'nonce': 2774951107}, {'type': 'v1', 'addr': '172.21.15.81:6835', 'nonce': 2774951107}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 57}] 2024-02-21T23:06:03.949 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.d mds_join_fs cephfs2 2024-02-21T23:06:04.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.184+0000 7fc357600640 1 Processor -- start 2024-02-21T23:06:04.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.184+0000 7fc357600640 1 -- start start 2024-02-21T23:06:04.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.185+0000 7fc357600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc3500c5f50 0x7fc35014fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:04.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.186+0000 7fc357600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc350152fb0 con 0x7fc350150530 2024-02-21T23:06:04.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.186+0000 7fc357600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc350153290 con 0x7fc35006b0e0 2024-02-21T23:06:04.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.186+0000 7fc357600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc350153bc0 con 0x7fc3500c5f50 2024-02-21T23:06:04.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.186+0000 7fc34fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc3500c5f50 0x7fc35014fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:04.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.186+0000 7fc355600640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fc350150530 0x7fc350156500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43884/0 (socket says 172.21.15.28:43884) 2024-02-21T23:06:04.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.186+0000 7fc355600640 1 -- 172.21.15.28:0/1818346850 learned_addr learned my addr 172.21.15.28:0/1818346850 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:06:04.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.187+0000 7fc34fe00640 1 -- 172.21.15.28:0/1818346850 >> v1:172.21.15.81:6789/0 conn(0x7fc35006b0e0 legacy=0x7fc3500c5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:04.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.187+0000 7fc34fe00640 1 -- 172.21.15.28:0/1818346850 >> v1:172.21.15.28:6789/0 conn(0x7fc350150530 legacy=0x7fc350156500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:04.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.187+0000 7fc34f400640 1 -- 172.21.15.28:0/1818346850 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1064247507 0 0) 0x7fc350152fb0 con 0x7fc350150530 2024-02-21T23:06:04.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.187+0000 7fc34fe00640 1 -- 172.21.15.28:0/1818346850 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc350153570 con 0x7fc3500c5f50 2024-02-21T23:06:04.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.188+0000 7fc34fe00640 1 --2- 172.21.15.28:0/1818346850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3500c5f50 0x7fc35014fff0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fc33c0097a0 tx=0x7fc33c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55121cb9dbf0d9a0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:04.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.188+0000 7fc34f400640 1 -- 172.21.15.28:0/1818346850 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc33c035020 con 0x7fc3500c5f50 2024-02-21T23:06:04.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.188+0000 7fc34f400640 1 -- 172.21.15.28:0/1818346850 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc33c031830 con 0x7fc3500c5f50 2024-02-21T23:06:04.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.188+0000 7fc34f400640 1 -- 172.21.15.28:0/1818346850 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc33c031c00 con 0x7fc3500c5f50 2024-02-21T23:06:04.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.189+0000 7fc357600640 1 -- 172.21.15.28:0/1818346850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3500c5f50 msgr2=0x7fc35014fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:04.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.189+0000 7fc357600640 1 --2- 172.21.15.28:0/1818346850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3500c5f50 0x7fc35014fff0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fc33c0097a0 tx=0x7fc33c02b8d0 comp rx=0 tx=0).stop 2024-02-21T23:06:04.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.189+0000 7fc357600640 1 -- 172.21.15.28:0/1818346850 shutdown_connections 2024-02-21T23:06:04.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.189+0000 7fc357600640 1 --2- 172.21.15.28:0/1818346850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3500c5f50 0x7fc35014fff0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:04.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.189+0000 7fc357600640 1 -- 172.21.15.28:0/1818346850 >> 172.21.15.28:0/1818346850 conn(0x7fc3500a85a0 msgr2=0x7fc3500a8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:04.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.189+0000 7fc357600640 1 -- 172.21.15.28:0/1818346850 shutdown_connections 2024-02-21T23:06:04.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.189+0000 7fc357600640 1 -- 172.21.15.28:0/1818346850 wait complete. 2024-02-21T23:06:04.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.192+0000 7fc357600640 1 Processor -- start 2024-02-21T23:06:04.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.192+0000 7fc357600640 1 -- start start 2024-02-21T23:06:04.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.193+0000 7fc357600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc35006b0e0 0x7fc3500bfe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:04.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.193+0000 7fc357600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc3500c5f50 0x7fc350143b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:04.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.193+0000 7fc357600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc350150530 0x7fc350144050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:04.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.193+0000 7fc357600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc35006b4c0 con 0x7fc35006b0e0 2024-02-21T23:06:04.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.193+0000 7fc357600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3501509c0 con 0x7fc3500c5f50 2024-02-21T23:06:04.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.193+0000 7fc357600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc35006b640 con 0x7fc350150530 2024-02-21T23:06:04.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.194+0000 7fc354c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc35006b0e0 0x7fc3500bfe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:04.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.194+0000 7fc354c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc35006b0e0 0x7fc3500bfe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54292/0 (socket says 172.21.15.28:54292) 2024-02-21T23:06:04.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.194+0000 7fc354c00640 1 -- 172.21.15.28:0/2018874059 learned_addr learned my addr 172.21.15.28:0/2018874059 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:04.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.194+0000 7fc34fe00640 1 --2- 172.21.15.28:0/2018874059 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc3500c5f50 0x7fc350143b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:04.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.194+0000 7fc355600640 1 --2- 172.21.15.28:0/2018874059 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc350150530 0x7fc350144050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:04.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.194+0000 7fc354c00640 1 -- 172.21.15.28:0/2018874059 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc350150530 msgr2=0x7fc350144050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:04.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.194+0000 7fc354c00640 1 --2- 172.21.15.28:0/2018874059 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc350150530 0x7fc350144050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:04.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.194+0000 7fc354c00640 1 -- 172.21.15.28:0/2018874059 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc3500c5f50 msgr2=0x7fc350143b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:04.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.194+0000 7fc354c00640 1 --2- 172.21.15.28:0/2018874059 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc3500c5f50 0x7fc350143b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:04.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.194+0000 7fc354c00640 1 -- 172.21.15.28:0/2018874059 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc350155320 con 0x7fc35006b0e0 2024-02-21T23:06:04.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.194+0000 7fc34fe00640 1 --2- 172.21.15.28:0/2018874059 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc3500c5f50 0x7fc350143b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:06:04.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.195+0000 7fc354c00640 1 --2- 172.21.15.28:0/2018874059 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc35006b0e0 0x7fc3500bfe90 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fc34000c450 tx=0x7fc34000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:04.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.195+0000 7fc34cc00640 1 -- 172.21.15.28:0/2018874059 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3400052d0 con 0x7fc35006b0e0 2024-02-21T23:06:04.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.195+0000 7fc34cc00640 1 -- 172.21.15.28:0/2018874059 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc340005470 con 0x7fc35006b0e0 2024-02-21T23:06:04.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.195+0000 7fc34cc00640 1 -- 172.21.15.28:0/2018874059 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc340012870 con 0x7fc35006b0e0 2024-02-21T23:06:04.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.195+0000 7fc357600640 1 -- 172.21.15.28:0/2018874059 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc350144590 con 0x7fc35006b0e0 2024-02-21T23:06:04.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.195+0000 7fc357600640 1 -- 172.21.15.28:0/2018874059 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3500a6670 con 0x7fc35006b0e0 2024-02-21T23:06:04.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.196+0000 7fc357600640 1 -- 172.21.15.28:0/2018874059 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc350067f30 con 0x7fc35006b0e0 2024-02-21T23:06:04.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.197+0000 7fc34cc00640 1 -- 172.21.15.28:0/2018874059 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc340012a10 con 0x7fc35006b0e0 2024-02-21T23:06:04.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.198+0000 7fc34cc00640 1 --2- 172.21.15.28:0/2018874059 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc33006b4d0 0x7fc33006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:04.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.199+0000 7fc34cc00640 1 -- 172.21.15.28:0/2018874059 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc340089fe0 con 0x7fc35006b0e0 2024-02-21T23:06:04.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.199+0000 7fc34fe00640 1 --2- 172.21.15.28:0/2018874059 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc33006b4d0 0x7fc33006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:04.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.200+0000 7fc34fe00640 1 --2- 172.21.15.28:0/2018874059 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc33006b4d0 0x7fc33006d990 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fc33c036000 tx=0x7fc33c008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:04.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.203+0000 7fc34cc00640 1 -- 172.21.15.28:0/2018874059 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc350155320 con 0x7fc35006b0e0 2024-02-21T23:06:04.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.404+0000 7fc357600640 1 -- 172.21.15.28:0/2018874059 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) v1 -- 0x7fc3500abda0 con 0x7fc35006b0e0 2024-02-21T23:06:04.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.410+0000 7fc34cc00640 1 -- 172.21.15.28:0/2018874059 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v24)=0 v24) v1 ==== 117+0+0 (secure 0 0 0) 0x7fc340058c90 con 0x7fc35006b0e0 2024-02-21T23:06:04.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.412+0000 7fc357600640 1 -- 172.21.15.28:0/2018874059 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc33006b4d0 msgr2=0x7fc33006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:04.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.412+0000 7fc357600640 1 --2- 172.21.15.28:0/2018874059 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc33006b4d0 0x7fc33006d990 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fc33c036000 tx=0x7fc33c008030 comp rx=0 tx=0).stop 2024-02-21T23:06:04.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.412+0000 7fc357600640 1 -- 172.21.15.28:0/2018874059 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc35006b0e0 msgr2=0x7fc3500bfe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:04.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.412+0000 7fc357600640 1 --2- 172.21.15.28:0/2018874059 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc35006b0e0 0x7fc3500bfe90 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fc34000c450 tx=0x7fc34000c910 comp rx=0 tx=0).stop 2024-02-21T23:06:04.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.412+0000 7fc357600640 1 -- 172.21.15.28:0/2018874059 shutdown_connections 2024-02-21T23:06:04.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.412+0000 7fc357600640 1 --2- 172.21.15.28:0/2018874059 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc33006b4d0 0x7fc33006d990 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:04.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.412+0000 7fc357600640 1 --2- 172.21.15.28:0/2018874059 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc350150530 0x7fc350144050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:04.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.412+0000 7fc357600640 1 --2- 172.21.15.28:0/2018874059 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc3500c5f50 0x7fc350143b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:04.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.412+0000 7fc357600640 1 --2- 172.21.15.28:0/2018874059 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc35006b0e0 0x7fc3500bfe90 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:04.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.413+0000 7fc357600640 1 -- 172.21.15.28:0/2018874059 >> 172.21.15.28:0/2018874059 conn(0x7fc3500a85a0 msgr2=0x7fc35014e1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:04.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.413+0000 7fc357600640 1 -- 172.21.15.28:0/2018874059 shutdown_connections 2024-02-21T23:06:04.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.413+0000 7fc357600640 1 -- 172.21.15.28:0/2018874059 wait complete. 2024-02-21T23:06:04.419 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.b mds_join_fs cephfs2 2024-02-21T23:06:04.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.653+0000 7f8fc4a00640 1 Processor -- start 2024-02-21T23:06:04.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.653+0000 7f8fc4a00640 1 -- start start 2024-02-21T23:06:04.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.655+0000 7f8fc4a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8fc00c4580 0x7f8fc00c12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:04.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.655+0000 7f8fc4a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f8fc00c2c10 0x7f8fc006ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:04.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.655+0000 7f8fc4a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fc006e220 con 0x7f8fc0150240 2024-02-21T23:06:04.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.655+0000 7f8fc4a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fc0150620 con 0x7f8fc00c4580 2024-02-21T23:06:04.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.655+0000 7f8fc4a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fc0153a90 con 0x7f8fc00c2c10 2024-02-21T23:06:04.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.655+0000 7f8fb7e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8fc00c4580 0x7f8fc00c12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:04.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.655+0000 7f8fb7e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8fc00c4580 0x7f8fc00c12c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54306/0 (socket says 172.21.15.28:54306) 2024-02-21T23:06:04.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.655+0000 7f8fb7e00640 1 -- 172.21.15.28:0/3850873917 learned_addr learned my addr 172.21.15.28:0/3850873917 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:04.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.656+0000 7f8fbf200640 1 --2- 172.21.15.28:0/3850873917 >> v2:172.21.15.81:3300/0 conn(0x7f8fc00c2c10 0x7f8fc006ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:04.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.656+0000 7f8fb7e00640 1 -- 172.21.15.28:0/3850873917 >> v2:172.21.15.81:3300/0 conn(0x7f8fc00c2c10 msgr2=0x7f8fc006ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:04.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.656+0000 7f8fb7e00640 1 --2- 172.21.15.28:0/3850873917 >> v2:172.21.15.81:3300/0 conn(0x7f8fc00c2c10 0x7f8fc006ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:04.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.656+0000 7f8fb7e00640 1 -- 172.21.15.28:0/3850873917 >> v1:172.21.15.81:6789/0 conn(0x7f8fc0150240 legacy=0x7f8fc006db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:04.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.656+0000 7f8fb7e00640 1 -- 172.21.15.28:0/3850873917 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fc006f780 con 0x7f8fc00c4580 2024-02-21T23:06:04.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.657+0000 7f8fb7e00640 1 --2- 172.21.15.28:0/3850873917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fc00c4580 0x7f8fc00c12c0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f8fc006d140 tx=0x7f8fa802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df82e8d69900c8e8 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:04.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.657+0000 7f8fbde00640 1 -- 172.21.15.28:0/3850873917 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fa802e070 con 0x7f8fc00c4580 2024-02-21T23:06:04.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.658+0000 7f8fbde00640 1 -- 172.21.15.28:0/3850873917 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fa8037070 con 0x7f8fc00c4580 2024-02-21T23:06:04.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.658+0000 7f8fbde00640 1 -- 172.21.15.28:0/3850873917 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fa8031a00 con 0x7f8fc00c4580 2024-02-21T23:06:04.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.658+0000 7f8fc4a00640 1 -- 172.21.15.28:0/3850873917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fc00c4580 msgr2=0x7f8fc00c12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:04.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.658+0000 7f8fc4a00640 1 --2- 172.21.15.28:0/3850873917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fc00c4580 0x7f8fc00c12c0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f8fc006d140 tx=0x7f8fa802b880 comp rx=0 tx=0).stop 2024-02-21T23:06:04.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.659+0000 7f8fc4a00640 1 -- 172.21.15.28:0/3850873917 shutdown_connections 2024-02-21T23:06:04.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.659+0000 7f8fc4a00640 1 --2- 172.21.15.28:0/3850873917 >> v2:172.21.15.81:3300/0 conn(0x7f8fc00c2c10 0x7f8fc006ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:04.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.659+0000 7f8fc4a00640 1 --2- 172.21.15.28:0/3850873917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fc00c4580 0x7f8fc00c12c0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:04.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.659+0000 7f8fc4a00640 1 -- 172.21.15.28:0/3850873917 >> 172.21.15.28:0/3850873917 conn(0x7f8fc00a8530 msgr2=0x7f8fc00a8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:04.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.659+0000 7f8fc4a00640 1 -- 172.21.15.28:0/3850873917 shutdown_connections 2024-02-21T23:06:04.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.659+0000 7f8fc4a00640 1 -- 172.21.15.28:0/3850873917 wait complete. 2024-02-21T23:06:04.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.660+0000 7f8fc4a00640 1 Processor -- start 2024-02-21T23:06:04.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.661+0000 7f8fc4a00640 1 -- start start 2024-02-21T23:06:04.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.661+0000 7f8fc4a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8fc00c2c10 0x7f8fc01dc500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:04.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.661+0000 7f8fc4a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fc00c4580 0x7f8fc01dce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:04.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.661+0000 7f8fc4a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8fc0150240 0x7f8fc01dd390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:04.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.662+0000 7f8fc4a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fc00c4a10 con 0x7f8fc00c4580 2024-02-21T23:06:04.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.662+0000 7f8fc4a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fc0153720 con 0x7f8fc00c2c10 2024-02-21T23:06:04.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.662+0000 7f8fc4a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fc00c4b90 con 0x7f8fc0150240 2024-02-21T23:06:04.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.662+0000 7f8fb7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fc00c4580 0x7f8fc01dce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:04.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.662+0000 7f8fb7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fc00c4580 0x7f8fc01dce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54316/0 (socket says 172.21.15.28:54316) 2024-02-21T23:06:04.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.662+0000 7f8fb7e00640 1 -- 172.21.15.28:0/2692091319 learned_addr learned my addr 172.21.15.28:0/2692091319 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:04.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.662+0000 7f8fbe800640 1 --2- 172.21.15.28:0/2692091319 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8fc00c2c10 0x7f8fc01dc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:04.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.662+0000 7f8fb7e00640 1 -- 172.21.15.28:0/2692091319 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8fc0150240 msgr2=0x7f8fc01dd390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:04.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.662+0000 7f8fbf200640 1 --2- 172.21.15.28:0/2692091319 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8fc0150240 0x7f8fc01dd390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:04.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.662+0000 7f8fb7e00640 1 --2- 172.21.15.28:0/2692091319 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8fc0150240 0x7f8fc01dd390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:04.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.662+0000 7f8fb7e00640 1 -- 172.21.15.28:0/2692091319 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8fc00c2c10 msgr2=0x7f8fc01dc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:04.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.662+0000 7f8fb7e00640 1 --2- 172.21.15.28:0/2692091319 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8fc00c2c10 0x7f8fc01dc500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:04.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.662+0000 7f8fb7e00640 1 -- 172.21.15.28:0/2692091319 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fc01ef2a0 con 0x7f8fc00c4580 2024-02-21T23:06:04.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.663+0000 7f8fb7e00640 1 --2- 172.21.15.28:0/2692091319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fc00c4580 0x7f8fc01dce50 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f8fa802b8c0 tx=0x7f8fa80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:04.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.663+0000 7f8fb6a00640 1 -- 172.21.15.28:0/2692091319 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fa8032670 con 0x7f8fc00c4580 2024-02-21T23:06:04.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.663+0000 7f8fb6a00640 1 -- 172.21.15.28:0/2692091319 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fa8032850 con 0x7f8fc00c4580 2024-02-21T23:06:04.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.663+0000 7f8fc4a00640 1 -- 172.21.15.28:0/2692091319 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fc01eed20 con 0x7f8fc00c4580 2024-02-21T23:06:04.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.663+0000 7f8fb6a00640 1 -- 172.21.15.28:0/2692091319 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fa8032c40 con 0x7f8fc00c4580 2024-02-21T23:06:04.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.663+0000 7f8fc4a00640 1 -- 172.21.15.28:0/2692091319 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fc01eefc0 con 0x7f8fc00c4580 2024-02-21T23:06:04.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.665+0000 7f8fc4a00640 1 -- 172.21.15.28:0/2692091319 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fc00bfbb0 con 0x7f8fc00c4580 2024-02-21T23:06:04.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.666+0000 7f8fb6a00640 1 -- 172.21.15.28:0/2692091319 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8fa804c060 con 0x7f8fc00c4580 2024-02-21T23:06:04.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.666+0000 7f8fb6a00640 1 --2- 172.21.15.28:0/2692091319 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8f9c06b4d0 0x7f8f9c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:04.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.667+0000 7f8fb6a00640 1 -- 172.21.15.28:0/2692091319 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f8fa80abc80 con 0x7f8fc00c4580 2024-02-21T23:06:04.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.667+0000 7f8fbe800640 1 --2- 172.21.15.28:0/2692091319 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8f9c06b4d0 0x7f8f9c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:04.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.668+0000 7f8fbe800640 1 --2- 172.21.15.28:0/2692091319 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8f9c06b4d0 0x7f8f9c06d990 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f8fb0006fd0 tx=0x7f8fb0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:04.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.676+0000 7f8fb6a00640 1 -- 172.21.15.28:0/2692091319 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f8fc00bfbb0 con 0x7f8fc00c4580 2024-02-21T23:06:04.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.875+0000 7f8fc4a00640 1 -- 172.21.15.28:0/2692091319 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) v1 -- 0x7f8fc004f290 con 0x7f8fc00c4580 2024-02-21T23:06:04.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.882+0000 7f8fb6a00640 1 -- 172.21.15.28:0/2692091319 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v25)=0 v25) v1 ==== 117+0+0 (secure 0 0 0) 0x7f8fa8040070 con 0x7f8fc00c4580 2024-02-21T23:06:04.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.884+0000 7f8fc4a00640 1 -- 172.21.15.28:0/2692091319 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8f9c06b4d0 msgr2=0x7f8f9c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:04.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.884+0000 7f8fc4a00640 1 --2- 172.21.15.28:0/2692091319 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8f9c06b4d0 0x7f8f9c06d990 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f8fb0006fd0 tx=0x7f8fb0008040 comp rx=0 tx=0).stop 2024-02-21T23:06:04.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.884+0000 7f8fc4a00640 1 -- 172.21.15.28:0/2692091319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fc00c4580 msgr2=0x7f8fc01dce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:04.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.884+0000 7f8fc4a00640 1 --2- 172.21.15.28:0/2692091319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fc00c4580 0x7f8fc01dce50 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f8fa802b8c0 tx=0x7f8fa80046c0 comp rx=0 tx=0).stop 2024-02-21T23:06:04.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.884+0000 7f8fc4a00640 1 -- 172.21.15.28:0/2692091319 shutdown_connections 2024-02-21T23:06:04.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.884+0000 7f8fc4a00640 1 --2- 172.21.15.28:0/2692091319 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8f9c06b4d0 0x7f8f9c06d990 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:04.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.884+0000 7f8fc4a00640 1 --2- 172.21.15.28:0/2692091319 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8fc0150240 0x7f8fc01dd390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:04.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.884+0000 7f8fc4a00640 1 --2- 172.21.15.28:0/2692091319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fc00c4580 0x7f8fc01dce50 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:04.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.884+0000 7f8fc4a00640 1 --2- 172.21.15.28:0/2692091319 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8fc00c2c10 0x7f8fc01dc500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:04.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.884+0000 7f8fc4a00640 1 -- 172.21.15.28:0/2692091319 >> 172.21.15.28:0/2692091319 conn(0x7f8fc00a8530 msgr2=0x7f8fc014f260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:04.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.885+0000 7f8fc4a00640 1 -- 172.21.15.28:0/2692091319 shutdown_connections 2024-02-21T23:06:04.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:04.885+0000 7f8fc4a00640 1 -- 172.21.15.28:0/2692091319 wait complete. 2024-02-21T23:06:04.897 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.c mds_join_fs cephfs2 2024-02-21T23:06:05.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.139+0000 7ff4d4e00640 1 Processor -- start 2024-02-21T23:06:05.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.139+0000 7ff4d4e00640 1 -- start start 2024-02-21T23:06:05.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.141+0000 7ff4d4e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff4d0158480 0x7ff4d0154f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:05.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.141+0000 7ff4d4e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff4d006e9d0 0x7ff4d006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:05.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.141+0000 7ff4d4e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4d006f380 con 0x7ff4d0152560 2024-02-21T23:06:05.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.141+0000 7ff4d4e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4d01554a0 con 0x7ff4d0158480 2024-02-21T23:06:05.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.141+0000 7ff4d4e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d00a6270 con 0x7ff4d006e9d0 2024-02-21T23:06:05.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.141+0000 7ff4cf400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff4d0152560 0x7ff4d006ae70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59762/0 (socket says 172.21.15.28:59762) 2024-02-21T23:06:05.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.141+0000 7ff4cf400640 1 -- 172.21.15.28:0/1632692657 learned_addr learned my addr 172.21.15.28:0/1632692657 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:06:05.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.141+0000 7ff4c7e00640 1 --2- 172.21.15.28:0/1632692657 >> v2:172.21.15.28:3300/0 conn(0x7ff4d0158480 0x7ff4d0154f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:05.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.141+0000 7ff4cfe00640 1 --2- 172.21.15.28:0/1632692657 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff4d006e9d0 0x7ff4d006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:05.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.142+0000 7ff4cea00640 1 -- 172.21.15.28:0/1632692657 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1383286467 0 0) 0x7ff4d006f380 con 0x7ff4d0152560 2024-02-21T23:06:05.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.142+0000 7ff4c7e00640 1 -- 172.21.15.28:0/1632692657 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff4d006e9d0 msgr2=0x7ff4d006ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:05.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.142+0000 7ff4c7e00640 1 --2- 172.21.15.28:0/1632692657 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff4d006e9d0 0x7ff4d006ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:05.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.142+0000 7ff4c7e00640 1 -- 172.21.15.28:0/1632692657 >> v1:172.21.15.28:6789/0 conn(0x7ff4d0152560 legacy=0x7ff4d006ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:05.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.142+0000 7ff4c7e00640 1 -- 172.21.15.28:0/1632692657 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4d01556d0 con 0x7ff4d0158480 2024-02-21T23:06:05.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.143+0000 7ff4c7e00640 1 --2- 172.21.15.28:0/1632692657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4d0158480 0x7ff4d0154f60 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7ff4b80097a0 tx=0x7ff4b802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfb18445e00373c4 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:05.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.143+0000 7ff4cea00640 1 -- 172.21.15.28:0/1632692657 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b8035020 con 0x7ff4d0158480 2024-02-21T23:06:05.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.143+0000 7ff4cea00640 1 -- 172.21.15.28:0/1632692657 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4b8031810 con 0x7ff4d0158480 2024-02-21T23:06:05.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.143+0000 7ff4cea00640 1 -- 172.21.15.28:0/1632692657 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b8031be0 con 0x7ff4d0158480 2024-02-21T23:06:05.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.144+0000 7ff4d4e00640 1 -- 172.21.15.28:0/1632692657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4d0158480 msgr2=0x7ff4d0154f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:05.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.144+0000 7ff4d4e00640 1 --2- 172.21.15.28:0/1632692657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4d0158480 0x7ff4d0154f60 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7ff4b80097a0 tx=0x7ff4b802b8b0 comp rx=0 tx=0).stop 2024-02-21T23:06:05.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.144+0000 7ff4d4e00640 1 -- 172.21.15.28:0/1632692657 shutdown_connections 2024-02-21T23:06:05.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.144+0000 7ff4d4e00640 1 --2- 172.21.15.28:0/1632692657 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff4d006e9d0 0x7ff4d006ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:05.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.144+0000 7ff4d4e00640 1 --2- 172.21.15.28:0/1632692657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4d0158480 0x7ff4d0154f60 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:05.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.144+0000 7ff4d4e00640 1 -- 172.21.15.28:0/1632692657 >> 172.21.15.28:0/1632692657 conn(0x7ff4d00a8530 msgr2=0x7ff4d00a8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:05.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.144+0000 7ff4d4e00640 1 -- 172.21.15.28:0/1632692657 shutdown_connections 2024-02-21T23:06:05.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.144+0000 7ff4d4e00640 1 -- 172.21.15.28:0/1632692657 wait complete. 2024-02-21T23:06:05.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.146+0000 7ff4d4e00640 1 Processor -- start 2024-02-21T23:06:05.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.146+0000 7ff4d4e00640 1 -- start start 2024-02-21T23:06:05.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.147+0000 7ff4d4e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff4d006e9d0 0x7ff4d01e7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:05.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.147+0000 7ff4d4e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff4d0152560 0x7ff4d0156f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:05.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.147+0000 7ff4d4e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4d0158480 0x7ff4d01eab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:05.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.147+0000 7ff4d4e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d006e4b0 con 0x7ff4d0158480 2024-02-21T23:06:05.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.147+0000 7ff4d4e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d0155c90 con 0x7ff4d0152560 2024-02-21T23:06:05.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.147+0000 7ff4d4e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d0152940 con 0x7ff4d006e9d0 2024-02-21T23:06:05.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.148+0000 7ff4cfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4d0158480 0x7ff4d01eab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:05.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.148+0000 7ff4cfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4d0158480 0x7ff4d01eab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54506/0 (socket says 172.21.15.28:54506) 2024-02-21T23:06:05.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.148+0000 7ff4cfe00640 1 -- 172.21.15.28:0/2893712518 learned_addr learned my addr 172.21.15.28:0/2893712518 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:05.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.148+0000 7ff4cf400640 1 --2- 172.21.15.28:0/2893712518 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff4d006e9d0 0x7ff4d01e7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:05.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.148+0000 7ff4c7e00640 1 --2- 172.21.15.28:0/2893712518 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff4d0152560 0x7ff4d0156f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:05.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.148+0000 7ff4cfe00640 1 -- 172.21.15.28:0/2893712518 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff4d006e9d0 msgr2=0x7ff4d01e7dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:05.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.148+0000 7ff4cfe00640 1 --2- 172.21.15.28:0/2893712518 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff4d006e9d0 0x7ff4d01e7dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:05.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.148+0000 7ff4cfe00640 1 -- 172.21.15.28:0/2893712518 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff4d0152560 msgr2=0x7ff4d0156f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:05.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.148+0000 7ff4cfe00640 1 --2- 172.21.15.28:0/2893712518 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff4d0152560 0x7ff4d0156f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:05.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.148+0000 7ff4cfe00640 1 -- 172.21.15.28:0/2893712518 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4d01eeb10 con 0x7ff4d0158480 2024-02-21T23:06:05.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.149+0000 7ff4cfe00640 1 --2- 172.21.15.28:0/2893712518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4d0158480 0x7ff4d01eab70 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7ff4c800efc0 tx=0x7ff4c800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:05.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.149+0000 7ff4c7400640 1 -- 172.21.15.28:0/2893712518 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c8018020 con 0x7ff4d0158480 2024-02-21T23:06:05.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.149+0000 7ff4c7400640 1 -- 172.21.15.28:0/2893712518 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4c8014480 con 0x7ff4d0158480 2024-02-21T23:06:05.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.149+0000 7ff4c7400640 1 -- 172.21.15.28:0/2893712518 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c8014870 con 0x7ff4d0158480 2024-02-21T23:06:05.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.149+0000 7ff4d4e00640 1 -- 172.21.15.28:0/2893712518 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4d00724d0 con 0x7ff4d0158480 2024-02-21T23:06:05.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.149+0000 7ff4d4e00640 1 -- 172.21.15.28:0/2893712518 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4d0072a90 con 0x7ff4d0158480 2024-02-21T23:06:05.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.150+0000 7ff4d4e00640 1 -- 172.21.15.28:0/2893712518 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4d00c2f00 con 0x7ff4d0158480 2024-02-21T23:06:05.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.152+0000 7ff4c7400640 1 -- 172.21.15.28:0/2893712518 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff4c8005d10 con 0x7ff4d0158480 2024-02-21T23:06:05.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.153+0000 7ff4c7400640 1 --2- 172.21.15.28:0/2893712518 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff4ac06b4d0 0x7ff4ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:05.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.153+0000 7ff4c7400640 1 -- 172.21.15.28:0/2893712518 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff4c808adf0 con 0x7ff4d0158480 2024-02-21T23:06:05.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.154+0000 7ff4cf400640 1 --2- 172.21.15.28:0/2893712518 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff4ac06b4d0 0x7ff4ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:05.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.155+0000 7ff4cf400640 1 --2- 172.21.15.28:0/2893712518 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff4ac06b4d0 0x7ff4ac06d990 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7ff4c0000f60 tx=0x7ff4c0007b50 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:05.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.157+0000 7ff4c7400640 1 -- 172.21.15.28:0/2893712518 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ff4c8022070 con 0x7ff4d0158480 2024-02-21T23:06:05.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.358+0000 7ff4d4e00640 1 -- 172.21.15.28:0/2893712518 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) v1 -- 0x7ff4d004f290 con 0x7ff4d0158480 2024-02-21T23:06:05.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.364+0000 7ff4c7400640 1 -- 172.21.15.28:0/2893712518 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v26)=0 v26) v1 ==== 117+0+0 (secure 0 0 0) 0x7ff4c8010040 con 0x7ff4d0158480 2024-02-21T23:06:05.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.367+0000 7ff4d4e00640 1 -- 172.21.15.28:0/2893712518 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff4ac06b4d0 msgr2=0x7ff4ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:05.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.367+0000 7ff4d4e00640 1 --2- 172.21.15.28:0/2893712518 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff4ac06b4d0 0x7ff4ac06d990 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7ff4c0000f60 tx=0x7ff4c0007b50 comp rx=0 tx=0).stop 2024-02-21T23:06:05.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.367+0000 7ff4d4e00640 1 -- 172.21.15.28:0/2893712518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4d0158480 msgr2=0x7ff4d01eab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:05.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.367+0000 7ff4d4e00640 1 --2- 172.21.15.28:0/2893712518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4d0158480 0x7ff4d01eab70 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7ff4c800efc0 tx=0x7ff4c800c490 comp rx=0 tx=0).stop 2024-02-21T23:06:05.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.367+0000 7ff4d4e00640 1 -- 172.21.15.28:0/2893712518 shutdown_connections 2024-02-21T23:06:05.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.367+0000 7ff4d4e00640 1 --2- 172.21.15.28:0/2893712518 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff4ac06b4d0 0x7ff4ac06d990 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:05.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.367+0000 7ff4d4e00640 1 --2- 172.21.15.28:0/2893712518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4d0158480 0x7ff4d01eab70 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:05.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.367+0000 7ff4d4e00640 1 --2- 172.21.15.28:0/2893712518 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff4d0152560 0x7ff4d0156f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:05.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.367+0000 7ff4d4e00640 1 --2- 172.21.15.28:0/2893712518 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff4d006e9d0 0x7ff4d01e7dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:05.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.367+0000 7ff4d4e00640 1 -- 172.21.15.28:0/2893712518 >> 172.21.15.28:0/2893712518 conn(0x7ff4d00a8530 msgr2=0x7ff4d00c4060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:05.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.367+0000 7ff4d4e00640 1 -- 172.21.15.28:0/2893712518 shutdown_connections 2024-02-21T23:06:05.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.368+0000 7ff4d4e00640 1 -- 172.21.15.28:0/2893712518 wait complete. 2024-02-21T23:06:05.377 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:06:05.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.589+0000 7f400ac00640 1 Processor -- start 2024-02-21T23:06:05.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.590+0000 7f400ac00640 1 -- start start 2024-02-21T23:06:05.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.591+0000 7f400ac00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f40040aff90 0x7f40040c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:05.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.591+0000 7f400ac00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f400406dbf0 0x7f4004158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:05.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.591+0000 7f400ac00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40040c74c0 con 0x7f40040c5720 2024-02-21T23:06:05.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.591+0000 7f400ac00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40040c7270 con 0x7f40040aff90 2024-02-21T23:06:05.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.591+0000 7f400ac00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4004158630 con 0x7f400406dbf0 2024-02-21T23:06:05.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.591+0000 7f4009800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f40040aff90 0x7f40040c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:05.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.591+0000 7f400a200640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f40040c5720 0x7f400406c270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59776/0 (socket says 172.21.15.28:59776) 2024-02-21T23:06:05.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.591+0000 7f400a200640 1 -- 172.21.15.28:0/800420892 learned_addr learned my addr 172.21.15.28:0/800420892 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:06:05.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.592+0000 7f4008e00640 1 --2- 172.21.15.28:0/800420892 >> v2:172.21.15.81:3300/0 conn(0x7f400406dbf0 0x7f4004158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:05.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.592+0000 7f4009800640 1 -- 172.21.15.28:0/800420892 >> v2:172.21.15.81:3300/0 conn(0x7f400406dbf0 msgr2=0x7f4004158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:05.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.592+0000 7f4009800640 1 --2- 172.21.15.28:0/800420892 >> v2:172.21.15.81:3300/0 conn(0x7f400406dbf0 0x7f4004158ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:05.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.592+0000 7f4009800640 1 -- 172.21.15.28:0/800420892 >> v1:172.21.15.28:6789/0 conn(0x7f40040c5720 legacy=0x7f400406c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:05.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.592+0000 7f4009800640 1 -- 172.21.15.28:0/800420892 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40040c77a0 con 0x7f40040aff90 2024-02-21T23:06:05.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.592+0000 7f4008e00640 1 --2- 172.21.15.28:0/800420892 >> v2:172.21.15.81:3300/0 conn(0x7f400406dbf0 0x7f4004158ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:06:05.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.593+0000 7f4009800640 1 --2- 172.21.15.28:0/800420892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40040aff90 0x7f40040c51e0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f40040c73f0 tx=0x7f3ff802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e80d2103cbb415ac server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:05.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.593+0000 7f4003e00640 1 -- 172.21.15.28:0/800420892 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ff8035020 con 0x7f40040aff90 2024-02-21T23:06:05.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.594+0000 7f4003e00640 1 -- 172.21.15.28:0/800420892 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ff8031830 con 0x7f40040aff90 2024-02-21T23:06:05.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.594+0000 7f4003e00640 1 -- 172.21.15.28:0/800420892 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ff8031c00 con 0x7f40040aff90 2024-02-21T23:06:05.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.594+0000 7f400ac00640 1 -- 172.21.15.28:0/800420892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40040aff90 msgr2=0x7f40040c51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:05.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.594+0000 7f400ac00640 1 --2- 172.21.15.28:0/800420892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40040aff90 0x7f40040c51e0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f40040c73f0 tx=0x7f3ff802b8d0 comp rx=0 tx=0).stop 2024-02-21T23:06:05.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.594+0000 7f400ac00640 1 -- 172.21.15.28:0/800420892 shutdown_connections 2024-02-21T23:06:05.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.594+0000 7f400ac00640 1 --2- 172.21.15.28:0/800420892 >> v2:172.21.15.81:3300/0 conn(0x7f400406dbf0 0x7f4004158ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:05.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.594+0000 7f400ac00640 1 --2- 172.21.15.28:0/800420892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40040aff90 0x7f40040c51e0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:05.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.594+0000 7f400ac00640 1 -- 172.21.15.28:0/800420892 >> 172.21.15.28:0/800420892 conn(0x7f40040ac370 msgr2=0x7f40040ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:05.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.594+0000 7f400ac00640 1 -- 172.21.15.28:0/800420892 shutdown_connections 2024-02-21T23:06:05.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.594+0000 7f400ac00640 1 -- 172.21.15.28:0/800420892 wait complete. 2024-02-21T23:06:05.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.597+0000 7f400ac00640 1 Processor -- start 2024-02-21T23:06:05.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.597+0000 7f400ac00640 1 -- start start 2024-02-21T23:06:05.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.599+0000 7f400ac00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f400406dbf0 0x7f4004152d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:05.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.599+0000 7f400ac00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40040aff90 0x7f40040c2f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:05.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.599+0000 7f400ac00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f40040c5720 0x7f40040ab7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:05.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.599+0000 7f400ac00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40040c8760 con 0x7f40040aff90 2024-02-21T23:06:05.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.599+0000 7f400ac00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40040be0c0 con 0x7f400406dbf0 2024-02-21T23:06:05.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.599+0000 7f400ac00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40040c88e0 con 0x7f40040c5720 2024-02-21T23:06:05.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.599+0000 7f4008e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40040aff90 0x7f40040c2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:05.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.599+0000 7f4008e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40040aff90 0x7f40040c2f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54528/0 (socket says 172.21.15.28:54528) 2024-02-21T23:06:05.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.599+0000 7f4008e00640 1 -- 172.21.15.28:0/3558844404 learned_addr learned my addr 172.21.15.28:0/3558844404 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:05.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.599+0000 7f4008e00640 1 -- 172.21.15.28:0/3558844404 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f40040c5720 msgr2=0x7f40040ab7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:05.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.599+0000 7f4008e00640 1 --2- 172.21.15.28:0/3558844404 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f40040c5720 0x7f40040ab7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:05.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.599+0000 7f4008e00640 1 -- 172.21.15.28:0/3558844404 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f400406dbf0 msgr2=0x7f4004152d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:05.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.599+0000 7f4008e00640 1 --2- 172.21.15.28:0/3558844404 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f400406dbf0 0x7f4004152d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:05.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.599+0000 7f4008e00640 1 -- 172.21.15.28:0/3558844404 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4004155490 con 0x7f40040aff90 2024-02-21T23:06:05.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.600+0000 7f4008e00640 1 --2- 172.21.15.28:0/3558844404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40040aff90 0x7f40040c2f00 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f3ff000c7f0 tx=0x7f3ff000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:05.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.600+0000 7f4001600640 1 -- 172.21.15.28:0/3558844404 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ff0015070 con 0x7f40040aff90 2024-02-21T23:06:05.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.600+0000 7f4001600640 1 -- 172.21.15.28:0/3558844404 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ff0007a30 con 0x7f40040aff90 2024-02-21T23:06:05.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.600+0000 7f400ac00640 1 -- 172.21.15.28:0/3558844404 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4004154500 con 0x7f40040aff90 2024-02-21T23:06:05.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.600+0000 7f4001600640 1 -- 172.21.15.28:0/3558844404 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ff0007e00 con 0x7f40040aff90 2024-02-21T23:06:05.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.600+0000 7f400ac00640 1 -- 172.21.15.28:0/3558844404 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40041ee230 con 0x7f40040aff90 2024-02-21T23:06:05.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.602+0000 7f400ac00640 1 -- 172.21.15.28:0/3558844404 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40041ee660 con 0x7f40040aff90 2024-02-21T23:06:05.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.602+0000 7f4001600640 1 -- 172.21.15.28:0/3558844404 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3ff0018070 con 0x7f40040aff90 2024-02-21T23:06:05.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.604+0000 7f4001600640 1 --2- 172.21.15.28:0/3558844404 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3fe406b500 0x7f3fe406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:05.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.604+0000 7f4001600640 1 -- 172.21.15.28:0/3558844404 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f3ff008a210 con 0x7f40040aff90 2024-02-21T23:06:05.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.604+0000 7f4009800640 1 --2- 172.21.15.28:0/3558844404 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3fe406b500 0x7f3fe406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:05.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.606+0000 7f4009800640 1 --2- 172.21.15.28:0/3558844404 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3fe406b500 0x7f3fe406d9c0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f3ff8036000 tx=0x7f3ff8030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:05.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.609+0000 7f4001600640 1 -- 172.21.15.28:0/3558844404 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3ff0058f90 con 0x7f40040aff90 2024-02-21T23:06:05.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.809+0000 7f400ac00640 1 -- 172.21.15.28:0/3558844404 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f400406d9d0 con 0x7f40040aff90 2024-02-21T23:06:05.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.811+0000 7f4001600640 1 -- 172.21.15.28:0/3558844404 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) v1 ==== 74+0+16885 (secure 0 0 0) 0x7f3ff0058890 con 0x7f40040aff90 2024-02-21T23:06:05.800 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:06:05.801 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":76,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:05:52.373861+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:05:48.772562+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-02-21T23:05:49.852659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:06:05.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.813+0000 7f400ac00640 1 -- 172.21.15.28:0/3558844404 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3fe406b500 msgr2=0x7f3fe406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:05.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.813+0000 7f400ac00640 1 --2- 172.21.15.28:0/3558844404 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3fe406b500 0x7f3fe406d9c0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f3ff8036000 tx=0x7f3ff8030000 comp rx=0 tx=0).stop 2024-02-21T23:06:05.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.813+0000 7f400ac00640 1 -- 172.21.15.28:0/3558844404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40040aff90 msgr2=0x7f40040c2f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:05.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.813+0000 7f400ac00640 1 --2- 172.21.15.28:0/3558844404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40040aff90 0x7f40040c2f00 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f3ff000c7f0 tx=0x7f3ff000ccb0 comp rx=0 tx=0).stop 2024-02-21T23:06:05.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.813+0000 7f400ac00640 1 -- 172.21.15.28:0/3558844404 shutdown_connections 2024-02-21T23:06:05.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.813+0000 7f400ac00640 1 --2- 172.21.15.28:0/3558844404 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3fe406b500 0x7f3fe406d9c0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:05.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.813+0000 7f400ac00640 1 --2- 172.21.15.28:0/3558844404 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f40040c5720 0x7f40040ab7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:05.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.813+0000 7f400ac00640 1 --2- 172.21.15.28:0/3558844404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40040aff90 0x7f40040c2f00 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:05.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.813+0000 7f400ac00640 1 --2- 172.21.15.28:0/3558844404 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f400406dbf0 0x7f4004152d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:05.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.813+0000 7f400ac00640 1 -- 172.21.15.28:0/3558844404 >> 172.21.15.28:0/3558844404 conn(0x7f40040ac370 msgr2=0x7f40040ac750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:05.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.814+0000 7f400ac00640 1 -- 172.21.15.28:0/3558844404 shutdown_connections 2024-02-21T23:06:05.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:05.814+0000 7f400ac00640 1 -- 172.21.15.28:0/3558844404 wait complete. 2024-02-21T23:06:05.830 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-21T23:02:30.728329+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-21T23:05:48.772562+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_data', 'create_time': '2024-02-21T23:05:49.852659+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-21T23:06:05.830 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-21T23:06:05.856 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-21T23:06:05.884 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-21T23:06:06.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.106+0000 7fc2b5a00640 1 Processor -- start 2024-02-21T23:06:06.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.107+0000 7fc2b5a00640 1 -- start start 2024-02-21T23:06:06.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.108+0000 7fc2b5a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc2b0152560 0x7fc2b0154f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:06.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.108+0000 7fc2b5a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fc2b006e9d0 0x7fc2b006ee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:06.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.108+0000 7fc2b5a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2b00a6270 con 0x7fc2b0158480 2024-02-21T23:06:06.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.108+0000 7fc2b5a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2b01554a0 con 0x7fc2b0152560 2024-02-21T23:06:06.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.108+0000 7fc2b5a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2b01556d0 con 0x7fc2b006e9d0 2024-02-21T23:06:06.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.108+0000 7fc2afe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc2b0152560 0x7fc2b0154f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:06.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.108+0000 7fc2afe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc2b0152560 0x7fc2b0154f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54530/0 (socket says 172.21.15.28:54530) 2024-02-21T23:06:06.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.108+0000 7fc2afe00640 1 -- 172.21.15.28:0/294095205 learned_addr learned my addr 172.21.15.28:0/294095205 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:06.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.109+0000 7fc2b5000640 1 --2- 172.21.15.28:0/294095205 >> v2:172.21.15.81:3300/0 conn(0x7fc2b006e9d0 0x7fc2b006ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:06.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.109+0000 7fc2afe00640 1 -- 172.21.15.28:0/294095205 >> v2:172.21.15.81:3300/0 conn(0x7fc2b006e9d0 msgr2=0x7fc2b006ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:06.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.109+0000 7fc2afe00640 1 --2- 172.21.15.28:0/294095205 >> v2:172.21.15.81:3300/0 conn(0x7fc2b006e9d0 0x7fc2b006ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:06.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.109+0000 7fc2afe00640 1 -- 172.21.15.28:0/294095205 >> v1:172.21.15.28:6789/0 conn(0x7fc2b0158480 legacy=0x7fc2b006ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:06.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.109+0000 7fc2afe00640 1 -- 172.21.15.28:0/294095205 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2b01559b0 con 0x7fc2b0152560 2024-02-21T23:06:06.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.109+0000 7fc2af400640 1 -- 172.21.15.28:0/294095205 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1733648960 0 0) 0x7fc2b00a6270 con 0x7fc2b0158480 2024-02-21T23:06:06.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.111+0000 7fc2afe00640 1 --2- 172.21.15.28:0/294095205 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2b0152560 0x7fc2b0154f60 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fc2b006d140 tx=0x7fc2a002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7226769fadd3a5f0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:06.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.111+0000 7fc2af400640 1 -- 172.21.15.28:0/294095205 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2a0034070 con 0x7fc2b0152560 2024-02-21T23:06:06.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.111+0000 7fc2af400640 1 -- 172.21.15.28:0/294095205 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2a0030860 con 0x7fc2b0152560 2024-02-21T23:06:06.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.111+0000 7fc2af400640 1 -- 172.21.15.28:0/294095205 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2a0030c30 con 0x7fc2b0152560 2024-02-21T23:06:06.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.112+0000 7fc2b5a00640 1 -- 172.21.15.28:0/294095205 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2b0152560 msgr2=0x7fc2b0154f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:06.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.112+0000 7fc2b5a00640 1 --2- 172.21.15.28:0/294095205 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2b0152560 0x7fc2b0154f60 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fc2b006d140 tx=0x7fc2a002b9a0 comp rx=0 tx=0).stop 2024-02-21T23:06:06.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.112+0000 7fc2b5a00640 1 -- 172.21.15.28:0/294095205 shutdown_connections 2024-02-21T23:06:06.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.112+0000 7fc2b5a00640 1 --2- 172.21.15.28:0/294095205 >> v2:172.21.15.81:3300/0 conn(0x7fc2b006e9d0 0x7fc2b006ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:06.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.112+0000 7fc2b5a00640 1 --2- 172.21.15.28:0/294095205 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2b0152560 0x7fc2b0154f60 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:06.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.112+0000 7fc2b5a00640 1 -- 172.21.15.28:0/294095205 >> 172.21.15.28:0/294095205 conn(0x7fc2b00a8530 msgr2=0x7fc2b00a8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:06.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.112+0000 7fc2b5a00640 1 -- 172.21.15.28:0/294095205 shutdown_connections 2024-02-21T23:06:06.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.112+0000 7fc2b5a00640 1 -- 172.21.15.28:0/294095205 wait complete. 2024-02-21T23:06:06.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.114+0000 7fc2b5a00640 1 Processor -- start 2024-02-21T23:06:06.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.114+0000 7fc2b5a00640 1 -- start start 2024-02-21T23:06:06.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.114+0000 7fc2b5a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2b006e9d0 0x7fc2b0150df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:06.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.114+0000 7fc2b5a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2b0152560 0x7fc2b00c53a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:06.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.115+0000 7fc2b5a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc2b0158480 0x7fc2b01ee220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:06.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.115+0000 7fc2b5a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2b006f340 con 0x7fc2b0152560 2024-02-21T23:06:06.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.115+0000 7fc2b5a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2b006e4b0 con 0x7fc2b006e9d0 2024-02-21T23:06:06.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.115+0000 7fc2b5a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2b0152940 con 0x7fc2b0158480 2024-02-21T23:06:06.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.115+0000 7fc2a7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2b0152560 0x7fc2b00c53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:06.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.115+0000 7fc2a7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2b0152560 0x7fc2b00c53a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54536/0 (socket says 172.21.15.28:54536) 2024-02-21T23:06:06.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.115+0000 7fc2a7e00640 1 -- 172.21.15.28:0/2855275906 learned_addr learned my addr 172.21.15.28:0/2855275906 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:06.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.115+0000 7fc2afe00640 1 --2- 172.21.15.28:0/2855275906 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2b006e9d0 0x7fc2b0150df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:06.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.115+0000 7fc2b5000640 1 --2- 172.21.15.28:0/2855275906 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc2b0158480 0x7fc2b01ee220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:06.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.115+0000 7fc2a7e00640 1 -- 172.21.15.28:0/2855275906 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc2b0158480 msgr2=0x7fc2b01ee220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:06.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.115+0000 7fc2a7e00640 1 --2- 172.21.15.28:0/2855275906 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc2b0158480 0x7fc2b01ee220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:06.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.115+0000 7fc2a7e00640 1 -- 172.21.15.28:0/2855275906 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2b006e9d0 msgr2=0x7fc2b0150df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:06.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.115+0000 7fc2a7e00640 1 --2- 172.21.15.28:0/2855275906 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2b006e9d0 0x7fc2b0150df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:06.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.115+0000 7fc2a7e00640 1 -- 172.21.15.28:0/2855275906 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2b01513f0 con 0x7fc2b0152560 2024-02-21T23:06:06.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.115+0000 7fc2afe00640 1 --2- 172.21.15.28:0/2855275906 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2b006e9d0 0x7fc2b0150df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:06:06.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.116+0000 7fc2a7e00640 1 --2- 172.21.15.28:0/2855275906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2b0152560 0x7fc2b00c53a0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fc29800c4b0 tx=0x7fc29800c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:06.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.116+0000 7fc2acc00640 1 -- 172.21.15.28:0/2855275906 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc298017070 con 0x7fc2b0152560 2024-02-21T23:06:06.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.116+0000 7fc2acc00640 1 -- 172.21.15.28:0/2855275906 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc298005480 con 0x7fc2b0152560 2024-02-21T23:06:06.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.116+0000 7fc2acc00640 1 -- 172.21.15.28:0/2855275906 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc298003830 con 0x7fc2b0152560 2024-02-21T23:06:06.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.116+0000 7fc2b5a00640 1 -- 172.21.15.28:0/2855275906 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2b01516d0 con 0x7fc2b0152560 2024-02-21T23:06:06.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.116+0000 7fc2b5a00640 1 -- 172.21.15.28:0/2855275906 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2b01edd00 con 0x7fc2b0152560 2024-02-21T23:06:06.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.117+0000 7fc2b5a00640 1 -- 172.21.15.28:0/2855275906 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2b004f290 con 0x7fc2b0152560 2024-02-21T23:06:06.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.118+0000 7fc2acc00640 1 -- 172.21.15.28:0/2855275906 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc29801a070 con 0x7fc2b0152560 2024-02-21T23:06:06.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.120+0000 7fc2acc00640 1 --2- 172.21.15.28:0/2855275906 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc28c06b4d0 0x7fc28c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:06.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.120+0000 7fc2acc00640 1 -- 172.21.15.28:0/2855275906 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc298089730 con 0x7fc2b0152560 2024-02-21T23:06:06.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.120+0000 7fc2afe00640 1 --2- 172.21.15.28:0/2855275906 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc28c06b4d0 0x7fc28c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:06.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.122+0000 7fc2afe00640 1 --2- 172.21.15.28:0/2855275906 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc28c06b4d0 0x7fc28c06d990 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fc2a0036000 tx=0x7fc2a003b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:06.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.125+0000 7fc2acc00640 1 -- 172.21.15.28:0/2855275906 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc298058420 con 0x7fc2b0152560 2024-02-21T23:06:06.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.323+0000 7fc2b5a00640 1 -- 172.21.15.28:0/2855275906 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc2b01518a0 con 0x7fc2b0152560 2024-02-21T23:06:06.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.324+0000 7fc2acc00640 1 -- 172.21.15.28:0/2855275906 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v76) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc298057d20 con 0x7fc2b0152560 2024-02-21T23:06:06.314 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-02-21T23:06:06.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.327+0000 7fc2b5a00640 1 -- 172.21.15.28:0/2855275906 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc28c06b4d0 msgr2=0x7fc28c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:06.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.327+0000 7fc2b5a00640 1 --2- 172.21.15.28:0/2855275906 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc28c06b4d0 0x7fc28c06d990 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fc2a0036000 tx=0x7fc2a003b040 comp rx=0 tx=0).stop 2024-02-21T23:06:06.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.327+0000 7fc2b5a00640 1 -- 172.21.15.28:0/2855275906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2b0152560 msgr2=0x7fc2b00c53a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:06.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.327+0000 7fc2b5a00640 1 --2- 172.21.15.28:0/2855275906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2b0152560 0x7fc2b00c53a0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fc29800c4b0 tx=0x7fc29800c970 comp rx=0 tx=0).stop 2024-02-21T23:06:06.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.327+0000 7fc2b5a00640 1 -- 172.21.15.28:0/2855275906 shutdown_connections 2024-02-21T23:06:06.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.327+0000 7fc2b5a00640 1 --2- 172.21.15.28:0/2855275906 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc28c06b4d0 0x7fc28c06d990 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:06.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.327+0000 7fc2b5a00640 1 --2- 172.21.15.28:0/2855275906 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc2b0158480 0x7fc2b01ee220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:06.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.327+0000 7fc2b5a00640 1 --2- 172.21.15.28:0/2855275906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2b0152560 0x7fc2b00c53a0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:06.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.327+0000 7fc2b5a00640 1 --2- 172.21.15.28:0/2855275906 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2b006e9d0 0x7fc2b0150df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:06.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.327+0000 7fc2b5a00640 1 -- 172.21.15.28:0/2855275906 >> 172.21.15.28:0/2855275906 conn(0x7fc2b00a8530 msgr2=0x7fc2b00a8910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:06.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.327+0000 7fc2b5a00640 1 -- 172.21.15.28:0/2855275906 shutdown_connections 2024-02-21T23:06:06.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.327+0000 7fc2b5a00640 1 -- 172.21.15.28:0/2855275906 wait complete. 2024-02-21T23:06:06.338 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-21T23:06:06.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.561+0000 7f78c7400640 1 Processor -- start 2024-02-21T23:06:06.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.561+0000 7f78c7400640 1 -- start start 2024-02-21T23:06:06.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.563+0000 7f78c7400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f78c00a7180 0x7f78c0072d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:06.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.563+0000 7f78c7400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f78c00c61a0 0x7f78c00c5770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:06.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.563+0000 7f78c7400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78c0156b50 con 0x7f78c0150590 2024-02-21T23:06:06.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.563+0000 7f78c7400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78c00c66f0 con 0x7f78c00c61a0 2024-02-21T23:06:06.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.563+0000 7f78c7400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78c00728e0 con 0x7f78c00a7180 2024-02-21T23:06:06.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.563+0000 7f78c5400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f78c00c61a0 0x7f78c00c5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:06.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.563+0000 7f78c5400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f78c00c61a0 0x7f78c00c5770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54546/0 (socket says 172.21.15.28:54546) 2024-02-21T23:06:06.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.563+0000 7f78c5400640 1 -- 172.21.15.28:0/1304105378 learned_addr learned my addr 172.21.15.28:0/1304105378 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:06.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.564+0000 7f78bfe00640 1 --2- 172.21.15.28:0/1304105378 >> v2:172.21.15.81:3300/0 conn(0x7f78c00a7180 0x7f78c0072d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:06.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.564+0000 7f78c5400640 1 -- 172.21.15.28:0/1304105378 >> v2:172.21.15.81:3300/0 conn(0x7f78c00a7180 msgr2=0x7f78c0072d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:06.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.564+0000 7f78c5400640 1 --2- 172.21.15.28:0/1304105378 >> v2:172.21.15.81:3300/0 conn(0x7f78c00a7180 0x7f78c0072d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:06.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.564+0000 7f78c5400640 1 -- 172.21.15.28:0/1304105378 >> v1:172.21.15.28:6789/0 conn(0x7f78c0150590 legacy=0x7f78c0156440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:06.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.564+0000 7f78c5400640 1 -- 172.21.15.28:0/1304105378 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78c00c6940 con 0x7f78c00c61a0 2024-02-21T23:06:06.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.565+0000 7f78c5400640 1 --2- 172.21.15.28:0/1304105378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78c00c61a0 0x7f78c00c5770 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f78c00c1fc0 tx=0x7f78b802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59eb17cb4b236628 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:06.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.566+0000 7f78bf400640 1 -- 172.21.15.28:0/1304105378 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b8035020 con 0x7f78c00c61a0 2024-02-21T23:06:06.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.566+0000 7f78bf400640 1 -- 172.21.15.28:0/1304105378 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78b8031870 con 0x7f78c00c61a0 2024-02-21T23:06:06.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.566+0000 7f78bf400640 1 -- 172.21.15.28:0/1304105378 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b8031c40 con 0x7f78c00c61a0 2024-02-21T23:06:06.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.566+0000 7f78c7400640 1 -- 172.21.15.28:0/1304105378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78c00c61a0 msgr2=0x7f78c00c5770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:06.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.566+0000 7f78c7400640 1 --2- 172.21.15.28:0/1304105378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78c00c61a0 0x7f78c00c5770 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f78c00c1fc0 tx=0x7f78b802b910 comp rx=0 tx=0).stop 2024-02-21T23:06:06.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.567+0000 7f78c7400640 1 -- 172.21.15.28:0/1304105378 shutdown_connections 2024-02-21T23:06:06.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.567+0000 7f78c7400640 1 --2- 172.21.15.28:0/1304105378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78c00c61a0 0x7f78c00c5770 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:06.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.567+0000 7f78c7400640 1 --2- 172.21.15.28:0/1304105378 >> v2:172.21.15.81:3300/0 conn(0x7f78c00a7180 0x7f78c0072d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:06.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.567+0000 7f78c7400640 1 -- 172.21.15.28:0/1304105378 >> 172.21.15.28:0/1304105378 conn(0x7f78c00a9a10 msgr2=0x7f78c00a9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:06.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.567+0000 7f78c7400640 1 -- 172.21.15.28:0/1304105378 shutdown_connections 2024-02-21T23:06:06.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.567+0000 7f78c7400640 1 -- 172.21.15.28:0/1304105378 wait complete. 2024-02-21T23:06:06.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.569+0000 7f78c7400640 1 Processor -- start 2024-02-21T23:06:06.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.570+0000 7f78c7400640 1 -- start start 2024-02-21T23:06:06.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.571+0000 7f78c7400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f78c00a7180 0x7f78c01eb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:06.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.571+0000 7f78c7400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f78c00c61a0 0x7f78c00abb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:06.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.571+0000 7f78c7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78c0150590 0x7f78c00ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:06.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.571+0000 7f78c7400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78c00c6f00 con 0x7f78c0150590 2024-02-21T23:06:06.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.571+0000 7f78c7400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78c0150970 con 0x7f78c00a7180 2024-02-21T23:06:06.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.571+0000 7f78c7400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78c0073430 con 0x7f78c00c61a0 2024-02-21T23:06:06.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.571+0000 7f78c5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78c0150590 0x7f78c00ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:06.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.572+0000 7f78c5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78c0150590 0x7f78c00ac0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54560/0 (socket says 172.21.15.28:54560) 2024-02-21T23:06:06.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.572+0000 7f78c5400640 1 -- 172.21.15.28:0/65988624 learned_addr learned my addr 172.21.15.28:0/65988624 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:06.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.572+0000 7f78c4a00640 1 --2- 172.21.15.28:0/65988624 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f78c00a7180 0x7f78c01eb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:06.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.572+0000 7f78bfe00640 1 --2- 172.21.15.28:0/65988624 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f78c00c61a0 0x7f78c00abb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:06.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.572+0000 7f78c5400640 1 -- 172.21.15.28:0/65988624 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f78c00c61a0 msgr2=0x7f78c00abb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:06.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.572+0000 7f78c5400640 1 --2- 172.21.15.28:0/65988624 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f78c00c61a0 0x7f78c00abb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:06.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.572+0000 7f78c5400640 1 -- 172.21.15.28:0/65988624 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f78c00a7180 msgr2=0x7f78c01eb1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:06.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.572+0000 7f78c5400640 1 --2- 172.21.15.28:0/65988624 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f78c00a7180 0x7f78c01eb1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:06.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.572+0000 7f78c5400640 1 -- 172.21.15.28:0/65988624 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78c014adc0 con 0x7f78c0150590 2024-02-21T23:06:06.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.572+0000 7f78c4a00640 1 --2- 172.21.15.28:0/65988624 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f78c00a7180 0x7f78c01eb1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:06:06.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.572+0000 7f78c5400640 1 --2- 172.21.15.28:0/65988624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78c0150590 0x7f78c00ac0c0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f78b802b950 tx=0x7f78b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:06.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.573+0000 7f78bcc00640 1 -- 172.21.15.28:0/65988624 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b8040070 con 0x7f78c0150590 2024-02-21T23:06:06.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.573+0000 7f78bcc00640 1 -- 172.21.15.28:0/65988624 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78b8030c50 con 0x7f78c0150590 2024-02-21T23:06:06.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.573+0000 7f78bcc00640 1 -- 172.21.15.28:0/65988624 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b8035040 con 0x7f78c0150590 2024-02-21T23:06:06.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.573+0000 7f78c7400640 1 -- 172.21.15.28:0/65988624 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78c01479d0 con 0x7f78c0150590 2024-02-21T23:06:06.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.573+0000 7f78c7400640 1 -- 172.21.15.28:0/65988624 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78c014c050 con 0x7f78c0150590 2024-02-21T23:06:06.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.574+0000 7f78c7400640 1 -- 172.21.15.28:0/65988624 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78c0068640 con 0x7f78c0150590 2024-02-21T23:06:06.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.575+0000 7f78bcc00640 1 -- 172.21.15.28:0/65988624 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f78b8038020 con 0x7f78c0150590 2024-02-21T23:06:06.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.577+0000 7f78bcc00640 1 --2- 172.21.15.28:0/65988624 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f78a006b4d0 0x7f78a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:06.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.577+0000 7f78bcc00640 1 -- 172.21.15.28:0/65988624 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f78b80ac1d0 con 0x7f78c0150590 2024-02-21T23:06:06.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.577+0000 7f78c4a00640 1 --2- 172.21.15.28:0/65988624 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f78a006b4d0 0x7f78a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:06.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.578+0000 7f78c4a00640 1 --2- 172.21.15.28:0/65988624 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f78a006b4d0 0x7f78a006d990 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f78b0001cd0 tx=0x7f78b0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:06.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.580+0000 7f78bcc00640 1 -- 172.21.15.28:0/65988624 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f78b807aeb0 con 0x7f78c0150590 2024-02-21T23:06:06.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.777+0000 7f78c7400640 1 -- 172.21.15.28:0/65988624 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f78c014ca20 con 0x7f78c0150590 2024-02-21T23:06:06.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.778+0000 7f78bcc00640 1 -- 172.21.15.28:0/65988624 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v76) v1 ==== 104+0+11 (secure 0 0 0) 0x7f78b807a7b0 con 0x7f78c0150590 2024-02-21T23:06:06.767 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-02-21T23:06:06.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.780+0000 7f78c7400640 1 -- 172.21.15.28:0/65988624 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f78a006b4d0 msgr2=0x7f78a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:06.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.780+0000 7f78c7400640 1 --2- 172.21.15.28:0/65988624 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f78a006b4d0 0x7f78a006d990 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f78b0001cd0 tx=0x7f78b0009040 comp rx=0 tx=0).stop 2024-02-21T23:06:06.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.780+0000 7f78c7400640 1 -- 172.21.15.28:0/65988624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78c0150590 msgr2=0x7f78c00ac0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:06.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.780+0000 7f78c7400640 1 --2- 172.21.15.28:0/65988624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78c0150590 0x7f78c00ac0c0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f78b802b950 tx=0x7f78b80046c0 comp rx=0 tx=0).stop 2024-02-21T23:06:06.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.780+0000 7f78c7400640 1 -- 172.21.15.28:0/65988624 shutdown_connections 2024-02-21T23:06:06.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.780+0000 7f78c7400640 1 --2- 172.21.15.28:0/65988624 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f78a006b4d0 0x7f78a006d990 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:06.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.780+0000 7f78c7400640 1 --2- 172.21.15.28:0/65988624 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f78c00c61a0 0x7f78c00abb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:06.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.780+0000 7f78c7400640 1 --2- 172.21.15.28:0/65988624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78c0150590 0x7f78c00ac0c0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:06.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.780+0000 7f78c7400640 1 --2- 172.21.15.28:0/65988624 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f78c00a7180 0x7f78c01eb1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:06.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.780+0000 7f78c7400640 1 -- 172.21.15.28:0/65988624 >> 172.21.15.28:0/65988624 conn(0x7f78c00a9a10 msgr2=0x7f78c00a89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:06.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.780+0000 7f78c7400640 1 -- 172.21.15.28:0/65988624 shutdown_connections 2024-02-21T23:06:06.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:06.780+0000 7f78c7400640 1 -- 172.21.15.28:0/65988624 wait complete. 2024-02-21T23:06:06.788 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-21T23:06:06.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.001+0000 7f0038600640 1 Processor -- start 2024-02-21T23:06:06.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.001+0000 7f0038600640 1 -- start start 2024-02-21T23:06:06.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.003+0000 7f0038600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f003014f3e0 0x7f00300c3530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:06.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.003+0000 7f0038600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f00300a97e0 0x7f003006be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:06.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.003+0000 7f0038600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0030152920 con 0x7f00300ac8d0 2024-02-21T23:06:06.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.003+0000 7f0038600640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00300ac380 con 0x7f00300a97e0 2024-02-21T23:06:06.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.003+0000 7f0038600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00300ac5d0 con 0x7f003014f3e0 2024-02-21T23:06:06.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.003+0000 7f0035200640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f00300ac8d0 0x7f00300728c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59792/0 (socket says 172.21.15.28:59792) 2024-02-21T23:06:06.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.003+0000 7f0035200640 1 -- 172.21.15.28:0/2714625069 learned_addr learned my addr 172.21.15.28:0/2714625069 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:06:06.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.004+0000 7f0035c00640 1 --2- 172.21.15.28:0/2714625069 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f003014f3e0 0x7f00300c3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:06.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.004+0000 7f0036600640 1 --2- 172.21.15.28:0/2714625069 >> v2:172.21.15.81:3300/0 conn(0x7f00300a97e0 0x7f003006be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:06.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.004+0000 7f002fe00640 1 -- 172.21.15.28:0/2714625069 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4289828758 0 0) 0x7f0030152920 con 0x7f00300ac8d0 2024-02-21T23:06:06.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.004+0000 7f002fe00640 1 -- 172.21.15.28:0/2714625069 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0018003520 con 0x7f00300ac8d0 2024-02-21T23:06:06.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.004+0000 7f002fe00640 1 -- 172.21.15.28:0/2714625069 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 89929213 0 0) 0x7f0018003520 con 0x7f00300ac8d0 2024-02-21T23:06:06.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.004+0000 7f002fe00640 1 -- 172.21.15.28:0/2714625069 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f003014f3e0 msgr2=0x7f00300c3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:06.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.004+0000 7f002fe00640 1 --2- 172.21.15.28:0/2714625069 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f003014f3e0 0x7f00300c3530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:06.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.004+0000 7f002fe00640 1 -- 172.21.15.28:0/2714625069 >> v2:172.21.15.81:3300/0 conn(0x7f00300a97e0 msgr2=0x7f003006be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:06.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.004+0000 7f002fe00640 1 --2- 172.21.15.28:0/2714625069 >> v2:172.21.15.81:3300/0 conn(0x7f00300a97e0 0x7f003006be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:06.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.004+0000 7f002fe00640 1 -- 172.21.15.28:0/2714625069 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f003006e280 con 0x7f00300ac8d0 2024-02-21T23:06:06.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.004+0000 7f002fe00640 1 -- 172.21.15.28:0/2714625069 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f001c001670 con 0x7f00300ac8d0 2024-02-21T23:06:06.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.004+0000 7f002fe00640 1 -- 172.21.15.28:0/2714625069 >> v1:172.21.15.28:6789/0 conn(0x7f00300ac8d0 legacy=0x7f00300728c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:06.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.004+0000 7f002fe00640 1 --2- 172.21.15.28:0/2714625069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00180039b0 0x7f0018003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:06.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.004+0000 7f002fe00640 1 --2- 172.21.15.28:0/2714625069 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0018004800 0x7f0018004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:07.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.004+0000 7f002fe00640 1 --2- 172.21.15.28:0/2714625069 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f003014f3e0 0x7f00180051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:07.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.004+0000 7f002fe00640 1 -- 172.21.15.28:0/2714625069 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00300ac380 con 0x7f00180039b0 2024-02-21T23:06:07.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.005+0000 7f002fe00640 1 -- 172.21.15.28:0/2714625069 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00300ac5d0 con 0x7f0018004800 2024-02-21T23:06:07.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.005+0000 7f002fe00640 1 -- 172.21.15.28:0/2714625069 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f001800baa0 con 0x7f003014f3e0 2024-02-21T23:06:07.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.005+0000 7f0035c00640 1 --2- 172.21.15.28:0/2714625069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00180039b0 0x7f0018003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:07.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.005+0000 7f0035200640 1 --2- 172.21.15.28:0/2714625069 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0018004800 0x7f0018004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:07.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.005+0000 7f0036600640 1 --2- 172.21.15.28:0/2714625069 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f003014f3e0 0x7f00180051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:07.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.005+0000 7f0035c00640 1 -- 172.21.15.28:0/2714625069 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f003014f3e0 msgr2=0x7f00180051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:07.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.005+0000 7f0035c00640 1 --2- 172.21.15.28:0/2714625069 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f003014f3e0 0x7f00180051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:07.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.005+0000 7f0035c00640 1 -- 172.21.15.28:0/2714625069 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0018004800 msgr2=0x7f0018004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:07.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.005+0000 7f0035c00640 1 --2- 172.21.15.28:0/2714625069 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0018004800 0x7f0018004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:07.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.005+0000 7f0035c00640 1 -- 172.21.15.28:0/2714625069 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0018003520 con 0x7f00180039b0 2024-02-21T23:06:07.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.005+0000 7f0035200640 1 --2- 172.21.15.28:0/2714625069 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0018004800 0x7f0018004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:06:07.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.006+0000 7f0035c00640 1 --2- 172.21.15.28:0/2714625069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00180039b0 0x7f0018003dd0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f0024009a90 tx=0x7f0018005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86d9355f211b6053 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:07.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.007+0000 7f002fe00640 1 -- 172.21.15.28:0/2714625069 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0024039070 con 0x7f00180039b0 2024-02-21T23:06:07.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.007+0000 7f002fe00640 1 -- 172.21.15.28:0/2714625069 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f002402c680 con 0x7f00180039b0 2024-02-21T23:06:07.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.007+0000 7f002fe00640 1 -- 172.21.15.28:0/2714625069 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0024009690 con 0x7f00180039b0 2024-02-21T23:06:07.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.007+0000 7f0038600640 1 -- 172.21.15.28:0/2714625069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00180039b0 msgr2=0x7f0018003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:07.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.007+0000 7f0038600640 1 --2- 172.21.15.28:0/2714625069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00180039b0 0x7f0018003dd0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f0024009a90 tx=0x7f0018005830 comp rx=0 tx=0).stop 2024-02-21T23:06:07.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.007+0000 7f0036600640 1 -- 172.21.15.28:0/2714625069 reap_dead start 2024-02-21T23:06:07.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.007+0000 7f0038600640 1 -- 172.21.15.28:0/2714625069 shutdown_connections 2024-02-21T23:06:07.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.007+0000 7f0038600640 1 -- 172.21.15.28:0/2714625069 >> 172.21.15.28:0/2714625069 conn(0x7f0030071b00 msgr2=0x7f0030071ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:07.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.007+0000 7f0038600640 1 -- 172.21.15.28:0/2714625069 shutdown_connections 2024-02-21T23:06:07.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.007+0000 7f0038600640 1 -- 172.21.15.28:0/2714625069 wait complete. 2024-02-21T23:06:07.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.009+0000 7f0038600640 1 Processor -- start 2024-02-21T23:06:07.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.009+0000 7f0038600640 1 -- start start 2024-02-21T23:06:07.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.010+0000 7f0038600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00180039b0 0x7f00301517e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:07.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.010+0000 7f0038600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0030144180 0x7f00301eedf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:07.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.010+0000 7f0038600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0030142c40 0x7f00300c5cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:07.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.010+0000 7f0038600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f003006e560 con 0x7f00180039b0 2024-02-21T23:06:07.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.010+0000 7f0038600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f003014f7c0 con 0x7f0030144180 2024-02-21T23:06:07.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.010+0000 7f0038600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f003006e6e0 con 0x7f0030142c40 2024-02-21T23:06:07.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.010+0000 7f0035c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00180039b0 0x7f00301517e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:07.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.010+0000 7f0035c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00180039b0 0x7f00301517e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54584/0 (socket says 172.21.15.28:54584) 2024-02-21T23:06:07.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.010+0000 7f0035c00640 1 -- 172.21.15.28:0/1264440017 learned_addr learned my addr 172.21.15.28:0/1264440017 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:07.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.011+0000 7f0035200640 1 --2- 172.21.15.28:0/1264440017 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0030144180 0x7f00301eedf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:07.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.011+0000 7f0035c00640 1 -- 172.21.15.28:0/1264440017 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0030142c40 msgr2=0x7f00300c5cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:07.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.011+0000 7f0035c00640 1 --2- 172.21.15.28:0/1264440017 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0030142c40 0x7f00300c5cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:07.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.011+0000 7f0035c00640 1 -- 172.21.15.28:0/1264440017 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0030144180 msgr2=0x7f00301eedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:07.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.011+0000 7f0035c00640 1 --2- 172.21.15.28:0/1264440017 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0030144180 0x7f00301eedf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:07.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.011+0000 7f0035c00640 1 -- 172.21.15.28:0/1264440017 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00301445c0 con 0x7f00180039b0 2024-02-21T23:06:07.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.011+0000 7f0035c00640 1 --2- 172.21.15.28:0/1264440017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00180039b0 0x7f00301517e0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f0024009bb0 tx=0x7f0024004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:07.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.011+0000 7f002d600640 1 -- 172.21.15.28:0/1264440017 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0024039050 con 0x7f00180039b0 2024-02-21T23:06:07.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.011+0000 7f002d600640 1 -- 172.21.15.28:0/1264440017 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0024045070 con 0x7f00180039b0 2024-02-21T23:06:07.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.011+0000 7f0038600640 1 -- 172.21.15.28:0/1264440017 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0030144880 con 0x7f00180039b0 2024-02-21T23:06:07.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.011+0000 7f002d600640 1 -- 172.21.15.28:0/1264440017 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0024003850 con 0x7f00180039b0 2024-02-21T23:06:07.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.011+0000 7f0038600640 1 -- 172.21.15.28:0/1264440017 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0030150fe0 con 0x7f00180039b0 2024-02-21T23:06:07.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.013+0000 7f0038600640 1 -- 172.21.15.28:0/1264440017 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0030068640 con 0x7f00180039b0 2024-02-21T23:06:07.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.013+0000 7f002d600640 1 -- 172.21.15.28:0/1264440017 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f00240039f0 con 0x7f00180039b0 2024-02-21T23:06:07.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.014+0000 7f002d600640 1 --2- 172.21.15.28:0/1264440017 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f001006b4d0 0x7f001006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:07.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.014+0000 7f002d600640 1 -- 172.21.15.28:0/1264440017 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f00240ac690 con 0x7f00180039b0 2024-02-21T23:06:07.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.015+0000 7f0035200640 1 --2- 172.21.15.28:0/1264440017 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f001006b4d0 0x7f001006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:07.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.016+0000 7f0035200640 1 --2- 172.21.15.28:0/1264440017 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f001006b4d0 0x7f001006d990 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f001c000ff0 tx=0x7f001c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:07.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.018+0000 7f002d600640 1 -- 172.21.15.28:0/1264440017 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f002407b310 con 0x7f00180039b0 2024-02-21T23:06:07.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.217+0000 7f0038600640 1 -- 172.21.15.28:0/1264440017 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0030152700 con 0x7f00180039b0 2024-02-21T23:06:07.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.218+0000 7f002d600640 1 -- 172.21.15.28:0/1264440017 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v76) v1 ==== 100+0+11 (secure 0 0 0) 0x7f002407ac10 con 0x7f00180039b0 2024-02-21T23:06:07.207 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-02-21T23:06:07.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.220+0000 7f0038600640 1 -- 172.21.15.28:0/1264440017 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f001006b4d0 msgr2=0x7f001006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:07.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.221+0000 7f0038600640 1 --2- 172.21.15.28:0/1264440017 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f001006b4d0 0x7f001006d990 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f001c000ff0 tx=0x7f001c009040 comp rx=0 tx=0).stop 2024-02-21T23:06:07.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.221+0000 7f0038600640 1 -- 172.21.15.28:0/1264440017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00180039b0 msgr2=0x7f00301517e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:07.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.221+0000 7f0038600640 1 --2- 172.21.15.28:0/1264440017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00180039b0 0x7f00301517e0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f0024009bb0 tx=0x7f0024004730 comp rx=0 tx=0).stop 2024-02-21T23:06:07.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.221+0000 7f0038600640 1 -- 172.21.15.28:0/1264440017 shutdown_connections 2024-02-21T23:06:07.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.221+0000 7f0038600640 1 --2- 172.21.15.28:0/1264440017 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f001006b4d0 0x7f001006d990 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:07.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.221+0000 7f0038600640 1 --2- 172.21.15.28:0/1264440017 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0030142c40 0x7f00300c5cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:07.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.221+0000 7f0038600640 1 --2- 172.21.15.28:0/1264440017 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0030144180 0x7f00301eedf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:07.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.221+0000 7f0038600640 1 --2- 172.21.15.28:0/1264440017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00180039b0 0x7f00301517e0 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:07.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.221+0000 7f0038600640 1 -- 172.21.15.28:0/1264440017 >> 172.21.15.28:0/1264440017 conn(0x7f0030071b00 msgr2=0x7f0030074360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:07.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.221+0000 7f0038600640 1 -- 172.21.15.28:0/1264440017 shutdown_connections 2024-02-21T23:06:07.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.221+0000 7f0038600640 1 -- 172.21.15.28:0/1264440017 wait complete. 2024-02-21T23:06:07.232 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-02-21T23:06:07.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.495+0000 7f1141800640 1 Processor -- start 2024-02-21T23:06:07.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.496+0000 7f1141800640 1 -- start start 2024-02-21T23:06:07.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.497+0000 7f1141800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f113c152ab0 0x7f113c0c7fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:07.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.497+0000 7f1141800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f113c158850 0x7f113c158cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:07.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.497+0000 7f1141800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f113c0a8180 con 0x7f113c0c6710 2024-02-21T23:06:07.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.497+0000 7f1141800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f113c0c3ce0 con 0x7f113c152ab0 2024-02-21T23:06:07.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.497+0000 7f1141800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f113c0c33b0 con 0x7f113c158850 2024-02-21T23:06:07.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.497+0000 7f113aa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f113c152ab0 0x7f113c0c7fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:07.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.497+0000 7f113aa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f113c152ab0 0x7f113c0c7fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54588/0 (socket says 172.21.15.28:54588) 2024-02-21T23:06:07.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.497+0000 7f113aa00640 1 -- 172.21.15.28:0/1846248801 learned_addr learned my addr 172.21.15.28:0/1846248801 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:07.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.498+0000 7f113b400640 1 --2- 172.21.15.28:0/1846248801 >> v2:172.21.15.81:3300/0 conn(0x7f113c158850 0x7f113c158cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:07.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.498+0000 7f113aa00640 1 -- 172.21.15.28:0/1846248801 >> v2:172.21.15.81:3300/0 conn(0x7f113c158850 msgr2=0x7f113c158cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:07.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.498+0000 7f113aa00640 1 --2- 172.21.15.28:0/1846248801 >> v2:172.21.15.81:3300/0 conn(0x7f113c158850 0x7f113c158cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:07.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.498+0000 7f113aa00640 1 -- 172.21.15.28:0/1846248801 >> v1:172.21.15.81:6789/0 conn(0x7f113c0c6710 legacy=0x7f113c0c5210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:07.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.498+0000 7f113aa00640 1 -- 172.21.15.28:0/1846248801 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f113c0c3690 con 0x7f113c152ab0 2024-02-21T23:06:07.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.499+0000 7f113aa00640 1 --2- 172.21.15.28:0/1846248801 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f113c152ab0 0x7f113c0c7fe0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f113c072d70 tx=0x7f112c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce27d2187808a889 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:07.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.499+0000 7f113a000640 1 -- 172.21.15.28:0/1846248801 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f112c035020 con 0x7f113c152ab0 2024-02-21T23:06:07.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.499+0000 7f113a000640 1 -- 172.21.15.28:0/1846248801 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f112c031870 con 0x7f113c152ab0 2024-02-21T23:06:07.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.499+0000 7f113a000640 1 -- 172.21.15.28:0/1846248801 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f112c031c40 con 0x7f113c152ab0 2024-02-21T23:06:07.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.500+0000 7f1141800640 1 -- 172.21.15.28:0/1846248801 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f113c152ab0 msgr2=0x7f113c0c7fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:07.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.500+0000 7f1141800640 1 --2- 172.21.15.28:0/1846248801 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f113c152ab0 0x7f113c0c7fe0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f113c072d70 tx=0x7f112c02b910 comp rx=0 tx=0).stop 2024-02-21T23:06:07.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.500+0000 7f1141800640 1 -- 172.21.15.28:0/1846248801 shutdown_connections 2024-02-21T23:06:07.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.500+0000 7f1141800640 1 --2- 172.21.15.28:0/1846248801 >> v2:172.21.15.81:3300/0 conn(0x7f113c158850 0x7f113c158cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:07.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.500+0000 7f1141800640 1 --2- 172.21.15.28:0/1846248801 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f113c152ab0 0x7f113c0c7fe0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:07.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.500+0000 7f1141800640 1 -- 172.21.15.28:0/1846248801 >> 172.21.15.28:0/1846248801 conn(0x7f113c0c8810 msgr2=0x7f113c0c8bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:07.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.500+0000 7f1141800640 1 -- 172.21.15.28:0/1846248801 shutdown_connections 2024-02-21T23:06:07.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.500+0000 7f1141800640 1 -- 172.21.15.28:0/1846248801 wait complete. 2024-02-21T23:06:07.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.502+0000 7f1141800640 1 Processor -- start 2024-02-21T23:06:07.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.502+0000 7f1141800640 1 -- start start 2024-02-21T23:06:07.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.503+0000 7f1141800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f113c0c6710 0x7f113c1eeb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:07.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.503+0000 7f1141800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f113c152ab0 0x7f113c1e0a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:07.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.503+0000 7f1141800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f113c158850 0x7f113c06f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:07.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.503+0000 7f1141800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f113c0c5aa0 con 0x7f113c0c6710 2024-02-21T23:06:07.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.503+0000 7f1141800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f113c0c5920 con 0x7f113c158850 2024-02-21T23:06:07.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.503+0000 7f1141800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f113c159200 con 0x7f113c152ab0 2024-02-21T23:06:07.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.503+0000 7f113aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f113c0c6710 0x7f113c1eeb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:07.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.503+0000 7f113aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f113c0c6710 0x7f113c1eeb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54602/0 (socket says 172.21.15.28:54602) 2024-02-21T23:06:07.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.503+0000 7f113aa00640 1 -- 172.21.15.28:0/1713088408 learned_addr learned my addr 172.21.15.28:0/1713088408 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:07.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.504+0000 7f1133e00640 1 --2- 172.21.15.28:0/1713088408 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f113c152ab0 0x7f113c1e0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:07.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.504+0000 7f113b400640 1 --2- 172.21.15.28:0/1713088408 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f113c158850 0x7f113c06f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:07.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.504+0000 7f113aa00640 1 -- 172.21.15.28:0/1713088408 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f113c152ab0 msgr2=0x7f113c1e0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:07.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.504+0000 7f113aa00640 1 --2- 172.21.15.28:0/1713088408 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f113c152ab0 0x7f113c1e0a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:07.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.504+0000 7f113aa00640 1 -- 172.21.15.28:0/1713088408 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f113c158850 msgr2=0x7f113c06f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:07.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.504+0000 7f113aa00640 1 --2- 172.21.15.28:0/1713088408 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f113c158850 0x7f113c06f330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:07.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.504+0000 7f113aa00640 1 -- 172.21.15.28:0/1713088408 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f113c150730 con 0x7f113c0c6710 2024-02-21T23:06:07.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.504+0000 7f113b400640 1 --2- 172.21.15.28:0/1713088408 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f113c158850 0x7f113c06f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:06:07.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.504+0000 7f113aa00640 1 --2- 172.21.15.28:0/1713088408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f113c0c6710 0x7f113c1eeb50 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f112c02b950 tx=0x7f112c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:07.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.504+0000 7f1132a00640 1 -- 172.21.15.28:0/1713088408 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f112c040070 con 0x7f113c0c6710 2024-02-21T23:06:07.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.505+0000 7f1132a00640 1 -- 172.21.15.28:0/1713088408 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f112c03a420 con 0x7f113c0c6710 2024-02-21T23:06:07.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.505+0000 7f1132a00640 1 -- 172.21.15.28:0/1713088408 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f112c035040 con 0x7f113c0c6710 2024-02-21T23:06:07.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.505+0000 7f1141800640 1 -- 172.21.15.28:0/1713088408 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f113c1ee070 con 0x7f113c0c6710 2024-02-21T23:06:07.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.505+0000 7f1141800640 1 -- 172.21.15.28:0/1713088408 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f113c1ee630 con 0x7f113c0c6710 2024-02-21T23:06:07.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.506+0000 7f1141800640 1 -- 172.21.15.28:0/1713088408 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f113c0c6190 con 0x7f113c0c6710 2024-02-21T23:06:07.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.508+0000 7f1132a00640 1 -- 172.21.15.28:0/1713088408 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f112c038020 con 0x7f113c0c6710 2024-02-21T23:06:07.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.509+0000 7f1132a00640 1 --2- 172.21.15.28:0/1713088408 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f111806b4d0 0x7f111806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:07.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.509+0000 7f1132a00640 1 -- 172.21.15.28:0/1713088408 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f112c0ac360 con 0x7f113c0c6710 2024-02-21T23:06:07.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.510+0000 7f1133e00640 1 --2- 172.21.15.28:0/1713088408 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f111806b4d0 0x7f111806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:07.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.511+0000 7f1133e00640 1 --2- 172.21.15.28:0/1713088408 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f111806b4d0 0x7f111806d990 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f1124007fd0 tx=0x7f1124009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:07.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.514+0000 7f1132a00640 1 -- 172.21.15.28:0/1713088408 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f113c0a8180 con 0x7f113c0c6710 2024-02-21T23:06:07.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.716+0000 7f1141800640 1 -- 172.21.15.28:0/1713088408 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f113c1e0f80 con 0x7f113c0c6710 2024-02-21T23:06:07.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.717+0000 7f1132a00640 1 -- 172.21.15.28:0/1713088408 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v76) v1 ==== 84+0+209 (secure 0 0 0) 0x7f112c07b0e0 con 0x7f113c0c6710 2024-02-21T23:06:07.707 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:06:07.707 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-02-21T23:06:07.707 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:06:07.707 INFO:teuthology.orchestra.run.smithi028.stdout: "poolnum": 1, 2024-02-21T23:06:07.707 INFO:teuthology.orchestra.run.smithi028.stdout: "poolname": ".mgr" 2024-02-21T23:06:07.707 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:06:07.707 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:06:07.707 INFO:teuthology.orchestra.run.smithi028.stdout: "poolnum": 11, 2024-02-21T23:06:07.707 INFO:teuthology.orchestra.run.smithi028.stdout: "poolname": "cephfs_metadata" 2024-02-21T23:06:07.708 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:06:07.708 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:06:07.708 INFO:teuthology.orchestra.run.smithi028.stdout: "poolnum": 12, 2024-02-21T23:06:07.708 INFO:teuthology.orchestra.run.smithi028.stdout: "poolname": "cephfs_data" 2024-02-21T23:06:07.708 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:06:07.708 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-02-21T23:06:07.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.720+0000 7f1141800640 1 -- 172.21.15.28:0/1713088408 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f111806b4d0 msgr2=0x7f111806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:07.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.720+0000 7f1141800640 1 --2- 172.21.15.28:0/1713088408 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f111806b4d0 0x7f111806d990 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f1124007fd0 tx=0x7f1124009040 comp rx=0 tx=0).stop 2024-02-21T23:06:07.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.720+0000 7f1141800640 1 -- 172.21.15.28:0/1713088408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f113c0c6710 msgr2=0x7f113c1eeb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:07.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.720+0000 7f1141800640 1 --2- 172.21.15.28:0/1713088408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f113c0c6710 0x7f113c1eeb50 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f112c02b950 tx=0x7f112c0046c0 comp rx=0 tx=0).stop 2024-02-21T23:06:07.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.720+0000 7f1141800640 1 -- 172.21.15.28:0/1713088408 shutdown_connections 2024-02-21T23:06:07.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.720+0000 7f1141800640 1 --2- 172.21.15.28:0/1713088408 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f111806b4d0 0x7f111806d990 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:07.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.720+0000 7f1141800640 1 --2- 172.21.15.28:0/1713088408 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f113c152ab0 0x7f113c1e0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:07.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.720+0000 7f1141800640 1 --2- 172.21.15.28:0/1713088408 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f113c158850 0x7f113c06f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:07.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.720+0000 7f1141800640 1 --2- 172.21.15.28:0/1713088408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f113c0c6710 0x7f113c1eeb50 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:07.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.720+0000 7f1141800640 1 -- 172.21.15.28:0/1713088408 >> 172.21.15.28:0/1713088408 conn(0x7f113c0c8810 msgr2=0x7f113c153720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:07.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.720+0000 7f1141800640 1 -- 172.21.15.28:0/1713088408 shutdown_connections 2024-02-21T23:06:07.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.720+0000 7f1141800640 1 -- 172.21.15.28:0/1713088408 wait complete. 2024-02-21T23:06:07.727 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs2' 2024-02-21T23:06:07.727 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs2_metadata --pg_num_min 64 2024-02-21T23:06:07.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.954+0000 7fa384800640 1 Processor -- start 2024-02-21T23:06:07.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.955+0000 7fa384800640 1 -- start start 2024-02-21T23:06:07.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.956+0000 7fa384800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa37c0bf970 0x7fa37c0bfff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:07.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.956+0000 7fa384800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fa37c0caca0 0x7fa37c158de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:07.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.956+0000 7fa384800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa37c0c6bb0 con 0x7fa37c0ca8c0 2024-02-21T23:06:07.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.956+0000 7fa384800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa37c0cb130 con 0x7fa37c0bf970 2024-02-21T23:06:07.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.956+0000 7fa384800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa37c0c6e90 con 0x7fa37c0caca0 2024-02-21T23:06:07.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.957+0000 7fa381400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa37c0bf970 0x7fa37c0bfff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:07.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.957+0000 7fa381400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa37c0bf970 0x7fa37c0bfff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54604/0 (socket says 172.21.15.28:54604) 2024-02-21T23:06:07.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.957+0000 7fa381400640 1 -- 172.21.15.28:0/1600785615 learned_addr learned my addr 172.21.15.28:0/1600785615 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:07.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.957+0000 7fa381e00640 1 --1- 172.21.15.28:0/1600785615 >> v1:172.21.15.28:6789/0 conn(0x7fa37c0ca8c0 0x7fa37c155a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59800/0 (socket says 172.21.15.28:59800) 2024-02-21T23:06:07.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.957+0000 7fa382800640 1 --2- 172.21.15.28:0/1600785615 >> v2:172.21.15.81:3300/0 conn(0x7fa37c0caca0 0x7fa37c158de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:07.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.957+0000 7fa381400640 1 -- 172.21.15.28:0/1600785615 >> v2:172.21.15.81:3300/0 conn(0x7fa37c0caca0 msgr2=0x7fa37c158de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:07.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.957+0000 7fa381400640 1 --2- 172.21.15.28:0/1600785615 >> v2:172.21.15.81:3300/0 conn(0x7fa37c0caca0 0x7fa37c158de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:07.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.957+0000 7fa381400640 1 -- 172.21.15.28:0/1600785615 >> v1:172.21.15.28:6789/0 conn(0x7fa37c0ca8c0 legacy=0x7fa37c155a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:07.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.957+0000 7fa381400640 1 -- 172.21.15.28:0/1600785615 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa37c0c7170 con 0x7fa37c0bf970 2024-02-21T23:06:07.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.959+0000 7fa381400640 1 --2- 172.21.15.28:0/1600785615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa37c0bf970 0x7fa37c0bfff0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fa37c072df0 tx=0x7fa36c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb12e8f8c0b296a7 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:07.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.959+0000 7fa380a00640 1 -- 172.21.15.28:0/1600785615 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa36c02e070 con 0x7fa37c0bf970 2024-02-21T23:06:07.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.959+0000 7fa380a00640 1 -- 172.21.15.28:0/1600785615 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa36c037070 con 0x7fa37c0bf970 2024-02-21T23:06:07.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.959+0000 7fa380a00640 1 -- 172.21.15.28:0/1600785615 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa36c031a00 con 0x7fa37c0bf970 2024-02-21T23:06:07.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.959+0000 7fa384800640 1 -- 172.21.15.28:0/1600785615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa37c0bf970 msgr2=0x7fa37c0bfff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:07.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.959+0000 7fa384800640 1 --2- 172.21.15.28:0/1600785615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa37c0bf970 0x7fa37c0bfff0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fa37c072df0 tx=0x7fa36c02b880 comp rx=0 tx=0).stop 2024-02-21T23:06:07.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.960+0000 7fa384800640 1 -- 172.21.15.28:0/1600785615 shutdown_connections 2024-02-21T23:06:07.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.960+0000 7fa384800640 1 --2- 172.21.15.28:0/1600785615 >> v2:172.21.15.81:3300/0 conn(0x7fa37c0caca0 0x7fa37c158de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:07.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.960+0000 7fa384800640 1 --2- 172.21.15.28:0/1600785615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa37c0bf970 0x7fa37c0bfff0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:07.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.960+0000 7fa384800640 1 -- 172.21.15.28:0/1600785615 >> 172.21.15.28:0/1600785615 conn(0x7fa37c08b1c0 msgr2=0x7fa37c0725a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:07.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.960+0000 7fa384800640 1 -- 172.21.15.28:0/1600785615 shutdown_connections 2024-02-21T23:06:07.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.960+0000 7fa384800640 1 -- 172.21.15.28:0/1600785615 wait complete. 2024-02-21T23:06:07.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.961+0000 7fa384800640 1 Processor -- start 2024-02-21T23:06:07.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.962+0000 7fa384800640 1 -- start start 2024-02-21T23:06:07.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.962+0000 7fa384800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa37c0bf970 0x7fa37c1ee020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:07.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.962+0000 7fa384800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa37c0ca8c0 0x7fa37c1ec140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:07.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.962+0000 7fa384800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa37c0caca0 0x7fa37c1ec680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:07.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.962+0000 7fa384800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa37c14f370 con 0x7fa37c0ca8c0 2024-02-21T23:06:07.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.962+0000 7fa384800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa37c1588c0 con 0x7fa37c0caca0 2024-02-21T23:06:07.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.962+0000 7fa384800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa37c14f4f0 con 0x7fa37c0bf970 2024-02-21T23:06:07.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.963+0000 7fa381400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa37c0ca8c0 0x7fa37c1ec140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:07.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.963+0000 7fa381400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa37c0ca8c0 0x7fa37c1ec140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54614/0 (socket says 172.21.15.28:54614) 2024-02-21T23:06:07.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.963+0000 7fa381400640 1 -- 172.21.15.28:0/1784980245 learned_addr learned my addr 172.21.15.28:0/1784980245 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:07.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.963+0000 7fa382800640 1 --2- 172.21.15.28:0/1784980245 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa37c0caca0 0x7fa37c1ec680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:07.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.963+0000 7fa381e00640 1 --2- 172.21.15.28:0/1784980245 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa37c0bf970 0x7fa37c1ee020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:07.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.963+0000 7fa381400640 1 -- 172.21.15.28:0/1784980245 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa37c0bf970 msgr2=0x7fa37c1ee020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:07.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.963+0000 7fa381400640 1 --2- 172.21.15.28:0/1784980245 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa37c0bf970 0x7fa37c1ee020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:07.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.963+0000 7fa381400640 1 -- 172.21.15.28:0/1784980245 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa37c0caca0 msgr2=0x7fa37c1ec680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:07.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.963+0000 7fa381400640 1 --2- 172.21.15.28:0/1784980245 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa37c0caca0 0x7fa37c1ec680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:07.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.963+0000 7fa381400640 1 -- 172.21.15.28:0/1784980245 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa37c1dfc00 con 0x7fa37c0ca8c0 2024-02-21T23:06:07.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.963+0000 7fa382800640 1 --2- 172.21.15.28:0/1784980245 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa37c0caca0 0x7fa37c1ec680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:06:07.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.964+0000 7fa381400640 1 --2- 172.21.15.28:0/1784980245 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa37c0ca8c0 0x7fa37c1ec140 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fa36c02b8c0 tx=0x7fa36c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:07.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.964+0000 7fa376000640 1 -- 172.21.15.28:0/1784980245 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa36c032880 con 0x7fa37c0ca8c0 2024-02-21T23:06:07.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.964+0000 7fa376000640 1 -- 172.21.15.28:0/1784980245 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa36c032a40 con 0x7fa37c0ca8c0 2024-02-21T23:06:07.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.964+0000 7fa376000640 1 -- 172.21.15.28:0/1784980245 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa36c032e30 con 0x7fa37c0ca8c0 2024-02-21T23:06:07.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.964+0000 7fa384800640 1 -- 172.21.15.28:0/1784980245 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa37c1de780 con 0x7fa37c0ca8c0 2024-02-21T23:06:07.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.964+0000 7fa384800640 1 -- 172.21.15.28:0/1784980245 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa37c1eeb40 con 0x7fa37c0ca8c0 2024-02-21T23:06:07.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.965+0000 7fa384800640 1 -- 172.21.15.28:0/1784980245 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa37c067f30 con 0x7fa37c0ca8c0 2024-02-21T23:06:07.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.966+0000 7fa376000640 1 -- 172.21.15.28:0/1784980245 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa36c04c060 con 0x7fa37c0ca8c0 2024-02-21T23:06:07.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.968+0000 7fa376000640 1 --2- 172.21.15.28:0/1784980245 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa36006b4d0 0x7fa36006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:07.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.968+0000 7fa376000640 1 -- 172.21.15.28:0/1784980245 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa36c0abdf0 con 0x7fa37c0ca8c0 2024-02-21T23:06:07.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.969+0000 7fa381e00640 1 --2- 172.21.15.28:0/1784980245 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa36006b4d0 0x7fa36006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:07.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.970+0000 7fa381e00640 1 --2- 172.21.15.28:0/1784980245 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa36006b4d0 0x7fa36006d990 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fa37c1e2620 tx=0x7fa370009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:07.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:07.972+0000 7fa376000640 1 -- 172.21.15.28:0/1784980245 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fa36c040070 con 0x7fa37c0ca8c0 2024-02-21T23:06:08.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:08.171+0000 7fa384800640 1 -- 172.21.15.28:0/1784980245 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs2_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fa37c073390 con 0x7fa37c0ca8c0 2024-02-21T23:06:08.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:08.905+0000 7fa376000640 1 -- 172.21.15.28:0/1784980245 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs2_metadata", "pg_num_min": 64}]=0 pool 'cephfs2_metadata' created v77) v1 ==== 140+0+0 (secure 0 0 0) 0x7fa36c07ac90 con 0x7fa37c0ca8c0 2024-02-21T23:06:08.895 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs2_metadata' created 2024-02-21T23:06:08.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:08.909+0000 7fa384800640 1 -- 172.21.15.28:0/1784980245 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa36006b4d0 msgr2=0x7fa36006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:08.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:08.909+0000 7fa384800640 1 --2- 172.21.15.28:0/1784980245 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa36006b4d0 0x7fa36006d990 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fa37c1e2620 tx=0x7fa370009040 comp rx=0 tx=0).stop 2024-02-21T23:06:08.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:08.909+0000 7fa384800640 1 -- 172.21.15.28:0/1784980245 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa37c0ca8c0 msgr2=0x7fa37c1ec140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:08.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:08.909+0000 7fa384800640 1 --2- 172.21.15.28:0/1784980245 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa37c0ca8c0 0x7fa37c1ec140 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fa36c02b8c0 tx=0x7fa36c0046c0 comp rx=0 tx=0).stop 2024-02-21T23:06:08.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:08.910+0000 7fa384800640 1 -- 172.21.15.28:0/1784980245 shutdown_connections 2024-02-21T23:06:08.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:08.910+0000 7fa384800640 1 --2- 172.21.15.28:0/1784980245 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa36006b4d0 0x7fa36006d990 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:08.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:08.910+0000 7fa384800640 1 --2- 172.21.15.28:0/1784980245 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa37c0caca0 0x7fa37c1ec680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:08.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:08.910+0000 7fa384800640 1 --2- 172.21.15.28:0/1784980245 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa37c0ca8c0 0x7fa37c1ec140 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:08.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:08.910+0000 7fa384800640 1 --2- 172.21.15.28:0/1784980245 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa37c0bf970 0x7fa37c1ee020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:08.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:08.910+0000 7fa384800640 1 -- 172.21.15.28:0/1784980245 >> 172.21.15.28:0/1784980245 conn(0x7fa37c08b1c0 msgr2=0x7fa37c159000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:08.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:08.910+0000 7fa384800640 1 -- 172.21.15.28:0/1784980245 shutdown_connections 2024-02-21T23:06:08.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:08.910+0000 7fa384800640 1 -- 172.21.15.28:0/1784980245 wait complete. 2024-02-21T23:06:08.925 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs2_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-02-21T23:06:09.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.140+0000 7f3072200640 1 Processor -- start 2024-02-21T23:06:09.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.140+0000 7f3072200640 1 -- start start 2024-02-21T23:06:09.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.142+0000 7f3072200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f306c0ab170 0x7f306c158c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:09.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.142+0000 7f3072200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f306c0c3b40 0x7f306c1591a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:09.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.142+0000 7f3072200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f306c0c8170 con 0x7f306c152a30 2024-02-21T23:06:09.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.142+0000 7f3072200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f306c1561a0 con 0x7f306c0c3b40 2024-02-21T23:06:09.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.142+0000 7f3072200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f306c0c7f40 con 0x7f306c0ab170 2024-02-21T23:06:09.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.142+0000 7f306be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f306c0c3b40 0x7f306c1591a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:09.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.142+0000 7f306be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f306c0c3b40 0x7f306c1591a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54622/0 (socket says 172.21.15.28:54622) 2024-02-21T23:06:09.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.142+0000 7f306be00640 1 -- 172.21.15.28:0/2614092646 learned_addr learned my addr 172.21.15.28:0/2614092646 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:09.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.143+0000 7f3063e00640 1 --2- 172.21.15.28:0/2614092646 >> v2:172.21.15.81:3300/0 conn(0x7f306c0ab170 0x7f306c158c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:09.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.143+0000 7f306be00640 1 -- 172.21.15.28:0/2614092646 >> v2:172.21.15.81:3300/0 conn(0x7f306c0ab170 msgr2=0x7f306c158c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:09.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.143+0000 7f306be00640 1 --2- 172.21.15.28:0/2614092646 >> v2:172.21.15.81:3300/0 conn(0x7f306c0ab170 0x7f306c158c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:09.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.143+0000 7f306be00640 1 -- 172.21.15.28:0/2614092646 >> v1:172.21.15.81:6789/0 conn(0x7f306c152a30 legacy=0x7f306c0c8690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:09.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.143+0000 7f306be00640 1 -- 172.21.15.28:0/2614092646 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f306c0c3130 con 0x7f306c0c3b40 2024-02-21T23:06:09.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.144+0000 7f306be00640 1 --2- 172.21.15.28:0/2614092646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f306c0c3b40 0x7f306c1591a0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f306c06bdf0 tx=0x7f306402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c368999482d6bb5b server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:09.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.145+0000 7f306aa00640 1 -- 172.21.15.28:0/2614092646 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f306402ca80 con 0x7f306c0c3b40 2024-02-21T23:06:09.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.145+0000 7f306aa00640 1 -- 172.21.15.28:0/2614092646 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f306402cc20 con 0x7f306c0c3b40 2024-02-21T23:06:09.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.145+0000 7f306aa00640 1 -- 172.21.15.28:0/2614092646 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30640343f0 con 0x7f306c0c3b40 2024-02-21T23:06:09.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.145+0000 7f3072200640 1 -- 172.21.15.28:0/2614092646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f306c0c3b40 msgr2=0x7f306c1591a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:09.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.145+0000 7f3072200640 1 --2- 172.21.15.28:0/2614092646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f306c0c3b40 0x7f306c1591a0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f306c06bdf0 tx=0x7f306402bee0 comp rx=0 tx=0).stop 2024-02-21T23:06:09.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.145+0000 7f3072200640 1 -- 172.21.15.28:0/2614092646 shutdown_connections 2024-02-21T23:06:09.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.145+0000 7f3072200640 1 --2- 172.21.15.28:0/2614092646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f306c0c3b40 0x7f306c1591a0 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:09.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.145+0000 7f3072200640 1 --2- 172.21.15.28:0/2614092646 >> v2:172.21.15.81:3300/0 conn(0x7f306c0ab170 0x7f306c158c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:09.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.146+0000 7f3072200640 1 -- 172.21.15.28:0/2614092646 >> 172.21.15.28:0/2614092646 conn(0x7f306c06b4a0 msgr2=0x7f306c0aa4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:09.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.146+0000 7f3072200640 1 -- 172.21.15.28:0/2614092646 shutdown_connections 2024-02-21T23:06:09.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.146+0000 7f3072200640 1 -- 172.21.15.28:0/2614092646 wait complete. 2024-02-21T23:06:09.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.147+0000 7f3072200640 1 Processor -- start 2024-02-21T23:06:09.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.147+0000 7f3072200640 1 -- start start 2024-02-21T23:06:09.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.148+0000 7f3072200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f306c0ab170 0x7f306c1ebbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:09.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.148+0000 7f3072200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f306c0c3b40 0x7f306c154300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:09.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.148+0000 7f3072200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f306c152a30 0x7f306c1ee670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:09.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.148+0000 7f3072200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f306c0c36f0 con 0x7f306c0c3b40 2024-02-21T23:06:09.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.148+0000 7f3072200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f306c1587d0 con 0x7f306c152a30 2024-02-21T23:06:09.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.148+0000 7f3072200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f306c0c7c60 con 0x7f306c0ab170 2024-02-21T23:06:09.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.148+0000 7f3063e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f306c0c3b40 0x7f306c154300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:09.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.149+0000 7f3063e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f306c0c3b40 0x7f306c154300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54634/0 (socket says 172.21.15.28:54634) 2024-02-21T23:06:09.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.149+0000 7f3063e00640 1 -- 172.21.15.28:0/1491249385 learned_addr learned my addr 172.21.15.28:0/1491249385 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:09.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.149+0000 7f306b400640 1 --2- 172.21.15.28:0/1491249385 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f306c0ab170 0x7f306c1ebbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:09.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.149+0000 7f306be00640 1 --2- 172.21.15.28:0/1491249385 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f306c152a30 0x7f306c1ee670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:09.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.149+0000 7f3063e00640 1 -- 172.21.15.28:0/1491249385 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f306c0ab170 msgr2=0x7f306c1ebbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:09.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.149+0000 7f3063e00640 1 --2- 172.21.15.28:0/1491249385 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f306c0ab170 0x7f306c1ebbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:09.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.149+0000 7f3063e00640 1 -- 172.21.15.28:0/1491249385 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f306c152a30 msgr2=0x7f306c1ee670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:09.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.149+0000 7f3063e00640 1 --2- 172.21.15.28:0/1491249385 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f306c152a30 0x7f306c1ee670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:09.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.149+0000 7f3063e00640 1 -- 172.21.15.28:0/1491249385 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f306c1eb100 con 0x7f306c0c3b40 2024-02-21T23:06:09.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.149+0000 7f3063e00640 1 --2- 172.21.15.28:0/1491249385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f306c0c3b40 0x7f306c154300 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f305400e980 tx=0x7f305400ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:09.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.150+0000 7f3063400640 1 -- 172.21.15.28:0/1491249385 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3054017070 con 0x7f306c0c3b40 2024-02-21T23:06:09.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.150+0000 7f3063400640 1 -- 172.21.15.28:0/1491249385 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f305400ca00 con 0x7f306c0c3b40 2024-02-21T23:06:09.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.150+0000 7f3072200640 1 -- 172.21.15.28:0/1491249385 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f306c0acae0 con 0x7f306c0c3b40 2024-02-21T23:06:09.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.150+0000 7f3063400640 1 -- 172.21.15.28:0/1491249385 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f305400cdd0 con 0x7f306c0c3b40 2024-02-21T23:06:09.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.150+0000 7f3072200640 1 -- 172.21.15.28:0/1491249385 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f306c1eb3c0 con 0x7f306c0c3b40 2024-02-21T23:06:09.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.151+0000 7f3072200640 1 -- 172.21.15.28:0/1491249385 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f306c0a7e80 con 0x7f306c0c3b40 2024-02-21T23:06:09.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.153+0000 7f3063400640 1 -- 172.21.15.28:0/1491249385 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f305401a070 con 0x7f306c0c3b40 2024-02-21T23:06:09.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.154+0000 7f3063400640 1 --2- 172.21.15.28:0/1491249385 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f304806b4d0 0x7f304806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:09.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.154+0000 7f3063400640 1 -- 172.21.15.28:0/1491249385 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 6472+0+0 (secure 0 0 0) 0x7f305408ab60 con 0x7f306c0c3b40 2024-02-21T23:06:09.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.155+0000 7f306b400640 1 --2- 172.21.15.28:0/1491249385 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f304806b4d0 0x7f304806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:09.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.156+0000 7f306b400640 1 --2- 172.21.15.28:0/1491249385 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f304806b4d0 0x7f304806d990 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f305c007fd0 tx=0x7f305c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:09.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.159+0000 7f3063400640 1 -- 172.21.15.28:0/1491249385 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f306c0a7e80 con 0x7f306c0c3b40 2024-02-21T23:06:09.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.381+0000 7f3072200640 1 -- 172.21.15.28:0/1491249385 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs2_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f306c067f30 con 0x7f306c0c3b40 2024-02-21T23:06:09.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.912+0000 7f3063400640 1 -- 172.21.15.28:0/1491249385 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs2_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs2_data' created v78) v1 ==== 172+0+0 (secure 0 0 0) 0x7f3054059850 con 0x7f306c0c3b40 2024-02-21T23:06:09.902 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs2_data' created 2024-02-21T23:06:09.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.916+0000 7f3072200640 1 -- 172.21.15.28:0/1491249385 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f304806b4d0 msgr2=0x7f304806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:09.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.916+0000 7f3072200640 1 --2- 172.21.15.28:0/1491249385 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f304806b4d0 0x7f304806d990 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f305c007fd0 tx=0x7f305c009040 comp rx=0 tx=0).stop 2024-02-21T23:06:09.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.916+0000 7f3072200640 1 -- 172.21.15.28:0/1491249385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f306c0c3b40 msgr2=0x7f306c154300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:09.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.916+0000 7f3072200640 1 --2- 172.21.15.28:0/1491249385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f306c0c3b40 0x7f306c154300 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f305400e980 tx=0x7f305400ee40 comp rx=0 tx=0).stop 2024-02-21T23:06:09.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.918+0000 7f3072200640 1 -- 172.21.15.28:0/1491249385 shutdown_connections 2024-02-21T23:06:09.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.918+0000 7f3072200640 1 --2- 172.21.15.28:0/1491249385 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f304806b4d0 0x7f304806d990 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:09.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.918+0000 7f3072200640 1 --2- 172.21.15.28:0/1491249385 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f306c152a30 0x7f306c1ee670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:09.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.918+0000 7f3072200640 1 --2- 172.21.15.28:0/1491249385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f306c0c3b40 0x7f306c154300 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:09.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.918+0000 7f3072200640 1 --2- 172.21.15.28:0/1491249385 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f306c0ab170 0x7f306c1ebbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:09.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.918+0000 7f3072200640 1 -- 172.21.15.28:0/1491249385 >> 172.21.15.28:0/1491249385 conn(0x7f306c06b4a0 msgr2=0x7f306c0c6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:09.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.919+0000 7f3072200640 1 -- 172.21.15.28:0/1491249385 shutdown_connections 2024-02-21T23:06:09.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:09.919+0000 7f3072200640 1 -- 172.21.15.28:0/1491249385 wait complete. 2024-02-21T23:06:09.941 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs2 cephfs2_metadata cephfs2_data 2024-02-21T23:06:10.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.136+0000 7f0524000640 1 Processor -- start 2024-02-21T23:06:10.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.136+0000 7f0524000640 1 -- start start 2024-02-21T23:06:10.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.138+0000 7f0524000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f051c06b0e0 0x7f051c0c5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:10.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.138+0000 7f0524000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f051c150700 0x7f051c156500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:10.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.138+0000 7f0524000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f051c0c61e0 con 0x7f051c0c5d80 2024-02-21T23:06:10.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.138+0000 7f0524000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f051c153bc0 con 0x7f051c150700 2024-02-21T23:06:10.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.138+0000 7f0524000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f051c152fb0 con 0x7f051c06b0e0 2024-02-21T23:06:10.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.139+0000 7f0522000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f051c150700 0x7f051c156500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:10.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.139+0000 7f0521600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f051c06b0e0 0x7f051c0c5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:10.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.139+0000 7f0521600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f051c06b0e0 0x7f051c0c5840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3301/0 says I am v2:172.21.15.28:42160/0 (socket says 172.21.15.28:42160) 2024-02-21T23:06:10.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.139+0000 7f0522000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f051c150700 0x7f051c156500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.28:51180/0 (socket says 172.21.15.28:51180) 2024-02-21T23:06:10.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.139+0000 7f0521600640 1 -- 172.21.15.28:0/1507814045 learned_addr learned my addr 172.21.15.28:0/1507814045 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:10.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.140+0000 7f0521600640 1 -- 172.21.15.28:0/1507814045 >> v2:172.21.15.81:3300/0 conn(0x7f051c150700 msgr2=0x7f051c156500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:10.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.140+0000 7f0521600640 1 --2- 172.21.15.28:0/1507814045 >> v2:172.21.15.81:3300/0 conn(0x7f051c150700 0x7f051c156500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:10.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.140+0000 7f0521600640 1 -- 172.21.15.28:0/1507814045 >> v1:172.21.15.81:6789/0 conn(0x7f051c0c5d80 legacy=0x7f051c14fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:10.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.140+0000 7f0521600640 1 -- 172.21.15.28:0/1507814045 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f051c153290 con 0x7f051c06b0e0 2024-02-21T23:06:10.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.141+0000 7f0521600640 1 --2- 172.21.15.28:0/1507814045 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f051c06b0e0 0x7f051c0c5840 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0510009870 tx=0x7f051002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=acea0c54bff98c7b server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:10.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.142+0000 7f051be00640 1 -- 172.21.15.28:0/1507814045 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0510035020 con 0x7f051c06b0e0 2024-02-21T23:06:10.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.142+0000 7f051be00640 1 -- 172.21.15.28:0/1507814045 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0510031640 con 0x7f051c06b0e0 2024-02-21T23:06:10.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.142+0000 7f051be00640 1 -- 172.21.15.28:0/1507814045 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f051002d040 con 0x7f051c06b0e0 2024-02-21T23:06:10.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.142+0000 7f0524000640 1 -- 172.21.15.28:0/1507814045 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f051c06b0e0 msgr2=0x7f051c0c5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:10.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.142+0000 7f0524000640 1 --2- 172.21.15.28:0/1507814045 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f051c06b0e0 0x7f051c0c5840 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0510009870 tx=0x7f051002b8f0 comp rx=0 tx=0).stop 2024-02-21T23:06:10.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.142+0000 7f0524000640 1 -- 172.21.15.28:0/1507814045 shutdown_connections 2024-02-21T23:06:10.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.142+0000 7f0524000640 1 --2- 172.21.15.28:0/1507814045 >> v2:172.21.15.81:3300/0 conn(0x7f051c150700 0x7f051c156500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:10.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.142+0000 7f0524000640 1 --2- 172.21.15.28:0/1507814045 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f051c06b0e0 0x7f051c0c5840 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:10.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.142+0000 7f0524000640 1 -- 172.21.15.28:0/1507814045 >> 172.21.15.28:0/1507814045 conn(0x7f051c0a85a0 msgr2=0x7f051c0a8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:10.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.142+0000 7f0524000640 1 -- 172.21.15.28:0/1507814045 shutdown_connections 2024-02-21T23:06:10.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.142+0000 7f0524000640 1 -- 172.21.15.28:0/1507814045 wait complete. 2024-02-21T23:06:10.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.144+0000 7f0524000640 1 Processor -- start 2024-02-21T23:06:10.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.144+0000 7f0524000640 1 -- start start 2024-02-21T23:06:10.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.145+0000 7f0524000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f051c06b0e0 0x7f051c1eabf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:10.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.145+0000 7f0524000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f051c0c5d80 0x7f051c155480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:10.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.145+0000 7f0524000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f051c150700 0x7f051c14dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:10.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.145+0000 7f0524000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f051c06b4c0 con 0x7f051c06b0e0 2024-02-21T23:06:10.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.145+0000 7f0524000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f051c153850 con 0x7f051c0c5d80 2024-02-21T23:06:10.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.145+0000 7f0524000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f051c06b640 con 0x7f051c150700 2024-02-21T23:06:10.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.145+0000 7f0521600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f051c06b0e0 0x7f051c1eabf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:10.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.145+0000 7f0521600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f051c06b0e0 0x7f051c1eabf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54636/0 (socket says 172.21.15.28:54636) 2024-02-21T23:06:10.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.145+0000 7f0521600640 1 -- 172.21.15.28:0/4232180857 learned_addr learned my addr 172.21.15.28:0/4232180857 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:10.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.145+0000 7f0520c00640 1 --2- 172.21.15.28:0/4232180857 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f051c0c5d80 0x7f051c155480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:10.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.145+0000 7f0522000640 1 --2- 172.21.15.28:0/4232180857 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f051c150700 0x7f051c14dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:10.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.145+0000 7f0521600640 1 -- 172.21.15.28:0/4232180857 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f051c150700 msgr2=0x7f051c14dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:10.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.145+0000 7f0521600640 1 --2- 172.21.15.28:0/4232180857 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f051c150700 0x7f051c14dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:10.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.146+0000 7f0521600640 1 -- 172.21.15.28:0/4232180857 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f051c0c5d80 msgr2=0x7f051c155480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:10.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.146+0000 7f0521600640 1 --2- 172.21.15.28:0/4232180857 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f051c0c5d80 0x7f051c155480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:10.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.146+0000 7f0521600640 1 -- 172.21.15.28:0/4232180857 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f051c156160 con 0x7f051c06b0e0 2024-02-21T23:06:10.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.146+0000 7f0520c00640 1 --2- 172.21.15.28:0/4232180857 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f051c0c5d80 0x7f051c155480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:06:10.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.146+0000 7f0521600640 1 --2- 172.21.15.28:0/4232180857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f051c06b0e0 0x7f051c1eabf0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f0510000c00 tx=0x7f05100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:10.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.146+0000 7f0519600640 1 -- 172.21.15.28:0/4232180857 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0510040070 con 0x7f051c06b0e0 2024-02-21T23:06:10.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.146+0000 7f0519600640 1 -- 172.21.15.28:0/4232180857 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05100399f0 con 0x7f051c06b0e0 2024-02-21T23:06:10.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.146+0000 7f0519600640 1 -- 172.21.15.28:0/4232180857 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0510035040 con 0x7f051c06b0e0 2024-02-21T23:06:10.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.146+0000 7f0524000640 1 -- 172.21.15.28:0/4232180857 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f051c1e04c0 con 0x7f051c06b0e0 2024-02-21T23:06:10.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.146+0000 7f0524000640 1 -- 172.21.15.28:0/4232180857 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f051c1e0c50 con 0x7f051c06b0e0 2024-02-21T23:06:10.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.148+0000 7f0524000640 1 -- 172.21.15.28:0/4232180857 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f051c067f30 con 0x7f051c06b0e0 2024-02-21T23:06:10.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.150+0000 7f0519600640 1 -- 172.21.15.28:0/4232180857 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f051004b060 con 0x7f051c06b0e0 2024-02-21T23:06:10.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.150+0000 7f0519600640 1 --2- 172.21.15.28:0/4232180857 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f04fc06b500 0x7f04fc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:10.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.151+0000 7f0519600640 1 -- 172.21.15.28:0/4232180857 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6855+0+0 (secure 0 0 0) 0x7f05100ac380 con 0x7f051c06b0e0 2024-02-21T23:06:10.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.151+0000 7f0520c00640 1 --2- 172.21.15.28:0/4232180857 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f04fc06b500 0x7f04fc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:10.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.152+0000 7f0520c00640 1 --2- 172.21.15.28:0/4232180857 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f04fc06b500 0x7f04fc06d9c0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f051c14d9d0 tx=0x7f0508009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:10.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.154+0000 7f0519600640 1 -- 172.21.15.28:0/4232180857 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f051007ae10 con 0x7f051c06b0e0 2024-02-21T23:06:10.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.481+0000 7f0524000640 1 -- 172.21.15.28:0/4232180857 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs2", "metadata": "cephfs2_metadata", "data": "cephfs2_data"} v 0) v1 -- 0x7f051c0abda0 con 0x7f051c06b0e0 2024-02-21T23:06:10.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.935+0000 7f0519600640 1 -- 172.21.15.28:0/4232180857 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs2", "metadata": "cephfs2_metadata", "data": "cephfs2_data"}]=0 new fs with metadata pool 13 and data pool 14 v63) v1 ==== 177+0+0 (secure 0 0 0) 0x7f051007a710 con 0x7f051c06b0e0 2024-02-21T23:06:10.925 INFO:teuthology.orchestra.run.smithi028.stderr:new fs with metadata pool 13 and data pool 14 2024-02-21T23:06:10.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.939+0000 7f0524000640 1 -- 172.21.15.28:0/4232180857 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f04fc06b500 msgr2=0x7f04fc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:10.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.939+0000 7f0524000640 1 --2- 172.21.15.28:0/4232180857 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f04fc06b500 0x7f04fc06d9c0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f051c14d9d0 tx=0x7f0508009040 comp rx=0 tx=0).stop 2024-02-21T23:06:10.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.939+0000 7f0524000640 1 -- 172.21.15.28:0/4232180857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f051c06b0e0 msgr2=0x7f051c1eabf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:10.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.939+0000 7f0524000640 1 --2- 172.21.15.28:0/4232180857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f051c06b0e0 0x7f051c1eabf0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f0510000c00 tx=0x7f05100046c0 comp rx=0 tx=0).stop 2024-02-21T23:06:10.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.940+0000 7f0524000640 1 -- 172.21.15.28:0/4232180857 shutdown_connections 2024-02-21T23:06:10.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.940+0000 7f0524000640 1 --2- 172.21.15.28:0/4232180857 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f04fc06b500 0x7f04fc06d9c0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:10.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.940+0000 7f0524000640 1 --2- 172.21.15.28:0/4232180857 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f051c150700 0x7f051c14dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:10.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.940+0000 7f0524000640 1 --2- 172.21.15.28:0/4232180857 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f051c0c5d80 0x7f051c155480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:10.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.940+0000 7f0524000640 1 --2- 172.21.15.28:0/4232180857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f051c06b0e0 0x7f051c1eabf0 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:10.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.940+0000 7f0524000640 1 -- 172.21.15.28:0/4232180857 >> 172.21.15.28:0/4232180857 conn(0x7f051c0a85a0 msgr2=0x7f051c14aeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:10.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.941+0000 7f0524000640 1 -- 172.21.15.28:0/4232180857 shutdown_connections 2024-02-21T23:06:10.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:10.941+0000 7f0524000640 1 -- 172.21.15.28:0/4232180857 wait complete. 2024-02-21T23:06:10.982 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:06:11.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.170+0000 7fe05a000640 1 Processor -- start 2024-02-21T23:06:11.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.170+0000 7fe05a000640 1 -- start start 2024-02-21T23:06:11.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.172+0000 7fe05a000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe05406dbf0 0x7fe054158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:11.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.172+0000 7fe05a000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fe054155960 0x7fe054159000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:11.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.172+0000 7fe05a000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0540c74c0 con 0x7fe0540aff90 2024-02-21T23:06:11.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.172+0000 7fe05a000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0540c7270 con 0x7fe05406dbf0 2024-02-21T23:06:11.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.172+0000 7fe05a000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe054158630 con 0x7fe054155960 2024-02-21T23:06:11.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.172+0000 7fe058c00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe0540aff90 0x7fe0540c51e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59816/0 (socket says 172.21.15.28:59816) 2024-02-21T23:06:11.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.172+0000 7fe058c00640 1 -- 172.21.15.28:0/633179104 learned_addr learned my addr 172.21.15.28:0/633179104 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:06:11.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.172+0000 7fe04be00640 1 --2- 172.21.15.28:0/633179104 >> v2:172.21.15.28:3300/0 conn(0x7fe05406dbf0 0x7fe054158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:11.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.172+0000 7fe04be00640 1 -- 172.21.15.28:0/633179104 >> v2:172.21.15.81:3300/0 conn(0x7fe054155960 msgr2=0x7fe054159000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:11.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.172+0000 7fe04be00640 1 --2- 172.21.15.28:0/633179104 >> v2:172.21.15.81:3300/0 conn(0x7fe054155960 0x7fe054159000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:11.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.172+0000 7fe04be00640 1 -- 172.21.15.28:0/633179104 >> v1:172.21.15.28:6789/0 conn(0x7fe0540aff90 legacy=0x7fe0540c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:11.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.172+0000 7fe04be00640 1 -- 172.21.15.28:0/633179104 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0540c77a0 con 0x7fe05406dbf0 2024-02-21T23:06:11.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.174+0000 7fe04be00640 1 --2- 172.21.15.28:0/633179104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe05406dbf0 0x7fe054158ac0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fe0540c6d40 tx=0x7fe04002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6be4a3db302ff28 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:11.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.174+0000 7fe053e00640 1 -- 172.21.15.28:0/633179104 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe040035020 con 0x7fe05406dbf0 2024-02-21T23:06:11.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.174+0000 7fe053e00640 1 -- 172.21.15.28:0/633179104 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe040031830 con 0x7fe05406dbf0 2024-02-21T23:06:11.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.174+0000 7fe053e00640 1 -- 172.21.15.28:0/633179104 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe040031c00 con 0x7fe05406dbf0 2024-02-21T23:06:11.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.174+0000 7fe05a000640 1 -- 172.21.15.28:0/633179104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe05406dbf0 msgr2=0x7fe054158ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:11.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.174+0000 7fe05a000640 1 --2- 172.21.15.28:0/633179104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe05406dbf0 0x7fe054158ac0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fe0540c6d40 tx=0x7fe04002b8d0 comp rx=0 tx=0).stop 2024-02-21T23:06:11.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.175+0000 7fe05a000640 1 -- 172.21.15.28:0/633179104 shutdown_connections 2024-02-21T23:06:11.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.175+0000 7fe05a000640 1 --2- 172.21.15.28:0/633179104 >> v2:172.21.15.81:3300/0 conn(0x7fe054155960 0x7fe054159000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:11.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.175+0000 7fe05a000640 1 --2- 172.21.15.28:0/633179104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe05406dbf0 0x7fe054158ac0 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:11.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.175+0000 7fe05a000640 1 -- 172.21.15.28:0/633179104 >> 172.21.15.28:0/633179104 conn(0x7fe0540ac370 msgr2=0x7fe0540ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:11.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.175+0000 7fe05a000640 1 -- 172.21.15.28:0/633179104 shutdown_connections 2024-02-21T23:06:11.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.175+0000 7fe05a000640 1 -- 172.21.15.28:0/633179104 wait complete. 2024-02-21T23:06:11.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.176+0000 7fe05a000640 1 Processor -- start 2024-02-21T23:06:11.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.176+0000 7fe05a000640 1 -- start start 2024-02-21T23:06:11.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.177+0000 7fe05a000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe05406dbf0 0x7fe0540bfc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:11.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.177+0000 7fe05a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0540aff90 0x7fe0540bee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:11.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.177+0000 7fe05a000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe054155960 0x7fe0540bf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:11.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.177+0000 7fe05a000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0540c58f0 con 0x7fe0540aff90 2024-02-21T23:06:11.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.177+0000 7fe05a000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0540be0c0 con 0x7fe054155960 2024-02-21T23:06:11.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.177+0000 7fe05a000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0540c5a70 con 0x7fe05406dbf0 2024-02-21T23:06:11.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.177+0000 7fe04be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0540aff90 0x7fe0540bee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:11.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.177+0000 7fe04be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0540aff90 0x7fe0540bee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54642/0 (socket says 172.21.15.28:54642) 2024-02-21T23:06:11.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.178+0000 7fe04be00640 1 -- 172.21.15.28:0/2329234975 learned_addr learned my addr 172.21.15.28:0/2329234975 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:11.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.178+0000 7fe058c00640 1 --2- 172.21.15.28:0/2329234975 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe05406dbf0 0x7fe0540bfc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:11.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.178+0000 7fe059600640 1 --2- 172.21.15.28:0/2329234975 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe054155960 0x7fe0540bf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:11.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.178+0000 7fe04be00640 1 -- 172.21.15.28:0/2329234975 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe05406dbf0 msgr2=0x7fe0540bfc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:11.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.178+0000 7fe04be00640 1 --2- 172.21.15.28:0/2329234975 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe05406dbf0 0x7fe0540bfc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:11.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.178+0000 7fe04be00640 1 -- 172.21.15.28:0/2329234975 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe054155960 msgr2=0x7fe0540bf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:11.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.178+0000 7fe04be00640 1 --2- 172.21.15.28:0/2329234975 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe054155960 0x7fe0540bf3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:11.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.178+0000 7fe04be00640 1 -- 172.21.15.28:0/2329234975 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0540cb0b0 con 0x7fe0540aff90 2024-02-21T23:06:11.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.178+0000 7fe059600640 1 --2- 172.21.15.28:0/2329234975 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe054155960 0x7fe0540bf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:06:11.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.178+0000 7fe04be00640 1 --2- 172.21.15.28:0/2329234975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0540aff90 0x7fe0540bee80 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fe04002bee0 tx=0x7fe0400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:11.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.179+0000 7fe051600640 1 -- 172.21.15.28:0/2329234975 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe040041070 con 0x7fe0540aff90 2024-02-21T23:06:11.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.179+0000 7fe051600640 1 -- 172.21.15.28:0/2329234975 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe04003a950 con 0x7fe0540aff90 2024-02-21T23:06:11.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.179+0000 7fe05a000640 1 -- 172.21.15.28:0/2329234975 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0541eb560 con 0x7fe0540aff90 2024-02-21T23:06:11.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.179+0000 7fe051600640 1 -- 172.21.15.28:0/2329234975 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe040035040 con 0x7fe0540aff90 2024-02-21T23:06:11.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.179+0000 7fe05a000640 1 -- 172.21.15.28:0/2329234975 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0540be960 con 0x7fe0540aff90 2024-02-21T23:06:11.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.180+0000 7fe05a000640 1 -- 172.21.15.28:0/2329234975 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0540beb60 con 0x7fe0540aff90 2024-02-21T23:06:11.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.181+0000 7fe051600640 1 -- 172.21.15.28:0/2329234975 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe040038020 con 0x7fe0540aff90 2024-02-21T23:06:11.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.182+0000 7fe051600640 1 --2- 172.21.15.28:0/2329234975 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe02806b4d0 0x7fe02806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:11.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.182+0000 7fe051600640 1 -- 172.21.15.28:0/2329234975 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6957+0+0 (secure 0 0 0) 0x7fe0400ac1a0 con 0x7fe0540aff90 2024-02-21T23:06:11.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.182+0000 7fe058c00640 1 --2- 172.21.15.28:0/2329234975 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe02806b4d0 0x7fe02806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:11.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.184+0000 7fe058c00640 1 --2- 172.21.15.28:0/2329234975 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe02806b4d0 0x7fe02806d990 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fe038000f60 tx=0x7fe038009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:11.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.187+0000 7fe051600640 1 -- 172.21.15.28:0/2329234975 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe0540beb60 con 0x7fe0540aff90 2024-02-21T23:06:11.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.377+0000 7fe05a000640 1 -- 172.21.15.28:0/2329234975 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe05404f290 con 0x7fe0540aff90 2024-02-21T23:06:11.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.379+0000 7fe051600640 1 -- 172.21.15.28:0/2329234975 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) v1 ==== 74+0+20622 (secure 0 0 0) 0x7fe04007ab30 con 0x7fe0540aff90 2024-02-21T23:06:11.369 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:06:11.369 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":79,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:06:10.484048+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:05:48.772562+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-02-21T23:05:49.852659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs2_metadata","create_time":"2024-02-21T23:06:08.240443+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs2_data","create_time":"2024-02-21T23:06:09.466016+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:06:11.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.382+0000 7fe05a000640 1 -- 172.21.15.28:0/2329234975 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe02806b4d0 msgr2=0x7fe02806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:11.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.382+0000 7fe05a000640 1 --2- 172.21.15.28:0/2329234975 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe02806b4d0 0x7fe02806d990 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fe038000f60 tx=0x7fe038009040 comp rx=0 tx=0).stop 2024-02-21T23:06:11.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.382+0000 7fe05a000640 1 -- 172.21.15.28:0/2329234975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0540aff90 msgr2=0x7fe0540bee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:11.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.382+0000 7fe05a000640 1 --2- 172.21.15.28:0/2329234975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0540aff90 0x7fe0540bee80 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fe04002bee0 tx=0x7fe0400046c0 comp rx=0 tx=0).stop 2024-02-21T23:06:11.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.382+0000 7fe05a000640 1 -- 172.21.15.28:0/2329234975 shutdown_connections 2024-02-21T23:06:11.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.382+0000 7fe05a000640 1 --2- 172.21.15.28:0/2329234975 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe02806b4d0 0x7fe02806d990 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:11.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.382+0000 7fe05a000640 1 --2- 172.21.15.28:0/2329234975 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe054155960 0x7fe0540bf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:11.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.382+0000 7fe05a000640 1 --2- 172.21.15.28:0/2329234975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0540aff90 0x7fe0540bee80 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:11.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.382+0000 7fe05a000640 1 --2- 172.21.15.28:0/2329234975 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe05406dbf0 0x7fe0540bfc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:11.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.382+0000 7fe05a000640 1 -- 172.21.15.28:0/2329234975 >> 172.21.15.28:0/2329234975 conn(0x7fe0540ac370 msgr2=0x7fe0540c50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:11.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.382+0000 7fe05a000640 1 -- 172.21.15.28:0/2329234975 shutdown_connections 2024-02-21T23:06:11.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.382+0000 7fe05a000640 1 -- 172.21.15.28:0/2329234975 wait complete. 2024-02-21T23:06:11.396 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:06:11.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.614+0000 7fce0ca00640 1 Processor -- start 2024-02-21T23:06:11.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.614+0000 7fce0ca00640 1 -- start start 2024-02-21T23:06:11.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.615+0000 7fce0ca00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fce0806dbf0 0x7fce08158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:11.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.616+0000 7fce0ca00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fce08155960 0x7fce08159000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:11.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.616+0000 7fce0ca00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce080c74c0 con 0x7fce080aff90 2024-02-21T23:06:11.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.616+0000 7fce0ca00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce080c7270 con 0x7fce0806dbf0 2024-02-21T23:06:11.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.616+0000 7fce0ca00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce08158630 con 0x7fce08155960 2024-02-21T23:06:11.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.616+0000 7fce05e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fce0806dbf0 0x7fce08158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:11.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.616+0000 7fce05e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fce0806dbf0 0x7fce08158ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54652/0 (socket says 172.21.15.28:54652) 2024-02-21T23:06:11.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.616+0000 7fce05e00640 1 -- 172.21.15.28:0/335625456 learned_addr learned my addr 172.21.15.28:0/335625456 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:11.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.616+0000 7fce07200640 1 --2- 172.21.15.28:0/335625456 >> v2:172.21.15.81:3300/0 conn(0x7fce08155960 0x7fce08159000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:11.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.617+0000 7fce05e00640 1 -- 172.21.15.28:0/335625456 >> v2:172.21.15.81:3300/0 conn(0x7fce08155960 msgr2=0x7fce08159000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:11.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.617+0000 7fce05e00640 1 --2- 172.21.15.28:0/335625456 >> v2:172.21.15.81:3300/0 conn(0x7fce08155960 0x7fce08159000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:11.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.617+0000 7fce05e00640 1 -- 172.21.15.28:0/335625456 >> v1:172.21.15.81:6789/0 conn(0x7fce080aff90 legacy=0x7fce080c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:11.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.617+0000 7fce05e00640 1 -- 172.21.15.28:0/335625456 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce080c77a0 con 0x7fce0806dbf0 2024-02-21T23:06:11.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.619+0000 7fce05e00640 1 --2- 172.21.15.28:0/335625456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce0806dbf0 0x7fce08158ac0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fce080c6d40 tx=0x7fcdf402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1cdc89e9328d312 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:11.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.619+0000 7fce05400640 1 -- 172.21.15.28:0/335625456 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf4035020 con 0x7fce0806dbf0 2024-02-21T23:06:11.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.619+0000 7fce05400640 1 -- 172.21.15.28:0/335625456 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdf4031870 con 0x7fce0806dbf0 2024-02-21T23:06:11.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.619+0000 7fce05400640 1 -- 172.21.15.28:0/335625456 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf4031c40 con 0x7fce0806dbf0 2024-02-21T23:06:11.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.620+0000 7fce0ca00640 1 -- 172.21.15.28:0/335625456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce0806dbf0 msgr2=0x7fce08158ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:11.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.620+0000 7fce0ca00640 1 --2- 172.21.15.28:0/335625456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce0806dbf0 0x7fce08158ac0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fce080c6d40 tx=0x7fcdf402b910 comp rx=0 tx=0).stop 2024-02-21T23:06:11.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.620+0000 7fce0ca00640 1 -- 172.21.15.28:0/335625456 shutdown_connections 2024-02-21T23:06:11.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.620+0000 7fce0ca00640 1 --2- 172.21.15.28:0/335625456 >> v2:172.21.15.81:3300/0 conn(0x7fce08155960 0x7fce08159000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:11.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.620+0000 7fce0ca00640 1 --2- 172.21.15.28:0/335625456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce0806dbf0 0x7fce08158ac0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:11.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.620+0000 7fce0ca00640 1 -- 172.21.15.28:0/335625456 >> 172.21.15.28:0/335625456 conn(0x7fce080ac370 msgr2=0x7fce080ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:11.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.620+0000 7fce0ca00640 1 -- 172.21.15.28:0/335625456 shutdown_connections 2024-02-21T23:06:11.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.620+0000 7fce0ca00640 1 -- 172.21.15.28:0/335625456 wait complete. 2024-02-21T23:06:11.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.621+0000 7fce0ca00640 1 Processor -- start 2024-02-21T23:06:11.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.622+0000 7fce0ca00640 1 -- start start 2024-02-21T23:06:11.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.622+0000 7fce0ca00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fce0806dbf0 0x7fce08156b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:11.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.622+0000 7fce0ca00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fce080aff90 0x7fce080c3630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:11.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.622+0000 7fce0ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce08155960 0x7fce080c1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:11.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.622+0000 7fce0ca00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce080c58f0 con 0x7fce08155960 2024-02-21T23:06:11.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.622+0000 7fce0ca00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce080be0c0 con 0x7fce0806dbf0 2024-02-21T23:06:11.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.622+0000 7fce0ca00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce080c5a70 con 0x7fce080aff90 2024-02-21T23:06:11.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.623+0000 7fce07200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce08155960 0x7fce080c1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:11.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.623+0000 7fce07200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce08155960 0x7fce080c1930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54666/0 (socket says 172.21.15.28:54666) 2024-02-21T23:06:11.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.623+0000 7fce07200640 1 -- 172.21.15.28:0/2805739578 learned_addr learned my addr 172.21.15.28:0/2805739578 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:11.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.623+0000 7fce05e00640 1 --2- 172.21.15.28:0/2805739578 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fce080aff90 0x7fce080c3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:11.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.623+0000 7fce06800640 1 --2- 172.21.15.28:0/2805739578 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fce0806dbf0 0x7fce08156b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:11.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.623+0000 7fce07200640 1 -- 172.21.15.28:0/2805739578 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fce080aff90 msgr2=0x7fce080c3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:11.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.623+0000 7fce07200640 1 --2- 172.21.15.28:0/2805739578 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fce080aff90 0x7fce080c3630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:11.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.623+0000 7fce07200640 1 -- 172.21.15.28:0/2805739578 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fce0806dbf0 msgr2=0x7fce08156b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:11.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.623+0000 7fce07200640 1 --2- 172.21.15.28:0/2805739578 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fce0806dbf0 0x7fce08156b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:11.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.623+0000 7fce07200640 1 -- 172.21.15.28:0/2805739578 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce081ee4a0 con 0x7fce08155960 2024-02-21T23:06:11.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.623+0000 7fce05e00640 1 --2- 172.21.15.28:0/2805739578 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fce080aff90 0x7fce080c3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:06:11.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.624+0000 7fce07200640 1 --2- 172.21.15.28:0/2805739578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce08155960 0x7fce080c1930 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fce00007e60 tx=0x7fce0000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:11.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.624+0000 7fcdfea00640 1 -- 172.21.15.28:0/2805739578 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce00014420 con 0x7fce08155960 2024-02-21T23:06:11.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.624+0000 7fcdfea00640 1 -- 172.21.15.28:0/2805739578 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce00019070 con 0x7fce08155960 2024-02-21T23:06:11.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.624+0000 7fce0ca00640 1 -- 172.21.15.28:0/2805739578 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce080be240 con 0x7fce08155960 2024-02-21T23:06:11.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.624+0000 7fcdfea00640 1 -- 172.21.15.28:0/2805739578 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce00014820 con 0x7fce08155960 2024-02-21T23:06:11.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.624+0000 7fce0ca00640 1 -- 172.21.15.28:0/2805739578 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce080c1560 con 0x7fce08155960 2024-02-21T23:06:11.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.625+0000 7fce0ca00640 1 -- 172.21.15.28:0/2805739578 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce0804f290 con 0x7fce08155960 2024-02-21T23:06:11.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.626+0000 7fcdfea00640 1 -- 172.21.15.28:0/2805739578 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fce000149c0 con 0x7fce08155960 2024-02-21T23:06:11.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.627+0000 7fcdfea00640 1 --2- 172.21.15.28:0/2805739578 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcde806b4d0 0x7fcde806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:11.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.627+0000 7fcdfea00640 1 -- 172.21.15.28:0/2805739578 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6957+0+0 (secure 0 0 0) 0x7fce0008af50 con 0x7fce08155960 2024-02-21T23:06:11.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.628+0000 7fce06800640 1 --2- 172.21.15.28:0/2805739578 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcde806b4d0 0x7fcde806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:11.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.629+0000 7fce06800640 1 --2- 172.21.15.28:0/2805739578 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcde806b4d0 0x7fcde806d990 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fce08152050 tx=0x7fcdf8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:11.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.632+0000 7fcdfea00640 1 -- 172.21.15.28:0/2805739578 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fce081ee4a0 con 0x7fce08155960 2024-02-21T23:06:11.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.832+0000 7fce0ca00640 1 -- 172.21.15.28:0/2805739578 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fce0806d9d0 con 0x7fce08155960 2024-02-21T23:06:11.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.833+0000 7fcdfea00640 1 -- 172.21.15.28:0/2805739578 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) v1 ==== 74+0+20622 (secure 0 0 0) 0x7fce000598e0 con 0x7fce08155960 2024-02-21T23:06:11.823 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:06:11.823 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":79,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:06:10.484048+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:05:48.772562+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-02-21T23:05:49.852659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs2_metadata","create_time":"2024-02-21T23:06:08.240443+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs2_data","create_time":"2024-02-21T23:06:09.466016+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:06:11.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.836+0000 7fce0ca00640 1 -- 172.21.15.28:0/2805739578 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcde806b4d0 msgr2=0x7fcde806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:11.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.836+0000 7fce0ca00640 1 --2- 172.21.15.28:0/2805739578 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcde806b4d0 0x7fcde806d990 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fce08152050 tx=0x7fcdf8009040 comp rx=0 tx=0).stop 2024-02-21T23:06:11.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.836+0000 7fce0ca00640 1 -- 172.21.15.28:0/2805739578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce08155960 msgr2=0x7fce080c1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:11.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.836+0000 7fce0ca00640 1 --2- 172.21.15.28:0/2805739578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce08155960 0x7fce080c1930 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fce00007e60 tx=0x7fce0000e4b0 comp rx=0 tx=0).stop 2024-02-21T23:06:11.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.837+0000 7fce0ca00640 1 -- 172.21.15.28:0/2805739578 shutdown_connections 2024-02-21T23:06:11.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.837+0000 7fce0ca00640 1 --2- 172.21.15.28:0/2805739578 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcde806b4d0 0x7fcde806d990 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:11.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.837+0000 7fce0ca00640 1 --2- 172.21.15.28:0/2805739578 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fce080aff90 0x7fce080c3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:11.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.837+0000 7fce0ca00640 1 --2- 172.21.15.28:0/2805739578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce08155960 0x7fce080c1930 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:11.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.837+0000 7fce0ca00640 1 --2- 172.21.15.28:0/2805739578 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fce0806dbf0 0x7fce08156b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:11.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.837+0000 7fce0ca00640 1 -- 172.21.15.28:0/2805739578 >> 172.21.15.28:0/2805739578 conn(0x7fce080ac370 msgr2=0x7fce080ada60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:11.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.837+0000 7fce0ca00640 1 -- 172.21.15.28:0/2805739578 shutdown_connections 2024-02-21T23:06:11.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:11.837+0000 7fce0ca00640 1 -- 172.21.15.28:0/2805739578 wait complete. 2024-02-21T23:06:11.852 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs2 standby_count_wanted 0 2024-02-21T23:06:12.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.141+0000 7f3cb7e00640 1 Processor -- start 2024-02-21T23:06:12.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.141+0000 7f3cb7e00640 1 -- start start 2024-02-21T23:06:12.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.143+0000 7f3cb7e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3cb8152650 0x7f3cb80c8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:12.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.143+0000 7f3cb7e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cb8158920 con 0x7f3cb806daa0 2024-02-21T23:06:12.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.143+0000 7f3cb7e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cb8158c00 con 0x7f3cb81550e0 2024-02-21T23:06:12.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.143+0000 7f3cb7e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb81586d0 con 0x7f3cb8152650 2024-02-21T23:06:12.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.143+0000 7f3cb6000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f3cb806daa0 0x7f3cb80c7160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59820/0 (socket says 172.21.15.28:59820) 2024-02-21T23:06:12.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.143+0000 7f3cb6000640 1 -- 172.21.15.28:0/3459305820 learned_addr learned my addr 172.21.15.28:0/3459305820 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:06:12.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.144+0000 7f3cb6a00640 1 --2- 172.21.15.28:0/3459305820 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3cb8152650 0x7f3cb80c8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:12.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.144+0000 7f3cb5600640 1 -- 172.21.15.28:0/3459305820 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4170552560 0 0) 0x7f3cb8158920 con 0x7f3cb806daa0 2024-02-21T23:06:12.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.144+0000 7f3cb5600640 1 -- 172.21.15.28:0/3459305820 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c9c003520 con 0x7f3cb806daa0 2024-02-21T23:06:12.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.144+0000 7f3cb5600640 1 -- 172.21.15.28:0/3459305820 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 704967175 0 0) 0x7f3cb8158c00 con 0x7f3cb81550e0 2024-02-21T23:06:12.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.144+0000 7f3cb5600640 1 -- 172.21.15.28:0/3459305820 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cb8158920 con 0x7f3cb81550e0 2024-02-21T23:06:12.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.144+0000 7f3cb5600640 1 -- 172.21.15.28:0/3459305820 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2409584228 0 0) 0x7f3c9c003520 con 0x7f3cb806daa0 2024-02-21T23:06:12.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.144+0000 7f3cb5600640 1 -- 172.21.15.28:0/3459305820 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3cb8152650 msgr2=0x7f3cb80c8ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:12.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.144+0000 7f3cb5600640 1 --2- 172.21.15.28:0/3459305820 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3cb8152650 0x7f3cb80c8ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:12.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.144+0000 7f3cb5600640 1 -- 172.21.15.28:0/3459305820 >> v1:172.21.15.81:6789/0 conn(0x7f3cb81550e0 legacy=0x7f3cb8155550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:12.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.144+0000 7f3cb5600640 1 -- 172.21.15.28:0/3459305820 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cb8158ee0 con 0x7f3cb806daa0 2024-02-21T23:06:12.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.144+0000 7f3cb5600640 1 -- 172.21.15.28:0/3459305820 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f3ca4001670 con 0x7f3cb806daa0 2024-02-21T23:06:12.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.145+0000 7f3cb5600640 1 -- 172.21.15.28:0/3459305820 >> v1:172.21.15.28:6789/0 conn(0x7f3cb806daa0 legacy=0x7f3cb80c7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:12.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.145+0000 7f3cb5600640 1 --2- 172.21.15.28:0/3459305820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c9c003b70 0x7f3c9c003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:12.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.145+0000 7f3cb5600640 1 --2- 172.21.15.28:0/3459305820 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c9c004800 0x7f3c9c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:12.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.145+0000 7f3cb5600640 1 --2- 172.21.15.28:0/3459305820 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3cb8152650 0x7f3c9c0051e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:12.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.145+0000 7f3cb5600640 1 -- 172.21.15.28:0/3459305820 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb81586d0 con 0x7f3c9c003b70 2024-02-21T23:06:12.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.145+0000 7f3cb5600640 1 -- 172.21.15.28:0/3459305820 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c9c00bad0 con 0x7f3c9c004800 2024-02-21T23:06:12.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.145+0000 7f3cb5600640 1 -- 172.21.15.28:0/3459305820 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c9c00bcd0 con 0x7f3cb8152650 2024-02-21T23:06:12.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.145+0000 7f3cb6a00640 1 --2- 172.21.15.28:0/3459305820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c9c003b70 0x7f3c9c003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:12.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.145+0000 7f3cb6000640 1 --2- 172.21.15.28:0/3459305820 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c9c004800 0x7f3c9c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:12.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.146+0000 7f3cb6a00640 1 -- 172.21.15.28:0/3459305820 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3cb8152650 msgr2=0x7f3c9c0051e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:12.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.146+0000 7f3cb6a00640 1 --2- 172.21.15.28:0/3459305820 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3cb8152650 0x7f3c9c0051e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:12.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.146+0000 7f3cb6a00640 1 -- 172.21.15.28:0/3459305820 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c9c004800 msgr2=0x7f3c9c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:12.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.146+0000 7f3cb6a00640 1 --2- 172.21.15.28:0/3459305820 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c9c004800 0x7f3c9c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:12.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.146+0000 7f3cb6a00640 1 -- 172.21.15.28:0/3459305820 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c9c003520 con 0x7f3c9c003b70 2024-02-21T23:06:12.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.147+0000 7f3cb6a00640 1 --2- 172.21.15.28:0/3459305820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c9c003b70 0x7f3c9c003f70 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f3ca8009ac0 tx=0x7f3c9c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9e2437a63f1ec5c server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:12.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.148+0000 7f3cb5600640 1 -- 172.21.15.28:0/3459305820 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ca8039070 con 0x7f3c9c003b70 2024-02-21T23:06:12.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.148+0000 7f3cb5600640 1 -- 172.21.15.28:0/3459305820 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ca802c710 con 0x7f3c9c003b70 2024-02-21T23:06:12.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.148+0000 7f3cb5600640 1 -- 172.21.15.28:0/3459305820 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ca8009690 con 0x7f3c9c003b70 2024-02-21T23:06:12.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.148+0000 7f3cb7e00640 1 -- 172.21.15.28:0/3459305820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c9c003b70 msgr2=0x7f3c9c003f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:12.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.148+0000 7f3cb7e00640 1 --2- 172.21.15.28:0/3459305820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c9c003b70 0x7f3c9c003f70 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f3ca8009ac0 tx=0x7f3c9c005830 comp rx=0 tx=0).stop 2024-02-21T23:06:12.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.148+0000 7f3cb7400640 1 -- 172.21.15.28:0/3459305820 reap_dead start 2024-02-21T23:06:12.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.148+0000 7f3cb7e00640 1 -- 172.21.15.28:0/3459305820 shutdown_connections 2024-02-21T23:06:12.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.148+0000 7f3cb7e00640 1 -- 172.21.15.28:0/3459305820 >> 172.21.15.28:0/3459305820 conn(0x7f3cb8072ce0 msgr2=0x7f3cb80ac340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:12.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.149+0000 7f3cb7e00640 1 -- 172.21.15.28:0/3459305820 shutdown_connections 2024-02-21T23:06:12.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.149+0000 7f3cb7e00640 1 -- 172.21.15.28:0/3459305820 wait complete. 2024-02-21T23:06:12.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.150+0000 7f3cb7e00640 1 Processor -- start 2024-02-21T23:06:12.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.150+0000 7f3cb7e00640 1 -- start start 2024-02-21T23:06:12.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.151+0000 7f3cb7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c9c003b70 0x7f3cb80c40c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:12.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.151+0000 7f3cb7e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3cb81debc0 0x7f3cb81df040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:12.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.151+0000 7f3cb7e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3cb814ba70 0x7f3cb814bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:12.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.151+0000 7f3cb7e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb8155c60 con 0x7f3c9c003b70 2024-02-21T23:06:12.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.151+0000 7f3cb7e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb80bdfb0 con 0x7f3cb81debc0 2024-02-21T23:06:12.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.151+0000 7f3cb7e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb8152a30 con 0x7f3cb814ba70 2024-02-21T23:06:12.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.151+0000 7f3cb6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c9c003b70 0x7f3cb80c40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:12.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.151+0000 7f3cb6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c9c003b70 0x7f3cb80c40c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54688/0 (socket says 172.21.15.28:54688) 2024-02-21T23:06:12.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.151+0000 7f3cb6a00640 1 -- 172.21.15.28:0/244338774 learned_addr learned my addr 172.21.15.28:0/244338774 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:12.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.152+0000 7f3cb6000640 1 --2- 172.21.15.28:0/244338774 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3cb81debc0 0x7f3cb81df040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:12.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.152+0000 7f3cb7400640 1 --2- 172.21.15.28:0/244338774 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3cb814ba70 0x7f3cb814bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:12.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.152+0000 7f3cb6a00640 1 -- 172.21.15.28:0/244338774 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3cb814ba70 msgr2=0x7f3cb814bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:12.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.152+0000 7f3cb6a00640 1 --2- 172.21.15.28:0/244338774 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3cb814ba70 0x7f3cb814bef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:12.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.152+0000 7f3cb6a00640 1 -- 172.21.15.28:0/244338774 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3cb81debc0 msgr2=0x7f3cb81df040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:12.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.152+0000 7f3cb6a00640 1 --2- 172.21.15.28:0/244338774 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3cb81debc0 0x7f3cb81df040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:12.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.152+0000 7f3cb6a00640 1 -- 172.21.15.28:0/244338774 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cb80c3e20 con 0x7f3c9c003b70 2024-02-21T23:06:12.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.152+0000 7f3cb6a00640 1 --2- 172.21.15.28:0/244338774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c9c003b70 0x7f3cb80c40c0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f3ca8037040 tx=0x7f3ca8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:12.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.153+0000 7f3caea00640 1 -- 172.21.15.28:0/244338774 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ca8039050 con 0x7f3c9c003b70 2024-02-21T23:06:12.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.153+0000 7f3caea00640 1 -- 172.21.15.28:0/244338774 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ca8045070 con 0x7f3c9c003b70 2024-02-21T23:06:12.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.153+0000 7f3caea00640 1 -- 172.21.15.28:0/244338774 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ca8003a70 con 0x7f3c9c003b70 2024-02-21T23:06:12.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.153+0000 7f3cb7e00640 1 -- 172.21.15.28:0/244338774 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cb80ac2b0 con 0x7f3c9c003b70 2024-02-21T23:06:12.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.153+0000 7f3cb7e00640 1 -- 172.21.15.28:0/244338774 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cb81e1440 con 0x7f3c9c003b70 2024-02-21T23:06:12.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.154+0000 7f3cb7e00640 1 -- 172.21.15.28:0/244338774 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cb81e1610 con 0x7f3c9c003b70 2024-02-21T23:06:12.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.155+0000 7f3caea00640 1 -- 172.21.15.28:0/244338774 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3ca8003c10 con 0x7f3c9c003b70 2024-02-21T23:06:12.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.156+0000 7f3caea00640 1 --2- 172.21.15.28:0/244338774 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3c9806b500 0x7f3c9806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:12.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.156+0000 7f3caea00640 1 -- 172.21.15.28:0/244338774 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f3ca80ac1a0 con 0x7f3c9c003b70 2024-02-21T23:06:12.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.157+0000 7f3cb6000640 1 --2- 172.21.15.28:0/244338774 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3c9806b500 0x7f3c9806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:12.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.158+0000 7f3cb6000640 1 --2- 172.21.15.28:0/244338774 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3c9806b500 0x7f3c9806d9c0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f3ca4001200 tx=0x7f3ca4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:12.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.165+0000 7f3caea00640 1 -- 172.21.15.28:0/244338774 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3ca804e020 con 0x7f3c9c003b70 2024-02-21T23:06:12.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.398+0000 7f3cb7e00640 1 -- 172.21.15.28:0/244338774 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs2", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f3cb804f290 con 0x7f3c9c003b70 2024-02-21T23:06:12.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.977+0000 7f3caea00640 1 -- 172.21.15.28:0/244338774 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs2", "var": "standby_count_wanted", "val": "0"}]=0 v66) v1 ==== 119+0+0 (secure 0 0 0) 0x7f3ca807abd0 con 0x7f3c9c003b70 2024-02-21T23:06:12.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.979+0000 7f3cb7e00640 1 -- 172.21.15.28:0/244338774 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3c9806b500 msgr2=0x7f3c9806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:12.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.979+0000 7f3cb7e00640 1 --2- 172.21.15.28:0/244338774 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3c9806b500 0x7f3c9806d9c0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f3ca4001200 tx=0x7f3ca4009040 comp rx=0 tx=0).stop 2024-02-21T23:06:12.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.979+0000 7f3cb7e00640 1 -- 172.21.15.28:0/244338774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c9c003b70 msgr2=0x7f3cb80c40c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:12.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.979+0000 7f3cb7e00640 1 --2- 172.21.15.28:0/244338774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c9c003b70 0x7f3cb80c40c0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f3ca8037040 tx=0x7f3ca8004730 comp rx=0 tx=0).stop 2024-02-21T23:06:12.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.980+0000 7f3cb7e00640 1 -- 172.21.15.28:0/244338774 shutdown_connections 2024-02-21T23:06:12.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.980+0000 7f3cb7e00640 1 --2- 172.21.15.28:0/244338774 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3c9806b500 0x7f3c9806d9c0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:12.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.980+0000 7f3cb7e00640 1 --2- 172.21.15.28:0/244338774 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3cb814ba70 0x7f3cb814bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:12.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.980+0000 7f3cb7e00640 1 --2- 172.21.15.28:0/244338774 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3cb81debc0 0x7f3cb81df040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:12.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.980+0000 7f3cb7e00640 1 --2- 172.21.15.28:0/244338774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c9c003b70 0x7f3cb80c40c0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:12.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.980+0000 7f3cb7e00640 1 -- 172.21.15.28:0/244338774 >> 172.21.15.28:0/244338774 conn(0x7f3cb8072ce0 msgr2=0x7f3cb806d0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:12.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.980+0000 7f3cb7e00640 1 -- 172.21.15.28:0/244338774 shutdown_connections 2024-02-21T23:06:12.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:12.980+0000 7f3cb7e00640 1 -- 172.21.15.28:0/244338774 wait complete. 2024-02-21T23:06:12.994 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-02-21T23:06:12.994 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs2 allow_standby_replay false 2024-02-21T23:06:13.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.199+0000 7f46a8600640 1 Processor -- start 2024-02-21T23:06:13.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.200+0000 7f46a8600640 1 -- start start 2024-02-21T23:06:13.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.202+0000 7f46a8600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f46a00c4d60 0x7f46a006cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:13.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.202+0000 7f46a8600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46a0155230 con 0x7f46a00c75d0 2024-02-21T23:06:13.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.202+0000 7f46a8600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46a00c6ca0 con 0x7f46a00bfbf0 2024-02-21T23:06:13.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.202+0000 7f46a8600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46a00c6960 con 0x7f46a00c4d60 2024-02-21T23:06:13.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.202+0000 7f46a5c00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f46a00c75d0 0x7f46a00af680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59830/0 (socket says 172.21.15.28:59830) 2024-02-21T23:06:13.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.202+0000 7f46a5c00640 1 -- 172.21.15.28:0/2198822427 learned_addr learned my addr 172.21.15.28:0/2198822427 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:06:13.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.203+0000 7f469fe00640 1 -- 172.21.15.28:0/2198822427 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3372589674 0 0) 0x7f4694001110 con 0x7f46a00c75d0 2024-02-21T23:06:13.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.203+0000 7f46a5200640 1 --2- 172.21.15.28:0/2198822427 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f46a00c4d60 0x7f46a006cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:13.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.203+0000 7f469fe00640 1 -- 172.21.15.28:0/2198822427 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4688003520 con 0x7f46a00c75d0 2024-02-21T23:06:13.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.203+0000 7f469fe00640 1 -- 172.21.15.28:0/2198822427 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1267252406 0 0) 0x7f4688003520 con 0x7f46a00c75d0 2024-02-21T23:06:13.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.203+0000 7f469fe00640 1 -- 172.21.15.28:0/2198822427 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f46a00c4d60 msgr2=0x7f46a006cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:13.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.203+0000 7f469fe00640 1 --2- 172.21.15.28:0/2198822427 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f46a00c4d60 0x7f46a006cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:13.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.203+0000 7f469fe00640 1 -- 172.21.15.28:0/2198822427 >> v1:172.21.15.81:6789/0 conn(0x7f46a00bfbf0 legacy=0x7f46a015fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:13.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.203+0000 7f469fe00640 1 -- 172.21.15.28:0/2198822427 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46a00c6f80 con 0x7f46a00c75d0 2024-02-21T23:06:13.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.203+0000 7f469fe00640 1 -- 172.21.15.28:0/2198822427 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f4694001840 con 0x7f46a00c75d0 2024-02-21T23:06:13.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.203+0000 7f46a5200640 1 --2- 172.21.15.28:0/2198822427 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f46a00c4d60 0x7f46a006cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:06:13.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.203+0000 7f469fe00640 1 -- 172.21.15.28:0/2198822427 >> v1:172.21.15.28:6789/0 conn(0x7f46a00c75d0 legacy=0x7f46a00af680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:13.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.203+0000 7f469fe00640 1 --2- 172.21.15.28:0/2198822427 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f46a00c4d60 0x7f46880039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:13.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.203+0000 7f469fe00640 1 --2- 172.21.15.28:0/2198822427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4688004020 0x7f4688004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:13.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.203+0000 7f469fe00640 1 --2- 172.21.15.28:0/2198822427 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4688004d40 0x7f46880051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:13.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.203+0000 7f469fe00640 1 -- 172.21.15.28:0/2198822427 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46a00c6960 con 0x7f4688004020 2024-02-21T23:06:13.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.203+0000 7f469fe00640 1 -- 172.21.15.28:0/2198822427 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f468800bc00 con 0x7f4688004d40 2024-02-21T23:06:13.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.203+0000 7f469fe00640 1 -- 172.21.15.28:0/2198822427 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f468800be00 con 0x7f46a00c4d60 2024-02-21T23:06:13.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.204+0000 7f46a5200640 1 --2- 172.21.15.28:0/2198822427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4688004020 0x7f4688004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:13.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.204+0000 7f46a5c00640 1 --2- 172.21.15.28:0/2198822427 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f46a00c4d60 0x7f46880039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:13.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.204+0000 7f46a6600640 1 --2- 172.21.15.28:0/2198822427 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4688004d40 0x7f46880051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:13.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.204+0000 7f46a5200640 1 -- 172.21.15.28:0/2198822427 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f46a00c4d60 msgr2=0x7f46880039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:13.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.204+0000 7f46a5200640 1 --2- 172.21.15.28:0/2198822427 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f46a00c4d60 0x7f46880039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:13.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.204+0000 7f46a5200640 1 -- 172.21.15.28:0/2198822427 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4688004d40 msgr2=0x7f46880051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:13.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.204+0000 7f46a5200640 1 --2- 172.21.15.28:0/2198822427 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4688004d40 0x7f46880051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:13.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.204+0000 7f46a5200640 1 -- 172.21.15.28:0/2198822427 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4688003520 con 0x7f4688004020 2024-02-21T23:06:13.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.204+0000 7f46a6600640 1 --2- 172.21.15.28:0/2198822427 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4688004d40 0x7f46880051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:06:13.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.205+0000 7f46a5200640 1 --2- 172.21.15.28:0/2198822427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4688004020 0x7f4688004800 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f46a00c51a0 tx=0x7f4688005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb8fba0223be8e57 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:13.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.206+0000 7f469fe00640 1 -- 172.21.15.28:0/2198822427 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f468c00f040 con 0x7f4688004020 2024-02-21T23:06:13.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.206+0000 7f469fe00640 1 -- 172.21.15.28:0/2198822427 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f468c004030 con 0x7f4688004020 2024-02-21T23:06:13.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.206+0000 7f469fe00640 1 -- 172.21.15.28:0/2198822427 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f468c004440 con 0x7f4688004020 2024-02-21T23:06:13.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.206+0000 7f46a8600640 1 -- 172.21.15.28:0/2198822427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4688004020 msgr2=0x7f4688004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:13.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.206+0000 7f46a8600640 1 --2- 172.21.15.28:0/2198822427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4688004020 0x7f4688004800 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f46a00c51a0 tx=0x7f4688005830 comp rx=0 tx=0).stop 2024-02-21T23:06:13.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.206+0000 7f46a6600640 1 -- 172.21.15.28:0/2198822427 reap_dead start 2024-02-21T23:06:13.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.206+0000 7f46a8600640 1 -- 172.21.15.28:0/2198822427 shutdown_connections 2024-02-21T23:06:13.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.206+0000 7f46a8600640 1 -- 172.21.15.28:0/2198822427 >> 172.21.15.28:0/2198822427 conn(0x7f46a00725a0 msgr2=0x7f46a00729b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:13.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.206+0000 7f46a8600640 1 -- 172.21.15.28:0/2198822427 shutdown_connections 2024-02-21T23:06:13.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.206+0000 7f46a8600640 1 -- 172.21.15.28:0/2198822427 wait complete. 2024-02-21T23:06:13.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.208+0000 7f46a8600640 1 Processor -- start 2024-02-21T23:06:13.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.208+0000 7f46a8600640 1 -- start start 2024-02-21T23:06:13.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.209+0000 7f46a8600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4688004020 0x7f46a0150540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:13.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.210+0000 7f46a8600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46a00cb360 0x7f46a0145100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:13.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.210+0000 7f46a8600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f46a00c8dc0 0x7f46a0145640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:13.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.210+0000 7f46a8600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46a0155a70 con 0x7f4688004020 2024-02-21T23:06:13.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.210+0000 7f46a8600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46a0155790 con 0x7f46a00cb360 2024-02-21T23:06:13.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.210+0000 7f46a8600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46a00afd90 con 0x7f46a00c8dc0 2024-02-21T23:06:13.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.210+0000 7f46a5c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4688004020 0x7f46a0150540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:13.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.210+0000 7f46a5c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4688004020 0x7f46a0150540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54706/0 (socket says 172.21.15.28:54706) 2024-02-21T23:06:13.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.210+0000 7f46a5c00640 1 -- 172.21.15.28:0/2439936489 learned_addr learned my addr 172.21.15.28:0/2439936489 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:13.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.210+0000 7f46a5200640 1 --2- 172.21.15.28:0/2439936489 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46a00cb360 0x7f46a0145100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:13.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.210+0000 7f46a6600640 1 --2- 172.21.15.28:0/2439936489 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f46a00c8dc0 0x7f46a0145640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:13.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.210+0000 7f46a5c00640 1 -- 172.21.15.28:0/2439936489 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f46a00c8dc0 msgr2=0x7f46a0145640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:13.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.210+0000 7f46a5c00640 1 --2- 172.21.15.28:0/2439936489 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f46a00c8dc0 0x7f46a0145640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:13.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.210+0000 7f46a5c00640 1 -- 172.21.15.28:0/2439936489 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46a00cb360 msgr2=0x7f46a0145100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:13.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.210+0000 7f46a5c00640 1 --2- 172.21.15.28:0/2439936489 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46a00cb360 0x7f46a0145100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:13.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.210+0000 7f46a5c00640 1 -- 172.21.15.28:0/2439936489 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46a00c8750 con 0x7f4688004020 2024-02-21T23:06:13.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.211+0000 7f46a5c00640 1 --2- 172.21.15.28:0/2439936489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4688004020 0x7f46a0150540 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f469400e7a0 tx=0x7f469400ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:13.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.211+0000 7f469d600640 1 -- 172.21.15.28:0/2439936489 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f469400c870 con 0x7f4688004020 2024-02-21T23:06:13.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.211+0000 7f469d600640 1 -- 172.21.15.28:0/2439936489 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4694018070 con 0x7f4688004020 2024-02-21T23:06:13.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.211+0000 7f469d600640 1 -- 172.21.15.28:0/2439936489 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f469400cc30 con 0x7f4688004020 2024-02-21T23:06:13.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.211+0000 7f46a8600640 1 -- 172.21.15.28:0/2439936489 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46a00c89f0 con 0x7f4688004020 2024-02-21T23:06:13.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.211+0000 7f46a8600640 1 -- 172.21.15.28:0/2439936489 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46a0159630 con 0x7f4688004020 2024-02-21T23:06:13.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.213+0000 7f46a8600640 1 -- 172.21.15.28:0/2439936489 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46a00c8bc0 con 0x7f4688004020 2024-02-21T23:06:13.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.213+0000 7f469d600640 1 -- 172.21.15.28:0/2439936489 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4694013420 con 0x7f4688004020 2024-02-21T23:06:13.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.215+0000 7f469d600640 1 --2- 172.21.15.28:0/2439936489 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f468006b500 0x7f468006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:13.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.215+0000 7f469d600640 1 -- 172.21.15.28:0/2439936489 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f469408a270 con 0x7f4688004020 2024-02-21T23:06:13.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.216+0000 7f46a5200640 1 --2- 172.21.15.28:0/2439936489 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f468006b500 0x7f468006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:13.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.217+0000 7f46a5200640 1 --2- 172.21.15.28:0/2439936489 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f468006b500 0x7f468006d9c0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f468c035000 tx=0x7f468c03a040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:13.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.222+0000 7f469d600640 1 -- 172.21.15.28:0/2439936489 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f46a00c8bc0 con 0x7f4688004020 2024-02-21T23:06:13.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.490+0000 7f46a8600640 1 -- 172.21.15.28:0/2439936489 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs2", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f46a01eaee0 con 0x7f4688004020 2024-02-21T23:06:13.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.979+0000 7f469d600640 1 -- 172.21.15.28:0/2439936489 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs2", "var": "allow_standby_replay", "val": "false"}]=0 v67) v1 ==== 123+0+0 (secure 0 0 0) 0x7f46a0155230 con 0x7f4688004020 2024-02-21T23:06:13.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.983+0000 7f46a8600640 1 -- 172.21.15.28:0/2439936489 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f468006b500 msgr2=0x7f468006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:13.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.983+0000 7f46a8600640 1 --2- 172.21.15.28:0/2439936489 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f468006b500 0x7f468006d9c0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f468c035000 tx=0x7f468c03a040 comp rx=0 tx=0).stop 2024-02-21T23:06:13.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.983+0000 7f46a8600640 1 -- 172.21.15.28:0/2439936489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4688004020 msgr2=0x7f46a0150540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:13.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.983+0000 7f46a8600640 1 --2- 172.21.15.28:0/2439936489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4688004020 0x7f46a0150540 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f469400e7a0 tx=0x7f469400ec60 comp rx=0 tx=0).stop 2024-02-21T23:06:13.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.984+0000 7f46a8600640 1 -- 172.21.15.28:0/2439936489 shutdown_connections 2024-02-21T23:06:13.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.984+0000 7f46a8600640 1 --2- 172.21.15.28:0/2439936489 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f468006b500 0x7f468006d9c0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:13.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.984+0000 7f46a8600640 1 --2- 172.21.15.28:0/2439936489 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f46a00c8dc0 0x7f46a0145640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:13.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.984+0000 7f46a8600640 1 --2- 172.21.15.28:0/2439936489 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46a00cb360 0x7f46a0145100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:13.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.984+0000 7f46a8600640 1 --2- 172.21.15.28:0/2439936489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4688004020 0x7f46a0150540 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:13.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.984+0000 7f46a8600640 1 -- 172.21.15.28:0/2439936489 >> 172.21.15.28:0/2439936489 conn(0x7f46a00725a0 msgr2=0x7f46a0154a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:13.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.984+0000 7f46a8600640 1 -- 172.21.15.28:0/2439936489 shutdown_connections 2024-02-21T23:06:13.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:13.984+0000 7f46a8600640 1 -- 172.21.15.28:0/2439936489 wait complete. 2024-02-21T23:06:13.995 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs2' 2024-02-21T23:06:13.995 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:06:14.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.209+0000 7f3cb3600640 1 Processor -- start 2024-02-21T23:06:14.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.209+0000 7f3cb3600640 1 -- start start 2024-02-21T23:06:14.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.211+0000 7f3cb3600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3cac153950 0x7f3cac0c15d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:14.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.211+0000 7f3cb3600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3cac15b6d0 0x7f3cac15dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:14.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.211+0000 7f3cb3600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cac152e70 con 0x7f3cac0c11f0 2024-02-21T23:06:14.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.211+0000 7f3cb3600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cac0a5ec0 con 0x7f3cac153950 2024-02-21T23:06:14.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.211+0000 7f3cb3600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cac153150 con 0x7f3cac15b6d0 2024-02-21T23:06:14.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.211+0000 7f3cabe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3cac153950 0x7f3cac0c15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:14.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.211+0000 7f3cabe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3cac153950 0x7f3cac0c15d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54712/0 (socket says 172.21.15.28:54712) 2024-02-21T23:06:14.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.211+0000 7f3cabe00640 1 -- 172.21.15.28:0/2038403242 learned_addr learned my addr 172.21.15.28:0/2038403242 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:14.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.212+0000 7f3cb1600640 1 --2- 172.21.15.28:0/2038403242 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3cac15b6d0 0x7f3cac15dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:14.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.212+0000 7f3cabe00640 1 -- 172.21.15.28:0/2038403242 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3cac15b6d0 msgr2=0x7f3cac15dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:14.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.212+0000 7f3cabe00640 1 --2- 172.21.15.28:0/2038403242 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3cac15b6d0 0x7f3cac15dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:14.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.212+0000 7f3cabe00640 1 -- 172.21.15.28:0/2038403242 >> v1:172.21.15.81:6789/0 conn(0x7f3cac0c11f0 legacy=0x7f3cac06dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:14.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.212+0000 7f3cabe00640 1 -- 172.21.15.28:0/2038403242 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cac153430 con 0x7f3cac153950 2024-02-21T23:06:14.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.213+0000 7f3cabe00640 1 --2- 172.21.15.28:0/2038403242 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cac153950 0x7f3cac0c15d0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f3c98009870 tx=0x7f3c9802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a55c17bd6cc78c16 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:14.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.214+0000 7f3cab400640 1 -- 172.21.15.28:0/2038403242 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c98035020 con 0x7f3cac153950 2024-02-21T23:06:14.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.214+0000 7f3cab400640 1 -- 172.21.15.28:0/2038403242 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c98031640 con 0x7f3cac153950 2024-02-21T23:06:14.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.214+0000 7f3cab400640 1 -- 172.21.15.28:0/2038403242 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c9802d040 con 0x7f3cac153950 2024-02-21T23:06:14.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.214+0000 7f3cb3600640 1 -- 172.21.15.28:0/2038403242 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cac153950 msgr2=0x7f3cac0c15d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:14.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.214+0000 7f3cb3600640 1 --2- 172.21.15.28:0/2038403242 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cac153950 0x7f3cac0c15d0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f3c98009870 tx=0x7f3c9802b8f0 comp rx=0 tx=0).stop 2024-02-21T23:06:14.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.215+0000 7f3cb3600640 1 -- 172.21.15.28:0/2038403242 shutdown_connections 2024-02-21T23:06:14.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.215+0000 7f3cb3600640 1 --2- 172.21.15.28:0/2038403242 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3cac15b6d0 0x7f3cac15dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:14.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.215+0000 7f3cb3600640 1 --2- 172.21.15.28:0/2038403242 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cac153950 0x7f3cac0c15d0 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:14.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.215+0000 7f3cb3600640 1 -- 172.21.15.28:0/2038403242 >> 172.21.15.28:0/2038403242 conn(0x7f3cac0c67f0 msgr2=0x7f3cac0c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:14.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.215+0000 7f3cb3600640 1 -- 172.21.15.28:0/2038403242 shutdown_connections 2024-02-21T23:06:14.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.215+0000 7f3cb3600640 1 -- 172.21.15.28:0/2038403242 wait complete. 2024-02-21T23:06:14.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.216+0000 7f3cb3600640 1 Processor -- start 2024-02-21T23:06:14.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.217+0000 7f3cb3600640 1 -- start start 2024-02-21T23:06:14.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.217+0000 7f3cb3600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3cac0c11f0 0x7f3cac0a7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:14.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.217+0000 7f3cb3600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3cac153950 0x7f3cac0a7fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:14.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.218+0000 7f3cb3600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cac15b6d0 0x7f3cac1409d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:14.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.218+0000 7f3cb3600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cac0a6110 con 0x7f3cac15b6d0 2024-02-21T23:06:14.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.218+0000 7f3cb3600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cac0c4c00 con 0x7f3cac153950 2024-02-21T23:06:14.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.218+0000 7f3cb3600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cac0c0d90 con 0x7f3cac0c11f0 2024-02-21T23:06:14.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.218+0000 7f3cb1600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cac15b6d0 0x7f3cac1409d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:14.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.218+0000 7f3cb1600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cac15b6d0 0x7f3cac1409d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54728/0 (socket says 172.21.15.28:54728) 2024-02-21T23:06:14.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.218+0000 7f3cb1600640 1 -- 172.21.15.28:0/1410432209 learned_addr learned my addr 172.21.15.28:0/1410432209 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:14.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.218+0000 7f3cb0c00640 1 --2- 172.21.15.28:0/1410432209 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3cac0c11f0 0x7f3cac0a7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:14.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.218+0000 7f3cabe00640 1 --2- 172.21.15.28:0/1410432209 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3cac153950 0x7f3cac0a7fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:14.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.218+0000 7f3cb1600640 1 -- 172.21.15.28:0/1410432209 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3cac0c11f0 msgr2=0x7f3cac0a7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:14.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.218+0000 7f3cb1600640 1 --2- 172.21.15.28:0/1410432209 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3cac0c11f0 0x7f3cac0a7a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:14.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.218+0000 7f3cb1600640 1 -- 172.21.15.28:0/1410432209 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3cac153950 msgr2=0x7f3cac0a7fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:14.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.218+0000 7f3cb1600640 1 --2- 172.21.15.28:0/1410432209 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3cac153950 0x7f3cac0a7fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:14.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.218+0000 7f3cb1600640 1 -- 172.21.15.28:0/1410432209 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cac141c40 con 0x7f3cac15b6d0 2024-02-21T23:06:14.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.218+0000 7f3cb0c00640 1 --2- 172.21.15.28:0/1410432209 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3cac0c11f0 0x7f3cac0a7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:06:14.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.219+0000 7f3cb1600640 1 --2- 172.21.15.28:0/1410432209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cac15b6d0 0x7f3cac1409d0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f3ca4007e60 tx=0x7f3ca400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:14.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.219+0000 7f3ca8c00640 1 -- 172.21.15.28:0/1410432209 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ca4014420 con 0x7f3cac15b6d0 2024-02-21T23:06:14.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.219+0000 7f3ca8c00640 1 -- 172.21.15.28:0/1410432209 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ca4019070 con 0x7f3cac15b6d0 2024-02-21T23:06:14.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.219+0000 7f3ca8c00640 1 -- 172.21.15.28:0/1410432209 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ca4014820 con 0x7f3cac15b6d0 2024-02-21T23:06:14.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.219+0000 7f3cb3600640 1 -- 172.21.15.28:0/1410432209 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cac141f20 con 0x7f3cac15b6d0 2024-02-21T23:06:14.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.219+0000 7f3cb3600640 1 -- 172.21.15.28:0/1410432209 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cac1424e0 con 0x7f3cac15b6d0 2024-02-21T23:06:14.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.221+0000 7f3cb3600640 1 -- 172.21.15.28:0/1410432209 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cac067f30 con 0x7f3cac15b6d0 2024-02-21T23:06:14.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.221+0000 7f3ca8c00640 1 -- 172.21.15.28:0/1410432209 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3ca40149c0 con 0x7f3cac15b6d0 2024-02-21T23:06:14.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.222+0000 7f3ca8c00640 1 --2- 172.21.15.28:0/1410432209 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3c8c06b4d0 0x7f3c8c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:14.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.223+0000 7f3ca8c00640 1 -- 172.21.15.28:0/1410432209 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f3ca408ae00 con 0x7f3cac15b6d0 2024-02-21T23:06:14.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.223+0000 7f3cb0c00640 1 --2- 172.21.15.28:0/1410432209 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3c8c06b4d0 0x7f3c8c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:14.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.224+0000 7f3cb0c00640 1 --2- 172.21.15.28:0/1410432209 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3c8c06b4d0 0x7f3c8c06d990 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f3cac0a5d10 tx=0x7f3ca0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:14.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.227+0000 7f3ca8c00640 1 -- 172.21.15.28:0/1410432209 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3ca4059830 con 0x7f3cac15b6d0 2024-02-21T23:06:14.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.482+0000 7f3cb3600640 1 -- 172.21.15.28:0/1410432209 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3cac0c0960 con 0x7f3cac15b6d0 2024-02-21T23:06:14.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.483+0000 7f3ca8c00640 1 -- 172.21.15.28:0/1410432209 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+6175 (secure 0 0 0) 0x7f3ca4059130 con 0x7f3cac15b6d0 2024-02-21T23:06:14.472 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:06:14.472 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":67,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5288,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":8,"addr":"172.21.15.81:6835/2774951107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2774951107},{"type":"v1","addr":"172.21.15.81:6835","nonce":2774951107}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67},{"gid":5294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":8,"addr":"172.21.15.81:6837/4292940632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":4292940632},{"type":"v1","addr":"172.21.15.81:6837","nonce":4292940632}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:05:50.828988+0000","modified":"2024-02-21T23:05:53.937407+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6366},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6366":{"gid":6366,"name":"a","rank":0,"incarnation":58,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/595682298","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":595682298},{"type":"v1","addr":"172.21.15.28:6835","nonce":595682298}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5},{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:06:10.484421+0000","modified":"2024-02-21T23:06:13.492899+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5539},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5539":{"gid":5539,"name":"c","rank":0,"incarnation":63,"state":"up:active","state_seq":8,"addr":"172.21.15.28:6833/624384999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":624384999},{"type":"v1","addr":"172.21.15.28:6833","nonce":624384999}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-02-21T23:06:14.472 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 67 2024-02-21T23:06:14.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.485+0000 7f3cb3600640 1 -- 172.21.15.28:0/1410432209 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3c8c06b4d0 msgr2=0x7f3c8c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:14.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.485+0000 7f3cb3600640 1 --2- 172.21.15.28:0/1410432209 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3c8c06b4d0 0x7f3c8c06d990 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f3cac0a5d10 tx=0x7f3ca0009040 comp rx=0 tx=0).stop 2024-02-21T23:06:14.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.485+0000 7f3cb3600640 1 -- 172.21.15.28:0/1410432209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cac15b6d0 msgr2=0x7f3cac1409d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:14.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.485+0000 7f3cb3600640 1 --2- 172.21.15.28:0/1410432209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cac15b6d0 0x7f3cac1409d0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f3ca4007e60 tx=0x7f3ca400e4b0 comp rx=0 tx=0).stop 2024-02-21T23:06:14.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.485+0000 7f3cb3600640 1 -- 172.21.15.28:0/1410432209 shutdown_connections 2024-02-21T23:06:14.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.485+0000 7f3cb3600640 1 --2- 172.21.15.28:0/1410432209 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3c8c06b4d0 0x7f3c8c06d990 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:14.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.485+0000 7f3cb3600640 1 --2- 172.21.15.28:0/1410432209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cac15b6d0 0x7f3cac1409d0 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:14.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.485+0000 7f3cb3600640 1 --2- 172.21.15.28:0/1410432209 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3cac153950 0x7f3cac0a7fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:14.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.485+0000 7f3cb3600640 1 --2- 172.21.15.28:0/1410432209 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3cac0c11f0 0x7f3cac0a7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:14.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.485+0000 7f3cb3600640 1 -- 172.21.15.28:0/1410432209 >> 172.21.15.28:0/1410432209 conn(0x7f3cac0c67f0 msgr2=0x7f3cac06e5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:14.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.486+0000 7f3cb3600640 1 -- 172.21.15.28:0/1410432209 shutdown_connections 2024-02-21T23:06:14.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.486+0000 7f3cb3600640 1 -- 172.21.15.28:0/1410432209 wait complete. 2024-02-21T23:06:14.493 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:06:14.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.719+0000 7ff669c00640 1 Processor -- start 2024-02-21T23:06:14.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.720+0000 7ff669c00640 1 -- start start 2024-02-21T23:06:14.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.721+0000 7ff669c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff66413e620 0x7ff664159070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:14.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.721+0000 7ff669c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff6640c6460 0x7ff6640ca8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:14.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.721+0000 7ff669c00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6641551b0 con 0x7ff6640cadf0 2024-02-21T23:06:14.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.721+0000 7ff669c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6640c71d0 con 0x7ff6640c6460 2024-02-21T23:06:14.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.721+0000 7ff669c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6641595b0 con 0x7ff66413e620 2024-02-21T23:06:14.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.722+0000 7ff65be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff6640c6460 0x7ff6640ca8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:14.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.722+0000 7ff65be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff6640c6460 0x7ff6640ca8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54738/0 (socket says 172.21.15.28:54738) 2024-02-21T23:06:14.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.722+0000 7ff65be00640 1 -- 172.21.15.28:0/397054707 learned_addr learned my addr 172.21.15.28:0/397054707 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:14.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.722+0000 7ff662a00640 1 --2- 172.21.15.28:0/397054707 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff66413e620 0x7ff664159070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:14.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.722+0000 7ff65be00640 1 -- 172.21.15.28:0/397054707 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff66413e620 msgr2=0x7ff664159070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:14.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.722+0000 7ff65be00640 1 --2- 172.21.15.28:0/397054707 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff66413e620 0x7ff664159070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:14.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.722+0000 7ff65be00640 1 -- 172.21.15.28:0/397054707 >> v1:172.21.15.81:6789/0 conn(0x7ff6640cadf0 legacy=0x7ff6640bf990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:14.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.722+0000 7ff65be00640 1 -- 172.21.15.28:0/397054707 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff664155420 con 0x7ff6640c6460 2024-02-21T23:06:14.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.723+0000 7ff65be00640 1 --2- 172.21.15.28:0/397054707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6640c6460 0x7ff6640ca8b0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7ff64c009870 tx=0x7ff64c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9df4fc3c061caa1 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:14.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.724+0000 7ff662000640 1 -- 172.21.15.28:0/397054707 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff64c0317c0 con 0x7ff6640c6460 2024-02-21T23:06:14.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.724+0000 7ff662000640 1 -- 172.21.15.28:0/397054707 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff64c037070 con 0x7ff6640c6460 2024-02-21T23:06:14.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.724+0000 7ff662000640 1 -- 172.21.15.28:0/397054707 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff64c031b80 con 0x7ff6640c6460 2024-02-21T23:06:14.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.725+0000 7ff669c00640 1 -- 172.21.15.28:0/397054707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6640c6460 msgr2=0x7ff6640ca8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:14.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.725+0000 7ff669c00640 1 --2- 172.21.15.28:0/397054707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6640c6460 0x7ff6640ca8b0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7ff64c009870 tx=0x7ff64c02b860 comp rx=0 tx=0).stop 2024-02-21T23:06:14.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.725+0000 7ff669c00640 1 -- 172.21.15.28:0/397054707 shutdown_connections 2024-02-21T23:06:14.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.725+0000 7ff669c00640 1 --2- 172.21.15.28:0/397054707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6640c6460 0x7ff6640ca8b0 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:14.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.725+0000 7ff669c00640 1 --2- 172.21.15.28:0/397054707 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff66413e620 0x7ff664159070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:14.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.725+0000 7ff669c00640 1 -- 172.21.15.28:0/397054707 >> 172.21.15.28:0/397054707 conn(0x7ff66408b1c0 msgr2=0x7ff6640725a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:14.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.725+0000 7ff669c00640 1 -- 172.21.15.28:0/397054707 shutdown_connections 2024-02-21T23:06:14.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.725+0000 7ff669c00640 1 -- 172.21.15.28:0/397054707 wait complete. 2024-02-21T23:06:14.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.726+0000 7ff669c00640 1 Processor -- start 2024-02-21T23:06:14.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.727+0000 7ff669c00640 1 -- start start 2024-02-21T23:06:14.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.727+0000 7ff669c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6640c6460 0x7ff6640c7860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:14.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.727+0000 7ff669c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff6640cadf0 0x7ff6640c8d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:14.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.727+0000 7ff669c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff66413e620 0x7ff6640c92b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:14.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.727+0000 7ff669c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff664155cc0 con 0x7ff6640c6460 2024-02-21T23:06:14.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.728+0000 7ff669c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6641559e0 con 0x7ff6640cadf0 2024-02-21T23:06:14.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.728+0000 7ff669c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff66413ea00 con 0x7ff66413e620 2024-02-21T23:06:14.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.728+0000 7ff662a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6640c6460 0x7ff6640c7860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:14.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.728+0000 7ff662a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6640c6460 0x7ff6640c7860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54754/0 (socket says 172.21.15.28:54754) 2024-02-21T23:06:14.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.728+0000 7ff662a00640 1 -- 172.21.15.28:0/2171839591 learned_addr learned my addr 172.21.15.28:0/2171839591 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:14.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.728+0000 7ff65be00640 1 --2- 172.21.15.28:0/2171839591 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff6640cadf0 0x7ff6640c8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:14.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.728+0000 7ff662a00640 1 -- 172.21.15.28:0/2171839591 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff66413e620 msgr2=0x7ff6640c92b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:14.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.728+0000 7ff662a00640 1 --2- 172.21.15.28:0/2171839591 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff66413e620 0x7ff6640c92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:14.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.728+0000 7ff662a00640 1 -- 172.21.15.28:0/2171839591 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff6640cadf0 msgr2=0x7ff6640c8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:14.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.728+0000 7ff662a00640 1 --2- 172.21.15.28:0/2171839591 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff6640cadf0 0x7ff6640c8d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:14.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.728+0000 7ff662a00640 1 -- 172.21.15.28:0/2171839591 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6640ca730 con 0x7ff6640c6460 2024-02-21T23:06:14.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.729+0000 7ff662a00640 1 --2- 172.21.15.28:0/2171839591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6640c6460 0x7ff6640c7860 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7ff65400d5f0 tx=0x7ff65400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:14.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.729+0000 7ff65aa00640 1 -- 172.21.15.28:0/2171839591 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6540125f0 con 0x7ff6640c6460 2024-02-21T23:06:14.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.729+0000 7ff65aa00640 1 -- 172.21.15.28:0/2171839591 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff654019070 con 0x7ff6640c6460 2024-02-21T23:06:14.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.729+0000 7ff669c00640 1 -- 172.21.15.28:0/2171839591 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6641eb9f0 con 0x7ff6640c6460 2024-02-21T23:06:14.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.729+0000 7ff65aa00640 1 -- 172.21.15.28:0/2171839591 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6540129d0 con 0x7ff6640c6460 2024-02-21T23:06:14.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.729+0000 7ff669c00640 1 -- 172.21.15.28:0/2171839591 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6640c7da0 con 0x7ff6640c6460 2024-02-21T23:06:14.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.731+0000 7ff669c00640 1 -- 172.21.15.28:0/2171839591 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff664067f30 con 0x7ff6640c6460 2024-02-21T23:06:14.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.732+0000 7ff65aa00640 1 -- 172.21.15.28:0/2171839591 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff654012b70 con 0x7ff6640c6460 2024-02-21T23:06:14.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.733+0000 7ff65aa00640 1 --2- 172.21.15.28:0/2171839591 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff64006b4d0 0x7ff64006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:14.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.733+0000 7ff65aa00640 1 -- 172.21.15.28:0/2171839591 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6957+0+0 (secure 0 0 0) 0x7ff65408aa90 con 0x7ff6640c6460 2024-02-21T23:06:14.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.734+0000 7ff65be00640 1 --2- 172.21.15.28:0/2171839591 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff64006b4d0 0x7ff64006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:14.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.735+0000 7ff65be00640 1 --2- 172.21.15.28:0/2171839591 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff64006b4d0 0x7ff64006d990 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7ff64c036000 tx=0x7ff64c008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:14.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:14.742+0000 7ff65aa00640 1 -- 172.21.15.28:0/2171839591 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ff654059420 con 0x7ff6640c6460 2024-02-21T23:06:15.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.027+0000 7ff669c00640 1 -- 172.21.15.28:0/2171839591 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff6640cb630 con 0x7ff6640c6460 2024-02-21T23:06:15.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.030+0000 7ff65aa00640 1 -- 172.21.15.28:0/2171839591 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) v1 ==== 74+0+20609 (secure 0 0 0) 0x7ff654058d20 con 0x7ff6640c6460 2024-02-21T23:06:15.020 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:06:15.020 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":80,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:06:11.956457+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:05:48.772562+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-02-21T23:05:49.852659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs2_metadata","create_time":"2024-02-21T23:06:08.240443+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs2_data","create_time":"2024-02-21T23:06:09.466016+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:06:15.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.033+0000 7ff669c00640 1 -- 172.21.15.28:0/2171839591 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff64006b4d0 msgr2=0x7ff64006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:15.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.033+0000 7ff669c00640 1 --2- 172.21.15.28:0/2171839591 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff64006b4d0 0x7ff64006d990 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7ff64c036000 tx=0x7ff64c008030 comp rx=0 tx=0).stop 2024-02-21T23:06:15.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.033+0000 7ff669c00640 1 -- 172.21.15.28:0/2171839591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6640c6460 msgr2=0x7ff6640c7860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:15.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.033+0000 7ff669c00640 1 --2- 172.21.15.28:0/2171839591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6640c6460 0x7ff6640c7860 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7ff65400d5f0 tx=0x7ff65400dab0 comp rx=0 tx=0).stop 2024-02-21T23:06:15.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.033+0000 7ff669c00640 1 -- 172.21.15.28:0/2171839591 shutdown_connections 2024-02-21T23:06:15.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.033+0000 7ff669c00640 1 --2- 172.21.15.28:0/2171839591 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff64006b4d0 0x7ff64006d990 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:15.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.033+0000 7ff669c00640 1 --2- 172.21.15.28:0/2171839591 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff66413e620 0x7ff6640c92b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:15.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.033+0000 7ff669c00640 1 --2- 172.21.15.28:0/2171839591 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff6640cadf0 0x7ff6640c8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:15.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.033+0000 7ff669c00640 1 --2- 172.21.15.28:0/2171839591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6640c6460 0x7ff6640c7860 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:15.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.033+0000 7ff669c00640 1 -- 172.21.15.28:0/2171839591 >> 172.21.15.28:0/2171839591 conn(0x7ff66408b1c0 msgr2=0x7ff6640c0000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:15.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.033+0000 7ff669c00640 1 -- 172.21.15.28:0/2171839591 shutdown_connections 2024-02-21T23:06:15.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.034+0000 7ff669c00640 1 -- 172.21.15.28:0/2171839591 wait complete. 2024-02-21T23:06:15.041 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-02-21T23:06:15.042 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-21T23:06:15.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.271+0000 7ffaf7e00640 1 Processor -- start 2024-02-21T23:06:15.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.271+0000 7ffaf7e00640 1 -- start start 2024-02-21T23:06:15.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.274+0000 7ffaf7e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ffaf814ca60 0x7ffaf8072120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:15.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.274+0000 7ffaf7e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffaf806f550 0x7ffaf806ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:15.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.274+0000 7ffaf7e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaf806bb40 con 0x7ffaf814ce40 2024-02-21T23:06:15.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.274+0000 7ffaf7e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffaf80c5520 con 0x7ffaf814ca60 2024-02-21T23:06:15.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.274+0000 7ffaf7e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf8153700 con 0x7ffaf806f550 2024-02-21T23:06:15.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.274+0000 7ffaf6a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ffaf814ca60 0x7ffaf8072120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:15.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.275+0000 7ffaf6a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ffaf814ca60 0x7ffaf8072120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54608/0 (socket says 172.21.15.28:54608) 2024-02-21T23:06:15.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.275+0000 7ffaf6a00640 1 -- 172.21.15.28:0/3111663634 learned_addr learned my addr 172.21.15.28:0/3111663634 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:15.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.275+0000 7ffaf6000640 1 --2- 172.21.15.28:0/3111663634 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffaf806f550 0x7ffaf806ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:15.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.275+0000 7ffaf6a00640 1 -- 172.21.15.28:0/3111663634 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffaf806f550 msgr2=0x7ffaf806ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:15.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.275+0000 7ffaf6a00640 1 --2- 172.21.15.28:0/3111663634 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffaf806f550 0x7ffaf806ae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:15.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.275+0000 7ffaf6a00640 1 -- 172.21.15.28:0/3111663634 >> v1:172.21.15.81:6789/0 conn(0x7ffaf814ce40 legacy=0x7ffaf81561d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:15.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.275+0000 7ffaf6a00640 1 -- 172.21.15.28:0/3111663634 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaf81539e0 con 0x7ffaf814ca60 2024-02-21T23:06:15.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.276+0000 7ffaf6a00640 1 --2- 172.21.15.28:0/3111663634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffaf814ca60 0x7ffaf8072120 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7ffae8009870 tx=0x7ffae802b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e93a934664527b1 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:15.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.277+0000 7ffaf5600640 1 -- 172.21.15.28:0/3111663634 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae8038070 con 0x7ffaf814ca60 2024-02-21T23:06:15.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.277+0000 7ffaf5600640 1 -- 172.21.15.28:0/3111663634 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffae8031740 con 0x7ffaf814ca60 2024-02-21T23:06:15.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.277+0000 7ffaf5600640 1 -- 172.21.15.28:0/3111663634 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae802f050 con 0x7ffaf814ca60 2024-02-21T23:06:15.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.277+0000 7ffaf7e00640 1 -- 172.21.15.28:0/3111663634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffaf814ca60 msgr2=0x7ffaf8072120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:15.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.277+0000 7ffaf7e00640 1 --2- 172.21.15.28:0/3111663634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffaf814ca60 0x7ffaf8072120 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7ffae8009870 tx=0x7ffae802b7e0 comp rx=0 tx=0).stop 2024-02-21T23:06:15.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.277+0000 7ffaf7e00640 1 -- 172.21.15.28:0/3111663634 shutdown_connections 2024-02-21T23:06:15.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.277+0000 7ffaf7e00640 1 --2- 172.21.15.28:0/3111663634 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffaf806f550 0x7ffaf806ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:15.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.277+0000 7ffaf7e00640 1 --2- 172.21.15.28:0/3111663634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffaf814ca60 0x7ffaf8072120 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:15.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.277+0000 7ffaf7e00640 1 -- 172.21.15.28:0/3111663634 >> 172.21.15.28:0/3111663634 conn(0x7ffaf8067050 msgr2=0x7ffaf806bf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:15.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.278+0000 7ffaf7e00640 1 -- 172.21.15.28:0/3111663634 shutdown_connections 2024-02-21T23:06:15.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.278+0000 7ffaf7e00640 1 -- 172.21.15.28:0/3111663634 wait complete. 2024-02-21T23:06:15.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.279+0000 7ffaf7e00640 1 Processor -- start 2024-02-21T23:06:15.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.279+0000 7ffaf7e00640 1 -- start start 2024-02-21T23:06:15.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.280+0000 7ffaf7e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffaf806f550 0x7ffaf81ebe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:15.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.280+0000 7ffaf7e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffaf814ca60 0x7ffaf8149230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:15.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.280+0000 7ffaf7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffaf814ce40 0x7ffaf81514f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:15.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.280+0000 7ffaf7e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf80727e0 con 0x7ffaf814ce40 2024-02-21T23:06:15.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.280+0000 7ffaf7e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf8072660 con 0x7ffaf806f550 2024-02-21T23:06:15.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.280+0000 7ffaf7e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf806f030 con 0x7ffaf814ca60 2024-02-21T23:06:15.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.281+0000 7ffaf7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffaf814ce40 0x7ffaf81514f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:15.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.281+0000 7ffaf7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffaf814ce40 0x7ffaf81514f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54616/0 (socket says 172.21.15.28:54616) 2024-02-21T23:06:15.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.281+0000 7ffaf7400640 1 -- 172.21.15.28:0/1521957444 learned_addr learned my addr 172.21.15.28:0/1521957444 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:15.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.281+0000 7ffaf6000640 1 --2- 172.21.15.28:0/1521957444 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffaf814ca60 0x7ffaf8149230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:15.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.281+0000 7ffaf6a00640 1 --2- 172.21.15.28:0/1521957444 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffaf806f550 0x7ffaf81ebe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:15.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.281+0000 7ffaf7400640 1 -- 172.21.15.28:0/1521957444 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffaf814ca60 msgr2=0x7ffaf8149230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:15.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.281+0000 7ffaf7400640 1 --2- 172.21.15.28:0/1521957444 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffaf814ca60 0x7ffaf8149230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:15.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.281+0000 7ffaf7400640 1 -- 172.21.15.28:0/1521957444 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffaf806f550 msgr2=0x7ffaf81ebe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:15.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.281+0000 7ffaf7400640 1 --2- 172.21.15.28:0/1521957444 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffaf806f550 0x7ffaf81ebe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:15.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.281+0000 7ffaf7400640 1 -- 172.21.15.28:0/1521957444 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaf81ec670 con 0x7ffaf814ce40 2024-02-21T23:06:15.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.281+0000 7ffaf6a00640 1 --2- 172.21.15.28:0/1521957444 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffaf806f550 0x7ffaf81ebe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:06:15.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.282+0000 7ffaf7400640 1 --2- 172.21.15.28:0/1521957444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffaf814ce40 0x7ffaf81514f0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7ffaf000c730 tx=0x7ffaf000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:15.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.282+0000 7ffaeea00640 1 -- 172.21.15.28:0/1521957444 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf0016020 con 0x7ffaf814ce40 2024-02-21T23:06:15.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.282+0000 7ffaeea00640 1 -- 172.21.15.28:0/1521957444 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaf000aa50 con 0x7ffaf814ce40 2024-02-21T23:06:15.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.282+0000 7ffaf7e00640 1 -- 172.21.15.28:0/1521957444 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaf81ec950 con 0x7ffaf814ce40 2024-02-21T23:06:15.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.282+0000 7ffaeea00640 1 -- 172.21.15.28:0/1521957444 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf000ce40 con 0x7ffaf814ce40 2024-02-21T23:06:15.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.282+0000 7ffaf7e00640 1 -- 172.21.15.28:0/1521957444 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaf81ec370 con 0x7ffaf814ce40 2024-02-21T23:06:15.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.283+0000 7ffaf7e00640 1 -- 172.21.15.28:0/1521957444 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffaf80c6420 con 0x7ffaf814ce40 2024-02-21T23:06:15.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.284+0000 7ffaeea00640 1 -- 172.21.15.28:0/1521957444 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffaf0019020 con 0x7ffaf814ce40 2024-02-21T23:06:15.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.286+0000 7ffaeea00640 1 --2- 172.21.15.28:0/1521957444 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ffad806b500 0x7ffad806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:15.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.286+0000 7ffaeea00640 1 -- 172.21.15.28:0/1521957444 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6957+0+0 (secure 0 0 0) 0x7ffaf008a640 con 0x7ffaf814ce40 2024-02-21T23:06:15.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.287+0000 7ffaf6a00640 1 --2- 172.21.15.28:0/1521957444 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ffad806b500 0x7ffad806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:15.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.288+0000 7ffaf6a00640 1 --2- 172.21.15.28:0/1521957444 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ffad806b500 0x7ffad806d9c0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7ffae8035000 tx=0x7ffae8008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:15.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.290+0000 7ffaeea00640 1 -- 172.21.15.28:0/1521957444 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ffaf0020030 con 0x7ffaf814ce40 2024-02-21T23:06:15.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.495+0000 7ffaf7e00640 1 -- 172.21.15.28:0/1521957444 --> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ffaf804f290 con 0x7ffad806b500 2024-02-21T23:06:15.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.522+0000 7ffaeea00640 1 -- 172.21.15.28:0/1521957444 <== mgr.4101 v2:172.21.15.81:6832/1848538577 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+485422 (secure 0 0 0) 0x7ffaf804f290 con 0x7ffad806b500 2024-02-21T23:06:15.523 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:06:15.527 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-21T23:06:15.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.533+0000 7ffaf7e00640 1 -- 172.21.15.28:0/1521957444 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ffad806b500 msgr2=0x7ffad806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:15.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.533+0000 7ffaf7e00640 1 --2- 172.21.15.28:0/1521957444 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ffad806b500 0x7ffad806d9c0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7ffae8035000 tx=0x7ffae8008030 comp rx=0 tx=0).stop 2024-02-21T23:06:15.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.533+0000 7ffaf7e00640 1 -- 172.21.15.28:0/1521957444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffaf814ce40 msgr2=0x7ffaf81514f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:15.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.533+0000 7ffaf7e00640 1 --2- 172.21.15.28:0/1521957444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffaf814ce40 0x7ffaf81514f0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7ffaf000c730 tx=0x7ffaf000cbf0 comp rx=0 tx=0).stop 2024-02-21T23:06:15.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.533+0000 7ffaf7e00640 1 -- 172.21.15.28:0/1521957444 shutdown_connections 2024-02-21T23:06:15.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.533+0000 7ffaf7e00640 1 --2- 172.21.15.28:0/1521957444 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ffad806b500 0x7ffad806d9c0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:15.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.533+0000 7ffaf7e00640 1 --2- 172.21.15.28:0/1521957444 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffaf814ca60 0x7ffaf8149230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:15.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.533+0000 7ffaf7e00640 1 --2- 172.21.15.28:0/1521957444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffaf814ce40 0x7ffaf81514f0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:15.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.533+0000 7ffaf7e00640 1 --2- 172.21.15.28:0/1521957444 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffaf806f550 0x7ffaf81ebe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:15.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.533+0000 7ffaf7e00640 1 -- 172.21.15.28:0/1521957444 >> 172.21.15.28:0/1521957444 conn(0x7ffaf8067050 msgr2=0x7ffaf8074360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:15.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.533+0000 7ffaf7e00640 1 -- 172.21.15.28:0/1521957444 shutdown_connections 2024-02-21T23:06:15.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.533+0000 7ffaf7e00640 1 -- 172.21.15.28:0/1521957444 wait complete. 2024-02-21T23:06:15.541 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":185,"stamp":"2024-02-21T23:06:14.741815+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":228,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":252936,"kb_used_data":7636,"kb_used_omap":12,"kb_used_meta":245171,"kb_avail":754721784,"statfs":{"total":773094113280,"available":772835106816,"internally_reserved":0,"allocated":7819264,"data_stored":4842858,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":251055665},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.833080"},"pg_stats":[{"pgid":"14.3d","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.940039+0000","last_change":"2024-02-21T23:06:09.940039+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.940039+0000","last_undegraded":"2024-02-21T23:06:09.940039+0000","last_fullsized":"2024-02-21T23:06:09.940039+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.3f","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.946468+0000","last_change":"2024-02-21T23:05:51.408831+0000","last_active":"2024-02-21T23:06:09.946468+0000","last_peered":"2024-02-21T23:06:09.946468+0000","last_clean":"2024-02-21T23:06:09.946468+0000","last_became_active":"2024-02-21T23:05:51.408353+0000","last_became_peered":"2024-02-21T23:05:51.408353+0000","last_unstale":"2024-02-21T23:06:09.946468+0000","last_undegraded":"2024-02-21T23:06:09.946468+0000","last_fullsized":"2024-02-21T23:06:09.946468+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:36:47.962935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.3c","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.931414+0000","last_change":"2024-02-21T23:06:09.931414+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.931414+0000","last_undegraded":"2024-02-21T23:06:09.931414+0000","last_fullsized":"2024-02-21T23:06:09.931414+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.944046+0000","last_change":"2024-02-21T23:05:51.420557+0000","last_active":"2024-02-21T23:06:09.944046+0000","last_peered":"2024-02-21T23:06:09.944046+0000","last_clean":"2024-02-21T23:06:09.944046+0000","last_became_active":"2024-02-21T23:05:51.420078+0000","last_became_peered":"2024-02-21T23:05:51.420078+0000","last_unstale":"2024-02-21T23:06:09.944046+0000","last_undegraded":"2024-02-21T23:06:09.944046+0000","last_fullsized":"2024-02-21T23:06:09.944046+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:36:23.792395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.3f","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.936313+0000","last_change":"2024-02-21T23:06:09.936313+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.936313+0000","last_undegraded":"2024-02-21T23:06:09.936313+0000","last_fullsized":"2024-02-21T23:06:09.936313+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.919954+0000","last_change":"2024-02-21T23:05:51.408934+0000","last_active":"2024-02-21T23:06:09.919954+0000","last_peered":"2024-02-21T23:06:09.919954+0000","last_clean":"2024-02-21T23:06:09.919954+0000","last_became_active":"2024-02-21T23:05:51.408597+0000","last_became_peered":"2024-02-21T23:05:51.408597+0000","last_unstale":"2024-02-21T23:06:09.919954+0000","last_undegraded":"2024-02-21T23:06:09.919954+0000","last_fullsized":"2024-02-21T23:06:09.919954+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:58:55.829941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.3e","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.924703+0000","last_change":"2024-02-21T23:06:09.924703+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.924703+0000","last_undegraded":"2024-02-21T23:06:09.924703+0000","last_fullsized":"2024-02-21T23:06:09.924703+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.934821+0000","last_change":"2024-02-21T23:05:51.423479+0000","last_active":"2024-02-21T23:06:09.934821+0000","last_peered":"2024-02-21T23:06:09.934821+0000","last_clean":"2024-02-21T23:06:09.934821+0000","last_became_active":"2024-02-21T23:05:51.423108+0000","last_became_peered":"2024-02-21T23:05:51.423108+0000","last_unstale":"2024-02-21T23:06:09.934821+0000","last_undegraded":"2024-02-21T23:06:09.934821+0000","last_fullsized":"2024-02-21T23:06:09.934821+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:36:47.962935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.39","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.936144+0000","last_change":"2024-02-21T23:06:09.936144+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.936144+0000","last_undegraded":"2024-02-21T23:06:09.936144+0000","last_fullsized":"2024-02-21T23:06:09.936144+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.940522+0000","last_change":"2024-02-21T23:05:51.419436+0000","last_active":"2024-02-21T23:06:09.940522+0000","last_peered":"2024-02-21T23:06:09.940522+0000","last_clean":"2024-02-21T23:06:09.940522+0000","last_became_active":"2024-02-21T23:05:51.418562+0000","last_became_peered":"2024-02-21T23:05:51.418562+0000","last_unstale":"2024-02-21T23:06:09.940522+0000","last_undegraded":"2024-02-21T23:06:09.940522+0000","last_fullsized":"2024-02-21T23:06:09.940522+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:19:45.189126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.38","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.932838+0000","last_change":"2024-02-21T23:06:09.932838+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.932838+0000","last_undegraded":"2024-02-21T23:06:09.932838+0000","last_fullsized":"2024-02-21T23:06:09.932838+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.945821+0000","last_change":"2024-02-21T23:05:51.410014+0000","last_active":"2024-02-21T23:06:09.945821+0000","last_peered":"2024-02-21T23:06:09.945821+0000","last_clean":"2024-02-21T23:06:09.945821+0000","last_became_active":"2024-02-21T23:05:51.409637+0000","last_became_peered":"2024-02-21T23:05:51.409637+0000","last_unstale":"2024-02-21T23:06:09.945821+0000","last_undegraded":"2024-02-21T23:06:09.945821+0000","last_fullsized":"2024-02-21T23:06:09.945821+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:03:35.243949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3b","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.943914+0000","last_change":"2024-02-21T23:06:09.943914+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.943914+0000","last_undegraded":"2024-02-21T23:06:09.943914+0000","last_fullsized":"2024-02-21T23:06:09.943914+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.934473+0000","last_change":"2024-02-21T23:05:51.419505+0000","last_active":"2024-02-21T23:06:09.934473+0000","last_peered":"2024-02-21T23:06:09.934473+0000","last_clean":"2024-02-21T23:06:09.934473+0000","last_became_active":"2024-02-21T23:05:51.418040+0000","last_became_peered":"2024-02-21T23:05:51.418040+0000","last_unstale":"2024-02-21T23:06:09.934473+0000","last_undegraded":"2024-02-21T23:06:09.934473+0000","last_fullsized":"2024-02-21T23:06:09.934473+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:57:06.211024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.3a","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.940874+0000","last_change":"2024-02-21T23:06:09.940874+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.940874+0000","last_undegraded":"2024-02-21T23:06:09.940874+0000","last_fullsized":"2024-02-21T23:06:09.940874+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.930980+0000","last_change":"2024-02-21T23:05:51.416524+0000","last_active":"2024-02-21T23:06:09.930980+0000","last_peered":"2024-02-21T23:06:09.930980+0000","last_clean":"2024-02-21T23:06:09.930980+0000","last_became_active":"2024-02-21T23:05:51.415391+0000","last_became_peered":"2024-02-21T23:05:51.415391+0000","last_unstale":"2024-02-21T23:06:09.930980+0000","last_undegraded":"2024-02-21T23:06:09.930980+0000","last_fullsized":"2024-02-21T23:06:09.930980+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:48:32.958103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.35","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.938982+0000","last_change":"2024-02-21T23:06:09.938982+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.938982+0000","last_undegraded":"2024-02-21T23:06:09.938982+0000","last_fullsized":"2024-02-21T23:06:09.938982+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.950904+0000","last_change":"2024-02-21T23:05:51.421068+0000","last_active":"2024-02-21T23:06:09.950904+0000","last_peered":"2024-02-21T23:06:09.950904+0000","last_clean":"2024-02-21T23:06:09.950904+0000","last_became_active":"2024-02-21T23:05:51.419746+0000","last_became_peered":"2024-02-21T23:05:51.419746+0000","last_unstale":"2024-02-21T23:06:09.950904+0000","last_undegraded":"2024-02-21T23:06:09.950904+0000","last_fullsized":"2024-02-21T23:06:09.950904+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:09:42.444733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.34","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.937566+0000","last_change":"2024-02-21T23:06:09.937566+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.937566+0000","last_undegraded":"2024-02-21T23:06:09.937566+0000","last_fullsized":"2024-02-21T23:06:09.937566+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.934177+0000","last_change":"2024-02-21T23:05:51.418295+0000","last_active":"2024-02-21T23:06:09.934177+0000","last_peered":"2024-02-21T23:06:09.934177+0000","last_clean":"2024-02-21T23:06:09.934177+0000","last_became_active":"2024-02-21T23:05:51.417780+0000","last_became_peered":"2024-02-21T23:05:51.417780+0000","last_unstale":"2024-02-21T23:06:09.934177+0000","last_undegraded":"2024-02-21T23:06:09.934177+0000","last_fullsized":"2024-02-21T23:06:09.934177+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:19:57.210565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.37","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.936387+0000","last_change":"2024-02-21T23:06:09.936387+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.936387+0000","last_undegraded":"2024-02-21T23:06:09.936387+0000","last_fullsized":"2024-02-21T23:06:09.936387+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.951775+0000","last_change":"2024-02-21T23:05:51.420518+0000","last_active":"2024-02-21T23:06:09.951775+0000","last_peered":"2024-02-21T23:06:09.951775+0000","last_clean":"2024-02-21T23:06:09.951775+0000","last_became_active":"2024-02-21T23:05:51.419564+0000","last_became_peered":"2024-02-21T23:05:51.419564+0000","last_unstale":"2024-02-21T23:06:09.951775+0000","last_undegraded":"2024-02-21T23:06:09.951775+0000","last_fullsized":"2024-02-21T23:06:09.951775+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:19:41.280587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.36","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.932127+0000","last_change":"2024-02-21T23:06:09.932127+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.932127+0000","last_undegraded":"2024-02-21T23:06:09.932127+0000","last_fullsized":"2024-02-21T23:06:09.932127+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.934938+0000","last_change":"2024-02-21T23:05:51.421691+0000","last_active":"2024-02-21T23:06:09.934938+0000","last_peered":"2024-02-21T23:06:09.934938+0000","last_clean":"2024-02-21T23:06:09.934938+0000","last_became_active":"2024-02-21T23:05:51.417621+0000","last_became_peered":"2024-02-21T23:05:51.417621+0000","last_unstale":"2024-02-21T23:06:09.934938+0000","last_undegraded":"2024-02-21T23:06:09.934938+0000","last_fullsized":"2024-02-21T23:06:09.934938+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:46:01.608149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.31","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.929205+0000","last_change":"2024-02-21T23:06:09.929205+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.929205+0000","last_undegraded":"2024-02-21T23:06:09.929205+0000","last_fullsized":"2024-02-21T23:06:09.929205+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.940619+0000","last_change":"2024-02-21T23:05:51.426871+0000","last_active":"2024-02-21T23:06:09.940619+0000","last_peered":"2024-02-21T23:06:09.940619+0000","last_clean":"2024-02-21T23:06:09.940619+0000","last_became_active":"2024-02-21T23:05:51.419412+0000","last_became_peered":"2024-02-21T23:05:51.419412+0000","last_unstale":"2024-02-21T23:06:09.940619+0000","last_undegraded":"2024-02-21T23:06:09.940619+0000","last_fullsized":"2024-02-21T23:06:09.940619+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:22:09.156621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.30","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.929405+0000","last_change":"2024-02-21T23:06:09.929405+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.929405+0000","last_undegraded":"2024-02-21T23:06:09.929405+0000","last_fullsized":"2024-02-21T23:06:09.929405+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.945911+0000","last_change":"2024-02-21T23:05:51.417593+0000","last_active":"2024-02-21T23:06:09.945911+0000","last_peered":"2024-02-21T23:06:09.945911+0000","last_clean":"2024-02-21T23:06:09.945911+0000","last_became_active":"2024-02-21T23:05:51.416945+0000","last_became_peered":"2024-02-21T23:05:51.416945+0000","last_unstale":"2024-02-21T23:06:09.945911+0000","last_undegraded":"2024-02-21T23:06:09.945911+0000","last_fullsized":"2024-02-21T23:06:09.945911+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:24:32.863072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.33","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.929330+0000","last_change":"2024-02-21T23:06:09.929330+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.929330+0000","last_undegraded":"2024-02-21T23:06:09.929330+0000","last_fullsized":"2024-02-21T23:06:09.929330+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.943577+0000","last_change":"2024-02-21T23:05:51.409645+0000","last_active":"2024-02-21T23:06:09.943577+0000","last_peered":"2024-02-21T23:06:09.943577+0000","last_clean":"2024-02-21T23:06:09.943577+0000","last_became_active":"2024-02-21T23:05:51.409356+0000","last_became_peered":"2024-02-21T23:05:51.409356+0000","last_unstale":"2024-02-21T23:06:09.943577+0000","last_undegraded":"2024-02-21T23:06:09.943577+0000","last_fullsized":"2024-02-21T23:06:09.943577+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:17:26.329609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.32","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.942772+0000","last_change":"2024-02-21T23:06:09.942772+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.942772+0000","last_undegraded":"2024-02-21T23:06:09.942772+0000","last_fullsized":"2024-02-21T23:06:09.942772+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.941967+0000","last_change":"2024-02-21T23:05:51.413656+0000","last_active":"2024-02-21T23:06:09.941967+0000","last_peered":"2024-02-21T23:06:09.941967+0000","last_clean":"2024-02-21T23:06:09.941967+0000","last_became_active":"2024-02-21T23:05:51.413350+0000","last_became_peered":"2024-02-21T23:05:51.413350+0000","last_unstale":"2024-02-21T23:06:09.941967+0000","last_undegraded":"2024-02-21T23:06:09.941967+0000","last_fullsized":"2024-02-21T23:06:09.941967+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:19:57.210565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.2d","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.938385+0000","last_change":"2024-02-21T23:06:09.938385+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.938385+0000","last_undegraded":"2024-02-21T23:06:09.938385+0000","last_fullsized":"2024-02-21T23:06:09.938385+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.930801+0000","last_change":"2024-02-21T23:05:51.427044+0000","last_active":"2024-02-21T23:06:09.930801+0000","last_peered":"2024-02-21T23:06:09.930801+0000","last_clean":"2024-02-21T23:06:09.930801+0000","last_became_active":"2024-02-21T23:05:51.419477+0000","last_became_peered":"2024-02-21T23:05:51.419477+0000","last_unstale":"2024-02-21T23:06:09.930801+0000","last_undegraded":"2024-02-21T23:06:09.930801+0000","last_fullsized":"2024-02-21T23:06:09.930801+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:04:00.714448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.2c","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.929202+0000","last_change":"2024-02-21T23:06:09.929202+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.929202+0000","last_undegraded":"2024-02-21T23:06:09.929202+0000","last_fullsized":"2024-02-21T23:06:09.929202+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.920081+0000","last_change":"2024-02-21T23:05:51.425126+0000","last_active":"2024-02-21T23:06:09.920081+0000","last_peered":"2024-02-21T23:06:09.920081+0000","last_clean":"2024-02-21T23:06:09.920081+0000","last_became_active":"2024-02-21T23:05:51.422511+0000","last_became_peered":"2024-02-21T23:05:51.422511+0000","last_unstale":"2024-02-21T23:06:09.920081+0000","last_undegraded":"2024-02-21T23:06:09.920081+0000","last_fullsized":"2024-02-21T23:06:09.920081+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:27:01.191337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.2f","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.939567+0000","last_change":"2024-02-21T23:06:09.939567+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.939567+0000","last_undegraded":"2024-02-21T23:06:09.939567+0000","last_fullsized":"2024-02-21T23:06:09.939567+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.946776+0000","last_change":"2024-02-21T23:05:51.423258+0000","last_active":"2024-02-21T23:06:09.946776+0000","last_peered":"2024-02-21T23:06:09.946776+0000","last_clean":"2024-02-21T23:06:09.946776+0000","last_became_active":"2024-02-21T23:05:51.422987+0000","last_became_peered":"2024-02-21T23:05:51.422987+0000","last_unstale":"2024-02-21T23:06:09.946776+0000","last_undegraded":"2024-02-21T23:06:09.946776+0000","last_fullsized":"2024-02-21T23:06:09.946776+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:41:31.260842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2e","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.941381+0000","last_change":"2024-02-21T23:06:09.941381+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.941381+0000","last_undegraded":"2024-02-21T23:06:09.941381+0000","last_fullsized":"2024-02-21T23:06:09.941381+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.937988+0000","last_change":"2024-02-21T23:05:51.427468+0000","last_active":"2024-02-21T23:06:09.937988+0000","last_peered":"2024-02-21T23:06:09.937988+0000","last_clean":"2024-02-21T23:06:09.937988+0000","last_became_active":"2024-02-21T23:05:51.425476+0000","last_became_peered":"2024-02-21T23:05:51.425476+0000","last_unstale":"2024-02-21T23:06:09.937988+0000","last_undegraded":"2024-02-21T23:06:09.937988+0000","last_fullsized":"2024-02-21T23:06:09.937988+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:06:18.158392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.29","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.929062+0000","last_change":"2024-02-21T23:06:09.929062+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.929062+0000","last_undegraded":"2024-02-21T23:06:09.929062+0000","last_fullsized":"2024-02-21T23:06:09.929062+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.948695+0000","last_change":"2024-02-21T23:05:51.415772+0000","last_active":"2024-02-21T23:06:09.948695+0000","last_peered":"2024-02-21T23:06:09.948695+0000","last_clean":"2024-02-21T23:06:09.948695+0000","last_became_active":"2024-02-21T23:05:51.415494+0000","last_became_peered":"2024-02-21T23:05:51.415494+0000","last_unstale":"2024-02-21T23:06:09.948695+0000","last_undegraded":"2024-02-21T23:06:09.948695+0000","last_fullsized":"2024-02-21T23:06:09.948695+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:42:37.006685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.28","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.932618+0000","last_change":"2024-02-21T23:06:09.932618+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.932618+0000","last_undegraded":"2024-02-21T23:06:09.932618+0000","last_fullsized":"2024-02-21T23:06:09.932618+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.937010+0000","last_change":"2024-02-21T23:05:51.428557+0000","last_active":"2024-02-21T23:06:09.937010+0000","last_peered":"2024-02-21T23:06:09.937010+0000","last_clean":"2024-02-21T23:06:09.937010+0000","last_became_active":"2024-02-21T23:05:51.427612+0000","last_became_peered":"2024-02-21T23:05:51.427612+0000","last_unstale":"2024-02-21T23:06:09.937010+0000","last_undegraded":"2024-02-21T23:06:09.937010+0000","last_fullsized":"2024-02-21T23:06:09.937010+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:38:44.895318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.2b","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.940735+0000","last_change":"2024-02-21T23:06:09.940735+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.940735+0000","last_undegraded":"2024-02-21T23:06:09.940735+0000","last_fullsized":"2024-02-21T23:06:09.940735+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.944849+0000","last_change":"2024-02-21T23:05:51.417302+0000","last_active":"2024-02-21T23:06:09.944849+0000","last_peered":"2024-02-21T23:06:09.944849+0000","last_clean":"2024-02-21T23:06:09.944849+0000","last_became_active":"2024-02-21T23:05:51.414092+0000","last_became_peered":"2024-02-21T23:05:51.414092+0000","last_unstale":"2024-02-21T23:06:09.944849+0000","last_undegraded":"2024-02-21T23:06:09.944849+0000","last_fullsized":"2024-02-21T23:06:09.944849+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:49:01.911867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2a","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.941237+0000","last_change":"2024-02-21T23:06:09.941237+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.941237+0000","last_undegraded":"2024-02-21T23:06:09.941237+0000","last_fullsized":"2024-02-21T23:06:09.941237+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.944272+0000","last_change":"2024-02-21T23:05:51.406411+0000","last_active":"2024-02-21T23:06:09.944272+0000","last_peered":"2024-02-21T23:06:09.944272+0000","last_clean":"2024-02-21T23:06:09.944272+0000","last_became_active":"2024-02-21T23:05:51.406104+0000","last_became_peered":"2024-02-21T23:05:51.406104+0000","last_unstale":"2024-02-21T23:06:09.944272+0000","last_undegraded":"2024-02-21T23:06:09.944272+0000","last_fullsized":"2024-02-21T23:06:09.944272+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:03:08.789066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.25","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.931274+0000","last_change":"2024-02-21T23:06:09.931274+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.931274+0000","last_undegraded":"2024-02-21T23:06:09.931274+0000","last_fullsized":"2024-02-21T23:06:09.931274+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.948003+0000","last_change":"2024-02-21T23:05:51.414112+0000","last_active":"2024-02-21T23:06:09.948003+0000","last_peered":"2024-02-21T23:06:09.948003+0000","last_clean":"2024-02-21T23:06:09.948003+0000","last_became_active":"2024-02-21T23:05:51.413796+0000","last_became_peered":"2024-02-21T23:05:51.413796+0000","last_unstale":"2024-02-21T23:06:09.948003+0000","last_undegraded":"2024-02-21T23:06:09.948003+0000","last_fullsized":"2024-02-21T23:06:09.948003+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:42:26.488510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.24","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.928558+0000","last_change":"2024-02-21T23:06:09.928558+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.928558+0000","last_undegraded":"2024-02-21T23:06:09.928558+0000","last_fullsized":"2024-02-21T23:06:09.928558+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.934345+0000","last_change":"2024-02-21T23:05:51.421208+0000","last_active":"2024-02-21T23:06:09.934345+0000","last_peered":"2024-02-21T23:06:09.934345+0000","last_clean":"2024-02-21T23:06:09.934345+0000","last_became_active":"2024-02-21T23:05:51.418171+0000","last_became_peered":"2024-02-21T23:05:51.418171+0000","last_unstale":"2024-02-21T23:06:09.934345+0000","last_undegraded":"2024-02-21T23:06:09.934345+0000","last_fullsized":"2024-02-21T23:06:09.934345+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:17:26.329609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.27","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.925950+0000","last_change":"2024-02-21T23:06:09.925950+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.925950+0000","last_undegraded":"2024-02-21T23:06:09.925950+0000","last_fullsized":"2024-02-21T23:06:09.925950+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.944600+0000","last_change":"2024-02-21T23:05:51.416857+0000","last_active":"2024-02-21T23:06:09.944600+0000","last_peered":"2024-02-21T23:06:09.944600+0000","last_clean":"2024-02-21T23:06:09.944600+0000","last_became_active":"2024-02-21T23:05:51.416577+0000","last_became_peered":"2024-02-21T23:05:51.416577+0000","last_unstale":"2024-02-21T23:06:09.944600+0000","last_undegraded":"2024-02-21T23:06:09.944600+0000","last_fullsized":"2024-02-21T23:06:09.944600+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:17:26.329609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.26","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.940425+0000","last_change":"2024-02-21T23:06:09.940425+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.940425+0000","last_undegraded":"2024-02-21T23:06:09.940425+0000","last_fullsized":"2024-02-21T23:06:09.940425+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.938091+0000","last_change":"2024-02-21T23:05:51.427489+0000","last_active":"2024-02-21T23:06:09.938091+0000","last_peered":"2024-02-21T23:06:09.938091+0000","last_clean":"2024-02-21T23:06:09.938091+0000","last_became_active":"2024-02-21T23:05:51.419458+0000","last_became_peered":"2024-02-21T23:05:51.419458+0000","last_unstale":"2024-02-21T23:06:09.938091+0000","last_undegraded":"2024-02-21T23:06:09.938091+0000","last_fullsized":"2024-02-21T23:06:09.938091+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:54:08.758447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.21","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.922993+0000","last_change":"2024-02-21T23:06:09.922993+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.922993+0000","last_undegraded":"2024-02-21T23:06:09.922993+0000","last_fullsized":"2024-02-21T23:06:09.922993+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.944278+0000","last_change":"2024-02-21T23:05:51.405741+0000","last_active":"2024-02-21T23:06:09.944278+0000","last_peered":"2024-02-21T23:06:09.944278+0000","last_clean":"2024-02-21T23:06:09.944278+0000","last_became_active":"2024-02-21T23:05:51.405301+0000","last_became_peered":"2024-02-21T23:05:51.405301+0000","last_unstale":"2024-02-21T23:06:09.944278+0000","last_undegraded":"2024-02-21T23:06:09.944278+0000","last_fullsized":"2024-02-21T23:06:09.944278+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:36:47.962935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.20","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.930360+0000","last_change":"2024-02-21T23:06:09.930360+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.930360+0000","last_undegraded":"2024-02-21T23:06:09.930360+0000","last_fullsized":"2024-02-21T23:06:09.930360+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.942207+0000","last_change":"2024-02-21T23:05:51.398663+0000","last_active":"2024-02-21T23:06:09.942207+0000","last_peered":"2024-02-21T23:06:09.942207+0000","last_clean":"2024-02-21T23:06:09.942207+0000","last_became_active":"2024-02-21T23:05:51.398146+0000","last_became_peered":"2024-02-21T23:05:51.398146+0000","last_unstale":"2024-02-21T23:06:09.942207+0000","last_undegraded":"2024-02-21T23:06:09.942207+0000","last_fullsized":"2024-02-21T23:06:09.942207+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:27:54.695721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.23","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.935616+0000","last_change":"2024-02-21T23:06:09.935616+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.935616+0000","last_undegraded":"2024-02-21T23:06:09.935616+0000","last_fullsized":"2024-02-21T23:06:09.935616+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.919121+0000","last_change":"2024-02-21T23:05:51.421330+0000","last_active":"2024-02-21T23:06:09.919121+0000","last_peered":"2024-02-21T23:06:09.919121+0000","last_clean":"2024-02-21T23:06:09.919121+0000","last_became_active":"2024-02-21T23:05:51.421012+0000","last_became_peered":"2024-02-21T23:05:51.421012+0000","last_unstale":"2024-02-21T23:06:09.919121+0000","last_undegraded":"2024-02-21T23:06:09.919121+0000","last_fullsized":"2024-02-21T23:06:09.919121+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:26:33.366331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.22","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.941990+0000","last_change":"2024-02-21T23:06:09.941990+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.941990+0000","last_undegraded":"2024-02-21T23:06:09.941990+0000","last_fullsized":"2024-02-21T23:06:09.941990+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.942077+0000","last_change":"2024-02-21T23:05:51.403173+0000","last_active":"2024-02-21T23:06:09.942077+0000","last_peered":"2024-02-21T23:06:09.942077+0000","last_clean":"2024-02-21T23:06:09.942077+0000","last_became_active":"2024-02-21T23:05:51.402610+0000","last_became_peered":"2024-02-21T23:05:51.402610+0000","last_unstale":"2024-02-21T23:06:09.942077+0000","last_undegraded":"2024-02-21T23:06:09.942077+0000","last_fullsized":"2024-02-21T23:06:09.942077+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:46:01.608149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.940077+0000","last_change":"2024-02-21T23:06:09.940077+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.940077+0000","last_undegraded":"2024-02-21T23:06:09.940077+0000","last_fullsized":"2024-02-21T23:06:09.940077+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.957716+0000","last_change":"2024-02-21T23:06:09.955402+0000","last_active":"2024-02-21T23:06:09.957716+0000","last_peered":"2024-02-21T23:06:09.957716+0000","last_clean":"2024-02-21T23:06:09.957716+0000","last_became_active":"2024-02-21T23:06:09.955113+0000","last_became_peered":"2024-02-21T23:06:09.955113+0000","last_unstale":"2024-02-21T23:06:09.957716+0000","last_undegraded":"2024-02-21T23:06:09.957716+0000","last_fullsized":"2024-02-21T23:06:09.957716+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:19:07.034670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.918368+0000","last_change":"2024-02-21T23:05:51.416534+0000","last_active":"2024-02-21T23:06:09.918368+0000","last_peered":"2024-02-21T23:06:09.918368+0000","last_clean":"2024-02-21T23:06:09.918368+0000","last_became_active":"2024-02-21T23:05:51.416244+0000","last_became_peered":"2024-02-21T23:05:51.416244+0000","last_unstale":"2024-02-21T23:06:09.918368+0000","last_undegraded":"2024-02-21T23:06:09.918368+0000","last_fullsized":"2024-02-21T23:06:09.918368+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:42:37.006685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"75'4","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.930907+0000","last_change":"2024-02-21T23:05:50.354109+0000","last_active":"2024-02-21T23:06:09.930907+0000","last_peered":"2024-02-21T23:06:09.930907+0000","last_clean":"2024-02-21T23:06:09.930907+0000","last_became_active":"2024-02-21T23:05:50.353809+0000","last_became_peered":"2024-02-21T23:05:50.353809+0000","last_unstale":"2024-02-21T23:06:09.930907+0000","last_undegraded":"2024-02-21T23:06:09.930907+0000","last_fullsized":"2024-02-21T23:06:09.930907+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:28:13.765557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.926981+0000","last_change":"2024-02-21T23:06:09.926981+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.926981+0000","last_undegraded":"2024-02-21T23:06:09.926981+0000","last_fullsized":"2024-02-21T23:06:09.926981+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.966118+0000","last_change":"2024-02-21T23:06:09.963140+0000","last_active":"2024-02-21T23:06:09.966118+0000","last_peered":"2024-02-21T23:06:09.966118+0000","last_clean":"2024-02-21T23:06:09.966118+0000","last_became_active":"2024-02-21T23:06:09.962637+0000","last_became_peered":"2024-02-21T23:06:09.962637+0000","last_unstale":"2024-02-21T23:06:09.966118+0000","last_undegraded":"2024-02-21T23:06:09.966118+0000","last_fullsized":"2024-02-21T23:06:09.966118+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:42:45.049032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.943704+0000","last_change":"2024-02-21T23:05:51.416268+0000","last_active":"2024-02-21T23:06:09.943704+0000","last_peered":"2024-02-21T23:06:09.943704+0000","last_clean":"2024-02-21T23:06:09.943704+0000","last_became_active":"2024-02-21T23:05:51.415980+0000","last_became_peered":"2024-02-21T23:05:51.415980+0000","last_unstale":"2024-02-21T23:06:09.943704+0000","last_undegraded":"2024-02-21T23:06:09.943704+0000","last_fullsized":"2024-02-21T23:06:09.943704+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:51:17.193881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"75'1","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.946611+0000","last_change":"2024-02-21T23:05:50.383438+0000","last_active":"2024-02-21T23:06:09.946611+0000","last_peered":"2024-02-21T23:06:09.946611+0000","last_clean":"2024-02-21T23:06:09.946611+0000","last_became_active":"2024-02-21T23:05:50.383134+0000","last_became_peered":"2024-02-21T23:05:50.383134+0000","last_unstale":"2024-02-21T23:06:09.946611+0000","last_undegraded":"2024-02-21T23:06:09.946611+0000","last_fullsized":"2024-02-21T23:06:09.946611+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:08:35.448428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.927781+0000","last_change":"2024-02-21T23:06:09.927781+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.927781+0000","last_undegraded":"2024-02-21T23:06:09.927781+0000","last_fullsized":"2024-02-21T23:06:09.927781+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.960638+0000","last_change":"2024-02-21T23:06:09.957178+0000","last_active":"2024-02-21T23:06:09.960638+0000","last_peered":"2024-02-21T23:06:09.960638+0000","last_clean":"2024-02-21T23:06:09.960638+0000","last_became_active":"2024-02-21T23:06:09.956818+0000","last_became_peered":"2024-02-21T23:06:09.956818+0000","last_unstale":"2024-02-21T23:06:09.960638+0000","last_undegraded":"2024-02-21T23:06:09.960638+0000","last_fullsized":"2024-02-21T23:06:09.960638+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:01:46.942433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.946131+0000","last_change":"2024-02-21T23:05:51.421571+0000","last_active":"2024-02-21T23:06:09.946131+0000","last_peered":"2024-02-21T23:06:09.946131+0000","last_clean":"2024-02-21T23:06:09.946131+0000","last_became_active":"2024-02-21T23:05:51.421110+0000","last_became_peered":"2024-02-21T23:05:51.421110+0000","last_unstale":"2024-02-21T23:06:09.946131+0000","last_undegraded":"2024-02-21T23:06:09.946131+0000","last_fullsized":"2024-02-21T23:06:09.946131+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:56:36.688152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"75'2","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.944712+0000","last_change":"2024-02-21T23:05:50.383231+0000","last_active":"2024-02-21T23:06:09.944712+0000","last_peered":"2024-02-21T23:06:09.944712+0000","last_clean":"2024-02-21T23:06:09.944712+0000","last_became_active":"2024-02-21T23:05:50.382757+0000","last_became_peered":"2024-02-21T23:05:50.382757+0000","last_unstale":"2024-02-21T23:06:09.944712+0000","last_undegraded":"2024-02-21T23:06:09.944712+0000","last_fullsized":"2024-02-21T23:06:09.944712+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:49:36.738197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.926671+0000","last_change":"2024-02-21T23:06:09.926671+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.926671+0000","last_undegraded":"2024-02-21T23:06:09.926671+0000","last_fullsized":"2024-02-21T23:06:09.926671+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.953270+0000","last_change":"2024-02-21T23:06:09.951255+0000","last_active":"2024-02-21T23:06:09.953270+0000","last_peered":"2024-02-21T23:06:09.953270+0000","last_clean":"2024-02-21T23:06:09.953270+0000","last_became_active":"2024-02-21T23:06:09.950934+0000","last_became_peered":"2024-02-21T23:06:09.950934+0000","last_unstale":"2024-02-21T23:06:09.953270+0000","last_undegraded":"2024-02-21T23:06:09.953270+0000","last_fullsized":"2024-02-21T23:06:09.953270+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:53:46.723561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.948793+0000","last_change":"2024-02-21T23:05:51.415892+0000","last_active":"2024-02-21T23:06:09.948793+0000","last_peered":"2024-02-21T23:06:09.948793+0000","last_clean":"2024-02-21T23:06:09.948793+0000","last_became_active":"2024-02-21T23:05:51.415520+0000","last_became_peered":"2024-02-21T23:05:51.415520+0000","last_unstale":"2024-02-21T23:06:09.948793+0000","last_undegraded":"2024-02-21T23:06:09.948793+0000","last_fullsized":"2024-02-21T23:06:09.948793+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:26:33.366331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"75'2","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.932382+0000","last_change":"2024-02-21T23:05:50.376050+0000","last_active":"2024-02-21T23:06:09.932382+0000","last_peered":"2024-02-21T23:06:09.932382+0000","last_clean":"2024-02-21T23:06:09.932382+0000","last_became_active":"2024-02-21T23:05:50.375731+0000","last_became_peered":"2024-02-21T23:05:50.375731+0000","last_unstale":"2024-02-21T23:06:09.932382+0000","last_undegraded":"2024-02-21T23:06:09.932382+0000","last_fullsized":"2024-02-21T23:06:09.932382+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:42:15.626763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.937587+0000","last_change":"2024-02-21T23:06:09.937587+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.937587+0000","last_undegraded":"2024-02-21T23:06:09.937587+0000","last_fullsized":"2024-02-21T23:06:09.937587+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.960875+0000","last_change":"2024-02-21T23:06:09.957048+0000","last_active":"2024-02-21T23:06:09.960875+0000","last_peered":"2024-02-21T23:06:09.960875+0000","last_clean":"2024-02-21T23:06:09.960875+0000","last_became_active":"2024-02-21T23:06:09.956666+0000","last_became_peered":"2024-02-21T23:06:09.956666+0000","last_unstale":"2024-02-21T23:06:09.960875+0000","last_undegraded":"2024-02-21T23:06:09.960875+0000","last_fullsized":"2024-02-21T23:06:09.960875+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:11:15.443744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.949179+0000","last_change":"2024-02-21T23:05:51.428435+0000","last_active":"2024-02-21T23:06:09.949179+0000","last_peered":"2024-02-21T23:06:09.949179+0000","last_clean":"2024-02-21T23:06:09.949179+0000","last_became_active":"2024-02-21T23:05:51.428133+0000","last_became_peered":"2024-02-21T23:05:51.428133+0000","last_unstale":"2024-02-21T23:06:09.949179+0000","last_undegraded":"2024-02-21T23:06:09.949179+0000","last_fullsized":"2024-02-21T23:06:09.949179+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:49:46.025250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"75'1","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.948878+0000","last_change":"2024-02-21T23:05:50.377574+0000","last_active":"2024-02-21T23:06:09.948878+0000","last_peered":"2024-02-21T23:06:09.948878+0000","last_clean":"2024-02-21T23:06:09.948878+0000","last_became_active":"2024-02-21T23:05:50.377250+0000","last_became_peered":"2024-02-21T23:05:50.377250+0000","last_unstale":"2024-02-21T23:06:09.948878+0000","last_undegraded":"2024-02-21T23:06:09.948878+0000","last_fullsized":"2024-02-21T23:06:09.948878+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:26:38.366160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.937636+0000","last_change":"2024-02-21T23:06:09.937636+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.937636+0000","last_undegraded":"2024-02-21T23:06:09.937636+0000","last_fullsized":"2024-02-21T23:06:09.937636+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.954390+0000","last_change":"2024-02-21T23:06:09.947108+0000","last_active":"2024-02-21T23:06:09.954390+0000","last_peered":"2024-02-21T23:06:09.954390+0000","last_clean":"2024-02-21T23:06:09.954390+0000","last_became_active":"2024-02-21T23:06:09.946815+0000","last_became_peered":"2024-02-21T23:06:09.946815+0000","last_unstale":"2024-02-21T23:06:09.954390+0000","last_undegraded":"2024-02-21T23:06:09.954390+0000","last_fullsized":"2024-02-21T23:06:09.954390+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:56:08.310911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.951879+0000","last_change":"2024-02-21T23:05:51.421508+0000","last_active":"2024-02-21T23:06:09.951879+0000","last_peered":"2024-02-21T23:06:09.951879+0000","last_clean":"2024-02-21T23:06:09.951879+0000","last_became_active":"2024-02-21T23:05:51.420041+0000","last_became_peered":"2024-02-21T23:05:51.420041+0000","last_unstale":"2024-02-21T23:06:09.951879+0000","last_undegraded":"2024-02-21T23:06:09.951879+0000","last_fullsized":"2024-02-21T23:06:09.951879+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:10:56.883222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"75'1","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.933207+0000","last_change":"2024-02-21T23:05:50.378668+0000","last_active":"2024-02-21T23:06:09.933207+0000","last_peered":"2024-02-21T23:06:09.933207+0000","last_clean":"2024-02-21T23:06:09.933207+0000","last_became_active":"2024-02-21T23:05:50.378369+0000","last_became_peered":"2024-02-21T23:05:50.378369+0000","last_unstale":"2024-02-21T23:06:09.933207+0000","last_undegraded":"2024-02-21T23:06:09.933207+0000","last_fullsized":"2024-02-21T23:06:09.933207+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:45:05.837402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.943022+0000","last_change":"2024-02-21T23:06:09.943022+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.943022+0000","last_undegraded":"2024-02-21T23:06:09.943022+0000","last_fullsized":"2024-02-21T23:06:09.943022+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.955897+0000","last_change":"2024-02-21T23:06:09.949620+0000","last_active":"2024-02-21T23:06:09.955897+0000","last_peered":"2024-02-21T23:06:09.955897+0000","last_clean":"2024-02-21T23:06:09.955897+0000","last_became_active":"2024-02-21T23:06:09.949252+0000","last_became_peered":"2024-02-21T23:06:09.949252+0000","last_unstale":"2024-02-21T23:06:09.955897+0000","last_undegraded":"2024-02-21T23:06:09.955897+0000","last_fullsized":"2024-02-21T23:06:09.955897+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:51:46.950044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.951942+0000","last_change":"2024-02-21T23:05:51.419084+0000","last_active":"2024-02-21T23:06:09.951942+0000","last_peered":"2024-02-21T23:06:09.951942+0000","last_clean":"2024-02-21T23:06:09.951942+0000","last_became_active":"2024-02-21T23:05:51.418752+0000","last_became_peered":"2024-02-21T23:05:51.418752+0000","last_unstale":"2024-02-21T23:06:09.951942+0000","last_undegraded":"2024-02-21T23:06:09.951942+0000","last_fullsized":"2024-02-21T23:06:09.951942+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:20:19.564331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.943810+0000","last_change":"2024-02-21T23:05:50.375534+0000","last_active":"2024-02-21T23:06:09.943810+0000","last_peered":"2024-02-21T23:06:09.943810+0000","last_clean":"2024-02-21T23:06:09.943810+0000","last_became_active":"2024-02-21T23:05:50.373762+0000","last_became_peered":"2024-02-21T23:05:50.373762+0000","last_unstale":"2024-02-21T23:06:09.943810+0000","last_undegraded":"2024-02-21T23:06:09.943810+0000","last_fullsized":"2024-02-21T23:06:09.943810+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:49:36.738197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.941977+0000","last_change":"2024-02-21T23:06:09.941977+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.941977+0000","last_undegraded":"2024-02-21T23:06:09.941977+0000","last_fullsized":"2024-02-21T23:06:09.941977+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.957774+0000","last_change":"2024-02-21T23:06:09.952374+0000","last_active":"2024-02-21T23:06:09.957774+0000","last_peered":"2024-02-21T23:06:09.957774+0000","last_clean":"2024-02-21T23:06:09.957774+0000","last_became_active":"2024-02-21T23:06:09.950283+0000","last_became_peered":"2024-02-21T23:06:09.950283+0000","last_unstale":"2024-02-21T23:06:09.957774+0000","last_undegraded":"2024-02-21T23:06:09.957774+0000","last_fullsized":"2024-02-21T23:06:09.957774+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:27:17.649346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.935109+0000","last_change":"2024-02-21T23:05:51.426711+0000","last_active":"2024-02-21T23:06:09.935109+0000","last_peered":"2024-02-21T23:06:09.935109+0000","last_clean":"2024-02-21T23:06:09.935109+0000","last_became_active":"2024-02-21T23:05:51.426415+0000","last_became_peered":"2024-02-21T23:05:51.426415+0000","last_unstale":"2024-02-21T23:06:09.935109+0000","last_undegraded":"2024-02-21T23:06:09.935109+0000","last_fullsized":"2024-02-21T23:06:09.935109+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:58:55.829941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.931209+0000","last_change":"2024-02-21T23:05:50.355567+0000","last_active":"2024-02-21T23:06:09.931209+0000","last_peered":"2024-02-21T23:06:09.931209+0000","last_clean":"2024-02-21T23:06:09.931209+0000","last_became_active":"2024-02-21T23:05:50.354835+0000","last_became_peered":"2024-02-21T23:05:50.354835+0000","last_unstale":"2024-02-21T23:06:09.931209+0000","last_undegraded":"2024-02-21T23:06:09.931209+0000","last_fullsized":"2024-02-21T23:06:09.931209+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:21:52.655098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.936369+0000","last_change":"2024-02-21T23:06:09.936369+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.936369+0000","last_undegraded":"2024-02-21T23:06:09.936369+0000","last_fullsized":"2024-02-21T23:06:09.936369+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.950750+0000","last_change":"2024-02-21T23:06:09.946843+0000","last_active":"2024-02-21T23:06:09.950750+0000","last_peered":"2024-02-21T23:06:09.950750+0000","last_clean":"2024-02-21T23:06:09.950750+0000","last_became_active":"2024-02-21T23:06:09.946452+0000","last_became_peered":"2024-02-21T23:06:09.946452+0000","last_unstale":"2024-02-21T23:06:09.950750+0000","last_undegraded":"2024-02-21T23:06:09.950750+0000","last_fullsized":"2024-02-21T23:06:09.950750+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:22:35.988029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.944383+0000","last_change":"2024-02-21T23:05:51.399923+0000","last_active":"2024-02-21T23:06:09.944383+0000","last_peered":"2024-02-21T23:06:09.944383+0000","last_clean":"2024-02-21T23:06:09.944383+0000","last_became_active":"2024-02-21T23:05:51.399642+0000","last_became_peered":"2024-02-21T23:05:51.399642+0000","last_unstale":"2024-02-21T23:06:09.944383+0000","last_undegraded":"2024-02-21T23:06:09.944383+0000","last_fullsized":"2024-02-21T23:06:09.944383+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:21:18.440977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.946594+0000","last_change":"2024-02-21T23:05:50.386639+0000","last_active":"2024-02-21T23:06:09.946594+0000","last_peered":"2024-02-21T23:06:09.946594+0000","last_clean":"2024-02-21T23:06:09.946594+0000","last_became_active":"2024-02-21T23:05:50.386158+0000","last_became_peered":"2024-02-21T23:05:50.386158+0000","last_unstale":"2024-02-21T23:06:09.946594+0000","last_undegraded":"2024-02-21T23:06:09.946594+0000","last_fullsized":"2024-02-21T23:06:09.946594+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:43:17.496894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.932767+0000","last_change":"2024-02-21T23:06:09.932767+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.932767+0000","last_undegraded":"2024-02-21T23:06:09.932767+0000","last_fullsized":"2024-02-21T23:06:09.932767+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.943824+0000","last_change":"2024-02-21T23:06:09.934904+0000","last_active":"2024-02-21T23:06:09.943824+0000","last_peered":"2024-02-21T23:06:09.943824+0000","last_clean":"2024-02-21T23:06:09.943824+0000","last_became_active":"2024-02-21T23:06:09.934606+0000","last_became_peered":"2024-02-21T23:06:09.934606+0000","last_unstale":"2024-02-21T23:06:09.943824+0000","last_undegraded":"2024-02-21T23:06:09.943824+0000","last_fullsized":"2024-02-21T23:06:09.943824+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:43:37.070238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.932966+0000","last_change":"2024-02-21T23:05:51.422844+0000","last_active":"2024-02-21T23:06:09.932966+0000","last_peered":"2024-02-21T23:06:09.932966+0000","last_clean":"2024-02-21T23:06:09.932966+0000","last_became_active":"2024-02-21T23:05:51.422488+0000","last_became_peered":"2024-02-21T23:05:51.422488+0000","last_unstale":"2024-02-21T23:06:09.932966+0000","last_undegraded":"2024-02-21T23:06:09.932966+0000","last_fullsized":"2024-02-21T23:06:09.932966+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:27:54.695721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.934464+0000","last_change":"2024-02-21T23:05:50.385146+0000","last_active":"2024-02-21T23:06:09.934464+0000","last_peered":"2024-02-21T23:06:09.934464+0000","last_clean":"2024-02-21T23:06:09.934464+0000","last_became_active":"2024-02-21T23:05:50.384418+0000","last_became_peered":"2024-02-21T23:05:50.384418+0000","last_unstale":"2024-02-21T23:06:09.934464+0000","last_undegraded":"2024-02-21T23:06:09.934464+0000","last_fullsized":"2024-02-21T23:06:09.934464+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:43:17.496894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.927206+0000","last_change":"2024-02-21T23:06:09.927206+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.927206+0000","last_undegraded":"2024-02-21T23:06:09.927206+0000","last_fullsized":"2024-02-21T23:06:09.927206+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.966125+0000","last_change":"2024-02-21T23:06:09.963033+0000","last_active":"2024-02-21T23:06:09.966125+0000","last_peered":"2024-02-21T23:06:09.966125+0000","last_clean":"2024-02-21T23:06:09.966125+0000","last_became_active":"2024-02-21T23:06:09.962695+0000","last_became_peered":"2024-02-21T23:06:09.962695+0000","last_unstale":"2024-02-21T23:06:09.966125+0000","last_undegraded":"2024-02-21T23:06:09.966125+0000","last_fullsized":"2024-02-21T23:06:09.966125+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:36:42.352917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.946054+0000","last_change":"2024-02-21T23:05:51.418004+0000","last_active":"2024-02-21T23:06:09.946054+0000","last_peered":"2024-02-21T23:06:09.946054+0000","last_clean":"2024-02-21T23:06:09.946054+0000","last_became_active":"2024-02-21T23:05:51.417463+0000","last_became_peered":"2024-02-21T23:05:51.417463+0000","last_unstale":"2024-02-21T23:06:09.946054+0000","last_undegraded":"2024-02-21T23:06:09.946054+0000","last_fullsized":"2024-02-21T23:06:09.946054+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:59:13.292494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"75'3","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.932692+0000","last_change":"2024-02-21T23:05:50.373189+0000","last_active":"2024-02-21T23:06:09.932692+0000","last_peered":"2024-02-21T23:06:09.932692+0000","last_clean":"2024-02-21T23:06:09.932692+0000","last_became_active":"2024-02-21T23:05:50.370680+0000","last_became_peered":"2024-02-21T23:05:50.370680+0000","last_unstale":"2024-02-21T23:06:09.932692+0000","last_undegraded":"2024-02-21T23:06:09.932692+0000","last_fullsized":"2024-02-21T23:06:09.932692+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:21:03.121546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.939836+0000","last_change":"2024-02-21T23:06:09.939836+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.939836+0000","last_undegraded":"2024-02-21T23:06:09.939836+0000","last_fullsized":"2024-02-21T23:06:09.939836+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.963277+0000","last_change":"2024-02-21T23:06:09.961844+0000","last_active":"2024-02-21T23:06:09.963277+0000","last_peered":"2024-02-21T23:06:09.963277+0000","last_clean":"2024-02-21T23:06:09.963277+0000","last_became_active":"2024-02-21T23:06:09.961480+0000","last_became_peered":"2024-02-21T23:06:09.961480+0000","last_unstale":"2024-02-21T23:06:09.963277+0000","last_undegraded":"2024-02-21T23:06:09.963277+0000","last_fullsized":"2024-02-21T23:06:09.963277+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:04:07.940431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.919599+0000","last_change":"2024-02-21T23:05:51.405242+0000","last_active":"2024-02-21T23:06:09.919599+0000","last_peered":"2024-02-21T23:06:09.919599+0000","last_clean":"2024-02-21T23:06:09.919599+0000","last_became_active":"2024-02-21T23:05:51.403777+0000","last_became_peered":"2024-02-21T23:05:51.403777+0000","last_unstale":"2024-02-21T23:06:09.919599+0000","last_undegraded":"2024-02-21T23:06:09.919599+0000","last_fullsized":"2024-02-21T23:06:09.919599+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:34:14.420091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"75'6","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.932534+0000","last_change":"2024-02-21T23:05:50.374208+0000","last_active":"2024-02-21T23:06:09.932534+0000","last_peered":"2024-02-21T23:06:09.932534+0000","last_clean":"2024-02-21T23:06:09.932534+0000","last_became_active":"2024-02-21T23:05:50.373869+0000","last_became_peered":"2024-02-21T23:05:50.373869+0000","last_unstale":"2024-02-21T23:06:09.932534+0000","last_undegraded":"2024-02-21T23:06:09.932534+0000","last_fullsized":"2024-02-21T23:06:09.932534+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:56:49.435540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.938351+0000","last_change":"2024-02-21T23:06:09.938351+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.938351+0000","last_undegraded":"2024-02-21T23:06:09.938351+0000","last_fullsized":"2024-02-21T23:06:09.938351+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.966390+0000","last_change":"2024-02-21T23:06:09.963215+0000","last_active":"2024-02-21T23:06:09.966390+0000","last_peered":"2024-02-21T23:06:09.966390+0000","last_clean":"2024-02-21T23:06:09.966390+0000","last_became_active":"2024-02-21T23:06:09.962779+0000","last_became_peered":"2024-02-21T23:06:09.962779+0000","last_unstale":"2024-02-21T23:06:09.966390+0000","last_undegraded":"2024-02-21T23:06:09.966390+0000","last_fullsized":"2024-02-21T23:06:09.966390+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:45:25.410745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.935046+0000","last_change":"2024-02-21T23:05:51.423479+0000","last_active":"2024-02-21T23:06:09.935046+0000","last_peered":"2024-02-21T23:06:09.935046+0000","last_clean":"2024-02-21T23:06:09.935046+0000","last_became_active":"2024-02-21T23:05:51.423190+0000","last_became_peered":"2024-02-21T23:05:51.423190+0000","last_unstale":"2024-02-21T23:06:09.935046+0000","last_undegraded":"2024-02-21T23:06:09.935046+0000","last_fullsized":"2024-02-21T23:06:09.935046+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:03:08.789066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"75'2","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.933594+0000","last_change":"2024-02-21T23:05:50.381643+0000","last_active":"2024-02-21T23:06:09.933594+0000","last_peered":"2024-02-21T23:06:09.933594+0000","last_clean":"2024-02-21T23:06:09.933594+0000","last_became_active":"2024-02-21T23:05:50.381345+0000","last_became_peered":"2024-02-21T23:05:50.381345+0000","last_unstale":"2024-02-21T23:06:09.933594+0000","last_undegraded":"2024-02-21T23:06:09.933594+0000","last_fullsized":"2024-02-21T23:06:09.933594+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:01:27.369089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.927030+0000","last_change":"2024-02-21T23:06:09.927030+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.927030+0000","last_undegraded":"2024-02-21T23:06:09.927030+0000","last_fullsized":"2024-02-21T23:06:09.927030+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.962022+0000","last_change":"2024-02-21T23:06:09.959967+0000","last_active":"2024-02-21T23:06:09.962022+0000","last_peered":"2024-02-21T23:06:09.962022+0000","last_clean":"2024-02-21T23:06:09.962022+0000","last_became_active":"2024-02-21T23:06:09.959175+0000","last_became_peered":"2024-02-21T23:06:09.959175+0000","last_unstale":"2024-02-21T23:06:09.962022+0000","last_undegraded":"2024-02-21T23:06:09.962022+0000","last_fullsized":"2024-02-21T23:06:09.962022+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:41:49.821364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.937192+0000","last_change":"2024-02-21T23:05:51.416753+0000","last_active":"2024-02-21T23:06:09.937192+0000","last_peered":"2024-02-21T23:06:09.937192+0000","last_clean":"2024-02-21T23:06:09.937192+0000","last_became_active":"2024-02-21T23:05:51.416427+0000","last_became_peered":"2024-02-21T23:05:51.416427+0000","last_unstale":"2024-02-21T23:06:09.937192+0000","last_undegraded":"2024-02-21T23:06:09.937192+0000","last_fullsized":"2024-02-21T23:06:09.937192+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:34:14.420091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.942362+0000","last_change":"2024-02-21T23:05:50.381962+0000","last_active":"2024-02-21T23:06:09.942362+0000","last_peered":"2024-02-21T23:06:09.942362+0000","last_clean":"2024-02-21T23:06:09.942362+0000","last_became_active":"2024-02-21T23:05:50.380838+0000","last_became_peered":"2024-02-21T23:05:50.380838+0000","last_unstale":"2024-02-21T23:06:09.942362+0000","last_undegraded":"2024-02-21T23:06:09.942362+0000","last_fullsized":"2024-02-21T23:06:09.942362+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:10:10.973567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.939210+0000","last_change":"2024-02-21T23:06:09.939210+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.939210+0000","last_undegraded":"2024-02-21T23:06:09.939210+0000","last_fullsized":"2024-02-21T23:06:09.939210+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.958221+0000","last_change":"2024-02-21T23:06:09.951230+0000","last_active":"2024-02-21T23:06:09.958221+0000","last_peered":"2024-02-21T23:06:09.958221+0000","last_clean":"2024-02-21T23:06:09.958221+0000","last_became_active":"2024-02-21T23:06:09.950920+0000","last_became_peered":"2024-02-21T23:06:09.950920+0000","last_unstale":"2024-02-21T23:06:09.958221+0000","last_undegraded":"2024-02-21T23:06:09.958221+0000","last_fullsized":"2024-02-21T23:06:09.958221+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:11:15.443744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.934193+0000","last_change":"2024-02-21T23:05:51.408507+0000","last_active":"2024-02-21T23:06:09.934193+0000","last_peered":"2024-02-21T23:06:09.934193+0000","last_clean":"2024-02-21T23:06:09.934193+0000","last_became_active":"2024-02-21T23:05:51.408198+0000","last_became_peered":"2024-02-21T23:05:51.408198+0000","last_unstale":"2024-02-21T23:06:09.934193+0000","last_undegraded":"2024-02-21T23:06:09.934193+0000","last_fullsized":"2024-02-21T23:06:09.934193+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:08:36.461250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.934108+0000","last_change":"2024-02-21T23:05:50.353116+0000","last_active":"2024-02-21T23:06:09.934108+0000","last_peered":"2024-02-21T23:06:09.934108+0000","last_clean":"2024-02-21T23:06:09.934108+0000","last_became_active":"2024-02-21T23:05:50.352768+0000","last_became_peered":"2024-02-21T23:05:50.352768+0000","last_unstale":"2024-02-21T23:06:09.934108+0000","last_undegraded":"2024-02-21T23:06:09.934108+0000","last_fullsized":"2024-02-21T23:06:09.934108+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:52:09.073053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.937280+0000","last_change":"2024-02-21T23:06:09.937280+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.937280+0000","last_undegraded":"2024-02-21T23:06:09.937280+0000","last_fullsized":"2024-02-21T23:06:09.937280+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.955574+0000","last_change":"2024-02-21T23:06:09.954937+0000","last_active":"2024-02-21T23:06:09.955574+0000","last_peered":"2024-02-21T23:06:09.955574+0000","last_clean":"2024-02-21T23:06:09.955574+0000","last_became_active":"2024-02-21T23:06:09.953847+0000","last_became_peered":"2024-02-21T23:06:09.953847+0000","last_unstale":"2024-02-21T23:06:09.955574+0000","last_undegraded":"2024-02-21T23:06:09.955574+0000","last_fullsized":"2024-02-21T23:06:09.955574+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:17:14.119658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.936892+0000","last_change":"2024-02-21T23:05:51.416424+0000","last_active":"2024-02-21T23:06:09.936892+0000","last_peered":"2024-02-21T23:06:09.936892+0000","last_clean":"2024-02-21T23:06:09.936892+0000","last_became_active":"2024-02-21T23:05:51.416070+0000","last_became_peered":"2024-02-21T23:05:51.416070+0000","last_unstale":"2024-02-21T23:06:09.936892+0000","last_undegraded":"2024-02-21T23:06:09.936892+0000","last_fullsized":"2024-02-21T23:06:09.936892+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:56:10.406772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"75'8","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.947720+0000","last_change":"2024-02-21T23:05:50.376063+0000","last_active":"2024-02-21T23:06:09.947720+0000","last_peered":"2024-02-21T23:06:09.947720+0000","last_clean":"2024-02-21T23:06:09.947720+0000","last_became_active":"2024-02-21T23:05:50.375772+0000","last_became_peered":"2024-02-21T23:05:50.375772+0000","last_unstale":"2024-02-21T23:06:09.947720+0000","last_undegraded":"2024-02-21T23:06:09.947720+0000","last_fullsized":"2024-02-21T23:06:09.947720+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:26:58.076002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.923016+0000","last_change":"2024-02-21T23:06:09.923016+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.923016+0000","last_undegraded":"2024-02-21T23:06:09.923016+0000","last_fullsized":"2024-02-21T23:06:09.923016+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.958286+0000","last_change":"2024-02-21T23:06:09.951762+0000","last_active":"2024-02-21T23:06:09.958286+0000","last_peered":"2024-02-21T23:06:09.958286+0000","last_clean":"2024-02-21T23:06:09.958286+0000","last_became_active":"2024-02-21T23:06:09.951155+0000","last_became_peered":"2024-02-21T23:06:09.951155+0000","last_unstale":"2024-02-21T23:06:09.958286+0000","last_undegraded":"2024-02-21T23:06:09.958286+0000","last_fullsized":"2024-02-21T23:06:09.958286+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:36:42.352917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.942415+0000","last_change":"2024-02-21T23:05:51.421565+0000","last_active":"2024-02-21T23:06:09.942415+0000","last_peered":"2024-02-21T23:06:09.942415+0000","last_clean":"2024-02-21T23:06:09.942415+0000","last_became_active":"2024-02-21T23:05:51.421252+0000","last_became_peered":"2024-02-21T23:05:51.421252+0000","last_unstale":"2024-02-21T23:06:09.942415+0000","last_undegraded":"2024-02-21T23:06:09.942415+0000","last_fullsized":"2024-02-21T23:06:09.942415+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:40:45.445418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"75'1","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.946704+0000","last_change":"2024-02-21T23:05:50.383778+0000","last_active":"2024-02-21T23:06:09.946704+0000","last_peered":"2024-02-21T23:06:09.946704+0000","last_clean":"2024-02-21T23:06:09.946704+0000","last_became_active":"2024-02-21T23:05:50.383436+0000","last_became_peered":"2024-02-21T23:05:50.383436+0000","last_unstale":"2024-02-21T23:06:09.946704+0000","last_undegraded":"2024-02-21T23:06:09.946704+0000","last_fullsized":"2024-02-21T23:06:09.946704+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:45:05.837402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.934671+0000","last_change":"2024-02-21T23:06:09.934671+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.934671+0000","last_undegraded":"2024-02-21T23:06:09.934671+0000","last_fullsized":"2024-02-21T23:06:09.934671+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.956162+0000","last_change":"2024-02-21T23:06:09.947681+0000","last_active":"2024-02-21T23:06:09.956162+0000","last_peered":"2024-02-21T23:06:09.956162+0000","last_clean":"2024-02-21T23:06:09.956162+0000","last_became_active":"2024-02-21T23:06:09.947370+0000","last_became_peered":"2024-02-21T23:06:09.947370+0000","last_unstale":"2024-02-21T23:06:09.956162+0000","last_undegraded":"2024-02-21T23:06:09.956162+0000","last_fullsized":"2024-02-21T23:06:09.956162+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:01:37.993072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.934009+0000","last_change":"2024-02-21T23:05:51.416676+0000","last_active":"2024-02-21T23:06:09.934009+0000","last_peered":"2024-02-21T23:06:09.934009+0000","last_clean":"2024-02-21T23:06:09.934009+0000","last_became_active":"2024-02-21T23:05:51.416297+0000","last_became_peered":"2024-02-21T23:05:51.416297+0000","last_unstale":"2024-02-21T23:06:09.934009+0000","last_undegraded":"2024-02-21T23:06:09.934009+0000","last_fullsized":"2024-02-21T23:06:09.934009+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:29:33.209545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"75'2","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.952053+0000","last_change":"2024-02-21T23:05:50.377181+0000","last_active":"2024-02-21T23:06:09.952053+0000","last_peered":"2024-02-21T23:06:09.952053+0000","last_clean":"2024-02-21T23:06:09.952053+0000","last_became_active":"2024-02-21T23:05:50.373524+0000","last_became_peered":"2024-02-21T23:05:50.373524+0000","last_unstale":"2024-02-21T23:06:09.952053+0000","last_undegraded":"2024-02-21T23:06:09.952053+0000","last_fullsized":"2024-02-21T23:06:09.952053+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:01:18.419728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.929958+0000","last_change":"2024-02-21T23:06:09.929958+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.929958+0000","last_undegraded":"2024-02-21T23:06:09.929958+0000","last_fullsized":"2024-02-21T23:06:09.929958+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.961219+0000","last_change":"2024-02-21T23:06:09.955696+0000","last_active":"2024-02-21T23:06:09.961219+0000","last_peered":"2024-02-21T23:06:09.961219+0000","last_clean":"2024-02-21T23:06:09.961219+0000","last_became_active":"2024-02-21T23:06:09.954715+0000","last_became_peered":"2024-02-21T23:06:09.954715+0000","last_unstale":"2024-02-21T23:06:09.961219+0000","last_undegraded":"2024-02-21T23:06:09.961219+0000","last_fullsized":"2024-02-21T23:06:09.961219+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:01:35.517150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.937478+0000","last_change":"2024-02-21T23:05:51.412971+0000","last_active":"2024-02-21T23:06:09.937478+0000","last_peered":"2024-02-21T23:06:09.937478+0000","last_clean":"2024-02-21T23:06:09.937478+0000","last_became_active":"2024-02-21T23:05:51.410966+0000","last_became_peered":"2024-02-21T23:05:51.410966+0000","last_unstale":"2024-02-21T23:06:09.937478+0000","last_undegraded":"2024-02-21T23:06:09.937478+0000","last_fullsized":"2024-02-21T23:06:09.937478+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:00:39.163092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"75'1","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.938268+0000","last_change":"2024-02-21T23:05:50.366307+0000","last_active":"2024-02-21T23:06:09.938268+0000","last_peered":"2024-02-21T23:06:09.938268+0000","last_clean":"2024-02-21T23:06:09.938268+0000","last_became_active":"2024-02-21T23:05:50.359426+0000","last_became_peered":"2024-02-21T23:05:50.359426+0000","last_unstale":"2024-02-21T23:06:09.938268+0000","last_undegraded":"2024-02-21T23:06:09.938268+0000","last_fullsized":"2024-02-21T23:06:09.938268+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:13:15.770526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.925807+0000","last_change":"2024-02-21T23:06:09.925807+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.925807+0000","last_undegraded":"2024-02-21T23:06:09.925807+0000","last_fullsized":"2024-02-21T23:06:09.925807+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.956240+0000","last_change":"2024-02-21T23:06:09.948670+0000","last_active":"2024-02-21T23:06:09.956240+0000","last_peered":"2024-02-21T23:06:09.956240+0000","last_clean":"2024-02-21T23:06:09.956240+0000","last_became_active":"2024-02-21T23:06:09.946109+0000","last_became_peered":"2024-02-21T23:06:09.946109+0000","last_unstale":"2024-02-21T23:06:09.956240+0000","last_undegraded":"2024-02-21T23:06:09.956240+0000","last_fullsized":"2024-02-21T23:06:09.956240+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:26:57.939504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.947158+0000","last_change":"2024-02-21T23:05:51.424067+0000","last_active":"2024-02-21T23:06:09.947158+0000","last_peered":"2024-02-21T23:06:09.947158+0000","last_clean":"2024-02-21T23:06:09.947158+0000","last_became_active":"2024-02-21T23:05:51.423644+0000","last_became_peered":"2024-02-21T23:05:51.423644+0000","last_unstale":"2024-02-21T23:06:09.947158+0000","last_undegraded":"2024-02-21T23:06:09.947158+0000","last_fullsized":"2024-02-21T23:06:09.947158+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:04:03.084512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.919750+0000","last_change":"2024-02-21T23:05:50.377924+0000","last_active":"2024-02-21T23:06:09.919750+0000","last_peered":"2024-02-21T23:06:09.919750+0000","last_clean":"2024-02-21T23:06:09.919750+0000","last_became_active":"2024-02-21T23:05:50.377440+0000","last_became_peered":"2024-02-21T23:05:50.377440+0000","last_unstale":"2024-02-21T23:06:09.919750+0000","last_undegraded":"2024-02-21T23:06:09.919750+0000","last_fullsized":"2024-02-21T23:06:09.919750+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:42:15.626763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.938677+0000","last_change":"2024-02-21T23:06:09.938677+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.938677+0000","last_undegraded":"2024-02-21T23:06:09.938677+0000","last_fullsized":"2024-02-21T23:06:09.938677+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.957820+0000","last_change":"2024-02-21T23:06:09.953285+0000","last_active":"2024-02-21T23:06:09.957820+0000","last_peered":"2024-02-21T23:06:09.957820+0000","last_clean":"2024-02-21T23:06:09.957820+0000","last_became_active":"2024-02-21T23:06:09.952940+0000","last_became_peered":"2024-02-21T23:06:09.952940+0000","last_unstale":"2024-02-21T23:06:09.957820+0000","last_undegraded":"2024-02-21T23:06:09.957820+0000","last_fullsized":"2024-02-21T23:06:09.957820+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:49:56.311541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.949283+0000","last_change":"2024-02-21T23:05:51.414619+0000","last_active":"2024-02-21T23:06:09.949283+0000","last_peered":"2024-02-21T23:06:09.949283+0000","last_clean":"2024-02-21T23:06:09.949283+0000","last_became_active":"2024-02-21T23:05:51.414059+0000","last_became_peered":"2024-02-21T23:05:51.414059+0000","last_unstale":"2024-02-21T23:06:09.949283+0000","last_undegraded":"2024-02-21T23:06:09.949283+0000","last_fullsized":"2024-02-21T23:06:09.949283+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:58:05.114297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"75'2","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.947250+0000","last_change":"2024-02-21T23:05:50.366877+0000","last_active":"2024-02-21T23:06:09.947250+0000","last_peered":"2024-02-21T23:06:09.947250+0000","last_clean":"2024-02-21T23:06:09.947250+0000","last_became_active":"2024-02-21T23:05:50.365566+0000","last_became_peered":"2024-02-21T23:05:50.365566+0000","last_unstale":"2024-02-21T23:06:09.947250+0000","last_undegraded":"2024-02-21T23:06:09.947250+0000","last_fullsized":"2024-02-21T23:06:09.947250+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:58:04.101475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.941480+0000","last_change":"2024-02-21T23:06:09.941480+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.941480+0000","last_undegraded":"2024-02-21T23:06:09.941480+0000","last_fullsized":"2024-02-21T23:06:09.941480+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.954312+0000","last_change":"2024-02-21T23:06:09.945701+0000","last_active":"2024-02-21T23:06:09.954312+0000","last_peered":"2024-02-21T23:06:09.954312+0000","last_clean":"2024-02-21T23:06:09.954312+0000","last_became_active":"2024-02-21T23:06:09.945391+0000","last_became_peered":"2024-02-21T23:06:09.945391+0000","last_unstale":"2024-02-21T23:06:09.954312+0000","last_undegraded":"2024-02-21T23:06:09.954312+0000","last_fullsized":"2024-02-21T23:06:09.954312+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:53:46.723561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.944153+0000","last_change":"2024-02-21T23:05:51.411005+0000","last_active":"2024-02-21T23:06:09.944153+0000","last_peered":"2024-02-21T23:06:09.944153+0000","last_clean":"2024-02-21T23:06:09.944153+0000","last_became_active":"2024-02-21T23:05:51.410503+0000","last_became_peered":"2024-02-21T23:05:51.410503+0000","last_unstale":"2024-02-21T23:06:09.944153+0000","last_undegraded":"2024-02-21T23:06:09.944153+0000","last_fullsized":"2024-02-21T23:06:09.944153+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:51:10.610852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.944912+0000","last_change":"2024-02-21T23:05:50.376888+0000","last_active":"2024-02-21T23:06:09.944912+0000","last_peered":"2024-02-21T23:06:09.944912+0000","last_clean":"2024-02-21T23:06:09.944912+0000","last_became_active":"2024-02-21T23:05:50.376610+0000","last_became_peered":"2024-02-21T23:05:50.376610+0000","last_unstale":"2024-02-21T23:06:09.944912+0000","last_undegraded":"2024-02-21T23:06:09.944912+0000","last_fullsized":"2024-02-21T23:06:09.944912+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:51:16.181060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.925072+0000","last_change":"2024-02-21T23:06:09.925072+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.925072+0000","last_undegraded":"2024-02-21T23:06:09.925072+0000","last_fullsized":"2024-02-21T23:06:09.925072+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.958272+0000","last_change":"2024-02-21T23:06:09.952555+0000","last_active":"2024-02-21T23:06:09.958272+0000","last_peered":"2024-02-21T23:06:09.958272+0000","last_clean":"2024-02-21T23:06:09.958272+0000","last_became_active":"2024-02-21T23:06:09.952281+0000","last_became_peered":"2024-02-21T23:06:09.952281+0000","last_unstale":"2024-02-21T23:06:09.958272+0000","last_undegraded":"2024-02-21T23:06:09.958272+0000","last_fullsized":"2024-02-21T23:06:09.958272+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:50:04.585772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.937015+0000","last_change":"2024-02-21T23:05:51.417167+0000","last_active":"2024-02-21T23:06:09.937015+0000","last_peered":"2024-02-21T23:06:09.937015+0000","last_clean":"2024-02-21T23:06:09.937015+0000","last_became_active":"2024-02-21T23:05:51.415957+0000","last_became_peered":"2024-02-21T23:05:51.415957+0000","last_unstale":"2024-02-21T23:06:09.937015+0000","last_undegraded":"2024-02-21T23:06:09.937015+0000","last_fullsized":"2024-02-21T23:06:09.937015+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:26:59.088824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.942505+0000","last_change":"2024-02-21T23:05:50.375808+0000","last_active":"2024-02-21T23:06:09.942505+0000","last_peered":"2024-02-21T23:06:09.942505+0000","last_clean":"2024-02-21T23:06:09.942505+0000","last_became_active":"2024-02-21T23:05:50.375467+0000","last_became_peered":"2024-02-21T23:05:50.375467+0000","last_unstale":"2024-02-21T23:06:09.942505+0000","last_undegraded":"2024-02-21T23:06:09.942505+0000","last_fullsized":"2024-02-21T23:06:09.942505+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:29:01.979463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.923630+0000","last_change":"2024-02-21T23:06:09.923630+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.923630+0000","last_undegraded":"2024-02-21T23:06:09.923630+0000","last_fullsized":"2024-02-21T23:06:09.923630+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.959070+0000","last_change":"2024-02-21T23:06:09.956377+0000","last_active":"2024-02-21T23:06:09.959070+0000","last_peered":"2024-02-21T23:06:09.959070+0000","last_clean":"2024-02-21T23:06:09.959070+0000","last_became_active":"2024-02-21T23:06:09.956129+0000","last_became_peered":"2024-02-21T23:06:09.956129+0000","last_unstale":"2024-02-21T23:06:09.959070+0000","last_undegraded":"2024-02-21T23:06:09.959070+0000","last_fullsized":"2024-02-21T23:06:09.959070+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:29:15.656090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.946180+0000","last_change":"2024-02-21T23:05:51.414453+0000","last_active":"2024-02-21T23:06:09.946180+0000","last_peered":"2024-02-21T23:06:09.946180+0000","last_clean":"2024-02-21T23:06:09.946180+0000","last_became_active":"2024-02-21T23:05:51.413499+0000","last_became_peered":"2024-02-21T23:05:51.413499+0000","last_unstale":"2024-02-21T23:06:09.946180+0000","last_undegraded":"2024-02-21T23:06:09.946180+0000","last_fullsized":"2024-02-21T23:06:09.946180+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:55:21.967274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"75'2","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.934245+0000","last_change":"2024-02-21T23:05:50.381948+0000","last_active":"2024-02-21T23:06:09.934245+0000","last_peered":"2024-02-21T23:06:09.934245+0000","last_clean":"2024-02-21T23:06:09.934245+0000","last_became_active":"2024-02-21T23:05:50.381452+0000","last_became_peered":"2024-02-21T23:05:50.381452+0000","last_unstale":"2024-02-21T23:06:09.934245+0000","last_undegraded":"2024-02-21T23:06:09.934245+0000","last_fullsized":"2024-02-21T23:06:09.934245+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:22:16.414686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.935470+0000","last_change":"2024-02-21T23:06:09.935470+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.935470+0000","last_undegraded":"2024-02-21T23:06:09.935470+0000","last_fullsized":"2024-02-21T23:06:09.935470+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.958416+0000","last_change":"2024-02-21T23:06:09.953924+0000","last_active":"2024-02-21T23:06:09.958416+0000","last_peered":"2024-02-21T23:06:09.958416+0000","last_clean":"2024-02-21T23:06:09.958416+0000","last_became_active":"2024-02-21T23:06:09.952952+0000","last_became_peered":"2024-02-21T23:06:09.952952+0000","last_unstale":"2024-02-21T23:06:09.958416+0000","last_undegraded":"2024-02-21T23:06:09.958416+0000","last_fullsized":"2024-02-21T23:06:09.958416+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:19:59.841109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.934595+0000","last_change":"2024-02-21T23:05:51.410619+0000","last_active":"2024-02-21T23:06:09.934595+0000","last_peered":"2024-02-21T23:06:09.934595+0000","last_clean":"2024-02-21T23:06:09.934595+0000","last_became_active":"2024-02-21T23:05:51.409937+0000","last_became_peered":"2024-02-21T23:05:51.409937+0000","last_unstale":"2024-02-21T23:06:09.934595+0000","last_undegraded":"2024-02-21T23:06:09.934595+0000","last_fullsized":"2024-02-21T23:06:09.934595+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:22:32.845173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"76'1","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.944263+0000","last_change":"2024-02-21T23:05:50.362370+0000","last_active":"2024-02-21T23:06:09.944263+0000","last_peered":"2024-02-21T23:06:09.944263+0000","last_clean":"2024-02-21T23:06:09.944263+0000","last_became_active":"2024-02-21T23:05:50.361848+0000","last_became_peered":"2024-02-21T23:05:50.361848+0000","last_unstale":"2024-02-21T23:06:09.944263+0000","last_undegraded":"2024-02-21T23:06:09.944263+0000","last_fullsized":"2024-02-21T23:06:09.944263+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:08:35.448428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.930610+0000","last_change":"2024-02-21T23:06:09.930610+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.930610+0000","last_undegraded":"2024-02-21T23:06:09.930610+0000","last_fullsized":"2024-02-21T23:06:09.930610+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.963409+0000","last_change":"2024-02-21T23:06:09.961946+0000","last_active":"2024-02-21T23:06:09.963409+0000","last_peered":"2024-02-21T23:06:09.963409+0000","last_clean":"2024-02-21T23:06:09.963409+0000","last_became_active":"2024-02-21T23:06:09.961662+0000","last_became_peered":"2024-02-21T23:06:09.961662+0000","last_unstale":"2024-02-21T23:06:09.963409+0000","last_undegraded":"2024-02-21T23:06:09.963409+0000","last_fullsized":"2024-02-21T23:06:09.963409+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:15:58.947158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.946318+0000","last_change":"2024-02-21T23:05:51.421357+0000","last_active":"2024-02-21T23:06:09.946318+0000","last_peered":"2024-02-21T23:06:09.946318+0000","last_clean":"2024-02-21T23:06:09.946318+0000","last_became_active":"2024-02-21T23:05:51.420734+0000","last_became_peered":"2024-02-21T23:05:51.420734+0000","last_unstale":"2024-02-21T23:06:09.946318+0000","last_undegraded":"2024-02-21T23:06:09.946318+0000","last_fullsized":"2024-02-21T23:06:09.946318+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:31:44.536584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.944953+0000","last_change":"2024-02-21T23:05:50.384680+0000","last_active":"2024-02-21T23:06:09.944953+0000","last_peered":"2024-02-21T23:06:09.944953+0000","last_clean":"2024-02-21T23:06:09.944953+0000","last_became_active":"2024-02-21T23:05:50.384208+0000","last_became_peered":"2024-02-21T23:05:50.384208+0000","last_unstale":"2024-02-21T23:06:09.944953+0000","last_undegraded":"2024-02-21T23:06:09.944953+0000","last_fullsized":"2024-02-21T23:06:09.944953+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:01:27.369089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.933769+0000","last_change":"2024-02-21T23:06:09.933769+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.933769+0000","last_undegraded":"2024-02-21T23:06:09.933769+0000","last_fullsized":"2024-02-21T23:06:09.933769+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.961163+0000","last_change":"2024-02-21T23:06:09.952597+0000","last_active":"2024-02-21T23:06:09.961163+0000","last_peered":"2024-02-21T23:06:09.961163+0000","last_clean":"2024-02-21T23:06:09.961163+0000","last_became_active":"2024-02-21T23:06:09.952270+0000","last_became_peered":"2024-02-21T23:06:09.952270+0000","last_unstale":"2024-02-21T23:06:09.961163+0000","last_undegraded":"2024-02-21T23:06:09.961163+0000","last_fullsized":"2024-02-21T23:06:09.961163+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:42:45.049032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.944384+0000","last_change":"2024-02-21T23:05:51.395859+0000","last_active":"2024-02-21T23:06:09.944384+0000","last_peered":"2024-02-21T23:06:09.944384+0000","last_clean":"2024-02-21T23:06:09.944384+0000","last_became_active":"2024-02-21T23:05:51.395012+0000","last_became_peered":"2024-02-21T23:05:51.395012+0000","last_unstale":"2024-02-21T23:06:09.944384+0000","last_undegraded":"2024-02-21T23:06:09.944384+0000","last_fullsized":"2024-02-21T23:06:09.944384+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:29:02.992284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.943897+0000","last_change":"2024-02-21T23:05:50.383920+0000","last_active":"2024-02-21T23:06:09.943897+0000","last_peered":"2024-02-21T23:06:09.943897+0000","last_clean":"2024-02-21T23:06:09.943897+0000","last_became_active":"2024-02-21T23:05:50.382488+0000","last_became_peered":"2024-02-21T23:05:50.382488+0000","last_unstale":"2024-02-21T23:06:09.943897+0000","last_undegraded":"2024-02-21T23:06:09.943897+0000","last_fullsized":"2024-02-21T23:06:09.943897+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:26:58.076002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.937602+0000","last_change":"2024-02-21T23:06:09.937602+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.937602+0000","last_undegraded":"2024-02-21T23:06:09.937602+0000","last_fullsized":"2024-02-21T23:06:09.937602+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.966343+0000","last_change":"2024-02-21T23:06:09.961904+0000","last_active":"2024-02-21T23:06:09.966343+0000","last_peered":"2024-02-21T23:06:09.966343+0000","last_clean":"2024-02-21T23:06:09.966343+0000","last_became_active":"2024-02-21T23:06:09.961650+0000","last_became_peered":"2024-02-21T23:06:09.961650+0000","last_unstale":"2024-02-21T23:06:09.966343+0000","last_undegraded":"2024-02-21T23:06:09.966343+0000","last_fullsized":"2024-02-21T23:06:09.966343+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:43:37.070238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.932210+0000","last_change":"2024-02-21T23:05:51.422495+0000","last_active":"2024-02-21T23:06:09.932210+0000","last_peered":"2024-02-21T23:06:09.932210+0000","last_clean":"2024-02-21T23:06:09.932210+0000","last_became_active":"2024-02-21T23:05:51.422196+0000","last_became_peered":"2024-02-21T23:05:51.422196+0000","last_unstale":"2024-02-21T23:06:09.932210+0000","last_undegraded":"2024-02-21T23:06:09.932210+0000","last_fullsized":"2024-02-21T23:06:09.932210+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:10:11.986389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.932846+0000","last_change":"2024-02-21T23:05:50.376027+0000","last_active":"2024-02-21T23:06:09.932846+0000","last_peered":"2024-02-21T23:06:09.932846+0000","last_clean":"2024-02-21T23:06:09.932846+0000","last_became_active":"2024-02-21T23:05:50.375170+0000","last_became_peered":"2024-02-21T23:05:50.375170+0000","last_unstale":"2024-02-21T23:06:09.932846+0000","last_undegraded":"2024-02-21T23:06:09.932846+0000","last_fullsized":"2024-02-21T23:06:09.932846+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:39:32.228635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.924758+0000","last_change":"2024-02-21T23:06:09.924758+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.924758+0000","last_undegraded":"2024-02-21T23:06:09.924758+0000","last_fullsized":"2024-02-21T23:06:09.924758+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.954412+0000","last_change":"2024-02-21T23:06:09.947879+0000","last_active":"2024-02-21T23:06:09.954412+0000","last_peered":"2024-02-21T23:06:09.954412+0000","last_clean":"2024-02-21T23:06:09.954412+0000","last_became_active":"2024-02-21T23:06:09.947599+0000","last_became_peered":"2024-02-21T23:06:09.947599+0000","last_unstale":"2024-02-21T23:06:09.954412+0000","last_undegraded":"2024-02-21T23:06:09.954412+0000","last_fullsized":"2024-02-21T23:06:09.954412+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:50:51.483608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.937296+0000","last_change":"2024-02-21T23:05:51.420500+0000","last_active":"2024-02-21T23:06:09.937296+0000","last_peered":"2024-02-21T23:06:09.937296+0000","last_clean":"2024-02-21T23:06:09.937296+0000","last_became_active":"2024-02-21T23:05:51.420103+0000","last_became_peered":"2024-02-21T23:05:51.420103+0000","last_unstale":"2024-02-21T23:06:09.937296+0000","last_undegraded":"2024-02-21T23:06:09.937296+0000","last_fullsized":"2024-02-21T23:06:09.937296+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:27:01.191337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.937594+0000","last_change":"2024-02-21T23:05:50.354805+0000","last_active":"2024-02-21T23:06:09.937594+0000","last_peered":"2024-02-21T23:06:09.937594+0000","last_clean":"2024-02-21T23:06:09.937594+0000","last_became_active":"2024-02-21T23:05:50.354525+0000","last_became_peered":"2024-02-21T23:05:50.354525+0000","last_unstale":"2024-02-21T23:06:09.937594+0000","last_undegraded":"2024-02-21T23:06:09.937594+0000","last_fullsized":"2024-02-21T23:06:09.937594+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:28:05.314552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.923982+0000","last_change":"2024-02-21T23:06:09.923982+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.923982+0000","last_undegraded":"2024-02-21T23:06:09.923982+0000","last_fullsized":"2024-02-21T23:06:09.923982+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.958392+0000","last_change":"2024-02-21T23:06:09.955164+0000","last_active":"2024-02-21T23:06:09.958392+0000","last_peered":"2024-02-21T23:06:09.958392+0000","last_clean":"2024-02-21T23:06:09.958392+0000","last_became_active":"2024-02-21T23:06:09.954858+0000","last_became_peered":"2024-02-21T23:06:09.954858+0000","last_unstale":"2024-02-21T23:06:09.958392+0000","last_undegraded":"2024-02-21T23:06:09.958392+0000","last_fullsized":"2024-02-21T23:06:09.958392+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:51:29.171374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.920149+0000","last_change":"2024-02-21T23:05:51.416548+0000","last_active":"2024-02-21T23:06:09.920149+0000","last_peered":"2024-02-21T23:06:09.920149+0000","last_clean":"2024-02-21T23:06:09.920149+0000","last_became_active":"2024-02-21T23:05:51.416222+0000","last_became_peered":"2024-02-21T23:05:51.416222+0000","last_unstale":"2024-02-21T23:06:09.920149+0000","last_undegraded":"2024-02-21T23:06:09.920149+0000","last_fullsized":"2024-02-21T23:06:09.920149+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:22:32.845173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.934813+0000","last_change":"2024-02-21T23:05:50.373554+0000","last_active":"2024-02-21T23:06:09.934813+0000","last_peered":"2024-02-21T23:06:09.934813+0000","last_clean":"2024-02-21T23:06:09.934813+0000","last_became_active":"2024-02-21T23:05:50.373022+0000","last_became_peered":"2024-02-21T23:05:50.373022+0000","last_unstale":"2024-02-21T23:06:09.934813+0000","last_undegraded":"2024-02-21T23:06:09.934813+0000","last_fullsized":"2024-02-21T23:06:09.934813+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:38:56.665723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.941480+0000","last_change":"2024-02-21T23:06:09.941480+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.941480+0000","last_undegraded":"2024-02-21T23:06:09.941480+0000","last_fullsized":"2024-02-21T23:06:09.941480+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.957795+0000","last_change":"2024-02-21T23:06:09.953218+0000","last_active":"2024-02-21T23:06:09.957795+0000","last_peered":"2024-02-21T23:06:09.957795+0000","last_clean":"2024-02-21T23:06:09.957795+0000","last_became_active":"2024-02-21T23:06:09.952908+0000","last_became_peered":"2024-02-21T23:06:09.952908+0000","last_unstale":"2024-02-21T23:06:09.957795+0000","last_undegraded":"2024-02-21T23:06:09.957795+0000","last_fullsized":"2024-02-21T23:06:09.957795+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:08:55.021772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.920356+0000","last_change":"2024-02-21T23:05:51.412683+0000","last_active":"2024-02-21T23:06:09.920356+0000","last_peered":"2024-02-21T23:06:09.920356+0000","last_clean":"2024-02-21T23:06:09.920356+0000","last_became_active":"2024-02-21T23:05:51.412389+0000","last_became_peered":"2024-02-21T23:05:51.412389+0000","last_unstale":"2024-02-21T23:06:09.920356+0000","last_undegraded":"2024-02-21T23:06:09.920356+0000","last_fullsized":"2024-02-21T23:06:09.920356+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:57:06.211024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.918795+0000","last_change":"2024-02-21T23:05:50.376356+0000","last_active":"2024-02-21T23:06:09.918795+0000","last_peered":"2024-02-21T23:06:09.918795+0000","last_clean":"2024-02-21T23:06:09.918795+0000","last_became_active":"2024-02-21T23:05:50.370467+0000","last_became_peered":"2024-02-21T23:05:50.370467+0000","last_unstale":"2024-02-21T23:06:09.918795+0000","last_undegraded":"2024-02-21T23:06:09.918795+0000","last_fullsized":"2024-02-21T23:06:09.918795+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:56:49.435540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.940614+0000","last_change":"2024-02-21T23:06:09.940614+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.940614+0000","last_undegraded":"2024-02-21T23:06:09.940614+0000","last_fullsized":"2024-02-21T23:06:09.940614+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.945283+0000","last_change":"2024-02-21T23:06:09.940641+0000","last_active":"2024-02-21T23:06:09.945283+0000","last_peered":"2024-02-21T23:06:09.945283+0000","last_clean":"2024-02-21T23:06:09.945283+0000","last_became_active":"2024-02-21T23:06:09.940353+0000","last_became_peered":"2024-02-21T23:06:09.940353+0000","last_unstale":"2024-02-21T23:06:09.945283+0000","last_undegraded":"2024-02-21T23:06:09.945283+0000","last_fullsized":"2024-02-21T23:06:09.945283+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:45:25.410745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.944841+0000","last_change":"2024-02-21T23:05:51.413008+0000","last_active":"2024-02-21T23:06:09.944841+0000","last_peered":"2024-02-21T23:06:09.944841+0000","last_clean":"2024-02-21T23:06:09.944841+0000","last_became_active":"2024-02-21T23:05:51.412714+0000","last_became_peered":"2024-02-21T23:05:51.412714+0000","last_unstale":"2024-02-21T23:06:09.944841+0000","last_undegraded":"2024-02-21T23:06:09.944841+0000","last_fullsized":"2024-02-21T23:06:09.944841+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:03:08.789066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.946178+0000","last_change":"2024-02-21T23:05:50.375581+0000","last_active":"2024-02-21T23:06:09.946178+0000","last_peered":"2024-02-21T23:06:09.946178+0000","last_clean":"2024-02-21T23:06:09.946178+0000","last_became_active":"2024-02-21T23:05:50.375282+0000","last_became_peered":"2024-02-21T23:05:50.375282+0000","last_unstale":"2024-02-21T23:06:09.946178+0000","last_undegraded":"2024-02-21T23:06:09.946178+0000","last_fullsized":"2024-02-21T23:06:09.946178+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:49:45.012428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":278,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.942188+0000","last_change":"2024-02-21T23:02:32.289335+0000","last_active":"2024-02-21T23:06:09.942188+0000","last_peered":"2024-02-21T23:06:09.942188+0000","last_clean":"2024-02-21T23:06:09.942188+0000","last_became_active":"2024-02-21T23:02:32.286913+0000","last_became_peered":"2024-02-21T23:02:32.286913+0000","last_unstale":"2024-02-21T23:06:09.942188+0000","last_undegraded":"2024-02-21T23:06:09.942188+0000","last_fullsized":"2024-02-21T23:06:09.942188+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_clean_scrub_stamp":"2024-02-21T23:02:31.172569+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:36:23.987374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":14,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574894,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32180,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339660,"statfs":{"total":96636764160,"available":96603811840,"internally_reserved":0,"allocated":1024000,"data_stored":1048493,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574895,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32128,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339712,"statfs":{"total":96636764160,"available":96603865088,"internally_reserved":0,"allocated":966656,"data_stored":459715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574894,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27960,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343880,"statfs":{"total":96636764160,"available":96608133120,"internally_reserved":0,"allocated":958464,"data_stored":458137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574895,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32136,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339704,"statfs":{"total":96636764160,"available":96603856896,"internally_reserved":0,"allocated":974848,"data_stored":459819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574895,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32120,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339720,"statfs":{"total":96636764160,"available":96603873280,"internally_reserved":0,"allocated":958464,"data_stored":458685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574896,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32128,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339712,"statfs":{"total":96636764160,"available":96603865088,"internally_reserved":0,"allocated":966656,"data_stored":458845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574894,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32108,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339732,"statfs":{"total":96636764160,"available":96603885568,"internally_reserved":0,"allocated":950272,"data_stored":450693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574894,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32176,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339664,"statfs":{"total":96636764160,"available":96603815936,"internally_reserved":0,"allocated":1019904,"data_stored":1048471,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-21T23:06:15.550 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-21T23:06:15.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.818+0000 7f6408a00640 1 Processor -- start 2024-02-21T23:06:15.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.819+0000 7f6408a00640 1 -- start start 2024-02-21T23:06:15.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.820+0000 7f6408a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6404152500 0x7f64041558c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:15.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.820+0000 7f6408a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64041528e0 con 0x7f640406bbf0 2024-02-21T23:06:15.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.820+0000 7f6408a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64040c86e0 con 0x7f64040ca050 2024-02-21T23:06:15.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.820+0000 7f6408a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64040c8980 con 0x7f6404152500 2024-02-21T23:06:15.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.820+0000 7f6403400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6404152500 0x7f64041558c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:15.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.820+0000 7f6403e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f640406bbf0 0x7f640406bff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42524/0 (socket says 172.21.15.28:42524) 2024-02-21T23:06:15.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.820+0000 7f6403e00640 1 -- 172.21.15.28:0/2136822261 learned_addr learned my addr 172.21.15.28:0/2136822261 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:06:15.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.821+0000 7f6403400640 1 -- 172.21.15.28:0/2136822261 >> v1:172.21.15.81:6789/0 conn(0x7f64040ca050 legacy=0x7f64040bf6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:15.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.821+0000 7f6403400640 1 -- 172.21.15.28:0/2136822261 >> v1:172.21.15.28:6789/0 conn(0x7f640406bbf0 legacy=0x7f640406bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:15.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.821+0000 7f6403400640 1 -- 172.21.15.28:0/2136822261 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64040c8c60 con 0x7f6404152500 2024-02-21T23:06:15.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.822+0000 7f6403400640 1 --2- 172.21.15.28:0/2136822261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6404152500 0x7f64041558c0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f63f40097a0 tx=0x7f63f402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60bf8ec07d6d70a1 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:15.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.823+0000 7f6402000640 1 -- 172.21.15.28:0/2136822261 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f40315a0 con 0x7f6404152500 2024-02-21T23:06:15.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.823+0000 7f6402000640 1 -- 172.21.15.28:0/2136822261 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4037070 con 0x7f6404152500 2024-02-21T23:06:15.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.823+0000 7f6402000640 1 -- 172.21.15.28:0/2136822261 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4031960 con 0x7f6404152500 2024-02-21T23:06:15.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.823+0000 7f6408a00640 1 -- 172.21.15.28:0/2136822261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6404152500 msgr2=0x7f64041558c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:15.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.823+0000 7f6408a00640 1 --2- 172.21.15.28:0/2136822261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6404152500 0x7f64041558c0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f63f40097a0 tx=0x7f63f402b840 comp rx=0 tx=0).stop 2024-02-21T23:06:15.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.823+0000 7f6408a00640 1 -- 172.21.15.28:0/2136822261 shutdown_connections 2024-02-21T23:06:15.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.823+0000 7f6408a00640 1 --2- 172.21.15.28:0/2136822261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6404152500 0x7f64041558c0 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:15.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.823+0000 7f6408a00640 1 -- 172.21.15.28:0/2136822261 >> 172.21.15.28:0/2136822261 conn(0x7f64040ac450 msgr2=0x7f64040ac860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:15.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.823+0000 7f6408a00640 1 -- 172.21.15.28:0/2136822261 shutdown_connections 2024-02-21T23:06:15.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.823+0000 7f6408a00640 1 -- 172.21.15.28:0/2136822261 wait complete. 2024-02-21T23:06:15.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.826+0000 7f6408a00640 1 Processor -- start 2024-02-21T23:06:15.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.826+0000 7f6408a00640 1 -- start start 2024-02-21T23:06:15.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.828+0000 7f6408a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f640406bbf0 0x7f64041564d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:15.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.828+0000 7f6408a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64040ca050 0x7f64041480c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:15.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.828+0000 7f6408a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6404152500 0x7f6404148600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:15.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.828+0000 7f6408a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64040bfdd0 con 0x7f64040ca050 2024-02-21T23:06:15.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.828+0000 7f6408a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64040c9220 con 0x7f6404152500 2024-02-21T23:06:15.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.828+0000 7f6408a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64040bff50 con 0x7f640406bbf0 2024-02-21T23:06:15.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.828+0000 7f6402a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64040ca050 0x7f64041480c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:15.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.828+0000 7f6402a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64040ca050 0x7f64041480c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54638/0 (socket says 172.21.15.28:54638) 2024-02-21T23:06:15.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.828+0000 7f6402a00640 1 -- 172.21.15.28:0/3045024171 learned_addr learned my addr 172.21.15.28:0/3045024171 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:15.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.828+0000 7f6403400640 1 --2- 172.21.15.28:0/3045024171 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f640406bbf0 0x7f64041564d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:15.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.828+0000 7f6403e00640 1 --2- 172.21.15.28:0/3045024171 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6404152500 0x7f6404148600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:15.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.829+0000 7f6402a00640 1 -- 172.21.15.28:0/3045024171 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f640406bbf0 msgr2=0x7f64041564d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:15.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.829+0000 7f6402a00640 1 --2- 172.21.15.28:0/3045024171 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f640406bbf0 0x7f64041564d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:15.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.829+0000 7f6402a00640 1 -- 172.21.15.28:0/3045024171 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6404152500 msgr2=0x7f6404148600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:15.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.829+0000 7f6402a00640 1 --2- 172.21.15.28:0/3045024171 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6404152500 0x7f6404148600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:15.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.829+0000 7f6402a00640 1 -- 172.21.15.28:0/3045024171 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f640414b720 con 0x7f64040ca050 2024-02-21T23:06:15.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.829+0000 7f6402a00640 1 --2- 172.21.15.28:0/3045024171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64040ca050 0x7f64041480c0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f63f000b3f0 tx=0x7f63f000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:15.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.829+0000 7f63fb400640 1 -- 172.21.15.28:0/3045024171 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0015020 con 0x7f64040ca050 2024-02-21T23:06:15.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.829+0000 7f63fb400640 1 -- 172.21.15.28:0/3045024171 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0011830 con 0x7f64040ca050 2024-02-21T23:06:15.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.830+0000 7f6408a00640 1 -- 172.21.15.28:0/3045024171 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f640414c280 con 0x7f64040ca050 2024-02-21T23:06:15.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.830+0000 7f63fb400640 1 -- 172.21.15.28:0/3045024171 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0011c20 con 0x7f64040ca050 2024-02-21T23:06:15.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.830+0000 7f6408a00640 1 -- 172.21.15.28:0/3045024171 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f640414bfa0 con 0x7f64040ca050 2024-02-21T23:06:15.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.831+0000 7f6408a00640 1 -- 172.21.15.28:0/3045024171 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f640414bd90 con 0x7f64040ca050 2024-02-21T23:06:15.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.833+0000 7f63fb400640 1 -- 172.21.15.28:0/3045024171 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f63f0018020 con 0x7f64040ca050 2024-02-21T23:06:15.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.834+0000 7f63fb400640 1 --2- 172.21.15.28:0/3045024171 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f63e406b500 0x7f63e406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:15.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.834+0000 7f63fb400640 1 -- 172.21.15.28:0/3045024171 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f63f008a380 con 0x7f64040ca050 2024-02-21T23:06:15.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.834+0000 7f6403400640 1 --2- 172.21.15.28:0/3045024171 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f63e406b500 0x7f63e406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:15.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.836+0000 7f6403400640 1 --2- 172.21.15.28:0/3045024171 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f63e406b500 0x7f63e406d9c0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f63f4036000 tx=0x7f63f403d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:15.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:15.839+0000 7f63fb400640 1 -- 172.21.15.28:0/3045024171 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f63f001f030 con 0x7f64040ca050 2024-02-21T23:06:16.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.088+0000 7f6408a00640 1 -- 172.21.15.28:0/3045024171 --> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f64041eae30 con 0x7f63e406b500 2024-02-21T23:06:16.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.122+0000 7f63fb400640 1 -- 172.21.15.28:0/3045024171 <== mgr.4101 v2:172.21.15.81:6832/1848538577 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+485422 (secure 0 0 0) 0x7f63f4037040 con 0x7f63e406b500 2024-02-21T23:06:16.118 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:06:16.122 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-21T23:06:16.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.132+0000 7f6408a00640 1 -- 172.21.15.28:0/3045024171 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f63e406b500 msgr2=0x7f63e406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:16.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.132+0000 7f6408a00640 1 --2- 172.21.15.28:0/3045024171 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f63e406b500 0x7f63e406d9c0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f63f4036000 tx=0x7f63f403d040 comp rx=0 tx=0).stop 2024-02-21T23:06:16.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.132+0000 7f6408a00640 1 -- 172.21.15.28:0/3045024171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64040ca050 msgr2=0x7f64041480c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:16.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.132+0000 7f6408a00640 1 --2- 172.21.15.28:0/3045024171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64040ca050 0x7f64041480c0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f63f000b3f0 tx=0x7f63f000b8b0 comp rx=0 tx=0).stop 2024-02-21T23:06:16.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.132+0000 7f6408a00640 1 -- 172.21.15.28:0/3045024171 shutdown_connections 2024-02-21T23:06:16.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.132+0000 7f6408a00640 1 --2- 172.21.15.28:0/3045024171 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f63e406b500 0x7f63e406d9c0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:16.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.132+0000 7f6408a00640 1 --2- 172.21.15.28:0/3045024171 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6404152500 0x7f6404148600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:16.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.132+0000 7f6408a00640 1 --2- 172.21.15.28:0/3045024171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64040ca050 0x7f64041480c0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:16.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.132+0000 7f6408a00640 1 --2- 172.21.15.28:0/3045024171 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f640406bbf0 0x7f64041564d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:16.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.132+0000 7f6408a00640 1 -- 172.21.15.28:0/3045024171 >> 172.21.15.28:0/3045024171 conn(0x7f64040ac450 msgr2=0x7f640406bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:16.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.132+0000 7f6408a00640 1 -- 172.21.15.28:0/3045024171 shutdown_connections 2024-02-21T23:06:16.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.132+0000 7f6408a00640 1 -- 172.21.15.28:0/3045024171 wait complete. 2024-02-21T23:06:16.144 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":185,"stamp":"2024-02-21T23:06:14.741815+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":228,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":252936,"kb_used_data":7636,"kb_used_omap":12,"kb_used_meta":245171,"kb_avail":754721784,"statfs":{"total":773094113280,"available":772835106816,"internally_reserved":0,"allocated":7819264,"data_stored":4842858,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":251055665},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.833080"},"pg_stats":[{"pgid":"14.3d","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.940039+0000","last_change":"2024-02-21T23:06:09.940039+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.940039+0000","last_undegraded":"2024-02-21T23:06:09.940039+0000","last_fullsized":"2024-02-21T23:06:09.940039+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.3f","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.946468+0000","last_change":"2024-02-21T23:05:51.408831+0000","last_active":"2024-02-21T23:06:09.946468+0000","last_peered":"2024-02-21T23:06:09.946468+0000","last_clean":"2024-02-21T23:06:09.946468+0000","last_became_active":"2024-02-21T23:05:51.408353+0000","last_became_peered":"2024-02-21T23:05:51.408353+0000","last_unstale":"2024-02-21T23:06:09.946468+0000","last_undegraded":"2024-02-21T23:06:09.946468+0000","last_fullsized":"2024-02-21T23:06:09.946468+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:36:47.962935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.3c","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.931414+0000","last_change":"2024-02-21T23:06:09.931414+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.931414+0000","last_undegraded":"2024-02-21T23:06:09.931414+0000","last_fullsized":"2024-02-21T23:06:09.931414+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.944046+0000","last_change":"2024-02-21T23:05:51.420557+0000","last_active":"2024-02-21T23:06:09.944046+0000","last_peered":"2024-02-21T23:06:09.944046+0000","last_clean":"2024-02-21T23:06:09.944046+0000","last_became_active":"2024-02-21T23:05:51.420078+0000","last_became_peered":"2024-02-21T23:05:51.420078+0000","last_unstale":"2024-02-21T23:06:09.944046+0000","last_undegraded":"2024-02-21T23:06:09.944046+0000","last_fullsized":"2024-02-21T23:06:09.944046+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:36:23.792395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.3f","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.936313+0000","last_change":"2024-02-21T23:06:09.936313+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.936313+0000","last_undegraded":"2024-02-21T23:06:09.936313+0000","last_fullsized":"2024-02-21T23:06:09.936313+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.919954+0000","last_change":"2024-02-21T23:05:51.408934+0000","last_active":"2024-02-21T23:06:09.919954+0000","last_peered":"2024-02-21T23:06:09.919954+0000","last_clean":"2024-02-21T23:06:09.919954+0000","last_became_active":"2024-02-21T23:05:51.408597+0000","last_became_peered":"2024-02-21T23:05:51.408597+0000","last_unstale":"2024-02-21T23:06:09.919954+0000","last_undegraded":"2024-02-21T23:06:09.919954+0000","last_fullsized":"2024-02-21T23:06:09.919954+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:58:55.829941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.3e","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.924703+0000","last_change":"2024-02-21T23:06:09.924703+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.924703+0000","last_undegraded":"2024-02-21T23:06:09.924703+0000","last_fullsized":"2024-02-21T23:06:09.924703+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.934821+0000","last_change":"2024-02-21T23:05:51.423479+0000","last_active":"2024-02-21T23:06:09.934821+0000","last_peered":"2024-02-21T23:06:09.934821+0000","last_clean":"2024-02-21T23:06:09.934821+0000","last_became_active":"2024-02-21T23:05:51.423108+0000","last_became_peered":"2024-02-21T23:05:51.423108+0000","last_unstale":"2024-02-21T23:06:09.934821+0000","last_undegraded":"2024-02-21T23:06:09.934821+0000","last_fullsized":"2024-02-21T23:06:09.934821+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:36:47.962935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.39","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.936144+0000","last_change":"2024-02-21T23:06:09.936144+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.936144+0000","last_undegraded":"2024-02-21T23:06:09.936144+0000","last_fullsized":"2024-02-21T23:06:09.936144+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.940522+0000","last_change":"2024-02-21T23:05:51.419436+0000","last_active":"2024-02-21T23:06:09.940522+0000","last_peered":"2024-02-21T23:06:09.940522+0000","last_clean":"2024-02-21T23:06:09.940522+0000","last_became_active":"2024-02-21T23:05:51.418562+0000","last_became_peered":"2024-02-21T23:05:51.418562+0000","last_unstale":"2024-02-21T23:06:09.940522+0000","last_undegraded":"2024-02-21T23:06:09.940522+0000","last_fullsized":"2024-02-21T23:06:09.940522+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:19:45.189126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.38","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.932838+0000","last_change":"2024-02-21T23:06:09.932838+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.932838+0000","last_undegraded":"2024-02-21T23:06:09.932838+0000","last_fullsized":"2024-02-21T23:06:09.932838+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.945821+0000","last_change":"2024-02-21T23:05:51.410014+0000","last_active":"2024-02-21T23:06:09.945821+0000","last_peered":"2024-02-21T23:06:09.945821+0000","last_clean":"2024-02-21T23:06:09.945821+0000","last_became_active":"2024-02-21T23:05:51.409637+0000","last_became_peered":"2024-02-21T23:05:51.409637+0000","last_unstale":"2024-02-21T23:06:09.945821+0000","last_undegraded":"2024-02-21T23:06:09.945821+0000","last_fullsized":"2024-02-21T23:06:09.945821+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:03:35.243949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3b","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.943914+0000","last_change":"2024-02-21T23:06:09.943914+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.943914+0000","last_undegraded":"2024-02-21T23:06:09.943914+0000","last_fullsized":"2024-02-21T23:06:09.943914+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.934473+0000","last_change":"2024-02-21T23:05:51.419505+0000","last_active":"2024-02-21T23:06:09.934473+0000","last_peered":"2024-02-21T23:06:09.934473+0000","last_clean":"2024-02-21T23:06:09.934473+0000","last_became_active":"2024-02-21T23:05:51.418040+0000","last_became_peered":"2024-02-21T23:05:51.418040+0000","last_unstale":"2024-02-21T23:06:09.934473+0000","last_undegraded":"2024-02-21T23:06:09.934473+0000","last_fullsized":"2024-02-21T23:06:09.934473+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:57:06.211024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.3a","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.940874+0000","last_change":"2024-02-21T23:06:09.940874+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.940874+0000","last_undegraded":"2024-02-21T23:06:09.940874+0000","last_fullsized":"2024-02-21T23:06:09.940874+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.930980+0000","last_change":"2024-02-21T23:05:51.416524+0000","last_active":"2024-02-21T23:06:09.930980+0000","last_peered":"2024-02-21T23:06:09.930980+0000","last_clean":"2024-02-21T23:06:09.930980+0000","last_became_active":"2024-02-21T23:05:51.415391+0000","last_became_peered":"2024-02-21T23:05:51.415391+0000","last_unstale":"2024-02-21T23:06:09.930980+0000","last_undegraded":"2024-02-21T23:06:09.930980+0000","last_fullsized":"2024-02-21T23:06:09.930980+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:48:32.958103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.35","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.938982+0000","last_change":"2024-02-21T23:06:09.938982+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.938982+0000","last_undegraded":"2024-02-21T23:06:09.938982+0000","last_fullsized":"2024-02-21T23:06:09.938982+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.950904+0000","last_change":"2024-02-21T23:05:51.421068+0000","last_active":"2024-02-21T23:06:09.950904+0000","last_peered":"2024-02-21T23:06:09.950904+0000","last_clean":"2024-02-21T23:06:09.950904+0000","last_became_active":"2024-02-21T23:05:51.419746+0000","last_became_peered":"2024-02-21T23:05:51.419746+0000","last_unstale":"2024-02-21T23:06:09.950904+0000","last_undegraded":"2024-02-21T23:06:09.950904+0000","last_fullsized":"2024-02-21T23:06:09.950904+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:09:42.444733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.34","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.937566+0000","last_change":"2024-02-21T23:06:09.937566+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.937566+0000","last_undegraded":"2024-02-21T23:06:09.937566+0000","last_fullsized":"2024-02-21T23:06:09.937566+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.934177+0000","last_change":"2024-02-21T23:05:51.418295+0000","last_active":"2024-02-21T23:06:09.934177+0000","last_peered":"2024-02-21T23:06:09.934177+0000","last_clean":"2024-02-21T23:06:09.934177+0000","last_became_active":"2024-02-21T23:05:51.417780+0000","last_became_peered":"2024-02-21T23:05:51.417780+0000","last_unstale":"2024-02-21T23:06:09.934177+0000","last_undegraded":"2024-02-21T23:06:09.934177+0000","last_fullsized":"2024-02-21T23:06:09.934177+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:19:57.210565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.37","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.936387+0000","last_change":"2024-02-21T23:06:09.936387+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.936387+0000","last_undegraded":"2024-02-21T23:06:09.936387+0000","last_fullsized":"2024-02-21T23:06:09.936387+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.951775+0000","last_change":"2024-02-21T23:05:51.420518+0000","last_active":"2024-02-21T23:06:09.951775+0000","last_peered":"2024-02-21T23:06:09.951775+0000","last_clean":"2024-02-21T23:06:09.951775+0000","last_became_active":"2024-02-21T23:05:51.419564+0000","last_became_peered":"2024-02-21T23:05:51.419564+0000","last_unstale":"2024-02-21T23:06:09.951775+0000","last_undegraded":"2024-02-21T23:06:09.951775+0000","last_fullsized":"2024-02-21T23:06:09.951775+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:19:41.280587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.36","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.932127+0000","last_change":"2024-02-21T23:06:09.932127+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.932127+0000","last_undegraded":"2024-02-21T23:06:09.932127+0000","last_fullsized":"2024-02-21T23:06:09.932127+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.934938+0000","last_change":"2024-02-21T23:05:51.421691+0000","last_active":"2024-02-21T23:06:09.934938+0000","last_peered":"2024-02-21T23:06:09.934938+0000","last_clean":"2024-02-21T23:06:09.934938+0000","last_became_active":"2024-02-21T23:05:51.417621+0000","last_became_peered":"2024-02-21T23:05:51.417621+0000","last_unstale":"2024-02-21T23:06:09.934938+0000","last_undegraded":"2024-02-21T23:06:09.934938+0000","last_fullsized":"2024-02-21T23:06:09.934938+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:46:01.608149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.31","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.929205+0000","last_change":"2024-02-21T23:06:09.929205+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.929205+0000","last_undegraded":"2024-02-21T23:06:09.929205+0000","last_fullsized":"2024-02-21T23:06:09.929205+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.940619+0000","last_change":"2024-02-21T23:05:51.426871+0000","last_active":"2024-02-21T23:06:09.940619+0000","last_peered":"2024-02-21T23:06:09.940619+0000","last_clean":"2024-02-21T23:06:09.940619+0000","last_became_active":"2024-02-21T23:05:51.419412+0000","last_became_peered":"2024-02-21T23:05:51.419412+0000","last_unstale":"2024-02-21T23:06:09.940619+0000","last_undegraded":"2024-02-21T23:06:09.940619+0000","last_fullsized":"2024-02-21T23:06:09.940619+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:22:09.156621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.30","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.929405+0000","last_change":"2024-02-21T23:06:09.929405+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.929405+0000","last_undegraded":"2024-02-21T23:06:09.929405+0000","last_fullsized":"2024-02-21T23:06:09.929405+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.945911+0000","last_change":"2024-02-21T23:05:51.417593+0000","last_active":"2024-02-21T23:06:09.945911+0000","last_peered":"2024-02-21T23:06:09.945911+0000","last_clean":"2024-02-21T23:06:09.945911+0000","last_became_active":"2024-02-21T23:05:51.416945+0000","last_became_peered":"2024-02-21T23:05:51.416945+0000","last_unstale":"2024-02-21T23:06:09.945911+0000","last_undegraded":"2024-02-21T23:06:09.945911+0000","last_fullsized":"2024-02-21T23:06:09.945911+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:24:32.863072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.33","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.929330+0000","last_change":"2024-02-21T23:06:09.929330+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.929330+0000","last_undegraded":"2024-02-21T23:06:09.929330+0000","last_fullsized":"2024-02-21T23:06:09.929330+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.943577+0000","last_change":"2024-02-21T23:05:51.409645+0000","last_active":"2024-02-21T23:06:09.943577+0000","last_peered":"2024-02-21T23:06:09.943577+0000","last_clean":"2024-02-21T23:06:09.943577+0000","last_became_active":"2024-02-21T23:05:51.409356+0000","last_became_peered":"2024-02-21T23:05:51.409356+0000","last_unstale":"2024-02-21T23:06:09.943577+0000","last_undegraded":"2024-02-21T23:06:09.943577+0000","last_fullsized":"2024-02-21T23:06:09.943577+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:17:26.329609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.32","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.942772+0000","last_change":"2024-02-21T23:06:09.942772+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.942772+0000","last_undegraded":"2024-02-21T23:06:09.942772+0000","last_fullsized":"2024-02-21T23:06:09.942772+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.941967+0000","last_change":"2024-02-21T23:05:51.413656+0000","last_active":"2024-02-21T23:06:09.941967+0000","last_peered":"2024-02-21T23:06:09.941967+0000","last_clean":"2024-02-21T23:06:09.941967+0000","last_became_active":"2024-02-21T23:05:51.413350+0000","last_became_peered":"2024-02-21T23:05:51.413350+0000","last_unstale":"2024-02-21T23:06:09.941967+0000","last_undegraded":"2024-02-21T23:06:09.941967+0000","last_fullsized":"2024-02-21T23:06:09.941967+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:19:57.210565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.2d","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.938385+0000","last_change":"2024-02-21T23:06:09.938385+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.938385+0000","last_undegraded":"2024-02-21T23:06:09.938385+0000","last_fullsized":"2024-02-21T23:06:09.938385+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.930801+0000","last_change":"2024-02-21T23:05:51.427044+0000","last_active":"2024-02-21T23:06:09.930801+0000","last_peered":"2024-02-21T23:06:09.930801+0000","last_clean":"2024-02-21T23:06:09.930801+0000","last_became_active":"2024-02-21T23:05:51.419477+0000","last_became_peered":"2024-02-21T23:05:51.419477+0000","last_unstale":"2024-02-21T23:06:09.930801+0000","last_undegraded":"2024-02-21T23:06:09.930801+0000","last_fullsized":"2024-02-21T23:06:09.930801+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:04:00.714448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.2c","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.929202+0000","last_change":"2024-02-21T23:06:09.929202+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.929202+0000","last_undegraded":"2024-02-21T23:06:09.929202+0000","last_fullsized":"2024-02-21T23:06:09.929202+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.920081+0000","last_change":"2024-02-21T23:05:51.425126+0000","last_active":"2024-02-21T23:06:09.920081+0000","last_peered":"2024-02-21T23:06:09.920081+0000","last_clean":"2024-02-21T23:06:09.920081+0000","last_became_active":"2024-02-21T23:05:51.422511+0000","last_became_peered":"2024-02-21T23:05:51.422511+0000","last_unstale":"2024-02-21T23:06:09.920081+0000","last_undegraded":"2024-02-21T23:06:09.920081+0000","last_fullsized":"2024-02-21T23:06:09.920081+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:27:01.191337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.2f","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.939567+0000","last_change":"2024-02-21T23:06:09.939567+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.939567+0000","last_undegraded":"2024-02-21T23:06:09.939567+0000","last_fullsized":"2024-02-21T23:06:09.939567+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.946776+0000","last_change":"2024-02-21T23:05:51.423258+0000","last_active":"2024-02-21T23:06:09.946776+0000","last_peered":"2024-02-21T23:06:09.946776+0000","last_clean":"2024-02-21T23:06:09.946776+0000","last_became_active":"2024-02-21T23:05:51.422987+0000","last_became_peered":"2024-02-21T23:05:51.422987+0000","last_unstale":"2024-02-21T23:06:09.946776+0000","last_undegraded":"2024-02-21T23:06:09.946776+0000","last_fullsized":"2024-02-21T23:06:09.946776+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:41:31.260842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2e","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.941381+0000","last_change":"2024-02-21T23:06:09.941381+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.941381+0000","last_undegraded":"2024-02-21T23:06:09.941381+0000","last_fullsized":"2024-02-21T23:06:09.941381+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.937988+0000","last_change":"2024-02-21T23:05:51.427468+0000","last_active":"2024-02-21T23:06:09.937988+0000","last_peered":"2024-02-21T23:06:09.937988+0000","last_clean":"2024-02-21T23:06:09.937988+0000","last_became_active":"2024-02-21T23:05:51.425476+0000","last_became_peered":"2024-02-21T23:05:51.425476+0000","last_unstale":"2024-02-21T23:06:09.937988+0000","last_undegraded":"2024-02-21T23:06:09.937988+0000","last_fullsized":"2024-02-21T23:06:09.937988+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:06:18.158392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.29","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.929062+0000","last_change":"2024-02-21T23:06:09.929062+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.929062+0000","last_undegraded":"2024-02-21T23:06:09.929062+0000","last_fullsized":"2024-02-21T23:06:09.929062+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.948695+0000","last_change":"2024-02-21T23:05:51.415772+0000","last_active":"2024-02-21T23:06:09.948695+0000","last_peered":"2024-02-21T23:06:09.948695+0000","last_clean":"2024-02-21T23:06:09.948695+0000","last_became_active":"2024-02-21T23:05:51.415494+0000","last_became_peered":"2024-02-21T23:05:51.415494+0000","last_unstale":"2024-02-21T23:06:09.948695+0000","last_undegraded":"2024-02-21T23:06:09.948695+0000","last_fullsized":"2024-02-21T23:06:09.948695+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:42:37.006685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.28","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.932618+0000","last_change":"2024-02-21T23:06:09.932618+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.932618+0000","last_undegraded":"2024-02-21T23:06:09.932618+0000","last_fullsized":"2024-02-21T23:06:09.932618+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.937010+0000","last_change":"2024-02-21T23:05:51.428557+0000","last_active":"2024-02-21T23:06:09.937010+0000","last_peered":"2024-02-21T23:06:09.937010+0000","last_clean":"2024-02-21T23:06:09.937010+0000","last_became_active":"2024-02-21T23:05:51.427612+0000","last_became_peered":"2024-02-21T23:05:51.427612+0000","last_unstale":"2024-02-21T23:06:09.937010+0000","last_undegraded":"2024-02-21T23:06:09.937010+0000","last_fullsized":"2024-02-21T23:06:09.937010+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:38:44.895318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.2b","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.940735+0000","last_change":"2024-02-21T23:06:09.940735+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.940735+0000","last_undegraded":"2024-02-21T23:06:09.940735+0000","last_fullsized":"2024-02-21T23:06:09.940735+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.944849+0000","last_change":"2024-02-21T23:05:51.417302+0000","last_active":"2024-02-21T23:06:09.944849+0000","last_peered":"2024-02-21T23:06:09.944849+0000","last_clean":"2024-02-21T23:06:09.944849+0000","last_became_active":"2024-02-21T23:05:51.414092+0000","last_became_peered":"2024-02-21T23:05:51.414092+0000","last_unstale":"2024-02-21T23:06:09.944849+0000","last_undegraded":"2024-02-21T23:06:09.944849+0000","last_fullsized":"2024-02-21T23:06:09.944849+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:49:01.911867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2a","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.941237+0000","last_change":"2024-02-21T23:06:09.941237+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.941237+0000","last_undegraded":"2024-02-21T23:06:09.941237+0000","last_fullsized":"2024-02-21T23:06:09.941237+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.944272+0000","last_change":"2024-02-21T23:05:51.406411+0000","last_active":"2024-02-21T23:06:09.944272+0000","last_peered":"2024-02-21T23:06:09.944272+0000","last_clean":"2024-02-21T23:06:09.944272+0000","last_became_active":"2024-02-21T23:05:51.406104+0000","last_became_peered":"2024-02-21T23:05:51.406104+0000","last_unstale":"2024-02-21T23:06:09.944272+0000","last_undegraded":"2024-02-21T23:06:09.944272+0000","last_fullsized":"2024-02-21T23:06:09.944272+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:03:08.789066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.25","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.931274+0000","last_change":"2024-02-21T23:06:09.931274+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.931274+0000","last_undegraded":"2024-02-21T23:06:09.931274+0000","last_fullsized":"2024-02-21T23:06:09.931274+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.948003+0000","last_change":"2024-02-21T23:05:51.414112+0000","last_active":"2024-02-21T23:06:09.948003+0000","last_peered":"2024-02-21T23:06:09.948003+0000","last_clean":"2024-02-21T23:06:09.948003+0000","last_became_active":"2024-02-21T23:05:51.413796+0000","last_became_peered":"2024-02-21T23:05:51.413796+0000","last_unstale":"2024-02-21T23:06:09.948003+0000","last_undegraded":"2024-02-21T23:06:09.948003+0000","last_fullsized":"2024-02-21T23:06:09.948003+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:42:26.488510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.24","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.928558+0000","last_change":"2024-02-21T23:06:09.928558+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.928558+0000","last_undegraded":"2024-02-21T23:06:09.928558+0000","last_fullsized":"2024-02-21T23:06:09.928558+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.934345+0000","last_change":"2024-02-21T23:05:51.421208+0000","last_active":"2024-02-21T23:06:09.934345+0000","last_peered":"2024-02-21T23:06:09.934345+0000","last_clean":"2024-02-21T23:06:09.934345+0000","last_became_active":"2024-02-21T23:05:51.418171+0000","last_became_peered":"2024-02-21T23:05:51.418171+0000","last_unstale":"2024-02-21T23:06:09.934345+0000","last_undegraded":"2024-02-21T23:06:09.934345+0000","last_fullsized":"2024-02-21T23:06:09.934345+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:17:26.329609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.27","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.925950+0000","last_change":"2024-02-21T23:06:09.925950+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.925950+0000","last_undegraded":"2024-02-21T23:06:09.925950+0000","last_fullsized":"2024-02-21T23:06:09.925950+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.944600+0000","last_change":"2024-02-21T23:05:51.416857+0000","last_active":"2024-02-21T23:06:09.944600+0000","last_peered":"2024-02-21T23:06:09.944600+0000","last_clean":"2024-02-21T23:06:09.944600+0000","last_became_active":"2024-02-21T23:05:51.416577+0000","last_became_peered":"2024-02-21T23:05:51.416577+0000","last_unstale":"2024-02-21T23:06:09.944600+0000","last_undegraded":"2024-02-21T23:06:09.944600+0000","last_fullsized":"2024-02-21T23:06:09.944600+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:17:26.329609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.26","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.940425+0000","last_change":"2024-02-21T23:06:09.940425+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.940425+0000","last_undegraded":"2024-02-21T23:06:09.940425+0000","last_fullsized":"2024-02-21T23:06:09.940425+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.938091+0000","last_change":"2024-02-21T23:05:51.427489+0000","last_active":"2024-02-21T23:06:09.938091+0000","last_peered":"2024-02-21T23:06:09.938091+0000","last_clean":"2024-02-21T23:06:09.938091+0000","last_became_active":"2024-02-21T23:05:51.419458+0000","last_became_peered":"2024-02-21T23:05:51.419458+0000","last_unstale":"2024-02-21T23:06:09.938091+0000","last_undegraded":"2024-02-21T23:06:09.938091+0000","last_fullsized":"2024-02-21T23:06:09.938091+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:54:08.758447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.21","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.922993+0000","last_change":"2024-02-21T23:06:09.922993+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.922993+0000","last_undegraded":"2024-02-21T23:06:09.922993+0000","last_fullsized":"2024-02-21T23:06:09.922993+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.944278+0000","last_change":"2024-02-21T23:05:51.405741+0000","last_active":"2024-02-21T23:06:09.944278+0000","last_peered":"2024-02-21T23:06:09.944278+0000","last_clean":"2024-02-21T23:06:09.944278+0000","last_became_active":"2024-02-21T23:05:51.405301+0000","last_became_peered":"2024-02-21T23:05:51.405301+0000","last_unstale":"2024-02-21T23:06:09.944278+0000","last_undegraded":"2024-02-21T23:06:09.944278+0000","last_fullsized":"2024-02-21T23:06:09.944278+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:36:47.962935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.20","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.930360+0000","last_change":"2024-02-21T23:06:09.930360+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.930360+0000","last_undegraded":"2024-02-21T23:06:09.930360+0000","last_fullsized":"2024-02-21T23:06:09.930360+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.942207+0000","last_change":"2024-02-21T23:05:51.398663+0000","last_active":"2024-02-21T23:06:09.942207+0000","last_peered":"2024-02-21T23:06:09.942207+0000","last_clean":"2024-02-21T23:06:09.942207+0000","last_became_active":"2024-02-21T23:05:51.398146+0000","last_became_peered":"2024-02-21T23:05:51.398146+0000","last_unstale":"2024-02-21T23:06:09.942207+0000","last_undegraded":"2024-02-21T23:06:09.942207+0000","last_fullsized":"2024-02-21T23:06:09.942207+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:27:54.695721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.23","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.935616+0000","last_change":"2024-02-21T23:06:09.935616+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.935616+0000","last_undegraded":"2024-02-21T23:06:09.935616+0000","last_fullsized":"2024-02-21T23:06:09.935616+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.919121+0000","last_change":"2024-02-21T23:05:51.421330+0000","last_active":"2024-02-21T23:06:09.919121+0000","last_peered":"2024-02-21T23:06:09.919121+0000","last_clean":"2024-02-21T23:06:09.919121+0000","last_became_active":"2024-02-21T23:05:51.421012+0000","last_became_peered":"2024-02-21T23:05:51.421012+0000","last_unstale":"2024-02-21T23:06:09.919121+0000","last_undegraded":"2024-02-21T23:06:09.919121+0000","last_fullsized":"2024-02-21T23:06:09.919121+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:26:33.366331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.22","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.941990+0000","last_change":"2024-02-21T23:06:09.941990+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.941990+0000","last_undegraded":"2024-02-21T23:06:09.941990+0000","last_fullsized":"2024-02-21T23:06:09.941990+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.942077+0000","last_change":"2024-02-21T23:05:51.403173+0000","last_active":"2024-02-21T23:06:09.942077+0000","last_peered":"2024-02-21T23:06:09.942077+0000","last_clean":"2024-02-21T23:06:09.942077+0000","last_became_active":"2024-02-21T23:05:51.402610+0000","last_became_peered":"2024-02-21T23:05:51.402610+0000","last_unstale":"2024-02-21T23:06:09.942077+0000","last_undegraded":"2024-02-21T23:06:09.942077+0000","last_fullsized":"2024-02-21T23:06:09.942077+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:46:01.608149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.940077+0000","last_change":"2024-02-21T23:06:09.940077+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.940077+0000","last_undegraded":"2024-02-21T23:06:09.940077+0000","last_fullsized":"2024-02-21T23:06:09.940077+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.957716+0000","last_change":"2024-02-21T23:06:09.955402+0000","last_active":"2024-02-21T23:06:09.957716+0000","last_peered":"2024-02-21T23:06:09.957716+0000","last_clean":"2024-02-21T23:06:09.957716+0000","last_became_active":"2024-02-21T23:06:09.955113+0000","last_became_peered":"2024-02-21T23:06:09.955113+0000","last_unstale":"2024-02-21T23:06:09.957716+0000","last_undegraded":"2024-02-21T23:06:09.957716+0000","last_fullsized":"2024-02-21T23:06:09.957716+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:19:07.034670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.918368+0000","last_change":"2024-02-21T23:05:51.416534+0000","last_active":"2024-02-21T23:06:09.918368+0000","last_peered":"2024-02-21T23:06:09.918368+0000","last_clean":"2024-02-21T23:06:09.918368+0000","last_became_active":"2024-02-21T23:05:51.416244+0000","last_became_peered":"2024-02-21T23:05:51.416244+0000","last_unstale":"2024-02-21T23:06:09.918368+0000","last_undegraded":"2024-02-21T23:06:09.918368+0000","last_fullsized":"2024-02-21T23:06:09.918368+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:42:37.006685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"75'4","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.930907+0000","last_change":"2024-02-21T23:05:50.354109+0000","last_active":"2024-02-21T23:06:09.930907+0000","last_peered":"2024-02-21T23:06:09.930907+0000","last_clean":"2024-02-21T23:06:09.930907+0000","last_became_active":"2024-02-21T23:05:50.353809+0000","last_became_peered":"2024-02-21T23:05:50.353809+0000","last_unstale":"2024-02-21T23:06:09.930907+0000","last_undegraded":"2024-02-21T23:06:09.930907+0000","last_fullsized":"2024-02-21T23:06:09.930907+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:28:13.765557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.926981+0000","last_change":"2024-02-21T23:06:09.926981+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.926981+0000","last_undegraded":"2024-02-21T23:06:09.926981+0000","last_fullsized":"2024-02-21T23:06:09.926981+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.966118+0000","last_change":"2024-02-21T23:06:09.963140+0000","last_active":"2024-02-21T23:06:09.966118+0000","last_peered":"2024-02-21T23:06:09.966118+0000","last_clean":"2024-02-21T23:06:09.966118+0000","last_became_active":"2024-02-21T23:06:09.962637+0000","last_became_peered":"2024-02-21T23:06:09.962637+0000","last_unstale":"2024-02-21T23:06:09.966118+0000","last_undegraded":"2024-02-21T23:06:09.966118+0000","last_fullsized":"2024-02-21T23:06:09.966118+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:42:45.049032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.943704+0000","last_change":"2024-02-21T23:05:51.416268+0000","last_active":"2024-02-21T23:06:09.943704+0000","last_peered":"2024-02-21T23:06:09.943704+0000","last_clean":"2024-02-21T23:06:09.943704+0000","last_became_active":"2024-02-21T23:05:51.415980+0000","last_became_peered":"2024-02-21T23:05:51.415980+0000","last_unstale":"2024-02-21T23:06:09.943704+0000","last_undegraded":"2024-02-21T23:06:09.943704+0000","last_fullsized":"2024-02-21T23:06:09.943704+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:51:17.193881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"75'1","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.946611+0000","last_change":"2024-02-21T23:05:50.383438+0000","last_active":"2024-02-21T23:06:09.946611+0000","last_peered":"2024-02-21T23:06:09.946611+0000","last_clean":"2024-02-21T23:06:09.946611+0000","last_became_active":"2024-02-21T23:05:50.383134+0000","last_became_peered":"2024-02-21T23:05:50.383134+0000","last_unstale":"2024-02-21T23:06:09.946611+0000","last_undegraded":"2024-02-21T23:06:09.946611+0000","last_fullsized":"2024-02-21T23:06:09.946611+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:08:35.448428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.927781+0000","last_change":"2024-02-21T23:06:09.927781+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.927781+0000","last_undegraded":"2024-02-21T23:06:09.927781+0000","last_fullsized":"2024-02-21T23:06:09.927781+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.960638+0000","last_change":"2024-02-21T23:06:09.957178+0000","last_active":"2024-02-21T23:06:09.960638+0000","last_peered":"2024-02-21T23:06:09.960638+0000","last_clean":"2024-02-21T23:06:09.960638+0000","last_became_active":"2024-02-21T23:06:09.956818+0000","last_became_peered":"2024-02-21T23:06:09.956818+0000","last_unstale":"2024-02-21T23:06:09.960638+0000","last_undegraded":"2024-02-21T23:06:09.960638+0000","last_fullsized":"2024-02-21T23:06:09.960638+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:01:46.942433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.946131+0000","last_change":"2024-02-21T23:05:51.421571+0000","last_active":"2024-02-21T23:06:09.946131+0000","last_peered":"2024-02-21T23:06:09.946131+0000","last_clean":"2024-02-21T23:06:09.946131+0000","last_became_active":"2024-02-21T23:05:51.421110+0000","last_became_peered":"2024-02-21T23:05:51.421110+0000","last_unstale":"2024-02-21T23:06:09.946131+0000","last_undegraded":"2024-02-21T23:06:09.946131+0000","last_fullsized":"2024-02-21T23:06:09.946131+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:56:36.688152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"75'2","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.944712+0000","last_change":"2024-02-21T23:05:50.383231+0000","last_active":"2024-02-21T23:06:09.944712+0000","last_peered":"2024-02-21T23:06:09.944712+0000","last_clean":"2024-02-21T23:06:09.944712+0000","last_became_active":"2024-02-21T23:05:50.382757+0000","last_became_peered":"2024-02-21T23:05:50.382757+0000","last_unstale":"2024-02-21T23:06:09.944712+0000","last_undegraded":"2024-02-21T23:06:09.944712+0000","last_fullsized":"2024-02-21T23:06:09.944712+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:49:36.738197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.926671+0000","last_change":"2024-02-21T23:06:09.926671+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.926671+0000","last_undegraded":"2024-02-21T23:06:09.926671+0000","last_fullsized":"2024-02-21T23:06:09.926671+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.953270+0000","last_change":"2024-02-21T23:06:09.951255+0000","last_active":"2024-02-21T23:06:09.953270+0000","last_peered":"2024-02-21T23:06:09.953270+0000","last_clean":"2024-02-21T23:06:09.953270+0000","last_became_active":"2024-02-21T23:06:09.950934+0000","last_became_peered":"2024-02-21T23:06:09.950934+0000","last_unstale":"2024-02-21T23:06:09.953270+0000","last_undegraded":"2024-02-21T23:06:09.953270+0000","last_fullsized":"2024-02-21T23:06:09.953270+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:53:46.723561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.948793+0000","last_change":"2024-02-21T23:05:51.415892+0000","last_active":"2024-02-21T23:06:09.948793+0000","last_peered":"2024-02-21T23:06:09.948793+0000","last_clean":"2024-02-21T23:06:09.948793+0000","last_became_active":"2024-02-21T23:05:51.415520+0000","last_became_peered":"2024-02-21T23:05:51.415520+0000","last_unstale":"2024-02-21T23:06:09.948793+0000","last_undegraded":"2024-02-21T23:06:09.948793+0000","last_fullsized":"2024-02-21T23:06:09.948793+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:26:33.366331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"75'2","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.932382+0000","last_change":"2024-02-21T23:05:50.376050+0000","last_active":"2024-02-21T23:06:09.932382+0000","last_peered":"2024-02-21T23:06:09.932382+0000","last_clean":"2024-02-21T23:06:09.932382+0000","last_became_active":"2024-02-21T23:05:50.375731+0000","last_became_peered":"2024-02-21T23:05:50.375731+0000","last_unstale":"2024-02-21T23:06:09.932382+0000","last_undegraded":"2024-02-21T23:06:09.932382+0000","last_fullsized":"2024-02-21T23:06:09.932382+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:42:15.626763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.937587+0000","last_change":"2024-02-21T23:06:09.937587+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.937587+0000","last_undegraded":"2024-02-21T23:06:09.937587+0000","last_fullsized":"2024-02-21T23:06:09.937587+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.960875+0000","last_change":"2024-02-21T23:06:09.957048+0000","last_active":"2024-02-21T23:06:09.960875+0000","last_peered":"2024-02-21T23:06:09.960875+0000","last_clean":"2024-02-21T23:06:09.960875+0000","last_became_active":"2024-02-21T23:06:09.956666+0000","last_became_peered":"2024-02-21T23:06:09.956666+0000","last_unstale":"2024-02-21T23:06:09.960875+0000","last_undegraded":"2024-02-21T23:06:09.960875+0000","last_fullsized":"2024-02-21T23:06:09.960875+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:11:15.443744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.949179+0000","last_change":"2024-02-21T23:05:51.428435+0000","last_active":"2024-02-21T23:06:09.949179+0000","last_peered":"2024-02-21T23:06:09.949179+0000","last_clean":"2024-02-21T23:06:09.949179+0000","last_became_active":"2024-02-21T23:05:51.428133+0000","last_became_peered":"2024-02-21T23:05:51.428133+0000","last_unstale":"2024-02-21T23:06:09.949179+0000","last_undegraded":"2024-02-21T23:06:09.949179+0000","last_fullsized":"2024-02-21T23:06:09.949179+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:49:46.025250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"75'1","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.948878+0000","last_change":"2024-02-21T23:05:50.377574+0000","last_active":"2024-02-21T23:06:09.948878+0000","last_peered":"2024-02-21T23:06:09.948878+0000","last_clean":"2024-02-21T23:06:09.948878+0000","last_became_active":"2024-02-21T23:05:50.377250+0000","last_became_peered":"2024-02-21T23:05:50.377250+0000","last_unstale":"2024-02-21T23:06:09.948878+0000","last_undegraded":"2024-02-21T23:06:09.948878+0000","last_fullsized":"2024-02-21T23:06:09.948878+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:26:38.366160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.937636+0000","last_change":"2024-02-21T23:06:09.937636+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.937636+0000","last_undegraded":"2024-02-21T23:06:09.937636+0000","last_fullsized":"2024-02-21T23:06:09.937636+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.954390+0000","last_change":"2024-02-21T23:06:09.947108+0000","last_active":"2024-02-21T23:06:09.954390+0000","last_peered":"2024-02-21T23:06:09.954390+0000","last_clean":"2024-02-21T23:06:09.954390+0000","last_became_active":"2024-02-21T23:06:09.946815+0000","last_became_peered":"2024-02-21T23:06:09.946815+0000","last_unstale":"2024-02-21T23:06:09.954390+0000","last_undegraded":"2024-02-21T23:06:09.954390+0000","last_fullsized":"2024-02-21T23:06:09.954390+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:56:08.310911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.951879+0000","last_change":"2024-02-21T23:05:51.421508+0000","last_active":"2024-02-21T23:06:09.951879+0000","last_peered":"2024-02-21T23:06:09.951879+0000","last_clean":"2024-02-21T23:06:09.951879+0000","last_became_active":"2024-02-21T23:05:51.420041+0000","last_became_peered":"2024-02-21T23:05:51.420041+0000","last_unstale":"2024-02-21T23:06:09.951879+0000","last_undegraded":"2024-02-21T23:06:09.951879+0000","last_fullsized":"2024-02-21T23:06:09.951879+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:10:56.883222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"75'1","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.933207+0000","last_change":"2024-02-21T23:05:50.378668+0000","last_active":"2024-02-21T23:06:09.933207+0000","last_peered":"2024-02-21T23:06:09.933207+0000","last_clean":"2024-02-21T23:06:09.933207+0000","last_became_active":"2024-02-21T23:05:50.378369+0000","last_became_peered":"2024-02-21T23:05:50.378369+0000","last_unstale":"2024-02-21T23:06:09.933207+0000","last_undegraded":"2024-02-21T23:06:09.933207+0000","last_fullsized":"2024-02-21T23:06:09.933207+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:45:05.837402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.943022+0000","last_change":"2024-02-21T23:06:09.943022+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.943022+0000","last_undegraded":"2024-02-21T23:06:09.943022+0000","last_fullsized":"2024-02-21T23:06:09.943022+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.955897+0000","last_change":"2024-02-21T23:06:09.949620+0000","last_active":"2024-02-21T23:06:09.955897+0000","last_peered":"2024-02-21T23:06:09.955897+0000","last_clean":"2024-02-21T23:06:09.955897+0000","last_became_active":"2024-02-21T23:06:09.949252+0000","last_became_peered":"2024-02-21T23:06:09.949252+0000","last_unstale":"2024-02-21T23:06:09.955897+0000","last_undegraded":"2024-02-21T23:06:09.955897+0000","last_fullsized":"2024-02-21T23:06:09.955897+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:51:46.950044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.951942+0000","last_change":"2024-02-21T23:05:51.419084+0000","last_active":"2024-02-21T23:06:09.951942+0000","last_peered":"2024-02-21T23:06:09.951942+0000","last_clean":"2024-02-21T23:06:09.951942+0000","last_became_active":"2024-02-21T23:05:51.418752+0000","last_became_peered":"2024-02-21T23:05:51.418752+0000","last_unstale":"2024-02-21T23:06:09.951942+0000","last_undegraded":"2024-02-21T23:06:09.951942+0000","last_fullsized":"2024-02-21T23:06:09.951942+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:20:19.564331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.943810+0000","last_change":"2024-02-21T23:05:50.375534+0000","last_active":"2024-02-21T23:06:09.943810+0000","last_peered":"2024-02-21T23:06:09.943810+0000","last_clean":"2024-02-21T23:06:09.943810+0000","last_became_active":"2024-02-21T23:05:50.373762+0000","last_became_peered":"2024-02-21T23:05:50.373762+0000","last_unstale":"2024-02-21T23:06:09.943810+0000","last_undegraded":"2024-02-21T23:06:09.943810+0000","last_fullsized":"2024-02-21T23:06:09.943810+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:49:36.738197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.941977+0000","last_change":"2024-02-21T23:06:09.941977+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.941977+0000","last_undegraded":"2024-02-21T23:06:09.941977+0000","last_fullsized":"2024-02-21T23:06:09.941977+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.957774+0000","last_change":"2024-02-21T23:06:09.952374+0000","last_active":"2024-02-21T23:06:09.957774+0000","last_peered":"2024-02-21T23:06:09.957774+0000","last_clean":"2024-02-21T23:06:09.957774+0000","last_became_active":"2024-02-21T23:06:09.950283+0000","last_became_peered":"2024-02-21T23:06:09.950283+0000","last_unstale":"2024-02-21T23:06:09.957774+0000","last_undegraded":"2024-02-21T23:06:09.957774+0000","last_fullsized":"2024-02-21T23:06:09.957774+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:27:17.649346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.935109+0000","last_change":"2024-02-21T23:05:51.426711+0000","last_active":"2024-02-21T23:06:09.935109+0000","last_peered":"2024-02-21T23:06:09.935109+0000","last_clean":"2024-02-21T23:06:09.935109+0000","last_became_active":"2024-02-21T23:05:51.426415+0000","last_became_peered":"2024-02-21T23:05:51.426415+0000","last_unstale":"2024-02-21T23:06:09.935109+0000","last_undegraded":"2024-02-21T23:06:09.935109+0000","last_fullsized":"2024-02-21T23:06:09.935109+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:58:55.829941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.931209+0000","last_change":"2024-02-21T23:05:50.355567+0000","last_active":"2024-02-21T23:06:09.931209+0000","last_peered":"2024-02-21T23:06:09.931209+0000","last_clean":"2024-02-21T23:06:09.931209+0000","last_became_active":"2024-02-21T23:05:50.354835+0000","last_became_peered":"2024-02-21T23:05:50.354835+0000","last_unstale":"2024-02-21T23:06:09.931209+0000","last_undegraded":"2024-02-21T23:06:09.931209+0000","last_fullsized":"2024-02-21T23:06:09.931209+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:21:52.655098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.936369+0000","last_change":"2024-02-21T23:06:09.936369+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.936369+0000","last_undegraded":"2024-02-21T23:06:09.936369+0000","last_fullsized":"2024-02-21T23:06:09.936369+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.950750+0000","last_change":"2024-02-21T23:06:09.946843+0000","last_active":"2024-02-21T23:06:09.950750+0000","last_peered":"2024-02-21T23:06:09.950750+0000","last_clean":"2024-02-21T23:06:09.950750+0000","last_became_active":"2024-02-21T23:06:09.946452+0000","last_became_peered":"2024-02-21T23:06:09.946452+0000","last_unstale":"2024-02-21T23:06:09.950750+0000","last_undegraded":"2024-02-21T23:06:09.950750+0000","last_fullsized":"2024-02-21T23:06:09.950750+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:22:35.988029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.944383+0000","last_change":"2024-02-21T23:05:51.399923+0000","last_active":"2024-02-21T23:06:09.944383+0000","last_peered":"2024-02-21T23:06:09.944383+0000","last_clean":"2024-02-21T23:06:09.944383+0000","last_became_active":"2024-02-21T23:05:51.399642+0000","last_became_peered":"2024-02-21T23:05:51.399642+0000","last_unstale":"2024-02-21T23:06:09.944383+0000","last_undegraded":"2024-02-21T23:06:09.944383+0000","last_fullsized":"2024-02-21T23:06:09.944383+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:21:18.440977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.946594+0000","last_change":"2024-02-21T23:05:50.386639+0000","last_active":"2024-02-21T23:06:09.946594+0000","last_peered":"2024-02-21T23:06:09.946594+0000","last_clean":"2024-02-21T23:06:09.946594+0000","last_became_active":"2024-02-21T23:05:50.386158+0000","last_became_peered":"2024-02-21T23:05:50.386158+0000","last_unstale":"2024-02-21T23:06:09.946594+0000","last_undegraded":"2024-02-21T23:06:09.946594+0000","last_fullsized":"2024-02-21T23:06:09.946594+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:43:17.496894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.932767+0000","last_change":"2024-02-21T23:06:09.932767+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.932767+0000","last_undegraded":"2024-02-21T23:06:09.932767+0000","last_fullsized":"2024-02-21T23:06:09.932767+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.943824+0000","last_change":"2024-02-21T23:06:09.934904+0000","last_active":"2024-02-21T23:06:09.943824+0000","last_peered":"2024-02-21T23:06:09.943824+0000","last_clean":"2024-02-21T23:06:09.943824+0000","last_became_active":"2024-02-21T23:06:09.934606+0000","last_became_peered":"2024-02-21T23:06:09.934606+0000","last_unstale":"2024-02-21T23:06:09.943824+0000","last_undegraded":"2024-02-21T23:06:09.943824+0000","last_fullsized":"2024-02-21T23:06:09.943824+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:43:37.070238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.932966+0000","last_change":"2024-02-21T23:05:51.422844+0000","last_active":"2024-02-21T23:06:09.932966+0000","last_peered":"2024-02-21T23:06:09.932966+0000","last_clean":"2024-02-21T23:06:09.932966+0000","last_became_active":"2024-02-21T23:05:51.422488+0000","last_became_peered":"2024-02-21T23:05:51.422488+0000","last_unstale":"2024-02-21T23:06:09.932966+0000","last_undegraded":"2024-02-21T23:06:09.932966+0000","last_fullsized":"2024-02-21T23:06:09.932966+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:27:54.695721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.934464+0000","last_change":"2024-02-21T23:05:50.385146+0000","last_active":"2024-02-21T23:06:09.934464+0000","last_peered":"2024-02-21T23:06:09.934464+0000","last_clean":"2024-02-21T23:06:09.934464+0000","last_became_active":"2024-02-21T23:05:50.384418+0000","last_became_peered":"2024-02-21T23:05:50.384418+0000","last_unstale":"2024-02-21T23:06:09.934464+0000","last_undegraded":"2024-02-21T23:06:09.934464+0000","last_fullsized":"2024-02-21T23:06:09.934464+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:43:17.496894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.927206+0000","last_change":"2024-02-21T23:06:09.927206+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.927206+0000","last_undegraded":"2024-02-21T23:06:09.927206+0000","last_fullsized":"2024-02-21T23:06:09.927206+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.966125+0000","last_change":"2024-02-21T23:06:09.963033+0000","last_active":"2024-02-21T23:06:09.966125+0000","last_peered":"2024-02-21T23:06:09.966125+0000","last_clean":"2024-02-21T23:06:09.966125+0000","last_became_active":"2024-02-21T23:06:09.962695+0000","last_became_peered":"2024-02-21T23:06:09.962695+0000","last_unstale":"2024-02-21T23:06:09.966125+0000","last_undegraded":"2024-02-21T23:06:09.966125+0000","last_fullsized":"2024-02-21T23:06:09.966125+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:36:42.352917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.946054+0000","last_change":"2024-02-21T23:05:51.418004+0000","last_active":"2024-02-21T23:06:09.946054+0000","last_peered":"2024-02-21T23:06:09.946054+0000","last_clean":"2024-02-21T23:06:09.946054+0000","last_became_active":"2024-02-21T23:05:51.417463+0000","last_became_peered":"2024-02-21T23:05:51.417463+0000","last_unstale":"2024-02-21T23:06:09.946054+0000","last_undegraded":"2024-02-21T23:06:09.946054+0000","last_fullsized":"2024-02-21T23:06:09.946054+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:59:13.292494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"75'3","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.932692+0000","last_change":"2024-02-21T23:05:50.373189+0000","last_active":"2024-02-21T23:06:09.932692+0000","last_peered":"2024-02-21T23:06:09.932692+0000","last_clean":"2024-02-21T23:06:09.932692+0000","last_became_active":"2024-02-21T23:05:50.370680+0000","last_became_peered":"2024-02-21T23:05:50.370680+0000","last_unstale":"2024-02-21T23:06:09.932692+0000","last_undegraded":"2024-02-21T23:06:09.932692+0000","last_fullsized":"2024-02-21T23:06:09.932692+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:21:03.121546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.939836+0000","last_change":"2024-02-21T23:06:09.939836+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.939836+0000","last_undegraded":"2024-02-21T23:06:09.939836+0000","last_fullsized":"2024-02-21T23:06:09.939836+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.963277+0000","last_change":"2024-02-21T23:06:09.961844+0000","last_active":"2024-02-21T23:06:09.963277+0000","last_peered":"2024-02-21T23:06:09.963277+0000","last_clean":"2024-02-21T23:06:09.963277+0000","last_became_active":"2024-02-21T23:06:09.961480+0000","last_became_peered":"2024-02-21T23:06:09.961480+0000","last_unstale":"2024-02-21T23:06:09.963277+0000","last_undegraded":"2024-02-21T23:06:09.963277+0000","last_fullsized":"2024-02-21T23:06:09.963277+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:04:07.940431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.919599+0000","last_change":"2024-02-21T23:05:51.405242+0000","last_active":"2024-02-21T23:06:09.919599+0000","last_peered":"2024-02-21T23:06:09.919599+0000","last_clean":"2024-02-21T23:06:09.919599+0000","last_became_active":"2024-02-21T23:05:51.403777+0000","last_became_peered":"2024-02-21T23:05:51.403777+0000","last_unstale":"2024-02-21T23:06:09.919599+0000","last_undegraded":"2024-02-21T23:06:09.919599+0000","last_fullsized":"2024-02-21T23:06:09.919599+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:34:14.420091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"75'6","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.932534+0000","last_change":"2024-02-21T23:05:50.374208+0000","last_active":"2024-02-21T23:06:09.932534+0000","last_peered":"2024-02-21T23:06:09.932534+0000","last_clean":"2024-02-21T23:06:09.932534+0000","last_became_active":"2024-02-21T23:05:50.373869+0000","last_became_peered":"2024-02-21T23:05:50.373869+0000","last_unstale":"2024-02-21T23:06:09.932534+0000","last_undegraded":"2024-02-21T23:06:09.932534+0000","last_fullsized":"2024-02-21T23:06:09.932534+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:56:49.435540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.938351+0000","last_change":"2024-02-21T23:06:09.938351+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.938351+0000","last_undegraded":"2024-02-21T23:06:09.938351+0000","last_fullsized":"2024-02-21T23:06:09.938351+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.966390+0000","last_change":"2024-02-21T23:06:09.963215+0000","last_active":"2024-02-21T23:06:09.966390+0000","last_peered":"2024-02-21T23:06:09.966390+0000","last_clean":"2024-02-21T23:06:09.966390+0000","last_became_active":"2024-02-21T23:06:09.962779+0000","last_became_peered":"2024-02-21T23:06:09.962779+0000","last_unstale":"2024-02-21T23:06:09.966390+0000","last_undegraded":"2024-02-21T23:06:09.966390+0000","last_fullsized":"2024-02-21T23:06:09.966390+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:45:25.410745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.935046+0000","last_change":"2024-02-21T23:05:51.423479+0000","last_active":"2024-02-21T23:06:09.935046+0000","last_peered":"2024-02-21T23:06:09.935046+0000","last_clean":"2024-02-21T23:06:09.935046+0000","last_became_active":"2024-02-21T23:05:51.423190+0000","last_became_peered":"2024-02-21T23:05:51.423190+0000","last_unstale":"2024-02-21T23:06:09.935046+0000","last_undegraded":"2024-02-21T23:06:09.935046+0000","last_fullsized":"2024-02-21T23:06:09.935046+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:03:08.789066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"75'2","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.933594+0000","last_change":"2024-02-21T23:05:50.381643+0000","last_active":"2024-02-21T23:06:09.933594+0000","last_peered":"2024-02-21T23:06:09.933594+0000","last_clean":"2024-02-21T23:06:09.933594+0000","last_became_active":"2024-02-21T23:05:50.381345+0000","last_became_peered":"2024-02-21T23:05:50.381345+0000","last_unstale":"2024-02-21T23:06:09.933594+0000","last_undegraded":"2024-02-21T23:06:09.933594+0000","last_fullsized":"2024-02-21T23:06:09.933594+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:01:27.369089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.927030+0000","last_change":"2024-02-21T23:06:09.927030+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.927030+0000","last_undegraded":"2024-02-21T23:06:09.927030+0000","last_fullsized":"2024-02-21T23:06:09.927030+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.962022+0000","last_change":"2024-02-21T23:06:09.959967+0000","last_active":"2024-02-21T23:06:09.962022+0000","last_peered":"2024-02-21T23:06:09.962022+0000","last_clean":"2024-02-21T23:06:09.962022+0000","last_became_active":"2024-02-21T23:06:09.959175+0000","last_became_peered":"2024-02-21T23:06:09.959175+0000","last_unstale":"2024-02-21T23:06:09.962022+0000","last_undegraded":"2024-02-21T23:06:09.962022+0000","last_fullsized":"2024-02-21T23:06:09.962022+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:41:49.821364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.937192+0000","last_change":"2024-02-21T23:05:51.416753+0000","last_active":"2024-02-21T23:06:09.937192+0000","last_peered":"2024-02-21T23:06:09.937192+0000","last_clean":"2024-02-21T23:06:09.937192+0000","last_became_active":"2024-02-21T23:05:51.416427+0000","last_became_peered":"2024-02-21T23:05:51.416427+0000","last_unstale":"2024-02-21T23:06:09.937192+0000","last_undegraded":"2024-02-21T23:06:09.937192+0000","last_fullsized":"2024-02-21T23:06:09.937192+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:34:14.420091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.942362+0000","last_change":"2024-02-21T23:05:50.381962+0000","last_active":"2024-02-21T23:06:09.942362+0000","last_peered":"2024-02-21T23:06:09.942362+0000","last_clean":"2024-02-21T23:06:09.942362+0000","last_became_active":"2024-02-21T23:05:50.380838+0000","last_became_peered":"2024-02-21T23:05:50.380838+0000","last_unstale":"2024-02-21T23:06:09.942362+0000","last_undegraded":"2024-02-21T23:06:09.942362+0000","last_fullsized":"2024-02-21T23:06:09.942362+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:10:10.973567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.939210+0000","last_change":"2024-02-21T23:06:09.939210+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.939210+0000","last_undegraded":"2024-02-21T23:06:09.939210+0000","last_fullsized":"2024-02-21T23:06:09.939210+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.958221+0000","last_change":"2024-02-21T23:06:09.951230+0000","last_active":"2024-02-21T23:06:09.958221+0000","last_peered":"2024-02-21T23:06:09.958221+0000","last_clean":"2024-02-21T23:06:09.958221+0000","last_became_active":"2024-02-21T23:06:09.950920+0000","last_became_peered":"2024-02-21T23:06:09.950920+0000","last_unstale":"2024-02-21T23:06:09.958221+0000","last_undegraded":"2024-02-21T23:06:09.958221+0000","last_fullsized":"2024-02-21T23:06:09.958221+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:11:15.443744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.934193+0000","last_change":"2024-02-21T23:05:51.408507+0000","last_active":"2024-02-21T23:06:09.934193+0000","last_peered":"2024-02-21T23:06:09.934193+0000","last_clean":"2024-02-21T23:06:09.934193+0000","last_became_active":"2024-02-21T23:05:51.408198+0000","last_became_peered":"2024-02-21T23:05:51.408198+0000","last_unstale":"2024-02-21T23:06:09.934193+0000","last_undegraded":"2024-02-21T23:06:09.934193+0000","last_fullsized":"2024-02-21T23:06:09.934193+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:08:36.461250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.934108+0000","last_change":"2024-02-21T23:05:50.353116+0000","last_active":"2024-02-21T23:06:09.934108+0000","last_peered":"2024-02-21T23:06:09.934108+0000","last_clean":"2024-02-21T23:06:09.934108+0000","last_became_active":"2024-02-21T23:05:50.352768+0000","last_became_peered":"2024-02-21T23:05:50.352768+0000","last_unstale":"2024-02-21T23:06:09.934108+0000","last_undegraded":"2024-02-21T23:06:09.934108+0000","last_fullsized":"2024-02-21T23:06:09.934108+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:52:09.073053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.937280+0000","last_change":"2024-02-21T23:06:09.937280+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.937280+0000","last_undegraded":"2024-02-21T23:06:09.937280+0000","last_fullsized":"2024-02-21T23:06:09.937280+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.955574+0000","last_change":"2024-02-21T23:06:09.954937+0000","last_active":"2024-02-21T23:06:09.955574+0000","last_peered":"2024-02-21T23:06:09.955574+0000","last_clean":"2024-02-21T23:06:09.955574+0000","last_became_active":"2024-02-21T23:06:09.953847+0000","last_became_peered":"2024-02-21T23:06:09.953847+0000","last_unstale":"2024-02-21T23:06:09.955574+0000","last_undegraded":"2024-02-21T23:06:09.955574+0000","last_fullsized":"2024-02-21T23:06:09.955574+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:17:14.119658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.936892+0000","last_change":"2024-02-21T23:05:51.416424+0000","last_active":"2024-02-21T23:06:09.936892+0000","last_peered":"2024-02-21T23:06:09.936892+0000","last_clean":"2024-02-21T23:06:09.936892+0000","last_became_active":"2024-02-21T23:05:51.416070+0000","last_became_peered":"2024-02-21T23:05:51.416070+0000","last_unstale":"2024-02-21T23:06:09.936892+0000","last_undegraded":"2024-02-21T23:06:09.936892+0000","last_fullsized":"2024-02-21T23:06:09.936892+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:56:10.406772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"75'8","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.947720+0000","last_change":"2024-02-21T23:05:50.376063+0000","last_active":"2024-02-21T23:06:09.947720+0000","last_peered":"2024-02-21T23:06:09.947720+0000","last_clean":"2024-02-21T23:06:09.947720+0000","last_became_active":"2024-02-21T23:05:50.375772+0000","last_became_peered":"2024-02-21T23:05:50.375772+0000","last_unstale":"2024-02-21T23:06:09.947720+0000","last_undegraded":"2024-02-21T23:06:09.947720+0000","last_fullsized":"2024-02-21T23:06:09.947720+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:26:58.076002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.923016+0000","last_change":"2024-02-21T23:06:09.923016+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.923016+0000","last_undegraded":"2024-02-21T23:06:09.923016+0000","last_fullsized":"2024-02-21T23:06:09.923016+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.958286+0000","last_change":"2024-02-21T23:06:09.951762+0000","last_active":"2024-02-21T23:06:09.958286+0000","last_peered":"2024-02-21T23:06:09.958286+0000","last_clean":"2024-02-21T23:06:09.958286+0000","last_became_active":"2024-02-21T23:06:09.951155+0000","last_became_peered":"2024-02-21T23:06:09.951155+0000","last_unstale":"2024-02-21T23:06:09.958286+0000","last_undegraded":"2024-02-21T23:06:09.958286+0000","last_fullsized":"2024-02-21T23:06:09.958286+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:36:42.352917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.942415+0000","last_change":"2024-02-21T23:05:51.421565+0000","last_active":"2024-02-21T23:06:09.942415+0000","last_peered":"2024-02-21T23:06:09.942415+0000","last_clean":"2024-02-21T23:06:09.942415+0000","last_became_active":"2024-02-21T23:05:51.421252+0000","last_became_peered":"2024-02-21T23:05:51.421252+0000","last_unstale":"2024-02-21T23:06:09.942415+0000","last_undegraded":"2024-02-21T23:06:09.942415+0000","last_fullsized":"2024-02-21T23:06:09.942415+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:40:45.445418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"75'1","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.946704+0000","last_change":"2024-02-21T23:05:50.383778+0000","last_active":"2024-02-21T23:06:09.946704+0000","last_peered":"2024-02-21T23:06:09.946704+0000","last_clean":"2024-02-21T23:06:09.946704+0000","last_became_active":"2024-02-21T23:05:50.383436+0000","last_became_peered":"2024-02-21T23:05:50.383436+0000","last_unstale":"2024-02-21T23:06:09.946704+0000","last_undegraded":"2024-02-21T23:06:09.946704+0000","last_fullsized":"2024-02-21T23:06:09.946704+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:45:05.837402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.934671+0000","last_change":"2024-02-21T23:06:09.934671+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.934671+0000","last_undegraded":"2024-02-21T23:06:09.934671+0000","last_fullsized":"2024-02-21T23:06:09.934671+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.956162+0000","last_change":"2024-02-21T23:06:09.947681+0000","last_active":"2024-02-21T23:06:09.956162+0000","last_peered":"2024-02-21T23:06:09.956162+0000","last_clean":"2024-02-21T23:06:09.956162+0000","last_became_active":"2024-02-21T23:06:09.947370+0000","last_became_peered":"2024-02-21T23:06:09.947370+0000","last_unstale":"2024-02-21T23:06:09.956162+0000","last_undegraded":"2024-02-21T23:06:09.956162+0000","last_fullsized":"2024-02-21T23:06:09.956162+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:01:37.993072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.934009+0000","last_change":"2024-02-21T23:05:51.416676+0000","last_active":"2024-02-21T23:06:09.934009+0000","last_peered":"2024-02-21T23:06:09.934009+0000","last_clean":"2024-02-21T23:06:09.934009+0000","last_became_active":"2024-02-21T23:05:51.416297+0000","last_became_peered":"2024-02-21T23:05:51.416297+0000","last_unstale":"2024-02-21T23:06:09.934009+0000","last_undegraded":"2024-02-21T23:06:09.934009+0000","last_fullsized":"2024-02-21T23:06:09.934009+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:29:33.209545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"75'2","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.952053+0000","last_change":"2024-02-21T23:05:50.377181+0000","last_active":"2024-02-21T23:06:09.952053+0000","last_peered":"2024-02-21T23:06:09.952053+0000","last_clean":"2024-02-21T23:06:09.952053+0000","last_became_active":"2024-02-21T23:05:50.373524+0000","last_became_peered":"2024-02-21T23:05:50.373524+0000","last_unstale":"2024-02-21T23:06:09.952053+0000","last_undegraded":"2024-02-21T23:06:09.952053+0000","last_fullsized":"2024-02-21T23:06:09.952053+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:01:18.419728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.929958+0000","last_change":"2024-02-21T23:06:09.929958+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.929958+0000","last_undegraded":"2024-02-21T23:06:09.929958+0000","last_fullsized":"2024-02-21T23:06:09.929958+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.961219+0000","last_change":"2024-02-21T23:06:09.955696+0000","last_active":"2024-02-21T23:06:09.961219+0000","last_peered":"2024-02-21T23:06:09.961219+0000","last_clean":"2024-02-21T23:06:09.961219+0000","last_became_active":"2024-02-21T23:06:09.954715+0000","last_became_peered":"2024-02-21T23:06:09.954715+0000","last_unstale":"2024-02-21T23:06:09.961219+0000","last_undegraded":"2024-02-21T23:06:09.961219+0000","last_fullsized":"2024-02-21T23:06:09.961219+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:01:35.517150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.937478+0000","last_change":"2024-02-21T23:05:51.412971+0000","last_active":"2024-02-21T23:06:09.937478+0000","last_peered":"2024-02-21T23:06:09.937478+0000","last_clean":"2024-02-21T23:06:09.937478+0000","last_became_active":"2024-02-21T23:05:51.410966+0000","last_became_peered":"2024-02-21T23:05:51.410966+0000","last_unstale":"2024-02-21T23:06:09.937478+0000","last_undegraded":"2024-02-21T23:06:09.937478+0000","last_fullsized":"2024-02-21T23:06:09.937478+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:00:39.163092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"75'1","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.938268+0000","last_change":"2024-02-21T23:05:50.366307+0000","last_active":"2024-02-21T23:06:09.938268+0000","last_peered":"2024-02-21T23:06:09.938268+0000","last_clean":"2024-02-21T23:06:09.938268+0000","last_became_active":"2024-02-21T23:05:50.359426+0000","last_became_peered":"2024-02-21T23:05:50.359426+0000","last_unstale":"2024-02-21T23:06:09.938268+0000","last_undegraded":"2024-02-21T23:06:09.938268+0000","last_fullsized":"2024-02-21T23:06:09.938268+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:13:15.770526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.925807+0000","last_change":"2024-02-21T23:06:09.925807+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.925807+0000","last_undegraded":"2024-02-21T23:06:09.925807+0000","last_fullsized":"2024-02-21T23:06:09.925807+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.956240+0000","last_change":"2024-02-21T23:06:09.948670+0000","last_active":"2024-02-21T23:06:09.956240+0000","last_peered":"2024-02-21T23:06:09.956240+0000","last_clean":"2024-02-21T23:06:09.956240+0000","last_became_active":"2024-02-21T23:06:09.946109+0000","last_became_peered":"2024-02-21T23:06:09.946109+0000","last_unstale":"2024-02-21T23:06:09.956240+0000","last_undegraded":"2024-02-21T23:06:09.956240+0000","last_fullsized":"2024-02-21T23:06:09.956240+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:26:57.939504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.947158+0000","last_change":"2024-02-21T23:05:51.424067+0000","last_active":"2024-02-21T23:06:09.947158+0000","last_peered":"2024-02-21T23:06:09.947158+0000","last_clean":"2024-02-21T23:06:09.947158+0000","last_became_active":"2024-02-21T23:05:51.423644+0000","last_became_peered":"2024-02-21T23:05:51.423644+0000","last_unstale":"2024-02-21T23:06:09.947158+0000","last_undegraded":"2024-02-21T23:06:09.947158+0000","last_fullsized":"2024-02-21T23:06:09.947158+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:04:03.084512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.919750+0000","last_change":"2024-02-21T23:05:50.377924+0000","last_active":"2024-02-21T23:06:09.919750+0000","last_peered":"2024-02-21T23:06:09.919750+0000","last_clean":"2024-02-21T23:06:09.919750+0000","last_became_active":"2024-02-21T23:05:50.377440+0000","last_became_peered":"2024-02-21T23:05:50.377440+0000","last_unstale":"2024-02-21T23:06:09.919750+0000","last_undegraded":"2024-02-21T23:06:09.919750+0000","last_fullsized":"2024-02-21T23:06:09.919750+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:42:15.626763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.938677+0000","last_change":"2024-02-21T23:06:09.938677+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.938677+0000","last_undegraded":"2024-02-21T23:06:09.938677+0000","last_fullsized":"2024-02-21T23:06:09.938677+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.957820+0000","last_change":"2024-02-21T23:06:09.953285+0000","last_active":"2024-02-21T23:06:09.957820+0000","last_peered":"2024-02-21T23:06:09.957820+0000","last_clean":"2024-02-21T23:06:09.957820+0000","last_became_active":"2024-02-21T23:06:09.952940+0000","last_became_peered":"2024-02-21T23:06:09.952940+0000","last_unstale":"2024-02-21T23:06:09.957820+0000","last_undegraded":"2024-02-21T23:06:09.957820+0000","last_fullsized":"2024-02-21T23:06:09.957820+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:49:56.311541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.949283+0000","last_change":"2024-02-21T23:05:51.414619+0000","last_active":"2024-02-21T23:06:09.949283+0000","last_peered":"2024-02-21T23:06:09.949283+0000","last_clean":"2024-02-21T23:06:09.949283+0000","last_became_active":"2024-02-21T23:05:51.414059+0000","last_became_peered":"2024-02-21T23:05:51.414059+0000","last_unstale":"2024-02-21T23:06:09.949283+0000","last_undegraded":"2024-02-21T23:06:09.949283+0000","last_fullsized":"2024-02-21T23:06:09.949283+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:58:05.114297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"75'2","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.947250+0000","last_change":"2024-02-21T23:05:50.366877+0000","last_active":"2024-02-21T23:06:09.947250+0000","last_peered":"2024-02-21T23:06:09.947250+0000","last_clean":"2024-02-21T23:06:09.947250+0000","last_became_active":"2024-02-21T23:05:50.365566+0000","last_became_peered":"2024-02-21T23:05:50.365566+0000","last_unstale":"2024-02-21T23:06:09.947250+0000","last_undegraded":"2024-02-21T23:06:09.947250+0000","last_fullsized":"2024-02-21T23:06:09.947250+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:58:04.101475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.941480+0000","last_change":"2024-02-21T23:06:09.941480+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.941480+0000","last_undegraded":"2024-02-21T23:06:09.941480+0000","last_fullsized":"2024-02-21T23:06:09.941480+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.954312+0000","last_change":"2024-02-21T23:06:09.945701+0000","last_active":"2024-02-21T23:06:09.954312+0000","last_peered":"2024-02-21T23:06:09.954312+0000","last_clean":"2024-02-21T23:06:09.954312+0000","last_became_active":"2024-02-21T23:06:09.945391+0000","last_became_peered":"2024-02-21T23:06:09.945391+0000","last_unstale":"2024-02-21T23:06:09.954312+0000","last_undegraded":"2024-02-21T23:06:09.954312+0000","last_fullsized":"2024-02-21T23:06:09.954312+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:53:46.723561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.944153+0000","last_change":"2024-02-21T23:05:51.411005+0000","last_active":"2024-02-21T23:06:09.944153+0000","last_peered":"2024-02-21T23:06:09.944153+0000","last_clean":"2024-02-21T23:06:09.944153+0000","last_became_active":"2024-02-21T23:05:51.410503+0000","last_became_peered":"2024-02-21T23:05:51.410503+0000","last_unstale":"2024-02-21T23:06:09.944153+0000","last_undegraded":"2024-02-21T23:06:09.944153+0000","last_fullsized":"2024-02-21T23:06:09.944153+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:51:10.610852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.944912+0000","last_change":"2024-02-21T23:05:50.376888+0000","last_active":"2024-02-21T23:06:09.944912+0000","last_peered":"2024-02-21T23:06:09.944912+0000","last_clean":"2024-02-21T23:06:09.944912+0000","last_became_active":"2024-02-21T23:05:50.376610+0000","last_became_peered":"2024-02-21T23:05:50.376610+0000","last_unstale":"2024-02-21T23:06:09.944912+0000","last_undegraded":"2024-02-21T23:06:09.944912+0000","last_fullsized":"2024-02-21T23:06:09.944912+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:51:16.181060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.925072+0000","last_change":"2024-02-21T23:06:09.925072+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.925072+0000","last_undegraded":"2024-02-21T23:06:09.925072+0000","last_fullsized":"2024-02-21T23:06:09.925072+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.958272+0000","last_change":"2024-02-21T23:06:09.952555+0000","last_active":"2024-02-21T23:06:09.958272+0000","last_peered":"2024-02-21T23:06:09.958272+0000","last_clean":"2024-02-21T23:06:09.958272+0000","last_became_active":"2024-02-21T23:06:09.952281+0000","last_became_peered":"2024-02-21T23:06:09.952281+0000","last_unstale":"2024-02-21T23:06:09.958272+0000","last_undegraded":"2024-02-21T23:06:09.958272+0000","last_fullsized":"2024-02-21T23:06:09.958272+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:50:04.585772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.937015+0000","last_change":"2024-02-21T23:05:51.417167+0000","last_active":"2024-02-21T23:06:09.937015+0000","last_peered":"2024-02-21T23:06:09.937015+0000","last_clean":"2024-02-21T23:06:09.937015+0000","last_became_active":"2024-02-21T23:05:51.415957+0000","last_became_peered":"2024-02-21T23:05:51.415957+0000","last_unstale":"2024-02-21T23:06:09.937015+0000","last_undegraded":"2024-02-21T23:06:09.937015+0000","last_fullsized":"2024-02-21T23:06:09.937015+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:26:59.088824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.942505+0000","last_change":"2024-02-21T23:05:50.375808+0000","last_active":"2024-02-21T23:06:09.942505+0000","last_peered":"2024-02-21T23:06:09.942505+0000","last_clean":"2024-02-21T23:06:09.942505+0000","last_became_active":"2024-02-21T23:05:50.375467+0000","last_became_peered":"2024-02-21T23:05:50.375467+0000","last_unstale":"2024-02-21T23:06:09.942505+0000","last_undegraded":"2024-02-21T23:06:09.942505+0000","last_fullsized":"2024-02-21T23:06:09.942505+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:29:01.979463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.923630+0000","last_change":"2024-02-21T23:06:09.923630+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.923630+0000","last_undegraded":"2024-02-21T23:06:09.923630+0000","last_fullsized":"2024-02-21T23:06:09.923630+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.959070+0000","last_change":"2024-02-21T23:06:09.956377+0000","last_active":"2024-02-21T23:06:09.959070+0000","last_peered":"2024-02-21T23:06:09.959070+0000","last_clean":"2024-02-21T23:06:09.959070+0000","last_became_active":"2024-02-21T23:06:09.956129+0000","last_became_peered":"2024-02-21T23:06:09.956129+0000","last_unstale":"2024-02-21T23:06:09.959070+0000","last_undegraded":"2024-02-21T23:06:09.959070+0000","last_fullsized":"2024-02-21T23:06:09.959070+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:29:15.656090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.946180+0000","last_change":"2024-02-21T23:05:51.414453+0000","last_active":"2024-02-21T23:06:09.946180+0000","last_peered":"2024-02-21T23:06:09.946180+0000","last_clean":"2024-02-21T23:06:09.946180+0000","last_became_active":"2024-02-21T23:05:51.413499+0000","last_became_peered":"2024-02-21T23:05:51.413499+0000","last_unstale":"2024-02-21T23:06:09.946180+0000","last_undegraded":"2024-02-21T23:06:09.946180+0000","last_fullsized":"2024-02-21T23:06:09.946180+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:55:21.967274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"75'2","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.934245+0000","last_change":"2024-02-21T23:05:50.381948+0000","last_active":"2024-02-21T23:06:09.934245+0000","last_peered":"2024-02-21T23:06:09.934245+0000","last_clean":"2024-02-21T23:06:09.934245+0000","last_became_active":"2024-02-21T23:05:50.381452+0000","last_became_peered":"2024-02-21T23:05:50.381452+0000","last_unstale":"2024-02-21T23:06:09.934245+0000","last_undegraded":"2024-02-21T23:06:09.934245+0000","last_fullsized":"2024-02-21T23:06:09.934245+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:22:16.414686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.935470+0000","last_change":"2024-02-21T23:06:09.935470+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.935470+0000","last_undegraded":"2024-02-21T23:06:09.935470+0000","last_fullsized":"2024-02-21T23:06:09.935470+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.958416+0000","last_change":"2024-02-21T23:06:09.953924+0000","last_active":"2024-02-21T23:06:09.958416+0000","last_peered":"2024-02-21T23:06:09.958416+0000","last_clean":"2024-02-21T23:06:09.958416+0000","last_became_active":"2024-02-21T23:06:09.952952+0000","last_became_peered":"2024-02-21T23:06:09.952952+0000","last_unstale":"2024-02-21T23:06:09.958416+0000","last_undegraded":"2024-02-21T23:06:09.958416+0000","last_fullsized":"2024-02-21T23:06:09.958416+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:19:59.841109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.934595+0000","last_change":"2024-02-21T23:05:51.410619+0000","last_active":"2024-02-21T23:06:09.934595+0000","last_peered":"2024-02-21T23:06:09.934595+0000","last_clean":"2024-02-21T23:06:09.934595+0000","last_became_active":"2024-02-21T23:05:51.409937+0000","last_became_peered":"2024-02-21T23:05:51.409937+0000","last_unstale":"2024-02-21T23:06:09.934595+0000","last_undegraded":"2024-02-21T23:06:09.934595+0000","last_fullsized":"2024-02-21T23:06:09.934595+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:22:32.845173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"76'1","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.944263+0000","last_change":"2024-02-21T23:05:50.362370+0000","last_active":"2024-02-21T23:06:09.944263+0000","last_peered":"2024-02-21T23:06:09.944263+0000","last_clean":"2024-02-21T23:06:09.944263+0000","last_became_active":"2024-02-21T23:05:50.361848+0000","last_became_peered":"2024-02-21T23:05:50.361848+0000","last_unstale":"2024-02-21T23:06:09.944263+0000","last_undegraded":"2024-02-21T23:06:09.944263+0000","last_fullsized":"2024-02-21T23:06:09.944263+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:08:35.448428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.930610+0000","last_change":"2024-02-21T23:06:09.930610+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.930610+0000","last_undegraded":"2024-02-21T23:06:09.930610+0000","last_fullsized":"2024-02-21T23:06:09.930610+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.963409+0000","last_change":"2024-02-21T23:06:09.961946+0000","last_active":"2024-02-21T23:06:09.963409+0000","last_peered":"2024-02-21T23:06:09.963409+0000","last_clean":"2024-02-21T23:06:09.963409+0000","last_became_active":"2024-02-21T23:06:09.961662+0000","last_became_peered":"2024-02-21T23:06:09.961662+0000","last_unstale":"2024-02-21T23:06:09.963409+0000","last_undegraded":"2024-02-21T23:06:09.963409+0000","last_fullsized":"2024-02-21T23:06:09.963409+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:15:58.947158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.946318+0000","last_change":"2024-02-21T23:05:51.421357+0000","last_active":"2024-02-21T23:06:09.946318+0000","last_peered":"2024-02-21T23:06:09.946318+0000","last_clean":"2024-02-21T23:06:09.946318+0000","last_became_active":"2024-02-21T23:05:51.420734+0000","last_became_peered":"2024-02-21T23:05:51.420734+0000","last_unstale":"2024-02-21T23:06:09.946318+0000","last_undegraded":"2024-02-21T23:06:09.946318+0000","last_fullsized":"2024-02-21T23:06:09.946318+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:31:44.536584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.944953+0000","last_change":"2024-02-21T23:05:50.384680+0000","last_active":"2024-02-21T23:06:09.944953+0000","last_peered":"2024-02-21T23:06:09.944953+0000","last_clean":"2024-02-21T23:06:09.944953+0000","last_became_active":"2024-02-21T23:05:50.384208+0000","last_became_peered":"2024-02-21T23:05:50.384208+0000","last_unstale":"2024-02-21T23:06:09.944953+0000","last_undegraded":"2024-02-21T23:06:09.944953+0000","last_fullsized":"2024-02-21T23:06:09.944953+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:01:27.369089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.933769+0000","last_change":"2024-02-21T23:06:09.933769+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.933769+0000","last_undegraded":"2024-02-21T23:06:09.933769+0000","last_fullsized":"2024-02-21T23:06:09.933769+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.961163+0000","last_change":"2024-02-21T23:06:09.952597+0000","last_active":"2024-02-21T23:06:09.961163+0000","last_peered":"2024-02-21T23:06:09.961163+0000","last_clean":"2024-02-21T23:06:09.961163+0000","last_became_active":"2024-02-21T23:06:09.952270+0000","last_became_peered":"2024-02-21T23:06:09.952270+0000","last_unstale":"2024-02-21T23:06:09.961163+0000","last_undegraded":"2024-02-21T23:06:09.961163+0000","last_fullsized":"2024-02-21T23:06:09.961163+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:42:45.049032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.944384+0000","last_change":"2024-02-21T23:05:51.395859+0000","last_active":"2024-02-21T23:06:09.944384+0000","last_peered":"2024-02-21T23:06:09.944384+0000","last_clean":"2024-02-21T23:06:09.944384+0000","last_became_active":"2024-02-21T23:05:51.395012+0000","last_became_peered":"2024-02-21T23:05:51.395012+0000","last_unstale":"2024-02-21T23:06:09.944384+0000","last_undegraded":"2024-02-21T23:06:09.944384+0000","last_fullsized":"2024-02-21T23:06:09.944384+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:29:02.992284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.943897+0000","last_change":"2024-02-21T23:05:50.383920+0000","last_active":"2024-02-21T23:06:09.943897+0000","last_peered":"2024-02-21T23:06:09.943897+0000","last_clean":"2024-02-21T23:06:09.943897+0000","last_became_active":"2024-02-21T23:05:50.382488+0000","last_became_peered":"2024-02-21T23:05:50.382488+0000","last_unstale":"2024-02-21T23:06:09.943897+0000","last_undegraded":"2024-02-21T23:06:09.943897+0000","last_fullsized":"2024-02-21T23:06:09.943897+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:26:58.076002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.937602+0000","last_change":"2024-02-21T23:06:09.937602+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.937602+0000","last_undegraded":"2024-02-21T23:06:09.937602+0000","last_fullsized":"2024-02-21T23:06:09.937602+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.966343+0000","last_change":"2024-02-21T23:06:09.961904+0000","last_active":"2024-02-21T23:06:09.966343+0000","last_peered":"2024-02-21T23:06:09.966343+0000","last_clean":"2024-02-21T23:06:09.966343+0000","last_became_active":"2024-02-21T23:06:09.961650+0000","last_became_peered":"2024-02-21T23:06:09.961650+0000","last_unstale":"2024-02-21T23:06:09.966343+0000","last_undegraded":"2024-02-21T23:06:09.966343+0000","last_fullsized":"2024-02-21T23:06:09.966343+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:43:37.070238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.932210+0000","last_change":"2024-02-21T23:05:51.422495+0000","last_active":"2024-02-21T23:06:09.932210+0000","last_peered":"2024-02-21T23:06:09.932210+0000","last_clean":"2024-02-21T23:06:09.932210+0000","last_became_active":"2024-02-21T23:05:51.422196+0000","last_became_peered":"2024-02-21T23:05:51.422196+0000","last_unstale":"2024-02-21T23:06:09.932210+0000","last_undegraded":"2024-02-21T23:06:09.932210+0000","last_fullsized":"2024-02-21T23:06:09.932210+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:10:11.986389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.932846+0000","last_change":"2024-02-21T23:05:50.376027+0000","last_active":"2024-02-21T23:06:09.932846+0000","last_peered":"2024-02-21T23:06:09.932846+0000","last_clean":"2024-02-21T23:06:09.932846+0000","last_became_active":"2024-02-21T23:05:50.375170+0000","last_became_peered":"2024-02-21T23:05:50.375170+0000","last_unstale":"2024-02-21T23:06:09.932846+0000","last_undegraded":"2024-02-21T23:06:09.932846+0000","last_fullsized":"2024-02-21T23:06:09.932846+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:39:32.228635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.924758+0000","last_change":"2024-02-21T23:06:09.924758+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.924758+0000","last_undegraded":"2024-02-21T23:06:09.924758+0000","last_fullsized":"2024-02-21T23:06:09.924758+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.954412+0000","last_change":"2024-02-21T23:06:09.947879+0000","last_active":"2024-02-21T23:06:09.954412+0000","last_peered":"2024-02-21T23:06:09.954412+0000","last_clean":"2024-02-21T23:06:09.954412+0000","last_became_active":"2024-02-21T23:06:09.947599+0000","last_became_peered":"2024-02-21T23:06:09.947599+0000","last_unstale":"2024-02-21T23:06:09.954412+0000","last_undegraded":"2024-02-21T23:06:09.954412+0000","last_fullsized":"2024-02-21T23:06:09.954412+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:50:51.483608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.937296+0000","last_change":"2024-02-21T23:05:51.420500+0000","last_active":"2024-02-21T23:06:09.937296+0000","last_peered":"2024-02-21T23:06:09.937296+0000","last_clean":"2024-02-21T23:06:09.937296+0000","last_became_active":"2024-02-21T23:05:51.420103+0000","last_became_peered":"2024-02-21T23:05:51.420103+0000","last_unstale":"2024-02-21T23:06:09.937296+0000","last_undegraded":"2024-02-21T23:06:09.937296+0000","last_fullsized":"2024-02-21T23:06:09.937296+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:27:01.191337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.937594+0000","last_change":"2024-02-21T23:05:50.354805+0000","last_active":"2024-02-21T23:06:09.937594+0000","last_peered":"2024-02-21T23:06:09.937594+0000","last_clean":"2024-02-21T23:06:09.937594+0000","last_became_active":"2024-02-21T23:05:50.354525+0000","last_became_peered":"2024-02-21T23:05:50.354525+0000","last_unstale":"2024-02-21T23:06:09.937594+0000","last_undegraded":"2024-02-21T23:06:09.937594+0000","last_fullsized":"2024-02-21T23:06:09.937594+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:28:05.314552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.923982+0000","last_change":"2024-02-21T23:06:09.923982+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.923982+0000","last_undegraded":"2024-02-21T23:06:09.923982+0000","last_fullsized":"2024-02-21T23:06:09.923982+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.958392+0000","last_change":"2024-02-21T23:06:09.955164+0000","last_active":"2024-02-21T23:06:09.958392+0000","last_peered":"2024-02-21T23:06:09.958392+0000","last_clean":"2024-02-21T23:06:09.958392+0000","last_became_active":"2024-02-21T23:06:09.954858+0000","last_became_peered":"2024-02-21T23:06:09.954858+0000","last_unstale":"2024-02-21T23:06:09.958392+0000","last_undegraded":"2024-02-21T23:06:09.958392+0000","last_fullsized":"2024-02-21T23:06:09.958392+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:51:29.171374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.920149+0000","last_change":"2024-02-21T23:05:51.416548+0000","last_active":"2024-02-21T23:06:09.920149+0000","last_peered":"2024-02-21T23:06:09.920149+0000","last_clean":"2024-02-21T23:06:09.920149+0000","last_became_active":"2024-02-21T23:05:51.416222+0000","last_became_peered":"2024-02-21T23:05:51.416222+0000","last_unstale":"2024-02-21T23:06:09.920149+0000","last_undegraded":"2024-02-21T23:06:09.920149+0000","last_fullsized":"2024-02-21T23:06:09.920149+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:22:32.845173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.934813+0000","last_change":"2024-02-21T23:05:50.373554+0000","last_active":"2024-02-21T23:06:09.934813+0000","last_peered":"2024-02-21T23:06:09.934813+0000","last_clean":"2024-02-21T23:06:09.934813+0000","last_became_active":"2024-02-21T23:05:50.373022+0000","last_became_peered":"2024-02-21T23:05:50.373022+0000","last_unstale":"2024-02-21T23:06:09.934813+0000","last_undegraded":"2024-02-21T23:06:09.934813+0000","last_fullsized":"2024-02-21T23:06:09.934813+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:38:56.665723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.941480+0000","last_change":"2024-02-21T23:06:09.941480+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.941480+0000","last_undegraded":"2024-02-21T23:06:09.941480+0000","last_fullsized":"2024-02-21T23:06:09.941480+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.957795+0000","last_change":"2024-02-21T23:06:09.953218+0000","last_active":"2024-02-21T23:06:09.957795+0000","last_peered":"2024-02-21T23:06:09.957795+0000","last_clean":"2024-02-21T23:06:09.957795+0000","last_became_active":"2024-02-21T23:06:09.952908+0000","last_became_peered":"2024-02-21T23:06:09.952908+0000","last_unstale":"2024-02-21T23:06:09.957795+0000","last_undegraded":"2024-02-21T23:06:09.957795+0000","last_fullsized":"2024-02-21T23:06:09.957795+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:08:55.021772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.920356+0000","last_change":"2024-02-21T23:05:51.412683+0000","last_active":"2024-02-21T23:06:09.920356+0000","last_peered":"2024-02-21T23:06:09.920356+0000","last_clean":"2024-02-21T23:06:09.920356+0000","last_became_active":"2024-02-21T23:05:51.412389+0000","last_became_peered":"2024-02-21T23:05:51.412389+0000","last_unstale":"2024-02-21T23:06:09.920356+0000","last_undegraded":"2024-02-21T23:06:09.920356+0000","last_fullsized":"2024-02-21T23:06:09.920356+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:57:06.211024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.918795+0000","last_change":"2024-02-21T23:05:50.376356+0000","last_active":"2024-02-21T23:06:09.918795+0000","last_peered":"2024-02-21T23:06:09.918795+0000","last_clean":"2024-02-21T23:06:09.918795+0000","last_became_active":"2024-02-21T23:05:50.370467+0000","last_became_peered":"2024-02-21T23:05:50.370467+0000","last_unstale":"2024-02-21T23:06:09.918795+0000","last_undegraded":"2024-02-21T23:06:09.918795+0000","last_fullsized":"2024-02-21T23:06:09.918795+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:56:49.435540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.940614+0000","last_change":"2024-02-21T23:06:09.940614+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.940614+0000","last_undegraded":"2024-02-21T23:06:09.940614+0000","last_fullsized":"2024-02-21T23:06:09.940614+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.945283+0000","last_change":"2024-02-21T23:06:09.940641+0000","last_active":"2024-02-21T23:06:09.945283+0000","last_peered":"2024-02-21T23:06:09.945283+0000","last_clean":"2024-02-21T23:06:09.945283+0000","last_became_active":"2024-02-21T23:06:09.940353+0000","last_became_peered":"2024-02-21T23:06:09.940353+0000","last_unstale":"2024-02-21T23:06:09.945283+0000","last_undegraded":"2024-02-21T23:06:09.945283+0000","last_fullsized":"2024-02-21T23:06:09.945283+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:45:25.410745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.944841+0000","last_change":"2024-02-21T23:05:51.413008+0000","last_active":"2024-02-21T23:06:09.944841+0000","last_peered":"2024-02-21T23:06:09.944841+0000","last_clean":"2024-02-21T23:06:09.944841+0000","last_became_active":"2024-02-21T23:05:51.412714+0000","last_became_peered":"2024-02-21T23:05:51.412714+0000","last_unstale":"2024-02-21T23:06:09.944841+0000","last_undegraded":"2024-02-21T23:06:09.944841+0000","last_fullsized":"2024-02-21T23:06:09.944841+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:03:08.789066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.946178+0000","last_change":"2024-02-21T23:05:50.375581+0000","last_active":"2024-02-21T23:06:09.946178+0000","last_peered":"2024-02-21T23:06:09.946178+0000","last_clean":"2024-02-21T23:06:09.946178+0000","last_became_active":"2024-02-21T23:05:50.375282+0000","last_became_peered":"2024-02-21T23:05:50.375282+0000","last_unstale":"2024-02-21T23:06:09.946178+0000","last_undegraded":"2024-02-21T23:06:09.946178+0000","last_fullsized":"2024-02-21T23:06:09.946178+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:49:45.012428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":278,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.942188+0000","last_change":"2024-02-21T23:02:32.289335+0000","last_active":"2024-02-21T23:06:09.942188+0000","last_peered":"2024-02-21T23:06:09.942188+0000","last_clean":"2024-02-21T23:06:09.942188+0000","last_became_active":"2024-02-21T23:02:32.286913+0000","last_became_peered":"2024-02-21T23:02:32.286913+0000","last_unstale":"2024-02-21T23:06:09.942188+0000","last_undegraded":"2024-02-21T23:06:09.942188+0000","last_fullsized":"2024-02-21T23:06:09.942188+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_clean_scrub_stamp":"2024-02-21T23:02:31.172569+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:36:23.987374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":14,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574894,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32180,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339660,"statfs":{"total":96636764160,"available":96603811840,"internally_reserved":0,"allocated":1024000,"data_stored":1048493,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574895,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32128,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339712,"statfs":{"total":96636764160,"available":96603865088,"internally_reserved":0,"allocated":966656,"data_stored":459715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574894,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27960,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343880,"statfs":{"total":96636764160,"available":96608133120,"internally_reserved":0,"allocated":958464,"data_stored":458137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574895,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32136,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339704,"statfs":{"total":96636764160,"available":96603856896,"internally_reserved":0,"allocated":974848,"data_stored":459819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574895,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32120,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339720,"statfs":{"total":96636764160,"available":96603873280,"internally_reserved":0,"allocated":958464,"data_stored":458685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574896,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32128,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339712,"statfs":{"total":96636764160,"available":96603865088,"internally_reserved":0,"allocated":966656,"data_stored":458845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574894,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32108,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339732,"statfs":{"total":96636764160,"available":96603885568,"internally_reserved":0,"allocated":950272,"data_stored":450693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574894,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32176,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339664,"statfs":{"total":96636764160,"available":96603815936,"internally_reserved":0,"allocated":1019904,"data_stored":1048471,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-21T23:06:16.152 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-21T23:06:16.319 INFO:tasks.ceph.mds.c.smithi028.stderr:2024-02-21T23:06:16.329+0000 7fd5d0000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-21T23:06:16.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.411+0000 7fb1d5000640 1 Processor -- start 2024-02-21T23:06:16.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.411+0000 7fb1d5000640 1 -- start start 2024-02-21T23:06:16.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.412+0000 7fb1d5000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb1d00aff90 0x7fb1d00c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:16.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.412+0000 7fb1d5000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb1d006dbf0 0x7fb1d0158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:16.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.412+0000 7fb1d5000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fb1d00c5720 0x7fb1d0159000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:16.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.412+0000 7fb1d5000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1d00c7270 con 0x7fb1d00aff90 2024-02-21T23:06:16.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.412+0000 7fb1d5000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1d0158630 con 0x7fb1d00c5720 2024-02-21T23:06:16.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.412+0000 7fb1d5000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d00c74c0 con 0x7fb1d006dbf0 2024-02-21T23:06:16.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.412+0000 7fb1cf400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb1d00aff90 0x7fb1d00c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:16.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.412+0000 7fb1cf400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb1d00aff90 0x7fb1d00c51e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54644/0 (socket says 172.21.15.28:54644) 2024-02-21T23:06:16.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.412+0000 7fb1cf400640 1 -- 172.21.15.28:0/798341846 learned_addr learned my addr 172.21.15.28:0/798341846 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:16.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.413+0000 7fb1cfe00640 1 --2- 172.21.15.28:0/798341846 >> v2:172.21.15.81:3300/0 conn(0x7fb1d00c5720 0x7fb1d0159000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:16.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.413+0000 7fb1cea00640 1 --2- 172.21.15.28:0/798341846 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb1d006dbf0 0x7fb1d0158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:16.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.413+0000 7fb1cf400640 1 -- 172.21.15.28:0/798341846 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb1d006dbf0 msgr2=0x7fb1d0158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:16.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.413+0000 7fb1cf400640 1 --2- 172.21.15.28:0/798341846 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb1d006dbf0 0x7fb1d0158ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:16.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.413+0000 7fb1cf400640 1 -- 172.21.15.28:0/798341846 >> v2:172.21.15.81:3300/0 conn(0x7fb1d00c5720 msgr2=0x7fb1d0159000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:16.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.413+0000 7fb1cf400640 1 --2- 172.21.15.28:0/798341846 >> v2:172.21.15.81:3300/0 conn(0x7fb1d00c5720 0x7fb1d0159000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:16.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.413+0000 7fb1cf400640 1 -- 172.21.15.28:0/798341846 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1d00c77a0 con 0x7fb1d00aff90 2024-02-21T23:06:16.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.413+0000 7fb1cea00640 1 --2- 172.21.15.28:0/798341846 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb1d006dbf0 0x7fb1d0158ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:06:16.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.413+0000 7fb1cfe00640 1 --2- 172.21.15.28:0/798341846 >> v2:172.21.15.81:3300/0 conn(0x7fb1d00c5720 0x7fb1d0159000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:06:16.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.414+0000 7fb1cf400640 1 --2- 172.21.15.28:0/798341846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1d00aff90 0x7fb1d00c51e0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fb1d00ca6e0 tx=0x7fb1c002b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=833fe72e23c7af78 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:16.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.414+0000 7fb1ce000640 1 -- 172.21.15.28:0/798341846 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1c0035020 con 0x7fb1d00aff90 2024-02-21T23:06:16.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.414+0000 7fb1ce000640 1 -- 172.21.15.28:0/798341846 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1c002d040 con 0x7fb1d00aff90 2024-02-21T23:06:16.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.414+0000 7fb1ce000640 1 -- 172.21.15.28:0/798341846 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1c0031a80 con 0x7fb1d00aff90 2024-02-21T23:06:16.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.415+0000 7fb1d5000640 1 -- 172.21.15.28:0/798341846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1d00aff90 msgr2=0x7fb1d00c51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:16.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.415+0000 7fb1d5000640 1 --2- 172.21.15.28:0/798341846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1d00aff90 0x7fb1d00c51e0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fb1d00ca6e0 tx=0x7fb1c002b900 comp rx=0 tx=0).stop 2024-02-21T23:06:16.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.415+0000 7fb1d5000640 1 -- 172.21.15.28:0/798341846 shutdown_connections 2024-02-21T23:06:16.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.415+0000 7fb1d5000640 1 --2- 172.21.15.28:0/798341846 >> v2:172.21.15.81:3300/0 conn(0x7fb1d00c5720 0x7fb1d0159000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:16.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.415+0000 7fb1d5000640 1 --2- 172.21.15.28:0/798341846 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb1d006dbf0 0x7fb1d0158ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:16.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.415+0000 7fb1d5000640 1 --2- 172.21.15.28:0/798341846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1d00aff90 0x7fb1d00c51e0 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:16.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.415+0000 7fb1d5000640 1 -- 172.21.15.28:0/798341846 >> 172.21.15.28:0/798341846 conn(0x7fb1d00ac370 msgr2=0x7fb1d00ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:16.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.415+0000 7fb1d5000640 1 -- 172.21.15.28:0/798341846 shutdown_connections 2024-02-21T23:06:16.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.415+0000 7fb1d5000640 1 -- 172.21.15.28:0/798341846 wait complete. 2024-02-21T23:06:16.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.418+0000 7fb1d5000640 1 Processor -- start 2024-02-21T23:06:16.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.418+0000 7fb1d5000640 1 -- start start 2024-02-21T23:06:16.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.419+0000 7fb1d5000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb1d006dbf0 0x7fb1d0154540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:16.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.419+0000 7fb1d5000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1d00aff90 0x7fb1d0153900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:16.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.419+0000 7fb1d5000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1d00c5720 0x7fb1d01eb080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:16.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.419+0000 7fb1d5000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d00c8760 con 0x7fb1d00c5720 2024-02-21T23:06:16.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.419+0000 7fb1d5000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d00be0c0 con 0x7fb1d00aff90 2024-02-21T23:06:16.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.419+0000 7fb1d5000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d00c88e0 con 0x7fb1d006dbf0 2024-02-21T23:06:16.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.419+0000 7fb1cfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1d00c5720 0x7fb1d01eb080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:16.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.419+0000 7fb1cfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1d00c5720 0x7fb1d01eb080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54650/0 (socket says 172.21.15.28:54650) 2024-02-21T23:06:16.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.420+0000 7fb1cfe00640 1 -- 172.21.15.28:0/3080624934 learned_addr learned my addr 172.21.15.28:0/3080624934 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:16.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.420+0000 7fb1cf400640 1 --2- 172.21.15.28:0/3080624934 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb1d006dbf0 0x7fb1d0154540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:16.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.420+0000 7fb1cea00640 1 --2- 172.21.15.28:0/3080624934 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1d00aff90 0x7fb1d0153900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:16.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.420+0000 7fb1cfe00640 1 -- 172.21.15.28:0/3080624934 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb1d006dbf0 msgr2=0x7fb1d0154540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:16.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.420+0000 7fb1cfe00640 1 --2- 172.21.15.28:0/3080624934 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb1d006dbf0 0x7fb1d0154540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:16.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.420+0000 7fb1cfe00640 1 -- 172.21.15.28:0/3080624934 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1d00aff90 msgr2=0x7fb1d0153900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:16.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.420+0000 7fb1cfe00640 1 --2- 172.21.15.28:0/3080624934 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1d00aff90 0x7fb1d0153900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:16.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.420+0000 7fb1cfe00640 1 -- 172.21.15.28:0/3080624934 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1d0154ee0 con 0x7fb1d00c5720 2024-02-21T23:06:16.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.420+0000 7fb1cfe00640 1 --2- 172.21.15.28:0/3080624934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1d00c5720 0x7fb1d01eb080 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fb1c800cd50 tx=0x7fb1c800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:16.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.421+0000 7fb1c7400640 1 -- 172.21.15.28:0/3080624934 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1c800f680 con 0x7fb1d00c5720 2024-02-21T23:06:16.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.421+0000 7fb1c7400640 1 -- 172.21.15.28:0/3080624934 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1c800f840 con 0x7fb1d00c5720 2024-02-21T23:06:16.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.421+0000 7fb1c7400640 1 -- 172.21.15.28:0/3080624934 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1c800fc30 con 0x7fb1d00c5720 2024-02-21T23:06:16.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.421+0000 7fb1d5000640 1 -- 172.21.15.28:0/3080624934 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1d0154c00 con 0x7fb1d00c5720 2024-02-21T23:06:16.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.421+0000 7fb1d5000640 1 -- 172.21.15.28:0/3080624934 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1d0153100 con 0x7fb1d00c5720 2024-02-21T23:06:16.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.422+0000 7fb1d5000640 1 -- 172.21.15.28:0/3080624934 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1d01552f0 con 0x7fb1d00c5720 2024-02-21T23:06:16.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.423+0000 7fb1c7400640 1 -- 172.21.15.28:0/3080624934 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb1c800e3e0 con 0x7fb1d00c5720 2024-02-21T23:06:16.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.425+0000 7fb1c7400640 1 --2- 172.21.15.28:0/3080624934 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb1b006b4d0 0x7fb1b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:16.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.425+0000 7fb1c7400640 1 -- 172.21.15.28:0/3080624934 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6957+0+0 (secure 0 0 0) 0x7fb1c808b4c0 con 0x7fb1d00c5720 2024-02-21T23:06:16.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.425+0000 7fb1cf400640 1 --2- 172.21.15.28:0/3080624934 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb1b006b4d0 0x7fb1b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:16.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.426+0000 7fb1cf400640 1 --2- 172.21.15.28:0/3080624934 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb1b006b4d0 0x7fb1b006d990 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fb1c0036000 tx=0x7fb1c003c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:16.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.429+0000 7fb1c7400640 1 -- 172.21.15.28:0/3080624934 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb1d01552f0 con 0x7fb1d00c5720 2024-02-21T23:06:16.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.626+0000 7fb1d5000640 1 -- 172.21.15.28:0/3080624934 --> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb1d01eb5c0 con 0x7fb1b006b4d0 2024-02-21T23:06:16.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.661+0000 7fb1c7400640 1 -- 172.21.15.28:0/3080624934 <== mgr.4101 v2:172.21.15.81:6832/1848538577 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+485422 (secure 0 0 0) 0x7fb1c0031a80 con 0x7fb1b006b4d0 2024-02-21T23:06:16.657 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:06:16.660 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-21T23:06:16.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.668+0000 7fb1d5000640 1 -- 172.21.15.28:0/3080624934 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb1b006b4d0 msgr2=0x7fb1b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:16.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.668+0000 7fb1d5000640 1 --2- 172.21.15.28:0/3080624934 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb1b006b4d0 0x7fb1b006d990 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fb1c0036000 tx=0x7fb1c003c040 comp rx=0 tx=0).stop 2024-02-21T23:06:16.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.668+0000 7fb1d5000640 1 -- 172.21.15.28:0/3080624934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1d00c5720 msgr2=0x7fb1d01eb080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:16.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.668+0000 7fb1d5000640 1 --2- 172.21.15.28:0/3080624934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1d00c5720 0x7fb1d01eb080 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fb1c800cd50 tx=0x7fb1c800a650 comp rx=0 tx=0).stop 2024-02-21T23:06:16.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.669+0000 7fb1d5000640 1 -- 172.21.15.28:0/3080624934 shutdown_connections 2024-02-21T23:06:16.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.669+0000 7fb1d5000640 1 --2- 172.21.15.28:0/3080624934 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb1b006b4d0 0x7fb1b006d990 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:16.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.669+0000 7fb1d5000640 1 --2- 172.21.15.28:0/3080624934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1d00c5720 0x7fb1d01eb080 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:16.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.669+0000 7fb1d5000640 1 --2- 172.21.15.28:0/3080624934 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1d00aff90 0x7fb1d0153900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:16.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.669+0000 7fb1d5000640 1 --2- 172.21.15.28:0/3080624934 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb1d006dbf0 0x7fb1d0154540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:16.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.669+0000 7fb1d5000640 1 -- 172.21.15.28:0/3080624934 >> 172.21.15.28:0/3080624934 conn(0x7fb1d00ac370 msgr2=0x7fb1d01571a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:16.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.669+0000 7fb1d5000640 1 -- 172.21.15.28:0/3080624934 shutdown_connections 2024-02-21T23:06:16.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.669+0000 7fb1d5000640 1 -- 172.21.15.28:0/3080624934 wait complete. 2024-02-21T23:06:16.678 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":185,"stamp":"2024-02-21T23:06:14.741815+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":228,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":252936,"kb_used_data":7636,"kb_used_omap":12,"kb_used_meta":245171,"kb_avail":754721784,"statfs":{"total":773094113280,"available":772835106816,"internally_reserved":0,"allocated":7819264,"data_stored":4842858,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":251055665},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.833080"},"pg_stats":[{"pgid":"14.3d","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.940039+0000","last_change":"2024-02-21T23:06:09.940039+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.940039+0000","last_undegraded":"2024-02-21T23:06:09.940039+0000","last_fullsized":"2024-02-21T23:06:09.940039+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.3f","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.946468+0000","last_change":"2024-02-21T23:05:51.408831+0000","last_active":"2024-02-21T23:06:09.946468+0000","last_peered":"2024-02-21T23:06:09.946468+0000","last_clean":"2024-02-21T23:06:09.946468+0000","last_became_active":"2024-02-21T23:05:51.408353+0000","last_became_peered":"2024-02-21T23:05:51.408353+0000","last_unstale":"2024-02-21T23:06:09.946468+0000","last_undegraded":"2024-02-21T23:06:09.946468+0000","last_fullsized":"2024-02-21T23:06:09.946468+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:36:47.962935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.3c","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.931414+0000","last_change":"2024-02-21T23:06:09.931414+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.931414+0000","last_undegraded":"2024-02-21T23:06:09.931414+0000","last_fullsized":"2024-02-21T23:06:09.931414+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.944046+0000","last_change":"2024-02-21T23:05:51.420557+0000","last_active":"2024-02-21T23:06:09.944046+0000","last_peered":"2024-02-21T23:06:09.944046+0000","last_clean":"2024-02-21T23:06:09.944046+0000","last_became_active":"2024-02-21T23:05:51.420078+0000","last_became_peered":"2024-02-21T23:05:51.420078+0000","last_unstale":"2024-02-21T23:06:09.944046+0000","last_undegraded":"2024-02-21T23:06:09.944046+0000","last_fullsized":"2024-02-21T23:06:09.944046+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:36:23.792395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.3f","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.936313+0000","last_change":"2024-02-21T23:06:09.936313+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.936313+0000","last_undegraded":"2024-02-21T23:06:09.936313+0000","last_fullsized":"2024-02-21T23:06:09.936313+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.919954+0000","last_change":"2024-02-21T23:05:51.408934+0000","last_active":"2024-02-21T23:06:09.919954+0000","last_peered":"2024-02-21T23:06:09.919954+0000","last_clean":"2024-02-21T23:06:09.919954+0000","last_became_active":"2024-02-21T23:05:51.408597+0000","last_became_peered":"2024-02-21T23:05:51.408597+0000","last_unstale":"2024-02-21T23:06:09.919954+0000","last_undegraded":"2024-02-21T23:06:09.919954+0000","last_fullsized":"2024-02-21T23:06:09.919954+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:58:55.829941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.3e","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.924703+0000","last_change":"2024-02-21T23:06:09.924703+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.924703+0000","last_undegraded":"2024-02-21T23:06:09.924703+0000","last_fullsized":"2024-02-21T23:06:09.924703+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.934821+0000","last_change":"2024-02-21T23:05:51.423479+0000","last_active":"2024-02-21T23:06:09.934821+0000","last_peered":"2024-02-21T23:06:09.934821+0000","last_clean":"2024-02-21T23:06:09.934821+0000","last_became_active":"2024-02-21T23:05:51.423108+0000","last_became_peered":"2024-02-21T23:05:51.423108+0000","last_unstale":"2024-02-21T23:06:09.934821+0000","last_undegraded":"2024-02-21T23:06:09.934821+0000","last_fullsized":"2024-02-21T23:06:09.934821+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:36:47.962935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.39","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.936144+0000","last_change":"2024-02-21T23:06:09.936144+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.936144+0000","last_undegraded":"2024-02-21T23:06:09.936144+0000","last_fullsized":"2024-02-21T23:06:09.936144+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.940522+0000","last_change":"2024-02-21T23:05:51.419436+0000","last_active":"2024-02-21T23:06:09.940522+0000","last_peered":"2024-02-21T23:06:09.940522+0000","last_clean":"2024-02-21T23:06:09.940522+0000","last_became_active":"2024-02-21T23:05:51.418562+0000","last_became_peered":"2024-02-21T23:05:51.418562+0000","last_unstale":"2024-02-21T23:06:09.940522+0000","last_undegraded":"2024-02-21T23:06:09.940522+0000","last_fullsized":"2024-02-21T23:06:09.940522+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:19:45.189126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.38","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.932838+0000","last_change":"2024-02-21T23:06:09.932838+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.932838+0000","last_undegraded":"2024-02-21T23:06:09.932838+0000","last_fullsized":"2024-02-21T23:06:09.932838+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.945821+0000","last_change":"2024-02-21T23:05:51.410014+0000","last_active":"2024-02-21T23:06:09.945821+0000","last_peered":"2024-02-21T23:06:09.945821+0000","last_clean":"2024-02-21T23:06:09.945821+0000","last_became_active":"2024-02-21T23:05:51.409637+0000","last_became_peered":"2024-02-21T23:05:51.409637+0000","last_unstale":"2024-02-21T23:06:09.945821+0000","last_undegraded":"2024-02-21T23:06:09.945821+0000","last_fullsized":"2024-02-21T23:06:09.945821+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:03:35.243949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3b","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.943914+0000","last_change":"2024-02-21T23:06:09.943914+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.943914+0000","last_undegraded":"2024-02-21T23:06:09.943914+0000","last_fullsized":"2024-02-21T23:06:09.943914+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.934473+0000","last_change":"2024-02-21T23:05:51.419505+0000","last_active":"2024-02-21T23:06:09.934473+0000","last_peered":"2024-02-21T23:06:09.934473+0000","last_clean":"2024-02-21T23:06:09.934473+0000","last_became_active":"2024-02-21T23:05:51.418040+0000","last_became_peered":"2024-02-21T23:05:51.418040+0000","last_unstale":"2024-02-21T23:06:09.934473+0000","last_undegraded":"2024-02-21T23:06:09.934473+0000","last_fullsized":"2024-02-21T23:06:09.934473+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:57:06.211024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.3a","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.940874+0000","last_change":"2024-02-21T23:06:09.940874+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.940874+0000","last_undegraded":"2024-02-21T23:06:09.940874+0000","last_fullsized":"2024-02-21T23:06:09.940874+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.930980+0000","last_change":"2024-02-21T23:05:51.416524+0000","last_active":"2024-02-21T23:06:09.930980+0000","last_peered":"2024-02-21T23:06:09.930980+0000","last_clean":"2024-02-21T23:06:09.930980+0000","last_became_active":"2024-02-21T23:05:51.415391+0000","last_became_peered":"2024-02-21T23:05:51.415391+0000","last_unstale":"2024-02-21T23:06:09.930980+0000","last_undegraded":"2024-02-21T23:06:09.930980+0000","last_fullsized":"2024-02-21T23:06:09.930980+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:48:32.958103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.35","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.938982+0000","last_change":"2024-02-21T23:06:09.938982+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.938982+0000","last_undegraded":"2024-02-21T23:06:09.938982+0000","last_fullsized":"2024-02-21T23:06:09.938982+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.950904+0000","last_change":"2024-02-21T23:05:51.421068+0000","last_active":"2024-02-21T23:06:09.950904+0000","last_peered":"2024-02-21T23:06:09.950904+0000","last_clean":"2024-02-21T23:06:09.950904+0000","last_became_active":"2024-02-21T23:05:51.419746+0000","last_became_peered":"2024-02-21T23:05:51.419746+0000","last_unstale":"2024-02-21T23:06:09.950904+0000","last_undegraded":"2024-02-21T23:06:09.950904+0000","last_fullsized":"2024-02-21T23:06:09.950904+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:09:42.444733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.34","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.937566+0000","last_change":"2024-02-21T23:06:09.937566+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.937566+0000","last_undegraded":"2024-02-21T23:06:09.937566+0000","last_fullsized":"2024-02-21T23:06:09.937566+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.934177+0000","last_change":"2024-02-21T23:05:51.418295+0000","last_active":"2024-02-21T23:06:09.934177+0000","last_peered":"2024-02-21T23:06:09.934177+0000","last_clean":"2024-02-21T23:06:09.934177+0000","last_became_active":"2024-02-21T23:05:51.417780+0000","last_became_peered":"2024-02-21T23:05:51.417780+0000","last_unstale":"2024-02-21T23:06:09.934177+0000","last_undegraded":"2024-02-21T23:06:09.934177+0000","last_fullsized":"2024-02-21T23:06:09.934177+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:19:57.210565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.37","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.936387+0000","last_change":"2024-02-21T23:06:09.936387+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.936387+0000","last_undegraded":"2024-02-21T23:06:09.936387+0000","last_fullsized":"2024-02-21T23:06:09.936387+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.951775+0000","last_change":"2024-02-21T23:05:51.420518+0000","last_active":"2024-02-21T23:06:09.951775+0000","last_peered":"2024-02-21T23:06:09.951775+0000","last_clean":"2024-02-21T23:06:09.951775+0000","last_became_active":"2024-02-21T23:05:51.419564+0000","last_became_peered":"2024-02-21T23:05:51.419564+0000","last_unstale":"2024-02-21T23:06:09.951775+0000","last_undegraded":"2024-02-21T23:06:09.951775+0000","last_fullsized":"2024-02-21T23:06:09.951775+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:19:41.280587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.36","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.932127+0000","last_change":"2024-02-21T23:06:09.932127+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.932127+0000","last_undegraded":"2024-02-21T23:06:09.932127+0000","last_fullsized":"2024-02-21T23:06:09.932127+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.934938+0000","last_change":"2024-02-21T23:05:51.421691+0000","last_active":"2024-02-21T23:06:09.934938+0000","last_peered":"2024-02-21T23:06:09.934938+0000","last_clean":"2024-02-21T23:06:09.934938+0000","last_became_active":"2024-02-21T23:05:51.417621+0000","last_became_peered":"2024-02-21T23:05:51.417621+0000","last_unstale":"2024-02-21T23:06:09.934938+0000","last_undegraded":"2024-02-21T23:06:09.934938+0000","last_fullsized":"2024-02-21T23:06:09.934938+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:46:01.608149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.31","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.929205+0000","last_change":"2024-02-21T23:06:09.929205+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.929205+0000","last_undegraded":"2024-02-21T23:06:09.929205+0000","last_fullsized":"2024-02-21T23:06:09.929205+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.940619+0000","last_change":"2024-02-21T23:05:51.426871+0000","last_active":"2024-02-21T23:06:09.940619+0000","last_peered":"2024-02-21T23:06:09.940619+0000","last_clean":"2024-02-21T23:06:09.940619+0000","last_became_active":"2024-02-21T23:05:51.419412+0000","last_became_peered":"2024-02-21T23:05:51.419412+0000","last_unstale":"2024-02-21T23:06:09.940619+0000","last_undegraded":"2024-02-21T23:06:09.940619+0000","last_fullsized":"2024-02-21T23:06:09.940619+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:22:09.156621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.30","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.929405+0000","last_change":"2024-02-21T23:06:09.929405+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.929405+0000","last_undegraded":"2024-02-21T23:06:09.929405+0000","last_fullsized":"2024-02-21T23:06:09.929405+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.945911+0000","last_change":"2024-02-21T23:05:51.417593+0000","last_active":"2024-02-21T23:06:09.945911+0000","last_peered":"2024-02-21T23:06:09.945911+0000","last_clean":"2024-02-21T23:06:09.945911+0000","last_became_active":"2024-02-21T23:05:51.416945+0000","last_became_peered":"2024-02-21T23:05:51.416945+0000","last_unstale":"2024-02-21T23:06:09.945911+0000","last_undegraded":"2024-02-21T23:06:09.945911+0000","last_fullsized":"2024-02-21T23:06:09.945911+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:24:32.863072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.33","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.929330+0000","last_change":"2024-02-21T23:06:09.929330+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.929330+0000","last_undegraded":"2024-02-21T23:06:09.929330+0000","last_fullsized":"2024-02-21T23:06:09.929330+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.943577+0000","last_change":"2024-02-21T23:05:51.409645+0000","last_active":"2024-02-21T23:06:09.943577+0000","last_peered":"2024-02-21T23:06:09.943577+0000","last_clean":"2024-02-21T23:06:09.943577+0000","last_became_active":"2024-02-21T23:05:51.409356+0000","last_became_peered":"2024-02-21T23:05:51.409356+0000","last_unstale":"2024-02-21T23:06:09.943577+0000","last_undegraded":"2024-02-21T23:06:09.943577+0000","last_fullsized":"2024-02-21T23:06:09.943577+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:17:26.329609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.32","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.942772+0000","last_change":"2024-02-21T23:06:09.942772+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.942772+0000","last_undegraded":"2024-02-21T23:06:09.942772+0000","last_fullsized":"2024-02-21T23:06:09.942772+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.941967+0000","last_change":"2024-02-21T23:05:51.413656+0000","last_active":"2024-02-21T23:06:09.941967+0000","last_peered":"2024-02-21T23:06:09.941967+0000","last_clean":"2024-02-21T23:06:09.941967+0000","last_became_active":"2024-02-21T23:05:51.413350+0000","last_became_peered":"2024-02-21T23:05:51.413350+0000","last_unstale":"2024-02-21T23:06:09.941967+0000","last_undegraded":"2024-02-21T23:06:09.941967+0000","last_fullsized":"2024-02-21T23:06:09.941967+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:19:57.210565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.2d","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.938385+0000","last_change":"2024-02-21T23:06:09.938385+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.938385+0000","last_undegraded":"2024-02-21T23:06:09.938385+0000","last_fullsized":"2024-02-21T23:06:09.938385+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.930801+0000","last_change":"2024-02-21T23:05:51.427044+0000","last_active":"2024-02-21T23:06:09.930801+0000","last_peered":"2024-02-21T23:06:09.930801+0000","last_clean":"2024-02-21T23:06:09.930801+0000","last_became_active":"2024-02-21T23:05:51.419477+0000","last_became_peered":"2024-02-21T23:05:51.419477+0000","last_unstale":"2024-02-21T23:06:09.930801+0000","last_undegraded":"2024-02-21T23:06:09.930801+0000","last_fullsized":"2024-02-21T23:06:09.930801+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:04:00.714448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.2c","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.929202+0000","last_change":"2024-02-21T23:06:09.929202+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.929202+0000","last_undegraded":"2024-02-21T23:06:09.929202+0000","last_fullsized":"2024-02-21T23:06:09.929202+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.920081+0000","last_change":"2024-02-21T23:05:51.425126+0000","last_active":"2024-02-21T23:06:09.920081+0000","last_peered":"2024-02-21T23:06:09.920081+0000","last_clean":"2024-02-21T23:06:09.920081+0000","last_became_active":"2024-02-21T23:05:51.422511+0000","last_became_peered":"2024-02-21T23:05:51.422511+0000","last_unstale":"2024-02-21T23:06:09.920081+0000","last_undegraded":"2024-02-21T23:06:09.920081+0000","last_fullsized":"2024-02-21T23:06:09.920081+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:27:01.191337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.2f","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.939567+0000","last_change":"2024-02-21T23:06:09.939567+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.939567+0000","last_undegraded":"2024-02-21T23:06:09.939567+0000","last_fullsized":"2024-02-21T23:06:09.939567+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.946776+0000","last_change":"2024-02-21T23:05:51.423258+0000","last_active":"2024-02-21T23:06:09.946776+0000","last_peered":"2024-02-21T23:06:09.946776+0000","last_clean":"2024-02-21T23:06:09.946776+0000","last_became_active":"2024-02-21T23:05:51.422987+0000","last_became_peered":"2024-02-21T23:05:51.422987+0000","last_unstale":"2024-02-21T23:06:09.946776+0000","last_undegraded":"2024-02-21T23:06:09.946776+0000","last_fullsized":"2024-02-21T23:06:09.946776+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:41:31.260842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2e","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.941381+0000","last_change":"2024-02-21T23:06:09.941381+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.941381+0000","last_undegraded":"2024-02-21T23:06:09.941381+0000","last_fullsized":"2024-02-21T23:06:09.941381+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.937988+0000","last_change":"2024-02-21T23:05:51.427468+0000","last_active":"2024-02-21T23:06:09.937988+0000","last_peered":"2024-02-21T23:06:09.937988+0000","last_clean":"2024-02-21T23:06:09.937988+0000","last_became_active":"2024-02-21T23:05:51.425476+0000","last_became_peered":"2024-02-21T23:05:51.425476+0000","last_unstale":"2024-02-21T23:06:09.937988+0000","last_undegraded":"2024-02-21T23:06:09.937988+0000","last_fullsized":"2024-02-21T23:06:09.937988+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:06:18.158392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.29","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.929062+0000","last_change":"2024-02-21T23:06:09.929062+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.929062+0000","last_undegraded":"2024-02-21T23:06:09.929062+0000","last_fullsized":"2024-02-21T23:06:09.929062+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.948695+0000","last_change":"2024-02-21T23:05:51.415772+0000","last_active":"2024-02-21T23:06:09.948695+0000","last_peered":"2024-02-21T23:06:09.948695+0000","last_clean":"2024-02-21T23:06:09.948695+0000","last_became_active":"2024-02-21T23:05:51.415494+0000","last_became_peered":"2024-02-21T23:05:51.415494+0000","last_unstale":"2024-02-21T23:06:09.948695+0000","last_undegraded":"2024-02-21T23:06:09.948695+0000","last_fullsized":"2024-02-21T23:06:09.948695+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:42:37.006685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.28","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.932618+0000","last_change":"2024-02-21T23:06:09.932618+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.932618+0000","last_undegraded":"2024-02-21T23:06:09.932618+0000","last_fullsized":"2024-02-21T23:06:09.932618+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.937010+0000","last_change":"2024-02-21T23:05:51.428557+0000","last_active":"2024-02-21T23:06:09.937010+0000","last_peered":"2024-02-21T23:06:09.937010+0000","last_clean":"2024-02-21T23:06:09.937010+0000","last_became_active":"2024-02-21T23:05:51.427612+0000","last_became_peered":"2024-02-21T23:05:51.427612+0000","last_unstale":"2024-02-21T23:06:09.937010+0000","last_undegraded":"2024-02-21T23:06:09.937010+0000","last_fullsized":"2024-02-21T23:06:09.937010+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:38:44.895318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.2b","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.940735+0000","last_change":"2024-02-21T23:06:09.940735+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.940735+0000","last_undegraded":"2024-02-21T23:06:09.940735+0000","last_fullsized":"2024-02-21T23:06:09.940735+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.944849+0000","last_change":"2024-02-21T23:05:51.417302+0000","last_active":"2024-02-21T23:06:09.944849+0000","last_peered":"2024-02-21T23:06:09.944849+0000","last_clean":"2024-02-21T23:06:09.944849+0000","last_became_active":"2024-02-21T23:05:51.414092+0000","last_became_peered":"2024-02-21T23:05:51.414092+0000","last_unstale":"2024-02-21T23:06:09.944849+0000","last_undegraded":"2024-02-21T23:06:09.944849+0000","last_fullsized":"2024-02-21T23:06:09.944849+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:49:01.911867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2a","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.941237+0000","last_change":"2024-02-21T23:06:09.941237+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.941237+0000","last_undegraded":"2024-02-21T23:06:09.941237+0000","last_fullsized":"2024-02-21T23:06:09.941237+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.944272+0000","last_change":"2024-02-21T23:05:51.406411+0000","last_active":"2024-02-21T23:06:09.944272+0000","last_peered":"2024-02-21T23:06:09.944272+0000","last_clean":"2024-02-21T23:06:09.944272+0000","last_became_active":"2024-02-21T23:05:51.406104+0000","last_became_peered":"2024-02-21T23:05:51.406104+0000","last_unstale":"2024-02-21T23:06:09.944272+0000","last_undegraded":"2024-02-21T23:06:09.944272+0000","last_fullsized":"2024-02-21T23:06:09.944272+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:03:08.789066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.25","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.931274+0000","last_change":"2024-02-21T23:06:09.931274+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.931274+0000","last_undegraded":"2024-02-21T23:06:09.931274+0000","last_fullsized":"2024-02-21T23:06:09.931274+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.948003+0000","last_change":"2024-02-21T23:05:51.414112+0000","last_active":"2024-02-21T23:06:09.948003+0000","last_peered":"2024-02-21T23:06:09.948003+0000","last_clean":"2024-02-21T23:06:09.948003+0000","last_became_active":"2024-02-21T23:05:51.413796+0000","last_became_peered":"2024-02-21T23:05:51.413796+0000","last_unstale":"2024-02-21T23:06:09.948003+0000","last_undegraded":"2024-02-21T23:06:09.948003+0000","last_fullsized":"2024-02-21T23:06:09.948003+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:42:26.488510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.24","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.928558+0000","last_change":"2024-02-21T23:06:09.928558+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.928558+0000","last_undegraded":"2024-02-21T23:06:09.928558+0000","last_fullsized":"2024-02-21T23:06:09.928558+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.934345+0000","last_change":"2024-02-21T23:05:51.421208+0000","last_active":"2024-02-21T23:06:09.934345+0000","last_peered":"2024-02-21T23:06:09.934345+0000","last_clean":"2024-02-21T23:06:09.934345+0000","last_became_active":"2024-02-21T23:05:51.418171+0000","last_became_peered":"2024-02-21T23:05:51.418171+0000","last_unstale":"2024-02-21T23:06:09.934345+0000","last_undegraded":"2024-02-21T23:06:09.934345+0000","last_fullsized":"2024-02-21T23:06:09.934345+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:17:26.329609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.27","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.925950+0000","last_change":"2024-02-21T23:06:09.925950+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.925950+0000","last_undegraded":"2024-02-21T23:06:09.925950+0000","last_fullsized":"2024-02-21T23:06:09.925950+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.944600+0000","last_change":"2024-02-21T23:05:51.416857+0000","last_active":"2024-02-21T23:06:09.944600+0000","last_peered":"2024-02-21T23:06:09.944600+0000","last_clean":"2024-02-21T23:06:09.944600+0000","last_became_active":"2024-02-21T23:05:51.416577+0000","last_became_peered":"2024-02-21T23:05:51.416577+0000","last_unstale":"2024-02-21T23:06:09.944600+0000","last_undegraded":"2024-02-21T23:06:09.944600+0000","last_fullsized":"2024-02-21T23:06:09.944600+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:17:26.329609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.26","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.940425+0000","last_change":"2024-02-21T23:06:09.940425+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.940425+0000","last_undegraded":"2024-02-21T23:06:09.940425+0000","last_fullsized":"2024-02-21T23:06:09.940425+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.938091+0000","last_change":"2024-02-21T23:05:51.427489+0000","last_active":"2024-02-21T23:06:09.938091+0000","last_peered":"2024-02-21T23:06:09.938091+0000","last_clean":"2024-02-21T23:06:09.938091+0000","last_became_active":"2024-02-21T23:05:51.419458+0000","last_became_peered":"2024-02-21T23:05:51.419458+0000","last_unstale":"2024-02-21T23:06:09.938091+0000","last_undegraded":"2024-02-21T23:06:09.938091+0000","last_fullsized":"2024-02-21T23:06:09.938091+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:54:08.758447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.21","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.922993+0000","last_change":"2024-02-21T23:06:09.922993+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.922993+0000","last_undegraded":"2024-02-21T23:06:09.922993+0000","last_fullsized":"2024-02-21T23:06:09.922993+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.944278+0000","last_change":"2024-02-21T23:05:51.405741+0000","last_active":"2024-02-21T23:06:09.944278+0000","last_peered":"2024-02-21T23:06:09.944278+0000","last_clean":"2024-02-21T23:06:09.944278+0000","last_became_active":"2024-02-21T23:05:51.405301+0000","last_became_peered":"2024-02-21T23:05:51.405301+0000","last_unstale":"2024-02-21T23:06:09.944278+0000","last_undegraded":"2024-02-21T23:06:09.944278+0000","last_fullsized":"2024-02-21T23:06:09.944278+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:36:47.962935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.20","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.930360+0000","last_change":"2024-02-21T23:06:09.930360+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.930360+0000","last_undegraded":"2024-02-21T23:06:09.930360+0000","last_fullsized":"2024-02-21T23:06:09.930360+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.942207+0000","last_change":"2024-02-21T23:05:51.398663+0000","last_active":"2024-02-21T23:06:09.942207+0000","last_peered":"2024-02-21T23:06:09.942207+0000","last_clean":"2024-02-21T23:06:09.942207+0000","last_became_active":"2024-02-21T23:05:51.398146+0000","last_became_peered":"2024-02-21T23:05:51.398146+0000","last_unstale":"2024-02-21T23:06:09.942207+0000","last_undegraded":"2024-02-21T23:06:09.942207+0000","last_fullsized":"2024-02-21T23:06:09.942207+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:27:54.695721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.23","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.935616+0000","last_change":"2024-02-21T23:06:09.935616+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.935616+0000","last_undegraded":"2024-02-21T23:06:09.935616+0000","last_fullsized":"2024-02-21T23:06:09.935616+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.919121+0000","last_change":"2024-02-21T23:05:51.421330+0000","last_active":"2024-02-21T23:06:09.919121+0000","last_peered":"2024-02-21T23:06:09.919121+0000","last_clean":"2024-02-21T23:06:09.919121+0000","last_became_active":"2024-02-21T23:05:51.421012+0000","last_became_peered":"2024-02-21T23:05:51.421012+0000","last_unstale":"2024-02-21T23:06:09.919121+0000","last_undegraded":"2024-02-21T23:06:09.919121+0000","last_fullsized":"2024-02-21T23:06:09.919121+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:26:33.366331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.22","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.941990+0000","last_change":"2024-02-21T23:06:09.941990+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.941990+0000","last_undegraded":"2024-02-21T23:06:09.941990+0000","last_fullsized":"2024-02-21T23:06:09.941990+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.942077+0000","last_change":"2024-02-21T23:05:51.403173+0000","last_active":"2024-02-21T23:06:09.942077+0000","last_peered":"2024-02-21T23:06:09.942077+0000","last_clean":"2024-02-21T23:06:09.942077+0000","last_became_active":"2024-02-21T23:05:51.402610+0000","last_became_peered":"2024-02-21T23:05:51.402610+0000","last_unstale":"2024-02-21T23:06:09.942077+0000","last_undegraded":"2024-02-21T23:06:09.942077+0000","last_fullsized":"2024-02-21T23:06:09.942077+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:46:01.608149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.940077+0000","last_change":"2024-02-21T23:06:09.940077+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.940077+0000","last_undegraded":"2024-02-21T23:06:09.940077+0000","last_fullsized":"2024-02-21T23:06:09.940077+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.957716+0000","last_change":"2024-02-21T23:06:09.955402+0000","last_active":"2024-02-21T23:06:09.957716+0000","last_peered":"2024-02-21T23:06:09.957716+0000","last_clean":"2024-02-21T23:06:09.957716+0000","last_became_active":"2024-02-21T23:06:09.955113+0000","last_became_peered":"2024-02-21T23:06:09.955113+0000","last_unstale":"2024-02-21T23:06:09.957716+0000","last_undegraded":"2024-02-21T23:06:09.957716+0000","last_fullsized":"2024-02-21T23:06:09.957716+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:19:07.034670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.918368+0000","last_change":"2024-02-21T23:05:51.416534+0000","last_active":"2024-02-21T23:06:09.918368+0000","last_peered":"2024-02-21T23:06:09.918368+0000","last_clean":"2024-02-21T23:06:09.918368+0000","last_became_active":"2024-02-21T23:05:51.416244+0000","last_became_peered":"2024-02-21T23:05:51.416244+0000","last_unstale":"2024-02-21T23:06:09.918368+0000","last_undegraded":"2024-02-21T23:06:09.918368+0000","last_fullsized":"2024-02-21T23:06:09.918368+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:42:37.006685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"75'4","reported_seq":22,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.930907+0000","last_change":"2024-02-21T23:05:50.354109+0000","last_active":"2024-02-21T23:06:09.930907+0000","last_peered":"2024-02-21T23:06:09.930907+0000","last_clean":"2024-02-21T23:06:09.930907+0000","last_became_active":"2024-02-21T23:05:50.353809+0000","last_became_peered":"2024-02-21T23:05:50.353809+0000","last_unstale":"2024-02-21T23:06:09.930907+0000","last_undegraded":"2024-02-21T23:06:09.930907+0000","last_fullsized":"2024-02-21T23:06:09.930907+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:28:13.765557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.926981+0000","last_change":"2024-02-21T23:06:09.926981+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.926981+0000","last_undegraded":"2024-02-21T23:06:09.926981+0000","last_fullsized":"2024-02-21T23:06:09.926981+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.966118+0000","last_change":"2024-02-21T23:06:09.963140+0000","last_active":"2024-02-21T23:06:09.966118+0000","last_peered":"2024-02-21T23:06:09.966118+0000","last_clean":"2024-02-21T23:06:09.966118+0000","last_became_active":"2024-02-21T23:06:09.962637+0000","last_became_peered":"2024-02-21T23:06:09.962637+0000","last_unstale":"2024-02-21T23:06:09.966118+0000","last_undegraded":"2024-02-21T23:06:09.966118+0000","last_fullsized":"2024-02-21T23:06:09.966118+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:42:45.049032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.943704+0000","last_change":"2024-02-21T23:05:51.416268+0000","last_active":"2024-02-21T23:06:09.943704+0000","last_peered":"2024-02-21T23:06:09.943704+0000","last_clean":"2024-02-21T23:06:09.943704+0000","last_became_active":"2024-02-21T23:05:51.415980+0000","last_became_peered":"2024-02-21T23:05:51.415980+0000","last_unstale":"2024-02-21T23:06:09.943704+0000","last_undegraded":"2024-02-21T23:06:09.943704+0000","last_fullsized":"2024-02-21T23:06:09.943704+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:51:17.193881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"75'1","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.946611+0000","last_change":"2024-02-21T23:05:50.383438+0000","last_active":"2024-02-21T23:06:09.946611+0000","last_peered":"2024-02-21T23:06:09.946611+0000","last_clean":"2024-02-21T23:06:09.946611+0000","last_became_active":"2024-02-21T23:05:50.383134+0000","last_became_peered":"2024-02-21T23:05:50.383134+0000","last_unstale":"2024-02-21T23:06:09.946611+0000","last_undegraded":"2024-02-21T23:06:09.946611+0000","last_fullsized":"2024-02-21T23:06:09.946611+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:08:35.448428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.927781+0000","last_change":"2024-02-21T23:06:09.927781+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.927781+0000","last_undegraded":"2024-02-21T23:06:09.927781+0000","last_fullsized":"2024-02-21T23:06:09.927781+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.960638+0000","last_change":"2024-02-21T23:06:09.957178+0000","last_active":"2024-02-21T23:06:09.960638+0000","last_peered":"2024-02-21T23:06:09.960638+0000","last_clean":"2024-02-21T23:06:09.960638+0000","last_became_active":"2024-02-21T23:06:09.956818+0000","last_became_peered":"2024-02-21T23:06:09.956818+0000","last_unstale":"2024-02-21T23:06:09.960638+0000","last_undegraded":"2024-02-21T23:06:09.960638+0000","last_fullsized":"2024-02-21T23:06:09.960638+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:01:46.942433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.946131+0000","last_change":"2024-02-21T23:05:51.421571+0000","last_active":"2024-02-21T23:06:09.946131+0000","last_peered":"2024-02-21T23:06:09.946131+0000","last_clean":"2024-02-21T23:06:09.946131+0000","last_became_active":"2024-02-21T23:05:51.421110+0000","last_became_peered":"2024-02-21T23:05:51.421110+0000","last_unstale":"2024-02-21T23:06:09.946131+0000","last_undegraded":"2024-02-21T23:06:09.946131+0000","last_fullsized":"2024-02-21T23:06:09.946131+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:56:36.688152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"75'2","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.944712+0000","last_change":"2024-02-21T23:05:50.383231+0000","last_active":"2024-02-21T23:06:09.944712+0000","last_peered":"2024-02-21T23:06:09.944712+0000","last_clean":"2024-02-21T23:06:09.944712+0000","last_became_active":"2024-02-21T23:05:50.382757+0000","last_became_peered":"2024-02-21T23:05:50.382757+0000","last_unstale":"2024-02-21T23:06:09.944712+0000","last_undegraded":"2024-02-21T23:06:09.944712+0000","last_fullsized":"2024-02-21T23:06:09.944712+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:49:36.738197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.926671+0000","last_change":"2024-02-21T23:06:09.926671+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.926671+0000","last_undegraded":"2024-02-21T23:06:09.926671+0000","last_fullsized":"2024-02-21T23:06:09.926671+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.953270+0000","last_change":"2024-02-21T23:06:09.951255+0000","last_active":"2024-02-21T23:06:09.953270+0000","last_peered":"2024-02-21T23:06:09.953270+0000","last_clean":"2024-02-21T23:06:09.953270+0000","last_became_active":"2024-02-21T23:06:09.950934+0000","last_became_peered":"2024-02-21T23:06:09.950934+0000","last_unstale":"2024-02-21T23:06:09.953270+0000","last_undegraded":"2024-02-21T23:06:09.953270+0000","last_fullsized":"2024-02-21T23:06:09.953270+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:53:46.723561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.948793+0000","last_change":"2024-02-21T23:05:51.415892+0000","last_active":"2024-02-21T23:06:09.948793+0000","last_peered":"2024-02-21T23:06:09.948793+0000","last_clean":"2024-02-21T23:06:09.948793+0000","last_became_active":"2024-02-21T23:05:51.415520+0000","last_became_peered":"2024-02-21T23:05:51.415520+0000","last_unstale":"2024-02-21T23:06:09.948793+0000","last_undegraded":"2024-02-21T23:06:09.948793+0000","last_fullsized":"2024-02-21T23:06:09.948793+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:26:33.366331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"75'2","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.932382+0000","last_change":"2024-02-21T23:05:50.376050+0000","last_active":"2024-02-21T23:06:09.932382+0000","last_peered":"2024-02-21T23:06:09.932382+0000","last_clean":"2024-02-21T23:06:09.932382+0000","last_became_active":"2024-02-21T23:05:50.375731+0000","last_became_peered":"2024-02-21T23:05:50.375731+0000","last_unstale":"2024-02-21T23:06:09.932382+0000","last_undegraded":"2024-02-21T23:06:09.932382+0000","last_fullsized":"2024-02-21T23:06:09.932382+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:42:15.626763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.937587+0000","last_change":"2024-02-21T23:06:09.937587+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.937587+0000","last_undegraded":"2024-02-21T23:06:09.937587+0000","last_fullsized":"2024-02-21T23:06:09.937587+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.960875+0000","last_change":"2024-02-21T23:06:09.957048+0000","last_active":"2024-02-21T23:06:09.960875+0000","last_peered":"2024-02-21T23:06:09.960875+0000","last_clean":"2024-02-21T23:06:09.960875+0000","last_became_active":"2024-02-21T23:06:09.956666+0000","last_became_peered":"2024-02-21T23:06:09.956666+0000","last_unstale":"2024-02-21T23:06:09.960875+0000","last_undegraded":"2024-02-21T23:06:09.960875+0000","last_fullsized":"2024-02-21T23:06:09.960875+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:11:15.443744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.949179+0000","last_change":"2024-02-21T23:05:51.428435+0000","last_active":"2024-02-21T23:06:09.949179+0000","last_peered":"2024-02-21T23:06:09.949179+0000","last_clean":"2024-02-21T23:06:09.949179+0000","last_became_active":"2024-02-21T23:05:51.428133+0000","last_became_peered":"2024-02-21T23:05:51.428133+0000","last_unstale":"2024-02-21T23:06:09.949179+0000","last_undegraded":"2024-02-21T23:06:09.949179+0000","last_fullsized":"2024-02-21T23:06:09.949179+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:49:46.025250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"75'1","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.948878+0000","last_change":"2024-02-21T23:05:50.377574+0000","last_active":"2024-02-21T23:06:09.948878+0000","last_peered":"2024-02-21T23:06:09.948878+0000","last_clean":"2024-02-21T23:06:09.948878+0000","last_became_active":"2024-02-21T23:05:50.377250+0000","last_became_peered":"2024-02-21T23:05:50.377250+0000","last_unstale":"2024-02-21T23:06:09.948878+0000","last_undegraded":"2024-02-21T23:06:09.948878+0000","last_fullsized":"2024-02-21T23:06:09.948878+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:26:38.366160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.937636+0000","last_change":"2024-02-21T23:06:09.937636+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.937636+0000","last_undegraded":"2024-02-21T23:06:09.937636+0000","last_fullsized":"2024-02-21T23:06:09.937636+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.954390+0000","last_change":"2024-02-21T23:06:09.947108+0000","last_active":"2024-02-21T23:06:09.954390+0000","last_peered":"2024-02-21T23:06:09.954390+0000","last_clean":"2024-02-21T23:06:09.954390+0000","last_became_active":"2024-02-21T23:06:09.946815+0000","last_became_peered":"2024-02-21T23:06:09.946815+0000","last_unstale":"2024-02-21T23:06:09.954390+0000","last_undegraded":"2024-02-21T23:06:09.954390+0000","last_fullsized":"2024-02-21T23:06:09.954390+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:56:08.310911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.951879+0000","last_change":"2024-02-21T23:05:51.421508+0000","last_active":"2024-02-21T23:06:09.951879+0000","last_peered":"2024-02-21T23:06:09.951879+0000","last_clean":"2024-02-21T23:06:09.951879+0000","last_became_active":"2024-02-21T23:05:51.420041+0000","last_became_peered":"2024-02-21T23:05:51.420041+0000","last_unstale":"2024-02-21T23:06:09.951879+0000","last_undegraded":"2024-02-21T23:06:09.951879+0000","last_fullsized":"2024-02-21T23:06:09.951879+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:10:56.883222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"75'1","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.933207+0000","last_change":"2024-02-21T23:05:50.378668+0000","last_active":"2024-02-21T23:06:09.933207+0000","last_peered":"2024-02-21T23:06:09.933207+0000","last_clean":"2024-02-21T23:06:09.933207+0000","last_became_active":"2024-02-21T23:05:50.378369+0000","last_became_peered":"2024-02-21T23:05:50.378369+0000","last_unstale":"2024-02-21T23:06:09.933207+0000","last_undegraded":"2024-02-21T23:06:09.933207+0000","last_fullsized":"2024-02-21T23:06:09.933207+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:45:05.837402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.943022+0000","last_change":"2024-02-21T23:06:09.943022+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.943022+0000","last_undegraded":"2024-02-21T23:06:09.943022+0000","last_fullsized":"2024-02-21T23:06:09.943022+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.955897+0000","last_change":"2024-02-21T23:06:09.949620+0000","last_active":"2024-02-21T23:06:09.955897+0000","last_peered":"2024-02-21T23:06:09.955897+0000","last_clean":"2024-02-21T23:06:09.955897+0000","last_became_active":"2024-02-21T23:06:09.949252+0000","last_became_peered":"2024-02-21T23:06:09.949252+0000","last_unstale":"2024-02-21T23:06:09.955897+0000","last_undegraded":"2024-02-21T23:06:09.955897+0000","last_fullsized":"2024-02-21T23:06:09.955897+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:51:46.950044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.951942+0000","last_change":"2024-02-21T23:05:51.419084+0000","last_active":"2024-02-21T23:06:09.951942+0000","last_peered":"2024-02-21T23:06:09.951942+0000","last_clean":"2024-02-21T23:06:09.951942+0000","last_became_active":"2024-02-21T23:05:51.418752+0000","last_became_peered":"2024-02-21T23:05:51.418752+0000","last_unstale":"2024-02-21T23:06:09.951942+0000","last_undegraded":"2024-02-21T23:06:09.951942+0000","last_fullsized":"2024-02-21T23:06:09.951942+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:20:19.564331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.943810+0000","last_change":"2024-02-21T23:05:50.375534+0000","last_active":"2024-02-21T23:06:09.943810+0000","last_peered":"2024-02-21T23:06:09.943810+0000","last_clean":"2024-02-21T23:06:09.943810+0000","last_became_active":"2024-02-21T23:05:50.373762+0000","last_became_peered":"2024-02-21T23:05:50.373762+0000","last_unstale":"2024-02-21T23:06:09.943810+0000","last_undegraded":"2024-02-21T23:06:09.943810+0000","last_fullsized":"2024-02-21T23:06:09.943810+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:49:36.738197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.941977+0000","last_change":"2024-02-21T23:06:09.941977+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.941977+0000","last_undegraded":"2024-02-21T23:06:09.941977+0000","last_fullsized":"2024-02-21T23:06:09.941977+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.957774+0000","last_change":"2024-02-21T23:06:09.952374+0000","last_active":"2024-02-21T23:06:09.957774+0000","last_peered":"2024-02-21T23:06:09.957774+0000","last_clean":"2024-02-21T23:06:09.957774+0000","last_became_active":"2024-02-21T23:06:09.950283+0000","last_became_peered":"2024-02-21T23:06:09.950283+0000","last_unstale":"2024-02-21T23:06:09.957774+0000","last_undegraded":"2024-02-21T23:06:09.957774+0000","last_fullsized":"2024-02-21T23:06:09.957774+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:27:17.649346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.935109+0000","last_change":"2024-02-21T23:05:51.426711+0000","last_active":"2024-02-21T23:06:09.935109+0000","last_peered":"2024-02-21T23:06:09.935109+0000","last_clean":"2024-02-21T23:06:09.935109+0000","last_became_active":"2024-02-21T23:05:51.426415+0000","last_became_peered":"2024-02-21T23:05:51.426415+0000","last_unstale":"2024-02-21T23:06:09.935109+0000","last_undegraded":"2024-02-21T23:06:09.935109+0000","last_fullsized":"2024-02-21T23:06:09.935109+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:58:55.829941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.931209+0000","last_change":"2024-02-21T23:05:50.355567+0000","last_active":"2024-02-21T23:06:09.931209+0000","last_peered":"2024-02-21T23:06:09.931209+0000","last_clean":"2024-02-21T23:06:09.931209+0000","last_became_active":"2024-02-21T23:05:50.354835+0000","last_became_peered":"2024-02-21T23:05:50.354835+0000","last_unstale":"2024-02-21T23:06:09.931209+0000","last_undegraded":"2024-02-21T23:06:09.931209+0000","last_fullsized":"2024-02-21T23:06:09.931209+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:21:52.655098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.936369+0000","last_change":"2024-02-21T23:06:09.936369+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.936369+0000","last_undegraded":"2024-02-21T23:06:09.936369+0000","last_fullsized":"2024-02-21T23:06:09.936369+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.950750+0000","last_change":"2024-02-21T23:06:09.946843+0000","last_active":"2024-02-21T23:06:09.950750+0000","last_peered":"2024-02-21T23:06:09.950750+0000","last_clean":"2024-02-21T23:06:09.950750+0000","last_became_active":"2024-02-21T23:06:09.946452+0000","last_became_peered":"2024-02-21T23:06:09.946452+0000","last_unstale":"2024-02-21T23:06:09.950750+0000","last_undegraded":"2024-02-21T23:06:09.950750+0000","last_fullsized":"2024-02-21T23:06:09.950750+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:22:35.988029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.944383+0000","last_change":"2024-02-21T23:05:51.399923+0000","last_active":"2024-02-21T23:06:09.944383+0000","last_peered":"2024-02-21T23:06:09.944383+0000","last_clean":"2024-02-21T23:06:09.944383+0000","last_became_active":"2024-02-21T23:05:51.399642+0000","last_became_peered":"2024-02-21T23:05:51.399642+0000","last_unstale":"2024-02-21T23:06:09.944383+0000","last_undegraded":"2024-02-21T23:06:09.944383+0000","last_fullsized":"2024-02-21T23:06:09.944383+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:21:18.440977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.946594+0000","last_change":"2024-02-21T23:05:50.386639+0000","last_active":"2024-02-21T23:06:09.946594+0000","last_peered":"2024-02-21T23:06:09.946594+0000","last_clean":"2024-02-21T23:06:09.946594+0000","last_became_active":"2024-02-21T23:05:50.386158+0000","last_became_peered":"2024-02-21T23:05:50.386158+0000","last_unstale":"2024-02-21T23:06:09.946594+0000","last_undegraded":"2024-02-21T23:06:09.946594+0000","last_fullsized":"2024-02-21T23:06:09.946594+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:43:17.496894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.932767+0000","last_change":"2024-02-21T23:06:09.932767+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.932767+0000","last_undegraded":"2024-02-21T23:06:09.932767+0000","last_fullsized":"2024-02-21T23:06:09.932767+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.943824+0000","last_change":"2024-02-21T23:06:09.934904+0000","last_active":"2024-02-21T23:06:09.943824+0000","last_peered":"2024-02-21T23:06:09.943824+0000","last_clean":"2024-02-21T23:06:09.943824+0000","last_became_active":"2024-02-21T23:06:09.934606+0000","last_became_peered":"2024-02-21T23:06:09.934606+0000","last_unstale":"2024-02-21T23:06:09.943824+0000","last_undegraded":"2024-02-21T23:06:09.943824+0000","last_fullsized":"2024-02-21T23:06:09.943824+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:43:37.070238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.932966+0000","last_change":"2024-02-21T23:05:51.422844+0000","last_active":"2024-02-21T23:06:09.932966+0000","last_peered":"2024-02-21T23:06:09.932966+0000","last_clean":"2024-02-21T23:06:09.932966+0000","last_became_active":"2024-02-21T23:05:51.422488+0000","last_became_peered":"2024-02-21T23:05:51.422488+0000","last_unstale":"2024-02-21T23:06:09.932966+0000","last_undegraded":"2024-02-21T23:06:09.932966+0000","last_fullsized":"2024-02-21T23:06:09.932966+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:27:54.695721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.934464+0000","last_change":"2024-02-21T23:05:50.385146+0000","last_active":"2024-02-21T23:06:09.934464+0000","last_peered":"2024-02-21T23:06:09.934464+0000","last_clean":"2024-02-21T23:06:09.934464+0000","last_became_active":"2024-02-21T23:05:50.384418+0000","last_became_peered":"2024-02-21T23:05:50.384418+0000","last_unstale":"2024-02-21T23:06:09.934464+0000","last_undegraded":"2024-02-21T23:06:09.934464+0000","last_fullsized":"2024-02-21T23:06:09.934464+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:43:17.496894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.927206+0000","last_change":"2024-02-21T23:06:09.927206+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.927206+0000","last_undegraded":"2024-02-21T23:06:09.927206+0000","last_fullsized":"2024-02-21T23:06:09.927206+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.966125+0000","last_change":"2024-02-21T23:06:09.963033+0000","last_active":"2024-02-21T23:06:09.966125+0000","last_peered":"2024-02-21T23:06:09.966125+0000","last_clean":"2024-02-21T23:06:09.966125+0000","last_became_active":"2024-02-21T23:06:09.962695+0000","last_became_peered":"2024-02-21T23:06:09.962695+0000","last_unstale":"2024-02-21T23:06:09.966125+0000","last_undegraded":"2024-02-21T23:06:09.966125+0000","last_fullsized":"2024-02-21T23:06:09.966125+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:36:42.352917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.946054+0000","last_change":"2024-02-21T23:05:51.418004+0000","last_active":"2024-02-21T23:06:09.946054+0000","last_peered":"2024-02-21T23:06:09.946054+0000","last_clean":"2024-02-21T23:06:09.946054+0000","last_became_active":"2024-02-21T23:05:51.417463+0000","last_became_peered":"2024-02-21T23:05:51.417463+0000","last_unstale":"2024-02-21T23:06:09.946054+0000","last_undegraded":"2024-02-21T23:06:09.946054+0000","last_fullsized":"2024-02-21T23:06:09.946054+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:59:13.292494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"75'3","reported_seq":21,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.932692+0000","last_change":"2024-02-21T23:05:50.373189+0000","last_active":"2024-02-21T23:06:09.932692+0000","last_peered":"2024-02-21T23:06:09.932692+0000","last_clean":"2024-02-21T23:06:09.932692+0000","last_became_active":"2024-02-21T23:05:50.370680+0000","last_became_peered":"2024-02-21T23:05:50.370680+0000","last_unstale":"2024-02-21T23:06:09.932692+0000","last_undegraded":"2024-02-21T23:06:09.932692+0000","last_fullsized":"2024-02-21T23:06:09.932692+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:21:03.121546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.939836+0000","last_change":"2024-02-21T23:06:09.939836+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.939836+0000","last_undegraded":"2024-02-21T23:06:09.939836+0000","last_fullsized":"2024-02-21T23:06:09.939836+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.963277+0000","last_change":"2024-02-21T23:06:09.961844+0000","last_active":"2024-02-21T23:06:09.963277+0000","last_peered":"2024-02-21T23:06:09.963277+0000","last_clean":"2024-02-21T23:06:09.963277+0000","last_became_active":"2024-02-21T23:06:09.961480+0000","last_became_peered":"2024-02-21T23:06:09.961480+0000","last_unstale":"2024-02-21T23:06:09.963277+0000","last_undegraded":"2024-02-21T23:06:09.963277+0000","last_fullsized":"2024-02-21T23:06:09.963277+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:04:07.940431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.919599+0000","last_change":"2024-02-21T23:05:51.405242+0000","last_active":"2024-02-21T23:06:09.919599+0000","last_peered":"2024-02-21T23:06:09.919599+0000","last_clean":"2024-02-21T23:06:09.919599+0000","last_became_active":"2024-02-21T23:05:51.403777+0000","last_became_peered":"2024-02-21T23:05:51.403777+0000","last_unstale":"2024-02-21T23:06:09.919599+0000","last_undegraded":"2024-02-21T23:06:09.919599+0000","last_fullsized":"2024-02-21T23:06:09.919599+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:34:14.420091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"75'6","reported_seq":24,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.932534+0000","last_change":"2024-02-21T23:05:50.374208+0000","last_active":"2024-02-21T23:06:09.932534+0000","last_peered":"2024-02-21T23:06:09.932534+0000","last_clean":"2024-02-21T23:06:09.932534+0000","last_became_active":"2024-02-21T23:05:50.373869+0000","last_became_peered":"2024-02-21T23:05:50.373869+0000","last_unstale":"2024-02-21T23:06:09.932534+0000","last_undegraded":"2024-02-21T23:06:09.932534+0000","last_fullsized":"2024-02-21T23:06:09.932534+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:56:49.435540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.938351+0000","last_change":"2024-02-21T23:06:09.938351+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.938351+0000","last_undegraded":"2024-02-21T23:06:09.938351+0000","last_fullsized":"2024-02-21T23:06:09.938351+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.966390+0000","last_change":"2024-02-21T23:06:09.963215+0000","last_active":"2024-02-21T23:06:09.966390+0000","last_peered":"2024-02-21T23:06:09.966390+0000","last_clean":"2024-02-21T23:06:09.966390+0000","last_became_active":"2024-02-21T23:06:09.962779+0000","last_became_peered":"2024-02-21T23:06:09.962779+0000","last_unstale":"2024-02-21T23:06:09.966390+0000","last_undegraded":"2024-02-21T23:06:09.966390+0000","last_fullsized":"2024-02-21T23:06:09.966390+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:45:25.410745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.935046+0000","last_change":"2024-02-21T23:05:51.423479+0000","last_active":"2024-02-21T23:06:09.935046+0000","last_peered":"2024-02-21T23:06:09.935046+0000","last_clean":"2024-02-21T23:06:09.935046+0000","last_became_active":"2024-02-21T23:05:51.423190+0000","last_became_peered":"2024-02-21T23:05:51.423190+0000","last_unstale":"2024-02-21T23:06:09.935046+0000","last_undegraded":"2024-02-21T23:06:09.935046+0000","last_fullsized":"2024-02-21T23:06:09.935046+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:03:08.789066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"75'2","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.933594+0000","last_change":"2024-02-21T23:05:50.381643+0000","last_active":"2024-02-21T23:06:09.933594+0000","last_peered":"2024-02-21T23:06:09.933594+0000","last_clean":"2024-02-21T23:06:09.933594+0000","last_became_active":"2024-02-21T23:05:50.381345+0000","last_became_peered":"2024-02-21T23:05:50.381345+0000","last_unstale":"2024-02-21T23:06:09.933594+0000","last_undegraded":"2024-02-21T23:06:09.933594+0000","last_fullsized":"2024-02-21T23:06:09.933594+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:01:27.369089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.927030+0000","last_change":"2024-02-21T23:06:09.927030+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.927030+0000","last_undegraded":"2024-02-21T23:06:09.927030+0000","last_fullsized":"2024-02-21T23:06:09.927030+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.962022+0000","last_change":"2024-02-21T23:06:09.959967+0000","last_active":"2024-02-21T23:06:09.962022+0000","last_peered":"2024-02-21T23:06:09.962022+0000","last_clean":"2024-02-21T23:06:09.962022+0000","last_became_active":"2024-02-21T23:06:09.959175+0000","last_became_peered":"2024-02-21T23:06:09.959175+0000","last_unstale":"2024-02-21T23:06:09.962022+0000","last_undegraded":"2024-02-21T23:06:09.962022+0000","last_fullsized":"2024-02-21T23:06:09.962022+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:41:49.821364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.937192+0000","last_change":"2024-02-21T23:05:51.416753+0000","last_active":"2024-02-21T23:06:09.937192+0000","last_peered":"2024-02-21T23:06:09.937192+0000","last_clean":"2024-02-21T23:06:09.937192+0000","last_became_active":"2024-02-21T23:05:51.416427+0000","last_became_peered":"2024-02-21T23:05:51.416427+0000","last_unstale":"2024-02-21T23:06:09.937192+0000","last_undegraded":"2024-02-21T23:06:09.937192+0000","last_fullsized":"2024-02-21T23:06:09.937192+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:34:14.420091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.942362+0000","last_change":"2024-02-21T23:05:50.381962+0000","last_active":"2024-02-21T23:06:09.942362+0000","last_peered":"2024-02-21T23:06:09.942362+0000","last_clean":"2024-02-21T23:06:09.942362+0000","last_became_active":"2024-02-21T23:05:50.380838+0000","last_became_peered":"2024-02-21T23:05:50.380838+0000","last_unstale":"2024-02-21T23:06:09.942362+0000","last_undegraded":"2024-02-21T23:06:09.942362+0000","last_fullsized":"2024-02-21T23:06:09.942362+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:10:10.973567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.939210+0000","last_change":"2024-02-21T23:06:09.939210+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.939210+0000","last_undegraded":"2024-02-21T23:06:09.939210+0000","last_fullsized":"2024-02-21T23:06:09.939210+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.958221+0000","last_change":"2024-02-21T23:06:09.951230+0000","last_active":"2024-02-21T23:06:09.958221+0000","last_peered":"2024-02-21T23:06:09.958221+0000","last_clean":"2024-02-21T23:06:09.958221+0000","last_became_active":"2024-02-21T23:06:09.950920+0000","last_became_peered":"2024-02-21T23:06:09.950920+0000","last_unstale":"2024-02-21T23:06:09.958221+0000","last_undegraded":"2024-02-21T23:06:09.958221+0000","last_fullsized":"2024-02-21T23:06:09.958221+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:11:15.443744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.934193+0000","last_change":"2024-02-21T23:05:51.408507+0000","last_active":"2024-02-21T23:06:09.934193+0000","last_peered":"2024-02-21T23:06:09.934193+0000","last_clean":"2024-02-21T23:06:09.934193+0000","last_became_active":"2024-02-21T23:05:51.408198+0000","last_became_peered":"2024-02-21T23:05:51.408198+0000","last_unstale":"2024-02-21T23:06:09.934193+0000","last_undegraded":"2024-02-21T23:06:09.934193+0000","last_fullsized":"2024-02-21T23:06:09.934193+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:08:36.461250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.934108+0000","last_change":"2024-02-21T23:05:50.353116+0000","last_active":"2024-02-21T23:06:09.934108+0000","last_peered":"2024-02-21T23:06:09.934108+0000","last_clean":"2024-02-21T23:06:09.934108+0000","last_became_active":"2024-02-21T23:05:50.352768+0000","last_became_peered":"2024-02-21T23:05:50.352768+0000","last_unstale":"2024-02-21T23:06:09.934108+0000","last_undegraded":"2024-02-21T23:06:09.934108+0000","last_fullsized":"2024-02-21T23:06:09.934108+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:52:09.073053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.937280+0000","last_change":"2024-02-21T23:06:09.937280+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.937280+0000","last_undegraded":"2024-02-21T23:06:09.937280+0000","last_fullsized":"2024-02-21T23:06:09.937280+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.955574+0000","last_change":"2024-02-21T23:06:09.954937+0000","last_active":"2024-02-21T23:06:09.955574+0000","last_peered":"2024-02-21T23:06:09.955574+0000","last_clean":"2024-02-21T23:06:09.955574+0000","last_became_active":"2024-02-21T23:06:09.953847+0000","last_became_peered":"2024-02-21T23:06:09.953847+0000","last_unstale":"2024-02-21T23:06:09.955574+0000","last_undegraded":"2024-02-21T23:06:09.955574+0000","last_fullsized":"2024-02-21T23:06:09.955574+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:17:14.119658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.936892+0000","last_change":"2024-02-21T23:05:51.416424+0000","last_active":"2024-02-21T23:06:09.936892+0000","last_peered":"2024-02-21T23:06:09.936892+0000","last_clean":"2024-02-21T23:06:09.936892+0000","last_became_active":"2024-02-21T23:05:51.416070+0000","last_became_peered":"2024-02-21T23:05:51.416070+0000","last_unstale":"2024-02-21T23:06:09.936892+0000","last_undegraded":"2024-02-21T23:06:09.936892+0000","last_fullsized":"2024-02-21T23:06:09.936892+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:56:10.406772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"75'8","reported_seq":26,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.947720+0000","last_change":"2024-02-21T23:05:50.376063+0000","last_active":"2024-02-21T23:06:09.947720+0000","last_peered":"2024-02-21T23:06:09.947720+0000","last_clean":"2024-02-21T23:06:09.947720+0000","last_became_active":"2024-02-21T23:05:50.375772+0000","last_became_peered":"2024-02-21T23:05:50.375772+0000","last_unstale":"2024-02-21T23:06:09.947720+0000","last_undegraded":"2024-02-21T23:06:09.947720+0000","last_fullsized":"2024-02-21T23:06:09.947720+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:26:58.076002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.923016+0000","last_change":"2024-02-21T23:06:09.923016+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.923016+0000","last_undegraded":"2024-02-21T23:06:09.923016+0000","last_fullsized":"2024-02-21T23:06:09.923016+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.958286+0000","last_change":"2024-02-21T23:06:09.951762+0000","last_active":"2024-02-21T23:06:09.958286+0000","last_peered":"2024-02-21T23:06:09.958286+0000","last_clean":"2024-02-21T23:06:09.958286+0000","last_became_active":"2024-02-21T23:06:09.951155+0000","last_became_peered":"2024-02-21T23:06:09.951155+0000","last_unstale":"2024-02-21T23:06:09.958286+0000","last_undegraded":"2024-02-21T23:06:09.958286+0000","last_fullsized":"2024-02-21T23:06:09.958286+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:36:42.352917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.942415+0000","last_change":"2024-02-21T23:05:51.421565+0000","last_active":"2024-02-21T23:06:09.942415+0000","last_peered":"2024-02-21T23:06:09.942415+0000","last_clean":"2024-02-21T23:06:09.942415+0000","last_became_active":"2024-02-21T23:05:51.421252+0000","last_became_peered":"2024-02-21T23:05:51.421252+0000","last_unstale":"2024-02-21T23:06:09.942415+0000","last_undegraded":"2024-02-21T23:06:09.942415+0000","last_fullsized":"2024-02-21T23:06:09.942415+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:40:45.445418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"75'1","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.946704+0000","last_change":"2024-02-21T23:05:50.383778+0000","last_active":"2024-02-21T23:06:09.946704+0000","last_peered":"2024-02-21T23:06:09.946704+0000","last_clean":"2024-02-21T23:06:09.946704+0000","last_became_active":"2024-02-21T23:05:50.383436+0000","last_became_peered":"2024-02-21T23:05:50.383436+0000","last_unstale":"2024-02-21T23:06:09.946704+0000","last_undegraded":"2024-02-21T23:06:09.946704+0000","last_fullsized":"2024-02-21T23:06:09.946704+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:45:05.837402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.934671+0000","last_change":"2024-02-21T23:06:09.934671+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.934671+0000","last_undegraded":"2024-02-21T23:06:09.934671+0000","last_fullsized":"2024-02-21T23:06:09.934671+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.956162+0000","last_change":"2024-02-21T23:06:09.947681+0000","last_active":"2024-02-21T23:06:09.956162+0000","last_peered":"2024-02-21T23:06:09.956162+0000","last_clean":"2024-02-21T23:06:09.956162+0000","last_became_active":"2024-02-21T23:06:09.947370+0000","last_became_peered":"2024-02-21T23:06:09.947370+0000","last_unstale":"2024-02-21T23:06:09.956162+0000","last_undegraded":"2024-02-21T23:06:09.956162+0000","last_fullsized":"2024-02-21T23:06:09.956162+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:01:37.993072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.934009+0000","last_change":"2024-02-21T23:05:51.416676+0000","last_active":"2024-02-21T23:06:09.934009+0000","last_peered":"2024-02-21T23:06:09.934009+0000","last_clean":"2024-02-21T23:06:09.934009+0000","last_became_active":"2024-02-21T23:05:51.416297+0000","last_became_peered":"2024-02-21T23:05:51.416297+0000","last_unstale":"2024-02-21T23:06:09.934009+0000","last_undegraded":"2024-02-21T23:06:09.934009+0000","last_fullsized":"2024-02-21T23:06:09.934009+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:29:33.209545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"75'2","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.952053+0000","last_change":"2024-02-21T23:05:50.377181+0000","last_active":"2024-02-21T23:06:09.952053+0000","last_peered":"2024-02-21T23:06:09.952053+0000","last_clean":"2024-02-21T23:06:09.952053+0000","last_became_active":"2024-02-21T23:05:50.373524+0000","last_became_peered":"2024-02-21T23:05:50.373524+0000","last_unstale":"2024-02-21T23:06:09.952053+0000","last_undegraded":"2024-02-21T23:06:09.952053+0000","last_fullsized":"2024-02-21T23:06:09.952053+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:01:18.419728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.929958+0000","last_change":"2024-02-21T23:06:09.929958+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.929958+0000","last_undegraded":"2024-02-21T23:06:09.929958+0000","last_fullsized":"2024-02-21T23:06:09.929958+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.961219+0000","last_change":"2024-02-21T23:06:09.955696+0000","last_active":"2024-02-21T23:06:09.961219+0000","last_peered":"2024-02-21T23:06:09.961219+0000","last_clean":"2024-02-21T23:06:09.961219+0000","last_became_active":"2024-02-21T23:06:09.954715+0000","last_became_peered":"2024-02-21T23:06:09.954715+0000","last_unstale":"2024-02-21T23:06:09.961219+0000","last_undegraded":"2024-02-21T23:06:09.961219+0000","last_fullsized":"2024-02-21T23:06:09.961219+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:01:35.517150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.937478+0000","last_change":"2024-02-21T23:05:51.412971+0000","last_active":"2024-02-21T23:06:09.937478+0000","last_peered":"2024-02-21T23:06:09.937478+0000","last_clean":"2024-02-21T23:06:09.937478+0000","last_became_active":"2024-02-21T23:05:51.410966+0000","last_became_peered":"2024-02-21T23:05:51.410966+0000","last_unstale":"2024-02-21T23:06:09.937478+0000","last_undegraded":"2024-02-21T23:06:09.937478+0000","last_fullsized":"2024-02-21T23:06:09.937478+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:00:39.163092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"75'1","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.938268+0000","last_change":"2024-02-21T23:05:50.366307+0000","last_active":"2024-02-21T23:06:09.938268+0000","last_peered":"2024-02-21T23:06:09.938268+0000","last_clean":"2024-02-21T23:06:09.938268+0000","last_became_active":"2024-02-21T23:05:50.359426+0000","last_became_peered":"2024-02-21T23:05:50.359426+0000","last_unstale":"2024-02-21T23:06:09.938268+0000","last_undegraded":"2024-02-21T23:06:09.938268+0000","last_fullsized":"2024-02-21T23:06:09.938268+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:13:15.770526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.925807+0000","last_change":"2024-02-21T23:06:09.925807+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.925807+0000","last_undegraded":"2024-02-21T23:06:09.925807+0000","last_fullsized":"2024-02-21T23:06:09.925807+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.956240+0000","last_change":"2024-02-21T23:06:09.948670+0000","last_active":"2024-02-21T23:06:09.956240+0000","last_peered":"2024-02-21T23:06:09.956240+0000","last_clean":"2024-02-21T23:06:09.956240+0000","last_became_active":"2024-02-21T23:06:09.946109+0000","last_became_peered":"2024-02-21T23:06:09.946109+0000","last_unstale":"2024-02-21T23:06:09.956240+0000","last_undegraded":"2024-02-21T23:06:09.956240+0000","last_fullsized":"2024-02-21T23:06:09.956240+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:26:57.939504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.947158+0000","last_change":"2024-02-21T23:05:51.424067+0000","last_active":"2024-02-21T23:06:09.947158+0000","last_peered":"2024-02-21T23:06:09.947158+0000","last_clean":"2024-02-21T23:06:09.947158+0000","last_became_active":"2024-02-21T23:05:51.423644+0000","last_became_peered":"2024-02-21T23:05:51.423644+0000","last_unstale":"2024-02-21T23:06:09.947158+0000","last_undegraded":"2024-02-21T23:06:09.947158+0000","last_fullsized":"2024-02-21T23:06:09.947158+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:04:03.084512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.919750+0000","last_change":"2024-02-21T23:05:50.377924+0000","last_active":"2024-02-21T23:06:09.919750+0000","last_peered":"2024-02-21T23:06:09.919750+0000","last_clean":"2024-02-21T23:06:09.919750+0000","last_became_active":"2024-02-21T23:05:50.377440+0000","last_became_peered":"2024-02-21T23:05:50.377440+0000","last_unstale":"2024-02-21T23:06:09.919750+0000","last_undegraded":"2024-02-21T23:06:09.919750+0000","last_fullsized":"2024-02-21T23:06:09.919750+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:42:15.626763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.938677+0000","last_change":"2024-02-21T23:06:09.938677+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.938677+0000","last_undegraded":"2024-02-21T23:06:09.938677+0000","last_fullsized":"2024-02-21T23:06:09.938677+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.957820+0000","last_change":"2024-02-21T23:06:09.953285+0000","last_active":"2024-02-21T23:06:09.957820+0000","last_peered":"2024-02-21T23:06:09.957820+0000","last_clean":"2024-02-21T23:06:09.957820+0000","last_became_active":"2024-02-21T23:06:09.952940+0000","last_became_peered":"2024-02-21T23:06:09.952940+0000","last_unstale":"2024-02-21T23:06:09.957820+0000","last_undegraded":"2024-02-21T23:06:09.957820+0000","last_fullsized":"2024-02-21T23:06:09.957820+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:49:56.311541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.949283+0000","last_change":"2024-02-21T23:05:51.414619+0000","last_active":"2024-02-21T23:06:09.949283+0000","last_peered":"2024-02-21T23:06:09.949283+0000","last_clean":"2024-02-21T23:06:09.949283+0000","last_became_active":"2024-02-21T23:05:51.414059+0000","last_became_peered":"2024-02-21T23:05:51.414059+0000","last_unstale":"2024-02-21T23:06:09.949283+0000","last_undegraded":"2024-02-21T23:06:09.949283+0000","last_fullsized":"2024-02-21T23:06:09.949283+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:58:05.114297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"75'2","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.947250+0000","last_change":"2024-02-21T23:05:50.366877+0000","last_active":"2024-02-21T23:06:09.947250+0000","last_peered":"2024-02-21T23:06:09.947250+0000","last_clean":"2024-02-21T23:06:09.947250+0000","last_became_active":"2024-02-21T23:05:50.365566+0000","last_became_peered":"2024-02-21T23:05:50.365566+0000","last_unstale":"2024-02-21T23:06:09.947250+0000","last_undegraded":"2024-02-21T23:06:09.947250+0000","last_fullsized":"2024-02-21T23:06:09.947250+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:58:04.101475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.941480+0000","last_change":"2024-02-21T23:06:09.941480+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.941480+0000","last_undegraded":"2024-02-21T23:06:09.941480+0000","last_fullsized":"2024-02-21T23:06:09.941480+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.954312+0000","last_change":"2024-02-21T23:06:09.945701+0000","last_active":"2024-02-21T23:06:09.954312+0000","last_peered":"2024-02-21T23:06:09.954312+0000","last_clean":"2024-02-21T23:06:09.954312+0000","last_became_active":"2024-02-21T23:06:09.945391+0000","last_became_peered":"2024-02-21T23:06:09.945391+0000","last_unstale":"2024-02-21T23:06:09.954312+0000","last_undegraded":"2024-02-21T23:06:09.954312+0000","last_fullsized":"2024-02-21T23:06:09.954312+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:53:46.723561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.944153+0000","last_change":"2024-02-21T23:05:51.411005+0000","last_active":"2024-02-21T23:06:09.944153+0000","last_peered":"2024-02-21T23:06:09.944153+0000","last_clean":"2024-02-21T23:06:09.944153+0000","last_became_active":"2024-02-21T23:05:51.410503+0000","last_became_peered":"2024-02-21T23:05:51.410503+0000","last_unstale":"2024-02-21T23:06:09.944153+0000","last_undegraded":"2024-02-21T23:06:09.944153+0000","last_fullsized":"2024-02-21T23:06:09.944153+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:51:10.610852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.944912+0000","last_change":"2024-02-21T23:05:50.376888+0000","last_active":"2024-02-21T23:06:09.944912+0000","last_peered":"2024-02-21T23:06:09.944912+0000","last_clean":"2024-02-21T23:06:09.944912+0000","last_became_active":"2024-02-21T23:05:50.376610+0000","last_became_peered":"2024-02-21T23:05:50.376610+0000","last_unstale":"2024-02-21T23:06:09.944912+0000","last_undegraded":"2024-02-21T23:06:09.944912+0000","last_fullsized":"2024-02-21T23:06:09.944912+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:51:16.181060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.925072+0000","last_change":"2024-02-21T23:06:09.925072+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.925072+0000","last_undegraded":"2024-02-21T23:06:09.925072+0000","last_fullsized":"2024-02-21T23:06:09.925072+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.958272+0000","last_change":"2024-02-21T23:06:09.952555+0000","last_active":"2024-02-21T23:06:09.958272+0000","last_peered":"2024-02-21T23:06:09.958272+0000","last_clean":"2024-02-21T23:06:09.958272+0000","last_became_active":"2024-02-21T23:06:09.952281+0000","last_became_peered":"2024-02-21T23:06:09.952281+0000","last_unstale":"2024-02-21T23:06:09.958272+0000","last_undegraded":"2024-02-21T23:06:09.958272+0000","last_fullsized":"2024-02-21T23:06:09.958272+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:50:04.585772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.937015+0000","last_change":"2024-02-21T23:05:51.417167+0000","last_active":"2024-02-21T23:06:09.937015+0000","last_peered":"2024-02-21T23:06:09.937015+0000","last_clean":"2024-02-21T23:06:09.937015+0000","last_became_active":"2024-02-21T23:05:51.415957+0000","last_became_peered":"2024-02-21T23:05:51.415957+0000","last_unstale":"2024-02-21T23:06:09.937015+0000","last_undegraded":"2024-02-21T23:06:09.937015+0000","last_fullsized":"2024-02-21T23:06:09.937015+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:26:59.088824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.942505+0000","last_change":"2024-02-21T23:05:50.375808+0000","last_active":"2024-02-21T23:06:09.942505+0000","last_peered":"2024-02-21T23:06:09.942505+0000","last_clean":"2024-02-21T23:06:09.942505+0000","last_became_active":"2024-02-21T23:05:50.375467+0000","last_became_peered":"2024-02-21T23:05:50.375467+0000","last_unstale":"2024-02-21T23:06:09.942505+0000","last_undegraded":"2024-02-21T23:06:09.942505+0000","last_fullsized":"2024-02-21T23:06:09.942505+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:29:01.979463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.923630+0000","last_change":"2024-02-21T23:06:09.923630+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.923630+0000","last_undegraded":"2024-02-21T23:06:09.923630+0000","last_fullsized":"2024-02-21T23:06:09.923630+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.959070+0000","last_change":"2024-02-21T23:06:09.956377+0000","last_active":"2024-02-21T23:06:09.959070+0000","last_peered":"2024-02-21T23:06:09.959070+0000","last_clean":"2024-02-21T23:06:09.959070+0000","last_became_active":"2024-02-21T23:06:09.956129+0000","last_became_peered":"2024-02-21T23:06:09.956129+0000","last_unstale":"2024-02-21T23:06:09.959070+0000","last_undegraded":"2024-02-21T23:06:09.959070+0000","last_fullsized":"2024-02-21T23:06:09.959070+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:29:15.656090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.946180+0000","last_change":"2024-02-21T23:05:51.414453+0000","last_active":"2024-02-21T23:06:09.946180+0000","last_peered":"2024-02-21T23:06:09.946180+0000","last_clean":"2024-02-21T23:06:09.946180+0000","last_became_active":"2024-02-21T23:05:51.413499+0000","last_became_peered":"2024-02-21T23:05:51.413499+0000","last_unstale":"2024-02-21T23:06:09.946180+0000","last_undegraded":"2024-02-21T23:06:09.946180+0000","last_fullsized":"2024-02-21T23:06:09.946180+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:55:21.967274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"75'2","reported_seq":20,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.934245+0000","last_change":"2024-02-21T23:05:50.381948+0000","last_active":"2024-02-21T23:06:09.934245+0000","last_peered":"2024-02-21T23:06:09.934245+0000","last_clean":"2024-02-21T23:06:09.934245+0000","last_became_active":"2024-02-21T23:05:50.381452+0000","last_became_peered":"2024-02-21T23:05:50.381452+0000","last_unstale":"2024-02-21T23:06:09.934245+0000","last_undegraded":"2024-02-21T23:06:09.934245+0000","last_fullsized":"2024-02-21T23:06:09.934245+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:22:16.414686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.935470+0000","last_change":"2024-02-21T23:06:09.935470+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.935470+0000","last_undegraded":"2024-02-21T23:06:09.935470+0000","last_fullsized":"2024-02-21T23:06:09.935470+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.958416+0000","last_change":"2024-02-21T23:06:09.953924+0000","last_active":"2024-02-21T23:06:09.958416+0000","last_peered":"2024-02-21T23:06:09.958416+0000","last_clean":"2024-02-21T23:06:09.958416+0000","last_became_active":"2024-02-21T23:06:09.952952+0000","last_became_peered":"2024-02-21T23:06:09.952952+0000","last_unstale":"2024-02-21T23:06:09.958416+0000","last_undegraded":"2024-02-21T23:06:09.958416+0000","last_fullsized":"2024-02-21T23:06:09.958416+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:19:59.841109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.934595+0000","last_change":"2024-02-21T23:05:51.410619+0000","last_active":"2024-02-21T23:06:09.934595+0000","last_peered":"2024-02-21T23:06:09.934595+0000","last_clean":"2024-02-21T23:06:09.934595+0000","last_became_active":"2024-02-21T23:05:51.409937+0000","last_became_peered":"2024-02-21T23:05:51.409937+0000","last_unstale":"2024-02-21T23:06:09.934595+0000","last_undegraded":"2024-02-21T23:06:09.934595+0000","last_fullsized":"2024-02-21T23:06:09.934595+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:22:32.845173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"76'1","reported_seq":19,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.944263+0000","last_change":"2024-02-21T23:05:50.362370+0000","last_active":"2024-02-21T23:06:09.944263+0000","last_peered":"2024-02-21T23:06:09.944263+0000","last_clean":"2024-02-21T23:06:09.944263+0000","last_became_active":"2024-02-21T23:05:50.361848+0000","last_became_peered":"2024-02-21T23:05:50.361848+0000","last_unstale":"2024-02-21T23:06:09.944263+0000","last_undegraded":"2024-02-21T23:06:09.944263+0000","last_fullsized":"2024-02-21T23:06:09.944263+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:08:35.448428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.930610+0000","last_change":"2024-02-21T23:06:09.930610+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.930610+0000","last_undegraded":"2024-02-21T23:06:09.930610+0000","last_fullsized":"2024-02-21T23:06:09.930610+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.963409+0000","last_change":"2024-02-21T23:06:09.961946+0000","last_active":"2024-02-21T23:06:09.963409+0000","last_peered":"2024-02-21T23:06:09.963409+0000","last_clean":"2024-02-21T23:06:09.963409+0000","last_became_active":"2024-02-21T23:06:09.961662+0000","last_became_peered":"2024-02-21T23:06:09.961662+0000","last_unstale":"2024-02-21T23:06:09.963409+0000","last_undegraded":"2024-02-21T23:06:09.963409+0000","last_fullsized":"2024-02-21T23:06:09.963409+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:15:58.947158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.946318+0000","last_change":"2024-02-21T23:05:51.421357+0000","last_active":"2024-02-21T23:06:09.946318+0000","last_peered":"2024-02-21T23:06:09.946318+0000","last_clean":"2024-02-21T23:06:09.946318+0000","last_became_active":"2024-02-21T23:05:51.420734+0000","last_became_peered":"2024-02-21T23:05:51.420734+0000","last_unstale":"2024-02-21T23:06:09.946318+0000","last_undegraded":"2024-02-21T23:06:09.946318+0000","last_fullsized":"2024-02-21T23:06:09.946318+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:31:44.536584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.944953+0000","last_change":"2024-02-21T23:05:50.384680+0000","last_active":"2024-02-21T23:06:09.944953+0000","last_peered":"2024-02-21T23:06:09.944953+0000","last_clean":"2024-02-21T23:06:09.944953+0000","last_became_active":"2024-02-21T23:05:50.384208+0000","last_became_peered":"2024-02-21T23:05:50.384208+0000","last_unstale":"2024-02-21T23:06:09.944953+0000","last_undegraded":"2024-02-21T23:06:09.944953+0000","last_fullsized":"2024-02-21T23:06:09.944953+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:01:27.369089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.933769+0000","last_change":"2024-02-21T23:06:09.933769+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.933769+0000","last_undegraded":"2024-02-21T23:06:09.933769+0000","last_fullsized":"2024-02-21T23:06:09.933769+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.961163+0000","last_change":"2024-02-21T23:06:09.952597+0000","last_active":"2024-02-21T23:06:09.961163+0000","last_peered":"2024-02-21T23:06:09.961163+0000","last_clean":"2024-02-21T23:06:09.961163+0000","last_became_active":"2024-02-21T23:06:09.952270+0000","last_became_peered":"2024-02-21T23:06:09.952270+0000","last_unstale":"2024-02-21T23:06:09.961163+0000","last_undegraded":"2024-02-21T23:06:09.961163+0000","last_fullsized":"2024-02-21T23:06:09.961163+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:42:45.049032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.944384+0000","last_change":"2024-02-21T23:05:51.395859+0000","last_active":"2024-02-21T23:06:09.944384+0000","last_peered":"2024-02-21T23:06:09.944384+0000","last_clean":"2024-02-21T23:06:09.944384+0000","last_became_active":"2024-02-21T23:05:51.395012+0000","last_became_peered":"2024-02-21T23:05:51.395012+0000","last_unstale":"2024-02-21T23:06:09.944384+0000","last_undegraded":"2024-02-21T23:06:09.944384+0000","last_fullsized":"2024-02-21T23:06:09.944384+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:29:02.992284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.943897+0000","last_change":"2024-02-21T23:05:50.383920+0000","last_active":"2024-02-21T23:06:09.943897+0000","last_peered":"2024-02-21T23:06:09.943897+0000","last_clean":"2024-02-21T23:06:09.943897+0000","last_became_active":"2024-02-21T23:05:50.382488+0000","last_became_peered":"2024-02-21T23:05:50.382488+0000","last_unstale":"2024-02-21T23:06:09.943897+0000","last_undegraded":"2024-02-21T23:06:09.943897+0000","last_fullsized":"2024-02-21T23:06:09.943897+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:26:58.076002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.937602+0000","last_change":"2024-02-21T23:06:09.937602+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.937602+0000","last_undegraded":"2024-02-21T23:06:09.937602+0000","last_fullsized":"2024-02-21T23:06:09.937602+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.966343+0000","last_change":"2024-02-21T23:06:09.961904+0000","last_active":"2024-02-21T23:06:09.966343+0000","last_peered":"2024-02-21T23:06:09.966343+0000","last_clean":"2024-02-21T23:06:09.966343+0000","last_became_active":"2024-02-21T23:06:09.961650+0000","last_became_peered":"2024-02-21T23:06:09.961650+0000","last_unstale":"2024-02-21T23:06:09.966343+0000","last_undegraded":"2024-02-21T23:06:09.966343+0000","last_fullsized":"2024-02-21T23:06:09.966343+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:43:37.070238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.932210+0000","last_change":"2024-02-21T23:05:51.422495+0000","last_active":"2024-02-21T23:06:09.932210+0000","last_peered":"2024-02-21T23:06:09.932210+0000","last_clean":"2024-02-21T23:06:09.932210+0000","last_became_active":"2024-02-21T23:05:51.422196+0000","last_became_peered":"2024-02-21T23:05:51.422196+0000","last_unstale":"2024-02-21T23:06:09.932210+0000","last_undegraded":"2024-02-21T23:06:09.932210+0000","last_fullsized":"2024-02-21T23:06:09.932210+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:10:11.986389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.932846+0000","last_change":"2024-02-21T23:05:50.376027+0000","last_active":"2024-02-21T23:06:09.932846+0000","last_peered":"2024-02-21T23:06:09.932846+0000","last_clean":"2024-02-21T23:06:09.932846+0000","last_became_active":"2024-02-21T23:05:50.375170+0000","last_became_peered":"2024-02-21T23:05:50.375170+0000","last_unstale":"2024-02-21T23:06:09.932846+0000","last_undegraded":"2024-02-21T23:06:09.932846+0000","last_fullsized":"2024-02-21T23:06:09.932846+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:39:32.228635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.924758+0000","last_change":"2024-02-21T23:06:09.924758+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.924758+0000","last_undegraded":"2024-02-21T23:06:09.924758+0000","last_fullsized":"2024-02-21T23:06:09.924758+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.954412+0000","last_change":"2024-02-21T23:06:09.947879+0000","last_active":"2024-02-21T23:06:09.954412+0000","last_peered":"2024-02-21T23:06:09.954412+0000","last_clean":"2024-02-21T23:06:09.954412+0000","last_became_active":"2024-02-21T23:06:09.947599+0000","last_became_peered":"2024-02-21T23:06:09.947599+0000","last_unstale":"2024-02-21T23:06:09.954412+0000","last_undegraded":"2024-02-21T23:06:09.954412+0000","last_fullsized":"2024-02-21T23:06:09.954412+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:50:51.483608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.937296+0000","last_change":"2024-02-21T23:05:51.420500+0000","last_active":"2024-02-21T23:06:09.937296+0000","last_peered":"2024-02-21T23:06:09.937296+0000","last_clean":"2024-02-21T23:06:09.937296+0000","last_became_active":"2024-02-21T23:05:51.420103+0000","last_became_peered":"2024-02-21T23:05:51.420103+0000","last_unstale":"2024-02-21T23:06:09.937296+0000","last_undegraded":"2024-02-21T23:06:09.937296+0000","last_fullsized":"2024-02-21T23:06:09.937296+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:27:01.191337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.937594+0000","last_change":"2024-02-21T23:05:50.354805+0000","last_active":"2024-02-21T23:06:09.937594+0000","last_peered":"2024-02-21T23:06:09.937594+0000","last_clean":"2024-02-21T23:06:09.937594+0000","last_became_active":"2024-02-21T23:05:50.354525+0000","last_became_peered":"2024-02-21T23:05:50.354525+0000","last_unstale":"2024-02-21T23:06:09.937594+0000","last_undegraded":"2024-02-21T23:06:09.937594+0000","last_fullsized":"2024-02-21T23:06:09.937594+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:28:05.314552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.923982+0000","last_change":"2024-02-21T23:06:09.923982+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.923982+0000","last_undegraded":"2024-02-21T23:06:09.923982+0000","last_fullsized":"2024-02-21T23:06:09.923982+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.958392+0000","last_change":"2024-02-21T23:06:09.955164+0000","last_active":"2024-02-21T23:06:09.958392+0000","last_peered":"2024-02-21T23:06:09.958392+0000","last_clean":"2024-02-21T23:06:09.958392+0000","last_became_active":"2024-02-21T23:06:09.954858+0000","last_became_peered":"2024-02-21T23:06:09.954858+0000","last_unstale":"2024-02-21T23:06:09.958392+0000","last_undegraded":"2024-02-21T23:06:09.958392+0000","last_fullsized":"2024-02-21T23:06:09.958392+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:51:29.171374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.920149+0000","last_change":"2024-02-21T23:05:51.416548+0000","last_active":"2024-02-21T23:06:09.920149+0000","last_peered":"2024-02-21T23:06:09.920149+0000","last_clean":"2024-02-21T23:06:09.920149+0000","last_became_active":"2024-02-21T23:05:51.416222+0000","last_became_peered":"2024-02-21T23:05:51.416222+0000","last_unstale":"2024-02-21T23:06:09.920149+0000","last_undegraded":"2024-02-21T23:06:09.920149+0000","last_fullsized":"2024-02-21T23:06:09.920149+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:22:32.845173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.934813+0000","last_change":"2024-02-21T23:05:50.373554+0000","last_active":"2024-02-21T23:06:09.934813+0000","last_peered":"2024-02-21T23:06:09.934813+0000","last_clean":"2024-02-21T23:06:09.934813+0000","last_became_active":"2024-02-21T23:05:50.373022+0000","last_became_peered":"2024-02-21T23:05:50.373022+0000","last_unstale":"2024-02-21T23:06:09.934813+0000","last_undegraded":"2024-02-21T23:06:09.934813+0000","last_fullsized":"2024-02-21T23:06:09.934813+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:38:56.665723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.941480+0000","last_change":"2024-02-21T23:06:09.941480+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.941480+0000","last_undegraded":"2024-02-21T23:06:09.941480+0000","last_fullsized":"2024-02-21T23:06:09.941480+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.957795+0000","last_change":"2024-02-21T23:06:09.953218+0000","last_active":"2024-02-21T23:06:09.957795+0000","last_peered":"2024-02-21T23:06:09.957795+0000","last_clean":"2024-02-21T23:06:09.957795+0000","last_became_active":"2024-02-21T23:06:09.952908+0000","last_became_peered":"2024-02-21T23:06:09.952908+0000","last_unstale":"2024-02-21T23:06:09.957795+0000","last_undegraded":"2024-02-21T23:06:09.957795+0000","last_fullsized":"2024-02-21T23:06:09.957795+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:08:55.021772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.920356+0000","last_change":"2024-02-21T23:05:51.412683+0000","last_active":"2024-02-21T23:06:09.920356+0000","last_peered":"2024-02-21T23:06:09.920356+0000","last_clean":"2024-02-21T23:06:09.920356+0000","last_became_active":"2024-02-21T23:05:51.412389+0000","last_became_peered":"2024-02-21T23:05:51.412389+0000","last_unstale":"2024-02-21T23:06:09.920356+0000","last_undegraded":"2024-02-21T23:06:09.920356+0000","last_fullsized":"2024-02-21T23:06:09.920356+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:57:06.211024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.918795+0000","last_change":"2024-02-21T23:05:50.376356+0000","last_active":"2024-02-21T23:06:09.918795+0000","last_peered":"2024-02-21T23:06:09.918795+0000","last_clean":"2024-02-21T23:06:09.918795+0000","last_became_active":"2024-02-21T23:05:50.370467+0000","last_became_peered":"2024-02-21T23:05:50.370467+0000","last_unstale":"2024-02-21T23:06:09.918795+0000","last_undegraded":"2024-02-21T23:06:09.918795+0000","last_fullsized":"2024-02-21T23:06:09.918795+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:56:49.435540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-02-21T23:06:09.940614+0000","last_change":"2024-02-21T23:06:09.940614+0000","last_active":"2024-02-21T23:06:09.906800+0000","last_peered":"2024-02-21T23:06:09.906800+0000","last_clean":"2024-02-21T23:06:09.906800+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:06:09.940614+0000","last_undegraded":"2024-02-21T23:06:09.940614+0000","last_fullsized":"2024-02-21T23:06:09.940614+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":9,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.945283+0000","last_change":"2024-02-21T23:06:09.940641+0000","last_active":"2024-02-21T23:06:09.945283+0000","last_peered":"2024-02-21T23:06:09.945283+0000","last_clean":"2024-02-21T23:06:09.945283+0000","last_became_active":"2024-02-21T23:06:09.940353+0000","last_became_peered":"2024-02-21T23:06:09.940353+0000","last_unstale":"2024-02-21T23:06:09.945283+0000","last_undegraded":"2024-02-21T23:06:09.945283+0000","last_fullsized":"2024-02-21T23:06:09.945283+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:45:25.410745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":16,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.944841+0000","last_change":"2024-02-21T23:05:51.413008+0000","last_active":"2024-02-21T23:06:09.944841+0000","last_peered":"2024-02-21T23:06:09.944841+0000","last_clean":"2024-02-21T23:06:09.944841+0000","last_became_active":"2024-02-21T23:05:51.412714+0000","last_became_peered":"2024-02-21T23:05:51.412714+0000","last_unstale":"2024-02-21T23:06:09.944841+0000","last_undegraded":"2024-02-21T23:06:09.944841+0000","last_fullsized":"2024-02-21T23:06:09.944841+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:03:08.789066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":18,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.946178+0000","last_change":"2024-02-21T23:05:50.375581+0000","last_active":"2024-02-21T23:06:09.946178+0000","last_peered":"2024-02-21T23:06:09.946178+0000","last_clean":"2024-02-21T23:06:09.946178+0000","last_became_active":"2024-02-21T23:05:50.375282+0000","last_became_peered":"2024-02-21T23:05:50.375282+0000","last_unstale":"2024-02-21T23:06:09.946178+0000","last_undegraded":"2024-02-21T23:06:09.946178+0000","last_fullsized":"2024-02-21T23:06:09.946178+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:49:45.012428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":278,"reported_epoch":78,"state":"active+clean","last_fresh":"2024-02-21T23:06:09.942188+0000","last_change":"2024-02-21T23:02:32.289335+0000","last_active":"2024-02-21T23:06:09.942188+0000","last_peered":"2024-02-21T23:06:09.942188+0000","last_clean":"2024-02-21T23:06:09.942188+0000","last_became_active":"2024-02-21T23:02:32.286913+0000","last_became_peered":"2024-02-21T23:02:32.286913+0000","last_unstale":"2024-02-21T23:06:09.942188+0000","last_undegraded":"2024-02-21T23:06:09.942188+0000","last_fullsized":"2024-02-21T23:06:09.942188+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_clean_scrub_stamp":"2024-02-21T23:02:31.172569+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:36:23.987374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":14,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574894,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32180,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339660,"statfs":{"total":96636764160,"available":96603811840,"internally_reserved":0,"allocated":1024000,"data_stored":1048493,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574895,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32128,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339712,"statfs":{"total":96636764160,"available":96603865088,"internally_reserved":0,"allocated":966656,"data_stored":459715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574894,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27960,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343880,"statfs":{"total":96636764160,"available":96608133120,"internally_reserved":0,"allocated":958464,"data_stored":458137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574895,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32136,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339704,"statfs":{"total":96636764160,"available":96603856896,"internally_reserved":0,"allocated":974848,"data_stored":459819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574895,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32120,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339720,"statfs":{"total":96636764160,"available":96603873280,"internally_reserved":0,"allocated":958464,"data_stored":458685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574896,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32128,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339712,"statfs":{"total":96636764160,"available":96603865088,"internally_reserved":0,"allocated":966656,"data_stored":458845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574894,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32108,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339732,"statfs":{"total":96636764160,"available":96603885568,"internally_reserved":0,"allocated":950272,"data_stored":450693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574894,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32176,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339664,"statfs":{"total":96636764160,"available":96603815936,"internally_reserved":0,"allocated":1019904,"data_stored":1048471,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-21T23:06:16.685 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.3d is not active+clean 2024-02-21T23:06:16.685 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:06:09.940039+0000', 'last_change': '2024-02-21T23:06:09.940039+0000', 'last_active': '2024-02-21T23:06:09.906800+0000', 'last_peered': '2024-02-21T23:06:09.906800+0000', 'last_clean': '2024-02-21T23:06:09.906800+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:06:09.940039+0000', 'last_undegraded': '2024-02-21T23:06:09.940039+0000', 'last_fullsized': '2024-02-21T23:06:09.940039+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_clean_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-02-21T23:06:16.685 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.3c is not active+clean 2024-02-21T23:06:16.686 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:06:09.931414+0000', 'last_change': '2024-02-21T23:06:09.931414+0000', 'last_active': '2024-02-21T23:06:09.906800+0000', 'last_peered': '2024-02-21T23:06:09.906800+0000', 'last_clean': '2024-02-21T23:06:09.906800+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:06:09.931414+0000', 'last_undegraded': '2024-02-21T23:06:09.931414+0000', 'last_fullsized': '2024-02-21T23:06:09.931414+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_clean_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-02-21T23:06:16.686 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.3f is not active+clean 2024-02-21T23:06:16.686 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:06:09.936313+0000', 'last_change': '2024-02-21T23:06:09.936313+0000', 'last_active': '2024-02-21T23:06:09.906800+0000', 'last_peered': '2024-02-21T23:06:09.906800+0000', 'last_clean': '2024-02-21T23:06:09.906800+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:06:09.936313+0000', 'last_undegraded': '2024-02-21T23:06:09.936313+0000', 'last_fullsized': '2024-02-21T23:06:09.936313+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_clean_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-02-21T23:06:16.686 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.3e is not active+clean 2024-02-21T23:06:16.686 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:06:09.924703+0000', 'last_change': '2024-02-21T23:06:09.924703+0000', 'last_active': '2024-02-21T23:06:09.906800+0000', 'last_peered': '2024-02-21T23:06:09.906800+0000', 'last_clean': '2024-02-21T23:06:09.906800+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:06:09.924703+0000', 'last_undegraded': '2024-02-21T23:06:09.924703+0000', 'last_fullsized': '2024-02-21T23:06:09.924703+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_clean_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-02-21T23:06:16.686 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.39 is not active+clean 2024-02-21T23:06:16.686 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:06:09.936144+0000', 'last_change': '2024-02-21T23:06:09.936144+0000', 'last_active': '2024-02-21T23:06:09.906800+0000', 'last_peered': '2024-02-21T23:06:09.906800+0000', 'last_clean': '2024-02-21T23:06:09.906800+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:06:09.936144+0000', 'last_undegraded': '2024-02-21T23:06:09.936144+0000', 'last_fullsized': '2024-02-21T23:06:09.936144+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_clean_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-02-21T23:06:16.686 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.38 is not active+clean 2024-02-21T23:06:16.686 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:06:09.932838+0000', 'last_change': '2024-02-21T23:06:09.932838+0000', 'last_active': '2024-02-21T23:06:09.906800+0000', 'last_peered': '2024-02-21T23:06:09.906800+0000', 'last_clean': '2024-02-21T23:06:09.906800+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:06:09.932838+0000', 'last_undegraded': '2024-02-21T23:06:09.932838+0000', 'last_fullsized': '2024-02-21T23:06:09.932838+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_clean_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-02-21T23:06:16.686 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.3b is not active+clean 2024-02-21T23:06:16.687 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:06:09.943914+0000', 'last_change': '2024-02-21T23:06:09.943914+0000', 'last_active': '2024-02-21T23:06:09.906800+0000', 'last_peered': '2024-02-21T23:06:09.906800+0000', 'last_clean': '2024-02-21T23:06:09.906800+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:06:09.943914+0000', 'last_undegraded': '2024-02-21T23:06:09.943914+0000', 'last_fullsized': '2024-02-21T23:06:09.943914+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_clean_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-02-21T23:06:16.687 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.3a is not active+clean 2024-02-21T23:06:16.687 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:06:09.940874+0000', 'last_change': '2024-02-21T23:06:09.940874+0000', 'last_active': '2024-02-21T23:06:09.906800+0000', 'last_peered': '2024-02-21T23:06:09.906800+0000', 'last_clean': '2024-02-21T23:06:09.906800+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:06:09.940874+0000', 'last_undegraded': '2024-02-21T23:06:09.940874+0000', 'last_fullsized': '2024-02-21T23:06:09.940874+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_clean_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-02-21T23:06:16.687 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.35 is not active+clean 2024-02-21T23:06:16.687 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:06:09.938982+0000', 'last_change': '2024-02-21T23:06:09.938982+0000', 'last_active': '2024-02-21T23:06:09.906800+0000', 'last_peered': '2024-02-21T23:06:09.906800+0000', 'last_clean': '2024-02-21T23:06:09.906800+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:06:09.938982+0000', 'last_undegraded': '2024-02-21T23:06:09.938982+0000', 'last_fullsized': '2024-02-21T23:06:09.938982+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_clean_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-02-21T23:06:16.687 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.34 is not active+clean 2024-02-21T23:06:16.687 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:06:09.937566+0000', 'last_change': '2024-02-21T23:06:09.937566+0000', 'last_active': '2024-02-21T23:06:09.906800+0000', 'last_peered': '2024-02-21T23:06:09.906800+0000', 'last_clean': '2024-02-21T23:06:09.906800+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:06:09.937566+0000', 'last_undegraded': '2024-02-21T23:06:09.937566+0000', 'last_fullsized': '2024-02-21T23:06:09.937566+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_clean_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-02-21T23:06:16.687 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.37 is not active+clean 2024-02-21T23:06:16.687 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:06:09.936387+0000', 'last_change': '2024-02-21T23:06:09.936387+0000', 'last_active': '2024-02-21T23:06:09.906800+0000', 'last_peered': '2024-02-21T23:06:09.906800+0000', 'last_clean': '2024-02-21T23:06:09.906800+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:06:09.936387+0000', 'last_undegraded': '2024-02-21T23:06:09.936387+0000', 'last_fullsized': '2024-02-21T23:06:09.936387+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_clean_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-02-21T23:06:16.687 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.36 is not active+clean 2024-02-21T23:06:16.687 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:06:09.932127+0000', 'last_change': '2024-02-21T23:06:09.932127+0000', 'last_active': '2024-02-21T23:06:09.906800+0000', 'last_peered': '2024-02-21T23:06:09.906800+0000', 'last_clean': '2024-02-21T23:06:09.906800+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:06:09.932127+0000', 'last_undegraded': '2024-02-21T23:06:09.932127+0000', 'last_fullsized': '2024-02-21T23:06:09.932127+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_clean_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-02-21T23:06:16.687 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.31 is not active+clean 2024-02-21T23:06:16.688 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:06:09.929205+0000', 'last_change': '2024-02-21T23:06:09.929205+0000', 'last_active': '2024-02-21T23:06:09.906800+0000', 'last_peered': '2024-02-21T23:06:09.906800+0000', 'last_clean': '2024-02-21T23:06:09.906800+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:06:09.929205+0000', 'last_undegraded': '2024-02-21T23:06:09.929205+0000', 'last_fullsized': '2024-02-21T23:06:09.929205+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_clean_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-02-21T23:06:16.688 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.30 is not active+clean 2024-02-21T23:06:16.688 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:06:09.929405+0000', 'last_change': '2024-02-21T23:06:09.929405+0000', 'last_active': '2024-02-21T23:06:09.906800+0000', 'last_peered': '2024-02-21T23:06:09.906800+0000', 'last_clean': '2024-02-21T23:06:09.906800+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:06:09.929405+0000', 'last_undegraded': '2024-02-21T23:06:09.929405+0000', 'last_fullsized': '2024-02-21T23:06:09.929405+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_clean_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-02-21T23:06:16.688 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.33 is not active+clean 2024-02-21T23:06:16.688 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:06:09.929330+0000', 'last_change': '2024-02-21T23:06:09.929330+0000', 'last_active': '2024-02-21T23:06:09.906800+0000', 'last_peered': '2024-02-21T23:06:09.906800+0000', 'last_clean': '2024-02-21T23:06:09.906800+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:06:09.929330+0000', 'last_undegraded': '2024-02-21T23:06:09.929330+0000', 'last_fullsized': '2024-02-21T23:06:09.929330+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_clean_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-02-21T23:06:16.688 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.32 is not active+clean 2024-02-21T23:06:16.688 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:06:09.942772+0000', 'last_change': '2024-02-21T23:06:09.942772+0000', 'last_active': '2024-02-21T23:06:09.906800+0000', 'last_peered': '2024-02-21T23:06:09.906800+0000', 'last_clean': '2024-02-21T23:06:09.906800+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:06:09.942772+0000', 'last_undegraded': '2024-02-21T23:06:09.942772+0000', 'last_fullsized': '2024-02-21T23:06:09.942772+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_clean_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-02-21T23:06:16.688 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.2d is not active+clean 2024-02-21T23:06:16.688 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:06:09.938385+0000', 'last_change': '2024-02-21T23:06:09.938385+0000', 'last_active': '2024-02-21T23:06:09.906800+0000', 'last_peered': '2024-02-21T23:06:09.906800+0000', 'last_clean': '2024-02-21T23:06:09.906800+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:06:09.938385+0000', 'last_undegraded': '2024-02-21T23:06:09.938385+0000', 'last_fullsized': '2024-02-21T23:06:09.938385+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_clean_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-02-21T23:06:16.688 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.2c is not active+clean 2024-02-21T23:06:16.688 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:06:09.929202+0000', 'last_change': '2024-02-21T23:06:09.929202+0000', 'last_active': '2024-02-21T23:06:09.906800+0000', 'last_peered': '2024-02-21T23:06:09.906800+0000', 'last_clean': '2024-02-21T23:06:09.906800+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:06:09.929202+0000', 'last_undegraded': '2024-02-21T23:06:09.929202+0000', 'last_fullsized': '2024-02-21T23:06:09.929202+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_clean_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-02-21T23:06:16.689 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.2f is not active+clean 2024-02-21T23:06:16.689 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:06:09.939567+0000', 'last_change': '2024-02-21T23:06:09.939567+0000', 'last_active': '2024-02-21T23:06:09.906800+0000', 'last_peered': '2024-02-21T23:06:09.906800+0000', 'last_clean': '2024-02-21T23:06:09.906800+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:06:09.939567+0000', 'last_undegraded': '2024-02-21T23:06:09.939567+0000', 'last_fullsized': '2024-02-21T23:06:09.939567+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_clean_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-02-21T23:06:16.689 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.2e is not active+clean 2024-02-21T23:06:16.689 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:06:09.941381+0000', 'last_change': '2024-02-21T23:06:09.941381+0000', 'last_active': '2024-02-21T23:06:09.906800+0000', 'last_peered': '2024-02-21T23:06:09.906800+0000', 'last_clean': '2024-02-21T23:06:09.906800+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:06:09.941381+0000', 'last_undegraded': '2024-02-21T23:06:09.941381+0000', 'last_fullsized': '2024-02-21T23:06:09.941381+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_clean_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-02-21T23:06:16.689 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.29 is not active+clean 2024-02-21T23:06:16.689 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:06:09.929062+0000', 'last_change': '2024-02-21T23:06:09.929062+0000', 'last_active': '2024-02-21T23:06:09.906800+0000', 'last_peered': '2024-02-21T23:06:09.906800+0000', 'last_clean': '2024-02-21T23:06:09.906800+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:06:09.929062+0000', 'last_undegraded': '2024-02-21T23:06:09.929062+0000', 'last_fullsized': '2024-02-21T23:06:09.929062+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_clean_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-02-21T23:06:16.689 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.28 is not active+clean 2024-02-21T23:06:16.689 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:06:09.932618+0000', 'last_change': '2024-02-21T23:06:09.932618+0000', 'last_active': '2024-02-21T23:06:09.906800+0000', 'last_peered': '2024-02-21T23:06:09.906800+0000', 'last_clean': '2024-02-21T23:06:09.906800+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:06:09.932618+0000', 'last_undegraded': '2024-02-21T23:06:09.932618+0000', 'last_fullsized': '2024-02-21T23:06:09.932618+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_clean_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-02-21T23:06:16.689 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.2b is not active+clean 2024-02-21T23:06:16.689 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:06:09.940735+0000', 'last_change': '2024-02-21T23:06:09.940735+0000', 'last_active': '2024-02-21T23:06:09.906800+0000', 'last_peered': '2024-02-21T23:06:09.906800+0000', 'last_clean': '2024-02-21T23:06:09.906800+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:06:09.940735+0000', 'last_undegraded': '2024-02-21T23:06:09.940735+0000', 'last_fullsized': '2024-02-21T23:06:09.940735+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_clean_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-02-21T23:06:16.689 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.2a is not active+clean 2024-02-21T23:06:16.690 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:06:09.941237+0000', 'last_change': '2024-02-21T23:06:09.941237+0000', 'last_active': '2024-02-21T23:06:09.906800+0000', 'last_peered': '2024-02-21T23:06:09.906800+0000', 'last_clean': '2024-02-21T23:06:09.906800+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:06:09.941237+0000', 'last_undegraded': '2024-02-21T23:06:09.941237+0000', 'last_fullsized': '2024-02-21T23:06:09.941237+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_clean_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-02-21T23:06:16.690 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.25 is not active+clean 2024-02-21T23:06:16.690 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:06:09.931274+0000', 'last_change': '2024-02-21T23:06:09.931274+0000', 'last_active': '2024-02-21T23:06:09.906800+0000', 'last_peered': '2024-02-21T23:06:09.906800+0000', 'last_clean': '2024-02-21T23:06:09.906800+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:06:09.931274+0000', 'last_undegraded': '2024-02-21T23:06:09.931274+0000', 'last_fullsized': '2024-02-21T23:06:09.931274+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_clean_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 6], 'acting': [2, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-02-21T23:06:16.690 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.24 is not active+clean 2024-02-21T23:06:16.690 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:06:09.928558+0000', 'last_change': '2024-02-21T23:06:09.928558+0000', 'last_active': '2024-02-21T23:06:09.906800+0000', 'last_peered': '2024-02-21T23:06:09.906800+0000', 'last_clean': '2024-02-21T23:06:09.906800+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:06:09.928558+0000', 'last_undegraded': '2024-02-21T23:06:09.928558+0000', 'last_fullsized': '2024-02-21T23:06:09.928558+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_clean_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-02-21T23:06:16.690 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.27 is not active+clean 2024-02-21T23:06:16.690 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:06:09.925950+0000', 'last_change': '2024-02-21T23:06:09.925950+0000', 'last_active': '2024-02-21T23:06:09.906800+0000', 'last_peered': '2024-02-21T23:06:09.906800+0000', 'last_clean': '2024-02-21T23:06:09.906800+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:06:09.925950+0000', 'last_undegraded': '2024-02-21T23:06:09.925950+0000', 'last_fullsized': '2024-02-21T23:06:09.925950+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_clean_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-02-21T23:06:16.690 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.26 is not active+clean 2024-02-21T23:06:16.690 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:06:09.940425+0000', 'last_change': '2024-02-21T23:06:09.940425+0000', 'last_active': '2024-02-21T23:06:09.906800+0000', 'last_peered': '2024-02-21T23:06:09.906800+0000', 'last_clean': '2024-02-21T23:06:09.906800+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:06:09.940425+0000', 'last_undegraded': '2024-02-21T23:06:09.940425+0000', 'last_fullsized': '2024-02-21T23:06:09.940425+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_clean_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-02-21T23:06:16.690 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.21 is not active+clean 2024-02-21T23:06:16.690 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:06:09.922993+0000', 'last_change': '2024-02-21T23:06:09.922993+0000', 'last_active': '2024-02-21T23:06:09.906800+0000', 'last_peered': '2024-02-21T23:06:09.906800+0000', 'last_clean': '2024-02-21T23:06:09.906800+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:06:09.922993+0000', 'last_undegraded': '2024-02-21T23:06:09.922993+0000', 'last_fullsized': '2024-02-21T23:06:09.922993+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_clean_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-02-21T23:06:16.691 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.20 is not active+clean 2024-02-21T23:06:16.691 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:06:09.930360+0000', 'last_change': '2024-02-21T23:06:09.930360+0000', 'last_active': '2024-02-21T23:06:09.906800+0000', 'last_peered': '2024-02-21T23:06:09.906800+0000', 'last_clean': '2024-02-21T23:06:09.906800+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:06:09.930360+0000', 'last_undegraded': '2024-02-21T23:06:09.930360+0000', 'last_fullsized': '2024-02-21T23:06:09.930360+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_clean_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-02-21T23:06:16.691 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.23 is not active+clean 2024-02-21T23:06:16.691 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:06:09.935616+0000', 'last_change': '2024-02-21T23:06:09.935616+0000', 'last_active': '2024-02-21T23:06:09.906800+0000', 'last_peered': '2024-02-21T23:06:09.906800+0000', 'last_clean': '2024-02-21T23:06:09.906800+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:06:09.935616+0000', 'last_undegraded': '2024-02-21T23:06:09.935616+0000', 'last_fullsized': '2024-02-21T23:06:09.935616+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_clean_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-02-21T23:06:16.691 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.22 is not active+clean 2024-02-21T23:06:16.691 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:06:09.941990+0000', 'last_change': '2024-02-21T23:06:09.941990+0000', 'last_active': '2024-02-21T23:06:09.906800+0000', 'last_peered': '2024-02-21T23:06:09.906800+0000', 'last_clean': '2024-02-21T23:06:09.906800+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:06:09.941990+0000', 'last_undegraded': '2024-02-21T23:06:09.941990+0000', 'last_fullsized': '2024-02-21T23:06:09.941990+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_clean_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-02-21T23:06:16.691 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.1a is not active+clean 2024-02-21T23:06:16.691 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:06:09.940077+0000', 'last_change': '2024-02-21T23:06:09.940077+0000', 'last_active': '2024-02-21T23:06:09.906800+0000', 'last_peered': '2024-02-21T23:06:09.906800+0000', 'last_clean': '2024-02-21T23:06:09.906800+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:06:09.940077+0000', 'last_undegraded': '2024-02-21T23:06:09.940077+0000', 'last_fullsized': '2024-02-21T23:06:09.940077+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_clean_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-02-21T23:06:16.691 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.1b is not active+clean 2024-02-21T23:06:16.691 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:06:09.926981+0000', 'last_change': '2024-02-21T23:06:09.926981+0000', 'last_active': '2024-02-21T23:06:09.906800+0000', 'last_peered': '2024-02-21T23:06:09.906800+0000', 'last_clean': '2024-02-21T23:06:09.906800+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:06:09.926981+0000', 'last_undegraded': '2024-02-21T23:06:09.926981+0000', 'last_fullsized': '2024-02-21T23:06:09.926981+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_clean_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-02-21T23:06:16.691 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.18 is not active+clean 2024-02-21T23:06:16.691 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:06:09.927781+0000', 'last_change': '2024-02-21T23:06:09.927781+0000', 'last_active': '2024-02-21T23:06:09.906800+0000', 'last_peered': '2024-02-21T23:06:09.906800+0000', 'last_clean': '2024-02-21T23:06:09.906800+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:06:09.927781+0000', 'last_undegraded': '2024-02-21T23:06:09.927781+0000', 'last_fullsized': '2024-02-21T23:06:09.927781+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_clean_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-02-21T23:06:16.692 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.19 is not active+clean 2024-02-21T23:06:16.692 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:06:09.926671+0000', 'last_change': '2024-02-21T23:06:09.926671+0000', 'last_active': '2024-02-21T23:06:09.906800+0000', 'last_peered': '2024-02-21T23:06:09.906800+0000', 'last_clean': '2024-02-21T23:06:09.906800+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:06:09.926671+0000', 'last_undegraded': '2024-02-21T23:06:09.926671+0000', 'last_fullsized': '2024-02-21T23:06:09.926671+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_clean_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-02-21T23:06:16.692 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.1e is not active+clean 2024-02-21T23:06:16.692 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:06:09.937587+0000', 'last_change': '2024-02-21T23:06:09.937587+0000', 'last_active': '2024-02-21T23:06:09.906800+0000', 'last_peered': '2024-02-21T23:06:09.906800+0000', 'last_clean': '2024-02-21T23:06:09.906800+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:06:09.937587+0000', 'last_undegraded': '2024-02-21T23:06:09.937587+0000', 'last_fullsized': '2024-02-21T23:06:09.937587+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_clean_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-02-21T23:06:16.692 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.1f is not active+clean 2024-02-21T23:06:16.692 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:06:09.937636+0000', 'last_change': '2024-02-21T23:06:09.937636+0000', 'last_active': '2024-02-21T23:06:09.906800+0000', 'last_peered': '2024-02-21T23:06:09.906800+0000', 'last_clean': '2024-02-21T23:06:09.906800+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:06:09.937636+0000', 'last_undegraded': '2024-02-21T23:06:09.937636+0000', 'last_fullsized': '2024-02-21T23:06:09.937636+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_clean_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-02-21T23:06:16.692 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.1c is not active+clean 2024-02-21T23:06:16.692 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:06:09.943022+0000', 'last_change': '2024-02-21T23:06:09.943022+0000', 'last_active': '2024-02-21T23:06:09.906800+0000', 'last_peered': '2024-02-21T23:06:09.906800+0000', 'last_clean': '2024-02-21T23:06:09.906800+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:06:09.943022+0000', 'last_undegraded': '2024-02-21T23:06:09.943022+0000', 'last_fullsized': '2024-02-21T23:06:09.943022+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_clean_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-02-21T23:06:16.692 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.1d is not active+clean 2024-02-21T23:06:16.692 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:06:09.941977+0000', 'last_change': '2024-02-21T23:06:09.941977+0000', 'last_active': '2024-02-21T23:06:09.906800+0000', 'last_peered': '2024-02-21T23:06:09.906800+0000', 'last_clean': '2024-02-21T23:06:09.906800+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:06:09.941977+0000', 'last_undegraded': '2024-02-21T23:06:09.941977+0000', 'last_fullsized': '2024-02-21T23:06:09.941977+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_clean_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-02-21T23:06:16.693 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.12 is not active+clean 2024-02-21T23:06:16.693 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:06:09.936369+0000', 'last_change': '2024-02-21T23:06:09.936369+0000', 'last_active': '2024-02-21T23:06:09.906800+0000', 'last_peered': '2024-02-21T23:06:09.906800+0000', 'last_clean': '2024-02-21T23:06:09.906800+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:06:09.936369+0000', 'last_undegraded': '2024-02-21T23:06:09.936369+0000', 'last_fullsized': '2024-02-21T23:06:09.936369+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_clean_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-02-21T23:06:16.693 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.13 is not active+clean 2024-02-21T23:06:16.693 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:06:09.932767+0000', 'last_change': '2024-02-21T23:06:09.932767+0000', 'last_active': '2024-02-21T23:06:09.906800+0000', 'last_peered': '2024-02-21T23:06:09.906800+0000', 'last_clean': '2024-02-21T23:06:09.906800+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:06:09.932767+0000', 'last_undegraded': '2024-02-21T23:06:09.932767+0000', 'last_fullsized': '2024-02-21T23:06:09.932767+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_clean_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-02-21T23:06:16.693 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.10 is not active+clean 2024-02-21T23:06:16.693 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:06:09.927206+0000', 'last_change': '2024-02-21T23:06:09.927206+0000', 'last_active': '2024-02-21T23:06:09.906800+0000', 'last_peered': '2024-02-21T23:06:09.906800+0000', 'last_clean': '2024-02-21T23:06:09.906800+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:06:09.927206+0000', 'last_undegraded': '2024-02-21T23:06:09.927206+0000', 'last_fullsized': '2024-02-21T23:06:09.927206+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_clean_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-02-21T23:06:16.693 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.11 is not active+clean 2024-02-21T23:06:16.693 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:06:09.939836+0000', 'last_change': '2024-02-21T23:06:09.939836+0000', 'last_active': '2024-02-21T23:06:09.906800+0000', 'last_peered': '2024-02-21T23:06:09.906800+0000', 'last_clean': '2024-02-21T23:06:09.906800+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:06:09.939836+0000', 'last_undegraded': '2024-02-21T23:06:09.939836+0000', 'last_fullsized': '2024-02-21T23:06:09.939836+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_clean_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-02-21T23:06:16.693 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.16 is not active+clean 2024-02-21T23:06:16.693 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:06:09.938351+0000', 'last_change': '2024-02-21T23:06:09.938351+0000', 'last_active': '2024-02-21T23:06:09.906800+0000', 'last_peered': '2024-02-21T23:06:09.906800+0000', 'last_clean': '2024-02-21T23:06:09.906800+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:06:09.938351+0000', 'last_undegraded': '2024-02-21T23:06:09.938351+0000', 'last_fullsized': '2024-02-21T23:06:09.938351+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_clean_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-02-21T23:06:16.693 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.17 is not active+clean 2024-02-21T23:06:16.693 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:06:09.927030+0000', 'last_change': '2024-02-21T23:06:09.927030+0000', 'last_active': '2024-02-21T23:06:09.906800+0000', 'last_peered': '2024-02-21T23:06:09.906800+0000', 'last_clean': '2024-02-21T23:06:09.906800+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:06:09.927030+0000', 'last_undegraded': '2024-02-21T23:06:09.927030+0000', 'last_fullsized': '2024-02-21T23:06:09.927030+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_clean_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-02-21T23:06:16.694 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.14 is not active+clean 2024-02-21T23:06:16.694 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:06:09.939210+0000', 'last_change': '2024-02-21T23:06:09.939210+0000', 'last_active': '2024-02-21T23:06:09.906800+0000', 'last_peered': '2024-02-21T23:06:09.906800+0000', 'last_clean': '2024-02-21T23:06:09.906800+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:06:09.939210+0000', 'last_undegraded': '2024-02-21T23:06:09.939210+0000', 'last_fullsized': '2024-02-21T23:06:09.939210+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_clean_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-02-21T23:06:16.694 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.15 is not active+clean 2024-02-21T23:06:16.694 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:06:09.937280+0000', 'last_change': '2024-02-21T23:06:09.937280+0000', 'last_active': '2024-02-21T23:06:09.906800+0000', 'last_peered': '2024-02-21T23:06:09.906800+0000', 'last_clean': '2024-02-21T23:06:09.906800+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:06:09.937280+0000', 'last_undegraded': '2024-02-21T23:06:09.937280+0000', 'last_fullsized': '2024-02-21T23:06:09.937280+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_clean_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-02-21T23:06:16.694 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.a is not active+clean 2024-02-21T23:06:16.694 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:06:09.923016+0000', 'last_change': '2024-02-21T23:06:09.923016+0000', 'last_active': '2024-02-21T23:06:09.906800+0000', 'last_peered': '2024-02-21T23:06:09.906800+0000', 'last_clean': '2024-02-21T23:06:09.906800+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:06:09.923016+0000', 'last_undegraded': '2024-02-21T23:06:09.923016+0000', 'last_fullsized': '2024-02-21T23:06:09.923016+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_clean_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-02-21T23:06:16.694 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.b is not active+clean 2024-02-21T23:06:16.694 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:06:09.934671+0000', 'last_change': '2024-02-21T23:06:09.934671+0000', 'last_active': '2024-02-21T23:06:09.906800+0000', 'last_peered': '2024-02-21T23:06:09.906800+0000', 'last_clean': '2024-02-21T23:06:09.906800+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:06:09.934671+0000', 'last_undegraded': '2024-02-21T23:06:09.934671+0000', 'last_fullsized': '2024-02-21T23:06:09.934671+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_clean_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-02-21T23:06:16.694 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.8 is not active+clean 2024-02-21T23:06:16.694 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:06:09.929958+0000', 'last_change': '2024-02-21T23:06:09.929958+0000', 'last_active': '2024-02-21T23:06:09.906800+0000', 'last_peered': '2024-02-21T23:06:09.906800+0000', 'last_clean': '2024-02-21T23:06:09.906800+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:06:09.929958+0000', 'last_undegraded': '2024-02-21T23:06:09.929958+0000', 'last_fullsized': '2024-02-21T23:06:09.929958+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_clean_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-02-21T23:06:16.694 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.9 is not active+clean 2024-02-21T23:06:16.695 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:06:09.925807+0000', 'last_change': '2024-02-21T23:06:09.925807+0000', 'last_active': '2024-02-21T23:06:09.906800+0000', 'last_peered': '2024-02-21T23:06:09.906800+0000', 'last_clean': '2024-02-21T23:06:09.906800+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:06:09.925807+0000', 'last_undegraded': '2024-02-21T23:06:09.925807+0000', 'last_fullsized': '2024-02-21T23:06:09.925807+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_clean_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-02-21T23:06:16.695 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.e is not active+clean 2024-02-21T23:06:16.695 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:06:09.938677+0000', 'last_change': '2024-02-21T23:06:09.938677+0000', 'last_active': '2024-02-21T23:06:09.906800+0000', 'last_peered': '2024-02-21T23:06:09.906800+0000', 'last_clean': '2024-02-21T23:06:09.906800+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:06:09.938677+0000', 'last_undegraded': '2024-02-21T23:06:09.938677+0000', 'last_fullsized': '2024-02-21T23:06:09.938677+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_clean_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-02-21T23:06:16.695 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.c is not active+clean 2024-02-21T23:06:16.695 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:06:09.941480+0000', 'last_change': '2024-02-21T23:06:09.941480+0000', 'last_active': '2024-02-21T23:06:09.906800+0000', 'last_peered': '2024-02-21T23:06:09.906800+0000', 'last_clean': '2024-02-21T23:06:09.906800+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:06:09.941480+0000', 'last_undegraded': '2024-02-21T23:06:09.941480+0000', 'last_fullsized': '2024-02-21T23:06:09.941480+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_clean_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-02-21T23:06:16.695 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.d is not active+clean 2024-02-21T23:06:16.695 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:06:09.925072+0000', 'last_change': '2024-02-21T23:06:09.925072+0000', 'last_active': '2024-02-21T23:06:09.906800+0000', 'last_peered': '2024-02-21T23:06:09.906800+0000', 'last_clean': '2024-02-21T23:06:09.906800+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:06:09.925072+0000', 'last_undegraded': '2024-02-21T23:06:09.925072+0000', 'last_fullsized': '2024-02-21T23:06:09.925072+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_clean_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-02-21T23:06:16.695 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.2 is not active+clean 2024-02-21T23:06:16.695 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:06:09.923630+0000', 'last_change': '2024-02-21T23:06:09.923630+0000', 'last_active': '2024-02-21T23:06:09.906800+0000', 'last_peered': '2024-02-21T23:06:09.906800+0000', 'last_clean': '2024-02-21T23:06:09.906800+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:06:09.923630+0000', 'last_undegraded': '2024-02-21T23:06:09.923630+0000', 'last_fullsized': '2024-02-21T23:06:09.923630+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_clean_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-02-21T23:06:16.695 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.3 is not active+clean 2024-02-21T23:06:16.695 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:06:09.935470+0000', 'last_change': '2024-02-21T23:06:09.935470+0000', 'last_active': '2024-02-21T23:06:09.906800+0000', 'last_peered': '2024-02-21T23:06:09.906800+0000', 'last_clean': '2024-02-21T23:06:09.906800+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:06:09.935470+0000', 'last_undegraded': '2024-02-21T23:06:09.935470+0000', 'last_fullsized': '2024-02-21T23:06:09.935470+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_clean_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-02-21T23:06:16.696 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.0 is not active+clean 2024-02-21T23:06:16.696 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:06:09.930610+0000', 'last_change': '2024-02-21T23:06:09.930610+0000', 'last_active': '2024-02-21T23:06:09.906800+0000', 'last_peered': '2024-02-21T23:06:09.906800+0000', 'last_clean': '2024-02-21T23:06:09.906800+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:06:09.930610+0000', 'last_undegraded': '2024-02-21T23:06:09.930610+0000', 'last_fullsized': '2024-02-21T23:06:09.930610+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_clean_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-02-21T23:06:16.696 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.1 is not active+clean 2024-02-21T23:06:16.696 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:06:09.933769+0000', 'last_change': '2024-02-21T23:06:09.933769+0000', 'last_active': '2024-02-21T23:06:09.906800+0000', 'last_peered': '2024-02-21T23:06:09.906800+0000', 'last_clean': '2024-02-21T23:06:09.906800+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:06:09.933769+0000', 'last_undegraded': '2024-02-21T23:06:09.933769+0000', 'last_fullsized': '2024-02-21T23:06:09.933769+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_clean_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-02-21T23:06:16.696 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.6 is not active+clean 2024-02-21T23:06:16.696 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:06:09.937602+0000', 'last_change': '2024-02-21T23:06:09.937602+0000', 'last_active': '2024-02-21T23:06:09.906800+0000', 'last_peered': '2024-02-21T23:06:09.906800+0000', 'last_clean': '2024-02-21T23:06:09.906800+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:06:09.937602+0000', 'last_undegraded': '2024-02-21T23:06:09.937602+0000', 'last_fullsized': '2024-02-21T23:06:09.937602+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_clean_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-02-21T23:06:16.696 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.7 is not active+clean 2024-02-21T23:06:16.696 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:06:09.924758+0000', 'last_change': '2024-02-21T23:06:09.924758+0000', 'last_active': '2024-02-21T23:06:09.906800+0000', 'last_peered': '2024-02-21T23:06:09.906800+0000', 'last_clean': '2024-02-21T23:06:09.906800+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:06:09.924758+0000', 'last_undegraded': '2024-02-21T23:06:09.924758+0000', 'last_fullsized': '2024-02-21T23:06:09.924758+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_clean_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-02-21T23:06:16.696 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.4 is not active+clean 2024-02-21T23:06:16.696 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:06:09.923982+0000', 'last_change': '2024-02-21T23:06:09.923982+0000', 'last_active': '2024-02-21T23:06:09.906800+0000', 'last_peered': '2024-02-21T23:06:09.906800+0000', 'last_clean': '2024-02-21T23:06:09.906800+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:06:09.923982+0000', 'last_undegraded': '2024-02-21T23:06:09.923982+0000', 'last_fullsized': '2024-02-21T23:06:09.923982+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_clean_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-02-21T23:06:16.696 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.5 is not active+clean 2024-02-21T23:06:16.697 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:06:09.941480+0000', 'last_change': '2024-02-21T23:06:09.941480+0000', 'last_active': '2024-02-21T23:06:09.906800+0000', 'last_peered': '2024-02-21T23:06:09.906800+0000', 'last_clean': '2024-02-21T23:06:09.906800+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:06:09.941480+0000', 'last_undegraded': '2024-02-21T23:06:09.941480+0000', 'last_fullsized': '2024-02-21T23:06:09.941480+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_clean_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-02-21T23:06:16.697 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.f is not active+clean 2024-02-21T23:06:16.697 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:06:09.940614+0000', 'last_change': '2024-02-21T23:06:09.940614+0000', 'last_active': '2024-02-21T23:06:09.906800+0000', 'last_peered': '2024-02-21T23:06:09.906800+0000', 'last_clean': '2024-02-21T23:06:09.906800+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:06:09.940614+0000', 'last_undegraded': '2024-02-21T23:06:09.940614+0000', 'last_fullsized': '2024-02-21T23:06:09.940614+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'last_clean_scrub_stamp': '2024-02-21T23:06:09.906800+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-02-21T23:06:16.697 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-02-21T23:06:16.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.945+0000 7fdec2000640 1 Processor -- start 2024-02-21T23:06:16.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.945+0000 7fdec2000640 1 -- start start 2024-02-21T23:06:16.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.947+0000 7fdec2000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdebc14ce40 0x7fdebc15e570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:16.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.947+0000 7fdec2000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdebc06bb40 con 0x7fdebc06f550 2024-02-21T23:06:16.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.947+0000 7fdec2000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdebc153700 con 0x7fdebc14ca60 2024-02-21T23:06:16.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.947+0000 7fdec2000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdebc0c5520 con 0x7fdebc14ce40 2024-02-21T23:06:16.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.947+0000 7fdebbe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fdebc06f550 0x7fdebc06ae70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42528/0 (socket says 172.21.15.28:42528) 2024-02-21T23:06:16.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.947+0000 7fdebbe00640 1 -- 172.21.15.28:0/2988986390 learned_addr learned my addr 172.21.15.28:0/2988986390 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:06:16.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.948+0000 7fdec1600640 1 --2- 172.21.15.28:0/2988986390 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdebc14ce40 0x7fdebc15e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:16.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.948+0000 7fdebb400640 1 -- 172.21.15.28:0/2988986390 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3771580976 0 0) 0x7fdebc06bb40 con 0x7fdebc06f550 2024-02-21T23:06:16.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.948+0000 7fdebb400640 1 -- 172.21.15.28:0/2988986390 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdea4003520 con 0x7fdebc06f550 2024-02-21T23:06:16.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.948+0000 7fdebb400640 1 -- 172.21.15.28:0/2988986390 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 834522014 0 0) 0x7fdea4003520 con 0x7fdebc06f550 2024-02-21T23:06:16.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.948+0000 7fdebb400640 1 -- 172.21.15.28:0/2988986390 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdebc14ce40 msgr2=0x7fdebc15e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:16.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.948+0000 7fdebb400640 1 --2- 172.21.15.28:0/2988986390 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdebc14ce40 0x7fdebc15e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:16.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.948+0000 7fdebb400640 1 -- 172.21.15.28:0/2988986390 >> v1:172.21.15.81:6789/0 conn(0x7fdebc14ca60 legacy=0x7fdebc072120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:16.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.948+0000 7fdebb400640 1 -- 172.21.15.28:0/2988986390 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdebc1539e0 con 0x7fdebc06f550 2024-02-21T23:06:16.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.948+0000 7fdebb400640 1 -- 172.21.15.28:0/2988986390 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fdea8001670 con 0x7fdebc06f550 2024-02-21T23:06:16.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.948+0000 7fdebb400640 1 -- 172.21.15.28:0/2988986390 >> v1:172.21.15.28:6789/0 conn(0x7fdebc06f550 legacy=0x7fdebc06ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:16.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.948+0000 7fdebb400640 1 --2- 172.21.15.28:0/2988986390 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdebc14ce40 0x7fdea40039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:16.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.948+0000 7fdebb400640 1 --2- 172.21.15.28:0/2988986390 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdea4004060 0x7fdea4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:16.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.948+0000 7fdebb400640 1 --2- 172.21.15.28:0/2988986390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdea4004e60 0x7fdea400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:16.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.948+0000 7fdebb400640 1 -- 172.21.15.28:0/2988986390 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdebc0c5520 con 0x7fdea4004e60 2024-02-21T23:06:16.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.948+0000 7fdebb400640 1 -- 172.21.15.28:0/2988986390 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdea4005680 con 0x7fdea4004060 2024-02-21T23:06:16.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.948+0000 7fdebb400640 1 -- 172.21.15.28:0/2988986390 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdea4010310 con 0x7fdebc14ce40 2024-02-21T23:06:16.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.949+0000 7fdec1600640 1 --2- 172.21.15.28:0/2988986390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdea4004e60 0x7fdea400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:16.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.949+0000 7fdebbe00640 1 --2- 172.21.15.28:0/2988986390 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdea4004060 0x7fdea4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:16.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.949+0000 7fdec1600640 1 -- 172.21.15.28:0/2988986390 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdebc14ce40 msgr2=0x7fdea40039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:16.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.949+0000 7fdec1600640 1 --2- 172.21.15.28:0/2988986390 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdebc14ce40 0x7fdea40039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:16.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.949+0000 7fdec1600640 1 -- 172.21.15.28:0/2988986390 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdea4004060 msgr2=0x7fdea4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:16.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.949+0000 7fdec1600640 1 --2- 172.21.15.28:0/2988986390 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdea4004060 0x7fdea4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:16.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.949+0000 7fdec1600640 1 -- 172.21.15.28:0/2988986390 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdea4003520 con 0x7fdea4004e60 2024-02-21T23:06:16.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.949+0000 7fdebbe00640 1 --2- 172.21.15.28:0/2988986390 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdea4004060 0x7fdea4004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:06:16.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.950+0000 7fdec1600640 1 --2- 172.21.15.28:0/2988986390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdea4004e60 0x7fdea400b9c0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fdeb4007360 tx=0x7fdea4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90b822048ea482ca server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:16.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.950+0000 7fdebb400640 1 -- 172.21.15.28:0/2988986390 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeb4039070 con 0x7fdea4004e60 2024-02-21T23:06:16.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.950+0000 7fdebb400640 1 -- 172.21.15.28:0/2988986390 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdeb402ccb0 con 0x7fdea4004e60 2024-02-21T23:06:16.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.950+0000 7fdebb400640 1 -- 172.21.15.28:0/2988986390 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeb4009be0 con 0x7fdea4004e60 2024-02-21T23:06:16.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.951+0000 7fdec2000640 1 -- 172.21.15.28:0/2988986390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdea4004e60 msgr2=0x7fdea400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:16.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.951+0000 7fdec2000640 1 --2- 172.21.15.28:0/2988986390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdea4004e60 0x7fdea400b9c0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fdeb4007360 tx=0x7fdea4005830 comp rx=0 tx=0).stop 2024-02-21T23:06:16.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.951+0000 7fdec1600640 1 -- 172.21.15.28:0/2988986390 reap_dead start 2024-02-21T23:06:16.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.951+0000 7fdec2000640 1 -- 172.21.15.28:0/2988986390 shutdown_connections 2024-02-21T23:06:16.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.951+0000 7fdec2000640 1 -- 172.21.15.28:0/2988986390 >> 172.21.15.28:0/2988986390 conn(0x7fdebc067050 msgr2=0x7fdebc06bf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:16.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.951+0000 7fdec2000640 1 -- 172.21.15.28:0/2988986390 shutdown_connections 2024-02-21T23:06:16.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.951+0000 7fdec2000640 1 -- 172.21.15.28:0/2988986390 wait complete. 2024-02-21T23:06:16.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.953+0000 7fdec2000640 1 Processor -- start 2024-02-21T23:06:16.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.953+0000 7fdec2000640 1 -- start start 2024-02-21T23:06:16.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.954+0000 7fdec2000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdea4004e60 0x7fdebc0c2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:16.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.954+0000 7fdec2000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdebc156c20 0x7fdebc14ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:16.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.954+0000 7fdec2000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdebc1e33b0 0x7fdebc1e3830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:16.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.954+0000 7fdec2000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdebc070230 con 0x7fdebc156c20 2024-02-21T23:06:16.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.954+0000 7fdec2000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdebc0700b0 con 0x7fdebc1e33b0 2024-02-21T23:06:16.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.954+0000 7fdec2000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdebc06f030 con 0x7fdea4004e60 2024-02-21T23:06:16.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.954+0000 7fdebbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdebc156c20 0x7fdebc14ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:16.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.954+0000 7fdebbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdebc156c20 0x7fdebc14ac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54660/0 (socket says 172.21.15.28:54660) 2024-02-21T23:06:16.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.954+0000 7fdebbe00640 1 -- 172.21.15.28:0/3589227002 learned_addr learned my addr 172.21.15.28:0/3589227002 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:16.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.954+0000 7fdec1600640 1 --2- 172.21.15.28:0/3589227002 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdebc1e33b0 0x7fdebc1e3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:16.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.954+0000 7fdec0c00640 1 --2- 172.21.15.28:0/3589227002 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdea4004e60 0x7fdebc0c2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:16.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.955+0000 7fdebbe00640 1 -- 172.21.15.28:0/3589227002 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdea4004e60 msgr2=0x7fdebc0c2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:16.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.955+0000 7fdebbe00640 1 --2- 172.21.15.28:0/3589227002 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdea4004e60 0x7fdebc0c2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:16.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.955+0000 7fdebbe00640 1 -- 172.21.15.28:0/3589227002 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdebc1e33b0 msgr2=0x7fdebc1e3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:16.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.955+0000 7fdebbe00640 1 --2- 172.21.15.28:0/3589227002 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdebc1e33b0 0x7fdebc1e3830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:16.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.955+0000 7fdebbe00640 1 -- 172.21.15.28:0/3589227002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdebc0c40d0 con 0x7fdebc156c20 2024-02-21T23:06:16.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.955+0000 7fdebbe00640 1 --2- 172.21.15.28:0/3589227002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdebc156c20 0x7fdebc14ac60 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fdea800d7c0 tx=0x7fdea800dc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:16.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.955+0000 7fdeb8c00640 1 -- 172.21.15.28:0/3589227002 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea8019070 con 0x7fdebc156c20 2024-02-21T23:06:16.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.955+0000 7fdeb8c00640 1 -- 172.21.15.28:0/3589227002 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdea80074e0 con 0x7fdebc156c20 2024-02-21T23:06:16.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.955+0000 7fdeb8c00640 1 -- 172.21.15.28:0/3589227002 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea80049e0 con 0x7fdebc156c20 2024-02-21T23:06:16.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.955+0000 7fdec2000640 1 -- 172.21.15.28:0/3589227002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdebc0c3660 con 0x7fdebc156c20 2024-02-21T23:06:16.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.955+0000 7fdec2000640 1 -- 172.21.15.28:0/3589227002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdebc1e66f0 con 0x7fdebc156c20 2024-02-21T23:06:16.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.957+0000 7fdec2000640 1 -- 172.21.15.28:0/3589227002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdebc0c6b00 con 0x7fdebc156c20 2024-02-21T23:06:16.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.959+0000 7fdeb8c00640 1 -- 172.21.15.28:0/3589227002 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdea8004b80 con 0x7fdebc156c20 2024-02-21T23:06:16.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.960+0000 7fdeb8c00640 1 --2- 172.21.15.28:0/3589227002 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fde9c06b4d0 0x7fde9c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:16.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.960+0000 7fdeb8c00640 1 -- 172.21.15.28:0/3589227002 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6957+0+0 (secure 0 0 0) 0x7fdea808a0b0 con 0x7fdebc156c20 2024-02-21T23:06:16.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.960+0000 7fdec0c00640 1 --2- 172.21.15.28:0/3589227002 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fde9c06b4d0 0x7fde9c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:16.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.961+0000 7fdec0c00640 1 --2- 172.21.15.28:0/3589227002 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fde9c06b4d0 0x7fde9c06d990 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fdeb0001620 tx=0x7fdeb0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:16.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:16.964+0000 7fdeb8c00640 1 -- 172.21.15.28:0/3589227002 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fdebc0c6b00 con 0x7fdebc156c20 2024-02-21T23:06:17.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.294+0000 7fdec2000640 1 -- 172.21.15.28:0/3589227002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fdebc04f290 con 0x7fdebc156c20 2024-02-21T23:06:17.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.295+0000 7fdeb8c00640 1 -- 172.21.15.28:0/3589227002 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1535 (secure 0 0 0) 0x7fdea8058a40 con 0x7fdebc156c20 2024-02-21T23:06:17.285 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:06:17.285 INFO:teuthology.orchestra.run.smithi028.stdout:{"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":239,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":80,"num_osds":8,"num_up_osds":8,"osd_up_since":1708556550,"num_in_osds":8,"osd_in_since":1708556541,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":193}],"num_pgs":193,"num_pools":5,"num_objects":46,"data_bytes":595188,"bytes_used":259280896,"bytes_avail":772834832384,"bytes_total":773094113280,"write_bytes_sec":2100,"read_op_per_sec":0,"write_op_per_sec":6},"fsmap":{"epoch":67,"id":5,"up":1,"in":1,"max":1,"id":6,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":6,"rank":0,"name":"c","status":"up:active","gid":5539},{"filesystem_id":5,"rank":0,"name":"a","status":"up:active","gid":6366}],"up:standby":2},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-02-21T23:05:28.726167+0000","services":{"mds":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-02-21T23:06:17.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.298+0000 7fdec2000640 1 -- 172.21.15.28:0/3589227002 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fde9c06b4d0 msgr2=0x7fde9c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:17.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.298+0000 7fdec2000640 1 --2- 172.21.15.28:0/3589227002 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fde9c06b4d0 0x7fde9c06d990 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fdeb0001620 tx=0x7fdeb0009040 comp rx=0 tx=0).stop 2024-02-21T23:06:17.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.298+0000 7fdec2000640 1 -- 172.21.15.28:0/3589227002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdebc156c20 msgr2=0x7fdebc14ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:17.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.298+0000 7fdec2000640 1 --2- 172.21.15.28:0/3589227002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdebc156c20 0x7fdebc14ac60 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fdea800d7c0 tx=0x7fdea800dc80 comp rx=0 tx=0).stop 2024-02-21T23:06:17.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.298+0000 7fdec2000640 1 -- 172.21.15.28:0/3589227002 shutdown_connections 2024-02-21T23:06:17.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.298+0000 7fdec2000640 1 --2- 172.21.15.28:0/3589227002 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fde9c06b4d0 0x7fde9c06d990 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:17.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.298+0000 7fdec2000640 1 --2- 172.21.15.28:0/3589227002 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdebc1e33b0 0x7fdebc1e3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:17.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.298+0000 7fdec2000640 1 --2- 172.21.15.28:0/3589227002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdebc156c20 0x7fdebc14ac60 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:17.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.298+0000 7fdec2000640 1 --2- 172.21.15.28:0/3589227002 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdea4004e60 0x7fdebc0c2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:17.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.298+0000 7fdec2000640 1 -- 172.21.15.28:0/3589227002 >> 172.21.15.28:0/3589227002 conn(0x7fdebc067050 msgr2=0x7fdebc074360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:17.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.298+0000 7fdec2000640 1 -- 172.21.15.28:0/3589227002 shutdown_connections 2024-02-21T23:06:17.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.298+0000 7fdec2000640 1 -- 172.21.15.28:0/3589227002 wait complete. 2024-02-21T23:06:17.305 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-02-21T23:06:17.305 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-21T23:06:17.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.517+0000 7f0121200640 1 Processor -- start 2024-02-21T23:06:17.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.517+0000 7f0121200640 1 -- start start 2024-02-21T23:06:17.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.519+0000 7f0121200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f011c0c3740 0x7f011c0c6520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:17.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.519+0000 7f0121200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f011c159350 0x7f011c0c6a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:17.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.519+0000 7f0121200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f011c156cd0 con 0x7f011c1568f0 2024-02-21T23:06:17.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.519+0000 7f0121200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f011c0c6fa0 con 0x7f011c0c3740 2024-02-21T23:06:17.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.519+0000 7f0121200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f011c0c3cb0 con 0x7f011c159350 2024-02-21T23:06:17.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.519+0000 7f011aa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f011c0c3740 0x7f011c0c6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:17.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.519+0000 7f011b400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f011c1568f0 0x7f011c06bf80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42530/0 (socket says 172.21.15.28:42530) 2024-02-21T23:06:17.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.519+0000 7f011b400640 1 -- 172.21.15.28:0/3259610962 learned_addr learned my addr 172.21.15.28:0/3259610962 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:06:17.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.519+0000 7f011aa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f011c0c3740 0x7f011c0c6520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54668/0 (socket says 172.21.15.28:54668) 2024-02-21T23:06:17.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.519+0000 7f011be00640 1 --2- 172.21.15.28:0/3259610962 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f011c159350 0x7f011c0c6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:17.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.519+0000 7f011aa00640 1 -- 172.21.15.28:0/3259610962 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f011c159350 msgr2=0x7f011c0c6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:17.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.519+0000 7f011aa00640 1 --2- 172.21.15.28:0/3259610962 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f011c159350 0x7f011c0c6a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:17.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.519+0000 7f011aa00640 1 -- 172.21.15.28:0/3259610962 >> v1:172.21.15.28:6789/0 conn(0x7f011c1568f0 legacy=0x7f011c06bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:17.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.519+0000 7f011aa00640 1 -- 172.21.15.28:0/3259610962 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f011c0c3f90 con 0x7f011c0c3740 2024-02-21T23:06:17.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.521+0000 7f011aa00640 1 --2- 172.21.15.28:0/3259610962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f011c0c3740 0x7f011c0c6520 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f01080097a0 tx=0x7f010802b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99f60e896a92f9b7 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:17.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.521+0000 7f011a000640 1 -- 172.21.15.28:0/3259610962 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0108031780 con 0x7f011c0c3740 2024-02-21T23:06:17.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.521+0000 7f011a000640 1 -- 172.21.15.28:0/3259610962 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0108037070 con 0x7f011c0c3740 2024-02-21T23:06:17.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.521+0000 7f011a000640 1 -- 172.21.15.28:0/3259610962 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0108031b40 con 0x7f011c0c3740 2024-02-21T23:06:17.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.523+0000 7f0121200640 1 -- 172.21.15.28:0/3259610962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f011c0c3740 msgr2=0x7f011c0c6520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:17.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.523+0000 7f0121200640 1 --2- 172.21.15.28:0/3259610962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f011c0c3740 0x7f011c0c6520 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f01080097a0 tx=0x7f010802b820 comp rx=0 tx=0).stop 2024-02-21T23:06:17.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.523+0000 7f0121200640 1 -- 172.21.15.28:0/3259610962 shutdown_connections 2024-02-21T23:06:17.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.523+0000 7f0121200640 1 --2- 172.21.15.28:0/3259610962 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f011c159350 0x7f011c0c6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:17.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.523+0000 7f0121200640 1 --2- 172.21.15.28:0/3259610962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f011c0c3740 0x7f011c0c6520 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:17.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.523+0000 7f0121200640 1 -- 172.21.15.28:0/3259610962 >> 172.21.15.28:0/3259610962 conn(0x7f011c0aa3d0 msgr2=0x7f011c0aa7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:17.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.523+0000 7f0121200640 1 -- 172.21.15.28:0/3259610962 shutdown_connections 2024-02-21T23:06:17.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.523+0000 7f0121200640 1 -- 172.21.15.28:0/3259610962 wait complete. 2024-02-21T23:06:17.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.525+0000 7f0121200640 1 Processor -- start 2024-02-21T23:06:17.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.525+0000 7f0121200640 1 -- start start 2024-02-21T23:06:17.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.526+0000 7f0121200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f011c0c3740 0x7f011c1f2830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:17.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.526+0000 7f0121200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f011c1568f0 0x7f011c1f2d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:17.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.526+0000 7f0121200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f011c159350 0x7f011c15b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:17.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.526+0000 7f0121200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f011c0c4d50 con 0x7f011c1568f0 2024-02-21T23:06:17.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.526+0000 7f0121200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f011c0c4550 con 0x7f011c0c3740 2024-02-21T23:06:17.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.526+0000 7f0121200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f011c0c3240 con 0x7f011c159350 2024-02-21T23:06:17.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.527+0000 7f011aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f011c1568f0 0x7f011c1f2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:17.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.527+0000 7f011aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f011c1568f0 0x7f011c1f2d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54670/0 (socket says 172.21.15.28:54670) 2024-02-21T23:06:17.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.527+0000 7f011aa00640 1 -- 172.21.15.28:0/177783851 learned_addr learned my addr 172.21.15.28:0/177783851 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:17.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.527+0000 7f011b400640 1 --2- 172.21.15.28:0/177783851 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f011c0c3740 0x7f011c1f2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:17.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.527+0000 7f011be00640 1 --2- 172.21.15.28:0/177783851 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f011c159350 0x7f011c15b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:17.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.527+0000 7f011aa00640 1 -- 172.21.15.28:0/177783851 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f011c159350 msgr2=0x7f011c15b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:17.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.527+0000 7f011aa00640 1 --2- 172.21.15.28:0/177783851 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f011c159350 0x7f011c15b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:17.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.527+0000 7f011aa00640 1 -- 172.21.15.28:0/177783851 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f011c0c3740 msgr2=0x7f011c1f2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:17.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.527+0000 7f011aa00640 1 --2- 172.21.15.28:0/177783851 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f011c0c3740 0x7f011c1f2830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:17.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.527+0000 7f011aa00640 1 -- 172.21.15.28:0/177783851 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f011c153720 con 0x7f011c1568f0 2024-02-21T23:06:17.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.527+0000 7f011b400640 1 --2- 172.21.15.28:0/177783851 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f011c0c3740 0x7f011c1f2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:06:17.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.527+0000 7f011be00640 1 --2- 172.21.15.28:0/177783851 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f011c159350 0x7f011c15b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:06:17.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.528+0000 7f011aa00640 1 --2- 172.21.15.28:0/177783851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f011c1568f0 0x7f011c1f2d70 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f01080098c0 tx=0x7f01080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:17.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.528+0000 7f0113400640 1 -- 172.21.15.28:0/177783851 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0108032610 con 0x7f011c1568f0 2024-02-21T23:06:17.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.528+0000 7f0113400640 1 -- 172.21.15.28:0/177783851 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01080327b0 con 0x7f011c1568f0 2024-02-21T23:06:17.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.528+0000 7f0121200640 1 -- 172.21.15.28:0/177783851 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f011c153a00 con 0x7f011c1568f0 2024-02-21T23:06:17.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.528+0000 7f0113400640 1 -- 172.21.15.28:0/177783851 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0108032ba0 con 0x7f011c1568f0 2024-02-21T23:06:17.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.528+0000 7f0121200640 1 -- 172.21.15.28:0/177783851 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f011c14a290 con 0x7f011c1568f0 2024-02-21T23:06:17.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.529+0000 7f0121200640 1 -- 172.21.15.28:0/177783851 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f011c06e110 con 0x7f011c1568f0 2024-02-21T23:06:17.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.531+0000 7f0113400640 1 -- 172.21.15.28:0/177783851 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f010804e440 con 0x7f011c1568f0 2024-02-21T23:06:17.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.531+0000 7f0113400640 1 --2- 172.21.15.28:0/177783851 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f00fc06b4d0 0x7f00fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:17.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.532+0000 7f0113400640 1 -- 172.21.15.28:0/177783851 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f01080ac450 con 0x7f011c1568f0 2024-02-21T23:06:17.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.532+0000 7f011b400640 1 --2- 172.21.15.28:0/177783851 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f00fc06b4d0 0x7f00fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:17.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.533+0000 7f011b400640 1 --2- 172.21.15.28:0/177783851 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f00fc06b4d0 0x7f00fc06d990 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f010c000f60 tx=0x7f010c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:17.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.541+0000 7f0113400640 1 -- 172.21.15.28:0/177783851 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f0108040070 con 0x7f011c1568f0 2024-02-21T23:06:17.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.780+0000 7f0121200640 1 -- 172.21.15.28:0/177783851 --> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f011c1f35e0 con 0x7f00fc06b4d0 2024-02-21T23:06:17.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.814+0000 7f0113400640 1 -- 172.21.15.28:0/177783851 <== mgr.4101 v2:172.21.15.81:6832/1848538577 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+490725 (secure 0 0 0) 0x7f010c0070d0 con 0x7f00fc06b4d0 2024-02-21T23:06:17.810 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:06:17.815 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-21T23:06:17.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.823+0000 7f0121200640 1 -- 172.21.15.28:0/177783851 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f00fc06b4d0 msgr2=0x7f00fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:17.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.824+0000 7f0121200640 1 --2- 172.21.15.28:0/177783851 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f00fc06b4d0 0x7f00fc06d990 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f010c000f60 tx=0x7f010c009040 comp rx=0 tx=0).stop 2024-02-21T23:06:17.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.824+0000 7f0121200640 1 -- 172.21.15.28:0/177783851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f011c1568f0 msgr2=0x7f011c1f2d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:17.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.824+0000 7f0121200640 1 --2- 172.21.15.28:0/177783851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f011c1568f0 0x7f011c1f2d70 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f01080098c0 tx=0x7f01080046c0 comp rx=0 tx=0).stop 2024-02-21T23:06:17.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.824+0000 7f0121200640 1 -- 172.21.15.28:0/177783851 shutdown_connections 2024-02-21T23:06:17.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.824+0000 7f0121200640 1 --2- 172.21.15.28:0/177783851 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f00fc06b4d0 0x7f00fc06d990 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:17.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.824+0000 7f0121200640 1 --2- 172.21.15.28:0/177783851 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f011c159350 0x7f011c15b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:17.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.824+0000 7f0121200640 1 --2- 172.21.15.28:0/177783851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f011c1568f0 0x7f011c1f2d70 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:17.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.824+0000 7f0121200640 1 --2- 172.21.15.28:0/177783851 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f011c0c3740 0x7f011c1f2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:17.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.824+0000 7f0121200640 1 -- 172.21.15.28:0/177783851 >> 172.21.15.28:0/177783851 conn(0x7f011c0aa3d0 msgr2=0x7f011c0c6210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:17.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.824+0000 7f0121200640 1 -- 172.21.15.28:0/177783851 shutdown_connections 2024-02-21T23:06:17.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:17.824+0000 7f0121200640 1 -- 172.21.15.28:0/177783851 wait complete. 2024-02-21T23:06:17.833 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":186,"stamp":"2024-02-21T23:06:16.742544+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595188,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":249,"num_write_kb":2444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":195,"ondisk_log_size":195,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":386,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":253204,"kb_used_data":7904,"kb_used_omap":12,"kb_used_meta":245171,"kb_avail":754721516,"statfs":{"total":773094113280,"available":772834832384,"internally_reserved":0,"allocated":8093696,"data_stored":4979792,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":251055665},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":47,"apply_latency_ms":47,"commit_latency_ns":47000000,"apply_latency_ns":47000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.826647"},"pg_stats":[{"pgid":"14.3d","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.978205+0000","last_change":"2024-02-21T23:06:10.982364+0000","last_active":"2024-02-21T23:06:11.978205+0000","last_peered":"2024-02-21T23:06:11.978205+0000","last_clean":"2024-02-21T23:06:11.978205+0000","last_became_active":"2024-02-21T23:06:10.982112+0000","last_became_peered":"2024-02-21T23:06:10.982112+0000","last_unstale":"2024-02-21T23:06:11.978205+0000","last_undegraded":"2024-02-21T23:06:11.978205+0000","last_fullsized":"2024-02-21T23:06:11.978205+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:29:34.759653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.3f","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.035955+0000","last_change":"2024-02-21T23:05:51.408831+0000","last_active":"2024-02-21T23:06:13.035955+0000","last_peered":"2024-02-21T23:06:13.035955+0000","last_clean":"2024-02-21T23:06:13.035955+0000","last_became_active":"2024-02-21T23:05:51.408353+0000","last_became_peered":"2024-02-21T23:05:51.408353+0000","last_unstale":"2024-02-21T23:06:13.035955+0000","last_undegraded":"2024-02-21T23:06:13.035955+0000","last_fullsized":"2024-02-21T23:06:13.035955+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:36:47.962935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.3c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.971866+0000","last_change":"2024-02-21T23:06:10.987327+0000","last_active":"2024-02-21T23:06:11.971866+0000","last_peered":"2024-02-21T23:06:11.971866+0000","last_clean":"2024-02-21T23:06:11.971866+0000","last_became_active":"2024-02-21T23:06:10.981952+0000","last_became_peered":"2024-02-21T23:06:10.981952+0000","last_unstale":"2024-02-21T23:06:11.971866+0000","last_undegraded":"2024-02-21T23:06:11.971866+0000","last_fullsized":"2024-02-21T23:06:11.971866+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:36:43.365830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.030722+0000","last_change":"2024-02-21T23:05:51.420557+0000","last_active":"2024-02-21T23:06:13.030722+0000","last_peered":"2024-02-21T23:06:13.030722+0000","last_clean":"2024-02-21T23:06:13.030722+0000","last_became_active":"2024-02-21T23:05:51.420078+0000","last_became_peered":"2024-02-21T23:05:51.420078+0000","last_unstale":"2024-02-21T23:06:13.030722+0000","last_undegraded":"2024-02-21T23:06:13.030722+0000","last_fullsized":"2024-02-21T23:06:13.030722+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:36:23.792395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.3f","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.978477+0000","last_change":"2024-02-21T23:06:10.959053+0000","last_active":"2024-02-21T23:06:11.978477+0000","last_peered":"2024-02-21T23:06:11.978477+0000","last_clean":"2024-02-21T23:06:11.978477+0000","last_became_active":"2024-02-21T23:06:10.958767+0000","last_became_peered":"2024-02-21T23:06:10.958767+0000","last_unstale":"2024-02-21T23:06:11.978477+0000","last_undegraded":"2024-02-21T23:06:11.978477+0000","last_fullsized":"2024-02-21T23:06:11.978477+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:41:50.244656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.971305+0000","last_change":"2024-02-21T23:05:51.408934+0000","last_active":"2024-02-21T23:06:11.971305+0000","last_peered":"2024-02-21T23:06:11.971305+0000","last_clean":"2024-02-21T23:06:11.971305+0000","last_became_active":"2024-02-21T23:05:51.408597+0000","last_became_peered":"2024-02-21T23:05:51.408597+0000","last_unstale":"2024-02-21T23:06:11.971305+0000","last_undegraded":"2024-02-21T23:06:11.971305+0000","last_fullsized":"2024-02-21T23:06:11.971305+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:58:55.829941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.3e","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.959663+0000","last_change":"2024-02-21T23:06:10.975663+0000","last_active":"2024-02-21T23:06:12.959663+0000","last_peered":"2024-02-21T23:06:12.959663+0000","last_clean":"2024-02-21T23:06:12.959663+0000","last_became_active":"2024-02-21T23:06:10.974064+0000","last_became_peered":"2024-02-21T23:06:10.974064+0000","last_unstale":"2024-02-21T23:06:12.959663+0000","last_undegraded":"2024-02-21T23:06:12.959663+0000","last_fullsized":"2024-02-21T23:06:12.959663+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:19:08.047583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.959623+0000","last_change":"2024-02-21T23:05:51.423479+0000","last_active":"2024-02-21T23:06:12.959623+0000","last_peered":"2024-02-21T23:06:12.959623+0000","last_clean":"2024-02-21T23:06:12.959623+0000","last_became_active":"2024-02-21T23:05:51.423108+0000","last_became_peered":"2024-02-21T23:05:51.423108+0000","last_unstale":"2024-02-21T23:06:12.959623+0000","last_undegraded":"2024-02-21T23:06:12.959623+0000","last_fullsized":"2024-02-21T23:06:12.959623+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:36:47.962935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.39","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.033222+0000","last_change":"2024-02-21T23:06:10.975296+0000","last_active":"2024-02-21T23:06:13.033222+0000","last_peered":"2024-02-21T23:06:13.033222+0000","last_clean":"2024-02-21T23:06:13.033222+0000","last_became_active":"2024-02-21T23:06:10.974998+0000","last_became_peered":"2024-02-21T23:06:10.974998+0000","last_unstale":"2024-02-21T23:06:13.033222+0000","last_undegraded":"2024-02-21T23:06:13.033222+0000","last_fullsized":"2024-02-21T23:06:13.033222+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:17:15.132571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.536186+0000","last_change":"2024-02-21T23:05:51.419436+0000","last_active":"2024-02-21T23:06:12.536186+0000","last_peered":"2024-02-21T23:06:12.536186+0000","last_clean":"2024-02-21T23:06:12.536186+0000","last_became_active":"2024-02-21T23:05:51.418562+0000","last_became_peered":"2024-02-21T23:05:51.418562+0000","last_unstale":"2024-02-21T23:06:12.536186+0000","last_undegraded":"2024-02-21T23:06:12.536186+0000","last_fullsized":"2024-02-21T23:06:12.536186+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:19:45.189126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.38","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.972213+0000","last_change":"2024-02-21T23:06:10.991824+0000","last_active":"2024-02-21T23:06:11.972213+0000","last_peered":"2024-02-21T23:06:11.972213+0000","last_clean":"2024-02-21T23:06:11.972213+0000","last_became_active":"2024-02-21T23:06:10.990580+0000","last_became_peered":"2024-02-21T23:06:10.990580+0000","last_unstale":"2024-02-21T23:06:11.972213+0000","last_undegraded":"2024-02-21T23:06:11.972213+0000","last_fullsized":"2024-02-21T23:06:11.972213+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:01:36.530063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.976468+0000","last_change":"2024-02-21T23:05:51.410014+0000","last_active":"2024-02-21T23:06:11.976468+0000","last_peered":"2024-02-21T23:06:11.976468+0000","last_clean":"2024-02-21T23:06:11.976468+0000","last_became_active":"2024-02-21T23:05:51.409637+0000","last_became_peered":"2024-02-21T23:05:51.409637+0000","last_unstale":"2024-02-21T23:06:11.976468+0000","last_undegraded":"2024-02-21T23:06:11.976468+0000","last_fullsized":"2024-02-21T23:06:11.976468+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:03:35.243949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3b","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.976370+0000","last_change":"2024-02-21T23:06:10.980590+0000","last_active":"2024-02-21T23:06:11.976370+0000","last_peered":"2024-02-21T23:06:11.976370+0000","last_clean":"2024-02-21T23:06:11.976370+0000","last_became_active":"2024-02-21T23:06:10.978843+0000","last_became_peered":"2024-02-21T23:06:10.978843+0000","last_unstale":"2024-02-21T23:06:11.976370+0000","last_undegraded":"2024-02-21T23:06:11.976370+0000","last_fullsized":"2024-02-21T23:06:11.976370+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:11:16.456657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.032749+0000","last_change":"2024-02-21T23:05:51.419505+0000","last_active":"2024-02-21T23:06:13.032749+0000","last_peered":"2024-02-21T23:06:13.032749+0000","last_clean":"2024-02-21T23:06:13.032749+0000","last_became_active":"2024-02-21T23:05:51.418040+0000","last_became_peered":"2024-02-21T23:05:51.418040+0000","last_unstale":"2024-02-21T23:06:13.032749+0000","last_undegraded":"2024-02-21T23:06:13.032749+0000","last_fullsized":"2024-02-21T23:06:13.032749+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:57:06.211024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.3a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.029781+0000","last_change":"2024-02-21T23:06:10.975464+0000","last_active":"2024-02-21T23:06:13.029781+0000","last_peered":"2024-02-21T23:06:13.029781+0000","last_clean":"2024-02-21T23:06:13.029781+0000","last_became_active":"2024-02-21T23:06:10.975183+0000","last_became_peered":"2024-02-21T23:06:10.975183+0000","last_unstale":"2024-02-21T23:06:13.029781+0000","last_undegraded":"2024-02-21T23:06:13.029781+0000","last_fullsized":"2024-02-21T23:06:13.029781+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:18:26.399820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.535896+0000","last_change":"2024-02-21T23:05:51.416524+0000","last_active":"2024-02-21T23:06:12.535896+0000","last_peered":"2024-02-21T23:06:12.535896+0000","last_clean":"2024-02-21T23:06:12.535896+0000","last_became_active":"2024-02-21T23:05:51.415391+0000","last_became_peered":"2024-02-21T23:05:51.415391+0000","last_unstale":"2024-02-21T23:06:12.535896+0000","last_undegraded":"2024-02-21T23:06:12.535896+0000","last_fullsized":"2024-02-21T23:06:12.535896+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:48:32.958103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.35","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.977890+0000","last_change":"2024-02-21T23:06:10.980906+0000","last_active":"2024-02-21T23:06:11.977890+0000","last_peered":"2024-02-21T23:06:11.977890+0000","last_clean":"2024-02-21T23:06:11.977890+0000","last_became_active":"2024-02-21T23:06:10.980361+0000","last_became_peered":"2024-02-21T23:06:10.980361+0000","last_unstale":"2024-02-21T23:06:11.977890+0000","last_undegraded":"2024-02-21T23:06:11.977890+0000","last_fullsized":"2024-02-21T23:06:11.977890+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:04:22.657947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.978465+0000","last_change":"2024-02-21T23:05:51.421068+0000","last_active":"2024-02-21T23:06:11.978465+0000","last_peered":"2024-02-21T23:06:11.978465+0000","last_clean":"2024-02-21T23:06:11.978465+0000","last_became_active":"2024-02-21T23:05:51.419746+0000","last_became_peered":"2024-02-21T23:05:51.419746+0000","last_unstale":"2024-02-21T23:06:11.978465+0000","last_undegraded":"2024-02-21T23:06:11.978465+0000","last_fullsized":"2024-02-21T23:06:11.978465+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:09:42.444733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.34","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.536406+0000","last_change":"2024-02-21T23:06:10.966147+0000","last_active":"2024-02-21T23:06:12.536406+0000","last_peered":"2024-02-21T23:06:12.536406+0000","last_clean":"2024-02-21T23:06:12.536406+0000","last_became_active":"2024-02-21T23:06:10.965783+0000","last_became_peered":"2024-02-21T23:06:10.965783+0000","last_unstale":"2024-02-21T23:06:12.536406+0000","last_undegraded":"2024-02-21T23:06:12.536406+0000","last_fullsized":"2024-02-21T23:06:12.536406+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:10:02.018168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.959863+0000","last_change":"2024-02-21T23:05:51.418295+0000","last_active":"2024-02-21T23:06:12.959863+0000","last_peered":"2024-02-21T23:06:12.959863+0000","last_clean":"2024-02-21T23:06:12.959863+0000","last_became_active":"2024-02-21T23:05:51.417780+0000","last_became_peered":"2024-02-21T23:05:51.417780+0000","last_unstale":"2024-02-21T23:06:12.959863+0000","last_undegraded":"2024-02-21T23:06:12.959863+0000","last_fullsized":"2024-02-21T23:06:12.959863+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:19:57.210565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.37","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.978058+0000","last_change":"2024-02-21T23:06:10.983699+0000","last_active":"2024-02-21T23:06:11.978058+0000","last_peered":"2024-02-21T23:06:11.978058+0000","last_clean":"2024-02-21T23:06:11.978058+0000","last_became_active":"2024-02-21T23:06:10.982870+0000","last_became_peered":"2024-02-21T23:06:10.982870+0000","last_unstale":"2024-02-21T23:06:11.978058+0000","last_undegraded":"2024-02-21T23:06:11.978058+0000","last_fullsized":"2024-02-21T23:06:11.978058+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:24:52.436507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.030605+0000","last_change":"2024-02-21T23:05:51.420518+0000","last_active":"2024-02-21T23:06:13.030605+0000","last_peered":"2024-02-21T23:06:13.030605+0000","last_clean":"2024-02-21T23:06:13.030605+0000","last_became_active":"2024-02-21T23:05:51.419564+0000","last_became_peered":"2024-02-21T23:05:51.419564+0000","last_unstale":"2024-02-21T23:06:13.030605+0000","last_undegraded":"2024-02-21T23:06:13.030605+0000","last_fullsized":"2024-02-21T23:06:13.030605+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:19:41.280587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.36","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.536772+0000","last_change":"2024-02-21T23:06:10.993431+0000","last_active":"2024-02-21T23:06:12.536772+0000","last_peered":"2024-02-21T23:06:12.536772+0000","last_clean":"2024-02-21T23:06:12.536772+0000","last_became_active":"2024-02-21T23:06:10.993067+0000","last_became_peered":"2024-02-21T23:06:10.993067+0000","last_unstale":"2024-02-21T23:06:12.536772+0000","last_undegraded":"2024-02-21T23:06:12.536772+0000","last_fullsized":"2024-02-21T23:06:12.536772+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:20:39.137766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.959701+0000","last_change":"2024-02-21T23:05:51.421691+0000","last_active":"2024-02-21T23:06:12.959701+0000","last_peered":"2024-02-21T23:06:12.959701+0000","last_clean":"2024-02-21T23:06:12.959701+0000","last_became_active":"2024-02-21T23:05:51.417621+0000","last_became_peered":"2024-02-21T23:05:51.417621+0000","last_unstale":"2024-02-21T23:06:12.959701+0000","last_undegraded":"2024-02-21T23:06:12.959701+0000","last_fullsized":"2024-02-21T23:06:12.959701+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:46:01.608149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.31","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.970970+0000","last_change":"2024-02-21T23:06:10.962402+0000","last_active":"2024-02-21T23:06:11.970970+0000","last_peered":"2024-02-21T23:06:11.970970+0000","last_clean":"2024-02-21T23:06:11.970970+0000","last_became_active":"2024-02-21T23:06:10.962127+0000","last_became_peered":"2024-02-21T23:06:10.962127+0000","last_unstale":"2024-02-21T23:06:11.970970+0000","last_undegraded":"2024-02-21T23:06:11.970970+0000","last_fullsized":"2024-02-21T23:06:11.970970+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:50:05.598685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.536283+0000","last_change":"2024-02-21T23:05:51.426871+0000","last_active":"2024-02-21T23:06:12.536283+0000","last_peered":"2024-02-21T23:06:12.536283+0000","last_clean":"2024-02-21T23:06:12.536283+0000","last_became_active":"2024-02-21T23:05:51.419412+0000","last_became_peered":"2024-02-21T23:05:51.419412+0000","last_unstale":"2024-02-21T23:06:12.536283+0000","last_undegraded":"2024-02-21T23:06:12.536283+0000","last_fullsized":"2024-02-21T23:06:12.536283+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:22:09.156621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.30","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.029598+0000","last_change":"2024-02-21T23:06:10.967734+0000","last_active":"2024-02-21T23:06:13.029598+0000","last_peered":"2024-02-21T23:06:13.029598+0000","last_clean":"2024-02-21T23:06:13.029598+0000","last_became_active":"2024-02-21T23:06:10.967386+0000","last_became_peered":"2024-02-21T23:06:10.967386+0000","last_unstale":"2024-02-21T23:06:13.029598+0000","last_undegraded":"2024-02-21T23:06:13.029598+0000","last_fullsized":"2024-02-21T23:06:13.029598+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:00:08.218703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.976550+0000","last_change":"2024-02-21T23:05:51.417593+0000","last_active":"2024-02-21T23:06:11.976550+0000","last_peered":"2024-02-21T23:06:11.976550+0000","last_clean":"2024-02-21T23:06:11.976550+0000","last_became_active":"2024-02-21T23:05:51.416945+0000","last_became_peered":"2024-02-21T23:05:51.416945+0000","last_unstale":"2024-02-21T23:06:11.976550+0000","last_undegraded":"2024-02-21T23:06:11.976550+0000","last_fullsized":"2024-02-21T23:06:11.976550+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:24:32.863072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.33","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.972115+0000","last_change":"2024-02-21T23:06:10.991670+0000","last_active":"2024-02-21T23:06:11.972115+0000","last_peered":"2024-02-21T23:06:11.972115+0000","last_clean":"2024-02-21T23:06:11.972115+0000","last_became_active":"2024-02-21T23:06:10.990780+0000","last_became_peered":"2024-02-21T23:06:10.990780+0000","last_unstale":"2024-02-21T23:06:11.972115+0000","last_undegraded":"2024-02-21T23:06:11.972115+0000","last_fullsized":"2024-02-21T23:06:11.972115+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:42:46.061945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.029483+0000","last_change":"2024-02-21T23:05:51.409645+0000","last_active":"2024-02-21T23:06:13.029483+0000","last_peered":"2024-02-21T23:06:13.029483+0000","last_clean":"2024-02-21T23:06:13.029483+0000","last_became_active":"2024-02-21T23:05:51.409356+0000","last_became_peered":"2024-02-21T23:05:51.409356+0000","last_unstale":"2024-02-21T23:06:13.029483+0000","last_undegraded":"2024-02-21T23:06:13.029483+0000","last_fullsized":"2024-02-21T23:06:13.029483+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:17:26.329609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.32","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.976374+0000","last_change":"2024-02-21T23:06:10.991249+0000","last_active":"2024-02-21T23:06:11.976374+0000","last_peered":"2024-02-21T23:06:11.976374+0000","last_clean":"2024-02-21T23:06:11.976374+0000","last_became_active":"2024-02-21T23:06:10.990938+0000","last_became_peered":"2024-02-21T23:06:10.990938+0000","last_unstale":"2024-02-21T23:06:11.976374+0000","last_undegraded":"2024-02-21T23:06:11.976374+0000","last_fullsized":"2024-02-21T23:06:11.976374+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:36:43.365830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.034178+0000","last_change":"2024-02-21T23:05:51.413656+0000","last_active":"2024-02-21T23:06:13.034178+0000","last_peered":"2024-02-21T23:06:13.034178+0000","last_clean":"2024-02-21T23:06:13.034178+0000","last_became_active":"2024-02-21T23:05:51.413350+0000","last_became_peered":"2024-02-21T23:05:51.413350+0000","last_unstale":"2024-02-21T23:06:13.034178+0000","last_undegraded":"2024-02-21T23:06:13.034178+0000","last_fullsized":"2024-02-21T23:06:13.034178+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:19:57.210565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.2d","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.978358+0000","last_change":"2024-02-21T23:06:10.968835+0000","last_active":"2024-02-21T23:06:11.978358+0000","last_peered":"2024-02-21T23:06:11.978358+0000","last_clean":"2024-02-21T23:06:11.978358+0000","last_became_active":"2024-02-21T23:06:10.968330+0000","last_became_peered":"2024-02-21T23:06:10.968330+0000","last_unstale":"2024-02-21T23:06:11.978358+0000","last_undegraded":"2024-02-21T23:06:11.978358+0000","last_fullsized":"2024-02-21T23:06:11.978358+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:41:50.834277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.536898+0000","last_change":"2024-02-21T23:05:51.427044+0000","last_active":"2024-02-21T23:06:12.536898+0000","last_peered":"2024-02-21T23:06:12.536898+0000","last_clean":"2024-02-21T23:06:12.536898+0000","last_became_active":"2024-02-21T23:05:51.419477+0000","last_became_peered":"2024-02-21T23:05:51.419477+0000","last_unstale":"2024-02-21T23:06:12.536898+0000","last_undegraded":"2024-02-21T23:06:12.536898+0000","last_fullsized":"2024-02-21T23:06:12.536898+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:04:00.714448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.2c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.970780+0000","last_change":"2024-02-21T23:06:10.975705+0000","last_active":"2024-02-21T23:06:11.970780+0000","last_peered":"2024-02-21T23:06:11.970780+0000","last_clean":"2024-02-21T23:06:11.970780+0000","last_became_active":"2024-02-21T23:06:10.970250+0000","last_became_peered":"2024-02-21T23:06:10.970250+0000","last_unstale":"2024-02-21T23:06:11.970780+0000","last_undegraded":"2024-02-21T23:06:11.970780+0000","last_fullsized":"2024-02-21T23:06:11.970780+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:58:24.687732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.971789+0000","last_change":"2024-02-21T23:05:51.425126+0000","last_active":"2024-02-21T23:06:11.971789+0000","last_peered":"2024-02-21T23:06:11.971789+0000","last_clean":"2024-02-21T23:06:11.971789+0000","last_became_active":"2024-02-21T23:05:51.422511+0000","last_became_peered":"2024-02-21T23:05:51.422511+0000","last_unstale":"2024-02-21T23:06:11.971789+0000","last_undegraded":"2024-02-21T23:06:11.971789+0000","last_fullsized":"2024-02-21T23:06:11.971789+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:27:01.191337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.2f","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.030469+0000","last_change":"2024-02-21T23:06:10.977919+0000","last_active":"2024-02-21T23:06:13.030469+0000","last_peered":"2024-02-21T23:06:13.030469+0000","last_clean":"2024-02-21T23:06:13.030469+0000","last_became_active":"2024-02-21T23:06:10.968729+0000","last_became_peered":"2024-02-21T23:06:10.968729+0000","last_unstale":"2024-02-21T23:06:13.030469+0000","last_undegraded":"2024-02-21T23:06:13.030469+0000","last_fullsized":"2024-02-21T23:06:13.030469+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:10:15.667957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.978241+0000","last_change":"2024-02-21T23:05:51.423258+0000","last_active":"2024-02-21T23:06:11.978241+0000","last_peered":"2024-02-21T23:06:11.978241+0000","last_clean":"2024-02-21T23:06:11.978241+0000","last_became_active":"2024-02-21T23:05:51.422987+0000","last_became_peered":"2024-02-21T23:05:51.422987+0000","last_unstale":"2024-02-21T23:06:11.978241+0000","last_undegraded":"2024-02-21T23:06:11.978241+0000","last_fullsized":"2024-02-21T23:06:11.978241+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:41:31.260842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2e","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.978880+0000","last_change":"2024-02-21T23:06:10.973113+0000","last_active":"2024-02-21T23:06:11.978880+0000","last_peered":"2024-02-21T23:06:11.978880+0000","last_clean":"2024-02-21T23:06:11.978880+0000","last_became_active":"2024-02-21T23:06:10.972815+0000","last_became_peered":"2024-02-21T23:06:10.972815+0000","last_unstale":"2024-02-21T23:06:11.978880+0000","last_undegraded":"2024-02-21T23:06:11.978880+0000","last_fullsized":"2024-02-21T23:06:11.978880+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:00:08.218703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.536378+0000","last_change":"2024-02-21T23:05:51.427468+0000","last_active":"2024-02-21T23:06:12.536378+0000","last_peered":"2024-02-21T23:06:12.536378+0000","last_clean":"2024-02-21T23:06:12.536378+0000","last_became_active":"2024-02-21T23:05:51.425476+0000","last_became_peered":"2024-02-21T23:05:51.425476+0000","last_unstale":"2024-02-21T23:06:12.536378+0000","last_undegraded":"2024-02-21T23:06:12.536378+0000","last_fullsized":"2024-02-21T23:06:12.536378+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:06:18.158392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.29","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.033158+0000","last_change":"2024-02-21T23:06:10.993284+0000","last_active":"2024-02-21T23:06:13.033158+0000","last_peered":"2024-02-21T23:06:13.033158+0000","last_clean":"2024-02-21T23:06:13.033158+0000","last_became_active":"2024-02-21T23:06:10.992937+0000","last_became_peered":"2024-02-21T23:06:10.992937+0000","last_unstale":"2024-02-21T23:06:13.033158+0000","last_undegraded":"2024-02-21T23:06:13.033158+0000","last_fullsized":"2024-02-21T23:06:13.033158+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:50:05.598685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.978079+0000","last_change":"2024-02-21T23:05:51.415772+0000","last_active":"2024-02-21T23:06:11.978079+0000","last_peered":"2024-02-21T23:06:11.978079+0000","last_clean":"2024-02-21T23:06:11.978079+0000","last_became_active":"2024-02-21T23:05:51.415494+0000","last_became_peered":"2024-02-21T23:05:51.415494+0000","last_unstale":"2024-02-21T23:06:11.978079+0000","last_undegraded":"2024-02-21T23:06:11.978079+0000","last_fullsized":"2024-02-21T23:06:11.978079+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:42:37.006685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.28","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.976025+0000","last_change":"2024-02-21T23:06:10.981808+0000","last_active":"2024-02-21T23:06:11.976025+0000","last_peered":"2024-02-21T23:06:11.976025+0000","last_clean":"2024-02-21T23:06:11.976025+0000","last_became_active":"2024-02-21T23:06:10.979421+0000","last_became_peered":"2024-02-21T23:06:10.979421+0000","last_unstale":"2024-02-21T23:06:11.976025+0000","last_undegraded":"2024-02-21T23:06:11.976025+0000","last_fullsized":"2024-02-21T23:06:11.976025+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:56:29.967189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.535991+0000","last_change":"2024-02-21T23:05:51.428557+0000","last_active":"2024-02-21T23:06:12.535991+0000","last_peered":"2024-02-21T23:06:12.535991+0000","last_clean":"2024-02-21T23:06:12.535991+0000","last_became_active":"2024-02-21T23:05:51.427612+0000","last_became_peered":"2024-02-21T23:05:51.427612+0000","last_unstale":"2024-02-21T23:06:12.535991+0000","last_undegraded":"2024-02-21T23:06:12.535991+0000","last_fullsized":"2024-02-21T23:06:12.535991+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:38:44.895318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.2b","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.977011+0000","last_change":"2024-02-21T23:06:10.979884+0000","last_active":"2024-02-21T23:06:11.977011+0000","last_peered":"2024-02-21T23:06:11.977011+0000","last_clean":"2024-02-21T23:06:11.977011+0000","last_became_active":"2024-02-21T23:06:10.979079+0000","last_became_peered":"2024-02-21T23:06:10.979079+0000","last_unstale":"2024-02-21T23:06:11.977011+0000","last_undegraded":"2024-02-21T23:06:11.977011+0000","last_fullsized":"2024-02-21T23:06:11.977011+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:03:54.817384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.975835+0000","last_change":"2024-02-21T23:05:51.417302+0000","last_active":"2024-02-21T23:06:11.975835+0000","last_peered":"2024-02-21T23:06:11.975835+0000","last_clean":"2024-02-21T23:06:11.975835+0000","last_became_active":"2024-02-21T23:05:51.414092+0000","last_became_peered":"2024-02-21T23:05:51.414092+0000","last_unstale":"2024-02-21T23:06:11.975835+0000","last_undegraded":"2024-02-21T23:06:11.975835+0000","last_fullsized":"2024-02-21T23:06:11.975835+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:49:01.911867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.976955+0000","last_change":"2024-02-21T23:06:10.989029+0000","last_active":"2024-02-21T23:06:11.976955+0000","last_peered":"2024-02-21T23:06:11.976955+0000","last_clean":"2024-02-21T23:06:11.976955+0000","last_became_active":"2024-02-21T23:06:10.988586+0000","last_became_peered":"2024-02-21T23:06:10.988586+0000","last_unstale":"2024-02-21T23:06:11.976955+0000","last_undegraded":"2024-02-21T23:06:11.976955+0000","last_fullsized":"2024-02-21T23:06:11.976955+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:41:05.018853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.029594+0000","last_change":"2024-02-21T23:05:51.406411+0000","last_active":"2024-02-21T23:06:13.029594+0000","last_peered":"2024-02-21T23:06:13.029594+0000","last_clean":"2024-02-21T23:06:13.029594+0000","last_became_active":"2024-02-21T23:05:51.406104+0000","last_became_peered":"2024-02-21T23:05:51.406104+0000","last_unstale":"2024-02-21T23:06:13.029594+0000","last_undegraded":"2024-02-21T23:06:13.029594+0000","last_fullsized":"2024-02-21T23:06:13.029594+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:03:08.789066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.25","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.536618+0000","last_change":"2024-02-21T23:06:10.978687+0000","last_active":"2024-02-21T23:06:12.536618+0000","last_peered":"2024-02-21T23:06:12.536618+0000","last_clean":"2024-02-21T23:06:12.536618+0000","last_became_active":"2024-02-21T23:06:10.978393+0000","last_became_peered":"2024-02-21T23:06:10.978393+0000","last_unstale":"2024-02-21T23:06:12.536618+0000","last_undegraded":"2024-02-21T23:06:12.536618+0000","last_fullsized":"2024-02-21T23:06:12.536618+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:32:04.110019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.030786+0000","last_change":"2024-02-21T23:05:51.414112+0000","last_active":"2024-02-21T23:06:13.030786+0000","last_peered":"2024-02-21T23:06:13.030786+0000","last_clean":"2024-02-21T23:06:13.030786+0000","last_became_active":"2024-02-21T23:05:51.413796+0000","last_became_peered":"2024-02-21T23:05:51.413796+0000","last_unstale":"2024-02-21T23:06:13.030786+0000","last_undegraded":"2024-02-21T23:06:13.030786+0000","last_fullsized":"2024-02-21T23:06:13.030786+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:42:26.488510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.24","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.971701+0000","last_change":"2024-02-21T23:06:10.982423+0000","last_active":"2024-02-21T23:06:11.971701+0000","last_peered":"2024-02-21T23:06:11.971701+0000","last_clean":"2024-02-21T23:06:11.971701+0000","last_became_active":"2024-02-21T23:06:10.981600+0000","last_became_peered":"2024-02-21T23:06:10.981600+0000","last_unstale":"2024-02-21T23:06:11.971701+0000","last_undegraded":"2024-02-21T23:06:11.971701+0000","last_fullsized":"2024-02-21T23:06:11.971701+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:11:16.456657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.960079+0000","last_change":"2024-02-21T23:05:51.421208+0000","last_active":"2024-02-21T23:06:12.960079+0000","last_peered":"2024-02-21T23:06:12.960079+0000","last_clean":"2024-02-21T23:06:12.960079+0000","last_became_active":"2024-02-21T23:05:51.418171+0000","last_became_peered":"2024-02-21T23:05:51.418171+0000","last_unstale":"2024-02-21T23:06:12.960079+0000","last_undegraded":"2024-02-21T23:06:12.960079+0000","last_fullsized":"2024-02-21T23:06:12.960079+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:17:26.329609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.27","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.960254+0000","last_change":"2024-02-21T23:06:10.986399+0000","last_active":"2024-02-21T23:06:12.960254+0000","last_peered":"2024-02-21T23:06:12.960254+0000","last_clean":"2024-02-21T23:06:12.960254+0000","last_became_active":"2024-02-21T23:06:10.985542+0000","last_became_peered":"2024-02-21T23:06:10.985542+0000","last_unstale":"2024-02-21T23:06:12.960254+0000","last_undegraded":"2024-02-21T23:06:12.960254+0000","last_fullsized":"2024-02-21T23:06:12.960254+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:37:11.082403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.034410+0000","last_change":"2024-02-21T23:05:51.416857+0000","last_active":"2024-02-21T23:06:13.034410+0000","last_peered":"2024-02-21T23:06:13.034410+0000","last_clean":"2024-02-21T23:06:13.034410+0000","last_became_active":"2024-02-21T23:05:51.416577+0000","last_became_peered":"2024-02-21T23:05:51.416577+0000","last_unstale":"2024-02-21T23:06:13.034410+0000","last_undegraded":"2024-02-21T23:06:13.034410+0000","last_fullsized":"2024-02-21T23:06:13.034410+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:17:26.329609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.26","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.977901+0000","last_change":"2024-02-21T23:06:10.991753+0000","last_active":"2024-02-21T23:06:11.977901+0000","last_peered":"2024-02-21T23:06:11.977901+0000","last_clean":"2024-02-21T23:06:11.977901+0000","last_became_active":"2024-02-21T23:06:10.990907+0000","last_became_peered":"2024-02-21T23:06:10.990907+0000","last_unstale":"2024-02-21T23:06:11.977901+0000","last_undegraded":"2024-02-21T23:06:11.977901+0000","last_fullsized":"2024-02-21T23:06:11.977901+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:59:32.865929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.536463+0000","last_change":"2024-02-21T23:05:51.427489+0000","last_active":"2024-02-21T23:06:12.536463+0000","last_peered":"2024-02-21T23:06:12.536463+0000","last_clean":"2024-02-21T23:06:12.536463+0000","last_became_active":"2024-02-21T23:05:51.419458+0000","last_became_peered":"2024-02-21T23:05:51.419458+0000","last_unstale":"2024-02-21T23:06:12.536463+0000","last_undegraded":"2024-02-21T23:06:12.536463+0000","last_fullsized":"2024-02-21T23:06:12.536463+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:54:08.758447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.21","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.959168+0000","last_change":"2024-02-21T23:06:10.991205+0000","last_active":"2024-02-21T23:06:12.959168+0000","last_peered":"2024-02-21T23:06:12.959168+0000","last_clean":"2024-02-21T23:06:12.959168+0000","last_became_active":"2024-02-21T23:06:10.990883+0000","last_became_peered":"2024-02-21T23:06:10.990883+0000","last_unstale":"2024-02-21T23:06:12.959168+0000","last_undegraded":"2024-02-21T23:06:12.959168+0000","last_fullsized":"2024-02-21T23:06:12.959168+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:15:06.720068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.030070+0000","last_change":"2024-02-21T23:05:51.405741+0000","last_active":"2024-02-21T23:06:13.030070+0000","last_peered":"2024-02-21T23:06:13.030070+0000","last_clean":"2024-02-21T23:06:13.030070+0000","last_became_active":"2024-02-21T23:05:51.405301+0000","last_became_peered":"2024-02-21T23:05:51.405301+0000","last_unstale":"2024-02-21T23:06:13.030070+0000","last_undegraded":"2024-02-21T23:06:13.030070+0000","last_fullsized":"2024-02-21T23:06:13.030070+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:36:47.962935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.20","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.029444+0000","last_change":"2024-02-21T23:06:10.967642+0000","last_active":"2024-02-21T23:06:13.029444+0000","last_peered":"2024-02-21T23:06:13.029444+0000","last_clean":"2024-02-21T23:06:13.029444+0000","last_became_active":"2024-02-21T23:06:10.967318+0000","last_became_peered":"2024-02-21T23:06:10.967318+0000","last_unstale":"2024-02-21T23:06:13.029444+0000","last_undegraded":"2024-02-21T23:06:13.029444+0000","last_fullsized":"2024-02-21T23:06:13.029444+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:40:55.366111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.032842+0000","last_change":"2024-02-21T23:05:51.398663+0000","last_active":"2024-02-21T23:06:13.032842+0000","last_peered":"2024-02-21T23:06:13.032842+0000","last_clean":"2024-02-21T23:06:13.032842+0000","last_became_active":"2024-02-21T23:05:51.398146+0000","last_became_peered":"2024-02-21T23:05:51.398146+0000","last_unstale":"2024-02-21T23:06:13.032842+0000","last_undegraded":"2024-02-21T23:06:13.032842+0000","last_fullsized":"2024-02-21T23:06:13.032842+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:27:54.695721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.23","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.033534+0000","last_change":"2024-02-21T23:06:10.976500+0000","last_active":"2024-02-21T23:06:13.033534+0000","last_peered":"2024-02-21T23:06:13.033534+0000","last_clean":"2024-02-21T23:06:13.033534+0000","last_became_active":"2024-02-21T23:06:10.976135+0000","last_became_peered":"2024-02-21T23:06:10.976135+0000","last_unstale":"2024-02-21T23:06:13.033534+0000","last_undegraded":"2024-02-21T23:06:13.033534+0000","last_fullsized":"2024-02-21T23:06:13.033534+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:19:08.047583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.971321+0000","last_change":"2024-02-21T23:05:51.421330+0000","last_active":"2024-02-21T23:06:11.971321+0000","last_peered":"2024-02-21T23:06:11.971321+0000","last_clean":"2024-02-21T23:06:11.971321+0000","last_became_active":"2024-02-21T23:05:51.421012+0000","last_became_peered":"2024-02-21T23:05:51.421012+0000","last_unstale":"2024-02-21T23:06:11.971321+0000","last_undegraded":"2024-02-21T23:06:11.971321+0000","last_fullsized":"2024-02-21T23:06:11.971321+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:26:33.366331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.22","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.976531+0000","last_change":"2024-02-21T23:06:10.983139+0000","last_active":"2024-02-21T23:06:11.976531+0000","last_peered":"2024-02-21T23:06:11.976531+0000","last_clean":"2024-02-21T23:06:11.976531+0000","last_became_active":"2024-02-21T23:06:10.982029+0000","last_became_peered":"2024-02-21T23:06:10.982029+0000","last_unstale":"2024-02-21T23:06:11.976531+0000","last_undegraded":"2024-02-21T23:06:11.976531+0000","last_fullsized":"2024-02-21T23:06:11.976531+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:42:46.061945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.034425+0000","last_change":"2024-02-21T23:05:51.403173+0000","last_active":"2024-02-21T23:06:13.034425+0000","last_peered":"2024-02-21T23:06:13.034425+0000","last_clean":"2024-02-21T23:06:13.034425+0000","last_became_active":"2024-02-21T23:05:51.402610+0000","last_became_peered":"2024-02-21T23:05:51.402610+0000","last_unstale":"2024-02-21T23:06:13.034425+0000","last_undegraded":"2024-02-21T23:06:13.034425+0000","last_fullsized":"2024-02-21T23:06:13.034425+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:46:01.608149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.030709+0000","last_change":"2024-02-21T23:06:10.977357+0000","last_active":"2024-02-21T23:06:13.030709+0000","last_peered":"2024-02-21T23:06:13.030709+0000","last_clean":"2024-02-21T23:06:13.030709+0000","last_became_active":"2024-02-21T23:06:10.977075+0000","last_became_peered":"2024-02-21T23:06:10.977075+0000","last_unstale":"2024-02-21T23:06:13.030709+0000","last_undegraded":"2024-02-21T23:06:13.030709+0000","last_fullsized":"2024-02-21T23:06:13.030709+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:33:34.425087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.976736+0000","last_change":"2024-02-21T23:06:09.955402+0000","last_active":"2024-02-21T23:06:11.976736+0000","last_peered":"2024-02-21T23:06:11.976736+0000","last_clean":"2024-02-21T23:06:11.976736+0000","last_became_active":"2024-02-21T23:06:09.955113+0000","last_became_peered":"2024-02-21T23:06:09.955113+0000","last_unstale":"2024-02-21T23:06:11.976736+0000","last_undegraded":"2024-02-21T23:06:11.976736+0000","last_fullsized":"2024-02-21T23:06:11.976736+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:37:10.069490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.971922+0000","last_change":"2024-02-21T23:05:51.416534+0000","last_active":"2024-02-21T23:06:11.971922+0000","last_peered":"2024-02-21T23:06:11.971922+0000","last_clean":"2024-02-21T23:06:11.971922+0000","last_became_active":"2024-02-21T23:05:51.416244+0000","last_became_peered":"2024-02-21T23:05:51.416244+0000","last_unstale":"2024-02-21T23:06:11.971922+0000","last_undegraded":"2024-02-21T23:06:11.971922+0000","last_fullsized":"2024-02-21T23:06:11.971922+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:42:37.006685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"75'4","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.536913+0000","last_change":"2024-02-21T23:05:50.354109+0000","last_active":"2024-02-21T23:06:12.536913+0000","last_peered":"2024-02-21T23:06:12.536913+0000","last_clean":"2024-02-21T23:06:12.536913+0000","last_became_active":"2024-02-21T23:05:50.353809+0000","last_became_peered":"2024-02-21T23:05:50.353809+0000","last_unstale":"2024-02-21T23:06:12.536913+0000","last_undegraded":"2024-02-21T23:06:12.536913+0000","last_fullsized":"2024-02-21T23:06:12.536913+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:28:13.765557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.972268+0000","last_change":"2024-02-21T23:06:10.985497+0000","last_active":"2024-02-21T23:06:11.972268+0000","last_peered":"2024-02-21T23:06:11.972268+0000","last_clean":"2024-02-21T23:06:11.972268+0000","last_became_active":"2024-02-21T23:06:10.985191+0000","last_became_peered":"2024-02-21T23:06:10.985191+0000","last_unstale":"2024-02-21T23:06:11.972268+0000","last_undegraded":"2024-02-21T23:06:11.972268+0000","last_fullsized":"2024-02-21T23:06:11.972268+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:41:05.018853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.033912+0000","last_change":"2024-02-21T23:06:09.963140+0000","last_active":"2024-02-21T23:06:13.033912+0000","last_peered":"2024-02-21T23:06:13.033912+0000","last_clean":"2024-02-21T23:06:13.033912+0000","last_became_active":"2024-02-21T23:06:09.962637+0000","last_became_peered":"2024-02-21T23:06:09.962637+0000","last_unstale":"2024-02-21T23:06:13.033912+0000","last_undegraded":"2024-02-21T23:06:13.033912+0000","last_fullsized":"2024-02-21T23:06:13.033912+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:41:49.821364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.029524+0000","last_change":"2024-02-21T23:05:51.416268+0000","last_active":"2024-02-21T23:06:13.029524+0000","last_peered":"2024-02-21T23:06:13.029524+0000","last_clean":"2024-02-21T23:06:13.029524+0000","last_became_active":"2024-02-21T23:05:51.415980+0000","last_became_peered":"2024-02-21T23:05:51.415980+0000","last_unstale":"2024-02-21T23:06:13.029524+0000","last_undegraded":"2024-02-21T23:06:13.029524+0000","last_fullsized":"2024-02-21T23:06:13.029524+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:51:17.193881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"75'1","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.035404+0000","last_change":"2024-02-21T23:05:50.383438+0000","last_active":"2024-02-21T23:06:13.035404+0000","last_peered":"2024-02-21T23:06:13.035404+0000","last_clean":"2024-02-21T23:06:13.035404+0000","last_became_active":"2024-02-21T23:05:50.383134+0000","last_became_peered":"2024-02-21T23:05:50.383134+0000","last_unstale":"2024-02-21T23:06:13.035404+0000","last_undegraded":"2024-02-21T23:06:13.035404+0000","last_fullsized":"2024-02-21T23:06:13.035404+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:08:35.448428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.971272+0000","last_change":"2024-02-21T23:06:10.970588+0000","last_active":"2024-02-21T23:06:11.971272+0000","last_peered":"2024-02-21T23:06:11.971272+0000","last_clean":"2024-02-21T23:06:11.971272+0000","last_became_active":"2024-02-21T23:06:10.970285+0000","last_became_peered":"2024-02-21T23:06:10.970285+0000","last_unstale":"2024-02-21T23:06:11.971272+0000","last_undegraded":"2024-02-21T23:06:11.971272+0000","last_fullsized":"2024-02-21T23:06:11.971272+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:20:00.854022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"79'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.976577+0000","last_change":"2024-02-21T23:06:09.957178+0000","last_active":"2024-02-21T23:06:11.976577+0000","last_peered":"2024-02-21T23:06:11.976577+0000","last_clean":"2024-02-21T23:06:11.976577+0000","last_became_active":"2024-02-21T23:06:09.956818+0000","last_became_peered":"2024-02-21T23:06:09.956818+0000","last_unstale":"2024-02-21T23:06:11.976577+0000","last_undegraded":"2024-02-21T23:06:11.976577+0000","last_fullsized":"2024-02-21T23:06:11.976577+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:24:34.391679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.977011+0000","last_change":"2024-02-21T23:05:51.421571+0000","last_active":"2024-02-21T23:06:11.977011+0000","last_peered":"2024-02-21T23:06:11.977011+0000","last_clean":"2024-02-21T23:06:11.977011+0000","last_became_active":"2024-02-21T23:05:51.421110+0000","last_became_peered":"2024-02-21T23:05:51.421110+0000","last_unstale":"2024-02-21T23:06:11.977011+0000","last_undegraded":"2024-02-21T23:06:11.977011+0000","last_fullsized":"2024-02-21T23:06:11.977011+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:56:36.688152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"75'2","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.034745+0000","last_change":"2024-02-21T23:05:50.383231+0000","last_active":"2024-02-21T23:06:13.034745+0000","last_peered":"2024-02-21T23:06:13.034745+0000","last_clean":"2024-02-21T23:06:13.034745+0000","last_became_active":"2024-02-21T23:05:50.382757+0000","last_became_peered":"2024-02-21T23:05:50.382757+0000","last_unstale":"2024-02-21T23:06:13.034745+0000","last_undegraded":"2024-02-21T23:06:13.034745+0000","last_fullsized":"2024-02-21T23:06:13.034745+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:49:36.738197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.034005+0000","last_change":"2024-02-21T23:06:10.979404+0000","last_active":"2024-02-21T23:06:13.034005+0000","last_peered":"2024-02-21T23:06:13.034005+0000","last_clean":"2024-02-21T23:06:13.034005+0000","last_became_active":"2024-02-21T23:06:10.979128+0000","last_became_peered":"2024-02-21T23:06:10.979128+0000","last_unstale":"2024-02-21T23:06:13.034005+0000","last_undegraded":"2024-02-21T23:06:13.034005+0000","last_fullsized":"2024-02-21T23:06:13.034005+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:20:00.854022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1a","version":"79'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.971772+0000","last_change":"2024-02-21T23:06:09.951255+0000","last_active":"2024-02-21T23:06:11.971772+0000","last_peered":"2024-02-21T23:06:11.971772+0000","last_clean":"2024-02-21T23:06:11.971772+0000","last_became_active":"2024-02-21T23:06:09.950934+0000","last_became_peered":"2024-02-21T23:06:09.950934+0000","last_unstale":"2024-02-21T23:06:11.971772+0000","last_undegraded":"2024-02-21T23:06:11.971772+0000","last_fullsized":"2024-02-21T23:06:11.971772+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:56:08.310911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.977534+0000","last_change":"2024-02-21T23:05:51.415892+0000","last_active":"2024-02-21T23:06:11.977534+0000","last_peered":"2024-02-21T23:06:11.977534+0000","last_clean":"2024-02-21T23:06:11.977534+0000","last_became_active":"2024-02-21T23:05:51.415520+0000","last_became_peered":"2024-02-21T23:05:51.415520+0000","last_unstale":"2024-02-21T23:06:11.977534+0000","last_undegraded":"2024-02-21T23:06:11.977534+0000","last_fullsized":"2024-02-21T23:06:11.977534+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:26:33.366331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"75'2","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.035819+0000","last_change":"2024-02-21T23:05:50.376050+0000","last_active":"2024-02-21T23:06:13.035819+0000","last_peered":"2024-02-21T23:06:13.035819+0000","last_clean":"2024-02-21T23:06:13.035819+0000","last_became_active":"2024-02-21T23:05:50.375731+0000","last_became_peered":"2024-02-21T23:05:50.375731+0000","last_unstale":"2024-02-21T23:06:13.035819+0000","last_undegraded":"2024-02-21T23:06:13.035819+0000","last_fullsized":"2024-02-21T23:06:13.035819+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:42:15.626763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.034775+0000","last_change":"2024-02-21T23:06:10.992590+0000","last_active":"2024-02-21T23:06:13.034775+0000","last_peered":"2024-02-21T23:06:13.034775+0000","last_clean":"2024-02-21T23:06:13.034775+0000","last_became_active":"2024-02-21T23:06:10.992286+0000","last_became_peered":"2024-02-21T23:06:10.992286+0000","last_unstale":"2024-02-21T23:06:13.034775+0000","last_undegraded":"2024-02-21T23:06:13.034775+0000","last_fullsized":"2024-02-21T23:06:13.034775+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:31:22.009771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.1d","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.034121+0000","last_change":"2024-02-21T23:06:09.957048+0000","last_active":"2024-02-21T23:06:13.034121+0000","last_peered":"2024-02-21T23:06:13.034121+0000","last_clean":"2024-02-21T23:06:13.034121+0000","last_became_active":"2024-02-21T23:06:09.956666+0000","last_became_peered":"2024-02-21T23:06:09.956666+0000","last_unstale":"2024-02-21T23:06:13.034121+0000","last_undegraded":"2024-02-21T23:06:13.034121+0000","last_fullsized":"2024-02-21T23:06:13.034121+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:24:51.423593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.030148+0000","last_change":"2024-02-21T23:05:51.428435+0000","last_active":"2024-02-21T23:06:13.030148+0000","last_peered":"2024-02-21T23:06:13.030148+0000","last_clean":"2024-02-21T23:06:13.030148+0000","last_became_active":"2024-02-21T23:05:51.428133+0000","last_became_peered":"2024-02-21T23:05:51.428133+0000","last_unstale":"2024-02-21T23:06:13.030148+0000","last_undegraded":"2024-02-21T23:06:13.030148+0000","last_fullsized":"2024-02-21T23:06:13.030148+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:49:46.025250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"75'1","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.978182+0000","last_change":"2024-02-21T23:05:50.377574+0000","last_active":"2024-02-21T23:06:11.978182+0000","last_peered":"2024-02-21T23:06:11.978182+0000","last_clean":"2024-02-21T23:06:11.978182+0000","last_became_active":"2024-02-21T23:05:50.377250+0000","last_became_peered":"2024-02-21T23:05:50.377250+0000","last_unstale":"2024-02-21T23:06:11.978182+0000","last_undegraded":"2024-02-21T23:06:11.978182+0000","last_fullsized":"2024-02-21T23:06:11.978182+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:26:38.366160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.030691+0000","last_change":"2024-02-21T23:06:10.983355+0000","last_active":"2024-02-21T23:06:13.030691+0000","last_peered":"2024-02-21T23:06:13.030691+0000","last_clean":"2024-02-21T23:06:13.030691+0000","last_became_active":"2024-02-21T23:06:10.982259+0000","last_became_peered":"2024-02-21T23:06:10.982259+0000","last_unstale":"2024-02-21T23:06:13.030691+0000","last_undegraded":"2024-02-21T23:06:13.030691+0000","last_fullsized":"2024-02-21T23:06:13.030691+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:19:24.322783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1c","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.035229+0000","last_change":"2024-02-21T23:06:09.947108+0000","last_active":"2024-02-21T23:06:13.035229+0000","last_peered":"2024-02-21T23:06:13.035229+0000","last_clean":"2024-02-21T23:06:13.035229+0000","last_became_active":"2024-02-21T23:06:09.946815+0000","last_became_peered":"2024-02-21T23:06:09.946815+0000","last_unstale":"2024-02-21T23:06:13.035229+0000","last_undegraded":"2024-02-21T23:06:13.035229+0000","last_fullsized":"2024-02-21T23:06:13.035229+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:03:27.349588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.030548+0000","last_change":"2024-02-21T23:05:51.421508+0000","last_active":"2024-02-21T23:06:13.030548+0000","last_peered":"2024-02-21T23:06:13.030548+0000","last_clean":"2024-02-21T23:06:13.030548+0000","last_became_active":"2024-02-21T23:05:51.420041+0000","last_became_peered":"2024-02-21T23:05:51.420041+0000","last_unstale":"2024-02-21T23:06:13.030548+0000","last_undegraded":"2024-02-21T23:06:13.030548+0000","last_fullsized":"2024-02-21T23:06:13.030548+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:10:56.883222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"75'1","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.959136+0000","last_change":"2024-02-21T23:05:50.378668+0000","last_active":"2024-02-21T23:06:12.959136+0000","last_peered":"2024-02-21T23:06:12.959136+0000","last_clean":"2024-02-21T23:06:12.959136+0000","last_became_active":"2024-02-21T23:05:50.378369+0000","last_became_peered":"2024-02-21T23:05:50.378369+0000","last_unstale":"2024-02-21T23:06:12.959136+0000","last_undegraded":"2024-02-21T23:06:12.959136+0000","last_fullsized":"2024-02-21T23:06:12.959136+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:45:05.837402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.029928+0000","last_change":"2024-02-21T23:06:10.988234+0000","last_active":"2024-02-21T23:06:13.029928+0000","last_peered":"2024-02-21T23:06:13.029928+0000","last_clean":"2024-02-21T23:06:13.029928+0000","last_became_active":"2024-02-21T23:06:10.985564+0000","last_became_peered":"2024-02-21T23:06:10.985564+0000","last_unstale":"2024-02-21T23:06:13.029928+0000","last_undegraded":"2024-02-21T23:06:13.029928+0000","last_fullsized":"2024-02-21T23:06:13.029928+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:35:03.330756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"79'4","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.035823+0000","last_change":"2024-02-21T23:06:09.949620+0000","last_active":"2024-02-21T23:06:13.035823+0000","last_peered":"2024-02-21T23:06:13.035823+0000","last_clean":"2024-02-21T23:06:13.035823+0000","last_became_active":"2024-02-21T23:06:09.949252+0000","last_became_peered":"2024-02-21T23:06:09.949252+0000","last_unstale":"2024-02-21T23:06:13.035823+0000","last_undegraded":"2024-02-21T23:06:13.035823+0000","last_fullsized":"2024-02-21T23:06:13.035823+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:46:20.168671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.979280+0000","last_change":"2024-02-21T23:05:51.419084+0000","last_active":"2024-02-21T23:06:11.979280+0000","last_peered":"2024-02-21T23:06:11.979280+0000","last_clean":"2024-02-21T23:06:11.979280+0000","last_became_active":"2024-02-21T23:05:51.418752+0000","last_became_peered":"2024-02-21T23:05:51.418752+0000","last_unstale":"2024-02-21T23:06:11.979280+0000","last_undegraded":"2024-02-21T23:06:11.979280+0000","last_fullsized":"2024-02-21T23:06:11.979280+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:20:19.564331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.029581+0000","last_change":"2024-02-21T23:05:50.375534+0000","last_active":"2024-02-21T23:06:13.029581+0000","last_peered":"2024-02-21T23:06:13.029581+0000","last_clean":"2024-02-21T23:06:13.029581+0000","last_became_active":"2024-02-21T23:05:50.373762+0000","last_became_peered":"2024-02-21T23:05:50.373762+0000","last_unstale":"2024-02-21T23:06:13.029581+0000","last_undegraded":"2024-02-21T23:06:13.029581+0000","last_fullsized":"2024-02-21T23:06:13.029581+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:49:36.738197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.030217+0000","last_change":"2024-02-21T23:06:10.964764+0000","last_active":"2024-02-21T23:06:13.030217+0000","last_peered":"2024-02-21T23:06:13.030217+0000","last_clean":"2024-02-21T23:06:13.030217+0000","last_became_active":"2024-02-21T23:06:10.964494+0000","last_became_peered":"2024-02-21T23:06:10.964494+0000","last_unstale":"2024-02-21T23:06:13.030217+0000","last_undegraded":"2024-02-21T23:06:13.030217+0000","last_fullsized":"2024-02-21T23:06:13.030217+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:53:45.860442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"79'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.977530+0000","last_change":"2024-02-21T23:06:09.952374+0000","last_active":"2024-02-21T23:06:11.977530+0000","last_peered":"2024-02-21T23:06:11.977530+0000","last_clean":"2024-02-21T23:06:11.977530+0000","last_became_active":"2024-02-21T23:06:09.950283+0000","last_became_peered":"2024-02-21T23:06:09.950283+0000","last_unstale":"2024-02-21T23:06:11.977530+0000","last_undegraded":"2024-02-21T23:06:11.977530+0000","last_fullsized":"2024-02-21T23:06:11.977530+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:50:51.483608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.036707+0000","last_change":"2024-02-21T23:05:51.426711+0000","last_active":"2024-02-21T23:06:13.036707+0000","last_peered":"2024-02-21T23:06:13.036707+0000","last_clean":"2024-02-21T23:06:13.036707+0000","last_became_active":"2024-02-21T23:05:51.426415+0000","last_became_peered":"2024-02-21T23:05:51.426415+0000","last_unstale":"2024-02-21T23:06:13.036707+0000","last_undegraded":"2024-02-21T23:06:13.036707+0000","last_fullsized":"2024-02-21T23:06:13.036707+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:58:55.829941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.535998+0000","last_change":"2024-02-21T23:05:50.355567+0000","last_active":"2024-02-21T23:06:12.535998+0000","last_peered":"2024-02-21T23:06:12.535998+0000","last_clean":"2024-02-21T23:06:12.535998+0000","last_became_active":"2024-02-21T23:05:50.354835+0000","last_became_peered":"2024-02-21T23:05:50.354835+0000","last_unstale":"2024-02-21T23:06:12.535998+0000","last_undegraded":"2024-02-21T23:06:12.535998+0000","last_fullsized":"2024-02-21T23:06:12.535998+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:21:52.655098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.535844+0000","last_change":"2024-02-21T23:06:10.992164+0000","last_active":"2024-02-21T23:06:12.535844+0000","last_peered":"2024-02-21T23:06:12.535844+0000","last_clean":"2024-02-21T23:06:12.535844+0000","last_became_active":"2024-02-21T23:06:10.991804+0000","last_became_peered":"2024-02-21T23:06:10.991804+0000","last_unstale":"2024-02-21T23:06:12.535844+0000","last_undegraded":"2024-02-21T23:06:12.535844+0000","last_fullsized":"2024-02-21T23:06:12.535844+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:49:21.485302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.971149+0000","last_change":"2024-02-21T23:06:09.946843+0000","last_active":"2024-02-21T23:06:11.971149+0000","last_peered":"2024-02-21T23:06:11.971149+0000","last_clean":"2024-02-21T23:06:11.971149+0000","last_became_active":"2024-02-21T23:06:09.946452+0000","last_became_peered":"2024-02-21T23:06:09.946452+0000","last_unstale":"2024-02-21T23:06:11.971149+0000","last_undegraded":"2024-02-21T23:06:11.971149+0000","last_fullsized":"2024-02-21T23:06:11.971149+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:50:51.483608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.029696+0000","last_change":"2024-02-21T23:05:51.399923+0000","last_active":"2024-02-21T23:06:13.029696+0000","last_peered":"2024-02-21T23:06:13.029696+0000","last_clean":"2024-02-21T23:06:13.029696+0000","last_became_active":"2024-02-21T23:05:51.399642+0000","last_became_peered":"2024-02-21T23:05:51.399642+0000","last_unstale":"2024-02-21T23:06:13.029696+0000","last_undegraded":"2024-02-21T23:06:13.029696+0000","last_fullsized":"2024-02-21T23:06:13.029696+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:21:18.440977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.035962+0000","last_change":"2024-02-21T23:05:50.386639+0000","last_active":"2024-02-21T23:06:13.035962+0000","last_peered":"2024-02-21T23:06:13.035962+0000","last_clean":"2024-02-21T23:06:13.035962+0000","last_became_active":"2024-02-21T23:05:50.386158+0000","last_became_peered":"2024-02-21T23:05:50.386158+0000","last_unstale":"2024-02-21T23:06:13.035962+0000","last_undegraded":"2024-02-21T23:06:13.035962+0000","last_fullsized":"2024-02-21T23:06:13.035962+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:43:17.496894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.536225+0000","last_change":"2024-02-21T23:06:10.983360+0000","last_active":"2024-02-21T23:06:12.536225+0000","last_peered":"2024-02-21T23:06:12.536225+0000","last_clean":"2024-02-21T23:06:12.536225+0000","last_became_active":"2024-02-21T23:06:10.983093+0000","last_became_peered":"2024-02-21T23:06:10.983093+0000","last_unstale":"2024-02-21T23:06:12.536225+0000","last_undegraded":"2024-02-21T23:06:12.536225+0000","last_fullsized":"2024-02-21T23:06:12.536225+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:56:56.261587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.10","version":"79'8","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.971653+0000","last_change":"2024-02-21T23:06:09.934904+0000","last_active":"2024-02-21T23:06:11.971653+0000","last_peered":"2024-02-21T23:06:11.971653+0000","last_clean":"2024-02-21T23:06:11.971653+0000","last_became_active":"2024-02-21T23:06:09.934606+0000","last_became_peered":"2024-02-21T23:06:09.934606+0000","last_unstale":"2024-02-21T23:06:11.971653+0000","last_undegraded":"2024-02-21T23:06:11.971653+0000","last_fullsized":"2024-02-21T23:06:11.971653+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:51:46.950044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.959439+0000","last_change":"2024-02-21T23:05:51.422844+0000","last_active":"2024-02-21T23:06:12.959439+0000","last_peered":"2024-02-21T23:06:12.959439+0000","last_clean":"2024-02-21T23:06:12.959439+0000","last_became_active":"2024-02-21T23:05:51.422488+0000","last_became_peered":"2024-02-21T23:05:51.422488+0000","last_unstale":"2024-02-21T23:06:12.959439+0000","last_undegraded":"2024-02-21T23:06:12.959439+0000","last_fullsized":"2024-02-21T23:06:12.959439+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:27:54.695721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.960134+0000","last_change":"2024-02-21T23:05:50.385146+0000","last_active":"2024-02-21T23:06:12.960134+0000","last_peered":"2024-02-21T23:06:12.960134+0000","last_clean":"2024-02-21T23:06:12.960134+0000","last_became_active":"2024-02-21T23:05:50.384418+0000","last_became_peered":"2024-02-21T23:05:50.384418+0000","last_unstale":"2024-02-21T23:06:12.960134+0000","last_undegraded":"2024-02-21T23:06:12.960134+0000","last_fullsized":"2024-02-21T23:06:12.960134+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:43:17.496894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.971506+0000","last_change":"2024-02-21T23:06:10.990710+0000","last_active":"2024-02-21T23:06:11.971506+0000","last_peered":"2024-02-21T23:06:11.971506+0000","last_clean":"2024-02-21T23:06:11.971506+0000","last_became_active":"2024-02-21T23:06:10.980292+0000","last_became_peered":"2024-02-21T23:06:10.980292+0000","last_unstale":"2024-02-21T23:06:11.971506+0000","last_undegraded":"2024-02-21T23:06:11.971506+0000","last_fullsized":"2024-02-21T23:06:11.971506+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:51:30.184287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.033819+0000","last_change":"2024-02-21T23:06:09.963033+0000","last_active":"2024-02-21T23:06:13.033819+0000","last_peered":"2024-02-21T23:06:13.033819+0000","last_clean":"2024-02-21T23:06:13.033819+0000","last_became_active":"2024-02-21T23:06:09.962695+0000","last_became_peered":"2024-02-21T23:06:09.962695+0000","last_unstale":"2024-02-21T23:06:13.033819+0000","last_undegraded":"2024-02-21T23:06:13.033819+0000","last_fullsized":"2024-02-21T23:06:13.033819+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:41:04.005940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.976677+0000","last_change":"2024-02-21T23:05:51.418004+0000","last_active":"2024-02-21T23:06:11.976677+0000","last_peered":"2024-02-21T23:06:11.976677+0000","last_clean":"2024-02-21T23:06:11.976677+0000","last_became_active":"2024-02-21T23:05:51.417463+0000","last_became_peered":"2024-02-21T23:05:51.417463+0000","last_unstale":"2024-02-21T23:06:11.976677+0000","last_undegraded":"2024-02-21T23:06:11.976677+0000","last_fullsized":"2024-02-21T23:06:11.976677+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:59:13.292494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"75'3","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.536685+0000","last_change":"2024-02-21T23:05:50.373189+0000","last_active":"2024-02-21T23:06:12.536685+0000","last_peered":"2024-02-21T23:06:12.536685+0000","last_clean":"2024-02-21T23:06:12.536685+0000","last_became_active":"2024-02-21T23:05:50.370680+0000","last_became_peered":"2024-02-21T23:05:50.370680+0000","last_unstale":"2024-02-21T23:06:12.536685+0000","last_undegraded":"2024-02-21T23:06:12.536685+0000","last_fullsized":"2024-02-21T23:06:12.536685+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:21:03.121546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.975807+0000","last_change":"2024-02-21T23:06:10.980721+0000","last_active":"2024-02-21T23:06:11.975807+0000","last_peered":"2024-02-21T23:06:11.975807+0000","last_clean":"2024-02-21T23:06:11.975807+0000","last_became_active":"2024-02-21T23:06:10.980424+0000","last_became_peered":"2024-02-21T23:06:10.980424+0000","last_unstale":"2024-02-21T23:06:11.975807+0000","last_undegraded":"2024-02-21T23:06:11.975807+0000","last_fullsized":"2024-02-21T23:06:11.975807+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:51:30.184287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.029909+0000","last_change":"2024-02-21T23:06:09.961844+0000","last_active":"2024-02-21T23:06:13.029909+0000","last_peered":"2024-02-21T23:06:13.029909+0000","last_clean":"2024-02-21T23:06:13.029909+0000","last_became_active":"2024-02-21T23:06:09.961480+0000","last_became_peered":"2024-02-21T23:06:09.961480+0000","last_unstale":"2024-02-21T23:06:13.029909+0000","last_undegraded":"2024-02-21T23:06:13.029909+0000","last_fullsized":"2024-02-21T23:06:13.029909+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:29:27.651018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.972413+0000","last_change":"2024-02-21T23:05:51.405242+0000","last_active":"2024-02-21T23:06:11.972413+0000","last_peered":"2024-02-21T23:06:11.972413+0000","last_clean":"2024-02-21T23:06:11.972413+0000","last_became_active":"2024-02-21T23:05:51.403777+0000","last_became_peered":"2024-02-21T23:05:51.403777+0000","last_unstale":"2024-02-21T23:06:11.972413+0000","last_undegraded":"2024-02-21T23:06:11.972413+0000","last_fullsized":"2024-02-21T23:06:11.972413+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:34:14.420091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"75'6","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.036075+0000","last_change":"2024-02-21T23:05:50.374208+0000","last_active":"2024-02-21T23:06:13.036075+0000","last_peered":"2024-02-21T23:06:13.036075+0000","last_clean":"2024-02-21T23:06:13.036075+0000","last_became_active":"2024-02-21T23:05:50.373869+0000","last_became_peered":"2024-02-21T23:05:50.373869+0000","last_unstale":"2024-02-21T23:06:13.036075+0000","last_undegraded":"2024-02-21T23:06:13.036075+0000","last_fullsized":"2024-02-21T23:06:13.036075+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:56:49.435540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.978674+0000","last_change":"2024-02-21T23:06:10.960877+0000","last_active":"2024-02-21T23:06:11.978674+0000","last_peered":"2024-02-21T23:06:11.978674+0000","last_clean":"2024-02-21T23:06:11.978674+0000","last_became_active":"2024-02-21T23:06:10.960614+0000","last_became_peered":"2024-02-21T23:06:10.960614+0000","last_unstale":"2024-02-21T23:06:11.978674+0000","last_undegraded":"2024-02-21T23:06:11.978674+0000","last_fullsized":"2024-02-21T23:06:11.978674+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:40:55.366111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.15","version":"79'3","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.978496+0000","last_change":"2024-02-21T23:06:09.963215+0000","last_active":"2024-02-21T23:06:11.978496+0000","last_peered":"2024-02-21T23:06:11.978496+0000","last_clean":"2024-02-21T23:06:11.978496+0000","last_became_active":"2024-02-21T23:06:09.962779+0000","last_became_peered":"2024-02-21T23:06:09.962779+0000","last_unstale":"2024-02-21T23:06:11.978496+0000","last_undegraded":"2024-02-21T23:06:11.978496+0000","last_fullsized":"2024-02-21T23:06:11.978496+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:22:35.988029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.959882+0000","last_change":"2024-02-21T23:05:51.423479+0000","last_active":"2024-02-21T23:06:12.959882+0000","last_peered":"2024-02-21T23:06:12.959882+0000","last_clean":"2024-02-21T23:06:12.959882+0000","last_became_active":"2024-02-21T23:05:51.423190+0000","last_became_peered":"2024-02-21T23:05:51.423190+0000","last_unstale":"2024-02-21T23:06:12.959882+0000","last_undegraded":"2024-02-21T23:06:12.959882+0000","last_fullsized":"2024-02-21T23:06:12.959882+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:03:08.789066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"75'2","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.958906+0000","last_change":"2024-02-21T23:05:50.381643+0000","last_active":"2024-02-21T23:06:12.958906+0000","last_peered":"2024-02-21T23:06:12.958906+0000","last_clean":"2024-02-21T23:06:12.958906+0000","last_became_active":"2024-02-21T23:05:50.381345+0000","last_became_peered":"2024-02-21T23:05:50.381345+0000","last_unstale":"2024-02-21T23:06:12.958906+0000","last_undegraded":"2024-02-21T23:06:12.958906+0000","last_fullsized":"2024-02-21T23:06:12.958906+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:01:27.369089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.972762+0000","last_change":"2024-02-21T23:06:10.987880+0000","last_active":"2024-02-21T23:06:11.972762+0000","last_peered":"2024-02-21T23:06:11.972762+0000","last_clean":"2024-02-21T23:06:11.972762+0000","last_became_active":"2024-02-21T23:06:10.987579+0000","last_became_peered":"2024-02-21T23:06:10.987579+0000","last_unstale":"2024-02-21T23:06:11.972762+0000","last_undegraded":"2024-02-21T23:06:11.972762+0000","last_fullsized":"2024-02-21T23:06:11.972762+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:03:54.817384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"79'6","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.030037+0000","last_change":"2024-02-21T23:06:09.959967+0000","last_active":"2024-02-21T23:06:13.030037+0000","last_peered":"2024-02-21T23:06:13.030037+0000","last_clean":"2024-02-21T23:06:13.030037+0000","last_became_active":"2024-02-21T23:06:09.959175+0000","last_became_peered":"2024-02-21T23:06:09.959175+0000","last_unstale":"2024-02-21T23:06:13.030037+0000","last_undegraded":"2024-02-21T23:06:13.030037+0000","last_fullsized":"2024-02-21T23:06:13.030037+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:00:41.536878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.035432+0000","last_change":"2024-02-21T23:05:51.416753+0000","last_active":"2024-02-21T23:06:13.035432+0000","last_peered":"2024-02-21T23:06:13.035432+0000","last_clean":"2024-02-21T23:06:13.035432+0000","last_became_active":"2024-02-21T23:05:51.416427+0000","last_became_peered":"2024-02-21T23:05:51.416427+0000","last_unstale":"2024-02-21T23:06:13.035432+0000","last_undegraded":"2024-02-21T23:06:13.035432+0000","last_fullsized":"2024-02-21T23:06:13.035432+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:34:14.420091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.033151+0000","last_change":"2024-02-21T23:05:50.381962+0000","last_active":"2024-02-21T23:06:13.033151+0000","last_peered":"2024-02-21T23:06:13.033151+0000","last_clean":"2024-02-21T23:06:13.033151+0000","last_became_active":"2024-02-21T23:05:50.380838+0000","last_became_peered":"2024-02-21T23:05:50.380838+0000","last_unstale":"2024-02-21T23:06:13.033151+0000","last_undegraded":"2024-02-21T23:06:13.033151+0000","last_fullsized":"2024-02-21T23:06:13.033151+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:10:10.973567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.976746+0000","last_change":"2024-02-21T23:06:10.974053+0000","last_active":"2024-02-21T23:06:11.976746+0000","last_peered":"2024-02-21T23:06:11.976746+0000","last_clean":"2024-02-21T23:06:11.976746+0000","last_became_active":"2024-02-21T23:06:10.973635+0000","last_became_peered":"2024-02-21T23:06:10.973635+0000","last_unstale":"2024-02-21T23:06:11.976746+0000","last_undegraded":"2024-02-21T23:06:11.976746+0000","last_fullsized":"2024-02-21T23:06:11.976746+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:01:36.530063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.960404+0000","last_change":"2024-02-21T23:06:09.951230+0000","last_active":"2024-02-21T23:06:12.960404+0000","last_peered":"2024-02-21T23:06:12.960404+0000","last_clean":"2024-02-21T23:06:12.960404+0000","last_became_active":"2024-02-21T23:06:09.950920+0000","last_became_peered":"2024-02-21T23:06:09.950920+0000","last_unstale":"2024-02-21T23:06:12.960404+0000","last_undegraded":"2024-02-21T23:06:12.960404+0000","last_fullsized":"2024-02-21T23:06:12.960404+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:24:51.423593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.036415+0000","last_change":"2024-02-21T23:05:51.408507+0000","last_active":"2024-02-21T23:06:13.036415+0000","last_peered":"2024-02-21T23:06:13.036415+0000","last_clean":"2024-02-21T23:06:13.036415+0000","last_became_active":"2024-02-21T23:05:51.408198+0000","last_became_peered":"2024-02-21T23:05:51.408198+0000","last_unstale":"2024-02-21T23:06:13.036415+0000","last_undegraded":"2024-02-21T23:06:13.036415+0000","last_fullsized":"2024-02-21T23:06:13.036415+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:08:36.461250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.535784+0000","last_change":"2024-02-21T23:05:50.353116+0000","last_active":"2024-02-21T23:06:12.535784+0000","last_peered":"2024-02-21T23:06:12.535784+0000","last_clean":"2024-02-21T23:06:12.535784+0000","last_became_active":"2024-02-21T23:05:50.352768+0000","last_became_peered":"2024-02-21T23:05:50.352768+0000","last_unstale":"2024-02-21T23:06:12.535784+0000","last_undegraded":"2024-02-21T23:06:12.535784+0000","last_fullsized":"2024-02-21T23:06:12.535784+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:52:09.073053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.035220+0000","last_change":"2024-02-21T23:06:10.987816+0000","last_active":"2024-02-21T23:06:13.035220+0000","last_peered":"2024-02-21T23:06:13.035220+0000","last_clean":"2024-02-21T23:06:13.035220+0000","last_became_active":"2024-02-21T23:06:10.987534+0000","last_became_peered":"2024-02-21T23:06:10.987534+0000","last_unstale":"2024-02-21T23:06:13.035220+0000","last_undegraded":"2024-02-21T23:06:13.035220+0000","last_fullsized":"2024-02-21T23:06:13.035220+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:59:19.322366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.978937+0000","last_change":"2024-02-21T23:06:09.954937+0000","last_active":"2024-02-21T23:06:11.978937+0000","last_peered":"2024-02-21T23:06:11.978937+0000","last_clean":"2024-02-21T23:06:11.978937+0000","last_became_active":"2024-02-21T23:06:09.953847+0000","last_became_peered":"2024-02-21T23:06:09.953847+0000","last_unstale":"2024-02-21T23:06:11.978937+0000","last_undegraded":"2024-02-21T23:06:11.978937+0000","last_fullsized":"2024-02-21T23:06:11.978937+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:31:20.996858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.036700+0000","last_change":"2024-02-21T23:05:51.416424+0000","last_active":"2024-02-21T23:06:13.036700+0000","last_peered":"2024-02-21T23:06:13.036700+0000","last_clean":"2024-02-21T23:06:13.036700+0000","last_became_active":"2024-02-21T23:05:51.416070+0000","last_became_peered":"2024-02-21T23:05:51.416070+0000","last_unstale":"2024-02-21T23:06:13.036700+0000","last_undegraded":"2024-02-21T23:06:13.036700+0000","last_fullsized":"2024-02-21T23:06:13.036700+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:56:10.406772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"75'8","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.029897+0000","last_change":"2024-02-21T23:05:50.376063+0000","last_active":"2024-02-21T23:06:13.029897+0000","last_peered":"2024-02-21T23:06:13.029897+0000","last_clean":"2024-02-21T23:06:13.029897+0000","last_became_active":"2024-02-21T23:05:50.375772+0000","last_became_peered":"2024-02-21T23:05:50.375772+0000","last_unstale":"2024-02-21T23:06:13.029897+0000","last_undegraded":"2024-02-21T23:06:13.029897+0000","last_fullsized":"2024-02-21T23:06:13.029897+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:26:58.076002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.959287+0000","last_change":"2024-02-21T23:06:10.991210+0000","last_active":"2024-02-21T23:06:12.959287+0000","last_peered":"2024-02-21T23:06:12.959287+0000","last_clean":"2024-02-21T23:06:12.959287+0000","last_became_active":"2024-02-21T23:06:10.990875+0000","last_became_peered":"2024-02-21T23:06:10.990875+0000","last_unstale":"2024-02-21T23:06:12.959287+0000","last_undegraded":"2024-02-21T23:06:12.959287+0000","last_fullsized":"2024-02-21T23:06:12.959287+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:17:15.132571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.958618+0000","last_change":"2024-02-21T23:06:09.951762+0000","last_active":"2024-02-21T23:06:12.958618+0000","last_peered":"2024-02-21T23:06:12.958618+0000","last_clean":"2024-02-21T23:06:12.958618+0000","last_became_active":"2024-02-21T23:06:09.951155+0000","last_became_peered":"2024-02-21T23:06:09.951155+0000","last_unstale":"2024-02-21T23:06:12.958618+0000","last_undegraded":"2024-02-21T23:06:12.958618+0000","last_fullsized":"2024-02-21T23:06:12.958618+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:41:04.005940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.976510+0000","last_change":"2024-02-21T23:05:51.421565+0000","last_active":"2024-02-21T23:06:11.976510+0000","last_peered":"2024-02-21T23:06:11.976510+0000","last_clean":"2024-02-21T23:06:11.976510+0000","last_became_active":"2024-02-21T23:05:51.421252+0000","last_became_peered":"2024-02-21T23:05:51.421252+0000","last_unstale":"2024-02-21T23:06:11.976510+0000","last_undegraded":"2024-02-21T23:06:11.976510+0000","last_fullsized":"2024-02-21T23:06:11.976510+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:40:45.445418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"75'1","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.036242+0000","last_change":"2024-02-21T23:05:50.383778+0000","last_active":"2024-02-21T23:06:13.036242+0000","last_peered":"2024-02-21T23:06:13.036242+0000","last_clean":"2024-02-21T23:06:13.036242+0000","last_became_active":"2024-02-21T23:05:50.383436+0000","last_became_peered":"2024-02-21T23:05:50.383436+0000","last_unstale":"2024-02-21T23:06:13.036242+0000","last_undegraded":"2024-02-21T23:06:13.036242+0000","last_fullsized":"2024-02-21T23:06:13.036242+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:45:05.837402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.030019+0000","last_change":"2024-02-21T23:06:10.965781+0000","last_active":"2024-02-21T23:06:13.030019+0000","last_peered":"2024-02-21T23:06:13.030019+0000","last_clean":"2024-02-21T23:06:13.030019+0000","last_became_active":"2024-02-21T23:06:10.965478+0000","last_became_peered":"2024-02-21T23:06:10.965478+0000","last_unstale":"2024-02-21T23:06:13.030019+0000","last_undegraded":"2024-02-21T23:06:13.030019+0000","last_fullsized":"2024-02-21T23:06:13.030019+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:34:34.740480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.535810+0000","last_change":"2024-02-21T23:06:09.947681+0000","last_active":"2024-02-21T23:06:12.535810+0000","last_peered":"2024-02-21T23:06:12.535810+0000","last_clean":"2024-02-21T23:06:12.535810+0000","last_became_active":"2024-02-21T23:06:09.947370+0000","last_became_peered":"2024-02-21T23:06:09.947370+0000","last_unstale":"2024-02-21T23:06:12.535810+0000","last_undegraded":"2024-02-21T23:06:12.535810+0000","last_fullsized":"2024-02-21T23:06:12.535810+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:16:51.794073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.535873+0000","last_change":"2024-02-21T23:05:51.416676+0000","last_active":"2024-02-21T23:06:12.535873+0000","last_peered":"2024-02-21T23:06:12.535873+0000","last_clean":"2024-02-21T23:06:12.535873+0000","last_became_active":"2024-02-21T23:05:51.416297+0000","last_became_peered":"2024-02-21T23:05:51.416297+0000","last_unstale":"2024-02-21T23:06:12.535873+0000","last_undegraded":"2024-02-21T23:06:12.535873+0000","last_fullsized":"2024-02-21T23:06:12.535873+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:29:33.209545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"75'2","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.979286+0000","last_change":"2024-02-21T23:05:50.377181+0000","last_active":"2024-02-21T23:06:11.979286+0000","last_peered":"2024-02-21T23:06:11.979286+0000","last_clean":"2024-02-21T23:06:11.979286+0000","last_became_active":"2024-02-21T23:05:50.373524+0000","last_became_peered":"2024-02-21T23:05:50.373524+0000","last_unstale":"2024-02-21T23:06:11.979286+0000","last_undegraded":"2024-02-21T23:06:11.979286+0000","last_fullsized":"2024-02-21T23:06:11.979286+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:01:18.419728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.029437+0000","last_change":"2024-02-21T23:06:10.991698+0000","last_active":"2024-02-21T23:06:13.029437+0000","last_peered":"2024-02-21T23:06:13.029437+0000","last_clean":"2024-02-21T23:06:13.029437+0000","last_became_active":"2024-02-21T23:06:10.991386+0000","last_became_peered":"2024-02-21T23:06:10.991386+0000","last_unstale":"2024-02-21T23:06:13.029437+0000","last_undegraded":"2024-02-21T23:06:13.029437+0000","last_fullsized":"2024-02-21T23:06:13.029437+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:41:50.244656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.958876+0000","last_change":"2024-02-21T23:06:09.955696+0000","last_active":"2024-02-21T23:06:12.958876+0000","last_peered":"2024-02-21T23:06:12.958876+0000","last_clean":"2024-02-21T23:06:12.958876+0000","last_became_active":"2024-02-21T23:06:09.954715+0000","last_became_peered":"2024-02-21T23:06:09.954715+0000","last_unstale":"2024-02-21T23:06:12.958876+0000","last_undegraded":"2024-02-21T23:06:12.958876+0000","last_fullsized":"2024-02-21T23:06:12.958876+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:04:21.645034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.536043+0000","last_change":"2024-02-21T23:05:51.412971+0000","last_active":"2024-02-21T23:06:12.536043+0000","last_peered":"2024-02-21T23:06:12.536043+0000","last_clean":"2024-02-21T23:06:12.536043+0000","last_became_active":"2024-02-21T23:05:51.410966+0000","last_became_peered":"2024-02-21T23:05:51.410966+0000","last_unstale":"2024-02-21T23:06:12.536043+0000","last_undegraded":"2024-02-21T23:06:12.536043+0000","last_fullsized":"2024-02-21T23:06:12.536043+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:00:39.163092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"75'1","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.536805+0000","last_change":"2024-02-21T23:05:50.366307+0000","last_active":"2024-02-21T23:06:12.536805+0000","last_peered":"2024-02-21T23:06:12.536805+0000","last_clean":"2024-02-21T23:06:12.536805+0000","last_became_active":"2024-02-21T23:05:50.359426+0000","last_became_peered":"2024-02-21T23:05:50.359426+0000","last_unstale":"2024-02-21T23:06:12.536805+0000","last_undegraded":"2024-02-21T23:06:12.536805+0000","last_fullsized":"2024-02-21T23:06:12.536805+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:13:15.770526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.958835+0000","last_change":"2024-02-21T23:06:10.969120+0000","last_active":"2024-02-21T23:06:12.958835+0000","last_peered":"2024-02-21T23:06:12.958835+0000","last_clean":"2024-02-21T23:06:12.958835+0000","last_became_active":"2024-02-21T23:06:10.968763+0000","last_became_peered":"2024-02-21T23:06:10.968763+0000","last_unstale":"2024-02-21T23:06:12.958835+0000","last_undegraded":"2024-02-21T23:06:12.958835+0000","last_fullsized":"2024-02-21T23:06:12.958835+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:00:42.549792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.535873+0000","last_change":"2024-02-21T23:06:09.948670+0000","last_active":"2024-02-21T23:06:12.535873+0000","last_peered":"2024-02-21T23:06:12.535873+0000","last_clean":"2024-02-21T23:06:12.535873+0000","last_became_active":"2024-02-21T23:06:09.946109+0000","last_became_peered":"2024-02-21T23:06:09.946109+0000","last_unstale":"2024-02-21T23:06:12.535873+0000","last_undegraded":"2024-02-21T23:06:12.535873+0000","last_fullsized":"2024-02-21T23:06:12.535873+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:16:18.547191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.976513+0000","last_change":"2024-02-21T23:05:51.424067+0000","last_active":"2024-02-21T23:06:11.976513+0000","last_peered":"2024-02-21T23:06:11.976513+0000","last_clean":"2024-02-21T23:06:11.976513+0000","last_became_active":"2024-02-21T23:05:51.423644+0000","last_became_peered":"2024-02-21T23:05:51.423644+0000","last_unstale":"2024-02-21T23:06:11.976513+0000","last_undegraded":"2024-02-21T23:06:11.976513+0000","last_fullsized":"2024-02-21T23:06:11.976513+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:04:03.084512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.972107+0000","last_change":"2024-02-21T23:05:50.377924+0000","last_active":"2024-02-21T23:06:11.972107+0000","last_peered":"2024-02-21T23:06:11.972107+0000","last_clean":"2024-02-21T23:06:11.972107+0000","last_became_active":"2024-02-21T23:05:50.377440+0000","last_became_peered":"2024-02-21T23:05:50.377440+0000","last_unstale":"2024-02-21T23:06:11.972107+0000","last_undegraded":"2024-02-21T23:06:11.972107+0000","last_fullsized":"2024-02-21T23:06:11.972107+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:42:15.626763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.979046+0000","last_change":"2024-02-21T23:06:10.981203+0000","last_active":"2024-02-21T23:06:11.979046+0000","last_peered":"2024-02-21T23:06:11.979046+0000","last_clean":"2024-02-21T23:06:11.979046+0000","last_became_active":"2024-02-21T23:06:10.980886+0000","last_became_peered":"2024-02-21T23:06:10.980886+0000","last_unstale":"2024-02-21T23:06:11.979046+0000","last_undegraded":"2024-02-21T23:06:11.979046+0000","last_fullsized":"2024-02-21T23:06:11.979046+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:04:08.953344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.d","version":"79'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.978625+0000","last_change":"2024-02-21T23:06:09.953285+0000","last_active":"2024-02-21T23:06:11.978625+0000","last_peered":"2024-02-21T23:06:11.978625+0000","last_clean":"2024-02-21T23:06:11.978625+0000","last_became_active":"2024-02-21T23:06:09.952940+0000","last_became_peered":"2024-02-21T23:06:09.952940+0000","last_unstale":"2024-02-21T23:06:11.978625+0000","last_undegraded":"2024-02-21T23:06:11.978625+0000","last_fullsized":"2024-02-21T23:06:11.978625+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:53:46.723561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.030184+0000","last_change":"2024-02-21T23:05:51.414619+0000","last_active":"2024-02-21T23:06:13.030184+0000","last_peered":"2024-02-21T23:06:13.030184+0000","last_clean":"2024-02-21T23:06:13.030184+0000","last_became_active":"2024-02-21T23:05:51.414059+0000","last_became_peered":"2024-02-21T23:05:51.414059+0000","last_unstale":"2024-02-21T23:06:13.030184+0000","last_undegraded":"2024-02-21T23:06:13.030184+0000","last_fullsized":"2024-02-21T23:06:13.030184+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:58:05.114297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"75'2","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.976661+0000","last_change":"2024-02-21T23:05:50.366877+0000","last_active":"2024-02-21T23:06:11.976661+0000","last_peered":"2024-02-21T23:06:11.976661+0000","last_clean":"2024-02-21T23:06:11.976661+0000","last_became_active":"2024-02-21T23:05:50.365566+0000","last_became_peered":"2024-02-21T23:05:50.365566+0000","last_unstale":"2024-02-21T23:06:11.976661+0000","last_undegraded":"2024-02-21T23:06:11.976661+0000","last_fullsized":"2024-02-21T23:06:11.976661+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:58:04.101475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.030006+0000","last_change":"2024-02-21T23:06:10.987713+0000","last_active":"2024-02-21T23:06:13.030006+0000","last_peered":"2024-02-21T23:06:13.030006+0000","last_clean":"2024-02-21T23:06:13.030006+0000","last_became_active":"2024-02-21T23:06:10.985949+0000","last_became_peered":"2024-02-21T23:06:10.985949+0000","last_unstale":"2024-02-21T23:06:13.030006+0000","last_undegraded":"2024-02-21T23:06:13.030006+0000","last_fullsized":"2024-02-21T23:06:13.030006+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:16:11.310383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"79'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.036400+0000","last_change":"2024-02-21T23:06:09.945701+0000","last_active":"2024-02-21T23:06:13.036400+0000","last_peered":"2024-02-21T23:06:13.036400+0000","last_clean":"2024-02-21T23:06:13.036400+0000","last_became_active":"2024-02-21T23:06:09.945391+0000","last_became_peered":"2024-02-21T23:06:09.945391+0000","last_unstale":"2024-02-21T23:06:13.036400+0000","last_undegraded":"2024-02-21T23:06:13.036400+0000","last_fullsized":"2024-02-21T23:06:13.036400+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:20:15.771087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.030743+0000","last_change":"2024-02-21T23:05:51.411005+0000","last_active":"2024-02-21T23:06:13.030743+0000","last_peered":"2024-02-21T23:06:13.030743+0000","last_clean":"2024-02-21T23:06:13.030743+0000","last_became_active":"2024-02-21T23:05:51.410503+0000","last_became_peered":"2024-02-21T23:05:51.410503+0000","last_unstale":"2024-02-21T23:06:13.030743+0000","last_undegraded":"2024-02-21T23:06:13.030743+0000","last_fullsized":"2024-02-21T23:06:13.030743+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:51:10.610852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.977826+0000","last_change":"2024-02-21T23:05:50.376888+0000","last_active":"2024-02-21T23:06:11.977826+0000","last_peered":"2024-02-21T23:06:11.977826+0000","last_clean":"2024-02-21T23:06:11.977826+0000","last_became_active":"2024-02-21T23:05:50.376610+0000","last_became_peered":"2024-02-21T23:05:50.376610+0000","last_unstale":"2024-02-21T23:06:11.977826+0000","last_undegraded":"2024-02-21T23:06:11.977826+0000","last_fullsized":"2024-02-21T23:06:11.977826+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:51:16.181060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.035166+0000","last_change":"2024-02-21T23:06:10.981890+0000","last_active":"2024-02-21T23:06:13.035166+0000","last_peered":"2024-02-21T23:06:13.035166+0000","last_clean":"2024-02-21T23:06:13.035166+0000","last_became_active":"2024-02-21T23:06:10.978458+0000","last_became_peered":"2024-02-21T23:06:10.978458+0000","last_unstale":"2024-02-21T23:06:13.035166+0000","last_undegraded":"2024-02-21T23:06:13.035166+0000","last_fullsized":"2024-02-21T23:06:13.035166+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:11:16.456657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.e","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.035094+0000","last_change":"2024-02-21T23:06:09.952555+0000","last_active":"2024-02-21T23:06:13.035094+0000","last_peered":"2024-02-21T23:06:13.035094+0000","last_clean":"2024-02-21T23:06:13.035094+0000","last_became_active":"2024-02-21T23:06:09.952281+0000","last_became_peered":"2024-02-21T23:06:09.952281+0000","last_unstale":"2024-02-21T23:06:13.035094+0000","last_undegraded":"2024-02-21T23:06:13.035094+0000","last_fullsized":"2024-02-21T23:06:13.035094+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:04:21.645034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.036839+0000","last_change":"2024-02-21T23:05:51.417167+0000","last_active":"2024-02-21T23:06:13.036839+0000","last_peered":"2024-02-21T23:06:13.036839+0000","last_clean":"2024-02-21T23:06:13.036839+0000","last_became_active":"2024-02-21T23:05:51.415957+0000","last_became_peered":"2024-02-21T23:05:51.415957+0000","last_unstale":"2024-02-21T23:06:13.036839+0000","last_undegraded":"2024-02-21T23:06:13.036839+0000","last_fullsized":"2024-02-21T23:06:13.036839+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:26:59.088824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.976802+0000","last_change":"2024-02-21T23:05:50.375808+0000","last_active":"2024-02-21T23:06:11.976802+0000","last_peered":"2024-02-21T23:06:11.976802+0000","last_clean":"2024-02-21T23:06:11.976802+0000","last_became_active":"2024-02-21T23:05:50.375467+0000","last_became_peered":"2024-02-21T23:05:50.375467+0000","last_unstale":"2024-02-21T23:06:11.976802+0000","last_undegraded":"2024-02-21T23:06:11.976802+0000","last_fullsized":"2024-02-21T23:06:11.976802+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:29:01.979463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.032733+0000","last_change":"2024-02-21T23:06:10.991982+0000","last_active":"2024-02-21T23:06:13.032733+0000","last_peered":"2024-02-21T23:06:13.032733+0000","last_clean":"2024-02-21T23:06:13.032733+0000","last_became_active":"2024-02-21T23:06:10.991622+0000","last_became_peered":"2024-02-21T23:06:10.991622+0000","last_unstale":"2024-02-21T23:06:13.032733+0000","last_undegraded":"2024-02-21T23:06:13.032733+0000","last_fullsized":"2024-02-21T23:06:13.032733+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:51:30.184287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.032396+0000","last_change":"2024-02-21T23:06:09.956377+0000","last_active":"2024-02-21T23:06:13.032396+0000","last_peered":"2024-02-21T23:06:13.032396+0000","last_clean":"2024-02-21T23:06:13.032396+0000","last_became_active":"2024-02-21T23:06:09.956129+0000","last_became_peered":"2024-02-21T23:06:09.956129+0000","last_unstale":"2024-02-21T23:06:13.032396+0000","last_undegraded":"2024-02-21T23:06:13.032396+0000","last_fullsized":"2024-02-21T23:06:13.032396+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:28:13.256243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.975684+0000","last_change":"2024-02-21T23:05:51.414453+0000","last_active":"2024-02-21T23:06:11.975684+0000","last_peered":"2024-02-21T23:06:11.975684+0000","last_clean":"2024-02-21T23:06:11.975684+0000","last_became_active":"2024-02-21T23:05:51.413499+0000","last_became_peered":"2024-02-21T23:05:51.413499+0000","last_unstale":"2024-02-21T23:06:11.975684+0000","last_undegraded":"2024-02-21T23:06:11.975684+0000","last_fullsized":"2024-02-21T23:06:11.975684+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:55:21.967274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"75'2","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.960670+0000","last_change":"2024-02-21T23:05:50.381948+0000","last_active":"2024-02-21T23:06:12.960670+0000","last_peered":"2024-02-21T23:06:12.960670+0000","last_clean":"2024-02-21T23:06:12.960670+0000","last_became_active":"2024-02-21T23:05:50.381452+0000","last_became_peered":"2024-02-21T23:05:50.381452+0000","last_unstale":"2024-02-21T23:06:12.960670+0000","last_undegraded":"2024-02-21T23:06:12.960670+0000","last_fullsized":"2024-02-21T23:06:12.960670+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:22:16.414686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.029844+0000","last_change":"2024-02-21T23:06:10.967217+0000","last_active":"2024-02-21T23:06:13.029844+0000","last_peered":"2024-02-21T23:06:13.029844+0000","last_clean":"2024-02-21T23:06:13.029844+0000","last_became_active":"2024-02-21T23:06:10.966901+0000","last_became_peered":"2024-02-21T23:06:10.966901+0000","last_unstale":"2024-02-21T23:06:13.029844+0000","last_undegraded":"2024-02-21T23:06:13.029844+0000","last_fullsized":"2024-02-21T23:06:13.029844+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:36:21.989353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.033134+0000","last_change":"2024-02-21T23:06:09.953924+0000","last_active":"2024-02-21T23:06:13.033134+0000","last_peered":"2024-02-21T23:06:13.033134+0000","last_clean":"2024-02-21T23:06:13.033134+0000","last_became_active":"2024-02-21T23:06:09.952952+0000","last_became_peered":"2024-02-21T23:06:09.952952+0000","last_unstale":"2024-02-21T23:06:13.033134+0000","last_undegraded":"2024-02-21T23:06:13.033134+0000","last_fullsized":"2024-02-21T23:06:13.033134+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:03:53.804470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.033406+0000","last_change":"2024-02-21T23:05:51.410619+0000","last_active":"2024-02-21T23:06:13.033406+0000","last_peered":"2024-02-21T23:06:13.033406+0000","last_clean":"2024-02-21T23:06:13.033406+0000","last_became_active":"2024-02-21T23:05:51.409937+0000","last_became_peered":"2024-02-21T23:05:51.409937+0000","last_unstale":"2024-02-21T23:06:13.033406+0000","last_undegraded":"2024-02-21T23:06:13.033406+0000","last_fullsized":"2024-02-21T23:06:13.033406+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:22:32.845173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"76'1","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.030826+0000","last_change":"2024-02-21T23:05:50.362370+0000","last_active":"2024-02-21T23:06:13.030826+0000","last_peered":"2024-02-21T23:06:13.030826+0000","last_clean":"2024-02-21T23:06:13.030826+0000","last_became_active":"2024-02-21T23:05:50.361848+0000","last_became_peered":"2024-02-21T23:05:50.361848+0000","last_unstale":"2024-02-21T23:06:13.030826+0000","last_undegraded":"2024-02-21T23:06:13.030826+0000","last_fullsized":"2024-02-21T23:06:13.030826+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:08:35.448428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.535691+0000","last_change":"2024-02-21T23:06:10.974560+0000","last_active":"2024-02-21T23:06:12.535691+0000","last_peered":"2024-02-21T23:06:12.535691+0000","last_clean":"2024-02-21T23:06:12.535691+0000","last_became_active":"2024-02-21T23:06:10.974224+0000","last_became_peered":"2024-02-21T23:06:10.974224+0000","last_unstale":"2024-02-21T23:06:12.535691+0000","last_undegraded":"2024-02-21T23:06:12.535691+0000","last_fullsized":"2024-02-21T23:06:12.535691+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:55:41.540709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.029834+0000","last_change":"2024-02-21T23:06:09.961946+0000","last_active":"2024-02-21T23:06:13.029834+0000","last_peered":"2024-02-21T23:06:13.029834+0000","last_clean":"2024-02-21T23:06:13.029834+0000","last_became_active":"2024-02-21T23:06:09.961662+0000","last_became_peered":"2024-02-21T23:06:09.961662+0000","last_unstale":"2024-02-21T23:06:13.029834+0000","last_undegraded":"2024-02-21T23:06:13.029834+0000","last_fullsized":"2024-02-21T23:06:13.029834+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:54:07.357455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.977217+0000","last_change":"2024-02-21T23:05:51.421357+0000","last_active":"2024-02-21T23:06:11.977217+0000","last_peered":"2024-02-21T23:06:11.977217+0000","last_clean":"2024-02-21T23:06:11.977217+0000","last_became_active":"2024-02-21T23:05:51.420734+0000","last_became_peered":"2024-02-21T23:05:51.420734+0000","last_unstale":"2024-02-21T23:06:11.977217+0000","last_undegraded":"2024-02-21T23:06:11.977217+0000","last_fullsized":"2024-02-21T23:06:11.977217+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:31:44.536584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.035689+0000","last_change":"2024-02-21T23:05:50.384680+0000","last_active":"2024-02-21T23:06:13.035689+0000","last_peered":"2024-02-21T23:06:13.035689+0000","last_clean":"2024-02-21T23:06:13.035689+0000","last_became_active":"2024-02-21T23:05:50.384208+0000","last_became_peered":"2024-02-21T23:05:50.384208+0000","last_unstale":"2024-02-21T23:06:13.035689+0000","last_undegraded":"2024-02-21T23:06:13.035689+0000","last_fullsized":"2024-02-21T23:06:13.035689+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:01:27.369089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.977561+0000","last_change":"2024-02-21T23:06:10.963711+0000","last_active":"2024-02-21T23:06:11.977561+0000","last_peered":"2024-02-21T23:06:11.977561+0000","last_clean":"2024-02-21T23:06:11.977561+0000","last_became_active":"2024-02-21T23:06:10.963131+0000","last_became_peered":"2024-02-21T23:06:10.963131+0000","last_unstale":"2024-02-21T23:06:11.977561+0000","last_undegraded":"2024-02-21T23:06:11.977561+0000","last_fullsized":"2024-02-21T23:06:11.977561+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:35:48.816255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.958788+0000","last_change":"2024-02-21T23:06:09.952597+0000","last_active":"2024-02-21T23:06:12.958788+0000","last_peered":"2024-02-21T23:06:12.958788+0000","last_clean":"2024-02-21T23:06:12.958788+0000","last_became_active":"2024-02-21T23:06:09.952270+0000","last_became_peered":"2024-02-21T23:06:09.952270+0000","last_unstale":"2024-02-21T23:06:12.958788+0000","last_undegraded":"2024-02-21T23:06:12.958788+0000","last_fullsized":"2024-02-21T23:06:12.958788+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:03:53.804470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.030138+0000","last_change":"2024-02-21T23:05:51.395859+0000","last_active":"2024-02-21T23:06:13.030138+0000","last_peered":"2024-02-21T23:06:13.030138+0000","last_clean":"2024-02-21T23:06:13.030138+0000","last_became_active":"2024-02-21T23:05:51.395012+0000","last_became_peered":"2024-02-21T23:05:51.395012+0000","last_unstale":"2024-02-21T23:06:13.030138+0000","last_undegraded":"2024-02-21T23:06:13.030138+0000","last_fullsized":"2024-02-21T23:06:13.030138+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:29:02.992284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.033782+0000","last_change":"2024-02-21T23:05:50.383920+0000","last_active":"2024-02-21T23:06:13.033782+0000","last_peered":"2024-02-21T23:06:13.033782+0000","last_clean":"2024-02-21T23:06:13.033782+0000","last_became_active":"2024-02-21T23:05:50.382488+0000","last_became_peered":"2024-02-21T23:05:50.382488+0000","last_unstale":"2024-02-21T23:06:13.033782+0000","last_undegraded":"2024-02-21T23:06:13.033782+0000","last_fullsized":"2024-02-21T23:06:13.033782+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:26:58.076002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.030523+0000","last_change":"2024-02-21T23:06:10.963734+0000","last_active":"2024-02-21T23:06:13.030523+0000","last_peered":"2024-02-21T23:06:13.030523+0000","last_clean":"2024-02-21T23:06:13.030523+0000","last_became_active":"2024-02-21T23:06:10.963441+0000","last_became_peered":"2024-02-21T23:06:10.963441+0000","last_unstale":"2024-02-21T23:06:13.030523+0000","last_undegraded":"2024-02-21T23:06:13.030523+0000","last_fullsized":"2024-02-21T23:06:13.030523+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:51:44.907006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.978786+0000","last_change":"2024-02-21T23:06:09.961904+0000","last_active":"2024-02-21T23:06:11.978786+0000","last_peered":"2024-02-21T23:06:11.978786+0000","last_clean":"2024-02-21T23:06:11.978786+0000","last_became_active":"2024-02-21T23:06:09.961650+0000","last_became_peered":"2024-02-21T23:06:09.961650+0000","last_unstale":"2024-02-21T23:06:11.978786+0000","last_undegraded":"2024-02-21T23:06:11.978786+0000","last_fullsized":"2024-02-21T23:06:11.978786+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:56:08.310911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.035568+0000","last_change":"2024-02-21T23:05:51.422495+0000","last_active":"2024-02-21T23:06:13.035568+0000","last_peered":"2024-02-21T23:06:13.035568+0000","last_clean":"2024-02-21T23:06:13.035568+0000","last_became_active":"2024-02-21T23:05:51.422196+0000","last_became_peered":"2024-02-21T23:05:51.422196+0000","last_unstale":"2024-02-21T23:06:13.035568+0000","last_undegraded":"2024-02-21T23:06:13.035568+0000","last_fullsized":"2024-02-21T23:06:13.035568+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:10:11.986389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.034279+0000","last_change":"2024-02-21T23:05:50.376027+0000","last_active":"2024-02-21T23:06:13.034279+0000","last_peered":"2024-02-21T23:06:13.034279+0000","last_clean":"2024-02-21T23:06:13.034279+0000","last_became_active":"2024-02-21T23:05:50.375170+0000","last_became_peered":"2024-02-21T23:05:50.375170+0000","last_unstale":"2024-02-21T23:06:13.034279+0000","last_undegraded":"2024-02-21T23:06:13.034279+0000","last_fullsized":"2024-02-21T23:06:13.034279+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:39:32.228635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.960474+0000","last_change":"2024-02-21T23:06:10.987733+0000","last_active":"2024-02-21T23:06:12.960474+0000","last_peered":"2024-02-21T23:06:12.960474+0000","last_clean":"2024-02-21T23:06:12.960474+0000","last_became_active":"2024-02-21T23:06:10.987434+0000","last_became_peered":"2024-02-21T23:06:10.987434+0000","last_unstale":"2024-02-21T23:06:12.960474+0000","last_undegraded":"2024-02-21T23:06:12.960474+0000","last_fullsized":"2024-02-21T23:06:12.960474+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:31:22.009771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.035325+0000","last_change":"2024-02-21T23:06:09.947879+0000","last_active":"2024-02-21T23:06:13.035325+0000","last_peered":"2024-02-21T23:06:13.035325+0000","last_clean":"2024-02-21T23:06:13.035325+0000","last_became_active":"2024-02-21T23:06:09.947599+0000","last_became_peered":"2024-02-21T23:06:09.947599+0000","last_unstale":"2024-02-21T23:06:13.035325+0000","last_undegraded":"2024-02-21T23:06:13.035325+0000","last_fullsized":"2024-02-21T23:06:13.035325+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:17:44.890131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.035705+0000","last_change":"2024-02-21T23:05:51.420500+0000","last_active":"2024-02-21T23:06:13.035705+0000","last_peered":"2024-02-21T23:06:13.035705+0000","last_clean":"2024-02-21T23:06:13.035705+0000","last_became_active":"2024-02-21T23:05:51.420103+0000","last_became_peered":"2024-02-21T23:05:51.420103+0000","last_unstale":"2024-02-21T23:06:13.035705+0000","last_undegraded":"2024-02-21T23:06:13.035705+0000","last_fullsized":"2024-02-21T23:06:13.035705+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:27:01.191337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.536099+0000","last_change":"2024-02-21T23:05:50.354805+0000","last_active":"2024-02-21T23:06:12.536099+0000","last_peered":"2024-02-21T23:06:12.536099+0000","last_clean":"2024-02-21T23:06:12.536099+0000","last_became_active":"2024-02-21T23:05:50.354525+0000","last_became_peered":"2024-02-21T23:05:50.354525+0000","last_unstale":"2024-02-21T23:06:12.536099+0000","last_undegraded":"2024-02-21T23:06:12.536099+0000","last_fullsized":"2024-02-21T23:06:12.536099+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:28:05.314552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.035802+0000","last_change":"2024-02-21T23:06:10.978267+0000","last_active":"2024-02-21T23:06:13.035802+0000","last_peered":"2024-02-21T23:06:13.035802+0000","last_clean":"2024-02-21T23:06:13.035802+0000","last_became_active":"2024-02-21T23:06:10.977902+0000","last_became_peered":"2024-02-21T23:06:10.977902+0000","last_unstale":"2024-02-21T23:06:13.035802+0000","last_undegraded":"2024-02-21T23:06:13.035802+0000","last_fullsized":"2024-02-21T23:06:13.035802+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:36:43.365830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.7","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.034181+0000","last_change":"2024-02-21T23:06:09.955164+0000","last_active":"2024-02-21T23:06:13.034181+0000","last_peered":"2024-02-21T23:06:13.034181+0000","last_clean":"2024-02-21T23:06:13.034181+0000","last_became_active":"2024-02-21T23:06:09.954858+0000","last_became_peered":"2024-02-21T23:06:09.954858+0000","last_unstale":"2024-02-21T23:06:13.034181+0000","last_undegraded":"2024-02-21T23:06:13.034181+0000","last_fullsized":"2024-02-21T23:06:13.034181+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:59:31.853016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.971935+0000","last_change":"2024-02-21T23:05:51.416548+0000","last_active":"2024-02-21T23:06:11.971935+0000","last_peered":"2024-02-21T23:06:11.971935+0000","last_clean":"2024-02-21T23:06:11.971935+0000","last_became_active":"2024-02-21T23:05:51.416222+0000","last_became_peered":"2024-02-21T23:05:51.416222+0000","last_unstale":"2024-02-21T23:06:11.971935+0000","last_undegraded":"2024-02-21T23:06:11.971935+0000","last_fullsized":"2024-02-21T23:06:11.971935+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:22:32.845173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.034239+0000","last_change":"2024-02-21T23:05:50.373554+0000","last_active":"2024-02-21T23:06:13.034239+0000","last_peered":"2024-02-21T23:06:13.034239+0000","last_clean":"2024-02-21T23:06:13.034239+0000","last_became_active":"2024-02-21T23:05:50.373022+0000","last_became_peered":"2024-02-21T23:05:50.373022+0000","last_unstale":"2024-02-21T23:06:13.034239+0000","last_undegraded":"2024-02-21T23:06:13.034239+0000","last_fullsized":"2024-02-21T23:06:13.034239+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:38:56.665723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.030396+0000","last_change":"2024-02-21T23:06:10.983730+0000","last_active":"2024-02-21T23:06:13.030396+0000","last_peered":"2024-02-21T23:06:13.030396+0000","last_clean":"2024-02-21T23:06:13.030396+0000","last_became_active":"2024-02-21T23:06:10.983405+0000","last_became_peered":"2024-02-21T23:06:10.983405+0000","last_unstale":"2024-02-21T23:06:13.030396+0000","last_undegraded":"2024-02-21T23:06:13.030396+0000","last_fullsized":"2024-02-21T23:06:13.030396+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:09:34.269512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"80'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.978594+0000","last_change":"2024-02-21T23:06:09.953218+0000","last_active":"2024-02-21T23:06:11.978594+0000","last_peered":"2024-02-21T23:06:11.978594+0000","last_clean":"2024-02-21T23:06:11.978594+0000","last_became_active":"2024-02-21T23:06:09.952908+0000","last_became_peered":"2024-02-21T23:06:09.952908+0000","last_unstale":"2024-02-21T23:06:11.978594+0000","last_undegraded":"2024-02-21T23:06:11.978594+0000","last_fullsized":"2024-02-21T23:06:11.978594+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:51:46.950044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.972781+0000","last_change":"2024-02-21T23:05:51.412683+0000","last_active":"2024-02-21T23:06:11.972781+0000","last_peered":"2024-02-21T23:06:11.972781+0000","last_clean":"2024-02-21T23:06:11.972781+0000","last_became_active":"2024-02-21T23:05:51.412389+0000","last_became_peered":"2024-02-21T23:05:51.412389+0000","last_unstale":"2024-02-21T23:06:11.972781+0000","last_undegraded":"2024-02-21T23:06:11.972781+0000","last_fullsized":"2024-02-21T23:06:11.972781+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:57:06.211024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.972069+0000","last_change":"2024-02-21T23:05:50.376356+0000","last_active":"2024-02-21T23:06:11.972069+0000","last_peered":"2024-02-21T23:06:11.972069+0000","last_clean":"2024-02-21T23:06:11.972069+0000","last_became_active":"2024-02-21T23:05:50.370467+0000","last_became_peered":"2024-02-21T23:05:50.370467+0000","last_unstale":"2024-02-21T23:06:11.972069+0000","last_undegraded":"2024-02-21T23:06:11.972069+0000","last_fullsized":"2024-02-21T23:06:11.972069+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:56:49.435540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.033889+0000","last_change":"2024-02-21T23:06:10.985628+0000","last_active":"2024-02-21T23:06:13.033889+0000","last_peered":"2024-02-21T23:06:13.033889+0000","last_clean":"2024-02-21T23:06:13.033889+0000","last_became_active":"2024-02-21T23:06:10.985322+0000","last_became_peered":"2024-02-21T23:06:10.985322+0000","last_unstale":"2024-02-21T23:06:13.033889+0000","last_undegraded":"2024-02-21T23:06:13.033889+0000","last_fullsized":"2024-02-21T23:06:13.033889+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:37:11.082403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.971838+0000","last_change":"2024-02-21T23:06:09.940641+0000","last_active":"2024-02-21T23:06:11.971838+0000","last_peered":"2024-02-21T23:06:11.971838+0000","last_clean":"2024-02-21T23:06:11.971838+0000","last_became_active":"2024-02-21T23:06:09.940353+0000","last_became_peered":"2024-02-21T23:06:09.940353+0000","last_unstale":"2024-02-21T23:06:11.971838+0000","last_undegraded":"2024-02-21T23:06:11.971838+0000","last_fullsized":"2024-02-21T23:06:11.971838+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:29:15.656090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.035455+0000","last_change":"2024-02-21T23:05:51.413008+0000","last_active":"2024-02-21T23:06:13.035455+0000","last_peered":"2024-02-21T23:06:13.035455+0000","last_clean":"2024-02-21T23:06:13.035455+0000","last_became_active":"2024-02-21T23:05:51.412714+0000","last_became_peered":"2024-02-21T23:05:51.412714+0000","last_unstale":"2024-02-21T23:06:13.035455+0000","last_undegraded":"2024-02-21T23:06:13.035455+0000","last_fullsized":"2024-02-21T23:06:13.035455+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:03:08.789066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.976840+0000","last_change":"2024-02-21T23:05:50.375581+0000","last_active":"2024-02-21T23:06:11.976840+0000","last_peered":"2024-02-21T23:06:11.976840+0000","last_clean":"2024-02-21T23:06:11.976840+0000","last_became_active":"2024-02-21T23:05:50.375282+0000","last_became_peered":"2024-02-21T23:05:50.375282+0000","last_unstale":"2024-02-21T23:06:11.976840+0000","last_undegraded":"2024-02-21T23:06:11.976840+0000","last_fullsized":"2024-02-21T23:06:11.976840+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:49:45.012428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":282,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.034669+0000","last_change":"2024-02-21T23:02:32.289335+0000","last_active":"2024-02-21T23:06:13.034669+0000","last_peered":"2024-02-21T23:06:13.034669+0000","last_clean":"2024-02-21T23:06:13.034669+0000","last_became_active":"2024-02-21T23:02:32.286913+0000","last_became_peered":"2024-02-21T23:02:32.286913+0000","last_unstale":"2024-02-21T23:06:13.034669+0000","last_undegraded":"2024-02-21T23:06:13.034669+0000","last_fullsized":"2024-02-21T23:06:13.034669+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_clean_scrub_stamp":"2024-02-21T23:02:31.172569+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:36:23.987374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":14,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574895,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32212,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339628,"statfs":{"total":96636764160,"available":96603779072,"internally_reserved":0,"allocated":1056768,"data_stored":1064123,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574896,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32168,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339672,"statfs":{"total":96636764160,"available":96603824128,"internally_reserved":0,"allocated":1011712,"data_stored":477505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574895,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27984,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343856,"statfs":{"total":96636764160,"available":96608108544,"internally_reserved":0,"allocated":983040,"data_stored":473711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574896,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32168,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339672,"statfs":{"total":96636764160,"available":96603824128,"internally_reserved":0,"allocated":1007616,"data_stored":475553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574896,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32144,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339696,"statfs":{"total":96636764160,"available":96603848704,"internally_reserved":0,"allocated":983040,"data_stored":474259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574897,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32152,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339688,"statfs":{"total":96636764160,"available":96603840512,"internally_reserved":0,"allocated":991232,"data_stored":474419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574895,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32152,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339688,"statfs":{"total":96636764160,"available":96603840512,"internally_reserved":0,"allocated":991232,"data_stored":473859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574895,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32224,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339616,"statfs":{"total":96636764160,"available":96603766784,"internally_reserved":0,"allocated":1069056,"data_stored":1066363,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-21T23:06:20.843 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-21T23:06:21.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.077+0000 7f56dae00640 1 Processor -- start 2024-02-21T23:06:21.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.078+0000 7f56dae00640 1 -- start start 2024-02-21T23:06:21.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.079+0000 7f56dae00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f56d4155960 0x7f56d41608a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:21.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.079+0000 7f56dae00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56d40c74c0 con 0x7f56d40aff90 2024-02-21T23:06:21.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.079+0000 7f56dae00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56d40c77a0 con 0x7f56d406dbf0 2024-02-21T23:06:21.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.079+0000 7f56dae00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56d40c7270 con 0x7f56d4155960 2024-02-21T23:06:21.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.080+0000 7f56d9a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f56d40aff90 0x7f56d40c51e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42540/0 (socket says 172.21.15.28:42540) 2024-02-21T23:06:21.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.080+0000 7f56d9a00640 1 -- 172.21.15.28:0/410404277 learned_addr learned my addr 172.21.15.28:0/410404277 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:06:21.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.080+0000 7f56da400640 1 --2- 172.21.15.28:0/410404277 >> v2:172.21.15.28:3300/0 conn(0x7f56d4155960 0x7f56d41608a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:21.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.080+0000 7f56d3e00640 1 -- 172.21.15.28:0/410404277 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3852472578 0 0) 0x7f56d40c74c0 con 0x7f56d40aff90 2024-02-21T23:06:21.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.080+0000 7f56d3e00640 1 -- 172.21.15.28:0/410404277 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56bc003520 con 0x7f56d40aff90 2024-02-21T23:06:21.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.080+0000 7f56da400640 1 -- 172.21.15.28:0/410404277 >> v1:172.21.15.81:6789/0 conn(0x7f56d406dbf0 legacy=0x7f56d4158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:21.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.081+0000 7f56da400640 1 -- 172.21.15.28:0/410404277 >> v1:172.21.15.28:6789/0 conn(0x7f56d40aff90 legacy=0x7f56d40c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:21.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.081+0000 7f56da400640 1 -- 172.21.15.28:0/410404277 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56d40c7a80 con 0x7f56d4155960 2024-02-21T23:06:21.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.082+0000 7f56da400640 1 --2- 172.21.15.28:0/410404277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56d4155960 0x7f56d41608a0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f56cc009d90 tx=0x7f56cc02bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37a6612a39d3e11 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:21.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.082+0000 7f56d3e00640 1 -- 172.21.15.28:0/410404277 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56cc039070 con 0x7f56d4155960 2024-02-21T23:06:21.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.082+0000 7f56d3e00640 1 -- 172.21.15.28:0/410404277 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56cc02cd00 con 0x7f56d4155960 2024-02-21T23:06:21.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.082+0000 7f56d3e00640 1 -- 172.21.15.28:0/410404277 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56cc034640 con 0x7f56d4155960 2024-02-21T23:06:21.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.083+0000 7f56dae00640 1 -- 172.21.15.28:0/410404277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56d4155960 msgr2=0x7f56d41608a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:21.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.083+0000 7f56dae00640 1 --2- 172.21.15.28:0/410404277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56d4155960 0x7f56d41608a0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f56cc009d90 tx=0x7f56cc02bf50 comp rx=0 tx=0).stop 2024-02-21T23:06:21.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.083+0000 7f56dae00640 1 -- 172.21.15.28:0/410404277 shutdown_connections 2024-02-21T23:06:21.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.083+0000 7f56dae00640 1 --2- 172.21.15.28:0/410404277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56d4155960 0x7f56d41608a0 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:21.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.083+0000 7f56dae00640 1 -- 172.21.15.28:0/410404277 >> 172.21.15.28:0/410404277 conn(0x7f56d40ac370 msgr2=0x7f56d40ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:21.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.083+0000 7f56dae00640 1 -- 172.21.15.28:0/410404277 shutdown_connections 2024-02-21T23:06:21.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.083+0000 7f56dae00640 1 -- 172.21.15.28:0/410404277 wait complete. 2024-02-21T23:06:21.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.085+0000 7f56dae00640 1 Processor -- start 2024-02-21T23:06:21.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.085+0000 7f56dae00640 1 -- start start 2024-02-21T23:06:21.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.086+0000 7f56dae00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56d406dbf0 0x7f56d40bfee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:21.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.086+0000 7f56dae00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f56d40aff90 0x7f56d40c3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:21.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.086+0000 7f56dae00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56d4155960 0x7f56d40c2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:21.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.086+0000 7f56dae00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56d40c58f0 con 0x7f56d4155960 2024-02-21T23:06:21.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.086+0000 7f56dae00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56d4158630 con 0x7f56d406dbf0 2024-02-21T23:06:21.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.086+0000 7f56dae00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56d40c5a70 con 0x7f56d40aff90 2024-02-21T23:06:21.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.086+0000 7f56da400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56d4155960 0x7f56d40c2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:21.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.086+0000 7f56da400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56d4155960 0x7f56d40c2380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54686/0 (socket says 172.21.15.28:54686) 2024-02-21T23:06:21.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.086+0000 7f56da400640 1 -- 172.21.15.28:0/327183671 learned_addr learned my addr 172.21.15.28:0/327183671 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:21.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.086+0000 7f56d9a00640 1 --2- 172.21.15.28:0/327183671 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56d406dbf0 0x7f56d40bfee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:21.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.087+0000 7f56d9000640 1 --2- 172.21.15.28:0/327183671 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f56d40aff90 0x7f56d40c3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:21.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.087+0000 7f56da400640 1 -- 172.21.15.28:0/327183671 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f56d40aff90 msgr2=0x7f56d40c3cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:21.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.087+0000 7f56da400640 1 --2- 172.21.15.28:0/327183671 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f56d40aff90 0x7f56d40c3cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:21.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.087+0000 7f56da400640 1 -- 172.21.15.28:0/327183671 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56d406dbf0 msgr2=0x7f56d40bfee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:21.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.087+0000 7f56da400640 1 --2- 172.21.15.28:0/327183671 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56d406dbf0 0x7f56d40bfee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:21.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.087+0000 7f56da400640 1 -- 172.21.15.28:0/327183671 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56d41eb830 con 0x7f56d4155960 2024-02-21T23:06:21.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.087+0000 7f56da400640 1 --2- 172.21.15.28:0/327183671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56d4155960 0x7f56d40c2380 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f56cc009d90 tx=0x7f56cc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:21.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.087+0000 7f56d1600640 1 -- 172.21.15.28:0/327183671 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56cc044020 con 0x7f56d4155960 2024-02-21T23:06:21.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.087+0000 7f56d1600640 1 -- 172.21.15.28:0/327183671 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56cc03a040 con 0x7f56d4155960 2024-02-21T23:06:21.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.088+0000 7f56d1600640 1 -- 172.21.15.28:0/327183671 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56cc039050 con 0x7f56d4155960 2024-02-21T23:06:21.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.088+0000 7f56dae00640 1 -- 172.21.15.28:0/327183671 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56d41ebad0 con 0x7f56d4155960 2024-02-21T23:06:21.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.088+0000 7f56dae00640 1 -- 172.21.15.28:0/327183671 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56d40ab130 con 0x7f56d4155960 2024-02-21T23:06:21.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.089+0000 7f56dae00640 1 -- 172.21.15.28:0/327183671 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56d404f290 con 0x7f56d4155960 2024-02-21T23:06:21.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.090+0000 7f56d1600640 1 -- 172.21.15.28:0/327183671 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f56cc003bd0 con 0x7f56d4155960 2024-02-21T23:06:21.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.092+0000 7f56d1600640 1 --2- 172.21.15.28:0/327183671 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f56b406b4d0 0x7f56b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:21.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.092+0000 7f56d1600640 1 -- 172.21.15.28:0/327183671 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f56cc0ac9c0 con 0x7f56d4155960 2024-02-21T23:06:21.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.092+0000 7f56d9a00640 1 --2- 172.21.15.28:0/327183671 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f56b406b4d0 0x7f56b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:21.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.094+0000 7f56d9a00640 1 --2- 172.21.15.28:0/327183671 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f56b406b4d0 0x7f56b406d990 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f56c80011e0 tx=0x7f56c8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:21.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.100+0000 7f56d1600640 1 -- 172.21.15.28:0/327183671 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f56cc0aee20 con 0x7f56d4155960 2024-02-21T23:06:21.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.297+0000 7f56dae00640 1 -- 172.21.15.28:0/327183671 --> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f56d406d9d0 con 0x7f56b406b4d0 2024-02-21T23:06:21.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.332+0000 7f56d1600640 1 -- 172.21.15.28:0/327183671 <== mgr.4101 v2:172.21.15.81:6832/1848538577 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+490725 (secure 0 0 0) 0x7f56d406d9d0 con 0x7f56b406b4d0 2024-02-21T23:06:21.327 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:06:21.330 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-21T23:06:21.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.340+0000 7f56dae00640 1 -- 172.21.15.28:0/327183671 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f56b406b4d0 msgr2=0x7f56b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:21.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.340+0000 7f56dae00640 1 --2- 172.21.15.28:0/327183671 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f56b406b4d0 0x7f56b406d990 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f56c80011e0 tx=0x7f56c8009040 comp rx=0 tx=0).stop 2024-02-21T23:06:21.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.340+0000 7f56dae00640 1 -- 172.21.15.28:0/327183671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56d4155960 msgr2=0x7f56d40c2380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:21.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.340+0000 7f56dae00640 1 --2- 172.21.15.28:0/327183671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56d4155960 0x7f56d40c2380 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f56cc009d90 tx=0x7f56cc004a40 comp rx=0 tx=0).stop 2024-02-21T23:06:21.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.340+0000 7f56dae00640 1 -- 172.21.15.28:0/327183671 shutdown_connections 2024-02-21T23:06:21.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.340+0000 7f56dae00640 1 --2- 172.21.15.28:0/327183671 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f56b406b4d0 0x7f56b406d990 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:21.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.340+0000 7f56dae00640 1 --2- 172.21.15.28:0/327183671 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f56d40aff90 0x7f56d40c3cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:21.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.340+0000 7f56dae00640 1 --2- 172.21.15.28:0/327183671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56d4155960 0x7f56d40c2380 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:21.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.340+0000 7f56dae00640 1 --2- 172.21.15.28:0/327183671 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56d406dbf0 0x7f56d40bfee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:21.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.341+0000 7f56dae00640 1 -- 172.21.15.28:0/327183671 >> 172.21.15.28:0/327183671 conn(0x7f56d40ac370 msgr2=0x7f56d4156230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:21.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.341+0000 7f56dae00640 1 -- 172.21.15.28:0/327183671 shutdown_connections 2024-02-21T23:06:21.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.341+0000 7f56dae00640 1 -- 172.21.15.28:0/327183671 wait complete. 2024-02-21T23:06:21.351 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":188,"stamp":"2024-02-21T23:06:20.744660+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595188,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":249,"num_write_kb":2444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":195,"ondisk_log_size":195,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":386,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":253204,"kb_used_data":7904,"kb_used_omap":12,"kb_used_meta":245171,"kb_avail":754721516,"statfs":{"total":773094113280,"available":772834832384,"internally_reserved":0,"allocated":8093696,"data_stored":4979792,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":251055665},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":47,"apply_latency_ms":47,"commit_latency_ns":47000000,"apply_latency_ns":47000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.806113"},"pg_stats":[{"pgid":"14.3d","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.978205+0000","last_change":"2024-02-21T23:06:10.982364+0000","last_active":"2024-02-21T23:06:11.978205+0000","last_peered":"2024-02-21T23:06:11.978205+0000","last_clean":"2024-02-21T23:06:11.978205+0000","last_became_active":"2024-02-21T23:06:10.982112+0000","last_became_peered":"2024-02-21T23:06:10.982112+0000","last_unstale":"2024-02-21T23:06:11.978205+0000","last_undegraded":"2024-02-21T23:06:11.978205+0000","last_fullsized":"2024-02-21T23:06:11.978205+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:29:34.759653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.3f","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.035955+0000","last_change":"2024-02-21T23:05:51.408831+0000","last_active":"2024-02-21T23:06:13.035955+0000","last_peered":"2024-02-21T23:06:13.035955+0000","last_clean":"2024-02-21T23:06:13.035955+0000","last_became_active":"2024-02-21T23:05:51.408353+0000","last_became_peered":"2024-02-21T23:05:51.408353+0000","last_unstale":"2024-02-21T23:06:13.035955+0000","last_undegraded":"2024-02-21T23:06:13.035955+0000","last_fullsized":"2024-02-21T23:06:13.035955+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:36:47.962935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.3c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.971866+0000","last_change":"2024-02-21T23:06:10.987327+0000","last_active":"2024-02-21T23:06:11.971866+0000","last_peered":"2024-02-21T23:06:11.971866+0000","last_clean":"2024-02-21T23:06:11.971866+0000","last_became_active":"2024-02-21T23:06:10.981952+0000","last_became_peered":"2024-02-21T23:06:10.981952+0000","last_unstale":"2024-02-21T23:06:11.971866+0000","last_undegraded":"2024-02-21T23:06:11.971866+0000","last_fullsized":"2024-02-21T23:06:11.971866+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:36:43.365830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.030722+0000","last_change":"2024-02-21T23:05:51.420557+0000","last_active":"2024-02-21T23:06:13.030722+0000","last_peered":"2024-02-21T23:06:13.030722+0000","last_clean":"2024-02-21T23:06:13.030722+0000","last_became_active":"2024-02-21T23:05:51.420078+0000","last_became_peered":"2024-02-21T23:05:51.420078+0000","last_unstale":"2024-02-21T23:06:13.030722+0000","last_undegraded":"2024-02-21T23:06:13.030722+0000","last_fullsized":"2024-02-21T23:06:13.030722+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:36:23.792395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.3f","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.978477+0000","last_change":"2024-02-21T23:06:10.959053+0000","last_active":"2024-02-21T23:06:11.978477+0000","last_peered":"2024-02-21T23:06:11.978477+0000","last_clean":"2024-02-21T23:06:11.978477+0000","last_became_active":"2024-02-21T23:06:10.958767+0000","last_became_peered":"2024-02-21T23:06:10.958767+0000","last_unstale":"2024-02-21T23:06:11.978477+0000","last_undegraded":"2024-02-21T23:06:11.978477+0000","last_fullsized":"2024-02-21T23:06:11.978477+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:41:50.244656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.971305+0000","last_change":"2024-02-21T23:05:51.408934+0000","last_active":"2024-02-21T23:06:11.971305+0000","last_peered":"2024-02-21T23:06:11.971305+0000","last_clean":"2024-02-21T23:06:11.971305+0000","last_became_active":"2024-02-21T23:05:51.408597+0000","last_became_peered":"2024-02-21T23:05:51.408597+0000","last_unstale":"2024-02-21T23:06:11.971305+0000","last_undegraded":"2024-02-21T23:06:11.971305+0000","last_fullsized":"2024-02-21T23:06:11.971305+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:58:55.829941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.3e","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.959663+0000","last_change":"2024-02-21T23:06:10.975663+0000","last_active":"2024-02-21T23:06:12.959663+0000","last_peered":"2024-02-21T23:06:12.959663+0000","last_clean":"2024-02-21T23:06:12.959663+0000","last_became_active":"2024-02-21T23:06:10.974064+0000","last_became_peered":"2024-02-21T23:06:10.974064+0000","last_unstale":"2024-02-21T23:06:12.959663+0000","last_undegraded":"2024-02-21T23:06:12.959663+0000","last_fullsized":"2024-02-21T23:06:12.959663+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:19:08.047583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.959623+0000","last_change":"2024-02-21T23:05:51.423479+0000","last_active":"2024-02-21T23:06:12.959623+0000","last_peered":"2024-02-21T23:06:12.959623+0000","last_clean":"2024-02-21T23:06:12.959623+0000","last_became_active":"2024-02-21T23:05:51.423108+0000","last_became_peered":"2024-02-21T23:05:51.423108+0000","last_unstale":"2024-02-21T23:06:12.959623+0000","last_undegraded":"2024-02-21T23:06:12.959623+0000","last_fullsized":"2024-02-21T23:06:12.959623+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:36:47.962935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.39","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.033222+0000","last_change":"2024-02-21T23:06:10.975296+0000","last_active":"2024-02-21T23:06:13.033222+0000","last_peered":"2024-02-21T23:06:13.033222+0000","last_clean":"2024-02-21T23:06:13.033222+0000","last_became_active":"2024-02-21T23:06:10.974998+0000","last_became_peered":"2024-02-21T23:06:10.974998+0000","last_unstale":"2024-02-21T23:06:13.033222+0000","last_undegraded":"2024-02-21T23:06:13.033222+0000","last_fullsized":"2024-02-21T23:06:13.033222+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:17:15.132571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.536186+0000","last_change":"2024-02-21T23:05:51.419436+0000","last_active":"2024-02-21T23:06:12.536186+0000","last_peered":"2024-02-21T23:06:12.536186+0000","last_clean":"2024-02-21T23:06:12.536186+0000","last_became_active":"2024-02-21T23:05:51.418562+0000","last_became_peered":"2024-02-21T23:05:51.418562+0000","last_unstale":"2024-02-21T23:06:12.536186+0000","last_undegraded":"2024-02-21T23:06:12.536186+0000","last_fullsized":"2024-02-21T23:06:12.536186+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:19:45.189126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.38","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.972213+0000","last_change":"2024-02-21T23:06:10.991824+0000","last_active":"2024-02-21T23:06:11.972213+0000","last_peered":"2024-02-21T23:06:11.972213+0000","last_clean":"2024-02-21T23:06:11.972213+0000","last_became_active":"2024-02-21T23:06:10.990580+0000","last_became_peered":"2024-02-21T23:06:10.990580+0000","last_unstale":"2024-02-21T23:06:11.972213+0000","last_undegraded":"2024-02-21T23:06:11.972213+0000","last_fullsized":"2024-02-21T23:06:11.972213+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:01:36.530063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.976468+0000","last_change":"2024-02-21T23:05:51.410014+0000","last_active":"2024-02-21T23:06:11.976468+0000","last_peered":"2024-02-21T23:06:11.976468+0000","last_clean":"2024-02-21T23:06:11.976468+0000","last_became_active":"2024-02-21T23:05:51.409637+0000","last_became_peered":"2024-02-21T23:05:51.409637+0000","last_unstale":"2024-02-21T23:06:11.976468+0000","last_undegraded":"2024-02-21T23:06:11.976468+0000","last_fullsized":"2024-02-21T23:06:11.976468+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:03:35.243949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3b","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.976370+0000","last_change":"2024-02-21T23:06:10.980590+0000","last_active":"2024-02-21T23:06:11.976370+0000","last_peered":"2024-02-21T23:06:11.976370+0000","last_clean":"2024-02-21T23:06:11.976370+0000","last_became_active":"2024-02-21T23:06:10.978843+0000","last_became_peered":"2024-02-21T23:06:10.978843+0000","last_unstale":"2024-02-21T23:06:11.976370+0000","last_undegraded":"2024-02-21T23:06:11.976370+0000","last_fullsized":"2024-02-21T23:06:11.976370+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:11:16.456657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.032749+0000","last_change":"2024-02-21T23:05:51.419505+0000","last_active":"2024-02-21T23:06:13.032749+0000","last_peered":"2024-02-21T23:06:13.032749+0000","last_clean":"2024-02-21T23:06:13.032749+0000","last_became_active":"2024-02-21T23:05:51.418040+0000","last_became_peered":"2024-02-21T23:05:51.418040+0000","last_unstale":"2024-02-21T23:06:13.032749+0000","last_undegraded":"2024-02-21T23:06:13.032749+0000","last_fullsized":"2024-02-21T23:06:13.032749+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:57:06.211024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.3a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.029781+0000","last_change":"2024-02-21T23:06:10.975464+0000","last_active":"2024-02-21T23:06:13.029781+0000","last_peered":"2024-02-21T23:06:13.029781+0000","last_clean":"2024-02-21T23:06:13.029781+0000","last_became_active":"2024-02-21T23:06:10.975183+0000","last_became_peered":"2024-02-21T23:06:10.975183+0000","last_unstale":"2024-02-21T23:06:13.029781+0000","last_undegraded":"2024-02-21T23:06:13.029781+0000","last_fullsized":"2024-02-21T23:06:13.029781+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:18:26.399820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.535896+0000","last_change":"2024-02-21T23:05:51.416524+0000","last_active":"2024-02-21T23:06:12.535896+0000","last_peered":"2024-02-21T23:06:12.535896+0000","last_clean":"2024-02-21T23:06:12.535896+0000","last_became_active":"2024-02-21T23:05:51.415391+0000","last_became_peered":"2024-02-21T23:05:51.415391+0000","last_unstale":"2024-02-21T23:06:12.535896+0000","last_undegraded":"2024-02-21T23:06:12.535896+0000","last_fullsized":"2024-02-21T23:06:12.535896+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:48:32.958103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.35","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.977890+0000","last_change":"2024-02-21T23:06:10.980906+0000","last_active":"2024-02-21T23:06:11.977890+0000","last_peered":"2024-02-21T23:06:11.977890+0000","last_clean":"2024-02-21T23:06:11.977890+0000","last_became_active":"2024-02-21T23:06:10.980361+0000","last_became_peered":"2024-02-21T23:06:10.980361+0000","last_unstale":"2024-02-21T23:06:11.977890+0000","last_undegraded":"2024-02-21T23:06:11.977890+0000","last_fullsized":"2024-02-21T23:06:11.977890+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:04:22.657947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.978465+0000","last_change":"2024-02-21T23:05:51.421068+0000","last_active":"2024-02-21T23:06:11.978465+0000","last_peered":"2024-02-21T23:06:11.978465+0000","last_clean":"2024-02-21T23:06:11.978465+0000","last_became_active":"2024-02-21T23:05:51.419746+0000","last_became_peered":"2024-02-21T23:05:51.419746+0000","last_unstale":"2024-02-21T23:06:11.978465+0000","last_undegraded":"2024-02-21T23:06:11.978465+0000","last_fullsized":"2024-02-21T23:06:11.978465+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:09:42.444733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.34","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.536406+0000","last_change":"2024-02-21T23:06:10.966147+0000","last_active":"2024-02-21T23:06:12.536406+0000","last_peered":"2024-02-21T23:06:12.536406+0000","last_clean":"2024-02-21T23:06:12.536406+0000","last_became_active":"2024-02-21T23:06:10.965783+0000","last_became_peered":"2024-02-21T23:06:10.965783+0000","last_unstale":"2024-02-21T23:06:12.536406+0000","last_undegraded":"2024-02-21T23:06:12.536406+0000","last_fullsized":"2024-02-21T23:06:12.536406+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:10:02.018168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.959863+0000","last_change":"2024-02-21T23:05:51.418295+0000","last_active":"2024-02-21T23:06:12.959863+0000","last_peered":"2024-02-21T23:06:12.959863+0000","last_clean":"2024-02-21T23:06:12.959863+0000","last_became_active":"2024-02-21T23:05:51.417780+0000","last_became_peered":"2024-02-21T23:05:51.417780+0000","last_unstale":"2024-02-21T23:06:12.959863+0000","last_undegraded":"2024-02-21T23:06:12.959863+0000","last_fullsized":"2024-02-21T23:06:12.959863+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:19:57.210565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.37","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.978058+0000","last_change":"2024-02-21T23:06:10.983699+0000","last_active":"2024-02-21T23:06:11.978058+0000","last_peered":"2024-02-21T23:06:11.978058+0000","last_clean":"2024-02-21T23:06:11.978058+0000","last_became_active":"2024-02-21T23:06:10.982870+0000","last_became_peered":"2024-02-21T23:06:10.982870+0000","last_unstale":"2024-02-21T23:06:11.978058+0000","last_undegraded":"2024-02-21T23:06:11.978058+0000","last_fullsized":"2024-02-21T23:06:11.978058+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:24:52.436507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.030605+0000","last_change":"2024-02-21T23:05:51.420518+0000","last_active":"2024-02-21T23:06:13.030605+0000","last_peered":"2024-02-21T23:06:13.030605+0000","last_clean":"2024-02-21T23:06:13.030605+0000","last_became_active":"2024-02-21T23:05:51.419564+0000","last_became_peered":"2024-02-21T23:05:51.419564+0000","last_unstale":"2024-02-21T23:06:13.030605+0000","last_undegraded":"2024-02-21T23:06:13.030605+0000","last_fullsized":"2024-02-21T23:06:13.030605+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:19:41.280587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.36","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.536772+0000","last_change":"2024-02-21T23:06:10.993431+0000","last_active":"2024-02-21T23:06:12.536772+0000","last_peered":"2024-02-21T23:06:12.536772+0000","last_clean":"2024-02-21T23:06:12.536772+0000","last_became_active":"2024-02-21T23:06:10.993067+0000","last_became_peered":"2024-02-21T23:06:10.993067+0000","last_unstale":"2024-02-21T23:06:12.536772+0000","last_undegraded":"2024-02-21T23:06:12.536772+0000","last_fullsized":"2024-02-21T23:06:12.536772+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:20:39.137766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.959701+0000","last_change":"2024-02-21T23:05:51.421691+0000","last_active":"2024-02-21T23:06:12.959701+0000","last_peered":"2024-02-21T23:06:12.959701+0000","last_clean":"2024-02-21T23:06:12.959701+0000","last_became_active":"2024-02-21T23:05:51.417621+0000","last_became_peered":"2024-02-21T23:05:51.417621+0000","last_unstale":"2024-02-21T23:06:12.959701+0000","last_undegraded":"2024-02-21T23:06:12.959701+0000","last_fullsized":"2024-02-21T23:06:12.959701+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:46:01.608149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.31","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.970970+0000","last_change":"2024-02-21T23:06:10.962402+0000","last_active":"2024-02-21T23:06:11.970970+0000","last_peered":"2024-02-21T23:06:11.970970+0000","last_clean":"2024-02-21T23:06:11.970970+0000","last_became_active":"2024-02-21T23:06:10.962127+0000","last_became_peered":"2024-02-21T23:06:10.962127+0000","last_unstale":"2024-02-21T23:06:11.970970+0000","last_undegraded":"2024-02-21T23:06:11.970970+0000","last_fullsized":"2024-02-21T23:06:11.970970+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:50:05.598685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.536283+0000","last_change":"2024-02-21T23:05:51.426871+0000","last_active":"2024-02-21T23:06:12.536283+0000","last_peered":"2024-02-21T23:06:12.536283+0000","last_clean":"2024-02-21T23:06:12.536283+0000","last_became_active":"2024-02-21T23:05:51.419412+0000","last_became_peered":"2024-02-21T23:05:51.419412+0000","last_unstale":"2024-02-21T23:06:12.536283+0000","last_undegraded":"2024-02-21T23:06:12.536283+0000","last_fullsized":"2024-02-21T23:06:12.536283+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:22:09.156621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.30","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.029598+0000","last_change":"2024-02-21T23:06:10.967734+0000","last_active":"2024-02-21T23:06:13.029598+0000","last_peered":"2024-02-21T23:06:13.029598+0000","last_clean":"2024-02-21T23:06:13.029598+0000","last_became_active":"2024-02-21T23:06:10.967386+0000","last_became_peered":"2024-02-21T23:06:10.967386+0000","last_unstale":"2024-02-21T23:06:13.029598+0000","last_undegraded":"2024-02-21T23:06:13.029598+0000","last_fullsized":"2024-02-21T23:06:13.029598+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:00:08.218703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.976550+0000","last_change":"2024-02-21T23:05:51.417593+0000","last_active":"2024-02-21T23:06:11.976550+0000","last_peered":"2024-02-21T23:06:11.976550+0000","last_clean":"2024-02-21T23:06:11.976550+0000","last_became_active":"2024-02-21T23:05:51.416945+0000","last_became_peered":"2024-02-21T23:05:51.416945+0000","last_unstale":"2024-02-21T23:06:11.976550+0000","last_undegraded":"2024-02-21T23:06:11.976550+0000","last_fullsized":"2024-02-21T23:06:11.976550+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:24:32.863072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.33","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.972115+0000","last_change":"2024-02-21T23:06:10.991670+0000","last_active":"2024-02-21T23:06:11.972115+0000","last_peered":"2024-02-21T23:06:11.972115+0000","last_clean":"2024-02-21T23:06:11.972115+0000","last_became_active":"2024-02-21T23:06:10.990780+0000","last_became_peered":"2024-02-21T23:06:10.990780+0000","last_unstale":"2024-02-21T23:06:11.972115+0000","last_undegraded":"2024-02-21T23:06:11.972115+0000","last_fullsized":"2024-02-21T23:06:11.972115+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:42:46.061945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.029483+0000","last_change":"2024-02-21T23:05:51.409645+0000","last_active":"2024-02-21T23:06:13.029483+0000","last_peered":"2024-02-21T23:06:13.029483+0000","last_clean":"2024-02-21T23:06:13.029483+0000","last_became_active":"2024-02-21T23:05:51.409356+0000","last_became_peered":"2024-02-21T23:05:51.409356+0000","last_unstale":"2024-02-21T23:06:13.029483+0000","last_undegraded":"2024-02-21T23:06:13.029483+0000","last_fullsized":"2024-02-21T23:06:13.029483+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:17:26.329609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.32","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.976374+0000","last_change":"2024-02-21T23:06:10.991249+0000","last_active":"2024-02-21T23:06:11.976374+0000","last_peered":"2024-02-21T23:06:11.976374+0000","last_clean":"2024-02-21T23:06:11.976374+0000","last_became_active":"2024-02-21T23:06:10.990938+0000","last_became_peered":"2024-02-21T23:06:10.990938+0000","last_unstale":"2024-02-21T23:06:11.976374+0000","last_undegraded":"2024-02-21T23:06:11.976374+0000","last_fullsized":"2024-02-21T23:06:11.976374+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:36:43.365830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.034178+0000","last_change":"2024-02-21T23:05:51.413656+0000","last_active":"2024-02-21T23:06:13.034178+0000","last_peered":"2024-02-21T23:06:13.034178+0000","last_clean":"2024-02-21T23:06:13.034178+0000","last_became_active":"2024-02-21T23:05:51.413350+0000","last_became_peered":"2024-02-21T23:05:51.413350+0000","last_unstale":"2024-02-21T23:06:13.034178+0000","last_undegraded":"2024-02-21T23:06:13.034178+0000","last_fullsized":"2024-02-21T23:06:13.034178+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:19:57.210565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.2d","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.978358+0000","last_change":"2024-02-21T23:06:10.968835+0000","last_active":"2024-02-21T23:06:11.978358+0000","last_peered":"2024-02-21T23:06:11.978358+0000","last_clean":"2024-02-21T23:06:11.978358+0000","last_became_active":"2024-02-21T23:06:10.968330+0000","last_became_peered":"2024-02-21T23:06:10.968330+0000","last_unstale":"2024-02-21T23:06:11.978358+0000","last_undegraded":"2024-02-21T23:06:11.978358+0000","last_fullsized":"2024-02-21T23:06:11.978358+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:41:50.834277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.536898+0000","last_change":"2024-02-21T23:05:51.427044+0000","last_active":"2024-02-21T23:06:12.536898+0000","last_peered":"2024-02-21T23:06:12.536898+0000","last_clean":"2024-02-21T23:06:12.536898+0000","last_became_active":"2024-02-21T23:05:51.419477+0000","last_became_peered":"2024-02-21T23:05:51.419477+0000","last_unstale":"2024-02-21T23:06:12.536898+0000","last_undegraded":"2024-02-21T23:06:12.536898+0000","last_fullsized":"2024-02-21T23:06:12.536898+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:04:00.714448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.2c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.970780+0000","last_change":"2024-02-21T23:06:10.975705+0000","last_active":"2024-02-21T23:06:11.970780+0000","last_peered":"2024-02-21T23:06:11.970780+0000","last_clean":"2024-02-21T23:06:11.970780+0000","last_became_active":"2024-02-21T23:06:10.970250+0000","last_became_peered":"2024-02-21T23:06:10.970250+0000","last_unstale":"2024-02-21T23:06:11.970780+0000","last_undegraded":"2024-02-21T23:06:11.970780+0000","last_fullsized":"2024-02-21T23:06:11.970780+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:58:24.687732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.971789+0000","last_change":"2024-02-21T23:05:51.425126+0000","last_active":"2024-02-21T23:06:11.971789+0000","last_peered":"2024-02-21T23:06:11.971789+0000","last_clean":"2024-02-21T23:06:11.971789+0000","last_became_active":"2024-02-21T23:05:51.422511+0000","last_became_peered":"2024-02-21T23:05:51.422511+0000","last_unstale":"2024-02-21T23:06:11.971789+0000","last_undegraded":"2024-02-21T23:06:11.971789+0000","last_fullsized":"2024-02-21T23:06:11.971789+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:27:01.191337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.2f","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.030469+0000","last_change":"2024-02-21T23:06:10.977919+0000","last_active":"2024-02-21T23:06:13.030469+0000","last_peered":"2024-02-21T23:06:13.030469+0000","last_clean":"2024-02-21T23:06:13.030469+0000","last_became_active":"2024-02-21T23:06:10.968729+0000","last_became_peered":"2024-02-21T23:06:10.968729+0000","last_unstale":"2024-02-21T23:06:13.030469+0000","last_undegraded":"2024-02-21T23:06:13.030469+0000","last_fullsized":"2024-02-21T23:06:13.030469+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:10:15.667957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.978241+0000","last_change":"2024-02-21T23:05:51.423258+0000","last_active":"2024-02-21T23:06:11.978241+0000","last_peered":"2024-02-21T23:06:11.978241+0000","last_clean":"2024-02-21T23:06:11.978241+0000","last_became_active":"2024-02-21T23:05:51.422987+0000","last_became_peered":"2024-02-21T23:05:51.422987+0000","last_unstale":"2024-02-21T23:06:11.978241+0000","last_undegraded":"2024-02-21T23:06:11.978241+0000","last_fullsized":"2024-02-21T23:06:11.978241+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:41:31.260842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2e","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.978880+0000","last_change":"2024-02-21T23:06:10.973113+0000","last_active":"2024-02-21T23:06:11.978880+0000","last_peered":"2024-02-21T23:06:11.978880+0000","last_clean":"2024-02-21T23:06:11.978880+0000","last_became_active":"2024-02-21T23:06:10.972815+0000","last_became_peered":"2024-02-21T23:06:10.972815+0000","last_unstale":"2024-02-21T23:06:11.978880+0000","last_undegraded":"2024-02-21T23:06:11.978880+0000","last_fullsized":"2024-02-21T23:06:11.978880+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:00:08.218703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.536378+0000","last_change":"2024-02-21T23:05:51.427468+0000","last_active":"2024-02-21T23:06:12.536378+0000","last_peered":"2024-02-21T23:06:12.536378+0000","last_clean":"2024-02-21T23:06:12.536378+0000","last_became_active":"2024-02-21T23:05:51.425476+0000","last_became_peered":"2024-02-21T23:05:51.425476+0000","last_unstale":"2024-02-21T23:06:12.536378+0000","last_undegraded":"2024-02-21T23:06:12.536378+0000","last_fullsized":"2024-02-21T23:06:12.536378+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:06:18.158392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.29","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.033158+0000","last_change":"2024-02-21T23:06:10.993284+0000","last_active":"2024-02-21T23:06:13.033158+0000","last_peered":"2024-02-21T23:06:13.033158+0000","last_clean":"2024-02-21T23:06:13.033158+0000","last_became_active":"2024-02-21T23:06:10.992937+0000","last_became_peered":"2024-02-21T23:06:10.992937+0000","last_unstale":"2024-02-21T23:06:13.033158+0000","last_undegraded":"2024-02-21T23:06:13.033158+0000","last_fullsized":"2024-02-21T23:06:13.033158+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:50:05.598685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.978079+0000","last_change":"2024-02-21T23:05:51.415772+0000","last_active":"2024-02-21T23:06:11.978079+0000","last_peered":"2024-02-21T23:06:11.978079+0000","last_clean":"2024-02-21T23:06:11.978079+0000","last_became_active":"2024-02-21T23:05:51.415494+0000","last_became_peered":"2024-02-21T23:05:51.415494+0000","last_unstale":"2024-02-21T23:06:11.978079+0000","last_undegraded":"2024-02-21T23:06:11.978079+0000","last_fullsized":"2024-02-21T23:06:11.978079+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:42:37.006685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.28","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.976025+0000","last_change":"2024-02-21T23:06:10.981808+0000","last_active":"2024-02-21T23:06:11.976025+0000","last_peered":"2024-02-21T23:06:11.976025+0000","last_clean":"2024-02-21T23:06:11.976025+0000","last_became_active":"2024-02-21T23:06:10.979421+0000","last_became_peered":"2024-02-21T23:06:10.979421+0000","last_unstale":"2024-02-21T23:06:11.976025+0000","last_undegraded":"2024-02-21T23:06:11.976025+0000","last_fullsized":"2024-02-21T23:06:11.976025+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:56:29.967189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.535991+0000","last_change":"2024-02-21T23:05:51.428557+0000","last_active":"2024-02-21T23:06:12.535991+0000","last_peered":"2024-02-21T23:06:12.535991+0000","last_clean":"2024-02-21T23:06:12.535991+0000","last_became_active":"2024-02-21T23:05:51.427612+0000","last_became_peered":"2024-02-21T23:05:51.427612+0000","last_unstale":"2024-02-21T23:06:12.535991+0000","last_undegraded":"2024-02-21T23:06:12.535991+0000","last_fullsized":"2024-02-21T23:06:12.535991+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:38:44.895318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.2b","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.977011+0000","last_change":"2024-02-21T23:06:10.979884+0000","last_active":"2024-02-21T23:06:11.977011+0000","last_peered":"2024-02-21T23:06:11.977011+0000","last_clean":"2024-02-21T23:06:11.977011+0000","last_became_active":"2024-02-21T23:06:10.979079+0000","last_became_peered":"2024-02-21T23:06:10.979079+0000","last_unstale":"2024-02-21T23:06:11.977011+0000","last_undegraded":"2024-02-21T23:06:11.977011+0000","last_fullsized":"2024-02-21T23:06:11.977011+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:03:54.817384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.975835+0000","last_change":"2024-02-21T23:05:51.417302+0000","last_active":"2024-02-21T23:06:11.975835+0000","last_peered":"2024-02-21T23:06:11.975835+0000","last_clean":"2024-02-21T23:06:11.975835+0000","last_became_active":"2024-02-21T23:05:51.414092+0000","last_became_peered":"2024-02-21T23:05:51.414092+0000","last_unstale":"2024-02-21T23:06:11.975835+0000","last_undegraded":"2024-02-21T23:06:11.975835+0000","last_fullsized":"2024-02-21T23:06:11.975835+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:49:01.911867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.976955+0000","last_change":"2024-02-21T23:06:10.989029+0000","last_active":"2024-02-21T23:06:11.976955+0000","last_peered":"2024-02-21T23:06:11.976955+0000","last_clean":"2024-02-21T23:06:11.976955+0000","last_became_active":"2024-02-21T23:06:10.988586+0000","last_became_peered":"2024-02-21T23:06:10.988586+0000","last_unstale":"2024-02-21T23:06:11.976955+0000","last_undegraded":"2024-02-21T23:06:11.976955+0000","last_fullsized":"2024-02-21T23:06:11.976955+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:41:05.018853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.029594+0000","last_change":"2024-02-21T23:05:51.406411+0000","last_active":"2024-02-21T23:06:13.029594+0000","last_peered":"2024-02-21T23:06:13.029594+0000","last_clean":"2024-02-21T23:06:13.029594+0000","last_became_active":"2024-02-21T23:05:51.406104+0000","last_became_peered":"2024-02-21T23:05:51.406104+0000","last_unstale":"2024-02-21T23:06:13.029594+0000","last_undegraded":"2024-02-21T23:06:13.029594+0000","last_fullsized":"2024-02-21T23:06:13.029594+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:03:08.789066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.25","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.536618+0000","last_change":"2024-02-21T23:06:10.978687+0000","last_active":"2024-02-21T23:06:12.536618+0000","last_peered":"2024-02-21T23:06:12.536618+0000","last_clean":"2024-02-21T23:06:12.536618+0000","last_became_active":"2024-02-21T23:06:10.978393+0000","last_became_peered":"2024-02-21T23:06:10.978393+0000","last_unstale":"2024-02-21T23:06:12.536618+0000","last_undegraded":"2024-02-21T23:06:12.536618+0000","last_fullsized":"2024-02-21T23:06:12.536618+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:32:04.110019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.030786+0000","last_change":"2024-02-21T23:05:51.414112+0000","last_active":"2024-02-21T23:06:13.030786+0000","last_peered":"2024-02-21T23:06:13.030786+0000","last_clean":"2024-02-21T23:06:13.030786+0000","last_became_active":"2024-02-21T23:05:51.413796+0000","last_became_peered":"2024-02-21T23:05:51.413796+0000","last_unstale":"2024-02-21T23:06:13.030786+0000","last_undegraded":"2024-02-21T23:06:13.030786+0000","last_fullsized":"2024-02-21T23:06:13.030786+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:42:26.488510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.24","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.971701+0000","last_change":"2024-02-21T23:06:10.982423+0000","last_active":"2024-02-21T23:06:11.971701+0000","last_peered":"2024-02-21T23:06:11.971701+0000","last_clean":"2024-02-21T23:06:11.971701+0000","last_became_active":"2024-02-21T23:06:10.981600+0000","last_became_peered":"2024-02-21T23:06:10.981600+0000","last_unstale":"2024-02-21T23:06:11.971701+0000","last_undegraded":"2024-02-21T23:06:11.971701+0000","last_fullsized":"2024-02-21T23:06:11.971701+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:11:16.456657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.960079+0000","last_change":"2024-02-21T23:05:51.421208+0000","last_active":"2024-02-21T23:06:12.960079+0000","last_peered":"2024-02-21T23:06:12.960079+0000","last_clean":"2024-02-21T23:06:12.960079+0000","last_became_active":"2024-02-21T23:05:51.418171+0000","last_became_peered":"2024-02-21T23:05:51.418171+0000","last_unstale":"2024-02-21T23:06:12.960079+0000","last_undegraded":"2024-02-21T23:06:12.960079+0000","last_fullsized":"2024-02-21T23:06:12.960079+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:17:26.329609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.27","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.960254+0000","last_change":"2024-02-21T23:06:10.986399+0000","last_active":"2024-02-21T23:06:12.960254+0000","last_peered":"2024-02-21T23:06:12.960254+0000","last_clean":"2024-02-21T23:06:12.960254+0000","last_became_active":"2024-02-21T23:06:10.985542+0000","last_became_peered":"2024-02-21T23:06:10.985542+0000","last_unstale":"2024-02-21T23:06:12.960254+0000","last_undegraded":"2024-02-21T23:06:12.960254+0000","last_fullsized":"2024-02-21T23:06:12.960254+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:37:11.082403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.034410+0000","last_change":"2024-02-21T23:05:51.416857+0000","last_active":"2024-02-21T23:06:13.034410+0000","last_peered":"2024-02-21T23:06:13.034410+0000","last_clean":"2024-02-21T23:06:13.034410+0000","last_became_active":"2024-02-21T23:05:51.416577+0000","last_became_peered":"2024-02-21T23:05:51.416577+0000","last_unstale":"2024-02-21T23:06:13.034410+0000","last_undegraded":"2024-02-21T23:06:13.034410+0000","last_fullsized":"2024-02-21T23:06:13.034410+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:17:26.329609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.26","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.977901+0000","last_change":"2024-02-21T23:06:10.991753+0000","last_active":"2024-02-21T23:06:11.977901+0000","last_peered":"2024-02-21T23:06:11.977901+0000","last_clean":"2024-02-21T23:06:11.977901+0000","last_became_active":"2024-02-21T23:06:10.990907+0000","last_became_peered":"2024-02-21T23:06:10.990907+0000","last_unstale":"2024-02-21T23:06:11.977901+0000","last_undegraded":"2024-02-21T23:06:11.977901+0000","last_fullsized":"2024-02-21T23:06:11.977901+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:59:32.865929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.536463+0000","last_change":"2024-02-21T23:05:51.427489+0000","last_active":"2024-02-21T23:06:12.536463+0000","last_peered":"2024-02-21T23:06:12.536463+0000","last_clean":"2024-02-21T23:06:12.536463+0000","last_became_active":"2024-02-21T23:05:51.419458+0000","last_became_peered":"2024-02-21T23:05:51.419458+0000","last_unstale":"2024-02-21T23:06:12.536463+0000","last_undegraded":"2024-02-21T23:06:12.536463+0000","last_fullsized":"2024-02-21T23:06:12.536463+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:54:08.758447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.21","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.959168+0000","last_change":"2024-02-21T23:06:10.991205+0000","last_active":"2024-02-21T23:06:12.959168+0000","last_peered":"2024-02-21T23:06:12.959168+0000","last_clean":"2024-02-21T23:06:12.959168+0000","last_became_active":"2024-02-21T23:06:10.990883+0000","last_became_peered":"2024-02-21T23:06:10.990883+0000","last_unstale":"2024-02-21T23:06:12.959168+0000","last_undegraded":"2024-02-21T23:06:12.959168+0000","last_fullsized":"2024-02-21T23:06:12.959168+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:15:06.720068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.030070+0000","last_change":"2024-02-21T23:05:51.405741+0000","last_active":"2024-02-21T23:06:13.030070+0000","last_peered":"2024-02-21T23:06:13.030070+0000","last_clean":"2024-02-21T23:06:13.030070+0000","last_became_active":"2024-02-21T23:05:51.405301+0000","last_became_peered":"2024-02-21T23:05:51.405301+0000","last_unstale":"2024-02-21T23:06:13.030070+0000","last_undegraded":"2024-02-21T23:06:13.030070+0000","last_fullsized":"2024-02-21T23:06:13.030070+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:36:47.962935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.20","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.029444+0000","last_change":"2024-02-21T23:06:10.967642+0000","last_active":"2024-02-21T23:06:13.029444+0000","last_peered":"2024-02-21T23:06:13.029444+0000","last_clean":"2024-02-21T23:06:13.029444+0000","last_became_active":"2024-02-21T23:06:10.967318+0000","last_became_peered":"2024-02-21T23:06:10.967318+0000","last_unstale":"2024-02-21T23:06:13.029444+0000","last_undegraded":"2024-02-21T23:06:13.029444+0000","last_fullsized":"2024-02-21T23:06:13.029444+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:40:55.366111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.032842+0000","last_change":"2024-02-21T23:05:51.398663+0000","last_active":"2024-02-21T23:06:13.032842+0000","last_peered":"2024-02-21T23:06:13.032842+0000","last_clean":"2024-02-21T23:06:13.032842+0000","last_became_active":"2024-02-21T23:05:51.398146+0000","last_became_peered":"2024-02-21T23:05:51.398146+0000","last_unstale":"2024-02-21T23:06:13.032842+0000","last_undegraded":"2024-02-21T23:06:13.032842+0000","last_fullsized":"2024-02-21T23:06:13.032842+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:27:54.695721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.23","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.033534+0000","last_change":"2024-02-21T23:06:10.976500+0000","last_active":"2024-02-21T23:06:13.033534+0000","last_peered":"2024-02-21T23:06:13.033534+0000","last_clean":"2024-02-21T23:06:13.033534+0000","last_became_active":"2024-02-21T23:06:10.976135+0000","last_became_peered":"2024-02-21T23:06:10.976135+0000","last_unstale":"2024-02-21T23:06:13.033534+0000","last_undegraded":"2024-02-21T23:06:13.033534+0000","last_fullsized":"2024-02-21T23:06:13.033534+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:19:08.047583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.971321+0000","last_change":"2024-02-21T23:05:51.421330+0000","last_active":"2024-02-21T23:06:11.971321+0000","last_peered":"2024-02-21T23:06:11.971321+0000","last_clean":"2024-02-21T23:06:11.971321+0000","last_became_active":"2024-02-21T23:05:51.421012+0000","last_became_peered":"2024-02-21T23:05:51.421012+0000","last_unstale":"2024-02-21T23:06:11.971321+0000","last_undegraded":"2024-02-21T23:06:11.971321+0000","last_fullsized":"2024-02-21T23:06:11.971321+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:26:33.366331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.22","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.976531+0000","last_change":"2024-02-21T23:06:10.983139+0000","last_active":"2024-02-21T23:06:11.976531+0000","last_peered":"2024-02-21T23:06:11.976531+0000","last_clean":"2024-02-21T23:06:11.976531+0000","last_became_active":"2024-02-21T23:06:10.982029+0000","last_became_peered":"2024-02-21T23:06:10.982029+0000","last_unstale":"2024-02-21T23:06:11.976531+0000","last_undegraded":"2024-02-21T23:06:11.976531+0000","last_fullsized":"2024-02-21T23:06:11.976531+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:42:46.061945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.034425+0000","last_change":"2024-02-21T23:05:51.403173+0000","last_active":"2024-02-21T23:06:13.034425+0000","last_peered":"2024-02-21T23:06:13.034425+0000","last_clean":"2024-02-21T23:06:13.034425+0000","last_became_active":"2024-02-21T23:05:51.402610+0000","last_became_peered":"2024-02-21T23:05:51.402610+0000","last_unstale":"2024-02-21T23:06:13.034425+0000","last_undegraded":"2024-02-21T23:06:13.034425+0000","last_fullsized":"2024-02-21T23:06:13.034425+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:46:01.608149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.030709+0000","last_change":"2024-02-21T23:06:10.977357+0000","last_active":"2024-02-21T23:06:13.030709+0000","last_peered":"2024-02-21T23:06:13.030709+0000","last_clean":"2024-02-21T23:06:13.030709+0000","last_became_active":"2024-02-21T23:06:10.977075+0000","last_became_peered":"2024-02-21T23:06:10.977075+0000","last_unstale":"2024-02-21T23:06:13.030709+0000","last_undegraded":"2024-02-21T23:06:13.030709+0000","last_fullsized":"2024-02-21T23:06:13.030709+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:33:34.425087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.976736+0000","last_change":"2024-02-21T23:06:09.955402+0000","last_active":"2024-02-21T23:06:11.976736+0000","last_peered":"2024-02-21T23:06:11.976736+0000","last_clean":"2024-02-21T23:06:11.976736+0000","last_became_active":"2024-02-21T23:06:09.955113+0000","last_became_peered":"2024-02-21T23:06:09.955113+0000","last_unstale":"2024-02-21T23:06:11.976736+0000","last_undegraded":"2024-02-21T23:06:11.976736+0000","last_fullsized":"2024-02-21T23:06:11.976736+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:37:10.069490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.971922+0000","last_change":"2024-02-21T23:05:51.416534+0000","last_active":"2024-02-21T23:06:11.971922+0000","last_peered":"2024-02-21T23:06:11.971922+0000","last_clean":"2024-02-21T23:06:11.971922+0000","last_became_active":"2024-02-21T23:05:51.416244+0000","last_became_peered":"2024-02-21T23:05:51.416244+0000","last_unstale":"2024-02-21T23:06:11.971922+0000","last_undegraded":"2024-02-21T23:06:11.971922+0000","last_fullsized":"2024-02-21T23:06:11.971922+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:42:37.006685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"75'4","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.536913+0000","last_change":"2024-02-21T23:05:50.354109+0000","last_active":"2024-02-21T23:06:12.536913+0000","last_peered":"2024-02-21T23:06:12.536913+0000","last_clean":"2024-02-21T23:06:12.536913+0000","last_became_active":"2024-02-21T23:05:50.353809+0000","last_became_peered":"2024-02-21T23:05:50.353809+0000","last_unstale":"2024-02-21T23:06:12.536913+0000","last_undegraded":"2024-02-21T23:06:12.536913+0000","last_fullsized":"2024-02-21T23:06:12.536913+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:28:13.765557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.972268+0000","last_change":"2024-02-21T23:06:10.985497+0000","last_active":"2024-02-21T23:06:11.972268+0000","last_peered":"2024-02-21T23:06:11.972268+0000","last_clean":"2024-02-21T23:06:11.972268+0000","last_became_active":"2024-02-21T23:06:10.985191+0000","last_became_peered":"2024-02-21T23:06:10.985191+0000","last_unstale":"2024-02-21T23:06:11.972268+0000","last_undegraded":"2024-02-21T23:06:11.972268+0000","last_fullsized":"2024-02-21T23:06:11.972268+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:41:05.018853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.033912+0000","last_change":"2024-02-21T23:06:09.963140+0000","last_active":"2024-02-21T23:06:13.033912+0000","last_peered":"2024-02-21T23:06:13.033912+0000","last_clean":"2024-02-21T23:06:13.033912+0000","last_became_active":"2024-02-21T23:06:09.962637+0000","last_became_peered":"2024-02-21T23:06:09.962637+0000","last_unstale":"2024-02-21T23:06:13.033912+0000","last_undegraded":"2024-02-21T23:06:13.033912+0000","last_fullsized":"2024-02-21T23:06:13.033912+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:41:49.821364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.029524+0000","last_change":"2024-02-21T23:05:51.416268+0000","last_active":"2024-02-21T23:06:13.029524+0000","last_peered":"2024-02-21T23:06:13.029524+0000","last_clean":"2024-02-21T23:06:13.029524+0000","last_became_active":"2024-02-21T23:05:51.415980+0000","last_became_peered":"2024-02-21T23:05:51.415980+0000","last_unstale":"2024-02-21T23:06:13.029524+0000","last_undegraded":"2024-02-21T23:06:13.029524+0000","last_fullsized":"2024-02-21T23:06:13.029524+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:51:17.193881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"75'1","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.035404+0000","last_change":"2024-02-21T23:05:50.383438+0000","last_active":"2024-02-21T23:06:13.035404+0000","last_peered":"2024-02-21T23:06:13.035404+0000","last_clean":"2024-02-21T23:06:13.035404+0000","last_became_active":"2024-02-21T23:05:50.383134+0000","last_became_peered":"2024-02-21T23:05:50.383134+0000","last_unstale":"2024-02-21T23:06:13.035404+0000","last_undegraded":"2024-02-21T23:06:13.035404+0000","last_fullsized":"2024-02-21T23:06:13.035404+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:08:35.448428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.971272+0000","last_change":"2024-02-21T23:06:10.970588+0000","last_active":"2024-02-21T23:06:11.971272+0000","last_peered":"2024-02-21T23:06:11.971272+0000","last_clean":"2024-02-21T23:06:11.971272+0000","last_became_active":"2024-02-21T23:06:10.970285+0000","last_became_peered":"2024-02-21T23:06:10.970285+0000","last_unstale":"2024-02-21T23:06:11.971272+0000","last_undegraded":"2024-02-21T23:06:11.971272+0000","last_fullsized":"2024-02-21T23:06:11.971272+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:20:00.854022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"79'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.976577+0000","last_change":"2024-02-21T23:06:09.957178+0000","last_active":"2024-02-21T23:06:11.976577+0000","last_peered":"2024-02-21T23:06:11.976577+0000","last_clean":"2024-02-21T23:06:11.976577+0000","last_became_active":"2024-02-21T23:06:09.956818+0000","last_became_peered":"2024-02-21T23:06:09.956818+0000","last_unstale":"2024-02-21T23:06:11.976577+0000","last_undegraded":"2024-02-21T23:06:11.976577+0000","last_fullsized":"2024-02-21T23:06:11.976577+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:24:34.391679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.977011+0000","last_change":"2024-02-21T23:05:51.421571+0000","last_active":"2024-02-21T23:06:11.977011+0000","last_peered":"2024-02-21T23:06:11.977011+0000","last_clean":"2024-02-21T23:06:11.977011+0000","last_became_active":"2024-02-21T23:05:51.421110+0000","last_became_peered":"2024-02-21T23:05:51.421110+0000","last_unstale":"2024-02-21T23:06:11.977011+0000","last_undegraded":"2024-02-21T23:06:11.977011+0000","last_fullsized":"2024-02-21T23:06:11.977011+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:56:36.688152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"75'2","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.034745+0000","last_change":"2024-02-21T23:05:50.383231+0000","last_active":"2024-02-21T23:06:13.034745+0000","last_peered":"2024-02-21T23:06:13.034745+0000","last_clean":"2024-02-21T23:06:13.034745+0000","last_became_active":"2024-02-21T23:05:50.382757+0000","last_became_peered":"2024-02-21T23:05:50.382757+0000","last_unstale":"2024-02-21T23:06:13.034745+0000","last_undegraded":"2024-02-21T23:06:13.034745+0000","last_fullsized":"2024-02-21T23:06:13.034745+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:49:36.738197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.034005+0000","last_change":"2024-02-21T23:06:10.979404+0000","last_active":"2024-02-21T23:06:13.034005+0000","last_peered":"2024-02-21T23:06:13.034005+0000","last_clean":"2024-02-21T23:06:13.034005+0000","last_became_active":"2024-02-21T23:06:10.979128+0000","last_became_peered":"2024-02-21T23:06:10.979128+0000","last_unstale":"2024-02-21T23:06:13.034005+0000","last_undegraded":"2024-02-21T23:06:13.034005+0000","last_fullsized":"2024-02-21T23:06:13.034005+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:20:00.854022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1a","version":"79'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.971772+0000","last_change":"2024-02-21T23:06:09.951255+0000","last_active":"2024-02-21T23:06:11.971772+0000","last_peered":"2024-02-21T23:06:11.971772+0000","last_clean":"2024-02-21T23:06:11.971772+0000","last_became_active":"2024-02-21T23:06:09.950934+0000","last_became_peered":"2024-02-21T23:06:09.950934+0000","last_unstale":"2024-02-21T23:06:11.971772+0000","last_undegraded":"2024-02-21T23:06:11.971772+0000","last_fullsized":"2024-02-21T23:06:11.971772+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:56:08.310911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.977534+0000","last_change":"2024-02-21T23:05:51.415892+0000","last_active":"2024-02-21T23:06:11.977534+0000","last_peered":"2024-02-21T23:06:11.977534+0000","last_clean":"2024-02-21T23:06:11.977534+0000","last_became_active":"2024-02-21T23:05:51.415520+0000","last_became_peered":"2024-02-21T23:05:51.415520+0000","last_unstale":"2024-02-21T23:06:11.977534+0000","last_undegraded":"2024-02-21T23:06:11.977534+0000","last_fullsized":"2024-02-21T23:06:11.977534+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:26:33.366331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"75'2","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.035819+0000","last_change":"2024-02-21T23:05:50.376050+0000","last_active":"2024-02-21T23:06:13.035819+0000","last_peered":"2024-02-21T23:06:13.035819+0000","last_clean":"2024-02-21T23:06:13.035819+0000","last_became_active":"2024-02-21T23:05:50.375731+0000","last_became_peered":"2024-02-21T23:05:50.375731+0000","last_unstale":"2024-02-21T23:06:13.035819+0000","last_undegraded":"2024-02-21T23:06:13.035819+0000","last_fullsized":"2024-02-21T23:06:13.035819+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:42:15.626763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.034775+0000","last_change":"2024-02-21T23:06:10.992590+0000","last_active":"2024-02-21T23:06:13.034775+0000","last_peered":"2024-02-21T23:06:13.034775+0000","last_clean":"2024-02-21T23:06:13.034775+0000","last_became_active":"2024-02-21T23:06:10.992286+0000","last_became_peered":"2024-02-21T23:06:10.992286+0000","last_unstale":"2024-02-21T23:06:13.034775+0000","last_undegraded":"2024-02-21T23:06:13.034775+0000","last_fullsized":"2024-02-21T23:06:13.034775+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:31:22.009771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.1d","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.034121+0000","last_change":"2024-02-21T23:06:09.957048+0000","last_active":"2024-02-21T23:06:13.034121+0000","last_peered":"2024-02-21T23:06:13.034121+0000","last_clean":"2024-02-21T23:06:13.034121+0000","last_became_active":"2024-02-21T23:06:09.956666+0000","last_became_peered":"2024-02-21T23:06:09.956666+0000","last_unstale":"2024-02-21T23:06:13.034121+0000","last_undegraded":"2024-02-21T23:06:13.034121+0000","last_fullsized":"2024-02-21T23:06:13.034121+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:24:51.423593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.030148+0000","last_change":"2024-02-21T23:05:51.428435+0000","last_active":"2024-02-21T23:06:13.030148+0000","last_peered":"2024-02-21T23:06:13.030148+0000","last_clean":"2024-02-21T23:06:13.030148+0000","last_became_active":"2024-02-21T23:05:51.428133+0000","last_became_peered":"2024-02-21T23:05:51.428133+0000","last_unstale":"2024-02-21T23:06:13.030148+0000","last_undegraded":"2024-02-21T23:06:13.030148+0000","last_fullsized":"2024-02-21T23:06:13.030148+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:49:46.025250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"75'1","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.978182+0000","last_change":"2024-02-21T23:05:50.377574+0000","last_active":"2024-02-21T23:06:11.978182+0000","last_peered":"2024-02-21T23:06:11.978182+0000","last_clean":"2024-02-21T23:06:11.978182+0000","last_became_active":"2024-02-21T23:05:50.377250+0000","last_became_peered":"2024-02-21T23:05:50.377250+0000","last_unstale":"2024-02-21T23:06:11.978182+0000","last_undegraded":"2024-02-21T23:06:11.978182+0000","last_fullsized":"2024-02-21T23:06:11.978182+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:26:38.366160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.030691+0000","last_change":"2024-02-21T23:06:10.983355+0000","last_active":"2024-02-21T23:06:13.030691+0000","last_peered":"2024-02-21T23:06:13.030691+0000","last_clean":"2024-02-21T23:06:13.030691+0000","last_became_active":"2024-02-21T23:06:10.982259+0000","last_became_peered":"2024-02-21T23:06:10.982259+0000","last_unstale":"2024-02-21T23:06:13.030691+0000","last_undegraded":"2024-02-21T23:06:13.030691+0000","last_fullsized":"2024-02-21T23:06:13.030691+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:19:24.322783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1c","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.035229+0000","last_change":"2024-02-21T23:06:09.947108+0000","last_active":"2024-02-21T23:06:13.035229+0000","last_peered":"2024-02-21T23:06:13.035229+0000","last_clean":"2024-02-21T23:06:13.035229+0000","last_became_active":"2024-02-21T23:06:09.946815+0000","last_became_peered":"2024-02-21T23:06:09.946815+0000","last_unstale":"2024-02-21T23:06:13.035229+0000","last_undegraded":"2024-02-21T23:06:13.035229+0000","last_fullsized":"2024-02-21T23:06:13.035229+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:03:27.349588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.030548+0000","last_change":"2024-02-21T23:05:51.421508+0000","last_active":"2024-02-21T23:06:13.030548+0000","last_peered":"2024-02-21T23:06:13.030548+0000","last_clean":"2024-02-21T23:06:13.030548+0000","last_became_active":"2024-02-21T23:05:51.420041+0000","last_became_peered":"2024-02-21T23:05:51.420041+0000","last_unstale":"2024-02-21T23:06:13.030548+0000","last_undegraded":"2024-02-21T23:06:13.030548+0000","last_fullsized":"2024-02-21T23:06:13.030548+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:10:56.883222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"75'1","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.959136+0000","last_change":"2024-02-21T23:05:50.378668+0000","last_active":"2024-02-21T23:06:12.959136+0000","last_peered":"2024-02-21T23:06:12.959136+0000","last_clean":"2024-02-21T23:06:12.959136+0000","last_became_active":"2024-02-21T23:05:50.378369+0000","last_became_peered":"2024-02-21T23:05:50.378369+0000","last_unstale":"2024-02-21T23:06:12.959136+0000","last_undegraded":"2024-02-21T23:06:12.959136+0000","last_fullsized":"2024-02-21T23:06:12.959136+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:45:05.837402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.029928+0000","last_change":"2024-02-21T23:06:10.988234+0000","last_active":"2024-02-21T23:06:13.029928+0000","last_peered":"2024-02-21T23:06:13.029928+0000","last_clean":"2024-02-21T23:06:13.029928+0000","last_became_active":"2024-02-21T23:06:10.985564+0000","last_became_peered":"2024-02-21T23:06:10.985564+0000","last_unstale":"2024-02-21T23:06:13.029928+0000","last_undegraded":"2024-02-21T23:06:13.029928+0000","last_fullsized":"2024-02-21T23:06:13.029928+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:35:03.330756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"79'4","reported_seq":18,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.035823+0000","last_change":"2024-02-21T23:06:09.949620+0000","last_active":"2024-02-21T23:06:13.035823+0000","last_peered":"2024-02-21T23:06:13.035823+0000","last_clean":"2024-02-21T23:06:13.035823+0000","last_became_active":"2024-02-21T23:06:09.949252+0000","last_became_peered":"2024-02-21T23:06:09.949252+0000","last_unstale":"2024-02-21T23:06:13.035823+0000","last_undegraded":"2024-02-21T23:06:13.035823+0000","last_fullsized":"2024-02-21T23:06:13.035823+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:46:20.168671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.979280+0000","last_change":"2024-02-21T23:05:51.419084+0000","last_active":"2024-02-21T23:06:11.979280+0000","last_peered":"2024-02-21T23:06:11.979280+0000","last_clean":"2024-02-21T23:06:11.979280+0000","last_became_active":"2024-02-21T23:05:51.418752+0000","last_became_peered":"2024-02-21T23:05:51.418752+0000","last_unstale":"2024-02-21T23:06:11.979280+0000","last_undegraded":"2024-02-21T23:06:11.979280+0000","last_fullsized":"2024-02-21T23:06:11.979280+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:20:19.564331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.029581+0000","last_change":"2024-02-21T23:05:50.375534+0000","last_active":"2024-02-21T23:06:13.029581+0000","last_peered":"2024-02-21T23:06:13.029581+0000","last_clean":"2024-02-21T23:06:13.029581+0000","last_became_active":"2024-02-21T23:05:50.373762+0000","last_became_peered":"2024-02-21T23:05:50.373762+0000","last_unstale":"2024-02-21T23:06:13.029581+0000","last_undegraded":"2024-02-21T23:06:13.029581+0000","last_fullsized":"2024-02-21T23:06:13.029581+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:49:36.738197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.030217+0000","last_change":"2024-02-21T23:06:10.964764+0000","last_active":"2024-02-21T23:06:13.030217+0000","last_peered":"2024-02-21T23:06:13.030217+0000","last_clean":"2024-02-21T23:06:13.030217+0000","last_became_active":"2024-02-21T23:06:10.964494+0000","last_became_peered":"2024-02-21T23:06:10.964494+0000","last_unstale":"2024-02-21T23:06:13.030217+0000","last_undegraded":"2024-02-21T23:06:13.030217+0000","last_fullsized":"2024-02-21T23:06:13.030217+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:53:45.860442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"79'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.977530+0000","last_change":"2024-02-21T23:06:09.952374+0000","last_active":"2024-02-21T23:06:11.977530+0000","last_peered":"2024-02-21T23:06:11.977530+0000","last_clean":"2024-02-21T23:06:11.977530+0000","last_became_active":"2024-02-21T23:06:09.950283+0000","last_became_peered":"2024-02-21T23:06:09.950283+0000","last_unstale":"2024-02-21T23:06:11.977530+0000","last_undegraded":"2024-02-21T23:06:11.977530+0000","last_fullsized":"2024-02-21T23:06:11.977530+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:50:51.483608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.036707+0000","last_change":"2024-02-21T23:05:51.426711+0000","last_active":"2024-02-21T23:06:13.036707+0000","last_peered":"2024-02-21T23:06:13.036707+0000","last_clean":"2024-02-21T23:06:13.036707+0000","last_became_active":"2024-02-21T23:05:51.426415+0000","last_became_peered":"2024-02-21T23:05:51.426415+0000","last_unstale":"2024-02-21T23:06:13.036707+0000","last_undegraded":"2024-02-21T23:06:13.036707+0000","last_fullsized":"2024-02-21T23:06:13.036707+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:58:55.829941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.535998+0000","last_change":"2024-02-21T23:05:50.355567+0000","last_active":"2024-02-21T23:06:12.535998+0000","last_peered":"2024-02-21T23:06:12.535998+0000","last_clean":"2024-02-21T23:06:12.535998+0000","last_became_active":"2024-02-21T23:05:50.354835+0000","last_became_peered":"2024-02-21T23:05:50.354835+0000","last_unstale":"2024-02-21T23:06:12.535998+0000","last_undegraded":"2024-02-21T23:06:12.535998+0000","last_fullsized":"2024-02-21T23:06:12.535998+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:21:52.655098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.535844+0000","last_change":"2024-02-21T23:06:10.992164+0000","last_active":"2024-02-21T23:06:12.535844+0000","last_peered":"2024-02-21T23:06:12.535844+0000","last_clean":"2024-02-21T23:06:12.535844+0000","last_became_active":"2024-02-21T23:06:10.991804+0000","last_became_peered":"2024-02-21T23:06:10.991804+0000","last_unstale":"2024-02-21T23:06:12.535844+0000","last_undegraded":"2024-02-21T23:06:12.535844+0000","last_fullsized":"2024-02-21T23:06:12.535844+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:49:21.485302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.971149+0000","last_change":"2024-02-21T23:06:09.946843+0000","last_active":"2024-02-21T23:06:11.971149+0000","last_peered":"2024-02-21T23:06:11.971149+0000","last_clean":"2024-02-21T23:06:11.971149+0000","last_became_active":"2024-02-21T23:06:09.946452+0000","last_became_peered":"2024-02-21T23:06:09.946452+0000","last_unstale":"2024-02-21T23:06:11.971149+0000","last_undegraded":"2024-02-21T23:06:11.971149+0000","last_fullsized":"2024-02-21T23:06:11.971149+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:50:51.483608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.029696+0000","last_change":"2024-02-21T23:05:51.399923+0000","last_active":"2024-02-21T23:06:13.029696+0000","last_peered":"2024-02-21T23:06:13.029696+0000","last_clean":"2024-02-21T23:06:13.029696+0000","last_became_active":"2024-02-21T23:05:51.399642+0000","last_became_peered":"2024-02-21T23:05:51.399642+0000","last_unstale":"2024-02-21T23:06:13.029696+0000","last_undegraded":"2024-02-21T23:06:13.029696+0000","last_fullsized":"2024-02-21T23:06:13.029696+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:21:18.440977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.035962+0000","last_change":"2024-02-21T23:05:50.386639+0000","last_active":"2024-02-21T23:06:13.035962+0000","last_peered":"2024-02-21T23:06:13.035962+0000","last_clean":"2024-02-21T23:06:13.035962+0000","last_became_active":"2024-02-21T23:05:50.386158+0000","last_became_peered":"2024-02-21T23:05:50.386158+0000","last_unstale":"2024-02-21T23:06:13.035962+0000","last_undegraded":"2024-02-21T23:06:13.035962+0000","last_fullsized":"2024-02-21T23:06:13.035962+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:43:17.496894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.536225+0000","last_change":"2024-02-21T23:06:10.983360+0000","last_active":"2024-02-21T23:06:12.536225+0000","last_peered":"2024-02-21T23:06:12.536225+0000","last_clean":"2024-02-21T23:06:12.536225+0000","last_became_active":"2024-02-21T23:06:10.983093+0000","last_became_peered":"2024-02-21T23:06:10.983093+0000","last_unstale":"2024-02-21T23:06:12.536225+0000","last_undegraded":"2024-02-21T23:06:12.536225+0000","last_fullsized":"2024-02-21T23:06:12.536225+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:56:56.261587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.10","version":"79'8","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.971653+0000","last_change":"2024-02-21T23:06:09.934904+0000","last_active":"2024-02-21T23:06:11.971653+0000","last_peered":"2024-02-21T23:06:11.971653+0000","last_clean":"2024-02-21T23:06:11.971653+0000","last_became_active":"2024-02-21T23:06:09.934606+0000","last_became_peered":"2024-02-21T23:06:09.934606+0000","last_unstale":"2024-02-21T23:06:11.971653+0000","last_undegraded":"2024-02-21T23:06:11.971653+0000","last_fullsized":"2024-02-21T23:06:11.971653+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:51:46.950044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.959439+0000","last_change":"2024-02-21T23:05:51.422844+0000","last_active":"2024-02-21T23:06:12.959439+0000","last_peered":"2024-02-21T23:06:12.959439+0000","last_clean":"2024-02-21T23:06:12.959439+0000","last_became_active":"2024-02-21T23:05:51.422488+0000","last_became_peered":"2024-02-21T23:05:51.422488+0000","last_unstale":"2024-02-21T23:06:12.959439+0000","last_undegraded":"2024-02-21T23:06:12.959439+0000","last_fullsized":"2024-02-21T23:06:12.959439+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:27:54.695721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.960134+0000","last_change":"2024-02-21T23:05:50.385146+0000","last_active":"2024-02-21T23:06:12.960134+0000","last_peered":"2024-02-21T23:06:12.960134+0000","last_clean":"2024-02-21T23:06:12.960134+0000","last_became_active":"2024-02-21T23:05:50.384418+0000","last_became_peered":"2024-02-21T23:05:50.384418+0000","last_unstale":"2024-02-21T23:06:12.960134+0000","last_undegraded":"2024-02-21T23:06:12.960134+0000","last_fullsized":"2024-02-21T23:06:12.960134+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:43:17.496894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.971506+0000","last_change":"2024-02-21T23:06:10.990710+0000","last_active":"2024-02-21T23:06:11.971506+0000","last_peered":"2024-02-21T23:06:11.971506+0000","last_clean":"2024-02-21T23:06:11.971506+0000","last_became_active":"2024-02-21T23:06:10.980292+0000","last_became_peered":"2024-02-21T23:06:10.980292+0000","last_unstale":"2024-02-21T23:06:11.971506+0000","last_undegraded":"2024-02-21T23:06:11.971506+0000","last_fullsized":"2024-02-21T23:06:11.971506+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:51:30.184287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.033819+0000","last_change":"2024-02-21T23:06:09.963033+0000","last_active":"2024-02-21T23:06:13.033819+0000","last_peered":"2024-02-21T23:06:13.033819+0000","last_clean":"2024-02-21T23:06:13.033819+0000","last_became_active":"2024-02-21T23:06:09.962695+0000","last_became_peered":"2024-02-21T23:06:09.962695+0000","last_unstale":"2024-02-21T23:06:13.033819+0000","last_undegraded":"2024-02-21T23:06:13.033819+0000","last_fullsized":"2024-02-21T23:06:13.033819+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:41:04.005940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.976677+0000","last_change":"2024-02-21T23:05:51.418004+0000","last_active":"2024-02-21T23:06:11.976677+0000","last_peered":"2024-02-21T23:06:11.976677+0000","last_clean":"2024-02-21T23:06:11.976677+0000","last_became_active":"2024-02-21T23:05:51.417463+0000","last_became_peered":"2024-02-21T23:05:51.417463+0000","last_unstale":"2024-02-21T23:06:11.976677+0000","last_undegraded":"2024-02-21T23:06:11.976677+0000","last_fullsized":"2024-02-21T23:06:11.976677+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:59:13.292494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"75'3","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.536685+0000","last_change":"2024-02-21T23:05:50.373189+0000","last_active":"2024-02-21T23:06:12.536685+0000","last_peered":"2024-02-21T23:06:12.536685+0000","last_clean":"2024-02-21T23:06:12.536685+0000","last_became_active":"2024-02-21T23:05:50.370680+0000","last_became_peered":"2024-02-21T23:05:50.370680+0000","last_unstale":"2024-02-21T23:06:12.536685+0000","last_undegraded":"2024-02-21T23:06:12.536685+0000","last_fullsized":"2024-02-21T23:06:12.536685+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:21:03.121546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.975807+0000","last_change":"2024-02-21T23:06:10.980721+0000","last_active":"2024-02-21T23:06:11.975807+0000","last_peered":"2024-02-21T23:06:11.975807+0000","last_clean":"2024-02-21T23:06:11.975807+0000","last_became_active":"2024-02-21T23:06:10.980424+0000","last_became_peered":"2024-02-21T23:06:10.980424+0000","last_unstale":"2024-02-21T23:06:11.975807+0000","last_undegraded":"2024-02-21T23:06:11.975807+0000","last_fullsized":"2024-02-21T23:06:11.975807+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:51:30.184287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.029909+0000","last_change":"2024-02-21T23:06:09.961844+0000","last_active":"2024-02-21T23:06:13.029909+0000","last_peered":"2024-02-21T23:06:13.029909+0000","last_clean":"2024-02-21T23:06:13.029909+0000","last_became_active":"2024-02-21T23:06:09.961480+0000","last_became_peered":"2024-02-21T23:06:09.961480+0000","last_unstale":"2024-02-21T23:06:13.029909+0000","last_undegraded":"2024-02-21T23:06:13.029909+0000","last_fullsized":"2024-02-21T23:06:13.029909+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:29:27.651018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.972413+0000","last_change":"2024-02-21T23:05:51.405242+0000","last_active":"2024-02-21T23:06:11.972413+0000","last_peered":"2024-02-21T23:06:11.972413+0000","last_clean":"2024-02-21T23:06:11.972413+0000","last_became_active":"2024-02-21T23:05:51.403777+0000","last_became_peered":"2024-02-21T23:05:51.403777+0000","last_unstale":"2024-02-21T23:06:11.972413+0000","last_undegraded":"2024-02-21T23:06:11.972413+0000","last_fullsized":"2024-02-21T23:06:11.972413+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:34:14.420091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"75'6","reported_seq":28,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.036075+0000","last_change":"2024-02-21T23:05:50.374208+0000","last_active":"2024-02-21T23:06:13.036075+0000","last_peered":"2024-02-21T23:06:13.036075+0000","last_clean":"2024-02-21T23:06:13.036075+0000","last_became_active":"2024-02-21T23:05:50.373869+0000","last_became_peered":"2024-02-21T23:05:50.373869+0000","last_unstale":"2024-02-21T23:06:13.036075+0000","last_undegraded":"2024-02-21T23:06:13.036075+0000","last_fullsized":"2024-02-21T23:06:13.036075+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:56:49.435540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.978674+0000","last_change":"2024-02-21T23:06:10.960877+0000","last_active":"2024-02-21T23:06:11.978674+0000","last_peered":"2024-02-21T23:06:11.978674+0000","last_clean":"2024-02-21T23:06:11.978674+0000","last_became_active":"2024-02-21T23:06:10.960614+0000","last_became_peered":"2024-02-21T23:06:10.960614+0000","last_unstale":"2024-02-21T23:06:11.978674+0000","last_undegraded":"2024-02-21T23:06:11.978674+0000","last_fullsized":"2024-02-21T23:06:11.978674+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:40:55.366111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.15","version":"79'3","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.978496+0000","last_change":"2024-02-21T23:06:09.963215+0000","last_active":"2024-02-21T23:06:11.978496+0000","last_peered":"2024-02-21T23:06:11.978496+0000","last_clean":"2024-02-21T23:06:11.978496+0000","last_became_active":"2024-02-21T23:06:09.962779+0000","last_became_peered":"2024-02-21T23:06:09.962779+0000","last_unstale":"2024-02-21T23:06:11.978496+0000","last_undegraded":"2024-02-21T23:06:11.978496+0000","last_fullsized":"2024-02-21T23:06:11.978496+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:22:35.988029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.959882+0000","last_change":"2024-02-21T23:05:51.423479+0000","last_active":"2024-02-21T23:06:12.959882+0000","last_peered":"2024-02-21T23:06:12.959882+0000","last_clean":"2024-02-21T23:06:12.959882+0000","last_became_active":"2024-02-21T23:05:51.423190+0000","last_became_peered":"2024-02-21T23:05:51.423190+0000","last_unstale":"2024-02-21T23:06:12.959882+0000","last_undegraded":"2024-02-21T23:06:12.959882+0000","last_fullsized":"2024-02-21T23:06:12.959882+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:03:08.789066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"75'2","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.958906+0000","last_change":"2024-02-21T23:05:50.381643+0000","last_active":"2024-02-21T23:06:12.958906+0000","last_peered":"2024-02-21T23:06:12.958906+0000","last_clean":"2024-02-21T23:06:12.958906+0000","last_became_active":"2024-02-21T23:05:50.381345+0000","last_became_peered":"2024-02-21T23:05:50.381345+0000","last_unstale":"2024-02-21T23:06:12.958906+0000","last_undegraded":"2024-02-21T23:06:12.958906+0000","last_fullsized":"2024-02-21T23:06:12.958906+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:01:27.369089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.972762+0000","last_change":"2024-02-21T23:06:10.987880+0000","last_active":"2024-02-21T23:06:11.972762+0000","last_peered":"2024-02-21T23:06:11.972762+0000","last_clean":"2024-02-21T23:06:11.972762+0000","last_became_active":"2024-02-21T23:06:10.987579+0000","last_became_peered":"2024-02-21T23:06:10.987579+0000","last_unstale":"2024-02-21T23:06:11.972762+0000","last_undegraded":"2024-02-21T23:06:11.972762+0000","last_fullsized":"2024-02-21T23:06:11.972762+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:03:54.817384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"79'6","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.030037+0000","last_change":"2024-02-21T23:06:09.959967+0000","last_active":"2024-02-21T23:06:13.030037+0000","last_peered":"2024-02-21T23:06:13.030037+0000","last_clean":"2024-02-21T23:06:13.030037+0000","last_became_active":"2024-02-21T23:06:09.959175+0000","last_became_peered":"2024-02-21T23:06:09.959175+0000","last_unstale":"2024-02-21T23:06:13.030037+0000","last_undegraded":"2024-02-21T23:06:13.030037+0000","last_fullsized":"2024-02-21T23:06:13.030037+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:00:41.536878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.035432+0000","last_change":"2024-02-21T23:05:51.416753+0000","last_active":"2024-02-21T23:06:13.035432+0000","last_peered":"2024-02-21T23:06:13.035432+0000","last_clean":"2024-02-21T23:06:13.035432+0000","last_became_active":"2024-02-21T23:05:51.416427+0000","last_became_peered":"2024-02-21T23:05:51.416427+0000","last_unstale":"2024-02-21T23:06:13.035432+0000","last_undegraded":"2024-02-21T23:06:13.035432+0000","last_fullsized":"2024-02-21T23:06:13.035432+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:34:14.420091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.033151+0000","last_change":"2024-02-21T23:05:50.381962+0000","last_active":"2024-02-21T23:06:13.033151+0000","last_peered":"2024-02-21T23:06:13.033151+0000","last_clean":"2024-02-21T23:06:13.033151+0000","last_became_active":"2024-02-21T23:05:50.380838+0000","last_became_peered":"2024-02-21T23:05:50.380838+0000","last_unstale":"2024-02-21T23:06:13.033151+0000","last_undegraded":"2024-02-21T23:06:13.033151+0000","last_fullsized":"2024-02-21T23:06:13.033151+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:10:10.973567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.976746+0000","last_change":"2024-02-21T23:06:10.974053+0000","last_active":"2024-02-21T23:06:11.976746+0000","last_peered":"2024-02-21T23:06:11.976746+0000","last_clean":"2024-02-21T23:06:11.976746+0000","last_became_active":"2024-02-21T23:06:10.973635+0000","last_became_peered":"2024-02-21T23:06:10.973635+0000","last_unstale":"2024-02-21T23:06:11.976746+0000","last_undegraded":"2024-02-21T23:06:11.976746+0000","last_fullsized":"2024-02-21T23:06:11.976746+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:01:36.530063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.960404+0000","last_change":"2024-02-21T23:06:09.951230+0000","last_active":"2024-02-21T23:06:12.960404+0000","last_peered":"2024-02-21T23:06:12.960404+0000","last_clean":"2024-02-21T23:06:12.960404+0000","last_became_active":"2024-02-21T23:06:09.950920+0000","last_became_peered":"2024-02-21T23:06:09.950920+0000","last_unstale":"2024-02-21T23:06:12.960404+0000","last_undegraded":"2024-02-21T23:06:12.960404+0000","last_fullsized":"2024-02-21T23:06:12.960404+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:24:51.423593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.036415+0000","last_change":"2024-02-21T23:05:51.408507+0000","last_active":"2024-02-21T23:06:13.036415+0000","last_peered":"2024-02-21T23:06:13.036415+0000","last_clean":"2024-02-21T23:06:13.036415+0000","last_became_active":"2024-02-21T23:05:51.408198+0000","last_became_peered":"2024-02-21T23:05:51.408198+0000","last_unstale":"2024-02-21T23:06:13.036415+0000","last_undegraded":"2024-02-21T23:06:13.036415+0000","last_fullsized":"2024-02-21T23:06:13.036415+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:08:36.461250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.535784+0000","last_change":"2024-02-21T23:05:50.353116+0000","last_active":"2024-02-21T23:06:12.535784+0000","last_peered":"2024-02-21T23:06:12.535784+0000","last_clean":"2024-02-21T23:06:12.535784+0000","last_became_active":"2024-02-21T23:05:50.352768+0000","last_became_peered":"2024-02-21T23:05:50.352768+0000","last_unstale":"2024-02-21T23:06:12.535784+0000","last_undegraded":"2024-02-21T23:06:12.535784+0000","last_fullsized":"2024-02-21T23:06:12.535784+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:52:09.073053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.035220+0000","last_change":"2024-02-21T23:06:10.987816+0000","last_active":"2024-02-21T23:06:13.035220+0000","last_peered":"2024-02-21T23:06:13.035220+0000","last_clean":"2024-02-21T23:06:13.035220+0000","last_became_active":"2024-02-21T23:06:10.987534+0000","last_became_peered":"2024-02-21T23:06:10.987534+0000","last_unstale":"2024-02-21T23:06:13.035220+0000","last_undegraded":"2024-02-21T23:06:13.035220+0000","last_fullsized":"2024-02-21T23:06:13.035220+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:59:19.322366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.978937+0000","last_change":"2024-02-21T23:06:09.954937+0000","last_active":"2024-02-21T23:06:11.978937+0000","last_peered":"2024-02-21T23:06:11.978937+0000","last_clean":"2024-02-21T23:06:11.978937+0000","last_became_active":"2024-02-21T23:06:09.953847+0000","last_became_peered":"2024-02-21T23:06:09.953847+0000","last_unstale":"2024-02-21T23:06:11.978937+0000","last_undegraded":"2024-02-21T23:06:11.978937+0000","last_fullsized":"2024-02-21T23:06:11.978937+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:31:20.996858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.036700+0000","last_change":"2024-02-21T23:05:51.416424+0000","last_active":"2024-02-21T23:06:13.036700+0000","last_peered":"2024-02-21T23:06:13.036700+0000","last_clean":"2024-02-21T23:06:13.036700+0000","last_became_active":"2024-02-21T23:05:51.416070+0000","last_became_peered":"2024-02-21T23:05:51.416070+0000","last_unstale":"2024-02-21T23:06:13.036700+0000","last_undegraded":"2024-02-21T23:06:13.036700+0000","last_fullsized":"2024-02-21T23:06:13.036700+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:56:10.406772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"75'8","reported_seq":30,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.029897+0000","last_change":"2024-02-21T23:05:50.376063+0000","last_active":"2024-02-21T23:06:13.029897+0000","last_peered":"2024-02-21T23:06:13.029897+0000","last_clean":"2024-02-21T23:06:13.029897+0000","last_became_active":"2024-02-21T23:05:50.375772+0000","last_became_peered":"2024-02-21T23:05:50.375772+0000","last_unstale":"2024-02-21T23:06:13.029897+0000","last_undegraded":"2024-02-21T23:06:13.029897+0000","last_fullsized":"2024-02-21T23:06:13.029897+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:26:58.076002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.959287+0000","last_change":"2024-02-21T23:06:10.991210+0000","last_active":"2024-02-21T23:06:12.959287+0000","last_peered":"2024-02-21T23:06:12.959287+0000","last_clean":"2024-02-21T23:06:12.959287+0000","last_became_active":"2024-02-21T23:06:10.990875+0000","last_became_peered":"2024-02-21T23:06:10.990875+0000","last_unstale":"2024-02-21T23:06:12.959287+0000","last_undegraded":"2024-02-21T23:06:12.959287+0000","last_fullsized":"2024-02-21T23:06:12.959287+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:17:15.132571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.958618+0000","last_change":"2024-02-21T23:06:09.951762+0000","last_active":"2024-02-21T23:06:12.958618+0000","last_peered":"2024-02-21T23:06:12.958618+0000","last_clean":"2024-02-21T23:06:12.958618+0000","last_became_active":"2024-02-21T23:06:09.951155+0000","last_became_peered":"2024-02-21T23:06:09.951155+0000","last_unstale":"2024-02-21T23:06:12.958618+0000","last_undegraded":"2024-02-21T23:06:12.958618+0000","last_fullsized":"2024-02-21T23:06:12.958618+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:41:04.005940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.976510+0000","last_change":"2024-02-21T23:05:51.421565+0000","last_active":"2024-02-21T23:06:11.976510+0000","last_peered":"2024-02-21T23:06:11.976510+0000","last_clean":"2024-02-21T23:06:11.976510+0000","last_became_active":"2024-02-21T23:05:51.421252+0000","last_became_peered":"2024-02-21T23:05:51.421252+0000","last_unstale":"2024-02-21T23:06:11.976510+0000","last_undegraded":"2024-02-21T23:06:11.976510+0000","last_fullsized":"2024-02-21T23:06:11.976510+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:40:45.445418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"75'1","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.036242+0000","last_change":"2024-02-21T23:05:50.383778+0000","last_active":"2024-02-21T23:06:13.036242+0000","last_peered":"2024-02-21T23:06:13.036242+0000","last_clean":"2024-02-21T23:06:13.036242+0000","last_became_active":"2024-02-21T23:05:50.383436+0000","last_became_peered":"2024-02-21T23:05:50.383436+0000","last_unstale":"2024-02-21T23:06:13.036242+0000","last_undegraded":"2024-02-21T23:06:13.036242+0000","last_fullsized":"2024-02-21T23:06:13.036242+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:45:05.837402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.030019+0000","last_change":"2024-02-21T23:06:10.965781+0000","last_active":"2024-02-21T23:06:13.030019+0000","last_peered":"2024-02-21T23:06:13.030019+0000","last_clean":"2024-02-21T23:06:13.030019+0000","last_became_active":"2024-02-21T23:06:10.965478+0000","last_became_peered":"2024-02-21T23:06:10.965478+0000","last_unstale":"2024-02-21T23:06:13.030019+0000","last_undegraded":"2024-02-21T23:06:13.030019+0000","last_fullsized":"2024-02-21T23:06:13.030019+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:34:34.740480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.535810+0000","last_change":"2024-02-21T23:06:09.947681+0000","last_active":"2024-02-21T23:06:12.535810+0000","last_peered":"2024-02-21T23:06:12.535810+0000","last_clean":"2024-02-21T23:06:12.535810+0000","last_became_active":"2024-02-21T23:06:09.947370+0000","last_became_peered":"2024-02-21T23:06:09.947370+0000","last_unstale":"2024-02-21T23:06:12.535810+0000","last_undegraded":"2024-02-21T23:06:12.535810+0000","last_fullsized":"2024-02-21T23:06:12.535810+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:16:51.794073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.535873+0000","last_change":"2024-02-21T23:05:51.416676+0000","last_active":"2024-02-21T23:06:12.535873+0000","last_peered":"2024-02-21T23:06:12.535873+0000","last_clean":"2024-02-21T23:06:12.535873+0000","last_became_active":"2024-02-21T23:05:51.416297+0000","last_became_peered":"2024-02-21T23:05:51.416297+0000","last_unstale":"2024-02-21T23:06:12.535873+0000","last_undegraded":"2024-02-21T23:06:12.535873+0000","last_fullsized":"2024-02-21T23:06:12.535873+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:29:33.209545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"75'2","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.979286+0000","last_change":"2024-02-21T23:05:50.377181+0000","last_active":"2024-02-21T23:06:11.979286+0000","last_peered":"2024-02-21T23:06:11.979286+0000","last_clean":"2024-02-21T23:06:11.979286+0000","last_became_active":"2024-02-21T23:05:50.373524+0000","last_became_peered":"2024-02-21T23:05:50.373524+0000","last_unstale":"2024-02-21T23:06:11.979286+0000","last_undegraded":"2024-02-21T23:06:11.979286+0000","last_fullsized":"2024-02-21T23:06:11.979286+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:01:18.419728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.029437+0000","last_change":"2024-02-21T23:06:10.991698+0000","last_active":"2024-02-21T23:06:13.029437+0000","last_peered":"2024-02-21T23:06:13.029437+0000","last_clean":"2024-02-21T23:06:13.029437+0000","last_became_active":"2024-02-21T23:06:10.991386+0000","last_became_peered":"2024-02-21T23:06:10.991386+0000","last_unstale":"2024-02-21T23:06:13.029437+0000","last_undegraded":"2024-02-21T23:06:13.029437+0000","last_fullsized":"2024-02-21T23:06:13.029437+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:41:50.244656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.958876+0000","last_change":"2024-02-21T23:06:09.955696+0000","last_active":"2024-02-21T23:06:12.958876+0000","last_peered":"2024-02-21T23:06:12.958876+0000","last_clean":"2024-02-21T23:06:12.958876+0000","last_became_active":"2024-02-21T23:06:09.954715+0000","last_became_peered":"2024-02-21T23:06:09.954715+0000","last_unstale":"2024-02-21T23:06:12.958876+0000","last_undegraded":"2024-02-21T23:06:12.958876+0000","last_fullsized":"2024-02-21T23:06:12.958876+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:04:21.645034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.536043+0000","last_change":"2024-02-21T23:05:51.412971+0000","last_active":"2024-02-21T23:06:12.536043+0000","last_peered":"2024-02-21T23:06:12.536043+0000","last_clean":"2024-02-21T23:06:12.536043+0000","last_became_active":"2024-02-21T23:05:51.410966+0000","last_became_peered":"2024-02-21T23:05:51.410966+0000","last_unstale":"2024-02-21T23:06:12.536043+0000","last_undegraded":"2024-02-21T23:06:12.536043+0000","last_fullsized":"2024-02-21T23:06:12.536043+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:00:39.163092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"75'1","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.536805+0000","last_change":"2024-02-21T23:05:50.366307+0000","last_active":"2024-02-21T23:06:12.536805+0000","last_peered":"2024-02-21T23:06:12.536805+0000","last_clean":"2024-02-21T23:06:12.536805+0000","last_became_active":"2024-02-21T23:05:50.359426+0000","last_became_peered":"2024-02-21T23:05:50.359426+0000","last_unstale":"2024-02-21T23:06:12.536805+0000","last_undegraded":"2024-02-21T23:06:12.536805+0000","last_fullsized":"2024-02-21T23:06:12.536805+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:13:15.770526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.958835+0000","last_change":"2024-02-21T23:06:10.969120+0000","last_active":"2024-02-21T23:06:12.958835+0000","last_peered":"2024-02-21T23:06:12.958835+0000","last_clean":"2024-02-21T23:06:12.958835+0000","last_became_active":"2024-02-21T23:06:10.968763+0000","last_became_peered":"2024-02-21T23:06:10.968763+0000","last_unstale":"2024-02-21T23:06:12.958835+0000","last_undegraded":"2024-02-21T23:06:12.958835+0000","last_fullsized":"2024-02-21T23:06:12.958835+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:00:42.549792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.535873+0000","last_change":"2024-02-21T23:06:09.948670+0000","last_active":"2024-02-21T23:06:12.535873+0000","last_peered":"2024-02-21T23:06:12.535873+0000","last_clean":"2024-02-21T23:06:12.535873+0000","last_became_active":"2024-02-21T23:06:09.946109+0000","last_became_peered":"2024-02-21T23:06:09.946109+0000","last_unstale":"2024-02-21T23:06:12.535873+0000","last_undegraded":"2024-02-21T23:06:12.535873+0000","last_fullsized":"2024-02-21T23:06:12.535873+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:16:18.547191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.976513+0000","last_change":"2024-02-21T23:05:51.424067+0000","last_active":"2024-02-21T23:06:11.976513+0000","last_peered":"2024-02-21T23:06:11.976513+0000","last_clean":"2024-02-21T23:06:11.976513+0000","last_became_active":"2024-02-21T23:05:51.423644+0000","last_became_peered":"2024-02-21T23:05:51.423644+0000","last_unstale":"2024-02-21T23:06:11.976513+0000","last_undegraded":"2024-02-21T23:06:11.976513+0000","last_fullsized":"2024-02-21T23:06:11.976513+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:04:03.084512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.972107+0000","last_change":"2024-02-21T23:05:50.377924+0000","last_active":"2024-02-21T23:06:11.972107+0000","last_peered":"2024-02-21T23:06:11.972107+0000","last_clean":"2024-02-21T23:06:11.972107+0000","last_became_active":"2024-02-21T23:05:50.377440+0000","last_became_peered":"2024-02-21T23:05:50.377440+0000","last_unstale":"2024-02-21T23:06:11.972107+0000","last_undegraded":"2024-02-21T23:06:11.972107+0000","last_fullsized":"2024-02-21T23:06:11.972107+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:42:15.626763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.979046+0000","last_change":"2024-02-21T23:06:10.981203+0000","last_active":"2024-02-21T23:06:11.979046+0000","last_peered":"2024-02-21T23:06:11.979046+0000","last_clean":"2024-02-21T23:06:11.979046+0000","last_became_active":"2024-02-21T23:06:10.980886+0000","last_became_peered":"2024-02-21T23:06:10.980886+0000","last_unstale":"2024-02-21T23:06:11.979046+0000","last_undegraded":"2024-02-21T23:06:11.979046+0000","last_fullsized":"2024-02-21T23:06:11.979046+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:04:08.953344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.d","version":"79'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.978625+0000","last_change":"2024-02-21T23:06:09.953285+0000","last_active":"2024-02-21T23:06:11.978625+0000","last_peered":"2024-02-21T23:06:11.978625+0000","last_clean":"2024-02-21T23:06:11.978625+0000","last_became_active":"2024-02-21T23:06:09.952940+0000","last_became_peered":"2024-02-21T23:06:09.952940+0000","last_unstale":"2024-02-21T23:06:11.978625+0000","last_undegraded":"2024-02-21T23:06:11.978625+0000","last_fullsized":"2024-02-21T23:06:11.978625+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:53:46.723561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.030184+0000","last_change":"2024-02-21T23:05:51.414619+0000","last_active":"2024-02-21T23:06:13.030184+0000","last_peered":"2024-02-21T23:06:13.030184+0000","last_clean":"2024-02-21T23:06:13.030184+0000","last_became_active":"2024-02-21T23:05:51.414059+0000","last_became_peered":"2024-02-21T23:05:51.414059+0000","last_unstale":"2024-02-21T23:06:13.030184+0000","last_undegraded":"2024-02-21T23:06:13.030184+0000","last_fullsized":"2024-02-21T23:06:13.030184+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:58:05.114297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"75'2","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.976661+0000","last_change":"2024-02-21T23:05:50.366877+0000","last_active":"2024-02-21T23:06:11.976661+0000","last_peered":"2024-02-21T23:06:11.976661+0000","last_clean":"2024-02-21T23:06:11.976661+0000","last_became_active":"2024-02-21T23:05:50.365566+0000","last_became_peered":"2024-02-21T23:05:50.365566+0000","last_unstale":"2024-02-21T23:06:11.976661+0000","last_undegraded":"2024-02-21T23:06:11.976661+0000","last_fullsized":"2024-02-21T23:06:11.976661+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:58:04.101475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.030006+0000","last_change":"2024-02-21T23:06:10.987713+0000","last_active":"2024-02-21T23:06:13.030006+0000","last_peered":"2024-02-21T23:06:13.030006+0000","last_clean":"2024-02-21T23:06:13.030006+0000","last_became_active":"2024-02-21T23:06:10.985949+0000","last_became_peered":"2024-02-21T23:06:10.985949+0000","last_unstale":"2024-02-21T23:06:13.030006+0000","last_undegraded":"2024-02-21T23:06:13.030006+0000","last_fullsized":"2024-02-21T23:06:13.030006+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:16:11.310383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"79'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.036400+0000","last_change":"2024-02-21T23:06:09.945701+0000","last_active":"2024-02-21T23:06:13.036400+0000","last_peered":"2024-02-21T23:06:13.036400+0000","last_clean":"2024-02-21T23:06:13.036400+0000","last_became_active":"2024-02-21T23:06:09.945391+0000","last_became_peered":"2024-02-21T23:06:09.945391+0000","last_unstale":"2024-02-21T23:06:13.036400+0000","last_undegraded":"2024-02-21T23:06:13.036400+0000","last_fullsized":"2024-02-21T23:06:13.036400+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:20:15.771087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.030743+0000","last_change":"2024-02-21T23:05:51.411005+0000","last_active":"2024-02-21T23:06:13.030743+0000","last_peered":"2024-02-21T23:06:13.030743+0000","last_clean":"2024-02-21T23:06:13.030743+0000","last_became_active":"2024-02-21T23:05:51.410503+0000","last_became_peered":"2024-02-21T23:05:51.410503+0000","last_unstale":"2024-02-21T23:06:13.030743+0000","last_undegraded":"2024-02-21T23:06:13.030743+0000","last_fullsized":"2024-02-21T23:06:13.030743+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:51:10.610852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.977826+0000","last_change":"2024-02-21T23:05:50.376888+0000","last_active":"2024-02-21T23:06:11.977826+0000","last_peered":"2024-02-21T23:06:11.977826+0000","last_clean":"2024-02-21T23:06:11.977826+0000","last_became_active":"2024-02-21T23:05:50.376610+0000","last_became_peered":"2024-02-21T23:05:50.376610+0000","last_unstale":"2024-02-21T23:06:11.977826+0000","last_undegraded":"2024-02-21T23:06:11.977826+0000","last_fullsized":"2024-02-21T23:06:11.977826+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:51:16.181060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.035166+0000","last_change":"2024-02-21T23:06:10.981890+0000","last_active":"2024-02-21T23:06:13.035166+0000","last_peered":"2024-02-21T23:06:13.035166+0000","last_clean":"2024-02-21T23:06:13.035166+0000","last_became_active":"2024-02-21T23:06:10.978458+0000","last_became_peered":"2024-02-21T23:06:10.978458+0000","last_unstale":"2024-02-21T23:06:13.035166+0000","last_undegraded":"2024-02-21T23:06:13.035166+0000","last_fullsized":"2024-02-21T23:06:13.035166+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:11:16.456657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.e","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.035094+0000","last_change":"2024-02-21T23:06:09.952555+0000","last_active":"2024-02-21T23:06:13.035094+0000","last_peered":"2024-02-21T23:06:13.035094+0000","last_clean":"2024-02-21T23:06:13.035094+0000","last_became_active":"2024-02-21T23:06:09.952281+0000","last_became_peered":"2024-02-21T23:06:09.952281+0000","last_unstale":"2024-02-21T23:06:13.035094+0000","last_undegraded":"2024-02-21T23:06:13.035094+0000","last_fullsized":"2024-02-21T23:06:13.035094+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:04:21.645034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.036839+0000","last_change":"2024-02-21T23:05:51.417167+0000","last_active":"2024-02-21T23:06:13.036839+0000","last_peered":"2024-02-21T23:06:13.036839+0000","last_clean":"2024-02-21T23:06:13.036839+0000","last_became_active":"2024-02-21T23:05:51.415957+0000","last_became_peered":"2024-02-21T23:05:51.415957+0000","last_unstale":"2024-02-21T23:06:13.036839+0000","last_undegraded":"2024-02-21T23:06:13.036839+0000","last_fullsized":"2024-02-21T23:06:13.036839+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:26:59.088824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.976802+0000","last_change":"2024-02-21T23:05:50.375808+0000","last_active":"2024-02-21T23:06:11.976802+0000","last_peered":"2024-02-21T23:06:11.976802+0000","last_clean":"2024-02-21T23:06:11.976802+0000","last_became_active":"2024-02-21T23:05:50.375467+0000","last_became_peered":"2024-02-21T23:05:50.375467+0000","last_unstale":"2024-02-21T23:06:11.976802+0000","last_undegraded":"2024-02-21T23:06:11.976802+0000","last_fullsized":"2024-02-21T23:06:11.976802+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:29:01.979463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.032733+0000","last_change":"2024-02-21T23:06:10.991982+0000","last_active":"2024-02-21T23:06:13.032733+0000","last_peered":"2024-02-21T23:06:13.032733+0000","last_clean":"2024-02-21T23:06:13.032733+0000","last_became_active":"2024-02-21T23:06:10.991622+0000","last_became_peered":"2024-02-21T23:06:10.991622+0000","last_unstale":"2024-02-21T23:06:13.032733+0000","last_undegraded":"2024-02-21T23:06:13.032733+0000","last_fullsized":"2024-02-21T23:06:13.032733+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:51:30.184287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.032396+0000","last_change":"2024-02-21T23:06:09.956377+0000","last_active":"2024-02-21T23:06:13.032396+0000","last_peered":"2024-02-21T23:06:13.032396+0000","last_clean":"2024-02-21T23:06:13.032396+0000","last_became_active":"2024-02-21T23:06:09.956129+0000","last_became_peered":"2024-02-21T23:06:09.956129+0000","last_unstale":"2024-02-21T23:06:13.032396+0000","last_undegraded":"2024-02-21T23:06:13.032396+0000","last_fullsized":"2024-02-21T23:06:13.032396+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:28:13.256243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.975684+0000","last_change":"2024-02-21T23:05:51.414453+0000","last_active":"2024-02-21T23:06:11.975684+0000","last_peered":"2024-02-21T23:06:11.975684+0000","last_clean":"2024-02-21T23:06:11.975684+0000","last_became_active":"2024-02-21T23:05:51.413499+0000","last_became_peered":"2024-02-21T23:05:51.413499+0000","last_unstale":"2024-02-21T23:06:11.975684+0000","last_undegraded":"2024-02-21T23:06:11.975684+0000","last_fullsized":"2024-02-21T23:06:11.975684+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:55:21.967274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"75'2","reported_seq":24,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.960670+0000","last_change":"2024-02-21T23:05:50.381948+0000","last_active":"2024-02-21T23:06:12.960670+0000","last_peered":"2024-02-21T23:06:12.960670+0000","last_clean":"2024-02-21T23:06:12.960670+0000","last_became_active":"2024-02-21T23:05:50.381452+0000","last_became_peered":"2024-02-21T23:05:50.381452+0000","last_unstale":"2024-02-21T23:06:12.960670+0000","last_undegraded":"2024-02-21T23:06:12.960670+0000","last_fullsized":"2024-02-21T23:06:12.960670+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:22:16.414686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.029844+0000","last_change":"2024-02-21T23:06:10.967217+0000","last_active":"2024-02-21T23:06:13.029844+0000","last_peered":"2024-02-21T23:06:13.029844+0000","last_clean":"2024-02-21T23:06:13.029844+0000","last_became_active":"2024-02-21T23:06:10.966901+0000","last_became_peered":"2024-02-21T23:06:10.966901+0000","last_unstale":"2024-02-21T23:06:13.029844+0000","last_undegraded":"2024-02-21T23:06:13.029844+0000","last_fullsized":"2024-02-21T23:06:13.029844+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:36:21.989353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.033134+0000","last_change":"2024-02-21T23:06:09.953924+0000","last_active":"2024-02-21T23:06:13.033134+0000","last_peered":"2024-02-21T23:06:13.033134+0000","last_clean":"2024-02-21T23:06:13.033134+0000","last_became_active":"2024-02-21T23:06:09.952952+0000","last_became_peered":"2024-02-21T23:06:09.952952+0000","last_unstale":"2024-02-21T23:06:13.033134+0000","last_undegraded":"2024-02-21T23:06:13.033134+0000","last_fullsized":"2024-02-21T23:06:13.033134+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:03:53.804470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.033406+0000","last_change":"2024-02-21T23:05:51.410619+0000","last_active":"2024-02-21T23:06:13.033406+0000","last_peered":"2024-02-21T23:06:13.033406+0000","last_clean":"2024-02-21T23:06:13.033406+0000","last_became_active":"2024-02-21T23:05:51.409937+0000","last_became_peered":"2024-02-21T23:05:51.409937+0000","last_unstale":"2024-02-21T23:06:13.033406+0000","last_undegraded":"2024-02-21T23:06:13.033406+0000","last_fullsized":"2024-02-21T23:06:13.033406+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:22:32.845173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"76'1","reported_seq":23,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.030826+0000","last_change":"2024-02-21T23:05:50.362370+0000","last_active":"2024-02-21T23:06:13.030826+0000","last_peered":"2024-02-21T23:06:13.030826+0000","last_clean":"2024-02-21T23:06:13.030826+0000","last_became_active":"2024-02-21T23:05:50.361848+0000","last_became_peered":"2024-02-21T23:05:50.361848+0000","last_unstale":"2024-02-21T23:06:13.030826+0000","last_undegraded":"2024-02-21T23:06:13.030826+0000","last_fullsized":"2024-02-21T23:06:13.030826+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:08:35.448428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.535691+0000","last_change":"2024-02-21T23:06:10.974560+0000","last_active":"2024-02-21T23:06:12.535691+0000","last_peered":"2024-02-21T23:06:12.535691+0000","last_clean":"2024-02-21T23:06:12.535691+0000","last_became_active":"2024-02-21T23:06:10.974224+0000","last_became_peered":"2024-02-21T23:06:10.974224+0000","last_unstale":"2024-02-21T23:06:12.535691+0000","last_undegraded":"2024-02-21T23:06:12.535691+0000","last_fullsized":"2024-02-21T23:06:12.535691+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:55:41.540709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.029834+0000","last_change":"2024-02-21T23:06:09.961946+0000","last_active":"2024-02-21T23:06:13.029834+0000","last_peered":"2024-02-21T23:06:13.029834+0000","last_clean":"2024-02-21T23:06:13.029834+0000","last_became_active":"2024-02-21T23:06:09.961662+0000","last_became_peered":"2024-02-21T23:06:09.961662+0000","last_unstale":"2024-02-21T23:06:13.029834+0000","last_undegraded":"2024-02-21T23:06:13.029834+0000","last_fullsized":"2024-02-21T23:06:13.029834+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:54:07.357455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.977217+0000","last_change":"2024-02-21T23:05:51.421357+0000","last_active":"2024-02-21T23:06:11.977217+0000","last_peered":"2024-02-21T23:06:11.977217+0000","last_clean":"2024-02-21T23:06:11.977217+0000","last_became_active":"2024-02-21T23:05:51.420734+0000","last_became_peered":"2024-02-21T23:05:51.420734+0000","last_unstale":"2024-02-21T23:06:11.977217+0000","last_undegraded":"2024-02-21T23:06:11.977217+0000","last_fullsized":"2024-02-21T23:06:11.977217+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:31:44.536584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.035689+0000","last_change":"2024-02-21T23:05:50.384680+0000","last_active":"2024-02-21T23:06:13.035689+0000","last_peered":"2024-02-21T23:06:13.035689+0000","last_clean":"2024-02-21T23:06:13.035689+0000","last_became_active":"2024-02-21T23:05:50.384208+0000","last_became_peered":"2024-02-21T23:05:50.384208+0000","last_unstale":"2024-02-21T23:06:13.035689+0000","last_undegraded":"2024-02-21T23:06:13.035689+0000","last_fullsized":"2024-02-21T23:06:13.035689+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:01:27.369089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.977561+0000","last_change":"2024-02-21T23:06:10.963711+0000","last_active":"2024-02-21T23:06:11.977561+0000","last_peered":"2024-02-21T23:06:11.977561+0000","last_clean":"2024-02-21T23:06:11.977561+0000","last_became_active":"2024-02-21T23:06:10.963131+0000","last_became_peered":"2024-02-21T23:06:10.963131+0000","last_unstale":"2024-02-21T23:06:11.977561+0000","last_undegraded":"2024-02-21T23:06:11.977561+0000","last_fullsized":"2024-02-21T23:06:11.977561+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:35:48.816255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.958788+0000","last_change":"2024-02-21T23:06:09.952597+0000","last_active":"2024-02-21T23:06:12.958788+0000","last_peered":"2024-02-21T23:06:12.958788+0000","last_clean":"2024-02-21T23:06:12.958788+0000","last_became_active":"2024-02-21T23:06:09.952270+0000","last_became_peered":"2024-02-21T23:06:09.952270+0000","last_unstale":"2024-02-21T23:06:12.958788+0000","last_undegraded":"2024-02-21T23:06:12.958788+0000","last_fullsized":"2024-02-21T23:06:12.958788+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:03:53.804470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.030138+0000","last_change":"2024-02-21T23:05:51.395859+0000","last_active":"2024-02-21T23:06:13.030138+0000","last_peered":"2024-02-21T23:06:13.030138+0000","last_clean":"2024-02-21T23:06:13.030138+0000","last_became_active":"2024-02-21T23:05:51.395012+0000","last_became_peered":"2024-02-21T23:05:51.395012+0000","last_unstale":"2024-02-21T23:06:13.030138+0000","last_undegraded":"2024-02-21T23:06:13.030138+0000","last_fullsized":"2024-02-21T23:06:13.030138+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:29:02.992284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.033782+0000","last_change":"2024-02-21T23:05:50.383920+0000","last_active":"2024-02-21T23:06:13.033782+0000","last_peered":"2024-02-21T23:06:13.033782+0000","last_clean":"2024-02-21T23:06:13.033782+0000","last_became_active":"2024-02-21T23:05:50.382488+0000","last_became_peered":"2024-02-21T23:05:50.382488+0000","last_unstale":"2024-02-21T23:06:13.033782+0000","last_undegraded":"2024-02-21T23:06:13.033782+0000","last_fullsized":"2024-02-21T23:06:13.033782+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:26:58.076002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.030523+0000","last_change":"2024-02-21T23:06:10.963734+0000","last_active":"2024-02-21T23:06:13.030523+0000","last_peered":"2024-02-21T23:06:13.030523+0000","last_clean":"2024-02-21T23:06:13.030523+0000","last_became_active":"2024-02-21T23:06:10.963441+0000","last_became_peered":"2024-02-21T23:06:10.963441+0000","last_unstale":"2024-02-21T23:06:13.030523+0000","last_undegraded":"2024-02-21T23:06:13.030523+0000","last_fullsized":"2024-02-21T23:06:13.030523+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:51:44.907006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.978786+0000","last_change":"2024-02-21T23:06:09.961904+0000","last_active":"2024-02-21T23:06:11.978786+0000","last_peered":"2024-02-21T23:06:11.978786+0000","last_clean":"2024-02-21T23:06:11.978786+0000","last_became_active":"2024-02-21T23:06:09.961650+0000","last_became_peered":"2024-02-21T23:06:09.961650+0000","last_unstale":"2024-02-21T23:06:11.978786+0000","last_undegraded":"2024-02-21T23:06:11.978786+0000","last_fullsized":"2024-02-21T23:06:11.978786+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:56:08.310911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.035568+0000","last_change":"2024-02-21T23:05:51.422495+0000","last_active":"2024-02-21T23:06:13.035568+0000","last_peered":"2024-02-21T23:06:13.035568+0000","last_clean":"2024-02-21T23:06:13.035568+0000","last_became_active":"2024-02-21T23:05:51.422196+0000","last_became_peered":"2024-02-21T23:05:51.422196+0000","last_unstale":"2024-02-21T23:06:13.035568+0000","last_undegraded":"2024-02-21T23:06:13.035568+0000","last_fullsized":"2024-02-21T23:06:13.035568+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:10:11.986389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.034279+0000","last_change":"2024-02-21T23:05:50.376027+0000","last_active":"2024-02-21T23:06:13.034279+0000","last_peered":"2024-02-21T23:06:13.034279+0000","last_clean":"2024-02-21T23:06:13.034279+0000","last_became_active":"2024-02-21T23:05:50.375170+0000","last_became_peered":"2024-02-21T23:05:50.375170+0000","last_unstale":"2024-02-21T23:06:13.034279+0000","last_undegraded":"2024-02-21T23:06:13.034279+0000","last_fullsized":"2024-02-21T23:06:13.034279+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:39:32.228635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.960474+0000","last_change":"2024-02-21T23:06:10.987733+0000","last_active":"2024-02-21T23:06:12.960474+0000","last_peered":"2024-02-21T23:06:12.960474+0000","last_clean":"2024-02-21T23:06:12.960474+0000","last_became_active":"2024-02-21T23:06:10.987434+0000","last_became_peered":"2024-02-21T23:06:10.987434+0000","last_unstale":"2024-02-21T23:06:12.960474+0000","last_undegraded":"2024-02-21T23:06:12.960474+0000","last_fullsized":"2024-02-21T23:06:12.960474+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:31:22.009771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.035325+0000","last_change":"2024-02-21T23:06:09.947879+0000","last_active":"2024-02-21T23:06:13.035325+0000","last_peered":"2024-02-21T23:06:13.035325+0000","last_clean":"2024-02-21T23:06:13.035325+0000","last_became_active":"2024-02-21T23:06:09.947599+0000","last_became_peered":"2024-02-21T23:06:09.947599+0000","last_unstale":"2024-02-21T23:06:13.035325+0000","last_undegraded":"2024-02-21T23:06:13.035325+0000","last_fullsized":"2024-02-21T23:06:13.035325+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:17:44.890131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.035705+0000","last_change":"2024-02-21T23:05:51.420500+0000","last_active":"2024-02-21T23:06:13.035705+0000","last_peered":"2024-02-21T23:06:13.035705+0000","last_clean":"2024-02-21T23:06:13.035705+0000","last_became_active":"2024-02-21T23:05:51.420103+0000","last_became_peered":"2024-02-21T23:05:51.420103+0000","last_unstale":"2024-02-21T23:06:13.035705+0000","last_undegraded":"2024-02-21T23:06:13.035705+0000","last_fullsized":"2024-02-21T23:06:13.035705+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:27:01.191337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:12.536099+0000","last_change":"2024-02-21T23:05:50.354805+0000","last_active":"2024-02-21T23:06:12.536099+0000","last_peered":"2024-02-21T23:06:12.536099+0000","last_clean":"2024-02-21T23:06:12.536099+0000","last_became_active":"2024-02-21T23:05:50.354525+0000","last_became_peered":"2024-02-21T23:05:50.354525+0000","last_unstale":"2024-02-21T23:06:12.536099+0000","last_undegraded":"2024-02-21T23:06:12.536099+0000","last_fullsized":"2024-02-21T23:06:12.536099+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:28:05.314552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.035802+0000","last_change":"2024-02-21T23:06:10.978267+0000","last_active":"2024-02-21T23:06:13.035802+0000","last_peered":"2024-02-21T23:06:13.035802+0000","last_clean":"2024-02-21T23:06:13.035802+0000","last_became_active":"2024-02-21T23:06:10.977902+0000","last_became_peered":"2024-02-21T23:06:10.977902+0000","last_unstale":"2024-02-21T23:06:13.035802+0000","last_undegraded":"2024-02-21T23:06:13.035802+0000","last_fullsized":"2024-02-21T23:06:13.035802+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:36:43.365830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.7","version":"79'2","reported_seq":16,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.034181+0000","last_change":"2024-02-21T23:06:09.955164+0000","last_active":"2024-02-21T23:06:13.034181+0000","last_peered":"2024-02-21T23:06:13.034181+0000","last_clean":"2024-02-21T23:06:13.034181+0000","last_became_active":"2024-02-21T23:06:09.954858+0000","last_became_peered":"2024-02-21T23:06:09.954858+0000","last_unstale":"2024-02-21T23:06:13.034181+0000","last_undegraded":"2024-02-21T23:06:13.034181+0000","last_fullsized":"2024-02-21T23:06:13.034181+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:59:31.853016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.971935+0000","last_change":"2024-02-21T23:05:51.416548+0000","last_active":"2024-02-21T23:06:11.971935+0000","last_peered":"2024-02-21T23:06:11.971935+0000","last_clean":"2024-02-21T23:06:11.971935+0000","last_became_active":"2024-02-21T23:05:51.416222+0000","last_became_peered":"2024-02-21T23:05:51.416222+0000","last_unstale":"2024-02-21T23:06:11.971935+0000","last_undegraded":"2024-02-21T23:06:11.971935+0000","last_fullsized":"2024-02-21T23:06:11.971935+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:22:32.845173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.034239+0000","last_change":"2024-02-21T23:05:50.373554+0000","last_active":"2024-02-21T23:06:13.034239+0000","last_peered":"2024-02-21T23:06:13.034239+0000","last_clean":"2024-02-21T23:06:13.034239+0000","last_became_active":"2024-02-21T23:05:50.373022+0000","last_became_peered":"2024-02-21T23:05:50.373022+0000","last_unstale":"2024-02-21T23:06:13.034239+0000","last_undegraded":"2024-02-21T23:06:13.034239+0000","last_fullsized":"2024-02-21T23:06:13.034239+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:38:56.665723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.030396+0000","last_change":"2024-02-21T23:06:10.983730+0000","last_active":"2024-02-21T23:06:13.030396+0000","last_peered":"2024-02-21T23:06:13.030396+0000","last_clean":"2024-02-21T23:06:13.030396+0000","last_became_active":"2024-02-21T23:06:10.983405+0000","last_became_peered":"2024-02-21T23:06:10.983405+0000","last_unstale":"2024-02-21T23:06:13.030396+0000","last_undegraded":"2024-02-21T23:06:13.030396+0000","last_fullsized":"2024-02-21T23:06:13.030396+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:09:34.269512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"80'1","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.978594+0000","last_change":"2024-02-21T23:06:09.953218+0000","last_active":"2024-02-21T23:06:11.978594+0000","last_peered":"2024-02-21T23:06:11.978594+0000","last_clean":"2024-02-21T23:06:11.978594+0000","last_became_active":"2024-02-21T23:06:09.952908+0000","last_became_peered":"2024-02-21T23:06:09.952908+0000","last_unstale":"2024-02-21T23:06:11.978594+0000","last_undegraded":"2024-02-21T23:06:11.978594+0000","last_fullsized":"2024-02-21T23:06:11.978594+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:51:46.950044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.972781+0000","last_change":"2024-02-21T23:05:51.412683+0000","last_active":"2024-02-21T23:06:11.972781+0000","last_peered":"2024-02-21T23:06:11.972781+0000","last_clean":"2024-02-21T23:06:11.972781+0000","last_became_active":"2024-02-21T23:05:51.412389+0000","last_became_peered":"2024-02-21T23:05:51.412389+0000","last_unstale":"2024-02-21T23:06:11.972781+0000","last_undegraded":"2024-02-21T23:06:11.972781+0000","last_fullsized":"2024-02-21T23:06:11.972781+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:57:06.211024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.972069+0000","last_change":"2024-02-21T23:05:50.376356+0000","last_active":"2024-02-21T23:06:11.972069+0000","last_peered":"2024-02-21T23:06:11.972069+0000","last_clean":"2024-02-21T23:06:11.972069+0000","last_became_active":"2024-02-21T23:05:50.370467+0000","last_became_peered":"2024-02-21T23:05:50.370467+0000","last_unstale":"2024-02-21T23:06:11.972069+0000","last_undegraded":"2024-02-21T23:06:11.972069+0000","last_fullsized":"2024-02-21T23:06:11.972069+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:56:49.435540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.033889+0000","last_change":"2024-02-21T23:06:10.985628+0000","last_active":"2024-02-21T23:06:13.033889+0000","last_peered":"2024-02-21T23:06:13.033889+0000","last_clean":"2024-02-21T23:06:13.033889+0000","last_became_active":"2024-02-21T23:06:10.985322+0000","last_became_peered":"2024-02-21T23:06:10.985322+0000","last_unstale":"2024-02-21T23:06:13.033889+0000","last_undegraded":"2024-02-21T23:06:13.033889+0000","last_fullsized":"2024-02-21T23:06:13.033889+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:09.906800+0000","last_clean_scrub_stamp":"2024-02-21T23:06:09.906800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:37:11.082403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.971838+0000","last_change":"2024-02-21T23:06:09.940641+0000","last_active":"2024-02-21T23:06:11.971838+0000","last_peered":"2024-02-21T23:06:11.971838+0000","last_clean":"2024-02-21T23:06:11.971838+0000","last_became_active":"2024-02-21T23:06:09.940353+0000","last_became_peered":"2024-02-21T23:06:09.940353+0000","last_unstale":"2024-02-21T23:06:11.971838+0000","last_undegraded":"2024-02-21T23:06:11.971838+0000","last_fullsized":"2024-02-21T23:06:11.971838+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:08.893887+0000","last_clean_scrub_stamp":"2024-02-21T23:06:08.893887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:29:15.656090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":20,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.035455+0000","last_change":"2024-02-21T23:05:51.413008+0000","last_active":"2024-02-21T23:06:13.035455+0000","last_peered":"2024-02-21T23:06:13.035455+0000","last_clean":"2024-02-21T23:06:13.035455+0000","last_became_active":"2024-02-21T23:05:51.412714+0000","last_became_peered":"2024-02-21T23:05:51.412714+0000","last_unstale":"2024-02-21T23:06:13.035455+0000","last_undegraded":"2024-02-21T23:06:13.035455+0000","last_fullsized":"2024-02-21T23:06:13.035455+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:50.333365+0000","last_clean_scrub_stamp":"2024-02-21T23:05:50.333365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:03:08.789066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":22,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:11.976840+0000","last_change":"2024-02-21T23:05:50.375581+0000","last_active":"2024-02-21T23:06:11.976840+0000","last_peered":"2024-02-21T23:06:11.976840+0000","last_clean":"2024-02-21T23:06:11.976840+0000","last_became_active":"2024-02-21T23:05:50.375282+0000","last_became_peered":"2024-02-21T23:05:50.375282+0000","last_unstale":"2024-02-21T23:06:11.976840+0000","last_undegraded":"2024-02-21T23:06:11.976840+0000","last_fullsized":"2024-02-21T23:06:11.976840+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:05:49.320544+0000","last_clean_scrub_stamp":"2024-02-21T23:05:49.320544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:49:45.012428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":282,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-02-21T23:06:13.034669+0000","last_change":"2024-02-21T23:02:32.289335+0000","last_active":"2024-02-21T23:06:13.034669+0000","last_peered":"2024-02-21T23:06:13.034669+0000","last_clean":"2024-02-21T23:06:13.034669+0000","last_became_active":"2024-02-21T23:02:32.286913+0000","last_became_peered":"2024-02-21T23:02:32.286913+0000","last_unstale":"2024-02-21T23:06:13.034669+0000","last_undegraded":"2024-02-21T23:06:13.034669+0000","last_fullsized":"2024-02-21T23:06:13.034669+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_clean_scrub_stamp":"2024-02-21T23:02:31.172569+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:36:23.987374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":14,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574895,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32212,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339628,"statfs":{"total":96636764160,"available":96603779072,"internally_reserved":0,"allocated":1056768,"data_stored":1064123,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574896,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32168,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339672,"statfs":{"total":96636764160,"available":96603824128,"internally_reserved":0,"allocated":1011712,"data_stored":477505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574895,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27984,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343856,"statfs":{"total":96636764160,"available":96608108544,"internally_reserved":0,"allocated":983040,"data_stored":473711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574896,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32168,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339672,"statfs":{"total":96636764160,"available":96603824128,"internally_reserved":0,"allocated":1007616,"data_stored":475553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574896,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32144,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339696,"statfs":{"total":96636764160,"available":96603848704,"internally_reserved":0,"allocated":983040,"data_stored":474259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574897,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32152,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339688,"statfs":{"total":96636764160,"available":96603840512,"internally_reserved":0,"allocated":991232,"data_stored":474419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574895,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32152,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339688,"statfs":{"total":96636764160,"available":96603840512,"internally_reserved":0,"allocated":991232,"data_stored":473859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574895,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32224,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339616,"statfs":{"total":96636764160,"available":96603766784,"internally_reserved":0,"allocated":1069056,"data_stored":1066363,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-21T23:06:21.403 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-02-21T23:06:21.403 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs refuse_standby_for_another_fs true 2024-02-21T23:06:21.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.614+0000 7f003ae00640 1 Processor -- start 2024-02-21T23:06:21.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.615+0000 7f003ae00640 1 -- start start 2024-02-21T23:06:21.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.617+0000 7f003ae00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f003414f000 0x7f003406ec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:21.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.617+0000 7f003ae00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f003406f160 0x7f00341556a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:21.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.617+0000 7f003ae00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00340c5b80 con 0x7f00340c5df0 2024-02-21T23:06:21.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.617+0000 7f003ae00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00340c4fe0 con 0x7f003414f000 2024-02-21T23:06:21.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.617+0000 7f003ae00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00340c5650 con 0x7f003406f160 2024-02-21T23:06:21.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.617+0000 7f003a400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f00340c5df0 0x7f003406ae70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42542/0 (socket says 172.21.15.28:42542) 2024-02-21T23:06:21.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.617+0000 7f003a400640 1 -- 172.21.15.28:0/997287831 learned_addr learned my addr 172.21.15.28:0/997287831 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:06:21.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.617+0000 7f0039000640 1 --2- 172.21.15.28:0/997287831 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f003406f160 0x7f00341556a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:21.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.617+0000 7f0039a00640 1 --2- 172.21.15.28:0/997287831 >> v2:172.21.15.81:3300/0 conn(0x7f003414f000 0x7f003406ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:21.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.618+0000 7f0033e00640 1 -- 172.21.15.28:0/997287831 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4124597406 0 0) 0x7f00340c5b80 con 0x7f00340c5df0 2024-02-21T23:06:21.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.618+0000 7f0033e00640 1 -- 172.21.15.28:0/997287831 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f001c003520 con 0x7f00340c5df0 2024-02-21T23:06:21.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.618+0000 7f0033e00640 1 -- 172.21.15.28:0/997287831 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 309513959 0 0) 0x7f001c003520 con 0x7f00340c5df0 2024-02-21T23:06:21.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.618+0000 7f0033e00640 1 -- 172.21.15.28:0/997287831 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f003406f160 msgr2=0x7f00341556a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:21.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.618+0000 7f0033e00640 1 --2- 172.21.15.28:0/997287831 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f003406f160 0x7f00341556a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:21.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.618+0000 7f0033e00640 1 -- 172.21.15.28:0/997287831 >> v2:172.21.15.81:3300/0 conn(0x7f003414f000 msgr2=0x7f003406ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:21.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.618+0000 7f0033e00640 1 --2- 172.21.15.28:0/997287831 >> v2:172.21.15.81:3300/0 conn(0x7f003414f000 0x7f003406ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:21.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.618+0000 7f0033e00640 1 -- 172.21.15.28:0/997287831 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0034152950 con 0x7f00340c5df0 2024-02-21T23:06:21.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.618+0000 7f0033e00640 1 -- 172.21.15.28:0/997287831 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f002c0019f0 con 0x7f00340c5df0 2024-02-21T23:06:21.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.618+0000 7f0033e00640 1 -- 172.21.15.28:0/997287831 >> v1:172.21.15.28:6789/0 conn(0x7f00340c5df0 legacy=0x7f003406ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:21.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.618+0000 7f0033e00640 1 --2- 172.21.15.28:0/997287831 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f003406f160 0x7f001c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:21.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.618+0000 7f0033e00640 1 --2- 172.21.15.28:0/997287831 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f001c003f30 0x7f001c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:21.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.618+0000 7f0033e00640 1 --2- 172.21.15.28:0/997287831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f001c004d40 0x7f001c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:21.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.619+0000 7f0033e00640 1 -- 172.21.15.28:0/997287831 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00340c5650 con 0x7f001c004d40 2024-02-21T23:06:21.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.619+0000 7f0033e00640 1 -- 172.21.15.28:0/997287831 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f001c00b9c0 con 0x7f001c003f30 2024-02-21T23:06:21.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.619+0000 7f0033e00640 1 -- 172.21.15.28:0/997287831 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f001c00bb40 con 0x7f003406f160 2024-02-21T23:06:21.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.619+0000 7f003a400640 1 --2- 172.21.15.28:0/997287831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f001c004d40 0x7f001c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:21.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.619+0000 7f0039a00640 1 --2- 172.21.15.28:0/997287831 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f003406f160 0x7f001c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:21.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.619+0000 7f0039000640 1 --2- 172.21.15.28:0/997287831 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f001c003f30 0x7f001c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:21.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.619+0000 7f003a400640 1 -- 172.21.15.28:0/997287831 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f003406f160 msgr2=0x7f001c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:21.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.619+0000 7f003a400640 1 --2- 172.21.15.28:0/997287831 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f003406f160 0x7f001c0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:21.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.619+0000 7f003a400640 1 -- 172.21.15.28:0/997287831 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f001c003f30 msgr2=0x7f001c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:21.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.619+0000 7f003a400640 1 --2- 172.21.15.28:0/997287831 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f001c003f30 0x7f001c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:21.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.619+0000 7f003a400640 1 -- 172.21.15.28:0/997287831 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f001c003520 con 0x7f001c004d40 2024-02-21T23:06:21.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.619+0000 7f0039a00640 1 --2- 172.21.15.28:0/997287831 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f003406f160 0x7f001c0039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:06:21.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.620+0000 7f003a400640 1 --2- 172.21.15.28:0/997287831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f001c004d40 0x7f001c0051c0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f002c00a920 tx=0x7f001c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=799b0f2bff97abb3 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:21.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.621+0000 7f0033e00640 1 -- 172.21.15.28:0/997287831 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f002c00a4c0 con 0x7f001c004d40 2024-02-21T23:06:21.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.621+0000 7f0033e00640 1 -- 172.21.15.28:0/997287831 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f002c034650 con 0x7f001c004d40 2024-02-21T23:06:21.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.621+0000 7f0033e00640 1 -- 172.21.15.28:0/997287831 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f002c034a80 con 0x7f001c004d40 2024-02-21T23:06:21.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.621+0000 7f003ae00640 1 -- 172.21.15.28:0/997287831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f001c004d40 msgr2=0x7f001c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:21.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.621+0000 7f003ae00640 1 --2- 172.21.15.28:0/997287831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f001c004d40 0x7f001c0051c0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f002c00a920 tx=0x7f001c005830 comp rx=0 tx=0).stop 2024-02-21T23:06:21.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.621+0000 7f003a400640 1 -- 172.21.15.28:0/997287831 reap_dead start 2024-02-21T23:06:21.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.621+0000 7f003ae00640 1 -- 172.21.15.28:0/997287831 shutdown_connections 2024-02-21T23:06:21.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.621+0000 7f003ae00640 1 -- 172.21.15.28:0/997287831 >> 172.21.15.28:0/997287831 conn(0x7f00340a8530 msgr2=0x7f00340a8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:21.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.622+0000 7f003ae00640 1 -- 172.21.15.28:0/997287831 shutdown_connections 2024-02-21T23:06:21.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.622+0000 7f003ae00640 1 -- 172.21.15.28:0/997287831 wait complete. 2024-02-21T23:06:21.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.624+0000 7f003ae00640 1 Processor -- start 2024-02-21T23:06:21.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.624+0000 7f003ae00640 1 -- start start 2024-02-21T23:06:21.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.626+0000 7f003ae00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0034147d70 0x7f0034148150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:21.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.626+0000 7f003ae00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0034144ff0 0x7f003414cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:21.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.626+0000 7f003ae00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0034154b20 0x7f003414d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:21.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.626+0000 7f003ae00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00340a6170 con 0x7f0034154b20 2024-02-21T23:06:21.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.626+0000 7f003ae00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0034155180 con 0x7f0034147d70 2024-02-21T23:06:21.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.626+0000 7f003ae00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00340a62f0 con 0x7f0034144ff0 2024-02-21T23:06:21.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.626+0000 7f003a400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0034154b20 0x7f003414d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:21.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.626+0000 7f003a400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0034154b20 0x7f003414d2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54704/0 (socket says 172.21.15.28:54704) 2024-02-21T23:06:21.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.626+0000 7f003a400640 1 -- 172.21.15.28:0/2380132419 learned_addr learned my addr 172.21.15.28:0/2380132419 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:21.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.627+0000 7f0039a00640 1 --2- 172.21.15.28:0/2380132419 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0034147d70 0x7f0034148150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:21.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.627+0000 7f0039000640 1 --2- 172.21.15.28:0/2380132419 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0034144ff0 0x7f003414cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:21.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.627+0000 7f003a400640 1 -- 172.21.15.28:0/2380132419 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0034144ff0 msgr2=0x7f003414cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:21.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.627+0000 7f003a400640 1 --2- 172.21.15.28:0/2380132419 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0034144ff0 0x7f003414cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:21.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.627+0000 7f003a400640 1 -- 172.21.15.28:0/2380132419 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0034147d70 msgr2=0x7f0034148150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:21.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.627+0000 7f003a400640 1 --2- 172.21.15.28:0/2380132419 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0034147d70 0x7f0034148150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:21.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.627+0000 7f003a400640 1 -- 172.21.15.28:0/2380132419 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0034144d10 con 0x7f0034154b20 2024-02-21T23:06:21.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.627+0000 7f003a400640 1 --2- 172.21.15.28:0/2380132419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0034154b20 0x7f003414d2b0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f002c00a920 tx=0x7f00340714f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:21.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.628+0000 7f0031600640 1 -- 172.21.15.28:0/2380132419 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f002c00a4c0 con 0x7f0034154b20 2024-02-21T23:06:21.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.628+0000 7f0031600640 1 -- 172.21.15.28:0/2380132419 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f002c045070 con 0x7f0034154b20 2024-02-21T23:06:21.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.628+0000 7f0031600640 1 -- 172.21.15.28:0/2380132419 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f002c002000 con 0x7f0034154b20 2024-02-21T23:06:21.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.628+0000 7f003ae00640 1 -- 172.21.15.28:0/2380132419 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00341ea930 con 0x7f0034154b20 2024-02-21T23:06:21.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.628+0000 7f003ae00640 1 -- 172.21.15.28:0/2380132419 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00341540e0 con 0x7f0034154b20 2024-02-21T23:06:21.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.629+0000 7f003ae00640 1 -- 172.21.15.28:0/2380132419 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f003404f290 con 0x7f0034154b20 2024-02-21T23:06:21.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.631+0000 7f0031600640 1 -- 172.21.15.28:0/2380132419 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f002c034c00 con 0x7f0034154b20 2024-02-21T23:06:21.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.632+0000 7f0031600640 1 --2- 172.21.15.28:0/2380132419 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f001406b4d0 0x7f001406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:21.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.632+0000 7f0031600640 1 -- 172.21.15.28:0/2380132419 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f002c0ac950 con 0x7f0034154b20 2024-02-21T23:06:21.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.633+0000 7f0039a00640 1 --2- 172.21.15.28:0/2380132419 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f001406b4d0 0x7f001406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:21.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.634+0000 7f0039a00640 1 --2- 172.21.15.28:0/2380132419 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f001406b4d0 0x7f001406d990 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f00340c2ca0 tx=0x7f0028008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:21.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:21.641+0000 7f0031600640 1 -- 172.21.15.28:0/2380132419 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f002c032380 con 0x7f0034154b20 2024-02-21T23:06:22.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:22.064+0000 7f003ae00640 1 -- 172.21.15.28:0/2380132419 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "refuse_standby_for_another_fs", "val": "true"} v 0) v1 -- 0x7f0034071180 con 0x7f0034154b20 2024-02-21T23:06:23.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.036+0000 7f0031600640 1 -- 172.21.15.28:0/2380132419 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "refuse_standby_for_another_fs", "val": "true"}]=0 set to refuse standby for another fs v68) v1 ==== 166+0+0 (secure 0 0 0) 0x7f002c07b4a0 con 0x7f0034154b20 2024-02-21T23:06:23.025 INFO:teuthology.orchestra.run.smithi028.stderr:set to refuse standby for another fs 2024-02-21T23:06:23.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.041+0000 7f003ae00640 1 -- 172.21.15.28:0/2380132419 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f001406b4d0 msgr2=0x7f001406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:23.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.041+0000 7f003ae00640 1 --2- 172.21.15.28:0/2380132419 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f001406b4d0 0x7f001406d990 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f00340c2ca0 tx=0x7f0028008040 comp rx=0 tx=0).stop 2024-02-21T23:06:23.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.041+0000 7f003ae00640 1 -- 172.21.15.28:0/2380132419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0034154b20 msgr2=0x7f003414d2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:23.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.041+0000 7f003ae00640 1 --2- 172.21.15.28:0/2380132419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0034154b20 0x7f003414d2b0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f002c00a920 tx=0x7f00340714f0 comp rx=0 tx=0).stop 2024-02-21T23:06:23.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.041+0000 7f003ae00640 1 -- 172.21.15.28:0/2380132419 shutdown_connections 2024-02-21T23:06:23.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.041+0000 7f003ae00640 1 --2- 172.21.15.28:0/2380132419 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f001406b4d0 0x7f001406d990 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:23.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.041+0000 7f003ae00640 1 --2- 172.21.15.28:0/2380132419 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0034144ff0 0x7f003414cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:23.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.041+0000 7f003ae00640 1 --2- 172.21.15.28:0/2380132419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0034154b20 0x7f003414d2b0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:23.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.041+0000 7f003ae00640 1 --2- 172.21.15.28:0/2380132419 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0034147d70 0x7f0034148150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:23.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.041+0000 7f003ae00640 1 -- 172.21.15.28:0/2380132419 >> 172.21.15.28:0/2380132419 conn(0x7f00340a8530 msgr2=0x7f00341555a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:23.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.041+0000 7f003ae00640 1 -- 172.21.15.28:0/2380132419 shutdown_connections 2024-02-21T23:06:23.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.041+0000 7f003ae00640 1 -- 172.21.15.28:0/2380132419 wait complete. 2024-02-21T23:06:23.053 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail 5:0 2024-02-21T23:06:23.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.317+0000 7f416f200640 1 Processor -- start 2024-02-21T23:06:23.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.317+0000 7f416f200640 1 -- start start 2024-02-21T23:06:23.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.319+0000 7f416f200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f41700c6b60 0x7f4170158930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:23.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.319+0000 7f416f200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41701529e0 con 0x7f41700c6190 2024-02-21T23:06:23.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.319+0000 7f416f200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41700c7200 con 0x7f4170152600 2024-02-21T23:06:23.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.319+0000 7f416f200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41700c74e0 con 0x7f41700c6b60 2024-02-21T23:06:23.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.320+0000 7f416d400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f41700c6b60 0x7f4170158930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:23.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.320+0000 7f416e800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f41700c6190 0x7f4170158e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42552/0 (socket says 172.21.15.28:42552) 2024-02-21T23:06:23.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.320+0000 7f416d400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f41700c6b60 0x7f4170158930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54708/0 (socket says 172.21.15.28:54708) 2024-02-21T23:06:23.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.320+0000 7f416d400640 1 -- 172.21.15.28:0/3102231633 learned_addr learned my addr 172.21.15.28:0/3102231633 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:23.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.320+0000 7f416d400640 1 -- 172.21.15.28:0/3102231633 >> v1:172.21.15.81:6789/0 conn(0x7f4170152600 legacy=0x7f41700c0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:23.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.320+0000 7f416d400640 1 -- 172.21.15.28:0/3102231633 >> v1:172.21.15.28:6789/0 conn(0x7f41700c6190 legacy=0x7f4170158e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:23.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.320+0000 7f416d400640 1 -- 172.21.15.28:0/3102231633 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41700c77c0 con 0x7f41700c6b60 2024-02-21T23:06:23.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.322+0000 7f416d400640 1 --2- 172.21.15.28:0/3102231633 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41700c6b60 0x7f4170158930 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f415c009870 tx=0x7f415c02ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86fe58842cd3782b server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:23.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.322+0000 7f416ca00640 1 -- 172.21.15.28:0/3102231633 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f415c02f850 con 0x7f41700c6b60 2024-02-21T23:06:23.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.322+0000 7f416ca00640 1 -- 172.21.15.28:0/3102231633 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f415c02f9f0 con 0x7f41700c6b60 2024-02-21T23:06:23.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.322+0000 7f416ca00640 1 -- 172.21.15.28:0/3102231633 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f415c02fdc0 con 0x7f41700c6b60 2024-02-21T23:06:23.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.323+0000 7f416f200640 1 -- 172.21.15.28:0/3102231633 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41700c6b60 msgr2=0x7f4170158930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:23.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.323+0000 7f416f200640 1 --2- 172.21.15.28:0/3102231633 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41700c6b60 0x7f4170158930 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f415c009870 tx=0x7f415c02ba30 comp rx=0 tx=0).stop 2024-02-21T23:06:23.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.323+0000 7f416f200640 1 -- 172.21.15.28:0/3102231633 shutdown_connections 2024-02-21T23:06:23.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.323+0000 7f416f200640 1 --2- 172.21.15.28:0/3102231633 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41700c6b60 0x7f4170158930 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:23.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.323+0000 7f416f200640 1 -- 172.21.15.28:0/3102231633 >> 172.21.15.28:0/3102231633 conn(0x7f417006d1a0 msgr2=0x7f41700725a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:23.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.323+0000 7f416f200640 1 -- 172.21.15.28:0/3102231633 shutdown_connections 2024-02-21T23:06:23.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.323+0000 7f416f200640 1 -- 172.21.15.28:0/3102231633 wait complete. 2024-02-21T23:06:23.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.325+0000 7f416f200640 1 Processor -- start 2024-02-21T23:06:23.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.325+0000 7f416f200640 1 -- start start 2024-02-21T23:06:23.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.326+0000 7f416f200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f41700c6190 0x7f41701e1f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:23.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.326+0000 7f416f200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f41700c6b60 0x7f4170156760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:23.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.326+0000 7f416f200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4170152600 0x7f41701ead80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:23.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.326+0000 7f416f200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41700c1680 con 0x7f4170152600 2024-02-21T23:06:23.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.326+0000 7f416f200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41700c7aa0 con 0x7f41700c6b60 2024-02-21T23:06:23.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.326+0000 7f416f200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41700c1800 con 0x7f41700c6190 2024-02-21T23:06:23.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.326+0000 7f416e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4170152600 0x7f41701ead80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:23.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.327+0000 7f416e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4170152600 0x7f41701ead80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54718/0 (socket says 172.21.15.28:54718) 2024-02-21T23:06:23.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.327+0000 7f416e800640 1 -- 172.21.15.28:0/2143434702 learned_addr learned my addr 172.21.15.28:0/2143434702 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:23.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.327+0000 7f416d400640 1 --2- 172.21.15.28:0/2143434702 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f41700c6b60 0x7f4170156760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:23.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.327+0000 7f416de00640 1 --2- 172.21.15.28:0/2143434702 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f41700c6190 0x7f41701e1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:23.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.327+0000 7f416e800640 1 -- 172.21.15.28:0/2143434702 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f41700c6190 msgr2=0x7f41701e1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:23.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.327+0000 7f416e800640 1 --2- 172.21.15.28:0/2143434702 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f41700c6190 0x7f41701e1f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:23.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.327+0000 7f416e800640 1 -- 172.21.15.28:0/2143434702 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f41700c6b60 msgr2=0x7f4170156760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:23.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.327+0000 7f416e800640 1 --2- 172.21.15.28:0/2143434702 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f41700c6b60 0x7f4170156760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:23.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.327+0000 7f416e800640 1 -- 172.21.15.28:0/2143434702 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41700c9ed0 con 0x7f4170152600 2024-02-21T23:06:23.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.327+0000 7f416e800640 1 --2- 172.21.15.28:0/2143434702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4170152600 0x7f41701ead80 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f416800c7b0 tx=0x7f416800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:23.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.328+0000 7f4166000640 1 -- 172.21.15.28:0/2143434702 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4168017070 con 0x7f4170152600 2024-02-21T23:06:23.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.328+0000 7f4166000640 1 -- 172.21.15.28:0/2143434702 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f416800a640 con 0x7f4170152600 2024-02-21T23:06:23.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.328+0000 7f4166000640 1 -- 172.21.15.28:0/2143434702 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f416800a9f0 con 0x7f4170152600 2024-02-21T23:06:23.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.328+0000 7f416f200640 1 -- 172.21.15.28:0/2143434702 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4170154cf0 con 0x7f4170152600 2024-02-21T23:06:23.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.328+0000 7f416f200640 1 -- 172.21.15.28:0/2143434702 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41701ef030 con 0x7f4170152600 2024-02-21T23:06:23.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.329+0000 7f416f200640 1 -- 172.21.15.28:0/2143434702 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f417004f290 con 0x7f4170152600 2024-02-21T23:06:23.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.331+0000 7f4166000640 1 -- 172.21.15.28:0/2143434702 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f416801a070 con 0x7f4170152600 2024-02-21T23:06:23.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.332+0000 7f4166000640 1 --2- 172.21.15.28:0/2143434702 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f415006b520 0x7f415006d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:23.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.332+0000 7f4166000640 1 -- 172.21.15.28:0/2143434702 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f4168089c50 con 0x7f4170152600 2024-02-21T23:06:23.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.333+0000 7f416de00640 1 --2- 172.21.15.28:0/2143434702 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f415006b520 0x7f415006d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:23.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.334+0000 7f416de00640 1 --2- 172.21.15.28:0/2143434702 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f415006b520 0x7f415006d9e0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f41701e2780 tx=0x7f4160009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:23.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.342+0000 7f4166000640 1 -- 172.21.15.28:0/2143434702 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4168057f80 con 0x7f4170152600 2024-02-21T23:06:23.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.689+0000 7f416f200640 1 -- 172.21.15.28:0/2143434702 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "5:0"} v 0) v1 -- 0x7f4170151db0 con 0x7f4170152600 2024-02-21T23:06:23.677 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-02-21T23:06:23.689+0000 7f5ddc400640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-21T23:06:23.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.692+0000 7f4166000640 1 -- 172.21.15.28:0/2143434702 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "5:0"}]=0 failed mds gid 6366 v69) v1 ==== 97+0+0 (secure 0 0 0) 0x7f416805c020 con 0x7f4170152600 2024-02-21T23:06:23.681 INFO:teuthology.orchestra.run.smithi028.stderr:failed mds gid 6366 2024-02-21T23:06:23.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.696+0000 7f416f200640 1 -- 172.21.15.28:0/2143434702 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f415006b520 msgr2=0x7f415006d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:23.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.696+0000 7f416f200640 1 --2- 172.21.15.28:0/2143434702 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f415006b520 0x7f415006d9e0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f41701e2780 tx=0x7f4160009040 comp rx=0 tx=0).stop 2024-02-21T23:06:23.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.696+0000 7f416f200640 1 -- 172.21.15.28:0/2143434702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4170152600 msgr2=0x7f41701ead80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:23.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.696+0000 7f416f200640 1 --2- 172.21.15.28:0/2143434702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4170152600 0x7f41701ead80 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f416800c7b0 tx=0x7f416800cc70 comp rx=0 tx=0).stop 2024-02-21T23:06:23.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.696+0000 7f416f200640 1 -- 172.21.15.28:0/2143434702 shutdown_connections 2024-02-21T23:06:23.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.696+0000 7f416f200640 1 --2- 172.21.15.28:0/2143434702 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f415006b520 0x7f415006d9e0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:23.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.696+0000 7f416f200640 1 --2- 172.21.15.28:0/2143434702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4170152600 0x7f41701ead80 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:23.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.696+0000 7f416f200640 1 --2- 172.21.15.28:0/2143434702 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f41700c6b60 0x7f4170156760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:23.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.696+0000 7f416f200640 1 --2- 172.21.15.28:0/2143434702 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f41700c6190 0x7f41701e1f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:23.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.696+0000 7f416f200640 1 -- 172.21.15.28:0/2143434702 >> 172.21.15.28:0/2143434702 conn(0x7f417006d1a0 msgr2=0x7f41700c06d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:23.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.697+0000 7f416f200640 1 -- 172.21.15.28:0/2143434702 shutdown_connections 2024-02-21T23:06:23.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.699+0000 7f416f200640 1 -- 172.21.15.28:0/2143434702 wait complete. 2024-02-21T23:06:23.693 INFO:tasks.ceph.mds.a.smithi028.stderr: -899> 2024-02-21T23:05:57.330+0000 7ff6f2600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-21T23:06:23.708 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:06:23.750 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-02-21T23:06:23.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.904+0000 7fe366e00640 1 Processor -- start 2024-02-21T23:06:23.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.904+0000 7fe366e00640 1 -- start start 2024-02-21T23:06:23.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.905+0000 7fe366e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe3601525c0 0x7fe3600cb390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:23.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.905+0000 7fe366e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe360155120 con 0x7fe360159010 2024-02-21T23:06:23.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.905+0000 7fe366e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe360155390 con 0x7fe3600c3280 2024-02-21T23:06:23.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.905+0000 7fe366e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe360155670 con 0x7fe3601525c0 2024-02-21T23:06:23.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.906+0000 7fe357e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe360159010 0x7fe3600c6a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42566/0 (socket says 172.21.15.28:42566) 2024-02-21T23:06:23.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.906+0000 7fe357e00640 1 -- 172.21.15.28:0/2381190402 learned_addr learned my addr 172.21.15.28:0/2381190402 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:06:23.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.906+0000 7fe35fe00640 1 --2- 172.21.15.28:0/2381190402 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe3601525c0 0x7fe3600cb390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:23.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.906+0000 7fe35f400640 1 -- 172.21.15.28:0/2381190402 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2661238114 0 0) 0x7fe360155120 con 0x7fe360159010 2024-02-21T23:06:23.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.906+0000 7fe35f400640 1 -- 172.21.15.28:0/2381190402 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe344003520 con 0x7fe360159010 2024-02-21T23:06:23.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.907+0000 7fe35f400640 1 -- 172.21.15.28:0/2381190402 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 230217709 0 0) 0x7fe344003520 con 0x7fe360159010 2024-02-21T23:06:23.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.907+0000 7fe35f400640 1 -- 172.21.15.28:0/2381190402 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe3601525c0 msgr2=0x7fe3600cb390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:23.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.907+0000 7fe35f400640 1 --2- 172.21.15.28:0/2381190402 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe3601525c0 0x7fe3600cb390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:23.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.907+0000 7fe35f400640 1 -- 172.21.15.28:0/2381190402 >> v1:172.21.15.81:6789/0 conn(0x7fe3600c3280 legacy=0x7fe3600c36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:23.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.907+0000 7fe35f400640 1 -- 172.21.15.28:0/2381190402 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe360155950 con 0x7fe360159010 2024-02-21T23:06:23.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.907+0000 7fe35f400640 1 -- 172.21.15.28:0/2381190402 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fe348001670 con 0x7fe360159010 2024-02-21T23:06:23.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.907+0000 7fe35f400640 1 -- 172.21.15.28:0/2381190402 >> v1:172.21.15.28:6789/0 conn(0x7fe360159010 legacy=0x7fe3600c6a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:23.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.907+0000 7fe35f400640 1 --2- 172.21.15.28:0/2381190402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3440039e0 0x7fe344003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:23.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.907+0000 7fe35f400640 1 --2- 172.21.15.28:0/2381190402 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe3601525c0 0x7fe344004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:23.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.907+0000 7fe35f400640 1 --2- 172.21.15.28:0/2381190402 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe344004d40 0x7fe3440051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:23.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.907+0000 7fe35f400640 1 -- 172.21.15.28:0/2381190402 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe360155670 con 0x7fe3440039e0 2024-02-21T23:06:23.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.907+0000 7fe35f400640 1 -- 172.21.15.28:0/2381190402 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe34400baa0 con 0x7fe344004d40 2024-02-21T23:06:23.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.907+0000 7fe35f400640 1 -- 172.21.15.28:0/2381190402 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe34400bca0 con 0x7fe3601525c0 2024-02-21T23:06:23.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.907+0000 7fe35fe00640 1 --2- 172.21.15.28:0/2381190402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3440039e0 0x7fe344003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:23.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.907+0000 7fe364e00640 1 --2- 172.21.15.28:0/2381190402 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe344004d40 0x7fe3440051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:23.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.908+0000 7fe357e00640 1 --2- 172.21.15.28:0/2381190402 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe3601525c0 0x7fe344004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:23.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.908+0000 7fe35fe00640 1 -- 172.21.15.28:0/2381190402 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe3601525c0 msgr2=0x7fe344004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:23.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.908+0000 7fe35fe00640 1 --2- 172.21.15.28:0/2381190402 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe3601525c0 0x7fe344004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:23.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.908+0000 7fe35fe00640 1 -- 172.21.15.28:0/2381190402 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe344004d40 msgr2=0x7fe3440051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:23.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.908+0000 7fe35fe00640 1 --2- 172.21.15.28:0/2381190402 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe344004d40 0x7fe3440051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:23.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.908+0000 7fe35fe00640 1 -- 172.21.15.28:0/2381190402 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe344003520 con 0x7fe3440039e0 2024-02-21T23:06:23.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.908+0000 7fe357e00640 1 --2- 172.21.15.28:0/2381190402 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe3601525c0 0x7fe344004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:06:23.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.908+0000 7fe364e00640 1 --2- 172.21.15.28:0/2381190402 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe344004d40 0x7fe3440051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:06:23.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.909+0000 7fe35fe00640 1 --2- 172.21.15.28:0/2381190402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3440039e0 0x7fe344003e00 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fe350009a90 tx=0x7fe344005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1ab89b843124e55 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:23.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.910+0000 7fe35f400640 1 -- 172.21.15.28:0/2381190402 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe350039070 con 0x7fe3440039e0 2024-02-21T23:06:23.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.910+0000 7fe35f400640 1 -- 172.21.15.28:0/2381190402 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe35002c680 con 0x7fe3440039e0 2024-02-21T23:06:23.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.910+0000 7fe35f400640 1 -- 172.21.15.28:0/2381190402 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe350009690 con 0x7fe3440039e0 2024-02-21T23:06:23.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.910+0000 7fe366e00640 1 -- 172.21.15.28:0/2381190402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3440039e0 msgr2=0x7fe344003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:23.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.910+0000 7fe366e00640 1 --2- 172.21.15.28:0/2381190402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3440039e0 0x7fe344003e00 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fe350009a90 tx=0x7fe344005830 comp rx=0 tx=0).stop 2024-02-21T23:06:23.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.910+0000 7fe364e00640 1 -- 172.21.15.28:0/2381190402 reap_dead start 2024-02-21T23:06:23.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.910+0000 7fe366e00640 1 -- 172.21.15.28:0/2381190402 shutdown_connections 2024-02-21T23:06:23.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.910+0000 7fe366e00640 1 -- 172.21.15.28:0/2381190402 >> 172.21.15.28:0/2381190402 conn(0x7fe3600729c0 msgr2=0x7fe360072dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:23.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.910+0000 7fe366e00640 1 -- 172.21.15.28:0/2381190402 shutdown_connections 2024-02-21T23:06:23.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.910+0000 7fe366e00640 1 -- 172.21.15.28:0/2381190402 wait complete. 2024-02-21T23:06:23.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.912+0000 7fe366e00640 1 Processor -- start 2024-02-21T23:06:23.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.912+0000 7fe366e00640 1 -- start start 2024-02-21T23:06:23.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.914+0000 7fe366e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3440039e0 0x7fe3601ee380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:23.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.914+0000 7fe366e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe36014f050 0x7fe3601584c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:23.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.914+0000 7fe366e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3600ca3c0 0x7fe3600c10e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:23.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.914+0000 7fe366e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3601594a0 con 0x7fe3600ca3c0 2024-02-21T23:06:23.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.914+0000 7fe366e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3600caff0 con 0x7fe3440039e0 2024-02-21T23:06:23.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.914+0000 7fe366e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3600c7110 con 0x7fe36014f050 2024-02-21T23:06:23.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.914+0000 7fe364e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3600ca3c0 0x7fe3600c10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:23.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.914+0000 7fe364e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3600ca3c0 0x7fe3600c10e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54760/0 (socket says 172.21.15.28:54760) 2024-02-21T23:06:23.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.914+0000 7fe364e00640 1 -- 172.21.15.28:0/784847422 learned_addr learned my addr 172.21.15.28:0/784847422 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:23.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.914+0000 7fe357e00640 1 --2- 172.21.15.28:0/784847422 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe36014f050 0x7fe3601584c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:23.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.914+0000 7fe35fe00640 1 --2- 172.21.15.28:0/784847422 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3440039e0 0x7fe3601ee380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:23.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.915+0000 7fe364e00640 1 -- 172.21.15.28:0/784847422 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe36014f050 msgr2=0x7fe3601584c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:23.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.915+0000 7fe364e00640 1 --2- 172.21.15.28:0/784847422 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe36014f050 0x7fe3601584c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:23.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.915+0000 7fe364e00640 1 -- 172.21.15.28:0/784847422 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3440039e0 msgr2=0x7fe3601ee380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:23.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.915+0000 7fe364e00640 1 --2- 172.21.15.28:0/784847422 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3440039e0 0x7fe3601ee380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:23.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.915+0000 7fe364e00640 1 -- 172.21.15.28:0/784847422 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe36014ee00 con 0x7fe3600ca3c0 2024-02-21T23:06:23.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.915+0000 7fe357e00640 1 --2- 172.21.15.28:0/784847422 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe36014f050 0x7fe3601584c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:06:23.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.915+0000 7fe35fe00640 1 --2- 172.21.15.28:0/784847422 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3440039e0 0x7fe3601ee380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:06:23.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.915+0000 7fe364e00640 1 --2- 172.21.15.28:0/784847422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3600ca3c0 0x7fe3600c10e0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fe3600b0350 tx=0x7fe35800d9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:23.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.915+0000 7fe35cc00640 1 -- 172.21.15.28:0/784847422 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe358003970 con 0x7fe3600ca3c0 2024-02-21T23:06:23.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.915+0000 7fe35cc00640 1 -- 172.21.15.28:0/784847422 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe358003b30 con 0x7fe3600ca3c0 2024-02-21T23:06:23.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.915+0000 7fe35cc00640 1 -- 172.21.15.28:0/784847422 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe358005070 con 0x7fe3600ca3c0 2024-02-21T23:06:23.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.915+0000 7fe366e00640 1 -- 172.21.15.28:0/784847422 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe36014eb20 con 0x7fe3600ca3c0 2024-02-21T23:06:23.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.915+0000 7fe366e00640 1 -- 172.21.15.28:0/784847422 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe360147360 con 0x7fe3600ca3c0 2024-02-21T23:06:23.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.916+0000 7fe366e00640 1 -- 172.21.15.28:0/784847422 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe360067f30 con 0x7fe3600ca3c0 2024-02-21T23:06:23.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.917+0000 7fe35cc00640 1 -- 172.21.15.28:0/784847422 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe35801b020 con 0x7fe3600ca3c0 2024-02-21T23:06:23.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.918+0000 7fe35cc00640 1 --2- 172.21.15.28:0/784847422 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe33c06b4d0 0x7fe33c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:23.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.918+0000 7fe35cc00640 1 -- 172.21.15.28:0/784847422 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7043+0+0 (secure 0 0 0) 0x7fe35808a270 con 0x7fe3600ca3c0 2024-02-21T23:06:23.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.919+0000 7fe35fe00640 1 --2- 172.21.15.28:0/784847422 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe33c06b4d0 0x7fe33c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:23.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.920+0000 7fe35fe00640 1 --2- 172.21.15.28:0/784847422 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe33c06b4d0 0x7fe33c06d990 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fe350036000 tx=0x7fe350030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:23.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:23.923+0000 7fe35cc00640 1 -- 172.21.15.28:0/784847422 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe358058b10 con 0x7fe3600ca3c0 2024-02-21T23:06:24.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.163+0000 7fe366e00640 1 -- 172.21.15.28:0/784847422 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe3600b0130 con 0x7fe3600ca3c0 2024-02-21T23:06:24.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.164+0000 7fe35cc00640 1 -- 172.21.15.28:0/784847422 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+5379 (secure 0 0 0) 0x7fe358058410 con 0x7fe3600ca3c0 2024-02-21T23:06:24.152 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:06:24.153 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":69,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5288,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":8,"addr":"172.21.15.81:6835/2774951107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2774951107},{"type":"v1","addr":"172.21.15.81:6835","nonce":2774951107}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67},{"gid":5294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":8,"addr":"172.21.15.81:6837/4292940632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":4292940632},{"type":"v1","addr":"172.21.15.81:6837","nonce":4292940632}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":69,"flags":146,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":true,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:05:50.828988+0000","modified":"2024-02-21T23:06:23.690122+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5},{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:06:10.484421+0000","modified":"2024-02-21T23:06:13.492899+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5539},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5539":{"gid":5539,"name":"c","rank":0,"incarnation":63,"state":"up:active","state_seq":8,"addr":"172.21.15.28:6833/624384999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":624384999},{"type":"v1","addr":"172.21.15.28:6833","nonce":624384999}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-02-21T23:06:24.153 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 69 2024-02-21T23:06:24.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.167+0000 7fe366e00640 1 -- 172.21.15.28:0/784847422 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe33c06b4d0 msgr2=0x7fe33c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:24.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.167+0000 7fe366e00640 1 --2- 172.21.15.28:0/784847422 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe33c06b4d0 0x7fe33c06d990 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fe350036000 tx=0x7fe350030000 comp rx=0 tx=0).stop 2024-02-21T23:06:24.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.167+0000 7fe366e00640 1 -- 172.21.15.28:0/784847422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3600ca3c0 msgr2=0x7fe3600c10e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:24.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.167+0000 7fe366e00640 1 --2- 172.21.15.28:0/784847422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3600ca3c0 0x7fe3600c10e0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fe3600b0350 tx=0x7fe35800d9d0 comp rx=0 tx=0).stop 2024-02-21T23:06:24.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.167+0000 7fe366e00640 1 -- 172.21.15.28:0/784847422 shutdown_connections 2024-02-21T23:06:24.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.167+0000 7fe366e00640 1 --2- 172.21.15.28:0/784847422 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe33c06b4d0 0x7fe33c06d990 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:24.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.167+0000 7fe366e00640 1 --2- 172.21.15.28:0/784847422 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe36014f050 0x7fe3601584c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:24.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.167+0000 7fe366e00640 1 --2- 172.21.15.28:0/784847422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3600ca3c0 0x7fe3600c10e0 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:24.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.167+0000 7fe366e00640 1 --2- 172.21.15.28:0/784847422 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3440039e0 0x7fe3601ee380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:24.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.167+0000 7fe366e00640 1 -- 172.21.15.28:0/784847422 >> 172.21.15.28:0/784847422 conn(0x7fe3600729c0 msgr2=0x7fe3600c5a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:24.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.167+0000 7fe366e00640 1 -- 172.21.15.28:0/784847422 shutdown_connections 2024-02-21T23:06:24.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.167+0000 7fe366e00640 1 -- 172.21.15.28:0/784847422 wait complete. 2024-02-21T23:06:24.175 INFO:tasks.cephfs.test_failover:Waiting for former active to reclaim its spot 2024-02-21T23:06:24.175 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok --format=json config get mds_beacon_grace 2024-02-21T23:06:24.453 INFO:teuthology.orchestra.run.smithi028.stdout:{"mds_beacon_grace":"15.000000"} 2024-02-21T23:06:24.453 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_beacon_grace": "15.000000" } 2024-02-21T23:06:24.454 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:06:24.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.674+0000 7f8db9800640 1 Processor -- start 2024-02-21T23:06:24.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.674+0000 7f8db9800640 1 -- start start 2024-02-21T23:06:24.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.675+0000 7f8db9800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8db4153950 0x7f8db40c15d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:24.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.675+0000 7f8db9800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8db415b6d0 0x7f8db415dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:24.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.675+0000 7f8db9800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8db4152e70 con 0x7f8db40c11f0 2024-02-21T23:06:24.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.675+0000 7f8db9800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8db40a5ec0 con 0x7f8db415b6d0 2024-02-21T23:06:24.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.675+0000 7f8db9800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8db4153150 con 0x7f8db4153950 2024-02-21T23:06:24.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.676+0000 7f8db2a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f8db40c11f0 0x7f8db406dce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42582/0 (socket says 172.21.15.28:42582) 2024-02-21T23:06:24.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.676+0000 7f8db2a00640 1 -- 172.21.15.28:0/425486400 learned_addr learned my addr 172.21.15.28:0/425486400 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:06:24.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.676+0000 7f8db3400640 1 --2- 172.21.15.28:0/425486400 >> v2:172.21.15.28:3300/0 conn(0x7f8db415b6d0 0x7f8db415dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:24.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.676+0000 7f8db2000640 1 --2- 172.21.15.28:0/425486400 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8db4153950 0x7f8db40c15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:24.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.676+0000 7f8db1600640 1 -- 172.21.15.28:0/425486400 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2968782050 0 0) 0x7f8db4152e70 con 0x7f8db40c11f0 2024-02-21T23:06:24.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.676+0000 7f8db1600640 1 -- 172.21.15.28:0/425486400 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d98003520 con 0x7f8db40c11f0 2024-02-21T23:06:24.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.676+0000 7f8db3400640 1 -- 172.21.15.28:0/425486400 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8db4153950 msgr2=0x7f8db40c15d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:24.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.676+0000 7f8db3400640 1 --2- 172.21.15.28:0/425486400 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8db4153950 0x7f8db40c15d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:24.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.676+0000 7f8db3400640 1 -- 172.21.15.28:0/425486400 >> v1:172.21.15.28:6789/0 conn(0x7f8db40c11f0 legacy=0x7f8db406dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:24.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.676+0000 7f8db3400640 1 -- 172.21.15.28:0/425486400 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8db4153430 con 0x7f8db415b6d0 2024-02-21T23:06:24.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.676+0000 7f8db2000640 1 --2- 172.21.15.28:0/425486400 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8db4153950 0x7f8db40c15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:06:24.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.678+0000 7f8db3400640 1 --2- 172.21.15.28:0/425486400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8db415b6d0 0x7f8db415dac0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f8dac009d90 tx=0x7f8dac02be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a7f843509850bce server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:24.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.678+0000 7f8db1600640 1 -- 172.21.15.28:0/425486400 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dac039070 con 0x7f8db415b6d0 2024-02-21T23:06:24.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.678+0000 7f8db1600640 1 -- 172.21.15.28:0/425486400 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dac02cbd0 con 0x7f8db415b6d0 2024-02-21T23:06:24.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.678+0000 7f8db1600640 1 -- 172.21.15.28:0/425486400 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dac034440 con 0x7f8db415b6d0 2024-02-21T23:06:24.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.679+0000 7f8db9800640 1 -- 172.21.15.28:0/425486400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8db415b6d0 msgr2=0x7f8db415dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:24.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.679+0000 7f8db9800640 1 --2- 172.21.15.28:0/425486400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8db415b6d0 0x7f8db415dac0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f8dac009d90 tx=0x7f8dac02be20 comp rx=0 tx=0).stop 2024-02-21T23:06:24.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.679+0000 7f8db9800640 1 -- 172.21.15.28:0/425486400 shutdown_connections 2024-02-21T23:06:24.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.679+0000 7f8db9800640 1 --2- 172.21.15.28:0/425486400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8db415b6d0 0x7f8db415dac0 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:24.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.679+0000 7f8db9800640 1 --2- 172.21.15.28:0/425486400 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8db4153950 0x7f8db40c15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:24.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.679+0000 7f8db9800640 1 -- 172.21.15.28:0/425486400 >> 172.21.15.28:0/425486400 conn(0x7f8db40c67f0 msgr2=0x7f8db40c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:24.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.679+0000 7f8db9800640 1 -- 172.21.15.28:0/425486400 shutdown_connections 2024-02-21T23:06:24.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.679+0000 7f8db9800640 1 -- 172.21.15.28:0/425486400 wait complete. 2024-02-21T23:06:24.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.681+0000 7f8db9800640 1 Processor -- start 2024-02-21T23:06:24.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.681+0000 7f8db9800640 1 -- start start 2024-02-21T23:06:24.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.682+0000 7f8db9800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8db40c11f0 0x7f8db41eadf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:24.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.682+0000 7f8db9800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8db4153950 0x7f8db40bfc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:24.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.683+0000 7f8db9800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8db415b6d0 0x7f8db41eea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:24.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.683+0000 7f8db9800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8db40a6110 con 0x7f8db40c11f0 2024-02-21T23:06:24.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.683+0000 7f8db9800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8db40c4c00 con 0x7f8db4153950 2024-02-21T23:06:24.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.683+0000 7f8db9800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8db40c0d90 con 0x7f8db415b6d0 2024-02-21T23:06:24.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.683+0000 7f8db2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8db40c11f0 0x7f8db41eadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:24.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.683+0000 7f8db2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8db40c11f0 0x7f8db41eadf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54784/0 (socket says 172.21.15.28:54784) 2024-02-21T23:06:24.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.683+0000 7f8db2a00640 1 -- 172.21.15.28:0/3829243117 learned_addr learned my addr 172.21.15.28:0/3829243117 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:24.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.683+0000 7f8db2000640 1 --2- 172.21.15.28:0/3829243117 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8db4153950 0x7f8db40bfc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:24.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.683+0000 7f8db2a00640 1 -- 172.21.15.28:0/3829243117 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8db415b6d0 msgr2=0x7f8db41eea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:24.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.683+0000 7f8db3400640 1 --2- 172.21.15.28:0/3829243117 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8db415b6d0 0x7f8db41eea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:24.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.683+0000 7f8db2a00640 1 --2- 172.21.15.28:0/3829243117 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8db415b6d0 0x7f8db41eea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:24.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.683+0000 7f8db2a00640 1 -- 172.21.15.28:0/3829243117 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8db4153950 msgr2=0x7f8db40bfc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:24.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.683+0000 7f8db2a00640 1 --2- 172.21.15.28:0/3829243117 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8db4153950 0x7f8db40bfc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:24.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.683+0000 7f8db2a00640 1 -- 172.21.15.28:0/3829243117 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8db4142840 con 0x7f8db40c11f0 2024-02-21T23:06:24.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.684+0000 7f8db2a00640 1 --2- 172.21.15.28:0/3829243117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8db40c11f0 0x7f8db41eadf0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f8da400c390 tx=0x7f8da400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:24.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.684+0000 7f8daaa00640 1 -- 172.21.15.28:0/3829243117 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8da4017070 con 0x7f8db40c11f0 2024-02-21T23:06:24.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.684+0000 7f8daaa00640 1 -- 172.21.15.28:0/3829243117 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8da4005ce0 con 0x7f8db40c11f0 2024-02-21T23:06:24.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.684+0000 7f8daaa00640 1 -- 172.21.15.28:0/3829243117 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8da40072f0 con 0x7f8db40c11f0 2024-02-21T23:06:24.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.684+0000 7f8db9800640 1 -- 172.21.15.28:0/3829243117 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8db4144a50 con 0x7f8db40c11f0 2024-02-21T23:06:24.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.684+0000 7f8db9800640 1 -- 172.21.15.28:0/3829243117 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8db4148120 con 0x7f8db40c11f0 2024-02-21T23:06:24.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.685+0000 7f8db9800640 1 -- 172.21.15.28:0/3829243117 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8db4067f30 con 0x7f8db40c11f0 2024-02-21T23:06:24.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.686+0000 7f8daaa00640 1 -- 172.21.15.28:0/3829243117 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8da401a070 con 0x7f8db40c11f0 2024-02-21T23:06:24.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.688+0000 7f8daaa00640 1 --2- 172.21.15.28:0/3829243117 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8d9406b520 0x7f8d9406d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:24.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.688+0000 7f8daaa00640 1 -- 172.21.15.28:0/3829243117 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7043+0+0 (secure 0 0 0) 0x7f8da4089b40 con 0x7f8db40c11f0 2024-02-21T23:06:24.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.688+0000 7f8db2000640 1 --2- 172.21.15.28:0/3829243117 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8d9406b520 0x7f8d9406d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:24.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.690+0000 7f8db2000640 1 --2- 172.21.15.28:0/3829243117 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8d9406b520 0x7f8d9406d9e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f8db4151fc0 tx=0x7f8da0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:24.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.692+0000 7f8daaa00640 1 -- 172.21.15.28:0/3829243117 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f8da40584f0 con 0x7f8db40c11f0 2024-02-21T23:06:24.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.940+0000 7f8db9800640 1 -- 172.21.15.28:0/3829243117 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8db40c0960 con 0x7f8db40c11f0 2024-02-21T23:06:24.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.941+0000 7f8daaa00640 1 -- 172.21.15.28:0/3829243117 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+6179 (secure 0 0 0) 0x7f8da4057df0 con 0x7f8db40c11f0 2024-02-21T23:06:24.930 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:06:24.930 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":71,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5288,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":8,"addr":"172.21.15.81:6835/2774951107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2774951107},{"type":"v1","addr":"172.21.15.81:6835","nonce":2774951107}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67},{"gid":5294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":8,"addr":"172.21.15.81:6837/4292940632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":4292940632},{"type":"v1","addr":"172.21.15.81:6837","nonce":4292940632}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":71,"flags":146,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":true,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:05:50.828988+0000","modified":"2024-02-21T23:06:24.697717+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6729},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6729":{"gid":6729,"name":"a","rank":0,"incarnation":71,"state":"up:replay","state_seq":1,"addr":"172.21.15.28:6835/1066741568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":1066741568},{"type":"v1","addr":"172.21.15.28:6835","nonce":1066741568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5},{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:06:10.484421+0000","modified":"2024-02-21T23:06:13.492899+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5539},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5539":{"gid":5539,"name":"c","rank":0,"incarnation":63,"state":"up:active","state_seq":8,"addr":"172.21.15.28:6833/624384999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":624384999},{"type":"v1","addr":"172.21.15.28:6833","nonce":624384999}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-02-21T23:06:24.930 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 71 2024-02-21T23:06:24.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.944+0000 7f8db9800640 1 -- 172.21.15.28:0/3829243117 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8d9406b520 msgr2=0x7f8d9406d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:24.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.944+0000 7f8db9800640 1 --2- 172.21.15.28:0/3829243117 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8d9406b520 0x7f8d9406d9e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f8db4151fc0 tx=0x7f8da0008040 comp rx=0 tx=0).stop 2024-02-21T23:06:24.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.944+0000 7f8db9800640 1 -- 172.21.15.28:0/3829243117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8db40c11f0 msgr2=0x7f8db41eadf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:24.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.944+0000 7f8db9800640 1 --2- 172.21.15.28:0/3829243117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8db40c11f0 0x7f8db41eadf0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f8da400c390 tx=0x7f8da400c850 comp rx=0 tx=0).stop 2024-02-21T23:06:24.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.944+0000 7f8db9800640 1 -- 172.21.15.28:0/3829243117 shutdown_connections 2024-02-21T23:06:24.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.944+0000 7f8db9800640 1 --2- 172.21.15.28:0/3829243117 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8d9406b520 0x7f8d9406d9e0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:24.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.944+0000 7f8db9800640 1 --2- 172.21.15.28:0/3829243117 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8db415b6d0 0x7f8db41eea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:24.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.944+0000 7f8db9800640 1 --2- 172.21.15.28:0/3829243117 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8db4153950 0x7f8db40bfc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:24.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.944+0000 7f8db9800640 1 --2- 172.21.15.28:0/3829243117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8db40c11f0 0x7f8db41eadf0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:24.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.944+0000 7f8db9800640 1 -- 172.21.15.28:0/3829243117 >> 172.21.15.28:0/3829243117 conn(0x7f8db40c67f0 msgr2=0x7f8db40c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:24.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.944+0000 7f8db9800640 1 -- 172.21.15.28:0/3829243117 shutdown_connections 2024-02-21T23:06:24.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:24.944+0000 7f8db9800640 1 -- 172.21.15.28:0/3829243117 wait complete. 2024-02-21T23:06:24.955 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-02-21T23:06:24.955 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-02-21T23:06:24.955 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-21T23:06:25.080 DEBUG:teuthology.orchestra.run.smithi081:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-21T23:06:25.190 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-02-21T23:06:25.191 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-21T23:06:25.191 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-21T23:06:25.243 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054608,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-02-21T23:06:25.243 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-02-21T23:06:25.244 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-21T23:06:25.300 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053833,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498812k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249404k,nr_inodes=812351,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498812k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-02-21T23:06:25.301 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:06:25.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.529+0000 7fa517800640 1 Processor -- start 2024-02-21T23:06:25.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.530+0000 7fa517800640 1 -- start start 2024-02-21T23:06:25.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.531+0000 7fa517800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fa5100af970 0x7fa5100c7350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:25.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.531+0000 7fa517800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa510155ca0 0x7fa51006cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:25.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.532+0000 7fa517800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5100c5e60 con 0x7fa5100c7890 2024-02-21T23:06:25.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.532+0000 7fa517800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5101561a0 con 0x7fa510155ca0 2024-02-21T23:06:25.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.532+0000 7fa517800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5100c6140 con 0x7fa5100af970 2024-02-21T23:06:25.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.532+0000 7fa50fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa510155ca0 0x7fa51006cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:25.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.532+0000 7fa50fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa510155ca0 0x7fa51006cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47228/0 (socket says 172.21.15.28:47228) 2024-02-21T23:06:25.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.532+0000 7fa50fe00640 1 -- 172.21.15.28:0/159213579 learned_addr learned my addr 172.21.15.28:0/159213579 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:25.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.532+0000 7fa514e00640 1 --2- 172.21.15.28:0/159213579 >> v2:172.21.15.81:3300/0 conn(0x7fa5100af970 0x7fa5100c7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:25.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.532+0000 7fa50fe00640 1 -- 172.21.15.28:0/159213579 >> v2:172.21.15.81:3300/0 conn(0x7fa5100af970 msgr2=0x7fa5100c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:25.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.532+0000 7fa50fe00640 1 --2- 172.21.15.28:0/159213579 >> v2:172.21.15.81:3300/0 conn(0x7fa5100af970 0x7fa5100c7350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:25.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.532+0000 7fa50fe00640 1 -- 172.21.15.28:0/159213579 >> v1:172.21.15.28:6789/0 conn(0x7fa5100c7890 legacy=0x7fa51006d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:25.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.533+0000 7fa50fe00640 1 -- 172.21.15.28:0/159213579 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5100c6420 con 0x7fa510155ca0 2024-02-21T23:06:25.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.534+0000 7fa50fe00640 1 --2- 172.21.15.28:0/159213579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa510155ca0 0x7fa51006cea0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fa5100cba40 tx=0x7fa4fc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2d2e53d1c473f40 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:25.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.534+0000 7fa50f400640 1 -- 172.21.15.28:0/159213579 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4fc035020 con 0x7fa510155ca0 2024-02-21T23:06:25.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.534+0000 7fa50f400640 1 -- 172.21.15.28:0/159213579 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4fc031870 con 0x7fa510155ca0 2024-02-21T23:06:25.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.534+0000 7fa50f400640 1 -- 172.21.15.28:0/159213579 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4fc031c40 con 0x7fa510155ca0 2024-02-21T23:06:25.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.535+0000 7fa517800640 1 -- 172.21.15.28:0/159213579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa510155ca0 msgr2=0x7fa51006cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:25.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.535+0000 7fa517800640 1 --2- 172.21.15.28:0/159213579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa510155ca0 0x7fa51006cea0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fa5100cba40 tx=0x7fa4fc02b910 comp rx=0 tx=0).stop 2024-02-21T23:06:25.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.535+0000 7fa517800640 1 -- 172.21.15.28:0/159213579 shutdown_connections 2024-02-21T23:06:25.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.535+0000 7fa517800640 1 --2- 172.21.15.28:0/159213579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa510155ca0 0x7fa51006cea0 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:25.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.535+0000 7fa517800640 1 --2- 172.21.15.28:0/159213579 >> v2:172.21.15.81:3300/0 conn(0x7fa5100af970 0x7fa5100c7350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:25.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.535+0000 7fa517800640 1 -- 172.21.15.28:0/159213579 >> 172.21.15.28:0/159213579 conn(0x7fa5100725a0 msgr2=0x7fa5100729b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:25.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.535+0000 7fa517800640 1 -- 172.21.15.28:0/159213579 shutdown_connections 2024-02-21T23:06:25.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.535+0000 7fa517800640 1 -- 172.21.15.28:0/159213579 wait complete. 2024-02-21T23:06:25.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.539+0000 7fa517800640 1 Processor -- start 2024-02-21T23:06:25.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.539+0000 7fa517800640 1 -- start start 2024-02-21T23:06:25.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.541+0000 7fa517800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa5100af970 0x7fa51014daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:25.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.541+0000 7fa517800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5100c7890 0x7fa51014b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:25.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.541+0000 7fa517800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa510155ca0 0x7fa51014bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:25.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.541+0000 7fa517800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5100afd50 con 0x7fa5100c7890 2024-02-21T23:06:25.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.541+0000 7fa517800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa51006daf0 con 0x7fa510155ca0 2024-02-21T23:06:25.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.541+0000 7fa517800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5100afed0 con 0x7fa5100af970 2024-02-21T23:06:25.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.541+0000 7fa50fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5100c7890 0x7fa51014b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:25.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.541+0000 7fa50fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5100c7890 0x7fa51014b610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47236/0 (socket says 172.21.15.28:47236) 2024-02-21T23:06:25.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.541+0000 7fa50fe00640 1 -- 172.21.15.28:0/4037468484 learned_addr learned my addr 172.21.15.28:0/4037468484 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:25.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.542+0000 7fa514e00640 1 --2- 172.21.15.28:0/4037468484 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa5100af970 0x7fa51014daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:25.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.542+0000 7fa515800640 1 --2- 172.21.15.28:0/4037468484 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa510155ca0 0x7fa51014bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:25.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.542+0000 7fa50fe00640 1 -- 172.21.15.28:0/4037468484 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa5100af970 msgr2=0x7fa51014daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:25.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.542+0000 7fa50fe00640 1 --2- 172.21.15.28:0/4037468484 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa5100af970 0x7fa51014daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:25.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.542+0000 7fa50fe00640 1 -- 172.21.15.28:0/4037468484 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa510155ca0 msgr2=0x7fa51014bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:25.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.542+0000 7fa50fe00640 1 --2- 172.21.15.28:0/4037468484 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa510155ca0 0x7fa51014bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:25.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.542+0000 7fa50fe00640 1 -- 172.21.15.28:0/4037468484 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5101ee910 con 0x7fa5100c7890 2024-02-21T23:06:25.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.542+0000 7fa50fe00640 1 --2- 172.21.15.28:0/4037468484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5100c7890 0x7fa51014b610 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc02b950 tx=0x7fa4fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:25.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.543+0000 7fa50cc00640 1 -- 172.21.15.28:0/4037468484 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4fc035040 con 0x7fa5100c7890 2024-02-21T23:06:25.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.543+0000 7fa50cc00640 1 -- 172.21.15.28:0/4037468484 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4fc030d00 con 0x7fa5100c7890 2024-02-21T23:06:25.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.543+0000 7fa517800640 1 -- 172.21.15.28:0/4037468484 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5101eebf0 con 0x7fa5100c7890 2024-02-21T23:06:25.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.543+0000 7fa50cc00640 1 -- 172.21.15.28:0/4037468484 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4fc0456a0 con 0x7fa5100c7890 2024-02-21T23:06:25.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.543+0000 7fa517800640 1 -- 172.21.15.28:0/4037468484 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5101ef1b0 con 0x7fa5100c7890 2024-02-21T23:06:25.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.544+0000 7fa517800640 1 -- 172.21.15.28:0/4037468484 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa510068640 con 0x7fa5100c7890 2024-02-21T23:06:25.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.545+0000 7fa50cc00640 1 -- 172.21.15.28:0/4037468484 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa4fc038020 con 0x7fa5100c7890 2024-02-21T23:06:25.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.547+0000 7fa50cc00640 1 --2- 172.21.15.28:0/4037468484 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa4f006b4d0 0x7fa4f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:25.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.547+0000 7fa50cc00640 1 -- 172.21.15.28:0/4037468484 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7159+0+0 (secure 0 0 0) 0x7fa4fc0acb70 con 0x7fa5100c7890 2024-02-21T23:06:25.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.548+0000 7fa514e00640 1 --2- 172.21.15.28:0/4037468484 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa4f006b4d0 0x7fa4f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:25.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.549+0000 7fa514e00640 1 --2- 172.21.15.28:0/4037468484 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa4f006b4d0 0x7fa4f006d990 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fa504006fd0 tx=0x7fa504008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:25.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.551+0000 7fa50cc00640 1 -- 172.21.15.28:0/4037468484 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fa4fc07b3b0 con 0x7fa5100c7890 2024-02-21T23:06:25.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.796+0000 7fa517800640 1 -- 172.21.15.28:0/4037468484 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa5101eb150 con 0x7fa5100c7890 2024-02-21T23:06:25.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.798+0000 7fa50cc00640 1 -- 172.21.15.28:0/4037468484 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+6182 (secure 0 0 0) 0x7fa5101eb150 con 0x7fa5100c7890 2024-02-21T23:06:25.786 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:06:25.786 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":72,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5288,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":8,"addr":"172.21.15.81:6835/2774951107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2774951107},{"type":"v1","addr":"172.21.15.81:6835","nonce":2774951107}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67},{"gid":5294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":8,"addr":"172.21.15.81:6837/4292940632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":4292940632},{"type":"v1","addr":"172.21.15.81:6837","nonce":4292940632}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":72,"flags":146,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":true,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:05:50.828988+0000","modified":"2024-02-21T23:06:24.716258+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6729},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6729":{"gid":6729,"name":"a","rank":0,"incarnation":71,"state":"up:reconnect","state_seq":2,"addr":"172.21.15.28:6835/1066741568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":1066741568},{"type":"v1","addr":"172.21.15.28:6835","nonce":1066741568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5},{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:06:10.484421+0000","modified":"2024-02-21T23:06:13.492899+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5539},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5539":{"gid":5539,"name":"c","rank":0,"incarnation":63,"state":"up:active","state_seq":8,"addr":"172.21.15.28:6833/624384999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":624384999},{"type":"v1","addr":"172.21.15.28:6833","nonce":624384999}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-02-21T23:06:25.786 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 72 2024-02-21T23:06:25.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.801+0000 7fa517800640 1 -- 172.21.15.28:0/4037468484 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa4f006b4d0 msgr2=0x7fa4f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:25.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.801+0000 7fa517800640 1 --2- 172.21.15.28:0/4037468484 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa4f006b4d0 0x7fa4f006d990 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fa504006fd0 tx=0x7fa504008040 comp rx=0 tx=0).stop 2024-02-21T23:06:25.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.801+0000 7fa517800640 1 -- 172.21.15.28:0/4037468484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5100c7890 msgr2=0x7fa51014b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:25.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.801+0000 7fa517800640 1 --2- 172.21.15.28:0/4037468484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5100c7890 0x7fa51014b610 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc02b950 tx=0x7fa4fc0046c0 comp rx=0 tx=0).stop 2024-02-21T23:06:25.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.801+0000 7fa517800640 1 -- 172.21.15.28:0/4037468484 shutdown_connections 2024-02-21T23:06:25.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.801+0000 7fa517800640 1 --2- 172.21.15.28:0/4037468484 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa4f006b4d0 0x7fa4f006d990 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:25.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.801+0000 7fa517800640 1 --2- 172.21.15.28:0/4037468484 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa510155ca0 0x7fa51014bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:25.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.801+0000 7fa517800640 1 --2- 172.21.15.28:0/4037468484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5100c7890 0x7fa51014b610 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:25.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.801+0000 7fa517800640 1 --2- 172.21.15.28:0/4037468484 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa5100af970 0x7fa51014daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:25.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.801+0000 7fa517800640 1 -- 172.21.15.28:0/4037468484 >> 172.21.15.28:0/4037468484 conn(0x7fa5100725a0 msgr2=0x7fa5101547b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:25.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.801+0000 7fa517800640 1 -- 172.21.15.28:0/4037468484 shutdown_connections 2024-02-21T23:06:25.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:25.801+0000 7fa517800640 1 -- 172.21.15.28:0/4037468484 wait complete. 2024-02-21T23:06:25.808 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:06:26.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.034+0000 7f17fb200640 1 Processor -- start 2024-02-21T23:06:26.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.034+0000 7f17fb200640 1 -- start start 2024-02-21T23:06:26.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.035+0000 7f17fb200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f17fc150450 0x7f17fc0a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:26.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.035+0000 7f17fb200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f17fc06ee40 0x7f17fc1536f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:26.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.035+0000 7f17fb200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17fc0c2760 con 0x7f17fc156af0 2024-02-21T23:06:26.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.035+0000 7f17fb200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17fc0c22c0 con 0x7f17fc150450 2024-02-21T23:06:26.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.035+0000 7f17fb200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17fc156660 con 0x7f17fc06ee40 2024-02-21T23:06:26.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.036+0000 7f17f9e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f17fc150450 0x7f17fc0a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:26.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.036+0000 7f17f9e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f17fc150450 0x7f17fc0a5fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47250/0 (socket says 172.21.15.28:47250) 2024-02-21T23:06:26.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.036+0000 7f17f9e00640 1 -- 172.21.15.28:0/3821999393 learned_addr learned my addr 172.21.15.28:0/3821999393 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:26.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.036+0000 7f17f1400640 1 --2- 172.21.15.28:0/3821999393 >> v2:172.21.15.81:3300/0 conn(0x7f17fc06ee40 0x7f17fc1536f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:26.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.036+0000 7f17f9e00640 1 -- 172.21.15.28:0/3821999393 >> v2:172.21.15.81:3300/0 conn(0x7f17fc06ee40 msgr2=0x7f17fc1536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:26.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.036+0000 7f17f9e00640 1 --2- 172.21.15.28:0/3821999393 >> v2:172.21.15.81:3300/0 conn(0x7f17fc06ee40 0x7f17fc1536f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:26.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.036+0000 7f17f9e00640 1 -- 172.21.15.28:0/3821999393 >> v1:172.21.15.81:6789/0 conn(0x7f17fc156af0 legacy=0x7f17fc06dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:26.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.036+0000 7f17f9e00640 1 -- 172.21.15.28:0/3821999393 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17fc153c30 con 0x7f17fc150450 2024-02-21T23:06:26.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.038+0000 7f17f9e00640 1 --2- 172.21.15.28:0/3821999393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17fc150450 0x7f17fc0a5fd0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f17fc06bcc0 tx=0x7f17ec02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=998853652416f1a server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:26.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.038+0000 7f17f9400640 1 -- 172.21.15.28:0/3821999393 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17ec035020 con 0x7f17fc150450 2024-02-21T23:06:26.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.038+0000 7f17f9400640 1 -- 172.21.15.28:0/3821999393 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17ec031870 con 0x7f17fc150450 2024-02-21T23:06:26.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.038+0000 7f17f9400640 1 -- 172.21.15.28:0/3821999393 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17ec031c40 con 0x7f17fc150450 2024-02-21T23:06:26.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.039+0000 7f17fb200640 1 -- 172.21.15.28:0/3821999393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17fc150450 msgr2=0x7f17fc0a5fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:26.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.039+0000 7f17fb200640 1 --2- 172.21.15.28:0/3821999393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17fc150450 0x7f17fc0a5fd0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f17fc06bcc0 tx=0x7f17ec02b910 comp rx=0 tx=0).stop 2024-02-21T23:06:26.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.039+0000 7f17fb200640 1 -- 172.21.15.28:0/3821999393 shutdown_connections 2024-02-21T23:06:26.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.039+0000 7f17fb200640 1 --2- 172.21.15.28:0/3821999393 >> v2:172.21.15.81:3300/0 conn(0x7f17fc06ee40 0x7f17fc1536f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:26.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.039+0000 7f17fb200640 1 --2- 172.21.15.28:0/3821999393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17fc150450 0x7f17fc0a5fd0 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:26.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.039+0000 7f17fb200640 1 -- 172.21.15.28:0/3821999393 >> 172.21.15.28:0/3821999393 conn(0x7f17fc06b630 msgr2=0x7f17fc0c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:26.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.039+0000 7f17fb200640 1 -- 172.21.15.28:0/3821999393 shutdown_connections 2024-02-21T23:06:26.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.039+0000 7f17fb200640 1 -- 172.21.15.28:0/3821999393 wait complete. 2024-02-21T23:06:26.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.041+0000 7f17fb200640 1 Processor -- start 2024-02-21T23:06:26.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.041+0000 7f17fb200640 1 -- start start 2024-02-21T23:06:26.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.042+0000 7f17fb200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f17fc06ee40 0x7f17fc1e0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:26.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.042+0000 7f17fb200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17fc150450 0x7f17fc0c40f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:26.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.042+0000 7f17fb200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f17fc156af0 0x7f17fc0c4630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:26.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.042+0000 7f17fb200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17fc0c2cb0 con 0x7f17fc150450 2024-02-21T23:06:26.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.042+0000 7f17fb200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17fc0c29d0 con 0x7f17fc156af0 2024-02-21T23:06:26.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.042+0000 7f17fb200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17fc06e700 con 0x7f17fc06ee40 2024-02-21T23:06:26.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.042+0000 7f17f1400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17fc150450 0x7f17fc0c40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:26.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.042+0000 7f17f1400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17fc150450 0x7f17fc0c40f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47260/0 (socket says 172.21.15.28:47260) 2024-02-21T23:06:26.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.042+0000 7f17f1400640 1 -- 172.21.15.28:0/1620416298 learned_addr learned my addr 172.21.15.28:0/1620416298 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:26.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.043+0000 7f17f9e00640 1 --2- 172.21.15.28:0/1620416298 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f17fc06ee40 0x7f17fc1e0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:26.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.043+0000 7f17fa800640 1 --2- 172.21.15.28:0/1620416298 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f17fc156af0 0x7f17fc0c4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:26.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.043+0000 7f17f1400640 1 -- 172.21.15.28:0/1620416298 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f17fc06ee40 msgr2=0x7f17fc1e0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:26.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.043+0000 7f17f1400640 1 --2- 172.21.15.28:0/1620416298 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f17fc06ee40 0x7f17fc1e0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:26.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.043+0000 7f17f1400640 1 -- 172.21.15.28:0/1620416298 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f17fc156af0 msgr2=0x7f17fc0c4630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:26.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.043+0000 7f17f1400640 1 --2- 172.21.15.28:0/1620416298 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f17fc156af0 0x7f17fc0c4630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:26.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.043+0000 7f17f1400640 1 -- 172.21.15.28:0/1620416298 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17fc0c6250 con 0x7f17fc150450 2024-02-21T23:06:26.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.043+0000 7f17f1400640 1 --2- 172.21.15.28:0/1620416298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17fc150450 0x7f17fc0c40f0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f17e400d5f0 tx=0x7f17e400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:26.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.044+0000 7f17f2a00640 1 -- 172.21.15.28:0/1620416298 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e40125f0 con 0x7f17fc150450 2024-02-21T23:06:26.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.044+0000 7f17f2a00640 1 -- 172.21.15.28:0/1620416298 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17e4019070 con 0x7f17fc150450 2024-02-21T23:06:26.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.044+0000 7f17f2a00640 1 -- 172.21.15.28:0/1620416298 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e40129d0 con 0x7f17fc150450 2024-02-21T23:06:26.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.044+0000 7f17fb200640 1 -- 172.21.15.28:0/1620416298 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17fc0c4b70 con 0x7f17fc150450 2024-02-21T23:06:26.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.044+0000 7f17fb200640 1 -- 172.21.15.28:0/1620416298 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17fc0c5a50 con 0x7f17fc150450 2024-02-21T23:06:26.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.045+0000 7f17fb200640 1 -- 172.21.15.28:0/1620416298 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17fc04f290 con 0x7f17fc150450 2024-02-21T23:06:26.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.047+0000 7f17f2a00640 1 -- 172.21.15.28:0/1620416298 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f17e4012b70 con 0x7f17fc150450 2024-02-21T23:06:26.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.048+0000 7f17f2a00640 1 --2- 172.21.15.28:0/1620416298 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f17d806b4d0 0x7f17d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:26.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.048+0000 7f17f2a00640 1 -- 172.21.15.28:0/1620416298 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7159+0+0 (secure 0 0 0) 0x7f17e408ab30 con 0x7f17fc150450 2024-02-21T23:06:26.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.049+0000 7f17f9e00640 1 --2- 172.21.15.28:0/1620416298 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f17d806b4d0 0x7f17d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:26.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.050+0000 7f17f9e00640 1 --2- 172.21.15.28:0/1620416298 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f17d806b4d0 0x7f17d806d990 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f17ec036000 tx=0x7f17ec03c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:26.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.053+0000 7f17f2a00640 1 -- 172.21.15.28:0/1620416298 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f17e4059490 con 0x7f17fc150450 2024-02-21T23:06:26.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.269+0000 7f17fb200640 1 -- 172.21.15.28:0/1620416298 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f17fc14a110 con 0x7f17fc150450 2024-02-21T23:06:26.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.273+0000 7f17f2a00640 1 -- 172.21.15.28:0/1620416298 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+20929 (secure 0 0 0) 0x7f17e4058d90 con 0x7f17fc150450 2024-02-21T23:06:26.263 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:06:26.263 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":83,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:06:26.056922+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:05:48.772562+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-02-21T23:05:49.852659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs2_metadata","create_time":"2024-02-21T23:06:08.240443+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs2_data","create_time":"2024-02-21T23:06:09.466016+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.2","mappings":[{"from":1,"to":7}]},{"pgid":"14.12","mappings":[{"from":1,"to":5}]},{"pgid":"14.1e","mappings":[{"from":1,"to":2}]},{"pgid":"14.24","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6835/595682298":"2024-02-22T23:06:23.690106+0000","172.21.15.28:6834/595682298":"2024-02-22T23:06:23.690106+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:06:26.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.277+0000 7f17fb200640 1 -- 172.21.15.28:0/1620416298 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f17d806b4d0 msgr2=0x7f17d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:26.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.277+0000 7f17fb200640 1 --2- 172.21.15.28:0/1620416298 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f17d806b4d0 0x7f17d806d990 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f17ec036000 tx=0x7f17ec03c040 comp rx=0 tx=0).stop 2024-02-21T23:06:26.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.277+0000 7f17fb200640 1 -- 172.21.15.28:0/1620416298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17fc150450 msgr2=0x7f17fc0c40f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:26.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.277+0000 7f17fb200640 1 --2- 172.21.15.28:0/1620416298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17fc150450 0x7f17fc0c40f0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f17e400d5f0 tx=0x7f17e400dab0 comp rx=0 tx=0).stop 2024-02-21T23:06:26.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.277+0000 7f17fb200640 1 -- 172.21.15.28:0/1620416298 shutdown_connections 2024-02-21T23:06:26.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.277+0000 7f17fb200640 1 --2- 172.21.15.28:0/1620416298 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f17d806b4d0 0x7f17d806d990 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:26.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.277+0000 7f17fb200640 1 --2- 172.21.15.28:0/1620416298 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f17fc156af0 0x7f17fc0c4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:26.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.277+0000 7f17fb200640 1 --2- 172.21.15.28:0/1620416298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17fc150450 0x7f17fc0c40f0 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:26.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.277+0000 7f17fb200640 1 --2- 172.21.15.28:0/1620416298 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f17fc06ee40 0x7f17fc1e0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:26.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.277+0000 7f17fb200640 1 -- 172.21.15.28:0/1620416298 >> 172.21.15.28:0/1620416298 conn(0x7f17fc06b630 msgr2=0x7f17fc0c64e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:26.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.277+0000 7f17fb200640 1 -- 172.21.15.28:0/1620416298 shutdown_connections 2024-02-21T23:06:26.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.277+0000 7f17fb200640 1 -- 172.21.15.28:0/1620416298 wait complete. 2024-02-21T23:06:26.284 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-21T23:02:30.728329+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-21T23:05:48.772562+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_data', 'create_time': '2024-02-21T23:05:49.852659+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs2_metadata', 'create_time': '2024-02-21T23:06:08.240443+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs2'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs2_data', 'create_time': '2024-02-21T23:06:09.466016+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs2'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-21T23:06:26.284 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-21T23:06:26.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.500+0000 7f7d09a00640 1 Processor -- start 2024-02-21T23:06:26.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.500+0000 7f7d09a00640 1 -- start start 2024-02-21T23:06:26.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.502+0000 7f7d09a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7d040c3800 0x7f7d040c3c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:26.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.502+0000 7f7d09a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f7d041530e0 0x7f7d04153550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:26.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.502+0000 7f7d09a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d040c22e0 con 0x7f7d04150500 2024-02-21T23:06:26.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.502+0000 7f7d09a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d04152be0 con 0x7f7d040c3800 2024-02-21T23:06:26.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.502+0000 7f7d09a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d0406c5c0 con 0x7f7d041530e0 2024-02-21T23:06:26.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.502+0000 7f7d03e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f7d04150500 0x7f7d040a6d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37444/0 (socket says 172.21.15.28:37444) 2024-02-21T23:06:26.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.502+0000 7f7cfbe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7d040c3800 0x7f7d040c3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:26.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.502+0000 7f7d03e00640 1 -- 172.21.15.28:0/1224436711 learned_addr learned my addr 172.21.15.28:0/1224436711 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:06:26.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.503+0000 7f7d09000640 1 --2- 172.21.15.28:0/1224436711 >> v2:172.21.15.81:3300/0 conn(0x7f7d041530e0 0x7f7d04153550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:26.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.503+0000 7f7cfbe00640 1 -- 172.21.15.28:0/1224436711 >> v2:172.21.15.81:3300/0 conn(0x7f7d041530e0 msgr2=0x7f7d04153550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:26.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.503+0000 7f7cfbe00640 1 --2- 172.21.15.28:0/1224436711 >> v2:172.21.15.81:3300/0 conn(0x7f7d041530e0 0x7f7d04153550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:26.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.503+0000 7f7d03400640 1 -- 172.21.15.28:0/1224436711 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1208099828 0 0) 0x7f7d040c22e0 con 0x7f7d04150500 2024-02-21T23:06:26.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.503+0000 7f7cfbe00640 1 -- 172.21.15.28:0/1224436711 >> v1:172.21.15.28:6789/0 conn(0x7f7d04150500 legacy=0x7f7d040a6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:26.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.503+0000 7f7cfbe00640 1 -- 172.21.15.28:0/1224436711 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d0406c8a0 con 0x7f7d040c3800 2024-02-21T23:06:26.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.505+0000 7f7cfbe00640 1 --2- 172.21.15.28:0/1224436711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d040c3800 0x7f7d040c3c60 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f7d040c8780 tx=0x7f7cec02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d62087230a91653f server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:26.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.505+0000 7f7d03400640 1 -- 172.21.15.28:0/1224436711 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cec035020 con 0x7f7d040c3800 2024-02-21T23:06:26.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.505+0000 7f7d03400640 1 -- 172.21.15.28:0/1224436711 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cec031830 con 0x7f7d040c3800 2024-02-21T23:06:26.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.505+0000 7f7d03400640 1 -- 172.21.15.28:0/1224436711 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cec031c00 con 0x7f7d040c3800 2024-02-21T23:06:26.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.506+0000 7f7d09a00640 1 -- 172.21.15.28:0/1224436711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d040c3800 msgr2=0x7f7d040c3c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:26.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.506+0000 7f7d09a00640 1 --2- 172.21.15.28:0/1224436711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d040c3800 0x7f7d040c3c60 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f7d040c8780 tx=0x7f7cec02b8d0 comp rx=0 tx=0).stop 2024-02-21T23:06:26.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.506+0000 7f7d09a00640 1 -- 172.21.15.28:0/1224436711 shutdown_connections 2024-02-21T23:06:26.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.506+0000 7f7d09a00640 1 --2- 172.21.15.28:0/1224436711 >> v2:172.21.15.81:3300/0 conn(0x7f7d041530e0 0x7f7d04153550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:26.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.506+0000 7f7d09a00640 1 --2- 172.21.15.28:0/1224436711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d040c3800 0x7f7d040c3c60 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:26.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.506+0000 7f7d09a00640 1 -- 172.21.15.28:0/1224436711 >> 172.21.15.28:0/1224436711 conn(0x7f7d040a9690 msgr2=0x7f7d040a9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:26.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.506+0000 7f7d09a00640 1 -- 172.21.15.28:0/1224436711 shutdown_connections 2024-02-21T23:06:26.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.506+0000 7f7d09a00640 1 -- 172.21.15.28:0/1224436711 wait complete. 2024-02-21T23:06:26.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.507+0000 7f7d09a00640 1 Processor -- start 2024-02-21T23:06:26.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.508+0000 7f7d09a00640 1 -- start start 2024-02-21T23:06:26.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.508+0000 7f7d09a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7d040c3800 0x7f7d041de760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:26.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.508+0000 7f7d09a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d04150500 0x7f7d0406f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:26.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.508+0000 7f7d09a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7d041530e0 0x7f7d0406f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:26.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.508+0000 7f7d09a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d04153a90 con 0x7f7d04150500 2024-02-21T23:06:26.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.508+0000 7f7d09a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d0406ce60 con 0x7f7d040c3800 2024-02-21T23:06:26.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.509+0000 7f7d09a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d040c7470 con 0x7f7d041530e0 2024-02-21T23:06:26.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.509+0000 7f7cfbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d04150500 0x7f7d0406f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:26.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.509+0000 7f7cfbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d04150500 0x7f7d0406f360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47266/0 (socket says 172.21.15.28:47266) 2024-02-21T23:06:26.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.509+0000 7f7cfbe00640 1 -- 172.21.15.28:0/501103565 learned_addr learned my addr 172.21.15.28:0/501103565 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:26.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.509+0000 7f7d03e00640 1 --2- 172.21.15.28:0/501103565 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7d040c3800 0x7f7d041de760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:26.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.509+0000 7f7d09000640 1 --2- 172.21.15.28:0/501103565 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7d041530e0 0x7f7d0406f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:26.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.509+0000 7f7cfbe00640 1 -- 172.21.15.28:0/501103565 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7d041530e0 msgr2=0x7f7d0406f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:26.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.509+0000 7f7cfbe00640 1 --2- 172.21.15.28:0/501103565 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7d041530e0 0x7f7d0406f8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:26.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.509+0000 7f7cfbe00640 1 -- 172.21.15.28:0/501103565 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7d040c3800 msgr2=0x7f7d041de760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:26.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.509+0000 7f7cfbe00640 1 --2- 172.21.15.28:0/501103565 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7d040c3800 0x7f7d041de760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:26.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.509+0000 7f7cfbe00640 1 -- 172.21.15.28:0/501103565 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d041dd5d0 con 0x7f7d04150500 2024-02-21T23:06:26.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.510+0000 7f7cfbe00640 1 --2- 172.21.15.28:0/501103565 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d04150500 0x7f7d0406f360 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f7cec000c00 tx=0x7f7cec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:26.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.510+0000 7f7d00c00640 1 -- 172.21.15.28:0/501103565 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cec040070 con 0x7f7d04150500 2024-02-21T23:06:26.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.510+0000 7f7d00c00640 1 -- 172.21.15.28:0/501103565 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cec039a20 con 0x7f7d04150500 2024-02-21T23:06:26.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.510+0000 7f7d09a00640 1 -- 172.21.15.28:0/501103565 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d041dd870 con 0x7f7d04150500 2024-02-21T23:06:26.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.510+0000 7f7d00c00640 1 -- 172.21.15.28:0/501103565 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cec035040 con 0x7f7d04150500 2024-02-21T23:06:26.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.510+0000 7f7d09a00640 1 -- 172.21.15.28:0/501103565 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d041dde30 con 0x7f7d04150500 2024-02-21T23:06:26.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.512+0000 7f7d09a00640 1 -- 172.21.15.28:0/501103565 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d0404f290 con 0x7f7d04150500 2024-02-21T23:06:26.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.513+0000 7f7d00c00640 1 -- 172.21.15.28:0/501103565 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7cec04b060 con 0x7f7d04150500 2024-02-21T23:06:26.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.513+0000 7f7d00c00640 1 --2- 172.21.15.28:0/501103565 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7ce006b520 0x7f7ce006d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:26.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.513+0000 7f7d00c00640 1 -- 172.21.15.28:0/501103565 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7159+0+0 (secure 0 0 0) 0x7f7cec0ac0a0 con 0x7f7d04150500 2024-02-21T23:06:26.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.514+0000 7f7d03e00640 1 --2- 172.21.15.28:0/501103565 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7ce006b520 0x7f7ce006d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:26.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.515+0000 7f7d03e00640 1 --2- 172.21.15.28:0/501103565 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7ce006b520 0x7f7ce006d9e0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f7cf4000f60 tx=0x7f7cf4007b50 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:26.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.518+0000 7f7d00c00640 1 -- 172.21.15.28:0/501103565 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7cec07a9d0 con 0x7f7d04150500 2024-02-21T23:06:26.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.720+0000 7f7d09a00640 1 -- 172.21.15.28:0/501103565 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7d040c71d0 con 0x7f7d04150500 2024-02-21T23:06:26.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.721+0000 7f7d00c00640 1 -- 172.21.15.28:0/501103565 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7cec07a2d0 con 0x7f7d04150500 2024-02-21T23:06:26.709 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-02-21T23:06:26.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.724+0000 7f7d09a00640 1 -- 172.21.15.28:0/501103565 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7ce006b520 msgr2=0x7f7ce006d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:26.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.724+0000 7f7d09a00640 1 --2- 172.21.15.28:0/501103565 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7ce006b520 0x7f7ce006d9e0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f7cf4000f60 tx=0x7f7cf4007b50 comp rx=0 tx=0).stop 2024-02-21T23:06:26.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.724+0000 7f7d09a00640 1 -- 172.21.15.28:0/501103565 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d04150500 msgr2=0x7f7d0406f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:26.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.724+0000 7f7d09a00640 1 --2- 172.21.15.28:0/501103565 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d04150500 0x7f7d0406f360 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f7cec000c00 tx=0x7f7cec0046c0 comp rx=0 tx=0).stop 2024-02-21T23:06:26.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.724+0000 7f7d09a00640 1 -- 172.21.15.28:0/501103565 shutdown_connections 2024-02-21T23:06:26.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.724+0000 7f7d09a00640 1 --2- 172.21.15.28:0/501103565 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7ce006b520 0x7f7ce006d9e0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:26.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.724+0000 7f7d09a00640 1 --2- 172.21.15.28:0/501103565 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7d041530e0 0x7f7d0406f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:26.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.724+0000 7f7d09a00640 1 --2- 172.21.15.28:0/501103565 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d04150500 0x7f7d0406f360 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:26.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.724+0000 7f7d09a00640 1 --2- 172.21.15.28:0/501103565 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7d040c3800 0x7f7d041de760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:26.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.724+0000 7f7d09a00640 1 -- 172.21.15.28:0/501103565 >> 172.21.15.28:0/501103565 conn(0x7f7d040a9690 msgr2=0x7f7d0414c950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:26.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.724+0000 7f7d09a00640 1 -- 172.21.15.28:0/501103565 shutdown_connections 2024-02-21T23:06:26.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.724+0000 7f7d09a00640 1 -- 172.21.15.28:0/501103565 wait complete. 2024-02-21T23:06:26.733 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-21T23:06:26.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.967+0000 7f4709800640 1 Processor -- start 2024-02-21T23:06:26.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.968+0000 7f4709800640 1 -- start start 2024-02-21T23:06:26.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.970+0000 7f4709800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4704158480 0x7f4704154f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:26.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.970+0000 7f4709800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f470406e9d0 0x7f470406ee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:26.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.970+0000 7f4709800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f470406f3a0 con 0x7f4704152560 2024-02-21T23:06:26.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.970+0000 7f4709800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47041554a0 con 0x7f470406e9d0 2024-02-21T23:06:26.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.970+0000 7f4709800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47040a6270 con 0x7f4704158480 2024-02-21T23:06:26.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.971+0000 7f4708e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f470406e9d0 0x7f470406ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:26.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.971+0000 7f4703400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4704158480 0x7f4704154f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:26.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.971+0000 7f4708e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f470406e9d0 0x7f470406ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.28:37778/0 (socket says 172.21.15.28:37778) 2024-02-21T23:06:26.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.971+0000 7f4708e00640 1 -- 172.21.15.28:0/618058077 learned_addr learned my addr 172.21.15.28:0/618058077 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:26.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.971+0000 7f4703400640 1 -- 172.21.15.28:0/618058077 >> v2:172.21.15.81:3300/0 conn(0x7f470406e9d0 msgr2=0x7f470406ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:26.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.971+0000 7f4703400640 1 --2- 172.21.15.28:0/618058077 >> v2:172.21.15.81:3300/0 conn(0x7f470406e9d0 0x7f470406ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:26.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.971+0000 7f4703400640 1 -- 172.21.15.28:0/618058077 >> v1:172.21.15.81:6789/0 conn(0x7f4704152560 legacy=0x7f470406ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:26.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.971+0000 7f4703400640 1 -- 172.21.15.28:0/618058077 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47041556d0 con 0x7f4704158480 2024-02-21T23:06:26.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.972+0000 7f4708e00640 1 --2- 172.21.15.28:0/618058077 >> v2:172.21.15.81:3300/0 conn(0x7f470406e9d0 0x7f470406ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-21T23:06:26.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.973+0000 7f4703400640 1 --2- 172.21.15.28:0/618058077 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4704158480 0x7f4704154f60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f46f00097a0 tx=0x7f46f002b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=845f53f50b86f725 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:26.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.973+0000 7f4702a00640 1 -- 172.21.15.28:0/618058077 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46f0035020 con 0x7f4704158480 2024-02-21T23:06:26.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.974+0000 7f4702a00640 1 -- 172.21.15.28:0/618058077 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46f0031810 con 0x7f4704158480 2024-02-21T23:06:26.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.974+0000 7f4702a00640 1 -- 172.21.15.28:0/618058077 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46f0031be0 con 0x7f4704158480 2024-02-21T23:06:26.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.974+0000 7f4709800640 1 -- 172.21.15.28:0/618058077 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4704158480 msgr2=0x7f4704154f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:26.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.974+0000 7f4709800640 1 --2- 172.21.15.28:0/618058077 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4704158480 0x7f4704154f60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f46f00097a0 tx=0x7f46f002b8b0 comp rx=0 tx=0).stop 2024-02-21T23:06:26.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.974+0000 7f4709800640 1 -- 172.21.15.28:0/618058077 shutdown_connections 2024-02-21T23:06:26.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.974+0000 7f4709800640 1 --2- 172.21.15.28:0/618058077 >> v2:172.21.15.81:3300/0 conn(0x7f470406e9d0 0x7f470406ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:26.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.974+0000 7f4709800640 1 --2- 172.21.15.28:0/618058077 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4704158480 0x7f4704154f60 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:26.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.974+0000 7f4709800640 1 -- 172.21.15.28:0/618058077 >> 172.21.15.28:0/618058077 conn(0x7f47040a8530 msgr2=0x7f47040a8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:26.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.975+0000 7f4709800640 1 -- 172.21.15.28:0/618058077 shutdown_connections 2024-02-21T23:06:26.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.975+0000 7f4709800640 1 -- 172.21.15.28:0/618058077 wait complete. 2024-02-21T23:06:26.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.976+0000 7f4709800640 1 Processor -- start 2024-02-21T23:06:26.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.976+0000 7f4709800640 1 -- start start 2024-02-21T23:06:26.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.978+0000 7f4709800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f470406e9d0 0x7f47041edd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:26.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.978+0000 7f4709800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4704152560 0x7f4704147340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:26.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.978+0000 7f4709800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4704158480 0x7f47041505b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:26.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.978+0000 7f4709800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f470406e4b0 con 0x7f4704152560 2024-02-21T23:06:26.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.978+0000 7f4709800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4704155c90 con 0x7f4704158480 2024-02-21T23:06:26.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.978+0000 7f4709800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4704152940 con 0x7f470406e9d0 2024-02-21T23:06:26.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.978+0000 7f4703400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4704152560 0x7f4704147340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:26.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.978+0000 7f4703400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4704152560 0x7f4704147340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47280/0 (socket says 172.21.15.28:47280) 2024-02-21T23:06:26.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.978+0000 7f4703400640 1 -- 172.21.15.28:0/3926358494 learned_addr learned my addr 172.21.15.28:0/3926358494 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:26.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.979+0000 7f4703e00640 1 --2- 172.21.15.28:0/3926358494 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f470406e9d0 0x7f47041edd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:26.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.979+0000 7f4708e00640 1 --2- 172.21.15.28:0/3926358494 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4704158480 0x7f47041505b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:26.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.979+0000 7f4703400640 1 -- 172.21.15.28:0/3926358494 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f470406e9d0 msgr2=0x7f47041edd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:26.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.979+0000 7f4703400640 1 --2- 172.21.15.28:0/3926358494 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f470406e9d0 0x7f47041edd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:26.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.979+0000 7f4703400640 1 -- 172.21.15.28:0/3926358494 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4704158480 msgr2=0x7f47041505b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:26.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.979+0000 7f4703400640 1 --2- 172.21.15.28:0/3926358494 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4704158480 0x7f47041505b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:26.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.979+0000 7f4703400640 1 -- 172.21.15.28:0/3926358494 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f470414fe20 con 0x7f4704152560 2024-02-21T23:06:26.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.979+0000 7f4703400640 1 --2- 172.21.15.28:0/3926358494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4704152560 0x7f4704147340 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f46f002b8f0 tx=0x7f46f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:26.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.980+0000 7f46fbe00640 1 -- 172.21.15.28:0/3926358494 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46f0041070 con 0x7f4704152560 2024-02-21T23:06:26.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.980+0000 7f46fbe00640 1 -- 172.21.15.28:0/3926358494 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46f003ad00 con 0x7f4704152560 2024-02-21T23:06:26.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.980+0000 7f46fbe00640 1 -- 172.21.15.28:0/3926358494 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46f0035040 con 0x7f4704152560 2024-02-21T23:06:26.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.980+0000 7f4709800640 1 -- 172.21.15.28:0/3926358494 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47041ee260 con 0x7f4704152560 2024-02-21T23:06:26.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.980+0000 7f4709800640 1 -- 172.21.15.28:0/3926358494 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f470406b570 con 0x7f4704152560 2024-02-21T23:06:26.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.982+0000 7f4709800640 1 -- 172.21.15.28:0/3926358494 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f470406c000 con 0x7f4704152560 2024-02-21T23:06:26.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.983+0000 7f46fbe00640 1 -- 172.21.15.28:0/3926358494 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f46f004c060 con 0x7f4704152560 2024-02-21T23:06:26.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.984+0000 7f46fbe00640 1 --2- 172.21.15.28:0/3926358494 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f46e406b4d0 0x7f46e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:26.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.984+0000 7f46fbe00640 1 -- 172.21.15.28:0/3926358494 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7159+0+0 (secure 0 0 0) 0x7f46f00ac3d0 con 0x7f4704152560 2024-02-21T23:06:26.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.985+0000 7f4703e00640 1 --2- 172.21.15.28:0/3926358494 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f46e406b4d0 0x7f46e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:26.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.986+0000 7f4703e00640 1 --2- 172.21.15.28:0/3926358494 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f46e406b4d0 0x7f46e406d990 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f46f4001640 tx=0x7f46f4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:26.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:26.989+0000 7f46fbe00640 1 -- 172.21.15.28:0/3926358494 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f46f004b2c0 con 0x7f4704152560 2024-02-21T23:06:27.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.191+0000 7f4709800640 1 -- 172.21.15.28:0/3926358494 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f470406e0f0 con 0x7f4704152560 2024-02-21T23:06:27.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.192+0000 7f46fbe00640 1 -- 172.21.15.28:0/3926358494 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v83) v1 ==== 104+0+11 (secure 0 0 0) 0x7f46f007ac90 con 0x7f4704152560 2024-02-21T23:06:27.180 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-02-21T23:06:27.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.194+0000 7f4709800640 1 -- 172.21.15.28:0/3926358494 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f46e406b4d0 msgr2=0x7f46e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:27.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.194+0000 7f4709800640 1 --2- 172.21.15.28:0/3926358494 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f46e406b4d0 0x7f46e406d990 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f46f4001640 tx=0x7f46f4009040 comp rx=0 tx=0).stop 2024-02-21T23:06:27.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.194+0000 7f4709800640 1 -- 172.21.15.28:0/3926358494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4704152560 msgr2=0x7f4704147340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:27.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.194+0000 7f4709800640 1 --2- 172.21.15.28:0/3926358494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4704152560 0x7f4704147340 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f46f002b8f0 tx=0x7f46f00046c0 comp rx=0 tx=0).stop 2024-02-21T23:06:27.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.194+0000 7f4709800640 1 -- 172.21.15.28:0/3926358494 shutdown_connections 2024-02-21T23:06:27.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.194+0000 7f4709800640 1 --2- 172.21.15.28:0/3926358494 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f46e406b4d0 0x7f46e406d990 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:27.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.194+0000 7f4709800640 1 --2- 172.21.15.28:0/3926358494 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4704158480 0x7f47041505b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:27.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.195+0000 7f4709800640 1 --2- 172.21.15.28:0/3926358494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4704152560 0x7f4704147340 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:27.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.195+0000 7f4709800640 1 --2- 172.21.15.28:0/3926358494 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f470406e9d0 0x7f47041edd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:27.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.195+0000 7f4709800640 1 -- 172.21.15.28:0/3926358494 >> 172.21.15.28:0/3926358494 conn(0x7f47040a8530 msgr2=0x7f47040c3180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:27.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.195+0000 7f4709800640 1 -- 172.21.15.28:0/3926358494 shutdown_connections 2024-02-21T23:06:27.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.195+0000 7f4709800640 1 -- 172.21.15.28:0/3926358494 wait complete. 2024-02-21T23:06:27.205 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-21T23:06:27.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.438+0000 7f68f7200640 1 Processor -- start 2024-02-21T23:06:27.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.439+0000 7f68f7200640 1 -- start start 2024-02-21T23:06:27.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.440+0000 7f68f7200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f68f80c4580 0x7f68f80c12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:27.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.440+0000 7f68f7200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f68f80c2c10 0x7f68f806ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:27.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.440+0000 7f68f7200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68f806e220 con 0x7f68f8150240 2024-02-21T23:06:27.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.440+0000 7f68f7200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68f8150620 con 0x7f68f80c4580 2024-02-21T23:06:27.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.440+0000 7f68f7200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68f8153a90 con 0x7f68f80c2c10 2024-02-21T23:06:27.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.441+0000 7f68f6800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f68f80c2c10 0x7f68f806ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:27.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.441+0000 7f68f5400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f68f80c4580 0x7f68f80c12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:27.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.441+0000 7f68f5400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f68f80c4580 0x7f68f80c12c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.28:37792/0 (socket says 172.21.15.28:37792) 2024-02-21T23:06:27.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.441+0000 7f68f5400640 1 -- 172.21.15.28:0/1506520410 learned_addr learned my addr 172.21.15.28:0/1506520410 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:27.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.442+0000 7f68f5400640 1 -- 172.21.15.28:0/1506520410 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f68f80c2c10 msgr2=0x7f68f806ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:27.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.442+0000 7f68f5400640 1 --2- 172.21.15.28:0/1506520410 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f68f80c2c10 0x7f68f806ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:27.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.442+0000 7f68f5400640 1 -- 172.21.15.28:0/1506520410 >> v1:172.21.15.81:6789/0 conn(0x7f68f8150240 legacy=0x7f68f806db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:27.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.442+0000 7f68f5400640 1 -- 172.21.15.28:0/1506520410 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68f806f780 con 0x7f68f80c4580 2024-02-21T23:06:27.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.442+0000 7f68f6800640 1 --2- 172.21.15.28:0/1506520410 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f68f80c2c10 0x7f68f806ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-21T23:06:27.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.443+0000 7f68f5400640 1 --2- 172.21.15.28:0/1506520410 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f68f80c4580 0x7f68f80c12c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f68e4009870 tx=0x7f68e402b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=419249214fb65065 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:27.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.444+0000 7f68f4a00640 1 -- 172.21.15.28:0/1506520410 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68e4035020 con 0x7f68f80c4580 2024-02-21T23:06:27.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.444+0000 7f68f4a00640 1 -- 172.21.15.28:0/1506520410 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68e4031640 con 0x7f68f80c4580 2024-02-21T23:06:27.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.444+0000 7f68f4a00640 1 -- 172.21.15.28:0/1506520410 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68e402d040 con 0x7f68f80c4580 2024-02-21T23:06:27.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.444+0000 7f68f7200640 1 -- 172.21.15.28:0/1506520410 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f68f80c4580 msgr2=0x7f68f80c12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:27.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.444+0000 7f68f7200640 1 --2- 172.21.15.28:0/1506520410 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f68f80c4580 0x7f68f80c12c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f68e4009870 tx=0x7f68e402b8f0 comp rx=0 tx=0).stop 2024-02-21T23:06:27.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.444+0000 7f68f7200640 1 -- 172.21.15.28:0/1506520410 shutdown_connections 2024-02-21T23:06:27.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.444+0000 7f68f7200640 1 --2- 172.21.15.28:0/1506520410 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f68f80c2c10 0x7f68f806ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:27.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.444+0000 7f68f7200640 1 --2- 172.21.15.28:0/1506520410 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f68f80c4580 0x7f68f80c12c0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:27.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.444+0000 7f68f7200640 1 -- 172.21.15.28:0/1506520410 >> 172.21.15.28:0/1506520410 conn(0x7f68f80a8530 msgr2=0x7f68f80a8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:27.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.444+0000 7f68f7200640 1 -- 172.21.15.28:0/1506520410 shutdown_connections 2024-02-21T23:06:27.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.444+0000 7f68f7200640 1 -- 172.21.15.28:0/1506520410 wait complete. 2024-02-21T23:06:27.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.446+0000 7f68f7200640 1 Processor -- start 2024-02-21T23:06:27.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.446+0000 7f68f7200640 1 -- start start 2024-02-21T23:06:27.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.447+0000 7f68f7200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f68f80c2c10 0x7f68f81dcbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:27.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.447+0000 7f68f7200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68f80c4580 0x7f68f814bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:27.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.447+0000 7f68f7200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f68f8150240 0x7f68f8151890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:27.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.447+0000 7f68f7200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68f80c4a10 con 0x7f68f80c4580 2024-02-21T23:06:27.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.447+0000 7f68f7200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68f8153720 con 0x7f68f80c2c10 2024-02-21T23:06:27.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.447+0000 7f68f7200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68f80c4b90 con 0x7f68f8150240 2024-02-21T23:06:27.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.447+0000 7f68f5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68f80c4580 0x7f68f814bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:27.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.448+0000 7f68f5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68f80c4580 0x7f68f814bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47282/0 (socket says 172.21.15.28:47282) 2024-02-21T23:06:27.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.448+0000 7f68f5400640 1 -- 172.21.15.28:0/3274112822 learned_addr learned my addr 172.21.15.28:0/3274112822 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:27.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.448+0000 7f68f6800640 1 --2- 172.21.15.28:0/3274112822 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f68f8150240 0x7f68f8151890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:27.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.448+0000 7f68f5e00640 1 --2- 172.21.15.28:0/3274112822 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f68f80c2c10 0x7f68f81dcbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:27.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.448+0000 7f68f5400640 1 -- 172.21.15.28:0/3274112822 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f68f8150240 msgr2=0x7f68f8151890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:27.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.448+0000 7f68f5400640 1 --2- 172.21.15.28:0/3274112822 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f68f8150240 0x7f68f8151890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:27.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.448+0000 7f68f5400640 1 -- 172.21.15.28:0/3274112822 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f68f80c2c10 msgr2=0x7f68f81dcbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:27.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.448+0000 7f68f5400640 1 --2- 172.21.15.28:0/3274112822 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f68f80c2c10 0x7f68f81dcbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:27.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.448+0000 7f68f5400640 1 -- 172.21.15.28:0/3274112822 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68f81564d0 con 0x7f68f80c4580 2024-02-21T23:06:27.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.448+0000 7f68f6800640 1 --2- 172.21.15.28:0/3274112822 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f68f8150240 0x7f68f8151890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:06:27.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.448+0000 7f68f5e00640 1 --2- 172.21.15.28:0/3274112822 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f68f80c2c10 0x7f68f81dcbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:06:27.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.449+0000 7f68f5400640 1 --2- 172.21.15.28:0/3274112822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68f80c4580 0x7f68f814bd90 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f68e4000c00 tx=0x7f68e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:27.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.449+0000 7f68ee000640 1 -- 172.21.15.28:0/3274112822 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68e4040070 con 0x7f68f80c4580 2024-02-21T23:06:27.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.449+0000 7f68ee000640 1 -- 172.21.15.28:0/3274112822 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68e40399f0 con 0x7f68f80c4580 2024-02-21T23:06:27.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.449+0000 7f68f7200640 1 -- 172.21.15.28:0/3274112822 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68f8152f70 con 0x7f68f80c4580 2024-02-21T23:06:27.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.449+0000 7f68ee000640 1 -- 172.21.15.28:0/3274112822 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68e4035040 con 0x7f68f80c4580 2024-02-21T23:06:27.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.449+0000 7f68f7200640 1 -- 172.21.15.28:0/3274112822 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68f81dc270 con 0x7f68f80c4580 2024-02-21T23:06:27.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.451+0000 7f68f7200640 1 -- 172.21.15.28:0/3274112822 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68f806bcb0 con 0x7f68f80c4580 2024-02-21T23:06:27.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.452+0000 7f68ee000640 1 -- 172.21.15.28:0/3274112822 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f68e404b060 con 0x7f68f80c4580 2024-02-21T23:06:27.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.453+0000 7f68ee000640 1 --2- 172.21.15.28:0/3274112822 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f68d806b4d0 0x7f68d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:27.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.453+0000 7f68ee000640 1 -- 172.21.15.28:0/3274112822 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7159+0+0 (secure 0 0 0) 0x7f68e40ac0a0 con 0x7f68f80c4580 2024-02-21T23:06:27.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.454+0000 7f68f5e00640 1 --2- 172.21.15.28:0/3274112822 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f68d806b4d0 0x7f68d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:27.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.455+0000 7f68f5e00640 1 --2- 172.21.15.28:0/3274112822 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f68d806b4d0 0x7f68d806d990 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f68f81eed00 tx=0x7f68e8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:27.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.457+0000 7f68ee000640 1 -- 172.21.15.28:0/3274112822 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f68f806bcb0 con 0x7f68f80c4580 2024-02-21T23:06:27.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.658+0000 7f68f7200640 1 -- 172.21.15.28:0/3274112822 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f68f804f290 con 0x7f68f80c4580 2024-02-21T23:06:27.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.659+0000 7f68ee000640 1 -- 172.21.15.28:0/3274112822 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v83) v1 ==== 100+0+11 (secure 0 0 0) 0x7f68e407a960 con 0x7f68f80c4580 2024-02-21T23:06:27.647 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-02-21T23:06:27.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.661+0000 7f68f7200640 1 -- 172.21.15.28:0/3274112822 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f68d806b4d0 msgr2=0x7f68d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:27.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.661+0000 7f68f7200640 1 --2- 172.21.15.28:0/3274112822 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f68d806b4d0 0x7f68d806d990 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f68f81eed00 tx=0x7f68e8009040 comp rx=0 tx=0).stop 2024-02-21T23:06:27.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.661+0000 7f68f7200640 1 -- 172.21.15.28:0/3274112822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68f80c4580 msgr2=0x7f68f814bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:27.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.661+0000 7f68f7200640 1 --2- 172.21.15.28:0/3274112822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68f80c4580 0x7f68f814bd90 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f68e4000c00 tx=0x7f68e40046c0 comp rx=0 tx=0).stop 2024-02-21T23:06:27.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.661+0000 7f68f7200640 1 -- 172.21.15.28:0/3274112822 shutdown_connections 2024-02-21T23:06:27.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.661+0000 7f68f7200640 1 --2- 172.21.15.28:0/3274112822 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f68d806b4d0 0x7f68d806d990 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:27.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.661+0000 7f68f7200640 1 --2- 172.21.15.28:0/3274112822 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f68f8150240 0x7f68f8151890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:27.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.661+0000 7f68f7200640 1 --2- 172.21.15.28:0/3274112822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68f80c4580 0x7f68f814bd90 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:27.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.661+0000 7f68f7200640 1 --2- 172.21.15.28:0/3274112822 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f68f80c2c10 0x7f68f81dcbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:27.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.661+0000 7f68f7200640 1 -- 172.21.15.28:0/3274112822 >> 172.21.15.28:0/3274112822 conn(0x7f68f80a8530 msgr2=0x7f68f814dd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:27.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.661+0000 7f68f7200640 1 -- 172.21.15.28:0/3274112822 shutdown_connections 2024-02-21T23:06:27.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.662+0000 7f68f7200640 1 -- 172.21.15.28:0/3274112822 wait complete. 2024-02-21T23:06:27.669 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs2_metadata pg_num 2024-02-21T23:06:27.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.959+0000 7f3088e00640 1 Processor -- start 2024-02-21T23:06:27.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.959+0000 7f3088e00640 1 -- start start 2024-02-21T23:06:27.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.961+0000 7f3088e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3084150240 0x7f30840c12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:27.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.961+0000 7f3088e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f30840c4580 0x7f308406db10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:27.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.961+0000 7f3088e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f308406f780 con 0x7f30840c2c10 2024-02-21T23:06:27.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.961+0000 7f3088e00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3084150620 con 0x7f30840c4580 2024-02-21T23:06:27.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.961+0000 7f3088e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3084153a90 con 0x7f3084150240 2024-02-21T23:06:27.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.961+0000 7f3083e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f30840c2c10 0x7f308406ae70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37458/0 (socket says 172.21.15.28:37458) 2024-02-21T23:06:27.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.961+0000 7f3083e00640 1 -- 172.21.15.28:0/376151035 learned_addr learned my addr 172.21.15.28:0/376151035 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:06:27.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.961+0000 7f3082a00640 1 --2- 172.21.15.28:0/376151035 >> v2:172.21.15.81:3300/0 conn(0x7f30840c4580 0x7f308406db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:27.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.961+0000 7f3083400640 1 --2- 172.21.15.28:0/376151035 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3084150240 0x7f30840c12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:27.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.962+0000 7f3082000640 1 -- 172.21.15.28:0/376151035 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3868091172 0 0) 0x7f308406f780 con 0x7f30840c2c10 2024-02-21T23:06:27.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.962+0000 7f3082000640 1 -- 172.21.15.28:0/376151035 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3068003520 con 0x7f30840c2c10 2024-02-21T23:06:27.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.962+0000 7f3082000640 1 -- 172.21.15.28:0/376151035 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2725643245 0 0) 0x7f3068003520 con 0x7f30840c2c10 2024-02-21T23:06:27.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.962+0000 7f3082000640 1 -- 172.21.15.28:0/376151035 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3084150240 msgr2=0x7f30840c12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:27.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.962+0000 7f3082000640 1 --2- 172.21.15.28:0/376151035 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3084150240 0x7f30840c12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:27.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.962+0000 7f3082000640 1 -- 172.21.15.28:0/376151035 >> v2:172.21.15.81:3300/0 conn(0x7f30840c4580 msgr2=0x7f308406db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:27.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.962+0000 7f3082000640 1 --2- 172.21.15.28:0/376151035 >> v2:172.21.15.81:3300/0 conn(0x7f30840c4580 0x7f308406db10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:27.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.962+0000 7f3082000640 1 -- 172.21.15.28:0/376151035 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f308406f4a0 con 0x7f30840c2c10 2024-02-21T23:06:27.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.962+0000 7f3082000640 1 -- 172.21.15.28:0/376151035 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f307c0019f0 con 0x7f30840c2c10 2024-02-21T23:06:27.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.962+0000 7f3083400640 1 --2- 172.21.15.28:0/376151035 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3084150240 0x7f30840c12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-21T23:06:27.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.962+0000 7f3082000640 1 -- 172.21.15.28:0/376151035 >> v1:172.21.15.28:6789/0 conn(0x7f30840c2c10 legacy=0x7f308406ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:27.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.962+0000 7f3082000640 1 --2- 172.21.15.28:0/376151035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30680039b0 0x7f3068003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:27.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.963+0000 7f3082000640 1 --2- 172.21.15.28:0/376151035 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3084150240 0x7f3068004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:27.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.963+0000 7f3082000640 1 --2- 172.21.15.28:0/376151035 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3068004d40 0x7f30680051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:27.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.963+0000 7f3082000640 1 -- 172.21.15.28:0/376151035 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3084153a90 con 0x7f30680039b0 2024-02-21T23:06:27.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.963+0000 7f3082000640 1 -- 172.21.15.28:0/376151035 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f306800b9c0 con 0x7f3068004d40 2024-02-21T23:06:27.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.963+0000 7f3082000640 1 -- 172.21.15.28:0/376151035 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f306800bb40 con 0x7f3084150240 2024-02-21T23:06:27.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.963+0000 7f3083400640 1 --2- 172.21.15.28:0/376151035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30680039b0 0x7f3068003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:27.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.963+0000 7f3082a00640 1 --2- 172.21.15.28:0/376151035 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3084150240 0x7f3068004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:27.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.963+0000 7f3083e00640 1 --2- 172.21.15.28:0/376151035 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3068004d40 0x7f30680051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:27.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.963+0000 7f3083400640 1 -- 172.21.15.28:0/376151035 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3084150240 msgr2=0x7f3068004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:27.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.963+0000 7f3083400640 1 --2- 172.21.15.28:0/376151035 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3084150240 0x7f3068004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:27.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.963+0000 7f3083400640 1 -- 172.21.15.28:0/376151035 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3068004d40 msgr2=0x7f30680051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:27.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.963+0000 7f3083400640 1 --2- 172.21.15.28:0/376151035 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3068004d40 0x7f30680051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:27.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.963+0000 7f3083400640 1 -- 172.21.15.28:0/376151035 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3068003520 con 0x7f30680039b0 2024-02-21T23:06:27.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.963+0000 7f3083e00640 1 --2- 172.21.15.28:0/376151035 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3068004d40 0x7f30680051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:06:27.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.965+0000 7f3083400640 1 --2- 172.21.15.28:0/376151035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30680039b0 0x7f3068003dd0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f3074009cc0 tx=0x7f3068005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e67984c74e15f511 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:27.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.966+0000 7f3082000640 1 -- 172.21.15.28:0/376151035 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3074039070 con 0x7f30680039b0 2024-02-21T23:06:27.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.966+0000 7f3082000640 1 -- 172.21.15.28:0/376151035 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f307402c8f0 con 0x7f30680039b0 2024-02-21T23:06:27.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.966+0000 7f3082000640 1 -- 172.21.15.28:0/376151035 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3074009860 con 0x7f30680039b0 2024-02-21T23:06:27.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.966+0000 7f3088e00640 1 -- 172.21.15.28:0/376151035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30680039b0 msgr2=0x7f3068003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:27.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.966+0000 7f3088e00640 1 --2- 172.21.15.28:0/376151035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30680039b0 0x7f3068003dd0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f3074009cc0 tx=0x7f3068005830 comp rx=0 tx=0).stop 2024-02-21T23:06:27.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.966+0000 7f3083e00640 1 -- 172.21.15.28:0/376151035 reap_dead start 2024-02-21T23:06:27.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.966+0000 7f3088e00640 1 -- 172.21.15.28:0/376151035 shutdown_connections 2024-02-21T23:06:27.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.966+0000 7f3088e00640 1 -- 172.21.15.28:0/376151035 >> 172.21.15.28:0/376151035 conn(0x7f30840a8530 msgr2=0x7f30840a8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:27.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.966+0000 7f3088e00640 1 -- 172.21.15.28:0/376151035 shutdown_connections 2024-02-21T23:06:27.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.967+0000 7f3088e00640 1 -- 172.21.15.28:0/376151035 wait complete. 2024-02-21T23:06:27.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.968+0000 7f3088e00640 1 Processor -- start 2024-02-21T23:06:27.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.968+0000 7f3088e00640 1 -- start start 2024-02-21T23:06:27.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.969+0000 7f3088e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f30680039b0 0x7f30840724b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:27.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.969+0000 7f3088e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3084153c40 0x7f3084156020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:27.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.969+0000 7f3088e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3084156560 0x7f30841edd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:27.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.969+0000 7f3088e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30840c4b90 con 0x7f3084153c40 2024-02-21T23:06:27.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.969+0000 7f3088e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30840c4a10 con 0x7f3084156560 2024-02-21T23:06:27.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.969+0000 7f3088e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f308406b580 con 0x7f30680039b0 2024-02-21T23:06:27.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.970+0000 7f3082a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3084153c40 0x7f3084156020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:27.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.970+0000 7f3082a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3084153c40 0x7f3084156020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47302/0 (socket says 172.21.15.28:47302) 2024-02-21T23:06:27.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.970+0000 7f3082a00640 1 -- 172.21.15.28:0/2808773003 learned_addr learned my addr 172.21.15.28:0/2808773003 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:27.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.970+0000 7f3083400640 1 --2- 172.21.15.28:0/2808773003 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f30680039b0 0x7f30840724b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:27.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.970+0000 7f3083e00640 1 --2- 172.21.15.28:0/2808773003 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3084156560 0x7f30841edd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:27.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.970+0000 7f3082a00640 1 -- 172.21.15.28:0/2808773003 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f30680039b0 msgr2=0x7f30840724b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:27.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.970+0000 7f3082a00640 1 --2- 172.21.15.28:0/2808773003 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f30680039b0 0x7f30840724b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:27.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.970+0000 7f3082a00640 1 -- 172.21.15.28:0/2808773003 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3084156560 msgr2=0x7f30841edd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:27.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.970+0000 7f3082a00640 1 --2- 172.21.15.28:0/2808773003 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3084156560 0x7f30841edd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:27.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.970+0000 7f3082a00640 1 -- 172.21.15.28:0/2808773003 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30840abf70 con 0x7f3084153c40 2024-02-21T23:06:27.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.970+0000 7f3083400640 1 --2- 172.21.15.28:0/2808773003 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f30680039b0 0x7f30840724b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:06:27.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.971+0000 7f3082a00640 1 --2- 172.21.15.28:0/2808773003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3084153c40 0x7f3084156020 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f30700115e0 tx=0x7f3070011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:27.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.971+0000 7f307b400640 1 -- 172.21.15.28:0/2808773003 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3070004220 con 0x7f3084153c40 2024-02-21T23:06:27.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.971+0000 7f307b400640 1 -- 172.21.15.28:0/2808773003 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30700043c0 con 0x7f3084153c40 2024-02-21T23:06:27.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.971+0000 7f3088e00640 1 -- 172.21.15.28:0/2808773003 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3084073000 con 0x7f3084153c40 2024-02-21T23:06:27.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.971+0000 7f307b400640 1 -- 172.21.15.28:0/2808773003 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f307000f930 con 0x7f3084153c40 2024-02-21T23:06:27.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.971+0000 7f3088e00640 1 -- 172.21.15.28:0/2808773003 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30841eb3f0 con 0x7f3084153c40 2024-02-21T23:06:27.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.972+0000 7f3078c00640 1 -- 172.21.15.28:0/2808773003 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f308404f290 con 0x7f3084153c40 2024-02-21T23:06:27.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.974+0000 7f307b400640 1 -- 172.21.15.28:0/2808773003 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f307000fad0 con 0x7f3084153c40 2024-02-21T23:06:27.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.974+0000 7f307b400640 1 --2- 172.21.15.28:0/2808773003 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f306406b500 0x7f306406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:27.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.975+0000 7f307b400640 1 -- 172.21.15.28:0/2808773003 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7159+0+0 (secure 0 0 0) 0x7f307008aa80 con 0x7f3084153c40 2024-02-21T23:06:27.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.975+0000 7f3083400640 1 --2- 172.21.15.28:0/2808773003 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f306406b500 0x7f306406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:27.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.976+0000 7f3083400640 1 --2- 172.21.15.28:0/2808773003 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f306406b500 0x7f306406d9c0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f3074036000 tx=0x7f3074030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:27.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:27.983+0000 7f307b400640 1 -- 172.21.15.28:0/2808773003 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f307008ce20 con 0x7f3084153c40 2024-02-21T23:06:28.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.185+0000 7f3078c00640 1 -- 172.21.15.28:0/2808773003 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs2_metadata", "var": "pg_num"} v 0) v1 -- 0x7f308406f0e0 con 0x7f3084153c40 2024-02-21T23:06:28.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.186+0000 7f307b400640 1 -- 172.21.15.28:0/2808773003 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs2_metadata", "var": "pg_num"}]=0 v83) v1 ==== 105+0+11 (secure 0 0 0) 0x7f3070019030 con 0x7f3084153c40 2024-02-21T23:06:28.174 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-02-21T23:06:28.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.188+0000 7f3078c00640 1 -- 172.21.15.28:0/2808773003 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f306406b500 msgr2=0x7f306406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:28.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.188+0000 7f3078c00640 1 --2- 172.21.15.28:0/2808773003 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f306406b500 0x7f306406d9c0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f3074036000 tx=0x7f3074030000 comp rx=0 tx=0).stop 2024-02-21T23:06:28.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.188+0000 7f3078c00640 1 -- 172.21.15.28:0/2808773003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3084153c40 msgr2=0x7f3084156020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:28.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.188+0000 7f3078c00640 1 --2- 172.21.15.28:0/2808773003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3084153c40 0x7f3084156020 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f30700115e0 tx=0x7f3070011aa0 comp rx=0 tx=0).stop 2024-02-21T23:06:28.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.189+0000 7f3078c00640 1 -- 172.21.15.28:0/2808773003 shutdown_connections 2024-02-21T23:06:28.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.189+0000 7f3078c00640 1 --2- 172.21.15.28:0/2808773003 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f306406b500 0x7f306406d9c0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:28.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.189+0000 7f3078c00640 1 --2- 172.21.15.28:0/2808773003 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3084156560 0x7f30841edd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:28.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.189+0000 7f3078c00640 1 --2- 172.21.15.28:0/2808773003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3084153c40 0x7f3084156020 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:28.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.189+0000 7f3078c00640 1 --2- 172.21.15.28:0/2808773003 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f30680039b0 0x7f30840724b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:28.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.189+0000 7f3078c00640 1 -- 172.21.15.28:0/2808773003 >> 172.21.15.28:0/2808773003 conn(0x7f30840a8530 msgr2=0x7f3084151f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:28.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.189+0000 7f3078c00640 1 -- 172.21.15.28:0/2808773003 shutdown_connections 2024-02-21T23:06:28.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.189+0000 7f3078c00640 1 -- 172.21.15.28:0/2808773003 wait complete. 2024-02-21T23:06:28.197 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs2_data pg_num 2024-02-21T23:06:28.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.421+0000 7fca9ac00640 1 Processor -- start 2024-02-21T23:06:28.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.421+0000 7fca9ac00640 1 -- start start 2024-02-21T23:06:28.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.422+0000 7fca9ac00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fca9413e0c0 0x7fca940c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:28.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.422+0000 7fca9ac00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fca940c76f0 0x7fca9406c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:28.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.422+0000 7fca9ac00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca94158550 con 0x7fca9406dbc0 2024-02-21T23:06:28.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.422+0000 7fca9ac00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca940c89a0 con 0x7fca940c76f0 2024-02-21T23:06:28.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.422+0000 7fca9ac00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca941587d0 con 0x7fca9413e0c0 2024-02-21T23:06:28.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.423+0000 7fca9a200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fca940c76f0 0x7fca9406c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:28.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.423+0000 7fca9a200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fca940c76f0 0x7fca9406c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47318/0 (socket says 172.21.15.28:47318) 2024-02-21T23:06:28.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.423+0000 7fca9a200640 1 -- 172.21.15.28:0/3491679985 learned_addr learned my addr 172.21.15.28:0/3491679985 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:28.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.423+0000 7fca99800640 1 --2- 172.21.15.28:0/3491679985 >> v2:172.21.15.81:3300/0 conn(0x7fca9413e0c0 0x7fca940c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:28.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.423+0000 7fca9a200640 1 -- 172.21.15.28:0/3491679985 >> v2:172.21.15.81:3300/0 conn(0x7fca9413e0c0 msgr2=0x7fca940c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:28.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.423+0000 7fca9a200640 1 --2- 172.21.15.28:0/3491679985 >> v2:172.21.15.81:3300/0 conn(0x7fca9413e0c0 0x7fca940c71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:28.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.423+0000 7fca9a200640 1 -- 172.21.15.28:0/3491679985 >> v1:172.21.15.81:6789/0 conn(0x7fca9406dbc0 legacy=0x7fca9406c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:28.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.423+0000 7fca9a200640 1 -- 172.21.15.28:0/3491679985 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca94158ab0 con 0x7fca940c76f0 2024-02-21T23:06:28.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.425+0000 7fca9a200640 1 --2- 172.21.15.28:0/3491679985 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca940c76f0 0x7fca9406c990 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fca940ad180 tx=0x7fca8c02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2368a8804a7adc9e server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:28.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.425+0000 7fca93e00640 1 -- 172.21.15.28:0/3491679985 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca8c02c9f0 con 0x7fca940c76f0 2024-02-21T23:06:28.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.425+0000 7fca93e00640 1 -- 172.21.15.28:0/3491679985 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca8c02cb90 con 0x7fca940c76f0 2024-02-21T23:06:28.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.425+0000 7fca93e00640 1 -- 172.21.15.28:0/3491679985 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca8c0343f0 con 0x7fca940c76f0 2024-02-21T23:06:28.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.426+0000 7fca9ac00640 1 -- 172.21.15.28:0/3491679985 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca940c76f0 msgr2=0x7fca9406c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:28.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.426+0000 7fca9ac00640 1 --2- 172.21.15.28:0/3491679985 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca940c76f0 0x7fca9406c990 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fca940ad180 tx=0x7fca8c02be50 comp rx=0 tx=0).stop 2024-02-21T23:06:28.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.426+0000 7fca9ac00640 1 -- 172.21.15.28:0/3491679985 shutdown_connections 2024-02-21T23:06:28.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.426+0000 7fca9ac00640 1 --2- 172.21.15.28:0/3491679985 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca940c76f0 0x7fca9406c990 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:28.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.426+0000 7fca9ac00640 1 --2- 172.21.15.28:0/3491679985 >> v2:172.21.15.81:3300/0 conn(0x7fca9413e0c0 0x7fca940c71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:28.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.426+0000 7fca9ac00640 1 -- 172.21.15.28:0/3491679985 >> 172.21.15.28:0/3491679985 conn(0x7fca94073110 msgr2=0x7fca940ac490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:28.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.426+0000 7fca9ac00640 1 -- 172.21.15.28:0/3491679985 shutdown_connections 2024-02-21T23:06:28.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.426+0000 7fca9ac00640 1 -- 172.21.15.28:0/3491679985 wait complete. 2024-02-21T23:06:28.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.429+0000 7fca9ac00640 1 Processor -- start 2024-02-21T23:06:28.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.429+0000 7fca9ac00640 1 -- start start 2024-02-21T23:06:28.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.430+0000 7fca9ac00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fca9406dbc0 0x7fca94136150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:28.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.430+0000 7fca9ac00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fca940c76f0 0x7fca941355c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:28.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.431+0000 7fca9ac00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca9413e0c0 0x7fca940aeff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:28.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.431+0000 7fca9ac00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca9406ced0 con 0x7fca9413e0c0 2024-02-21T23:06:28.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.431+0000 7fca9ac00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca94159070 con 0x7fca940c76f0 2024-02-21T23:06:28.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.431+0000 7fca9ac00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca9413e520 con 0x7fca9406dbc0 2024-02-21T23:06:28.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.431+0000 7fca9a200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca9413e0c0 0x7fca940aeff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:28.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.431+0000 7fca9a200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca9413e0c0 0x7fca940aeff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47322/0 (socket says 172.21.15.28:47322) 2024-02-21T23:06:28.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.431+0000 7fca9a200640 1 -- 172.21.15.28:0/97608499 learned_addr learned my addr 172.21.15.28:0/97608499 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:28.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.431+0000 7fca99800640 1 --2- 172.21.15.28:0/97608499 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fca9406dbc0 0x7fca94136150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:28.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.431+0000 7fca98e00640 1 --2- 172.21.15.28:0/97608499 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fca940c76f0 0x7fca941355c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:28.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.431+0000 7fca9a200640 1 -- 172.21.15.28:0/97608499 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fca9406dbc0 msgr2=0x7fca94136150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:28.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.431+0000 7fca9a200640 1 --2- 172.21.15.28:0/97608499 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fca9406dbc0 0x7fca94136150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:28.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.431+0000 7fca9a200640 1 -- 172.21.15.28:0/97608499 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fca940c76f0 msgr2=0x7fca941355c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:28.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.431+0000 7fca9a200640 1 --2- 172.21.15.28:0/97608499 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fca940c76f0 0x7fca941355c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:28.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.431+0000 7fca9a200640 1 -- 172.21.15.28:0/97608499 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca94135b00 con 0x7fca9413e0c0 2024-02-21T23:06:28.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.431+0000 7fca98e00640 1 --2- 172.21.15.28:0/97608499 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fca940c76f0 0x7fca941355c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:06:28.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.432+0000 7fca9a200640 1 --2- 172.21.15.28:0/97608499 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca9413e0c0 0x7fca940aeff0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fca8c036040 tx=0x7fca8c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:28.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.432+0000 7fca91600640 1 -- 172.21.15.28:0/97608499 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca8c043020 con 0x7fca9413e0c0 2024-02-21T23:06:28.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.432+0000 7fca91600640 1 -- 172.21.15.28:0/97608499 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca8c03e450 con 0x7fca9413e0c0 2024-02-21T23:06:28.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.432+0000 7fca91600640 1 -- 172.21.15.28:0/97608499 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca8c009510 con 0x7fca9413e0c0 2024-02-21T23:06:28.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.432+0000 7fca9ac00640 1 -- 172.21.15.28:0/97608499 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca941536a0 con 0x7fca9413e0c0 2024-02-21T23:06:28.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.432+0000 7fca9ac00640 1 -- 172.21.15.28:0/97608499 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca940af530 con 0x7fca9413e0c0 2024-02-21T23:06:28.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.434+0000 7fca9ac00640 1 -- 172.21.15.28:0/97608499 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca9404f290 con 0x7fca9413e0c0 2024-02-21T23:06:28.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.434+0000 7fca91600640 1 -- 172.21.15.28:0/97608499 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fca8c046020 con 0x7fca9413e0c0 2024-02-21T23:06:28.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.435+0000 7fca91600640 1 --2- 172.21.15.28:0/97608499 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fca7406b4d0 0x7fca7406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:28.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.435+0000 7fca91600640 1 -- 172.21.15.28:0/97608499 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7159+0+0 (secure 0 0 0) 0x7fca8c0acf70 con 0x7fca9413e0c0 2024-02-21T23:06:28.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.436+0000 7fca99800640 1 --2- 172.21.15.28:0/97608499 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fca7406b4d0 0x7fca7406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:28.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.437+0000 7fca99800640 1 --2- 172.21.15.28:0/97608499 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fca7406b4d0 0x7fca7406d990 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fca88006fd0 tx=0x7fca88008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:28.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.440+0000 7fca91600640 1 -- 172.21.15.28:0/97608499 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fca8c04d030 con 0x7fca9413e0c0 2024-02-21T23:06:28.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.649+0000 7fca9ac00640 1 -- 172.21.15.28:0/97608499 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs2_data", "var": "pg_num"} v 0) v1 -- 0x7fca940c65a0 con 0x7fca9413e0c0 2024-02-21T23:06:28.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.650+0000 7fca91600640 1 -- 172.21.15.28:0/97608499 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs2_data", "var": "pg_num"}]=0 v83) v1 ==== 101+0+11 (secure 0 0 0) 0x7fca940c65a0 con 0x7fca9413e0c0 2024-02-21T23:06:28.638 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-02-21T23:06:28.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.652+0000 7fca9ac00640 1 -- 172.21.15.28:0/97608499 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fca7406b4d0 msgr2=0x7fca7406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:28.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.652+0000 7fca9ac00640 1 --2- 172.21.15.28:0/97608499 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fca7406b4d0 0x7fca7406d990 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fca88006fd0 tx=0x7fca88008040 comp rx=0 tx=0).stop 2024-02-21T23:06:28.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.652+0000 7fca9ac00640 1 -- 172.21.15.28:0/97608499 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca9413e0c0 msgr2=0x7fca940aeff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:28.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.652+0000 7fca9ac00640 1 --2- 172.21.15.28:0/97608499 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca9413e0c0 0x7fca940aeff0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fca8c036040 tx=0x7fca8c004a40 comp rx=0 tx=0).stop 2024-02-21T23:06:28.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.652+0000 7fca9ac00640 1 -- 172.21.15.28:0/97608499 shutdown_connections 2024-02-21T23:06:28.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.652+0000 7fca9ac00640 1 --2- 172.21.15.28:0/97608499 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fca7406b4d0 0x7fca7406d990 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:28.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.652+0000 7fca9ac00640 1 --2- 172.21.15.28:0/97608499 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca9413e0c0 0x7fca940aeff0 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:28.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.652+0000 7fca9ac00640 1 --2- 172.21.15.28:0/97608499 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fca940c76f0 0x7fca941355c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:28.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.652+0000 7fca9ac00640 1 --2- 172.21.15.28:0/97608499 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fca9406dbc0 0x7fca94136150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:28.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.652+0000 7fca9ac00640 1 -- 172.21.15.28:0/97608499 >> 172.21.15.28:0/97608499 conn(0x7fca94073110 msgr2=0x7fca940c5020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:28.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.652+0000 7fca9ac00640 1 -- 172.21.15.28:0/97608499 shutdown_connections 2024-02-21T23:06:28.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.653+0000 7fca9ac00640 1 -- 172.21.15.28:0/97608499 wait complete. 2024-02-21T23:06:28.658 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:06:28.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.887+0000 7fe890200640 1 Processor -- start 2024-02-21T23:06:28.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.888+0000 7fe890200640 1 -- start start 2024-02-21T23:06:28.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.889+0000 7fe890200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe8880c4650 0x7fe88806e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:28.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.889+0000 7fe890200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fe8880c5be0 0x7fe8880c2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:28.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.889+0000 7fe890200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8880a70d0 con 0x7fe888155ce0 2024-02-21T23:06:28.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.889+0000 7fe890200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe88806edf0 con 0x7fe8880c4650 2024-02-21T23:06:28.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.889+0000 7fe890200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8880c61d0 con 0x7fe8880c5be0 2024-02-21T23:06:28.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.889+0000 7fe88d800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe8880c4650 0x7fe88806e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:28.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.889+0000 7fe88d800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe8880c4650 0x7fe88806e8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47338/0 (socket says 172.21.15.28:47338) 2024-02-21T23:06:28.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.889+0000 7fe88d800640 1 -- 172.21.15.28:0/868724130 learned_addr learned my addr 172.21.15.28:0/868724130 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:28.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.890+0000 7fe88ce00640 1 --2- 172.21.15.28:0/868724130 >> v2:172.21.15.81:3300/0 conn(0x7fe8880c5be0 0x7fe8880c2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:28.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.890+0000 7fe88d800640 1 -- 172.21.15.28:0/868724130 >> v2:172.21.15.81:3300/0 conn(0x7fe8880c5be0 msgr2=0x7fe8880c2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:28.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.890+0000 7fe88d800640 1 --2- 172.21.15.28:0/868724130 >> v2:172.21.15.81:3300/0 conn(0x7fe8880c5be0 0x7fe8880c2cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:28.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.890+0000 7fe88d800640 1 -- 172.21.15.28:0/868724130 >> v1:172.21.15.81:6789/0 conn(0x7fe888155ce0 legacy=0x7fe888072470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:28.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.890+0000 7fe88d800640 1 -- 172.21.15.28:0/868724130 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8880c64b0 con 0x7fe8880c4650 2024-02-21T23:06:28.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.891+0000 7fe88d800640 1 --2- 172.21.15.28:0/868724130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8880c4650 0x7fe88806e8b0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fe88806e630 tx=0x7fe87c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25968598e3e0516f server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:28.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.891+0000 7fe887e00640 1 -- 172.21.15.28:0/868724130 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe87c02e070 con 0x7fe8880c4650 2024-02-21T23:06:28.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.892+0000 7fe887e00640 1 -- 172.21.15.28:0/868724130 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe87c037070 con 0x7fe8880c4650 2024-02-21T23:06:28.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.892+0000 7fe887e00640 1 -- 172.21.15.28:0/868724130 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe87c031a00 con 0x7fe8880c4650 2024-02-21T23:06:28.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.892+0000 7fe890200640 1 -- 172.21.15.28:0/868724130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8880c4650 msgr2=0x7fe88806e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:28.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.892+0000 7fe890200640 1 --2- 172.21.15.28:0/868724130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8880c4650 0x7fe88806e8b0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fe88806e630 tx=0x7fe87c02b880 comp rx=0 tx=0).stop 2024-02-21T23:06:28.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.892+0000 7fe890200640 1 -- 172.21.15.28:0/868724130 shutdown_connections 2024-02-21T23:06:28.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.892+0000 7fe890200640 1 --2- 172.21.15.28:0/868724130 >> v2:172.21.15.81:3300/0 conn(0x7fe8880c5be0 0x7fe8880c2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:28.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.892+0000 7fe890200640 1 --2- 172.21.15.28:0/868724130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8880c4650 0x7fe88806e8b0 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:28.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.892+0000 7fe890200640 1 -- 172.21.15.28:0/868724130 >> 172.21.15.28:0/868724130 conn(0x7fe8880c7950 msgr2=0x7fe8880c7d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:28.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.892+0000 7fe890200640 1 -- 172.21.15.28:0/868724130 shutdown_connections 2024-02-21T23:06:28.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.892+0000 7fe890200640 1 -- 172.21.15.28:0/868724130 wait complete. 2024-02-21T23:06:28.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.895+0000 7fe890200640 1 Processor -- start 2024-02-21T23:06:28.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.895+0000 7fe890200640 1 -- start start 2024-02-21T23:06:28.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.896+0000 7fe890200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8880c4650 0x7fe8881eba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:28.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.896+0000 7fe890200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe8880c5be0 0x7fe8880aa020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:28.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.896+0000 7fe890200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe888155ce0 0x7fe888142660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:28.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.896+0000 7fe890200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8880a6d30 con 0x7fe8880c4650 2024-02-21T23:06:28.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.896+0000 7fe890200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8880c3230 con 0x7fe8880c5be0 2024-02-21T23:06:28.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.896+0000 7fe890200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8880a6eb0 con 0x7fe888155ce0 2024-02-21T23:06:28.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.897+0000 7fe88d800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8880c4650 0x7fe8881eba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:28.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.897+0000 7fe88d800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8880c4650 0x7fe8881eba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47344/0 (socket says 172.21.15.28:47344) 2024-02-21T23:06:28.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.897+0000 7fe88d800640 1 -- 172.21.15.28:0/2468648368 learned_addr learned my addr 172.21.15.28:0/2468648368 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:28.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.897+0000 7fe88d800640 1 -- 172.21.15.28:0/2468648368 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe888155ce0 msgr2=0x7fe888142660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:28.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.897+0000 7fe88d800640 1 --2- 172.21.15.28:0/2468648368 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe888155ce0 0x7fe888142660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:28.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.897+0000 7fe88d800640 1 -- 172.21.15.28:0/2468648368 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe8880c5be0 msgr2=0x7fe8880aa020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:28.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.897+0000 7fe88d800640 1 --2- 172.21.15.28:0/2468648368 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe8880c5be0 0x7fe8880aa020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:28.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.897+0000 7fe88d800640 1 -- 172.21.15.28:0/2468648368 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8880aae70 con 0x7fe8880c4650 2024-02-21T23:06:28.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.897+0000 7fe88d800640 1 --2- 172.21.15.28:0/2468648368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8880c4650 0x7fe8881eba30 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fe87c000c00 tx=0x7fe87c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:28.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.898+0000 7fe885600640 1 -- 172.21.15.28:0/2468648368 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe87c032880 con 0x7fe8880c4650 2024-02-21T23:06:28.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.898+0000 7fe885600640 1 -- 172.21.15.28:0/2468648368 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe87c032a40 con 0x7fe8880c4650 2024-02-21T23:06:28.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.898+0000 7fe890200640 1 -- 172.21.15.28:0/2468648368 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8881eaf50 con 0x7fe8880c4650 2024-02-21T23:06:28.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.898+0000 7fe885600640 1 -- 172.21.15.28:0/2468648368 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe87c032e30 con 0x7fe8880c4650 2024-02-21T23:06:28.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.898+0000 7fe890200640 1 -- 172.21.15.28:0/2468648368 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8880ab150 con 0x7fe8880c4650 2024-02-21T23:06:28.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.899+0000 7fe890200640 1 -- 172.21.15.28:0/2468648368 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe888067f30 con 0x7fe8880c4650 2024-02-21T23:06:28.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.900+0000 7fe885600640 1 -- 172.21.15.28:0/2468648368 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe87c04c060 con 0x7fe8880c4650 2024-02-21T23:06:28.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.901+0000 7fe885600640 1 --2- 172.21.15.28:0/2468648368 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe86806b4d0 0x7fe86806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:28.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.901+0000 7fe885600640 1 -- 172.21.15.28:0/2468648368 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7159+0+0 (secure 0 0 0) 0x7fe87c0ac020 con 0x7fe8880c4650 2024-02-21T23:06:28.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.902+0000 7fe88ce00640 1 --2- 172.21.15.28:0/2468648368 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe86806b4d0 0x7fe86806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:28.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.903+0000 7fe88ce00640 1 --2- 172.21.15.28:0/2468648368 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe86806b4d0 0x7fe86806d990 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fe8881ebf70 tx=0x7fe874008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:28.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:28.909+0000 7fe885600640 1 -- 172.21.15.28:0/2468648368 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe87c040070 con 0x7fe8880c4650 2024-02-21T23:06:29.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.255+0000 7fe890200640 1 -- 172.21.15.28:0/2468648368 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe8880c2680 con 0x7fe8880c4650 2024-02-21T23:06:29.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.256+0000 7fe885600640 1 -- 172.21.15.28:0/2468648368 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+6179 (secure 0 0 0) 0x7fe87c07a980 con 0x7fe8880c4650 2024-02-21T23:06:29.244 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:06:29.244 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":74,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5288,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":8,"addr":"172.21.15.81:6835/2774951107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2774951107},{"type":"v1","addr":"172.21.15.81:6835","nonce":2774951107}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67},{"gid":5294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":8,"addr":"172.21.15.81:6837/4292940632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":4292940632},{"type":"v1","addr":"172.21.15.81:6837","nonce":4292940632}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":74,"flags":146,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":true,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:05:50.828988+0000","modified":"2024-02-21T23:06:26.718835+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6729},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6729":{"gid":6729,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6835/1066741568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":1066741568},{"type":"v1","addr":"172.21.15.28:6835","nonce":1066741568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5},{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:06:10.484421+0000","modified":"2024-02-21T23:06:13.492899+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5539},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5539":{"gid":5539,"name":"c","rank":0,"incarnation":63,"state":"up:active","state_seq":8,"addr":"172.21.15.28:6833/624384999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":624384999},{"type":"v1","addr":"172.21.15.28:6833","nonce":624384999}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-02-21T23:06:29.245 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 74 2024-02-21T23:06:29.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.259+0000 7fe890200640 1 -- 172.21.15.28:0/2468648368 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe86806b4d0 msgr2=0x7fe86806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:29.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.259+0000 7fe890200640 1 --2- 172.21.15.28:0/2468648368 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe86806b4d0 0x7fe86806d990 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fe8881ebf70 tx=0x7fe874008040 comp rx=0 tx=0).stop 2024-02-21T23:06:29.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.259+0000 7fe890200640 1 -- 172.21.15.28:0/2468648368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8880c4650 msgr2=0x7fe8881eba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:29.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.259+0000 7fe890200640 1 --2- 172.21.15.28:0/2468648368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8880c4650 0x7fe8881eba30 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fe87c000c00 tx=0x7fe87c0046c0 comp rx=0 tx=0).stop 2024-02-21T23:06:29.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.259+0000 7fe890200640 1 -- 172.21.15.28:0/2468648368 shutdown_connections 2024-02-21T23:06:29.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.259+0000 7fe890200640 1 --2- 172.21.15.28:0/2468648368 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe86806b4d0 0x7fe86806d990 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:29.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.259+0000 7fe890200640 1 --2- 172.21.15.28:0/2468648368 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe888155ce0 0x7fe888142660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:29.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.259+0000 7fe890200640 1 --2- 172.21.15.28:0/2468648368 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe8880c5be0 0x7fe8880aa020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:29.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.259+0000 7fe890200640 1 --2- 172.21.15.28:0/2468648368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8880c4650 0x7fe8881eba30 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:29.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.259+0000 7fe890200640 1 -- 172.21.15.28:0/2468648368 >> 172.21.15.28:0/2468648368 conn(0x7fe8880c7950 msgr2=0x7fe88814f560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:29.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.259+0000 7fe890200640 1 -- 172.21.15.28:0/2468648368 shutdown_connections 2024-02-21T23:06:29.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.259+0000 7fe890200640 1 -- 172.21.15.28:0/2468648368 wait complete. 2024-02-21T23:06:29.276 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:06:29.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.485+0000 7f624a000640 1 Processor -- start 2024-02-21T23:06:29.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.485+0000 7f624a000640 1 -- start start 2024-02-21T23:06:29.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.488+0000 7f624a000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f62440c3990 0x7f62440cb390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:29.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.488+0000 7f624a000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6244155120 con 0x7f62441525c0 2024-02-21T23:06:29.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.488+0000 7f624a000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6244155390 con 0x7f6244159010 2024-02-21T23:06:29.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.488+0000 7f624a000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6244155670 con 0x7f62440c3990 2024-02-21T23:06:29.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.489+0000 7f6243400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f62441525c0 0x7f62440c6a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37466/0 (socket says 172.21.15.28:37466) 2024-02-21T23:06:29.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.489+0000 7f6243400640 1 -- 172.21.15.28:0/1628348393 learned_addr learned my addr 172.21.15.28:0/1628348393 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:06:29.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.489+0000 7f6243e00640 1 --2- 172.21.15.28:0/1628348393 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f62440c3990 0x7f62440cb390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:29.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.489+0000 7f6242a00640 1 -- 172.21.15.28:0/1628348393 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4226606552 0 0) 0x7f6244155120 con 0x7f62441525c0 2024-02-21T23:06:29.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.489+0000 7f6242a00640 1 -- 172.21.15.28:0/1628348393 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6228003520 con 0x7f62441525c0 2024-02-21T23:06:29.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.490+0000 7f6242a00640 1 -- 172.21.15.28:0/1628348393 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 159651754 0 0) 0x7f6228003520 con 0x7f62441525c0 2024-02-21T23:06:29.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.490+0000 7f6242a00640 1 -- 172.21.15.28:0/1628348393 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f62440c3990 msgr2=0x7f62440cb390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:29.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.490+0000 7f6242a00640 1 --2- 172.21.15.28:0/1628348393 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f62440c3990 0x7f62440cb390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:29.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.490+0000 7f6242a00640 1 -- 172.21.15.28:0/1628348393 >> v1:172.21.15.81:6789/0 conn(0x7f6244159010 legacy=0x7f62440c3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:29.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.490+0000 7f6242a00640 1 -- 172.21.15.28:0/1628348393 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6244155950 con 0x7f62441525c0 2024-02-21T23:06:29.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.490+0000 7f6242a00640 1 -- 172.21.15.28:0/1628348393 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f6234001670 con 0x7f62441525c0 2024-02-21T23:06:29.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.490+0000 7f6242a00640 1 -- 172.21.15.28:0/1628348393 >> v1:172.21.15.28:6789/0 conn(0x7f62441525c0 legacy=0x7f62440c6a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:29.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.490+0000 7f6242a00640 1 --2- 172.21.15.28:0/1628348393 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f62440c3990 0x7f62280039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:29.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.490+0000 7f6242a00640 1 --2- 172.21.15.28:0/1628348393 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6228003f60 0x7f6228004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:29.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.490+0000 7f6242a00640 1 --2- 172.21.15.28:0/1628348393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6228004d40 0x7f62280051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:29.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.490+0000 7f6242a00640 1 -- 172.21.15.28:0/1628348393 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6244155670 con 0x7f6228004d40 2024-02-21T23:06:29.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.490+0000 7f6242a00640 1 -- 172.21.15.28:0/1628348393 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f622800baa0 con 0x7f6228003f60 2024-02-21T23:06:29.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.490+0000 7f6242a00640 1 -- 172.21.15.28:0/1628348393 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f622800bca0 con 0x7f62440c3990 2024-02-21T23:06:29.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.490+0000 7f6243e00640 1 --2- 172.21.15.28:0/1628348393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6228004d40 0x7f62280051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:29.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.490+0000 7f623be00640 1 --2- 172.21.15.28:0/1628348393 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6228003f60 0x7f6228004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:29.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.491+0000 7f6243400640 1 --2- 172.21.15.28:0/1628348393 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f62440c3990 0x7f62280039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:29.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.491+0000 7f6243e00640 1 -- 172.21.15.28:0/1628348393 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f62440c3990 msgr2=0x7f62280039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:29.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.491+0000 7f6243e00640 1 --2- 172.21.15.28:0/1628348393 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f62440c3990 0x7f62280039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:29.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.491+0000 7f6243e00640 1 -- 172.21.15.28:0/1628348393 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6228003f60 msgr2=0x7f6228004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:29.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.491+0000 7f6243e00640 1 --2- 172.21.15.28:0/1628348393 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6228003f60 0x7f6228004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:29.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.491+0000 7f6243e00640 1 -- 172.21.15.28:0/1628348393 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6228003520 con 0x7f6228004d40 2024-02-21T23:06:29.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.491+0000 7f6243400640 1 --2- 172.21.15.28:0/1628348393 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f62440c3990 0x7f62280039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:06:29.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.492+0000 7f6243e00640 1 --2- 172.21.15.28:0/1628348393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6228004d40 0x7f62280051c0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f623c009fc0 tx=0x7f6228005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd8b450c2e9a7ccb server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:29.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.493+0000 7f6242a00640 1 -- 172.21.15.28:0/1628348393 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f623c039070 con 0x7f6228004d40 2024-02-21T23:06:29.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.493+0000 7f6242a00640 1 -- 172.21.15.28:0/1628348393 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f623c02cc20 con 0x7f6228004d40 2024-02-21T23:06:29.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.493+0000 7f6242a00640 1 -- 172.21.15.28:0/1628348393 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f623c009c10 con 0x7f6228004d40 2024-02-21T23:06:29.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.494+0000 7f624a000640 1 -- 172.21.15.28:0/1628348393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6228004d40 msgr2=0x7f62280051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:29.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.494+0000 7f624a000640 1 --2- 172.21.15.28:0/1628348393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6228004d40 0x7f62280051c0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f623c009fc0 tx=0x7f6228005830 comp rx=0 tx=0).stop 2024-02-21T23:06:29.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.494+0000 7f6243e00640 1 -- 172.21.15.28:0/1628348393 reap_dead start 2024-02-21T23:06:29.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.494+0000 7f624a000640 1 -- 172.21.15.28:0/1628348393 shutdown_connections 2024-02-21T23:06:29.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.494+0000 7f624a000640 1 -- 172.21.15.28:0/1628348393 >> 172.21.15.28:0/1628348393 conn(0x7f62440729c0 msgr2=0x7f6244072dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:29.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.494+0000 7f624a000640 1 -- 172.21.15.28:0/1628348393 shutdown_connections 2024-02-21T23:06:29.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.494+0000 7f624a000640 1 -- 172.21.15.28:0/1628348393 wait complete. 2024-02-21T23:06:29.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.496+0000 7f624a000640 1 Processor -- start 2024-02-21T23:06:29.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.496+0000 7f624a000640 1 -- start start 2024-02-21T23:06:29.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.498+0000 7f624a000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6228004d40 0x7f6244157bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:29.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.498+0000 7f624a000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62441482a0 0x7f62440c0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:29.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.498+0000 7f624a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6244147d50 0x7f62441de670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:29.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.498+0000 7f624a000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62441594a0 con 0x7f6244147d50 2024-02-21T23:06:29.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.498+0000 7f624a000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62440caff0 con 0x7f62441482a0 2024-02-21T23:06:29.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.498+0000 7f624a000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62440c7110 con 0x7f6228004d40 2024-02-21T23:06:29.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.498+0000 7f6243e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6244147d50 0x7f62441de670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:29.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.498+0000 7f6243e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6244147d50 0x7f62441de670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47356/0 (socket says 172.21.15.28:47356) 2024-02-21T23:06:29.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.498+0000 7f6243e00640 1 -- 172.21.15.28:0/1250858474 learned_addr learned my addr 172.21.15.28:0/1250858474 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:29.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.498+0000 7f6243400640 1 --2- 172.21.15.28:0/1250858474 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6228004d40 0x7f6244157bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:29.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.499+0000 7f623be00640 1 --2- 172.21.15.28:0/1250858474 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62441482a0 0x7f62440c0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:29.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.499+0000 7f6243e00640 1 -- 172.21.15.28:0/1250858474 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6228004d40 msgr2=0x7f6244157bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:29.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.499+0000 7f6243e00640 1 --2- 172.21.15.28:0/1250858474 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6228004d40 0x7f6244157bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:29.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.499+0000 7f6243e00640 1 -- 172.21.15.28:0/1250858474 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62441482a0 msgr2=0x7f62440c0c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:29.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.499+0000 7f6243e00640 1 --2- 172.21.15.28:0/1250858474 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62441482a0 0x7f62440c0c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:29.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.499+0000 7f6243e00640 1 -- 172.21.15.28:0/1250858474 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6244158810 con 0x7f6244147d50 2024-02-21T23:06:29.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.499+0000 7f623be00640 1 --2- 172.21.15.28:0/1250858474 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62441482a0 0x7f62440c0c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:06:29.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.499+0000 7f6243e00640 1 --2- 172.21.15.28:0/1250858474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6244147d50 0x7f62441de670 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f623c002230 tx=0x7f623c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:29.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.499+0000 7f623b400640 1 -- 172.21.15.28:0/1250858474 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f623c039030 con 0x7f6244147d50 2024-02-21T23:06:29.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.499+0000 7f623b400640 1 -- 172.21.15.28:0/1250858474 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f623c043070 con 0x7f6244147d50 2024-02-21T23:06:29.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.499+0000 7f623b400640 1 -- 172.21.15.28:0/1250858474 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f623c009c10 con 0x7f6244147d50 2024-02-21T23:06:29.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.499+0000 7f624a000640 1 -- 172.21.15.28:0/1250858474 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6244158af0 con 0x7f6244147d50 2024-02-21T23:06:29.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.500+0000 7f624a000640 1 -- 172.21.15.28:0/1250858474 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62441570f0 con 0x7f6244147d50 2024-02-21T23:06:29.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.501+0000 7f624a000640 1 -- 172.21.15.28:0/1250858474 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6244067f30 con 0x7f6244147d50 2024-02-21T23:06:29.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.503+0000 7f623b400640 1 -- 172.21.15.28:0/1250858474 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f623c005d10 con 0x7f6244147d50 2024-02-21T23:06:29.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.504+0000 7f623b400640 1 --2- 172.21.15.28:0/1250858474 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f622006b500 0x7f622006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:29.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.504+0000 7f623b400640 1 -- 172.21.15.28:0/1250858474 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7159+0+0 (secure 0 0 0) 0x7f623c0ac950 con 0x7f6244147d50 2024-02-21T23:06:29.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.505+0000 7f6243400640 1 --2- 172.21.15.28:0/1250858474 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f622006b500 0x7f622006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:29.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.506+0000 7f6243400640 1 --2- 172.21.15.28:0/1250858474 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f622006b500 0x7f622006d9c0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f6234005c70 tx=0x7f6234009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:29.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.509+0000 7f623b400640 1 -- 172.21.15.28:0/1250858474 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f623c07ab30 con 0x7f6244147d50 2024-02-21T23:06:29.694 INFO:tasks.ceph.mds.a.smithi028.stderr:2024-02-21T23:06:29.706+0000 7f87d1600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-21T23:06:29.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.804+0000 7f624a000640 1 -- 172.21.15.28:0/1250858474 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6244157950 con 0x7f6244147d50 2024-02-21T23:06:29.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.807+0000 7f623b400640 1 -- 172.21.15.28:0/1250858474 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+20929 (secure 0 0 0) 0x7f623c07ebd0 con 0x7f6244147d50 2024-02-21T23:06:29.796 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:06:29.796 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":83,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:06:26.056922+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:05:48.772562+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-02-21T23:05:49.852659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs2_metadata","create_time":"2024-02-21T23:06:08.240443+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs2_data","create_time":"2024-02-21T23:06:09.466016+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.2","mappings":[{"from":1,"to":7}]},{"pgid":"14.12","mappings":[{"from":1,"to":5}]},{"pgid":"14.1e","mappings":[{"from":1,"to":2}]},{"pgid":"14.24","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6835/595682298":"2024-02-22T23:06:23.690106+0000","172.21.15.28:6834/595682298":"2024-02-22T23:06:23.690106+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:06:29.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.809+0000 7f624a000640 1 -- 172.21.15.28:0/1250858474 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f622006b500 msgr2=0x7f622006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:29.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.809+0000 7f624a000640 1 --2- 172.21.15.28:0/1250858474 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f622006b500 0x7f622006d9c0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f6234005c70 tx=0x7f6234009040 comp rx=0 tx=0).stop 2024-02-21T23:06:29.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.810+0000 7f624a000640 1 -- 172.21.15.28:0/1250858474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6244147d50 msgr2=0x7f62441de670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:29.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.810+0000 7f624a000640 1 --2- 172.21.15.28:0/1250858474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6244147d50 0x7f62441de670 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f623c002230 tx=0x7f623c009750 comp rx=0 tx=0).stop 2024-02-21T23:06:29.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.810+0000 7f624a000640 1 -- 172.21.15.28:0/1250858474 shutdown_connections 2024-02-21T23:06:29.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.810+0000 7f624a000640 1 --2- 172.21.15.28:0/1250858474 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f622006b500 0x7f622006d9c0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:29.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.810+0000 7f624a000640 1 --2- 172.21.15.28:0/1250858474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6244147d50 0x7f62441de670 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:29.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.810+0000 7f624a000640 1 --2- 172.21.15.28:0/1250858474 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62441482a0 0x7f62440c0c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:29.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.810+0000 7f624a000640 1 --2- 172.21.15.28:0/1250858474 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6228004d40 0x7f6244157bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:29.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.810+0000 7f624a000640 1 -- 172.21.15.28:0/1250858474 >> 172.21.15.28:0/1250858474 conn(0x7f62440729c0 msgr2=0x7f6244154840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:29.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.810+0000 7f624a000640 1 -- 172.21.15.28:0/1250858474 shutdown_connections 2024-02-21T23:06:29.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:29.810+0000 7f624a000640 1 -- 172.21.15.28:0/1250858474 wait complete. 2024-02-21T23:06:29.817 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-02-21T23:06:29.817 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:06:30.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.034+0000 7f0a1a600640 1 Processor -- start 2024-02-21T23:06:30.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.034+0000 7f0a1a600640 1 -- start start 2024-02-21T23:06:30.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.035+0000 7f0a1a600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0a140aff90 0x7f0a140c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:30.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.035+0000 7f0a1a600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0a1406dbf0 0x7f0a14158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:30.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.035+0000 7f0a1a600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0a140c5720 0x7f0a14159000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:30.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.035+0000 7f0a1a600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a140c7270 con 0x7f0a1406dbf0 2024-02-21T23:06:30.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.036+0000 7f0a1a600640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a14158630 con 0x7f0a140c5720 2024-02-21T23:06:30.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.036+0000 7f0a1a600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a140c74c0 con 0x7f0a140aff90 2024-02-21T23:06:30.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.036+0000 7f0a13e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0a1406dbf0 0x7f0a14158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:30.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.036+0000 7f0a13e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0a1406dbf0 0x7f0a14158ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47366/0 (socket says 172.21.15.28:47366) 2024-02-21T23:06:30.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.036+0000 7f0a13e00640 1 -- 172.21.15.28:0/2802911450 learned_addr learned my addr 172.21.15.28:0/2802911450 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:30.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.036+0000 7f0a19c00640 1 --2- 172.21.15.28:0/2802911450 >> v2:172.21.15.81:3300/0 conn(0x7f0a140c5720 0x7f0a14159000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:30.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.036+0000 7f0a19200640 1 --2- 172.21.15.28:0/2802911450 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0a140aff90 0x7f0a140c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:30.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.037+0000 7f0a13e00640 1 -- 172.21.15.28:0/2802911450 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0a140aff90 msgr2=0x7f0a140c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:30.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.037+0000 7f0a13e00640 1 --2- 172.21.15.28:0/2802911450 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0a140aff90 0x7f0a140c51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:30.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.037+0000 7f0a13e00640 1 -- 172.21.15.28:0/2802911450 >> v2:172.21.15.81:3300/0 conn(0x7f0a140c5720 msgr2=0x7f0a14159000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:30.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.037+0000 7f0a13e00640 1 --2- 172.21.15.28:0/2802911450 >> v2:172.21.15.81:3300/0 conn(0x7f0a140c5720 0x7f0a14159000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:30.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.037+0000 7f0a13e00640 1 -- 172.21.15.28:0/2802911450 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a140c77a0 con 0x7f0a1406dbf0 2024-02-21T23:06:30.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.037+0000 7f0a19c00640 1 --2- 172.21.15.28:0/2802911450 >> v2:172.21.15.81:3300/0 conn(0x7f0a140c5720 0x7f0a14159000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:06:30.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.038+0000 7f0a13e00640 1 --2- 172.21.15.28:0/2802911450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a1406dbf0 0x7f0a14158ac0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f0a140ca6e0 tx=0x7f0a0002b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be51426c187d8946 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:30.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.038+0000 7f0a13400640 1 -- 172.21.15.28:0/2802911450 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a00035020 con 0x7f0a1406dbf0 2024-02-21T23:06:30.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.039+0000 7f0a13400640 1 -- 172.21.15.28:0/2802911450 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a0002d040 con 0x7f0a1406dbf0 2024-02-21T23:06:30.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.039+0000 7f0a13400640 1 -- 172.21.15.28:0/2802911450 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a00031a80 con 0x7f0a1406dbf0 2024-02-21T23:06:30.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.039+0000 7f0a1a600640 1 -- 172.21.15.28:0/2802911450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a1406dbf0 msgr2=0x7f0a14158ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:30.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.039+0000 7f0a1a600640 1 --2- 172.21.15.28:0/2802911450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a1406dbf0 0x7f0a14158ac0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f0a140ca6e0 tx=0x7f0a0002b900 comp rx=0 tx=0).stop 2024-02-21T23:06:30.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.039+0000 7f0a1a600640 1 -- 172.21.15.28:0/2802911450 shutdown_connections 2024-02-21T23:06:30.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.039+0000 7f0a1a600640 1 --2- 172.21.15.28:0/2802911450 >> v2:172.21.15.81:3300/0 conn(0x7f0a140c5720 0x7f0a14159000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:30.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.039+0000 7f0a1a600640 1 --2- 172.21.15.28:0/2802911450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a1406dbf0 0x7f0a14158ac0 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:30.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.039+0000 7f0a1a600640 1 --2- 172.21.15.28:0/2802911450 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0a140aff90 0x7f0a140c51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:30.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.039+0000 7f0a1a600640 1 -- 172.21.15.28:0/2802911450 >> 172.21.15.28:0/2802911450 conn(0x7f0a140ac370 msgr2=0x7f0a140ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:30.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.040+0000 7f0a1a600640 1 -- 172.21.15.28:0/2802911450 shutdown_connections 2024-02-21T23:06:30.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.040+0000 7f0a1a600640 1 -- 172.21.15.28:0/2802911450 wait complete. 2024-02-21T23:06:30.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.042+0000 7f0a1a600640 1 Processor -- start 2024-02-21T23:06:30.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.043+0000 7f0a1a600640 1 -- start start 2024-02-21T23:06:30.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.044+0000 7f0a1a600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0a1406dbf0 0x7f0a1414eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:30.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.044+0000 7f0a1a600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a140aff90 0x7f0a140c3860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:30.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.044+0000 7f0a1a600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a140c5720 0x7f0a141548c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:30.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.044+0000 7f0a1a600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a140c8760 con 0x7f0a140c5720 2024-02-21T23:06:30.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.044+0000 7f0a1a600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a140be0c0 con 0x7f0a140aff90 2024-02-21T23:06:30.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.044+0000 7f0a1a600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a140c88e0 con 0x7f0a1406dbf0 2024-02-21T23:06:30.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.045+0000 7f0a19c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a140c5720 0x7f0a141548c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:30.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.045+0000 7f0a19c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a140c5720 0x7f0a141548c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47382/0 (socket says 172.21.15.28:47382) 2024-02-21T23:06:30.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.045+0000 7f0a19c00640 1 -- 172.21.15.28:0/4283331812 learned_addr learned my addr 172.21.15.28:0/4283331812 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:30.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.045+0000 7f0a19200640 1 --2- 172.21.15.28:0/4283331812 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0a1406dbf0 0x7f0a1414eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:30.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.045+0000 7f0a13e00640 1 --2- 172.21.15.28:0/4283331812 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a140aff90 0x7f0a140c3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:30.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.045+0000 7f0a19c00640 1 -- 172.21.15.28:0/4283331812 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0a1406dbf0 msgr2=0x7f0a1414eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:30.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.045+0000 7f0a19c00640 1 --2- 172.21.15.28:0/4283331812 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0a1406dbf0 0x7f0a1414eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:30.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.045+0000 7f0a19c00640 1 -- 172.21.15.28:0/4283331812 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a140aff90 msgr2=0x7f0a140c3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:30.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.045+0000 7f0a19c00640 1 --2- 172.21.15.28:0/4283331812 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a140aff90 0x7f0a140c3860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:30.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.045+0000 7f0a19c00640 1 -- 172.21.15.28:0/4283331812 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a140c3da0 con 0x7f0a140c5720 2024-02-21T23:06:30.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.045+0000 7f0a19200640 1 --2- 172.21.15.28:0/4283331812 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0a1406dbf0 0x7f0a1414eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:06:30.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.046+0000 7f0a19c00640 1 --2- 172.21.15.28:0/4283331812 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a140c5720 0x7f0a141548c0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f0a0c007e20 tx=0x7f0a0c00d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:30.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.046+0000 7f0a10c00640 1 -- 172.21.15.28:0/4283331812 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a0c01a070 con 0x7f0a140c5720 2024-02-21T23:06:30.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.046+0000 7f0a10c00640 1 -- 172.21.15.28:0/4283331812 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a0c003b90 con 0x7f0a140c5720 2024-02-21T23:06:30.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.046+0000 7f0a10c00640 1 -- 172.21.15.28:0/4283331812 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a0c00de80 con 0x7f0a140c5720 2024-02-21T23:06:30.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.046+0000 7f0a1a600640 1 -- 172.21.15.28:0/4283331812 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a141eb790 con 0x7f0a140c5720 2024-02-21T23:06:30.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.046+0000 7f0a1a600640 1 -- 172.21.15.28:0/4283331812 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a14153190 con 0x7f0a140c5720 2024-02-21T23:06:30.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.048+0000 7f0a1a600640 1 -- 172.21.15.28:0/4283331812 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a1404f290 con 0x7f0a140c5720 2024-02-21T23:06:30.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.048+0000 7f0a10c00640 1 -- 172.21.15.28:0/4283331812 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0a0c01d070 con 0x7f0a140c5720 2024-02-21T23:06:30.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.050+0000 7f0a10c00640 1 --2- 172.21.15.28:0/4283331812 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f09f406b4d0 0x7f09f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:30.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.050+0000 7f0a10c00640 1 -- 172.21.15.28:0/4283331812 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7159+0+0 (secure 0 0 0) 0x7f0a0c08aa40 con 0x7f0a140c5720 2024-02-21T23:06:30.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.051+0000 7f0a19200640 1 --2- 172.21.15.28:0/4283331812 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f09f406b4d0 0x7f09f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:30.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.052+0000 7f0a19200640 1 --2- 172.21.15.28:0/4283331812 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f09f406b4d0 0x7f09f406d990 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f0a08006fd0 tx=0x7f0a08008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:30.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.053+0000 7f0a10c00640 1 -- 172.21.15.28:0/4283331812 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f0a0c010070 con 0x7f0a140c5720 2024-02-21T23:06:30.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.484+0000 7f0a1a600640 1 -- 172.21.15.28:0/4283331812 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0a1406d9d0 con 0x7f0a140c5720 2024-02-21T23:06:30.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.485+0000 7f0a10c00640 1 -- 172.21.15.28:0/4283331812 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+6179 (secure 0 0 0) 0x7f0a0c059300 con 0x7f0a140c5720 2024-02-21T23:06:30.474 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:06:30.474 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":74,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5288,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":8,"addr":"172.21.15.81:6835/2774951107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2774951107},{"type":"v1","addr":"172.21.15.81:6835","nonce":2774951107}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67},{"gid":5294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":8,"addr":"172.21.15.81:6837/4292940632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":4292940632},{"type":"v1","addr":"172.21.15.81:6837","nonce":4292940632}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":74,"flags":146,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":true,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:05:50.828988+0000","modified":"2024-02-21T23:06:26.718835+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6729},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6729":{"gid":6729,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6835/1066741568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":1066741568},{"type":"v1","addr":"172.21.15.28:6835","nonce":1066741568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5},{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:06:10.484421+0000","modified":"2024-02-21T23:06:13.492899+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5539},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5539":{"gid":5539,"name":"c","rank":0,"incarnation":63,"state":"up:active","state_seq":8,"addr":"172.21.15.28:6833/624384999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":624384999},{"type":"v1","addr":"172.21.15.28:6833","nonce":624384999}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-02-21T23:06:30.474 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 74 2024-02-21T23:06:30.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.487+0000 7f0a1a600640 1 -- 172.21.15.28:0/4283331812 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f09f406b4d0 msgr2=0x7f09f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:30.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.487+0000 7f0a1a600640 1 --2- 172.21.15.28:0/4283331812 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f09f406b4d0 0x7f09f406d990 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f0a08006fd0 tx=0x7f0a08008040 comp rx=0 tx=0).stop 2024-02-21T23:06:30.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.487+0000 7f0a1a600640 1 -- 172.21.15.28:0/4283331812 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a140c5720 msgr2=0x7f0a141548c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:30.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.487+0000 7f0a1a600640 1 --2- 172.21.15.28:0/4283331812 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a140c5720 0x7f0a141548c0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f0a0c007e20 tx=0x7f0a0c00d4a0 comp rx=0 tx=0).stop 2024-02-21T23:06:30.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.487+0000 7f0a1a600640 1 -- 172.21.15.28:0/4283331812 shutdown_connections 2024-02-21T23:06:30.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.487+0000 7f0a1a600640 1 --2- 172.21.15.28:0/4283331812 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f09f406b4d0 0x7f09f406d990 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:30.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.487+0000 7f0a1a600640 1 --2- 172.21.15.28:0/4283331812 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a140c5720 0x7f0a141548c0 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:30.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.487+0000 7f0a1a600640 1 --2- 172.21.15.28:0/4283331812 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a140aff90 0x7f0a140c3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:30.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.487+0000 7f0a1a600640 1 --2- 172.21.15.28:0/4283331812 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0a1406dbf0 0x7f0a1414eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:30.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.487+0000 7f0a1a600640 1 -- 172.21.15.28:0/4283331812 >> 172.21.15.28:0/4283331812 conn(0x7f0a140ac370 msgr2=0x7f0a140c8ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:30.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.488+0000 7f0a1a600640 1 -- 172.21.15.28:0/4283331812 shutdown_connections 2024-02-21T23:06:30.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.488+0000 7f0a1a600640 1 -- 172.21.15.28:0/4283331812 wait complete. 2024-02-21T23:06:30.496 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:06:30.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.693+0000 7f4a85800640 1 Processor -- start 2024-02-21T23:06:30.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.693+0000 7f4a85800640 1 -- start start 2024-02-21T23:06:30.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.695+0000 7f4a85800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4a80152500 0x7f4a801558c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:30.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.695+0000 7f4a85800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4a800ca050 0x7f4a800bf6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:30.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.695+0000 7f4a85800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f4a800bfc00 0x7f4a8006bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:30.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.695+0000 7f4a85800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a801528e0 con 0x7f4a800ca050 2024-02-21T23:06:30.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.695+0000 7f4a85800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a800c86e0 con 0x7f4a800bfc00 2024-02-21T23:06:30.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.695+0000 7f4a85800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a800c8980 con 0x7f4a80152500 2024-02-21T23:06:30.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.695+0000 7f4a7f400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4a800ca050 0x7f4a800bf6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:30.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.695+0000 7f4a7f400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4a800ca050 0x7f4a800bf6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47392/0 (socket says 172.21.15.28:47392) 2024-02-21T23:06:30.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.695+0000 7f4a7f400640 1 -- 172.21.15.28:0/202858394 learned_addr learned my addr 172.21.15.28:0/202858394 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:30.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.696+0000 7f4a7fe00640 1 --2- 172.21.15.28:0/202858394 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4a80152500 0x7f4a801558c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:30.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.696+0000 7f4a84e00640 1 --2- 172.21.15.28:0/202858394 >> v2:172.21.15.81:3300/0 conn(0x7f4a800bfc00 0x7f4a8006bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:30.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.696+0000 7f4a7f400640 1 -- 172.21.15.28:0/202858394 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4a80152500 msgr2=0x7f4a801558c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:30.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.696+0000 7f4a7f400640 1 --2- 172.21.15.28:0/202858394 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4a80152500 0x7f4a801558c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:30.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.696+0000 7f4a7f400640 1 -- 172.21.15.28:0/202858394 >> v2:172.21.15.81:3300/0 conn(0x7f4a800bfc00 msgr2=0x7f4a8006bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:30.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.696+0000 7f4a7f400640 1 --2- 172.21.15.28:0/202858394 >> v2:172.21.15.81:3300/0 conn(0x7f4a800bfc00 0x7f4a8006bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:30.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.696+0000 7f4a7f400640 1 -- 172.21.15.28:0/202858394 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a800c8c60 con 0x7f4a800ca050 2024-02-21T23:06:30.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.697+0000 7f4a7f400640 1 --2- 172.21.15.28:0/202858394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a800ca050 0x7f4a800bf6c0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f4a800c99a0 tx=0x7f4a6802b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69396e8b4e2cc95b server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:30.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.697+0000 7f4a7ea00640 1 -- 172.21.15.28:0/202858394 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a680317d0 con 0x7f4a800ca050 2024-02-21T23:06:30.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.697+0000 7f4a7ea00640 1 -- 172.21.15.28:0/202858394 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a68037070 con 0x7f4a800ca050 2024-02-21T23:06:30.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.697+0000 7f4a7ea00640 1 -- 172.21.15.28:0/202858394 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a68031b90 con 0x7f4a800ca050 2024-02-21T23:06:30.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.698+0000 7f4a85800640 1 -- 172.21.15.28:0/202858394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a800ca050 msgr2=0x7f4a800bf6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:30.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.698+0000 7f4a85800640 1 --2- 172.21.15.28:0/202858394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a800ca050 0x7f4a800bf6c0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f4a800c99a0 tx=0x7f4a6802b870 comp rx=0 tx=0).stop 2024-02-21T23:06:30.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.698+0000 7f4a85800640 1 -- 172.21.15.28:0/202858394 shutdown_connections 2024-02-21T23:06:30.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.698+0000 7f4a85800640 1 --2- 172.21.15.28:0/202858394 >> v2:172.21.15.81:3300/0 conn(0x7f4a800bfc00 0x7f4a8006bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:30.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.698+0000 7f4a85800640 1 --2- 172.21.15.28:0/202858394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a800ca050 0x7f4a800bf6c0 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:30.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.698+0000 7f4a85800640 1 --2- 172.21.15.28:0/202858394 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4a80152500 0x7f4a801558c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:30.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.698+0000 7f4a85800640 1 -- 172.21.15.28:0/202858394 >> 172.21.15.28:0/202858394 conn(0x7f4a800ac450 msgr2=0x7f4a800ac860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:30.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.698+0000 7f4a85800640 1 -- 172.21.15.28:0/202858394 shutdown_connections 2024-02-21T23:06:30.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.698+0000 7f4a85800640 1 -- 172.21.15.28:0/202858394 wait complete. 2024-02-21T23:06:30.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.701+0000 7f4a85800640 1 Processor -- start 2024-02-21T23:06:30.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.701+0000 7f4a85800640 1 -- start start 2024-02-21T23:06:30.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.702+0000 7f4a85800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a800bfc00 0x7f4a80151030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:30.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.702+0000 7f4a85800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4a800ca050 0x7f4a80147c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:30.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.702+0000 7f4a85800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a80152500 0x7f4a80148140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:30.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.702+0000 7f4a85800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a8006c130 con 0x7f4a80152500 2024-02-21T23:06:30.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.702+0000 7f4a85800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a800c9220 con 0x7f4a800bfc00 2024-02-21T23:06:30.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.702+0000 7f4a85800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a8006c2b0 con 0x7f4a800ca050 2024-02-21T23:06:30.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.703+0000 7f4a84e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a80152500 0x7f4a80148140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:30.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.703+0000 7f4a84e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a80152500 0x7f4a80148140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47402/0 (socket says 172.21.15.28:47402) 2024-02-21T23:06:30.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.703+0000 7f4a84e00640 1 -- 172.21.15.28:0/1866377494 learned_addr learned my addr 172.21.15.28:0/1866377494 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:30.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.703+0000 7f4a7fe00640 1 --2- 172.21.15.28:0/1866377494 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a800bfc00 0x7f4a80151030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:30.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.703+0000 7f4a7f400640 1 --2- 172.21.15.28:0/1866377494 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4a800ca050 0x7f4a80147c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:30.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.703+0000 7f4a84e00640 1 -- 172.21.15.28:0/1866377494 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4a800ca050 msgr2=0x7f4a80147c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:30.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.703+0000 7f4a84e00640 1 --2- 172.21.15.28:0/1866377494 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4a800ca050 0x7f4a80147c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:30.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.703+0000 7f4a84e00640 1 -- 172.21.15.28:0/1866377494 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a800bfc00 msgr2=0x7f4a80151030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:30.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.703+0000 7f4a84e00640 1 --2- 172.21.15.28:0/1866377494 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a800bfc00 0x7f4a80151030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:30.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.703+0000 7f4a84e00640 1 -- 172.21.15.28:0/1866377494 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a80148680 con 0x7f4a80152500 2024-02-21T23:06:30.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.703+0000 7f4a7f400640 1 --2- 172.21.15.28:0/1866377494 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4a800ca050 0x7f4a80147c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:06:30.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.704+0000 7f4a84e00640 1 --2- 172.21.15.28:0/1866377494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a80152500 0x7f4a80148140 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f4a7800d800 tx=0x7f4a7800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:30.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.704+0000 7f4a77e00640 1 -- 172.21.15.28:0/1866377494 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a7800b6c0 con 0x7f4a80152500 2024-02-21T23:06:30.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.704+0000 7f4a77e00640 1 -- 172.21.15.28:0/1866377494 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a78019070 con 0x7f4a80152500 2024-02-21T23:06:30.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.704+0000 7f4a77e00640 1 -- 172.21.15.28:0/1866377494 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a7800bac0 con 0x7f4a80152500 2024-02-21T23:06:30.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.704+0000 7f4a85800640 1 -- 172.21.15.28:0/1866377494 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a801ee890 con 0x7f4a80152500 2024-02-21T23:06:30.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.704+0000 7f4a85800640 1 -- 172.21.15.28:0/1866377494 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a800c6630 con 0x7f4a80152500 2024-02-21T23:06:30.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.706+0000 7f4a85800640 1 -- 172.21.15.28:0/1866377494 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a8004f290 con 0x7f4a80152500 2024-02-21T23:06:30.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.707+0000 7f4a77e00640 1 -- 172.21.15.28:0/1866377494 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4a7800bc60 con 0x7f4a80152500 2024-02-21T23:06:30.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.708+0000 7f4a77e00640 1 --2- 172.21.15.28:0/1866377494 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4a5c06b4d0 0x7f4a5c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:30.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.708+0000 7f4a77e00640 1 -- 172.21.15.28:0/1866377494 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7159+0+0 (secure 0 0 0) 0x7f4a7808ac30 con 0x7f4a80152500 2024-02-21T23:06:30.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.709+0000 7f4a7fe00640 1 --2- 172.21.15.28:0/1866377494 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4a5c06b4d0 0x7f4a5c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:30.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.710+0000 7f4a7fe00640 1 --2- 172.21.15.28:0/1866377494 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4a5c06b4d0 0x7f4a5c06d990 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f4a70009940 tx=0x7f4a70003cd0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:30.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.713+0000 7f4a77e00640 1 -- 172.21.15.28:0/1866377494 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4a780594f0 con 0x7f4a80152500 2024-02-21T23:06:30.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.945+0000 7f4a85800640 1 -- 172.21.15.28:0/1866377494 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4a80151b20 con 0x7f4a80152500 2024-02-21T23:06:30.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.946+0000 7f4a77e00640 1 -- 172.21.15.28:0/1866377494 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+6179 (secure 0 0 0) 0x7f4a78058df0 con 0x7f4a80152500 2024-02-21T23:06:30.935 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:06:30.935 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":74,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5288,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":8,"addr":"172.21.15.81:6835/2774951107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2774951107},{"type":"v1","addr":"172.21.15.81:6835","nonce":2774951107}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67},{"gid":5294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":8,"addr":"172.21.15.81:6837/4292940632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":4292940632},{"type":"v1","addr":"172.21.15.81:6837","nonce":4292940632}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":74,"flags":146,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":true,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:05:50.828988+0000","modified":"2024-02-21T23:06:26.718835+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6729},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6729":{"gid":6729,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6835/1066741568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":1066741568},{"type":"v1","addr":"172.21.15.28:6835","nonce":1066741568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5},{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:06:10.484421+0000","modified":"2024-02-21T23:06:13.492899+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5539},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5539":{"gid":5539,"name":"c","rank":0,"incarnation":63,"state":"up:active","state_seq":8,"addr":"172.21.15.28:6833/624384999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":624384999},{"type":"v1","addr":"172.21.15.28:6833","nonce":624384999}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-02-21T23:06:30.935 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 74 2024-02-21T23:06:30.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.948+0000 7f4a85800640 1 -- 172.21.15.28:0/1866377494 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4a5c06b4d0 msgr2=0x7f4a5c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:30.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.948+0000 7f4a85800640 1 --2- 172.21.15.28:0/1866377494 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4a5c06b4d0 0x7f4a5c06d990 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f4a70009940 tx=0x7f4a70003cd0 comp rx=0 tx=0).stop 2024-02-21T23:06:30.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.949+0000 7f4a85800640 1 -- 172.21.15.28:0/1866377494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a80152500 msgr2=0x7f4a80148140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:30.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.949+0000 7f4a85800640 1 --2- 172.21.15.28:0/1866377494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a80152500 0x7f4a80148140 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f4a7800d800 tx=0x7f4a7800dcc0 comp rx=0 tx=0).stop 2024-02-21T23:06:30.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.949+0000 7f4a85800640 1 -- 172.21.15.28:0/1866377494 shutdown_connections 2024-02-21T23:06:30.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.949+0000 7f4a85800640 1 --2- 172.21.15.28:0/1866377494 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4a5c06b4d0 0x7f4a5c06d990 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:30.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.949+0000 7f4a85800640 1 --2- 172.21.15.28:0/1866377494 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4a800ca050 0x7f4a80147c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:30.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.949+0000 7f4a85800640 1 --2- 172.21.15.28:0/1866377494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a80152500 0x7f4a80148140 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:30.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.949+0000 7f4a85800640 1 --2- 172.21.15.28:0/1866377494 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a800bfc00 0x7f4a80151030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:30.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.949+0000 7f4a85800640 1 -- 172.21.15.28:0/1866377494 >> 172.21.15.28:0/1866377494 conn(0x7f4a800ac450 msgr2=0x7f4a8006c5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:30.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.949+0000 7f4a85800640 1 -- 172.21.15.28:0/1866377494 shutdown_connections 2024-02-21T23:06:30.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:30.949+0000 7f4a85800640 1 -- 172.21.15.28:0/1866377494 wait complete. 2024-02-21T23:06:30.957 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:06:31.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.210+0000 7f155ce00640 1 Processor -- start 2024-02-21T23:06:31.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.210+0000 7f155ce00640 1 -- start start 2024-02-21T23:06:31.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.211+0000 7f155ce00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f15580aff90 0x7f15580c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:31.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.211+0000 7f155ce00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15580c74c0 con 0x7f155806dbf0 2024-02-21T23:06:31.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.212+0000 7f155ce00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15580c77a0 con 0x7f15580c5720 2024-02-21T23:06:31.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.212+0000 7f155ce00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15580c7270 con 0x7f15580aff90 2024-02-21T23:06:31.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.212+0000 7f1557400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f15580aff90 0x7f15580c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:31.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.212+0000 7f1556a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f155806dbf0 0x7f1558158ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37478/0 (socket says 172.21.15.28:37478) 2024-02-21T23:06:31.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.212+0000 7f1556a00640 1 -- 172.21.15.28:0/938328012 learned_addr learned my addr 172.21.15.28:0/938328012 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:06:31.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.212+0000 7f1557400640 1 -- 172.21.15.28:0/938328012 >> v1:172.21.15.81:6789/0 conn(0x7f15580c5720 legacy=0x7f155815f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:31.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.212+0000 7f1557400640 1 -- 172.21.15.28:0/938328012 >> v1:172.21.15.28:6789/0 conn(0x7f155806dbf0 legacy=0x7f1558158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:31.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.212+0000 7f1556000640 1 -- 172.21.15.28:0/938328012 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2430460228 0 0) 0x7f15580c74c0 con 0x7f155806dbf0 2024-02-21T23:06:31.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.213+0000 7f1557400640 1 -- 172.21.15.28:0/938328012 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15580c7a80 con 0x7f15580aff90 2024-02-21T23:06:31.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.214+0000 7f1557400640 1 --2- 172.21.15.28:0/938328012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15580aff90 0x7f15580c51e0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f15480097a0 tx=0x7f154802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25d5291283a300f4 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:31.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.214+0000 7f1556000640 1 -- 172.21.15.28:0/938328012 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1548035020 con 0x7f15580aff90 2024-02-21T23:06:31.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.214+0000 7f1556000640 1 -- 172.21.15.28:0/938328012 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1548031830 con 0x7f15580aff90 2024-02-21T23:06:31.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.214+0000 7f1556000640 1 -- 172.21.15.28:0/938328012 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1548031c00 con 0x7f15580aff90 2024-02-21T23:06:31.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.214+0000 7f155ce00640 1 -- 172.21.15.28:0/938328012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15580aff90 msgr2=0x7f15580c51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:31.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.214+0000 7f155ce00640 1 --2- 172.21.15.28:0/938328012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15580aff90 0x7f15580c51e0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f15480097a0 tx=0x7f154802b8d0 comp rx=0 tx=0).stop 2024-02-21T23:06:31.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.215+0000 7f155ce00640 1 -- 172.21.15.28:0/938328012 shutdown_connections 2024-02-21T23:06:31.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.215+0000 7f155ce00640 1 --2- 172.21.15.28:0/938328012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15580aff90 0x7f15580c51e0 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:31.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.215+0000 7f155ce00640 1 -- 172.21.15.28:0/938328012 >> 172.21.15.28:0/938328012 conn(0x7f15580ac370 msgr2=0x7f15580ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:31.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.215+0000 7f155ce00640 1 -- 172.21.15.28:0/938328012 shutdown_connections 2024-02-21T23:06:31.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.215+0000 7f155ce00640 1 -- 172.21.15.28:0/938328012 wait complete. 2024-02-21T23:06:31.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.216+0000 7f155ce00640 1 Processor -- start 2024-02-21T23:06:31.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.217+0000 7f155ce00640 1 -- start start 2024-02-21T23:06:31.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.217+0000 7f155ce00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f155806dbf0 0x7f155814f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:31.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.217+0000 7f155ce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15580aff90 0x7f15580ab430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:31.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.217+0000 7f155ce00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f15580c5720 0x7f15580c2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:31.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.217+0000 7f155ce00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15580c8760 con 0x7f15580aff90 2024-02-21T23:06:31.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.217+0000 7f155ce00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1558158630 con 0x7f15580c5720 2024-02-21T23:06:31.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.217+0000 7f155ce00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15580c88e0 con 0x7f155806dbf0 2024-02-21T23:06:31.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.218+0000 7f1556a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15580aff90 0x7f15580ab430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:31.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.218+0000 7f1556a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15580aff90 0x7f15580ab430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47434/0 (socket says 172.21.15.28:47434) 2024-02-21T23:06:31.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.218+0000 7f1556a00640 1 -- 172.21.15.28:0/3312810817 learned_addr learned my addr 172.21.15.28:0/3312810817 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:31.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.218+0000 7f1557400640 1 --2- 172.21.15.28:0/3312810817 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f155806dbf0 0x7f155814f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:31.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.218+0000 7f1557e00640 1 --2- 172.21.15.28:0/3312810817 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f15580c5720 0x7f15580c2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:31.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.219+0000 7f1556a00640 1 -- 172.21.15.28:0/3312810817 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f155806dbf0 msgr2=0x7f155814f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:31.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.219+0000 7f1556a00640 1 --2- 172.21.15.28:0/3312810817 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f155806dbf0 0x7f155814f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:31.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.219+0000 7f1556a00640 1 -- 172.21.15.28:0/3312810817 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f15580c5720 msgr2=0x7f15580c2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:31.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.219+0000 7f1556a00640 1 --2- 172.21.15.28:0/3312810817 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f15580c5720 0x7f15580c2160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:31.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.219+0000 7f1556a00640 1 -- 172.21.15.28:0/3312810817 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15580be840 con 0x7f15580aff90 2024-02-21T23:06:31.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.219+0000 7f1556a00640 1 --2- 172.21.15.28:0/3312810817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15580aff90 0x7f15580ab430 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f154400c4b0 tx=0x7f154400c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:31.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.219+0000 7f154f400640 1 -- 172.21.15.28:0/3312810817 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1544018070 con 0x7f15580aff90 2024-02-21T23:06:31.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.219+0000 7f154f400640 1 -- 172.21.15.28:0/3312810817 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1544007500 con 0x7f15580aff90 2024-02-21T23:06:31.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.219+0000 7f155ce00640 1 -- 172.21.15.28:0/3312810817 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f155814ef90 con 0x7f15580aff90 2024-02-21T23:06:31.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.219+0000 7f154f400640 1 -- 172.21.15.28:0/3312810817 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1544004a20 con 0x7f15580aff90 2024-02-21T23:06:31.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.219+0000 7f155ce00640 1 -- 172.21.15.28:0/3312810817 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15581ee1a0 con 0x7f15580aff90 2024-02-21T23:06:31.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.221+0000 7f155ce00640 1 -- 172.21.15.28:0/3312810817 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15581ee4b0 con 0x7f15580aff90 2024-02-21T23:06:31.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.222+0000 7f154f400640 1 -- 172.21.15.28:0/3312810817 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f154401b070 con 0x7f15580aff90 2024-02-21T23:06:31.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.222+0000 7f154f400640 1 --2- 172.21.15.28:0/3312810817 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f153806b4d0 0x7f153806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:31.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.223+0000 7f154f400640 1 -- 172.21.15.28:0/3312810817 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7159+0+0 (secure 0 0 0) 0x7f1544089af0 con 0x7f15580aff90 2024-02-21T23:06:31.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.223+0000 7f1557400640 1 --2- 172.21.15.28:0/3312810817 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f153806b4d0 0x7f153806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:31.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.224+0000 7f1557400640 1 --2- 172.21.15.28:0/3312810817 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f153806b4d0 0x7f153806d990 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f1548036000 tx=0x7f154803c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:31.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.232+0000 7f154f400640 1 -- 172.21.15.28:0/3312810817 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1544058470 con 0x7f15580aff90 2024-02-21T23:06:31.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.494+0000 7f155ce00640 1 -- 172.21.15.28:0/3312810817 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f155804f290 con 0x7f15580aff90 2024-02-21T23:06:31.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.495+0000 7f154f400640 1 -- 172.21.15.28:0/3312810817 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+20929 (secure 0 0 0) 0x7f1544057d70 con 0x7f15580aff90 2024-02-21T23:06:31.484 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:06:31.484 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":83,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:06:26.056922+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:05:48.772562+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-02-21T23:05:49.852659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs2_metadata","create_time":"2024-02-21T23:06:08.240443+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs2_data","create_time":"2024-02-21T23:06:09.466016+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.2","mappings":[{"from":1,"to":7}]},{"pgid":"14.12","mappings":[{"from":1,"to":5}]},{"pgid":"14.1e","mappings":[{"from":1,"to":2}]},{"pgid":"14.24","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6835/595682298":"2024-02-22T23:06:23.690106+0000","172.21.15.28:6834/595682298":"2024-02-22T23:06:23.690106+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:06:31.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.499+0000 7f155ce00640 1 -- 172.21.15.28:0/3312810817 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f153806b4d0 msgr2=0x7f153806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:31.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.499+0000 7f155ce00640 1 --2- 172.21.15.28:0/3312810817 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f153806b4d0 0x7f153806d990 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f1548036000 tx=0x7f154803c040 comp rx=0 tx=0).stop 2024-02-21T23:06:31.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.499+0000 7f155ce00640 1 -- 172.21.15.28:0/3312810817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15580aff90 msgr2=0x7f15580ab430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:31.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.499+0000 7f155ce00640 1 --2- 172.21.15.28:0/3312810817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15580aff90 0x7f15580ab430 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f154400c4b0 tx=0x7f154400c970 comp rx=0 tx=0).stop 2024-02-21T23:06:31.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.499+0000 7f155ce00640 1 -- 172.21.15.28:0/3312810817 shutdown_connections 2024-02-21T23:06:31.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.499+0000 7f155ce00640 1 --2- 172.21.15.28:0/3312810817 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f153806b4d0 0x7f153806d990 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:31.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.499+0000 7f155ce00640 1 --2- 172.21.15.28:0/3312810817 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f15580c5720 0x7f15580c2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:31.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.499+0000 7f155ce00640 1 --2- 172.21.15.28:0/3312810817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15580aff90 0x7f15580ab430 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:31.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.499+0000 7f155ce00640 1 --2- 172.21.15.28:0/3312810817 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f155806dbf0 0x7f155814f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:31.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.499+0000 7f155ce00640 1 -- 172.21.15.28:0/3312810817 >> 172.21.15.28:0/3312810817 conn(0x7f15580ac370 msgr2=0x7f1558157da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:31.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.499+0000 7f155ce00640 1 -- 172.21.15.28:0/3312810817 shutdown_connections 2024-02-21T23:06:31.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.499+0000 7f155ce00640 1 -- 172.21.15.28:0/3312810817 wait complete. 2024-02-21T23:06:31.512 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-02-21T23:06:31.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.725+0000 7f64b0a00640 1 Processor -- start 2024-02-21T23:06:31.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.725+0000 7f64b0a00640 1 -- start start 2024-02-21T23:06:31.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.727+0000 7f64b0a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f64ac0c5040 0x7f64ac155a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:31.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.727+0000 7f64b0a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64ac06df50 con 0x7f64ac0c85c0 2024-02-21T23:06:31.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.727+0000 7f64b0a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64ac0c5520 con 0x7f64ac1524c0 2024-02-21T23:06:31.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.727+0000 7f64b0a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64ac0c5800 con 0x7f64ac0c5040 2024-02-21T23:06:31.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.727+0000 7f64abe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f64ac0c85c0 0x7f64ac0c70d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37486/0 (socket says 172.21.15.28:37486) 2024-02-21T23:06:31.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.727+0000 7f64abe00640 1 -- 172.21.15.28:0/1784332011 learned_addr learned my addr 172.21.15.28:0/1784332011 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:06:31.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.728+0000 7f64a3e00640 1 --2- 172.21.15.28:0/1784332011 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f64ac0c5040 0x7f64ac155a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:31.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.728+0000 7f64aaa00640 1 -- 172.21.15.28:0/1784332011 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1114120336 0 0) 0x7f64ac06df50 con 0x7f64ac0c85c0 2024-02-21T23:06:31.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.728+0000 7f64aaa00640 1 -- 172.21.15.28:0/1784332011 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6490003520 con 0x7f64ac0c85c0 2024-02-21T23:06:31.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.728+0000 7f64aaa00640 1 -- 172.21.15.28:0/1784332011 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3782379772 0 0) 0x7f6490003520 con 0x7f64ac0c85c0 2024-02-21T23:06:31.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.728+0000 7f64aaa00640 1 -- 172.21.15.28:0/1784332011 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f64ac0c5040 msgr2=0x7f64ac155a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:31.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.728+0000 7f64aaa00640 1 --2- 172.21.15.28:0/1784332011 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f64ac0c5040 0x7f64ac155a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:31.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.728+0000 7f64aaa00640 1 -- 172.21.15.28:0/1784332011 >> v1:172.21.15.81:6789/0 conn(0x7f64ac1524c0 legacy=0x7f64ac06ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:31.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.728+0000 7f64aaa00640 1 -- 172.21.15.28:0/1784332011 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64ac06c670 con 0x7f64ac0c85c0 2024-02-21T23:06:31.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.728+0000 7f64aaa00640 1 -- 172.21.15.28:0/1784332011 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f64a40019f0 con 0x7f64ac0c85c0 2024-02-21T23:06:31.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.728+0000 7f64aaa00640 1 -- 172.21.15.28:0/1784332011 >> v1:172.21.15.28:6789/0 conn(0x7f64ac0c85c0 legacy=0x7f64ac0c70d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:31.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.728+0000 7f64aaa00640 1 --2- 172.21.15.28:0/1784332011 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64900039e0 0x7f6490003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:31.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.728+0000 7f64aaa00640 1 --2- 172.21.15.28:0/1784332011 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f64ac0c5040 0x7f6490004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:31.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.728+0000 7f64aaa00640 1 --2- 172.21.15.28:0/1784332011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6490004d40 0x7f64900051a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:31.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.728+0000 7f64aaa00640 1 -- 172.21.15.28:0/1784332011 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64ac0c5800 con 0x7f6490004d40 2024-02-21T23:06:31.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.729+0000 7f64aaa00640 1 -- 172.21.15.28:0/1784332011 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f649000bbd0 con 0x7f64900039e0 2024-02-21T23:06:31.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.729+0000 7f64aaa00640 1 -- 172.21.15.28:0/1784332011 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f649000bdd0 con 0x7f64ac0c5040 2024-02-21T23:06:31.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.729+0000 7f64abe00640 1 --2- 172.21.15.28:0/1784332011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6490004d40 0x7f64900051a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:31.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.729+0000 7f64ab400640 1 --2- 172.21.15.28:0/1784332011 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64900039e0 0x7f6490003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:31.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.729+0000 7f64a3e00640 1 --2- 172.21.15.28:0/1784332011 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f64ac0c5040 0x7f6490004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:31.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.729+0000 7f64abe00640 1 -- 172.21.15.28:0/1784332011 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f64ac0c5040 msgr2=0x7f6490004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:31.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.729+0000 7f64abe00640 1 --2- 172.21.15.28:0/1784332011 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f64ac0c5040 0x7f6490004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:31.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.729+0000 7f64abe00640 1 -- 172.21.15.28:0/1784332011 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64900039e0 msgr2=0x7f6490003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:31.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.729+0000 7f64abe00640 1 --2- 172.21.15.28:0/1784332011 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64900039e0 0x7f6490003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:31.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.729+0000 7f64abe00640 1 -- 172.21.15.28:0/1784332011 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6490003520 con 0x7f6490004d40 2024-02-21T23:06:31.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.730+0000 7f64abe00640 1 --2- 172.21.15.28:0/1784332011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6490004d40 0x7f64900051a0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f64a400a770 tx=0x7f6490005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a667fd07509f6af server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:31.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.731+0000 7f64aaa00640 1 -- 172.21.15.28:0/1784332011 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a400a390 con 0x7f6490004d40 2024-02-21T23:06:31.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.731+0000 7f64aaa00640 1 -- 172.21.15.28:0/1784332011 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64a4034650 con 0x7f6490004d40 2024-02-21T23:06:31.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.731+0000 7f64aaa00640 1 -- 172.21.15.28:0/1784332011 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a4034a40 con 0x7f6490004d40 2024-02-21T23:06:31.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.731+0000 7f64b0a00640 1 -- 172.21.15.28:0/1784332011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6490004d40 msgr2=0x7f64900051a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:31.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.731+0000 7f64b0a00640 1 --2- 172.21.15.28:0/1784332011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6490004d40 0x7f64900051a0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f64a400a770 tx=0x7f6490005830 comp rx=0 tx=0).stop 2024-02-21T23:06:31.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.731+0000 7f64abe00640 1 -- 172.21.15.28:0/1784332011 reap_dead start 2024-02-21T23:06:31.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.731+0000 7f64b0a00640 1 -- 172.21.15.28:0/1784332011 shutdown_connections 2024-02-21T23:06:31.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.731+0000 7f64b0a00640 1 -- 172.21.15.28:0/1784332011 >> 172.21.15.28:0/1784332011 conn(0x7f64ac0ac2b0 msgr2=0x7f64ac0ac6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:31.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.731+0000 7f64b0a00640 1 -- 172.21.15.28:0/1784332011 shutdown_connections 2024-02-21T23:06:31.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.731+0000 7f64b0a00640 1 -- 172.21.15.28:0/1784332011 wait complete. 2024-02-21T23:06:31.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.733+0000 7f64b0a00640 1 Processor -- start 2024-02-21T23:06:31.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.733+0000 7f64b0a00640 1 -- start start 2024-02-21T23:06:31.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.734+0000 7f64b0a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6490004d40 0x7f64ac0add80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:31.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.734+0000 7f64b0a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f64ac0c21a0 0x7f64ac0ae2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:31.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.734+0000 7f64b0a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64ac0ae800 0x7f64ac0c38c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:31.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.734+0000 7f64b0a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64ac0c8a70 con 0x7f64ac0ae800 2024-02-21T23:06:31.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.734+0000 7f64b0a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64ac1528a0 con 0x7f6490004d40 2024-02-21T23:06:31.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.734+0000 7f64b0a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64ac0c77e0 con 0x7f64ac0c21a0 2024-02-21T23:06:31.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.734+0000 7f64abe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64ac0ae800 0x7f64ac0c38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:31.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.735+0000 7f64abe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64ac0ae800 0x7f64ac0c38c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47456/0 (socket says 172.21.15.28:47456) 2024-02-21T23:06:31.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.735+0000 7f64abe00640 1 -- 172.21.15.28:0/2736149028 learned_addr learned my addr 172.21.15.28:0/2736149028 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:31.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.735+0000 7f64ab400640 1 --2- 172.21.15.28:0/2736149028 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6490004d40 0x7f64ac0add80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:31.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.735+0000 7f64a3e00640 1 --2- 172.21.15.28:0/2736149028 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f64ac0c21a0 0x7f64ac0ae2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:31.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.735+0000 7f64abe00640 1 -- 172.21.15.28:0/2736149028 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f64ac0c21a0 msgr2=0x7f64ac0ae2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:31.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.735+0000 7f64abe00640 1 --2- 172.21.15.28:0/2736149028 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f64ac0c21a0 0x7f64ac0ae2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:31.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.735+0000 7f64abe00640 1 -- 172.21.15.28:0/2736149028 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6490004d40 msgr2=0x7f64ac0add80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:31.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.735+0000 7f64abe00640 1 --2- 172.21.15.28:0/2736149028 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6490004d40 0x7f64ac0add80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:31.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.735+0000 7f64abe00640 1 -- 172.21.15.28:0/2736149028 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64ac0c2630 con 0x7f64ac0ae800 2024-02-21T23:06:31.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.735+0000 7f64abe00640 1 --2- 172.21.15.28:0/2736149028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64ac0ae800 0x7f64ac0c38c0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f64a400a770 tx=0x7f64ac0c6f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:31.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.736+0000 7f64a3400640 1 -- 172.21.15.28:0/2736149028 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a4001ee0 con 0x7f64ac0ae800 2024-02-21T23:06:31.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.736+0000 7f64a3400640 1 -- 172.21.15.28:0/2736149028 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64a4044070 con 0x7f64ac0ae800 2024-02-21T23:06:31.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.736+0000 7f64a3400640 1 -- 172.21.15.28:0/2736149028 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a4034800 con 0x7f64ac0ae800 2024-02-21T23:06:31.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.736+0000 7f64b0a00640 1 -- 172.21.15.28:0/2736149028 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64ac0c2910 con 0x7f64ac0ae800 2024-02-21T23:06:31.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.736+0000 7f64b0a00640 1 -- 172.21.15.28:0/2736149028 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64ac0c2ed0 con 0x7f64ac0ae800 2024-02-21T23:06:31.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.737+0000 7f64b0a00640 1 -- 172.21.15.28:0/2736149028 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64ac0b0380 con 0x7f64ac0ae800 2024-02-21T23:06:31.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.739+0000 7f64a3400640 1 -- 172.21.15.28:0/2736149028 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f64a40349a0 con 0x7f64ac0ae800 2024-02-21T23:06:31.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.740+0000 7f64a3400640 1 --2- 172.21.15.28:0/2736149028 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f648806b4d0 0x7f648806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:31.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.740+0000 7f64a3400640 1 -- 172.21.15.28:0/2736149028 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7159+0+0 (secure 0 0 0) 0x7f64a40acac0 con 0x7f64ac0ae800 2024-02-21T23:06:31.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.741+0000 7f64ab400640 1 --2- 172.21.15.28:0/2736149028 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f648806b4d0 0x7f648806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:31.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.742+0000 7f64ab400640 1 --2- 172.21.15.28:0/2736149028 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f648806b4d0 0x7f648806d990 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f649c007fd0 tx=0x7f649c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:31.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.745+0000 7f64a3400640 1 -- 172.21.15.28:0/2736149028 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f64a407b4a0 con 0x7f64ac0ae800 2024-02-21T23:06:31.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.974+0000 7f64b0a00640 1 -- 172.21.15.28:0/2736149028 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f64ac0bdcc0 con 0x7f64ac0ae800 2024-02-21T23:06:31.964 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-02-21T23:06:31.975+0000 7f5ddc400640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-21T23:06:31.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.978+0000 7f64a3400640 1 -- 172.21.15.28:0/2736149028 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v75) v1 ==== 161+0+0 (secure 0 0 0) 0x7f64a403ee10 con 0x7f64ac0ae800 2024-02-21T23:06:31.966 INFO:teuthology.orchestra.run.smithi028.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-02-21T23:06:31.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.982+0000 7f64b0a00640 1 -- 172.21.15.28:0/2736149028 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f648806b4d0 msgr2=0x7f648806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:31.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.982+0000 7f64b0a00640 1 --2- 172.21.15.28:0/2736149028 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f648806b4d0 0x7f648806d990 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f649c007fd0 tx=0x7f649c009040 comp rx=0 tx=0).stop 2024-02-21T23:06:31.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.982+0000 7f64b0a00640 1 -- 172.21.15.28:0/2736149028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64ac0ae800 msgr2=0x7f64ac0c38c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:31.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.982+0000 7f64b0a00640 1 --2- 172.21.15.28:0/2736149028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64ac0ae800 0x7f64ac0c38c0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f64a400a770 tx=0x7f64ac0c6f20 comp rx=0 tx=0).stop 2024-02-21T23:06:31.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.982+0000 7f64b0a00640 1 -- 172.21.15.28:0/2736149028 shutdown_connections 2024-02-21T23:06:31.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.982+0000 7f64b0a00640 1 --2- 172.21.15.28:0/2736149028 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f648806b4d0 0x7f648806d990 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:31.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.982+0000 7f64b0a00640 1 --2- 172.21.15.28:0/2736149028 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f64ac0c21a0 0x7f64ac0ae2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:31.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.982+0000 7f64b0a00640 1 --2- 172.21.15.28:0/2736149028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64ac0ae800 0x7f64ac0c38c0 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:31.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.982+0000 7f64b0a00640 1 --2- 172.21.15.28:0/2736149028 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6490004d40 0x7f64ac0add80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:31.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.983+0000 7f64b0a00640 1 -- 172.21.15.28:0/2736149028 >> 172.21.15.28:0/2736149028 conn(0x7f64ac0ac2b0 msgr2=0x7f64ac151790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:31.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.983+0000 7f64b0a00640 1 -- 172.21.15.28:0/2736149028 shutdown_connections 2024-02-21T23:06:31.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:31.983+0000 7f64b0a00640 1 -- 172.21.15.28:0/2736149028 wait complete. 2024-02-21T23:06:31.979 INFO:tasks.ceph.mds.a.smithi028.stderr: -87> 2024-02-21T23:06:29.706+0000 7f87d1600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-21T23:06:31.992 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-02-21T23:06:32.025 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-02-21T23:06:32.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.190+0000 7fae9b200640 1 Processor -- start 2024-02-21T23:06:32.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.190+0000 7fae9b200640 1 -- start start 2024-02-21T23:06:32.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.191+0000 7fae9b200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fae941501c0 0x7fae94156c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:32.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.191+0000 7fae9b200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fae94154110 0x7fae9406f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:32.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.191+0000 7fae9b200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fae9406f670 0x7fae941533c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:32.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.191+0000 7fae9b200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae94152f30 con 0x7fae9406f670 2024-02-21T23:06:32.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.191+0000 7fae9b200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae94152c20 con 0x7fae94154110 2024-02-21T23:06:32.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.191+0000 7fae9b200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae9406fcd0 con 0x7fae941501c0 2024-02-21T23:06:32.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.191+0000 7fae9a800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fae9406f670 0x7fae941533c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:32.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.191+0000 7fae9a800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fae9406f670 0x7fae941533c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47480/0 (socket says 172.21.15.28:47480) 2024-02-21T23:06:32.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.191+0000 7fae9a800640 1 -- 172.21.15.28:0/1341034197 learned_addr learned my addr 172.21.15.28:0/1341034197 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:32.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.192+0000 7fae99e00640 1 --2- 172.21.15.28:0/1341034197 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fae941501c0 0x7fae94156c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:32.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.192+0000 7fae99400640 1 --2- 172.21.15.28:0/1341034197 >> v2:172.21.15.81:3300/0 conn(0x7fae94154110 0x7fae9406f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:32.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.192+0000 7fae9a800640 1 -- 172.21.15.28:0/1341034197 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fae941501c0 msgr2=0x7fae94156c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:32.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.192+0000 7fae9a800640 1 --2- 172.21.15.28:0/1341034197 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fae941501c0 0x7fae94156c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:32.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.192+0000 7fae9a800640 1 -- 172.21.15.28:0/1341034197 >> v2:172.21.15.81:3300/0 conn(0x7fae94154110 msgr2=0x7fae9406f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:32.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.192+0000 7fae9a800640 1 --2- 172.21.15.28:0/1341034197 >> v2:172.21.15.81:3300/0 conn(0x7fae94154110 0x7fae9406f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:32.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.192+0000 7fae9a800640 1 -- 172.21.15.28:0/1341034197 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae940c3230 con 0x7fae9406f670 2024-02-21T23:06:32.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.192+0000 7fae99400640 1 --2- 172.21.15.28:0/1341034197 >> v2:172.21.15.81:3300/0 conn(0x7fae94154110 0x7fae9406f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:06:32.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.193+0000 7fae9a800640 1 --2- 172.21.15.28:0/1341034197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae9406f670 0x7fae941533c0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7fae940c5b30 tx=0x7fae9002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba6eaef9078acb66 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:32.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.193+0000 7fae98a00640 1 -- 172.21.15.28:0/1341034197 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae9002ca70 con 0x7fae9406f670 2024-02-21T23:06:32.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.193+0000 7fae98a00640 1 -- 172.21.15.28:0/1341034197 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae9002cc10 con 0x7fae9406f670 2024-02-21T23:06:32.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.193+0000 7fae98a00640 1 -- 172.21.15.28:0/1341034197 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae900343f0 con 0x7fae9406f670 2024-02-21T23:06:32.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.194+0000 7fae9b200640 1 -- 172.21.15.28:0/1341034197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae9406f670 msgr2=0x7fae941533c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:32.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.194+0000 7fae9b200640 1 --2- 172.21.15.28:0/1341034197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae9406f670 0x7fae941533c0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7fae940c5b30 tx=0x7fae9002bed0 comp rx=0 tx=0).stop 2024-02-21T23:06:32.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.194+0000 7fae9b200640 1 -- 172.21.15.28:0/1341034197 shutdown_connections 2024-02-21T23:06:32.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.194+0000 7fae9b200640 1 --2- 172.21.15.28:0/1341034197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae9406f670 0x7fae941533c0 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:32.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.194+0000 7fae9b200640 1 --2- 172.21.15.28:0/1341034197 >> v2:172.21.15.81:3300/0 conn(0x7fae94154110 0x7fae9406f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:32.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.194+0000 7fae9b200640 1 --2- 172.21.15.28:0/1341034197 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fae941501c0 0x7fae94156c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:32.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.194+0000 7fae9b200640 1 -- 172.21.15.28:0/1341034197 >> 172.21.15.28:0/1341034197 conn(0x7fae940c8620 msgr2=0x7fae940c8a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:32.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.194+0000 7fae9b200640 1 -- 172.21.15.28:0/1341034197 shutdown_connections 2024-02-21T23:06:32.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.194+0000 7fae9b200640 1 -- 172.21.15.28:0/1341034197 wait complete. 2024-02-21T23:06:32.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.196+0000 7fae9b200640 1 Processor -- start 2024-02-21T23:06:32.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.196+0000 7fae9b200640 1 -- start start 2024-02-21T23:06:32.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.197+0000 7fae9b200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fae9406f670 0x7fae9414ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:32.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.197+0000 7fae9b200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fae941501c0 0x7fae940abc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:32.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.197+0000 7fae9b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae94154110 0x7fae940adac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:32.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.197+0000 7fae9b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae940c8310 con 0x7fae94154110 2024-02-21T23:06:32.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.197+0000 7fae9b200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae941505a0 con 0x7fae9406f670 2024-02-21T23:06:32.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.197+0000 7fae9b200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae940711b0 con 0x7fae941501c0 2024-02-21T23:06:32.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.197+0000 7fae9a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae94154110 0x7fae940adac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:32.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.197+0000 7fae9a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae94154110 0x7fae940adac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47490/0 (socket says 172.21.15.28:47490) 2024-02-21T23:06:32.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.197+0000 7fae9a800640 1 -- 172.21.15.28:0/449909447 learned_addr learned my addr 172.21.15.28:0/449909447 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:32.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.198+0000 7fae99400640 1 --2- 172.21.15.28:0/449909447 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fae941501c0 0x7fae940abc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:32.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.198+0000 7fae99e00640 1 --2- 172.21.15.28:0/449909447 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fae9406f670 0x7fae9414ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:32.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.198+0000 7fae9a800640 1 -- 172.21.15.28:0/449909447 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fae941501c0 msgr2=0x7fae940abc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:32.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.198+0000 7fae9a800640 1 --2- 172.21.15.28:0/449909447 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fae941501c0 0x7fae940abc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:32.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.198+0000 7fae9a800640 1 -- 172.21.15.28:0/449909447 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fae9406f670 msgr2=0x7fae9414ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:32.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.198+0000 7fae9a800640 1 --2- 172.21.15.28:0/449909447 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fae9406f670 0x7fae9414ab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:32.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.198+0000 7fae9a800640 1 -- 172.21.15.28:0/449909447 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae9414b0a0 con 0x7fae94154110 2024-02-21T23:06:32.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.198+0000 7fae9a800640 1 --2- 172.21.15.28:0/449909447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae94154110 0x7fae940adac0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fae90036040 tx=0x7fae90004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:32.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.198+0000 7fae8e000640 1 -- 172.21.15.28:0/449909447 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae90043020 con 0x7fae94154110 2024-02-21T23:06:32.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.198+0000 7fae8e000640 1 -- 172.21.15.28:0/449909447 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae9003e450 con 0x7fae94154110 2024-02-21T23:06:32.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.198+0000 7fae8e000640 1 -- 172.21.15.28:0/449909447 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae90009510 con 0x7fae94154110 2024-02-21T23:06:32.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.198+0000 7fae9b200640 1 -- 172.21.15.28:0/449909447 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae941df820 con 0x7fae94154110 2024-02-21T23:06:32.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.198+0000 7fae9b200640 1 -- 172.21.15.28:0/449909447 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae9414f840 con 0x7fae94154110 2024-02-21T23:06:32.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.200+0000 7fae9b200640 1 -- 172.21.15.28:0/449909447 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae9404f290 con 0x7fae94154110 2024-02-21T23:06:32.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.202+0000 7fae8e000640 1 -- 172.21.15.28:0/449909447 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fae90046020 con 0x7fae94154110 2024-02-21T23:06:32.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.203+0000 7fae8e000640 1 --2- 172.21.15.28:0/449909447 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fae7806b4d0 0x7fae7806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:32.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.203+0000 7fae8e000640 1 -- 172.21.15.28:0/449909447 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 7245+0+0 (secure 0 0 0) 0x7fae900ad080 con 0x7fae94154110 2024-02-21T23:06:32.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.203+0000 7fae99e00640 1 --2- 172.21.15.28:0/449909447 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fae7806b4d0 0x7fae7806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:32.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.204+0000 7fae99e00640 1 --2- 172.21.15.28:0/449909447 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fae7806b4d0 0x7fae7806d990 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fae88006fd0 tx=0x7fae88008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:32.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.207+0000 7fae8e000640 1 -- 172.21.15.28:0/449909447 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fae9004d030 con 0x7fae94154110 2024-02-21T23:06:32.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.449+0000 7fae9b200640 1 -- 172.21.15.28:0/449909447 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fae940c8ec0 con 0x7fae94154110 2024-02-21T23:06:32.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.473+0000 7fae8e000640 1 -- 172.21.15.28:0/449909447 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v76) v1 ==== 104+0+0 (secure 0 0 0) 0x7fae9007ba10 con 0x7fae94154110 2024-02-21T23:06:32.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.477+0000 7fae9b200640 1 -- 172.21.15.28:0/449909447 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fae7806b4d0 msgr2=0x7fae7806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:32.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.477+0000 7fae9b200640 1 --2- 172.21.15.28:0/449909447 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fae7806b4d0 0x7fae7806d990 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fae88006fd0 tx=0x7fae88008040 comp rx=0 tx=0).stop 2024-02-21T23:06:32.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.477+0000 7fae9b200640 1 -- 172.21.15.28:0/449909447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae94154110 msgr2=0x7fae940adac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:32.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.477+0000 7fae9b200640 1 --2- 172.21.15.28:0/449909447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae94154110 0x7fae940adac0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fae90036040 tx=0x7fae90004a40 comp rx=0 tx=0).stop 2024-02-21T23:06:32.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.478+0000 7fae9b200640 1 -- 172.21.15.28:0/449909447 shutdown_connections 2024-02-21T23:06:32.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.478+0000 7fae9b200640 1 --2- 172.21.15.28:0/449909447 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fae7806b4d0 0x7fae7806d990 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:32.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.478+0000 7fae9b200640 1 --2- 172.21.15.28:0/449909447 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fae941501c0 0x7fae940abc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:32.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.478+0000 7fae9b200640 1 --2- 172.21.15.28:0/449909447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae94154110 0x7fae940adac0 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:32.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.478+0000 7fae9b200640 1 --2- 172.21.15.28:0/449909447 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fae9406f670 0x7fae9414ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:32.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.478+0000 7fae9b200640 1 -- 172.21.15.28:0/449909447 >> 172.21.15.28:0/449909447 conn(0x7fae940c8620 msgr2=0x7fae9414c550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:32.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.478+0000 7fae9b200640 1 -- 172.21.15.28:0/449909447 shutdown_connections 2024-02-21T23:06:32.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.479+0000 7fae9b200640 1 -- 172.21.15.28:0/449909447 wait complete. 2024-02-21T23:06:32.473 INFO:tasks.ceph.mds.c.smithi028.stderr: -520> 2024-02-21T23:06:16.329+0000 7fd5d0000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-21T23:06:32.486 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-02-21T23:06:32.519 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-02-21T23:06:32.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.724+0000 7fd0a0600640 1 Processor -- start 2024-02-21T23:06:32.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.724+0000 7fd0a0600640 1 -- start start 2024-02-21T23:06:32.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.725+0000 7fd0a0600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd098156c10 0x7fd0980c52c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:32.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.726+0000 7fd0a0600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd09806f220 con 0x7fd09806e680 2024-02-21T23:06:32.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.726+0000 7fd0a0600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd09815a4c0 con 0x7fd0980c5a60 2024-02-21T23:06:32.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.726+0000 7fd0a0600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd098159f90 con 0x7fd098156c10 2024-02-21T23:06:32.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.726+0000 7fd09e600640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd09806e680 0x7fd09806eb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37506/0 (socket says 172.21.15.28:37506) 2024-02-21T23:06:32.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.726+0000 7fd09e600640 1 -- 172.21.15.28:0/174814314 learned_addr learned my addr 172.21.15.28:0/174814314 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:06:32.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.726+0000 7fd09dc00640 1 --2- 172.21.15.28:0/174814314 >> v2:172.21.15.28:3300/0 conn(0x7fd098156c10 0x7fd0980c52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:32.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.726+0000 7fd097e00640 1 -- 172.21.15.28:0/174814314 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4067473419 0 0) 0x7fd090001490 con 0x7fd09806e680 2024-02-21T23:06:32.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.726+0000 7fd097e00640 1 -- 172.21.15.28:0/174814314 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd080003520 con 0x7fd09806e680 2024-02-21T23:06:32.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.726+0000 7fd09dc00640 1 -- 172.21.15.28:0/174814314 >> v1:172.21.15.81:6789/0 conn(0x7fd0980c5a60 legacy=0x7fd0980c27e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:32.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.727+0000 7fd09dc00640 1 -- 172.21.15.28:0/174814314 >> v1:172.21.15.28:6789/0 conn(0x7fd09806e680 legacy=0x7fd09806eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:32.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.727+0000 7fd09dc00640 1 -- 172.21.15.28:0/174814314 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0980c4820 con 0x7fd098156c10 2024-02-21T23:06:32.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.728+0000 7fd09dc00640 1 --2- 172.21.15.28:0/174814314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd098156c10 0x7fd0980c52c0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fd08c0097a0 tx=0x7fd08c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bec211eff349d03c server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:32.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.728+0000 7fd097e00640 1 -- 172.21.15.28:0/174814314 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd08c035020 con 0x7fd098156c10 2024-02-21T23:06:32.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.728+0000 7fd097e00640 1 -- 172.21.15.28:0/174814314 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd08c031830 con 0x7fd098156c10 2024-02-21T23:06:32.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.728+0000 7fd097e00640 1 -- 172.21.15.28:0/174814314 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd08c031c00 con 0x7fd098156c10 2024-02-21T23:06:32.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.728+0000 7fd0a0600640 1 -- 172.21.15.28:0/174814314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd098156c10 msgr2=0x7fd0980c52c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:32.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.728+0000 7fd0a0600640 1 --2- 172.21.15.28:0/174814314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd098156c10 0x7fd0980c52c0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fd08c0097a0 tx=0x7fd08c02b8d0 comp rx=0 tx=0).stop 2024-02-21T23:06:32.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.729+0000 7fd0a0600640 1 -- 172.21.15.28:0/174814314 shutdown_connections 2024-02-21T23:06:32.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.729+0000 7fd0a0600640 1 --2- 172.21.15.28:0/174814314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd098156c10 0x7fd0980c52c0 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:32.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.729+0000 7fd0a0600640 1 -- 172.21.15.28:0/174814314 >> 172.21.15.28:0/174814314 conn(0x7fd09806b830 msgr2=0x7fd0980c67f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:32.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.729+0000 7fd0a0600640 1 -- 172.21.15.28:0/174814314 shutdown_connections 2024-02-21T23:06:32.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.729+0000 7fd0a0600640 1 -- 172.21.15.28:0/174814314 wait complete. 2024-02-21T23:06:32.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.730+0000 7fd0a0600640 1 Processor -- start 2024-02-21T23:06:32.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.730+0000 7fd0a0600640 1 -- start start 2024-02-21T23:06:32.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.731+0000 7fd0a0600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd09806e680 0x7fd0980aa9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:32.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.731+0000 7fd0a0600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0980c5a60 0x7fd0981ef2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:32.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.731+0000 7fd0a0600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd098156c10 0x7fd0980a9660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:32.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.731+0000 7fd0a0600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0980c4da0 con 0x7fd09806e680 2024-02-21T23:06:32.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.731+0000 7fd0a0600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0980c4ac0 con 0x7fd0980c5a60 2024-02-21T23:06:32.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.731+0000 7fd0a0600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0980c2ef0 con 0x7fd098156c10 2024-02-21T23:06:32.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.732+0000 7fd09dc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd09806e680 0x7fd0980aa9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:32.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.732+0000 7fd09dc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd09806e680 0x7fd0980aa9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47534/0 (socket says 172.21.15.28:47534) 2024-02-21T23:06:32.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.732+0000 7fd09dc00640 1 -- 172.21.15.28:0/496937435 learned_addr learned my addr 172.21.15.28:0/496937435 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:32.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.732+0000 7fd09dc00640 1 -- 172.21.15.28:0/496937435 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd098156c10 msgr2=0x7fd0980a9660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:32.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.732+0000 7fd09dc00640 1 --2- 172.21.15.28:0/496937435 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd098156c10 0x7fd0980a9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:32.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.732+0000 7fd09dc00640 1 -- 172.21.15.28:0/496937435 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0980c5a60 msgr2=0x7fd0981ef2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:32.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.732+0000 7fd09d200640 1 --2- 172.21.15.28:0/496937435 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0980c5a60 0x7fd0981ef2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:32.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.732+0000 7fd09dc00640 1 --2- 172.21.15.28:0/496937435 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0980c5a60 0x7fd0981ef2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:32.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.732+0000 7fd09dc00640 1 -- 172.21.15.28:0/496937435 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0981f2990 con 0x7fd09806e680 2024-02-21T23:06:32.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.732+0000 7fd09dc00640 1 --2- 172.21.15.28:0/496937435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd09806e680 0x7fd0980aa9c0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fd08c034040 tx=0x7fd08c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:32.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.733+0000 7fd095600640 1 -- 172.21.15.28:0/496937435 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd08c035040 con 0x7fd09806e680 2024-02-21T23:06:32.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.733+0000 7fd095600640 1 -- 172.21.15.28:0/496937435 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd08c004230 con 0x7fd09806e680 2024-02-21T23:06:32.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.733+0000 7fd0a0600640 1 -- 172.21.15.28:0/496937435 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd098158420 con 0x7fd09806e680 2024-02-21T23:06:32.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.733+0000 7fd095600640 1 -- 172.21.15.28:0/496937435 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd08c03a640 con 0x7fd09806e680 2024-02-21T23:06:32.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.733+0000 7fd0a0600640 1 -- 172.21.15.28:0/496937435 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd098159440 con 0x7fd09806e680 2024-02-21T23:06:32.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.734+0000 7fd0a0600640 1 -- 172.21.15.28:0/496937435 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0980c0d00 con 0x7fd09806e680 2024-02-21T23:06:32.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.735+0000 7fd095600640 1 -- 172.21.15.28:0/496937435 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd08c038020 con 0x7fd09806e680 2024-02-21T23:06:32.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.736+0000 7fd095600640 1 --2- 172.21.15.28:0/496937435 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd07806b4d0 0x7fd07806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:32.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.736+0000 7fd095600640 1 -- 172.21.15.28:0/496937435 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7331+0+0 (secure 0 0 0) 0x7fd08c0ac530 con 0x7fd09806e680 2024-02-21T23:06:32.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.736+0000 7fd09d200640 1 --2- 172.21.15.28:0/496937435 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd07806b4d0 0x7fd07806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:32.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.738+0000 7fd09d200640 1 --2- 172.21.15.28:0/496937435 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd07806b4d0 0x7fd07806d990 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fd098155c60 tx=0x7fd084009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:32.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.740+0000 7fd095600640 1 -- 172.21.15.28:0/496937435 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd0980c0d00 con 0x7fd09806e680 2024-02-21T23:06:32.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:32.933+0000 7fd0a0600640 1 -- 172.21.15.28:0/496937435 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd098067f30 con 0x7fd09806e680 2024-02-21T23:06:33.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.484+0000 7fd095600640 1 -- 172.21.15.28:0/496937435 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v86) v1 ==== 181+0+0 (secure 0 0 0) 0x7fd08c07ad40 con 0x7fd09806e680 2024-02-21T23:06:33.473 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' removed 2024-02-21T23:06:33.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.490+0000 7fd0a0600640 1 -- 172.21.15.28:0/496937435 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd07806b4d0 msgr2=0x7fd07806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:33.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.490+0000 7fd0a0600640 1 --2- 172.21.15.28:0/496937435 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd07806b4d0 0x7fd07806d990 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fd098155c60 tx=0x7fd084009040 comp rx=0 tx=0).stop 2024-02-21T23:06:33.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.490+0000 7fd0a0600640 1 -- 172.21.15.28:0/496937435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd09806e680 msgr2=0x7fd0980aa9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:33.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.490+0000 7fd0a0600640 1 --2- 172.21.15.28:0/496937435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd09806e680 0x7fd0980aa9c0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fd08c034040 tx=0x7fd08c0046c0 comp rx=0 tx=0).stop 2024-02-21T23:06:33.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.490+0000 7fd0a0600640 1 -- 172.21.15.28:0/496937435 shutdown_connections 2024-02-21T23:06:33.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.490+0000 7fd0a0600640 1 --2- 172.21.15.28:0/496937435 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd07806b4d0 0x7fd07806d990 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:33.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.490+0000 7fd0a0600640 1 --2- 172.21.15.28:0/496937435 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd098156c10 0x7fd0980a9660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:33.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.490+0000 7fd0a0600640 1 --2- 172.21.15.28:0/496937435 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd0980c5a60 0x7fd0981ef2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:33.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.490+0000 7fd0a0600640 1 --2- 172.21.15.28:0/496937435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd09806e680 0x7fd0980aa9c0 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:33.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.490+0000 7fd0a0600640 1 -- 172.21.15.28:0/496937435 >> 172.21.15.28:0/496937435 conn(0x7fd09806b830 msgr2=0x7fd0980a8cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:33.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.494+0000 7fd0a0600640 1 -- 172.21.15.28:0/496937435 shutdown_connections 2024-02-21T23:06:33.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.494+0000 7fd0a0600640 1 -- 172.21.15.28:0/496937435 wait complete. 2024-02-21T23:06:33.507 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-02-21T23:06:33.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.723+0000 7f45d5c00640 1 Processor -- start 2024-02-21T23:06:33.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.723+0000 7f45d5c00640 1 -- start start 2024-02-21T23:06:33.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.724+0000 7f45d5c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f45d006ea40 0x7f45d015e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:33.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.725+0000 7f45d5c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45d0152fd0 con 0x7f45d01501c0 2024-02-21T23:06:33.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.725+0000 7f45d5c00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45d01532b0 con 0x7f45d00c1010 2024-02-21T23:06:33.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.725+0000 7f45d5c00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45d0153590 con 0x7f45d006ea40 2024-02-21T23:06:33.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.725+0000 7f45cfe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f45d01501c0 0x7f45d00a96c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37514/0 (socket says 172.21.15.28:37514) 2024-02-21T23:06:33.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.725+0000 7f45cfe00640 1 -- 172.21.15.28:0/1158560505 learned_addr learned my addr 172.21.15.28:0/1158560505 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:06:33.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.725+0000 7f45d5200640 1 --2- 172.21.15.28:0/1158560505 >> v2:172.21.15.81:3300/0 conn(0x7f45d006ea40 0x7f45d015e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:33.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.725+0000 7f45cea00640 1 -- 172.21.15.28:0/1158560505 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3376244699 0 0) 0x7f45d0152fd0 con 0x7f45d01501c0 2024-02-21T23:06:33.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.725+0000 7f45cea00640 1 -- 172.21.15.28:0/1158560505 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45b4003520 con 0x7f45d01501c0 2024-02-21T23:06:33.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.726+0000 7f45cea00640 1 -- 172.21.15.28:0/1158560505 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2713852549 0 0) 0x7f45b4003520 con 0x7f45d01501c0 2024-02-21T23:06:33.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.726+0000 7f45cea00640 1 -- 172.21.15.28:0/1158560505 >> v2:172.21.15.81:3300/0 conn(0x7f45d006ea40 msgr2=0x7f45d015e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:33.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.726+0000 7f45cea00640 1 --2- 172.21.15.28:0/1158560505 >> v2:172.21.15.81:3300/0 conn(0x7f45d006ea40 0x7f45d015e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:33.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.726+0000 7f45cea00640 1 -- 172.21.15.28:0/1158560505 >> v1:172.21.15.81:6789/0 conn(0x7f45d00c1010 legacy=0x7f45d006e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:33.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.726+0000 7f45cea00640 1 -- 172.21.15.28:0/1158560505 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45d0153870 con 0x7f45d01501c0 2024-02-21T23:06:33.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.726+0000 7f45cea00640 1 -- 172.21.15.28:0/1158560505 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f45c0001670 con 0x7f45d01501c0 2024-02-21T23:06:33.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.726+0000 7f45cea00640 1 -- 172.21.15.28:0/1158560505 >> v1:172.21.15.28:6789/0 conn(0x7f45d01501c0 legacy=0x7f45d00a96c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:33.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.726+0000 7f45cea00640 1 --2- 172.21.15.28:0/1158560505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45b40039b0 0x7f45b4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:33.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.726+0000 7f45cea00640 1 --2- 172.21.15.28:0/1158560505 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f45b4004800 0x7f45b4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:33.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.726+0000 7f45cea00640 1 --2- 172.21.15.28:0/1158560505 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f45b40051c0 0x7f45b400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:33.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.726+0000 7f45cea00640 1 -- 172.21.15.28:0/1158560505 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45d0153590 con 0x7f45b40039b0 2024-02-21T23:06:33.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.726+0000 7f45cea00640 1 -- 172.21.15.28:0/1158560505 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45b400c160 con 0x7f45b40051c0 2024-02-21T23:06:33.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.726+0000 7f45cea00640 1 -- 172.21.15.28:0/1158560505 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45b400c360 con 0x7f45b4004800 2024-02-21T23:06:33.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.726+0000 7f45cfe00640 1 --2- 172.21.15.28:0/1158560505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45b40039b0 0x7f45b4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:33.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.726+0000 7f45d5200640 1 --2- 172.21.15.28:0/1158560505 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f45b40051c0 0x7f45b400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:33.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.727+0000 7f45cfe00640 1 -- 172.21.15.28:0/1158560505 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f45b4004800 msgr2=0x7f45b4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:33.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.727+0000 7f45cf400640 1 --2- 172.21.15.28:0/1158560505 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f45b4004800 0x7f45b4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:33.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.727+0000 7f45cfe00640 1 --2- 172.21.15.28:0/1158560505 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f45b4004800 0x7f45b4004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:33.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.727+0000 7f45cfe00640 1 -- 172.21.15.28:0/1158560505 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f45b40051c0 msgr2=0x7f45b400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:33.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.727+0000 7f45cfe00640 1 --2- 172.21.15.28:0/1158560505 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f45b40051c0 0x7f45b400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:33.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.727+0000 7f45cfe00640 1 -- 172.21.15.28:0/1158560505 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45b4003520 con 0x7f45b40039b0 2024-02-21T23:06:33.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.727+0000 7f45d5200640 1 --2- 172.21.15.28:0/1158560505 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f45b40051c0 0x7f45b400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:06:33.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.727+0000 7f45d5200640 1 -- 172.21.15.28:0/1158560505 reap_dead start 2024-02-21T23:06:33.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.728+0000 7f45cfe00640 1 --2- 172.21.15.28:0/1158560505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45b40039b0 0x7f45b4003dd0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f45c000c240 tx=0x7f45b4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58f71f3bac6c1d33 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:33.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.728+0000 7f45cea00640 1 -- 172.21.15.28:0/1158560505 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c0007cf0 con 0x7f45b40039b0 2024-02-21T23:06:33.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.728+0000 7f45cea00640 1 -- 172.21.15.28:0/1158560505 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45c0030070 con 0x7f45b40039b0 2024-02-21T23:06:33.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.728+0000 7f45cea00640 1 -- 172.21.15.28:0/1158560505 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c0005d70 con 0x7f45b40039b0 2024-02-21T23:06:33.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.729+0000 7f45d5c00640 1 -- 172.21.15.28:0/1158560505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45b40039b0 msgr2=0x7f45b4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:33.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.729+0000 7f45d5c00640 1 --2- 172.21.15.28:0/1158560505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45b40039b0 0x7f45b4003dd0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f45c000c240 tx=0x7f45b4005830 comp rx=0 tx=0).stop 2024-02-21T23:06:33.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.729+0000 7f45d5c00640 1 -- 172.21.15.28:0/1158560505 shutdown_connections 2024-02-21T23:06:33.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.729+0000 7f45d5c00640 1 --2- 172.21.15.28:0/1158560505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45b40039b0 0x7f45b4003dd0 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:33.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.729+0000 7f45d5c00640 1 -- 172.21.15.28:0/1158560505 >> 172.21.15.28:0/1158560505 conn(0x7f45d00c6610 msgr2=0x7f45d00c69f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:33.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.729+0000 7f45d5c00640 1 -- 172.21.15.28:0/1158560505 shutdown_connections 2024-02-21T23:06:33.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.729+0000 7f45d5c00640 1 -- 172.21.15.28:0/1158560505 wait complete. 2024-02-21T23:06:33.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.731+0000 7f45d5c00640 1 Processor -- start 2024-02-21T23:06:33.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.731+0000 7f45d5c00640 1 -- start start 2024-02-21T23:06:33.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.732+0000 7f45d5c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f45b40039b0 0x7f45d00c1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:33.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.732+0000 7f45d5c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f45d01424e0 0x7f45d014ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:33.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.732+0000 7f45d5c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45d00c2290 0x7f45d014bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:33.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.732+0000 7f45d5c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45d00c7340 con 0x7f45d00c2290 2024-02-21T23:06:33.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.732+0000 7f45d5c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45d00c0c20 con 0x7f45d01424e0 2024-02-21T23:06:33.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.732+0000 7f45d5c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45d01505a0 con 0x7f45b40039b0 2024-02-21T23:06:33.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.732+0000 7f45d5200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45d00c2290 0x7f45d014bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:33.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.732+0000 7f45d5200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45d00c2290 0x7f45d014bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47554/0 (socket says 172.21.15.28:47554) 2024-02-21T23:06:33.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.732+0000 7f45d5200640 1 -- 172.21.15.28:0/802575102 learned_addr learned my addr 172.21.15.28:0/802575102 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:33.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.732+0000 7f45cfe00640 1 --2- 172.21.15.28:0/802575102 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f45b40039b0 0x7f45d00c1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:33.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.732+0000 7f45cf400640 1 --2- 172.21.15.28:0/802575102 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f45d01424e0 0x7f45d014ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:33.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.733+0000 7f45d5200640 1 -- 172.21.15.28:0/802575102 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f45b40039b0 msgr2=0x7f45d00c1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:33.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.733+0000 7f45d5200640 1 --2- 172.21.15.28:0/802575102 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f45b40039b0 0x7f45d00c1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:33.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.733+0000 7f45d5200640 1 -- 172.21.15.28:0/802575102 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f45d01424e0 msgr2=0x7f45d014ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:33.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.733+0000 7f45d5200640 1 --2- 172.21.15.28:0/802575102 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f45d01424e0 0x7f45d014ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:33.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.733+0000 7f45d5200640 1 -- 172.21.15.28:0/802575102 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45d01ec600 con 0x7f45d00c2290 2024-02-21T23:06:33.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.733+0000 7f45cfe00640 1 --2- 172.21.15.28:0/802575102 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f45b40039b0 0x7f45d00c1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:06:33.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.733+0000 7f45d5200640 1 --2- 172.21.15.28:0/802575102 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45d00c2290 0x7f45d014bfb0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f45d00c30e0 tx=0x7f45c8010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:33.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.733+0000 7f45c7e00640 1 -- 172.21.15.28:0/802575102 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c801a030 con 0x7f45d00c2290 2024-02-21T23:06:33.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.733+0000 7f45c7e00640 1 -- 172.21.15.28:0/802575102 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45c8010da0 con 0x7f45d00c2290 2024-02-21T23:06:33.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.733+0000 7f45c7e00640 1 -- 172.21.15.28:0/802575102 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c8003bd0 con 0x7f45d00c2290 2024-02-21T23:06:33.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.733+0000 7f45d5c00640 1 -- 172.21.15.28:0/802575102 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45d01ec080 con 0x7f45d00c2290 2024-02-21T23:06:33.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.733+0000 7f45d5c00640 1 -- 172.21.15.28:0/802575102 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45d01ec320 con 0x7f45d00c2290 2024-02-21T23:06:33.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.735+0000 7f45d5c00640 1 -- 172.21.15.28:0/802575102 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45d004f290 con 0x7f45d00c2290 2024-02-21T23:06:33.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.736+0000 7f45c7e00640 1 -- 172.21.15.28:0/802575102 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f45c801d070 con 0x7f45d00c2290 2024-02-21T23:06:33.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.737+0000 7f45c7e00640 1 --2- 172.21.15.28:0/802575102 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f45b006b500 0x7f45b006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:33.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.737+0000 7f45c7e00640 1 -- 172.21.15.28:0/802575102 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6893+0+0 (secure 0 0 0) 0x7f45c808b180 con 0x7f45d00c2290 2024-02-21T23:06:33.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.738+0000 7f45cfe00640 1 --2- 172.21.15.28:0/802575102 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f45b006b500 0x7f45b006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:33.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.739+0000 7f45cfe00640 1 --2- 172.21.15.28:0/802575102 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f45b006b500 0x7f45b006d9c0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f45c0034000 tx=0x7f45c0039040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:33.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.742+0000 7f45c7e00640 1 -- 172.21.15.28:0/802575102 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f45c8059b50 con 0x7f45d00c2290 2024-02-21T23:06:33.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:33.952+0000 7f45d5c00640 1 -- 172.21.15.28:0/802575102 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f45d00c6b90 con 0x7f45d00c2290 2024-02-21T23:06:34.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.487+0000 7f45c7e00640 1 -- 172.21.15.28:0/802575102 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v87) v1 ==== 169+0+0 (secure 0 0 0) 0x7f45c8059450 con 0x7f45d00c2290 2024-02-21T23:06:34.476 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' removed 2024-02-21T23:06:34.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.493+0000 7f45d5c00640 1 -- 172.21.15.28:0/802575102 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f45b006b500 msgr2=0x7f45b006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:34.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.493+0000 7f45d5c00640 1 --2- 172.21.15.28:0/802575102 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f45b006b500 0x7f45b006d9c0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f45c0034000 tx=0x7f45c0039040 comp rx=0 tx=0).stop 2024-02-21T23:06:34.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.493+0000 7f45d5c00640 1 -- 172.21.15.28:0/802575102 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45d00c2290 msgr2=0x7f45d014bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:34.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.493+0000 7f45d5c00640 1 --2- 172.21.15.28:0/802575102 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45d00c2290 0x7f45d014bfb0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f45d00c30e0 tx=0x7f45c8010590 comp rx=0 tx=0).stop 2024-02-21T23:06:34.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.493+0000 7f45d5c00640 1 -- 172.21.15.28:0/802575102 shutdown_connections 2024-02-21T23:06:34.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.493+0000 7f45d5c00640 1 --2- 172.21.15.28:0/802575102 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f45b006b500 0x7f45b006d9c0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:34.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.493+0000 7f45d5c00640 1 --2- 172.21.15.28:0/802575102 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45d00c2290 0x7f45d014bfb0 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:34.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.493+0000 7f45d5c00640 1 --2- 172.21.15.28:0/802575102 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f45d01424e0 0x7f45d014ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:34.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.493+0000 7f45d5c00640 1 --2- 172.21.15.28:0/802575102 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f45b40039b0 0x7f45d00c1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:34.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.494+0000 7f45d5c00640 1 -- 172.21.15.28:0/802575102 >> 172.21.15.28:0/802575102 conn(0x7f45d00c6610 msgr2=0x7f45d01525e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:34.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.494+0000 7f45d5c00640 1 -- 172.21.15.28:0/802575102 shutdown_connections 2024-02-21T23:06:34.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.494+0000 7f45d5c00640 1 -- 172.21.15.28:0/802575102 wait complete. 2024-02-21T23:06:34.509 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:06:34.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.713+0000 7f9190a00640 1 Processor -- start 2024-02-21T23:06:34.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.713+0000 7f9190a00640 1 -- start start 2024-02-21T23:06:34.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.715+0000 7f9190a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f918c0aff90 0x7f918c0c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:34.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.715+0000 7f9190a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f918c06dbf0 0x7f918c158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:34.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.715+0000 7f9190a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f918c0c5720 0x7f918c159000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:34.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.715+0000 7f9190a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f918c0c7270 con 0x7f918c0c5720 2024-02-21T23:06:34.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.715+0000 7f9190a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f918c158630 con 0x7f918c0aff90 2024-02-21T23:06:34.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.715+0000 7f9190a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f918c0c74c0 con 0x7f918c06dbf0 2024-02-21T23:06:34.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.715+0000 7f918b200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f918c0c5720 0x7f918c159000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:34.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.715+0000 7f918b200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f918c0c5720 0x7f918c159000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47564/0 (socket says 172.21.15.28:47564) 2024-02-21T23:06:34.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.715+0000 7f918b200640 1 -- 172.21.15.28:0/2907571521 learned_addr learned my addr 172.21.15.28:0/2907571521 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:34.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.715+0000 7f918a800640 1 --2- 172.21.15.28:0/2907571521 >> v2:172.21.15.81:3300/0 conn(0x7f918c0aff90 0x7f918c0c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:34.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.715+0000 7f9189e00640 1 --2- 172.21.15.28:0/2907571521 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f918c06dbf0 0x7f918c158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:34.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.715+0000 7f918b200640 1 -- 172.21.15.28:0/2907571521 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f918c06dbf0 msgr2=0x7f918c158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:34.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.715+0000 7f918b200640 1 --2- 172.21.15.28:0/2907571521 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f918c06dbf0 0x7f918c158ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:34.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.716+0000 7f918b200640 1 -- 172.21.15.28:0/2907571521 >> v2:172.21.15.81:3300/0 conn(0x7f918c0aff90 msgr2=0x7f918c0c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:34.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.716+0000 7f918b200640 1 --2- 172.21.15.28:0/2907571521 >> v2:172.21.15.81:3300/0 conn(0x7f918c0aff90 0x7f918c0c51e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:34.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.716+0000 7f918b200640 1 -- 172.21.15.28:0/2907571521 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f918c0c77a0 con 0x7f918c0c5720 2024-02-21T23:06:34.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.716+0000 7f9189e00640 1 --2- 172.21.15.28:0/2907571521 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f918c06dbf0 0x7f918c158ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:06:34.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.717+0000 7f918b200640 1 --2- 172.21.15.28:0/2907571521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f918c0c5720 0x7f918c159000 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f918c0aa1b0 tx=0x7f918402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5938768faf3ad63 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:34.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.717+0000 7f9189400640 1 -- 172.21.15.28:0/2907571521 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f918402ca80 con 0x7f918c0c5720 2024-02-21T23:06:34.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.717+0000 7f9189400640 1 -- 172.21.15.28:0/2907571521 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f918402cc20 con 0x7f918c0c5720 2024-02-21T23:06:34.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.717+0000 7f9189400640 1 -- 172.21.15.28:0/2907571521 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91840343f0 con 0x7f918c0c5720 2024-02-21T23:06:34.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.718+0000 7f9190a00640 1 -- 172.21.15.28:0/2907571521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f918c0c5720 msgr2=0x7f918c159000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:34.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.718+0000 7f9190a00640 1 --2- 172.21.15.28:0/2907571521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f918c0c5720 0x7f918c159000 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f918c0aa1b0 tx=0x7f918402bee0 comp rx=0 tx=0).stop 2024-02-21T23:06:34.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.718+0000 7f9190a00640 1 -- 172.21.15.28:0/2907571521 shutdown_connections 2024-02-21T23:06:34.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.718+0000 7f9190a00640 1 --2- 172.21.15.28:0/2907571521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f918c0c5720 0x7f918c159000 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:34.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.718+0000 7f9190a00640 1 --2- 172.21.15.28:0/2907571521 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f918c06dbf0 0x7f918c158ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:34.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.718+0000 7f9190a00640 1 --2- 172.21.15.28:0/2907571521 >> v2:172.21.15.81:3300/0 conn(0x7f918c0aff90 0x7f918c0c51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:34.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.718+0000 7f9190a00640 1 -- 172.21.15.28:0/2907571521 >> 172.21.15.28:0/2907571521 conn(0x7f918c0ac370 msgr2=0x7f918c0ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:34.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.718+0000 7f9190a00640 1 -- 172.21.15.28:0/2907571521 shutdown_connections 2024-02-21T23:06:34.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.718+0000 7f9190a00640 1 -- 172.21.15.28:0/2907571521 wait complete. 2024-02-21T23:06:34.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.720+0000 7f9190a00640 1 Processor -- start 2024-02-21T23:06:34.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.720+0000 7f9190a00640 1 -- start start 2024-02-21T23:06:34.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.721+0000 7f9190a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f918c06dbf0 0x7f918c1eb080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:34.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.721+0000 7f9190a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f918c0aff90 0x7f918c0c8da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:34.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.721+0000 7f9190a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f918c0c5720 0x7f918c1ee290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:34.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.721+0000 7f9190a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f918c0c8760 con 0x7f918c06dbf0 2024-02-21T23:06:34.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.721+0000 7f9190a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f918c0be0c0 con 0x7f918c0c5720 2024-02-21T23:06:34.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.721+0000 7f9190a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f918c0c88e0 con 0x7f918c0aff90 2024-02-21T23:06:34.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.721+0000 7f918a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f918c06dbf0 0x7f918c1eb080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:34.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.721+0000 7f918a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f918c06dbf0 0x7f918c1eb080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47570/0 (socket says 172.21.15.28:47570) 2024-02-21T23:06:34.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.721+0000 7f918a800640 1 -- 172.21.15.28:0/1424837918 learned_addr learned my addr 172.21.15.28:0/1424837918 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:34.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.721+0000 7f9189e00640 1 --2- 172.21.15.28:0/1424837918 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f918c0aff90 0x7f918c0c8da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:34.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.721+0000 7f918b200640 1 --2- 172.21.15.28:0/1424837918 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f918c0c5720 0x7f918c1ee290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:34.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.722+0000 7f918a800640 1 -- 172.21.15.28:0/1424837918 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f918c0aff90 msgr2=0x7f918c0c8da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:34.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.722+0000 7f918a800640 1 --2- 172.21.15.28:0/1424837918 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f918c0aff90 0x7f918c0c8da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:34.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.722+0000 7f918a800640 1 -- 172.21.15.28:0/1424837918 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f918c0c5720 msgr2=0x7f918c1ee290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:34.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.722+0000 7f918a800640 1 --2- 172.21.15.28:0/1424837918 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f918c0c5720 0x7f918c1ee290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:34.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.722+0000 7f918a800640 1 -- 172.21.15.28:0/1424837918 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f918c1eb5c0 con 0x7f918c06dbf0 2024-02-21T23:06:34.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.722+0000 7f918b200640 1 --2- 172.21.15.28:0/1424837918 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f918c0c5720 0x7f918c1ee290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:06:34.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.722+0000 7f918a800640 1 --2- 172.21.15.28:0/1424837918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f918c06dbf0 0x7f918c1eb080 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f917c00c7f0 tx=0x7f917c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:34.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.722+0000 7f9182a00640 1 -- 172.21.15.28:0/1424837918 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f917c015070 con 0x7f918c06dbf0 2024-02-21T23:06:34.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.722+0000 7f9182a00640 1 -- 172.21.15.28:0/1424837918 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f917c004240 con 0x7f918c06dbf0 2024-02-21T23:06:34.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.722+0000 7f9182a00640 1 -- 172.21.15.28:0/1424837918 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f917c007620 con 0x7f918c06dbf0 2024-02-21T23:06:34.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.722+0000 7f9190a00640 1 -- 172.21.15.28:0/1424837918 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f918c0c3c80 con 0x7f918c06dbf0 2024-02-21T23:06:34.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.722+0000 7f9190a00640 1 -- 172.21.15.28:0/1424837918 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f918c0c4240 con 0x7f918c06dbf0 2024-02-21T23:06:34.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.724+0000 7f9190a00640 1 -- 172.21.15.28:0/1424837918 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f918c0c7fb0 con 0x7f918c06dbf0 2024-02-21T23:06:34.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.724+0000 7f9182a00640 1 -- 172.21.15.28:0/1424837918 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f917c0079d0 con 0x7f918c06dbf0 2024-02-21T23:06:34.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.725+0000 7f9182a00640 1 --2- 172.21.15.28:0/1424837918 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f916c06b4d0 0x7f916c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:34.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.725+0000 7f9182a00640 1 -- 172.21.15.28:0/1424837918 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6479+0+0 (secure 0 0 0) 0x7f917c08a570 con 0x7f918c06dbf0 2024-02-21T23:06:34.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.726+0000 7f9189e00640 1 --2- 172.21.15.28:0/1424837918 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f916c06b4d0 0x7f916c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:34.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.727+0000 7f9189e00640 1 --2- 172.21.15.28:0/1424837918 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f916c06b4d0 0x7f916c06d990 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f918c1eed10 tx=0x7f9178008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:34.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.730+0000 7f9182a00640 1 -- 172.21.15.28:0/1424837918 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f918c0c7fb0 con 0x7f918c06dbf0 2024-02-21T23:06:34.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.931+0000 7f9190a00640 1 -- 172.21.15.28:0/1424837918 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f918c04f290 con 0x7f918c06dbf0 2024-02-21T23:06:34.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.934+0000 7f9182a00640 1 -- 172.21.15.28:0/1424837918 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+17407 (secure 0 0 0) 0x7f918c1eb5c0 con 0x7f918c06dbf0 2024-02-21T23:06:34.922 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:06:34.922 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":87,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:06:34.484343+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs2_metadata","create_time":"2024-02-21T23:06:08.240443+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs2_data","create_time":"2024-02-21T23:06:09.466016+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.2","mappings":[{"from":1,"to":7}]},{"pgid":"14.12","mappings":[{"from":1,"to":5}]},{"pgid":"14.1e","mappings":[{"from":1,"to":2}]},{"pgid":"14.24","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6833/624384999":"2024-02-22T23:06:32.469969+0000","172.21.15.28:6832/624384999":"2024-02-22T23:06:32.469969+0000","172.21.15.28:6835/1066741568":"2024-02-22T23:06:31.975858+0000","172.21.15.28:6834/1066741568":"2024-02-22T23:06:31.975858+0000","172.21.15.28:6835/595682298":"2024-02-22T23:06:23.690106+0000","172.21.15.28:6834/595682298":"2024-02-22T23:06:23.690106+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:06:34.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.936+0000 7f9190a00640 1 -- 172.21.15.28:0/1424837918 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f916c06b4d0 msgr2=0x7f916c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:34.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.936+0000 7f9190a00640 1 --2- 172.21.15.28:0/1424837918 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f916c06b4d0 0x7f916c06d990 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f918c1eed10 tx=0x7f9178008040 comp rx=0 tx=0).stop 2024-02-21T23:06:34.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.936+0000 7f9190a00640 1 -- 172.21.15.28:0/1424837918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f918c06dbf0 msgr2=0x7f918c1eb080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:34.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.936+0000 7f9190a00640 1 --2- 172.21.15.28:0/1424837918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f918c06dbf0 0x7f918c1eb080 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f917c00c7f0 tx=0x7f917c00ccb0 comp rx=0 tx=0).stop 2024-02-21T23:06:34.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.936+0000 7f9190a00640 1 -- 172.21.15.28:0/1424837918 shutdown_connections 2024-02-21T23:06:34.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.936+0000 7f9190a00640 1 --2- 172.21.15.28:0/1424837918 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f916c06b4d0 0x7f916c06d990 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:34.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.936+0000 7f9190a00640 1 --2- 172.21.15.28:0/1424837918 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f918c0aff90 0x7f918c0c8da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:34.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.936+0000 7f9190a00640 1 --2- 172.21.15.28:0/1424837918 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f918c0c5720 0x7f918c1ee290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:34.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.936+0000 7f9190a00640 1 --2- 172.21.15.28:0/1424837918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f918c06dbf0 0x7f918c1eb080 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:34.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.936+0000 7f9190a00640 1 -- 172.21.15.28:0/1424837918 >> 172.21.15.28:0/1424837918 conn(0x7f918c0ac370 msgr2=0x7f918c0ac750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:34.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.936+0000 7f9190a00640 1 -- 172.21.15.28:0/1424837918 shutdown_connections 2024-02-21T23:06:34.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:34.936+0000 7f9190a00640 1 -- 172.21.15.28:0/1424837918 wait complete. 2024-02-21T23:06:34.943 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-21T23:02:30.728329+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs2_metadata', 'create_time': '2024-02-21T23:06:08.240443+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs2'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs2_data', 'create_time': '2024-02-21T23:06:09.466016+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs2'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-21T23:06:34.944 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-21T23:06:35.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.171+0000 7f90efe00640 1 Processor -- start 2024-02-21T23:06:35.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.171+0000 7f90efe00640 1 -- start start 2024-02-21T23:06:35.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.173+0000 7f90efe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f90f013e0c0 0x7f90f00c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:35.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.173+0000 7f90efe00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f90f00c76f0 0x7f90f006c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:35.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.173+0000 7f90efe00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90f0158550 con 0x7f90f006dbc0 2024-02-21T23:06:35.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.173+0000 7f90efe00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90f00c89a0 con 0x7f90f013e0c0 2024-02-21T23:06:35.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.173+0000 7f90efe00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90f01587d0 con 0x7f90f00c76f0 2024-02-21T23:06:35.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.173+0000 7f90eea00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f90f013e0c0 0x7f90f00c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:35.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.173+0000 7f90e7e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f90f006dbc0 0x7f90f006c280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59482/0 (socket says 172.21.15.28:59482) 2024-02-21T23:06:35.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.173+0000 7f90e7e00640 1 -- 172.21.15.28:0/1282426450 learned_addr learned my addr 172.21.15.28:0/1282426450 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:06:35.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.174+0000 7f90ef400640 1 --2- 172.21.15.28:0/1282426450 >> v2:172.21.15.81:3300/0 conn(0x7f90f00c76f0 0x7f90f006c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:35.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.174+0000 7f90eea00640 1 -- 172.21.15.28:0/1282426450 >> v2:172.21.15.81:3300/0 conn(0x7f90f00c76f0 msgr2=0x7f90f006c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:35.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.174+0000 7f90eea00640 1 --2- 172.21.15.28:0/1282426450 >> v2:172.21.15.81:3300/0 conn(0x7f90f00c76f0 0x7f90f006c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:35.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.174+0000 7f90eea00640 1 -- 172.21.15.28:0/1282426450 >> v1:172.21.15.28:6789/0 conn(0x7f90f006dbc0 legacy=0x7f90f006c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:35.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.174+0000 7f90eea00640 1 -- 172.21.15.28:0/1282426450 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90f0158ab0 con 0x7f90f013e0c0 2024-02-21T23:06:35.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.175+0000 7f90eea00640 1 --2- 172.21.15.28:0/1282426450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90f013e0c0 0x7f90f00c71b0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f90f00c8260 tx=0x7f90e002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e2c79235ada8c24 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:35.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.175+0000 7f90ee000640 1 -- 172.21.15.28:0/1282426450 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90e00315a0 con 0x7f90f013e0c0 2024-02-21T23:06:35.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.175+0000 7f90ee000640 1 -- 172.21.15.28:0/1282426450 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90e0037070 con 0x7f90f013e0c0 2024-02-21T23:06:35.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.175+0000 7f90ee000640 1 -- 172.21.15.28:0/1282426450 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90e0031960 con 0x7f90f013e0c0 2024-02-21T23:06:35.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.176+0000 7f90efe00640 1 -- 172.21.15.28:0/1282426450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90f013e0c0 msgr2=0x7f90f00c71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:35.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.176+0000 7f90efe00640 1 --2- 172.21.15.28:0/1282426450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90f013e0c0 0x7f90f00c71b0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f90f00c8260 tx=0x7f90e002b840 comp rx=0 tx=0).stop 2024-02-21T23:06:35.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.176+0000 7f90efe00640 1 -- 172.21.15.28:0/1282426450 shutdown_connections 2024-02-21T23:06:35.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.176+0000 7f90efe00640 1 --2- 172.21.15.28:0/1282426450 >> v2:172.21.15.81:3300/0 conn(0x7f90f00c76f0 0x7f90f006c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:35.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.176+0000 7f90efe00640 1 --2- 172.21.15.28:0/1282426450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90f013e0c0 0x7f90f00c71b0 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:35.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.176+0000 7f90efe00640 1 -- 172.21.15.28:0/1282426450 >> 172.21.15.28:0/1282426450 conn(0x7f90f0073110 msgr2=0x7f90f00ac490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:35.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.176+0000 7f90efe00640 1 -- 172.21.15.28:0/1282426450 shutdown_connections 2024-02-21T23:06:35.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.176+0000 7f90efe00640 1 -- 172.21.15.28:0/1282426450 wait complete. 2024-02-21T23:06:35.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.178+0000 7f90efe00640 1 Processor -- start 2024-02-21T23:06:35.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.178+0000 7f90efe00640 1 -- start start 2024-02-21T23:06:35.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.179+0000 7f90efe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90f006dbc0 0x7f90f01511c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:35.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.179+0000 7f90efe00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f90f00c76f0 0x7f90f014ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:35.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.179+0000 7f90efe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f90f013e0c0 0x7f90f014f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:35.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.179+0000 7f90efe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90f006ced0 con 0x7f90f006dbc0 2024-02-21T23:06:35.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.179+0000 7f90efe00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90f0159070 con 0x7f90f013e0c0 2024-02-21T23:06:35.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.179+0000 7f90efe00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90f013e520 con 0x7f90f00c76f0 2024-02-21T23:06:35.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.179+0000 7f90eea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90f006dbc0 0x7f90f01511c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:35.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.179+0000 7f90eea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90f006dbc0 0x7f90f01511c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59186/0 (socket says 172.21.15.28:59186) 2024-02-21T23:06:35.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.179+0000 7f90eea00640 1 -- 172.21.15.28:0/897708564 learned_addr learned my addr 172.21.15.28:0/897708564 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:35.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.179+0000 7f90ef400640 1 --2- 172.21.15.28:0/897708564 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f90f013e0c0 0x7f90f014f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:35.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.180+0000 7f90e7e00640 1 --2- 172.21.15.28:0/897708564 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f90f00c76f0 0x7f90f014ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:35.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.180+0000 7f90eea00640 1 -- 172.21.15.28:0/897708564 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f90f00c76f0 msgr2=0x7f90f014ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:35.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.180+0000 7f90eea00640 1 --2- 172.21.15.28:0/897708564 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f90f00c76f0 0x7f90f014ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:35.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.180+0000 7f90eea00640 1 -- 172.21.15.28:0/897708564 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f90f013e0c0 msgr2=0x7f90f014f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:35.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.180+0000 7f90eea00640 1 --2- 172.21.15.28:0/897708564 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f90f013e0c0 0x7f90f014f350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:35.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.180+0000 7f90eea00640 1 -- 172.21.15.28:0/897708564 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90f0138040 con 0x7f90f006dbc0 2024-02-21T23:06:35.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.180+0000 7f90ef400640 1 --2- 172.21.15.28:0/897708564 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f90f013e0c0 0x7f90f014f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:06:35.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.180+0000 7f90eea00640 1 --2- 172.21.15.28:0/897708564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90f006dbc0 0x7f90f01511c0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f90e0009890 tx=0x7f90e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:35.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.180+0000 7f90e6a00640 1 -- 172.21.15.28:0/897708564 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90e0032630 con 0x7f90f006dbc0 2024-02-21T23:06:35.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.180+0000 7f90e6a00640 1 -- 172.21.15.28:0/897708564 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90e00327d0 con 0x7f90f006dbc0 2024-02-21T23:06:35.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.180+0000 7f90e6a00640 1 -- 172.21.15.28:0/897708564 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90e0032ba0 con 0x7f90f006dbc0 2024-02-21T23:06:35.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.180+0000 7f90efe00640 1 -- 172.21.15.28:0/897708564 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90f01eb230 con 0x7f90f006dbc0 2024-02-21T23:06:35.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.181+0000 7f90efe00640 1 -- 172.21.15.28:0/897708564 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90f014f890 con 0x7f90f006dbc0 2024-02-21T23:06:35.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.182+0000 7f90efe00640 1 -- 172.21.15.28:0/897708564 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90f004f290 con 0x7f90f006dbc0 2024-02-21T23:06:35.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.183+0000 7f90e6a00640 1 -- 172.21.15.28:0/897708564 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f90e003b3f0 con 0x7f90f006dbc0 2024-02-21T23:06:35.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.184+0000 7f90e6a00640 1 --2- 172.21.15.28:0/897708564 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f90cc06b4d0 0x7f90cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:35.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.184+0000 7f90e6a00640 1 -- 172.21.15.28:0/897708564 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6479+0+0 (secure 0 0 0) 0x7f90e002f030 con 0x7f90f006dbc0 2024-02-21T23:06:35.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.184+0000 7f90e7e00640 1 --2- 172.21.15.28:0/897708564 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f90cc06b4d0 0x7f90cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:35.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.185+0000 7f90e7e00640 1 --2- 172.21.15.28:0/897708564 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f90cc06b4d0 0x7f90cc06d990 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f90f0156120 tx=0x7f90d8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:35.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.188+0000 7f90e6a00640 1 -- 172.21.15.28:0/897708564 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f90e0040070 con 0x7f90f006dbc0 2024-02-21T23:06:35.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.391+0000 7f90efe00640 1 -- 172.21.15.28:0/897708564 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f90f006d9a0 con 0x7f90f006dbc0 2024-02-21T23:06:35.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.393+0000 7f90e6a00640 1 -- 172.21.15.28:0/897708564 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) v1 ==== 93+0+10 (secure 0 0 0) 0x7f90e007aed0 con 0x7f90f006dbc0 2024-02-21T23:06:35.381 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-02-21T23:06:35.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.395+0000 7f90efe00640 1 -- 172.21.15.28:0/897708564 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f90cc06b4d0 msgr2=0x7f90cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:35.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.395+0000 7f90efe00640 1 --2- 172.21.15.28:0/897708564 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f90cc06b4d0 0x7f90cc06d990 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f90f0156120 tx=0x7f90d8009040 comp rx=0 tx=0).stop 2024-02-21T23:06:35.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.395+0000 7f90efe00640 1 -- 172.21.15.28:0/897708564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90f006dbc0 msgr2=0x7f90f01511c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:35.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.395+0000 7f90efe00640 1 --2- 172.21.15.28:0/897708564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90f006dbc0 0x7f90f01511c0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f90e0009890 tx=0x7f90e00046c0 comp rx=0 tx=0).stop 2024-02-21T23:06:35.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.395+0000 7f90efe00640 1 -- 172.21.15.28:0/897708564 shutdown_connections 2024-02-21T23:06:35.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.396+0000 7f90efe00640 1 --2- 172.21.15.28:0/897708564 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f90cc06b4d0 0x7f90cc06d990 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:35.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.396+0000 7f90efe00640 1 --2- 172.21.15.28:0/897708564 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f90f00c76f0 0x7f90f014ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:35.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.396+0000 7f90efe00640 1 --2- 172.21.15.28:0/897708564 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f90f013e0c0 0x7f90f014f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:35.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.396+0000 7f90efe00640 1 --2- 172.21.15.28:0/897708564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90f006dbc0 0x7f90f01511c0 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:35.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.396+0000 7f90efe00640 1 -- 172.21.15.28:0/897708564 >> 172.21.15.28:0/897708564 conn(0x7f90f0073110 msgr2=0x7f90f00c5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:35.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.396+0000 7f90efe00640 1 -- 172.21.15.28:0/897708564 shutdown_connections 2024-02-21T23:06:35.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.396+0000 7f90efe00640 1 -- 172.21.15.28:0/897708564 wait complete. 2024-02-21T23:06:35.402 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs2_metadata pg_num 2024-02-21T23:06:35.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.638+0000 7efebf200640 1 Processor -- start 2024-02-21T23:06:35.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.639+0000 7efebf200640 1 -- start start 2024-02-21T23:06:35.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.640+0000 7efebf200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efec013e0c0 0x7efec00c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:35.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.640+0000 7efebf200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7efec00c76f0 0x7efec006c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:35.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.640+0000 7efebf200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efec0158550 con 0x7efec006dbc0 2024-02-21T23:06:35.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.640+0000 7efebf200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efec00c89a0 con 0x7efec00c76f0 2024-02-21T23:06:35.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.640+0000 7efebf200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efec01587d0 con 0x7efec013e0c0 2024-02-21T23:06:35.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.641+0000 7efebd400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7efec006dbc0 0x7efec006c280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59490/0 (socket says 172.21.15.28:59490) 2024-02-21T23:06:35.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.641+0000 7efebd400640 1 -- 172.21.15.28:0/826428721 learned_addr learned my addr 172.21.15.28:0/826428721 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:06:35.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.641+0000 7efebde00640 1 --2- 172.21.15.28:0/826428721 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efec013e0c0 0x7efec00c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:35.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.641+0000 7efebe800640 1 --2- 172.21.15.28:0/826428721 >> v2:172.21.15.81:3300/0 conn(0x7efec00c76f0 0x7efec006c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:35.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.641+0000 7efebca00640 1 -- 172.21.15.28:0/826428721 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2461728840 0 0) 0x7efea8001490 con 0x7efec006dbc0 2024-02-21T23:06:35.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.641+0000 7efebca00640 1 -- 172.21.15.28:0/826428721 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efea0003520 con 0x7efec006dbc0 2024-02-21T23:06:35.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.641+0000 7efebca00640 1 -- 172.21.15.28:0/826428721 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1657800936 0 0) 0x7efea0003520 con 0x7efec006dbc0 2024-02-21T23:06:35.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.641+0000 7efebca00640 1 -- 172.21.15.28:0/826428721 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efec013e0c0 msgr2=0x7efec00c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:35.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.641+0000 7efebca00640 1 --2- 172.21.15.28:0/826428721 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efec013e0c0 0x7efec00c71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:35.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.642+0000 7efebca00640 1 -- 172.21.15.28:0/826428721 >> v2:172.21.15.81:3300/0 conn(0x7efec00c76f0 msgr2=0x7efec006c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:35.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.642+0000 7efebca00640 1 --2- 172.21.15.28:0/826428721 >> v2:172.21.15.81:3300/0 conn(0x7efec00c76f0 0x7efec006c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:35.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.642+0000 7efebca00640 1 -- 172.21.15.28:0/826428721 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efec0158ab0 con 0x7efec006dbc0 2024-02-21T23:06:35.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.642+0000 7efebca00640 1 -- 172.21.15.28:0/826428721 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7efea8001bc0 con 0x7efec006dbc0 2024-02-21T23:06:35.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.642+0000 7efebca00640 1 -- 172.21.15.28:0/826428721 >> v1:172.21.15.28:6789/0 conn(0x7efec006dbc0 legacy=0x7efec006c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:35.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.642+0000 7efebca00640 1 --2- 172.21.15.28:0/826428721 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efea00039b0 0x7efea0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:35.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.642+0000 7efebca00640 1 --2- 172.21.15.28:0/826428721 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efea0004800 0x7efea0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:35.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.642+0000 7efebca00640 1 --2- 172.21.15.28:0/826428721 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efec013e0c0 0x7efea00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:35.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.642+0000 7efebca00640 1 -- 172.21.15.28:0/826428721 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efec01587d0 con 0x7efea00039b0 2024-02-21T23:06:35.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.642+0000 7efebca00640 1 -- 172.21.15.28:0/826428721 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efea000b9c0 con 0x7efea0004800 2024-02-21T23:06:35.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.642+0000 7efebca00640 1 -- 172.21.15.28:0/826428721 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efea000bb40 con 0x7efec013e0c0 2024-02-21T23:06:35.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.642+0000 7efebde00640 1 --2- 172.21.15.28:0/826428721 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efea00039b0 0x7efea0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:35.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.642+0000 7efebe800640 1 --2- 172.21.15.28:0/826428721 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efec013e0c0 0x7efea00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:35.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.642+0000 7efebd400640 1 --2- 172.21.15.28:0/826428721 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efea0004800 0x7efea0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:35.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.642+0000 7efebde00640 1 -- 172.21.15.28:0/826428721 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efec013e0c0 msgr2=0x7efea00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:35.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.642+0000 7efebde00640 1 --2- 172.21.15.28:0/826428721 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efec013e0c0 0x7efea00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:35.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.642+0000 7efebde00640 1 -- 172.21.15.28:0/826428721 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efea0004800 msgr2=0x7efea0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:35.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.643+0000 7efebde00640 1 --2- 172.21.15.28:0/826428721 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efea0004800 0x7efea0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:35.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.643+0000 7efebde00640 1 -- 172.21.15.28:0/826428721 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efea0003520 con 0x7efea00039b0 2024-02-21T23:06:35.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.644+0000 7efebde00640 1 --2- 172.21.15.28:0/826428721 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efea00039b0 0x7efea0003dd0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7efeb0009af0 tx=0x7efea0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a684b4cc8ae3a09c server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:35.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.644+0000 7efebca00640 1 -- 172.21.15.28:0/826428721 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb0039070 con 0x7efea00039b0 2024-02-21T23:06:35.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.644+0000 7efebca00640 1 -- 172.21.15.28:0/826428721 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efeb002c6b0 con 0x7efea00039b0 2024-02-21T23:06:35.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.644+0000 7efebca00640 1 -- 172.21.15.28:0/826428721 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb0009690 con 0x7efea00039b0 2024-02-21T23:06:35.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.644+0000 7efebf200640 1 -- 172.21.15.28:0/826428721 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efea00039b0 msgr2=0x7efea0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:35.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.644+0000 7efebf200640 1 --2- 172.21.15.28:0/826428721 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efea00039b0 0x7efea0003dd0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7efeb0009af0 tx=0x7efea0005830 comp rx=0 tx=0).stop 2024-02-21T23:06:35.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.644+0000 7efebe800640 1 -- 172.21.15.28:0/826428721 reap_dead start 2024-02-21T23:06:35.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.645+0000 7efebf200640 1 -- 172.21.15.28:0/826428721 shutdown_connections 2024-02-21T23:06:35.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.645+0000 7efebf200640 1 -- 172.21.15.28:0/826428721 >> 172.21.15.28:0/826428721 conn(0x7efec0073110 msgr2=0x7efec00ac490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:35.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.645+0000 7efebf200640 1 -- 172.21.15.28:0/826428721 shutdown_connections 2024-02-21T23:06:35.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.645+0000 7efebf200640 1 -- 172.21.15.28:0/826428721 wait complete. 2024-02-21T23:06:35.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.646+0000 7efebf200640 1 Processor -- start 2024-02-21T23:06:35.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.647+0000 7efebf200640 1 -- start start 2024-02-21T23:06:35.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.647+0000 7efebf200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efea00039b0 0x7efec0151f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:35.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.648+0000 7efebf200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efec00abdd0 0x7efec0138590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:35.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.648+0000 7efebf200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efec01eec90 0x7efec01578c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:35.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.648+0000 7efebf200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efec006ced0 con 0x7efea00039b0 2024-02-21T23:06:35.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.648+0000 7efebf200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efec0159070 con 0x7efec00abdd0 2024-02-21T23:06:35.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.648+0000 7efebf200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efec013e520 con 0x7efec01eec90 2024-02-21T23:06:35.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.648+0000 7efebde00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efea00039b0 0x7efec0151f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:35.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.648+0000 7efebde00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efea00039b0 0x7efec0151f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59202/0 (socket says 172.21.15.28:59202) 2024-02-21T23:06:35.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.648+0000 7efebde00640 1 -- 172.21.15.28:0/2581245986 learned_addr learned my addr 172.21.15.28:0/2581245986 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:35.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.648+0000 7efebd400640 1 --2- 172.21.15.28:0/2581245986 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efec00abdd0 0x7efec0138590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:35.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.648+0000 7efebe800640 1 --2- 172.21.15.28:0/2581245986 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efec01eec90 0x7efec01578c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:35.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.648+0000 7efebde00640 1 -- 172.21.15.28:0/2581245986 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efec01eec90 msgr2=0x7efec01578c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:35.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.648+0000 7efebde00640 1 --2- 172.21.15.28:0/2581245986 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efec01eec90 0x7efec01578c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:35.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.648+0000 7efebde00640 1 -- 172.21.15.28:0/2581245986 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efec00abdd0 msgr2=0x7efec0138590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:35.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.648+0000 7efebde00640 1 --2- 172.21.15.28:0/2581245986 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efec00abdd0 0x7efec0138590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:35.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.648+0000 7efebde00640 1 -- 172.21.15.28:0/2581245986 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efec00ac2d0 con 0x7efea00039b0 2024-02-21T23:06:35.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.649+0000 7efebde00640 1 --2- 172.21.15.28:0/2581245986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efea00039b0 0x7efec0151f60 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7efeb0009af0 tx=0x7efeb0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:35.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.649+0000 7efeb6000640 1 -- 172.21.15.28:0/2581245986 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb0042020 con 0x7efea00039b0 2024-02-21T23:06:35.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.649+0000 7efeb6000640 1 -- 172.21.15.28:0/2581245986 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efeb003a040 con 0x7efea00039b0 2024-02-21T23:06:35.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.649+0000 7efebf200640 1 -- 172.21.15.28:0/2581245986 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efec01ee1b0 con 0x7efea00039b0 2024-02-21T23:06:35.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.649+0000 7efeb6000640 1 -- 172.21.15.28:0/2581245986 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb0039070 con 0x7efea00039b0 2024-02-21T23:06:35.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.649+0000 7efebf200640 1 -- 172.21.15.28:0/2581245986 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efec0138fe0 con 0x7efea00039b0 2024-02-21T23:06:35.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.650+0000 7efebf200640 1 -- 172.21.15.28:0/2581245986 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efec004f290 con 0x7efea00039b0 2024-02-21T23:06:35.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.652+0000 7efeb6000640 1 -- 172.21.15.28:0/2581245986 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efeb0045020 con 0x7efea00039b0 2024-02-21T23:06:35.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.653+0000 7efeb6000640 1 --2- 172.21.15.28:0/2581245986 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7efe9c06b4d0 0x7efe9c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:35.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.653+0000 7efeb6000640 1 -- 172.21.15.28:0/2581245986 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6479+0+0 (secure 0 0 0) 0x7efeb00ab6e0 con 0x7efea00039b0 2024-02-21T23:06:35.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.653+0000 7efebd400640 1 --2- 172.21.15.28:0/2581245986 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7efe9c06b4d0 0x7efe9c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:35.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.655+0000 7efebd400640 1 --2- 172.21.15.28:0/2581245986 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7efe9c06b4d0 0x7efe9c06d990 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7efec00c5a90 tx=0x7efea8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:35.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.658+0000 7efeb6000640 1 -- 172.21.15.28:0/2581245986 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7efeb004c020 con 0x7efea00039b0 2024-02-21T23:06:35.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.857+0000 7efebf200640 1 -- 172.21.15.28:0/2581245986 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs2_metadata", "var": "pg_num"} v 0) v1 -- 0x7efec006d9a0 con 0x7efea00039b0 2024-02-21T23:06:36.171 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-21T23:06:36.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.858+0000 7efeb6000640 1 -- 172.21.15.28:0/2581245986 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs2_metadata", "var": "pg_num"}]=0 v87) v1 ==== 105+0+11 (secure 0 0 0) 0x7efeb007a250 con 0x7efea00039b0 2024-02-21T23:06:36.172 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-02-21T23:06:36.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.860+0000 7efebf200640 1 -- 172.21.15.28:0/2581245986 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7efe9c06b4d0 msgr2=0x7efe9c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:36.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.860+0000 7efebf200640 1 --2- 172.21.15.28:0/2581245986 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7efe9c06b4d0 0x7efe9c06d990 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7efec00c5a90 tx=0x7efea8009040 comp rx=0 tx=0).stop 2024-02-21T23:06:36.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.860+0000 7efebf200640 1 -- 172.21.15.28:0/2581245986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efea00039b0 msgr2=0x7efec0151f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:36.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.860+0000 7efebf200640 1 --2- 172.21.15.28:0/2581245986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efea00039b0 0x7efec0151f60 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7efeb0009af0 tx=0x7efeb0004730 comp rx=0 tx=0).stop 2024-02-21T23:06:36.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.860+0000 7efebf200640 1 -- 172.21.15.28:0/2581245986 shutdown_connections 2024-02-21T23:06:36.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.860+0000 7efebf200640 1 --2- 172.21.15.28:0/2581245986 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7efe9c06b4d0 0x7efe9c06d990 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:36.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.860+0000 7efebf200640 1 --2- 172.21.15.28:0/2581245986 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efec01eec90 0x7efec01578c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:36.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.860+0000 7efebf200640 1 --2- 172.21.15.28:0/2581245986 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efec00abdd0 0x7efec0138590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:36.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.860+0000 7efebf200640 1 --2- 172.21.15.28:0/2581245986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efea00039b0 0x7efec0151f60 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:36.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.860+0000 7efebf200640 1 -- 172.21.15.28:0/2581245986 >> 172.21.15.28:0/2581245986 conn(0x7efec0073110 msgr2=0x7efec0139e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:36.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.860+0000 7efebf200640 1 -- 172.21.15.28:0/2581245986 shutdown_connections 2024-02-21T23:06:36.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:35.860+0000 7efebf200640 1 -- 172.21.15.28:0/2581245986 wait complete. 2024-02-21T23:06:36.175 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs2_data pg_num 2024-02-21T23:06:36.221 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-21T23:06:36.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.433+0000 7ff643200640 1 Processor -- start 2024-02-21T23:06:36.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.433+0000 7ff643200640 1 -- start start 2024-02-21T23:06:36.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.435+0000 7ff643200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff64406dbc0 0x7ff64406c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:36.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.435+0000 7ff643200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff64413e520 0x7ff64406c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:36.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.435+0000 7ff643200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff644158550 con 0x7ff64413e0c0 2024-02-21T23:06:36.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.435+0000 7ff643200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6440c89a0 con 0x7ff64413e520 2024-02-21T23:06:36.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.435+0000 7ff643200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6441587d0 con 0x7ff64406dbc0 2024-02-21T23:06:36.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.435+0000 7ff641e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff64413e0c0 0x7ff6440c71b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59492/0 (socket says 172.21.15.28:59492) 2024-02-21T23:06:36.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.435+0000 7ff641e00640 1 -- 172.21.15.28:0/2654571365 learned_addr learned my addr 172.21.15.28:0/2654571365 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:06:36.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.435+0000 7ff642800640 1 --2- 172.21.15.28:0/2654571365 >> v2:172.21.15.28:3300/0 conn(0x7ff64413e520 0x7ff64406c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:36.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.436+0000 7ff639400640 1 --2- 172.21.15.28:0/2654571365 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff64406dbc0 0x7ff64406c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:36.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.436+0000 7ff641400640 1 -- 172.21.15.28:0/2654571365 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2135932160 0 0) 0x7ff634001110 con 0x7ff64413e0c0 2024-02-21T23:06:36.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.436+0000 7ff641400640 1 -- 172.21.15.28:0/2654571365 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff628003520 con 0x7ff64413e0c0 2024-02-21T23:06:36.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.436+0000 7ff642800640 1 -- 172.21.15.28:0/2654571365 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff64406dbc0 msgr2=0x7ff64406c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:36.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.436+0000 7ff642800640 1 --2- 172.21.15.28:0/2654571365 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff64406dbc0 0x7ff64406c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:36.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.436+0000 7ff642800640 1 -- 172.21.15.28:0/2654571365 >> v1:172.21.15.28:6789/0 conn(0x7ff64413e0c0 legacy=0x7ff6440c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:36.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.436+0000 7ff642800640 1 -- 172.21.15.28:0/2654571365 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff644158ab0 con 0x7ff64413e520 2024-02-21T23:06:36.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.436+0000 7ff639400640 1 --2- 172.21.15.28:0/2654571365 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff64406dbc0 0x7ff64406c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:06:36.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.437+0000 7ff642800640 1 --2- 172.21.15.28:0/2654571365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff64413e520 0x7ff64406c7c0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7ff63c009d90 tx=0x7ff63c02be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1eeec5a1205134f server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:36.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.437+0000 7ff641400640 1 -- 172.21.15.28:0/2654571365 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff63c039070 con 0x7ff64413e520 2024-02-21T23:06:36.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.437+0000 7ff641400640 1 -- 172.21.15.28:0/2654571365 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff63c02cbc0 con 0x7ff64413e520 2024-02-21T23:06:36.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.437+0000 7ff641400640 1 -- 172.21.15.28:0/2654571365 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff63c034440 con 0x7ff64413e520 2024-02-21T23:06:36.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.438+0000 7ff643200640 1 -- 172.21.15.28:0/2654571365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff64413e520 msgr2=0x7ff64406c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:36.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.438+0000 7ff643200640 1 --2- 172.21.15.28:0/2654571365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff64413e520 0x7ff64406c7c0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7ff63c009d90 tx=0x7ff63c02be10 comp rx=0 tx=0).stop 2024-02-21T23:06:36.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.438+0000 7ff643200640 1 -- 172.21.15.28:0/2654571365 shutdown_connections 2024-02-21T23:06:36.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.438+0000 7ff643200640 1 --2- 172.21.15.28:0/2654571365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff64413e520 0x7ff64406c7c0 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:36.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.438+0000 7ff643200640 1 --2- 172.21.15.28:0/2654571365 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff64406dbc0 0x7ff64406c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:36.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.438+0000 7ff643200640 1 -- 172.21.15.28:0/2654571365 >> 172.21.15.28:0/2654571365 conn(0x7ff644073110 msgr2=0x7ff6440ac490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:36.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.438+0000 7ff643200640 1 -- 172.21.15.28:0/2654571365 shutdown_connections 2024-02-21T23:06:36.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.438+0000 7ff643200640 1 -- 172.21.15.28:0/2654571365 wait complete. 2024-02-21T23:06:36.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.440+0000 7ff643200640 1 Processor -- start 2024-02-21T23:06:36.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.441+0000 7ff643200640 1 -- start start 2024-02-21T23:06:36.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.442+0000 7ff643200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff64406dbc0 0x7ff644155b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:36.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.442+0000 7ff643200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff64413e0c0 0x7ff644151ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:36.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.442+0000 7ff643200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff64413e520 0x7ff644139a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:36.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.442+0000 7ff643200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6440c78c0 con 0x7ff64406dbc0 2024-02-21T23:06:36.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.442+0000 7ff643200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff644159070 con 0x7ff64413e0c0 2024-02-21T23:06:36.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.442+0000 7ff643200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6440c7a40 con 0x7ff64413e520 2024-02-21T23:06:36.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.442+0000 7ff641e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff64406dbc0 0x7ff644155b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:36.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.442+0000 7ff641e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff64406dbc0 0x7ff644155b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59218/0 (socket says 172.21.15.28:59218) 2024-02-21T23:06:36.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.442+0000 7ff641e00640 1 -- 172.21.15.28:0/443357283 learned_addr learned my addr 172.21.15.28:0/443357283 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:36.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.442+0000 7ff641e00640 1 -- 172.21.15.28:0/443357283 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff64413e520 msgr2=0x7ff644139a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:36.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.442+0000 7ff641e00640 1 --2- 172.21.15.28:0/443357283 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff64413e520 0x7ff644139a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:36.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.442+0000 7ff641e00640 1 -- 172.21.15.28:0/443357283 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff64413e0c0 msgr2=0x7ff644151ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:36.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.442+0000 7ff641e00640 1 --2- 172.21.15.28:0/443357283 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff64413e0c0 0x7ff644151ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:36.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.442+0000 7ff641e00640 1 -- 172.21.15.28:0/443357283 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6441edf30 con 0x7ff64406dbc0 2024-02-21T23:06:36.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.443+0000 7ff641e00640 1 --2- 172.21.15.28:0/443357283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff64406dbc0 0x7ff644155b40 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7ff63400c3a0 tx=0x7ff63400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:36.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.443+0000 7ff63aa00640 1 -- 172.21.15.28:0/443357283 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff634017070 con 0x7ff64406dbc0 2024-02-21T23:06:36.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.443+0000 7ff63aa00640 1 -- 172.21.15.28:0/443357283 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff634005ce0 con 0x7ff64406dbc0 2024-02-21T23:06:36.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.443+0000 7ff63aa00640 1 -- 172.21.15.28:0/443357283 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff634007310 con 0x7ff64406dbc0 2024-02-21T23:06:36.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.443+0000 7ff643200640 1 -- 172.21.15.28:0/443357283 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6441ee210 con 0x7ff64406dbc0 2024-02-21T23:06:36.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.443+0000 7ff643200640 1 -- 172.21.15.28:0/443357283 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6441ee7d0 con 0x7ff64406dbc0 2024-02-21T23:06:36.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.445+0000 7ff643200640 1 -- 172.21.15.28:0/443357283 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff64404f290 con 0x7ff64406dbc0 2024-02-21T23:06:36.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.445+0000 7ff63aa00640 1 -- 172.21.15.28:0/443357283 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff63401a070 con 0x7ff64406dbc0 2024-02-21T23:06:36.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.446+0000 7ff63aa00640 1 --2- 172.21.15.28:0/443357283 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff62006b4d0 0x7ff62006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:36.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.446+0000 7ff63aa00640 1 -- 172.21.15.28:0/443357283 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6479+0+0 (secure 0 0 0) 0x7ff6340898d0 con 0x7ff64406dbc0 2024-02-21T23:06:36.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.447+0000 7ff639400640 1 --2- 172.21.15.28:0/443357283 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff62006b4d0 0x7ff62006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:36.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.449+0000 7ff639400640 1 --2- 172.21.15.28:0/443357283 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff62006b4d0 0x7ff62006d990 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7ff62c006fd0 tx=0x7ff62c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:36.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.451+0000 7ff63aa00640 1 -- 172.21.15.28:0/443357283 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ff644158550 con 0x7ff64406dbc0 2024-02-21T23:06:36.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.652+0000 7ff643200640 1 -- 172.21.15.28:0/443357283 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs2_data", "var": "pg_num"} v 0) v1 -- 0x7ff64406d9a0 con 0x7ff64406dbc0 2024-02-21T23:06:36.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.653+0000 7ff63aa00640 1 -- 172.21.15.28:0/443357283 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs2_data", "var": "pg_num"}]=0 v87) v1 ==== 101+0+11 (secure 0 0 0) 0x7ff6340584e0 con 0x7ff64406dbc0 2024-02-21T23:06:36.642 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-02-21T23:06:36.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.656+0000 7ff643200640 1 -- 172.21.15.28:0/443357283 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff62006b4d0 msgr2=0x7ff62006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:36.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.656+0000 7ff643200640 1 --2- 172.21.15.28:0/443357283 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff62006b4d0 0x7ff62006d990 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7ff62c006fd0 tx=0x7ff62c008040 comp rx=0 tx=0).stop 2024-02-21T23:06:36.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.656+0000 7ff643200640 1 -- 172.21.15.28:0/443357283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff64406dbc0 msgr2=0x7ff644155b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:36.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.656+0000 7ff643200640 1 --2- 172.21.15.28:0/443357283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff64406dbc0 0x7ff644155b40 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7ff63400c3a0 tx=0x7ff63400c860 comp rx=0 tx=0).stop 2024-02-21T23:06:36.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.656+0000 7ff643200640 1 -- 172.21.15.28:0/443357283 shutdown_connections 2024-02-21T23:06:36.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.656+0000 7ff643200640 1 --2- 172.21.15.28:0/443357283 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff62006b4d0 0x7ff62006d990 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:36.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.656+0000 7ff643200640 1 --2- 172.21.15.28:0/443357283 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff64413e520 0x7ff644139a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:36.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.656+0000 7ff643200640 1 --2- 172.21.15.28:0/443357283 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff64413e0c0 0x7ff644151ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:36.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.656+0000 7ff643200640 1 --2- 172.21.15.28:0/443357283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff64406dbc0 0x7ff644155b40 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:36.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.656+0000 7ff643200640 1 -- 172.21.15.28:0/443357283 >> 172.21.15.28:0/443357283 conn(0x7ff644073110 msgr2=0x7ff6440c9000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:36.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.657+0000 7ff643200640 1 -- 172.21.15.28:0/443357283 shutdown_connections 2024-02-21T23:06:36.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.657+0000 7ff643200640 1 -- 172.21.15.28:0/443357283 wait complete. 2024-02-21T23:06:36.664 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:06:36.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.898+0000 7ff312a00640 1 Processor -- start 2024-02-21T23:06:36.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.898+0000 7ff312a00640 1 -- start start 2024-02-21T23:06:36.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.900+0000 7ff312a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff30c0c11f0 0x7ff30c153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:36.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.900+0000 7ff312a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff30c0c15d0 0x7ff30c06dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:36.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.900+0000 7ff312a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff30c0c1a30 con 0x7ff30c156350 2024-02-21T23:06:36.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.900+0000 7ff312a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff30c0a5ec0 con 0x7ff30c0c15d0 2024-02-21T23:06:36.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.900+0000 7ff312a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff30c152e70 con 0x7ff30c0c11f0 2024-02-21T23:06:36.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.900+0000 7ff30b400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff30c0c15d0 0x7ff30c06dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:36.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.900+0000 7ff310a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff30c156350 0x7ff30c1567a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59496/0 (socket says 172.21.15.28:59496) 2024-02-21T23:06:36.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.900+0000 7ff310a00640 1 -- 172.21.15.28:0/3376230409 learned_addr learned my addr 172.21.15.28:0/3376230409 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:06:36.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.900+0000 7ff30be00640 1 --2- 172.21.15.28:0/3376230409 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff30c0c11f0 0x7ff30c153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:36.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.901+0000 7ff30b400640 1 -- 172.21.15.28:0/3376230409 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff30c0c11f0 msgr2=0x7ff30c153950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:36.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.901+0000 7ff30b400640 1 --2- 172.21.15.28:0/3376230409 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff30c0c11f0 0x7ff30c153950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:36.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.901+0000 7ff30b400640 1 -- 172.21.15.28:0/3376230409 >> v1:172.21.15.28:6789/0 conn(0x7ff30c156350 legacy=0x7ff30c1567a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:36.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.901+0000 7ff30aa00640 1 -- 172.21.15.28:0/3376230409 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1283739697 0 0) 0x7ff30c0c1a30 con 0x7ff30c156350 2024-02-21T23:06:36.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.901+0000 7ff30b400640 1 -- 172.21.15.28:0/3376230409 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff30c153150 con 0x7ff30c0c15d0 2024-02-21T23:06:36.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.902+0000 7ff30b400640 1 --2- 172.21.15.28:0/3376230409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff30c0c15d0 0x7ff30c06dce0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7ff2f80097a0 tx=0x7ff2f802b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ea007dd3e48ff98 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:36.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.902+0000 7ff30aa00640 1 -- 172.21.15.28:0/3376230409 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2f8038070 con 0x7ff30c0c15d0 2024-02-21T23:06:36.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.902+0000 7ff30aa00640 1 -- 172.21.15.28:0/3376230409 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2f8031700 con 0x7ff30c0c15d0 2024-02-21T23:06:36.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.902+0000 7ff30aa00640 1 -- 172.21.15.28:0/3376230409 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2f8031ad0 con 0x7ff30c0c15d0 2024-02-21T23:06:36.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.903+0000 7ff312a00640 1 -- 172.21.15.28:0/3376230409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff30c0c15d0 msgr2=0x7ff30c06dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:36.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.903+0000 7ff312a00640 1 --2- 172.21.15.28:0/3376230409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff30c0c15d0 0x7ff30c06dce0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7ff2f80097a0 tx=0x7ff2f802b7a0 comp rx=0 tx=0).stop 2024-02-21T23:06:36.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.903+0000 7ff312a00640 1 -- 172.21.15.28:0/3376230409 shutdown_connections 2024-02-21T23:06:36.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.903+0000 7ff312a00640 1 --2- 172.21.15.28:0/3376230409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff30c0c15d0 0x7ff30c06dce0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:36.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.903+0000 7ff312a00640 1 --2- 172.21.15.28:0/3376230409 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff30c0c11f0 0x7ff30c153950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:36.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.903+0000 7ff312a00640 1 -- 172.21.15.28:0/3376230409 >> 172.21.15.28:0/3376230409 conn(0x7ff30c0c67f0 msgr2=0x7ff30c0c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:36.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.903+0000 7ff312a00640 1 -- 172.21.15.28:0/3376230409 shutdown_connections 2024-02-21T23:06:36.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.903+0000 7ff312a00640 1 -- 172.21.15.28:0/3376230409 wait complete. 2024-02-21T23:06:36.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.905+0000 7ff312a00640 1 Processor -- start 2024-02-21T23:06:36.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.905+0000 7ff312a00640 1 -- start start 2024-02-21T23:06:36.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.906+0000 7ff312a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff30c0c11f0 0x7ff30c1e8dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:36.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.906+0000 7ff312a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff30c0c15d0 0x7ff30c1de9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:36.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.906+0000 7ff312a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff30c156350 0x7ff30c143ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:36.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.906+0000 7ff312a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff30c0c4c00 con 0x7ff30c0c11f0 2024-02-21T23:06:36.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.906+0000 7ff312a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff30c14d1c0 con 0x7ff30c156350 2024-02-21T23:06:36.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.906+0000 7ff312a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff30c0a6110 con 0x7ff30c0c15d0 2024-02-21T23:06:36.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.906+0000 7ff30be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff30c0c11f0 0x7ff30c1e8dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:36.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.906+0000 7ff30be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff30c0c11f0 0x7ff30c1e8dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59232/0 (socket says 172.21.15.28:59232) 2024-02-21T23:06:36.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.906+0000 7ff30be00640 1 -- 172.21.15.28:0/1568302686 learned_addr learned my addr 172.21.15.28:0/1568302686 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:36.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.906+0000 7ff30b400640 1 --2- 172.21.15.28:0/1568302686 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff30c0c15d0 0x7ff30c1de9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:36.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.907+0000 7ff310a00640 1 --2- 172.21.15.28:0/1568302686 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff30c156350 0x7ff30c143ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:36.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.907+0000 7ff30be00640 1 -- 172.21.15.28:0/1568302686 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff30c0c15d0 msgr2=0x7ff30c1de9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:36.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.907+0000 7ff30be00640 1 --2- 172.21.15.28:0/1568302686 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff30c0c15d0 0x7ff30c1de9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:36.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.907+0000 7ff30be00640 1 -- 172.21.15.28:0/1568302686 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff30c156350 msgr2=0x7ff30c143ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:36.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.907+0000 7ff30be00640 1 --2- 172.21.15.28:0/1568302686 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff30c156350 0x7ff30c143ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:36.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.907+0000 7ff30be00640 1 -- 172.21.15.28:0/1568302686 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff30c1ec7d0 con 0x7ff30c0c11f0 2024-02-21T23:06:36.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.907+0000 7ff30be00640 1 --2- 172.21.15.28:0/1568302686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff30c0c11f0 0x7ff30c1e8dd0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7ff2fc007e70 tx=0x7ff2fc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:36.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.907+0000 7ff303e00640 1 -- 172.21.15.28:0/1568302686 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2fc014620 con 0x7ff30c0c11f0 2024-02-21T23:06:36.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.907+0000 7ff303e00640 1 -- 172.21.15.28:0/1568302686 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2fc019070 con 0x7ff30c0c11f0 2024-02-21T23:06:36.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.907+0000 7ff303e00640 1 -- 172.21.15.28:0/1568302686 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2fc014a00 con 0x7ff30c0c11f0 2024-02-21T23:06:36.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.907+0000 7ff312a00640 1 -- 172.21.15.28:0/1568302686 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff30c1ecab0 con 0x7ff30c0c11f0 2024-02-21T23:06:36.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.908+0000 7ff312a00640 1 -- 172.21.15.28:0/1568302686 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff30c1ed070 con 0x7ff30c0c11f0 2024-02-21T23:06:36.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.909+0000 7ff312a00640 1 -- 172.21.15.28:0/1568302686 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff30c067f30 con 0x7ff30c0c11f0 2024-02-21T23:06:36.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.910+0000 7ff303e00640 1 -- 172.21.15.28:0/1568302686 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff2fc014ba0 con 0x7ff30c0c11f0 2024-02-21T23:06:36.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.911+0000 7ff303e00640 1 --2- 172.21.15.28:0/1568302686 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff2ec06b4d0 0x7ff2ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:36.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.911+0000 7ff303e00640 1 -- 172.21.15.28:0/1568302686 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6479+0+0 (secure 0 0 0) 0x7ff2fc08ad80 con 0x7ff30c0c11f0 2024-02-21T23:06:36.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.911+0000 7ff30b400640 1 --2- 172.21.15.28:0/1568302686 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff2ec06b4d0 0x7ff2ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:36.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.912+0000 7ff30b400640 1 --2- 172.21.15.28:0/1568302686 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff2ec06b4d0 0x7ff2ec06d990 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7ff2f8035000 tx=0x7ff2f8008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:36.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:36.915+0000 7ff303e00640 1 -- 172.21.15.28:0/1568302686 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ff2fc0598f0 con 0x7ff30c0c11f0 2024-02-21T23:06:37.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.150+0000 7ff312a00640 1 -- 172.21.15.28:0/1568302686 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff30c154bc0 con 0x7ff30c0c11f0 2024-02-21T23:06:37.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.152+0000 7ff303e00640 1 -- 172.21.15.28:0/1568302686 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+4941 (secure 0 0 0) 0x7ff2fc0591f0 con 0x7ff30c0c11f0 2024-02-21T23:06:37.140 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:06:37.141 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":79,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":8,"addr":"172.21.15.81:6837/4292940632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":4292940632},{"type":"v1","addr":"172.21.15.81:6837","nonce":4292940632}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67},{"gid":6831,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2340244964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2340244964},{"type":"v1","addr":"172.21.15.28:6835","nonce":2340244964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":76},{"gid":6846,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/2977800995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2977800995},{"type":"v1","addr":"172.21.15.28:6833","nonce":2977800995}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:06:10.484421+0000","modified":"2024-02-21T23:06:34.512209+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5288},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5288":{"gid":5288,"name":"d","rank":0,"incarnation":76,"state":"up:active","state_seq":15,"addr":"172.21.15.81:6835/2774951107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2774951107},{"type":"v1","addr":"172.21.15.81:6835","nonce":2774951107}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-02-21T23:06:37.141 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 79 2024-02-21T23:06:37.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.154+0000 7ff312a00640 1 -- 172.21.15.28:0/1568302686 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff2ec06b4d0 msgr2=0x7ff2ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:37.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.154+0000 7ff312a00640 1 --2- 172.21.15.28:0/1568302686 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff2ec06b4d0 0x7ff2ec06d990 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7ff2f8035000 tx=0x7ff2f8008030 comp rx=0 tx=0).stop 2024-02-21T23:06:37.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.154+0000 7ff312a00640 1 -- 172.21.15.28:0/1568302686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff30c0c11f0 msgr2=0x7ff30c1e8dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:37.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.154+0000 7ff312a00640 1 --2- 172.21.15.28:0/1568302686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff30c0c11f0 0x7ff30c1e8dd0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7ff2fc007e70 tx=0x7ff2fc00e4b0 comp rx=0 tx=0).stop 2024-02-21T23:06:37.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.154+0000 7ff312a00640 1 -- 172.21.15.28:0/1568302686 shutdown_connections 2024-02-21T23:06:37.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.154+0000 7ff312a00640 1 --2- 172.21.15.28:0/1568302686 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff2ec06b4d0 0x7ff2ec06d990 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:37.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.154+0000 7ff312a00640 1 --2- 172.21.15.28:0/1568302686 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff30c0c15d0 0x7ff30c1de9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:37.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.154+0000 7ff312a00640 1 --2- 172.21.15.28:0/1568302686 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff30c156350 0x7ff30c143ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:37.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.154+0000 7ff312a00640 1 --2- 172.21.15.28:0/1568302686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff30c0c11f0 0x7ff30c1e8dd0 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:37.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.154+0000 7ff312a00640 1 -- 172.21.15.28:0/1568302686 >> 172.21.15.28:0/1568302686 conn(0x7ff30c0c67f0 msgr2=0x7ff30c0c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:37.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.155+0000 7ff312a00640 1 -- 172.21.15.28:0/1568302686 shutdown_connections 2024-02-21T23:06:37.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.155+0000 7ff312a00640 1 -- 172.21.15.28:0/1568302686 wait complete. 2024-02-21T23:06:37.160 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:06:37.326 INFO:tasks.ceph.mds.d.smithi081.stderr:2024-02-21T23:06:37.337+0000 7f56fca00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-21T23:06:37.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.390+0000 7f901e400640 1 Processor -- start 2024-02-21T23:06:37.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.390+0000 7f901e400640 1 -- start start 2024-02-21T23:06:37.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.391+0000 7f901e400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9018153950 0x7f90180c15d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:37.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.391+0000 7f901e400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f901815b6d0 0x7f901815dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:37.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.391+0000 7f901e400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9018152e70 con 0x7f90180c11f0 2024-02-21T23:06:37.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.391+0000 7f901e400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90180a5ec0 con 0x7f9018153950 2024-02-21T23:06:37.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.391+0000 7f901e400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9018153150 con 0x7f901815b6d0 2024-02-21T23:06:37.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.391+0000 7f900fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9018153950 0x7f90180c15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:37.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.391+0000 7f9017400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f90180c11f0 0x7f901806dce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59508/0 (socket says 172.21.15.28:59508) 2024-02-21T23:06:37.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.391+0000 7f9017400640 1 -- 172.21.15.28:0/2698088292 learned_addr learned my addr 172.21.15.28:0/2698088292 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:06:37.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.392+0000 7f9017e00640 1 --2- 172.21.15.28:0/2698088292 >> v2:172.21.15.81:3300/0 conn(0x7f901815b6d0 0x7f901815dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:37.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.392+0000 7f900fe00640 1 -- 172.21.15.28:0/2698088292 >> v2:172.21.15.81:3300/0 conn(0x7f901815b6d0 msgr2=0x7f901815dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:37.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.392+0000 7f900fe00640 1 --2- 172.21.15.28:0/2698088292 >> v2:172.21.15.81:3300/0 conn(0x7f901815b6d0 0x7f901815dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:37.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.392+0000 7f900fe00640 1 -- 172.21.15.28:0/2698088292 >> v1:172.21.15.28:6789/0 conn(0x7f90180c11f0 legacy=0x7f901806dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:37.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.392+0000 7f900fe00640 1 -- 172.21.15.28:0/2698088292 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9018153430 con 0x7f9018153950 2024-02-21T23:06:37.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.392+0000 7f9016a00640 1 -- 172.21.15.28:0/2698088292 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3823055942 0 0) 0x7f9018152e70 con 0x7f90180c11f0 2024-02-21T23:06:37.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.393+0000 7f900fe00640 1 --2- 172.21.15.28:0/2698088292 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9018153950 0x7f90180c15d0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f90180a6040 tx=0x7f900002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63bf8435d6054686 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:37.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.393+0000 7f9016a00640 1 -- 172.21.15.28:0/2698088292 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90000315a0 con 0x7f9018153950 2024-02-21T23:06:37.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.393+0000 7f9016a00640 1 -- 172.21.15.28:0/2698088292 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9000037070 con 0x7f9018153950 2024-02-21T23:06:37.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.393+0000 7f9016a00640 1 -- 172.21.15.28:0/2698088292 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9000031960 con 0x7f9018153950 2024-02-21T23:06:37.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.394+0000 7f901e400640 1 -- 172.21.15.28:0/2698088292 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9018153950 msgr2=0x7f90180c15d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:37.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.394+0000 7f901e400640 1 --2- 172.21.15.28:0/2698088292 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9018153950 0x7f90180c15d0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f90180a6040 tx=0x7f900002b840 comp rx=0 tx=0).stop 2024-02-21T23:06:37.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.394+0000 7f901e400640 1 -- 172.21.15.28:0/2698088292 shutdown_connections 2024-02-21T23:06:37.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.394+0000 7f901e400640 1 --2- 172.21.15.28:0/2698088292 >> v2:172.21.15.81:3300/0 conn(0x7f901815b6d0 0x7f901815dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:37.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.394+0000 7f901e400640 1 --2- 172.21.15.28:0/2698088292 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9018153950 0x7f90180c15d0 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:37.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.394+0000 7f901e400640 1 -- 172.21.15.28:0/2698088292 >> 172.21.15.28:0/2698088292 conn(0x7f90180c67f0 msgr2=0x7f90180c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:37.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.394+0000 7f901e400640 1 -- 172.21.15.28:0/2698088292 shutdown_connections 2024-02-21T23:06:37.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.394+0000 7f901e400640 1 -- 172.21.15.28:0/2698088292 wait complete. 2024-02-21T23:06:37.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.396+0000 7f901e400640 1 Processor -- start 2024-02-21T23:06:37.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.396+0000 7f901e400640 1 -- start start 2024-02-21T23:06:37.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.397+0000 7f901e400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f90180c11f0 0x7f9018142320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:37.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.397+0000 7f901e400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9018153950 0x7f90180a7270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:37.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.397+0000 7f901e400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f901815b6d0 0x7f90180a77b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:37.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.397+0000 7f901e400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90180a6110 con 0x7f9018153950 2024-02-21T23:06:37.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.397+0000 7f901e400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90180c4c00 con 0x7f90180c11f0 2024-02-21T23:06:37.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.397+0000 7f901e400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90180c0d90 con 0x7f901815b6d0 2024-02-21T23:06:37.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.397+0000 7f900fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9018153950 0x7f90180a7270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:37.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.397+0000 7f900fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9018153950 0x7f90180a7270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59262/0 (socket says 172.21.15.28:59262) 2024-02-21T23:06:37.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.397+0000 7f900fe00640 1 -- 172.21.15.28:0/2453423440 learned_addr learned my addr 172.21.15.28:0/2453423440 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:37.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.397+0000 7f9017400640 1 --2- 172.21.15.28:0/2453423440 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f90180c11f0 0x7f9018142320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:37.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.397+0000 7f9017e00640 1 --2- 172.21.15.28:0/2453423440 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f901815b6d0 0x7f90180a77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:37.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.398+0000 7f900fe00640 1 -- 172.21.15.28:0/2453423440 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f901815b6d0 msgr2=0x7f90180a77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:37.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.398+0000 7f900fe00640 1 --2- 172.21.15.28:0/2453423440 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f901815b6d0 0x7f90180a77b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:37.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.398+0000 7f900fe00640 1 -- 172.21.15.28:0/2453423440 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f90180c11f0 msgr2=0x7f9018142320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:37.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.398+0000 7f900fe00640 1 --2- 172.21.15.28:0/2453423440 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f90180c11f0 0x7f9018142320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:37.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.398+0000 7f900fe00640 1 -- 172.21.15.28:0/2453423440 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90180a6790 con 0x7f9018153950 2024-02-21T23:06:37.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.398+0000 7f900fe00640 1 --2- 172.21.15.28:0/2453423440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9018153950 0x7f90180a7270 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f9000000c00 tx=0x7f90000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:37.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.398+0000 7f900f400640 1 -- 172.21.15.28:0/2453423440 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9000004230 con 0x7f9018153950 2024-02-21T23:06:37.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.398+0000 7f900f400640 1 -- 172.21.15.28:0/2453423440 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90000043d0 con 0x7f9018153950 2024-02-21T23:06:37.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.398+0000 7f900f400640 1 -- 172.21.15.28:0/2453423440 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9000003850 con 0x7f9018153950 2024-02-21T23:06:37.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.398+0000 7f901e400640 1 -- 172.21.15.28:0/2453423440 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90180a6a70 con 0x7f9018153950 2024-02-21T23:06:37.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.399+0000 7f901e400640 1 -- 172.21.15.28:0/2453423440 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9018142860 con 0x7f9018153950 2024-02-21T23:06:37.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.400+0000 7f901e400640 1 -- 172.21.15.28:0/2453423440 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9018067f30 con 0x7f9018153950 2024-02-21T23:06:37.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.401+0000 7f900f400640 1 -- 172.21.15.28:0/2453423440 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f90000039f0 con 0x7f9018153950 2024-02-21T23:06:37.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.402+0000 7f900f400640 1 --2- 172.21.15.28:0/2453423440 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8ff406b4d0 0x7f8ff406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:37.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.402+0000 7f900f400640 1 -- 172.21.15.28:0/2453423440 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6479+0+0 (secure 0 0 0) 0x7f900002f030 con 0x7f9018153950 2024-02-21T23:06:37.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.403+0000 7f9017400640 1 --2- 172.21.15.28:0/2453423440 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8ff406b4d0 0x7f8ff406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:37.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.404+0000 7f9017400640 1 --2- 172.21.15.28:0/2453423440 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8ff406b4d0 0x7f8ff406d990 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f9008007fd0 tx=0x7f9008007de0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:37.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.407+0000 7f900f400640 1 -- 172.21.15.28:0/2453423440 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f9000041070 con 0x7f9018153950 2024-02-21T23:06:37.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.613+0000 7f901e400640 1 -- 172.21.15.28:0/2453423440 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f90180c0960 con 0x7f9018153950 2024-02-21T23:06:37.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.615+0000 7f900f400640 1 -- 172.21.15.28:0/2453423440 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+17407 (secure 0 0 0) 0x7f900007b390 con 0x7f9018153950 2024-02-21T23:06:37.604 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:06:37.604 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":87,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:06:34.484343+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs2_metadata","create_time":"2024-02-21T23:06:08.240443+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs2_data","create_time":"2024-02-21T23:06:09.466016+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.2","mappings":[{"from":1,"to":7}]},{"pgid":"14.12","mappings":[{"from":1,"to":5}]},{"pgid":"14.1e","mappings":[{"from":1,"to":2}]},{"pgid":"14.24","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6833/624384999":"2024-02-22T23:06:32.469969+0000","172.21.15.28:6832/624384999":"2024-02-22T23:06:32.469969+0000","172.21.15.28:6835/1066741568":"2024-02-22T23:06:31.975858+0000","172.21.15.28:6834/1066741568":"2024-02-22T23:06:31.975858+0000","172.21.15.28:6835/595682298":"2024-02-22T23:06:23.690106+0000","172.21.15.28:6834/595682298":"2024-02-22T23:06:23.690106+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:06:37.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.618+0000 7f901e400640 1 -- 172.21.15.28:0/2453423440 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8ff406b4d0 msgr2=0x7f8ff406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:37.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.618+0000 7f901e400640 1 --2- 172.21.15.28:0/2453423440 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8ff406b4d0 0x7f8ff406d990 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f9008007fd0 tx=0x7f9008007de0 comp rx=0 tx=0).stop 2024-02-21T23:06:37.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.618+0000 7f901e400640 1 -- 172.21.15.28:0/2453423440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9018153950 msgr2=0x7f90180a7270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:37.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.618+0000 7f901e400640 1 --2- 172.21.15.28:0/2453423440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9018153950 0x7f90180a7270 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f9000000c00 tx=0x7f90000046c0 comp rx=0 tx=0).stop 2024-02-21T23:06:37.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.618+0000 7f901e400640 1 -- 172.21.15.28:0/2453423440 shutdown_connections 2024-02-21T23:06:37.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.618+0000 7f901e400640 1 --2- 172.21.15.28:0/2453423440 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8ff406b4d0 0x7f8ff406d990 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:37.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.618+0000 7f901e400640 1 --2- 172.21.15.28:0/2453423440 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f901815b6d0 0x7f90180a77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:37.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.618+0000 7f901e400640 1 --2- 172.21.15.28:0/2453423440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9018153950 0x7f90180a7270 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:37.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.618+0000 7f901e400640 1 --2- 172.21.15.28:0/2453423440 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f90180c11f0 0x7f9018142320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:37.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.618+0000 7f901e400640 1 -- 172.21.15.28:0/2453423440 >> 172.21.15.28:0/2453423440 conn(0x7f90180c67f0 msgr2=0x7f901814bd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:37.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.618+0000 7f901e400640 1 -- 172.21.15.28:0/2453423440 shutdown_connections 2024-02-21T23:06:37.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.618+0000 7f901e400640 1 -- 172.21.15.28:0/2453423440 wait complete. 2024-02-21T23:06:37.626 INFO:tasks.cephfs.filesystem:Destroying file system cephfs2 and related pools 2024-02-21T23:06:37.626 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:06:37.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.862+0000 7fcdd9400640 1 Processor -- start 2024-02-21T23:06:37.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.862+0000 7fcdd9400640 1 -- start start 2024-02-21T23:06:37.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.864+0000 7fcdd9400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fcdd40a5fd0 0x7fcdd406dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:37.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.864+0000 7fcdd9400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdd40c2760 con 0x7fcdd406ee40 2024-02-21T23:06:37.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.864+0000 7fcdd9400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdd4156310 con 0x7fcdd4150450 2024-02-21T23:06:37.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.864+0000 7fcdd9400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdd40c22c0 con 0x7fcdd40a5fd0 2024-02-21T23:06:37.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.864+0000 7fcdd3400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fcdd406ee40 0x7fcdd4156af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59524/0 (socket says 172.21.15.28:59524) 2024-02-21T23:06:37.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.864+0000 7fcdd3400640 1 -- 172.21.15.28:0/2467272964 learned_addr learned my addr 172.21.15.28:0/2467272964 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:06:37.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.865+0000 7fcdd8a00640 1 --2- 172.21.15.28:0/2467272964 >> v2:172.21.15.81:3300/0 conn(0x7fcdd40a5fd0 0x7fcdd406dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:37.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.865+0000 7fcdd2a00640 1 -- 172.21.15.28:0/2467272964 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4026369584 0 0) 0x7fcdd40c2760 con 0x7fcdd406ee40 2024-02-21T23:06:37.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.865+0000 7fcdd2a00640 1 -- 172.21.15.28:0/2467272964 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdb8003520 con 0x7fcdd406ee40 2024-02-21T23:06:37.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.865+0000 7fcdd2a00640 1 -- 172.21.15.28:0/2467272964 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3913535056 0 0) 0x7fcdb8003520 con 0x7fcdd406ee40 2024-02-21T23:06:37.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.865+0000 7fcdd2a00640 1 -- 172.21.15.28:0/2467272964 >> v2:172.21.15.81:3300/0 conn(0x7fcdd40a5fd0 msgr2=0x7fcdd406dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:37.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.865+0000 7fcdd2a00640 1 --2- 172.21.15.28:0/2467272964 >> v2:172.21.15.81:3300/0 conn(0x7fcdd40a5fd0 0x7fcdd406dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:37.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.865+0000 7fcdd2a00640 1 -- 172.21.15.28:0/2467272964 >> v1:172.21.15.81:6789/0 conn(0x7fcdd4150450 legacy=0x7fcdd41536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:37.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.865+0000 7fcdd2a00640 1 -- 172.21.15.28:0/2467272964 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdd40c29d0 con 0x7fcdd406ee40 2024-02-21T23:06:37.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.865+0000 7fcdd2a00640 1 -- 172.21.15.28:0/2467272964 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fcdc0001670 con 0x7fcdd406ee40 2024-02-21T23:06:37.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.865+0000 7fcdd2a00640 1 -- 172.21.15.28:0/2467272964 >> v1:172.21.15.28:6789/0 conn(0x7fcdd406ee40 legacy=0x7fcdd4156af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:37.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.865+0000 7fcdd2a00640 1 --2- 172.21.15.28:0/2467272964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdb80039b0 0x7fcdb8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:37.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.865+0000 7fcdd2a00640 1 --2- 172.21.15.28:0/2467272964 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcdb8004800 0x7fcdb8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:37.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.865+0000 7fcdd2a00640 1 --2- 172.21.15.28:0/2467272964 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcdb80051c0 0x7fcdb800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:37.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.865+0000 7fcdd2a00640 1 -- 172.21.15.28:0/2467272964 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb800c030 con 0x7fcdb80039b0 2024-02-21T23:06:37.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.865+0000 7fcdd2a00640 1 -- 172.21.15.28:0/2467272964 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb800c1e0 con 0x7fcdb80051c0 2024-02-21T23:06:37.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.865+0000 7fcdd2a00640 1 -- 172.21.15.28:0/2467272964 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb800c3e0 con 0x7fcdb8004800 2024-02-21T23:06:37.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.866+0000 7fcdd3e00640 1 --2- 172.21.15.28:0/2467272964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdb80039b0 0x7fcdb8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:37.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.866+0000 7fcdd8a00640 1 --2- 172.21.15.28:0/2467272964 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcdb80051c0 0x7fcdb800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:37.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.866+0000 7fcdd3e00640 1 -- 172.21.15.28:0/2467272964 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcdb8004800 msgr2=0x7fcdb8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:37.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.866+0000 7fcdd3e00640 1 --2- 172.21.15.28:0/2467272964 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcdb8004800 0x7fcdb8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:37.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.866+0000 7fcdd3e00640 1 -- 172.21.15.28:0/2467272964 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcdb80051c0 msgr2=0x7fcdb800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:37.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.866+0000 7fcdd3e00640 1 --2- 172.21.15.28:0/2467272964 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcdb80051c0 0x7fcdb800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:37.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.866+0000 7fcdd3e00640 1 -- 172.21.15.28:0/2467272964 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdb8003520 con 0x7fcdb80039b0 2024-02-21T23:06:37.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.866+0000 7fcdd8a00640 1 --2- 172.21.15.28:0/2467272964 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcdb80051c0 0x7fcdb800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:06:37.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.866+0000 7fcdd8a00640 1 -- 172.21.15.28:0/2467272964 reap_dead start 2024-02-21T23:06:37.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.867+0000 7fcdd3e00640 1 --2- 172.21.15.28:0/2467272964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdb80039b0 0x7fcdb8003dd0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fcdc4001750 tx=0x7fcdb8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e13f0a2b5e70aa9c server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:37.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.867+0000 7fcdd2a00640 1 -- 172.21.15.28:0/2467272964 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc4001370 con 0x7fcdb80039b0 2024-02-21T23:06:37.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.868+0000 7fcdd2a00640 1 -- 172.21.15.28:0/2467272964 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdc40052d0 con 0x7fcdb80039b0 2024-02-21T23:06:37.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.868+0000 7fcdd2a00640 1 -- 172.21.15.28:0/2467272964 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc4003830 con 0x7fcdb80039b0 2024-02-21T23:06:37.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.868+0000 7fcdd9400640 1 -- 172.21.15.28:0/2467272964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdb80039b0 msgr2=0x7fcdb8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:37.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.868+0000 7fcdd9400640 1 --2- 172.21.15.28:0/2467272964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdb80039b0 0x7fcdb8003dd0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fcdc4001750 tx=0x7fcdb8005830 comp rx=0 tx=0).stop 2024-02-21T23:06:37.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.868+0000 7fcdd9400640 1 -- 172.21.15.28:0/2467272964 shutdown_connections 2024-02-21T23:06:37.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.868+0000 7fcdd9400640 1 --2- 172.21.15.28:0/2467272964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdb80039b0 0x7fcdb8003dd0 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:37.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.868+0000 7fcdd9400640 1 -- 172.21.15.28:0/2467272964 >> 172.21.15.28:0/2467272964 conn(0x7fcdd406b630 msgr2=0x7fcdd40c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:37.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.868+0000 7fcdd9400640 1 -- 172.21.15.28:0/2467272964 shutdown_connections 2024-02-21T23:06:37.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.868+0000 7fcdd9400640 1 -- 172.21.15.28:0/2467272964 wait complete. 2024-02-21T23:06:37.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.870+0000 7fcdd9400640 1 Processor -- start 2024-02-21T23:06:37.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.870+0000 7fcdd9400640 1 -- start start 2024-02-21T23:06:37.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.871+0000 7fcdd9400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdb80039b0 0x7fcdd40c40f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:37.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.871+0000 7fcdd9400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcdd41ee270 0x7fcdd4152890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:37.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.871+0000 7fcdd9400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcdd4151880 0x7fcdd4148990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:37.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.871+0000 7fcdd9400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd41536f0 con 0x7fcdb80039b0 2024-02-21T23:06:37.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.871+0000 7fcdd9400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd4156660 con 0x7fcdd41ee270 2024-02-21T23:06:37.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.871+0000 7fcdd9400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd4153870 con 0x7fcdd4151880 2024-02-21T23:06:37.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.871+0000 7fcdd3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdb80039b0 0x7fcdd40c40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:37.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.871+0000 7fcdd3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdb80039b0 0x7fcdd40c40f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59274/0 (socket says 172.21.15.28:59274) 2024-02-21T23:06:37.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.871+0000 7fcdd3e00640 1 -- 172.21.15.28:0/127848611 learned_addr learned my addr 172.21.15.28:0/127848611 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:37.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.872+0000 7fcdd3400640 1 --2- 172.21.15.28:0/127848611 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcdd41ee270 0x7fcdd4152890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:37.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.872+0000 7fcdd8a00640 1 --2- 172.21.15.28:0/127848611 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcdd4151880 0x7fcdd4148990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:37.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.872+0000 7fcdd3e00640 1 -- 172.21.15.28:0/127848611 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcdd4151880 msgr2=0x7fcdd4148990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:37.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.872+0000 7fcdd3e00640 1 --2- 172.21.15.28:0/127848611 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcdd4151880 0x7fcdd4148990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:37.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.872+0000 7fcdd3e00640 1 -- 172.21.15.28:0/127848611 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcdd41ee270 msgr2=0x7fcdd4152890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:37.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.872+0000 7fcdd3e00640 1 --2- 172.21.15.28:0/127848611 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcdd41ee270 0x7fcdd4152890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:37.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.872+0000 7fcdd3e00640 1 -- 172.21.15.28:0/127848611 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdd414acc0 con 0x7fcdb80039b0 2024-02-21T23:06:37.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.872+0000 7fcdd3e00640 1 --2- 172.21.15.28:0/127848611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdb80039b0 0x7fcdd40c40f0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fcdc4034040 tx=0x7fcdc4001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:37.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.873+0000 7fcdcbe00640 1 -- 172.21.15.28:0/127848611 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc4041070 con 0x7fcdb80039b0 2024-02-21T23:06:37.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.873+0000 7fcdcbe00640 1 -- 172.21.15.28:0/127848611 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdc4005480 con 0x7fcdb80039b0 2024-02-21T23:06:37.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.873+0000 7fcdcbe00640 1 -- 172.21.15.28:0/127848611 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc403d640 con 0x7fcdb80039b0 2024-02-21T23:06:37.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.873+0000 7fcdd9400640 1 -- 172.21.15.28:0/127848611 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdd414afa0 con 0x7fcdb80039b0 2024-02-21T23:06:37.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.873+0000 7fcdd9400640 1 -- 172.21.15.28:0/127848611 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdd4151080 con 0x7fcdb80039b0 2024-02-21T23:06:37.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.874+0000 7fcdd9400640 1 -- 172.21.15.28:0/127848611 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdd404f290 con 0x7fcdb80039b0 2024-02-21T23:06:37.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.875+0000 7fcdcbe00640 1 -- 172.21.15.28:0/127848611 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcdc4044070 con 0x7fcdb80039b0 2024-02-21T23:06:37.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.876+0000 7fcdcbe00640 1 --2- 172.21.15.28:0/127848611 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcdb406b500 0x7fcdb406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:37.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.876+0000 7fcdcbe00640 1 -- 172.21.15.28:0/127848611 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6479+0+0 (secure 0 0 0) 0x7fcdc40abfd0 con 0x7fcdb80039b0 2024-02-21T23:06:37.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.876+0000 7fcdd3400640 1 --2- 172.21.15.28:0/127848611 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcdb406b500 0x7fcdb406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:37.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.878+0000 7fcdd3400640 1 --2- 172.21.15.28:0/127848611 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcdb406b500 0x7fcdb406d9c0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fcdd40a84c0 tx=0x7fcdc0001640 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:37.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:37.881+0000 7fcdcbe00640 1 -- 172.21.15.28:0/127848611 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fcdc407abd0 con 0x7fcdb80039b0 2024-02-21T23:06:38.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.122+0000 7fcdd9400640 1 -- 172.21.15.28:0/127848611 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcdd40c1f10 con 0x7fcdb80039b0 2024-02-21T23:06:38.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.123+0000 7fcdcbe00640 1 -- 172.21.15.28:0/127848611 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+4941 (secure 0 0 0) 0x7fcdc407a4d0 con 0x7fcdb80039b0 2024-02-21T23:06:38.112 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:06:38.112 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":79,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":8,"addr":"172.21.15.81:6837/4292940632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":4292940632},{"type":"v1","addr":"172.21.15.81:6837","nonce":4292940632}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67},{"gid":6831,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2340244964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2340244964},{"type":"v1","addr":"172.21.15.28:6835","nonce":2340244964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":76},{"gid":6846,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/2977800995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2977800995},{"type":"v1","addr":"172.21.15.28:6833","nonce":2977800995}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:06:10.484421+0000","modified":"2024-02-21T23:06:34.512209+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5288},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5288":{"gid":5288,"name":"d","rank":0,"incarnation":76,"state":"up:active","state_seq":15,"addr":"172.21.15.81:6835/2774951107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2774951107},{"type":"v1","addr":"172.21.15.81:6835","nonce":2774951107}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-02-21T23:06:38.113 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 79 2024-02-21T23:06:38.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.126+0000 7fcdd9400640 1 -- 172.21.15.28:0/127848611 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcdb406b500 msgr2=0x7fcdb406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:38.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.126+0000 7fcdd9400640 1 --2- 172.21.15.28:0/127848611 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcdb406b500 0x7fcdb406d9c0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fcdd40a84c0 tx=0x7fcdc0001640 comp rx=0 tx=0).stop 2024-02-21T23:06:38.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.126+0000 7fcdd9400640 1 -- 172.21.15.28:0/127848611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdb80039b0 msgr2=0x7fcdd40c40f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:38.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.126+0000 7fcdd9400640 1 --2- 172.21.15.28:0/127848611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdb80039b0 0x7fcdd40c40f0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fcdc4034040 tx=0x7fcdc4001120 comp rx=0 tx=0).stop 2024-02-21T23:06:38.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.126+0000 7fcdd9400640 1 -- 172.21.15.28:0/127848611 shutdown_connections 2024-02-21T23:06:38.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.126+0000 7fcdd9400640 1 --2- 172.21.15.28:0/127848611 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcdb406b500 0x7fcdb406d9c0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:38.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.126+0000 7fcdd9400640 1 --2- 172.21.15.28:0/127848611 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcdd4151880 0x7fcdd4148990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:38.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.126+0000 7fcdd9400640 1 --2- 172.21.15.28:0/127848611 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcdd41ee270 0x7fcdd4152890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:38.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.126+0000 7fcdd9400640 1 --2- 172.21.15.28:0/127848611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdb80039b0 0x7fcdd40c40f0 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:38.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.126+0000 7fcdd9400640 1 -- 172.21.15.28:0/127848611 >> 172.21.15.28:0/127848611 conn(0x7fcdd406b630 msgr2=0x7fcdd40c64e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:38.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.126+0000 7fcdd9400640 1 -- 172.21.15.28:0/127848611 shutdown_connections 2024-02-21T23:06:38.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.126+0000 7fcdd9400640 1 -- 172.21.15.28:0/127848611 wait complete. 2024-02-21T23:06:38.135 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:06:38.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.367+0000 7f549f200640 1 Processor -- start 2024-02-21T23:06:38.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.368+0000 7f549f200640 1 -- start start 2024-02-21T23:06:38.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.369+0000 7f549f200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f54a006ee40 0x7f54a00a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:38.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.369+0000 7f549f200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54a00c2760 con 0x7f54a0156af0 2024-02-21T23:06:38.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.369+0000 7f549f200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54a0156310 con 0x7f54a0150450 2024-02-21T23:06:38.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.369+0000 7f549f200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54a00c22c0 con 0x7f54a006ee40 2024-02-21T23:06:38.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.369+0000 7f549e800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f54a0156af0 0x7f54a006dff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59530/0 (socket says 172.21.15.28:59530) 2024-02-21T23:06:38.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.369+0000 7f549e800640 1 -- 172.21.15.28:0/3999549109 learned_addr learned my addr 172.21.15.28:0/3999549109 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:06:38.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.370+0000 7f5497e00640 1 --2- 172.21.15.28:0/3999549109 >> v2:172.21.15.81:3300/0 conn(0x7f54a006ee40 0x7f54a00a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:38.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.370+0000 7f549d400640 1 -- 172.21.15.28:0/3999549109 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2630110809 0 0) 0x7f54a00c2760 con 0x7f54a0156af0 2024-02-21T23:06:38.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.370+0000 7f549d400640 1 -- 172.21.15.28:0/3999549109 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5484003520 con 0x7f54a0156af0 2024-02-21T23:06:38.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.370+0000 7f549d400640 1 -- 172.21.15.28:0/3999549109 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3778086234 0 0) 0x7f5484003520 con 0x7f54a0156af0 2024-02-21T23:06:38.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.370+0000 7f549d400640 1 -- 172.21.15.28:0/3999549109 >> v2:172.21.15.81:3300/0 conn(0x7f54a006ee40 msgr2=0x7f54a00a5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:38.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.370+0000 7f549d400640 1 --2- 172.21.15.28:0/3999549109 >> v2:172.21.15.81:3300/0 conn(0x7f54a006ee40 0x7f54a00a5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:38.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.370+0000 7f549d400640 1 -- 172.21.15.28:0/3999549109 >> v1:172.21.15.81:6789/0 conn(0x7f54a0150450 legacy=0x7f54a01536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:38.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.370+0000 7f549d400640 1 -- 172.21.15.28:0/3999549109 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54a00c29d0 con 0x7f54a0156af0 2024-02-21T23:06:38.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.371+0000 7f549d400640 1 -- 172.21.15.28:0/3999549109 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f54980019f0 con 0x7f54a0156af0 2024-02-21T23:06:38.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.371+0000 7f549d400640 1 -- 172.21.15.28:0/3999549109 >> v1:172.21.15.28:6789/0 conn(0x7f54a0156af0 legacy=0x7f54a006dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:38.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.371+0000 7f549d400640 1 --2- 172.21.15.28:0/3999549109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54840039b0 0x7f5484003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:38.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.371+0000 7f549d400640 1 --2- 172.21.15.28:0/3999549109 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5484004800 0x7f5484004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:38.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.371+0000 7f549d400640 1 --2- 172.21.15.28:0/3999549109 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54840051c0 0x7f548400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:38.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.371+0000 7f549d400640 1 -- 172.21.15.28:0/3999549109 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f548400c030 con 0x7f54840039b0 2024-02-21T23:06:38.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.371+0000 7f549d400640 1 -- 172.21.15.28:0/3999549109 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f548400c1e0 con 0x7f54840051c0 2024-02-21T23:06:38.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.371+0000 7f549d400640 1 -- 172.21.15.28:0/3999549109 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f548400c3e0 con 0x7f5484004800 2024-02-21T23:06:38.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.371+0000 7f549de00640 1 --2- 172.21.15.28:0/3999549109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54840039b0 0x7f5484003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:38.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.371+0000 7f5497e00640 1 --2- 172.21.15.28:0/3999549109 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5484004800 0x7f5484004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:38.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.371+0000 7f549de00640 1 -- 172.21.15.28:0/3999549109 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5484004800 msgr2=0x7f5484004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:38.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.371+0000 7f549de00640 1 --2- 172.21.15.28:0/3999549109 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5484004800 0x7f5484004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:38.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.371+0000 7f549de00640 1 -- 172.21.15.28:0/3999549109 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54840051c0 msgr2=0x7f548400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:38.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.371+0000 7f549e800640 1 --2- 172.21.15.28:0/3999549109 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54840051c0 0x7f548400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:38.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.372+0000 7f549de00640 1 --2- 172.21.15.28:0/3999549109 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54840051c0 0x7f548400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:38.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.372+0000 7f549de00640 1 -- 172.21.15.28:0/3999549109 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5484003520 con 0x7f54840039b0 2024-02-21T23:06:38.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.372+0000 7f549e800640 1 --2- 172.21.15.28:0/3999549109 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54840051c0 0x7f548400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:06:38.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.372+0000 7f549e800640 1 -- 172.21.15.28:0/3999549109 reap_dead start 2024-02-21T23:06:38.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.373+0000 7f549de00640 1 --2- 172.21.15.28:0/3999549109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54840039b0 0x7f5484003dd0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f5490001750 tx=0x7f5484005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52219b19f0167e5c server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:38.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.373+0000 7f549d400640 1 -- 172.21.15.28:0/3999549109 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5490001370 con 0x7f54840039b0 2024-02-21T23:06:38.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.373+0000 7f549d400640 1 -- 172.21.15.28:0/3999549109 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54900052d0 con 0x7f54840039b0 2024-02-21T23:06:38.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.373+0000 7f549d400640 1 -- 172.21.15.28:0/3999549109 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5490003830 con 0x7f54840039b0 2024-02-21T23:06:38.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.373+0000 7f549f200640 1 -- 172.21.15.28:0/3999549109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54840039b0 msgr2=0x7f5484003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:38.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.373+0000 7f549f200640 1 --2- 172.21.15.28:0/3999549109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54840039b0 0x7f5484003dd0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f5490001750 tx=0x7f5484005830 comp rx=0 tx=0).stop 2024-02-21T23:06:38.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.374+0000 7f549f200640 1 -- 172.21.15.28:0/3999549109 shutdown_connections 2024-02-21T23:06:38.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.374+0000 7f549f200640 1 --2- 172.21.15.28:0/3999549109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54840039b0 0x7f5484003dd0 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:38.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.374+0000 7f549f200640 1 -- 172.21.15.28:0/3999549109 >> 172.21.15.28:0/3999549109 conn(0x7f54a006b630 msgr2=0x7f54a00c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:38.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.374+0000 7f549f200640 1 -- 172.21.15.28:0/3999549109 shutdown_connections 2024-02-21T23:06:38.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.374+0000 7f549f200640 1 -- 172.21.15.28:0/3999549109 wait complete. 2024-02-21T23:06:38.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.376+0000 7f549f200640 1 Processor -- start 2024-02-21T23:06:38.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.376+0000 7f549f200640 1 -- start start 2024-02-21T23:06:38.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.377+0000 7f549f200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54840039b0 0x7f54a01e9480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:38.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.377+0000 7f549f200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54a01dc2e0 0x7f54a01dfd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:38.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.377+0000 7f549f200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f54a014a960 0x7f54a01ddbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:38.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.377+0000 7f549f200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54a0156f80 con 0x7f54a01dc2e0 2024-02-21T23:06:38.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.377+0000 7f549f200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54a0156660 con 0x7f54840039b0 2024-02-21T23:06:38.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.377+0000 7f549f200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54a0157100 con 0x7f54a014a960 2024-02-21T23:06:38.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.377+0000 7f5497e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54a01dc2e0 0x7f54a01dfd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:38.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.377+0000 7f5497e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54a01dc2e0 0x7f54a01dfd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59286/0 (socket says 172.21.15.28:59286) 2024-02-21T23:06:38.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.377+0000 7f5497e00640 1 -- 172.21.15.28:0/1255740746 learned_addr learned my addr 172.21.15.28:0/1255740746 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:38.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.377+0000 7f549de00640 1 --2- 172.21.15.28:0/1255740746 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54840039b0 0x7f54a01e9480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:38.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.377+0000 7f549e800640 1 --2- 172.21.15.28:0/1255740746 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f54a014a960 0x7f54a01ddbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:38.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.377+0000 7f5497e00640 1 -- 172.21.15.28:0/1255740746 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f54a014a960 msgr2=0x7f54a01ddbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:38.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.378+0000 7f5497e00640 1 --2- 172.21.15.28:0/1255740746 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f54a014a960 0x7f54a01ddbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:38.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.378+0000 7f5497e00640 1 -- 172.21.15.28:0/1255740746 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54840039b0 msgr2=0x7f54a01e9480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:38.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.378+0000 7f5497e00640 1 --2- 172.21.15.28:0/1255740746 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54840039b0 0x7f54a01e9480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:38.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.378+0000 7f5497e00640 1 -- 172.21.15.28:0/1255740746 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54a01ec9c0 con 0x7f54a01dc2e0 2024-02-21T23:06:38.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.378+0000 7f549e800640 1 --2- 172.21.15.28:0/1255740746 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f54a014a960 0x7f54a01ddbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:06:38.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.378+0000 7f549de00640 1 --2- 172.21.15.28:0/1255740746 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54840039b0 0x7f54a01e9480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:06:38.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.378+0000 7f5497e00640 1 --2- 172.21.15.28:0/1255740746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54a01dc2e0 0x7f54a01dfd30 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f54880108e0 tx=0x7f5488010da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:38.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.378+0000 7f5496000640 1 -- 172.21.15.28:0/1255740746 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f548800f040 con 0x7f54a01dc2e0 2024-02-21T23:06:38.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.378+0000 7f5496000640 1 -- 172.21.15.28:0/1255740746 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f548800eb00 con 0x7f54a01dc2e0 2024-02-21T23:06:38.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.378+0000 7f5496000640 1 -- 172.21.15.28:0/1255740746 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f548801c400 con 0x7f54a01dc2e0 2024-02-21T23:06:38.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.378+0000 7f549f200640 1 -- 172.21.15.28:0/1255740746 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54a01ecca0 con 0x7f54a01dc2e0 2024-02-21T23:06:38.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.378+0000 7f549f200640 1 -- 172.21.15.28:0/1255740746 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54a01e8930 con 0x7f54a01dc2e0 2024-02-21T23:06:38.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.379+0000 7f549f200640 1 -- 172.21.15.28:0/1255740746 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54a004f290 con 0x7f54a01dc2e0 2024-02-21T23:06:38.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.380+0000 7f5496000640 1 -- 172.21.15.28:0/1255740746 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f548801c5a0 con 0x7f54a01dc2e0 2024-02-21T23:06:38.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.381+0000 7f5496000640 1 --2- 172.21.15.28:0/1255740746 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f547c06b4d0 0x7f547c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:38.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.381+0000 7f5496000640 1 -- 172.21.15.28:0/1255740746 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6479+0+0 (secure 0 0 0) 0x7f548808ab30 con 0x7f54a01dc2e0 2024-02-21T23:06:38.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.382+0000 7f549de00640 1 --2- 172.21.15.28:0/1255740746 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f547c06b4d0 0x7f547c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:38.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.383+0000 7f549de00640 1 --2- 172.21.15.28:0/1255740746 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f547c06b4d0 0x7f547c06d990 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f5490033000 tx=0x7f5490009030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:38.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.386+0000 7f5496000640 1 -- 172.21.15.28:0/1255740746 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f5488018030 con 0x7f54a01dc2e0 2024-02-21T23:06:38.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.626+0000 7f549f200640 1 -- 172.21.15.28:0/1255740746 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f54a00c0be0 con 0x7f54a01dc2e0 2024-02-21T23:06:38.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.627+0000 7f5496000640 1 -- 172.21.15.28:0/1255740746 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+4941 (secure 0 0 0) 0x7f5488058fc0 con 0x7f54a01dc2e0 2024-02-21T23:06:38.616 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:06:38.616 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":79,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":8,"addr":"172.21.15.81:6837/4292940632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":4292940632},{"type":"v1","addr":"172.21.15.81:6837","nonce":4292940632}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67},{"gid":6831,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2340244964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2340244964},{"type":"v1","addr":"172.21.15.28:6835","nonce":2340244964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":76},{"gid":6846,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/2977800995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2977800995},{"type":"v1","addr":"172.21.15.28:6833","nonce":2977800995}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:06:10.484421+0000","modified":"2024-02-21T23:06:34.512209+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5288},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5288":{"gid":5288,"name":"d","rank":0,"incarnation":76,"state":"up:active","state_seq":15,"addr":"172.21.15.81:6835/2774951107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2774951107},{"type":"v1","addr":"172.21.15.81:6835","nonce":2774951107}]},"join_fscid":6,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-02-21T23:06:38.616 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 79 2024-02-21T23:06:38.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.630+0000 7f549f200640 1 -- 172.21.15.28:0/1255740746 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f547c06b4d0 msgr2=0x7f547c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:38.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.630+0000 7f549f200640 1 --2- 172.21.15.28:0/1255740746 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f547c06b4d0 0x7f547c06d990 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f5490033000 tx=0x7f5490009030 comp rx=0 tx=0).stop 2024-02-21T23:06:38.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.630+0000 7f549f200640 1 -- 172.21.15.28:0/1255740746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54a01dc2e0 msgr2=0x7f54a01dfd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:38.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.630+0000 7f549f200640 1 --2- 172.21.15.28:0/1255740746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54a01dc2e0 0x7f54a01dfd30 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f54880108e0 tx=0x7f5488010da0 comp rx=0 tx=0).stop 2024-02-21T23:06:38.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.630+0000 7f549f200640 1 -- 172.21.15.28:0/1255740746 shutdown_connections 2024-02-21T23:06:38.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.630+0000 7f549f200640 1 --2- 172.21.15.28:0/1255740746 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f547c06b4d0 0x7f547c06d990 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:38.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.630+0000 7f549f200640 1 --2- 172.21.15.28:0/1255740746 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f54a014a960 0x7f54a01ddbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:38.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.630+0000 7f549f200640 1 --2- 172.21.15.28:0/1255740746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54a01dc2e0 0x7f54a01dfd30 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:38.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.630+0000 7f549f200640 1 --2- 172.21.15.28:0/1255740746 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54840039b0 0x7f54a01e9480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:38.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.630+0000 7f549f200640 1 -- 172.21.15.28:0/1255740746 >> 172.21.15.28:0/1255740746 conn(0x7f54a006b630 msgr2=0x7f54a014c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:38.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.630+0000 7f549f200640 1 -- 172.21.15.28:0/1255740746 shutdown_connections 2024-02-21T23:06:38.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.630+0000 7f549f200640 1 -- 172.21.15.28:0/1255740746 wait complete. 2024-02-21T23:06:38.640 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:06:38.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.878+0000 7f4dd7200640 1 Processor -- start 2024-02-21T23:06:38.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.879+0000 7f4dd7200640 1 -- start start 2024-02-21T23:06:38.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.880+0000 7f4dd7200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4dd00aff90 0x7f4dd00c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:38.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.880+0000 7f4dd7200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dd00c74c0 con 0x7f4dd00c5720 2024-02-21T23:06:38.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.880+0000 7f4dd7200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dd00c77a0 con 0x7f4dd006dbf0 2024-02-21T23:06:38.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.880+0000 7f4dd7200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd00c7270 con 0x7f4dd00aff90 2024-02-21T23:06:38.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.880+0000 7f4dd6800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f4dd00c5720 0x7f4dd015f890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59540/0 (socket says 172.21.15.28:59540) 2024-02-21T23:06:38.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.880+0000 7f4dd6800640 1 -- 172.21.15.28:0/3789483146 learned_addr learned my addr 172.21.15.28:0/3789483146 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:06:38.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.881+0000 7f4dd5e00640 1 --2- 172.21.15.28:0/3789483146 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4dd00aff90 0x7f4dd00c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:38.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.881+0000 7f4dd4a00640 1 -- 172.21.15.28:0/3789483146 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 883497374 0 0) 0x7f4dd00c74c0 con 0x7f4dd00c5720 2024-02-21T23:06:38.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.881+0000 7f4dd4a00640 1 -- 172.21.15.28:0/3789483146 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4db8003520 con 0x7f4dd00c5720 2024-02-21T23:06:38.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.881+0000 7f4dd4a00640 1 -- 172.21.15.28:0/3789483146 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1005554945 0 0) 0x7f4db8003520 con 0x7f4dd00c5720 2024-02-21T23:06:38.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.881+0000 7f4dd4a00640 1 -- 172.21.15.28:0/3789483146 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4dd00aff90 msgr2=0x7f4dd00c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:38.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.881+0000 7f4dd4a00640 1 --2- 172.21.15.28:0/3789483146 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4dd00aff90 0x7f4dd00c51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:38.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.881+0000 7f4dd4a00640 1 -- 172.21.15.28:0/3789483146 >> v1:172.21.15.81:6789/0 conn(0x7f4dd006dbf0 legacy=0x7f4dd0158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:38.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.881+0000 7f4dd4a00640 1 -- 172.21.15.28:0/3789483146 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dd00c7a80 con 0x7f4dd00c5720 2024-02-21T23:06:38.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.881+0000 7f4dd4a00640 1 -- 172.21.15.28:0/3789483146 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f4dcc0019f0 con 0x7f4dd00c5720 2024-02-21T23:06:38.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.881+0000 7f4dd4a00640 1 -- 172.21.15.28:0/3789483146 >> v1:172.21.15.28:6789/0 conn(0x7f4dd00c5720 legacy=0x7f4dd015f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:38.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.881+0000 7f4dd4a00640 1 --2- 172.21.15.28:0/3789483146 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4db80039e0 0x7f4db8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:38.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.881+0000 7f4dd4a00640 1 --2- 172.21.15.28:0/3789483146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4db8004800 0x7f4db8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:38.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.882+0000 7f4dd4a00640 1 --2- 172.21.15.28:0/3789483146 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4dd00aff90 0x7f4db80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:38.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.882+0000 7f4dd4a00640 1 -- 172.21.15.28:0/3789483146 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd00c7270 con 0x7f4db8004800 2024-02-21T23:06:38.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.882+0000 7f4dd4a00640 1 -- 172.21.15.28:0/3789483146 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db800ba70 con 0x7f4db80039e0 2024-02-21T23:06:38.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.882+0000 7f4dd4a00640 1 -- 172.21.15.28:0/3789483146 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4db800bc70 con 0x7f4dd00aff90 2024-02-21T23:06:38.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.882+0000 7f4dd5400640 1 --2- 172.21.15.28:0/3789483146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4db8004800 0x7f4db8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:38.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.882+0000 7f4dd5e00640 1 --2- 172.21.15.28:0/3789483146 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4db80039e0 0x7f4db8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:38.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.882+0000 7f4dd5400640 1 -- 172.21.15.28:0/3789483146 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4dd00aff90 msgr2=0x7f4db80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:38.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.882+0000 7f4dd6800640 1 --2- 172.21.15.28:0/3789483146 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4dd00aff90 0x7f4db80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:38.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.882+0000 7f4dd5400640 1 --2- 172.21.15.28:0/3789483146 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4dd00aff90 0x7f4db80051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:38.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.882+0000 7f4dd5400640 1 -- 172.21.15.28:0/3789483146 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4db80039e0 msgr2=0x7f4db8003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:38.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.882+0000 7f4dd5400640 1 --2- 172.21.15.28:0/3789483146 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4db80039e0 0x7f4db8003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:38.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.882+0000 7f4dd5400640 1 -- 172.21.15.28:0/3789483146 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4db8003520 con 0x7f4db8004800 2024-02-21T23:06:38.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.882+0000 7f4dd5e00640 1 --2- 172.21.15.28:0/3789483146 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4db80039e0 0x7f4db8003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:06:38.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.884+0000 7f4dd5400640 1 --2- 172.21.15.28:0/3789483146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4db8004800 0x7f4db8004c80 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f4dc0001720 tx=0x7f4db8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5d31c20d66f770 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:38.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.884+0000 7f4dd4a00640 1 -- 172.21.15.28:0/3789483146 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc0001370 con 0x7f4db8004800 2024-02-21T23:06:38.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.884+0000 7f4dd4a00640 1 -- 172.21.15.28:0/3789483146 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dc00052d0 con 0x7f4db8004800 2024-02-21T23:06:38.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.884+0000 7f4dd4a00640 1 -- 172.21.15.28:0/3789483146 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc0003830 con 0x7f4db8004800 2024-02-21T23:06:38.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.884+0000 7f4dd7200640 1 -- 172.21.15.28:0/3789483146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4db8004800 msgr2=0x7f4db8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:38.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.884+0000 7f4dd7200640 1 --2- 172.21.15.28:0/3789483146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4db8004800 0x7f4db8004c80 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f4dc0001720 tx=0x7f4db8005830 comp rx=0 tx=0).stop 2024-02-21T23:06:38.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.884+0000 7f4dd6800640 1 -- 172.21.15.28:0/3789483146 reap_dead start 2024-02-21T23:06:38.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.885+0000 7f4dd7200640 1 -- 172.21.15.28:0/3789483146 shutdown_connections 2024-02-21T23:06:38.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.885+0000 7f4dd7200640 1 -- 172.21.15.28:0/3789483146 >> 172.21.15.28:0/3789483146 conn(0x7f4dd00ac370 msgr2=0x7f4dd00ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:38.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.885+0000 7f4dd7200640 1 -- 172.21.15.28:0/3789483146 shutdown_connections 2024-02-21T23:06:38.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.885+0000 7f4dd7200640 1 -- 172.21.15.28:0/3789483146 wait complete. 2024-02-21T23:06:38.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.886+0000 7f4dd7200640 1 Processor -- start 2024-02-21T23:06:38.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.886+0000 7f4dd7200640 1 -- start start 2024-02-21T23:06:38.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.887+0000 7f4dd7200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4db8004800 0x7f4dd00ae4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:38.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.887+0000 7f4dd7200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4dd00aea10 0x7f4dd00c2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:38.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.887+0000 7f4dd7200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4dd00c2540 0x7f4dd01edbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:38.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.887+0000 7f4dd7200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd00c8760 con 0x7f4dd00c2540 2024-02-21T23:06:38.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.887+0000 7f4dd7200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd0158630 con 0x7f4dd00aea10 2024-02-21T23:06:38.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.887+0000 7f4dd7200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd00c88e0 con 0x7f4db8004800 2024-02-21T23:06:38.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.888+0000 7f4dd6800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4dd00c2540 0x7f4dd01edbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:38.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.888+0000 7f4dd6800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4dd00c2540 0x7f4dd01edbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59302/0 (socket says 172.21.15.28:59302) 2024-02-21T23:06:38.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.888+0000 7f4dd6800640 1 -- 172.21.15.28:0/3142964414 learned_addr learned my addr 172.21.15.28:0/3142964414 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:38.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.888+0000 7f4dd5e00640 1 --2- 172.21.15.28:0/3142964414 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4db8004800 0x7f4dd00ae4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:38.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.888+0000 7f4dd5400640 1 --2- 172.21.15.28:0/3142964414 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4dd00aea10 0x7f4dd00c2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:38.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.888+0000 7f4dd6800640 1 -- 172.21.15.28:0/3142964414 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4db8004800 msgr2=0x7f4dd00ae4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:38.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.888+0000 7f4dd6800640 1 --2- 172.21.15.28:0/3142964414 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4db8004800 0x7f4dd00ae4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:38.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.888+0000 7f4dd6800640 1 -- 172.21.15.28:0/3142964414 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4dd00aea10 msgr2=0x7f4dd00c2000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:38.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.888+0000 7f4dd6800640 1 --2- 172.21.15.28:0/3142964414 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4dd00aea10 0x7f4dd00c2000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:38.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.888+0000 7f4dd6800640 1 -- 172.21.15.28:0/3142964414 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dd0152d40 con 0x7f4dd00c2540 2024-02-21T23:06:38.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.888+0000 7f4dd5e00640 1 --2- 172.21.15.28:0/3142964414 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4db8004800 0x7f4dd00ae4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:06:38.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.889+0000 7f4dd6800640 1 --2- 172.21.15.28:0/3142964414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4dd00c2540 0x7f4dd01edbf0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f4dd006d630 tx=0x7f4dcc00b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:38.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.889+0000 7f4dca000640 1 -- 172.21.15.28:0/3142964414 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dcc004d60 con 0x7f4dd00c2540 2024-02-21T23:06:38.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.889+0000 7f4dca000640 1 -- 172.21.15.28:0/3142964414 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dcc00bb80 con 0x7f4dd00c2540 2024-02-21T23:06:38.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.889+0000 7f4dd7200640 1 -- 172.21.15.28:0/3142964414 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dd0153020 con 0x7f4dd00c2540 2024-02-21T23:06:38.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.889+0000 7f4dca000640 1 -- 172.21.15.28:0/3142964414 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dcc007110 con 0x7f4dd00c2540 2024-02-21T23:06:38.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.889+0000 7f4dd7200640 1 -- 172.21.15.28:0/3142964414 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dd01535e0 con 0x7f4dd00c2540 2024-02-21T23:06:38.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.890+0000 7f4dd7200640 1 -- 172.21.15.28:0/3142964414 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dd004f290 con 0x7f4dd00c2540 2024-02-21T23:06:38.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.891+0000 7f4dca000640 1 -- 172.21.15.28:0/3142964414 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4dcc01b060 con 0x7f4dd00c2540 2024-02-21T23:06:38.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.892+0000 7f4dca000640 1 --2- 172.21.15.28:0/3142964414 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4db406b500 0x7f4db406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:38.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.892+0000 7f4dca000640 1 -- 172.21.15.28:0/3142964414 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6479+0+0 (secure 0 0 0) 0x7f4dcc08a4f0 con 0x7f4dd00c2540 2024-02-21T23:06:38.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.893+0000 7f4dd5e00640 1 --2- 172.21.15.28:0/3142964414 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4db406b500 0x7f4db406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:38.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.894+0000 7f4dd5e00640 1 --2- 172.21.15.28:0/3142964414 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4db406b500 0x7f4db406d9c0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f4dd00c3f10 tx=0x7f4dc4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:38.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:38.896+0000 7f4dca000640 1 -- 172.21.15.28:0/3142964414 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4dcc059060 con 0x7f4dd00c2540 2024-02-21T23:06:39.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.098+0000 7f4dd7200640 1 -- 172.21.15.28:0/3142964414 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4dd00c5210 con 0x7f4dd00c2540 2024-02-21T23:06:39.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.099+0000 7f4dca000640 1 -- 172.21.15.28:0/3142964414 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+17407 (secure 0 0 0) 0x7f4dcc058960 con 0x7f4dd00c2540 2024-02-21T23:06:39.088 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:06:39.088 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":87,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:06:34.484343+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs2_metadata","create_time":"2024-02-21T23:06:08.240443+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs2_data","create_time":"2024-02-21T23:06:09.466016+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.2","mappings":[{"from":1,"to":7}]},{"pgid":"14.12","mappings":[{"from":1,"to":5}]},{"pgid":"14.1e","mappings":[{"from":1,"to":2}]},{"pgid":"14.24","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6833/624384999":"2024-02-22T23:06:32.469969+0000","172.21.15.28:6832/624384999":"2024-02-22T23:06:32.469969+0000","172.21.15.28:6835/1066741568":"2024-02-22T23:06:31.975858+0000","172.21.15.28:6834/1066741568":"2024-02-22T23:06:31.975858+0000","172.21.15.28:6835/595682298":"2024-02-22T23:06:23.690106+0000","172.21.15.28:6834/595682298":"2024-02-22T23:06:23.690106+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:06:39.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.102+0000 7f4dd7200640 1 -- 172.21.15.28:0/3142964414 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4db406b500 msgr2=0x7f4db406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:39.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.102+0000 7f4dd7200640 1 --2- 172.21.15.28:0/3142964414 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4db406b500 0x7f4db406d9c0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f4dd00c3f10 tx=0x7f4dc4008040 comp rx=0 tx=0).stop 2024-02-21T23:06:39.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.102+0000 7f4dd7200640 1 -- 172.21.15.28:0/3142964414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4dd00c2540 msgr2=0x7f4dd01edbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:39.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.102+0000 7f4dd7200640 1 --2- 172.21.15.28:0/3142964414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4dd00c2540 0x7f4dd01edbf0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f4dd006d630 tx=0x7f4dcc00b4a0 comp rx=0 tx=0).stop 2024-02-21T23:06:39.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.102+0000 7f4dd7200640 1 -- 172.21.15.28:0/3142964414 shutdown_connections 2024-02-21T23:06:39.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.102+0000 7f4dd7200640 1 --2- 172.21.15.28:0/3142964414 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4db406b500 0x7f4db406d9c0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:39.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.102+0000 7f4dd7200640 1 --2- 172.21.15.28:0/3142964414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4dd00c2540 0x7f4dd01edbf0 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:39.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.102+0000 7f4dd7200640 1 --2- 172.21.15.28:0/3142964414 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4dd00aea10 0x7f4dd00c2000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:39.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.102+0000 7f4dd7200640 1 --2- 172.21.15.28:0/3142964414 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4db8004800 0x7f4dd00ae4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:39.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.102+0000 7f4dd7200640 1 -- 172.21.15.28:0/3142964414 >> 172.21.15.28:0/3142964414 conn(0x7f4dd00ac370 msgr2=0x7f4dd00ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:39.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.102+0000 7f4dd7200640 1 -- 172.21.15.28:0/3142964414 shutdown_connections 2024-02-21T23:06:39.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.102+0000 7f4dd7200640 1 -- 172.21.15.28:0/3142964414 wait complete. 2024-02-21T23:06:39.109 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs2 2024-02-21T23:06:39.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.333+0000 7fd007200640 1 Processor -- start 2024-02-21T23:06:39.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.334+0000 7fd007200640 1 -- start start 2024-02-21T23:06:39.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.335+0000 7fd007200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fd000149830 0x7fd0000a8e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:39.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.335+0000 7fd007200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd0000c43e0 0x7fd0000c5a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:39.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.335+0000 7fd007200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd000149c10 con 0x7fd00006d410 2024-02-21T23:06:39.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.335+0000 7fd007200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0001537d0 con 0x7fd0000c43e0 2024-02-21T23:06:39.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.335+0000 7fd007200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd000153ab0 con 0x7fd000149830 2024-02-21T23:06:39.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.335+0000 7fd006800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd0000c43e0 0x7fd0000c5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:39.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.336+0000 7fd006800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd0000c43e0 0x7fd0000c5a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59310/0 (socket says 172.21.15.28:59310) 2024-02-21T23:06:39.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.336+0000 7fd006800640 1 -- 172.21.15.28:0/1478221025 learned_addr learned my addr 172.21.15.28:0/1478221025 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:39.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.336+0000 7fd005e00640 1 --2- 172.21.15.28:0/1478221025 >> v2:172.21.15.81:3300/0 conn(0x7fd000149830 0x7fd0000a8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:39.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.336+0000 7fd006800640 1 -- 172.21.15.28:0/1478221025 >> v2:172.21.15.81:3300/0 conn(0x7fd000149830 msgr2=0x7fd0000a8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:39.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.336+0000 7fd006800640 1 --2- 172.21.15.28:0/1478221025 >> v2:172.21.15.81:3300/0 conn(0x7fd000149830 0x7fd0000a8e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:39.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.336+0000 7fd006800640 1 -- 172.21.15.28:0/1478221025 >> v1:172.21.15.81:6789/0 conn(0x7fd00006d410 legacy=0x7fd0000c0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:39.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.336+0000 7fd006800640 1 -- 172.21.15.28:0/1478221025 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0000c4fa0 con 0x7fd0000c43e0 2024-02-21T23:06:39.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.337+0000 7fd006800640 1 --2- 172.21.15.28:0/1478221025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0000c43e0 0x7fd0000c5a80 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fd00006f7c0 tx=0x7fcffc02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bc6db6b69c5eb6 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:39.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.337+0000 7fd004a00640 1 -- 172.21.15.28:0/1478221025 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcffc02ca80 con 0x7fd0000c43e0 2024-02-21T23:06:39.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.338+0000 7fd004a00640 1 -- 172.21.15.28:0/1478221025 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcffc02cc20 con 0x7fd0000c43e0 2024-02-21T23:06:39.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.338+0000 7fd004a00640 1 -- 172.21.15.28:0/1478221025 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcffc0343f0 con 0x7fd0000c43e0 2024-02-21T23:06:39.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.338+0000 7fd007200640 1 -- 172.21.15.28:0/1478221025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0000c43e0 msgr2=0x7fd0000c5a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:39.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.338+0000 7fd007200640 1 --2- 172.21.15.28:0/1478221025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0000c43e0 0x7fd0000c5a80 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fd00006f7c0 tx=0x7fcffc02bee0 comp rx=0 tx=0).stop 2024-02-21T23:06:39.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.338+0000 7fd007200640 1 -- 172.21.15.28:0/1478221025 shutdown_connections 2024-02-21T23:06:39.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.338+0000 7fd007200640 1 --2- 172.21.15.28:0/1478221025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0000c43e0 0x7fd0000c5a80 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:39.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.338+0000 7fd007200640 1 --2- 172.21.15.28:0/1478221025 >> v2:172.21.15.81:3300/0 conn(0x7fd000149830 0x7fd0000a8e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:39.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.338+0000 7fd007200640 1 -- 172.21.15.28:0/1478221025 >> 172.21.15.28:0/1478221025 conn(0x7fd0000a8320 msgr2=0x7fd0000a8730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:39.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.338+0000 7fd007200640 1 -- 172.21.15.28:0/1478221025 shutdown_connections 2024-02-21T23:06:39.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.338+0000 7fd007200640 1 -- 172.21.15.28:0/1478221025 wait complete. 2024-02-21T23:06:39.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.340+0000 7fd007200640 1 Processor -- start 2024-02-21T23:06:39.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.340+0000 7fd007200640 1 -- start start 2024-02-21T23:06:39.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.341+0000 7fd007200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd00006d410 0x7fd000145e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:39.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.341+0000 7fd007200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0000c43e0 0x7fd0001dfb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:39.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.341+0000 7fd007200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd000149830 0x7fd0000a9620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:39.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.341+0000 7fd007200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0000c15f0 con 0x7fd0000c43e0 2024-02-21T23:06:39.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.341+0000 7fd007200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0000c5560 con 0x7fd000149830 2024-02-21T23:06:39.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.341+0000 7fd007200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0000c1770 con 0x7fd00006d410 2024-02-21T23:06:39.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.341+0000 7fd005400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0000c43e0 0x7fd0001dfb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:39.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.341+0000 7fd005400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0000c43e0 0x7fd0001dfb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59318/0 (socket says 172.21.15.28:59318) 2024-02-21T23:06:39.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.341+0000 7fd005400640 1 -- 172.21.15.28:0/771776900 learned_addr learned my addr 172.21.15.28:0/771776900 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:39.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.342+0000 7fd005e00640 1 --2- 172.21.15.28:0/771776900 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd00006d410 0x7fd000145e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:39.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.342+0000 7fd006800640 1 --2- 172.21.15.28:0/771776900 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd000149830 0x7fd0000a9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:39.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.342+0000 7fd005400640 1 -- 172.21.15.28:0/771776900 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd00006d410 msgr2=0x7fd000145e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:39.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.342+0000 7fd005400640 1 --2- 172.21.15.28:0/771776900 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd00006d410 0x7fd000145e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:39.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.342+0000 7fd005400640 1 -- 172.21.15.28:0/771776900 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd000149830 msgr2=0x7fd0000a9620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:39.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.342+0000 7fd005400640 1 --2- 172.21.15.28:0/771776900 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd000149830 0x7fd0000a9620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:39.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.342+0000 7fd005400640 1 -- 172.21.15.28:0/771776900 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0000aa210 con 0x7fd0000c43e0 2024-02-21T23:06:39.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.342+0000 7fd005400640 1 --2- 172.21.15.28:0/771776900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0000c43e0 0x7fd0001dfb50 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fcff000c400 tx=0x7fcff000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:39.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.342+0000 7fcffa000640 1 -- 172.21.15.28:0/771776900 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff0011790 con 0x7fd0000c43e0 2024-02-21T23:06:39.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.342+0000 7fcffa000640 1 -- 172.21.15.28:0/771776900 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcff0017070 con 0x7fd0000c43e0 2024-02-21T23:06:39.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.342+0000 7fcffa000640 1 -- 172.21.15.28:0/771776900 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff0011bb0 con 0x7fd0000c43e0 2024-02-21T23:06:39.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.342+0000 7fd007200640 1 -- 172.21.15.28:0/771776900 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0001475e0 con 0x7fd0000c43e0 2024-02-21T23:06:39.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.343+0000 7fd007200640 1 -- 172.21.15.28:0/771776900 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0001e0090 con 0x7fd0000c43e0 2024-02-21T23:06:39.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.344+0000 7fd007200640 1 -- 172.21.15.28:0/771776900 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0000c73f0 con 0x7fd0000c43e0 2024-02-21T23:06:39.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.345+0000 7fcffa000640 1 -- 172.21.15.28:0/771776900 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcff001c460 con 0x7fd0000c43e0 2024-02-21T23:06:39.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.345+0000 7fcffa000640 1 --2- 172.21.15.28:0/771776900 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcfe406b4d0 0x7fcfe406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:39.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.346+0000 7fcffa000640 1 -- 172.21.15.28:0/771776900 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6479+0+0 (secure 0 0 0) 0x7fcff008a3c0 con 0x7fd0000c43e0 2024-02-21T23:06:39.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.346+0000 7fd005e00640 1 --2- 172.21.15.28:0/771776900 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcfe406b4d0 0x7fcfe406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:39.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.347+0000 7fd005e00640 1 --2- 172.21.15.28:0/771776900 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcfe406b4d0 0x7fcfe406d990 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fcff4006fd0 tx=0x7fcff4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:39.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.350+0000 7fcffa000640 1 -- 172.21.15.28:0/771776900 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd0000c73f0 con 0x7fd0000c43e0 2024-02-21T23:06:39.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.592+0000 7fd007200640 1 -- 172.21.15.28:0/771776900 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs2"} v 0) v1 -- 0x7fd00004f290 con 0x7fd0000c43e0 2024-02-21T23:06:39.582 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-02-21T23:06:39.593+0000 7f5ddc400640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-21T23:06:39.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.596+0000 7fcffa000640 1 -- 172.21.15.28:0/771776900 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs2"}]=0 cephfs2 marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v80) v1 ==== 163+0+0 (secure 0 0 0) 0x7fcff0058f30 con 0x7fd0000c43e0 2024-02-21T23:06:39.586 INFO:teuthology.orchestra.run.smithi028.stderr:cephfs2 marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-02-21T23:06:39.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.601+0000 7fd007200640 1 -- 172.21.15.28:0/771776900 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcfe406b4d0 msgr2=0x7fcfe406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:39.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.601+0000 7fd007200640 1 --2- 172.21.15.28:0/771776900 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcfe406b4d0 0x7fcfe406d990 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fcff4006fd0 tx=0x7fcff4008040 comp rx=0 tx=0).stop 2024-02-21T23:06:39.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.601+0000 7fd007200640 1 -- 172.21.15.28:0/771776900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0000c43e0 msgr2=0x7fd0001dfb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:39.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.601+0000 7fd007200640 1 --2- 172.21.15.28:0/771776900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0000c43e0 0x7fd0001dfb50 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fcff000c400 tx=0x7fcff000c8c0 comp rx=0 tx=0).stop 2024-02-21T23:06:39.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.601+0000 7fd007200640 1 -- 172.21.15.28:0/771776900 shutdown_connections 2024-02-21T23:06:39.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.601+0000 7fd007200640 1 --2- 172.21.15.28:0/771776900 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcfe406b4d0 0x7fcfe406d990 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:39.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.601+0000 7fd007200640 1 --2- 172.21.15.28:0/771776900 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd000149830 0x7fd0000a9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:39.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.601+0000 7fd007200640 1 --2- 172.21.15.28:0/771776900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0000c43e0 0x7fd0001dfb50 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:39.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.601+0000 7fd007200640 1 --2- 172.21.15.28:0/771776900 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd00006d410 0x7fd000145e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:39.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.601+0000 7fd007200640 1 -- 172.21.15.28:0/771776900 >> 172.21.15.28:0/771776900 conn(0x7fd0000a8320 msgr2=0x7fd0000a7230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:39.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.602+0000 7fd007200640 1 -- 172.21.15.28:0/771776900 shutdown_connections 2024-02-21T23:06:39.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.602+0000 7fd007200640 1 -- 172.21.15.28:0/771776900 wait complete. 2024-02-21T23:06:39.608 INFO:tasks.ceph.mds.d.smithi081.stderr: -89> 2024-02-21T23:06:37.337+0000 7f56fca00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-21T23:06:39.612 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs2 --yes-i-really-mean-it 2024-02-21T23:06:39.680 INFO:tasks.ceph.mds.d.smithi081.stdout:starting mds.d at 2024-02-21T23:06:39.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.838+0000 7f3fa7200640 1 Processor -- start 2024-02-21T23:06:39.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.838+0000 7f3fa7200640 1 -- start start 2024-02-21T23:06:39.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.840+0000 7f3fa7200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3fa8152360 0x7f3fa80c5800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:39.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.840+0000 7f3fa7200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f3fa8154e60 0x7f3fa81552d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:39.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.840+0000 7f3fa7200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fa80c1ee0 con 0x7f3fa80c48a0 2024-02-21T23:06:39.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.840+0000 7f3fa7200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fa80c21c0 con 0x7f3fa8154e60 2024-02-21T23:06:39.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.840+0000 7f3fa7200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fa80c24a0 con 0x7f3fa8152360 2024-02-21T23:06:39.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.840+0000 7f3f9d400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f3fa80c48a0 0x7f3fa806ae70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59550/0 (socket says 172.21.15.28:59550) 2024-02-21T23:06:39.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.840+0000 7f3f9d400640 1 -- 172.21.15.28:0/3488277368 learned_addr learned my addr 172.21.15.28:0/3488277368 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:06:39.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.841+0000 7f3fa5e00640 1 --2- 172.21.15.28:0/3488277368 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3fa8152360 0x7f3fa80c5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:39.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.841+0000 7f3fa6800640 1 --2- 172.21.15.28:0/3488277368 >> v2:172.21.15.81:3300/0 conn(0x7f3fa8154e60 0x7f3fa81552d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:39.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.841+0000 7f3fa5400640 1 -- 172.21.15.28:0/3488277368 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1334542143 0 0) 0x7f3fa80c1ee0 con 0x7f3fa80c48a0 2024-02-21T23:06:39.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.841+0000 7f3fa5400640 1 -- 172.21.15.28:0/3488277368 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f8c003520 con 0x7f3fa80c48a0 2024-02-21T23:06:39.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.841+0000 7f3fa5400640 1 -- 172.21.15.28:0/3488277368 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2155923278 0 0) 0x7f3f8c003520 con 0x7f3fa80c48a0 2024-02-21T23:06:39.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.841+0000 7f3fa5400640 1 -- 172.21.15.28:0/3488277368 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3fa8152360 msgr2=0x7f3fa80c5800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:39.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.841+0000 7f3fa5400640 1 --2- 172.21.15.28:0/3488277368 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3fa8152360 0x7f3fa80c5800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:39.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.841+0000 7f3fa5400640 1 -- 172.21.15.28:0/3488277368 >> v2:172.21.15.81:3300/0 conn(0x7f3fa8154e60 msgr2=0x7f3fa81552d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:39.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.841+0000 7f3fa5400640 1 --2- 172.21.15.28:0/3488277368 >> v2:172.21.15.81:3300/0 conn(0x7f3fa8154e60 0x7f3fa81552d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:39.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.841+0000 7f3fa5400640 1 -- 172.21.15.28:0/3488277368 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fa80c2780 con 0x7f3fa80c48a0 2024-02-21T23:06:39.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.841+0000 7f3fa5400640 1 -- 172.21.15.28:0/3488277368 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f3f90001690 con 0x7f3fa80c48a0 2024-02-21T23:06:39.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.841+0000 7f3fa5400640 1 -- 172.21.15.28:0/3488277368 >> v1:172.21.15.28:6789/0 conn(0x7f3fa80c48a0 legacy=0x7f3fa806ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:39.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.841+0000 7f3fa5400640 1 --2- 172.21.15.28:0/3488277368 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f8c0039b0 0x7f3f8c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:39.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.841+0000 7f3fa5400640 1 --2- 172.21.15.28:0/3488277368 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3fa8152360 0x7f3f8c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:39.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.841+0000 7f3fa5400640 1 --2- 172.21.15.28:0/3488277368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f8c004d40 0x7f3f8c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:39.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.841+0000 7f3fa5400640 1 -- 172.21.15.28:0/3488277368 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fa80c21c0 con 0x7f3f8c004d40 2024-02-21T23:06:39.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.841+0000 7f3fa5400640 1 -- 172.21.15.28:0/3488277368 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fa80c24a0 con 0x7f3f8c0039b0 2024-02-21T23:06:39.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.841+0000 7f3fa5400640 1 -- 172.21.15.28:0/3488277368 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f8c00b9c0 con 0x7f3fa8152360 2024-02-21T23:06:39.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.842+0000 7f3fa6800640 1 --2- 172.21.15.28:0/3488277368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f8c004d40 0x7f3f8c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:39.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.842+0000 7f3fa5e00640 1 --2- 172.21.15.28:0/3488277368 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f8c0039b0 0x7f3f8c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:39.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.842+0000 7f3f9d400640 1 --2- 172.21.15.28:0/3488277368 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3fa8152360 0x7f3f8c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:39.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.842+0000 7f3fa6800640 1 -- 172.21.15.28:0/3488277368 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3fa8152360 msgr2=0x7f3f8c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:39.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.842+0000 7f3fa6800640 1 --2- 172.21.15.28:0/3488277368 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3fa8152360 0x7f3f8c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:39.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.842+0000 7f3fa6800640 1 -- 172.21.15.28:0/3488277368 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f8c0039b0 msgr2=0x7f3f8c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:39.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.842+0000 7f3fa6800640 1 --2- 172.21.15.28:0/3488277368 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f8c0039b0 0x7f3f8c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:39.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.842+0000 7f3fa6800640 1 -- 172.21.15.28:0/3488277368 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f8c003520 con 0x7f3f8c004d40 2024-02-21T23:06:39.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.843+0000 7f3fa6800640 1 --2- 172.21.15.28:0/3488277368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f8c004d40 0x7f3f8c0051c0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f3fa0009fc0 tx=0x7f3f8c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=948ad02c78a95646 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:39.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.844+0000 7f3fa5400640 1 -- 172.21.15.28:0/3488277368 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fa0039070 con 0x7f3f8c004d40 2024-02-21T23:06:39.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.844+0000 7f3fa5400640 1 -- 172.21.15.28:0/3488277368 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fa002cc60 con 0x7f3f8c004d40 2024-02-21T23:06:39.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.844+0000 7f3fa5400640 1 -- 172.21.15.28:0/3488277368 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fa0009c10 con 0x7f3f8c004d40 2024-02-21T23:06:39.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.844+0000 7f3fa7200640 1 -- 172.21.15.28:0/3488277368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f8c004d40 msgr2=0x7f3f8c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:39.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.844+0000 7f3fa7200640 1 --2- 172.21.15.28:0/3488277368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f8c004d40 0x7f3f8c0051c0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f3fa0009fc0 tx=0x7f3f8c005830 comp rx=0 tx=0).stop 2024-02-21T23:06:39.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.844+0000 7f3fa6800640 1 -- 172.21.15.28:0/3488277368 reap_dead start 2024-02-21T23:06:39.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.845+0000 7f3fa7200640 1 -- 172.21.15.28:0/3488277368 shutdown_connections 2024-02-21T23:06:39.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.845+0000 7f3fa7200640 1 -- 172.21.15.28:0/3488277368 >> 172.21.15.28:0/3488277368 conn(0x7f3fa80a8530 msgr2=0x7f3fa80a8910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:39.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.845+0000 7f3fa7200640 1 -- 172.21.15.28:0/3488277368 shutdown_connections 2024-02-21T23:06:39.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.845+0000 7f3fa7200640 1 -- 172.21.15.28:0/3488277368 wait complete. 2024-02-21T23:06:39.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.847+0000 7f3fa7200640 1 Processor -- start 2024-02-21T23:06:39.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.847+0000 7f3fa7200640 1 -- start start 2024-02-21T23:06:39.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.848+0000 7f3fa7200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f8c004d40 0x7f3fa81ee720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:39.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.848+0000 7f3fa7200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3fa8158e50 0x7f3fa80a7a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:39.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.848+0000 7f3fa7200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3fa81ea810 0x7f3fa8147740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:39.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.848+0000 7f3fa7200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fa806b580 con 0x7f3f8c004d40 2024-02-21T23:06:39.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.848+0000 7f3fa7200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fa80c1b70 con 0x7f3fa8158e50 2024-02-21T23:06:39.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.848+0000 7f3fa7200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fa806b700 con 0x7f3fa81ea810 2024-02-21T23:06:39.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.848+0000 7f3fa5e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f8c004d40 0x7f3fa81ee720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:39.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.848+0000 7f3fa5e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f8c004d40 0x7f3fa81ee720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59322/0 (socket says 172.21.15.28:59322) 2024-02-21T23:06:39.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.848+0000 7f3fa5e00640 1 -- 172.21.15.28:0/1091866396 learned_addr learned my addr 172.21.15.28:0/1091866396 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:39.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.849+0000 7f3fa6800640 1 --2- 172.21.15.28:0/1091866396 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3fa81ea810 0x7f3fa8147740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:39.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.849+0000 7f3f9d400640 1 --2- 172.21.15.28:0/1091866396 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3fa8158e50 0x7f3fa80a7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:39.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.849+0000 7f3fa5e00640 1 -- 172.21.15.28:0/1091866396 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3fa81ea810 msgr2=0x7f3fa8147740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:39.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.849+0000 7f3fa5e00640 1 --2- 172.21.15.28:0/1091866396 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3fa81ea810 0x7f3fa8147740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:39.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.849+0000 7f3fa5e00640 1 -- 172.21.15.28:0/1091866396 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3fa8158e50 msgr2=0x7f3fa80a7a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:39.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.849+0000 7f3fa5e00640 1 --2- 172.21.15.28:0/1091866396 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3fa8158e50 0x7f3fa80a7a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:39.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.849+0000 7f3fa5e00640 1 -- 172.21.15.28:0/1091866396 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fa8071da0 con 0x7f3f8c004d40 2024-02-21T23:06:39.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.849+0000 7f3f9d400640 1 --2- 172.21.15.28:0/1091866396 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3fa8158e50 0x7f3fa80a7a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:06:39.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.849+0000 7f3fa6800640 1 --2- 172.21.15.28:0/1091866396 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3fa81ea810 0x7f3fa8147740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:06:39.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.849+0000 7f3fa5e00640 1 --2- 172.21.15.28:0/1091866396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f8c004d40 0x7f3fa81ee720 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f3f98011540 tx=0x7f3f98011a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:39.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.849+0000 7f3f9ea00640 1 -- 172.21.15.28:0/1091866396 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f98015930 con 0x7f3f8c004d40 2024-02-21T23:06:39.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.849+0000 7f3f9ea00640 1 -- 172.21.15.28:0/1091866396 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f98015ad0 con 0x7f3f8c004d40 2024-02-21T23:06:39.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.849+0000 7f3fa7200640 1 -- 172.21.15.28:0/1091866396 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fa81eec60 con 0x7f3f8c004d40 2024-02-21T23:06:39.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.849+0000 7f3f9ea00640 1 -- 172.21.15.28:0/1091866396 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f9801d400 con 0x7f3f8c004d40 2024-02-21T23:06:39.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.849+0000 7f3fa7200640 1 -- 172.21.15.28:0/1091866396 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fa8072320 con 0x7f3f8c004d40 2024-02-21T23:06:39.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.851+0000 7f3fa7200640 1 -- 172.21.15.28:0/1091866396 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fa8072560 con 0x7f3f8c004d40 2024-02-21T23:06:39.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.852+0000 7f3f9ea00640 1 -- 172.21.15.28:0/1091866396 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3f9801d5a0 con 0x7f3f8c004d40 2024-02-21T23:06:39.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.852+0000 7f3f9ea00640 1 --2- 172.21.15.28:0/1091866396 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3f8406b500 0x7f3f8406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:39.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.853+0000 7f3f9ea00640 1 -- 172.21.15.28:0/1091866396 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6565+0+0 (secure 0 0 0) 0x7f3f9808ae40 con 0x7f3f8c004d40 2024-02-21T23:06:39.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.853+0000 7f3f9d400640 1 --2- 172.21.15.28:0/1091866396 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3f8406b500 0x7f3f8406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:39.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.854+0000 7f3f9d400640 1 --2- 172.21.15.28:0/1091866396 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3f8406b500 0x7f3f8406d9c0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f3f90001050 tx=0x7f3f90000fe0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:39.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:39.857+0000 7f3f9ea00640 1 -- 172.21.15.28:0/1091866396 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3f98019030 con 0x7f3f8c004d40 2024-02-21T23:06:40.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.099+0000 7f3fa7200640 1 -- 172.21.15.28:0/1091866396 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs2", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f3fa804f290 con 0x7f3f8c004d40 2024-02-21T23:06:40.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.600+0000 7f3f9ea00640 1 -- 172.21.15.28:0/1091866396 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs2", "yes_i_really_mean_it": true}]=0 v81) v1 ==== 105+0+0 (secure 0 0 0) 0x7f3f980598c0 con 0x7f3f8c004d40 2024-02-21T23:06:40.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.603+0000 7f3fa7200640 1 -- 172.21.15.28:0/1091866396 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3f8406b500 msgr2=0x7f3f8406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:40.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.603+0000 7f3fa7200640 1 --2- 172.21.15.28:0/1091866396 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3f8406b500 0x7f3f8406d9c0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f3f90001050 tx=0x7f3f90000fe0 comp rx=0 tx=0).stop 2024-02-21T23:06:40.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.603+0000 7f3fa7200640 1 -- 172.21.15.28:0/1091866396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f8c004d40 msgr2=0x7f3fa81ee720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:40.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.603+0000 7f3fa7200640 1 --2- 172.21.15.28:0/1091866396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f8c004d40 0x7f3fa81ee720 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f3f98011540 tx=0x7f3f98011a00 comp rx=0 tx=0).stop 2024-02-21T23:06:40.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.603+0000 7f3fa7200640 1 -- 172.21.15.28:0/1091866396 shutdown_connections 2024-02-21T23:06:40.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.603+0000 7f3fa7200640 1 --2- 172.21.15.28:0/1091866396 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3f8406b500 0x7f3f8406d9c0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:40.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.603+0000 7f3fa7200640 1 --2- 172.21.15.28:0/1091866396 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3fa81ea810 0x7f3fa8147740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:40.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.603+0000 7f3fa7200640 1 --2- 172.21.15.28:0/1091866396 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3fa8158e50 0x7f3fa80a7a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:40.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.603+0000 7f3fa7200640 1 --2- 172.21.15.28:0/1091866396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f8c004d40 0x7f3fa81ee720 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:40.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.603+0000 7f3fa7200640 1 -- 172.21.15.28:0/1091866396 >> 172.21.15.28:0/1091866396 conn(0x7f3fa80a8530 msgr2=0x7f3fa814beb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:40.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.603+0000 7f3fa7200640 1 -- 172.21.15.28:0/1091866396 shutdown_connections 2024-02-21T23:06:40.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.603+0000 7f3fa7200640 1 -- 172.21.15.28:0/1091866396 wait complete. 2024-02-21T23:06:40.616 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs2_metadata cephfs2_metadata --yes-i-really-really-mean-it 2024-02-21T23:06:40.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.855+0000 7fd412200640 1 Processor -- start 2024-02-21T23:06:40.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.856+0000 7fd412200640 1 -- start start 2024-02-21T23:06:40.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.857+0000 7fd412200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd40c06cfa0 0x7fd40c0ca540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:40.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.857+0000 7fd412200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fd40c0c6290 0x7fd40c1522c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:40.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.857+0000 7fd412200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd40c0caf40 con 0x7fd40c158460 2024-02-21T23:06:40.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.857+0000 7fd412200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd40c0c8ae0 con 0x7fd40c06cfa0 2024-02-21T23:06:40.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.857+0000 7fd412200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd40c0c87b0 con 0x7fd40c0c6290 2024-02-21T23:06:40.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.858+0000 7fd410e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd40c06cfa0 0x7fd40c0ca540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:40.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.858+0000 7fd410e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd40c06cfa0 0x7fd40c0ca540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59330/0 (socket says 172.21.15.28:59330) 2024-02-21T23:06:40.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.858+0000 7fd410e00640 1 -- 172.21.15.28:0/2067414263 learned_addr learned my addr 172.21.15.28:0/2067414263 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:40.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.858+0000 7fd411800640 1 --2- 172.21.15.28:0/2067414263 >> v2:172.21.15.81:3300/0 conn(0x7fd40c0c6290 0x7fd40c1522c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:40.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.858+0000 7fd410e00640 1 -- 172.21.15.28:0/2067414263 >> v2:172.21.15.81:3300/0 conn(0x7fd40c0c6290 msgr2=0x7fd40c1522c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:40.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.858+0000 7fd410e00640 1 --2- 172.21.15.28:0/2067414263 >> v2:172.21.15.81:3300/0 conn(0x7fd40c0c6290 0x7fd40c1522c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:40.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.858+0000 7fd410e00640 1 -- 172.21.15.28:0/2067414263 >> v1:172.21.15.81:6789/0 conn(0x7fd40c158460 legacy=0x7fd40c0c8f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:40.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.858+0000 7fd410e00640 1 -- 172.21.15.28:0/2067414263 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd40c1588f0 con 0x7fd40c06cfa0 2024-02-21T23:06:40.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.859+0000 7fd410e00640 1 --2- 172.21.15.28:0/2067414263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd40c06cfa0 0x7fd40c0ca540 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fd40c06c8a0 tx=0x7fd3f402b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c7694db28741b45 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:40.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.860+0000 7fd40b400640 1 -- 172.21.15.28:0/2067414263 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3f4038070 con 0x7fd40c06cfa0 2024-02-21T23:06:40.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.860+0000 7fd40b400640 1 -- 172.21.15.28:0/2067414263 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3f402f050 con 0x7fd40c06cfa0 2024-02-21T23:06:40.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.860+0000 7fd40b400640 1 -- 172.21.15.28:0/2067414263 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3f4031970 con 0x7fd40c06cfa0 2024-02-21T23:06:40.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.860+0000 7fd412200640 1 -- 172.21.15.28:0/2067414263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd40c06cfa0 msgr2=0x7fd40c0ca540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:40.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.860+0000 7fd412200640 1 --2- 172.21.15.28:0/2067414263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd40c06cfa0 0x7fd40c0ca540 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fd40c06c8a0 tx=0x7fd3f402b7f0 comp rx=0 tx=0).stop 2024-02-21T23:06:40.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.861+0000 7fd412200640 1 -- 172.21.15.28:0/2067414263 shutdown_connections 2024-02-21T23:06:40.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.861+0000 7fd412200640 1 --2- 172.21.15.28:0/2067414263 >> v2:172.21.15.81:3300/0 conn(0x7fd40c0c6290 0x7fd40c1522c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:40.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.861+0000 7fd412200640 1 --2- 172.21.15.28:0/2067414263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd40c06cfa0 0x7fd40c0ca540 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:40.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.861+0000 7fd412200640 1 -- 172.21.15.28:0/2067414263 >> 172.21.15.28:0/2067414263 conn(0x7fd40c0725a0 msgr2=0x7fd40c0729b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:40.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.861+0000 7fd412200640 1 -- 172.21.15.28:0/2067414263 shutdown_connections 2024-02-21T23:06:40.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.861+0000 7fd412200640 1 -- 172.21.15.28:0/2067414263 wait complete. 2024-02-21T23:06:40.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.862+0000 7fd412200640 1 Processor -- start 2024-02-21T23:06:40.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.862+0000 7fd412200640 1 -- start start 2024-02-21T23:06:40.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.863+0000 7fd412200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd40c06cfa0 0x7fd40c14d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:40.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.863+0000 7fd412200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd40c0c6290 0x7fd40c0af0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:40.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.863+0000 7fd412200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd40c158460 0x7fd40c1ed9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:40.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.863+0000 7fd412200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd40c0c6c90 con 0x7fd40c158460 2024-02-21T23:06:40.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.863+0000 7fd412200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd40c0c69b0 con 0x7fd40c0c6290 2024-02-21T23:06:40.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.863+0000 7fd412200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd40c0c66a0 con 0x7fd40c06cfa0 2024-02-21T23:06:40.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.864+0000 7fd411800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd40c158460 0x7fd40c1ed9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:40.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.864+0000 7fd411800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd40c158460 0x7fd40c1ed9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59344/0 (socket says 172.21.15.28:59344) 2024-02-21T23:06:40.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.864+0000 7fd411800640 1 -- 172.21.15.28:0/3707088612 learned_addr learned my addr 172.21.15.28:0/3707088612 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:40.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.864+0000 7fd410e00640 1 --2- 172.21.15.28:0/3707088612 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd40c06cfa0 0x7fd40c14d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:40.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.864+0000 7fd40be00640 1 --2- 172.21.15.28:0/3707088612 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd40c0c6290 0x7fd40c0af0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:40.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.864+0000 7fd411800640 1 -- 172.21.15.28:0/3707088612 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd40c06cfa0 msgr2=0x7fd40c14d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:40.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.864+0000 7fd411800640 1 --2- 172.21.15.28:0/3707088612 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd40c06cfa0 0x7fd40c14d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:40.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.864+0000 7fd411800640 1 -- 172.21.15.28:0/3707088612 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd40c0c6290 msgr2=0x7fd40c0af0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:40.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.864+0000 7fd411800640 1 --2- 172.21.15.28:0/3707088612 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd40c0c6290 0x7fd40c0af0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:40.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.864+0000 7fd411800640 1 -- 172.21.15.28:0/3707088612 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd40c14af20 con 0x7fd40c158460 2024-02-21T23:06:40.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.864+0000 7fd410e00640 1 --2- 172.21.15.28:0/3707088612 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd40c06cfa0 0x7fd40c14d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:06:40.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.865+0000 7fd411800640 1 --2- 172.21.15.28:0/3707088612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd40c158460 0x7fd40c1ed9b0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fd404007e60 tx=0x7fd40400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:40.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.865+0000 7fd408c00640 1 -- 172.21.15.28:0/3707088612 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd404014420 con 0x7fd40c158460 2024-02-21T23:06:40.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.865+0000 7fd408c00640 1 -- 172.21.15.28:0/3707088612 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd404019070 con 0x7fd40c158460 2024-02-21T23:06:40.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.865+0000 7fd408c00640 1 -- 172.21.15.28:0/3707088612 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd404014820 con 0x7fd40c158460 2024-02-21T23:06:40.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.865+0000 7fd412200640 1 -- 172.21.15.28:0/3707088612 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd40c14ac40 con 0x7fd40c158460 2024-02-21T23:06:40.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.865+0000 7fd412200640 1 -- 172.21.15.28:0/3707088612 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd40c0ad160 con 0x7fd40c158460 2024-02-21T23:06:40.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.866+0000 7fd412200640 1 -- 172.21.15.28:0/3707088612 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd40c04f290 con 0x7fd40c158460 2024-02-21T23:06:40.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.867+0000 7fd408c00640 1 -- 172.21.15.28:0/3707088612 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd4040149c0 con 0x7fd40c158460 2024-02-21T23:06:40.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.869+0000 7fd408c00640 1 --2- 172.21.15.28:0/3707088612 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd3e806b4d0 0x7fd3e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:40.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.869+0000 7fd408c00640 1 -- 172.21.15.28:0/3707088612 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6565+0+0 (secure 0 0 0) 0x7fd40408add0 con 0x7fd40c158460 2024-02-21T23:06:40.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.869+0000 7fd410e00640 1 --2- 172.21.15.28:0/3707088612 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd3e806b4d0 0x7fd3e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:40.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.870+0000 7fd410e00640 1 --2- 172.21.15.28:0/3707088612 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd3e806b4d0 0x7fd3e806d990 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fd3f4035000 tx=0x7fd3f4008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:40.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:40.873+0000 7fd408c00640 1 -- 172.21.15.28:0/3707088612 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd4040598e0 con 0x7fd40c158460 2024-02-21T23:06:41.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.078+0000 7fd412200640 1 -- 172.21.15.28:0/3707088612 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs2_metadata", "pool2": "cephfs2_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd40c0af5e0 con 0x7fd40c158460 2024-02-21T23:06:41.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.609+0000 7fd408c00640 1 -- 172.21.15.28:0/3707088612 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs2_metadata", "pool2": "cephfs2_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs2_metadata' removed v89) v1 ==== 184+0+0 (secure 0 0 0) 0x7fd4040591e0 con 0x7fd40c158460 2024-02-21T23:06:41.599 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs2_metadata' removed 2024-02-21T23:06:41.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.615+0000 7fd412200640 1 -- 172.21.15.28:0/3707088612 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd3e806b4d0 msgr2=0x7fd3e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:41.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.615+0000 7fd412200640 1 --2- 172.21.15.28:0/3707088612 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd3e806b4d0 0x7fd3e806d990 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fd3f4035000 tx=0x7fd3f4008030 comp rx=0 tx=0).stop 2024-02-21T23:06:41.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.615+0000 7fd412200640 1 -- 172.21.15.28:0/3707088612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd40c158460 msgr2=0x7fd40c1ed9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:41.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.615+0000 7fd412200640 1 --2- 172.21.15.28:0/3707088612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd40c158460 0x7fd40c1ed9b0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fd404007e60 tx=0x7fd40400e4b0 comp rx=0 tx=0).stop 2024-02-21T23:06:41.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.616+0000 7fd412200640 1 -- 172.21.15.28:0/3707088612 shutdown_connections 2024-02-21T23:06:41.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.616+0000 7fd412200640 1 --2- 172.21.15.28:0/3707088612 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd3e806b4d0 0x7fd3e806d990 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:41.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.616+0000 7fd412200640 1 --2- 172.21.15.28:0/3707088612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd40c158460 0x7fd40c1ed9b0 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:41.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.616+0000 7fd412200640 1 --2- 172.21.15.28:0/3707088612 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd40c0c6290 0x7fd40c0af0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:41.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.616+0000 7fd412200640 1 --2- 172.21.15.28:0/3707088612 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd40c06cfa0 0x7fd40c14d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:41.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.616+0000 7fd412200640 1 -- 172.21.15.28:0/3707088612 >> 172.21.15.28:0/3707088612 conn(0x7fd40c0725a0 msgr2=0x7fd40c072980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:41.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.617+0000 7fd412200640 1 -- 172.21.15.28:0/3707088612 shutdown_connections 2024-02-21T23:06:41.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.617+0000 7fd412200640 1 -- 172.21.15.28:0/3707088612 wait complete. 2024-02-21T23:06:41.631 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs2_data cephfs2_data --yes-i-really-really-mean-it 2024-02-21T23:06:41.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.858+0000 7fd868200640 1 Processor -- start 2024-02-21T23:06:41.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.858+0000 7fd868200640 1 -- start start 2024-02-21T23:06:41.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.860+0000 7fd868200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd86006e780 0x7fd86006eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:41.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.860+0000 7fd868200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd8600a5fc0 0x7fd8600c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:41.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.860+0000 7fd868200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fd8600c50c0 0x7fd8601564d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:41.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.860+0000 7fd868200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8601543d0 con 0x7fd8600a5fc0 2024-02-21T23:06:41.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.860+0000 7fd868200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8600c5ad0 con 0x7fd8600c50c0 2024-02-21T23:06:41.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.860+0000 7fd868200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8600c46f0 con 0x7fd86006e780 2024-02-21T23:06:41.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.860+0000 7fd864e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd8600a5fc0 0x7fd8600c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:41.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.860+0000 7fd864e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd8600a5fc0 0x7fd8600c4b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59354/0 (socket says 172.21.15.28:59354) 2024-02-21T23:06:41.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.860+0000 7fd864e00640 1 -- 172.21.15.28:0/747974817 learned_addr learned my addr 172.21.15.28:0/747974817 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:41.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.861+0000 7fd865800640 1 --2- 172.21.15.28:0/747974817 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd86006e780 0x7fd86006eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:41.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.861+0000 7fd866200640 1 --2- 172.21.15.28:0/747974817 >> v2:172.21.15.81:3300/0 conn(0x7fd8600c50c0 0x7fd8601564d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:41.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.861+0000 7fd864e00640 1 -- 172.21.15.28:0/747974817 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd86006e780 msgr2=0x7fd86006eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:41.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.861+0000 7fd864e00640 1 --2- 172.21.15.28:0/747974817 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd86006e780 0x7fd86006eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:41.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.861+0000 7fd864e00640 1 -- 172.21.15.28:0/747974817 >> v2:172.21.15.81:3300/0 conn(0x7fd8600c50c0 msgr2=0x7fd8601564d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:41.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.861+0000 7fd864e00640 1 --2- 172.21.15.28:0/747974817 >> v2:172.21.15.81:3300/0 conn(0x7fd8600c50c0 0x7fd8601564d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:41.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.861+0000 7fd864e00640 1 -- 172.21.15.28:0/747974817 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd860153cb0 con 0x7fd8600a5fc0 2024-02-21T23:06:41.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.862+0000 7fd864e00640 1 --2- 172.21.15.28:0/747974817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8600a5fc0 0x7fd8600c4b80 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fd8600c74d0 tx=0x7fd84c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9992be367bc34853 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:41.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.862+0000 7fd85fe00640 1 -- 172.21.15.28:0/747974817 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd84c037070 con 0x7fd8600a5fc0 2024-02-21T23:06:41.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.863+0000 7fd85fe00640 1 -- 172.21.15.28:0/747974817 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd84c004230 con 0x7fd8600a5fc0 2024-02-21T23:06:41.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.863+0000 7fd85fe00640 1 -- 172.21.15.28:0/747974817 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd84c003610 con 0x7fd8600a5fc0 2024-02-21T23:06:41.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.863+0000 7fd868200640 1 -- 172.21.15.28:0/747974817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8600a5fc0 msgr2=0x7fd8600c4b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:41.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.863+0000 7fd868200640 1 --2- 172.21.15.28:0/747974817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8600a5fc0 0x7fd8600c4b80 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fd8600c74d0 tx=0x7fd84c02b860 comp rx=0 tx=0).stop 2024-02-21T23:06:41.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.863+0000 7fd868200640 1 -- 172.21.15.28:0/747974817 shutdown_connections 2024-02-21T23:06:41.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.863+0000 7fd868200640 1 --2- 172.21.15.28:0/747974817 >> v2:172.21.15.81:3300/0 conn(0x7fd8600c50c0 0x7fd8601564d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:41.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.863+0000 7fd868200640 1 --2- 172.21.15.28:0/747974817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8600a5fc0 0x7fd8600c4b80 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:41.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.863+0000 7fd868200640 1 --2- 172.21.15.28:0/747974817 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd86006e780 0x7fd86006eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:41.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.863+0000 7fd868200640 1 -- 172.21.15.28:0/747974817 >> 172.21.15.28:0/747974817 conn(0x7fd86006b830 msgr2=0x7fd8600c67f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:41.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.863+0000 7fd868200640 1 -- 172.21.15.28:0/747974817 shutdown_connections 2024-02-21T23:06:41.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.863+0000 7fd868200640 1 -- 172.21.15.28:0/747974817 wait complete. 2024-02-21T23:06:41.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.865+0000 7fd868200640 1 Processor -- start 2024-02-21T23:06:41.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.865+0000 7fd868200640 1 -- start start 2024-02-21T23:06:41.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.866+0000 7fd868200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd86006e780 0x7fd8600c0130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:41.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.866+0000 7fd868200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8600a5fc0 0x7fd86014b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:41.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.866+0000 7fd868200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd8600c50c0 0x7fd86014bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:41.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.866+0000 7fd868200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd860153960 con 0x7fd8600a5fc0 2024-02-21T23:06:41.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.866+0000 7fd868200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8600c6000 con 0x7fd8600c50c0 2024-02-21T23:06:41.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.866+0000 7fd868200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd86006d110 con 0x7fd86006e780 2024-02-21T23:06:41.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.866+0000 7fd864e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8600a5fc0 0x7fd86014b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:41.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.867+0000 7fd864e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8600a5fc0 0x7fd86014b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59362/0 (socket says 172.21.15.28:59362) 2024-02-21T23:06:41.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.867+0000 7fd864e00640 1 -- 172.21.15.28:0/31313558 learned_addr learned my addr 172.21.15.28:0/31313558 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:41.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.867+0000 7fd865800640 1 --2- 172.21.15.28:0/31313558 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd86006e780 0x7fd8600c0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:41.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.867+0000 7fd866200640 1 --2- 172.21.15.28:0/31313558 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd8600c50c0 0x7fd86014bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:41.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.867+0000 7fd864e00640 1 -- 172.21.15.28:0/31313558 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd86006e780 msgr2=0x7fd8600c0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:41.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.867+0000 7fd864e00640 1 --2- 172.21.15.28:0/31313558 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd86006e780 0x7fd8600c0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:41.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.867+0000 7fd864e00640 1 -- 172.21.15.28:0/31313558 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd8600c50c0 msgr2=0x7fd86014bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:41.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.867+0000 7fd864e00640 1 --2- 172.21.15.28:0/31313558 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd8600c50c0 0x7fd86014bec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:41.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.867+0000 7fd864e00640 1 -- 172.21.15.28:0/31313558 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd86014c400 con 0x7fd8600a5fc0 2024-02-21T23:06:41.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.868+0000 7fd864e00640 1 --2- 172.21.15.28:0/31313558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8600a5fc0 0x7fd86014b980 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fd84c035040 tx=0x7fd84c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:41.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.868+0000 7fd85d600640 1 -- 172.21.15.28:0/31313558 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd84c042020 con 0x7fd8600a5fc0 2024-02-21T23:06:41.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.868+0000 7fd85d600640 1 -- 172.21.15.28:0/31313558 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd84c003a40 con 0x7fd8600a5fc0 2024-02-21T23:06:41.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.868+0000 7fd85d600640 1 -- 172.21.15.28:0/31313558 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd84c037040 con 0x7fd8600a5fc0 2024-02-21T23:06:41.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.868+0000 7fd868200640 1 -- 172.21.15.28:0/31313558 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd860071ed0 con 0x7fd8600a5fc0 2024-02-21T23:06:41.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.868+0000 7fd868200640 1 -- 172.21.15.28:0/31313558 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd860072860 con 0x7fd8600a5fc0 2024-02-21T23:06:41.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.869+0000 7fd868200640 1 -- 172.21.15.28:0/31313558 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd860067f30 con 0x7fd8600a5fc0 2024-02-21T23:06:41.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.870+0000 7fd85d600640 1 -- 172.21.15.28:0/31313558 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd84c045020 con 0x7fd8600a5fc0 2024-02-21T23:06:41.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.871+0000 7fd85d600640 1 --2- 172.21.15.28:0/31313558 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd84006b4d0 0x7fd84006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:41.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.871+0000 7fd85d600640 1 -- 172.21.15.28:0/31313558 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6125+0+0 (secure 0 0 0) 0x7fd84c0ac1c0 con 0x7fd8600a5fc0 2024-02-21T23:06:41.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.872+0000 7fd865800640 1 --2- 172.21.15.28:0/31313558 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd84006b4d0 0x7fd84006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:41.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.873+0000 7fd865800640 1 --2- 172.21.15.28:0/31313558 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd84006b4d0 0x7fd84006d990 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fd854006fd0 tx=0x7fd854008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:41.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:41.876+0000 7fd85d600640 1 -- 172.21.15.28:0/31313558 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd84c04c030 con 0x7fd8600a5fc0 2024-02-21T23:06:42.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.077+0000 7fd868200640 1 -- 172.21.15.28:0/31313558 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs2_data", "pool2": "cephfs2_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd8600c58b0 con 0x7fd8600a5fc0 2024-02-21T23:06:42.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.612+0000 7fd85d600640 1 -- 172.21.15.28:0/31313558 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs2_data", "pool2": "cephfs2_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs2_data' removed v90) v1 ==== 172+0+0 (secure 0 0 0) 0x7fd84c07a840 con 0x7fd8600a5fc0 2024-02-21T23:06:42.602 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs2_data' removed 2024-02-21T23:06:42.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.618+0000 7fd868200640 1 -- 172.21.15.28:0/31313558 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd84006b4d0 msgr2=0x7fd84006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:42.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.618+0000 7fd868200640 1 --2- 172.21.15.28:0/31313558 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd84006b4d0 0x7fd84006d990 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fd854006fd0 tx=0x7fd854008040 comp rx=0 tx=0).stop 2024-02-21T23:06:42.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.618+0000 7fd868200640 1 -- 172.21.15.28:0/31313558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8600a5fc0 msgr2=0x7fd86014b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:42.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.618+0000 7fd868200640 1 --2- 172.21.15.28:0/31313558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8600a5fc0 0x7fd86014b980 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fd84c035040 tx=0x7fd84c0046c0 comp rx=0 tx=0).stop 2024-02-21T23:06:42.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.618+0000 7fd868200640 1 -- 172.21.15.28:0/31313558 shutdown_connections 2024-02-21T23:06:42.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.618+0000 7fd868200640 1 --2- 172.21.15.28:0/31313558 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd84006b4d0 0x7fd84006d990 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:42.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.618+0000 7fd868200640 1 --2- 172.21.15.28:0/31313558 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd8600c50c0 0x7fd86014bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:42.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.618+0000 7fd868200640 1 --2- 172.21.15.28:0/31313558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8600a5fc0 0x7fd86014b980 secure :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fd84c035040 tx=0x7fd84c0046c0 comp rx=0 tx=0).stop 2024-02-21T23:06:42.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.618+0000 7fd868200640 1 --2- 172.21.15.28:0/31313558 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd86006e780 0x7fd8600c0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:42.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.618+0000 7fd868200640 1 -- 172.21.15.28:0/31313558 >> 172.21.15.28:0/31313558 conn(0x7fd86006b830 msgr2=0x7fd8600c2f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:42.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.620+0000 7fd868200640 1 -- 172.21.15.28:0/31313558 shutdown_connections 2024-02-21T23:06:42.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.620+0000 7fd868200640 1 -- 172.21.15.28:0/31313558 wait complete. 2024-02-21T23:06:42.636 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.b mds_join_fs 2024-02-21T23:06:42.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.845+0000 7fd14a400640 1 Processor -- start 2024-02-21T23:06:42.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.845+0000 7fd14a400640 1 -- start start 2024-02-21T23:06:42.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.847+0000 7fd14a400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd14406f400 0x7fd14406e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:42.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.847+0000 7fd14a400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd14406db10 con 0x7fd1440c2c60 2024-02-21T23:06:42.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.847+0000 7fd14a400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1441539a0 con 0x7fd1441500f0 2024-02-21T23:06:42.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.847+0000 7fd14a400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1440c1080 con 0x7fd14406f400 2024-02-21T23:06:42.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.847+0000 7fd143e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd1440c2c60 0x7fd14406dff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59566/0 (socket says 172.21.15.28:59566) 2024-02-21T23:06:42.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.847+0000 7fd143e00640 1 -- 172.21.15.28:0/4005028402 learned_addr learned my addr 172.21.15.28:0/4005028402 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:06:42.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.847+0000 7fd149a00640 1 --2- 172.21.15.28:0/4005028402 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd14406f400 0x7fd14406e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:42.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.847+0000 7fd143400640 1 -- 172.21.15.28:0/4005028402 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1095369559 0 0) 0x7fd14406db10 con 0x7fd1440c2c60 2024-02-21T23:06:42.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.847+0000 7fd143400640 1 -- 172.21.15.28:0/4005028402 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd12c003520 con 0x7fd1440c2c60 2024-02-21T23:06:42.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.848+0000 7fd143400640 1 -- 172.21.15.28:0/4005028402 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1485847693 0 0) 0x7fd12c003520 con 0x7fd1440c2c60 2024-02-21T23:06:42.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.848+0000 7fd143400640 1 -- 172.21.15.28:0/4005028402 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd14406f400 msgr2=0x7fd14406e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:42.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.848+0000 7fd143400640 1 --2- 172.21.15.28:0/4005028402 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd14406f400 0x7fd14406e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:42.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.848+0000 7fd143400640 1 -- 172.21.15.28:0/4005028402 >> v1:172.21.15.81:6789/0 conn(0x7fd1441500f0 legacy=0x7fd14406ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:42.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.848+0000 7fd143400640 1 -- 172.21.15.28:0/4005028402 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1440c12d0 con 0x7fd1440c2c60 2024-02-21T23:06:42.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.848+0000 7fd143400640 1 -- 172.21.15.28:0/4005028402 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fd130001670 con 0x7fd1440c2c60 2024-02-21T23:06:42.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.848+0000 7fd143400640 1 -- 172.21.15.28:0/4005028402 >> v1:172.21.15.28:6789/0 conn(0x7fd1440c2c60 legacy=0x7fd14406dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:42.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.848+0000 7fd143400640 1 --2- 172.21.15.28:0/4005028402 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd12c0039e0 0x7fd12c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:42.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.848+0000 7fd143400640 1 --2- 172.21.15.28:0/4005028402 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd14406f400 0x7fd12c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:42.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.848+0000 7fd143400640 1 --2- 172.21.15.28:0/4005028402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd12c004d40 0x7fd12c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:42.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.848+0000 7fd143400640 1 -- 172.21.15.28:0/4005028402 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1440c1080 con 0x7fd12c004d40 2024-02-21T23:06:42.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.848+0000 7fd143400640 1 -- 172.21.15.28:0/4005028402 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd12c00baa0 con 0x7fd12c0039e0 2024-02-21T23:06:42.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.848+0000 7fd143400640 1 -- 172.21.15.28:0/4005028402 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd12c00bca0 con 0x7fd14406f400 2024-02-21T23:06:42.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.848+0000 7fd149a00640 1 --2- 172.21.15.28:0/4005028402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd12c004d40 0x7fd12c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:42.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.848+0000 7fd149000640 1 --2- 172.21.15.28:0/4005028402 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd12c0039e0 0x7fd12c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:42.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.849+0000 7fd143e00640 1 --2- 172.21.15.28:0/4005028402 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd14406f400 0x7fd12c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:42.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.849+0000 7fd149a00640 1 -- 172.21.15.28:0/4005028402 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd14406f400 msgr2=0x7fd12c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:42.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.849+0000 7fd149a00640 1 --2- 172.21.15.28:0/4005028402 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd14406f400 0x7fd12c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:42.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.849+0000 7fd149a00640 1 -- 172.21.15.28:0/4005028402 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd12c0039e0 msgr2=0x7fd12c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:42.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.849+0000 7fd149a00640 1 --2- 172.21.15.28:0/4005028402 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd12c0039e0 0x7fd12c003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:42.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.849+0000 7fd149a00640 1 -- 172.21.15.28:0/4005028402 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd12c003520 con 0x7fd12c004d40 2024-02-21T23:06:42.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.850+0000 7fd149a00640 1 --2- 172.21.15.28:0/4005028402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd12c004d40 0x7fd12c0051c0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fd13c009f90 tx=0x7fd12c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c898ba0023797f3 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:42.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.850+0000 7fd143400640 1 -- 172.21.15.28:0/4005028402 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd13c039070 con 0x7fd12c004d40 2024-02-21T23:06:42.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.850+0000 7fd143400640 1 -- 172.21.15.28:0/4005028402 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd13c02cbd0 con 0x7fd12c004d40 2024-02-21T23:06:42.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.850+0000 7fd143400640 1 -- 172.21.15.28:0/4005028402 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd13c009c10 con 0x7fd12c004d40 2024-02-21T23:06:42.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.851+0000 7fd14a400640 1 -- 172.21.15.28:0/4005028402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd12c004d40 msgr2=0x7fd12c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:42.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.851+0000 7fd14a400640 1 --2- 172.21.15.28:0/4005028402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd12c004d40 0x7fd12c0051c0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fd13c009f90 tx=0x7fd12c005830 comp rx=0 tx=0).stop 2024-02-21T23:06:42.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.851+0000 7fd149a00640 1 -- 172.21.15.28:0/4005028402 reap_dead start 2024-02-21T23:06:42.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.851+0000 7fd14a400640 1 -- 172.21.15.28:0/4005028402 shutdown_connections 2024-02-21T23:06:42.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.851+0000 7fd14a400640 1 -- 172.21.15.28:0/4005028402 >> 172.21.15.28:0/4005028402 conn(0x7fd1440a8530 msgr2=0x7fd1440a8910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:42.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.851+0000 7fd14a400640 1 -- 172.21.15.28:0/4005028402 shutdown_connections 2024-02-21T23:06:42.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.851+0000 7fd14a400640 1 -- 172.21.15.28:0/4005028402 wait complete. 2024-02-21T23:06:42.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.853+0000 7fd14a400640 1 Processor -- start 2024-02-21T23:06:42.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.853+0000 7fd14a400640 1 -- start start 2024-02-21T23:06:42.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.854+0000 7fd14a400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd12c004d40 0x7fd14414f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:42.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.854+0000 7fd14a400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd14414d910 0x7fd1440ab840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:42.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.854+0000 7fd14a400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1441478b0 0x7fd1440a6480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:42.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.854+0000 7fd14a400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd14406b580 con 0x7fd1441478b0 2024-02-21T23:06:42.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.854+0000 7fd14a400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1441504d0 con 0x7fd12c004d40 2024-02-21T23:06:42.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.854+0000 7fd14a400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd14406b700 con 0x7fd14414d910 2024-02-21T23:06:42.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.854+0000 7fd149a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1441478b0 0x7fd1440a6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:42.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.854+0000 7fd149a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1441478b0 0x7fd1440a6480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59378/0 (socket says 172.21.15.28:59378) 2024-02-21T23:06:42.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.854+0000 7fd149a00640 1 -- 172.21.15.28:0/1353720216 learned_addr learned my addr 172.21.15.28:0/1353720216 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:42.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.854+0000 7fd149000640 1 --2- 172.21.15.28:0/1353720216 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd12c004d40 0x7fd14414f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:42.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.854+0000 7fd143e00640 1 --2- 172.21.15.28:0/1353720216 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd14414d910 0x7fd1440ab840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:42.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.855+0000 7fd149a00640 1 -- 172.21.15.28:0/1353720216 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd14414d910 msgr2=0x7fd1440ab840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:42.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.855+0000 7fd149a00640 1 --2- 172.21.15.28:0/1353720216 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd14414d910 0x7fd1440ab840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:42.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.855+0000 7fd149a00640 1 -- 172.21.15.28:0/1353720216 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd12c004d40 msgr2=0x7fd14414f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:42.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.855+0000 7fd149a00640 1 --2- 172.21.15.28:0/1353720216 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd12c004d40 0x7fd14414f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:42.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.855+0000 7fd149a00640 1 -- 172.21.15.28:0/1353720216 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1440ac0d0 con 0x7fd1441478b0 2024-02-21T23:06:42.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.855+0000 7fd143e00640 1 --2- 172.21.15.28:0/1353720216 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd14414d910 0x7fd1440ab840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:06:42.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.855+0000 7fd149a00640 1 --2- 172.21.15.28:0/1353720216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1441478b0 0x7fd1440a6480 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fd13c037040 tx=0x7fd13c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:42.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.855+0000 7fd140c00640 1 -- 172.21.15.28:0/1353720216 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd13c039030 con 0x7fd1441478b0 2024-02-21T23:06:42.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.855+0000 7fd140c00640 1 -- 172.21.15.28:0/1353720216 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd13c03a040 con 0x7fd1441478b0 2024-02-21T23:06:42.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.855+0000 7fd140c00640 1 -- 172.21.15.28:0/1353720216 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd13c003990 con 0x7fd1441478b0 2024-02-21T23:06:42.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.855+0000 7fd14a400640 1 -- 172.21.15.28:0/1353720216 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1440ac3b0 con 0x7fd1441478b0 2024-02-21T23:06:42.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.855+0000 7fd14a400640 1 -- 172.21.15.28:0/1353720216 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd14414f030 con 0x7fd1441478b0 2024-02-21T23:06:42.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.857+0000 7fd14a400640 1 -- 172.21.15.28:0/1353720216 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd14404f290 con 0x7fd1441478b0 2024-02-21T23:06:42.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.858+0000 7fd140c00640 1 -- 172.21.15.28:0/1353720216 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd13c045020 con 0x7fd1441478b0 2024-02-21T23:06:42.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.858+0000 7fd140c00640 1 --2- 172.21.15.28:0/1353720216 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd12406b4d0 0x7fd12406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:42.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.859+0000 7fd140c00640 1 -- 172.21.15.28:0/1353720216 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5593+0+0 (secure 0 0 0) 0x7fd13c0abfe0 con 0x7fd1441478b0 2024-02-21T23:06:42.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.859+0000 7fd149000640 1 --2- 172.21.15.28:0/1353720216 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd12406b4d0 0x7fd12406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:42.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.860+0000 7fd149000640 1 --2- 172.21.15.28:0/1353720216 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd12406b4d0 0x7fd12406d990 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fd138007fd0 tx=0x7fd138009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:42.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:42.863+0000 7fd140c00640 1 -- 172.21.15.28:0/1353720216 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd13c07aec0 con 0x7fd1441478b0 2024-02-21T23:06:43.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.068+0000 7fd14a400640 1 -- 172.21.15.28:0/1353720216 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.b", "name": "mds_join_fs"} v 0) v1 -- 0x7fd14414f200 con 0x7fd1441478b0 2024-02-21T23:06:43.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.073+0000 7fd140c00640 1 -- 172.21.15.28:0/1353720216 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.b", "name": "mds_join_fs"}]=0 v27) v1 ==== 96+0+0 (secure 0 0 0) 0x7fd13c07a7c0 con 0x7fd1441478b0 2024-02-21T23:06:43.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.075+0000 7fd14a400640 1 -- 172.21.15.28:0/1353720216 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd12406b4d0 msgr2=0x7fd12406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:43.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.075+0000 7fd14a400640 1 --2- 172.21.15.28:0/1353720216 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd12406b4d0 0x7fd12406d990 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fd138007fd0 tx=0x7fd138009040 comp rx=0 tx=0).stop 2024-02-21T23:06:43.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.075+0000 7fd14a400640 1 -- 172.21.15.28:0/1353720216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1441478b0 msgr2=0x7fd1440a6480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:43.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.075+0000 7fd14a400640 1 --2- 172.21.15.28:0/1353720216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1441478b0 0x7fd1440a6480 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fd13c037040 tx=0x7fd13c009750 comp rx=0 tx=0).stop 2024-02-21T23:06:43.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.076+0000 7fd14a400640 1 -- 172.21.15.28:0/1353720216 shutdown_connections 2024-02-21T23:06:43.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.076+0000 7fd14a400640 1 --2- 172.21.15.28:0/1353720216 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd12406b4d0 0x7fd12406d990 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:43.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.076+0000 7fd14a400640 1 --2- 172.21.15.28:0/1353720216 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd14414d910 0x7fd1440ab840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:43.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.076+0000 7fd14a400640 1 --2- 172.21.15.28:0/1353720216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1441478b0 0x7fd1440a6480 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:43.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.076+0000 7fd14a400640 1 --2- 172.21.15.28:0/1353720216 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd12c004d40 0x7fd14414f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:43.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.076+0000 7fd14a400640 1 -- 172.21.15.28:0/1353720216 >> 172.21.15.28:0/1353720216 conn(0x7fd1440a8530 msgr2=0x7fd1440c39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:43.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.076+0000 7fd14a400640 1 -- 172.21.15.28:0/1353720216 shutdown_connections 2024-02-21T23:06:43.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.076+0000 7fd14a400640 1 -- 172.21.15.28:0/1353720216 wait complete. 2024-02-21T23:06:43.086 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.d mds_join_fs 2024-02-21T23:06:43.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.320+0000 7fee8fe00640 1 Processor -- start 2024-02-21T23:06:43.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.320+0000 7fee8fe00640 1 -- start start 2024-02-21T23:06:43.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.322+0000 7fee8fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fee9013e0c0 0x7fee900c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:43.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.322+0000 7fee8fe00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee90158550 con 0x7fee9006dbc0 2024-02-21T23:06:43.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.322+0000 7fee8fe00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee901587d0 con 0x7fee900c76f0 2024-02-21T23:06:43.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.322+0000 7fee8fe00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee900c89a0 con 0x7fee9013e0c0 2024-02-21T23:06:43.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.322+0000 7fee8ea00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fee9013e0c0 0x7fee900c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:43.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.322+0000 7fee8e000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fee9006dbc0 0x7fee9006c280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59574/0 (socket says 172.21.15.28:59574) 2024-02-21T23:06:43.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.323+0000 7fee8e000640 1 -- 172.21.15.28:0/2627705327 learned_addr learned my addr 172.21.15.28:0/2627705327 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:06:43.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.323+0000 7fee8ea00640 1 -- 172.21.15.28:0/2627705327 >> v1:172.21.15.81:6789/0 conn(0x7fee900c76f0 legacy=0x7fee9006c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:43.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.323+0000 7fee8ea00640 1 -- 172.21.15.28:0/2627705327 >> v1:172.21.15.28:6789/0 conn(0x7fee9006dbc0 legacy=0x7fee9006c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:43.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.323+0000 7fee8ea00640 1 -- 172.21.15.28:0/2627705327 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee90158ab0 con 0x7fee9013e0c0 2024-02-21T23:06:43.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.324+0000 7fee8ea00640 1 --2- 172.21.15.28:0/2627705327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee9013e0c0 0x7fee900c71b0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fee800097a0 tx=0x7fee8002b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d1626d8de49fdde server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:43.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.324+0000 7fee8d600640 1 -- 172.21.15.28:0/2627705327 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee80038070 con 0x7fee9013e0c0 2024-02-21T23:06:43.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.325+0000 7fee8d600640 1 -- 172.21.15.28:0/2627705327 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee80031700 con 0x7fee9013e0c0 2024-02-21T23:06:43.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.325+0000 7fee8d600640 1 -- 172.21.15.28:0/2627705327 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee80031ad0 con 0x7fee9013e0c0 2024-02-21T23:06:43.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.325+0000 7fee8fe00640 1 -- 172.21.15.28:0/2627705327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee9013e0c0 msgr2=0x7fee900c71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:43.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.325+0000 7fee8fe00640 1 --2- 172.21.15.28:0/2627705327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee9013e0c0 0x7fee900c71b0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fee800097a0 tx=0x7fee8002b7a0 comp rx=0 tx=0).stop 2024-02-21T23:06:43.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.325+0000 7fee8fe00640 1 -- 172.21.15.28:0/2627705327 shutdown_connections 2024-02-21T23:06:43.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.325+0000 7fee8fe00640 1 --2- 172.21.15.28:0/2627705327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee9013e0c0 0x7fee900c71b0 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:43.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.325+0000 7fee8fe00640 1 -- 172.21.15.28:0/2627705327 >> 172.21.15.28:0/2627705327 conn(0x7fee90073110 msgr2=0x7fee900ac490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:43.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.325+0000 7fee8fe00640 1 -- 172.21.15.28:0/2627705327 shutdown_connections 2024-02-21T23:06:43.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.325+0000 7fee8fe00640 1 -- 172.21.15.28:0/2627705327 wait complete. 2024-02-21T23:06:43.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.327+0000 7fee8fe00640 1 Processor -- start 2024-02-21T23:06:43.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.327+0000 7fee8fe00640 1 -- start start 2024-02-21T23:06:43.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.328+0000 7fee8fe00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fee9006dbc0 0x7fee901eb850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:43.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.328+0000 7fee8fe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee900c76f0 0x7fee90157640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:43.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.328+0000 7fee8fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee9013e0c0 0x7fee90139b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:43.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.328+0000 7fee8fe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee9013e520 con 0x7fee9013e0c0 2024-02-21T23:06:43.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.328+0000 7fee8fe00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee90159070 con 0x7fee900c76f0 2024-02-21T23:06:43.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.328+0000 7fee8fe00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee9013e6a0 con 0x7fee9006dbc0 2024-02-21T23:06:43.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.328+0000 7fee8f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee9013e0c0 0x7fee90139b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:43.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.328+0000 7fee8f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee9013e0c0 0x7fee90139b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59396/0 (socket says 172.21.15.28:59396) 2024-02-21T23:06:43.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.328+0000 7fee8f400640 1 -- 172.21.15.28:0/3571191505 learned_addr learned my addr 172.21.15.28:0/3571191505 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:43.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.329+0000 7fee8e000640 1 --2- 172.21.15.28:0/3571191505 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee900c76f0 0x7fee90157640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:43.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.329+0000 7fee8ea00640 1 --2- 172.21.15.28:0/3571191505 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fee9006dbc0 0x7fee901eb850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:43.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.329+0000 7fee8f400640 1 -- 172.21.15.28:0/3571191505 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fee9006dbc0 msgr2=0x7fee901eb850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:43.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.329+0000 7fee8f400640 1 --2- 172.21.15.28:0/3571191505 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fee9006dbc0 0x7fee901eb850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:43.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.329+0000 7fee8f400640 1 -- 172.21.15.28:0/3571191505 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee900c76f0 msgr2=0x7fee90157640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:43.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.329+0000 7fee8f400640 1 --2- 172.21.15.28:0/3571191505 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee900c76f0 0x7fee90157640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:43.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.329+0000 7fee8f400640 1 -- 172.21.15.28:0/3571191505 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee900abd20 con 0x7fee9013e0c0 2024-02-21T23:06:43.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.329+0000 7fee8f400640 1 --2- 172.21.15.28:0/3571191505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee9013e0c0 0x7fee90139b70 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fee8800b9c0 tx=0x7fee8800be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:43.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.329+0000 7fee86a00640 1 -- 172.21.15.28:0/3571191505 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee88019070 con 0x7fee9013e0c0 2024-02-21T23:06:43.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.329+0000 7fee86a00640 1 -- 172.21.15.28:0/3571191505 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee8800cba0 con 0x7fee9013e0c0 2024-02-21T23:06:43.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.329+0000 7fee86a00640 1 -- 172.21.15.28:0/3571191505 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee88014440 con 0x7fee9013e0c0 2024-02-21T23:06:43.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.330+0000 7fee8fe00640 1 -- 172.21.15.28:0/3571191505 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee900ac000 con 0x7fee9013e0c0 2024-02-21T23:06:43.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.330+0000 7fee8fe00640 1 -- 172.21.15.28:0/3571191505 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee900ac5c0 con 0x7fee9013e0c0 2024-02-21T23:06:43.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.331+0000 7fee8fe00640 1 -- 172.21.15.28:0/3571191505 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee9004f290 con 0x7fee9013e0c0 2024-02-21T23:06:43.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.332+0000 7fee86a00640 1 -- 172.21.15.28:0/3571191505 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fee88003970 con 0x7fee9013e0c0 2024-02-21T23:06:43.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.333+0000 7fee86a00640 1 --2- 172.21.15.28:0/3571191505 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fee6c06b500 0x7fee6c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:43.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.333+0000 7fee86a00640 1 -- 172.21.15.28:0/3571191505 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5593+0+0 (secure 0 0 0) 0x7fee88089d10 con 0x7fee9013e0c0 2024-02-21T23:06:43.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.333+0000 7fee8ea00640 1 --2- 172.21.15.28:0/3571191505 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fee6c06b500 0x7fee6c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:43.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.335+0000 7fee8ea00640 1 --2- 172.21.15.28:0/3571191505 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fee6c06b500 0x7fee6c06d9c0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fee80035000 tx=0x7fee80008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:43.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.338+0000 7fee86a00640 1 -- 172.21.15.28:0/3571191505 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fee88058c90 con 0x7fee9013e0c0 2024-02-21T23:06:43.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.540+0000 7fee8fe00640 1 -- 172.21.15.28:0/3571191505 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.d", "name": "mds_join_fs"} v 0) v1 -- 0x7fee9006d9a0 con 0x7fee9013e0c0 2024-02-21T23:06:43.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.547+0000 7fee86a00640 1 -- 172.21.15.28:0/3571191505 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.d", "name": "mds_join_fs"}]=0 v28) v1 ==== 96+0+0 (secure 0 0 0) 0x7fee88058590 con 0x7fee9013e0c0 2024-02-21T23:06:43.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.550+0000 7fee8fe00640 1 -- 172.21.15.28:0/3571191505 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fee6c06b500 msgr2=0x7fee6c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:43.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.550+0000 7fee8fe00640 1 --2- 172.21.15.28:0/3571191505 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fee6c06b500 0x7fee6c06d9c0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fee80035000 tx=0x7fee80008030 comp rx=0 tx=0).stop 2024-02-21T23:06:43.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.550+0000 7fee8fe00640 1 -- 172.21.15.28:0/3571191505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee9013e0c0 msgr2=0x7fee90139b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:43.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.550+0000 7fee8fe00640 1 --2- 172.21.15.28:0/3571191505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee9013e0c0 0x7fee90139b70 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fee8800b9c0 tx=0x7fee8800be80 comp rx=0 tx=0).stop 2024-02-21T23:06:43.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.551+0000 7fee8fe00640 1 -- 172.21.15.28:0/3571191505 shutdown_connections 2024-02-21T23:06:43.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.551+0000 7fee8fe00640 1 --2- 172.21.15.28:0/3571191505 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fee6c06b500 0x7fee6c06d9c0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:43.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.551+0000 7fee8fe00640 1 --2- 172.21.15.28:0/3571191505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee9013e0c0 0x7fee90139b70 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:43.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.551+0000 7fee8fe00640 1 --2- 172.21.15.28:0/3571191505 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee900c76f0 0x7fee90157640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:43.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.551+0000 7fee8fe00640 1 --2- 172.21.15.28:0/3571191505 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fee9006dbc0 0x7fee901eb850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:43.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.551+0000 7fee8fe00640 1 -- 172.21.15.28:0/3571191505 >> 172.21.15.28:0/3571191505 conn(0x7fee90073110 msgr2=0x7fee901de5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:43.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.551+0000 7fee8fe00640 1 -- 172.21.15.28:0/3571191505 shutdown_connections 2024-02-21T23:06:43.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.551+0000 7fee8fe00640 1 -- 172.21.15.28:0/3571191505 wait complete. 2024-02-21T23:06:43.561 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.c mds_join_fs 2024-02-21T23:06:43.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.791+0000 7fa861400640 1 Processor -- start 2024-02-21T23:06:43.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.791+0000 7fa861400640 1 -- start start 2024-02-21T23:06:43.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.793+0000 7fa861400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa85c1500f0 0x7fa85c06ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:43.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.793+0000 7fa861400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa85c06db10 con 0x7fa85c06b3b0 2024-02-21T23:06:43.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.793+0000 7fa861400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa85c1539a0 con 0x7fa85c0c2c60 2024-02-21T23:06:43.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.793+0000 7fa861400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa85c0c1080 con 0x7fa85c1500f0 2024-02-21T23:06:43.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.793+0000 7fa85be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa85c1500f0 0x7fa85c06ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:43.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.793+0000 7fa85be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa85c1500f0 0x7fa85c06ae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59408/0 (socket says 172.21.15.28:59408) 2024-02-21T23:06:43.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.793+0000 7fa85be00640 1 -- 172.21.15.28:0/2383330863 learned_addr learned my addr 172.21.15.28:0/2383330863 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:43.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.794+0000 7fa85be00640 1 -- 172.21.15.28:0/2383330863 >> v1:172.21.15.81:6789/0 conn(0x7fa85c0c2c60 legacy=0x7fa85c06dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:43.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.794+0000 7fa85be00640 1 -- 172.21.15.28:0/2383330863 >> v1:172.21.15.28:6789/0 conn(0x7fa85c06b3b0 legacy=0x7fa85c15d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:43.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.794+0000 7fa85be00640 1 -- 172.21.15.28:0/2383330863 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa85c0c12d0 con 0x7fa85c1500f0 2024-02-21T23:06:43.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.794+0000 7fa85b400640 1 -- 172.21.15.28:0/2383330863 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1576665510 0 0) 0x7fa85c06db10 con 0x7fa85c06b3b0 2024-02-21T23:06:43.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.795+0000 7fa85be00640 1 --2- 172.21.15.28:0/2383330863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa85c1500f0 0x7fa85c06ae70 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fa85c06c010 tx=0x7fa84c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=381d23bca1b6c87e server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:43.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.795+0000 7fa85b400640 1 -- 172.21.15.28:0/2383330863 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa84c034070 con 0x7fa85c1500f0 2024-02-21T23:06:43.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.796+0000 7fa85b400640 1 -- 172.21.15.28:0/2383330863 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa84c030840 con 0x7fa85c1500f0 2024-02-21T23:06:43.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.796+0000 7fa85b400640 1 -- 172.21.15.28:0/2383330863 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa84c030c10 con 0x7fa85c1500f0 2024-02-21T23:06:43.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.796+0000 7fa861400640 1 -- 172.21.15.28:0/2383330863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa85c1500f0 msgr2=0x7fa85c06ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:43.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.796+0000 7fa861400640 1 --2- 172.21.15.28:0/2383330863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa85c1500f0 0x7fa85c06ae70 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fa85c06c010 tx=0x7fa84c02b980 comp rx=0 tx=0).stop 2024-02-21T23:06:43.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.796+0000 7fa861400640 1 -- 172.21.15.28:0/2383330863 shutdown_connections 2024-02-21T23:06:43.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.796+0000 7fa861400640 1 --2- 172.21.15.28:0/2383330863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa85c1500f0 0x7fa85c06ae70 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:43.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.796+0000 7fa861400640 1 -- 172.21.15.28:0/2383330863 >> 172.21.15.28:0/2383330863 conn(0x7fa85c0a8530 msgr2=0x7fa85c0a8910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:43.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.796+0000 7fa861400640 1 -- 172.21.15.28:0/2383330863 shutdown_connections 2024-02-21T23:06:43.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.796+0000 7fa861400640 1 -- 172.21.15.28:0/2383330863 wait complete. 2024-02-21T23:06:43.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.798+0000 7fa861400640 1 Processor -- start 2024-02-21T23:06:43.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.798+0000 7fa861400640 1 -- start start 2024-02-21T23:06:43.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.799+0000 7fa861400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa85c06b3b0 0x7fa85c1ee900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:43.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.799+0000 7fa861400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa85c0c2c60 0x7fa85c0c4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:43.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.799+0000 7fa861400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa85c1500f0 0x7fa85c1ea9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:43.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.799+0000 7fa861400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa85c1535e0 con 0x7fa85c1500f0 2024-02-21T23:06:43.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.799+0000 7fa861400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa85c1504d0 con 0x7fa85c06b3b0 2024-02-21T23:06:43.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.799+0000 7fa861400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa85c153760 con 0x7fa85c0c2c60 2024-02-21T23:06:43.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.799+0000 7fa860a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa85c1500f0 0x7fa85c1ea9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:43.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.800+0000 7fa860a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa85c1500f0 0x7fa85c1ea9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59414/0 (socket says 172.21.15.28:59414) 2024-02-21T23:06:43.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.800+0000 7fa860a00640 1 -- 172.21.15.28:0/216573003 learned_addr learned my addr 172.21.15.28:0/216573003 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:43.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.800+0000 7fa853e00640 1 --2- 172.21.15.28:0/216573003 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa85c0c2c60 0x7fa85c0c4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:43.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.800+0000 7fa85be00640 1 --2- 172.21.15.28:0/216573003 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa85c06b3b0 0x7fa85c1ee900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:43.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.800+0000 7fa860a00640 1 -- 172.21.15.28:0/216573003 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa85c0c2c60 msgr2=0x7fa85c0c4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:43.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.800+0000 7fa860a00640 1 --2- 172.21.15.28:0/216573003 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa85c0c2c60 0x7fa85c0c4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:43.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.800+0000 7fa860a00640 1 -- 172.21.15.28:0/216573003 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa85c06b3b0 msgr2=0x7fa85c1ee900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:43.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.800+0000 7fa860a00640 1 --2- 172.21.15.28:0/216573003 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa85c06b3b0 0x7fa85c1ee900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:43.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.800+0000 7fa860a00640 1 -- 172.21.15.28:0/216573003 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa85c072ef0 con 0x7fa85c1500f0 2024-02-21T23:06:43.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.800+0000 7fa860a00640 1 --2- 172.21.15.28:0/216573003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa85c1500f0 0x7fa85c1ea9f0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fa85400c760 tx=0x7fa85400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:43.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.801+0000 7fa858c00640 1 -- 172.21.15.28:0/216573003 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa854003da0 con 0x7fa85c1500f0 2024-02-21T23:06:43.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.801+0000 7fa858c00640 1 -- 172.21.15.28:0/216573003 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa854005070 con 0x7fa85c1500f0 2024-02-21T23:06:43.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.801+0000 7fa858c00640 1 -- 172.21.15.28:0/216573003 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa854005460 con 0x7fa85c1500f0 2024-02-21T23:06:43.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.801+0000 7fa861400640 1 -- 172.21.15.28:0/216573003 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa85c1eee40 con 0x7fa85c1500f0 2024-02-21T23:06:43.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.801+0000 7fa861400640 1 -- 172.21.15.28:0/216573003 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa85c072930 con 0x7fa85c1500f0 2024-02-21T23:06:43.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.802+0000 7fa861400640 1 -- 172.21.15.28:0/216573003 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa85c04f290 con 0x7fa85c1500f0 2024-02-21T23:06:43.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.803+0000 7fa858c00640 1 -- 172.21.15.28:0/216573003 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa854005600 con 0x7fa85c1500f0 2024-02-21T23:06:43.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.804+0000 7fa858c00640 1 --2- 172.21.15.28:0/216573003 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa83806b4d0 0x7fa83806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:43.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.804+0000 7fa858c00640 1 -- 172.21.15.28:0/216573003 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5593+0+0 (secure 0 0 0) 0x7fa85408a0a0 con 0x7fa85c1500f0 2024-02-21T23:06:43.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.805+0000 7fa85be00640 1 --2- 172.21.15.28:0/216573003 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa83806b4d0 0x7fa83806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:43.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.806+0000 7fa85be00640 1 --2- 172.21.15.28:0/216573003 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa83806b4d0 0x7fa83806d990 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fa84c036000 tx=0x7fa84c03b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:43.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:43.808+0000 7fa858c00640 1 -- 172.21.15.28:0/216573003 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fa854058ef0 con 0x7fa85c1500f0 2024-02-21T23:06:44.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.011+0000 7fa861400640 1 -- 172.21.15.28:0/216573003 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.c", "name": "mds_join_fs"} v 0) v1 -- 0x7fa85c06f1e0 con 0x7fa85c1500f0 2024-02-21T23:06:44.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.018+0000 7fa858c00640 1 -- 172.21.15.28:0/216573003 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.c", "name": "mds_join_fs"}]=0 v29) v1 ==== 96+0+0 (secure 0 0 0) 0x7fa8540587f0 con 0x7fa85c1500f0 2024-02-21T23:06:44.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.021+0000 7fa861400640 1 -- 172.21.15.28:0/216573003 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa83806b4d0 msgr2=0x7fa83806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:44.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.021+0000 7fa861400640 1 --2- 172.21.15.28:0/216573003 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa83806b4d0 0x7fa83806d990 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fa84c036000 tx=0x7fa84c03b040 comp rx=0 tx=0).stop 2024-02-21T23:06:44.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.021+0000 7fa861400640 1 -- 172.21.15.28:0/216573003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa85c1500f0 msgr2=0x7fa85c1ea9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:44.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.021+0000 7fa861400640 1 --2- 172.21.15.28:0/216573003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa85c1500f0 0x7fa85c1ea9f0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fa85400c760 tx=0x7fa85400cc20 comp rx=0 tx=0).stop 2024-02-21T23:06:44.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.021+0000 7fa861400640 1 -- 172.21.15.28:0/216573003 shutdown_connections 2024-02-21T23:06:44.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.021+0000 7fa861400640 1 --2- 172.21.15.28:0/216573003 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa83806b4d0 0x7fa83806d990 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:44.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.021+0000 7fa861400640 1 --2- 172.21.15.28:0/216573003 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa85c0c2c60 0x7fa85c0c4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:44.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.021+0000 7fa861400640 1 --2- 172.21.15.28:0/216573003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa85c1500f0 0x7fa85c1ea9f0 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:44.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.021+0000 7fa861400640 1 --2- 172.21.15.28:0/216573003 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa85c06b3b0 0x7fa85c1ee900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:44.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.021+0000 7fa861400640 1 -- 172.21.15.28:0/216573003 >> 172.21.15.28:0/216573003 conn(0x7fa85c0a8530 msgr2=0x7fa85c14b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:44.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.022+0000 7fa861400640 1 -- 172.21.15.28:0/216573003 shutdown_connections 2024-02-21T23:06:44.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.022+0000 7fa861400640 1 -- 172.21.15.28:0/216573003 wait complete. 2024-02-21T23:06:44.032 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-02-21T23:06:44.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.268+0000 7f8ed2800640 1 Processor -- start 2024-02-21T23:06:44.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.268+0000 7f8ed2800640 1 -- start start 2024-02-21T23:06:44.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.270+0000 7f8ed2800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8ecc06b0e0 0x7f8ecc0c5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:44.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.270+0000 7f8ed2800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8ecc150700 0x7f8ecc156500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:44.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.270+0000 7f8ed2800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ecc0c61e0 con 0x7f8ecc0c5d80 2024-02-21T23:06:44.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.270+0000 7f8ed2800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ecc153bc0 con 0x7f8ecc06b0e0 2024-02-21T23:06:44.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.270+0000 7f8ed2800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ecc152fb0 con 0x7f8ecc150700 2024-02-21T23:06:44.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.270+0000 7f8ec2a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f8ecc0c5d80 0x7f8ecc14fff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59584/0 (socket says 172.21.15.28:59584) 2024-02-21T23:06:44.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.270+0000 7f8ec2a00640 1 -- 172.21.15.28:0/235938991 learned_addr learned my addr 172.21.15.28:0/235938991 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:06:44.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.270+0000 7f8ecb400640 1 --2- 172.21.15.28:0/235938991 >> v2:172.21.15.28:3300/0 conn(0x7f8ecc06b0e0 0x7f8ecc0c5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:44.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.271+0000 7f8ecbe00640 1 --2- 172.21.15.28:0/235938991 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8ecc150700 0x7f8ecc156500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:44.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.271+0000 7f8ecaa00640 1 -- 172.21.15.28:0/235938991 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2241986240 0 0) 0x7f8ecc0c61e0 con 0x7f8ecc0c5d80 2024-02-21T23:06:44.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.271+0000 7f8ecaa00640 1 -- 172.21.15.28:0/235938991 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8eb0003520 con 0x7f8ecc0c5d80 2024-02-21T23:06:44.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.271+0000 7f8ecb400640 1 -- 172.21.15.28:0/235938991 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8ecc150700 msgr2=0x7f8ecc156500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:44.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.271+0000 7f8ecb400640 1 --2- 172.21.15.28:0/235938991 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8ecc150700 0x7f8ecc156500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:44.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.271+0000 7f8ecb400640 1 -- 172.21.15.28:0/235938991 >> v1:172.21.15.28:6789/0 conn(0x7f8ecc0c5d80 legacy=0x7f8ecc14fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:44.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.271+0000 7f8ecb400640 1 -- 172.21.15.28:0/235938991 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ecc153290 con 0x7f8ecc06b0e0 2024-02-21T23:06:44.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.271+0000 7f8ecbe00640 1 --2- 172.21.15.28:0/235938991 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8ecc150700 0x7f8ecc156500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:06:44.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.272+0000 7f8ecb400640 1 --2- 172.21.15.28:0/235938991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ecc06b0e0 0x7f8ecc0c5840 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc153d40 tx=0x7f8ebc02be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3073c14ebd61c130 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:44.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.272+0000 7f8ecaa00640 1 -- 172.21.15.28:0/235938991 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ebc039070 con 0x7f8ecc06b0e0 2024-02-21T23:06:44.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.273+0000 7f8ecaa00640 1 -- 172.21.15.28:0/235938991 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ebc02cbb0 con 0x7f8ecc06b0e0 2024-02-21T23:06:44.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.273+0000 7f8ecaa00640 1 -- 172.21.15.28:0/235938991 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ebc034440 con 0x7f8ecc06b0e0 2024-02-21T23:06:44.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.273+0000 7f8ed2800640 1 -- 172.21.15.28:0/235938991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ecc06b0e0 msgr2=0x7f8ecc0c5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:44.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.273+0000 7f8ed2800640 1 --2- 172.21.15.28:0/235938991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ecc06b0e0 0x7f8ecc0c5840 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc153d40 tx=0x7f8ebc02be00 comp rx=0 tx=0).stop 2024-02-21T23:06:44.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.273+0000 7f8ed2800640 1 -- 172.21.15.28:0/235938991 shutdown_connections 2024-02-21T23:06:44.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.273+0000 7f8ed2800640 1 --2- 172.21.15.28:0/235938991 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8ecc150700 0x7f8ecc156500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:44.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.273+0000 7f8ed2800640 1 --2- 172.21.15.28:0/235938991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ecc06b0e0 0x7f8ecc0c5840 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:44.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.273+0000 7f8ed2800640 1 -- 172.21.15.28:0/235938991 >> 172.21.15.28:0/235938991 conn(0x7f8ecc0a85a0 msgr2=0x7f8ecc0a8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:44.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.273+0000 7f8ed2800640 1 -- 172.21.15.28:0/235938991 shutdown_connections 2024-02-21T23:06:44.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.274+0000 7f8ed2800640 1 -- 172.21.15.28:0/235938991 wait complete. 2024-02-21T23:06:44.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.275+0000 7f8ed2800640 1 Processor -- start 2024-02-21T23:06:44.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.275+0000 7f8ed2800640 1 -- start start 2024-02-21T23:06:44.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.276+0000 7f8ed2800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8ecc06b0e0 0x7f8ecc14f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:44.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.276+0000 7f8ed2800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8ecc0c5d80 0x7f8ecc154150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:44.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.276+0000 7f8ed2800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ecc150700 0x7f8ecc1465f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:44.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.276+0000 7f8ed2800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ecc06b4c0 con 0x7f8ecc150700 2024-02-21T23:06:44.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.276+0000 7f8ed2800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ecc153850 con 0x7f8ecc06b0e0 2024-02-21T23:06:44.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.276+0000 7f8ed2800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ecc06b640 con 0x7f8ecc0c5d80 2024-02-21T23:06:44.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.277+0000 7f8ecbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ecc150700 0x7f8ecc1465f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:44.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.277+0000 7f8ecbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ecc150700 0x7f8ecc1465f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59430/0 (socket says 172.21.15.28:59430) 2024-02-21T23:06:44.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.277+0000 7f8ecbe00640 1 -- 172.21.15.28:0/3869154374 learned_addr learned my addr 172.21.15.28:0/3869154374 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:44.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.277+0000 7f8ecb400640 1 --2- 172.21.15.28:0/3869154374 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8ecc06b0e0 0x7f8ecc14f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:44.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.277+0000 7f8ec2a00640 1 --2- 172.21.15.28:0/3869154374 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8ecc0c5d80 0x7f8ecc154150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:44.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.277+0000 7f8ecbe00640 1 -- 172.21.15.28:0/3869154374 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8ecc0c5d80 msgr2=0x7f8ecc154150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:44.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.277+0000 7f8ecbe00640 1 --2- 172.21.15.28:0/3869154374 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8ecc0c5d80 0x7f8ecc154150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:44.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.277+0000 7f8ecbe00640 1 -- 172.21.15.28:0/3869154374 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8ecc06b0e0 msgr2=0x7f8ecc14f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:44.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.277+0000 7f8ecbe00640 1 --2- 172.21.15.28:0/3869154374 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8ecc06b0e0 0x7f8ecc14f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:44.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.277+0000 7f8ecbe00640 1 -- 172.21.15.28:0/3869154374 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ecc155e80 con 0x7f8ecc150700 2024-02-21T23:06:44.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.278+0000 7f8ecbe00640 1 --2- 172.21.15.28:0/3869154374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ecc150700 0x7f8ecc1465f0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f8ec400d5c0 tx=0x7f8ec400da80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:44.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.278+0000 7f8ec3e00640 1 -- 172.21.15.28:0/3869154374 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec4004030 con 0x7f8ecc150700 2024-02-21T23:06:44.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.278+0000 7f8ec3e00640 1 -- 172.21.15.28:0/3869154374 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ec40041d0 con 0x7f8ecc150700 2024-02-21T23:06:44.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.278+0000 7f8ec3e00640 1 -- 172.21.15.28:0/3869154374 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec4003610 con 0x7f8ecc150700 2024-02-21T23:06:44.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.278+0000 7f8ed2800640 1 -- 172.21.15.28:0/3869154374 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ecc156160 con 0x7f8ecc150700 2024-02-21T23:06:44.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.278+0000 7f8ed2800640 1 -- 172.21.15.28:0/3869154374 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ecc14f950 con 0x7f8ecc150700 2024-02-21T23:06:44.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.280+0000 7f8ed2800640 1 -- 172.21.15.28:0/3869154374 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ecc067f30 con 0x7f8ecc150700 2024-02-21T23:06:44.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.280+0000 7f8ec3e00640 1 -- 172.21.15.28:0/3869154374 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8ec40037b0 con 0x7f8ecc150700 2024-02-21T23:06:44.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.282+0000 7f8ec3e00640 1 --2- 172.21.15.28:0/3869154374 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8ea806b4d0 0x7f8ea806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:44.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.282+0000 7f8ec3e00640 1 -- 172.21.15.28:0/3869154374 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f8ec408a4c0 con 0x7f8ecc150700 2024-02-21T23:06:44.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.282+0000 7f8ecb400640 1 --2- 172.21.15.28:0/3869154374 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8ea806b4d0 0x7f8ea806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:44.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.284+0000 7f8ecb400640 1 --2- 172.21.15.28:0/3869154374 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8ea806b4d0 0x7f8ea806d990 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc036000 tx=0x7f8ebc030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:44.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.286+0000 7f8ec3e00640 1 -- 172.21.15.28:0/3869154374 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f8ec4059320 con 0x7f8ecc150700 2024-02-21T23:06:44.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.496+0000 7f8ed2800640 1 -- 172.21.15.28:0/3869154374 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f8ecc0c53e0 con 0x7f8ecc150700 2024-02-21T23:06:44.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.503+0000 7f8ec3e00640 1 -- 172.21.15.28:0/3869154374 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v30) v1 ==== 104+0+0 (secure 0 0 0) 0x7f8ec4058c20 con 0x7f8ecc150700 2024-02-21T23:06:44.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.506+0000 7f8ed2800640 1 -- 172.21.15.28:0/3869154374 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8ea806b4d0 msgr2=0x7f8ea806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:44.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.506+0000 7f8ed2800640 1 --2- 172.21.15.28:0/3869154374 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8ea806b4d0 0x7f8ea806d990 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc036000 tx=0x7f8ebc030000 comp rx=0 tx=0).stop 2024-02-21T23:06:44.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.506+0000 7f8ed2800640 1 -- 172.21.15.28:0/3869154374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ecc150700 msgr2=0x7f8ecc1465f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:44.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.506+0000 7f8ed2800640 1 --2- 172.21.15.28:0/3869154374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ecc150700 0x7f8ecc1465f0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f8ec400d5c0 tx=0x7f8ec400da80 comp rx=0 tx=0).stop 2024-02-21T23:06:44.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.506+0000 7f8ed2800640 1 -- 172.21.15.28:0/3869154374 shutdown_connections 2024-02-21T23:06:44.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.506+0000 7f8ed2800640 1 --2- 172.21.15.28:0/3869154374 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8ea806b4d0 0x7f8ea806d990 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:44.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.506+0000 7f8ed2800640 1 --2- 172.21.15.28:0/3869154374 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8ecc0c5d80 0x7f8ecc154150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:44.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.506+0000 7f8ed2800640 1 --2- 172.21.15.28:0/3869154374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ecc150700 0x7f8ecc1465f0 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:44.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.506+0000 7f8ed2800640 1 --2- 172.21.15.28:0/3869154374 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8ecc06b0e0 0x7f8ecc14f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:44.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.506+0000 7f8ed2800640 1 -- 172.21.15.28:0/3869154374 >> 172.21.15.28:0/3869154374 conn(0x7f8ecc0a85a0 msgr2=0x7f8ecc14ab70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:44.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.507+0000 7f8ed2800640 1 -- 172.21.15.28:0/3869154374 shutdown_connections 2024-02-21T23:06:44.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.507+0000 7f8ed2800640 1 -- 172.21.15.28:0/3869154374 wait complete. 2024-02-21T23:06:44.516 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_last_resort_refused' 2024-02-21T23:06:44.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.750+0000 7fe1d3e00640 1 Processor -- start 2024-02-21T23:06:44.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.751+0000 7fe1d3e00640 1 -- start start 2024-02-21T23:06:44.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.752+0000 7fe1d3e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe1cc153950 0x7fe1cc0c15d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:44.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.752+0000 7fe1d3e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1cc152e70 con 0x7fe1cc0c11f0 2024-02-21T23:06:44.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.752+0000 7fe1d3e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1cc153150 con 0x7fe1cc15b6d0 2024-02-21T23:06:44.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.752+0000 7fe1d3e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1cc0a5ec0 con 0x7fe1cc153950 2024-02-21T23:06:44.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.753+0000 7fe1d1400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe1cc0c11f0 0x7fe1cc06dce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59588/0 (socket says 172.21.15.28:59588) 2024-02-21T23:06:44.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.753+0000 7fe1d1400640 1 -- 172.21.15.28:0/1298219432 learned_addr learned my addr 172.21.15.28:0/1298219432 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:06:44.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.753+0000 7fe1d0a00640 1 --2- 172.21.15.28:0/1298219432 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe1cc153950 0x7fe1cc0c15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:44.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.753+0000 7fe1cbe00640 1 -- 172.21.15.28:0/1298219432 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3161299410 0 0) 0x7fe1cc152e70 con 0x7fe1cc0c11f0 2024-02-21T23:06:44.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.753+0000 7fe1cbe00640 1 -- 172.21.15.28:0/1298219432 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1b4003520 con 0x7fe1cc0c11f0 2024-02-21T23:06:44.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.754+0000 7fe1cbe00640 1 -- 172.21.15.28:0/1298219432 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3735427091 0 0) 0x7fe1b4003520 con 0x7fe1cc0c11f0 2024-02-21T23:06:44.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.754+0000 7fe1cbe00640 1 -- 172.21.15.28:0/1298219432 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe1cc153950 msgr2=0x7fe1cc0c15d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:44.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.754+0000 7fe1cbe00640 1 --2- 172.21.15.28:0/1298219432 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe1cc153950 0x7fe1cc0c15d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:44.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.754+0000 7fe1cbe00640 1 -- 172.21.15.28:0/1298219432 >> v1:172.21.15.81:6789/0 conn(0x7fe1cc15b6d0 legacy=0x7fe1cc15dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:44.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.754+0000 7fe1cbe00640 1 -- 172.21.15.28:0/1298219432 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1cc153430 con 0x7fe1cc0c11f0 2024-02-21T23:06:44.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.754+0000 7fe1cbe00640 1 -- 172.21.15.28:0/1298219432 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fe1c0001670 con 0x7fe1cc0c11f0 2024-02-21T23:06:44.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.754+0000 7fe1cbe00640 1 -- 172.21.15.28:0/1298219432 >> v1:172.21.15.28:6789/0 conn(0x7fe1cc0c11f0 legacy=0x7fe1cc06dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:44.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.754+0000 7fe1cbe00640 1 --2- 172.21.15.28:0/1298219432 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe1b40039e0 0x7fe1b4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:44.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.754+0000 7fe1cbe00640 1 --2- 172.21.15.28:0/1298219432 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe1cc153950 0x7fe1b4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:44.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.754+0000 7fe1cbe00640 1 --2- 172.21.15.28:0/1298219432 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1b4004d40 0x7fe1b40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:44.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.754+0000 7fe1cbe00640 1 -- 172.21.15.28:0/1298219432 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1cc0a5ec0 con 0x7fe1b4004d40 2024-02-21T23:06:44.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.754+0000 7fe1cbe00640 1 -- 172.21.15.28:0/1298219432 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b400bb50 con 0x7fe1b40039e0 2024-02-21T23:06:44.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.754+0000 7fe1cbe00640 1 -- 172.21.15.28:0/1298219432 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b400bd50 con 0x7fe1cc153950 2024-02-21T23:06:44.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.754+0000 7fe1d1e00640 1 --2- 172.21.15.28:0/1298219432 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1b4004d40 0x7fe1b40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:44.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.754+0000 7fe1d0a00640 1 --2- 172.21.15.28:0/1298219432 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe1cc153950 0x7fe1b4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:44.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.754+0000 7fe1d1400640 1 --2- 172.21.15.28:0/1298219432 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe1b40039e0 0x7fe1b4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:44.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.755+0000 7fe1d1e00640 1 -- 172.21.15.28:0/1298219432 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe1cc153950 msgr2=0x7fe1b4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:44.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.755+0000 7fe1d1e00640 1 --2- 172.21.15.28:0/1298219432 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe1cc153950 0x7fe1b4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:44.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.755+0000 7fe1d1e00640 1 -- 172.21.15.28:0/1298219432 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe1b40039e0 msgr2=0x7fe1b4003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:44.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.755+0000 7fe1d1e00640 1 --2- 172.21.15.28:0/1298219432 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe1b40039e0 0x7fe1b4003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:44.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.755+0000 7fe1d1e00640 1 -- 172.21.15.28:0/1298219432 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1b4003520 con 0x7fe1b4004d40 2024-02-21T23:06:44.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.756+0000 7fe1d1e00640 1 --2- 172.21.15.28:0/1298219432 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1b4004d40 0x7fe1b40051c0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc06fad0 tx=0x7fe1b4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b620b46d61caf829 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:44.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.756+0000 7fe1cbe00640 1 -- 172.21.15.28:0/1298219432 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1c4001a70 con 0x7fe1b4004d40 2024-02-21T23:06:44.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.756+0000 7fe1cbe00640 1 -- 172.21.15.28:0/1298219432 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1c4003b70 con 0x7fe1b4004d40 2024-02-21T23:06:44.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.756+0000 7fe1cbe00640 1 -- 172.21.15.28:0/1298219432 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1c40050d0 con 0x7fe1b4004d40 2024-02-21T23:06:44.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.756+0000 7fe1d3e00640 1 -- 172.21.15.28:0/1298219432 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1b4004d40 msgr2=0x7fe1b40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:44.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.756+0000 7fe1d3e00640 1 --2- 172.21.15.28:0/1298219432 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1b4004d40 0x7fe1b40051c0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc06fad0 tx=0x7fe1b4005830 comp rx=0 tx=0).stop 2024-02-21T23:06:44.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.756+0000 7fe1d1e00640 1 -- 172.21.15.28:0/1298219432 reap_dead start 2024-02-21T23:06:44.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.757+0000 7fe1d3e00640 1 -- 172.21.15.28:0/1298219432 shutdown_connections 2024-02-21T23:06:44.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.757+0000 7fe1d3e00640 1 -- 172.21.15.28:0/1298219432 >> 172.21.15.28:0/1298219432 conn(0x7fe1cc0c67f0 msgr2=0x7fe1cc0c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:44.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.757+0000 7fe1d3e00640 1 -- 172.21.15.28:0/1298219432 shutdown_connections 2024-02-21T23:06:44.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.757+0000 7fe1d3e00640 1 -- 172.21.15.28:0/1298219432 wait complete. 2024-02-21T23:06:44.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.758+0000 7fe1d3e00640 1 Processor -- start 2024-02-21T23:06:44.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.759+0000 7fe1d3e00640 1 -- start start 2024-02-21T23:06:44.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.760+0000 7fe1d3e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe1b4004d40 0x7fe1cc1428d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:44.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.760+0000 7fe1d3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1cc1e8b50 0x7fe1cc0ac100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:44.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.760+0000 7fe1d3e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe1cc0a6390 0x7fe1cc0aa960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:44.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.760+0000 7fe1d3e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1cc0a6110 con 0x7fe1cc1e8b50 2024-02-21T23:06:44.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.760+0000 7fe1d3e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1cc0c4c00 con 0x7fe1b4004d40 2024-02-21T23:06:44.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.760+0000 7fe1d3e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1cc0c0d90 con 0x7fe1cc0a6390 2024-02-21T23:06:44.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.760+0000 7fe1d0a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1cc1e8b50 0x7fe1cc0ac100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:44.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.760+0000 7fe1d0a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1cc1e8b50 0x7fe1cc0ac100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59448/0 (socket says 172.21.15.28:59448) 2024-02-21T23:06:44.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.760+0000 7fe1d0a00640 1 -- 172.21.15.28:0/1455133394 learned_addr learned my addr 172.21.15.28:0/1455133394 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:44.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.760+0000 7fe1d1e00640 1 --2- 172.21.15.28:0/1455133394 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe1cc0a6390 0x7fe1cc0aa960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:44.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.760+0000 7fe1d1400640 1 --2- 172.21.15.28:0/1455133394 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe1b4004d40 0x7fe1cc1428d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:44.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.760+0000 7fe1d0a00640 1 -- 172.21.15.28:0/1455133394 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe1cc0a6390 msgr2=0x7fe1cc0aa960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:44.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.760+0000 7fe1d0a00640 1 --2- 172.21.15.28:0/1455133394 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe1cc0a6390 0x7fe1cc0aa960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:44.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.760+0000 7fe1d0a00640 1 -- 172.21.15.28:0/1455133394 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe1b4004d40 msgr2=0x7fe1cc1428d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:44.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.760+0000 7fe1d0a00640 1 --2- 172.21.15.28:0/1455133394 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe1b4004d40 0x7fe1cc1428d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:44.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.760+0000 7fe1d0a00640 1 -- 172.21.15.28:0/1455133394 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1cc1e8fd0 con 0x7fe1cc1e8b50 2024-02-21T23:06:44.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.761+0000 7fe1d0a00640 1 --2- 172.21.15.28:0/1455133394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1cc1e8b50 0x7fe1cc0ac100 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7fe1b8011600 tx=0x7fe1b8011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:44.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.761+0000 7fe1c9600640 1 -- 172.21.15.28:0/1455133394 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1b800f8c0 con 0x7fe1cc1e8b50 2024-02-21T23:06:44.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.761+0000 7fe1c9600640 1 -- 172.21.15.28:0/1455133394 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1b800fa60 con 0x7fe1cc1e8b50 2024-02-21T23:06:44.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.761+0000 7fe1d3e00640 1 -- 172.21.15.28:0/1455133394 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1cc144ba0 con 0x7fe1cc1e8b50 2024-02-21T23:06:44.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.761+0000 7fe1c9600640 1 -- 172.21.15.28:0/1455133394 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1b800fe10 con 0x7fe1cc1e8b50 2024-02-21T23:06:44.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.761+0000 7fe1d3e00640 1 -- 172.21.15.28:0/1455133394 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1cc0a6e80 con 0x7fe1cc1e8b50 2024-02-21T23:06:44.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.762+0000 7fe1d3e00640 1 -- 172.21.15.28:0/1455133394 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1cc0a6ca0 con 0x7fe1cc1e8b50 2024-02-21T23:06:44.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.763+0000 7fe1c9600640 1 -- 172.21.15.28:0/1455133394 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe1b8015460 con 0x7fe1cc1e8b50 2024-02-21T23:06:44.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.764+0000 7fe1c9600640 1 --2- 172.21.15.28:0/1455133394 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe1ac06b4d0 0x7fe1ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:44.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.764+0000 7fe1c9600640 1 -- 172.21.15.28:0/1455133394 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5593+0+0 (secure 0 0 0) 0x7fe1b808aac0 con 0x7fe1cc1e8b50 2024-02-21T23:06:44.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.765+0000 7fe1d1400640 1 --2- 172.21.15.28:0/1455133394 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe1ac06b4d0 0x7fe1ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:44.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.766+0000 7fe1d1400640 1 --2- 172.21.15.28:0/1455133394 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe1ac06b4d0 0x7fe1ac06d990 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fe1c0001cc0 tx=0x7fe1c0001d90 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:44.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:44.769+0000 7fe1c9600640 1 -- 172.21.15.28:0/1455133394 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe1cc0a6ca0 con 0x7fe1cc1e8b50 2024-02-21T23:06:45.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.048+0000 7fe1d3e00640 1 -- 172.21.15.28:0/1455133394 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_last_resort_refused"]} v 0) v1 -- 0x7fe1cc067f30 con 0x7fe1cc1e8b50 2024-02-21T23:06:45.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.648+0000 7fe1c9600640 1 -- 172.21.15.28:0/1455133394 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_last_resort_refused"]}]=0 v253) v1 ==== 158+0+0 (secure 0 0 0) 0x7fe1cc1e8fd0 con 0x7fe1cc1e8b50 2024-02-21T23:06:45.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.652+0000 7fe1d3e00640 1 -- 172.21.15.28:0/1455133394 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe1ac06b4d0 msgr2=0x7fe1ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:45.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.652+0000 7fe1d3e00640 1 --2- 172.21.15.28:0/1455133394 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe1ac06b4d0 0x7fe1ac06d990 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fe1c0001cc0 tx=0x7fe1c0001d90 comp rx=0 tx=0).stop 2024-02-21T23:06:45.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.652+0000 7fe1d3e00640 1 -- 172.21.15.28:0/1455133394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1cc1e8b50 msgr2=0x7fe1cc0ac100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:45.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.652+0000 7fe1d3e00640 1 --2- 172.21.15.28:0/1455133394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1cc1e8b50 0x7fe1cc0ac100 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7fe1b8011600 tx=0x7fe1b8011ac0 comp rx=0 tx=0).stop 2024-02-21T23:06:45.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.652+0000 7fe1d3e00640 1 -- 172.21.15.28:0/1455133394 shutdown_connections 2024-02-21T23:06:45.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.652+0000 7fe1d3e00640 1 --2- 172.21.15.28:0/1455133394 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe1ac06b4d0 0x7fe1ac06d990 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:45.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.652+0000 7fe1d3e00640 1 --2- 172.21.15.28:0/1455133394 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe1cc0a6390 0x7fe1cc0aa960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:45.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.652+0000 7fe1d3e00640 1 --2- 172.21.15.28:0/1455133394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1cc1e8b50 0x7fe1cc0ac100 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:45.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.652+0000 7fe1d3e00640 1 --2- 172.21.15.28:0/1455133394 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe1b4004d40 0x7fe1cc1428d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:45.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.652+0000 7fe1d3e00640 1 -- 172.21.15.28:0/1455133394 >> 172.21.15.28:0/1455133394 conn(0x7fe1cc0c67f0 msgr2=0x7fe1cc148b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:45.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.652+0000 7fe1d3e00640 1 -- 172.21.15.28:0/1455133394 shutdown_connections 2024-02-21T23:06:45.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.652+0000 7fe1d3e00640 1 -- 172.21.15.28:0/1455133394 wait complete. 2024-02-21T23:06:45.664 INFO:tasks.cephfs_test_runner:That a standby with mds_join_fs set to another fs is not used if refuse_standby_for_another_fs is set. ... ok 2024-02-21T23:06:45.665 INFO:tasks.cephfs_test_runner:Starting test: test_join_fs_runtime (tasks.cephfs.test_failover.TestClusterAffinity) That setting mds_join_fs at runtime affects the cluster layout. 2024-02-21T23:06:45.665 INFO:tasks.cephfs_test_runner:test_join_fs_runtime (tasks.cephfs.test_failover.TestClusterAffinity) 2024-02-21T23:06:45.665 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:06:45.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.889+0000 7fba4a200640 1 Processor -- start 2024-02-21T23:06:45.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.889+0000 7fba4a200640 1 -- start start 2024-02-21T23:06:45.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.890+0000 7fba4a200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fba4414f0b0 0x7fba44155180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:45.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.890+0000 7fba4a200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fba440a6160 0x7fba440c2010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:45.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.890+0000 7fba4a200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba44155760 con 0x7fba440c2550 2024-02-21T23:06:45.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.890+0000 7fba4a200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba44151d30 con 0x7fba440a6160 2024-02-21T23:06:45.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.890+0000 7fba4a200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba44155a40 con 0x7fba4414f0b0 2024-02-21T23:06:45.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.891+0000 7fba49800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fba440c2550 0x7fba441521c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34208/0 (socket says 172.21.15.28:34208) 2024-02-21T23:06:45.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.891+0000 7fba49800640 1 -- 172.21.15.28:0/969324633 learned_addr learned my addr 172.21.15.28:0/969324633 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:06:45.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.891+0000 7fba43e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fba440a6160 0x7fba440c2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:45.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.891+0000 7fba48e00640 1 --2- 172.21.15.28:0/969324633 >> v2:172.21.15.81:3300/0 conn(0x7fba4414f0b0 0x7fba44155180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:45.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.891+0000 7fba43e00640 1 -- 172.21.15.28:0/969324633 >> v2:172.21.15.81:3300/0 conn(0x7fba4414f0b0 msgr2=0x7fba44155180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:45.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.891+0000 7fba43e00640 1 --2- 172.21.15.28:0/969324633 >> v2:172.21.15.81:3300/0 conn(0x7fba4414f0b0 0x7fba44155180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:45.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.891+0000 7fba43e00640 1 -- 172.21.15.28:0/969324633 >> v1:172.21.15.28:6789/0 conn(0x7fba440c2550 legacy=0x7fba441521c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:45.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.891+0000 7fba43e00640 1 -- 172.21.15.28:0/969324633 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba44155d20 con 0x7fba440a6160 2024-02-21T23:06:45.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.892+0000 7fba43e00640 1 --2- 172.21.15.28:0/969324633 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba440a6160 0x7fba440c2010 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7fba44151eb0 tx=0x7fba2c02dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cceaa113cc42ecd1 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:45.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.893+0000 7fba43400640 1 -- 172.21.15.28:0/969324633 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba2c02f8a0 con 0x7fba440a6160 2024-02-21T23:06:45.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.893+0000 7fba43400640 1 -- 172.21.15.28:0/969324633 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba2c02fa40 con 0x7fba440a6160 2024-02-21T23:06:45.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.893+0000 7fba43400640 1 -- 172.21.15.28:0/969324633 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba2c02fe10 con 0x7fba440a6160 2024-02-21T23:06:45.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.893+0000 7fba4a200640 1 -- 172.21.15.28:0/969324633 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba440a6160 msgr2=0x7fba440c2010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:45.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.893+0000 7fba4a200640 1 --2- 172.21.15.28:0/969324633 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba440a6160 0x7fba440c2010 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7fba44151eb0 tx=0x7fba2c02dae0 comp rx=0 tx=0).stop 2024-02-21T23:06:45.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.893+0000 7fba4a200640 1 -- 172.21.15.28:0/969324633 shutdown_connections 2024-02-21T23:06:45.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.893+0000 7fba4a200640 1 --2- 172.21.15.28:0/969324633 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba440a6160 0x7fba440c2010 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:45.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.893+0000 7fba4a200640 1 --2- 172.21.15.28:0/969324633 >> v2:172.21.15.81:3300/0 conn(0x7fba4414f0b0 0x7fba44155180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:45.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.893+0000 7fba4a200640 1 -- 172.21.15.28:0/969324633 >> 172.21.15.28:0/969324633 conn(0x7fba4406b630 msgr2=0x7fba440c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:45.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.894+0000 7fba4a200640 1 -- 172.21.15.28:0/969324633 shutdown_connections 2024-02-21T23:06:45.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.894+0000 7fba4a200640 1 -- 172.21.15.28:0/969324633 wait complete. 2024-02-21T23:06:45.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.896+0000 7fba4a200640 1 Processor -- start 2024-02-21T23:06:45.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.896+0000 7fba4a200640 1 -- start start 2024-02-21T23:06:45.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.897+0000 7fba4a200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fba440a6160 0x7fba440c4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:45.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.897+0000 7fba4a200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fba440c2550 0x7fba44153ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:45.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.897+0000 7fba4a200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba4414f0b0 0x7fba441ede50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:45.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.897+0000 7fba4a200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba440a7f70 con 0x7fba4414f0b0 2024-02-21T23:06:45.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.897+0000 7fba4a200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba440a68d0 con 0x7fba440c2550 2024-02-21T23:06:45.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.897+0000 7fba4a200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba440a80f0 con 0x7fba440a6160 2024-02-21T23:06:45.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.898+0000 7fba49800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba4414f0b0 0x7fba441ede50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:45.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.898+0000 7fba49800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba4414f0b0 0x7fba441ede50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36170/0 (socket says 172.21.15.28:36170) 2024-02-21T23:06:45.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.898+0000 7fba49800640 1 -- 172.21.15.28:0/3331396598 learned_addr learned my addr 172.21.15.28:0/3331396598 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:45.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.898+0000 7fba48e00640 1 --2- 172.21.15.28:0/3331396598 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fba440a6160 0x7fba440c4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:45.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.898+0000 7fba43e00640 1 --2- 172.21.15.28:0/3331396598 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fba440c2550 0x7fba44153ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:45.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.898+0000 7fba49800640 1 -- 172.21.15.28:0/3331396598 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fba440a6160 msgr2=0x7fba440c4e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:45.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.898+0000 7fba49800640 1 --2- 172.21.15.28:0/3331396598 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fba440a6160 0x7fba440c4e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:45.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.898+0000 7fba49800640 1 -- 172.21.15.28:0/3331396598 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fba440c2550 msgr2=0x7fba44153ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:45.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.898+0000 7fba49800640 1 --2- 172.21.15.28:0/3331396598 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fba440c2550 0x7fba44153ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:45.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.898+0000 7fba49800640 1 -- 172.21.15.28:0/3331396598 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba44152f80 con 0x7fba4414f0b0 2024-02-21T23:06:45.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.898+0000 7fba43e00640 1 --2- 172.21.15.28:0/3331396598 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fba440c2550 0x7fba44153ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:06:45.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.899+0000 7fba49800640 1 --2- 172.21.15.28:0/3331396598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba4414f0b0 0x7fba441ede50 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7fba3c00cbc0 tx=0x7fba3c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:45.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.899+0000 7fba40c00640 1 -- 172.21.15.28:0/3331396598 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba3c017070 con 0x7fba4414f0b0 2024-02-21T23:06:45.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.899+0000 7fba40c00640 1 -- 172.21.15.28:0/3331396598 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba3c00ad40 con 0x7fba4414f0b0 2024-02-21T23:06:45.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.899+0000 7fba40c00640 1 -- 172.21.15.28:0/3331396598 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba3c012600 con 0x7fba4414f0b0 2024-02-21T23:06:45.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.899+0000 7fba4a200640 1 -- 172.21.15.28:0/3331396598 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba440c5b40 con 0x7fba4414f0b0 2024-02-21T23:06:45.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.899+0000 7fba4a200640 1 -- 172.21.15.28:0/3331396598 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba44154420 con 0x7fba4414f0b0 2024-02-21T23:06:45.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.900+0000 7fba4a200640 1 -- 172.21.15.28:0/3331396598 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba4404f290 con 0x7fba4414f0b0 2024-02-21T23:06:45.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.901+0000 7fba40c00640 1 -- 172.21.15.28:0/3331396598 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fba3c01a070 con 0x7fba4414f0b0 2024-02-21T23:06:45.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.902+0000 7fba40c00640 1 --2- 172.21.15.28:0/3331396598 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fba2006b4d0 0x7fba2006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:45.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.902+0000 7fba40c00640 1 -- 172.21.15.28:0/3331396598 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5593+0+0 (secure 0 0 0) 0x7fba3c089e10 con 0x7fba4414f0b0 2024-02-21T23:06:45.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.903+0000 7fba48e00640 1 --2- 172.21.15.28:0/3331396598 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fba2006b4d0 0x7fba2006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:45.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.904+0000 7fba48e00640 1 --2- 172.21.15.28:0/3331396598 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fba2006b4d0 0x7fba2006d990 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fba34006fd0 tx=0x7fba34008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:45.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:45.908+0000 7fba40c00640 1 -- 172.21.15.28:0/3331396598 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fba3c08d030 con 0x7fba4414f0b0 2024-02-21T23:06:46.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.108+0000 7fba4a200640 1 -- 172.21.15.28:0/3331396598 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fba440c1d60 con 0x7fba4414f0b0 2024-02-21T23:06:46.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.110+0000 7fba40c00640 1 -- 172.21.15.28:0/3331396598 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+13620 (secure 0 0 0) 0x7fba3c0586a0 con 0x7fba4414f0b0 2024-02-21T23:06:46.099 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:06:46.099 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":90,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:06:42.610731+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.81:6834/2774951107":"2024-02-22T23:06:39.594368+0000","172.21.15.81:6835/2774951107":"2024-02-22T23:06:39.594368+0000","172.21.15.28:6833/624384999":"2024-02-22T23:06:32.469969+0000","172.21.15.28:6832/624384999":"2024-02-22T23:06:32.469969+0000","172.21.15.28:6835/1066741568":"2024-02-22T23:06:31.975858+0000","172.21.15.28:6834/1066741568":"2024-02-22T23:06:31.975858+0000","172.21.15.28:6835/595682298":"2024-02-22T23:06:23.690106+0000","172.21.15.28:6834/595682298":"2024-02-22T23:06:23.690106+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:06:46.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.112+0000 7fba4a200640 1 -- 172.21.15.28:0/3331396598 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fba2006b4d0 msgr2=0x7fba2006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:46.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.112+0000 7fba4a200640 1 --2- 172.21.15.28:0/3331396598 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fba2006b4d0 0x7fba2006d990 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fba34006fd0 tx=0x7fba34008040 comp rx=0 tx=0).stop 2024-02-21T23:06:46.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.112+0000 7fba4a200640 1 -- 172.21.15.28:0/3331396598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba4414f0b0 msgr2=0x7fba441ede50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:46.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.112+0000 7fba4a200640 1 --2- 172.21.15.28:0/3331396598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba4414f0b0 0x7fba441ede50 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7fba3c00cbc0 tx=0x7fba3c00a430 comp rx=0 tx=0).stop 2024-02-21T23:06:46.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.112+0000 7fba4a200640 1 -- 172.21.15.28:0/3331396598 shutdown_connections 2024-02-21T23:06:46.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.112+0000 7fba4a200640 1 --2- 172.21.15.28:0/3331396598 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fba2006b4d0 0x7fba2006d990 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:46.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.112+0000 7fba4a200640 1 --2- 172.21.15.28:0/3331396598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba4414f0b0 0x7fba441ede50 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:46.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.112+0000 7fba4a200640 1 --2- 172.21.15.28:0/3331396598 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fba440c2550 0x7fba44153ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:46.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.112+0000 7fba4a200640 1 --2- 172.21.15.28:0/3331396598 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fba440a6160 0x7fba440c4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:46.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.112+0000 7fba4a200640 1 -- 172.21.15.28:0/3331396598 >> 172.21.15.28:0/3331396598 conn(0x7fba4406b630 msgr2=0x7fba440c3630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:46.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.112+0000 7fba4a200640 1 -- 172.21.15.28:0/3331396598 shutdown_connections 2024-02-21T23:06:46.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.112+0000 7fba4a200640 1 -- 172.21.15.28:0/3331396598 wait complete. 2024-02-21T23:06:46.120 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-21T23:02:30.728329+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-21T23:06:46.123 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-21T23:06:46.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.358+0000 7fde47e00640 1 Processor -- start 2024-02-21T23:06:46.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.359+0000 7fde47e00640 1 -- start start 2024-02-21T23:06:46.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.360+0000 7fde47e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fde4813e0c0 0x7fde480c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:46.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.360+0000 7fde47e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fde4806dbc0 0x7fde4806c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:46.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.360+0000 7fde47e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde48158550 con 0x7fde480c76f0 2024-02-21T23:06:46.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.360+0000 7fde47e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde480c89a0 con 0x7fde4806dbc0 2024-02-21T23:06:46.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.360+0000 7fde47e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde481587d0 con 0x7fde4813e0c0 2024-02-21T23:06:46.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.361+0000 7fde46000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fde4806dbc0 0x7fde4806c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:46.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.361+0000 7fde46000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fde4806dbc0 0x7fde4806c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36182/0 (socket says 172.21.15.28:36182) 2024-02-21T23:06:46.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.361+0000 7fde46000640 1 -- 172.21.15.28:0/4060516829 learned_addr learned my addr 172.21.15.28:0/4060516829 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:46.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.361+0000 7fde46a00640 1 --2- 172.21.15.28:0/4060516829 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fde4813e0c0 0x7fde480c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:46.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.361+0000 7fde46000640 1 -- 172.21.15.28:0/4060516829 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fde4813e0c0 msgr2=0x7fde480c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:46.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.361+0000 7fde46000640 1 --2- 172.21.15.28:0/4060516829 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fde4813e0c0 0x7fde480c71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:46.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.361+0000 7fde46000640 1 -- 172.21.15.28:0/4060516829 >> v1:172.21.15.81:6789/0 conn(0x7fde480c76f0 legacy=0x7fde4806c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:46.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.361+0000 7fde46000640 1 -- 172.21.15.28:0/4060516829 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde48158ab0 con 0x7fde4806dbc0 2024-02-21T23:06:46.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.361+0000 7fde46a00640 1 --2- 172.21.15.28:0/4060516829 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fde4813e0c0 0x7fde480c71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:06:46.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.363+0000 7fde46000640 1 --2- 172.21.15.28:0/4060516829 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde4806dbc0 0x7fde4806c280 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7fde38009870 tx=0x7fde3802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eeb1294d275c1e3b server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:46.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.363+0000 7fde45600640 1 -- 172.21.15.28:0/4060516829 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde38035020 con 0x7fde4806dbc0 2024-02-21T23:06:46.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.363+0000 7fde45600640 1 -- 172.21.15.28:0/4060516829 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde38031640 con 0x7fde4806dbc0 2024-02-21T23:06:46.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.363+0000 7fde45600640 1 -- 172.21.15.28:0/4060516829 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde3802d040 con 0x7fde4806dbc0 2024-02-21T23:06:46.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.363+0000 7fde47e00640 1 -- 172.21.15.28:0/4060516829 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde4806dbc0 msgr2=0x7fde4806c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:46.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.363+0000 7fde47e00640 1 --2- 172.21.15.28:0/4060516829 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde4806dbc0 0x7fde4806c280 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7fde38009870 tx=0x7fde3802b8f0 comp rx=0 tx=0).stop 2024-02-21T23:06:46.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.364+0000 7fde47e00640 1 -- 172.21.15.28:0/4060516829 shutdown_connections 2024-02-21T23:06:46.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.364+0000 7fde47e00640 1 --2- 172.21.15.28:0/4060516829 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde4806dbc0 0x7fde4806c280 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:46.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.364+0000 7fde47e00640 1 --2- 172.21.15.28:0/4060516829 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fde4813e0c0 0x7fde480c71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:46.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.364+0000 7fde47e00640 1 -- 172.21.15.28:0/4060516829 >> 172.21.15.28:0/4060516829 conn(0x7fde48073110 msgr2=0x7fde480ac490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:46.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.364+0000 7fde47e00640 1 -- 172.21.15.28:0/4060516829 shutdown_connections 2024-02-21T23:06:46.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.364+0000 7fde47e00640 1 -- 172.21.15.28:0/4060516829 wait complete. 2024-02-21T23:06:46.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.365+0000 7fde47e00640 1 Processor -- start 2024-02-21T23:06:46.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.366+0000 7fde47e00640 1 -- start start 2024-02-21T23:06:46.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.366+0000 7fde47e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fde4806dbc0 0x7fde480ab5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:46.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.367+0000 7fde47e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde480c76f0 0x7fde48138360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:46.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.367+0000 7fde47e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde4813e0c0 0x7fde481388a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:46.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.367+0000 7fde47e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde4806ced0 con 0x7fde480c76f0 2024-02-21T23:06:46.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.367+0000 7fde47e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde48159070 con 0x7fde4813e0c0 2024-02-21T23:06:46.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.367+0000 7fde47e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde4813e520 con 0x7fde4806dbc0 2024-02-21T23:06:46.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.367+0000 7fde46000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde480c76f0 0x7fde48138360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:46.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.367+0000 7fde46000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde480c76f0 0x7fde48138360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36194/0 (socket says 172.21.15.28:36194) 2024-02-21T23:06:46.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.367+0000 7fde46000640 1 -- 172.21.15.28:0/1118452032 learned_addr learned my addr 172.21.15.28:0/1118452032 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:46.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.367+0000 7fde46a00640 1 --2- 172.21.15.28:0/1118452032 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fde4806dbc0 0x7fde480ab5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:46.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.367+0000 7fde46000640 1 -- 172.21.15.28:0/1118452032 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fde4806dbc0 msgr2=0x7fde480ab5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:46.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.367+0000 7fde47400640 1 --2- 172.21.15.28:0/1118452032 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde4813e0c0 0x7fde481388a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:46.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.367+0000 7fde46000640 1 --2- 172.21.15.28:0/1118452032 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fde4806dbc0 0x7fde480ab5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:46.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.367+0000 7fde46000640 1 -- 172.21.15.28:0/1118452032 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde4813e0c0 msgr2=0x7fde481388a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:46.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.367+0000 7fde46000640 1 --2- 172.21.15.28:0/1118452032 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde4813e0c0 0x7fde481388a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:46.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.367+0000 7fde46000640 1 -- 172.21.15.28:0/1118452032 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde481ea930 con 0x7fde480c76f0 2024-02-21T23:06:46.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.367+0000 7fde47400640 1 --2- 172.21.15.28:0/1118452032 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde4813e0c0 0x7fde481388a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:06:46.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.368+0000 7fde46000640 1 --2- 172.21.15.28:0/1118452032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde480c76f0 0x7fde48138360 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7fde38000c00 tx=0x7fde380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:46.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.368+0000 7fde3f400640 1 -- 172.21.15.28:0/1118452032 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde38040070 con 0x7fde480c76f0 2024-02-21T23:06:46.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.368+0000 7fde3f400640 1 -- 172.21.15.28:0/1118452032 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde38039bf0 con 0x7fde480c76f0 2024-02-21T23:06:46.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.368+0000 7fde3f400640 1 -- 172.21.15.28:0/1118452032 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde38035040 con 0x7fde480c76f0 2024-02-21T23:06:46.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.368+0000 7fde47e00640 1 -- 172.21.15.28:0/1118452032 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde480ade90 con 0x7fde480c76f0 2024-02-21T23:06:46.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.368+0000 7fde47e00640 1 -- 172.21.15.28:0/1118452032 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde481eaed0 con 0x7fde480c76f0 2024-02-21T23:06:46.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.369+0000 7fde47e00640 1 -- 172.21.15.28:0/1118452032 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde4804f290 con 0x7fde480c76f0 2024-02-21T23:06:46.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.370+0000 7fde3f400640 1 -- 172.21.15.28:0/1118452032 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fde3804b060 con 0x7fde480c76f0 2024-02-21T23:06:46.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.371+0000 7fde3f400640 1 --2- 172.21.15.28:0/1118452032 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fde2406b4d0 0x7fde2406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:46.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.371+0000 7fde3f400640 1 -- 172.21.15.28:0/1118452032 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5593+0+0 (secure 0 0 0) 0x7fde380abc90 con 0x7fde480c76f0 2024-02-21T23:06:46.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.372+0000 7fde46a00640 1 --2- 172.21.15.28:0/1118452032 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fde2406b4d0 0x7fde2406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:46.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.373+0000 7fde46a00640 1 --2- 172.21.15.28:0/1118452032 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fde2406b4d0 0x7fde2406d990 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fde3000a950 tx=0x7fde30008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:46.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.375+0000 7fde3f400640 1 -- 172.21.15.28:0/1118452032 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fde38031610 con 0x7fde480c76f0 2024-02-21T23:06:46.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.576+0000 7fde47e00640 1 -- 172.21.15.28:0/1118452032 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fde4806d9a0 con 0x7fde480c76f0 2024-02-21T23:06:46.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.577+0000 7fde3f400640 1 -- 172.21.15.28:0/1118452032 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) v1 ==== 93+0+10 (secure 0 0 0) 0x7fde3807ab70 con 0x7fde480c76f0 2024-02-21T23:06:46.566 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-02-21T23:06:46.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.579+0000 7fde47e00640 1 -- 172.21.15.28:0/1118452032 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fde2406b4d0 msgr2=0x7fde2406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:46.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.579+0000 7fde47e00640 1 --2- 172.21.15.28:0/1118452032 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fde2406b4d0 0x7fde2406d990 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fde3000a950 tx=0x7fde30008040 comp rx=0 tx=0).stop 2024-02-21T23:06:46.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.579+0000 7fde47e00640 1 -- 172.21.15.28:0/1118452032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde480c76f0 msgr2=0x7fde48138360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:46.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.579+0000 7fde47e00640 1 --2- 172.21.15.28:0/1118452032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde480c76f0 0x7fde48138360 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7fde38000c00 tx=0x7fde380046c0 comp rx=0 tx=0).stop 2024-02-21T23:06:46.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.579+0000 7fde47e00640 1 -- 172.21.15.28:0/1118452032 shutdown_connections 2024-02-21T23:06:46.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.579+0000 7fde47e00640 1 --2- 172.21.15.28:0/1118452032 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fde2406b4d0 0x7fde2406d990 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:46.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.579+0000 7fde47e00640 1 --2- 172.21.15.28:0/1118452032 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde4813e0c0 0x7fde481388a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:46.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.579+0000 7fde47e00640 1 --2- 172.21.15.28:0/1118452032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde480c76f0 0x7fde48138360 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:46.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.579+0000 7fde47e00640 1 --2- 172.21.15.28:0/1118452032 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fde4806dbc0 0x7fde480ab5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:46.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.579+0000 7fde47e00640 1 -- 172.21.15.28:0/1118452032 >> 172.21.15.28:0/1118452032 conn(0x7fde48073110 msgr2=0x7fde481511e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:46.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.580+0000 7fde47e00640 1 -- 172.21.15.28:0/1118452032 shutdown_connections 2024-02-21T23:06:46.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.580+0000 7fde47e00640 1 -- 172.21.15.28:0/1118452032 wait complete. 2024-02-21T23:06:46.589 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_runtime' 2024-02-21T23:06:46.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.820+0000 7f115fe00640 1 Processor -- start 2024-02-21T23:06:46.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.820+0000 7f115fe00640 1 -- start start 2024-02-21T23:06:46.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.821+0000 7f115fe00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f11600c0fe0 0x7f116006e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:46.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.821+0000 7f115fe00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11600a8b40 con 0x7f1160158580 2024-02-21T23:06:46.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.821+0000 7f115fe00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11600a8e00 con 0x7f116006dff0 2024-02-21T23:06:46.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.821+0000 7f115fe00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11600a90e0 con 0x7f11600c0fe0 2024-02-21T23:06:46.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.822+0000 7f115f400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f1160158580 0x7f116014b2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34216/0 (socket says 172.21.15.28:34216) 2024-02-21T23:06:46.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.822+0000 7f115f400640 1 -- 172.21.15.28:0/4195759799 learned_addr learned my addr 172.21.15.28:0/4195759799 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:06:46.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.822+0000 7f1156000640 1 --2- 172.21.15.28:0/4195759799 >> v2:172.21.15.81:3300/0 conn(0x7f11600c0fe0 0x7f116006e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:46.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.822+0000 7f115e000640 1 -- 172.21.15.28:0/4195759799 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2524014872 0 0) 0x7f11600a8b40 con 0x7f1160158580 2024-02-21T23:06:46.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.822+0000 7f115e000640 1 -- 172.21.15.28:0/4195759799 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1144003520 con 0x7f1160158580 2024-02-21T23:06:46.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.823+0000 7f115e000640 1 -- 172.21.15.28:0/4195759799 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1500132132 0 0) 0x7f1144003520 con 0x7f1160158580 2024-02-21T23:06:46.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.823+0000 7f115e000640 1 -- 172.21.15.28:0/4195759799 >> v2:172.21.15.81:3300/0 conn(0x7f11600c0fe0 msgr2=0x7f116006e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:46.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.823+0000 7f115e000640 1 --2- 172.21.15.28:0/4195759799 >> v2:172.21.15.81:3300/0 conn(0x7f11600c0fe0 0x7f116006e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:46.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.823+0000 7f115e000640 1 -- 172.21.15.28:0/4195759799 >> v1:172.21.15.81:6789/0 conn(0x7f116006dff0 legacy=0x7f1160071350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:46.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.823+0000 7f115e000640 1 -- 172.21.15.28:0/4195759799 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1160072ed0 con 0x7f1160158580 2024-02-21T23:06:46.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.823+0000 7f115e000640 1 -- 172.21.15.28:0/4195759799 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f11580019f0 con 0x7f1160158580 2024-02-21T23:06:46.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.823+0000 7f115e000640 1 -- 172.21.15.28:0/4195759799 >> v1:172.21.15.28:6789/0 conn(0x7f1160158580 legacy=0x7f116014b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:46.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.823+0000 7f115e000640 1 --2- 172.21.15.28:0/4195759799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11440039b0 0x7f1144003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:46.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.823+0000 7f115e000640 1 --2- 172.21.15.28:0/4195759799 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1144004800 0x7f1144004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:46.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.823+0000 7f115e000640 1 --2- 172.21.15.28:0/4195759799 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f11440051c0 0x7f114400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:46.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.823+0000 7f115e000640 1 -- 172.21.15.28:0/4195759799 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11600a90e0 con 0x7f11440039b0 2024-02-21T23:06:46.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.823+0000 7f115e000640 1 -- 172.21.15.28:0/4195759799 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f114400c060 con 0x7f11440051c0 2024-02-21T23:06:46.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.823+0000 7f115e000640 1 -- 172.21.15.28:0/4195759799 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f114400c260 con 0x7f1144004800 2024-02-21T23:06:46.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.823+0000 7f115ea00640 1 --2- 172.21.15.28:0/4195759799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11440039b0 0x7f1144003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:46.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.824+0000 7f115f400640 1 --2- 172.21.15.28:0/4195759799 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f11440051c0 0x7f114400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:46.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.824+0000 7f1156000640 1 --2- 172.21.15.28:0/4195759799 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1144004800 0x7f1144004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:46.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.824+0000 7f115ea00640 1 -- 172.21.15.28:0/4195759799 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1144004800 msgr2=0x7f1144004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:46.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.824+0000 7f115ea00640 1 --2- 172.21.15.28:0/4195759799 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1144004800 0x7f1144004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:46.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.824+0000 7f115ea00640 1 -- 172.21.15.28:0/4195759799 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f11440051c0 msgr2=0x7f114400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:46.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.824+0000 7f115ea00640 1 --2- 172.21.15.28:0/4195759799 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f11440051c0 0x7f114400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:46.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.824+0000 7f115ea00640 1 -- 172.21.15.28:0/4195759799 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1144003520 con 0x7f11440039b0 2024-02-21T23:06:46.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.824+0000 7f115f400640 1 -- 172.21.15.28:0/4195759799 reap_dead start 2024-02-21T23:06:46.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.824+0000 7f1156000640 1 --2- 172.21.15.28:0/4195759799 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1144004800 0x7f1144004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:06:46.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.825+0000 7f115ea00640 1 --2- 172.21.15.28:0/4195759799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11440039b0 0x7f1144003dd0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f1150001750 tx=0x7f1144005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea9518a11ab7444c server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:46.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.825+0000 7f115e000640 1 -- 172.21.15.28:0/4195759799 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1150001370 con 0x7f11440039b0 2024-02-21T23:06:46.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.825+0000 7f115e000640 1 -- 172.21.15.28:0/4195759799 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11500052d0 con 0x7f11440039b0 2024-02-21T23:06:46.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.825+0000 7f115e000640 1 -- 172.21.15.28:0/4195759799 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1150003830 con 0x7f11440039b0 2024-02-21T23:06:46.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.826+0000 7f115fe00640 1 -- 172.21.15.28:0/4195759799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11440039b0 msgr2=0x7f1144003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:46.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.826+0000 7f115fe00640 1 --2- 172.21.15.28:0/4195759799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11440039b0 0x7f1144003dd0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f1150001750 tx=0x7f1144005830 comp rx=0 tx=0).stop 2024-02-21T23:06:46.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.826+0000 7f115fe00640 1 -- 172.21.15.28:0/4195759799 shutdown_connections 2024-02-21T23:06:46.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.826+0000 7f115fe00640 1 --2- 172.21.15.28:0/4195759799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11440039b0 0x7f1144003dd0 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:46.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.826+0000 7f115fe00640 1 -- 172.21.15.28:0/4195759799 >> 172.21.15.28:0/4195759799 conn(0x7f116006b630 msgr2=0x7f11600c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:46.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.826+0000 7f115fe00640 1 -- 172.21.15.28:0/4195759799 shutdown_connections 2024-02-21T23:06:46.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.826+0000 7f115fe00640 1 -- 172.21.15.28:0/4195759799 wait complete. 2024-02-21T23:06:46.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.827+0000 7f115fe00640 1 Processor -- start 2024-02-21T23:06:46.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.828+0000 7f115fe00640 1 -- start start 2024-02-21T23:06:46.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.828+0000 7f115fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11440039b0 0x7f11601eb0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:46.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.828+0000 7f115fe00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f11601eb620 0x7f11601ec370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:46.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.828+0000 7f115fe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f11601ec8b0 0x7f1160147730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:46.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.828+0000 7f115fe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1160158980 con 0x7f11440039b0 2024-02-21T23:06:46.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.828+0000 7f115fe00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1160073520 con 0x7f11601ec8b0 2024-02-21T23:06:46.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.828+0000 7f115fe00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1160158b00 con 0x7f11601eb620 2024-02-21T23:06:46.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.829+0000 7f115ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11440039b0 0x7f11601eb0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:46.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.829+0000 7f115ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11440039b0 0x7f11601eb0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36218/0 (socket says 172.21.15.28:36218) 2024-02-21T23:06:46.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.829+0000 7f115ea00640 1 -- 172.21.15.28:0/1094243611 learned_addr learned my addr 172.21.15.28:0/1094243611 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:46.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.829+0000 7f1156000640 1 --2- 172.21.15.28:0/1094243611 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f11601eb620 0x7f11601ec370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:46.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.829+0000 7f115f400640 1 --2- 172.21.15.28:0/1094243611 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f11601ec8b0 0x7f1160147730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:46.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.829+0000 7f115ea00640 1 -- 172.21.15.28:0/1094243611 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f11601eb620 msgr2=0x7f11601ec370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:46.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.829+0000 7f115ea00640 1 --2- 172.21.15.28:0/1094243611 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f11601eb620 0x7f11601ec370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:46.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.829+0000 7f115ea00640 1 -- 172.21.15.28:0/1094243611 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f11601ec8b0 msgr2=0x7f1160147730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:46.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.829+0000 7f115ea00640 1 --2- 172.21.15.28:0/1094243611 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f11601ec8b0 0x7f1160147730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:46.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.829+0000 7f115ea00640 1 -- 172.21.15.28:0/1094243611 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11600aa460 con 0x7f11440039b0 2024-02-21T23:06:46.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.829+0000 7f1156000640 1 --2- 172.21.15.28:0/1094243611 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f11601eb620 0x7f11601ec370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:06:46.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.829+0000 7f115f400640 1 --2- 172.21.15.28:0/1094243611 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f11601ec8b0 0x7f1160147730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:06:46.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.830+0000 7f115ea00640 1 --2- 172.21.15.28:0/1094243611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11440039b0 0x7f11601eb0e0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f1150005040 tx=0x7f1150001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:46.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.830+0000 7f1157400640 1 -- 172.21.15.28:0/1094243611 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1150041020 con 0x7f11440039b0 2024-02-21T23:06:46.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.830+0000 7f1157400640 1 -- 172.21.15.28:0/1094243611 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1150005480 con 0x7f11440039b0 2024-02-21T23:06:46.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.830+0000 7f1157400640 1 -- 172.21.15.28:0/1094243611 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1150007a30 con 0x7f11440039b0 2024-02-21T23:06:46.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.830+0000 7f115fe00640 1 -- 172.21.15.28:0/1094243611 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11600aa740 con 0x7f11440039b0 2024-02-21T23:06:46.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.830+0000 7f115fe00640 1 -- 172.21.15.28:0/1094243611 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11600aad00 con 0x7f11440039b0 2024-02-21T23:06:46.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.831+0000 7f115fe00640 1 -- 172.21.15.28:0/1094243611 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f116004f290 con 0x7f11440039b0 2024-02-21T23:06:46.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.832+0000 7f1157400640 1 -- 172.21.15.28:0/1094243611 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1150044020 con 0x7f11440039b0 2024-02-21T23:06:46.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.833+0000 7f1157400640 1 --2- 172.21.15.28:0/1094243611 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f113c06b500 0x7f113c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:46.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.833+0000 7f1157400640 1 -- 172.21.15.28:0/1094243611 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f11500abbd0 con 0x7f11440039b0 2024-02-21T23:06:46.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.834+0000 7f1156000640 1 --2- 172.21.15.28:0/1094243611 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f113c06b500 0x7f113c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:46.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.835+0000 7f1156000640 1 --2- 172.21.15.28:0/1094243611 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f113c06b500 0x7f113c06d9c0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f1160152cd0 tx=0x7f1148008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:46.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:46.837+0000 7f1157400640 1 -- 172.21.15.28:0/1094243611 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f115004b030 con 0x7f11440039b0 2024-02-21T23:06:47.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.100+0000 7f115fe00640 1 -- 172.21.15.28:0/1094243611 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_runtime"]} v 0) v1 -- 0x7f11600a8320 con 0x7f11440039b0 2024-02-21T23:06:47.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.661+0000 7f1157400640 1 -- 172.21.15.28:0/1094243611 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_runtime"]}]=0 v255) v1 ==== 149+0+0 (secure 0 0 0) 0x7f11500303c0 con 0x7f11440039b0 2024-02-21T23:06:47.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.664+0000 7f115fe00640 1 -- 172.21.15.28:0/1094243611 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f113c06b500 msgr2=0x7f113c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:47.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.664+0000 7f115fe00640 1 --2- 172.21.15.28:0/1094243611 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f113c06b500 0x7f113c06d9c0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f1160152cd0 tx=0x7f1148008040 comp rx=0 tx=0).stop 2024-02-21T23:06:47.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.664+0000 7f115fe00640 1 -- 172.21.15.28:0/1094243611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11440039b0 msgr2=0x7f11601eb0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:47.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.664+0000 7f115fe00640 1 --2- 172.21.15.28:0/1094243611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11440039b0 0x7f11601eb0e0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f1150005040 tx=0x7f1150001120 comp rx=0 tx=0).stop 2024-02-21T23:06:47.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.665+0000 7f115fe00640 1 -- 172.21.15.28:0/1094243611 shutdown_connections 2024-02-21T23:06:47.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.665+0000 7f115fe00640 1 --2- 172.21.15.28:0/1094243611 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f113c06b500 0x7f113c06d9c0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:47.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.665+0000 7f115fe00640 1 --2- 172.21.15.28:0/1094243611 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f11601eb620 0x7f11601ec370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:47.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.665+0000 7f115fe00640 1 --2- 172.21.15.28:0/1094243611 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f11601ec8b0 0x7f1160147730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:47.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.665+0000 7f115fe00640 1 --2- 172.21.15.28:0/1094243611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11440039b0 0x7f11601eb0e0 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:47.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.665+0000 7f115fe00640 1 -- 172.21.15.28:0/1094243611 >> 172.21.15.28:0/1094243611 conn(0x7f116006b630 msgr2=0x7f11601513a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:47.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.665+0000 7f115fe00640 1 -- 172.21.15.28:0/1094243611 shutdown_connections 2024-02-21T23:06:47.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.665+0000 7f115fe00640 1 -- 172.21.15.28:0/1094243611 wait complete. 2024-02-21T23:06:47.675 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-02-21T23:06:47.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.913+0000 7f262b200640 1 Processor -- start 2024-02-21T23:06:47.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.913+0000 7f262b200640 1 -- start start 2024-02-21T23:06:47.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.914+0000 7f262b200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f262c1520a0 0x7f262c071a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:47.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.914+0000 7f262b200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f262c072f20 con 0x7f262c06e8d0 2024-02-21T23:06:47.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.914+0000 7f262b200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f262c06db10 con 0x7f262c0729b0 2024-02-21T23:06:47.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.914+0000 7f262b200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f262c06ddf0 con 0x7f262c1520a0 2024-02-21T23:06:47.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.915+0000 7f262a800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f262c06e8d0 0x7f262c06ed80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34230/0 (socket says 172.21.15.28:34230) 2024-02-21T23:06:47.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.915+0000 7f262a800640 1 -- 172.21.15.28:0/1378780751 learned_addr learned my addr 172.21.15.28:0/1378780751 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:06:47.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.915+0000 7f2629e00640 1 --2- 172.21.15.28:0/1378780751 >> v2:172.21.15.81:3300/0 conn(0x7f262c1520a0 0x7f262c071a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:47.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.915+0000 7f2629400640 1 -- 172.21.15.28:0/1378780751 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1655598788 0 0) 0x7f262c072f20 con 0x7f262c06e8d0 2024-02-21T23:06:47.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.915+0000 7f2629400640 1 -- 172.21.15.28:0/1378780751 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2610003520 con 0x7f262c06e8d0 2024-02-21T23:06:47.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.916+0000 7f2629400640 1 -- 172.21.15.28:0/1378780751 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 814047134 0 0) 0x7f2610003520 con 0x7f262c06e8d0 2024-02-21T23:06:47.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.916+0000 7f2629400640 1 -- 172.21.15.28:0/1378780751 >> v2:172.21.15.81:3300/0 conn(0x7f262c1520a0 msgr2=0x7f262c071a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:47.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.916+0000 7f2629400640 1 --2- 172.21.15.28:0/1378780751 >> v2:172.21.15.81:3300/0 conn(0x7f262c1520a0 0x7f262c071a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:47.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.916+0000 7f2629400640 1 -- 172.21.15.28:0/1378780751 >> v1:172.21.15.81:6789/0 conn(0x7f262c0729b0 legacy=0x7f262c0c16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:47.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.916+0000 7f2629400640 1 -- 172.21.15.28:0/1378780751 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f262c06e0d0 con 0x7f262c06e8d0 2024-02-21T23:06:47.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.916+0000 7f2629400640 1 -- 172.21.15.28:0/1378780751 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f26240019f0 con 0x7f262c06e8d0 2024-02-21T23:06:47.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.916+0000 7f2629400640 1 -- 172.21.15.28:0/1378780751 >> v1:172.21.15.28:6789/0 conn(0x7f262c06e8d0 legacy=0x7f262c06ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:47.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.916+0000 7f2629400640 1 --2- 172.21.15.28:0/1378780751 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2610003a00 0x7f2610003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:47.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.916+0000 7f2629400640 1 --2- 172.21.15.28:0/1378780751 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2610004800 0x7f2610004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:47.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.916+0000 7f2629400640 1 --2- 172.21.15.28:0/1378780751 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2610005210 0x7f261000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:47.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.916+0000 7f2629400640 1 -- 172.21.15.28:0/1378780751 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f262c06ddf0 con 0x7f2610005210 2024-02-21T23:06:47.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.916+0000 7f2629400640 1 -- 172.21.15.28:0/1378780751 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f261000c160 con 0x7f2610003a00 2024-02-21T23:06:47.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.916+0000 7f2629400640 1 -- 172.21.15.28:0/1378780751 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f261000c360 con 0x7f2610004800 2024-02-21T23:06:47.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.916+0000 7f262a800640 1 --2- 172.21.15.28:0/1378780751 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2610005210 0x7f261000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:47.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.917+0000 7f2629e00640 1 --2- 172.21.15.28:0/1378780751 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2610003a00 0x7f2610003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:47.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.917+0000 7f262a800640 1 -- 172.21.15.28:0/1378780751 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2610004800 msgr2=0x7f2610004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:47.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.917+0000 7f2623e00640 1 --2- 172.21.15.28:0/1378780751 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2610004800 0x7f2610004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:47.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.917+0000 7f262a800640 1 --2- 172.21.15.28:0/1378780751 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2610004800 0x7f2610004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:47.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.917+0000 7f262a800640 1 -- 172.21.15.28:0/1378780751 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2610003a00 msgr2=0x7f2610003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:47.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.917+0000 7f262a800640 1 --2- 172.21.15.28:0/1378780751 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2610003a00 0x7f2610003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:47.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.917+0000 7f262a800640 1 -- 172.21.15.28:0/1378780751 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2610003520 con 0x7f2610005210 2024-02-21T23:06:47.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.918+0000 7f262a800640 1 -- 172.21.15.28:0/1378780751 reap_dead start 2024-02-21T23:06:47.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.918+0000 7f262a800640 1 --2- 172.21.15.28:0/1378780751 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2610005210 0x7f261000b9c0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f262400a760 tx=0x7f2610005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49c8d5351929129a server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:47.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.918+0000 7f2629400640 1 -- 172.21.15.28:0/1378780751 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f262400a3b0 con 0x7f2610005210 2024-02-21T23:06:47.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.918+0000 7f2629400640 1 -- 172.21.15.28:0/1378780751 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2624034630 con 0x7f2610005210 2024-02-21T23:06:47.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.918+0000 7f2629400640 1 -- 172.21.15.28:0/1378780751 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26240349e0 con 0x7f2610005210 2024-02-21T23:06:47.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.919+0000 7f262b200640 1 -- 172.21.15.28:0/1378780751 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2610005210 msgr2=0x7f261000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:47.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.919+0000 7f262b200640 1 --2- 172.21.15.28:0/1378780751 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2610005210 0x7f261000b9c0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f262400a760 tx=0x7f2610005830 comp rx=0 tx=0).stop 2024-02-21T23:06:47.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.919+0000 7f262b200640 1 -- 172.21.15.28:0/1378780751 shutdown_connections 2024-02-21T23:06:47.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.919+0000 7f262b200640 1 --2- 172.21.15.28:0/1378780751 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2610005210 0x7f261000b9c0 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:47.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.919+0000 7f262b200640 1 -- 172.21.15.28:0/1378780751 >> 172.21.15.28:0/1378780751 conn(0x7f262c0a8530 msgr2=0x7f262c0a8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:47.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.919+0000 7f262b200640 1 -- 172.21.15.28:0/1378780751 shutdown_connections 2024-02-21T23:06:47.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.919+0000 7f262b200640 1 -- 172.21.15.28:0/1378780751 wait complete. 2024-02-21T23:06:47.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.921+0000 7f262b200640 1 Processor -- start 2024-02-21T23:06:47.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.921+0000 7f262b200640 1 -- start start 2024-02-21T23:06:47.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.921+0000 7f262b200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2610005210 0x7f262c1468a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:47.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.922+0000 7f262b200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f262c1474a0 0x7f262c1509d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:47.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.922+0000 7f262b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f262c0ac3f0 0x7f262c150f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:47.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.922+0000 7f262b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f262c0c21b0 con 0x7f262c0ac3f0 2024-02-21T23:06:47.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.922+0000 7f262b200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f262c0c4b90 con 0x7f2610005210 2024-02-21T23:06:47.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.922+0000 7f262b200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f262c0c2330 con 0x7f262c1474a0 2024-02-21T23:06:47.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.922+0000 7f262a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f262c0ac3f0 0x7f262c150f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:47.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.922+0000 7f262a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f262c0ac3f0 0x7f262c150f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36236/0 (socket says 172.21.15.28:36236) 2024-02-21T23:06:47.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.922+0000 7f262a800640 1 -- 172.21.15.28:0/3253924109 learned_addr learned my addr 172.21.15.28:0/3253924109 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:47.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.922+0000 7f2623e00640 1 --2- 172.21.15.28:0/3253924109 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f262c1474a0 0x7f262c1509d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:47.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.922+0000 7f2629e00640 1 --2- 172.21.15.28:0/3253924109 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2610005210 0x7f262c1468a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:47.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.922+0000 7f262a800640 1 -- 172.21.15.28:0/3253924109 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f262c1474a0 msgr2=0x7f262c1509d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:47.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.922+0000 7f262a800640 1 --2- 172.21.15.28:0/3253924109 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f262c1474a0 0x7f262c1509d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:47.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.922+0000 7f262a800640 1 -- 172.21.15.28:0/3253924109 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2610005210 msgr2=0x7f262c1468a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:47.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.922+0000 7f262a800640 1 --2- 172.21.15.28:0/3253924109 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2610005210 0x7f262c1468a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:47.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.922+0000 7f262a800640 1 -- 172.21.15.28:0/3253924109 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f262c151450 con 0x7f262c0ac3f0 2024-02-21T23:06:47.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.923+0000 7f262a800640 1 --2- 172.21.15.28:0/3253924109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f262c0ac3f0 0x7f262c150f10 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f262400a880 tx=0x7f262c072e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:47.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.923+0000 7f2622000640 1 -- 172.21.15.28:0/3253924109 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2624039040 con 0x7f262c0ac3f0 2024-02-21T23:06:47.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.923+0000 7f2622000640 1 -- 172.21.15.28:0/3253924109 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2624034430 con 0x7f262c0ac3f0 2024-02-21T23:06:47.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.923+0000 7f262b200640 1 -- 172.21.15.28:0/3253924109 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f262c0a9be0 con 0x7f262c0ac3f0 2024-02-21T23:06:47.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.923+0000 7f2622000640 1 -- 172.21.15.28:0/3253924109 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f262400a3b0 con 0x7f262c0ac3f0 2024-02-21T23:06:47.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.923+0000 7f262b200640 1 -- 172.21.15.28:0/3253924109 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f262c14ee00 con 0x7f262c0ac3f0 2024-02-21T23:06:47.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.925+0000 7f262b200640 1 -- 172.21.15.28:0/3253924109 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f262c04f290 con 0x7f262c0ac3f0 2024-02-21T23:06:47.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.926+0000 7f2622000640 1 -- 172.21.15.28:0/3253924109 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2624045020 con 0x7f262c0ac3f0 2024-02-21T23:06:47.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.926+0000 7f2622000640 1 --2- 172.21.15.28:0/3253924109 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f260806b500 0x7f260806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:47.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.927+0000 7f2622000640 1 -- 172.21.15.28:0/3253924109 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f26240ab7a0 con 0x7f262c0ac3f0 2024-02-21T23:06:47.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.927+0000 7f2629e00640 1 --2- 172.21.15.28:0/3253924109 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f260806b500 0x7f260806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:47.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.928+0000 7f2629e00640 1 --2- 172.21.15.28:0/3253924109 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f260806b500 0x7f260806d9c0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f261c009990 tx=0x7f261c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:47.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:47.932+0000 7f2622000640 1 -- 172.21.15.28:0/3253924109 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f262404c020 con 0x7f262c0ac3f0 2024-02-21T23:06:48.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:48.130+0000 7f262b200640 1 -- 172.21.15.28:0/3253924109 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f262c0c0d60 con 0x7f262c0ac3f0 2024-02-21T23:06:48.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:48.134+0000 7f2622000640 1 -- 172.21.15.28:0/3253924109 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v31)=0 v31) v1 ==== 122+0+0 (secure 0 0 0) 0x7f262c151450 con 0x7f262c0ac3f0 2024-02-21T23:06:48.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:48.137+0000 7f262b200640 1 -- 172.21.15.28:0/3253924109 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f260806b500 msgr2=0x7f260806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:48.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:48.137+0000 7f262b200640 1 --2- 172.21.15.28:0/3253924109 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f260806b500 0x7f260806d9c0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f261c009990 tx=0x7f261c008040 comp rx=0 tx=0).stop 2024-02-21T23:06:48.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:48.137+0000 7f262b200640 1 -- 172.21.15.28:0/3253924109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f262c0ac3f0 msgr2=0x7f262c150f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:48.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:48.137+0000 7f262b200640 1 --2- 172.21.15.28:0/3253924109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f262c0ac3f0 0x7f262c150f10 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f262400a880 tx=0x7f262c072e10 comp rx=0 tx=0).stop 2024-02-21T23:06:48.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:48.137+0000 7f262b200640 1 -- 172.21.15.28:0/3253924109 shutdown_connections 2024-02-21T23:06:48.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:48.137+0000 7f262b200640 1 --2- 172.21.15.28:0/3253924109 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f260806b500 0x7f260806d9c0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:48.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:48.137+0000 7f262b200640 1 --2- 172.21.15.28:0/3253924109 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f262c1474a0 0x7f262c1509d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:48.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:48.137+0000 7f262b200640 1 --2- 172.21.15.28:0/3253924109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f262c0ac3f0 0x7f262c150f10 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:48.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:48.137+0000 7f262b200640 1 --2- 172.21.15.28:0/3253924109 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2610005210 0x7f262c1468a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:48.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:48.137+0000 7f262b200640 1 -- 172.21.15.28:0/3253924109 >> 172.21.15.28:0/3253924109 conn(0x7f262c0a8530 msgr2=0x7f262c0c14b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:48.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:48.138+0000 7f262b200640 1 -- 172.21.15.28:0/3253924109 shutdown_connections 2024-02-21T23:06:48.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:48.138+0000 7f262b200640 1 -- 172.21.15.28:0/3253924109 wait complete. 2024-02-21T23:06:48.146 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-02-21T23:06:48.147 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-21T23:06:48.264 DEBUG:teuthology.orchestra.run.smithi081:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-21T23:06:48.379 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-02-21T23:06:48.379 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-21T23:06:48.379 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-21T23:06:48.439 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054608,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-02-21T23:06:48.439 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-02-21T23:06:48.439 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-21T23:06:48.495 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053833,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498812k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249404k,nr_inodes=812351,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498812k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-02-21T23:06:48.496 INFO:tasks.cephfs.cephfs_test_case:[, ] 2024-02-21T23:06:48.496 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:06:48.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:48.734+0000 7f627b000640 1 Processor -- start 2024-02-21T23:06:48.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:48.734+0000 7f627b000640 1 -- start start 2024-02-21T23:06:48.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:48.735+0000 7f627b000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f627414f0b0 0x7f6274155180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:48.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:48.736+0000 7f627b000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6274155760 con 0x7f62741521c0 2024-02-21T23:06:48.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:48.736+0000 7f627b000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6274155a40 con 0x7f62740a6160 2024-02-21T23:06:48.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:48.736+0000 7f627b000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6274151d30 con 0x7f627414f0b0 2024-02-21T23:06:48.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:48.736+0000 7f627a600640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f62741521c0 0x7f627415c510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34232/0 (socket says 172.21.15.28:34232) 2024-02-21T23:06:48.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:48.736+0000 7f627a600640 1 -- 172.21.15.28:0/2806436070 learned_addr learned my addr 172.21.15.28:0/2806436070 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:06:48.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:48.736+0000 7f6279c00640 1 --2- 172.21.15.28:0/2806436070 >> v2:172.21.15.28:3300/0 conn(0x7f627414f0b0 0x7f6274155180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:48.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:48.736+0000 7f6273e00640 1 -- 172.21.15.28:0/2806436070 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2076589692 0 0) 0x7f6274155760 con 0x7f62741521c0 2024-02-21T23:06:48.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:48.737+0000 7f6273e00640 1 -- 172.21.15.28:0/2806436070 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f625c005730 con 0x7f62741521c0 2024-02-21T23:06:48.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:48.737+0000 7f6279c00640 1 -- 172.21.15.28:0/2806436070 >> v1:172.21.15.81:6789/0 conn(0x7f62740a6160 legacy=0x7f62740c2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:48.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:48.737+0000 7f6279c00640 1 -- 172.21.15.28:0/2806436070 >> v1:172.21.15.28:6789/0 conn(0x7f62741521c0 legacy=0x7f627415c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:48.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:48.737+0000 7f6279c00640 1 -- 172.21.15.28:0/2806436070 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6274155d20 con 0x7f627414f0b0 2024-02-21T23:06:48.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:48.738+0000 7f6279c00640 1 --2- 172.21.15.28:0/2806436070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f627414f0b0 0x7f6274155180 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f62680097a0 tx=0x7f626802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16584740a1faf2b7 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:48.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:48.738+0000 7f6273e00640 1 -- 172.21.15.28:0/2806436070 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6268034070 con 0x7f627414f0b0 2024-02-21T23:06:48.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:48.738+0000 7f6273e00640 1 -- 172.21.15.28:0/2806436070 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6268030610 con 0x7f627414f0b0 2024-02-21T23:06:48.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:48.738+0000 7f6273e00640 1 -- 172.21.15.28:0/2806436070 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62680309c0 con 0x7f627414f0b0 2024-02-21T23:06:48.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:48.739+0000 7f627b000640 1 -- 172.21.15.28:0/2806436070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f627414f0b0 msgr2=0x7f6274155180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:48.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:48.739+0000 7f627b000640 1 --2- 172.21.15.28:0/2806436070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f627414f0b0 0x7f6274155180 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f62680097a0 tx=0x7f626802b960 comp rx=0 tx=0).stop 2024-02-21T23:06:48.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:48.739+0000 7f627b000640 1 -- 172.21.15.28:0/2806436070 shutdown_connections 2024-02-21T23:06:48.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:48.739+0000 7f627b000640 1 --2- 172.21.15.28:0/2806436070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f627414f0b0 0x7f6274155180 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:48.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:48.739+0000 7f627b000640 1 -- 172.21.15.28:0/2806436070 >> 172.21.15.28:0/2806436070 conn(0x7f627406b630 msgr2=0x7f62740c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:48.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:48.739+0000 7f627b000640 1 -- 172.21.15.28:0/2806436070 shutdown_connections 2024-02-21T23:06:48.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:48.739+0000 7f627b000640 1 -- 172.21.15.28:0/2806436070 wait complete. 2024-02-21T23:06:48.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:48.740+0000 7f627b000640 1 Processor -- start 2024-02-21T23:06:48.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:48.741+0000 7f627b000640 1 -- start start 2024-02-21T23:06:48.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:48.741+0000 7f627b000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62740a6160 0x7f62740c53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:48.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:48.741+0000 7f627b000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f627414f0b0 0x7f627414aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:48.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:48.742+0000 7f627b000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f62741521c0 0x7f6274149dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:48.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:48.742+0000 7f627b000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62740c2720 con 0x7f62740a6160 2024-02-21T23:06:48.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:48.742+0000 7f627b000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62740a68d0 con 0x7f627414f0b0 2024-02-21T23:06:48.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:48.742+0000 7f627b000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62740c28a0 con 0x7f62741521c0 2024-02-21T23:06:48.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:48.742+0000 7f6279c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62740a6160 0x7f62740c53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:48.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:48.742+0000 7f6279c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62740a6160 0x7f62740c53b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36268/0 (socket says 172.21.15.28:36268) 2024-02-21T23:06:48.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:48.742+0000 7f6279c00640 1 -- 172.21.15.28:0/1198366724 learned_addr learned my addr 172.21.15.28:0/1198366724 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:48.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:48.742+0000 7f6279c00640 1 -- 172.21.15.28:0/1198366724 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f62741521c0 msgr2=0x7f6274149dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:48.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:48.742+0000 7f6279c00640 1 --2- 172.21.15.28:0/1198366724 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f62741521c0 0x7f6274149dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:48.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:48.742+0000 7f6279c00640 1 -- 172.21.15.28:0/1198366724 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f627414f0b0 msgr2=0x7f627414aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:48.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:48.742+0000 7f6279c00640 1 --2- 172.21.15.28:0/1198366724 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f627414f0b0 0x7f627414aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:48.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:48.742+0000 7f6279c00640 1 -- 172.21.15.28:0/1198366724 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6274153090 con 0x7f62740a6160 2024-02-21T23:06:48.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:48.743+0000 7f6279c00640 1 --2- 172.21.15.28:0/1198366724 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62740a6160 0x7f62740c53b0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f62680097a0 tx=0x7f62680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:48.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:48.743+0000 7f6271600640 1 -- 172.21.15.28:0/1198366724 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6268034040 con 0x7f62740a6160 2024-02-21T23:06:48.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:48.743+0000 7f6271600640 1 -- 172.21.15.28:0/1198366724 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6268030410 con 0x7f62740a6160 2024-02-21T23:06:48.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:48.743+0000 7f6271600640 1 -- 172.21.15.28:0/1198366724 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6268004250 con 0x7f62740a6160 2024-02-21T23:06:48.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:48.743+0000 7f627b000640 1 -- 172.21.15.28:0/1198366724 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6274153330 con 0x7f62740a6160 2024-02-21T23:06:48.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:48.743+0000 7f627b000640 1 -- 172.21.15.28:0/1198366724 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62741538f0 con 0x7f62740a6160 2024-02-21T23:06:48.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:48.744+0000 7f627b000640 1 -- 172.21.15.28:0/1198366724 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f627404f290 con 0x7f62740a6160 2024-02-21T23:06:48.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:48.745+0000 7f6271600640 1 -- 172.21.15.28:0/1198366724 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6268037070 con 0x7f62740a6160 2024-02-21T23:06:48.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:48.746+0000 7f6271600640 1 --2- 172.21.15.28:0/1198366724 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f625406b4d0 0x7f625406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:48.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:48.747+0000 7f6271600640 1 -- 172.21.15.28:0/1198366724 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f62680abc70 con 0x7f62740a6160 2024-02-21T23:06:48.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:48.747+0000 7f6279200640 1 --2- 172.21.15.28:0/1198366724 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f625406b4d0 0x7f625406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:48.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:48.748+0000 7f6279200640 1 --2- 172.21.15.28:0/1198366724 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f625406b4d0 0x7f625406d990 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f6260000f60 tx=0x7f62600079d0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:48.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:48.751+0000 7f6271600640 1 -- 172.21.15.28:0/1198366724 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f626807ab50 con 0x7f62740a6160 2024-02-21T23:06:48.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:48.998+0000 7f627b000640 1 -- 172.21.15.28:0/1198366724 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f627414c1b0 con 0x7f62740a6160 2024-02-21T23:06:48.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:48.999+0000 7f6271600640 1 -- 172.21.15.28:0/1198366724 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+3699 (secure 0 0 0) 0x7f626807a450 con 0x7f62740a6160 2024-02-21T23:06:48.989 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:06:48.989 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":81,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":8,"addr":"172.21.15.81:6837/4292940632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":4292940632},{"type":"v1","addr":"172.21.15.81:6837","nonce":4292940632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81},{"gid":5507,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/3961681673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":3961681673},{"type":"v1","addr":"172.21.15.81:6835","nonce":3961681673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81},{"gid":6831,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2340244964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2340244964},{"type":"v1","addr":"172.21.15.28:6835","nonce":2340244964}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":76},{"gid":6846,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/2977800995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2977800995},{"type":"v1","addr":"172.21.15.28:6833","nonce":2977800995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":81}],"filesystems":[]} 2024-02-21T23:06:48.989 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 81 2024-02-21T23:06:48.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.002+0000 7f627b000640 1 -- 172.21.15.28:0/1198366724 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f625406b4d0 msgr2=0x7f625406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:48.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.002+0000 7f627b000640 1 --2- 172.21.15.28:0/1198366724 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f625406b4d0 0x7f625406d990 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f6260000f60 tx=0x7f62600079d0 comp rx=0 tx=0).stop 2024-02-21T23:06:48.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.002+0000 7f627b000640 1 -- 172.21.15.28:0/1198366724 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62740a6160 msgr2=0x7f62740c53b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:48.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.002+0000 7f627b000640 1 --2- 172.21.15.28:0/1198366724 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62740a6160 0x7f62740c53b0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f62680097a0 tx=0x7f62680046c0 comp rx=0 tx=0).stop 2024-02-21T23:06:48.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.003+0000 7f627b000640 1 -- 172.21.15.28:0/1198366724 shutdown_connections 2024-02-21T23:06:48.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.003+0000 7f627b000640 1 --2- 172.21.15.28:0/1198366724 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f625406b4d0 0x7f625406d990 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:48.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.003+0000 7f627b000640 1 --2- 172.21.15.28:0/1198366724 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f62741521c0 0x7f6274149dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:48.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.003+0000 7f627b000640 1 --2- 172.21.15.28:0/1198366724 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f627414f0b0 0x7f627414aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:48.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.003+0000 7f627b000640 1 --2- 172.21.15.28:0/1198366724 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62740a6160 0x7f62740c53b0 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:48.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.003+0000 7f627b000640 1 -- 172.21.15.28:0/1198366724 >> 172.21.15.28:0/1198366724 conn(0x7f627406b630 msgr2=0x7f6274072360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:48.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.003+0000 7f627b000640 1 -- 172.21.15.28:0/1198366724 shutdown_connections 2024-02-21T23:06:48.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.003+0000 7f627b000640 1 -- 172.21.15.28:0/1198366724 wait complete. 2024-02-21T23:06:49.012 INFO:tasks.ceph.mds.a:Restarting daemon 2024-02-21T23:06:49.013 INFO:tasks.ceph.mds.a:Stopping old one... 2024-02-21T23:06:49.013 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-02-21T23:06:49.013 INFO:teuthology.orchestra.run:waiting for 300 2024-02-21T23:06:49.013 INFO:tasks.ceph.mds.c:Restarting daemon 2024-02-21T23:06:49.013 INFO:tasks.ceph.mds.c:Stopping old one... 2024-02-21T23:06:49.014 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-02-21T23:06:49.014 INFO:teuthology.orchestra.run:waiting for 300 2024-02-21T23:06:49.014 INFO:tasks.ceph.mds.b:Restarting daemon 2024-02-21T23:06:49.014 INFO:tasks.ceph.mds.b:Stopping old one... 2024-02-21T23:06:49.014 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-02-21T23:06:49.014 INFO:teuthology.orchestra.run:waiting for 300 2024-02-21T23:06:49.015 INFO:tasks.ceph.mds.d:Restarting daemon 2024-02-21T23:06:49.015 INFO:tasks.ceph.mds.d:Stopping old one... 2024-02-21T23:06:49.015 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-02-21T23:06:49.015 INFO:teuthology.orchestra.run:waiting for 300 2024-02-21T23:06:49.032 INFO:tasks.ceph.mds.c:Stopped 2024-02-21T23:06:49.033 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-02-21T23:06:49.034 INFO:tasks.ceph.mds.a:Stopped 2024-02-21T23:06:49.034 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-02-21T23:06:49.035 INFO:tasks.ceph.mds.b:Stopped 2024-02-21T23:06:49.035 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-02-21T23:06:49.047 INFO:tasks.ceph.mds.d:Stopped 2024-02-21T23:06:49.047 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-02-21T23:06:49.081 INFO:tasks.ceph.mds.c:Started 2024-02-21T23:06:49.081 DEBUG:teuthology.parallel:result is None 2024-02-21T23:06:49.087 INFO:tasks.ceph.mds.a:Started 2024-02-21T23:06:49.088 DEBUG:teuthology.parallel:result is None 2024-02-21T23:06:49.093 INFO:tasks.ceph.mds.b:Started 2024-02-21T23:06:49.094 DEBUG:teuthology.parallel:result is None 2024-02-21T23:06:49.099 INFO:tasks.ceph.mds.d:Started 2024-02-21T23:06:49.099 DEBUG:teuthology.parallel:result is None 2024-02-21T23:06:49.099 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-02-21T23:06:49.247 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-02-21T23:06:49.248 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-02-21T23:06:49.261 INFO:tasks.ceph.mds.b.smithi081.stdout:starting mds.b at 2024-02-21T23:06:49.266 INFO:tasks.ceph.mds.d.smithi081.stdout:starting mds.d at 2024-02-21T23:06:49.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.304+0000 7f7706c00640 1 Processor -- start 2024-02-21T23:06:49.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.304+0000 7f7706c00640 1 -- start start 2024-02-21T23:06:49.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.305+0000 7f7706c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7700152a10 0x7f7700155640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:49.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.306+0000 7f7706c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f77000c54f0 0x7f77000c59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:49.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.306+0000 7f7706c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77000a64b0 con 0x7f77000a5e30 2024-02-21T23:06:49.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.306+0000 7f7706c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7700156290 con 0x7f7700152a10 2024-02-21T23:06:49.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.306+0000 7f7706c00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7700152df0 con 0x7f77000c54f0 2024-02-21T23:06:49.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.306+0000 7f76ffe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7700152a10 0x7f7700155640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:49.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.306+0000 7f76ffe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7700152a10 0x7f7700155640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36308/0 (socket says 172.21.15.28:36308) 2024-02-21T23:06:49.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.306+0000 7f76f7e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f77000a5e30 0x7f7700155b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34260/0 (socket says 172.21.15.28:34260) 2024-02-21T23:06:49.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.306+0000 7f76ffe00640 1 -- 172.21.15.28:0/319523621 learned_addr learned my addr 172.21.15.28:0/319523621 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:49.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.306+0000 7f7704c00640 1 --2- 172.21.15.28:0/319523621 >> v2:172.21.15.81:3300/0 conn(0x7f77000c54f0 0x7f77000c59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:49.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.306+0000 7f76ffe00640 1 -- 172.21.15.28:0/319523621 >> v2:172.21.15.81:3300/0 conn(0x7f77000c54f0 msgr2=0x7f77000c59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:49.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.306+0000 7f76ffe00640 1 --2- 172.21.15.28:0/319523621 >> v2:172.21.15.81:3300/0 conn(0x7f77000c54f0 0x7f77000c59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:49.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.306+0000 7f76ffe00640 1 -- 172.21.15.28:0/319523621 >> v1:172.21.15.28:6789/0 conn(0x7f77000a5e30 legacy=0x7f7700155b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:49.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.306+0000 7f76ffe00640 1 -- 172.21.15.28:0/319523621 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7700071960 con 0x7f7700152a10 2024-02-21T23:06:49.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.306+0000 7f76ff400640 1 -- 172.21.15.28:0/319523621 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2718341275 0 0) 0x7f77000a64b0 con 0x7f77000a5e30 2024-02-21T23:06:49.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.306+0000 7f7704c00640 1 --2- 172.21.15.28:0/319523621 >> v2:172.21.15.81:3300/0 conn(0x7f77000c54f0 0x7f77000c59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:06:49.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.308+0000 7f76ffe00640 1 --2- 172.21.15.28:0/319523621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7700152a10 0x7f7700155640 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f77000679d0 tx=0x7f76f002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbebb405af448dcf server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:49.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.308+0000 7f76ff400640 1 -- 172.21.15.28:0/319523621 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76f0035020 con 0x7f7700152a10 2024-02-21T23:06:49.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.308+0000 7f76ff400640 1 -- 172.21.15.28:0/319523621 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76f0031870 con 0x7f7700152a10 2024-02-21T23:06:49.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.308+0000 7f76ff400640 1 -- 172.21.15.28:0/319523621 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76f0031c40 con 0x7f7700152a10 2024-02-21T23:06:49.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.308+0000 7f7706c00640 1 -- 172.21.15.28:0/319523621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7700152a10 msgr2=0x7f7700155640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:49.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.308+0000 7f7706c00640 1 --2- 172.21.15.28:0/319523621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7700152a10 0x7f7700155640 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f77000679d0 tx=0x7f76f002b910 comp rx=0 tx=0).stop 2024-02-21T23:06:49.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.308+0000 7f7706c00640 1 -- 172.21.15.28:0/319523621 shutdown_connections 2024-02-21T23:06:49.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.308+0000 7f7706c00640 1 --2- 172.21.15.28:0/319523621 >> v2:172.21.15.81:3300/0 conn(0x7f77000c54f0 0x7f77000c59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:49.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.308+0000 7f7706c00640 1 --2- 172.21.15.28:0/319523621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7700152a10 0x7f7700155640 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:49.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.308+0000 7f7706c00640 1 -- 172.21.15.28:0/319523621 >> 172.21.15.28:0/319523621 conn(0x7f77000a8970 msgr2=0x7f77000a8d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:49.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.309+0000 7f7706c00640 1 -- 172.21.15.28:0/319523621 shutdown_connections 2024-02-21T23:06:49.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.309+0000 7f7706c00640 1 -- 172.21.15.28:0/319523621 wait complete. 2024-02-21T23:06:49.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.310+0000 7f7706c00640 1 Processor -- start 2024-02-21T23:06:49.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.310+0000 7f7706c00640 1 -- start start 2024-02-21T23:06:49.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.311+0000 7f7706c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f77000a5e30 0x7f7700159180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:49.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.311+0000 7f7706c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f77000c54f0 0x7f77000a7770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:49.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.311+0000 7f7706c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7700152a10 0x7f77001eb500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:49.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.311+0000 7f7706c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7700071f20 con 0x7f7700152a10 2024-02-21T23:06:49.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.311+0000 7f7706c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77000c68d0 con 0x7f77000a5e30 2024-02-21T23:06:49.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.311+0000 7f7706c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77000c5ee0 con 0x7f77000c54f0 2024-02-21T23:06:49.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.311+0000 7f7704c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7700152a10 0x7f77001eb500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:49.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.312+0000 7f7704c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7700152a10 0x7f77001eb500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36312/0 (socket says 172.21.15.28:36312) 2024-02-21T23:06:49.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.312+0000 7f7704c00640 1 -- 172.21.15.28:0/3852973906 learned_addr learned my addr 172.21.15.28:0/3852973906 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:49.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.312+0000 7f76f7e00640 1 --2- 172.21.15.28:0/3852973906 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f77000c54f0 0x7f77000a7770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:49.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.312+0000 7f76ffe00640 1 --2- 172.21.15.28:0/3852973906 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f77000a5e30 0x7f7700159180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:49.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.312+0000 7f7704c00640 1 -- 172.21.15.28:0/3852973906 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f77000c54f0 msgr2=0x7f77000a7770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:49.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.312+0000 7f7704c00640 1 --2- 172.21.15.28:0/3852973906 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f77000c54f0 0x7f77000a7770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:49.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.312+0000 7f7704c00640 1 -- 172.21.15.28:0/3852973906 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f77000a5e30 msgr2=0x7f7700159180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:49.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.312+0000 7f7704c00640 1 --2- 172.21.15.28:0/3852973906 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f77000a5e30 0x7f7700159180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:49.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.312+0000 7f7704c00640 1 -- 172.21.15.28:0/3852973906 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77001486b0 con 0x7f7700152a10 2024-02-21T23:06:49.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.312+0000 7f7704c00640 1 --2- 172.21.15.28:0/3852973906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7700152a10 0x7f77001eb500 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f76f800cf20 tx=0x7f76f800a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:49.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.313+0000 7f76fcc00640 1 -- 172.21.15.28:0/3852973906 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76f801a070 con 0x7f7700152a10 2024-02-21T23:06:49.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.313+0000 7f76fcc00640 1 -- 172.21.15.28:0/3852973906 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76f800d840 con 0x7f7700152a10 2024-02-21T23:06:49.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.313+0000 7f76fcc00640 1 -- 172.21.15.28:0/3852973906 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76f800dc30 con 0x7f7700152a10 2024-02-21T23:06:49.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.313+0000 7f7706c00640 1 -- 172.21.15.28:0/3852973906 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7700148990 con 0x7f7700152a10 2024-02-21T23:06:49.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.313+0000 7f7706c00640 1 -- 172.21.15.28:0/3852973906 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77000a8040 con 0x7f7700152a10 2024-02-21T23:06:49.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.314+0000 7f7706c00640 1 -- 172.21.15.28:0/3852973906 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77000c66f0 con 0x7f7700152a10 2024-02-21T23:06:49.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.315+0000 7f76fcc00640 1 -- 172.21.15.28:0/3852973906 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f76f801d070 con 0x7f7700152a10 2024-02-21T23:06:49.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.316+0000 7f76fcc00640 1 --2- 172.21.15.28:0/3852973906 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f76dc06b500 0x7f76dc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:49.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.316+0000 7f76fcc00640 1 -- 172.21.15.28:0/3852973906 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f76f808aa40 con 0x7f7700152a10 2024-02-21T23:06:49.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.317+0000 7f76ffe00640 1 --2- 172.21.15.28:0/3852973906 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f76dc06b500 0x7f76dc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:49.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.317+0000 7f76ffe00640 1 --2- 172.21.15.28:0/3852973906 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f76dc06b500 0x7f76dc06d9c0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f76f0036000 tx=0x7f76f0008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:49.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.320+0000 7f76fcc00640 1 -- 172.21.15.28:0/3852973906 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f77000c66f0 con 0x7f7700152a10 2024-02-21T23:06:49.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.535+0000 7f7706c00640 1 -- 172.21.15.28:0/3852973906 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f77001eadd0 con 0x7f7700152a10 2024-02-21T23:06:49.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.689+0000 7f76fcc00640 1 -- 172.21.15.28:0/3852973906 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v92) v1 ==== 97+0+0 (secure 0 0 0) 0x7f76f8059240 con 0x7f7700152a10 2024-02-21T23:06:49.679 INFO:teuthology.orchestra.run.smithi028.stderr: removed all blocklist entries 2024-02-21T23:06:49.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.693+0000 7f7706c00640 1 -- 172.21.15.28:0/3852973906 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f76dc06b500 msgr2=0x7f76dc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:49.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.693+0000 7f7706c00640 1 --2- 172.21.15.28:0/3852973906 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f76dc06b500 0x7f76dc06d9c0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f76f0036000 tx=0x7f76f0008030 comp rx=0 tx=0).stop 2024-02-21T23:06:49.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.693+0000 7f7706c00640 1 -- 172.21.15.28:0/3852973906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7700152a10 msgr2=0x7f77001eb500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:49.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.693+0000 7f7706c00640 1 --2- 172.21.15.28:0/3852973906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7700152a10 0x7f77001eb500 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f76f800cf20 tx=0x7f76f800a7d0 comp rx=0 tx=0).stop 2024-02-21T23:06:49.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.693+0000 7f7706c00640 1 -- 172.21.15.28:0/3852973906 shutdown_connections 2024-02-21T23:06:49.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.693+0000 7f7706c00640 1 --2- 172.21.15.28:0/3852973906 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f76dc06b500 0x7f76dc06d9c0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:49.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.693+0000 7f7706c00640 1 --2- 172.21.15.28:0/3852973906 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f77000c54f0 0x7f77000a7770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:49.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.693+0000 7f7706c00640 1 --2- 172.21.15.28:0/3852973906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7700152a10 0x7f77001eb500 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:49.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.693+0000 7f7706c00640 1 --2- 172.21.15.28:0/3852973906 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f77000a5e30 0x7f7700159180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:49.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.693+0000 7f7706c00640 1 -- 172.21.15.28:0/3852973906 >> 172.21.15.28:0/3852973906 conn(0x7f77000a8970 msgr2=0x7f7700074460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:49.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.694+0000 7f7706c00640 1 -- 172.21.15.28:0/3852973906 shutdown_connections 2024-02-21T23:06:49.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.694+0000 7f7706c00640 1 -- 172.21.15.28:0/3852973906 wait complete. 2024-02-21T23:06:49.710 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-02-21T23:06:49.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.938+0000 7f051c200640 1 Processor -- start 2024-02-21T23:06:49.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.938+0000 7f051c200640 1 -- start start 2024-02-21T23:06:49.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.940+0000 7f051c200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f05140c7e00 0x7f05141608d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:49.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.940+0000 7f051c200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05140c6f30 con 0x7f05141526b0 2024-02-21T23:06:49.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.940+0000 7f051c200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05140c6450 con 0x7f05141559c0 2024-02-21T23:06:49.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.940+0000 7f051c200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05140c61b0 con 0x7f05140c7e00 2024-02-21T23:06:49.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.940+0000 7f0519800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f05141526b0 0x7f05140ca940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34266/0 (socket says 172.21.15.28:34266) 2024-02-21T23:06:49.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.940+0000 7f0519800640 1 -- 172.21.15.28:0/3412263360 learned_addr learned my addr 172.21.15.28:0/3412263360 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:06:49.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.940+0000 7f051a200640 1 --2- 172.21.15.28:0/3412263360 >> v2:172.21.15.28:3300/0 conn(0x7f05140c7e00 0x7f05141608d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:49.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.940+0000 7f0513e00640 1 -- 172.21.15.28:0/3412263360 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3484726145 0 0) 0x7f05140c6f30 con 0x7f05141526b0 2024-02-21T23:06:49.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.941+0000 7f0513e00640 1 -- 172.21.15.28:0/3412263360 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04fc003520 con 0x7f05141526b0 2024-02-21T23:06:49.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.941+0000 7f051a200640 1 -- 172.21.15.28:0/3412263360 >> v1:172.21.15.81:6789/0 conn(0x7f05141559c0 legacy=0x7f05140c76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:49.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.941+0000 7f051a200640 1 -- 172.21.15.28:0/3412263360 >> v1:172.21.15.28:6789/0 conn(0x7f05141526b0 legacy=0x7f05140ca940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:49.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.941+0000 7f051a200640 1 -- 172.21.15.28:0/3412263360 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05140c6710 con 0x7f05140c7e00 2024-02-21T23:06:49.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.942+0000 7f051a200640 1 --2- 172.21.15.28:0/3412263360 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05140c7e00 0x7f05141608d0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f050c0097a0 tx=0x7f050c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7be821741a3a1085 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:49.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.942+0000 7f0513e00640 1 -- 172.21.15.28:0/3412263360 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f050c035020 con 0x7f05140c7e00 2024-02-21T23:06:49.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.942+0000 7f0513e00640 1 -- 172.21.15.28:0/3412263360 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f050c031830 con 0x7f05140c7e00 2024-02-21T23:06:49.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.942+0000 7f0513e00640 1 -- 172.21.15.28:0/3412263360 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f050c031c00 con 0x7f05140c7e00 2024-02-21T23:06:49.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.943+0000 7f051c200640 1 -- 172.21.15.28:0/3412263360 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05140c7e00 msgr2=0x7f05141608d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:49.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.943+0000 7f051c200640 1 --2- 172.21.15.28:0/3412263360 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05140c7e00 0x7f05141608d0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f050c0097a0 tx=0x7f050c02b8d0 comp rx=0 tx=0).stop 2024-02-21T23:06:49.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.943+0000 7f051c200640 1 -- 172.21.15.28:0/3412263360 shutdown_connections 2024-02-21T23:06:49.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.943+0000 7f051c200640 1 --2- 172.21.15.28:0/3412263360 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05140c7e00 0x7f05141608d0 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:49.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.943+0000 7f051c200640 1 -- 172.21.15.28:0/3412263360 >> 172.21.15.28:0/3412263360 conn(0x7f051406d3d0 msgr2=0x7f051408afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:49.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.943+0000 7f051c200640 1 -- 172.21.15.28:0/3412263360 shutdown_connections 2024-02-21T23:06:49.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.943+0000 7f051c200640 1 -- 172.21.15.28:0/3412263360 wait complete. 2024-02-21T23:06:49.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.945+0000 7f051c200640 1 Processor -- start 2024-02-21T23:06:49.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.945+0000 7f051c200640 1 -- start start 2024-02-21T23:06:49.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.946+0000 7f051c200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05140c7e00 0x7f0514146ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:49.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.946+0000 7f051c200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05141526b0 0x7f051414cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:49.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.946+0000 7f051c200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f05141559c0 0x7f05140c8750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:49.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.946+0000 7f051c200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05140c8270 con 0x7f05141526b0 2024-02-21T23:06:49.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.946+0000 7f051c200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05140c71d0 con 0x7f05140c7e00 2024-02-21T23:06:49.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.946+0000 7f051c200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05140c5c90 con 0x7f05141559c0 2024-02-21T23:06:49.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.946+0000 7f0518e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05141526b0 0x7f051414cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:49.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.946+0000 7f0518e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05141526b0 0x7f051414cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36316/0 (socket says 172.21.15.28:36316) 2024-02-21T23:06:49.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.946+0000 7f0518e00640 1 -- 172.21.15.28:0/3800029528 learned_addr learned my addr 172.21.15.28:0/3800029528 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:49.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.946+0000 7f0519800640 1 --2- 172.21.15.28:0/3800029528 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05140c7e00 0x7f0514146ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:49.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.946+0000 7f051a200640 1 --2- 172.21.15.28:0/3800029528 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f05141559c0 0x7f05140c8750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:49.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.946+0000 7f0518e00640 1 -- 172.21.15.28:0/3800029528 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f05141559c0 msgr2=0x7f05140c8750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:49.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.946+0000 7f0518e00640 1 --2- 172.21.15.28:0/3800029528 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f05141559c0 0x7f05140c8750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:49.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.946+0000 7f0518e00640 1 -- 172.21.15.28:0/3800029528 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05140c7e00 msgr2=0x7f0514146ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:49.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.946+0000 7f0518e00640 1 --2- 172.21.15.28:0/3800029528 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05140c7e00 0x7f0514146ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:49.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.946+0000 7f0518e00640 1 -- 172.21.15.28:0/3800029528 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0514147530 con 0x7f05141526b0 2024-02-21T23:06:49.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.946+0000 7f0519800640 1 --2- 172.21.15.28:0/3800029528 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05140c7e00 0x7f0514146ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:06:49.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.947+0000 7f0518e00640 1 --2- 172.21.15.28:0/3800029528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05141526b0 0x7f051414cae0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f050000b3f0 tx=0x7f050000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:49.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.947+0000 7f0511600640 1 -- 172.21.15.28:0/3800029528 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0500015020 con 0x7f05141526b0 2024-02-21T23:06:49.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.947+0000 7f0511600640 1 -- 172.21.15.28:0/3800029528 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0500011830 con 0x7f05141526b0 2024-02-21T23:06:49.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.947+0000 7f051c200640 1 -- 172.21.15.28:0/3800029528 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05141eaaf0 con 0x7f05141526b0 2024-02-21T23:06:49.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.947+0000 7f0511600640 1 -- 172.21.15.28:0/3800029528 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0500011c20 con 0x7f05141526b0 2024-02-21T23:06:49.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.947+0000 7f051c200640 1 -- 172.21.15.28:0/3800029528 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05141eb0b0 con 0x7f05141526b0 2024-02-21T23:06:49.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.948+0000 7f051c200640 1 -- 172.21.15.28:0/3800029528 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0514067f30 con 0x7f05141526b0 2024-02-21T23:06:49.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.949+0000 7f0511600640 1 -- 172.21.15.28:0/3800029528 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0500018020 con 0x7f05141526b0 2024-02-21T23:06:49.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.950+0000 7f0511600640 1 --2- 172.21.15.28:0/3800029528 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f04f406b4d0 0x7f04f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:49.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.950+0000 7f0511600640 1 -- 172.21.15.28:0/3800029528 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f0500089c40 con 0x7f05141526b0 2024-02-21T23:06:49.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.951+0000 7f0519800640 1 --2- 172.21.15.28:0/3800029528 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f04f406b4d0 0x7f04f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:49.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.952+0000 7f0519800640 1 --2- 172.21.15.28:0/3800029528 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f04f406b4d0 0x7f04f406d990 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f0508005c00 tx=0x7f0508009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:49.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:49.955+0000 7f0511600640 1 -- 172.21.15.28:0/3800029528 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f050001f030 con 0x7f05141526b0 2024-02-21T23:06:50.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.226+0000 7f051c200640 1 -- 172.21.15.28:0/3800029528 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f05141482f0 con 0x7f05141526b0 2024-02-21T23:06:50.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.227+0000 7f0511600640 1 -- 172.21.15.28:0/3800029528 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v13) v1 ==== 80+0+4724 (secure 0 0 0) 0x7f05000172c0 con 0x7f05141526b0 2024-02-21T23:06:50.216 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:06:50.217 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-21T23:06:50.217 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_dump": [ 2024-02-21T23:06:50.217 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:06:50.217 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.a", 2024-02-21T23:06:50.217 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDSgNZl4FY1KRAAJSK5Ibb0ibKiKXYpTLgtmQ==", 2024-02-21T23:06:50.217 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:06:50.217 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:06:50.217 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-02-21T23:06:50.217 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-02-21T23:06:50.217 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:06:50.217 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:06:50.217 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:06:50.217 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:06:50.218 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.b", 2024-02-21T23:06:50.218 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDTgNZlwRcAERAAgsYmmkuobKelBEJVCRpXUg==", 2024-02-21T23:06:50.218 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:06:50.218 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:06:50.218 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-02-21T23:06:50.218 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-02-21T23:06:50.218 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:06:50.218 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:06:50.218 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:06:50.218 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:06:50.218 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.c", 2024-02-21T23:06:50.218 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDSgNZloFpQOxAAK2WBArXRVNTgqH9WWDEPDA==", 2024-02-21T23:06:50.218 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:06:50.218 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:06:50.219 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-02-21T23:06:50.219 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-02-21T23:06:50.219 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:06:50.219 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:06:50.219 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:06:50.219 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:06:50.219 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.d", 2024-02-21T23:06:50.219 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDTgNZlblOmIRAAxe2HfSKGSkkzJDN4aVdzgQ==", 2024-02-21T23:06:50.219 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:06:50.219 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:06:50.219 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-02-21T23:06:50.219 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-02-21T23:06:50.219 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:06:50.219 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:06:50.220 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:06:50.220 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:06:50.220 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.0", 2024-02-21T23:06:50.220 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDVgNZlxNJbLxAAXot7Q3chrLEBsFNO8kT6QQ==", 2024-02-21T23:06:50.220 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:06:50.220 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:06:50.220 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:06:50.220 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:06:50.220 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:06:50.220 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:06:50.220 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:06:50.220 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.1", 2024-02-21T23:06:50.220 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDYgNZlZZfWORAAr4XkOxLYLmumVbu362N2HQ==", 2024-02-21T23:06:50.220 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:06:50.221 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:06:50.221 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:06:50.221 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:06:50.221 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:06:50.221 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:06:50.221 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:06:50.221 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.2", 2024-02-21T23:06:50.221 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDcgNZl9mcDGBAAIZGOF8IUMenwo386x95mwg==", 2024-02-21T23:06:50.221 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:06:50.221 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:06:50.221 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:06:50.221 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:06:50.221 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:06:50.221 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:06:50.222 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:06:50.222 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.3", 2024-02-21T23:06:50.222 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDfgNZl8ALSMBAAag9cmGH0/4xALWSUmrZowg==", 2024-02-21T23:06:50.222 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:06:50.222 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:06:50.222 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:06:50.222 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:06:50.222 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:06:50.222 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:06:50.222 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:06:50.222 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.4", 2024-02-21T23:06:50.222 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDngNZlBgeKABAAeW99RVfCBvB1lp7w7mQBlQ==", 2024-02-21T23:06:50.222 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:06:50.222 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:06:50.223 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:06:50.223 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:06:50.223 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:06:50.223 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:06:50.223 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:06:50.223 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.5", 2024-02-21T23:06:50.223 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDqgNZljQG3ChAAY/IZEIcvcGNvlFGIWib3gg==", 2024-02-21T23:06:50.223 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:06:50.223 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:06:50.223 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:06:50.223 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:06:50.223 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:06:50.223 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:06:50.223 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:06:50.224 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.6", 2024-02-21T23:06:50.224 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDtgNZlKnakIxAAkW5xlkshzvEHJdmpgPxR+g==", 2024-02-21T23:06:50.224 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:06:50.224 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:06:50.224 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:06:50.224 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:06:50.224 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:06:50.224 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:06:50.224 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:06:50.224 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.7", 2024-02-21T23:06:50.224 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDxgNZlTWxYARAASX1/gmqO+8mQPP8uA4xjiQ==", 2024-02-21T23:06:50.224 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:06:50.224 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:06:50.224 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:06:50.225 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:06:50.225 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:06:50.225 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:06:50.225 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:06:50.225 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.0", 2024-02-21T23:06:50.225 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDTgNZlILvYLxAAoREWxKKT7YcXnhkgk8YvoQ==", 2024-02-21T23:06:50.225 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:06:50.225 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:06:50.225 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow r", 2024-02-21T23:06:50.225 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-02-21T23:06:50.225 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:06:50.225 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:06:50.225 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:06:50.225 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.1", 2024-02-21T23:06:50.226 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDTgNZlm7ImOxAA8iRcXlzFPZPaXQfsKhuO/g==", 2024-02-21T23:06:50.226 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:06:50.226 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:06:50.226 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-02-21T23:06:50.226 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-02-21T23:06:50.226 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-02-21T23:06:50.226 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:06:50.226 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:06:50.226 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:06:50.226 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.admin", 2024-02-21T23:06:50.226 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDRgNZlBYd1GhAAbK5qBCIkf0Ej5nEyPnTxIQ==", 2024-02-21T23:06:50.226 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:06:50.226 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-02-21T23:06:50.226 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-02-21T23:06:50.227 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-02-21T23:06:50.227 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:06:50.227 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:06:50.227 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:06:50.227 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:06:50.227 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.x", 2024-02-21T23:06:50.227 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDSgNZlJYKeHBAAYPskmLRr7Jw44TdKzM/JMg==", 2024-02-21T23:06:50.227 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:06:50.227 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-02-21T23:06:50.227 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-02-21T23:06:50.227 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:06:50.227 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:06:50.227 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:06:50.227 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:06:50.228 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.y", 2024-02-21T23:06:50.228 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDSgNZlSRkOEBAAOhRj9PsZDpjoV6WT4drrZA==", 2024-02-21T23:06:50.228 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:06:50.228 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-02-21T23:06:50.228 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-02-21T23:06:50.228 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:06:50.228 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:06:50.228 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:06:50.228 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-21T23:06:50.228 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-21T23:06:50.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.230+0000 7f051c200640 1 -- 172.21.15.28:0/3800029528 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f04f406b4d0 msgr2=0x7f04f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:50.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.230+0000 7f051c200640 1 --2- 172.21.15.28:0/3800029528 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f04f406b4d0 0x7f04f406d990 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f0508005c00 tx=0x7f0508009040 comp rx=0 tx=0).stop 2024-02-21T23:06:50.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.230+0000 7f051c200640 1 -- 172.21.15.28:0/3800029528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05141526b0 msgr2=0x7f051414cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:50.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.230+0000 7f051c200640 1 --2- 172.21.15.28:0/3800029528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05141526b0 0x7f051414cae0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f050000b3f0 tx=0x7f050000b8b0 comp rx=0 tx=0).stop 2024-02-21T23:06:50.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.230+0000 7f051c200640 1 -- 172.21.15.28:0/3800029528 shutdown_connections 2024-02-21T23:06:50.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.230+0000 7f051c200640 1 --2- 172.21.15.28:0/3800029528 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f04f406b4d0 0x7f04f406d990 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:50.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.230+0000 7f051c200640 1 --2- 172.21.15.28:0/3800029528 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f05141559c0 0x7f05140c8750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:50.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.230+0000 7f051c200640 1 --2- 172.21.15.28:0/3800029528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05141526b0 0x7f051414cae0 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:50.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.230+0000 7f051c200640 1 --2- 172.21.15.28:0/3800029528 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05140c7e00 0x7f0514146ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:50.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.230+0000 7f051c200640 1 -- 172.21.15.28:0/3800029528 >> 172.21.15.28:0/3800029528 conn(0x7f051406d3d0 msgr2=0x7f051408afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:50.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.230+0000 7f051c200640 1 -- 172.21.15.28:0/3800029528 shutdown_connections 2024-02-21T23:06:50.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.230+0000 7f051c200640 1 -- 172.21.15.28:0/3800029528 wait complete. 2024-02-21T23:06:50.242 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:06:50.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.471+0000 7f64afe00640 1 Processor -- start 2024-02-21T23:06:50.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.472+0000 7f64afe00640 1 -- start start 2024-02-21T23:06:50.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.473+0000 7f64afe00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f64b006ee40 0x7f64b00a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:50.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.473+0000 7f64afe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f64b0156af0 0x7f64b006dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:50.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.473+0000 7f64afe00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64b00c2760 con 0x7f64b0150450 2024-02-21T23:06:50.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.473+0000 7f64afe00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64b00c22c0 con 0x7f64b0156af0 2024-02-21T23:06:50.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.473+0000 7f64afe00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64b0156660 con 0x7f64b006ee40 2024-02-21T23:06:50.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.473+0000 7f64aea00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f64b0150450 0x7f64b01536f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34272/0 (socket says 172.21.15.28:34272) 2024-02-21T23:06:50.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.473+0000 7f64aea00640 1 -- 172.21.15.28:0/2264511720 learned_addr learned my addr 172.21.15.28:0/2264511720 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:06:50.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.473+0000 7f64af400640 1 --2- 172.21.15.28:0/2264511720 >> v2:172.21.15.28:3300/0 conn(0x7f64b0156af0 0x7f64b006dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:50.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.473+0000 7f64ae000640 1 --2- 172.21.15.28:0/2264511720 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f64b006ee40 0x7f64b00a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:50.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.474+0000 7f64af400640 1 -- 172.21.15.28:0/2264511720 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f64b006ee40 msgr2=0x7f64b00a5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:50.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.474+0000 7f64af400640 1 --2- 172.21.15.28:0/2264511720 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f64b006ee40 0x7f64b00a5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:50.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.474+0000 7f64af400640 1 -- 172.21.15.28:0/2264511720 >> v1:172.21.15.28:6789/0 conn(0x7f64b0150450 legacy=0x7f64b01536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:50.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.474+0000 7f64af400640 1 -- 172.21.15.28:0/2264511720 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64b0156310 con 0x7f64b0156af0 2024-02-21T23:06:50.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.474+0000 7f64ad600640 1 -- 172.21.15.28:0/2264511720 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2090886133 0 0) 0x7f64b00c2760 con 0x7f64b0150450 2024-02-21T23:06:50.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.475+0000 7f64af400640 1 --2- 172.21.15.28:0/2264511720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64b0156af0 0x7f64b006dff0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f64a8009d90 tx=0x7f64a802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=766aec67f1e34fc6 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:50.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.475+0000 7f64ad600640 1 -- 172.21.15.28:0/2264511720 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a8039070 con 0x7f64b0156af0 2024-02-21T23:06:50.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.475+0000 7f64ad600640 1 -- 172.21.15.28:0/2264511720 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64a802cc50 con 0x7f64b0156af0 2024-02-21T23:06:50.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.475+0000 7f64ad600640 1 -- 172.21.15.28:0/2264511720 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a8034440 con 0x7f64b0156af0 2024-02-21T23:06:50.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.476+0000 7f64afe00640 1 -- 172.21.15.28:0/2264511720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64b0156af0 msgr2=0x7f64b006dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:50.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.476+0000 7f64afe00640 1 --2- 172.21.15.28:0/2264511720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64b0156af0 0x7f64b006dff0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f64a8009d90 tx=0x7f64a802bea0 comp rx=0 tx=0).stop 2024-02-21T23:06:50.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.476+0000 7f64afe00640 1 -- 172.21.15.28:0/2264511720 shutdown_connections 2024-02-21T23:06:50.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.476+0000 7f64afe00640 1 --2- 172.21.15.28:0/2264511720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64b0156af0 0x7f64b006dff0 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:50.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.476+0000 7f64afe00640 1 --2- 172.21.15.28:0/2264511720 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f64b006ee40 0x7f64b00a5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:50.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.476+0000 7f64afe00640 1 -- 172.21.15.28:0/2264511720 >> 172.21.15.28:0/2264511720 conn(0x7f64b006b630 msgr2=0x7f64b00c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:50.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.476+0000 7f64afe00640 1 -- 172.21.15.28:0/2264511720 shutdown_connections 2024-02-21T23:06:50.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.476+0000 7f64afe00640 1 -- 172.21.15.28:0/2264511720 wait complete. 2024-02-21T23:06:50.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.478+0000 7f64afe00640 1 Processor -- start 2024-02-21T23:06:50.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.478+0000 7f64afe00640 1 -- start start 2024-02-21T23:06:50.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.479+0000 7f64afe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64b006ee40 0x7f64b01e9030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:50.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.479+0000 7f64afe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64b0150450 0x7f64b00c4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:50.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.479+0000 7f64afe00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f64b0156af0 0x7f64b0150d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:50.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.479+0000 7f64afe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64b0156f80 con 0x7f64b006ee40 2024-02-21T23:06:50.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.479+0000 7f64afe00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64b00c2cb0 con 0x7f64b0150450 2024-02-21T23:06:50.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.479+0000 7f64afe00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64b0157100 con 0x7f64b0156af0 2024-02-21T23:06:50.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.479+0000 7f64aea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64b006ee40 0x7f64b01e9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:50.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.479+0000 7f64aea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64b006ee40 0x7f64b01e9030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36320/0 (socket says 172.21.15.28:36320) 2024-02-21T23:06:50.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.479+0000 7f64aea00640 1 -- 172.21.15.28:0/2730304143 learned_addr learned my addr 172.21.15.28:0/2730304143 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:50.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.479+0000 7f64ae000640 1 --2- 172.21.15.28:0/2730304143 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64b0150450 0x7f64b00c4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:50.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.479+0000 7f64aea00640 1 -- 172.21.15.28:0/2730304143 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f64b0156af0 msgr2=0x7f64b0150d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:50.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.479+0000 7f64aea00640 1 --2- 172.21.15.28:0/2730304143 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f64b0156af0 0x7f64b0150d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:50.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.479+0000 7f64aea00640 1 -- 172.21.15.28:0/2730304143 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64b0150450 msgr2=0x7f64b00c4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:50.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.479+0000 7f64aea00640 1 --2- 172.21.15.28:0/2730304143 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64b0150450 0x7f64b00c4bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:50.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.479+0000 7f64aea00640 1 -- 172.21.15.28:0/2730304143 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64b0155f70 con 0x7f64b006ee40 2024-02-21T23:06:50.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.480+0000 7f64aea00640 1 --2- 172.21.15.28:0/2730304143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64b006ee40 0x7f64b01e9030 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f64a000c4d0 tx=0x7f64a000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:50.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.480+0000 7f64a6a00640 1 -- 172.21.15.28:0/2730304143 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a0018070 con 0x7f64b006ee40 2024-02-21T23:06:50.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.480+0000 7f64a6a00640 1 -- 172.21.15.28:0/2730304143 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64a0007500 con 0x7f64b006ee40 2024-02-21T23:06:50.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.480+0000 7f64a6a00640 1 -- 172.21.15.28:0/2730304143 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a0004a20 con 0x7f64b006ee40 2024-02-21T23:06:50.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.480+0000 7f64afe00640 1 -- 172.21.15.28:0/2730304143 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64b0153920 con 0x7f64b006ee40 2024-02-21T23:06:50.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.480+0000 7f64afe00640 1 -- 172.21.15.28:0/2730304143 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64b00c5100 con 0x7f64b006ee40 2024-02-21T23:06:50.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.481+0000 7f64afe00640 1 -- 172.21.15.28:0/2730304143 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64b004f290 con 0x7f64b006ee40 2024-02-21T23:06:50.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.482+0000 7f64a6a00640 1 -- 172.21.15.28:0/2730304143 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f64a0004bc0 con 0x7f64b006ee40 2024-02-21T23:06:50.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.484+0000 7f64a6a00640 1 --2- 172.21.15.28:0/2730304143 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f649006b500 0x7f649006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:50.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.484+0000 7f64a6a00640 1 -- 172.21.15.28:0/2730304143 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f64a0089200 con 0x7f64b006ee40 2024-02-21T23:06:50.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.485+0000 7f64ae000640 1 --2- 172.21.15.28:0/2730304143 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f649006b500 0x7f649006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:50.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.486+0000 7f64ae000640 1 --2- 172.21.15.28:0/2730304143 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f649006b500 0x7f649006d9c0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f64b014ee20 tx=0x7f649c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:50.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.489+0000 7f64a6a00640 1 -- 172.21.15.28:0/2730304143 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f64a00582d0 con 0x7f64b006ee40 2024-02-21T23:06:50.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.694+0000 7f64afe00640 1 -- 172.21.15.28:0/2730304143 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f64b00c1f10 con 0x7f64b006ee40 2024-02-21T23:06:50.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.697+0000 7f64a6a00640 1 -- 172.21.15.28:0/2730304143 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) v1 ==== 74+0+13105 (secure 0 0 0) 0x7f64a0057bd0 con 0x7f64b006ee40 2024-02-21T23:06:50.686 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:06:50.687 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":94,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:06:49.706048+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:06:50.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.700+0000 7f64afe00640 1 -- 172.21.15.28:0/2730304143 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f649006b500 msgr2=0x7f649006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:50.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.700+0000 7f64afe00640 1 --2- 172.21.15.28:0/2730304143 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f649006b500 0x7f649006d9c0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f64b014ee20 tx=0x7f649c008040 comp rx=0 tx=0).stop 2024-02-21T23:06:50.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.700+0000 7f64afe00640 1 -- 172.21.15.28:0/2730304143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64b006ee40 msgr2=0x7f64b01e9030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:50.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.700+0000 7f64afe00640 1 --2- 172.21.15.28:0/2730304143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64b006ee40 0x7f64b01e9030 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f64a000c4d0 tx=0x7f64a000c990 comp rx=0 tx=0).stop 2024-02-21T23:06:50.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.700+0000 7f64afe00640 1 -- 172.21.15.28:0/2730304143 shutdown_connections 2024-02-21T23:06:50.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.700+0000 7f64afe00640 1 --2- 172.21.15.28:0/2730304143 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f649006b500 0x7f649006d9c0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:50.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.700+0000 7f64afe00640 1 --2- 172.21.15.28:0/2730304143 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f64b0156af0 0x7f64b0150d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:50.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.700+0000 7f64afe00640 1 --2- 172.21.15.28:0/2730304143 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64b0150450 0x7f64b00c4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:50.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.700+0000 7f64afe00640 1 --2- 172.21.15.28:0/2730304143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64b006ee40 0x7f64b01e9030 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:50.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.700+0000 7f64afe00640 1 -- 172.21.15.28:0/2730304143 >> 172.21.15.28:0/2730304143 conn(0x7f64b006b630 msgr2=0x7f64b014e580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:50.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.700+0000 7f64afe00640 1 -- 172.21.15.28:0/2730304143 shutdown_connections 2024-02-21T23:06:50.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.700+0000 7f64afe00640 1 -- 172.21.15.28:0/2730304143 wait complete. 2024-02-21T23:06:50.709 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-21T23:02:30.728329+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-21T23:06:50.709 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-21T23:06:50.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.937+0000 7f736d600640 1 Processor -- start 2024-02-21T23:06:50.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.938+0000 7f736d600640 1 -- start start 2024-02-21T23:06:50.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.939+0000 7f736d600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f73680c7900 0x7f7368158630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:50.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.939+0000 7f736d600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73680c6860 con 0x7f73681590b0 2024-02-21T23:06:50.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.939+0000 7f736d600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73680c6b40 con 0x7f7368152600 2024-02-21T23:06:50.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.939+0000 7f736d600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73680c6e20 con 0x7f73680c7900 2024-02-21T23:06:50.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.940+0000 7f7366000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f73681590b0 0x7f73680c2f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34286/0 (socket says 172.21.15.28:34286) 2024-02-21T23:06:50.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.940+0000 7f7366000640 1 -- 172.21.15.28:0/2491260221 learned_addr learned my addr 172.21.15.28:0/2491260221 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:06:50.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.940+0000 7f7367400640 1 --2- 172.21.15.28:0/2491260221 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f73680c7900 0x7f7368158630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:50.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.940+0000 7f7365600640 1 -- 172.21.15.28:0/2491260221 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1581733238 0 0) 0x7f73680c6860 con 0x7f73681590b0 2024-02-21T23:06:50.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.940+0000 7f7365600640 1 -- 172.21.15.28:0/2491260221 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f734c003520 con 0x7f73681590b0 2024-02-21T23:06:50.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.941+0000 7f7365600640 1 -- 172.21.15.28:0/2491260221 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3749345518 0 0) 0x7f73680c6b40 con 0x7f7368152600 2024-02-21T23:06:50.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.941+0000 7f7365600640 1 -- 172.21.15.28:0/2491260221 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73680c6860 con 0x7f7368152600 2024-02-21T23:06:50.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.941+0000 7f7365600640 1 -- 172.21.15.28:0/2491260221 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2801104696 0 0) 0x7f734c003520 con 0x7f73681590b0 2024-02-21T23:06:50.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.941+0000 7f7365600640 1 -- 172.21.15.28:0/2491260221 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f73680c7900 msgr2=0x7f7368158630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:50.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.941+0000 7f7365600640 1 --2- 172.21.15.28:0/2491260221 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f73680c7900 0x7f7368158630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:50.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.941+0000 7f7365600640 1 -- 172.21.15.28:0/2491260221 >> v1:172.21.15.81:6789/0 conn(0x7f7368152600 legacy=0x7f73680c5d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:50.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.941+0000 7f7365600640 1 -- 172.21.15.28:0/2491260221 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73680c7100 con 0x7f73681590b0 2024-02-21T23:06:50.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.941+0000 7f7365600640 1 -- 172.21.15.28:0/2491260221 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f7354001670 con 0x7f73681590b0 2024-02-21T23:06:50.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.941+0000 7f7365600640 1 -- 172.21.15.28:0/2491260221 >> v1:172.21.15.28:6789/0 conn(0x7f73681590b0 legacy=0x7f73680c2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:50.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.941+0000 7f7365600640 1 --2- 172.21.15.28:0/2491260221 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f734c003b70 0x7f734c003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:50.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.941+0000 7f7365600640 1 --2- 172.21.15.28:0/2491260221 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f734c004800 0x7f734c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:50.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.941+0000 7f7365600640 1 --2- 172.21.15.28:0/2491260221 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f734c005290 0x7f734c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:50.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.941+0000 7f7365600640 1 -- 172.21.15.28:0/2491260221 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73680c6e20 con 0x7f734c003b70 2024-02-21T23:06:50.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.941+0000 7f7365600640 1 -- 172.21.15.28:0/2491260221 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f734c00c160 con 0x7f734c004800 2024-02-21T23:06:50.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.941+0000 7f7365600640 1 -- 172.21.15.28:0/2491260221 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f734c00c360 con 0x7f734c005290 2024-02-21T23:06:50.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.941+0000 7f7367400640 1 --2- 172.21.15.28:0/2491260221 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f73680c7900 0x7f7368158630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-21T23:06:50.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.941+0000 7f7366a00640 1 --2- 172.21.15.28:0/2491260221 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f734c003b70 0x7f734c003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:50.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.942+0000 7f7366000640 1 --2- 172.21.15.28:0/2491260221 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f734c004800 0x7f734c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:50.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.942+0000 7f7367400640 1 --2- 172.21.15.28:0/2491260221 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f734c005290 0x7f734c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:50.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.942+0000 7f7366a00640 1 -- 172.21.15.28:0/2491260221 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f734c005290 msgr2=0x7f734c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:50.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.942+0000 7f7366a00640 1 --2- 172.21.15.28:0/2491260221 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f734c005290 0x7f734c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:50.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.942+0000 7f7366a00640 1 -- 172.21.15.28:0/2491260221 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f734c004800 msgr2=0x7f734c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:50.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.942+0000 7f7366a00640 1 --2- 172.21.15.28:0/2491260221 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f734c004800 0x7f734c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:50.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.942+0000 7f7366a00640 1 -- 172.21.15.28:0/2491260221 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f734c003520 con 0x7f734c003b70 2024-02-21T23:06:50.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.943+0000 7f7366a00640 1 --2- 172.21.15.28:0/2491260221 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f734c003b70 0x7f734c003f70 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f73580018d0 tx=0x7f734c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40633cc6e4ca6d85 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:50.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.943+0000 7f7365600640 1 -- 172.21.15.28:0/2491260221 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7358037070 con 0x7f734c003b70 2024-02-21T23:06:50.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.943+0000 7f7365600640 1 -- 172.21.15.28:0/2491260221 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73580052d0 con 0x7f734c003b70 2024-02-21T23:06:50.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.943+0000 7f7365600640 1 -- 172.21.15.28:0/2491260221 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73580014f0 con 0x7f734c003b70 2024-02-21T23:06:50.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.944+0000 7f736d600640 1 -- 172.21.15.28:0/2491260221 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f734c003b70 msgr2=0x7f734c003f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:50.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.944+0000 7f736d600640 1 --2- 172.21.15.28:0/2491260221 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f734c003b70 0x7f734c003f70 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f73580018d0 tx=0x7f734c005830 comp rx=0 tx=0).stop 2024-02-21T23:06:50.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.944+0000 7f7367400640 1 -- 172.21.15.28:0/2491260221 reap_dead start 2024-02-21T23:06:50.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.944+0000 7f736d600640 1 -- 172.21.15.28:0/2491260221 shutdown_connections 2024-02-21T23:06:50.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.944+0000 7f736d600640 1 -- 172.21.15.28:0/2491260221 >> 172.21.15.28:0/2491260221 conn(0x7f7368072bd0 msgr2=0x7f736806d0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:50.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.944+0000 7f736d600640 1 -- 172.21.15.28:0/2491260221 shutdown_connections 2024-02-21T23:06:50.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.944+0000 7f736d600640 1 -- 172.21.15.28:0/2491260221 wait complete. 2024-02-21T23:06:50.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.946+0000 7f736d600640 1 Processor -- start 2024-02-21T23:06:50.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.946+0000 7f736d600640 1 -- start start 2024-02-21T23:06:50.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.947+0000 7f736d600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f734c003b70 0x7f7368153780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:50.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.947+0000 7f736d600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f73681eede0 0x7f73681eb5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:50.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.947+0000 7f736d600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f73681ebb20 0x7f73681450e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:50.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.947+0000 7f736d600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73680c3640 con 0x7f73681ebb20 2024-02-21T23:06:50.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.947+0000 7f736d600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73681529e0 con 0x7f734c003b70 2024-02-21T23:06:50.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.947+0000 7f736d600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7368158630 con 0x7f73681eede0 2024-02-21T23:06:50.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.947+0000 7f7367400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f73681ebb20 0x7f73681450e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:50.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.948+0000 7f7367400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f73681ebb20 0x7f73681450e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36338/0 (socket says 172.21.15.28:36338) 2024-02-21T23:06:50.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.948+0000 7f7367400640 1 -- 172.21.15.28:0/902795118 learned_addr learned my addr 172.21.15.28:0/902795118 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:50.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.948+0000 7f7366a00640 1 --2- 172.21.15.28:0/902795118 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f734c003b70 0x7f7368153780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:50.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.948+0000 7f7366000640 1 --2- 172.21.15.28:0/902795118 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f73681eede0 0x7f73681eb5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:50.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.948+0000 7f7367400640 1 -- 172.21.15.28:0/902795118 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f73681eede0 msgr2=0x7f73681eb5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:50.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.948+0000 7f7367400640 1 --2- 172.21.15.28:0/902795118 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f73681eede0 0x7f73681eb5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:50.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.948+0000 7f7367400640 1 -- 172.21.15.28:0/902795118 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f734c003b70 msgr2=0x7f7368153780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:50.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.948+0000 7f7367400640 1 --2- 172.21.15.28:0/902795118 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f734c003b70 0x7f7368153780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:50.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.948+0000 7f7367400640 1 -- 172.21.15.28:0/902795118 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73681ef2c0 con 0x7f73681ebb20 2024-02-21T23:06:50.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.948+0000 7f7367400640 1 --2- 172.21.15.28:0/902795118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f73681ebb20 0x7f73681450e0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f73680c7d90 tx=0x7f7360011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:50.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.949+0000 7f735ea00640 1 -- 172.21.15.28:0/902795118 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7360003990 con 0x7f73681ebb20 2024-02-21T23:06:50.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.949+0000 7f735ea00640 1 -- 172.21.15.28:0/902795118 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f736001b070 con 0x7f73681ebb20 2024-02-21T23:06:50.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.949+0000 7f736d600640 1 -- 172.21.15.28:0/902795118 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73681493c0 con 0x7f73681ebb20 2024-02-21T23:06:50.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.949+0000 7f735ea00640 1 -- 172.21.15.28:0/902795118 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7360003b50 con 0x7f73681ebb20 2024-02-21T23:06:50.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.949+0000 7f736d600640 1 -- 172.21.15.28:0/902795118 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73681eb0c0 con 0x7f73681ebb20 2024-02-21T23:06:50.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.950+0000 7f736d600640 1 -- 172.21.15.28:0/902795118 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73681498e0 con 0x7f73681ebb20 2024-02-21T23:06:50.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.951+0000 7f735ea00640 1 -- 172.21.15.28:0/902795118 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7360005d10 con 0x7f73681ebb20 2024-02-21T23:06:50.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.952+0000 7f735ea00640 1 --2- 172.21.15.28:0/902795118 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f734806b4d0 0x7f734806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:50.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.952+0000 7f735ea00640 1 -- 172.21.15.28:0/902795118 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f736008a8d0 con 0x7f73681ebb20 2024-02-21T23:06:50.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.953+0000 7f7366a00640 1 --2- 172.21.15.28:0/902795118 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f734806b4d0 0x7f734806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:50.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.953+0000 7f7366a00640 1 --2- 172.21.15.28:0/902795118 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f734806b4d0 0x7f734806d990 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f7358034000 tx=0x7f7358031000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:50.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:50.956+0000 7f735ea00640 1 -- 172.21.15.28:0/902795118 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f73681498e0 con 0x7f73681ebb20 2024-02-21T23:06:51.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.159+0000 7f736d600640 1 -- 172.21.15.28:0/902795118 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f73681eaed0 con 0x7f73681ebb20 2024-02-21T23:06:51.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.160+0000 7f735ea00640 1 -- 172.21.15.28:0/902795118 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) v1 ==== 93+0+10 (secure 0 0 0) 0x7f73600597e0 con 0x7f73681ebb20 2024-02-21T23:06:51.149 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-02-21T23:06:51.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.162+0000 7f736d600640 1 -- 172.21.15.28:0/902795118 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f734806b4d0 msgr2=0x7f734806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:51.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.162+0000 7f736d600640 1 --2- 172.21.15.28:0/902795118 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f734806b4d0 0x7f734806d990 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f7358034000 tx=0x7f7358031000 comp rx=0 tx=0).stop 2024-02-21T23:06:51.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.162+0000 7f736d600640 1 -- 172.21.15.28:0/902795118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f73681ebb20 msgr2=0x7f73681450e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:51.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.162+0000 7f736d600640 1 --2- 172.21.15.28:0/902795118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f73681ebb20 0x7f73681450e0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f73680c7d90 tx=0x7f7360011890 comp rx=0 tx=0).stop 2024-02-21T23:06:51.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.162+0000 7f736d600640 1 -- 172.21.15.28:0/902795118 shutdown_connections 2024-02-21T23:06:51.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.162+0000 7f736d600640 1 --2- 172.21.15.28:0/902795118 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f734806b4d0 0x7f734806d990 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:51.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.162+0000 7f736d600640 1 --2- 172.21.15.28:0/902795118 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f73681eede0 0x7f73681eb5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:51.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.162+0000 7f736d600640 1 --2- 172.21.15.28:0/902795118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f73681ebb20 0x7f73681450e0 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:51.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.162+0000 7f736d600640 1 --2- 172.21.15.28:0/902795118 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f734c003b70 0x7f7368153780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:51.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.162+0000 7f736d600640 1 -- 172.21.15.28:0/902795118 >> 172.21.15.28:0/902795118 conn(0x7f7368072bd0 msgr2=0x7f736814ea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:51.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.162+0000 7f736d600640 1 -- 172.21.15.28:0/902795118 shutdown_connections 2024-02-21T23:06:51.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.162+0000 7f736d600640 1 -- 172.21.15.28:0/902795118 wait complete. 2024-02-21T23:06:51.173 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-02-21T23:06:51.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.410+0000 7f3eade00640 1 Processor -- start 2024-02-21T23:06:51.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.410+0000 7f3eade00640 1 -- start start 2024-02-21T23:06:51.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.411+0000 7f3eade00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3ea80b0130 0x7f3ea80c8740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:51.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.412+0000 7f3eade00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f3ea8158c20 0x7f3ea80c5670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:51.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.412+0000 7f3eade00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ea81550b0 con 0x7f3ea80c51c0 2024-02-21T23:06:51.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.412+0000 7f3eade00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ea8152a40 con 0x7f3ea80b0130 2024-02-21T23:06:51.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.412+0000 7f3eade00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ea8155350 con 0x7f3ea8158c20 2024-02-21T23:06:51.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.412+0000 7f3eaca00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3ea80b0130 0x7f3ea80c8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:51.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.412+0000 7f3ea7e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f3ea80c51c0 0x7f3ea8158510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34296/0 (socket says 172.21.15.28:34296) 2024-02-21T23:06:51.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.412+0000 7f3ea7e00640 1 -- 172.21.15.28:0/1342530735 learned_addr learned my addr 172.21.15.28:0/1342530735 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:06:51.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.412+0000 7f3ead400640 1 --2- 172.21.15.28:0/1342530735 >> v2:172.21.15.81:3300/0 conn(0x7f3ea8158c20 0x7f3ea80c5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:51.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.412+0000 7f3eaca00640 1 -- 172.21.15.28:0/1342530735 >> v2:172.21.15.81:3300/0 conn(0x7f3ea8158c20 msgr2=0x7f3ea80c5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:51.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.413+0000 7f3eaca00640 1 --2- 172.21.15.28:0/1342530735 >> v2:172.21.15.81:3300/0 conn(0x7f3ea8158c20 0x7f3ea80c5670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:51.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.413+0000 7f3eaca00640 1 -- 172.21.15.28:0/1342530735 >> v1:172.21.15.28:6789/0 conn(0x7f3ea80c51c0 legacy=0x7f3ea8158510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:51.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.413+0000 7f3eaca00640 1 -- 172.21.15.28:0/1342530735 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ea8155630 con 0x7f3ea80b0130 2024-02-21T23:06:51.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.414+0000 7f3eaca00640 1 --2- 172.21.15.28:0/1342530735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ea80b0130 0x7f3ea80c8740 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f3ea81595d0 tx=0x7f3e9802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15fe92dca0bda173 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:51.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.415+0000 7f3ea7400640 1 -- 172.21.15.28:0/1342530735 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e980315a0 con 0x7f3ea80b0130 2024-02-21T23:06:51.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.415+0000 7f3ea7400640 1 -- 172.21.15.28:0/1342530735 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e98037070 con 0x7f3ea80b0130 2024-02-21T23:06:51.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.415+0000 7f3ea7400640 1 -- 172.21.15.28:0/1342530735 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e98031960 con 0x7f3ea80b0130 2024-02-21T23:06:51.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.415+0000 7f3eade00640 1 -- 172.21.15.28:0/1342530735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ea80b0130 msgr2=0x7f3ea80c8740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:51.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.415+0000 7f3eade00640 1 --2- 172.21.15.28:0/1342530735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ea80b0130 0x7f3ea80c8740 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f3ea81595d0 tx=0x7f3e9802b840 comp rx=0 tx=0).stop 2024-02-21T23:06:51.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.416+0000 7f3eade00640 1 -- 172.21.15.28:0/1342530735 shutdown_connections 2024-02-21T23:06:51.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.416+0000 7f3eade00640 1 --2- 172.21.15.28:0/1342530735 >> v2:172.21.15.81:3300/0 conn(0x7f3ea8158c20 0x7f3ea80c5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:51.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.416+0000 7f3eade00640 1 --2- 172.21.15.28:0/1342530735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ea80b0130 0x7f3ea80c8740 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:51.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.416+0000 7f3eade00640 1 -- 172.21.15.28:0/1342530735 >> 172.21.15.28:0/1342530735 conn(0x7f3ea80ac370 msgr2=0x7f3ea80ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:51.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.416+0000 7f3eade00640 1 -- 172.21.15.28:0/1342530735 shutdown_connections 2024-02-21T23:06:51.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.416+0000 7f3eade00640 1 -- 172.21.15.28:0/1342530735 wait complete. 2024-02-21T23:06:51.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.418+0000 7f3eade00640 1 Processor -- start 2024-02-21T23:06:51.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.418+0000 7f3eade00640 1 -- start start 2024-02-21T23:06:51.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.419+0000 7f3eade00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3ea80b0130 0x7f3ea8158200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:51.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.419+0000 7f3eade00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3ea80c51c0 0x7f3ea8151090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:51.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.419+0000 7f3eade00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ea8158c20 0x7f3ea81515d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:51.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.419+0000 7f3eade00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea80c8dd0 con 0x7f3ea8158c20 2024-02-21T23:06:51.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.419+0000 7f3eade00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea8155e30 con 0x7f3ea80c51c0 2024-02-21T23:06:51.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.419+0000 7f3eade00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea80be060 con 0x7f3ea80b0130 2024-02-21T23:06:51.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.420+0000 7f3ead400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ea8158c20 0x7f3ea81515d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:51.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.420+0000 7f3ead400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ea8158c20 0x7f3ea81515d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36350/0 (socket says 172.21.15.28:36350) 2024-02-21T23:06:51.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.420+0000 7f3ead400640 1 -- 172.21.15.28:0/3910147796 learned_addr learned my addr 172.21.15.28:0/3910147796 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:51.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.420+0000 7f3ea7e00640 1 --2- 172.21.15.28:0/3910147796 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3ea80c51c0 0x7f3ea8151090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:51.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.420+0000 7f3eaca00640 1 --2- 172.21.15.28:0/3910147796 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3ea80b0130 0x7f3ea8158200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:51.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.420+0000 7f3ead400640 1 -- 172.21.15.28:0/3910147796 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3ea80b0130 msgr2=0x7f3ea8158200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:51.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.420+0000 7f3ead400640 1 --2- 172.21.15.28:0/3910147796 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3ea80b0130 0x7f3ea8158200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:51.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.420+0000 7f3ead400640 1 -- 172.21.15.28:0/3910147796 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3ea80c51c0 msgr2=0x7f3ea8151090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:51.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.420+0000 7f3ead400640 1 --2- 172.21.15.28:0/3910147796 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3ea80c51c0 0x7f3ea8151090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:51.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.420+0000 7f3ead400640 1 -- 172.21.15.28:0/3910147796 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ea80c3f40 con 0x7f3ea8158c20 2024-02-21T23:06:51.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.420+0000 7f3ea7e00640 1 --2- 172.21.15.28:0/3910147796 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3ea80c51c0 0x7f3ea8151090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:06:51.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.421+0000 7f3ead400640 1 --2- 172.21.15.28:0/3910147796 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ea8158c20 0x7f3ea81515d0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f3ea000cc60 tx=0x7f3ea000a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:51.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.421+0000 7f3ea4c00640 1 -- 172.21.15.28:0/3910147796 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea0003990 con 0x7f3ea8158c20 2024-02-21T23:06:51.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.421+0000 7f3ea4c00640 1 -- 172.21.15.28:0/3910147796 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ea0003b30 con 0x7f3ea8158c20 2024-02-21T23:06:51.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.421+0000 7f3ea4c00640 1 -- 172.21.15.28:0/3910147796 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea0004020 con 0x7f3ea8158c20 2024-02-21T23:06:51.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.421+0000 7f3eade00640 1 -- 172.21.15.28:0/3910147796 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ea81eeef0 con 0x7f3ea8158c20 2024-02-21T23:06:51.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.421+0000 7f3eade00640 1 -- 172.21.15.28:0/3910147796 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ea8151e50 con 0x7f3ea8158c20 2024-02-21T23:06:51.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.422+0000 7f3eade00640 1 -- 172.21.15.28:0/3910147796 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ea804f290 con 0x7f3ea8158c20 2024-02-21T23:06:51.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.423+0000 7f3ea4c00640 1 -- 172.21.15.28:0/3910147796 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3ea00041c0 con 0x7f3ea8158c20 2024-02-21T23:06:51.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.424+0000 7f3ea4c00640 1 --2- 172.21.15.28:0/3910147796 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3e8406b4d0 0x7f3e8406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:51.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.424+0000 7f3ea4c00640 1 -- 172.21.15.28:0/3910147796 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f3ea008a490 con 0x7f3ea8158c20 2024-02-21T23:06:51.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.425+0000 7f3eaca00640 1 --2- 172.21.15.28:0/3910147796 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3e8406b4d0 0x7f3e8406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:51.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.426+0000 7f3eaca00640 1 --2- 172.21.15.28:0/3910147796 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3e8406b4d0 0x7f3e8406d990 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f3e98036000 tx=0x7f3e98008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:51.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.428+0000 7f3ea4c00640 1 -- 172.21.15.28:0/3910147796 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3ea80c3f40 con 0x7f3ea8158c20 2024-02-21T23:06:51.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.628+0000 7f3eade00640 1 -- 172.21.15.28:0/3910147796 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f3ea80c16d0 con 0x7f3ea8158c20 2024-02-21T23:06:51.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.628+0000 7f3ea4c00640 1 -- 172.21.15.28:0/3910147796 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v94) v1 ==== 84+0+65 (secure 0 0 0) 0x7f3ea0014030 con 0x7f3ea8158c20 2024-02-21T23:06:51.618 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:06:51.618 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-02-21T23:06:51.618 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:06:51.618 INFO:teuthology.orchestra.run.smithi028.stdout: "poolnum": 1, 2024-02-21T23:06:51.618 INFO:teuthology.orchestra.run.smithi028.stdout: "poolname": ".mgr" 2024-02-21T23:06:51.618 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:06:51.618 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-02-21T23:06:51.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.632+0000 7f3eade00640 1 -- 172.21.15.28:0/3910147796 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3e8406b4d0 msgr2=0x7f3e8406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:51.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.632+0000 7f3eade00640 1 --2- 172.21.15.28:0/3910147796 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3e8406b4d0 0x7f3e8406d990 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f3e98036000 tx=0x7f3e98008030 comp rx=0 tx=0).stop 2024-02-21T23:06:51.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.632+0000 7f3eade00640 1 -- 172.21.15.28:0/3910147796 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ea8158c20 msgr2=0x7f3ea81515d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:51.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.632+0000 7f3eade00640 1 --2- 172.21.15.28:0/3910147796 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ea8158c20 0x7f3ea81515d0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f3ea000cc60 tx=0x7f3ea000a5c0 comp rx=0 tx=0).stop 2024-02-21T23:06:51.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.632+0000 7f3eade00640 1 -- 172.21.15.28:0/3910147796 shutdown_connections 2024-02-21T23:06:51.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.632+0000 7f3eade00640 1 --2- 172.21.15.28:0/3910147796 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3e8406b4d0 0x7f3e8406d990 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:51.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.632+0000 7f3eade00640 1 --2- 172.21.15.28:0/3910147796 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ea8158c20 0x7f3ea81515d0 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:51.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.632+0000 7f3eade00640 1 --2- 172.21.15.28:0/3910147796 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3ea80c51c0 0x7f3ea8151090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:51.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.632+0000 7f3eade00640 1 --2- 172.21.15.28:0/3910147796 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3ea80b0130 0x7f3ea8158200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:51.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.632+0000 7f3eade00640 1 -- 172.21.15.28:0/3910147796 >> 172.21.15.28:0/3910147796 conn(0x7f3ea80ac370 msgr2=0x7f3ea80ab990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:51.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.632+0000 7f3eade00640 1 -- 172.21.15.28:0/3910147796 shutdown_connections 2024-02-21T23:06:51.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.632+0000 7f3eade00640 1 -- 172.21.15.28:0/3910147796 wait complete. 2024-02-21T23:06:51.638 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-02-21T23:06:51.639 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-02-21T23:06:51.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.872+0000 7f4ef3e00640 1 Processor -- start 2024-02-21T23:06:51.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.872+0000 7f4ef3e00640 1 -- start start 2024-02-21T23:06:51.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.874+0000 7f4ef3e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4ef406cfa0 0x7f4ef40ca540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:51.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.874+0000 7f4ef3e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4ef4158460 0x7f4ef40c6290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:51.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.874+0000 7f4ef3e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ef40caf40 con 0x7f4ef40c8f70 2024-02-21T23:06:51.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.874+0000 7f4ef3e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ef40c8ae0 con 0x7f4ef4158460 2024-02-21T23:06:51.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.874+0000 7f4ef3e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef40c87b0 con 0x7f4ef406cfa0 2024-02-21T23:06:51.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.875+0000 7f4eea000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4ef4158460 0x7f4ef40c6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:51.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.875+0000 7f4eea000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4ef4158460 0x7f4ef40c6290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36356/0 (socket says 172.21.15.28:36356) 2024-02-21T23:06:51.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.875+0000 7f4eea000640 1 -- 172.21.15.28:0/2498392439 learned_addr learned my addr 172.21.15.28:0/2498392439 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:51.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.875+0000 7f4ef2a00640 1 --2- 172.21.15.28:0/2498392439 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4ef406cfa0 0x7f4ef40ca540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:51.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.875+0000 7f4eea000640 1 -- 172.21.15.28:0/2498392439 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4ef406cfa0 msgr2=0x7f4ef40ca540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:51.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.875+0000 7f4eea000640 1 --2- 172.21.15.28:0/2498392439 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4ef406cfa0 0x7f4ef40ca540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:51.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.875+0000 7f4eea000640 1 -- 172.21.15.28:0/2498392439 >> v1:172.21.15.81:6789/0 conn(0x7f4ef40c8f70 legacy=0x7f4ef41522c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:51.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.875+0000 7f4eea000640 1 -- 172.21.15.28:0/2498392439 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ef41588c0 con 0x7f4ef4158460 2024-02-21T23:06:51.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.875+0000 7f4ef2a00640 1 --2- 172.21.15.28:0/2498392439 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4ef406cfa0 0x7f4ef40ca540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:06:51.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.876+0000 7f4eea000640 1 --2- 172.21.15.28:0/2498392439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ef4158460 0x7f4ef40c6290 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f4ef40c8c60 tx=0x7f4edc02b850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf1bdbb37f7d2d9b server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:51.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.877+0000 7f4ef2000640 1 -- 172.21.15.28:0/2498392439 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4edc032640 con 0x7f4ef4158460 2024-02-21T23:06:51.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.877+0000 7f4ef2000640 1 -- 172.21.15.28:0/2498392439 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4edc037070 con 0x7f4ef4158460 2024-02-21T23:06:51.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.877+0000 7f4ef2000640 1 -- 172.21.15.28:0/2498392439 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4edc032a00 con 0x7f4ef4158460 2024-02-21T23:06:51.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.877+0000 7f4ef3e00640 1 -- 172.21.15.28:0/2498392439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ef4158460 msgr2=0x7f4ef40c6290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:51.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.878+0000 7f4ef3e00640 1 --2- 172.21.15.28:0/2498392439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ef4158460 0x7f4ef40c6290 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f4ef40c8c60 tx=0x7f4edc02b850 comp rx=0 tx=0).stop 2024-02-21T23:06:51.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.878+0000 7f4ef3e00640 1 -- 172.21.15.28:0/2498392439 shutdown_connections 2024-02-21T23:06:51.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.878+0000 7f4ef3e00640 1 --2- 172.21.15.28:0/2498392439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ef4158460 0x7f4ef40c6290 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:51.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.878+0000 7f4ef3e00640 1 --2- 172.21.15.28:0/2498392439 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4ef406cfa0 0x7f4ef40ca540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:51.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.878+0000 7f4ef3e00640 1 -- 172.21.15.28:0/2498392439 >> 172.21.15.28:0/2498392439 conn(0x7f4ef40725a0 msgr2=0x7f4ef40729b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:51.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.878+0000 7f4ef3e00640 1 -- 172.21.15.28:0/2498392439 shutdown_connections 2024-02-21T23:06:51.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.878+0000 7f4ef3e00640 1 -- 172.21.15.28:0/2498392439 wait complete. 2024-02-21T23:06:51.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.879+0000 7f4ef3e00640 1 Processor -- start 2024-02-21T23:06:51.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.881+0000 7f4ef3e00640 1 -- start start 2024-02-21T23:06:51.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.881+0000 7f4ef3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ef406cfa0 0x7f4ef40aba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:51.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.881+0000 7f4ef3e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4ef40c8f70 0x7f4ef41574f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:51.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.881+0000 7f4ef3e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ef4158460 0x7f4ef41ee450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:51.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.881+0000 7f4ef3e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef40c6c90 con 0x7f4ef406cfa0 2024-02-21T23:06:51.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.881+0000 7f4ef3e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef40c69b0 con 0x7f4ef4158460 2024-02-21T23:06:51.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.881+0000 7f4ef3e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef41529d0 con 0x7f4ef40c8f70 2024-02-21T23:06:51.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.881+0000 7f4ef2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ef406cfa0 0x7f4ef40aba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:51.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.881+0000 7f4ef2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ef406cfa0 0x7f4ef40aba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36366/0 (socket says 172.21.15.28:36366) 2024-02-21T23:06:51.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.881+0000 7f4ef2a00640 1 -- 172.21.15.28:0/3903714274 learned_addr learned my addr 172.21.15.28:0/3903714274 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:51.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.881+0000 7f4ef2a00640 1 -- 172.21.15.28:0/3903714274 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4ef40c8f70 msgr2=0x7f4ef41574f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:51.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.881+0000 7f4ef2a00640 1 --2- 172.21.15.28:0/3903714274 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4ef40c8f70 0x7f4ef41574f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:51.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.881+0000 7f4ef2a00640 1 -- 172.21.15.28:0/3903714274 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ef4158460 msgr2=0x7f4ef41ee450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:51.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.881+0000 7f4ef2a00640 1 --2- 172.21.15.28:0/3903714274 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ef4158460 0x7f4ef41ee450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:51.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.881+0000 7f4ef2a00640 1 -- 172.21.15.28:0/3903714274 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ef414d720 con 0x7f4ef406cfa0 2024-02-21T23:06:51.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.882+0000 7f4ef2a00640 1 --2- 172.21.15.28:0/3903714274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ef406cfa0 0x7f4ef40aba50 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f4ee400cf60 tx=0x7f4ee400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:51.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.882+0000 7f4eeb400640 1 -- 172.21.15.28:0/3903714274 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee401a070 con 0x7f4ef406cfa0 2024-02-21T23:06:51.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.882+0000 7f4eeb400640 1 -- 172.21.15.28:0/3903714274 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ee400d840 con 0x7f4ef406cfa0 2024-02-21T23:06:51.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.882+0000 7f4eeb400640 1 -- 172.21.15.28:0/3903714274 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee400dc10 con 0x7f4ef406cfa0 2024-02-21T23:06:51.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.882+0000 7f4ef3e00640 1 -- 172.21.15.28:0/3903714274 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ef414da00 con 0x7f4ef406cfa0 2024-02-21T23:06:51.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.882+0000 7f4ef3e00640 1 -- 172.21.15.28:0/3903714274 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ef406dd50 con 0x7f4ef406cfa0 2024-02-21T23:06:51.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.883+0000 7f4ef3e00640 1 -- 172.21.15.28:0/3903714274 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ef41edf60 con 0x7f4ef406cfa0 2024-02-21T23:06:51.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.884+0000 7f4eeb400640 1 -- 172.21.15.28:0/3903714274 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4ee401d070 con 0x7f4ef406cfa0 2024-02-21T23:06:51.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.887+0000 7f4eeb400640 1 --2- 172.21.15.28:0/3903714274 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4ed006b4d0 0x7f4ed006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:51.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.887+0000 7f4eeb400640 1 -- 172.21.15.28:0/3903714274 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f4ee408a860 con 0x7f4ef406cfa0 2024-02-21T23:06:51.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.887+0000 7f4eea000640 1 --2- 172.21.15.28:0/3903714274 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4ed006b4d0 0x7f4ed006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:51.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.887+0000 7f4eea000640 1 --2- 172.21.15.28:0/3903714274 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4ed006b4d0 0x7f4ed006d990 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f4edc036000 tx=0x7f4edc030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:51.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:51.892+0000 7f4eeb400640 1 -- 172.21.15.28:0/3903714274 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4ee4059240 con 0x7f4ef406cfa0 2024-02-21T23:06:52.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.093+0000 7f4ef3e00640 1 -- 172.21.15.28:0/3903714274 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f4ef404f290 con 0x7f4ef406cfa0 2024-02-21T23:06:52.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.706+0000 7f4eeb400640 1 -- 172.21.15.28:0/3903714274 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v95) v1 ==== 138+0+0 (secure 0 0 0) 0x7f4ee405d2e0 con 0x7f4ef406cfa0 2024-02-21T23:06:52.695 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' created 2024-02-21T23:06:52.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.710+0000 7f4ef3e00640 1 -- 172.21.15.28:0/3903714274 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4ed006b4d0 msgr2=0x7f4ed006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:52.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.710+0000 7f4ef3e00640 1 --2- 172.21.15.28:0/3903714274 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4ed006b4d0 0x7f4ed006d990 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f4edc036000 tx=0x7f4edc030000 comp rx=0 tx=0).stop 2024-02-21T23:06:52.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.710+0000 7f4ef3e00640 1 -- 172.21.15.28:0/3903714274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ef406cfa0 msgr2=0x7f4ef40aba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:52.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.710+0000 7f4ef3e00640 1 --2- 172.21.15.28:0/3903714274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ef406cfa0 0x7f4ef40aba50 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f4ee400cf60 tx=0x7f4ee400cf90 comp rx=0 tx=0).stop 2024-02-21T23:06:52.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.711+0000 7f4ef3e00640 1 -- 172.21.15.28:0/3903714274 shutdown_connections 2024-02-21T23:06:52.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.711+0000 7f4ef3e00640 1 --2- 172.21.15.28:0/3903714274 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4ed006b4d0 0x7f4ed006d990 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:52.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.711+0000 7f4ef3e00640 1 --2- 172.21.15.28:0/3903714274 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4ef40c8f70 0x7f4ef41574f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:52.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.711+0000 7f4ef3e00640 1 --2- 172.21.15.28:0/3903714274 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ef4158460 0x7f4ef41ee450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:52.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.711+0000 7f4ef3e00640 1 --2- 172.21.15.28:0/3903714274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ef406cfa0 0x7f4ef40aba50 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:52.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.711+0000 7f4ef3e00640 1 -- 172.21.15.28:0/3903714274 >> 172.21.15.28:0/3903714274 conn(0x7f4ef40725a0 msgr2=0x7f4ef41de2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:52.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.711+0000 7f4ef3e00640 1 -- 172.21.15.28:0/3903714274 shutdown_connections 2024-02-21T23:06:52.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.712+0000 7f4ef3e00640 1 -- 172.21.15.28:0/3903714274 wait complete. 2024-02-21T23:06:52.725 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-02-21T23:06:52.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.948+0000 7f7e18600640 1 Processor -- start 2024-02-21T23:06:52.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.949+0000 7f7e18600640 1 -- start start 2024-02-21T23:06:52.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.950+0000 7f7e18600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f7e100c22e0 0x7f7e1006d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:52.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.950+0000 7f7e18600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7e101561a0 0x7f7e1006d660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:52.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.950+0000 7f7e18600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e10071970 con 0x7f7e100c5a70 2024-02-21T23:06:52.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.950+0000 7f7e18600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e10072a10 con 0x7f7e101561a0 2024-02-21T23:06:52.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.950+0000 7f7e18600640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e10072580 con 0x7f7e100c22e0 2024-02-21T23:06:52.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.950+0000 7f7e15c00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f7e100c5a70 0x7f7e10155a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34310/0 (socket says 172.21.15.28:34310) 2024-02-21T23:06:52.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.950+0000 7f7e15c00640 1 -- 172.21.15.28:0/501886115 learned_addr learned my addr 172.21.15.28:0/501886115 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:06:52.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.950+0000 7f7e16600640 1 --2- 172.21.15.28:0/501886115 >> v2:172.21.15.28:3300/0 conn(0x7f7e101561a0 0x7f7e1006d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:52.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.951+0000 7f7e15200640 1 --2- 172.21.15.28:0/501886115 >> v2:172.21.15.81:3300/0 conn(0x7f7e100c22e0 0x7f7e1006d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:52.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.951+0000 7f7e0fe00640 1 -- 172.21.15.28:0/501886115 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2467869606 0 0) 0x7f7e10071970 con 0x7f7e100c5a70 2024-02-21T23:06:52.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.951+0000 7f7e0fe00640 1 -- 172.21.15.28:0/501886115 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7df4003520 con 0x7f7e100c5a70 2024-02-21T23:06:52.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.951+0000 7f7e16600640 1 -- 172.21.15.28:0/501886115 >> v2:172.21.15.81:3300/0 conn(0x7f7e100c22e0 msgr2=0x7f7e1006d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:52.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.951+0000 7f7e16600640 1 --2- 172.21.15.28:0/501886115 >> v2:172.21.15.81:3300/0 conn(0x7f7e100c22e0 0x7f7e1006d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:52.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.951+0000 7f7e16600640 1 -- 172.21.15.28:0/501886115 >> v1:172.21.15.28:6789/0 conn(0x7f7e100c5a70 legacy=0x7f7e10155a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:52.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.951+0000 7f7e16600640 1 -- 172.21.15.28:0/501886115 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e10071c50 con 0x7f7e101561a0 2024-02-21T23:06:52.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.952+0000 7f7e16600640 1 --2- 172.21.15.28:0/501886115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e101561a0 0x7f7e1006d660 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f7e1006f5d0 tx=0x7f7e0802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74dcce4d45670b05 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:52.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.952+0000 7f7e0fe00640 1 -- 172.21.15.28:0/501886115 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e080315a0 con 0x7f7e101561a0 2024-02-21T23:06:52.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.952+0000 7f7e0fe00640 1 -- 172.21.15.28:0/501886115 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e08037070 con 0x7f7e101561a0 2024-02-21T23:06:52.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.952+0000 7f7e0fe00640 1 -- 172.21.15.28:0/501886115 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e08031960 con 0x7f7e101561a0 2024-02-21T23:06:52.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.953+0000 7f7e18600640 1 -- 172.21.15.28:0/501886115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e101561a0 msgr2=0x7f7e1006d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:52.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.953+0000 7f7e18600640 1 --2- 172.21.15.28:0/501886115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e101561a0 0x7f7e1006d660 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f7e1006f5d0 tx=0x7f7e0802b840 comp rx=0 tx=0).stop 2024-02-21T23:06:52.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.953+0000 7f7e18600640 1 -- 172.21.15.28:0/501886115 shutdown_connections 2024-02-21T23:06:52.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.953+0000 7f7e18600640 1 --2- 172.21.15.28:0/501886115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e101561a0 0x7f7e1006d660 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:52.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.953+0000 7f7e18600640 1 --2- 172.21.15.28:0/501886115 >> v2:172.21.15.81:3300/0 conn(0x7f7e100c22e0 0x7f7e1006d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:52.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.953+0000 7f7e18600640 1 -- 172.21.15.28:0/501886115 >> 172.21.15.28:0/501886115 conn(0x7f7e100a85a0 msgr2=0x7f7e100a8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:52.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.953+0000 7f7e18600640 1 -- 172.21.15.28:0/501886115 shutdown_connections 2024-02-21T23:06:52.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.953+0000 7f7e18600640 1 -- 172.21.15.28:0/501886115 wait complete. 2024-02-21T23:06:52.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.955+0000 7f7e18600640 1 Processor -- start 2024-02-21T23:06:52.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.955+0000 7f7e18600640 1 -- start start 2024-02-21T23:06:52.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.956+0000 7f7e18600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e100c22e0 0x7f7e100a9690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:52.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.956+0000 7f7e18600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e100c5a70 0x7f7e100bfe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:52.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.956+0000 7f7e18600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7e101561a0 0x7f7e101eb590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:52.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.956+0000 7f7e18600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1006dba0 con 0x7f7e100c22e0 2024-02-21T23:06:52.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.956+0000 7f7e18600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e10072210 con 0x7f7e100c5a70 2024-02-21T23:06:52.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.956+0000 7f7e18600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e100a6160 con 0x7f7e101561a0 2024-02-21T23:06:52.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.956+0000 7f7e15c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e100c22e0 0x7f7e100a9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:52.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.956+0000 7f7e15c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e100c22e0 0x7f7e100a9690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36386/0 (socket says 172.21.15.28:36386) 2024-02-21T23:06:52.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.956+0000 7f7e15c00640 1 -- 172.21.15.28:0/3976628379 learned_addr learned my addr 172.21.15.28:0/3976628379 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:52.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.957+0000 7f7e15200640 1 --2- 172.21.15.28:0/3976628379 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e100c5a70 0x7f7e100bfe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:52.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.957+0000 7f7e16600640 1 --2- 172.21.15.28:0/3976628379 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7e101561a0 0x7f7e101eb590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:52.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.957+0000 7f7e15c00640 1 -- 172.21.15.28:0/3976628379 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7e101561a0 msgr2=0x7f7e101eb590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:52.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.957+0000 7f7e15c00640 1 --2- 172.21.15.28:0/3976628379 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7e101561a0 0x7f7e101eb590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:52.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.957+0000 7f7e15c00640 1 -- 172.21.15.28:0/3976628379 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e100c5a70 msgr2=0x7f7e100bfe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:52.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.957+0000 7f7e15c00640 1 --2- 172.21.15.28:0/3976628379 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e100c5a70 0x7f7e100bfe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:52.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.957+0000 7f7e15c00640 1 -- 172.21.15.28:0/3976628379 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e100a9bd0 con 0x7f7e100c22e0 2024-02-21T23:06:52.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.957+0000 7f7e16600640 1 --2- 172.21.15.28:0/3976628379 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7e101561a0 0x7f7e101eb590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:06:52.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.957+0000 7f7e15200640 1 --2- 172.21.15.28:0/3976628379 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e100c5a70 0x7f7e100bfe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:06:52.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.957+0000 7f7e15c00640 1 --2- 172.21.15.28:0/3976628379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e100c22e0 0x7f7e100a9690 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f7e0000c460 tx=0x7f7e0000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:52.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.958+0000 7f7e0d600640 1 -- 172.21.15.28:0/3976628379 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e00017070 con 0x7f7e100c22e0 2024-02-21T23:06:52.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.958+0000 7f7e0d600640 1 -- 172.21.15.28:0/3976628379 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e00005ce0 con 0x7f7e100c22e0 2024-02-21T23:06:52.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.958+0000 7f7e18600640 1 -- 172.21.15.28:0/3976628379 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e1014d350 con 0x7f7e100c22e0 2024-02-21T23:06:52.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.958+0000 7f7e0d600640 1 -- 172.21.15.28:0/3976628379 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e00007290 con 0x7f7e100c22e0 2024-02-21T23:06:52.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.958+0000 7f7e18600640 1 -- 172.21.15.28:0/3976628379 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e1014d5f0 con 0x7f7e100c22e0 2024-02-21T23:06:52.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.959+0000 7f7e18600640 1 -- 172.21.15.28:0/3976628379 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e10067f30 con 0x7f7e100c22e0 2024-02-21T23:06:52.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.960+0000 7f7e0d600640 1 -- 172.21.15.28:0/3976628379 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7e0001a070 con 0x7f7e100c22e0 2024-02-21T23:06:52.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.961+0000 7f7e0d600640 1 --2- 172.21.15.28:0/3976628379 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7df006b500 0x7f7df006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:52.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.961+0000 7f7e0d600640 1 -- 172.21.15.28:0/3976628379 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5619+0+0 (secure 0 0 0) 0x7f7e000894d0 con 0x7f7e100c22e0 2024-02-21T23:06:52.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.962+0000 7f7e15200640 1 --2- 172.21.15.28:0/3976628379 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7df006b500 0x7f7df006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:52.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.963+0000 7f7e15200640 1 --2- 172.21.15.28:0/3976628379 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7df006b500 0x7f7df006d9c0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f7e100c06b0 tx=0x7f7dfc0039d0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:52.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:52.966+0000 7f7e0d600640 1 -- 172.21.15.28:0/3976628379 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7e00058390 con 0x7f7e100c22e0 2024-02-21T23:06:53.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.165+0000 7f7e18600640 1 -- 172.21.15.28:0/3976628379 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f7e100c1460 con 0x7f7e100c22e0 2024-02-21T23:06:53.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.713+0000 7f7e0d600640 1 -- 172.21.15.28:0/3976628379 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v96) v1 ==== 170+0+0 (secure 0 0 0) 0x7f7e00057c90 con 0x7f7e100c22e0 2024-02-21T23:06:53.702 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' created 2024-02-21T23:06:53.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.717+0000 7f7e18600640 1 -- 172.21.15.28:0/3976628379 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7df006b500 msgr2=0x7f7df006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:53.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.717+0000 7f7e18600640 1 --2- 172.21.15.28:0/3976628379 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7df006b500 0x7f7df006d9c0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f7e100c06b0 tx=0x7f7dfc0039d0 comp rx=0 tx=0).stop 2024-02-21T23:06:53.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.717+0000 7f7e18600640 1 -- 172.21.15.28:0/3976628379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e100c22e0 msgr2=0x7f7e100a9690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:53.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.717+0000 7f7e18600640 1 --2- 172.21.15.28:0/3976628379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e100c22e0 0x7f7e100a9690 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f7e0000c460 tx=0x7f7e0000c920 comp rx=0 tx=0).stop 2024-02-21T23:06:53.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.718+0000 7f7e18600640 1 -- 172.21.15.28:0/3976628379 shutdown_connections 2024-02-21T23:06:53.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.718+0000 7f7e18600640 1 --2- 172.21.15.28:0/3976628379 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7df006b500 0x7f7df006d9c0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:53.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.718+0000 7f7e18600640 1 --2- 172.21.15.28:0/3976628379 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7e101561a0 0x7f7e101eb590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:53.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.718+0000 7f7e18600640 1 --2- 172.21.15.28:0/3976628379 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e100c5a70 0x7f7e100bfe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:53.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.718+0000 7f7e18600640 1 --2- 172.21.15.28:0/3976628379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e100c22e0 0x7f7e100a9690 secure :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f7e0000c460 tx=0x7f7e0000c920 comp rx=0 tx=0).stop 2024-02-21T23:06:53.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.718+0000 7f7e18600640 1 -- 172.21.15.28:0/3976628379 >> 172.21.15.28:0/3976628379 conn(0x7f7e100a85a0 msgr2=0x7f7e1006d100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:53.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.719+0000 7f7e18600640 1 -- 172.21.15.28:0/3976628379 shutdown_connections 2024-02-21T23:06:53.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.719+0000 7f7e18600640 1 -- 172.21.15.28:0/3976628379 wait complete. 2024-02-21T23:06:53.742 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-02-21T23:06:53.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.940+0000 7f36fe800640 1 Processor -- start 2024-02-21T23:06:53.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.940+0000 7f36fe800640 1 -- start start 2024-02-21T23:06:53.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.941+0000 7f36fe800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f36f80c5a70 0x7f36f8155a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:53.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.941+0000 7f36fe800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f36f80c2b30 0x7f36f806d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:53.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.941+0000 7f36fe800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36f8071970 con 0x7f36f80c22e0 2024-02-21T23:06:53.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.941+0000 7f36fe800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36f8072a10 con 0x7f36f80c2b30 2024-02-21T23:06:53.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.941+0000 7f36fe800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36f8072580 con 0x7f36f80c5a70 2024-02-21T23:06:53.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.941+0000 7f36f7e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f36f80c2b30 0x7f36f806d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:53.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.942+0000 7f36f7e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f36f80c2b30 0x7f36f806d830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36398/0 (socket says 172.21.15.28:36398) 2024-02-21T23:06:53.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.942+0000 7f36f7e00640 1 -- 172.21.15.28:0/3459636101 learned_addr learned my addr 172.21.15.28:0/3459636101 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:53.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.942+0000 7f36f7400640 1 --2- 172.21.15.28:0/3459636101 >> v2:172.21.15.81:3300/0 conn(0x7f36f80c5a70 0x7f36f8155a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:53.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.942+0000 7f36f7e00640 1 -- 172.21.15.28:0/3459636101 >> v2:172.21.15.81:3300/0 conn(0x7f36f80c5a70 msgr2=0x7f36f8155a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:53.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.942+0000 7f36f7e00640 1 --2- 172.21.15.28:0/3459636101 >> v2:172.21.15.81:3300/0 conn(0x7f36f80c5a70 0x7f36f8155a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:53.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.942+0000 7f36f7e00640 1 -- 172.21.15.28:0/3459636101 >> v1:172.21.15.81:6789/0 conn(0x7f36f80c22e0 legacy=0x7f36f806d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:53.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.942+0000 7f36f7e00640 1 -- 172.21.15.28:0/3459636101 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36f8071c50 con 0x7f36f80c2b30 2024-02-21T23:06:53.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.943+0000 7f36f7e00640 1 --2- 172.21.15.28:0/3459636101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36f80c2b30 0x7f36f806d830 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f36f80c7810 tx=0x7f36f002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88ea4d20f9348686 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:53.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.943+0000 7f36f6a00640 1 -- 172.21.15.28:0/3459636101 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36f002c9f0 con 0x7f36f80c2b30 2024-02-21T23:06:53.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.943+0000 7f36f6a00640 1 -- 172.21.15.28:0/3459636101 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36f002cb90 con 0x7f36f80c2b30 2024-02-21T23:06:53.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.943+0000 7f36f6a00640 1 -- 172.21.15.28:0/3459636101 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36f00343f0 con 0x7f36f80c2b30 2024-02-21T23:06:53.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.944+0000 7f36fe800640 1 -- 172.21.15.28:0/3459636101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36f80c2b30 msgr2=0x7f36f806d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:53.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.944+0000 7f36fe800640 1 --2- 172.21.15.28:0/3459636101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36f80c2b30 0x7f36f806d830 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f36f80c7810 tx=0x7f36f002be50 comp rx=0 tx=0).stop 2024-02-21T23:06:53.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.944+0000 7f36fe800640 1 -- 172.21.15.28:0/3459636101 shutdown_connections 2024-02-21T23:06:53.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.944+0000 7f36fe800640 1 --2- 172.21.15.28:0/3459636101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36f80c2b30 0x7f36f806d830 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:53.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.944+0000 7f36fe800640 1 --2- 172.21.15.28:0/3459636101 >> v2:172.21.15.81:3300/0 conn(0x7f36f80c5a70 0x7f36f8155a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:53.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.944+0000 7f36fe800640 1 -- 172.21.15.28:0/3459636101 >> 172.21.15.28:0/3459636101 conn(0x7f36f80a85a0 msgr2=0x7f36f80a8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:53.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.944+0000 7f36fe800640 1 -- 172.21.15.28:0/3459636101 shutdown_connections 2024-02-21T23:06:53.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.944+0000 7f36fe800640 1 -- 172.21.15.28:0/3459636101 wait complete. 2024-02-21T23:06:53.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.946+0000 7f36fe800640 1 Processor -- start 2024-02-21T23:06:53.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.946+0000 7f36fe800640 1 -- start start 2024-02-21T23:06:53.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.947+0000 7f36fe800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36f80c22e0 0x7f36f80ab540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:53.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.947+0000 7f36fe800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f36f80c2b30 0x7f36f80bc300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:53.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.947+0000 7f36fe800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f36f80c5a70 0x7f36f813e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:53.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.947+0000 7f36fe800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36f80a6160 con 0x7f36f80c22e0 2024-02-21T23:06:53.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.947+0000 7f36fe800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36f8072210 con 0x7f36f80c2b30 2024-02-21T23:06:53.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.947+0000 7f36fe800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36f80a62e0 con 0x7f36f80c5a70 2024-02-21T23:06:53.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.947+0000 7f36f7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36f80c22e0 0x7f36f80ab540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:53.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.947+0000 7f36f7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36f80c22e0 0x7f36f80ab540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36412/0 (socket says 172.21.15.28:36412) 2024-02-21T23:06:53.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.947+0000 7f36f7400640 1 -- 172.21.15.28:0/2614819684 learned_addr learned my addr 172.21.15.28:0/2614819684 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:53.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.948+0000 7f36efe00640 1 --2- 172.21.15.28:0/2614819684 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f36f80c2b30 0x7f36f80bc300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:53.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.948+0000 7f36f7400640 1 -- 172.21.15.28:0/2614819684 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f36f80c5a70 msgr2=0x7f36f813e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:53.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.948+0000 7f36f7e00640 1 --2- 172.21.15.28:0/2614819684 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f36f80c5a70 0x7f36f813e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:53.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.948+0000 7f36f7400640 1 --2- 172.21.15.28:0/2614819684 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f36f80c5a70 0x7f36f813e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:53.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.948+0000 7f36f7400640 1 -- 172.21.15.28:0/2614819684 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f36f80c2b30 msgr2=0x7f36f80bc300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:53.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.948+0000 7f36f7400640 1 --2- 172.21.15.28:0/2614819684 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f36f80c2b30 0x7f36f80bc300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:53.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.948+0000 7f36f7400640 1 -- 172.21.15.28:0/2614819684 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36f80bb8b0 con 0x7f36f80c22e0 2024-02-21T23:06:53.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.948+0000 7f36f7400640 1 --2- 172.21.15.28:0/2614819684 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36f80c22e0 0x7f36f80ab540 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f36e800d620 tx=0x7f36e800dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:53.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.948+0000 7f36ef400640 1 -- 172.21.15.28:0/2614819684 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36e800b620 con 0x7f36f80c22e0 2024-02-21T23:06:53.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.948+0000 7f36ef400640 1 -- 172.21.15.28:0/2614819684 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36e8019070 con 0x7f36f80c22e0 2024-02-21T23:06:53.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.948+0000 7f36ef400640 1 -- 172.21.15.28:0/2614819684 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36e800ba00 con 0x7f36f80c22e0 2024-02-21T23:06:53.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.948+0000 7f36fe800640 1 -- 172.21.15.28:0/2614819684 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36f81ebeb0 con 0x7f36f80c22e0 2024-02-21T23:06:53.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.949+0000 7f36fe800640 1 -- 172.21.15.28:0/2614819684 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36f80aba80 con 0x7f36f80c22e0 2024-02-21T23:06:53.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.950+0000 7f36fe800640 1 -- 172.21.15.28:0/2614819684 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36f8067f30 con 0x7f36f80c22e0 2024-02-21T23:06:53.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.951+0000 7f36ef400640 1 -- 172.21.15.28:0/2614819684 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f36e800bba0 con 0x7f36f80c22e0 2024-02-21T23:06:53.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.952+0000 7f36ef400640 1 --2- 172.21.15.28:0/2614819684 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f36d406b500 0x7f36d406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:53.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.952+0000 7f36ef400640 1 -- 172.21.15.28:0/2614819684 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6001+0+0 (secure 0 0 0) 0x7f36e808a6e0 con 0x7f36f80c22e0 2024-02-21T23:06:53.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.952+0000 7f36efe00640 1 --2- 172.21.15.28:0/2614819684 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f36d406b500 0x7f36d406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:53.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.954+0000 7f36efe00640 1 --2- 172.21.15.28:0/2614819684 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f36d406b500 0x7f36d406d9c0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f36e0007fd0 tx=0x7f36e0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:53.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:53.957+0000 7f36ef400640 1 -- 172.21.15.28:0/2614819684 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f36e8059420 con 0x7f36f80c22e0 2024-02-21T23:06:54.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:54.199+0000 7f36fe800640 1 -- 172.21.15.28:0/2614819684 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f36f80a5b10 con 0x7f36f80c22e0 2024-02-21T23:06:54.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:54.780+0000 7f36ef400640 1 -- 172.21.15.28:0/2614819684 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 15 and data pool 16 v84) v1 ==== 174+0+0 (secure 0 0 0) 0x7f36e8058d20 con 0x7f36f80c22e0 2024-02-21T23:06:54.771 INFO:teuthology.orchestra.run.smithi028.stderr:new fs with metadata pool 15 and data pool 16 2024-02-21T23:06:54.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:54.787+0000 7f36fe800640 1 -- 172.21.15.28:0/2614819684 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f36d406b500 msgr2=0x7f36d406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:54.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:54.787+0000 7f36fe800640 1 --2- 172.21.15.28:0/2614819684 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f36d406b500 0x7f36d406d9c0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f36e0007fd0 tx=0x7f36e0009040 comp rx=0 tx=0).stop 2024-02-21T23:06:54.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:54.787+0000 7f36fe800640 1 -- 172.21.15.28:0/2614819684 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36f80c22e0 msgr2=0x7f36f80ab540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:54.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:54.787+0000 7f36fe800640 1 --2- 172.21.15.28:0/2614819684 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36f80c22e0 0x7f36f80ab540 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f36e800d620 tx=0x7f36e800dae0 comp rx=0 tx=0).stop 2024-02-21T23:06:54.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:54.788+0000 7f36fe800640 1 -- 172.21.15.28:0/2614819684 shutdown_connections 2024-02-21T23:06:54.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:54.788+0000 7f36fe800640 1 --2- 172.21.15.28:0/2614819684 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f36d406b500 0x7f36d406d9c0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:54.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:54.788+0000 7f36fe800640 1 --2- 172.21.15.28:0/2614819684 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f36f80c5a70 0x7f36f813e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:54.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:54.788+0000 7f36fe800640 1 --2- 172.21.15.28:0/2614819684 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f36f80c2b30 0x7f36f80bc300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:54.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:54.788+0000 7f36fe800640 1 --2- 172.21.15.28:0/2614819684 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36f80c22e0 0x7f36f80ab540 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:54.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:54.788+0000 7f36fe800640 1 -- 172.21.15.28:0/2614819684 >> 172.21.15.28:0/2614819684 conn(0x7f36f80a85a0 msgr2=0x7f36f80c43e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:54.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:54.792+0000 7f36fe800640 1 -- 172.21.15.28:0/2614819684 shutdown_connections 2024-02-21T23:06:54.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:54.792+0000 7f36fe800640 1 -- 172.21.15.28:0/2614819684 wait complete. 2024-02-21T23:06:54.812 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:06:54.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.004+0000 7fa238800640 1 Processor -- start 2024-02-21T23:06:54.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.005+0000 7fa238800640 1 -- start start 2024-02-21T23:06:54.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.006+0000 7fa238800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fa230152590 0x7fa230072f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:54.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.006+0000 7fa238800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa2300c5c00 0x7fa230158f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:54.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.006+0000 7fa238800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2301550d0 con 0x7fa2300c5770 2024-02-21T23:06:54.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.006+0000 7fa238800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa230155390 con 0x7fa2300c5c00 2024-02-21T23:06:54.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.006+0000 7fa238800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa230155670 con 0x7fa230152590 2024-02-21T23:06:54.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.006+0000 7fa236800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa2300c5c00 0x7fa230158f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:54.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.007+0000 7fa236800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa2300c5c00 0x7fa230158f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47520/0 (socket says 172.21.15.28:47520) 2024-02-21T23:06:54.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.007+0000 7fa236800640 1 -- 172.21.15.28:0/1260413620 learned_addr learned my addr 172.21.15.28:0/1260413620 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:54.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.007+0000 7fa235e00640 1 --2- 172.21.15.28:0/1260413620 >> v2:172.21.15.81:3300/0 conn(0x7fa230152590 0x7fa230072f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:54.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.007+0000 7fa236800640 1 -- 172.21.15.28:0/1260413620 >> v2:172.21.15.81:3300/0 conn(0x7fa230152590 msgr2=0x7fa230072f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:54.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.007+0000 7fa236800640 1 --2- 172.21.15.28:0/1260413620 >> v2:172.21.15.81:3300/0 conn(0x7fa230152590 0x7fa230072f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:54.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.007+0000 7fa236800640 1 -- 172.21.15.28:0/1260413620 >> v1:172.21.15.81:6789/0 conn(0x7fa2300c5770 legacy=0x7fa230158870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:54.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.007+0000 7fa236800640 1 -- 172.21.15.28:0/1260413620 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa230155950 con 0x7fa2300c5c00 2024-02-21T23:06:54.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.008+0000 7fa236800640 1 --2- 172.21.15.28:0/1260413620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2300c5c00 0x7fa230158f80 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fa23006cf40 tx=0x7fa22c02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0ba397b81422661 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:54.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.009+0000 7fa234a00640 1 -- 172.21.15.28:0/1260413620 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa22c02c9f0 con 0x7fa2300c5c00 2024-02-21T23:06:54.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.009+0000 7fa234a00640 1 -- 172.21.15.28:0/1260413620 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa22c02cb90 con 0x7fa2300c5c00 2024-02-21T23:06:54.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.009+0000 7fa234a00640 1 -- 172.21.15.28:0/1260413620 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa22c0343f0 con 0x7fa2300c5c00 2024-02-21T23:06:54.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.009+0000 7fa238800640 1 -- 172.21.15.28:0/1260413620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2300c5c00 msgr2=0x7fa230158f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:54.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.009+0000 7fa238800640 1 --2- 172.21.15.28:0/1260413620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2300c5c00 0x7fa230158f80 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fa23006cf40 tx=0x7fa22c02be50 comp rx=0 tx=0).stop 2024-02-21T23:06:54.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.009+0000 7fa238800640 1 -- 172.21.15.28:0/1260413620 shutdown_connections 2024-02-21T23:06:54.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.009+0000 7fa238800640 1 --2- 172.21.15.28:0/1260413620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2300c5c00 0x7fa230158f80 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:54.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.009+0000 7fa238800640 1 --2- 172.21.15.28:0/1260413620 >> v2:172.21.15.81:3300/0 conn(0x7fa230152590 0x7fa230072f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:54.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.009+0000 7fa238800640 1 -- 172.21.15.28:0/1260413620 >> 172.21.15.28:0/1260413620 conn(0x7fa23006c5b0 msgr2=0x7fa2300c7870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:54.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.009+0000 7fa238800640 1 -- 172.21.15.28:0/1260413620 shutdown_connections 2024-02-21T23:06:54.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.009+0000 7fa238800640 1 -- 172.21.15.28:0/1260413620 wait complete. 2024-02-21T23:06:54.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.011+0000 7fa238800640 1 Processor -- start 2024-02-21T23:06:54.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.011+0000 7fa238800640 1 -- start start 2024-02-21T23:06:55.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.012+0000 7fa238800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa2300c5770 0x7fa230150aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:55.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.012+0000 7fa238800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2300c5c00 0x7fa2300a9130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:55.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.012+0000 7fa238800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa230152590 0x7fa2300a9670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:55.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.012+0000 7fa238800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa230073460 con 0x7fa2300c5c00 2024-02-21T23:06:55.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.012+0000 7fa238800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2301594c0 con 0x7fa2300c5770 2024-02-21T23:06:55.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.012+0000 7fa238800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2300729e0 con 0x7fa230152590 2024-02-21T23:06:55.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.013+0000 7fa235400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2300c5c00 0x7fa2300a9130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:55.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.013+0000 7fa235400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2300c5c00 0x7fa2300a9130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47532/0 (socket says 172.21.15.28:47532) 2024-02-21T23:06:55.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.013+0000 7fa235400640 1 -- 172.21.15.28:0/3470846798 learned_addr learned my addr 172.21.15.28:0/3470846798 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:55.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.013+0000 7fa235e00640 1 --2- 172.21.15.28:0/3470846798 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa2300c5770 0x7fa230150aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:55.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.013+0000 7fa236800640 1 --2- 172.21.15.28:0/3470846798 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa230152590 0x7fa2300a9670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:55.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.013+0000 7fa235400640 1 -- 172.21.15.28:0/3470846798 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa230152590 msgr2=0x7fa2300a9670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:55.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.013+0000 7fa235400640 1 --2- 172.21.15.28:0/3470846798 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa230152590 0x7fa2300a9670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:55.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.013+0000 7fa235400640 1 -- 172.21.15.28:0/3470846798 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa2300c5770 msgr2=0x7fa230150aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:55.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.013+0000 7fa235400640 1 --2- 172.21.15.28:0/3470846798 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa2300c5770 0x7fa230150aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:55.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.013+0000 7fa235400640 1 -- 172.21.15.28:0/3470846798 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2301eadc0 con 0x7fa2300c5c00 2024-02-21T23:06:55.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.013+0000 7fa235e00640 1 --2- 172.21.15.28:0/3470846798 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa2300c5770 0x7fa230150aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:06:55.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.014+0000 7fa235400640 1 --2- 172.21.15.28:0/3470846798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2300c5c00 0x7fa2300a9130 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fa22000b410 tx=0x7fa22000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:55.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.014+0000 7fa22a000640 1 -- 172.21.15.28:0/3470846798 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa220015020 con 0x7fa2300c5c00 2024-02-21T23:06:55.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.014+0000 7fa22a000640 1 -- 172.21.15.28:0/3470846798 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa220011830 con 0x7fa2300c5c00 2024-02-21T23:06:55.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.014+0000 7fa22a000640 1 -- 172.21.15.28:0/3470846798 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa220011c20 con 0x7fa2300c5c00 2024-02-21T23:06:55.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.014+0000 7fa238800640 1 -- 172.21.15.28:0/3470846798 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2301eb0a0 con 0x7fa2300c5c00 2024-02-21T23:06:55.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.014+0000 7fa238800640 1 -- 172.21.15.28:0/3470846798 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2301eb660 con 0x7fa2300c5c00 2024-02-21T23:06:55.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.015+0000 7fa238800640 1 -- 172.21.15.28:0/3470846798 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa230068640 con 0x7fa2300c5c00 2024-02-21T23:06:55.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.016+0000 7fa22a000640 1 -- 172.21.15.28:0/3470846798 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa220018020 con 0x7fa2300c5c00 2024-02-21T23:06:55.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.017+0000 7fa22a000640 1 --2- 172.21.15.28:0/3470846798 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa21406b500 0x7fa21406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:55.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.017+0000 7fa22a000640 1 -- 172.21.15.28:0/3470846798 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa220089cb0 con 0x7fa2300c5c00 2024-02-21T23:06:55.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.018+0000 7fa235e00640 1 --2- 172.21.15.28:0/3470846798 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa21406b500 0x7fa21406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:55.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.019+0000 7fa235e00640 1 --2- 172.21.15.28:0/3470846798 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa21406b500 0x7fa21406d9c0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fa224006fd0 tx=0x7fa224008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:55.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.022+0000 7fa22a000640 1 -- 172.21.15.28:0/3470846798 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fa22001f070 con 0x7fa2300c5c00 2024-02-21T23:06:55.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.216+0000 7fa238800640 1 -- 172.21.15.28:0/3470846798 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa2300c6b00 con 0x7fa2300c5c00 2024-02-21T23:06:55.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.218+0000 7fa22a000640 1 -- 172.21.15.28:0/3470846798 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) v1 ==== 74+0+16826 (secure 0 0 0) 0x7fa220018310 con 0x7fa2300c5c00 2024-02-21T23:06:55.207 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:06:55.207 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":97,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:06:54.200752+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:06:52.151181+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-02-21T23:06:53.221776+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:06:55.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.221+0000 7fa238800640 1 -- 172.21.15.28:0/3470846798 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa21406b500 msgr2=0x7fa21406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:55.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.221+0000 7fa238800640 1 --2- 172.21.15.28:0/3470846798 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa21406b500 0x7fa21406d9c0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fa224006fd0 tx=0x7fa224008040 comp rx=0 tx=0).stop 2024-02-21T23:06:55.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.221+0000 7fa238800640 1 -- 172.21.15.28:0/3470846798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2300c5c00 msgr2=0x7fa2300a9130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:55.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.221+0000 7fa238800640 1 --2- 172.21.15.28:0/3470846798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2300c5c00 0x7fa2300a9130 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fa22000b410 tx=0x7fa22000b8d0 comp rx=0 tx=0).stop 2024-02-21T23:06:55.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.221+0000 7fa238800640 1 -- 172.21.15.28:0/3470846798 shutdown_connections 2024-02-21T23:06:55.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.221+0000 7fa238800640 1 --2- 172.21.15.28:0/3470846798 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa21406b500 0x7fa21406d9c0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:55.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.221+0000 7fa238800640 1 --2- 172.21.15.28:0/3470846798 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa230152590 0x7fa2300a9670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:55.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.221+0000 7fa238800640 1 --2- 172.21.15.28:0/3470846798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2300c5c00 0x7fa2300a9130 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:55.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.221+0000 7fa238800640 1 --2- 172.21.15.28:0/3470846798 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa2300c5770 0x7fa230150aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:55.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.221+0000 7fa238800640 1 -- 172.21.15.28:0/3470846798 >> 172.21.15.28:0/3470846798 conn(0x7fa23006c5b0 msgr2=0x7fa2300c42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:55.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.221+0000 7fa238800640 1 -- 172.21.15.28:0/3470846798 shutdown_connections 2024-02-21T23:06:55.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.221+0000 7fa238800640 1 -- 172.21.15.28:0/3470846798 wait complete. 2024-02-21T23:06:55.227 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:06:55.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.433+0000 7f809b200640 1 Processor -- start 2024-02-21T23:06:55.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.433+0000 7f809b200640 1 -- start start 2024-02-21T23:06:55.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.434+0000 7f809b200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f809c06dbf0 0x7f809c158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:55.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.434+0000 7f809b200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f809c155960 0x7f809c159000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:55.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.434+0000 7f809b200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f809c0c74c0 con 0x7f809c0aff90 2024-02-21T23:06:55.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.434+0000 7f809b200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f809c0c7270 con 0x7f809c155960 2024-02-21T23:06:55.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.434+0000 7f809b200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f809c158630 con 0x7f809c06dbf0 2024-02-21T23:06:55.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.434+0000 7f8099e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f809c0aff90 0x7f809c0c51e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34596/0 (socket says 172.21.15.28:34596) 2024-02-21T23:06:55.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.434+0000 7f8099e00640 1 -- 172.21.15.28:0/4256652288 learned_addr learned my addr 172.21.15.28:0/4256652288 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:06:55.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.434+0000 7f809a800640 1 --2- 172.21.15.28:0/4256652288 >> v2:172.21.15.28:3300/0 conn(0x7f809c155960 0x7f809c159000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:55.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.435+0000 7f8099400640 1 --2- 172.21.15.28:0/4256652288 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f809c06dbf0 0x7f809c158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:55.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.435+0000 7f809a800640 1 -- 172.21.15.28:0/4256652288 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f809c06dbf0 msgr2=0x7f809c158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:55.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.435+0000 7f809a800640 1 --2- 172.21.15.28:0/4256652288 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f809c06dbf0 0x7f809c158ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:55.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.435+0000 7f809a800640 1 -- 172.21.15.28:0/4256652288 >> v1:172.21.15.28:6789/0 conn(0x7f809c0aff90 legacy=0x7f809c0c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:55.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.435+0000 7f809a800640 1 -- 172.21.15.28:0/4256652288 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f809c0c77a0 con 0x7f809c155960 2024-02-21T23:06:55.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.435+0000 7f8098a00640 1 -- 172.21.15.28:0/4256652288 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4098211293 0 0) 0x7f809c0c74c0 con 0x7f809c0aff90 2024-02-21T23:06:55.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.435+0000 7f8099400640 1 --2- 172.21.15.28:0/4256652288 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f809c06dbf0 0x7f809c158ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:06:55.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.437+0000 7f809a800640 1 --2- 172.21.15.28:0/4256652288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f809c155960 0x7f809c159000 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f8094009d90 tx=0x7f809402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f1e3f3339176a06 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:55.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.437+0000 7f8098a00640 1 -- 172.21.15.28:0/4256652288 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8094039070 con 0x7f809c155960 2024-02-21T23:06:55.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.437+0000 7f8098a00640 1 -- 172.21.15.28:0/4256652288 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f809402cc50 con 0x7f809c155960 2024-02-21T23:06:55.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.437+0000 7f8098a00640 1 -- 172.21.15.28:0/4256652288 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8094034440 con 0x7f809c155960 2024-02-21T23:06:55.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.437+0000 7f809b200640 1 -- 172.21.15.28:0/4256652288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f809c155960 msgr2=0x7f809c159000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:55.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.437+0000 7f809b200640 1 --2- 172.21.15.28:0/4256652288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f809c155960 0x7f809c159000 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f8094009d90 tx=0x7f809402bea0 comp rx=0 tx=0).stop 2024-02-21T23:06:55.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.438+0000 7f809b200640 1 -- 172.21.15.28:0/4256652288 shutdown_connections 2024-02-21T23:06:55.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.438+0000 7f809b200640 1 --2- 172.21.15.28:0/4256652288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f809c155960 0x7f809c159000 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:55.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.438+0000 7f809b200640 1 --2- 172.21.15.28:0/4256652288 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f809c06dbf0 0x7f809c158ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:55.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.438+0000 7f809b200640 1 -- 172.21.15.28:0/4256652288 >> 172.21.15.28:0/4256652288 conn(0x7f809c0ac370 msgr2=0x7f809c0ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:55.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.438+0000 7f809b200640 1 -- 172.21.15.28:0/4256652288 shutdown_connections 2024-02-21T23:06:55.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.438+0000 7f809b200640 1 -- 172.21.15.28:0/4256652288 wait complete. 2024-02-21T23:06:55.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.439+0000 7f809b200640 1 Processor -- start 2024-02-21T23:06:55.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.439+0000 7f809b200640 1 -- start start 2024-02-21T23:06:55.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.440+0000 7f809b200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f809c06dbf0 0x7f809c1ee1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:55.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.440+0000 7f809b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f809c0aff90 0x7f809c0ab4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:55.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.440+0000 7f809b200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f809c155960 0x7f809c14e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:55.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.440+0000 7f809b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f809c0c58f0 con 0x7f809c0aff90 2024-02-21T23:06:55.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.440+0000 7f809b200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f809c0be0c0 con 0x7f809c155960 2024-02-21T23:06:55.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.440+0000 7f809b200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f809c0c5a70 con 0x7f809c06dbf0 2024-02-21T23:06:55.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.440+0000 7f8099400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f809c0aff90 0x7f809c0ab4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:55.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.441+0000 7f8099400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f809c0aff90 0x7f809c0ab4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47548/0 (socket says 172.21.15.28:47548) 2024-02-21T23:06:55.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.441+0000 7f8099400640 1 -- 172.21.15.28:0/1077226747 learned_addr learned my addr 172.21.15.28:0/1077226747 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:55.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.441+0000 7f8099e00640 1 --2- 172.21.15.28:0/1077226747 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f809c06dbf0 0x7f809c1ee1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:55.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.441+0000 7f809a800640 1 --2- 172.21.15.28:0/1077226747 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f809c155960 0x7f809c14e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:55.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.441+0000 7f8099400640 1 -- 172.21.15.28:0/1077226747 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f809c06dbf0 msgr2=0x7f809c1ee1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:55.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.441+0000 7f8099400640 1 --2- 172.21.15.28:0/1077226747 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f809c06dbf0 0x7f809c1ee1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:55.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.441+0000 7f8099400640 1 -- 172.21.15.28:0/1077226747 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f809c155960 msgr2=0x7f809c14e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:55.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.441+0000 7f8099400640 1 --2- 172.21.15.28:0/1077226747 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f809c155960 0x7f809c14e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:55.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.441+0000 7f8099400640 1 -- 172.21.15.28:0/1077226747 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f809c1ee840 con 0x7f809c0aff90 2024-02-21T23:06:55.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.441+0000 7f8099400640 1 --2- 172.21.15.28:0/1077226747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f809c0aff90 0x7f809c0ab4a0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f808800e810 tx=0x7f808800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:55.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.442+0000 7f8092000640 1 -- 172.21.15.28:0/1077226747 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f808800f040 con 0x7f809c0aff90 2024-02-21T23:06:55.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.442+0000 7f8092000640 1 -- 172.21.15.28:0/1077226747 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8088004030 con 0x7f809c0aff90 2024-02-21T23:06:55.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.442+0000 7f8092000640 1 -- 172.21.15.28:0/1077226747 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8088004400 con 0x7f809c0aff90 2024-02-21T23:06:55.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.442+0000 7f809b200640 1 -- 172.21.15.28:0/1077226747 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f809c1eeb20 con 0x7f809c0aff90 2024-02-21T23:06:55.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.442+0000 7f809b200640 1 -- 172.21.15.28:0/1077226747 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f809c152eb0 con 0x7f809c0aff90 2024-02-21T23:06:55.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.443+0000 7f809b200640 1 -- 172.21.15.28:0/1077226747 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f809c04f290 con 0x7f809c0aff90 2024-02-21T23:06:55.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.444+0000 7f8092000640 1 -- 172.21.15.28:0/1077226747 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f808800c6d0 con 0x7f809c0aff90 2024-02-21T23:06:55.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.445+0000 7f8092000640 1 --2- 172.21.15.28:0/1077226747 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f807c06b4d0 0x7f807c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:55.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.445+0000 7f8092000640 1 -- 172.21.15.28:0/1077226747 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f808808a180 con 0x7f809c0aff90 2024-02-21T23:06:55.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.445+0000 7f8099e00640 1 --2- 172.21.15.28:0/1077226747 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f807c06b4d0 0x7f807c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:55.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.447+0000 7f8099e00640 1 --2- 172.21.15.28:0/1077226747 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f807c06b4d0 0x7f807c06d990 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f808c000f60 tx=0x7f808c007b50 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:55.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.449+0000 7f8092000640 1 -- 172.21.15.28:0/1077226747 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f8088017020 con 0x7f809c0aff90 2024-02-21T23:06:55.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.660+0000 7f809b200640 1 -- 172.21.15.28:0/1077226747 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f809c06d9d0 con 0x7f809c0aff90 2024-02-21T23:06:55.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.662+0000 7f8092000640 1 -- 172.21.15.28:0/1077226747 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) v1 ==== 74+0+16826 (secure 0 0 0) 0x7f8088058e60 con 0x7f809c0aff90 2024-02-21T23:06:55.650 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:06:55.650 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":97,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:06:54.200752+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:06:52.151181+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-02-21T23:06:53.221776+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:06:55.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.664+0000 7f809b200640 1 -- 172.21.15.28:0/1077226747 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f807c06b4d0 msgr2=0x7f807c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:55.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.664+0000 7f809b200640 1 --2- 172.21.15.28:0/1077226747 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f807c06b4d0 0x7f807c06d990 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f808c000f60 tx=0x7f808c007b50 comp rx=0 tx=0).stop 2024-02-21T23:06:55.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.665+0000 7f809b200640 1 -- 172.21.15.28:0/1077226747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f809c0aff90 msgr2=0x7f809c0ab4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:55.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.665+0000 7f809b200640 1 --2- 172.21.15.28:0/1077226747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f809c0aff90 0x7f809c0ab4a0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f808800e810 tx=0x7f808800ecd0 comp rx=0 tx=0).stop 2024-02-21T23:06:55.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.665+0000 7f809b200640 1 -- 172.21.15.28:0/1077226747 shutdown_connections 2024-02-21T23:06:55.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.665+0000 7f809b200640 1 --2- 172.21.15.28:0/1077226747 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f807c06b4d0 0x7f807c06d990 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:55.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.665+0000 7f809b200640 1 --2- 172.21.15.28:0/1077226747 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f809c155960 0x7f809c14e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:55.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.665+0000 7f809b200640 1 --2- 172.21.15.28:0/1077226747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f809c0aff90 0x7f809c0ab4a0 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:55.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.665+0000 7f809b200640 1 --2- 172.21.15.28:0/1077226747 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f809c06dbf0 0x7f809c1ee1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:55.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.665+0000 7f809b200640 1 -- 172.21.15.28:0/1077226747 >> 172.21.15.28:0/1077226747 conn(0x7f809c0ac370 msgr2=0x7f809c1572b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:55.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.665+0000 7f809b200640 1 -- 172.21.15.28:0/1077226747 shutdown_connections 2024-02-21T23:06:55.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.665+0000 7f809b200640 1 -- 172.21.15.28:0/1077226747 wait complete. 2024-02-21T23:06:55.672 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-02-21T23:06:55.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.878+0000 7f6bbfe00640 1 Processor -- start 2024-02-21T23:06:55.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.879+0000 7f6bbfe00640 1 -- start start 2024-02-21T23:06:55.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.880+0000 7f6bbfe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6bc006dbc0 0x7f6bc006c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:55.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.880+0000 7f6bbfe00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f6bc013e520 0x7f6bc006c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:55.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.880+0000 7f6bbfe00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bc0158550 con 0x7f6bc013e0c0 2024-02-21T23:06:55.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.880+0000 7f6bbfe00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bc00c89a0 con 0x7f6bc006dbc0 2024-02-21T23:06:55.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.880+0000 7f6bbfe00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bc01587d0 con 0x7f6bc013e520 2024-02-21T23:06:55.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.880+0000 7f6bb7e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6bc006dbc0 0x7f6bc006c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:55.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.880+0000 7f6bb7e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6bc006dbc0 0x7f6bc006c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47558/0 (socket says 172.21.15.28:47558) 2024-02-21T23:06:55.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.880+0000 7f6bb7e00640 1 -- 172.21.15.28:0/3442378676 learned_addr learned my addr 172.21.15.28:0/3442378676 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:55.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.881+0000 7f6bbf400640 1 --2- 172.21.15.28:0/3442378676 >> v2:172.21.15.81:3300/0 conn(0x7f6bc013e520 0x7f6bc006c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:55.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.881+0000 7f6bb7e00640 1 -- 172.21.15.28:0/3442378676 >> v2:172.21.15.81:3300/0 conn(0x7f6bc013e520 msgr2=0x7f6bc006c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:55.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.881+0000 7f6bb7e00640 1 --2- 172.21.15.28:0/3442378676 >> v2:172.21.15.81:3300/0 conn(0x7f6bc013e520 0x7f6bc006c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:55.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.881+0000 7f6bb7e00640 1 -- 172.21.15.28:0/3442378676 >> v1:172.21.15.81:6789/0 conn(0x7f6bc013e0c0 legacy=0x7f6bc00c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:55.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.881+0000 7f6bb7e00640 1 -- 172.21.15.28:0/3442378676 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bc0158ab0 con 0x7f6bc006dbc0 2024-02-21T23:06:55.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.882+0000 7f6bb7e00640 1 --2- 172.21.15.28:0/3442378676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6bc006dbc0 0x7f6bc006c280 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f6bc00c8260 tx=0x7f6ba802b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64980a24e1254b7a server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:55.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.882+0000 7f6bbe000640 1 -- 172.21.15.28:0/3442378676 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba8038070 con 0x7f6bc006dbc0 2024-02-21T23:06:55.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.882+0000 7f6bbe000640 1 -- 172.21.15.28:0/3442378676 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ba802f050 con 0x7f6bc006dbc0 2024-02-21T23:06:55.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.882+0000 7f6bbe000640 1 -- 172.21.15.28:0/3442378676 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba8031970 con 0x7f6bc006dbc0 2024-02-21T23:06:55.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.883+0000 7f6bbfe00640 1 -- 172.21.15.28:0/3442378676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6bc006dbc0 msgr2=0x7f6bc006c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:55.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.883+0000 7f6bbfe00640 1 --2- 172.21.15.28:0/3442378676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6bc006dbc0 0x7f6bc006c280 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f6bc00c8260 tx=0x7f6ba802b7f0 comp rx=0 tx=0).stop 2024-02-21T23:06:55.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.883+0000 7f6bbfe00640 1 -- 172.21.15.28:0/3442378676 shutdown_connections 2024-02-21T23:06:55.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.883+0000 7f6bbfe00640 1 --2- 172.21.15.28:0/3442378676 >> v2:172.21.15.81:3300/0 conn(0x7f6bc013e520 0x7f6bc006c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:55.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.883+0000 7f6bbfe00640 1 --2- 172.21.15.28:0/3442378676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6bc006dbc0 0x7f6bc006c280 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:55.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.883+0000 7f6bbfe00640 1 -- 172.21.15.28:0/3442378676 >> 172.21.15.28:0/3442378676 conn(0x7f6bc0073110 msgr2=0x7f6bc00ac490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:55.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.883+0000 7f6bbfe00640 1 -- 172.21.15.28:0/3442378676 shutdown_connections 2024-02-21T23:06:55.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.883+0000 7f6bbfe00640 1 -- 172.21.15.28:0/3442378676 wait complete. 2024-02-21T23:06:55.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.885+0000 7f6bbfe00640 1 Processor -- start 2024-02-21T23:06:55.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.885+0000 7f6bbfe00640 1 -- start start 2024-02-21T23:06:55.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.886+0000 7f6bbfe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6bc006dbc0 0x7f6bc01dfa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:55.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.886+0000 7f6bbfe00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6bc013e0c0 0x7f6bc0157c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:55.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.886+0000 7f6bbfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6bc013e520 0x7f6bc00ab480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:55.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.886+0000 7f6bbfe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc00c78c0 con 0x7f6bc013e520 2024-02-21T23:06:55.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.886+0000 7f6bbfe00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc0159070 con 0x7f6bc006dbc0 2024-02-21T23:06:55.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.886+0000 7f6bbfe00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc00c7a40 con 0x7f6bc013e0c0 2024-02-21T23:06:55.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.886+0000 7f6bbf400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6bc013e520 0x7f6bc00ab480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:55.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.886+0000 7f6bbf400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6bc013e520 0x7f6bc00ab480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47566/0 (socket says 172.21.15.28:47566) 2024-02-21T23:06:55.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.886+0000 7f6bbf400640 1 -- 172.21.15.28:0/1136079053 learned_addr learned my addr 172.21.15.28:0/1136079053 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:55.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.886+0000 7f6bbea00640 1 --2- 172.21.15.28:0/1136079053 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6bc006dbc0 0x7f6bc01dfa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:55.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.886+0000 7f6bb7e00640 1 --2- 172.21.15.28:0/1136079053 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6bc013e0c0 0x7f6bc0157c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:55.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.887+0000 7f6bbf400640 1 -- 172.21.15.28:0/1136079053 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6bc013e0c0 msgr2=0x7f6bc0157c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:55.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.887+0000 7f6bbf400640 1 --2- 172.21.15.28:0/1136079053 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6bc013e0c0 0x7f6bc0157c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:55.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.887+0000 7f6bbf400640 1 -- 172.21.15.28:0/1136079053 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6bc006dbc0 msgr2=0x7f6bc01dfa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:55.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.887+0000 7f6bbf400640 1 --2- 172.21.15.28:0/1136079053 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6bc006dbc0 0x7f6bc01dfa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:55.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.887+0000 7f6bbf400640 1 -- 172.21.15.28:0/1136079053 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bc00ac160 con 0x7f6bc013e520 2024-02-21T23:06:55.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.887+0000 7f6bb7e00640 1 --2- 172.21.15.28:0/1136079053 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6bc013e0c0 0x7f6bc0157c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:06:55.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.887+0000 7f6bbf400640 1 --2- 172.21.15.28:0/1136079053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6bc013e520 0x7f6bc00ab480 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f6bb8007e90 tx=0x7f6bb800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:55.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.887+0000 7f6bb6a00640 1 -- 172.21.15.28:0/1136079053 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bb8014420 con 0x7f6bc013e520 2024-02-21T23:06:55.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.887+0000 7f6bb6a00640 1 -- 172.21.15.28:0/1136079053 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bb8019070 con 0x7f6bc013e520 2024-02-21T23:06:55.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.887+0000 7f6bbfe00640 1 -- 172.21.15.28:0/1136079053 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bc01de790 con 0x7f6bc013e520 2024-02-21T23:06:55.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.887+0000 7f6bb6a00640 1 -- 172.21.15.28:0/1136079053 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bb8014800 con 0x7f6bc013e520 2024-02-21T23:06:55.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.887+0000 7f6bbfe00640 1 -- 172.21.15.28:0/1136079053 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bc01edfb0 con 0x7f6bc013e520 2024-02-21T23:06:55.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.889+0000 7f6bbfe00640 1 -- 172.21.15.28:0/1136079053 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bc004f290 con 0x7f6bc013e520 2024-02-21T23:06:55.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.890+0000 7f6bb6a00640 1 -- 172.21.15.28:0/1136079053 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6bb80149a0 con 0x7f6bc013e520 2024-02-21T23:06:55.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.891+0000 7f6bb6a00640 1 --2- 172.21.15.28:0/1136079053 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6b9c06b500 0x7f6b9c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:55.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.892+0000 7f6bb6a00640 1 -- 172.21.15.28:0/1136079053 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6bb808acf0 con 0x7f6bc013e520 2024-02-21T23:06:55.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.892+0000 7f6bbea00640 1 --2- 172.21.15.28:0/1136079053 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6b9c06b500 0x7f6b9c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:55.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.893+0000 7f6bbea00640 1 --2- 172.21.15.28:0/1136079053 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6b9c06b500 0x7f6b9c06d9c0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f6bb0006fd0 tx=0x7f6bb0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:55.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:55.896+0000 7f6bb6a00640 1 -- 172.21.15.28:0/1136079053 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f6bc00ac160 con 0x7f6bc013e520 2024-02-21T23:06:56.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:56.144+0000 7f6bbfe00640 1 -- 172.21.15.28:0/1136079053 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f6bc006d9a0 con 0x7f6bc013e520 2024-02-21T23:06:56.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:56.792+0000 7f6bb6a00640 1 -- 172.21.15.28:0/1136079053 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v87) v1 ==== 118+0+0 (secure 0 0 0) 0x7f6bb80598b0 con 0x7f6bc013e520 2024-02-21T23:06:56.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:56.795+0000 7f6bbfe00640 1 -- 172.21.15.28:0/1136079053 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6b9c06b500 msgr2=0x7f6b9c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:56.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:56.795+0000 7f6bbfe00640 1 --2- 172.21.15.28:0/1136079053 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6b9c06b500 0x7f6b9c06d9c0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f6bb0006fd0 tx=0x7f6bb0008040 comp rx=0 tx=0).stop 2024-02-21T23:06:56.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:56.795+0000 7f6bbfe00640 1 -- 172.21.15.28:0/1136079053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6bc013e520 msgr2=0x7f6bc00ab480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:56.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:56.795+0000 7f6bbfe00640 1 --2- 172.21.15.28:0/1136079053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6bc013e520 0x7f6bc00ab480 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f6bb8007e90 tx=0x7f6bb800e4b0 comp rx=0 tx=0).stop 2024-02-21T23:06:56.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:56.795+0000 7f6bbfe00640 1 -- 172.21.15.28:0/1136079053 shutdown_connections 2024-02-21T23:06:56.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:56.795+0000 7f6bbfe00640 1 --2- 172.21.15.28:0/1136079053 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6b9c06b500 0x7f6b9c06d9c0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:56.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:56.795+0000 7f6bbfe00640 1 --2- 172.21.15.28:0/1136079053 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6bc013e0c0 0x7f6bc0157c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:56.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:56.795+0000 7f6bbfe00640 1 --2- 172.21.15.28:0/1136079053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6bc013e520 0x7f6bc00ab480 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:56.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:56.795+0000 7f6bbfe00640 1 --2- 172.21.15.28:0/1136079053 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6bc006dbc0 0x7f6bc01dfa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:56.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:56.795+0000 7f6bbfe00640 1 -- 172.21.15.28:0/1136079053 >> 172.21.15.28:0/1136079053 conn(0x7f6bc0073110 msgr2=0x7f6bc013bb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:56.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:56.795+0000 7f6bbfe00640 1 -- 172.21.15.28:0/1136079053 shutdown_connections 2024-02-21T23:06:56.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:56.795+0000 7f6bbfe00640 1 -- 172.21.15.28:0/1136079053 wait complete. 2024-02-21T23:06:56.802 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-02-21T23:06:56.802 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-02-21T23:06:57.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.018+0000 7f7737200640 1 Processor -- start 2024-02-21T23:06:57.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.018+0000 7f7737200640 1 -- start start 2024-02-21T23:06:57.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.019+0000 7f7737200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7730152650 0x7f77300c8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:57.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.019+0000 7f7737200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f77301550e0 0x7f7730155550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:57.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.019+0000 7f7737200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7730158920 con 0x7f773006daa0 2024-02-21T23:06:57.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.019+0000 7f7737200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77301586d0 con 0x7f77301550e0 2024-02-21T23:06:57.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.019+0000 7f7737200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7730158c00 con 0x7f7730152650 2024-02-21T23:06:57.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.020+0000 7f7735e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7730152650 0x7f77300c8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:57.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.020+0000 7f7736800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f77301550e0 0x7f7730155550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:57.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.020+0000 7f7735e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7730152650 0x7f77300c8ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3301/0 says I am v2:172.21.15.28:41430/0 (socket says 172.21.15.28:41430) 2024-02-21T23:06:57.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.020+0000 7f7735e00640 1 -- 172.21.15.28:0/2270393267 learned_addr learned my addr 172.21.15.28:0/2270393267 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:57.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.021+0000 7f7736800640 1 -- 172.21.15.28:0/2270393267 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7730152650 msgr2=0x7f77300c8ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:57.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.021+0000 7f7736800640 1 --2- 172.21.15.28:0/2270393267 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7730152650 0x7f77300c8ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:57.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.021+0000 7f7736800640 1 -- 172.21.15.28:0/2270393267 >> v1:172.21.15.81:6789/0 conn(0x7f773006daa0 legacy=0x7f77300c7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:57.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.021+0000 7f7736800640 1 -- 172.21.15.28:0/2270393267 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7730158ee0 con 0x7f77301550e0 2024-02-21T23:06:57.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.021+0000 7f7734a00640 1 -- 172.21.15.28:0/2270393267 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3980294459 0 0) 0x7f7730158920 con 0x7f773006daa0 2024-02-21T23:06:57.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.021+0000 7f7735e00640 1 --2- 172.21.15.28:0/2270393267 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7730152650 0x7f77300c8ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-21T23:06:57.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.022+0000 7f7736800640 1 --2- 172.21.15.28:0/2270393267 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f77301550e0 0x7f7730155550 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f772c009d90 tx=0x7f772c02be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f1ddeb7679a77dfc server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:57.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.023+0000 7f7734a00640 1 -- 172.21.15.28:0/2270393267 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f772c039070 con 0x7f77301550e0 2024-02-21T23:06:57.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.023+0000 7f7734a00640 1 -- 172.21.15.28:0/2270393267 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f772c02cbd0 con 0x7f77301550e0 2024-02-21T23:06:57.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.023+0000 7f7734a00640 1 -- 172.21.15.28:0/2270393267 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f772c034440 con 0x7f77301550e0 2024-02-21T23:06:57.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.023+0000 7f7737200640 1 -- 172.21.15.28:0/2270393267 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f77301550e0 msgr2=0x7f7730155550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:57.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.023+0000 7f7737200640 1 --2- 172.21.15.28:0/2270393267 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f77301550e0 0x7f7730155550 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f772c009d90 tx=0x7f772c02be20 comp rx=0 tx=0).stop 2024-02-21T23:06:57.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.023+0000 7f7737200640 1 -- 172.21.15.28:0/2270393267 shutdown_connections 2024-02-21T23:06:57.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.023+0000 7f7737200640 1 --2- 172.21.15.28:0/2270393267 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f77301550e0 0x7f7730155550 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:57.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.023+0000 7f7737200640 1 --2- 172.21.15.28:0/2270393267 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7730152650 0x7f77300c8ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:57.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.023+0000 7f7737200640 1 -- 172.21.15.28:0/2270393267 >> 172.21.15.28:0/2270393267 conn(0x7f7730072ce0 msgr2=0x7f77300ac340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:57.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.023+0000 7f7737200640 1 -- 172.21.15.28:0/2270393267 shutdown_connections 2024-02-21T23:06:57.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.024+0000 7f7737200640 1 -- 172.21.15.28:0/2270393267 wait complete. 2024-02-21T23:06:57.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.025+0000 7f7737200640 1 Processor -- start 2024-02-21T23:06:57.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.025+0000 7f7737200640 1 -- start start 2024-02-21T23:06:57.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.026+0000 7f7737200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f773006daa0 0x7f77301e7ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:57.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.026+0000 7f7737200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7730152650 0x7f7730150cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:57.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.026+0000 7f7737200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77301550e0 0x7f77300aa590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:57.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.026+0000 7f7737200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7730152a30 con 0x7f77301550e0 2024-02-21T23:06:57.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.026+0000 7f7737200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77300bdfb0 con 0x7f7730152650 2024-02-21T23:06:57.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.026+0000 7f7737200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77300c7870 con 0x7f773006daa0 2024-02-21T23:06:57.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.026+0000 7f7736800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77301550e0 0x7f77300aa590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:57.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.026+0000 7f7736800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77301550e0 0x7f77300aa590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47572/0 (socket says 172.21.15.28:47572) 2024-02-21T23:06:57.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.026+0000 7f7736800640 1 -- 172.21.15.28:0/950249900 learned_addr learned my addr 172.21.15.28:0/950249900 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:57.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.027+0000 7f7735e00640 1 --2- 172.21.15.28:0/950249900 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f773006daa0 0x7f77301e7ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:57.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.027+0000 7f7735400640 1 --2- 172.21.15.28:0/950249900 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7730152650 0x7f7730150cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:57.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.027+0000 7f7736800640 1 -- 172.21.15.28:0/950249900 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f773006daa0 msgr2=0x7f77301e7ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:57.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.027+0000 7f7736800640 1 --2- 172.21.15.28:0/950249900 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f773006daa0 0x7f77301e7ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:57.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.027+0000 7f7736800640 1 -- 172.21.15.28:0/950249900 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7730152650 msgr2=0x7f7730150cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:57.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.027+0000 7f7736800640 1 --2- 172.21.15.28:0/950249900 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7730152650 0x7f7730150cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:57.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.027+0000 7f7736800640 1 -- 172.21.15.28:0/950249900 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77301eb700 con 0x7f77301550e0 2024-02-21T23:06:57.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.027+0000 7f7736800640 1 --2- 172.21.15.28:0/950249900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77301550e0 0x7f77300aa590 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f772c037040 tx=0x7f772c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:57.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.027+0000 7f772a000640 1 -- 172.21.15.28:0/950249900 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f772c044020 con 0x7f77301550e0 2024-02-21T23:06:57.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.028+0000 7f772a000640 1 -- 172.21.15.28:0/950249900 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f772c03a040 con 0x7f77301550e0 2024-02-21T23:06:57.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.028+0000 7f772a000640 1 -- 172.21.15.28:0/950249900 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f772c039050 con 0x7f77301550e0 2024-02-21T23:06:57.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.028+0000 7f7737200640 1 -- 172.21.15.28:0/950249900 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77300ab600 con 0x7f77301550e0 2024-02-21T23:06:57.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.028+0000 7f7737200640 1 -- 172.21.15.28:0/950249900 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77300ab8a0 con 0x7f77301550e0 2024-02-21T23:06:57.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.029+0000 7f7737200640 1 -- 172.21.15.28:0/950249900 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77300ab290 con 0x7f77301550e0 2024-02-21T23:06:57.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.030+0000 7f772a000640 1 -- 172.21.15.28:0/950249900 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f772c003bb0 con 0x7f77301550e0 2024-02-21T23:06:57.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.031+0000 7f772a000640 1 --2- 172.21.15.28:0/950249900 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f771406b4d0 0x7f771406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:57.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.031+0000 7f772a000640 1 -- 172.21.15.28:0/950249900 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f772c0ac740 con 0x7f77301550e0 2024-02-21T23:06:57.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.032+0000 7f7735e00640 1 --2- 172.21.15.28:0/950249900 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f771406b4d0 0x7f771406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:57.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.033+0000 7f7735e00640 1 --2- 172.21.15.28:0/950249900 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f771406b4d0 0x7f771406d990 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f7724009b70 tx=0x7f7724008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:57.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.036+0000 7f772a000640 1 -- 172.21.15.28:0/950249900 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f772c07b420 con 0x7f77301550e0 2024-02-21T23:06:57.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.272+0000 7f7737200640 1 -- 172.21.15.28:0/950249900 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f773004f290 con 0x7f77301550e0 2024-02-21T23:06:57.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.804+0000 7f772a000640 1 -- 172.21.15.28:0/950249900 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v88) v1 ==== 122+0+0 (secure 0 0 0) 0x7f772c07ad20 con 0x7f77301550e0 2024-02-21T23:06:57.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.809+0000 7f7737200640 1 -- 172.21.15.28:0/950249900 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f771406b4d0 msgr2=0x7f771406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:57.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.809+0000 7f7737200640 1 --2- 172.21.15.28:0/950249900 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f771406b4d0 0x7f771406d990 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f7724009b70 tx=0x7f7724008040 comp rx=0 tx=0).stop 2024-02-21T23:06:57.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.809+0000 7f7737200640 1 -- 172.21.15.28:0/950249900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77301550e0 msgr2=0x7f77300aa590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:57.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.809+0000 7f7737200640 1 --2- 172.21.15.28:0/950249900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77301550e0 0x7f77300aa590 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f772c037040 tx=0x7f772c004a40 comp rx=0 tx=0).stop 2024-02-21T23:06:57.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.809+0000 7f7737200640 1 -- 172.21.15.28:0/950249900 shutdown_connections 2024-02-21T23:06:57.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.809+0000 7f7737200640 1 --2- 172.21.15.28:0/950249900 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f771406b4d0 0x7f771406d990 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:57.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.809+0000 7f7737200640 1 --2- 172.21.15.28:0/950249900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77301550e0 0x7f77300aa590 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:57.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.809+0000 7f7737200640 1 --2- 172.21.15.28:0/950249900 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7730152650 0x7f7730150cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:57.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.809+0000 7f7737200640 1 --2- 172.21.15.28:0/950249900 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f773006daa0 0x7f77301e7ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:57.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.809+0000 7f7737200640 1 -- 172.21.15.28:0/950249900 >> 172.21.15.28:0/950249900 conn(0x7f7730072ce0 msgr2=0x7f773006ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:57.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.809+0000 7f7737200640 1 -- 172.21.15.28:0/950249900 shutdown_connections 2024-02-21T23:06:57.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:57.809+0000 7f7737200640 1 -- 172.21.15.28:0/950249900 wait complete. 2024-02-21T23:06:57.822 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-02-21T23:06:57.822 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:06:58.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.058+0000 7f3fc3200640 1 Processor -- start 2024-02-21T23:06:58.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.058+0000 7f3fc3200640 1 -- start start 2024-02-21T23:06:58.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.060+0000 7f3fc3200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f3fc4155960 0x7f3fc41608a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:58.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.060+0000 7f3fc3200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fc40c74c0 con 0x7f3fc40aff90 2024-02-21T23:06:58.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.060+0000 7f3fc3200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fc40c77a0 con 0x7f3fc406dbf0 2024-02-21T23:06:58.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.060+0000 7f3fc3200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fc40c7270 con 0x7f3fc4155960 2024-02-21T23:06:58.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.060+0000 7f3fc1e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f3fc40aff90 0x7f3fc40c51e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34600/0 (socket says 172.21.15.28:34600) 2024-02-21T23:06:58.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.060+0000 7f3fc1e00640 1 -- 172.21.15.28:0/4273809457 learned_addr learned my addr 172.21.15.28:0/4273809457 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:06:58.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.061+0000 7f3fc2800640 1 --2- 172.21.15.28:0/4273809457 >> v2:172.21.15.81:3300/0 conn(0x7f3fc4155960 0x7f3fc41608a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:58.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.061+0000 7f3fc0a00640 1 -- 172.21.15.28:0/4273809457 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3521952947 0 0) 0x7f3fc40c74c0 con 0x7f3fc40aff90 2024-02-21T23:06:58.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.061+0000 7f3fc0a00640 1 -- 172.21.15.28:0/4273809457 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fa8003520 con 0x7f3fc40aff90 2024-02-21T23:06:58.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.061+0000 7f3fc0a00640 1 -- 172.21.15.28:0/4273809457 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3860216270 0 0) 0x7f3fa8003520 con 0x7f3fc40aff90 2024-02-21T23:06:58.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.061+0000 7f3fc0a00640 1 -- 172.21.15.28:0/4273809457 >> v2:172.21.15.81:3300/0 conn(0x7f3fc4155960 msgr2=0x7f3fc41608a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:58.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.061+0000 7f3fc0a00640 1 --2- 172.21.15.28:0/4273809457 >> v2:172.21.15.81:3300/0 conn(0x7f3fc4155960 0x7f3fc41608a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:58.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.061+0000 7f3fc0a00640 1 -- 172.21.15.28:0/4273809457 >> v1:172.21.15.81:6789/0 conn(0x7f3fc406dbf0 legacy=0x7f3fc4158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:58.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.061+0000 7f3fc0a00640 1 -- 172.21.15.28:0/4273809457 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fc40c7a80 con 0x7f3fc40aff90 2024-02-21T23:06:58.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.061+0000 7f3fc0a00640 1 -- 172.21.15.28:0/4273809457 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f3fb4001670 con 0x7f3fc40aff90 2024-02-21T23:06:58.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.061+0000 7f3fc0a00640 1 -- 172.21.15.28:0/4273809457 >> v1:172.21.15.28:6789/0 conn(0x7f3fc40aff90 legacy=0x7f3fc40c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:58.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.061+0000 7f3fc0a00640 1 --2- 172.21.15.28:0/4273809457 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3fa8003a00 0x7f3fa8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:58.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.061+0000 7f3fc0a00640 1 --2- 172.21.15.28:0/4273809457 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3fa8004800 0x7f3fa8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:58.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.061+0000 7f3fc0a00640 1 --2- 172.21.15.28:0/4273809457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3fa80051c0 0x7f3fa800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:58.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.061+0000 7f3fc0a00640 1 -- 172.21.15.28:0/4273809457 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc40c7270 con 0x7f3fa80051c0 2024-02-21T23:06:58.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.061+0000 7f3fc0a00640 1 -- 172.21.15.28:0/4273809457 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fa800c0c0 con 0x7f3fa8004800 2024-02-21T23:06:58.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.061+0000 7f3fc0a00640 1 -- 172.21.15.28:0/4273809457 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fa800c2c0 con 0x7f3fa8003a00 2024-02-21T23:06:58.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.062+0000 7f3fc2800640 1 --2- 172.21.15.28:0/4273809457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3fa80051c0 0x7f3fa800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:58.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.062+0000 7f3fc1e00640 1 --2- 172.21.15.28:0/4273809457 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3fa8003a00 0x7f3fa8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:58.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.062+0000 7f3fc1400640 1 --2- 172.21.15.28:0/4273809457 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3fa8004800 0x7f3fa8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:58.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.062+0000 7f3fc2800640 1 -- 172.21.15.28:0/4273809457 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3fa8003a00 msgr2=0x7f3fa8003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:58.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.062+0000 7f3fc2800640 1 --2- 172.21.15.28:0/4273809457 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3fa8003a00 0x7f3fa8003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:58.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.062+0000 7f3fc2800640 1 -- 172.21.15.28:0/4273809457 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3fa8004800 msgr2=0x7f3fa8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:58.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.062+0000 7f3fc2800640 1 --2- 172.21.15.28:0/4273809457 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3fa8004800 0x7f3fa8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:58.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.062+0000 7f3fc2800640 1 -- 172.21.15.28:0/4273809457 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fa8003520 con 0x7f3fa80051c0 2024-02-21T23:06:58.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.062+0000 7f3fc1e00640 1 --2- 172.21.15.28:0/4273809457 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3fa8003a00 0x7f3fa8003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:06:58.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.062+0000 7f3fc1400640 1 --2- 172.21.15.28:0/4273809457 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3fa8004800 0x7f3fa8004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:06:58.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.064+0000 7f3fc2800640 1 -- 172.21.15.28:0/4273809457 reap_dead start 2024-02-21T23:06:58.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.064+0000 7f3fc2800640 1 --2- 172.21.15.28:0/4273809457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3fa80051c0 0x7f3fa800b9c0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f3fbc009fc0 tx=0x7f3fa8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce09c2f28d98172 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:58.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.064+0000 7f3fc0a00640 1 -- 172.21.15.28:0/4273809457 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fbc009c10 con 0x7f3fa80051c0 2024-02-21T23:06:58.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.064+0000 7f3fc0a00640 1 -- 172.21.15.28:0/4273809457 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fbc02cc60 con 0x7f3fa80051c0 2024-02-21T23:06:58.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.064+0000 7f3fc0a00640 1 -- 172.21.15.28:0/4273809457 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fbc003bd0 con 0x7f3fa80051c0 2024-02-21T23:06:58.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.065+0000 7f3fc3200640 1 -- 172.21.15.28:0/4273809457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3fa80051c0 msgr2=0x7f3fa800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:58.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.065+0000 7f3fc3200640 1 --2- 172.21.15.28:0/4273809457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3fa80051c0 0x7f3fa800b9c0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f3fbc009fc0 tx=0x7f3fa8005830 comp rx=0 tx=0).stop 2024-02-21T23:06:58.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.065+0000 7f3fc3200640 1 -- 172.21.15.28:0/4273809457 shutdown_connections 2024-02-21T23:06:58.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.065+0000 7f3fc3200640 1 --2- 172.21.15.28:0/4273809457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3fa80051c0 0x7f3fa800b9c0 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:58.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.065+0000 7f3fc3200640 1 -- 172.21.15.28:0/4273809457 >> 172.21.15.28:0/4273809457 conn(0x7f3fc40ac370 msgr2=0x7f3fc40ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:58.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.065+0000 7f3fc3200640 1 -- 172.21.15.28:0/4273809457 shutdown_connections 2024-02-21T23:06:58.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.065+0000 7f3fc3200640 1 -- 172.21.15.28:0/4273809457 wait complete. 2024-02-21T23:06:58.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.067+0000 7f3fc3200640 1 Processor -- start 2024-02-21T23:06:58.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.067+0000 7f3fc3200640 1 -- start start 2024-02-21T23:06:58.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.068+0000 7f3fc3200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3fa80051c0 0x7f3fc41eaf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:58.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.068+0000 7f3fc3200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3fc40c4220 0x7f3fc40ab780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:58.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.068+0000 7f3fc3200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3fc40abcc0 0x7f3fc41533d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:58.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.068+0000 7f3fc3200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc40c8760 con 0x7f3fc40c4220 2024-02-21T23:06:58.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.068+0000 7f3fc3200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc4158630 con 0x7f3fc40abcc0 2024-02-21T23:06:58.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.068+0000 7f3fc3200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc40c88e0 con 0x7f3fa80051c0 2024-02-21T23:06:58.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.068+0000 7f3fc1400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3fc40c4220 0x7f3fc40ab780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:58.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.068+0000 7f3fc1400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3fc40c4220 0x7f3fc40ab780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47584/0 (socket says 172.21.15.28:47584) 2024-02-21T23:06:58.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.068+0000 7f3fc1400640 1 -- 172.21.15.28:0/1051942804 learned_addr learned my addr 172.21.15.28:0/1051942804 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:58.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.068+0000 7f3fc1e00640 1 --2- 172.21.15.28:0/1051942804 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3fa80051c0 0x7f3fc41eaf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:58.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.069+0000 7f3fc2800640 1 --2- 172.21.15.28:0/1051942804 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3fc40abcc0 0x7f3fc41533d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:58.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.069+0000 7f3fc1400640 1 -- 172.21.15.28:0/1051942804 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3fa80051c0 msgr2=0x7f3fc41eaf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:58.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.069+0000 7f3fc1400640 1 --2- 172.21.15.28:0/1051942804 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3fa80051c0 0x7f3fc41eaf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:58.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.069+0000 7f3fc1400640 1 -- 172.21.15.28:0/1051942804 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3fc40abcc0 msgr2=0x7f3fc41533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:58.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.069+0000 7f3fc1400640 1 --2- 172.21.15.28:0/1051942804 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3fc40abcc0 0x7f3fc41533d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:58.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.069+0000 7f3fc1400640 1 -- 172.21.15.28:0/1051942804 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fc40ac140 con 0x7f3fc40c4220 2024-02-21T23:06:58.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.069+0000 7f3fc2800640 1 --2- 172.21.15.28:0/1051942804 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3fc40abcc0 0x7f3fc41533d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:06:58.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.069+0000 7f3fc1400640 1 --2- 172.21.15.28:0/1051942804 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3fc40c4220 0x7f3fc40ab780 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f3fb000d7d0 tx=0x7f3fb000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:58.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.069+0000 7f3fba000640 1 -- 172.21.15.28:0/1051942804 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb0005270 con 0x7f3fc40c4220 2024-02-21T23:06:58.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.069+0000 7f3fba000640 1 -- 172.21.15.28:0/1051942804 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fb0017070 con 0x7f3fc40c4220 2024-02-21T23:06:58.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.069+0000 7f3fba000640 1 -- 172.21.15.28:0/1051942804 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb000b650 con 0x7f3fc40c4220 2024-02-21T23:06:58.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.069+0000 7f3fc3200640 1 -- 172.21.15.28:0/1051942804 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fc40c2410 con 0x7f3fc40c4220 2024-02-21T23:06:58.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.070+0000 7f3fc3200640 1 -- 172.21.15.28:0/1051942804 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fc4152eb0 con 0x7f3fc40c4220 2024-02-21T23:06:58.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.071+0000 7f3fc3200640 1 -- 172.21.15.28:0/1051942804 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fc404f290 con 0x7f3fc40c4220 2024-02-21T23:06:58.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.072+0000 7f3fba000640 1 -- 172.21.15.28:0/1051942804 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3fb000b7f0 con 0x7f3fc40c4220 2024-02-21T23:06:58.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.073+0000 7f3fba000640 1 --2- 172.21.15.28:0/1051942804 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3fa406b4d0 0x7f3fa406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:58.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.073+0000 7f3fba000640 1 -- 172.21.15.28:0/1051942804 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f3fb008a0a0 con 0x7f3fc40c4220 2024-02-21T23:06:58.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.073+0000 7f3fc1e00640 1 --2- 172.21.15.28:0/1051942804 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3fa406b4d0 0x7f3fa406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:58.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.075+0000 7f3fc1e00640 1 --2- 172.21.15.28:0/1051942804 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3fa406b4d0 0x7f3fa406d990 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f3fb4001200 tx=0x7f3fb4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:58.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.078+0000 7f3fba000640 1 -- 172.21.15.28:0/1051942804 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3fb0058e20 con 0x7f3fc40c4220 2024-02-21T23:06:58.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.315+0000 7f3fc3200640 1 -- 172.21.15.28:0/1051942804 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3fc406d9d0 con 0x7f3fc40c4220 2024-02-21T23:06:58.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.317+0000 7f3fba000640 1 -- 172.21.15.28:0/1051942804 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+4934 (secure 0 0 0) 0x7f3fb0001230 con 0x7f3fc40c4220 2024-02-21T23:06:58.305 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:06:58.305 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":88,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5555,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/85689296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":85689296},{"type":"v1","addr":"172.21.15.28:6833","nonce":85689296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82},{"gid":5567,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1194613007","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1194613007},{"type":"v1","addr":"172.21.15.81:6835","nonce":1194613007}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":5570,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/2315735157","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":2315735157},{"type":"v1","addr":"172.21.15.81:6837","nonce":2315735157}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:06:54.201589+0000","modified":"2024-02-21T23:06:57.273466+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5932},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5932":{"gid":5932,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/2477712025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2477712025},{"type":"v1","addr":"172.21.15.28:6835","nonce":2477712025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-02-21T23:06:58.306 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 88 2024-02-21T23:06:58.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.320+0000 7f3fc3200640 1 -- 172.21.15.28:0/1051942804 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3fa406b4d0 msgr2=0x7f3fa406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:58.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.320+0000 7f3fc3200640 1 --2- 172.21.15.28:0/1051942804 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3fa406b4d0 0x7f3fa406d990 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f3fb4001200 tx=0x7f3fb4009040 comp rx=0 tx=0).stop 2024-02-21T23:06:58.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.320+0000 7f3fc3200640 1 -- 172.21.15.28:0/1051942804 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3fc40c4220 msgr2=0x7f3fc40ab780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:58.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.320+0000 7f3fc3200640 1 --2- 172.21.15.28:0/1051942804 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3fc40c4220 0x7f3fc40ab780 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f3fb000d7d0 tx=0x7f3fb000dc90 comp rx=0 tx=0).stop 2024-02-21T23:06:58.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.320+0000 7f3fc3200640 1 -- 172.21.15.28:0/1051942804 shutdown_connections 2024-02-21T23:06:58.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.320+0000 7f3fc3200640 1 --2- 172.21.15.28:0/1051942804 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3fa406b4d0 0x7f3fa406d990 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:58.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.320+0000 7f3fc3200640 1 --2- 172.21.15.28:0/1051942804 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3fc40abcc0 0x7f3fc41533d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:58.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.320+0000 7f3fc3200640 1 --2- 172.21.15.28:0/1051942804 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3fc40c4220 0x7f3fc40ab780 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:58.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.320+0000 7f3fc3200640 1 --2- 172.21.15.28:0/1051942804 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3fa80051c0 0x7f3fc41eaf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:58.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.320+0000 7f3fc3200640 1 -- 172.21.15.28:0/1051942804 >> 172.21.15.28:0/1051942804 conn(0x7f3fc40ac370 msgr2=0x7f3fc4157e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:58.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.320+0000 7f3fc3200640 1 -- 172.21.15.28:0/1051942804 shutdown_connections 2024-02-21T23:06:58.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.320+0000 7f3fc3200640 1 -- 172.21.15.28:0/1051942804 wait complete. 2024-02-21T23:06:58.328 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:06:58.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.559+0000 7f0ca5a00640 1 Processor -- start 2024-02-21T23:06:58.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.559+0000 7f0ca5a00640 1 -- start start 2024-02-21T23:06:58.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.561+0000 7f0ca5a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0ca006dbf0 0x7f0ca0158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:58.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.561+0000 7f0ca5a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ca00c74c0 con 0x7f0ca00aff90 2024-02-21T23:06:58.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.561+0000 7f0ca5a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ca00c77a0 con 0x7f0ca0155960 2024-02-21T23:06:58.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.561+0000 7f0ca5a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ca00c7270 con 0x7f0ca006dbf0 2024-02-21T23:06:58.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.561+0000 7f0c9fe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f0ca00aff90 0x7f0ca00c51e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34604/0 (socket says 172.21.15.28:34604) 2024-02-21T23:06:58.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.561+0000 7f0c9fe00640 1 -- 172.21.15.28:0/1573329076 learned_addr learned my addr 172.21.15.28:0/1573329076 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:06:58.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.561+0000 7f0c9f400640 1 --2- 172.21.15.28:0/1573329076 >> v2:172.21.15.28:3300/0 conn(0x7f0ca006dbf0 0x7f0ca0158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:58.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.562+0000 7f0c9ea00640 1 -- 172.21.15.28:0/1573329076 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1726610206 0 0) 0x7f0ca00c74c0 con 0x7f0ca00aff90 2024-02-21T23:06:58.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.562+0000 7f0c9f400640 1 -- 172.21.15.28:0/1573329076 >> v1:172.21.15.81:6789/0 conn(0x7f0ca0155960 legacy=0x7f0ca015f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:58.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.562+0000 7f0c9f400640 1 -- 172.21.15.28:0/1573329076 >> v1:172.21.15.28:6789/0 conn(0x7f0ca00aff90 legacy=0x7f0ca00c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:58.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.562+0000 7f0c9f400640 1 -- 172.21.15.28:0/1573329076 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ca00c7a80 con 0x7f0ca006dbf0 2024-02-21T23:06:58.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.563+0000 7f0c9f400640 1 --2- 172.21.15.28:0/1573329076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ca006dbf0 0x7f0ca0158ac0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c0097a0 tx=0x7f0c8c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66d4384fd22aa733 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:58.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.563+0000 7f0c9ea00640 1 -- 172.21.15.28:0/1573329076 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c8c035020 con 0x7f0ca006dbf0 2024-02-21T23:06:58.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.563+0000 7f0c9ea00640 1 -- 172.21.15.28:0/1573329076 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c8c031830 con 0x7f0ca006dbf0 2024-02-21T23:06:58.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.564+0000 7f0c9ea00640 1 -- 172.21.15.28:0/1573329076 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c8c031c00 con 0x7f0ca006dbf0 2024-02-21T23:06:58.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.564+0000 7f0ca5a00640 1 -- 172.21.15.28:0/1573329076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ca006dbf0 msgr2=0x7f0ca0158ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:58.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.564+0000 7f0ca5a00640 1 --2- 172.21.15.28:0/1573329076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ca006dbf0 0x7f0ca0158ac0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c0097a0 tx=0x7f0c8c02b8d0 comp rx=0 tx=0).stop 2024-02-21T23:06:58.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.564+0000 7f0ca5a00640 1 -- 172.21.15.28:0/1573329076 shutdown_connections 2024-02-21T23:06:58.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.564+0000 7f0ca5a00640 1 --2- 172.21.15.28:0/1573329076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ca006dbf0 0x7f0ca0158ac0 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:58.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.564+0000 7f0ca5a00640 1 -- 172.21.15.28:0/1573329076 >> 172.21.15.28:0/1573329076 conn(0x7f0ca00ac370 msgr2=0x7f0ca00ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:58.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.564+0000 7f0ca5a00640 1 -- 172.21.15.28:0/1573329076 shutdown_connections 2024-02-21T23:06:58.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.564+0000 7f0ca5a00640 1 -- 172.21.15.28:0/1573329076 wait complete. 2024-02-21T23:06:58.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.567+0000 7f0ca5a00640 1 Processor -- start 2024-02-21T23:06:58.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.567+0000 7f0ca5a00640 1 -- start start 2024-02-21T23:06:58.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.569+0000 7f0ca5a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ca006dbf0 0x7f0ca00c1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:58.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.569+0000 7f0ca5a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ca00aff90 0x7f0ca00c0230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:58.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.569+0000 7f0ca5a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0ca0155960 0x7f0ca00bf540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:58.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.569+0000 7f0ca5a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca00c58f0 con 0x7f0ca006dbf0 2024-02-21T23:06:58.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.569+0000 7f0ca5a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca0158630 con 0x7f0ca00aff90 2024-02-21T23:06:58.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.569+0000 7f0ca5a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca00c5a70 con 0x7f0ca0155960 2024-02-21T23:06:58.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.569+0000 7f0c9fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ca006dbf0 0x7f0ca00c1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:58.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.569+0000 7f0c9fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ca006dbf0 0x7f0ca00c1930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47606/0 (socket says 172.21.15.28:47606) 2024-02-21T23:06:58.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.569+0000 7f0c9fe00640 1 -- 172.21.15.28:0/1313760297 learned_addr learned my addr 172.21.15.28:0/1313760297 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:58.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.570+0000 7f0c9f400640 1 --2- 172.21.15.28:0/1313760297 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ca00aff90 0x7f0ca00c0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:58.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.570+0000 7f0c9fe00640 1 -- 172.21.15.28:0/1313760297 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0ca0155960 msgr2=0x7f0ca00bf540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:58.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.570+0000 7f0ca5000640 1 --2- 172.21.15.28:0/1313760297 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0ca0155960 0x7f0ca00bf540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:58.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.570+0000 7f0c9fe00640 1 --2- 172.21.15.28:0/1313760297 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0ca0155960 0x7f0ca00bf540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:58.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.570+0000 7f0c9fe00640 1 -- 172.21.15.28:0/1313760297 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ca00aff90 msgr2=0x7f0ca00c0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:58.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.570+0000 7f0c9fe00640 1 --2- 172.21.15.28:0/1313760297 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ca00aff90 0x7f0ca00c0230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:58.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.570+0000 7f0c9fe00640 1 -- 172.21.15.28:0/1313760297 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ca0155430 con 0x7f0ca006dbf0 2024-02-21T23:06:58.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.570+0000 7f0c9f400640 1 --2- 172.21.15.28:0/1313760297 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ca00aff90 0x7f0ca00c0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:06:58.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.570+0000 7f0ca5000640 1 --2- 172.21.15.28:0/1313760297 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0ca0155960 0x7f0ca00bf540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:06:58.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.570+0000 7f0c9fe00640 1 --2- 172.21.15.28:0/1313760297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ca006dbf0 0x7f0ca00c1930 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f0c9000c500 tx=0x7f0c9000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:58.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.570+0000 7f0c97e00640 1 -- 172.21.15.28:0/1313760297 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c90018070 con 0x7f0ca006dbf0 2024-02-21T23:06:58.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.570+0000 7f0c97e00640 1 -- 172.21.15.28:0/1313760297 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c90007580 con 0x7f0ca006dbf0 2024-02-21T23:06:58.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.570+0000 7f0ca5a00640 1 -- 172.21.15.28:0/1313760297 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ca00be4a0 con 0x7f0ca006dbf0 2024-02-21T23:06:58.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.570+0000 7f0c97e00640 1 -- 172.21.15.28:0/1313760297 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c90004a20 con 0x7f0ca006dbf0 2024-02-21T23:06:58.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.571+0000 7f0ca5a00640 1 -- 172.21.15.28:0/1313760297 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ca00bea60 con 0x7f0ca006dbf0 2024-02-21T23:06:58.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.572+0000 7f0ca5a00640 1 -- 172.21.15.28:0/1313760297 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ca004f290 con 0x7f0ca006dbf0 2024-02-21T23:06:58.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.573+0000 7f0c97e00640 1 -- 172.21.15.28:0/1313760297 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0c9001b070 con 0x7f0ca006dbf0 2024-02-21T23:06:58.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.574+0000 7f0c97e00640 1 --2- 172.21.15.28:0/1313760297 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0c8006b4d0 0x7f0c8006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:58.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.574+0000 7f0c97e00640 1 -- 172.21.15.28:0/1313760297 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f0c90089680 con 0x7f0ca006dbf0 2024-02-21T23:06:58.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.575+0000 7f0c9f400640 1 --2- 172.21.15.28:0/1313760297 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0c8006b4d0 0x7f0c8006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:58.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.576+0000 7f0c9f400640 1 --2- 172.21.15.28:0/1313760297 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0c8006b4d0 0x7f0c8006d990 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c036000 tx=0x7f0c8c008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:58.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.578+0000 7f0c97e00640 1 -- 172.21.15.28:0/1313760297 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f0c90058360 con 0x7f0ca006dbf0 2024-02-21T23:06:58.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.778+0000 7f0ca5a00640 1 -- 172.21.15.28:0/1313760297 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0ca006d9d0 con 0x7f0ca006dbf0 2024-02-21T23:06:58.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.779+0000 7f0c97e00640 1 -- 172.21.15.28:0/1313760297 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) v1 ==== 74+0+16813 (secure 0 0 0) 0x7f0c90057c60 con 0x7f0ca006dbf0 2024-02-21T23:06:58.768 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:06:58.768 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":98,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:06:55.751401+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:06:52.151181+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-02-21T23:06:53.221776+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:06:58.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.781+0000 7f0ca5a00640 1 -- 172.21.15.28:0/1313760297 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0c8006b4d0 msgr2=0x7f0c8006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:58.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.781+0000 7f0ca5a00640 1 --2- 172.21.15.28:0/1313760297 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0c8006b4d0 0x7f0c8006d990 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c036000 tx=0x7f0c8c008030 comp rx=0 tx=0).stop 2024-02-21T23:06:58.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.781+0000 7f0ca5a00640 1 -- 172.21.15.28:0/1313760297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ca006dbf0 msgr2=0x7f0ca00c1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:58.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.781+0000 7f0ca5a00640 1 --2- 172.21.15.28:0/1313760297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ca006dbf0 0x7f0ca00c1930 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f0c9000c500 tx=0x7f0c9000c9c0 comp rx=0 tx=0).stop 2024-02-21T23:06:58.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.782+0000 7f0ca5a00640 1 -- 172.21.15.28:0/1313760297 shutdown_connections 2024-02-21T23:06:58.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.782+0000 7f0ca5a00640 1 --2- 172.21.15.28:0/1313760297 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0c8006b4d0 0x7f0c8006d990 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:58.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.782+0000 7f0ca5a00640 1 --2- 172.21.15.28:0/1313760297 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0ca0155960 0x7f0ca00bf540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:58.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.782+0000 7f0ca5a00640 1 --2- 172.21.15.28:0/1313760297 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ca00aff90 0x7f0ca00c0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:58.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.782+0000 7f0ca5a00640 1 --2- 172.21.15.28:0/1313760297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ca006dbf0 0x7f0ca00c1930 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:58.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.782+0000 7f0ca5a00640 1 -- 172.21.15.28:0/1313760297 >> 172.21.15.28:0/1313760297 conn(0x7f0ca00ac370 msgr2=0x7f0ca0157220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:58.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.782+0000 7f0ca5a00640 1 -- 172.21.15.28:0/1313760297 shutdown_connections 2024-02-21T23:06:58.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:58.782+0000 7f0ca5a00640 1 -- 172.21.15.28:0/1313760297 wait complete. 2024-02-21T23:06:58.791 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-02-21T23:06:58.791 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-21T23:06:59.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.033+0000 7f8269200640 1 Processor -- start 2024-02-21T23:06:59.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.033+0000 7f8269200640 1 -- start start 2024-02-21T23:06:59.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.034+0000 7f8269200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f82641569d0 0x7f82640c3280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:59.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.034+0000 7f8269200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f8264159400 0x7f826406e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:59.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.034+0000 7f8269200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f826406ed10 con 0x7f82640c4fe0 2024-02-21T23:06:59.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.034+0000 7f8269200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8264159900 con 0x7f8264159400 2024-02-21T23:06:59.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.034+0000 7f8269200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8264159be0 con 0x7f82641569d0 2024-02-21T23:06:59.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.035+0000 7f8262800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f8264159400 0x7f826406e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:59.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.035+0000 7f8261400640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f82640c4fe0 0x7f826406e0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.28:43380/0 (socket says 172.21.15.28:43380) 2024-02-21T23:06:59.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.035+0000 7f8261400640 1 -- 172.21.15.28:0/882858349 learned_addr learned my addr 172.21.15.28:0/882858349 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:06:59.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.035+0000 7f8261e00640 1 --2- 172.21.15.28:0/882858349 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f82641569d0 0x7f82640c3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:59.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.036+0000 7f8260a00640 1 -- 172.21.15.28:0/882858349 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 309037780 0 0) 0x7f826406ed10 con 0x7f82640c4fe0 2024-02-21T23:06:59.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.036+0000 7f8261e00640 1 -- 172.21.15.28:0/882858349 >> v2:172.21.15.81:3300/0 conn(0x7f8264159400 msgr2=0x7f826406e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:59.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.036+0000 7f8261e00640 1 --2- 172.21.15.28:0/882858349 >> v2:172.21.15.81:3300/0 conn(0x7f8264159400 0x7f826406e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:59.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.036+0000 7f8261e00640 1 -- 172.21.15.28:0/882858349 >> v1:172.21.15.81:6789/0 conn(0x7f82640c4fe0 legacy=0x7f826406e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:59.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.036+0000 7f8261e00640 1 -- 172.21.15.28:0/882858349 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8264159ec0 con 0x7f82641569d0 2024-02-21T23:06:59.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.037+0000 7f8261e00640 1 --2- 172.21.15.28:0/882858349 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f82641569d0 0x7f82640c3280 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8254009d90 tx=0x7f825402be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4616bdeae5c333c6 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:59.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.038+0000 7f8260a00640 1 -- 172.21.15.28:0/882858349 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8254039070 con 0x7f82641569d0 2024-02-21T23:06:59.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.038+0000 7f8260a00640 1 -- 172.21.15.28:0/882858349 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f825402cbc0 con 0x7f82641569d0 2024-02-21T23:06:59.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.038+0000 7f8260a00640 1 -- 172.21.15.28:0/882858349 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8254034440 con 0x7f82641569d0 2024-02-21T23:06:59.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.038+0000 7f8269200640 1 -- 172.21.15.28:0/882858349 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f82641569d0 msgr2=0x7f82640c3280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:59.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.038+0000 7f8269200640 1 --2- 172.21.15.28:0/882858349 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f82641569d0 0x7f82640c3280 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8254009d90 tx=0x7f825402be10 comp rx=0 tx=0).stop 2024-02-21T23:06:59.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.039+0000 7f8269200640 1 -- 172.21.15.28:0/882858349 shutdown_connections 2024-02-21T23:06:59.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.039+0000 7f8269200640 1 --2- 172.21.15.28:0/882858349 >> v2:172.21.15.81:3300/0 conn(0x7f8264159400 0x7f826406e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:59.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.039+0000 7f8269200640 1 --2- 172.21.15.28:0/882858349 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f82641569d0 0x7f82640c3280 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:59.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.039+0000 7f8269200640 1 -- 172.21.15.28:0/882858349 >> 172.21.15.28:0/882858349 conn(0x7f82640c67f0 msgr2=0x7f82640c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:59.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.039+0000 7f8269200640 1 -- 172.21.15.28:0/882858349 shutdown_connections 2024-02-21T23:06:59.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.039+0000 7f8269200640 1 -- 172.21.15.28:0/882858349 wait complete. 2024-02-21T23:06:59.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.041+0000 7f8269200640 1 Processor -- start 2024-02-21T23:06:59.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.041+0000 7f8269200640 1 -- start start 2024-02-21T23:06:59.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.042+0000 7f8269200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82640c4fe0 0x7f8264153a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:59.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.042+0000 7f8269200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f82641569d0 0x7f82641eef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:59.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.042+0000 7f8269200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8264159400 0x7f82641ef490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:59.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.042+0000 7f8269200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82640c37c0 con 0x7f8264159400 2024-02-21T23:06:59.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.042+0000 7f8269200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8264156db0 con 0x7f82640c4fe0 2024-02-21T23:06:59.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.042+0000 7f8269200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82640c3940 con 0x7f82641569d0 2024-02-21T23:06:59.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.042+0000 7f8262800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8264159400 0x7f82641ef490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:59.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.042+0000 7f8262800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8264159400 0x7f82641ef490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47610/0 (socket says 172.21.15.28:47610) 2024-02-21T23:06:59.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.042+0000 7f8262800640 1 -- 172.21.15.28:0/2060473702 learned_addr learned my addr 172.21.15.28:0/2060473702 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:59.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.042+0000 7f8261e00640 1 --2- 172.21.15.28:0/2060473702 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82640c4fe0 0x7f8264153a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:59.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.043+0000 7f8261400640 1 --2- 172.21.15.28:0/2060473702 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f82641569d0 0x7f82641eef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:59.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.043+0000 7f8262800640 1 -- 172.21.15.28:0/2060473702 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f82641569d0 msgr2=0x7f82641eef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:59.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.043+0000 7f8262800640 1 --2- 172.21.15.28:0/2060473702 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f82641569d0 0x7f82641eef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:59.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.043+0000 7f8262800640 1 -- 172.21.15.28:0/2060473702 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82640c4fe0 msgr2=0x7f8264153a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:59.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.043+0000 7f8262800640 1 --2- 172.21.15.28:0/2060473702 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82640c4fe0 0x7f8264153a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:59.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.043+0000 7f8262800640 1 -- 172.21.15.28:0/2060473702 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8264150d10 con 0x7f8264159400 2024-02-21T23:06:59.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.043+0000 7f8262800640 1 --2- 172.21.15.28:0/2060473702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8264159400 0x7f82641ef490 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f825c00e820 tx=0x7f825c00ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:59.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.043+0000 7f825a000640 1 -- 172.21.15.28:0/2060473702 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f825c017070 con 0x7f8264159400 2024-02-21T23:06:59.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.043+0000 7f825a000640 1 -- 172.21.15.28:0/2060473702 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f825c00c730 con 0x7f8264159400 2024-02-21T23:06:59.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.043+0000 7f825a000640 1 -- 172.21.15.28:0/2060473702 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f825c00cb00 con 0x7f8264159400 2024-02-21T23:06:59.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.043+0000 7f8269200640 1 -- 172.21.15.28:0/2060473702 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82641526f0 con 0x7f8264159400 2024-02-21T23:06:59.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.044+0000 7f8269200640 1 -- 172.21.15.28:0/2060473702 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8264153f50 con 0x7f8264159400 2024-02-21T23:06:59.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.045+0000 7f8269200640 1 -- 172.21.15.28:0/2060473702 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8264067f30 con 0x7f8264159400 2024-02-21T23:06:59.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.046+0000 7f825a000640 1 -- 172.21.15.28:0/2060473702 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f825c01a070 con 0x7f8264159400 2024-02-21T23:06:59.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.047+0000 7f825a000640 1 --2- 172.21.15.28:0/2060473702 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f824406b4d0 0x7f824406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:59.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.047+0000 7f825a000640 1 -- 172.21.15.28:0/2060473702 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f825c08a5d0 con 0x7f8264159400 2024-02-21T23:06:59.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.048+0000 7f8261e00640 1 --2- 172.21.15.28:0/2060473702 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f824406b4d0 0x7f824406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:59.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.049+0000 7f8261e00640 1 --2- 172.21.15.28:0/2060473702 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f824406b4d0 0x7f824406d990 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f8254036000 tx=0x7f8254030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:59.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.052+0000 7f825a000640 1 -- 172.21.15.28:0/2060473702 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f825c0592b0 con 0x7f8264159400 2024-02-21T23:06:59.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.283+0000 7f8269200640 1 -- 172.21.15.28:0/2060473702 --> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f826406f1d0 con 0x7f824406b4d0 2024-02-21T23:06:59.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.297+0000 7f825a000640 1 -- 172.21.15.28:0/2060473702 <== mgr.4101 v2:172.21.15.81:6832/1848538577 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252115 (secure 0 0 0) 0x7f826406f1d0 con 0x7f824406b4d0 2024-02-21T23:06:59.289 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:06:59.291 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-21T23:06:59.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.303+0000 7f8269200640 1 -- 172.21.15.28:0/2060473702 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f824406b4d0 msgr2=0x7f824406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:59.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.303+0000 7f8269200640 1 --2- 172.21.15.28:0/2060473702 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f824406b4d0 0x7f824406d990 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f8254036000 tx=0x7f8254030000 comp rx=0 tx=0).stop 2024-02-21T23:06:59.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.303+0000 7f8269200640 1 -- 172.21.15.28:0/2060473702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8264159400 msgr2=0x7f82641ef490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:59.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.303+0000 7f8269200640 1 --2- 172.21.15.28:0/2060473702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8264159400 0x7f82641ef490 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f825c00e820 tx=0x7f825c00ece0 comp rx=0 tx=0).stop 2024-02-21T23:06:59.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.303+0000 7f8269200640 1 -- 172.21.15.28:0/2060473702 shutdown_connections 2024-02-21T23:06:59.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.303+0000 7f8269200640 1 --2- 172.21.15.28:0/2060473702 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f824406b4d0 0x7f824406d990 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:59.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.303+0000 7f8269200640 1 --2- 172.21.15.28:0/2060473702 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f82641569d0 0x7f82641eef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:59.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.303+0000 7f8269200640 1 --2- 172.21.15.28:0/2060473702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8264159400 0x7f82641ef490 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:59.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.303+0000 7f8269200640 1 --2- 172.21.15.28:0/2060473702 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f82640c4fe0 0x7f8264153a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:59.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.303+0000 7f8269200640 1 -- 172.21.15.28:0/2060473702 >> 172.21.15.28:0/2060473702 conn(0x7f82640c67f0 msgr2=0x7f82640c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:59.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.304+0000 7f8269200640 1 -- 172.21.15.28:0/2060473702 shutdown_connections 2024-02-21T23:06:59.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.304+0000 7f8269200640 1 -- 172.21.15.28:0/2060473702 wait complete. 2024-02-21T23:06:59.312 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":225,"stamp":"2024-02-21T23:06:58.758555+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":79,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":259084,"kb_used_data":9612,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754715636,"statfs":{"total":773094113280,"available":772828811264,"internally_reserved":0,"allocated":9842688,"data_stored":5896440,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002335"},"pg_stats":[{"pgid":"16.3f","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-21T23:06:55.759282+0000","last_change":"2024-02-21T23:06:54.782740+0000","last_active":"2024-02-21T23:06:55.759282+0000","last_peered":"2024-02-21T23:06:55.759282+0000","last_clean":"2024-02-21T23:06:55.759282+0000","last_became_active":"2024-02-21T23:06:54.782364+0000","last_became_peered":"2024-02-21T23:06:54.782364+0000","last_unstale":"2024-02-21T23:06:55.759282+0000","last_undegraded":"2024-02-21T23:06:55.759282+0000","last_fullsized":"2024-02-21T23:06:55.759282+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:42:22.612435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3e","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.797843+0000","last_change":"2024-02-21T23:06:54.790581+0000","last_active":"2024-02-21T23:06:54.797843+0000","last_peered":"2024-02-21T23:06:54.797843+0000","last_clean":"2024-02-21T23:06:54.797843+0000","last_became_active":"2024-02-21T23:06:54.790272+0000","last_became_peered":"2024-02-21T23:06:54.790272+0000","last_unstale":"2024-02-21T23:06:54.797843+0000","last_undegraded":"2024-02-21T23:06:54.797843+0000","last_fullsized":"2024-02-21T23:06:54.797843+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:16:27.612073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.3d","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-21T23:06:55.758640+0000","last_change":"2024-02-21T23:06:54.805419+0000","last_active":"2024-02-21T23:06:55.758640+0000","last_peered":"2024-02-21T23:06:55.758640+0000","last_clean":"2024-02-21T23:06:55.758640+0000","last_became_active":"2024-02-21T23:06:54.801524+0000","last_became_peered":"2024-02-21T23:06:54.801524+0000","last_unstale":"2024-02-21T23:06:55.758640+0000","last_undegraded":"2024-02-21T23:06:55.758640+0000","last_fullsized":"2024-02-21T23:06:55.758640+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:32:08.968265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3c","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-21T23:06:55.760276+0000","last_change":"2024-02-21T23:06:54.788418+0000","last_active":"2024-02-21T23:06:55.760276+0000","last_peered":"2024-02-21T23:06:55.760276+0000","last_clean":"2024-02-21T23:06:55.760276+0000","last_became_active":"2024-02-21T23:06:54.782881+0000","last_became_peered":"2024-02-21T23:06:54.782881+0000","last_unstale":"2024-02-21T23:06:55.760276+0000","last_undegraded":"2024-02-21T23:06:55.760276+0000","last_fullsized":"2024-02-21T23:06:55.760276+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:54:58.664903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.3b","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.817510+0000","last_change":"2024-02-21T23:06:54.803107+0000","last_active":"2024-02-21T23:06:54.817510+0000","last_peered":"2024-02-21T23:06:54.817510+0000","last_clean":"2024-02-21T23:06:54.817510+0000","last_became_active":"2024-02-21T23:06:54.802533+0000","last_became_peered":"2024-02-21T23:06:54.802533+0000","last_unstale":"2024-02-21T23:06:54.817510+0000","last_undegraded":"2024-02-21T23:06:54.817510+0000","last_fullsized":"2024-02-21T23:06:54.817510+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:00:52.018344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.3a","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.816169+0000","last_change":"2024-02-21T23:06:54.811691+0000","last_active":"2024-02-21T23:06:54.816169+0000","last_peered":"2024-02-21T23:06:54.816169+0000","last_clean":"2024-02-21T23:06:54.816169+0000","last_became_active":"2024-02-21T23:06:54.810596+0000","last_became_peered":"2024-02-21T23:06:54.810596+0000","last_unstale":"2024-02-21T23:06:54.816169+0000","last_undegraded":"2024-02-21T23:06:54.816169+0000","last_fullsized":"2024-02-21T23:06:54.816169+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:37:05.788994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.39","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.823254+0000","last_change":"2024-02-21T23:06:54.822341+0000","last_active":"2024-02-21T23:06:54.823254+0000","last_peered":"2024-02-21T23:06:54.823254+0000","last_clean":"2024-02-21T23:06:54.823254+0000","last_became_active":"2024-02-21T23:06:54.822050+0000","last_became_peered":"2024-02-21T23:06:54.822050+0000","last_unstale":"2024-02-21T23:06:54.823254+0000","last_undegraded":"2024-02-21T23:06:54.823254+0000","last_fullsized":"2024-02-21T23:06:54.823254+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:16:55.110025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.38","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.819911+0000","last_change":"2024-02-21T23:06:54.815560+0000","last_active":"2024-02-21T23:06:54.819911+0000","last_peered":"2024-02-21T23:06:54.819911+0000","last_clean":"2024-02-21T23:06:54.819911+0000","last_became_active":"2024-02-21T23:06:54.815204+0000","last_became_peered":"2024-02-21T23:06:54.815204+0000","last_unstale":"2024-02-21T23:06:54.819911+0000","last_undegraded":"2024-02-21T23:06:54.819911+0000","last_fullsized":"2024-02-21T23:06:54.819911+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:37:27.165471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.37","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.799561+0000","last_change":"2024-02-21T23:06:54.795637+0000","last_active":"2024-02-21T23:06:54.799561+0000","last_peered":"2024-02-21T23:06:54.799561+0000","last_clean":"2024-02-21T23:06:54.799561+0000","last_became_active":"2024-02-21T23:06:54.795280+0000","last_became_peered":"2024-02-21T23:06:54.795280+0000","last_unstale":"2024-02-21T23:06:54.799561+0000","last_undegraded":"2024-02-21T23:06:54.799561+0000","last_fullsized":"2024-02-21T23:06:54.799561+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:36:32.615896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.36","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.815914+0000","last_change":"2024-02-21T23:06:54.802917+0000","last_active":"2024-02-21T23:06:54.815914+0000","last_peered":"2024-02-21T23:06:54.815914+0000","last_clean":"2024-02-21T23:06:54.815914+0000","last_became_active":"2024-02-21T23:06:54.802593+0000","last_became_peered":"2024-02-21T23:06:54.802593+0000","last_unstale":"2024-02-21T23:06:54.815914+0000","last_undegraded":"2024-02-21T23:06:54.815914+0000","last_fullsized":"2024-02-21T23:06:54.815914+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:41:39.165752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.35","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.800192+0000","last_change":"2024-02-21T23:06:54.795064+0000","last_active":"2024-02-21T23:06:54.800192+0000","last_peered":"2024-02-21T23:06:54.800192+0000","last_clean":"2024-02-21T23:06:54.800192+0000","last_became_active":"2024-02-21T23:06:54.794219+0000","last_became_peered":"2024-02-21T23:06:54.794219+0000","last_unstale":"2024-02-21T23:06:54.800192+0000","last_undegraded":"2024-02-21T23:06:54.800192+0000","last_fullsized":"2024-02-21T23:06:54.800192+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:20:08.122424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.34","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-21T23:06:55.762812+0000","last_change":"2024-02-21T23:06:54.789417+0000","last_active":"2024-02-21T23:06:55.762812+0000","last_peered":"2024-02-21T23:06:55.762812+0000","last_clean":"2024-02-21T23:06:55.762812+0000","last_became_active":"2024-02-21T23:06:54.788775+0000","last_became_peered":"2024-02-21T23:06:54.788775+0000","last_unstale":"2024-02-21T23:06:55.762812+0000","last_undegraded":"2024-02-21T23:06:55.762812+0000","last_fullsized":"2024-02-21T23:06:55.762812+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:51:32.395965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.33","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-21T23:06:55.760429+0000","last_change":"2024-02-21T23:06:54.793906+0000","last_active":"2024-02-21T23:06:55.760429+0000","last_peered":"2024-02-21T23:06:55.760429+0000","last_clean":"2024-02-21T23:06:55.760429+0000","last_became_active":"2024-02-21T23:06:54.793526+0000","last_became_peered":"2024-02-21T23:06:54.793526+0000","last_unstale":"2024-02-21T23:06:55.760429+0000","last_undegraded":"2024-02-21T23:06:55.760429+0000","last_fullsized":"2024-02-21T23:06:55.760429+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:24:58.275117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.32","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.789983+0000","last_change":"2024-02-21T23:06:54.787542+0000","last_active":"2024-02-21T23:06:54.789983+0000","last_peered":"2024-02-21T23:06:54.789983+0000","last_clean":"2024-02-21T23:06:54.789983+0000","last_became_active":"2024-02-21T23:06:54.787234+0000","last_became_peered":"2024-02-21T23:06:54.787234+0000","last_unstale":"2024-02-21T23:06:54.789983+0000","last_undegraded":"2024-02-21T23:06:54.789983+0000","last_fullsized":"2024-02-21T23:06:54.789983+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:20:08.122424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.31","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.799711+0000","last_change":"2024-02-21T23:06:54.794883+0000","last_active":"2024-02-21T23:06:54.799711+0000","last_peered":"2024-02-21T23:06:54.799711+0000","last_clean":"2024-02-21T23:06:54.799711+0000","last_became_active":"2024-02-21T23:06:54.794465+0000","last_became_peered":"2024-02-21T23:06:54.794465+0000","last_unstale":"2024-02-21T23:06:54.799711+0000","last_undegraded":"2024-02-21T23:06:54.799711+0000","last_fullsized":"2024-02-21T23:06:54.799711+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:40:27.170195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.30","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.799762+0000","last_change":"2024-02-21T23:06:54.796425+0000","last_active":"2024-02-21T23:06:54.799762+0000","last_peered":"2024-02-21T23:06:54.799762+0000","last_clean":"2024-02-21T23:06:54.799762+0000","last_became_active":"2024-02-21T23:06:54.795767+0000","last_became_peered":"2024-02-21T23:06:54.795767+0000","last_unstale":"2024-02-21T23:06:54.799762+0000","last_undegraded":"2024-02-21T23:06:54.799762+0000","last_fullsized":"2024-02-21T23:06:54.799762+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:14:40.765415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.2f","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-21T23:06:55.759502+0000","last_change":"2024-02-21T23:06:54.791090+0000","last_active":"2024-02-21T23:06:55.759502+0000","last_peered":"2024-02-21T23:06:55.759502+0000","last_clean":"2024-02-21T23:06:55.759502+0000","last_became_active":"2024-02-21T23:06:54.790700+0000","last_became_peered":"2024-02-21T23:06:54.790700+0000","last_unstale":"2024-02-21T23:06:55.759502+0000","last_undegraded":"2024-02-21T23:06:55.759502+0000","last_fullsized":"2024-02-21T23:06:55.759502+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:33:13.357252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2e","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.816207+0000","last_change":"2024-02-21T23:06:54.811699+0000","last_active":"2024-02-21T23:06:54.816207+0000","last_peered":"2024-02-21T23:06:54.816207+0000","last_clean":"2024-02-21T23:06:54.816207+0000","last_became_active":"2024-02-21T23:06:54.811369+0000","last_became_peered":"2024-02-21T23:06:54.811369+0000","last_unstale":"2024-02-21T23:06:54.816207+0000","last_undegraded":"2024-02-21T23:06:54.816207+0000","last_fullsized":"2024-02-21T23:06:54.816207+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:35:47.130397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.2d","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-21T23:06:55.758672+0000","last_change":"2024-02-21T23:06:54.805707+0000","last_active":"2024-02-21T23:06:55.758672+0000","last_peered":"2024-02-21T23:06:55.758672+0000","last_clean":"2024-02-21T23:06:55.758672+0000","last_became_active":"2024-02-21T23:06:54.803725+0000","last_became_peered":"2024-02-21T23:06:54.803725+0000","last_unstale":"2024-02-21T23:06:55.758672+0000","last_undegraded":"2024-02-21T23:06:55.758672+0000","last_fullsized":"2024-02-21T23:06:55.758672+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:28:27.997114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2c","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-21T23:06:55.762834+0000","last_change":"2024-02-21T23:06:54.789182+0000","last_active":"2024-02-21T23:06:55.762834+0000","last_peered":"2024-02-21T23:06:55.762834+0000","last_clean":"2024-02-21T23:06:55.762834+0000","last_became_active":"2024-02-21T23:06:54.788144+0000","last_became_peered":"2024-02-21T23:06:54.788144+0000","last_unstale":"2024-02-21T23:06:55.762834+0000","last_undegraded":"2024-02-21T23:06:55.762834+0000","last_fullsized":"2024-02-21T23:06:55.762834+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:27:38.906198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2b","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.811525+0000","last_change":"2024-02-21T23:06:54.802442+0000","last_active":"2024-02-21T23:06:54.811525+0000","last_peered":"2024-02-21T23:06:54.811525+0000","last_clean":"2024-02-21T23:06:54.811525+0000","last_became_active":"2024-02-21T23:06:54.802137+0000","last_became_peered":"2024-02-21T23:06:54.802137+0000","last_unstale":"2024-02-21T23:06:54.811525+0000","last_undegraded":"2024-02-21T23:06:54.811525+0000","last_fullsized":"2024-02-21T23:06:54.811525+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:52:13.983928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2a","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.821322+0000","last_change":"2024-02-21T23:06:54.804229+0000","last_active":"2024-02-21T23:06:54.821322+0000","last_peered":"2024-02-21T23:06:54.821322+0000","last_clean":"2024-02-21T23:06:54.821322+0000","last_became_active":"2024-02-21T23:06:54.803918+0000","last_became_peered":"2024-02-21T23:06:54.803918+0000","last_unstale":"2024-02-21T23:06:54.821322+0000","last_undegraded":"2024-02-21T23:06:54.821322+0000","last_fullsized":"2024-02-21T23:06:54.821322+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:37:05.788994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.29","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-21T23:06:55.757763+0000","last_change":"2024-02-21T23:06:54.797757+0000","last_active":"2024-02-21T23:06:55.757763+0000","last_peered":"2024-02-21T23:06:55.757763+0000","last_clean":"2024-02-21T23:06:55.757763+0000","last_became_active":"2024-02-21T23:06:54.797437+0000","last_became_peered":"2024-02-21T23:06:54.797437+0000","last_unstale":"2024-02-21T23:06:55.757763+0000","last_undegraded":"2024-02-21T23:06:55.757763+0000","last_fullsized":"2024-02-21T23:06:55.757763+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:07:42.797378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.28","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.821355+0000","last_change":"2024-02-21T23:06:54.808884+0000","last_active":"2024-02-21T23:06:54.821355+0000","last_peered":"2024-02-21T23:06:54.821355+0000","last_clean":"2024-02-21T23:06:54.821355+0000","last_became_active":"2024-02-21T23:06:54.804268+0000","last_became_peered":"2024-02-21T23:06:54.804268+0000","last_unstale":"2024-02-21T23:06:54.821355+0000","last_undegraded":"2024-02-21T23:06:54.821355+0000","last_fullsized":"2024-02-21T23:06:54.821355+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:35:47.130397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.27","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-21T23:06:55.762944+0000","last_change":"2024-02-21T23:06:54.796911+0000","last_active":"2024-02-21T23:06:55.762944+0000","last_peered":"2024-02-21T23:06:55.762944+0000","last_clean":"2024-02-21T23:06:55.762944+0000","last_became_active":"2024-02-21T23:06:54.796598+0000","last_became_peered":"2024-02-21T23:06:54.796598+0000","last_unstale":"2024-02-21T23:06:55.762944+0000","last_undegraded":"2024-02-21T23:06:55.762944+0000","last_fullsized":"2024-02-21T23:06:55.762944+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:14:16.213619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.26","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-21T23:06:55.762972+0000","last_change":"2024-02-21T23:06:54.793514+0000","last_active":"2024-02-21T23:06:55.762972+0000","last_peered":"2024-02-21T23:06:55.762972+0000","last_clean":"2024-02-21T23:06:55.762972+0000","last_became_active":"2024-02-21T23:06:54.793190+0000","last_became_peered":"2024-02-21T23:06:54.793190+0000","last_unstale":"2024-02-21T23:06:55.762972+0000","last_undegraded":"2024-02-21T23:06:55.762972+0000","last_fullsized":"2024-02-21T23:06:55.762972+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:16:44.498936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.25","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.800063+0000","last_change":"2024-02-21T23:06:54.791421+0000","last_active":"2024-02-21T23:06:54.800063+0000","last_peered":"2024-02-21T23:06:54.800063+0000","last_clean":"2024-02-21T23:06:54.800063+0000","last_became_active":"2024-02-21T23:06:54.790327+0000","last_became_peered":"2024-02-21T23:06:54.790327+0000","last_unstale":"2024-02-21T23:06:54.800063+0000","last_undegraded":"2024-02-21T23:06:54.800063+0000","last_fullsized":"2024-02-21T23:06:54.800063+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:10:18.069153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.24","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.799304+0000","last_change":"2024-02-21T23:06:54.796127+0000","last_active":"2024-02-21T23:06:54.799304+0000","last_peered":"2024-02-21T23:06:54.799304+0000","last_clean":"2024-02-21T23:06:54.799304+0000","last_became_active":"2024-02-21T23:06:54.795809+0000","last_became_peered":"2024-02-21T23:06:54.795809+0000","last_unstale":"2024-02-21T23:06:54.799304+0000","last_undegraded":"2024-02-21T23:06:54.799304+0000","last_fullsized":"2024-02-21T23:06:54.799304+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:30:18.559295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.23","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.801220+0000","last_change":"2024-02-21T23:06:54.791461+0000","last_active":"2024-02-21T23:06:54.801220+0000","last_peered":"2024-02-21T23:06:54.801220+0000","last_clean":"2024-02-21T23:06:54.801220+0000","last_became_active":"2024-02-21T23:06:54.788604+0000","last_became_peered":"2024-02-21T23:06:54.788604+0000","last_unstale":"2024-02-21T23:06:54.801220+0000","last_undegraded":"2024-02-21T23:06:54.801220+0000","last_fullsized":"2024-02-21T23:06:54.801220+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:50:49.398326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.22","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-21T23:06:55.761947+0000","last_change":"2024-02-21T23:06:54.790416+0000","last_active":"2024-02-21T23:06:55.761947+0000","last_peered":"2024-02-21T23:06:55.761947+0000","last_clean":"2024-02-21T23:06:55.761947+0000","last_became_active":"2024-02-21T23:06:54.790117+0000","last_became_peered":"2024-02-21T23:06:54.790117+0000","last_unstale":"2024-02-21T23:06:55.761947+0000","last_undegraded":"2024-02-21T23:06:55.761947+0000","last_fullsized":"2024-02-21T23:06:55.761947+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:43:15.037996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.21","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.805315+0000","last_change":"2024-02-21T23:06:54.801270+0000","last_active":"2024-02-21T23:06:54.805315+0000","last_peered":"2024-02-21T23:06:54.805315+0000","last_clean":"2024-02-21T23:06:54.805315+0000","last_became_active":"2024-02-21T23:06:54.801062+0000","last_became_peered":"2024-02-21T23:06:54.801062+0000","last_unstale":"2024-02-21T23:06:54.805315+0000","last_undegraded":"2024-02-21T23:06:54.805315+0000","last_fullsized":"2024-02-21T23:06:54.805315+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:00:52.018344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.20","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.803977+0000","last_change":"2024-02-21T23:06:54.795665+0000","last_active":"2024-02-21T23:06:54.803977+0000","last_peered":"2024-02-21T23:06:54.803977+0000","last_clean":"2024-02-21T23:06:54.803977+0000","last_became_active":"2024-02-21T23:06:54.794717+0000","last_became_peered":"2024-02-21T23:06:54.794717+0000","last_unstale":"2024-02-21T23:06:54.803977+0000","last_undegraded":"2024-02-21T23:06:54.803977+0000","last_fullsized":"2024-02-21T23:06:54.803977+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:20:44.653663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-21T23:06:55.757573+0000","last_change":"2024-02-21T23:06:54.796453+0000","last_active":"2024-02-21T23:06:55.757573+0000","last_peered":"2024-02-21T23:06:55.757573+0000","last_clean":"2024-02-21T23:06:55.757573+0000","last_became_active":"2024-02-21T23:06:54.790829+0000","last_became_peered":"2024-02-21T23:06:54.790829+0000","last_unstale":"2024-02-21T23:06:55.757573+0000","last_undegraded":"2024-02-21T23:06:55.757573+0000","last_fullsized":"2024-02-21T23:06:55.757573+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:47:30.278741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"97'4","reported_seq":18,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-21T23:06:55.761090+0000","last_change":"2024-02-21T23:06:53.748885+0000","last_active":"2024-02-21T23:06:55.761090+0000","last_peered":"2024-02-21T23:06:55.761090+0000","last_clean":"2024-02-21T23:06:55.761090+0000","last_became_active":"2024-02-21T23:06:53.748561+0000","last_became_peered":"2024-02-21T23:06:53.748561+0000","last_unstale":"2024-02-21T23:06:55.761090+0000","last_undegraded":"2024-02-21T23:06:55.761090+0000","last_fullsized":"2024-02-21T23:06:55.761090+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_clean_scrub_stamp":"2024-02-21T23:06:52.701904+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:51:31.391428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.819934+0000","last_change":"2024-02-21T23:06:54.815770+0000","last_active":"2024-02-21T23:06:54.819934+0000","last_peered":"2024-02-21T23:06:54.819934+0000","last_clean":"2024-02-21T23:06:54.819934+0000","last_became_active":"2024-02-21T23:06:54.815342+0000","last_became_peered":"2024-02-21T23:06:54.815342+0000","last_unstale":"2024-02-21T23:06:54.819934+0000","last_undegraded":"2024-02-21T23:06:54.819934+0000","last_fullsized":"2024-02-21T23:06:54.819934+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:43:29.861586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.1e","version":"97'1","reported_seq":13,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.818602+0000","last_change":"2024-02-21T23:06:53.747576+0000","last_active":"2024-02-21T23:06:54.818602+0000","last_peered":"2024-02-21T23:06:54.818602+0000","last_clean":"2024-02-21T23:06:54.818602+0000","last_became_active":"2024-02-21T23:06:53.746959+0000","last_became_peered":"2024-02-21T23:06:53.746959+0000","last_unstale":"2024-02-21T23:06:54.818602+0000","last_undegraded":"2024-02-21T23:06:54.818602+0000","last_fullsized":"2024-02-21T23:06:54.818602+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_clean_scrub_stamp":"2024-02-21T23:06:52.701904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:18:28.698148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-21T23:06:55.761799+0000","last_change":"2024-02-21T23:06:54.797713+0000","last_active":"2024-02-21T23:06:55.761799+0000","last_peered":"2024-02-21T23:06:55.761799+0000","last_clean":"2024-02-21T23:06:55.761799+0000","last_became_active":"2024-02-21T23:06:54.797279+0000","last_became_peered":"2024-02-21T23:06:54.797279+0000","last_unstale":"2024-02-21T23:06:55.761799+0000","last_undegraded":"2024-02-21T23:06:55.761799+0000","last_fullsized":"2024-02-21T23:06:55.761799+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:14:40.765415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1d","version":"97'2","reported_seq":16,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-21T23:06:55.762465+0000","last_change":"2024-02-21T23:06:53.755179+0000","last_active":"2024-02-21T23:06:55.762465+0000","last_peered":"2024-02-21T23:06:55.762465+0000","last_clean":"2024-02-21T23:06:55.762465+0000","last_became_active":"2024-02-21T23:06:53.752835+0000","last_became_peered":"2024-02-21T23:06:53.752835+0000","last_unstale":"2024-02-21T23:06:55.762465+0000","last_undegraded":"2024-02-21T23:06:55.762465+0000","last_fullsized":"2024-02-21T23:06:55.762465+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_clean_scrub_stamp":"2024-02-21T23:06:52.701904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:37:04.784457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.800227+0000","last_change":"2024-02-21T23:06:54.788244+0000","last_active":"2024-02-21T23:06:54.800227+0000","last_peered":"2024-02-21T23:06:54.800227+0000","last_clean":"2024-02-21T23:06:54.800227+0000","last_became_active":"2024-02-21T23:06:54.787875+0000","last_became_peered":"2024-02-21T23:06:54.787875+0000","last_unstale":"2024-02-21T23:06:54.800227+0000","last_undegraded":"2024-02-21T23:06:54.800227+0000","last_fullsized":"2024-02-21T23:06:54.800227+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:54:29.660083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1c","version":"97'2","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.821699+0000","last_change":"2024-02-21T23:06:53.744843+0000","last_active":"2024-02-21T23:06:54.821699+0000","last_peered":"2024-02-21T23:06:54.821699+0000","last_clean":"2024-02-21T23:06:54.821699+0000","last_became_active":"2024-02-21T23:06:53.744486+0000","last_became_peered":"2024-02-21T23:06:53.744486+0000","last_unstale":"2024-02-21T23:06:54.821699+0000","last_undegraded":"2024-02-21T23:06:54.821699+0000","last_fullsized":"2024-02-21T23:06:54.821699+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_clean_scrub_stamp":"2024-02-21T23:06:52.701904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:52:35.919454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-21T23:06:55.760463+0000","last_change":"2024-02-21T23:06:54.784963+0000","last_active":"2024-02-21T23:06:55.760463+0000","last_peered":"2024-02-21T23:06:55.760463+0000","last_clean":"2024-02-21T23:06:55.760463+0000","last_became_active":"2024-02-21T23:06:54.783318+0000","last_became_peered":"2024-02-21T23:06:54.783318+0000","last_unstale":"2024-02-21T23:06:55.760463+0000","last_undegraded":"2024-02-21T23:06:55.760463+0000","last_fullsized":"2024-02-21T23:06:55.760463+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:54:29.272490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1b","version":"97'1","reported_seq":13,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.816309+0000","last_change":"2024-02-21T23:06:53.751073+0000","last_active":"2024-02-21T23:06:54.816309+0000","last_peered":"2024-02-21T23:06:54.816309+0000","last_clean":"2024-02-21T23:06:54.816309+0000","last_became_active":"2024-02-21T23:06:53.749702+0000","last_became_peered":"2024-02-21T23:06:53.749702+0000","last_unstale":"2024-02-21T23:06:54.816309+0000","last_undegraded":"2024-02-21T23:06:54.816309+0000","last_fullsized":"2024-02-21T23:06:54.816309+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_clean_scrub_stamp":"2024-02-21T23:06:52.701904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:10:17.064616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-21T23:06:55.758965+0000","last_change":"2024-02-21T23:06:54.805456+0000","last_active":"2024-02-21T23:06:55.758965+0000","last_peered":"2024-02-21T23:06:55.758965+0000","last_clean":"2024-02-21T23:06:55.758965+0000","last_became_active":"2024-02-21T23:06:54.803352+0000","last_became_peered":"2024-02-21T23:06:54.803352+0000","last_unstale":"2024-02-21T23:06:55.758965+0000","last_undegraded":"2024-02-21T23:06:55.758965+0000","last_fullsized":"2024-02-21T23:06:55.758965+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:41:59.760739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"97'1","reported_seq":13,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.814272+0000","last_change":"2024-02-21T23:06:53.750209+0000","last_active":"2024-02-21T23:06:54.814272+0000","last_peered":"2024-02-21T23:06:54.814272+0000","last_clean":"2024-02-21T23:06:54.814272+0000","last_became_active":"2024-02-21T23:06:53.749948+0000","last_became_peered":"2024-02-21T23:06:53.749948+0000","last_unstale":"2024-02-21T23:06:54.814272+0000","last_undegraded":"2024-02-21T23:06:54.814272+0000","last_fullsized":"2024-02-21T23:06:54.814272+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_clean_scrub_stamp":"2024-02-21T23:06:52.701904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:16:54.105487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.802231+0000","last_change":"2024-02-21T23:06:54.799408+0000","last_active":"2024-02-21T23:06:54.802231+0000","last_peered":"2024-02-21T23:06:54.802231+0000","last_clean":"2024-02-21T23:06:54.802231+0000","last_became_active":"2024-02-21T23:06:54.798798+0000","last_became_peered":"2024-02-21T23:06:54.798798+0000","last_unstale":"2024-02-21T23:06:54.802231+0000","last_undegraded":"2024-02-21T23:06:54.802231+0000","last_fullsized":"2024-02-21T23:06:54.802231+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:42:34.044297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-21T23:06:55.761580+0000","last_change":"2024-02-21T23:06:53.751886+0000","last_active":"2024-02-21T23:06:55.761580+0000","last_peered":"2024-02-21T23:06:55.761580+0000","last_clean":"2024-02-21T23:06:55.761580+0000","last_became_active":"2024-02-21T23:06:53.750408+0000","last_became_peered":"2024-02-21T23:06:53.750408+0000","last_unstale":"2024-02-21T23:06:55.761580+0000","last_undegraded":"2024-02-21T23:06:55.761580+0000","last_fullsized":"2024-02-21T23:06:55.761580+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_clean_scrub_stamp":"2024-02-21T23:06:52.701904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:19:09.194924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-21T23:06:55.761189+0000","last_change":"2024-02-21T23:06:54.781428+0000","last_active":"2024-02-21T23:06:55.761189+0000","last_peered":"2024-02-21T23:06:55.761189+0000","last_clean":"2024-02-21T23:06:55.761189+0000","last_became_active":"2024-02-21T23:06:54.780157+0000","last_became_peered":"2024-02-21T23:06:54.780157+0000","last_unstale":"2024-02-21T23:06:55.761189+0000","last_undegraded":"2024-02-21T23:06:55.761189+0000","last_fullsized":"2024-02-21T23:06:55.761189+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:17:54.125344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-21T23:06:55.759989+0000","last_change":"2024-02-21T23:06:53.740420+0000","last_active":"2024-02-21T23:06:55.759989+0000","last_peered":"2024-02-21T23:06:55.759989+0000","last_clean":"2024-02-21T23:06:55.759989+0000","last_became_active":"2024-02-21T23:06:53.738614+0000","last_became_peered":"2024-02-21T23:06:53.738614+0000","last_unstale":"2024-02-21T23:06:55.759989+0000","last_undegraded":"2024-02-21T23:06:55.759989+0000","last_fullsized":"2024-02-21T23:06:55.759989+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_clean_scrub_stamp":"2024-02-21T23:06:52.701904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:43:14.033459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-21T23:06:55.757525+0000","last_change":"2024-02-21T23:06:54.794779+0000","last_active":"2024-02-21T23:06:55.757525+0000","last_peered":"2024-02-21T23:06:55.757525+0000","last_clean":"2024-02-21T23:06:55.757525+0000","last_became_active":"2024-02-21T23:06:54.790409+0000","last_became_peered":"2024-02-21T23:06:54.790409+0000","last_unstale":"2024-02-21T23:06:55.757525+0000","last_undegraded":"2024-02-21T23:06:55.757525+0000","last_fullsized":"2024-02-21T23:06:55.757525+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:59:29.839517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-21T23:06:55.762872+0000","last_change":"2024-02-21T23:06:53.753188+0000","last_active":"2024-02-21T23:06:55.762872+0000","last_peered":"2024-02-21T23:06:55.762872+0000","last_clean":"2024-02-21T23:06:55.762872+0000","last_became_active":"2024-02-21T23:06:53.752893+0000","last_became_peered":"2024-02-21T23:06:53.752893+0000","last_unstale":"2024-02-21T23:06:55.762872+0000","last_undegraded":"2024-02-21T23:06:55.762872+0000","last_fullsized":"2024-02-21T23:06:55.762872+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_clean_scrub_stamp":"2024-02-21T23:06:52.701904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:16:54.105487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.815589+0000","last_change":"2024-02-21T23:06:54.798146+0000","last_active":"2024-02-21T23:06:54.815589+0000","last_peered":"2024-02-21T23:06:54.815589+0000","last_clean":"2024-02-21T23:06:54.815589+0000","last_became_active":"2024-02-21T23:06:54.797837+0000","last_became_peered":"2024-02-21T23:06:54.797837+0000","last_unstale":"2024-02-21T23:06:54.815589+0000","last_undegraded":"2024-02-21T23:06:54.815589+0000","last_fullsized":"2024-02-21T23:06:54.815589+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:42:34.044297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-21T23:06:55.759002+0000","last_change":"2024-02-21T23:06:53.744963+0000","last_active":"2024-02-21T23:06:55.759002+0000","last_peered":"2024-02-21T23:06:55.759002+0000","last_clean":"2024-02-21T23:06:55.759002+0000","last_became_active":"2024-02-21T23:06:53.742193+0000","last_became_peered":"2024-02-21T23:06:53.742193+0000","last_unstale":"2024-02-21T23:06:55.759002+0000","last_undegraded":"2024-02-21T23:06:55.759002+0000","last_fullsized":"2024-02-21T23:06:55.759002+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_clean_scrub_stamp":"2024-02-21T23:06:52.701904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:45:17.720735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.820300+0000","last_change":"2024-02-21T23:06:54.818699+0000","last_active":"2024-02-21T23:06:54.820300+0000","last_peered":"2024-02-21T23:06:54.820300+0000","last_clean":"2024-02-21T23:06:54.820300+0000","last_became_active":"2024-02-21T23:06:54.818260+0000","last_became_peered":"2024-02-21T23:06:54.818260+0000","last_unstale":"2024-02-21T23:06:54.820300+0000","last_undegraded":"2024-02-21T23:06:54.820300+0000","last_fullsized":"2024-02-21T23:06:54.820300+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:04:38.617025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.15","version":"97'3","reported_seq":15,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.815789+0000","last_change":"2024-02-21T23:06:53.746719+0000","last_active":"2024-02-21T23:06:54.815789+0000","last_peered":"2024-02-21T23:06:54.815789+0000","last_clean":"2024-02-21T23:06:54.815789+0000","last_became_active":"2024-02-21T23:06:53.746425+0000","last_became_peered":"2024-02-21T23:06:53.746425+0000","last_unstale":"2024-02-21T23:06:54.815789+0000","last_undegraded":"2024-02-21T23:06:54.815789+0000","last_fullsized":"2024-02-21T23:06:54.815789+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_clean_scrub_stamp":"2024-02-21T23:06:52.701904+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:10:58.463061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.821262+0000","last_change":"2024-02-21T23:06:54.803344+0000","last_active":"2024-02-21T23:06:54.821262+0000","last_peered":"2024-02-21T23:06:54.821262+0000","last_clean":"2024-02-21T23:06:54.821262+0000","last_became_active":"2024-02-21T23:06:54.802485+0000","last_became_peered":"2024-02-21T23:06:54.802485+0000","last_unstale":"2024-02-21T23:06:54.821262+0000","last_undegraded":"2024-02-21T23:06:54.821262+0000","last_fullsized":"2024-02-21T23:06:54.821262+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:19:10.199462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.14","version":"97'6","reported_seq":18,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.820229+0000","last_change":"2024-02-21T23:06:53.756426+0000","last_active":"2024-02-21T23:06:54.820229+0000","last_peered":"2024-02-21T23:06:54.820229+0000","last_clean":"2024-02-21T23:06:54.820229+0000","last_became_active":"2024-02-21T23:06:53.756111+0000","last_became_peered":"2024-02-21T23:06:53.756111+0000","last_unstale":"2024-02-21T23:06:54.820229+0000","last_undegraded":"2024-02-21T23:06:54.820229+0000","last_fullsized":"2024-02-21T23:06:54.820229+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_clean_scrub_stamp":"2024-02-21T23:06:52.701904+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:52:35.919454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.800131+0000","last_change":"2024-02-21T23:06:54.792920+0000","last_active":"2024-02-21T23:06:54.800131+0000","last_peered":"2024-02-21T23:06:54.800131+0000","last_clean":"2024-02-21T23:06:54.800131+0000","last_became_active":"2024-02-21T23:06:54.792547+0000","last_became_peered":"2024-02-21T23:06:54.792547+0000","last_unstale":"2024-02-21T23:06:54.800131+0000","last_undegraded":"2024-02-21T23:06:54.800131+0000","last_fullsized":"2024-02-21T23:06:54.800131+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:52:28.706647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.13","version":"97'2","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.822186+0000","last_change":"2024-02-21T23:06:53.743679+0000","last_active":"2024-02-21T23:06:54.822186+0000","last_peered":"2024-02-21T23:06:54.822186+0000","last_clean":"2024-02-21T23:06:54.822186+0000","last_became_active":"2024-02-21T23:06:53.743301+0000","last_became_peered":"2024-02-21T23:06:53.743301+0000","last_unstale":"2024-02-21T23:06:54.822186+0000","last_undegraded":"2024-02-21T23:06:54.822186+0000","last_fullsized":"2024-02-21T23:06:54.822186+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_clean_scrub_stamp":"2024-02-21T23:06:52.701904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:36:31.611359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.805581+0000","last_change":"2024-02-21T23:06:54.799388+0000","last_active":"2024-02-21T23:06:54.805581+0000","last_peered":"2024-02-21T23:06:54.805581+0000","last_clean":"2024-02-21T23:06:54.805581+0000","last_became_active":"2024-02-21T23:06:54.795335+0000","last_became_peered":"2024-02-21T23:06:54.795335+0000","last_unstale":"2024-02-21T23:06:54.805581+0000","last_undegraded":"2024-02-21T23:06:54.805581+0000","last_fullsized":"2024-02-21T23:06:54.805581+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:41:39.165752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-21T23:06:55.761294+0000","last_change":"2024-02-21T23:06:53.744219+0000","last_active":"2024-02-21T23:06:55.761294+0000","last_peered":"2024-02-21T23:06:55.761294+0000","last_clean":"2024-02-21T23:06:55.761294+0000","last_became_active":"2024-02-21T23:06:53.743238+0000","last_became_peered":"2024-02-21T23:06:53.743238+0000","last_unstale":"2024-02-21T23:06:55.761294+0000","last_undegraded":"2024-02-21T23:06:55.761294+0000","last_fullsized":"2024-02-21T23:06:55.761294+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_clean_scrub_stamp":"2024-02-21T23:06:52.701904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:27:37.901660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.791395+0000","last_change":"2024-02-21T23:06:54.789691+0000","last_active":"2024-02-21T23:06:54.791395+0000","last_peered":"2024-02-21T23:06:54.791395+0000","last_clean":"2024-02-21T23:06:54.791395+0000","last_became_active":"2024-02-21T23:06:54.788584+0000","last_became_peered":"2024-02-21T23:06:54.788584+0000","last_unstale":"2024-02-21T23:06:54.791395+0000","last_undegraded":"2024-02-21T23:06:54.791395+0000","last_fullsized":"2024-02-21T23:06:54.791395+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:59:08.487373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.778789+0000","last_change":"2024-02-21T23:06:53.741935+0000","last_active":"2024-02-21T23:06:54.778789+0000","last_peered":"2024-02-21T23:06:54.778789+0000","last_clean":"2024-02-21T23:06:54.778789+0000","last_became_active":"2024-02-21T23:06:53.741662+0000","last_became_peered":"2024-02-21T23:06:53.741662+0000","last_unstale":"2024-02-21T23:06:54.778789+0000","last_undegraded":"2024-02-21T23:06:54.778789+0000","last_fullsized":"2024-02-21T23:06:54.778789+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_clean_scrub_stamp":"2024-02-21T23:06:52.701904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:47:03.976688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-21T23:06:55.759743+0000","last_change":"2024-02-21T23:06:54.799679+0000","last_active":"2024-02-21T23:06:55.759743+0000","last_peered":"2024-02-21T23:06:55.759743+0000","last_clean":"2024-02-21T23:06:55.759743+0000","last_became_active":"2024-02-21T23:06:54.799387+0000","last_became_peered":"2024-02-21T23:06:54.799387+0000","last_unstale":"2024-02-21T23:06:55.759743+0000","last_undegraded":"2024-02-21T23:06:55.759743+0000","last_fullsized":"2024-02-21T23:06:55.759743+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:14:21.111864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.10","version":"97'8","reported_seq":22,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-21T23:06:55.761500+0000","last_change":"2024-02-21T23:06:53.753404+0000","last_active":"2024-02-21T23:06:55.761500+0000","last_peered":"2024-02-21T23:06:55.761500+0000","last_clean":"2024-02-21T23:06:55.761500+0000","last_became_active":"2024-02-21T23:06:53.752630+0000","last_became_peered":"2024-02-21T23:06:53.752630+0000","last_unstale":"2024-02-21T23:06:55.761500+0000","last_undegraded":"2024-02-21T23:06:55.761500+0000","last_fullsized":"2024-02-21T23:06:55.761500+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_clean_scrub_stamp":"2024-02-21T23:06:52.701904+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:00:51.013807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.794424+0000","last_change":"2024-02-21T23:06:54.789052+0000","last_active":"2024-02-21T23:06:54.794424+0000","last_peered":"2024-02-21T23:06:54.794424+0000","last_clean":"2024-02-21T23:06:54.794424+0000","last_became_active":"2024-02-21T23:06:54.788752+0000","last_became_peered":"2024-02-21T23:06:54.788752+0000","last_unstale":"2024-02-21T23:06:54.794424+0000","last_undegraded":"2024-02-21T23:06:54.794424+0000","last_fullsized":"2024-02-21T23:06:54.794424+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:52:20.566957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.f","version":"97'1","reported_seq":13,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.813799+0000","last_change":"2024-02-21T23:06:53.745825+0000","last_active":"2024-02-21T23:06:54.813799+0000","last_peered":"2024-02-21T23:06:54.813799+0000","last_clean":"2024-02-21T23:06:54.813799+0000","last_became_active":"2024-02-21T23:06:53.745469+0000","last_became_peered":"2024-02-21T23:06:53.745469+0000","last_unstale":"2024-02-21T23:06:54.813799+0000","last_undegraded":"2024-02-21T23:06:54.813799+0000","last_fullsized":"2024-02-21T23:06:54.813799+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_clean_scrub_stamp":"2024-02-21T23:06:52.701904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:57:12.762293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.813600+0000","last_change":"2024-02-21T23:06:54.809364+0000","last_active":"2024-02-21T23:06:54.813600+0000","last_peered":"2024-02-21T23:06:54.813600+0000","last_clean":"2024-02-21T23:06:54.813600+0000","last_became_active":"2024-02-21T23:06:54.809080+0000","last_became_peered":"2024-02-21T23:06:54.809080+0000","last_unstale":"2024-02-21T23:06:54.813600+0000","last_undegraded":"2024-02-21T23:06:54.813600+0000","last_fullsized":"2024-02-21T23:06:54.813600+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:19:10.199462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"97'1","reported_seq":13,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.815884+0000","last_change":"2024-02-21T23:06:53.749878+0000","last_active":"2024-02-21T23:06:54.815884+0000","last_peered":"2024-02-21T23:06:54.815884+0000","last_clean":"2024-02-21T23:06:54.815884+0000","last_became_active":"2024-02-21T23:06:53.749348+0000","last_became_peered":"2024-02-21T23:06:53.749348+0000","last_unstale":"2024-02-21T23:06:54.815884+0000","last_undegraded":"2024-02-21T23:06:54.815884+0000","last_fullsized":"2024-02-21T23:06:54.815884+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_clean_scrub_stamp":"2024-02-21T23:06:52.701904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:52:27.702110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.809484+0000","last_change":"2024-02-21T23:06:54.792795+0000","last_active":"2024-02-21T23:06:54.809484+0000","last_peered":"2024-02-21T23:06:54.809484+0000","last_clean":"2024-02-21T23:06:54.809484+0000","last_became_active":"2024-02-21T23:06:54.792269+0000","last_became_peered":"2024-02-21T23:06:54.792269+0000","last_unstale":"2024-02-21T23:06:54.809484+0000","last_undegraded":"2024-02-21T23:06:54.809484+0000","last_fullsized":"2024-02-21T23:06:54.809484+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:30:18.559295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.773678+0000","last_change":"2024-02-21T23:06:53.758219+0000","last_active":"2024-02-21T23:06:54.773678+0000","last_peered":"2024-02-21T23:06:54.773678+0000","last_clean":"2024-02-21T23:06:54.773678+0000","last_became_active":"2024-02-21T23:06:53.757650+0000","last_became_peered":"2024-02-21T23:06:53.757650+0000","last_unstale":"2024-02-21T23:06:54.773678+0000","last_undegraded":"2024-02-21T23:06:54.773678+0000","last_fullsized":"2024-02-21T23:06:54.773678+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_clean_scrub_stamp":"2024-02-21T23:06:52.701904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:35:17.535584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-21T23:06:55.758580+0000","last_change":"2024-02-21T23:06:54.790172+0000","last_active":"2024-02-21T23:06:55.758580+0000","last_peered":"2024-02-21T23:06:55.758580+0000","last_clean":"2024-02-21T23:06:55.758580+0000","last_became_active":"2024-02-21T23:06:54.789866+0000","last_became_peered":"2024-02-21T23:06:54.789866+0000","last_unstale":"2024-02-21T23:06:55.758580+0000","last_undegraded":"2024-02-21T23:06:55.758580+0000","last_fullsized":"2024-02-21T23:06:55.758580+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:35:51.171071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"97'2","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.819341+0000","last_change":"2024-02-21T23:06:53.730541+0000","last_active":"2024-02-21T23:06:54.819341+0000","last_peered":"2024-02-21T23:06:54.819341+0000","last_clean":"2024-02-21T23:06:54.819341+0000","last_became_active":"2024-02-21T23:06:53.730073+0000","last_became_peered":"2024-02-21T23:06:53.730073+0000","last_unstale":"2024-02-21T23:06:54.819341+0000","last_undegraded":"2024-02-21T23:06:54.819341+0000","last_fullsized":"2024-02-21T23:06:54.819341+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_clean_scrub_stamp":"2024-02-21T23:06:52.701904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:30:05.360823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-21T23:06:55.758870+0000","last_change":"2024-02-21T23:06:54.808993+0000","last_active":"2024-02-21T23:06:55.758870+0000","last_peered":"2024-02-21T23:06:55.758870+0000","last_clean":"2024-02-21T23:06:55.758870+0000","last_became_active":"2024-02-21T23:06:54.808704+0000","last_became_peered":"2024-02-21T23:06:54.808704+0000","last_unstale":"2024-02-21T23:06:55.758870+0000","last_undegraded":"2024-02-21T23:06:55.758870+0000","last_fullsized":"2024-02-21T23:06:55.758870+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:43:13.678228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.777961+0000","last_change":"2024-02-21T23:06:53.740354+0000","last_active":"2024-02-21T23:06:54.777961+0000","last_peered":"2024-02-21T23:06:54.777961+0000","last_clean":"2024-02-21T23:06:54.777961+0000","last_became_active":"2024-02-21T23:06:53.740058+0000","last_became_peered":"2024-02-21T23:06:53.740058+0000","last_unstale":"2024-02-21T23:06:54.777961+0000","last_undegraded":"2024-02-21T23:06:54.777961+0000","last_fullsized":"2024-02-21T23:06:54.777961+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_clean_scrub_stamp":"2024-02-21T23:06:52.701904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:28:57.064260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-21T23:06:55.761181+0000","last_change":"2024-02-21T23:06:54.791521+0000","last_active":"2024-02-21T23:06:55.761181+0000","last_peered":"2024-02-21T23:06:55.761181+0000","last_clean":"2024-02-21T23:06:55.761181+0000","last_became_active":"2024-02-21T23:06:54.791181+0000","last_became_peered":"2024-02-21T23:06:54.791181+0000","last_unstale":"2024-02-21T23:06:55.761181+0000","last_undegraded":"2024-02-21T23:06:55.761181+0000","last_fullsized":"2024-02-21T23:06:55.761181+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:54:37.184551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.787113+0000","last_change":"2024-02-21T23:06:53.756174+0000","last_active":"2024-02-21T23:06:54.787113+0000","last_peered":"2024-02-21T23:06:54.787113+0000","last_clean":"2024-02-21T23:06:54.787113+0000","last_became_active":"2024-02-21T23:06:53.755343+0000","last_became_peered":"2024-02-21T23:06:53.755343+0000","last_unstale":"2024-02-21T23:06:54.787113+0000","last_undegraded":"2024-02-21T23:06:54.787113+0000","last_fullsized":"2024-02-21T23:06:54.787113+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_clean_scrub_stamp":"2024-02-21T23:06:52.701904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:29:23.260556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.820287+0000","last_change":"2024-02-21T23:06:54.817397+0000","last_active":"2024-02-21T23:06:54.820287+0000","last_peered":"2024-02-21T23:06:54.820287+0000","last_clean":"2024-02-21T23:06:54.820287+0000","last_became_active":"2024-02-21T23:06:54.817090+0000","last_became_peered":"2024-02-21T23:06:54.817090+0000","last_unstale":"2024-02-21T23:06:54.820287+0000","last_undegraded":"2024-02-21T23:06:54.820287+0000","last_fullsized":"2024-02-21T23:06:54.820287+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:41:48.818494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.779569+0000","last_change":"2024-02-21T23:06:53.725221+0000","last_active":"2024-02-21T23:06:54.779569+0000","last_peered":"2024-02-21T23:06:54.779569+0000","last_clean":"2024-02-21T23:06:54.779569+0000","last_became_active":"2024-02-21T23:06:53.724900+0000","last_became_peered":"2024-02-21T23:06:53.724900+0000","last_unstale":"2024-02-21T23:06:54.779569+0000","last_undegraded":"2024-02-21T23:06:54.779569+0000","last_fullsized":"2024-02-21T23:06:54.779569+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_clean_scrub_stamp":"2024-02-21T23:06:52.701904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:04:11.157605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-21T23:06:55.757405+0000","last_change":"2024-02-21T23:06:54.772130+0000","last_active":"2024-02-21T23:06:55.757405+0000","last_peered":"2024-02-21T23:06:55.757405+0000","last_clean":"2024-02-21T23:06:55.757405+0000","last_became_active":"2024-02-21T23:06:54.770534+0000","last_became_peered":"2024-02-21T23:06:54.770534+0000","last_unstale":"2024-02-21T23:06:55.757405+0000","last_undegraded":"2024-02-21T23:06:55.757405+0000","last_fullsized":"2024-02-21T23:06:55.757405+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:59:55.738424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.7","version":"97'2","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.821774+0000","last_change":"2024-02-21T23:06:53.744593+0000","last_active":"2024-02-21T23:06:54.821774+0000","last_peered":"2024-02-21T23:06:54.821774+0000","last_clean":"2024-02-21T23:06:54.821774+0000","last_became_active":"2024-02-21T23:06:53.744310+0000","last_became_peered":"2024-02-21T23:06:53.744310+0000","last_unstale":"2024-02-21T23:06:54.821774+0000","last_undegraded":"2024-02-21T23:06:54.821774+0000","last_fullsized":"2024-02-21T23:06:54.821774+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_clean_scrub_stamp":"2024-02-21T23:06:52.701904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:57:12.762293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-21T23:06:55.761669+0000","last_change":"2024-02-21T23:06:54.776671+0000","last_active":"2024-02-21T23:06:55.761669+0000","last_peered":"2024-02-21T23:06:55.761669+0000","last_clean":"2024-02-21T23:06:55.761669+0000","last_became_active":"2024-02-21T23:06:54.775959+0000","last_became_peered":"2024-02-21T23:06:54.775959+0000","last_unstale":"2024-02-21T23:06:55.761669+0000","last_undegraded":"2024-02-21T23:06:55.761669+0000","last_fullsized":"2024-02-21T23:06:55.761669+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:40:27.170195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-21T23:06:55.759080+0000","last_change":"2024-02-21T23:06:53.753025+0000","last_active":"2024-02-21T23:06:55.759080+0000","last_peered":"2024-02-21T23:06:55.759080+0000","last_clean":"2024-02-21T23:06:55.759080+0000","last_became_active":"2024-02-21T23:06:53.752597+0000","last_became_peered":"2024-02-21T23:06:53.752597+0000","last_unstale":"2024-02-21T23:06:55.759080+0000","last_undegraded":"2024-02-21T23:06:55.759080+0000","last_fullsized":"2024-02-21T23:06:55.759080+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_clean_scrub_stamp":"2024-02-21T23:06:52.701904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:54:36.180014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-21T23:06:55.761082+0000","last_change":"2024-02-21T23:06:54.791481+0000","last_active":"2024-02-21T23:06:55.761082+0000","last_peered":"2024-02-21T23:06:55.761082+0000","last_clean":"2024-02-21T23:06:55.761082+0000","last_became_active":"2024-02-21T23:06:54.791200+0000","last_became_peered":"2024-02-21T23:06:54.791200+0000","last_unstale":"2024-02-21T23:06:55.761082+0000","last_undegraded":"2024-02-21T23:06:55.761082+0000","last_fullsized":"2024-02-21T23:06:55.761082+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:45:18.725272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.789091+0000","last_change":"2024-02-21T23:06:53.748756+0000","last_active":"2024-02-21T23:06:54.789091+0000","last_peered":"2024-02-21T23:06:54.789091+0000","last_clean":"2024-02-21T23:06:54.789091+0000","last_became_active":"2024-02-21T23:06:53.748409+0000","last_became_peered":"2024-02-21T23:06:53.748409+0000","last_unstale":"2024-02-21T23:06:54.789091+0000","last_undegraded":"2024-02-21T23:06:54.789091+0000","last_fullsized":"2024-02-21T23:06:54.789091+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_clean_scrub_stamp":"2024-02-21T23:06:52.701904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:04:57.981362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.800321+0000","last_change":"2024-02-21T23:06:54.795666+0000","last_active":"2024-02-21T23:06:54.800321+0000","last_peered":"2024-02-21T23:06:54.800321+0000","last_clean":"2024-02-21T23:06:54.800321+0000","last_became_active":"2024-02-21T23:06:54.795393+0000","last_became_peered":"2024-02-21T23:06:54.795393+0000","last_unstale":"2024-02-21T23:06:54.800321+0000","last_undegraded":"2024-02-21T23:06:54.800321+0000","last_fullsized":"2024-02-21T23:06:54.800321+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:34:18.224729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.776743+0000","last_change":"2024-02-21T23:06:53.750570+0000","last_active":"2024-02-21T23:06:54.776743+0000","last_peered":"2024-02-21T23:06:54.776743+0000","last_clean":"2024-02-21T23:06:54.776743+0000","last_became_active":"2024-02-21T23:06:53.750291+0000","last_became_peered":"2024-02-21T23:06:53.750291+0000","last_unstale":"2024-02-21T23:06:54.776743+0000","last_undegraded":"2024-02-21T23:06:54.776743+0000","last_fullsized":"2024-02-21T23:06:54.776743+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_clean_scrub_stamp":"2024-02-21T23:06:52.701904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:16:26.607536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.820323+0000","last_change":"2024-02-21T23:06:54.817197+0000","last_active":"2024-02-21T23:06:54.820323+0000","last_peered":"2024-02-21T23:06:54.820323+0000","last_clean":"2024-02-21T23:06:54.820323+0000","last_became_active":"2024-02-21T23:06:54.815274+0000","last_became_peered":"2024-02-21T23:06:54.815274+0000","last_unstale":"2024-02-21T23:06:54.820323+0000","last_undegraded":"2024-02-21T23:06:54.820323+0000","last_fullsized":"2024-02-21T23:06:54.820323+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:02:20.329704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-21T23:06:55.762187+0000","last_change":"2024-02-21T23:06:53.750107+0000","last_active":"2024-02-21T23:06:55.762187+0000","last_peered":"2024-02-21T23:06:55.762187+0000","last_clean":"2024-02-21T23:06:55.762187+0000","last_became_active":"2024-02-21T23:06:53.749809+0000","last_became_peered":"2024-02-21T23:06:53.749809+0000","last_unstale":"2024-02-21T23:06:55.762187+0000","last_undegraded":"2024-02-21T23:06:55.762187+0000","last_fullsized":"2024-02-21T23:06:55.762187+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_clean_scrub_stamp":"2024-02-21T23:06:52.701904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:35:46.125860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.819848+0000","last_change":"2024-02-21T23:06:54.815320+0000","last_active":"2024-02-21T23:06:54.819848+0000","last_peered":"2024-02-21T23:06:54.819848+0000","last_clean":"2024-02-21T23:06:54.819848+0000","last_became_active":"2024-02-21T23:06:54.815007+0000","last_became_peered":"2024-02-21T23:06:54.815007+0000","last_unstale":"2024-02-21T23:06:54.819848+0000","last_undegraded":"2024-02-21T23:06:54.819848+0000","last_fullsized":"2024-02-21T23:06:54.819848+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:12:00.256298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.778304+0000","last_change":"2024-02-21T23:06:53.745467+0000","last_active":"2024-02-21T23:06:54.778304+0000","last_peered":"2024-02-21T23:06:54.778304+0000","last_clean":"2024-02-21T23:06:54.778304+0000","last_became_active":"2024-02-21T23:06:53.745166+0000","last_became_peered":"2024-02-21T23:06:53.745166+0000","last_unstale":"2024-02-21T23:06:54.778304+0000","last_undegraded":"2024-02-21T23:06:54.778304+0000","last_fullsized":"2024-02-21T23:06:54.778304+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_clean_scrub_stamp":"2024-02-21T23:06:52.701904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:20:59.579104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.799463+0000","last_change":"2024-02-21T23:06:54.794516+0000","last_active":"2024-02-21T23:06:54.799463+0000","last_peered":"2024-02-21T23:06:54.799463+0000","last_clean":"2024-02-21T23:06:54.799463+0000","last_became_active":"2024-02-21T23:06:54.794169+0000","last_became_peered":"2024-02-21T23:06:54.794169+0000","last_unstale":"2024-02-21T23:06:54.799463+0000","last_undegraded":"2024-02-21T23:06:54.799463+0000","last_fullsized":"2024-02-21T23:06:54.799463+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:34:18.224729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.787272+0000","last_change":"2024-02-21T23:06:53.749902+0000","last_active":"2024-02-21T23:06:54.787272+0000","last_peered":"2024-02-21T23:06:54.787272+0000","last_clean":"2024-02-21T23:06:54.787272+0000","last_became_active":"2024-02-21T23:06:53.749613+0000","last_became_peered":"2024-02-21T23:06:53.749613+0000","last_unstale":"2024-02-21T23:06:54.787272+0000","last_undegraded":"2024-02-21T23:06:54.787272+0000","last_fullsized":"2024-02-21T23:06:54.787272+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_clean_scrub_stamp":"2024-02-21T23:06:52.701904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:30:59.747020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.799955+0000","last_change":"2024-02-21T23:06:54.792685+0000","last_active":"2024-02-21T23:06:54.799955+0000","last_peered":"2024-02-21T23:06:54.799955+0000","last_clean":"2024-02-21T23:06:54.799955+0000","last_became_active":"2024-02-21T23:06:54.792304+0000","last_became_peered":"2024-02-21T23:06:54.792304+0000","last_unstale":"2024-02-21T23:06:54.799955+0000","last_undegraded":"2024-02-21T23:06:54.799955+0000","last_fullsized":"2024-02-21T23:06:54.799955+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:10:59.467598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.779767+0000","last_change":"2024-02-21T23:06:53.742463+0000","last_active":"2024-02-21T23:06:54.779767+0000","last_peered":"2024-02-21T23:06:54.779767+0000","last_clean":"2024-02-21T23:06:54.779767+0000","last_became_active":"2024-02-21T23:06:53.742119+0000","last_became_peered":"2024-02-21T23:06:53.742119+0000","last_unstale":"2024-02-21T23:06:54.779767+0000","last_undegraded":"2024-02-21T23:06:54.779767+0000","last_fullsized":"2024-02-21T23:06:54.779767+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_clean_scrub_stamp":"2024-02-21T23:06:52.701904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:22:20.809516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.810998+0000","last_change":"2024-02-21T23:06:54.802344+0000","last_active":"2024-02-21T23:06:54.810998+0000","last_peered":"2024-02-21T23:06:54.810998+0000","last_clean":"2024-02-21T23:06:54.810998+0000","last_became_active":"2024-02-21T23:06:54.802038+0000","last_became_peered":"2024-02-21T23:06:54.802038+0000","last_unstale":"2024-02-21T23:06:54.810998+0000","last_undegraded":"2024-02-21T23:06:54.810998+0000","last_fullsized":"2024-02-21T23:06:54.810998+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:40:27.170195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.e","version":"97'2","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.818923+0000","last_change":"2024-02-21T23:06:53.736840+0000","last_active":"2024-02-21T23:06:54.818923+0000","last_peered":"2024-02-21T23:06:54.818923+0000","last_clean":"2024-02-21T23:06:54.818923+0000","last_became_active":"2024-02-21T23:06:53.736495+0000","last_became_peered":"2024-02-21T23:06:53.736495+0000","last_unstale":"2024-02-21T23:06:54.818923+0000","last_undegraded":"2024-02-21T23:06:54.818923+0000","last_fullsized":"2024-02-21T23:06:54.818923+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_clean_scrub_stamp":"2024-02-21T23:06:52.701904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:37:50.331474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":315,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.768546+0000","last_change":"2024-02-21T23:02:32.289335+0000","last_active":"2024-02-21T23:06:54.768546+0000","last_peered":"2024-02-21T23:06:54.768546+0000","last_clean":"2024-02-21T23:06:54.768546+0000","last_became_active":"2024-02-21T23:02:32.286913+0000","last_became_peered":"2024-02-21T23:02:32.286913+0000","last_unstale":"2024-02-21T23:06:54.768546+0000","last_undegraded":"2024-02-21T23:06:54.768546+0000","last_fullsized":"2024-02-21T23:06:54.768546+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_clean_scrub_stamp":"2024-02-21T23:02:31.172569+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:36:23.987374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":16,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574903,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32436,"kb_used_data":1248,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339404,"statfs":{"total":96636764160,"available":96603549696,"internally_reserved":0,"allocated":1277952,"data_stored":1178552,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574904,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32376,"kb_used_data":1196,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339464,"statfs":{"total":96636764160,"available":96603611136,"internally_reserved":0,"allocated":1224704,"data_stored":590240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574903,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32360,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339480,"statfs":{"total":96636764160,"available":96603627520,"internally_reserved":0,"allocated":1204224,"data_stored":588024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574904,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32368,"kb_used_data":1184,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339472,"statfs":{"total":96636764160,"available":96603619328,"internally_reserved":0,"allocated":1212416,"data_stored":588116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574904,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32364,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339476,"statfs":{"total":96636764160,"available":96603623424,"internally_reserved":0,"allocated":1208320,"data_stored":588606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574905,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32364,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339476,"statfs":{"total":96636764160,"available":96603623424,"internally_reserved":0,"allocated":1208320,"data_stored":588058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574903,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32368,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339472,"statfs":{"total":96636764160,"available":96603619328,"internally_reserved":0,"allocated":1208320,"data_stored":588058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574903,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32448,"kb_used_data":1268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339392,"statfs":{"total":96636764160,"available":96603537408,"internally_reserved":0,"allocated":1298432,"data_stored":1186786,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-21T23:06:59.318 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-21T23:06:59.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.597+0000 7f8a77400640 1 Processor -- start 2024-02-21T23:06:59.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.597+0000 7f8a77400640 1 -- start start 2024-02-21T23:06:59.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.599+0000 7f8a77400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8a700af970 0x7f8a700c7350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:59.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.599+0000 7f8a77400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8a70155ca0 0x7f8a7006cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:59.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.599+0000 7f8a77400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a700c5e60 con 0x7f8a700c7890 2024-02-21T23:06:59.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.599+0000 7f8a77400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a701561a0 con 0x7f8a70155ca0 2024-02-21T23:06:59.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.599+0000 7f8a77400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a700c6140 con 0x7f8a700af970 2024-02-21T23:06:59.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.599+0000 7f8a6fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8a70155ca0 0x7f8a7006cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:59.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.599+0000 7f8a6fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8a70155ca0 0x7f8a7006cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47612/0 (socket says 172.21.15.28:47612) 2024-02-21T23:06:59.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.599+0000 7f8a6fe00640 1 -- 172.21.15.28:0/3298843729 learned_addr learned my addr 172.21.15.28:0/3298843729 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:59.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.599+0000 7f8a75400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f8a700c7890 0x7f8a7006d3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34610/0 (socket says 172.21.15.28:34610) 2024-02-21T23:06:59.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.599+0000 7f8a74a00640 1 --2- 172.21.15.28:0/3298843729 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8a700af970 0x7f8a700c7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:59.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.600+0000 7f8a6fe00640 1 -- 172.21.15.28:0/3298843729 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8a700af970 msgr2=0x7f8a700c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:06:59.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.600+0000 7f8a6fe00640 1 --2- 172.21.15.28:0/3298843729 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8a700af970 0x7f8a700c7350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:59.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.600+0000 7f8a6fe00640 1 -- 172.21.15.28:0/3298843729 >> v1:172.21.15.28:6789/0 conn(0x7f8a700c7890 legacy=0x7f8a7006d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:59.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.600+0000 7f8a6fe00640 1 -- 172.21.15.28:0/3298843729 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a700c6420 con 0x7f8a70155ca0 2024-02-21T23:06:59.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.601+0000 7f8a6fe00640 1 --2- 172.21.15.28:0/3298843729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a70155ca0 0x7f8a7006cea0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c009870 tx=0x7f8a5c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39eaeeb476101c62 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:59.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.601+0000 7f8a6f400640 1 -- 172.21.15.28:0/3298843729 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a5c035020 con 0x7f8a70155ca0 2024-02-21T23:06:59.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.601+0000 7f8a6f400640 1 -- 172.21.15.28:0/3298843729 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a5c031640 con 0x7f8a70155ca0 2024-02-21T23:06:59.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.601+0000 7f8a6f400640 1 -- 172.21.15.28:0/3298843729 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a5c02d040 con 0x7f8a70155ca0 2024-02-21T23:06:59.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.602+0000 7f8a77400640 1 -- 172.21.15.28:0/3298843729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a70155ca0 msgr2=0x7f8a7006cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:59.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.602+0000 7f8a77400640 1 --2- 172.21.15.28:0/3298843729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a70155ca0 0x7f8a7006cea0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c009870 tx=0x7f8a5c02b8f0 comp rx=0 tx=0).stop 2024-02-21T23:06:59.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.602+0000 7f8a77400640 1 -- 172.21.15.28:0/3298843729 shutdown_connections 2024-02-21T23:06:59.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.602+0000 7f8a77400640 1 --2- 172.21.15.28:0/3298843729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a70155ca0 0x7f8a7006cea0 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:59.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.602+0000 7f8a77400640 1 --2- 172.21.15.28:0/3298843729 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8a700af970 0x7f8a700c7350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:59.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.602+0000 7f8a77400640 1 -- 172.21.15.28:0/3298843729 >> 172.21.15.28:0/3298843729 conn(0x7f8a700725a0 msgr2=0x7f8a700729b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:59.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.602+0000 7f8a77400640 1 -- 172.21.15.28:0/3298843729 shutdown_connections 2024-02-21T23:06:59.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.602+0000 7f8a77400640 1 -- 172.21.15.28:0/3298843729 wait complete. 2024-02-21T23:06:59.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.604+0000 7f8a77400640 1 Processor -- start 2024-02-21T23:06:59.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.604+0000 7f8a77400640 1 -- start start 2024-02-21T23:06:59.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.604+0000 7f8a77400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8a700af970 0x7f8a7014ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:59.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.605+0000 7f8a77400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8a700c7890 0x7f8a701e04d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:59.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.605+0000 7f8a77400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a70155ca0 0x7f8a701ee5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:59.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.605+0000 7f8a77400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a700afd50 con 0x7f8a70155ca0 2024-02-21T23:06:59.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.605+0000 7f8a77400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a7006daf0 con 0x7f8a700c7890 2024-02-21T23:06:59.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.605+0000 7f8a77400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a700afed0 con 0x7f8a700af970 2024-02-21T23:06:59.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.605+0000 7f8a75400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a70155ca0 0x7f8a701ee5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:59.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.605+0000 7f8a75400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a70155ca0 0x7f8a701ee5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47618/0 (socket says 172.21.15.28:47618) 2024-02-21T23:06:59.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.605+0000 7f8a75400640 1 -- 172.21.15.28:0/2980771823 learned_addr learned my addr 172.21.15.28:0/2980771823 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:06:59.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.605+0000 7f8a74a00640 1 --2- 172.21.15.28:0/2980771823 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8a700af970 0x7f8a7014ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:59.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.605+0000 7f8a75400640 1 -- 172.21.15.28:0/2980771823 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8a700af970 msgr2=0x7f8a7014ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:59.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.605+0000 7f8a6fe00640 1 --2- 172.21.15.28:0/2980771823 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8a700c7890 0x7f8a701e04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:59.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.605+0000 7f8a75400640 1 --2- 172.21.15.28:0/2980771823 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8a700af970 0x7f8a7014ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:59.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.605+0000 7f8a75400640 1 -- 172.21.15.28:0/2980771823 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8a700c7890 msgr2=0x7f8a701e04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:59.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.605+0000 7f8a75400640 1 --2- 172.21.15.28:0/2980771823 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8a700c7890 0x7f8a701e04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:59.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.605+0000 7f8a75400640 1 -- 172.21.15.28:0/2980771823 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a70158fb0 con 0x7f8a70155ca0 2024-02-21T23:06:59.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.606+0000 7f8a75400640 1 --2- 172.21.15.28:0/2980771823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a70155ca0 0x7f8a701ee5f0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f8a6800c7b0 tx=0x7f8a6800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:59.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.606+0000 7f8a6cc00640 1 -- 172.21.15.28:0/2980771823 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a68017070 con 0x7f8a70155ca0 2024-02-21T23:06:59.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.606+0000 7f8a6cc00640 1 -- 172.21.15.28:0/2980771823 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a6800a640 con 0x7f8a70155ca0 2024-02-21T23:06:59.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.606+0000 7f8a6cc00640 1 -- 172.21.15.28:0/2980771823 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a6800a9f0 con 0x7f8a70155ca0 2024-02-21T23:06:59.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.606+0000 7f8a77400640 1 -- 172.21.15.28:0/2980771823 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a70159290 con 0x7f8a70155ca0 2024-02-21T23:06:59.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.606+0000 7f8a77400640 1 -- 172.21.15.28:0/2980771823 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a700cb330 con 0x7f8a70155ca0 2024-02-21T23:06:59.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.608+0000 7f8a77400640 1 -- 172.21.15.28:0/2980771823 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a70068640 con 0x7f8a70155ca0 2024-02-21T23:06:59.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.609+0000 7f8a6cc00640 1 -- 172.21.15.28:0/2980771823 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8a6801a070 con 0x7f8a70155ca0 2024-02-21T23:06:59.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.610+0000 7f8a6cc00640 1 --2- 172.21.15.28:0/2980771823 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8a5006b4d0 0x7f8a5006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:06:59.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.610+0000 7f8a6cc00640 1 -- 172.21.15.28:0/2980771823 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f8a680898d0 con 0x7f8a70155ca0 2024-02-21T23:06:59.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.611+0000 7f8a74a00640 1 --2- 172.21.15.28:0/2980771823 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8a5006b4d0 0x7f8a5006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:06:59.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.612+0000 7f8a74a00640 1 --2- 172.21.15.28:0/2980771823 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8a5006b4d0 0x7f8a5006d990 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f8a64006fd0 tx=0x7f8a64008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:06:59.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.615+0000 7f8a6cc00640 1 -- 172.21.15.28:0/2980771823 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f8a68057f60 con 0x7f8a70155ca0 2024-02-21T23:06:59.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.813+0000 7f8a77400640 1 -- 172.21.15.28:0/2980771823 --> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8a700c48b0 con 0x7f8a5006b4d0 2024-02-21T23:06:59.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.833+0000 7f8a6cc00640 1 -- 172.21.15.28:0/2980771823 <== mgr.4101 v2:172.21.15.81:6832/1848538577 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252115 (secure 0 0 0) 0x7f8a700c48b0 con 0x7f8a5006b4d0 2024-02-21T23:06:59.825 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:06:59.826 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-21T23:06:59.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.838+0000 7f8a77400640 1 -- 172.21.15.28:0/2980771823 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8a5006b4d0 msgr2=0x7f8a5006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:59.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.838+0000 7f8a77400640 1 --2- 172.21.15.28:0/2980771823 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8a5006b4d0 0x7f8a5006d990 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f8a64006fd0 tx=0x7f8a64008040 comp rx=0 tx=0).stop 2024-02-21T23:06:59.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.838+0000 7f8a77400640 1 -- 172.21.15.28:0/2980771823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a70155ca0 msgr2=0x7f8a701ee5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:06:59.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.838+0000 7f8a77400640 1 --2- 172.21.15.28:0/2980771823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a70155ca0 0x7f8a701ee5f0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f8a6800c7b0 tx=0x7f8a6800cc70 comp rx=0 tx=0).stop 2024-02-21T23:06:59.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.838+0000 7f8a77400640 1 -- 172.21.15.28:0/2980771823 shutdown_connections 2024-02-21T23:06:59.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.838+0000 7f8a77400640 1 --2- 172.21.15.28:0/2980771823 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8a5006b4d0 0x7f8a5006d990 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:59.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.838+0000 7f8a77400640 1 --2- 172.21.15.28:0/2980771823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a70155ca0 0x7f8a701ee5f0 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:59.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.838+0000 7f8a77400640 1 --2- 172.21.15.28:0/2980771823 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8a700c7890 0x7f8a701e04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:59.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.838+0000 7f8a77400640 1 --2- 172.21.15.28:0/2980771823 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8a700af970 0x7f8a7014ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:06:59.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.838+0000 7f8a77400640 1 -- 172.21.15.28:0/2980771823 >> 172.21.15.28:0/2980771823 conn(0x7f8a700725a0 msgr2=0x7f8a70151440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:06:59.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.838+0000 7f8a77400640 1 -- 172.21.15.28:0/2980771823 shutdown_connections 2024-02-21T23:06:59.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:06:59.839+0000 7f8a77400640 1 -- 172.21.15.28:0/2980771823 wait complete. 2024-02-21T23:06:59.845 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":225,"stamp":"2024-02-21T23:06:58.758555+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":79,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":259084,"kb_used_data":9612,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754715636,"statfs":{"total":773094113280,"available":772828811264,"internally_reserved":0,"allocated":9842688,"data_stored":5896440,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002335"},"pg_stats":[{"pgid":"16.3f","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-21T23:06:55.759282+0000","last_change":"2024-02-21T23:06:54.782740+0000","last_active":"2024-02-21T23:06:55.759282+0000","last_peered":"2024-02-21T23:06:55.759282+0000","last_clean":"2024-02-21T23:06:55.759282+0000","last_became_active":"2024-02-21T23:06:54.782364+0000","last_became_peered":"2024-02-21T23:06:54.782364+0000","last_unstale":"2024-02-21T23:06:55.759282+0000","last_undegraded":"2024-02-21T23:06:55.759282+0000","last_fullsized":"2024-02-21T23:06:55.759282+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:42:22.612435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3e","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.797843+0000","last_change":"2024-02-21T23:06:54.790581+0000","last_active":"2024-02-21T23:06:54.797843+0000","last_peered":"2024-02-21T23:06:54.797843+0000","last_clean":"2024-02-21T23:06:54.797843+0000","last_became_active":"2024-02-21T23:06:54.790272+0000","last_became_peered":"2024-02-21T23:06:54.790272+0000","last_unstale":"2024-02-21T23:06:54.797843+0000","last_undegraded":"2024-02-21T23:06:54.797843+0000","last_fullsized":"2024-02-21T23:06:54.797843+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:16:27.612073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.3d","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-21T23:06:55.758640+0000","last_change":"2024-02-21T23:06:54.805419+0000","last_active":"2024-02-21T23:06:55.758640+0000","last_peered":"2024-02-21T23:06:55.758640+0000","last_clean":"2024-02-21T23:06:55.758640+0000","last_became_active":"2024-02-21T23:06:54.801524+0000","last_became_peered":"2024-02-21T23:06:54.801524+0000","last_unstale":"2024-02-21T23:06:55.758640+0000","last_undegraded":"2024-02-21T23:06:55.758640+0000","last_fullsized":"2024-02-21T23:06:55.758640+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:32:08.968265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3c","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-21T23:06:55.760276+0000","last_change":"2024-02-21T23:06:54.788418+0000","last_active":"2024-02-21T23:06:55.760276+0000","last_peered":"2024-02-21T23:06:55.760276+0000","last_clean":"2024-02-21T23:06:55.760276+0000","last_became_active":"2024-02-21T23:06:54.782881+0000","last_became_peered":"2024-02-21T23:06:54.782881+0000","last_unstale":"2024-02-21T23:06:55.760276+0000","last_undegraded":"2024-02-21T23:06:55.760276+0000","last_fullsized":"2024-02-21T23:06:55.760276+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:54:58.664903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.3b","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.817510+0000","last_change":"2024-02-21T23:06:54.803107+0000","last_active":"2024-02-21T23:06:54.817510+0000","last_peered":"2024-02-21T23:06:54.817510+0000","last_clean":"2024-02-21T23:06:54.817510+0000","last_became_active":"2024-02-21T23:06:54.802533+0000","last_became_peered":"2024-02-21T23:06:54.802533+0000","last_unstale":"2024-02-21T23:06:54.817510+0000","last_undegraded":"2024-02-21T23:06:54.817510+0000","last_fullsized":"2024-02-21T23:06:54.817510+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:00:52.018344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.3a","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.816169+0000","last_change":"2024-02-21T23:06:54.811691+0000","last_active":"2024-02-21T23:06:54.816169+0000","last_peered":"2024-02-21T23:06:54.816169+0000","last_clean":"2024-02-21T23:06:54.816169+0000","last_became_active":"2024-02-21T23:06:54.810596+0000","last_became_peered":"2024-02-21T23:06:54.810596+0000","last_unstale":"2024-02-21T23:06:54.816169+0000","last_undegraded":"2024-02-21T23:06:54.816169+0000","last_fullsized":"2024-02-21T23:06:54.816169+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:37:05.788994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.39","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.823254+0000","last_change":"2024-02-21T23:06:54.822341+0000","last_active":"2024-02-21T23:06:54.823254+0000","last_peered":"2024-02-21T23:06:54.823254+0000","last_clean":"2024-02-21T23:06:54.823254+0000","last_became_active":"2024-02-21T23:06:54.822050+0000","last_became_peered":"2024-02-21T23:06:54.822050+0000","last_unstale":"2024-02-21T23:06:54.823254+0000","last_undegraded":"2024-02-21T23:06:54.823254+0000","last_fullsized":"2024-02-21T23:06:54.823254+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:16:55.110025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.38","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.819911+0000","last_change":"2024-02-21T23:06:54.815560+0000","last_active":"2024-02-21T23:06:54.819911+0000","last_peered":"2024-02-21T23:06:54.819911+0000","last_clean":"2024-02-21T23:06:54.819911+0000","last_became_active":"2024-02-21T23:06:54.815204+0000","last_became_peered":"2024-02-21T23:06:54.815204+0000","last_unstale":"2024-02-21T23:06:54.819911+0000","last_undegraded":"2024-02-21T23:06:54.819911+0000","last_fullsized":"2024-02-21T23:06:54.819911+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:37:27.165471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.37","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.799561+0000","last_change":"2024-02-21T23:06:54.795637+0000","last_active":"2024-02-21T23:06:54.799561+0000","last_peered":"2024-02-21T23:06:54.799561+0000","last_clean":"2024-02-21T23:06:54.799561+0000","last_became_active":"2024-02-21T23:06:54.795280+0000","last_became_peered":"2024-02-21T23:06:54.795280+0000","last_unstale":"2024-02-21T23:06:54.799561+0000","last_undegraded":"2024-02-21T23:06:54.799561+0000","last_fullsized":"2024-02-21T23:06:54.799561+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:36:32.615896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.36","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.815914+0000","last_change":"2024-02-21T23:06:54.802917+0000","last_active":"2024-02-21T23:06:54.815914+0000","last_peered":"2024-02-21T23:06:54.815914+0000","last_clean":"2024-02-21T23:06:54.815914+0000","last_became_active":"2024-02-21T23:06:54.802593+0000","last_became_peered":"2024-02-21T23:06:54.802593+0000","last_unstale":"2024-02-21T23:06:54.815914+0000","last_undegraded":"2024-02-21T23:06:54.815914+0000","last_fullsized":"2024-02-21T23:06:54.815914+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:41:39.165752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.35","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.800192+0000","last_change":"2024-02-21T23:06:54.795064+0000","last_active":"2024-02-21T23:06:54.800192+0000","last_peered":"2024-02-21T23:06:54.800192+0000","last_clean":"2024-02-21T23:06:54.800192+0000","last_became_active":"2024-02-21T23:06:54.794219+0000","last_became_peered":"2024-02-21T23:06:54.794219+0000","last_unstale":"2024-02-21T23:06:54.800192+0000","last_undegraded":"2024-02-21T23:06:54.800192+0000","last_fullsized":"2024-02-21T23:06:54.800192+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:20:08.122424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.34","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-21T23:06:55.762812+0000","last_change":"2024-02-21T23:06:54.789417+0000","last_active":"2024-02-21T23:06:55.762812+0000","last_peered":"2024-02-21T23:06:55.762812+0000","last_clean":"2024-02-21T23:06:55.762812+0000","last_became_active":"2024-02-21T23:06:54.788775+0000","last_became_peered":"2024-02-21T23:06:54.788775+0000","last_unstale":"2024-02-21T23:06:55.762812+0000","last_undegraded":"2024-02-21T23:06:55.762812+0000","last_fullsized":"2024-02-21T23:06:55.762812+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:51:32.395965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.33","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-21T23:06:55.760429+0000","last_change":"2024-02-21T23:06:54.793906+0000","last_active":"2024-02-21T23:06:55.760429+0000","last_peered":"2024-02-21T23:06:55.760429+0000","last_clean":"2024-02-21T23:06:55.760429+0000","last_became_active":"2024-02-21T23:06:54.793526+0000","last_became_peered":"2024-02-21T23:06:54.793526+0000","last_unstale":"2024-02-21T23:06:55.760429+0000","last_undegraded":"2024-02-21T23:06:55.760429+0000","last_fullsized":"2024-02-21T23:06:55.760429+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:24:58.275117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.32","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.789983+0000","last_change":"2024-02-21T23:06:54.787542+0000","last_active":"2024-02-21T23:06:54.789983+0000","last_peered":"2024-02-21T23:06:54.789983+0000","last_clean":"2024-02-21T23:06:54.789983+0000","last_became_active":"2024-02-21T23:06:54.787234+0000","last_became_peered":"2024-02-21T23:06:54.787234+0000","last_unstale":"2024-02-21T23:06:54.789983+0000","last_undegraded":"2024-02-21T23:06:54.789983+0000","last_fullsized":"2024-02-21T23:06:54.789983+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:20:08.122424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.31","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.799711+0000","last_change":"2024-02-21T23:06:54.794883+0000","last_active":"2024-02-21T23:06:54.799711+0000","last_peered":"2024-02-21T23:06:54.799711+0000","last_clean":"2024-02-21T23:06:54.799711+0000","last_became_active":"2024-02-21T23:06:54.794465+0000","last_became_peered":"2024-02-21T23:06:54.794465+0000","last_unstale":"2024-02-21T23:06:54.799711+0000","last_undegraded":"2024-02-21T23:06:54.799711+0000","last_fullsized":"2024-02-21T23:06:54.799711+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:40:27.170195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.30","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.799762+0000","last_change":"2024-02-21T23:06:54.796425+0000","last_active":"2024-02-21T23:06:54.799762+0000","last_peered":"2024-02-21T23:06:54.799762+0000","last_clean":"2024-02-21T23:06:54.799762+0000","last_became_active":"2024-02-21T23:06:54.795767+0000","last_became_peered":"2024-02-21T23:06:54.795767+0000","last_unstale":"2024-02-21T23:06:54.799762+0000","last_undegraded":"2024-02-21T23:06:54.799762+0000","last_fullsized":"2024-02-21T23:06:54.799762+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:14:40.765415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.2f","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-21T23:06:55.759502+0000","last_change":"2024-02-21T23:06:54.791090+0000","last_active":"2024-02-21T23:06:55.759502+0000","last_peered":"2024-02-21T23:06:55.759502+0000","last_clean":"2024-02-21T23:06:55.759502+0000","last_became_active":"2024-02-21T23:06:54.790700+0000","last_became_peered":"2024-02-21T23:06:54.790700+0000","last_unstale":"2024-02-21T23:06:55.759502+0000","last_undegraded":"2024-02-21T23:06:55.759502+0000","last_fullsized":"2024-02-21T23:06:55.759502+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:33:13.357252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2e","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.816207+0000","last_change":"2024-02-21T23:06:54.811699+0000","last_active":"2024-02-21T23:06:54.816207+0000","last_peered":"2024-02-21T23:06:54.816207+0000","last_clean":"2024-02-21T23:06:54.816207+0000","last_became_active":"2024-02-21T23:06:54.811369+0000","last_became_peered":"2024-02-21T23:06:54.811369+0000","last_unstale":"2024-02-21T23:06:54.816207+0000","last_undegraded":"2024-02-21T23:06:54.816207+0000","last_fullsized":"2024-02-21T23:06:54.816207+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:35:47.130397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.2d","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-21T23:06:55.758672+0000","last_change":"2024-02-21T23:06:54.805707+0000","last_active":"2024-02-21T23:06:55.758672+0000","last_peered":"2024-02-21T23:06:55.758672+0000","last_clean":"2024-02-21T23:06:55.758672+0000","last_became_active":"2024-02-21T23:06:54.803725+0000","last_became_peered":"2024-02-21T23:06:54.803725+0000","last_unstale":"2024-02-21T23:06:55.758672+0000","last_undegraded":"2024-02-21T23:06:55.758672+0000","last_fullsized":"2024-02-21T23:06:55.758672+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:28:27.997114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2c","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-21T23:06:55.762834+0000","last_change":"2024-02-21T23:06:54.789182+0000","last_active":"2024-02-21T23:06:55.762834+0000","last_peered":"2024-02-21T23:06:55.762834+0000","last_clean":"2024-02-21T23:06:55.762834+0000","last_became_active":"2024-02-21T23:06:54.788144+0000","last_became_peered":"2024-02-21T23:06:54.788144+0000","last_unstale":"2024-02-21T23:06:55.762834+0000","last_undegraded":"2024-02-21T23:06:55.762834+0000","last_fullsized":"2024-02-21T23:06:55.762834+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:27:38.906198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2b","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.811525+0000","last_change":"2024-02-21T23:06:54.802442+0000","last_active":"2024-02-21T23:06:54.811525+0000","last_peered":"2024-02-21T23:06:54.811525+0000","last_clean":"2024-02-21T23:06:54.811525+0000","last_became_active":"2024-02-21T23:06:54.802137+0000","last_became_peered":"2024-02-21T23:06:54.802137+0000","last_unstale":"2024-02-21T23:06:54.811525+0000","last_undegraded":"2024-02-21T23:06:54.811525+0000","last_fullsized":"2024-02-21T23:06:54.811525+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:52:13.983928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2a","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.821322+0000","last_change":"2024-02-21T23:06:54.804229+0000","last_active":"2024-02-21T23:06:54.821322+0000","last_peered":"2024-02-21T23:06:54.821322+0000","last_clean":"2024-02-21T23:06:54.821322+0000","last_became_active":"2024-02-21T23:06:54.803918+0000","last_became_peered":"2024-02-21T23:06:54.803918+0000","last_unstale":"2024-02-21T23:06:54.821322+0000","last_undegraded":"2024-02-21T23:06:54.821322+0000","last_fullsized":"2024-02-21T23:06:54.821322+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:37:05.788994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.29","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-21T23:06:55.757763+0000","last_change":"2024-02-21T23:06:54.797757+0000","last_active":"2024-02-21T23:06:55.757763+0000","last_peered":"2024-02-21T23:06:55.757763+0000","last_clean":"2024-02-21T23:06:55.757763+0000","last_became_active":"2024-02-21T23:06:54.797437+0000","last_became_peered":"2024-02-21T23:06:54.797437+0000","last_unstale":"2024-02-21T23:06:55.757763+0000","last_undegraded":"2024-02-21T23:06:55.757763+0000","last_fullsized":"2024-02-21T23:06:55.757763+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:07:42.797378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.28","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.821355+0000","last_change":"2024-02-21T23:06:54.808884+0000","last_active":"2024-02-21T23:06:54.821355+0000","last_peered":"2024-02-21T23:06:54.821355+0000","last_clean":"2024-02-21T23:06:54.821355+0000","last_became_active":"2024-02-21T23:06:54.804268+0000","last_became_peered":"2024-02-21T23:06:54.804268+0000","last_unstale":"2024-02-21T23:06:54.821355+0000","last_undegraded":"2024-02-21T23:06:54.821355+0000","last_fullsized":"2024-02-21T23:06:54.821355+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:35:47.130397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.27","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-21T23:06:55.762944+0000","last_change":"2024-02-21T23:06:54.796911+0000","last_active":"2024-02-21T23:06:55.762944+0000","last_peered":"2024-02-21T23:06:55.762944+0000","last_clean":"2024-02-21T23:06:55.762944+0000","last_became_active":"2024-02-21T23:06:54.796598+0000","last_became_peered":"2024-02-21T23:06:54.796598+0000","last_unstale":"2024-02-21T23:06:55.762944+0000","last_undegraded":"2024-02-21T23:06:55.762944+0000","last_fullsized":"2024-02-21T23:06:55.762944+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:14:16.213619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.26","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-21T23:06:55.762972+0000","last_change":"2024-02-21T23:06:54.793514+0000","last_active":"2024-02-21T23:06:55.762972+0000","last_peered":"2024-02-21T23:06:55.762972+0000","last_clean":"2024-02-21T23:06:55.762972+0000","last_became_active":"2024-02-21T23:06:54.793190+0000","last_became_peered":"2024-02-21T23:06:54.793190+0000","last_unstale":"2024-02-21T23:06:55.762972+0000","last_undegraded":"2024-02-21T23:06:55.762972+0000","last_fullsized":"2024-02-21T23:06:55.762972+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:16:44.498936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.25","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.800063+0000","last_change":"2024-02-21T23:06:54.791421+0000","last_active":"2024-02-21T23:06:54.800063+0000","last_peered":"2024-02-21T23:06:54.800063+0000","last_clean":"2024-02-21T23:06:54.800063+0000","last_became_active":"2024-02-21T23:06:54.790327+0000","last_became_peered":"2024-02-21T23:06:54.790327+0000","last_unstale":"2024-02-21T23:06:54.800063+0000","last_undegraded":"2024-02-21T23:06:54.800063+0000","last_fullsized":"2024-02-21T23:06:54.800063+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:10:18.069153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.24","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.799304+0000","last_change":"2024-02-21T23:06:54.796127+0000","last_active":"2024-02-21T23:06:54.799304+0000","last_peered":"2024-02-21T23:06:54.799304+0000","last_clean":"2024-02-21T23:06:54.799304+0000","last_became_active":"2024-02-21T23:06:54.795809+0000","last_became_peered":"2024-02-21T23:06:54.795809+0000","last_unstale":"2024-02-21T23:06:54.799304+0000","last_undegraded":"2024-02-21T23:06:54.799304+0000","last_fullsized":"2024-02-21T23:06:54.799304+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:30:18.559295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.23","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.801220+0000","last_change":"2024-02-21T23:06:54.791461+0000","last_active":"2024-02-21T23:06:54.801220+0000","last_peered":"2024-02-21T23:06:54.801220+0000","last_clean":"2024-02-21T23:06:54.801220+0000","last_became_active":"2024-02-21T23:06:54.788604+0000","last_became_peered":"2024-02-21T23:06:54.788604+0000","last_unstale":"2024-02-21T23:06:54.801220+0000","last_undegraded":"2024-02-21T23:06:54.801220+0000","last_fullsized":"2024-02-21T23:06:54.801220+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:50:49.398326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.22","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-21T23:06:55.761947+0000","last_change":"2024-02-21T23:06:54.790416+0000","last_active":"2024-02-21T23:06:55.761947+0000","last_peered":"2024-02-21T23:06:55.761947+0000","last_clean":"2024-02-21T23:06:55.761947+0000","last_became_active":"2024-02-21T23:06:54.790117+0000","last_became_peered":"2024-02-21T23:06:54.790117+0000","last_unstale":"2024-02-21T23:06:55.761947+0000","last_undegraded":"2024-02-21T23:06:55.761947+0000","last_fullsized":"2024-02-21T23:06:55.761947+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:43:15.037996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.21","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.805315+0000","last_change":"2024-02-21T23:06:54.801270+0000","last_active":"2024-02-21T23:06:54.805315+0000","last_peered":"2024-02-21T23:06:54.805315+0000","last_clean":"2024-02-21T23:06:54.805315+0000","last_became_active":"2024-02-21T23:06:54.801062+0000","last_became_peered":"2024-02-21T23:06:54.801062+0000","last_unstale":"2024-02-21T23:06:54.805315+0000","last_undegraded":"2024-02-21T23:06:54.805315+0000","last_fullsized":"2024-02-21T23:06:54.805315+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:00:52.018344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.20","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.803977+0000","last_change":"2024-02-21T23:06:54.795665+0000","last_active":"2024-02-21T23:06:54.803977+0000","last_peered":"2024-02-21T23:06:54.803977+0000","last_clean":"2024-02-21T23:06:54.803977+0000","last_became_active":"2024-02-21T23:06:54.794717+0000","last_became_peered":"2024-02-21T23:06:54.794717+0000","last_unstale":"2024-02-21T23:06:54.803977+0000","last_undegraded":"2024-02-21T23:06:54.803977+0000","last_fullsized":"2024-02-21T23:06:54.803977+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:20:44.653663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-21T23:06:55.757573+0000","last_change":"2024-02-21T23:06:54.796453+0000","last_active":"2024-02-21T23:06:55.757573+0000","last_peered":"2024-02-21T23:06:55.757573+0000","last_clean":"2024-02-21T23:06:55.757573+0000","last_became_active":"2024-02-21T23:06:54.790829+0000","last_became_peered":"2024-02-21T23:06:54.790829+0000","last_unstale":"2024-02-21T23:06:55.757573+0000","last_undegraded":"2024-02-21T23:06:55.757573+0000","last_fullsized":"2024-02-21T23:06:55.757573+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:47:30.278741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"97'4","reported_seq":18,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-21T23:06:55.761090+0000","last_change":"2024-02-21T23:06:53.748885+0000","last_active":"2024-02-21T23:06:55.761090+0000","last_peered":"2024-02-21T23:06:55.761090+0000","last_clean":"2024-02-21T23:06:55.761090+0000","last_became_active":"2024-02-21T23:06:53.748561+0000","last_became_peered":"2024-02-21T23:06:53.748561+0000","last_unstale":"2024-02-21T23:06:55.761090+0000","last_undegraded":"2024-02-21T23:06:55.761090+0000","last_fullsized":"2024-02-21T23:06:55.761090+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_clean_scrub_stamp":"2024-02-21T23:06:52.701904+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:51:31.391428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.819934+0000","last_change":"2024-02-21T23:06:54.815770+0000","last_active":"2024-02-21T23:06:54.819934+0000","last_peered":"2024-02-21T23:06:54.819934+0000","last_clean":"2024-02-21T23:06:54.819934+0000","last_became_active":"2024-02-21T23:06:54.815342+0000","last_became_peered":"2024-02-21T23:06:54.815342+0000","last_unstale":"2024-02-21T23:06:54.819934+0000","last_undegraded":"2024-02-21T23:06:54.819934+0000","last_fullsized":"2024-02-21T23:06:54.819934+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:43:29.861586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.1e","version":"97'1","reported_seq":13,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.818602+0000","last_change":"2024-02-21T23:06:53.747576+0000","last_active":"2024-02-21T23:06:54.818602+0000","last_peered":"2024-02-21T23:06:54.818602+0000","last_clean":"2024-02-21T23:06:54.818602+0000","last_became_active":"2024-02-21T23:06:53.746959+0000","last_became_peered":"2024-02-21T23:06:53.746959+0000","last_unstale":"2024-02-21T23:06:54.818602+0000","last_undegraded":"2024-02-21T23:06:54.818602+0000","last_fullsized":"2024-02-21T23:06:54.818602+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_clean_scrub_stamp":"2024-02-21T23:06:52.701904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:18:28.698148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-21T23:06:55.761799+0000","last_change":"2024-02-21T23:06:54.797713+0000","last_active":"2024-02-21T23:06:55.761799+0000","last_peered":"2024-02-21T23:06:55.761799+0000","last_clean":"2024-02-21T23:06:55.761799+0000","last_became_active":"2024-02-21T23:06:54.797279+0000","last_became_peered":"2024-02-21T23:06:54.797279+0000","last_unstale":"2024-02-21T23:06:55.761799+0000","last_undegraded":"2024-02-21T23:06:55.761799+0000","last_fullsized":"2024-02-21T23:06:55.761799+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:14:40.765415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1d","version":"97'2","reported_seq":16,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-21T23:06:55.762465+0000","last_change":"2024-02-21T23:06:53.755179+0000","last_active":"2024-02-21T23:06:55.762465+0000","last_peered":"2024-02-21T23:06:55.762465+0000","last_clean":"2024-02-21T23:06:55.762465+0000","last_became_active":"2024-02-21T23:06:53.752835+0000","last_became_peered":"2024-02-21T23:06:53.752835+0000","last_unstale":"2024-02-21T23:06:55.762465+0000","last_undegraded":"2024-02-21T23:06:55.762465+0000","last_fullsized":"2024-02-21T23:06:55.762465+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_clean_scrub_stamp":"2024-02-21T23:06:52.701904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:37:04.784457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.800227+0000","last_change":"2024-02-21T23:06:54.788244+0000","last_active":"2024-02-21T23:06:54.800227+0000","last_peered":"2024-02-21T23:06:54.800227+0000","last_clean":"2024-02-21T23:06:54.800227+0000","last_became_active":"2024-02-21T23:06:54.787875+0000","last_became_peered":"2024-02-21T23:06:54.787875+0000","last_unstale":"2024-02-21T23:06:54.800227+0000","last_undegraded":"2024-02-21T23:06:54.800227+0000","last_fullsized":"2024-02-21T23:06:54.800227+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:54:29.660083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1c","version":"97'2","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.821699+0000","last_change":"2024-02-21T23:06:53.744843+0000","last_active":"2024-02-21T23:06:54.821699+0000","last_peered":"2024-02-21T23:06:54.821699+0000","last_clean":"2024-02-21T23:06:54.821699+0000","last_became_active":"2024-02-21T23:06:53.744486+0000","last_became_peered":"2024-02-21T23:06:53.744486+0000","last_unstale":"2024-02-21T23:06:54.821699+0000","last_undegraded":"2024-02-21T23:06:54.821699+0000","last_fullsized":"2024-02-21T23:06:54.821699+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_clean_scrub_stamp":"2024-02-21T23:06:52.701904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:52:35.919454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-21T23:06:55.760463+0000","last_change":"2024-02-21T23:06:54.784963+0000","last_active":"2024-02-21T23:06:55.760463+0000","last_peered":"2024-02-21T23:06:55.760463+0000","last_clean":"2024-02-21T23:06:55.760463+0000","last_became_active":"2024-02-21T23:06:54.783318+0000","last_became_peered":"2024-02-21T23:06:54.783318+0000","last_unstale":"2024-02-21T23:06:55.760463+0000","last_undegraded":"2024-02-21T23:06:55.760463+0000","last_fullsized":"2024-02-21T23:06:55.760463+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:54:29.272490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1b","version":"97'1","reported_seq":13,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.816309+0000","last_change":"2024-02-21T23:06:53.751073+0000","last_active":"2024-02-21T23:06:54.816309+0000","last_peered":"2024-02-21T23:06:54.816309+0000","last_clean":"2024-02-21T23:06:54.816309+0000","last_became_active":"2024-02-21T23:06:53.749702+0000","last_became_peered":"2024-02-21T23:06:53.749702+0000","last_unstale":"2024-02-21T23:06:54.816309+0000","last_undegraded":"2024-02-21T23:06:54.816309+0000","last_fullsized":"2024-02-21T23:06:54.816309+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_clean_scrub_stamp":"2024-02-21T23:06:52.701904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:10:17.064616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-21T23:06:55.758965+0000","last_change":"2024-02-21T23:06:54.805456+0000","last_active":"2024-02-21T23:06:55.758965+0000","last_peered":"2024-02-21T23:06:55.758965+0000","last_clean":"2024-02-21T23:06:55.758965+0000","last_became_active":"2024-02-21T23:06:54.803352+0000","last_became_peered":"2024-02-21T23:06:54.803352+0000","last_unstale":"2024-02-21T23:06:55.758965+0000","last_undegraded":"2024-02-21T23:06:55.758965+0000","last_fullsized":"2024-02-21T23:06:55.758965+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:41:59.760739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"97'1","reported_seq":13,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.814272+0000","last_change":"2024-02-21T23:06:53.750209+0000","last_active":"2024-02-21T23:06:54.814272+0000","last_peered":"2024-02-21T23:06:54.814272+0000","last_clean":"2024-02-21T23:06:54.814272+0000","last_became_active":"2024-02-21T23:06:53.749948+0000","last_became_peered":"2024-02-21T23:06:53.749948+0000","last_unstale":"2024-02-21T23:06:54.814272+0000","last_undegraded":"2024-02-21T23:06:54.814272+0000","last_fullsized":"2024-02-21T23:06:54.814272+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_clean_scrub_stamp":"2024-02-21T23:06:52.701904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:16:54.105487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.802231+0000","last_change":"2024-02-21T23:06:54.799408+0000","last_active":"2024-02-21T23:06:54.802231+0000","last_peered":"2024-02-21T23:06:54.802231+0000","last_clean":"2024-02-21T23:06:54.802231+0000","last_became_active":"2024-02-21T23:06:54.798798+0000","last_became_peered":"2024-02-21T23:06:54.798798+0000","last_unstale":"2024-02-21T23:06:54.802231+0000","last_undegraded":"2024-02-21T23:06:54.802231+0000","last_fullsized":"2024-02-21T23:06:54.802231+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:42:34.044297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-21T23:06:55.761580+0000","last_change":"2024-02-21T23:06:53.751886+0000","last_active":"2024-02-21T23:06:55.761580+0000","last_peered":"2024-02-21T23:06:55.761580+0000","last_clean":"2024-02-21T23:06:55.761580+0000","last_became_active":"2024-02-21T23:06:53.750408+0000","last_became_peered":"2024-02-21T23:06:53.750408+0000","last_unstale":"2024-02-21T23:06:55.761580+0000","last_undegraded":"2024-02-21T23:06:55.761580+0000","last_fullsized":"2024-02-21T23:06:55.761580+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_clean_scrub_stamp":"2024-02-21T23:06:52.701904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:19:09.194924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-21T23:06:55.761189+0000","last_change":"2024-02-21T23:06:54.781428+0000","last_active":"2024-02-21T23:06:55.761189+0000","last_peered":"2024-02-21T23:06:55.761189+0000","last_clean":"2024-02-21T23:06:55.761189+0000","last_became_active":"2024-02-21T23:06:54.780157+0000","last_became_peered":"2024-02-21T23:06:54.780157+0000","last_unstale":"2024-02-21T23:06:55.761189+0000","last_undegraded":"2024-02-21T23:06:55.761189+0000","last_fullsized":"2024-02-21T23:06:55.761189+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:17:54.125344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-21T23:06:55.759989+0000","last_change":"2024-02-21T23:06:53.740420+0000","last_active":"2024-02-21T23:06:55.759989+0000","last_peered":"2024-02-21T23:06:55.759989+0000","last_clean":"2024-02-21T23:06:55.759989+0000","last_became_active":"2024-02-21T23:06:53.738614+0000","last_became_peered":"2024-02-21T23:06:53.738614+0000","last_unstale":"2024-02-21T23:06:55.759989+0000","last_undegraded":"2024-02-21T23:06:55.759989+0000","last_fullsized":"2024-02-21T23:06:55.759989+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_clean_scrub_stamp":"2024-02-21T23:06:52.701904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:43:14.033459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-21T23:06:55.757525+0000","last_change":"2024-02-21T23:06:54.794779+0000","last_active":"2024-02-21T23:06:55.757525+0000","last_peered":"2024-02-21T23:06:55.757525+0000","last_clean":"2024-02-21T23:06:55.757525+0000","last_became_active":"2024-02-21T23:06:54.790409+0000","last_became_peered":"2024-02-21T23:06:54.790409+0000","last_unstale":"2024-02-21T23:06:55.757525+0000","last_undegraded":"2024-02-21T23:06:55.757525+0000","last_fullsized":"2024-02-21T23:06:55.757525+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:59:29.839517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-21T23:06:55.762872+0000","last_change":"2024-02-21T23:06:53.753188+0000","last_active":"2024-02-21T23:06:55.762872+0000","last_peered":"2024-02-21T23:06:55.762872+0000","last_clean":"2024-02-21T23:06:55.762872+0000","last_became_active":"2024-02-21T23:06:53.752893+0000","last_became_peered":"2024-02-21T23:06:53.752893+0000","last_unstale":"2024-02-21T23:06:55.762872+0000","last_undegraded":"2024-02-21T23:06:55.762872+0000","last_fullsized":"2024-02-21T23:06:55.762872+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_clean_scrub_stamp":"2024-02-21T23:06:52.701904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:16:54.105487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.815589+0000","last_change":"2024-02-21T23:06:54.798146+0000","last_active":"2024-02-21T23:06:54.815589+0000","last_peered":"2024-02-21T23:06:54.815589+0000","last_clean":"2024-02-21T23:06:54.815589+0000","last_became_active":"2024-02-21T23:06:54.797837+0000","last_became_peered":"2024-02-21T23:06:54.797837+0000","last_unstale":"2024-02-21T23:06:54.815589+0000","last_undegraded":"2024-02-21T23:06:54.815589+0000","last_fullsized":"2024-02-21T23:06:54.815589+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:42:34.044297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-21T23:06:55.759002+0000","last_change":"2024-02-21T23:06:53.744963+0000","last_active":"2024-02-21T23:06:55.759002+0000","last_peered":"2024-02-21T23:06:55.759002+0000","last_clean":"2024-02-21T23:06:55.759002+0000","last_became_active":"2024-02-21T23:06:53.742193+0000","last_became_peered":"2024-02-21T23:06:53.742193+0000","last_unstale":"2024-02-21T23:06:55.759002+0000","last_undegraded":"2024-02-21T23:06:55.759002+0000","last_fullsized":"2024-02-21T23:06:55.759002+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_clean_scrub_stamp":"2024-02-21T23:06:52.701904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:45:17.720735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.820300+0000","last_change":"2024-02-21T23:06:54.818699+0000","last_active":"2024-02-21T23:06:54.820300+0000","last_peered":"2024-02-21T23:06:54.820300+0000","last_clean":"2024-02-21T23:06:54.820300+0000","last_became_active":"2024-02-21T23:06:54.818260+0000","last_became_peered":"2024-02-21T23:06:54.818260+0000","last_unstale":"2024-02-21T23:06:54.820300+0000","last_undegraded":"2024-02-21T23:06:54.820300+0000","last_fullsized":"2024-02-21T23:06:54.820300+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:04:38.617025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.15","version":"97'3","reported_seq":15,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.815789+0000","last_change":"2024-02-21T23:06:53.746719+0000","last_active":"2024-02-21T23:06:54.815789+0000","last_peered":"2024-02-21T23:06:54.815789+0000","last_clean":"2024-02-21T23:06:54.815789+0000","last_became_active":"2024-02-21T23:06:53.746425+0000","last_became_peered":"2024-02-21T23:06:53.746425+0000","last_unstale":"2024-02-21T23:06:54.815789+0000","last_undegraded":"2024-02-21T23:06:54.815789+0000","last_fullsized":"2024-02-21T23:06:54.815789+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_clean_scrub_stamp":"2024-02-21T23:06:52.701904+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:10:58.463061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.821262+0000","last_change":"2024-02-21T23:06:54.803344+0000","last_active":"2024-02-21T23:06:54.821262+0000","last_peered":"2024-02-21T23:06:54.821262+0000","last_clean":"2024-02-21T23:06:54.821262+0000","last_became_active":"2024-02-21T23:06:54.802485+0000","last_became_peered":"2024-02-21T23:06:54.802485+0000","last_unstale":"2024-02-21T23:06:54.821262+0000","last_undegraded":"2024-02-21T23:06:54.821262+0000","last_fullsized":"2024-02-21T23:06:54.821262+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:19:10.199462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.14","version":"97'6","reported_seq":18,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.820229+0000","last_change":"2024-02-21T23:06:53.756426+0000","last_active":"2024-02-21T23:06:54.820229+0000","last_peered":"2024-02-21T23:06:54.820229+0000","last_clean":"2024-02-21T23:06:54.820229+0000","last_became_active":"2024-02-21T23:06:53.756111+0000","last_became_peered":"2024-02-21T23:06:53.756111+0000","last_unstale":"2024-02-21T23:06:54.820229+0000","last_undegraded":"2024-02-21T23:06:54.820229+0000","last_fullsized":"2024-02-21T23:06:54.820229+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_clean_scrub_stamp":"2024-02-21T23:06:52.701904+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:52:35.919454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.800131+0000","last_change":"2024-02-21T23:06:54.792920+0000","last_active":"2024-02-21T23:06:54.800131+0000","last_peered":"2024-02-21T23:06:54.800131+0000","last_clean":"2024-02-21T23:06:54.800131+0000","last_became_active":"2024-02-21T23:06:54.792547+0000","last_became_peered":"2024-02-21T23:06:54.792547+0000","last_unstale":"2024-02-21T23:06:54.800131+0000","last_undegraded":"2024-02-21T23:06:54.800131+0000","last_fullsized":"2024-02-21T23:06:54.800131+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:52:28.706647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.13","version":"97'2","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.822186+0000","last_change":"2024-02-21T23:06:53.743679+0000","last_active":"2024-02-21T23:06:54.822186+0000","last_peered":"2024-02-21T23:06:54.822186+0000","last_clean":"2024-02-21T23:06:54.822186+0000","last_became_active":"2024-02-21T23:06:53.743301+0000","last_became_peered":"2024-02-21T23:06:53.743301+0000","last_unstale":"2024-02-21T23:06:54.822186+0000","last_undegraded":"2024-02-21T23:06:54.822186+0000","last_fullsized":"2024-02-21T23:06:54.822186+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_clean_scrub_stamp":"2024-02-21T23:06:52.701904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:36:31.611359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.805581+0000","last_change":"2024-02-21T23:06:54.799388+0000","last_active":"2024-02-21T23:06:54.805581+0000","last_peered":"2024-02-21T23:06:54.805581+0000","last_clean":"2024-02-21T23:06:54.805581+0000","last_became_active":"2024-02-21T23:06:54.795335+0000","last_became_peered":"2024-02-21T23:06:54.795335+0000","last_unstale":"2024-02-21T23:06:54.805581+0000","last_undegraded":"2024-02-21T23:06:54.805581+0000","last_fullsized":"2024-02-21T23:06:54.805581+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:41:39.165752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-21T23:06:55.761294+0000","last_change":"2024-02-21T23:06:53.744219+0000","last_active":"2024-02-21T23:06:55.761294+0000","last_peered":"2024-02-21T23:06:55.761294+0000","last_clean":"2024-02-21T23:06:55.761294+0000","last_became_active":"2024-02-21T23:06:53.743238+0000","last_became_peered":"2024-02-21T23:06:53.743238+0000","last_unstale":"2024-02-21T23:06:55.761294+0000","last_undegraded":"2024-02-21T23:06:55.761294+0000","last_fullsized":"2024-02-21T23:06:55.761294+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_clean_scrub_stamp":"2024-02-21T23:06:52.701904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:27:37.901660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.791395+0000","last_change":"2024-02-21T23:06:54.789691+0000","last_active":"2024-02-21T23:06:54.791395+0000","last_peered":"2024-02-21T23:06:54.791395+0000","last_clean":"2024-02-21T23:06:54.791395+0000","last_became_active":"2024-02-21T23:06:54.788584+0000","last_became_peered":"2024-02-21T23:06:54.788584+0000","last_unstale":"2024-02-21T23:06:54.791395+0000","last_undegraded":"2024-02-21T23:06:54.791395+0000","last_fullsized":"2024-02-21T23:06:54.791395+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:59:08.487373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.778789+0000","last_change":"2024-02-21T23:06:53.741935+0000","last_active":"2024-02-21T23:06:54.778789+0000","last_peered":"2024-02-21T23:06:54.778789+0000","last_clean":"2024-02-21T23:06:54.778789+0000","last_became_active":"2024-02-21T23:06:53.741662+0000","last_became_peered":"2024-02-21T23:06:53.741662+0000","last_unstale":"2024-02-21T23:06:54.778789+0000","last_undegraded":"2024-02-21T23:06:54.778789+0000","last_fullsized":"2024-02-21T23:06:54.778789+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_clean_scrub_stamp":"2024-02-21T23:06:52.701904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:47:03.976688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-21T23:06:55.759743+0000","last_change":"2024-02-21T23:06:54.799679+0000","last_active":"2024-02-21T23:06:55.759743+0000","last_peered":"2024-02-21T23:06:55.759743+0000","last_clean":"2024-02-21T23:06:55.759743+0000","last_became_active":"2024-02-21T23:06:54.799387+0000","last_became_peered":"2024-02-21T23:06:54.799387+0000","last_unstale":"2024-02-21T23:06:55.759743+0000","last_undegraded":"2024-02-21T23:06:55.759743+0000","last_fullsized":"2024-02-21T23:06:55.759743+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:14:21.111864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.10","version":"97'8","reported_seq":22,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-21T23:06:55.761500+0000","last_change":"2024-02-21T23:06:53.753404+0000","last_active":"2024-02-21T23:06:55.761500+0000","last_peered":"2024-02-21T23:06:55.761500+0000","last_clean":"2024-02-21T23:06:55.761500+0000","last_became_active":"2024-02-21T23:06:53.752630+0000","last_became_peered":"2024-02-21T23:06:53.752630+0000","last_unstale":"2024-02-21T23:06:55.761500+0000","last_undegraded":"2024-02-21T23:06:55.761500+0000","last_fullsized":"2024-02-21T23:06:55.761500+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_clean_scrub_stamp":"2024-02-21T23:06:52.701904+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:00:51.013807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.794424+0000","last_change":"2024-02-21T23:06:54.789052+0000","last_active":"2024-02-21T23:06:54.794424+0000","last_peered":"2024-02-21T23:06:54.794424+0000","last_clean":"2024-02-21T23:06:54.794424+0000","last_became_active":"2024-02-21T23:06:54.788752+0000","last_became_peered":"2024-02-21T23:06:54.788752+0000","last_unstale":"2024-02-21T23:06:54.794424+0000","last_undegraded":"2024-02-21T23:06:54.794424+0000","last_fullsized":"2024-02-21T23:06:54.794424+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:52:20.566957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.f","version":"97'1","reported_seq":13,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.813799+0000","last_change":"2024-02-21T23:06:53.745825+0000","last_active":"2024-02-21T23:06:54.813799+0000","last_peered":"2024-02-21T23:06:54.813799+0000","last_clean":"2024-02-21T23:06:54.813799+0000","last_became_active":"2024-02-21T23:06:53.745469+0000","last_became_peered":"2024-02-21T23:06:53.745469+0000","last_unstale":"2024-02-21T23:06:54.813799+0000","last_undegraded":"2024-02-21T23:06:54.813799+0000","last_fullsized":"2024-02-21T23:06:54.813799+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_clean_scrub_stamp":"2024-02-21T23:06:52.701904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:57:12.762293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.813600+0000","last_change":"2024-02-21T23:06:54.809364+0000","last_active":"2024-02-21T23:06:54.813600+0000","last_peered":"2024-02-21T23:06:54.813600+0000","last_clean":"2024-02-21T23:06:54.813600+0000","last_became_active":"2024-02-21T23:06:54.809080+0000","last_became_peered":"2024-02-21T23:06:54.809080+0000","last_unstale":"2024-02-21T23:06:54.813600+0000","last_undegraded":"2024-02-21T23:06:54.813600+0000","last_fullsized":"2024-02-21T23:06:54.813600+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:19:10.199462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"97'1","reported_seq":13,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.815884+0000","last_change":"2024-02-21T23:06:53.749878+0000","last_active":"2024-02-21T23:06:54.815884+0000","last_peered":"2024-02-21T23:06:54.815884+0000","last_clean":"2024-02-21T23:06:54.815884+0000","last_became_active":"2024-02-21T23:06:53.749348+0000","last_became_peered":"2024-02-21T23:06:53.749348+0000","last_unstale":"2024-02-21T23:06:54.815884+0000","last_undegraded":"2024-02-21T23:06:54.815884+0000","last_fullsized":"2024-02-21T23:06:54.815884+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_clean_scrub_stamp":"2024-02-21T23:06:52.701904+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:52:27.702110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.809484+0000","last_change":"2024-02-21T23:06:54.792795+0000","last_active":"2024-02-21T23:06:54.809484+0000","last_peered":"2024-02-21T23:06:54.809484+0000","last_clean":"2024-02-21T23:06:54.809484+0000","last_became_active":"2024-02-21T23:06:54.792269+0000","last_became_peered":"2024-02-21T23:06:54.792269+0000","last_unstale":"2024-02-21T23:06:54.809484+0000","last_undegraded":"2024-02-21T23:06:54.809484+0000","last_fullsized":"2024-02-21T23:06:54.809484+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:30:18.559295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.773678+0000","last_change":"2024-02-21T23:06:53.758219+0000","last_active":"2024-02-21T23:06:54.773678+0000","last_peered":"2024-02-21T23:06:54.773678+0000","last_clean":"2024-02-21T23:06:54.773678+0000","last_became_active":"2024-02-21T23:06:53.757650+0000","last_became_peered":"2024-02-21T23:06:53.757650+0000","last_unstale":"2024-02-21T23:06:54.773678+0000","last_undegraded":"2024-02-21T23:06:54.773678+0000","last_fullsized":"2024-02-21T23:06:54.773678+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_clean_scrub_stamp":"2024-02-21T23:06:52.701904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:35:17.535584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-21T23:06:55.758580+0000","last_change":"2024-02-21T23:06:54.790172+0000","last_active":"2024-02-21T23:06:55.758580+0000","last_peered":"2024-02-21T23:06:55.758580+0000","last_clean":"2024-02-21T23:06:55.758580+0000","last_became_active":"2024-02-21T23:06:54.789866+0000","last_became_peered":"2024-02-21T23:06:54.789866+0000","last_unstale":"2024-02-21T23:06:55.758580+0000","last_undegraded":"2024-02-21T23:06:55.758580+0000","last_fullsized":"2024-02-21T23:06:55.758580+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:35:51.171071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"97'2","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.819341+0000","last_change":"2024-02-21T23:06:53.730541+0000","last_active":"2024-02-21T23:06:54.819341+0000","last_peered":"2024-02-21T23:06:54.819341+0000","last_clean":"2024-02-21T23:06:54.819341+0000","last_became_active":"2024-02-21T23:06:53.730073+0000","last_became_peered":"2024-02-21T23:06:53.730073+0000","last_unstale":"2024-02-21T23:06:54.819341+0000","last_undegraded":"2024-02-21T23:06:54.819341+0000","last_fullsized":"2024-02-21T23:06:54.819341+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_clean_scrub_stamp":"2024-02-21T23:06:52.701904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:30:05.360823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-21T23:06:55.758870+0000","last_change":"2024-02-21T23:06:54.808993+0000","last_active":"2024-02-21T23:06:55.758870+0000","last_peered":"2024-02-21T23:06:55.758870+0000","last_clean":"2024-02-21T23:06:55.758870+0000","last_became_active":"2024-02-21T23:06:54.808704+0000","last_became_peered":"2024-02-21T23:06:54.808704+0000","last_unstale":"2024-02-21T23:06:55.758870+0000","last_undegraded":"2024-02-21T23:06:55.758870+0000","last_fullsized":"2024-02-21T23:06:55.758870+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:43:13.678228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.777961+0000","last_change":"2024-02-21T23:06:53.740354+0000","last_active":"2024-02-21T23:06:54.777961+0000","last_peered":"2024-02-21T23:06:54.777961+0000","last_clean":"2024-02-21T23:06:54.777961+0000","last_became_active":"2024-02-21T23:06:53.740058+0000","last_became_peered":"2024-02-21T23:06:53.740058+0000","last_unstale":"2024-02-21T23:06:54.777961+0000","last_undegraded":"2024-02-21T23:06:54.777961+0000","last_fullsized":"2024-02-21T23:06:54.777961+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_clean_scrub_stamp":"2024-02-21T23:06:52.701904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:28:57.064260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-21T23:06:55.761181+0000","last_change":"2024-02-21T23:06:54.791521+0000","last_active":"2024-02-21T23:06:55.761181+0000","last_peered":"2024-02-21T23:06:55.761181+0000","last_clean":"2024-02-21T23:06:55.761181+0000","last_became_active":"2024-02-21T23:06:54.791181+0000","last_became_peered":"2024-02-21T23:06:54.791181+0000","last_unstale":"2024-02-21T23:06:55.761181+0000","last_undegraded":"2024-02-21T23:06:55.761181+0000","last_fullsized":"2024-02-21T23:06:55.761181+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:54:37.184551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.787113+0000","last_change":"2024-02-21T23:06:53.756174+0000","last_active":"2024-02-21T23:06:54.787113+0000","last_peered":"2024-02-21T23:06:54.787113+0000","last_clean":"2024-02-21T23:06:54.787113+0000","last_became_active":"2024-02-21T23:06:53.755343+0000","last_became_peered":"2024-02-21T23:06:53.755343+0000","last_unstale":"2024-02-21T23:06:54.787113+0000","last_undegraded":"2024-02-21T23:06:54.787113+0000","last_fullsized":"2024-02-21T23:06:54.787113+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_clean_scrub_stamp":"2024-02-21T23:06:52.701904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:29:23.260556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.820287+0000","last_change":"2024-02-21T23:06:54.817397+0000","last_active":"2024-02-21T23:06:54.820287+0000","last_peered":"2024-02-21T23:06:54.820287+0000","last_clean":"2024-02-21T23:06:54.820287+0000","last_became_active":"2024-02-21T23:06:54.817090+0000","last_became_peered":"2024-02-21T23:06:54.817090+0000","last_unstale":"2024-02-21T23:06:54.820287+0000","last_undegraded":"2024-02-21T23:06:54.820287+0000","last_fullsized":"2024-02-21T23:06:54.820287+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:41:48.818494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.779569+0000","last_change":"2024-02-21T23:06:53.725221+0000","last_active":"2024-02-21T23:06:54.779569+0000","last_peered":"2024-02-21T23:06:54.779569+0000","last_clean":"2024-02-21T23:06:54.779569+0000","last_became_active":"2024-02-21T23:06:53.724900+0000","last_became_peered":"2024-02-21T23:06:53.724900+0000","last_unstale":"2024-02-21T23:06:54.779569+0000","last_undegraded":"2024-02-21T23:06:54.779569+0000","last_fullsized":"2024-02-21T23:06:54.779569+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_clean_scrub_stamp":"2024-02-21T23:06:52.701904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:04:11.157605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-21T23:06:55.757405+0000","last_change":"2024-02-21T23:06:54.772130+0000","last_active":"2024-02-21T23:06:55.757405+0000","last_peered":"2024-02-21T23:06:55.757405+0000","last_clean":"2024-02-21T23:06:55.757405+0000","last_became_active":"2024-02-21T23:06:54.770534+0000","last_became_peered":"2024-02-21T23:06:54.770534+0000","last_unstale":"2024-02-21T23:06:55.757405+0000","last_undegraded":"2024-02-21T23:06:55.757405+0000","last_fullsized":"2024-02-21T23:06:55.757405+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:59:55.738424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.7","version":"97'2","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.821774+0000","last_change":"2024-02-21T23:06:53.744593+0000","last_active":"2024-02-21T23:06:54.821774+0000","last_peered":"2024-02-21T23:06:54.821774+0000","last_clean":"2024-02-21T23:06:54.821774+0000","last_became_active":"2024-02-21T23:06:53.744310+0000","last_became_peered":"2024-02-21T23:06:53.744310+0000","last_unstale":"2024-02-21T23:06:54.821774+0000","last_undegraded":"2024-02-21T23:06:54.821774+0000","last_fullsized":"2024-02-21T23:06:54.821774+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_clean_scrub_stamp":"2024-02-21T23:06:52.701904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:57:12.762293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-21T23:06:55.761669+0000","last_change":"2024-02-21T23:06:54.776671+0000","last_active":"2024-02-21T23:06:55.761669+0000","last_peered":"2024-02-21T23:06:55.761669+0000","last_clean":"2024-02-21T23:06:55.761669+0000","last_became_active":"2024-02-21T23:06:54.775959+0000","last_became_peered":"2024-02-21T23:06:54.775959+0000","last_unstale":"2024-02-21T23:06:55.761669+0000","last_undegraded":"2024-02-21T23:06:55.761669+0000","last_fullsized":"2024-02-21T23:06:55.761669+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:40:27.170195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-21T23:06:55.759080+0000","last_change":"2024-02-21T23:06:53.753025+0000","last_active":"2024-02-21T23:06:55.759080+0000","last_peered":"2024-02-21T23:06:55.759080+0000","last_clean":"2024-02-21T23:06:55.759080+0000","last_became_active":"2024-02-21T23:06:53.752597+0000","last_became_peered":"2024-02-21T23:06:53.752597+0000","last_unstale":"2024-02-21T23:06:55.759080+0000","last_undegraded":"2024-02-21T23:06:55.759080+0000","last_fullsized":"2024-02-21T23:06:55.759080+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_clean_scrub_stamp":"2024-02-21T23:06:52.701904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:54:36.180014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-21T23:06:55.761082+0000","last_change":"2024-02-21T23:06:54.791481+0000","last_active":"2024-02-21T23:06:55.761082+0000","last_peered":"2024-02-21T23:06:55.761082+0000","last_clean":"2024-02-21T23:06:55.761082+0000","last_became_active":"2024-02-21T23:06:54.791200+0000","last_became_peered":"2024-02-21T23:06:54.791200+0000","last_unstale":"2024-02-21T23:06:55.761082+0000","last_undegraded":"2024-02-21T23:06:55.761082+0000","last_fullsized":"2024-02-21T23:06:55.761082+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:45:18.725272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.789091+0000","last_change":"2024-02-21T23:06:53.748756+0000","last_active":"2024-02-21T23:06:54.789091+0000","last_peered":"2024-02-21T23:06:54.789091+0000","last_clean":"2024-02-21T23:06:54.789091+0000","last_became_active":"2024-02-21T23:06:53.748409+0000","last_became_peered":"2024-02-21T23:06:53.748409+0000","last_unstale":"2024-02-21T23:06:54.789091+0000","last_undegraded":"2024-02-21T23:06:54.789091+0000","last_fullsized":"2024-02-21T23:06:54.789091+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_clean_scrub_stamp":"2024-02-21T23:06:52.701904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:04:57.981362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.800321+0000","last_change":"2024-02-21T23:06:54.795666+0000","last_active":"2024-02-21T23:06:54.800321+0000","last_peered":"2024-02-21T23:06:54.800321+0000","last_clean":"2024-02-21T23:06:54.800321+0000","last_became_active":"2024-02-21T23:06:54.795393+0000","last_became_peered":"2024-02-21T23:06:54.795393+0000","last_unstale":"2024-02-21T23:06:54.800321+0000","last_undegraded":"2024-02-21T23:06:54.800321+0000","last_fullsized":"2024-02-21T23:06:54.800321+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:34:18.224729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.776743+0000","last_change":"2024-02-21T23:06:53.750570+0000","last_active":"2024-02-21T23:06:54.776743+0000","last_peered":"2024-02-21T23:06:54.776743+0000","last_clean":"2024-02-21T23:06:54.776743+0000","last_became_active":"2024-02-21T23:06:53.750291+0000","last_became_peered":"2024-02-21T23:06:53.750291+0000","last_unstale":"2024-02-21T23:06:54.776743+0000","last_undegraded":"2024-02-21T23:06:54.776743+0000","last_fullsized":"2024-02-21T23:06:54.776743+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_clean_scrub_stamp":"2024-02-21T23:06:52.701904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:16:26.607536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.820323+0000","last_change":"2024-02-21T23:06:54.817197+0000","last_active":"2024-02-21T23:06:54.820323+0000","last_peered":"2024-02-21T23:06:54.820323+0000","last_clean":"2024-02-21T23:06:54.820323+0000","last_became_active":"2024-02-21T23:06:54.815274+0000","last_became_peered":"2024-02-21T23:06:54.815274+0000","last_unstale":"2024-02-21T23:06:54.820323+0000","last_undegraded":"2024-02-21T23:06:54.820323+0000","last_fullsized":"2024-02-21T23:06:54.820323+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:02:20.329704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-02-21T23:06:55.762187+0000","last_change":"2024-02-21T23:06:53.750107+0000","last_active":"2024-02-21T23:06:55.762187+0000","last_peered":"2024-02-21T23:06:55.762187+0000","last_clean":"2024-02-21T23:06:55.762187+0000","last_became_active":"2024-02-21T23:06:53.749809+0000","last_became_peered":"2024-02-21T23:06:53.749809+0000","last_unstale":"2024-02-21T23:06:55.762187+0000","last_undegraded":"2024-02-21T23:06:55.762187+0000","last_fullsized":"2024-02-21T23:06:55.762187+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_clean_scrub_stamp":"2024-02-21T23:06:52.701904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:35:46.125860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.819848+0000","last_change":"2024-02-21T23:06:54.815320+0000","last_active":"2024-02-21T23:06:54.819848+0000","last_peered":"2024-02-21T23:06:54.819848+0000","last_clean":"2024-02-21T23:06:54.819848+0000","last_became_active":"2024-02-21T23:06:54.815007+0000","last_became_peered":"2024-02-21T23:06:54.815007+0000","last_unstale":"2024-02-21T23:06:54.819848+0000","last_undegraded":"2024-02-21T23:06:54.819848+0000","last_fullsized":"2024-02-21T23:06:54.819848+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:12:00.256298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.778304+0000","last_change":"2024-02-21T23:06:53.745467+0000","last_active":"2024-02-21T23:06:54.778304+0000","last_peered":"2024-02-21T23:06:54.778304+0000","last_clean":"2024-02-21T23:06:54.778304+0000","last_became_active":"2024-02-21T23:06:53.745166+0000","last_became_peered":"2024-02-21T23:06:53.745166+0000","last_unstale":"2024-02-21T23:06:54.778304+0000","last_undegraded":"2024-02-21T23:06:54.778304+0000","last_fullsized":"2024-02-21T23:06:54.778304+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_clean_scrub_stamp":"2024-02-21T23:06:52.701904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:20:59.579104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.799463+0000","last_change":"2024-02-21T23:06:54.794516+0000","last_active":"2024-02-21T23:06:54.799463+0000","last_peered":"2024-02-21T23:06:54.799463+0000","last_clean":"2024-02-21T23:06:54.799463+0000","last_became_active":"2024-02-21T23:06:54.794169+0000","last_became_peered":"2024-02-21T23:06:54.794169+0000","last_unstale":"2024-02-21T23:06:54.799463+0000","last_undegraded":"2024-02-21T23:06:54.799463+0000","last_fullsized":"2024-02-21T23:06:54.799463+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:34:18.224729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.787272+0000","last_change":"2024-02-21T23:06:53.749902+0000","last_active":"2024-02-21T23:06:54.787272+0000","last_peered":"2024-02-21T23:06:54.787272+0000","last_clean":"2024-02-21T23:06:54.787272+0000","last_became_active":"2024-02-21T23:06:53.749613+0000","last_became_peered":"2024-02-21T23:06:53.749613+0000","last_unstale":"2024-02-21T23:06:54.787272+0000","last_undegraded":"2024-02-21T23:06:54.787272+0000","last_fullsized":"2024-02-21T23:06:54.787272+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_clean_scrub_stamp":"2024-02-21T23:06:52.701904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:30:59.747020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.799955+0000","last_change":"2024-02-21T23:06:54.792685+0000","last_active":"2024-02-21T23:06:54.799955+0000","last_peered":"2024-02-21T23:06:54.799955+0000","last_clean":"2024-02-21T23:06:54.799955+0000","last_became_active":"2024-02-21T23:06:54.792304+0000","last_became_peered":"2024-02-21T23:06:54.792304+0000","last_unstale":"2024-02-21T23:06:54.799955+0000","last_undegraded":"2024-02-21T23:06:54.799955+0000","last_fullsized":"2024-02-21T23:06:54.799955+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:10:59.467598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.779767+0000","last_change":"2024-02-21T23:06:53.742463+0000","last_active":"2024-02-21T23:06:54.779767+0000","last_peered":"2024-02-21T23:06:54.779767+0000","last_clean":"2024-02-21T23:06:54.779767+0000","last_became_active":"2024-02-21T23:06:53.742119+0000","last_became_peered":"2024-02-21T23:06:53.742119+0000","last_unstale":"2024-02-21T23:06:54.779767+0000","last_undegraded":"2024-02-21T23:06:54.779767+0000","last_fullsized":"2024-02-21T23:06:54.779767+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_clean_scrub_stamp":"2024-02-21T23:06:52.701904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:22:20.809516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.810998+0000","last_change":"2024-02-21T23:06:54.802344+0000","last_active":"2024-02-21T23:06:54.810998+0000","last_peered":"2024-02-21T23:06:54.810998+0000","last_clean":"2024-02-21T23:06:54.810998+0000","last_became_active":"2024-02-21T23:06:54.802038+0000","last_became_peered":"2024-02-21T23:06:54.802038+0000","last_unstale":"2024-02-21T23:06:54.810998+0000","last_undegraded":"2024-02-21T23:06:54.810998+0000","last_fullsized":"2024-02-21T23:06:54.810998+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:53.706442+0000","last_clean_scrub_stamp":"2024-02-21T23:06:53.706442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:40:27.170195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.e","version":"97'2","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.818923+0000","last_change":"2024-02-21T23:06:53.736840+0000","last_active":"2024-02-21T23:06:54.818923+0000","last_peered":"2024-02-21T23:06:54.818923+0000","last_clean":"2024-02-21T23:06:54.818923+0000","last_became_active":"2024-02-21T23:06:53.736495+0000","last_became_peered":"2024-02-21T23:06:53.736495+0000","last_unstale":"2024-02-21T23:06:54.818923+0000","last_undegraded":"2024-02-21T23:06:54.818923+0000","last_fullsized":"2024-02-21T23:06:54.818923+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:06:52.701904+0000","last_clean_scrub_stamp":"2024-02-21T23:06:52.701904+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:37:50.331474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":315,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-02-21T23:06:54.768546+0000","last_change":"2024-02-21T23:02:32.289335+0000","last_active":"2024-02-21T23:06:54.768546+0000","last_peered":"2024-02-21T23:06:54.768546+0000","last_clean":"2024-02-21T23:06:54.768546+0000","last_became_active":"2024-02-21T23:02:32.286913+0000","last_became_peered":"2024-02-21T23:02:32.286913+0000","last_unstale":"2024-02-21T23:06:54.768546+0000","last_undegraded":"2024-02-21T23:06:54.768546+0000","last_fullsized":"2024-02-21T23:06:54.768546+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_clean_scrub_stamp":"2024-02-21T23:02:31.172569+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:36:23.987374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":16,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574903,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32436,"kb_used_data":1248,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339404,"statfs":{"total":96636764160,"available":96603549696,"internally_reserved":0,"allocated":1277952,"data_stored":1178552,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574904,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32376,"kb_used_data":1196,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339464,"statfs":{"total":96636764160,"available":96603611136,"internally_reserved":0,"allocated":1224704,"data_stored":590240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574903,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32360,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339480,"statfs":{"total":96636764160,"available":96603627520,"internally_reserved":0,"allocated":1204224,"data_stored":588024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574904,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32368,"kb_used_data":1184,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339472,"statfs":{"total":96636764160,"available":96603619328,"internally_reserved":0,"allocated":1212416,"data_stored":588116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574904,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32364,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339476,"statfs":{"total":96636764160,"available":96603623424,"internally_reserved":0,"allocated":1208320,"data_stored":588606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574905,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32364,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339476,"statfs":{"total":96636764160,"available":96603623424,"internally_reserved":0,"allocated":1208320,"data_stored":588058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574903,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32368,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339472,"statfs":{"total":96636764160,"available":96603619328,"internally_reserved":0,"allocated":1208320,"data_stored":588058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574903,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32448,"kb_used_data":1268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339392,"statfs":{"total":96636764160,"available":96603537408,"internally_reserved":0,"allocated":1298432,"data_stored":1186786,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-21T23:06:59.850 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-02-21T23:06:59.850 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-02-21T23:07:00.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.179+0000 7f5588e00640 1 Processor -- start 2024-02-21T23:07:00.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.179+0000 7f5588e00640 1 -- start start 2024-02-21T23:07:00.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.181+0000 7f5588e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f558414f3c0 0x7f55840c2570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:00.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.181+0000 7f5588e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f55840c1aa0 0x7f55840c3350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:00.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.181+0000 7f5588e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f55840c3890 0x7f5584151d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:00.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.181+0000 7f5588e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5584071180 con 0x7f55840c1aa0 2024-02-21T23:07:00.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.181+0000 7f5588e00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55840c2ab0 con 0x7f558414f3c0 2024-02-21T23:07:00.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.181+0000 7f5588e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55840c2d80 con 0x7f55840c3890 2024-02-21T23:07:00.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.181+0000 7f5582a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f55840c1aa0 0x7f55840c3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:00.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.181+0000 7f5582a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f55840c1aa0 0x7f55840c3350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47628/0 (socket says 172.21.15.28:47628) 2024-02-21T23:07:00.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.181+0000 7f5582a00640 1 -- 172.21.15.28:0/4151709112 learned_addr learned my addr 172.21.15.28:0/4151709112 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:00.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.182+0000 7f5582a00640 1 -- 172.21.15.28:0/4151709112 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f55840c3890 msgr2=0x7f5584151d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:00.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.182+0000 7f5582a00640 1 --2- 172.21.15.28:0/4151709112 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f55840c3890 0x7f5584151d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:00.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.182+0000 7f5582a00640 1 -- 172.21.15.28:0/4151709112 >> v2:172.21.15.81:3300/0 conn(0x7f558414f3c0 msgr2=0x7f55840c2570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:00.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.182+0000 7f5583400640 1 --2- 172.21.15.28:0/4151709112 >> v2:172.21.15.81:3300/0 conn(0x7f558414f3c0 0x7f55840c2570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:00.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.182+0000 7f5582a00640 1 --2- 172.21.15.28:0/4151709112 >> v2:172.21.15.81:3300/0 conn(0x7f558414f3c0 0x7f55840c2570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:00.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.182+0000 7f5582a00640 1 -- 172.21.15.28:0/4151709112 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5584071420 con 0x7f55840c1aa0 2024-02-21T23:07:00.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.182+0000 7f5583400640 1 --2- 172.21.15.28:0/4151709112 >> v2:172.21.15.81:3300/0 conn(0x7f558414f3c0 0x7f55840c2570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:07:00.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.183+0000 7f5582a00640 1 --2- 172.21.15.28:0/4151709112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55840c1aa0 0x7f55840c3350 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f5570009870 tx=0x7f557002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a59bfadeb08c529 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:00.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.183+0000 7f5582000640 1 -- 172.21.15.28:0/4151709112 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55700317c0 con 0x7f55840c1aa0 2024-02-21T23:07:00.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.183+0000 7f5582000640 1 -- 172.21.15.28:0/4151709112 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5570037070 con 0x7f55840c1aa0 2024-02-21T23:07:00.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.183+0000 7f5582000640 1 -- 172.21.15.28:0/4151709112 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5570031b80 con 0x7f55840c1aa0 2024-02-21T23:07:00.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.184+0000 7f5588e00640 1 -- 172.21.15.28:0/4151709112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55840c1aa0 msgr2=0x7f55840c3350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:00.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.184+0000 7f5588e00640 1 --2- 172.21.15.28:0/4151709112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55840c1aa0 0x7f55840c3350 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f5570009870 tx=0x7f557002b860 comp rx=0 tx=0).stop 2024-02-21T23:07:00.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.184+0000 7f5588e00640 1 -- 172.21.15.28:0/4151709112 shutdown_connections 2024-02-21T23:07:00.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.184+0000 7f5588e00640 1 --2- 172.21.15.28:0/4151709112 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f55840c3890 0x7f5584151d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:00.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.184+0000 7f5588e00640 1 --2- 172.21.15.28:0/4151709112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55840c1aa0 0x7f55840c3350 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:00.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.184+0000 7f5588e00640 1 --2- 172.21.15.28:0/4151709112 >> v2:172.21.15.81:3300/0 conn(0x7f558414f3c0 0x7f55840c2570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:00.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.184+0000 7f5588e00640 1 -- 172.21.15.28:0/4151709112 >> 172.21.15.28:0/4151709112 conn(0x7f55840a8690 msgr2=0x7f55840a8a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:00.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.184+0000 7f5588e00640 1 -- 172.21.15.28:0/4151709112 shutdown_connections 2024-02-21T23:07:00.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.184+0000 7f5588e00640 1 -- 172.21.15.28:0/4151709112 wait complete. 2024-02-21T23:07:00.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.186+0000 7f5588e00640 1 Processor -- start 2024-02-21T23:07:00.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.186+0000 7f5588e00640 1 -- start start 2024-02-21T23:07:00.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.187+0000 7f5588e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f55840c1aa0 0x7f55840c4d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:00.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.187+0000 7f5588e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55840c3890 0x7f55840ab290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:00.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.187+0000 7f5588e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f558414f3c0 0x7f5584149fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:00.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.187+0000 7f5588e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55841522b0 con 0x7f55840c3890 2024-02-21T23:07:00.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.187+0000 7f5588e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55840719e0 con 0x7f558414f3c0 2024-02-21T23:07:00.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.187+0000 7f5588e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5584152430 con 0x7f55840c1aa0 2024-02-21T23:07:00.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.187+0000 7f5582a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55840c3890 0x7f55840ab290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:00.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.187+0000 7f5582a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55840c3890 0x7f55840ab290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47642/0 (socket says 172.21.15.28:47642) 2024-02-21T23:07:00.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.187+0000 7f5582a00640 1 -- 172.21.15.28:0/2551151978 learned_addr learned my addr 172.21.15.28:0/2551151978 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:00.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.187+0000 7f5583400640 1 --2- 172.21.15.28:0/2551151978 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f55840c1aa0 0x7f55840c4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:00.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.187+0000 7f5583e00640 1 --2- 172.21.15.28:0/2551151978 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f558414f3c0 0x7f5584149fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:00.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.187+0000 7f5582a00640 1 -- 172.21.15.28:0/2551151978 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f55840c1aa0 msgr2=0x7f55840c4d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:00.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.187+0000 7f5582a00640 1 --2- 172.21.15.28:0/2551151978 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f55840c1aa0 0x7f55840c4d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:00.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.188+0000 7f5582a00640 1 -- 172.21.15.28:0/2551151978 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f558414f3c0 msgr2=0x7f5584149fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:00.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.188+0000 7f5582a00640 1 --2- 172.21.15.28:0/2551151978 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f558414f3c0 0x7f5584149fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:00.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.188+0000 7f5582a00640 1 -- 172.21.15.28:0/2551151978 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55840c5fc0 con 0x7f55840c3890 2024-02-21T23:07:00.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.188+0000 7f5583e00640 1 --2- 172.21.15.28:0/2551151978 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f558414f3c0 0x7f5584149fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:07:00.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.188+0000 7f5582a00640 1 --2- 172.21.15.28:0/2551151978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55840c3890 0x7f55840ab290 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f5570000c00 tx=0x7f55700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:00.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.188+0000 7f557b400640 1 -- 172.21.15.28:0/2551151978 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5570032860 con 0x7f55840c3890 2024-02-21T23:07:00.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.188+0000 7f557b400640 1 -- 172.21.15.28:0/2551151978 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5570032a00 con 0x7f55840c3890 2024-02-21T23:07:00.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.188+0000 7f5588e00640 1 -- 172.21.15.28:0/2551151978 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55840c62a0 con 0x7f55840c3890 2024-02-21T23:07:00.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.188+0000 7f557b400640 1 -- 172.21.15.28:0/2551151978 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5570032df0 con 0x7f55840c3890 2024-02-21T23:07:00.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.188+0000 7f5588e00640 1 -- 172.21.15.28:0/2551151978 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55840c4580 con 0x7f55840c3890 2024-02-21T23:07:00.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.190+0000 7f5588e00640 1 -- 172.21.15.28:0/2551151978 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f558404f290 con 0x7f55840c3890 2024-02-21T23:07:00.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.191+0000 7f557b400640 1 -- 172.21.15.28:0/2551151978 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f557004d3f0 con 0x7f55840c3890 2024-02-21T23:07:00.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.191+0000 7f557b400640 1 --2- 172.21.15.28:0/2551151978 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f556406b500 0x7f556406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:00.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.192+0000 7f557b400640 1 -- 172.21.15.28:0/2551151978 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f55700ac130 con 0x7f55840c3890 2024-02-21T23:07:00.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.192+0000 7f5583400640 1 --2- 172.21.15.28:0/2551151978 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f556406b500 0x7f556406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:00.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.193+0000 7f5583400640 1 --2- 172.21.15.28:0/2551151978 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f556406b500 0x7f556406d9c0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f557400a950 tx=0x7f5574008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:00.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.196+0000 7f557b400640 1 -- 172.21.15.28:0/2551151978 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f5570040070 con 0x7f55840c3890 2024-02-21T23:07:00.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.536+0000 7f5588e00640 1 -- 172.21.15.28:0/2551151978 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f55840c1880 con 0x7f55840c3890 2024-02-21T23:07:00.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.540+0000 7f557b400640 1 -- 172.21.15.28:0/2551151978 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v14) v1 ==== 190+0+147 (secure 0 0 0) 0x7f557007aea0 con 0x7f55840c3890 2024-02-21T23:07:00.528 INFO:teuthology.orchestra.run.smithi028.stdout:[client.0] 2024-02-21T23:07:00.528 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQDTgNZlILvYLxAAoREWxKKT7YcXnhkgk8YvoQ== 2024-02-21T23:07:00.528 INFO:teuthology.orchestra.run.smithi028.stdout: caps mds = "allow" 2024-02-21T23:07:00.528 INFO:teuthology.orchestra.run.smithi028.stdout: caps mon = "allow r" 2024-02-21T23:07:00.529 INFO:teuthology.orchestra.run.smithi028.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-02-21T23:07:00.529 INFO:teuthology.orchestra.run.smithi028.stderr:updated caps for client.0 2024-02-21T23:07:00.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.543+0000 7f5588e00640 1 -- 172.21.15.28:0/2551151978 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f556406b500 msgr2=0x7f556406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:00.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.543+0000 7f5588e00640 1 --2- 172.21.15.28:0/2551151978 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f556406b500 0x7f556406d9c0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f557400a950 tx=0x7f5574008040 comp rx=0 tx=0).stop 2024-02-21T23:07:00.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.543+0000 7f5588e00640 1 -- 172.21.15.28:0/2551151978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55840c3890 msgr2=0x7f55840ab290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:00.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.543+0000 7f5588e00640 1 --2- 172.21.15.28:0/2551151978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55840c3890 0x7f55840ab290 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f5570000c00 tx=0x7f55700046c0 comp rx=0 tx=0).stop 2024-02-21T23:07:00.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.543+0000 7f5588e00640 1 -- 172.21.15.28:0/2551151978 shutdown_connections 2024-02-21T23:07:00.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.543+0000 7f5588e00640 1 --2- 172.21.15.28:0/2551151978 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f556406b500 0x7f556406d9c0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:00.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.543+0000 7f5588e00640 1 --2- 172.21.15.28:0/2551151978 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f558414f3c0 0x7f5584149fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:00.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.543+0000 7f5588e00640 1 --2- 172.21.15.28:0/2551151978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55840c3890 0x7f55840ab290 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:00.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.543+0000 7f5588e00640 1 --2- 172.21.15.28:0/2551151978 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f55840c1aa0 0x7f55840c4d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:00.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.543+0000 7f5588e00640 1 -- 172.21.15.28:0/2551151978 >> 172.21.15.28:0/2551151978 conn(0x7f55840a8690 msgr2=0x7f55840a8a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:00.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.543+0000 7f5588e00640 1 -- 172.21.15.28:0/2551151978 shutdown_connections 2024-02-21T23:07:00.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.543+0000 7f5588e00640 1 -- 172.21.15.28:0/2551151978 wait complete. 2024-02-21T23:07:00.551 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:07:00.693 INFO:tasks.ceph.mds.a.smithi028.stderr:2024-02-21T23:07:00.705+0000 7fe55f400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-21T23:07:00.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.774+0000 7f52f4c00640 1 Processor -- start 2024-02-21T23:07:00.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.774+0000 7f52f4c00640 1 -- start start 2024-02-21T23:07:00.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.776+0000 7f52f4c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f52f006f330 0x7f52f00c7290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:00.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.776+0000 7f52f4c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f52f00c7be0 0x7f52f014c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:00.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.776+0000 7f52f4c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52f00c5020 con 0x7f52f0070d40 2024-02-21T23:07:00.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.776+0000 7f52f4c00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52f0153a10 con 0x7f52f006f330 2024-02-21T23:07:00.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.776+0000 7f52f4c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52f0153cd0 con 0x7f52f00c7be0 2024-02-21T23:07:00.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.776+0000 7f52ede00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f52f0070d40 0x7f52f00c4910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34626/0 (socket says 172.21.15.28:34626) 2024-02-21T23:07:00.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.776+0000 7f52ede00640 1 -- 172.21.15.28:0/288158567 learned_addr learned my addr 172.21.15.28:0/288158567 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:07:00.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.776+0000 7f52e7e00640 1 --2- 172.21.15.28:0/288158567 >> v2:172.21.15.81:3300/0 conn(0x7f52f006f330 0x7f52f00c7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:00.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.776+0000 7f52ee800640 1 --2- 172.21.15.28:0/288158567 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f52f00c7be0 0x7f52f014c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:00.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.776+0000 7f52ed400640 1 -- 172.21.15.28:0/288158567 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 376756560 0 0) 0x7f52f00c5020 con 0x7f52f0070d40 2024-02-21T23:07:00.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.777+0000 7f52ed400640 1 -- 172.21.15.28:0/288158567 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52d4003520 con 0x7f52f0070d40 2024-02-21T23:07:00.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.777+0000 7f52ed400640 1 -- 172.21.15.28:0/288158567 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1192956343 0 0) 0x7f52d4003520 con 0x7f52f0070d40 2024-02-21T23:07:00.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.777+0000 7f52ed400640 1 -- 172.21.15.28:0/288158567 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f52f00c7be0 msgr2=0x7f52f014c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:00.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.777+0000 7f52ed400640 1 --2- 172.21.15.28:0/288158567 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f52f00c7be0 0x7f52f014c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:00.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.777+0000 7f52ed400640 1 -- 172.21.15.28:0/288158567 >> v2:172.21.15.81:3300/0 conn(0x7f52f006f330 msgr2=0x7f52f00c7290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:00.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.777+0000 7f52ed400640 1 --2- 172.21.15.28:0/288158567 >> v2:172.21.15.81:3300/0 conn(0x7f52f006f330 0x7f52f00c7290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:00.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.777+0000 7f52ed400640 1 -- 172.21.15.28:0/288158567 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52f00c67b0 con 0x7f52f0070d40 2024-02-21T23:07:00.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.777+0000 7f52ed400640 1 -- 172.21.15.28:0/288158567 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f52e0001670 con 0x7f52f0070d40 2024-02-21T23:07:00.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.777+0000 7f52ed400640 1 -- 172.21.15.28:0/288158567 >> v1:172.21.15.28:6789/0 conn(0x7f52f0070d40 legacy=0x7f52f00c4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:00.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.777+0000 7f52ed400640 1 --2- 172.21.15.28:0/288158567 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f52f00c7be0 0x7f52d40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:00.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.777+0000 7f52ed400640 1 --2- 172.21.15.28:0/288158567 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f52d4003f80 0x7f52d4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:00.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.777+0000 7f52ed400640 1 --2- 172.21.15.28:0/288158567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52d4004d40 0x7f52d40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:00.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.777+0000 7f52ed400640 1 -- 172.21.15.28:0/288158567 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52f0153a10 con 0x7f52d4004d40 2024-02-21T23:07:00.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.777+0000 7f52ed400640 1 -- 172.21.15.28:0/288158567 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52f0153cd0 con 0x7f52d4003f80 2024-02-21T23:07:00.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.777+0000 7f52ed400640 1 -- 172.21.15.28:0/288158567 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52d400b9c0 con 0x7f52f00c7be0 2024-02-21T23:07:00.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.777+0000 7f52ee800640 1 --2- 172.21.15.28:0/288158567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52d4004d40 0x7f52d40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:00.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.778+0000 7f52ede00640 1 --2- 172.21.15.28:0/288158567 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f52f00c7be0 0x7f52d40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:00.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.778+0000 7f52e7e00640 1 --2- 172.21.15.28:0/288158567 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f52d4003f80 0x7f52d4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:00.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.778+0000 7f52ee800640 1 -- 172.21.15.28:0/288158567 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f52f00c7be0 msgr2=0x7f52d40039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:00.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.778+0000 7f52ee800640 1 --2- 172.21.15.28:0/288158567 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f52f00c7be0 0x7f52d40039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:00.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.778+0000 7f52ee800640 1 -- 172.21.15.28:0/288158567 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f52d4003f80 msgr2=0x7f52d4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:00.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.778+0000 7f52ee800640 1 --2- 172.21.15.28:0/288158567 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f52d4003f80 0x7f52d4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:00.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.778+0000 7f52ee800640 1 -- 172.21.15.28:0/288158567 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52d4003520 con 0x7f52d4004d40 2024-02-21T23:07:00.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.780+0000 7f52ee800640 1 --2- 172.21.15.28:0/288158567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52d4004d40 0x7f52d40051c0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f52e800e170 tx=0x7f52d4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef761fedaa7f8aea server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:00.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.780+0000 7f52ed400640 1 -- 172.21.15.28:0/288158567 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e8039070 con 0x7f52d4004d40 2024-02-21T23:07:00.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.780+0000 7f52ed400640 1 -- 172.21.15.28:0/288158567 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52e802cbf0 con 0x7f52d4004d40 2024-02-21T23:07:00.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.780+0000 7f52ed400640 1 -- 172.21.15.28:0/288158567 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e8009c10 con 0x7f52d4004d40 2024-02-21T23:07:00.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.781+0000 7f52f4c00640 1 -- 172.21.15.28:0/288158567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52d4004d40 msgr2=0x7f52d40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:00.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.781+0000 7f52f4c00640 1 --2- 172.21.15.28:0/288158567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52d4004d40 0x7f52d40051c0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f52e800e170 tx=0x7f52d4005830 comp rx=0 tx=0).stop 2024-02-21T23:07:00.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.781+0000 7f52ee800640 1 -- 172.21.15.28:0/288158567 reap_dead start 2024-02-21T23:07:00.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.781+0000 7f52f4c00640 1 -- 172.21.15.28:0/288158567 shutdown_connections 2024-02-21T23:07:00.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.781+0000 7f52f4c00640 1 -- 172.21.15.28:0/288158567 >> 172.21.15.28:0/288158567 conn(0x7f52f00c8810 msgr2=0x7f52f00c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:00.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.781+0000 7f52f4c00640 1 -- 172.21.15.28:0/288158567 shutdown_connections 2024-02-21T23:07:00.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.781+0000 7f52f4c00640 1 -- 172.21.15.28:0/288158567 wait complete. 2024-02-21T23:07:00.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.784+0000 7f52f4c00640 1 Processor -- start 2024-02-21T23:07:00.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.784+0000 7f52f4c00640 1 -- start start 2024-02-21T23:07:00.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.786+0000 7f52f4c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52d4004d40 0x7f52f00bf590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:00.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.786+0000 7f52f4c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f52f00bde20 0x7f52f00be2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:00.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.786+0000 7f52f4c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f52f00be7e0 0x7f52f01ede40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:00.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.786+0000 7f52f4c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52f0071550 con 0x7f52d4004d40 2024-02-21T23:07:00.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.786+0000 7f52f4c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52f00c6d70 con 0x7f52f00be7e0 2024-02-21T23:07:00.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.786+0000 7f52f4c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52f0071120 con 0x7f52f00bde20 2024-02-21T23:07:00.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.786+0000 7f52ede00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52d4004d40 0x7f52f00bf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:00.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.786+0000 7f52ede00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52d4004d40 0x7f52f00bf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47652/0 (socket says 172.21.15.28:47652) 2024-02-21T23:07:00.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.786+0000 7f52ede00640 1 -- 172.21.15.28:0/703649080 learned_addr learned my addr 172.21.15.28:0/703649080 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:00.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.786+0000 7f52e7e00640 1 --2- 172.21.15.28:0/703649080 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f52f00bde20 0x7f52f00be2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:00.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.786+0000 7f52ede00640 1 -- 172.21.15.28:0/703649080 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f52f00bde20 msgr2=0x7f52f00be2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:00.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.786+0000 7f52ede00640 1 --2- 172.21.15.28:0/703649080 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f52f00bde20 0x7f52f00be2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:00.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.786+0000 7f52ede00640 1 -- 172.21.15.28:0/703649080 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f52f00be7e0 msgr2=0x7f52f01ede40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:00.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.786+0000 7f52ee800640 1 --2- 172.21.15.28:0/703649080 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f52f00be7e0 0x7f52f01ede40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:00.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.787+0000 7f52ede00640 1 --2- 172.21.15.28:0/703649080 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f52f00be7e0 0x7f52f01ede40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:00.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.787+0000 7f52ede00640 1 -- 172.21.15.28:0/703649080 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52f01eaa70 con 0x7f52d4004d40 2024-02-21T23:07:00.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.787+0000 7f52e7e00640 1 --2- 172.21.15.28:0/703649080 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f52f00bde20 0x7f52f00be2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:07:00.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.787+0000 7f52ee800640 1 --2- 172.21.15.28:0/703649080 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f52f00be7e0 0x7f52f01ede40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:07:00.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.787+0000 7f52ede00640 1 --2- 172.21.15.28:0/703649080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52d4004d40 0x7f52f00bf590 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f52e000e870 tx=0x7f52e000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:00.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.787+0000 7f52e6000640 1 -- 172.21.15.28:0/703649080 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e000c870 con 0x7f52d4004d40 2024-02-21T23:07:00.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.787+0000 7f52e6000640 1 -- 172.21.15.28:0/703649080 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52e0018070 con 0x7f52d4004d40 2024-02-21T23:07:00.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.787+0000 7f52e6000640 1 -- 172.21.15.28:0/703649080 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e000cc70 con 0x7f52d4004d40 2024-02-21T23:07:00.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.787+0000 7f52f4c00640 1 -- 172.21.15.28:0/703649080 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52f01ead50 con 0x7f52d4004d40 2024-02-21T23:07:00.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.787+0000 7f52f4c00640 1 -- 172.21.15.28:0/703649080 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52f01eb310 con 0x7f52d4004d40 2024-02-21T23:07:00.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.789+0000 7f52f4c00640 1 -- 172.21.15.28:0/703649080 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52f0068640 con 0x7f52d4004d40 2024-02-21T23:07:00.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.790+0000 7f52e6000640 1 -- 172.21.15.28:0/703649080 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f52e001d460 con 0x7f52d4004d40 2024-02-21T23:07:00.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.791+0000 7f52e6000640 1 --2- 172.21.15.28:0/703649080 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f52cc06b4d0 0x7f52cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:00.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.791+0000 7f52e6000640 1 -- 172.21.15.28:0/703649080 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f52e008a260 con 0x7f52d4004d40 2024-02-21T23:07:00.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.791+0000 7f52e7e00640 1 --2- 172.21.15.28:0/703649080 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f52cc06b4d0 0x7f52cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:00.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.792+0000 7f52e7e00640 1 --2- 172.21.15.28:0/703649080 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f52cc06b4d0 0x7f52cc06d990 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f52d8009a00 tx=0x7f52d8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:00.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:00.795+0000 7f52e6000640 1 -- 172.21.15.28:0/703649080 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f52e0058fe0 con 0x7f52d4004d40 2024-02-21T23:07:01.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.035+0000 7f52f4c00640 1 -- 172.21.15.28:0/703649080 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f52f00c8360 con 0x7f52d4004d40 2024-02-21T23:07:01.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.037+0000 7f52e6000640 1 -- 172.21.15.28:0/703649080 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+4934 (secure 0 0 0) 0x7f52e00588e0 con 0x7f52d4004d40 2024-02-21T23:07:01.025 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:07:01.025 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":88,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5555,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/85689296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":85689296},{"type":"v1","addr":"172.21.15.28:6833","nonce":85689296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82},{"gid":5567,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1194613007","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1194613007},{"type":"v1","addr":"172.21.15.81:6835","nonce":1194613007}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":5570,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/2315735157","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":2315735157},{"type":"v1","addr":"172.21.15.81:6837","nonce":2315735157}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:06:54.201589+0000","modified":"2024-02-21T23:06:57.273466+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5932},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5932":{"gid":5932,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/2477712025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2477712025},{"type":"v1","addr":"172.21.15.28:6835","nonce":2477712025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-02-21T23:07:01.025 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 88 2024-02-21T23:07:01.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.040+0000 7f52f4c00640 1 -- 172.21.15.28:0/703649080 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f52cc06b4d0 msgr2=0x7f52cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:01.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.040+0000 7f52f4c00640 1 --2- 172.21.15.28:0/703649080 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f52cc06b4d0 0x7f52cc06d990 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f52d8009a00 tx=0x7f52d8008040 comp rx=0 tx=0).stop 2024-02-21T23:07:01.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.040+0000 7f52f4c00640 1 -- 172.21.15.28:0/703649080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52d4004d40 msgr2=0x7f52f00bf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:01.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.040+0000 7f52f4c00640 1 --2- 172.21.15.28:0/703649080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52d4004d40 0x7f52f00bf590 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f52e000e870 tx=0x7f52e000ed30 comp rx=0 tx=0).stop 2024-02-21T23:07:01.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.040+0000 7f52f4c00640 1 -- 172.21.15.28:0/703649080 shutdown_connections 2024-02-21T23:07:01.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.040+0000 7f52f4c00640 1 --2- 172.21.15.28:0/703649080 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f52cc06b4d0 0x7f52cc06d990 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:01.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.040+0000 7f52f4c00640 1 --2- 172.21.15.28:0/703649080 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f52f00bde20 0x7f52f00be2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:01.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.040+0000 7f52f4c00640 1 --2- 172.21.15.28:0/703649080 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f52f00be7e0 0x7f52f01ede40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:01.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.040+0000 7f52f4c00640 1 --2- 172.21.15.28:0/703649080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52d4004d40 0x7f52f00bf590 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:01.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.040+0000 7f52f4c00640 1 -- 172.21.15.28:0/703649080 >> 172.21.15.28:0/703649080 conn(0x7f52f00c8810 msgr2=0x7f52f00aa5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:01.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.040+0000 7f52f4c00640 1 -- 172.21.15.28:0/703649080 shutdown_connections 2024-02-21T23:07:01.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.040+0000 7f52f4c00640 1 -- 172.21.15.28:0/703649080 wait complete. 2024-02-21T23:07:01.450 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:06:54.201589+0000', 'modified': '2024-02-21T23:06:57.273466+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5932}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5932': {'gid': 5932, 'name': 'a', 'rank': 0, 'incarnation': 84, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/2477712025', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 2477712025}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 2477712025}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:07:01.451 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-21T23:07:01.451 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-02-21T23:07:01.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.692+0000 7f6f09800640 1 Processor -- start 2024-02-21T23:07:01.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.692+0000 7f6f09800640 1 -- start start 2024-02-21T23:07:01.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.694+0000 7f6f09800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6f041524c0 0x7f6f04155a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:01.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.694+0000 7f6f09800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6f040c5040 0x7f6f0406ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:01.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.694+0000 7f6f09800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f0406df50 con 0x7f6f040c85c0 2024-02-21T23:07:01.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.694+0000 7f6f09800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f0406d3b0 con 0x7f6f041524c0 2024-02-21T23:07:01.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.694+0000 7f6f09800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f040c5520 con 0x7f6f040c5040 2024-02-21T23:07:01.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.694+0000 7f6f03e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6f041524c0 0x7f6f04155a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:01.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.694+0000 7f6f03e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6f041524c0 0x7f6f04155a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47654/0 (socket says 172.21.15.28:47654) 2024-02-21T23:07:01.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.694+0000 7f6f03e00640 1 -- 172.21.15.28:0/2896097751 learned_addr learned my addr 172.21.15.28:0/2896097751 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:01.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.694+0000 7f6f08e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f6f040c85c0 0x7f6f040c70d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34642/0 (socket says 172.21.15.28:34642) 2024-02-21T23:07:01.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.695+0000 7f6efb400640 1 --2- 172.21.15.28:0/2896097751 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6f040c5040 0x7f6f0406ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:01.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.695+0000 7f6f03e00640 1 -- 172.21.15.28:0/2896097751 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6f040c5040 msgr2=0x7f6f0406ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:01.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.695+0000 7f6f03e00640 1 --2- 172.21.15.28:0/2896097751 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6f040c5040 0x7f6f0406ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:01.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.695+0000 7f6f03e00640 1 -- 172.21.15.28:0/2896097751 >> v1:172.21.15.28:6789/0 conn(0x7f6f040c85c0 legacy=0x7f6f040c70d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:01.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.695+0000 7f6f03400640 1 -- 172.21.15.28:0/2896097751 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2395107283 0 0) 0x7f6f0406df50 con 0x7f6f040c85c0 2024-02-21T23:07:01.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.695+0000 7f6f03e00640 1 -- 172.21.15.28:0/2896097751 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f040c5800 con 0x7f6f041524c0 2024-02-21T23:07:01.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.696+0000 7f6f03e00640 1 --2- 172.21.15.28:0/2896097751 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f041524c0 0x7f6f04155a10 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f6eec009870 tx=0x7f6eec02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e300042fa0293f36 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:01.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.696+0000 7f6f03400640 1 -- 172.21.15.28:0/2896097751 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eec0317c0 con 0x7f6f041524c0 2024-02-21T23:07:01.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.696+0000 7f6f03400640 1 -- 172.21.15.28:0/2896097751 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6eec037070 con 0x7f6f041524c0 2024-02-21T23:07:01.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.696+0000 7f6f03400640 1 -- 172.21.15.28:0/2896097751 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eec031b80 con 0x7f6f041524c0 2024-02-21T23:07:01.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.697+0000 7f6f09800640 1 -- 172.21.15.28:0/2896097751 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f041524c0 msgr2=0x7f6f04155a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:01.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.697+0000 7f6f09800640 1 --2- 172.21.15.28:0/2896097751 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f041524c0 0x7f6f04155a10 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f6eec009870 tx=0x7f6eec02b860 comp rx=0 tx=0).stop 2024-02-21T23:07:01.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.697+0000 7f6f09800640 1 -- 172.21.15.28:0/2896097751 shutdown_connections 2024-02-21T23:07:01.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.697+0000 7f6f09800640 1 --2- 172.21.15.28:0/2896097751 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6f040c5040 0x7f6f0406ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:01.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.697+0000 7f6f09800640 1 --2- 172.21.15.28:0/2896097751 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f041524c0 0x7f6f04155a10 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:01.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.697+0000 7f6f09800640 1 -- 172.21.15.28:0/2896097751 >> 172.21.15.28:0/2896097751 conn(0x7f6f040ac2b0 msgr2=0x7f6f040ac6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:01.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.697+0000 7f6f09800640 1 -- 172.21.15.28:0/2896097751 shutdown_connections 2024-02-21T23:07:01.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.697+0000 7f6f09800640 1 -- 172.21.15.28:0/2896097751 wait complete. 2024-02-21T23:07:01.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.699+0000 7f6f09800640 1 Processor -- start 2024-02-21T23:07:01.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.699+0000 7f6f09800640 1 -- start start 2024-02-21T23:07:01.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.700+0000 7f6f09800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6f040c5040 0x7f6f040c28d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:01.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.700+0000 7f6f09800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f040c85c0 0x7f6f041e7a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:01.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.700+0000 7f6f09800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f041524c0 0x7f6f041e7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:01.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.700+0000 7f6f09800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f041528a0 con 0x7f6f041524c0 2024-02-21T23:07:01.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.700+0000 7f6f09800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f0406c950 con 0x7f6f040c85c0 2024-02-21T23:07:01.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.700+0000 7f6f09800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f040c8a70 con 0x7f6f040c5040 2024-02-21T23:07:01.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.700+0000 7f6f08e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f041524c0 0x7f6f041e7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:01.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.700+0000 7f6f08e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f041524c0 0x7f6f041e7fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47656/0 (socket says 172.21.15.28:47656) 2024-02-21T23:07:01.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.700+0000 7f6f08e00640 1 -- 172.21.15.28:0/446297645 learned_addr learned my addr 172.21.15.28:0/446297645 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:01.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.700+0000 7f6f03e00640 1 --2- 172.21.15.28:0/446297645 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6f040c5040 0x7f6f040c28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:01.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.700+0000 7f6efb400640 1 --2- 172.21.15.28:0/446297645 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f040c85c0 0x7f6f041e7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:01.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.701+0000 7f6f08e00640 1 -- 172.21.15.28:0/446297645 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6f040c5040 msgr2=0x7f6f040c28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:01.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.701+0000 7f6f08e00640 1 --2- 172.21.15.28:0/446297645 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6f040c5040 0x7f6f040c28d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:01.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.701+0000 7f6f08e00640 1 -- 172.21.15.28:0/446297645 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f040c85c0 msgr2=0x7f6f041e7a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:01.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.701+0000 7f6f08e00640 1 --2- 172.21.15.28:0/446297645 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f040c85c0 0x7f6f041e7a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:01.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.701+0000 7f6f08e00640 1 -- 172.21.15.28:0/446297645 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f040bf690 con 0x7f6f041524c0 2024-02-21T23:07:01.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.701+0000 7f6f08e00640 1 --2- 172.21.15.28:0/446297645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f041524c0 0x7f6f041e7fa0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f6efc00c7a0 tx=0x7f6efc00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:01.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.701+0000 7f6f00c00640 1 -- 172.21.15.28:0/446297645 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6efc018070 con 0x7f6f041524c0 2024-02-21T23:07:01.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.701+0000 7f6f00c00640 1 -- 172.21.15.28:0/446297645 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6efc00a640 con 0x7f6f041524c0 2024-02-21T23:07:01.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.701+0000 7f6f00c00640 1 -- 172.21.15.28:0/446297645 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6efc00aab0 con 0x7f6f041524c0 2024-02-21T23:07:01.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.701+0000 7f6f09800640 1 -- 172.21.15.28:0/446297645 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f040bf970 con 0x7f6f041524c0 2024-02-21T23:07:01.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.701+0000 7f6f09800640 1 -- 172.21.15.28:0/446297645 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f040c1790 con 0x7f6f041524c0 2024-02-21T23:07:01.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.704+0000 7f6f00c00640 1 -- 172.21.15.28:0/446297645 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6efc00ac50 con 0x7f6f041524c0 2024-02-21T23:07:01.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.705+0000 7f6f00c00640 1 --2- 172.21.15.28:0/446297645 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6ee006b500 0x7f6ee006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:01.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.705+0000 7f6f00c00640 1 -- 172.21.15.28:0/446297645 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6efc089970 con 0x7f6f041524c0 2024-02-21T23:07:01.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.706+0000 7f6f0dd87740 1 Processor -- start 2024-02-21T23:07:01.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.706+0000 7f6f03e00640 1 --2- 172.21.15.28:0/446297645 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6ee006b500 0x7f6ee006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:01.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.706+0000 7f6f0dd87740 1 -- start start 2024-02-21T23:07:01.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.707+0000 7f6f0dd87740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55f3cb5a9020 0x55f3cb5a9400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:01.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.707+0000 7f6f0dd87740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f3cb5aa330 0x55f3cb5e5170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:01.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.707+0000 7f6f0dd87740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3cb5e56b0 0x55f3cb5e3720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:01.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.707+0000 7f6f0dd87740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3cb50d470 con 0x55f3cb5e56b0 2024-02-21T23:07:01.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.707+0000 7f6f0dd87740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3cb4b6af0 con 0x55f3cb5aa330 2024-02-21T23:07:01.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.707+0000 7f6f0dd87740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f3cb509510 con 0x55f3cb5a9020 2024-02-21T23:07:01.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.707+0000 7f6f08e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3cb5e56b0 0x55f3cb5e3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:01.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.707+0000 7f6f08e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3cb5e56b0 0x55f3cb5e3720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47668/0 (socket says 172.21.15.28:47668) 2024-02-21T23:07:01.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.707+0000 7f6f08e00640 1 -- 172.21.15.28:0/915896679 learned_addr learned my addr 172.21.15.28:0/915896679 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:01.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.707+0000 7f6f03e00640 1 --2- 172.21.15.28:0/446297645 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6ee006b500 0x7f6ee006d9c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f6eec036000 tx=0x7f6eec03d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:01.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.708+0000 7f6f03e00640 1 --2- 172.21.15.28:0/915896679 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55f3cb5a9020 0x55f3cb5a9400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:01.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.708+0000 7f6efb400640 1 --2- 172.21.15.28:0/915896679 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f3cb5aa330 0x55f3cb5e5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:01.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.708+0000 7f6f08e00640 1 -- 172.21.15.28:0/915896679 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55f3cb5a9020 msgr2=0x55f3cb5a9400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:01.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.708+0000 7f6f08e00640 1 --2- 172.21.15.28:0/915896679 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55f3cb5a9020 0x55f3cb5a9400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:01.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.708+0000 7f6f08e00640 1 -- 172.21.15.28:0/915896679 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f3cb5aa330 msgr2=0x55f3cb5e5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:01.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.708+0000 7f6f08e00640 1 --2- 172.21.15.28:0/915896679 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f3cb5aa330 0x55f3cb5e5170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:01.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.708+0000 7f6f08e00640 1 -- 172.21.15.28:0/915896679 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f3cb4bdd90 con 0x55f3cb5e56b0 2024-02-21T23:07:01.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.708+0000 7f6efb400640 1 --2- 172.21.15.28:0/915896679 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f3cb5aa330 0x55f3cb5e5170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:07:01.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.708+0000 7f6f08e00640 1 --2- 172.21.15.28:0/915896679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3cb5e56b0 0x55f3cb5e3720 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f6efc087c40 tx=0x7f6efc054320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34adc83c27b34ead server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:01.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.708+0000 7f6ef8c00640 1 -- 172.21.15.28:0/915896679 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6efc05c0a0 con 0x55f3cb5e56b0 2024-02-21T23:07:01.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.708+0000 7f6ef8c00640 1 -- 172.21.15.28:0/915896679 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6efc060660 con 0x55f3cb5e56b0 2024-02-21T23:07:01.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.708+0000 7f6ef8c00640 1 -- 172.21.15.28:0/915896679 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6efc0545c0 con 0x55f3cb5e56b0 2024-02-21T23:07:01.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.709+0000 7f6f0dd87740 1 -- 172.21.15.28:0/915896679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3cb5e56b0 msgr2=0x55f3cb5e3720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:01.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.709+0000 7f6f0dd87740 1 --2- 172.21.15.28:0/915896679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3cb5e56b0 0x55f3cb5e3720 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f6efc087c40 tx=0x7f6efc054320 comp rx=0 tx=0).stop 2024-02-21T23:07:01.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.709+0000 7f6f0dd87740 1 -- 172.21.15.28:0/915896679 shutdown_connections 2024-02-21T23:07:01.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.709+0000 7f6f0dd87740 1 --2- 172.21.15.28:0/915896679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3cb5e56b0 0x55f3cb5e3720 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:01.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.709+0000 7f6f0dd87740 1 --2- 172.21.15.28:0/915896679 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f3cb5aa330 0x55f3cb5e5170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:01.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.709+0000 7f6f0dd87740 1 --2- 172.21.15.28:0/915896679 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55f3cb5a9020 0x55f3cb5a9400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:01.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.709+0000 7f6f0dd87740 1 -- 172.21.15.28:0/915896679 >> 172.21.15.28:0/915896679 conn(0x55f3cb5a5970 msgr2=0x55f3cb5ca380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:01.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.709+0000 7f6f0dd87740 1 -- 172.21.15.28:0/915896679 shutdown_connections 2024-02-21T23:07:01.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.709+0000 7f6f0dd87740 1 -- 172.21.15.28:0/915896679 wait complete. 2024-02-21T23:07:01.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.710+0000 7f6f0dd87740 1 -- start start 2024-02-21T23:07:01.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.710+0000 7f6f0dd87740 1 Processor -- start 2024-02-21T23:07:01.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.711+0000 7f6f0dd87740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3cb5aa330 0x55f3cb5eb8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:01.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.711+0000 7f6f0dd87740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f3cb5e56b0 0x55f3cb5f8b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:01.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.711+0000 7f6f0dd87740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55f3cb5ebe20 0x55f3cb5f9050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:01.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.711+0000 7f6f0dd87740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3cb4ac040 con 0x55f3cb5aa330 2024-02-21T23:07:01.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.711+0000 7f6f0dd87740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3cb5caa90 con 0x55f3cb5e56b0 2024-02-21T23:07:01.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.711+0000 7f6f0dd87740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f3cb5ec600 con 0x55f3cb5ebe20 2024-02-21T23:07:01.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.711+0000 7f6f0dd87740 10 client.0 _ll_register_callbacks cb 0x55f3cb4c9920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:07:01.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.711+0000 7f6f03e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3cb5aa330 0x55f3cb5eb8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:01.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.711+0000 7f6f03e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3cb5aa330 0x55f3cb5eb8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47678/0 (socket says 172.21.15.28:47678) 2024-02-21T23:07:01.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.711+0000 7f6f03e00640 1 -- 172.21.15.28:0/2075199265 learned_addr learned my addr 172.21.15.28:0/2075199265 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:01.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.712+0000 7f6efb400640 1 --2- 172.21.15.28:0/2075199265 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f3cb5e56b0 0x55f3cb5f8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:01.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.712+0000 7f6f08e00640 1 --2- 172.21.15.28:0/2075199265 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55f3cb5ebe20 0x55f3cb5f9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:01.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.712+0000 7f6f03e00640 1 -- 172.21.15.28:0/2075199265 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55f3cb5ebe20 msgr2=0x55f3cb5f9050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:01.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.712+0000 7f6f03e00640 1 --2- 172.21.15.28:0/2075199265 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55f3cb5ebe20 0x55f3cb5f9050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:01.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.712+0000 7f6f03e00640 1 -- 172.21.15.28:0/2075199265 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f3cb5e56b0 msgr2=0x55f3cb5f8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:01.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.712+0000 7f6f03e00640 1 --2- 172.21.15.28:0/2075199265 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f3cb5e56b0 0x55f3cb5f8b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:01.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.712+0000 7f6f03e00640 1 -- 172.21.15.28:0/2075199265 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f3cb4cbaa0 con 0x55f3cb5aa330 2024-02-21T23:07:01.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.712+0000 7f6f03e00640 1 --2- 172.21.15.28:0/2075199265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3cb5aa330 0x55f3cb5eb8e0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f6eec009870 tx=0x7f6eec02e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b28a944d85f54b3b server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:01.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.712+0000 7f6ef1600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:07:01.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.712+0000 7f6ef1600640 1 -- 172.21.15.28:0/2075199265 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eec054060 con 0x55f3cb5aa330 2024-02-21T23:07:01.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.712+0000 7f6f0dd87740 1 -- 172.21.15.28:0/2075199265 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6efc060660 con 0x55f3cb5aa330 2024-02-21T23:07:01.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.712+0000 7f6ef1600640 1 -- 172.21.15.28:0/2075199265 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6eec045600 con 0x55f3cb5aa330 2024-02-21T23:07:01.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.713+0000 7f6ef1600640 1 -- 172.21.15.28:0/2075199265 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=88) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6eec032e30 con 0x55f3cb5aa330 2024-02-21T23:07:01.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.713+0000 7f6f0dd87740 10 client.7200 fetch_fsmap learned FSMap version 88 2024-02-21T23:07:01.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.713+0000 7f6f0dd87740 1 -- 172.21.15.28:0/2075199265 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=88,monmap=2+}) v3 -- 0x55f3cb2222d0 con 0x55f3cb5aa330 2024-02-21T23:07:01.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.713+0000 7f6ef1600640 1 -- 172.21.15.28:0/2075199265 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6efc060660 con 0x55f3cb5aa330 2024-02-21T23:07:01.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.713+0000 7f6ef1600640 1 -- 172.21.15.28:0/2075199265 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 88) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f6eec045800 con 0x55f3cb5aa330 2024-02-21T23:07:01.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.713+0000 7f6f0dd87740 10 client.7200 fetch_fsmap finished waiting for FSMap version 88 2024-02-21T23:07:01.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.713+0000 7f6f0dd87740 10 client.7200 resolve_mds: resolved name 'a' to daemon mds.a 2024-02-21T23:07:01.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.713+0000 7f6f0dd87740 20 client.7200 populate_metadata read hostname 'smithi028' 2024-02-21T23:07:01.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.713+0000 7f6f0dd87740 1 --2- 172.21.15.28:0/2075199265 >> [v2:172.21.15.28:6834/2477712025,v1:172.21.15.28:6835/2477712025] conn(0x55f3cb5c89c0 0x55f3cb5c8da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:01.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.713+0000 7f6f0dd87740 4 client.7200 mds_command: new command op to 5932 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-21T23:07:01.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.713+0000 7f6f0dd87740 1 -- 172.21.15.28:0/2075199265 --> [v2:172.21.15.28:6834/2477712025,v1:172.21.15.28:6835/2477712025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f3cb51a9b0 con 0x55f3cb5c89c0 2024-02-21T23:07:01.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.714+0000 7f6efb400640 1 --2- 172.21.15.28:0/2075199265 >> [v2:172.21.15.28:6834/2477712025,v1:172.21.15.28:6835/2477712025] conn(0x55f3cb5c89c0 0x55f3cb5c8da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:01.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.714+0000 7f6efb400640 1 --2- 172.21.15.28:0/2075199265 >> [v2:172.21.15.28:6834/2477712025,v1:172.21.15.28:6835/2477712025] conn(0x55f3cb5c89c0 0x55f3cb5c8da0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d8c3dfa2e422e68 server_cookie=a485e988953910e8 in_seq=0 out_seq=0 2024-02-21T23:07:01.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.714+0000 7f6ef1600640 10 client.7200 ms_handle_connect on v2:172.21.15.28:6834/2477712025 2024-02-21T23:07:01.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.715+0000 7f6ef1600640 1 -- 172.21.15.28:0/2075199265 <== mds.0 v2:172.21.15.28:6834/2477712025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f3cb51a9b0 con 0x55f3cb5c89c0 2024-02-21T23:07:01.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.715+0000 7f6ef1600640 10 client.7200 handle_command_reply: tid=0 2024-02-21T23:07:01.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.715+0000 7f6f0dd87740 1 client.7200 shutdown 2024-02-21T23:07:01.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.716+0000 7f6f0dd87740 1 -- 172.21.15.28:0/2075199265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3cb5aa330 msgr2=0x55f3cb5eb8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:01.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.716+0000 7f6f0dd87740 1 --2- 172.21.15.28:0/2075199265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3cb5aa330 0x55f3cb5eb8e0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f6eec009870 tx=0x7f6eec02e000 comp rx=0 tx=0).stop 2024-02-21T23:07:01.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.716+0000 7f6f0dd87740 1 -- 172.21.15.28:0/2075199265 shutdown_connections 2024-02-21T23:07:01.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.716+0000 7f6f0dd87740 1 --2- 172.21.15.28:0/2075199265 >> [v2:172.21.15.28:6834/2477712025,v1:172.21.15.28:6835/2477712025] conn(0x55f3cb5c89c0 0x55f3cb5c8da0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:01.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.716+0000 7f6f0dd87740 1 --2- 172.21.15.28:0/2075199265 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55f3cb5ebe20 0x55f3cb5f9050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:01.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.716+0000 7f6f0dd87740 1 --2- 172.21.15.28:0/2075199265 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f3cb5e56b0 0x55f3cb5f8b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:01.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.716+0000 7f6f0dd87740 1 --2- 172.21.15.28:0/2075199265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3cb5aa330 0x55f3cb5eb8e0 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:01.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.716+0000 7f6ef1600640 0 client.7200 ms_handle_reset on v2:172.21.15.28:6834/2477712025 2024-02-21T23:07:01.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.716+0000 7f6f0dd87740 1 -- 172.21.15.28:0/2075199265 >> 172.21.15.28:0/2075199265 conn(0x55f3cb5a9020 msgr2=0x55f3cb5e8790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:01.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.716+0000 7f6f0dd87740 1 -- 172.21.15.28:0/2075199265 shutdown_connections 2024-02-21T23:07:01.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.716+0000 7f6f0dd87740 1 -- 172.21.15.28:0/2075199265 wait complete. 2024-02-21T23:07:01.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.716+0000 7f6f0dd87740 20 client.7200 trim_cache size 0 max 16384 2024-02-21T23:07:01.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.735+0000 7f6f0dd87740 1 Processor -- start 2024-02-21T23:07:01.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.735+0000 7f6f0dd87740 1 -- start start 2024-02-21T23:07:01.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.735+0000 7f6f0dd87740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3cb5aa330 0x55f3cb5f8b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:01.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.736+0000 7f6f0dd87740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55f3cb5e56b0 0x55f3cb5c8fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:01.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.736+0000 7f6f0dd87740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f3cb5ebe20 0x55f3cb5c9510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:01.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.736+0000 7f6f0dd87740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f041f69b0 con 0x55f3cb5aa330 2024-02-21T23:07:01.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.736+0000 7f6f0dd87740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f041eb2c0 con 0x55f3cb5ebe20 2024-02-21T23:07:01.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.736+0000 7f6f0dd87740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f041f63b0 con 0x55f3cb5e56b0 2024-02-21T23:07:01.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.736+0000 7f6f03e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3cb5aa330 0x55f3cb5f8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:01.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.736+0000 7f6f03e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3cb5aa330 0x55f3cb5f8b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47694/0 (socket says 172.21.15.28:47694) 2024-02-21T23:07:01.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.736+0000 7f6f03e00640 1 -- 172.21.15.28:0/4160884004 learned_addr learned my addr 172.21.15.28:0/4160884004 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:01.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.736+0000 7f6f08e00640 1 --2- 172.21.15.28:0/4160884004 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f3cb5ebe20 0x55f3cb5c9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:01.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.736+0000 7f6f03e00640 1 -- 172.21.15.28:0/4160884004 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55f3cb5e56b0 msgr2=0x55f3cb5c8fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:01.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.736+0000 7f6f03e00640 1 --2- 172.21.15.28:0/4160884004 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55f3cb5e56b0 0x55f3cb5c8fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:01.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.736+0000 7f6f03e00640 1 -- 172.21.15.28:0/4160884004 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f3cb5ebe20 msgr2=0x55f3cb5c9510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:01.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.736+0000 7f6f03e00640 1 --2- 172.21.15.28:0/4160884004 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f3cb5ebe20 0x55f3cb5c9510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:01.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.736+0000 7f6f03e00640 1 -- 172.21.15.28:0/4160884004 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f3cb2ec6b0 con 0x55f3cb5aa330 2024-02-21T23:07:01.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.737+0000 7f6f03e00640 1 --2- 172.21.15.28:0/4160884004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3cb5aa330 0x55f3cb5f8b10 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f6eec006b10 tx=0x7f6eec009240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=899226c38e1797ab server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:01.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.737+0000 7f6ef3e00640 1 -- 172.21.15.28:0/4160884004 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eec0458c0 con 0x55f3cb5aa330 2024-02-21T23:07:01.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.737+0000 7f6ef3e00640 1 -- 172.21.15.28:0/4160884004 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6eec037040 con 0x55f3cb5aa330 2024-02-21T23:07:01.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.737+0000 7f6ef3e00640 1 -- 172.21.15.28:0/4160884004 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eec004230 con 0x55f3cb5aa330 2024-02-21T23:07:01.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.738+0000 7f6f0dd87740 1 -- 172.21.15.28:0/4160884004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3cb5aa330 msgr2=0x55f3cb5f8b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:01.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.738+0000 7f6f0dd87740 1 --2- 172.21.15.28:0/4160884004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3cb5aa330 0x55f3cb5f8b10 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f6eec006b10 tx=0x7f6eec009240 comp rx=0 tx=0).stop 2024-02-21T23:07:01.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.738+0000 7f6f0dd87740 1 -- 172.21.15.28:0/4160884004 shutdown_connections 2024-02-21T23:07:01.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.738+0000 7f6f0dd87740 1 --2- 172.21.15.28:0/4160884004 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55f3cb5e56b0 0x55f3cb5c8fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:01.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.738+0000 7f6f0dd87740 1 --2- 172.21.15.28:0/4160884004 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f3cb5ebe20 0x55f3cb5c9510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:01.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.738+0000 7f6f0dd87740 1 --2- 172.21.15.28:0/4160884004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3cb5aa330 0x55f3cb5f8b10 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:01.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.738+0000 7f6f0dd87740 1 -- 172.21.15.28:0/4160884004 >> 172.21.15.28:0/4160884004 conn(0x55f3cb5a9020 msgr2=0x55f3cb5f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:01.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.738+0000 7f6f0dd87740 1 -- 172.21.15.28:0/4160884004 shutdown_connections 2024-02-21T23:07:01.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.738+0000 7f6f0dd87740 1 -- 172.21.15.28:0/4160884004 wait complete. 2024-02-21T23:07:01.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.738+0000 7f6f0dd87740 1 -- start start 2024-02-21T23:07:01.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.739+0000 7f6f0dd87740 1 Processor -- start 2024-02-21T23:07:01.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.739+0000 7f6f0dd87740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3cb5aa330 0x55f3cb5c96b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:01.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.739+0000 7f6f0dd87740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55f3cb5e56b0 0x55f3cb5c1d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:01.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.739+0000 7f6f0dd87740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f3cb5ebe20 0x55f3cb653fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:01.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.739+0000 7f6f0dd87740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3cb5cc2a0 con 0x55f3cb5aa330 2024-02-21T23:07:01.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.739+0000 7f6f0dd87740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3cb5ec340 con 0x55f3cb5ebe20 2024-02-21T23:07:01.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.739+0000 7f6f0dd87740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f3cb5c57a0 con 0x55f3cb5e56b0 2024-02-21T23:07:01.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.740+0000 7f6f0dd87740 10 client.0 _ll_register_callbacks cb 0x7f6eec0096f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:07:01.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.740+0000 7f6f03e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3cb5aa330 0x55f3cb5c96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:01.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.740+0000 7f6f03e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3cb5aa330 0x55f3cb5c96b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47708/0 (socket says 172.21.15.28:47708) 2024-02-21T23:07:01.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.740+0000 7f6f03e00640 1 -- 172.21.15.28:0/267361921 learned_addr learned my addr 172.21.15.28:0/267361921 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:01.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.740+0000 7f6f03e00640 1 -- 172.21.15.28:0/267361921 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55f3cb5e56b0 msgr2=0x55f3cb5c1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:01.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.740+0000 7f6f03e00640 1 --2- 172.21.15.28:0/267361921 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55f3cb5e56b0 0x55f3cb5c1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:01.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.740+0000 7f6f03e00640 1 -- 172.21.15.28:0/267361921 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f3cb5ebe20 msgr2=0x55f3cb653fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:01.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.740+0000 7f6f03e00640 1 --2- 172.21.15.28:0/267361921 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f3cb5ebe20 0x55f3cb653fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:01.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.740+0000 7f6f03e00640 1 -- 172.21.15.28:0/267361921 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f3cb658220 con 0x55f3cb5aa330 2024-02-21T23:07:01.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.740+0000 7f6f03e00640 1 --2- 172.21.15.28:0/267361921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3cb5aa330 0x55f3cb5c96b0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f6eec047010 tx=0x7f6eec004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9fd17e9c6697a34 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:01.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.740+0000 7f6ef1600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:07:01.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.741+0000 7f6ef1600640 1 -- 172.21.15.28:0/267361921 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eec063060 con 0x55f3cb5aa330 2024-02-21T23:07:01.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.741+0000 7f6ef1600640 1 -- 172.21.15.28:0/267361921 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6eec004230 con 0x55f3cb5aa330 2024-02-21T23:07:01.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.741+0000 7f6f0dd87740 1 -- 172.21.15.28:0/267361921 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6eec037040 con 0x55f3cb5aa330 2024-02-21T23:07:01.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.741+0000 7f6ef1600640 1 -- 172.21.15.28:0/267361921 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=88) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6eec069050 con 0x55f3cb5aa330 2024-02-21T23:07:01.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.741+0000 7f6f0dd87740 10 client.7206 fetch_fsmap learned FSMap version 88 2024-02-21T23:07:01.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.741+0000 7f6f0dd87740 1 -- 172.21.15.28:0/267361921 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=88,monmap=2+}) v3 -- 0x55f3cb5ff160 con 0x55f3cb5aa330 2024-02-21T23:07:01.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.742+0000 7f6ef1600640 1 -- 172.21.15.28:0/267361921 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6eec037040 con 0x55f3cb5aa330 2024-02-21T23:07:01.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.742+0000 7f6ef1600640 1 -- 172.21.15.28:0/267361921 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 88) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f6eec06c020 con 0x55f3cb5aa330 2024-02-21T23:07:01.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.742+0000 7f6f0dd87740 10 client.7206 fetch_fsmap finished waiting for FSMap version 88 2024-02-21T23:07:01.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.742+0000 7f6f0dd87740 10 client.7206 resolve_mds: resolved name 'a' to daemon mds.a 2024-02-21T23:07:01.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.742+0000 7f6f0dd87740 20 client.7206 populate_metadata read hostname 'smithi028' 2024-02-21T23:07:01.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.742+0000 7f6f0dd87740 1 --2- 172.21.15.28:0/267361921 >> [v2:172.21.15.28:6834/2477712025,v1:172.21.15.28:6835/2477712025] conn(0x55f3cb654840 0x55f3cb607d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:01.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.742+0000 7f6f0dd87740 4 client.7206 mds_command: new command op to 5932 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-21T23:07:01.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.742+0000 7f6f0dd87740 1 -- 172.21.15.28:0/267361921 --> [v2:172.21.15.28:6834/2477712025,v1:172.21.15.28:6835/2477712025] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55f3cb514d00 con 0x55f3cb654840 2024-02-21T23:07:01.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.742+0000 7f6efb400640 1 --2- 172.21.15.28:0/267361921 >> [v2:172.21.15.28:6834/2477712025,v1:172.21.15.28:6835/2477712025] conn(0x55f3cb654840 0x55f3cb607d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:01.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.743+0000 7f6ef1600640 10 client.7206 ms_handle_connect on v2:172.21.15.28:6834/2477712025 2024-02-21T23:07:01.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.743+0000 7f6efb400640 1 --2- 172.21.15.28:0/267361921 >> [v2:172.21.15.28:6834/2477712025,v1:172.21.15.28:6835/2477712025] conn(0x55f3cb654840 0x55f3cb607d70 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf42504c38452cfd server_cookie=4e2d62ded2b80840 in_seq=0 out_seq=0 2024-02-21T23:07:01.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.743+0000 7f6ef1600640 1 -- 172.21.15.28:0/267361921 <== mds.0 v2:172.21.15.28:6834/2477712025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x55f3cb514d00 con 0x55f3cb654840 2024-02-21T23:07:01.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.743+0000 7f6ef1600640 10 client.7206 handle_command_reply: tid=0 2024-02-21T23:07:01.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.743+0000 7f6f0dd87740 1 client.7206 shutdown 2024-02-21T23:07:01.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.743+0000 7f6f0dd87740 1 -- 172.21.15.28:0/267361921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3cb5aa330 msgr2=0x55f3cb5c96b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:01.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.743+0000 7f6f0dd87740 1 --2- 172.21.15.28:0/267361921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3cb5aa330 0x55f3cb5c96b0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f6eec047010 tx=0x7f6eec004730 comp rx=0 tx=0).stop 2024-02-21T23:07:01.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.744+0000 7f6f0dd87740 1 -- 172.21.15.28:0/267361921 shutdown_connections 2024-02-21T23:07:01.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.744+0000 7f6f0dd87740 1 --2- 172.21.15.28:0/267361921 >> [v2:172.21.15.28:6834/2477712025,v1:172.21.15.28:6835/2477712025] conn(0x55f3cb654840 0x55f3cb607d70 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:01.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.744+0000 7f6f0dd87740 1 --2- 172.21.15.28:0/267361921 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55f3cb5e56b0 0x55f3cb5c1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:01.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.744+0000 7f6f0dd87740 1 --2- 172.21.15.28:0/267361921 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55f3cb5ebe20 0x55f3cb653fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:01.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.744+0000 7f6f0dd87740 1 --2- 172.21.15.28:0/267361921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3cb5aa330 0x55f3cb5c96b0 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:01.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.744+0000 7f6ef1600640 0 client.7206 ms_handle_reset on v2:172.21.15.28:6834/2477712025 2024-02-21T23:07:01.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.744+0000 7f6f0dd87740 1 -- 172.21.15.28:0/267361921 >> 172.21.15.28:0/267361921 conn(0x55f3cb5a9020 msgr2=0x55f3cb5e3670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:01.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.744+0000 7f6f0dd87740 1 -- 172.21.15.28:0/267361921 shutdown_connections 2024-02-21T23:07:01.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.744+0000 7f6f0dd87740 1 -- 172.21.15.28:0/267361921 wait complete. 2024-02-21T23:07:01.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.744+0000 7f6f0dd87740 20 client.7206 trim_cache size 0 max 16384 2024-02-21T23:07:01.737 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-21T23:07:01.737 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac", 2024-02-21T23:07:01.737 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-02-21T23:07:01.737 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 5932, 2024-02-21T23:07:01.737 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-02-21T23:07:01.738 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-02-21T23:07:01.738 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-02-21T23:07:01.738 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 6.9571696489999999, 2024-02-21T23:07:01.738 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 88, 2024-02-21T23:07:01.738 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 98, 2024-02-21T23:07:01.738 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 98, 2024-02-21T23:07:01.738 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 12.476221675 2024-02-21T23:07:01.738 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-21T23:07:01.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.745+0000 7f6f09800640 1 -- 172.21.15.28:0/446297645 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6ee006b500 msgr2=0x7f6ee006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:01.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.745+0000 7f6f09800640 1 --2- 172.21.15.28:0/446297645 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6ee006b500 0x7f6ee006d9c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f6eec036000 tx=0x7f6eec03d040 comp rx=0 tx=0).stop 2024-02-21T23:07:01.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.745+0000 7f6f09800640 1 -- 172.21.15.28:0/446297645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f041524c0 msgr2=0x7f6f041e7fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:01.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.745+0000 7f6f09800640 1 --2- 172.21.15.28:0/446297645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f041524c0 0x7f6f041e7fa0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f6efc00c7a0 tx=0x7f6efc00cc60 comp rx=0 tx=0).stop 2024-02-21T23:07:01.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.745+0000 7f6f09800640 1 -- 172.21.15.28:0/446297645 shutdown_connections 2024-02-21T23:07:01.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.745+0000 7f6f09800640 1 --2- 172.21.15.28:0/446297645 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6ee006b500 0x7f6ee006d9c0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:01.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.745+0000 7f6f09800640 1 --2- 172.21.15.28:0/446297645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f041524c0 0x7f6f041e7fa0 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:01.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.745+0000 7f6f09800640 1 --2- 172.21.15.28:0/446297645 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f040c85c0 0x7f6f041e7a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:01.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.745+0000 7f6f09800640 1 --2- 172.21.15.28:0/446297645 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6f040c5040 0x7f6f040c28d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:01.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.745+0000 7f6f09800640 1 -- 172.21.15.28:0/446297645 >> 172.21.15.28:0/446297645 conn(0x7f6f040ac2b0 msgr2=0x7f6f04155550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:01.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.745+0000 7f6f09800640 1 -- 172.21.15.28:0/446297645 shutdown_connections 2024-02-21T23:07:01.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.745+0000 7f6f09800640 1 -- 172.21.15.28:0/446297645 wait complete. 2024-02-21T23:07:01.751 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:07:01.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.978+0000 7f9f35800640 1 Processor -- start 2024-02-21T23:07:01.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.978+0000 7f9f35800640 1 -- start start 2024-02-21T23:07:01.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.981+0000 7f9f35800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9f300a5f40 0x7f9f3006dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:01.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.981+0000 7f9f35800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f9f300c0270 0x7f9f300c06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:01.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.981+0000 7f9f35800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f3006e750 con 0x7f9f300c3490 2024-02-21T23:07:01.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.981+0000 7f9f35800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f3006eff0 con 0x7f9f300a5f40 2024-02-21T23:07:01.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.981+0000 7f9f35800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f300bfde0 con 0x7f9f300c0270 2024-02-21T23:07:01.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.981+0000 7f9f2e000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9f300a5f40 0x7f9f3006dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:01.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.981+0000 7f9f2e000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9f300a5f40 0x7f9f3006dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47720/0 (socket says 172.21.15.28:47720) 2024-02-21T23:07:01.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.981+0000 7f9f2e000640 1 -- 172.21.15.28:0/391795311 learned_addr learned my addr 172.21.15.28:0/391795311 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:01.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.981+0000 7f9f2f400640 1 --2- 172.21.15.28:0/391795311 >> v2:172.21.15.81:3300/0 conn(0x7f9f300c0270 0x7f9f300c06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:01.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.982+0000 7f9f2e000640 1 -- 172.21.15.28:0/391795311 >> v2:172.21.15.81:3300/0 conn(0x7f9f300c0270 msgr2=0x7f9f300c06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:01.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.982+0000 7f9f2e000640 1 --2- 172.21.15.28:0/391795311 >> v2:172.21.15.81:3300/0 conn(0x7f9f300c0270 0x7f9f300c06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:01.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.982+0000 7f9f2e000640 1 -- 172.21.15.28:0/391795311 >> v1:172.21.15.81:6789/0 conn(0x7f9f300c3490 legacy=0x7f9f30152140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:01.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.982+0000 7f9f2e000640 1 -- 172.21.15.28:0/391795311 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f3006ea30 con 0x7f9f300a5f40 2024-02-21T23:07:01.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.983+0000 7f9f2e000640 1 --2- 172.21.15.28:0/391795311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f300a5f40 0x7f9f3006dfb0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f9f3006fa70 tx=0x7f9f1802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5aeb224f9a883efc server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:01.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.983+0000 7f9f2d600640 1 -- 172.21.15.28:0/391795311 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f1802e070 con 0x7f9f300a5f40 2024-02-21T23:07:01.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.983+0000 7f9f2d600640 1 -- 172.21.15.28:0/391795311 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f18037070 con 0x7f9f300a5f40 2024-02-21T23:07:01.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.983+0000 7f9f2d600640 1 -- 172.21.15.28:0/391795311 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f18031a00 con 0x7f9f300a5f40 2024-02-21T23:07:01.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.984+0000 7f9f35800640 1 -- 172.21.15.28:0/391795311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f300a5f40 msgr2=0x7f9f3006dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:01.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.984+0000 7f9f35800640 1 --2- 172.21.15.28:0/391795311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f300a5f40 0x7f9f3006dfb0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f9f3006fa70 tx=0x7f9f1802b880 comp rx=0 tx=0).stop 2024-02-21T23:07:01.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.984+0000 7f9f35800640 1 -- 172.21.15.28:0/391795311 shutdown_connections 2024-02-21T23:07:01.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.984+0000 7f9f35800640 1 --2- 172.21.15.28:0/391795311 >> v2:172.21.15.81:3300/0 conn(0x7f9f300c0270 0x7f9f300c06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:01.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.984+0000 7f9f35800640 1 --2- 172.21.15.28:0/391795311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f300a5f40 0x7f9f3006dfb0 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:01.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.984+0000 7f9f35800640 1 -- 172.21.15.28:0/391795311 >> 172.21.15.28:0/391795311 conn(0x7f9f300c67f0 msgr2=0x7f9f300c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:01.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.984+0000 7f9f35800640 1 -- 172.21.15.28:0/391795311 shutdown_connections 2024-02-21T23:07:01.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.984+0000 7f9f35800640 1 -- 172.21.15.28:0/391795311 wait complete. 2024-02-21T23:07:01.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.987+0000 7f9f35800640 1 Processor -- start 2024-02-21T23:07:01.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.987+0000 7f9f35800640 1 -- start start 2024-02-21T23:07:01.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.988+0000 7f9f35800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f300a5f40 0x7f9f30149bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:01.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.988+0000 7f9f35800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f300c0270 0x7f9f301e1d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:01.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.988+0000 7f9f35800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9f300c3490 0x7f9f3014ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:01.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.988+0000 7f9f35800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f300c0c20 con 0x7f9f300a5f40 2024-02-21T23:07:01.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.988+0000 7f9f35800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f3006e4f0 con 0x7f9f300c0270 2024-02-21T23:07:01.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.988+0000 7f9f35800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f300c1c70 con 0x7f9f300c3490 2024-02-21T23:07:01.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.988+0000 7f9f2ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f300a5f40 0x7f9f30149bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:01.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.988+0000 7f9f2ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f300a5f40 0x7f9f30149bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47726/0 (socket says 172.21.15.28:47726) 2024-02-21T23:07:01.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.988+0000 7f9f2ea00640 1 -- 172.21.15.28:0/3487908026 learned_addr learned my addr 172.21.15.28:0/3487908026 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:01.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.988+0000 7f9f2e000640 1 --2- 172.21.15.28:0/3487908026 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f300c0270 0x7f9f301e1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:01.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.988+0000 7f9f2f400640 1 --2- 172.21.15.28:0/3487908026 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9f300c3490 0x7f9f3014ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:01.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.989+0000 7f9f2ea00640 1 -- 172.21.15.28:0/3487908026 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9f300c3490 msgr2=0x7f9f3014ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:01.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.989+0000 7f9f2ea00640 1 --2- 172.21.15.28:0/3487908026 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9f300c3490 0x7f9f3014ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:01.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.989+0000 7f9f2ea00640 1 -- 172.21.15.28:0/3487908026 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f300c0270 msgr2=0x7f9f301e1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:01.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.989+0000 7f9f2ea00640 1 --2- 172.21.15.28:0/3487908026 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f300c0270 0x7f9f301e1d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:01.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.989+0000 7f9f2ea00640 1 -- 172.21.15.28:0/3487908026 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f301df980 con 0x7f9f300a5f40 2024-02-21T23:07:01.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.989+0000 7f9f2ea00640 1 --2- 172.21.15.28:0/3487908026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f300a5f40 0x7f9f30149bb0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f9f2000c7f0 tx=0x7f9f2000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:01.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.989+0000 7f9f26a00640 1 -- 172.21.15.28:0/3487908026 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f20016020 con 0x7f9f300a5f40 2024-02-21T23:07:01.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.989+0000 7f9f26a00640 1 -- 172.21.15.28:0/3487908026 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f2000aa50 con 0x7f9f300a5f40 2024-02-21T23:07:01.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.989+0000 7f9f26a00640 1 -- 172.21.15.28:0/3487908026 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f2000ae40 con 0x7f9f300a5f40 2024-02-21T23:07:01.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.989+0000 7f9f35800640 1 -- 172.21.15.28:0/3487908026 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f301ee8d0 con 0x7f9f300a5f40 2024-02-21T23:07:01.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.990+0000 7f9f35800640 1 -- 172.21.15.28:0/3487908026 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f3014a2b0 con 0x7f9f300a5f40 2024-02-21T23:07:01.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.991+0000 7f9f35800640 1 -- 172.21.15.28:0/3487908026 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f30067f30 con 0x7f9f300a5f40 2024-02-21T23:07:01.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.992+0000 7f9f26a00640 1 -- 172.21.15.28:0/3487908026 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9f20019020 con 0x7f9f300a5f40 2024-02-21T23:07:01.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.993+0000 7f9f26a00640 1 --2- 172.21.15.28:0/3487908026 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9f0c06b4d0 0x7f9f0c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:01.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.993+0000 7f9f26a00640 1 -- 172.21.15.28:0/3487908026 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f9f2008a2d0 con 0x7f9f300a5f40 2024-02-21T23:07:01.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.993+0000 7f9f2e000640 1 --2- 172.21.15.28:0/3487908026 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9f0c06b4d0 0x7f9f0c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:01.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.994+0000 7f9f2e000640 1 --2- 172.21.15.28:0/3487908026 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9f0c06b4d0 0x7f9f0c06d990 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f9f18036000 tx=0x7f9f1803d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:01.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:01.997+0000 7f9f26a00640 1 -- 172.21.15.28:0/3487908026 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f9f20020030 con 0x7f9f300a5f40 2024-02-21T23:07:02.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.235+0000 7f9f35800640 1 -- 172.21.15.28:0/3487908026 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9f3006dd00 con 0x7f9f300a5f40 2024-02-21T23:07:02.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.236+0000 7f9f26a00640 1 -- 172.21.15.28:0/3487908026 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+4934 (secure 0 0 0) 0x7f9f20059050 con 0x7f9f300a5f40 2024-02-21T23:07:02.227 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:07:02.227 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":88,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5555,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/85689296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":85689296},{"type":"v1","addr":"172.21.15.28:6833","nonce":85689296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82},{"gid":5567,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1194613007","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1194613007},{"type":"v1","addr":"172.21.15.81:6835","nonce":1194613007}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":5570,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/2315735157","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":2315735157},{"type":"v1","addr":"172.21.15.81:6837","nonce":2315735157}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:06:54.201589+0000","modified":"2024-02-21T23:06:57.273466+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5932},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5932":{"gid":5932,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/2477712025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2477712025},{"type":"v1","addr":"172.21.15.28:6835","nonce":2477712025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-02-21T23:07:02.228 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 88 2024-02-21T23:07:02.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.238+0000 7f9f35800640 1 -- 172.21.15.28:0/3487908026 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9f0c06b4d0 msgr2=0x7f9f0c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:02.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.238+0000 7f9f35800640 1 --2- 172.21.15.28:0/3487908026 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9f0c06b4d0 0x7f9f0c06d990 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f9f18036000 tx=0x7f9f1803d040 comp rx=0 tx=0).stop 2024-02-21T23:07:02.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.238+0000 7f9f35800640 1 -- 172.21.15.28:0/3487908026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f300a5f40 msgr2=0x7f9f30149bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:02.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.238+0000 7f9f35800640 1 --2- 172.21.15.28:0/3487908026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f300a5f40 0x7f9f30149bb0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f9f2000c7f0 tx=0x7f9f2000ccb0 comp rx=0 tx=0).stop 2024-02-21T23:07:02.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.239+0000 7f9f35800640 1 -- 172.21.15.28:0/3487908026 shutdown_connections 2024-02-21T23:07:02.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.239+0000 7f9f35800640 1 --2- 172.21.15.28:0/3487908026 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9f0c06b4d0 0x7f9f0c06d990 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:02.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.239+0000 7f9f35800640 1 --2- 172.21.15.28:0/3487908026 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9f300c3490 0x7f9f3014ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:02.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.239+0000 7f9f35800640 1 --2- 172.21.15.28:0/3487908026 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f300c0270 0x7f9f301e1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:02.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.239+0000 7f9f35800640 1 --2- 172.21.15.28:0/3487908026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f300a5f40 0x7f9f30149bb0 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:02.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.239+0000 7f9f35800640 1 -- 172.21.15.28:0/3487908026 >> 172.21.15.28:0/3487908026 conn(0x7f9f300c67f0 msgr2=0x7f9f3014b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:02.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.239+0000 7f9f35800640 1 -- 172.21.15.28:0/3487908026 shutdown_connections 2024-02-21T23:07:02.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.239+0000 7f9f35800640 1 -- 172.21.15.28:0/3487908026 wait complete. 2024-02-21T23:07:02.249 INFO:tasks.cephfs.test_failover:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 7, "epoch": 88, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 7, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-02-21T23:06:54.201589+0000", "damaged": [], "data_pools": [ 16 ], "enabled": true, "epoch": 88, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5932": { "addr": "172.21.15.28:6835/2477712025", "addrs": { "addrvec": [ { "addr": "172.21.15.28:6834", "nonce": 2477712025, "type": "v2" }, { "addr": "172.21.15.28:6835", "nonce": 2477712025, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5932, "incarnation": 84, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 15, "modified": "2024-02-21T23:06:57.273466+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5932 } } } ], "standbys": [ { "addr": "172.21.15.28:6833/85689296", "addrs": { "addrvec": [ { "addr": "172.21.15.28:6832", "nonce": 85689296, "type": "v2" }, { "addr": "172.21.15.28:6833", "nonce": 85689296, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 82, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5555, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.81:6835/1194613007", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6834", "nonce": 1194613007, "type": "v2" }, { "addr": "172.21.15.81:6835", "nonce": 1194613007, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 83, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5567, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.81:6837/2315735157", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6836", "nonce": 2315735157, "type": "v2" }, { "addr": "172.21.15.81:6837", "nonce": 2315735157, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 83, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5570, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-02-21T23:07:02.250 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-02-21T23:07:02.250 INFO:tasks.cephfs.test_failover:current = [{'gid': 5932, 'name': 'a', 'rank': 0, 'incarnation': 84, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/2477712025', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 2477712025}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 2477712025}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 5567, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.81:6835/1194613007', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6834', 'nonce': 1194613007}, {'type': 'v1', 'addr': '172.21.15.81:6835', 'nonce': 1194613007}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 83}, {'gid': 5555, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.28:6833/85689296', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 85689296}, {'type': 'v1', 'addr': '172.21.15.28:6833', 'nonce': 85689296}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 82}, {'gid': 5570, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.81:6837/2315735157', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 2315735157}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 2315735157}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 83}] 2024-02-21T23:07:02.250 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.c mds_join_fs cephfs 2024-02-21T23:07:02.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.484+0000 7f09cf600640 1 Processor -- start 2024-02-21T23:07:02.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.484+0000 7f09cf600640 1 -- start start 2024-02-21T23:07:02.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.486+0000 7f09cf600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f09c80c5cc0 0x7f09c80c6c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:02.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.486+0000 7f09cf600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f09c80c3030 0x7f09c80c3440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:02.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.486+0000 7f09cf600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09c8159580 con 0x7f09c80c7590 2024-02-21T23:07:02.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.486+0000 7f09cf600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09c80c6230 con 0x7f09c80c3030 2024-02-21T23:07:02.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.486+0000 7f09cf600640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09c81587c0 con 0x7f09c80c5cc0 2024-02-21T23:07:02.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.486+0000 7f09cd600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f09c80c3030 0x7f09c80c3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:02.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.486+0000 7f09cd600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f09c80c3030 0x7f09c80c3440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47740/0 (socket says 172.21.15.28:47740) 2024-02-21T23:07:02.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.486+0000 7f09cd600640 1 -- 172.21.15.28:0/3073099050 learned_addr learned my addr 172.21.15.28:0/3073099050 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:02.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.486+0000 7f09c7e00640 1 --2- 172.21.15.28:0/3073099050 >> v2:172.21.15.81:3300/0 conn(0x7f09c80c5cc0 0x7f09c80c6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:02.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.487+0000 7f09cd600640 1 -- 172.21.15.28:0/3073099050 >> v2:172.21.15.81:3300/0 conn(0x7f09c80c5cc0 msgr2=0x7f09c80c6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:02.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.487+0000 7f09cd600640 1 --2- 172.21.15.28:0/3073099050 >> v2:172.21.15.81:3300/0 conn(0x7f09c80c5cc0 0x7f09c80c6c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:02.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.487+0000 7f09cd600640 1 -- 172.21.15.28:0/3073099050 >> v1:172.21.15.81:6789/0 conn(0x7f09c80c7590 legacy=0x7f09c81524e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:02.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.487+0000 7f09cd600640 1 -- 172.21.15.28:0/3073099050 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09c8158aa0 con 0x7f09c80c3030 2024-02-21T23:07:02.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.487+0000 7f09c7e00640 1 --2- 172.21.15.28:0/3073099050 >> v2:172.21.15.81:3300/0 conn(0x7f09c80c5cc0 0x7f09c80c6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:07:02.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.489+0000 7f09cd600640 1 --2- 172.21.15.28:0/3073099050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09c80c3030 0x7f09c80c3440 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f09c80c2cc0 tx=0x7f09b802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c611f9c47404571 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:02.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.489+0000 7f09c7400640 1 -- 172.21.15.28:0/3073099050 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09b8035020 con 0x7f09c80c3030 2024-02-21T23:07:02.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.489+0000 7f09c7400640 1 -- 172.21.15.28:0/3073099050 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09b8031870 con 0x7f09c80c3030 2024-02-21T23:07:02.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.489+0000 7f09c7400640 1 -- 172.21.15.28:0/3073099050 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09b8031c40 con 0x7f09c80c3030 2024-02-21T23:07:02.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.490+0000 7f09cf600640 1 -- 172.21.15.28:0/3073099050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09c80c3030 msgr2=0x7f09c80c3440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:02.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.490+0000 7f09cf600640 1 --2- 172.21.15.28:0/3073099050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09c80c3030 0x7f09c80c3440 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f09c80c2cc0 tx=0x7f09b802b910 comp rx=0 tx=0).stop 2024-02-21T23:07:02.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.490+0000 7f09cf600640 1 -- 172.21.15.28:0/3073099050 shutdown_connections 2024-02-21T23:07:02.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.490+0000 7f09cf600640 1 --2- 172.21.15.28:0/3073099050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09c80c3030 0x7f09c80c3440 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:02.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.490+0000 7f09cf600640 1 --2- 172.21.15.28:0/3073099050 >> v2:172.21.15.81:3300/0 conn(0x7f09c80c5cc0 0x7f09c80c6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:02.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.490+0000 7f09cf600640 1 -- 172.21.15.28:0/3073099050 >> 172.21.15.28:0/3073099050 conn(0x7f09c808afe0 msgr2=0x7f09c80725a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:02.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.490+0000 7f09cf600640 1 -- 172.21.15.28:0/3073099050 shutdown_connections 2024-02-21T23:07:02.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.490+0000 7f09cf600640 1 -- 172.21.15.28:0/3073099050 wait complete. 2024-02-21T23:07:02.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.493+0000 7f09cf600640 1 Processor -- start 2024-02-21T23:07:02.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.493+0000 7f09cf600640 1 -- start start 2024-02-21T23:07:02.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.494+0000 7f09cf600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09c80c3030 0x7f09c8156c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:02.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.494+0000 7f09cf600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f09c80c5cc0 0x7f09c81575c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:02.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.494+0000 7f09cf600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f09c80c7590 0x7f09c81eae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:02.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.494+0000 7f09cf600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09c80c6720 con 0x7f09c80c3030 2024-02-21T23:07:02.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.494+0000 7f09cf600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09c8159060 con 0x7f09c80c7590 2024-02-21T23:07:02.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.494+0000 7f09cf600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09c80c68a0 con 0x7f09c80c5cc0 2024-02-21T23:07:02.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.495+0000 7f09ccc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09c80c3030 0x7f09c8156c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:02.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.495+0000 7f09ccc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09c80c3030 0x7f09c8156c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47754/0 (socket says 172.21.15.28:47754) 2024-02-21T23:07:02.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.495+0000 7f09ccc00640 1 -- 172.21.15.28:0/3174884274 learned_addr learned my addr 172.21.15.28:0/3174884274 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:02.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.495+0000 7f09c7e00640 1 --2- 172.21.15.28:0/3174884274 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f09c80c5cc0 0x7f09c81575c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:02.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.495+0000 7f09cd600640 1 --2- 172.21.15.28:0/3174884274 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f09c80c7590 0x7f09c81eae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:02.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.495+0000 7f09ccc00640 1 -- 172.21.15.28:0/3174884274 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f09c80c5cc0 msgr2=0x7f09c81575c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:02.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.495+0000 7f09ccc00640 1 --2- 172.21.15.28:0/3174884274 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f09c80c5cc0 0x7f09c81575c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:02.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.495+0000 7f09ccc00640 1 -- 172.21.15.28:0/3174884274 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f09c80c7590 msgr2=0x7f09c81eae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:02.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.495+0000 7f09ccc00640 1 --2- 172.21.15.28:0/3174884274 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f09c80c7590 0x7f09c81eae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:02.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.495+0000 7f09ccc00640 1 -- 172.21.15.28:0/3174884274 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09c8156950 con 0x7f09c80c3030 2024-02-21T23:07:02.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.496+0000 7f09ccc00640 1 --2- 172.21.15.28:0/3174884274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09c80c3030 0x7f09c8156c30 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f09c000c730 tx=0x7f09c000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:02.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.496+0000 7f09c4c00640 1 -- 172.21.15.28:0/3174884274 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c0003b70 con 0x7f09c80c3030 2024-02-21T23:07:02.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.496+0000 7f09c4c00640 1 -- 172.21.15.28:0/3174884274 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09c0003d30 con 0x7f09c80c3030 2024-02-21T23:07:02.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.496+0000 7f09c4c00640 1 -- 172.21.15.28:0/3174884274 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c000aa90 con 0x7f09c80c3030 2024-02-21T23:07:02.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.496+0000 7f09cf600640 1 -- 172.21.15.28:0/3174884274 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09c80c9f10 con 0x7f09c80c3030 2024-02-21T23:07:02.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.496+0000 7f09cf600640 1 -- 172.21.15.28:0/3174884274 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09c8156500 con 0x7f09c80c3030 2024-02-21T23:07:02.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.498+0000 7f09cf600640 1 -- 172.21.15.28:0/3174884274 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09c80c9d00 con 0x7f09c80c3030 2024-02-21T23:07:02.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.498+0000 7f09c4c00640 1 -- 172.21.15.28:0/3174884274 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f09c000ac30 con 0x7f09c80c3030 2024-02-21T23:07:02.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.499+0000 7f09c4c00640 1 --2- 172.21.15.28:0/3174884274 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f09a406b4d0 0x7f09a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:02.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.499+0000 7f09c4c00640 1 -- 172.21.15.28:0/3174884274 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f09c008a280 con 0x7f09c80c3030 2024-02-21T23:07:02.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.500+0000 7f09c7e00640 1 --2- 172.21.15.28:0/3174884274 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f09a406b4d0 0x7f09a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:02.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.501+0000 7f09c7e00640 1 --2- 172.21.15.28:0/3174884274 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f09a406b4d0 0x7f09a406d990 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f09c814ec00 tx=0x7f09b0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:02.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.504+0000 7f09c4c00640 1 -- 172.21.15.28:0/3174884274 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f09c0058f60 con 0x7f09c80c3030 2024-02-21T23:07:02.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.705+0000 7f09cf600640 1 -- 172.21.15.28:0/3174884274 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) v1 -- 0x7f09c8067f30 con 0x7f09c80c3030 2024-02-21T23:07:02.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.709+0000 7f09c4c00640 1 -- 172.21.15.28:0/3174884274 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v32)=0 v32) v1 ==== 116+0+0 (secure 0 0 0) 0x7f09c0058860 con 0x7f09c80c3030 2024-02-21T23:07:02.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.712+0000 7f09cf600640 1 -- 172.21.15.28:0/3174884274 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f09a406b4d0 msgr2=0x7f09a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:02.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.712+0000 7f09cf600640 1 --2- 172.21.15.28:0/3174884274 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f09a406b4d0 0x7f09a406d990 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f09c814ec00 tx=0x7f09b0008040 comp rx=0 tx=0).stop 2024-02-21T23:07:02.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.712+0000 7f09cf600640 1 -- 172.21.15.28:0/3174884274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09c80c3030 msgr2=0x7f09c8156c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:02.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.712+0000 7f09cf600640 1 --2- 172.21.15.28:0/3174884274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09c80c3030 0x7f09c8156c30 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f09c000c730 tx=0x7f09c000cbf0 comp rx=0 tx=0).stop 2024-02-21T23:07:02.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.712+0000 7f09cf600640 1 -- 172.21.15.28:0/3174884274 shutdown_connections 2024-02-21T23:07:02.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.712+0000 7f09cf600640 1 --2- 172.21.15.28:0/3174884274 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f09a406b4d0 0x7f09a406d990 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:02.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.712+0000 7f09cf600640 1 --2- 172.21.15.28:0/3174884274 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f09c80c5cc0 0x7f09c81575c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:02.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.712+0000 7f09cf600640 1 --2- 172.21.15.28:0/3174884274 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f09c80c7590 0x7f09c81eae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:02.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.712+0000 7f09cf600640 1 --2- 172.21.15.28:0/3174884274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09c80c3030 0x7f09c8156c30 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:02.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.712+0000 7f09cf600640 1 -- 172.21.15.28:0/3174884274 >> 172.21.15.28:0/3174884274 conn(0x7f09c808afe0 msgr2=0x7f09c81514e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:02.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.713+0000 7f09cf600640 1 -- 172.21.15.28:0/3174884274 shutdown_connections 2024-02-21T23:07:02.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.713+0000 7f09cf600640 1 -- 172.21.15.28:0/3174884274 wait complete. 2024-02-21T23:07:02.722 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:07:02.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.949+0000 7f5527200640 1 Processor -- start 2024-02-21T23:07:02.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.949+0000 7f5527200640 1 -- start start 2024-02-21T23:07:02.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.950+0000 7f5527200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5520150450 0x7f55200a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:02.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.951+0000 7f5527200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f552006ee40 0x7f55201536f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:02.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.951+0000 7f5527200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55200c2760 con 0x7f5520156af0 2024-02-21T23:07:02.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.951+0000 7f5527200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55200c22c0 con 0x7f5520150450 2024-02-21T23:07:02.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.951+0000 7f5527200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5520156660 con 0x7f552006ee40 2024-02-21T23:07:02.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.951+0000 7f5526800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f5520156af0 0x7f552006dff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34644/0 (socket says 172.21.15.28:34644) 2024-02-21T23:07:02.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.951+0000 7f5526800640 1 -- 172.21.15.28:0/4191848194 learned_addr learned my addr 172.21.15.28:0/4191848194 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:07:02.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.951+0000 7f5525e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5520150450 0x7f55200a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:02.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.951+0000 7f5525400640 1 --2- 172.21.15.28:0/4191848194 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f552006ee40 0x7f55201536f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:02.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.952+0000 7f5525e00640 1 -- 172.21.15.28:0/4191848194 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f552006ee40 msgr2=0x7f55201536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:02.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.952+0000 7f5524a00640 1 -- 172.21.15.28:0/4191848194 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 121611828 0 0) 0x7f55200c2760 con 0x7f5520156af0 2024-02-21T23:07:02.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.952+0000 7f5525e00640 1 --2- 172.21.15.28:0/4191848194 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f552006ee40 0x7f55201536f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:02.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.952+0000 7f5525e00640 1 -- 172.21.15.28:0/4191848194 >> v1:172.21.15.28:6789/0 conn(0x7f5520156af0 legacy=0x7f552006dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:02.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.952+0000 7f5525e00640 1 -- 172.21.15.28:0/4191848194 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5520153c30 con 0x7f5520150450 2024-02-21T23:07:02.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.954+0000 7f5525e00640 1 --2- 172.21.15.28:0/4191848194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5520150450 0x7f55200a5fd0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f55140097a0 tx=0x7f551402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c187261399905323 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:02.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.954+0000 7f5524a00640 1 -- 172.21.15.28:0/4191848194 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5514035020 con 0x7f5520150450 2024-02-21T23:07:02.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.954+0000 7f5524a00640 1 -- 172.21.15.28:0/4191848194 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5514031810 con 0x7f5520150450 2024-02-21T23:07:02.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.954+0000 7f5524a00640 1 -- 172.21.15.28:0/4191848194 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5514031be0 con 0x7f5520150450 2024-02-21T23:07:02.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.955+0000 7f5527200640 1 -- 172.21.15.28:0/4191848194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5520150450 msgr2=0x7f55200a5fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:02.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.955+0000 7f5527200640 1 --2- 172.21.15.28:0/4191848194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5520150450 0x7f55200a5fd0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f55140097a0 tx=0x7f551402b8b0 comp rx=0 tx=0).stop 2024-02-21T23:07:02.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.955+0000 7f5527200640 1 -- 172.21.15.28:0/4191848194 shutdown_connections 2024-02-21T23:07:02.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.955+0000 7f5527200640 1 --2- 172.21.15.28:0/4191848194 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f552006ee40 0x7f55201536f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:02.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.955+0000 7f5527200640 1 --2- 172.21.15.28:0/4191848194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5520150450 0x7f55200a5fd0 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:02.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.955+0000 7f5527200640 1 -- 172.21.15.28:0/4191848194 >> 172.21.15.28:0/4191848194 conn(0x7f552006b630 msgr2=0x7f55200c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:02.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.955+0000 7f5527200640 1 -- 172.21.15.28:0/4191848194 shutdown_connections 2024-02-21T23:07:02.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.955+0000 7f5527200640 1 -- 172.21.15.28:0/4191848194 wait complete. 2024-02-21T23:07:02.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.957+0000 7f5527200640 1 Processor -- start 2024-02-21T23:07:02.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.957+0000 7f5527200640 1 -- start start 2024-02-21T23:07:02.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.958+0000 7f5527200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f552006ee40 0x7f5520147500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:02.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.958+0000 7f5527200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5520150450 0x7f5520155a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:02.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.958+0000 7f5527200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5520156af0 0x7f5520154760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:02.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.958+0000 7f5527200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55200c2cb0 con 0x7f552006ee40 2024-02-21T23:07:02.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.958+0000 7f5527200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55200c29d0 con 0x7f5520156af0 2024-02-21T23:07:02.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.958+0000 7f5527200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f552006e700 con 0x7f5520150450 2024-02-21T23:07:02.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.958+0000 7f5525e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f552006ee40 0x7f5520147500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:02.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.958+0000 7f5525e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f552006ee40 0x7f5520147500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47780/0 (socket says 172.21.15.28:47780) 2024-02-21T23:07:02.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.958+0000 7f5525e00640 1 -- 172.21.15.28:0/1950730357 learned_addr learned my addr 172.21.15.28:0/1950730357 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:02.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.958+0000 7f5525400640 1 --2- 172.21.15.28:0/1950730357 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5520150450 0x7f5520155a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:02.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.958+0000 7f5526800640 1 --2- 172.21.15.28:0/1950730357 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5520156af0 0x7f5520154760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:02.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.959+0000 7f5525e00640 1 -- 172.21.15.28:0/1950730357 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5520150450 msgr2=0x7f5520155a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:02.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.959+0000 7f5525e00640 1 --2- 172.21.15.28:0/1950730357 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5520150450 0x7f5520155a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:02.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.959+0000 7f5525e00640 1 -- 172.21.15.28:0/1950730357 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5520156af0 msgr2=0x7f5520154760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:02.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.959+0000 7f5525e00640 1 --2- 172.21.15.28:0/1950730357 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5520156af0 0x7f5520154760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:02.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.959+0000 7f5525e00640 1 -- 172.21.15.28:0/1950730357 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55201eb680 con 0x7f552006ee40 2024-02-21T23:07:02.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.959+0000 7f5526800640 1 --2- 172.21.15.28:0/1950730357 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5520156af0 0x7f5520154760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:07:02.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.959+0000 7f5525e00640 1 --2- 172.21.15.28:0/1950730357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f552006ee40 0x7f5520147500 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f55140098c0 tx=0x7f55140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:02.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.960+0000 7f551a000640 1 -- 172.21.15.28:0/1950730357 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5514041070 con 0x7f552006ee40 2024-02-21T23:07:02.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.960+0000 7f551a000640 1 -- 172.21.15.28:0/1950730357 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f551403ad00 con 0x7f552006ee40 2024-02-21T23:07:02.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.960+0000 7f551a000640 1 -- 172.21.15.28:0/1950730357 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5514035040 con 0x7f552006ee40 2024-02-21T23:07:02.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.960+0000 7f5527200640 1 -- 172.21.15.28:0/1950730357 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55200c4730 con 0x7f552006ee40 2024-02-21T23:07:02.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.960+0000 7f5527200640 1 -- 172.21.15.28:0/1950730357 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5520154ca0 con 0x7f552006ee40 2024-02-21T23:07:02.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.961+0000 7f5527200640 1 -- 172.21.15.28:0/1950730357 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f552004f290 con 0x7f552006ee40 2024-02-21T23:07:02.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.962+0000 7f551a000640 1 -- 172.21.15.28:0/1950730357 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f551404c060 con 0x7f552006ee40 2024-02-21T23:07:02.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.963+0000 7f551a000640 1 --2- 172.21.15.28:0/1950730357 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f550406b500 0x7f550406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:02.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.963+0000 7f551a000640 1 -- 172.21.15.28:0/1950730357 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f55140abdf0 con 0x7f552006ee40 2024-02-21T23:07:02.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.964+0000 7f5525400640 1 --2- 172.21.15.28:0/1950730357 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f550406b500 0x7f550406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:02.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.965+0000 7f5525400640 1 --2- 172.21.15.28:0/1950730357 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f550406b500 0x7f550406d9c0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f552014f970 tx=0x7f5510008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:02.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:02.972+0000 7f551a000640 1 -- 172.21.15.28:0/1950730357 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f551404b2c0 con 0x7f552006ee40 2024-02-21T23:07:03.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.212+0000 7f5527200640 1 -- 172.21.15.28:0/1950730357 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f55200c1f10 con 0x7f552006ee40 2024-02-21T23:07:03.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.213+0000 7f551a000640 1 -- 172.21.15.28:0/1950730357 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+4934 (secure 0 0 0) 0x7f551407ab50 con 0x7f552006ee40 2024-02-21T23:07:03.202 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:07:03.202 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":88,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5555,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/85689296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":85689296},{"type":"v1","addr":"172.21.15.28:6833","nonce":85689296}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":82},{"gid":5567,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1194613007","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1194613007},{"type":"v1","addr":"172.21.15.81:6835","nonce":1194613007}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":5570,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/2315735157","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":2315735157},{"type":"v1","addr":"172.21.15.81:6837","nonce":2315735157}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:06:54.201589+0000","modified":"2024-02-21T23:06:57.273466+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5932},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5932":{"gid":5932,"name":"a","rank":0,"incarnation":84,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/2477712025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2477712025},{"type":"v1","addr":"172.21.15.28:6835","nonce":2477712025}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-02-21T23:07:03.202 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 88 2024-02-21T23:07:03.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.216+0000 7f5527200640 1 -- 172.21.15.28:0/1950730357 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f550406b500 msgr2=0x7f550406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:03.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.216+0000 7f5527200640 1 --2- 172.21.15.28:0/1950730357 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f550406b500 0x7f550406d9c0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f552014f970 tx=0x7f5510008040 comp rx=0 tx=0).stop 2024-02-21T23:07:03.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.216+0000 7f5527200640 1 -- 172.21.15.28:0/1950730357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f552006ee40 msgr2=0x7f5520147500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:03.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.216+0000 7f5527200640 1 --2- 172.21.15.28:0/1950730357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f552006ee40 0x7f5520147500 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f55140098c0 tx=0x7f55140046c0 comp rx=0 tx=0).stop 2024-02-21T23:07:03.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.216+0000 7f5527200640 1 -- 172.21.15.28:0/1950730357 shutdown_connections 2024-02-21T23:07:03.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.216+0000 7f5527200640 1 --2- 172.21.15.28:0/1950730357 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f550406b500 0x7f550406d9c0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:03.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.216+0000 7f5527200640 1 --2- 172.21.15.28:0/1950730357 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5520150450 0x7f5520155a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:03.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.216+0000 7f5527200640 1 --2- 172.21.15.28:0/1950730357 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5520156af0 0x7f5520154760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:03.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.216+0000 7f5527200640 1 --2- 172.21.15.28:0/1950730357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f552006ee40 0x7f5520147500 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:03.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.216+0000 7f5527200640 1 -- 172.21.15.28:0/1950730357 >> 172.21.15.28:0/1950730357 conn(0x7f552006b630 msgr2=0x7f552006dff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:03.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.216+0000 7f5527200640 1 -- 172.21.15.28:0/1950730357 shutdown_connections 2024-02-21T23:07:03.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.216+0000 7f5527200640 1 -- 172.21.15.28:0/1950730357 wait complete. 2024-02-21T23:07:03.627 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:06:54.201589+0000', 'modified': '2024-02-21T23:06:57.273466+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5932}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5932': {'gid': 5932, 'name': 'a', 'rank': 0, 'incarnation': 84, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/2477712025', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 2477712025}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 2477712025}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:07:03.627 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-21T23:07:03.627 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-02-21T23:07:03.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.873+0000 7f011b000640 1 Processor -- start 2024-02-21T23:07:03.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.873+0000 7f011b000640 1 -- start start 2024-02-21T23:07:03.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.875+0000 7f011b000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f011406d410 0x7f01140a8e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:03.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.875+0000 7f011b000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f01140c43e0 0x7f01140c5a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:03.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.875+0000 7f011b000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0114149c10 con 0x7f0114149830 2024-02-21T23:07:03.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.875+0000 7f011b000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01141537d0 con 0x7f011406d410 2024-02-21T23:07:03.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.875+0000 7f011b000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0114153ab0 con 0x7f01140c43e0 2024-02-21T23:07:03.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.875+0000 7f0119c00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f0114149830 0x7f01140c0ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34650/0 (socket says 172.21.15.28:34650) 2024-02-21T23:07:03.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.875+0000 7f0119c00640 1 -- 172.21.15.28:0/2362283074 learned_addr learned my addr 172.21.15.28:0/2362283074 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:07:03.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.876+0000 7f0119200640 1 --2- 172.21.15.28:0/2362283074 >> v2:172.21.15.81:3300/0 conn(0x7f011406d410 0x7f01140a8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:03.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.876+0000 7f011a600640 1 --2- 172.21.15.28:0/2362283074 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f01140c43e0 0x7f01140c5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:03.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.876+0000 7f0113e00640 1 -- 172.21.15.28:0/2362283074 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3403294617 0 0) 0x7f0114149c10 con 0x7f0114149830 2024-02-21T23:07:03.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.876+0000 7f0113e00640 1 -- 172.21.15.28:0/2362283074 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00fc003520 con 0x7f0114149830 2024-02-21T23:07:03.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.876+0000 7f0113e00640 1 -- 172.21.15.28:0/2362283074 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1316444055 0 0) 0x7f00fc003520 con 0x7f0114149830 2024-02-21T23:07:03.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.876+0000 7f0113e00640 1 -- 172.21.15.28:0/2362283074 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f01140c43e0 msgr2=0x7f01140c5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:03.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.876+0000 7f0113e00640 1 --2- 172.21.15.28:0/2362283074 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f01140c43e0 0x7f01140c5a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:03.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.876+0000 7f0113e00640 1 -- 172.21.15.28:0/2362283074 >> v2:172.21.15.81:3300/0 conn(0x7f011406d410 msgr2=0x7f01140a8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:03.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.876+0000 7f0113e00640 1 --2- 172.21.15.28:0/2362283074 >> v2:172.21.15.81:3300/0 conn(0x7f011406d410 0x7f01140a8e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:03.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.876+0000 7f0113e00640 1 -- 172.21.15.28:0/2362283074 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01140c4fa0 con 0x7f0114149830 2024-02-21T23:07:03.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.876+0000 7f0113e00640 1 -- 172.21.15.28:0/2362283074 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f0108001670 con 0x7f0114149830 2024-02-21T23:07:03.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.876+0000 7f0119200640 1 --2- 172.21.15.28:0/2362283074 >> v2:172.21.15.81:3300/0 conn(0x7f011406d410 0x7f01140a8e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-21T23:07:03.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.876+0000 7f0113e00640 1 -- 172.21.15.28:0/2362283074 >> v1:172.21.15.28:6789/0 conn(0x7f0114149830 legacy=0x7f01140c0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:03.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.876+0000 7f0113e00640 1 --2- 172.21.15.28:0/2362283074 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f00fc0039b0 0x7f00fc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:03.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.876+0000 7f0113e00640 1 --2- 172.21.15.28:0/2362283074 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f01140c43e0 0x7f00fc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:03.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.876+0000 7f0113e00640 1 --2- 172.21.15.28:0/2362283074 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00fc004d40 0x7f00fc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:03.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.876+0000 7f0113e00640 1 -- 172.21.15.28:0/2362283074 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01141537d0 con 0x7f00fc004d40 2024-02-21T23:07:03.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.876+0000 7f0113e00640 1 -- 172.21.15.28:0/2362283074 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0114153ab0 con 0x7f00fc0039b0 2024-02-21T23:07:03.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.876+0000 7f0113e00640 1 -- 172.21.15.28:0/2362283074 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00fc00ba40 con 0x7f01140c43e0 2024-02-21T23:07:03.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.877+0000 7f011a600640 1 --2- 172.21.15.28:0/2362283074 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00fc004d40 0x7f00fc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:03.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.877+0000 7f011a600640 1 -- 172.21.15.28:0/2362283074 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f01140c43e0 msgr2=0x7f00fc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:03.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.877+0000 7f011a600640 1 --2- 172.21.15.28:0/2362283074 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f01140c43e0 0x7f00fc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:03.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.877+0000 7f011a600640 1 -- 172.21.15.28:0/2362283074 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f00fc0039b0 msgr2=0x7f00fc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:03.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.877+0000 7f0119c00640 1 --2- 172.21.15.28:0/2362283074 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f00fc0039b0 0x7f00fc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:03.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.877+0000 7f011a600640 1 --2- 172.21.15.28:0/2362283074 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f00fc0039b0 0x7f00fc003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:03.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.877+0000 7f011a600640 1 -- 172.21.15.28:0/2362283074 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00fc003520 con 0x7f00fc004d40 2024-02-21T23:07:03.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.877+0000 7f0119c00640 1 --2- 172.21.15.28:0/2362283074 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f00fc0039b0 0x7f00fc003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:07:03.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.878+0000 7f011a600640 1 --2- 172.21.15.28:0/2362283074 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00fc004d40 0x7f00fc0051c0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f010c009f90 tx=0x7f00fc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1024fb14172c21fe server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:03.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.879+0000 7f0113e00640 1 -- 172.21.15.28:0/2362283074 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f010c039070 con 0x7f00fc004d40 2024-02-21T23:07:03.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.879+0000 7f0113e00640 1 -- 172.21.15.28:0/2362283074 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f010c02cc20 con 0x7f00fc004d40 2024-02-21T23:07:03.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.879+0000 7f0113e00640 1 -- 172.21.15.28:0/2362283074 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f010c009c10 con 0x7f00fc004d40 2024-02-21T23:07:03.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.879+0000 7f011b000640 1 -- 172.21.15.28:0/2362283074 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00fc004d40 msgr2=0x7f00fc0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:03.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.880+0000 7f011b000640 1 --2- 172.21.15.28:0/2362283074 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00fc004d40 0x7f00fc0051c0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f010c009f90 tx=0x7f00fc005830 comp rx=0 tx=0).stop 2024-02-21T23:07:03.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.880+0000 7f011a600640 1 -- 172.21.15.28:0/2362283074 reap_dead start 2024-02-21T23:07:03.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.880+0000 7f011b000640 1 -- 172.21.15.28:0/2362283074 shutdown_connections 2024-02-21T23:07:03.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.880+0000 7f011b000640 1 -- 172.21.15.28:0/2362283074 >> 172.21.15.28:0/2362283074 conn(0x7f01140a8320 msgr2=0x7f01140a8730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:03.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.880+0000 7f011b000640 1 -- 172.21.15.28:0/2362283074 shutdown_connections 2024-02-21T23:07:03.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.880+0000 7f011b000640 1 -- 172.21.15.28:0/2362283074 wait complete. 2024-02-21T23:07:03.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.881+0000 7f011b000640 1 Processor -- start 2024-02-21T23:07:03.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.882+0000 7f011b000640 1 -- start start 2024-02-21T23:07:03.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.882+0000 7f011b000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00fc004d40 0x7f01141d8f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:03.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.883+0000 7f011b000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0114148260 0x7f01140bc920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:03.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.883+0000 7f011b000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f01141e9810 0x7f01141ebb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:03.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.883+0000 7f011b000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01140c15f0 con 0x7f00fc004d40 2024-02-21T23:07:03.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.883+0000 7f011b000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01140c5560 con 0x7f0114148260 2024-02-21T23:07:03.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.883+0000 7f011b000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01140c1770 con 0x7f01141e9810 2024-02-21T23:07:03.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.883+0000 7f0119c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00fc004d40 0x7f01141d8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:03.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.883+0000 7f0119c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00fc004d40 0x7f01141d8f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47790/0 (socket says 172.21.15.28:47790) 2024-02-21T23:07:03.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.883+0000 7f0119c00640 1 -- 172.21.15.28:0/2413564373 learned_addr learned my addr 172.21.15.28:0/2413564373 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:03.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.883+0000 7f0119c00640 1 -- 172.21.15.28:0/2413564373 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f01141e9810 msgr2=0x7f01141ebb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:03.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.883+0000 7f0119200640 1 --2- 172.21.15.28:0/2413564373 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0114148260 0x7f01140bc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:03.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.883+0000 7f0119c00640 1 --2- 172.21.15.28:0/2413564373 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f01141e9810 0x7f01141ebb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:03.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.883+0000 7f0119c00640 1 -- 172.21.15.28:0/2413564373 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0114148260 msgr2=0x7f01140bc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:03.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.883+0000 7f0119c00640 1 --2- 172.21.15.28:0/2413564373 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0114148260 0x7f01140bc920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:03.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.883+0000 7f0119c00640 1 -- 172.21.15.28:0/2413564373 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01141526e0 con 0x7f00fc004d40 2024-02-21T23:07:03.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.884+0000 7f0119c00640 1 --2- 172.21.15.28:0/2413564373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00fc004d40 0x7f01141d8f50 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f010800d860 tx=0x7f010800dd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:03.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.884+0000 7f0111600640 1 -- 172.21.15.28:0/2413564373 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0108019070 con 0x7f00fc004d40 2024-02-21T23:07:03.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.884+0000 7f0111600640 1 -- 172.21.15.28:0/2413564373 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01080074e0 con 0x7f00fc004d40 2024-02-21T23:07:03.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.884+0000 7f011b000640 1 -- 172.21.15.28:0/2413564373 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01141ec600 con 0x7f00fc004d40 2024-02-21T23:07:03.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.884+0000 7f0111600640 1 -- 172.21.15.28:0/2413564373 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0108004a00 con 0x7f00fc004d40 2024-02-21T23:07:03.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.884+0000 7f011b000640 1 -- 172.21.15.28:0/2413564373 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01140bc400 con 0x7f00fc004d40 2024-02-21T23:07:03.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.886+0000 7f0111600640 1 -- 172.21.15.28:0/2413564373 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f010801c070 con 0x7f00fc004d40 2024-02-21T23:07:03.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.887+0000 7f0111600640 1 --2- 172.21.15.28:0/2413564373 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f00f406b500 0x7f00f406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:03.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.887+0000 7f0111600640 1 -- 172.21.15.28:0/2413564373 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f0108089df0 con 0x7f00fc004d40 2024-02-21T23:07:03.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.888+0000 7f0119200640 1 --2- 172.21.15.28:0/2413564373 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f00f406b500 0x7f00f406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:03.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.889+0000 7f0119200640 1 --2- 172.21.15.28:0/2413564373 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f00f406b500 0x7f00f406d9c0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f0100004790 tx=0x7f0100008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:03.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.890+0000 7f011f446740 1 Processor -- start 2024-02-21T23:07:03.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.890+0000 7f011f446740 1 -- start start 2024-02-21T23:07:03.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.891+0000 7f011f446740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf84f7c020 0x55bf84f7c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:03.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.891+0000 7f011f446740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55bf84f7d330 0x55bf84fb8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:03.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.891+0000 7f011f446740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bf84fb86b0 0x55bf84fb6720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:03.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.891+0000 7f011f446740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf84ee0470 con 0x55bf84f7c020 2024-02-21T23:07:03.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.891+0000 7f011f446740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf84e89af0 con 0x55bf84fb86b0 2024-02-21T23:07:03.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.891+0000 7f011f446740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bf84edc510 con 0x55bf84f7d330 2024-02-21T23:07:03.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.892+0000 7f0119c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf84f7c020 0x55bf84f7c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:03.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.892+0000 7f0119c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf84f7c020 0x55bf84f7c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47792/0 (socket says 172.21.15.28:47792) 2024-02-21T23:07:03.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.892+0000 7f0119c00640 1 -- 172.21.15.28:0/707719842 learned_addr learned my addr 172.21.15.28:0/707719842 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:03.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.892+0000 7f0119c00640 1 -- 172.21.15.28:0/707719842 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55bf84f7d330 msgr2=0x55bf84fb8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:03.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.892+0000 7f0119c00640 1 --2- 172.21.15.28:0/707719842 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55bf84f7d330 0x55bf84fb8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:03.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.892+0000 7f0119c00640 1 -- 172.21.15.28:0/707719842 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bf84fb86b0 msgr2=0x55bf84fb6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:03.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.892+0000 7f0119c00640 1 --2- 172.21.15.28:0/707719842 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bf84fb86b0 0x55bf84fb6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:03.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.892+0000 7f0119c00640 1 -- 172.21.15.28:0/707719842 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bf84e90d90 con 0x55bf84f7c020 2024-02-21T23:07:03.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.892+0000 7f0119c00640 1 --2- 172.21.15.28:0/707719842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf84f7c020 0x55bf84f7c400 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f0108088190 tx=0x7f0108058530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e34370acb42a929d server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:03.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.893+0000 7f0106000640 1 -- 172.21.15.28:0/707719842 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0108060c40 con 0x55bf84f7c020 2024-02-21T23:07:03.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.893+0000 7f0106000640 1 -- 172.21.15.28:0/707719842 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0108086070 con 0x55bf84f7c020 2024-02-21T23:07:03.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.893+0000 7f0106000640 1 -- 172.21.15.28:0/707719842 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0108054ba0 con 0x55bf84f7c020 2024-02-21T23:07:03.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.893+0000 7f011f446740 1 -- 172.21.15.28:0/707719842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf84f7c020 msgr2=0x55bf84f7c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:03.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.893+0000 7f011f446740 1 --2- 172.21.15.28:0/707719842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf84f7c020 0x55bf84f7c400 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f0108088190 tx=0x7f0108058530 comp rx=0 tx=0).stop 2024-02-21T23:07:03.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.894+0000 7f011f446740 1 -- 172.21.15.28:0/707719842 shutdown_connections 2024-02-21T23:07:03.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.894+0000 7f011f446740 1 --2- 172.21.15.28:0/707719842 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55bf84f7d330 0x55bf84fb8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:03.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.894+0000 7f011f446740 1 --2- 172.21.15.28:0/707719842 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bf84fb86b0 0x55bf84fb6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:03.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.894+0000 7f011f446740 1 --2- 172.21.15.28:0/707719842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf84f7c020 0x55bf84f7c400 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:03.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.894+0000 7f011f446740 1 -- 172.21.15.28:0/707719842 >> 172.21.15.28:0/707719842 conn(0x55bf84f78970 msgr2=0x55bf84f9d380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:03.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.894+0000 7f011f446740 1 -- 172.21.15.28:0/707719842 shutdown_connections 2024-02-21T23:07:03.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.894+0000 7f011f446740 1 -- 172.21.15.28:0/707719842 wait complete. 2024-02-21T23:07:03.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.894+0000 7f011f446740 1 -- start start 2024-02-21T23:07:03.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.895+0000 7f011f446740 1 Processor -- start 2024-02-21T23:07:03.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.896+0000 7f011f446740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bf84f7d330 0x55bf84f9ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:03.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.896+0000 7f011f446740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55bf84fb86b0 0x55bf84fcb930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:03.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.896+0000 7f011f446740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf84f98550 0x55bf84fc9950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:03.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.896+0000 7f011f446740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf84e7f040 con 0x55bf84f98550 2024-02-21T23:07:03.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.896+0000 7f011f446740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf84f9da90 con 0x55bf84f7d330 2024-02-21T23:07:03.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.896+0000 7f011f446740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bf84fca670 con 0x55bf84fb86b0 2024-02-21T23:07:03.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.896+0000 7f011f446740 10 client.0 _ll_register_callbacks cb 0x55bf84e9c920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:07:03.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.896+0000 7f011a600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf84f98550 0x55bf84fc9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:03.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.896+0000 7f011a600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf84f98550 0x55bf84fc9950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47802/0 (socket says 172.21.15.28:47802) 2024-02-21T23:07:03.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.896+0000 7f011a600640 1 -- 172.21.15.28:0/1416809859 learned_addr learned my addr 172.21.15.28:0/1416809859 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:03.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.896+0000 7f0119c00640 1 --2- 172.21.15.28:0/1416809859 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bf84f7d330 0x55bf84f9ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:03.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.896+0000 7f0119200640 1 --2- 172.21.15.28:0/1416809859 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55bf84fb86b0 0x55bf84fcb930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:03.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.896+0000 7f011a600640 1 -- 172.21.15.28:0/1416809859 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55bf84fb86b0 msgr2=0x55bf84fcb930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:03.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.896+0000 7f011a600640 1 --2- 172.21.15.28:0/1416809859 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55bf84fb86b0 0x55bf84fcb930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:03.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.896+0000 7f011a600640 1 -- 172.21.15.28:0/1416809859 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bf84f7d330 msgr2=0x55bf84f9ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:03.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.896+0000 7f011a600640 1 --2- 172.21.15.28:0/1416809859 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bf84f7d330 0x55bf84f9ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:03.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.896+0000 7f011a600640 1 -- 172.21.15.28:0/1416809859 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bf84e9eaa0 con 0x55bf84f98550 2024-02-21T23:07:03.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.896+0000 7f0119c00640 1 --2- 172.21.15.28:0/1416809859 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bf84f7d330 0x55bf84f9ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:07:03.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.897+0000 7f011a600640 1 --2- 172.21.15.28:0/1416809859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf84f98550 0x55bf84fc9950 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f010c037040 tx=0x7f010c007040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=feb7913c90660a0c server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:03.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.897+0000 7f00faa00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:07:03.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.897+0000 7f00faa00640 1 -- 172.21.15.28:0/1416809859 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f010c039070 con 0x55bf84f98550 2024-02-21T23:07:03.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.897+0000 7f00faa00640 1 -- 172.21.15.28:0/1416809859 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f010c034850 con 0x55bf84f98550 2024-02-21T23:07:03.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.897+0000 7f011f446740 1 -- 172.21.15.28:0/1416809859 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0108086070 con 0x55bf84f98550 2024-02-21T23:07:03.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.897+0000 7f00faa00640 1 -- 172.21.15.28:0/1416809859 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=88) v2 ==== 24+0+0 (secure 0 0 0) 0x7f010c02de30 con 0x55bf84f98550 2024-02-21T23:07:03.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.897+0000 7f011f446740 10 client.7239 fetch_fsmap learned FSMap version 88 2024-02-21T23:07:03.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.897+0000 7f011f446740 1 -- 172.21.15.28:0/1416809859 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=88,monmap=2+}) v3 -- 0x55bf84bf52d0 con 0x55bf84f98550 2024-02-21T23:07:03.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.898+0000 7f00faa00640 1 -- 172.21.15.28:0/1416809859 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0108086070 con 0x55bf84f98550 2024-02-21T23:07:03.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.898+0000 7f00faa00640 1 -- 172.21.15.28:0/1416809859 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 88) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f010c034a50 con 0x55bf84f98550 2024-02-21T23:07:03.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.898+0000 7f011f446740 10 client.7239 fetch_fsmap finished waiting for FSMap version 88 2024-02-21T23:07:03.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.898+0000 7f011f446740 10 client.7239 resolve_mds: resolved name 'a' to daemon mds.a 2024-02-21T23:07:03.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.898+0000 7f011f446740 20 client.7239 populate_metadata read hostname 'smithi028' 2024-02-21T23:07:03.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.898+0000 7f011f446740 1 --2- 172.21.15.28:0/1416809859 >> [v2:172.21.15.28:6834/2477712025,v1:172.21.15.28:6835/2477712025] conn(0x55bf84f9b720 0x55bf84f97ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:03.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.898+0000 7f011f446740 4 client.7239 mds_command: new command op to 5932 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-21T23:07:03.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.898+0000 7f011f446740 1 -- 172.21.15.28:0/1416809859 --> [v2:172.21.15.28:6834/2477712025,v1:172.21.15.28:6835/2477712025] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bf84eed9b0 con 0x55bf84f9b720 2024-02-21T23:07:03.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.898+0000 7f0119c00640 1 --2- 172.21.15.28:0/1416809859 >> [v2:172.21.15.28:6834/2477712025,v1:172.21.15.28:6835/2477712025] conn(0x55bf84f9b720 0x55bf84f97ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:03.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.899+0000 7f0119c00640 1 --2- 172.21.15.28:0/1416809859 >> [v2:172.21.15.28:6834/2477712025,v1:172.21.15.28:6835/2477712025] conn(0x55bf84f9b720 0x55bf84f97ab0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eed8d0b07d7a6ba7 server_cookie=8cd13552c2c4fd55 in_seq=0 out_seq=0 2024-02-21T23:07:03.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.899+0000 7f00faa00640 10 client.7239 ms_handle_connect on v2:172.21.15.28:6834/2477712025 2024-02-21T23:07:03.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.899+0000 7f00faa00640 1 -- 172.21.15.28:0/1416809859 <== mds.0 v2:172.21.15.28:6834/2477712025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55bf84eed9b0 con 0x55bf84f9b720 2024-02-21T23:07:03.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.899+0000 7f00faa00640 10 client.7239 handle_command_reply: tid=0 2024-02-21T23:07:03.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.900+0000 7f011f446740 1 client.7239 shutdown 2024-02-21T23:07:03.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.900+0000 7f011f446740 1 -- 172.21.15.28:0/1416809859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf84f98550 msgr2=0x55bf84fc9950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:03.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.900+0000 7f011f446740 1 --2- 172.21.15.28:0/1416809859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf84f98550 0x55bf84fc9950 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f010c037040 tx=0x7f010c007040 comp rx=0 tx=0).stop 2024-02-21T23:07:03.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.900+0000 7f011f446740 1 -- 172.21.15.28:0/1416809859 shutdown_connections 2024-02-21T23:07:03.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.900+0000 7f011f446740 1 --2- 172.21.15.28:0/1416809859 >> [v2:172.21.15.28:6834/2477712025,v1:172.21.15.28:6835/2477712025] conn(0x55bf84f9b720 0x55bf84f97ab0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:03.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.900+0000 7f011f446740 1 --2- 172.21.15.28:0/1416809859 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55bf84fb86b0 0x55bf84fcb930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:03.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.900+0000 7f011f446740 1 --2- 172.21.15.28:0/1416809859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf84f98550 0x55bf84fc9950 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:03.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.900+0000 7f011f446740 1 --2- 172.21.15.28:0/1416809859 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bf84f7d330 0x55bf84f9ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:03.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.900+0000 7f00faa00640 0 client.7239 ms_handle_reset on v2:172.21.15.28:6834/2477712025 2024-02-21T23:07:03.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.900+0000 7f011f446740 1 -- 172.21.15.28:0/1416809859 >> 172.21.15.28:0/1416809859 conn(0x55bf84f7c020 msgr2=0x55bf84fb6ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:03.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.900+0000 7f011f446740 1 -- 172.21.15.28:0/1416809859 shutdown_connections 2024-02-21T23:07:03.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.900+0000 7f011f446740 1 -- 172.21.15.28:0/1416809859 wait complete. 2024-02-21T23:07:03.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.900+0000 7f011f446740 20 client.7239 trim_cache size 0 max 16384 2024-02-21T23:07:03.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.931+0000 7f011f446740 1 Processor -- start 2024-02-21T23:07:03.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.932+0000 7f011f446740 1 -- start start 2024-02-21T23:07:03.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.933+0000 7f011f446740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bf84f7d330 0x55bf84f9ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:03.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.933+0000 7f011f446740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf84f98550 0x55bf84f99a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:03.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.933+0000 7f011f446740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55bf84fb86b0 0x55bf84f99f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:03.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.933+0000 7f011f446740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01141f4ff0 con 0x55bf84f98550 2024-02-21T23:07:03.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.933+0000 7f011f446740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01141e90a0 con 0x55bf84f7d330 2024-02-21T23:07:03.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.933+0000 7f011f446740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01141f49f0 con 0x55bf84fb86b0 2024-02-21T23:07:03.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.933+0000 7f0119200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf84f98550 0x55bf84f99a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:03.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.933+0000 7f0119200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf84f98550 0x55bf84f99a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47810/0 (socket says 172.21.15.28:47810) 2024-02-21T23:07:03.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.933+0000 7f0119200640 1 -- 172.21.15.28:0/1475613443 learned_addr learned my addr 172.21.15.28:0/1475613443 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:03.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.934+0000 7f0119c00640 1 --2- 172.21.15.28:0/1475613443 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bf84f7d330 0x55bf84f9ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:03.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.934+0000 7f011a600640 1 --2- 172.21.15.28:0/1475613443 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55bf84fb86b0 0x55bf84f99f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:03.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.934+0000 7f0119200640 1 -- 172.21.15.28:0/1475613443 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55bf84fb86b0 msgr2=0x55bf84f99f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:03.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.934+0000 7f0119200640 1 --2- 172.21.15.28:0/1475613443 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55bf84fb86b0 0x55bf84f99f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:03.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.934+0000 7f0119200640 1 -- 172.21.15.28:0/1475613443 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bf84f7d330 msgr2=0x55bf84f9ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:03.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.934+0000 7f0119200640 1 --2- 172.21.15.28:0/1475613443 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bf84f7d330 0x55bf84f9ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:03.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.934+0000 7f0119200640 1 -- 172.21.15.28:0/1475613443 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bf84cbf6b0 con 0x55bf84f98550 2024-02-21T23:07:03.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.934+0000 7f0119200640 1 --2- 172.21.15.28:0/1475613443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf84f98550 0x55bf84f99a50 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f0100010a70 tx=0x7f0100013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=364a0ea316e65f1a server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:03.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.935+0000 7f0105600640 1 -- 172.21.15.28:0/1475613443 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0100027070 con 0x55bf84f98550 2024-02-21T23:07:03.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.935+0000 7f0105600640 1 -- 172.21.15.28:0/1475613443 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0100004440 con 0x55bf84f98550 2024-02-21T23:07:03.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.935+0000 7f0105600640 1 -- 172.21.15.28:0/1475613443 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f010000f040 con 0x55bf84f98550 2024-02-21T23:07:03.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.935+0000 7f011f446740 1 -- 172.21.15.28:0/1475613443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf84f98550 msgr2=0x55bf84f99a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:03.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.935+0000 7f011f446740 1 --2- 172.21.15.28:0/1475613443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf84f98550 0x55bf84f99a50 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f0100010a70 tx=0x7f0100013000 comp rx=0 tx=0).stop 2024-02-21T23:07:03.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.935+0000 7f011f446740 1 -- 172.21.15.28:0/1475613443 shutdown_connections 2024-02-21T23:07:03.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.935+0000 7f011f446740 1 --2- 172.21.15.28:0/1475613443 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55bf84fb86b0 0x55bf84f99f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:03.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.935+0000 7f011f446740 1 --2- 172.21.15.28:0/1475613443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf84f98550 0x55bf84f99a50 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:03.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.935+0000 7f011f446740 1 --2- 172.21.15.28:0/1475613443 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bf84f7d330 0x55bf84f9ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:03.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.935+0000 7f011f446740 1 -- 172.21.15.28:0/1475613443 >> 172.21.15.28:0/1475613443 conn(0x55bf84f7c020 msgr2=0x55bf84f71d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:03.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.935+0000 7f011f446740 1 -- 172.21.15.28:0/1475613443 shutdown_connections 2024-02-21T23:07:03.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.935+0000 7f011f446740 1 -- 172.21.15.28:0/1475613443 wait complete. 2024-02-21T23:07:03.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.936+0000 7f011f446740 1 -- start start 2024-02-21T23:07:03.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.936+0000 7f011f446740 1 Processor -- start 2024-02-21T23:07:03.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.937+0000 7f011f446740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55bf84f7d330 0x55bf84fcccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:03.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.937+0000 7f011f446740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bf84f98550 0x55bf84f9a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:03.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.937+0000 7f011f446740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf84fb86b0 0x55bf8501df90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:03.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.937+0000 7f011f446740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf84f9b5a0 con 0x55bf84fb86b0 2024-02-21T23:07:03.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.937+0000 7f011f446740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf84fca470 con 0x55bf84f98550 2024-02-21T23:07:03.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.937+0000 7f011f446740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bf84f9bba0 con 0x55bf84f7d330 2024-02-21T23:07:03.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.937+0000 7f011f446740 10 client.0 _ll_register_callbacks cb 0x7f00fc0036f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:07:03.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.938+0000 7f011a600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf84fb86b0 0x55bf8501df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:03.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.938+0000 7f011a600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf84fb86b0 0x55bf8501df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47826/0 (socket says 172.21.15.28:47826) 2024-02-21T23:07:03.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.938+0000 7f011a600640 1 -- 172.21.15.28:0/3220592753 learned_addr learned my addr 172.21.15.28:0/3220592753 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:03.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.938+0000 7f0119c00640 1 --2- 172.21.15.28:0/3220592753 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55bf84f7d330 0x55bf84fcccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:03.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.938+0000 7f0119200640 1 --2- 172.21.15.28:0/3220592753 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bf84f98550 0x55bf84f9a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:03.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.938+0000 7f011a600640 1 -- 172.21.15.28:0/3220592753 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55bf84f7d330 msgr2=0x55bf84fcccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:03.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.938+0000 7f011a600640 1 --2- 172.21.15.28:0/3220592753 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55bf84f7d330 0x55bf84fcccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:03.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.938+0000 7f011a600640 1 -- 172.21.15.28:0/3220592753 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bf84f98550 msgr2=0x55bf84f9a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:03.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.938+0000 7f011a600640 1 --2- 172.21.15.28:0/3220592753 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bf84f98550 0x55bf84f9a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:03.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.938+0000 7f011a600640 1 -- 172.21.15.28:0/3220592753 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bf84fd2240 con 0x55bf84fb86b0 2024-02-21T23:07:03.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.938+0000 7f011a600640 1 --2- 172.21.15.28:0/3220592753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf84fb86b0 0x55bf8501df90 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f010c037040 tx=0x7f010c02e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7670e8b103b797e6 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:03.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.938+0000 7f00faa00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:07:03.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.939+0000 7f00faa00640 1 -- 172.21.15.28:0/3220592753 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f010c034b20 con 0x55bf84fb86b0 2024-02-21T23:07:03.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.939+0000 7f00faa00640 1 -- 172.21.15.28:0/3220592753 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f010c03e490 con 0x55bf84fb86b0 2024-02-21T23:07:03.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.939+0000 7f011f446740 1 -- 172.21.15.28:0/3220592753 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0100004440 con 0x55bf84fb86b0 2024-02-21T23:07:03.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.939+0000 7f00faa00640 1 -- 172.21.15.28:0/3220592753 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=88) v2 ==== 24+0+0 (secure 0 0 0) 0x7f010c05e050 con 0x55bf84fb86b0 2024-02-21T23:07:03.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.939+0000 7f011f446740 10 client.7245 fetch_fsmap learned FSMap version 88 2024-02-21T23:07:03.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.939+0000 7f011f446740 1 -- 172.21.15.28:0/3220592753 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=88,monmap=2+}) v3 -- 0x55bf84f97de0 con 0x55bf84fb86b0 2024-02-21T23:07:03.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.939+0000 7f00faa00640 1 -- 172.21.15.28:0/3220592753 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0100004440 con 0x55bf84fb86b0 2024-02-21T23:07:03.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.939+0000 7f00faa00640 1 -- 172.21.15.28:0/3220592753 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 88) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f010c061020 con 0x55bf84fb86b0 2024-02-21T23:07:03.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.939+0000 7f011f446740 10 client.7245 fetch_fsmap finished waiting for FSMap version 88 2024-02-21T23:07:03.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.939+0000 7f011f446740 10 client.7245 resolve_mds: resolved name 'a' to daemon mds.a 2024-02-21T23:07:03.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.940+0000 7f011f446740 20 client.7245 populate_metadata read hostname 'smithi028' 2024-02-21T23:07:03.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.940+0000 7f011f446740 1 --2- 172.21.15.28:0/3220592753 >> [v2:172.21.15.28:6834/2477712025,v1:172.21.15.28:6835/2477712025] conn(0x55bf8501e8f0 0x55bf85037b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:03.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.940+0000 7f011f446740 4 client.7245 mds_command: new command op to 5932 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-21T23:07:03.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.940+0000 7f011f446740 1 -- 172.21.15.28:0/3220592753 --> [v2:172.21.15.28:6834/2477712025,v1:172.21.15.28:6835/2477712025] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55bf84fbb020 con 0x55bf8501e8f0 2024-02-21T23:07:03.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.940+0000 7f0119c00640 1 --2- 172.21.15.28:0/3220592753 >> [v2:172.21.15.28:6834/2477712025,v1:172.21.15.28:6835/2477712025] conn(0x55bf8501e8f0 0x55bf85037b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:03.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.940+0000 7f0119c00640 1 --2- 172.21.15.28:0/3220592753 >> [v2:172.21.15.28:6834/2477712025,v1:172.21.15.28:6835/2477712025] conn(0x55bf8501e8f0 0x55bf85037b50 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7bae423cec4623be server_cookie=a7d965f5c406d2d3 in_seq=0 out_seq=0 2024-02-21T23:07:03.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.940+0000 7f00faa00640 10 client.7245 ms_handle_connect on v2:172.21.15.28:6834/2477712025 2024-02-21T23:07:03.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.941+0000 7f00faa00640 1 -- 172.21.15.28:0/3220592753 <== mds.0 v2:172.21.15.28:6834/2477712025 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x55bf84fbb020 con 0x55bf8501e8f0 2024-02-21T23:07:03.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.941+0000 7f00faa00640 10 client.7245 handle_command_reply: tid=0 2024-02-21T23:07:03.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.941+0000 7f011f446740 1 client.7245 shutdown 2024-02-21T23:07:03.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.941+0000 7f011f446740 1 -- 172.21.15.28:0/3220592753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf84fb86b0 msgr2=0x55bf8501df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:03.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.941+0000 7f011f446740 1 --2- 172.21.15.28:0/3220592753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf84fb86b0 0x55bf8501df90 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f010c037040 tx=0x7f010c02e070 comp rx=0 tx=0).stop 2024-02-21T23:07:03.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.941+0000 7f011f446740 1 -- 172.21.15.28:0/3220592753 shutdown_connections 2024-02-21T23:07:03.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.941+0000 7f011f446740 1 --2- 172.21.15.28:0/3220592753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf84fb86b0 0x55bf8501df90 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:03.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.941+0000 7f011f446740 1 --2- 172.21.15.28:0/3220592753 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bf84f98550 0x55bf84f9a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:03.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.941+0000 7f011f446740 1 --2- 172.21.15.28:0/3220592753 >> [v2:172.21.15.28:6834/2477712025,v1:172.21.15.28:6835/2477712025] conn(0x55bf8501e8f0 0x55bf85037b50 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:03.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.941+0000 7f011f446740 1 --2- 172.21.15.28:0/3220592753 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55bf84f7d330 0x55bf84fcccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:03.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.941+0000 7f011f446740 1 -- 172.21.15.28:0/3220592753 >> 172.21.15.28:0/3220592753 conn(0x55bf84f7c020 msgr2=0x55bf84fb7ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:03.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.941+0000 7f00faa00640 0 client.7245 ms_handle_reset on v2:172.21.15.28:6834/2477712025 2024-02-21T23:07:03.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.941+0000 7f011f446740 1 -- 172.21.15.28:0/3220592753 shutdown_connections 2024-02-21T23:07:03.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.941+0000 7f011f446740 1 -- 172.21.15.28:0/3220592753 wait complete. 2024-02-21T23:07:03.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.941+0000 7f011f446740 20 client.7245 trim_cache size 0 max 16384 2024-02-21T23:07:03.933 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-21T23:07:03.933 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac", 2024-02-21T23:07:03.933 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-02-21T23:07:03.933 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 5932, 2024-02-21T23:07:03.933 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-02-21T23:07:03.933 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-02-21T23:07:03.933 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-02-21T23:07:03.933 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 9.1546688090000004, 2024-02-21T23:07:03.934 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 88, 2024-02-21T23:07:03.934 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 98, 2024-02-21T23:07:03.934 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 98, 2024-02-21T23:07:03.934 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 14.673722436 2024-02-21T23:07:03.934 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-21T23:07:03.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.942+0000 7f011b000640 1 -- 172.21.15.28:0/2413564373 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f00f406b500 msgr2=0x7f00f406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:03.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.942+0000 7f011b000640 1 --2- 172.21.15.28:0/2413564373 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f00f406b500 0x7f00f406d9c0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f0100004790 tx=0x7f0100008040 comp rx=0 tx=0).stop 2024-02-21T23:07:03.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.942+0000 7f011b000640 1 -- 172.21.15.28:0/2413564373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00fc004d40 msgr2=0x7f01141d8f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:03.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.942+0000 7f011b000640 1 --2- 172.21.15.28:0/2413564373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00fc004d40 0x7f01141d8f50 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f010800d860 tx=0x7f010800dd20 comp rx=0 tx=0).stop 2024-02-21T23:07:03.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.942+0000 7f011b000640 1 -- 172.21.15.28:0/2413564373 shutdown_connections 2024-02-21T23:07:03.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.942+0000 7f011b000640 1 --2- 172.21.15.28:0/2413564373 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f00f406b500 0x7f00f406d9c0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:03.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.942+0000 7f011b000640 1 --2- 172.21.15.28:0/2413564373 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f01141e9810 0x7f01141ebb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:03.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.942+0000 7f011b000640 1 --2- 172.21.15.28:0/2413564373 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0114148260 0x7f01140bc920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:03.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.942+0000 7f011b000640 1 --2- 172.21.15.28:0/2413564373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00fc004d40 0x7f01141d8f50 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:03.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.942+0000 7f011b000640 1 -- 172.21.15.28:0/2413564373 >> 172.21.15.28:0/2413564373 conn(0x7f01140a8320 msgr2=0x7f011406eb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:03.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.942+0000 7f011b000640 1 -- 172.21.15.28:0/2413564373 shutdown_connections 2024-02-21T23:07:03.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:03.942+0000 7f011b000640 1 -- 172.21.15.28:0/2413564373 wait complete. 2024-02-21T23:07:03.947 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 7, "epoch": 88, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 7, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-02-21T23:06:54.201589+0000", "damaged": [], "data_pools": [ 16 ], "enabled": true, "epoch": 88, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5932": { "addr": "172.21.15.28:6835/2477712025", "addrs": { "addrvec": [ { "addr": "172.21.15.28:6834", "nonce": 2477712025, "type": "v2" }, { "addr": "172.21.15.28:6835", "nonce": 2477712025, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5932, "incarnation": 84, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 15, "modified": "2024-02-21T23:06:57.273466+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5932 } } } ], "standbys": [ { "addr": "172.21.15.28:6833/85689296", "addrs": { "addrvec": [ { "addr": "172.21.15.28:6832", "nonce": 85689296, "type": "v2" }, { "addr": "172.21.15.28:6833", "nonce": 85689296, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 82, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5555, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.81:6835/1194613007", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6834", "nonce": 1194613007, "type": "v2" }, { "addr": "172.21.15.81:6835", "nonce": 1194613007, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 83, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5567, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.81:6837/2315735157", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6836", "nonce": 2315735157, "type": "v2" }, { "addr": "172.21.15.81:6837", "nonce": 2315735157, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 83, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5570, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-02-21T23:07:03.948 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': 7, 'name': 'c', 'state': 'up:active'}, {'join_fscid': -1, 'name': 'd'}] 2024-02-21T23:07:03.948 INFO:tasks.cephfs.test_failover:current = [{'gid': 5932, 'name': 'a', 'rank': 0, 'incarnation': 84, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/2477712025', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 2477712025}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 2477712025}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 5567, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.81:6835/1194613007', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6834', 'nonce': 1194613007}, {'type': 'v1', 'addr': '172.21.15.81:6835', 'nonce': 1194613007}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 83}, {'gid': 5555, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.28:6833/85689296', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 85689296}, {'type': 'v1', 'addr': '172.21.15.28:6833', 'nonce': 85689296}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 82}, {'gid': 5570, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.81:6837/2315735157', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 2315735157}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 2315735157}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 83}] 2024-02-21T23:07:03.948 DEBUG:tasks.cephfs.test_failover:7 != -1 2024-02-21T23:07:03.948 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-02-21T23:07:05.853 INFO:tasks.ceph.mds.a.smithi028.stderr: -261> 2024-02-21T23:07:00.705+0000 7fe55f400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-21T23:07:05.903 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-02-21T23:07:06.349 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-21T23:07:06.357 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-21T23:07:08.949 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:07:09.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.177+0000 7fd5b5400640 1 Processor -- start 2024-02-21T23:07:09.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.177+0000 7fd5b5400640 1 -- start start 2024-02-21T23:07:09.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.179+0000 7fd5b5400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd5b00aff90 0x7fd5b00c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:09.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.179+0000 7fd5b5400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fd5b00c5720 0x7fd5b015f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:09.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.179+0000 7fd5b5400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5b00c74c0 con 0x7fd5b006dbf0 2024-02-21T23:07:09.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.179+0000 7fd5b5400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5b00c7270 con 0x7fd5b00c5720 2024-02-21T23:07:09.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.179+0000 7fd5b5400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b0158630 con 0x7fd5b00aff90 2024-02-21T23:07:09.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.180+0000 7fd5b4a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fd5b00c5720 0x7fd5b015f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:09.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.180+0000 7fd5b4a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fd5b00c5720 0x7fd5b015f890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.28:54704/0 (socket says 172.21.15.28:54704) 2024-02-21T23:07:09.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.180+0000 7fd5b4a00640 1 -- 172.21.15.28:0/3936054648 learned_addr learned my addr 172.21.15.28:0/3936054648 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:09.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.180+0000 7fd5afe00640 1 --2- 172.21.15.28:0/3936054648 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd5b00aff90 0x7fd5b00c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:09.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.181+0000 7fd5b4a00640 1 -- 172.21.15.28:0/3936054648 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd5b00aff90 msgr2=0x7fd5b00c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:09.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.181+0000 7fd5b4a00640 1 --2- 172.21.15.28:0/3936054648 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd5b00aff90 0x7fd5b00c51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:09.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.181+0000 7fd5b4a00640 1 -- 172.21.15.28:0/3936054648 >> v1:172.21.15.81:6789/0 conn(0x7fd5b006dbf0 legacy=0x7fd5b0158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:09.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.181+0000 7fd5b4a00640 1 -- 172.21.15.28:0/3936054648 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5b00c77a0 con 0x7fd5b00c5720 2024-02-21T23:07:09.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.182+0000 7fd5b4a00640 1 --2- 172.21.15.28:0/3936054648 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5b00c5720 0x7fd5b015f890 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fd5a8009e50 tx=0x7fd5a802bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=32dfa248a4b0b9f0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:09.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.183+0000 7fd5af400640 1 -- 172.21.15.28:0/3936054648 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a802cb00 con 0x7fd5b00c5720 2024-02-21T23:07:09.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.183+0000 7fd5af400640 1 -- 172.21.15.28:0/3936054648 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5a802cca0 con 0x7fd5b00c5720 2024-02-21T23:07:09.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.183+0000 7fd5af400640 1 -- 172.21.15.28:0/3936054648 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a8034610 con 0x7fd5b00c5720 2024-02-21T23:07:09.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.183+0000 7fd5b5400640 1 -- 172.21.15.28:0/3936054648 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5b00c5720 msgr2=0x7fd5b015f890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:09.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.183+0000 7fd5b5400640 1 --2- 172.21.15.28:0/3936054648 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5b00c5720 0x7fd5b015f890 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fd5a8009e50 tx=0x7fd5a802bf60 comp rx=0 tx=0).stop 2024-02-21T23:07:09.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.183+0000 7fd5b5400640 1 -- 172.21.15.28:0/3936054648 shutdown_connections 2024-02-21T23:07:09.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.183+0000 7fd5b5400640 1 --2- 172.21.15.28:0/3936054648 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5b00c5720 0x7fd5b015f890 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:09.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.183+0000 7fd5b5400640 1 --2- 172.21.15.28:0/3936054648 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd5b00aff90 0x7fd5b00c51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:09.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.183+0000 7fd5b5400640 1 -- 172.21.15.28:0/3936054648 >> 172.21.15.28:0/3936054648 conn(0x7fd5b00ac370 msgr2=0x7fd5b00ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:09.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.184+0000 7fd5b5400640 1 -- 172.21.15.28:0/3936054648 shutdown_connections 2024-02-21T23:07:09.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.184+0000 7fd5b5400640 1 -- 172.21.15.28:0/3936054648 wait complete. 2024-02-21T23:07:09.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.185+0000 7fd5b5400640 1 Processor -- start 2024-02-21T23:07:09.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.185+0000 7fd5b5400640 1 -- start start 2024-02-21T23:07:09.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.186+0000 7fd5b5400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5b006dbf0 0x7fd5b01eaeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:09.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.186+0000 7fd5b5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5b00aff90 0x7fd5b00ae740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:09.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.186+0000 7fd5b5400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd5b00c5720 0x7fd5b01533d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:09.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.186+0000 7fd5b5400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b00c8760 con 0x7fd5b00aff90 2024-02-21T23:07:09.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.186+0000 7fd5b5400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b00be0c0 con 0x7fd5b006dbf0 2024-02-21T23:07:09.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.186+0000 7fd5b5400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b00c88e0 con 0x7fd5b00c5720 2024-02-21T23:07:09.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.187+0000 7fd5a7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5b00aff90 0x7fd5b00ae740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:09.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.187+0000 7fd5a7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5b00aff90 0x7fd5b00ae740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55224/0 (socket says 172.21.15.28:55224) 2024-02-21T23:07:09.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.187+0000 7fd5a7e00640 1 -- 172.21.15.28:0/456301910 learned_addr learned my addr 172.21.15.28:0/456301910 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:09.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.187+0000 7fd5afe00640 1 --2- 172.21.15.28:0/456301910 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5b006dbf0 0x7fd5b01eaeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:09.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.187+0000 7fd5b4a00640 1 --2- 172.21.15.28:0/456301910 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd5b00c5720 0x7fd5b01533d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:09.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.187+0000 7fd5a7e00640 1 -- 172.21.15.28:0/456301910 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd5b00c5720 msgr2=0x7fd5b01533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:09.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.187+0000 7fd5a7e00640 1 --2- 172.21.15.28:0/456301910 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd5b00c5720 0x7fd5b01533d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:09.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.187+0000 7fd5a7e00640 1 -- 172.21.15.28:0/456301910 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5b006dbf0 msgr2=0x7fd5b01eaeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:09.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.187+0000 7fd5a7e00640 1 --2- 172.21.15.28:0/456301910 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5b006dbf0 0x7fd5b01eaeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:09.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.187+0000 7fd5a7e00640 1 -- 172.21.15.28:0/456301910 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5b0154190 con 0x7fd5b00aff90 2024-02-21T23:07:09.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.187+0000 7fd5a7e00640 1 --2- 172.21.15.28:0/456301910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5b00aff90 0x7fd5b00ae740 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7fd59800c430 tx=0x7fd59800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:09.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.188+0000 7fd5acc00640 1 -- 172.21.15.28:0/456301910 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5980052d0 con 0x7fd5b00aff90 2024-02-21T23:07:09.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.188+0000 7fd5acc00640 1 -- 172.21.15.28:0/456301910 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd598005490 con 0x7fd5b00aff90 2024-02-21T23:07:09.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.188+0000 7fd5acc00640 1 -- 172.21.15.28:0/456301910 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd598007260 con 0x7fd5b00aff90 2024-02-21T23:07:09.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.188+0000 7fd5b5400640 1 -- 172.21.15.28:0/456301910 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5b0154470 con 0x7fd5b00aff90 2024-02-21T23:07:09.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.188+0000 7fd5b5400640 1 -- 172.21.15.28:0/456301910 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5b00aec80 con 0x7fd5b00aff90 2024-02-21T23:07:09.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.189+0000 7fd5b5400640 1 -- 172.21.15.28:0/456301910 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5b004f290 con 0x7fd5b00aff90 2024-02-21T23:07:09.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.190+0000 7fd5acc00640 1 -- 172.21.15.28:0/456301910 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd598007400 con 0x7fd5b00aff90 2024-02-21T23:07:09.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.191+0000 7fd5acc00640 1 --2- 172.21.15.28:0/456301910 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd58c06b4d0 0x7fd58c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:09.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.191+0000 7fd5acc00640 1 -- 172.21.15.28:0/456301910 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fd5980898e0 con 0x7fd5b00aff90 2024-02-21T23:07:09.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.192+0000 7fd5afe00640 1 --2- 172.21.15.28:0/456301910 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd58c06b4d0 0x7fd58c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:09.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.193+0000 7fd5afe00640 1 --2- 172.21.15.28:0/456301910 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd58c06b4d0 0x7fd58c06d990 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fd5a0009ad0 tx=0x7fd5a0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:09.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.201+0000 7fd5acc00640 1 -- 172.21.15.28:0/456301910 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd598058570 con 0x7fd5b00aff90 2024-02-21T23:07:09.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.441+0000 7fd5b5400640 1 -- 172.21.15.28:0/456301910 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd5b01524f0 con 0x7fd5b00aff90 2024-02-21T23:07:09.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.443+0000 7fd5acc00640 1 -- 172.21.15.28:0/456301910 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+4934 (secure 0 0 0) 0x7fd598057e70 con 0x7fd5b00aff90 2024-02-21T23:07:09.431 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:07:09.431 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":93,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5567,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1194613007","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1194613007},{"type":"v1","addr":"172.21.15.81:6835","nonce":1194613007}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":5570,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/2315735157","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":2315735157},{"type":"v1","addr":"172.21.15.81:6837","nonce":2315735157}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":7251,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2558379712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2558379712},{"type":"v1","addr":"172.21.15.28:6835","nonce":2558379712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":91}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:06:54.201589+0000","modified":"2024-02-21T23:07:07.869167+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":99,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5555},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5555":{"gid":5555,"name":"c","rank":0,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.28:6833/85689296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":85689296},{"type":"v1","addr":"172.21.15.28:6833","nonce":85689296}]},"join_fscid":7,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-02-21T23:07:09.431 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 93 2024-02-21T23:07:09.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.446+0000 7fd5b5400640 1 -- 172.21.15.28:0/456301910 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd58c06b4d0 msgr2=0x7fd58c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:09.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.446+0000 7fd5b5400640 1 --2- 172.21.15.28:0/456301910 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd58c06b4d0 0x7fd58c06d990 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fd5a0009ad0 tx=0x7fd5a0008040 comp rx=0 tx=0).stop 2024-02-21T23:07:09.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.446+0000 7fd5b5400640 1 -- 172.21.15.28:0/456301910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5b00aff90 msgr2=0x7fd5b00ae740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:09.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.446+0000 7fd5b5400640 1 --2- 172.21.15.28:0/456301910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5b00aff90 0x7fd5b00ae740 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7fd59800c430 tx=0x7fd59800c8f0 comp rx=0 tx=0).stop 2024-02-21T23:07:09.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.446+0000 7fd5b5400640 1 -- 172.21.15.28:0/456301910 shutdown_connections 2024-02-21T23:07:09.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.446+0000 7fd5b5400640 1 --2- 172.21.15.28:0/456301910 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd58c06b4d0 0x7fd58c06d990 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:09.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.446+0000 7fd5b5400640 1 --2- 172.21.15.28:0/456301910 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd5b00c5720 0x7fd5b01533d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:09.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.446+0000 7fd5b5400640 1 --2- 172.21.15.28:0/456301910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5b00aff90 0x7fd5b00ae740 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:09.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.446+0000 7fd5b5400640 1 --2- 172.21.15.28:0/456301910 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5b006dbf0 0x7fd5b01eaeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:09.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.446+0000 7fd5b5400640 1 -- 172.21.15.28:0/456301910 >> 172.21.15.28:0/456301910 conn(0x7fd5b00ac370 msgr2=0x7fd5b00ac750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:09.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.446+0000 7fd5b5400640 1 -- 172.21.15.28:0/456301910 shutdown_connections 2024-02-21T23:07:09.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:09.446+0000 7fd5b5400640 1 -- 172.21.15.28:0/456301910 wait complete. 2024-02-21T23:07:09.681 INFO:tasks.ceph.mds.c.smithi028.stderr:2024-02-21T23:07:09.693+0000 7fbe84c00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-21T23:07:09.858 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 93, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:06:54.201589+0000', 'modified': '2024-02-21T23:07:07.869167+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 99, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5555}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5555': {'gid': 5555, 'name': 'c', 'rank': 0, 'incarnation': 90, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.28:6833/85689296', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 85689296}, {'type': 'v1', 'addr': '172.21.15.28:6833', 'nonce': 85689296}]}, 'join_fscid': 7, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:07:09.858 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-21T23:07:09.858 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-02-21T23:07:10.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.101+0000 7fd147e00640 1 Processor -- start 2024-02-21T23:07:10.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.101+0000 7fd147e00640 1 -- start start 2024-02-21T23:07:10.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.102+0000 7fd147e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd1481524c0 0x7fd148155a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:10.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.102+0000 7fd147e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fd1480c85c0 0x7fd1480c70d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:10.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.102+0000 7fd147e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd14806df50 con 0x7fd1480c5040 2024-02-21T23:07:10.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.102+0000 7fd147e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1480c5520 con 0x7fd1481524c0 2024-02-21T23:07:10.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.102+0000 7fd147e00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1480c5800 con 0x7fd1480c85c0 2024-02-21T23:07:10.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.103+0000 7fd146a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd1481524c0 0x7fd148155a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:10.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.103+0000 7fd13fe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd1480c5040 0x7fd14806ce70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:40458/0 (socket says 172.21.15.28:40458) 2024-02-21T23:07:10.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.103+0000 7fd13fe00640 1 -- 172.21.15.28:0/3115695572 learned_addr learned my addr 172.21.15.28:0/3115695572 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:07:10.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.103+0000 7fd147400640 1 --2- 172.21.15.28:0/3115695572 >> v2:172.21.15.81:3300/0 conn(0x7fd1480c85c0 0x7fd1480c70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:10.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.103+0000 7fd146a00640 1 -- 172.21.15.28:0/3115695572 >> v2:172.21.15.81:3300/0 conn(0x7fd1480c85c0 msgr2=0x7fd1480c70d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:10.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.103+0000 7fd146a00640 1 --2- 172.21.15.28:0/3115695572 >> v2:172.21.15.81:3300/0 conn(0x7fd1480c85c0 0x7fd1480c70d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:10.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.103+0000 7fd146a00640 1 -- 172.21.15.28:0/3115695572 >> v1:172.21.15.28:6789/0 conn(0x7fd1480c5040 legacy=0x7fd14806ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:10.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.103+0000 7fd146a00640 1 -- 172.21.15.28:0/3115695572 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd14806c670 con 0x7fd1481524c0 2024-02-21T23:07:10.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.103+0000 7fd146000640 1 -- 172.21.15.28:0/3115695572 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 190542564 0 0) 0x7fd14806df50 con 0x7fd1480c5040 2024-02-21T23:07:10.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.105+0000 7fd146a00640 1 --2- 172.21.15.28:0/3115695572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1481524c0 0x7fd148155a10 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7fd1480c4af0 tx=0x7fd13802b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=404c61722bf2c981 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:10.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.105+0000 7fd146000640 1 -- 172.21.15.28:0/3115695572 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd138038070 con 0x7fd1481524c0 2024-02-21T23:07:10.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.105+0000 7fd146000640 1 -- 172.21.15.28:0/3115695572 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd138031710 con 0x7fd1481524c0 2024-02-21T23:07:10.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.105+0000 7fd146000640 1 -- 172.21.15.28:0/3115695572 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd138031ae0 con 0x7fd1481524c0 2024-02-21T23:07:10.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.106+0000 7fd147e00640 1 -- 172.21.15.28:0/3115695572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1481524c0 msgr2=0x7fd148155a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:10.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.106+0000 7fd147e00640 1 --2- 172.21.15.28:0/3115695572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1481524c0 0x7fd148155a10 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7fd1480c4af0 tx=0x7fd13802b7b0 comp rx=0 tx=0).stop 2024-02-21T23:07:10.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.106+0000 7fd147e00640 1 -- 172.21.15.28:0/3115695572 shutdown_connections 2024-02-21T23:07:10.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.106+0000 7fd147e00640 1 --2- 172.21.15.28:0/3115695572 >> v2:172.21.15.81:3300/0 conn(0x7fd1480c85c0 0x7fd1480c70d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:10.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.106+0000 7fd147e00640 1 --2- 172.21.15.28:0/3115695572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1481524c0 0x7fd148155a10 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:10.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.106+0000 7fd147e00640 1 -- 172.21.15.28:0/3115695572 >> 172.21.15.28:0/3115695572 conn(0x7fd1480ac2b0 msgr2=0x7fd1480ac6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:10.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.107+0000 7fd147e00640 1 -- 172.21.15.28:0/3115695572 shutdown_connections 2024-02-21T23:07:10.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.107+0000 7fd147e00640 1 -- 172.21.15.28:0/3115695572 wait complete. 2024-02-21T23:07:10.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.108+0000 7fd147e00640 1 Processor -- start 2024-02-21T23:07:10.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.108+0000 7fd147e00640 1 -- start start 2024-02-21T23:07:10.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.109+0000 7fd147e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1480c5040 0x7fd1480c0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:10.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.109+0000 7fd147e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1480c85c0 0x7fd14814f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:10.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.109+0000 7fd147e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd1481524c0 0x7fd148147e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:10.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.109+0000 7fd147e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1480c8a70 con 0x7fd1480c85c0 2024-02-21T23:07:10.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.109+0000 7fd147e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1481528a0 con 0x7fd1480c5040 2024-02-21T23:07:10.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.109+0000 7fd147e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1480c7610 con 0x7fd1481524c0 2024-02-21T23:07:10.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.110+0000 7fd13fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1480c85c0 0x7fd14814f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:10.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.110+0000 7fd13fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1480c85c0 0x7fd14814f190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55234/0 (socket says 172.21.15.28:55234) 2024-02-21T23:07:10.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.110+0000 7fd13fe00640 1 -- 172.21.15.28:0/592452904 learned_addr learned my addr 172.21.15.28:0/592452904 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:10.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.110+0000 7fd146a00640 1 --2- 172.21.15.28:0/592452904 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1480c5040 0x7fd1480c0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:10.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.110+0000 7fd13fe00640 1 -- 172.21.15.28:0/592452904 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd1481524c0 msgr2=0x7fd148147e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:10.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.110+0000 7fd147400640 1 --2- 172.21.15.28:0/592452904 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd1481524c0 0x7fd148147e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:10.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.110+0000 7fd13fe00640 1 --2- 172.21.15.28:0/592452904 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd1481524c0 0x7fd148147e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:10.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.110+0000 7fd13fe00640 1 -- 172.21.15.28:0/592452904 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1480c5040 msgr2=0x7fd1480c0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:10.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.110+0000 7fd13fe00640 1 --2- 172.21.15.28:0/592452904 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1480c5040 0x7fd1480c0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:10.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.110+0000 7fd13fe00640 1 -- 172.21.15.28:0/592452904 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd148148380 con 0x7fd1480c85c0 2024-02-21T23:07:10.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.110+0000 7fd146a00640 1 --2- 172.21.15.28:0/592452904 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1480c5040 0x7fd1480c0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:07:10.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.110+0000 7fd13fe00640 1 --2- 172.21.15.28:0/592452904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1480c85c0 0x7fd14814f190 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7fd13000c500 tx=0x7fd13000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:10.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.111+0000 7fd13ea00640 1 -- 172.21.15.28:0/592452904 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd130018070 con 0x7fd1480c85c0 2024-02-21T23:07:10.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.111+0000 7fd13ea00640 1 -- 172.21.15.28:0/592452904 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd130007580 con 0x7fd1480c85c0 2024-02-21T23:07:10.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.111+0000 7fd13ea00640 1 -- 172.21.15.28:0/592452904 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd130004a20 con 0x7fd1480c85c0 2024-02-21T23:07:10.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.111+0000 7fd147e00640 1 -- 172.21.15.28:0/592452904 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd14814f7c0 con 0x7fd1480c85c0 2024-02-21T23:07:10.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.111+0000 7fd147e00640 1 -- 172.21.15.28:0/592452904 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd14814d330 con 0x7fd1480c85c0 2024-02-21T23:07:10.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.113+0000 7fd13ea00640 1 -- 172.21.15.28:0/592452904 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd13001b070 con 0x7fd1480c85c0 2024-02-21T23:07:10.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.115+0000 7fd15083f740 1 Processor -- start 2024-02-21T23:07:10.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.115+0000 7fd13ea00640 1 --2- 172.21.15.28:0/592452904 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd12406b4d0 0x7fd12406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:10.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.115+0000 7fd15083f740 1 -- start start 2024-02-21T23:07:10.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.115+0000 7fd13ea00640 1 -- 172.21.15.28:0/592452904 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fd130089700 con 0x7fd1480c85c0 2024-02-21T23:07:10.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.115+0000 7fd147400640 1 --2- 172.21.15.28:0/592452904 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd12406b4d0 0x7fd12406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:10.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.116+0000 7fd15083f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b086db020 0x557b086db400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:10.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.116+0000 7fd15083f740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557b086dc330 0x557b08717170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:10.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.116+0000 7fd15083f740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x557b087176b0 0x557b08715720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:10.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.116+0000 7fd15083f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b0863f470 con 0x557b086db020 2024-02-21T23:07:10.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.116+0000 7fd15083f740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b085e8af0 con 0x557b086dc330 2024-02-21T23:07:10.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.116+0000 7fd15083f740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x557b0863b510 con 0x557b087176b0 2024-02-21T23:07:10.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.116+0000 7fd146a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b086db020 0x557b086db400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:10.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.116+0000 7fd146a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b086db020 0x557b086db400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55248/0 (socket says 172.21.15.28:55248) 2024-02-21T23:07:10.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.116+0000 7fd146a00640 1 -- 172.21.15.28:0/2377376838 learned_addr learned my addr 172.21.15.28:0/2377376838 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:10.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.116+0000 7fd147400640 1 --2- 172.21.15.28:0/2377376838 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x557b087176b0 0x557b08715720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:10.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.116+0000 7fd13fe00640 1 --2- 172.21.15.28:0/2377376838 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557b086dc330 0x557b08717170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:10.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.116+0000 7fd146a00640 1 -- 172.21.15.28:0/2377376838 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x557b087176b0 msgr2=0x557b08715720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:10.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.116+0000 7fd146a00640 1 --2- 172.21.15.28:0/2377376838 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x557b087176b0 0x557b08715720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:10.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.116+0000 7fd146a00640 1 -- 172.21.15.28:0/2377376838 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557b086dc330 msgr2=0x557b08717170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:10.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.116+0000 7fd146a00640 1 --2- 172.21.15.28:0/2377376838 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557b086dc330 0x557b08717170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:10.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.116+0000 7fd146a00640 1 -- 172.21.15.28:0/2377376838 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557b085efd90 con 0x557b086db020 2024-02-21T23:07:10.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.117+0000 7fd146a00640 1 --2- 172.21.15.28:0/2377376838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b086db020 0x557b086db400 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fd138000c00 tx=0x7fd138030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d751167cbc19cb5 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:10.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.117+0000 7fd147400640 1 --2- 172.21.15.28:0/592452904 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd12406b4d0 0x7fd12406d990 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fd140007440 tx=0x7fd140007840 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:10.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.117+0000 7fd137400640 1 -- 172.21.15.28:0/2377376838 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd138038040 con 0x557b086db020 2024-02-21T23:07:10.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.117+0000 7fd137400640 1 -- 172.21.15.28:0/2377376838 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd138031910 con 0x557b086db020 2024-02-21T23:07:10.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.117+0000 7fd137400640 1 -- 172.21.15.28:0/2377376838 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd138031cc0 con 0x557b086db020 2024-02-21T23:07:10.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.118+0000 7fd15083f740 1 -- 172.21.15.28:0/2377376838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b086db020 msgr2=0x557b086db400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:10.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.118+0000 7fd15083f740 1 --2- 172.21.15.28:0/2377376838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b086db020 0x557b086db400 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fd138000c00 tx=0x7fd138030000 comp rx=0 tx=0).stop 2024-02-21T23:07:10.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.118+0000 7fd15083f740 1 -- 172.21.15.28:0/2377376838 shutdown_connections 2024-02-21T23:07:10.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.118+0000 7fd15083f740 1 --2- 172.21.15.28:0/2377376838 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x557b087176b0 0x557b08715720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:10.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.118+0000 7fd15083f740 1 --2- 172.21.15.28:0/2377376838 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557b086dc330 0x557b08717170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:10.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.118+0000 7fd15083f740 1 --2- 172.21.15.28:0/2377376838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b086db020 0x557b086db400 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:10.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.118+0000 7fd15083f740 1 -- 172.21.15.28:0/2377376838 >> 172.21.15.28:0/2377376838 conn(0x557b086d7970 msgr2=0x557b086fc380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:10.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.118+0000 7fd15083f740 1 -- 172.21.15.28:0/2377376838 shutdown_connections 2024-02-21T23:07:10.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.118+0000 7fd15083f740 1 -- 172.21.15.28:0/2377376838 wait complete. 2024-02-21T23:07:10.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.119+0000 7fd15083f740 1 -- start start 2024-02-21T23:07:10.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.119+0000 7fd15083f740 1 Processor -- start 2024-02-21T23:07:10.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.120+0000 7fd15083f740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x557b086dc330 0x557b0871e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:10.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.120+0000 7fd15083f740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557b087176b0 0x557b0871cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:10.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.120+0000 7fd15083f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b086f7890 0x557b08710a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:10.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.120+0000 7fd15083f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b085de040 con 0x557b086f7890 2024-02-21T23:07:10.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.120+0000 7fd15083f740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b086fca90 con 0x557b087176b0 2024-02-21T23:07:10.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.120+0000 7fd15083f740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x557b0871db00 con 0x557b086dc330 2024-02-21T23:07:10.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.120+0000 7fd15083f740 10 client.0 _ll_register_callbacks cb 0x557b085fb920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:07:10.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.120+0000 7fd147400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b086f7890 0x557b08710a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:10.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.120+0000 7fd147400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b086f7890 0x557b08710a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55252/0 (socket says 172.21.15.28:55252) 2024-02-21T23:07:10.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.120+0000 7fd147400640 1 -- 172.21.15.28:0/3411783401 learned_addr learned my addr 172.21.15.28:0/3411783401 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:10.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.120+0000 7fd146a00640 1 --2- 172.21.15.28:0/3411783401 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x557b086dc330 0x557b0871e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:10.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.121+0000 7fd13fe00640 1 --2- 172.21.15.28:0/3411783401 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557b087176b0 0x557b0871cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:10.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.121+0000 7fd147400640 1 -- 172.21.15.28:0/3411783401 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x557b086dc330 msgr2=0x557b0871e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:10.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.121+0000 7fd147400640 1 --2- 172.21.15.28:0/3411783401 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x557b086dc330 0x557b0871e4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:10.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.121+0000 7fd147400640 1 -- 172.21.15.28:0/3411783401 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557b087176b0 msgr2=0x557b0871cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:10.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.121+0000 7fd147400640 1 --2- 172.21.15.28:0/3411783401 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557b087176b0 0x557b0871cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:10.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.121+0000 7fd147400640 1 -- 172.21.15.28:0/3411783401 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557b085fdaa0 con 0x557b086f7890 2024-02-21T23:07:10.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.121+0000 7fd147400640 1 --2- 172.21.15.28:0/3411783401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b086f7890 0x557b08710a60 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7fd140017b20 tx=0x7fd140004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db7df0e8e4b24ffc server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:10.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.121+0000 7fd12be00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:07:10.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.121+0000 7fd12be00640 1 -- 172.21.15.28:0/3411783401 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd140027060 con 0x557b086f7890 2024-02-21T23:07:10.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.121+0000 7fd12be00640 1 -- 172.21.15.28:0/3411783401 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fd140015d20 con 0x557b086f7890 2024-02-21T23:07:10.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.121+0000 7fd15083f740 1 -- 172.21.15.28:0/3411783401 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd138031910 con 0x557b086f7890 2024-02-21T23:07:10.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.122+0000 7fd12be00640 1 -- 172.21.15.28:0/3411783401 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=93) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd14002d050 con 0x557b086f7890 2024-02-21T23:07:10.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.122+0000 7fd15083f740 10 client.7269 fetch_fsmap learned FSMap version 93 2024-02-21T23:07:10.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.122+0000 7fd15083f740 1 -- 172.21.15.28:0/3411783401 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=93,monmap=2+}) v3 -- 0x557b083542d0 con 0x557b086f7890 2024-02-21T23:07:10.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.122+0000 7fd12be00640 1 -- 172.21.15.28:0/3411783401 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd138031910 con 0x557b086f7890 2024-02-21T23:07:10.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.122+0000 7fd12be00640 1 -- 172.21.15.28:0/3411783401 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 93) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fd140030020 con 0x557b086f7890 2024-02-21T23:07:10.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.122+0000 7fd15083f740 10 client.7269 fetch_fsmap finished waiting for FSMap version 93 2024-02-21T23:07:10.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.122+0000 7fd15083f740 10 client.7269 resolve_mds: resolved name 'c' to daemon mds.c 2024-02-21T23:07:10.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.122+0000 7fd15083f740 20 client.7269 populate_metadata read hostname 'smithi028' 2024-02-21T23:07:10.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.122+0000 7fd15083f740 1 --2- 172.21.15.28:0/3411783401 >> [v2:172.21.15.28:6832/85689296,v1:172.21.15.28:6833/85689296] conn(0x557b0872a9c0 0x557b0872ada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:10.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.122+0000 7fd15083f740 4 client.7269 mds_command: new command op to 5555 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-21T23:07:10.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.122+0000 7fd15083f740 1 -- 172.21.15.28:0/3411783401 --> [v2:172.21.15.28:6832/85689296,v1:172.21.15.28:6833/85689296] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557b0864c9b0 con 0x557b0872a9c0 2024-02-21T23:07:10.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.122+0000 7fd146a00640 1 --2- 172.21.15.28:0/3411783401 >> [v2:172.21.15.28:6832/85689296,v1:172.21.15.28:6833/85689296] conn(0x557b0872a9c0 0x557b0872ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:10.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.123+0000 7fd146a00640 1 --2- 172.21.15.28:0/3411783401 >> [v2:172.21.15.28:6832/85689296,v1:172.21.15.28:6833/85689296] conn(0x557b0872a9c0 0x557b0872ada0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3eab47934323eeb8 server_cookie=40fbb0bd94335888 in_seq=0 out_seq=0 2024-02-21T23:07:10.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.123+0000 7fd12be00640 10 client.7269 ms_handle_connect on v2:172.21.15.28:6832/85689296 2024-02-21T23:07:10.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.124+0000 7fd12be00640 1 -- 172.21.15.28:0/3411783401 <== mds.0 v2:172.21.15.28:6832/85689296 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x557b0864c9b0 con 0x557b0872a9c0 2024-02-21T23:07:10.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.124+0000 7fd12be00640 10 client.7269 handle_command_reply: tid=0 2024-02-21T23:07:10.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.124+0000 7fd15083f740 1 client.7269 shutdown 2024-02-21T23:07:10.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.124+0000 7fd15083f740 1 -- 172.21.15.28:0/3411783401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b086f7890 msgr2=0x557b08710a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:10.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.124+0000 7fd15083f740 1 --2- 172.21.15.28:0/3411783401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b086f7890 0x557b08710a60 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7fd140017b20 tx=0x7fd140004ab0 comp rx=0 tx=0).stop 2024-02-21T23:07:10.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.124+0000 7fd15083f740 1 -- 172.21.15.28:0/3411783401 shutdown_connections 2024-02-21T23:07:10.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.124+0000 7fd15083f740 1 --2- 172.21.15.28:0/3411783401 >> [v2:172.21.15.28:6832/85689296,v1:172.21.15.28:6833/85689296] conn(0x557b0872a9c0 0x557b0872ada0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:10.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.124+0000 7fd15083f740 1 --2- 172.21.15.28:0/3411783401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b086f7890 0x557b08710a60 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:10.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.124+0000 7fd15083f740 1 --2- 172.21.15.28:0/3411783401 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557b087176b0 0x557b0871cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:10.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.124+0000 7fd15083f740 1 --2- 172.21.15.28:0/3411783401 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x557b086dc330 0x557b0871e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:10.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.124+0000 7fd12be00640 0 client.7269 ms_handle_reset on v2:172.21.15.28:6832/85689296 2024-02-21T23:07:10.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.124+0000 7fd15083f740 1 -- 172.21.15.28:0/3411783401 >> 172.21.15.28:0/3411783401 conn(0x557b086db020 msgr2=0x557b08715fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:10.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.125+0000 7fd15083f740 1 -- 172.21.15.28:0/3411783401 shutdown_connections 2024-02-21T23:07:10.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.125+0000 7fd15083f740 1 -- 172.21.15.28:0/3411783401 wait complete. 2024-02-21T23:07:10.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.125+0000 7fd15083f740 20 client.7269 trim_cache size 0 max 16384 2024-02-21T23:07:10.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.143+0000 7fd15083f740 1 Processor -- start 2024-02-21T23:07:10.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.143+0000 7fd15083f740 1 -- start start 2024-02-21T23:07:10.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.144+0000 7fd15083f740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x557b086dc330 0x557b086f7c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:10.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.144+0000 7fd15083f740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557b087176b0 0x557b086f81c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:10.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.144+0000 7fd15083f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b08716710 0x557b08716b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:10.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.144+0000 7fd15083f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1481f6b80 con 0x557b08716710 2024-02-21T23:07:10.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.144+0000 7fd15083f740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1481eea50 con 0x557b087176b0 2024-02-21T23:07:10.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.144+0000 7fd15083f740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1481f6580 con 0x557b086dc330 2024-02-21T23:07:10.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.144+0000 7fd147400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b08716710 0x557b08716b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:10.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.144+0000 7fd147400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b08716710 0x557b08716b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55254/0 (socket says 172.21.15.28:55254) 2024-02-21T23:07:10.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.144+0000 7fd147400640 1 -- 172.21.15.28:0/893821417 learned_addr learned my addr 172.21.15.28:0/893821417 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:10.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.145+0000 7fd13fe00640 1 --2- 172.21.15.28:0/893821417 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557b087176b0 0x557b086f81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:10.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.145+0000 7fd146a00640 1 --2- 172.21.15.28:0/893821417 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x557b086dc330 0x557b086f7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:10.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.145+0000 7fd147400640 1 -- 172.21.15.28:0/893821417 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x557b086dc330 msgr2=0x557b086f7c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:10.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.145+0000 7fd147400640 1 --2- 172.21.15.28:0/893821417 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x557b086dc330 0x557b086f7c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:10.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.145+0000 7fd147400640 1 -- 172.21.15.28:0/893821417 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557b087176b0 msgr2=0x557b086f81c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:10.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.145+0000 7fd147400640 1 --2- 172.21.15.28:0/893821417 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557b087176b0 0x557b086f81c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:10.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.145+0000 7fd147400640 1 -- 172.21.15.28:0/893821417 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557b0841e6b0 con 0x557b08716710 2024-02-21T23:07:10.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.145+0000 7fd146a00640 1 --2- 172.21.15.28:0/893821417 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x557b086dc330 0x557b086f7c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:07:10.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.145+0000 7fd13fe00640 1 --2- 172.21.15.28:0/893821417 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557b087176b0 0x557b086f81c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:07:10.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.145+0000 7fd147400640 1 --2- 172.21.15.28:0/893821417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b08716710 0x557b08716b90 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7fd140007de0 tx=0x7fd140020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=173233b464f8a91f server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:10.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.145+0000 7fd136a00640 1 -- 172.21.15.28:0/893821417 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd140036070 con 0x557b08716710 2024-02-21T23:07:10.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.145+0000 7fd136a00640 1 -- 172.21.15.28:0/893821417 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1400090b0 con 0x557b08716710 2024-02-21T23:07:10.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.145+0000 7fd136a00640 1 -- 172.21.15.28:0/893821417 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd140012040 con 0x557b08716710 2024-02-21T23:07:10.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.146+0000 7fd15083f740 1 -- 172.21.15.28:0/893821417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b08716710 msgr2=0x557b08716b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:10.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.146+0000 7fd15083f740 1 --2- 172.21.15.28:0/893821417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b08716710 0x557b08716b90 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7fd140007de0 tx=0x7fd140020000 comp rx=0 tx=0).stop 2024-02-21T23:07:10.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.146+0000 7fd15083f740 1 -- 172.21.15.28:0/893821417 shutdown_connections 2024-02-21T23:07:10.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.146+0000 7fd15083f740 1 --2- 172.21.15.28:0/893821417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b08716710 0x557b08716b90 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:10.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.146+0000 7fd15083f740 1 --2- 172.21.15.28:0/893821417 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557b087176b0 0x557b086f81c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:10.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.146+0000 7fd15083f740 1 --2- 172.21.15.28:0/893821417 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x557b086dc330 0x557b086f7c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:10.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.146+0000 7fd15083f740 1 -- 172.21.15.28:0/893821417 >> 172.21.15.28:0/893821417 conn(0x557b086db020 msgr2=0x557b08715fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:10.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.146+0000 7fd15083f740 1 -- 172.21.15.28:0/893821417 shutdown_connections 2024-02-21T23:07:10.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.146+0000 7fd15083f740 1 -- 172.21.15.28:0/893821417 wait complete. 2024-02-21T23:07:10.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.147+0000 7fd15083f740 1 -- start start 2024-02-21T23:07:10.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.147+0000 7fd15083f740 1 Processor -- start 2024-02-21T23:07:10.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.148+0000 7fd15083f740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x557b086dc330 0x557b086fbcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:10.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.148+0000 7fd15083f740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557b08716710 0x557b08715d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:10.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.149+0000 7fd15083f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b087176b0 0x557b08783b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:10.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.149+0000 7fd15083f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b0871d840 con 0x557b087176b0 2024-02-21T23:07:10.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.149+0000 7fd15083f740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b0872a840 con 0x557b08716710 2024-02-21T23:07:10.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.149+0000 7fd15083f740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x557b0871d460 con 0x557b086dc330 2024-02-21T23:07:10.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.149+0000 7fd15083f740 10 client.0 _ll_register_callbacks cb 0x7fd1400177f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:07:10.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.149+0000 7fd147400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b087176b0 0x557b08783b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:10.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.149+0000 7fd147400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b087176b0 0x557b08783b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55270/0 (socket says 172.21.15.28:55270) 2024-02-21T23:07:10.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.149+0000 7fd147400640 1 -- 172.21.15.28:0/2269046275 learned_addr learned my addr 172.21.15.28:0/2269046275 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:10.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.149+0000 7fd13fe00640 1 --2- 172.21.15.28:0/2269046275 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557b08716710 0x557b08715d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:10.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.149+0000 7fd146a00640 1 --2- 172.21.15.28:0/2269046275 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x557b086dc330 0x557b086fbcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:10.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.149+0000 7fd147400640 1 -- 172.21.15.28:0/2269046275 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x557b086dc330 msgr2=0x557b086fbcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:10.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.149+0000 7fd147400640 1 --2- 172.21.15.28:0/2269046275 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x557b086dc330 0x557b086fbcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:10.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.149+0000 7fd147400640 1 -- 172.21.15.28:0/2269046275 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557b08716710 msgr2=0x557b08715d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:10.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.149+0000 7fd147400640 1 --2- 172.21.15.28:0/2269046275 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557b08716710 0x557b08715d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:10.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.149+0000 7fd147400640 1 -- 172.21.15.28:0/2269046275 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557b087868a0 con 0x557b087176b0 2024-02-21T23:07:10.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.150+0000 7fd147400640 1 --2- 172.21.15.28:0/2269046275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b087176b0 0x557b08783b60 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7fd140034040 tx=0x7fd140009950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e692c0b0bf09c32a server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:10.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.150+0000 7fd12be00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:07:10.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.150+0000 7fd12be00640 1 -- 172.21.15.28:0/2269046275 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd140036070 con 0x557b087176b0 2024-02-21T23:07:10.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.150+0000 7fd12be00640 1 -- 172.21.15.28:0/2269046275 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fd14003dca0 con 0x557b087176b0 2024-02-21T23:07:10.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.150+0000 7fd15083f740 1 -- 172.21.15.28:0/2269046275 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd1400090b0 con 0x557b087176b0 2024-02-21T23:07:10.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.150+0000 7fd12be00640 1 -- 172.21.15.28:0/2269046275 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=93) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd140044050 con 0x557b087176b0 2024-02-21T23:07:10.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.150+0000 7fd15083f740 10 client.7275 fetch_fsmap learned FSMap version 93 2024-02-21T23:07:10.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.150+0000 7fd15083f740 1 -- 172.21.15.28:0/2269046275 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=93,monmap=2+}) v3 -- 0x557b087162a0 con 0x557b087176b0 2024-02-21T23:07:10.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.151+0000 7fd12be00640 1 -- 172.21.15.28:0/2269046275 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1400090b0 con 0x557b087176b0 2024-02-21T23:07:10.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.151+0000 7fd12be00640 1 -- 172.21.15.28:0/2269046275 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 93) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fd140047020 con 0x557b087176b0 2024-02-21T23:07:10.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.151+0000 7fd15083f740 10 client.7275 fetch_fsmap finished waiting for FSMap version 93 2024-02-21T23:07:10.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.151+0000 7fd15083f740 10 client.7275 resolve_mds: resolved name 'c' to daemon mds.c 2024-02-21T23:07:10.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.151+0000 7fd15083f740 20 client.7275 populate_metadata read hostname 'smithi028' 2024-02-21T23:07:10.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.151+0000 7fd15083f740 1 --2- 172.21.15.28:0/2269046275 >> [v2:172.21.15.28:6832/85689296,v1:172.21.15.28:6833/85689296] conn(0x557b086f8a50 0x557b086f8e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:10.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.151+0000 7fd15083f740 4 client.7275 mds_command: new command op to 5555 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-21T23:07:10.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.151+0000 7fd15083f740 1 -- 172.21.15.28:0/2269046275 --> [v2:172.21.15.28:6832/85689296,v1:172.21.15.28:6833/85689296] -- command(tid 0: {"prefix": "status"}) v1 -- 0x557b08646d00 con 0x557b086f8a50 2024-02-21T23:07:10.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.151+0000 7fd146a00640 1 --2- 172.21.15.28:0/2269046275 >> [v2:172.21.15.28:6832/85689296,v1:172.21.15.28:6833/85689296] conn(0x557b086f8a50 0x557b086f8e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:10.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.152+0000 7fd146a00640 1 --2- 172.21.15.28:0/2269046275 >> [v2:172.21.15.28:6832/85689296,v1:172.21.15.28:6833/85689296] conn(0x557b086f8a50 0x557b086f8e30 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9f556536c1b0e49 server_cookie=c06f192279fcffb4 in_seq=0 out_seq=0 2024-02-21T23:07:10.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.152+0000 7fd12be00640 10 client.7275 ms_handle_connect on v2:172.21.15.28:6832/85689296 2024-02-21T23:07:10.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.152+0000 7fd12be00640 1 -- 172.21.15.28:0/2269046275 <== mds.0 v2:172.21.15.28:6832/85689296 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x557b08646d00 con 0x557b086f8a50 2024-02-21T23:07:10.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.152+0000 7fd12be00640 10 client.7275 handle_command_reply: tid=0 2024-02-21T23:07:10.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.152+0000 7fd15083f740 1 client.7275 shutdown 2024-02-21T23:07:10.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.152+0000 7fd15083f740 1 -- 172.21.15.28:0/2269046275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b087176b0 msgr2=0x557b08783b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:10.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.152+0000 7fd15083f740 1 --2- 172.21.15.28:0/2269046275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b087176b0 0x557b08783b60 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7fd140034040 tx=0x7fd140009950 comp rx=0 tx=0).stop 2024-02-21T23:07:10.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.152+0000 7fd15083f740 1 -- 172.21.15.28:0/2269046275 shutdown_connections 2024-02-21T23:07:10.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.152+0000 7fd15083f740 1 --2- 172.21.15.28:0/2269046275 >> [v2:172.21.15.28:6832/85689296,v1:172.21.15.28:6833/85689296] conn(0x557b086f8a50 0x557b086f8e30 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:10.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.152+0000 7fd15083f740 1 --2- 172.21.15.28:0/2269046275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b087176b0 0x557b08783b60 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:10.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.152+0000 7fd15083f740 1 --2- 172.21.15.28:0/2269046275 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557b08716710 0x557b08715d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:10.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.152+0000 7fd15083f740 1 --2- 172.21.15.28:0/2269046275 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x557b086dc330 0x557b086fbcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:10.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.152+0000 7fd12be00640 0 client.7275 ms_handle_reset on v2:172.21.15.28:6832/85689296 2024-02-21T23:07:10.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.152+0000 7fd15083f740 1 -- 172.21.15.28:0/2269046275 >> 172.21.15.28:0/2269046275 conn(0x557b086db020 msgr2=0x557b086f8120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:10.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.152+0000 7fd15083f740 1 -- 172.21.15.28:0/2269046275 shutdown_connections 2024-02-21T23:07:10.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.152+0000 7fd15083f740 1 -- 172.21.15.28:0/2269046275 wait complete. 2024-02-21T23:07:10.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.153+0000 7fd15083f740 20 client.7275 trim_cache size 0 max 16384 2024-02-21T23:07:10.144 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-21T23:07:10.145 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac", 2024-02-21T23:07:10.145 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-02-21T23:07:10.145 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 5555, 2024-02-21T23:07:10.145 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-02-21T23:07:10.145 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-02-21T23:07:10.145 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-02-21T23:07:10.145 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 4.3077202809999999, 2024-02-21T23:07:10.145 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 93, 2024-02-21T23:07:10.145 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 99, 2024-02-21T23:07:10.145 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 99, 2024-02-21T23:07:10.145 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 20.889618536 2024-02-21T23:07:10.145 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-21T23:07:10.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.153+0000 7fd147e00640 1 -- 172.21.15.28:0/592452904 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd12406b4d0 msgr2=0x7fd12406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:10.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.153+0000 7fd147e00640 1 --2- 172.21.15.28:0/592452904 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd12406b4d0 0x7fd12406d990 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fd140007440 tx=0x7fd140007840 comp rx=0 tx=0).stop 2024-02-21T23:07:10.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.153+0000 7fd147e00640 1 -- 172.21.15.28:0/592452904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1480c85c0 msgr2=0x7fd14814f190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:10.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.153+0000 7fd147e00640 1 --2- 172.21.15.28:0/592452904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1480c85c0 0x7fd14814f190 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7fd13000c500 tx=0x7fd13000c9c0 comp rx=0 tx=0).stop 2024-02-21T23:07:10.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.153+0000 7fd147e00640 1 -- 172.21.15.28:0/592452904 shutdown_connections 2024-02-21T23:07:10.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.153+0000 7fd147e00640 1 --2- 172.21.15.28:0/592452904 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd12406b4d0 0x7fd12406d990 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:10.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.153+0000 7fd147e00640 1 --2- 172.21.15.28:0/592452904 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd1481524c0 0x7fd148147e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:10.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.153+0000 7fd147e00640 1 --2- 172.21.15.28:0/592452904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1480c85c0 0x7fd14814f190 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:10.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.153+0000 7fd147e00640 1 --2- 172.21.15.28:0/592452904 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1480c5040 0x7fd1480c0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:10.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.153+0000 7fd147e00640 1 -- 172.21.15.28:0/592452904 >> 172.21.15.28:0/592452904 conn(0x7fd1480ac2b0 msgr2=0x7fd1480c9590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:10.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.153+0000 7fd147e00640 1 -- 172.21.15.28:0/592452904 shutdown_connections 2024-02-21T23:07:10.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.154+0000 7fd147e00640 1 -- 172.21.15.28:0/592452904 wait complete. 2024-02-21T23:07:10.160 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 7, "epoch": 93, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 7, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-02-21T23:06:54.201589+0000", "damaged": [], "data_pools": [ 16 ], "enabled": true, "epoch": 93, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5555": { "addr": "172.21.15.28:6833/85689296", "addrs": { "addrvec": [ { "addr": "172.21.15.28:6832", "nonce": 85689296, "type": "v2" }, { "addr": "172.21.15.28:6833", "nonce": 85689296, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5555, "incarnation": 90, "join_fscid": 7, "name": "c", "rank": 0, "state": "up:active", "state_seq": 8 } }, "last_failure": 0, "last_failure_osd_epoch": 99, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 15, "modified": "2024-02-21T23:07:07.869167+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5555 } } } ], "standbys": [ { "addr": "172.21.15.81:6835/1194613007", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6834", "nonce": 1194613007, "type": "v2" }, { "addr": "172.21.15.81:6835", "nonce": 1194613007, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 83, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5567, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.81:6837/2315735157", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6836", "nonce": 2315735157, "type": "v2" }, { "addr": "172.21.15.81:6837", "nonce": 2315735157, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 83, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5570, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.28:6835/2558379712", "addrs": { "addrvec": [ { "addr": "172.21.15.28:6834", "nonce": 2558379712, "type": "v2" }, { "addr": "172.21.15.28:6835", "nonce": 2558379712, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 91, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7251, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-02-21T23:07:10.161 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': 7, 'name': 'c', 'state': 'up:active'}, {'join_fscid': -1, 'name': 'd'}] 2024-02-21T23:07:10.161 INFO:tasks.cephfs.test_failover:current = [{'gid': 7251, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.28:6835/2558379712', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 2558379712}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 2558379712}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 91}, {'gid': 5567, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.81:6835/1194613007', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6834', 'nonce': 1194613007}, {'type': 'v1', 'addr': '172.21.15.81:6835', 'nonce': 1194613007}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 83}, {'gid': 5555, 'name': 'c', 'rank': 0, 'incarnation': 90, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.28:6833/85689296', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 85689296}, {'type': 'v1', 'addr': '172.21.15.28:6833', 'nonce': 85689296}]}, 'join_fscid': 7, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 5570, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.81:6837/2315735157', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 2315735157}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 2315735157}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 83}] 2024-02-21T23:07:10.162 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-02-21T23:07:10.162 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-02-21T23:07:10.162 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-21T23:07:10.274 DEBUG:teuthology.orchestra.run.smithi081:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-21T23:07:10.399 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-02-21T23:07:10.399 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-21T23:07:10.399 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-21T23:07:10.454 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054608,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-02-21T23:07:10.454 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-02-21T23:07:10.455 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-21T23:07:10.510 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053833,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498812k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249404k,nr_inodes=812351,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498812k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-02-21T23:07:10.511 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:07:10.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.734+0000 7ffa69c00640 1 Processor -- start 2024-02-21T23:07:10.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.734+0000 7ffa69c00640 1 -- start start 2024-02-21T23:07:10.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.736+0000 7ffa69c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7ffa64155c20 0x7ffa64160870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:10.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.736+0000 7ffa69c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa6406ba00 con 0x7ffa641524a0 2024-02-21T23:07:10.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.736+0000 7ffa69c00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa6406f2d0 con 0x7ffa640c45d0 2024-02-21T23:07:10.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.736+0000 7ffa69c00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa64154ff0 con 0x7ffa64155c20 2024-02-21T23:07:10.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.736+0000 7ffa63e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ffa641524a0 0x7ffa640c7620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:40468/0 (socket says 172.21.15.28:40468) 2024-02-21T23:07:10.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.736+0000 7ffa63e00640 1 -- 172.21.15.28:0/2525696274 learned_addr learned my addr 172.21.15.28:0/2525696274 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:07:10.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.737+0000 7ffa69200640 1 --2- 172.21.15.28:0/2525696274 >> v2:172.21.15.81:3300/0 conn(0x7ffa64155c20 0x7ffa64160870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:10.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.737+0000 7ffa63400640 1 -- 172.21.15.28:0/2525696274 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3527445869 0 0) 0x7ffa6406ba00 con 0x7ffa641524a0 2024-02-21T23:07:10.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.737+0000 7ffa63400640 1 -- 172.21.15.28:0/2525696274 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa48003520 con 0x7ffa641524a0 2024-02-21T23:07:10.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.737+0000 7ffa63400640 1 -- 172.21.15.28:0/2525696274 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3406643631 0 0) 0x7ffa48003520 con 0x7ffa641524a0 2024-02-21T23:07:10.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.737+0000 7ffa63400640 1 -- 172.21.15.28:0/2525696274 >> v2:172.21.15.81:3300/0 conn(0x7ffa64155c20 msgr2=0x7ffa64160870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:10.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.737+0000 7ffa63400640 1 --2- 172.21.15.28:0/2525696274 >> v2:172.21.15.81:3300/0 conn(0x7ffa64155c20 0x7ffa64160870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:10.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.737+0000 7ffa63400640 1 -- 172.21.15.28:0/2525696274 >> v1:172.21.15.81:6789/0 conn(0x7ffa640c45d0 legacy=0x7ffa64155510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:10.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.737+0000 7ffa63400640 1 -- 172.21.15.28:0/2525696274 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa6406aee0 con 0x7ffa641524a0 2024-02-21T23:07:10.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.737+0000 7ffa63400640 1 -- 172.21.15.28:0/2525696274 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7ffa54001670 con 0x7ffa641524a0 2024-02-21T23:07:10.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.737+0000 7ffa63400640 1 -- 172.21.15.28:0/2525696274 >> v1:172.21.15.28:6789/0 conn(0x7ffa641524a0 legacy=0x7ffa640c7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:10.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.737+0000 7ffa63400640 1 --2- 172.21.15.28:0/2525696274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa480039b0 0x7ffa48003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:10.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.737+0000 7ffa63400640 1 --2- 172.21.15.28:0/2525696274 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffa48004800 0x7ffa48004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:10.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.737+0000 7ffa63400640 1 --2- 172.21.15.28:0/2525696274 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffa480051c0 0x7ffa4800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:10.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.737+0000 7ffa63400640 1 -- 172.21.15.28:0/2525696274 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa64154ff0 con 0x7ffa480039b0 2024-02-21T23:07:10.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.737+0000 7ffa63400640 1 -- 172.21.15.28:0/2525696274 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa4800c0f0 con 0x7ffa48004800 2024-02-21T23:07:10.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.737+0000 7ffa63400640 1 -- 172.21.15.28:0/2525696274 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa4800c2f0 con 0x7ffa480051c0 2024-02-21T23:07:10.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.738+0000 7ffa63e00640 1 --2- 172.21.15.28:0/2525696274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa480039b0 0x7ffa48003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:10.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.738+0000 7ffa5be00640 1 --2- 172.21.15.28:0/2525696274 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffa48004800 0x7ffa48004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:10.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.738+0000 7ffa63e00640 1 -- 172.21.15.28:0/2525696274 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffa480051c0 msgr2=0x7ffa4800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:10.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.738+0000 7ffa69200640 1 --2- 172.21.15.28:0/2525696274 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffa480051c0 0x7ffa4800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:10.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.738+0000 7ffa63e00640 1 --2- 172.21.15.28:0/2525696274 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffa480051c0 0x7ffa4800b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:10.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.738+0000 7ffa63e00640 1 -- 172.21.15.28:0/2525696274 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffa48004800 msgr2=0x7ffa48004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:10.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.738+0000 7ffa63e00640 1 --2- 172.21.15.28:0/2525696274 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffa48004800 0x7ffa48004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:10.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.738+0000 7ffa63e00640 1 -- 172.21.15.28:0/2525696274 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa48003520 con 0x7ffa480039b0 2024-02-21T23:07:10.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.738+0000 7ffa5be00640 1 --2- 172.21.15.28:0/2525696274 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffa48004800 0x7ffa48004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:07:10.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.738+0000 7ffa69200640 1 -- 172.21.15.28:0/2525696274 reap_dead start 2024-02-21T23:07:10.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.739+0000 7ffa63e00640 1 --2- 172.21.15.28:0/2525696274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa480039b0 0x7ffa48003db0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7ffa5400c210 tx=0x7ffa48005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc053f7629011bdf server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:10.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.739+0000 7ffa63400640 1 -- 172.21.15.28:0/2525696274 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa54007d30 con 0x7ffa480039b0 2024-02-21T23:07:10.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.739+0000 7ffa63400640 1 -- 172.21.15.28:0/2525696274 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa54030070 con 0x7ffa480039b0 2024-02-21T23:07:10.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.740+0000 7ffa63400640 1 -- 172.21.15.28:0/2525696274 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa54005d70 con 0x7ffa480039b0 2024-02-21T23:07:10.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.740+0000 7ffa69c00640 1 -- 172.21.15.28:0/2525696274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa480039b0 msgr2=0x7ffa48003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:10.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.740+0000 7ffa69c00640 1 --2- 172.21.15.28:0/2525696274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa480039b0 0x7ffa48003db0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7ffa5400c210 tx=0x7ffa48005830 comp rx=0 tx=0).stop 2024-02-21T23:07:10.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.740+0000 7ffa69c00640 1 -- 172.21.15.28:0/2525696274 shutdown_connections 2024-02-21T23:07:10.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.740+0000 7ffa69c00640 1 --2- 172.21.15.28:0/2525696274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa480039b0 0x7ffa48003db0 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:10.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.740+0000 7ffa69c00640 1 -- 172.21.15.28:0/2525696274 >> 172.21.15.28:0/2525696274 conn(0x7ffa64067050 msgr2=0x7ffa640a84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:10.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.740+0000 7ffa69c00640 1 -- 172.21.15.28:0/2525696274 shutdown_connections 2024-02-21T23:07:10.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.740+0000 7ffa69c00640 1 -- 172.21.15.28:0/2525696274 wait complete. 2024-02-21T23:07:10.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.742+0000 7ffa69c00640 1 Processor -- start 2024-02-21T23:07:10.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.742+0000 7ffa69c00640 1 -- start start 2024-02-21T23:07:10.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.743+0000 7ffa69c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffa480039b0 0x7ffa6414f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:10.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.743+0000 7ffa69c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffa641ee940 0x7ffa64157980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:10.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.743+0000 7ffa69c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa640c7210 0x7ffa640aa4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:10.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.743+0000 7ffa69c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa640c4fa0 con 0x7ffa640c7210 2024-02-21T23:07:10.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.743+0000 7ffa69c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa6406f5b0 con 0x7ffa641ee940 2024-02-21T23:07:10.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.743+0000 7ffa69c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa6406b460 con 0x7ffa480039b0 2024-02-21T23:07:10.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.743+0000 7ffa69200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa640c7210 0x7ffa640aa4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:10.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.743+0000 7ffa69200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa640c7210 0x7ffa640aa4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55282/0 (socket says 172.21.15.28:55282) 2024-02-21T23:07:10.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.743+0000 7ffa69200640 1 -- 172.21.15.28:0/3373495767 learned_addr learned my addr 172.21.15.28:0/3373495767 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:10.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.743+0000 7ffa63e00640 1 --2- 172.21.15.28:0/3373495767 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffa480039b0 0x7ffa6414f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:10.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.743+0000 7ffa5be00640 1 --2- 172.21.15.28:0/3373495767 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffa641ee940 0x7ffa64157980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:10.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.743+0000 7ffa69200640 1 -- 172.21.15.28:0/3373495767 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffa480039b0 msgr2=0x7ffa6414f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:10.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.743+0000 7ffa69200640 1 --2- 172.21.15.28:0/3373495767 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffa480039b0 0x7ffa6414f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:10.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.743+0000 7ffa69200640 1 -- 172.21.15.28:0/3373495767 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffa641ee940 msgr2=0x7ffa64157980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:10.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.743+0000 7ffa69200640 1 --2- 172.21.15.28:0/3373495767 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffa641ee940 0x7ffa64157980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:10.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.743+0000 7ffa69200640 1 -- 172.21.15.28:0/3373495767 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa641df770 con 0x7ffa640c7210 2024-02-21T23:07:10.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.744+0000 7ffa69200640 1 --2- 172.21.15.28:0/3373495767 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa640c7210 0x7ffa640aa4e0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7ffa640c43d0 tx=0x7ffa5c009dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:10.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.744+0000 7ffa60c00640 1 -- 172.21.15.28:0/3373495767 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa5c019070 con 0x7ffa640c7210 2024-02-21T23:07:10.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.744+0000 7ffa60c00640 1 -- 172.21.15.28:0/3373495767 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa5c007d70 con 0x7ffa640c7210 2024-02-21T23:07:10.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.744+0000 7ffa60c00640 1 -- 172.21.15.28:0/3373495767 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa5c014640 con 0x7ffa640c7210 2024-02-21T23:07:10.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.744+0000 7ffa69c00640 1 -- 172.21.15.28:0/3373495767 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa640ac440 con 0x7ffa640c7210 2024-02-21T23:07:10.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.744+0000 7ffa69c00640 1 -- 172.21.15.28:0/3373495767 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa641e2220 con 0x7ffa640c7210 2024-02-21T23:07:10.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.745+0000 7ffa69c00640 1 -- 172.21.15.28:0/3373495767 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa641e2010 con 0x7ffa640c7210 2024-02-21T23:07:10.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.746+0000 7ffa60c00640 1 -- 172.21.15.28:0/3373495767 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffa5c01c070 con 0x7ffa640c7210 2024-02-21T23:07:10.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.748+0000 7ffa60c00640 1 --2- 172.21.15.28:0/3373495767 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ffa4006b4d0 0x7ffa4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:10.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.748+0000 7ffa60c00640 1 -- 172.21.15.28:0/3373495767 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6187+0+0 (secure 0 0 0) 0x7ffa5c089f50 con 0x7ffa640c7210 2024-02-21T23:07:10.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.749+0000 7ffa63e00640 1 --2- 172.21.15.28:0/3373495767 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ffa4006b4d0 0x7ffa4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:10.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.750+0000 7ffa63e00640 1 --2- 172.21.15.28:0/3373495767 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ffa4006b4d0 0x7ffa4006d990 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7ffa54034000 tx=0x7ffa54039040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:10.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:10.752+0000 7ffa60c00640 1 -- 172.21.15.28:0/3373495767 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ffa5c058be0 con 0x7ffa640c7210 2024-02-21T23:07:10.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.002+0000 7ffa69c00640 1 -- 172.21.15.28:0/3373495767 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffa6404f290 con 0x7ffa640c7210 2024-02-21T23:07:10.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.003+0000 7ffa60c00640 1 -- 172.21.15.28:0/3373495767 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+4934 (secure 0 0 0) 0x7ffa5c0584e0 con 0x7ffa640c7210 2024-02-21T23:07:10.992 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:07:10.992 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":93,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5567,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1194613007","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1194613007},{"type":"v1","addr":"172.21.15.81:6835","nonce":1194613007}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":5570,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/2315735157","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":2315735157},{"type":"v1","addr":"172.21.15.81:6837","nonce":2315735157}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":7251,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2558379712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2558379712},{"type":"v1","addr":"172.21.15.28:6835","nonce":2558379712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":91}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:06:54.201589+0000","modified":"2024-02-21T23:07:07.869167+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":99,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5555},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5555":{"gid":5555,"name":"c","rank":0,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.28:6833/85689296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":85689296},{"type":"v1","addr":"172.21.15.28:6833","nonce":85689296}]},"join_fscid":7,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-02-21T23:07:10.992 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 93 2024-02-21T23:07:10.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.006+0000 7ffa69c00640 1 -- 172.21.15.28:0/3373495767 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ffa4006b4d0 msgr2=0x7ffa4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:10.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.006+0000 7ffa69c00640 1 --2- 172.21.15.28:0/3373495767 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ffa4006b4d0 0x7ffa4006d990 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7ffa54034000 tx=0x7ffa54039040 comp rx=0 tx=0).stop 2024-02-21T23:07:10.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.006+0000 7ffa69c00640 1 -- 172.21.15.28:0/3373495767 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa640c7210 msgr2=0x7ffa640aa4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:10.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.006+0000 7ffa69c00640 1 --2- 172.21.15.28:0/3373495767 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa640c7210 0x7ffa640aa4e0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7ffa640c43d0 tx=0x7ffa5c009dc0 comp rx=0 tx=0).stop 2024-02-21T23:07:10.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.006+0000 7ffa69c00640 1 -- 172.21.15.28:0/3373495767 shutdown_connections 2024-02-21T23:07:10.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.006+0000 7ffa69c00640 1 --2- 172.21.15.28:0/3373495767 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ffa4006b4d0 0x7ffa4006d990 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:10.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.006+0000 7ffa69c00640 1 --2- 172.21.15.28:0/3373495767 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa640c7210 0x7ffa640aa4e0 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:10.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.006+0000 7ffa69c00640 1 --2- 172.21.15.28:0/3373495767 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffa641ee940 0x7ffa64157980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:10.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.006+0000 7ffa69c00640 1 --2- 172.21.15.28:0/3373495767 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffa480039b0 0x7ffa6414f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:10.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.006+0000 7ffa69c00640 1 -- 172.21.15.28:0/3373495767 >> 172.21.15.28:0/3373495767 conn(0x7ffa64067050 msgr2=0x7ffa64150290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:10.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.006+0000 7ffa69c00640 1 -- 172.21.15.28:0/3373495767 shutdown_connections 2024-02-21T23:07:10.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.006+0000 7ffa69c00640 1 -- 172.21.15.28:0/3373495767 wait complete. 2024-02-21T23:07:11.013 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:07:11.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.236+0000 7fd772e00640 1 Processor -- start 2024-02-21T23:07:11.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.237+0000 7fd772e00640 1 -- start start 2024-02-21T23:07:11.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.238+0000 7fd772e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd76c0aff90 0x7fd76c0c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:11.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.238+0000 7fd772e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fd76c0c5720 0x7fd76c15f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:11.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.238+0000 7fd772e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd76c0c74c0 con 0x7fd76c06dbf0 2024-02-21T23:07:11.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.238+0000 7fd772e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd76c0c7270 con 0x7fd76c0aff90 2024-02-21T23:07:11.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.238+0000 7fd772e00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd76c158630 con 0x7fd76c0c5720 2024-02-21T23:07:11.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.238+0000 7fd771a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd76c0aff90 0x7fd76c0c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:11.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.239+0000 7fd771a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd76c0aff90 0x7fd76c0c51e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55286/0 (socket says 172.21.15.28:55286) 2024-02-21T23:07:11.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.239+0000 7fd771a00640 1 -- 172.21.15.28:0/2193350455 learned_addr learned my addr 172.21.15.28:0/2193350455 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:11.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.239+0000 7fd772400640 1 --2- 172.21.15.28:0/2193350455 >> v2:172.21.15.81:3300/0 conn(0x7fd76c0c5720 0x7fd76c15f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:11.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.239+0000 7fd771a00640 1 -- 172.21.15.28:0/2193350455 >> v2:172.21.15.81:3300/0 conn(0x7fd76c0c5720 msgr2=0x7fd76c15f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:11.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.239+0000 7fd771a00640 1 --2- 172.21.15.28:0/2193350455 >> v2:172.21.15.81:3300/0 conn(0x7fd76c0c5720 0x7fd76c15f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:11.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.239+0000 7fd771a00640 1 -- 172.21.15.28:0/2193350455 >> v1:172.21.15.28:6789/0 conn(0x7fd76c06dbf0 legacy=0x7fd76c158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:11.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.239+0000 7fd771a00640 1 -- 172.21.15.28:0/2193350455 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd76c0c77a0 con 0x7fd76c0aff90 2024-02-21T23:07:11.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.239+0000 7fd772400640 1 --2- 172.21.15.28:0/2193350455 >> v2:172.21.15.81:3300/0 conn(0x7fd76c0c5720 0x7fd76c15f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:07:11.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.240+0000 7fd771a00640 1 --2- 172.21.15.28:0/2193350455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd76c0aff90 0x7fd76c0c51e0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7fd76c0c73f0 tx=0x7fd76002b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d25f1515774d59c server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:11.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.241+0000 7fd76be00640 1 -- 172.21.15.28:0/2193350455 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd760034070 con 0x7fd76c0aff90 2024-02-21T23:07:11.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.241+0000 7fd76be00640 1 -- 172.21.15.28:0/2193350455 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd760030850 con 0x7fd76c0aff90 2024-02-21T23:07:11.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.241+0000 7fd76be00640 1 -- 172.21.15.28:0/2193350455 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd760030c20 con 0x7fd76c0aff90 2024-02-21T23:07:11.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.241+0000 7fd772e00640 1 -- 172.21.15.28:0/2193350455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd76c0aff90 msgr2=0x7fd76c0c51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:11.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.241+0000 7fd772e00640 1 --2- 172.21.15.28:0/2193350455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd76c0aff90 0x7fd76c0c51e0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7fd76c0c73f0 tx=0x7fd76002b990 comp rx=0 tx=0).stop 2024-02-21T23:07:11.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.242+0000 7fd772e00640 1 -- 172.21.15.28:0/2193350455 shutdown_connections 2024-02-21T23:07:11.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.242+0000 7fd772e00640 1 --2- 172.21.15.28:0/2193350455 >> v2:172.21.15.81:3300/0 conn(0x7fd76c0c5720 0x7fd76c15f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:11.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.242+0000 7fd772e00640 1 --2- 172.21.15.28:0/2193350455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd76c0aff90 0x7fd76c0c51e0 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:11.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.242+0000 7fd772e00640 1 -- 172.21.15.28:0/2193350455 >> 172.21.15.28:0/2193350455 conn(0x7fd76c0ac370 msgr2=0x7fd76c0ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:11.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.242+0000 7fd772e00640 1 -- 172.21.15.28:0/2193350455 shutdown_connections 2024-02-21T23:07:11.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.242+0000 7fd772e00640 1 -- 172.21.15.28:0/2193350455 wait complete. 2024-02-21T23:07:11.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.244+0000 7fd772e00640 1 Processor -- start 2024-02-21T23:07:11.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.244+0000 7fd772e00640 1 -- start start 2024-02-21T23:07:11.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.246+0000 7fd772e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd76c06dbf0 0x7fd76c1edec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:11.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.246+0000 7fd772e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd76c0aff90 0x7fd76c151ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:11.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.246+0000 7fd772e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd76c0c5720 0x7fd76c152020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:11.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.246+0000 7fd772e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd76c0c8760 con 0x7fd76c0aff90 2024-02-21T23:07:11.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.246+0000 7fd772e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd76c0be0c0 con 0x7fd76c06dbf0 2024-02-21T23:07:11.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.246+0000 7fd772e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd76c0c88e0 con 0x7fd76c0c5720 2024-02-21T23:07:11.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.246+0000 7fd771000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd76c0aff90 0x7fd76c151ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:11.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.246+0000 7fd771000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd76c0aff90 0x7fd76c151ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55300/0 (socket says 172.21.15.28:55300) 2024-02-21T23:07:11.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.246+0000 7fd771000640 1 -- 172.21.15.28:0/1462102456 learned_addr learned my addr 172.21.15.28:0/1462102456 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:11.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.246+0000 7fd771a00640 1 --2- 172.21.15.28:0/1462102456 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd76c06dbf0 0x7fd76c1edec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:11.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.247+0000 7fd772400640 1 --2- 172.21.15.28:0/1462102456 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd76c0c5720 0x7fd76c152020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:11.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.247+0000 7fd771000640 1 -- 172.21.15.28:0/1462102456 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd76c0c5720 msgr2=0x7fd76c152020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:11.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.247+0000 7fd771000640 1 --2- 172.21.15.28:0/1462102456 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd76c0c5720 0x7fd76c152020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:11.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.247+0000 7fd771000640 1 -- 172.21.15.28:0/1462102456 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd76c06dbf0 msgr2=0x7fd76c1edec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:11.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.247+0000 7fd771000640 1 --2- 172.21.15.28:0/1462102456 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd76c06dbf0 0x7fd76c1edec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:11.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.247+0000 7fd771000640 1 -- 172.21.15.28:0/1462102456 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd76c1ee940 con 0x7fd76c0aff90 2024-02-21T23:07:11.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.247+0000 7fd771a00640 1 --2- 172.21.15.28:0/1462102456 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd76c06dbf0 0x7fd76c1edec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:07:11.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.247+0000 7fd771000640 1 --2- 172.21.15.28:0/1462102456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd76c0aff90 0x7fd76c151ae0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7fd75800c440 tx=0x7fd758001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:11.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.247+0000 7fd769600640 1 -- 172.21.15.28:0/1462102456 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7580052d0 con 0x7fd76c0aff90 2024-02-21T23:07:11.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.247+0000 7fd769600640 1 -- 172.21.15.28:0/1462102456 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd758005470 con 0x7fd76c0aff90 2024-02-21T23:07:11.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.247+0000 7fd769600640 1 -- 172.21.15.28:0/1462102456 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd758007260 con 0x7fd76c0aff90 2024-02-21T23:07:11.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.247+0000 7fd772e00640 1 -- 172.21.15.28:0/1462102456 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd76c0c3b30 con 0x7fd76c0aff90 2024-02-21T23:07:11.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.248+0000 7fd772e00640 1 -- 172.21.15.28:0/1462102456 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd76c0c40f0 con 0x7fd76c0aff90 2024-02-21T23:07:11.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.249+0000 7fd772e00640 1 -- 172.21.15.28:0/1462102456 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd76c04f290 con 0x7fd76c0aff90 2024-02-21T23:07:11.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.250+0000 7fd769600640 1 -- 172.21.15.28:0/1462102456 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd758007400 con 0x7fd76c0aff90 2024-02-21T23:07:11.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.251+0000 7fd769600640 1 --2- 172.21.15.28:0/1462102456 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd74c06b4d0 0x7fd74c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:11.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.251+0000 7fd769600640 1 -- 172.21.15.28:0/1462102456 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fd758089860 con 0x7fd76c0aff90 2024-02-21T23:07:11.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.251+0000 7fd771a00640 1 --2- 172.21.15.28:0/1462102456 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd74c06b4d0 0x7fd74c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:11.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.253+0000 7fd771a00640 1 --2- 172.21.15.28:0/1462102456 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd74c06b4d0 0x7fd74c06d990 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fd760036000 tx=0x7fd76003b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:11.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.255+0000 7fd769600640 1 -- 172.21.15.28:0/1462102456 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd758058590 con 0x7fd76c0aff90 2024-02-21T23:07:11.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.456+0000 7fd772e00640 1 -- 172.21.15.28:0/1462102456 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd76c156d20 con 0x7fd76c0aff90 2024-02-21T23:07:11.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.459+0000 7fd769600640 1 -- 172.21.15.28:0/1462102456 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) v1 ==== 74+0+16942 (secure 0 0 0) 0x7fd758057e90 con 0x7fd76c0aff90 2024-02-21T23:07:11.448 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:07:11.448 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":99,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:07:05.837927+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:06:52.151181+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-02-21T23:06:53.221776+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6835/2477712025":"2024-02-22T23:07:05.837870+0000","172.21.15.28:6834/2477712025":"2024-02-22T23:07:05.837870+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:07:11.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.461+0000 7fd772e00640 1 -- 172.21.15.28:0/1462102456 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd74c06b4d0 msgr2=0x7fd74c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:11.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.461+0000 7fd772e00640 1 --2- 172.21.15.28:0/1462102456 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd74c06b4d0 0x7fd74c06d990 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fd760036000 tx=0x7fd76003b040 comp rx=0 tx=0).stop 2024-02-21T23:07:11.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.461+0000 7fd772e00640 1 -- 172.21.15.28:0/1462102456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd76c0aff90 msgr2=0x7fd76c151ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:11.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.462+0000 7fd772e00640 1 --2- 172.21.15.28:0/1462102456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd76c0aff90 0x7fd76c151ae0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7fd75800c440 tx=0x7fd758001e70 comp rx=0 tx=0).stop 2024-02-21T23:07:11.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.462+0000 7fd772e00640 1 -- 172.21.15.28:0/1462102456 shutdown_connections 2024-02-21T23:07:11.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.462+0000 7fd772e00640 1 --2- 172.21.15.28:0/1462102456 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd74c06b4d0 0x7fd74c06d990 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:11.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.462+0000 7fd772e00640 1 --2- 172.21.15.28:0/1462102456 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd76c0c5720 0x7fd76c152020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:11.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.462+0000 7fd772e00640 1 --2- 172.21.15.28:0/1462102456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd76c0aff90 0x7fd76c151ae0 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:11.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.462+0000 7fd772e00640 1 --2- 172.21.15.28:0/1462102456 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd76c06dbf0 0x7fd76c1edec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:11.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.462+0000 7fd772e00640 1 -- 172.21.15.28:0/1462102456 >> 172.21.15.28:0/1462102456 conn(0x7fd76c0ac370 msgr2=0x7fd76c0c8ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:11.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.462+0000 7fd772e00640 1 -- 172.21.15.28:0/1462102456 shutdown_connections 2024-02-21T23:07:11.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.462+0000 7fd772e00640 1 -- 172.21.15.28:0/1462102456 wait complete. 2024-02-21T23:07:11.472 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-21T23:02:30.728329+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-21T23:06:52.151181+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '97', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_data', 'create_time': '2024-02-21T23:06:53.221776+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '98', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-21T23:07:11.472 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-21T23:07:11.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.687+0000 7fc08f000640 1 Processor -- start 2024-02-21T23:07:11.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.688+0000 7fc08f000640 1 -- start start 2024-02-21T23:07:11.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.690+0000 7fc08f000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fc0880c5cc0 0x7fc0880c6c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:11.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.690+0000 7fc08f000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc0880c3030 0x7fc0880c3460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:11.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.690+0000 7fc08f000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc088159580 con 0x7fc0880c7590 2024-02-21T23:07:11.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.690+0000 7fc08f000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0880c6230 con 0x7fc0880c5cc0 2024-02-21T23:07:11.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.690+0000 7fc08f000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0881587c0 con 0x7fc0880c3030 2024-02-21T23:07:11.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.690+0000 7fc087e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fc0880c7590 0x7fc0881524e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:40496/0 (socket says 172.21.15.28:40496) 2024-02-21T23:07:11.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.690+0000 7fc087e00640 1 -- 172.21.15.28:0/3008273442 learned_addr learned my addr 172.21.15.28:0/3008273442 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:07:11.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.691+0000 7fc087400640 1 --2- 172.21.15.28:0/3008273442 >> v2:172.21.15.81:3300/0 conn(0x7fc0880c5cc0 0x7fc0880c6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:11.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.691+0000 7fc08d000640 1 --2- 172.21.15.28:0/3008273442 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc0880c3030 0x7fc0880c3460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:11.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.691+0000 7fc086a00640 1 -- 172.21.15.28:0/3008273442 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4252316171 0 0) 0x7fc088159580 con 0x7fc0880c7590 2024-02-21T23:07:11.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.691+0000 7fc086a00640 1 -- 172.21.15.28:0/3008273442 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc06c003520 con 0x7fc0880c7590 2024-02-21T23:07:11.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.691+0000 7fc086a00640 1 -- 172.21.15.28:0/3008273442 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3298943776 0 0) 0x7fc06c003520 con 0x7fc0880c7590 2024-02-21T23:07:11.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.691+0000 7fc086a00640 1 -- 172.21.15.28:0/3008273442 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc0880c3030 msgr2=0x7fc0880c3460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:11.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.691+0000 7fc086a00640 1 --2- 172.21.15.28:0/3008273442 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc0880c3030 0x7fc0880c3460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:11.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.691+0000 7fc086a00640 1 -- 172.21.15.28:0/3008273442 >> v2:172.21.15.81:3300/0 conn(0x7fc0880c5cc0 msgr2=0x7fc0880c6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:11.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.691+0000 7fc086a00640 1 --2- 172.21.15.28:0/3008273442 >> v2:172.21.15.81:3300/0 conn(0x7fc0880c5cc0 0x7fc0880c6c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:11.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.691+0000 7fc086a00640 1 -- 172.21.15.28:0/3008273442 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc088158aa0 con 0x7fc0880c7590 2024-02-21T23:07:11.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.691+0000 7fc086a00640 1 -- 172.21.15.28:0/3008273442 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fc078001670 con 0x7fc0880c7590 2024-02-21T23:07:11.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.691+0000 7fc086a00640 1 -- 172.21.15.28:0/3008273442 >> v1:172.21.15.28:6789/0 conn(0x7fc0880c7590 legacy=0x7fc0881524e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:11.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.691+0000 7fc086a00640 1 --2- 172.21.15.28:0/3008273442 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc0880c3030 0x7fc06c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:11.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.691+0000 7fc086a00640 1 --2- 172.21.15.28:0/3008273442 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc06c003f30 0x7fc06c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:11.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.691+0000 7fc086a00640 1 --2- 172.21.15.28:0/3008273442 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc06c004d40 0x7fc06c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:11.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.692+0000 7fc086a00640 1 -- 172.21.15.28:0/3008273442 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0880c6230 con 0x7fc06c004d40 2024-02-21T23:07:11.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.692+0000 7fc086a00640 1 -- 172.21.15.28:0/3008273442 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0881587c0 con 0x7fc06c003f30 2024-02-21T23:07:11.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.692+0000 7fc086a00640 1 -- 172.21.15.28:0/3008273442 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc06c00ba10 con 0x7fc0880c3030 2024-02-21T23:07:11.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.692+0000 7fc08d000640 1 --2- 172.21.15.28:0/3008273442 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc06c004d40 0x7fc06c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:11.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.692+0000 7fc087e00640 1 --2- 172.21.15.28:0/3008273442 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc0880c3030 0x7fc06c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:11.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.692+0000 7fc087400640 1 --2- 172.21.15.28:0/3008273442 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc06c003f30 0x7fc06c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:11.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.692+0000 7fc08d000640 1 -- 172.21.15.28:0/3008273442 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc0880c3030 msgr2=0x7fc06c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:11.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.692+0000 7fc08d000640 1 --2- 172.21.15.28:0/3008273442 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc0880c3030 0x7fc06c0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:11.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.692+0000 7fc08d000640 1 -- 172.21.15.28:0/3008273442 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc06c003f30 msgr2=0x7fc06c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:11.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.692+0000 7fc08d000640 1 --2- 172.21.15.28:0/3008273442 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc06c003f30 0x7fc06c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:11.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.692+0000 7fc08d000640 1 -- 172.21.15.28:0/3008273442 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc06c003520 con 0x7fc06c004d40 2024-02-21T23:07:11.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.692+0000 7fc087400640 1 --2- 172.21.15.28:0/3008273442 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc06c003f30 0x7fc06c004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:07:11.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.694+0000 7fc08d000640 1 --2- 172.21.15.28:0/3008273442 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc06c004d40 0x7fc06c0051c0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7fc08000e170 tx=0x7fc06c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f39b7c0afb8597a server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:11.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.694+0000 7fc086a00640 1 -- 172.21.15.28:0/3008273442 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc080039070 con 0x7fc06c004d40 2024-02-21T23:07:11.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.694+0000 7fc086a00640 1 -- 172.21.15.28:0/3008273442 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc08002cbf0 con 0x7fc06c004d40 2024-02-21T23:07:11.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.694+0000 7fc086a00640 1 -- 172.21.15.28:0/3008273442 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc080009c10 con 0x7fc06c004d40 2024-02-21T23:07:11.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.694+0000 7fc08f000640 1 -- 172.21.15.28:0/3008273442 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc06c004d40 msgr2=0x7fc06c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:11.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.694+0000 7fc08f000640 1 --2- 172.21.15.28:0/3008273442 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc06c004d40 0x7fc06c0051c0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7fc08000e170 tx=0x7fc06c005830 comp rx=0 tx=0).stop 2024-02-21T23:07:11.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.694+0000 7fc08d000640 1 -- 172.21.15.28:0/3008273442 reap_dead start 2024-02-21T23:07:11.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.694+0000 7fc08f000640 1 -- 172.21.15.28:0/3008273442 shutdown_connections 2024-02-21T23:07:11.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.694+0000 7fc08f000640 1 -- 172.21.15.28:0/3008273442 >> 172.21.15.28:0/3008273442 conn(0x7fc08808afe0 msgr2=0x7fc0880725a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:11.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.695+0000 7fc08f000640 1 -- 172.21.15.28:0/3008273442 shutdown_connections 2024-02-21T23:07:11.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.695+0000 7fc08f000640 1 -- 172.21.15.28:0/3008273442 wait complete. 2024-02-21T23:07:11.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.697+0000 7fc08f000640 1 Processor -- start 2024-02-21T23:07:11.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.697+0000 7fc08f000640 1 -- start start 2024-02-21T23:07:11.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.698+0000 7fc08f000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc06c004d40 0x7fc0880c88c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:11.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.698+0000 7fc08f000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc088155640 0x7fc088157e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:11.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.698+0000 7fc08f000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc08814ee10 0x7fc0881ee180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:11.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.698+0000 7fc08f000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0880c6720 con 0x7fc088155640 2024-02-21T23:07:11.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.698+0000 7fc08f000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc088159060 con 0x7fc08814ee10 2024-02-21T23:07:11.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.698+0000 7fc08f000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0880c68a0 con 0x7fc06c004d40 2024-02-21T23:07:11.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.698+0000 7fc087400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc088155640 0x7fc088157e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:11.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.698+0000 7fc087400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc088155640 0x7fc088157e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55320/0 (socket says 172.21.15.28:55320) 2024-02-21T23:07:11.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.698+0000 7fc087400640 1 -- 172.21.15.28:0/3993331037 learned_addr learned my addr 172.21.15.28:0/3993331037 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:11.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.698+0000 7fc087e00640 1 --2- 172.21.15.28:0/3993331037 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc06c004d40 0x7fc0880c88c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:11.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.698+0000 7fc08d000640 1 --2- 172.21.15.28:0/3993331037 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc08814ee10 0x7fc0881ee180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:11.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.699+0000 7fc087400640 1 -- 172.21.15.28:0/3993331037 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc06c004d40 msgr2=0x7fc0880c88c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:11.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.699+0000 7fc087400640 1 --2- 172.21.15.28:0/3993331037 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc06c004d40 0x7fc0880c88c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:11.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.699+0000 7fc087400640 1 -- 172.21.15.28:0/3993331037 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc08814ee10 msgr2=0x7fc0881ee180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:11.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.699+0000 7fc087400640 1 --2- 172.21.15.28:0/3993331037 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc08814ee10 0x7fc0881ee180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:11.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.699+0000 7fc087400640 1 -- 172.21.15.28:0/3993331037 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0881ed9a0 con 0x7fc088155640 2024-02-21T23:07:11.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.699+0000 7fc087400640 1 --2- 172.21.15.28:0/3993331037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc088155640 0x7fc088157e10 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7fc074010a00 tx=0x7fc074010ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:11.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.699+0000 7fc07fe00640 1 -- 172.21.15.28:0/3993331037 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc07400f040 con 0x7fc088155640 2024-02-21T23:07:11.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.699+0000 7fc07fe00640 1 -- 172.21.15.28:0/3993331037 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc07400ebc0 con 0x7fc088155640 2024-02-21T23:07:11.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.699+0000 7fc07fe00640 1 -- 172.21.15.28:0/3993331037 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc07401c400 con 0x7fc088155640 2024-02-21T23:07:11.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.699+0000 7fc08f000640 1 -- 172.21.15.28:0/3993331037 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc088156cf0 con 0x7fc088155640 2024-02-21T23:07:11.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.700+0000 7fc08f000640 1 -- 172.21.15.28:0/3993331037 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0881572b0 con 0x7fc088155640 2024-02-21T23:07:11.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.701+0000 7fc08f000640 1 -- 172.21.15.28:0/3993331037 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc088067f30 con 0x7fc088155640 2024-02-21T23:07:11.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.702+0000 7fc07fe00640 1 -- 172.21.15.28:0/3993331037 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc07401c5a0 con 0x7fc088155640 2024-02-21T23:07:11.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.703+0000 7fc07fe00640 1 --2- 172.21.15.28:0/3993331037 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc06806b500 0x7fc06806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:11.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.703+0000 7fc07fe00640 1 -- 172.21.15.28:0/3993331037 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fc07408aa70 con 0x7fc088155640 2024-02-21T23:07:11.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.703+0000 7fc087e00640 1 --2- 172.21.15.28:0/3993331037 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc06806b500 0x7fc06806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:11.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.704+0000 7fc087e00640 1 --2- 172.21.15.28:0/3993331037 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc06806b500 0x7fc06806d9c0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fc078000fb0 tx=0x7fc078009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:11.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.707+0000 7fc07fe00640 1 -- 172.21.15.28:0/3993331037 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc074018030 con 0x7fc088155640 2024-02-21T23:07:11.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.909+0000 7fc08f000640 1 -- 172.21.15.28:0/3993331037 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc088157480 con 0x7fc088155640 2024-02-21T23:07:11.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.910+0000 7fc07fe00640 1 -- 172.21.15.28:0/3993331037 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v99) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc0740590b0 con 0x7fc088155640 2024-02-21T23:07:11.898 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-02-21T23:07:11.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.913+0000 7fc08f000640 1 -- 172.21.15.28:0/3993331037 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc06806b500 msgr2=0x7fc06806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:11.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.913+0000 7fc08f000640 1 --2- 172.21.15.28:0/3993331037 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc06806b500 0x7fc06806d9c0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fc078000fb0 tx=0x7fc078009040 comp rx=0 tx=0).stop 2024-02-21T23:07:11.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.913+0000 7fc08f000640 1 -- 172.21.15.28:0/3993331037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc088155640 msgr2=0x7fc088157e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:11.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.913+0000 7fc08f000640 1 --2- 172.21.15.28:0/3993331037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc088155640 0x7fc088157e10 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7fc074010a00 tx=0x7fc074010ec0 comp rx=0 tx=0).stop 2024-02-21T23:07:11.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.913+0000 7fc08f000640 1 -- 172.21.15.28:0/3993331037 shutdown_connections 2024-02-21T23:07:11.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.913+0000 7fc08f000640 1 --2- 172.21.15.28:0/3993331037 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc06806b500 0x7fc06806d9c0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:11.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.913+0000 7fc08f000640 1 --2- 172.21.15.28:0/3993331037 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc08814ee10 0x7fc0881ee180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:11.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.913+0000 7fc08f000640 1 --2- 172.21.15.28:0/3993331037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc088155640 0x7fc088157e10 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:11.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.913+0000 7fc08f000640 1 --2- 172.21.15.28:0/3993331037 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc06c004d40 0x7fc0880c88c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:11.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.913+0000 7fc08f000640 1 -- 172.21.15.28:0/3993331037 >> 172.21.15.28:0/3993331037 conn(0x7fc08808afe0 msgr2=0x7fc088147200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:11.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.913+0000 7fc08f000640 1 -- 172.21.15.28:0/3993331037 shutdown_connections 2024-02-21T23:07:11.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:11.913+0000 7fc08f000640 1 -- 172.21.15.28:0/3993331037 wait complete. 2024-02-21T23:07:11.921 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-21T23:07:12.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.151+0000 7fbcf4800640 1 Processor -- start 2024-02-21T23:07:12.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.151+0000 7fbcf4800640 1 -- start start 2024-02-21T23:07:12.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.153+0000 7fbcf4800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbcec06b0e0 0x7fbcec0c5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:12.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.153+0000 7fbcf4800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fbcec150700 0x7fbcec156500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:12.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.153+0000 7fbcf4800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcec0c61e0 con 0x7fbcec0c5d80 2024-02-21T23:07:12.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.153+0000 7fbcf4800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcec153bc0 con 0x7fbcec06b0e0 2024-02-21T23:07:12.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.153+0000 7fbcf4800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcec152fb0 con 0x7fbcec150700 2024-02-21T23:07:12.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.153+0000 7fbcf1e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbcec06b0e0 0x7fbcec0c5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:12.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.153+0000 7fbcf1e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbcec06b0e0 0x7fbcec0c5840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55326/0 (socket says 172.21.15.28:55326) 2024-02-21T23:07:12.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.153+0000 7fbcf1e00640 1 -- 172.21.15.28:0/1673281301 learned_addr learned my addr 172.21.15.28:0/1673281301 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:12.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.153+0000 7fbcf2800640 1 --2- 172.21.15.28:0/1673281301 >> v2:172.21.15.81:3300/0 conn(0x7fbcec150700 0x7fbcec156500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:12.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.154+0000 7fbcf1e00640 1 -- 172.21.15.28:0/1673281301 >> v2:172.21.15.81:3300/0 conn(0x7fbcec150700 msgr2=0x7fbcec156500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:12.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.154+0000 7fbcf1e00640 1 --2- 172.21.15.28:0/1673281301 >> v2:172.21.15.81:3300/0 conn(0x7fbcec150700 0x7fbcec156500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:12.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.154+0000 7fbcf1e00640 1 -- 172.21.15.28:0/1673281301 >> v1:172.21.15.81:6789/0 conn(0x7fbcec0c5d80 legacy=0x7fbcec14fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:12.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.154+0000 7fbcf1e00640 1 -- 172.21.15.28:0/1673281301 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcec153290 con 0x7fbcec06b0e0 2024-02-21T23:07:12.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.154+0000 7fbcf2800640 1 --2- 172.21.15.28:0/1673281301 >> v2:172.21.15.81:3300/0 conn(0x7fbcec150700 0x7fbcec156500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:07:12.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.155+0000 7fbcf1e00640 1 --2- 172.21.15.28:0/1673281301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcec06b0e0 0x7fbcec0c5840 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7fbcec0a64b0 tx=0x7fbce002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c994457b24b848 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:12.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.156+0000 7fbcf0a00640 1 -- 172.21.15.28:0/1673281301 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbce002e070 con 0x7fbcec06b0e0 2024-02-21T23:07:12.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.156+0000 7fbcf0a00640 1 -- 172.21.15.28:0/1673281301 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbce0037070 con 0x7fbcec06b0e0 2024-02-21T23:07:12.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.156+0000 7fbcf0a00640 1 -- 172.21.15.28:0/1673281301 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbce0031a00 con 0x7fbcec06b0e0 2024-02-21T23:07:12.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.157+0000 7fbcf4800640 1 -- 172.21.15.28:0/1673281301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcec06b0e0 msgr2=0x7fbcec0c5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:12.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.157+0000 7fbcf4800640 1 --2- 172.21.15.28:0/1673281301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcec06b0e0 0x7fbcec0c5840 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7fbcec0a64b0 tx=0x7fbce002b880 comp rx=0 tx=0).stop 2024-02-21T23:07:12.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.157+0000 7fbcf4800640 1 -- 172.21.15.28:0/1673281301 shutdown_connections 2024-02-21T23:07:12.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.157+0000 7fbcf4800640 1 --2- 172.21.15.28:0/1673281301 >> v2:172.21.15.81:3300/0 conn(0x7fbcec150700 0x7fbcec156500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:12.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.157+0000 7fbcf4800640 1 --2- 172.21.15.28:0/1673281301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcec06b0e0 0x7fbcec0c5840 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:12.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.157+0000 7fbcf4800640 1 -- 172.21.15.28:0/1673281301 >> 172.21.15.28:0/1673281301 conn(0x7fbcec0a85a0 msgr2=0x7fbcec0a8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:12.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.157+0000 7fbcf4800640 1 -- 172.21.15.28:0/1673281301 shutdown_connections 2024-02-21T23:07:12.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.157+0000 7fbcf4800640 1 -- 172.21.15.28:0/1673281301 wait complete. 2024-02-21T23:07:12.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.159+0000 7fbcf4800640 1 Processor -- start 2024-02-21T23:07:12.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.159+0000 7fbcf4800640 1 -- start start 2024-02-21T23:07:12.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.159+0000 7fbcf4800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbcec06b0e0 0x7fbcec1543c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:12.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.160+0000 7fbcf4800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbcec0c5d80 0x7fbcec1eae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:12.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.160+0000 7fbcf4800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcec150700 0x7fbcec1eb360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:12.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.160+0000 7fbcf4800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcec06b4c0 con 0x7fbcec150700 2024-02-21T23:07:12.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.160+0000 7fbcf4800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcec153850 con 0x7fbcec0c5d80 2024-02-21T23:07:12.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.160+0000 7fbcf4800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcec06b640 con 0x7fbcec06b0e0 2024-02-21T23:07:12.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.160+0000 7fbcf2800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcec150700 0x7fbcec1eb360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:12.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.160+0000 7fbcf2800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcec150700 0x7fbcec1eb360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55342/0 (socket says 172.21.15.28:55342) 2024-02-21T23:07:12.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.160+0000 7fbcf2800640 1 -- 172.21.15.28:0/3907428130 learned_addr learned my addr 172.21.15.28:0/3907428130 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:12.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.160+0000 7fbcf1400640 1 --2- 172.21.15.28:0/3907428130 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbcec0c5d80 0x7fbcec1eae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:12.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.160+0000 7fbcf1e00640 1 --2- 172.21.15.28:0/3907428130 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbcec06b0e0 0x7fbcec1543c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:12.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.160+0000 7fbcf2800640 1 -- 172.21.15.28:0/3907428130 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbcec06b0e0 msgr2=0x7fbcec1543c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:12.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.160+0000 7fbcf2800640 1 --2- 172.21.15.28:0/3907428130 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbcec06b0e0 0x7fbcec1543c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:12.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.161+0000 7fbcf2800640 1 -- 172.21.15.28:0/3907428130 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbcec0c5d80 msgr2=0x7fbcec1eae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:12.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.161+0000 7fbcf2800640 1 --2- 172.21.15.28:0/3907428130 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbcec0c5d80 0x7fbcec1eae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:12.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.161+0000 7fbcf2800640 1 -- 172.21.15.28:0/3907428130 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcec150c10 con 0x7fbcec150700 2024-02-21T23:07:12.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.161+0000 7fbcf1400640 1 --2- 172.21.15.28:0/3907428130 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbcec0c5d80 0x7fbcec1eae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:07:12.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.161+0000 7fbcf2800640 1 --2- 172.21.15.28:0/3907428130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcec150700 0x7fbcec1eb360 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7fbce8007e70 tx=0x7fbce800d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:12.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.161+0000 7fbce6000640 1 -- 172.21.15.28:0/3907428130 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbce801a070 con 0x7fbcec150700 2024-02-21T23:07:12.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.161+0000 7fbce6000640 1 -- 172.21.15.28:0/3907428130 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbce8003bb0 con 0x7fbcec150700 2024-02-21T23:07:12.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.161+0000 7fbce6000640 1 -- 172.21.15.28:0/3907428130 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbce8004030 con 0x7fbcec150700 2024-02-21T23:07:12.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.161+0000 7fbcf4800640 1 -- 172.21.15.28:0/3907428130 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcec155dc0 con 0x7fbcec150700 2024-02-21T23:07:12.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.161+0000 7fbcf4800640 1 -- 172.21.15.28:0/3907428130 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcec150eb0 con 0x7fbcec150700 2024-02-21T23:07:12.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.162+0000 7fbcf4800640 1 -- 172.21.15.28:0/3907428130 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcec067f30 con 0x7fbcec150700 2024-02-21T23:07:12.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.164+0000 7fbce6000640 1 -- 172.21.15.28:0/3907428130 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbce801d070 con 0x7fbcec150700 2024-02-21T23:07:12.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.165+0000 7fbce6000640 1 --2- 172.21.15.28:0/3907428130 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbcd006b500 0x7fbcd006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:12.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.165+0000 7fbce6000640 1 -- 172.21.15.28:0/3907428130 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fbce808a620 con 0x7fbcec150700 2024-02-21T23:07:12.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.166+0000 7fbcf1e00640 1 --2- 172.21.15.28:0/3907428130 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbcd006b500 0x7fbcd006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:12.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.167+0000 7fbcf1e00640 1 --2- 172.21.15.28:0/3907428130 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbcd006b500 0x7fbcd006d9c0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fbce0036000 tx=0x7fbce003d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:12.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.175+0000 7fbce6000640 1 -- 172.21.15.28:0/3907428130 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fbce8010070 con 0x7fbcec150700 2024-02-21T23:07:12.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.373+0000 7fbcf4800640 1 -- 172.21.15.28:0/3907428130 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fbcec14f9d0 con 0x7fbcec150700 2024-02-21T23:07:12.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.374+0000 7fbce6000640 1 -- 172.21.15.28:0/3907428130 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v99) v1 ==== 104+0+11 (secure 0 0 0) 0x7fbce80592b0 con 0x7fbcec150700 2024-02-21T23:07:12.363 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-02-21T23:07:12.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.377+0000 7fbcf4800640 1 -- 172.21.15.28:0/3907428130 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbcd006b500 msgr2=0x7fbcd006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:12.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.377+0000 7fbcf4800640 1 --2- 172.21.15.28:0/3907428130 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbcd006b500 0x7fbcd006d9c0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fbce0036000 tx=0x7fbce003d040 comp rx=0 tx=0).stop 2024-02-21T23:07:12.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.377+0000 7fbcf4800640 1 -- 172.21.15.28:0/3907428130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcec150700 msgr2=0x7fbcec1eb360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:12.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.377+0000 7fbcf4800640 1 --2- 172.21.15.28:0/3907428130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcec150700 0x7fbcec1eb360 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7fbce8007e70 tx=0x7fbce800d4a0 comp rx=0 tx=0).stop 2024-02-21T23:07:12.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.377+0000 7fbcf4800640 1 -- 172.21.15.28:0/3907428130 shutdown_connections 2024-02-21T23:07:12.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.377+0000 7fbcf4800640 1 --2- 172.21.15.28:0/3907428130 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbcd006b500 0x7fbcd006d9c0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:12.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.377+0000 7fbcf4800640 1 --2- 172.21.15.28:0/3907428130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcec150700 0x7fbcec1eb360 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:12.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.377+0000 7fbcf4800640 1 --2- 172.21.15.28:0/3907428130 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbcec0c5d80 0x7fbcec1eae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:12.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.377+0000 7fbcf4800640 1 --2- 172.21.15.28:0/3907428130 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbcec06b0e0 0x7fbcec1543c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:12.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.377+0000 7fbcf4800640 1 -- 172.21.15.28:0/3907428130 >> 172.21.15.28:0/3907428130 conn(0x7fbcec0a85a0 msgr2=0x7fbcec14acf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:12.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.377+0000 7fbcf4800640 1 -- 172.21.15.28:0/3907428130 shutdown_connections 2024-02-21T23:07:12.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.378+0000 7fbcf4800640 1 -- 172.21.15.28:0/3907428130 wait complete. 2024-02-21T23:07:12.383 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-21T23:07:12.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.644+0000 7f56ede00640 1 Processor -- start 2024-02-21T23:07:12.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.645+0000 7f56ede00640 1 -- start start 2024-02-21T23:07:12.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.646+0000 7f56ede00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f56e8152690 0x7f56e80c50b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:12.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.646+0000 7f56ede00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f56e80c55f0 0x7f56e80c3090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:12.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.646+0000 7f56ede00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56e806ca30 con 0x7f56e80c35d0 2024-02-21T23:07:12.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.646+0000 7f56ede00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56e8158470 con 0x7f56e80c55f0 2024-02-21T23:07:12.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.646+0000 7f56ede00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56e80c3a60 con 0x7f56e8152690 2024-02-21T23:07:12.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.647+0000 7f56e7e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f56e80c55f0 0x7f56e80c3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:12.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.647+0000 7f56e7e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f56e80c55f0 0x7f56e80c3090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55346/0 (socket says 172.21.15.28:55346) 2024-02-21T23:07:12.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.647+0000 7f56e7e00640 1 -- 172.21.15.28:0/2935107520 learned_addr learned my addr 172.21.15.28:0/2935107520 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:12.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.647+0000 7f56eca00640 1 --2- 172.21.15.28:0/2935107520 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f56e8152690 0x7f56e80c50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:12.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.647+0000 7f56e7e00640 1 -- 172.21.15.28:0/2935107520 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f56e8152690 msgr2=0x7f56e80c50b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:12.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.647+0000 7f56e7e00640 1 --2- 172.21.15.28:0/2935107520 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f56e8152690 0x7f56e80c50b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:12.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.647+0000 7f56e7e00640 1 -- 172.21.15.28:0/2935107520 >> v1:172.21.15.81:6789/0 conn(0x7f56e80c35d0 legacy=0x7f56e8155310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:12.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.647+0000 7f56e7e00640 1 -- 172.21.15.28:0/2935107520 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56e806ccd0 con 0x7f56e80c55f0 2024-02-21T23:07:12.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.647+0000 7f56eca00640 1 --2- 172.21.15.28:0/2935107520 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f56e8152690 0x7f56e80c50b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:07:12.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.648+0000 7f56e7e00640 1 --2- 172.21.15.28:0/2935107520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56e80c55f0 0x7f56e80c3090 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f56d4009870 tx=0x7f56d402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17506eafe5e46871 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:12.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.649+0000 7f56e7400640 1 -- 172.21.15.28:0/2935107520 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56d4035020 con 0x7f56e80c55f0 2024-02-21T23:07:12.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.649+0000 7f56e7400640 1 -- 172.21.15.28:0/2935107520 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56d4031640 con 0x7f56e80c55f0 2024-02-21T23:07:12.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.649+0000 7f56e7400640 1 -- 172.21.15.28:0/2935107520 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56d402d040 con 0x7f56e80c55f0 2024-02-21T23:07:12.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.649+0000 7f56ede00640 1 -- 172.21.15.28:0/2935107520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56e80c55f0 msgr2=0x7f56e80c3090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:12.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.649+0000 7f56ede00640 1 --2- 172.21.15.28:0/2935107520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56e80c55f0 0x7f56e80c3090 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f56d4009870 tx=0x7f56d402b8f0 comp rx=0 tx=0).stop 2024-02-21T23:07:12.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.650+0000 7f56ede00640 1 -- 172.21.15.28:0/2935107520 shutdown_connections 2024-02-21T23:07:12.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.650+0000 7f56ede00640 1 --2- 172.21.15.28:0/2935107520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56e80c55f0 0x7f56e80c3090 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:12.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.650+0000 7f56ede00640 1 --2- 172.21.15.28:0/2935107520 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f56e8152690 0x7f56e80c50b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:12.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.650+0000 7f56ede00640 1 -- 172.21.15.28:0/2935107520 >> 172.21.15.28:0/2935107520 conn(0x7f56e80c8620 msgr2=0x7f56e80c8a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:12.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.650+0000 7f56ede00640 1 -- 172.21.15.28:0/2935107520 shutdown_connections 2024-02-21T23:07:12.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.650+0000 7f56ede00640 1 -- 172.21.15.28:0/2935107520 wait complete. 2024-02-21T23:07:12.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.651+0000 7f56ede00640 1 Processor -- start 2024-02-21T23:07:12.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.652+0000 7f56ede00640 1 -- start start 2024-02-21T23:07:12.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.652+0000 7f56ede00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56e80c35d0 0x7f56e814b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:12.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.652+0000 7f56ede00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56e80c55f0 0x7f56e8154140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:12.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.652+0000 7f56ede00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f56e8152690 0x7f56e80aa840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:12.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.652+0000 7f56ede00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56e806d570 con 0x7f56e80c35d0 2024-02-21T23:07:12.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.652+0000 7f56ede00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56e806d290 con 0x7f56e80c55f0 2024-02-21T23:07:12.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.652+0000 7f56ede00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56e8155a20 con 0x7f56e8152690 2024-02-21T23:07:12.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.653+0000 7f56eca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56e80c35d0 0x7f56e814b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:12.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.653+0000 7f56eca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56e80c35d0 0x7f56e814b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55362/0 (socket says 172.21.15.28:55362) 2024-02-21T23:07:12.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.653+0000 7f56eca00640 1 -- 172.21.15.28:0/2752692556 learned_addr learned my addr 172.21.15.28:0/2752692556 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:12.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.653+0000 7f56e7e00640 1 --2- 172.21.15.28:0/2752692556 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56e80c55f0 0x7f56e8154140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:12.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.653+0000 7f56eca00640 1 -- 172.21.15.28:0/2752692556 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f56e8152690 msgr2=0x7f56e80aa840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:12.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.653+0000 7f56eca00640 1 --2- 172.21.15.28:0/2752692556 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f56e8152690 0x7f56e80aa840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:12.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.653+0000 7f56eca00640 1 -- 172.21.15.28:0/2752692556 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56e80c55f0 msgr2=0x7f56e8154140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:12.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.653+0000 7f56eca00640 1 --2- 172.21.15.28:0/2752692556 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56e80c55f0 0x7f56e8154140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:12.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.653+0000 7f56eca00640 1 -- 172.21.15.28:0/2752692556 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56e80a8ea0 con 0x7f56e80c35d0 2024-02-21T23:07:12.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.653+0000 7f56e7e00640 1 --2- 172.21.15.28:0/2752692556 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56e80c55f0 0x7f56e8154140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:07:12.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.654+0000 7f56eca00640 1 --2- 172.21.15.28:0/2752692556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56e80c35d0 0x7f56e814b340 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f56dc00c7f0 tx=0x7f56dc00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:12.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.654+0000 7f56e4c00640 1 -- 172.21.15.28:0/2752692556 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56dc015070 con 0x7f56e80c35d0 2024-02-21T23:07:12.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.654+0000 7f56e4c00640 1 -- 172.21.15.28:0/2752692556 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56dc007820 con 0x7f56e80c35d0 2024-02-21T23:07:12.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.654+0000 7f56e4c00640 1 -- 172.21.15.28:0/2752692556 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56dc007bd0 con 0x7f56e80c35d0 2024-02-21T23:07:12.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.654+0000 7f56ede00640 1 -- 172.21.15.28:0/2752692556 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56e80a9180 con 0x7f56e80c35d0 2024-02-21T23:07:12.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.654+0000 7f56ede00640 1 -- 172.21.15.28:0/2752692556 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56e80a8620 con 0x7f56e80c35d0 2024-02-21T23:07:12.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.656+0000 7f56ede00640 1 -- 172.21.15.28:0/2752692556 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56e804f290 con 0x7f56e80c35d0 2024-02-21T23:07:12.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.657+0000 7f56e4c00640 1 -- 172.21.15.28:0/2752692556 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f56dc018070 con 0x7f56e80c35d0 2024-02-21T23:07:12.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.658+0000 7f56e4c00640 1 --2- 172.21.15.28:0/2752692556 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f56c806b500 0x7f56c806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:12.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.658+0000 7f56e4c00640 1 -- 172.21.15.28:0/2752692556 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f56dc08a570 con 0x7f56e80c35d0 2024-02-21T23:07:12.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.659+0000 7f56e7e00640 1 --2- 172.21.15.28:0/2752692556 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f56c806b500 0x7f56c806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:12.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.660+0000 7f56e7e00640 1 --2- 172.21.15.28:0/2752692556 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f56c806b500 0x7f56c806d9c0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f56d4036000 tx=0x7f56d4008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:12.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.663+0000 7f56e4c00640 1 -- 172.21.15.28:0/2752692556 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f56dc059180 con 0x7f56e80c35d0 2024-02-21T23:07:12.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.863+0000 7f56ede00640 1 -- 172.21.15.28:0/2752692556 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f56e80c7410 con 0x7f56e80c35d0 2024-02-21T23:07:12.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.864+0000 7f56e4c00640 1 -- 172.21.15.28:0/2752692556 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v99) v1 ==== 100+0+11 (secure 0 0 0) 0x7f56dc058a80 con 0x7f56e80c35d0 2024-02-21T23:07:12.852 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-02-21T23:07:12.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.866+0000 7f56ede00640 1 -- 172.21.15.28:0/2752692556 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f56c806b500 msgr2=0x7f56c806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:12.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.866+0000 7f56ede00640 1 --2- 172.21.15.28:0/2752692556 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f56c806b500 0x7f56c806d9c0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f56d4036000 tx=0x7f56d4008030 comp rx=0 tx=0).stop 2024-02-21T23:07:12.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.866+0000 7f56ede00640 1 -- 172.21.15.28:0/2752692556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56e80c35d0 msgr2=0x7f56e814b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:12.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.866+0000 7f56ede00640 1 --2- 172.21.15.28:0/2752692556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56e80c35d0 0x7f56e814b340 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f56dc00c7f0 tx=0x7f56dc00ccb0 comp rx=0 tx=0).stop 2024-02-21T23:07:12.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.866+0000 7f56ede00640 1 -- 172.21.15.28:0/2752692556 shutdown_connections 2024-02-21T23:07:12.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.866+0000 7f56ede00640 1 --2- 172.21.15.28:0/2752692556 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f56c806b500 0x7f56c806d9c0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:12.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.866+0000 7f56ede00640 1 --2- 172.21.15.28:0/2752692556 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f56e8152690 0x7f56e80aa840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:12.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.866+0000 7f56ede00640 1 --2- 172.21.15.28:0/2752692556 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56e80c55f0 0x7f56e8154140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:12.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.866+0000 7f56ede00640 1 --2- 172.21.15.28:0/2752692556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56e80c35d0 0x7f56e814b340 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:12.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.866+0000 7f56ede00640 1 -- 172.21.15.28:0/2752692556 >> 172.21.15.28:0/2752692556 conn(0x7f56e80c8620 msgr2=0x7f56e80c8a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:12.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.867+0000 7f56ede00640 1 -- 172.21.15.28:0/2752692556 shutdown_connections 2024-02-21T23:07:12.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:12.867+0000 7f56ede00640 1 -- 172.21.15.28:0/2752692556 wait complete. 2024-02-21T23:07:12.874 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:07:13.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.108+0000 7effdba00640 1 Processor -- start 2024-02-21T23:07:13.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.109+0000 7effdba00640 1 -- start start 2024-02-21T23:07:13.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.110+0000 7effdba00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7effd40c11f0 0x7effd4153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:13.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.110+0000 7effdba00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effd4152e70 con 0x7effd415b6d0 2024-02-21T23:07:13.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.110+0000 7effdba00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effd4153150 con 0x7effd40c15d0 2024-02-21T23:07:13.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.110+0000 7effdba00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effd40c1a60 con 0x7effd40c11f0 2024-02-21T23:07:13.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.110+0000 7effd9000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7effd40c11f0 0x7effd4153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:13.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.110+0000 7effd9a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7effd415b6d0 0x7effd415dac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:40508/0 (socket says 172.21.15.28:40508) 2024-02-21T23:07:13.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.110+0000 7effd9a00640 1 -- 172.21.15.28:0/3391260615 learned_addr learned my addr 172.21.15.28:0/3391260615 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:07:13.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.111+0000 7effd3400640 1 -- 172.21.15.28:0/3391260615 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 548871939 0 0) 0x7effd4152e70 con 0x7effd415b6d0 2024-02-21T23:07:13.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.111+0000 7effd9000640 1 -- 172.21.15.28:0/3391260615 >> v1:172.21.15.81:6789/0 conn(0x7effd40c15d0 legacy=0x7effd406dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:13.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.111+0000 7effd9000640 1 -- 172.21.15.28:0/3391260615 >> v1:172.21.15.28:6789/0 conn(0x7effd415b6d0 legacy=0x7effd415dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:13.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.111+0000 7effd9000640 1 -- 172.21.15.28:0/3391260615 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effd4153430 con 0x7effd40c11f0 2024-02-21T23:07:13.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.112+0000 7effd9000640 1 --2- 172.21.15.28:0/3391260615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effd40c11f0 0x7effd4153950 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7effc80097a0 tx=0x7effc802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=151920e07516b9b0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:13.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.112+0000 7effd3400640 1 -- 172.21.15.28:0/3391260615 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc80315a0 con 0x7effd40c11f0 2024-02-21T23:07:13.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.112+0000 7effd3400640 1 -- 172.21.15.28:0/3391260615 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effc8037070 con 0x7effd40c11f0 2024-02-21T23:07:13.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.112+0000 7effd3400640 1 -- 172.21.15.28:0/3391260615 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc8031960 con 0x7effd40c11f0 2024-02-21T23:07:13.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.113+0000 7effdba00640 1 -- 172.21.15.28:0/3391260615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effd40c11f0 msgr2=0x7effd4153950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:13.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.113+0000 7effdba00640 1 --2- 172.21.15.28:0/3391260615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effd40c11f0 0x7effd4153950 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7effc80097a0 tx=0x7effc802b840 comp rx=0 tx=0).stop 2024-02-21T23:07:13.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.113+0000 7effdba00640 1 -- 172.21.15.28:0/3391260615 shutdown_connections 2024-02-21T23:07:13.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.113+0000 7effdba00640 1 --2- 172.21.15.28:0/3391260615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effd40c11f0 0x7effd4153950 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:13.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.113+0000 7effdba00640 1 -- 172.21.15.28:0/3391260615 >> 172.21.15.28:0/3391260615 conn(0x7effd40c67f0 msgr2=0x7effd40c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:13.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.114+0000 7effdba00640 1 -- 172.21.15.28:0/3391260615 shutdown_connections 2024-02-21T23:07:13.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.114+0000 7effdba00640 1 -- 172.21.15.28:0/3391260615 wait complete. 2024-02-21T23:07:13.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.115+0000 7effdba00640 1 Processor -- start 2024-02-21T23:07:13.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.115+0000 7effdba00640 1 -- start start 2024-02-21T23:07:13.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.116+0000 7effdba00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effd40c11f0 0x7effd40bfe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:13.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.116+0000 7effdba00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7effd40c15d0 0x7effd41ee3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:13.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.116+0000 7effdba00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7effd415b6d0 0x7effd41ee8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:13.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.116+0000 7effdba00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effd40c4c00 con 0x7effd40c11f0 2024-02-21T23:07:13.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.116+0000 7effdba00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effd40a5ec0 con 0x7effd415b6d0 2024-02-21T23:07:13.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.116+0000 7effdba00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effd40a6110 con 0x7effd40c15d0 2024-02-21T23:07:13.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.116+0000 7effd9000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effd40c11f0 0x7effd40bfe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:13.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.116+0000 7effd9000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effd40c11f0 0x7effd40bfe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55384/0 (socket says 172.21.15.28:55384) 2024-02-21T23:07:13.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.116+0000 7effd9000640 1 -- 172.21.15.28:0/3876287773 learned_addr learned my addr 172.21.15.28:0/3876287773 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:13.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.117+0000 7effd3e00640 1 --2- 172.21.15.28:0/3876287773 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7effd40c15d0 0x7effd41ee3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:13.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.117+0000 7effd9a00640 1 --2- 172.21.15.28:0/3876287773 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7effd415b6d0 0x7effd41ee8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:13.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.117+0000 7effd9000640 1 -- 172.21.15.28:0/3876287773 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7effd40c15d0 msgr2=0x7effd41ee3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:13.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.117+0000 7effd9000640 1 --2- 172.21.15.28:0/3876287773 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7effd40c15d0 0x7effd41ee3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:13.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.117+0000 7effd9000640 1 -- 172.21.15.28:0/3876287773 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7effd415b6d0 msgr2=0x7effd41ee8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:13.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.117+0000 7effd9000640 1 --2- 172.21.15.28:0/3876287773 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7effd415b6d0 0x7effd41ee8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:13.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.117+0000 7effd9000640 1 -- 172.21.15.28:0/3876287773 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effd40a7e80 con 0x7effd40c11f0 2024-02-21T23:07:13.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.117+0000 7effd9000640 1 --2- 172.21.15.28:0/3876287773 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effd40c11f0 0x7effd40bfe90 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7effc80096f0 tx=0x7effc80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:13.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.117+0000 7effd0c00640 1 -- 172.21.15.28:0/3876287773 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc8032840 con 0x7effd40c11f0 2024-02-21T23:07:13.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.118+0000 7effd0c00640 1 -- 172.21.15.28:0/3876287773 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effc80329e0 con 0x7effd40c11f0 2024-02-21T23:07:13.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.118+0000 7effd0c00640 1 -- 172.21.15.28:0/3876287773 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc8032dd0 con 0x7effd40c11f0 2024-02-21T23:07:13.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.118+0000 7effdba00640 1 -- 172.21.15.28:0/3876287773 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effd40a8160 con 0x7effd40c11f0 2024-02-21T23:07:13.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.118+0000 7effdba00640 1 -- 172.21.15.28:0/3876287773 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effd40c03d0 con 0x7effd40c11f0 2024-02-21T23:07:13.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.119+0000 7effdba00640 1 -- 172.21.15.28:0/3876287773 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effd4067f30 con 0x7effd40c11f0 2024-02-21T23:07:13.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.120+0000 7effd0c00640 1 -- 172.21.15.28:0/3876287773 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7effc803b3f0 con 0x7effd40c11f0 2024-02-21T23:07:13.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.121+0000 7effd0c00640 1 --2- 172.21.15.28:0/3876287773 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7effb406b4d0 0x7effb406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:13.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.121+0000 7effd0c00640 1 -- 172.21.15.28:0/3876287773 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6187+0+0 (secure 0 0 0) 0x7effc802f030 con 0x7effd40c11f0 2024-02-21T23:07:13.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.122+0000 7effd3e00640 1 --2- 172.21.15.28:0/3876287773 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7effb406b4d0 0x7effb406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:13.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.123+0000 7effd3e00640 1 --2- 172.21.15.28:0/3876287773 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7effb406b4d0 0x7effb406d990 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7effd41ef460 tx=0x7effc0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:13.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.126+0000 7effd0c00640 1 -- 172.21.15.28:0/3876287773 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7effc8040070 con 0x7effd40c11f0 2024-02-21T23:07:13.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.365+0000 7effdba00640 1 -- 172.21.15.28:0/3876287773 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7effd40c0960 con 0x7effd40c11f0 2024-02-21T23:07:13.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.366+0000 7effd0c00640 1 -- 172.21.15.28:0/3876287773 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+4934 (secure 0 0 0) 0x7effc807afc0 con 0x7effd40c11f0 2024-02-21T23:07:13.354 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:07:13.354 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":93,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5567,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1194613007","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1194613007},{"type":"v1","addr":"172.21.15.81:6835","nonce":1194613007}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":5570,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/2315735157","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":2315735157},{"type":"v1","addr":"172.21.15.81:6837","nonce":2315735157}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":7251,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2558379712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2558379712},{"type":"v1","addr":"172.21.15.28:6835","nonce":2558379712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":91}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:06:54.201589+0000","modified":"2024-02-21T23:07:07.869167+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":99,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5555},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5555":{"gid":5555,"name":"c","rank":0,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.28:6833/85689296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":85689296},{"type":"v1","addr":"172.21.15.28:6833","nonce":85689296}]},"join_fscid":7,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-02-21T23:07:13.354 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 93 2024-02-21T23:07:13.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.368+0000 7effdba00640 1 -- 172.21.15.28:0/3876287773 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7effb406b4d0 msgr2=0x7effb406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:13.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.368+0000 7effdba00640 1 --2- 172.21.15.28:0/3876287773 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7effb406b4d0 0x7effb406d990 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7effd41ef460 tx=0x7effc0009040 comp rx=0 tx=0).stop 2024-02-21T23:07:13.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.368+0000 7effdba00640 1 -- 172.21.15.28:0/3876287773 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effd40c11f0 msgr2=0x7effd40bfe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:13.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.368+0000 7effdba00640 1 --2- 172.21.15.28:0/3876287773 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effd40c11f0 0x7effd40bfe90 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7effc80096f0 tx=0x7effc80046c0 comp rx=0 tx=0).stop 2024-02-21T23:07:13.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.368+0000 7effdba00640 1 -- 172.21.15.28:0/3876287773 shutdown_connections 2024-02-21T23:07:13.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.368+0000 7effdba00640 1 --2- 172.21.15.28:0/3876287773 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7effb406b4d0 0x7effb406d990 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:13.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.368+0000 7effdba00640 1 --2- 172.21.15.28:0/3876287773 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7effd40c15d0 0x7effd41ee3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:13.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.368+0000 7effdba00640 1 --2- 172.21.15.28:0/3876287773 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7effd415b6d0 0x7effd41ee8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:13.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.368+0000 7effdba00640 1 --2- 172.21.15.28:0/3876287773 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effd40c11f0 0x7effd40bfe90 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:13.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.368+0000 7effdba00640 1 -- 172.21.15.28:0/3876287773 >> 172.21.15.28:0/3876287773 conn(0x7effd40c67f0 msgr2=0x7effd40c5370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:13.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.369+0000 7effdba00640 1 -- 172.21.15.28:0/3876287773 shutdown_connections 2024-02-21T23:07:13.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.369+0000 7effdba00640 1 -- 172.21.15.28:0/3876287773 wait complete. 2024-02-21T23:07:13.378 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:07:13.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.605+0000 7ff53be00640 1 Processor -- start 2024-02-21T23:07:13.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.605+0000 7ff53be00640 1 -- start start 2024-02-21T23:07:13.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.608+0000 7ff53be00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7ff53c0c8a80 0x7ff53c156c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:13.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.608+0000 7ff53be00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff53c0c8e60 0x7ff53c153640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:13.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.608+0000 7ff53be00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff53c153b80 con 0x7ff53c1540b0 2024-02-21T23:07:13.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.608+0000 7ff53be00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff53c0c92e0 con 0x7ff53c0c8a80 2024-02-21T23:07:13.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.608+0000 7ff53be00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff53c0c8000 con 0x7ff53c0c8e60 2024-02-21T23:07:13.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.609+0000 7ff53b400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff53c1540b0 0x7ff53c0705a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:40522/0 (socket says 172.21.15.28:40522) 2024-02-21T23:07:13.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.609+0000 7ff53b400640 1 -- 172.21.15.28:0/4077841998 learned_addr learned my addr 172.21.15.28:0/4077841998 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:07:13.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.609+0000 7ff53aa00640 1 --2- 172.21.15.28:0/4077841998 >> v2:172.21.15.81:3300/0 conn(0x7ff53c0c8a80 0x7ff53c156c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:13.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.609+0000 7ff53a000640 1 --2- 172.21.15.28:0/4077841998 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff53c0c8e60 0x7ff53c153640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:13.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.609+0000 7ff539600640 1 -- 172.21.15.28:0/4077841998 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3245975949 0 0) 0x7ff534001490 con 0x7ff53c1540b0 2024-02-21T23:07:13.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.609+0000 7ff539600640 1 -- 172.21.15.28:0/4077841998 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff520003520 con 0x7ff53c1540b0 2024-02-21T23:07:13.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.610+0000 7ff539600640 1 -- 172.21.15.28:0/4077841998 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3546274816 0 0) 0x7ff520003520 con 0x7ff53c1540b0 2024-02-21T23:07:13.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.610+0000 7ff539600640 1 -- 172.21.15.28:0/4077841998 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff53c0c8e60 msgr2=0x7ff53c153640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:13.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.610+0000 7ff539600640 1 --2- 172.21.15.28:0/4077841998 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff53c0c8e60 0x7ff53c153640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:13.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.610+0000 7ff539600640 1 -- 172.21.15.28:0/4077841998 >> v2:172.21.15.81:3300/0 conn(0x7ff53c0c8a80 msgr2=0x7ff53c156c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:13.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.610+0000 7ff539600640 1 --2- 172.21.15.28:0/4077841998 >> v2:172.21.15.81:3300/0 conn(0x7ff53c0c8a80 0x7ff53c156c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:13.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.610+0000 7ff539600640 1 -- 172.21.15.28:0/4077841998 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff53c0c82e0 con 0x7ff53c1540b0 2024-02-21T23:07:13.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.610+0000 7ff539600640 1 -- 172.21.15.28:0/4077841998 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7ff534001bc0 con 0x7ff53c1540b0 2024-02-21T23:07:13.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.610+0000 7ff53aa00640 1 --2- 172.21.15.28:0/4077841998 >> v2:172.21.15.81:3300/0 conn(0x7ff53c0c8a80 0x7ff53c156c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-21T23:07:13.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.610+0000 7ff539600640 1 -- 172.21.15.28:0/4077841998 >> v1:172.21.15.28:6789/0 conn(0x7ff53c1540b0 legacy=0x7ff53c0705a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:13.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.610+0000 7ff539600640 1 --2- 172.21.15.28:0/4077841998 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5200039b0 0x7ff520003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:13.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.610+0000 7ff539600640 1 --2- 172.21.15.28:0/4077841998 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff53c0c8e60 0x7ff520004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:13.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.610+0000 7ff539600640 1 --2- 172.21.15.28:0/4077841998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff520004d40 0x7ff5200051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:13.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.610+0000 7ff539600640 1 -- 172.21.15.28:0/4077841998 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff53c0c92e0 con 0x7ff520004d40 2024-02-21T23:07:13.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.610+0000 7ff539600640 1 -- 172.21.15.28:0/4077841998 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff53c0c8000 con 0x7ff5200039b0 2024-02-21T23:07:13.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.610+0000 7ff539600640 1 -- 172.21.15.28:0/4077841998 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff52000b9c0 con 0x7ff53c0c8e60 2024-02-21T23:07:13.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.610+0000 7ff53b400640 1 --2- 172.21.15.28:0/4077841998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff520004d40 0x7ff5200051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:13.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.610+0000 7ff53aa00640 1 --2- 172.21.15.28:0/4077841998 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5200039b0 0x7ff520003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:13.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.610+0000 7ff53a000640 1 --2- 172.21.15.28:0/4077841998 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff53c0c8e60 0x7ff520004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:13.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.611+0000 7ff53b400640 1 -- 172.21.15.28:0/4077841998 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff53c0c8e60 msgr2=0x7ff520004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:13.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.611+0000 7ff53b400640 1 --2- 172.21.15.28:0/4077841998 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff53c0c8e60 0x7ff520004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:13.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.611+0000 7ff53b400640 1 -- 172.21.15.28:0/4077841998 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5200039b0 msgr2=0x7ff520003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:13.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.611+0000 7ff53b400640 1 --2- 172.21.15.28:0/4077841998 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff5200039b0 0x7ff520003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:13.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.611+0000 7ff53b400640 1 -- 172.21.15.28:0/4077841998 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff520003520 con 0x7ff520004d40 2024-02-21T23:07:13.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.612+0000 7ff53b400640 1 --2- 172.21.15.28:0/4077841998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff520004d40 0x7ff5200051c0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7ff53400a9a0 tx=0x7ff520005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fda3d6f02864378 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:13.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.612+0000 7ff539600640 1 -- 172.21.15.28:0/4077841998 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff53400a5c0 con 0x7ff520004d40 2024-02-21T23:07:13.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.612+0000 7ff539600640 1 -- 172.21.15.28:0/4077841998 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff534035420 con 0x7ff520004d40 2024-02-21T23:07:13.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.612+0000 7ff539600640 1 -- 172.21.15.28:0/4077841998 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff534035830 con 0x7ff520004d40 2024-02-21T23:07:13.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.613+0000 7ff53be00640 1 -- 172.21.15.28:0/4077841998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff520004d40 msgr2=0x7ff5200051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:13.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.613+0000 7ff53be00640 1 --2- 172.21.15.28:0/4077841998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff520004d40 0x7ff5200051c0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7ff53400a9a0 tx=0x7ff520005830 comp rx=0 tx=0).stop 2024-02-21T23:07:13.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.613+0000 7ff53b400640 1 -- 172.21.15.28:0/4077841998 reap_dead start 2024-02-21T23:07:13.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.613+0000 7ff53be00640 1 -- 172.21.15.28:0/4077841998 shutdown_connections 2024-02-21T23:07:13.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.613+0000 7ff53be00640 1 -- 172.21.15.28:0/4077841998 >> 172.21.15.28:0/4077841998 conn(0x7ff53c06b260 msgr2=0x7ff53c06b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:13.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.613+0000 7ff53be00640 1 -- 172.21.15.28:0/4077841998 shutdown_connections 2024-02-21T23:07:13.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.613+0000 7ff53be00640 1 -- 172.21.15.28:0/4077841998 wait complete. 2024-02-21T23:07:13.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.615+0000 7ff53be00640 1 Processor -- start 2024-02-21T23:07:13.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.616+0000 7ff53be00640 1 -- start start 2024-02-21T23:07:13.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.617+0000 7ff53be00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff520004d40 0x7ff53c0a9720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:13.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.617+0000 7ff53be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff53c1eaac0 0x7ff53c1eb170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:13.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.617+0000 7ff53be00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff53c14fbf0 0x7ff53c1eb6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:13.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.617+0000 7ff53be00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff53c156770 con 0x7ff53c1eaac0 2024-02-21T23:07:13.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.617+0000 7ff53be00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff53c156490 con 0x7ff53c14fbf0 2024-02-21T23:07:13.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.617+0000 7ff53be00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff53c0c7c90 con 0x7ff520004d40 2024-02-21T23:07:13.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.617+0000 7ff53a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff53c1eaac0 0x7ff53c1eb170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:13.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.617+0000 7ff53a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff53c1eaac0 0x7ff53c1eb170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55400/0 (socket says 172.21.15.28:55400) 2024-02-21T23:07:13.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.617+0000 7ff53a000640 1 -- 172.21.15.28:0/3108631837 learned_addr learned my addr 172.21.15.28:0/3108631837 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:13.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.617+0000 7ff53aa00640 1 --2- 172.21.15.28:0/3108631837 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff520004d40 0x7ff53c0a9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:13.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.617+0000 7ff53b400640 1 --2- 172.21.15.28:0/3108631837 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff53c14fbf0 0x7ff53c1eb6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:13.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.618+0000 7ff53a000640 1 -- 172.21.15.28:0/3108631837 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff520004d40 msgr2=0x7ff53c0a9720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:13.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.618+0000 7ff53a000640 1 --2- 172.21.15.28:0/3108631837 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff520004d40 0x7ff53c0a9720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:13.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.618+0000 7ff53a000640 1 -- 172.21.15.28:0/3108631837 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff53c14fbf0 msgr2=0x7ff53c1eb6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:13.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.618+0000 7ff53a000640 1 --2- 172.21.15.28:0/3108631837 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff53c14fbf0 0x7ff53c1eb6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:13.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.618+0000 7ff53a000640 1 -- 172.21.15.28:0/3108631837 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff53c14f560 con 0x7ff53c1eaac0 2024-02-21T23:07:13.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.618+0000 7ff53a000640 1 --2- 172.21.15.28:0/3108631837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff53c1eaac0 0x7ff53c1eb170 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7ff528011600 tx=0x7ff528011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:13.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.618+0000 7ff532a00640 1 -- 172.21.15.28:0/3108631837 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff52800f920 con 0x7ff53c1eaac0 2024-02-21T23:07:13.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.618+0000 7ff532a00640 1 -- 172.21.15.28:0/3108631837 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff52800fac0 con 0x7ff53c1eaac0 2024-02-21T23:07:13.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.618+0000 7ff532a00640 1 -- 172.21.15.28:0/3108631837 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff52800fe70 con 0x7ff53c1eaac0 2024-02-21T23:07:13.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.618+0000 7ff53be00640 1 -- 172.21.15.28:0/3108631837 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff53c14dd50 con 0x7ff53c1eaac0 2024-02-21T23:07:13.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.619+0000 7ff53be00640 1 -- 172.21.15.28:0/3108631837 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff53c14e310 con 0x7ff53c1eaac0 2024-02-21T23:07:13.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.620+0000 7ff53be00640 1 -- 172.21.15.28:0/3108631837 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff53c04f290 con 0x7ff53c1eaac0 2024-02-21T23:07:13.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.621+0000 7ff532a00640 1 -- 172.21.15.28:0/3108631837 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff528015460 con 0x7ff53c1eaac0 2024-02-21T23:07:13.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.622+0000 7ff532a00640 1 --2- 172.21.15.28:0/3108631837 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff51c06b4d0 0x7ff51c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:13.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.622+0000 7ff532a00640 1 -- 172.21.15.28:0/3108631837 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6187+0+0 (secure 0 0 0) 0x7ff52808ad90 con 0x7ff53c1eaac0 2024-02-21T23:07:13.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.622+0000 7ff53aa00640 1 --2- 172.21.15.28:0/3108631837 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff51c06b4d0 0x7ff51c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:13.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.624+0000 7ff53aa00640 1 --2- 172.21.15.28:0/3108631837 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff51c06b4d0 0x7ff51c06d990 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7ff52c009ba0 tx=0x7ff52c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:13.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.627+0000 7ff532a00640 1 -- 172.21.15.28:0/3108631837 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ff528019030 con 0x7ff53c1eaac0 2024-02-21T23:07:13.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.830+0000 7ff53be00640 1 -- 172.21.15.28:0/3108631837 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff53c0c7520 con 0x7ff53c1eaac0 2024-02-21T23:07:13.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.833+0000 7ff532a00640 1 -- 172.21.15.28:0/3108631837 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) v1 ==== 74+0+16942 (secure 0 0 0) 0x7ff528059a20 con 0x7ff53c1eaac0 2024-02-21T23:07:13.822 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:07:13.823 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":99,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:07:05.837927+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:06:52.151181+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-02-21T23:06:53.221776+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6835/2477712025":"2024-02-22T23:07:05.837870+0000","172.21.15.28:6834/2477712025":"2024-02-22T23:07:05.837870+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:07:13.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.836+0000 7ff53be00640 1 -- 172.21.15.28:0/3108631837 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff51c06b4d0 msgr2=0x7ff51c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:13.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.836+0000 7ff53be00640 1 --2- 172.21.15.28:0/3108631837 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff51c06b4d0 0x7ff51c06d990 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7ff52c009ba0 tx=0x7ff52c008040 comp rx=0 tx=0).stop 2024-02-21T23:07:13.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.836+0000 7ff53be00640 1 -- 172.21.15.28:0/3108631837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff53c1eaac0 msgr2=0x7ff53c1eb170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:13.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.836+0000 7ff53be00640 1 --2- 172.21.15.28:0/3108631837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff53c1eaac0 0x7ff53c1eb170 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7ff528011600 tx=0x7ff528011ac0 comp rx=0 tx=0).stop 2024-02-21T23:07:13.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.837+0000 7ff53be00640 1 -- 172.21.15.28:0/3108631837 shutdown_connections 2024-02-21T23:07:13.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.837+0000 7ff53be00640 1 --2- 172.21.15.28:0/3108631837 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff51c06b4d0 0x7ff51c06d990 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:13.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.837+0000 7ff53be00640 1 --2- 172.21.15.28:0/3108631837 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff53c14fbf0 0x7ff53c1eb6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:13.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.837+0000 7ff53be00640 1 --2- 172.21.15.28:0/3108631837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff53c1eaac0 0x7ff53c1eb170 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:13.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.837+0000 7ff53be00640 1 --2- 172.21.15.28:0/3108631837 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff520004d40 0x7ff53c0a9720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:13.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.837+0000 7ff53be00640 1 -- 172.21.15.28:0/3108631837 >> 172.21.15.28:0/3108631837 conn(0x7ff53c06b260 msgr2=0x7ff53c1de230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:13.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.837+0000 7ff53be00640 1 -- 172.21.15.28:0/3108631837 shutdown_connections 2024-02-21T23:07:13.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:13.837+0000 7ff53be00640 1 -- 172.21.15.28:0/3108631837 wait complete. 2024-02-21T23:07:13.855 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-02-21T23:07:13.855 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:07:14.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.087+0000 7fbc86600640 1 Processor -- start 2024-02-21T23:07:14.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.088+0000 7fbc86600640 1 -- start start 2024-02-21T23:07:14.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.089+0000 7fbc86600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fbc800aff90 0x7fbc800c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:14.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.089+0000 7fbc86600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbc8006dbf0 0x7fbc80158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:14.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.090+0000 7fbc86600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc800c74c0 con 0x7fbc800c5720 2024-02-21T23:07:14.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.090+0000 7fbc86600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc800c7270 con 0x7fbc8006dbf0 2024-02-21T23:07:14.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.090+0000 7fbc86600640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc80158630 con 0x7fbc800aff90 2024-02-21T23:07:14.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.090+0000 7fbc7fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbc8006dbf0 0x7fbc80158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:14.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.090+0000 7fbc7fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbc8006dbf0 0x7fbc80158ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55406/0 (socket says 172.21.15.28:55406) 2024-02-21T23:07:14.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.090+0000 7fbc7fe00640 1 -- 172.21.15.28:0/240665515 learned_addr learned my addr 172.21.15.28:0/240665515 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:14.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.090+0000 7fbc85200640 1 --2- 172.21.15.28:0/240665515 >> v2:172.21.15.81:3300/0 conn(0x7fbc800aff90 0x7fbc800c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:14.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.091+0000 7fbc7fe00640 1 -- 172.21.15.28:0/240665515 >> v2:172.21.15.81:3300/0 conn(0x7fbc800aff90 msgr2=0x7fbc800c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:14.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.091+0000 7fbc7fe00640 1 --2- 172.21.15.28:0/240665515 >> v2:172.21.15.81:3300/0 conn(0x7fbc800aff90 0x7fbc800c51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:14.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.091+0000 7fbc7fe00640 1 -- 172.21.15.28:0/240665515 >> v1:172.21.15.81:6789/0 conn(0x7fbc800c5720 legacy=0x7fbc8006c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:14.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.091+0000 7fbc7fe00640 1 -- 172.21.15.28:0/240665515 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc800c77a0 con 0x7fbc8006dbf0 2024-02-21T23:07:14.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.092+0000 7fbc7fe00640 1 --2- 172.21.15.28:0/240665515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc8006dbf0 0x7fbc80158ac0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fbc800c73f0 tx=0x7fbc6802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92d9bffc603216aa server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:14.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.092+0000 7fbc7f400640 1 -- 172.21.15.28:0/240665515 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc68034070 con 0x7fbc8006dbf0 2024-02-21T23:07:14.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.092+0000 7fbc7f400640 1 -- 172.21.15.28:0/240665515 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc68030860 con 0x7fbc8006dbf0 2024-02-21T23:07:14.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.092+0000 7fbc7f400640 1 -- 172.21.15.28:0/240665515 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc68030c30 con 0x7fbc8006dbf0 2024-02-21T23:07:14.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.093+0000 7fbc86600640 1 -- 172.21.15.28:0/240665515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc8006dbf0 msgr2=0x7fbc80158ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:14.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.093+0000 7fbc86600640 1 --2- 172.21.15.28:0/240665515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc8006dbf0 0x7fbc80158ac0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fbc800c73f0 tx=0x7fbc6802b9a0 comp rx=0 tx=0).stop 2024-02-21T23:07:14.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.093+0000 7fbc86600640 1 -- 172.21.15.28:0/240665515 shutdown_connections 2024-02-21T23:07:14.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.093+0000 7fbc86600640 1 --2- 172.21.15.28:0/240665515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc8006dbf0 0x7fbc80158ac0 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:14.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.093+0000 7fbc86600640 1 --2- 172.21.15.28:0/240665515 >> v2:172.21.15.81:3300/0 conn(0x7fbc800aff90 0x7fbc800c51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:14.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.093+0000 7fbc86600640 1 -- 172.21.15.28:0/240665515 >> 172.21.15.28:0/240665515 conn(0x7fbc800ac370 msgr2=0x7fbc800ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:14.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.093+0000 7fbc86600640 1 -- 172.21.15.28:0/240665515 shutdown_connections 2024-02-21T23:07:14.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.093+0000 7fbc86600640 1 -- 172.21.15.28:0/240665515 wait complete. 2024-02-21T23:07:14.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.096+0000 7fbc86600640 1 Processor -- start 2024-02-21T23:07:14.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.096+0000 7fbc86600640 1 -- start start 2024-02-21T23:07:14.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.097+0000 7fbc86600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc8006dbf0 0x7fbc800ae9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:14.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.098+0000 7fbc86600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbc800aff90 0x7fbc800c4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:14.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.098+0000 7fbc86600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc800c5720 0x7fbc80152d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:14.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.098+0000 7fbc86600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc800c8760 con 0x7fbc800c5720 2024-02-21T23:07:14.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.098+0000 7fbc86600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc800be0c0 con 0x7fbc8006dbf0 2024-02-21T23:07:14.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.098+0000 7fbc86600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc800c88e0 con 0x7fbc800aff90 2024-02-21T23:07:14.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.098+0000 7fbc85c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc800c5720 0x7fbc80152d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:14.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.098+0000 7fbc85c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc800c5720 0x7fbc80152d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55410/0 (socket says 172.21.15.28:55410) 2024-02-21T23:07:14.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.098+0000 7fbc85c00640 1 -- 172.21.15.28:0/2678348651 learned_addr learned my addr 172.21.15.28:0/2678348651 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:14.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.098+0000 7fbc85200640 1 --2- 172.21.15.28:0/2678348651 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc8006dbf0 0x7fbc800ae9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:14.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.098+0000 7fbc7fe00640 1 --2- 172.21.15.28:0/2678348651 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbc800aff90 0x7fbc800c4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:14.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.099+0000 7fbc85c00640 1 -- 172.21.15.28:0/2678348651 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbc800aff90 msgr2=0x7fbc800c4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:14.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.099+0000 7fbc85c00640 1 --2- 172.21.15.28:0/2678348651 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbc800aff90 0x7fbc800c4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:14.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.099+0000 7fbc85c00640 1 -- 172.21.15.28:0/2678348651 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc8006dbf0 msgr2=0x7fbc800ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:14.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.099+0000 7fbc85c00640 1 --2- 172.21.15.28:0/2678348651 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc8006dbf0 0x7fbc800ae9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:14.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.099+0000 7fbc85c00640 1 -- 172.21.15.28:0/2678348651 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc8014eee0 con 0x7fbc800c5720 2024-02-21T23:07:14.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.099+0000 7fbc85c00640 1 --2- 172.21.15.28:0/2678348651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc800c5720 0x7fbc80152d80 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7fbc7800c6b0 tx=0x7fbc7800cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:14.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.099+0000 7fbc7cc00640 1 -- 172.21.15.28:0/2678348651 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc78003b70 con 0x7fbc800c5720 2024-02-21T23:07:14.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.099+0000 7fbc7cc00640 1 -- 172.21.15.28:0/2678348651 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc78003d30 con 0x7fbc800c5720 2024-02-21T23:07:14.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.099+0000 7fbc86600640 1 -- 172.21.15.28:0/2678348651 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc801eb780 con 0x7fbc800c5720 2024-02-21T23:07:14.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.099+0000 7fbc7cc00640 1 -- 172.21.15.28:0/2678348651 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc78005310 con 0x7fbc800c5720 2024-02-21T23:07:14.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.100+0000 7fbc86600640 1 -- 172.21.15.28:0/2678348651 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc801ea9c0 con 0x7fbc800c5720 2024-02-21T23:07:14.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.101+0000 7fbc86600640 1 -- 172.21.15.28:0/2678348651 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc8004f290 con 0x7fbc800c5720 2024-02-21T23:07:14.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.102+0000 7fbc7cc00640 1 -- 172.21.15.28:0/2678348651 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbc780054b0 con 0x7fbc800c5720 2024-02-21T23:07:14.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.103+0000 7fbc7cc00640 1 --2- 172.21.15.28:0/2678348651 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbc5c06b500 0x7fbc5c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:14.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.103+0000 7fbc7cc00640 1 -- 172.21.15.28:0/2678348651 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fbc7808a210 con 0x7fbc800c5720 2024-02-21T23:07:14.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.103+0000 7fbc85200640 1 --2- 172.21.15.28:0/2678348651 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbc5c06b500 0x7fbc5c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:14.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.105+0000 7fbc85200640 1 --2- 172.21.15.28:0/2678348651 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbc5c06b500 0x7fbc5c06d9c0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fbc70006fd0 tx=0x7fbc70008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:14.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.107+0000 7fbc7cc00640 1 -- 172.21.15.28:0/2678348651 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fbc78058ea0 con 0x7fbc800c5720 2024-02-21T23:07:14.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.366+0000 7fbc86600640 1 -- 172.21.15.28:0/2678348651 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbc801592f0 con 0x7fbc800c5720 2024-02-21T23:07:14.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.367+0000 7fbc7cc00640 1 -- 172.21.15.28:0/2678348651 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+4934 (secure 0 0 0) 0x7fbc780587a0 con 0x7fbc800c5720 2024-02-21T23:07:14.357 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:07:14.357 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":93,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5567,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1194613007","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1194613007},{"type":"v1","addr":"172.21.15.81:6835","nonce":1194613007}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":5570,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/2315735157","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":2315735157},{"type":"v1","addr":"172.21.15.81:6837","nonce":2315735157}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":7251,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2558379712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2558379712},{"type":"v1","addr":"172.21.15.28:6835","nonce":2558379712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":91}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:06:54.201589+0000","modified":"2024-02-21T23:07:07.869167+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":99,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5555},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5555":{"gid":5555,"name":"c","rank":0,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.28:6833/85689296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":85689296},{"type":"v1","addr":"172.21.15.28:6833","nonce":85689296}]},"join_fscid":7,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-02-21T23:07:14.357 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 93 2024-02-21T23:07:14.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.370+0000 7fbc86600640 1 -- 172.21.15.28:0/2678348651 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbc5c06b500 msgr2=0x7fbc5c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:14.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.370+0000 7fbc86600640 1 --2- 172.21.15.28:0/2678348651 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbc5c06b500 0x7fbc5c06d9c0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fbc70006fd0 tx=0x7fbc70008040 comp rx=0 tx=0).stop 2024-02-21T23:07:14.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.370+0000 7fbc86600640 1 -- 172.21.15.28:0/2678348651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc800c5720 msgr2=0x7fbc80152d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:14.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.370+0000 7fbc86600640 1 --2- 172.21.15.28:0/2678348651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc800c5720 0x7fbc80152d80 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7fbc7800c6b0 tx=0x7fbc7800cb70 comp rx=0 tx=0).stop 2024-02-21T23:07:14.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.370+0000 7fbc86600640 1 -- 172.21.15.28:0/2678348651 shutdown_connections 2024-02-21T23:07:14.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.370+0000 7fbc86600640 1 --2- 172.21.15.28:0/2678348651 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbc5c06b500 0x7fbc5c06d9c0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:14.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.370+0000 7fbc86600640 1 --2- 172.21.15.28:0/2678348651 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbc800aff90 0x7fbc800c4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:14.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.370+0000 7fbc86600640 1 --2- 172.21.15.28:0/2678348651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc800c5720 0x7fbc80152d80 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:14.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.370+0000 7fbc86600640 1 --2- 172.21.15.28:0/2678348651 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc8006dbf0 0x7fbc800ae9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:14.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.370+0000 7fbc86600640 1 -- 172.21.15.28:0/2678348651 >> 172.21.15.28:0/2678348651 conn(0x7fbc800ac370 msgr2=0x7fbc801573e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:14.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.371+0000 7fbc86600640 1 -- 172.21.15.28:0/2678348651 shutdown_connections 2024-02-21T23:07:14.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.371+0000 7fbc86600640 1 -- 172.21.15.28:0/2678348651 wait complete. 2024-02-21T23:07:14.380 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:07:14.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.600+0000 7f64f3200640 1 Processor -- start 2024-02-21T23:07:14.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.600+0000 7f64f3200640 1 -- start start 2024-02-21T23:07:14.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.602+0000 7f64f3200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f64f40ca050 0x7f64f41558c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:14.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.602+0000 7f64f3200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f64f406bbf0 0x7f64f406bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:14.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.602+0000 7f64f3200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64f41528e0 con 0x7f64f4152500 2024-02-21T23:07:14.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.602+0000 7f64f3200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64f40c86e0 con 0x7f64f406bbf0 2024-02-21T23:07:14.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.602+0000 7f64f3200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64f40c8980 con 0x7f64f40ca050 2024-02-21T23:07:14.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.602+0000 7f64f2800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f64f406bbf0 0x7f64f406bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:14.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.602+0000 7f64f2800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f64f406bbf0 0x7f64f406bff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55420/0 (socket says 172.21.15.28:55420) 2024-02-21T23:07:14.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.602+0000 7f64f2800640 1 -- 172.21.15.28:0/22426088 learned_addr learned my addr 172.21.15.28:0/22426088 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:14.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.603+0000 7f64f1400640 1 --2- 172.21.15.28:0/22426088 >> v2:172.21.15.81:3300/0 conn(0x7f64f40ca050 0x7f64f41558c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:14.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.603+0000 7f64f2800640 1 -- 172.21.15.28:0/22426088 >> v2:172.21.15.81:3300/0 conn(0x7f64f40ca050 msgr2=0x7f64f41558c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:14.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.603+0000 7f64f2800640 1 --2- 172.21.15.28:0/22426088 >> v2:172.21.15.81:3300/0 conn(0x7f64f40ca050 0x7f64f41558c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:14.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.603+0000 7f64f2800640 1 -- 172.21.15.28:0/22426088 >> v1:172.21.15.81:6789/0 conn(0x7f64f4152500 legacy=0x7f64f40bf6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:14.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.603+0000 7f64f2800640 1 -- 172.21.15.28:0/22426088 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64f40c8c60 con 0x7f64f406bbf0 2024-02-21T23:07:14.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.605+0000 7f64f2800640 1 --2- 172.21.15.28:0/22426088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64f406bbf0 0x7f64f406bff0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f64f40cb8b0 tx=0x7f64ec02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cdf63fc28be825a server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:14.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.605+0000 7f64f0a00640 1 -- 172.21.15.28:0/22426088 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64ec02ca80 con 0x7f64f406bbf0 2024-02-21T23:07:14.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.605+0000 7f64f0a00640 1 -- 172.21.15.28:0/22426088 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64ec02cc20 con 0x7f64f406bbf0 2024-02-21T23:07:14.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.605+0000 7f64f0a00640 1 -- 172.21.15.28:0/22426088 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64ec0343f0 con 0x7f64f406bbf0 2024-02-21T23:07:14.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.606+0000 7f64f3200640 1 -- 172.21.15.28:0/22426088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64f406bbf0 msgr2=0x7f64f406bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:14.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.606+0000 7f64f3200640 1 --2- 172.21.15.28:0/22426088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64f406bbf0 0x7f64f406bff0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f64f40cb8b0 tx=0x7f64ec02bee0 comp rx=0 tx=0).stop 2024-02-21T23:07:14.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.606+0000 7f64f3200640 1 -- 172.21.15.28:0/22426088 shutdown_connections 2024-02-21T23:07:14.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.606+0000 7f64f3200640 1 --2- 172.21.15.28:0/22426088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64f406bbf0 0x7f64f406bff0 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:14.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.606+0000 7f64f3200640 1 --2- 172.21.15.28:0/22426088 >> v2:172.21.15.81:3300/0 conn(0x7f64f40ca050 0x7f64f41558c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:14.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.606+0000 7f64f3200640 1 -- 172.21.15.28:0/22426088 >> 172.21.15.28:0/22426088 conn(0x7f64f40ac450 msgr2=0x7f64f40ac860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:14.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.606+0000 7f64f3200640 1 -- 172.21.15.28:0/22426088 shutdown_connections 2024-02-21T23:07:14.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.606+0000 7f64f3200640 1 -- 172.21.15.28:0/22426088 wait complete. 2024-02-21T23:07:14.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.608+0000 7f64f3200640 1 Processor -- start 2024-02-21T23:07:14.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.608+0000 7f64f3200640 1 -- start start 2024-02-21T23:07:14.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.609+0000 7f64f3200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64f406bbf0 0x7f64f414d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:14.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.609+0000 7f64f3200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f64f40ca050 0x7f64f4157b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:14.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.609+0000 7f64f3200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64f4152500 0x7f64f4158090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:14.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.609+0000 7f64f3200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64f40bfdd0 con 0x7f64f406bbf0 2024-02-21T23:07:14.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.609+0000 7f64f3200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64f40c9220 con 0x7f64f4152500 2024-02-21T23:07:14.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.609+0000 7f64f3200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64f40bff50 con 0x7f64f40ca050 2024-02-21T23:07:14.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.609+0000 7f64f1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64f406bbf0 0x7f64f414d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:14.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.609+0000 7f64f1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64f406bbf0 0x7f64f414d2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55430/0 (socket says 172.21.15.28:55430) 2024-02-21T23:07:14.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.609+0000 7f64f1e00640 1 -- 172.21.15.28:0/2013488238 learned_addr learned my addr 172.21.15.28:0/2013488238 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:14.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.610+0000 7f64f1400640 1 --2- 172.21.15.28:0/2013488238 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f64f40ca050 0x7f64f4157b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:14.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.610+0000 7f64f2800640 1 --2- 172.21.15.28:0/2013488238 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64f4152500 0x7f64f4158090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:14.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.610+0000 7f64f1e00640 1 -- 172.21.15.28:0/2013488238 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f64f40ca050 msgr2=0x7f64f4157b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:14.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.610+0000 7f64f1e00640 1 --2- 172.21.15.28:0/2013488238 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f64f40ca050 0x7f64f4157b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:14.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.610+0000 7f64f1e00640 1 -- 172.21.15.28:0/2013488238 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64f4152500 msgr2=0x7f64f4158090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:14.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.610+0000 7f64f1e00640 1 --2- 172.21.15.28:0/2013488238 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64f4152500 0x7f64f4158090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:14.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.610+0000 7f64f1e00640 1 -- 172.21.15.28:0/2013488238 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64f4157210 con 0x7f64f406bbf0 2024-02-21T23:07:14.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.610+0000 7f64f1e00640 1 --2- 172.21.15.28:0/2013488238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64f406bbf0 0x7f64f414d2c0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f64e400c430 tx=0x7f64e400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:14.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.610+0000 7f64ea000640 1 -- 172.21.15.28:0/2013488238 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e4017070 con 0x7f64f406bbf0 2024-02-21T23:07:14.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.610+0000 7f64ea000640 1 -- 172.21.15.28:0/2013488238 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64e40052d0 con 0x7f64f406bbf0 2024-02-21T23:07:14.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.610+0000 7f64ea000640 1 -- 172.21.15.28:0/2013488238 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e4003830 con 0x7f64f406bbf0 2024-02-21T23:07:14.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.611+0000 7f64f3200640 1 -- 172.21.15.28:0/2013488238 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64f4149220 con 0x7f64f406bbf0 2024-02-21T23:07:14.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.611+0000 7f64f3200640 1 -- 172.21.15.28:0/2013488238 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64f41494e0 con 0x7f64f406bbf0 2024-02-21T23:07:14.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.612+0000 7f64f3200640 1 -- 172.21.15.28:0/2013488238 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64f404f290 con 0x7f64f406bbf0 2024-02-21T23:07:14.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.613+0000 7f64ea000640 1 -- 172.21.15.28:0/2013488238 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f64e401a070 con 0x7f64f406bbf0 2024-02-21T23:07:14.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.614+0000 7f64ea000640 1 --2- 172.21.15.28:0/2013488238 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f64d406b500 0x7f64d406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:14.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.614+0000 7f64ea000640 1 -- 172.21.15.28:0/2013488238 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f64e4089770 con 0x7f64f406bbf0 2024-02-21T23:07:14.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.615+0000 7f64f1400640 1 --2- 172.21.15.28:0/2013488238 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f64d406b500 0x7f64d406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:14.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.616+0000 7f64f1400640 1 --2- 172.21.15.28:0/2013488238 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f64d406b500 0x7f64d406d9c0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f64e0006fd0 tx=0x7f64e0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:14.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.619+0000 7f64ea000640 1 -- 172.21.15.28:0/2013488238 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f64e4058400 con 0x7f64f406bbf0 2024-02-21T23:07:14.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.927+0000 7f64f3200640 1 -- 172.21.15.28:0/2013488238 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f64f40c5160 con 0x7f64f406bbf0 2024-02-21T23:07:14.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.928+0000 7f64ea000640 1 -- 172.21.15.28:0/2013488238 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+4934 (secure 0 0 0) 0x7f64e4057d00 con 0x7f64f406bbf0 2024-02-21T23:07:14.918 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:07:14.918 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":93,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5567,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1194613007","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1194613007},{"type":"v1","addr":"172.21.15.81:6835","nonce":1194613007}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":5570,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/2315735157","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":2315735157},{"type":"v1","addr":"172.21.15.81:6837","nonce":2315735157}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":7251,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2558379712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2558379712},{"type":"v1","addr":"172.21.15.28:6835","nonce":2558379712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":91}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:06:54.201589+0000","modified":"2024-02-21T23:07:07.869167+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":99,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5555},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5555":{"gid":5555,"name":"c","rank":0,"incarnation":90,"state":"up:active","state_seq":8,"addr":"172.21.15.28:6833/85689296","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":85689296},{"type":"v1","addr":"172.21.15.28:6833","nonce":85689296}]},"join_fscid":7,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-02-21T23:07:14.918 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 93 2024-02-21T23:07:14.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.931+0000 7f64f3200640 1 -- 172.21.15.28:0/2013488238 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f64d406b500 msgr2=0x7f64d406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:14.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.931+0000 7f64f3200640 1 --2- 172.21.15.28:0/2013488238 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f64d406b500 0x7f64d406d9c0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f64e0006fd0 tx=0x7f64e0008040 comp rx=0 tx=0).stop 2024-02-21T23:07:14.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.931+0000 7f64f3200640 1 -- 172.21.15.28:0/2013488238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64f406bbf0 msgr2=0x7f64f414d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:14.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.931+0000 7f64f3200640 1 --2- 172.21.15.28:0/2013488238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64f406bbf0 0x7f64f414d2c0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f64e400c430 tx=0x7f64e400c8f0 comp rx=0 tx=0).stop 2024-02-21T23:07:14.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.931+0000 7f64f3200640 1 -- 172.21.15.28:0/2013488238 shutdown_connections 2024-02-21T23:07:14.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.931+0000 7f64f3200640 1 --2- 172.21.15.28:0/2013488238 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f64d406b500 0x7f64d406d9c0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:14.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.931+0000 7f64f3200640 1 --2- 172.21.15.28:0/2013488238 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f64f40ca050 0x7f64f4157b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:14.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.931+0000 7f64f3200640 1 --2- 172.21.15.28:0/2013488238 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64f4152500 0x7f64f4158090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:14.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.931+0000 7f64f3200640 1 --2- 172.21.15.28:0/2013488238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64f406bbf0 0x7f64f414d2c0 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:14.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.931+0000 7f64f3200640 1 -- 172.21.15.28:0/2013488238 >> 172.21.15.28:0/2013488238 conn(0x7f64f40ac450 msgr2=0x7f64f41505f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:14.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.931+0000 7f64f3200640 1 -- 172.21.15.28:0/2013488238 shutdown_connections 2024-02-21T23:07:14.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:14.931+0000 7f64f3200640 1 -- 172.21.15.28:0/2013488238 wait complete. 2024-02-21T23:07:14.941 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:07:15.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.172+0000 7fe682800640 1 Processor -- start 2024-02-21T23:07:15.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.172+0000 7fe682800640 1 -- start start 2024-02-21T23:07:15.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.175+0000 7fe682800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fe67c072400 0x7fe67c0c0710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:15.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.175+0000 7fe682800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe67c0c3430 0x7fe67c0c0c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:15.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.175+0000 7fe682800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe67c155ad0 con 0x7fe67c0aca20 2024-02-21T23:07:15.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.175+0000 7fe682800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe67c0c19e0 con 0x7fe67c0c3430 2024-02-21T23:07:15.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.175+0000 7fe682800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe67c0c1cc0 con 0x7fe67c072400 2024-02-21T23:07:15.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.175+0000 7fe681400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe67c0aca20 0x7fe67c06be80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59894/0 (socket says 172.21.15.28:59894) 2024-02-21T23:07:15.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.175+0000 7fe681400640 1 -- 172.21.15.28:0/3614533602 learned_addr learned my addr 172.21.15.28:0/3614533602 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:07:15.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.176+0000 7fe681e00640 1 --2- 172.21.15.28:0/3614533602 >> v2:172.21.15.28:3300/0 conn(0x7fe67c0c3430 0x7fe67c0c0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:15.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.176+0000 7fe67be00640 1 -- 172.21.15.28:0/3614533602 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1540962019 0 0) 0x7fe67c155ad0 con 0x7fe67c0aca20 2024-02-21T23:07:15.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.176+0000 7fe67be00640 1 -- 172.21.15.28:0/3614533602 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe664003520 con 0x7fe67c0aca20 2024-02-21T23:07:15.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.176+0000 7fe680a00640 1 --2- 172.21.15.28:0/3614533602 >> v2:172.21.15.81:3300/0 conn(0x7fe67c072400 0x7fe67c0c0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:15.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.176+0000 7fe681e00640 1 -- 172.21.15.28:0/3614533602 >> v2:172.21.15.81:3300/0 conn(0x7fe67c072400 msgr2=0x7fe67c0c0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:15.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.176+0000 7fe681e00640 1 --2- 172.21.15.28:0/3614533602 >> v2:172.21.15.81:3300/0 conn(0x7fe67c072400 0x7fe67c0c0710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:15.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.176+0000 7fe681e00640 1 -- 172.21.15.28:0/3614533602 >> v1:172.21.15.28:6789/0 conn(0x7fe67c0aca20 legacy=0x7fe67c06be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:15.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.176+0000 7fe681e00640 1 -- 172.21.15.28:0/3614533602 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe67c067250 con 0x7fe67c0c3430 2024-02-21T23:07:15.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.177+0000 7fe681e00640 1 --2- 172.21.15.28:0/3614533602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe67c0c3430 0x7fe67c0c0c50 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7fe67c06d590 tx=0x7fe67402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=735091b38fdbbd9 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:15.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.178+0000 7fe67be00640 1 -- 172.21.15.28:0/3614533602 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6740315a0 con 0x7fe67c0c3430 2024-02-21T23:07:15.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.178+0000 7fe67be00640 1 -- 172.21.15.28:0/3614533602 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe674037070 con 0x7fe67c0c3430 2024-02-21T23:07:15.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.178+0000 7fe67be00640 1 -- 172.21.15.28:0/3614533602 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe674031940 con 0x7fe67c0c3430 2024-02-21T23:07:15.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.178+0000 7fe682800640 1 -- 172.21.15.28:0/3614533602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe67c0c3430 msgr2=0x7fe67c0c0c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:15.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.178+0000 7fe682800640 1 --2- 172.21.15.28:0/3614533602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe67c0c3430 0x7fe67c0c0c50 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7fe67c06d590 tx=0x7fe67402b840 comp rx=0 tx=0).stop 2024-02-21T23:07:15.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.178+0000 7fe682800640 1 -- 172.21.15.28:0/3614533602 shutdown_connections 2024-02-21T23:07:15.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.178+0000 7fe682800640 1 --2- 172.21.15.28:0/3614533602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe67c0c3430 0x7fe67c0c0c50 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:15.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.178+0000 7fe682800640 1 --2- 172.21.15.28:0/3614533602 >> v2:172.21.15.81:3300/0 conn(0x7fe67c072400 0x7fe67c0c0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:15.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.178+0000 7fe682800640 1 -- 172.21.15.28:0/3614533602 >> 172.21.15.28:0/3614533602 conn(0x7fe67c071910 msgr2=0x7fe67c071cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:15.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.179+0000 7fe682800640 1 -- 172.21.15.28:0/3614533602 shutdown_connections 2024-02-21T23:07:15.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.179+0000 7fe682800640 1 -- 172.21.15.28:0/3614533602 wait complete. 2024-02-21T23:07:15.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.180+0000 7fe682800640 1 Processor -- start 2024-02-21T23:07:15.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.180+0000 7fe682800640 1 -- start start 2024-02-21T23:07:15.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.181+0000 7fe682800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe67c072400 0x7fe67c154ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:15.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.181+0000 7fe682800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe67c0aca20 0x7fe67c0aa250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:15.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.181+0000 7fe682800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe67c0c3430 0x7fe67c0aa790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:15.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.181+0000 7fe682800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe67c06c710 con 0x7fe67c072400 2024-02-21T23:07:15.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.181+0000 7fe682800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe67c06c590 con 0x7fe67c0aca20 2024-02-21T23:07:15.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.181+0000 7fe682800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe67c0ace00 con 0x7fe67c0c3430 2024-02-21T23:07:15.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.182+0000 7fe681400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe67c072400 0x7fe67c154ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:15.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.182+0000 7fe681400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe67c072400 0x7fe67c154ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50772/0 (socket says 172.21.15.28:50772) 2024-02-21T23:07:15.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.182+0000 7fe681400640 1 -- 172.21.15.28:0/1959097100 learned_addr learned my addr 172.21.15.28:0/1959097100 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:15.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.182+0000 7fe681400640 1 -- 172.21.15.28:0/1959097100 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe67c0c3430 msgr2=0x7fe67c0aa790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:15.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.182+0000 7fe681400640 1 --2- 172.21.15.28:0/1959097100 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe67c0c3430 0x7fe67c0aa790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:15.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.182+0000 7fe681400640 1 -- 172.21.15.28:0/1959097100 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe67c0aca20 msgr2=0x7fe67c0aa250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:15.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.182+0000 7fe680a00640 1 --2- 172.21.15.28:0/1959097100 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe67c0aca20 0x7fe67c0aa250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:15.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.182+0000 7fe681400640 1 --2- 172.21.15.28:0/1959097100 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe67c0aca20 0x7fe67c0aa250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:15.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.182+0000 7fe681400640 1 -- 172.21.15.28:0/1959097100 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe67c0ac1e0 con 0x7fe67c072400 2024-02-21T23:07:15.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.182+0000 7fe680a00640 1 --2- 172.21.15.28:0/1959097100 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe67c0aca20 0x7fe67c0aa250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:07:15.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.183+0000 7fe681400640 1 --2- 172.21.15.28:0/1959097100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe67c072400 0x7fe67c154ab0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7fe67000cad0 tx=0x7fe67000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:15.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.183+0000 7fe679600640 1 -- 172.21.15.28:0/1959097100 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe670017070 con 0x7fe67c072400 2024-02-21T23:07:15.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.183+0000 7fe679600640 1 -- 172.21.15.28:0/1959097100 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe67000ac30 con 0x7fe67c072400 2024-02-21T23:07:15.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.183+0000 7fe679600640 1 -- 172.21.15.28:0/1959097100 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe670012400 con 0x7fe67c072400 2024-02-21T23:07:15.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.183+0000 7fe682800640 1 -- 172.21.15.28:0/1959097100 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe67c1edd30 con 0x7fe67c072400 2024-02-21T23:07:15.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.183+0000 7fe682800640 1 -- 172.21.15.28:0/1959097100 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe67c1ee2f0 con 0x7fe67c072400 2024-02-21T23:07:15.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.184+0000 7fe682800640 1 -- 172.21.15.28:0/1959097100 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe67c04f290 con 0x7fe67c072400 2024-02-21T23:07:15.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.185+0000 7fe679600640 1 -- 172.21.15.28:0/1959097100 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe67001a070 con 0x7fe67c072400 2024-02-21T23:07:15.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.186+0000 7fe679600640 1 --2- 172.21.15.28:0/1959097100 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe65c06b4d0 0x7fe65c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:15.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.186+0000 7fe679600640 1 -- 172.21.15.28:0/1959097100 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fe67008a1c0 con 0x7fe67c072400 2024-02-21T23:07:15.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.187+0000 7fe680a00640 1 --2- 172.21.15.28:0/1959097100 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe65c06b4d0 0x7fe65c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:15.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.188+0000 7fe680a00640 1 --2- 172.21.15.28:0/1959097100 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe65c06b4d0 0x7fe65c06d990 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fe668006fd0 tx=0x7fe668008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:15.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.191+0000 7fe679600640 1 -- 172.21.15.28:0/1959097100 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe670058e50 con 0x7fe67c072400 2024-02-21T23:07:15.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.395+0000 7fe682800640 1 -- 172.21.15.28:0/1959097100 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe67c1ee4c0 con 0x7fe67c072400 2024-02-21T23:07:15.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.398+0000 7fe679600640 1 -- 172.21.15.28:0/1959097100 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) v1 ==== 74+0+16942 (secure 0 0 0) 0x7fe670058750 con 0x7fe67c072400 2024-02-21T23:07:15.387 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:07:15.388 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":99,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:07:05.837927+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:06:52.151181+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-02-21T23:06:53.221776+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6835/2477712025":"2024-02-22T23:07:05.837870+0000","172.21.15.28:6834/2477712025":"2024-02-22T23:07:05.837870+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:07:15.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.401+0000 7fe682800640 1 -- 172.21.15.28:0/1959097100 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe65c06b4d0 msgr2=0x7fe65c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:15.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.401+0000 7fe682800640 1 --2- 172.21.15.28:0/1959097100 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe65c06b4d0 0x7fe65c06d990 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fe668006fd0 tx=0x7fe668008040 comp rx=0 tx=0).stop 2024-02-21T23:07:15.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.401+0000 7fe682800640 1 -- 172.21.15.28:0/1959097100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe67c072400 msgr2=0x7fe67c154ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:15.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.401+0000 7fe682800640 1 --2- 172.21.15.28:0/1959097100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe67c072400 0x7fe67c154ab0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7fe67000cad0 tx=0x7fe67000cf90 comp rx=0 tx=0).stop 2024-02-21T23:07:15.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.401+0000 7fe682800640 1 -- 172.21.15.28:0/1959097100 shutdown_connections 2024-02-21T23:07:15.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.401+0000 7fe682800640 1 --2- 172.21.15.28:0/1959097100 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe65c06b4d0 0x7fe65c06d990 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:15.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.401+0000 7fe682800640 1 --2- 172.21.15.28:0/1959097100 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe67c0c3430 0x7fe67c0aa790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:15.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.401+0000 7fe682800640 1 --2- 172.21.15.28:0/1959097100 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe67c0aca20 0x7fe67c0aa250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:15.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.401+0000 7fe682800640 1 --2- 172.21.15.28:0/1959097100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe67c072400 0x7fe67c154ab0 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:15.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.401+0000 7fe682800640 1 -- 172.21.15.28:0/1959097100 >> 172.21.15.28:0/1959097100 conn(0x7fe67c071910 msgr2=0x7fe67c06bbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:15.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.401+0000 7fe682800640 1 -- 172.21.15.28:0/1959097100 shutdown_connections 2024-02-21T23:07:15.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.401+0000 7fe682800640 1 -- 172.21.15.28:0/1959097100 wait complete. 2024-02-21T23:07:15.411 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-02-21T23:07:15.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.636+0000 7fb46ea00640 1 Processor -- start 2024-02-21T23:07:15.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.637+0000 7fb46ea00640 1 -- start start 2024-02-21T23:07:15.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.638+0000 7fb46ea00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb4680c00b0 0x7fb46806db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:15.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.638+0000 7fb46ea00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4681593c0 con 0x7fb4681527a0 2024-02-21T23:07:15.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.638+0000 7fb46ea00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4680c2070 con 0x7fb468158f70 2024-02-21T23:07:15.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.638+0000 7fb46ea00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb46806e790 con 0x7fb4680c00b0 2024-02-21T23:07:15.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.639+0000 7fb46e000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb4680c00b0 0x7fb46806db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:15.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.639+0000 7fb46d600640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb4681527a0 0x7fb4680bf9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59906/0 (socket says 172.21.15.28:59906) 2024-02-21T23:07:15.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.639+0000 7fb46d600640 1 -- 172.21.15.28:0/2972227752 learned_addr learned my addr 172.21.15.28:0/2972227752 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:07:15.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.639+0000 7fb46e000640 1 -- 172.21.15.28:0/2972227752 >> v1:172.21.15.81:6789/0 conn(0x7fb468158f70 legacy=0x7fb46806d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:15.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.639+0000 7fb46e000640 1 -- 172.21.15.28:0/2972227752 >> v1:172.21.15.28:6789/0 conn(0x7fb4681527a0 legacy=0x7fb4680bf9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:15.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.639+0000 7fb46e000640 1 -- 172.21.15.28:0/2972227752 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4680c2350 con 0x7fb4680c00b0 2024-02-21T23:07:15.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.641+0000 7fb46e000640 1 --2- 172.21.15.28:0/2972227752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4680c00b0 0x7fb46806db20 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7fb460009d90 tx=0x7fb46002bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dc01c60681002c0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:15.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.641+0000 7fb467e00640 1 -- 172.21.15.28:0/2972227752 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb460039070 con 0x7fb4680c00b0 2024-02-21T23:07:15.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.641+0000 7fb467e00640 1 -- 172.21.15.28:0/2972227752 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb46002cd80 con 0x7fb4680c00b0 2024-02-21T23:07:15.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.641+0000 7fb467e00640 1 -- 172.21.15.28:0/2972227752 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb460034660 con 0x7fb4680c00b0 2024-02-21T23:07:15.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.641+0000 7fb46ea00640 1 -- 172.21.15.28:0/2972227752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4680c00b0 msgr2=0x7fb46806db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:15.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.641+0000 7fb46ea00640 1 --2- 172.21.15.28:0/2972227752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4680c00b0 0x7fb46806db20 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7fb460009d90 tx=0x7fb46002bfd0 comp rx=0 tx=0).stop 2024-02-21T23:07:15.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.642+0000 7fb46ea00640 1 -- 172.21.15.28:0/2972227752 shutdown_connections 2024-02-21T23:07:15.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.642+0000 7fb46ea00640 1 --2- 172.21.15.28:0/2972227752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4680c00b0 0x7fb46806db20 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:15.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.642+0000 7fb46ea00640 1 -- 172.21.15.28:0/2972227752 >> 172.21.15.28:0/2972227752 conn(0x7fb4680a8320 msgr2=0x7fb4680a8730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:15.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.642+0000 7fb46ea00640 1 -- 172.21.15.28:0/2972227752 shutdown_connections 2024-02-21T23:07:15.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.642+0000 7fb46ea00640 1 -- 172.21.15.28:0/2972227752 wait complete. 2024-02-21T23:07:15.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.643+0000 7fb46ea00640 1 Processor -- start 2024-02-21T23:07:15.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.643+0000 7fb46ea00640 1 -- start start 2024-02-21T23:07:15.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.644+0000 7fb46ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4680c00b0 0x7fb4680c4340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:15.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.644+0000 7fb46ea00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb4681527a0 0x7fb46814f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:15.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.644+0000 7fb46ea00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb468158f70 0x7fb46814f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:15.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.644+0000 7fb46ea00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4680c04c0 con 0x7fb4680c00b0 2024-02-21T23:07:15.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.644+0000 7fb46ea00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4680c29a0 con 0x7fb4681527a0 2024-02-21T23:07:15.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.644+0000 7fb46ea00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb46806e060 con 0x7fb468158f70 2024-02-21T23:07:15.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.645+0000 7fb46d600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4680c00b0 0x7fb4680c4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:15.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.645+0000 7fb46d600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4680c00b0 0x7fb4680c4340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50792/0 (socket says 172.21.15.28:50792) 2024-02-21T23:07:15.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.645+0000 7fb46d600640 1 -- 172.21.15.28:0/3598685826 learned_addr learned my addr 172.21.15.28:0/3598685826 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:15.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.645+0000 7fb46d600640 1 -- 172.21.15.28:0/3598685826 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb468158f70 msgr2=0x7fb46814f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:15.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.645+0000 7fb46d600640 1 --2- 172.21.15.28:0/3598685826 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb468158f70 0x7fb46814f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:15.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.645+0000 7fb46cc00640 1 --2- 172.21.15.28:0/3598685826 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb4681527a0 0x7fb46814f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:15.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.645+0000 7fb46d600640 1 -- 172.21.15.28:0/3598685826 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb4681527a0 msgr2=0x7fb46814f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:15.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.645+0000 7fb46d600640 1 --2- 172.21.15.28:0/3598685826 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb4681527a0 0x7fb46814f110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:15.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.645+0000 7fb46d600640 1 -- 172.21.15.28:0/3598685826 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4681ee100 con 0x7fb4680c00b0 2024-02-21T23:07:15.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.645+0000 7fb46d600640 1 --2- 172.21.15.28:0/3598685826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4680c00b0 0x7fb4680c4340 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7fb45c00c4a0 tx=0x7fb45c00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:15.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.646+0000 7fb465600640 1 -- 172.21.15.28:0/3598685826 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb45c017070 con 0x7fb4680c00b0 2024-02-21T23:07:15.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.646+0000 7fb465600640 1 -- 172.21.15.28:0/3598685826 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb45c005ce0 con 0x7fb4680c00b0 2024-02-21T23:07:15.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.646+0000 7fb46ea00640 1 -- 172.21.15.28:0/3598685826 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4681ee3e0 con 0x7fb4680c00b0 2024-02-21T23:07:15.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.646+0000 7fb465600640 1 -- 172.21.15.28:0/3598685826 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb45c007270 con 0x7fb4680c00b0 2024-02-21T23:07:15.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.646+0000 7fb46ea00640 1 -- 172.21.15.28:0/3598685826 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4681ee9a0 con 0x7fb4680c00b0 2024-02-21T23:07:15.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.647+0000 7fb46ea00640 1 -- 172.21.15.28:0/3598685826 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4680c73f0 con 0x7fb4680c00b0 2024-02-21T23:07:15.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.649+0000 7fb465600640 1 -- 172.21.15.28:0/3598685826 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb45c01a070 con 0x7fb4680c00b0 2024-02-21T23:07:15.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.651+0000 7fb465600640 1 --2- 172.21.15.28:0/3598685826 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb44c06b500 0x7fb44c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:15.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.651+0000 7fb465600640 1 -- 172.21.15.28:0/3598685826 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fb45c0895b0 con 0x7fb4680c00b0 2024-02-21T23:07:15.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.651+0000 7fb46cc00640 1 --2- 172.21.15.28:0/3598685826 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb44c06b500 0x7fb44c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:15.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.652+0000 7fb46cc00640 1 --2- 172.21.15.28:0/3598685826 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb44c06b500 0x7fb44c06d9c0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fb4681ebc20 tx=0x7fb458009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:15.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.657+0000 7fb465600640 1 -- 172.21.15.28:0/3598685826 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb4680c73f0 con 0x7fb4680c00b0 2024-02-21T23:07:15.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.910+0000 7fb46ea00640 1 -- 172.21.15.28:0/3598685826 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fb46804f290 con 0x7fb4680c00b0 2024-02-21T23:07:15.901 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-02-21T23:07:15.912+0000 7f5ddc400640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-21T23:07:15.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.918+0000 7fb465600640 1 -- 172.21.15.28:0/3598685826 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v94) v1 ==== 161+0+0 (secure 0 0 0) 0x7fb45c057c90 con 0x7fb4680c00b0 2024-02-21T23:07:15.907 INFO:teuthology.orchestra.run.smithi028.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-02-21T23:07:15.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.921+0000 7fb46ea00640 1 -- 172.21.15.28:0/3598685826 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb44c06b500 msgr2=0x7fb44c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:15.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.921+0000 7fb46ea00640 1 --2- 172.21.15.28:0/3598685826 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb44c06b500 0x7fb44c06d9c0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fb4681ebc20 tx=0x7fb458009040 comp rx=0 tx=0).stop 2024-02-21T23:07:15.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.921+0000 7fb46ea00640 1 -- 172.21.15.28:0/3598685826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4680c00b0 msgr2=0x7fb4680c4340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:15.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.921+0000 7fb46ea00640 1 --2- 172.21.15.28:0/3598685826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4680c00b0 0x7fb4680c4340 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7fb45c00c4a0 tx=0x7fb45c00c960 comp rx=0 tx=0).stop 2024-02-21T23:07:15.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.921+0000 7fb46ea00640 1 -- 172.21.15.28:0/3598685826 shutdown_connections 2024-02-21T23:07:15.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.921+0000 7fb46ea00640 1 --2- 172.21.15.28:0/3598685826 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb44c06b500 0x7fb44c06d9c0 secure :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fb4681ebc20 tx=0x7fb458009040 comp rx=0 tx=0).stop 2024-02-21T23:07:15.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.921+0000 7fb46ea00640 1 --2- 172.21.15.28:0/3598685826 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb468158f70 0x7fb46814f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:15.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.921+0000 7fb46ea00640 1 --2- 172.21.15.28:0/3598685826 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb4681527a0 0x7fb46814f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:15.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.921+0000 7fb46ea00640 1 --2- 172.21.15.28:0/3598685826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4680c00b0 0x7fb4680c4340 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:15.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.921+0000 7fb46ea00640 1 -- 172.21.15.28:0/3598685826 >> 172.21.15.28:0/3598685826 conn(0x7fb4680a8320 msgr2=0x7fb4680bf9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:15.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.922+0000 7fb46ea00640 1 -- 172.21.15.28:0/3598685826 shutdown_connections 2024-02-21T23:07:15.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:15.922+0000 7fb46ea00640 1 -- 172.21.15.28:0/3598685826 wait complete. 2024-02-21T23:07:15.922 INFO:tasks.ceph.mds.c.smithi028.stderr: -254> 2024-02-21T23:07:09.693+0000 7fbe84c00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-21T23:07:15.930 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-02-21T23:07:15.985 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-02-21T23:07:16.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.134+0000 7f13fa600640 1 Processor -- start 2024-02-21T23:07:16.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.134+0000 7f13fa600640 1 -- start start 2024-02-21T23:07:16.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.136+0000 7f13fa600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f13f40c6220 0x7f13f41586b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:16.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.136+0000 7f13fa600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13f40c20d0 con 0x7f13f40c4960 2024-02-21T23:07:16.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.136+0000 7f13fa600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13f40c1290 con 0x7f13f4158bf0 2024-02-21T23:07:16.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.136+0000 7f13fa600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13f40a62e0 con 0x7f13f40c6220 2024-02-21T23:07:16.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.136+0000 7f13ebe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f13f40c6220 0x7f13f41586b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:16.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.136+0000 7f13f3400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f13f40c4960 0x7f13f41524f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59934/0 (socket says 172.21.15.28:59934) 2024-02-21T23:07:16.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.136+0000 7f13f3400640 1 -- 172.21.15.28:0/697951506 learned_addr learned my addr 172.21.15.28:0/697951506 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:07:16.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.137+0000 7f13ebe00640 1 -- 172.21.15.28:0/697951506 >> v1:172.21.15.81:6789/0 conn(0x7f13f4158bf0 legacy=0x7f13f415f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:16.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.137+0000 7f13ebe00640 1 -- 172.21.15.28:0/697951506 >> v1:172.21.15.28:6789/0 conn(0x7f13f40c4960 legacy=0x7f13f41524f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:16.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.137+0000 7f13ebe00640 1 -- 172.21.15.28:0/697951506 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13f40c1570 con 0x7f13f40c6220 2024-02-21T23:07:16.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.137+0000 7f13f2a00640 1 -- 172.21.15.28:0/697951506 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3899959382 0 0) 0x7f13f40c20d0 con 0x7f13f40c4960 2024-02-21T23:07:16.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.138+0000 7f13ebe00640 1 --2- 172.21.15.28:0/697951506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13f40c6220 0x7f13f41586b0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f13dc0097a0 tx=0x7f13dc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d9b24d392b2984f server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:16.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.138+0000 7f13f2a00640 1 -- 172.21.15.28:0/697951506 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13dc0315a0 con 0x7f13f40c6220 2024-02-21T23:07:16.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.138+0000 7f13f2a00640 1 -- 172.21.15.28:0/697951506 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13dc037070 con 0x7f13f40c6220 2024-02-21T23:07:16.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.138+0000 7f13f2a00640 1 -- 172.21.15.28:0/697951506 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13dc031960 con 0x7f13f40c6220 2024-02-21T23:07:16.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.139+0000 7f13fa600640 1 -- 172.21.15.28:0/697951506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13f40c6220 msgr2=0x7f13f41586b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:16.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.139+0000 7f13fa600640 1 --2- 172.21.15.28:0/697951506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13f40c6220 0x7f13f41586b0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f13dc0097a0 tx=0x7f13dc02b840 comp rx=0 tx=0).stop 2024-02-21T23:07:16.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.139+0000 7f13fa600640 1 -- 172.21.15.28:0/697951506 shutdown_connections 2024-02-21T23:07:16.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.139+0000 7f13fa600640 1 --2- 172.21.15.28:0/697951506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13f40c6220 0x7f13f41586b0 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:16.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.139+0000 7f13fa600640 1 -- 172.21.15.28:0/697951506 >> 172.21.15.28:0/697951506 conn(0x7f13f40a85a0 msgr2=0x7f13f40a8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:16.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.139+0000 7f13fa600640 1 -- 172.21.15.28:0/697951506 shutdown_connections 2024-02-21T23:07:16.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.139+0000 7f13fa600640 1 -- 172.21.15.28:0/697951506 wait complete. 2024-02-21T23:07:16.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.141+0000 7f13fa600640 1 Processor -- start 2024-02-21T23:07:16.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.141+0000 7f13fa600640 1 -- start start 2024-02-21T23:07:16.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.142+0000 7f13fa600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f13f40c4960 0x7f13f414ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:16.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.142+0000 7f13fa600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13f40c6220 0x7f13f41566d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:16.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.142+0000 7f13fa600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f13f4158bf0 0x7f13f40ab6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:16.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.142+0000 7f13fa600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13f40a5f20 con 0x7f13f40c6220 2024-02-21T23:07:16.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.142+0000 7f13fa600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13f40c1960 con 0x7f13f40c4960 2024-02-21T23:07:16.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.142+0000 7f13fa600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13f40a60a0 con 0x7f13f4158bf0 2024-02-21T23:07:16.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.143+0000 7f13ebe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13f40c6220 0x7f13f41566d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:16.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.143+0000 7f13ebe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13f40c6220 0x7f13f41566d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50822/0 (socket says 172.21.15.28:50822) 2024-02-21T23:07:16.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.143+0000 7f13ebe00640 1 -- 172.21.15.28:0/2220512565 learned_addr learned my addr 172.21.15.28:0/2220512565 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:16.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.143+0000 7f13f3e00640 1 --2- 172.21.15.28:0/2220512565 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f13f4158bf0 0x7f13f40ab6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:16.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.143+0000 7f13f3400640 1 --2- 172.21.15.28:0/2220512565 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f13f40c4960 0x7f13f414ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:16.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.143+0000 7f13ebe00640 1 -- 172.21.15.28:0/2220512565 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f13f4158bf0 msgr2=0x7f13f40ab6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:16.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.143+0000 7f13ebe00640 1 --2- 172.21.15.28:0/2220512565 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f13f4158bf0 0x7f13f40ab6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:16.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.143+0000 7f13ebe00640 1 -- 172.21.15.28:0/2220512565 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f13f40c4960 msgr2=0x7f13f414ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:16.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.143+0000 7f13ebe00640 1 --2- 172.21.15.28:0/2220512565 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f13f40c4960 0x7f13f414ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:16.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.143+0000 7f13ebe00640 1 -- 172.21.15.28:0/2220512565 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13f4157be0 con 0x7f13f40c6220 2024-02-21T23:07:16.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.143+0000 7f13f3e00640 1 --2- 172.21.15.28:0/2220512565 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f13f4158bf0 0x7f13f40ab6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:07:16.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.143+0000 7f13f3400640 1 --2- 172.21.15.28:0/2220512565 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f13f40c4960 0x7f13f414ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:07:16.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.144+0000 7f13ebe00640 1 --2- 172.21.15.28:0/2220512565 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13f40c6220 0x7f13f41566d0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f13dc0097a0 tx=0x7f13dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:16.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.144+0000 7f13eb400640 1 -- 172.21.15.28:0/2220512565 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13dc032840 con 0x7f13f40c6220 2024-02-21T23:07:16.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.144+0000 7f13eb400640 1 -- 172.21.15.28:0/2220512565 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13dc0329e0 con 0x7f13f40c6220 2024-02-21T23:07:16.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.144+0000 7f13eb400640 1 -- 172.21.15.28:0/2220512565 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13dc032dd0 con 0x7f13f40c6220 2024-02-21T23:07:16.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.144+0000 7f13fa600640 1 -- 172.21.15.28:0/2220512565 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13f40aac00 con 0x7f13f40c6220 2024-02-21T23:07:16.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.144+0000 7f13fa600640 1 -- 172.21.15.28:0/2220512565 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13f41588c0 con 0x7f13f40c6220 2024-02-21T23:07:16.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.145+0000 7f13fa600640 1 -- 172.21.15.28:0/2220512565 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13f4067f30 con 0x7f13f40c6220 2024-02-21T23:07:16.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.146+0000 7f13eb400640 1 -- 172.21.15.28:0/2220512565 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f13dc04d3f0 con 0x7f13f40c6220 2024-02-21T23:07:16.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.147+0000 7f13eb400640 1 --2- 172.21.15.28:0/2220512565 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f13d006b4d0 0x7f13d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:16.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.147+0000 7f13eb400640 1 -- 172.21.15.28:0/2220512565 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f13dc02f030 con 0x7f13f40c6220 2024-02-21T23:07:16.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.148+0000 7f13f3400640 1 --2- 172.21.15.28:0/2220512565 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f13d006b4d0 0x7f13d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:16.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.149+0000 7f13f3400640 1 --2- 172.21.15.28:0/2220512565 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f13d006b4d0 0x7f13d006d990 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f13e4000f60 tx=0x7f13e4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:16.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.151+0000 7f13eb400640 1 -- 172.21.15.28:0/2220512565 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f13dc040070 con 0x7f13f40c6220 2024-02-21T23:07:16.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.394+0000 7f13fa600640 1 -- 172.21.15.28:0/2220512565 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f13f40a8250 con 0x7f13f40c6220 2024-02-21T23:07:16.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.921+0000 7f13eb400640 1 -- 172.21.15.28:0/2220512565 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v95) v1 ==== 104+0+0 (secure 0 0 0) 0x7f13dc07acc0 con 0x7f13f40c6220 2024-02-21T23:07:16.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.924+0000 7f13fa600640 1 -- 172.21.15.28:0/2220512565 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f13d006b4d0 msgr2=0x7f13d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:16.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.924+0000 7f13fa600640 1 --2- 172.21.15.28:0/2220512565 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f13d006b4d0 0x7f13d006d990 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f13e4000f60 tx=0x7f13e4009040 comp rx=0 tx=0).stop 2024-02-21T23:07:16.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.924+0000 7f13fa600640 1 -- 172.21.15.28:0/2220512565 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13f40c6220 msgr2=0x7f13f41566d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:16.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.924+0000 7f13fa600640 1 --2- 172.21.15.28:0/2220512565 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13f40c6220 0x7f13f41566d0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f13dc0097a0 tx=0x7f13dc0046c0 comp rx=0 tx=0).stop 2024-02-21T23:07:16.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.924+0000 7f13fa600640 1 -- 172.21.15.28:0/2220512565 shutdown_connections 2024-02-21T23:07:16.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.924+0000 7f13fa600640 1 --2- 172.21.15.28:0/2220512565 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f13d006b4d0 0x7f13d006d990 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:16.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.924+0000 7f13fa600640 1 --2- 172.21.15.28:0/2220512565 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f13f4158bf0 0x7f13f40ab6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:16.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.924+0000 7f13fa600640 1 --2- 172.21.15.28:0/2220512565 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13f40c6220 0x7f13f41566d0 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:16.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.924+0000 7f13fa600640 1 --2- 172.21.15.28:0/2220512565 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f13f40c4960 0x7f13f414ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:16.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.924+0000 7f13fa600640 1 -- 172.21.15.28:0/2220512565 >> 172.21.15.28:0/2220512565 conn(0x7f13f40a85a0 msgr2=0x7f13f40a8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:16.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.924+0000 7f13fa600640 1 -- 172.21.15.28:0/2220512565 shutdown_connections 2024-02-21T23:07:16.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:16.925+0000 7f13fa600640 1 -- 172.21.15.28:0/2220512565 wait complete. 2024-02-21T23:07:16.938 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-02-21T23:07:17.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.172+0000 7f871d000640 1 Processor -- start 2024-02-21T23:07:17.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.172+0000 7f871d000640 1 -- start start 2024-02-21T23:07:17.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.173+0000 7f871d000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f871806ae70 0x7f871806b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:17.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.174+0000 7f871d000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87181527d0 con 0x7f8718155870 2024-02-21T23:07:17.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.174+0000 7f871d000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87180724f0 con 0x7f87180c2a90 2024-02-21T23:07:17.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.174+0000 7f871d000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87180727d0 con 0x7f871806ae70 2024-02-21T23:07:17.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.174+0000 7f8717e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f8718155870 0x7f871815c3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59948/0 (socket says 172.21.15.28:59948) 2024-02-21T23:07:17.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.174+0000 7f8717e00640 1 -- 172.21.15.28:0/2924308917 learned_addr learned my addr 172.21.15.28:0/2924308917 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:07:17.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.174+0000 7f8717400640 1 --2- 172.21.15.28:0/2924308917 >> v2:172.21.15.81:3300/0 conn(0x7f871806ae70 0x7f871806b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:17.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.175+0000 7f8716000640 1 -- 172.21.15.28:0/2924308917 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2298252862 0 0) 0x7f87181527d0 con 0x7f8718155870 2024-02-21T23:07:17.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.175+0000 7f8716000640 1 -- 172.21.15.28:0/2924308917 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86fc005730 con 0x7f8718155870 2024-02-21T23:07:17.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.175+0000 7f8716000640 1 -- 172.21.15.28:0/2924308917 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1909185318 0 0) 0x7f86fc005730 con 0x7f8718155870 2024-02-21T23:07:17.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.175+0000 7f8716000640 1 -- 172.21.15.28:0/2924308917 >> v2:172.21.15.81:3300/0 conn(0x7f871806ae70 msgr2=0x7f871806b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:17.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.175+0000 7f8716000640 1 --2- 172.21.15.28:0/2924308917 >> v2:172.21.15.81:3300/0 conn(0x7f871806ae70 0x7f871806b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:17.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.175+0000 7f8716000640 1 -- 172.21.15.28:0/2924308917 >> v1:172.21.15.81:6789/0 conn(0x7f87180c2a90 legacy=0x7f8718155160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:17.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.175+0000 7f8716000640 1 -- 172.21.15.28:0/2924308917 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f871806f720 con 0x7f8718155870 2024-02-21T23:07:17.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.175+0000 7f8716000640 1 -- 172.21.15.28:0/2924308917 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f8710001670 con 0x7f8718155870 2024-02-21T23:07:17.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.175+0000 7f8716000640 1 -- 172.21.15.28:0/2924308917 >> v1:172.21.15.28:6789/0 conn(0x7f8718155870 legacy=0x7f871815c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:17.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.175+0000 7f8716000640 1 --2- 172.21.15.28:0/2924308917 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86fc005bc0 0x7f86fc005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:17.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.175+0000 7f8716000640 1 --2- 172.21.15.28:0/2924308917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86fc006a10 0x7f86fc006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:17.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.175+0000 7f8716000640 1 --2- 172.21.15.28:0/2924308917 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f86fc0073d0 0x7f86fc00dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:17.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.175+0000 7f8716000640 1 -- 172.21.15.28:0/2924308917 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87180727d0 con 0x7f86fc006a10 2024-02-21T23:07:17.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.175+0000 7f8716000640 1 -- 172.21.15.28:0/2924308917 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86fc00e270 con 0x7f86fc005bc0 2024-02-21T23:07:17.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.175+0000 7f8716000640 1 -- 172.21.15.28:0/2924308917 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86fc00e470 con 0x7f86fc0073d0 2024-02-21T23:07:17.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.175+0000 7f8716a00640 1 --2- 172.21.15.28:0/2924308917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86fc006a10 0x7f86fc006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:17.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.176+0000 7f8717400640 1 --2- 172.21.15.28:0/2924308917 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86fc005bc0 0x7f86fc005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:17.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.176+0000 7f8717e00640 1 --2- 172.21.15.28:0/2924308917 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f86fc0073d0 0x7f86fc00dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:17.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.176+0000 7f8716a00640 1 -- 172.21.15.28:0/2924308917 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f86fc0073d0 msgr2=0x7f86fc00dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:17.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.176+0000 7f8716a00640 1 --2- 172.21.15.28:0/2924308917 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f86fc0073d0 0x7f86fc00dbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:17.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.176+0000 7f8716a00640 1 -- 172.21.15.28:0/2924308917 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86fc005bc0 msgr2=0x7f86fc005fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:17.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.176+0000 7f8716a00640 1 --2- 172.21.15.28:0/2924308917 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86fc005bc0 0x7f86fc005fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:17.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.176+0000 7f8716a00640 1 -- 172.21.15.28:0/2924308917 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86fc005730 con 0x7f86fc006a10 2024-02-21T23:07:17.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.176+0000 7f8717e00640 1 -- 172.21.15.28:0/2924308917 reap_dead start 2024-02-21T23:07:17.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.177+0000 7f8716a00640 1 --2- 172.21.15.28:0/2924308917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86fc006a10 0x7f86fc006e90 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f8700001750 tx=0x7f86fc007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cdc0690eef6782b server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:17.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.177+0000 7f8716000640 1 -- 172.21.15.28:0/2924308917 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8700001370 con 0x7f86fc006a10 2024-02-21T23:07:17.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.177+0000 7f8716000640 1 -- 172.21.15.28:0/2924308917 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87000052d0 con 0x7f86fc006a10 2024-02-21T23:07:17.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.177+0000 7f8716000640 1 -- 172.21.15.28:0/2924308917 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8700003830 con 0x7f86fc006a10 2024-02-21T23:07:17.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.178+0000 7f871d000640 1 -- 172.21.15.28:0/2924308917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86fc006a10 msgr2=0x7f86fc006e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:17.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.178+0000 7f871d000640 1 --2- 172.21.15.28:0/2924308917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86fc006a10 0x7f86fc006e90 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f8700001750 tx=0x7f86fc007a40 comp rx=0 tx=0).stop 2024-02-21T23:07:17.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.178+0000 7f871d000640 1 -- 172.21.15.28:0/2924308917 shutdown_connections 2024-02-21T23:07:17.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.178+0000 7f871d000640 1 --2- 172.21.15.28:0/2924308917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86fc006a10 0x7f86fc006e90 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:17.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.178+0000 7f871d000640 1 -- 172.21.15.28:0/2924308917 >> 172.21.15.28:0/2924308917 conn(0x7f87180c6610 msgr2=0x7f87180c69f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:17.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.178+0000 7f871d000640 1 -- 172.21.15.28:0/2924308917 shutdown_connections 2024-02-21T23:07:17.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.178+0000 7f871d000640 1 -- 172.21.15.28:0/2924308917 wait complete. 2024-02-21T23:07:17.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.180+0000 7f871d000640 1 Processor -- start 2024-02-21T23:07:17.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.180+0000 7f871d000640 1 -- start start 2024-02-21T23:07:17.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.181+0000 7f871d000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f86fc006a10 0x7f8718135160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:17.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.181+0000 7f871d000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8718142180 0x7f87181451c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:17.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.181+0000 7f871d000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8718145700 0x7f87181eaaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:17.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.181+0000 7f871d000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8718155d00 con 0x7f8718142180 2024-02-21T23:07:17.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.181+0000 7f871d000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87180c2240 con 0x7f8718145700 2024-02-21T23:07:17.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.181+0000 7f871d000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8718147ae0 con 0x7f86fc006a10 2024-02-21T23:07:17.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.181+0000 7f8716a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8718142180 0x7f87181451c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:17.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.181+0000 7f8716a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8718142180 0x7f87181451c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50840/0 (socket says 172.21.15.28:50840) 2024-02-21T23:07:17.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.182+0000 7f8716a00640 1 -- 172.21.15.28:0/3276172406 learned_addr learned my addr 172.21.15.28:0/3276172406 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:17.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.182+0000 7f8717400640 1 --2- 172.21.15.28:0/3276172406 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f86fc006a10 0x7f8718135160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:17.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.182+0000 7f8717e00640 1 --2- 172.21.15.28:0/3276172406 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8718145700 0x7f87181eaaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:17.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.182+0000 7f8716a00640 1 -- 172.21.15.28:0/3276172406 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f86fc006a10 msgr2=0x7f8718135160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:17.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.182+0000 7f8716a00640 1 --2- 172.21.15.28:0/3276172406 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f86fc006a10 0x7f8718135160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:17.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.182+0000 7f8716a00640 1 -- 172.21.15.28:0/3276172406 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8718145700 msgr2=0x7f87181eaaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:17.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.182+0000 7f8716a00640 1 --2- 172.21.15.28:0/3276172406 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8718145700 0x7f87181eaaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:17.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.182+0000 7f8716a00640 1 -- 172.21.15.28:0/3276172406 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87180aa810 con 0x7f8718142180 2024-02-21T23:07:17.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.182+0000 7f8716a00640 1 --2- 172.21.15.28:0/3276172406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8718142180 0x7f87181451c0 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f8700000c00 tx=0x7f8700001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:17.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.183+0000 7f870f400640 1 -- 172.21.15.28:0/3276172406 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8700041020 con 0x7f8718142180 2024-02-21T23:07:17.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.183+0000 7f870f400640 1 -- 172.21.15.28:0/3276172406 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8700005480 con 0x7f8718142180 2024-02-21T23:07:17.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.183+0000 7f870f400640 1 -- 172.21.15.28:0/3276172406 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8700007a30 con 0x7f8718142180 2024-02-21T23:07:17.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.183+0000 7f871d000640 1 -- 172.21.15.28:0/3276172406 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87180c4200 con 0x7f8718142180 2024-02-21T23:07:17.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.183+0000 7f871d000640 1 -- 172.21.15.28:0/3276172406 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8718147f70 con 0x7f8718142180 2024-02-21T23:07:17.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.184+0000 7f871d000640 1 -- 172.21.15.28:0/3276172406 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8718147c60 con 0x7f8718142180 2024-02-21T23:07:17.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.185+0000 7f870f400640 1 -- 172.21.15.28:0/3276172406 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8700044020 con 0x7f8718142180 2024-02-21T23:07:17.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.186+0000 7f870f400640 1 --2- 172.21.15.28:0/3276172406 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f86f406b500 0x7f86f406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:17.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.186+0000 7f870f400640 1 -- 172.21.15.28:0/3276172406 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f87000abee0 con 0x7f8718142180 2024-02-21T23:07:17.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.187+0000 7f8717400640 1 --2- 172.21.15.28:0/3276172406 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f86f406b500 0x7f86f406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:17.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.188+0000 7f8717400640 1 --2- 172.21.15.28:0/3276172406 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f86f406b500 0x7f86f406d9c0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f8708007520 tx=0x7f870800b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:17.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.190+0000 7f870f400640 1 -- 172.21.15.28:0/3276172406 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f870004b030 con 0x7f8718142180 2024-02-21T23:07:17.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.389+0000 7f871d000640 1 -- 172.21.15.28:0/3276172406 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f871804f290 con 0x7f8718142180 2024-02-21T23:07:17.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.931+0000 7f870f400640 1 -- 172.21.15.28:0/3276172406 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v101) v1 ==== 181+0+0 (secure 0 0 0) 0x7f87000303c0 con 0x7f8718142180 2024-02-21T23:07:17.921 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' removed 2024-02-21T23:07:17.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.937+0000 7f871d000640 1 -- 172.21.15.28:0/3276172406 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f86f406b500 msgr2=0x7f86f406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:17.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.937+0000 7f871d000640 1 --2- 172.21.15.28:0/3276172406 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f86f406b500 0x7f86f406d9c0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f8708007520 tx=0x7f870800b040 comp rx=0 tx=0).stop 2024-02-21T23:07:17.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.937+0000 7f871d000640 1 -- 172.21.15.28:0/3276172406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8718142180 msgr2=0x7f87181451c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:17.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.937+0000 7f871d000640 1 --2- 172.21.15.28:0/3276172406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8718142180 0x7f87181451c0 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f8700000c00 tx=0x7f8700001120 comp rx=0 tx=0).stop 2024-02-21T23:07:17.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.940+0000 7f871d000640 1 -- 172.21.15.28:0/3276172406 shutdown_connections 2024-02-21T23:07:17.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.940+0000 7f871d000640 1 --2- 172.21.15.28:0/3276172406 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f86f406b500 0x7f86f406d9c0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:17.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.940+0000 7f871d000640 1 --2- 172.21.15.28:0/3276172406 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8718145700 0x7f87181eaaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:17.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.940+0000 7f871d000640 1 --2- 172.21.15.28:0/3276172406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8718142180 0x7f87181451c0 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:17.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.940+0000 7f871d000640 1 --2- 172.21.15.28:0/3276172406 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f86fc006a10 0x7f8718135160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:17.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.940+0000 7f871d000640 1 -- 172.21.15.28:0/3276172406 >> 172.21.15.28:0/3276172406 conn(0x7f87180c6610 msgr2=0x7f87180c69f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:17.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.941+0000 7f871d000640 1 -- 172.21.15.28:0/3276172406 shutdown_connections 2024-02-21T23:07:17.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:17.941+0000 7f871d000640 1 -- 172.21.15.28:0/3276172406 wait complete. 2024-02-21T23:07:17.952 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-02-21T23:07:18.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.186+0000 7fac59a00640 1 Processor -- start 2024-02-21T23:07:18.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.186+0000 7fac59a00640 1 -- start start 2024-02-21T23:07:18.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.187+0000 7fac59a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fac540aa480 0x7fac540c1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:18.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.188+0000 7fac59a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fac540c36a0 0x7fac54152410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:18.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.188+0000 7fac59a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac540ab0f0 con 0x7fac540c7370 2024-02-21T23:07:18.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.188+0000 7fac59a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac54155150 con 0x7fac540c36a0 2024-02-21T23:07:18.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.188+0000 7fac59a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac54156040 con 0x7fac540aa480 2024-02-21T23:07:18.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.188+0000 7fac4b400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fac540c7370 0x7fac54155930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59962/0 (socket says 172.21.15.28:59962) 2024-02-21T23:07:18.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.188+0000 7fac4b400640 1 -- 172.21.15.28:0/2343088920 learned_addr learned my addr 172.21.15.28:0/2343088920 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:07:18.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.188+0000 7fac53e00640 1 --2- 172.21.15.28:0/2343088920 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fac540aa480 0x7fac540c1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:18.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.188+0000 7fac59000640 1 --2- 172.21.15.28:0/2343088920 >> v2:172.21.15.81:3300/0 conn(0x7fac540c36a0 0x7fac54152410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:18.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.188+0000 7fac53400640 1 -- 172.21.15.28:0/2343088920 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2327199229 0 0) 0x7fac540ab0f0 con 0x7fac540c7370 2024-02-21T23:07:18.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.188+0000 7fac53400640 1 -- 172.21.15.28:0/2343088920 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac38003520 con 0x7fac540c7370 2024-02-21T23:07:18.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.189+0000 7fac53400640 1 -- 172.21.15.28:0/2343088920 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2089847775 0 0) 0x7fac38003520 con 0x7fac540c7370 2024-02-21T23:07:18.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.189+0000 7fac53400640 1 -- 172.21.15.28:0/2343088920 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fac540aa480 msgr2=0x7fac540c1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:18.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.189+0000 7fac53400640 1 --2- 172.21.15.28:0/2343088920 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fac540aa480 0x7fac540c1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:18.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.189+0000 7fac53400640 1 -- 172.21.15.28:0/2343088920 >> v2:172.21.15.81:3300/0 conn(0x7fac540c36a0 msgr2=0x7fac54152410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:18.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.189+0000 7fac53400640 1 --2- 172.21.15.28:0/2343088920 >> v2:172.21.15.81:3300/0 conn(0x7fac540c36a0 0x7fac54152410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:18.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.189+0000 7fac53400640 1 -- 172.21.15.28:0/2343088920 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac540c3c60 con 0x7fac540c7370 2024-02-21T23:07:18.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.189+0000 7fac53400640 1 -- 172.21.15.28:0/2343088920 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fac3c001670 con 0x7fac540c7370 2024-02-21T23:07:18.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.189+0000 7fac53400640 1 -- 172.21.15.28:0/2343088920 >> v1:172.21.15.28:6789/0 conn(0x7fac540c7370 legacy=0x7fac54155930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:18.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.189+0000 7fac53400640 1 --2- 172.21.15.28:0/2343088920 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fac540aa480 0x7fac380039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:18.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.189+0000 7fac53400640 1 --2- 172.21.15.28:0/2343088920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac38003f30 0x7fac38004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:18.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.189+0000 7fac53400640 1 --2- 172.21.15.28:0/2343088920 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac38004d40 0x7fac380051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:18.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.189+0000 7fac53400640 1 -- 172.21.15.28:0/2343088920 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac54155150 con 0x7fac38003f30 2024-02-21T23:07:18.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.189+0000 7fac53400640 1 -- 172.21.15.28:0/2343088920 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac54156040 con 0x7fac38004d40 2024-02-21T23:07:18.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.189+0000 7fac53400640 1 -- 172.21.15.28:0/2343088920 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac3800baa0 con 0x7fac540aa480 2024-02-21T23:07:18.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.189+0000 7fac4b400640 1 --2- 172.21.15.28:0/2343088920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac38003f30 0x7fac38004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:18.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.190+0000 7fac59000640 1 --2- 172.21.15.28:0/2343088920 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac38004d40 0x7fac380051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:18.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.190+0000 7fac53e00640 1 --2- 172.21.15.28:0/2343088920 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fac540aa480 0x7fac380039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:18.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.190+0000 7fac4b400640 1 -- 172.21.15.28:0/2343088920 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fac540aa480 msgr2=0x7fac380039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:18.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.190+0000 7fac4b400640 1 --2- 172.21.15.28:0/2343088920 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fac540aa480 0x7fac380039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:18.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.190+0000 7fac4b400640 1 -- 172.21.15.28:0/2343088920 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac38004d40 msgr2=0x7fac380051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:18.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.190+0000 7fac4b400640 1 --2- 172.21.15.28:0/2343088920 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac38004d40 0x7fac380051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:18.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.190+0000 7fac4b400640 1 -- 172.21.15.28:0/2343088920 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac38003520 con 0x7fac38003f30 2024-02-21T23:07:18.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.190+0000 7fac53e00640 1 --2- 172.21.15.28:0/2343088920 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fac540aa480 0x7fac380039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:07:18.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.191+0000 7fac4b400640 1 --2- 172.21.15.28:0/2343088920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac38003f30 0x7fac38004800 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7fac3c00c210 tx=0x7fac38005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d01bf701f77ed289 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:18.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.191+0000 7fac53400640 1 -- 172.21.15.28:0/2343088920 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac3c007d50 con 0x7fac38003f30 2024-02-21T23:07:18.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.191+0000 7fac53400640 1 -- 172.21.15.28:0/2343088920 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac3c030070 con 0x7fac38003f30 2024-02-21T23:07:18.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.191+0000 7fac53400640 1 -- 172.21.15.28:0/2343088920 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac3c02cdc0 con 0x7fac38003f30 2024-02-21T23:07:18.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.192+0000 7fac59a00640 1 -- 172.21.15.28:0/2343088920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac38003f30 msgr2=0x7fac38004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:18.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.192+0000 7fac59a00640 1 --2- 172.21.15.28:0/2343088920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac38003f30 0x7fac38004800 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7fac3c00c210 tx=0x7fac38005830 comp rx=0 tx=0).stop 2024-02-21T23:07:18.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.192+0000 7fac59000640 1 -- 172.21.15.28:0/2343088920 reap_dead start 2024-02-21T23:07:18.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.192+0000 7fac59a00640 1 -- 172.21.15.28:0/2343088920 shutdown_connections 2024-02-21T23:07:18.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.192+0000 7fac59a00640 1 -- 172.21.15.28:0/2343088920 >> 172.21.15.28:0/2343088920 conn(0x7fac540c85b0 msgr2=0x7fac540c8990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:18.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.192+0000 7fac59a00640 1 -- 172.21.15.28:0/2343088920 shutdown_connections 2024-02-21T23:07:18.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.192+0000 7fac59a00640 1 -- 172.21.15.28:0/2343088920 wait complete. 2024-02-21T23:07:18.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.194+0000 7fac59a00640 1 Processor -- start 2024-02-21T23:07:18.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.194+0000 7fac59a00640 1 -- start start 2024-02-21T23:07:18.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.195+0000 7fac59a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fac38003f30 0x7fac5414d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:18.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.195+0000 7fac59a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac54099e00 0x7fac540a8940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:18.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.195+0000 7fac59a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac540a8e80 0x7fac54150450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:18.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.195+0000 7fac59a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac54155410 con 0x7fac54099e00 2024-02-21T23:07:18.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.195+0000 7fac59a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac540c4220 con 0x7fac540a8e80 2024-02-21T23:07:18.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.195+0000 7fac59a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac540aa860 con 0x7fac38003f30 2024-02-21T23:07:18.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.195+0000 7fac4b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac54099e00 0x7fac540a8940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:18.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.195+0000 7fac4b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac54099e00 0x7fac540a8940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50860/0 (socket says 172.21.15.28:50860) 2024-02-21T23:07:18.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.195+0000 7fac4b400640 1 -- 172.21.15.28:0/4140251019 learned_addr learned my addr 172.21.15.28:0/4140251019 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:18.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.196+0000 7fac53e00640 1 --2- 172.21.15.28:0/4140251019 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fac38003f30 0x7fac5414d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:18.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.196+0000 7fac59000640 1 --2- 172.21.15.28:0/4140251019 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac540a8e80 0x7fac54150450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:18.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.196+0000 7fac4b400640 1 -- 172.21.15.28:0/4140251019 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fac38003f30 msgr2=0x7fac5414d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:18.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.196+0000 7fac4b400640 1 --2- 172.21.15.28:0/4140251019 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fac38003f30 0x7fac5414d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:18.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.196+0000 7fac4b400640 1 -- 172.21.15.28:0/4140251019 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac540a8e80 msgr2=0x7fac54150450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:18.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.196+0000 7fac4b400640 1 --2- 172.21.15.28:0/4140251019 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac540a8e80 0x7fac54150450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:18.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.196+0000 7fac4b400640 1 -- 172.21.15.28:0/4140251019 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac5409b240 con 0x7fac54099e00 2024-02-21T23:07:18.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.196+0000 7fac53e00640 1 --2- 172.21.15.28:0/4140251019 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fac38003f30 0x7fac5414d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:07:18.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.196+0000 7fac4b400640 1 --2- 172.21.15.28:0/4140251019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac54099e00 0x7fac540a8940 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fac3c001970 tx=0x7fac3c001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:18.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.196+0000 7fac50c00640 1 -- 172.21.15.28:0/4140251019 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac3c045070 con 0x7fac54099e00 2024-02-21T23:07:18.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.197+0000 7fac50c00640 1 -- 172.21.15.28:0/4140251019 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac3c004bf0 con 0x7fac54099e00 2024-02-21T23:07:18.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.197+0000 7fac59a00640 1 -- 172.21.15.28:0/4140251019 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac541518a0 con 0x7fac54099e00 2024-02-21T23:07:18.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.197+0000 7fac50c00640 1 -- 172.21.15.28:0/4140251019 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac3c007d50 con 0x7fac54099e00 2024-02-21T23:07:18.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.197+0000 7fac59a00640 1 -- 172.21.15.28:0/4140251019 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac5409ad20 con 0x7fac54099e00 2024-02-21T23:07:18.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.198+0000 7fac59a00640 1 -- 172.21.15.28:0/4140251019 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac5404f290 con 0x7fac54099e00 2024-02-21T23:07:18.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.199+0000 7fac50c00640 1 -- 172.21.15.28:0/4140251019 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fac3c007070 con 0x7fac54099e00 2024-02-21T23:07:18.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.199+0000 7fac50c00640 1 --2- 172.21.15.28:0/4140251019 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fac3006b500 0x7fac3006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:18.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.200+0000 7fac50c00640 1 -- 172.21.15.28:0/4140251019 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5835+0+0 (secure 0 0 0) 0x7fac3c0ab2e0 con 0x7fac54099e00 2024-02-21T23:07:18.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.200+0000 7fac53e00640 1 --2- 172.21.15.28:0/4140251019 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fac3006b500 0x7fac3006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:18.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.201+0000 7fac53e00640 1 --2- 172.21.15.28:0/4140251019 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fac3006b500 0x7fac3006d9c0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fac540ae1d0 tx=0x7fac4400d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:18.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.205+0000 7fac50c00640 1 -- 172.21.15.28:0/4140251019 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fac3c07a150 con 0x7fac54099e00 2024-02-21T23:07:18.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.403+0000 7fac59a00640 1 -- 172.21.15.28:0/4140251019 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fac540c7f30 con 0x7fac54099e00 2024-02-21T23:07:18.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.935+0000 7fac50c00640 1 -- 172.21.15.28:0/4140251019 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v102) v1 ==== 169+0+0 (secure 0 0 0) 0x7fac3c079a50 con 0x7fac54099e00 2024-02-21T23:07:18.924 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' removed 2024-02-21T23:07:18.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.940+0000 7fac59a00640 1 -- 172.21.15.28:0/4140251019 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fac3006b500 msgr2=0x7fac3006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:18.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.940+0000 7fac59a00640 1 --2- 172.21.15.28:0/4140251019 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fac3006b500 0x7fac3006d9c0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fac540ae1d0 tx=0x7fac4400d040 comp rx=0 tx=0).stop 2024-02-21T23:07:18.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.940+0000 7fac59a00640 1 -- 172.21.15.28:0/4140251019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac54099e00 msgr2=0x7fac540a8940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:18.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.940+0000 7fac59a00640 1 --2- 172.21.15.28:0/4140251019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac54099e00 0x7fac540a8940 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fac3c001970 tx=0x7fac3c001a40 comp rx=0 tx=0).stop 2024-02-21T23:07:18.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.941+0000 7fac59a00640 1 -- 172.21.15.28:0/4140251019 shutdown_connections 2024-02-21T23:07:18.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.941+0000 7fac59a00640 1 --2- 172.21.15.28:0/4140251019 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fac3006b500 0x7fac3006d9c0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:18.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.941+0000 7fac59a00640 1 --2- 172.21.15.28:0/4140251019 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac540a8e80 0x7fac54150450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:18.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.941+0000 7fac59a00640 1 --2- 172.21.15.28:0/4140251019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac54099e00 0x7fac540a8940 secure :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fac3c001970 tx=0x7fac3c001a40 comp rx=0 tx=0).stop 2024-02-21T23:07:18.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.941+0000 7fac59a00640 1 --2- 172.21.15.28:0/4140251019 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fac38003f30 0x7fac5414d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:18.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.941+0000 7fac59a00640 1 -- 172.21.15.28:0/4140251019 >> 172.21.15.28:0/4140251019 conn(0x7fac540c85b0 msgr2=0x7fac541529c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:18.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.944+0000 7fac59a00640 1 -- 172.21.15.28:0/4140251019 shutdown_connections 2024-02-21T23:07:18.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:18.944+0000 7fac59a00640 1 -- 172.21.15.28:0/4140251019 wait complete. 2024-02-21T23:07:18.960 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.c mds_join_fs 2024-02-21T23:07:19.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.178+0000 7f47a7200640 1 Processor -- start 2024-02-21T23:07:19.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.178+0000 7f47a7200640 1 -- start start 2024-02-21T23:07:19.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.180+0000 7f47a7200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f47a813e520 0x7f47a806c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:19.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.180+0000 7f47a7200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47a8158550 con 0x7f47a806dbc0 2024-02-21T23:07:19.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.180+0000 7f47a7200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47a81587d0 con 0x7f47a813e0c0 2024-02-21T23:07:19.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.180+0000 7f47a7200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47a80c89a0 con 0x7f47a813e520 2024-02-21T23:07:19.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.180+0000 7f479d400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f47a806dbc0 0x7f47a806c280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59974/0 (socket says 172.21.15.28:59974) 2024-02-21T23:07:19.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.180+0000 7f47a6800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f47a813e520 0x7f47a806c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:19.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.180+0000 7f479d400640 1 -- 172.21.15.28:0/934327192 learned_addr learned my addr 172.21.15.28:0/934327192 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:07:19.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.181+0000 7f47a6800640 1 -- 172.21.15.28:0/934327192 >> v1:172.21.15.81:6789/0 conn(0x7f47a813e0c0 legacy=0x7f47a80c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:19.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.181+0000 7f47a6800640 1 -- 172.21.15.28:0/934327192 >> v1:172.21.15.28:6789/0 conn(0x7f47a806dbc0 legacy=0x7f47a806c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:19.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.181+0000 7f47a6800640 1 -- 172.21.15.28:0/934327192 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47a8158ab0 con 0x7f47a813e520 2024-02-21T23:07:19.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.181+0000 7f47a5400640 1 -- 172.21.15.28:0/934327192 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3922552556 0 0) 0x7f4790001110 con 0x7f47a806dbc0 2024-02-21T23:07:19.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.182+0000 7f47a6800640 1 --2- 172.21.15.28:0/934327192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47a813e520 0x7f47a806c990 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f47a00097a0 tx=0x7f47a002b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18e390bda4d96c1b server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:19.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.182+0000 7f47a5400640 1 -- 172.21.15.28:0/934327192 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47a0038070 con 0x7f47a813e520 2024-02-21T23:07:19.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.183+0000 7f47a5400640 1 -- 172.21.15.28:0/934327192 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47a0031700 con 0x7f47a813e520 2024-02-21T23:07:19.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.183+0000 7f47a5400640 1 -- 172.21.15.28:0/934327192 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47a0031ad0 con 0x7f47a813e520 2024-02-21T23:07:19.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.183+0000 7f47a7200640 1 -- 172.21.15.28:0/934327192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47a813e520 msgr2=0x7f47a806c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:19.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.183+0000 7f47a7200640 1 --2- 172.21.15.28:0/934327192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47a813e520 0x7f47a806c990 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f47a00097a0 tx=0x7f47a002b7a0 comp rx=0 tx=0).stop 2024-02-21T23:07:19.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.183+0000 7f47a7200640 1 -- 172.21.15.28:0/934327192 shutdown_connections 2024-02-21T23:07:19.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.183+0000 7f47a7200640 1 --2- 172.21.15.28:0/934327192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47a813e520 0x7f47a806c990 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:19.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.183+0000 7f47a7200640 1 -- 172.21.15.28:0/934327192 >> 172.21.15.28:0/934327192 conn(0x7f47a8073110 msgr2=0x7f47a80ac490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:19.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.183+0000 7f47a7200640 1 -- 172.21.15.28:0/934327192 shutdown_connections 2024-02-21T23:07:19.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.183+0000 7f47a7200640 1 -- 172.21.15.28:0/934327192 wait complete. 2024-02-21T23:07:19.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.185+0000 7f47a7200640 1 Processor -- start 2024-02-21T23:07:19.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.185+0000 7f47a7200640 1 -- start start 2024-02-21T23:07:19.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.186+0000 7f47a7200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47a806dbc0 0x7f47a80afee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:19.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.186+0000 7f47a7200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47a813e0c0 0x7f47a80ae390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:19.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.186+0000 7f47a7200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f47a813e520 0x7f47a80af220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:19.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.186+0000 7f47a7200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a806ced0 con 0x7f47a806dbc0 2024-02-21T23:07:19.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.186+0000 7f47a7200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a8159070 con 0x7f47a813e0c0 2024-02-21T23:07:19.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.186+0000 7f47a7200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47a80c78c0 con 0x7f47a813e520 2024-02-21T23:07:19.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.187+0000 7f47a5e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47a806dbc0 0x7f47a80afee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:19.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.187+0000 7f47a5e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47a806dbc0 0x7f47a80afee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50876/0 (socket says 172.21.15.28:50876) 2024-02-21T23:07:19.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.187+0000 7f47a5e00640 1 -- 172.21.15.28:0/3034627910 learned_addr learned my addr 172.21.15.28:0/3034627910 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:19.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.187+0000 7f479d400640 1 --2- 172.21.15.28:0/3034627910 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47a813e0c0 0x7f47a80ae390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:19.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.187+0000 7f47a6800640 1 --2- 172.21.15.28:0/3034627910 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f47a813e520 0x7f47a80af220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:19.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.187+0000 7f47a5e00640 1 -- 172.21.15.28:0/3034627910 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f47a813e520 msgr2=0x7f47a80af220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:19.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.187+0000 7f47a5e00640 1 --2- 172.21.15.28:0/3034627910 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f47a813e520 0x7f47a80af220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:19.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.187+0000 7f47a5e00640 1 -- 172.21.15.28:0/3034627910 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47a813e0c0 msgr2=0x7f47a80ae390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:19.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.187+0000 7f47a5e00640 1 --2- 172.21.15.28:0/3034627910 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47a813e0c0 0x7f47a80ae390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:19.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.187+0000 7f47a5e00640 1 -- 172.21.15.28:0/3034627910 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47a81ead20 con 0x7f47a806dbc0 2024-02-21T23:07:19.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.187+0000 7f47a5e00640 1 --2- 172.21.15.28:0/3034627910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47a806dbc0 0x7f47a80afee0 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f479800ba50 tx=0x7f479800bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:19.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.188+0000 7f479ea00640 1 -- 172.21.15.28:0/3034627910 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4798019070 con 0x7f47a806dbc0 2024-02-21T23:07:19.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.188+0000 7f479ea00640 1 -- 172.21.15.28:0/3034627910 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f479800cc20 con 0x7f47a806dbc0 2024-02-21T23:07:19.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.188+0000 7f479ea00640 1 -- 172.21.15.28:0/3034627910 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4798014440 con 0x7f47a806dbc0 2024-02-21T23:07:19.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.188+0000 7f47a7200640 1 -- 172.21.15.28:0/3034627910 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47a81eafe0 con 0x7f47a806dbc0 2024-02-21T23:07:19.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.188+0000 7f47a7200640 1 -- 172.21.15.28:0/3034627910 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47a81eb5a0 con 0x7f47a806dbc0 2024-02-21T23:07:19.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.189+0000 7f47a7200640 1 -- 172.21.15.28:0/3034627910 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47a804f290 con 0x7f47a806dbc0 2024-02-21T23:07:19.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.190+0000 7f479ea00640 1 -- 172.21.15.28:0/3034627910 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f479801c070 con 0x7f47a806dbc0 2024-02-21T23:07:19.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.191+0000 7f479ea00640 1 --2- 172.21.15.28:0/3034627910 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f478406b4d0 0x7f478406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:19.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.191+0000 7f479ea00640 1 -- 172.21.15.28:0/3034627910 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f4798089de0 con 0x7f47a806dbc0 2024-02-21T23:07:19.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.192+0000 7f479d400640 1 --2- 172.21.15.28:0/3034627910 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f478406b4d0 0x7f478406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:19.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.193+0000 7f479d400640 1 --2- 172.21.15.28:0/3034627910 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f478406b4d0 0x7f478406d990 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f47a80b0420 tx=0x7f4790007c00 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:19.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.196+0000 7f479ea00640 1 -- 172.21.15.28:0/3034627910 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4798058690 con 0x7f47a806dbc0 2024-02-21T23:07:19.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.396+0000 7f47a7200640 1 -- 172.21.15.28:0/3034627910 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.c", "name": "mds_join_fs"} v 0) v1 -- 0x7f47a813cf70 con 0x7f47a806dbc0 2024-02-21T23:07:19.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.401+0000 7f479ea00640 1 -- 172.21.15.28:0/3034627910 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.c", "name": "mds_join_fs"}]=0 v33) v1 ==== 96+0+0 (secure 0 0 0) 0x7f479805c730 con 0x7f47a806dbc0 2024-02-21T23:07:19.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.404+0000 7f47a7200640 1 -- 172.21.15.28:0/3034627910 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f478406b4d0 msgr2=0x7f478406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:19.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.404+0000 7f47a7200640 1 --2- 172.21.15.28:0/3034627910 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f478406b4d0 0x7f478406d990 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f47a80b0420 tx=0x7f4790007c00 comp rx=0 tx=0).stop 2024-02-21T23:07:19.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.404+0000 7f47a7200640 1 -- 172.21.15.28:0/3034627910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47a806dbc0 msgr2=0x7f47a80afee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:19.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.404+0000 7f47a7200640 1 --2- 172.21.15.28:0/3034627910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47a806dbc0 0x7f47a80afee0 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f479800ba50 tx=0x7f479800bf10 comp rx=0 tx=0).stop 2024-02-21T23:07:19.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.405+0000 7f47a7200640 1 -- 172.21.15.28:0/3034627910 shutdown_connections 2024-02-21T23:07:19.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.405+0000 7f47a7200640 1 --2- 172.21.15.28:0/3034627910 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f478406b4d0 0x7f478406d990 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:19.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.405+0000 7f47a7200640 1 --2- 172.21.15.28:0/3034627910 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f47a813e520 0x7f47a80af220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:19.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.405+0000 7f47a7200640 1 --2- 172.21.15.28:0/3034627910 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47a813e0c0 0x7f47a80ae390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:19.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.405+0000 7f47a7200640 1 --2- 172.21.15.28:0/3034627910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47a806dbc0 0x7f47a80afee0 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:19.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.405+0000 7f47a7200640 1 -- 172.21.15.28:0/3034627910 >> 172.21.15.28:0/3034627910 conn(0x7f47a8073110 msgr2=0x7f47a80c5020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:19.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.405+0000 7f47a7200640 1 -- 172.21.15.28:0/3034627910 shutdown_connections 2024-02-21T23:07:19.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.405+0000 7f47a7200640 1 -- 172.21.15.28:0/3034627910 wait complete. 2024-02-21T23:07:19.414 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-02-21T23:07:19.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.645+0000 7fd68ae00640 1 Processor -- start 2024-02-21T23:07:19.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.646+0000 7fd68ae00640 1 -- start start 2024-02-21T23:07:19.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.647+0000 7fd68ae00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd68413e0c0 0x7fd6840c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:19.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.647+0000 7fd68ae00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd684158550 con 0x7fd6840c76f0 2024-02-21T23:07:19.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.647+0000 7fd68ae00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6841587d0 con 0x7fd68406dbc0 2024-02-21T23:07:19.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.647+0000 7fd68ae00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6840c89a0 con 0x7fd68413e0c0 2024-02-21T23:07:19.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.647+0000 7fd68a400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd6840c76f0 0x7fd68406c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59982/0 (socket says 172.21.15.28:59982) 2024-02-21T23:07:19.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.647+0000 7fd68a400640 1 -- 172.21.15.28:0/4064230475 learned_addr learned my addr 172.21.15.28:0/4064230475 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:07:19.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.648+0000 7fd689a00640 1 --2- 172.21.15.28:0/4064230475 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd68413e0c0 0x7fd6840c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:19.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.648+0000 7fd683e00640 1 -- 172.21.15.28:0/4064230475 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2916837324 0 0) 0x7fd67c001490 con 0x7fd6840c76f0 2024-02-21T23:07:19.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.648+0000 7fd683e00640 1 -- 172.21.15.28:0/4064230475 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd66c003520 con 0x7fd6840c76f0 2024-02-21T23:07:19.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.648+0000 7fd683e00640 1 -- 172.21.15.28:0/4064230475 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2020413800 0 0) 0x7fd6841587d0 con 0x7fd68406dbc0 2024-02-21T23:07:19.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.648+0000 7fd683e00640 1 -- 172.21.15.28:0/4064230475 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd67c001490 con 0x7fd68406dbc0 2024-02-21T23:07:19.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.648+0000 7fd683e00640 1 -- 172.21.15.28:0/4064230475 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1286926174 0 0) 0x7fd66c003520 con 0x7fd6840c76f0 2024-02-21T23:07:19.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.648+0000 7fd683e00640 1 -- 172.21.15.28:0/4064230475 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd68413e0c0 msgr2=0x7fd6840c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:19.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.648+0000 7fd683e00640 1 --2- 172.21.15.28:0/4064230475 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd68413e0c0 0x7fd6840c71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:19.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.648+0000 7fd683e00640 1 -- 172.21.15.28:0/4064230475 >> v1:172.21.15.81:6789/0 conn(0x7fd68406dbc0 legacy=0x7fd68406c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:19.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.648+0000 7fd683e00640 1 -- 172.21.15.28:0/4064230475 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd684158ab0 con 0x7fd6840c76f0 2024-02-21T23:07:19.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.648+0000 7fd683e00640 1 -- 172.21.15.28:0/4064230475 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fd67c001bc0 con 0x7fd6840c76f0 2024-02-21T23:07:19.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.648+0000 7fd683e00640 1 -- 172.21.15.28:0/4064230475 >> v1:172.21.15.28:6789/0 conn(0x7fd6840c76f0 legacy=0x7fd68406c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:19.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.648+0000 7fd683e00640 1 --2- 172.21.15.28:0/4064230475 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd66c003b70 0x7fd66c003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:19.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.648+0000 7fd683e00640 1 --2- 172.21.15.28:0/4064230475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd66c004800 0x7fd66c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:19.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.649+0000 7fd683e00640 1 --2- 172.21.15.28:0/4064230475 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd68413e0c0 0x7fd66c0052c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:19.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.649+0000 7fd683e00640 1 -- 172.21.15.28:0/4064230475 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd66c00bb90 con 0x7fd66c004800 2024-02-21T23:07:19.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.649+0000 7fd683e00640 1 -- 172.21.15.28:0/4064230475 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd66c00bd40 con 0x7fd66c003b70 2024-02-21T23:07:19.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.649+0000 7fd683e00640 1 -- 172.21.15.28:0/4064230475 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd66c00bf40 con 0x7fd68413e0c0 2024-02-21T23:07:19.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.649+0000 7fd689000640 1 --2- 172.21.15.28:0/4064230475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd66c004800 0x7fd66c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:19.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.649+0000 7fd689a00640 1 --2- 172.21.15.28:0/4064230475 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd66c003b70 0x7fd66c003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:19.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.649+0000 7fd689000640 1 -- 172.21.15.28:0/4064230475 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd68413e0c0 msgr2=0x7fd66c0052c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:19.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.649+0000 7fd68a400640 1 --2- 172.21.15.28:0/4064230475 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd68413e0c0 0x7fd66c0052c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:19.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.649+0000 7fd689000640 1 --2- 172.21.15.28:0/4064230475 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd68413e0c0 0x7fd66c0052c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:19.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.649+0000 7fd689000640 1 -- 172.21.15.28:0/4064230475 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd66c003b70 msgr2=0x7fd66c003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:19.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.649+0000 7fd689000640 1 --2- 172.21.15.28:0/4064230475 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd66c003b70 0x7fd66c003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:19.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.649+0000 7fd689000640 1 -- 172.21.15.28:0/4064230475 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd66c003520 con 0x7fd66c004800 2024-02-21T23:07:19.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.649+0000 7fd689a00640 1 --2- 172.21.15.28:0/4064230475 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd66c003b70 0x7fd66c003f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:07:19.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.650+0000 7fd689000640 1 --2- 172.21.15.28:0/4064230475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd66c004800 0x7fd66c004c80 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7fd6700018d0 tx=0x7fd66c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdedf69babfef3b3 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:19.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.651+0000 7fd683e00640 1 -- 172.21.15.28:0/4064230475 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd670037070 con 0x7fd66c004800 2024-02-21T23:07:19.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.651+0000 7fd683e00640 1 -- 172.21.15.28:0/4064230475 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6700052d0 con 0x7fd66c004800 2024-02-21T23:07:19.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.651+0000 7fd683e00640 1 -- 172.21.15.28:0/4064230475 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6700014f0 con 0x7fd66c004800 2024-02-21T23:07:19.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.651+0000 7fd68ae00640 1 -- 172.21.15.28:0/4064230475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd66c004800 msgr2=0x7fd66c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:19.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.651+0000 7fd68ae00640 1 --2- 172.21.15.28:0/4064230475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd66c004800 0x7fd66c004c80 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7fd6700018d0 tx=0x7fd66c005830 comp rx=0 tx=0).stop 2024-02-21T23:07:19.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.651+0000 7fd68a400640 1 -- 172.21.15.28:0/4064230475 reap_dead start 2024-02-21T23:07:19.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.651+0000 7fd68ae00640 1 -- 172.21.15.28:0/4064230475 shutdown_connections 2024-02-21T23:07:19.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.651+0000 7fd68ae00640 1 -- 172.21.15.28:0/4064230475 >> 172.21.15.28:0/4064230475 conn(0x7fd684073110 msgr2=0x7fd6840ac490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:19.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.651+0000 7fd68ae00640 1 -- 172.21.15.28:0/4064230475 shutdown_connections 2024-02-21T23:07:19.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.652+0000 7fd68ae00640 1 -- 172.21.15.28:0/4064230475 wait complete. 2024-02-21T23:07:19.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.653+0000 7fd68ae00640 1 Processor -- start 2024-02-21T23:07:19.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.653+0000 7fd68ae00640 1 -- start start 2024-02-21T23:07:19.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.654+0000 7fd68ae00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd66c004800 0x7fd6841eb210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:19.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.654+0000 7fd68ae00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd6841556a0 0x7fd6840ca820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:19.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.654+0000 7fd68ae00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd684152d90 0x7fd6841ee4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:19.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.654+0000 7fd68ae00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd68413e520 con 0x7fd684152d90 2024-02-21T23:07:19.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.654+0000 7fd68ae00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd684159070 con 0x7fd66c004800 2024-02-21T23:07:19.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.654+0000 7fd68ae00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd68413e6a0 con 0x7fd6841556a0 2024-02-21T23:07:19.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.655+0000 7fd68a400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd684152d90 0x7fd6841ee4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:19.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.655+0000 7fd68a400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd684152d90 0x7fd6841ee4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50896/0 (socket says 172.21.15.28:50896) 2024-02-21T23:07:19.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.655+0000 7fd68a400640 1 -- 172.21.15.28:0/484449542 learned_addr learned my addr 172.21.15.28:0/484449542 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:19.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.655+0000 7fd689a00640 1 --2- 172.21.15.28:0/484449542 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd66c004800 0x7fd6841eb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:19.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.655+0000 7fd689000640 1 --2- 172.21.15.28:0/484449542 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd6841556a0 0x7fd6840ca820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:19.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.655+0000 7fd68a400640 1 -- 172.21.15.28:0/484449542 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd6841556a0 msgr2=0x7fd6840ca820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:19.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.655+0000 7fd68a400640 1 --2- 172.21.15.28:0/484449542 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd6841556a0 0x7fd6840ca820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:19.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.655+0000 7fd68a400640 1 -- 172.21.15.28:0/484449542 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd66c004800 msgr2=0x7fd6841eb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:19.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.655+0000 7fd68a400640 1 --2- 172.21.15.28:0/484449542 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd66c004800 0x7fd6841eb210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:19.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.655+0000 7fd68a400640 1 -- 172.21.15.28:0/484449542 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6840aefa0 con 0x7fd684152d90 2024-02-21T23:07:19.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.655+0000 7fd68a400640 1 --2- 172.21.15.28:0/484449542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd684152d90 0x7fd6841ee4b0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7fd6840c8260 tx=0x7fd67c00b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:19.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.656+0000 7fd681600640 1 -- 172.21.15.28:0/484449542 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd67c001ba0 con 0x7fd684152d90 2024-02-21T23:07:19.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.656+0000 7fd681600640 1 -- 172.21.15.28:0/484449542 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd67c00bb80 con 0x7fd684152d90 2024-02-21T23:07:19.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.656+0000 7fd68ae00640 1 -- 172.21.15.28:0/484449542 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6840af280 con 0x7fd684152d90 2024-02-21T23:07:19.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.656+0000 7fd681600640 1 -- 172.21.15.28:0/484449542 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd67c007110 con 0x7fd684152d90 2024-02-21T23:07:19.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.656+0000 7fd68ae00640 1 -- 172.21.15.28:0/484449542 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6841edf90 con 0x7fd684152d90 2024-02-21T23:07:19.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.657+0000 7fd68ae00640 1 -- 172.21.15.28:0/484449542 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6840c4b30 con 0x7fd684152d90 2024-02-21T23:07:19.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.658+0000 7fd681600640 1 -- 172.21.15.28:0/484449542 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd67c01b020 con 0x7fd684152d90 2024-02-21T23:07:19.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.659+0000 7fd681600640 1 --2- 172.21.15.28:0/484449542 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd66406b500 0x7fd66406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:19.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.659+0000 7fd681600640 1 -- 172.21.15.28:0/484449542 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5421+0+0 (secure 0 0 0) 0x7fd67c08a2d0 con 0x7fd684152d90 2024-02-21T23:07:19.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.660+0000 7fd689a00640 1 --2- 172.21.15.28:0/484449542 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd66406b500 0x7fd66406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:19.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.661+0000 7fd689a00640 1 --2- 172.21.15.28:0/484449542 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd66406b500 0x7fd66406d9c0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fd6780099c0 tx=0x7fd678008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:19.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.663+0000 7fd681600640 1 -- 172.21.15.28:0/484449542 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd6840c4b30 con 0x7fd684152d90 2024-02-21T23:07:19.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.867+0000 7fd68ae00640 1 -- 172.21.15.28:0/484449542 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fd68404f290 con 0x7fd684152d90 2024-02-21T23:07:19.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.873+0000 7fd681600640 1 -- 172.21.15.28:0/484449542 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v34) v1 ==== 104+0+0 (secure 0 0 0) 0x7fd684158550 con 0x7fd684152d90 2024-02-21T23:07:19.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.876+0000 7fd68ae00640 1 -- 172.21.15.28:0/484449542 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd66406b500 msgr2=0x7fd66406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:19.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.876+0000 7fd68ae00640 1 --2- 172.21.15.28:0/484449542 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd66406b500 0x7fd66406d9c0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fd6780099c0 tx=0x7fd678008040 comp rx=0 tx=0).stop 2024-02-21T23:07:19.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.876+0000 7fd68ae00640 1 -- 172.21.15.28:0/484449542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd684152d90 msgr2=0x7fd6841ee4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:19.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.876+0000 7fd68ae00640 1 --2- 172.21.15.28:0/484449542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd684152d90 0x7fd6841ee4b0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7fd6840c8260 tx=0x7fd67c00b4a0 comp rx=0 tx=0).stop 2024-02-21T23:07:19.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.876+0000 7fd68ae00640 1 -- 172.21.15.28:0/484449542 shutdown_connections 2024-02-21T23:07:19.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.876+0000 7fd68ae00640 1 --2- 172.21.15.28:0/484449542 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd66406b500 0x7fd66406d9c0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:19.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.876+0000 7fd68ae00640 1 --2- 172.21.15.28:0/484449542 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd6841556a0 0x7fd6840ca820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:19.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.876+0000 7fd68ae00640 1 --2- 172.21.15.28:0/484449542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd684152d90 0x7fd6841ee4b0 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:19.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.876+0000 7fd68ae00640 1 --2- 172.21.15.28:0/484449542 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd66c004800 0x7fd6841eb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:19.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.876+0000 7fd68ae00640 1 -- 172.21.15.28:0/484449542 >> 172.21.15.28:0/484449542 conn(0x7fd684073110 msgr2=0x7fd68406c180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:19.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.877+0000 7fd68ae00640 1 -- 172.21.15.28:0/484449542 shutdown_connections 2024-02-21T23:07:19.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:19.877+0000 7fd68ae00640 1 -- 172.21.15.28:0/484449542 wait complete. 2024-02-21T23:07:19.885 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_runtime' 2024-02-21T23:07:20.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.118+0000 7f1133200640 1 Processor -- start 2024-02-21T23:07:20.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.119+0000 7f1133200640 1 -- start start 2024-02-21T23:07:20.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.120+0000 7f1133200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f112c1569d0 0x7f112c0c3280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:20.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.120+0000 7f1133200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f112c159900 con 0x7f112c0c4fe0 2024-02-21T23:07:20.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.120+0000 7f1133200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f112c159be0 con 0x7f112c159400 2024-02-21T23:07:20.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.120+0000 7f1133200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f112c159ec0 con 0x7f112c1569d0 2024-02-21T23:07:20.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.120+0000 7f1123e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f112c0c4fe0 0x7f112c06e0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59998/0 (socket says 172.21.15.28:59998) 2024-02-21T23:07:20.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.120+0000 7f1123e00640 1 -- 172.21.15.28:0/518940279 learned_addr learned my addr 172.21.15.28:0/518940279 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:07:20.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.120+0000 7f112be00640 1 --2- 172.21.15.28:0/518940279 >> v2:172.21.15.81:3300/0 conn(0x7f112c1569d0 0x7f112c0c3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:20.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.120+0000 7f112b400640 1 -- 172.21.15.28:0/518940279 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3556948506 0 0) 0x7f112c159900 con 0x7f112c0c4fe0 2024-02-21T23:07:20.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.121+0000 7f112b400640 1 -- 172.21.15.28:0/518940279 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1110003520 con 0x7f112c0c4fe0 2024-02-21T23:07:20.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.121+0000 7f112b400640 1 -- 172.21.15.28:0/518940279 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 435075043 0 0) 0x7f1110003520 con 0x7f112c0c4fe0 2024-02-21T23:07:20.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.121+0000 7f112b400640 1 -- 172.21.15.28:0/518940279 >> v2:172.21.15.81:3300/0 conn(0x7f112c1569d0 msgr2=0x7f112c0c3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:20.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.121+0000 7f112b400640 1 --2- 172.21.15.28:0/518940279 >> v2:172.21.15.81:3300/0 conn(0x7f112c1569d0 0x7f112c0c3280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:20.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.121+0000 7f112b400640 1 -- 172.21.15.28:0/518940279 >> v1:172.21.15.81:6789/0 conn(0x7f112c159400 legacy=0x7f112c06e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:20.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.121+0000 7f112b400640 1 -- 172.21.15.28:0/518940279 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f112c15a1a0 con 0x7f112c0c4fe0 2024-02-21T23:07:20.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.121+0000 7f112b400640 1 -- 172.21.15.28:0/518940279 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f1114001690 con 0x7f112c0c4fe0 2024-02-21T23:07:20.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.121+0000 7f112b400640 1 -- 172.21.15.28:0/518940279 >> v1:172.21.15.28:6789/0 conn(0x7f112c0c4fe0 legacy=0x7f112c06e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:20.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.121+0000 7f112b400640 1 --2- 172.21.15.28:0/518940279 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f11100039b0 0x7f1110003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:20.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.121+0000 7f112b400640 1 --2- 172.21.15.28:0/518940279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1110004800 0x7f1110004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:20.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.121+0000 7f112b400640 1 --2- 172.21.15.28:0/518940279 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f11100051c0 0x7f111000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:20.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.121+0000 7f112b400640 1 -- 172.21.15.28:0/518940279 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f112c159ec0 con 0x7f1110004800 2024-02-21T23:07:20.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.121+0000 7f112b400640 1 -- 172.21.15.28:0/518940279 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f111000c160 con 0x7f11100051c0 2024-02-21T23:07:20.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.121+0000 7f112b400640 1 -- 172.21.15.28:0/518940279 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f111000c360 con 0x7f11100039b0 2024-02-21T23:07:20.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.121+0000 7f1123e00640 1 --2- 172.21.15.28:0/518940279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1110004800 0x7f1110004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:20.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.122+0000 7f1131200640 1 --2- 172.21.15.28:0/518940279 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f11100051c0 0x7f111000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:20.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.122+0000 7f112be00640 1 --2- 172.21.15.28:0/518940279 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f11100039b0 0x7f1110003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:20.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.122+0000 7f1123e00640 1 -- 172.21.15.28:0/518940279 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f11100039b0 msgr2=0x7f1110003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:20.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.122+0000 7f1123e00640 1 --2- 172.21.15.28:0/518940279 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f11100039b0 0x7f1110003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:20.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.122+0000 7f1123e00640 1 -- 172.21.15.28:0/518940279 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f11100051c0 msgr2=0x7f111000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:20.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.122+0000 7f1123e00640 1 --2- 172.21.15.28:0/518940279 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f11100051c0 0x7f111000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:20.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.122+0000 7f1123e00640 1 -- 172.21.15.28:0/518940279 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1110003520 con 0x7f1110004800 2024-02-21T23:07:20.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.122+0000 7f1131200640 1 --2- 172.21.15.28:0/518940279 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f11100051c0 0x7f111000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:07:20.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.122+0000 7f1131200640 1 -- 172.21.15.28:0/518940279 reap_dead start 2024-02-21T23:07:20.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.123+0000 7f1123e00640 1 --2- 172.21.15.28:0/518940279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1110004800 0x7f1110004c80 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f111400c210 tx=0x7f1110005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d57e438f53156176 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:20.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.123+0000 7f112b400640 1 -- 172.21.15.28:0/518940279 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1114007d10 con 0x7f1110004800 2024-02-21T23:07:20.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.123+0000 7f112b400640 1 -- 172.21.15.28:0/518940279 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1114030070 con 0x7f1110004800 2024-02-21T23:07:20.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.123+0000 7f112b400640 1 -- 172.21.15.28:0/518940279 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1114005d70 con 0x7f1110004800 2024-02-21T23:07:20.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.124+0000 7f1133200640 1 -- 172.21.15.28:0/518940279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1110004800 msgr2=0x7f1110004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:20.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.124+0000 7f1133200640 1 --2- 172.21.15.28:0/518940279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1110004800 0x7f1110004c80 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f111400c210 tx=0x7f1110005830 comp rx=0 tx=0).stop 2024-02-21T23:07:20.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.124+0000 7f1133200640 1 -- 172.21.15.28:0/518940279 shutdown_connections 2024-02-21T23:07:20.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.124+0000 7f1133200640 1 --2- 172.21.15.28:0/518940279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1110004800 0x7f1110004c80 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:20.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.124+0000 7f1133200640 1 -- 172.21.15.28:0/518940279 >> 172.21.15.28:0/518940279 conn(0x7f112c0c67f0 msgr2=0x7f112c0c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:20.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.124+0000 7f1133200640 1 -- 172.21.15.28:0/518940279 shutdown_connections 2024-02-21T23:07:20.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.124+0000 7f1133200640 1 -- 172.21.15.28:0/518940279 wait complete. 2024-02-21T23:07:20.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.126+0000 7f1133200640 1 Processor -- start 2024-02-21T23:07:20.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.126+0000 7f1133200640 1 -- start start 2024-02-21T23:07:20.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.127+0000 7f1133200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1110004800 0x7f112c153250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:20.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.127+0000 7f1133200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f112c15a690 0x7f112c15b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:20.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.127+0000 7f1133200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f112c14cd70 0x7f112c1f21c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:20.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.127+0000 7f1133200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f112c0c4870 con 0x7f112c14cd70 2024-02-21T23:07:20.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.127+0000 7f1133200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f112c0c46f0 con 0x7f112c15a690 2024-02-21T23:07:20.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.127+0000 7f1133200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f112c0c49f0 con 0x7f1110004800 2024-02-21T23:07:20.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.127+0000 7f1131200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f112c14cd70 0x7f112c1f21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:20.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.127+0000 7f1131200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f112c14cd70 0x7f112c1f21c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50908/0 (socket says 172.21.15.28:50908) 2024-02-21T23:07:20.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.127+0000 7f1131200640 1 -- 172.21.15.28:0/386855240 learned_addr learned my addr 172.21.15.28:0/386855240 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:20.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.127+0000 7f112be00640 1 --2- 172.21.15.28:0/386855240 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1110004800 0x7f112c153250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:20.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.127+0000 7f1123e00640 1 --2- 172.21.15.28:0/386855240 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f112c15a690 0x7f112c15b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:20.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.128+0000 7f1131200640 1 -- 172.21.15.28:0/386855240 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1110004800 msgr2=0x7f112c153250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:20.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.128+0000 7f1131200640 1 --2- 172.21.15.28:0/386855240 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1110004800 0x7f112c153250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:20.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.128+0000 7f1131200640 1 -- 172.21.15.28:0/386855240 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f112c15a690 msgr2=0x7f112c15b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:20.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.128+0000 7f1131200640 1 --2- 172.21.15.28:0/386855240 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f112c15a690 0x7f112c15b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:20.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.128+0000 7f1131200640 1 -- 172.21.15.28:0/386855240 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f112c150d10 con 0x7f112c14cd70 2024-02-21T23:07:20.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.128+0000 7f112be00640 1 --2- 172.21.15.28:0/386855240 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1110004800 0x7f112c153250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:07:20.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.128+0000 7f1131200640 1 --2- 172.21.15.28:0/386855240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f112c14cd70 0x7f112c1f21c0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f112c0c66d0 tx=0x7f112400d9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:20.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.128+0000 7f1128c00640 1 -- 172.21.15.28:0/386855240 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1124018070 con 0x7f112c14cd70 2024-02-21T23:07:20.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.128+0000 7f1128c00640 1 -- 172.21.15.28:0/386855240 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1124003d80 con 0x7f112c14cd70 2024-02-21T23:07:20.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.128+0000 7f1128c00640 1 -- 172.21.15.28:0/386855240 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1124006d90 con 0x7f112c14cd70 2024-02-21T23:07:20.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.128+0000 7f1133200640 1 -- 172.21.15.28:0/386855240 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f112c1f2d70 con 0x7f112c14cd70 2024-02-21T23:07:20.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.128+0000 7f1133200640 1 -- 172.21.15.28:0/386855240 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f112c1f3050 con 0x7f112c14cd70 2024-02-21T23:07:20.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.129+0000 7f1133200640 1 -- 172.21.15.28:0/386855240 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f112c067f30 con 0x7f112c14cd70 2024-02-21T23:07:20.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.131+0000 7f1128c00640 1 -- 172.21.15.28:0/386855240 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1124005070 con 0x7f112c14cd70 2024-02-21T23:07:20.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.131+0000 7f1128c00640 1 --2- 172.21.15.28:0/386855240 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f110806b500 0x7f110806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:20.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.132+0000 7f1128c00640 1 -- 172.21.15.28:0/386855240 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f1124089310 con 0x7f112c14cd70 2024-02-21T23:07:20.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.132+0000 7f112be00640 1 --2- 172.21.15.28:0/386855240 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f110806b500 0x7f110806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:20.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.133+0000 7f112be00640 1 --2- 172.21.15.28:0/386855240 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f110806b500 0x7f110806d9c0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f112c152530 tx=0x7f111c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:20.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.136+0000 7f1128c00640 1 -- 172.21.15.28:0/386855240 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f11240582a0 con 0x7f112c14cd70 2024-02-21T23:07:20.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.421+0000 7f1133200640 1 -- 172.21.15.28:0/386855240 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_runtime"]} v 0) v1 -- 0x7f112c154200 con 0x7f112c14cd70 2024-02-21T23:07:20.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.952+0000 7f1128c00640 1 -- 172.21.15.28:0/386855240 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_runtime"]}]=0 v286) v1 ==== 146+0+0 (secure 0 0 0) 0x7f1124057ba0 con 0x7f112c14cd70 2024-02-21T23:07:20.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.955+0000 7f1133200640 1 -- 172.21.15.28:0/386855240 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f110806b500 msgr2=0x7f110806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:20.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.955+0000 7f1133200640 1 --2- 172.21.15.28:0/386855240 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f110806b500 0x7f110806d9c0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f112c152530 tx=0x7f111c008040 comp rx=0 tx=0).stop 2024-02-21T23:07:20.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.955+0000 7f1133200640 1 -- 172.21.15.28:0/386855240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f112c14cd70 msgr2=0x7f112c1f21c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:20.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.955+0000 7f1133200640 1 --2- 172.21.15.28:0/386855240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f112c14cd70 0x7f112c1f21c0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f112c0c66d0 tx=0x7f112400d9d0 comp rx=0 tx=0).stop 2024-02-21T23:07:20.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.956+0000 7f1133200640 1 -- 172.21.15.28:0/386855240 shutdown_connections 2024-02-21T23:07:20.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.956+0000 7f1133200640 1 --2- 172.21.15.28:0/386855240 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f110806b500 0x7f110806d9c0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:20.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.956+0000 7f1133200640 1 --2- 172.21.15.28:0/386855240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f112c14cd70 0x7f112c1f21c0 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:20.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.956+0000 7f1133200640 1 --2- 172.21.15.28:0/386855240 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f112c15a690 0x7f112c15b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:20.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.956+0000 7f1133200640 1 --2- 172.21.15.28:0/386855240 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1110004800 0x7f112c153250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:20.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.956+0000 7f1133200640 1 -- 172.21.15.28:0/386855240 >> 172.21.15.28:0/386855240 conn(0x7f112c0c67f0 msgr2=0x7f112c0c10f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:20.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.956+0000 7f1133200640 1 -- 172.21.15.28:0/386855240 shutdown_connections 2024-02-21T23:07:20.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:20.956+0000 7f1133200640 1 -- 172.21.15.28:0/386855240 wait complete. 2024-02-21T23:07:20.969 INFO:tasks.cephfs_test_runner:That setting mds_join_fs at runtime affects the cluster layout. ... ok 2024-02-21T23:07:20.970 INFO:tasks.cephfs_test_runner:Starting test: test_join_fs_standby_replay (tasks.cephfs.test_failover.TestClusterAffinity) That a standby-replay daemon with weak affinity is replaced by a stronger one. 2024-02-21T23:07:20.970 INFO:tasks.cephfs_test_runner:test_join_fs_standby_replay (tasks.cephfs.test_failover.TestClusterAffinity) 2024-02-21T23:07:20.970 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:07:21.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.204+0000 7fc23a800640 1 Processor -- start 2024-02-21T23:07:21.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.204+0000 7fc23a800640 1 -- start start 2024-02-21T23:07:21.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.205+0000 7fc23a800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc2340c11f0 0x7fc234153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:21.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.205+0000 7fc23a800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc234152e70 con 0x7fc23415b6d0 2024-02-21T23:07:21.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.205+0000 7fc23a800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc234153150 con 0x7fc2340c15d0 2024-02-21T23:07:21.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.205+0000 7fc23a800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2340c1a60 con 0x7fc2340c11f0 2024-02-21T23:07:21.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.205+0000 7fc233400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc2340c11f0 0x7fc234153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:21.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.205+0000 7fc233e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fc23415b6d0 0x7fc23415dac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60006/0 (socket says 172.21.15.28:60006) 2024-02-21T23:07:21.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.205+0000 7fc233e00640 1 -- 172.21.15.28:0/1821414947 learned_addr learned my addr 172.21.15.28:0/1821414947 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:07:21.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.206+0000 7fc233400640 1 -- 172.21.15.28:0/1821414947 >> v1:172.21.15.81:6789/0 conn(0x7fc2340c15d0 legacy=0x7fc23406dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:21.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.206+0000 7fc233400640 1 -- 172.21.15.28:0/1821414947 >> v1:172.21.15.28:6789/0 conn(0x7fc23415b6d0 legacy=0x7fc23415dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:21.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.206+0000 7fc233400640 1 -- 172.21.15.28:0/1821414947 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc234153430 con 0x7fc2340c11f0 2024-02-21T23:07:21.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.207+0000 7fc233400640 1 --2- 172.21.15.28:0/1821414947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2340c11f0 0x7fc234153950 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7fc2240097a0 tx=0x7fc22402b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1b5c91bc478ee1b server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:21.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.207+0000 7fc232000640 1 -- 172.21.15.28:0/1821414947 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc224038070 con 0x7fc2340c11f0 2024-02-21T23:07:21.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.207+0000 7fc232000640 1 -- 172.21.15.28:0/1821414947 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc224031700 con 0x7fc2340c11f0 2024-02-21T23:07:21.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.207+0000 7fc232000640 1 -- 172.21.15.28:0/1821414947 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc224031ad0 con 0x7fc2340c11f0 2024-02-21T23:07:21.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.208+0000 7fc23a800640 1 -- 172.21.15.28:0/1821414947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2340c11f0 msgr2=0x7fc234153950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:21.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.208+0000 7fc23a800640 1 --2- 172.21.15.28:0/1821414947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2340c11f0 0x7fc234153950 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7fc2240097a0 tx=0x7fc22402b7a0 comp rx=0 tx=0).stop 2024-02-21T23:07:21.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.208+0000 7fc23a800640 1 -- 172.21.15.28:0/1821414947 shutdown_connections 2024-02-21T23:07:21.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.208+0000 7fc23a800640 1 --2- 172.21.15.28:0/1821414947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2340c11f0 0x7fc234153950 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:21.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.208+0000 7fc23a800640 1 -- 172.21.15.28:0/1821414947 >> 172.21.15.28:0/1821414947 conn(0x7fc2340c67f0 msgr2=0x7fc2340c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:21.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.208+0000 7fc23a800640 1 -- 172.21.15.28:0/1821414947 shutdown_connections 2024-02-21T23:07:21.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.208+0000 7fc23a800640 1 -- 172.21.15.28:0/1821414947 wait complete. 2024-02-21T23:07:21.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.210+0000 7fc23a800640 1 Processor -- start 2024-02-21T23:07:21.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.210+0000 7fc23a800640 1 -- start start 2024-02-21T23:07:21.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.211+0000 7fc23a800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2340c11f0 0x7fc2341ebaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:21.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.211+0000 7fc23a800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc2340c15d0 0x7fc234148bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:21.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.211+0000 7fc23a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc23415b6d0 0x7fc23413fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:21.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.211+0000 7fc23a800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2340c4c00 con 0x7fc23415b6d0 2024-02-21T23:07:21.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.211+0000 7fc23a800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2340a5ec0 con 0x7fc2340c11f0 2024-02-21T23:07:21.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.211+0000 7fc23a800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2340a6110 con 0x7fc2340c15d0 2024-02-21T23:07:21.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.211+0000 7fc233e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc23415b6d0 0x7fc23413fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:21.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.211+0000 7fc233e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc23415b6d0 0x7fc23413fad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50918/0 (socket says 172.21.15.28:50918) 2024-02-21T23:07:21.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.211+0000 7fc233e00640 1 -- 172.21.15.28:0/3317227747 learned_addr learned my addr 172.21.15.28:0/3317227747 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:21.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.211+0000 7fc233400640 1 --2- 172.21.15.28:0/3317227747 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2340c11f0 0x7fc2341ebaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:21.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.211+0000 7fc232a00640 1 --2- 172.21.15.28:0/3317227747 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc2340c15d0 0x7fc234148bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:21.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.212+0000 7fc233e00640 1 -- 172.21.15.28:0/3317227747 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc2340c15d0 msgr2=0x7fc234148bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:21.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.212+0000 7fc233e00640 1 --2- 172.21.15.28:0/3317227747 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc2340c15d0 0x7fc234148bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:21.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.212+0000 7fc233e00640 1 -- 172.21.15.28:0/3317227747 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2340c11f0 msgr2=0x7fc2341ebaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:21.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.212+0000 7fc233e00640 1 --2- 172.21.15.28:0/3317227747 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2340c11f0 0x7fc2341ebaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:21.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.212+0000 7fc233e00640 1 -- 172.21.15.28:0/3317227747 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2340a75c0 con 0x7fc23415b6d0 2024-02-21T23:07:21.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.212+0000 7fc233400640 1 --2- 172.21.15.28:0/3317227747 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2340c11f0 0x7fc2341ebaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:07:21.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.212+0000 7fc233e00640 1 --2- 172.21.15.28:0/3317227747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc23415b6d0 0x7fc23413fad0 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7fc22c00cbc0 tx=0x7fc22c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:21.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.212+0000 7fc22b400640 1 -- 172.21.15.28:0/3317227747 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc22c017070 con 0x7fc23415b6d0 2024-02-21T23:07:21.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.212+0000 7fc22b400640 1 -- 172.21.15.28:0/3317227747 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc22c00ace0 con 0x7fc23415b6d0 2024-02-21T23:07:21.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.212+0000 7fc22b400640 1 -- 172.21.15.28:0/3317227747 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc22c012600 con 0x7fc23415b6d0 2024-02-21T23:07:21.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.212+0000 7fc23a800640 1 -- 172.21.15.28:0/3317227747 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2340a78a0 con 0x7fc23415b6d0 2024-02-21T23:07:21.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.213+0000 7fc23a800640 1 -- 172.21.15.28:0/3317227747 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc234149100 con 0x7fc23415b6d0 2024-02-21T23:07:21.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.214+0000 7fc23a800640 1 -- 172.21.15.28:0/3317227747 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc234067f30 con 0x7fc23415b6d0 2024-02-21T23:07:21.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.215+0000 7fc22b400640 1 -- 172.21.15.28:0/3317227747 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc22c01a070 con 0x7fc23415b6d0 2024-02-21T23:07:21.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.215+0000 7fc22b400640 1 --2- 172.21.15.28:0/3317227747 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc21406b4d0 0x7fc21406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:21.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.216+0000 7fc22b400640 1 -- 172.21.15.28:0/3317227747 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5421+0+0 (secure 0 0 0) 0x7fc22c089d90 con 0x7fc23415b6d0 2024-02-21T23:07:21.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.216+0000 7fc233400640 1 --2- 172.21.15.28:0/3317227747 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc21406b4d0 0x7fc21406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:21.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.217+0000 7fc233400640 1 --2- 172.21.15.28:0/3317227747 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc21406b4d0 0x7fc21406d990 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fc224035000 tx=0x7fc224008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:21.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.220+0000 7fc22b400640 1 -- 172.21.15.28:0/3317227747 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc22c058640 con 0x7fc23415b6d0 2024-02-21T23:07:21.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.422+0000 7fc23a800640 1 -- 172.21.15.28:0/3317227747 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc234147340 con 0x7fc23415b6d0 2024-02-21T23:07:21.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.424+0000 7fc22b400640 1 -- 172.21.15.28:0/3317227747 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v102) v1 ==== 74+0+13361 (secure 0 0 0) 0x7fc22c05c6e0 con 0x7fc23415b6d0 2024-02-21T23:07:21.414 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:07:21.414 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":102,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:07:18.932807+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6833/85689296":"2024-02-22T23:07:15.914225+0000","172.21.15.28:6832/85689296":"2024-02-22T23:07:15.914225+0000","172.21.15.28:6835/2477712025":"2024-02-22T23:07:05.837870+0000","172.21.15.28:6834/2477712025":"2024-02-22T23:07:05.837870+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:07:21.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.427+0000 7fc23a800640 1 -- 172.21.15.28:0/3317227747 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc21406b4d0 msgr2=0x7fc21406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:21.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.427+0000 7fc23a800640 1 --2- 172.21.15.28:0/3317227747 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc21406b4d0 0x7fc21406d990 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fc224035000 tx=0x7fc224008030 comp rx=0 tx=0).stop 2024-02-21T23:07:21.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.427+0000 7fc23a800640 1 -- 172.21.15.28:0/3317227747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc23415b6d0 msgr2=0x7fc23413fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:21.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.427+0000 7fc23a800640 1 --2- 172.21.15.28:0/3317227747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc23415b6d0 0x7fc23413fad0 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7fc22c00cbc0 tx=0x7fc22c00a430 comp rx=0 tx=0).stop 2024-02-21T23:07:21.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.427+0000 7fc23a800640 1 -- 172.21.15.28:0/3317227747 shutdown_connections 2024-02-21T23:07:21.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.427+0000 7fc23a800640 1 --2- 172.21.15.28:0/3317227747 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc21406b4d0 0x7fc21406d990 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:21.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.427+0000 7fc23a800640 1 --2- 172.21.15.28:0/3317227747 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc2340c15d0 0x7fc234148bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:21.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.427+0000 7fc23a800640 1 --2- 172.21.15.28:0/3317227747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc23415b6d0 0x7fc23413fad0 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:21.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.427+0000 7fc23a800640 1 --2- 172.21.15.28:0/3317227747 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2340c11f0 0x7fc2341ebaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:21.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.427+0000 7fc23a800640 1 -- 172.21.15.28:0/3317227747 >> 172.21.15.28:0/3317227747 conn(0x7fc2340c67f0 msgr2=0x7fc23414a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:21.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.427+0000 7fc23a800640 1 -- 172.21.15.28:0/3317227747 shutdown_connections 2024-02-21T23:07:21.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.427+0000 7fc23a800640 1 -- 172.21.15.28:0/3317227747 wait complete. 2024-02-21T23:07:21.437 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-21T23:02:30.728329+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-21T23:07:21.438 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-21T23:07:21.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.670+0000 7f2c3d400640 1 Processor -- start 2024-02-21T23:07:21.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.671+0000 7f2c3d400640 1 -- start start 2024-02-21T23:07:21.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.672+0000 7f2c3d400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2c3813e0c0 0x7f2c380c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:21.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.672+0000 7f2c3d400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2c380c76f0 0x7f2c3806c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:21.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.672+0000 7f2c3d400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c38158550 con 0x7f2c3806dbc0 2024-02-21T23:07:21.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.672+0000 7f2c3d400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c380c89a0 con 0x7f2c380c76f0 2024-02-21T23:07:21.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.672+0000 7f2c3d400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c381587d0 con 0x7f2c3813e0c0 2024-02-21T23:07:21.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.673+0000 7f2c3ca00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2c380c76f0 0x7f2c3806c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:21.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.673+0000 7f2c3ca00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2c380c76f0 0x7f2c3806c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50932/0 (socket says 172.21.15.28:50932) 2024-02-21T23:07:21.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.673+0000 7f2c3ca00640 1 -- 172.21.15.28:0/2130817967 learned_addr learned my addr 172.21.15.28:0/2130817967 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:21.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.673+0000 7f2c37e00640 1 --2- 172.21.15.28:0/2130817967 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2c3813e0c0 0x7f2c380c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:21.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.673+0000 7f2c3ca00640 1 -- 172.21.15.28:0/2130817967 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2c3813e0c0 msgr2=0x7f2c380c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:21.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.673+0000 7f2c3ca00640 1 --2- 172.21.15.28:0/2130817967 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2c3813e0c0 0x7f2c380c71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:21.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.673+0000 7f2c3ca00640 1 -- 172.21.15.28:0/2130817967 >> v1:172.21.15.81:6789/0 conn(0x7f2c3806dbc0 legacy=0x7f2c3806c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:21.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.673+0000 7f2c3ca00640 1 -- 172.21.15.28:0/2130817967 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c38158ab0 con 0x7f2c380c76f0 2024-02-21T23:07:21.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.674+0000 7f2c3ca00640 1 --2- 172.21.15.28:0/2130817967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c380c76f0 0x7f2c3806c990 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f2c30009e50 tx=0x7f2c3002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1ce626bc2091eeb server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:21.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.675+0000 7f2c36a00640 1 -- 172.21.15.28:0/2130817967 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c3002c9e0 con 0x7f2c380c76f0 2024-02-21T23:07:21.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.675+0000 7f2c36a00640 1 -- 172.21.15.28:0/2130817967 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c3002cb80 con 0x7f2c380c76f0 2024-02-21T23:07:21.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.675+0000 7f2c36a00640 1 -- 172.21.15.28:0/2130817967 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c300343f0 con 0x7f2c380c76f0 2024-02-21T23:07:21.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.675+0000 7f2c3d400640 1 -- 172.21.15.28:0/2130817967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c380c76f0 msgr2=0x7f2c3806c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:21.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.675+0000 7f2c3d400640 1 --2- 172.21.15.28:0/2130817967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c380c76f0 0x7f2c3806c990 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f2c30009e50 tx=0x7f2c3002be40 comp rx=0 tx=0).stop 2024-02-21T23:07:21.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.676+0000 7f2c3d400640 1 -- 172.21.15.28:0/2130817967 shutdown_connections 2024-02-21T23:07:21.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.676+0000 7f2c3d400640 1 --2- 172.21.15.28:0/2130817967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c380c76f0 0x7f2c3806c990 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:21.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.676+0000 7f2c3d400640 1 --2- 172.21.15.28:0/2130817967 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2c3813e0c0 0x7f2c380c71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:21.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.676+0000 7f2c3d400640 1 -- 172.21.15.28:0/2130817967 >> 172.21.15.28:0/2130817967 conn(0x7f2c38073110 msgr2=0x7f2c380ac490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:21.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.676+0000 7f2c3d400640 1 -- 172.21.15.28:0/2130817967 shutdown_connections 2024-02-21T23:07:21.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.676+0000 7f2c3d400640 1 -- 172.21.15.28:0/2130817967 wait complete. 2024-02-21T23:07:21.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.677+0000 7f2c3d400640 1 Processor -- start 2024-02-21T23:07:21.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.678+0000 7f2c3d400640 1 -- start start 2024-02-21T23:07:21.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.678+0000 7f2c3d400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2c3806dbc0 0x7f2c381559a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:21.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.678+0000 7f2c3d400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2c380c76f0 0x7f2c38137510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:21.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.679+0000 7f2c3d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c3813e0c0 0x7f2c381354e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:21.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.679+0000 7f2c3d400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c3806ced0 con 0x7f2c3813e0c0 2024-02-21T23:07:21.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.679+0000 7f2c3d400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c38159070 con 0x7f2c380c76f0 2024-02-21T23:07:21.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.679+0000 7f2c3d400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c3813e520 con 0x7f2c3806dbc0 2024-02-21T23:07:21.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.679+0000 7f2c3ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c3813e0c0 0x7f2c381354e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:21.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.679+0000 7f2c3ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c3813e0c0 0x7f2c381354e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50946/0 (socket says 172.21.15.28:50946) 2024-02-21T23:07:21.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.679+0000 7f2c3ca00640 1 -- 172.21.15.28:0/4258009177 learned_addr learned my addr 172.21.15.28:0/4258009177 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:21.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.679+0000 7f2c37400640 1 --2- 172.21.15.28:0/4258009177 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2c380c76f0 0x7f2c38137510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:21.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.679+0000 7f2c37e00640 1 --2- 172.21.15.28:0/4258009177 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2c3806dbc0 0x7f2c381559a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:21.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.679+0000 7f2c3ca00640 1 -- 172.21.15.28:0/4258009177 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2c3806dbc0 msgr2=0x7f2c381559a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:21.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.679+0000 7f2c3ca00640 1 --2- 172.21.15.28:0/4258009177 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2c3806dbc0 0x7f2c381559a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:21.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.679+0000 7f2c3ca00640 1 -- 172.21.15.28:0/4258009177 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2c380c76f0 msgr2=0x7f2c38137510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:21.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.679+0000 7f2c3ca00640 1 --2- 172.21.15.28:0/4258009177 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2c380c76f0 0x7f2c38137510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:21.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.679+0000 7f2c3ca00640 1 -- 172.21.15.28:0/4258009177 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c381eb870 con 0x7f2c3813e0c0 2024-02-21T23:07:21.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.680+0000 7f2c3ca00640 1 --2- 172.21.15.28:0/4258009177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c3813e0c0 0x7f2c381354e0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f2c30036040 tx=0x7f2c30004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:21.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.680+0000 7f2c2fe00640 1 -- 172.21.15.28:0/4258009177 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c30043020 con 0x7f2c3813e0c0 2024-02-21T23:07:21.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.680+0000 7f2c2fe00640 1 -- 172.21.15.28:0/4258009177 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c3003e450 con 0x7f2c3813e0c0 2024-02-21T23:07:21.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.680+0000 7f2c2fe00640 1 -- 172.21.15.28:0/4258009177 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c30009510 con 0x7f2c3813e0c0 2024-02-21T23:07:21.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.680+0000 7f2c3d400640 1 -- 172.21.15.28:0/4258009177 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c381ebaf0 con 0x7f2c3813e0c0 2024-02-21T23:07:21.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.680+0000 7f2c3d400640 1 -- 172.21.15.28:0/4258009177 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c38134fc0 con 0x7f2c3813e0c0 2024-02-21T23:07:21.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.681+0000 7f2c3d400640 1 -- 172.21.15.28:0/4258009177 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c3804f290 con 0x7f2c3813e0c0 2024-02-21T23:07:21.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.682+0000 7f2c2fe00640 1 -- 172.21.15.28:0/4258009177 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2c30046020 con 0x7f2c3813e0c0 2024-02-21T23:07:21.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.683+0000 7f2c2fe00640 1 --2- 172.21.15.28:0/4258009177 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2c1406b4d0 0x7f2c1406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:21.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.683+0000 7f2c2fe00640 1 -- 172.21.15.28:0/4258009177 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f2c300ac8d0 con 0x7f2c3813e0c0 2024-02-21T23:07:21.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.684+0000 7f2c37e00640 1 --2- 172.21.15.28:0/4258009177 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2c1406b4d0 0x7f2c1406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:21.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.685+0000 7f2c37e00640 1 --2- 172.21.15.28:0/4258009177 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2c1406b4d0 0x7f2c1406d990 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f2c28006fd0 tx=0x7f2c28008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:21.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.688+0000 7f2c2fe00640 1 -- 172.21.15.28:0/4258009177 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f2c3004d030 con 0x7f2c3813e0c0 2024-02-21T23:07:21.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.891+0000 7f2c3d400640 1 -- 172.21.15.28:0/4258009177 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2c3806d9a0 con 0x7f2c3813e0c0 2024-02-21T23:07:21.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.892+0000 7f2c2fe00640 1 -- 172.21.15.28:0/4258009177 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v102) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2c381eb870 con 0x7f2c3813e0c0 2024-02-21T23:07:21.885 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-02-21T23:07:21.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.895+0000 7f2c3d400640 1 -- 172.21.15.28:0/4258009177 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2c1406b4d0 msgr2=0x7f2c1406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:21.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.895+0000 7f2c3d400640 1 --2- 172.21.15.28:0/4258009177 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2c1406b4d0 0x7f2c1406d990 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f2c28006fd0 tx=0x7f2c28008040 comp rx=0 tx=0).stop 2024-02-21T23:07:21.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.895+0000 7f2c3d400640 1 -- 172.21.15.28:0/4258009177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c3813e0c0 msgr2=0x7f2c381354e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:21.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.895+0000 7f2c3d400640 1 --2- 172.21.15.28:0/4258009177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c3813e0c0 0x7f2c381354e0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f2c30036040 tx=0x7f2c30004a40 comp rx=0 tx=0).stop 2024-02-21T23:07:21.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.895+0000 7f2c3d400640 1 -- 172.21.15.28:0/4258009177 shutdown_connections 2024-02-21T23:07:21.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.895+0000 7f2c3d400640 1 --2- 172.21.15.28:0/4258009177 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2c1406b4d0 0x7f2c1406d990 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:21.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.895+0000 7f2c3d400640 1 --2- 172.21.15.28:0/4258009177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c3813e0c0 0x7f2c381354e0 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:21.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.895+0000 7f2c3d400640 1 --2- 172.21.15.28:0/4258009177 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2c380c76f0 0x7f2c38137510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:21.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.895+0000 7f2c3d400640 1 --2- 172.21.15.28:0/4258009177 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2c3806dbc0 0x7f2c381559a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:21.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.895+0000 7f2c3d400640 1 -- 172.21.15.28:0/4258009177 >> 172.21.15.28:0/4258009177 conn(0x7f2c38073110 msgr2=0x7f2c3813bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:21.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.896+0000 7f2c3d400640 1 -- 172.21.15.28:0/4258009177 shutdown_connections 2024-02-21T23:07:21.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:21.896+0000 7f2c3d400640 1 -- 172.21.15.28:0/4258009177 wait complete. 2024-02-21T23:07:21.905 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_standby_replay' 2024-02-21T23:07:22.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.137+0000 7fde96800640 1 Processor -- start 2024-02-21T23:07:22.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.138+0000 7fde96800640 1 -- start start 2024-02-21T23:07:22.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.139+0000 7fde96800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fde90150450 0x7fde900a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:22.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.139+0000 7fde96800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fde90156af0 0x7fde9006dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:22.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.139+0000 7fde96800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde900c2760 con 0x7fde9006ee40 2024-02-21T23:07:22.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.139+0000 7fde96800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde900c22c0 con 0x7fde90156af0 2024-02-21T23:07:22.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.139+0000 7fde96800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde90156660 con 0x7fde90150450 2024-02-21T23:07:22.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.139+0000 7fde94a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fde9006ee40 0x7fde901536f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60018/0 (socket says 172.21.15.28:60018) 2024-02-21T23:07:22.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.139+0000 7fde94a00640 1 -- 172.21.15.28:0/3175484529 learned_addr learned my addr 172.21.15.28:0/3175484529 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:07:22.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.140+0000 7fde95e00640 1 --2- 172.21.15.28:0/3175484529 >> v2:172.21.15.81:3300/0 conn(0x7fde90156af0 0x7fde9006dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:22.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.140+0000 7fde95400640 1 --2- 172.21.15.28:0/3175484529 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fde90150450 0x7fde900a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:22.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.140+0000 7fde8fe00640 1 -- 172.21.15.28:0/3175484529 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 773882009 0 0) 0x7fde900c2760 con 0x7fde9006ee40 2024-02-21T23:07:22.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.140+0000 7fde8fe00640 1 -- 172.21.15.28:0/3175484529 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde78003520 con 0x7fde9006ee40 2024-02-21T23:07:22.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.140+0000 7fde8fe00640 1 -- 172.21.15.28:0/3175484529 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4116112459 0 0) 0x7fde78003520 con 0x7fde9006ee40 2024-02-21T23:07:22.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.140+0000 7fde8fe00640 1 -- 172.21.15.28:0/3175484529 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fde90150450 msgr2=0x7fde900a5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:22.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.140+0000 7fde8fe00640 1 --2- 172.21.15.28:0/3175484529 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fde90150450 0x7fde900a5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:22.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.140+0000 7fde8fe00640 1 -- 172.21.15.28:0/3175484529 >> v2:172.21.15.81:3300/0 conn(0x7fde90156af0 msgr2=0x7fde9006dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:22.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.140+0000 7fde8fe00640 1 --2- 172.21.15.28:0/3175484529 >> v2:172.21.15.81:3300/0 conn(0x7fde90156af0 0x7fde9006dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:22.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.140+0000 7fde8fe00640 1 -- 172.21.15.28:0/3175484529 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde90156310 con 0x7fde9006ee40 2024-02-21T23:07:22.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.140+0000 7fde8fe00640 1 -- 172.21.15.28:0/3175484529 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fde7c001670 con 0x7fde9006ee40 2024-02-21T23:07:22.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.140+0000 7fde8fe00640 1 -- 172.21.15.28:0/3175484529 >> v1:172.21.15.28:6789/0 conn(0x7fde9006ee40 legacy=0x7fde901536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:22.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.141+0000 7fde8fe00640 1 --2- 172.21.15.28:0/3175484529 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fde90150450 0x7fde780039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:22.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.141+0000 7fde8fe00640 1 --2- 172.21.15.28:0/3175484529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde78003f30 0x7fde78004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:22.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.141+0000 7fde8fe00640 1 --2- 172.21.15.28:0/3175484529 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde78004d40 0x7fde780051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:22.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.141+0000 7fde8fe00640 1 -- 172.21.15.28:0/3175484529 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde90156660 con 0x7fde78003f30 2024-02-21T23:07:22.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.141+0000 7fde8fe00640 1 -- 172.21.15.28:0/3175484529 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde7800b9c0 con 0x7fde78004d40 2024-02-21T23:07:22.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.141+0000 7fde8fe00640 1 -- 172.21.15.28:0/3175484529 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde7800bb90 con 0x7fde90150450 2024-02-21T23:07:22.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.141+0000 7fde94a00640 1 --2- 172.21.15.28:0/3175484529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde78003f30 0x7fde78004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:22.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.141+0000 7fde95e00640 1 --2- 172.21.15.28:0/3175484529 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde78004d40 0x7fde780051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:22.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.141+0000 7fde95400640 1 --2- 172.21.15.28:0/3175484529 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fde90150450 0x7fde780039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:22.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.141+0000 7fde94a00640 1 -- 172.21.15.28:0/3175484529 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fde90150450 msgr2=0x7fde780039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:22.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.141+0000 7fde94a00640 1 --2- 172.21.15.28:0/3175484529 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fde90150450 0x7fde780039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:22.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.141+0000 7fde94a00640 1 -- 172.21.15.28:0/3175484529 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde78004d40 msgr2=0x7fde780051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:22.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.141+0000 7fde94a00640 1 --2- 172.21.15.28:0/3175484529 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde78004d40 0x7fde780051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:22.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.141+0000 7fde94a00640 1 -- 172.21.15.28:0/3175484529 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde78003520 con 0x7fde78003f30 2024-02-21T23:07:22.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.143+0000 7fde94a00640 1 --2- 172.21.15.28:0/3175484529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde78003f30 0x7fde78004800 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7fde7c00c290 tx=0x7fde78005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34d2a905116323ed server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:22.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.143+0000 7fde8fe00640 1 -- 172.21.15.28:0/3175484529 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde7c007d50 con 0x7fde78003f30 2024-02-21T23:07:22.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.143+0000 7fde8fe00640 1 -- 172.21.15.28:0/3175484529 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde7c030070 con 0x7fde78003f30 2024-02-21T23:07:22.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.143+0000 7fde8fe00640 1 -- 172.21.15.28:0/3175484529 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde7c02cde0 con 0x7fde78003f30 2024-02-21T23:07:22.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.143+0000 7fde96800640 1 -- 172.21.15.28:0/3175484529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde78003f30 msgr2=0x7fde78004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:22.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.143+0000 7fde96800640 1 --2- 172.21.15.28:0/3175484529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde78003f30 0x7fde78004800 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7fde7c00c290 tx=0x7fde78005830 comp rx=0 tx=0).stop 2024-02-21T23:07:22.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.143+0000 7fde95e00640 1 -- 172.21.15.28:0/3175484529 reap_dead start 2024-02-21T23:07:22.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.144+0000 7fde96800640 1 -- 172.21.15.28:0/3175484529 shutdown_connections 2024-02-21T23:07:22.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.144+0000 7fde96800640 1 -- 172.21.15.28:0/3175484529 >> 172.21.15.28:0/3175484529 conn(0x7fde9006b630 msgr2=0x7fde900c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:22.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.144+0000 7fde96800640 1 -- 172.21.15.28:0/3175484529 shutdown_connections 2024-02-21T23:07:22.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.144+0000 7fde96800640 1 -- 172.21.15.28:0/3175484529 wait complete. 2024-02-21T23:07:22.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.145+0000 7fde96800640 1 Processor -- start 2024-02-21T23:07:22.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.145+0000 7fde96800640 1 -- start start 2024-02-21T23:07:22.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.146+0000 7fde96800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fde78003f30 0x7fde900bfba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:22.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.146+0000 7fde96800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde900714a0 0x7fde901ed860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:22.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.146+0000 7fde96800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde900a93a0 0x7fde901edda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:22.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.146+0000 7fde96800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde90156f80 con 0x7fde900714a0 2024-02-21T23:07:22.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.146+0000 7fde96800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde900c2cb0 con 0x7fde900a93a0 2024-02-21T23:07:22.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.146+0000 7fde96800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde90157100 con 0x7fde78003f30 2024-02-21T23:07:22.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.147+0000 7fde94a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde900714a0 0x7fde901ed860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:22.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.147+0000 7fde94a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde900714a0 0x7fde901ed860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50954/0 (socket says 172.21.15.28:50954) 2024-02-21T23:07:22.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.147+0000 7fde94a00640 1 -- 172.21.15.28:0/2419690947 learned_addr learned my addr 172.21.15.28:0/2419690947 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:22.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.147+0000 7fde94a00640 1 -- 172.21.15.28:0/2419690947 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fde78003f30 msgr2=0x7fde900bfba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:22.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.147+0000 7fde94a00640 1 --2- 172.21.15.28:0/2419690947 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fde78003f30 0x7fde900bfba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:22.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.147+0000 7fde94a00640 1 -- 172.21.15.28:0/2419690947 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde900a93a0 msgr2=0x7fde901edda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:22.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.147+0000 7fde95e00640 1 --2- 172.21.15.28:0/2419690947 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde900a93a0 0x7fde901edda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:22.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.147+0000 7fde94a00640 1 --2- 172.21.15.28:0/2419690947 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde900a93a0 0x7fde901edda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:22.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.147+0000 7fde94a00640 1 -- 172.21.15.28:0/2419690947 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde901eb240 con 0x7fde900714a0 2024-02-21T23:07:22.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.147+0000 7fde94a00640 1 --2- 172.21.15.28:0/2419690947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde900714a0 0x7fde901ed860 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7fde7c00c380 tx=0x7fde90070ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:22.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.148+0000 7fde8d600640 1 -- 172.21.15.28:0/2419690947 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde7c045070 con 0x7fde900714a0 2024-02-21T23:07:22.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.148+0000 7fde8d600640 1 -- 172.21.15.28:0/2419690947 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde7c004bf0 con 0x7fde900714a0 2024-02-21T23:07:22.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.148+0000 7fde8d600640 1 -- 172.21.15.28:0/2419690947 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde7c001c80 con 0x7fde900714a0 2024-02-21T23:07:22.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.148+0000 7fde96800640 1 -- 172.21.15.28:0/2419690947 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde9014ce00 con 0x7fde900714a0 2024-02-21T23:07:22.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.148+0000 7fde96800640 1 -- 172.21.15.28:0/2419690947 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde901eaf60 con 0x7fde900714a0 2024-02-21T23:07:22.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.149+0000 7fde96800640 1 -- 172.21.15.28:0/2419690947 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde9004f290 con 0x7fde900714a0 2024-02-21T23:07:22.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.150+0000 7fde8d600640 1 -- 172.21.15.28:0/2419690947 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fde7c048070 con 0x7fde900714a0 2024-02-21T23:07:22.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.151+0000 7fde8d600640 1 --2- 172.21.15.28:0/2419690947 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fde7006b520 0x7fde7006d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:22.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.151+0000 7fde8d600640 1 -- 172.21.15.28:0/2419690947 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5421+0+0 (secure 0 0 0) 0x7fde7c0ab260 con 0x7fde900714a0 2024-02-21T23:07:22.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.152+0000 7fde95400640 1 --2- 172.21.15.28:0/2419690947 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fde7006b520 0x7fde7006d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:22.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.153+0000 7fde95400640 1 --2- 172.21.15.28:0/2419690947 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fde7006b520 0x7fde7006d9e0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fde84006b10 tx=0x7fde84006a20 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:22.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.156+0000 7fde8d600640 1 -- 172.21.15.28:0/2419690947 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fde7c07a270 con 0x7fde900714a0 2024-02-21T23:07:22.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.430+0000 7fde96800640 1 -- 172.21.15.28:0/2419690947 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_standby_replay"]} v 0) v1 -- 0x7fde900c1f10 con 0x7fde900714a0 2024-02-21T23:07:22.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.966+0000 7fde8d600640 1 -- 172.21.15.28:0/2419690947 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_standby_replay"]}]=0 v288) v1 ==== 156+0+0 (secure 0 0 0) 0x7fde7c079b70 con 0x7fde900714a0 2024-02-21T23:07:22.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.970+0000 7fde96800640 1 -- 172.21.15.28:0/2419690947 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fde7006b520 msgr2=0x7fde7006d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:22.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.970+0000 7fde96800640 1 --2- 172.21.15.28:0/2419690947 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fde7006b520 0x7fde7006d9e0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fde84006b10 tx=0x7fde84006a20 comp rx=0 tx=0).stop 2024-02-21T23:07:22.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.970+0000 7fde96800640 1 -- 172.21.15.28:0/2419690947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde900714a0 msgr2=0x7fde901ed860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:22.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.970+0000 7fde96800640 1 --2- 172.21.15.28:0/2419690947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde900714a0 0x7fde901ed860 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7fde7c00c380 tx=0x7fde90070ec0 comp rx=0 tx=0).stop 2024-02-21T23:07:22.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.970+0000 7fde96800640 1 -- 172.21.15.28:0/2419690947 shutdown_connections 2024-02-21T23:07:22.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.970+0000 7fde96800640 1 --2- 172.21.15.28:0/2419690947 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fde7006b520 0x7fde7006d9e0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:22.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.970+0000 7fde96800640 1 --2- 172.21.15.28:0/2419690947 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde900a93a0 0x7fde901edda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:22.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.970+0000 7fde96800640 1 --2- 172.21.15.28:0/2419690947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde900714a0 0x7fde901ed860 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:22.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.970+0000 7fde96800640 1 --2- 172.21.15.28:0/2419690947 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fde78003f30 0x7fde900bfba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:22.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.970+0000 7fde96800640 1 -- 172.21.15.28:0/2419690947 >> 172.21.15.28:0/2419690947 conn(0x7fde9006b630 msgr2=0x7fde9014dc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:22.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.970+0000 7fde96800640 1 -- 172.21.15.28:0/2419690947 shutdown_connections 2024-02-21T23:07:22.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:22.970+0000 7fde96800640 1 -- 172.21.15.28:0/2419690947 wait complete. 2024-02-21T23:07:22.981 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-02-21T23:07:23.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.219+0000 7f0c09e00640 1 Processor -- start 2024-02-21T23:07:23.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.219+0000 7f0c09e00640 1 -- start start 2024-02-21T23:07:23.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.220+0000 7f0c09e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0c040c4580 0x7f0c040c12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:23.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.220+0000 7f0c09e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0c040c2c10 0x7f0c0406ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:23.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.221+0000 7f0c09e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c0406e220 con 0x7f0c04150240 2024-02-21T23:07:23.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.221+0000 7f0c09e00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c04150620 con 0x7f0c040c4580 2024-02-21T23:07:23.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.221+0000 7f0c09e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c04153a90 con 0x7f0c040c2c10 2024-02-21T23:07:23.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.221+0000 7f0c08a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f0c04150240 0x7f0c0406db10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60032/0 (socket says 172.21.15.28:60032) 2024-02-21T23:07:23.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.221+0000 7f0c08a00640 1 -- 172.21.15.28:0/1008775856 learned_addr learned my addr 172.21.15.28:0/1008775856 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:07:23.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.221+0000 7f0c03e00640 1 --2- 172.21.15.28:0/1008775856 >> v2:172.21.15.81:3300/0 conn(0x7f0c040c4580 0x7f0c040c12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:23.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.221+0000 7f0c09400640 1 --2- 172.21.15.28:0/1008775856 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0c040c2c10 0x7f0c0406ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:23.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.222+0000 7f0c03400640 1 -- 172.21.15.28:0/1008775856 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 134100480 0 0) 0x7f0c0406e220 con 0x7f0c04150240 2024-02-21T23:07:23.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.222+0000 7f0c03400640 1 -- 172.21.15.28:0/1008775856 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0be8003520 con 0x7f0c04150240 2024-02-21T23:07:23.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.222+0000 7f0c03e00640 1 -- 172.21.15.28:0/1008775856 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0c040c2c10 msgr2=0x7f0c0406ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:23.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.222+0000 7f0c03e00640 1 --2- 172.21.15.28:0/1008775856 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0c040c2c10 0x7f0c0406ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:23.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.222+0000 7f0c03400640 1 -- 172.21.15.28:0/1008775856 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 555796055 0 0) 0x7f0be8003520 con 0x7f0c04150240 2024-02-21T23:07:23.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.222+0000 7f0c03e00640 1 -- 172.21.15.28:0/1008775856 >> v1:172.21.15.28:6789/0 conn(0x7f0c04150240 legacy=0x7f0c0406db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:23.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.222+0000 7f0c03e00640 1 -- 172.21.15.28:0/1008775856 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c0406f780 con 0x7f0c040c4580 2024-02-21T23:07:23.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.224+0000 7f0c03e00640 1 --2- 172.21.15.28:0/1008775856 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0c040c4580 0x7f0c040c12c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f0bf00097a0 tx=0x7f0bf002b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf3ab753d2cbcba0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:23.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.224+0000 7f0c03400640 1 -- 172.21.15.28:0/1008775856 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf0037070 con 0x7f0c040c4580 2024-02-21T23:07:23.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.224+0000 7f0c03400640 1 -- 172.21.15.28:0/1008775856 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bf0030510 con 0x7f0c040c4580 2024-02-21T23:07:23.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.224+0000 7f0c03400640 1 -- 172.21.15.28:0/1008775856 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf4001670 con 0x7f0c040c4580 2024-02-21T23:07:23.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.225+0000 7f0c09e00640 1 -- 172.21.15.28:0/1008775856 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0c040c4580 msgr2=0x7f0c040c12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:23.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.225+0000 7f0c09e00640 1 --2- 172.21.15.28:0/1008775856 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0c040c4580 0x7f0c040c12c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f0bf00097a0 tx=0x7f0bf002b710 comp rx=0 tx=0).stop 2024-02-21T23:07:23.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.225+0000 7f0c09e00640 1 -- 172.21.15.28:0/1008775856 shutdown_connections 2024-02-21T23:07:23.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.225+0000 7f0c09e00640 1 --2- 172.21.15.28:0/1008775856 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0c040c2c10 0x7f0c0406ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:23.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.225+0000 7f0c09e00640 1 --2- 172.21.15.28:0/1008775856 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0c040c4580 0x7f0c040c12c0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:23.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.225+0000 7f0c09e00640 1 -- 172.21.15.28:0/1008775856 >> 172.21.15.28:0/1008775856 conn(0x7f0c040a8530 msgr2=0x7f0c040a8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:23.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.225+0000 7f0c09e00640 1 -- 172.21.15.28:0/1008775856 shutdown_connections 2024-02-21T23:07:23.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.225+0000 7f0c09e00640 1 -- 172.21.15.28:0/1008775856 wait complete. 2024-02-21T23:07:23.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.227+0000 7f0c09e00640 1 Processor -- start 2024-02-21T23:07:23.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.227+0000 7f0c09e00640 1 -- start start 2024-02-21T23:07:23.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.228+0000 7f0c09e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0c040c2c10 0x7f0c0414beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:23.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.228+0000 7f0c09e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c040c4580 0x7f0c040ab7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:23.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.228+0000 7f0c09e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0c04150240 0x7f0c041ec090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:23.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.228+0000 7f0c09e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c040c4a10 con 0x7f0c040c4580 2024-02-21T23:07:23.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.228+0000 7f0c09e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c04153720 con 0x7f0c040c2c10 2024-02-21T23:07:23.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.228+0000 7f0c09e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c040c4b90 con 0x7f0c04150240 2024-02-21T23:07:23.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.228+0000 7f0c03e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c040c4580 0x7f0c040ab7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:23.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.228+0000 7f0c03e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c040c4580 0x7f0c040ab7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50962/0 (socket says 172.21.15.28:50962) 2024-02-21T23:07:23.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.228+0000 7f0c03e00640 1 -- 172.21.15.28:0/1377704954 learned_addr learned my addr 172.21.15.28:0/1377704954 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:23.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.228+0000 7f0c08a00640 1 --2- 172.21.15.28:0/1377704954 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0c040c2c10 0x7f0c0414beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:23.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.228+0000 7f0c09400640 1 --2- 172.21.15.28:0/1377704954 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0c04150240 0x7f0c041ec090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:23.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.228+0000 7f0c03e00640 1 -- 172.21.15.28:0/1377704954 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0c04150240 msgr2=0x7f0c041ec090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:23.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.228+0000 7f0c03e00640 1 --2- 172.21.15.28:0/1377704954 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0c04150240 0x7f0c041ec090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:23.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.228+0000 7f0c03e00640 1 -- 172.21.15.28:0/1377704954 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0c040c2c10 msgr2=0x7f0c0414beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:23.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.228+0000 7f0c03e00640 1 --2- 172.21.15.28:0/1377704954 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0c040c2c10 0x7f0c0414beb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:23.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.228+0000 7f0c03e00640 1 -- 172.21.15.28:0/1377704954 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c04155670 con 0x7f0c040c4580 2024-02-21T23:07:23.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.229+0000 7f0c03e00640 1 --2- 172.21.15.28:0/1377704954 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c040c4580 0x7f0c040ab7d0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f0bf0035040 tx=0x7f0bf00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:23.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.229+0000 7f0c00c00640 1 -- 172.21.15.28:0/1377704954 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf0041030 con 0x7f0c040c4580 2024-02-21T23:07:23.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.229+0000 7f0c00c00640 1 -- 172.21.15.28:0/1377704954 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bf003c670 con 0x7f0c040c4580 2024-02-21T23:07:23.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.229+0000 7f0c00c00640 1 -- 172.21.15.28:0/1377704954 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf0037040 con 0x7f0c040c4580 2024-02-21T23:07:23.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.229+0000 7f0c09e00640 1 -- 172.21.15.28:0/1377704954 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c04156a70 con 0x7f0c040c4580 2024-02-21T23:07:23.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.229+0000 7f0c09e00640 1 -- 172.21.15.28:0/1377704954 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c04155910 con 0x7f0c040c4580 2024-02-21T23:07:23.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.230+0000 7f0c09e00640 1 -- 172.21.15.28:0/1377704954 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c0404f290 con 0x7f0c040c4580 2024-02-21T23:07:23.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.231+0000 7f0c00c00640 1 -- 172.21.15.28:0/1377704954 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0bf0044070 con 0x7f0c040c4580 2024-02-21T23:07:23.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.232+0000 7f0c00c00640 1 --2- 172.21.15.28:0/1377704954 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0be406b4d0 0x7f0be406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:23.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.233+0000 7f0c00c00640 1 -- 172.21.15.28:0/1377704954 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f0bf00ab980 con 0x7f0c040c4580 2024-02-21T23:07:23.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.233+0000 7f0c08a00640 1 --2- 172.21.15.28:0/1377704954 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0be406b4d0 0x7f0be406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:23.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.234+0000 7f0c08a00640 1 --2- 172.21.15.28:0/1377704954 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0be406b4d0 0x7f0be406d990 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f0bf40011e0 tx=0x7f0bf4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:23.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.237+0000 7f0c00c00640 1 -- 172.21.15.28:0/1377704954 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f0bf007a910 con 0x7f0c040c4580 2024-02-21T23:07:23.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.441+0000 7f0c09e00640 1 -- 172.21.15.28:0/1377704954 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f0c041565d0 con 0x7f0c040c4580 2024-02-21T23:07:23.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.447+0000 7f0c00c00640 1 -- 172.21.15.28:0/1377704954 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v35)=0 v35) v1 ==== 122+0+0 (secure 0 0 0) 0x7f0bf007a210 con 0x7f0c040c4580 2024-02-21T23:07:23.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.450+0000 7f0c09e00640 1 -- 172.21.15.28:0/1377704954 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0be406b4d0 msgr2=0x7f0be406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:23.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.450+0000 7f0c09e00640 1 --2- 172.21.15.28:0/1377704954 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0be406b4d0 0x7f0be406d990 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f0bf40011e0 tx=0x7f0bf4009040 comp rx=0 tx=0).stop 2024-02-21T23:07:23.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.450+0000 7f0c09e00640 1 -- 172.21.15.28:0/1377704954 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c040c4580 msgr2=0x7f0c040ab7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:23.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.450+0000 7f0c09e00640 1 --2- 172.21.15.28:0/1377704954 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c040c4580 0x7f0c040ab7d0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f0bf0035040 tx=0x7f0bf00046c0 comp rx=0 tx=0).stop 2024-02-21T23:07:23.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.450+0000 7f0c09e00640 1 -- 172.21.15.28:0/1377704954 shutdown_connections 2024-02-21T23:07:23.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.450+0000 7f0c09e00640 1 --2- 172.21.15.28:0/1377704954 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0be406b4d0 0x7f0be406d990 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:23.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.450+0000 7f0c09e00640 1 --2- 172.21.15.28:0/1377704954 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0c04150240 0x7f0c041ec090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:23.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.450+0000 7f0c09e00640 1 --2- 172.21.15.28:0/1377704954 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c040c4580 0x7f0c040ab7d0 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:23.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.450+0000 7f0c09e00640 1 --2- 172.21.15.28:0/1377704954 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0c040c2c10 0x7f0c0414beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:23.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.450+0000 7f0c09e00640 1 -- 172.21.15.28:0/1377704954 >> 172.21.15.28:0/1377704954 conn(0x7f0c040a8530 msgr2=0x7f0c0406ae70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:23.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.450+0000 7f0c09e00640 1 -- 172.21.15.28:0/1377704954 shutdown_connections 2024-02-21T23:07:23.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:23.450+0000 7f0c09e00640 1 -- 172.21.15.28:0/1377704954 wait complete. 2024-02-21T23:07:23.458 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-02-21T23:07:23.458 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-21T23:07:23.579 DEBUG:teuthology.orchestra.run.smithi081:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-21T23:07:23.705 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-02-21T23:07:23.706 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-21T23:07:23.706 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-21T23:07:23.767 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054608,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-02-21T23:07:23.767 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-02-21T23:07:23.767 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-21T23:07:23.823 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053833,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498812k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249404k,nr_inodes=812351,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498812k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-02-21T23:07:23.823 INFO:tasks.cephfs.cephfs_test_case:[, ] 2024-02-21T23:07:23.823 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:07:24.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.063+0000 7fc332c00640 1 Processor -- start 2024-02-21T23:07:24.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.064+0000 7fc332c00640 1 -- start start 2024-02-21T23:07:24.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.065+0000 7fc332c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fc32c06dbf0 0x7fc32c158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:24.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.065+0000 7fc332c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc32c155960 0x7fc32c159000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:24.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.065+0000 7fc332c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc32c0c74c0 con 0x7fc32c0aff90 2024-02-21T23:07:24.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.065+0000 7fc332c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc32c0c7270 con 0x7fc32c155960 2024-02-21T23:07:24.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.065+0000 7fc332c00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc32c158630 con 0x7fc32c06dbf0 2024-02-21T23:07:24.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.066+0000 7fc331800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fc32c0aff90 0x7fc32c0c51e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60040/0 (socket says 172.21.15.28:60040) 2024-02-21T23:07:24.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.066+0000 7fc331800640 1 -- 172.21.15.28:0/3763196208 learned_addr learned my addr 172.21.15.28:0/3763196208 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:07:24.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.066+0000 7fc332200640 1 --2- 172.21.15.28:0/3763196208 >> v2:172.21.15.28:3300/0 conn(0x7fc32c155960 0x7fc32c159000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:24.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.066+0000 7fc330e00640 1 --2- 172.21.15.28:0/3763196208 >> v2:172.21.15.81:3300/0 conn(0x7fc32c06dbf0 0x7fc32c158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:24.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.066+0000 7fc332200640 1 -- 172.21.15.28:0/3763196208 >> v2:172.21.15.81:3300/0 conn(0x7fc32c06dbf0 msgr2=0x7fc32c158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:24.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.066+0000 7fc32be00640 1 -- 172.21.15.28:0/3763196208 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2806387255 0 0) 0x7fc32c0c74c0 con 0x7fc32c0aff90 2024-02-21T23:07:24.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.066+0000 7fc332200640 1 --2- 172.21.15.28:0/3763196208 >> v2:172.21.15.81:3300/0 conn(0x7fc32c06dbf0 0x7fc32c158ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:24.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.066+0000 7fc332200640 1 -- 172.21.15.28:0/3763196208 >> v1:172.21.15.28:6789/0 conn(0x7fc32c0aff90 legacy=0x7fc32c0c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:24.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.066+0000 7fc332200640 1 -- 172.21.15.28:0/3763196208 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc32c0c77a0 con 0x7fc32c155960 2024-02-21T23:07:24.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.068+0000 7fc332200640 1 --2- 172.21.15.28:0/3763196208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc32c155960 0x7fc32c159000 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7fc32c0cb8c0 tx=0x7fc32402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aac0f11917283891 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:24.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.068+0000 7fc32be00640 1 -- 172.21.15.28:0/3763196208 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc324039070 con 0x7fc32c155960 2024-02-21T23:07:24.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.068+0000 7fc32be00640 1 -- 172.21.15.28:0/3763196208 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc32402cc80 con 0x7fc32c155960 2024-02-21T23:07:24.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.068+0000 7fc32be00640 1 -- 172.21.15.28:0/3763196208 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc324034660 con 0x7fc32c155960 2024-02-21T23:07:24.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.069+0000 7fc332c00640 1 -- 172.21.15.28:0/3763196208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc32c155960 msgr2=0x7fc32c159000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:24.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.069+0000 7fc332c00640 1 --2- 172.21.15.28:0/3763196208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc32c155960 0x7fc32c159000 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7fc32c0cb8c0 tx=0x7fc32402bed0 comp rx=0 tx=0).stop 2024-02-21T23:07:24.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.069+0000 7fc332c00640 1 -- 172.21.15.28:0/3763196208 shutdown_connections 2024-02-21T23:07:24.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.069+0000 7fc332c00640 1 --2- 172.21.15.28:0/3763196208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc32c155960 0x7fc32c159000 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:24.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.069+0000 7fc332c00640 1 --2- 172.21.15.28:0/3763196208 >> v2:172.21.15.81:3300/0 conn(0x7fc32c06dbf0 0x7fc32c158ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:24.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.069+0000 7fc332c00640 1 -- 172.21.15.28:0/3763196208 >> 172.21.15.28:0/3763196208 conn(0x7fc32c0ac370 msgr2=0x7fc32c0ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:24.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.069+0000 7fc332c00640 1 -- 172.21.15.28:0/3763196208 shutdown_connections 2024-02-21T23:07:24.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.069+0000 7fc332c00640 1 -- 172.21.15.28:0/3763196208 wait complete. 2024-02-21T23:07:24.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.071+0000 7fc332c00640 1 Processor -- start 2024-02-21T23:07:24.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.071+0000 7fc332c00640 1 -- start start 2024-02-21T23:07:24.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.072+0000 7fc332c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc32c06dbf0 0x7fc32c1eace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:24.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.072+0000 7fc332c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc32c0aff90 0x7fc32c1ed8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:24.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.072+0000 7fc332c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc32c155960 0x7fc32c1545a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:24.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.072+0000 7fc332c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc32c0c58f0 con 0x7fc32c06dbf0 2024-02-21T23:07:24.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.072+0000 7fc332c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc32c0be0c0 con 0x7fc32c0aff90 2024-02-21T23:07:24.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.072+0000 7fc332c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc32c0c5a70 con 0x7fc32c155960 2024-02-21T23:07:24.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.072+0000 7fc331800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc32c06dbf0 0x7fc32c1eace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:24.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.072+0000 7fc331800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc32c06dbf0 0x7fc32c1eace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50982/0 (socket says 172.21.15.28:50982) 2024-02-21T23:07:24.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.072+0000 7fc331800640 1 -- 172.21.15.28:0/2591037786 learned_addr learned my addr 172.21.15.28:0/2591037786 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:24.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.073+0000 7fc330e00640 1 --2- 172.21.15.28:0/2591037786 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc32c0aff90 0x7fc32c1ed8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:24.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.073+0000 7fc332200640 1 --2- 172.21.15.28:0/2591037786 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc32c155960 0x7fc32c1545a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:24.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.073+0000 7fc331800640 1 -- 172.21.15.28:0/2591037786 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc32c155960 msgr2=0x7fc32c1545a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:24.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.073+0000 7fc331800640 1 --2- 172.21.15.28:0/2591037786 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc32c155960 0x7fc32c1545a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:24.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.073+0000 7fc331800640 1 -- 172.21.15.28:0/2591037786 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc32c0aff90 msgr2=0x7fc32c1ed8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:24.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.073+0000 7fc331800640 1 --2- 172.21.15.28:0/2591037786 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc32c0aff90 0x7fc32c1ed8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:24.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.073+0000 7fc331800640 1 -- 172.21.15.28:0/2591037786 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc32c1eb220 con 0x7fc32c06dbf0 2024-02-21T23:07:24.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.073+0000 7fc332200640 1 --2- 172.21.15.28:0/2591037786 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc32c155960 0x7fc32c1545a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:07:24.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.073+0000 7fc331800640 1 --2- 172.21.15.28:0/2591037786 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc32c06dbf0 0x7fc32c1eace0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7fc32000c4d0 tx=0x7fc32000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:24.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.073+0000 7fc329600640 1 -- 172.21.15.28:0/2591037786 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc320018070 con 0x7fc32c06dbf0 2024-02-21T23:07:24.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.074+0000 7fc329600640 1 -- 172.21.15.28:0/2591037786 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc320007500 con 0x7fc32c06dbf0 2024-02-21T23:07:24.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.074+0000 7fc332c00640 1 -- 172.21.15.28:0/2591037786 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc32c0bda00 con 0x7fc32c06dbf0 2024-02-21T23:07:24.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.074+0000 7fc329600640 1 -- 172.21.15.28:0/2591037786 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc320004a20 con 0x7fc32c06dbf0 2024-02-21T23:07:24.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.074+0000 7fc332c00640 1 -- 172.21.15.28:0/2591037786 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc32c1ea4e0 con 0x7fc32c06dbf0 2024-02-21T23:07:24.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.075+0000 7fc332c00640 1 -- 172.21.15.28:0/2591037786 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc32c1ea2d0 con 0x7fc32c06dbf0 2024-02-21T23:07:24.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.076+0000 7fc329600640 1 -- 172.21.15.28:0/2591037786 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc32001b070 con 0x7fc32c06dbf0 2024-02-21T23:07:24.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.077+0000 7fc329600640 1 --2- 172.21.15.28:0/2591037786 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc30c06b4d0 0x7fc30c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:24.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.077+0000 7fc329600640 1 -- 172.21.15.28:0/2591037786 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5421+0+0 (secure 0 0 0) 0x7fc320089400 con 0x7fc32c06dbf0 2024-02-21T23:07:24.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.078+0000 7fc330e00640 1 --2- 172.21.15.28:0/2591037786 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc30c06b4d0 0x7fc30c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:24.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.079+0000 7fc330e00640 1 --2- 172.21.15.28:0/2591037786 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc30c06b4d0 0x7fc30c06d990 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fc32c1e8110 tx=0x7fc318008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:24.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.082+0000 7fc329600640 1 -- 172.21.15.28:0/2591037786 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc320058390 con 0x7fc32c06dbf0 2024-02-21T23:07:24.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.322+0000 7fc332c00640 1 -- 172.21.15.28:0/2591037786 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc32c04f290 con 0x7fc32c06dbf0 2024-02-21T23:07:24.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.323+0000 7fc329600640 1 -- 172.21.15.28:0/2591037786 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+3696 (secure 0 0 0) 0x7fc320057c90 con 0x7fc32c06dbf0 2024-02-21T23:07:24.313 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:07:24.313 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":95,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5567,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1194613007","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1194613007},{"type":"v1","addr":"172.21.15.81:6835","nonce":1194613007}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":5570,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/2315735157","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":2315735157},{"type":"v1","addr":"172.21.15.81:6837","nonce":2315735157}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":83},{"gid":7251,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2558379712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2558379712},{"type":"v1","addr":"172.21.15.28:6835","nonce":2558379712}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":91},{"gid":7359,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/120062681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":120062681},{"type":"v1","addr":"172.21.15.28:6833","nonce":120062681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":95}],"filesystems":[]} 2024-02-21T23:07:24.313 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 95 2024-02-21T23:07:24.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.326+0000 7fc332c00640 1 -- 172.21.15.28:0/2591037786 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc30c06b4d0 msgr2=0x7fc30c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:24.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.326+0000 7fc332c00640 1 --2- 172.21.15.28:0/2591037786 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc30c06b4d0 0x7fc30c06d990 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fc32c1e8110 tx=0x7fc318008040 comp rx=0 tx=0).stop 2024-02-21T23:07:24.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.326+0000 7fc332c00640 1 -- 172.21.15.28:0/2591037786 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc32c06dbf0 msgr2=0x7fc32c1eace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:24.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.326+0000 7fc332c00640 1 --2- 172.21.15.28:0/2591037786 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc32c06dbf0 0x7fc32c1eace0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7fc32000c4d0 tx=0x7fc32000c990 comp rx=0 tx=0).stop 2024-02-21T23:07:24.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.326+0000 7fc332c00640 1 -- 172.21.15.28:0/2591037786 shutdown_connections 2024-02-21T23:07:24.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.326+0000 7fc332c00640 1 --2- 172.21.15.28:0/2591037786 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc30c06b4d0 0x7fc30c06d990 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:24.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.326+0000 7fc332c00640 1 --2- 172.21.15.28:0/2591037786 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc32c155960 0x7fc32c1545a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:24.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.326+0000 7fc332c00640 1 --2- 172.21.15.28:0/2591037786 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc32c0aff90 0x7fc32c1ed8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:24.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.326+0000 7fc332c00640 1 --2- 172.21.15.28:0/2591037786 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc32c06dbf0 0x7fc32c1eace0 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:24.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.326+0000 7fc332c00640 1 -- 172.21.15.28:0/2591037786 >> 172.21.15.28:0/2591037786 conn(0x7fc32c0ac370 msgr2=0x7fc32c0cb200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:24.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.326+0000 7fc332c00640 1 -- 172.21.15.28:0/2591037786 shutdown_connections 2024-02-21T23:07:24.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.326+0000 7fc332c00640 1 -- 172.21.15.28:0/2591037786 wait complete. 2024-02-21T23:07:24.337 INFO:tasks.ceph.mds.a:Restarting daemon 2024-02-21T23:07:24.337 INFO:tasks.ceph.mds.a:Stopping old one... 2024-02-21T23:07:24.338 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-02-21T23:07:24.338 INFO:teuthology.orchestra.run:waiting for 300 2024-02-21T23:07:24.338 INFO:tasks.ceph.mds.c:Restarting daemon 2024-02-21T23:07:24.338 INFO:tasks.ceph.mds.c:Stopping old one... 2024-02-21T23:07:24.338 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-02-21T23:07:24.338 INFO:teuthology.orchestra.run:waiting for 300 2024-02-21T23:07:24.339 INFO:tasks.ceph.mds.b:Restarting daemon 2024-02-21T23:07:24.339 INFO:tasks.ceph.mds.b:Stopping old one... 2024-02-21T23:07:24.339 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-02-21T23:07:24.339 INFO:teuthology.orchestra.run:waiting for 300 2024-02-21T23:07:24.339 INFO:tasks.ceph.mds.d:Restarting daemon 2024-02-21T23:07:24.339 INFO:tasks.ceph.mds.d:Stopping old one... 2024-02-21T23:07:24.340 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-02-21T23:07:24.340 INFO:teuthology.orchestra.run:waiting for 300 2024-02-21T23:07:24.357 INFO:tasks.ceph.mds.c:Stopped 2024-02-21T23:07:24.357 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-02-21T23:07:24.360 INFO:tasks.ceph.mds.a:Stopped 2024-02-21T23:07:24.360 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-02-21T23:07:24.364 INFO:tasks.ceph.mds.d:Stopped 2024-02-21T23:07:24.364 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-02-21T23:07:24.371 INFO:tasks.ceph.mds.b:Stopped 2024-02-21T23:07:24.371 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-02-21T23:07:24.407 INFO:tasks.ceph.mds.c:Started 2024-02-21T23:07:24.408 DEBUG:teuthology.parallel:result is None 2024-02-21T23:07:24.414 INFO:tasks.ceph.mds.a:Started 2024-02-21T23:07:24.415 DEBUG:teuthology.parallel:result is None 2024-02-21T23:07:24.418 INFO:tasks.ceph.mds.d:Started 2024-02-21T23:07:24.418 DEBUG:teuthology.parallel:result is None 2024-02-21T23:07:24.424 INFO:tasks.ceph.mds.b:Started 2024-02-21T23:07:24.425 DEBUG:teuthology.parallel:result is None 2024-02-21T23:07:24.425 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-02-21T23:07:24.566 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-02-21T23:07:24.571 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-02-21T23:07:24.577 INFO:tasks.ceph.mds.d.smithi081.stdout:starting mds.d at 2024-02-21T23:07:24.580 INFO:tasks.ceph.mds.b.smithi081.stdout:starting mds.b at 2024-02-21T23:07:24.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.624+0000 7fd943a00640 1 Processor -- start 2024-02-21T23:07:24.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.624+0000 7fd943a00640 1 -- start start 2024-02-21T23:07:24.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.625+0000 7fd943a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fd93c156c80 0x7fd93c06e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:24.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.625+0000 7fd943a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd93c0c11f0 0x7fd93c06ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:24.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.625+0000 7fd943a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd93c159fd0 con 0x7fd93c072e60 2024-02-21T23:07:24.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.625+0000 7fd943a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd93c0c5b60 con 0x7fd93c0c11f0 2024-02-21T23:07:24.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.625+0000 7fd943a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd93c0c5db0 con 0x7fd93c156c80 2024-02-21T23:07:24.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.625+0000 7fd941a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd93c0c11f0 0x7fd93c06ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:24.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.625+0000 7fd941a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd93c0c11f0 0x7fd93c06ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51024/0 (socket says 172.21.15.28:51024) 2024-02-21T23:07:24.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.625+0000 7fd941a00640 1 -- 172.21.15.28:0/1975444253 learned_addr learned my addr 172.21.15.28:0/1975444253 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:24.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.625+0000 7fd941000640 1 --2- 172.21.15.28:0/1975444253 >> v2:172.21.15.81:3300/0 conn(0x7fd93c156c80 0x7fd93c06e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:24.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.626+0000 7fd941a00640 1 -- 172.21.15.28:0/1975444253 >> v2:172.21.15.81:3300/0 conn(0x7fd93c156c80 msgr2=0x7fd93c06e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:24.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.626+0000 7fd941a00640 1 --2- 172.21.15.28:0/1975444253 >> v2:172.21.15.81:3300/0 conn(0x7fd93c156c80 0x7fd93c06e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:24.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.626+0000 7fd941a00640 1 -- 172.21.15.28:0/1975444253 >> v1:172.21.15.81:6789/0 conn(0x7fd93c072e60 legacy=0x7fd93c0c4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:24.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.626+0000 7fd941a00640 1 -- 172.21.15.28:0/1975444253 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd93c15a290 con 0x7fd93c0c11f0 2024-02-21T23:07:24.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.626+0000 7fd93b400640 1 -- 172.21.15.28:0/1975444253 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1694290307 0 0) 0x7fd93c159fd0 con 0x7fd93c072e60 2024-02-21T23:07:24.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.627+0000 7fd941a00640 1 --2- 172.21.15.28:0/1975444253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd93c0c11f0 0x7fd93c06ed80 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7fd93c0c5ce0 tx=0x7fd93402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbb169ea0a7f2ff3 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:24.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.627+0000 7fd93b400640 1 -- 172.21.15.28:0/1975444253 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd934034070 con 0x7fd93c0c11f0 2024-02-21T23:07:24.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.627+0000 7fd93b400640 1 -- 172.21.15.28:0/1975444253 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd934030860 con 0x7fd93c0c11f0 2024-02-21T23:07:24.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.627+0000 7fd93b400640 1 -- 172.21.15.28:0/1975444253 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd934030c30 con 0x7fd93c0c11f0 2024-02-21T23:07:24.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.627+0000 7fd943a00640 1 -- 172.21.15.28:0/1975444253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd93c0c11f0 msgr2=0x7fd93c06ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:24.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.627+0000 7fd943a00640 1 --2- 172.21.15.28:0/1975444253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd93c0c11f0 0x7fd93c06ed80 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7fd93c0c5ce0 tx=0x7fd93402b9a0 comp rx=0 tx=0).stop 2024-02-21T23:07:24.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.628+0000 7fd943a00640 1 -- 172.21.15.28:0/1975444253 shutdown_connections 2024-02-21T23:07:24.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.628+0000 7fd943a00640 1 --2- 172.21.15.28:0/1975444253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd93c0c11f0 0x7fd93c06ed80 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:24.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.628+0000 7fd943a00640 1 --2- 172.21.15.28:0/1975444253 >> v2:172.21.15.81:3300/0 conn(0x7fd93c156c80 0x7fd93c06e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:24.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.628+0000 7fd943a00640 1 -- 172.21.15.28:0/1975444253 >> 172.21.15.28:0/1975444253 conn(0x7fd93c06b820 msgr2=0x7fd93c0c67f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:24.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.628+0000 7fd943a00640 1 -- 172.21.15.28:0/1975444253 shutdown_connections 2024-02-21T23:07:24.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.628+0000 7fd943a00640 1 -- 172.21.15.28:0/1975444253 wait complete. 2024-02-21T23:07:24.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.629+0000 7fd943a00640 1 Processor -- start 2024-02-21T23:07:24.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.630+0000 7fd943a00640 1 -- start start 2024-02-21T23:07:24.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.630+0000 7fd943a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd93c0c11f0 0x7fd93c15c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:24.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.630+0000 7fd943a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd93c156c80 0x7fd93c15d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:24.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.630+0000 7fd943a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd93c0a95f0 0x7fd93c14ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:24.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.630+0000 7fd943a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd93c0c51e0 con 0x7fd93c0c11f0 2024-02-21T23:07:24.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.630+0000 7fd943a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd93c0c0e10 con 0x7fd93c156c80 2024-02-21T23:07:24.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.630+0000 7fd943a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd93c0c5360 con 0x7fd93c0a95f0 2024-02-21T23:07:24.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.631+0000 7fd941000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd93c0c11f0 0x7fd93c15c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:24.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.631+0000 7fd941000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd93c0c11f0 0x7fd93c15c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51034/0 (socket says 172.21.15.28:51034) 2024-02-21T23:07:24.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.631+0000 7fd941000640 1 -- 172.21.15.28:0/3769222695 learned_addr learned my addr 172.21.15.28:0/3769222695 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:24.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.631+0000 7fd941000640 1 -- 172.21.15.28:0/3769222695 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd93c0a95f0 msgr2=0x7fd93c14ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:24.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.631+0000 7fd941a00640 1 --2- 172.21.15.28:0/3769222695 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd93c0a95f0 0x7fd93c14ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:24.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.631+0000 7fd93be00640 1 --2- 172.21.15.28:0/3769222695 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd93c156c80 0x7fd93c15d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:24.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.631+0000 7fd941000640 1 --2- 172.21.15.28:0/3769222695 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd93c0a95f0 0x7fd93c14ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:24.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.631+0000 7fd941000640 1 -- 172.21.15.28:0/3769222695 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd93c156c80 msgr2=0x7fd93c15d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:24.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.631+0000 7fd941000640 1 --2- 172.21.15.28:0/3769222695 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd93c156c80 0x7fd93c15d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:24.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.631+0000 7fd941000640 1 -- 172.21.15.28:0/3769222695 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd93c14f810 con 0x7fd93c0c11f0 2024-02-21T23:07:24.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.631+0000 7fd941a00640 1 --2- 172.21.15.28:0/3769222695 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd93c0a95f0 0x7fd93c14ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:07:24.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.631+0000 7fd93be00640 1 --2- 172.21.15.28:0/3769222695 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd93c156c80 0x7fd93c15d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:07:24.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.631+0000 7fd941000640 1 --2- 172.21.15.28:0/3769222695 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd93c0c11f0 0x7fd93c15c790 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7fd92c00efc0 tx=0x7fd92c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:24.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.632+0000 7fd938c00640 1 -- 172.21.15.28:0/3769222695 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd92c018020 con 0x7fd93c0c11f0 2024-02-21T23:07:24.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.632+0000 7fd938c00640 1 -- 172.21.15.28:0/3769222695 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd92c014480 con 0x7fd93c0c11f0 2024-02-21T23:07:24.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.632+0000 7fd938c00640 1 -- 172.21.15.28:0/3769222695 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd92c014850 con 0x7fd93c0c11f0 2024-02-21T23:07:24.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.632+0000 7fd943a00640 1 -- 172.21.15.28:0/3769222695 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd93c15aa60 con 0x7fd93c0c11f0 2024-02-21T23:07:24.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.632+0000 7fd943a00640 1 -- 172.21.15.28:0/3769222695 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd93c15d6a0 con 0x7fd93c0c11f0 2024-02-21T23:07:24.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.633+0000 7fd943a00640 1 -- 172.21.15.28:0/3769222695 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd93c15ac30 con 0x7fd93c0c11f0 2024-02-21T23:07:24.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.634+0000 7fd938c00640 1 -- 172.21.15.28:0/3769222695 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd92c005d10 con 0x7fd93c0c11f0 2024-02-21T23:07:24.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.635+0000 7fd938c00640 1 --2- 172.21.15.28:0/3769222695 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd91806b4d0 0x7fd91806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:24.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.635+0000 7fd938c00640 1 -- 172.21.15.28:0/3769222695 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5421+0+0 (secure 0 0 0) 0x7fd92c08abe0 con 0x7fd93c0c11f0 2024-02-21T23:07:24.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.635+0000 7fd93be00640 1 --2- 172.21.15.28:0/3769222695 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd91806b4d0 0x7fd91806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:24.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.636+0000 7fd93be00640 1 --2- 172.21.15.28:0/3769222695 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd91806b4d0 0x7fd91806d990 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fd93c0c6e30 tx=0x7fd924009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:24.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.639+0000 7fd938c00640 1 -- 172.21.15.28:0/3769222695 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd92c022070 con 0x7fd93c0c11f0 2024-02-21T23:07:24.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.857+0000 7fd943a00640 1 -- 172.21.15.28:0/3769222695 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fd93c06f400 con 0x7fd93c0c11f0 2024-02-21T23:07:24.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.993+0000 7fd938c00640 1 -- 172.21.15.28:0/3769222695 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v104) v1 ==== 97+0+0 (secure 0 0 0) 0x7fd92c010040 con 0x7fd93c0c11f0 2024-02-21T23:07:24.983 INFO:teuthology.orchestra.run.smithi028.stderr: removed all blocklist entries 2024-02-21T23:07:24.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.998+0000 7fd943a00640 1 -- 172.21.15.28:0/3769222695 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd91806b4d0 msgr2=0x7fd91806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:24.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.998+0000 7fd943a00640 1 --2- 172.21.15.28:0/3769222695 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd91806b4d0 0x7fd91806d990 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fd93c0c6e30 tx=0x7fd924009040 comp rx=0 tx=0).stop 2024-02-21T23:07:24.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.998+0000 7fd943a00640 1 -- 172.21.15.28:0/3769222695 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd93c0c11f0 msgr2=0x7fd93c15c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:24.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.998+0000 7fd943a00640 1 --2- 172.21.15.28:0/3769222695 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd93c0c11f0 0x7fd93c15c790 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7fd92c00efc0 tx=0x7fd92c00c490 comp rx=0 tx=0).stop 2024-02-21T23:07:24.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.998+0000 7fd943a00640 1 -- 172.21.15.28:0/3769222695 shutdown_connections 2024-02-21T23:07:24.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.998+0000 7fd943a00640 1 --2- 172.21.15.28:0/3769222695 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd91806b4d0 0x7fd91806d990 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:24.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.998+0000 7fd943a00640 1 --2- 172.21.15.28:0/3769222695 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd93c0a95f0 0x7fd93c14ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:24.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.998+0000 7fd943a00640 1 --2- 172.21.15.28:0/3769222695 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd93c156c80 0x7fd93c15d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:24.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.998+0000 7fd943a00640 1 --2- 172.21.15.28:0/3769222695 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd93c0c11f0 0x7fd93c15c790 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:24.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.998+0000 7fd943a00640 1 -- 172.21.15.28:0/3769222695 >> 172.21.15.28:0/3769222695 conn(0x7fd93c06b820 msgr2=0x7fd93c0c4ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:24.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.998+0000 7fd943a00640 1 -- 172.21.15.28:0/3769222695 shutdown_connections 2024-02-21T23:07:24.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:24.998+0000 7fd943a00640 1 -- 172.21.15.28:0/3769222695 wait complete. 2024-02-21T23:07:25.013 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-02-21T23:07:25.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.242+0000 7fd2b8200640 1 Processor -- start 2024-02-21T23:07:25.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.242+0000 7fd2b8200640 1 -- start start 2024-02-21T23:07:25.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.244+0000 7fd2b8200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd2b00c11f0 0x7fd2b0153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:25.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.244+0000 7fd2b8200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fd2b00c15d0 0x7fd2b006dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:25.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.244+0000 7fd2b8200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd2b0156350 0x7fd2b01567a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:25.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.244+0000 7fd2b8200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2b00a5ec0 con 0x7fd2b0156350 2024-02-21T23:07:25.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.244+0000 7fd2b8200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2b00c1a30 con 0x7fd2b00c15d0 2024-02-21T23:07:25.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.244+0000 7fd2b8200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b0152e70 con 0x7fd2b00c11f0 2024-02-21T23:07:25.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.244+0000 7fd2b6200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd2b0156350 0x7fd2b01567a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:25.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.244+0000 7fd2b6200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd2b0156350 0x7fd2b01567a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59304/0 (socket says 172.21.15.28:59304) 2024-02-21T23:07:25.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.244+0000 7fd2b6200640 1 -- 172.21.15.28:0/315113859 learned_addr learned my addr 172.21.15.28:0/315113859 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:25.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.245+0000 7fd2b5800640 1 --2- 172.21.15.28:0/315113859 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd2b00c11f0 0x7fd2b0153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:25.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.245+0000 7fd2b4e00640 1 --2- 172.21.15.28:0/315113859 >> v2:172.21.15.81:3300/0 conn(0x7fd2b00c15d0 0x7fd2b006dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:25.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.245+0000 7fd2b6200640 1 -- 172.21.15.28:0/315113859 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd2b00c11f0 msgr2=0x7fd2b0153950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:25.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.245+0000 7fd2b6200640 1 --2- 172.21.15.28:0/315113859 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd2b00c11f0 0x7fd2b0153950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:25.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.245+0000 7fd2b6200640 1 -- 172.21.15.28:0/315113859 >> v2:172.21.15.81:3300/0 conn(0x7fd2b00c15d0 msgr2=0x7fd2b006dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:25.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.245+0000 7fd2b6200640 1 --2- 172.21.15.28:0/315113859 >> v2:172.21.15.81:3300/0 conn(0x7fd2b00c15d0 0x7fd2b006dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:25.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.245+0000 7fd2b6200640 1 -- 172.21.15.28:0/315113859 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2b0153150 con 0x7fd2b0156350 2024-02-21T23:07:25.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.246+0000 7fd2b6200640 1 --2- 172.21.15.28:0/315113859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2b0156350 0x7fd2b01567a0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7fd2b006fb00 tx=0x7fd2a802bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=116720094d3472c0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:25.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.246+0000 7fd2afe00640 1 -- 172.21.15.28:0/315113859 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2a802c950 con 0x7fd2b0156350 2024-02-21T23:07:25.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.247+0000 7fd2afe00640 1 -- 172.21.15.28:0/315113859 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2a802caf0 con 0x7fd2b0156350 2024-02-21T23:07:25.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.247+0000 7fd2afe00640 1 -- 172.21.15.28:0/315113859 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2a80343f0 con 0x7fd2b0156350 2024-02-21T23:07:25.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.247+0000 7fd2b8200640 1 -- 172.21.15.28:0/315113859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2b0156350 msgr2=0x7fd2b01567a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:25.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.247+0000 7fd2b8200640 1 --2- 172.21.15.28:0/315113859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2b0156350 0x7fd2b01567a0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7fd2b006fb00 tx=0x7fd2a802bdb0 comp rx=0 tx=0).stop 2024-02-21T23:07:25.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.247+0000 7fd2b8200640 1 -- 172.21.15.28:0/315113859 shutdown_connections 2024-02-21T23:07:25.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.247+0000 7fd2b8200640 1 --2- 172.21.15.28:0/315113859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2b0156350 0x7fd2b01567a0 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:25.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.247+0000 7fd2b8200640 1 --2- 172.21.15.28:0/315113859 >> v2:172.21.15.81:3300/0 conn(0x7fd2b00c15d0 0x7fd2b006dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:25.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.247+0000 7fd2b8200640 1 --2- 172.21.15.28:0/315113859 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd2b00c11f0 0x7fd2b0153950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:25.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.247+0000 7fd2b8200640 1 -- 172.21.15.28:0/315113859 >> 172.21.15.28:0/315113859 conn(0x7fd2b00c67f0 msgr2=0x7fd2b00c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:25.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.247+0000 7fd2b8200640 1 -- 172.21.15.28:0/315113859 shutdown_connections 2024-02-21T23:07:25.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.247+0000 7fd2b8200640 1 -- 172.21.15.28:0/315113859 wait complete. 2024-02-21T23:07:25.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.249+0000 7fd2b8200640 1 Processor -- start 2024-02-21T23:07:25.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.249+0000 7fd2b8200640 1 -- start start 2024-02-21T23:07:25.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.250+0000 7fd2b8200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd2b00c11f0 0x7fd2b00c5da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:25.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.250+0000 7fd2b8200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2b00c15d0 0x7fd2b0147f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:25.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.250+0000 7fd2b8200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2b0156350 0x7fd2b0148440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:25.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.250+0000 7fd2b8200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b00c4c00 con 0x7fd2b00c15d0 2024-02-21T23:07:25.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.250+0000 7fd2b8200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b014d1c0 con 0x7fd2b0156350 2024-02-21T23:07:25.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.250+0000 7fd2b8200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b00a6110 con 0x7fd2b00c11f0 2024-02-21T23:07:25.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.250+0000 7fd2b4e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2b00c15d0 0x7fd2b0147f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:25.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.251+0000 7fd2b4e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2b00c15d0 0x7fd2b0147f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59314/0 (socket says 172.21.15.28:59314) 2024-02-21T23:07:25.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.251+0000 7fd2b4e00640 1 -- 172.21.15.28:0/2441944785 learned_addr learned my addr 172.21.15.28:0/2441944785 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:25.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.251+0000 7fd2b6200640 1 --2- 172.21.15.28:0/2441944785 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2b0156350 0x7fd2b0148440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:25.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.251+0000 7fd2b5800640 1 --2- 172.21.15.28:0/2441944785 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd2b00c11f0 0x7fd2b00c5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:25.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.251+0000 7fd2b4e00640 1 -- 172.21.15.28:0/2441944785 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd2b00c11f0 msgr2=0x7fd2b00c5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:25.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.251+0000 7fd2b4e00640 1 --2- 172.21.15.28:0/2441944785 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd2b00c11f0 0x7fd2b00c5da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:25.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.251+0000 7fd2b4e00640 1 -- 172.21.15.28:0/2441944785 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2b0156350 msgr2=0x7fd2b0148440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:25.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.251+0000 7fd2b4e00640 1 --2- 172.21.15.28:0/2441944785 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2b0156350 0x7fd2b0148440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:25.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.251+0000 7fd2b4e00640 1 -- 172.21.15.28:0/2441944785 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2b00713f0 con 0x7fd2b00c15d0 2024-02-21T23:07:25.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.251+0000 7fd2b4e00640 1 --2- 172.21.15.28:0/2441944785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2b00c15d0 0x7fd2b0147f00 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7fd29c00d5f0 tx=0x7fd29c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:25.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.252+0000 7fd2ad600640 1 -- 172.21.15.28:0/2441944785 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29c0125f0 con 0x7fd2b00c15d0 2024-02-21T23:07:25.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.252+0000 7fd2ad600640 1 -- 172.21.15.28:0/2441944785 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd29c019070 con 0x7fd2b00c15d0 2024-02-21T23:07:25.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.252+0000 7fd2ad600640 1 -- 172.21.15.28:0/2441944785 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29c0129d0 con 0x7fd2b00c15d0 2024-02-21T23:07:25.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.252+0000 7fd2b8200640 1 -- 172.21.15.28:0/2441944785 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2b01eee50 con 0x7fd2b00c15d0 2024-02-21T23:07:25.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.252+0000 7fd2b8200640 1 -- 172.21.15.28:0/2441944785 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2b00c62e0 con 0x7fd2b00c15d0 2024-02-21T23:07:25.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.253+0000 7fd2b8200640 1 -- 172.21.15.28:0/2441944785 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2b0067f30 con 0x7fd2b00c15d0 2024-02-21T23:07:25.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.254+0000 7fd2ad600640 1 -- 172.21.15.28:0/2441944785 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd29c012b70 con 0x7fd2b00c15d0 2024-02-21T23:07:25.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.255+0000 7fd2ad600640 1 --2- 172.21.15.28:0/2441944785 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd29006b500 0x7fd29006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:25.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.255+0000 7fd2ad600640 1 -- 172.21.15.28:0/2441944785 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fd29c08a410 con 0x7fd2b00c15d0 2024-02-21T23:07:25.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.255+0000 7fd2b5800640 1 --2- 172.21.15.28:0/2441944785 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd29006b500 0x7fd29006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:25.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.257+0000 7fd2b5800640 1 --2- 172.21.15.28:0/2441944785 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd29006b500 0x7fd29006d9c0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fd2a4006fd0 tx=0x7fd2a4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:25.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.260+0000 7fd2ad600640 1 -- 172.21.15.28:0/2441944785 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd29c059440 con 0x7fd2b00c15d0 2024-02-21T23:07:25.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.535+0000 7fd2b8200640 1 -- 172.21.15.28:0/2441944785 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fd2b00c0960 con 0x7fd2b00c15d0 2024-02-21T23:07:25.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.536+0000 7fd2ad600640 1 -- 172.21.15.28:0/2441944785 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v14) v1 ==== 80+0+4724 (secure 0 0 0) 0x7fd29c058d40 con 0x7fd2b00c15d0 2024-02-21T23:07:25.525 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:07:25.525 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-21T23:07:25.525 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_dump": [ 2024-02-21T23:07:25.525 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:07:25.525 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.a", 2024-02-21T23:07:25.525 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDSgNZl4FY1KRAAJSK5Ibb0ibKiKXYpTLgtmQ==", 2024-02-21T23:07:25.525 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:07:25.525 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:07:25.525 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-02-21T23:07:25.526 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-02-21T23:07:25.526 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:07:25.526 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:07:25.526 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:07:25.526 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:07:25.526 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.b", 2024-02-21T23:07:25.526 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDTgNZlwRcAERAAgsYmmkuobKelBEJVCRpXUg==", 2024-02-21T23:07:25.526 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:07:25.526 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:07:25.526 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-02-21T23:07:25.526 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-02-21T23:07:25.526 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:07:25.526 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:07:25.526 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:07:25.527 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:07:25.527 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.c", 2024-02-21T23:07:25.527 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDSgNZloFpQOxAAK2WBArXRVNTgqH9WWDEPDA==", 2024-02-21T23:07:25.527 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:07:25.527 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:07:25.527 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-02-21T23:07:25.527 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-02-21T23:07:25.527 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:07:25.527 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:07:25.527 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:07:25.527 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:07:25.527 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.d", 2024-02-21T23:07:25.527 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDTgNZlblOmIRAAxe2HfSKGSkkzJDN4aVdzgQ==", 2024-02-21T23:07:25.528 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:07:25.528 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:07:25.528 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-02-21T23:07:25.528 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-02-21T23:07:25.528 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:07:25.528 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:07:25.528 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:07:25.528 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:07:25.529 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.0", 2024-02-21T23:07:25.529 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDVgNZlxNJbLxAAXot7Q3chrLEBsFNO8kT6QQ==", 2024-02-21T23:07:25.529 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:07:25.529 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:07:25.529 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:07:25.529 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:07:25.529 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:07:25.529 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:07:25.529 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:07:25.529 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.1", 2024-02-21T23:07:25.529 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDYgNZlZZfWORAAr4XkOxLYLmumVbu362N2HQ==", 2024-02-21T23:07:25.529 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:07:25.529 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:07:25.529 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:07:25.530 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:07:25.530 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:07:25.530 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:07:25.530 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:07:25.530 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.2", 2024-02-21T23:07:25.530 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDcgNZl9mcDGBAAIZGOF8IUMenwo386x95mwg==", 2024-02-21T23:07:25.530 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:07:25.530 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:07:25.530 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:07:25.530 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:07:25.530 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:07:25.530 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:07:25.530 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:07:25.530 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.3", 2024-02-21T23:07:25.530 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDfgNZl8ALSMBAAag9cmGH0/4xALWSUmrZowg==", 2024-02-21T23:07:25.531 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:07:25.531 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:07:25.531 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:07:25.531 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:07:25.531 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:07:25.531 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:07:25.531 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:07:25.531 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.4", 2024-02-21T23:07:25.531 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDngNZlBgeKABAAeW99RVfCBvB1lp7w7mQBlQ==", 2024-02-21T23:07:25.531 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:07:25.531 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:07:25.531 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:07:25.531 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:07:25.531 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:07:25.531 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:07:25.532 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:07:25.532 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.5", 2024-02-21T23:07:25.532 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDqgNZljQG3ChAAY/IZEIcvcGNvlFGIWib3gg==", 2024-02-21T23:07:25.532 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:07:25.532 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:07:25.532 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:07:25.532 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:07:25.532 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:07:25.532 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:07:25.532 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:07:25.532 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.6", 2024-02-21T23:07:25.532 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDtgNZlKnakIxAAkW5xlkshzvEHJdmpgPxR+g==", 2024-02-21T23:07:25.532 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:07:25.532 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:07:25.533 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:07:25.533 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:07:25.533 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:07:25.533 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:07:25.533 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:07:25.533 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.7", 2024-02-21T23:07:25.533 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDxgNZlTWxYARAASX1/gmqO+8mQPP8uA4xjiQ==", 2024-02-21T23:07:25.533 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:07:25.533 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:07:25.533 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:07:25.533 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:07:25.533 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:07:25.533 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:07:25.533 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:07:25.533 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.0", 2024-02-21T23:07:25.534 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDTgNZlILvYLxAAoREWxKKT7YcXnhkgk8YvoQ==", 2024-02-21T23:07:25.534 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:07:25.534 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:07:25.534 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow r", 2024-02-21T23:07:25.534 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-02-21T23:07:25.534 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:07:25.534 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:07:25.534 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:07:25.534 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.1", 2024-02-21T23:07:25.534 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDTgNZlm7ImOxAA8iRcXlzFPZPaXQfsKhuO/g==", 2024-02-21T23:07:25.534 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:07:25.534 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:07:25.534 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-02-21T23:07:25.535 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-02-21T23:07:25.535 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-02-21T23:07:25.535 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:07:25.535 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:07:25.535 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:07:25.535 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.admin", 2024-02-21T23:07:25.535 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDRgNZlBYd1GhAAbK5qBCIkf0Ej5nEyPnTxIQ==", 2024-02-21T23:07:25.535 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:07:25.535 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-02-21T23:07:25.535 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-02-21T23:07:25.535 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-02-21T23:07:25.535 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:07:25.535 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:07:25.535 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:07:25.535 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:07:25.536 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.x", 2024-02-21T23:07:25.536 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDSgNZlJYKeHBAAYPskmLRr7Jw44TdKzM/JMg==", 2024-02-21T23:07:25.536 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:07:25.536 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-02-21T23:07:25.536 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-02-21T23:07:25.536 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:07:25.536 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:07:25.536 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:07:25.536 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:07:25.536 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.y", 2024-02-21T23:07:25.536 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDSgNZlSRkOEBAAOhRj9PsZDpjoV6WT4drrZA==", 2024-02-21T23:07:25.536 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:07:25.536 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-02-21T23:07:25.536 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-02-21T23:07:25.536 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:07:25.537 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:07:25.537 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:07:25.537 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-21T23:07:25.537 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-21T23:07:25.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.538+0000 7fd2b8200640 1 -- 172.21.15.28:0/2441944785 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd29006b500 msgr2=0x7fd29006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:25.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.538+0000 7fd2b8200640 1 --2- 172.21.15.28:0/2441944785 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd29006b500 0x7fd29006d9c0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fd2a4006fd0 tx=0x7fd2a4008040 comp rx=0 tx=0).stop 2024-02-21T23:07:25.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.538+0000 7fd2b8200640 1 -- 172.21.15.28:0/2441944785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2b00c15d0 msgr2=0x7fd2b0147f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:25.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.538+0000 7fd2b8200640 1 --2- 172.21.15.28:0/2441944785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2b00c15d0 0x7fd2b0147f00 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7fd29c00d5f0 tx=0x7fd29c00dab0 comp rx=0 tx=0).stop 2024-02-21T23:07:25.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.538+0000 7fd2b8200640 1 -- 172.21.15.28:0/2441944785 shutdown_connections 2024-02-21T23:07:25.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.538+0000 7fd2b8200640 1 --2- 172.21.15.28:0/2441944785 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd29006b500 0x7fd29006d9c0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:25.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.538+0000 7fd2b8200640 1 --2- 172.21.15.28:0/2441944785 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2b0156350 0x7fd2b0148440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:25.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.538+0000 7fd2b8200640 1 --2- 172.21.15.28:0/2441944785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2b00c15d0 0x7fd2b0147f00 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:25.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.538+0000 7fd2b8200640 1 --2- 172.21.15.28:0/2441944785 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd2b00c11f0 0x7fd2b00c5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:25.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.538+0000 7fd2b8200640 1 -- 172.21.15.28:0/2441944785 >> 172.21.15.28:0/2441944785 conn(0x7fd2b00c67f0 msgr2=0x7fd2b00bfc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:25.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.538+0000 7fd2b8200640 1 -- 172.21.15.28:0/2441944785 shutdown_connections 2024-02-21T23:07:25.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.538+0000 7fd2b8200640 1 -- 172.21.15.28:0/2441944785 wait complete. 2024-02-21T23:07:25.548 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:07:25.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.784+0000 7fbc56800640 1 Processor -- start 2024-02-21T23:07:25.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.784+0000 7fbc56800640 1 -- start start 2024-02-21T23:07:25.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.785+0000 7fbc56800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fbc500c11f0 0x7fbc50153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:25.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.785+0000 7fbc56800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc50152e70 con 0x7fbc5015b6d0 2024-02-21T23:07:25.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.785+0000 7fbc56800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc50153150 con 0x7fbc500c15d0 2024-02-21T23:07:25.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.785+0000 7fbc56800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc500c1a60 con 0x7fbc500c11f0 2024-02-21T23:07:25.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.786+0000 7fbc4fe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fbc5015b6d0 0x7fbc5015dac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:45874/0 (socket says 172.21.15.28:45874) 2024-02-21T23:07:25.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.786+0000 7fbc4fe00640 1 -- 172.21.15.28:0/1505395661 learned_addr learned my addr 172.21.15.28:0/1505395661 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:07:25.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.786+0000 7fbc4f400640 1 --2- 172.21.15.28:0/1505395661 >> v2:172.21.15.81:3300/0 conn(0x7fbc500c11f0 0x7fbc50153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:25.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.786+0000 7fbc4ea00640 1 -- 172.21.15.28:0/1505395661 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 251923147 0 0) 0x7fbc50152e70 con 0x7fbc5015b6d0 2024-02-21T23:07:25.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.786+0000 7fbc4ea00640 1 -- 172.21.15.28:0/1505395661 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc34003520 con 0x7fbc5015b6d0 2024-02-21T23:07:25.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.786+0000 7fbc4ea00640 1 -- 172.21.15.28:0/1505395661 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3585423669 0 0) 0x7fbc34003520 con 0x7fbc5015b6d0 2024-02-21T23:07:25.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.786+0000 7fbc4ea00640 1 -- 172.21.15.28:0/1505395661 >> v2:172.21.15.81:3300/0 conn(0x7fbc500c11f0 msgr2=0x7fbc50153950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:25.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.786+0000 7fbc4ea00640 1 --2- 172.21.15.28:0/1505395661 >> v2:172.21.15.81:3300/0 conn(0x7fbc500c11f0 0x7fbc50153950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:25.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.786+0000 7fbc4ea00640 1 -- 172.21.15.28:0/1505395661 >> v1:172.21.15.81:6789/0 conn(0x7fbc500c15d0 legacy=0x7fbc5006dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:25.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.787+0000 7fbc4ea00640 1 -- 172.21.15.28:0/1505395661 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc50153430 con 0x7fbc5015b6d0 2024-02-21T23:07:25.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.787+0000 7fbc4ea00640 1 -- 172.21.15.28:0/1505395661 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fbc480019f0 con 0x7fbc5015b6d0 2024-02-21T23:07:25.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.787+0000 7fbc4ea00640 1 -- 172.21.15.28:0/1505395661 >> v1:172.21.15.28:6789/0 conn(0x7fbc5015b6d0 legacy=0x7fbc5015dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:25.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.787+0000 7fbc4ea00640 1 --2- 172.21.15.28:0/1505395661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc34003a00 0x7fbc34003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:25.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.787+0000 7fbc4ea00640 1 --2- 172.21.15.28:0/1505395661 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc34004800 0x7fbc34004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:25.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.787+0000 7fbc4ea00640 1 --2- 172.21.15.28:0/1505395661 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbc340052e0 0x7fbc3400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:25.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.787+0000 7fbc4ea00640 1 -- 172.21.15.28:0/1505395661 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc500c1a60 con 0x7fbc34003a00 2024-02-21T23:07:25.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.787+0000 7fbc4ea00640 1 -- 172.21.15.28:0/1505395661 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc3400c1e0 con 0x7fbc34004800 2024-02-21T23:07:25.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.787+0000 7fbc4ea00640 1 -- 172.21.15.28:0/1505395661 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc3400c3e0 con 0x7fbc340052e0 2024-02-21T23:07:25.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.787+0000 7fbc4f400640 1 --2- 172.21.15.28:0/1505395661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc34003a00 0x7fbc34003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:25.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.787+0000 7fbc4f400640 1 -- 172.21.15.28:0/1505395661 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbc340052e0 msgr2=0x7fbc3400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:25.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.787+0000 7fbc4f400640 1 --2- 172.21.15.28:0/1505395661 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbc340052e0 0x7fbc3400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:25.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.787+0000 7fbc4f400640 1 -- 172.21.15.28:0/1505395661 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc34004800 msgr2=0x7fbc34004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:25.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.787+0000 7fbc47e00640 1 --2- 172.21.15.28:0/1505395661 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc34004800 0x7fbc34004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:25.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.787+0000 7fbc4f400640 1 --2- 172.21.15.28:0/1505395661 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc34004800 0x7fbc34004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:25.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.788+0000 7fbc4f400640 1 -- 172.21.15.28:0/1505395661 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc34003520 con 0x7fbc34003a00 2024-02-21T23:07:25.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.788+0000 7fbc4fe00640 1 -- 172.21.15.28:0/1505395661 reap_dead start 2024-02-21T23:07:25.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.788+0000 7fbc47e00640 1 --2- 172.21.15.28:0/1505395661 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc34004800 0x7fbc34004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:07:25.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.789+0000 7fbc4f400640 1 --2- 172.21.15.28:0/1505395661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc34003a00 0x7fbc34003e20 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7fbc40009ac0 tx=0x7fbc34005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3614ea61d5fca11a server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:25.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.789+0000 7fbc4ea00640 1 -- 172.21.15.28:0/1505395661 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc40039070 con 0x7fbc34003a00 2024-02-21T23:07:25.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.789+0000 7fbc4ea00640 1 -- 172.21.15.28:0/1505395661 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc4002c6b0 con 0x7fbc34003a00 2024-02-21T23:07:25.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.789+0000 7fbc4ea00640 1 -- 172.21.15.28:0/1505395661 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc40009690 con 0x7fbc34003a00 2024-02-21T23:07:25.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.789+0000 7fbc56800640 1 -- 172.21.15.28:0/1505395661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc34003a00 msgr2=0x7fbc34003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:25.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.789+0000 7fbc56800640 1 --2- 172.21.15.28:0/1505395661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc34003a00 0x7fbc34003e20 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7fbc40009ac0 tx=0x7fbc34005830 comp rx=0 tx=0).stop 2024-02-21T23:07:25.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.790+0000 7fbc56800640 1 -- 172.21.15.28:0/1505395661 shutdown_connections 2024-02-21T23:07:25.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.790+0000 7fbc56800640 1 --2- 172.21.15.28:0/1505395661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc34003a00 0x7fbc34003e20 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:25.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.790+0000 7fbc56800640 1 -- 172.21.15.28:0/1505395661 >> 172.21.15.28:0/1505395661 conn(0x7fbc500c67f0 msgr2=0x7fbc500c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:25.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.790+0000 7fbc56800640 1 -- 172.21.15.28:0/1505395661 shutdown_connections 2024-02-21T23:07:25.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.790+0000 7fbc56800640 1 -- 172.21.15.28:0/1505395661 wait complete. 2024-02-21T23:07:25.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.791+0000 7fbc56800640 1 Processor -- start 2024-02-21T23:07:25.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.792+0000 7fbc56800640 1 -- start start 2024-02-21T23:07:25.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.792+0000 7fbc56800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbc34003a00 0x7fbc500a6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:25.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.792+0000 7fbc56800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc50141d70 0x7fbc50147e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:25.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.792+0000 7fbc56800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc50141500 0x7fbc500a76e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:25.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.793+0000 7fbc56800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc500c4c00 con 0x7fbc50141500 2024-02-21T23:07:25.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.793+0000 7fbc56800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc500a5ec0 con 0x7fbc50141d70 2024-02-21T23:07:25.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.793+0000 7fbc56800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc500a6110 con 0x7fbc34003a00 2024-02-21T23:07:25.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.793+0000 7fbc4fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc50141500 0x7fbc500a76e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:25.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.793+0000 7fbc4fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc50141500 0x7fbc500a76e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59324/0 (socket says 172.21.15.28:59324) 2024-02-21T23:07:25.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.793+0000 7fbc4fe00640 1 -- 172.21.15.28:0/1804809606 learned_addr learned my addr 172.21.15.28:0/1804809606 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:25.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.793+0000 7fbc47e00640 1 --2- 172.21.15.28:0/1804809606 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc50141d70 0x7fbc50147e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:25.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.793+0000 7fbc4f400640 1 --2- 172.21.15.28:0/1804809606 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbc34003a00 0x7fbc500a6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:25.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.793+0000 7fbc4fe00640 1 -- 172.21.15.28:0/1804809606 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbc34003a00 msgr2=0x7fbc500a6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:25.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.793+0000 7fbc4fe00640 1 --2- 172.21.15.28:0/1804809606 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbc34003a00 0x7fbc500a6dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:25.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.793+0000 7fbc4fe00640 1 -- 172.21.15.28:0/1804809606 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc50141d70 msgr2=0x7fbc50147e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:25.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.793+0000 7fbc4fe00640 1 --2- 172.21.15.28:0/1804809606 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc50141d70 0x7fbc50147e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:25.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.793+0000 7fbc4fe00640 1 -- 172.21.15.28:0/1804809606 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc50072f50 con 0x7fbc50141500 2024-02-21T23:07:25.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.793+0000 7fbc4f400640 1 --2- 172.21.15.28:0/1804809606 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbc34003a00 0x7fbc500a6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:07:25.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.794+0000 7fbc4fe00640 1 --2- 172.21.15.28:0/1804809606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc50141500 0x7fbc500a76e0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7fbc500c41e0 tx=0x7fbc4800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:25.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.794+0000 7fbc47400640 1 -- 172.21.15.28:0/1804809606 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc4800a5b0 con 0x7fbc50141500 2024-02-21T23:07:25.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.794+0000 7fbc47400640 1 -- 172.21.15.28:0/1804809606 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc4800a770 con 0x7fbc50141500 2024-02-21T23:07:25.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.794+0000 7fbc47400640 1 -- 172.21.15.28:0/1804809606 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc4800ab60 con 0x7fbc50141500 2024-02-21T23:07:25.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.794+0000 7fbc56800640 1 -- 172.21.15.28:0/1804809606 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc50073230 con 0x7fbc50141500 2024-02-21T23:07:25.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.794+0000 7fbc56800640 1 -- 172.21.15.28:0/1804809606 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc500a68b0 con 0x7fbc50141500 2024-02-21T23:07:25.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.795+0000 7fbc56800640 1 -- 172.21.15.28:0/1804809606 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc500a6ab0 con 0x7fbc50141500 2024-02-21T23:07:25.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.796+0000 7fbc47400640 1 -- 172.21.15.28:0/1804809606 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbc48011070 con 0x7fbc50141500 2024-02-21T23:07:25.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.799+0000 7fbc47400640 1 --2- 172.21.15.28:0/1804809606 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbc2c06b550 0x7fbc2c06da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:25.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.799+0000 7fbc47400640 1 -- 172.21.15.28:0/1804809606 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fbc48089900 con 0x7fbc50141500 2024-02-21T23:07:25.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.799+0000 7fbc4f400640 1 --2- 172.21.15.28:0/1804809606 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbc2c06b550 0x7fbc2c06da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:25.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.800+0000 7fbc4f400640 1 --2- 172.21.15.28:0/1804809606 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbc2c06b550 0x7fbc2c06da10 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fbc40036000 tx=0x7fbc40030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:25.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:25.802+0000 7fbc47400640 1 -- 172.21.15.28:0/1804809606 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fbc500a6ab0 con 0x7fbc50141500 2024-02-21T23:07:25.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.006+0000 7fbc56800640 1 -- 172.21.15.28:0/1804809606 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbc50067f30 con 0x7fbc50141500 2024-02-21T23:07:25.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.007+0000 7fbc47400640 1 -- 172.21.15.28:0/1804809606 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v106) v1 ==== 74+0+13106 (secure 0 0 0) 0x7fbc4801a060 con 0x7fbc50141500 2024-02-21T23:07:25.997 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:07:25.997 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":106,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:07:25.011663+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":96,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:07:25.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.011+0000 7fbc56800640 1 -- 172.21.15.28:0/1804809606 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbc2c06b550 msgr2=0x7fbc2c06da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:26.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.011+0000 7fbc56800640 1 --2- 172.21.15.28:0/1804809606 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbc2c06b550 0x7fbc2c06da10 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fbc40036000 tx=0x7fbc40030000 comp rx=0 tx=0).stop 2024-02-21T23:07:26.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.011+0000 7fbc56800640 1 -- 172.21.15.28:0/1804809606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc50141500 msgr2=0x7fbc500a76e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:26.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.011+0000 7fbc56800640 1 --2- 172.21.15.28:0/1804809606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc50141500 0x7fbc500a76e0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7fbc500c41e0 tx=0x7fbc4800cd50 comp rx=0 tx=0).stop 2024-02-21T23:07:26.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.011+0000 7fbc56800640 1 -- 172.21.15.28:0/1804809606 shutdown_connections 2024-02-21T23:07:26.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.011+0000 7fbc56800640 1 --2- 172.21.15.28:0/1804809606 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbc2c06b550 0x7fbc2c06da10 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:26.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.011+0000 7fbc56800640 1 --2- 172.21.15.28:0/1804809606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc50141500 0x7fbc500a76e0 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:26.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.011+0000 7fbc56800640 1 --2- 172.21.15.28:0/1804809606 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbc50141d70 0x7fbc50147e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:26.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.011+0000 7fbc56800640 1 --2- 172.21.15.28:0/1804809606 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbc34003a00 0x7fbc500a6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:26.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.011+0000 7fbc56800640 1 -- 172.21.15.28:0/1804809606 >> 172.21.15.28:0/1804809606 conn(0x7fbc500c67f0 msgr2=0x7fbc500c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:26.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.011+0000 7fbc56800640 1 -- 172.21.15.28:0/1804809606 shutdown_connections 2024-02-21T23:07:26.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.011+0000 7fbc56800640 1 -- 172.21.15.28:0/1804809606 wait complete. 2024-02-21T23:07:26.017 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-21T23:02:30.728329+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-21T23:07:26.018 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-21T23:07:26.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.256+0000 7fda51a00640 1 Processor -- start 2024-02-21T23:07:26.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.256+0000 7fda51a00640 1 -- start start 2024-02-21T23:07:26.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.257+0000 7fda51a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fda4c150240 0x7fda4c0c12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:26.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.257+0000 7fda51a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fda4c0c4580 0x7fda4c06db10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:26.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.257+0000 7fda51a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda4c06f780 con 0x7fda4c0c2c10 2024-02-21T23:07:26.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.257+0000 7fda51a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda4c150620 con 0x7fda4c150240 2024-02-21T23:07:26.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.257+0000 7fda51a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda4c153a90 con 0x7fda4c0c4580 2024-02-21T23:07:26.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.257+0000 7fda4be00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fda4c150240 0x7fda4c0c12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:26.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.257+0000 7fda4b400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fda4c0c4580 0x7fda4c06db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:26.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.258+0000 7fda4be00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fda4c150240 0x7fda4c0c12c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.28:60954/0 (socket says 172.21.15.28:60954) 2024-02-21T23:07:26.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.258+0000 7fda4be00640 1 -- 172.21.15.28:0/1711094219 learned_addr learned my addr 172.21.15.28:0/1711094219 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:26.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.258+0000 7fda4be00640 1 -- 172.21.15.28:0/1711094219 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fda4c0c4580 msgr2=0x7fda4c06db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:26.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.258+0000 7fda4be00640 1 --2- 172.21.15.28:0/1711094219 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fda4c0c4580 0x7fda4c06db10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:26.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.258+0000 7fda4be00640 1 -- 172.21.15.28:0/1711094219 >> v1:172.21.15.81:6789/0 conn(0x7fda4c0c2c10 legacy=0x7fda4c06ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:26.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.258+0000 7fda4be00640 1 -- 172.21.15.28:0/1711094219 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda4c06f4a0 con 0x7fda4c150240 2024-02-21T23:07:26.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.260+0000 7fda4be00640 1 --2- 172.21.15.28:0/1711094219 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda4c150240 0x7fda4c0c12c0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fda3c009870 tx=0x7fda3c02b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bfb89f5fe4ef0399 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:26.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.260+0000 7fda4aa00640 1 -- 172.21.15.28:0/1711094219 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda3c035020 con 0x7fda4c150240 2024-02-21T23:07:26.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.260+0000 7fda4aa00640 1 -- 172.21.15.28:0/1711094219 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda3c031640 con 0x7fda4c150240 2024-02-21T23:07:26.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.260+0000 7fda4aa00640 1 -- 172.21.15.28:0/1711094219 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda3c02d040 con 0x7fda4c150240 2024-02-21T23:07:26.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.261+0000 7fda51a00640 1 -- 172.21.15.28:0/1711094219 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda4c150240 msgr2=0x7fda4c0c12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:26.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.261+0000 7fda51a00640 1 --2- 172.21.15.28:0/1711094219 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda4c150240 0x7fda4c0c12c0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fda3c009870 tx=0x7fda3c02b8f0 comp rx=0 tx=0).stop 2024-02-21T23:07:26.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.261+0000 7fda51a00640 1 -- 172.21.15.28:0/1711094219 shutdown_connections 2024-02-21T23:07:26.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.261+0000 7fda51a00640 1 --2- 172.21.15.28:0/1711094219 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fda4c0c4580 0x7fda4c06db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:26.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.261+0000 7fda51a00640 1 --2- 172.21.15.28:0/1711094219 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda4c150240 0x7fda4c0c12c0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:26.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.261+0000 7fda51a00640 1 -- 172.21.15.28:0/1711094219 >> 172.21.15.28:0/1711094219 conn(0x7fda4c0a8530 msgr2=0x7fda4c0a8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:26.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.261+0000 7fda51a00640 1 -- 172.21.15.28:0/1711094219 shutdown_connections 2024-02-21T23:07:26.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.261+0000 7fda51a00640 1 -- 172.21.15.28:0/1711094219 wait complete. 2024-02-21T23:07:26.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.263+0000 7fda51a00640 1 Processor -- start 2024-02-21T23:07:26.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.263+0000 7fda51a00640 1 -- start start 2024-02-21T23:07:26.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.264+0000 7fda51a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda4c0c2c10 0x7fda4c14ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:26.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.264+0000 7fda51a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda4c0c4580 0x7fda4c1ec3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:26.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.264+0000 7fda51a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fda4c150240 0x7fda4c0c5020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:26.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.264+0000 7fda51a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda4c0c4b90 con 0x7fda4c0c4580 2024-02-21T23:07:26.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.264+0000 7fda51a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda4c0c4a10 con 0x7fda4c0c2c10 2024-02-21T23:07:26.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.264+0000 7fda51a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda4c06b580 con 0x7fda4c150240 2024-02-21T23:07:26.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.264+0000 7fda4b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda4c0c4580 0x7fda4c1ec3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:26.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.264+0000 7fda4b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda4c0c4580 0x7fda4c1ec3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59328/0 (socket says 172.21.15.28:59328) 2024-02-21T23:07:26.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.264+0000 7fda4b400640 1 -- 172.21.15.28:0/305679492 learned_addr learned my addr 172.21.15.28:0/305679492 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:26.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.264+0000 7fda4be00640 1 --2- 172.21.15.28:0/305679492 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda4c0c2c10 0x7fda4c14ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:26.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.264+0000 7fda51000640 1 --2- 172.21.15.28:0/305679492 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fda4c150240 0x7fda4c0c5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:26.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.265+0000 7fda4b400640 1 -- 172.21.15.28:0/305679492 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fda4c150240 msgr2=0x7fda4c0c5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:26.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.265+0000 7fda4b400640 1 --2- 172.21.15.28:0/305679492 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fda4c150240 0x7fda4c0c5020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:26.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.265+0000 7fda4b400640 1 -- 172.21.15.28:0/305679492 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda4c0c2c10 msgr2=0x7fda4c14ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:26.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.265+0000 7fda4b400640 1 --2- 172.21.15.28:0/305679492 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda4c0c2c10 0x7fda4c14ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:26.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.265+0000 7fda4b400640 1 -- 172.21.15.28:0/305679492 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda4c155120 con 0x7fda4c0c4580 2024-02-21T23:07:26.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.265+0000 7fda51000640 1 --2- 172.21.15.28:0/305679492 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fda4c150240 0x7fda4c0c5020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:07:26.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.265+0000 7fda4b400640 1 --2- 172.21.15.28:0/305679492 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda4c0c4580 0x7fda4c1ec3b0 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7fda3800e980 tx=0x7fda3800ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:26.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.265+0000 7fda43e00640 1 -- 172.21.15.28:0/305679492 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda38017070 con 0x7fda4c0c4580 2024-02-21T23:07:26.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.265+0000 7fda43e00640 1 -- 172.21.15.28:0/305679492 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda3800ca00 con 0x7fda4c0c4580 2024-02-21T23:07:26.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.265+0000 7fda51a00640 1 -- 172.21.15.28:0/305679492 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda4c1dc270 con 0x7fda4c0c4580 2024-02-21T23:07:26.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.265+0000 7fda43e00640 1 -- 172.21.15.28:0/305679492 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda3800cdd0 con 0x7fda4c0c4580 2024-02-21T23:07:26.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.266+0000 7fda51a00640 1 -- 172.21.15.28:0/305679492 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda4c1dc830 con 0x7fda4c0c4580 2024-02-21T23:07:26.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.267+0000 7fda51a00640 1 -- 172.21.15.28:0/305679492 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda4c04f290 con 0x7fda4c0c4580 2024-02-21T23:07:26.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.268+0000 7fda43e00640 1 -- 172.21.15.28:0/305679492 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fda3801a070 con 0x7fda4c0c4580 2024-02-21T23:07:26.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.271+0000 7fda43e00640 1 --2- 172.21.15.28:0/305679492 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fda2c06b4d0 0x7fda2c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:26.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.271+0000 7fda43e00640 1 -- 172.21.15.28:0/305679492 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fda3808a0c0 con 0x7fda4c0c4580 2024-02-21T23:07:26.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.271+0000 7fda4be00640 1 --2- 172.21.15.28:0/305679492 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fda2c06b4d0 0x7fda2c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:26.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.273+0000 7fda4be00640 1 --2- 172.21.15.28:0/305679492 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fda2c06b4d0 0x7fda2c06d990 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fda3c036000 tx=0x7fda3c03c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:26.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.276+0000 7fda43e00640 1 -- 172.21.15.28:0/305679492 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fda38059070 con 0x7fda4c0c4580 2024-02-21T23:07:26.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.479+0000 7fda51a00640 1 -- 172.21.15.28:0/305679492 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fda4c06f0e0 con 0x7fda4c0c4580 2024-02-21T23:07:26.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.479+0000 7fda43e00640 1 -- 172.21.15.28:0/305679492 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v106) v1 ==== 93+0+10 (secure 0 0 0) 0x7fda38058970 con 0x7fda4c0c4580 2024-02-21T23:07:26.467 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-02-21T23:07:26.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.482+0000 7fda51a00640 1 -- 172.21.15.28:0/305679492 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fda2c06b4d0 msgr2=0x7fda2c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:26.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.482+0000 7fda51a00640 1 --2- 172.21.15.28:0/305679492 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fda2c06b4d0 0x7fda2c06d990 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fda3c036000 tx=0x7fda3c03c040 comp rx=0 tx=0).stop 2024-02-21T23:07:26.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.482+0000 7fda51a00640 1 -- 172.21.15.28:0/305679492 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda4c0c4580 msgr2=0x7fda4c1ec3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:26.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.482+0000 7fda51a00640 1 --2- 172.21.15.28:0/305679492 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda4c0c4580 0x7fda4c1ec3b0 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7fda3800e980 tx=0x7fda3800ee40 comp rx=0 tx=0).stop 2024-02-21T23:07:26.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.482+0000 7fda51a00640 1 -- 172.21.15.28:0/305679492 shutdown_connections 2024-02-21T23:07:26.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.482+0000 7fda51a00640 1 --2- 172.21.15.28:0/305679492 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fda2c06b4d0 0x7fda2c06d990 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:26.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.482+0000 7fda51a00640 1 --2- 172.21.15.28:0/305679492 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fda4c150240 0x7fda4c0c5020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:26.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.482+0000 7fda51a00640 1 --2- 172.21.15.28:0/305679492 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda4c0c4580 0x7fda4c1ec3b0 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:26.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.482+0000 7fda51a00640 1 --2- 172.21.15.28:0/305679492 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda4c0c2c10 0x7fda4c14ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:26.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.482+0000 7fda51a00640 1 -- 172.21.15.28:0/305679492 >> 172.21.15.28:0/305679492 conn(0x7fda4c0a8530 msgr2=0x7fda4c14e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:26.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.482+0000 7fda51a00640 1 -- 172.21.15.28:0/305679492 shutdown_connections 2024-02-21T23:07:26.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.482+0000 7fda51a00640 1 -- 172.21.15.28:0/305679492 wait complete. 2024-02-21T23:07:26.491 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-02-21T23:07:26.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.729+0000 7f8feb000640 1 Processor -- start 2024-02-21T23:07:26.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.729+0000 7f8feb000640 1 -- start start 2024-02-21T23:07:26.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.731+0000 7f8feb000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f8fe40c8740 0x7f8fe4158510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:26.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.731+0000 7f8feb000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8fe4158a50 0x7f8fe415f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:26.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.731+0000 7f8feb000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fe41550b0 con 0x7f8fe40b0130 2024-02-21T23:07:26.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.731+0000 7f8feb000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fe4152a40 con 0x7f8fe4158a50 2024-02-21T23:07:26.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.731+0000 7f8feb000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fe4155350 con 0x7f8fe40c8740 2024-02-21T23:07:26.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.731+0000 7f8fea600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8fe4158a50 0x7f8fe415f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:26.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.731+0000 7f8fea600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8fe4158a50 0x7f8fe415f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59338/0 (socket says 172.21.15.28:59338) 2024-02-21T23:07:26.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.731+0000 7f8fea600640 1 -- 172.21.15.28:0/1527546723 learned_addr learned my addr 172.21.15.28:0/1527546723 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:26.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.732+0000 7f8fe9200640 1 --2- 172.21.15.28:0/1527546723 >> v2:172.21.15.81:3300/0 conn(0x7f8fe40c8740 0x7f8fe4158510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:26.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.732+0000 7f8fea600640 1 -- 172.21.15.28:0/1527546723 >> v2:172.21.15.81:3300/0 conn(0x7f8fe40c8740 msgr2=0x7f8fe4158510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:26.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.732+0000 7f8fea600640 1 --2- 172.21.15.28:0/1527546723 >> v2:172.21.15.81:3300/0 conn(0x7f8fe40c8740 0x7f8fe4158510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:26.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.732+0000 7f8fea600640 1 -- 172.21.15.28:0/1527546723 >> v1:172.21.15.81:6789/0 conn(0x7f8fe40b0130 legacy=0x7f8fe40c51c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:26.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.732+0000 7f8fea600640 1 -- 172.21.15.28:0/1527546723 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fe4155630 con 0x7f8fe4158a50 2024-02-21T23:07:26.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.733+0000 7f8fea600640 1 --2- 172.21.15.28:0/1527546723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fe4158a50 0x7f8fe415f880 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f8fe41595d0 tx=0x7f8fdc02be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec730018b24e5541 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:26.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.733+0000 7f8fe3e00640 1 -- 172.21.15.28:0/1527546723 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fdc02c9c0 con 0x7f8fe4158a50 2024-02-21T23:07:26.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.733+0000 7f8fe3e00640 1 -- 172.21.15.28:0/1527546723 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fdc02cb60 con 0x7f8fe4158a50 2024-02-21T23:07:26.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.733+0000 7f8fe3e00640 1 -- 172.21.15.28:0/1527546723 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fdc0343f0 con 0x7f8fe4158a50 2024-02-21T23:07:26.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.734+0000 7f8feb000640 1 -- 172.21.15.28:0/1527546723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fe4158a50 msgr2=0x7f8fe415f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:26.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.734+0000 7f8feb000640 1 --2- 172.21.15.28:0/1527546723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fe4158a50 0x7f8fe415f880 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f8fe41595d0 tx=0x7f8fdc02be20 comp rx=0 tx=0).stop 2024-02-21T23:07:26.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.734+0000 7f8feb000640 1 -- 172.21.15.28:0/1527546723 shutdown_connections 2024-02-21T23:07:26.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.734+0000 7f8feb000640 1 --2- 172.21.15.28:0/1527546723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fe4158a50 0x7f8fe415f880 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:26.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.734+0000 7f8feb000640 1 --2- 172.21.15.28:0/1527546723 >> v2:172.21.15.81:3300/0 conn(0x7f8fe40c8740 0x7f8fe4158510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:26.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.734+0000 7f8feb000640 1 -- 172.21.15.28:0/1527546723 >> 172.21.15.28:0/1527546723 conn(0x7f8fe40ac370 msgr2=0x7f8fe40ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:26.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.734+0000 7f8feb000640 1 -- 172.21.15.28:0/1527546723 shutdown_connections 2024-02-21T23:07:26.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.734+0000 7f8feb000640 1 -- 172.21.15.28:0/1527546723 wait complete. 2024-02-21T23:07:26.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.737+0000 7f8feb000640 1 Processor -- start 2024-02-21T23:07:26.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.737+0000 7f8feb000640 1 -- start start 2024-02-21T23:07:26.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.738+0000 7f8feb000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8fe40b0130 0x7f8fe40c2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:26.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.738+0000 7f8feb000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fe40c8740 0x7f8fe41ee4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:26.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.738+0000 7f8feb000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8fe4158a50 0x7f8fe41e1590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:26.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.738+0000 7f8feb000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe40c8dd0 con 0x7f8fe40c8740 2024-02-21T23:07:26.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.738+0000 7f8feb000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe4155e30 con 0x7f8fe4158a50 2024-02-21T23:07:26.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.738+0000 7f8feb000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe4158e60 con 0x7f8fe40b0130 2024-02-21T23:07:26.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.738+0000 7f8fe9200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fe40c8740 0x7f8fe41ee4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:26.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.738+0000 7f8fe9200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fe40c8740 0x7f8fe41ee4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59346/0 (socket says 172.21.15.28:59346) 2024-02-21T23:07:26.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.738+0000 7f8fe9200640 1 -- 172.21.15.28:0/1049526007 learned_addr learned my addr 172.21.15.28:0/1049526007 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:26.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.739+0000 7f8fe9c00640 1 --2- 172.21.15.28:0/1049526007 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8fe40b0130 0x7f8fe40c2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:26.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.739+0000 7f8fe9200640 1 -- 172.21.15.28:0/1049526007 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8fe40b0130 msgr2=0x7f8fe40c2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:26.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.739+0000 7f8fe9200640 1 --2- 172.21.15.28:0/1049526007 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8fe40b0130 0x7f8fe40c2320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:26.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.739+0000 7f8fe9200640 1 -- 172.21.15.28:0/1049526007 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8fe4158a50 msgr2=0x7f8fe41e1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:26.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.739+0000 7f8fe9200640 1 --2- 172.21.15.28:0/1049526007 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8fe4158a50 0x7f8fe41e1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:26.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.739+0000 7f8fe9200640 1 -- 172.21.15.28:0/1049526007 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fe40abe60 con 0x7f8fe40c8740 2024-02-21T23:07:26.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.739+0000 7f8fe9c00640 1 --2- 172.21.15.28:0/1049526007 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8fe40b0130 0x7f8fe40c2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:07:26.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.739+0000 7f8fe9200640 1 --2- 172.21.15.28:0/1049526007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fe40c8740 0x7f8fe41ee4e0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f8fd000d830 tx=0x7f8fd000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:26.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.739+0000 7f8fe1600640 1 -- 172.21.15.28:0/1049526007 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fd000b6c0 con 0x7f8fe40c8740 2024-02-21T23:07:26.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.739+0000 7f8fe1600640 1 -- 172.21.15.28:0/1049526007 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fd0019070 con 0x7f8fe40c8740 2024-02-21T23:07:26.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.739+0000 7f8fe1600640 1 -- 172.21.15.28:0/1049526007 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fd000ba80 con 0x7f8fe40c8740 2024-02-21T23:07:26.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.740+0000 7f8feb000640 1 -- 172.21.15.28:0/1049526007 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fe4151f20 con 0x7f8fe40c8740 2024-02-21T23:07:26.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.740+0000 7f8feb000640 1 -- 172.21.15.28:0/1049526007 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fe4152290 con 0x7f8fe40c8740 2024-02-21T23:07:26.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.741+0000 7f8feb000640 1 -- 172.21.15.28:0/1049526007 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fe404f290 con 0x7f8fe40c8740 2024-02-21T23:07:26.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.742+0000 7f8fe1600640 1 -- 172.21.15.28:0/1049526007 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8fd001e460 con 0x7f8fe40c8740 2024-02-21T23:07:26.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.743+0000 7f8fe1600640 1 --2- 172.21.15.28:0/1049526007 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8fc406b520 0x7f8fc406d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:26.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.743+0000 7f8fe1600640 1 -- 172.21.15.28:0/1049526007 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f8fd008a6a0 con 0x7f8fe40c8740 2024-02-21T23:07:26.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.744+0000 7f8fe9c00640 1 --2- 172.21.15.28:0/1049526007 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8fc406b520 0x7f8fc406d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:26.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.745+0000 7f8fe9c00640 1 --2- 172.21.15.28:0/1049526007 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8fc406b520 0x7f8fc406d9e0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f8fd800a830 tx=0x7f8fd8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:26.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.747+0000 7f8fe1600640 1 -- 172.21.15.28:0/1049526007 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f8fd0059750 con 0x7f8fe40c8740 2024-02-21T23:07:26.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.952+0000 7f8feb000640 1 -- 172.21.15.28:0/1049526007 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f8fe40c4ad0 con 0x7f8fe40c8740 2024-02-21T23:07:26.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.953+0000 7f8fe1600640 1 -- 172.21.15.28:0/1049526007 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v106) v1 ==== 84+0+65 (secure 0 0 0) 0x7f8fd0059050 con 0x7f8fe40c8740 2024-02-21T23:07:26.941 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:07:26.941 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-02-21T23:07:26.942 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:07:26.942 INFO:teuthology.orchestra.run.smithi028.stdout: "poolnum": 1, 2024-02-21T23:07:26.942 INFO:teuthology.orchestra.run.smithi028.stdout: "poolname": ".mgr" 2024-02-21T23:07:26.942 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:07:26.942 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-02-21T23:07:26.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.955+0000 7f8feb000640 1 -- 172.21.15.28:0/1049526007 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8fc406b520 msgr2=0x7f8fc406d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:26.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.955+0000 7f8feb000640 1 --2- 172.21.15.28:0/1049526007 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8fc406b520 0x7f8fc406d9e0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f8fd800a830 tx=0x7f8fd8008040 comp rx=0 tx=0).stop 2024-02-21T23:07:26.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.955+0000 7f8feb000640 1 -- 172.21.15.28:0/1049526007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fe40c8740 msgr2=0x7f8fe41ee4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:26.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.955+0000 7f8feb000640 1 --2- 172.21.15.28:0/1049526007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fe40c8740 0x7f8fe41ee4e0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f8fd000d830 tx=0x7f8fd000dcf0 comp rx=0 tx=0).stop 2024-02-21T23:07:26.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.956+0000 7f8feb000640 1 -- 172.21.15.28:0/1049526007 shutdown_connections 2024-02-21T23:07:26.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.956+0000 7f8feb000640 1 --2- 172.21.15.28:0/1049526007 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8fc406b520 0x7f8fc406d9e0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:26.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.956+0000 7f8feb000640 1 --2- 172.21.15.28:0/1049526007 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8fe4158a50 0x7f8fe41e1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:26.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.956+0000 7f8feb000640 1 --2- 172.21.15.28:0/1049526007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fe40c8740 0x7f8fe41ee4e0 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:26.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.956+0000 7f8feb000640 1 --2- 172.21.15.28:0/1049526007 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8fe40b0130 0x7f8fe40c2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:26.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.956+0000 7f8feb000640 1 -- 172.21.15.28:0/1049526007 >> 172.21.15.28:0/1049526007 conn(0x7f8fe40ac370 msgr2=0x7f8fe406cde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:26.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.956+0000 7f8feb000640 1 -- 172.21.15.28:0/1049526007 shutdown_connections 2024-02-21T23:07:26.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:26.956+0000 7f8feb000640 1 -- 172.21.15.28:0/1049526007 wait complete. 2024-02-21T23:07:26.964 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-02-21T23:07:26.965 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-02-21T23:07:27.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:27.199+0000 7fe2f4a00640 1 Processor -- start 2024-02-21T23:07:27.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:27.199+0000 7fe2f4a00640 1 -- start start 2024-02-21T23:07:27.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:27.201+0000 7fe2f4a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe2f00c1010 0x7fe2f006e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:27.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:27.201+0000 7fe2f4a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe2f006e870 0x7fe2f015d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:27.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:27.201+0000 7fe2f4a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2f0152fd0 con 0x7fe2f01501c0 2024-02-21T23:07:27.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:27.201+0000 7fe2f4a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2f01532b0 con 0x7fe2f006e870 2024-02-21T23:07:27.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:27.201+0000 7fe2f4a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f0153590 con 0x7fe2f00c1010 2024-02-21T23:07:27.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:27.201+0000 7fe2efe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe2f006e870 0x7fe2f015d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:27.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:27.201+0000 7fe2efe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe2f006e870 0x7fe2f015d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59356/0 (socket says 172.21.15.28:59356) 2024-02-21T23:07:27.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:27.201+0000 7fe2efe00640 1 -- 172.21.15.28:0/1802642002 learned_addr learned my addr 172.21.15.28:0/1802642002 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:27.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:27.201+0000 7fe2eea00640 1 --2- 172.21.15.28:0/1802642002 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe2f00c1010 0x7fe2f006e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:27.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:27.202+0000 7fe2efe00640 1 -- 172.21.15.28:0/1802642002 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe2f00c1010 msgr2=0x7fe2f006e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:27.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:27.202+0000 7fe2efe00640 1 --2- 172.21.15.28:0/1802642002 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe2f00c1010 0x7fe2f006e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:27.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:27.202+0000 7fe2efe00640 1 -- 172.21.15.28:0/1802642002 >> v1:172.21.15.81:6789/0 conn(0x7fe2f01501c0 legacy=0x7fe2f00a96c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:27.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:27.202+0000 7fe2efe00640 1 -- 172.21.15.28:0/1802642002 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2f0153870 con 0x7fe2f006e870 2024-02-21T23:07:27.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:27.203+0000 7fe2efe00640 1 --2- 172.21.15.28:0/1802642002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2f006e870 0x7fe2f015d5b0 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7fe2e8009e50 tx=0x7fe2e802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97022f4844d62ae6 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:27.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:27.204+0000 7fe2ee000640 1 -- 172.21.15.28:0/1802642002 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e802ca70 con 0x7fe2f006e870 2024-02-21T23:07:27.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:27.204+0000 7fe2ee000640 1 -- 172.21.15.28:0/1802642002 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2e802cc10 con 0x7fe2f006e870 2024-02-21T23:07:27.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:27.204+0000 7fe2ee000640 1 -- 172.21.15.28:0/1802642002 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e80343f0 con 0x7fe2f006e870 2024-02-21T23:07:27.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:27.204+0000 7fe2f4a00640 1 -- 172.21.15.28:0/1802642002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2f006e870 msgr2=0x7fe2f015d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:27.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:27.204+0000 7fe2f4a00640 1 --2- 172.21.15.28:0/1802642002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2f006e870 0x7fe2f015d5b0 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7fe2e8009e50 tx=0x7fe2e802bed0 comp rx=0 tx=0).stop 2024-02-21T23:07:27.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:27.205+0000 7fe2f4a00640 1 -- 172.21.15.28:0/1802642002 shutdown_connections 2024-02-21T23:07:27.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:27.205+0000 7fe2f4a00640 1 --2- 172.21.15.28:0/1802642002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2f006e870 0x7fe2f015d5b0 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:27.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:27.205+0000 7fe2f4a00640 1 --2- 172.21.15.28:0/1802642002 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe2f00c1010 0x7fe2f006e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:27.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:27.205+0000 7fe2f4a00640 1 -- 172.21.15.28:0/1802642002 >> 172.21.15.28:0/1802642002 conn(0x7fe2f00c6610 msgr2=0x7fe2f00c69f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:27.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:27.205+0000 7fe2f4a00640 1 -- 172.21.15.28:0/1802642002 shutdown_connections 2024-02-21T23:07:27.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:27.205+0000 7fe2f4a00640 1 -- 172.21.15.28:0/1802642002 wait complete. 2024-02-21T23:07:27.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:27.207+0000 7fe2f4a00640 1 Processor -- start 2024-02-21T23:07:27.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:27.207+0000 7fe2f4a00640 1 -- start start 2024-02-21T23:07:27.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:27.208+0000 7fe2f4a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe2f006e870 0x7fe2f01439d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:27.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:27.208+0000 7fe2f4a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2f00c1010 0x7fe2f0151af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:27.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:27.208+0000 7fe2f4a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2f01501c0 0x7fe2f01ea760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:27.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:27.208+0000 7fe2f4a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f00c7340 con 0x7fe2f00c1010 2024-02-21T23:07:27.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:27.208+0000 7fe2f4a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f00c0c20 con 0x7fe2f01501c0 2024-02-21T23:07:27.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:27.208+0000 7fe2f4a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f01505a0 con 0x7fe2f006e870 2024-02-21T23:07:27.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:27.209+0000 7fe2eea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2f00c1010 0x7fe2f0151af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:27.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:27.209+0000 7fe2eea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2f00c1010 0x7fe2f0151af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59360/0 (socket says 172.21.15.28:59360) 2024-02-21T23:07:27.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:27.209+0000 7fe2eea00640 1 -- 172.21.15.28:0/2407349899 learned_addr learned my addr 172.21.15.28:0/2407349899 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:27.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:27.209+0000 7fe2ef400640 1 --2- 172.21.15.28:0/2407349899 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe2f006e870 0x7fe2f01439d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:27.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:27.209+0000 7fe2efe00640 1 --2- 172.21.15.28:0/2407349899 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2f01501c0 0x7fe2f01ea760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:27.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:27.209+0000 7fe2eea00640 1 -- 172.21.15.28:0/2407349899 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe2f006e870 msgr2=0x7fe2f01439d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:27.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:27.209+0000 7fe2eea00640 1 --2- 172.21.15.28:0/2407349899 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe2f006e870 0x7fe2f01439d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:27.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:27.209+0000 7fe2eea00640 1 -- 172.21.15.28:0/2407349899 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2f01501c0 msgr2=0x7fe2f01ea760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:27.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:27.209+0000 7fe2eea00640 1 --2- 172.21.15.28:0/2407349899 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2f01501c0 0x7fe2f01ea760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:27.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:27.209+0000 7fe2eea00640 1 -- 172.21.15.28:0/2407349899 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2f00c61c0 con 0x7fe2f00c1010 2024-02-21T23:07:27.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:27.209+0000 7fe2efe00640 1 --2- 172.21.15.28:0/2407349899 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2f01501c0 0x7fe2f01ea760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:07:27.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:27.209+0000 7fe2ef400640 1 --2- 172.21.15.28:0/2407349899 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe2f006e870 0x7fe2f01439d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:07:27.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:27.210+0000 7fe2eea00640 1 --2- 172.21.15.28:0/2407349899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2f00c1010 0x7fe2f0151af0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7fe2dc00d830 tx=0x7fe2dc00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:27.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:27.210+0000 7fe2e7400640 1 -- 172.21.15.28:0/2407349899 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2dc00b6c0 con 0x7fe2f00c1010 2024-02-21T23:07:27.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:27.210+0000 7fe2e7400640 1 -- 172.21.15.28:0/2407349899 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2dc019070 con 0x7fe2f00c1010 2024-02-21T23:07:27.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:27.210+0000 7fe2e7400640 1 -- 172.21.15.28:0/2407349899 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2dc00ba80 con 0x7fe2f00c1010 2024-02-21T23:07:27.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:27.210+0000 7fe2f4a00640 1 -- 172.21.15.28:0/2407349899 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2f01ee680 con 0x7fe2f00c1010 2024-02-21T23:07:27.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:27.210+0000 7fe2f4a00640 1 -- 172.21.15.28:0/2407349899 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2f00c5510 con 0x7fe2f00c1010 2024-02-21T23:07:27.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:27.211+0000 7fe2f4a00640 1 -- 172.21.15.28:0/2407349899 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2f00c14f0 con 0x7fe2f00c1010 2024-02-21T23:07:27.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:27.212+0000 7fe2e7400640 1 -- 172.21.15.28:0/2407349899 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe2dc01e460 con 0x7fe2f00c1010 2024-02-21T23:07:27.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:27.213+0000 7fe2e7400640 1 --2- 172.21.15.28:0/2407349899 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe2d006b500 0x7fe2d006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:27.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:27.213+0000 7fe2e7400640 1 -- 172.21.15.28:0/2407349899 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fe2dc08a6f0 con 0x7fe2f00c1010 2024-02-21T23:07:27.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:27.214+0000 7fe2ef400640 1 --2- 172.21.15.28:0/2407349899 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe2d006b500 0x7fe2d006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:27.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:27.215+0000 7fe2ef400640 1 --2- 172.21.15.28:0/2407349899 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe2d006b500 0x7fe2d006d9c0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fe2e0000f60 tx=0x7fe2e0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:27.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:27.221+0000 7fe2e7400640 1 -- 172.21.15.28:0/2407349899 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe2f00c14f0 con 0x7fe2f00c1010 2024-02-21T23:07:27.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:27.423+0000 7fe2f4a00640 1 -- 172.21.15.28:0/2407349899 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fe2f014a680 con 0x7fe2f00c1010 2024-02-21T23:07:28.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:28.020+0000 7fe2e7400640 1 -- 172.21.15.28:0/2407349899 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v107) v1 ==== 138+0+0 (secure 0 0 0) 0x7fe2dc059720 con 0x7fe2f00c1010 2024-02-21T23:07:28.013 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' created 2024-02-21T23:07:28.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:28.024+0000 7fe2f4a00640 1 -- 172.21.15.28:0/2407349899 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe2d006b500 msgr2=0x7fe2d006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:28.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:28.024+0000 7fe2f4a00640 1 --2- 172.21.15.28:0/2407349899 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe2d006b500 0x7fe2d006d9c0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fe2e0000f60 tx=0x7fe2e0009040 comp rx=0 tx=0).stop 2024-02-21T23:07:28.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:28.024+0000 7fe2f4a00640 1 -- 172.21.15.28:0/2407349899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2f00c1010 msgr2=0x7fe2f0151af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:28.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:28.024+0000 7fe2f4a00640 1 --2- 172.21.15.28:0/2407349899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2f00c1010 0x7fe2f0151af0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7fe2dc00d830 tx=0x7fe2dc00dcf0 comp rx=0 tx=0).stop 2024-02-21T23:07:28.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:28.025+0000 7fe2f4a00640 1 -- 172.21.15.28:0/2407349899 shutdown_connections 2024-02-21T23:07:28.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:28.025+0000 7fe2f4a00640 1 --2- 172.21.15.28:0/2407349899 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe2d006b500 0x7fe2d006d9c0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:28.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:28.025+0000 7fe2f4a00640 1 --2- 172.21.15.28:0/2407349899 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2f01501c0 0x7fe2f01ea760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:28.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:28.025+0000 7fe2f4a00640 1 --2- 172.21.15.28:0/2407349899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2f00c1010 0x7fe2f0151af0 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:28.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:28.025+0000 7fe2f4a00640 1 --2- 172.21.15.28:0/2407349899 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe2f006e870 0x7fe2f01439d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:28.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:28.025+0000 7fe2f4a00640 1 -- 172.21.15.28:0/2407349899 >> 172.21.15.28:0/2407349899 conn(0x7fe2f00c6610 msgr2=0x7fe2f00c69f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:28.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:28.025+0000 7fe2f4a00640 1 -- 172.21.15.28:0/2407349899 shutdown_connections 2024-02-21T23:07:28.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:28.025+0000 7fe2f4a00640 1 -- 172.21.15.28:0/2407349899 wait complete. 2024-02-21T23:07:28.041 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-02-21T23:07:28.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:28.268+0000 7f7607a00640 1 Processor -- start 2024-02-21T23:07:28.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:28.268+0000 7f7607a00640 1 -- start start 2024-02-21T23:07:28.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:28.270+0000 7f7607a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7600150700 0x7f7600156500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:28.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:28.270+0000 7f7607a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7600152fb0 con 0x7f76000c5f50 2024-02-21T23:07:28.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:28.270+0000 7f7607a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7600153290 con 0x7f760006b0e0 2024-02-21T23:07:28.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:28.270+0000 7f7607a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7600153bc0 con 0x7f7600150700 2024-02-21T23:07:28.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:28.270+0000 7f75ffe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f76000c5f50 0x7f760014fff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:45880/0 (socket says 172.21.15.28:45880) 2024-02-21T23:07:28.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:28.270+0000 7f75ffe00640 1 -- 172.21.15.28:0/3612859944 learned_addr learned my addr 172.21.15.28:0/3612859944 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:07:28.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:28.270+0000 7f7605a00640 1 --2- 172.21.15.28:0/3612859944 >> v2:172.21.15.28:3300/0 conn(0x7f7600150700 0x7f7600156500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:28.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:28.271+0000 7f75ff400640 1 -- 172.21.15.28:0/3612859944 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2490225368 0 0) 0x7f7600152fb0 con 0x7f76000c5f50 2024-02-21T23:07:28.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:28.271+0000 7f7605a00640 1 -- 172.21.15.28:0/3612859944 >> v1:172.21.15.81:6789/0 conn(0x7f760006b0e0 legacy=0x7f76000c5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:28.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:28.271+0000 7f7605a00640 1 -- 172.21.15.28:0/3612859944 >> v1:172.21.15.28:6789/0 conn(0x7f76000c5f50 legacy=0x7f760014fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:28.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:28.271+0000 7f7605a00640 1 -- 172.21.15.28:0/3612859944 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7600153570 con 0x7f7600150700 2024-02-21T23:07:28.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:28.272+0000 7f7605a00640 1 --2- 172.21.15.28:0/3612859944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7600150700 0x7f7600156500 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f75f80097a0 tx=0x7f75f802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9592286506edc129 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:28.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:28.273+0000 7f75ff400640 1 -- 172.21.15.28:0/3612859944 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f8035020 con 0x7f7600150700 2024-02-21T23:07:28.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:28.273+0000 7f75ff400640 1 -- 172.21.15.28:0/3612859944 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75f8031830 con 0x7f7600150700 2024-02-21T23:07:28.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:28.273+0000 7f75ff400640 1 -- 172.21.15.28:0/3612859944 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f8031c00 con 0x7f7600150700 2024-02-21T23:07:28.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:28.273+0000 7f7607a00640 1 -- 172.21.15.28:0/3612859944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7600150700 msgr2=0x7f7600156500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:28.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:28.273+0000 7f7607a00640 1 --2- 172.21.15.28:0/3612859944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7600150700 0x7f7600156500 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f75f80097a0 tx=0x7f75f802b8d0 comp rx=0 tx=0).stop 2024-02-21T23:07:28.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:28.274+0000 7f7607a00640 1 -- 172.21.15.28:0/3612859944 shutdown_connections 2024-02-21T23:07:28.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:28.274+0000 7f7607a00640 1 --2- 172.21.15.28:0/3612859944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7600150700 0x7f7600156500 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:28.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:28.274+0000 7f7607a00640 1 -- 172.21.15.28:0/3612859944 >> 172.21.15.28:0/3612859944 conn(0x7f76000a85a0 msgr2=0x7f76000a8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:28.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:28.274+0000 7f7607a00640 1 -- 172.21.15.28:0/3612859944 shutdown_connections 2024-02-21T23:07:28.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:28.274+0000 7f7607a00640 1 -- 172.21.15.28:0/3612859944 wait complete. 2024-02-21T23:07:28.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:28.275+0000 7f7607a00640 1 Processor -- start 2024-02-21T23:07:28.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:28.276+0000 7f7607a00640 1 -- start start 2024-02-21T23:07:28.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:28.276+0000 7f7607a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f760006b0e0 0x7f7600150c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:28.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:28.276+0000 7f7607a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76000c5f50 0x7f76000aa690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:28.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:28.277+0000 7f7607a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7600150700 0x7f7600149a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:28.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:28.277+0000 7f7607a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f760006b640 con 0x7f76000c5f50 2024-02-21T23:07:28.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:28.277+0000 7f7607a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f760006b4c0 con 0x7f760006b0e0 2024-02-21T23:07:28.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:28.277+0000 7f7607a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7600156a40 con 0x7f7600150700 2024-02-21T23:07:28.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:28.277+0000 7f75ffe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76000c5f50 0x7f76000aa690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:28.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:28.277+0000 7f75ffe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76000c5f50 0x7f76000aa690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59374/0 (socket says 172.21.15.28:59374) 2024-02-21T23:07:28.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:28.277+0000 7f75ffe00640 1 -- 172.21.15.28:0/3789951598 learned_addr learned my addr 172.21.15.28:0/3789951598 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:28.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:28.277+0000 7f7605a00640 1 --2- 172.21.15.28:0/3789951598 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7600150700 0x7f7600149a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:28.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:28.277+0000 7f7605000640 1 --2- 172.21.15.28:0/3789951598 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f760006b0e0 0x7f7600150c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:28.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:28.277+0000 7f75ffe00640 1 -- 172.21.15.28:0/3789951598 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7600150700 msgr2=0x7f7600149a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:28.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:28.277+0000 7f75ffe00640 1 --2- 172.21.15.28:0/3789951598 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7600150700 0x7f7600149a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:28.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:28.277+0000 7f75ffe00640 1 -- 172.21.15.28:0/3789951598 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f760006b0e0 msgr2=0x7f7600150c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:28.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:28.277+0000 7f75ffe00640 1 --2- 172.21.15.28:0/3789951598 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f760006b0e0 0x7f7600150c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:28.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:28.277+0000 7f75ffe00640 1 -- 172.21.15.28:0/3789951598 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7600154d40 con 0x7f76000c5f50 2024-02-21T23:07:28.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:28.277+0000 7f7605000640 1 --2- 172.21.15.28:0/3789951598 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f760006b0e0 0x7f7600150c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:07:28.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:28.278+0000 7f75ffe00640 1 --2- 172.21.15.28:0/3789951598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76000c5f50 0x7f76000aa690 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f75ec00c8c0 tx=0x7f75ec00cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:28.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:28.278+0000 7f75fcc00640 1 -- 172.21.15.28:0/3789951598 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ec018070 con 0x7f76000c5f50 2024-02-21T23:07:28.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:28.278+0000 7f75fcc00640 1 -- 172.21.15.28:0/3789951598 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75ec00a640 con 0x7f76000c5f50 2024-02-21T23:07:28.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:28.278+0000 7f75fcc00640 1 -- 172.21.15.28:0/3789951598 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ec00aa30 con 0x7f76000c5f50 2024-02-21T23:07:28.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:28.278+0000 7f7607a00640 1 -- 172.21.15.28:0/3789951598 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7600155860 con 0x7f76000c5f50 2024-02-21T23:07:28.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:28.278+0000 7f7607a00640 1 -- 172.21.15.28:0/3789951598 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7600154990 con 0x7f76000c5f50 2024-02-21T23:07:28.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:28.280+0000 7f7607a00640 1 -- 172.21.15.28:0/3789951598 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7600067f30 con 0x7f76000c5f50 2024-02-21T23:07:28.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:28.281+0000 7f75fcc00640 1 -- 172.21.15.28:0/3789951598 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f75ec00abd0 con 0x7f76000c5f50 2024-02-21T23:07:28.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:28.282+0000 7f75fcc00640 1 --2- 172.21.15.28:0/3789951598 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f75e006b4d0 0x7f75e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:28.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:28.282+0000 7f75fcc00640 1 -- 172.21.15.28:0/3789951598 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5619+0+0 (secure 0 0 0) 0x7f75ec089c40 con 0x7f76000c5f50 2024-02-21T23:07:28.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:28.283+0000 7f7605000640 1 --2- 172.21.15.28:0/3789951598 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f75e006b4d0 0x7f75e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:28.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:28.284+0000 7f7605000640 1 --2- 172.21.15.28:0/3789951598 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f75e006b4d0 0x7f75e006d990 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f7600152600 tx=0x7f75f0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:28.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:28.287+0000 7f75fcc00640 1 -- 172.21.15.28:0/3789951598 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f75ec021020 con 0x7f76000c5f50 2024-02-21T23:07:28.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:28.492+0000 7f7607a00640 1 -- 172.21.15.28:0/3789951598 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f760014f8c0 con 0x7f76000c5f50 2024-02-21T23:07:29.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.037+0000 7f75fcc00640 1 -- 172.21.15.28:0/3789951598 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v108) v1 ==== 170+0+0 (secure 0 0 0) 0x7f75ec058b80 con 0x7f76000c5f50 2024-02-21T23:07:29.025 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' created 2024-02-21T23:07:29.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.042+0000 7f7607a00640 1 -- 172.21.15.28:0/3789951598 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f75e006b4d0 msgr2=0x7f75e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:29.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.042+0000 7f7607a00640 1 --2- 172.21.15.28:0/3789951598 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f75e006b4d0 0x7f75e006d990 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f7600152600 tx=0x7f75f0009040 comp rx=0 tx=0).stop 2024-02-21T23:07:29.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.042+0000 7f7607a00640 1 -- 172.21.15.28:0/3789951598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76000c5f50 msgr2=0x7f76000aa690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:29.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.042+0000 7f7607a00640 1 --2- 172.21.15.28:0/3789951598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76000c5f50 0x7f76000aa690 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f75ec00c8c0 tx=0x7f75ec00cd80 comp rx=0 tx=0).stop 2024-02-21T23:07:29.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.043+0000 7f7607a00640 1 -- 172.21.15.28:0/3789951598 shutdown_connections 2024-02-21T23:07:29.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.043+0000 7f7607a00640 1 --2- 172.21.15.28:0/3789951598 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f75e006b4d0 0x7f75e006d990 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:29.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.043+0000 7f7607a00640 1 --2- 172.21.15.28:0/3789951598 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7600150700 0x7f7600149a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:29.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.043+0000 7f7607a00640 1 --2- 172.21.15.28:0/3789951598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76000c5f50 0x7f76000aa690 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:29.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.043+0000 7f7607a00640 1 --2- 172.21.15.28:0/3789951598 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f760006b0e0 0x7f7600150c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:29.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.043+0000 7f7607a00640 1 -- 172.21.15.28:0/3789951598 >> 172.21.15.28:0/3789951598 conn(0x7f76000a85a0 msgr2=0x7f760014a720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:29.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.043+0000 7f7607a00640 1 -- 172.21.15.28:0/3789951598 shutdown_connections 2024-02-21T23:07:29.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.043+0000 7f7607a00640 1 -- 172.21.15.28:0/3789951598 wait complete. 2024-02-21T23:07:29.073 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-02-21T23:07:29.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.271+0000 7f23b7000640 1 Processor -- start 2024-02-21T23:07:29.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.272+0000 7f23b7000640 1 -- start start 2024-02-21T23:07:29.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.273+0000 7f23b7000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f23b00c6220 0x7f23b01586b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:29.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.273+0000 7f23b7000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f23b0158bf0 0x7f23b015f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:29.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.273+0000 7f23b7000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23b00c20d0 con 0x7f23b00c4960 2024-02-21T23:07:29.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.273+0000 7f23b7000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23b00a62e0 con 0x7f23b00c6220 2024-02-21T23:07:29.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.273+0000 7f23b7000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23b00c1960 con 0x7f23b0158bf0 2024-02-21T23:07:29.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.274+0000 7f23af400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f23b00c6220 0x7f23b01586b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:29.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.274+0000 7f23af400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f23b00c6220 0x7f23b01586b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59390/0 (socket says 172.21.15.28:59390) 2024-02-21T23:07:29.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.274+0000 7f23af400640 1 -- 172.21.15.28:0/3085825007 learned_addr learned my addr 172.21.15.28:0/3085825007 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:29.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.274+0000 7f23b5000640 1 --2- 172.21.15.28:0/3085825007 >> v2:172.21.15.81:3300/0 conn(0x7f23b0158bf0 0x7f23b015f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:29.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.274+0000 7f23af400640 1 -- 172.21.15.28:0/3085825007 >> v2:172.21.15.81:3300/0 conn(0x7f23b0158bf0 msgr2=0x7f23b015f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:29.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.274+0000 7f23af400640 1 --2- 172.21.15.28:0/3085825007 >> v2:172.21.15.81:3300/0 conn(0x7f23b0158bf0 0x7f23b015f9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:29.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.274+0000 7f23af400640 1 -- 172.21.15.28:0/3085825007 >> v1:172.21.15.81:6789/0 conn(0x7f23b00c4960 legacy=0x7f23b01524f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:29.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.274+0000 7f23af400640 1 -- 172.21.15.28:0/3085825007 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23b00c1290 con 0x7f23b00c6220 2024-02-21T23:07:29.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.275+0000 7f23af400640 1 --2- 172.21.15.28:0/3085825007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23b00c6220 0x7f23b01586b0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f23b006f9f0 tx=0x7f239c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fecb25059aae9d3a server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:29.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.276+0000 7f23aea00640 1 -- 172.21.15.28:0/3085825007 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f239c02e070 con 0x7f23b00c6220 2024-02-21T23:07:29.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.276+0000 7f23aea00640 1 -- 172.21.15.28:0/3085825007 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f239c037070 con 0x7f23b00c6220 2024-02-21T23:07:29.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.276+0000 7f23aea00640 1 -- 172.21.15.28:0/3085825007 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f239c031a00 con 0x7f23b00c6220 2024-02-21T23:07:29.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.276+0000 7f23b7000640 1 -- 172.21.15.28:0/3085825007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23b00c6220 msgr2=0x7f23b01586b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:29.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.276+0000 7f23b7000640 1 --2- 172.21.15.28:0/3085825007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23b00c6220 0x7f23b01586b0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f23b006f9f0 tx=0x7f239c02b880 comp rx=0 tx=0).stop 2024-02-21T23:07:29.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.277+0000 7f23b7000640 1 -- 172.21.15.28:0/3085825007 shutdown_connections 2024-02-21T23:07:29.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.277+0000 7f23b7000640 1 --2- 172.21.15.28:0/3085825007 >> v2:172.21.15.81:3300/0 conn(0x7f23b0158bf0 0x7f23b015f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:29.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.277+0000 7f23b7000640 1 --2- 172.21.15.28:0/3085825007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23b00c6220 0x7f23b01586b0 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:29.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.277+0000 7f23b7000640 1 -- 172.21.15.28:0/3085825007 >> 172.21.15.28:0/3085825007 conn(0x7f23b00a85a0 msgr2=0x7f23b00a8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:29.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.277+0000 7f23b7000640 1 -- 172.21.15.28:0/3085825007 shutdown_connections 2024-02-21T23:07:29.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.277+0000 7f23b7000640 1 -- 172.21.15.28:0/3085825007 wait complete. 2024-02-21T23:07:29.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.279+0000 7f23b7000640 1 Processor -- start 2024-02-21T23:07:29.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.279+0000 7f23b7000640 1 -- start start 2024-02-21T23:07:29.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.280+0000 7f23b7000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f23b00c4960 0x7f23b0145b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:29.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.280+0000 7f23b7000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f23b00c6220 0x7f23b01eaf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:29.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.280+0000 7f23b7000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23b0158bf0 0x7f23b014df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:29.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.280+0000 7f23b7000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23b00a5f20 con 0x7f23b0158bf0 2024-02-21T23:07:29.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.280+0000 7f23b7000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23b00c1bb0 con 0x7f23b00c4960 2024-02-21T23:07:29.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.280+0000 7f23b7000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23b00a60a0 con 0x7f23b00c6220 2024-02-21T23:07:29.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.281+0000 7f23b5000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23b0158bf0 0x7f23b014df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:29.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.281+0000 7f23b5000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23b0158bf0 0x7f23b014df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59404/0 (socket says 172.21.15.28:59404) 2024-02-21T23:07:29.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.281+0000 7f23b5000640 1 -- 172.21.15.28:0/2925921579 learned_addr learned my addr 172.21.15.28:0/2925921579 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:29.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.281+0000 7f23afe00640 1 --2- 172.21.15.28:0/2925921579 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f23b00c4960 0x7f23b0145b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:29.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.281+0000 7f23af400640 1 --2- 172.21.15.28:0/2925921579 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f23b00c6220 0x7f23b01eaf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:29.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.281+0000 7f23b5000640 1 -- 172.21.15.28:0/2925921579 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f23b00c6220 msgr2=0x7f23b01eaf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:29.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.281+0000 7f23b5000640 1 --2- 172.21.15.28:0/2925921579 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f23b00c6220 0x7f23b01eaf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:29.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.281+0000 7f23b5000640 1 -- 172.21.15.28:0/2925921579 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f23b00c4960 msgr2=0x7f23b0145b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:29.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.281+0000 7f23b5000640 1 --2- 172.21.15.28:0/2925921579 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f23b00c4960 0x7f23b0145b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:29.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.281+0000 7f23b5000640 1 -- 172.21.15.28:0/2925921579 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23b00ab300 con 0x7f23b0158bf0 2024-02-21T23:07:29.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.282+0000 7f23b5000640 1 --2- 172.21.15.28:0/2925921579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23b0158bf0 0x7f23b014df90 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f23a8007e60 tx=0x7f23a800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:29.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.282+0000 7f23a7e00640 1 -- 172.21.15.28:0/2925921579 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23a8014420 con 0x7f23b0158bf0 2024-02-21T23:07:29.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.282+0000 7f23a7e00640 1 -- 172.21.15.28:0/2925921579 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23a8019070 con 0x7f23b0158bf0 2024-02-21T23:07:29.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.282+0000 7f23a7e00640 1 -- 172.21.15.28:0/2925921579 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23a8014820 con 0x7f23b0158bf0 2024-02-21T23:07:29.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.282+0000 7f23b7000640 1 -- 172.21.15.28:0/2925921579 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23b01ee7a0 con 0x7f23b0158bf0 2024-02-21T23:07:29.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.282+0000 7f23b7000640 1 -- 172.21.15.28:0/2925921579 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23b00ab720 con 0x7f23b0158bf0 2024-02-21T23:07:29.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.283+0000 7f23b7000640 1 -- 172.21.15.28:0/2925921579 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23b0067f30 con 0x7f23b0158bf0 2024-02-21T23:07:29.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.284+0000 7f23a7e00640 1 -- 172.21.15.28:0/2925921579 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f23a80149c0 con 0x7f23b0158bf0 2024-02-21T23:07:29.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.285+0000 7f23a7e00640 1 --2- 172.21.15.28:0/2925921579 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f239006b500 0x7f239006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:29.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.285+0000 7f23a7e00640 1 -- 172.21.15.28:0/2925921579 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6001+0+0 (secure 0 0 0) 0x7f23a808ab70 con 0x7f23b0158bf0 2024-02-21T23:07:29.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.286+0000 7f23afe00640 1 --2- 172.21.15.28:0/2925921579 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f239006b500 0x7f239006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:29.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.287+0000 7f23afe00640 1 --2- 172.21.15.28:0/2925921579 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f239006b500 0x7f239006d9c0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f23b0151ac0 tx=0x7f23a0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:29.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.290+0000 7f23a7e00640 1 -- 172.21.15.28:0/2925921579 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f23a80598b0 con 0x7f23b0158bf0 2024-02-21T23:07:29.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:29.531+0000 7f23b7000640 1 -- 172.21.15.28:0/2925921579 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f23b00ab8f0 con 0x7f23b0158bf0 2024-02-21T23:07:30.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.054+0000 7f23a7e00640 1 -- 172.21.15.28:0/2925921579 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 17 and data pool 18 v98) v1 ==== 174+0+0 (secure 0 0 0) 0x7f23a80591b0 con 0x7f23b0158bf0 2024-02-21T23:07:30.042 INFO:teuthology.orchestra.run.smithi028.stderr:new fs with metadata pool 17 and data pool 18 2024-02-21T23:07:30.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.061+0000 7f23b7000640 1 -- 172.21.15.28:0/2925921579 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f239006b500 msgr2=0x7f239006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:30.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.061+0000 7f23b7000640 1 --2- 172.21.15.28:0/2925921579 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f239006b500 0x7f239006d9c0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f23b0151ac0 tx=0x7f23a0009040 comp rx=0 tx=0).stop 2024-02-21T23:07:30.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.061+0000 7f23b7000640 1 -- 172.21.15.28:0/2925921579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23b0158bf0 msgr2=0x7f23b014df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:30.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.061+0000 7f23b7000640 1 --2- 172.21.15.28:0/2925921579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23b0158bf0 0x7f23b014df90 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f23a8007e60 tx=0x7f23a800e4b0 comp rx=0 tx=0).stop 2024-02-21T23:07:30.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.062+0000 7f23b7000640 1 -- 172.21.15.28:0/2925921579 shutdown_connections 2024-02-21T23:07:30.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.062+0000 7f23b7000640 1 --2- 172.21.15.28:0/2925921579 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f239006b500 0x7f239006d9c0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:30.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.062+0000 7f23b7000640 1 --2- 172.21.15.28:0/2925921579 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f23b00c6220 0x7f23b01eaf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:30.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.062+0000 7f23b7000640 1 --2- 172.21.15.28:0/2925921579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23b0158bf0 0x7f23b014df90 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:30.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.062+0000 7f23b7000640 1 --2- 172.21.15.28:0/2925921579 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f23b00c4960 0x7f23b0145b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:30.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.062+0000 7f23b7000640 1 -- 172.21.15.28:0/2925921579 >> 172.21.15.28:0/2925921579 conn(0x7f23b00a85a0 msgr2=0x7f23b006e160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:30.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.062+0000 7f23b7000640 1 -- 172.21.15.28:0/2925921579 shutdown_connections 2024-02-21T23:07:30.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.063+0000 7f23b7000640 1 -- 172.21.15.28:0/2925921579 wait complete. 2024-02-21T23:07:30.086 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:07:30.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.279+0000 7f702b400640 1 Processor -- start 2024-02-21T23:07:30.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.279+0000 7f702b400640 1 -- start start 2024-02-21T23:07:30.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.281+0000 7f702b400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7024150670 0x7f70241539a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:30.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.281+0000 7f702b400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f70240c6650 0x7f70240c3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:30.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.281+0000 7f702b400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70240c8910 con 0x7f70240c3650 2024-02-21T23:07:30.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.281+0000 7f702b400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70240c86c0 con 0x7f7024150670 2024-02-21T23:07:30.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.281+0000 7f702b400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7024070580 con 0x7f70240c6650 2024-02-21T23:07:30.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.281+0000 7f7028a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7024150670 0x7f70241539a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:30.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.281+0000 7f7028a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7024150670 0x7f70241539a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59412/0 (socket says 172.21.15.28:59412) 2024-02-21T23:07:30.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.281+0000 7f7028a00640 1 -- 172.21.15.28:0/1645031210 learned_addr learned my addr 172.21.15.28:0/1645031210 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:30.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.282+0000 7f7023e00640 1 --2- 172.21.15.28:0/1645031210 >> v2:172.21.15.81:3300/0 conn(0x7f70240c6650 0x7f70240c3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:30.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.282+0000 7f7028a00640 1 -- 172.21.15.28:0/1645031210 >> v2:172.21.15.81:3300/0 conn(0x7f70240c6650 msgr2=0x7f70240c3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:30.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.282+0000 7f7028a00640 1 --2- 172.21.15.28:0/1645031210 >> v2:172.21.15.81:3300/0 conn(0x7f70240c6650 0x7f70240c3110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:30.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.282+0000 7f7028a00640 1 -- 172.21.15.28:0/1645031210 >> v1:172.21.15.81:6789/0 conn(0x7f70240c3650 legacy=0x7f7024070a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:30.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.282+0000 7f7028a00640 1 -- 172.21.15.28:0/1645031210 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70240c8bf0 con 0x7f7024150670 2024-02-21T23:07:30.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.282+0000 7f7023e00640 1 --2- 172.21.15.28:0/1645031210 >> v2:172.21.15.81:3300/0 conn(0x7f70240c6650 0x7f70240c3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:07:30.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.283+0000 7f7028a00640 1 --2- 172.21.15.28:0/1645031210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7024150670 0x7f70241539a0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f7024067ac0 tx=0x7f701802b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56ac81da5451b82a server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:30.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.283+0000 7f7023400640 1 -- 172.21.15.28:0/1645031210 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7018035020 con 0x7f7024150670 2024-02-21T23:07:30.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.284+0000 7f7023400640 1 -- 172.21.15.28:0/1645031210 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7018031800 con 0x7f7024150670 2024-02-21T23:07:30.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.284+0000 7f7023400640 1 -- 172.21.15.28:0/1645031210 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7018031bd0 con 0x7f7024150670 2024-02-21T23:07:30.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.284+0000 7f702b400640 1 -- 172.21.15.28:0/1645031210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7024150670 msgr2=0x7f70241539a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:30.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.284+0000 7f702b400640 1 --2- 172.21.15.28:0/1645031210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7024150670 0x7f70241539a0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f7024067ac0 tx=0x7f701802b8a0 comp rx=0 tx=0).stop 2024-02-21T23:07:30.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.284+0000 7f702b400640 1 -- 172.21.15.28:0/1645031210 shutdown_connections 2024-02-21T23:07:30.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.284+0000 7f702b400640 1 --2- 172.21.15.28:0/1645031210 >> v2:172.21.15.81:3300/0 conn(0x7f70240c6650 0x7f70240c3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:30.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.284+0000 7f702b400640 1 --2- 172.21.15.28:0/1645031210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7024150670 0x7f70241539a0 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:30.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.284+0000 7f702b400640 1 -- 172.21.15.28:0/1645031210 >> 172.21.15.28:0/1645031210 conn(0x7f70240aa4c0 msgr2=0x7f70240aa8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:30.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.284+0000 7f702b400640 1 -- 172.21.15.28:0/1645031210 shutdown_connections 2024-02-21T23:07:30.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.284+0000 7f702b400640 1 -- 172.21.15.28:0/1645031210 wait complete. 2024-02-21T23:07:30.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.286+0000 7f702b400640 1 Processor -- start 2024-02-21T23:07:30.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.286+0000 7f702b400640 1 -- start start 2024-02-21T23:07:30.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.287+0000 7f702b400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70240c3650 0x7f70241542d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:30.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.287+0000 7f702b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70240c6650 0x7f70240c1db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:30.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.287+0000 7f702b400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7024150670 0x7f70241eb080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:30.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.287+0000 7f702b400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7024071120 con 0x7f70240c6650 2024-02-21T23:07:30.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.287+0000 7f702b400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70240c6d60 con 0x7f70240c3650 2024-02-21T23:07:30.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.287+0000 7f702b400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70240712a0 con 0x7f7024150670 2024-02-21T23:07:30.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.287+0000 7f7023e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70240c6650 0x7f70240c1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:30.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.287+0000 7f7023e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70240c6650 0x7f70240c1db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59420/0 (socket says 172.21.15.28:59420) 2024-02-21T23:07:30.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.287+0000 7f7023e00640 1 -- 172.21.15.28:0/2022338450 learned_addr learned my addr 172.21.15.28:0/2022338450 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:30.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.288+0000 7f7028a00640 1 --2- 172.21.15.28:0/2022338450 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70240c3650 0x7f70241542d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:30.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.288+0000 7f7029400640 1 --2- 172.21.15.28:0/2022338450 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7024150670 0x7f70241eb080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:30.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.288+0000 7f7023e00640 1 -- 172.21.15.28:0/2022338450 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7024150670 msgr2=0x7f70241eb080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:30.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.288+0000 7f7023e00640 1 --2- 172.21.15.28:0/2022338450 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7024150670 0x7f70241eb080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:30.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.288+0000 7f7023e00640 1 -- 172.21.15.28:0/2022338450 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70240c3650 msgr2=0x7f70241542d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:30.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.288+0000 7f7023e00640 1 --2- 172.21.15.28:0/2022338450 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70240c3650 0x7f70241542d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:30.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.288+0000 7f7023e00640 1 -- 172.21.15.28:0/2022338450 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70241ea8a0 con 0x7f70240c6650 2024-02-21T23:07:30.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.288+0000 7f7029400640 1 --2- 172.21.15.28:0/2022338450 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7024150670 0x7f70241eb080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:07:30.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.288+0000 7f7023e00640 1 --2- 172.21.15.28:0/2022338450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70240c6650 0x7f70240c1db0 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f701000c7f0 tx=0x7f701000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:30.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.288+0000 7f7020c00640 1 -- 172.21.15.28:0/2022338450 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7010004230 con 0x7f70240c6650 2024-02-21T23:07:30.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.289+0000 7f7020c00640 1 -- 172.21.15.28:0/2022338450 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70100043d0 con 0x7f70240c6650 2024-02-21T23:07:30.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.289+0000 7f702b400640 1 -- 172.21.15.28:0/2022338450 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f702414d040 con 0x7f70240c6650 2024-02-21T23:07:30.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.289+0000 7f7020c00640 1 -- 172.21.15.28:0/2022338450 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7010007a70 con 0x7f70240c6650 2024-02-21T23:07:30.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.289+0000 7f702b400640 1 -- 172.21.15.28:0/2022338450 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f702414d600 con 0x7f70240c6650 2024-02-21T23:07:30.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.290+0000 7f702b400640 1 -- 172.21.15.28:0/2022338450 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7024067f30 con 0x7f70240c6650 2024-02-21T23:07:30.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.291+0000 7f7020c00640 1 -- 172.21.15.28:0/2022338450 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7010007c10 con 0x7f70240c6650 2024-02-21T23:07:30.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.292+0000 7f7020c00640 1 --2- 172.21.15.28:0/2022338450 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f700406b500 0x7f700406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:30.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.292+0000 7f7020c00640 1 -- 172.21.15.28:0/2022338450 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f701008a450 con 0x7f70240c6650 2024-02-21T23:07:30.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.292+0000 7f7028a00640 1 --2- 172.21.15.28:0/2022338450 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f700406b500 0x7f700406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:30.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.293+0000 7f7028a00640 1 --2- 172.21.15.28:0/2022338450 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f700406b500 0x7f700406d9c0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f7018036000 tx=0x7f701803c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:30.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.296+0000 7f7020c00640 1 -- 172.21.15.28:0/2022338450 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7010015020 con 0x7f70240c6650 2024-02-21T23:07:30.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.498+0000 7f702b400640 1 -- 172.21.15.28:0/2022338450 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7024071b50 con 0x7f70240c6650 2024-02-21T23:07:30.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.500+0000 7f7020c00640 1 -- 172.21.15.28:0/2022338450 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v109) v1 ==== 74+0+16905 (secure 0 0 0) 0x7f7010059130 con 0x7f70240c6650 2024-02-21T23:07:30.489 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:07:30.489 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":109,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:07:29.532953+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:07:27.477155+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-02-21T23:07:28.549456+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:07:30.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.503+0000 7f702b400640 1 -- 172.21.15.28:0/2022338450 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f700406b500 msgr2=0x7f700406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:30.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.503+0000 7f702b400640 1 --2- 172.21.15.28:0/2022338450 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f700406b500 0x7f700406d9c0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f7018036000 tx=0x7f701803c040 comp rx=0 tx=0).stop 2024-02-21T23:07:30.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.503+0000 7f702b400640 1 -- 172.21.15.28:0/2022338450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70240c6650 msgr2=0x7f70240c1db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:30.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.503+0000 7f702b400640 1 --2- 172.21.15.28:0/2022338450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70240c6650 0x7f70240c1db0 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f701000c7f0 tx=0x7f701000ccb0 comp rx=0 tx=0).stop 2024-02-21T23:07:30.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.503+0000 7f702b400640 1 -- 172.21.15.28:0/2022338450 shutdown_connections 2024-02-21T23:07:30.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.503+0000 7f702b400640 1 --2- 172.21.15.28:0/2022338450 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f700406b500 0x7f700406d9c0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:30.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.503+0000 7f702b400640 1 --2- 172.21.15.28:0/2022338450 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7024150670 0x7f70241eb080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:30.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.503+0000 7f702b400640 1 --2- 172.21.15.28:0/2022338450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70240c6650 0x7f70240c1db0 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:30.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.503+0000 7f702b400640 1 --2- 172.21.15.28:0/2022338450 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70240c3650 0x7f70241542d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:30.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.503+0000 7f702b400640 1 -- 172.21.15.28:0/2022338450 >> 172.21.15.28:0/2022338450 conn(0x7f70240aa4c0 msgr2=0x7f7024074360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:30.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.504+0000 7f702b400640 1 -- 172.21.15.28:0/2022338450 shutdown_connections 2024-02-21T23:07:30.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.504+0000 7f702b400640 1 -- 172.21.15.28:0/2022338450 wait complete. 2024-02-21T23:07:30.510 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:07:30.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.739+0000 7f70a3e00640 1 Processor -- start 2024-02-21T23:07:30.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.740+0000 7f70a3e00640 1 -- start start 2024-02-21T23:07:30.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.741+0000 7f70a3e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f70a40aff90 0x7f70a40c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:30.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.741+0000 7f70a3e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70a40c74c0 con 0x7f70a406dbf0 2024-02-21T23:07:30.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.741+0000 7f70a3e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70a40c77a0 con 0x7f70a40c5720 2024-02-21T23:07:30.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.741+0000 7f70a3e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70a40c7270 con 0x7f70a40aff90 2024-02-21T23:07:30.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.741+0000 7f70a2000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f70a406dbf0 0x7f70a4158ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:45882/0 (socket says 172.21.15.28:45882) 2024-02-21T23:07:30.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.741+0000 7f70a2000640 1 -- 172.21.15.28:0/2331743311 learned_addr learned my addr 172.21.15.28:0/2331743311 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:07:30.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.742+0000 7f70a1600640 1 -- 172.21.15.28:0/2331743311 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1218435827 0 0) 0x7f70a40c74c0 con 0x7f70a406dbf0 2024-02-21T23:07:30.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.742+0000 7f70a2a00640 1 --2- 172.21.15.28:0/2331743311 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f70a40aff90 0x7f70a40c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:30.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.742+0000 7f70a1600640 1 -- 172.21.15.28:0/2331743311 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7088003520 con 0x7f70a406dbf0 2024-02-21T23:07:30.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.742+0000 7f70a1600640 1 -- 172.21.15.28:0/2331743311 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1920319131 0 0) 0x7f7088003520 con 0x7f70a406dbf0 2024-02-21T23:07:30.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.742+0000 7f70a1600640 1 -- 172.21.15.28:0/2331743311 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f70a40aff90 msgr2=0x7f70a40c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:30.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.742+0000 7f70a1600640 1 --2- 172.21.15.28:0/2331743311 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f70a40aff90 0x7f70a40c51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:30.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.742+0000 7f70a1600640 1 -- 172.21.15.28:0/2331743311 >> v1:172.21.15.81:6789/0 conn(0x7f70a40c5720 legacy=0x7f70a415f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:30.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.742+0000 7f70a1600640 1 -- 172.21.15.28:0/2331743311 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70a40c7a80 con 0x7f70a406dbf0 2024-02-21T23:07:30.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.742+0000 7f70a1600640 1 -- 172.21.15.28:0/2331743311 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f708c001670 con 0x7f70a406dbf0 2024-02-21T23:07:30.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.742+0000 7f70a1600640 1 -- 172.21.15.28:0/2331743311 >> v1:172.21.15.28:6789/0 conn(0x7f70a406dbf0 legacy=0x7f70a4158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:30.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.742+0000 7f70a1600640 1 --2- 172.21.15.28:0/2331743311 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f70a40aff90 0x7f70880039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:30.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.743+0000 7f70a1600640 1 --2- 172.21.15.28:0/2331743311 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7088004010 0x7f7088004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:30.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.743+0000 7f70a1600640 1 --2- 172.21.15.28:0/2331743311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7088004d40 0x7f70880051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:30.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.743+0000 7f70a1600640 1 -- 172.21.15.28:0/2331743311 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70a40c7270 con 0x7f7088004d40 2024-02-21T23:07:30.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.743+0000 7f70a1600640 1 -- 172.21.15.28:0/2331743311 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f708800bb80 con 0x7f7088004010 2024-02-21T23:07:30.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.743+0000 7f70a1600640 1 -- 172.21.15.28:0/2331743311 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f708800bd80 con 0x7f70a40aff90 2024-02-21T23:07:30.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.743+0000 7f70a3400640 1 --2- 172.21.15.28:0/2331743311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7088004d40 0x7f70880051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:30.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.743+0000 7f70a2a00640 1 --2- 172.21.15.28:0/2331743311 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f70a40aff90 0x7f70880039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:30.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.743+0000 7f70a2000640 1 --2- 172.21.15.28:0/2331743311 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7088004010 0x7f7088004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:30.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.743+0000 7f70a3400640 1 -- 172.21.15.28:0/2331743311 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f70a40aff90 msgr2=0x7f70880039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:30.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.743+0000 7f70a3400640 1 --2- 172.21.15.28:0/2331743311 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f70a40aff90 0x7f70880039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:30.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.743+0000 7f70a3400640 1 -- 172.21.15.28:0/2331743311 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7088004010 msgr2=0x7f7088004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:30.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.743+0000 7f70a3400640 1 --2- 172.21.15.28:0/2331743311 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7088004010 0x7f7088004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:30.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.743+0000 7f70a3400640 1 -- 172.21.15.28:0/2331743311 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7088003520 con 0x7f7088004d40 2024-02-21T23:07:30.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.745+0000 7f70a3400640 1 --2- 172.21.15.28:0/2331743311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7088004d40 0x7f70880051c0 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f709c001e50 tx=0x7f7088005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=268229d3bd07fc6f server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:30.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.745+0000 7f70a1600640 1 -- 172.21.15.28:0/2331743311 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f709c001a70 con 0x7f7088004d40 2024-02-21T23:07:30.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.745+0000 7f70a1600640 1 -- 172.21.15.28:0/2331743311 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f709c003b70 con 0x7f7088004d40 2024-02-21T23:07:30.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.745+0000 7f70a1600640 1 -- 172.21.15.28:0/2331743311 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f709c0050d0 con 0x7f7088004d40 2024-02-21T23:07:30.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.746+0000 7f70a3e00640 1 -- 172.21.15.28:0/2331743311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7088004d40 msgr2=0x7f70880051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:30.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.746+0000 7f70a3e00640 1 --2- 172.21.15.28:0/2331743311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7088004d40 0x7f70880051c0 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f709c001e50 tx=0x7f7088005830 comp rx=0 tx=0).stop 2024-02-21T23:07:30.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.746+0000 7f70a3400640 1 -- 172.21.15.28:0/2331743311 reap_dead start 2024-02-21T23:07:30.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.746+0000 7f70a3e00640 1 -- 172.21.15.28:0/2331743311 shutdown_connections 2024-02-21T23:07:30.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.746+0000 7f70a3e00640 1 -- 172.21.15.28:0/2331743311 >> 172.21.15.28:0/2331743311 conn(0x7f70a40ac370 msgr2=0x7f70a40ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:30.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.746+0000 7f70a3e00640 1 -- 172.21.15.28:0/2331743311 shutdown_connections 2024-02-21T23:07:30.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.746+0000 7f70a3e00640 1 -- 172.21.15.28:0/2331743311 wait complete. 2024-02-21T23:07:30.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.747+0000 7f70a3e00640 1 Processor -- start 2024-02-21T23:07:30.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.748+0000 7f70a3e00640 1 -- start start 2024-02-21T23:07:30.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.748+0000 7f70a3e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7088004d40 0x7f70a40c8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:30.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.749+0000 7f70a3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70a41ee870 0x7f70a40be360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:30.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.749+0000 7f70a3e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70a40be8a0 0x7f70a40c1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:30.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.749+0000 7f70a3e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70a40c8760 con 0x7f70a41ee870 2024-02-21T23:07:30.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.749+0000 7f70a3e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70a4158630 con 0x7f70a40be8a0 2024-02-21T23:07:30.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.749+0000 7f70a3e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70a40c88e0 con 0x7f7088004d40 2024-02-21T23:07:30.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.749+0000 7f70a2000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70a41ee870 0x7f70a40be360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:30.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.749+0000 7f70a2000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70a41ee870 0x7f70a40be360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59450/0 (socket says 172.21.15.28:59450) 2024-02-21T23:07:30.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.749+0000 7f70a2000640 1 -- 172.21.15.28:0/3162405946 learned_addr learned my addr 172.21.15.28:0/3162405946 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:30.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.749+0000 7f70a2a00640 1 --2- 172.21.15.28:0/3162405946 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7088004d40 0x7f70a40c8cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:30.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.749+0000 7f70a3400640 1 --2- 172.21.15.28:0/3162405946 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70a40be8a0 0x7f70a40c1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:30.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.749+0000 7f70a2000640 1 -- 172.21.15.28:0/3162405946 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7088004d40 msgr2=0x7f70a40c8cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:30.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.749+0000 7f70a2000640 1 --2- 172.21.15.28:0/3162405946 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7088004d40 0x7f70a40c8cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:30.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.749+0000 7f70a2000640 1 -- 172.21.15.28:0/3162405946 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70a40be8a0 msgr2=0x7f70a40c1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:30.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.749+0000 7f70a2000640 1 --2- 172.21.15.28:0/3162405946 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70a40be8a0 0x7f70a40c1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:30.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.749+0000 7f70a2000640 1 -- 172.21.15.28:0/3162405946 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70a40c4680 con 0x7f70a41ee870 2024-02-21T23:07:30.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.750+0000 7f70a2000640 1 --2- 172.21.15.28:0/3162405946 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70a41ee870 0x7f70a40be360 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f708c00e800 tx=0x7f708c00ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:30.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.750+0000 7f709aa00640 1 -- 172.21.15.28:0/3162405946 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f708c00c870 con 0x7f70a41ee870 2024-02-21T23:07:30.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.750+0000 7f709aa00640 1 -- 172.21.15.28:0/3162405946 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f708c018070 con 0x7f70a41ee870 2024-02-21T23:07:30.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.750+0000 7f70a3e00640 1 -- 172.21.15.28:0/3162405946 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70a41edd90 con 0x7f70a41ee870 2024-02-21T23:07:30.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.750+0000 7f709aa00640 1 -- 172.21.15.28:0/3162405946 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f708c00cc50 con 0x7f70a41ee870 2024-02-21T23:07:30.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.750+0000 7f70a3e00640 1 -- 172.21.15.28:0/3162405946 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70a40c9210 con 0x7f70a41ee870 2024-02-21T23:07:30.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.751+0000 7f70a3e00640 1 -- 172.21.15.28:0/3162405946 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70a404f290 con 0x7f70a41ee870 2024-02-21T23:07:30.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.753+0000 7f709aa00640 1 -- 172.21.15.28:0/3162405946 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f708c01d460 con 0x7f70a41ee870 2024-02-21T23:07:30.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.754+0000 7f709aa00640 1 --2- 172.21.15.28:0/3162405946 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f708006b500 0x7f708006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:30.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.754+0000 7f709aa00640 1 -- 172.21.15.28:0/3162405946 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f708c08a2c0 con 0x7f70a41ee870 2024-02-21T23:07:30.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.755+0000 7f70a2a00640 1 --2- 172.21.15.28:0/3162405946 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f708006b500 0x7f708006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:30.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.756+0000 7f70a2a00640 1 --2- 172.21.15.28:0/3162405946 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f708006b500 0x7f708006d9c0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f70a4157080 tx=0x7f7094008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:30.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.759+0000 7f709aa00640 1 -- 172.21.15.28:0/3162405946 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f708c058fa0 con 0x7f70a41ee870 2024-02-21T23:07:30.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.960+0000 7f70a3e00640 1 -- 172.21.15.28:0/3162405946 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f70a406d9d0 con 0x7f70a41ee870 2024-02-21T23:07:30.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.962+0000 7f709aa00640 1 -- 172.21.15.28:0/3162405946 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v109) v1 ==== 74+0+16905 (secure 0 0 0) 0x7f708c0588a0 con 0x7f70a41ee870 2024-02-21T23:07:30.951 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:07:30.951 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":109,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:07:29.532953+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:07:27.477155+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-02-21T23:07:28.549456+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:07:30.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.965+0000 7f70a3e00640 1 -- 172.21.15.28:0/3162405946 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f708006b500 msgr2=0x7f708006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:30.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.965+0000 7f70a3e00640 1 --2- 172.21.15.28:0/3162405946 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f708006b500 0x7f708006d9c0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f70a4157080 tx=0x7f7094008040 comp rx=0 tx=0).stop 2024-02-21T23:07:30.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.965+0000 7f70a3e00640 1 -- 172.21.15.28:0/3162405946 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70a41ee870 msgr2=0x7f70a40be360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:30.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.965+0000 7f70a3e00640 1 --2- 172.21.15.28:0/3162405946 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70a41ee870 0x7f70a40be360 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f708c00e800 tx=0x7f708c00ecc0 comp rx=0 tx=0).stop 2024-02-21T23:07:30.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.965+0000 7f70a3e00640 1 -- 172.21.15.28:0/3162405946 shutdown_connections 2024-02-21T23:07:30.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.965+0000 7f70a3e00640 1 --2- 172.21.15.28:0/3162405946 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f708006b500 0x7f708006d9c0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:30.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.965+0000 7f70a3e00640 1 --2- 172.21.15.28:0/3162405946 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70a40be8a0 0x7f70a40c1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:30.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.965+0000 7f70a3e00640 1 --2- 172.21.15.28:0/3162405946 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70a41ee870 0x7f70a40be360 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:30.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.965+0000 7f70a3e00640 1 --2- 172.21.15.28:0/3162405946 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7088004d40 0x7f70a40c8cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:30.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.965+0000 7f70a3e00640 1 -- 172.21.15.28:0/3162405946 >> 172.21.15.28:0/3162405946 conn(0x7f70a40ac370 msgr2=0x7f70a40ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:30.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.965+0000 7f70a3e00640 1 -- 172.21.15.28:0/3162405946 shutdown_connections 2024-02-21T23:07:30.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:30.966+0000 7f70a3e00640 1 -- 172.21.15.28:0/3162405946 wait complete. 2024-02-21T23:07:30.971 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-02-21T23:07:31.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:31.221+0000 7f63ffe00640 1 Processor -- start 2024-02-21T23:07:31.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:31.222+0000 7f63ffe00640 1 -- start start 2024-02-21T23:07:31.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:31.223+0000 7f63ffe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6400152560 0x7f6400154f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:31.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:31.223+0000 7f63ffe00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f6400158480 0x7f640006ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:31.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:31.223+0000 7f63ffe00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64000a6270 con 0x7f640006b3b0 2024-02-21T23:07:31.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:31.223+0000 7f63ffe00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64001554a0 con 0x7f6400152560 2024-02-21T23:07:31.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:31.223+0000 7f63ffe00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64001556d0 con 0x7f6400158480 2024-02-21T23:07:31.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:31.224+0000 7f63fea00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6400152560 0x7f6400154f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:31.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:31.224+0000 7f63ff400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f640006b3b0 0x7f640006e9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:45896/0 (socket says 172.21.15.28:45896) 2024-02-21T23:07:31.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:31.224+0000 7f63ff400640 1 -- 172.21.15.28:0/4181312028 learned_addr learned my addr 172.21.15.28:0/4181312028 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:07:31.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:31.224+0000 7f63fe000640 1 --2- 172.21.15.28:0/4181312028 >> v2:172.21.15.81:3300/0 conn(0x7f6400158480 0x7f640006ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:31.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:31.224+0000 7f63fea00640 1 -- 172.21.15.28:0/4181312028 >> v2:172.21.15.81:3300/0 conn(0x7f6400158480 msgr2=0x7f640006ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:31.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:31.224+0000 7f63fea00640 1 --2- 172.21.15.28:0/4181312028 >> v2:172.21.15.81:3300/0 conn(0x7f6400158480 0x7f640006ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:31.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:31.224+0000 7f63fea00640 1 -- 172.21.15.28:0/4181312028 >> v1:172.21.15.28:6789/0 conn(0x7f640006b3b0 legacy=0x7f640006e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:31.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:31.224+0000 7f63fea00640 1 -- 172.21.15.28:0/4181312028 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64001559b0 con 0x7f6400152560 2024-02-21T23:07:31.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:31.224+0000 7f63fd600640 1 -- 172.21.15.28:0/4181312028 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 692608848 0 0) 0x7f64000a6270 con 0x7f640006b3b0 2024-02-21T23:07:31.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:31.226+0000 7f63fea00640 1 --2- 172.21.15.28:0/4181312028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6400152560 0x7f6400154f60 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f640006f7f0 tx=0x7f63f002bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b127fcbcabeb264 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:31.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:31.227+0000 7f63fd600640 1 -- 172.21.15.28:0/4181312028 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0039070 con 0x7f6400152560 2024-02-21T23:07:31.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:31.227+0000 7f63fd600640 1 -- 172.21.15.28:0/4181312028 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f002cc70 con 0x7f6400152560 2024-02-21T23:07:31.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:31.227+0000 7f63fd600640 1 -- 172.21.15.28:0/4181312028 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0034660 con 0x7f6400152560 2024-02-21T23:07:31.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:31.227+0000 7f63ffe00640 1 -- 172.21.15.28:0/4181312028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6400152560 msgr2=0x7f6400154f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:31.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:31.227+0000 7f63ffe00640 1 --2- 172.21.15.28:0/4181312028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6400152560 0x7f6400154f60 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f640006f7f0 tx=0x7f63f002bec0 comp rx=0 tx=0).stop 2024-02-21T23:07:31.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:31.227+0000 7f63ffe00640 1 -- 172.21.15.28:0/4181312028 shutdown_connections 2024-02-21T23:07:31.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:31.227+0000 7f63ffe00640 1 --2- 172.21.15.28:0/4181312028 >> v2:172.21.15.81:3300/0 conn(0x7f6400158480 0x7f640006ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:31.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:31.227+0000 7f63ffe00640 1 --2- 172.21.15.28:0/4181312028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6400152560 0x7f6400154f60 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:31.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:31.227+0000 7f63ffe00640 1 -- 172.21.15.28:0/4181312028 >> 172.21.15.28:0/4181312028 conn(0x7f64000a8530 msgr2=0x7f64000a8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:31.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:31.227+0000 7f63ffe00640 1 -- 172.21.15.28:0/4181312028 shutdown_connections 2024-02-21T23:07:31.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:31.227+0000 7f63ffe00640 1 -- 172.21.15.28:0/4181312028 wait complete. 2024-02-21T23:07:31.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:31.230+0000 7f63ffe00640 1 Processor -- start 2024-02-21T23:07:31.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:31.230+0000 7f63ffe00640 1 -- start start 2024-02-21T23:07:31.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:31.232+0000 7f63ffe00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f640006b3b0 0x7f64001498f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:31.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:31.232+0000 7f63ffe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6400152560 0x7f64000719c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:31.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:31.232+0000 7f63ffe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6400158480 0x7f6400071f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:31.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:31.232+0000 7f63ffe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6400152940 con 0x7f6400152560 2024-02-21T23:07:31.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:31.232+0000 7f63ffe00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f640006e4b0 con 0x7f6400158480 2024-02-21T23:07:31.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:31.232+0000 7f63ffe00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64000c1bc0 con 0x7f640006b3b0 2024-02-21T23:07:31.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:31.232+0000 7f63fe000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6400152560 0x7f64000719c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:31.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:31.232+0000 7f63fe000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6400152560 0x7f64000719c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59470/0 (socket says 172.21.15.28:59470) 2024-02-21T23:07:31.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:31.232+0000 7f63fe000640 1 -- 172.21.15.28:0/1676815293 learned_addr learned my addr 172.21.15.28:0/1676815293 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:31.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:31.232+0000 7f63fea00640 1 --2- 172.21.15.28:0/1676815293 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f640006b3b0 0x7f64001498f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:31.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:31.233+0000 7f63ff400640 1 --2- 172.21.15.28:0/1676815293 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6400158480 0x7f6400071f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:31.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:31.233+0000 7f63fe000640 1 -- 172.21.15.28:0/1676815293 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f640006b3b0 msgr2=0x7f64001498f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:31.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:31.233+0000 7f63fe000640 1 --2- 172.21.15.28:0/1676815293 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f640006b3b0 0x7f64001498f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:31.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:31.233+0000 7f63fe000640 1 -- 172.21.15.28:0/1676815293 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6400158480 msgr2=0x7f6400071f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:31.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:31.233+0000 7f63fe000640 1 --2- 172.21.15.28:0/1676815293 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6400158480 0x7f6400071f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:31.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:31.233+0000 7f63fe000640 1 -- 172.21.15.28:0/1676815293 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64001ea9e0 con 0x7f6400152560 2024-02-21T23:07:31.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:31.233+0000 7f63ff400640 1 --2- 172.21.15.28:0/1676815293 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6400158480 0x7f6400071f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:07:31.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:31.233+0000 7f63fe000640 1 --2- 172.21.15.28:0/1676815293 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6400152560 0x7f64000719c0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f63e800d800 tx=0x7f63e800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:31.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:31.233+0000 7f63f6a00640 1 -- 172.21.15.28:0/1676815293 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63e800b660 con 0x7f6400152560 2024-02-21T23:07:31.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:31.233+0000 7f63f6a00640 1 -- 172.21.15.28:0/1676815293 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63e8019070 con 0x7f6400152560 2024-02-21T23:07:31.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:31.233+0000 7f63ffe00640 1 -- 172.21.15.28:0/1676815293 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64001eacc0 con 0x7f6400152560 2024-02-21T23:07:31.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:31.233+0000 7f63f6a00640 1 -- 172.21.15.28:0/1676815293 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63e800ba20 con 0x7f6400152560 2024-02-21T23:07:31.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:31.233+0000 7f63ffe00640 1 -- 172.21.15.28:0/1676815293 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6400149e30 con 0x7f6400152560 2024-02-21T23:07:31.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:31.234+0000 7f63ffe00640 1 -- 172.21.15.28:0/1676815293 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f640004f290 con 0x7f6400152560 2024-02-21T23:07:31.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:31.236+0000 7f63f6a00640 1 -- 172.21.15.28:0/1676815293 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f63e800bbc0 con 0x7f6400152560 2024-02-21T23:07:31.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:31.237+0000 7f63f6a00640 1 --2- 172.21.15.28:0/1676815293 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f63dc06b4d0 0x7f63dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:31.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:31.237+0000 7f63f6a00640 1 -- 172.21.15.28:0/1676815293 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f63e808a790 con 0x7f6400152560 2024-02-21T23:07:31.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:31.238+0000 7f63fea00640 1 --2- 172.21.15.28:0/1676815293 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f63dc06b4d0 0x7f63dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:31.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:31.239+0000 7f63fea00640 1 --2- 172.21.15.28:0/1676815293 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f63dc06b4d0 0x7f63dc06d990 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f63f0036000 tx=0x7f63f0030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:31.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:31.241+0000 7f63f6a00640 1 -- 172.21.15.28:0/1676815293 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f63e801c360 con 0x7f6400152560 2024-02-21T23:07:31.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:31.480+0000 7f63ffe00640 1 -- 172.21.15.28:0/1676815293 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f640006e0f0 con 0x7f6400152560 2024-02-21T23:07:32.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.072+0000 7f63f6a00640 1 -- 172.21.15.28:0/1676815293 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v101) v1 ==== 118+0+0 (secure 0 0 0) 0x7f63e8059510 con 0x7f6400152560 2024-02-21T23:07:32.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.075+0000 7f63ffe00640 1 -- 172.21.15.28:0/1676815293 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f63dc06b4d0 msgr2=0x7f63dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:32.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.075+0000 7f63ffe00640 1 --2- 172.21.15.28:0/1676815293 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f63dc06b4d0 0x7f63dc06d990 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f63f0036000 tx=0x7f63f0030000 comp rx=0 tx=0).stop 2024-02-21T23:07:32.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.075+0000 7f63ffe00640 1 -- 172.21.15.28:0/1676815293 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6400152560 msgr2=0x7f64000719c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:32.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.075+0000 7f63ffe00640 1 --2- 172.21.15.28:0/1676815293 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6400152560 0x7f64000719c0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f63e800d800 tx=0x7f63e800dcc0 comp rx=0 tx=0).stop 2024-02-21T23:07:32.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.075+0000 7f63ffe00640 1 -- 172.21.15.28:0/1676815293 shutdown_connections 2024-02-21T23:07:32.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.075+0000 7f63ffe00640 1 --2- 172.21.15.28:0/1676815293 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f63dc06b4d0 0x7f63dc06d990 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:32.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.075+0000 7f63ffe00640 1 --2- 172.21.15.28:0/1676815293 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6400158480 0x7f6400071f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:32.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.075+0000 7f63ffe00640 1 --2- 172.21.15.28:0/1676815293 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6400152560 0x7f64000719c0 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:32.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.075+0000 7f63ffe00640 1 --2- 172.21.15.28:0/1676815293 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f640006b3b0 0x7f64001498f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:32.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.076+0000 7f63ffe00640 1 -- 172.21.15.28:0/1676815293 >> 172.21.15.28:0/1676815293 conn(0x7f64000a8530 msgr2=0x7f64000c4910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:32.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.076+0000 7f63ffe00640 1 -- 172.21.15.28:0/1676815293 shutdown_connections 2024-02-21T23:07:32.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.076+0000 7f63ffe00640 1 -- 172.21.15.28:0/1676815293 wait complete. 2024-02-21T23:07:32.086 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-02-21T23:07:32.086 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-02-21T23:07:32.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.324+0000 7f9c7d000640 1 Processor -- start 2024-02-21T23:07:32.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.324+0000 7f9c7d000640 1 -- start start 2024-02-21T23:07:32.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.325+0000 7f9c7d000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9c78150500 0x7f9c780a6d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:32.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.325+0000 7f9c7d000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9c780c3800 0x7f9c780c3c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:32.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.325+0000 7f9c7d000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c780c22e0 con 0x7f9c781530e0 2024-02-21T23:07:32.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.325+0000 7f9c7d000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c78152be0 con 0x7f9c78150500 2024-02-21T23:07:32.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.325+0000 7f9c7d000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c7806c5c0 con 0x7f9c780c3800 2024-02-21T23:07:32.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.326+0000 7f9c77400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9c78150500 0x7f9c780a6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:32.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.326+0000 7f9c77400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9c78150500 0x7f9c780a6d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59474/0 (socket says 172.21.15.28:59474) 2024-02-21T23:07:32.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.326+0000 7f9c77400640 1 -- 172.21.15.28:0/2535675360 learned_addr learned my addr 172.21.15.28:0/2535675360 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:32.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.326+0000 7f9c6ea00640 1 --2- 172.21.15.28:0/2535675360 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9c780c3800 0x7f9c780c3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:32.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.326+0000 7f9c77400640 1 -- 172.21.15.28:0/2535675360 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9c780c3800 msgr2=0x7f9c780c3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:32.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.326+0000 7f9c77400640 1 --2- 172.21.15.28:0/2535675360 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9c780c3800 0x7f9c780c3c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:32.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.326+0000 7f9c77400640 1 -- 172.21.15.28:0/2535675360 >> v1:172.21.15.81:6789/0 conn(0x7f9c781530e0 legacy=0x7f9c78153570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:32.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.326+0000 7f9c77400640 1 -- 172.21.15.28:0/2535675360 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c7806c8a0 con 0x7f9c78150500 2024-02-21T23:07:32.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.327+0000 7f9c77400640 1 --2- 172.21.15.28:0/2535675360 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c78150500 0x7f9c780a6d60 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f9c68009e50 tx=0x7f9c6802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be8ecb712764b3f7 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:32.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.328+0000 7f9c76a00640 1 -- 172.21.15.28:0/2535675360 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c6802ca80 con 0x7f9c78150500 2024-02-21T23:07:32.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.328+0000 7f9c76a00640 1 -- 172.21.15.28:0/2535675360 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c6802cc20 con 0x7f9c78150500 2024-02-21T23:07:32.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.328+0000 7f9c76a00640 1 -- 172.21.15.28:0/2535675360 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c680343f0 con 0x7f9c78150500 2024-02-21T23:07:32.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.328+0000 7f9c7d000640 1 -- 172.21.15.28:0/2535675360 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c78150500 msgr2=0x7f9c780a6d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:32.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.328+0000 7f9c7d000640 1 --2- 172.21.15.28:0/2535675360 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c78150500 0x7f9c780a6d60 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f9c68009e50 tx=0x7f9c6802bee0 comp rx=0 tx=0).stop 2024-02-21T23:07:32.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.328+0000 7f9c7d000640 1 -- 172.21.15.28:0/2535675360 shutdown_connections 2024-02-21T23:07:32.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.328+0000 7f9c7d000640 1 --2- 172.21.15.28:0/2535675360 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9c780c3800 0x7f9c780c3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:32.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.328+0000 7f9c7d000640 1 --2- 172.21.15.28:0/2535675360 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c78150500 0x7f9c780a6d60 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:32.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.328+0000 7f9c7d000640 1 -- 172.21.15.28:0/2535675360 >> 172.21.15.28:0/2535675360 conn(0x7f9c780a9690 msgr2=0x7f9c780a9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:32.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.329+0000 7f9c7d000640 1 -- 172.21.15.28:0/2535675360 shutdown_connections 2024-02-21T23:07:32.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.329+0000 7f9c7d000640 1 -- 172.21.15.28:0/2535675360 wait complete. 2024-02-21T23:07:32.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.331+0000 7f9c7d000640 1 Processor -- start 2024-02-21T23:07:32.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.331+0000 7f9c7d000640 1 -- start start 2024-02-21T23:07:32.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.333+0000 7f9c7d000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c780c3800 0x7f9c780c5b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:32.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.333+0000 7f9c7d000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c78150500 0x7f9c781eda10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:32.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.333+0000 7f9c7d000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9c781530e0 0x7f9c781edf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:32.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.333+0000 7f9c7d000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c780a72a0 con 0x7f9c780c3800 2024-02-21T23:07:32.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.333+0000 7f9c7d000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c7806ce60 con 0x7f9c78150500 2024-02-21T23:07:32.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.333+0000 7f9c7d000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c780c7470 con 0x7f9c781530e0 2024-02-21T23:07:32.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.333+0000 7f9c77400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c780c3800 0x7f9c780c5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:32.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.333+0000 7f9c77400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c780c3800 0x7f9c780c5b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59482/0 (socket says 172.21.15.28:59482) 2024-02-21T23:07:32.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.333+0000 7f9c77400640 1 -- 172.21.15.28:0/135610000 learned_addr learned my addr 172.21.15.28:0/135610000 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:32.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.334+0000 7f9c6ea00640 1 --2- 172.21.15.28:0/135610000 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c78150500 0x7f9c781eda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:32.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.334+0000 7f9c77400640 1 -- 172.21.15.28:0/135610000 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9c781530e0 msgr2=0x7f9c781edf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:32.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.334+0000 7f9c77e00640 1 --2- 172.21.15.28:0/135610000 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9c781530e0 0x7f9c781edf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:32.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.334+0000 7f9c77400640 1 --2- 172.21.15.28:0/135610000 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9c781530e0 0x7f9c781edf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:32.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.334+0000 7f9c77400640 1 -- 172.21.15.28:0/135610000 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c78150500 msgr2=0x7f9c781eda10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:32.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.334+0000 7f9c77400640 1 --2- 172.21.15.28:0/135610000 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c78150500 0x7f9c781eda10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:32.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.334+0000 7f9c77400640 1 -- 172.21.15.28:0/135610000 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c781ee490 con 0x7f9c780c3800 2024-02-21T23:07:32.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.334+0000 7f9c6ea00640 1 --2- 172.21.15.28:0/135610000 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c78150500 0x7f9c781eda10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:07:32.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.334+0000 7f9c77e00640 1 --2- 172.21.15.28:0/135610000 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9c781530e0 0x7f9c781edf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:07:32.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.334+0000 7f9c77400640 1 --2- 172.21.15.28:0/135610000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c780c3800 0x7f9c780c5b80 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f9c68036040 tx=0x7f9c68004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:32.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.335+0000 7f9c6fe00640 1 -- 172.21.15.28:0/135610000 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c68043020 con 0x7f9c780c3800 2024-02-21T23:07:32.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.335+0000 7f9c6fe00640 1 -- 172.21.15.28:0/135610000 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c6803e450 con 0x7f9c780c3800 2024-02-21T23:07:32.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.335+0000 7f9c6fe00640 1 -- 172.21.15.28:0/135610000 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c6803e840 con 0x7f9c780c3800 2024-02-21T23:07:32.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.335+0000 7f9c7d000640 1 -- 172.21.15.28:0/135610000 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c780ac680 con 0x7f9c780c3800 2024-02-21T23:07:32.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.335+0000 7f9c7d000640 1 -- 172.21.15.28:0/135610000 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c7814edf0 con 0x7f9c780c3800 2024-02-21T23:07:32.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.336+0000 7f9c7d000640 1 -- 172.21.15.28:0/135610000 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c7814f780 con 0x7f9c780c3800 2024-02-21T23:07:32.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.337+0000 7f9c6fe00640 1 -- 172.21.15.28:0/135610000 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9c68046020 con 0x7f9c780c3800 2024-02-21T23:07:32.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.338+0000 7f9c6fe00640 1 --2- 172.21.15.28:0/135610000 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9c5406b4d0 0x7f9c5406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:32.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.338+0000 7f9c6fe00640 1 -- 172.21.15.28:0/135610000 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f9c680acdf0 con 0x7f9c780c3800 2024-02-21T23:07:32.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.339+0000 7f9c6ea00640 1 --2- 172.21.15.28:0/135610000 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9c5406b4d0 0x7f9c5406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:32.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.340+0000 7f9c6ea00640 1 --2- 172.21.15.28:0/135610000 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9c5406b4d0 0x7f9c5406d990 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f9c780c3260 tx=0x7f9c60008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:32.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.347+0000 7f9c6fe00640 1 -- 172.21.15.28:0/135610000 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f9c6804d030 con 0x7f9c780c3800 2024-02-21T23:07:32.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:32.589+0000 7f9c7d000640 1 -- 172.21.15.28:0/135610000 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f9c7804f290 con 0x7f9c780c3800 2024-02-21T23:07:33.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.084+0000 7f9c6fe00640 1 -- 172.21.15.28:0/135610000 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v102) v1 ==== 122+0+0 (secure 0 0 0) 0x7f9c6807bad0 con 0x7f9c780c3800 2024-02-21T23:07:33.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.087+0000 7f9c7d000640 1 -- 172.21.15.28:0/135610000 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9c5406b4d0 msgr2=0x7f9c5406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:33.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.087+0000 7f9c7d000640 1 --2- 172.21.15.28:0/135610000 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9c5406b4d0 0x7f9c5406d990 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f9c780c3260 tx=0x7f9c60008040 comp rx=0 tx=0).stop 2024-02-21T23:07:33.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.087+0000 7f9c7d000640 1 -- 172.21.15.28:0/135610000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c780c3800 msgr2=0x7f9c780c5b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:33.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.087+0000 7f9c7d000640 1 --2- 172.21.15.28:0/135610000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c780c3800 0x7f9c780c5b80 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f9c68036040 tx=0x7f9c68004a40 comp rx=0 tx=0).stop 2024-02-21T23:07:33.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.087+0000 7f9c7d000640 1 -- 172.21.15.28:0/135610000 shutdown_connections 2024-02-21T23:07:33.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.088+0000 7f9c7d000640 1 --2- 172.21.15.28:0/135610000 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9c5406b4d0 0x7f9c5406d990 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:33.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.088+0000 7f9c7d000640 1 --2- 172.21.15.28:0/135610000 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9c781530e0 0x7f9c781edf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:33.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.088+0000 7f9c7d000640 1 --2- 172.21.15.28:0/135610000 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c78150500 0x7f9c781eda10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:33.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.088+0000 7f9c7d000640 1 --2- 172.21.15.28:0/135610000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c780c3800 0x7f9c780c5b80 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:33.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.088+0000 7f9c7d000640 1 -- 172.21.15.28:0/135610000 >> 172.21.15.28:0/135610000 conn(0x7f9c780a9690 msgr2=0x7f9c780c66f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:33.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.088+0000 7f9c7d000640 1 -- 172.21.15.28:0/135610000 shutdown_connections 2024-02-21T23:07:33.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.088+0000 7f9c7d000640 1 -- 172.21.15.28:0/135610000 wait complete. 2024-02-21T23:07:33.096 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-02-21T23:07:33.097 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:07:33.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.320+0000 7f915f200640 1 Processor -- start 2024-02-21T23:07:33.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.320+0000 7f915f200640 1 -- start start 2024-02-21T23:07:33.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.321+0000 7f915f200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f91580aff90 0x7f91580c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:33.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.321+0000 7f915f200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f915806dbf0 0x7f9158158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:33.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.321+0000 7f915f200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91580c74c0 con 0x7f91580c5720 2024-02-21T23:07:33.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.321+0000 7f915f200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91580c7270 con 0x7f91580aff90 2024-02-21T23:07:33.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.321+0000 7f915f200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9158158630 con 0x7f915806dbf0 2024-02-21T23:07:33.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.322+0000 7f915de00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f91580aff90 0x7f91580c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:33.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.322+0000 7f915de00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f91580aff90 0x7f91580c51e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59492/0 (socket says 172.21.15.28:59492) 2024-02-21T23:07:33.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.322+0000 7f915de00640 1 -- 172.21.15.28:0/401736911 learned_addr learned my addr 172.21.15.28:0/401736911 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:33.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.322+0000 7f915d400640 1 --2- 172.21.15.28:0/401736911 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f915806dbf0 0x7f9158158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:33.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.322+0000 7f915de00640 1 -- 172.21.15.28:0/401736911 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f915806dbf0 msgr2=0x7f9158158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:33.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.322+0000 7f915de00640 1 --2- 172.21.15.28:0/401736911 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f915806dbf0 0x7f9158158ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:33.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.322+0000 7f915de00640 1 -- 172.21.15.28:0/401736911 >> v1:172.21.15.81:6789/0 conn(0x7f91580c5720 legacy=0x7f915806c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:33.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.322+0000 7f915de00640 1 -- 172.21.15.28:0/401736911 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91580c77a0 con 0x7f91580aff90 2024-02-21T23:07:33.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.323+0000 7f915de00640 1 --2- 172.21.15.28:0/401736911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91580aff90 0x7f91580c51e0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f914c009870 tx=0x7f914c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82f871f87e256d38 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:33.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.324+0000 7f915ca00640 1 -- 172.21.15.28:0/401736911 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f914c035020 con 0x7f91580aff90 2024-02-21T23:07:33.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.324+0000 7f915ca00640 1 -- 172.21.15.28:0/401736911 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f914c031640 con 0x7f91580aff90 2024-02-21T23:07:33.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.324+0000 7f915ca00640 1 -- 172.21.15.28:0/401736911 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f914c02d040 con 0x7f91580aff90 2024-02-21T23:07:33.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.324+0000 7f915f200640 1 -- 172.21.15.28:0/401736911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91580aff90 msgr2=0x7f91580c51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:33.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.324+0000 7f915f200640 1 --2- 172.21.15.28:0/401736911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91580aff90 0x7f91580c51e0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f914c009870 tx=0x7f914c02b8f0 comp rx=0 tx=0).stop 2024-02-21T23:07:33.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.325+0000 7f915f200640 1 -- 172.21.15.28:0/401736911 shutdown_connections 2024-02-21T23:07:33.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.325+0000 7f915f200640 1 --2- 172.21.15.28:0/401736911 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f915806dbf0 0x7f9158158ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:33.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.325+0000 7f915f200640 1 --2- 172.21.15.28:0/401736911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91580aff90 0x7f91580c51e0 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:33.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.325+0000 7f915f200640 1 -- 172.21.15.28:0/401736911 >> 172.21.15.28:0/401736911 conn(0x7f91580ac370 msgr2=0x7f91580ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:33.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.325+0000 7f915f200640 1 -- 172.21.15.28:0/401736911 shutdown_connections 2024-02-21T23:07:33.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.325+0000 7f915f200640 1 -- 172.21.15.28:0/401736911 wait complete. 2024-02-21T23:07:33.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.327+0000 7f915f200640 1 Processor -- start 2024-02-21T23:07:33.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.327+0000 7f915f200640 1 -- start start 2024-02-21T23:07:33.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.329+0000 7f915f200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f915806dbf0 0x7f915814f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:33.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.329+0000 7f915f200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f91580aff90 0x7f9158151240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:33.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.329+0000 7f915f200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f91580c5720 0x7f91580c25b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:33.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.329+0000 7f915f200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91580c8760 con 0x7f915806dbf0 2024-02-21T23:07:33.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.329+0000 7f915f200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91580be0c0 con 0x7f91580c5720 2024-02-21T23:07:33.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.329+0000 7f915f200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91580c88e0 con 0x7f91580aff90 2024-02-21T23:07:33.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.329+0000 7f915de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f915806dbf0 0x7f915814f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:33.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.329+0000 7f915de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f915806dbf0 0x7f915814f210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59502/0 (socket says 172.21.15.28:59502) 2024-02-21T23:07:33.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.329+0000 7f915de00640 1 -- 172.21.15.28:0/587002829 learned_addr learned my addr 172.21.15.28:0/587002829 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:33.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.330+0000 7f915d400640 1 --2- 172.21.15.28:0/587002829 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f91580aff90 0x7f9158151240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:33.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.330+0000 7f915de00640 1 -- 172.21.15.28:0/587002829 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f91580aff90 msgr2=0x7f9158151240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:33.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.330+0000 7f915de00640 1 --2- 172.21.15.28:0/587002829 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f91580aff90 0x7f9158151240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:33.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.330+0000 7f915de00640 1 -- 172.21.15.28:0/587002829 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f91580c5720 msgr2=0x7f91580c25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:33.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.330+0000 7f915e800640 1 --2- 172.21.15.28:0/587002829 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f91580c5720 0x7f91580c25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:33.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.330+0000 7f915de00640 1 --2- 172.21.15.28:0/587002829 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f91580c5720 0x7f91580c25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:33.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.330+0000 7f915de00640 1 -- 172.21.15.28:0/587002829 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f915814f750 con 0x7f915806dbf0 2024-02-21T23:07:33.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.330+0000 7f915d400640 1 --2- 172.21.15.28:0/587002829 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f91580aff90 0x7f9158151240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:07:33.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.330+0000 7f915de00640 1 --2- 172.21.15.28:0/587002829 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f915806dbf0 0x7f915814f210 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f914c02b930 tx=0x7f914c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:33.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.330+0000 7f9152000640 1 -- 172.21.15.28:0/587002829 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f914c03aa60 con 0x7f915806dbf0 2024-02-21T23:07:33.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.330+0000 7f9152000640 1 -- 172.21.15.28:0/587002829 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f914c03ac00 con 0x7f915806dbf0 2024-02-21T23:07:33.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.330+0000 7f915f200640 1 -- 172.21.15.28:0/587002829 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f915814ef70 con 0x7f915806dbf0 2024-02-21T23:07:33.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.330+0000 7f9152000640 1 -- 172.21.15.28:0/587002829 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f914c0443f0 con 0x7f915806dbf0 2024-02-21T23:07:33.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.331+0000 7f915f200640 1 -- 172.21.15.28:0/587002829 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91580af2f0 con 0x7f915806dbf0 2024-02-21T23:07:33.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.332+0000 7f915f200640 1 -- 172.21.15.28:0/587002829 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f915804f290 con 0x7f915806dbf0 2024-02-21T23:07:33.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.333+0000 7f9152000640 1 -- 172.21.15.28:0/587002829 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f914c04b060 con 0x7f915806dbf0 2024-02-21T23:07:33.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.334+0000 7f9152000640 1 --2- 172.21.15.28:0/587002829 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f913806b4d0 0x7f913806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:33.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.334+0000 7f9152000640 1 -- 172.21.15.28:0/587002829 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f914c0ac030 con 0x7f915806dbf0 2024-02-21T23:07:33.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.335+0000 7f915d400640 1 --2- 172.21.15.28:0/587002829 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f913806b4d0 0x7f913806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:33.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.336+0000 7f915d400640 1 --2- 172.21.15.28:0/587002829 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f913806b4d0 0x7f913806d990 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f9158157450 tx=0x7f9144008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:33.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.339+0000 7f9152000640 1 -- 172.21.15.28:0/587002829 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f915814f750 con 0x7f915806dbf0 2024-02-21T23:07:33.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.604+0000 7f915f200640 1 -- 172.21.15.28:0/587002829 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f915806d9d0 con 0x7f915806dbf0 2024-02-21T23:07:33.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.604+0000 7f9152000640 1 -- 172.21.15.28:0/587002829 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+4942 (secure 0 0 0) 0x7f914c03f070 con 0x7f915806dbf0 2024-02-21T23:07:33.593 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:07:33.593 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":102,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5729,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1754860807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1754860807},{"type":"v1","addr":"172.21.15.81:6835","nonce":1754860807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3963422563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3963422563},{"type":"v1","addr":"172.21.15.81:6837","nonce":3963422563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7446,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/1119726035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1119726035},{"type":"v1","addr":"172.21.15.28:6833","nonce":1119726035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:07:29.533547+0000","modified":"2024-02-21T23:07:32.591216+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7458},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7458":{"gid":7458,"name":"a","rank":0,"incarnation":98,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/1259832430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":1259832430},{"type":"v1","addr":"172.21.15.28:6835","nonce":1259832430}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-02-21T23:07:33.593 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 102 2024-02-21T23:07:33.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.607+0000 7f915f200640 1 -- 172.21.15.28:0/587002829 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f913806b4d0 msgr2=0x7f913806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:33.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.607+0000 7f915f200640 1 --2- 172.21.15.28:0/587002829 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f913806b4d0 0x7f913806d990 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f9158157450 tx=0x7f9144008040 comp rx=0 tx=0).stop 2024-02-21T23:07:33.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.607+0000 7f915f200640 1 -- 172.21.15.28:0/587002829 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f915806dbf0 msgr2=0x7f915814f210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:33.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.607+0000 7f915f200640 1 --2- 172.21.15.28:0/587002829 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f915806dbf0 0x7f915814f210 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f914c02b930 tx=0x7f914c0046c0 comp rx=0 tx=0).stop 2024-02-21T23:07:33.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.607+0000 7f915f200640 1 -- 172.21.15.28:0/587002829 shutdown_connections 2024-02-21T23:07:33.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.607+0000 7f915f200640 1 --2- 172.21.15.28:0/587002829 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f913806b4d0 0x7f913806d990 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:33.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.607+0000 7f915f200640 1 --2- 172.21.15.28:0/587002829 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f91580aff90 0x7f9158151240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:33.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.607+0000 7f915f200640 1 --2- 172.21.15.28:0/587002829 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f91580c5720 0x7f91580c25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:33.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.607+0000 7f915f200640 1 --2- 172.21.15.28:0/587002829 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f915806dbf0 0x7f915814f210 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:33.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.607+0000 7f915f200640 1 -- 172.21.15.28:0/587002829 >> 172.21.15.28:0/587002829 conn(0x7f91580ac370 msgr2=0x7f91580c8ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:33.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.607+0000 7f915f200640 1 -- 172.21.15.28:0/587002829 shutdown_connections 2024-02-21T23:07:33.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.607+0000 7f915f200640 1 -- 172.21.15.28:0/587002829 wait complete. 2024-02-21T23:07:33.613 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:07:33.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.852+0000 7f927a400640 1 Processor -- start 2024-02-21T23:07:33.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.852+0000 7f927a400640 1 -- start start 2024-02-21T23:07:33.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.854+0000 7f927a400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f92740aff90 0x7f92740c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:33.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.854+0000 7f927a400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f927406dbf0 0x7f9274158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:33.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.854+0000 7f927a400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f92740c5720 0x7f9274159000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:33.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.854+0000 7f927a400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92740c7270 con 0x7f92740c5720 2024-02-21T23:07:33.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.854+0000 7f927a400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9274158630 con 0x7f927406dbf0 2024-02-21T23:07:33.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.854+0000 7f927a400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92740c74c0 con 0x7f92740aff90 2024-02-21T23:07:33.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.855+0000 7f9279a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f92740c5720 0x7f9274159000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:33.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.855+0000 7f9279a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f92740c5720 0x7f9274159000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59514/0 (socket says 172.21.15.28:59514) 2024-02-21T23:07:33.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.855+0000 7f9279a00640 1 -- 172.21.15.28:0/2800182182 learned_addr learned my addr 172.21.15.28:0/2800182182 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:33.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.855+0000 7f9279000640 1 --2- 172.21.15.28:0/2800182182 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f92740aff90 0x7f92740c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:33.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.855+0000 7f9273e00640 1 --2- 172.21.15.28:0/2800182182 >> v2:172.21.15.81:3300/0 conn(0x7f927406dbf0 0x7f9274158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:33.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.855+0000 7f9279a00640 1 -- 172.21.15.28:0/2800182182 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f92740aff90 msgr2=0x7f92740c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:33.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.855+0000 7f9279a00640 1 --2- 172.21.15.28:0/2800182182 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f92740aff90 0x7f92740c51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:33.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.855+0000 7f9279a00640 1 -- 172.21.15.28:0/2800182182 >> v2:172.21.15.81:3300/0 conn(0x7f927406dbf0 msgr2=0x7f9274158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:33.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.855+0000 7f9279a00640 1 --2- 172.21.15.28:0/2800182182 >> v2:172.21.15.81:3300/0 conn(0x7f927406dbf0 0x7f9274158ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:33.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.855+0000 7f9279a00640 1 -- 172.21.15.28:0/2800182182 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92740c77a0 con 0x7f92740c5720 2024-02-21T23:07:33.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.855+0000 7f9279000640 1 --2- 172.21.15.28:0/2800182182 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f92740aff90 0x7f92740c51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:07:33.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.856+0000 7f9279a00640 1 --2- 172.21.15.28:0/2800182182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92740c5720 0x7f9274159000 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f92740cb8f0 tx=0x7f926c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f17893be0078b8bf server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:33.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.857+0000 7f9273400640 1 -- 172.21.15.28:0/2800182182 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f926c02ca80 con 0x7f92740c5720 2024-02-21T23:07:33.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.857+0000 7f9273400640 1 -- 172.21.15.28:0/2800182182 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f926c02cc20 con 0x7f92740c5720 2024-02-21T23:07:33.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.857+0000 7f9273400640 1 -- 172.21.15.28:0/2800182182 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f926c0343f0 con 0x7f92740c5720 2024-02-21T23:07:33.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.857+0000 7f927a400640 1 -- 172.21.15.28:0/2800182182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92740c5720 msgr2=0x7f9274159000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:33.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.857+0000 7f927a400640 1 --2- 172.21.15.28:0/2800182182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92740c5720 0x7f9274159000 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f92740cb8f0 tx=0x7f926c02bee0 comp rx=0 tx=0).stop 2024-02-21T23:07:33.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.857+0000 7f927a400640 1 -- 172.21.15.28:0/2800182182 shutdown_connections 2024-02-21T23:07:33.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.857+0000 7f927a400640 1 --2- 172.21.15.28:0/2800182182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92740c5720 0x7f9274159000 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:33.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.857+0000 7f927a400640 1 --2- 172.21.15.28:0/2800182182 >> v2:172.21.15.81:3300/0 conn(0x7f927406dbf0 0x7f9274158ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:33.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.857+0000 7f927a400640 1 --2- 172.21.15.28:0/2800182182 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f92740aff90 0x7f92740c51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:33.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.857+0000 7f927a400640 1 -- 172.21.15.28:0/2800182182 >> 172.21.15.28:0/2800182182 conn(0x7f92740ac370 msgr2=0x7f92740ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:33.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.857+0000 7f927a400640 1 -- 172.21.15.28:0/2800182182 shutdown_connections 2024-02-21T23:07:33.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.857+0000 7f927a400640 1 -- 172.21.15.28:0/2800182182 wait complete. 2024-02-21T23:07:33.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.860+0000 7f927a400640 1 Processor -- start 2024-02-21T23:07:33.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.860+0000 7f927a400640 1 -- start start 2024-02-21T23:07:33.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.862+0000 7f927a400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f927406dbf0 0x7f92740ae740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:33.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.862+0000 7f927a400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f92740aff90 0x7f92740c3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:33.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.862+0000 7f927a400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92740c5720 0x7f92740c3c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:33.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.862+0000 7f927a400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92740c8760 con 0x7f92740c5720 2024-02-21T23:07:33.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.862+0000 7f927a400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92740be0c0 con 0x7f927406dbf0 2024-02-21T23:07:33.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.862+0000 7f927a400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92740c88e0 con 0x7f92740aff90 2024-02-21T23:07:33.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.862+0000 7f9279a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92740c5720 0x7f92740c3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:33.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.862+0000 7f9279a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92740c5720 0x7f92740c3c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59522/0 (socket says 172.21.15.28:59522) 2024-02-21T23:07:33.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.862+0000 7f9279a00640 1 -- 172.21.15.28:0/1075505638 learned_addr learned my addr 172.21.15.28:0/1075505638 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:33.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.862+0000 7f9273e00640 1 --2- 172.21.15.28:0/1075505638 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f92740aff90 0x7f92740c3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:33.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.862+0000 7f9279000640 1 --2- 172.21.15.28:0/1075505638 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f927406dbf0 0x7f92740ae740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:33.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.862+0000 7f9279a00640 1 -- 172.21.15.28:0/1075505638 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f92740aff90 msgr2=0x7f92740c3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:33.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.862+0000 7f9279a00640 1 --2- 172.21.15.28:0/1075505638 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f92740aff90 0x7f92740c3710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:33.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.863+0000 7f9279a00640 1 -- 172.21.15.28:0/1075505638 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f927406dbf0 msgr2=0x7f92740ae740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:33.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.863+0000 7f9279a00640 1 --2- 172.21.15.28:0/1075505638 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f927406dbf0 0x7f92740ae740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:33.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.863+0000 7f9279a00640 1 -- 172.21.15.28:0/1075505638 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92740c2220 con 0x7f92740c5720 2024-02-21T23:07:33.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.863+0000 7f9279000640 1 --2- 172.21.15.28:0/1075505638 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f927406dbf0 0x7f92740ae740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:07:33.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.863+0000 7f9279a00640 1 --2- 172.21.15.28:0/1075505638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92740c5720 0x7f92740c3c50 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f926c036040 tx=0x7f926c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:33.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.863+0000 7f9270c00640 1 -- 172.21.15.28:0/1075505638 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f926c044020 con 0x7f92740c5720 2024-02-21T23:07:33.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.863+0000 7f9270c00640 1 -- 172.21.15.28:0/1075505638 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f926c040690 con 0x7f92740c5720 2024-02-21T23:07:33.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.863+0000 7f927a400640 1 -- 172.21.15.28:0/1075505638 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92741eed10 con 0x7f92740c5720 2024-02-21T23:07:33.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.863+0000 7f9270c00640 1 -- 172.21.15.28:0/1075505638 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f926c040a80 con 0x7f92740c5720 2024-02-21T23:07:33.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.864+0000 7f927a400640 1 -- 172.21.15.28:0/1075505638 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9274154820 con 0x7f92740c5720 2024-02-21T23:07:33.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.865+0000 7f927a400640 1 -- 172.21.15.28:0/1075505638 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92741ea9f0 con 0x7f92740c5720 2024-02-21T23:07:33.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.865+0000 7f9270c00640 1 -- 172.21.15.28:0/1075505638 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f926c047020 con 0x7f92740c5720 2024-02-21T23:07:33.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.866+0000 7f9270c00640 1 --2- 172.21.15.28:0/1075505638 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f925406b4d0 0x7f925406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:33.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.866+0000 7f9270c00640 1 -- 172.21.15.28:0/1075505638 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f926c0acfb0 con 0x7f92740c5720 2024-02-21T23:07:33.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.867+0000 7f9279000640 1 --2- 172.21.15.28:0/1075505638 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f925406b4d0 0x7f925406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:33.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.868+0000 7f9279000640 1 --2- 172.21.15.28:0/1075505638 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f925406b4d0 0x7f925406d990 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f9268006fd0 tx=0x7f9268008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:33.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:33.871+0000 7f9270c00640 1 -- 172.21.15.28:0/1075505638 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f92741ea9f0 con 0x7f92740c5720 2024-02-21T23:07:34.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.151+0000 7f927a400640 1 -- 172.21.15.28:0/1075505638 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f927404f290 con 0x7f92740c5720 2024-02-21T23:07:34.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.153+0000 7f9270c00640 1 -- 172.21.15.28:0/1075505638 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v110) v1 ==== 74+0+16892 (secure 0 0 0) 0x7f926c04e030 con 0x7f92740c5720 2024-02-21T23:07:34.141 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:07:34.142 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":110,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:07:31.029084+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:07:27.477155+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-02-21T23:07:28.549456+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:07:34.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.156+0000 7f927a400640 1 -- 172.21.15.28:0/1075505638 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f925406b4d0 msgr2=0x7f925406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:34.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.156+0000 7f927a400640 1 --2- 172.21.15.28:0/1075505638 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f925406b4d0 0x7f925406d990 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f9268006fd0 tx=0x7f9268008040 comp rx=0 tx=0).stop 2024-02-21T23:07:34.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.156+0000 7f927a400640 1 -- 172.21.15.28:0/1075505638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92740c5720 msgr2=0x7f92740c3c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:34.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.156+0000 7f927a400640 1 --2- 172.21.15.28:0/1075505638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92740c5720 0x7f92740c3c50 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f926c036040 tx=0x7f926c004a40 comp rx=0 tx=0).stop 2024-02-21T23:07:34.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.156+0000 7f927a400640 1 -- 172.21.15.28:0/1075505638 shutdown_connections 2024-02-21T23:07:34.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.156+0000 7f927a400640 1 --2- 172.21.15.28:0/1075505638 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f925406b4d0 0x7f925406d990 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:34.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.156+0000 7f927a400640 1 --2- 172.21.15.28:0/1075505638 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f92740aff90 0x7f92740c3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:34.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.156+0000 7f927a400640 1 --2- 172.21.15.28:0/1075505638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92740c5720 0x7f92740c3c50 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:34.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.156+0000 7f927a400640 1 --2- 172.21.15.28:0/1075505638 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f927406dbf0 0x7f92740ae740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:34.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.156+0000 7f927a400640 1 -- 172.21.15.28:0/1075505638 >> 172.21.15.28:0/1075505638 conn(0x7f92740ac370 msgr2=0x7f92740ac750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:34.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.156+0000 7f927a400640 1 -- 172.21.15.28:0/1075505638 shutdown_connections 2024-02-21T23:07:34.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.157+0000 7f927a400640 1 -- 172.21.15.28:0/1075505638 wait complete. 2024-02-21T23:07:34.165 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-02-21T23:07:34.165 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-21T23:07:34.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.382+0000 7f3d3f200640 1 Processor -- start 2024-02-21T23:07:34.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.382+0000 7f3d3f200640 1 -- start start 2024-02-21T23:07:34.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.384+0000 7f3d3f200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3d38155960 0x7f3d381608a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:34.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.384+0000 7f3d3f200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d380c74c0 con 0x7f3d3806dbf0 2024-02-21T23:07:34.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.384+0000 7f3d3f200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d380c77a0 con 0x7f3d380aff90 2024-02-21T23:07:34.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.384+0000 7f3d3f200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d380c7270 con 0x7f3d38155960 2024-02-21T23:07:34.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.384+0000 7f3d3d400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f3d3806dbf0 0x7f3d38158ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:45898/0 (socket says 172.21.15.28:45898) 2024-02-21T23:07:34.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.384+0000 7f3d3d400640 1 -- 172.21.15.28:0/3884274157 learned_addr learned my addr 172.21.15.28:0/3884274157 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:07:34.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.385+0000 7f3d3e800640 1 --2- 172.21.15.28:0/3884274157 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3d38155960 0x7f3d381608a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:34.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.385+0000 7f3d3ca00640 1 -- 172.21.15.28:0/3884274157 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 130400164 0 0) 0x7f3d380c74c0 con 0x7f3d3806dbf0 2024-02-21T23:07:34.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.385+0000 7f3d3ca00640 1 -- 172.21.15.28:0/3884274157 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d20003520 con 0x7f3d3806dbf0 2024-02-21T23:07:34.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.385+0000 7f3d3ca00640 1 -- 172.21.15.28:0/3884274157 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 775816422 0 0) 0x7f3d20003520 con 0x7f3d3806dbf0 2024-02-21T23:07:34.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.385+0000 7f3d3ca00640 1 -- 172.21.15.28:0/3884274157 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3d38155960 msgr2=0x7f3d381608a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:34.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.385+0000 7f3d3ca00640 1 --2- 172.21.15.28:0/3884274157 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3d38155960 0x7f3d381608a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:34.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.385+0000 7f3d3ca00640 1 -- 172.21.15.28:0/3884274157 >> v1:172.21.15.81:6789/0 conn(0x7f3d380aff90 legacy=0x7f3d380c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:34.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.385+0000 7f3d3ca00640 1 -- 172.21.15.28:0/3884274157 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d380c7a80 con 0x7f3d3806dbf0 2024-02-21T23:07:34.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.385+0000 7f3d3ca00640 1 -- 172.21.15.28:0/3884274157 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f3d28001670 con 0x7f3d3806dbf0 2024-02-21T23:07:34.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.385+0000 7f3d3ca00640 1 -- 172.21.15.28:0/3884274157 >> v1:172.21.15.28:6789/0 conn(0x7f3d3806dbf0 legacy=0x7f3d38158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:34.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.385+0000 7f3d3ca00640 1 --2- 172.21.15.28:0/3884274157 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d200039e0 0x7f3d20003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:34.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.385+0000 7f3d3ca00640 1 --2- 172.21.15.28:0/3884274157 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3d38155960 0x7f3d20004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:34.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.385+0000 7f3d3ca00640 1 --2- 172.21.15.28:0/3884274157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d20004d40 0x7f3d200051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:34.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.385+0000 7f3d3ca00640 1 -- 172.21.15.28:0/3884274157 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d380c7270 con 0x7f3d20004d40 2024-02-21T23:07:34.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.385+0000 7f3d3ca00640 1 -- 172.21.15.28:0/3884274157 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d2000baa0 con 0x7f3d200039e0 2024-02-21T23:07:34.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.385+0000 7f3d3ca00640 1 -- 172.21.15.28:0/3884274157 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d2000bca0 con 0x7f3d38155960 2024-02-21T23:07:34.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.386+0000 7f3d3e800640 1 --2- 172.21.15.28:0/3884274157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d20004d40 0x7f3d200051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:34.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.386+0000 7f3d3de00640 1 --2- 172.21.15.28:0/3884274157 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d200039e0 0x7f3d20003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:34.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.386+0000 7f3d3d400640 1 --2- 172.21.15.28:0/3884274157 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3d38155960 0x7f3d20004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:34.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.386+0000 7f3d3e800640 1 -- 172.21.15.28:0/3884274157 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3d38155960 msgr2=0x7f3d20004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:34.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.386+0000 7f3d3e800640 1 --2- 172.21.15.28:0/3884274157 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3d38155960 0x7f3d20004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:34.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.386+0000 7f3d3e800640 1 -- 172.21.15.28:0/3884274157 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d200039e0 msgr2=0x7f3d20003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:34.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.386+0000 7f3d3e800640 1 --2- 172.21.15.28:0/3884274157 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d200039e0 0x7f3d20003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:34.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.386+0000 7f3d3e800640 1 -- 172.21.15.28:0/3884274157 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d20003520 con 0x7f3d20004d40 2024-02-21T23:07:34.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.387+0000 7f3d3e800640 1 --2- 172.21.15.28:0/3884274157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d20004d40 0x7f3d200051c0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f3d34009f90 tx=0x7f3d20005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a035b4204ab522c6 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:34.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.388+0000 7f3d3ca00640 1 -- 172.21.15.28:0/3884274157 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d34039070 con 0x7f3d20004d40 2024-02-21T23:07:34.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.388+0000 7f3d3ca00640 1 -- 172.21.15.28:0/3884274157 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d3402cbd0 con 0x7f3d20004d40 2024-02-21T23:07:34.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.388+0000 7f3d3ca00640 1 -- 172.21.15.28:0/3884274157 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d34009c10 con 0x7f3d20004d40 2024-02-21T23:07:34.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.388+0000 7f3d3f200640 1 -- 172.21.15.28:0/3884274157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d20004d40 msgr2=0x7f3d200051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:34.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.388+0000 7f3d3f200640 1 --2- 172.21.15.28:0/3884274157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d20004d40 0x7f3d200051c0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f3d34009f90 tx=0x7f3d20005830 comp rx=0 tx=0).stop 2024-02-21T23:07:34.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.389+0000 7f3d3e800640 1 -- 172.21.15.28:0/3884274157 reap_dead start 2024-02-21T23:07:34.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.389+0000 7f3d3f200640 1 -- 172.21.15.28:0/3884274157 shutdown_connections 2024-02-21T23:07:34.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.389+0000 7f3d3f200640 1 -- 172.21.15.28:0/3884274157 >> 172.21.15.28:0/3884274157 conn(0x7f3d380ac370 msgr2=0x7f3d380ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:34.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.389+0000 7f3d3f200640 1 -- 172.21.15.28:0/3884274157 shutdown_connections 2024-02-21T23:07:34.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.389+0000 7f3d3f200640 1 -- 172.21.15.28:0/3884274157 wait complete. 2024-02-21T23:07:34.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.390+0000 7f3d3f200640 1 Processor -- start 2024-02-21T23:07:34.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.391+0000 7f3d3f200640 1 -- start start 2024-02-21T23:07:34.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.391+0000 7f3d3f200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3d20004d40 0x7f3d380aef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:34.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.391+0000 7f3d3f200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d381ea9b0 0x7f3d381eae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:34.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.391+0000 7f3d3f200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d380c1930 0x7f3d380c1db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:34.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.391+0000 7f3d3f200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d380c58f0 con 0x7f3d381ea9b0 2024-02-21T23:07:34.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.391+0000 7f3d3f200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d38158630 con 0x7f3d380c1930 2024-02-21T23:07:34.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.391+0000 7f3d3f200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d380c5a70 con 0x7f3d20004d40 2024-02-21T23:07:34.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.392+0000 7f3d3d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d381ea9b0 0x7f3d381eae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:34.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.392+0000 7f3d3d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d381ea9b0 0x7f3d381eae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59532/0 (socket says 172.21.15.28:59532) 2024-02-21T23:07:34.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.392+0000 7f3d3d400640 1 -- 172.21.15.28:0/1775028847 learned_addr learned my addr 172.21.15.28:0/1775028847 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:34.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.392+0000 7f3d3de00640 1 --2- 172.21.15.28:0/1775028847 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3d20004d40 0x7f3d380aef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:34.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.392+0000 7f3d3e800640 1 --2- 172.21.15.28:0/1775028847 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d380c1930 0x7f3d380c1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:34.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.392+0000 7f3d3d400640 1 -- 172.21.15.28:0/1775028847 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3d20004d40 msgr2=0x7f3d380aef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:34.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.392+0000 7f3d3d400640 1 --2- 172.21.15.28:0/1775028847 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3d20004d40 0x7f3d380aef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:34.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.393+0000 7f3d3d400640 1 -- 172.21.15.28:0/1775028847 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d380c1930 msgr2=0x7f3d380c1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:34.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.393+0000 7f3d3d400640 1 --2- 172.21.15.28:0/1775028847 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d380c1930 0x7f3d380c1db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:34.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.393+0000 7f3d3d400640 1 -- 172.21.15.28:0/1775028847 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d381eee10 con 0x7f3d381ea9b0 2024-02-21T23:07:34.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.393+0000 7f3d3d400640 1 --2- 172.21.15.28:0/1775028847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d381ea9b0 0x7f3d381eae30 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f3d2800e860 tx=0x7f3d2800ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:34.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.393+0000 7f3d32000640 1 -- 172.21.15.28:0/1775028847 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d2800c8d0 con 0x7f3d381ea9b0 2024-02-21T23:07:34.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.393+0000 7f3d32000640 1 -- 172.21.15.28:0/1775028847 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d28018070 con 0x7f3d381ea9b0 2024-02-21T23:07:34.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.393+0000 7f3d32000640 1 -- 172.21.15.28:0/1775028847 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d2800ccb0 con 0x7f3d381ea9b0 2024-02-21T23:07:34.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.393+0000 7f3d3f200640 1 -- 172.21.15.28:0/1775028847 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d381ee050 con 0x7f3d381ea9b0 2024-02-21T23:07:34.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.394+0000 7f3d3f200640 1 -- 172.21.15.28:0/1775028847 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d381ee610 con 0x7f3d381ea9b0 2024-02-21T23:07:34.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.395+0000 7f3d3f200640 1 -- 172.21.15.28:0/1775028847 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d3804f290 con 0x7f3d381ea9b0 2024-02-21T23:07:34.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.396+0000 7f3d32000640 1 -- 172.21.15.28:0/1775028847 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3d28013420 con 0x7f3d381ea9b0 2024-02-21T23:07:34.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.397+0000 7f3d32000640 1 --2- 172.21.15.28:0/1775028847 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3d1c06b500 0x7f3d1c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:34.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.397+0000 7f3d32000640 1 -- 172.21.15.28:0/1775028847 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f3d28089ff0 con 0x7f3d381ea9b0 2024-02-21T23:07:34.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.397+0000 7f3d3de00640 1 --2- 172.21.15.28:0/1775028847 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3d1c06b500 0x7f3d1c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:34.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.398+0000 7f3d3de00640 1 --2- 172.21.15.28:0/1775028847 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3d1c06b500 0x7f3d1c06d9c0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c007fd0 tx=0x7f3d2c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:34.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.401+0000 7f3d32000640 1 -- 172.21.15.28:0/1775028847 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3d28058cd0 con 0x7f3d381ea9b0 2024-02-21T23:07:34.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.600+0000 7f3d3f200640 1 -- 172.21.15.28:0/1775028847 --> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3d3806d9d0 con 0x7f3d1c06b500 2024-02-21T23:07:34.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.618+0000 7f3d32000640 1 -- 172.21.15.28:0/1775028847 <== mgr.4101 v2:172.21.15.81:6832/1848538577 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252493 (secure 0 0 0) 0x7f3d3806d9d0 con 0x7f3d1c06b500 2024-02-21T23:07:34.609 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:07:34.611 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-21T23:07:34.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.623+0000 7f3d3f200640 1 -- 172.21.15.28:0/1775028847 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3d1c06b500 msgr2=0x7f3d1c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:34.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.623+0000 7f3d3f200640 1 --2- 172.21.15.28:0/1775028847 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3d1c06b500 0x7f3d1c06d9c0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c007fd0 tx=0x7f3d2c009040 comp rx=0 tx=0).stop 2024-02-21T23:07:34.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.623+0000 7f3d3f200640 1 -- 172.21.15.28:0/1775028847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d381ea9b0 msgr2=0x7f3d381eae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:34.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.623+0000 7f3d3f200640 1 --2- 172.21.15.28:0/1775028847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d381ea9b0 0x7f3d381eae30 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f3d2800e860 tx=0x7f3d2800ec60 comp rx=0 tx=0).stop 2024-02-21T23:07:34.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.623+0000 7f3d3f200640 1 -- 172.21.15.28:0/1775028847 shutdown_connections 2024-02-21T23:07:34.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.623+0000 7f3d3f200640 1 --2- 172.21.15.28:0/1775028847 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3d1c06b500 0x7f3d1c06d9c0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:34.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.623+0000 7f3d3f200640 1 --2- 172.21.15.28:0/1775028847 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d380c1930 0x7f3d380c1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:34.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.623+0000 7f3d3f200640 1 --2- 172.21.15.28:0/1775028847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d381ea9b0 0x7f3d381eae30 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:34.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.623+0000 7f3d3f200640 1 --2- 172.21.15.28:0/1775028847 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3d20004d40 0x7f3d380aef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:34.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.623+0000 7f3d3f200640 1 -- 172.21.15.28:0/1775028847 >> 172.21.15.28:0/1775028847 conn(0x7f3d380ac370 msgr2=0x7f3d38157e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:34.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.623+0000 7f3d3f200640 1 -- 172.21.15.28:0/1775028847 shutdown_connections 2024-02-21T23:07:34.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.623+0000 7f3d3f200640 1 -- 172.21.15.28:0/1775028847 wait complete. 2024-02-21T23:07:34.631 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":254,"stamp":"2024-02-21T23:07:32.769668+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":70,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":260272,"kb_used_data":10800,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754714448,"statfs":{"total":773094113280,"available":772827594752,"internally_reserved":0,"allocated":11059200,"data_stored":6477191,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":40,"apply_latency_ms":40,"commit_latency_ns":40000000,"apply_latency_ns":40000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.747303"},"pg_stats":[{"pgid":"18.3f","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.067550+0000","last_change":"2024-02-21T23:07:30.059591+0000","last_active":"2024-02-21T23:07:30.067550+0000","last_peered":"2024-02-21T23:07:30.067550+0000","last_clean":"2024-02-21T23:07:30.067550+0000","last_became_active":"2024-02-21T23:07:30.059262+0000","last_became_peered":"2024-02-21T23:07:30.059262+0000","last_unstale":"2024-02-21T23:07:30.067550+0000","last_undegraded":"2024-02-21T23:07:30.067550+0000","last_fullsized":"2024-02-21T23:07:30.067550+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:47:16.637877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3e","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.085757+0000","last_change":"2024-02-21T23:07:30.084488+0000","last_active":"2024-02-21T23:07:30.085757+0000","last_peered":"2024-02-21T23:07:30.085757+0000","last_clean":"2024-02-21T23:07:30.085757+0000","last_became_active":"2024-02-21T23:07:30.080786+0000","last_became_peered":"2024-02-21T23:07:30.080786+0000","last_unstale":"2024-02-21T23:07:30.085757+0000","last_undegraded":"2024-02-21T23:07:30.085757+0000","last_fullsized":"2024-02-21T23:07:30.085757+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:43:13.411504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3d","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.085469+0000","last_change":"2024-02-21T23:07:30.080111+0000","last_active":"2024-02-21T23:07:30.085469+0000","last_peered":"2024-02-21T23:07:30.085469+0000","last_clean":"2024-02-21T23:07:30.085469+0000","last_became_active":"2024-02-21T23:07:30.079687+0000","last_became_peered":"2024-02-21T23:07:30.079687+0000","last_unstale":"2024-02-21T23:07:30.085469+0000","last_undegraded":"2024-02-21T23:07:30.085469+0000","last_fullsized":"2024-02-21T23:07:30.085469+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:08:35.154317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3c","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.074198+0000","last_change":"2024-02-21T23:07:30.067543+0000","last_active":"2024-02-21T23:07:30.074198+0000","last_peered":"2024-02-21T23:07:30.074198+0000","last_clean":"2024-02-21T23:07:30.074198+0000","last_became_active":"2024-02-21T23:07:30.066543+0000","last_became_peered":"2024-02-21T23:07:30.066543+0000","last_unstale":"2024-02-21T23:07:30.074198+0000","last_undegraded":"2024-02-21T23:07:30.074198+0000","last_fullsized":"2024-02-21T23:07:30.074198+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:36:26.486506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3b","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.084559+0000","last_change":"2024-02-21T23:07:30.081619+0000","last_active":"2024-02-21T23:07:30.084559+0000","last_peered":"2024-02-21T23:07:30.084559+0000","last_clean":"2024-02-21T23:07:30.084559+0000","last_became_active":"2024-02-21T23:07:30.080253+0000","last_became_peered":"2024-02-21T23:07:30.080253+0000","last_unstale":"2024-02-21T23:07:30.084559+0000","last_undegraded":"2024-02-21T23:07:30.084559+0000","last_fullsized":"2024-02-21T23:07:30.084559+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:48:05.594176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3a","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.068878+0000","last_change":"2024-02-21T23:07:30.065698+0000","last_active":"2024-02-21T23:07:30.068878+0000","last_peered":"2024-02-21T23:07:30.068878+0000","last_clean":"2024-02-21T23:07:30.068878+0000","last_became_active":"2024-02-21T23:07:30.064508+0000","last_became_peered":"2024-02-21T23:07:30.064508+0000","last_unstale":"2024-02-21T23:07:30.068878+0000","last_undegraded":"2024-02-21T23:07:30.068878+0000","last_fullsized":"2024-02-21T23:07:30.068878+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:07:55.564030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.39","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.072380+0000","last_change":"2024-02-21T23:07:30.068392+0000","last_active":"2024-02-21T23:07:30.072380+0000","last_peered":"2024-02-21T23:07:30.072380+0000","last_clean":"2024-02-21T23:07:30.072380+0000","last_became_active":"2024-02-21T23:07:30.066886+0000","last_became_peered":"2024-02-21T23:07:30.066886+0000","last_unstale":"2024-02-21T23:07:30.072380+0000","last_undegraded":"2024-02-21T23:07:30.072380+0000","last_fullsized":"2024-02-21T23:07:30.072380+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:47:51.045182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.38","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.087796+0000","last_change":"2024-02-21T23:07:30.082159+0000","last_active":"2024-02-21T23:07:30.087796+0000","last_peered":"2024-02-21T23:07:30.087796+0000","last_clean":"2024-02-21T23:07:30.087796+0000","last_became_active":"2024-02-21T23:07:30.081900+0000","last_became_peered":"2024-02-21T23:07:30.081900+0000","last_unstale":"2024-02-21T23:07:30.087796+0000","last_undegraded":"2024-02-21T23:07:30.087796+0000","last_fullsized":"2024-02-21T23:07:30.087796+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:14:24.006664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.37","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.071663+0000","last_change":"2024-02-21T23:07:30.064760+0000","last_active":"2024-02-21T23:07:30.071663+0000","last_peered":"2024-02-21T23:07:30.071663+0000","last_clean":"2024-02-21T23:07:30.071663+0000","last_became_active":"2024-02-21T23:07:30.063783+0000","last_became_peered":"2024-02-21T23:07:30.063783+0000","last_unstale":"2024-02-21T23:07:30.071663+0000","last_undegraded":"2024-02-21T23:07:30.071663+0000","last_fullsized":"2024-02-21T23:07:30.071663+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:00:31.053859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.36","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.075253+0000","last_change":"2024-02-21T23:07:30.069825+0000","last_active":"2024-02-21T23:07:30.075253+0000","last_peered":"2024-02-21T23:07:30.075253+0000","last_clean":"2024-02-21T23:07:30.075253+0000","last_became_active":"2024-02-21T23:07:30.069253+0000","last_became_peered":"2024-02-21T23:07:30.069253+0000","last_unstale":"2024-02-21T23:07:30.075253+0000","last_undegraded":"2024-02-21T23:07:30.075253+0000","last_fullsized":"2024-02-21T23:07:30.075253+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:12:27.752661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.35","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.082924+0000","last_change":"2024-02-21T23:07:30.072818+0000","last_active":"2024-02-21T23:07:30.082924+0000","last_peered":"2024-02-21T23:07:30.082924+0000","last_clean":"2024-02-21T23:07:30.082924+0000","last_became_active":"2024-02-21T23:07:30.071816+0000","last_became_peered":"2024-02-21T23:07:30.071816+0000","last_unstale":"2024-02-21T23:07:30.082924+0000","last_undegraded":"2024-02-21T23:07:30.082924+0000","last_fullsized":"2024-02-21T23:07:30.082924+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:00:31.053859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.34","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.063599+0000","last_change":"2024-02-21T23:07:30.057927+0000","last_active":"2024-02-21T23:07:30.063599+0000","last_peered":"2024-02-21T23:07:30.063599+0000","last_clean":"2024-02-21T23:07:30.063599+0000","last_became_active":"2024-02-21T23:07:30.057359+0000","last_became_peered":"2024-02-21T23:07:30.057359+0000","last_unstale":"2024-02-21T23:07:30.063599+0000","last_undegraded":"2024-02-21T23:07:30.063599+0000","last_fullsized":"2024-02-21T23:07:30.063599+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:32:49.509733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.33","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.075556+0000","last_change":"2024-02-21T23:07:30.073057+0000","last_active":"2024-02-21T23:07:30.075556+0000","last_peered":"2024-02-21T23:07:30.075556+0000","last_clean":"2024-02-21T23:07:30.075556+0000","last_became_active":"2024-02-21T23:07:30.072773+0000","last_became_peered":"2024-02-21T23:07:30.072773+0000","last_unstale":"2024-02-21T23:07:30.075556+0000","last_undegraded":"2024-02-21T23:07:30.075556+0000","last_fullsized":"2024-02-21T23:07:30.075556+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:41:21.176597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.32","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.085471+0000","last_change":"2024-02-21T23:07:30.080500+0000","last_active":"2024-02-21T23:07:30.085471+0000","last_peered":"2024-02-21T23:07:30.085471+0000","last_clean":"2024-02-21T23:07:30.085471+0000","last_became_active":"2024-02-21T23:07:30.079450+0000","last_became_peered":"2024-02-21T23:07:30.079450+0000","last_unstale":"2024-02-21T23:07:30.085471+0000","last_undegraded":"2024-02-21T23:07:30.085471+0000","last_fullsized":"2024-02-21T23:07:30.085471+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:20:44.474084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.31","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.083458+0000","last_change":"2024-02-21T23:07:30.075424+0000","last_active":"2024-02-21T23:07:30.083458+0000","last_peered":"2024-02-21T23:07:30.083458+0000","last_clean":"2024-02-21T23:07:30.083458+0000","last_became_active":"2024-02-21T23:07:30.074499+0000","last_became_peered":"2024-02-21T23:07:30.074499+0000","last_unstale":"2024-02-21T23:07:30.083458+0000","last_undegraded":"2024-02-21T23:07:30.083458+0000","last_fullsized":"2024-02-21T23:07:30.083458+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:00:05.154952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.30","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.063474+0000","last_change":"2024-02-21T23:07:30.059165+0000","last_active":"2024-02-21T23:07:30.063474+0000","last_peered":"2024-02-21T23:07:30.063474+0000","last_clean":"2024-02-21T23:07:30.063474+0000","last_became_active":"2024-02-21T23:07:30.058591+0000","last_became_peered":"2024-02-21T23:07:30.058591+0000","last_unstale":"2024-02-21T23:07:30.063474+0000","last_undegraded":"2024-02-21T23:07:30.063474+0000","last_fullsized":"2024-02-21T23:07:30.063474+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:12:22.139208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.2f","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.075714+0000","last_change":"2024-02-21T23:07:30.071924+0000","last_active":"2024-02-21T23:07:30.075714+0000","last_peered":"2024-02-21T23:07:30.075714+0000","last_clean":"2024-02-21T23:07:30.075714+0000","last_became_active":"2024-02-21T23:07:30.071641+0000","last_became_peered":"2024-02-21T23:07:30.071641+0000","last_unstale":"2024-02-21T23:07:30.075714+0000","last_undegraded":"2024-02-21T23:07:30.075714+0000","last_fullsized":"2024-02-21T23:07:30.075714+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:33:50.871582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.2e","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.085697+0000","last_change":"2024-02-21T23:07:30.081049+0000","last_active":"2024-02-21T23:07:30.085697+0000","last_peered":"2024-02-21T23:07:30.085697+0000","last_clean":"2024-02-21T23:07:30.085697+0000","last_became_active":"2024-02-21T23:07:30.080402+0000","last_became_peered":"2024-02-21T23:07:30.080402+0000","last_unstale":"2024-02-21T23:07:30.085697+0000","last_undegraded":"2024-02-21T23:07:30.085697+0000","last_fullsized":"2024-02-21T23:07:30.085697+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:19:35.573219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.2d","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.085625+0000","last_change":"2024-02-21T23:07:30.079530+0000","last_active":"2024-02-21T23:07:30.085625+0000","last_peered":"2024-02-21T23:07:30.085625+0000","last_clean":"2024-02-21T23:07:30.085625+0000","last_became_active":"2024-02-21T23:07:30.078744+0000","last_became_peered":"2024-02-21T23:07:30.078744+0000","last_unstale":"2024-02-21T23:07:30.085625+0000","last_undegraded":"2024-02-21T23:07:30.085625+0000","last_fullsized":"2024-02-21T23:07:30.085625+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:47:16.637877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.2c","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.074197+0000","last_change":"2024-02-21T23:07:30.066512+0000","last_active":"2024-02-21T23:07:30.074197+0000","last_peered":"2024-02-21T23:07:30.074197+0000","last_clean":"2024-02-21T23:07:30.074197+0000","last_became_active":"2024-02-21T23:07:30.065427+0000","last_became_peered":"2024-02-21T23:07:30.065427+0000","last_unstale":"2024-02-21T23:07:30.074197+0000","last_undegraded":"2024-02-21T23:07:30.074197+0000","last_fullsized":"2024-02-21T23:07:30.074197+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:37:41.104430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.2b","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.073406+0000","last_change":"2024-02-21T23:07:30.068288+0000","last_active":"2024-02-21T23:07:30.073406+0000","last_peered":"2024-02-21T23:07:30.073406+0000","last_clean":"2024-02-21T23:07:30.073406+0000","last_became_active":"2024-02-21T23:07:30.067970+0000","last_became_peered":"2024-02-21T23:07:30.067970+0000","last_unstale":"2024-02-21T23:07:30.073406+0000","last_undegraded":"2024-02-21T23:07:30.073406+0000","last_fullsized":"2024-02-21T23:07:30.073406+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:08:18.112813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.2a","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.072986+0000","last_change":"2024-02-21T23:07:30.064417+0000","last_active":"2024-02-21T23:07:30.072986+0000","last_peered":"2024-02-21T23:07:30.072986+0000","last_clean":"2024-02-21T23:07:30.072986+0000","last_became_active":"2024-02-21T23:07:30.064128+0000","last_became_peered":"2024-02-21T23:07:30.064128+0000","last_unstale":"2024-02-21T23:07:30.072986+0000","last_undegraded":"2024-02-21T23:07:30.072986+0000","last_fullsized":"2024-02-21T23:07:30.072986+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:32:40.636665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.29","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.074715+0000","last_change":"2024-02-21T23:07:30.072358+0000","last_active":"2024-02-21T23:07:30.074715+0000","last_peered":"2024-02-21T23:07:30.074715+0000","last_clean":"2024-02-21T23:07:30.074715+0000","last_became_active":"2024-02-21T23:07:30.069662+0000","last_became_peered":"2024-02-21T23:07:30.069662+0000","last_unstale":"2024-02-21T23:07:30.074715+0000","last_undegraded":"2024-02-21T23:07:30.074715+0000","last_fullsized":"2024-02-21T23:07:30.074715+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:29:03.312549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.28","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.087628+0000","last_change":"2024-02-21T23:07:30.082380+0000","last_active":"2024-02-21T23:07:30.087628+0000","last_peered":"2024-02-21T23:07:30.087628+0000","last_clean":"2024-02-21T23:07:30.087628+0000","last_became_active":"2024-02-21T23:07:30.080196+0000","last_became_peered":"2024-02-21T23:07:30.080196+0000","last_unstale":"2024-02-21T23:07:30.087628+0000","last_undegraded":"2024-02-21T23:07:30.087628+0000","last_fullsized":"2024-02-21T23:07:30.087628+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:29:03.312549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.27","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.087443+0000","last_change":"2024-02-21T23:07:30.077414+0000","last_active":"2024-02-21T23:07:30.087443+0000","last_peered":"2024-02-21T23:07:30.087443+0000","last_clean":"2024-02-21T23:07:30.087443+0000","last_became_active":"2024-02-21T23:07:30.074579+0000","last_became_peered":"2024-02-21T23:07:30.074579+0000","last_unstale":"2024-02-21T23:07:30.087443+0000","last_undegraded":"2024-02-21T23:07:30.087443+0000","last_fullsized":"2024-02-21T23:07:30.087443+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:00:05.154952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.26","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.075294+0000","last_change":"2024-02-21T23:07:30.060596+0000","last_active":"2024-02-21T23:07:30.075294+0000","last_peered":"2024-02-21T23:07:30.075294+0000","last_clean":"2024-02-21T23:07:30.075294+0000","last_became_active":"2024-02-21T23:07:30.060206+0000","last_became_peered":"2024-02-21T23:07:30.060206+0000","last_unstale":"2024-02-21T23:07:30.075294+0000","last_undegraded":"2024-02-21T23:07:30.075294+0000","last_fullsized":"2024-02-21T23:07:30.075294+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:01:27.333779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.25","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.085576+0000","last_change":"2024-02-21T23:07:30.078812+0000","last_active":"2024-02-21T23:07:30.085576+0000","last_peered":"2024-02-21T23:07:30.085576+0000","last_clean":"2024-02-21T23:07:30.085576+0000","last_became_active":"2024-02-21T23:07:30.078514+0000","last_became_peered":"2024-02-21T23:07:30.078514+0000","last_unstale":"2024-02-21T23:07:30.085576+0000","last_undegraded":"2024-02-21T23:07:30.085576+0000","last_fullsized":"2024-02-21T23:07:30.085576+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:28:57.130238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.24","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.076157+0000","last_change":"2024-02-21T23:07:30.070416+0000","last_active":"2024-02-21T23:07:30.076157+0000","last_peered":"2024-02-21T23:07:30.076157+0000","last_clean":"2024-02-21T23:07:30.076157+0000","last_became_active":"2024-02-21T23:07:30.070100+0000","last_became_peered":"2024-02-21T23:07:30.070100+0000","last_unstale":"2024-02-21T23:07:30.076157+0000","last_undegraded":"2024-02-21T23:07:30.076157+0000","last_fullsized":"2024-02-21T23:07:30.076157+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:46:25.120139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.23","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.072817+0000","last_change":"2024-02-21T23:07:30.064187+0000","last_active":"2024-02-21T23:07:30.072817+0000","last_peered":"2024-02-21T23:07:30.072817+0000","last_clean":"2024-02-21T23:07:30.072817+0000","last_became_active":"2024-02-21T23:07:30.063867+0000","last_became_peered":"2024-02-21T23:07:30.063867+0000","last_unstale":"2024-02-21T23:07:30.072817+0000","last_undegraded":"2024-02-21T23:07:30.072817+0000","last_fullsized":"2024-02-21T23:07:30.072817+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:19:45.514897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.22","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.062646+0000","last_change":"2024-02-21T23:07:30.057638+0000","last_active":"2024-02-21T23:07:30.062646+0000","last_peered":"2024-02-21T23:07:30.062646+0000","last_clean":"2024-02-21T23:07:30.062646+0000","last_became_active":"2024-02-21T23:07:30.057299+0000","last_became_peered":"2024-02-21T23:07:30.057299+0000","last_unstale":"2024-02-21T23:07:30.062646+0000","last_undegraded":"2024-02-21T23:07:30.062646+0000","last_fullsized":"2024-02-21T23:07:30.062646+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:33:48.672687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.21","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.087397+0000","last_change":"2024-02-21T23:07:30.076868+0000","last_active":"2024-02-21T23:07:30.087397+0000","last_peered":"2024-02-21T23:07:30.087397+0000","last_clean":"2024-02-21T23:07:30.087397+0000","last_became_active":"2024-02-21T23:07:30.076599+0000","last_became_peered":"2024-02-21T23:07:30.076599+0000","last_unstale":"2024-02-21T23:07:30.087397+0000","last_undegraded":"2024-02-21T23:07:30.087397+0000","last_fullsized":"2024-02-21T23:07:30.087397+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:00:31.053859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.20","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.087607+0000","last_change":"2024-02-21T23:07:30.082200+0000","last_active":"2024-02-21T23:07:30.087607+0000","last_peered":"2024-02-21T23:07:30.087607+0000","last_clean":"2024-02-21T23:07:30.087607+0000","last_became_active":"2024-02-21T23:07:30.079831+0000","last_became_peered":"2024-02-21T23:07:30.079831+0000","last_unstale":"2024-02-21T23:07:30.087607+0000","last_undegraded":"2024-02-21T23:07:30.087607+0000","last_fullsized":"2024-02-21T23:07:30.087607+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:32:44.283700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.075736+0000","last_change":"2024-02-21T23:07:30.067693+0000","last_active":"2024-02-21T23:07:30.075736+0000","last_peered":"2024-02-21T23:07:30.075736+0000","last_clean":"2024-02-21T23:07:30.075736+0000","last_became_active":"2024-02-21T23:07:30.067386+0000","last_became_peered":"2024-02-21T23:07:30.067386+0000","last_unstale":"2024-02-21T23:07:30.075736+0000","last_undegraded":"2024-02-21T23:07:30.075736+0000","last_fullsized":"2024-02-21T23:07:30.075736+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:29:57.959277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1f","version":"109'4","reported_seq":16,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.088644+0000","last_change":"2024-02-21T23:07:29.078080+0000","last_active":"2024-02-21T23:07:30.088644+0000","last_peered":"2024-02-21T23:07:30.088644+0000","last_clean":"2024-02-21T23:07:30.088644+0000","last_became_active":"2024-02-21T23:07:29.077684+0000","last_became_peered":"2024-02-21T23:07:29.077684+0000","last_unstale":"2024-02-21T23:07:30.088644+0000","last_undegraded":"2024-02-21T23:07:30.088644+0000","last_fullsized":"2024-02-21T23:07:30.088644+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_clean_scrub_stamp":"2024-02-21T23:07:28.006668+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:41:20.161387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.063001+0000","last_change":"2024-02-21T23:07:30.054594+0000","last_active":"2024-02-21T23:07:30.063001+0000","last_peered":"2024-02-21T23:07:30.063001+0000","last_clean":"2024-02-21T23:07:30.063001+0000","last_became_active":"2024-02-21T23:07:30.054236+0000","last_became_peered":"2024-02-21T23:07:30.054236+0000","last_unstale":"2024-02-21T23:07:30.063001+0000","last_undegraded":"2024-02-21T23:07:30.063001+0000","last_fullsized":"2024-02-21T23:07:30.063001+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:43:45.497542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1e","version":"109'1","reported_seq":13,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.081728+0000","last_change":"2024-02-21T23:07:29.066858+0000","last_active":"2024-02-21T23:07:30.081728+0000","last_peered":"2024-02-21T23:07:30.081728+0000","last_clean":"2024-02-21T23:07:30.081728+0000","last_became_active":"2024-02-21T23:07:29.066573+0000","last_became_peered":"2024-02-21T23:07:29.066573+0000","last_unstale":"2024-02-21T23:07:30.081728+0000","last_undegraded":"2024-02-21T23:07:30.081728+0000","last_fullsized":"2024-02-21T23:07:30.081728+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_clean_scrub_stamp":"2024-02-21T23:07:28.006668+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:55:03.572716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.073752+0000","last_change":"2024-02-21T23:07:30.069818+0000","last_active":"2024-02-21T23:07:30.073752+0000","last_peered":"2024-02-21T23:07:30.073752+0000","last_clean":"2024-02-21T23:07:30.073752+0000","last_became_active":"2024-02-21T23:07:30.069518+0000","last_became_peered":"2024-02-21T23:07:30.069518+0000","last_unstale":"2024-02-21T23:07:30.073752+0000","last_undegraded":"2024-02-21T23:07:30.073752+0000","last_fullsized":"2024-02-21T23:07:30.073752+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:14:32.130423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"109'2","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.083205+0000","last_change":"2024-02-21T23:07:29.075790+0000","last_active":"2024-02-21T23:07:30.083205+0000","last_peered":"2024-02-21T23:07:30.083205+0000","last_clean":"2024-02-21T23:07:30.083205+0000","last_became_active":"2024-02-21T23:07:29.074272+0000","last_became_peered":"2024-02-21T23:07:29.074272+0000","last_unstale":"2024-02-21T23:07:30.083205+0000","last_undegraded":"2024-02-21T23:07:30.083205+0000","last_fullsized":"2024-02-21T23:07:30.083205+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_clean_scrub_stamp":"2024-02-21T23:07:28.006668+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:43:51.259956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.071688+0000","last_change":"2024-02-21T23:07:30.065056+0000","last_active":"2024-02-21T23:07:30.071688+0000","last_peered":"2024-02-21T23:07:30.071688+0000","last_clean":"2024-02-21T23:07:30.071688+0000","last_became_active":"2024-02-21T23:07:30.064694+0000","last_became_peered":"2024-02-21T23:07:30.064694+0000","last_unstale":"2024-02-21T23:07:30.071688+0000","last_undegraded":"2024-02-21T23:07:30.071688+0000","last_fullsized":"2024-02-21T23:07:30.071688+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:00:05.154952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"109'2","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.085807+0000","last_change":"2024-02-21T23:07:29.059998+0000","last_active":"2024-02-21T23:07:30.085807+0000","last_peered":"2024-02-21T23:07:30.085807+0000","last_clean":"2024-02-21T23:07:30.085807+0000","last_became_active":"2024-02-21T23:07:29.059719+0000","last_became_peered":"2024-02-21T23:07:29.059719+0000","last_unstale":"2024-02-21T23:07:30.085807+0000","last_undegraded":"2024-02-21T23:07:30.085807+0000","last_fullsized":"2024-02-21T23:07:30.085807+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_clean_scrub_stamp":"2024-02-21T23:07:28.006668+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:45:53.025498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.075798+0000","last_change":"2024-02-21T23:07:30.073742+0000","last_active":"2024-02-21T23:07:30.075798+0000","last_peered":"2024-02-21T23:07:30.075798+0000","last_clean":"2024-02-21T23:07:30.075798+0000","last_became_active":"2024-02-21T23:07:30.073456+0000","last_became_peered":"2024-02-21T23:07:30.073456+0000","last_unstale":"2024-02-21T23:07:30.075798+0000","last_undegraded":"2024-02-21T23:07:30.075798+0000","last_fullsized":"2024-02-21T23:07:30.075798+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:43:52.275165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1b","version":"109'1","reported_seq":13,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.084429+0000","last_change":"2024-02-21T23:07:29.064738+0000","last_active":"2024-02-21T23:07:30.084429+0000","last_peered":"2024-02-21T23:07:30.084429+0000","last_clean":"2024-02-21T23:07:30.084429+0000","last_became_active":"2024-02-21T23:07:29.064453+0000","last_became_peered":"2024-02-21T23:07:29.064453+0000","last_unstale":"2024-02-21T23:07:30.084429+0000","last_undegraded":"2024-02-21T23:07:30.084429+0000","last_fullsized":"2024-02-21T23:07:30.084429+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_clean_scrub_stamp":"2024-02-21T23:07:28.006668+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:43:51.259956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.080744+0000","last_change":"2024-02-21T23:07:30.068959+0000","last_active":"2024-02-21T23:07:30.080744+0000","last_peered":"2024-02-21T23:07:30.080744+0000","last_clean":"2024-02-21T23:07:30.080744+0000","last_became_active":"2024-02-21T23:07:30.068604+0000","last_became_peered":"2024-02-21T23:07:30.068604+0000","last_unstale":"2024-02-21T23:07:30.080744+0000","last_undegraded":"2024-02-21T23:07:30.080744+0000","last_fullsized":"2024-02-21T23:07:30.080744+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:14:32.130423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"109'1","reported_seq":13,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.086251+0000","last_change":"2024-02-21T23:07:29.069504+0000","last_active":"2024-02-21T23:07:30.086251+0000","last_peered":"2024-02-21T23:07:30.086251+0000","last_clean":"2024-02-21T23:07:30.086251+0000","last_became_active":"2024-02-21T23:07:29.069197+0000","last_became_peered":"2024-02-21T23:07:29.069197+0000","last_unstale":"2024-02-21T23:07:30.086251+0000","last_undegraded":"2024-02-21T23:07:30.086251+0000","last_fullsized":"2024-02-21T23:07:30.086251+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_clean_scrub_stamp":"2024-02-21T23:07:28.006668+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:56:55.710409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.077191+0000","last_change":"2024-02-21T23:07:30.063377+0000","last_active":"2024-02-21T23:07:30.077191+0000","last_peered":"2024-02-21T23:07:30.077191+0000","last_clean":"2024-02-21T23:07:30.077191+0000","last_became_active":"2024-02-21T23:07:30.062671+0000","last_became_peered":"2024-02-21T23:07:30.062671+0000","last_unstale":"2024-02-21T23:07:30.077191+0000","last_undegraded":"2024-02-21T23:07:30.077191+0000","last_fullsized":"2024-02-21T23:07:30.077191+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:14:56.427299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.063169+0000","last_change":"2024-02-21T23:07:29.067526+0000","last_active":"2024-02-21T23:07:30.063169+0000","last_peered":"2024-02-21T23:07:30.063169+0000","last_clean":"2024-02-21T23:07:30.063169+0000","last_became_active":"2024-02-21T23:07:29.067188+0000","last_became_peered":"2024-02-21T23:07:29.067188+0000","last_unstale":"2024-02-21T23:07:30.063169+0000","last_undegraded":"2024-02-21T23:07:30.063169+0000","last_fullsized":"2024-02-21T23:07:30.063169+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_clean_scrub_stamp":"2024-02-21T23:07:28.006668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:43:51.259956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.066747+0000","last_change":"2024-02-21T23:07:30.065429+0000","last_active":"2024-02-21T23:07:30.066747+0000","last_peered":"2024-02-21T23:07:30.066747+0000","last_clean":"2024-02-21T23:07:30.066747+0000","last_became_active":"2024-02-21T23:07:30.064289+0000","last_became_peered":"2024-02-21T23:07:30.064289+0000","last_unstale":"2024-02-21T23:07:30.066747+0000","last_undegraded":"2024-02-21T23:07:30.066747+0000","last_fullsized":"2024-02-21T23:07:30.066747+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:21:34.378582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.037128+0000","last_change":"2024-02-21T23:07:29.066688+0000","last_active":"2024-02-21T23:07:30.037128+0000","last_peered":"2024-02-21T23:07:30.037128+0000","last_clean":"2024-02-21T23:07:30.037128+0000","last_became_active":"2024-02-21T23:07:29.065319+0000","last_became_peered":"2024-02-21T23:07:29.065319+0000","last_unstale":"2024-02-21T23:07:30.037128+0000","last_undegraded":"2024-02-21T23:07:30.037128+0000","last_fullsized":"2024-02-21T23:07:30.037128+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_clean_scrub_stamp":"2024-02-21T23:07:28.006668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:49:57.742371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.083462+0000","last_change":"2024-02-21T23:07:30.075112+0000","last_active":"2024-02-21T23:07:30.083462+0000","last_peered":"2024-02-21T23:07:30.083462+0000","last_clean":"2024-02-21T23:07:30.083462+0000","last_became_active":"2024-02-21T23:07:30.074791+0000","last_became_peered":"2024-02-21T23:07:30.074791+0000","last_unstale":"2024-02-21T23:07:30.083462+0000","last_undegraded":"2024-02-21T23:07:30.083462+0000","last_fullsized":"2024-02-21T23:07:30.083462+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:28:34.875509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.071071+0000","last_change":"2024-02-21T23:07:29.072136+0000","last_active":"2024-02-21T23:07:30.071071+0000","last_peered":"2024-02-21T23:07:30.071071+0000","last_clean":"2024-02-21T23:07:30.071071+0000","last_became_active":"2024-02-21T23:07:29.071591+0000","last_became_peered":"2024-02-21T23:07:29.071591+0000","last_unstale":"2024-02-21T23:07:30.071071+0000","last_undegraded":"2024-02-21T23:07:30.071071+0000","last_fullsized":"2024-02-21T23:07:30.071071+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_clean_scrub_stamp":"2024-02-21T23:07:28.006668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:43:51.259956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.073691+0000","last_change":"2024-02-21T23:07:30.068804+0000","last_active":"2024-02-21T23:07:30.073691+0000","last_peered":"2024-02-21T23:07:30.073691+0000","last_clean":"2024-02-21T23:07:30.073691+0000","last_became_active":"2024-02-21T23:07:30.068459+0000","last_became_peered":"2024-02-21T23:07:30.068459+0000","last_unstale":"2024-02-21T23:07:30.073691+0000","last_undegraded":"2024-02-21T23:07:30.073691+0000","last_fullsized":"2024-02-21T23:07:30.073691+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:32:40.636665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.065341+0000","last_change":"2024-02-21T23:07:29.071010+0000","last_active":"2024-02-21T23:07:30.065341+0000","last_peered":"2024-02-21T23:07:30.065341+0000","last_clean":"2024-02-21T23:07:30.065341+0000","last_became_active":"2024-02-21T23:07:29.070516+0000","last_became_peered":"2024-02-21T23:07:29.070516+0000","last_unstale":"2024-02-21T23:07:30.065341+0000","last_undegraded":"2024-02-21T23:07:30.065341+0000","last_fullsized":"2024-02-21T23:07:30.065341+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_clean_scrub_stamp":"2024-02-21T23:07:28.006668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:28:33.860300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.085321+0000","last_change":"2024-02-21T23:07:30.074543+0000","last_active":"2024-02-21T23:07:30.085321+0000","last_peered":"2024-02-21T23:07:30.085321+0000","last_clean":"2024-02-21T23:07:30.085321+0000","last_became_active":"2024-02-21T23:07:30.074227+0000","last_became_peered":"2024-02-21T23:07:30.074227+0000","last_unstale":"2024-02-21T23:07:30.085321+0000","last_undegraded":"2024-02-21T23:07:30.085321+0000","last_fullsized":"2024-02-21T23:07:30.085321+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:21:53.631128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.15","version":"109'3","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.088810+0000","last_change":"2024-02-21T23:07:29.073372+0000","last_active":"2024-02-21T23:07:30.088810+0000","last_peered":"2024-02-21T23:07:30.088810+0000","last_clean":"2024-02-21T23:07:30.088810+0000","last_became_active":"2024-02-21T23:07:29.073050+0000","last_became_peered":"2024-02-21T23:07:29.073050+0000","last_unstale":"2024-02-21T23:07:30.088810+0000","last_undegraded":"2024-02-21T23:07:30.088810+0000","last_fullsized":"2024-02-21T23:07:30.088810+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_clean_scrub_stamp":"2024-02-21T23:07:28.006668+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:28:56.115029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.087671+0000","last_change":"2024-02-21T23:07:30.078266+0000","last_active":"2024-02-21T23:07:30.087671+0000","last_peered":"2024-02-21T23:07:30.087671+0000","last_clean":"2024-02-21T23:07:30.087671+0000","last_became_active":"2024-02-21T23:07:30.077743+0000","last_became_peered":"2024-02-21T23:07:30.077743+0000","last_unstale":"2024-02-21T23:07:30.087671+0000","last_undegraded":"2024-02-21T23:07:30.087671+0000","last_fullsized":"2024-02-21T23:07:30.087671+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:08:18.112813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.14","version":"109'6","reported_seq":18,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.095228+0000","last_change":"2024-02-21T23:07:29.071931+0000","last_active":"2024-02-21T23:07:30.095228+0000","last_peered":"2024-02-21T23:07:30.095228+0000","last_clean":"2024-02-21T23:07:30.095228+0000","last_became_active":"2024-02-21T23:07:29.070378+0000","last_became_peered":"2024-02-21T23:07:29.070378+0000","last_unstale":"2024-02-21T23:07:30.095228+0000","last_undegraded":"2024-02-21T23:07:30.095228+0000","last_fullsized":"2024-02-21T23:07:30.095228+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_clean_scrub_stamp":"2024-02-21T23:07:28.006668+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:32:43.268491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.088454+0000","last_change":"2024-02-21T23:07:30.083594+0000","last_active":"2024-02-21T23:07:30.088454+0000","last_peered":"2024-02-21T23:07:30.088454+0000","last_clean":"2024-02-21T23:07:30.088454+0000","last_became_active":"2024-02-21T23:07:30.082142+0000","last_became_peered":"2024-02-21T23:07:30.082142+0000","last_unstale":"2024-02-21T23:07:30.088454+0000","last_undegraded":"2024-02-21T23:07:30.088454+0000","last_fullsized":"2024-02-21T23:07:30.088454+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:32:40.636665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.13","version":"109'2","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.093360+0000","last_change":"2024-02-21T23:07:29.069228+0000","last_active":"2024-02-21T23:07:30.093360+0000","last_peered":"2024-02-21T23:07:30.093360+0000","last_clean":"2024-02-21T23:07:30.093360+0000","last_became_active":"2024-02-21T23:07:29.068925+0000","last_became_peered":"2024-02-21T23:07:29.068925+0000","last_unstale":"2024-02-21T23:07:30.093360+0000","last_undegraded":"2024-02-21T23:07:30.093360+0000","last_fullsized":"2024-02-21T23:07:30.093360+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_clean_scrub_stamp":"2024-02-21T23:07:28.006668+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:36:25.471297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.062950+0000","last_change":"2024-02-21T23:07:30.060999+0000","last_active":"2024-02-21T23:07:30.062950+0000","last_peered":"2024-02-21T23:07:30.062950+0000","last_clean":"2024-02-21T23:07:30.062950+0000","last_became_active":"2024-02-21T23:07:30.059434+0000","last_became_peered":"2024-02-21T23:07:30.059434+0000","last_unstale":"2024-02-21T23:07:30.062950+0000","last_undegraded":"2024-02-21T23:07:30.062950+0000","last_fullsized":"2024-02-21T23:07:30.062950+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:18:29.440779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.046810+0000","last_change":"2024-02-21T23:07:29.074974+0000","last_active":"2024-02-21T23:07:30.046810+0000","last_peered":"2024-02-21T23:07:30.046810+0000","last_clean":"2024-02-21T23:07:30.046810+0000","last_became_active":"2024-02-21T23:07:29.074414+0000","last_became_peered":"2024-02-21T23:07:29.074414+0000","last_unstale":"2024-02-21T23:07:30.046810+0000","last_undegraded":"2024-02-21T23:07:30.046810+0000","last_fullsized":"2024-02-21T23:07:30.046810+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_clean_scrub_stamp":"2024-02-21T23:07:28.006668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:42:56.912661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.077411+0000","last_change":"2024-02-21T23:07:30.068424+0000","last_active":"2024-02-21T23:07:30.077411+0000","last_peered":"2024-02-21T23:07:30.077411+0000","last_clean":"2024-02-21T23:07:30.077411+0000","last_became_active":"2024-02-21T23:07:30.068076+0000","last_became_peered":"2024-02-21T23:07:30.068076+0000","last_unstale":"2024-02-21T23:07:30.077411+0000","last_undegraded":"2024-02-21T23:07:30.077411+0000","last_fullsized":"2024-02-21T23:07:30.077411+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:36:22.445832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.043805+0000","last_change":"2024-02-21T23:07:29.071783+0000","last_active":"2024-02-21T23:07:30.043805+0000","last_peered":"2024-02-21T23:07:30.043805+0000","last_clean":"2024-02-21T23:07:30.043805+0000","last_became_active":"2024-02-21T23:07:29.070845+0000","last_became_peered":"2024-02-21T23:07:29.070845+0000","last_unstale":"2024-02-21T23:07:30.043805+0000","last_undegraded":"2024-02-21T23:07:30.043805+0000","last_fullsized":"2024-02-21T23:07:30.043805+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_clean_scrub_stamp":"2024-02-21T23:07:28.006668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:29:02.297339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.073538+0000","last_change":"2024-02-21T23:07:30.068242+0000","last_active":"2024-02-21T23:07:30.073538+0000","last_peered":"2024-02-21T23:07:30.073538+0000","last_clean":"2024-02-21T23:07:30.073538+0000","last_became_active":"2024-02-21T23:07:30.067823+0000","last_became_peered":"2024-02-21T23:07:30.067823+0000","last_unstale":"2024-02-21T23:07:30.073538+0000","last_undegraded":"2024-02-21T23:07:30.073538+0000","last_fullsized":"2024-02-21T23:07:30.073538+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:47:16.637877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.f","version":"109'1","reported_seq":13,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.085123+0000","last_change":"2024-02-21T23:07:29.067956+0000","last_active":"2024-02-21T23:07:30.085123+0000","last_peered":"2024-02-21T23:07:30.085123+0000","last_clean":"2024-02-21T23:07:30.085123+0000","last_became_active":"2024-02-21T23:07:29.067670+0000","last_became_peered":"2024-02-21T23:07:29.067670+0000","last_unstale":"2024-02-21T23:07:30.085123+0000","last_undegraded":"2024-02-21T23:07:30.085123+0000","last_fullsized":"2024-02-21T23:07:30.085123+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_clean_scrub_stamp":"2024-02-21T23:07:28.006668+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:55:11.484777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.065666+0000","last_change":"2024-02-21T23:07:30.057744+0000","last_active":"2024-02-21T23:07:30.065666+0000","last_peered":"2024-02-21T23:07:30.065666+0000","last_clean":"2024-02-21T23:07:30.065666+0000","last_became_active":"2024-02-21T23:07:30.053562+0000","last_became_peered":"2024-02-21T23:07:30.053562+0000","last_unstale":"2024-02-21T23:07:30.065666+0000","last_undegraded":"2024-02-21T23:07:30.065666+0000","last_fullsized":"2024-02-21T23:07:30.065666+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:28:57.130238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.e","version":"109'2","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.081153+0000","last_change":"2024-02-21T23:07:29.066586+0000","last_active":"2024-02-21T23:07:30.081153+0000","last_peered":"2024-02-21T23:07:30.081153+0000","last_clean":"2024-02-21T23:07:30.081153+0000","last_became_active":"2024-02-21T23:07:29.066268+0000","last_became_peered":"2024-02-21T23:07:29.066268+0000","last_unstale":"2024-02-21T23:07:30.081153+0000","last_undegraded":"2024-02-21T23:07:30.081153+0000","last_fullsized":"2024-02-21T23:07:30.081153+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_clean_scrub_stamp":"2024-02-21T23:07:28.006668+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:42:13.465978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.077659+0000","last_change":"2024-02-21T23:07:30.074074+0000","last_active":"2024-02-21T23:07:30.077659+0000","last_peered":"2024-02-21T23:07:30.077659+0000","last_clean":"2024-02-21T23:07:30.077659+0000","last_became_active":"2024-02-21T23:07:30.073581+0000","last_became_peered":"2024-02-21T23:07:30.073581+0000","last_unstale":"2024-02-21T23:07:30.077659+0000","last_undegraded":"2024-02-21T23:07:30.077659+0000","last_fullsized":"2024-02-21T23:07:30.077659+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:15:22.969452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.d","version":"109'1","reported_seq":13,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.088883+0000","last_change":"2024-02-21T23:07:29.076731+0000","last_active":"2024-02-21T23:07:30.088883+0000","last_peered":"2024-02-21T23:07:30.088883+0000","last_clean":"2024-02-21T23:07:30.088883+0000","last_became_active":"2024-02-21T23:07:29.076536+0000","last_became_peered":"2024-02-21T23:07:29.076536+0000","last_unstale":"2024-02-21T23:07:30.088883+0000","last_undegraded":"2024-02-21T23:07:30.088883+0000","last_fullsized":"2024-02-21T23:07:30.088883+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_clean_scrub_stamp":"2024-02-21T23:07:28.006668+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:47:15.622668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.087472+0000","last_change":"2024-02-21T23:07:30.078047+0000","last_active":"2024-02-21T23:07:30.087472+0000","last_peered":"2024-02-21T23:07:30.087472+0000","last_clean":"2024-02-21T23:07:30.087472+0000","last_became_active":"2024-02-21T23:07:30.074058+0000","last_became_peered":"2024-02-21T23:07:30.074058+0000","last_unstale":"2024-02-21T23:07:30.087472+0000","last_undegraded":"2024-02-21T23:07:30.087472+0000","last_fullsized":"2024-02-21T23:07:30.087472+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:48:05.594176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.056851+0000","last_change":"2024-02-21T23:07:29.074595+0000","last_active":"2024-02-21T23:07:30.056851+0000","last_peered":"2024-02-21T23:07:30.056851+0000","last_clean":"2024-02-21T23:07:30.056851+0000","last_became_active":"2024-02-21T23:07:29.074070+0000","last_became_peered":"2024-02-21T23:07:29.074070+0000","last_unstale":"2024-02-21T23:07:30.056851+0000","last_undegraded":"2024-02-21T23:07:30.056851+0000","last_fullsized":"2024-02-21T23:07:30.056851+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_clean_scrub_stamp":"2024-02-21T23:07:28.006668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:14:31.115214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.084461+0000","last_change":"2024-02-21T23:07:30.081454+0000","last_active":"2024-02-21T23:07:30.084461+0000","last_peered":"2024-02-21T23:07:30.084461+0000","last_clean":"2024-02-21T23:07:30.084461+0000","last_became_active":"2024-02-21T23:07:30.081149+0000","last_became_peered":"2024-02-21T23:07:30.081149+0000","last_unstale":"2024-02-21T23:07:30.084461+0000","last_undegraded":"2024-02-21T23:07:30.084461+0000","last_fullsized":"2024-02-21T23:07:30.084461+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:01:21.611573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"109'2","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.088573+0000","last_change":"2024-02-21T23:07:29.071631+0000","last_active":"2024-02-21T23:07:30.088573+0000","last_peered":"2024-02-21T23:07:30.088573+0000","last_clean":"2024-02-21T23:07:30.088573+0000","last_became_active":"2024-02-21T23:07:29.071335+0000","last_became_peered":"2024-02-21T23:07:29.071335+0000","last_unstale":"2024-02-21T23:07:30.088573+0000","last_undegraded":"2024-02-21T23:07:30.088573+0000","last_fullsized":"2024-02-21T23:07:30.088573+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_clean_scrub_stamp":"2024-02-21T23:07:28.006668+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:26:56.660421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.081245+0000","last_change":"2024-02-21T23:07:30.069563+0000","last_active":"2024-02-21T23:07:30.081245+0000","last_peered":"2024-02-21T23:07:30.081245+0000","last_clean":"2024-02-21T23:07:30.081245+0000","last_became_active":"2024-02-21T23:07:30.069239+0000","last_became_peered":"2024-02-21T23:07:30.069239+0000","last_unstale":"2024-02-21T23:07:30.081245+0000","last_undegraded":"2024-02-21T23:07:30.081245+0000","last_fullsized":"2024-02-21T23:07:30.081245+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:20:59.588268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.052298+0000","last_change":"2024-02-21T23:07:29.070486+0000","last_active":"2024-02-21T23:07:30.052298+0000","last_peered":"2024-02-21T23:07:30.052298+0000","last_clean":"2024-02-21T23:07:30.052298+0000","last_became_active":"2024-02-21T23:07:29.070179+0000","last_became_peered":"2024-02-21T23:07:29.070179+0000","last_unstale":"2024-02-21T23:07:30.052298+0000","last_undegraded":"2024-02-21T23:07:30.052298+0000","last_fullsized":"2024-02-21T23:07:30.052298+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_clean_scrub_stamp":"2024-02-21T23:07:28.006668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:28:33.860300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.085451+0000","last_change":"2024-02-21T23:07:30.080250+0000","last_active":"2024-02-21T23:07:30.085451+0000","last_peered":"2024-02-21T23:07:30.085451+0000","last_clean":"2024-02-21T23:07:30.085451+0000","last_became_active":"2024-02-21T23:07:30.079915+0000","last_became_peered":"2024-02-21T23:07:30.079915+0000","last_unstale":"2024-02-21T23:07:30.085451+0000","last_undegraded":"2024-02-21T23:07:30.085451+0000","last_fullsized":"2024-02-21T23:07:30.085451+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:00:18.649979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.044147+0000","last_change":"2024-02-21T23:07:29.064409+0000","last_active":"2024-02-21T23:07:30.044147+0000","last_peered":"2024-02-21T23:07:30.044147+0000","last_clean":"2024-02-21T23:07:30.044147+0000","last_became_active":"2024-02-21T23:07:29.063935+0000","last_became_peered":"2024-02-21T23:07:29.063935+0000","last_unstale":"2024-02-21T23:07:30.044147+0000","last_undegraded":"2024-02-21T23:07:30.044147+0000","last_fullsized":"2024-02-21T23:07:30.044147+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_clean_scrub_stamp":"2024-02-21T23:07:28.006668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:43:47.978454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.087490+0000","last_change":"2024-02-21T23:07:30.079069+0000","last_active":"2024-02-21T23:07:30.087490+0000","last_peered":"2024-02-21T23:07:30.087490+0000","last_clean":"2024-02-21T23:07:30.087490+0000","last_became_active":"2024-02-21T23:07:30.076338+0000","last_became_peered":"2024-02-21T23:07:30.076338+0000","last_unstale":"2024-02-21T23:07:30.087490+0000","last_undegraded":"2024-02-21T23:07:30.087490+0000","last_fullsized":"2024-02-21T23:07:30.087490+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:36:26.486506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.052691+0000","last_change":"2024-02-21T23:07:29.074845+0000","last_active":"2024-02-21T23:07:30.052691+0000","last_peered":"2024-02-21T23:07:30.052691+0000","last_clean":"2024-02-21T23:07:30.052691+0000","last_became_active":"2024-02-21T23:07:29.074537+0000","last_became_peered":"2024-02-21T23:07:29.074537+0000","last_unstale":"2024-02-21T23:07:30.052691+0000","last_undegraded":"2024-02-21T23:07:30.052691+0000","last_fullsized":"2024-02-21T23:07:30.052691+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_clean_scrub_stamp":"2024-02-21T23:07:28.006668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:28:56.115029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.063640+0000","last_change":"2024-02-21T23:07:30.061637+0000","last_active":"2024-02-21T23:07:30.063640+0000","last_peered":"2024-02-21T23:07:30.063640+0000","last_clean":"2024-02-21T23:07:30.063640+0000","last_became_active":"2024-02-21T23:07:30.061326+0000","last_became_peered":"2024-02-21T23:07:30.061326+0000","last_unstale":"2024-02-21T23:07:30.063640+0000","last_undegraded":"2024-02-21T23:07:30.063640+0000","last_fullsized":"2024-02-21T23:07:30.063640+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:20:05.469348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.7","version":"109'2","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.084643+0000","last_change":"2024-02-21T23:07:29.069912+0000","last_active":"2024-02-21T23:07:30.084643+0000","last_peered":"2024-02-21T23:07:30.084643+0000","last_clean":"2024-02-21T23:07:30.084643+0000","last_became_active":"2024-02-21T23:07:29.069479+0000","last_became_peered":"2024-02-21T23:07:29.069479+0000","last_unstale":"2024-02-21T23:07:30.084643+0000","last_undegraded":"2024-02-21T23:07:30.084643+0000","last_fullsized":"2024-02-21T23:07:30.084643+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_clean_scrub_stamp":"2024-02-21T23:07:28.006668+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:14:47.017723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.086610+0000","last_change":"2024-02-21T23:07:30.079590+0000","last_active":"2024-02-21T23:07:30.086610+0000","last_peered":"2024-02-21T23:07:30.086610+0000","last_clean":"2024-02-21T23:07:30.086610+0000","last_became_active":"2024-02-21T23:07:30.078597+0000","last_became_peered":"2024-02-21T23:07:30.078597+0000","last_unstale":"2024-02-21T23:07:30.086610+0000","last_undegraded":"2024-02-21T23:07:30.086610+0000","last_fullsized":"2024-02-21T23:07:30.086610+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:28:34.875509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.070651+0000","last_change":"2024-02-21T23:07:29.071395+0000","last_active":"2024-02-21T23:07:30.070651+0000","last_peered":"2024-02-21T23:07:30.070651+0000","last_clean":"2024-02-21T23:07:30.070651+0000","last_became_active":"2024-02-21T23:07:29.071099+0000","last_became_peered":"2024-02-21T23:07:29.071099+0000","last_unstale":"2024-02-21T23:07:30.070651+0000","last_undegraded":"2024-02-21T23:07:30.070651+0000","last_fullsized":"2024-02-21T23:07:30.070651+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_clean_scrub_stamp":"2024-02-21T23:07:28.006668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:33:49.856373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.078707+0000","last_change":"2024-02-21T23:07:30.076179+0000","last_active":"2024-02-21T23:07:30.078707+0000","last_peered":"2024-02-21T23:07:30.078707+0000","last_clean":"2024-02-21T23:07:30.078707+0000","last_became_active":"2024-02-21T23:07:30.075869+0000","last_became_peered":"2024-02-21T23:07:30.075869+0000","last_unstale":"2024-02-21T23:07:30.078707+0000","last_undegraded":"2024-02-21T23:07:30.078707+0000","last_fullsized":"2024-02-21T23:07:30.078707+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:43:48.993663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.070708+0000","last_change":"2024-02-21T23:07:29.072468+0000","last_active":"2024-02-21T23:07:30.070708+0000","last_peered":"2024-02-21T23:07:30.070708+0000","last_clean":"2024-02-21T23:07:30.070708+0000","last_became_active":"2024-02-21T23:07:29.072165+0000","last_became_peered":"2024-02-21T23:07:29.072165+0000","last_unstale":"2024-02-21T23:07:30.070708+0000","last_undegraded":"2024-02-21T23:07:30.070708+0000","last_fullsized":"2024-02-21T23:07:30.070708+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_clean_scrub_stamp":"2024-02-21T23:07:28.006668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:00:09.026620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.078791+0000","last_change":"2024-02-21T23:07:30.072026+0000","last_active":"2024-02-21T23:07:30.078791+0000","last_peered":"2024-02-21T23:07:30.078791+0000","last_clean":"2024-02-21T23:07:30.078791+0000","last_became_active":"2024-02-21T23:07:30.071675+0000","last_became_peered":"2024-02-21T23:07:30.071675+0000","last_unstale":"2024-02-21T23:07:30.078791+0000","last_undegraded":"2024-02-21T23:07:30.078791+0000","last_fullsized":"2024-02-21T23:07:30.078791+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:17:30.425460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.039050+0000","last_change":"2024-02-21T23:07:29.055828+0000","last_active":"2024-02-21T23:07:30.039050+0000","last_peered":"2024-02-21T23:07:30.039050+0000","last_clean":"2024-02-21T23:07:30.039050+0000","last_became_active":"2024-02-21T23:07:29.054642+0000","last_became_peered":"2024-02-21T23:07:29.054642+0000","last_unstale":"2024-02-21T23:07:30.039050+0000","last_undegraded":"2024-02-21T23:07:30.039050+0000","last_fullsized":"2024-02-21T23:07:30.039050+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_clean_scrub_stamp":"2024-02-21T23:07:28.006668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:07:33.232700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.063449+0000","last_change":"2024-02-21T23:07:30.050940+0000","last_active":"2024-02-21T23:07:30.063449+0000","last_peered":"2024-02-21T23:07:30.063449+0000","last_clean":"2024-02-21T23:07:30.063449+0000","last_became_active":"2024-02-21T23:07:30.050426+0000","last_became_peered":"2024-02-21T23:07:30.050426+0000","last_unstale":"2024-02-21T23:07:30.063449+0000","last_undegraded":"2024-02-21T23:07:30.063449+0000","last_fullsized":"2024-02-21T23:07:30.063449+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:50:47.848516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.045644+0000","last_change":"2024-02-21T23:07:29.068231+0000","last_active":"2024-02-21T23:07:30.045644+0000","last_peered":"2024-02-21T23:07:30.045644+0000","last_clean":"2024-02-21T23:07:30.045644+0000","last_became_active":"2024-02-21T23:07:29.067904+0000","last_became_peered":"2024-02-21T23:07:29.067904+0000","last_unstale":"2024-02-21T23:07:30.045644+0000","last_undegraded":"2024-02-21T23:07:30.045644+0000","last_fullsized":"2024-02-21T23:07:30.045644+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_clean_scrub_stamp":"2024-02-21T23:07:28.006668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:28:33.860300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.085151+0000","last_change":"2024-02-21T23:07:30.074982+0000","last_active":"2024-02-21T23:07:30.085151+0000","last_peered":"2024-02-21T23:07:30.085151+0000","last_clean":"2024-02-21T23:07:30.085151+0000","last_became_active":"2024-02-21T23:07:30.074611+0000","last_became_peered":"2024-02-21T23:07:30.074611+0000","last_unstale":"2024-02-21T23:07:30.085151+0000","last_undegraded":"2024-02-21T23:07:30.085151+0000","last_fullsized":"2024-02-21T23:07:30.085151+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:33:00.995974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.043972+0000","last_change":"2024-02-21T23:07:29.073321+0000","last_active":"2024-02-21T23:07:30.043972+0000","last_peered":"2024-02-21T23:07:30.043972+0000","last_clean":"2024-02-21T23:07:30.043972+0000","last_became_active":"2024-02-21T23:07:29.072825+0000","last_became_peered":"2024-02-21T23:07:29.072825+0000","last_unstale":"2024-02-21T23:07:30.043972+0000","last_undegraded":"2024-02-21T23:07:30.043972+0000","last_fullsized":"2024-02-21T23:07:30.043972+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_clean_scrub_stamp":"2024-02-21T23:07:28.006668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:41:20.161387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.074751+0000","last_change":"2024-02-21T23:07:30.072238+0000","last_active":"2024-02-21T23:07:30.074751+0000","last_peered":"2024-02-21T23:07:30.074751+0000","last_clean":"2024-02-21T23:07:30.074751+0000","last_became_active":"2024-02-21T23:07:30.068670+0000","last_became_peered":"2024-02-21T23:07:30.068670+0000","last_unstale":"2024-02-21T23:07:30.074751+0000","last_undegraded":"2024-02-21T23:07:30.074751+0000","last_fullsized":"2024-02-21T23:07:30.074751+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:32:44.283700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.035590+0000","last_change":"2024-02-21T23:07:29.062336+0000","last_active":"2024-02-21T23:07:30.035590+0000","last_peered":"2024-02-21T23:07:30.035590+0000","last_clean":"2024-02-21T23:07:30.035590+0000","last_became_active":"2024-02-21T23:07:29.060554+0000","last_became_peered":"2024-02-21T23:07:29.060554+0000","last_unstale":"2024-02-21T23:07:30.035590+0000","last_undegraded":"2024-02-21T23:07:30.035590+0000","last_fullsized":"2024-02-21T23:07:30.035590+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_clean_scrub_stamp":"2024-02-21T23:07:28.006668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:22:10.884648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.075509+0000","last_change":"2024-02-21T23:07:30.072341+0000","last_active":"2024-02-21T23:07:30.075509+0000","last_peered":"2024-02-21T23:07:30.075509+0000","last_clean":"2024-02-21T23:07:30.075509+0000","last_became_active":"2024-02-21T23:07:30.071082+0000","last_became_peered":"2024-02-21T23:07:30.071082+0000","last_unstale":"2024-02-21T23:07:30.075509+0000","last_undegraded":"2024-02-21T23:07:30.075509+0000","last_fullsized":"2024-02-21T23:07:30.075509+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:00:10.041830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.053129+0000","last_change":"2024-02-21T23:07:29.072647+0000","last_active":"2024-02-21T23:07:30.053129+0000","last_peered":"2024-02-21T23:07:30.053129+0000","last_clean":"2024-02-21T23:07:30.053129+0000","last_became_active":"2024-02-21T23:07:29.072168+0000","last_became_peered":"2024-02-21T23:07:29.072168+0000","last_unstale":"2024-02-21T23:07:30.053129+0000","last_undegraded":"2024-02-21T23:07:30.053129+0000","last_fullsized":"2024-02-21T23:07:30.053129+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_clean_scrub_stamp":"2024-02-21T23:07:28.006668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:32:39.621456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.073381+0000","last_change":"2024-02-21T23:07:30.068176+0000","last_active":"2024-02-21T23:07:30.073381+0000","last_peered":"2024-02-21T23:07:30.073381+0000","last_clean":"2024-02-21T23:07:30.073381+0000","last_became_active":"2024-02-21T23:07:30.065279+0000","last_became_peered":"2024-02-21T23:07:30.065279+0000","last_unstale":"2024-02-21T23:07:30.073381+0000","last_undegraded":"2024-02-21T23:07:30.073381+0000","last_fullsized":"2024-02-21T23:07:30.073381+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:48:05.594176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.10","version":"109'8","reported_seq":20,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.094056+0000","last_change":"2024-02-21T23:07:29.068221+0000","last_active":"2024-02-21T23:07:30.094056+0000","last_peered":"2024-02-21T23:07:30.094056+0000","last_clean":"2024-02-21T23:07:30.094056+0000","last_became_active":"2024-02-21T23:07:29.067873+0000","last_became_peered":"2024-02-21T23:07:29.067873+0000","last_unstale":"2024-02-21T23:07:30.094056+0000","last_undegraded":"2024-02-21T23:07:30.094056+0000","last_fullsized":"2024-02-21T23:07:30.094056+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_clean_scrub_stamp":"2024-02-21T23:07:28.006668+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:42:34.060965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":336,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.053120+0000","last_change":"2024-02-21T23:02:32.289335+0000","last_active":"2024-02-21T23:07:30.053120+0000","last_peered":"2024-02-21T23:07:30.053120+0000","last_clean":"2024-02-21T23:07:30.053120+0000","last_became_active":"2024-02-21T23:02:32.286913+0000","last_became_peered":"2024-02-21T23:02:32.286913+0000","last_unstale":"2024-02-21T23:07:30.053120+0000","last_undegraded":"2024-02-21T23:07:30.053120+0000","last_fullsized":"2024-02-21T23:07:30.053120+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_clean_scrub_stamp":"2024-02-21T23:02:31.172569+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:36:23.987374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":18,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574910,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32596,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339244,"statfs":{"total":96636764160,"available":96603385856,"internally_reserved":0,"allocated":1441792,"data_stored":1253567,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574911,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32500,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339340,"statfs":{"total":96636764160,"available":96603484160,"internally_reserved":0,"allocated":1347584,"data_stored":655256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574910,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32516,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339324,"statfs":{"total":96636764160,"available":96603467776,"internally_reserved":0,"allocated":1368064,"data_stored":662583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574911,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32524,"kb_used_data":1340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339316,"statfs":{"total":96636764160,"available":96603459584,"internally_reserved":0,"allocated":1372160,"data_stored":664035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574911,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32528,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339312,"statfs":{"total":96636764160,"available":96603455488,"internally_reserved":0,"allocated":1376256,"data_stored":664069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574912,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32512,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339328,"statfs":{"total":96636764160,"available":96603471872,"internally_reserved":0,"allocated":1359872,"data_stored":662423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574910,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32516,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339324,"statfs":{"total":96636764160,"available":96603467776,"internally_reserved":0,"allocated":1363968,"data_stored":662445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574910,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32580,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339260,"statfs":{"total":96636764160,"available":96603402240,"internally_reserved":0,"allocated":1429504,"data_stored":1252813,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-21T23:07:34.637 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-21T23:07:34.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.910+0000 7f1c2cc00640 1 Processor -- start 2024-02-21T23:07:34.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.911+0000 7f1c2cc00640 1 -- start start 2024-02-21T23:07:34.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.912+0000 7f1c2cc00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1c280c8a80 0x7f1c28156c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:34.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.912+0000 7f1c2cc00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1c281540b0 0x7f1c2815d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:34.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.913+0000 7f1c2cc00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c280c8000 con 0x7f1c280c8e60 2024-02-21T23:07:34.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.913+0000 7f1c2cc00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c280c92e0 con 0x7f1c281540b0 2024-02-21T23:07:34.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.913+0000 7f1c2cc00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c280c82e0 con 0x7f1c280c8a80 2024-02-21T23:07:34.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.913+0000 7f1c27e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1c281540b0 0x7f1c2815d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:34.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.913+0000 7f1c27e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1c281540b0 0x7f1c2815d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58164/0 (socket says 172.21.15.28:58164) 2024-02-21T23:07:34.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.913+0000 7f1c27e00640 1 -- 172.21.15.28:0/3028853601 learned_addr learned my addr 172.21.15.28:0/3028853601 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:34.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.913+0000 7f1c27400640 1 --2- 172.21.15.28:0/3028853601 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1c280c8a80 0x7f1c28156c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:34.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.914+0000 7f1c27e00640 1 -- 172.21.15.28:0/3028853601 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1c280c8a80 msgr2=0x7f1c28156c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:34.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.914+0000 7f1c27e00640 1 --2- 172.21.15.28:0/3028853601 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1c280c8a80 0x7f1c28156c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:34.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.914+0000 7f1c27e00640 1 -- 172.21.15.28:0/3028853601 >> v1:172.21.15.81:6789/0 conn(0x7f1c280c8e60 legacy=0x7f1c28153640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:34.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.914+0000 7f1c27e00640 1 -- 172.21.15.28:0/3028853601 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c281561b0 con 0x7f1c281540b0 2024-02-21T23:07:34.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.915+0000 7f1c27e00640 1 --2- 172.21.15.28:0/3028853601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c281540b0 0x7f1c2815d540 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f1c20009e50 tx=0x7f1c2002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a190137b6da0c8d server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:34.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.915+0000 7f1c26a00640 1 -- 172.21.15.28:0/3028853601 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c2002ca70 con 0x7f1c281540b0 2024-02-21T23:07:34.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.915+0000 7f1c26a00640 1 -- 172.21.15.28:0/3028853601 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c2002cc10 con 0x7f1c281540b0 2024-02-21T23:07:34.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.915+0000 7f1c26a00640 1 -- 172.21.15.28:0/3028853601 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c200343f0 con 0x7f1c281540b0 2024-02-21T23:07:34.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.916+0000 7f1c2cc00640 1 -- 172.21.15.28:0/3028853601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c281540b0 msgr2=0x7f1c2815d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:34.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.916+0000 7f1c2cc00640 1 --2- 172.21.15.28:0/3028853601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c281540b0 0x7f1c2815d540 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f1c20009e50 tx=0x7f1c2002bed0 comp rx=0 tx=0).stop 2024-02-21T23:07:34.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.916+0000 7f1c2cc00640 1 -- 172.21.15.28:0/3028853601 shutdown_connections 2024-02-21T23:07:34.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.916+0000 7f1c2cc00640 1 --2- 172.21.15.28:0/3028853601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c281540b0 0x7f1c2815d540 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:34.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.916+0000 7f1c2cc00640 1 --2- 172.21.15.28:0/3028853601 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1c280c8a80 0x7f1c28156c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:34.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.916+0000 7f1c2cc00640 1 -- 172.21.15.28:0/3028853601 >> 172.21.15.28:0/3028853601 conn(0x7f1c2806b260 msgr2=0x7f1c2806b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:34.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.916+0000 7f1c2cc00640 1 -- 172.21.15.28:0/3028853601 shutdown_connections 2024-02-21T23:07:34.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.916+0000 7f1c2cc00640 1 -- 172.21.15.28:0/3028853601 wait complete. 2024-02-21T23:07:34.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.918+0000 7f1c2cc00640 1 Processor -- start 2024-02-21T23:07:34.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.918+0000 7f1c2cc00640 1 -- start start 2024-02-21T23:07:34.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.918+0000 7f1c2cc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c280c8a80 0x7f1c280acf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:34.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.919+0000 7f1c2cc00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1c280c8e60 0x7f1c280c3190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:34.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.919+0000 7f1c2cc00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c281540b0 0x7f1c281476c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:34.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.919+0000 7f1c2cc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c280c7c90 con 0x7f1c280c8a80 2024-02-21T23:07:34.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.919+0000 7f1c2cc00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c28156770 con 0x7f1c281540b0 2024-02-21T23:07:34.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.919+0000 7f1c2cc00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c280c2c30 con 0x7f1c280c8e60 2024-02-21T23:07:34.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.919+0000 7f1c27400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c280c8a80 0x7f1c280acf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:34.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.919+0000 7f1c27400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c280c8a80 0x7f1c280acf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58176/0 (socket says 172.21.15.28:58176) 2024-02-21T23:07:34.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.919+0000 7f1c27400640 1 -- 172.21.15.28:0/2094884171 learned_addr learned my addr 172.21.15.28:0/2094884171 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:34.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.919+0000 7f1c27400640 1 -- 172.21.15.28:0/2094884171 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1c280c8e60 msgr2=0x7f1c280c3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:34.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.919+0000 7f1c27400640 1 --2- 172.21.15.28:0/2094884171 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1c280c8e60 0x7f1c280c3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:34.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.919+0000 7f1c27400640 1 -- 172.21.15.28:0/2094884171 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c281540b0 msgr2=0x7f1c281476c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:34.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.919+0000 7f1c27400640 1 --2- 172.21.15.28:0/2094884171 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c281540b0 0x7f1c281476c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:34.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.919+0000 7f1c27400640 1 -- 172.21.15.28:0/2094884171 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c280ab530 con 0x7f1c280c8a80 2024-02-21T23:07:34.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.920+0000 7f1c27400640 1 --2- 172.21.15.28:0/2094884171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c280c8a80 0x7f1c280acf50 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f1c1800d5f0 tx=0x7f1c1800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:34.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.920+0000 7f1c1fe00640 1 -- 172.21.15.28:0/2094884171 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c180125f0 con 0x7f1c280c8a80 2024-02-21T23:07:34.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.920+0000 7f1c1fe00640 1 -- 172.21.15.28:0/2094884171 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c18019070 con 0x7f1c280c8a80 2024-02-21T23:07:34.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.920+0000 7f1c2cc00640 1 -- 172.21.15.28:0/2094884171 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c281eefd0 con 0x7f1c280c8a80 2024-02-21T23:07:34.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.920+0000 7f1c1fe00640 1 -- 172.21.15.28:0/2094884171 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c180129d0 con 0x7f1c280c8a80 2024-02-21T23:07:34.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.920+0000 7f1c2cc00640 1 -- 172.21.15.28:0/2094884171 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c280ad8b0 con 0x7f1c280c8a80 2024-02-21T23:07:34.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.922+0000 7f1c2cc00640 1 -- 172.21.15.28:0/2094884171 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c280c2880 con 0x7f1c280c8a80 2024-02-21T23:07:34.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.922+0000 7f1c1fe00640 1 -- 172.21.15.28:0/2094884171 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1c18012b70 con 0x7f1c280c8a80 2024-02-21T23:07:34.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.923+0000 7f1c1fe00640 1 --2- 172.21.15.28:0/2094884171 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1c0406b4d0 0x7f1c0406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:34.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.923+0000 7f1c1fe00640 1 -- 172.21.15.28:0/2094884171 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1c1808a650 con 0x7f1c280c8a80 2024-02-21T23:07:34.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.924+0000 7f1c1ea00640 1 --2- 172.21.15.28:0/2094884171 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1c0406b4d0 0x7f1c0406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:34.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.925+0000 7f1c1ea00640 1 --2- 172.21.15.28:0/2094884171 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1c0406b4d0 0x7f1c0406d990 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f1c10000f60 tx=0x7f1c100079d0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:34.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:34.928+0000 7f1c1fe00640 1 -- 172.21.15.28:0/2094884171 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1c280c2880 con 0x7f1c280c8a80 2024-02-21T23:07:35.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.160+0000 7f1c2cc00640 1 -- 172.21.15.28:0/2094884171 --> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1c2806e550 con 0x7f1c0406b4d0 2024-02-21T23:07:35.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.178+0000 7f1c1fe00640 1 -- 172.21.15.28:0/2094884171 <== mgr.4101 v2:172.21.15.81:6832/1848538577 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252493 (secure 0 0 0) 0x7f1c2806e550 con 0x7f1c0406b4d0 2024-02-21T23:07:35.170 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:07:35.172 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-21T23:07:35.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.184+0000 7f1c2cc00640 1 -- 172.21.15.28:0/2094884171 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1c0406b4d0 msgr2=0x7f1c0406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:35.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.184+0000 7f1c2cc00640 1 --2- 172.21.15.28:0/2094884171 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1c0406b4d0 0x7f1c0406d990 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f1c10000f60 tx=0x7f1c100079d0 comp rx=0 tx=0).stop 2024-02-21T23:07:35.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.184+0000 7f1c2cc00640 1 -- 172.21.15.28:0/2094884171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c280c8a80 msgr2=0x7f1c280acf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:35.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.184+0000 7f1c2cc00640 1 --2- 172.21.15.28:0/2094884171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c280c8a80 0x7f1c280acf50 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f1c1800d5f0 tx=0x7f1c1800dab0 comp rx=0 tx=0).stop 2024-02-21T23:07:35.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.185+0000 7f1c2cc00640 1 -- 172.21.15.28:0/2094884171 shutdown_connections 2024-02-21T23:07:35.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.185+0000 7f1c2cc00640 1 --2- 172.21.15.28:0/2094884171 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1c0406b4d0 0x7f1c0406d990 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:35.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.185+0000 7f1c2cc00640 1 --2- 172.21.15.28:0/2094884171 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1c280c8e60 0x7f1c280c3190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:35.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.185+0000 7f1c2cc00640 1 --2- 172.21.15.28:0/2094884171 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c281540b0 0x7f1c281476c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:35.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.185+0000 7f1c2cc00640 1 --2- 172.21.15.28:0/2094884171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c280c8a80 0x7f1c280acf50 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:35.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.185+0000 7f1c2cc00640 1 -- 172.21.15.28:0/2094884171 >> 172.21.15.28:0/2094884171 conn(0x7f1c2806b260 msgr2=0x7f1c2806b640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:35.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.185+0000 7f1c2cc00640 1 -- 172.21.15.28:0/2094884171 shutdown_connections 2024-02-21T23:07:35.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.185+0000 7f1c2cc00640 1 -- 172.21.15.28:0/2094884171 wait complete. 2024-02-21T23:07:35.192 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":255,"stamp":"2024-02-21T23:07:34.770286+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":70,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":260272,"kb_used_data":10800,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754714448,"statfs":{"total":773094113280,"available":772827594752,"internally_reserved":0,"allocated":11059200,"data_stored":6477191,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":40,"apply_latency_ms":40,"commit_latency_ns":40000000,"apply_latency_ns":40000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001817"},"pg_stats":[{"pgid":"18.3f","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.067550+0000","last_change":"2024-02-21T23:07:30.059591+0000","last_active":"2024-02-21T23:07:30.067550+0000","last_peered":"2024-02-21T23:07:30.067550+0000","last_clean":"2024-02-21T23:07:30.067550+0000","last_became_active":"2024-02-21T23:07:30.059262+0000","last_became_peered":"2024-02-21T23:07:30.059262+0000","last_unstale":"2024-02-21T23:07:30.067550+0000","last_undegraded":"2024-02-21T23:07:30.067550+0000","last_fullsized":"2024-02-21T23:07:30.067550+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:47:16.637877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3e","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.085757+0000","last_change":"2024-02-21T23:07:30.084488+0000","last_active":"2024-02-21T23:07:30.085757+0000","last_peered":"2024-02-21T23:07:30.085757+0000","last_clean":"2024-02-21T23:07:30.085757+0000","last_became_active":"2024-02-21T23:07:30.080786+0000","last_became_peered":"2024-02-21T23:07:30.080786+0000","last_unstale":"2024-02-21T23:07:30.085757+0000","last_undegraded":"2024-02-21T23:07:30.085757+0000","last_fullsized":"2024-02-21T23:07:30.085757+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:43:13.411504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3d","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.085469+0000","last_change":"2024-02-21T23:07:30.080111+0000","last_active":"2024-02-21T23:07:30.085469+0000","last_peered":"2024-02-21T23:07:30.085469+0000","last_clean":"2024-02-21T23:07:30.085469+0000","last_became_active":"2024-02-21T23:07:30.079687+0000","last_became_peered":"2024-02-21T23:07:30.079687+0000","last_unstale":"2024-02-21T23:07:30.085469+0000","last_undegraded":"2024-02-21T23:07:30.085469+0000","last_fullsized":"2024-02-21T23:07:30.085469+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:08:35.154317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3c","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.074198+0000","last_change":"2024-02-21T23:07:30.067543+0000","last_active":"2024-02-21T23:07:30.074198+0000","last_peered":"2024-02-21T23:07:30.074198+0000","last_clean":"2024-02-21T23:07:30.074198+0000","last_became_active":"2024-02-21T23:07:30.066543+0000","last_became_peered":"2024-02-21T23:07:30.066543+0000","last_unstale":"2024-02-21T23:07:30.074198+0000","last_undegraded":"2024-02-21T23:07:30.074198+0000","last_fullsized":"2024-02-21T23:07:30.074198+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:36:26.486506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3b","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.084559+0000","last_change":"2024-02-21T23:07:30.081619+0000","last_active":"2024-02-21T23:07:30.084559+0000","last_peered":"2024-02-21T23:07:30.084559+0000","last_clean":"2024-02-21T23:07:30.084559+0000","last_became_active":"2024-02-21T23:07:30.080253+0000","last_became_peered":"2024-02-21T23:07:30.080253+0000","last_unstale":"2024-02-21T23:07:30.084559+0000","last_undegraded":"2024-02-21T23:07:30.084559+0000","last_fullsized":"2024-02-21T23:07:30.084559+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:48:05.594176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3a","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.068878+0000","last_change":"2024-02-21T23:07:30.065698+0000","last_active":"2024-02-21T23:07:30.068878+0000","last_peered":"2024-02-21T23:07:30.068878+0000","last_clean":"2024-02-21T23:07:30.068878+0000","last_became_active":"2024-02-21T23:07:30.064508+0000","last_became_peered":"2024-02-21T23:07:30.064508+0000","last_unstale":"2024-02-21T23:07:30.068878+0000","last_undegraded":"2024-02-21T23:07:30.068878+0000","last_fullsized":"2024-02-21T23:07:30.068878+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:07:55.564030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.39","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.072380+0000","last_change":"2024-02-21T23:07:30.068392+0000","last_active":"2024-02-21T23:07:30.072380+0000","last_peered":"2024-02-21T23:07:30.072380+0000","last_clean":"2024-02-21T23:07:30.072380+0000","last_became_active":"2024-02-21T23:07:30.066886+0000","last_became_peered":"2024-02-21T23:07:30.066886+0000","last_unstale":"2024-02-21T23:07:30.072380+0000","last_undegraded":"2024-02-21T23:07:30.072380+0000","last_fullsized":"2024-02-21T23:07:30.072380+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:47:51.045182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.38","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.087796+0000","last_change":"2024-02-21T23:07:30.082159+0000","last_active":"2024-02-21T23:07:30.087796+0000","last_peered":"2024-02-21T23:07:30.087796+0000","last_clean":"2024-02-21T23:07:30.087796+0000","last_became_active":"2024-02-21T23:07:30.081900+0000","last_became_peered":"2024-02-21T23:07:30.081900+0000","last_unstale":"2024-02-21T23:07:30.087796+0000","last_undegraded":"2024-02-21T23:07:30.087796+0000","last_fullsized":"2024-02-21T23:07:30.087796+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:14:24.006664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.37","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.071663+0000","last_change":"2024-02-21T23:07:30.064760+0000","last_active":"2024-02-21T23:07:30.071663+0000","last_peered":"2024-02-21T23:07:30.071663+0000","last_clean":"2024-02-21T23:07:30.071663+0000","last_became_active":"2024-02-21T23:07:30.063783+0000","last_became_peered":"2024-02-21T23:07:30.063783+0000","last_unstale":"2024-02-21T23:07:30.071663+0000","last_undegraded":"2024-02-21T23:07:30.071663+0000","last_fullsized":"2024-02-21T23:07:30.071663+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:00:31.053859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.36","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.075253+0000","last_change":"2024-02-21T23:07:30.069825+0000","last_active":"2024-02-21T23:07:30.075253+0000","last_peered":"2024-02-21T23:07:30.075253+0000","last_clean":"2024-02-21T23:07:30.075253+0000","last_became_active":"2024-02-21T23:07:30.069253+0000","last_became_peered":"2024-02-21T23:07:30.069253+0000","last_unstale":"2024-02-21T23:07:30.075253+0000","last_undegraded":"2024-02-21T23:07:30.075253+0000","last_fullsized":"2024-02-21T23:07:30.075253+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:12:27.752661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.35","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.082924+0000","last_change":"2024-02-21T23:07:30.072818+0000","last_active":"2024-02-21T23:07:30.082924+0000","last_peered":"2024-02-21T23:07:30.082924+0000","last_clean":"2024-02-21T23:07:30.082924+0000","last_became_active":"2024-02-21T23:07:30.071816+0000","last_became_peered":"2024-02-21T23:07:30.071816+0000","last_unstale":"2024-02-21T23:07:30.082924+0000","last_undegraded":"2024-02-21T23:07:30.082924+0000","last_fullsized":"2024-02-21T23:07:30.082924+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:00:31.053859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.34","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.063599+0000","last_change":"2024-02-21T23:07:30.057927+0000","last_active":"2024-02-21T23:07:30.063599+0000","last_peered":"2024-02-21T23:07:30.063599+0000","last_clean":"2024-02-21T23:07:30.063599+0000","last_became_active":"2024-02-21T23:07:30.057359+0000","last_became_peered":"2024-02-21T23:07:30.057359+0000","last_unstale":"2024-02-21T23:07:30.063599+0000","last_undegraded":"2024-02-21T23:07:30.063599+0000","last_fullsized":"2024-02-21T23:07:30.063599+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:32:49.509733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.33","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.075556+0000","last_change":"2024-02-21T23:07:30.073057+0000","last_active":"2024-02-21T23:07:30.075556+0000","last_peered":"2024-02-21T23:07:30.075556+0000","last_clean":"2024-02-21T23:07:30.075556+0000","last_became_active":"2024-02-21T23:07:30.072773+0000","last_became_peered":"2024-02-21T23:07:30.072773+0000","last_unstale":"2024-02-21T23:07:30.075556+0000","last_undegraded":"2024-02-21T23:07:30.075556+0000","last_fullsized":"2024-02-21T23:07:30.075556+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:41:21.176597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.32","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.085471+0000","last_change":"2024-02-21T23:07:30.080500+0000","last_active":"2024-02-21T23:07:30.085471+0000","last_peered":"2024-02-21T23:07:30.085471+0000","last_clean":"2024-02-21T23:07:30.085471+0000","last_became_active":"2024-02-21T23:07:30.079450+0000","last_became_peered":"2024-02-21T23:07:30.079450+0000","last_unstale":"2024-02-21T23:07:30.085471+0000","last_undegraded":"2024-02-21T23:07:30.085471+0000","last_fullsized":"2024-02-21T23:07:30.085471+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:20:44.474084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.31","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.083458+0000","last_change":"2024-02-21T23:07:30.075424+0000","last_active":"2024-02-21T23:07:30.083458+0000","last_peered":"2024-02-21T23:07:30.083458+0000","last_clean":"2024-02-21T23:07:30.083458+0000","last_became_active":"2024-02-21T23:07:30.074499+0000","last_became_peered":"2024-02-21T23:07:30.074499+0000","last_unstale":"2024-02-21T23:07:30.083458+0000","last_undegraded":"2024-02-21T23:07:30.083458+0000","last_fullsized":"2024-02-21T23:07:30.083458+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:00:05.154952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.30","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.063474+0000","last_change":"2024-02-21T23:07:30.059165+0000","last_active":"2024-02-21T23:07:30.063474+0000","last_peered":"2024-02-21T23:07:30.063474+0000","last_clean":"2024-02-21T23:07:30.063474+0000","last_became_active":"2024-02-21T23:07:30.058591+0000","last_became_peered":"2024-02-21T23:07:30.058591+0000","last_unstale":"2024-02-21T23:07:30.063474+0000","last_undegraded":"2024-02-21T23:07:30.063474+0000","last_fullsized":"2024-02-21T23:07:30.063474+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:12:22.139208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.2f","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.075714+0000","last_change":"2024-02-21T23:07:30.071924+0000","last_active":"2024-02-21T23:07:30.075714+0000","last_peered":"2024-02-21T23:07:30.075714+0000","last_clean":"2024-02-21T23:07:30.075714+0000","last_became_active":"2024-02-21T23:07:30.071641+0000","last_became_peered":"2024-02-21T23:07:30.071641+0000","last_unstale":"2024-02-21T23:07:30.075714+0000","last_undegraded":"2024-02-21T23:07:30.075714+0000","last_fullsized":"2024-02-21T23:07:30.075714+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:33:50.871582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.2e","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.085697+0000","last_change":"2024-02-21T23:07:30.081049+0000","last_active":"2024-02-21T23:07:30.085697+0000","last_peered":"2024-02-21T23:07:30.085697+0000","last_clean":"2024-02-21T23:07:30.085697+0000","last_became_active":"2024-02-21T23:07:30.080402+0000","last_became_peered":"2024-02-21T23:07:30.080402+0000","last_unstale":"2024-02-21T23:07:30.085697+0000","last_undegraded":"2024-02-21T23:07:30.085697+0000","last_fullsized":"2024-02-21T23:07:30.085697+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:19:35.573219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.2d","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.085625+0000","last_change":"2024-02-21T23:07:30.079530+0000","last_active":"2024-02-21T23:07:30.085625+0000","last_peered":"2024-02-21T23:07:30.085625+0000","last_clean":"2024-02-21T23:07:30.085625+0000","last_became_active":"2024-02-21T23:07:30.078744+0000","last_became_peered":"2024-02-21T23:07:30.078744+0000","last_unstale":"2024-02-21T23:07:30.085625+0000","last_undegraded":"2024-02-21T23:07:30.085625+0000","last_fullsized":"2024-02-21T23:07:30.085625+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:47:16.637877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.2c","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.074197+0000","last_change":"2024-02-21T23:07:30.066512+0000","last_active":"2024-02-21T23:07:30.074197+0000","last_peered":"2024-02-21T23:07:30.074197+0000","last_clean":"2024-02-21T23:07:30.074197+0000","last_became_active":"2024-02-21T23:07:30.065427+0000","last_became_peered":"2024-02-21T23:07:30.065427+0000","last_unstale":"2024-02-21T23:07:30.074197+0000","last_undegraded":"2024-02-21T23:07:30.074197+0000","last_fullsized":"2024-02-21T23:07:30.074197+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:37:41.104430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.2b","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.073406+0000","last_change":"2024-02-21T23:07:30.068288+0000","last_active":"2024-02-21T23:07:30.073406+0000","last_peered":"2024-02-21T23:07:30.073406+0000","last_clean":"2024-02-21T23:07:30.073406+0000","last_became_active":"2024-02-21T23:07:30.067970+0000","last_became_peered":"2024-02-21T23:07:30.067970+0000","last_unstale":"2024-02-21T23:07:30.073406+0000","last_undegraded":"2024-02-21T23:07:30.073406+0000","last_fullsized":"2024-02-21T23:07:30.073406+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:08:18.112813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.2a","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.072986+0000","last_change":"2024-02-21T23:07:30.064417+0000","last_active":"2024-02-21T23:07:30.072986+0000","last_peered":"2024-02-21T23:07:30.072986+0000","last_clean":"2024-02-21T23:07:30.072986+0000","last_became_active":"2024-02-21T23:07:30.064128+0000","last_became_peered":"2024-02-21T23:07:30.064128+0000","last_unstale":"2024-02-21T23:07:30.072986+0000","last_undegraded":"2024-02-21T23:07:30.072986+0000","last_fullsized":"2024-02-21T23:07:30.072986+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:32:40.636665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.29","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.074715+0000","last_change":"2024-02-21T23:07:30.072358+0000","last_active":"2024-02-21T23:07:30.074715+0000","last_peered":"2024-02-21T23:07:30.074715+0000","last_clean":"2024-02-21T23:07:30.074715+0000","last_became_active":"2024-02-21T23:07:30.069662+0000","last_became_peered":"2024-02-21T23:07:30.069662+0000","last_unstale":"2024-02-21T23:07:30.074715+0000","last_undegraded":"2024-02-21T23:07:30.074715+0000","last_fullsized":"2024-02-21T23:07:30.074715+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:29:03.312549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.28","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.087628+0000","last_change":"2024-02-21T23:07:30.082380+0000","last_active":"2024-02-21T23:07:30.087628+0000","last_peered":"2024-02-21T23:07:30.087628+0000","last_clean":"2024-02-21T23:07:30.087628+0000","last_became_active":"2024-02-21T23:07:30.080196+0000","last_became_peered":"2024-02-21T23:07:30.080196+0000","last_unstale":"2024-02-21T23:07:30.087628+0000","last_undegraded":"2024-02-21T23:07:30.087628+0000","last_fullsized":"2024-02-21T23:07:30.087628+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:29:03.312549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.27","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.087443+0000","last_change":"2024-02-21T23:07:30.077414+0000","last_active":"2024-02-21T23:07:30.087443+0000","last_peered":"2024-02-21T23:07:30.087443+0000","last_clean":"2024-02-21T23:07:30.087443+0000","last_became_active":"2024-02-21T23:07:30.074579+0000","last_became_peered":"2024-02-21T23:07:30.074579+0000","last_unstale":"2024-02-21T23:07:30.087443+0000","last_undegraded":"2024-02-21T23:07:30.087443+0000","last_fullsized":"2024-02-21T23:07:30.087443+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:00:05.154952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.26","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.075294+0000","last_change":"2024-02-21T23:07:30.060596+0000","last_active":"2024-02-21T23:07:30.075294+0000","last_peered":"2024-02-21T23:07:30.075294+0000","last_clean":"2024-02-21T23:07:30.075294+0000","last_became_active":"2024-02-21T23:07:30.060206+0000","last_became_peered":"2024-02-21T23:07:30.060206+0000","last_unstale":"2024-02-21T23:07:30.075294+0000","last_undegraded":"2024-02-21T23:07:30.075294+0000","last_fullsized":"2024-02-21T23:07:30.075294+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:01:27.333779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.25","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.085576+0000","last_change":"2024-02-21T23:07:30.078812+0000","last_active":"2024-02-21T23:07:30.085576+0000","last_peered":"2024-02-21T23:07:30.085576+0000","last_clean":"2024-02-21T23:07:30.085576+0000","last_became_active":"2024-02-21T23:07:30.078514+0000","last_became_peered":"2024-02-21T23:07:30.078514+0000","last_unstale":"2024-02-21T23:07:30.085576+0000","last_undegraded":"2024-02-21T23:07:30.085576+0000","last_fullsized":"2024-02-21T23:07:30.085576+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:28:57.130238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.24","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.076157+0000","last_change":"2024-02-21T23:07:30.070416+0000","last_active":"2024-02-21T23:07:30.076157+0000","last_peered":"2024-02-21T23:07:30.076157+0000","last_clean":"2024-02-21T23:07:30.076157+0000","last_became_active":"2024-02-21T23:07:30.070100+0000","last_became_peered":"2024-02-21T23:07:30.070100+0000","last_unstale":"2024-02-21T23:07:30.076157+0000","last_undegraded":"2024-02-21T23:07:30.076157+0000","last_fullsized":"2024-02-21T23:07:30.076157+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:46:25.120139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.23","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.072817+0000","last_change":"2024-02-21T23:07:30.064187+0000","last_active":"2024-02-21T23:07:30.072817+0000","last_peered":"2024-02-21T23:07:30.072817+0000","last_clean":"2024-02-21T23:07:30.072817+0000","last_became_active":"2024-02-21T23:07:30.063867+0000","last_became_peered":"2024-02-21T23:07:30.063867+0000","last_unstale":"2024-02-21T23:07:30.072817+0000","last_undegraded":"2024-02-21T23:07:30.072817+0000","last_fullsized":"2024-02-21T23:07:30.072817+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:19:45.514897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.22","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.062646+0000","last_change":"2024-02-21T23:07:30.057638+0000","last_active":"2024-02-21T23:07:30.062646+0000","last_peered":"2024-02-21T23:07:30.062646+0000","last_clean":"2024-02-21T23:07:30.062646+0000","last_became_active":"2024-02-21T23:07:30.057299+0000","last_became_peered":"2024-02-21T23:07:30.057299+0000","last_unstale":"2024-02-21T23:07:30.062646+0000","last_undegraded":"2024-02-21T23:07:30.062646+0000","last_fullsized":"2024-02-21T23:07:30.062646+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:33:48.672687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.21","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.087397+0000","last_change":"2024-02-21T23:07:30.076868+0000","last_active":"2024-02-21T23:07:30.087397+0000","last_peered":"2024-02-21T23:07:30.087397+0000","last_clean":"2024-02-21T23:07:30.087397+0000","last_became_active":"2024-02-21T23:07:30.076599+0000","last_became_peered":"2024-02-21T23:07:30.076599+0000","last_unstale":"2024-02-21T23:07:30.087397+0000","last_undegraded":"2024-02-21T23:07:30.087397+0000","last_fullsized":"2024-02-21T23:07:30.087397+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:00:31.053859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.20","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.087607+0000","last_change":"2024-02-21T23:07:30.082200+0000","last_active":"2024-02-21T23:07:30.087607+0000","last_peered":"2024-02-21T23:07:30.087607+0000","last_clean":"2024-02-21T23:07:30.087607+0000","last_became_active":"2024-02-21T23:07:30.079831+0000","last_became_peered":"2024-02-21T23:07:30.079831+0000","last_unstale":"2024-02-21T23:07:30.087607+0000","last_undegraded":"2024-02-21T23:07:30.087607+0000","last_fullsized":"2024-02-21T23:07:30.087607+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:32:44.283700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.075736+0000","last_change":"2024-02-21T23:07:30.067693+0000","last_active":"2024-02-21T23:07:30.075736+0000","last_peered":"2024-02-21T23:07:30.075736+0000","last_clean":"2024-02-21T23:07:30.075736+0000","last_became_active":"2024-02-21T23:07:30.067386+0000","last_became_peered":"2024-02-21T23:07:30.067386+0000","last_unstale":"2024-02-21T23:07:30.075736+0000","last_undegraded":"2024-02-21T23:07:30.075736+0000","last_fullsized":"2024-02-21T23:07:30.075736+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:29:57.959277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1f","version":"109'4","reported_seq":16,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.088644+0000","last_change":"2024-02-21T23:07:29.078080+0000","last_active":"2024-02-21T23:07:30.088644+0000","last_peered":"2024-02-21T23:07:30.088644+0000","last_clean":"2024-02-21T23:07:30.088644+0000","last_became_active":"2024-02-21T23:07:29.077684+0000","last_became_peered":"2024-02-21T23:07:29.077684+0000","last_unstale":"2024-02-21T23:07:30.088644+0000","last_undegraded":"2024-02-21T23:07:30.088644+0000","last_fullsized":"2024-02-21T23:07:30.088644+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_clean_scrub_stamp":"2024-02-21T23:07:28.006668+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:41:20.161387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.063001+0000","last_change":"2024-02-21T23:07:30.054594+0000","last_active":"2024-02-21T23:07:30.063001+0000","last_peered":"2024-02-21T23:07:30.063001+0000","last_clean":"2024-02-21T23:07:30.063001+0000","last_became_active":"2024-02-21T23:07:30.054236+0000","last_became_peered":"2024-02-21T23:07:30.054236+0000","last_unstale":"2024-02-21T23:07:30.063001+0000","last_undegraded":"2024-02-21T23:07:30.063001+0000","last_fullsized":"2024-02-21T23:07:30.063001+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:43:45.497542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1e","version":"109'1","reported_seq":13,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.081728+0000","last_change":"2024-02-21T23:07:29.066858+0000","last_active":"2024-02-21T23:07:30.081728+0000","last_peered":"2024-02-21T23:07:30.081728+0000","last_clean":"2024-02-21T23:07:30.081728+0000","last_became_active":"2024-02-21T23:07:29.066573+0000","last_became_peered":"2024-02-21T23:07:29.066573+0000","last_unstale":"2024-02-21T23:07:30.081728+0000","last_undegraded":"2024-02-21T23:07:30.081728+0000","last_fullsized":"2024-02-21T23:07:30.081728+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_clean_scrub_stamp":"2024-02-21T23:07:28.006668+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:55:03.572716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.073752+0000","last_change":"2024-02-21T23:07:30.069818+0000","last_active":"2024-02-21T23:07:30.073752+0000","last_peered":"2024-02-21T23:07:30.073752+0000","last_clean":"2024-02-21T23:07:30.073752+0000","last_became_active":"2024-02-21T23:07:30.069518+0000","last_became_peered":"2024-02-21T23:07:30.069518+0000","last_unstale":"2024-02-21T23:07:30.073752+0000","last_undegraded":"2024-02-21T23:07:30.073752+0000","last_fullsized":"2024-02-21T23:07:30.073752+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:14:32.130423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"109'2","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.083205+0000","last_change":"2024-02-21T23:07:29.075790+0000","last_active":"2024-02-21T23:07:30.083205+0000","last_peered":"2024-02-21T23:07:30.083205+0000","last_clean":"2024-02-21T23:07:30.083205+0000","last_became_active":"2024-02-21T23:07:29.074272+0000","last_became_peered":"2024-02-21T23:07:29.074272+0000","last_unstale":"2024-02-21T23:07:30.083205+0000","last_undegraded":"2024-02-21T23:07:30.083205+0000","last_fullsized":"2024-02-21T23:07:30.083205+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_clean_scrub_stamp":"2024-02-21T23:07:28.006668+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:43:51.259956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.071688+0000","last_change":"2024-02-21T23:07:30.065056+0000","last_active":"2024-02-21T23:07:30.071688+0000","last_peered":"2024-02-21T23:07:30.071688+0000","last_clean":"2024-02-21T23:07:30.071688+0000","last_became_active":"2024-02-21T23:07:30.064694+0000","last_became_peered":"2024-02-21T23:07:30.064694+0000","last_unstale":"2024-02-21T23:07:30.071688+0000","last_undegraded":"2024-02-21T23:07:30.071688+0000","last_fullsized":"2024-02-21T23:07:30.071688+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:00:05.154952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"109'2","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.085807+0000","last_change":"2024-02-21T23:07:29.059998+0000","last_active":"2024-02-21T23:07:30.085807+0000","last_peered":"2024-02-21T23:07:30.085807+0000","last_clean":"2024-02-21T23:07:30.085807+0000","last_became_active":"2024-02-21T23:07:29.059719+0000","last_became_peered":"2024-02-21T23:07:29.059719+0000","last_unstale":"2024-02-21T23:07:30.085807+0000","last_undegraded":"2024-02-21T23:07:30.085807+0000","last_fullsized":"2024-02-21T23:07:30.085807+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_clean_scrub_stamp":"2024-02-21T23:07:28.006668+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:45:53.025498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.075798+0000","last_change":"2024-02-21T23:07:30.073742+0000","last_active":"2024-02-21T23:07:30.075798+0000","last_peered":"2024-02-21T23:07:30.075798+0000","last_clean":"2024-02-21T23:07:30.075798+0000","last_became_active":"2024-02-21T23:07:30.073456+0000","last_became_peered":"2024-02-21T23:07:30.073456+0000","last_unstale":"2024-02-21T23:07:30.075798+0000","last_undegraded":"2024-02-21T23:07:30.075798+0000","last_fullsized":"2024-02-21T23:07:30.075798+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:43:52.275165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1b","version":"109'1","reported_seq":13,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.084429+0000","last_change":"2024-02-21T23:07:29.064738+0000","last_active":"2024-02-21T23:07:30.084429+0000","last_peered":"2024-02-21T23:07:30.084429+0000","last_clean":"2024-02-21T23:07:30.084429+0000","last_became_active":"2024-02-21T23:07:29.064453+0000","last_became_peered":"2024-02-21T23:07:29.064453+0000","last_unstale":"2024-02-21T23:07:30.084429+0000","last_undegraded":"2024-02-21T23:07:30.084429+0000","last_fullsized":"2024-02-21T23:07:30.084429+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_clean_scrub_stamp":"2024-02-21T23:07:28.006668+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:43:51.259956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.080744+0000","last_change":"2024-02-21T23:07:30.068959+0000","last_active":"2024-02-21T23:07:30.080744+0000","last_peered":"2024-02-21T23:07:30.080744+0000","last_clean":"2024-02-21T23:07:30.080744+0000","last_became_active":"2024-02-21T23:07:30.068604+0000","last_became_peered":"2024-02-21T23:07:30.068604+0000","last_unstale":"2024-02-21T23:07:30.080744+0000","last_undegraded":"2024-02-21T23:07:30.080744+0000","last_fullsized":"2024-02-21T23:07:30.080744+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:14:32.130423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"109'1","reported_seq":13,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.086251+0000","last_change":"2024-02-21T23:07:29.069504+0000","last_active":"2024-02-21T23:07:30.086251+0000","last_peered":"2024-02-21T23:07:30.086251+0000","last_clean":"2024-02-21T23:07:30.086251+0000","last_became_active":"2024-02-21T23:07:29.069197+0000","last_became_peered":"2024-02-21T23:07:29.069197+0000","last_unstale":"2024-02-21T23:07:30.086251+0000","last_undegraded":"2024-02-21T23:07:30.086251+0000","last_fullsized":"2024-02-21T23:07:30.086251+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_clean_scrub_stamp":"2024-02-21T23:07:28.006668+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:56:55.710409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.077191+0000","last_change":"2024-02-21T23:07:30.063377+0000","last_active":"2024-02-21T23:07:30.077191+0000","last_peered":"2024-02-21T23:07:30.077191+0000","last_clean":"2024-02-21T23:07:30.077191+0000","last_became_active":"2024-02-21T23:07:30.062671+0000","last_became_peered":"2024-02-21T23:07:30.062671+0000","last_unstale":"2024-02-21T23:07:30.077191+0000","last_undegraded":"2024-02-21T23:07:30.077191+0000","last_fullsized":"2024-02-21T23:07:30.077191+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:14:56.427299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.063169+0000","last_change":"2024-02-21T23:07:29.067526+0000","last_active":"2024-02-21T23:07:30.063169+0000","last_peered":"2024-02-21T23:07:30.063169+0000","last_clean":"2024-02-21T23:07:30.063169+0000","last_became_active":"2024-02-21T23:07:29.067188+0000","last_became_peered":"2024-02-21T23:07:29.067188+0000","last_unstale":"2024-02-21T23:07:30.063169+0000","last_undegraded":"2024-02-21T23:07:30.063169+0000","last_fullsized":"2024-02-21T23:07:30.063169+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_clean_scrub_stamp":"2024-02-21T23:07:28.006668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:43:51.259956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.066747+0000","last_change":"2024-02-21T23:07:30.065429+0000","last_active":"2024-02-21T23:07:30.066747+0000","last_peered":"2024-02-21T23:07:30.066747+0000","last_clean":"2024-02-21T23:07:30.066747+0000","last_became_active":"2024-02-21T23:07:30.064289+0000","last_became_peered":"2024-02-21T23:07:30.064289+0000","last_unstale":"2024-02-21T23:07:30.066747+0000","last_undegraded":"2024-02-21T23:07:30.066747+0000","last_fullsized":"2024-02-21T23:07:30.066747+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:21:34.378582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.037128+0000","last_change":"2024-02-21T23:07:29.066688+0000","last_active":"2024-02-21T23:07:30.037128+0000","last_peered":"2024-02-21T23:07:30.037128+0000","last_clean":"2024-02-21T23:07:30.037128+0000","last_became_active":"2024-02-21T23:07:29.065319+0000","last_became_peered":"2024-02-21T23:07:29.065319+0000","last_unstale":"2024-02-21T23:07:30.037128+0000","last_undegraded":"2024-02-21T23:07:30.037128+0000","last_fullsized":"2024-02-21T23:07:30.037128+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_clean_scrub_stamp":"2024-02-21T23:07:28.006668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:49:57.742371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.083462+0000","last_change":"2024-02-21T23:07:30.075112+0000","last_active":"2024-02-21T23:07:30.083462+0000","last_peered":"2024-02-21T23:07:30.083462+0000","last_clean":"2024-02-21T23:07:30.083462+0000","last_became_active":"2024-02-21T23:07:30.074791+0000","last_became_peered":"2024-02-21T23:07:30.074791+0000","last_unstale":"2024-02-21T23:07:30.083462+0000","last_undegraded":"2024-02-21T23:07:30.083462+0000","last_fullsized":"2024-02-21T23:07:30.083462+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:28:34.875509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.071071+0000","last_change":"2024-02-21T23:07:29.072136+0000","last_active":"2024-02-21T23:07:30.071071+0000","last_peered":"2024-02-21T23:07:30.071071+0000","last_clean":"2024-02-21T23:07:30.071071+0000","last_became_active":"2024-02-21T23:07:29.071591+0000","last_became_peered":"2024-02-21T23:07:29.071591+0000","last_unstale":"2024-02-21T23:07:30.071071+0000","last_undegraded":"2024-02-21T23:07:30.071071+0000","last_fullsized":"2024-02-21T23:07:30.071071+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_clean_scrub_stamp":"2024-02-21T23:07:28.006668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:43:51.259956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.073691+0000","last_change":"2024-02-21T23:07:30.068804+0000","last_active":"2024-02-21T23:07:30.073691+0000","last_peered":"2024-02-21T23:07:30.073691+0000","last_clean":"2024-02-21T23:07:30.073691+0000","last_became_active":"2024-02-21T23:07:30.068459+0000","last_became_peered":"2024-02-21T23:07:30.068459+0000","last_unstale":"2024-02-21T23:07:30.073691+0000","last_undegraded":"2024-02-21T23:07:30.073691+0000","last_fullsized":"2024-02-21T23:07:30.073691+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:32:40.636665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.065341+0000","last_change":"2024-02-21T23:07:29.071010+0000","last_active":"2024-02-21T23:07:30.065341+0000","last_peered":"2024-02-21T23:07:30.065341+0000","last_clean":"2024-02-21T23:07:30.065341+0000","last_became_active":"2024-02-21T23:07:29.070516+0000","last_became_peered":"2024-02-21T23:07:29.070516+0000","last_unstale":"2024-02-21T23:07:30.065341+0000","last_undegraded":"2024-02-21T23:07:30.065341+0000","last_fullsized":"2024-02-21T23:07:30.065341+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_clean_scrub_stamp":"2024-02-21T23:07:28.006668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:28:33.860300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.085321+0000","last_change":"2024-02-21T23:07:30.074543+0000","last_active":"2024-02-21T23:07:30.085321+0000","last_peered":"2024-02-21T23:07:30.085321+0000","last_clean":"2024-02-21T23:07:30.085321+0000","last_became_active":"2024-02-21T23:07:30.074227+0000","last_became_peered":"2024-02-21T23:07:30.074227+0000","last_unstale":"2024-02-21T23:07:30.085321+0000","last_undegraded":"2024-02-21T23:07:30.085321+0000","last_fullsized":"2024-02-21T23:07:30.085321+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:21:53.631128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.15","version":"109'3","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.088810+0000","last_change":"2024-02-21T23:07:29.073372+0000","last_active":"2024-02-21T23:07:30.088810+0000","last_peered":"2024-02-21T23:07:30.088810+0000","last_clean":"2024-02-21T23:07:30.088810+0000","last_became_active":"2024-02-21T23:07:29.073050+0000","last_became_peered":"2024-02-21T23:07:29.073050+0000","last_unstale":"2024-02-21T23:07:30.088810+0000","last_undegraded":"2024-02-21T23:07:30.088810+0000","last_fullsized":"2024-02-21T23:07:30.088810+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_clean_scrub_stamp":"2024-02-21T23:07:28.006668+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:28:56.115029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.087671+0000","last_change":"2024-02-21T23:07:30.078266+0000","last_active":"2024-02-21T23:07:30.087671+0000","last_peered":"2024-02-21T23:07:30.087671+0000","last_clean":"2024-02-21T23:07:30.087671+0000","last_became_active":"2024-02-21T23:07:30.077743+0000","last_became_peered":"2024-02-21T23:07:30.077743+0000","last_unstale":"2024-02-21T23:07:30.087671+0000","last_undegraded":"2024-02-21T23:07:30.087671+0000","last_fullsized":"2024-02-21T23:07:30.087671+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:08:18.112813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.14","version":"109'6","reported_seq":18,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.095228+0000","last_change":"2024-02-21T23:07:29.071931+0000","last_active":"2024-02-21T23:07:30.095228+0000","last_peered":"2024-02-21T23:07:30.095228+0000","last_clean":"2024-02-21T23:07:30.095228+0000","last_became_active":"2024-02-21T23:07:29.070378+0000","last_became_peered":"2024-02-21T23:07:29.070378+0000","last_unstale":"2024-02-21T23:07:30.095228+0000","last_undegraded":"2024-02-21T23:07:30.095228+0000","last_fullsized":"2024-02-21T23:07:30.095228+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_clean_scrub_stamp":"2024-02-21T23:07:28.006668+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:32:43.268491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.088454+0000","last_change":"2024-02-21T23:07:30.083594+0000","last_active":"2024-02-21T23:07:30.088454+0000","last_peered":"2024-02-21T23:07:30.088454+0000","last_clean":"2024-02-21T23:07:30.088454+0000","last_became_active":"2024-02-21T23:07:30.082142+0000","last_became_peered":"2024-02-21T23:07:30.082142+0000","last_unstale":"2024-02-21T23:07:30.088454+0000","last_undegraded":"2024-02-21T23:07:30.088454+0000","last_fullsized":"2024-02-21T23:07:30.088454+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:32:40.636665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.13","version":"109'2","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.093360+0000","last_change":"2024-02-21T23:07:29.069228+0000","last_active":"2024-02-21T23:07:30.093360+0000","last_peered":"2024-02-21T23:07:30.093360+0000","last_clean":"2024-02-21T23:07:30.093360+0000","last_became_active":"2024-02-21T23:07:29.068925+0000","last_became_peered":"2024-02-21T23:07:29.068925+0000","last_unstale":"2024-02-21T23:07:30.093360+0000","last_undegraded":"2024-02-21T23:07:30.093360+0000","last_fullsized":"2024-02-21T23:07:30.093360+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_clean_scrub_stamp":"2024-02-21T23:07:28.006668+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:36:25.471297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.062950+0000","last_change":"2024-02-21T23:07:30.060999+0000","last_active":"2024-02-21T23:07:30.062950+0000","last_peered":"2024-02-21T23:07:30.062950+0000","last_clean":"2024-02-21T23:07:30.062950+0000","last_became_active":"2024-02-21T23:07:30.059434+0000","last_became_peered":"2024-02-21T23:07:30.059434+0000","last_unstale":"2024-02-21T23:07:30.062950+0000","last_undegraded":"2024-02-21T23:07:30.062950+0000","last_fullsized":"2024-02-21T23:07:30.062950+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:18:29.440779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.046810+0000","last_change":"2024-02-21T23:07:29.074974+0000","last_active":"2024-02-21T23:07:30.046810+0000","last_peered":"2024-02-21T23:07:30.046810+0000","last_clean":"2024-02-21T23:07:30.046810+0000","last_became_active":"2024-02-21T23:07:29.074414+0000","last_became_peered":"2024-02-21T23:07:29.074414+0000","last_unstale":"2024-02-21T23:07:30.046810+0000","last_undegraded":"2024-02-21T23:07:30.046810+0000","last_fullsized":"2024-02-21T23:07:30.046810+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_clean_scrub_stamp":"2024-02-21T23:07:28.006668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:42:56.912661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.077411+0000","last_change":"2024-02-21T23:07:30.068424+0000","last_active":"2024-02-21T23:07:30.077411+0000","last_peered":"2024-02-21T23:07:30.077411+0000","last_clean":"2024-02-21T23:07:30.077411+0000","last_became_active":"2024-02-21T23:07:30.068076+0000","last_became_peered":"2024-02-21T23:07:30.068076+0000","last_unstale":"2024-02-21T23:07:30.077411+0000","last_undegraded":"2024-02-21T23:07:30.077411+0000","last_fullsized":"2024-02-21T23:07:30.077411+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:36:22.445832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.043805+0000","last_change":"2024-02-21T23:07:29.071783+0000","last_active":"2024-02-21T23:07:30.043805+0000","last_peered":"2024-02-21T23:07:30.043805+0000","last_clean":"2024-02-21T23:07:30.043805+0000","last_became_active":"2024-02-21T23:07:29.070845+0000","last_became_peered":"2024-02-21T23:07:29.070845+0000","last_unstale":"2024-02-21T23:07:30.043805+0000","last_undegraded":"2024-02-21T23:07:30.043805+0000","last_fullsized":"2024-02-21T23:07:30.043805+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_clean_scrub_stamp":"2024-02-21T23:07:28.006668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:29:02.297339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.073538+0000","last_change":"2024-02-21T23:07:30.068242+0000","last_active":"2024-02-21T23:07:30.073538+0000","last_peered":"2024-02-21T23:07:30.073538+0000","last_clean":"2024-02-21T23:07:30.073538+0000","last_became_active":"2024-02-21T23:07:30.067823+0000","last_became_peered":"2024-02-21T23:07:30.067823+0000","last_unstale":"2024-02-21T23:07:30.073538+0000","last_undegraded":"2024-02-21T23:07:30.073538+0000","last_fullsized":"2024-02-21T23:07:30.073538+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:47:16.637877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.f","version":"109'1","reported_seq":13,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.085123+0000","last_change":"2024-02-21T23:07:29.067956+0000","last_active":"2024-02-21T23:07:30.085123+0000","last_peered":"2024-02-21T23:07:30.085123+0000","last_clean":"2024-02-21T23:07:30.085123+0000","last_became_active":"2024-02-21T23:07:29.067670+0000","last_became_peered":"2024-02-21T23:07:29.067670+0000","last_unstale":"2024-02-21T23:07:30.085123+0000","last_undegraded":"2024-02-21T23:07:30.085123+0000","last_fullsized":"2024-02-21T23:07:30.085123+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_clean_scrub_stamp":"2024-02-21T23:07:28.006668+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:55:11.484777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.065666+0000","last_change":"2024-02-21T23:07:30.057744+0000","last_active":"2024-02-21T23:07:30.065666+0000","last_peered":"2024-02-21T23:07:30.065666+0000","last_clean":"2024-02-21T23:07:30.065666+0000","last_became_active":"2024-02-21T23:07:30.053562+0000","last_became_peered":"2024-02-21T23:07:30.053562+0000","last_unstale":"2024-02-21T23:07:30.065666+0000","last_undegraded":"2024-02-21T23:07:30.065666+0000","last_fullsized":"2024-02-21T23:07:30.065666+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:28:57.130238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.e","version":"109'2","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.081153+0000","last_change":"2024-02-21T23:07:29.066586+0000","last_active":"2024-02-21T23:07:30.081153+0000","last_peered":"2024-02-21T23:07:30.081153+0000","last_clean":"2024-02-21T23:07:30.081153+0000","last_became_active":"2024-02-21T23:07:29.066268+0000","last_became_peered":"2024-02-21T23:07:29.066268+0000","last_unstale":"2024-02-21T23:07:30.081153+0000","last_undegraded":"2024-02-21T23:07:30.081153+0000","last_fullsized":"2024-02-21T23:07:30.081153+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_clean_scrub_stamp":"2024-02-21T23:07:28.006668+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:42:13.465978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.077659+0000","last_change":"2024-02-21T23:07:30.074074+0000","last_active":"2024-02-21T23:07:30.077659+0000","last_peered":"2024-02-21T23:07:30.077659+0000","last_clean":"2024-02-21T23:07:30.077659+0000","last_became_active":"2024-02-21T23:07:30.073581+0000","last_became_peered":"2024-02-21T23:07:30.073581+0000","last_unstale":"2024-02-21T23:07:30.077659+0000","last_undegraded":"2024-02-21T23:07:30.077659+0000","last_fullsized":"2024-02-21T23:07:30.077659+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:15:22.969452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.d","version":"109'1","reported_seq":13,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.088883+0000","last_change":"2024-02-21T23:07:29.076731+0000","last_active":"2024-02-21T23:07:30.088883+0000","last_peered":"2024-02-21T23:07:30.088883+0000","last_clean":"2024-02-21T23:07:30.088883+0000","last_became_active":"2024-02-21T23:07:29.076536+0000","last_became_peered":"2024-02-21T23:07:29.076536+0000","last_unstale":"2024-02-21T23:07:30.088883+0000","last_undegraded":"2024-02-21T23:07:30.088883+0000","last_fullsized":"2024-02-21T23:07:30.088883+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_clean_scrub_stamp":"2024-02-21T23:07:28.006668+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:47:15.622668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.087472+0000","last_change":"2024-02-21T23:07:30.078047+0000","last_active":"2024-02-21T23:07:30.087472+0000","last_peered":"2024-02-21T23:07:30.087472+0000","last_clean":"2024-02-21T23:07:30.087472+0000","last_became_active":"2024-02-21T23:07:30.074058+0000","last_became_peered":"2024-02-21T23:07:30.074058+0000","last_unstale":"2024-02-21T23:07:30.087472+0000","last_undegraded":"2024-02-21T23:07:30.087472+0000","last_fullsized":"2024-02-21T23:07:30.087472+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:48:05.594176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.056851+0000","last_change":"2024-02-21T23:07:29.074595+0000","last_active":"2024-02-21T23:07:30.056851+0000","last_peered":"2024-02-21T23:07:30.056851+0000","last_clean":"2024-02-21T23:07:30.056851+0000","last_became_active":"2024-02-21T23:07:29.074070+0000","last_became_peered":"2024-02-21T23:07:29.074070+0000","last_unstale":"2024-02-21T23:07:30.056851+0000","last_undegraded":"2024-02-21T23:07:30.056851+0000","last_fullsized":"2024-02-21T23:07:30.056851+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_clean_scrub_stamp":"2024-02-21T23:07:28.006668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:14:31.115214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.084461+0000","last_change":"2024-02-21T23:07:30.081454+0000","last_active":"2024-02-21T23:07:30.084461+0000","last_peered":"2024-02-21T23:07:30.084461+0000","last_clean":"2024-02-21T23:07:30.084461+0000","last_became_active":"2024-02-21T23:07:30.081149+0000","last_became_peered":"2024-02-21T23:07:30.081149+0000","last_unstale":"2024-02-21T23:07:30.084461+0000","last_undegraded":"2024-02-21T23:07:30.084461+0000","last_fullsized":"2024-02-21T23:07:30.084461+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:01:21.611573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"109'2","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.088573+0000","last_change":"2024-02-21T23:07:29.071631+0000","last_active":"2024-02-21T23:07:30.088573+0000","last_peered":"2024-02-21T23:07:30.088573+0000","last_clean":"2024-02-21T23:07:30.088573+0000","last_became_active":"2024-02-21T23:07:29.071335+0000","last_became_peered":"2024-02-21T23:07:29.071335+0000","last_unstale":"2024-02-21T23:07:30.088573+0000","last_undegraded":"2024-02-21T23:07:30.088573+0000","last_fullsized":"2024-02-21T23:07:30.088573+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_clean_scrub_stamp":"2024-02-21T23:07:28.006668+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:26:56.660421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.081245+0000","last_change":"2024-02-21T23:07:30.069563+0000","last_active":"2024-02-21T23:07:30.081245+0000","last_peered":"2024-02-21T23:07:30.081245+0000","last_clean":"2024-02-21T23:07:30.081245+0000","last_became_active":"2024-02-21T23:07:30.069239+0000","last_became_peered":"2024-02-21T23:07:30.069239+0000","last_unstale":"2024-02-21T23:07:30.081245+0000","last_undegraded":"2024-02-21T23:07:30.081245+0000","last_fullsized":"2024-02-21T23:07:30.081245+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:20:59.588268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.052298+0000","last_change":"2024-02-21T23:07:29.070486+0000","last_active":"2024-02-21T23:07:30.052298+0000","last_peered":"2024-02-21T23:07:30.052298+0000","last_clean":"2024-02-21T23:07:30.052298+0000","last_became_active":"2024-02-21T23:07:29.070179+0000","last_became_peered":"2024-02-21T23:07:29.070179+0000","last_unstale":"2024-02-21T23:07:30.052298+0000","last_undegraded":"2024-02-21T23:07:30.052298+0000","last_fullsized":"2024-02-21T23:07:30.052298+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_clean_scrub_stamp":"2024-02-21T23:07:28.006668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:28:33.860300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.085451+0000","last_change":"2024-02-21T23:07:30.080250+0000","last_active":"2024-02-21T23:07:30.085451+0000","last_peered":"2024-02-21T23:07:30.085451+0000","last_clean":"2024-02-21T23:07:30.085451+0000","last_became_active":"2024-02-21T23:07:30.079915+0000","last_became_peered":"2024-02-21T23:07:30.079915+0000","last_unstale":"2024-02-21T23:07:30.085451+0000","last_undegraded":"2024-02-21T23:07:30.085451+0000","last_fullsized":"2024-02-21T23:07:30.085451+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:00:18.649979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.044147+0000","last_change":"2024-02-21T23:07:29.064409+0000","last_active":"2024-02-21T23:07:30.044147+0000","last_peered":"2024-02-21T23:07:30.044147+0000","last_clean":"2024-02-21T23:07:30.044147+0000","last_became_active":"2024-02-21T23:07:29.063935+0000","last_became_peered":"2024-02-21T23:07:29.063935+0000","last_unstale":"2024-02-21T23:07:30.044147+0000","last_undegraded":"2024-02-21T23:07:30.044147+0000","last_fullsized":"2024-02-21T23:07:30.044147+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_clean_scrub_stamp":"2024-02-21T23:07:28.006668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:43:47.978454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.087490+0000","last_change":"2024-02-21T23:07:30.079069+0000","last_active":"2024-02-21T23:07:30.087490+0000","last_peered":"2024-02-21T23:07:30.087490+0000","last_clean":"2024-02-21T23:07:30.087490+0000","last_became_active":"2024-02-21T23:07:30.076338+0000","last_became_peered":"2024-02-21T23:07:30.076338+0000","last_unstale":"2024-02-21T23:07:30.087490+0000","last_undegraded":"2024-02-21T23:07:30.087490+0000","last_fullsized":"2024-02-21T23:07:30.087490+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:36:26.486506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.052691+0000","last_change":"2024-02-21T23:07:29.074845+0000","last_active":"2024-02-21T23:07:30.052691+0000","last_peered":"2024-02-21T23:07:30.052691+0000","last_clean":"2024-02-21T23:07:30.052691+0000","last_became_active":"2024-02-21T23:07:29.074537+0000","last_became_peered":"2024-02-21T23:07:29.074537+0000","last_unstale":"2024-02-21T23:07:30.052691+0000","last_undegraded":"2024-02-21T23:07:30.052691+0000","last_fullsized":"2024-02-21T23:07:30.052691+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_clean_scrub_stamp":"2024-02-21T23:07:28.006668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:28:56.115029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.063640+0000","last_change":"2024-02-21T23:07:30.061637+0000","last_active":"2024-02-21T23:07:30.063640+0000","last_peered":"2024-02-21T23:07:30.063640+0000","last_clean":"2024-02-21T23:07:30.063640+0000","last_became_active":"2024-02-21T23:07:30.061326+0000","last_became_peered":"2024-02-21T23:07:30.061326+0000","last_unstale":"2024-02-21T23:07:30.063640+0000","last_undegraded":"2024-02-21T23:07:30.063640+0000","last_fullsized":"2024-02-21T23:07:30.063640+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:20:05.469348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.7","version":"109'2","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.084643+0000","last_change":"2024-02-21T23:07:29.069912+0000","last_active":"2024-02-21T23:07:30.084643+0000","last_peered":"2024-02-21T23:07:30.084643+0000","last_clean":"2024-02-21T23:07:30.084643+0000","last_became_active":"2024-02-21T23:07:29.069479+0000","last_became_peered":"2024-02-21T23:07:29.069479+0000","last_unstale":"2024-02-21T23:07:30.084643+0000","last_undegraded":"2024-02-21T23:07:30.084643+0000","last_fullsized":"2024-02-21T23:07:30.084643+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_clean_scrub_stamp":"2024-02-21T23:07:28.006668+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:14:47.017723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.086610+0000","last_change":"2024-02-21T23:07:30.079590+0000","last_active":"2024-02-21T23:07:30.086610+0000","last_peered":"2024-02-21T23:07:30.086610+0000","last_clean":"2024-02-21T23:07:30.086610+0000","last_became_active":"2024-02-21T23:07:30.078597+0000","last_became_peered":"2024-02-21T23:07:30.078597+0000","last_unstale":"2024-02-21T23:07:30.086610+0000","last_undegraded":"2024-02-21T23:07:30.086610+0000","last_fullsized":"2024-02-21T23:07:30.086610+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:28:34.875509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.070651+0000","last_change":"2024-02-21T23:07:29.071395+0000","last_active":"2024-02-21T23:07:30.070651+0000","last_peered":"2024-02-21T23:07:30.070651+0000","last_clean":"2024-02-21T23:07:30.070651+0000","last_became_active":"2024-02-21T23:07:29.071099+0000","last_became_peered":"2024-02-21T23:07:29.071099+0000","last_unstale":"2024-02-21T23:07:30.070651+0000","last_undegraded":"2024-02-21T23:07:30.070651+0000","last_fullsized":"2024-02-21T23:07:30.070651+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_clean_scrub_stamp":"2024-02-21T23:07:28.006668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:33:49.856373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.078707+0000","last_change":"2024-02-21T23:07:30.076179+0000","last_active":"2024-02-21T23:07:30.078707+0000","last_peered":"2024-02-21T23:07:30.078707+0000","last_clean":"2024-02-21T23:07:30.078707+0000","last_became_active":"2024-02-21T23:07:30.075869+0000","last_became_peered":"2024-02-21T23:07:30.075869+0000","last_unstale":"2024-02-21T23:07:30.078707+0000","last_undegraded":"2024-02-21T23:07:30.078707+0000","last_fullsized":"2024-02-21T23:07:30.078707+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:43:48.993663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.070708+0000","last_change":"2024-02-21T23:07:29.072468+0000","last_active":"2024-02-21T23:07:30.070708+0000","last_peered":"2024-02-21T23:07:30.070708+0000","last_clean":"2024-02-21T23:07:30.070708+0000","last_became_active":"2024-02-21T23:07:29.072165+0000","last_became_peered":"2024-02-21T23:07:29.072165+0000","last_unstale":"2024-02-21T23:07:30.070708+0000","last_undegraded":"2024-02-21T23:07:30.070708+0000","last_fullsized":"2024-02-21T23:07:30.070708+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_clean_scrub_stamp":"2024-02-21T23:07:28.006668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:00:09.026620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.078791+0000","last_change":"2024-02-21T23:07:30.072026+0000","last_active":"2024-02-21T23:07:30.078791+0000","last_peered":"2024-02-21T23:07:30.078791+0000","last_clean":"2024-02-21T23:07:30.078791+0000","last_became_active":"2024-02-21T23:07:30.071675+0000","last_became_peered":"2024-02-21T23:07:30.071675+0000","last_unstale":"2024-02-21T23:07:30.078791+0000","last_undegraded":"2024-02-21T23:07:30.078791+0000","last_fullsized":"2024-02-21T23:07:30.078791+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:17:30.425460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.039050+0000","last_change":"2024-02-21T23:07:29.055828+0000","last_active":"2024-02-21T23:07:30.039050+0000","last_peered":"2024-02-21T23:07:30.039050+0000","last_clean":"2024-02-21T23:07:30.039050+0000","last_became_active":"2024-02-21T23:07:29.054642+0000","last_became_peered":"2024-02-21T23:07:29.054642+0000","last_unstale":"2024-02-21T23:07:30.039050+0000","last_undegraded":"2024-02-21T23:07:30.039050+0000","last_fullsized":"2024-02-21T23:07:30.039050+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_clean_scrub_stamp":"2024-02-21T23:07:28.006668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:07:33.232700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.063449+0000","last_change":"2024-02-21T23:07:30.050940+0000","last_active":"2024-02-21T23:07:30.063449+0000","last_peered":"2024-02-21T23:07:30.063449+0000","last_clean":"2024-02-21T23:07:30.063449+0000","last_became_active":"2024-02-21T23:07:30.050426+0000","last_became_peered":"2024-02-21T23:07:30.050426+0000","last_unstale":"2024-02-21T23:07:30.063449+0000","last_undegraded":"2024-02-21T23:07:30.063449+0000","last_fullsized":"2024-02-21T23:07:30.063449+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:50:47.848516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.045644+0000","last_change":"2024-02-21T23:07:29.068231+0000","last_active":"2024-02-21T23:07:30.045644+0000","last_peered":"2024-02-21T23:07:30.045644+0000","last_clean":"2024-02-21T23:07:30.045644+0000","last_became_active":"2024-02-21T23:07:29.067904+0000","last_became_peered":"2024-02-21T23:07:29.067904+0000","last_unstale":"2024-02-21T23:07:30.045644+0000","last_undegraded":"2024-02-21T23:07:30.045644+0000","last_fullsized":"2024-02-21T23:07:30.045644+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_clean_scrub_stamp":"2024-02-21T23:07:28.006668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:28:33.860300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.085151+0000","last_change":"2024-02-21T23:07:30.074982+0000","last_active":"2024-02-21T23:07:30.085151+0000","last_peered":"2024-02-21T23:07:30.085151+0000","last_clean":"2024-02-21T23:07:30.085151+0000","last_became_active":"2024-02-21T23:07:30.074611+0000","last_became_peered":"2024-02-21T23:07:30.074611+0000","last_unstale":"2024-02-21T23:07:30.085151+0000","last_undegraded":"2024-02-21T23:07:30.085151+0000","last_fullsized":"2024-02-21T23:07:30.085151+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:33:00.995974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.043972+0000","last_change":"2024-02-21T23:07:29.073321+0000","last_active":"2024-02-21T23:07:30.043972+0000","last_peered":"2024-02-21T23:07:30.043972+0000","last_clean":"2024-02-21T23:07:30.043972+0000","last_became_active":"2024-02-21T23:07:29.072825+0000","last_became_peered":"2024-02-21T23:07:29.072825+0000","last_unstale":"2024-02-21T23:07:30.043972+0000","last_undegraded":"2024-02-21T23:07:30.043972+0000","last_fullsized":"2024-02-21T23:07:30.043972+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_clean_scrub_stamp":"2024-02-21T23:07:28.006668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:41:20.161387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.074751+0000","last_change":"2024-02-21T23:07:30.072238+0000","last_active":"2024-02-21T23:07:30.074751+0000","last_peered":"2024-02-21T23:07:30.074751+0000","last_clean":"2024-02-21T23:07:30.074751+0000","last_became_active":"2024-02-21T23:07:30.068670+0000","last_became_peered":"2024-02-21T23:07:30.068670+0000","last_unstale":"2024-02-21T23:07:30.074751+0000","last_undegraded":"2024-02-21T23:07:30.074751+0000","last_fullsized":"2024-02-21T23:07:30.074751+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:32:44.283700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.035590+0000","last_change":"2024-02-21T23:07:29.062336+0000","last_active":"2024-02-21T23:07:30.035590+0000","last_peered":"2024-02-21T23:07:30.035590+0000","last_clean":"2024-02-21T23:07:30.035590+0000","last_became_active":"2024-02-21T23:07:29.060554+0000","last_became_peered":"2024-02-21T23:07:29.060554+0000","last_unstale":"2024-02-21T23:07:30.035590+0000","last_undegraded":"2024-02-21T23:07:30.035590+0000","last_fullsized":"2024-02-21T23:07:30.035590+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_clean_scrub_stamp":"2024-02-21T23:07:28.006668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:22:10.884648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.075509+0000","last_change":"2024-02-21T23:07:30.072341+0000","last_active":"2024-02-21T23:07:30.075509+0000","last_peered":"2024-02-21T23:07:30.075509+0000","last_clean":"2024-02-21T23:07:30.075509+0000","last_became_active":"2024-02-21T23:07:30.071082+0000","last_became_peered":"2024-02-21T23:07:30.071082+0000","last_unstale":"2024-02-21T23:07:30.075509+0000","last_undegraded":"2024-02-21T23:07:30.075509+0000","last_fullsized":"2024-02-21T23:07:30.075509+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:00:10.041830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.053129+0000","last_change":"2024-02-21T23:07:29.072647+0000","last_active":"2024-02-21T23:07:30.053129+0000","last_peered":"2024-02-21T23:07:30.053129+0000","last_clean":"2024-02-21T23:07:30.053129+0000","last_became_active":"2024-02-21T23:07:29.072168+0000","last_became_peered":"2024-02-21T23:07:29.072168+0000","last_unstale":"2024-02-21T23:07:30.053129+0000","last_undegraded":"2024-02-21T23:07:30.053129+0000","last_fullsized":"2024-02-21T23:07:30.053129+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_clean_scrub_stamp":"2024-02-21T23:07:28.006668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:32:39.621456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.073381+0000","last_change":"2024-02-21T23:07:30.068176+0000","last_active":"2024-02-21T23:07:30.073381+0000","last_peered":"2024-02-21T23:07:30.073381+0000","last_clean":"2024-02-21T23:07:30.073381+0000","last_became_active":"2024-02-21T23:07:30.065279+0000","last_became_peered":"2024-02-21T23:07:30.065279+0000","last_unstale":"2024-02-21T23:07:30.073381+0000","last_undegraded":"2024-02-21T23:07:30.073381+0000","last_fullsized":"2024-02-21T23:07:30.073381+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:29.021877+0000","last_clean_scrub_stamp":"2024-02-21T23:07:29.021877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:48:05.594176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.10","version":"109'8","reported_seq":20,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.094056+0000","last_change":"2024-02-21T23:07:29.068221+0000","last_active":"2024-02-21T23:07:30.094056+0000","last_peered":"2024-02-21T23:07:30.094056+0000","last_clean":"2024-02-21T23:07:30.094056+0000","last_became_active":"2024-02-21T23:07:29.067873+0000","last_became_peered":"2024-02-21T23:07:29.067873+0000","last_unstale":"2024-02-21T23:07:30.094056+0000","last_undegraded":"2024-02-21T23:07:30.094056+0000","last_fullsized":"2024-02-21T23:07:30.094056+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:07:28.006668+0000","last_clean_scrub_stamp":"2024-02-21T23:07:28.006668+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:42:34.060965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":336,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-02-21T23:07:30.053120+0000","last_change":"2024-02-21T23:02:32.289335+0000","last_active":"2024-02-21T23:07:30.053120+0000","last_peered":"2024-02-21T23:07:30.053120+0000","last_clean":"2024-02-21T23:07:30.053120+0000","last_became_active":"2024-02-21T23:02:32.286913+0000","last_became_peered":"2024-02-21T23:02:32.286913+0000","last_unstale":"2024-02-21T23:07:30.053120+0000","last_undegraded":"2024-02-21T23:07:30.053120+0000","last_fullsized":"2024-02-21T23:07:30.053120+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_clean_scrub_stamp":"2024-02-21T23:02:31.172569+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:36:23.987374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":18,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574910,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32596,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339244,"statfs":{"total":96636764160,"available":96603385856,"internally_reserved":0,"allocated":1441792,"data_stored":1253567,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574911,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32500,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339340,"statfs":{"total":96636764160,"available":96603484160,"internally_reserved":0,"allocated":1347584,"data_stored":655256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574910,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32516,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339324,"statfs":{"total":96636764160,"available":96603467776,"internally_reserved":0,"allocated":1368064,"data_stored":662583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574911,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32524,"kb_used_data":1340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339316,"statfs":{"total":96636764160,"available":96603459584,"internally_reserved":0,"allocated":1372160,"data_stored":664035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574911,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32528,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339312,"statfs":{"total":96636764160,"available":96603455488,"internally_reserved":0,"allocated":1376256,"data_stored":664069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574912,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32512,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339328,"statfs":{"total":96636764160,"available":96603471872,"internally_reserved":0,"allocated":1359872,"data_stored":662423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574910,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32516,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339324,"statfs":{"total":96636764160,"available":96603467776,"internally_reserved":0,"allocated":1363968,"data_stored":662445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574910,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32580,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339260,"statfs":{"total":96636764160,"available":96603402240,"internally_reserved":0,"allocated":1429504,"data_stored":1252813,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-21T23:07:35.196 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-02-21T23:07:35.196 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-02-21T23:07:35.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.477+0000 7f8384400640 1 Processor -- start 2024-02-21T23:07:35.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.478+0000 7f8384400640 1 -- start start 2024-02-21T23:07:35.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.479+0000 7f8384400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f837c06b0e0 0x7f837c0c5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:35.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.479+0000 7f8384400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f837c0c5d80 0x7f837c14fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:35.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.479+0000 7f8384400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f837c0c61e0 con 0x7f837c150530 2024-02-21T23:07:35.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.479+0000 7f8384400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f837c153bc0 con 0x7f837c06b0e0 2024-02-21T23:07:35.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.479+0000 7f8384400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f837c152fb0 con 0x7f837c0c5d80 2024-02-21T23:07:35.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.480+0000 7f8381a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f837c06b0e0 0x7f837c0c5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:35.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.480+0000 7f8381a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f837c06b0e0 0x7f837c0c5840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58190/0 (socket says 172.21.15.28:58190) 2024-02-21T23:07:35.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.480+0000 7f8381a00640 1 -- 172.21.15.28:0/1994812584 learned_addr learned my addr 172.21.15.28:0/1994812584 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:35.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.480+0000 7f8381000640 1 --2- 172.21.15.28:0/1994812584 >> v2:172.21.15.81:3300/0 conn(0x7f837c0c5d80 0x7f837c14fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:35.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.480+0000 7f8381a00640 1 -- 172.21.15.28:0/1994812584 >> v2:172.21.15.81:3300/0 conn(0x7f837c0c5d80 msgr2=0x7f837c14fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:35.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.480+0000 7f8381a00640 1 --2- 172.21.15.28:0/1994812584 >> v2:172.21.15.81:3300/0 conn(0x7f837c0c5d80 0x7f837c14fff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:35.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.480+0000 7f8381a00640 1 -- 172.21.15.28:0/1994812584 >> v1:172.21.15.81:6789/0 conn(0x7f837c150530 legacy=0x7f837c156500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:35.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.480+0000 7f8381a00640 1 -- 172.21.15.28:0/1994812584 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f837c153290 con 0x7f837c06b0e0 2024-02-21T23:07:35.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.481+0000 7f8381a00640 1 --2- 172.21.15.28:0/1994812584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f837c06b0e0 0x7f837c0c5840 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f837c0c65f0 tx=0x7f836c02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54ba8efe751aee93 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:35.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.481+0000 7f837be00640 1 -- 172.21.15.28:0/1994812584 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f836c02c9f0 con 0x7f837c06b0e0 2024-02-21T23:07:35.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.482+0000 7f837be00640 1 -- 172.21.15.28:0/1994812584 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f836c02cb90 con 0x7f837c06b0e0 2024-02-21T23:07:35.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.482+0000 7f837be00640 1 -- 172.21.15.28:0/1994812584 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f836c0343f0 con 0x7f837c06b0e0 2024-02-21T23:07:35.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.482+0000 7f8384400640 1 -- 172.21.15.28:0/1994812584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f837c06b0e0 msgr2=0x7f837c0c5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:35.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.482+0000 7f8384400640 1 --2- 172.21.15.28:0/1994812584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f837c06b0e0 0x7f837c0c5840 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f837c0c65f0 tx=0x7f836c02be50 comp rx=0 tx=0).stop 2024-02-21T23:07:35.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.482+0000 7f8384400640 1 -- 172.21.15.28:0/1994812584 shutdown_connections 2024-02-21T23:07:35.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.482+0000 7f8384400640 1 --2- 172.21.15.28:0/1994812584 >> v2:172.21.15.81:3300/0 conn(0x7f837c0c5d80 0x7f837c14fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:35.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.482+0000 7f8384400640 1 --2- 172.21.15.28:0/1994812584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f837c06b0e0 0x7f837c0c5840 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:35.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.482+0000 7f8384400640 1 -- 172.21.15.28:0/1994812584 >> 172.21.15.28:0/1994812584 conn(0x7f837c0a85a0 msgr2=0x7f837c0a8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:35.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.482+0000 7f8384400640 1 -- 172.21.15.28:0/1994812584 shutdown_connections 2024-02-21T23:07:35.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.482+0000 7f8384400640 1 -- 172.21.15.28:0/1994812584 wait complete. 2024-02-21T23:07:35.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.484+0000 7f8384400640 1 Processor -- start 2024-02-21T23:07:35.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.484+0000 7f8384400640 1 -- start start 2024-02-21T23:07:35.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.485+0000 7f8384400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f837c06b0e0 0x7f837c149580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:35.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.485+0000 7f8384400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f837c0c5d80 0x7f837c1540e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:35.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.485+0000 7f8384400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f837c150530 0x7f837c154620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:35.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.485+0000 7f8384400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f837c150970 con 0x7f837c150530 2024-02-21T23:07:35.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.485+0000 7f8384400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f837c153850 con 0x7f837c06b0e0 2024-02-21T23:07:35.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.485+0000 7f8384400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f837c06b4c0 con 0x7f837c0c5d80 2024-02-21T23:07:35.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.485+0000 7f8382400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f837c150530 0x7f837c154620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:35.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.485+0000 7f8382400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f837c150530 0x7f837c154620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58198/0 (socket says 172.21.15.28:58198) 2024-02-21T23:07:35.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.486+0000 7f8382400640 1 -- 172.21.15.28:0/1446442502 learned_addr learned my addr 172.21.15.28:0/1446442502 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:35.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.486+0000 7f8381a00640 1 --2- 172.21.15.28:0/1446442502 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f837c06b0e0 0x7f837c149580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:35.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.486+0000 7f8381000640 1 --2- 172.21.15.28:0/1446442502 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f837c0c5d80 0x7f837c1540e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:35.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.486+0000 7f8382400640 1 -- 172.21.15.28:0/1446442502 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f837c0c5d80 msgr2=0x7f837c1540e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:35.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.486+0000 7f8382400640 1 --2- 172.21.15.28:0/1446442502 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f837c0c5d80 0x7f837c1540e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:35.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.486+0000 7f8382400640 1 -- 172.21.15.28:0/1446442502 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f837c06b0e0 msgr2=0x7f837c149580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:35.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.486+0000 7f8382400640 1 --2- 172.21.15.28:0/1446442502 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f837c06b0e0 0x7f837c149580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:35.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.486+0000 7f8382400640 1 -- 172.21.15.28:0/1446442502 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f837c1eb660 con 0x7f837c150530 2024-02-21T23:07:35.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.486+0000 7f8382400640 1 --2- 172.21.15.28:0/1446442502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f837c150530 0x7f837c154620 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f837400b440 tx=0x7f837400b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:35.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.487+0000 7f8379600640 1 -- 172.21.15.28:0/1446442502 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8374015020 con 0x7f837c150530 2024-02-21T23:07:35.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.487+0000 7f8379600640 1 -- 172.21.15.28:0/1446442502 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8374011830 con 0x7f837c150530 2024-02-21T23:07:35.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.487+0000 7f8379600640 1 -- 172.21.15.28:0/1446442502 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8374011c20 con 0x7f837c150530 2024-02-21T23:07:35.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.487+0000 7f8384400640 1 -- 172.21.15.28:0/1446442502 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f837c149e70 con 0x7f837c150530 2024-02-21T23:07:35.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.487+0000 7f8384400640 1 -- 172.21.15.28:0/1446442502 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f837c155c00 con 0x7f837c150530 2024-02-21T23:07:35.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.488+0000 7f8384400640 1 -- 172.21.15.28:0/1446442502 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f837c067f30 con 0x7f837c150530 2024-02-21T23:07:35.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.489+0000 7f8379600640 1 -- 172.21.15.28:0/1446442502 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8374018020 con 0x7f837c150530 2024-02-21T23:07:35.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.491+0000 7f8379600640 1 --2- 172.21.15.28:0/1446442502 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f835c06b500 0x7f835c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:35.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.491+0000 7f8379600640 1 -- 172.21.15.28:0/1446442502 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f8374089e40 con 0x7f837c150530 2024-02-21T23:07:35.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.491+0000 7f8381a00640 1 --2- 172.21.15.28:0/1446442502 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f835c06b500 0x7f835c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:35.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.492+0000 7f8381a00640 1 --2- 172.21.15.28:0/1446442502 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f835c06b500 0x7f835c06d9c0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f836c035000 tx=0x7f836c030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:35.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.495+0000 7f8379600640 1 -- 172.21.15.28:0/1446442502 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f837401f030 con 0x7f837c150530 2024-02-21T23:07:35.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.771+0000 7f8384400640 1 -- 172.21.15.28:0/1446442502 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f837c155dd0 con 0x7f837c150530 2024-02-21T23:07:35.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.775+0000 7f8379600640 1 -- 172.21.15.28:0/1446442502 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v15) v1 ==== 190+0+147 (secure 0 0 0) 0x7f83740172c0 con 0x7f837c150530 2024-02-21T23:07:35.764 INFO:teuthology.orchestra.run.smithi028.stdout:[client.0] 2024-02-21T23:07:35.764 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQDTgNZlILvYLxAAoREWxKKT7YcXnhkgk8YvoQ== 2024-02-21T23:07:35.764 INFO:teuthology.orchestra.run.smithi028.stdout: caps mds = "allow" 2024-02-21T23:07:35.764 INFO:teuthology.orchestra.run.smithi028.stdout: caps mon = "allow r" 2024-02-21T23:07:35.764 INFO:teuthology.orchestra.run.smithi028.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-02-21T23:07:35.764 INFO:teuthology.orchestra.run.smithi028.stderr:updated caps for client.0 2024-02-21T23:07:35.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.778+0000 7f8384400640 1 -- 172.21.15.28:0/1446442502 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f835c06b500 msgr2=0x7f835c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:35.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.778+0000 7f8384400640 1 --2- 172.21.15.28:0/1446442502 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f835c06b500 0x7f835c06d9c0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f836c035000 tx=0x7f836c030000 comp rx=0 tx=0).stop 2024-02-21T23:07:35.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.778+0000 7f8384400640 1 -- 172.21.15.28:0/1446442502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f837c150530 msgr2=0x7f837c154620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:35.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.778+0000 7f8384400640 1 --2- 172.21.15.28:0/1446442502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f837c150530 0x7f837c154620 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f837400b440 tx=0x7f837400b900 comp rx=0 tx=0).stop 2024-02-21T23:07:35.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.778+0000 7f8384400640 1 -- 172.21.15.28:0/1446442502 shutdown_connections 2024-02-21T23:07:35.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.778+0000 7f8384400640 1 --2- 172.21.15.28:0/1446442502 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f835c06b500 0x7f835c06d9c0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:35.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.778+0000 7f8384400640 1 --2- 172.21.15.28:0/1446442502 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f837c0c5d80 0x7f837c1540e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:35.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.778+0000 7f8384400640 1 --2- 172.21.15.28:0/1446442502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f837c150530 0x7f837c154620 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:35.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.778+0000 7f8384400640 1 --2- 172.21.15.28:0/1446442502 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f837c06b0e0 0x7f837c149580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:35.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.778+0000 7f8384400640 1 -- 172.21.15.28:0/1446442502 >> 172.21.15.28:0/1446442502 conn(0x7f837c0a85a0 msgr2=0x7f837c0a6eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:35.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.778+0000 7f8384400640 1 -- 172.21.15.28:0/1446442502 shutdown_connections 2024-02-21T23:07:35.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:35.778+0000 7f8384400640 1 -- 172.21.15.28:0/1446442502 wait complete. 2024-02-21T23:07:35.788 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:07:35.998 INFO:tasks.ceph.mds.a.smithi028.stderr:2024-02-21T23:07:36.009+0000 7f44eb400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-21T23:07:36.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.016+0000 7f04f0e00640 1 Processor -- start 2024-02-21T23:07:36.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.016+0000 7f04f0e00640 1 -- start start 2024-02-21T23:07:36.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.019+0000 7f04f0e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f04ec152500 0x7f04ec1558c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:36.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.019+0000 7f04f0e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f04ec06bbf0 0x7f04ec06c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:36.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.019+0000 7f04f0e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04ec1528e0 con 0x7f04ec0ca050 2024-02-21T23:07:36.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.019+0000 7f04f0e00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04ec0c86e0 con 0x7f04ec06bbf0 2024-02-21T23:07:36.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.019+0000 7f04f0e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04ec0c8980 con 0x7f04ec152500 2024-02-21T23:07:36.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.020+0000 7f04eb400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f04ec152500 0x7f04ec1558c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:36.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.020+0000 7f04ebe00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f04ec06bbf0 0x7f04ec06c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:36.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.020+0000 7f04eb400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f04ec152500 0x7f04ec1558c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3301/0 says I am v2:172.21.15.28:60786/0 (socket says 172.21.15.28:60786) 2024-02-21T23:07:36.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.020+0000 7f04eb400640 1 -- 172.21.15.28:0/685234855 learned_addr learned my addr 172.21.15.28:0/685234855 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:36.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.021+0000 7f04eb400640 1 -- 172.21.15.28:0/685234855 >> v2:172.21.15.81:3300/0 conn(0x7f04ec06bbf0 msgr2=0x7f04ec06c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:36.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.021+0000 7f04eb400640 1 --2- 172.21.15.28:0/685234855 >> v2:172.21.15.81:3300/0 conn(0x7f04ec06bbf0 0x7f04ec06c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:36.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.021+0000 7f04eb400640 1 -- 172.21.15.28:0/685234855 >> v1:172.21.15.81:6789/0 conn(0x7f04ec0ca050 legacy=0x7f04ec0bf6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:36.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.021+0000 7f04eb400640 1 -- 172.21.15.28:0/685234855 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04ec0c8c60 con 0x7f04ec152500 2024-02-21T23:07:36.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.021+0000 7f04ebe00640 1 --2- 172.21.15.28:0/685234855 >> v2:172.21.15.81:3300/0 conn(0x7f04ec06bbf0 0x7f04ec06c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-21T23:07:36.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.022+0000 7f04eb400640 1 --2- 172.21.15.28:0/685234855 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f04ec152500 0x7f04ec1558c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f04dc009870 tx=0x7f04dc02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a064b3fbe53dcd9 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:36.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.023+0000 7f04eaa00640 1 -- 172.21.15.28:0/685234855 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04dc035020 con 0x7f04ec152500 2024-02-21T23:07:36.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.023+0000 7f04eaa00640 1 -- 172.21.15.28:0/685234855 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04dc031640 con 0x7f04ec152500 2024-02-21T23:07:36.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.023+0000 7f04eaa00640 1 -- 172.21.15.28:0/685234855 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04dc02d040 con 0x7f04ec152500 2024-02-21T23:07:36.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.023+0000 7f04f0e00640 1 -- 172.21.15.28:0/685234855 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f04ec152500 msgr2=0x7f04ec1558c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:36.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.023+0000 7f04f0e00640 1 --2- 172.21.15.28:0/685234855 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f04ec152500 0x7f04ec1558c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f04dc009870 tx=0x7f04dc02b8f0 comp rx=0 tx=0).stop 2024-02-21T23:07:36.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.023+0000 7f04f0e00640 1 -- 172.21.15.28:0/685234855 shutdown_connections 2024-02-21T23:07:36.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.023+0000 7f04f0e00640 1 --2- 172.21.15.28:0/685234855 >> v2:172.21.15.81:3300/0 conn(0x7f04ec06bbf0 0x7f04ec06c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:36.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.023+0000 7f04f0e00640 1 --2- 172.21.15.28:0/685234855 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f04ec152500 0x7f04ec1558c0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:36.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.023+0000 7f04f0e00640 1 -- 172.21.15.28:0/685234855 >> 172.21.15.28:0/685234855 conn(0x7f04ec0ac450 msgr2=0x7f04ec0ac860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:36.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.024+0000 7f04f0e00640 1 -- 172.21.15.28:0/685234855 shutdown_connections 2024-02-21T23:07:36.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.024+0000 7f04f0e00640 1 -- 172.21.15.28:0/685234855 wait complete. 2024-02-21T23:07:36.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.025+0000 7f04f0e00640 1 Processor -- start 2024-02-21T23:07:36.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.025+0000 7f04f0e00640 1 -- start start 2024-02-21T23:07:36.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.026+0000 7f04f0e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f04ec06bbf0 0x7f04ec1eee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:36.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.026+0000 7f04f0e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f04ec0ca050 0x7f04ec0ab9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:36.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.026+0000 7f04f0e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04ec152500 0x7f04ec1e8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:36.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.026+0000 7f04f0e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04ec0bfdd0 con 0x7f04ec152500 2024-02-21T23:07:36.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.026+0000 7f04f0e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04ec0c9220 con 0x7f04ec0ca050 2024-02-21T23:07:36.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.026+0000 7f04f0e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04ec0bff50 con 0x7f04ec06bbf0 2024-02-21T23:07:36.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.027+0000 7f04ebe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04ec152500 0x7f04ec1e8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:36.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.027+0000 7f04ebe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04ec152500 0x7f04ec1e8ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58212/0 (socket says 172.21.15.28:58212) 2024-02-21T23:07:36.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.027+0000 7f04ebe00640 1 -- 172.21.15.28:0/3760038280 learned_addr learned my addr 172.21.15.28:0/3760038280 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:36.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.027+0000 7f04e3e00640 1 --2- 172.21.15.28:0/3760038280 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f04ec0ca050 0x7f04ec0ab9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:36.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.027+0000 7f04eb400640 1 --2- 172.21.15.28:0/3760038280 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f04ec06bbf0 0x7f04ec1eee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:36.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.027+0000 7f04ebe00640 1 -- 172.21.15.28:0/3760038280 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f04ec06bbf0 msgr2=0x7f04ec1eee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:36.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.027+0000 7f04ebe00640 1 --2- 172.21.15.28:0/3760038280 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f04ec06bbf0 0x7f04ec1eee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:36.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.027+0000 7f04ebe00640 1 -- 172.21.15.28:0/3760038280 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f04ec0ca050 msgr2=0x7f04ec0ab9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:36.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.027+0000 7f04ebe00640 1 --2- 172.21.15.28:0/3760038280 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f04ec0ca050 0x7f04ec0ab9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:36.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.027+0000 7f04ebe00640 1 -- 172.21.15.28:0/3760038280 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04ec1eb730 con 0x7f04ec152500 2024-02-21T23:07:36.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.028+0000 7f04ebe00640 1 --2- 172.21.15.28:0/3760038280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04ec152500 0x7f04ec1e8ce0 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f04e400efc0 tx=0x7f04e400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:36.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.028+0000 7f04e3400640 1 -- 172.21.15.28:0/3760038280 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04e4018030 con 0x7f04ec152500 2024-02-21T23:07:36.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.028+0000 7f04e3400640 1 -- 172.21.15.28:0/3760038280 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04e400cdf0 con 0x7f04ec152500 2024-02-21T23:07:36.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.028+0000 7f04e3400640 1 -- 172.21.15.28:0/3760038280 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04e4003bb0 con 0x7f04ec152500 2024-02-21T23:07:36.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.028+0000 7f04f0e00640 1 -- 172.21.15.28:0/3760038280 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04ec1eb9d0 con 0x7f04ec152500 2024-02-21T23:07:36.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.028+0000 7f04f0e00640 1 -- 172.21.15.28:0/3760038280 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04ec1ee680 con 0x7f04ec152500 2024-02-21T23:07:36.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.029+0000 7f04f0e00640 1 -- 172.21.15.28:0/3760038280 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04ec04f290 con 0x7f04ec152500 2024-02-21T23:07:36.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.030+0000 7f04e3400640 1 -- 172.21.15.28:0/3760038280 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f04e4005d10 con 0x7f04ec152500 2024-02-21T23:07:36.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.032+0000 7f04e3400640 1 --2- 172.21.15.28:0/3760038280 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f04c806b4d0 0x7f04c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:36.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.032+0000 7f04e3400640 1 -- 172.21.15.28:0/3760038280 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f04e408ad00 con 0x7f04ec152500 2024-02-21T23:07:36.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.032+0000 7f04eb400640 1 --2- 172.21.15.28:0/3760038280 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f04c806b4d0 0x7f04c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:36.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.034+0000 7f04eb400640 1 --2- 172.21.15.28:0/3760038280 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f04c806b4d0 0x7f04c806d990 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f04dc036000 tx=0x7f04dc03c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:36.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.036+0000 7f04e3400640 1 -- 172.21.15.28:0/3760038280 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f04e4022070 con 0x7f04ec152500 2024-02-21T23:07:36.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.285+0000 7f04f0e00640 1 -- 172.21.15.28:0/3760038280 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f04ec153780 con 0x7f04ec152500 2024-02-21T23:07:36.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.286+0000 7f04e3400640 1 -- 172.21.15.28:0/3760038280 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+4942 (secure 0 0 0) 0x7f04e4010040 con 0x7f04ec152500 2024-02-21T23:07:36.274 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:07:36.275 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":102,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5729,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1754860807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1754860807},{"type":"v1","addr":"172.21.15.81:6835","nonce":1754860807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3963422563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3963422563},{"type":"v1","addr":"172.21.15.81:6837","nonce":3963422563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7446,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/1119726035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1119726035},{"type":"v1","addr":"172.21.15.28:6833","nonce":1119726035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:07:29.533547+0000","modified":"2024-02-21T23:07:32.591216+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7458},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7458":{"gid":7458,"name":"a","rank":0,"incarnation":98,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/1259832430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":1259832430},{"type":"v1","addr":"172.21.15.28:6835","nonce":1259832430}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-02-21T23:07:36.275 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 102 2024-02-21T23:07:36.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.289+0000 7f04f0e00640 1 -- 172.21.15.28:0/3760038280 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f04c806b4d0 msgr2=0x7f04c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:36.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.289+0000 7f04f0e00640 1 --2- 172.21.15.28:0/3760038280 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f04c806b4d0 0x7f04c806d990 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f04dc036000 tx=0x7f04dc03c040 comp rx=0 tx=0).stop 2024-02-21T23:07:36.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.289+0000 7f04f0e00640 1 -- 172.21.15.28:0/3760038280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04ec152500 msgr2=0x7f04ec1e8ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:36.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.289+0000 7f04f0e00640 1 --2- 172.21.15.28:0/3760038280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04ec152500 0x7f04ec1e8ce0 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f04e400efc0 tx=0x7f04e400c490 comp rx=0 tx=0).stop 2024-02-21T23:07:36.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.289+0000 7f04f0e00640 1 -- 172.21.15.28:0/3760038280 shutdown_connections 2024-02-21T23:07:36.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.289+0000 7f04f0e00640 1 --2- 172.21.15.28:0/3760038280 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f04c806b4d0 0x7f04c806d990 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:36.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.289+0000 7f04f0e00640 1 --2- 172.21.15.28:0/3760038280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04ec152500 0x7f04ec1e8ce0 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:36.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.289+0000 7f04f0e00640 1 --2- 172.21.15.28:0/3760038280 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f04ec0ca050 0x7f04ec0ab9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:36.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.289+0000 7f04f0e00640 1 --2- 172.21.15.28:0/3760038280 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f04ec06bbf0 0x7f04ec1eee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:36.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.289+0000 7f04f0e00640 1 -- 172.21.15.28:0/3760038280 >> 172.21.15.28:0/3760038280 conn(0x7f04ec0ac450 msgr2=0x7f04ec06c8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:36.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.290+0000 7f04f0e00640 1 -- 172.21.15.28:0/3760038280 shutdown_connections 2024-02-21T23:07:36.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.290+0000 7f04f0e00640 1 -- 172.21.15.28:0/3760038280 wait complete. 2024-02-21T23:07:36.500 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-21T23:07:36.507 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-21T23:07:36.699 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 102, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:07:29.533547+0000', 'modified': '2024-02-21T23:07:32.591216+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7458}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7458': {'gid': 7458, 'name': 'a', 'rank': 0, 'incarnation': 98, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/1259832430', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 1259832430}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 1259832430}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:07:36.699 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-21T23:07:36.700 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-02-21T23:07:36.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.932+0000 7f940b000640 1 Processor -- start 2024-02-21T23:07:36.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.933+0000 7f940b000640 1 -- start start 2024-02-21T23:07:36.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.935+0000 7f940b000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f94041524c0 0x7f9404155a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:36.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.935+0000 7f940b000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f94040c85c0 0x7f94040c70d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:36.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.935+0000 7f940b000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f940406df50 con 0x7f94040c5040 2024-02-21T23:07:36.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.935+0000 7f940b000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94040c5520 con 0x7f94041524c0 2024-02-21T23:07:36.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.935+0000 7f940b000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94040c5800 con 0x7f94040c85c0 2024-02-21T23:07:36.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.936+0000 7f9409c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f94041524c0 0x7f9404155a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:36.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.936+0000 7f940a600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f94040c85c0 0x7f94040c70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:36.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.936+0000 7f940a600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f94040c85c0 0x7f94040c70d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3301/0 says I am v2:172.21.15.28:60792/0 (socket says 172.21.15.28:60792) 2024-02-21T23:07:36.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.936+0000 7f940a600640 1 -- 172.21.15.28:0/3252154109 learned_addr learned my addr 172.21.15.28:0/3252154109 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:36.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.937+0000 7f940a600640 1 -- 172.21.15.28:0/3252154109 >> v2:172.21.15.81:3300/0 conn(0x7f94041524c0 msgr2=0x7f9404155a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:36.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.937+0000 7f940a600640 1 --2- 172.21.15.28:0/3252154109 >> v2:172.21.15.81:3300/0 conn(0x7f94041524c0 0x7f9404155a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:36.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.937+0000 7f940a600640 1 -- 172.21.15.28:0/3252154109 >> v1:172.21.15.81:6789/0 conn(0x7f94040c5040 legacy=0x7f940406ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:36.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.937+0000 7f940a600640 1 -- 172.21.15.28:0/3252154109 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f940406c670 con 0x7f94040c85c0 2024-02-21T23:07:36.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.937+0000 7f9409c00640 1 --2- 172.21.15.28:0/3252154109 >> v2:172.21.15.81:3300/0 conn(0x7f94041524c0 0x7f9404155a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-21T23:07:36.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.938+0000 7f940a600640 1 --2- 172.21.15.28:0/3252154109 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f94040c85c0 0x7f94040c70d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f93fc009e50 tx=0x7f93fc02be50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8948060b89ebfa2f server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:36.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.939+0000 7f9403e00640 1 -- 172.21.15.28:0/3252154109 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93fc02c9f0 con 0x7f94040c85c0 2024-02-21T23:07:36.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.939+0000 7f9403e00640 1 -- 172.21.15.28:0/3252154109 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93fc02cb90 con 0x7f94040c85c0 2024-02-21T23:07:36.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.939+0000 7f9403e00640 1 -- 172.21.15.28:0/3252154109 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93fc0343f0 con 0x7f94040c85c0 2024-02-21T23:07:36.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.940+0000 7f940b000640 1 -- 172.21.15.28:0/3252154109 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f94040c85c0 msgr2=0x7f94040c70d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:36.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.940+0000 7f940b000640 1 --2- 172.21.15.28:0/3252154109 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f94040c85c0 0x7f94040c70d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f93fc009e50 tx=0x7f93fc02be50 comp rx=0 tx=0).stop 2024-02-21T23:07:36.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.940+0000 7f940b000640 1 -- 172.21.15.28:0/3252154109 shutdown_connections 2024-02-21T23:07:36.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.940+0000 7f940b000640 1 --2- 172.21.15.28:0/3252154109 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f94040c85c0 0x7f94040c70d0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:36.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.940+0000 7f940b000640 1 --2- 172.21.15.28:0/3252154109 >> v2:172.21.15.81:3300/0 conn(0x7f94041524c0 0x7f9404155a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:36.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.940+0000 7f940b000640 1 -- 172.21.15.28:0/3252154109 >> 172.21.15.28:0/3252154109 conn(0x7f94040ac2b0 msgr2=0x7f94040ac6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:36.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.940+0000 7f940b000640 1 -- 172.21.15.28:0/3252154109 shutdown_connections 2024-02-21T23:07:36.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.940+0000 7f940b000640 1 -- 172.21.15.28:0/3252154109 wait complete. 2024-02-21T23:07:36.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.943+0000 7f940b000640 1 Processor -- start 2024-02-21T23:07:36.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.944+0000 7f940b000640 1 -- start start 2024-02-21T23:07:36.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.946+0000 7f940b000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f94040c5040 0x7f9404158e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:36.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.946+0000 7f940b000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f94040c85c0 0x7f94041eb4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:36.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.946+0000 7f940b000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94041524c0 0x7f9404154470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:36.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.946+0000 7f940b000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94040c8a70 con 0x7f94041524c0 2024-02-21T23:07:36.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.946+0000 7f940b000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94041528a0 con 0x7f94040c5040 2024-02-21T23:07:36.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.946+0000 7f940b000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94040c7610 con 0x7f94040c85c0 2024-02-21T23:07:36.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.946+0000 7f940a600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94041524c0 0x7f9404154470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:36.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.946+0000 7f940a600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94041524c0 0x7f9404154470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58214/0 (socket says 172.21.15.28:58214) 2024-02-21T23:07:36.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.946+0000 7f940a600640 1 -- 172.21.15.28:0/2625513164 learned_addr learned my addr 172.21.15.28:0/2625513164 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:36.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.946+0000 7f9409c00640 1 --2- 172.21.15.28:0/2625513164 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f94040c5040 0x7f9404158e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:36.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.946+0000 7f9409200640 1 --2- 172.21.15.28:0/2625513164 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f94040c85c0 0x7f94041eb4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:36.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.947+0000 7f940a600640 1 -- 172.21.15.28:0/2625513164 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f94040c85c0 msgr2=0x7f94041eb4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:36.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.947+0000 7f940a600640 1 --2- 172.21.15.28:0/2625513164 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f94040c85c0 0x7f94041eb4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:36.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.947+0000 7f940a600640 1 -- 172.21.15.28:0/2625513164 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f94040c5040 msgr2=0x7f9404158e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:36.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.947+0000 7f940a600640 1 --2- 172.21.15.28:0/2625513164 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f94040c5040 0x7f9404158e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:36.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.947+0000 7f940a600640 1 -- 172.21.15.28:0/2625513164 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9404154f40 con 0x7f94041524c0 2024-02-21T23:07:36.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.947+0000 7f940a600640 1 --2- 172.21.15.28:0/2625513164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94041524c0 0x7f9404154470 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f93fc036040 tx=0x7f93fc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:36.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.948+0000 7f9401600640 1 -- 172.21.15.28:0/2625513164 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93fc044020 con 0x7f94041524c0 2024-02-21T23:07:36.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.948+0000 7f9401600640 1 -- 172.21.15.28:0/2625513164 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93fc040470 con 0x7f94041524c0 2024-02-21T23:07:36.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.948+0000 7f9401600640 1 -- 172.21.15.28:0/2625513164 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93fc040860 con 0x7f94041524c0 2024-02-21T23:07:36.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.948+0000 7f940b000640 1 -- 172.21.15.28:0/2625513164 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f940414c800 con 0x7f94041524c0 2024-02-21T23:07:36.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.948+0000 7f940b000640 1 -- 172.21.15.28:0/2625513164 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f940414cdc0 con 0x7f94041524c0 2024-02-21T23:07:36.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.950+0000 7f9401600640 1 -- 172.21.15.28:0/2625513164 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f93fc047020 con 0x7f94041524c0 2024-02-21T23:07:36.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.951+0000 7f9401600640 1 --2- 172.21.15.28:0/2625513164 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f93e406b500 0x7f93e406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:36.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.951+0000 7f9401600640 1 -- 172.21.15.28:0/2625513164 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f93fc0ace30 con 0x7f94041524c0 2024-02-21T23:07:36.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.952+0000 7f940f5f1740 1 Processor -- start 2024-02-21T23:07:36.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.952+0000 7f940f5f1740 1 -- start start 2024-02-21T23:07:36.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.952+0000 7f9409c00640 1 --2- 172.21.15.28:0/2625513164 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f93e406b500 0x7f93e406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:36.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.953+0000 7f940f5f1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b7fdd4020 0x556b7fdd4400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:36.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.953+0000 7f940f5f1740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x556b7fdd5330 0x556b7fe10170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:36.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.953+0000 7f940f5f1740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556b7fe106b0 0x556b7fe0e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:36.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.953+0000 7f940f5f1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b7fd38470 con 0x556b7fdd4020 2024-02-21T23:07:36.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.953+0000 7f940f5f1740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b7fce1af0 con 0x556b7fe106b0 2024-02-21T23:07:36.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.953+0000 7f940f5f1740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x556b7fd34510 con 0x556b7fdd5330 2024-02-21T23:07:36.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.953+0000 7f9409c00640 1 --2- 172.21.15.28:0/2625513164 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f93e406b500 0x7f93e406d9c0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f93f8009a30 tx=0x7f93f8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:36.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.953+0000 7f9409c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b7fdd4020 0x556b7fdd4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:36.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.953+0000 7f9409c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b7fdd4020 0x556b7fdd4400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58220/0 (socket says 172.21.15.28:58220) 2024-02-21T23:07:36.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.953+0000 7f9409c00640 1 -- 172.21.15.28:0/4119154912 learned_addr learned my addr 172.21.15.28:0/4119154912 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:36.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.954+0000 7f9409200640 1 --2- 172.21.15.28:0/4119154912 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x556b7fdd5330 0x556b7fe10170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:36.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.954+0000 7f940a600640 1 --2- 172.21.15.28:0/4119154912 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556b7fe106b0 0x556b7fe0e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:36.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.954+0000 7f9409c00640 1 -- 172.21.15.28:0/4119154912 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x556b7fdd5330 msgr2=0x556b7fe10170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:36.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.954+0000 7f9409c00640 1 --2- 172.21.15.28:0/4119154912 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x556b7fdd5330 0x556b7fe10170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:36.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.954+0000 7f9409c00640 1 -- 172.21.15.28:0/4119154912 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556b7fe106b0 msgr2=0x556b7fe0e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:36.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.954+0000 7f9409c00640 1 --2- 172.21.15.28:0/4119154912 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556b7fe106b0 0x556b7fe0e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:36.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.954+0000 7f9409c00640 1 -- 172.21.15.28:0/4119154912 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556b7fce8d90 con 0x556b7fdd4020 2024-02-21T23:07:36.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.954+0000 7f9409c00640 1 --2- 172.21.15.28:0/4119154912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b7fdd4020 0x556b7fdd4400 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f93f8011d60 tx=0x7f93f8014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb0d2115a9bd53f4 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:36.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.955+0000 7f93f6000640 1 -- 172.21.15.28:0/4119154912 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93f8011d90 con 0x556b7fdd4020 2024-02-21T23:07:36.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.955+0000 7f93f6000640 1 -- 172.21.15.28:0/4119154912 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93f8026070 con 0x556b7fdd4020 2024-02-21T23:07:36.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.955+0000 7f93f6000640 1 -- 172.21.15.28:0/4119154912 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93f8021680 con 0x556b7fdd4020 2024-02-21T23:07:36.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.955+0000 7f940f5f1740 1 -- 172.21.15.28:0/4119154912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b7fdd4020 msgr2=0x556b7fdd4400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:36.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.955+0000 7f940f5f1740 1 --2- 172.21.15.28:0/4119154912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b7fdd4020 0x556b7fdd4400 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f93f8011d60 tx=0x7f93f8014000 comp rx=0 tx=0).stop 2024-02-21T23:07:36.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.955+0000 7f940f5f1740 1 -- 172.21.15.28:0/4119154912 shutdown_connections 2024-02-21T23:07:36.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.955+0000 7f940f5f1740 1 --2- 172.21.15.28:0/4119154912 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x556b7fdd5330 0x556b7fe10170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:36.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.955+0000 7f940f5f1740 1 --2- 172.21.15.28:0/4119154912 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556b7fe106b0 0x556b7fe0e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:36.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.955+0000 7f940f5f1740 1 --2- 172.21.15.28:0/4119154912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b7fdd4020 0x556b7fdd4400 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:36.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.955+0000 7f940f5f1740 1 -- 172.21.15.28:0/4119154912 >> 172.21.15.28:0/4119154912 conn(0x556b7fdd0970 msgr2=0x556b7fdf5380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:36.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.955+0000 7f940f5f1740 1 -- 172.21.15.28:0/4119154912 shutdown_connections 2024-02-21T23:07:36.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.956+0000 7f940f5f1740 1 -- 172.21.15.28:0/4119154912 wait complete. 2024-02-21T23:07:36.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.956+0000 7f940f5f1740 1 -- start start 2024-02-21T23:07:36.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.956+0000 7f940f5f1740 1 Processor -- start 2024-02-21T23:07:36.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.957+0000 7f940f5f1740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x556b7fdd5330 0x556b7fdf32f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:36.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.957+0000 7f940f5f1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b7fe106b0 0x556b7fdce730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:36.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.957+0000 7f940f5f1740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556b7fe20bc0 0x556b7fdefc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:36.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.957+0000 7f940f5f1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b7fcd7040 con 0x556b7fe106b0 2024-02-21T23:07:36.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.957+0000 7f940f5f1740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b7fdf5a90 con 0x556b7fe20bc0 2024-02-21T23:07:36.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.957+0000 7f940f5f1740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x556b7fdf3bf0 con 0x556b7fdd5330 2024-02-21T23:07:36.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.957+0000 7f940f5f1740 10 client.0 _ll_register_callbacks cb 0x556b7fcf4920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:07:36.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.958+0000 7f9409200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b7fe106b0 0x556b7fdce730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:36.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.958+0000 7f9409200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b7fe106b0 0x556b7fdce730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58232/0 (socket says 172.21.15.28:58232) 2024-02-21T23:07:36.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.958+0000 7f9409200640 1 -- 172.21.15.28:0/2499190984 learned_addr learned my addr 172.21.15.28:0/2499190984 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:36.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.958+0000 7f940a600640 1 --2- 172.21.15.28:0/2499190984 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556b7fe20bc0 0x556b7fdefc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:36.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.958+0000 7f9409c00640 1 --2- 172.21.15.28:0/2499190984 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x556b7fdd5330 0x556b7fdf32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:36.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.958+0000 7f9409200640 1 -- 172.21.15.28:0/2499190984 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x556b7fdd5330 msgr2=0x556b7fdf32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:36.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.958+0000 7f9409200640 1 --2- 172.21.15.28:0/2499190984 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x556b7fdd5330 0x556b7fdf32f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:36.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.958+0000 7f9409200640 1 -- 172.21.15.28:0/2499190984 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556b7fe20bc0 msgr2=0x556b7fdefc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:36.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.958+0000 7f9409200640 1 --2- 172.21.15.28:0/2499190984 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556b7fe20bc0 0x556b7fdefc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:36.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.958+0000 7f9409200640 1 -- 172.21.15.28:0/2499190984 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556b7fcf6aa0 con 0x556b7fe106b0 2024-02-21T23:07:36.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.958+0000 7f9409c00640 1 --2- 172.21.15.28:0/2499190984 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x556b7fdd5330 0x556b7fdf32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:07:36.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.958+0000 7f940a600640 1 --2- 172.21.15.28:0/2499190984 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556b7fe20bc0 0x556b7fdefc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:07:36.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.958+0000 7f9409200640 1 --2- 172.21.15.28:0/2499190984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b7fe106b0 0x556b7fdce730 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f93f0003b50 tx=0x7f93f00014f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6c498336eba28a5 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:36.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.958+0000 7f93eaa00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:07:36.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.959+0000 7f93eaa00640 1 -- 172.21.15.28:0/2499190984 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93f001b060 con 0x556b7fe106b0 2024-02-21T23:07:36.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.959+0000 7f93eaa00640 1 -- 172.21.15.28:0/2499190984 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(110..110 src has 1..110) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f93f0016ca0 con 0x556b7fe106b0 2024-02-21T23:07:36.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.959+0000 7f940f5f1740 1 -- 172.21.15.28:0/2499190984 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f93f8026070 con 0x556b7fe106b0 2024-02-21T23:07:36.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.959+0000 7f93eaa00640 1 -- 172.21.15.28:0/2499190984 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=102) v2 ==== 24+0+0 (secure 0 0 0) 0x7f93f0001d20 con 0x556b7fe106b0 2024-02-21T23:07:36.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.959+0000 7f940f5f1740 10 client.7581 fetch_fsmap learned FSMap version 102 2024-02-21T23:07:36.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.959+0000 7f940f5f1740 1 -- 172.21.15.28:0/2499190984 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=102,monmap=2+}) v3 -- 0x556b7fa4d2d0 con 0x556b7fe106b0 2024-02-21T23:07:36.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.959+0000 7f93eaa00640 1 -- 172.21.15.28:0/2499190984 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93f8026070 con 0x556b7fe106b0 2024-02-21T23:07:36.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.960+0000 7f93eaa00640 1 -- 172.21.15.28:0/2499190984 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 102) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f93f00124d0 con 0x556b7fe106b0 2024-02-21T23:07:36.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.960+0000 7f940f5f1740 10 client.7581 fetch_fsmap finished waiting for FSMap version 102 2024-02-21T23:07:36.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.960+0000 7f940f5f1740 10 client.7581 resolve_mds: resolved name 'a' to daemon mds.a 2024-02-21T23:07:36.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.960+0000 7f940f5f1740 20 client.7581 populate_metadata read hostname 'smithi028' 2024-02-21T23:07:36.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.960+0000 7f940f5f1740 1 --2- 172.21.15.28:0/2499190984 >> [v2:172.21.15.28:6834/1259832430,v1:172.21.15.28:6835/1259832430] conn(0x556b7fe0bdd0 0x556b7fe23420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:36.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.960+0000 7f940f5f1740 4 client.7581 mds_command: new command op to 7458 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-21T23:07:36.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.960+0000 7f940f5f1740 1 -- 172.21.15.28:0/2499190984 --> [v2:172.21.15.28:6834/1259832430,v1:172.21.15.28:6835/1259832430] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556b7fd459b0 con 0x556b7fe0bdd0 2024-02-21T23:07:36.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.960+0000 7f9409c00640 1 --2- 172.21.15.28:0/2499190984 >> [v2:172.21.15.28:6834/1259832430,v1:172.21.15.28:6835/1259832430] conn(0x556b7fe0bdd0 0x556b7fe23420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:36.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.961+0000 7f9409c00640 1 --2- 172.21.15.28:0/2499190984 >> [v2:172.21.15.28:6834/1259832430,v1:172.21.15.28:6835/1259832430] conn(0x556b7fe0bdd0 0x556b7fe23420 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=beba624a022f1838 server_cookie=dbddf5b6a2e91deb in_seq=0 out_seq=0 2024-02-21T23:07:36.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.961+0000 7f93eaa00640 10 client.7581 ms_handle_connect on v2:172.21.15.28:6834/1259832430 2024-02-21T23:07:36.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.962+0000 7f93eaa00640 1 -- 172.21.15.28:0/2499190984 <== mds.0 v2:172.21.15.28:6834/1259832430 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x556b7fd459b0 con 0x556b7fe0bdd0 2024-02-21T23:07:36.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.962+0000 7f93eaa00640 10 client.7581 handle_command_reply: tid=0 2024-02-21T23:07:36.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.962+0000 7f940f5f1740 1 client.7581 shutdown 2024-02-21T23:07:36.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.962+0000 7f940f5f1740 1 -- 172.21.15.28:0/2499190984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b7fe106b0 msgr2=0x556b7fdce730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:36.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.962+0000 7f940f5f1740 1 --2- 172.21.15.28:0/2499190984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b7fe106b0 0x556b7fdce730 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f93f0003b50 tx=0x7f93f00014f0 comp rx=0 tx=0).stop 2024-02-21T23:07:36.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.962+0000 7f940f5f1740 1 -- 172.21.15.28:0/2499190984 shutdown_connections 2024-02-21T23:07:36.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.962+0000 7f940f5f1740 1 --2- 172.21.15.28:0/2499190984 >> [v2:172.21.15.28:6834/1259832430,v1:172.21.15.28:6835/1259832430] conn(0x556b7fe0bdd0 0x556b7fe23420 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:36.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.962+0000 7f940f5f1740 1 --2- 172.21.15.28:0/2499190984 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556b7fe20bc0 0x556b7fdefc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:36.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.962+0000 7f940f5f1740 1 --2- 172.21.15.28:0/2499190984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b7fe106b0 0x556b7fdce730 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:36.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.962+0000 7f940f5f1740 1 --2- 172.21.15.28:0/2499190984 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x556b7fdd5330 0x556b7fdf32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:36.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.962+0000 7f940f5f1740 1 -- 172.21.15.28:0/2499190984 >> 172.21.15.28:0/2499190984 conn(0x556b7fdd4020 msgr2=0x556b7fdcd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:36.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.962+0000 7f93eaa00640 0 client.7581 ms_handle_reset on v2:172.21.15.28:6834/1259832430 2024-02-21T23:07:36.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.962+0000 7f940f5f1740 1 -- 172.21.15.28:0/2499190984 shutdown_connections 2024-02-21T23:07:36.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.962+0000 7f940f5f1740 1 -- 172.21.15.28:0/2499190984 wait complete. 2024-02-21T23:07:36.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.962+0000 7f940f5f1740 20 client.7581 trim_cache size 0 max 16384 2024-02-21T23:07:36.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.981+0000 7f940f5f1740 1 Processor -- start 2024-02-21T23:07:36.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.981+0000 7f940f5f1740 1 -- start start 2024-02-21T23:07:36.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.982+0000 7f940f5f1740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x556b7fdd5330 0x556b7fdf2f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:36.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.982+0000 7f940f5f1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b7fe106b0 0x556b7fe0d230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:36.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.982+0000 7f940f5f1740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556b7fe20bc0 0x556b7fdf08d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:36.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.982+0000 7f940f5f1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94041f6cb0 con 0x556b7fe106b0 2024-02-21T23:07:36.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.982+0000 7f940f5f1740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94041eeb20 con 0x556b7fe20bc0 2024-02-21T23:07:36.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.982+0000 7f940f5f1740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94041f66b0 con 0x556b7fdd5330 2024-02-21T23:07:36.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.982+0000 7f9409200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b7fe106b0 0x556b7fe0d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:36.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.982+0000 7f9409200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b7fe106b0 0x556b7fe0d230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58234/0 (socket says 172.21.15.28:58234) 2024-02-21T23:07:36.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.982+0000 7f9409200640 1 -- 172.21.15.28:0/2833759837 learned_addr learned my addr 172.21.15.28:0/2833759837 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:36.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.982+0000 7f9409200640 1 -- 172.21.15.28:0/2833759837 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x556b7fdd5330 msgr2=0x556b7fdf2f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:36.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.982+0000 7f9409200640 1 --2- 172.21.15.28:0/2833759837 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x556b7fdd5330 0x556b7fdf2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:36.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.982+0000 7f9409200640 1 -- 172.21.15.28:0/2833759837 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556b7fe20bc0 msgr2=0x556b7fdf08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:36.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.982+0000 7f9409200640 1 --2- 172.21.15.28:0/2833759837 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556b7fe20bc0 0x556b7fdf08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:36.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.982+0000 7f9409200640 1 -- 172.21.15.28:0/2833759837 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556b7fb176b0 con 0x556b7fe106b0 2024-02-21T23:07:36.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.983+0000 7f9409200640 1 --2- 172.21.15.28:0/2833759837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b7fe106b0 0x556b7fe0d230 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f93f0019040 tx=0x7f93f0010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b695aecedfbb3d50 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:36.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.983+0000 7f93f5600640 1 -- 172.21.15.28:0/2833759837 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93f0025080 con 0x556b7fe106b0 2024-02-21T23:07:36.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.983+0000 7f93f5600640 1 -- 172.21.15.28:0/2833759837 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93f00019f0 con 0x556b7fe106b0 2024-02-21T23:07:36.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.983+0000 7f93f5600640 1 -- 172.21.15.28:0/2833759837 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93f0015460 con 0x556b7fe106b0 2024-02-21T23:07:36.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.984+0000 7f940f5f1740 1 -- 172.21.15.28:0/2833759837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b7fe106b0 msgr2=0x556b7fe0d230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:36.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.984+0000 7f940f5f1740 1 --2- 172.21.15.28:0/2833759837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b7fe106b0 0x556b7fe0d230 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f93f0019040 tx=0x7f93f0010000 comp rx=0 tx=0).stop 2024-02-21T23:07:36.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.984+0000 7f940f5f1740 1 -- 172.21.15.28:0/2833759837 shutdown_connections 2024-02-21T23:07:36.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.984+0000 7f940f5f1740 1 --2- 172.21.15.28:0/2833759837 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556b7fe20bc0 0x556b7fdf08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:36.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.984+0000 7f940f5f1740 1 --2- 172.21.15.28:0/2833759837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b7fe106b0 0x556b7fe0d230 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:36.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.984+0000 7f940f5f1740 1 --2- 172.21.15.28:0/2833759837 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x556b7fdd5330 0x556b7fdf2f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:36.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.984+0000 7f940f5f1740 1 -- 172.21.15.28:0/2833759837 >> 172.21.15.28:0/2833759837 conn(0x556b7fdd4020 msgr2=0x556b7fdcd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:36.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.984+0000 7f940f5f1740 1 -- 172.21.15.28:0/2833759837 shutdown_connections 2024-02-21T23:07:36.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.984+0000 7f940f5f1740 1 -- 172.21.15.28:0/2833759837 wait complete. 2024-02-21T23:07:36.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.984+0000 7f940f5f1740 1 -- start start 2024-02-21T23:07:36.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.985+0000 7f940f5f1740 1 Processor -- start 2024-02-21T23:07:36.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.986+0000 7f940f5f1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b7fdd5330 0x556b7fdf3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:36.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.986+0000 7f940f5f1740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x556b7fe106b0 0x556b7fdf4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:36.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.986+0000 7f940f5f1740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556b7fe20bc0 0x556b7fe82470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:36.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.986+0000 7f940f5f1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b7fdcec70 con 0x556b7fdd5330 2024-02-21T23:07:36.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.986+0000 7f940f5f1740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b7fdf39e0 con 0x556b7fe20bc0 2024-02-21T23:07:36.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.986+0000 7f940f5f1740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x556b7fdf0630 con 0x556b7fe106b0 2024-02-21T23:07:36.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.986+0000 7f940f5f1740 10 client.0 _ll_register_callbacks cb 0x7f93f0003730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:07:36.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.986+0000 7f9409c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b7fdd5330 0x556b7fdf3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:36.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.986+0000 7f9409c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b7fdd5330 0x556b7fdf3120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58248/0 (socket says 172.21.15.28:58248) 2024-02-21T23:07:36.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.986+0000 7f9409c00640 1 -- 172.21.15.28:0/3371645288 learned_addr learned my addr 172.21.15.28:0/3371645288 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:36.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.986+0000 7f9409c00640 1 -- 172.21.15.28:0/3371645288 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x556b7fe106b0 msgr2=0x556b7fdf4bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:36.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.986+0000 7f9409200640 1 --2- 172.21.15.28:0/3371645288 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x556b7fe106b0 0x556b7fdf4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:36.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.986+0000 7f9409c00640 1 --2- 172.21.15.28:0/3371645288 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x556b7fe106b0 0x556b7fdf4bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:36.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.986+0000 7f9409c00640 1 -- 172.21.15.28:0/3371645288 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556b7fe20bc0 msgr2=0x556b7fe82470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:36.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.986+0000 7f940a600640 1 --2- 172.21.15.28:0/3371645288 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556b7fe20bc0 0x556b7fe82470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:36.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.986+0000 7f9409c00640 1 --2- 172.21.15.28:0/3371645288 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556b7fe20bc0 0x556b7fe82470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:36.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.986+0000 7f9409c00640 1 -- 172.21.15.28:0/3371645288 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556b7fdefcb0 con 0x556b7fdd5330 2024-02-21T23:07:36.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.987+0000 7f9409c00640 1 --2- 172.21.15.28:0/3371645288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b7fdd5330 0x556b7fdf3120 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f93f8022000 tx=0x7f93f8009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38ea352bcce9a066 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:36.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.987+0000 7f93eaa00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:07:36.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.987+0000 7f93eaa00640 1 -- 172.21.15.28:0/3371645288 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93f8039020 con 0x556b7fdd5330 2024-02-21T23:07:36.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.987+0000 7f93eaa00640 1 -- 172.21.15.28:0/3371645288 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(110..110 src has 1..110) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f93f8011e10 con 0x556b7fdd5330 2024-02-21T23:07:36.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.987+0000 7f940f5f1740 1 -- 172.21.15.28:0/3371645288 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f93f00019f0 con 0x556b7fdd5330 2024-02-21T23:07:36.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.987+0000 7f93eaa00640 1 -- 172.21.15.28:0/3371645288 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=102) v2 ==== 24+0+0 (secure 0 0 0) 0x7f93f8035590 con 0x556b7fdd5330 2024-02-21T23:07:36.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.987+0000 7f940f5f1740 10 client.7587 fetch_fsmap learned FSMap version 102 2024-02-21T23:07:36.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.987+0000 7f940f5f1740 1 -- 172.21.15.28:0/3371645288 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=102,monmap=2+}) v3 -- 0x556b7fe82dc0 con 0x556b7fdd5330 2024-02-21T23:07:36.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.988+0000 7f93eaa00640 1 -- 172.21.15.28:0/3371645288 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93f00019f0 con 0x556b7fdd5330 2024-02-21T23:07:36.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.988+0000 7f93eaa00640 1 -- 172.21.15.28:0/3371645288 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 102) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f93f8046080 con 0x556b7fdd5330 2024-02-21T23:07:36.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.988+0000 7f940f5f1740 10 client.7587 fetch_fsmap finished waiting for FSMap version 102 2024-02-21T23:07:36.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.988+0000 7f940f5f1740 10 client.7587 resolve_mds: resolved name 'a' to daemon mds.a 2024-02-21T23:07:36.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.988+0000 7f940f5f1740 20 client.7587 populate_metadata read hostname 'smithi028' 2024-02-21T23:07:36.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.988+0000 7f940f5f1740 1 --2- 172.21.15.28:0/3371645288 >> [v2:172.21.15.28:6834/1259832430,v1:172.21.15.28:6835/1259832430] conn(0x556b7fe08110 0x556b7fe7bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:36.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.988+0000 7f940f5f1740 4 client.7587 mds_command: new command op to 7458 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-21T23:07:36.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.988+0000 7f940f5f1740 1 -- 172.21.15.28:0/3371645288 --> [v2:172.21.15.28:6834/1259832430,v1:172.21.15.28:6835/1259832430] -- command(tid 0: {"prefix": "status"}) v1 -- 0x556b7fd3fd00 con 0x556b7fe08110 2024-02-21T23:07:36.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.988+0000 7f9409200640 1 --2- 172.21.15.28:0/3371645288 >> [v2:172.21.15.28:6834/1259832430,v1:172.21.15.28:6835/1259832430] conn(0x556b7fe08110 0x556b7fe7bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:36.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.989+0000 7f9409200640 1 --2- 172.21.15.28:0/3371645288 >> [v2:172.21.15.28:6834/1259832430,v1:172.21.15.28:6835/1259832430] conn(0x556b7fe08110 0x556b7fe7bc40 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f9fa7dcd70cc5874 server_cookie=4fc594e42a4f0e54 in_seq=0 out_seq=0 2024-02-21T23:07:36.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.989+0000 7f93eaa00640 10 client.7587 ms_handle_connect on v2:172.21.15.28:6834/1259832430 2024-02-21T23:07:36.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.989+0000 7f93eaa00640 1 -- 172.21.15.28:0/3371645288 <== mds.0 v2:172.21.15.28:6834/1259832430 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+334 (crc 0 0 0) 0x556b7fd3fd00 con 0x556b7fe08110 2024-02-21T23:07:36.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.989+0000 7f93eaa00640 10 client.7587 handle_command_reply: tid=0 2024-02-21T23:07:36.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.989+0000 7f940f5f1740 1 client.7587 shutdown 2024-02-21T23:07:36.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.989+0000 7f940f5f1740 1 -- 172.21.15.28:0/3371645288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b7fdd5330 msgr2=0x556b7fdf3120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:36.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.989+0000 7f940f5f1740 1 --2- 172.21.15.28:0/3371645288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b7fdd5330 0x556b7fdf3120 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f93f8022000 tx=0x7f93f8009190 comp rx=0 tx=0).stop 2024-02-21T23:07:36.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.989+0000 7f940f5f1740 1 -- 172.21.15.28:0/3371645288 shutdown_connections 2024-02-21T23:07:36.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.989+0000 7f940f5f1740 1 --2- 172.21.15.28:0/3371645288 >> [v2:172.21.15.28:6834/1259832430,v1:172.21.15.28:6835/1259832430] conn(0x556b7fe08110 0x556b7fe7bc40 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:36.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.989+0000 7f940f5f1740 1 --2- 172.21.15.28:0/3371645288 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x556b7fe106b0 0x556b7fdf4bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:36.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.989+0000 7f940f5f1740 1 --2- 172.21.15.28:0/3371645288 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x556b7fe20bc0 0x556b7fe82470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:36.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.989+0000 7f940f5f1740 1 --2- 172.21.15.28:0/3371645288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b7fdd5330 0x556b7fdf3120 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:36.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.989+0000 7f93eaa00640 0 client.7587 ms_handle_reset on v2:172.21.15.28:6834/1259832430 2024-02-21T23:07:36.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.989+0000 7f940f5f1740 1 -- 172.21.15.28:0/3371645288 >> 172.21.15.28:0/3371645288 conn(0x556b7fdd4020 msgr2=0x556b7fe06b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:36.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.990+0000 7f940f5f1740 1 -- 172.21.15.28:0/3371645288 shutdown_connections 2024-02-21T23:07:36.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.990+0000 7f940f5f1740 1 -- 172.21.15.28:0/3371645288 wait complete. 2024-02-21T23:07:36.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.990+0000 7f940f5f1740 20 client.7587 trim_cache size 0 max 16384 2024-02-21T23:07:36.986 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-21T23:07:36.986 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac", 2024-02-21T23:07:36.986 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-02-21T23:07:36.986 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 7458, 2024-02-21T23:07:36.986 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-02-21T23:07:36.986 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-02-21T23:07:36.986 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-02-21T23:07:36.986 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 6.9349385410000002, 2024-02-21T23:07:36.986 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 102, 2024-02-21T23:07:36.986 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 110, 2024-02-21T23:07:36.986 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 110, 2024-02-21T23:07:36.986 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 12.401567824000001 2024-02-21T23:07:36.986 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-21T23:07:36.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.990+0000 7f940b000640 1 -- 172.21.15.28:0/2625513164 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f93e406b500 msgr2=0x7f93e406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:36.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.990+0000 7f940b000640 1 --2- 172.21.15.28:0/2625513164 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f93e406b500 0x7f93e406d9c0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f93f8009a30 tx=0x7f93f8008040 comp rx=0 tx=0).stop 2024-02-21T23:07:36.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.990+0000 7f940b000640 1 -- 172.21.15.28:0/2625513164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94041524c0 msgr2=0x7f9404154470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:36.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.990+0000 7f940b000640 1 --2- 172.21.15.28:0/2625513164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94041524c0 0x7f9404154470 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f93fc036040 tx=0x7f93fc004a40 comp rx=0 tx=0).stop 2024-02-21T23:07:36.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.990+0000 7f940b000640 1 -- 172.21.15.28:0/2625513164 shutdown_connections 2024-02-21T23:07:36.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.990+0000 7f940b000640 1 --2- 172.21.15.28:0/2625513164 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f93e406b500 0x7f93e406d9c0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:36.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.990+0000 7f940b000640 1 --2- 172.21.15.28:0/2625513164 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f94040c85c0 0x7f94041eb4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:36.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.990+0000 7f940b000640 1 --2- 172.21.15.28:0/2625513164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94041524c0 0x7f9404154470 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:36.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.990+0000 7f940b000640 1 --2- 172.21.15.28:0/2625513164 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f94040c5040 0x7f9404158e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:36.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.991+0000 7f940b000640 1 -- 172.21.15.28:0/2625513164 >> 172.21.15.28:0/2625513164 conn(0x7f94040ac2b0 msgr2=0x7f94040c7cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:36.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.991+0000 7f940b000640 1 -- 172.21.15.28:0/2625513164 shutdown_connections 2024-02-21T23:07:36.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:36.991+0000 7f940b000640 1 -- 172.21.15.28:0/2625513164 wait complete. 2024-02-21T23:07:37.001 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:07:37.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.227+0000 7f4181200640 1 Processor -- start 2024-02-21T23:07:37.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.228+0000 7f4181200640 1 -- start start 2024-02-21T23:07:37.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.229+0000 7f4181200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f417c0aff90 0x7f417c0c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:37.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.229+0000 7f4181200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f417c06dbf0 0x7f417c158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:37.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.230+0000 7f4181200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f417c0c5720 0x7f417c159000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:37.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.230+0000 7f4181200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f417c0c7270 con 0x7f417c0c5720 2024-02-21T23:07:37.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.230+0000 7f4181200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f417c158630 con 0x7f417c0aff90 2024-02-21T23:07:37.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.230+0000 7f4181200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f417c0c74c0 con 0x7f417c06dbf0 2024-02-21T23:07:37.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.230+0000 7f417be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f417c0c5720 0x7f417c159000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:37.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.230+0000 7f417be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f417c0c5720 0x7f417c159000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58256/0 (socket says 172.21.15.28:58256) 2024-02-21T23:07:37.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.230+0000 7f417be00640 1 -- 172.21.15.28:0/3785159746 learned_addr learned my addr 172.21.15.28:0/3785159746 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:37.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.230+0000 7f4173e00640 1 --2- 172.21.15.28:0/3785159746 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f417c06dbf0 0x7f417c158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:37.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.230+0000 7f417b400640 1 --2- 172.21.15.28:0/3785159746 >> v2:172.21.15.81:3300/0 conn(0x7f417c0aff90 0x7f417c0c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:37.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.230+0000 7f417be00640 1 -- 172.21.15.28:0/3785159746 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f417c06dbf0 msgr2=0x7f417c158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:37.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.230+0000 7f417be00640 1 --2- 172.21.15.28:0/3785159746 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f417c06dbf0 0x7f417c158ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:37.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.230+0000 7f417be00640 1 -- 172.21.15.28:0/3785159746 >> v2:172.21.15.81:3300/0 conn(0x7f417c0aff90 msgr2=0x7f417c0c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:37.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.230+0000 7f417be00640 1 --2- 172.21.15.28:0/3785159746 >> v2:172.21.15.81:3300/0 conn(0x7f417c0aff90 0x7f417c0c51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:37.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.230+0000 7f417be00640 1 -- 172.21.15.28:0/3785159746 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f417c0c77a0 con 0x7f417c0c5720 2024-02-21T23:07:37.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.230+0000 7f417b400640 1 --2- 172.21.15.28:0/3785159746 >> v2:172.21.15.81:3300/0 conn(0x7f417c0aff90 0x7f417c0c51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:07:37.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.232+0000 7f417be00640 1 --2- 172.21.15.28:0/3785159746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f417c0c5720 0x7f417c159000 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f417c0aa1b0 tx=0x7f417402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3672a49e67616c77 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:37.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.232+0000 7f417aa00640 1 -- 172.21.15.28:0/3785159746 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417402ca80 con 0x7f417c0c5720 2024-02-21T23:07:37.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.232+0000 7f417aa00640 1 -- 172.21.15.28:0/3785159746 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f417402cc20 con 0x7f417c0c5720 2024-02-21T23:07:37.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.232+0000 7f417aa00640 1 -- 172.21.15.28:0/3785159746 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41740343f0 con 0x7f417c0c5720 2024-02-21T23:07:37.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.232+0000 7f4181200640 1 -- 172.21.15.28:0/3785159746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f417c0c5720 msgr2=0x7f417c159000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:37.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.232+0000 7f4181200640 1 --2- 172.21.15.28:0/3785159746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f417c0c5720 0x7f417c159000 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f417c0aa1b0 tx=0x7f417402bee0 comp rx=0 tx=0).stop 2024-02-21T23:07:37.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.233+0000 7f4181200640 1 -- 172.21.15.28:0/3785159746 shutdown_connections 2024-02-21T23:07:37.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.233+0000 7f4181200640 1 --2- 172.21.15.28:0/3785159746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f417c0c5720 0x7f417c159000 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:37.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.233+0000 7f4181200640 1 --2- 172.21.15.28:0/3785159746 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f417c06dbf0 0x7f417c158ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:37.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.233+0000 7f4181200640 1 --2- 172.21.15.28:0/3785159746 >> v2:172.21.15.81:3300/0 conn(0x7f417c0aff90 0x7f417c0c51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:37.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.233+0000 7f4181200640 1 -- 172.21.15.28:0/3785159746 >> 172.21.15.28:0/3785159746 conn(0x7f417c0ac370 msgr2=0x7f417c0ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:37.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.233+0000 7f4181200640 1 -- 172.21.15.28:0/3785159746 shutdown_connections 2024-02-21T23:07:37.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.233+0000 7f4181200640 1 -- 172.21.15.28:0/3785159746 wait complete. 2024-02-21T23:07:37.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.234+0000 7f4181200640 1 Processor -- start 2024-02-21T23:07:37.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.235+0000 7f4181200640 1 -- start start 2024-02-21T23:07:37.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.236+0000 7f4181200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f417c06dbf0 0x7f417c0be240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:37.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.236+0000 7f4181200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f417c0aff90 0x7f417c0bf960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:37.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.236+0000 7f4181200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f417c0c5720 0x7f417c1eacc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:37.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.236+0000 7f4181200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f417c0c8760 con 0x7f417c0c5720 2024-02-21T23:07:37.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.236+0000 7f4181200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f417c0be0c0 con 0x7f417c06dbf0 2024-02-21T23:07:37.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.236+0000 7f4181200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f417c0c88e0 con 0x7f417c0aff90 2024-02-21T23:07:37.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.236+0000 7f417be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f417c0c5720 0x7f417c1eacc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:37.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.236+0000 7f417be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f417c0c5720 0x7f417c1eacc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58262/0 (socket says 172.21.15.28:58262) 2024-02-21T23:07:37.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.236+0000 7f417be00640 1 -- 172.21.15.28:0/3096384590 learned_addr learned my addr 172.21.15.28:0/3096384590 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:37.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.236+0000 7f4173e00640 1 --2- 172.21.15.28:0/3096384590 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f417c0aff90 0x7f417c0bf960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:37.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.236+0000 7f417b400640 1 --2- 172.21.15.28:0/3096384590 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f417c06dbf0 0x7f417c0be240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:37.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.236+0000 7f417be00640 1 -- 172.21.15.28:0/3096384590 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f417c0aff90 msgr2=0x7f417c0bf960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:37.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.236+0000 7f417be00640 1 --2- 172.21.15.28:0/3096384590 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f417c0aff90 0x7f417c0bf960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:37.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.236+0000 7f417be00640 1 -- 172.21.15.28:0/3096384590 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f417c06dbf0 msgr2=0x7f417c0be240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:37.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.236+0000 7f417be00640 1 --2- 172.21.15.28:0/3096384590 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f417c06dbf0 0x7f417c0be240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:37.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.236+0000 7f417be00640 1 -- 172.21.15.28:0/3096384590 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f417c0bde80 con 0x7f417c0c5720 2024-02-21T23:07:37.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.236+0000 7f4173e00640 1 --2- 172.21.15.28:0/3096384590 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f417c0aff90 0x7f417c0bf960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:07:37.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.237+0000 7f417be00640 1 --2- 172.21.15.28:0/3096384590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f417c0c5720 0x7f417c1eacc0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f4174009f40 tx=0x7f4174004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:37.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.237+0000 7f4173400640 1 -- 172.21.15.28:0/3096384590 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4174043020 con 0x7f417c0c5720 2024-02-21T23:07:37.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.237+0000 7f4173400640 1 -- 172.21.15.28:0/3096384590 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f417403e450 con 0x7f417c0c5720 2024-02-21T23:07:37.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.237+0000 7f4173400640 1 -- 172.21.15.28:0/3096384590 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4174009510 con 0x7f417c0c5720 2024-02-21T23:07:37.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.237+0000 7f4181200640 1 -- 172.21.15.28:0/3096384590 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f417c0bdba0 con 0x7f417c0c5720 2024-02-21T23:07:37.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.237+0000 7f4181200640 1 -- 172.21.15.28:0/3096384590 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f417c1ea7a0 con 0x7f417c0c5720 2024-02-21T23:07:37.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.238+0000 7f4181200640 1 -- 172.21.15.28:0/3096384590 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f417c04f290 con 0x7f417c0c5720 2024-02-21T23:07:37.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.239+0000 7f4173400640 1 -- 172.21.15.28:0/3096384590 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4174046020 con 0x7f417c0c5720 2024-02-21T23:07:37.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.240+0000 7f4173400640 1 --2- 172.21.15.28:0/3096384590 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f415806b4d0 0x7f415806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:37.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.240+0000 7f4173400640 1 -- 172.21.15.28:0/3096384590 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f41740acc90 con 0x7f417c0c5720 2024-02-21T23:07:37.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.241+0000 7f417b400640 1 --2- 172.21.15.28:0/3096384590 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f415806b4d0 0x7f415806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:37.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.242+0000 7f417b400640 1 --2- 172.21.15.28:0/3096384590 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f415806b4d0 0x7f415806d990 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f416c008850 tx=0x7f416c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:37.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.245+0000 7f4173400640 1 -- 172.21.15.28:0/3096384590 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f417404d030 con 0x7f417c0c5720 2024-02-21T23:07:37.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.484+0000 7f4181200640 1 -- 172.21.15.28:0/3096384590 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f417c1ea970 con 0x7f417c0c5720 2024-02-21T23:07:37.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.485+0000 7f4173400640 1 -- 172.21.15.28:0/3096384590 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+4942 (secure 0 0 0) 0x7f417407ba00 con 0x7f417c0c5720 2024-02-21T23:07:37.474 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:07:37.474 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":102,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5729,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1754860807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1754860807},{"type":"v1","addr":"172.21.15.81:6835","nonce":1754860807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3963422563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3963422563},{"type":"v1","addr":"172.21.15.81:6837","nonce":3963422563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7446,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/1119726035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1119726035},{"type":"v1","addr":"172.21.15.28:6833","nonce":1119726035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:07:29.533547+0000","modified":"2024-02-21T23:07:32.591216+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7458},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7458":{"gid":7458,"name":"a","rank":0,"incarnation":98,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/1259832430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":1259832430},{"type":"v1","addr":"172.21.15.28:6835","nonce":1259832430}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-02-21T23:07:37.474 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 102 2024-02-21T23:07:37.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.488+0000 7f4181200640 1 -- 172.21.15.28:0/3096384590 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f415806b4d0 msgr2=0x7f415806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:37.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.488+0000 7f4181200640 1 --2- 172.21.15.28:0/3096384590 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f415806b4d0 0x7f415806d990 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f416c008850 tx=0x7f416c008040 comp rx=0 tx=0).stop 2024-02-21T23:07:37.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.488+0000 7f4181200640 1 -- 172.21.15.28:0/3096384590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f417c0c5720 msgr2=0x7f417c1eacc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:37.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.488+0000 7f4181200640 1 --2- 172.21.15.28:0/3096384590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f417c0c5720 0x7f417c1eacc0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f4174009f40 tx=0x7f4174004a40 comp rx=0 tx=0).stop 2024-02-21T23:07:37.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.488+0000 7f4181200640 1 -- 172.21.15.28:0/3096384590 shutdown_connections 2024-02-21T23:07:37.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.488+0000 7f4181200640 1 --2- 172.21.15.28:0/3096384590 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f415806b4d0 0x7f415806d990 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:37.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.488+0000 7f4181200640 1 --2- 172.21.15.28:0/3096384590 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f417c0aff90 0x7f417c0bf960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:37.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.488+0000 7f4181200640 1 --2- 172.21.15.28:0/3096384590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f417c0c5720 0x7f417c1eacc0 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:37.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.488+0000 7f4181200640 1 --2- 172.21.15.28:0/3096384590 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f417c06dbf0 0x7f417c0be240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:37.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.488+0000 7f4181200640 1 -- 172.21.15.28:0/3096384590 >> 172.21.15.28:0/3096384590 conn(0x7f417c0ac370 msgr2=0x7f417c0ab7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:37.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.488+0000 7f4181200640 1 -- 172.21.15.28:0/3096384590 shutdown_connections 2024-02-21T23:07:37.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.488+0000 7f4181200640 1 -- 172.21.15.28:0/3096384590 wait complete. 2024-02-21T23:07:37.497 INFO:tasks.cephfs.test_failover:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 8, "epoch": 102, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 8, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-02-21T23:07:29.533547+0000", "damaged": [], "data_pools": [ 18 ], "enabled": true, "epoch": 102, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7458": { "addr": "172.21.15.28:6835/1259832430", "addrs": { "addrvec": [ { "addr": "172.21.15.28:6834", "nonce": 1259832430, "type": "v2" }, { "addr": "172.21.15.28:6835", "nonce": 1259832430, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7458, "incarnation": 98, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 17, "modified": "2024-02-21T23:07:32.591216+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7458 } } } ], "standbys": [ { "addr": "172.21.15.81:6835/1754860807", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6834", "nonce": 1754860807, "type": "v2" }, { "addr": "172.21.15.81:6835", "nonce": 1754860807, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 97, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5729, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.81:6837/3963422563", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6836", "nonce": 3963422563, "type": "v2" }, { "addr": "172.21.15.81:6837", "nonce": 3963422563, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 97, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5732, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.28:6833/1119726035", "addrs": { "addrvec": [ { "addr": "172.21.15.28:6832", "nonce": 1119726035, "type": "v2" }, { "addr": "172.21.15.28:6833", "nonce": 1119726035, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 96, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7446, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-02-21T23:07:37.497 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-02-21T23:07:37.497 INFO:tasks.cephfs.test_failover:current = [{'gid': 7458, 'name': 'a', 'rank': 0, 'incarnation': 98, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/1259832430', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 1259832430}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 1259832430}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 5732, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.81:6837/3963422563', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3963422563}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3963422563}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 97}, {'gid': 7446, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.28:6833/1119726035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 1119726035}, {'type': 'v1', 'addr': '172.21.15.28:6833', 'nonce': 1119726035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 96}, {'gid': 5729, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.81:6835/1754860807', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6834', 'nonce': 1754860807}, {'type': 'v1', 'addr': '172.21.15.81:6835', 'nonce': 1754860807}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 97}] 2024-02-21T23:07:37.498 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.d mds_join_fs cephfs 2024-02-21T23:07:37.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.758+0000 7f1977e00640 1 Processor -- start 2024-02-21T23:07:37.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.758+0000 7f1977e00640 1 -- start start 2024-02-21T23:07:37.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.760+0000 7f1977e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1978152bd0 0x7f197814cb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:37.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.760+0000 7f1977e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19780c4e00 con 0x7f1978070550 2024-02-21T23:07:37.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.760+0000 7f1977e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1978153220 con 0x7f19780c4a20 2024-02-21T23:07:37.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.760+0000 7f1977e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19780c77b0 con 0x7f1978152bd0 2024-02-21T23:07:37.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.760+0000 7f1976000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f1978070550 0x7f197814c400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:48436/0 (socket says 172.21.15.28:48436) 2024-02-21T23:07:37.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.760+0000 7f1976000640 1 -- 172.21.15.28:0/260848069 learned_addr learned my addr 172.21.15.28:0/260848069 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:07:37.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.761+0000 7f1977400640 1 --2- 172.21.15.28:0/260848069 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1978152bd0 0x7f197814cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:37.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.761+0000 7f1975600640 1 -- 172.21.15.28:0/260848069 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3280024955 0 0) 0x7f19780c4e00 con 0x7f1978070550 2024-02-21T23:07:37.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.761+0000 7f1975600640 1 -- 172.21.15.28:0/260848069 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f195c003520 con 0x7f1978070550 2024-02-21T23:07:37.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.762+0000 7f1977400640 1 -- 172.21.15.28:0/260848069 >> v1:172.21.15.81:6789/0 conn(0x7f19780c4a20 legacy=0x7f19780c5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:37.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.762+0000 7f1977400640 1 -- 172.21.15.28:0/260848069 >> v1:172.21.15.28:6789/0 conn(0x7f1978070550 legacy=0x7f197814c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:37.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.762+0000 7f1975600640 1 -- 172.21.15.28:0/260848069 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 731342111 0 0) 0x7f1978153220 con 0x7f19780c4a20 2024-02-21T23:07:37.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.762+0000 7f1977400640 1 -- 172.21.15.28:0/260848069 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1978153500 con 0x7f1978152bd0 2024-02-21T23:07:37.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.763+0000 7f1977400640 1 --2- 172.21.15.28:0/260848069 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1978152bd0 0x7f197814cb10 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1970009dc0 tx=0x7f197002bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3697dd48fe230d66 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:37.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.763+0000 7f1975600640 1 -- 172.21.15.28:0/260848069 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1970039070 con 0x7f1978152bd0 2024-02-21T23:07:37.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.763+0000 7f1975600640 1 -- 172.21.15.28:0/260848069 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f197002ccc0 con 0x7f1978152bd0 2024-02-21T23:07:37.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.763+0000 7f1975600640 1 -- 172.21.15.28:0/260848069 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1970034660 con 0x7f1978152bd0 2024-02-21T23:07:37.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.764+0000 7f1977e00640 1 -- 172.21.15.28:0/260848069 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1978152bd0 msgr2=0x7f197814cb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:37.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.764+0000 7f1977e00640 1 --2- 172.21.15.28:0/260848069 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1978152bd0 0x7f197814cb10 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1970009dc0 tx=0x7f197002bf10 comp rx=0 tx=0).stop 2024-02-21T23:07:37.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.764+0000 7f1977e00640 1 -- 172.21.15.28:0/260848069 shutdown_connections 2024-02-21T23:07:37.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.764+0000 7f1977e00640 1 --2- 172.21.15.28:0/260848069 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1978152bd0 0x7f197814cb10 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:37.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.764+0000 7f1977e00640 1 -- 172.21.15.28:0/260848069 >> 172.21.15.28:0/260848069 conn(0x7f197806b290 msgr2=0x7f197806b6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:37.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.764+0000 7f1977e00640 1 -- 172.21.15.28:0/260848069 shutdown_connections 2024-02-21T23:07:37.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.764+0000 7f1977e00640 1 -- 172.21.15.28:0/260848069 wait complete. 2024-02-21T23:07:37.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.767+0000 7f1977e00640 1 Processor -- start 2024-02-21T23:07:37.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.767+0000 7f1977e00640 1 -- start start 2024-02-21T23:07:37.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.769+0000 7f1977e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1978070550 0x7f197814b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:37.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.769+0000 7f1977e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19780c4a20 0x7f197814b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:37.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.769+0000 7f1977e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1978152bd0 0x7f19781edec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:37.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.769+0000 7f1977e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1978153ac0 con 0x7f19780c4a20 2024-02-21T23:07:37.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.769+0000 7f1977e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f197814d050 con 0x7f1978070550 2024-02-21T23:07:37.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.769+0000 7f1977e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19780c2c30 con 0x7f1978152bd0 2024-02-21T23:07:37.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.769+0000 7f1976000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19780c4a20 0x7f197814b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:37.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.769+0000 7f1976000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19780c4a20 0x7f197814b680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58272/0 (socket says 172.21.15.28:58272) 2024-02-21T23:07:37.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.769+0000 7f1976000640 1 -- 172.21.15.28:0/1229781553 learned_addr learned my addr 172.21.15.28:0/1229781553 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:37.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.769+0000 7f1976a00640 1 --2- 172.21.15.28:0/1229781553 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1978070550 0x7f197814b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:37.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.769+0000 7f1977400640 1 --2- 172.21.15.28:0/1229781553 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1978152bd0 0x7f19781edec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:37.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.770+0000 7f1976000640 1 -- 172.21.15.28:0/1229781553 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1978152bd0 msgr2=0x7f19781edec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:37.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.770+0000 7f1976000640 1 --2- 172.21.15.28:0/1229781553 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1978152bd0 0x7f19781edec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:37.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.770+0000 7f1976000640 1 -- 172.21.15.28:0/1229781553 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1978070550 msgr2=0x7f197814b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:37.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.770+0000 7f1976000640 1 --2- 172.21.15.28:0/1229781553 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1978070550 0x7f197814b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:37.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.770+0000 7f1976000640 1 -- 172.21.15.28:0/1229781553 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19780abed0 con 0x7f19780c4a20 2024-02-21T23:07:37.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.770+0000 7f1976a00640 1 --2- 172.21.15.28:0/1229781553 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1978070550 0x7f197814b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:07:37.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.770+0000 7f1976000640 1 --2- 172.21.15.28:0/1229781553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19780c4a20 0x7f197814b680 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f196000c390 tx=0x7f196000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:37.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.771+0000 7f196ea00640 1 -- 172.21.15.28:0/1229781553 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1960017070 con 0x7f19780c4a20 2024-02-21T23:07:37.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.771+0000 7f196ea00640 1 -- 172.21.15.28:0/1229781553 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1960005ce0 con 0x7f19780c4a20 2024-02-21T23:07:37.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.771+0000 7f196ea00640 1 -- 172.21.15.28:0/1229781553 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19600072f0 con 0x7f19780c4a20 2024-02-21T23:07:37.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.771+0000 7f1977e00640 1 -- 172.21.15.28:0/1229781553 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19780ac1b0 con 0x7f19780c4a20 2024-02-21T23:07:37.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.771+0000 7f1977e00640 1 -- 172.21.15.28:0/1229781553 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19780ac770 con 0x7f19780c4a20 2024-02-21T23:07:37.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.772+0000 7f1977e00640 1 -- 172.21.15.28:0/1229781553 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f197804f290 con 0x7f19780c4a20 2024-02-21T23:07:37.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.774+0000 7f196ea00640 1 -- 172.21.15.28:0/1229781553 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f196001a070 con 0x7f19780c4a20 2024-02-21T23:07:37.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.775+0000 7f196ea00640 1 --2- 172.21.15.28:0/1229781553 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f195406b500 0x7f195406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:37.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.775+0000 7f196ea00640 1 -- 172.21.15.28:0/1229781553 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1960089620 con 0x7f19780c4a20 2024-02-21T23:07:37.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.776+0000 7f1976a00640 1 --2- 172.21.15.28:0/1229781553 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f195406b500 0x7f195406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:37.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.777+0000 7f1976a00640 1 --2- 172.21.15.28:0/1229781553 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f195406b500 0x7f195406d9c0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f1968000f60 tx=0x7f1968009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:37.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.780+0000 7f196ea00640 1 -- 172.21.15.28:0/1229781553 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1960057d50 con 0x7f19780c4a20 2024-02-21T23:07:37.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:37.999+0000 7f1977e00640 1 -- 172.21.15.28:0/1229781553 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) v1 -- 0x7f19780c2e40 con 0x7f19780c4a20 2024-02-21T23:07:37.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.005+0000 7f196ea00640 1 -- 172.21.15.28:0/1229781553 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v36)=0 v36) v1 ==== 116+0+0 (secure 0 0 0) 0x7f196005bdf0 con 0x7f19780c4a20 2024-02-21T23:07:37.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.007+0000 7f1977e00640 1 -- 172.21.15.28:0/1229781553 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f195406b500 msgr2=0x7f195406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:37.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.007+0000 7f1977e00640 1 --2- 172.21.15.28:0/1229781553 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f195406b500 0x7f195406d9c0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f1968000f60 tx=0x7f1968009040 comp rx=0 tx=0).stop 2024-02-21T23:07:37.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.007+0000 7f1977e00640 1 -- 172.21.15.28:0/1229781553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19780c4a20 msgr2=0x7f197814b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:37.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.007+0000 7f1977e00640 1 --2- 172.21.15.28:0/1229781553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19780c4a20 0x7f197814b680 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f196000c390 tx=0x7f196000c850 comp rx=0 tx=0).stop 2024-02-21T23:07:37.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.008+0000 7f1977e00640 1 -- 172.21.15.28:0/1229781553 shutdown_connections 2024-02-21T23:07:37.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.008+0000 7f1977e00640 1 --2- 172.21.15.28:0/1229781553 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f195406b500 0x7f195406d9c0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:37.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.008+0000 7f1977e00640 1 --2- 172.21.15.28:0/1229781553 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1978152bd0 0x7f19781edec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:37.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.008+0000 7f1977e00640 1 --2- 172.21.15.28:0/1229781553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19780c4a20 0x7f197814b680 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:37.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.008+0000 7f1977e00640 1 --2- 172.21.15.28:0/1229781553 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1978070550 0x7f197814b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:37.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.008+0000 7f1977e00640 1 -- 172.21.15.28:0/1229781553 >> 172.21.15.28:0/1229781553 conn(0x7f197806b290 msgr2=0x7f19780aac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:37.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.008+0000 7f1977e00640 1 -- 172.21.15.28:0/1229781553 shutdown_connections 2024-02-21T23:07:37.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.008+0000 7f1977e00640 1 -- 172.21.15.28:0/1229781553 wait complete. 2024-02-21T23:07:38.020 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:07:38.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.225+0000 7f3261200640 1 Processor -- start 2024-02-21T23:07:38.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.225+0000 7f3261200640 1 -- start start 2024-02-21T23:07:38.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.226+0000 7f3261200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f325c06ee40 0x7f325c0a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:38.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.226+0000 7f3261200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f325c156af0 0x7f325c06dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:38.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.227+0000 7f3261200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f325c0c2760 con 0x7f325c150450 2024-02-21T23:07:38.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.227+0000 7f3261200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f325c0c22c0 con 0x7f325c06ee40 2024-02-21T23:07:38.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.227+0000 7f3261200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f325c156660 con 0x7f325c156af0 2024-02-21T23:07:38.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.227+0000 7f325aa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f325c06ee40 0x7f325c0a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:38.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.227+0000 7f325aa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f325c06ee40 0x7f325c0a5fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58282/0 (socket says 172.21.15.28:58282) 2024-02-21T23:07:38.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.227+0000 7f325aa00640 1 -- 172.21.15.28:0/3681595271 learned_addr learned my addr 172.21.15.28:0/3681595271 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:38.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.227+0000 7f325be00640 1 --2- 172.21.15.28:0/3681595271 >> v2:172.21.15.81:3300/0 conn(0x7f325c156af0 0x7f325c06dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:38.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.227+0000 7f325aa00640 1 -- 172.21.15.28:0/3681595271 >> v2:172.21.15.81:3300/0 conn(0x7f325c156af0 msgr2=0x7f325c06dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:38.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.227+0000 7f325aa00640 1 --2- 172.21.15.28:0/3681595271 >> v2:172.21.15.81:3300/0 conn(0x7f325c156af0 0x7f325c06dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:38.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.227+0000 7f325aa00640 1 -- 172.21.15.28:0/3681595271 >> v1:172.21.15.81:6789/0 conn(0x7f325c150450 legacy=0x7f325c1536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:38.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.227+0000 7f325aa00640 1 -- 172.21.15.28:0/3681595271 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f325c156310 con 0x7f325c06ee40 2024-02-21T23:07:38.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.229+0000 7f325aa00640 1 --2- 172.21.15.28:0/3681595271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f325c06ee40 0x7f325c0a5fd0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f325c0734b0 tx=0x7f324402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a79dba5c29f7abb7 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:38.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.229+0000 7f325a000640 1 -- 172.21.15.28:0/3681595271 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3244035020 con 0x7f325c06ee40 2024-02-21T23:07:38.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.229+0000 7f325a000640 1 -- 172.21.15.28:0/3681595271 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3244031870 con 0x7f325c06ee40 2024-02-21T23:07:38.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.229+0000 7f325a000640 1 -- 172.21.15.28:0/3681595271 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3244031c40 con 0x7f325c06ee40 2024-02-21T23:07:38.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.229+0000 7f3261200640 1 -- 172.21.15.28:0/3681595271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f325c06ee40 msgr2=0x7f325c0a5fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:38.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.229+0000 7f3261200640 1 --2- 172.21.15.28:0/3681595271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f325c06ee40 0x7f325c0a5fd0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f325c0734b0 tx=0x7f324402b910 comp rx=0 tx=0).stop 2024-02-21T23:07:38.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.230+0000 7f3261200640 1 -- 172.21.15.28:0/3681595271 shutdown_connections 2024-02-21T23:07:38.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.230+0000 7f3261200640 1 --2- 172.21.15.28:0/3681595271 >> v2:172.21.15.81:3300/0 conn(0x7f325c156af0 0x7f325c06dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:38.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.230+0000 7f3261200640 1 --2- 172.21.15.28:0/3681595271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f325c06ee40 0x7f325c0a5fd0 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:38.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.230+0000 7f3261200640 1 -- 172.21.15.28:0/3681595271 >> 172.21.15.28:0/3681595271 conn(0x7f325c06b630 msgr2=0x7f325c0c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:38.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.230+0000 7f3261200640 1 -- 172.21.15.28:0/3681595271 shutdown_connections 2024-02-21T23:07:38.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.230+0000 7f3261200640 1 -- 172.21.15.28:0/3681595271 wait complete. 2024-02-21T23:07:38.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.231+0000 7f3261200640 1 Processor -- start 2024-02-21T23:07:38.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.232+0000 7f3261200640 1 -- start start 2024-02-21T23:07:38.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.232+0000 7f3261200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f325c06ee40 0x7f325c1559b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:38.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.232+0000 7f3261200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f325c150450 0x7f325c140430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:38.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.232+0000 7f3261200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f325c156af0 0x7f325c140970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:38.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.232+0000 7f3261200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f325c156f80 con 0x7f325c150450 2024-02-21T23:07:38.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.232+0000 7f3261200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f325c0c2cb0 con 0x7f325c06ee40 2024-02-21T23:07:38.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.232+0000 7f3261200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f325c157100 con 0x7f325c156af0 2024-02-21T23:07:38.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.233+0000 7f325aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f325c150450 0x7f325c140430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:38.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.233+0000 7f325aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f325c150450 0x7f325c140430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58292/0 (socket says 172.21.15.28:58292) 2024-02-21T23:07:38.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.233+0000 7f325aa00640 1 -- 172.21.15.28:0/1436343800 learned_addr learned my addr 172.21.15.28:0/1436343800 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:38.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.233+0000 7f325b400640 1 --2- 172.21.15.28:0/1436343800 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f325c06ee40 0x7f325c1559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:38.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.233+0000 7f325aa00640 1 -- 172.21.15.28:0/1436343800 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f325c156af0 msgr2=0x7f325c140970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:38.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.233+0000 7f325aa00640 1 --2- 172.21.15.28:0/1436343800 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f325c156af0 0x7f325c140970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:38.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.233+0000 7f325aa00640 1 -- 172.21.15.28:0/1436343800 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f325c06ee40 msgr2=0x7f325c1559b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:38.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.233+0000 7f325aa00640 1 --2- 172.21.15.28:0/1436343800 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f325c06ee40 0x7f325c1559b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:38.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.233+0000 7f325aa00640 1 -- 172.21.15.28:0/1436343800 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f325c0c61c0 con 0x7f325c150450 2024-02-21T23:07:38.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.233+0000 7f325aa00640 1 --2- 172.21.15.28:0/1436343800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f325c150450 0x7f325c140430 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f3244000c00 tx=0x7f32440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:38.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.234+0000 7f3253400640 1 -- 172.21.15.28:0/1436343800 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f324403ad60 con 0x7f325c150450 2024-02-21T23:07:38.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.234+0000 7f3253400640 1 -- 172.21.15.28:0/1436343800 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32440443f0 con 0x7f325c150450 2024-02-21T23:07:38.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.234+0000 7f3261200640 1 -- 172.21.15.28:0/1436343800 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f325c1eb590 con 0x7f325c150450 2024-02-21T23:07:38.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.234+0000 7f3253400640 1 -- 172.21.15.28:0/1436343800 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32440447c0 con 0x7f325c150450 2024-02-21T23:07:38.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.234+0000 7f3261200640 1 -- 172.21.15.28:0/1436343800 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f325c14a900 con 0x7f325c150450 2024-02-21T23:07:38.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.235+0000 7f3261200640 1 -- 172.21.15.28:0/1436343800 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f325c14a6f0 con 0x7f325c150450 2024-02-21T23:07:38.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.237+0000 7f3253400640 1 -- 172.21.15.28:0/1436343800 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f324404b060 con 0x7f325c150450 2024-02-21T23:07:38.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.238+0000 7f3253400640 1 --2- 172.21.15.28:0/1436343800 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f323806b4d0 0x7f323806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:38.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.238+0000 7f3253400640 1 -- 172.21.15.28:0/1436343800 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f32440ac530 con 0x7f325c150450 2024-02-21T23:07:38.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.239+0000 7f325b400640 1 --2- 172.21.15.28:0/1436343800 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f323806b4d0 0x7f323806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:38.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.240+0000 7f325b400640 1 --2- 172.21.15.28:0/1436343800 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f323806b4d0 0x7f323806d990 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f324c007fd0 tx=0x7f324c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:38.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.243+0000 7f3253400640 1 -- 172.21.15.28:0/1436343800 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f324403f070 con 0x7f325c150450 2024-02-21T23:07:38.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.485+0000 7f3261200640 1 -- 172.21.15.28:0/1436343800 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f325c04f290 con 0x7f325c150450 2024-02-21T23:07:38.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.486+0000 7f3253400640 1 -- 172.21.15.28:0/1436343800 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+4942 (secure 0 0 0) 0x7f324402f320 con 0x7f325c150450 2024-02-21T23:07:38.474 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:07:38.474 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":102,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5729,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1754860807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1754860807},{"type":"v1","addr":"172.21.15.81:6835","nonce":1754860807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3963422563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3963422563},{"type":"v1","addr":"172.21.15.81:6837","nonce":3963422563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7446,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/1119726035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1119726035},{"type":"v1","addr":"172.21.15.28:6833","nonce":1119726035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:07:29.533547+0000","modified":"2024-02-21T23:07:32.591216+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7458},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7458":{"gid":7458,"name":"a","rank":0,"incarnation":98,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/1259832430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":1259832430},{"type":"v1","addr":"172.21.15.28:6835","nonce":1259832430}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-02-21T23:07:38.474 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 102 2024-02-21T23:07:38.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.488+0000 7f3261200640 1 -- 172.21.15.28:0/1436343800 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f323806b4d0 msgr2=0x7f323806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:38.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.488+0000 7f3261200640 1 --2- 172.21.15.28:0/1436343800 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f323806b4d0 0x7f323806d990 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f324c007fd0 tx=0x7f324c009040 comp rx=0 tx=0).stop 2024-02-21T23:07:38.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.488+0000 7f3261200640 1 -- 172.21.15.28:0/1436343800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f325c150450 msgr2=0x7f325c140430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:38.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.488+0000 7f3261200640 1 --2- 172.21.15.28:0/1436343800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f325c150450 0x7f325c140430 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f3244000c00 tx=0x7f32440046c0 comp rx=0 tx=0).stop 2024-02-21T23:07:38.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.488+0000 7f3261200640 1 -- 172.21.15.28:0/1436343800 shutdown_connections 2024-02-21T23:07:38.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.488+0000 7f3261200640 1 --2- 172.21.15.28:0/1436343800 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f323806b4d0 0x7f323806d990 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:38.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.488+0000 7f3261200640 1 --2- 172.21.15.28:0/1436343800 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f325c156af0 0x7f325c140970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:38.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.488+0000 7f3261200640 1 --2- 172.21.15.28:0/1436343800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f325c150450 0x7f325c140430 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:38.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.488+0000 7f3261200640 1 --2- 172.21.15.28:0/1436343800 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f325c06ee40 0x7f325c1559b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:38.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.488+0000 7f3261200640 1 -- 172.21.15.28:0/1436343800 >> 172.21.15.28:0/1436343800 conn(0x7f325c06b630 msgr2=0x7f325c0a7180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:38.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.489+0000 7f3261200640 1 -- 172.21.15.28:0/1436343800 shutdown_connections 2024-02-21T23:07:38.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:38.489+0000 7f3261200640 1 -- 172.21.15.28:0/1436343800 wait complete. 2024-02-21T23:07:38.899 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 102, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:07:29.533547+0000', 'modified': '2024-02-21T23:07:32.591216+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7458}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7458': {'gid': 7458, 'name': 'a', 'rank': 0, 'incarnation': 98, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/1259832430', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 1259832430}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 1259832430}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:07:38.899 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-21T23:07:38.899 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-02-21T23:07:39.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.139+0000 7fa44d200640 1 Processor -- start 2024-02-21T23:07:39.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.140+0000 7fa44d200640 1 -- start start 2024-02-21T23:07:39.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.141+0000 7fa44d200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fa4480c5f20 0x7fa448156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:39.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.141+0000 7fa44d200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa448153220 0x7fa448153630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:39.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.141+0000 7fa44d200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4480c5c10 con 0x7fa4480c2df0 2024-02-21T23:07:39.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.141+0000 7fa44d200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa44806e6a0 con 0x7fa448153220 2024-02-21T23:07:39.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.141+0000 7fa44d200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa44806e8d0 con 0x7fa4480c5f20 2024-02-21T23:07:39.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.141+0000 7fa446800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa448153220 0x7fa448153630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:39.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.142+0000 7fa446800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa448153220 0x7fa448153630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58308/0 (socket says 172.21.15.28:58308) 2024-02-21T23:07:39.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.142+0000 7fa446800640 1 -- 172.21.15.28:0/462668137 learned_addr learned my addr 172.21.15.28:0/462668137 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:39.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.142+0000 7fa445400640 1 --2- 172.21.15.28:0/462668137 >> v2:172.21.15.81:3300/0 conn(0x7fa4480c5f20 0x7fa448156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:39.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.142+0000 7fa446800640 1 -- 172.21.15.28:0/462668137 >> v2:172.21.15.81:3300/0 conn(0x7fa4480c5f20 msgr2=0x7fa448156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:39.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.142+0000 7fa446800640 1 --2- 172.21.15.28:0/462668137 >> v2:172.21.15.81:3300/0 conn(0x7fa4480c5f20 0x7fa448156ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:39.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.142+0000 7fa446800640 1 -- 172.21.15.28:0/462668137 >> v1:172.21.15.81:6789/0 conn(0x7fa4480c2df0 legacy=0x7fa4480c31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:39.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.142+0000 7fa446800640 1 -- 172.21.15.28:0/462668137 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa44806ebb0 con 0x7fa448153220 2024-02-21T23:07:39.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.142+0000 7fa445400640 1 --2- 172.21.15.28:0/462668137 >> v2:172.21.15.81:3300/0 conn(0x7fa4480c5f20 0x7fa448156ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:07:39.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.144+0000 7fa446800640 1 --2- 172.21.15.28:0/462668137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa448153220 0x7fa448153630 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7fa44806f8b0 tx=0x7fa44002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3c22f6974680a49 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:39.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.144+0000 7fa444a00640 1 -- 172.21.15.28:0/462668137 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa44002ca80 con 0x7fa448153220 2024-02-21T23:07:39.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.144+0000 7fa444a00640 1 -- 172.21.15.28:0/462668137 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa44002cc20 con 0x7fa448153220 2024-02-21T23:07:39.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.144+0000 7fa444a00640 1 -- 172.21.15.28:0/462668137 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4400343f0 con 0x7fa448153220 2024-02-21T23:07:39.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.145+0000 7fa44d200640 1 -- 172.21.15.28:0/462668137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa448153220 msgr2=0x7fa448153630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:39.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.145+0000 7fa44d200640 1 --2- 172.21.15.28:0/462668137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa448153220 0x7fa448153630 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7fa44806f8b0 tx=0x7fa44002bee0 comp rx=0 tx=0).stop 2024-02-21T23:07:39.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.145+0000 7fa44d200640 1 -- 172.21.15.28:0/462668137 shutdown_connections 2024-02-21T23:07:39.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.145+0000 7fa44d200640 1 --2- 172.21.15.28:0/462668137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa448153220 0x7fa448153630 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:39.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.145+0000 7fa44d200640 1 --2- 172.21.15.28:0/462668137 >> v2:172.21.15.81:3300/0 conn(0x7fa4480c5f20 0x7fa448156ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:39.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.145+0000 7fa44d200640 1 -- 172.21.15.28:0/462668137 >> 172.21.15.28:0/462668137 conn(0x7fa44806b820 msgr2=0x7fa4480c67f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:39.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.145+0000 7fa44d200640 1 -- 172.21.15.28:0/462668137 shutdown_connections 2024-02-21T23:07:39.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.145+0000 7fa44d200640 1 -- 172.21.15.28:0/462668137 wait complete. 2024-02-21T23:07:39.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.147+0000 7fa44d200640 1 Processor -- start 2024-02-21T23:07:39.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.147+0000 7fa44d200640 1 -- start start 2024-02-21T23:07:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.148+0000 7fa44d200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa4480c2df0 0x7fa4480bfc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.148+0000 7fa44d200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4480c5f20 0x7fa4481eb320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.148+0000 7fa44d200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa448153220 0x7fa4481eb860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.148+0000 7fa44d200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4480c38e0 con 0x7fa4480c5f20 2024-02-21T23:07:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.148+0000 7fa44d200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa44806f170 con 0x7fa4480c2df0 2024-02-21T23:07:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.148+0000 7fa44d200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4480c3a60 con 0x7fa448153220 2024-02-21T23:07:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.148+0000 7fa445400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4480c5f20 0x7fa4481eb320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.148+0000 7fa445400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4480c5f20 0x7fa4481eb320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58316/0 (socket says 172.21.15.28:58316) 2024-02-21T23:07:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.148+0000 7fa445400640 1 -- 172.21.15.28:0/2129276449 learned_addr learned my addr 172.21.15.28:0/2129276449 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.148+0000 7fa445e00640 1 --2- 172.21.15.28:0/2129276449 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa4480c2df0 0x7fa4480bfc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.148+0000 7fa445400640 1 -- 172.21.15.28:0/2129276449 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa448153220 msgr2=0x7fa4481eb860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.148+0000 7fa446800640 1 --2- 172.21.15.28:0/2129276449 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa448153220 0x7fa4481eb860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.148+0000 7fa445400640 1 --2- 172.21.15.28:0/2129276449 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa448153220 0x7fa4481eb860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.149+0000 7fa445400640 1 -- 172.21.15.28:0/2129276449 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa4480c2df0 msgr2=0x7fa4480bfc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.149+0000 7fa445400640 1 --2- 172.21.15.28:0/2129276449 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa4480c2df0 0x7fa4480bfc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.149+0000 7fa445400640 1 -- 172.21.15.28:0/2129276449 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4480723d0 con 0x7fa4480c5f20 2024-02-21T23:07:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.149+0000 7fa445e00640 1 --2- 172.21.15.28:0/2129276449 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa4480c2df0 0x7fa4480bfc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:07:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.149+0000 7fa445400640 1 --2- 172.21.15.28:0/2129276449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4480c5f20 0x7fa4481eb320 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7fa43400d7d0 tx=0x7fa43400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.149+0000 7fa43e000640 1 -- 172.21.15.28:0/2129276449 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa434004030 con 0x7fa4480c5f20 2024-02-21T23:07:39.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.149+0000 7fa43e000640 1 -- 172.21.15.28:0/2129276449 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4340041d0 con 0x7fa4480c5f20 2024-02-21T23:07:39.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.149+0000 7fa43e000640 1 -- 172.21.15.28:0/2129276449 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa43400b6c0 con 0x7fa4480c5f20 2024-02-21T23:07:39.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.149+0000 7fa44d200640 1 -- 172.21.15.28:0/2129276449 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4481ee870 con 0x7fa4480c5f20 2024-02-21T23:07:39.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.149+0000 7fa44d200640 1 -- 172.21.15.28:0/2129276449 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4481eee30 con 0x7fa4480c5f20 2024-02-21T23:07:39.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.151+0000 7fa43e000640 1 -- 172.21.15.28:0/2129276449 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa43400b860 con 0x7fa4480c5f20 2024-02-21T23:07:39.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.152+0000 7fa43e000640 1 --2- 172.21.15.28:0/2129276449 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa42806b4d0 0x7fa42806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:39.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.152+0000 7fa43e000640 1 -- 172.21.15.28:0/2129276449 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa43408a6a0 con 0x7fa4480c5f20 2024-02-21T23:07:39.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.153+0000 7fa445e00640 1 --2- 172.21.15.28:0/2129276449 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa42806b4d0 0x7fa42806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:39.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.153+0000 7fa450174740 1 Processor -- start 2024-02-21T23:07:39.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.153+0000 7fa450174740 1 -- start start 2024-02-21T23:07:39.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.154+0000 7fa445e00640 1 --2- 172.21.15.28:0/2129276449 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa42806b4d0 0x7fa42806d990 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fa4480c0740 tx=0x7fa438008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:39.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.154+0000 7fa450174740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ec0b42be40 0x55ec0b42c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:39.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.154+0000 7fa450174740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec0b42d150 0x55ec0b467f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:39.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.154+0000 7fa450174740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55ec0b4684d0 0x55ec0b466540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:39.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.154+0000 7fa450174740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec0b390470 con 0x55ec0b42d150 2024-02-21T23:07:39.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.154+0000 7fa450174740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec0b339af0 con 0x55ec0b42be40 2024-02-21T23:07:39.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.154+0000 7fa450174740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ec0b38c510 con 0x55ec0b4684d0 2024-02-21T23:07:39.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.155+0000 7fa445400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec0b42d150 0x55ec0b467f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:39.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.155+0000 7fa445400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec0b42d150 0x55ec0b467f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58324/0 (socket says 172.21.15.28:58324) 2024-02-21T23:07:39.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.155+0000 7fa445400640 1 -- 172.21.15.28:0/1692435179 learned_addr learned my addr 172.21.15.28:0/1692435179 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:39.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.155+0000 7fa445400640 1 -- 172.21.15.28:0/1692435179 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55ec0b4684d0 msgr2=0x55ec0b466540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:39.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.155+0000 7fa445400640 1 --2- 172.21.15.28:0/1692435179 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55ec0b4684d0 0x55ec0b466540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:39.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.155+0000 7fa445400640 1 -- 172.21.15.28:0/1692435179 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ec0b42be40 msgr2=0x55ec0b42c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:39.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.155+0000 7fa445400640 1 --2- 172.21.15.28:0/1692435179 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ec0b42be40 0x55ec0b42c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:39.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.155+0000 7fa445400640 1 -- 172.21.15.28:0/1692435179 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ec0b340d90 con 0x55ec0b42d150 2024-02-21T23:07:39.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.155+0000 7fa445400640 1 --2- 172.21.15.28:0/1692435179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec0b42d150 0x55ec0b467f90 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7fa434088910 tx=0x7fa434058860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d68138b22160e5f8 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:39.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.156+0000 7fa432a00640 1 -- 172.21.15.28:0/1692435179 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa43405cd40 con 0x55ec0b42d150 2024-02-21T23:07:39.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.156+0000 7fa432a00640 1 -- 172.21.15.28:0/1692435179 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa434061300 con 0x55ec0b42d150 2024-02-21T23:07:39.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.156+0000 7fa432a00640 1 -- 172.21.15.28:0/1692435179 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa434056ef0 con 0x55ec0b42d150 2024-02-21T23:07:39.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.156+0000 7fa450174740 1 -- 172.21.15.28:0/1692435179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec0b42d150 msgr2=0x55ec0b467f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:39.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.156+0000 7fa450174740 1 --2- 172.21.15.28:0/1692435179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec0b42d150 0x55ec0b467f90 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7fa434088910 tx=0x7fa434058860 comp rx=0 tx=0).stop 2024-02-21T23:07:39.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.156+0000 7fa450174740 1 -- 172.21.15.28:0/1692435179 shutdown_connections 2024-02-21T23:07:39.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.156+0000 7fa450174740 1 --2- 172.21.15.28:0/1692435179 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55ec0b4684d0 0x55ec0b466540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:39.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.156+0000 7fa450174740 1 --2- 172.21.15.28:0/1692435179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec0b42d150 0x55ec0b467f90 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:39.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.156+0000 7fa450174740 1 --2- 172.21.15.28:0/1692435179 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ec0b42be40 0x55ec0b42c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:39.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.156+0000 7fa450174740 1 -- 172.21.15.28:0/1692435179 >> 172.21.15.28:0/1692435179 conn(0x55ec0b428790 msgr2=0x55ec0b44d1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:39.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.156+0000 7fa450174740 1 -- 172.21.15.28:0/1692435179 shutdown_connections 2024-02-21T23:07:39.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.156+0000 7fa450174740 1 -- 172.21.15.28:0/1692435179 wait complete. 2024-02-21T23:07:39.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.157+0000 7fa450174740 1 -- start start 2024-02-21T23:07:39.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.157+0000 7fa450174740 1 Processor -- start 2024-02-21T23:07:39.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.158+0000 7fa450174740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55ec0b42d150 0x55ec0b47b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:39.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.158+0000 7fa450174740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec0b4684d0 0x55ec0b44c170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:39.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.158+0000 7fa450174740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ec0b448490 0x55ec0b447220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:39.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.158+0000 7fa450174740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec0b32f040 con 0x55ec0b4684d0 2024-02-21T23:07:39.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.158+0000 7fa450174740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec0b44d8b0 con 0x55ec0b448490 2024-02-21T23:07:39.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.158+0000 7fa450174740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ec0b46da60 con 0x55ec0b42d150 2024-02-21T23:07:39.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.158+0000 7fa450174740 10 client.0 _ll_register_callbacks cb 0x55ec0b34c920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:07:39.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.159+0000 7fa445400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec0b4684d0 0x55ec0b44c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:39.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.159+0000 7fa445400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec0b4684d0 0x55ec0b44c170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58334/0 (socket says 172.21.15.28:58334) 2024-02-21T23:07:39.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.159+0000 7fa445400640 1 -- 172.21.15.28:0/930432686 learned_addr learned my addr 172.21.15.28:0/930432686 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:39.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.159+0000 7fa446800640 1 --2- 172.21.15.28:0/930432686 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ec0b448490 0x55ec0b447220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:39.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.159+0000 7fa445400640 1 -- 172.21.15.28:0/930432686 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55ec0b42d150 msgr2=0x55ec0b47b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:39.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.159+0000 7fa445e00640 1 --2- 172.21.15.28:0/930432686 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55ec0b42d150 0x55ec0b47b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:39.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.159+0000 7fa445400640 1 --2- 172.21.15.28:0/930432686 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55ec0b42d150 0x55ec0b47b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:39.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.159+0000 7fa445400640 1 -- 172.21.15.28:0/930432686 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ec0b448490 msgr2=0x55ec0b447220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:39.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.159+0000 7fa445400640 1 --2- 172.21.15.28:0/930432686 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ec0b448490 0x55ec0b447220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:39.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.159+0000 7fa445400640 1 -- 172.21.15.28:0/930432686 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ec0b34eaa0 con 0x55ec0b4684d0 2024-02-21T23:07:39.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.159+0000 7fa446800640 1 --2- 172.21.15.28:0/930432686 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ec0b448490 0x55ec0b447220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:07:39.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.160+0000 7fa445400640 1 --2- 172.21.15.28:0/930432686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec0b4684d0 0x55ec0b44c170 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7fa43400b4b0 tx=0x7fa43405b070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6865a9763342ca6 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:39.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.160+0000 7fa427400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:07:39.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.160+0000 7fa427400640 1 -- 172.21.15.28:0/930432686 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa434058730 con 0x55ec0b4684d0 2024-02-21T23:07:39.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.160+0000 7fa427400640 1 -- 172.21.15.28:0/930432686 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(110..110 src has 1..110) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa434055260 con 0x55ec0b4684d0 2024-02-21T23:07:39.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.160+0000 7fa450174740 1 -- 172.21.15.28:0/930432686 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa434061300 con 0x55ec0b4684d0 2024-02-21T23:07:39.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.160+0000 7fa427400640 1 -- 172.21.15.28:0/930432686 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=102) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa43405cd40 con 0x55ec0b4684d0 2024-02-21T23:07:39.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.160+0000 7fa450174740 10 client.7617 fetch_fsmap learned FSMap version 102 2024-02-21T23:07:39.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.160+0000 7fa450174740 1 -- 172.21.15.28:0/930432686 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=102,monmap=2+}) v3 -- 0x55ec0b0a52d0 con 0x55ec0b4684d0 2024-02-21T23:07:39.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.160+0000 7fa427400640 1 -- 172.21.15.28:0/930432686 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa434061300 con 0x55ec0b4684d0 2024-02-21T23:07:39.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.160+0000 7fa427400640 1 -- 172.21.15.28:0/930432686 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 102) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fa434013070 con 0x55ec0b4684d0 2024-02-21T23:07:39.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.160+0000 7fa450174740 10 client.7617 fetch_fsmap finished waiting for FSMap version 102 2024-02-21T23:07:39.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.161+0000 7fa450174740 10 client.7617 resolve_mds: resolved name 'a' to daemon mds.a 2024-02-21T23:07:39.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.161+0000 7fa450174740 20 client.7617 populate_metadata read hostname 'smithi028' 2024-02-21T23:07:39.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.161+0000 7fa450174740 1 --2- 172.21.15.28:0/930432686 >> [v2:172.21.15.28:6834/1259832430,v1:172.21.15.28:6835/1259832430] conn(0x55ec0b478640 0x55ec0b478a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:39.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.161+0000 7fa450174740 4 client.7617 mds_command: new command op to 7458 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-21T23:07:39.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.161+0000 7fa450174740 1 -- 172.21.15.28:0/930432686 --> [v2:172.21.15.28:6834/1259832430,v1:172.21.15.28:6835/1259832430] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ec0b39d9b0 con 0x55ec0b478640 2024-02-21T23:07:39.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.161+0000 7fa445e00640 1 --2- 172.21.15.28:0/930432686 >> [v2:172.21.15.28:6834/1259832430,v1:172.21.15.28:6835/1259832430] conn(0x55ec0b478640 0x55ec0b478a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:39.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.162+0000 7fa445e00640 1 --2- 172.21.15.28:0/930432686 >> [v2:172.21.15.28:6834/1259832430,v1:172.21.15.28:6835/1259832430] conn(0x55ec0b478640 0x55ec0b478a20 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ee504ba9fa17d96 server_cookie=824bf9e28ad7742b in_seq=0 out_seq=0 2024-02-21T23:07:39.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.162+0000 7fa427400640 10 client.7617 ms_handle_connect on v2:172.21.15.28:6834/1259832430 2024-02-21T23:07:39.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.162+0000 7fa427400640 1 -- 172.21.15.28:0/930432686 <== mds.0 v2:172.21.15.28:6834/1259832430 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55ec0b39d9b0 con 0x55ec0b478640 2024-02-21T23:07:39.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.162+0000 7fa427400640 10 client.7617 handle_command_reply: tid=0 2024-02-21T23:07:39.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.162+0000 7fa450174740 1 client.7617 shutdown 2024-02-21T23:07:39.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.163+0000 7fa450174740 1 -- 172.21.15.28:0/930432686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec0b4684d0 msgr2=0x55ec0b44c170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:39.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.163+0000 7fa450174740 1 --2- 172.21.15.28:0/930432686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec0b4684d0 0x55ec0b44c170 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7fa43400b4b0 tx=0x7fa43405b070 comp rx=0 tx=0).stop 2024-02-21T23:07:39.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.163+0000 7fa450174740 1 -- 172.21.15.28:0/930432686 shutdown_connections 2024-02-21T23:07:39.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.163+0000 7fa450174740 1 --2- 172.21.15.28:0/930432686 >> [v2:172.21.15.28:6834/1259832430,v1:172.21.15.28:6835/1259832430] conn(0x55ec0b478640 0x55ec0b478a20 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:39.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.163+0000 7fa450174740 1 --2- 172.21.15.28:0/930432686 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ec0b448490 0x55ec0b447220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:39.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.163+0000 7fa450174740 1 --2- 172.21.15.28:0/930432686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec0b4684d0 0x55ec0b44c170 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:39.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.163+0000 7fa450174740 1 --2- 172.21.15.28:0/930432686 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55ec0b42d150 0x55ec0b47b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:39.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.163+0000 7fa427400640 0 client.7617 ms_handle_reset on v2:172.21.15.28:6834/1259832430 2024-02-21T23:07:39.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.163+0000 7fa450174740 1 -- 172.21.15.28:0/930432686 >> 172.21.15.28:0/930432686 conn(0x55ec0b42be40 msgr2=0x55ec0b466e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:39.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.163+0000 7fa450174740 1 -- 172.21.15.28:0/930432686 shutdown_connections 2024-02-21T23:07:39.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.163+0000 7fa450174740 1 -- 172.21.15.28:0/930432686 wait complete. 2024-02-21T23:07:39.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.163+0000 7fa450174740 20 client.7617 trim_cache size 0 max 16384 2024-02-21T23:07:39.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.181+0000 7fa450174740 1 Processor -- start 2024-02-21T23:07:39.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.182+0000 7fa450174740 1 -- start start 2024-02-21T23:07:39.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.182+0000 7fa450174740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55ec0b42d150 0x55ec0b447250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:39.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.182+0000 7fa450174740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ec0b448490 0x55ec0b448be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:39.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.183+0000 7fa450174740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec0b4684d0 0x55ec0b4255f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:39.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.183+0000 7fa450174740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4481f6f10 con 0x55ec0b4684d0 2024-02-21T23:07:39.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.183+0000 7fa450174740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4480ab690 con 0x55ec0b448490 2024-02-21T23:07:39.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.183+0000 7fa450174740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4481f6910 con 0x55ec0b42d150 2024-02-21T23:07:39.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.183+0000 7fa446800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec0b4684d0 0x55ec0b4255f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:39.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.183+0000 7fa446800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec0b4684d0 0x55ec0b4255f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58336/0 (socket says 172.21.15.28:58336) 2024-02-21T23:07:39.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.183+0000 7fa446800640 1 -- 172.21.15.28:0/374009282 learned_addr learned my addr 172.21.15.28:0/374009282 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:39.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.183+0000 7fa446800640 1 -- 172.21.15.28:0/374009282 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55ec0b42d150 msgr2=0x55ec0b447250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:39.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.183+0000 7fa445e00640 1 --2- 172.21.15.28:0/374009282 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55ec0b42d150 0x55ec0b447250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:39.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.183+0000 7fa445400640 1 --2- 172.21.15.28:0/374009282 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ec0b448490 0x55ec0b448be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:39.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.183+0000 7fa446800640 1 --2- 172.21.15.28:0/374009282 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55ec0b42d150 0x55ec0b447250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:39.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.183+0000 7fa446800640 1 -- 172.21.15.28:0/374009282 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ec0b448490 msgr2=0x55ec0b448be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:39.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.183+0000 7fa446800640 1 --2- 172.21.15.28:0/374009282 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ec0b448490 0x55ec0b448be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:39.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.183+0000 7fa446800640 1 -- 172.21.15.28:0/374009282 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ec0b47bca0 con 0x55ec0b4684d0 2024-02-21T23:07:39.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.184+0000 7fa446800640 1 --2- 172.21.15.28:0/374009282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec0b4684d0 0x55ec0b4255f0 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7fa44002c4f0 tx=0x7fa44002e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=698d1d6438d65f6b server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:39.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.184+0000 7fa432000640 1 -- 172.21.15.28:0/374009282 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa440044070 con 0x55ec0b4684d0 2024-02-21T23:07:39.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.184+0000 7fa432000640 1 -- 172.21.15.28:0/374009282 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa44002ca80 con 0x55ec0b4684d0 2024-02-21T23:07:39.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.184+0000 7fa432000640 1 -- 172.21.15.28:0/374009282 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa440009510 con 0x55ec0b4684d0 2024-02-21T23:07:39.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.185+0000 7fa450174740 1 -- 172.21.15.28:0/374009282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec0b4684d0 msgr2=0x55ec0b4255f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:39.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.185+0000 7fa450174740 1 --2- 172.21.15.28:0/374009282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec0b4684d0 0x55ec0b4255f0 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7fa44002c4f0 tx=0x7fa44002e070 comp rx=0 tx=0).stop 2024-02-21T23:07:39.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.185+0000 7fa450174740 1 -- 172.21.15.28:0/374009282 shutdown_connections 2024-02-21T23:07:39.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.185+0000 7fa450174740 1 --2- 172.21.15.28:0/374009282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec0b4684d0 0x55ec0b4255f0 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:39.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.185+0000 7fa450174740 1 --2- 172.21.15.28:0/374009282 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ec0b448490 0x55ec0b448be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:39.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.185+0000 7fa450174740 1 --2- 172.21.15.28:0/374009282 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55ec0b42d150 0x55ec0b447250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:39.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.185+0000 7fa450174740 1 -- 172.21.15.28:0/374009282 >> 172.21.15.28:0/374009282 conn(0x55ec0b42be40 msgr2=0x55ec0b466e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:39.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.185+0000 7fa450174740 1 -- 172.21.15.28:0/374009282 shutdown_connections 2024-02-21T23:07:39.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.185+0000 7fa450174740 1 -- 172.21.15.28:0/374009282 wait complete. 2024-02-21T23:07:39.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.185+0000 7fa450174740 1 -- start start 2024-02-21T23:07:39.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.186+0000 7fa450174740 1 Processor -- start 2024-02-21T23:07:39.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.187+0000 7fa450174740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55ec0b42d150 0x55ec0b4820d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:39.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.187+0000 7fa450174740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec0b448490 0x55ec0b48b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:39.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.187+0000 7fa450174740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ec0b4684d0 0x55ec0b426250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:39.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.187+0000 7fa450174740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec0b44c7a0 con 0x55ec0b448490 2024-02-21T23:07:39.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.187+0000 7fa450174740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec0b32f040 con 0x55ec0b4684d0 2024-02-21T23:07:39.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.187+0000 7fa450174740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ec0b447950 con 0x55ec0b42d150 2024-02-21T23:07:39.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.187+0000 7fa450174740 10 client.0 _ll_register_callbacks cb 0x7fa434058d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:07:39.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.188+0000 7fa445400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec0b448490 0x55ec0b48b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:39.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.188+0000 7fa445400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec0b448490 0x55ec0b48b4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58350/0 (socket says 172.21.15.28:58350) 2024-02-21T23:07:39.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.188+0000 7fa445400640 1 -- 172.21.15.28:0/3116145955 learned_addr learned my addr 172.21.15.28:0/3116145955 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:39.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.188+0000 7fa445400640 1 -- 172.21.15.28:0/3116145955 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55ec0b42d150 msgr2=0x55ec0b4820d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:39.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.188+0000 7fa445e00640 1 --2- 172.21.15.28:0/3116145955 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55ec0b42d150 0x55ec0b4820d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:39.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.188+0000 7fa446800640 1 --2- 172.21.15.28:0/3116145955 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ec0b4684d0 0x55ec0b426250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:39.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.188+0000 7fa445400640 1 --2- 172.21.15.28:0/3116145955 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55ec0b42d150 0x55ec0b4820d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:39.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.188+0000 7fa445400640 1 -- 172.21.15.28:0/3116145955 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ec0b4684d0 msgr2=0x55ec0b426250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:39.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.188+0000 7fa445400640 1 --2- 172.21.15.28:0/3116145955 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ec0b4684d0 0x55ec0b426250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:39.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.188+0000 7fa445400640 1 -- 172.21.15.28:0/3116145955 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ec0b42c590 con 0x55ec0b448490 2024-02-21T23:07:39.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.188+0000 7fa445400640 1 --2- 172.21.15.28:0/3116145955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec0b448490 0x55ec0b48b4e0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7fa434055570 tx=0x7fa434054fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23c5f697a4697649 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:39.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.188+0000 7fa427400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:07:39.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.189+0000 7fa427400640 1 -- 172.21.15.28:0/3116145955 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa434058730 con 0x55ec0b448490 2024-02-21T23:07:39.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.189+0000 7fa427400640 1 -- 172.21.15.28:0/3116145955 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(110..110 src has 1..110) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa43400bb50 con 0x55ec0b448490 2024-02-21T23:07:39.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.189+0000 7fa450174740 1 -- 172.21.15.28:0/3116145955 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa44002ca80 con 0x55ec0b448490 2024-02-21T23:07:39.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.189+0000 7fa427400640 1 -- 172.21.15.28:0/3116145955 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=102) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa434061300 con 0x55ec0b448490 2024-02-21T23:07:39.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.189+0000 7fa450174740 10 client.7623 fetch_fsmap learned FSMap version 102 2024-02-21T23:07:39.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.189+0000 7fa450174740 1 -- 172.21.15.28:0/3116145955 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=102,monmap=2+}) v3 -- 0x55ec0b16f6b0 con 0x55ec0b448490 2024-02-21T23:07:39.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.189+0000 7fa427400640 1 -- 172.21.15.28:0/3116145955 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa44002ca80 con 0x55ec0b448490 2024-02-21T23:07:39.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.189+0000 7fa427400640 1 -- 172.21.15.28:0/3116145955 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 102) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fa434058040 con 0x55ec0b448490 2024-02-21T23:07:39.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.189+0000 7fa450174740 10 client.7623 fetch_fsmap finished waiting for FSMap version 102 2024-02-21T23:07:39.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.189+0000 7fa450174740 10 client.7623 resolve_mds: resolved name 'a' to daemon mds.a 2024-02-21T23:07:39.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.189+0000 7fa450174740 20 client.7623 populate_metadata read hostname 'smithi028' 2024-02-21T23:07:39.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.189+0000 7fa450174740 1 --2- 172.21.15.28:0/3116145955 >> [v2:172.21.15.28:6834/1259832430,v1:172.21.15.28:6835/1259832430] conn(0x55ec0b426790 0x55ec0b426b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:39.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.189+0000 7fa450174740 4 client.7623 mds_command: new command op to 7458 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-21T23:07:39.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.189+0000 7fa450174740 1 -- 172.21.15.28:0/3116145955 --> [v2:172.21.15.28:6834/1259832430,v1:172.21.15.28:6835/1259832430] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55ec0b397d00 con 0x55ec0b426790 2024-02-21T23:07:39.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.190+0000 7fa445e00640 1 --2- 172.21.15.28:0/3116145955 >> [v2:172.21.15.28:6834/1259832430,v1:172.21.15.28:6835/1259832430] conn(0x55ec0b426790 0x55ec0b426b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:39.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.190+0000 7fa445e00640 1 --2- 172.21.15.28:0/3116145955 >> [v2:172.21.15.28:6834/1259832430,v1:172.21.15.28:6835/1259832430] conn(0x55ec0b426790 0x55ec0b426b70 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db8e4901468050a2 server_cookie=a057a621b35bdbd8 in_seq=0 out_seq=0 2024-02-21T23:07:39.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.190+0000 7fa427400640 10 client.7623 ms_handle_connect on v2:172.21.15.28:6834/1259832430 2024-02-21T23:07:39.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.191+0000 7fa427400640 1 -- 172.21.15.28:0/3116145955 <== mds.0 v2:172.21.15.28:6834/1259832430 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+328 (crc 0 0 0) 0x55ec0b397d00 con 0x55ec0b426790 2024-02-21T23:07:39.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.191+0000 7fa427400640 10 client.7623 handle_command_reply: tid=0 2024-02-21T23:07:39.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.191+0000 7fa450174740 1 client.7623 shutdown 2024-02-21T23:07:39.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.191+0000 7fa450174740 1 -- 172.21.15.28:0/3116145955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec0b448490 msgr2=0x55ec0b48b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:39.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.191+0000 7fa450174740 1 --2- 172.21.15.28:0/3116145955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec0b448490 0x55ec0b48b4e0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7fa434055570 tx=0x7fa434054fc0 comp rx=0 tx=0).stop 2024-02-21T23:07:39.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.191+0000 7fa450174740 1 -- 172.21.15.28:0/3116145955 shutdown_connections 2024-02-21T23:07:39.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.191+0000 7fa450174740 1 --2- 172.21.15.28:0/3116145955 >> [v2:172.21.15.28:6834/1259832430,v1:172.21.15.28:6835/1259832430] conn(0x55ec0b426790 0x55ec0b426b70 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:39.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.191+0000 7fa450174740 1 --2- 172.21.15.28:0/3116145955 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55ec0b4684d0 0x55ec0b426250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:39.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.191+0000 7fa450174740 1 --2- 172.21.15.28:0/3116145955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ec0b448490 0x55ec0b48b4e0 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:39.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.191+0000 7fa450174740 1 --2- 172.21.15.28:0/3116145955 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55ec0b42d150 0x55ec0b4820d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:39.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.191+0000 7fa450174740 1 -- 172.21.15.28:0/3116145955 >> 172.21.15.28:0/3116145955 conn(0x55ec0b42be40 msgr2=0x55ec0b44b880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:39.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.191+0000 7fa427400640 0 client.7623 ms_handle_reset on v2:172.21.15.28:6834/1259832430 2024-02-21T23:07:39.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.191+0000 7fa450174740 1 -- 172.21.15.28:0/3116145955 shutdown_connections 2024-02-21T23:07:39.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.191+0000 7fa450174740 1 -- 172.21.15.28:0/3116145955 wait complete. 2024-02-21T23:07:39.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.191+0000 7fa450174740 20 client.7623 trim_cache size 0 max 16384 2024-02-21T23:07:39.183 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-21T23:07:39.183 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac", 2024-02-21T23:07:39.183 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-02-21T23:07:39.183 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 7458, 2024-02-21T23:07:39.183 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-02-21T23:07:39.183 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-02-21T23:07:39.183 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-02-21T23:07:39.183 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 9.1364424599999996, 2024-02-21T23:07:39.183 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 102, 2024-02-21T23:07:39.183 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 110, 2024-02-21T23:07:39.183 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 110, 2024-02-21T23:07:39.184 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 14.603071923 2024-02-21T23:07:39.184 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-21T23:07:39.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.192+0000 7fa44d200640 1 -- 172.21.15.28:0/2129276449 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa42806b4d0 msgr2=0x7fa42806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:39.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.192+0000 7fa44d200640 1 --2- 172.21.15.28:0/2129276449 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa42806b4d0 0x7fa42806d990 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fa4480c0740 tx=0x7fa438008040 comp rx=0 tx=0).stop 2024-02-21T23:07:39.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.192+0000 7fa44d200640 1 -- 172.21.15.28:0/2129276449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4480c5f20 msgr2=0x7fa4481eb320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:39.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.192+0000 7fa44d200640 1 --2- 172.21.15.28:0/2129276449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4480c5f20 0x7fa4481eb320 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7fa43400d7d0 tx=0x7fa43400dc90 comp rx=0 tx=0).stop 2024-02-21T23:07:39.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.192+0000 7fa44d200640 1 -- 172.21.15.28:0/2129276449 shutdown_connections 2024-02-21T23:07:39.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.192+0000 7fa44d200640 1 --2- 172.21.15.28:0/2129276449 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa42806b4d0 0x7fa42806d990 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:39.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.192+0000 7fa44d200640 1 --2- 172.21.15.28:0/2129276449 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa448153220 0x7fa4481eb860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:39.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.192+0000 7fa44d200640 1 --2- 172.21.15.28:0/2129276449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4480c5f20 0x7fa4481eb320 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:39.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.192+0000 7fa44d200640 1 --2- 172.21.15.28:0/2129276449 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa4480c2df0 0x7fa4480bfc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:39.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.192+0000 7fa44d200640 1 -- 172.21.15.28:0/2129276449 >> 172.21.15.28:0/2129276449 conn(0x7fa44806b820 msgr2=0x7fa4480c43e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:39.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.192+0000 7fa44d200640 1 -- 172.21.15.28:0/2129276449 shutdown_connections 2024-02-21T23:07:39.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:39.192+0000 7fa44d200640 1 -- 172.21.15.28:0/2129276449 wait complete. 2024-02-21T23:07:39.197 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 8, "epoch": 102, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 8, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-02-21T23:07:29.533547+0000", "damaged": [], "data_pools": [ 18 ], "enabled": true, "epoch": 102, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7458": { "addr": "172.21.15.28:6835/1259832430", "addrs": { "addrvec": [ { "addr": "172.21.15.28:6834", "nonce": 1259832430, "type": "v2" }, { "addr": "172.21.15.28:6835", "nonce": 1259832430, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7458, "incarnation": 98, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 17, "modified": "2024-02-21T23:07:32.591216+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7458 } } } ], "standbys": [ { "addr": "172.21.15.81:6835/1754860807", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6834", "nonce": 1754860807, "type": "v2" }, { "addr": "172.21.15.81:6835", "nonce": 1754860807, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 97, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5729, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.81:6837/3963422563", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6836", "nonce": 3963422563, "type": "v2" }, { "addr": "172.21.15.81:6837", "nonce": 3963422563, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 97, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5732, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.28:6833/1119726035", "addrs": { "addrvec": [ { "addr": "172.21.15.28:6832", "nonce": 1119726035, "type": "v2" }, { "addr": "172.21.15.28:6833", "nonce": 1119726035, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 96, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7446, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-02-21T23:07:39.198 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': 8, 'name': 'd', 'state': 'up:active'}] 2024-02-21T23:07:39.198 INFO:tasks.cephfs.test_failover:current = [{'gid': 7458, 'name': 'a', 'rank': 0, 'incarnation': 98, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/1259832430', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 1259832430}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 1259832430}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 5732, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.81:6837/3963422563', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3963422563}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3963422563}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 97}, {'gid': 7446, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.28:6833/1119726035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 1119726035}, {'type': 'v1', 'addr': '172.21.15.28:6833', 'nonce': 1119726035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 96}, {'gid': 5729, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.81:6835/1754860807', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6834', 'nonce': 1754860807}, {'type': 'v1', 'addr': '172.21.15.81:6835', 'nonce': 1754860807}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 97}] 2024-02-21T23:07:39.198 DEBUG:tasks.cephfs.test_failover:8 != -1 2024-02-21T23:07:39.198 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-02-21T23:07:41.156 INFO:tasks.ceph.mds.a.smithi028.stderr: -261> 2024-02-21T23:07:36.009+0000 7f44eb400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-21T23:07:41.221 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-02-21T23:07:44.199 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:07:44.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.441+0000 7f1ceba00640 1 Processor -- start 2024-02-21T23:07:44.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.441+0000 7f1ceba00640 1 -- start start 2024-02-21T23:07:44.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.443+0000 7f1ceba00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1ce40c11f0 0x7f1ce4153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:44.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.443+0000 7f1ceba00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f1ce40c15d0 0x7f1ce406dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:44.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.443+0000 7f1ceba00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1ce4156350 0x7f1ce4156780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:44.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.443+0000 7f1ceba00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ce40c1a60 con 0x7f1ce40c11f0 2024-02-21T23:07:44.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.443+0000 7f1ceba00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ce40a5ec0 con 0x7f1ce40c15d0 2024-02-21T23:07:44.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.443+0000 7f1ceba00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce4152e70 con 0x7f1ce4156350 2024-02-21T23:07:44.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.443+0000 7f1ce9000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1ce40c11f0 0x7f1ce4153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:44.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.443+0000 7f1ce9000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1ce40c11f0 0x7f1ce4153950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58370/0 (socket says 172.21.15.28:58370) 2024-02-21T23:07:44.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.443+0000 7f1ce9000640 1 -- 172.21.15.28:0/851252503 learned_addr learned my addr 172.21.15.28:0/851252503 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:44.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.444+0000 7f1ce9a00640 1 --2- 172.21.15.28:0/851252503 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1ce4156350 0x7f1ce4156780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:44.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.444+0000 7f1ce3e00640 1 --2- 172.21.15.28:0/851252503 >> v2:172.21.15.81:3300/0 conn(0x7f1ce40c15d0 0x7f1ce406dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:44.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.444+0000 7f1ce9000640 1 -- 172.21.15.28:0/851252503 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1ce4156350 msgr2=0x7f1ce4156780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:44.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.444+0000 7f1ce9000640 1 --2- 172.21.15.28:0/851252503 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1ce4156350 0x7f1ce4156780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:44.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.444+0000 7f1ce9000640 1 -- 172.21.15.28:0/851252503 >> v2:172.21.15.81:3300/0 conn(0x7f1ce40c15d0 msgr2=0x7f1ce406dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:44.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.444+0000 7f1ce9000640 1 --2- 172.21.15.28:0/851252503 >> v2:172.21.15.81:3300/0 conn(0x7f1ce40c15d0 0x7f1ce406dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:44.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.444+0000 7f1ce9000640 1 -- 172.21.15.28:0/851252503 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ce4153150 con 0x7f1ce40c11f0 2024-02-21T23:07:44.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.446+0000 7f1ce9000640 1 --2- 172.21.15.28:0/851252503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ce40c11f0 0x7f1ce4153950 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f1ce40c24a0 tx=0x7f1cd802b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=832747e703284fd9 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:44.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.446+0000 7f1ce3400640 1 -- 172.21.15.28:0/851252503 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cd8038070 con 0x7f1ce40c11f0 2024-02-21T23:07:44.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.446+0000 7f1ce3400640 1 -- 172.21.15.28:0/851252503 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cd802f050 con 0x7f1ce40c11f0 2024-02-21T23:07:44.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.446+0000 7f1ce3400640 1 -- 172.21.15.28:0/851252503 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cd8031970 con 0x7f1ce40c11f0 2024-02-21T23:07:44.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.447+0000 7f1ceba00640 1 -- 172.21.15.28:0/851252503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ce40c11f0 msgr2=0x7f1ce4153950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:44.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.447+0000 7f1ceba00640 1 --2- 172.21.15.28:0/851252503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ce40c11f0 0x7f1ce4153950 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f1ce40c24a0 tx=0x7f1cd802b7f0 comp rx=0 tx=0).stop 2024-02-21T23:07:44.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.447+0000 7f1ceba00640 1 -- 172.21.15.28:0/851252503 shutdown_connections 2024-02-21T23:07:44.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.447+0000 7f1ceba00640 1 --2- 172.21.15.28:0/851252503 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1ce4156350 0x7f1ce4156780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:44.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.447+0000 7f1ceba00640 1 --2- 172.21.15.28:0/851252503 >> v2:172.21.15.81:3300/0 conn(0x7f1ce40c15d0 0x7f1ce406dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:44.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.447+0000 7f1ceba00640 1 --2- 172.21.15.28:0/851252503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ce40c11f0 0x7f1ce4153950 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:44.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.447+0000 7f1ceba00640 1 -- 172.21.15.28:0/851252503 >> 172.21.15.28:0/851252503 conn(0x7f1ce40c67f0 msgr2=0x7f1ce40c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:44.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.447+0000 7f1ceba00640 1 -- 172.21.15.28:0/851252503 shutdown_connections 2024-02-21T23:07:44.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.447+0000 7f1ceba00640 1 -- 172.21.15.28:0/851252503 wait complete. 2024-02-21T23:07:44.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.449+0000 7f1ceba00640 1 Processor -- start 2024-02-21T23:07:44.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.449+0000 7f1ceba00640 1 -- start start 2024-02-21T23:07:44.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.450+0000 7f1ceba00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1ce40c11f0 0x7f1ce4146960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:44.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.450+0000 7f1ceba00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ce40c15d0 0x7f1ce40a63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:44.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.450+0000 7f1ceba00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ce4156350 0x7f1ce40a6910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:44.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.450+0000 7f1ceba00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce40c4c00 con 0x7f1ce4156350 2024-02-21T23:07:44.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.450+0000 7f1ceba00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce414d1c0 con 0x7f1ce40c15d0 2024-02-21T23:07:44.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.450+0000 7f1ceba00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce40a6110 con 0x7f1ce40c11f0 2024-02-21T23:07:44.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.450+0000 7f1ce9a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ce4156350 0x7f1ce40a6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:44.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.450+0000 7f1ce9a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ce4156350 0x7f1ce40a6910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58374/0 (socket says 172.21.15.28:58374) 2024-02-21T23:07:44.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.450+0000 7f1ce9a00640 1 -- 172.21.15.28:0/2196850008 learned_addr learned my addr 172.21.15.28:0/2196850008 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:44.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.450+0000 7f1ce3e00640 1 --2- 172.21.15.28:0/2196850008 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ce40c15d0 0x7f1ce40a63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:44.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.450+0000 7f1ce9000640 1 --2- 172.21.15.28:0/2196850008 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1ce40c11f0 0x7f1ce4146960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:44.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.451+0000 7f1ce9a00640 1 -- 172.21.15.28:0/2196850008 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1ce40c11f0 msgr2=0x7f1ce4146960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:44.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.451+0000 7f1ce9a00640 1 --2- 172.21.15.28:0/2196850008 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1ce40c11f0 0x7f1ce4146960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:44.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.451+0000 7f1ce9a00640 1 -- 172.21.15.28:0/2196850008 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ce40c15d0 msgr2=0x7f1ce40a63d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:44.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.451+0000 7f1ce9a00640 1 --2- 172.21.15.28:0/2196850008 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ce40c15d0 0x7f1ce40a63d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:44.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.451+0000 7f1ce9a00640 1 -- 172.21.15.28:0/2196850008 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ce41eacd0 con 0x7f1ce4156350 2024-02-21T23:07:44.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.451+0000 7f1ce9a00640 1 --2- 172.21.15.28:0/2196850008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ce4156350 0x7f1ce40a6910 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc007e40 tx=0x7f1cdc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:44.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.451+0000 7f1ce0c00640 1 -- 172.21.15.28:0/2196850008 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cdc014620 con 0x7f1ce4156350 2024-02-21T23:07:44.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.451+0000 7f1ce0c00640 1 -- 172.21.15.28:0/2196850008 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cdc019070 con 0x7f1ce4156350 2024-02-21T23:07:44.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.451+0000 7f1ceba00640 1 -- 172.21.15.28:0/2196850008 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ce41ee3e0 con 0x7f1ce4156350 2024-02-21T23:07:44.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.452+0000 7f1ce0c00640 1 -- 172.21.15.28:0/2196850008 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cdc014a20 con 0x7f1ce4156350 2024-02-21T23:07:44.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.452+0000 7f1ceba00640 1 -- 172.21.15.28:0/2196850008 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ce41eb270 con 0x7f1ce4156350 2024-02-21T23:07:44.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.453+0000 7f1ceba00640 1 -- 172.21.15.28:0/2196850008 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ce4067f30 con 0x7f1ce4156350 2024-02-21T23:07:44.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.454+0000 7f1ce0c00640 1 -- 172.21.15.28:0/2196850008 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1cdc014bc0 con 0x7f1ce4156350 2024-02-21T23:07:44.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.455+0000 7f1ce0c00640 1 --2- 172.21.15.28:0/2196850008 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1cc406b4d0 0x7f1cc406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:44.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.455+0000 7f1ce0c00640 1 -- 172.21.15.28:0/2196850008 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f1cdc08ac90 con 0x7f1ce4156350 2024-02-21T23:07:44.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.456+0000 7f1ce9000640 1 --2- 172.21.15.28:0/2196850008 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1cc406b4d0 0x7f1cc406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:44.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.457+0000 7f1ce9000640 1 --2- 172.21.15.28:0/2196850008 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1cc406b4d0 0x7f1cc406d990 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f1cd8035000 tx=0x7f1cd8008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:44.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.460+0000 7f1ce0c00640 1 -- 172.21.15.28:0/2196850008 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1cdc0599c0 con 0x7f1ce4156350 2024-02-21T23:07:44.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.699+0000 7f1ceba00640 1 -- 172.21.15.28:0/2196850008 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1ce406f0e0 con 0x7f1ce4156350 2024-02-21T23:07:44.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.701+0000 7f1ce0c00640 1 -- 172.21.15.28:0/2196850008 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+4942 (secure 0 0 0) 0x7f1cdc0592c0 con 0x7f1ce4156350 2024-02-21T23:07:44.689 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:07:44.689 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":107,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3963422563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3963422563},{"type":"v1","addr":"172.21.15.81:6837","nonce":3963422563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7446,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/1119726035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1119726035},{"type":"v1","addr":"172.21.15.28:6833","nonce":1119726035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":96},{"gid":7632,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/780497578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":780497578},{"type":"v1","addr":"172.21.15.28:6835","nonce":780497578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:07:29.533547+0000","modified":"2024-02-21T23:07:43.164772+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5729},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5729":{"gid":5729,"name":"d","rank":0,"incarnation":104,"state":"up:active","state_seq":8,"addr":"172.21.15.81:6835/1754860807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1754860807},{"type":"v1","addr":"172.21.15.81:6835","nonce":1754860807}]},"join_fscid":8,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-02-21T23:07:44.690 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 107 2024-02-21T23:07:44.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.703+0000 7f1ceba00640 1 -- 172.21.15.28:0/2196850008 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1cc406b4d0 msgr2=0x7f1cc406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:44.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.703+0000 7f1ceba00640 1 --2- 172.21.15.28:0/2196850008 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1cc406b4d0 0x7f1cc406d990 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f1cd8035000 tx=0x7f1cd8008030 comp rx=0 tx=0).stop 2024-02-21T23:07:44.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.704+0000 7f1ceba00640 1 -- 172.21.15.28:0/2196850008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ce4156350 msgr2=0x7f1ce40a6910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:44.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.704+0000 7f1ceba00640 1 --2- 172.21.15.28:0/2196850008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ce4156350 0x7f1ce40a6910 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc007e40 tx=0x7f1cdc00e4b0 comp rx=0 tx=0).stop 2024-02-21T23:07:44.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.704+0000 7f1ceba00640 1 -- 172.21.15.28:0/2196850008 shutdown_connections 2024-02-21T23:07:44.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.704+0000 7f1ceba00640 1 --2- 172.21.15.28:0/2196850008 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1cc406b4d0 0x7f1cc406d990 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:44.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.704+0000 7f1ceba00640 1 --2- 172.21.15.28:0/2196850008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ce4156350 0x7f1ce40a6910 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:44.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.704+0000 7f1ceba00640 1 --2- 172.21.15.28:0/2196850008 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ce40c15d0 0x7f1ce40a63d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:44.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.704+0000 7f1ceba00640 1 --2- 172.21.15.28:0/2196850008 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1ce40c11f0 0x7f1ce4146960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:44.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.704+0000 7f1ceba00640 1 -- 172.21.15.28:0/2196850008 >> 172.21.15.28:0/2196850008 conn(0x7f1ce40c67f0 msgr2=0x7f1ce41489f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:44.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.704+0000 7f1ceba00640 1 -- 172.21.15.28:0/2196850008 shutdown_connections 2024-02-21T23:07:44.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:44.704+0000 7f1ceba00640 1 -- 172.21.15.28:0/2196850008 wait complete. 2024-02-21T23:07:45.118 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:07:29.533547+0000', 'modified': '2024-02-21T23:07:43.164772+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 111, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5729}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5729': {'gid': 5729, 'name': 'd', 'rank': 0, 'incarnation': 104, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.81:6835/1754860807', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6834', 'nonce': 1754860807}, {'type': 'v1', 'addr': '172.21.15.81:6835', 'nonce': 1754860807}]}, 'join_fscid': 8, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:07:45.118 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-21T23:07:45.118 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-02-21T23:07:45.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.356+0000 7fe92f000640 1 Processor -- start 2024-02-21T23:07:45.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.356+0000 7fe92f000640 1 -- start start 2024-02-21T23:07:45.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.358+0000 7fe92f000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe92806d410 0x7fe9280a8e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:45.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.358+0000 7fe92f000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe928149c10 con 0x7fe928149830 2024-02-21T23:07:45.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.358+0000 7fe92f000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9281537d0 con 0x7fe9280c43e0 2024-02-21T23:07:45.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.358+0000 7fe92f000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe928153ab0 con 0x7fe92806d410 2024-02-21T23:07:45.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.358+0000 7fe92dc00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe928149830 0x7fe9280c0ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53396/0 (socket says 172.21.15.28:53396) 2024-02-21T23:07:45.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.358+0000 7fe92dc00640 1 -- 172.21.15.28:0/3431703097 learned_addr learned my addr 172.21.15.28:0/3431703097 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:07:45.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.358+0000 7fe92d200640 1 --2- 172.21.15.28:0/3431703097 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe92806d410 0x7fe9280a8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:45.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.359+0000 7fe927e00640 1 -- 172.21.15.28:0/3431703097 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1749814246 0 0) 0x7fe928149c10 con 0x7fe928149830 2024-02-21T23:07:45.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.359+0000 7fe927e00640 1 -- 172.21.15.28:0/3431703097 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe910003520 con 0x7fe928149830 2024-02-21T23:07:45.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.359+0000 7fe927e00640 1 -- 172.21.15.28:0/3431703097 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2570578512 0 0) 0x7fe910003520 con 0x7fe928149830 2024-02-21T23:07:45.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.359+0000 7fe927e00640 1 -- 172.21.15.28:0/3431703097 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe92806d410 msgr2=0x7fe9280a8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:45.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.359+0000 7fe927e00640 1 --2- 172.21.15.28:0/3431703097 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe92806d410 0x7fe9280a8e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:45.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.359+0000 7fe927e00640 1 -- 172.21.15.28:0/3431703097 >> v1:172.21.15.81:6789/0 conn(0x7fe9280c43e0 legacy=0x7fe9280c5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:45.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.359+0000 7fe927e00640 1 -- 172.21.15.28:0/3431703097 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9280c4fa0 con 0x7fe928149830 2024-02-21T23:07:45.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.359+0000 7fe927e00640 1 -- 172.21.15.28:0/3431703097 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fe91c001670 con 0x7fe928149830 2024-02-21T23:07:45.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.359+0000 7fe927e00640 1 -- 172.21.15.28:0/3431703097 >> v1:172.21.15.28:6789/0 conn(0x7fe928149830 legacy=0x7fe9280c0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:45.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.359+0000 7fe927e00640 1 --2- 172.21.15.28:0/3431703097 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9100039e0 0x7fe910003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:45.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.359+0000 7fe927e00640 1 --2- 172.21.15.28:0/3431703097 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe910004800 0x7fe910004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:45.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.359+0000 7fe927e00640 1 --2- 172.21.15.28:0/3431703097 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe92806d410 0x7fe9100051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:45.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.359+0000 7fe927e00640 1 -- 172.21.15.28:0/3431703097 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe928153ab0 con 0x7fe910004800 2024-02-21T23:07:45.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.359+0000 7fe927e00640 1 -- 172.21.15.28:0/3431703097 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe91000bb00 con 0x7fe9100039e0 2024-02-21T23:07:45.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.359+0000 7fe927e00640 1 -- 172.21.15.28:0/3431703097 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe91000bd00 con 0x7fe92806d410 2024-02-21T23:07:45.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.359+0000 7fe92d200640 1 --2- 172.21.15.28:0/3431703097 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe910004800 0x7fe910004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:45.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.360+0000 7fe92dc00640 1 --2- 172.21.15.28:0/3431703097 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9100039e0 0x7fe910003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:45.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.360+0000 7fe92d200640 1 -- 172.21.15.28:0/3431703097 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe92806d410 msgr2=0x7fe9100051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:45.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.360+0000 7fe92d200640 1 --2- 172.21.15.28:0/3431703097 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe92806d410 0x7fe9100051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:45.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.360+0000 7fe92d200640 1 -- 172.21.15.28:0/3431703097 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9100039e0 msgr2=0x7fe910003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:45.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.360+0000 7fe92d200640 1 --2- 172.21.15.28:0/3431703097 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9100039e0 0x7fe910003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:45.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.360+0000 7fe92d200640 1 -- 172.21.15.28:0/3431703097 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe910003520 con 0x7fe910004800 2024-02-21T23:07:45.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.360+0000 7fe92dc00640 1 --2- 172.21.15.28:0/3431703097 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9100039e0 0x7fe910003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:07:45.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.361+0000 7fe92d200640 1 --2- 172.21.15.28:0/3431703097 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe910004800 0x7fe910004c80 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7fe92806d830 tx=0x7fe910005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afed9696eebf8e4c server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:45.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.361+0000 7fe927e00640 1 -- 172.21.15.28:0/3431703097 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe914039070 con 0x7fe910004800 2024-02-21T23:07:45.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.362+0000 7fe927e00640 1 -- 172.21.15.28:0/3431703097 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe91402c630 con 0x7fe910004800 2024-02-21T23:07:45.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.362+0000 7fe927e00640 1 -- 172.21.15.28:0/3431703097 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe914009690 con 0x7fe910004800 2024-02-21T23:07:45.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.362+0000 7fe92f000640 1 -- 172.21.15.28:0/3431703097 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe910004800 msgr2=0x7fe910004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:45.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.362+0000 7fe92f000640 1 --2- 172.21.15.28:0/3431703097 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe910004800 0x7fe910004c80 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7fe92806d830 tx=0x7fe910005830 comp rx=0 tx=0).stop 2024-02-21T23:07:45.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.362+0000 7fe92e600640 1 -- 172.21.15.28:0/3431703097 reap_dead start 2024-02-21T23:07:45.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.362+0000 7fe92f000640 1 -- 172.21.15.28:0/3431703097 shutdown_connections 2024-02-21T23:07:45.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.362+0000 7fe92f000640 1 -- 172.21.15.28:0/3431703097 >> 172.21.15.28:0/3431703097 conn(0x7fe9280a8320 msgr2=0x7fe9280a8730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:45.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.362+0000 7fe92f000640 1 -- 172.21.15.28:0/3431703097 shutdown_connections 2024-02-21T23:07:45.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.362+0000 7fe92f000640 1 -- 172.21.15.28:0/3431703097 wait complete. 2024-02-21T23:07:45.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.364+0000 7fe92f000640 1 Processor -- start 2024-02-21T23:07:45.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.364+0000 7fe92f000640 1 -- start start 2024-02-21T23:07:45.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.365+0000 7fe92f000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe910004800 0x7fe9281ea620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:45.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.365+0000 7fe92f000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe928154120 0x7fe92806da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:45.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.365+0000 7fe92f000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9281ee430 0x7fe9280aaa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:45.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.365+0000 7fe92f000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9280c15f0 con 0x7fe9281ee430 2024-02-21T23:07:45.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.365+0000 7fe92f000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9280c5560 con 0x7fe910004800 2024-02-21T23:07:45.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.365+0000 7fe92f000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9280c1770 con 0x7fe928154120 2024-02-21T23:07:45.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.365+0000 7fe92e600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9281ee430 0x7fe9280aaa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:45.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.365+0000 7fe92e600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9281ee430 0x7fe9280aaa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59804/0 (socket says 172.21.15.28:59804) 2024-02-21T23:07:45.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.366+0000 7fe92e600640 1 -- 172.21.15.28:0/2045045529 learned_addr learned my addr 172.21.15.28:0/2045045529 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:45.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.366+0000 7fe92d200640 1 --2- 172.21.15.28:0/2045045529 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe928154120 0x7fe92806da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:45.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.366+0000 7fe92dc00640 1 --2- 172.21.15.28:0/2045045529 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe910004800 0x7fe9281ea620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:45.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.366+0000 7fe92e600640 1 -- 172.21.15.28:0/2045045529 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe928154120 msgr2=0x7fe92806da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:45.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.366+0000 7fe92e600640 1 --2- 172.21.15.28:0/2045045529 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe928154120 0x7fe92806da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:45.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.366+0000 7fe92e600640 1 -- 172.21.15.28:0/2045045529 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe910004800 msgr2=0x7fe9281ea620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:45.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.366+0000 7fe92e600640 1 --2- 172.21.15.28:0/2045045529 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe910004800 0x7fe9281ea620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:45.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.366+0000 7fe92e600640 1 -- 172.21.15.28:0/2045045529 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9280726d0 con 0x7fe9281ee430 2024-02-21T23:07:45.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.366+0000 7fe92e600640 1 --2- 172.21.15.28:0/2045045529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9281ee430 0x7fe9280aaa10 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7fe9280c0ca0 tx=0x7fe92000cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:45.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.367+0000 7fe925600640 1 -- 172.21.15.28:0/2045045529 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe920017070 con 0x7fe9281ee430 2024-02-21T23:07:45.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.367+0000 7fe925600640 1 -- 172.21.15.28:0/2045045529 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe920005d60 con 0x7fe9281ee430 2024-02-21T23:07:45.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.367+0000 7fe925600640 1 -- 172.21.15.28:0/2045045529 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe92000a7f0 con 0x7fe9281ee430 2024-02-21T23:07:45.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.367+0000 7fe92f000640 1 -- 172.21.15.28:0/2045045529 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9280729b0 con 0x7fe9281ee430 2024-02-21T23:07:45.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.367+0000 7fe92f000640 1 -- 172.21.15.28:0/2045045529 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9281edc30 con 0x7fe9281ee430 2024-02-21T23:07:45.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.369+0000 7fe925600640 1 -- 172.21.15.28:0/2045045529 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe92001a070 con 0x7fe9281ee430 2024-02-21T23:07:45.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.370+0000 7fe925600640 1 --2- 172.21.15.28:0/2045045529 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe90806b4d0 0x7fe90806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:45.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.370+0000 7fe925600640 1 -- 172.21.15.28:0/2045045529 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fe920089600 con 0x7fe9281ee430 2024-02-21T23:07:45.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.371+0000 7fe92dc00640 1 --2- 172.21.15.28:0/2045045529 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe90806b4d0 0x7fe90806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:45.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.371+0000 7fe9335bd740 1 Processor -- start 2024-02-21T23:07:45.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.371+0000 7fe9335bd740 1 -- start start 2024-02-21T23:07:45.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.372+0000 7fe9335bd740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5574f4481020 0x5574f4481400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:45.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.372+0000 7fe9335bd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574f4482330 0x5574f44bd170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:45.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.372+0000 7fe9335bd740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5574f44bd6b0 0x5574f44bb720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:45.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.372+0000 7fe9335bd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574f43e5470 con 0x5574f4482330 2024-02-21T23:07:45.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.372+0000 7fe9335bd740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574f438eaf0 con 0x5574f4481020 2024-02-21T23:07:45.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.372+0000 7fe9335bd740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x5574f43e1510 con 0x5574f44bd6b0 2024-02-21T23:07:45.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.372+0000 7fe92dc00640 1 --2- 172.21.15.28:0/2045045529 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe90806b4d0 0x7fe90806d990 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fe91c000c00 tx=0x7fe91c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:45.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.372+0000 7fe92d200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574f4482330 0x5574f44bd170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:45.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.372+0000 7fe92d200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574f4482330 0x5574f44bd170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59816/0 (socket says 172.21.15.28:59816) 2024-02-21T23:07:45.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.372+0000 7fe92d200640 1 -- 172.21.15.28:0/794308596 learned_addr learned my addr 172.21.15.28:0/794308596 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:45.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.372+0000 7fe92dc00640 1 --2- 172.21.15.28:0/794308596 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5574f4481020 0x5574f4481400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:45.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.372+0000 7fe92e600640 1 --2- 172.21.15.28:0/794308596 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5574f44bd6b0 0x5574f44bb720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:45.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.373+0000 7fe92d200640 1 -- 172.21.15.28:0/794308596 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5574f44bd6b0 msgr2=0x5574f44bb720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:45.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.373+0000 7fe92d200640 1 --2- 172.21.15.28:0/794308596 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5574f44bd6b0 0x5574f44bb720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:45.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.373+0000 7fe92d200640 1 -- 172.21.15.28:0/794308596 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5574f4481020 msgr2=0x5574f4481400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:45.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.373+0000 7fe92d200640 1 --2- 172.21.15.28:0/794308596 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5574f4481020 0x5574f4481400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:45.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.373+0000 7fe92d200640 1 -- 172.21.15.28:0/794308596 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5574f4395d90 con 0x5574f4482330 2024-02-21T23:07:45.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.373+0000 7fe92e600640 1 --2- 172.21.15.28:0/794308596 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5574f44bd6b0 0x5574f44bb720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:07:45.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.373+0000 7fe92dc00640 1 --2- 172.21.15.28:0/794308596 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5574f4481020 0x5574f4481400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:07:45.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.373+0000 7fe92d200640 1 --2- 172.21.15.28:0/794308596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574f4482330 0x5574f44bd170 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7fe9280a9380 tx=0x7fe91402e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26050bdf92897767 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:45.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.373+0000 7fe91a000640 1 -- 172.21.15.28:0/794308596 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe914039040 con 0x5574f4482330 2024-02-21T23:07:45.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.373+0000 7fe91a000640 1 -- 172.21.15.28:0/794308596 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe914005990 con 0x5574f4482330 2024-02-21T23:07:45.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.373+0000 7fe91a000640 1 -- 172.21.15.28:0/794308596 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe914005d60 con 0x5574f4482330 2024-02-21T23:07:45.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.374+0000 7fe9335bd740 1 -- 172.21.15.28:0/794308596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574f4482330 msgr2=0x5574f44bd170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:45.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.374+0000 7fe9335bd740 1 --2- 172.21.15.28:0/794308596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574f4482330 0x5574f44bd170 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7fe9280a9380 tx=0x7fe91402e000 comp rx=0 tx=0).stop 2024-02-21T23:07:45.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.374+0000 7fe9335bd740 1 -- 172.21.15.28:0/794308596 shutdown_connections 2024-02-21T23:07:45.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.374+0000 7fe9335bd740 1 --2- 172.21.15.28:0/794308596 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5574f44bd6b0 0x5574f44bb720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:45.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.374+0000 7fe9335bd740 1 --2- 172.21.15.28:0/794308596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574f4482330 0x5574f44bd170 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:45.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.374+0000 7fe9335bd740 1 --2- 172.21.15.28:0/794308596 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5574f4481020 0x5574f4481400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:45.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.374+0000 7fe9335bd740 1 -- 172.21.15.28:0/794308596 >> 172.21.15.28:0/794308596 conn(0x5574f447d970 msgr2=0x5574f44a2380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:45.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.374+0000 7fe9335bd740 1 -- 172.21.15.28:0/794308596 shutdown_connections 2024-02-21T23:07:45.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.374+0000 7fe9335bd740 1 -- 172.21.15.28:0/794308596 wait complete. 2024-02-21T23:07:45.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.375+0000 7fe9335bd740 1 -- start start 2024-02-21T23:07:45.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.375+0000 7fe9335bd740 1 Processor -- start 2024-02-21T23:07:45.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.376+0000 7fe9335bd740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5574f4482330 0x5574f447baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:45.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.376+0000 7fe9335bd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574f44bd6b0 0x5574f44cd990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:45.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.376+0000 7fe9335bd740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5574f447bfe0 0x5574f44d0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:45.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.376+0000 7fe9335bd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574f4384040 con 0x5574f44bd6b0 2024-02-21T23:07:45.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.376+0000 7fe9335bd740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574f44a2a90 con 0x5574f447bfe0 2024-02-21T23:07:45.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.376+0000 7fe9335bd740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x5574f447eba0 con 0x5574f4482330 2024-02-21T23:07:45.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.376+0000 7fe9335bd740 10 client.0 _ll_register_callbacks cb 0x5574f43a1920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:07:45.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.376+0000 7fe92d200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574f44bd6b0 0x5574f44cd990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:45.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.377+0000 7fe92d200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574f44bd6b0 0x5574f44cd990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59828/0 (socket says 172.21.15.28:59828) 2024-02-21T23:07:45.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.377+0000 7fe92d200640 1 -- 172.21.15.28:0/100526415 learned_addr learned my addr 172.21.15.28:0/100526415 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:45.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.377+0000 7fe92e600640 1 --2- 172.21.15.28:0/100526415 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5574f447bfe0 0x5574f44d0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:45.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.377+0000 7fe92d200640 1 -- 172.21.15.28:0/100526415 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5574f4482330 msgr2=0x5574f447baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:45.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.377+0000 7fe92dc00640 1 --2- 172.21.15.28:0/100526415 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5574f4482330 0x5574f447baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:45.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.377+0000 7fe92d200640 1 --2- 172.21.15.28:0/100526415 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5574f4482330 0x5574f447baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:45.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.377+0000 7fe92d200640 1 -- 172.21.15.28:0/100526415 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5574f447bfe0 msgr2=0x5574f44d0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:45.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.377+0000 7fe92d200640 1 --2- 172.21.15.28:0/100526415 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5574f447bfe0 0x5574f44d0ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:45.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.377+0000 7fe92d200640 1 -- 172.21.15.28:0/100526415 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5574f43a3aa0 con 0x5574f44bd6b0 2024-02-21T23:07:45.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.377+0000 7fe92d200640 1 --2- 172.21.15.28:0/100526415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574f44bd6b0 0x5574f44cd990 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7fe914000c00 tx=0x7fe9140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9da49830cb777e69 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:45.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.377+0000 7fe90ea00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:07:45.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.378+0000 7fe90ea00640 1 -- 172.21.15.28:0/100526415 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe914039040 con 0x5574f44bd6b0 2024-02-21T23:07:45.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.378+0000 7fe90ea00640 1 -- 172.21.15.28:0/100526415 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(111..111 src has 1..111) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fe91402cbb0 con 0x5574f44bd6b0 2024-02-21T23:07:45.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.378+0000 7fe9335bd740 1 -- 172.21.15.28:0/100526415 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe914005990 con 0x5574f44bd6b0 2024-02-21T23:07:45.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.378+0000 7fe90ea00640 1 -- 172.21.15.28:0/100526415 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=107) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe914057050 con 0x5574f44bd6b0 2024-02-21T23:07:45.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.378+0000 7fe9335bd740 10 client.7650 fetch_fsmap learned FSMap version 107 2024-02-21T23:07:45.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.378+0000 7fe9335bd740 1 -- 172.21.15.28:0/100526415 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=107,monmap=2+}) v3 -- 0x5574f40fa2d0 con 0x5574f44bd6b0 2024-02-21T23:07:45.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.378+0000 7fe90ea00640 1 -- 172.21.15.28:0/100526415 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe914005990 con 0x5574f44bd6b0 2024-02-21T23:07:45.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.378+0000 7fe90ea00640 1 -- 172.21.15.28:0/100526415 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 107) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fe91405a020 con 0x5574f44bd6b0 2024-02-21T23:07:45.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.378+0000 7fe9335bd740 10 client.7650 fetch_fsmap finished waiting for FSMap version 107 2024-02-21T23:07:45.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.378+0000 7fe9335bd740 10 client.7650 resolve_mds: resolved name 'd' to daemon mds.d 2024-02-21T23:07:45.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.379+0000 7fe9335bd740 20 client.7650 populate_metadata read hostname 'smithi028' 2024-02-21T23:07:45.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.379+0000 7fe9335bd740 1 --2- 172.21.15.28:0/100526415 >> [v2:172.21.15.81:6834/1754860807,v1:172.21.15.81:6835/1754860807] conn(0x5574f44d0ff0 0x5574f44d13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:45.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.379+0000 7fe9335bd740 4 client.7650 mds_command: new command op to 5729 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-21T23:07:45.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.379+0000 7fe9335bd740 1 -- 172.21.15.28:0/100526415 --> [v2:172.21.15.81:6834/1754860807,v1:172.21.15.81:6835/1754860807] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5574f43f29b0 con 0x5574f44d0ff0 2024-02-21T23:07:45.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.379+0000 7fe92dc00640 1 --2- 172.21.15.28:0/100526415 >> [v2:172.21.15.81:6834/1754860807,v1:172.21.15.81:6835/1754860807] conn(0x5574f44d0ff0 0x5574f44d13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:45.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.380+0000 7fe92dc00640 1 --2- 172.21.15.28:0/100526415 >> [v2:172.21.15.81:6834/1754860807,v1:172.21.15.81:6835/1754860807] conn(0x5574f44d0ff0 0x5574f44d13d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=886c238e45764886 server_cookie=b7185b4ee3c69ad6 in_seq=0 out_seq=0 2024-02-21T23:07:45.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.380+0000 7fe90ea00640 10 client.7650 ms_handle_connect on v2:172.21.15.81:6834/1754860807 2024-02-21T23:07:45.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.382+0000 7fe90ea00640 1 -- 172.21.15.28:0/100526415 <== mds.0 v2:172.21.15.81:6834/1754860807 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5574f43f29b0 con 0x5574f44d0ff0 2024-02-21T23:07:45.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.382+0000 7fe90ea00640 10 client.7650 handle_command_reply: tid=0 2024-02-21T23:07:45.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.382+0000 7fe9335bd740 1 client.7650 shutdown 2024-02-21T23:07:45.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.382+0000 7fe9335bd740 1 -- 172.21.15.28:0/100526415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574f44bd6b0 msgr2=0x5574f44cd990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:45.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.382+0000 7fe9335bd740 1 --2- 172.21.15.28:0/100526415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574f44bd6b0 0x5574f44cd990 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7fe914000c00 tx=0x7fe9140046c0 comp rx=0 tx=0).stop 2024-02-21T23:07:45.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.382+0000 7fe9335bd740 1 -- 172.21.15.28:0/100526415 shutdown_connections 2024-02-21T23:07:45.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.382+0000 7fe9335bd740 1 --2- 172.21.15.28:0/100526415 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5574f447bfe0 0x5574f44d0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:45.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.382+0000 7fe9335bd740 1 --2- 172.21.15.28:0/100526415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574f44bd6b0 0x5574f44cd990 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:45.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.382+0000 7fe9335bd740 1 --2- 172.21.15.28:0/100526415 >> [v2:172.21.15.81:6834/1754860807,v1:172.21.15.81:6835/1754860807] conn(0x5574f44d0ff0 0x5574f44d13d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:45.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.382+0000 7fe9335bd740 1 --2- 172.21.15.28:0/100526415 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5574f4482330 0x5574f447baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:45.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.382+0000 7fe9335bd740 1 -- 172.21.15.28:0/100526415 >> 172.21.15.28:0/100526415 conn(0x5574f4481020 msgr2=0x5574f44bbd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:45.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.382+0000 7fe90ea00640 0 client.7650 ms_handle_reset on v2:172.21.15.81:6834/1754860807 2024-02-21T23:07:45.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.382+0000 7fe9335bd740 1 -- 172.21.15.28:0/100526415 shutdown_connections 2024-02-21T23:07:45.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.382+0000 7fe9335bd740 1 -- 172.21.15.28:0/100526415 wait complete. 2024-02-21T23:07:45.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.383+0000 7fe9335bd740 20 client.7650 trim_cache size 0 max 16384 2024-02-21T23:07:45.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.401+0000 7fe9335bd740 1 Processor -- start 2024-02-21T23:07:45.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.401+0000 7fe9335bd740 1 -- start start 2024-02-21T23:07:45.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.402+0000 7fe9335bd740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5574f447bfe0 0x5574f44ba0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:45.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.402+0000 7fe9335bd740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5574f4482330 0x5574f4476d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:45.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.402+0000 7fe9335bd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574f44bd6b0 0x5574f447b270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:45.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.402+0000 7fe9335bd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9281f74c0 con 0x5574f44bd6b0 2024-02-21T23:07:45.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.402+0000 7fe9335bd740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe928142e60 con 0x5574f447bfe0 2024-02-21T23:07:45.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.402+0000 7fe9335bd740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe928143460 con 0x5574f4482330 2024-02-21T23:07:45.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.402+0000 7fe92e600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574f44bd6b0 0x5574f447b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:45.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.402+0000 7fe92e600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574f44bd6b0 0x5574f447b270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59836/0 (socket says 172.21.15.28:59836) 2024-02-21T23:07:45.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.402+0000 7fe92e600640 1 -- 172.21.15.28:0/1468827232 learned_addr learned my addr 172.21.15.28:0/1468827232 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:45.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.402+0000 7fe92d200640 1 --2- 172.21.15.28:0/1468827232 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5574f4482330 0x5574f4476d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:45.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.402+0000 7fe92e600640 1 -- 172.21.15.28:0/1468827232 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5574f4482330 msgr2=0x5574f4476d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:45.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.402+0000 7fe92e600640 1 --2- 172.21.15.28:0/1468827232 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5574f4482330 0x5574f4476d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:45.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.403+0000 7fe92dc00640 1 --2- 172.21.15.28:0/1468827232 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5574f447bfe0 0x5574f44ba0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:45.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.403+0000 7fe92e600640 1 -- 172.21.15.28:0/1468827232 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5574f447bfe0 msgr2=0x5574f44ba0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:45.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.403+0000 7fe92e600640 1 --2- 172.21.15.28:0/1468827232 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5574f447bfe0 0x5574f44ba0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:45.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.403+0000 7fe92e600640 1 -- 172.21.15.28:0/1468827232 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5574f41c46b0 con 0x5574f44bd6b0 2024-02-21T23:07:45.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.403+0000 7fe92d200640 1 --2- 172.21.15.28:0/1468827232 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5574f4482330 0x5574f4476d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:07:45.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.403+0000 7fe92e600640 1 --2- 172.21.15.28:0/1468827232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574f44bd6b0 0x5574f447b270 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7fe920057be0 tx=0x7fe92005e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7335a08c3eaa0c89 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:45.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.403+0000 7fe919600640 1 -- 172.21.15.28:0/1468827232 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe920057c50 con 0x5574f44bd6b0 2024-02-21T23:07:45.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.403+0000 7fe919600640 1 -- 172.21.15.28:0/1468827232 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe92005bda0 con 0x5574f44bd6b0 2024-02-21T23:07:45.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.403+0000 7fe919600640 1 -- 172.21.15.28:0/1468827232 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe920060360 con 0x5574f44bd6b0 2024-02-21T23:07:45.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.404+0000 7fe9335bd740 1 -- 172.21.15.28:0/1468827232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574f44bd6b0 msgr2=0x5574f447b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:45.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.404+0000 7fe9335bd740 1 --2- 172.21.15.28:0/1468827232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574f44bd6b0 0x5574f447b270 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7fe920057be0 tx=0x7fe92005e040 comp rx=0 tx=0).stop 2024-02-21T23:07:45.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.404+0000 7fe9335bd740 1 -- 172.21.15.28:0/1468827232 shutdown_connections 2024-02-21T23:07:45.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.404+0000 7fe9335bd740 1 --2- 172.21.15.28:0/1468827232 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5574f4482330 0x5574f4476d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:45.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.404+0000 7fe9335bd740 1 --2- 172.21.15.28:0/1468827232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574f44bd6b0 0x5574f447b270 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:45.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.404+0000 7fe9335bd740 1 --2- 172.21.15.28:0/1468827232 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5574f447bfe0 0x5574f44ba0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:45.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.404+0000 7fe9335bd740 1 -- 172.21.15.28:0/1468827232 >> 172.21.15.28:0/1468827232 conn(0x5574f4481020 msgr2=0x5574f44cd750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:45.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.404+0000 7fe9335bd740 1 -- 172.21.15.28:0/1468827232 shutdown_connections 2024-02-21T23:07:45.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.404+0000 7fe9335bd740 1 -- 172.21.15.28:0/1468827232 wait complete. 2024-02-21T23:07:45.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.405+0000 7fe9335bd740 1 -- start start 2024-02-21T23:07:45.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.405+0000 7fe9335bd740 1 Processor -- start 2024-02-21T23:07:45.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.406+0000 7fe9335bd740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5574f447bfe0 0x5574f4477110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:45.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.406+0000 7fe9335bd740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5574f4482330 0x5574f44e0b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:45.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.406+0000 7fe9335bd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574f44bd6b0 0x5574f447ed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:45.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.406+0000 7fe9335bd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574f44a2a90 con 0x5574f44bd6b0 2024-02-21T23:07:45.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.406+0000 7fe9335bd740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574f449d5a0 con 0x5574f447bfe0 2024-02-21T23:07:45.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.406+0000 7fe9335bd740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x5574f4384040 con 0x5574f4482330 2024-02-21T23:07:45.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.406+0000 7fe9335bd740 10 client.0 _ll_register_callbacks cb 0x7fe91000b9c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:07:45.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.406+0000 7fe92e600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574f44bd6b0 0x5574f447ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:45.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.406+0000 7fe92e600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574f44bd6b0 0x5574f447ed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59848/0 (socket says 172.21.15.28:59848) 2024-02-21T23:07:45.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.406+0000 7fe92e600640 1 -- 172.21.15.28:0/3425153892 learned_addr learned my addr 172.21.15.28:0/3425153892 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:45.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.406+0000 7fe92dc00640 1 --2- 172.21.15.28:0/3425153892 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5574f447bfe0 0x5574f4477110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:45.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.406+0000 7fe92d200640 1 --2- 172.21.15.28:0/3425153892 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5574f4482330 0x5574f44e0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:45.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.406+0000 7fe92e600640 1 -- 172.21.15.28:0/3425153892 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5574f4482330 msgr2=0x5574f44e0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:45.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.406+0000 7fe92e600640 1 --2- 172.21.15.28:0/3425153892 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5574f4482330 0x5574f44e0b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:45.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.406+0000 7fe92e600640 1 -- 172.21.15.28:0/3425153892 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5574f447bfe0 msgr2=0x5574f4477110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:45.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.406+0000 7fe92e600640 1 --2- 172.21.15.28:0/3425153892 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5574f447bfe0 0x5574f4477110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:45.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.406+0000 7fe92e600640 1 -- 172.21.15.28:0/3425153892 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5574f449f1b0 con 0x5574f44bd6b0 2024-02-21T23:07:45.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.407+0000 7fe92e600640 1 --2- 172.21.15.28:0/3425153892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574f44bd6b0 0x5574f447ed30 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7fe92005e200 tx=0x7fe920014070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d331c45383e2c9b server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:45.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.407+0000 7fe90ea00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:07:45.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.407+0000 7fe90ea00640 1 -- 172.21.15.28:0/3425153892 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9200572a0 con 0x5574f44bd6b0 2024-02-21T23:07:45.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.407+0000 7fe90ea00640 1 -- 172.21.15.28:0/3425153892 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(111..111 src has 1..111) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fe9200078c0 con 0x5574f44bd6b0 2024-02-21T23:07:45.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.407+0000 7fe9335bd740 1 -- 172.21.15.28:0/3425153892 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe92005bda0 con 0x5574f44bd6b0 2024-02-21T23:07:45.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.407+0000 7fe90ea00640 1 -- 172.21.15.28:0/3425153892 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=107) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe92007c050 con 0x5574f44bd6b0 2024-02-21T23:07:45.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.407+0000 7fe9335bd740 10 client.7656 fetch_fsmap learned FSMap version 107 2024-02-21T23:07:45.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.407+0000 7fe9335bd740 1 -- 172.21.15.28:0/3425153892 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=107,monmap=2+}) v3 -- 0x5574f447b7d0 con 0x5574f44bd6b0 2024-02-21T23:07:45.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.407+0000 7fe90ea00640 1 -- 172.21.15.28:0/3425153892 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe92005bda0 con 0x5574f44bd6b0 2024-02-21T23:07:45.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.407+0000 7fe90ea00640 1 -- 172.21.15.28:0/3425153892 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 107) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fe92007f020 con 0x5574f44bd6b0 2024-02-21T23:07:45.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.408+0000 7fe9335bd740 10 client.7656 fetch_fsmap finished waiting for FSMap version 107 2024-02-21T23:07:45.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.408+0000 7fe9335bd740 10 client.7656 resolve_mds: resolved name 'd' to daemon mds.d 2024-02-21T23:07:45.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.408+0000 7fe9335bd740 20 client.7656 populate_metadata read hostname 'smithi028' 2024-02-21T23:07:45.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.408+0000 7fe9335bd740 1 --2- 172.21.15.28:0/3425153892 >> [v2:172.21.15.81:6834/1754860807,v1:172.21.15.81:6835/1754860807] conn(0x5574f452a420 0x5574f452fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:45.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.408+0000 7fe9335bd740 4 client.7656 mds_command: new command op to 5729 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-21T23:07:45.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.408+0000 7fe9335bd740 1 -- 172.21.15.28:0/3425153892 --> [v2:172.21.15.81:6834/1754860807,v1:172.21.15.81:6835/1754860807] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5574f43ecd00 con 0x5574f452a420 2024-02-21T23:07:45.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.408+0000 7fe92dc00640 1 --2- 172.21.15.28:0/3425153892 >> [v2:172.21.15.81:6834/1754860807,v1:172.21.15.81:6835/1754860807] conn(0x5574f452a420 0x5574f452fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:45.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.409+0000 7fe92dc00640 1 --2- 172.21.15.28:0/3425153892 >> [v2:172.21.15.81:6834/1754860807,v1:172.21.15.81:6835/1754860807] conn(0x5574f452a420 0x5574f452fff0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6fa745d35f172469 server_cookie=3af2debb5230e01 in_seq=0 out_seq=0 2024-02-21T23:07:45.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.409+0000 7fe90ea00640 10 client.7656 ms_handle_connect on v2:172.21.15.81:6834/1754860807 2024-02-21T23:07:45.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.410+0000 7fe90ea00640 1 -- 172.21.15.28:0/3425153892 <== mds.0 v2:172.21.15.81:6834/1754860807 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x5574f43ecd00 con 0x5574f452a420 2024-02-21T23:07:45.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.410+0000 7fe90ea00640 10 client.7656 handle_command_reply: tid=0 2024-02-21T23:07:45.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.410+0000 7fe9335bd740 1 client.7656 shutdown 2024-02-21T23:07:45.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.410+0000 7fe9335bd740 1 -- 172.21.15.28:0/3425153892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574f44bd6b0 msgr2=0x5574f447ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:45.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.410+0000 7fe9335bd740 1 --2- 172.21.15.28:0/3425153892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574f44bd6b0 0x5574f447ed30 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7fe92005e200 tx=0x7fe920014070 comp rx=0 tx=0).stop 2024-02-21T23:07:45.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.410+0000 7fe9335bd740 1 -- 172.21.15.28:0/3425153892 shutdown_connections 2024-02-21T23:07:45.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.410+0000 7fe9335bd740 1 --2- 172.21.15.28:0/3425153892 >> [v2:172.21.15.81:6834/1754860807,v1:172.21.15.81:6835/1754860807] conn(0x5574f452a420 0x5574f452fff0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:45.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.410+0000 7fe9335bd740 1 --2- 172.21.15.28:0/3425153892 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5574f4482330 0x5574f44e0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:45.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.410+0000 7fe9335bd740 1 --2- 172.21.15.28:0/3425153892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574f44bd6b0 0x5574f447ed30 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:45.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.410+0000 7fe9335bd740 1 --2- 172.21.15.28:0/3425153892 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5574f447bfe0 0x5574f4477110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:45.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.410+0000 7fe90ea00640 0 client.7656 ms_handle_reset on v2:172.21.15.81:6834/1754860807 2024-02-21T23:07:45.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.410+0000 7fe9335bd740 1 -- 172.21.15.28:0/3425153892 >> 172.21.15.28:0/3425153892 conn(0x5574f4481020 msgr2=0x5574f44bebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:45.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.410+0000 7fe9335bd740 1 -- 172.21.15.28:0/3425153892 shutdown_connections 2024-02-21T23:07:45.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.410+0000 7fe9335bd740 1 -- 172.21.15.28:0/3425153892 wait complete. 2024-02-21T23:07:45.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.411+0000 7fe9335bd740 20 client.7656 trim_cache size 0 max 16384 2024-02-21T23:07:45.404 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-21T23:07:45.404 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac", 2024-02-21T23:07:45.404 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-02-21T23:07:45.404 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 5729, 2024-02-21T23:07:45.404 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-02-21T23:07:45.404 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-02-21T23:07:45.404 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-02-21T23:07:45.404 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 4.263354509, 2024-02-21T23:07:45.405 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 107, 2024-02-21T23:07:45.405 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 111, 2024-02-21T23:07:45.405 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 111, 2024-02-21T23:07:45.405 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 20.816426057000001 2024-02-21T23:07:45.405 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-21T23:07:45.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.411+0000 7fe92f000640 1 -- 172.21.15.28:0/2045045529 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe90806b4d0 msgr2=0x7fe90806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:45.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.411+0000 7fe92f000640 1 --2- 172.21.15.28:0/2045045529 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe90806b4d0 0x7fe90806d990 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fe91c000c00 tx=0x7fe91c009040 comp rx=0 tx=0).stop 2024-02-21T23:07:45.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.411+0000 7fe92f000640 1 -- 172.21.15.28:0/2045045529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9281ee430 msgr2=0x7fe9280aaa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:45.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.411+0000 7fe92f000640 1 --2- 172.21.15.28:0/2045045529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9281ee430 0x7fe9280aaa10 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7fe9280c0ca0 tx=0x7fe92000cb90 comp rx=0 tx=0).stop 2024-02-21T23:07:45.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.411+0000 7fe92f000640 1 -- 172.21.15.28:0/2045045529 shutdown_connections 2024-02-21T23:07:45.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.411+0000 7fe92f000640 1 --2- 172.21.15.28:0/2045045529 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe90806b4d0 0x7fe90806d990 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:45.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.411+0000 7fe92f000640 1 --2- 172.21.15.28:0/2045045529 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe928154120 0x7fe92806da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:45.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.411+0000 7fe92f000640 1 --2- 172.21.15.28:0/2045045529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9281ee430 0x7fe9280aaa10 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:45.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.411+0000 7fe92f000640 1 --2- 172.21.15.28:0/2045045529 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe910004800 0x7fe9281ea620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:45.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.411+0000 7fe92f000640 1 -- 172.21.15.28:0/2045045529 >> 172.21.15.28:0/2045045529 conn(0x7fe9280a8320 msgr2=0x7fe9281521f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:45.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.411+0000 7fe92f000640 1 -- 172.21.15.28:0/2045045529 shutdown_connections 2024-02-21T23:07:45.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.411+0000 7fe92f000640 1 -- 172.21.15.28:0/2045045529 wait complete. 2024-02-21T23:07:45.416 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 8, "epoch": 107, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 8, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-02-21T23:07:29.533547+0000", "damaged": [], "data_pools": [ 18 ], "enabled": true, "epoch": 107, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5729": { "addr": "172.21.15.81:6835/1754860807", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6834", "nonce": 1754860807, "type": "v2" }, { "addr": "172.21.15.81:6835", "nonce": 1754860807, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5729, "incarnation": 104, "join_fscid": 8, "name": "d", "rank": 0, "state": "up:active", "state_seq": 8 } }, "last_failure": 0, "last_failure_osd_epoch": 111, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 17, "modified": "2024-02-21T23:07:43.164772+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5729 } } } ], "standbys": [ { "addr": "172.21.15.81:6837/3963422563", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6836", "nonce": 3963422563, "type": "v2" }, { "addr": "172.21.15.81:6837", "nonce": 3963422563, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 97, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5732, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.28:6833/1119726035", "addrs": { "addrvec": [ { "addr": "172.21.15.28:6832", "nonce": 1119726035, "type": "v2" }, { "addr": "172.21.15.28:6833", "nonce": 1119726035, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 96, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7446, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.28:6835/780497578", "addrs": { "addrvec": [ { "addr": "172.21.15.28:6834", "nonce": 780497578, "type": "v2" }, { "addr": "172.21.15.28:6835", "nonce": 780497578, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 105, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7632, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-02-21T23:07:45.418 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': 8, 'name': 'd', 'state': 'up:active'}] 2024-02-21T23:07:45.418 INFO:tasks.cephfs.test_failover:current = [{'gid': 7632, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.28:6835/780497578', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 780497578}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 780497578}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 105}, {'gid': 5732, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.81:6837/3963422563', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3963422563}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3963422563}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 97}, {'gid': 7446, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.28:6833/1119726035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 1119726035}, {'type': 'v1', 'addr': '172.21.15.28:6833', 'nonce': 1119726035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 96}, {'gid': 5729, 'name': 'd', 'rank': 0, 'incarnation': 104, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.81:6835/1754860807', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6834', 'nonce': 1754860807}, {'type': 'v1', 'addr': '172.21.15.81:6835', 'nonce': 1754860807}]}, 'join_fscid': 8, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}] 2024-02-21T23:07:45.418 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-02-21T23:07:45.418 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-02-21T23:07:45.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.652+0000 7f500b200640 1 Processor -- start 2024-02-21T23:07:45.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.652+0000 7f500b200640 1 -- start start 2024-02-21T23:07:45.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.653+0000 7f500b200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f500c06dbc0 0x7f500c06c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:45.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.653+0000 7f500b200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f500c13e520 0x7f500c06c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:45.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.653+0000 7f500b200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f500c158550 con 0x7f500c13e0c0 2024-02-21T23:07:45.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.653+0000 7f500b200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f500c0c89a0 con 0x7f500c06dbc0 2024-02-21T23:07:45.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.653+0000 7f500b200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f500c1587d0 con 0x7f500c13e520 2024-02-21T23:07:45.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.654+0000 7f5009e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f500c13e0c0 0x7f500c0c71b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53400/0 (socket says 172.21.15.28:53400) 2024-02-21T23:07:45.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.654+0000 7f5009e00640 1 -- 172.21.15.28:0/783470243 learned_addr learned my addr 172.21.15.28:0/783470243 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:07:45.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.654+0000 7f5009400640 1 --2- 172.21.15.28:0/783470243 >> v2:172.21.15.81:3300/0 conn(0x7f500c06dbc0 0x7f500c06c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:45.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.654+0000 7f500a800640 1 --2- 172.21.15.28:0/783470243 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f500c13e520 0x7f500c06c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:45.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.654+0000 7f5008a00640 1 -- 172.21.15.28:0/783470243 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3705421511 0 0) 0x7f4ffc001110 con 0x7f500c13e0c0 2024-02-21T23:07:45.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.654+0000 7f5008a00640 1 -- 172.21.15.28:0/783470243 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ff0003520 con 0x7f500c13e0c0 2024-02-21T23:07:45.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.655+0000 7f5008a00640 1 -- 172.21.15.28:0/783470243 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2522050977 0 0) 0x7f4ff0003520 con 0x7f500c13e0c0 2024-02-21T23:07:45.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.655+0000 7f5008a00640 1 -- 172.21.15.28:0/783470243 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f500c13e520 msgr2=0x7f500c06c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:45.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.655+0000 7f5008a00640 1 --2- 172.21.15.28:0/783470243 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f500c13e520 0x7f500c06c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:45.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.655+0000 7f5008a00640 1 -- 172.21.15.28:0/783470243 >> v2:172.21.15.81:3300/0 conn(0x7f500c06dbc0 msgr2=0x7f500c06c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:45.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.655+0000 7f5008a00640 1 --2- 172.21.15.28:0/783470243 >> v2:172.21.15.81:3300/0 conn(0x7f500c06dbc0 0x7f500c06c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:45.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.655+0000 7f5008a00640 1 -- 172.21.15.28:0/783470243 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f500c158ab0 con 0x7f500c13e0c0 2024-02-21T23:07:45.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.655+0000 7f5008a00640 1 -- 172.21.15.28:0/783470243 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f4ffc001840 con 0x7f500c13e0c0 2024-02-21T23:07:45.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.655+0000 7f5008a00640 1 -- 172.21.15.28:0/783470243 >> v1:172.21.15.28:6789/0 conn(0x7f500c13e0c0 legacy=0x7f500c0c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:45.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.655+0000 7f5008a00640 1 --2- 172.21.15.28:0/783470243 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ff00039b0 0x7f4ff0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:45.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.655+0000 7f5008a00640 1 --2- 172.21.15.28:0/783470243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ff0004800 0x7f4ff0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:45.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.655+0000 7f5008a00640 1 --2- 172.21.15.28:0/783470243 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4ff00051c0 0x7f4ff000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:45.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.655+0000 7f5008a00640 1 -- 172.21.15.28:0/783470243 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f500c1587d0 con 0x7f4ff0004800 2024-02-21T23:07:45.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.655+0000 7f5008a00640 1 -- 172.21.15.28:0/783470243 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff000bf00 con 0x7f4ff00039b0 2024-02-21T23:07:45.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.655+0000 7f5008a00640 1 -- 172.21.15.28:0/783470243 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff000c0b0 con 0x7f4ff00051c0 2024-02-21T23:07:45.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.655+0000 7f5009400640 1 --2- 172.21.15.28:0/783470243 >> v2:172.21.15.81:3300/0 conn(0x7f500c06dbc0 0x7f500c06c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-21T23:07:45.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.655+0000 7f500a800640 1 --2- 172.21.15.28:0/783470243 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f500c13e520 0x7f500c06c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-21T23:07:45.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.655+0000 7f5009400640 1 --2- 172.21.15.28:0/783470243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ff0004800 0x7f4ff0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:45.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.656+0000 7f500a800640 1 --2- 172.21.15.28:0/783470243 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4ff00051c0 0x7f4ff000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:45.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.656+0000 7f5009e00640 1 --2- 172.21.15.28:0/783470243 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ff00039b0 0x7f4ff0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:45.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.656+0000 7f5009400640 1 -- 172.21.15.28:0/783470243 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4ff00051c0 msgr2=0x7f4ff000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:45.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.656+0000 7f5009400640 1 --2- 172.21.15.28:0/783470243 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4ff00051c0 0x7f4ff000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:45.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.656+0000 7f5009400640 1 -- 172.21.15.28:0/783470243 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ff00039b0 msgr2=0x7f4ff0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:45.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.656+0000 7f5009400640 1 --2- 172.21.15.28:0/783470243 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ff00039b0 0x7f4ff0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:45.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.656+0000 7f5009400640 1 -- 172.21.15.28:0/783470243 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ff0003520 con 0x7f4ff0004800 2024-02-21T23:07:45.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.656+0000 7f500a800640 1 --2- 172.21.15.28:0/783470243 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4ff00051c0 0x7f4ff000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:07:45.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.656+0000 7f5009e00640 1 --2- 172.21.15.28:0/783470243 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ff00039b0 0x7f4ff0003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:07:45.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.657+0000 7f5009400640 1 --2- 172.21.15.28:0/783470243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ff0004800 0x7f4ff0004c80 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f4ff8009c70 tx=0x7f4ff0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74bcc273d6135185 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:45.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.657+0000 7f5008a00640 1 -- 172.21.15.28:0/783470243 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff8009840 con 0x7f4ff0004800 2024-02-21T23:07:45.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.657+0000 7f5008a00640 1 -- 172.21.15.28:0/783470243 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ff8004020 con 0x7f4ff0004800 2024-02-21T23:07:45.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.657+0000 7f5008a00640 1 -- 172.21.15.28:0/783470243 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff8004410 con 0x7f4ff0004800 2024-02-21T23:07:45.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.658+0000 7f500b200640 1 -- 172.21.15.28:0/783470243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ff0004800 msgr2=0x7f4ff0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:45.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.658+0000 7f500b200640 1 --2- 172.21.15.28:0/783470243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ff0004800 0x7f4ff0004c80 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f4ff8009c70 tx=0x7f4ff0005830 comp rx=0 tx=0).stop 2024-02-21T23:07:45.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.658+0000 7f500a800640 1 -- 172.21.15.28:0/783470243 reap_dead start 2024-02-21T23:07:45.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.658+0000 7f500b200640 1 -- 172.21.15.28:0/783470243 shutdown_connections 2024-02-21T23:07:45.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.658+0000 7f500b200640 1 -- 172.21.15.28:0/783470243 >> 172.21.15.28:0/783470243 conn(0x7f500c073110 msgr2=0x7f500c0ac490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:45.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.658+0000 7f500b200640 1 -- 172.21.15.28:0/783470243 shutdown_connections 2024-02-21T23:07:45.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.658+0000 7f500b200640 1 -- 172.21.15.28:0/783470243 wait complete. 2024-02-21T23:07:45.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.660+0000 7f500b200640 1 Processor -- start 2024-02-21T23:07:45.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.660+0000 7f500b200640 1 -- start start 2024-02-21T23:07:45.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.661+0000 7f500b200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ff0004800 0x7f500c1ee700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:45.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.661+0000 7f500b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f500c154c50 0x7f500c153d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:45.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.661+0000 7f500b200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f500c1542c0 0x7f500c151e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:45.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.661+0000 7f500b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f500c0c78c0 con 0x7f500c154c50 2024-02-21T23:07:45.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.661+0000 7f500b200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f500c159070 con 0x7f4ff0004800 2024-02-21T23:07:45.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.661+0000 7f500b200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f500c0c7a40 con 0x7f500c1542c0 2024-02-21T23:07:45.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.662+0000 7f5009400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f500c154c50 0x7f500c153d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:45.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.662+0000 7f5009400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f500c154c50 0x7f500c153d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59852/0 (socket says 172.21.15.28:59852) 2024-02-21T23:07:45.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.662+0000 7f5009400640 1 -- 172.21.15.28:0/1071614933 learned_addr learned my addr 172.21.15.28:0/1071614933 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:45.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.662+0000 7f5009e00640 1 --2- 172.21.15.28:0/1071614933 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ff0004800 0x7f500c1ee700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:45.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.662+0000 7f500a800640 1 --2- 172.21.15.28:0/1071614933 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f500c1542c0 0x7f500c151e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:45.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.662+0000 7f5009400640 1 -- 172.21.15.28:0/1071614933 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f500c1542c0 msgr2=0x7f500c151e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:45.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.662+0000 7f5009400640 1 --2- 172.21.15.28:0/1071614933 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f500c1542c0 0x7f500c151e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:45.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.662+0000 7f5009400640 1 -- 172.21.15.28:0/1071614933 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ff0004800 msgr2=0x7f500c1ee700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:45.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.662+0000 7f5009400640 1 --2- 172.21.15.28:0/1071614933 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ff0004800 0x7f500c1ee700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:45.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.662+0000 7f5009400640 1 -- 172.21.15.28:0/1071614933 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f500c154770 con 0x7f500c154c50 2024-02-21T23:07:45.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.662+0000 7f5009e00640 1 --2- 172.21.15.28:0/1071614933 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ff0004800 0x7f500c1ee700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:07:45.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.663+0000 7f5009400640 1 --2- 172.21.15.28:0/1071614933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f500c154c50 0x7f500c153d80 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f4ff8009d90 tx=0x7f4ff8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:45.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.663+0000 7f5002000640 1 -- 172.21.15.28:0/1071614933 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff8043020 con 0x7f500c154c50 2024-02-21T23:07:45.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.663+0000 7f5002000640 1 -- 172.21.15.28:0/1071614933 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ff802c950 con 0x7f500c154c50 2024-02-21T23:07:45.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.663+0000 7f5002000640 1 -- 172.21.15.28:0/1071614933 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff8006e10 con 0x7f500c154c50 2024-02-21T23:07:45.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.663+0000 7f500b200640 1 -- 172.21.15.28:0/1071614933 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f500c153880 con 0x7f500c154c50 2024-02-21T23:07:45.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.663+0000 7f500b200640 1 -- 172.21.15.28:0/1071614933 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f500c0aff00 con 0x7f500c154c50 2024-02-21T23:07:45.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.664+0000 7f500b200640 1 -- 172.21.15.28:0/1071614933 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f500c153a50 con 0x7f500c154c50 2024-02-21T23:07:45.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.665+0000 7f5002000640 1 -- 172.21.15.28:0/1071614933 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4ff8046020 con 0x7f500c154c50 2024-02-21T23:07:45.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.666+0000 7f5002000640 1 --2- 172.21.15.28:0/1071614933 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4fec06b500 0x7f4fec06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:45.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.666+0000 7f5002000640 1 -- 172.21.15.28:0/1071614933 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f4ff80ac3c0 con 0x7f500c154c50 2024-02-21T23:07:45.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.667+0000 7f5009e00640 1 --2- 172.21.15.28:0/1071614933 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4fec06b500 0x7f4fec06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:45.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.668+0000 7f5009e00640 1 --2- 172.21.15.28:0/1071614933 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4fec06b500 0x7f4fec06d9c0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f4ffc005bb0 tx=0x7f4ffc009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:45.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.671+0000 7f5002000640 1 -- 172.21.15.28:0/1071614933 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4ff804d030 con 0x7f500c154c50 2024-02-21T23:07:45.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:45.914+0000 7f500b200640 1 -- 172.21.15.28:0/1071614933 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7f500c04f290 con 0x7f500c154c50 2024-02-21T23:07:46.000 INFO:tasks.ceph.mds.d.smithi081.stderr:2024-02-21T23:07:46.011+0000 7fd00b600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-21T23:07:46.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.180+0000 7f5002000640 1 -- 172.21.15.28:0/1071614933 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v108) v1 ==== 121+0+0 (secure 0 0 0) 0x7f4ff802cd00 con 0x7f500c154c50 2024-02-21T23:07:46.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.184+0000 7f500b200640 1 -- 172.21.15.28:0/1071614933 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4fec06b500 msgr2=0x7f4fec06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:46.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.184+0000 7f500b200640 1 --2- 172.21.15.28:0/1071614933 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4fec06b500 0x7f4fec06d9c0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f4ffc005bb0 tx=0x7f4ffc009040 comp rx=0 tx=0).stop 2024-02-21T23:07:46.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.184+0000 7f500b200640 1 -- 172.21.15.28:0/1071614933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f500c154c50 msgr2=0x7f500c153d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:46.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.184+0000 7f500b200640 1 --2- 172.21.15.28:0/1071614933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f500c154c50 0x7f500c153d80 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f4ff8009d90 tx=0x7f4ff8004730 comp rx=0 tx=0).stop 2024-02-21T23:07:46.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.184+0000 7f500b200640 1 -- 172.21.15.28:0/1071614933 shutdown_connections 2024-02-21T23:07:46.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.184+0000 7f500b200640 1 --2- 172.21.15.28:0/1071614933 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4fec06b500 0x7f4fec06d9c0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:46.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.184+0000 7f500b200640 1 --2- 172.21.15.28:0/1071614933 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f500c1542c0 0x7f500c151e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:46.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.184+0000 7f500b200640 1 --2- 172.21.15.28:0/1071614933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f500c154c50 0x7f500c153d80 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:46.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.184+0000 7f500b200640 1 --2- 172.21.15.28:0/1071614933 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ff0004800 0x7f500c1ee700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:46.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.184+0000 7f500b200640 1 -- 172.21.15.28:0/1071614933 >> 172.21.15.28:0/1071614933 conn(0x7f500c073110 msgr2=0x7f500c0c8fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:46.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.184+0000 7f500b200640 1 -- 172.21.15.28:0/1071614933 shutdown_connections 2024-02-21T23:07:46.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.184+0000 7f500b200640 1 -- 172.21.15.28:0/1071614933 wait complete. 2024-02-21T23:07:46.197 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:07:46.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.415+0000 7f454d400640 1 Processor -- start 2024-02-21T23:07:46.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.415+0000 7f454d400640 1 -- start start 2024-02-21T23:07:46.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.417+0000 7f454d400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f45480c0f00 0x7f4548159db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:46.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.417+0000 7f454d400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f4548071070 0x7f4548071500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:46.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.417+0000 7f454d400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45480c1390 con 0x7f45481569b0 2024-02-21T23:07:46.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.417+0000 7f454d400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45480c1fa0 con 0x7f4548071070 2024-02-21T23:07:46.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.417+0000 7f454d400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45480c1670 con 0x7f45480c0f00 2024-02-21T23:07:46.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.417+0000 7f4547e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f45481569b0 0x7f454806dff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53416/0 (socket says 172.21.15.28:53416) 2024-02-21T23:07:46.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.417+0000 7f4547e00640 1 -- 172.21.15.28:0/1058626370 learned_addr learned my addr 172.21.15.28:0/1058626370 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:07:46.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.418+0000 7f453fe00640 1 --2- 172.21.15.28:0/1058626370 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f45480c0f00 0x7f4548159db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:46.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.418+0000 7f454ca00640 1 --2- 172.21.15.28:0/1058626370 >> v2:172.21.15.81:3300/0 conn(0x7f4548071070 0x7f4548071500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:46.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.418+0000 7f4547400640 1 -- 172.21.15.28:0/1058626370 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 717598569 0 0) 0x7f45480c1390 con 0x7f45481569b0 2024-02-21T23:07:46.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.418+0000 7f4547400640 1 -- 172.21.15.28:0/1058626370 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f452c003520 con 0x7f45481569b0 2024-02-21T23:07:46.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.418+0000 7f4547400640 1 -- 172.21.15.28:0/1058626370 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3166121306 0 0) 0x7f452c003520 con 0x7f45481569b0 2024-02-21T23:07:46.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.418+0000 7f453fe00640 1 -- 172.21.15.28:0/1058626370 >> v2:172.21.15.81:3300/0 conn(0x7f4548071070 msgr2=0x7f4548071500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:46.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.418+0000 7f453fe00640 1 --2- 172.21.15.28:0/1058626370 >> v2:172.21.15.81:3300/0 conn(0x7f4548071070 0x7f4548071500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:46.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.418+0000 7f453fe00640 1 -- 172.21.15.28:0/1058626370 >> v1:172.21.15.28:6789/0 conn(0x7f45481569b0 legacy=0x7f454806dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:46.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.418+0000 7f453fe00640 1 -- 172.21.15.28:0/1058626370 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45480c1950 con 0x7f45480c0f00 2024-02-21T23:07:46.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.420+0000 7f453fe00640 1 --2- 172.21.15.28:0/1058626370 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f45480c0f00 0x7f4548159db0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f45300097a0 tx=0x7f453002b710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4aea15a9f0c19495 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:46.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.420+0000 7f4547400640 1 -- 172.21.15.28:0/1058626370 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4530037070 con 0x7f45480c0f00 2024-02-21T23:07:46.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.420+0000 7f4547400640 1 -- 172.21.15.28:0/1058626370 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4530030510 con 0x7f45480c0f00 2024-02-21T23:07:46.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.420+0000 7f4547400640 1 -- 172.21.15.28:0/1058626370 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4538001670 con 0x7f45480c0f00 2024-02-21T23:07:46.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.421+0000 7f454d400640 1 -- 172.21.15.28:0/1058626370 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f45480c0f00 msgr2=0x7f4548159db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:46.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.421+0000 7f454d400640 1 --2- 172.21.15.28:0/1058626370 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f45480c0f00 0x7f4548159db0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f45300097a0 tx=0x7f453002b710 comp rx=0 tx=0).stop 2024-02-21T23:07:46.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.421+0000 7f454d400640 1 -- 172.21.15.28:0/1058626370 shutdown_connections 2024-02-21T23:07:46.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.421+0000 7f454d400640 1 --2- 172.21.15.28:0/1058626370 >> v2:172.21.15.81:3300/0 conn(0x7f4548071070 0x7f4548071500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:46.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.421+0000 7f454d400640 1 --2- 172.21.15.28:0/1058626370 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f45480c0f00 0x7f4548159db0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:46.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.421+0000 7f454d400640 1 -- 172.21.15.28:0/1058626370 >> 172.21.15.28:0/1058626370 conn(0x7f454806b630 msgr2=0x7f45480c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:46.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.421+0000 7f454d400640 1 -- 172.21.15.28:0/1058626370 shutdown_connections 2024-02-21T23:07:46.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.421+0000 7f454d400640 1 -- 172.21.15.28:0/1058626370 wait complete. 2024-02-21T23:07:46.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.423+0000 7f454d400640 1 Processor -- start 2024-02-21T23:07:46.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.423+0000 7f454d400640 1 -- start start 2024-02-21T23:07:46.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.424+0000 7f454d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4548071070 0x7f45480a2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:46.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.424+0000 7f454d400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f45481569b0 0x7f454815cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:46.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.424+0000 7f454d400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f45480a4210 0x7f45480a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:46.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.424+0000 7f454d400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f454806e700 con 0x7f4548071070 2024-02-21T23:07:46.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.424+0000 7f454d400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45480c2ba0 con 0x7f45481569b0 2024-02-21T23:07:46.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.424+0000 7f454d400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f454806e880 con 0x7f45480a4210 2024-02-21T23:07:46.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.424+0000 7f4547e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4548071070 0x7f45480a2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:46.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.424+0000 7f4547e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4548071070 0x7f45480a2c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59866/0 (socket says 172.21.15.28:59866) 2024-02-21T23:07:46.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.424+0000 7f4547e00640 1 -- 172.21.15.28:0/1834759629 learned_addr learned my addr 172.21.15.28:0/1834759629 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:46.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.424+0000 7f453fe00640 1 --2- 172.21.15.28:0/1834759629 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f45481569b0 0x7f454815cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:46.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.424+0000 7f454ca00640 1 --2- 172.21.15.28:0/1834759629 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f45480a4210 0x7f45480a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:46.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.425+0000 7f4547e00640 1 -- 172.21.15.28:0/1834759629 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f45480a4210 msgr2=0x7f45480a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:46.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.425+0000 7f4547e00640 1 --2- 172.21.15.28:0/1834759629 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f45480a4210 0x7f45480a46a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:46.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.425+0000 7f4547e00640 1 -- 172.21.15.28:0/1834759629 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f45481569b0 msgr2=0x7f454815cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:46.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.425+0000 7f4547e00640 1 --2- 172.21.15.28:0/1834759629 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f45481569b0 0x7f454815cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:46.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.425+0000 7f4547e00640 1 -- 172.21.15.28:0/1834759629 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45480a3250 con 0x7f4548071070 2024-02-21T23:07:46.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.425+0000 7f4547e00640 1 --2- 172.21.15.28:0/1834759629 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4548071070 0x7f45480a2c70 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f453800c700 tx=0x7f453800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:46.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.425+0000 7f4544c00640 1 -- 172.21.15.28:0/1834759629 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4538017070 con 0x7f4548071070 2024-02-21T23:07:46.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.425+0000 7f4544c00640 1 -- 172.21.15.28:0/1834759629 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4538005ad0 con 0x7f4548071070 2024-02-21T23:07:46.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.425+0000 7f4544c00640 1 -- 172.21.15.28:0/1834759629 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4538007070 con 0x7f4548071070 2024-02-21T23:07:46.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.425+0000 7f454d400640 1 -- 172.21.15.28:0/1834759629 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4548152eb0 con 0x7f4548071070 2024-02-21T23:07:46.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.425+0000 7f454d400640 1 -- 172.21.15.28:0/1834759629 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4548153470 con 0x7f4548071070 2024-02-21T23:07:46.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.426+0000 7f454d400640 1 -- 172.21.15.28:0/1834759629 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f454804f290 con 0x7f4548071070 2024-02-21T23:07:46.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.427+0000 7f4544c00640 1 -- 172.21.15.28:0/1834759629 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f453801a070 con 0x7f4548071070 2024-02-21T23:07:46.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.428+0000 7f4544c00640 1 --2- 172.21.15.28:0/1834759629 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f452406b4d0 0x7f452406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:46.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.428+0000 7f4544c00640 1 -- 172.21.15.28:0/1834759629 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f45380898a0 con 0x7f4548071070 2024-02-21T23:07:46.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.429+0000 7f453fe00640 1 --2- 172.21.15.28:0/1834759629 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f452406b4d0 0x7f452406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:46.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.430+0000 7f453fe00640 1 --2- 172.21.15.28:0/1834759629 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f452406b4d0 0x7f452406d990 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f4530034000 tx=0x7f4530008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:46.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.435+0000 7f4544c00640 1 -- 172.21.15.28:0/1834759629 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4538058530 con 0x7f4548071070 2024-02-21T23:07:46.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.682+0000 7f454d400640 1 -- 172.21.15.28:0/1834759629 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4548073540 con 0x7f4548071070 2024-02-21T23:07:46.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.682+0000 7f4544c00640 1 -- 172.21.15.28:0/1834759629 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 109 v109) v1 ==== 95+0+4948 (secure 0 0 0) 0x7f4538057e30 con 0x7f4548071070 2024-02-21T23:07:46.671 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:07:46.671 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":109,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3963422563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3963422563},{"type":"v1","addr":"172.21.15.81:6837","nonce":3963422563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7446,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/1119726035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1119726035},{"type":"v1","addr":"172.21.15.28:6833","nonce":1119726035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":109,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-02-21T23:07:29.533547+0000","modified":"2024-02-21T23:07:46.185982+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5729},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5729":{"gid":5729,"name":"d","rank":0,"incarnation":104,"state":"up:active","state_seq":8,"addr":"172.21.15.81:6835/1754860807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1754860807},{"type":"v1","addr":"172.21.15.81:6835","nonce":1754860807}]},"join_fscid":8,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7632":{"gid":7632,"name":"a","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/780497578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":780497578},{"type":"v1","addr":"172.21.15.28:6835","nonce":780497578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-02-21T23:07:46.671 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 109 2024-02-21T23:07:46.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.685+0000 7f454d400640 1 -- 172.21.15.28:0/1834759629 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f452406b4d0 msgr2=0x7f452406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:46.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.685+0000 7f454d400640 1 --2- 172.21.15.28:0/1834759629 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f452406b4d0 0x7f452406d990 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f4530034000 tx=0x7f4530008030 comp rx=0 tx=0).stop 2024-02-21T23:07:46.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.685+0000 7f454d400640 1 -- 172.21.15.28:0/1834759629 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4548071070 msgr2=0x7f45480a2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:46.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.685+0000 7f454d400640 1 --2- 172.21.15.28:0/1834759629 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4548071070 0x7f45480a2c70 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f453800c700 tx=0x7f453800cbc0 comp rx=0 tx=0).stop 2024-02-21T23:07:46.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.685+0000 7f454d400640 1 -- 172.21.15.28:0/1834759629 shutdown_connections 2024-02-21T23:07:46.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.685+0000 7f454d400640 1 --2- 172.21.15.28:0/1834759629 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f452406b4d0 0x7f452406d990 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:46.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.685+0000 7f454d400640 1 --2- 172.21.15.28:0/1834759629 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f45480a4210 0x7f45480a46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:46.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.685+0000 7f454d400640 1 --2- 172.21.15.28:0/1834759629 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f45481569b0 0x7f454815cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:46.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.685+0000 7f454d400640 1 --2- 172.21.15.28:0/1834759629 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4548071070 0x7f45480a2c70 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:46.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.685+0000 7f454d400640 1 -- 172.21.15.28:0/1834759629 >> 172.21.15.28:0/1834759629 conn(0x7f454806b630 msgr2=0x7f454806dff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:46.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.685+0000 7f454d400640 1 -- 172.21.15.28:0/1834759629 shutdown_connections 2024-02-21T23:07:46.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.685+0000 7f454d400640 1 -- 172.21.15.28:0/1834759629 wait complete. 2024-02-21T23:07:46.692 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.b mds_join_fs cephfs 2024-02-21T23:07:46.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.936+0000 7fd1c2000640 1 Processor -- start 2024-02-21T23:07:46.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.936+0000 7fd1c2000640 1 -- start start 2024-02-21T23:07:46.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.937+0000 7fd1c2000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd1bc0c22e0 0x7fd1bc06d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:46.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.937+0000 7fd1c2000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1bc071970 con 0x7fd1bc0c5a70 2024-02-21T23:07:46.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.937+0000 7fd1c2000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1bc071c50 con 0x7fd1bc1561a0 2024-02-21T23:07:46.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.937+0000 7fd1c2000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1bc072a10 con 0x7fd1bc0c22e0 2024-02-21T23:07:46.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.937+0000 7fd1bb400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd1bc0c5a70 0x7fd1bc155a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53420/0 (socket says 172.21.15.28:53420) 2024-02-21T23:07:46.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.938+0000 7fd1bb400640 1 -- 172.21.15.28:0/4260489608 learned_addr learned my addr 172.21.15.28:0/4260489608 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:07:46.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.938+0000 7fd1b3e00640 1 --2- 172.21.15.28:0/4260489608 >> v2:172.21.15.28:3300/0 conn(0x7fd1bc0c22e0 0x7fd1bc06d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:46.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.938+0000 7fd1baa00640 1 -- 172.21.15.28:0/4260489608 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2855922626 0 0) 0x7fd1bc071970 con 0x7fd1bc0c5a70 2024-02-21T23:07:46.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.938+0000 7fd1baa00640 1 -- 172.21.15.28:0/4260489608 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1a0003520 con 0x7fd1bc0c5a70 2024-02-21T23:07:46.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.938+0000 7fd1b3e00640 1 -- 172.21.15.28:0/4260489608 >> v1:172.21.15.81:6789/0 conn(0x7fd1bc1561a0 legacy=0x7fd1bc06d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:46.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.938+0000 7fd1b3e00640 1 -- 172.21.15.28:0/4260489608 >> v1:172.21.15.28:6789/0 conn(0x7fd1bc0c5a70 legacy=0x7fd1bc155a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:46.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.938+0000 7fd1b3e00640 1 -- 172.21.15.28:0/4260489608 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1bc071f30 con 0x7fd1bc0c22e0 2024-02-21T23:07:46.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.940+0000 7fd1b3e00640 1 --2- 172.21.15.28:0/4260489608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1bc0c22e0 0x7fd1bc06d120 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7fd1a40097a0 tx=0x7fd1a402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53157b28823caf07 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:46.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.940+0000 7fd1baa00640 1 -- 172.21.15.28:0/4260489608 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a4035020 con 0x7fd1bc0c22e0 2024-02-21T23:07:46.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.940+0000 7fd1baa00640 1 -- 172.21.15.28:0/4260489608 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1a4031830 con 0x7fd1bc0c22e0 2024-02-21T23:07:46.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.940+0000 7fd1baa00640 1 -- 172.21.15.28:0/4260489608 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a4031c00 con 0x7fd1bc0c22e0 2024-02-21T23:07:46.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.940+0000 7fd1c2000640 1 -- 172.21.15.28:0/4260489608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1bc0c22e0 msgr2=0x7fd1bc06d120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:46.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.940+0000 7fd1c2000640 1 --2- 172.21.15.28:0/4260489608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1bc0c22e0 0x7fd1bc06d120 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7fd1a40097a0 tx=0x7fd1a402b8d0 comp rx=0 tx=0).stop 2024-02-21T23:07:46.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.941+0000 7fd1c2000640 1 -- 172.21.15.28:0/4260489608 shutdown_connections 2024-02-21T23:07:46.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.941+0000 7fd1c2000640 1 --2- 172.21.15.28:0/4260489608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1bc0c22e0 0x7fd1bc06d120 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:46.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.941+0000 7fd1c2000640 1 -- 172.21.15.28:0/4260489608 >> 172.21.15.28:0/4260489608 conn(0x7fd1bc0a85a0 msgr2=0x7fd1bc0a8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:46.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.941+0000 7fd1c2000640 1 -- 172.21.15.28:0/4260489608 shutdown_connections 2024-02-21T23:07:46.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.941+0000 7fd1c2000640 1 -- 172.21.15.28:0/4260489608 wait complete. 2024-02-21T23:07:46.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.942+0000 7fd1c2000640 1 Processor -- start 2024-02-21T23:07:46.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.942+0000 7fd1c2000640 1 -- start start 2024-02-21T23:07:46.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.943+0000 7fd1c2000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1bc0c22e0 0x7fd1bc0ab660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:46.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.943+0000 7fd1c2000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd1bc0c5a70 0x7fd1bc14bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:46.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.943+0000 7fd1c2000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1bc1561a0 0x7fd1bc148570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:46.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.943+0000 7fd1c2000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1bc0a6160 con 0x7fd1bc1561a0 2024-02-21T23:07:46.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.943+0000 7fd1c2000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1bc072580 con 0x7fd1bc0c22e0 2024-02-21T23:07:46.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.943+0000 7fd1c2000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1bc0a62e0 con 0x7fd1bc0c5a70 2024-02-21T23:07:46.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.944+0000 7fd1bbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1bc1561a0 0x7fd1bc148570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:46.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.944+0000 7fd1bbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1bc1561a0 0x7fd1bc148570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59884/0 (socket says 172.21.15.28:59884) 2024-02-21T23:07:46.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.944+0000 7fd1bbe00640 1 -- 172.21.15.28:0/1395375324 learned_addr learned my addr 172.21.15.28:0/1395375324 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:46.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.944+0000 7fd1bb400640 1 --2- 172.21.15.28:0/1395375324 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1bc0c22e0 0x7fd1bc0ab660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:46.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.944+0000 7fd1b3e00640 1 --2- 172.21.15.28:0/1395375324 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd1bc0c5a70 0x7fd1bc14bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:46.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.944+0000 7fd1bbe00640 1 -- 172.21.15.28:0/1395375324 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd1bc0c5a70 msgr2=0x7fd1bc14bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:46.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.944+0000 7fd1bbe00640 1 --2- 172.21.15.28:0/1395375324 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd1bc0c5a70 0x7fd1bc14bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:46.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.944+0000 7fd1bbe00640 1 -- 172.21.15.28:0/1395375324 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1bc0c22e0 msgr2=0x7fd1bc0ab660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:46.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.944+0000 7fd1bbe00640 1 --2- 172.21.15.28:0/1395375324 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1bc0c22e0 0x7fd1bc0ab660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:46.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.944+0000 7fd1bbe00640 1 -- 172.21.15.28:0/1395375324 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1bc151e70 con 0x7fd1bc1561a0 2024-02-21T23:07:46.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.945+0000 7fd1bbe00640 1 --2- 172.21.15.28:0/1395375324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1bc1561a0 0x7fd1bc148570 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7fd1b400c6e0 tx=0x7fd1b400cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:46.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.945+0000 7fd1b3400640 1 -- 172.21.15.28:0/1395375324 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1b4003b70 con 0x7fd1bc1561a0 2024-02-21T23:07:46.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.945+0000 7fd1b3400640 1 -- 172.21.15.28:0/1395375324 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1b4003d30 con 0x7fd1bc1561a0 2024-02-21T23:07:46.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.945+0000 7fd1b3400640 1 -- 172.21.15.28:0/1395375324 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1b4005310 con 0x7fd1bc1561a0 2024-02-21T23:07:46.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.945+0000 7fd1c2000640 1 -- 172.21.15.28:0/1395375324 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1bc0a9d50 con 0x7fd1bc1561a0 2024-02-21T23:07:46.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.945+0000 7fd1c2000640 1 -- 172.21.15.28:0/1395375324 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1bc0a7340 con 0x7fd1bc1561a0 2024-02-21T23:07:46.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.946+0000 7fd1c2000640 1 -- 172.21.15.28:0/1395375324 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1bc0c4db0 con 0x7fd1bc1561a0 2024-02-21T23:07:46.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.947+0000 7fd1b3400640 1 -- 172.21.15.28:0/1395375324 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd1b40054b0 con 0x7fd1bc1561a0 2024-02-21T23:07:46.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.948+0000 7fd1b3400640 1 --2- 172.21.15.28:0/1395375324 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd19806b500 0x7fd19806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:46.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.948+0000 7fd1b3400640 1 -- 172.21.15.28:0/1395375324 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fd1b408a230 con 0x7fd1bc1561a0 2024-02-21T23:07:46.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.949+0000 7fd1bb400640 1 --2- 172.21.15.28:0/1395375324 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd19806b500 0x7fd19806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:46.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.950+0000 7fd1bb400640 1 --2- 172.21.15.28:0/1395375324 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd19806b500 0x7fd19806d9c0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac0011e0 tx=0x7fd1ac009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:46.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:46.953+0000 7fd1b3400640 1 -- 172.21.15.28:0/1395375324 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd1b4058ec0 con 0x7fd1bc1561a0 2024-02-21T23:07:47.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.154+0000 7fd1c2000640 1 -- 172.21.15.28:0/1395375324 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) v1 -- 0x7fd1bc067f30 con 0x7fd1bc1561a0 2024-02-21T23:07:47.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.159+0000 7fd1b3400640 1 -- 172.21.15.28:0/1395375324 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v37)=0 v37) v1 ==== 116+0+0 (secure 0 0 0) 0x7fd1b40587c0 con 0x7fd1bc1561a0 2024-02-21T23:07:47.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.162+0000 7fd1c2000640 1 -- 172.21.15.28:0/1395375324 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd19806b500 msgr2=0x7fd19806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:47.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.162+0000 7fd1c2000640 1 --2- 172.21.15.28:0/1395375324 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd19806b500 0x7fd19806d9c0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac0011e0 tx=0x7fd1ac009040 comp rx=0 tx=0).stop 2024-02-21T23:07:47.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.162+0000 7fd1c2000640 1 -- 172.21.15.28:0/1395375324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1bc1561a0 msgr2=0x7fd1bc148570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:47.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.162+0000 7fd1c2000640 1 --2- 172.21.15.28:0/1395375324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1bc1561a0 0x7fd1bc148570 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7fd1b400c6e0 tx=0x7fd1b400cba0 comp rx=0 tx=0).stop 2024-02-21T23:07:47.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.162+0000 7fd1c2000640 1 -- 172.21.15.28:0/1395375324 shutdown_connections 2024-02-21T23:07:47.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.162+0000 7fd1c2000640 1 --2- 172.21.15.28:0/1395375324 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd19806b500 0x7fd19806d9c0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:47.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.162+0000 7fd1c2000640 1 --2- 172.21.15.28:0/1395375324 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd1bc0c5a70 0x7fd1bc14bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:47.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.162+0000 7fd1c2000640 1 --2- 172.21.15.28:0/1395375324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1bc1561a0 0x7fd1bc148570 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:47.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.162+0000 7fd1c2000640 1 --2- 172.21.15.28:0/1395375324 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1bc0c22e0 0x7fd1bc0ab660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:47.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.162+0000 7fd1c2000640 1 -- 172.21.15.28:0/1395375324 >> 172.21.15.28:0/1395375324 conn(0x7fd1bc0a85a0 msgr2=0x7fd1bc06d100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:47.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.162+0000 7fd1c2000640 1 -- 172.21.15.28:0/1395375324 shutdown_connections 2024-02-21T23:07:47.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.162+0000 7fd1c2000640 1 -- 172.21.15.28:0/1395375324 wait complete. 2024-02-21T23:07:47.171 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:07:47.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.403+0000 7fea01200640 1 Processor -- start 2024-02-21T23:07:47.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.404+0000 7fea01200640 1 -- start start 2024-02-21T23:07:47.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.405+0000 7fea01200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fe9fc0c11f0 0x7fe9fc153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:47.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.405+0000 7fea01200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe9fc15b6d0 0x7fe9fc15dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:47.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.405+0000 7fea01200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9fc152e70 con 0x7fe9fc0c15d0 2024-02-21T23:07:47.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.405+0000 7fea01200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9fc0c1a60 con 0x7fe9fc0c11f0 2024-02-21T23:07:47.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.405+0000 7fea01200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9fc0a5ec0 con 0x7fe9fc15b6d0 2024-02-21T23:07:47.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.406+0000 7fe9f9400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe9fc0c15d0 0x7fe9fc06dce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53422/0 (socket says 172.21.15.28:53422) 2024-02-21T23:07:47.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.406+0000 7fe9f9400640 1 -- 172.21.15.28:0/2205182561 learned_addr learned my addr 172.21.15.28:0/2205182561 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:07:47.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.406+0000 7fe9f9e00640 1 --2- 172.21.15.28:0/2205182561 >> v2:172.21.15.81:3300/0 conn(0x7fe9fc0c11f0 0x7fe9fc153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:47.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.406+0000 7fe9fa800640 1 --2- 172.21.15.28:0/2205182561 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe9fc15b6d0 0x7fe9fc15dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:47.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.406+0000 7fe9f8a00640 1 -- 172.21.15.28:0/2205182561 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3176226168 0 0) 0x7fe9fc152e70 con 0x7fe9fc0c15d0 2024-02-21T23:07:47.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.406+0000 7fe9f8a00640 1 -- 172.21.15.28:0/2205182561 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9e0003520 con 0x7fe9fc0c15d0 2024-02-21T23:07:47.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.406+0000 7fe9f8a00640 1 -- 172.21.15.28:0/2205182561 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3879230202 0 0) 0x7fe9e0003520 con 0x7fe9fc0c15d0 2024-02-21T23:07:47.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.406+0000 7fe9f8a00640 1 -- 172.21.15.28:0/2205182561 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe9fc15b6d0 msgr2=0x7fe9fc15dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:47.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.406+0000 7fe9f8a00640 1 --2- 172.21.15.28:0/2205182561 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe9fc15b6d0 0x7fe9fc15dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:47.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.406+0000 7fe9f8a00640 1 -- 172.21.15.28:0/2205182561 >> v2:172.21.15.81:3300/0 conn(0x7fe9fc0c11f0 msgr2=0x7fe9fc153950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:47.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.406+0000 7fe9f8a00640 1 --2- 172.21.15.28:0/2205182561 >> v2:172.21.15.81:3300/0 conn(0x7fe9fc0c11f0 0x7fe9fc153950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:47.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.407+0000 7fe9f8a00640 1 -- 172.21.15.28:0/2205182561 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9fc153150 con 0x7fe9fc0c15d0 2024-02-21T23:07:47.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.407+0000 7fe9f8a00640 1 -- 172.21.15.28:0/2205182561 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fe9e4001670 con 0x7fe9fc0c15d0 2024-02-21T23:07:47.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.407+0000 7fe9f8a00640 1 -- 172.21.15.28:0/2205182561 >> v1:172.21.15.28:6789/0 conn(0x7fe9fc0c15d0 legacy=0x7fe9fc06dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:47.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.407+0000 7fe9f8a00640 1 --2- 172.21.15.28:0/2205182561 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9e00039b0 0x7fe9e0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:47.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.407+0000 7fe9f8a00640 1 --2- 172.21.15.28:0/2205182561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9e0004800 0x7fe9e0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:47.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.407+0000 7fe9f8a00640 1 --2- 172.21.15.28:0/2205182561 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe9fc15b6d0 0x7fe9e00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:47.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.407+0000 7fe9f8a00640 1 -- 172.21.15.28:0/2205182561 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9fc0c1a60 con 0x7fe9e0004800 2024-02-21T23:07:47.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.407+0000 7fe9f8a00640 1 -- 172.21.15.28:0/2205182561 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9fc0a5ec0 con 0x7fe9e00039b0 2024-02-21T23:07:47.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.407+0000 7fe9f8a00640 1 -- 172.21.15.28:0/2205182561 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e000baa0 con 0x7fe9fc15b6d0 2024-02-21T23:07:47.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.407+0000 7fe9f9400640 1 --2- 172.21.15.28:0/2205182561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9e0004800 0x7fe9e0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:47.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.407+0000 7fe9f9e00640 1 --2- 172.21.15.28:0/2205182561 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9e00039b0 0x7fe9e0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:47.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.407+0000 7fe9fa800640 1 --2- 172.21.15.28:0/2205182561 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe9fc15b6d0 0x7fe9e00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:47.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.407+0000 7fe9f9400640 1 -- 172.21.15.28:0/2205182561 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe9fc15b6d0 msgr2=0x7fe9e00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:47.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.407+0000 7fe9f9400640 1 --2- 172.21.15.28:0/2205182561 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe9fc15b6d0 0x7fe9e00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:47.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.407+0000 7fe9f9400640 1 -- 172.21.15.28:0/2205182561 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9e00039b0 msgr2=0x7fe9e0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:47.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.407+0000 7fe9f9400640 1 --2- 172.21.15.28:0/2205182561 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9e00039b0 0x7fe9e0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:47.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.407+0000 7fe9f9400640 1 -- 172.21.15.28:0/2205182561 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9e0003520 con 0x7fe9e0004800 2024-02-21T23:07:47.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.407+0000 7fe9f9e00640 1 --2- 172.21.15.28:0/2205182561 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9e00039b0 0x7fe9e0003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:07:47.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.407+0000 7fe9fa800640 1 --2- 172.21.15.28:0/2205182561 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe9fc15b6d0 0x7fe9e00051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:07:47.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.409+0000 7fe9f9400640 1 --2- 172.21.15.28:0/2205182561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9e0004800 0x7fe9e0004c80 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7fe9e4001050 tx=0x7fe9e0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b70d3fdbb184d28d server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:47.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.409+0000 7fe9f8a00640 1 -- 172.21.15.28:0/2205182561 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9e4007c80 con 0x7fe9e0004800 2024-02-21T23:07:47.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.409+0000 7fe9f8a00640 1 -- 172.21.15.28:0/2205182561 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9e4030070 con 0x7fe9e0004800 2024-02-21T23:07:47.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.409+0000 7fe9f8a00640 1 -- 172.21.15.28:0/2205182561 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9e4005d30 con 0x7fe9e0004800 2024-02-21T23:07:47.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.409+0000 7fea01200640 1 -- 172.21.15.28:0/2205182561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9e0004800 msgr2=0x7fe9e0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:47.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.409+0000 7fea01200640 1 --2- 172.21.15.28:0/2205182561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9e0004800 0x7fe9e0004c80 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7fe9e4001050 tx=0x7fe9e0005830 comp rx=0 tx=0).stop 2024-02-21T23:07:47.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.409+0000 7fe9fa800640 1 -- 172.21.15.28:0/2205182561 reap_dead start 2024-02-21T23:07:47.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.410+0000 7fea01200640 1 -- 172.21.15.28:0/2205182561 shutdown_connections 2024-02-21T23:07:47.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.410+0000 7fea01200640 1 -- 172.21.15.28:0/2205182561 >> 172.21.15.28:0/2205182561 conn(0x7fe9fc0c67f0 msgr2=0x7fe9fc0c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:47.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.410+0000 7fea01200640 1 -- 172.21.15.28:0/2205182561 shutdown_connections 2024-02-21T23:07:47.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.410+0000 7fea01200640 1 -- 172.21.15.28:0/2205182561 wait complete. 2024-02-21T23:07:47.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.411+0000 7fea01200640 1 Processor -- start 2024-02-21T23:07:47.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.411+0000 7fea01200640 1 -- start start 2024-02-21T23:07:47.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.412+0000 7fea01200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9e0004800 0x7fe9fc13f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:47.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.412+0000 7fea01200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9fc152a10 0x7fe9fc0a74f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:47.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.412+0000 7fea01200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe9fc0a7a30 0x7fe9fc148210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:47.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.412+0000 7fea01200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9fc0c4c00 con 0x7fe9e0004800 2024-02-21T23:07:47.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.412+0000 7fea01200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9fc14d1c0 con 0x7fe9fc152a10 2024-02-21T23:07:47.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.412+0000 7fea01200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9fc0a6110 con 0x7fe9fc0a7a30 2024-02-21T23:07:47.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.413+0000 7fe9f9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9e0004800 0x7fe9fc13f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:47.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.413+0000 7fe9f9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9e0004800 0x7fe9fc13f4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59904/0 (socket says 172.21.15.28:59904) 2024-02-21T23:07:47.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.413+0000 7fe9f9e00640 1 -- 172.21.15.28:0/3039034304 learned_addr learned my addr 172.21.15.28:0/3039034304 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:47.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.413+0000 7fe9f9400640 1 --2- 172.21.15.28:0/3039034304 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9fc152a10 0x7fe9fc0a74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:47.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.413+0000 7fe9f9e00640 1 -- 172.21.15.28:0/3039034304 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe9fc0a7a30 msgr2=0x7fe9fc148210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:47.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.413+0000 7fe9fa800640 1 --2- 172.21.15.28:0/3039034304 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe9fc0a7a30 0x7fe9fc148210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:47.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.413+0000 7fe9f9e00640 1 --2- 172.21.15.28:0/3039034304 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe9fc0a7a30 0x7fe9fc148210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:47.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.413+0000 7fe9f9e00640 1 -- 172.21.15.28:0/3039034304 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9fc152a10 msgr2=0x7fe9fc0a74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:47.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.413+0000 7fe9f9e00640 1 --2- 172.21.15.28:0/3039034304 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9fc152a10 0x7fe9fc0a74f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:47.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.413+0000 7fe9f9e00640 1 -- 172.21.15.28:0/3039034304 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9fc0ac540 con 0x7fe9e0004800 2024-02-21T23:07:47.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.413+0000 7fe9f9400640 1 --2- 172.21.15.28:0/3039034304 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9fc152a10 0x7fe9fc0a74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:07:47.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.413+0000 7fe9f9e00640 1 --2- 172.21.15.28:0/3039034304 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9e0004800 0x7fe9fc13f4e0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec00f430 tx=0x7fe9ec00f8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:47.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.414+0000 7fe9f2000640 1 -- 172.21.15.28:0/3039034304 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9ec003990 con 0x7fe9e0004800 2024-02-21T23:07:47.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.414+0000 7fe9f2000640 1 -- 172.21.15.28:0/3039034304 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9ec003b30 con 0x7fe9e0004800 2024-02-21T23:07:47.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.414+0000 7fe9f2000640 1 -- 172.21.15.28:0/3039034304 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9ec004020 con 0x7fe9e0004800 2024-02-21T23:07:47.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.414+0000 7fea01200640 1 -- 172.21.15.28:0/3039034304 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9fc1e9bd0 con 0x7fe9e0004800 2024-02-21T23:07:47.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.414+0000 7fea01200640 1 -- 172.21.15.28:0/3039034304 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9fc14a070 con 0x7fe9e0004800 2024-02-21T23:07:47.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.415+0000 7fea01200640 1 -- 172.21.15.28:0/3039034304 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9fc067f30 con 0x7fe9e0004800 2024-02-21T23:07:47.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.416+0000 7fe9f2000640 1 -- 172.21.15.28:0/3039034304 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe9ec005d10 con 0x7fe9e0004800 2024-02-21T23:07:47.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.417+0000 7fe9f2000640 1 --2- 172.21.15.28:0/3039034304 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe9d806b500 0x7fe9d806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:47.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.418+0000 7fe9f2000640 1 -- 172.21.15.28:0/3039034304 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fe9ec08acf0 con 0x7fe9e0004800 2024-02-21T23:07:47.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.418+0000 7fe9f9400640 1 --2- 172.21.15.28:0/3039034304 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe9d806b500 0x7fe9d806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:47.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.419+0000 7fe9f9400640 1 --2- 172.21.15.28:0/3039034304 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe9d806b500 0x7fe9d806d9c0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fe9e4034000 tx=0x7fe9e4007000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:47.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.422+0000 7fe9f2000640 1 -- 172.21.15.28:0/3039034304 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe9ec08f070 con 0x7fe9e0004800 2024-02-21T23:07:47.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.658+0000 7fea01200640 1 -- 172.21.15.28:0/3039034304 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe9fc0c0960 con 0x7fe9e0004800 2024-02-21T23:07:47.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.659+0000 7fe9f2000640 1 -- 172.21.15.28:0/3039034304 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 109 v109) v1 ==== 95+0+4948 (secure 0 0 0) 0x7fe9ec024020 con 0x7fe9e0004800 2024-02-21T23:07:47.647 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:07:47.648 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":109,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5732,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3963422563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3963422563},{"type":"v1","addr":"172.21.15.81:6837","nonce":3963422563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7446,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/1119726035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1119726035},{"type":"v1","addr":"172.21.15.28:6833","nonce":1119726035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":109,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-02-21T23:07:29.533547+0000","modified":"2024-02-21T23:07:46.185982+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5729},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5729":{"gid":5729,"name":"d","rank":0,"incarnation":104,"state":"up:active","state_seq":8,"addr":"172.21.15.81:6835/1754860807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1754860807},{"type":"v1","addr":"172.21.15.81:6835","nonce":1754860807}]},"join_fscid":8,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7632":{"gid":7632,"name":"a","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6835/780497578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":780497578},{"type":"v1","addr":"172.21.15.28:6835","nonce":780497578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-02-21T23:07:47.648 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 109 2024-02-21T23:07:47.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.662+0000 7fea01200640 1 -- 172.21.15.28:0/3039034304 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe9d806b500 msgr2=0x7fe9d806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:47.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.662+0000 7fea01200640 1 --2- 172.21.15.28:0/3039034304 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe9d806b500 0x7fe9d806d9c0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fe9e4034000 tx=0x7fe9e4007000 comp rx=0 tx=0).stop 2024-02-21T23:07:47.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.662+0000 7fea01200640 1 -- 172.21.15.28:0/3039034304 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9e0004800 msgr2=0x7fe9fc13f4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:47.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.662+0000 7fea01200640 1 --2- 172.21.15.28:0/3039034304 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9e0004800 0x7fe9fc13f4e0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec00f430 tx=0x7fe9ec00f8f0 comp rx=0 tx=0).stop 2024-02-21T23:07:47.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.662+0000 7fea01200640 1 -- 172.21.15.28:0/3039034304 shutdown_connections 2024-02-21T23:07:47.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.662+0000 7fea01200640 1 --2- 172.21.15.28:0/3039034304 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe9d806b500 0x7fe9d806d9c0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:47.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.662+0000 7fea01200640 1 --2- 172.21.15.28:0/3039034304 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe9fc0a7a30 0x7fe9fc148210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:47.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.662+0000 7fea01200640 1 --2- 172.21.15.28:0/3039034304 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9fc152a10 0x7fe9fc0a74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:47.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.662+0000 7fea01200640 1 --2- 172.21.15.28:0/3039034304 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9e0004800 0x7fe9fc13f4e0 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:47.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.662+0000 7fea01200640 1 -- 172.21.15.28:0/3039034304 >> 172.21.15.28:0/3039034304 conn(0x7fe9fc0c67f0 msgr2=0x7fe9fc143950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:47.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.663+0000 7fea01200640 1 -- 172.21.15.28:0/3039034304 shutdown_connections 2024-02-21T23:07:47.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:47.663+0000 7fea01200640 1 -- 172.21.15.28:0/3039034304 wait complete. 2024-02-21T23:07:48.075 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 109, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-02-21T23:07:29.533547+0000', 'modified': '2024-02-21T23:07:46.185982+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 111, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5729}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5729': {'gid': 5729, 'name': 'd', 'rank': 0, 'incarnation': 104, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.81:6835/1754860807', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6834', 'nonce': 1754860807}, {'type': 'v1', 'addr': '172.21.15.81:6835', 'nonce': 1754860807}]}, 'join_fscid': 8, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_7632': {'gid': 7632, 'name': 'a', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.28:6835/780497578', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 780497578}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 780497578}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:07:48.075 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-21T23:07:48.075 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-02-21T23:07:48.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.314+0000 7f0ca5a00640 1 Processor -- start 2024-02-21T23:07:48.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.314+0000 7f0ca5a00640 1 -- start start 2024-02-21T23:07:48.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.316+0000 7f0ca5a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0ca0158c20 0x7f0ca0160890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:48.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.316+0000 7f0ca5a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ca01550b0 con 0x7f0ca00c8740 2024-02-21T23:07:48.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.316+0000 7f0ca5a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ca0155350 con 0x7f0ca00b0130 2024-02-21T23:07:48.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.316+0000 7f0ca5a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ca0152a40 con 0x7f0ca0158c20 2024-02-21T23:07:48.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.316+0000 7f0c97e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f0ca00c8740 0x7f0ca0158510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53424/0 (socket says 172.21.15.28:53424) 2024-02-21T23:07:48.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.316+0000 7f0c97e00640 1 -- 172.21.15.28:0/266423703 learned_addr learned my addr 172.21.15.28:0/266423703 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:07:48.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.316+0000 7f0ca5000640 1 --2- 172.21.15.28:0/266423703 >> v2:172.21.15.81:3300/0 conn(0x7f0ca0158c20 0x7f0ca0160890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:48.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.317+0000 7f0c9f400640 1 -- 172.21.15.28:0/266423703 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2972049776 0 0) 0x7f0ca01550b0 con 0x7f0ca00c8740 2024-02-21T23:07:48.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.317+0000 7f0c9f400640 1 -- 172.21.15.28:0/266423703 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c84003520 con 0x7f0ca00c8740 2024-02-21T23:07:48.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.317+0000 7f0c9f400640 1 -- 172.21.15.28:0/266423703 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2816891197 0 0) 0x7f0c84003520 con 0x7f0ca00c8740 2024-02-21T23:07:48.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.317+0000 7f0c9f400640 1 -- 172.21.15.28:0/266423703 >> v2:172.21.15.81:3300/0 conn(0x7f0ca0158c20 msgr2=0x7f0ca0160890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:48.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.317+0000 7f0c9f400640 1 --2- 172.21.15.28:0/266423703 >> v2:172.21.15.81:3300/0 conn(0x7f0ca0158c20 0x7f0ca0160890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:48.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.317+0000 7f0c9f400640 1 -- 172.21.15.28:0/266423703 >> v1:172.21.15.81:6789/0 conn(0x7f0ca00b0130 legacy=0x7f0ca00c51c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:48.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.317+0000 7f0c9f400640 1 -- 172.21.15.28:0/266423703 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ca0155630 con 0x7f0ca00c8740 2024-02-21T23:07:48.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.317+0000 7f0c9f400640 1 -- 172.21.15.28:0/266423703 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f0c88001690 con 0x7f0ca00c8740 2024-02-21T23:07:48.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.317+0000 7f0c9f400640 1 -- 172.21.15.28:0/266423703 >> v1:172.21.15.28:6789/0 conn(0x7f0ca00c8740 legacy=0x7f0ca0158510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:48.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.317+0000 7f0c9f400640 1 --2- 172.21.15.28:0/266423703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c840039b0 0x7f0c84003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:48.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.317+0000 7f0c9f400640 1 --2- 172.21.15.28:0/266423703 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0c84004800 0x7f0c84004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:48.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.317+0000 7f0c9f400640 1 --2- 172.21.15.28:0/266423703 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0c840051c0 0x7f0c8400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:48.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.317+0000 7f0c9f400640 1 -- 172.21.15.28:0/266423703 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca0152a40 con 0x7f0c840039b0 2024-02-21T23:07:48.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.317+0000 7f0c9f400640 1 -- 172.21.15.28:0/266423703 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c8400c160 con 0x7f0c840051c0 2024-02-21T23:07:48.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.317+0000 7f0c9f400640 1 -- 172.21.15.28:0/266423703 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c8400c360 con 0x7f0c84004800 2024-02-21T23:07:48.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.318+0000 7f0c9fe00640 1 --2- 172.21.15.28:0/266423703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c840039b0 0x7f0c84003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:48.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.318+0000 7f0c97e00640 1 --2- 172.21.15.28:0/266423703 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0c84004800 0x7f0c84004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:48.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.318+0000 7f0ca5000640 1 --2- 172.21.15.28:0/266423703 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0c840051c0 0x7f0c8400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:48.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.318+0000 7f0c9fe00640 1 -- 172.21.15.28:0/266423703 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0c84004800 msgr2=0x7f0c84004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:48.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.318+0000 7f0c9fe00640 1 --2- 172.21.15.28:0/266423703 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0c84004800 0x7f0c84004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:48.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.318+0000 7f0c9fe00640 1 -- 172.21.15.28:0/266423703 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0c840051c0 msgr2=0x7f0c8400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:48.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.318+0000 7f0c9fe00640 1 --2- 172.21.15.28:0/266423703 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0c840051c0 0x7f0c8400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:48.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.318+0000 7f0c9fe00640 1 -- 172.21.15.28:0/266423703 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c84003520 con 0x7f0c840039b0 2024-02-21T23:07:48.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.318+0000 7f0ca5000640 1 --2- 172.21.15.28:0/266423703 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0c840051c0 0x7f0c8400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:07:48.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.318+0000 7f0ca5000640 1 -- 172.21.15.28:0/266423703 reap_dead start 2024-02-21T23:07:48.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.318+0000 7f0c97e00640 1 --2- 172.21.15.28:0/266423703 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0c84004800 0x7f0c84004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:07:48.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.319+0000 7f0c9fe00640 1 --2- 172.21.15.28:0/266423703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c840039b0 0x7f0c84003dd0 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f0c90001750 tx=0x7f0c84005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac2a6f0b7b19fb5 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:48.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.319+0000 7f0c9f400640 1 -- 172.21.15.28:0/266423703 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c90001370 con 0x7f0c840039b0 2024-02-21T23:07:48.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.319+0000 7f0c9f400640 1 -- 172.21.15.28:0/266423703 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c900052d0 con 0x7f0c840039b0 2024-02-21T23:07:48.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.319+0000 7f0c9f400640 1 -- 172.21.15.28:0/266423703 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c90003830 con 0x7f0c840039b0 2024-02-21T23:07:48.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.320+0000 7f0ca5a00640 1 -- 172.21.15.28:0/266423703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c840039b0 msgr2=0x7f0c84003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:48.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.320+0000 7f0ca5a00640 1 --2- 172.21.15.28:0/266423703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c840039b0 0x7f0c84003dd0 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f0c90001750 tx=0x7f0c84005830 comp rx=0 tx=0).stop 2024-02-21T23:07:48.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.320+0000 7f0ca5a00640 1 -- 172.21.15.28:0/266423703 shutdown_connections 2024-02-21T23:07:48.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.320+0000 7f0ca5a00640 1 --2- 172.21.15.28:0/266423703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c840039b0 0x7f0c84003dd0 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:48.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.320+0000 7f0ca5a00640 1 -- 172.21.15.28:0/266423703 >> 172.21.15.28:0/266423703 conn(0x7f0ca00ac370 msgr2=0x7f0ca00ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:48.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.320+0000 7f0ca5a00640 1 -- 172.21.15.28:0/266423703 shutdown_connections 2024-02-21T23:07:48.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.320+0000 7f0ca5a00640 1 -- 172.21.15.28:0/266423703 wait complete. 2024-02-21T23:07:48.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.322+0000 7f0ca5a00640 1 Processor -- start 2024-02-21T23:07:48.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.322+0000 7f0ca5a00640 1 -- start start 2024-02-21T23:07:48.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.323+0000 7f0ca5a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0c840039b0 0x7f0ca01eab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:48.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.323+0000 7f0ca5a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ca00c13c0 0x7f0ca01ee8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:48.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.323+0000 7f0ca5a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0ca00bf610 0x7f0ca00bfa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:48.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.323+0000 7f0ca5a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca00c8dd0 con 0x7f0ca00c13c0 2024-02-21T23:07:48.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.323+0000 7f0ca5a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca0155e30 con 0x7f0c840039b0 2024-02-21T23:07:48.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.323+0000 7f0ca5a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca00be060 con 0x7f0ca00bf610 2024-02-21T23:07:48.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.324+0000 7f0c97e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ca00c13c0 0x7f0ca01ee8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:48.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.324+0000 7f0c97e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ca00c13c0 0x7f0ca01ee8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59912/0 (socket says 172.21.15.28:59912) 2024-02-21T23:07:48.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.324+0000 7f0c97e00640 1 -- 172.21.15.28:0/3080561421 learned_addr learned my addr 172.21.15.28:0/3080561421 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:48.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.324+0000 7f0ca5000640 1 --2- 172.21.15.28:0/3080561421 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0ca00bf610 0x7f0ca00bfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:48.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.324+0000 7f0c9fe00640 1 --2- 172.21.15.28:0/3080561421 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0c840039b0 0x7f0ca01eab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:48.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.324+0000 7f0c97e00640 1 -- 172.21.15.28:0/3080561421 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0ca00bf610 msgr2=0x7f0ca00bfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:48.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.324+0000 7f0c97e00640 1 --2- 172.21.15.28:0/3080561421 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0ca00bf610 0x7f0ca00bfa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:48.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.324+0000 7f0c97e00640 1 -- 172.21.15.28:0/3080561421 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0c840039b0 msgr2=0x7f0ca01eab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:48.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.324+0000 7f0c97e00640 1 --2- 172.21.15.28:0/3080561421 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0c840039b0 0x7f0ca01eab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:48.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.324+0000 7f0c97e00640 1 -- 172.21.15.28:0/3080561421 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ca00c37e0 con 0x7f0ca00c13c0 2024-02-21T23:07:48.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.324+0000 7f0c97e00640 1 --2- 172.21.15.28:0/3080561421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ca00c13c0 0x7f0ca01ee8b0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f0c880011e0 tx=0x7f0c8800db40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:48.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.325+0000 7f0c9cc00640 1 -- 172.21.15.28:0/3080561421 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c88019070 con 0x7f0ca00c13c0 2024-02-21T23:07:48.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.325+0000 7f0c9cc00640 1 -- 172.21.15.28:0/3080561421 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c880072d0 con 0x7f0ca00c13c0 2024-02-21T23:07:48.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.325+0000 7f0c9cc00640 1 -- 172.21.15.28:0/3080561421 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c880047e0 con 0x7f0ca00c13c0 2024-02-21T23:07:48.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.325+0000 7f0ca5a00640 1 -- 172.21.15.28:0/3080561421 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ca00c3ac0 con 0x7f0ca00c13c0 2024-02-21T23:07:48.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.325+0000 7f0ca5a00640 1 -- 172.21.15.28:0/3080561421 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ca01ee0b0 con 0x7f0ca00c13c0 2024-02-21T23:07:48.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.327+0000 7f0c9cc00640 1 -- 172.21.15.28:0/3080561421 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0c8801c070 con 0x7f0ca00c13c0 2024-02-21T23:07:48.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.329+0000 7f0c9cc00640 1 --2- 172.21.15.28:0/3080561421 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0c7c06b500 0x7f0c7c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:48.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.329+0000 7f0c9cc00640 1 -- 172.21.15.28:0/3080561421 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f0c88089ac0 con 0x7f0ca00c13c0 2024-02-21T23:07:48.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.329+0000 7f0c9fe00640 1 --2- 172.21.15.28:0/3080561421 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0c7c06b500 0x7f0c7c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:48.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.329+0000 7f0caa024740 1 Processor -- start 2024-02-21T23:07:48.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.330+0000 7f0caa024740 1 -- start start 2024-02-21T23:07:48.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.330+0000 7f0caa024740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0373d5020 0x55b0373d5400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:48.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.330+0000 7f0caa024740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b0373d6330 0x55b037411170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:48.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.331+0000 7f0caa024740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55b0374116b0 0x55b03740f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:48.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.331+0000 7f0caa024740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b037339470 con 0x55b0373d5020 2024-02-21T23:07:48.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.331+0000 7f0caa024740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0372e2af0 con 0x55b0373d6330 2024-02-21T23:07:48.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.331+0000 7f0caa024740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b037335510 con 0x55b0374116b0 2024-02-21T23:07:48.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.331+0000 7f0c9fe00640 1 --2- 172.21.15.28:0/3080561421 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0c7c06b500 0x7f0c7c06d9c0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f0c90033000 tx=0x7f0c90009030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:48.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.331+0000 7f0c9fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0373d5020 0x55b0373d5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:48.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.331+0000 7f0c9fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0373d5020 0x55b0373d5400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59926/0 (socket says 172.21.15.28:59926) 2024-02-21T23:07:48.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.331+0000 7f0c9fe00640 1 -- 172.21.15.28:0/1465059343 learned_addr learned my addr 172.21.15.28:0/1465059343 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:48.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.331+0000 7f0c97e00640 1 --2- 172.21.15.28:0/1465059343 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b0373d6330 0x55b037411170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:48.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.331+0000 7f0ca5000640 1 --2- 172.21.15.28:0/1465059343 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55b0374116b0 0x55b03740f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:48.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.331+0000 7f0c9fe00640 1 -- 172.21.15.28:0/1465059343 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55b0374116b0 msgr2=0x55b03740f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:48.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.331+0000 7f0c9fe00640 1 --2- 172.21.15.28:0/1465059343 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55b0374116b0 0x55b03740f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:48.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.331+0000 7f0c9fe00640 1 -- 172.21.15.28:0/1465059343 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b0373d6330 msgr2=0x55b037411170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:48.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.331+0000 7f0c9fe00640 1 --2- 172.21.15.28:0/1465059343 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b0373d6330 0x55b037411170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:48.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.331+0000 7f0c9fe00640 1 -- 172.21.15.28:0/1465059343 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b0372e9d90 con 0x55b0373d5020 2024-02-21T23:07:48.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.332+0000 7f0c9fe00640 1 --2- 172.21.15.28:0/1465059343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0373d5020 0x55b0373d5400 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f0c90001840 tx=0x7f0c9002f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b23cce4ba3270857 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:48.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.332+0000 7f0c94c00640 1 -- 172.21.15.28:0/1465059343 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c9004c070 con 0x55b0373d5020 2024-02-21T23:07:48.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.332+0000 7f0c94c00640 1 -- 172.21.15.28:0/1465059343 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c90007680 con 0x55b0373d5020 2024-02-21T23:07:48.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.332+0000 7f0c94c00640 1 -- 172.21.15.28:0/1465059343 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c90042040 con 0x55b0373d5020 2024-02-21T23:07:48.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.333+0000 7f0caa024740 1 -- 172.21.15.28:0/1465059343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0373d5020 msgr2=0x55b0373d5400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:48.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.333+0000 7f0caa024740 1 --2- 172.21.15.28:0/1465059343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0373d5020 0x55b0373d5400 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f0c90001840 tx=0x7f0c9002f000 comp rx=0 tx=0).stop 2024-02-21T23:07:48.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.333+0000 7f0caa024740 1 -- 172.21.15.28:0/1465059343 shutdown_connections 2024-02-21T23:07:48.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.333+0000 7f0caa024740 1 --2- 172.21.15.28:0/1465059343 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55b0374116b0 0x55b03740f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:48.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.333+0000 7f0caa024740 1 --2- 172.21.15.28:0/1465059343 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b0373d6330 0x55b037411170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:48.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.333+0000 7f0caa024740 1 --2- 172.21.15.28:0/1465059343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0373d5020 0x55b0373d5400 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:48.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.333+0000 7f0caa024740 1 -- 172.21.15.28:0/1465059343 >> 172.21.15.28:0/1465059343 conn(0x55b0373d1970 msgr2=0x55b0373f6380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:48.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.333+0000 7f0caa024740 1 -- 172.21.15.28:0/1465059343 shutdown_connections 2024-02-21T23:07:48.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.333+0000 7f0caa024740 1 -- 172.21.15.28:0/1465059343 wait complete. 2024-02-21T23:07:48.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.334+0000 7f0caa024740 1 -- start start 2024-02-21T23:07:48.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.334+0000 7f0caa024740 1 Processor -- start 2024-02-21T23:07:48.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.335+0000 7f0caa024740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b0373d6330 0x55b037424820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:48.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.335+0000 7f0caa024740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0374116b0 0x55b0373f4d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:48.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.335+0000 7f0caa024740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55b03740be90 0x55b0373f0500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:48.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.335+0000 7f0caa024740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0372d8040 con 0x55b0374116b0 2024-02-21T23:07:48.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.335+0000 7f0caa024740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0373f6a90 con 0x55b0373d6330 2024-02-21T23:07:48.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.335+0000 7f0caa024740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b0373f5ab0 con 0x55b03740be90 2024-02-21T23:07:48.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.335+0000 7f0caa024740 10 client.0 _ll_register_callbacks cb 0x55b0372f5920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:07:48.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.335+0000 7f0c97e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0374116b0 0x55b0373f4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:48.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.335+0000 7f0c97e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0374116b0 0x55b0373f4d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59934/0 (socket says 172.21.15.28:59934) 2024-02-21T23:07:48.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.336+0000 7f0c97e00640 1 -- 172.21.15.28:0/2612834658 learned_addr learned my addr 172.21.15.28:0/2612834658 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:48.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.336+0000 7f0c9fe00640 1 --2- 172.21.15.28:0/2612834658 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b0373d6330 0x55b037424820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:48.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.336+0000 7f0ca5000640 1 --2- 172.21.15.28:0/2612834658 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55b03740be90 0x55b0373f0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:48.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.336+0000 7f0c97e00640 1 -- 172.21.15.28:0/2612834658 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55b03740be90 msgr2=0x55b0373f0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:48.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.336+0000 7f0c97e00640 1 --2- 172.21.15.28:0/2612834658 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55b03740be90 0x55b0373f0500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:48.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.336+0000 7f0c97e00640 1 -- 172.21.15.28:0/2612834658 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b0373d6330 msgr2=0x55b037424820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:48.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.336+0000 7f0c97e00640 1 --2- 172.21.15.28:0/2612834658 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b0373d6330 0x55b037424820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:48.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.336+0000 7f0c97e00640 1 -- 172.21.15.28:0/2612834658 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b0372f7aa0 con 0x55b0374116b0 2024-02-21T23:07:48.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.336+0000 7f0ca5000640 1 --2- 172.21.15.28:0/2612834658 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55b03740be90 0x55b0373f0500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:07:48.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.336+0000 7f0c97e00640 1 --2- 172.21.15.28:0/2612834658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0374116b0 0x55b0373f4d90 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f0c88087e00 tx=0x7f0c880581c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9302c040e76867c server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:48.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.336+0000 7f0c8d600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:07:48.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.336+0000 7f0c8d600640 1 -- 172.21.15.28:0/2612834658 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c88086060 con 0x55b0374116b0 2024-02-21T23:07:48.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.336+0000 7f0c8d600640 1 -- 172.21.15.28:0/2612834658 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(111..111 src has 1..111) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f0c88087a90 con 0x55b0374116b0 2024-02-21T23:07:48.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.337+0000 7f0caa024740 1 -- 172.21.15.28:0/2612834658 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b0373469b0 con 0x55b0374116b0 2024-02-21T23:07:48.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.337+0000 7f0c8d600640 1 -- 172.21.15.28:0/2612834658 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=109) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0c88074050 con 0x55b0374116b0 2024-02-21T23:07:48.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.337+0000 7f0caa024740 10 client.7695 fetch_fsmap learned FSMap version 109 2024-02-21T23:07:48.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.337+0000 7f0caa024740 1 -- 172.21.15.28:0/2612834658 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=109,monmap=2+}) v3 -- 0x55b03704e2d0 con 0x55b0374116b0 2024-02-21T23:07:48.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.337+0000 7f0c8d600640 1 -- 172.21.15.28:0/2612834658 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55b0373469b0 con 0x55b0374116b0 2024-02-21T23:07:48.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.337+0000 7f0c8d600640 1 -- 172.21.15.28:0/2612834658 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 109) v1 ==== 3282+0+0 (secure 0 0 0) 0x7f0c88057550 con 0x55b0374116b0 2024-02-21T23:07:48.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.337+0000 7f0caa024740 10 client.7695 fetch_fsmap finished waiting for FSMap version 109 2024-02-21T23:07:48.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.337+0000 7f0caa024740 10 client.7695 resolve_mds: resolved name 'd' to daemon mds.d 2024-02-21T23:07:48.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.337+0000 7f0caa024740 20 client.7695 populate_metadata read hostname 'smithi028' 2024-02-21T23:07:48.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.337+0000 7f0caa024740 1 --2- 172.21.15.28:0/2612834658 >> [v2:172.21.15.81:6834/1754860807,v1:172.21.15.81:6835/1754860807] conn(0x55b0374216f0 0x55b03742faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:48.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.337+0000 7f0caa024740 4 client.7695 mds_command: new command op to 5729 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-21T23:07:48.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.337+0000 7f0caa024740 1 -- 172.21.15.28:0/2612834658 --> [v2:172.21.15.81:6834/1754860807,v1:172.21.15.81:6835/1754860807] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b037340d00 con 0x55b0374216f0 2024-02-21T23:07:48.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.338+0000 7f0c9fe00640 1 --2- 172.21.15.28:0/2612834658 >> [v2:172.21.15.81:6834/1754860807,v1:172.21.15.81:6835/1754860807] conn(0x55b0374216f0 0x55b03742faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:48.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.339+0000 7f0c9fe00640 1 --2- 172.21.15.28:0/2612834658 >> [v2:172.21.15.81:6834/1754860807,v1:172.21.15.81:6835/1754860807] conn(0x55b0374216f0 0x55b03742faa0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=41aa29bc8e8ccff1 server_cookie=28d528e8e18dbb89 in_seq=0 out_seq=0 2024-02-21T23:07:48.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.339+0000 7f0c8d600640 10 client.7695 ms_handle_connect on v2:172.21.15.81:6834/1754860807 2024-02-21T23:07:48.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.340+0000 7f0c8d600640 1 -- 172.21.15.28:0/2612834658 <== mds.0 v2:172.21.15.81:6834/1754860807 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55b037340d00 con 0x55b0374216f0 2024-02-21T23:07:48.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.340+0000 7f0c8d600640 10 client.7695 handle_command_reply: tid=0 2024-02-21T23:07:48.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.340+0000 7f0caa024740 1 client.7695 shutdown 2024-02-21T23:07:48.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.341+0000 7f0caa024740 1 -- 172.21.15.28:0/2612834658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0374116b0 msgr2=0x55b0373f4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:48.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.341+0000 7f0caa024740 1 --2- 172.21.15.28:0/2612834658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0374116b0 0x55b0373f4d90 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f0c88087e00 tx=0x7f0c880581c0 comp rx=0 tx=0).stop 2024-02-21T23:07:48.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.341+0000 7f0caa024740 1 -- 172.21.15.28:0/2612834658 shutdown_connections 2024-02-21T23:07:48.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.341+0000 7f0caa024740 1 --2- 172.21.15.28:0/2612834658 >> [v2:172.21.15.81:6834/1754860807,v1:172.21.15.81:6835/1754860807] conn(0x55b0374216f0 0x55b03742faa0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:48.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.341+0000 7f0caa024740 1 --2- 172.21.15.28:0/2612834658 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55b03740be90 0x55b0373f0500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:48.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.341+0000 7f0caa024740 1 --2- 172.21.15.28:0/2612834658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0374116b0 0x55b0373f4d90 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:48.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.341+0000 7f0caa024740 1 --2- 172.21.15.28:0/2612834658 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b0373d6330 0x55b037424820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:48.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.341+0000 7f0c8d600640 0 client.7695 ms_handle_reset on v2:172.21.15.81:6834/1754860807 2024-02-21T23:07:48.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.341+0000 7f0caa024740 1 -- 172.21.15.28:0/2612834658 >> 172.21.15.28:0/2612834658 conn(0x55b0373d5020 msgr2=0x55b03740ff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:48.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.341+0000 7f0caa024740 1 -- 172.21.15.28:0/2612834658 shutdown_connections 2024-02-21T23:07:48.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.341+0000 7f0caa024740 1 -- 172.21.15.28:0/2612834658 wait complete. 2024-02-21T23:07:48.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.341+0000 7f0caa024740 20 client.7695 trim_cache size 0 max 16384 2024-02-21T23:07:48.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.360+0000 7f0caa024740 1 Processor -- start 2024-02-21T23:07:48.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.360+0000 7f0caa024740 1 -- start start 2024-02-21T23:07:48.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.361+0000 7f0caa024740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b0373d6330 0x55b037410db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:48.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.361+0000 7f0caa024740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b03740be90 0x55b0373f00a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:48.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.361+0000 7f0caa024740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55b0374116b0 0x55b0373ecb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:48.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.361+0000 7f0caa024740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca01f8270 con 0x55b03740be90 2024-02-21T23:07:48.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.361+0000 7f0caa024740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca01f68e0 con 0x55b0373d6330 2024-02-21T23:07:48.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.361+0000 7f0caa024740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca01f6ee0 con 0x55b0374116b0 2024-02-21T23:07:48.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.361+0000 7f0c97e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b03740be90 0x55b0373f00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:48.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.361+0000 7f0c97e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b03740be90 0x55b0373f00a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59940/0 (socket says 172.21.15.28:59940) 2024-02-21T23:07:48.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.361+0000 7f0c97e00640 1 -- 172.21.15.28:0/493400690 learned_addr learned my addr 172.21.15.28:0/493400690 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:48.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.361+0000 7f0ca5000640 1 --2- 172.21.15.28:0/493400690 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55b0374116b0 0x55b0373ecb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:48.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.361+0000 7f0c9fe00640 1 --2- 172.21.15.28:0/493400690 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b0373d6330 0x55b037410db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:48.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.362+0000 7f0c97e00640 1 -- 172.21.15.28:0/493400690 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55b0374116b0 msgr2=0x55b0373ecb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:48.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.362+0000 7f0c97e00640 1 --2- 172.21.15.28:0/493400690 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55b0374116b0 0x55b0373ecb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:48.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.362+0000 7f0c97e00640 1 -- 172.21.15.28:0/493400690 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b0373d6330 msgr2=0x55b037410db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:48.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.362+0000 7f0c97e00640 1 --2- 172.21.15.28:0/493400690 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b0373d6330 0x55b037410db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:48.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.362+0000 7f0c97e00640 1 -- 172.21.15.28:0/493400690 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b0371186b0 con 0x55b03740be90 2024-02-21T23:07:48.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.362+0000 7f0c97e00640 1 --2- 172.21.15.28:0/493400690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b03740be90 0x55b0373f00a0 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f0c880563b0 tx=0x7f0c8805c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=380a046fd32b5528 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:48.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.362+0000 7f0c8fe00640 1 -- 172.21.15.28:0/493400690 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c88086070 con 0x55b03740be90 2024-02-21T23:07:48.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.362+0000 7f0c8fe00640 1 -- 172.21.15.28:0/493400690 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c8805c1a0 con 0x55b03740be90 2024-02-21T23:07:48.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.362+0000 7f0c8fe00640 1 -- 172.21.15.28:0/493400690 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c8805e370 con 0x55b03740be90 2024-02-21T23:07:48.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.363+0000 7f0caa024740 1 -- 172.21.15.28:0/493400690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b03740be90 msgr2=0x55b0373f00a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:48.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.363+0000 7f0caa024740 1 --2- 172.21.15.28:0/493400690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b03740be90 0x55b0373f00a0 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f0c880563b0 tx=0x7f0c8805c000 comp rx=0 tx=0).stop 2024-02-21T23:07:48.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.363+0000 7f0caa024740 1 -- 172.21.15.28:0/493400690 shutdown_connections 2024-02-21T23:07:48.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.363+0000 7f0caa024740 1 --2- 172.21.15.28:0/493400690 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55b0374116b0 0x55b0373ecb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:48.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.363+0000 7f0caa024740 1 --2- 172.21.15.28:0/493400690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b03740be90 0x55b0373f00a0 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:48.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.363+0000 7f0caa024740 1 --2- 172.21.15.28:0/493400690 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b0373d6330 0x55b037410db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:48.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.363+0000 7f0caa024740 1 -- 172.21.15.28:0/493400690 >> 172.21.15.28:0/493400690 conn(0x55b0373d5020 msgr2=0x55b0373e6ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:48.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.363+0000 7f0caa024740 1 -- 172.21.15.28:0/493400690 shutdown_connections 2024-02-21T23:07:48.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.363+0000 7f0caa024740 1 -- 172.21.15.28:0/493400690 wait complete. 2024-02-21T23:07:48.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.364+0000 7f0caa024740 1 -- start start 2024-02-21T23:07:48.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.364+0000 7f0caa024740 1 Processor -- start 2024-02-21T23:07:48.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.365+0000 7f0caa024740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0373d6330 0x55b0373e99a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:48.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.365+0000 7f0caa024740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b0374116b0 0x55b0373ecb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:48.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.365+0000 7f0caa024740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55b037486400 0x55b037486880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:48.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.365+0000 7f0caa024740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0373f5850 con 0x55b0373d6330 2024-02-21T23:07:48.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.365+0000 7f0caa024740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0373f0c70 con 0x55b0374116b0 2024-02-21T23:07:48.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.365+0000 7f0caa024740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b0373f54b0 con 0x55b037486400 2024-02-21T23:07:48.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.365+0000 7f0caa024740 10 client.0 _ll_register_callbacks cb 0x7f0c88057ed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:07:48.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.365+0000 7f0c9fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0373d6330 0x55b0373e99a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:48.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.365+0000 7f0c9fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0373d6330 0x55b0373e99a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59952/0 (socket says 172.21.15.28:59952) 2024-02-21T23:07:48.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.365+0000 7f0c9fe00640 1 -- 172.21.15.28:0/3303374903 learned_addr learned my addr 172.21.15.28:0/3303374903 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:48.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.365+0000 7f0c97e00640 1 --2- 172.21.15.28:0/3303374903 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b0374116b0 0x55b0373ecb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:48.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.365+0000 7f0ca5000640 1 --2- 172.21.15.28:0/3303374903 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55b037486400 0x55b037486880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:48.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.365+0000 7f0c9fe00640 1 -- 172.21.15.28:0/3303374903 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55b037486400 msgr2=0x55b037486880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:48.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.365+0000 7f0c9fe00640 1 --2- 172.21.15.28:0/3303374903 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55b037486400 0x55b037486880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:48.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.365+0000 7f0c9fe00640 1 -- 172.21.15.28:0/3303374903 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b0374116b0 msgr2=0x55b0373ecb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:48.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.365+0000 7f0c9fe00640 1 --2- 172.21.15.28:0/3303374903 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b0374116b0 0x55b0373ecb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:48.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.365+0000 7f0c9fe00640 1 -- 172.21.15.28:0/3303374903 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b0373ed390 con 0x55b0373d6330 2024-02-21T23:07:48.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.366+0000 7f0c9fe00640 1 --2- 172.21.15.28:0/3303374903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0373d6330 0x55b0373e99a0 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f0c90007ba0 tx=0x7f0c90001190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6549711b5c1d32f1 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:48.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.366+0000 7f0c8d600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:07:48.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.366+0000 7f0c8d600640 1 -- 172.21.15.28:0/3303374903 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c90058020 con 0x55b0373d6330 2024-02-21T23:07:48.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.366+0000 7f0c8d600640 1 -- 172.21.15.28:0/3303374903 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(111..111 src has 1..111) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f0c90006970 con 0x55b0373d6330 2024-02-21T23:07:48.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.366+0000 7f0caa024740 1 -- 172.21.15.28:0/3303374903 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0c8805c1a0 con 0x55b0373d6330 2024-02-21T23:07:48.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.366+0000 7f0c8d600640 1 -- 172.21.15.28:0/3303374903 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=109) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0c9004c040 con 0x55b0373d6330 2024-02-21T23:07:48.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.366+0000 7f0caa024740 10 client.7701 fetch_fsmap learned FSMap version 109 2024-02-21T23:07:48.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.366+0000 7f0caa024740 1 -- 172.21.15.28:0/3303374903 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=109,monmap=2+}) v3 -- 0x55b0373d4b40 con 0x55b0373d6330 2024-02-21T23:07:48.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.367+0000 7f0c8d600640 1 -- 172.21.15.28:0/3303374903 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c8805c1a0 con 0x55b0373d6330 2024-02-21T23:07:48.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.367+0000 7f0c8d600640 1 -- 172.21.15.28:0/3303374903 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 109) v1 ==== 3282+0+0 (secure 0 0 0) 0x7f0c90052040 con 0x55b0373d6330 2024-02-21T23:07:48.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.367+0000 7f0caa024740 10 client.7701 fetch_fsmap finished waiting for FSMap version 109 2024-02-21T23:07:48.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.367+0000 7f0caa024740 10 client.7701 resolve_mds: resolved name 'd' to daemon mds.d 2024-02-21T23:07:48.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.367+0000 7f0caa024740 20 client.7701 populate_metadata read hostname 'smithi028' 2024-02-21T23:07:48.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.367+0000 7f0caa024740 1 --2- 172.21.15.28:0/3303374903 >> [v2:172.21.15.81:6834/1754860807,v1:172.21.15.81:6835/1754860807] conn(0x55b03748c660 0x55b03748ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:48.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.367+0000 7f0caa024740 4 client.7701 mds_command: new command op to 5729 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-21T23:07:48.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.367+0000 7f0caa024740 1 -- 172.21.15.28:0/3303374903 --> [v2:172.21.15.81:6834/1754860807,v1:172.21.15.81:6835/1754860807] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55b0372bf830 con 0x55b03748c660 2024-02-21T23:07:48.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.367+0000 7f0c97e00640 1 --2- 172.21.15.28:0/3303374903 >> [v2:172.21.15.81:6834/1754860807,v1:172.21.15.81:6835/1754860807] conn(0x55b03748c660 0x55b03748ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:48.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.368+0000 7f0c97e00640 1 --2- 172.21.15.28:0/3303374903 >> [v2:172.21.15.81:6834/1754860807,v1:172.21.15.81:6835/1754860807] conn(0x55b03748c660 0x55b03748ca40 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=72989d19388333aa server_cookie=724d5ceaa0ea98c6 in_seq=0 out_seq=0 2024-02-21T23:07:48.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.368+0000 7f0c8d600640 10 client.7701 ms_handle_connect on v2:172.21.15.81:6834/1754860807 2024-02-21T23:07:48.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.369+0000 7f0c8d600640 1 -- 172.21.15.28:0/3303374903 <== mds.0 v2:172.21.15.81:6834/1754860807 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+334 (crc 0 0 0) 0x55b0372bf830 con 0x55b03748c660 2024-02-21T23:07:48.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.369+0000 7f0c8d600640 10 client.7701 handle_command_reply: tid=0 2024-02-21T23:07:48.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.369+0000 7f0caa024740 1 client.7701 shutdown 2024-02-21T23:07:48.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.369+0000 7f0caa024740 1 -- 172.21.15.28:0/3303374903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0373d6330 msgr2=0x55b0373e99a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:48.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.369+0000 7f0caa024740 1 --2- 172.21.15.28:0/3303374903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0373d6330 0x55b0373e99a0 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f0c90007ba0 tx=0x7f0c90001190 comp rx=0 tx=0).stop 2024-02-21T23:07:48.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.369+0000 7f0caa024740 1 -- 172.21.15.28:0/3303374903 shutdown_connections 2024-02-21T23:07:48.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.369+0000 7f0caa024740 1 --2- 172.21.15.28:0/3303374903 >> [v2:172.21.15.81:6834/1754860807,v1:172.21.15.81:6835/1754860807] conn(0x55b03748c660 0x55b03748ca40 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:48.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.369+0000 7f0caa024740 1 --2- 172.21.15.28:0/3303374903 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55b037486400 0x55b037486880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:48.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.369+0000 7f0caa024740 1 --2- 172.21.15.28:0/3303374903 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b0374116b0 0x55b0373ecb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:48.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.369+0000 7f0caa024740 1 --2- 172.21.15.28:0/3303374903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0373d6330 0x55b0373e99a0 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:48.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.369+0000 7f0caa024740 1 -- 172.21.15.28:0/3303374903 >> 172.21.15.28:0/3303374903 conn(0x55b0373d5020 msgr2=0x55b0373f4a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:48.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.369+0000 7f0c8d600640 0 client.7701 ms_handle_reset on v2:172.21.15.81:6834/1754860807 2024-02-21T23:07:48.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.369+0000 7f0caa024740 1 -- 172.21.15.28:0/3303374903 shutdown_connections 2024-02-21T23:07:48.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.369+0000 7f0caa024740 1 -- 172.21.15.28:0/3303374903 wait complete. 2024-02-21T23:07:48.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.369+0000 7f0caa024740 20 client.7701 trim_cache size 0 max 16384 2024-02-21T23:07:48.361 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-21T23:07:48.361 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac", 2024-02-21T23:07:48.361 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-02-21T23:07:48.361 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 5729, 2024-02-21T23:07:48.361 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-02-21T23:07:48.361 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-02-21T23:07:48.361 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-02-21T23:07:48.362 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 7.2235167249999996, 2024-02-21T23:07:48.362 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 109, 2024-02-21T23:07:48.362 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 111, 2024-02-21T23:07:48.362 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 111, 2024-02-21T23:07:48.362 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 23.776583160000001 2024-02-21T23:07:48.362 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-21T23:07:48.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.370+0000 7f0ca5a00640 1 -- 172.21.15.28:0/3080561421 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0c7c06b500 msgr2=0x7f0c7c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:48.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.370+0000 7f0ca5a00640 1 --2- 172.21.15.28:0/3080561421 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0c7c06b500 0x7f0c7c06d9c0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f0c90033000 tx=0x7f0c90009030 comp rx=0 tx=0).stop 2024-02-21T23:07:48.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.370+0000 7f0ca5a00640 1 -- 172.21.15.28:0/3080561421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ca00c13c0 msgr2=0x7f0ca01ee8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:48.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.370+0000 7f0ca5a00640 1 --2- 172.21.15.28:0/3080561421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ca00c13c0 0x7f0ca01ee8b0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f0c880011e0 tx=0x7f0c8800db40 comp rx=0 tx=0).stop 2024-02-21T23:07:48.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.370+0000 7f0ca5a00640 1 -- 172.21.15.28:0/3080561421 shutdown_connections 2024-02-21T23:07:48.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.370+0000 7f0ca5a00640 1 --2- 172.21.15.28:0/3080561421 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0c7c06b500 0x7f0c7c06d9c0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:48.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.370+0000 7f0ca5a00640 1 --2- 172.21.15.28:0/3080561421 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0ca00bf610 0x7f0ca00bfa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:48.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.370+0000 7f0ca5a00640 1 --2- 172.21.15.28:0/3080561421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ca00c13c0 0x7f0ca01ee8b0 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:48.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.370+0000 7f0ca5a00640 1 --2- 172.21.15.28:0/3080561421 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0c840039b0 0x7f0ca01eab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:48.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.370+0000 7f0ca5a00640 1 -- 172.21.15.28:0/3080561421 >> 172.21.15.28:0/3080561421 conn(0x7f0ca00ac370 msgr2=0x7f0ca01542c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:48.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.370+0000 7f0ca5a00640 1 -- 172.21.15.28:0/3080561421 shutdown_connections 2024-02-21T23:07:48.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:48.370+0000 7f0ca5a00640 1 -- 172.21.15.28:0/3080561421 wait complete. 2024-02-21T23:07:48.376 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 8, "epoch": 109, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 8, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-02-21T23:07:29.533547+0000", "damaged": [], "data_pools": [ 18 ], "enabled": true, "epoch": 109, "ever_allowed_features": 32, "explicitly_allowed_features": 32, "failed": [], "flags": 50, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": true, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5729": { "addr": "172.21.15.81:6835/1754860807", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6834", "nonce": 1754860807, "type": "v2" }, { "addr": "172.21.15.81:6835", "nonce": 1754860807, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5729, "incarnation": 104, "join_fscid": 8, "name": "d", "rank": 0, "state": "up:active", "state_seq": 8 }, "gid_7632": { "addr": "172.21.15.28:6835/780497578", "addrs": { "addrvec": [ { "addr": "172.21.15.28:6834", "nonce": 780497578, "type": "v2" }, { "addr": "172.21.15.28:6835", "nonce": 780497578, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7632, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:standby-replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 111, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 17, "modified": "2024-02-21T23:07:46.185982+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5729 } } } ], "standbys": [ { "addr": "172.21.15.81:6837/3963422563", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6836", "nonce": 3963422563, "type": "v2" }, { "addr": "172.21.15.81:6837", "nonce": 3963422563, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 97, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5732, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.28:6833/1119726035", "addrs": { "addrvec": [ { "addr": "172.21.15.28:6832", "nonce": 1119726035, "type": "v2" }, { "addr": "172.21.15.28:6833", "nonce": 1119726035, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 96, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7446, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-02-21T23:07:48.377 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': 8, 'name': 'b', 'state': 'up:standby-replay'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': 8, 'name': 'd', 'state': 'up:active'}] 2024-02-21T23:07:48.377 INFO:tasks.cephfs.test_failover:current = [{'gid': 7632, 'name': 'a', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.28:6835/780497578', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 780497578}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 780497578}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 5732, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.81:6837/3963422563', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3963422563}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3963422563}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 97}, {'gid': 7446, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.28:6833/1119726035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 1119726035}, {'type': 'v1', 'addr': '172.21.15.28:6833', 'nonce': 1119726035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 96}, {'gid': 5729, 'name': 'd', 'rank': 0, 'incarnation': 104, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.81:6835/1754860807', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6834', 'nonce': 1754860807}, {'type': 'v1', 'addr': '172.21.15.81:6835', 'nonce': 1754860807}]}, 'join_fscid': 8, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}] 2024-02-21T23:07:48.377 DEBUG:tasks.cephfs.test_failover:8 != -1 2024-02-21T23:07:48.377 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-02-21T23:07:49.277 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-02-21T23:07:53.378 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:07:53.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.622+0000 7f1b75800640 1 Processor -- start 2024-02-21T23:07:53.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.623+0000 7f1b75800640 1 -- start start 2024-02-21T23:07:53.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.624+0000 7f1b75800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1b700c7340 0x7f1b700c4370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:53.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.624+0000 7f1b75800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1b7006ae70 0x7f1b7006b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:53.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.624+0000 7f1b75800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b700c38d0 con 0x7f1b70072a50 2024-02-21T23:07:53.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.624+0000 7f1b75800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b70153f00 con 0x7f1b7006ae70 2024-02-21T23:07:53.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.624+0000 7f1b75800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b700c5850 con 0x7f1b700c7340 2024-02-21T23:07:53.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.624+0000 7f1b6f400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1b7006ae70 0x7f1b7006b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:53.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.624+0000 7f1b6f400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1b7006ae70 0x7f1b7006b2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59994/0 (socket says 172.21.15.28:59994) 2024-02-21T23:07:53.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.624+0000 7f1b6f400640 1 -- 172.21.15.28:0/82646476 learned_addr learned my addr 172.21.15.28:0/82646476 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:53.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.624+0000 7f1b6ea00640 1 --2- 172.21.15.28:0/82646476 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1b700c7340 0x7f1b700c4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:53.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.625+0000 7f1b6f400640 1 -- 172.21.15.28:0/82646476 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1b700c7340 msgr2=0x7f1b700c4370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:53.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.625+0000 7f1b6f400640 1 --2- 172.21.15.28:0/82646476 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1b700c7340 0x7f1b700c4370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:53.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.625+0000 7f1b6f400640 1 -- 172.21.15.28:0/82646476 >> v1:172.21.15.81:6789/0 conn(0x7f1b70072a50 legacy=0x7f1b700c48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:53.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.625+0000 7f1b6f400640 1 -- 172.21.15.28:0/82646476 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b70153c20 con 0x7f1b7006ae70 2024-02-21T23:07:53.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.625+0000 7f1b6ea00640 1 --2- 172.21.15.28:0/82646476 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1b700c7340 0x7f1b700c4370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:07:53.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.626+0000 7f1b6f400640 1 --2- 172.21.15.28:0/82646476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b7006ae70 0x7f1b7006b2c0 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f1b68009e50 tx=0x7f1b6802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f94c846237e07de server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:53.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.626+0000 7f1b6e000640 1 -- 172.21.15.28:0/82646476 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b6802cb00 con 0x7f1b7006ae70 2024-02-21T23:07:53.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.626+0000 7f1b6e000640 1 -- 172.21.15.28:0/82646476 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b6802cca0 con 0x7f1b7006ae70 2024-02-21T23:07:53.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.626+0000 7f1b6e000640 1 -- 172.21.15.28:0/82646476 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b68034610 con 0x7f1b7006ae70 2024-02-21T23:07:53.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.627+0000 7f1b75800640 1 -- 172.21.15.28:0/82646476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b7006ae70 msgr2=0x7f1b7006b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:53.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.627+0000 7f1b75800640 1 --2- 172.21.15.28:0/82646476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b7006ae70 0x7f1b7006b2c0 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f1b68009e50 tx=0x7f1b6802bf60 comp rx=0 tx=0).stop 2024-02-21T23:07:53.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.627+0000 7f1b75800640 1 -- 172.21.15.28:0/82646476 shutdown_connections 2024-02-21T23:07:53.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.627+0000 7f1b75800640 1 --2- 172.21.15.28:0/82646476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b7006ae70 0x7f1b7006b2c0 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:53.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.627+0000 7f1b75800640 1 --2- 172.21.15.28:0/82646476 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1b700c7340 0x7f1b700c4370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:53.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.627+0000 7f1b75800640 1 -- 172.21.15.28:0/82646476 >> 172.21.15.28:0/82646476 conn(0x7f1b700c7950 msgr2=0x7f1b700c7d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:53.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.627+0000 7f1b75800640 1 -- 172.21.15.28:0/82646476 shutdown_connections 2024-02-21T23:07:53.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.627+0000 7f1b75800640 1 -- 172.21.15.28:0/82646476 wait complete. 2024-02-21T23:07:53.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.629+0000 7f1b75800640 1 Processor -- start 2024-02-21T23:07:53.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.629+0000 7f1b75800640 1 -- start start 2024-02-21T23:07:53.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.630+0000 7f1b75800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b7006ae70 0x7f1b701ee4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:53.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.630+0000 7f1b75800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1b700c7340 0x7f1b701478e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:53.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.630+0000 7f1b75800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1b701eea00 0x7f1b70147e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:53.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.630+0000 7f1b75800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b700c3e50 con 0x7f1b7006ae70 2024-02-21T23:07:53.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.630+0000 7f1b75800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b700c2fa0 con 0x7f1b700c7340 2024-02-21T23:07:53.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.630+0000 7f1b75800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b70153900 con 0x7f1b701eea00 2024-02-21T23:07:53.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.630+0000 7f1b6ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b7006ae70 0x7f1b701ee4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:53.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.630+0000 7f1b6ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b7006ae70 0x7f1b701ee4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60010/0 (socket says 172.21.15.28:60010) 2024-02-21T23:07:53.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.630+0000 7f1b6ea00640 1 -- 172.21.15.28:0/219192452 learned_addr learned my addr 172.21.15.28:0/219192452 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:53.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.630+0000 7f1b6ea00640 1 -- 172.21.15.28:0/219192452 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1b701eea00 msgr2=0x7f1b70147e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:53.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.630+0000 7f1b6ea00640 1 --2- 172.21.15.28:0/219192452 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1b701eea00 0x7f1b70147e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:53.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.630+0000 7f1b6ea00640 1 -- 172.21.15.28:0/219192452 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1b700c7340 msgr2=0x7f1b701478e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:53.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.630+0000 7f1b6ea00640 1 --2- 172.21.15.28:0/219192452 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1b700c7340 0x7f1b701478e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:53.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.630+0000 7f1b6ea00640 1 -- 172.21.15.28:0/219192452 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b7006f750 con 0x7f1b7006ae70 2024-02-21T23:07:53.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.631+0000 7f1b6ea00640 1 --2- 172.21.15.28:0/219192452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b7006ae70 0x7f1b701ee4c0 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f1b6000d7d0 tx=0x7f1b6000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:53.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.631+0000 7f1b67400640 1 -- 172.21.15.28:0/219192452 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b60004030 con 0x7f1b7006ae70 2024-02-21T23:07:53.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.631+0000 7f1b67400640 1 -- 172.21.15.28:0/219192452 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b600041d0 con 0x7f1b7006ae70 2024-02-21T23:07:53.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.631+0000 7f1b67400640 1 -- 172.21.15.28:0/219192452 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b600096a0 con 0x7f1b7006ae70 2024-02-21T23:07:53.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.631+0000 7f1b75800640 1 -- 172.21.15.28:0/219192452 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b701eeeb0 con 0x7f1b7006ae70 2024-02-21T23:07:53.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.631+0000 7f1b75800640 1 -- 172.21.15.28:0/219192452 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b70151ac0 con 0x7f1b7006ae70 2024-02-21T23:07:53.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.632+0000 7f1b75800640 1 -- 172.21.15.28:0/219192452 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b70067f30 con 0x7f1b7006ae70 2024-02-21T23:07:53.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.633+0000 7f1b67400640 1 -- 172.21.15.28:0/219192452 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1b60009840 con 0x7f1b7006ae70 2024-02-21T23:07:53.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.634+0000 7f1b67400640 1 --2- 172.21.15.28:0/219192452 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1b4c06b4d0 0x7f1b4c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:53.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.634+0000 7f1b67400640 1 -- 172.21.15.28:0/219192452 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f1b6008a760 con 0x7f1b7006ae70 2024-02-21T23:07:53.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.635+0000 7f1b66000640 1 --2- 172.21.15.28:0/219192452 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1b4c06b4d0 0x7f1b4c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:53.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.636+0000 7f1b66000640 1 --2- 172.21.15.28:0/219192452 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1b4c06b4d0 0x7f1b4c06d990 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f1b700aa750 tx=0x7f1b580079b0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:53.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.639+0000 7f1b67400640 1 -- 172.21.15.28:0/219192452 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1b60059360 con 0x7f1b7006ae70 2024-02-21T23:07:53.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.877+0000 7f1b75800640 1 -- 172.21.15.28:0/219192452 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1b70073170 con 0x7f1b7006ae70 2024-02-21T23:07:53.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.879+0000 7f1b67400640 1 -- 172.21.15.28:0/219192452 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+4949 (secure 0 0 0) 0x7f1b60058c60 con 0x7f1b7006ae70 2024-02-21T23:07:53.868 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:07:53.868 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":112,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5843,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3059988835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3059988835},{"type":"v1","addr":"172.21.15.28:6835","nonce":3059988835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":112},{"gid":7446,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/1119726035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1119726035},{"type":"v1","addr":"172.21.15.28:6833","nonce":1119726035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":111,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-02-21T23:07:29.533547+0000","modified":"2024-02-21T23:07:49.218076+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5729},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5729":{"gid":5729,"name":"d","rank":0,"incarnation":104,"state":"up:active","state_seq":8,"addr":"172.21.15.81:6835/1754860807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1754860807},{"type":"v1","addr":"172.21.15.81:6835","nonce":1754860807}]},"join_fscid":8,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5732":{"gid":5732,"name":"b","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":7,"addr":"172.21.15.81:6837/3963422563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3963422563},{"type":"v1","addr":"172.21.15.81:6837","nonce":3963422563}]},"join_fscid":8,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-02-21T23:07:53.868 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 112 2024-02-21T23:07:53.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.881+0000 7f1b75800640 1 -- 172.21.15.28:0/219192452 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1b4c06b4d0 msgr2=0x7f1b4c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:53.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.881+0000 7f1b75800640 1 --2- 172.21.15.28:0/219192452 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1b4c06b4d0 0x7f1b4c06d990 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f1b700aa750 tx=0x7f1b580079b0 comp rx=0 tx=0).stop 2024-02-21T23:07:53.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.881+0000 7f1b75800640 1 -- 172.21.15.28:0/219192452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b7006ae70 msgr2=0x7f1b701ee4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:53.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.881+0000 7f1b75800640 1 --2- 172.21.15.28:0/219192452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b7006ae70 0x7f1b701ee4c0 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f1b6000d7d0 tx=0x7f1b6000dc90 comp rx=0 tx=0).stop 2024-02-21T23:07:53.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.881+0000 7f1b75800640 1 -- 172.21.15.28:0/219192452 shutdown_connections 2024-02-21T23:07:53.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.881+0000 7f1b75800640 1 --2- 172.21.15.28:0/219192452 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1b4c06b4d0 0x7f1b4c06d990 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:53.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.881+0000 7f1b75800640 1 --2- 172.21.15.28:0/219192452 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1b701eea00 0x7f1b70147e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:53.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.881+0000 7f1b75800640 1 --2- 172.21.15.28:0/219192452 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1b700c7340 0x7f1b701478e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:53.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.881+0000 7f1b75800640 1 --2- 172.21.15.28:0/219192452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b7006ae70 0x7f1b701ee4c0 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:53.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.881+0000 7f1b75800640 1 -- 172.21.15.28:0/219192452 >> 172.21.15.28:0/219192452 conn(0x7f1b700c7950 msgr2=0x7f1b700aa8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:53.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.881+0000 7f1b75800640 1 -- 172.21.15.28:0/219192452 shutdown_connections 2024-02-21T23:07:53.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:53.882+0000 7f1b75800640 1 -- 172.21.15.28:0/219192452 wait complete. 2024-02-21T23:07:54.294 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 111, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-02-21T23:07:29.533547+0000', 'modified': '2024-02-21T23:07:49.218076+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5729}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5729': {'gid': 5729, 'name': 'd', 'rank': 0, 'incarnation': 104, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.81:6835/1754860807', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6834', 'nonce': 1754860807}, {'type': 'v1', 'addr': '172.21.15.81:6835', 'nonce': 1754860807}]}, 'join_fscid': 8, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_5732': {'gid': 5732, 'name': 'b', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 7, 'addr': '172.21.15.81:6837/3963422563', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3963422563}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3963422563}]}, 'join_fscid': 8, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:07:54.294 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-21T23:07:54.295 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-02-21T23:07:54.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.532+0000 7fcf46000640 1 Processor -- start 2024-02-21T23:07:54.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.532+0000 7fcf46000640 1 -- start start 2024-02-21T23:07:54.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.534+0000 7fcf46000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcf401527a0 0x7fcf40158f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:54.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.534+0000 7fcf46000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcf4006d410 0x7fcf4006d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:54.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.534+0000 7fcf46000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf400c2070 con 0x7fcf400bf9a0 2024-02-21T23:07:54.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.534+0000 7fcf46000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf4006e790 con 0x7fcf401527a0 2024-02-21T23:07:54.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.534+0000 7fcf46000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf400c29a0 con 0x7fcf4006d410 2024-02-21T23:07:54.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.534+0000 7fcf44c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcf401527a0 0x7fcf40158f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:54.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.534+0000 7fcf44c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcf401527a0 0x7fcf40158f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60014/0 (socket says 172.21.15.28:60014) 2024-02-21T23:07:54.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.534+0000 7fcf44c00640 1 -- 172.21.15.28:0/4163106378 learned_addr learned my addr 172.21.15.28:0/4163106378 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:54.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.534+0000 7fcf45600640 1 --2- 172.21.15.28:0/4163106378 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcf4006d410 0x7fcf4006d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:54.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.535+0000 7fcf44c00640 1 -- 172.21.15.28:0/4163106378 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcf4006d410 msgr2=0x7fcf4006d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:54.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.535+0000 7fcf44c00640 1 --2- 172.21.15.28:0/4163106378 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcf4006d410 0x7fcf4006d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:54.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.535+0000 7fcf44c00640 1 -- 172.21.15.28:0/4163106378 >> v1:172.21.15.81:6789/0 conn(0x7fcf400bf9a0 legacy=0x7fcf400bfdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:54.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.535+0000 7fcf44c00640 1 -- 172.21.15.28:0/4163106378 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf400c2350 con 0x7fcf401527a0 2024-02-21T23:07:54.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.536+0000 7fcf44c00640 1 --2- 172.21.15.28:0/4163106378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf401527a0 0x7fcf40158f70 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7fcf34009870 tx=0x7fcf3402ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6fe6077b0e865a6 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:54.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.537+0000 7fcf3f400640 1 -- 172.21.15.28:0/4163106378 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf3402f840 con 0x7fcf401527a0 2024-02-21T23:07:54.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.537+0000 7fcf3f400640 1 -- 172.21.15.28:0/4163106378 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf3402f9e0 con 0x7fcf401527a0 2024-02-21T23:07:54.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.537+0000 7fcf3f400640 1 -- 172.21.15.28:0/4163106378 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf3402fdb0 con 0x7fcf401527a0 2024-02-21T23:07:54.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.537+0000 7fcf46000640 1 -- 172.21.15.28:0/4163106378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf401527a0 msgr2=0x7fcf40158f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:54.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.537+0000 7fcf46000640 1 --2- 172.21.15.28:0/4163106378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf401527a0 0x7fcf40158f70 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7fcf34009870 tx=0x7fcf3402ba20 comp rx=0 tx=0).stop 2024-02-21T23:07:54.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.538+0000 7fcf46000640 1 -- 172.21.15.28:0/4163106378 shutdown_connections 2024-02-21T23:07:54.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.538+0000 7fcf46000640 1 --2- 172.21.15.28:0/4163106378 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcf4006d410 0x7fcf4006d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:54.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.538+0000 7fcf46000640 1 --2- 172.21.15.28:0/4163106378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf401527a0 0x7fcf40158f70 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:54.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.538+0000 7fcf46000640 1 -- 172.21.15.28:0/4163106378 >> 172.21.15.28:0/4163106378 conn(0x7fcf400a8320 msgr2=0x7fcf400a8730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:54.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.538+0000 7fcf46000640 1 -- 172.21.15.28:0/4163106378 shutdown_connections 2024-02-21T23:07:54.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.538+0000 7fcf46000640 1 -- 172.21.15.28:0/4163106378 wait complete. 2024-02-21T23:07:54.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.539+0000 7fcf46000640 1 Processor -- start 2024-02-21T23:07:54.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.540+0000 7fcf46000640 1 -- start start 2024-02-21T23:07:54.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.540+0000 7fcf46000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf4006d410 0x7fcf400c4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:54.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.540+0000 7fcf46000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf400bf9a0 0x7fcf401eafb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:54.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.540+0000 7fcf46000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcf401527a0 0x7fcf401eb4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:54.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.540+0000 7fcf46000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf400c1d20 con 0x7fcf400bf9a0 2024-02-21T23:07:54.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.540+0000 7fcf46000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf400c0500 con 0x7fcf4006d410 2024-02-21T23:07:54.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.540+0000 7fcf46000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf4006dda0 con 0x7fcf401527a0 2024-02-21T23:07:54.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.541+0000 7fcf3fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf400bf9a0 0x7fcf401eafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:54.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.541+0000 7fcf3fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf400bf9a0 0x7fcf401eafb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60026/0 (socket says 172.21.15.28:60026) 2024-02-21T23:07:54.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.541+0000 7fcf3fe00640 1 -- 172.21.15.28:0/3847540227 learned_addr learned my addr 172.21.15.28:0/3847540227 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:54.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.541+0000 7fcf44c00640 1 --2- 172.21.15.28:0/3847540227 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf4006d410 0x7fcf400c4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:54.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.541+0000 7fcf45600640 1 --2- 172.21.15.28:0/3847540227 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcf401527a0 0x7fcf401eb4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:54.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.541+0000 7fcf3fe00640 1 -- 172.21.15.28:0/3847540227 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcf401527a0 msgr2=0x7fcf401eb4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:54.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.541+0000 7fcf3fe00640 1 --2- 172.21.15.28:0/3847540227 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcf401527a0 0x7fcf401eb4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:54.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.541+0000 7fcf3fe00640 1 -- 172.21.15.28:0/3847540227 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf4006d410 msgr2=0x7fcf400c4d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:54.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.541+0000 7fcf3fe00640 1 --2- 172.21.15.28:0/3847540227 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf4006d410 0x7fcf400c4d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:54.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.541+0000 7fcf3fe00640 1 -- 172.21.15.28:0/3847540227 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf40153610 con 0x7fcf400bf9a0 2024-02-21T23:07:54.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.541+0000 7fcf44c00640 1 --2- 172.21.15.28:0/3847540227 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf4006d410 0x7fcf400c4d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:07:54.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.541+0000 7fcf45600640 1 --2- 172.21.15.28:0/3847540227 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcf401527a0 0x7fcf401eb4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:07:54.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.542+0000 7fcf3fe00640 1 --2- 172.21.15.28:0/3847540227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf400bf9a0 0x7fcf401eafb0 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c00c400 tx=0x7fcf2c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:54.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.542+0000 7fcf3cc00640 1 -- 172.21.15.28:0/3847540227 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf2c0052d0 con 0x7fcf400bf9a0 2024-02-21T23:07:54.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.542+0000 7fcf3cc00640 1 -- 172.21.15.28:0/3847540227 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf2c005490 con 0x7fcf400bf9a0 2024-02-21T23:07:54.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.542+0000 7fcf3cc00640 1 -- 172.21.15.28:0/3847540227 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf2c007260 con 0x7fcf400bf9a0 2024-02-21T23:07:54.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.542+0000 7fcf46000640 1 -- 172.21.15.28:0/3847540227 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf401eaaf0 con 0x7fcf400bf9a0 2024-02-21T23:07:54.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.542+0000 7fcf46000640 1 -- 172.21.15.28:0/3847540227 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf401eedc0 con 0x7fcf400bf9a0 2024-02-21T23:07:54.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.544+0000 7fcf3cc00640 1 -- 172.21.15.28:0/3847540227 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcf2c007400 con 0x7fcf400bf9a0 2024-02-21T23:07:54.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.545+0000 7fcf3cc00640 1 --2- 172.21.15.28:0/3847540227 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcf2006b520 0x7fcf2006d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:54.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.545+0000 7fcf3cc00640 1 -- 172.21.15.28:0/3847540227 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fcf2c0898f0 con 0x7fcf400bf9a0 2024-02-21T23:07:54.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.546+0000 7fcf44c00640 1 --2- 172.21.15.28:0/3847540227 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcf2006b520 0x7fcf2006d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:54.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.546+0000 7fcf4a593740 1 Processor -- start 2024-02-21T23:07:54.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.546+0000 7fcf4a593740 1 -- start start 2024-02-21T23:07:54.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.547+0000 7fcf4a593740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55add7b04020 0x55add7b04400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:54.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.547+0000 7fcf4a593740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55add7b05330 0x55add7b40170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:54.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.547+0000 7fcf4a593740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55add7b406b0 0x55add7b3e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:54.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.547+0000 7fcf4a593740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55add7a68470 con 0x55add7b406b0 2024-02-21T23:07:54.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.547+0000 7fcf4a593740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55add7a11af0 con 0x55add7b04020 2024-02-21T23:07:54.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.547+0000 7fcf4a593740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x55add7a64510 con 0x55add7b05330 2024-02-21T23:07:54.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.547+0000 7fcf44c00640 1 --2- 172.21.15.28:0/3847540227 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcf2006b520 0x7fcf2006d9e0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fcf34036000 tx=0x7fcf34008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:54.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.547+0000 7fcf45600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55add7b406b0 0x55add7b3e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:54.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.547+0000 7fcf45600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55add7b406b0 0x55add7b3e720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60038/0 (socket says 172.21.15.28:60038) 2024-02-21T23:07:54.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.547+0000 7fcf45600640 1 -- 172.21.15.28:0/58628646 learned_addr learned my addr 172.21.15.28:0/58628646 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:54.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.547+0000 7fcf44c00640 1 --2- 172.21.15.28:0/58628646 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55add7b04020 0x55add7b04400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:54.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.547+0000 7fcf3fe00640 1 --2- 172.21.15.28:0/58628646 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55add7b05330 0x55add7b40170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:54.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.547+0000 7fcf45600640 1 -- 172.21.15.28:0/58628646 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55add7b05330 msgr2=0x55add7b40170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:54.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.547+0000 7fcf45600640 1 --2- 172.21.15.28:0/58628646 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55add7b05330 0x55add7b40170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:54.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.547+0000 7fcf45600640 1 -- 172.21.15.28:0/58628646 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55add7b04020 msgr2=0x55add7b04400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:54.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.547+0000 7fcf45600640 1 --2- 172.21.15.28:0/58628646 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55add7b04020 0x55add7b04400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:54.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.547+0000 7fcf45600640 1 -- 172.21.15.28:0/58628646 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55add7a18d90 con 0x55add7b406b0 2024-02-21T23:07:54.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.548+0000 7fcf45600640 1 --2- 172.21.15.28:0/58628646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55add7b406b0 0x55add7b3e720 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7fcf380076e0 tx=0x7fcf38009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9de52ee76b4272ec server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:54.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.548+0000 7fcf31200640 1 -- 172.21.15.28:0/58628646 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf3801a020 con 0x55add7b406b0 2024-02-21T23:07:54.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.548+0000 7fcf31200640 1 -- 172.21.15.28:0/58628646 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf38007e70 con 0x55add7b406b0 2024-02-21T23:07:54.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.548+0000 7fcf31200640 1 -- 172.21.15.28:0/58628646 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf38011950 con 0x55add7b406b0 2024-02-21T23:07:54.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.549+0000 7fcf4a593740 1 -- 172.21.15.28:0/58628646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55add7b406b0 msgr2=0x55add7b3e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:54.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.549+0000 7fcf4a593740 1 --2- 172.21.15.28:0/58628646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55add7b406b0 0x55add7b3e720 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7fcf380076e0 tx=0x7fcf38009750 comp rx=0 tx=0).stop 2024-02-21T23:07:54.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.549+0000 7fcf4a593740 1 -- 172.21.15.28:0/58628646 shutdown_connections 2024-02-21T23:07:54.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.549+0000 7fcf4a593740 1 --2- 172.21.15.28:0/58628646 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55add7b05330 0x55add7b40170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:54.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.549+0000 7fcf4a593740 1 --2- 172.21.15.28:0/58628646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55add7b406b0 0x55add7b3e720 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:54.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.549+0000 7fcf4a593740 1 --2- 172.21.15.28:0/58628646 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55add7b04020 0x55add7b04400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:54.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.549+0000 7fcf4a593740 1 -- 172.21.15.28:0/58628646 >> 172.21.15.28:0/58628646 conn(0x55add7b00970 msgr2=0x55add7b25380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:54.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.549+0000 7fcf4a593740 1 -- 172.21.15.28:0/58628646 shutdown_connections 2024-02-21T23:07:54.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.549+0000 7fcf4a593740 1 -- 172.21.15.28:0/58628646 wait complete. 2024-02-21T23:07:54.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.550+0000 7fcf4a593740 1 -- start start 2024-02-21T23:07:54.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.550+0000 7fcf4a593740 1 Processor -- start 2024-02-21T23:07:54.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.551+0000 7fcf4a593740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55add7b05330 0x55add7b53b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:54.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.551+0000 7fcf4a593740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55add7b406b0 0x55add7b38ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:54.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.551+0000 7fcf4a593740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55add7b390e0 0x55add7b3d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:54.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.551+0000 7fcf4a593740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55add7a07040 con 0x55add7b390e0 2024-02-21T23:07:54.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.551+0000 7fcf4a593740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55add7b25a90 con 0x55add7b05330 2024-02-21T23:07:54.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.551+0000 7fcf4a593740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x55add7b232f0 con 0x55add7b406b0 2024-02-21T23:07:54.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.551+0000 7fcf4a593740 10 client.0 _ll_register_callbacks cb 0x55add7a24920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:07:54.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.551+0000 7fcf45600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55add7b390e0 0x55add7b3d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:54.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.551+0000 7fcf45600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55add7b390e0 0x55add7b3d120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60052/0 (socket says 172.21.15.28:60052) 2024-02-21T23:07:54.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.551+0000 7fcf45600640 1 -- 172.21.15.28:0/2566421690 learned_addr learned my addr 172.21.15.28:0/2566421690 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:54.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.551+0000 7fcf44c00640 1 --2- 172.21.15.28:0/2566421690 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55add7b05330 0x55add7b53b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:54.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.551+0000 7fcf3fe00640 1 --2- 172.21.15.28:0/2566421690 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55add7b406b0 0x55add7b38ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:54.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.551+0000 7fcf45600640 1 -- 172.21.15.28:0/2566421690 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55add7b406b0 msgr2=0x55add7b38ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:54.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.551+0000 7fcf45600640 1 --2- 172.21.15.28:0/2566421690 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55add7b406b0 0x55add7b38ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:54.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.551+0000 7fcf45600640 1 -- 172.21.15.28:0/2566421690 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55add7b05330 msgr2=0x55add7b53b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:54.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.551+0000 7fcf45600640 1 --2- 172.21.15.28:0/2566421690 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55add7b05330 0x55add7b53b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:54.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.552+0000 7fcf45600640 1 -- 172.21.15.28:0/2566421690 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55add7a26aa0 con 0x55add7b390e0 2024-02-21T23:07:54.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.552+0000 7fcf44c00640 1 --2- 172.21.15.28:0/2566421690 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55add7b05330 0x55add7b53b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:07:54.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.552+0000 7fcf45600640 1 --2- 172.21.15.28:0/2566421690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55add7b390e0 0x55add7b3d120 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7fcf38000f80 tx=0x7fcf38004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d725407640d76b5 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:54.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.552+0000 7fcf25600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:07:54.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.552+0000 7fcf25600640 1 -- 172.21.15.28:0/2566421690 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf3801a050 con 0x55add7b390e0 2024-02-21T23:07:54.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.552+0000 7fcf25600640 1 -- 172.21.15.28:0/2566421690 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(111..111 src has 1..111) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fcf38016880 con 0x55add7b390e0 2024-02-21T23:07:54.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.552+0000 7fcf4a593740 1 -- 172.21.15.28:0/2566421690 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcf38007e70 con 0x55add7b390e0 2024-02-21T23:07:54.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.552+0000 7fcf25600640 1 -- 172.21.15.28:0/2566421690 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=112) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcf38016aa0 con 0x55add7b390e0 2024-02-21T23:07:54.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.552+0000 7fcf4a593740 10 client.7725 fetch_fsmap learned FSMap version 112 2024-02-21T23:07:54.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.552+0000 7fcf4a593740 1 -- 172.21.15.28:0/2566421690 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=112,monmap=2+}) v3 -- 0x55add777d2d0 con 0x55add7b390e0 2024-02-21T23:07:54.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.553+0000 7fcf25600640 1 -- 172.21.15.28:0/2566421690 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf38007e70 con 0x55add7b390e0 2024-02-21T23:07:54.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.553+0000 7fcf25600640 1 -- 172.21.15.28:0/2566421690 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 112) v1 ==== 3282+0+0 (secure 0 0 0) 0x7fcf38034020 con 0x55add7b390e0 2024-02-21T23:07:54.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.553+0000 7fcf4a593740 10 client.7725 fetch_fsmap finished waiting for FSMap version 112 2024-02-21T23:07:54.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.553+0000 7fcf4a593740 10 client.7725 resolve_mds: resolved name 'd' to daemon mds.d 2024-02-21T23:07:54.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.553+0000 7fcf4a593740 20 client.7725 populate_metadata read hostname 'smithi028' 2024-02-21T23:07:54.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.553+0000 7fcf4a593740 1 --2- 172.21.15.28:0/2566421690 >> [v2:172.21.15.81:6834/1754860807,v1:172.21.15.81:6835/1754860807] conn(0x55add7b252b0 0x55add7b503b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:54.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.553+0000 7fcf4a593740 4 client.7725 mds_command: new command op to 5729 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-21T23:07:54.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.553+0000 7fcf4a593740 1 -- 172.21.15.28:0/2566421690 --> [v2:172.21.15.81:6834/1754860807,v1:172.21.15.81:6835/1754860807] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55add7a759b0 con 0x55add7b252b0 2024-02-21T23:07:54.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.554+0000 7fcf44c00640 1 --2- 172.21.15.28:0/2566421690 >> [v2:172.21.15.81:6834/1754860807,v1:172.21.15.81:6835/1754860807] conn(0x55add7b252b0 0x55add7b503b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:54.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.554+0000 7fcf44c00640 1 --2- 172.21.15.28:0/2566421690 >> [v2:172.21.15.81:6834/1754860807,v1:172.21.15.81:6835/1754860807] conn(0x55add7b252b0 0x55add7b503b0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=15cf444b5ddcb859 server_cookie=c1cc0b85eee3fc5d in_seq=0 out_seq=0 2024-02-21T23:07:54.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.554+0000 7fcf25600640 10 client.7725 ms_handle_connect on v2:172.21.15.81:6834/1754860807 2024-02-21T23:07:54.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.556+0000 7fcf25600640 1 -- 172.21.15.28:0/2566421690 <== mds.0 v2:172.21.15.81:6834/1754860807 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55add7a759b0 con 0x55add7b252b0 2024-02-21T23:07:54.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.556+0000 7fcf25600640 10 client.7725 handle_command_reply: tid=0 2024-02-21T23:07:54.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.556+0000 7fcf4a593740 1 client.7725 shutdown 2024-02-21T23:07:54.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.556+0000 7fcf4a593740 1 -- 172.21.15.28:0/2566421690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55add7b390e0 msgr2=0x55add7b3d120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:54.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.556+0000 7fcf4a593740 1 --2- 172.21.15.28:0/2566421690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55add7b390e0 0x55add7b3d120 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7fcf38000f80 tx=0x7fcf38004ab0 comp rx=0 tx=0).stop 2024-02-21T23:07:54.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.557+0000 7fcf4a593740 1 -- 172.21.15.28:0/2566421690 shutdown_connections 2024-02-21T23:07:54.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.557+0000 7fcf4a593740 1 --2- 172.21.15.28:0/2566421690 >> [v2:172.21.15.81:6834/1754860807,v1:172.21.15.81:6835/1754860807] conn(0x55add7b252b0 0x55add7b503b0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:54.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.557+0000 7fcf4a593740 1 --2- 172.21.15.28:0/2566421690 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55add7b406b0 0x55add7b38ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:54.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.557+0000 7fcf4a593740 1 --2- 172.21.15.28:0/2566421690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55add7b390e0 0x55add7b3d120 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:54.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.557+0000 7fcf4a593740 1 --2- 172.21.15.28:0/2566421690 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55add7b05330 0x55add7b53b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:54.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.557+0000 7fcf25600640 0 client.7725 ms_handle_reset on v2:172.21.15.81:6834/1754860807 2024-02-21T23:07:54.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.557+0000 7fcf4a593740 1 -- 172.21.15.28:0/2566421690 >> 172.21.15.28:0/2566421690 conn(0x55add7b04020 msgr2=0x55add7afd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:54.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.557+0000 7fcf4a593740 1 -- 172.21.15.28:0/2566421690 shutdown_connections 2024-02-21T23:07:54.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.557+0000 7fcf4a593740 1 -- 172.21.15.28:0/2566421690 wait complete. 2024-02-21T23:07:54.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.557+0000 7fcf4a593740 20 client.7725 trim_cache size 0 max 16384 2024-02-21T23:07:54.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.576+0000 7fcf4a593740 1 Processor -- start 2024-02-21T23:07:54.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.576+0000 7fcf4a593740 1 -- start start 2024-02-21T23:07:54.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.576+0000 7fcf4a593740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55add7b05330 0x55add7b42000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:54.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.576+0000 7fcf4a593740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55add7b390e0 0x55add7b20d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:54.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.577+0000 7fcf4a593740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55add7b406b0 0x55add7b62db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:54.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.577+0000 7fcf4a593740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf401f7b10 con 0x55add7b390e0 2024-02-21T23:07:54.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.577+0000 7fcf4a593740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf4014dc20 con 0x55add7b406b0 2024-02-21T23:07:54.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.577+0000 7fcf4a593740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf401f67b0 con 0x55add7b05330 2024-02-21T23:07:54.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.577+0000 7fcf3fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55add7b390e0 0x55add7b20d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:54.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.577+0000 7fcf3fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55add7b390e0 0x55add7b20d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60058/0 (socket says 172.21.15.28:60058) 2024-02-21T23:07:54.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.577+0000 7fcf3fe00640 1 -- 172.21.15.28:0/3016609930 learned_addr learned my addr 172.21.15.28:0/3016609930 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:54.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.577+0000 7fcf44c00640 1 --2- 172.21.15.28:0/3016609930 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55add7b05330 0x55add7b42000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:54.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.577+0000 7fcf45600640 1 --2- 172.21.15.28:0/3016609930 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55add7b406b0 0x55add7b62db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:54.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.578+0000 7fcf3fe00640 1 -- 172.21.15.28:0/3016609930 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55add7b05330 msgr2=0x55add7b42000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:54.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.578+0000 7fcf3fe00640 1 --2- 172.21.15.28:0/3016609930 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55add7b05330 0x55add7b42000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:54.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.578+0000 7fcf3fe00640 1 -- 172.21.15.28:0/3016609930 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55add7b406b0 msgr2=0x55add7b62db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:54.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.578+0000 7fcf3fe00640 1 --2- 172.21.15.28:0/3016609930 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55add7b406b0 0x55add7b62db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:54.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.578+0000 7fcf3fe00640 1 -- 172.21.15.28:0/3016609930 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55add78476b0 con 0x55add7b390e0 2024-02-21T23:07:54.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.578+0000 7fcf3fe00640 1 --2- 172.21.15.28:0/3016609930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55add7b390e0 0x55add7b20d80 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c057e90 tx=0x7fcf2c000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=665ddaf71e2f61a5 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:54.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.578+0000 7fcf27e00640 1 -- 172.21.15.28:0/3016609930 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf2c086070 con 0x55add7b390e0 2024-02-21T23:07:54.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.578+0000 7fcf27e00640 1 -- 172.21.15.28:0/3016609930 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf2c060610 con 0x55add7b390e0 2024-02-21T23:07:54.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.578+0000 7fcf27e00640 1 -- 172.21.15.28:0/3016609930 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf2c06f070 con 0x55add7b390e0 2024-02-21T23:07:54.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.579+0000 7fcf4a593740 1 -- 172.21.15.28:0/3016609930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55add7b390e0 msgr2=0x55add7b20d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:54.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.579+0000 7fcf4a593740 1 --2- 172.21.15.28:0/3016609930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55add7b390e0 0x55add7b20d80 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c057e90 tx=0x7fcf2c000ed0 comp rx=0 tx=0).stop 2024-02-21T23:07:54.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.579+0000 7fcf4a593740 1 -- 172.21.15.28:0/3016609930 shutdown_connections 2024-02-21T23:07:54.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.579+0000 7fcf4a593740 1 --2- 172.21.15.28:0/3016609930 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55add7b406b0 0x55add7b62db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:54.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.579+0000 7fcf4a593740 1 --2- 172.21.15.28:0/3016609930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55add7b390e0 0x55add7b20d80 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:54.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.579+0000 7fcf4a593740 1 --2- 172.21.15.28:0/3016609930 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55add7b05330 0x55add7b42000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:54.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.579+0000 7fcf4a593740 1 -- 172.21.15.28:0/3016609930 >> 172.21.15.28:0/3016609930 conn(0x55add7b04020 msgr2=0x55add7b3fd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:54.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.579+0000 7fcf4a593740 1 -- 172.21.15.28:0/3016609930 shutdown_connections 2024-02-21T23:07:54.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.579+0000 7fcf4a593740 1 -- 172.21.15.28:0/3016609930 wait complete. 2024-02-21T23:07:54.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.579+0000 7fcf4a593740 1 -- start start 2024-02-21T23:07:54.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.580+0000 7fcf4a593740 1 Processor -- start 2024-02-21T23:07:54.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.580+0000 7fcf4a593740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55add7b05330 0x55add7bac6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:54.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.580+0000 7fcf4a593740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55add7b390e0 0x55add7bacc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:54.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.581+0000 7fcf4a593740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55add7b406b0 0x55add7b631b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:54.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.581+0000 7fcf4a593740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55add7b50230 con 0x55add7b390e0 2024-02-21T23:07:54.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.581+0000 7fcf4a593740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55add7b3aa70 con 0x55add7b05330 2024-02-21T23:07:54.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.581+0000 7fcf4a593740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x55add7b54090 con 0x55add7b406b0 2024-02-21T23:07:54.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.581+0000 7fcf4a593740 10 client.0 _ll_register_callbacks cb 0x7fcf38011b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:07:54.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.581+0000 7fcf3fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55add7b390e0 0x55add7bacc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:54.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.581+0000 7fcf3fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55add7b390e0 0x55add7bacc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60062/0 (socket says 172.21.15.28:60062) 2024-02-21T23:07:54.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.581+0000 7fcf3fe00640 1 -- 172.21.15.28:0/2486604529 learned_addr learned my addr 172.21.15.28:0/2486604529 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:54.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.581+0000 7fcf3fe00640 1 -- 172.21.15.28:0/2486604529 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55add7b406b0 msgr2=0x55add7b631b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:54.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.581+0000 7fcf3fe00640 1 --2- 172.21.15.28:0/2486604529 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55add7b406b0 0x55add7b631b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:54.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.581+0000 7fcf3fe00640 1 -- 172.21.15.28:0/2486604529 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55add7b05330 msgr2=0x55add7bac6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:54.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.581+0000 7fcf3fe00640 1 --2- 172.21.15.28:0/2486604529 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55add7b05330 0x55add7bac6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:54.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.581+0000 7fcf3fe00640 1 -- 172.21.15.28:0/2486604529 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55add7b4c610 con 0x55add7b390e0 2024-02-21T23:07:54.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.581+0000 7fcf3fe00640 1 --2- 172.21.15.28:0/2486604529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55add7b390e0 0x55add7bacc20 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c00cdc0 tx=0x7fcf2c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=557fc41218454180 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:54.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.581+0000 7fcf25600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:07:54.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.582+0000 7fcf25600640 1 -- 172.21.15.28:0/2486604529 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf2c086030 con 0x55add7b390e0 2024-02-21T23:07:54.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.582+0000 7fcf25600640 1 -- 172.21.15.28:0/2486604529 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(111..111 src has 1..111) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fcf2c05e270 con 0x55add7b390e0 2024-02-21T23:07:54.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.582+0000 7fcf4a593740 1 -- 172.21.15.28:0/2486604529 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcf2c060610 con 0x55add7b390e0 2024-02-21T23:07:54.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.582+0000 7fcf25600640 1 -- 172.21.15.28:0/2486604529 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=112) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcf2c06f050 con 0x55add7b390e0 2024-02-21T23:07:54.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.582+0000 7fcf4a593740 10 client.7731 fetch_fsmap learned FSMap version 112 2024-02-21T23:07:54.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.582+0000 7fcf4a593740 1 -- 172.21.15.28:0/2486604529 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=112,monmap=2+}) v3 -- 0x55add7b387b0 con 0x55add7b390e0 2024-02-21T23:07:54.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.582+0000 7fcf25600640 1 -- 172.21.15.28:0/2486604529 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf2c060610 con 0x55add7b390e0 2024-02-21T23:07:54.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.582+0000 7fcf25600640 1 -- 172.21.15.28:0/2486604529 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 112) v1 ==== 3282+0+0 (secure 0 0 0) 0x7fcf2c057350 con 0x55add7b390e0 2024-02-21T23:07:54.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.582+0000 7fcf4a593740 10 client.7731 fetch_fsmap finished waiting for FSMap version 112 2024-02-21T23:07:54.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.582+0000 7fcf4a593740 10 client.7731 resolve_mds: resolved name 'd' to daemon mds.d 2024-02-21T23:07:54.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.582+0000 7fcf4a593740 20 client.7731 populate_metadata read hostname 'smithi028' 2024-02-21T23:07:54.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.582+0000 7fcf4a593740 1 --2- 172.21.15.28:0/2486604529 >> [v2:172.21.15.81:6834/1754860807,v1:172.21.15.81:6835/1754860807] conn(0x55add7b50230 0x55add7b50610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:54.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.582+0000 7fcf4a593740 4 client.7731 mds_command: new command op to 5729 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-21T23:07:54.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.582+0000 7fcf4a593740 1 -- 172.21.15.28:0/2486604529 --> [v2:172.21.15.81:6834/1754860807,v1:172.21.15.81:6835/1754860807] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55add7a6fd00 con 0x55add7b50230 2024-02-21T23:07:54.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.583+0000 7fcf44c00640 1 --2- 172.21.15.28:0/2486604529 >> [v2:172.21.15.81:6834/1754860807,v1:172.21.15.81:6835/1754860807] conn(0x55add7b50230 0x55add7b50610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:54.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.584+0000 7fcf44c00640 1 --2- 172.21.15.28:0/2486604529 >> [v2:172.21.15.81:6834/1754860807,v1:172.21.15.81:6835/1754860807] conn(0x55add7b50230 0x55add7b50610 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6162b3692858437d server_cookie=60277677fd090fea in_seq=0 out_seq=0 2024-02-21T23:07:54.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.584+0000 7fcf25600640 10 client.7731 ms_handle_connect on v2:172.21.15.81:6834/1754860807 2024-02-21T23:07:54.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.584+0000 7fcf25600640 1 -- 172.21.15.28:0/2486604529 <== mds.0 v2:172.21.15.81:6834/1754860807 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+328 (crc 0 0 0) 0x55add7a6fd00 con 0x55add7b50230 2024-02-21T23:07:54.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.584+0000 7fcf25600640 10 client.7731 handle_command_reply: tid=0 2024-02-21T23:07:54.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.584+0000 7fcf4a593740 1 client.7731 shutdown 2024-02-21T23:07:54.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.585+0000 7fcf4a593740 1 -- 172.21.15.28:0/2486604529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55add7b390e0 msgr2=0x55add7bacc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:54.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.585+0000 7fcf4a593740 1 --2- 172.21.15.28:0/2486604529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55add7b390e0 0x55add7bacc20 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c00cdc0 tx=0x7fcf2c000bc0 comp rx=0 tx=0).stop 2024-02-21T23:07:54.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.585+0000 7fcf4a593740 1 -- 172.21.15.28:0/2486604529 shutdown_connections 2024-02-21T23:07:54.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.585+0000 7fcf4a593740 1 --2- 172.21.15.28:0/2486604529 >> [v2:172.21.15.81:6834/1754860807,v1:172.21.15.81:6835/1754860807] conn(0x55add7b50230 0x55add7b50610 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:54.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.585+0000 7fcf4a593740 1 --2- 172.21.15.28:0/2486604529 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55add7b406b0 0x55add7b631b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:54.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.585+0000 7fcf4a593740 1 --2- 172.21.15.28:0/2486604529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55add7b390e0 0x55add7bacc20 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:54.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.585+0000 7fcf4a593740 1 --2- 172.21.15.28:0/2486604529 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55add7b05330 0x55add7bac6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:54.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.585+0000 7fcf25600640 0 client.7731 ms_handle_reset on v2:172.21.15.81:6834/1754860807 2024-02-21T23:07:54.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.585+0000 7fcf4a593740 1 -- 172.21.15.28:0/2486604529 >> 172.21.15.28:0/2486604529 conn(0x55add7b04020 msgr2=0x55add7af9d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:54.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.585+0000 7fcf4a593740 1 -- 172.21.15.28:0/2486604529 shutdown_connections 2024-02-21T23:07:54.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.585+0000 7fcf4a593740 1 -- 172.21.15.28:0/2486604529 wait complete. 2024-02-21T23:07:54.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.585+0000 7fcf4a593740 20 client.7731 trim_cache size 0 max 16384 2024-02-21T23:07:54.578 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-21T23:07:54.578 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac", 2024-02-21T23:07:54.578 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-02-21T23:07:54.578 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 5729, 2024-02-21T23:07:54.578 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-02-21T23:07:54.578 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-02-21T23:07:54.579 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-02-21T23:07:54.579 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 13.439072971, 2024-02-21T23:07:54.579 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 111, 2024-02-21T23:07:54.579 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 111, 2024-02-21T23:07:54.579 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 111, 2024-02-21T23:07:54.579 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 29.992142723000001 2024-02-21T23:07:54.579 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-21T23:07:54.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.585+0000 7fcf46000640 1 -- 172.21.15.28:0/3847540227 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcf2006b520 msgr2=0x7fcf2006d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:54.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.586+0000 7fcf46000640 1 --2- 172.21.15.28:0/3847540227 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcf2006b520 0x7fcf2006d9e0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fcf34036000 tx=0x7fcf34008030 comp rx=0 tx=0).stop 2024-02-21T23:07:54.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.586+0000 7fcf46000640 1 -- 172.21.15.28:0/3847540227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf400bf9a0 msgr2=0x7fcf401eafb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:54.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.586+0000 7fcf46000640 1 --2- 172.21.15.28:0/3847540227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf400bf9a0 0x7fcf401eafb0 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c00c400 tx=0x7fcf2c00c8c0 comp rx=0 tx=0).stop 2024-02-21T23:07:54.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.586+0000 7fcf46000640 1 -- 172.21.15.28:0/3847540227 shutdown_connections 2024-02-21T23:07:54.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.586+0000 7fcf46000640 1 --2- 172.21.15.28:0/3847540227 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcf2006b520 0x7fcf2006d9e0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:54.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.586+0000 7fcf46000640 1 --2- 172.21.15.28:0/3847540227 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcf401527a0 0x7fcf401eb4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:54.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.586+0000 7fcf46000640 1 --2- 172.21.15.28:0/3847540227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf400bf9a0 0x7fcf401eafb0 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:54.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.586+0000 7fcf46000640 1 --2- 172.21.15.28:0/3847540227 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf4006d410 0x7fcf400c4d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:54.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.586+0000 7fcf46000640 1 -- 172.21.15.28:0/3847540227 >> 172.21.15.28:0/3847540227 conn(0x7fcf400a8320 msgr2=0x7fcf401507a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:54.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.586+0000 7fcf46000640 1 -- 172.21.15.28:0/3847540227 shutdown_connections 2024-02-21T23:07:54.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:54.586+0000 7fcf46000640 1 -- 172.21.15.28:0/3847540227 wait complete. 2024-02-21T23:07:54.592 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 8, "epoch": 112, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 8, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-02-21T23:07:29.533547+0000", "damaged": [], "data_pools": [ 18 ], "enabled": true, "epoch": 111, "ever_allowed_features": 32, "explicitly_allowed_features": 32, "failed": [], "flags": 50, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": true, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5729": { "addr": "172.21.15.81:6835/1754860807", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6834", "nonce": 1754860807, "type": "v2" }, { "addr": "172.21.15.81:6835", "nonce": 1754860807, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5729, "incarnation": 104, "join_fscid": 8, "name": "d", "rank": 0, "state": "up:active", "state_seq": 8 }, "gid_5732": { "addr": "172.21.15.81:6837/3963422563", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6836", "nonce": 3963422563, "type": "v2" }, { "addr": "172.21.15.81:6837", "nonce": 3963422563, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5732, "incarnation": 0, "join_fscid": 8, "name": "b", "rank": 0, "state": "up:standby-replay", "state_seq": 7 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 17, "modified": "2024-02-21T23:07:49.218076+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5729 } } } ], "standbys": [ { "addr": "172.21.15.28:6835/3059988835", "addrs": { "addrvec": [ { "addr": "172.21.15.28:6834", "nonce": 3059988835, "type": "v2" }, { "addr": "172.21.15.28:6835", "nonce": 3059988835, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 112, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5843, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.28:6833/1119726035", "addrs": { "addrvec": [ { "addr": "172.21.15.28:6832", "nonce": 1119726035, "type": "v2" }, { "addr": "172.21.15.28:6833", "nonce": 1119726035, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 96, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7446, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-02-21T23:07:54.593 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': 8, 'name': 'b', 'state': 'up:standby-replay'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': 8, 'name': 'd', 'state': 'up:active'}] 2024-02-21T23:07:54.593 INFO:tasks.cephfs.test_failover:current = [{'gid': 5843, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.28:6835/3059988835', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3059988835}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3059988835}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 112}, {'gid': 5732, 'name': 'b', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 7, 'addr': '172.21.15.81:6837/3963422563', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3963422563}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3963422563}]}, 'join_fscid': 8, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 7446, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.28:6833/1119726035', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 1119726035}, {'type': 'v1', 'addr': '172.21.15.28:6833', 'nonce': 1119726035}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 96}, {'gid': 5729, 'name': 'd', 'rank': 0, 'incarnation': 104, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.81:6835/1754860807', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6834', 'nonce': 1754860807}, {'type': 'v1', 'addr': '172.21.15.81:6835', 'nonce': 1754860807}]}, 'join_fscid': 8, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}] 2024-02-21T23:07:54.593 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-02-21T23:07:54.594 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-02-21T23:07:54.594 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-21T23:07:54.705 DEBUG:teuthology.orchestra.run.smithi081:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-21T23:07:54.807 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-02-21T23:07:54.808 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-21T23:07:54.808 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-21T23:07:54.866 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054608,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-02-21T23:07:54.866 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-02-21T23:07:54.866 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-21T23:07:54.927 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053833,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498812k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249404k,nr_inodes=812351,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498812k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-02-21T23:07:54.928 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:07:55.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.163+0000 7f8365e00640 1 Processor -- start 2024-02-21T23:07:55.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.163+0000 7f8365e00640 1 -- start start 2024-02-21T23:07:55.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.164+0000 7f8365e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f83600c8a80 0x7f8360156c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:55.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.164+0000 7f8365e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f83600c8e60 0x7f8360153640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:55.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.165+0000 7f8365e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8360153b80 con 0x7f83601540b0 2024-02-21T23:07:55.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.165+0000 7f8365e00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83600c92e0 con 0x7f83600c8e60 2024-02-21T23:07:55.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.165+0000 7f8365e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83600c8000 con 0x7f83600c8a80 2024-02-21T23:07:55.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.165+0000 7f835fe00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f83600c8e60 0x7f8360153640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:55.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.165+0000 7f8364a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f83600c8a80 0x7f8360156c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:55.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.165+0000 7f835fe00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f83600c8e60 0x7f8360153640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.28:34238/0 (socket says 172.21.15.28:34238) 2024-02-21T23:07:55.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.165+0000 7f835fe00640 1 -- 172.21.15.28:0/2237315145 learned_addr learned my addr 172.21.15.28:0/2237315145 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:55.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.166+0000 7f835fe00640 1 -- 172.21.15.28:0/2237315145 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f83600c8a80 msgr2=0x7f8360156c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:55.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.166+0000 7f835fe00640 1 --2- 172.21.15.28:0/2237315145 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f83600c8a80 0x7f8360156c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:55.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.166+0000 7f835fe00640 1 -- 172.21.15.28:0/2237315145 >> v1:172.21.15.81:6789/0 conn(0x7f83601540b0 legacy=0x7f83600705a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:55.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.166+0000 7f835fe00640 1 -- 172.21.15.28:0/2237315145 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83600c82e0 con 0x7f83600c8e60 2024-02-21T23:07:55.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.167+0000 7f835fe00640 1 --2- 172.21.15.28:0/2237315145 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f83600c8e60 0x7f8360153640 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f8350009870 tx=0x7f835002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9d8d6252e46e9c2 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:55.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.168+0000 7f835f400640 1 -- 172.21.15.28:0/2237315145 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8350034070 con 0x7f83600c8e60 2024-02-21T23:07:55.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.168+0000 7f835f400640 1 -- 172.21.15.28:0/2237315145 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8350030840 con 0x7f83600c8e60 2024-02-21T23:07:55.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.168+0000 7f835f400640 1 -- 172.21.15.28:0/2237315145 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8350030c10 con 0x7f83600c8e60 2024-02-21T23:07:55.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.168+0000 7f8365e00640 1 -- 172.21.15.28:0/2237315145 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f83600c8e60 msgr2=0x7f8360153640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:55.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.168+0000 7f8365e00640 1 --2- 172.21.15.28:0/2237315145 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f83600c8e60 0x7f8360153640 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f8350009870 tx=0x7f835002b980 comp rx=0 tx=0).stop 2024-02-21T23:07:55.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.169+0000 7f8365e00640 1 -- 172.21.15.28:0/2237315145 shutdown_connections 2024-02-21T23:07:55.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.169+0000 7f8365e00640 1 --2- 172.21.15.28:0/2237315145 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f83600c8e60 0x7f8360153640 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:55.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.169+0000 7f8365e00640 1 --2- 172.21.15.28:0/2237315145 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f83600c8a80 0x7f8360156c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:55.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.169+0000 7f8365e00640 1 -- 172.21.15.28:0/2237315145 >> 172.21.15.28:0/2237315145 conn(0x7f836006b260 msgr2=0x7f836006b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:55.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.169+0000 7f8365e00640 1 -- 172.21.15.28:0/2237315145 shutdown_connections 2024-02-21T23:07:55.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.169+0000 7f8365e00640 1 -- 172.21.15.28:0/2237315145 wait complete. 2024-02-21T23:07:55.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.170+0000 7f8365e00640 1 Processor -- start 2024-02-21T23:07:55.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.171+0000 7f8365e00640 1 -- start start 2024-02-21T23:07:55.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.171+0000 7f8365e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83600c8a80 0x7f836014f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:55.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.171+0000 7f8365e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f83600c8e60 0x7f83600abe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:55.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.171+0000 7f8365e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f83601540b0 0x7f836014da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:55.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.171+0000 7f8365e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8360156770 con 0x7f83600c8a80 2024-02-21T23:07:55.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.171+0000 7f8365e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8360156490 con 0x7f83600c8e60 2024-02-21T23:07:55.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.171+0000 7f8365e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83600c7c90 con 0x7f83601540b0 2024-02-21T23:07:55.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.172+0000 7f8364a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83600c8a80 0x7f836014f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:55.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.172+0000 7f8364a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83600c8a80 0x7f836014f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48532/0 (socket says 172.21.15.28:48532) 2024-02-21T23:07:55.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.172+0000 7f8364a00640 1 -- 172.21.15.28:0/2213085503 learned_addr learned my addr 172.21.15.28:0/2213085503 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:55.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.172+0000 7f835fe00640 1 --2- 172.21.15.28:0/2213085503 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f83600c8e60 0x7f83600abe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:55.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.172+0000 7f8364a00640 1 -- 172.21.15.28:0/2213085503 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f83601540b0 msgr2=0x7f836014da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:55.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.172+0000 7f8364a00640 1 --2- 172.21.15.28:0/2213085503 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f83601540b0 0x7f836014da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:55.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.172+0000 7f8364a00640 1 -- 172.21.15.28:0/2213085503 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f83600c8e60 msgr2=0x7f83600abe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:55.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.172+0000 7f8364a00640 1 --2- 172.21.15.28:0/2213085503 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f83600c8e60 0x7f83600abe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:55.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.172+0000 7f8364a00640 1 -- 172.21.15.28:0/2213085503 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f836014c210 con 0x7f83600c8a80 2024-02-21T23:07:55.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.173+0000 7f8364a00640 1 --2- 172.21.15.28:0/2213085503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83600c8a80 0x7f836014f5c0 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f834800e9e0 tx=0x7f834800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:55.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.173+0000 7f835cc00640 1 -- 172.21.15.28:0/2213085503 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8348017070 con 0x7f83600c8a80 2024-02-21T23:07:55.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.173+0000 7f835cc00640 1 -- 172.21.15.28:0/2213085503 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f834800c9d0 con 0x7f83600c8a80 2024-02-21T23:07:55.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.173+0000 7f8365e00640 1 -- 172.21.15.28:0/2213085503 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f836014d450 con 0x7f83600c8a80 2024-02-21T23:07:55.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.173+0000 7f835cc00640 1 -- 172.21.15.28:0/2213085503 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834800cdc0 con 0x7f83600c8a80 2024-02-21T23:07:55.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.173+0000 7f8365e00640 1 -- 172.21.15.28:0/2213085503 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83600acf70 con 0x7f83600c8a80 2024-02-21T23:07:55.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.174+0000 7f8365e00640 1 -- 172.21.15.28:0/2213085503 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83601e7910 con 0x7f83600c8a80 2024-02-21T23:07:55.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.175+0000 7f835cc00640 1 -- 172.21.15.28:0/2213085503 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f834801a070 con 0x7f83600c8a80 2024-02-21T23:07:55.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.176+0000 7f835cc00640 1 --2- 172.21.15.28:0/2213085503 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f833c06b4d0 0x7f833c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:55.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.176+0000 7f835cc00640 1 -- 172.21.15.28:0/2213085503 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f834808a4c0 con 0x7f83600c8a80 2024-02-21T23:07:55.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.177+0000 7f835fe00640 1 --2- 172.21.15.28:0/2213085503 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f833c06b4d0 0x7f833c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:55.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.178+0000 7f835fe00640 1 --2- 172.21.15.28:0/2213085503 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f833c06b4d0 0x7f833c06d990 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f8350036000 tx=0x7f835003b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:55.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.180+0000 7f835cc00640 1 -- 172.21.15.28:0/2213085503 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f83601e7910 con 0x7f83600c8a80 2024-02-21T23:07:55.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.420+0000 7f8365e00640 1 -- 172.21.15.28:0/2213085503 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f83601535d0 con 0x7f83600c8a80 2024-02-21T23:07:55.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.422+0000 7f835cc00640 1 -- 172.21.15.28:0/2213085503 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+4949 (secure 0 0 0) 0x7f83480590c0 con 0x7f83600c8a80 2024-02-21T23:07:55.411 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:07:55.412 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":112,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5843,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3059988835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3059988835},{"type":"v1","addr":"172.21.15.28:6835","nonce":3059988835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":112},{"gid":7446,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/1119726035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1119726035},{"type":"v1","addr":"172.21.15.28:6833","nonce":1119726035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":111,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-02-21T23:07:29.533547+0000","modified":"2024-02-21T23:07:49.218076+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5729},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5729":{"gid":5729,"name":"d","rank":0,"incarnation":104,"state":"up:active","state_seq":8,"addr":"172.21.15.81:6835/1754860807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1754860807},{"type":"v1","addr":"172.21.15.81:6835","nonce":1754860807}]},"join_fscid":8,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5732":{"gid":5732,"name":"b","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":7,"addr":"172.21.15.81:6837/3963422563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3963422563},{"type":"v1","addr":"172.21.15.81:6837","nonce":3963422563}]},"join_fscid":8,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-02-21T23:07:55.412 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 112 2024-02-21T23:07:55.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.424+0000 7f8365e00640 1 -- 172.21.15.28:0/2213085503 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f833c06b4d0 msgr2=0x7f833c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:55.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.424+0000 7f8365e00640 1 --2- 172.21.15.28:0/2213085503 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f833c06b4d0 0x7f833c06d990 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f8350036000 tx=0x7f835003b040 comp rx=0 tx=0).stop 2024-02-21T23:07:55.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.424+0000 7f8365e00640 1 -- 172.21.15.28:0/2213085503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83600c8a80 msgr2=0x7f836014f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:55.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.424+0000 7f8365e00640 1 --2- 172.21.15.28:0/2213085503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83600c8a80 0x7f836014f5c0 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f834800e9e0 tx=0x7f834800eea0 comp rx=0 tx=0).stop 2024-02-21T23:07:55.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.425+0000 7f8365e00640 1 -- 172.21.15.28:0/2213085503 shutdown_connections 2024-02-21T23:07:55.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.425+0000 7f8365e00640 1 --2- 172.21.15.28:0/2213085503 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f833c06b4d0 0x7f833c06d990 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:55.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.425+0000 7f8365e00640 1 --2- 172.21.15.28:0/2213085503 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f83601540b0 0x7f836014da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:55.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.425+0000 7f8365e00640 1 --2- 172.21.15.28:0/2213085503 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f83600c8e60 0x7f83600abe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:55.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.425+0000 7f8365e00640 1 --2- 172.21.15.28:0/2213085503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83600c8a80 0x7f836014f5c0 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:55.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.425+0000 7f8365e00640 1 -- 172.21.15.28:0/2213085503 >> 172.21.15.28:0/2213085503 conn(0x7f836006b260 msgr2=0x7f83600a8db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:55.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.425+0000 7f8365e00640 1 -- 172.21.15.28:0/2213085503 shutdown_connections 2024-02-21T23:07:55.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.425+0000 7f8365e00640 1 -- 172.21.15.28:0/2213085503 wait complete. 2024-02-21T23:07:55.434 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:07:55.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.656+0000 7f08eb200640 1 Processor -- start 2024-02-21T23:07:55.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.656+0000 7f08eb200640 1 -- start start 2024-02-21T23:07:55.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.657+0000 7f08eb200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f08ec0aca20 0x7f08ec06be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:55.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.657+0000 7f08eb200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f08ec06c3c0 0x7f08ec0c0710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:55.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.657+0000 7f08eb200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f08ec072400 0x7f08ec0c0c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:55.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.657+0000 7f08eb200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08ec155ad0 con 0x7f08ec06c3c0 2024-02-21T23:07:55.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.657+0000 7f08eb200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08ec0c19e0 con 0x7f08ec0aca20 2024-02-21T23:07:55.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.657+0000 7f08eb200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08ec0c1cc0 con 0x7f08ec072400 2024-02-21T23:07:55.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.658+0000 7f08e9400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f08ec06c3c0 0x7f08ec0c0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:55.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.658+0000 7f08e9400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f08ec06c3c0 0x7f08ec0c0710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48536/0 (socket says 172.21.15.28:48536) 2024-02-21T23:07:55.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.658+0000 7f08e9400640 1 -- 172.21.15.28:0/2244896856 learned_addr learned my addr 172.21.15.28:0/2244896856 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:55.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.658+0000 7f08e9e00640 1 --2- 172.21.15.28:0/2244896856 >> v2:172.21.15.81:3300/0 conn(0x7f08ec0aca20 0x7f08ec06be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:55.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.658+0000 7f08ea800640 1 --2- 172.21.15.28:0/2244896856 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f08ec072400 0x7f08ec0c0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:55.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.658+0000 7f08e9400640 1 -- 172.21.15.28:0/2244896856 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f08ec072400 msgr2=0x7f08ec0c0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:55.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.658+0000 7f08e9400640 1 --2- 172.21.15.28:0/2244896856 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f08ec072400 0x7f08ec0c0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:55.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.658+0000 7f08e9400640 1 -- 172.21.15.28:0/2244896856 >> v2:172.21.15.81:3300/0 conn(0x7f08ec0aca20 msgr2=0x7f08ec06be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:55.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.658+0000 7f08e9400640 1 --2- 172.21.15.28:0/2244896856 >> v2:172.21.15.81:3300/0 conn(0x7f08ec0aca20 0x7f08ec06be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:55.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.658+0000 7f08e9400640 1 -- 172.21.15.28:0/2244896856 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08ec067250 con 0x7f08ec06c3c0 2024-02-21T23:07:55.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.659+0000 7f08e9400640 1 --2- 172.21.15.28:0/2244896856 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08ec06c3c0 0x7f08ec0c0710 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f08ec071570 tx=0x7f08dc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1f67b9dd2878e1 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:55.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.660+0000 7f08e8a00640 1 -- 172.21.15.28:0/2244896856 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08dc02e070 con 0x7f08ec06c3c0 2024-02-21T23:07:55.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.660+0000 7f08e8a00640 1 -- 172.21.15.28:0/2244896856 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08dc037070 con 0x7f08ec06c3c0 2024-02-21T23:07:55.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.660+0000 7f08e8a00640 1 -- 172.21.15.28:0/2244896856 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08dc031a00 con 0x7f08ec06c3c0 2024-02-21T23:07:55.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.660+0000 7f08eb200640 1 -- 172.21.15.28:0/2244896856 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08ec06c3c0 msgr2=0x7f08ec0c0710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:55.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.660+0000 7f08eb200640 1 --2- 172.21.15.28:0/2244896856 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08ec06c3c0 0x7f08ec0c0710 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f08ec071570 tx=0x7f08dc02b880 comp rx=0 tx=0).stop 2024-02-21T23:07:55.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.661+0000 7f08eb200640 1 -- 172.21.15.28:0/2244896856 shutdown_connections 2024-02-21T23:07:55.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.661+0000 7f08eb200640 1 --2- 172.21.15.28:0/2244896856 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f08ec072400 0x7f08ec0c0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:55.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.661+0000 7f08eb200640 1 --2- 172.21.15.28:0/2244896856 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08ec06c3c0 0x7f08ec0c0710 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:55.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.661+0000 7f08eb200640 1 --2- 172.21.15.28:0/2244896856 >> v2:172.21.15.81:3300/0 conn(0x7f08ec0aca20 0x7f08ec06be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:55.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.661+0000 7f08eb200640 1 -- 172.21.15.28:0/2244896856 >> 172.21.15.28:0/2244896856 conn(0x7f08ec071910 msgr2=0x7f08ec071cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:55.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.661+0000 7f08eb200640 1 -- 172.21.15.28:0/2244896856 shutdown_connections 2024-02-21T23:07:55.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.661+0000 7f08eb200640 1 -- 172.21.15.28:0/2244896856 wait complete. 2024-02-21T23:07:55.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.662+0000 7f08eb200640 1 Processor -- start 2024-02-21T23:07:55.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.663+0000 7f08eb200640 1 -- start start 2024-02-21T23:07:55.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.663+0000 7f08eb200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f08ec06c3c0 0x7f08ec154540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:55.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.663+0000 7f08eb200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f08ec072400 0x7f08ec13ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:55.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.663+0000 7f08eb200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08ec0aca20 0x7f08ec1eaa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:55.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.663+0000 7f08eb200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08ec0acf80 con 0x7f08ec0aca20 2024-02-21T23:07:55.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.663+0000 7f08eb200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08ec0ace00 con 0x7f08ec06c3c0 2024-02-21T23:07:55.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.663+0000 7f08eb200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08ec0c1190 con 0x7f08ec072400 2024-02-21T23:07:55.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.664+0000 7f08ea800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08ec0aca20 0x7f08ec1eaa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:55.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.664+0000 7f08ea800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08ec0aca20 0x7f08ec1eaa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48542/0 (socket says 172.21.15.28:48542) 2024-02-21T23:07:55.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.664+0000 7f08ea800640 1 -- 172.21.15.28:0/3567585523 learned_addr learned my addr 172.21.15.28:0/3567585523 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:55.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.664+0000 7f08e9e00640 1 --2- 172.21.15.28:0/3567585523 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f08ec06c3c0 0x7f08ec154540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:55.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.664+0000 7f08ea800640 1 -- 172.21.15.28:0/3567585523 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f08ec072400 msgr2=0x7f08ec13ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:55.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.664+0000 7f08e9400640 1 --2- 172.21.15.28:0/3567585523 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f08ec072400 0x7f08ec13ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:55.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.664+0000 7f08ea800640 1 --2- 172.21.15.28:0/3567585523 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f08ec072400 0x7f08ec13ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:55.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.664+0000 7f08ea800640 1 -- 172.21.15.28:0/3567585523 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f08ec06c3c0 msgr2=0x7f08ec154540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:55.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.664+0000 7f08ea800640 1 --2- 172.21.15.28:0/3567585523 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f08ec06c3c0 0x7f08ec154540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:55.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.664+0000 7f08ea800640 1 -- 172.21.15.28:0/3567585523 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08ec0abd30 con 0x7f08ec0aca20 2024-02-21T23:07:55.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.664+0000 7f08e9e00640 1 --2- 172.21.15.28:0/3567585523 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f08ec06c3c0 0x7f08ec154540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:07:55.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.664+0000 7f08e9400640 1 --2- 172.21.15.28:0/3567585523 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f08ec072400 0x7f08ec13ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:07:55.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.665+0000 7f08ea800640 1 --2- 172.21.15.28:0/3567585523 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08ec0aca20 0x7f08ec1eaa40 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f08e4007e60 tx=0x7f08e400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:55.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.665+0000 7f08e2000640 1 -- 172.21.15.28:0/3567585523 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08e4014620 con 0x7f08ec0aca20 2024-02-21T23:07:55.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.665+0000 7f08e2000640 1 -- 172.21.15.28:0/3567585523 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08e4019070 con 0x7f08ec0aca20 2024-02-21T23:07:55.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.665+0000 7f08e2000640 1 -- 172.21.15.28:0/3567585523 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08e4014a20 con 0x7f08ec0aca20 2024-02-21T23:07:55.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.665+0000 7f08eb200640 1 -- 172.21.15.28:0/3567585523 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08ec0ac2b0 con 0x7f08ec0aca20 2024-02-21T23:07:55.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.665+0000 7f08eb200640 1 -- 172.21.15.28:0/3567585523 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08ec1eed80 con 0x7f08ec0aca20 2024-02-21T23:07:55.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.666+0000 7f08eb200640 1 -- 172.21.15.28:0/3567585523 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08ec04f290 con 0x7f08ec0aca20 2024-02-21T23:07:55.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.667+0000 7f08e2000640 1 -- 172.21.15.28:0/3567585523 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f08e4014bc0 con 0x7f08ec0aca20 2024-02-21T23:07:55.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.668+0000 7f08e2000640 1 --2- 172.21.15.28:0/3567585523 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f08cc06b4d0 0x7f08cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:55.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.668+0000 7f08e2000640 1 -- 172.21.15.28:0/3567585523 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f08e408ac80 con 0x7f08ec0aca20 2024-02-21T23:07:55.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.668+0000 7f08e9e00640 1 --2- 172.21.15.28:0/3567585523 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f08cc06b4d0 0x7f08cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:55.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.669+0000 7f08e9e00640 1 --2- 172.21.15.28:0/3567585523 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f08cc06b4d0 0x7f08cc06d990 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f08d8006fd0 tx=0x7f08d8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:55.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.673+0000 7f08e2000640 1 -- 172.21.15.28:0/3567585523 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f08e4059910 con 0x7f08ec0aca20 2024-02-21T23:07:55.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.875+0000 7f08eb200640 1 -- 172.21.15.28:0/3567585523 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f08ec06c080 con 0x7f08ec0aca20 2024-02-21T23:07:55.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.877+0000 7f08e2000640 1 -- 172.21.15.28:0/3567585523 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v111) v1 ==== 74+0+17021 (secure 0 0 0) 0x7f08e4059210 con 0x7f08ec0aca20 2024-02-21T23:07:55.867 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:07:55.867 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":111,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:07:41.139084+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:07:27.477155+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-02-21T23:07:28.549456+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6835/1259832430":"2024-02-22T23:07:41.139002+0000","172.21.15.28:6834/1259832430":"2024-02-22T23:07:41.139002+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:07:55.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.880+0000 7f08eb200640 1 -- 172.21.15.28:0/3567585523 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f08cc06b4d0 msgr2=0x7f08cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:55.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.880+0000 7f08eb200640 1 --2- 172.21.15.28:0/3567585523 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f08cc06b4d0 0x7f08cc06d990 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f08d8006fd0 tx=0x7f08d8008040 comp rx=0 tx=0).stop 2024-02-21T23:07:55.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.880+0000 7f08eb200640 1 -- 172.21.15.28:0/3567585523 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08ec0aca20 msgr2=0x7f08ec1eaa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:55.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.880+0000 7f08eb200640 1 --2- 172.21.15.28:0/3567585523 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08ec0aca20 0x7f08ec1eaa40 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f08e4007e60 tx=0x7f08e400e4b0 comp rx=0 tx=0).stop 2024-02-21T23:07:55.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.880+0000 7f08eb200640 1 -- 172.21.15.28:0/3567585523 shutdown_connections 2024-02-21T23:07:55.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.880+0000 7f08eb200640 1 --2- 172.21.15.28:0/3567585523 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f08cc06b4d0 0x7f08cc06d990 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:55.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.880+0000 7f08eb200640 1 --2- 172.21.15.28:0/3567585523 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f08ec072400 0x7f08ec13ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:55.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.880+0000 7f08eb200640 1 --2- 172.21.15.28:0/3567585523 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08ec0aca20 0x7f08ec1eaa40 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:55.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.880+0000 7f08eb200640 1 --2- 172.21.15.28:0/3567585523 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f08ec06c3c0 0x7f08ec154540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:55.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.880+0000 7f08eb200640 1 -- 172.21.15.28:0/3567585523 >> 172.21.15.28:0/3567585523 conn(0x7f08ec071910 msgr2=0x7f08ec06f1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:55.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.881+0000 7f08eb200640 1 -- 172.21.15.28:0/3567585523 shutdown_connections 2024-02-21T23:07:55.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:55.881+0000 7f08eb200640 1 -- 172.21.15.28:0/3567585523 wait complete. 2024-02-21T23:07:55.890 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-21T23:02:30.728329+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-21T23:07:27.477155+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_data', 'create_time': '2024-02-21T23:07:28.549456+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '110', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-21T23:07:55.890 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-21T23:07:56.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.126+0000 7fda05200640 1 Processor -- start 2024-02-21T23:07:56.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.126+0000 7fda05200640 1 -- start start 2024-02-21T23:07:56.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.127+0000 7fda05200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fda00150240 0x7fda000c12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:56.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.127+0000 7fda05200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fda000c4580 0x7fda0006db10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:56.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.127+0000 7fda05200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda0006f780 con 0x7fda000c2c10 2024-02-21T23:07:56.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.128+0000 7fda05200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda00150620 con 0x7fda000c4580 2024-02-21T23:07:56.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.128+0000 7fda05200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda00153a90 con 0x7fda00150240 2024-02-21T23:07:56.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.128+0000 7fd9fea00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fda000c4580 0x7fda0006db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:56.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.128+0000 7fd9fea00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fda000c4580 0x7fda0006db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48558/0 (socket says 172.21.15.28:48558) 2024-02-21T23:07:56.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.128+0000 7fd9fea00640 1 -- 172.21.15.28:0/2297889872 learned_addr learned my addr 172.21.15.28:0/2297889872 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:56.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.128+0000 7fd9ff400640 1 --2- 172.21.15.28:0/2297889872 >> v2:172.21.15.81:3300/0 conn(0x7fda00150240 0x7fda000c12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:56.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.129+0000 7fd9fea00640 1 -- 172.21.15.28:0/2297889872 >> v2:172.21.15.81:3300/0 conn(0x7fda00150240 msgr2=0x7fda000c12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:56.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.129+0000 7fd9fea00640 1 --2- 172.21.15.28:0/2297889872 >> v2:172.21.15.81:3300/0 conn(0x7fda00150240 0x7fda000c12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:56.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.129+0000 7fd9fea00640 1 -- 172.21.15.28:0/2297889872 >> v1:172.21.15.28:6789/0 conn(0x7fda000c2c10 legacy=0x7fda0006ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:56.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.129+0000 7fd9fea00640 1 -- 172.21.15.28:0/2297889872 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda0006f4a0 con 0x7fda000c4580 2024-02-21T23:07:56.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.129+0000 7fd9ff400640 1 --2- 172.21.15.28:0/2297889872 >> v2:172.21.15.81:3300/0 conn(0x7fda00150240 0x7fda000c12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:07:56.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.130+0000 7fd9fea00640 1 --2- 172.21.15.28:0/2297889872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda000c4580 0x7fda0006db10 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7fda0006bef0 tx=0x7fd9ec02b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b914702975507388 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:56.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.130+0000 7fd9fe000640 1 -- 172.21.15.28:0/2297889872 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9ec035020 con 0x7fda000c4580 2024-02-21T23:07:56.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.130+0000 7fd9fe000640 1 -- 172.21.15.28:0/2297889872 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9ec031890 con 0x7fda000c4580 2024-02-21T23:07:56.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.130+0000 7fd9fe000640 1 -- 172.21.15.28:0/2297889872 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9ec031c60 con 0x7fda000c4580 2024-02-21T23:07:56.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.131+0000 7fda05200640 1 -- 172.21.15.28:0/2297889872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda000c4580 msgr2=0x7fda0006db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:56.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.131+0000 7fda05200640 1 --2- 172.21.15.28:0/2297889872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda000c4580 0x7fda0006db10 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7fda0006bef0 tx=0x7fd9ec02b930 comp rx=0 tx=0).stop 2024-02-21T23:07:56.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.131+0000 7fda05200640 1 -- 172.21.15.28:0/2297889872 shutdown_connections 2024-02-21T23:07:56.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.131+0000 7fda05200640 1 --2- 172.21.15.28:0/2297889872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda000c4580 0x7fda0006db10 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:56.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.131+0000 7fda05200640 1 --2- 172.21.15.28:0/2297889872 >> v2:172.21.15.81:3300/0 conn(0x7fda00150240 0x7fda000c12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:56.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.131+0000 7fda05200640 1 -- 172.21.15.28:0/2297889872 >> 172.21.15.28:0/2297889872 conn(0x7fda000a8530 msgr2=0x7fda000a8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:56.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.131+0000 7fda05200640 1 -- 172.21.15.28:0/2297889872 shutdown_connections 2024-02-21T23:07:56.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.131+0000 7fda05200640 1 -- 172.21.15.28:0/2297889872 wait complete. 2024-02-21T23:07:56.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.133+0000 7fda05200640 1 Processor -- start 2024-02-21T23:07:56.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.134+0000 7fda05200640 1 -- start start 2024-02-21T23:07:56.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.135+0000 7fda05200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fda000c2c10 0x7fda001e9e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:56.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.135+0000 7fda05200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda000c4580 0x7fda001ed200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:56.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.135+0000 7fda05200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda00150240 0x7fda00151d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:56.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.135+0000 7fda05200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda000c4b90 con 0x7fda00150240 2024-02-21T23:07:56.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.135+0000 7fda05200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda000c4a10 con 0x7fda000c4580 2024-02-21T23:07:56.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.135+0000 7fda05200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda0006b580 con 0x7fda000c2c10 2024-02-21T23:07:56.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.135+0000 7fd9ffe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda00150240 0x7fda00151d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:56.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.135+0000 7fd9ffe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda00150240 0x7fda00151d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48574/0 (socket says 172.21.15.28:48574) 2024-02-21T23:07:56.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.135+0000 7fd9ffe00640 1 -- 172.21.15.28:0/1267129216 learned_addr learned my addr 172.21.15.28:0/1267129216 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:56.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.135+0000 7fd9ff400640 1 --2- 172.21.15.28:0/1267129216 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fda000c2c10 0x7fda001e9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:56.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.135+0000 7fd9fea00640 1 --2- 172.21.15.28:0/1267129216 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda000c4580 0x7fda001ed200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:56.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.136+0000 7fd9ffe00640 1 -- 172.21.15.28:0/1267129216 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fda000c2c10 msgr2=0x7fda001e9e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:56.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.136+0000 7fd9ffe00640 1 --2- 172.21.15.28:0/1267129216 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fda000c2c10 0x7fda001e9e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:56.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.136+0000 7fd9ffe00640 1 -- 172.21.15.28:0/1267129216 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda000c4580 msgr2=0x7fda001ed200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:56.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.136+0000 7fd9ffe00640 1 --2- 172.21.15.28:0/1267129216 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda000c4580 0x7fda001ed200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:56.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.136+0000 7fd9ffe00640 1 -- 172.21.15.28:0/1267129216 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda001e9390 con 0x7fda00150240 2024-02-21T23:07:56.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.136+0000 7fd9ffe00640 1 --2- 172.21.15.28:0/1267129216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda00150240 0x7fda00151d90 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7fd9f800c730 tx=0x7fd9f800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:56.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.136+0000 7fd9f7400640 1 -- 172.21.15.28:0/1267129216 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9f8003b90 con 0x7fda00150240 2024-02-21T23:07:56.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.136+0000 7fd9f7400640 1 -- 172.21.15.28:0/1267129216 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9f8003d30 con 0x7fda00150240 2024-02-21T23:07:56.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.136+0000 7fd9f7400640 1 -- 172.21.15.28:0/1267129216 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9f8005310 con 0x7fda00150240 2024-02-21T23:07:56.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.136+0000 7fda05200640 1 -- 172.21.15.28:0/1267129216 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda001dc5f0 con 0x7fda00150240 2024-02-21T23:07:56.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.137+0000 7fda05200640 1 -- 172.21.15.28:0/1267129216 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda001522d0 con 0x7fda00150240 2024-02-21T23:07:56.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.138+0000 7fda05200640 1 -- 172.21.15.28:0/1267129216 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda0014e470 con 0x7fda00150240 2024-02-21T23:07:56.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.139+0000 7fd9f7400640 1 -- 172.21.15.28:0/1267129216 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd9f80056c0 con 0x7fda00150240 2024-02-21T23:07:56.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.140+0000 7fd9f7400640 1 --2- 172.21.15.28:0/1267129216 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd9e006b500 0x7fd9e006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:56.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.140+0000 7fd9f7400640 1 -- 172.21.15.28:0/1267129216 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fd9f808a3e0 con 0x7fda00150240 2024-02-21T23:07:56.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.141+0000 7fd9ff400640 1 --2- 172.21.15.28:0/1267129216 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd9e006b500 0x7fd9e006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:56.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.142+0000 7fd9ff400640 1 --2- 172.21.15.28:0/1267129216 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd9e006b500 0x7fd9e006d9c0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fda001e70c0 tx=0x7fd9f0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:56.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.146+0000 7fd9f7400640 1 -- 172.21.15.28:0/1267129216 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fda0014e470 con 0x7fda00150240 2024-02-21T23:07:56.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.353+0000 7fda05200640 1 -- 172.21.15.28:0/1267129216 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fda0004f290 con 0x7fda00150240 2024-02-21T23:07:56.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.354+0000 7fd9f7400640 1 -- 172.21.15.28:0/1267129216 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v111) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd9f8059070 con 0x7fda00150240 2024-02-21T23:07:56.344 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-02-21T23:07:56.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.357+0000 7fda05200640 1 -- 172.21.15.28:0/1267129216 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd9e006b500 msgr2=0x7fd9e006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:56.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.357+0000 7fda05200640 1 --2- 172.21.15.28:0/1267129216 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd9e006b500 0x7fd9e006d9c0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fda001e70c0 tx=0x7fd9f0008040 comp rx=0 tx=0).stop 2024-02-21T23:07:56.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.357+0000 7fda05200640 1 -- 172.21.15.28:0/1267129216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda00150240 msgr2=0x7fda00151d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:56.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.357+0000 7fda05200640 1 --2- 172.21.15.28:0/1267129216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda00150240 0x7fda00151d90 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7fd9f800c730 tx=0x7fd9f800cbf0 comp rx=0 tx=0).stop 2024-02-21T23:07:56.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.357+0000 7fda05200640 1 -- 172.21.15.28:0/1267129216 shutdown_connections 2024-02-21T23:07:56.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.357+0000 7fda05200640 1 --2- 172.21.15.28:0/1267129216 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd9e006b500 0x7fd9e006d9c0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:56.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.357+0000 7fda05200640 1 --2- 172.21.15.28:0/1267129216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda00150240 0x7fda00151d90 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:56.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.357+0000 7fda05200640 1 --2- 172.21.15.28:0/1267129216 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda000c4580 0x7fda001ed200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:56.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.357+0000 7fda05200640 1 --2- 172.21.15.28:0/1267129216 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fda000c2c10 0x7fda001e9e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:56.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.357+0000 7fda05200640 1 -- 172.21.15.28:0/1267129216 >> 172.21.15.28:0/1267129216 conn(0x7fda000a8530 msgr2=0x7fda000c30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:56.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.357+0000 7fda05200640 1 -- 172.21.15.28:0/1267129216 shutdown_connections 2024-02-21T23:07:56.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.357+0000 7fda05200640 1 -- 172.21.15.28:0/1267129216 wait complete. 2024-02-21T23:07:56.365 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-21T23:07:56.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.605+0000 7f6fd7000640 1 Processor -- start 2024-02-21T23:07:56.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.605+0000 7f6fd7000640 1 -- start start 2024-02-21T23:07:56.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.607+0000 7f6fd7000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6fd00c5a70 0x7f6fd0155a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:56.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.607+0000 7f6fd7000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6fd00c2b30 0x7f6fd006d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:56.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.607+0000 7f6fd7000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fd0071970 con 0x7f6fd00c22e0 2024-02-21T23:07:56.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.607+0000 7f6fd7000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fd0072a10 con 0x7f6fd00c2b30 2024-02-21T23:07:56.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.607+0000 7f6fd7000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd0072580 con 0x7f6fd00c5a70 2024-02-21T23:07:56.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.607+0000 7f6fd5000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6fd00c2b30 0x7f6fd006d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:56.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.607+0000 7f6fd5000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6fd00c2b30 0x7f6fd006d830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48582/0 (socket says 172.21.15.28:48582) 2024-02-21T23:07:56.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.607+0000 7f6fd5000640 1 -- 172.21.15.28:0/3464601838 learned_addr learned my addr 172.21.15.28:0/3464601838 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:56.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.607+0000 7f6fcfe00640 1 --2- 172.21.15.28:0/3464601838 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6fd00c5a70 0x7f6fd0155a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:56.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.608+0000 7f6fd5000640 1 -- 172.21.15.28:0/3464601838 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6fd00c5a70 msgr2=0x7f6fd0155a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:56.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.608+0000 7f6fd5000640 1 --2- 172.21.15.28:0/3464601838 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6fd00c5a70 0x7f6fd0155a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:56.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.608+0000 7f6fd5000640 1 -- 172.21.15.28:0/3464601838 >> v1:172.21.15.81:6789/0 conn(0x7f6fd00c22e0 legacy=0x7f6fd006d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:56.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.608+0000 7f6fd5000640 1 -- 172.21.15.28:0/3464601838 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fd0071c50 con 0x7f6fd00c2b30 2024-02-21T23:07:56.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.608+0000 7f6fcfe00640 1 --2- 172.21.15.28:0/3464601838 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6fd00c5a70 0x7f6fd0155a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:07:56.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.609+0000 7f6fd5000640 1 --2- 172.21.15.28:0/3464601838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6fd00c2b30 0x7f6fd006d830 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f6fc8009870 tx=0x7f6fc802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98af98d4fdf994bb server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:56.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.609+0000 7f6fcea00640 1 -- 172.21.15.28:0/3464601838 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fc80317c0 con 0x7f6fd00c2b30 2024-02-21T23:07:56.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.609+0000 7f6fcea00640 1 -- 172.21.15.28:0/3464601838 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fc8037070 con 0x7f6fd00c2b30 2024-02-21T23:07:56.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.609+0000 7f6fcea00640 1 -- 172.21.15.28:0/3464601838 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fc8031b80 con 0x7f6fd00c2b30 2024-02-21T23:07:56.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.610+0000 7f6fd7000640 1 -- 172.21.15.28:0/3464601838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6fd00c2b30 msgr2=0x7f6fd006d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:56.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.610+0000 7f6fd7000640 1 --2- 172.21.15.28:0/3464601838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6fd00c2b30 0x7f6fd006d830 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f6fc8009870 tx=0x7f6fc802b860 comp rx=0 tx=0).stop 2024-02-21T23:07:56.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.610+0000 7f6fd7000640 1 -- 172.21.15.28:0/3464601838 shutdown_connections 2024-02-21T23:07:56.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.610+0000 7f6fd7000640 1 --2- 172.21.15.28:0/3464601838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6fd00c2b30 0x7f6fd006d830 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:56.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.610+0000 7f6fd7000640 1 --2- 172.21.15.28:0/3464601838 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6fd00c5a70 0x7f6fd0155a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:56.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.610+0000 7f6fd7000640 1 -- 172.21.15.28:0/3464601838 >> 172.21.15.28:0/3464601838 conn(0x7f6fd00a85a0 msgr2=0x7f6fd00a8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:56.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.610+0000 7f6fd7000640 1 -- 172.21.15.28:0/3464601838 shutdown_connections 2024-02-21T23:07:56.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.610+0000 7f6fd7000640 1 -- 172.21.15.28:0/3464601838 wait complete. 2024-02-21T23:07:56.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.612+0000 7f6fd7000640 1 Processor -- start 2024-02-21T23:07:56.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.612+0000 7f6fd7000640 1 -- start start 2024-02-21T23:07:56.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.613+0000 7f6fd7000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6fd00c22e0 0x7f6fd01ec3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:56.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.613+0000 7f6fd7000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6fd00c2b30 0x7f6fd00aa830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:56.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.613+0000 7f6fd7000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6fd00c5a70 0x7f6fd00aad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:56.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.613+0000 7f6fd7000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd00a6160 con 0x7f6fd00c22e0 2024-02-21T23:07:56.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.613+0000 7f6fd7000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd0072210 con 0x7f6fd00c2b30 2024-02-21T23:07:56.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.613+0000 7f6fd7000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd00a62e0 con 0x7f6fd00c5a70 2024-02-21T23:07:56.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.613+0000 7f6fcfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6fd00c22e0 0x7f6fd01ec3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:56.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.613+0000 7f6fcfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6fd00c22e0 0x7f6fd01ec3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48590/0 (socket says 172.21.15.28:48590) 2024-02-21T23:07:56.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.613+0000 7f6fcfe00640 1 -- 172.21.15.28:0/3844547760 learned_addr learned my addr 172.21.15.28:0/3844547760 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:56.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.613+0000 7f6fd5000640 1 --2- 172.21.15.28:0/3844547760 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6fd00c5a70 0x7f6fd00aad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:56.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.614+0000 7f6fcf400640 1 --2- 172.21.15.28:0/3844547760 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6fd00c2b30 0x7f6fd00aa830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:56.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.614+0000 7f6fcfe00640 1 -- 172.21.15.28:0/3844547760 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6fd00c5a70 msgr2=0x7f6fd00aad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:56.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.614+0000 7f6fcfe00640 1 --2- 172.21.15.28:0/3844547760 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6fd00c5a70 0x7f6fd00aad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:56.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.614+0000 7f6fcfe00640 1 -- 172.21.15.28:0/3844547760 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6fd00c2b30 msgr2=0x7f6fd00aa830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:56.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.614+0000 7f6fcfe00640 1 --2- 172.21.15.28:0/3844547760 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6fd00c2b30 0x7f6fd00aa830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:56.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.614+0000 7f6fcfe00640 1 -- 172.21.15.28:0/3844547760 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fd01523b0 con 0x7f6fd00c22e0 2024-02-21T23:07:56.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.614+0000 7f6fcf400640 1 --2- 172.21.15.28:0/3844547760 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6fd00c2b30 0x7f6fd00aa830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:07:56.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.614+0000 7f6fcfe00640 1 --2- 172.21.15.28:0/3844547760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6fd00c22e0 0x7f6fd01ec3d0 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f6fc000d100 tx=0x7f6fc000d5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:56.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.614+0000 7f6fc7e00640 1 -- 172.21.15.28:0/3844547760 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fc001a070 con 0x7f6fd00c22e0 2024-02-21T23:07:56.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.614+0000 7f6fc7e00640 1 -- 172.21.15.28:0/3844547760 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fc0003b90 con 0x7f6fd00c22e0 2024-02-21T23:07:56.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.614+0000 7f6fc7e00640 1 -- 172.21.15.28:0/3844547760 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fc0004030 con 0x7f6fd00c22e0 2024-02-21T23:07:56.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.614+0000 7f6fd7000640 1 -- 172.21.15.28:0/3844547760 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fd01538c0 con 0x7f6fd00c22e0 2024-02-21T23:07:56.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.615+0000 7f6fd7000640 1 -- 172.21.15.28:0/3844547760 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fd00ab980 con 0x7f6fd00c22e0 2024-02-21T23:07:56.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.616+0000 7f6fd7000640 1 -- 172.21.15.28:0/3844547760 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fd006c560 con 0x7f6fd00c22e0 2024-02-21T23:07:56.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.617+0000 7f6fc7e00640 1 -- 172.21.15.28:0/3844547760 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6fc001d070 con 0x7f6fd00c22e0 2024-02-21T23:07:56.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.617+0000 7f6fc7e00640 1 --2- 172.21.15.28:0/3844547760 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6fb006b4d0 0x7f6fb006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:56.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.618+0000 7f6fc7e00640 1 -- 172.21.15.28:0/3844547760 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f6fc008a6d0 con 0x7f6fd00c22e0 2024-02-21T23:07:56.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.618+0000 7f6fcf400640 1 --2- 172.21.15.28:0/3844547760 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6fb006b4d0 0x7f6fb006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:56.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.619+0000 7f6fcf400640 1 --2- 172.21.15.28:0/3844547760 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6fb006b4d0 0x7f6fb006d990 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f6fd00c62a0 tx=0x7f6fbc008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:56.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.622+0000 7f6fc7e00640 1 -- 172.21.15.28:0/3844547760 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f6fd006c560 con 0x7f6fd00c22e0 2024-02-21T23:07:56.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.824+0000 7f6fd7000640 1 -- 172.21.15.28:0/3844547760 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6fd0067f30 con 0x7f6fd00c22e0 2024-02-21T23:07:56.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.825+0000 7f6fc7e00640 1 -- 172.21.15.28:0/3844547760 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v111) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6fc0010070 con 0x7f6fd00c22e0 2024-02-21T23:07:56.814 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-02-21T23:07:56.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.828+0000 7f6fd7000640 1 -- 172.21.15.28:0/3844547760 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6fb006b4d0 msgr2=0x7f6fb006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:56.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.828+0000 7f6fd7000640 1 --2- 172.21.15.28:0/3844547760 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6fb006b4d0 0x7f6fb006d990 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f6fd00c62a0 tx=0x7f6fbc008040 comp rx=0 tx=0).stop 2024-02-21T23:07:56.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.828+0000 7f6fd7000640 1 -- 172.21.15.28:0/3844547760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6fd00c22e0 msgr2=0x7f6fd01ec3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:56.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.828+0000 7f6fd7000640 1 --2- 172.21.15.28:0/3844547760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6fd00c22e0 0x7f6fd01ec3d0 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f6fc000d100 tx=0x7f6fc000d5c0 comp rx=0 tx=0).stop 2024-02-21T23:07:56.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.828+0000 7f6fd7000640 1 -- 172.21.15.28:0/3844547760 shutdown_connections 2024-02-21T23:07:56.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.828+0000 7f6fd7000640 1 --2- 172.21.15.28:0/3844547760 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6fb006b4d0 0x7f6fb006d990 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:56.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.828+0000 7f6fd7000640 1 --2- 172.21.15.28:0/3844547760 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6fd00c5a70 0x7f6fd00aad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:56.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.828+0000 7f6fd7000640 1 --2- 172.21.15.28:0/3844547760 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6fd00c2b30 0x7f6fd00aa830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:56.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.828+0000 7f6fd7000640 1 --2- 172.21.15.28:0/3844547760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6fd00c22e0 0x7f6fd01ec3d0 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:56.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.828+0000 7f6fd7000640 1 -- 172.21.15.28:0/3844547760 >> 172.21.15.28:0/3844547760 conn(0x7f6fd00a85a0 msgr2=0x7f6fd0155e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:56.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.828+0000 7f6fd7000640 1 -- 172.21.15.28:0/3844547760 shutdown_connections 2024-02-21T23:07:56.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:56.828+0000 7f6fd7000640 1 -- 172.21.15.28:0/3844547760 wait complete. 2024-02-21T23:07:56.841 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-21T23:07:57.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.072+0000 7f59b6c00640 1 Processor -- start 2024-02-21T23:07:57.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.073+0000 7f59b6c00640 1 -- start start 2024-02-21T23:07:57.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.074+0000 7f59b6c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f59b00c4960 0x7f59b00c6220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:57.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.074+0000 7f59b6c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f59b01524f0 0x7f59b0152930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:57.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.074+0000 7f59b6c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f59b01586b0 0x7f59b0158b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:57.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.074+0000 7f59b6c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59b00a62e0 con 0x7f59b01524f0 2024-02-21T23:07:57.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.074+0000 7f59b6c00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59b00c1960 con 0x7f59b01586b0 2024-02-21T23:07:57.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.074+0000 7f59b6c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59b00c20d0 con 0x7f59b00c4960 2024-02-21T23:07:57.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.074+0000 7f59a7e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f59b01524f0 0x7f59b0152930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:57.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.075+0000 7f59a7e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f59b01524f0 0x7f59b0152930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48598/0 (socket says 172.21.15.28:48598) 2024-02-21T23:07:57.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.075+0000 7f59a7e00640 1 -- 172.21.15.28:0/2262899054 learned_addr learned my addr 172.21.15.28:0/2262899054 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:57.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.075+0000 7f59afe00640 1 --2- 172.21.15.28:0/2262899054 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f59b00c4960 0x7f59b00c6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:57.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.075+0000 7f59b4c00640 1 --2- 172.21.15.28:0/2262899054 >> v2:172.21.15.81:3300/0 conn(0x7f59b01586b0 0x7f59b0158b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:57.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.075+0000 7f59a7e00640 1 -- 172.21.15.28:0/2262899054 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f59b00c4960 msgr2=0x7f59b00c6220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:57.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.075+0000 7f59a7e00640 1 --2- 172.21.15.28:0/2262899054 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f59b00c4960 0x7f59b00c6220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:57.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.075+0000 7f59a7e00640 1 -- 172.21.15.28:0/2262899054 >> v2:172.21.15.81:3300/0 conn(0x7f59b01586b0 msgr2=0x7f59b0158b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:57.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.075+0000 7f59a7e00640 1 --2- 172.21.15.28:0/2262899054 >> v2:172.21.15.81:3300/0 conn(0x7f59b01586b0 0x7f59b0158b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:57.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.075+0000 7f59a7e00640 1 -- 172.21.15.28:0/2262899054 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59b00c1290 con 0x7f59b01524f0 2024-02-21T23:07:57.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.076+0000 7f59a7e00640 1 --2- 172.21.15.28:0/2262899054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59b01524f0 0x7f59b0152930 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f59b0073730 tx=0x7f599802b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d4dc119a24c3a5b server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:57.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.077+0000 7f59af400640 1 -- 172.21.15.28:0/2262899054 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59980317d0 con 0x7f59b01524f0 2024-02-21T23:07:57.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.077+0000 7f59af400640 1 -- 172.21.15.28:0/2262899054 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5998037070 con 0x7f59b01524f0 2024-02-21T23:07:57.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.077+0000 7f59af400640 1 -- 172.21.15.28:0/2262899054 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5998031b90 con 0x7f59b01524f0 2024-02-21T23:07:57.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.077+0000 7f59b6c00640 1 -- 172.21.15.28:0/2262899054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59b01524f0 msgr2=0x7f59b0152930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:57.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.077+0000 7f59b6c00640 1 --2- 172.21.15.28:0/2262899054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59b01524f0 0x7f59b0152930 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f59b0073730 tx=0x7f599802b870 comp rx=0 tx=0).stop 2024-02-21T23:07:57.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.078+0000 7f59b6c00640 1 -- 172.21.15.28:0/2262899054 shutdown_connections 2024-02-21T23:07:57.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.078+0000 7f59b6c00640 1 --2- 172.21.15.28:0/2262899054 >> v2:172.21.15.81:3300/0 conn(0x7f59b01586b0 0x7f59b0158b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:57.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.078+0000 7f59b6c00640 1 --2- 172.21.15.28:0/2262899054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59b01524f0 0x7f59b0152930 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:57.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.078+0000 7f59b6c00640 1 --2- 172.21.15.28:0/2262899054 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f59b00c4960 0x7f59b00c6220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:57.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.078+0000 7f59b6c00640 1 -- 172.21.15.28:0/2262899054 >> 172.21.15.28:0/2262899054 conn(0x7f59b00a85a0 msgr2=0x7f59b00a8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:57.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.078+0000 7f59b6c00640 1 -- 172.21.15.28:0/2262899054 shutdown_connections 2024-02-21T23:07:57.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.078+0000 7f59b6c00640 1 -- 172.21.15.28:0/2262899054 wait complete. 2024-02-21T23:07:57.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.080+0000 7f59b6c00640 1 Processor -- start 2024-02-21T23:07:57.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.080+0000 7f59b6c00640 1 -- start start 2024-02-21T23:07:57.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.081+0000 7f59b6c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59b00c4960 0x7f59b01ead60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:57.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.081+0000 7f59b6c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f59b01524f0 0x7f59b0153220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:57.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.081+0000 7f59b6c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f59b01586b0 0x7f59b014b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:57.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.081+0000 7f59b6c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59b00a5f20 con 0x7f59b00c4960 2024-02-21T23:07:57.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.081+0000 7f59b6c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59b00c1bb0 con 0x7f59b01586b0 2024-02-21T23:07:57.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.081+0000 7f59b6c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59b00a60a0 con 0x7f59b01524f0 2024-02-21T23:07:57.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.081+0000 7f59afe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59b00c4960 0x7f59b01ead60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:57.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.082+0000 7f59afe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59b00c4960 0x7f59b01ead60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48614/0 (socket says 172.21.15.28:48614) 2024-02-21T23:07:57.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.082+0000 7f59afe00640 1 -- 172.21.15.28:0/984462482 learned_addr learned my addr 172.21.15.28:0/984462482 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:57.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.082+0000 7f59a7e00640 1 --2- 172.21.15.28:0/984462482 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f59b01524f0 0x7f59b0153220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:57.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.082+0000 7f59b4c00640 1 --2- 172.21.15.28:0/984462482 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f59b01586b0 0x7f59b014b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:57.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.082+0000 7f59afe00640 1 -- 172.21.15.28:0/984462482 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f59b01524f0 msgr2=0x7f59b0153220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:57.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.082+0000 7f59afe00640 1 --2- 172.21.15.28:0/984462482 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f59b01524f0 0x7f59b0153220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:57.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.082+0000 7f59afe00640 1 -- 172.21.15.28:0/984462482 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f59b01586b0 msgr2=0x7f59b014b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:57.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.082+0000 7f59afe00640 1 --2- 172.21.15.28:0/984462482 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f59b01586b0 0x7f59b014b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:57.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.082+0000 7f59afe00640 1 -- 172.21.15.28:0/984462482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59b01eb2a0 con 0x7f59b00c4960 2024-02-21T23:07:57.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.083+0000 7f59afe00640 1 --2- 172.21.15.28:0/984462482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59b00c4960 0x7f59b01ead60 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f59a000d800 tx=0x7f59a000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:57.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.083+0000 7f59acc00640 1 -- 172.21.15.28:0/984462482 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59a0018020 con 0x7f59b00c4960 2024-02-21T23:07:57.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.083+0000 7f59acc00640 1 -- 172.21.15.28:0/984462482 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59a00096a0 con 0x7f59b00c4960 2024-02-21T23:07:57.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.083+0000 7f59b6c00640 1 -- 172.21.15.28:0/984462482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59b014d0d0 con 0x7f59b00c4960 2024-02-21T23:07:57.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.083+0000 7f59acc00640 1 -- 172.21.15.28:0/984462482 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59a0009a70 con 0x7f59b00c4960 2024-02-21T23:07:57.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.083+0000 7f59b6c00640 1 -- 172.21.15.28:0/984462482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59b014d690 con 0x7f59b00c4960 2024-02-21T23:07:57.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.084+0000 7f59b6c00640 1 -- 172.21.15.28:0/984462482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59b0067f30 con 0x7f59b00c4960 2024-02-21T23:07:57.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.085+0000 7f59acc00640 1 -- 172.21.15.28:0/984462482 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f59a001b020 con 0x7f59b00c4960 2024-02-21T23:07:57.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.087+0000 7f59acc00640 1 --2- 172.21.15.28:0/984462482 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f598c06b500 0x7f598c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:57.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.087+0000 7f59acc00640 1 -- 172.21.15.28:0/984462482 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f59a008a5c0 con 0x7f59b00c4960 2024-02-21T23:07:57.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.087+0000 7f59a7e00640 1 --2- 172.21.15.28:0/984462482 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f598c06b500 0x7f598c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:57.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.088+0000 7f59a7e00640 1 --2- 172.21.15.28:0/984462482 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f598c06b500 0x7f598c06d9c0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f5998036000 tx=0x7f599803d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:57.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.092+0000 7f59acc00640 1 -- 172.21.15.28:0/984462482 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f59a0022030 con 0x7f59b00c4960 2024-02-21T23:07:57.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.292+0000 7f59b6c00640 1 -- 172.21.15.28:0/984462482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f59b0153760 con 0x7f59b00c4960 2024-02-21T23:07:57.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.293+0000 7f59acc00640 1 -- 172.21.15.28:0/984462482 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v111) v1 ==== 100+0+11 (secure 0 0 0) 0x7f59a0059250 con 0x7f59b00c4960 2024-02-21T23:07:57.282 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-02-21T23:07:57.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.296+0000 7f59b6c00640 1 -- 172.21.15.28:0/984462482 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f598c06b500 msgr2=0x7f598c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:57.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.296+0000 7f59b6c00640 1 --2- 172.21.15.28:0/984462482 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f598c06b500 0x7f598c06d9c0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f5998036000 tx=0x7f599803d040 comp rx=0 tx=0).stop 2024-02-21T23:07:57.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.296+0000 7f59b6c00640 1 -- 172.21.15.28:0/984462482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59b00c4960 msgr2=0x7f59b01ead60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:57.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.296+0000 7f59b6c00640 1 --2- 172.21.15.28:0/984462482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59b00c4960 0x7f59b01ead60 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f59a000d800 tx=0x7f59a000dcc0 comp rx=0 tx=0).stop 2024-02-21T23:07:57.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.296+0000 7f59b6c00640 1 -- 172.21.15.28:0/984462482 shutdown_connections 2024-02-21T23:07:57.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.296+0000 7f59b6c00640 1 --2- 172.21.15.28:0/984462482 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f598c06b500 0x7f598c06d9c0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:57.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.296+0000 7f59b6c00640 1 --2- 172.21.15.28:0/984462482 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f59b01524f0 0x7f59b0153220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:57.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.296+0000 7f59b6c00640 1 --2- 172.21.15.28:0/984462482 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f59b01586b0 0x7f59b014b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:57.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.296+0000 7f59b6c00640 1 --2- 172.21.15.28:0/984462482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59b00c4960 0x7f59b01ead60 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:57.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.296+0000 7f59b6c00640 1 -- 172.21.15.28:0/984462482 >> 172.21.15.28:0/984462482 conn(0x7f59b00a85a0 msgr2=0x7f59b006ee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:57.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.296+0000 7f59b6c00640 1 -- 172.21.15.28:0/984462482 shutdown_connections 2024-02-21T23:07:57.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.297+0000 7f59b6c00640 1 -- 172.21.15.28:0/984462482 wait complete. 2024-02-21T23:07:57.304 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:07:57.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.534+0000 7fc2d4c00640 1 Processor -- start 2024-02-21T23:07:57.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.534+0000 7fc2d4c00640 1 -- start start 2024-02-21T23:07:57.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.536+0000 7fc2d4c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc2d0150450 0x7fc2d00a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:57.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.536+0000 7fc2d4c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fc2d006ee40 0x7fc2d01536f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:57.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.536+0000 7fc2d4c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2d00c2760 con 0x7fc2d0156af0 2024-02-21T23:07:57.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.536+0000 7fc2d4c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2d00c22c0 con 0x7fc2d0150450 2024-02-21T23:07:57.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.536+0000 7fc2d4c00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2d0156660 con 0x7fc2d006ee40 2024-02-21T23:07:57.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.536+0000 7fc2cf400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc2d0150450 0x7fc2d00a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:57.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.536+0000 7fc2cf400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc2d0150450 0x7fc2d00a5fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48630/0 (socket says 172.21.15.28:48630) 2024-02-21T23:07:57.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.536+0000 7fc2cf400640 1 -- 172.21.15.28:0/3163182638 learned_addr learned my addr 172.21.15.28:0/3163182638 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:57.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.536+0000 7fc2cfe00640 1 --1- 172.21.15.28:0/3163182638 >> v1:172.21.15.28:6789/0 conn(0x7fc2d0156af0 0x7fc2d006dff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60730/0 (socket says 172.21.15.28:60730) 2024-02-21T23:07:57.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.537+0000 7fc2cea00640 1 --2- 172.21.15.28:0/3163182638 >> v2:172.21.15.81:3300/0 conn(0x7fc2d006ee40 0x7fc2d01536f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:57.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.537+0000 7fc2cf400640 1 -- 172.21.15.28:0/3163182638 >> v2:172.21.15.81:3300/0 conn(0x7fc2d006ee40 msgr2=0x7fc2d01536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:57.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.537+0000 7fc2cf400640 1 --2- 172.21.15.28:0/3163182638 >> v2:172.21.15.81:3300/0 conn(0x7fc2d006ee40 0x7fc2d01536f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:57.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.537+0000 7fc2cf400640 1 -- 172.21.15.28:0/3163182638 >> v1:172.21.15.28:6789/0 conn(0x7fc2d0156af0 legacy=0x7fc2d006dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:57.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.537+0000 7fc2cf400640 1 -- 172.21.15.28:0/3163182638 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2d0153c30 con 0x7fc2d0150450 2024-02-21T23:07:57.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.538+0000 7fc2cf400640 1 --2- 172.21.15.28:0/3163182638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2d0150450 0x7fc2d00a5fd0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7fc2d006bcc0 tx=0x7fc2c002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d211ca55d3ce1d server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:57.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.538+0000 7fc2ce000640 1 -- 172.21.15.28:0/3163182638 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2c0034070 con 0x7fc2d0150450 2024-02-21T23:07:57.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.538+0000 7fc2ce000640 1 -- 172.21.15.28:0/3163182638 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2c0030860 con 0x7fc2d0150450 2024-02-21T23:07:57.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.538+0000 7fc2ce000640 1 -- 172.21.15.28:0/3163182638 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2c0030c30 con 0x7fc2d0150450 2024-02-21T23:07:57.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.539+0000 7fc2d4c00640 1 -- 172.21.15.28:0/3163182638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2d0150450 msgr2=0x7fc2d00a5fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:57.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.539+0000 7fc2d4c00640 1 --2- 172.21.15.28:0/3163182638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2d0150450 0x7fc2d00a5fd0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7fc2d006bcc0 tx=0x7fc2c002b9a0 comp rx=0 tx=0).stop 2024-02-21T23:07:57.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.539+0000 7fc2d4c00640 1 -- 172.21.15.28:0/3163182638 shutdown_connections 2024-02-21T23:07:57.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.539+0000 7fc2d4c00640 1 --2- 172.21.15.28:0/3163182638 >> v2:172.21.15.81:3300/0 conn(0x7fc2d006ee40 0x7fc2d01536f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:57.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.539+0000 7fc2d4c00640 1 --2- 172.21.15.28:0/3163182638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2d0150450 0x7fc2d00a5fd0 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:57.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.539+0000 7fc2d4c00640 1 -- 172.21.15.28:0/3163182638 >> 172.21.15.28:0/3163182638 conn(0x7fc2d006b630 msgr2=0x7fc2d00c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:57.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.539+0000 7fc2d4c00640 1 -- 172.21.15.28:0/3163182638 shutdown_connections 2024-02-21T23:07:57.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.539+0000 7fc2d4c00640 1 -- 172.21.15.28:0/3163182638 wait complete. 2024-02-21T23:07:57.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.542+0000 7fc2d4c00640 1 Processor -- start 2024-02-21T23:07:57.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.542+0000 7fc2d4c00640 1 -- start start 2024-02-21T23:07:57.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.543+0000 7fc2d4c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc2d006ee40 0x7fc2d01ebf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:57.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.543+0000 7fc2d4c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2d0150450 0x7fc2d01eea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:57.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.543+0000 7fc2d4c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2d0156af0 0x7fc2d00c5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:57.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.543+0000 7fc2d4c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d00c2cb0 con 0x7fc2d0156af0 2024-02-21T23:07:57.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.543+0000 7fc2d4c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d00c29d0 con 0x7fc2d0150450 2024-02-21T23:07:57.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.543+0000 7fc2d4c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d006e700 con 0x7fc2d006ee40 2024-02-21T23:07:57.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.544+0000 7fc2cfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2d0156af0 0x7fc2d00c5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:57.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.544+0000 7fc2cfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2d0156af0 0x7fc2d00c5870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48646/0 (socket says 172.21.15.28:48646) 2024-02-21T23:07:57.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.544+0000 7fc2cfe00640 1 -- 172.21.15.28:0/125601819 learned_addr learned my addr 172.21.15.28:0/125601819 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:57.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.544+0000 7fc2cea00640 1 --2- 172.21.15.28:0/125601819 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2d0150450 0x7fc2d01eea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:57.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.544+0000 7fc2cf400640 1 --2- 172.21.15.28:0/125601819 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc2d006ee40 0x7fc2d01ebf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:57.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.544+0000 7fc2cfe00640 1 -- 172.21.15.28:0/125601819 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc2d006ee40 msgr2=0x7fc2d01ebf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:57.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.544+0000 7fc2cfe00640 1 --2- 172.21.15.28:0/125601819 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc2d006ee40 0x7fc2d01ebf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:57.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.544+0000 7fc2cfe00640 1 -- 172.21.15.28:0/125601819 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2d0150450 msgr2=0x7fc2d01eea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:57.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.544+0000 7fc2cfe00640 1 --2- 172.21.15.28:0/125601819 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2d0150450 0x7fc2d01eea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:57.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.544+0000 7fc2cfe00640 1 -- 172.21.15.28:0/125601819 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2d01eabb0 con 0x7fc2d0156af0 2024-02-21T23:07:57.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.544+0000 7fc2cf400640 1 --2- 172.21.15.28:0/125601819 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc2d006ee40 0x7fc2d01ebf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:07:57.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.544+0000 7fc2cfe00640 1 --2- 172.21.15.28:0/125601819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2d0156af0 0x7fc2d00c5870 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7fc2c800c7b0 tx=0x7fc2c800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:57.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.545+0000 7fc2c7400640 1 -- 172.21.15.28:0/125601819 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2c8017070 con 0x7fc2d0156af0 2024-02-21T23:07:57.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.545+0000 7fc2c7400640 1 -- 172.21.15.28:0/125601819 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2c800a640 con 0x7fc2d0156af0 2024-02-21T23:07:57.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.545+0000 7fc2c7400640 1 -- 172.21.15.28:0/125601819 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2c800a9f0 con 0x7fc2d0156af0 2024-02-21T23:07:57.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.545+0000 7fc2d4c00640 1 -- 172.21.15.28:0/125601819 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2d0152860 con 0x7fc2d0156af0 2024-02-21T23:07:57.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.545+0000 7fc2d4c00640 1 -- 172.21.15.28:0/125601819 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2d00c5db0 con 0x7fc2d0156af0 2024-02-21T23:07:57.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.546+0000 7fc2d4c00640 1 -- 172.21.15.28:0/125601819 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2d00a8c00 con 0x7fc2d0156af0 2024-02-21T23:07:57.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.547+0000 7fc2c7400640 1 -- 172.21.15.28:0/125601819 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc2c801a070 con 0x7fc2d0156af0 2024-02-21T23:07:57.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.548+0000 7fc2c7400640 1 --2- 172.21.15.28:0/125601819 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc2ac06b550 0x7fc2ac06da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:57.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.548+0000 7fc2c7400640 1 -- 172.21.15.28:0/125601819 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fc2c8089850 con 0x7fc2d0156af0 2024-02-21T23:07:57.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.549+0000 7fc2cf400640 1 --2- 172.21.15.28:0/125601819 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc2ac06b550 0x7fc2ac06da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:57.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.550+0000 7fc2cf400640 1 --2- 172.21.15.28:0/125601819 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc2ac06b550 0x7fc2ac06da10 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fc2c0036000 tx=0x7fc2c003b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:57.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.553+0000 7fc2c7400640 1 -- 172.21.15.28:0/125601819 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc2d00a8c00 con 0x7fc2d0156af0 2024-02-21T23:07:57.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.794+0000 7fc2d4c00640 1 -- 172.21.15.28:0/125601819 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc2d004f290 con 0x7fc2d0156af0 2024-02-21T23:07:57.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.796+0000 7fc2c7400640 1 -- 172.21.15.28:0/125601819 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+4949 (secure 0 0 0) 0x7fc2c8057f80 con 0x7fc2d0156af0 2024-02-21T23:07:57.785 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:07:57.785 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":112,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5843,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3059988835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3059988835},{"type":"v1","addr":"172.21.15.28:6835","nonce":3059988835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":112},{"gid":7446,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/1119726035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1119726035},{"type":"v1","addr":"172.21.15.28:6833","nonce":1119726035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":111,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-02-21T23:07:29.533547+0000","modified":"2024-02-21T23:07:49.218076+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5729},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5729":{"gid":5729,"name":"d","rank":0,"incarnation":104,"state":"up:active","state_seq":8,"addr":"172.21.15.81:6835/1754860807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1754860807},{"type":"v1","addr":"172.21.15.81:6835","nonce":1754860807}]},"join_fscid":8,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5732":{"gid":5732,"name":"b","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":7,"addr":"172.21.15.81:6837/3963422563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3963422563},{"type":"v1","addr":"172.21.15.81:6837","nonce":3963422563}]},"join_fscid":8,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-02-21T23:07:57.785 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 112 2024-02-21T23:07:57.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.798+0000 7fc2d4c00640 1 -- 172.21.15.28:0/125601819 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc2ac06b550 msgr2=0x7fc2ac06da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:57.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.798+0000 7fc2d4c00640 1 --2- 172.21.15.28:0/125601819 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc2ac06b550 0x7fc2ac06da10 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fc2c0036000 tx=0x7fc2c003b040 comp rx=0 tx=0).stop 2024-02-21T23:07:57.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.798+0000 7fc2d4c00640 1 -- 172.21.15.28:0/125601819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2d0156af0 msgr2=0x7fc2d00c5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:57.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.798+0000 7fc2d4c00640 1 --2- 172.21.15.28:0/125601819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2d0156af0 0x7fc2d00c5870 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7fc2c800c7b0 tx=0x7fc2c800cc70 comp rx=0 tx=0).stop 2024-02-21T23:07:57.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.798+0000 7fc2d4c00640 1 -- 172.21.15.28:0/125601819 shutdown_connections 2024-02-21T23:07:57.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.798+0000 7fc2d4c00640 1 --2- 172.21.15.28:0/125601819 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc2ac06b550 0x7fc2ac06da10 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:57.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.798+0000 7fc2d4c00640 1 --2- 172.21.15.28:0/125601819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2d0156af0 0x7fc2d00c5870 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:57.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.798+0000 7fc2d4c00640 1 --2- 172.21.15.28:0/125601819 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2d0150450 0x7fc2d01eea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:57.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.798+0000 7fc2d4c00640 1 --2- 172.21.15.28:0/125601819 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc2d006ee40 0x7fc2d01ebf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:57.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.799+0000 7fc2d4c00640 1 -- 172.21.15.28:0/125601819 >> 172.21.15.28:0/125601819 conn(0x7fc2d006b630 msgr2=0x7fc2d014e480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:57.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.799+0000 7fc2d4c00640 1 -- 172.21.15.28:0/125601819 shutdown_connections 2024-02-21T23:07:57.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:57.799+0000 7fc2d4c00640 1 -- 172.21.15.28:0/125601819 wait complete. 2024-02-21T23:07:57.806 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:07:58.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.033+0000 7fcd28000640 1 Processor -- start 2024-02-21T23:07:58.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.033+0000 7fcd28000640 1 -- start start 2024-02-21T23:07:58.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.034+0000 7fcd28000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcd201559c0 0x7fcd200c76f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:58.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.034+0000 7fcd28000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd200c6f30 con 0x7fcd201526b0 2024-02-21T23:07:58.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.034+0000 7fcd28000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd200c6450 con 0x7fcd200c7c30 2024-02-21T23:07:58.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.034+0000 7fcd28000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd200c61b0 con 0x7fcd201559c0 2024-02-21T23:07:58.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.034+0000 7fcd25600640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fcd201526b0 0x7fcd200ca940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60746/0 (socket says 172.21.15.28:60746) 2024-02-21T23:07:58.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.035+0000 7fcd25600640 1 -- 172.21.15.28:0/3427464235 learned_addr learned my addr 172.21.15.28:0/3427464235 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:07:58.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.035+0000 7fcd1fe00640 1 -- 172.21.15.28:0/3427464235 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2250326131 0 0) 0x7fcd200c6f30 con 0x7fcd201526b0 2024-02-21T23:07:58.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.035+0000 7fcd1fe00640 1 -- 172.21.15.28:0/3427464235 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd08003520 con 0x7fcd201526b0 2024-02-21T23:07:58.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.035+0000 7fcd24c00640 1 --2- 172.21.15.28:0/3427464235 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcd201559c0 0x7fcd200c76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:58.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.035+0000 7fcd1fe00640 1 -- 172.21.15.28:0/3427464235 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3465043884 0 0) 0x7fcd08003520 con 0x7fcd201526b0 2024-02-21T23:07:58.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.035+0000 7fcd1fe00640 1 -- 172.21.15.28:0/3427464235 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcd201559c0 msgr2=0x7fcd200c76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:58.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.035+0000 7fcd1fe00640 1 --2- 172.21.15.28:0/3427464235 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcd201559c0 0x7fcd200c76f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:58.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.035+0000 7fcd1fe00640 1 -- 172.21.15.28:0/3427464235 >> v1:172.21.15.81:6789/0 conn(0x7fcd200c7c30 legacy=0x7fcd2015f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:58.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.035+0000 7fcd1fe00640 1 -- 172.21.15.28:0/3427464235 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd200c6710 con 0x7fcd201526b0 2024-02-21T23:07:58.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.035+0000 7fcd1fe00640 1 -- 172.21.15.28:0/3427464235 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fcd14001670 con 0x7fcd201526b0 2024-02-21T23:07:58.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.036+0000 7fcd1fe00640 1 -- 172.21.15.28:0/3427464235 >> v1:172.21.15.28:6789/0 conn(0x7fcd201526b0 legacy=0x7fcd200ca940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:58.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.036+0000 7fcd1fe00640 1 --2- 172.21.15.28:0/3427464235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd080039e0 0x7fcd08003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:58.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.036+0000 7fcd1fe00640 1 --2- 172.21.15.28:0/3427464235 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcd08004800 0x7fcd08004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:58.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.036+0000 7fcd1fe00640 1 --2- 172.21.15.28:0/3427464235 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcd201559c0 0x7fcd08005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:58.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.036+0000 7fcd1fe00640 1 -- 172.21.15.28:0/3427464235 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd0800bac0 con 0x7fcd080039e0 2024-02-21T23:07:58.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.036+0000 7fcd1fe00640 1 -- 172.21.15.28:0/3427464235 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd0800bc70 con 0x7fcd08004800 2024-02-21T23:07:58.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.036+0000 7fcd1fe00640 1 -- 172.21.15.28:0/3427464235 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd0800be70 con 0x7fcd201559c0 2024-02-21T23:07:58.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.036+0000 7fcd25600640 1 --2- 172.21.15.28:0/3427464235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd080039e0 0x7fcd08003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:58.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.036+0000 7fcd24c00640 1 --2- 172.21.15.28:0/3427464235 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcd08004800 0x7fcd08004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:58.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.036+0000 7fcd25600640 1 -- 172.21.15.28:0/3427464235 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcd201559c0 msgr2=0x7fcd08005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:58.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.036+0000 7fcd25600640 1 --2- 172.21.15.28:0/3427464235 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcd201559c0 0x7fcd08005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:58.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.036+0000 7fcd25600640 1 -- 172.21.15.28:0/3427464235 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcd08004800 msgr2=0x7fcd08004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:58.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.036+0000 7fcd25600640 1 --2- 172.21.15.28:0/3427464235 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcd08004800 0x7fcd08004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:58.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.036+0000 7fcd25600640 1 -- 172.21.15.28:0/3427464235 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd08003520 con 0x7fcd080039e0 2024-02-21T23:07:58.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.036+0000 7fcd24c00640 1 --2- 172.21.15.28:0/3427464235 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcd08004800 0x7fcd08004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:07:58.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.037+0000 7fcd25600640 1 --2- 172.21.15.28:0/3427464235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd080039e0 0x7fcd08003e00 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7fcd1400c240 tx=0x7fcd08005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca1a911e83f3ee1d server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:58.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.038+0000 7fcd1fe00640 1 -- 172.21.15.28:0/3427464235 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd14007d10 con 0x7fcd080039e0 2024-02-21T23:07:58.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.038+0000 7fcd1fe00640 1 -- 172.21.15.28:0/3427464235 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd14030070 con 0x7fcd080039e0 2024-02-21T23:07:58.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.038+0000 7fcd1fe00640 1 -- 172.21.15.28:0/3427464235 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd1402cd80 con 0x7fcd080039e0 2024-02-21T23:07:58.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.038+0000 7fcd28000640 1 -- 172.21.15.28:0/3427464235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd080039e0 msgr2=0x7fcd08003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:58.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.038+0000 7fcd28000640 1 --2- 172.21.15.28:0/3427464235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd080039e0 0x7fcd08003e00 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7fcd1400c240 tx=0x7fcd08005830 comp rx=0 tx=0).stop 2024-02-21T23:07:58.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.038+0000 7fcd26000640 1 -- 172.21.15.28:0/3427464235 reap_dead start 2024-02-21T23:07:58.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.038+0000 7fcd28000640 1 -- 172.21.15.28:0/3427464235 shutdown_connections 2024-02-21T23:07:58.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.038+0000 7fcd28000640 1 -- 172.21.15.28:0/3427464235 >> 172.21.15.28:0/3427464235 conn(0x7fcd2006d3d0 msgr2=0x7fcd2008afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:58.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.039+0000 7fcd28000640 1 -- 172.21.15.28:0/3427464235 shutdown_connections 2024-02-21T23:07:58.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.039+0000 7fcd28000640 1 -- 172.21.15.28:0/3427464235 wait complete. 2024-02-21T23:07:58.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.040+0000 7fcd28000640 1 Processor -- start 2024-02-21T23:07:58.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.041+0000 7fcd28000640 1 -- start start 2024-02-21T23:07:58.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.041+0000 7fcd28000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcd080039e0 0x7fcd2014cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:58.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.042+0000 7fcd28000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcd20156620 0x7fcd201ee3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:58.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.042+0000 7fcd28000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd201ee900 0x7fcd200c8560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:58.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.042+0000 7fcd28000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd200c5c90 con 0x7fcd201ee900 2024-02-21T23:07:58.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.042+0000 7fcd28000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd200c71d0 con 0x7fcd20156620 2024-02-21T23:07:58.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.042+0000 7fcd28000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd200c4e70 con 0x7fcd080039e0 2024-02-21T23:07:58.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.042+0000 7fcd26000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd201ee900 0x7fcd200c8560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:58.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.042+0000 7fcd26000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd201ee900 0x7fcd200c8560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48674/0 (socket says 172.21.15.28:48674) 2024-02-21T23:07:58.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.042+0000 7fcd26000640 1 -- 172.21.15.28:0/713634837 learned_addr learned my addr 172.21.15.28:0/713634837 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:58.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.042+0000 7fcd25600640 1 --2- 172.21.15.28:0/713634837 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcd080039e0 0x7fcd2014cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:58.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.042+0000 7fcd24c00640 1 --2- 172.21.15.28:0/713634837 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcd20156620 0x7fcd201ee3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:58.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.042+0000 7fcd26000640 1 -- 172.21.15.28:0/713634837 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcd080039e0 msgr2=0x7fcd2014cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:58.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.042+0000 7fcd26000640 1 --2- 172.21.15.28:0/713634837 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcd080039e0 0x7fcd2014cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:58.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.042+0000 7fcd26000640 1 -- 172.21.15.28:0/713634837 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcd20156620 msgr2=0x7fcd201ee3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:58.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.042+0000 7fcd26000640 1 --2- 172.21.15.28:0/713634837 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcd20156620 0x7fcd201ee3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:58.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.042+0000 7fcd26000640 1 -- 172.21.15.28:0/713634837 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd201e1c40 con 0x7fcd201ee900 2024-02-21T23:07:58.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.042+0000 7fcd24c00640 1 --2- 172.21.15.28:0/713634837 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcd20156620 0x7fcd201ee3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:07:58.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.043+0000 7fcd26000640 1 --2- 172.21.15.28:0/713634837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd201ee900 0x7fcd200c8560 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7fcd200c24b0 tx=0x7fcd1000c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:58.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.043+0000 7fcd1d600640 1 -- 172.21.15.28:0/713634837 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd10004220 con 0x7fcd201ee900 2024-02-21T23:07:58.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.043+0000 7fcd1d600640 1 -- 172.21.15.28:0/713634837 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd100043e0 con 0x7fcd201ee900 2024-02-21T23:07:58.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.043+0000 7fcd28000640 1 -- 172.21.15.28:0/713634837 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd201e1f20 con 0x7fcd201ee900 2024-02-21T23:07:58.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.043+0000 7fcd1d600640 1 -- 172.21.15.28:0/713634837 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd10007310 con 0x7fcd201ee900 2024-02-21T23:07:58.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.043+0000 7fcd28000640 1 -- 172.21.15.28:0/713634837 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd201edea0 con 0x7fcd201ee900 2024-02-21T23:07:58.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.044+0000 7fcd28000640 1 -- 172.21.15.28:0/713634837 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd201ee0a0 con 0x7fcd201ee900 2024-02-21T23:07:58.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.045+0000 7fcd1d600640 1 -- 172.21.15.28:0/713634837 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcd100074b0 con 0x7fcd201ee900 2024-02-21T23:07:58.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.047+0000 7fcd1d600640 1 --2- 172.21.15.28:0/713634837 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcd0406b4d0 0x7fcd0406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:58.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.047+0000 7fcd1d600640 1 -- 172.21.15.28:0/713634837 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fcd10089560 con 0x7fcd201ee900 2024-02-21T23:07:58.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.048+0000 7fcd25600640 1 --2- 172.21.15.28:0/713634837 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcd0406b4d0 0x7fcd0406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:58.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.049+0000 7fcd25600640 1 --2- 172.21.15.28:0/713634837 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcd0406b4d0 0x7fcd0406d990 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fcd14034000 tx=0x7fcd14009030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:58.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.052+0000 7fcd1d600640 1 -- 172.21.15.28:0/713634837 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fcd201ee0a0 con 0x7fcd201ee900 2024-02-21T23:07:58.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.253+0000 7fcd28000640 1 -- 172.21.15.28:0/713634837 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcd20067f30 con 0x7fcd201ee900 2024-02-21T23:07:58.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.255+0000 7fcd1d600640 1 -- 172.21.15.28:0/713634837 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v111) v1 ==== 74+0+17021 (secure 0 0 0) 0x7fcd100581f0 con 0x7fcd201ee900 2024-02-21T23:07:58.245 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:07:58.245 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":111,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:07:41.139084+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:07:27.477155+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-02-21T23:07:28.549456+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6835/1259832430":"2024-02-22T23:07:41.139002+0000","172.21.15.28:6834/1259832430":"2024-02-22T23:07:41.139002+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:07:58.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.258+0000 7fcd28000640 1 -- 172.21.15.28:0/713634837 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcd0406b4d0 msgr2=0x7fcd0406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:58.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.258+0000 7fcd28000640 1 --2- 172.21.15.28:0/713634837 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcd0406b4d0 0x7fcd0406d990 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fcd14034000 tx=0x7fcd14009030 comp rx=0 tx=0).stop 2024-02-21T23:07:58.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.258+0000 7fcd28000640 1 -- 172.21.15.28:0/713634837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd201ee900 msgr2=0x7fcd200c8560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:58.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.258+0000 7fcd28000640 1 --2- 172.21.15.28:0/713634837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd201ee900 0x7fcd200c8560 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7fcd200c24b0 tx=0x7fcd1000c820 comp rx=0 tx=0).stop 2024-02-21T23:07:58.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.258+0000 7fcd28000640 1 -- 172.21.15.28:0/713634837 shutdown_connections 2024-02-21T23:07:58.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.258+0000 7fcd28000640 1 --2- 172.21.15.28:0/713634837 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcd0406b4d0 0x7fcd0406d990 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:58.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.258+0000 7fcd28000640 1 --2- 172.21.15.28:0/713634837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd201ee900 0x7fcd200c8560 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:58.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.258+0000 7fcd28000640 1 --2- 172.21.15.28:0/713634837 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcd20156620 0x7fcd201ee3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:58.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.258+0000 7fcd28000640 1 --2- 172.21.15.28:0/713634837 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcd080039e0 0x7fcd2014cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:58.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.258+0000 7fcd28000640 1 -- 172.21.15.28:0/713634837 >> 172.21.15.28:0/713634837 conn(0x7fcd2006d3d0 msgr2=0x7fcd20154030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:58.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.258+0000 7fcd28000640 1 -- 172.21.15.28:0/713634837 shutdown_connections 2024-02-21T23:07:58.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.258+0000 7fcd28000640 1 -- 172.21.15.28:0/713634837 wait complete. 2024-02-21T23:07:58.267 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-02-21T23:07:58.268 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:07:58.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.497+0000 7f66f2a00640 1 Processor -- start 2024-02-21T23:07:58.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.498+0000 7f66f2a00640 1 -- start start 2024-02-21T23:07:58.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.499+0000 7f66f2a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f66ec06dbf0 0x7f66ec158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:58.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.499+0000 7f66f2a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f66ec155960 0x7f66ec159000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:58.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.499+0000 7f66f2a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66ec0c74c0 con 0x7f66ec0aff90 2024-02-21T23:07:58.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.499+0000 7f66f2a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66ec0c7270 con 0x7f66ec06dbf0 2024-02-21T23:07:58.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.499+0000 7f66f2a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66ec158630 con 0x7f66ec155960 2024-02-21T23:07:58.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.500+0000 7f66f1600640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f66ec0aff90 0x7f66ec0c51e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60748/0 (socket says 172.21.15.28:60748) 2024-02-21T23:07:58.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.500+0000 7f66f1600640 1 -- 172.21.15.28:0/588236701 learned_addr learned my addr 172.21.15.28:0/588236701 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:07:58.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.500+0000 7f66f2000640 1 --2- 172.21.15.28:0/588236701 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f66ec155960 0x7f66ec159000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:58.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.500+0000 7f66f0c00640 1 --2- 172.21.15.28:0/588236701 >> v2:172.21.15.81:3300/0 conn(0x7f66ec06dbf0 0x7f66ec158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:58.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.500+0000 7f66ebe00640 1 -- 172.21.15.28:0/588236701 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2788771868 0 0) 0x7f66ec0c74c0 con 0x7f66ec0aff90 2024-02-21T23:07:58.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.500+0000 7f66ebe00640 1 -- 172.21.15.28:0/588236701 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66d4003520 con 0x7f66ec0aff90 2024-02-21T23:07:58.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.501+0000 7f66ebe00640 1 -- 172.21.15.28:0/588236701 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1152908091 0 0) 0x7f66d4003520 con 0x7f66ec0aff90 2024-02-21T23:07:58.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.501+0000 7f66ebe00640 1 -- 172.21.15.28:0/588236701 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f66ec155960 msgr2=0x7f66ec159000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:58.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.501+0000 7f66ebe00640 1 --2- 172.21.15.28:0/588236701 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f66ec155960 0x7f66ec159000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:58.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.501+0000 7f66ebe00640 1 -- 172.21.15.28:0/588236701 >> v2:172.21.15.81:3300/0 conn(0x7f66ec06dbf0 msgr2=0x7f66ec158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:58.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.501+0000 7f66ebe00640 1 --2- 172.21.15.28:0/588236701 >> v2:172.21.15.81:3300/0 conn(0x7f66ec06dbf0 0x7f66ec158ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:58.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.501+0000 7f66ebe00640 1 -- 172.21.15.28:0/588236701 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66ec0c77a0 con 0x7f66ec0aff90 2024-02-21T23:07:58.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.501+0000 7f66ebe00640 1 -- 172.21.15.28:0/588236701 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f66e0001670 con 0x7f66ec0aff90 2024-02-21T23:07:58.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.501+0000 7f66ebe00640 1 -- 172.21.15.28:0/588236701 >> v1:172.21.15.28:6789/0 conn(0x7f66ec0aff90 legacy=0x7f66ec0c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:58.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.501+0000 7f66ebe00640 1 --2- 172.21.15.28:0/588236701 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66d40039b0 0x7f66d4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:58.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.501+0000 7f66ebe00640 1 --2- 172.21.15.28:0/588236701 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f66ec155960 0x7f66d4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:58.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.501+0000 7f66ebe00640 1 --2- 172.21.15.28:0/588236701 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f66d4004d40 0x7f66d40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:58.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.501+0000 7f66ebe00640 1 -- 172.21.15.28:0/588236701 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66ec0c7270 con 0x7f66d40039b0 2024-02-21T23:07:58.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.501+0000 7f66ebe00640 1 -- 172.21.15.28:0/588236701 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66ec158630 con 0x7f66d4004d40 2024-02-21T23:07:58.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.501+0000 7f66ebe00640 1 -- 172.21.15.28:0/588236701 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66d400b9c0 con 0x7f66ec155960 2024-02-21T23:07:58.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.501+0000 7f66f1600640 1 --2- 172.21.15.28:0/588236701 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66d40039b0 0x7f66d4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:58.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.502+0000 7f66f0c00640 1 --2- 172.21.15.28:0/588236701 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f66ec155960 0x7f66d4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:58.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.502+0000 7f66f2000640 1 --2- 172.21.15.28:0/588236701 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f66d4004d40 0x7f66d40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:58.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.502+0000 7f66f1600640 1 -- 172.21.15.28:0/588236701 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f66ec155960 msgr2=0x7f66d4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:58.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.502+0000 7f66f1600640 1 --2- 172.21.15.28:0/588236701 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f66ec155960 0x7f66d4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:58.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.502+0000 7f66f1600640 1 -- 172.21.15.28:0/588236701 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f66d4004d40 msgr2=0x7f66d40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:58.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.502+0000 7f66f1600640 1 --2- 172.21.15.28:0/588236701 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f66d4004d40 0x7f66d40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:58.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.502+0000 7f66f1600640 1 -- 172.21.15.28:0/588236701 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66d4003520 con 0x7f66d40039b0 2024-02-21T23:07:58.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.503+0000 7f66f1600640 1 --2- 172.21.15.28:0/588236701 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66d40039b0 0x7f66d4003dd0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f66e000c190 tx=0x7f66d4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5118c2851ce0421 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:58.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.503+0000 7f66ebe00640 1 -- 172.21.15.28:0/588236701 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e0007c80 con 0x7f66d40039b0 2024-02-21T23:07:58.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.503+0000 7f66ebe00640 1 -- 172.21.15.28:0/588236701 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66e0030070 con 0x7f66d40039b0 2024-02-21T23:07:58.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.503+0000 7f66ebe00640 1 -- 172.21.15.28:0/588236701 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e0005d10 con 0x7f66d40039b0 2024-02-21T23:07:58.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.504+0000 7f66f2a00640 1 -- 172.21.15.28:0/588236701 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66d40039b0 msgr2=0x7f66d4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:58.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.504+0000 7f66f2a00640 1 --2- 172.21.15.28:0/588236701 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66d40039b0 0x7f66d4003dd0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f66e000c190 tx=0x7f66d4005830 comp rx=0 tx=0).stop 2024-02-21T23:07:58.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.504+0000 7f66f2000640 1 -- 172.21.15.28:0/588236701 reap_dead start 2024-02-21T23:07:58.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.504+0000 7f66f2a00640 1 -- 172.21.15.28:0/588236701 shutdown_connections 2024-02-21T23:07:58.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.504+0000 7f66f2a00640 1 -- 172.21.15.28:0/588236701 >> 172.21.15.28:0/588236701 conn(0x7f66ec0ac370 msgr2=0x7f66ec0ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:58.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.504+0000 7f66f2a00640 1 -- 172.21.15.28:0/588236701 shutdown_connections 2024-02-21T23:07:58.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.504+0000 7f66f2a00640 1 -- 172.21.15.28:0/588236701 wait complete. 2024-02-21T23:07:58.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.506+0000 7f66f2a00640 1 Processor -- start 2024-02-21T23:07:58.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.506+0000 7f66f2a00640 1 -- start start 2024-02-21T23:07:58.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.507+0000 7f66f2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66d40039b0 0x7f66ec0c1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:58.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.507+0000 7f66f2a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f66ec0c4470 0x7f66ec1ed900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:58.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.507+0000 7f66f2a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f66ec0ca7f0 0x7f66ec0cac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:58.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.507+0000 7f66f2a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66ec0c58f0 con 0x7f66d40039b0 2024-02-21T23:07:58.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.507+0000 7f66f2a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66ec0be0c0 con 0x7f66ec0c4470 2024-02-21T23:07:58.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.507+0000 7f66f2a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66ec0c5a70 con 0x7f66ec0ca7f0 2024-02-21T23:07:58.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.507+0000 7f66f1600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66d40039b0 0x7f66ec0c1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:58.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.507+0000 7f66f1600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66d40039b0 0x7f66ec0c1320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48680/0 (socket says 172.21.15.28:48680) 2024-02-21T23:07:58.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.507+0000 7f66f1600640 1 -- 172.21.15.28:0/3595724902 learned_addr learned my addr 172.21.15.28:0/3595724902 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:58.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.508+0000 7f66f2000640 1 --2- 172.21.15.28:0/3595724902 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f66ec0ca7f0 0x7f66ec0cac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:58.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.508+0000 7f66f0c00640 1 --2- 172.21.15.28:0/3595724902 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f66ec0c4470 0x7f66ec1ed900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:58.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.508+0000 7f66f1600640 1 -- 172.21.15.28:0/3595724902 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f66ec0ca7f0 msgr2=0x7f66ec0cac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:58.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.508+0000 7f66f1600640 1 --2- 172.21.15.28:0/3595724902 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f66ec0ca7f0 0x7f66ec0cac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:58.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.508+0000 7f66f1600640 1 -- 172.21.15.28:0/3595724902 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f66ec0c4470 msgr2=0x7f66ec1ed900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:58.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.508+0000 7f66f1600640 1 --2- 172.21.15.28:0/3595724902 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f66ec0c4470 0x7f66ec1ed900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:58.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.508+0000 7f66f1600640 1 -- 172.21.15.28:0/3595724902 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66ec0c3d60 con 0x7f66d40039b0 2024-02-21T23:07:58.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.508+0000 7f66f1600640 1 --2- 172.21.15.28:0/3595724902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66d40039b0 0x7f66ec0c1320 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f66e000c2b0 tx=0x7f66e0001970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:58.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.508+0000 7f66e9600640 1 -- 172.21.15.28:0/3595724902 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e0045070 con 0x7f66d40039b0 2024-02-21T23:07:58.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.509+0000 7f66e9600640 1 -- 172.21.15.28:0/3595724902 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66e0004bf0 con 0x7f66d40039b0 2024-02-21T23:07:58.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.509+0000 7f66e9600640 1 -- 172.21.15.28:0/3595724902 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e0001bc0 con 0x7f66d40039b0 2024-02-21T23:07:58.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.509+0000 7f66f2a00640 1 -- 172.21.15.28:0/3595724902 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66ec1eafe0 con 0x7f66d40039b0 2024-02-21T23:07:58.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.509+0000 7f66f2a00640 1 -- 172.21.15.28:0/3595724902 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66ec0c3a80 con 0x7f66d40039b0 2024-02-21T23:07:58.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.510+0000 7f66f2a00640 1 -- 172.21.15.28:0/3595724902 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66ec04f290 con 0x7f66d40039b0 2024-02-21T23:07:58.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.511+0000 7f66e9600640 1 -- 172.21.15.28:0/3595724902 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f66e0007070 con 0x7f66d40039b0 2024-02-21T23:07:58.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.512+0000 7f66e9600640 1 --2- 172.21.15.28:0/3595724902 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f66cc06b500 0x7f66cc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:58.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.512+0000 7f66e9600640 1 -- 172.21.15.28:0/3595724902 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f66e00ab3e0 con 0x7f66d40039b0 2024-02-21T23:07:58.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.512+0000 7f66f0c00640 1 --2- 172.21.15.28:0/3595724902 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f66cc06b500 0x7f66cc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:58.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.513+0000 7f66f0c00640 1 --2- 172.21.15.28:0/3595724902 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f66cc06b500 0x7f66cc06d9c0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f66ec0ac2b0 tx=0x7f66d8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:58.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.516+0000 7f66e9600640 1 -- 172.21.15.28:0/3595724902 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f66e0079a10 con 0x7f66d40039b0 2024-02-21T23:07:58.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.758+0000 7f66f2a00640 1 -- 172.21.15.28:0/3595724902 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f66ec06d9d0 con 0x7f66d40039b0 2024-02-21T23:07:58.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.759+0000 7f66e9600640 1 -- 172.21.15.28:0/3595724902 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+4949 (secure 0 0 0) 0x7f66e007dab0 con 0x7f66d40039b0 2024-02-21T23:07:58.748 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:07:58.748 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":112,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5843,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3059988835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3059988835},{"type":"v1","addr":"172.21.15.28:6835","nonce":3059988835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":112},{"gid":7446,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/1119726035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1119726035},{"type":"v1","addr":"172.21.15.28:6833","nonce":1119726035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":111,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-02-21T23:07:29.533547+0000","modified":"2024-02-21T23:07:49.218076+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5729},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5729":{"gid":5729,"name":"d","rank":0,"incarnation":104,"state":"up:active","state_seq":8,"addr":"172.21.15.81:6835/1754860807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1754860807},{"type":"v1","addr":"172.21.15.81:6835","nonce":1754860807}]},"join_fscid":8,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5732":{"gid":5732,"name":"b","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":7,"addr":"172.21.15.81:6837/3963422563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3963422563},{"type":"v1","addr":"172.21.15.81:6837","nonce":3963422563}]},"join_fscid":8,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-02-21T23:07:58.749 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 112 2024-02-21T23:07:58.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.762+0000 7f66f2a00640 1 -- 172.21.15.28:0/3595724902 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f66cc06b500 msgr2=0x7f66cc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:58.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.762+0000 7f66f2a00640 1 --2- 172.21.15.28:0/3595724902 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f66cc06b500 0x7f66cc06d9c0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f66ec0ac2b0 tx=0x7f66d8008040 comp rx=0 tx=0).stop 2024-02-21T23:07:58.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.762+0000 7f66f2a00640 1 -- 172.21.15.28:0/3595724902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66d40039b0 msgr2=0x7f66ec0c1320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:58.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.762+0000 7f66f2a00640 1 --2- 172.21.15.28:0/3595724902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66d40039b0 0x7f66ec0c1320 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f66e000c2b0 tx=0x7f66e0001970 comp rx=0 tx=0).stop 2024-02-21T23:07:58.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.762+0000 7f66f2a00640 1 -- 172.21.15.28:0/3595724902 shutdown_connections 2024-02-21T23:07:58.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.762+0000 7f66f2a00640 1 --2- 172.21.15.28:0/3595724902 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f66cc06b500 0x7f66cc06d9c0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:58.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.762+0000 7f66f2a00640 1 --2- 172.21.15.28:0/3595724902 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f66ec0ca7f0 0x7f66ec0cac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:58.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.762+0000 7f66f2a00640 1 --2- 172.21.15.28:0/3595724902 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f66ec0c4470 0x7f66ec1ed900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:58.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.762+0000 7f66f2a00640 1 --2- 172.21.15.28:0/3595724902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66d40039b0 0x7f66ec0c1320 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:58.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.762+0000 7f66f2a00640 1 -- 172.21.15.28:0/3595724902 >> 172.21.15.28:0/3595724902 conn(0x7f66ec0ac370 msgr2=0x7f66ec0ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:58.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.762+0000 7f66f2a00640 1 -- 172.21.15.28:0/3595724902 shutdown_connections 2024-02-21T23:07:58.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:58.762+0000 7f66f2a00640 1 -- 172.21.15.28:0/3595724902 wait complete. 2024-02-21T23:07:58.768 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:07:58.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.002+0000 7f394b200640 1 Processor -- start 2024-02-21T23:07:58.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.002+0000 7f394b200640 1 -- start start 2024-02-21T23:07:58.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.003+0000 7f394b200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f394c150450 0x7f394c0a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:58.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.004+0000 7f394b200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f394c0c2760 con 0x7f394c156af0 2024-02-21T23:07:58.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.004+0000 7f394b200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f394c156310 con 0x7f394c06ee40 2024-02-21T23:07:58.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.004+0000 7f394b200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f394c0c22c0 con 0x7f394c150450 2024-02-21T23:07:58.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.004+0000 7f394a800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f394c156af0 0x7f394c06dff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60762/0 (socket says 172.21.15.28:60762) 2024-02-21T23:07:58.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.004+0000 7f394a800640 1 -- 172.21.15.28:0/3112671603 learned_addr learned my addr 172.21.15.28:0/3112671603 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:07:58.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.004+0000 7f3949e00640 1 --2- 172.21.15.28:0/3112671603 >> v2:172.21.15.81:3300/0 conn(0x7f394c150450 0x7f394c0a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:58.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.004+0000 7f3948a00640 1 -- 172.21.15.28:0/3112671603 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1018258594 0 0) 0x7f394c0c2760 con 0x7f394c156af0 2024-02-21T23:07:58.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.005+0000 7f3948a00640 1 -- 172.21.15.28:0/3112671603 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3930003520 con 0x7f394c156af0 2024-02-21T23:07:58.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.005+0000 7f3948a00640 1 -- 172.21.15.28:0/3112671603 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3606091703 0 0) 0x7f3930003520 con 0x7f394c156af0 2024-02-21T23:07:58.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.005+0000 7f3948a00640 1 -- 172.21.15.28:0/3112671603 >> v2:172.21.15.81:3300/0 conn(0x7f394c150450 msgr2=0x7f394c0a5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:58.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.005+0000 7f3948a00640 1 --2- 172.21.15.28:0/3112671603 >> v2:172.21.15.81:3300/0 conn(0x7f394c150450 0x7f394c0a5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:58.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.005+0000 7f3948a00640 1 -- 172.21.15.28:0/3112671603 >> v1:172.21.15.81:6789/0 conn(0x7f394c06ee40 legacy=0x7f394c1536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:58.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.005+0000 7f3948a00640 1 -- 172.21.15.28:0/3112671603 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f394c0c29d0 con 0x7f394c156af0 2024-02-21T23:07:58.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.005+0000 7f3948a00640 1 -- 172.21.15.28:0/3112671603 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f39440019f0 con 0x7f394c156af0 2024-02-21T23:07:58.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.005+0000 7f3948a00640 1 -- 172.21.15.28:0/3112671603 >> v1:172.21.15.28:6789/0 conn(0x7f394c156af0 legacy=0x7f394c06dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:58.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.005+0000 7f3948a00640 1 --2- 172.21.15.28:0/3112671603 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3930003a00 0x7f3930003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:58.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.005+0000 7f3948a00640 1 --2- 172.21.15.28:0/3112671603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3930004800 0x7f3930004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:58.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.005+0000 7f3948a00640 1 --2- 172.21.15.28:0/3112671603 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f39300051c0 0x7f393000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:58.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.005+0000 7f3948a00640 1 -- 172.21.15.28:0/3112671603 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f393000c090 con 0x7f3930004800 2024-02-21T23:07:58.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.005+0000 7f3948a00640 1 -- 172.21.15.28:0/3112671603 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f393000c240 con 0x7f3930003a00 2024-02-21T23:07:58.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.005+0000 7f3948a00640 1 -- 172.21.15.28:0/3112671603 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f393000c440 con 0x7f39300051c0 2024-02-21T23:07:58.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.006+0000 7f3949400640 1 --2- 172.21.15.28:0/3112671603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3930004800 0x7f3930004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:58.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.006+0000 7f3949e00640 1 --2- 172.21.15.28:0/3112671603 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3930003a00 0x7f3930003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:58.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.006+0000 7f394a800640 1 --2- 172.21.15.28:0/3112671603 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f39300051c0 0x7f393000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:58.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.006+0000 7f3949400640 1 -- 172.21.15.28:0/3112671603 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f39300051c0 msgr2=0x7f393000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:58.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.006+0000 7f3949400640 1 --2- 172.21.15.28:0/3112671603 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f39300051c0 0x7f393000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:58.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.006+0000 7f3949400640 1 -- 172.21.15.28:0/3112671603 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3930003a00 msgr2=0x7f3930003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:58.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.006+0000 7f3949400640 1 --2- 172.21.15.28:0/3112671603 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3930003a00 0x7f3930003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:58.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.006+0000 7f3949400640 1 -- 172.21.15.28:0/3112671603 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3930003520 con 0x7f3930004800 2024-02-21T23:07:58.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.006+0000 7f394a800640 1 --2- 172.21.15.28:0/3112671603 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f39300051c0 0x7f393000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:07:58.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.006+0000 7f3949e00640 1 --2- 172.21.15.28:0/3112671603 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3930003a00 0x7f3930003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:07:58.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.006+0000 7f394a800640 1 -- 172.21.15.28:0/3112671603 reap_dead start 2024-02-21T23:07:59.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.007+0000 7f3949400640 1 --2- 172.21.15.28:0/3112671603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3930004800 0x7f3930004c80 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f393c001750 tx=0x7f3930005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d45ab5c7cd63ab1 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:59.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.007+0000 7f3948a00640 1 -- 172.21.15.28:0/3112671603 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f393c001370 con 0x7f3930004800 2024-02-21T23:07:59.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.008+0000 7f3948a00640 1 -- 172.21.15.28:0/3112671603 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f393c0052d0 con 0x7f3930004800 2024-02-21T23:07:59.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.008+0000 7f3948a00640 1 -- 172.21.15.28:0/3112671603 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f393c003830 con 0x7f3930004800 2024-02-21T23:07:59.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.008+0000 7f394b200640 1 -- 172.21.15.28:0/3112671603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3930004800 msgr2=0x7f3930004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:59.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.008+0000 7f394b200640 1 --2- 172.21.15.28:0/3112671603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3930004800 0x7f3930004c80 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f393c001750 tx=0x7f3930005830 comp rx=0 tx=0).stop 2024-02-21T23:07:59.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.008+0000 7f394b200640 1 -- 172.21.15.28:0/3112671603 shutdown_connections 2024-02-21T23:07:59.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.008+0000 7f394b200640 1 --2- 172.21.15.28:0/3112671603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3930004800 0x7f3930004c80 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:59.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.008+0000 7f394b200640 1 -- 172.21.15.28:0/3112671603 >> 172.21.15.28:0/3112671603 conn(0x7f394c06b630 msgr2=0x7f394c0c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:59.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.008+0000 7f394b200640 1 -- 172.21.15.28:0/3112671603 shutdown_connections 2024-02-21T23:07:59.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.008+0000 7f394b200640 1 -- 172.21.15.28:0/3112671603 wait complete. 2024-02-21T23:07:59.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.010+0000 7f394b200640 1 Processor -- start 2024-02-21T23:07:59.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.010+0000 7f394b200640 1 -- start start 2024-02-21T23:07:59.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.011+0000 7f394b200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3930004800 0x7f394c0bfba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:59.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.011+0000 7f394b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f394c1ede70 0x7f394c145860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:59.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.011+0000 7f394b200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f394c145da0 0x7f394c146a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:59.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.011+0000 7f394b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f394c156f80 con 0x7f394c1ede70 2024-02-21T23:07:59.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.011+0000 7f394b200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f394c156660 con 0x7f394c145da0 2024-02-21T23:07:59.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.011+0000 7f394b200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f394c157100 con 0x7f3930004800 2024-02-21T23:07:59.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.011+0000 7f3949400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f394c1ede70 0x7f394c145860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:59.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.011+0000 7f3949400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f394c1ede70 0x7f394c145860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48694/0 (socket says 172.21.15.28:48694) 2024-02-21T23:07:59.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.011+0000 7f3949400640 1 -- 172.21.15.28:0/917626164 learned_addr learned my addr 172.21.15.28:0/917626164 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:59.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.012+0000 7f3949e00640 1 --2- 172.21.15.28:0/917626164 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3930004800 0x7f394c0bfba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:59.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.012+0000 7f394a800640 1 --2- 172.21.15.28:0/917626164 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f394c145da0 0x7f394c146a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:59.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.012+0000 7f3949400640 1 -- 172.21.15.28:0/917626164 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3930004800 msgr2=0x7f394c0bfba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:59.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.012+0000 7f3949400640 1 --2- 172.21.15.28:0/917626164 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3930004800 0x7f394c0bfba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:59.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.012+0000 7f3949400640 1 -- 172.21.15.28:0/917626164 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f394c145da0 msgr2=0x7f394c146a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:59.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.012+0000 7f3949400640 1 --2- 172.21.15.28:0/917626164 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f394c145da0 0x7f394c146a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:59.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.012+0000 7f3949400640 1 -- 172.21.15.28:0/917626164 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f394c0c03f0 con 0x7f394c1ede70 2024-02-21T23:07:59.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.012+0000 7f3949400640 1 --2- 172.21.15.28:0/917626164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f394c1ede70 0x7f394c145860 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f393c02bce0 tx=0x7f393c001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:59.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.013+0000 7f3942000640 1 -- 172.21.15.28:0/917626164 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f393c041020 con 0x7f394c1ede70 2024-02-21T23:07:59.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.013+0000 7f3942000640 1 -- 172.21.15.28:0/917626164 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f393c005480 con 0x7f394c1ede70 2024-02-21T23:07:59.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.013+0000 7f3942000640 1 -- 172.21.15.28:0/917626164 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f393c03d680 con 0x7f394c1ede70 2024-02-21T23:07:59.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.013+0000 7f394b200640 1 -- 172.21.15.28:0/917626164 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f394c1ead10 con 0x7f394c1ede70 2024-02-21T23:07:59.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.013+0000 7f394b200640 1 -- 172.21.15.28:0/917626164 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f394c1eb2d0 con 0x7f394c1ede70 2024-02-21T23:07:59.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.014+0000 7f394b200640 1 -- 172.21.15.28:0/917626164 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f394c04f290 con 0x7f394c1ede70 2024-02-21T23:07:59.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.015+0000 7f3942000640 1 -- 172.21.15.28:0/917626164 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f393c044020 con 0x7f394c1ede70 2024-02-21T23:07:59.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.016+0000 7f3942000640 1 --2- 172.21.15.28:0/917626164 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f392806b4d0 0x7f392806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:59.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.016+0000 7f3942000640 1 -- 172.21.15.28:0/917626164 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f393c0ac1f0 con 0x7f394c1ede70 2024-02-21T23:07:59.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.016+0000 7f3949e00640 1 --2- 172.21.15.28:0/917626164 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f392806b4d0 0x7f392806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:59.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.017+0000 7f3949e00640 1 --2- 172.21.15.28:0/917626164 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f392806b4d0 0x7f392806d990 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f3934009b20 tx=0x7f3934008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:59.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.020+0000 7f3942000640 1 -- 172.21.15.28:0/917626164 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f393c04b020 con 0x7f394c1ede70 2024-02-21T23:07:59.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.258+0000 7f394b200640 1 -- 172.21.15.28:0/917626164 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f394c0c1f10 con 0x7f394c1ede70 2024-02-21T23:07:59.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.259+0000 7f3942000640 1 -- 172.21.15.28:0/917626164 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+4949 (secure 0 0 0) 0x7f393c07ae00 con 0x7f394c1ede70 2024-02-21T23:07:59.248 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:07:59.248 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":112,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5843,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3059988835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3059988835},{"type":"v1","addr":"172.21.15.28:6835","nonce":3059988835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":112},{"gid":7446,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/1119726035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1119726035},{"type":"v1","addr":"172.21.15.28:6833","nonce":1119726035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":96}],"filesystems":[{"mdsmap":{"epoch":111,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-02-21T23:07:29.533547+0000","modified":"2024-02-21T23:07:49.218076+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5729},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5729":{"gid":5729,"name":"d","rank":0,"incarnation":104,"state":"up:active","state_seq":8,"addr":"172.21.15.81:6835/1754860807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1754860807},{"type":"v1","addr":"172.21.15.81:6835","nonce":1754860807}]},"join_fscid":8,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5732":{"gid":5732,"name":"b","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":7,"addr":"172.21.15.81:6837/3963422563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3963422563},{"type":"v1","addr":"172.21.15.81:6837","nonce":3963422563}]},"join_fscid":8,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-02-21T23:07:59.249 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 112 2024-02-21T23:07:59.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.262+0000 7f394b200640 1 -- 172.21.15.28:0/917626164 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f392806b4d0 msgr2=0x7f392806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:59.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.262+0000 7f394b200640 1 --2- 172.21.15.28:0/917626164 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f392806b4d0 0x7f392806d990 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f3934009b20 tx=0x7f3934008040 comp rx=0 tx=0).stop 2024-02-21T23:07:59.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.262+0000 7f394b200640 1 -- 172.21.15.28:0/917626164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f394c1ede70 msgr2=0x7f394c145860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:59.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.262+0000 7f394b200640 1 --2- 172.21.15.28:0/917626164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f394c1ede70 0x7f394c145860 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f393c02bce0 tx=0x7f393c001120 comp rx=0 tx=0).stop 2024-02-21T23:07:59.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.262+0000 7f394b200640 1 -- 172.21.15.28:0/917626164 shutdown_connections 2024-02-21T23:07:59.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.262+0000 7f394b200640 1 --2- 172.21.15.28:0/917626164 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f392806b4d0 0x7f392806d990 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:59.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.262+0000 7f394b200640 1 --2- 172.21.15.28:0/917626164 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f394c145da0 0x7f394c146a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:59.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.262+0000 7f394b200640 1 --2- 172.21.15.28:0/917626164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f394c1ede70 0x7f394c145860 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:59.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.262+0000 7f394b200640 1 --2- 172.21.15.28:0/917626164 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3930004800 0x7f394c0bfba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:59.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.262+0000 7f394b200640 1 -- 172.21.15.28:0/917626164 >> 172.21.15.28:0/917626164 conn(0x7f394c06b630 msgr2=0x7f394c149190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:59.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.263+0000 7f394b200640 1 -- 172.21.15.28:0/917626164 shutdown_connections 2024-02-21T23:07:59.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.263+0000 7f394b200640 1 -- 172.21.15.28:0/917626164 wait complete. 2024-02-21T23:07:59.275 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:07:59.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.512+0000 7fba60e00640 1 Processor -- start 2024-02-21T23:07:59.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.512+0000 7fba60e00640 1 -- start start 2024-02-21T23:07:59.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.513+0000 7fba60e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fba5c0c8e60 0x7fba5c15e550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:59.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.513+0000 7fba60e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba5c0c92a0 con 0x7fba5c0c8a80 2024-02-21T23:07:59.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.514+0000 7fba60e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba5c0c8000 con 0x7fba5c156c90 2024-02-21T23:07:59.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.514+0000 7fba60e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba5c1547c0 con 0x7fba5c0c8e60 2024-02-21T23:07:59.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.514+0000 7fba5b400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fba5c0c8a80 0x7fba5c153640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60772/0 (socket says 172.21.15.28:60772) 2024-02-21T23:07:59.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.514+0000 7fba5b400640 1 -- 172.21.15.28:0/2257450715 learned_addr learned my addr 172.21.15.28:0/2257450715 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:07:59.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.514+0000 7fba5be00640 1 --2- 172.21.15.28:0/2257450715 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fba5c0c8e60 0x7fba5c15e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:59.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.514+0000 7fba5a000640 1 -- 172.21.15.28:0/2257450715 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1060726638 0 0) 0x7fba5c0c92a0 con 0x7fba5c0c8a80 2024-02-21T23:07:59.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.514+0000 7fba5a000640 1 -- 172.21.15.28:0/2257450715 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba40003520 con 0x7fba5c0c8a80 2024-02-21T23:07:59.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.515+0000 7fba5a000640 1 -- 172.21.15.28:0/2257450715 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4206913153 0 0) 0x7fba40003520 con 0x7fba5c0c8a80 2024-02-21T23:07:59.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.515+0000 7fba5a000640 1 -- 172.21.15.28:0/2257450715 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fba5c0c8e60 msgr2=0x7fba5c15e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:59.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.515+0000 7fba5a000640 1 --2- 172.21.15.28:0/2257450715 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fba5c0c8e60 0x7fba5c15e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:59.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.515+0000 7fba5a000640 1 -- 172.21.15.28:0/2257450715 >> v1:172.21.15.81:6789/0 conn(0x7fba5c156c90 legacy=0x7fba5c1540b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:59.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.515+0000 7fba5a000640 1 -- 172.21.15.28:0/2257450715 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba5c0c82e0 con 0x7fba5c0c8a80 2024-02-21T23:07:59.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.515+0000 7fba5a000640 1 -- 172.21.15.28:0/2257450715 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fba4c001670 con 0x7fba5c0c8a80 2024-02-21T23:07:59.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.515+0000 7fba5a000640 1 -- 172.21.15.28:0/2257450715 >> v1:172.21.15.28:6789/0 conn(0x7fba5c0c8a80 legacy=0x7fba5c153640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:59.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.515+0000 7fba5a000640 1 --2- 172.21.15.28:0/2257450715 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fba400039e0 0x7fba40003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:59.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.515+0000 7fba5a000640 1 --2- 172.21.15.28:0/2257450715 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fba5c0c8e60 0x7fba40004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:59.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.515+0000 7fba5a000640 1 --2- 172.21.15.28:0/2257450715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba40004d40 0x7fba400051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:59.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.515+0000 7fba5a000640 1 -- 172.21.15.28:0/2257450715 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba5c1547c0 con 0x7fba40004d40 2024-02-21T23:07:59.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.515+0000 7fba5a000640 1 -- 172.21.15.28:0/2257450715 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba4000baf0 con 0x7fba400039e0 2024-02-21T23:07:59.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.515+0000 7fba5a000640 1 -- 172.21.15.28:0/2257450715 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba4000bcf0 con 0x7fba5c0c8e60 2024-02-21T23:07:59.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.515+0000 7fba5be00640 1 --2- 172.21.15.28:0/2257450715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba40004d40 0x7fba400051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:59.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.515+0000 7fba5aa00640 1 --2- 172.21.15.28:0/2257450715 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fba5c0c8e60 0x7fba40004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:59.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.515+0000 7fba5b400640 1 --2- 172.21.15.28:0/2257450715 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fba400039e0 0x7fba40003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:59.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.516+0000 7fba5be00640 1 -- 172.21.15.28:0/2257450715 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fba5c0c8e60 msgr2=0x7fba40004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:59.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.516+0000 7fba5be00640 1 --2- 172.21.15.28:0/2257450715 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fba5c0c8e60 0x7fba40004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:59.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.516+0000 7fba5be00640 1 -- 172.21.15.28:0/2257450715 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fba400039e0 msgr2=0x7fba40003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:59.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.516+0000 7fba5be00640 1 --2- 172.21.15.28:0/2257450715 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fba400039e0 0x7fba40003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:59.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.516+0000 7fba5be00640 1 -- 172.21.15.28:0/2257450715 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba40003520 con 0x7fba40004d40 2024-02-21T23:07:59.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.517+0000 7fba5be00640 1 --2- 172.21.15.28:0/2257450715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba40004d40 0x7fba400051c0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7fba54009fc0 tx=0x7fba40005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=107c59b6e5dae676 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:59.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.517+0000 7fba5a000640 1 -- 172.21.15.28:0/2257450715 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba54039070 con 0x7fba40004d40 2024-02-21T23:07:59.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.517+0000 7fba5a000640 1 -- 172.21.15.28:0/2257450715 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba5402cbf0 con 0x7fba40004d40 2024-02-21T23:07:59.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.517+0000 7fba5a000640 1 -- 172.21.15.28:0/2257450715 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba54009c10 con 0x7fba40004d40 2024-02-21T23:07:59.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.518+0000 7fba60e00640 1 -- 172.21.15.28:0/2257450715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba40004d40 msgr2=0x7fba400051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:59.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.518+0000 7fba60e00640 1 --2- 172.21.15.28:0/2257450715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba40004d40 0x7fba400051c0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7fba54009fc0 tx=0x7fba40005830 comp rx=0 tx=0).stop 2024-02-21T23:07:59.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.518+0000 7fba5be00640 1 -- 172.21.15.28:0/2257450715 reap_dead start 2024-02-21T23:07:59.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.518+0000 7fba60e00640 1 -- 172.21.15.28:0/2257450715 shutdown_connections 2024-02-21T23:07:59.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.518+0000 7fba60e00640 1 -- 172.21.15.28:0/2257450715 >> 172.21.15.28:0/2257450715 conn(0x7fba5c06b260 msgr2=0x7fba5c06b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:59.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.518+0000 7fba60e00640 1 -- 172.21.15.28:0/2257450715 shutdown_connections 2024-02-21T23:07:59.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.518+0000 7fba60e00640 1 -- 172.21.15.28:0/2257450715 wait complete. 2024-02-21T23:07:59.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.519+0000 7fba60e00640 1 Processor -- start 2024-02-21T23:07:59.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.520+0000 7fba60e00640 1 -- start start 2024-02-21T23:07:59.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.520+0000 7fba60e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fba40004d40 0x7fba5c14f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:59.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.520+0000 7fba60e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fba5c14f8f0 0x7fba5c1ee010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:59.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.521+0000 7fba60e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba5c1ee550 0x7fba5c1eaaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:59.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.521+0000 7fba60e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba5c156490 con 0x7fba5c1ee550 2024-02-21T23:07:59.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.521+0000 7fba60e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba5c1570c0 con 0x7fba5c14f8f0 2024-02-21T23:07:59.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.521+0000 7fba60e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba5c156770 con 0x7fba40004d40 2024-02-21T23:07:59.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.521+0000 7fba5be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba5c1ee550 0x7fba5c1eaaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:59.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.521+0000 7fba5be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba5c1ee550 0x7fba5c1eaaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48720/0 (socket says 172.21.15.28:48720) 2024-02-21T23:07:59.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.521+0000 7fba5be00640 1 -- 172.21.15.28:0/576617866 learned_addr learned my addr 172.21.15.28:0/576617866 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:59.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.521+0000 7fba5b400640 1 --2- 172.21.15.28:0/576617866 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fba40004d40 0x7fba5c14f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:59.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.521+0000 7fba5aa00640 1 --2- 172.21.15.28:0/576617866 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fba5c14f8f0 0x7fba5c1ee010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:59.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.521+0000 7fba5be00640 1 -- 172.21.15.28:0/576617866 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fba40004d40 msgr2=0x7fba5c14f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:59.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.521+0000 7fba5be00640 1 --2- 172.21.15.28:0/576617866 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fba40004d40 0x7fba5c14f3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:59.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.521+0000 7fba5be00640 1 -- 172.21.15.28:0/576617866 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fba5c14f8f0 msgr2=0x7fba5c1ee010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:59.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.521+0000 7fba5be00640 1 --2- 172.21.15.28:0/576617866 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fba5c14f8f0 0x7fba5c1ee010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:59.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.521+0000 7fba5be00640 1 -- 172.21.15.28:0/576617866 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba5c14eed0 con 0x7fba5c1ee550 2024-02-21T23:07:59.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.522+0000 7fba5be00640 1 --2- 172.21.15.28:0/576617866 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba5c1ee550 0x7fba5c1eaaa0 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7fba5400e230 tx=0x7fba54009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:59.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.522+0000 7fba53400640 1 -- 172.21.15.28:0/576617866 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba54039070 con 0x7fba5c1ee550 2024-02-21T23:07:59.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.522+0000 7fba53400640 1 -- 172.21.15.28:0/576617866 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba5403a040 con 0x7fba5c1ee550 2024-02-21T23:07:59.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.522+0000 7fba53400640 1 -- 172.21.15.28:0/576617866 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba54003990 con 0x7fba5c1ee550 2024-02-21T23:07:59.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.522+0000 7fba60e00640 1 -- 172.21.15.28:0/576617866 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba5c1eed00 con 0x7fba5c1ee550 2024-02-21T23:07:59.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.522+0000 7fba60e00640 1 -- 172.21.15.28:0/576617866 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba5c0aaae0 con 0x7fba5c1ee550 2024-02-21T23:07:59.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.523+0000 7fba60e00640 1 -- 172.21.15.28:0/576617866 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba5c04f290 con 0x7fba5c1ee550 2024-02-21T23:07:59.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.524+0000 7fba53400640 1 -- 172.21.15.28:0/576617866 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fba54046020 con 0x7fba5c1ee550 2024-02-21T23:07:59.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.525+0000 7fba53400640 1 --2- 172.21.15.28:0/576617866 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fba3c06b4d0 0x7fba3c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:59.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.525+0000 7fba53400640 1 -- 172.21.15.28:0/576617866 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fba5402c9f0 con 0x7fba5c1ee550 2024-02-21T23:07:59.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.526+0000 7fba5b400640 1 --2- 172.21.15.28:0/576617866 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fba3c06b4d0 0x7fba3c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:59.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.527+0000 7fba5b400640 1 --2- 172.21.15.28:0/576617866 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fba3c06b4d0 0x7fba3c06d990 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fba4c000c00 tx=0x7fba4c007040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:59.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.530+0000 7fba53400640 1 -- 172.21.15.28:0/576617866 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fba5407a790 con 0x7fba5c1ee550 2024-02-21T23:07:59.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.731+0000 7fba60e00640 1 -- 172.21.15.28:0/576617866 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fba5c0c7520 con 0x7fba5c1ee550 2024-02-21T23:07:59.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.733+0000 7fba53400640 1 -- 172.21.15.28:0/576617866 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v111) v1 ==== 74+0+17021 (secure 0 0 0) 0x7fba5407e830 con 0x7fba5c1ee550 2024-02-21T23:07:59.722 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:07:59.722 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":111,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:07:41.139084+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:07:27.477155+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-02-21T23:07:28.549456+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6835/1259832430":"2024-02-22T23:07:41.139002+0000","172.21.15.28:6834/1259832430":"2024-02-22T23:07:41.139002+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:07:59.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.735+0000 7fba60e00640 1 -- 172.21.15.28:0/576617866 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fba3c06b4d0 msgr2=0x7fba3c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:59.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.735+0000 7fba60e00640 1 --2- 172.21.15.28:0/576617866 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fba3c06b4d0 0x7fba3c06d990 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fba4c000c00 tx=0x7fba4c007040 comp rx=0 tx=0).stop 2024-02-21T23:07:59.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.735+0000 7fba60e00640 1 -- 172.21.15.28:0/576617866 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba5c1ee550 msgr2=0x7fba5c1eaaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:59.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.735+0000 7fba60e00640 1 --2- 172.21.15.28:0/576617866 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba5c1ee550 0x7fba5c1eaaa0 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7fba5400e230 tx=0x7fba54009750 comp rx=0 tx=0).stop 2024-02-21T23:07:59.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.735+0000 7fba60e00640 1 -- 172.21.15.28:0/576617866 shutdown_connections 2024-02-21T23:07:59.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.735+0000 7fba60e00640 1 --2- 172.21.15.28:0/576617866 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fba3c06b4d0 0x7fba3c06d990 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:59.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.736+0000 7fba60e00640 1 --2- 172.21.15.28:0/576617866 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba5c1ee550 0x7fba5c1eaaa0 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:59.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.736+0000 7fba60e00640 1 --2- 172.21.15.28:0/576617866 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fba5c14f8f0 0x7fba5c1ee010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:59.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.736+0000 7fba60e00640 1 --2- 172.21.15.28:0/576617866 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fba40004d40 0x7fba5c14f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:59.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.736+0000 7fba60e00640 1 -- 172.21.15.28:0/576617866 >> 172.21.15.28:0/576617866 conn(0x7fba5c06b260 msgr2=0x7fba5c154940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:59.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.736+0000 7fba60e00640 1 -- 172.21.15.28:0/576617866 shutdown_connections 2024-02-21T23:07:59.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.736+0000 7fba60e00640 1 -- 172.21.15.28:0/576617866 wait complete. 2024-02-21T23:07:59.747 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-02-21T23:07:59.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.981+0000 7f0139c00640 1 Processor -- start 2024-02-21T23:07:59.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.982+0000 7f0139c00640 1 -- start start 2024-02-21T23:07:59.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.983+0000 7f0139c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f01341524c0 0x7f0134155a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:59.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.983+0000 7f0139c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f01340c85c0 0x7f01340c70d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:59.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.983+0000 7f0139c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f013406df50 con 0x7f01340c5040 2024-02-21T23:07:59.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.983+0000 7f0139c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01340c5520 con 0x7f01340c85c0 2024-02-21T23:07:59.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.983+0000 7f0139c00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01340c5800 con 0x7f01341524c0 2024-02-21T23:07:59.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.983+0000 7f012be00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f01340c5040 0x7f013406ce70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60774/0 (socket says 172.21.15.28:60774) 2024-02-21T23:07:59.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.983+0000 7f012be00640 1 -- 172.21.15.28:0/3854921044 learned_addr learned my addr 172.21.15.28:0/3854921044 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:07:59.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.983+0000 7f0139200640 1 --2- 172.21.15.28:0/3854921044 >> v2:172.21.15.28:3300/0 conn(0x7f01340c85c0 0x7f01340c70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:59.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.984+0000 7f0133e00640 1 --2- 172.21.15.28:0/3854921044 >> v2:172.21.15.81:3300/0 conn(0x7f01341524c0 0x7f0134155a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:59.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.984+0000 7f0133400640 1 -- 172.21.15.28:0/3854921044 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3759766658 0 0) 0x7f013406df50 con 0x7f01340c5040 2024-02-21T23:07:59.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.984+0000 7f0133400640 1 -- 172.21.15.28:0/3854921044 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0118003520 con 0x7f01340c5040 2024-02-21T23:07:59.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.984+0000 7f0139200640 1 -- 172.21.15.28:0/3854921044 >> v2:172.21.15.81:3300/0 conn(0x7f01341524c0 msgr2=0x7f0134155a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:07:59.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.984+0000 7f0139200640 1 --2- 172.21.15.28:0/3854921044 >> v2:172.21.15.81:3300/0 conn(0x7f01341524c0 0x7f0134155a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:59.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.984+0000 7f0139200640 1 -- 172.21.15.28:0/3854921044 >> v1:172.21.15.28:6789/0 conn(0x7f01340c5040 legacy=0x7f013406ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:59.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.984+0000 7f0139200640 1 -- 172.21.15.28:0/3854921044 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f013406c670 con 0x7f01340c85c0 2024-02-21T23:07:59.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.985+0000 7f0139200640 1 --2- 172.21.15.28:0/3854921044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01340c85c0 0x7f01340c70d0 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f01340cb790 tx=0x7f012c02bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f9da259ada7193a server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:59.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.985+0000 7f0133400640 1 -- 172.21.15.28:0/3854921044 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f012c039070 con 0x7f01340c85c0 2024-02-21T23:07:59.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.985+0000 7f0133400640 1 -- 172.21.15.28:0/3854921044 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f012c02cb60 con 0x7f01340c85c0 2024-02-21T23:07:59.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.985+0000 7f0133400640 1 -- 172.21.15.28:0/3854921044 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f012c034440 con 0x7f01340c85c0 2024-02-21T23:07:59.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.986+0000 7f0139c00640 1 -- 172.21.15.28:0/3854921044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01340c85c0 msgr2=0x7f01340c70d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:59.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.986+0000 7f0139c00640 1 --2- 172.21.15.28:0/3854921044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01340c85c0 0x7f01340c70d0 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f01340cb790 tx=0x7f012c02bdb0 comp rx=0 tx=0).stop 2024-02-21T23:07:59.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.986+0000 7f0139c00640 1 -- 172.21.15.28:0/3854921044 shutdown_connections 2024-02-21T23:07:59.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.986+0000 7f0139c00640 1 --2- 172.21.15.28:0/3854921044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01340c85c0 0x7f01340c70d0 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:59.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.986+0000 7f0139c00640 1 --2- 172.21.15.28:0/3854921044 >> v2:172.21.15.81:3300/0 conn(0x7f01341524c0 0x7f0134155a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:59.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.986+0000 7f0139c00640 1 -- 172.21.15.28:0/3854921044 >> 172.21.15.28:0/3854921044 conn(0x7f01340ac2b0 msgr2=0x7f01340ac6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:07:59.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.986+0000 7f0139c00640 1 -- 172.21.15.28:0/3854921044 shutdown_connections 2024-02-21T23:07:59.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.986+0000 7f0139c00640 1 -- 172.21.15.28:0/3854921044 wait complete. 2024-02-21T23:07:59.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.988+0000 7f0139c00640 1 Processor -- start 2024-02-21T23:07:59.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.988+0000 7f0139c00640 1 -- start start 2024-02-21T23:07:59.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.989+0000 7f0139c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01340c5040 0x7f01341572a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:59.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.989+0000 7f0139c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01340c85c0 0x7f013414d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:59.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.989+0000 7f0139c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f01341524c0 0x7f0134158e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:59.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.989+0000 7f0139c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01340c8a70 con 0x7f01340c85c0 2024-02-21T23:07:59.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.989+0000 7f0139c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01341528a0 con 0x7f01340c5040 2024-02-21T23:07:59.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.989+0000 7f0139c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01340c7610 con 0x7f01341524c0 2024-02-21T23:07:59.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.989+0000 7f012be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01340c85c0 0x7f013414d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:59.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.989+0000 7f012be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01340c85c0 0x7f013414d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48738/0 (socket says 172.21.15.28:48738) 2024-02-21T23:07:59.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.989+0000 7f012be00640 1 -- 172.21.15.28:0/2478097416 learned_addr learned my addr 172.21.15.28:0/2478097416 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:07:59.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.989+0000 7f0133e00640 1 --2- 172.21.15.28:0/2478097416 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01340c5040 0x7f01341572a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:59.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.990+0000 7f0139200640 1 --2- 172.21.15.28:0/2478097416 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f01341524c0 0x7f0134158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:59.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.990+0000 7f012be00640 1 -- 172.21.15.28:0/2478097416 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f01341524c0 msgr2=0x7f0134158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:59.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.990+0000 7f012be00640 1 --2- 172.21.15.28:0/2478097416 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f01341524c0 0x7f0134158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:59.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.990+0000 7f012be00640 1 -- 172.21.15.28:0/2478097416 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01340c5040 msgr2=0x7f01341572a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:07:59.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.990+0000 7f012be00640 1 --2- 172.21.15.28:0/2478097416 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01340c5040 0x7f01341572a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:07:59.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.990+0000 7f012be00640 1 -- 172.21.15.28:0/2478097416 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f013414dcf0 con 0x7f01340c85c0 2024-02-21T23:07:59.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.990+0000 7f012be00640 1 --2- 172.21.15.28:0/2478097416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01340c85c0 0x7f013414d7b0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f011c00c430 tx=0x7f011c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:59.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.990+0000 7f0130c00640 1 -- 172.21.15.28:0/2478097416 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f011c017070 con 0x7f01340c85c0 2024-02-21T23:07:59.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.990+0000 7f0130c00640 1 -- 172.21.15.28:0/2478097416 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f011c005ce0 con 0x7f01340c85c0 2024-02-21T23:07:59.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.990+0000 7f0130c00640 1 -- 172.21.15.28:0/2478097416 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f011c0072f0 con 0x7f01340c85c0 2024-02-21T23:07:59.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.990+0000 7f0139c00640 1 -- 172.21.15.28:0/2478097416 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01340bee50 con 0x7f01340c85c0 2024-02-21T23:07:59.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.991+0000 7f0139c00640 1 -- 172.21.15.28:0/2478097416 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01340beb70 con 0x7f01340c85c0 2024-02-21T23:07:59.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.992+0000 7f0139c00640 1 -- 172.21.15.28:0/2478097416 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f013404f290 con 0x7f01340c85c0 2024-02-21T23:07:59.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.993+0000 7f0130c00640 1 -- 172.21.15.28:0/2478097416 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f011c01a070 con 0x7f01340c85c0 2024-02-21T23:07:59.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.994+0000 7f0130c00640 1 --2- 172.21.15.28:0/2478097416 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f011006b520 0x7f011006d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:07:59.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.994+0000 7f0130c00640 1 -- 172.21.15.28:0/2478097416 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f011c089860 con 0x7f01340c85c0 2024-02-21T23:07:59.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.995+0000 7f0133e00640 1 --2- 172.21.15.28:0/2478097416 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f011006b520 0x7f011006d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:07:59.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.996+0000 7f0133e00640 1 --2- 172.21.15.28:0/2478097416 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f011006b520 0x7f011006d9e0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f0124006fd0 tx=0x7f0124008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:07:59.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:07:59.999+0000 7f0130c00640 1 -- 172.21.15.28:0/2478097416 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f011c058570 con 0x7f01340c85c0 2024-02-21T23:08:00.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.237+0000 7f0139c00640 1 -- 172.21.15.28:0/2478097416 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f013414be20 con 0x7f01340c85c0 2024-02-21T23:08:00.227 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-02-21T23:08:00.238+0000 7f5ddc400640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-21T23:08:00.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.241+0000 7f0130c00640 1 -- 172.21.15.28:0/2478097416 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v113) v1 ==== 161+0+0 (secure 0 0 0) 0x7f011c057e70 con 0x7f01340c85c0 2024-02-21T23:08:00.231 INFO:teuthology.orchestra.run.smithi028.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-02-21T23:08:00.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.245+0000 7f0139c00640 1 -- 172.21.15.28:0/2478097416 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f011006b520 msgr2=0x7f011006d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:00.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.245+0000 7f0139c00640 1 --2- 172.21.15.28:0/2478097416 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f011006b520 0x7f011006d9e0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f0124006fd0 tx=0x7f0124008040 comp rx=0 tx=0).stop 2024-02-21T23:08:00.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.245+0000 7f0139c00640 1 -- 172.21.15.28:0/2478097416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01340c85c0 msgr2=0x7f013414d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:00.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.245+0000 7f0139c00640 1 --2- 172.21.15.28:0/2478097416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01340c85c0 0x7f013414d7b0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f011c00c430 tx=0x7f011c00c8f0 comp rx=0 tx=0).stop 2024-02-21T23:08:00.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.245+0000 7f0139c00640 1 -- 172.21.15.28:0/2478097416 shutdown_connections 2024-02-21T23:08:00.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.245+0000 7f0139c00640 1 --2- 172.21.15.28:0/2478097416 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f011006b520 0x7f011006d9e0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:00.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.245+0000 7f0139c00640 1 --2- 172.21.15.28:0/2478097416 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f01341524c0 0x7f0134158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:00.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.245+0000 7f0139c00640 1 --2- 172.21.15.28:0/2478097416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01340c85c0 0x7f013414d7b0 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:00.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.245+0000 7f0139c00640 1 --2- 172.21.15.28:0/2478097416 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01340c5040 0x7f01341572a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:00.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.245+0000 7f0139c00640 1 -- 172.21.15.28:0/2478097416 >> 172.21.15.28:0/2478097416 conn(0x7f01340ac2b0 msgr2=0x7f01340ab740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:00.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.245+0000 7f0139c00640 1 -- 172.21.15.28:0/2478097416 shutdown_connections 2024-02-21T23:08:00.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.245+0000 7f0139c00640 1 -- 172.21.15.28:0/2478097416 wait complete. 2024-02-21T23:08:00.252 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-02-21T23:08:00.253 INFO:tasks.ceph.mds.d.smithi081.stderr: -608> 2024-02-21T23:07:46.011+0000 7fd00b600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-21T23:08:00.295 INFO:tasks.ceph.mds.b.smithi081.stdout:starting mds.b at 2024-02-21T23:08:00.310 INFO:tasks.ceph.mds.d.smithi081.stdout:starting mds.d at 2024-02-21T23:08:00.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.479+0000 7f6087e00640 1 Processor -- start 2024-02-21T23:08:00.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.479+0000 7f6087e00640 1 -- start start 2024-02-21T23:08:00.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.480+0000 7f6087e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f608813e520 0x7f608806c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:00.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.480+0000 7f6087e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6088158550 con 0x7f608813e0c0 2024-02-21T23:08:00.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.480+0000 7f6087e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60881587d0 con 0x7f608806dbc0 2024-02-21T23:08:00.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.480+0000 7f6087e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60880c89a0 con 0x7f608813e520 2024-02-21T23:08:00.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.480+0000 7f6086a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f608813e0c0 0x7f60880c71b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60776/0 (socket says 172.21.15.28:60776) 2024-02-21T23:08:00.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.480+0000 7f6086a00640 1 -- 172.21.15.28:0/1784012093 learned_addr learned my addr 172.21.15.28:0/1784012093 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:08:00.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.480+0000 7f6087400640 1 --2- 172.21.15.28:0/1784012093 >> v2:172.21.15.28:3300/0 conn(0x7f608813e520 0x7f608806c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:00.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.481+0000 7f6087400640 1 -- 172.21.15.28:0/1784012093 >> v1:172.21.15.81:6789/0 conn(0x7f608806dbc0 legacy=0x7f608806c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:00.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.481+0000 7f6087400640 1 -- 172.21.15.28:0/1784012093 >> v1:172.21.15.28:6789/0 conn(0x7f608813e0c0 legacy=0x7f60880c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:00.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.481+0000 7f6087400640 1 -- 172.21.15.28:0/1784012093 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6088158ab0 con 0x7f608813e520 2024-02-21T23:08:00.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.481+0000 7f6085600640 1 -- 172.21.15.28:0/1784012093 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2610246701 0 0) 0x7f6078001110 con 0x7f608813e0c0 2024-02-21T23:08:00.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.484+0000 7f6087400640 1 --2- 172.21.15.28:0/1784012093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f608813e520 0x7f608806c990 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f6080009d90 tx=0x7f608002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5918afe56326aad4 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:00.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.484+0000 7f6085600640 1 -- 172.21.15.28:0/1784012093 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6080039070 con 0x7f608813e520 2024-02-21T23:08:00.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.484+0000 7f6085600640 1 -- 172.21.15.28:0/1784012093 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f608002cbd0 con 0x7f608813e520 2024-02-21T23:08:00.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.484+0000 7f6085600640 1 -- 172.21.15.28:0/1784012093 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6080034440 con 0x7f608813e520 2024-02-21T23:08:00.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.484+0000 7f6087e00640 1 -- 172.21.15.28:0/1784012093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f608813e520 msgr2=0x7f608806c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:00.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.484+0000 7f6087e00640 1 --2- 172.21.15.28:0/1784012093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f608813e520 0x7f608806c990 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f6080009d90 tx=0x7f608002be20 comp rx=0 tx=0).stop 2024-02-21T23:08:00.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.484+0000 7f6087e00640 1 -- 172.21.15.28:0/1784012093 shutdown_connections 2024-02-21T23:08:00.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.484+0000 7f6087e00640 1 --2- 172.21.15.28:0/1784012093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f608813e520 0x7f608806c990 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:00.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.484+0000 7f6087e00640 1 -- 172.21.15.28:0/1784012093 >> 172.21.15.28:0/1784012093 conn(0x7f6088073110 msgr2=0x7f60880ac490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:00.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.484+0000 7f6087e00640 1 -- 172.21.15.28:0/1784012093 shutdown_connections 2024-02-21T23:08:00.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.484+0000 7f6087e00640 1 -- 172.21.15.28:0/1784012093 wait complete. 2024-02-21T23:08:00.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.487+0000 7f6087e00640 1 Processor -- start 2024-02-21T23:08:00.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.487+0000 7f6087e00640 1 -- start start 2024-02-21T23:08:00.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.487+0000 7f6087e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f608806dbc0 0x7f608814f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:00.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.487+0000 7f6087e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f608813e0c0 0x7f60881e1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:00.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.487+0000 7f6087e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f608813e520 0x7f6088157640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:00.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.487+0000 7f6087e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f608806ced0 con 0x7f608813e0c0 2024-02-21T23:08:00.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.487+0000 7f6087e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6088159070 con 0x7f608813e520 2024-02-21T23:08:00.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.487+0000 7f6087e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60880c78c0 con 0x7f608806dbc0 2024-02-21T23:08:00.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.487+0000 7f6086000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f608813e0c0 0x7f60881e1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:00.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.488+0000 7f6086000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f608813e0c0 0x7f60881e1ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48744/0 (socket says 172.21.15.28:48744) 2024-02-21T23:08:00.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.488+0000 7f6086000640 1 -- 172.21.15.28:0/89900822 learned_addr learned my addr 172.21.15.28:0/89900822 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:00.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.488+0000 7f6086000640 1 -- 172.21.15.28:0/89900822 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f608806dbc0 msgr2=0x7f608814f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:00.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.488+0000 7f6086a00640 1 --2- 172.21.15.28:0/89900822 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f608806dbc0 0x7f608814f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:00.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.488+0000 7f6087400640 1 --2- 172.21.15.28:0/89900822 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f608813e520 0x7f6088157640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:00.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.488+0000 7f6086000640 1 --2- 172.21.15.28:0/89900822 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f608806dbc0 0x7f608814f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:00.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.488+0000 7f6086000640 1 -- 172.21.15.28:0/89900822 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f608813e520 msgr2=0x7f6088157640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:00.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.488+0000 7f6086000640 1 --2- 172.21.15.28:0/89900822 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f608813e520 0x7f6088157640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:00.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.488+0000 7f6086000640 1 -- 172.21.15.28:0/89900822 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60881582d0 con 0x7f608813e0c0 2024-02-21T23:08:00.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.488+0000 7f6086a00640 1 --2- 172.21.15.28:0/89900822 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f608806dbc0 0x7f608814f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:08:00.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.488+0000 7f6086000640 1 --2- 172.21.15.28:0/89900822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f608813e0c0 0x7f60881e1ad0 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f607400c430 tx=0x7f607400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:00.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.489+0000 7f607f400640 1 -- 172.21.15.28:0/89900822 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6074017070 con 0x7f608813e0c0 2024-02-21T23:08:00.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.489+0000 7f607f400640 1 -- 172.21.15.28:0/89900822 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60740052d0 con 0x7f608813e0c0 2024-02-21T23:08:00.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.489+0000 7f607f400640 1 -- 172.21.15.28:0/89900822 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6074003830 con 0x7f608813e0c0 2024-02-21T23:08:00.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.489+0000 7f6087e00640 1 -- 172.21.15.28:0/89900822 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f608814f6d0 con 0x7f608813e0c0 2024-02-21T23:08:00.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.489+0000 7f6087e00640 1 -- 172.21.15.28:0/89900822 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60881dfc70 con 0x7f608813e0c0 2024-02-21T23:08:00.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.490+0000 7f6087e00640 1 -- 172.21.15.28:0/89900822 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f608813b690 con 0x7f608813e0c0 2024-02-21T23:08:00.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.491+0000 7f607f400640 1 -- 172.21.15.28:0/89900822 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f607401a070 con 0x7f608813e0c0 2024-02-21T23:08:00.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.492+0000 7f607f400640 1 --2- 172.21.15.28:0/89900822 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f606806b500 0x7f606806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:00.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.492+0000 7f607f400640 1 -- 172.21.15.28:0/89900822 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f6074089870 con 0x7f608813e0c0 2024-02-21T23:08:00.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.493+0000 7f6086a00640 1 --2- 172.21.15.28:0/89900822 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f606806b500 0x7f606806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:00.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.494+0000 7f6086a00640 1 --2- 172.21.15.28:0/89900822 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f606806b500 0x7f606806d9c0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f6078000f60 tx=0x7f6078009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:00.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.496+0000 7f607f400640 1 -- 172.21.15.28:0/89900822 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f6074058410 con 0x7f608813e0c0 2024-02-21T23:08:00.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:00.739+0000 7f6087e00640 1 -- 172.21.15.28:0/89900822 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f608804f290 con 0x7f608813e0c0 2024-02-21T23:08:01.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.283+0000 7f607f400640 1 -- 172.21.15.28:0/89900822 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v114) v1 ==== 104+0+0 (secure 0 0 0) 0x7f6074057d10 con 0x7f608813e0c0 2024-02-21T23:08:01.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.286+0000 7f6087e00640 1 -- 172.21.15.28:0/89900822 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f606806b500 msgr2=0x7f606806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:01.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.286+0000 7f6087e00640 1 --2- 172.21.15.28:0/89900822 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f606806b500 0x7f606806d9c0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f6078000f60 tx=0x7f6078009040 comp rx=0 tx=0).stop 2024-02-21T23:08:01.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.286+0000 7f6087e00640 1 -- 172.21.15.28:0/89900822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f608813e0c0 msgr2=0x7f60881e1ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:01.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.286+0000 7f6087e00640 1 --2- 172.21.15.28:0/89900822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f608813e0c0 0x7f60881e1ad0 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f607400c430 tx=0x7f607400c8f0 comp rx=0 tx=0).stop 2024-02-21T23:08:01.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.287+0000 7f6087e00640 1 -- 172.21.15.28:0/89900822 shutdown_connections 2024-02-21T23:08:01.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.287+0000 7f6087e00640 1 --2- 172.21.15.28:0/89900822 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f606806b500 0x7f606806d9c0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:01.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.287+0000 7f6087e00640 1 --2- 172.21.15.28:0/89900822 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f608813e520 0x7f6088157640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:01.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.287+0000 7f6087e00640 1 --2- 172.21.15.28:0/89900822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f608813e0c0 0x7f60881e1ad0 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:01.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.287+0000 7f6087e00640 1 --2- 172.21.15.28:0/89900822 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f608806dbc0 0x7f608814f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:01.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.287+0000 7f6087e00640 1 -- 172.21.15.28:0/89900822 >> 172.21.15.28:0/89900822 conn(0x7f6088073110 msgr2=0x7f60880c9720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:01.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.287+0000 7f6087e00640 1 -- 172.21.15.28:0/89900822 shutdown_connections 2024-02-21T23:08:01.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.287+0000 7f6087e00640 1 -- 172.21.15.28:0/89900822 wait complete. 2024-02-21T23:08:01.298 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-02-21T23:08:01.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.534+0000 7ff11a400640 1 Processor -- start 2024-02-21T23:08:01.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.535+0000 7ff11a400640 1 -- start start 2024-02-21T23:08:01.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.536+0000 7ff11a400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff11406e780 0x7ff11406eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:01.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.536+0000 7ff11a400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7ff1140c5290 0x7ff11415d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:01.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.536+0000 7ff11a400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff114153cb0 con 0x7ff1140a5fc0 2024-02-21T23:08:01.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.536+0000 7ff11a400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1141543d0 con 0x7ff1140c5290 2024-02-21T23:08:01.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.536+0000 7ff11a400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1140c5ad0 con 0x7ff11406e780 2024-02-21T23:08:01.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.536+0000 7ff10be00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff1140a5fc0 0x7ff1140c4b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60792/0 (socket says 172.21.15.28:60792) 2024-02-21T23:08:01.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.536+0000 7ff10be00640 1 -- 172.21.15.28:0/1580226293 learned_addr learned my addr 172.21.15.28:0/1580226293 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:08:01.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.537+0000 7ff113400640 1 --2- 172.21.15.28:0/1580226293 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff11406e780 0x7ff11406eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:01.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.537+0000 7ff113e00640 1 --2- 172.21.15.28:0/1580226293 >> v2:172.21.15.81:3300/0 conn(0x7ff1140c5290 0x7ff11415d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:01.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.537+0000 7ff112a00640 1 -- 172.21.15.28:0/1580226293 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3249449654 0 0) 0x7ff114153cb0 con 0x7ff1140a5fc0 2024-02-21T23:08:01.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.537+0000 7ff112a00640 1 -- 172.21.15.28:0/1580226293 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0f8003520 con 0x7ff1140a5fc0 2024-02-21T23:08:01.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.537+0000 7ff112a00640 1 -- 172.21.15.28:0/1580226293 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2389850033 0 0) 0x7ff0f8003520 con 0x7ff1140a5fc0 2024-02-21T23:08:01.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.537+0000 7ff112a00640 1 -- 172.21.15.28:0/1580226293 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff11406e780 msgr2=0x7ff11406eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:01.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.537+0000 7ff112a00640 1 --2- 172.21.15.28:0/1580226293 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff11406e780 0x7ff11406eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:01.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.537+0000 7ff112a00640 1 -- 172.21.15.28:0/1580226293 >> v2:172.21.15.81:3300/0 conn(0x7ff1140c5290 msgr2=0x7ff11415d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:01.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.537+0000 7ff112a00640 1 --2- 172.21.15.28:0/1580226293 >> v2:172.21.15.81:3300/0 conn(0x7ff1140c5290 0x7ff11415d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:01.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.538+0000 7ff112a00640 1 -- 172.21.15.28:0/1580226293 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1140c5d20 con 0x7ff1140a5fc0 2024-02-21T23:08:01.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.538+0000 7ff112a00640 1 -- 172.21.15.28:0/1580226293 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7ff0fc0017f0 con 0x7ff1140a5fc0 2024-02-21T23:08:01.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.538+0000 7ff112a00640 1 -- 172.21.15.28:0/1580226293 >> v1:172.21.15.28:6789/0 conn(0x7ff1140a5fc0 legacy=0x7ff1140c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:01.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.538+0000 7ff112a00640 1 --2- 172.21.15.28:0/1580226293 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0f80039b0 0x7ff0f8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:01.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.538+0000 7ff112a00640 1 --2- 172.21.15.28:0/1580226293 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0f8004800 0x7ff0f8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:01.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.538+0000 7ff112a00640 1 --2- 172.21.15.28:0/1580226293 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff11406e780 0x7ff0f80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:01.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.538+0000 7ff112a00640 1 -- 172.21.15.28:0/1580226293 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1140c5ad0 con 0x7ff0f8004800 2024-02-21T23:08:01.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.538+0000 7ff112a00640 1 -- 172.21.15.28:0/1580226293 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f800b9c0 con 0x7ff0f80039b0 2024-02-21T23:08:01.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.538+0000 7ff112a00640 1 -- 172.21.15.28:0/1580226293 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f800bb90 con 0x7ff11406e780 2024-02-21T23:08:01.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.538+0000 7ff10be00640 1 --2- 172.21.15.28:0/1580226293 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0f8004800 0x7ff0f8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:01.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.538+0000 7ff113400640 1 --2- 172.21.15.28:0/1580226293 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0f80039b0 0x7ff0f8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:01.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.538+0000 7ff113e00640 1 --2- 172.21.15.28:0/1580226293 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff11406e780 0x7ff0f80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:01.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.538+0000 7ff10be00640 1 -- 172.21.15.28:0/1580226293 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff11406e780 msgr2=0x7ff0f80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:01.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.538+0000 7ff10be00640 1 --2- 172.21.15.28:0/1580226293 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff11406e780 0x7ff0f80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:01.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.538+0000 7ff10be00640 1 -- 172.21.15.28:0/1580226293 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0f80039b0 msgr2=0x7ff0f8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:01.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.538+0000 7ff10be00640 1 --2- 172.21.15.28:0/1580226293 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0f80039b0 0x7ff0f8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:01.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.538+0000 7ff10be00640 1 -- 172.21.15.28:0/1580226293 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0f8003520 con 0x7ff0f8004800 2024-02-21T23:08:01.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.540+0000 7ff10be00640 1 --2- 172.21.15.28:0/1580226293 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0f8004800 0x7ff0f8004c80 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc00c190 tx=0x7ff0f8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b4983ffc1d07f4 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:01.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.540+0000 7ff112a00640 1 -- 172.21.15.28:0/1580226293 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0fc007c80 con 0x7ff0f8004800 2024-02-21T23:08:01.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.540+0000 7ff112a00640 1 -- 172.21.15.28:0/1580226293 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0fc030070 con 0x7ff0f8004800 2024-02-21T23:08:01.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.540+0000 7ff112a00640 1 -- 172.21.15.28:0/1580226293 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0fc0074b0 con 0x7ff0f8004800 2024-02-21T23:08:01.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.540+0000 7ff11a400640 1 -- 172.21.15.28:0/1580226293 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0f8004800 msgr2=0x7ff0f8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:01.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.540+0000 7ff11a400640 1 --2- 172.21.15.28:0/1580226293 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0f8004800 0x7ff0f8004c80 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc00c190 tx=0x7ff0f8005830 comp rx=0 tx=0).stop 2024-02-21T23:08:01.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.540+0000 7ff113e00640 1 -- 172.21.15.28:0/1580226293 reap_dead start 2024-02-21T23:08:01.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.540+0000 7ff11a400640 1 -- 172.21.15.28:0/1580226293 shutdown_connections 2024-02-21T23:08:01.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.541+0000 7ff11a400640 1 -- 172.21.15.28:0/1580226293 >> 172.21.15.28:0/1580226293 conn(0x7ff11406b830 msgr2=0x7ff1140c67f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:01.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.541+0000 7ff11a400640 1 -- 172.21.15.28:0/1580226293 shutdown_connections 2024-02-21T23:08:01.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.541+0000 7ff11a400640 1 -- 172.21.15.28:0/1580226293 wait complete. 2024-02-21T23:08:01.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.542+0000 7ff11a400640 1 Processor -- start 2024-02-21T23:08:01.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.543+0000 7ff11a400640 1 -- start start 2024-02-21T23:08:01.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.543+0000 7ff11a400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0f8004800 0x7ff11414a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:01.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.543+0000 7ff11a400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff11414b560 0x7ff1140a6470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:01.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.543+0000 7ff11a400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff11414aad0 0x7ff114151130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:01.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.543+0000 7ff11a400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff114153960 con 0x7ff11414b560 2024-02-21T23:08:01.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.543+0000 7ff11a400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1140c46f0 con 0x7ff0f8004800 2024-02-21T23:08:01.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.543+0000 7ff11a400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff11406d110 con 0x7ff11414aad0 2024-02-21T23:08:01.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.544+0000 7ff10be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff11414b560 0x7ff1140a6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:01.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.544+0000 7ff10be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff11414b560 0x7ff1140a6470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48750/0 (socket says 172.21.15.28:48750) 2024-02-21T23:08:01.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.544+0000 7ff10be00640 1 -- 172.21.15.28:0/945878054 learned_addr learned my addr 172.21.15.28:0/945878054 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:01.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.544+0000 7ff113400640 1 --2- 172.21.15.28:0/945878054 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0f8004800 0x7ff11414a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:01.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.544+0000 7ff113e00640 1 --2- 172.21.15.28:0/945878054 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff11414aad0 0x7ff114151130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:01.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.544+0000 7ff10be00640 1 -- 172.21.15.28:0/945878054 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff11414aad0 msgr2=0x7ff114151130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:01.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.544+0000 7ff10be00640 1 --2- 172.21.15.28:0/945878054 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff11414aad0 0x7ff114151130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:01.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.544+0000 7ff10be00640 1 -- 172.21.15.28:0/945878054 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0f8004800 msgr2=0x7ff11414a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:01.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.544+0000 7ff10be00640 1 --2- 172.21.15.28:0/945878054 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0f8004800 0x7ff11414a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:01.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.544+0000 7ff10be00640 1 -- 172.21.15.28:0/945878054 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1141e8e70 con 0x7ff11414b560 2024-02-21T23:08:01.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.544+0000 7ff113e00640 1 --2- 172.21.15.28:0/945878054 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff11414aad0 0x7ff114151130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:08:01.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.545+0000 7ff10be00640 1 --2- 172.21.15.28:0/945878054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff11414b560 0x7ff1140a6470 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc02c480 tx=0x7ff0fc001160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:01.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.545+0000 7ff10b400640 1 -- 172.21.15.28:0/945878054 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0fc007c80 con 0x7ff11414b560 2024-02-21T23:08:01.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.545+0000 7ff10b400640 1 -- 172.21.15.28:0/945878054 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0fc007480 con 0x7ff11414b560 2024-02-21T23:08:01.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.545+0000 7ff10b400640 1 -- 172.21.15.28:0/945878054 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0fc0057a0 con 0x7ff11414b560 2024-02-21T23:08:01.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.545+0000 7ff11a400640 1 -- 172.21.15.28:0/945878054 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1141e9150 con 0x7ff11414b560 2024-02-21T23:08:01.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.545+0000 7ff11a400640 1 -- 172.21.15.28:0/945878054 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1141e9710 con 0x7ff11414b560 2024-02-21T23:08:01.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.546+0000 7ff11a400640 1 -- 172.21.15.28:0/945878054 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1140c0a80 con 0x7ff11414b560 2024-02-21T23:08:01.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.547+0000 7ff10b400640 1 -- 172.21.15.28:0/945878054 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff0fc035470 con 0x7ff11414b560 2024-02-21T23:08:01.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.548+0000 7ff10b400640 1 --2- 172.21.15.28:0/945878054 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff0f006b550 0x7ff0f006da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:01.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.548+0000 7ff10b400640 1 -- 172.21.15.28:0/945878054 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6273+0+0 (secure 0 0 0) 0x7ff0fc0abb80 con 0x7ff11414b560 2024-02-21T23:08:01.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.549+0000 7ff113400640 1 --2- 172.21.15.28:0/945878054 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff0f006b550 0x7ff0f006da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:01.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.550+0000 7ff113400640 1 --2- 172.21.15.28:0/945878054 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff0f006b550 0x7ff0f006da10 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7ff1141dd5b0 tx=0x7ff104008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:01.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.553+0000 7ff10b400640 1 -- 172.21.15.28:0/945878054 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ff0fc04e060 con 0x7ff11414b560 2024-02-21T23:08:01.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:01.755+0000 7ff11a400640 1 -- 172.21.15.28:0/945878054 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ff114067f30 con 0x7ff11414b560 2024-02-21T23:08:02.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.304+0000 7ff10b400640 1 -- 172.21.15.28:0/945878054 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v113) v1 ==== 181+0+0 (secure 0 0 0) 0x7ff0fc07a830 con 0x7ff11414b560 2024-02-21T23:08:02.293 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' removed 2024-02-21T23:08:02.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.307+0000 7ff11a400640 1 -- 172.21.15.28:0/945878054 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff0f006b550 msgr2=0x7ff0f006da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:02.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.307+0000 7ff11a400640 1 --2- 172.21.15.28:0/945878054 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff0f006b550 0x7ff0f006da10 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7ff1141dd5b0 tx=0x7ff104008040 comp rx=0 tx=0).stop 2024-02-21T23:08:02.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.307+0000 7ff11a400640 1 -- 172.21.15.28:0/945878054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff11414b560 msgr2=0x7ff1140a6470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:02.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.307+0000 7ff11a400640 1 --2- 172.21.15.28:0/945878054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff11414b560 0x7ff1140a6470 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc02c480 tx=0x7ff0fc001160 comp rx=0 tx=0).stop 2024-02-21T23:08:02.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.309+0000 7ff11a400640 1 -- 172.21.15.28:0/945878054 shutdown_connections 2024-02-21T23:08:02.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.309+0000 7ff11a400640 1 --2- 172.21.15.28:0/945878054 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff0f006b550 0x7ff0f006da10 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:02.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.309+0000 7ff11a400640 1 --2- 172.21.15.28:0/945878054 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff11414aad0 0x7ff114151130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:02.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.309+0000 7ff11a400640 1 --2- 172.21.15.28:0/945878054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff11414b560 0x7ff1140a6470 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:02.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.309+0000 7ff11a400640 1 --2- 172.21.15.28:0/945878054 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0f8004800 0x7ff11414a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:02.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.309+0000 7ff11a400640 1 -- 172.21.15.28:0/945878054 >> 172.21.15.28:0/945878054 conn(0x7ff11406b830 msgr2=0x7ff11414eed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:02.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.310+0000 7ff11a400640 1 -- 172.21.15.28:0/945878054 shutdown_connections 2024-02-21T23:08:02.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.310+0000 7ff11a400640 1 -- 172.21.15.28:0/945878054 wait complete. 2024-02-21T23:08:02.324 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-02-21T23:08:02.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.563+0000 7f7007600640 1 Processor -- start 2024-02-21T23:08:02.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.563+0000 7f7007600640 1 -- start start 2024-02-21T23:08:02.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.565+0000 7f7007600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f70000bfba0 0x7f7000155dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:02.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.565+0000 7f7007600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70000c6d10 con 0x7f70000c4d10 2024-02-21T23:08:02.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.565+0000 7f7007600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70001552f0 con 0x7f70000afbc0 2024-02-21T23:08:02.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.565+0000 7f7007600640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7000155940 con 0x7f70000bfba0 2024-02-21T23:08:02.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.565+0000 7f6fffe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f70000c4d10 0x7f70000c7310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60802/0 (socket says 172.21.15.28:60802) 2024-02-21T23:08:02.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.565+0000 7f6fffe00640 1 -- 172.21.15.28:0/2286800274 learned_addr learned my addr 172.21.15.28:0/2286800274 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:08:02.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.565+0000 7f6fff400640 1 -- 172.21.15.28:0/2286800274 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2330870023 0 0) 0x7f70000c6d10 con 0x7f70000c4d10 2024-02-21T23:08:02.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.565+0000 7f6fff400640 1 -- 172.21.15.28:0/2286800274 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fe4003520 con 0x7f70000c4d10 2024-02-21T23:08:02.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.565+0000 7f7005600640 1 --2- 172.21.15.28:0/2286800274 >> v2:172.21.15.81:3300/0 conn(0x7f70000bfba0 0x7f7000155dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:02.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.566+0000 7f6fff400640 1 -- 172.21.15.28:0/2286800274 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 84003248 0 0) 0x7f6fe4003520 con 0x7f70000c4d10 2024-02-21T23:08:02.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.566+0000 7f6fff400640 1 -- 172.21.15.28:0/2286800274 >> v2:172.21.15.81:3300/0 conn(0x7f70000bfba0 msgr2=0x7f7000155dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:02.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.566+0000 7f6fff400640 1 --2- 172.21.15.28:0/2286800274 >> v2:172.21.15.81:3300/0 conn(0x7f70000bfba0 0x7f7000155dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:02.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.566+0000 7f6fff400640 1 -- 172.21.15.28:0/2286800274 >> v1:172.21.15.81:6789/0 conn(0x7f70000afbc0 legacy=0x7f700006cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:02.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.566+0000 7f6fff400640 1 -- 172.21.15.28:0/2286800274 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70001555d0 con 0x7f70000c4d10 2024-02-21T23:08:02.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.566+0000 7f6fff400640 1 -- 172.21.15.28:0/2286800274 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f6fec001670 con 0x7f70000c4d10 2024-02-21T23:08:02.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.566+0000 7f6fff400640 1 -- 172.21.15.28:0/2286800274 >> v1:172.21.15.28:6789/0 conn(0x7f70000c4d10 legacy=0x7f70000c7310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:02.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.566+0000 7f6fff400640 1 --2- 172.21.15.28:0/2286800274 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6fe4003a00 0x7f6fe4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:02.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.566+0000 7f6fff400640 1 --2- 172.21.15.28:0/2286800274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6fe4004800 0x7f6fe4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:02.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.566+0000 7f6fff400640 1 --2- 172.21.15.28:0/2286800274 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6fe4005210 0x7f6fe400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:02.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.566+0000 7f6fff400640 1 -- 172.21.15.28:0/2286800274 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7000155940 con 0x7f6fe4004800 2024-02-21T23:08:02.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.566+0000 7f6fff400640 1 -- 172.21.15.28:0/2286800274 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe400c160 con 0x7f6fe4005210 2024-02-21T23:08:02.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.566+0000 7f6fff400640 1 -- 172.21.15.28:0/2286800274 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe400c360 con 0x7f6fe4003a00 2024-02-21T23:08:02.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.566+0000 7f6fffe00640 1 --2- 172.21.15.28:0/2286800274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6fe4004800 0x7f6fe4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:02.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.567+0000 7f6fffe00640 1 -- 172.21.15.28:0/2286800274 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6fe4003a00 msgr2=0x7f6fe4003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:02.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.567+0000 7f7004c00640 1 --2- 172.21.15.28:0/2286800274 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6fe4003a00 0x7f6fe4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:02.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.567+0000 7f6fffe00640 1 --2- 172.21.15.28:0/2286800274 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6fe4003a00 0x7f6fe4003e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:02.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.567+0000 7f6fffe00640 1 -- 172.21.15.28:0/2286800274 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6fe4005210 msgr2=0x7f6fe400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:02.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.567+0000 7f6fffe00640 1 --2- 172.21.15.28:0/2286800274 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6fe4005210 0x7f6fe400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:02.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.567+0000 7f6fffe00640 1 -- 172.21.15.28:0/2286800274 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fe4003520 con 0x7f6fe4004800 2024-02-21T23:08:02.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.567+0000 7f7005600640 1 -- 172.21.15.28:0/2286800274 reap_dead start 2024-02-21T23:08:02.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.568+0000 7f6fffe00640 1 --2- 172.21.15.28:0/2286800274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6fe4004800 0x7f6fe4004c80 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f70000c5150 tx=0x7f6fe4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e85f3453e36feae server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:02.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.568+0000 7f6fff400640 1 -- 172.21.15.28:0/2286800274 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fec007c40 con 0x7f6fe4004800 2024-02-21T23:08:02.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.568+0000 7f6fff400640 1 -- 172.21.15.28:0/2286800274 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fec030070 con 0x7f6fe4004800 2024-02-21T23:08:02.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.568+0000 7f6fff400640 1 -- 172.21.15.28:0/2286800274 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fec005d70 con 0x7f6fe4004800 2024-02-21T23:08:02.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.569+0000 7f7007600640 1 -- 172.21.15.28:0/2286800274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6fe4004800 msgr2=0x7f6fe4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:02.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.569+0000 7f7007600640 1 --2- 172.21.15.28:0/2286800274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6fe4004800 0x7f6fe4004c80 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f70000c5150 tx=0x7f6fe4005830 comp rx=0 tx=0).stop 2024-02-21T23:08:02.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.569+0000 7f7007600640 1 -- 172.21.15.28:0/2286800274 shutdown_connections 2024-02-21T23:08:02.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.569+0000 7f7007600640 1 --2- 172.21.15.28:0/2286800274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6fe4004800 0x7f6fe4004c80 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:02.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.569+0000 7f7007600640 1 -- 172.21.15.28:0/2286800274 >> 172.21.15.28:0/2286800274 conn(0x7f70000725a0 msgr2=0x7f70000729b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:02.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.569+0000 7f7007600640 1 -- 172.21.15.28:0/2286800274 shutdown_connections 2024-02-21T23:08:02.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.569+0000 7f7007600640 1 -- 172.21.15.28:0/2286800274 wait complete. 2024-02-21T23:08:02.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.570+0000 7f7007600640 1 Processor -- start 2024-02-21T23:08:02.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.571+0000 7f7007600640 1 -- start start 2024-02-21T23:08:02.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.572+0000 7f7007600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6fe4004800 0x7f70001488c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:02.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.572+0000 7f7007600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70001583a0 0x7f70001ee4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:02.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.572+0000 7f7007600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f70000c99c0 0x7f7000148040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:02.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.572+0000 7f7007600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70000c6960 con 0x7f70001583a0 2024-02-21T23:08:02.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.572+0000 7f7007600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70000c67e0 con 0x7f6fe4004800 2024-02-21T23:08:02.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.572+0000 7f7007600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7000155dd0 con 0x7f70000c99c0 2024-02-21T23:08:02.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.572+0000 7f6fffe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70001583a0 0x7f70001ee4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:02.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.572+0000 7f6fffe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70001583a0 0x7f70001ee4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48766/0 (socket says 172.21.15.28:48766) 2024-02-21T23:08:02.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.572+0000 7f6fffe00640 1 -- 172.21.15.28:0/3215600002 learned_addr learned my addr 172.21.15.28:0/3215600002 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:02.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.572+0000 7f7004c00640 1 --2- 172.21.15.28:0/3215600002 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6fe4004800 0x7f70001488c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:02.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.572+0000 7f6fffe00640 1 -- 172.21.15.28:0/3215600002 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f70000c99c0 msgr2=0x7f7000148040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:02.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.572+0000 7f7005600640 1 --2- 172.21.15.28:0/3215600002 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f70000c99c0 0x7f7000148040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:02.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.572+0000 7f6fffe00640 1 --2- 172.21.15.28:0/3215600002 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f70000c99c0 0x7f7000148040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:02.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.572+0000 7f6fffe00640 1 -- 172.21.15.28:0/3215600002 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6fe4004800 msgr2=0x7f70001488c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:02.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.572+0000 7f6fffe00640 1 --2- 172.21.15.28:0/3215600002 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6fe4004800 0x7f70001488c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:02.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.572+0000 7f6fffe00640 1 -- 172.21.15.28:0/3215600002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7000159180 con 0x7f70001583a0 2024-02-21T23:08:02.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.572+0000 7f7004c00640 1 --2- 172.21.15.28:0/3215600002 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6fe4004800 0x7f70001488c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:08:02.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.573+0000 7f6fffe00640 1 --2- 172.21.15.28:0/3215600002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70001583a0 0x7f70001ee4e0 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f6fec00c1a0 tx=0x7f6fec02e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:02.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.573+0000 7f6ffcc00640 1 -- 172.21.15.28:0/3215600002 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fec007c40 con 0x7f70001583a0 2024-02-21T23:08:02.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.573+0000 7f6ffcc00640 1 -- 172.21.15.28:0/3215600002 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fec000ed0 con 0x7f70001583a0 2024-02-21T23:08:02.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.573+0000 7f7007600640 1 -- 172.21.15.28:0/3215600002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7000159460 con 0x7f70001583a0 2024-02-21T23:08:02.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.573+0000 7f6ffcc00640 1 -- 172.21.15.28:0/3215600002 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fec02cdf0 con 0x7f70001583a0 2024-02-21T23:08:02.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.573+0000 7f7007600640 1 -- 172.21.15.28:0/3215600002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7000158060 con 0x7f70001583a0 2024-02-21T23:08:02.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.574+0000 7f7007600640 1 -- 172.21.15.28:0/3215600002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7000156340 con 0x7f70001583a0 2024-02-21T23:08:02.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.576+0000 7f6ffcc00640 1 -- 172.21.15.28:0/3215600002 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6fec007070 con 0x7f70001583a0 2024-02-21T23:08:02.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.577+0000 7f6ffcc00640 1 --2- 172.21.15.28:0/3215600002 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6fe006b500 0x7f6fe006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:02.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.577+0000 7f6ffcc00640 1 -- 172.21.15.28:0/3215600002 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5835+0+0 (secure 0 0 0) 0x7f6fec0aba30 con 0x7f70001583a0 2024-02-21T23:08:02.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.577+0000 7f7004c00640 1 --2- 172.21.15.28:0/3215600002 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6fe006b500 0x7f6fe006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:02.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.578+0000 7f7004c00640 1 --2- 172.21.15.28:0/3215600002 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6fe006b500 0x7f6fe006d9c0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f700014ea20 tx=0x7f6ff0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:02.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.581+0000 7f6ffcc00640 1 -- 172.21.15.28:0/3215600002 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7000156340 con 0x7f70001583a0 2024-02-21T23:08:02.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:02.779+0000 7f7007600640 1 -- 172.21.15.28:0/3215600002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f7000068640 con 0x7f70001583a0 2024-02-21T23:08:03.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.314+0000 7f6ffcc00640 1 -- 172.21.15.28:0/3215600002 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v114) v1 ==== 169+0+0 (secure 0 0 0) 0x7f6fec07a790 con 0x7f70001583a0 2024-02-21T23:08:03.303 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' removed 2024-02-21T23:08:03.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.318+0000 7f7007600640 1 -- 172.21.15.28:0/3215600002 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6fe006b500 msgr2=0x7f6fe006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:03.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.318+0000 7f7007600640 1 --2- 172.21.15.28:0/3215600002 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6fe006b500 0x7f6fe006d9c0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f700014ea20 tx=0x7f6ff0009040 comp rx=0 tx=0).stop 2024-02-21T23:08:03.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.318+0000 7f7007600640 1 -- 172.21.15.28:0/3215600002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70001583a0 msgr2=0x7f70001ee4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:03.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.318+0000 7f7007600640 1 --2- 172.21.15.28:0/3215600002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70001583a0 0x7f70001ee4e0 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f6fec00c1a0 tx=0x7f6fec02e070 comp rx=0 tx=0).stop 2024-02-21T23:08:03.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.318+0000 7f7007600640 1 -- 172.21.15.28:0/3215600002 shutdown_connections 2024-02-21T23:08:03.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.318+0000 7f7007600640 1 --2- 172.21.15.28:0/3215600002 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6fe006b500 0x7f6fe006d9c0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:03.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.318+0000 7f7007600640 1 --2- 172.21.15.28:0/3215600002 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f70000c99c0 0x7f7000148040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:03.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.318+0000 7f7007600640 1 --2- 172.21.15.28:0/3215600002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70001583a0 0x7f70001ee4e0 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:03.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.318+0000 7f7007600640 1 --2- 172.21.15.28:0/3215600002 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6fe4004800 0x7f70001488c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:03.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.318+0000 7f7007600640 1 -- 172.21.15.28:0/3215600002 >> 172.21.15.28:0/3215600002 conn(0x7f70000725a0 msgr2=0x7f70000c3690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:03.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.319+0000 7f7007600640 1 -- 172.21.15.28:0/3215600002 shutdown_connections 2024-02-21T23:08:03.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.319+0000 7f7007600640 1 -- 172.21.15.28:0/3215600002 wait complete. 2024-02-21T23:08:03.336 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.b mds_join_fs 2024-02-21T23:08:03.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.571+0000 7f6a7fe00640 1 Processor -- start 2024-02-21T23:08:03.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.572+0000 7f6a7fe00640 1 -- start start 2024-02-21T23:08:03.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.573+0000 7f6a7fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6a8006ec20 0x7f6a80152780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:03.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.573+0000 7f6a7fe00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f6a800c59b0 0x7f6a8006f000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:03.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.573+0000 7f6a7fe00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a8014f600 con 0x7f6a80155b90 2024-02-21T23:08:03.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.573+0000 7f6a7fe00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a800c2fd0 con 0x7f6a8006ec20 2024-02-21T23:08:03.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.573+0000 7f6a7fe00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a800729b0 con 0x7f6a800c59b0 2024-02-21T23:08:03.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.573+0000 7f6a7ea00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6a8006ec20 0x7f6a80152780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:03.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.573+0000 7f6a7ea00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6a8006ec20 0x7f6a80152780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48782/0 (socket says 172.21.15.28:48782) 2024-02-21T23:08:03.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.573+0000 7f6a7ea00640 1 -- 172.21.15.28:0/287911784 learned_addr learned my addr 172.21.15.28:0/287911784 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:03.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.574+0000 7f6a7e000640 1 --2- 172.21.15.28:0/287911784 >> v2:172.21.15.81:3300/0 conn(0x7f6a800c59b0 0x7f6a8006f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:03.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.574+0000 7f6a7ea00640 1 -- 172.21.15.28:0/287911784 >> v2:172.21.15.81:3300/0 conn(0x7f6a800c59b0 msgr2=0x7f6a8006f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:03.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.574+0000 7f6a7ea00640 1 --2- 172.21.15.28:0/287911784 >> v2:172.21.15.81:3300/0 conn(0x7f6a800c59b0 0x7f6a8006f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:03.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.574+0000 7f6a7ea00640 1 -- 172.21.15.28:0/287911784 >> v1:172.21.15.28:6789/0 conn(0x7f6a80155b90 legacy=0x7f6a800722a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:03.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.574+0000 7f6a7d600640 1 -- 172.21.15.28:0/287911784 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2972624797 0 0) 0x7f6a8014f600 con 0x7f6a80155b90 2024-02-21T23:08:03.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.574+0000 7f6a7ea00640 1 -- 172.21.15.28:0/287911784 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a800c5e90 con 0x7f6a8006ec20 2024-02-21T23:08:03.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.575+0000 7f6a7ea00640 1 --2- 172.21.15.28:0/287911784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a8006ec20 0x7f6a80152780 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f6a8006bf10 tx=0x7f6a7002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae49a7ad921b009e server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:03.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.575+0000 7f6a7d600640 1 -- 172.21.15.28:0/287911784 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a7002ca80 con 0x7f6a8006ec20 2024-02-21T23:08:03.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.575+0000 7f6a7d600640 1 -- 172.21.15.28:0/287911784 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a7002cc20 con 0x7f6a8006ec20 2024-02-21T23:08:03.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.575+0000 7f6a7d600640 1 -- 172.21.15.28:0/287911784 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a700343f0 con 0x7f6a8006ec20 2024-02-21T23:08:03.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.576+0000 7f6a7fe00640 1 -- 172.21.15.28:0/287911784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a8006ec20 msgr2=0x7f6a80152780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:03.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.576+0000 7f6a7fe00640 1 --2- 172.21.15.28:0/287911784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a8006ec20 0x7f6a80152780 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f6a8006bf10 tx=0x7f6a7002bee0 comp rx=0 tx=0).stop 2024-02-21T23:08:03.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.576+0000 7f6a7fe00640 1 -- 172.21.15.28:0/287911784 shutdown_connections 2024-02-21T23:08:03.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.576+0000 7f6a7fe00640 1 --2- 172.21.15.28:0/287911784 >> v2:172.21.15.81:3300/0 conn(0x7f6a800c59b0 0x7f6a8006f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:03.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.576+0000 7f6a7fe00640 1 --2- 172.21.15.28:0/287911784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a8006ec20 0x7f6a80152780 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:03.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.576+0000 7f6a7fe00640 1 -- 172.21.15.28:0/287911784 >> 172.21.15.28:0/287911784 conn(0x7f6a800a8530 msgr2=0x7f6a800a8910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:03.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.576+0000 7f6a7fe00640 1 -- 172.21.15.28:0/287911784 shutdown_connections 2024-02-21T23:08:03.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.576+0000 7f6a7fe00640 1 -- 172.21.15.28:0/287911784 wait complete. 2024-02-21T23:08:03.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.578+0000 7f6a7fe00640 1 Processor -- start 2024-02-21T23:08:03.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.578+0000 7f6a7fe00640 1 -- start start 2024-02-21T23:08:03.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.579+0000 7f6a7fe00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6a8006ec20 0x7f6a801e1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:03.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.579+0000 7f6a7fe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6a800c59b0 0x7f6a8014d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:03.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.579+0000 7f6a7fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a80155b90 0x7f6a8014dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:03.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.579+0000 7f6a7fe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a80151f80 con 0x7f6a80155b90 2024-02-21T23:08:03.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.579+0000 7f6a7fe00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a80151ca0 con 0x7f6a800c59b0 2024-02-21T23:08:03.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.579+0000 7f6a7fe00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a80152260 con 0x7f6a8006ec20 2024-02-21T23:08:03.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.579+0000 7f6a7f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a80155b90 0x7f6a8014dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:03.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.579+0000 7f6a7f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a80155b90 0x7f6a8014dbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48794/0 (socket says 172.21.15.28:48794) 2024-02-21T23:08:03.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.579+0000 7f6a7f400640 1 -- 172.21.15.28:0/531119849 learned_addr learned my addr 172.21.15.28:0/531119849 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:03.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.580+0000 7f6a7ea00640 1 --2- 172.21.15.28:0/531119849 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6a8006ec20 0x7f6a801e1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:03.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.580+0000 7f6a7e000640 1 --2- 172.21.15.28:0/531119849 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6a800c59b0 0x7f6a8014d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:03.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.580+0000 7f6a7f400640 1 -- 172.21.15.28:0/531119849 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6a8006ec20 msgr2=0x7f6a801e1ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:03.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.580+0000 7f6a7f400640 1 --2- 172.21.15.28:0/531119849 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6a8006ec20 0x7f6a801e1ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:03.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.580+0000 7f6a7f400640 1 -- 172.21.15.28:0/531119849 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6a800c59b0 msgr2=0x7f6a8014d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:03.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.580+0000 7f6a7f400640 1 --2- 172.21.15.28:0/531119849 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6a800c59b0 0x7f6a8014d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:03.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.580+0000 7f6a7f400640 1 -- 172.21.15.28:0/531119849 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a8014e100 con 0x7f6a80155b90 2024-02-21T23:08:03.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.580+0000 7f6a7f400640 1 --2- 172.21.15.28:0/531119849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a80155b90 0x7f6a8014dbc0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f6a7800c480 tx=0x7f6a7800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:03.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.581+0000 7f6a76a00640 1 -- 172.21.15.28:0/531119849 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a78005480 con 0x7f6a80155b90 2024-02-21T23:08:03.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.581+0000 7f6a76a00640 1 -- 172.21.15.28:0/531119849 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a780035f0 con 0x7f6a80155b90 2024-02-21T23:08:03.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.581+0000 7f6a76a00640 1 -- 172.21.15.28:0/531119849 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a780039c0 con 0x7f6a80155b90 2024-02-21T23:08:03.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.581+0000 7f6a7fe00640 1 -- 172.21.15.28:0/531119849 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a8014e3e0 con 0x7f6a80155b90 2024-02-21T23:08:03.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.581+0000 7f6a7fe00640 1 -- 172.21.15.28:0/531119849 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a801e2220 con 0x7f6a80155b90 2024-02-21T23:08:03.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.582+0000 7f6a7fe00640 1 -- 172.21.15.28:0/531119849 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a8004f290 con 0x7f6a80155b90 2024-02-21T23:08:03.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.583+0000 7f6a76a00640 1 -- 172.21.15.28:0/531119849 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6a78003b60 con 0x7f6a80155b90 2024-02-21T23:08:03.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.584+0000 7f6a76a00640 1 --2- 172.21.15.28:0/531119849 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6a5c06b670 0x7f6a5c06db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:03.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.584+0000 7f6a76a00640 1 -- 172.21.15.28:0/531119849 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f6a78089d80 con 0x7f6a80155b90 2024-02-21T23:08:03.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.584+0000 7f6a7ea00640 1 --2- 172.21.15.28:0/531119849 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6a5c06b670 0x7f6a5c06db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:03.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.585+0000 7f6a7ea00640 1 --2- 172.21.15.28:0/531119849 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6a5c06b670 0x7f6a5c06db30 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f6a70035000 tx=0x7f6a70030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:03.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.588+0000 7f6a76a00640 1 -- 172.21.15.28:0/531119849 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f6a780182b0 con 0x7f6a80155b90 2024-02-21T23:08:03.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.791+0000 7f6a7fe00640 1 -- 172.21.15.28:0/531119849 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.b", "name": "mds_join_fs"} v 0) v1 -- 0x7f6a800c5240 con 0x7f6a80155b90 2024-02-21T23:08:03.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.798+0000 7f6a76a00640 1 -- 172.21.15.28:0/531119849 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.b", "name": "mds_join_fs"}]=0 v38) v1 ==== 96+0+0 (secure 0 0 0) 0x7f6a78058d10 con 0x7f6a80155b90 2024-02-21T23:08:03.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.801+0000 7f6a7fe00640 1 -- 172.21.15.28:0/531119849 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6a5c06b670 msgr2=0x7f6a5c06db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:03.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.801+0000 7f6a7fe00640 1 --2- 172.21.15.28:0/531119849 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6a5c06b670 0x7f6a5c06db30 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f6a70035000 tx=0x7f6a70030000 comp rx=0 tx=0).stop 2024-02-21T23:08:03.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.801+0000 7f6a7fe00640 1 -- 172.21.15.28:0/531119849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a80155b90 msgr2=0x7f6a8014dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:03.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.801+0000 7f6a7fe00640 1 --2- 172.21.15.28:0/531119849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a80155b90 0x7f6a8014dbc0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f6a7800c480 tx=0x7f6a7800c940 comp rx=0 tx=0).stop 2024-02-21T23:08:03.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.801+0000 7f6a7fe00640 1 -- 172.21.15.28:0/531119849 shutdown_connections 2024-02-21T23:08:03.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.801+0000 7f6a7fe00640 1 --2- 172.21.15.28:0/531119849 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6a5c06b670 0x7f6a5c06db30 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:03.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.801+0000 7f6a7fe00640 1 --2- 172.21.15.28:0/531119849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a80155b90 0x7f6a8014dbc0 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:03.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.801+0000 7f6a7fe00640 1 --2- 172.21.15.28:0/531119849 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6a800c59b0 0x7f6a8014d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:03.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.801+0000 7f6a7fe00640 1 --2- 172.21.15.28:0/531119849 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6a8006ec20 0x7f6a801e1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:03.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.801+0000 7f6a7fe00640 1 -- 172.21.15.28:0/531119849 >> 172.21.15.28:0/531119849 conn(0x7f6a800a8530 msgr2=0x7f6a800c3b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:03.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.801+0000 7f6a7fe00640 1 -- 172.21.15.28:0/531119849 shutdown_connections 2024-02-21T23:08:03.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:03.801+0000 7f6a7fe00640 1 -- 172.21.15.28:0/531119849 wait complete. 2024-02-21T23:08:03.808 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.d mds_join_fs 2024-02-21T23:08:04.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.042+0000 7f3b33200640 1 Processor -- start 2024-02-21T23:08:04.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.042+0000 7f3b33200640 1 -- start start 2024-02-21T23:08:04.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.044+0000 7f3b33200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3b34153470 0x7f3b340c4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:04.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.044+0000 7f3b33200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3b340c1080 0x7f3b34159c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:04.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.044+0000 7f3b33200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f3b3415d170 0x7f3b3406ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:04.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.044+0000 7f3b33200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b3415cce0 con 0x7f3b340c1080 2024-02-21T23:08:04.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.044+0000 7f3b33200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b340c5150 con 0x7f3b3415d170 2024-02-21T23:08:04.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.044+0000 7f3b33200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b3415a1a0 con 0x7f3b34153470 2024-02-21T23:08:04.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.044+0000 7f3b31400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3b340c1080 0x7f3b34159c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:04.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.044+0000 7f3b31400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3b340c1080 0x7f3b34159c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48802/0 (socket says 172.21.15.28:48802) 2024-02-21T23:08:04.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.044+0000 7f3b31400640 1 -- 172.21.15.28:0/3335919095 learned_addr learned my addr 172.21.15.28:0/3335919095 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:04.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.045+0000 7f3b32800640 1 --2- 172.21.15.28:0/3335919095 >> v2:172.21.15.81:3300/0 conn(0x7f3b3415d170 0x7f3b3406ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:04.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.045+0000 7f3b31e00640 1 --2- 172.21.15.28:0/3335919095 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3b34153470 0x7f3b340c4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:04.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.045+0000 7f3b31400640 1 -- 172.21.15.28:0/3335919095 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3b34153470 msgr2=0x7f3b340c4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:04.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.045+0000 7f3b31400640 1 --2- 172.21.15.28:0/3335919095 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3b34153470 0x7f3b340c4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:04.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.045+0000 7f3b31400640 1 -- 172.21.15.28:0/3335919095 >> v2:172.21.15.81:3300/0 conn(0x7f3b3415d170 msgr2=0x7f3b3406ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:04.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.045+0000 7f3b31400640 1 --2- 172.21.15.28:0/3335919095 >> v2:172.21.15.81:3300/0 conn(0x7f3b3415d170 0x7f3b3406ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:04.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.045+0000 7f3b31400640 1 -- 172.21.15.28:0/3335919095 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b340c1590 con 0x7f3b340c1080 2024-02-21T23:08:04.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.046+0000 7f3b31400640 1 --2- 172.21.15.28:0/3335919095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b340c1080 0x7f3b34159c60 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f3b340c6540 tx=0x7f3b2402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=894291f3bdf02339 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:04.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.047+0000 7f3b30a00640 1 -- 172.21.15.28:0/3335919095 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b24035020 con 0x7f3b340c1080 2024-02-21T23:08:04.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.047+0000 7f3b30a00640 1 -- 172.21.15.28:0/3335919095 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b24031870 con 0x7f3b340c1080 2024-02-21T23:08:04.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.047+0000 7f3b30a00640 1 -- 172.21.15.28:0/3335919095 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b24031c40 con 0x7f3b340c1080 2024-02-21T23:08:04.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.047+0000 7f3b33200640 1 -- 172.21.15.28:0/3335919095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b340c1080 msgr2=0x7f3b34159c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:04.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.047+0000 7f3b33200640 1 --2- 172.21.15.28:0/3335919095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b340c1080 0x7f3b34159c60 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f3b340c6540 tx=0x7f3b2402b910 comp rx=0 tx=0).stop 2024-02-21T23:08:04.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.047+0000 7f3b33200640 1 -- 172.21.15.28:0/3335919095 shutdown_connections 2024-02-21T23:08:04.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.047+0000 7f3b33200640 1 --2- 172.21.15.28:0/3335919095 >> v2:172.21.15.81:3300/0 conn(0x7f3b3415d170 0x7f3b3406ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:04.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.047+0000 7f3b33200640 1 --2- 172.21.15.28:0/3335919095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b340c1080 0x7f3b34159c60 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:04.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.047+0000 7f3b33200640 1 --2- 172.21.15.28:0/3335919095 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3b34153470 0x7f3b340c4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:04.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.047+0000 7f3b33200640 1 -- 172.21.15.28:0/3335919095 >> 172.21.15.28:0/3335919095 conn(0x7f3b340a8530 msgr2=0x7f3b340a8910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:04.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.048+0000 7f3b33200640 1 -- 172.21.15.28:0/3335919095 shutdown_connections 2024-02-21T23:08:04.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.048+0000 7f3b33200640 1 -- 172.21.15.28:0/3335919095 wait complete. 2024-02-21T23:08:04.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.049+0000 7f3b33200640 1 Processor -- start 2024-02-21T23:08:04.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.049+0000 7f3b33200640 1 -- start start 2024-02-21T23:08:04.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.050+0000 7f3b33200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3b340c1080 0x7f3b34159490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:04.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.050+0000 7f3b33200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b34153470 0x7f3b3414c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:04.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.050+0000 7f3b33200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b3415d170 0x7f3b3414ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:04.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.050+0000 7f3b33200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b3415d620 con 0x7f3b3415d170 2024-02-21T23:08:04.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.050+0000 7f3b33200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b3415c970 con 0x7f3b34153470 2024-02-21T23:08:04.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.050+0000 7f3b33200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b3406b3b0 con 0x7f3b340c1080 2024-02-21T23:08:04.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.051+0000 7f3b32800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b3415d170 0x7f3b3414ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:04.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.051+0000 7f3b32800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b3415d170 0x7f3b3414ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48810/0 (socket says 172.21.15.28:48810) 2024-02-21T23:08:04.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.051+0000 7f3b32800640 1 -- 172.21.15.28:0/3375022603 learned_addr learned my addr 172.21.15.28:0/3375022603 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:04.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.051+0000 7f3b32800640 1 -- 172.21.15.28:0/3375022603 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3b340c1080 msgr2=0x7f3b34159490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:04.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.051+0000 7f3b31e00640 1 --2- 172.21.15.28:0/3375022603 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3b340c1080 0x7f3b34159490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:04.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.051+0000 7f3b31400640 1 --2- 172.21.15.28:0/3375022603 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b34153470 0x7f3b3414c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:04.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.051+0000 7f3b32800640 1 --2- 172.21.15.28:0/3375022603 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3b340c1080 0x7f3b34159490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:04.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.051+0000 7f3b32800640 1 -- 172.21.15.28:0/3375022603 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b34153470 msgr2=0x7f3b3414c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:04.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.051+0000 7f3b32800640 1 --2- 172.21.15.28:0/3375022603 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b34153470 0x7f3b3414c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:04.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.051+0000 7f3b32800640 1 -- 172.21.15.28:0/3375022603 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b341f2530 con 0x7f3b3415d170 2024-02-21T23:08:04.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.051+0000 7f3b31400640 1 --2- 172.21.15.28:0/3375022603 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b34153470 0x7f3b3414c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:08:04.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.052+0000 7f3b32800640 1 --2- 172.21.15.28:0/3375022603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b3415d170 0x7f3b3414ce60 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c007e90 tx=0x7f3b2c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:04.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.052+0000 7f3b2a000640 1 -- 172.21.15.28:0/3375022603 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b2c014420 con 0x7f3b3415d170 2024-02-21T23:08:04.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.052+0000 7f3b2a000640 1 -- 172.21.15.28:0/3375022603 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b2c019070 con 0x7f3b3415d170 2024-02-21T23:08:04.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.052+0000 7f3b2a000640 1 -- 172.21.15.28:0/3375022603 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b2c014800 con 0x7f3b3415d170 2024-02-21T23:08:04.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.052+0000 7f3b33200640 1 -- 172.21.15.28:0/3375022603 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b341f2810 con 0x7f3b3415d170 2024-02-21T23:08:04.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.052+0000 7f3b33200640 1 -- 172.21.15.28:0/3375022603 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b341f2dd0 con 0x7f3b3415d170 2024-02-21T23:08:04.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.053+0000 7f3b33200640 1 -- 172.21.15.28:0/3375022603 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b3404f290 con 0x7f3b3415d170 2024-02-21T23:08:04.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.054+0000 7f3b2a000640 1 -- 172.21.15.28:0/3375022603 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3b2c0149a0 con 0x7f3b3415d170 2024-02-21T23:08:04.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.055+0000 7f3b2a000640 1 --2- 172.21.15.28:0/3375022603 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3b1006b4d0 0x7f3b1006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:04.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.056+0000 7f3b2a000640 1 -- 172.21.15.28:0/3375022603 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f3b2c08a7b0 con 0x7f3b3415d170 2024-02-21T23:08:04.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.056+0000 7f3b31e00640 1 --2- 172.21.15.28:0/3375022603 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3b1006b4d0 0x7f3b1006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:04.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.057+0000 7f3b31e00640 1 --2- 172.21.15.28:0/3375022603 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3b1006b4d0 0x7f3b1006d990 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c006fd0 tx=0x7f3b1c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:04.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.060+0000 7f3b2a000640 1 -- 172.21.15.28:0/3375022603 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3b2c0597e0 con 0x7f3b3415d170 2024-02-21T23:08:04.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.259+0000 7f3b33200640 1 -- 172.21.15.28:0/3375022603 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.d", "name": "mds_join_fs"} v 0) v1 -- 0x7f3b3406f500 con 0x7f3b3415d170 2024-02-21T23:08:04.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.265+0000 7f3b2a000640 1 -- 172.21.15.28:0/3375022603 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.d", "name": "mds_join_fs"}]=0 v39) v1 ==== 96+0+0 (secure 0 0 0) 0x7f3b2c0590e0 con 0x7f3b3415d170 2024-02-21T23:08:04.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.267+0000 7f3b33200640 1 -- 172.21.15.28:0/3375022603 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3b1006b4d0 msgr2=0x7f3b1006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:04.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.267+0000 7f3b33200640 1 --2- 172.21.15.28:0/3375022603 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3b1006b4d0 0x7f3b1006d990 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c006fd0 tx=0x7f3b1c008040 comp rx=0 tx=0).stop 2024-02-21T23:08:04.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.267+0000 7f3b33200640 1 -- 172.21.15.28:0/3375022603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b3415d170 msgr2=0x7f3b3414ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:04.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.267+0000 7f3b33200640 1 --2- 172.21.15.28:0/3375022603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b3415d170 0x7f3b3414ce60 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c007e90 tx=0x7f3b2c00e4b0 comp rx=0 tx=0).stop 2024-02-21T23:08:04.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.268+0000 7f3b33200640 1 -- 172.21.15.28:0/3375022603 shutdown_connections 2024-02-21T23:08:04.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.268+0000 7f3b33200640 1 --2- 172.21.15.28:0/3375022603 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3b1006b4d0 0x7f3b1006d990 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:04.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.268+0000 7f3b33200640 1 --2- 172.21.15.28:0/3375022603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b3415d170 0x7f3b3414ce60 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:04.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.268+0000 7f3b33200640 1 --2- 172.21.15.28:0/3375022603 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b34153470 0x7f3b3414c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:04.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.268+0000 7f3b33200640 1 --2- 172.21.15.28:0/3375022603 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3b340c1080 0x7f3b34159490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:04.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.268+0000 7f3b33200640 1 -- 172.21.15.28:0/3375022603 >> 172.21.15.28:0/3375022603 conn(0x7f3b340a8530 msgr2=0x7f3b340c5a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:04.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.268+0000 7f3b33200640 1 -- 172.21.15.28:0/3375022603 shutdown_connections 2024-02-21T23:08:04.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.268+0000 7f3b33200640 1 -- 172.21.15.28:0/3375022603 wait complete. 2024-02-21T23:08:04.277 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-02-21T23:08:04.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.515+0000 7fe500c00640 1 Processor -- start 2024-02-21T23:08:04.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.515+0000 7fe500c00640 1 -- start start 2024-02-21T23:08:04.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.516+0000 7fe500c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fe4fc1500f0 0x7fe4fc06ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:04.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.516+0000 7fe500c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4fc06db10 con 0x7fe4fc06b3b0 2024-02-21T23:08:04.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.516+0000 7fe500c00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4fc1539a0 con 0x7fe4fc0c2c60 2024-02-21T23:08:04.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.516+0000 7fe500c00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4fc0c1080 con 0x7fe4fc1500f0 2024-02-21T23:08:04.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.517+0000 7fe4fbe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe4fc06b3b0 0x7fe4fc15d4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60814/0 (socket says 172.21.15.28:60814) 2024-02-21T23:08:04.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.517+0000 7fe4fbe00640 1 -- 172.21.15.28:0/1651584729 learned_addr learned my addr 172.21.15.28:0/1651584729 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:08:04.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.517+0000 7fe4fb400640 1 --2- 172.21.15.28:0/1651584729 >> v2:172.21.15.81:3300/0 conn(0x7fe4fc1500f0 0x7fe4fc06ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:04.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.517+0000 7fe4faa00640 1 -- 172.21.15.28:0/1651584729 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1999303192 0 0) 0x7fe4fc06db10 con 0x7fe4fc06b3b0 2024-02-21T23:08:04.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.517+0000 7fe4faa00640 1 -- 172.21.15.28:0/1651584729 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4e0003520 con 0x7fe4fc06b3b0 2024-02-21T23:08:04.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.518+0000 7fe4faa00640 1 -- 172.21.15.28:0/1651584729 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 200955632 0 0) 0x7fe4e0003520 con 0x7fe4fc06b3b0 2024-02-21T23:08:04.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.518+0000 7fe4faa00640 1 -- 172.21.15.28:0/1651584729 >> v2:172.21.15.81:3300/0 conn(0x7fe4fc1500f0 msgr2=0x7fe4fc06ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:04.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.518+0000 7fe4faa00640 1 --2- 172.21.15.28:0/1651584729 >> v2:172.21.15.81:3300/0 conn(0x7fe4fc1500f0 0x7fe4fc06ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:04.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.518+0000 7fe4faa00640 1 -- 172.21.15.28:0/1651584729 >> v1:172.21.15.81:6789/0 conn(0x7fe4fc0c2c60 legacy=0x7fe4fc06dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:04.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.518+0000 7fe4faa00640 1 -- 172.21.15.28:0/1651584729 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4fc0c12d0 con 0x7fe4fc06b3b0 2024-02-21T23:08:04.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.518+0000 7fe4faa00640 1 -- 172.21.15.28:0/1651584729 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fe4f40019f0 con 0x7fe4fc06b3b0 2024-02-21T23:08:04.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.518+0000 7fe4faa00640 1 -- 172.21.15.28:0/1651584729 >> v1:172.21.15.28:6789/0 conn(0x7fe4fc06b3b0 legacy=0x7fe4fc15d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:04.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.518+0000 7fe4faa00640 1 --2- 172.21.15.28:0/1651584729 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe4e0003a00 0x7fe4e0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:04.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.518+0000 7fe4faa00640 1 --2- 172.21.15.28:0/1651584729 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe4e0004800 0x7fe4e0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:04.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.518+0000 7fe4faa00640 1 --2- 172.21.15.28:0/1651584729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4e00051c0 0x7fe4e000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:04.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.518+0000 7fe4faa00640 1 -- 172.21.15.28:0/1651584729 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4fc0c1080 con 0x7fe4e00051c0 2024-02-21T23:08:04.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.518+0000 7fe4faa00640 1 -- 172.21.15.28:0/1651584729 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e000c0c0 con 0x7fe4e0004800 2024-02-21T23:08:04.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.518+0000 7fe4faa00640 1 -- 172.21.15.28:0/1651584729 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e000c2c0 con 0x7fe4e0003a00 2024-02-21T23:08:04.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.518+0000 7fe4fbe00640 1 --2- 172.21.15.28:0/1651584729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4e00051c0 0x7fe4e000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:04.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.518+0000 7fe4fb400640 1 --2- 172.21.15.28:0/1651584729 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe4e0003a00 0x7fe4e0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:04.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.518+0000 7fe4f3e00640 1 --2- 172.21.15.28:0/1651584729 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe4e0004800 0x7fe4e0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:04.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.519+0000 7fe4fbe00640 1 -- 172.21.15.28:0/1651584729 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe4e0003a00 msgr2=0x7fe4e0003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:04.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.519+0000 7fe4fbe00640 1 --2- 172.21.15.28:0/1651584729 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe4e0003a00 0x7fe4e0003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:04.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.519+0000 7fe4fbe00640 1 -- 172.21.15.28:0/1651584729 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe4e0004800 msgr2=0x7fe4e0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:04.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.519+0000 7fe4fbe00640 1 --2- 172.21.15.28:0/1651584729 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe4e0004800 0x7fe4e0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:04.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.519+0000 7fe4fbe00640 1 -- 172.21.15.28:0/1651584729 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4e0003520 con 0x7fe4e00051c0 2024-02-21T23:08:04.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.520+0000 7fe4fbe00640 1 -- 172.21.15.28:0/1651584729 reap_dead start 2024-02-21T23:08:04.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.520+0000 7fe4fbe00640 1 --2- 172.21.15.28:0/1651584729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4e00051c0 0x7fe4e000b9c0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7fe4f400a790 tx=0x7fe4e0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1416c67b7650f6d server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:04.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.521+0000 7fe4faa00640 1 -- 172.21.15.28:0/1651584729 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4f400a3b0 con 0x7fe4e00051c0 2024-02-21T23:08:04.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.521+0000 7fe4faa00640 1 -- 172.21.15.28:0/1651584729 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4f4034650 con 0x7fe4e00051c0 2024-02-21T23:08:04.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.521+0000 7fe4faa00640 1 -- 172.21.15.28:0/1651584729 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4f4034a40 con 0x7fe4e00051c0 2024-02-21T23:08:04.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.521+0000 7fe500c00640 1 -- 172.21.15.28:0/1651584729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4e00051c0 msgr2=0x7fe4e000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:04.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.521+0000 7fe500c00640 1 --2- 172.21.15.28:0/1651584729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4e00051c0 0x7fe4e000b9c0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7fe4f400a790 tx=0x7fe4e0005830 comp rx=0 tx=0).stop 2024-02-21T23:08:04.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.521+0000 7fe500c00640 1 -- 172.21.15.28:0/1651584729 shutdown_connections 2024-02-21T23:08:04.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.521+0000 7fe500c00640 1 --2- 172.21.15.28:0/1651584729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4e00051c0 0x7fe4e000b9c0 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:04.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.521+0000 7fe500c00640 1 -- 172.21.15.28:0/1651584729 >> 172.21.15.28:0/1651584729 conn(0x7fe4fc0a8530 msgr2=0x7fe4fc0a8910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:04.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.522+0000 7fe500c00640 1 -- 172.21.15.28:0/1651584729 shutdown_connections 2024-02-21T23:08:04.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.522+0000 7fe500c00640 1 -- 172.21.15.28:0/1651584729 wait complete. 2024-02-21T23:08:04.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.523+0000 7fe500c00640 1 Processor -- start 2024-02-21T23:08:04.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.523+0000 7fe500c00640 1 -- start start 2024-02-21T23:08:04.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.524+0000 7fe500c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4e00051c0 0x7fe4fc0a6450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:04.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.524+0000 7fe500c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe4fc14f280 0x7fe4fc071c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:04.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.524+0000 7fe500c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe4fc1eb5e0 0x7fe4fc1ee240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:04.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.524+0000 7fe500c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4fc1535e0 con 0x7fe4e00051c0 2024-02-21T23:08:04.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.524+0000 7fe500c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4fc1504d0 con 0x7fe4fc14f280 2024-02-21T23:08:04.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.524+0000 7fe500c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4fc153760 con 0x7fe4fc1eb5e0 2024-02-21T23:08:04.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.525+0000 7fe4fb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4e00051c0 0x7fe4fc0a6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:04.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.525+0000 7fe4fb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4e00051c0 0x7fe4fc0a6450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48826/0 (socket says 172.21.15.28:48826) 2024-02-21T23:08:04.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.525+0000 7fe4fb400640 1 -- 172.21.15.28:0/2097625731 learned_addr learned my addr 172.21.15.28:0/2097625731 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:04.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.525+0000 7fe4fb400640 1 -- 172.21.15.28:0/2097625731 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe4fc1eb5e0 msgr2=0x7fe4fc1ee240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:04.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.525+0000 7fe4fbe00640 1 --2- 172.21.15.28:0/2097625731 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe4fc1eb5e0 0x7fe4fc1ee240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:04.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.525+0000 7fe4fb400640 1 --2- 172.21.15.28:0/2097625731 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe4fc1eb5e0 0x7fe4fc1ee240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:04.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.525+0000 7fe4fb400640 1 -- 172.21.15.28:0/2097625731 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe4fc14f280 msgr2=0x7fe4fc071c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:04.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.525+0000 7fe4fb400640 1 --2- 172.21.15.28:0/2097625731 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe4fc14f280 0x7fe4fc071c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:04.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.525+0000 7fe4fb400640 1 -- 172.21.15.28:0/2097625731 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4fc072b00 con 0x7fe4e00051c0 2024-02-21T23:08:04.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.526+0000 7fe4fb400640 1 --2- 172.21.15.28:0/2097625731 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4e00051c0 0x7fe4fc0a6450 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7fe4ec0118e0 tx=0x7fe4ec011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:04.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.526+0000 7fe4f3400640 1 -- 172.21.15.28:0/2097625731 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4ec00faf0 con 0x7fe4e00051c0 2024-02-21T23:08:04.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.526+0000 7fe4f3400640 1 -- 172.21.15.28:0/2097625731 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4ec00fc90 con 0x7fe4e00051c0 2024-02-21T23:08:04.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.526+0000 7fe500c00640 1 -- 172.21.15.28:0/2097625731 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4fc0a7c60 con 0x7fe4e00051c0 2024-02-21T23:08:04.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.526+0000 7fe4f3400640 1 -- 172.21.15.28:0/2097625731 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4ec014440 con 0x7fe4e00051c0 2024-02-21T23:08:04.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.526+0000 7fe500c00640 1 -- 172.21.15.28:0/2097625731 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4fc1eade0 con 0x7fe4e00051c0 2024-02-21T23:08:04.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.527+0000 7fe500c00640 1 -- 172.21.15.28:0/2097625731 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4fc067f30 con 0x7fe4e00051c0 2024-02-21T23:08:04.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.528+0000 7fe4f3400640 1 -- 172.21.15.28:0/2097625731 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe4ec012040 con 0x7fe4e00051c0 2024-02-21T23:08:04.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.529+0000 7fe4f3400640 1 --2- 172.21.15.28:0/2097625731 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe4d806b4d0 0x7fe4d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:04.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.529+0000 7fe4f3400640 1 -- 172.21.15.28:0/2097625731 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5421+0+0 (secure 0 0 0) 0x7fe4ec08a740 con 0x7fe4e00051c0 2024-02-21T23:08:04.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.530+0000 7fe4f3e00640 1 --2- 172.21.15.28:0/2097625731 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe4d806b4d0 0x7fe4d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:04.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.531+0000 7fe4f3e00640 1 --2- 172.21.15.28:0/2097625731 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe4d806b4d0 0x7fe4d806d990 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fe4e4000c00 tx=0x7fe4e4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:04.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.534+0000 7fe4f3400640 1 -- 172.21.15.28:0/2097625731 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe4fc067f30 con 0x7fe4e00051c0 2024-02-21T23:08:04.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.738+0000 7fe500c00640 1 -- 172.21.15.28:0/2097625731 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fe4fc06c440 con 0x7fe4e00051c0 2024-02-21T23:08:04.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.743+0000 7fe4f3400640 1 -- 172.21.15.28:0/2097625731 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v40) v1 ==== 104+0+0 (secure 0 0 0) 0x7fe4fc06c440 con 0x7fe4e00051c0 2024-02-21T23:08:04.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.746+0000 7fe500c00640 1 -- 172.21.15.28:0/2097625731 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe4d806b4d0 msgr2=0x7fe4d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:04.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.746+0000 7fe500c00640 1 --2- 172.21.15.28:0/2097625731 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe4d806b4d0 0x7fe4d806d990 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fe4e4000c00 tx=0x7fe4e4009040 comp rx=0 tx=0).stop 2024-02-21T23:08:04.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.746+0000 7fe500c00640 1 -- 172.21.15.28:0/2097625731 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4e00051c0 msgr2=0x7fe4fc0a6450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:04.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.746+0000 7fe500c00640 1 --2- 172.21.15.28:0/2097625731 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4e00051c0 0x7fe4fc0a6450 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7fe4ec0118e0 tx=0x7fe4ec011da0 comp rx=0 tx=0).stop 2024-02-21T23:08:04.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.746+0000 7fe500c00640 1 -- 172.21.15.28:0/2097625731 shutdown_connections 2024-02-21T23:08:04.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.747+0000 7fe500c00640 1 --2- 172.21.15.28:0/2097625731 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe4d806b4d0 0x7fe4d806d990 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:04.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.747+0000 7fe500c00640 1 --2- 172.21.15.28:0/2097625731 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe4fc1eb5e0 0x7fe4fc1ee240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:04.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.747+0000 7fe500c00640 1 --2- 172.21.15.28:0/2097625731 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe4fc14f280 0x7fe4fc071c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:04.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.747+0000 7fe500c00640 1 --2- 172.21.15.28:0/2097625731 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4e00051c0 0x7fe4fc0a6450 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:04.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.747+0000 7fe500c00640 1 -- 172.21.15.28:0/2097625731 >> 172.21.15.28:0/2097625731 conn(0x7fe4fc0a8530 msgr2=0x7fe4fc0c3120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:04.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.747+0000 7fe500c00640 1 -- 172.21.15.28:0/2097625731 shutdown_connections 2024-02-21T23:08:04.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.747+0000 7fe500c00640 1 -- 172.21.15.28:0/2097625731 wait complete. 2024-02-21T23:08:04.753 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_standby_replay' 2024-02-21T23:08:04.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.984+0000 7f958f200640 1 Processor -- start 2024-02-21T23:08:04.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.984+0000 7f958f200640 1 -- start start 2024-02-21T23:08:04.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.986+0000 7f958f200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f959006dbf0 0x7f9590158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:04.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.986+0000 7f958f200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9590155960 0x7f9590159000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:04.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.986+0000 7f958f200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95900c74c0 con 0x7f95900aff90 2024-02-21T23:08:04.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.986+0000 7f958f200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95900c7270 con 0x7f959006dbf0 2024-02-21T23:08:04.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.986+0000 7f958f200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9590158630 con 0x7f9590155960 2024-02-21T23:08:04.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.986+0000 7f958de00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f95900aff90 0x7f95900c51e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:46644/0 (socket says 172.21.15.28:46644) 2024-02-21T23:08:04.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.986+0000 7f958de00640 1 -- 172.21.15.28:0/3777512831 learned_addr learned my addr 172.21.15.28:0/3777512831 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:08:04.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.987+0000 7f9587e00640 1 --2- 172.21.15.28:0/3777512831 >> v2:172.21.15.81:3300/0 conn(0x7f959006dbf0 0x7f9590158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:04.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.987+0000 7f958e800640 1 --2- 172.21.15.28:0/3777512831 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9590155960 0x7f9590159000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:04.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.987+0000 7f958d400640 1 -- 172.21.15.28:0/3777512831 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 478356208 0 0) 0x7f95900c74c0 con 0x7f95900aff90 2024-02-21T23:08:04.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.987+0000 7f958d400640 1 -- 172.21.15.28:0/3777512831 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9574003520 con 0x7f95900aff90 2024-02-21T23:08:04.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.987+0000 7f958d400640 1 -- 172.21.15.28:0/3777512831 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4292327278 0 0) 0x7f9574003520 con 0x7f95900aff90 2024-02-21T23:08:04.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.987+0000 7f958d400640 1 -- 172.21.15.28:0/3777512831 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9590155960 msgr2=0x7f9590159000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:04.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.987+0000 7f958d400640 1 --2- 172.21.15.28:0/3777512831 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9590155960 0x7f9590159000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:04.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.987+0000 7f958d400640 1 -- 172.21.15.28:0/3777512831 >> v2:172.21.15.81:3300/0 conn(0x7f959006dbf0 msgr2=0x7f9590158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:04.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.987+0000 7f958d400640 1 --2- 172.21.15.28:0/3777512831 >> v2:172.21.15.81:3300/0 conn(0x7f959006dbf0 0x7f9590158ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:04.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.987+0000 7f958d400640 1 -- 172.21.15.28:0/3777512831 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95900c77a0 con 0x7f95900aff90 2024-02-21T23:08:04.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.987+0000 7f958d400640 1 -- 172.21.15.28:0/3777512831 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f9580001670 con 0x7f95900aff90 2024-02-21T23:08:04.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.987+0000 7f958d400640 1 -- 172.21.15.28:0/3777512831 >> v1:172.21.15.28:6789/0 conn(0x7f95900aff90 legacy=0x7f95900c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:04.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.987+0000 7f958d400640 1 --2- 172.21.15.28:0/3777512831 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95740039b0 0x7f9574003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:04.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.987+0000 7f958d400640 1 --2- 172.21.15.28:0/3777512831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9574004800 0x7f9574004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:04.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.987+0000 7f958d400640 1 --2- 172.21.15.28:0/3777512831 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9590155960 0x7f95740051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:04.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.987+0000 7f958d400640 1 -- 172.21.15.28:0/3777512831 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95900c7270 con 0x7f9574004800 2024-02-21T23:08:04.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.987+0000 7f958d400640 1 -- 172.21.15.28:0/3777512831 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9590158630 con 0x7f95740039b0 2024-02-21T23:08:04.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.987+0000 7f958d400640 1 -- 172.21.15.28:0/3777512831 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f957400b9c0 con 0x7f9590155960 2024-02-21T23:08:04.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.988+0000 7f9587e00640 1 --2- 172.21.15.28:0/3777512831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9574004800 0x7f9574004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:04.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.988+0000 7f958de00640 1 --2- 172.21.15.28:0/3777512831 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95740039b0 0x7f9574003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:04.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.988+0000 7f958e800640 1 --2- 172.21.15.28:0/3777512831 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9590155960 0x7f95740051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:04.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.988+0000 7f9587e00640 1 -- 172.21.15.28:0/3777512831 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9590155960 msgr2=0x7f95740051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:04.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.988+0000 7f9587e00640 1 --2- 172.21.15.28:0/3777512831 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9590155960 0x7f95740051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:04.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.988+0000 7f9587e00640 1 -- 172.21.15.28:0/3777512831 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95740039b0 msgr2=0x7f9574003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:04.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.988+0000 7f9587e00640 1 --2- 172.21.15.28:0/3777512831 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95740039b0 0x7f9574003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:04.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.988+0000 7f9587e00640 1 -- 172.21.15.28:0/3777512831 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9574003520 con 0x7f9574004800 2024-02-21T23:08:04.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.989+0000 7f9587e00640 1 --2- 172.21.15.28:0/3777512831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9574004800 0x7f9574004c80 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f9578009a90 tx=0x7f9574005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=220dd06ee9a771ab server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:04.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.990+0000 7f958d400640 1 -- 172.21.15.28:0/3777512831 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9578039070 con 0x7f9574004800 2024-02-21T23:08:04.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.990+0000 7f958d400640 1 -- 172.21.15.28:0/3777512831 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f957802c6b0 con 0x7f9574004800 2024-02-21T23:08:04.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.990+0000 7f958d400640 1 -- 172.21.15.28:0/3777512831 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9578009690 con 0x7f9574004800 2024-02-21T23:08:04.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.990+0000 7f958f200640 1 -- 172.21.15.28:0/3777512831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9574004800 msgr2=0x7f9574004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:04.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.990+0000 7f958f200640 1 --2- 172.21.15.28:0/3777512831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9574004800 0x7f9574004c80 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f9578009a90 tx=0x7f9574005830 comp rx=0 tx=0).stop 2024-02-21T23:08:04.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.990+0000 7f958e800640 1 -- 172.21.15.28:0/3777512831 reap_dead start 2024-02-21T23:08:04.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.990+0000 7f958f200640 1 -- 172.21.15.28:0/3777512831 shutdown_connections 2024-02-21T23:08:04.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.990+0000 7f958f200640 1 -- 172.21.15.28:0/3777512831 >> 172.21.15.28:0/3777512831 conn(0x7f95900ac370 msgr2=0x7f95900ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:04.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.990+0000 7f958f200640 1 -- 172.21.15.28:0/3777512831 shutdown_connections 2024-02-21T23:08:04.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.990+0000 7f958f200640 1 -- 172.21.15.28:0/3777512831 wait complete. 2024-02-21T23:08:04.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.992+0000 7f958f200640 1 Processor -- start 2024-02-21T23:08:04.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.992+0000 7f958f200640 1 -- start start 2024-02-21T23:08:04.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.993+0000 7f958f200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9574004800 0x7f9590144980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:04.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.993+0000 7f958f200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9590144f30 0x7f9590150f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:04.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.993+0000 7f958f200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95900c3350 0x7f95900c3800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:04.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.993+0000 7f958f200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95900c58f0 con 0x7f9590144f30 2024-02-21T23:08:04.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.993+0000 7f958f200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95900be0c0 con 0x7f95900c3350 2024-02-21T23:08:04.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.993+0000 7f958f200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95900c5a70 con 0x7f9574004800 2024-02-21T23:08:04.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.993+0000 7f9587e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9590144f30 0x7f9590150f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:04.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.993+0000 7f9587e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9590144f30 0x7f9590150f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54394/0 (socket says 172.21.15.28:54394) 2024-02-21T23:08:04.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.993+0000 7f9587e00640 1 -- 172.21.15.28:0/1591727373 learned_addr learned my addr 172.21.15.28:0/1591727373 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:04.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.994+0000 7f9587e00640 1 -- 172.21.15.28:0/1591727373 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9574004800 msgr2=0x7f9590144980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:04.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.994+0000 7f958de00640 1 --2- 172.21.15.28:0/1591727373 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9574004800 0x7f9590144980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:04.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.994+0000 7f9587e00640 1 --2- 172.21.15.28:0/1591727373 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9574004800 0x7f9590144980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:04.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.994+0000 7f9587e00640 1 -- 172.21.15.28:0/1591727373 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95900c3350 msgr2=0x7f95900c3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:04.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.994+0000 7f9587e00640 1 --2- 172.21.15.28:0/1591727373 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95900c3350 0x7f95900c3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:04.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.994+0000 7f9587e00640 1 -- 172.21.15.28:0/1591727373 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95901ee540 con 0x7f9590144f30 2024-02-21T23:08:04.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.994+0000 7f9587e00640 1 --2- 172.21.15.28:0/1591727373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9590144f30 0x7f9590150f80 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f9578037040 tx=0x7f9578004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:04.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.994+0000 7f9586000640 1 -- 172.21.15.28:0/1591727373 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9578039050 con 0x7f9590144f30 2024-02-21T23:08:04.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.995+0000 7f9586000640 1 -- 172.21.15.28:0/1591727373 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f957803a040 con 0x7f9590144f30 2024-02-21T23:08:04.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.995+0000 7f9586000640 1 -- 172.21.15.28:0/1591727373 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95780044c0 con 0x7f9590144f30 2024-02-21T23:08:04.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.995+0000 7f958f200640 1 -- 172.21.15.28:0/1591727373 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95901ee7e0 con 0x7f9590144f30 2024-02-21T23:08:04.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.995+0000 7f958f200640 1 -- 172.21.15.28:0/1591727373 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95901543f0 con 0x7f9590144f30 2024-02-21T23:08:04.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.996+0000 7f958f200640 1 -- 172.21.15.28:0/1591727373 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f959004f290 con 0x7f9590144f30 2024-02-21T23:08:04.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.997+0000 7f9586000640 1 -- 172.21.15.28:0/1591727373 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9578047020 con 0x7f9590144f30 2024-02-21T23:08:04.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.998+0000 7f9586000640 1 --2- 172.21.15.28:0/1591727373 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f956c06b4d0 0x7f956c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:04.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.998+0000 7f9586000640 1 -- 172.21.15.28:0/1591727373 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f95780ab790 con 0x7f9590144f30 2024-02-21T23:08:04.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:04.998+0000 7f958de00640 1 --2- 172.21.15.28:0/1591727373 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f956c06b4d0 0x7f956c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:04.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.000+0000 7f958de00640 1 --2- 172.21.15.28:0/1591727373 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f956c06b4d0 0x7f956c06d990 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f9580001eb0 tx=0x7f9580000ed0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:04.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.002+0000 7f9586000640 1 -- 172.21.15.28:0/1591727373 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f957804e020 con 0x7f9590144f30 2024-02-21T23:08:05.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.275+0000 7f958f200640 1 -- 172.21.15.28:0/1591727373 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_standby_replay"]} v 0) v1 -- 0x7f95901550d0 con 0x7f9590144f30 2024-02-21T23:08:05.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.319+0000 7f9586000640 1 -- 172.21.15.28:0/1591727373 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_standby_replay"]}]=0 v326) v1 ==== 153+0+0 (secure 0 0 0) 0x7f957807a720 con 0x7f9590144f30 2024-02-21T23:08:05.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.322+0000 7f958f200640 1 -- 172.21.15.28:0/1591727373 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f956c06b4d0 msgr2=0x7f956c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:05.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.322+0000 7f958f200640 1 --2- 172.21.15.28:0/1591727373 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f956c06b4d0 0x7f956c06d990 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f9580001eb0 tx=0x7f9580000ed0 comp rx=0 tx=0).stop 2024-02-21T23:08:05.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.322+0000 7f958f200640 1 -- 172.21.15.28:0/1591727373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9590144f30 msgr2=0x7f9590150f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:05.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.322+0000 7f958f200640 1 --2- 172.21.15.28:0/1591727373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9590144f30 0x7f9590150f80 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f9578037040 tx=0x7f9578004730 comp rx=0 tx=0).stop 2024-02-21T23:08:05.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.323+0000 7f958f200640 1 -- 172.21.15.28:0/1591727373 shutdown_connections 2024-02-21T23:08:05.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.323+0000 7f958f200640 1 --2- 172.21.15.28:0/1591727373 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f956c06b4d0 0x7f956c06d990 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:05.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.323+0000 7f958f200640 1 --2- 172.21.15.28:0/1591727373 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95900c3350 0x7f95900c3800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:05.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.323+0000 7f958f200640 1 --2- 172.21.15.28:0/1591727373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9590144f30 0x7f9590150f80 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:05.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.323+0000 7f958f200640 1 --2- 172.21.15.28:0/1591727373 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9574004800 0x7f9590144980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:05.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.323+0000 7f958f200640 1 -- 172.21.15.28:0/1591727373 >> 172.21.15.28:0/1591727373 conn(0x7f95900ac370 msgr2=0x7f95901568f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:05.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.323+0000 7f958f200640 1 -- 172.21.15.28:0/1591727373 shutdown_connections 2024-02-21T23:08:05.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.323+0000 7f958f200640 1 -- 172.21.15.28:0/1591727373 wait complete. 2024-02-21T23:08:05.331 INFO:tasks.cephfs_test_runner:That a standby-replay daemon with weak affinity is replaced by a stronger one. ... ok 2024-02-21T23:08:05.332 INFO:tasks.cephfs_test_runner:Starting test: test_join_fs_steady (tasks.cephfs.test_failover.TestClusterAffinity) That a sole MDS with mds_join_fs set will come back as active eventually even after failover. 2024-02-21T23:08:05.332 INFO:tasks.cephfs_test_runner:test_join_fs_steady (tasks.cephfs.test_failover.TestClusterAffinity) 2024-02-21T23:08:05.332 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:08:05.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.572+0000 7f65e4e00640 1 Processor -- start 2024-02-21T23:08:05.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.573+0000 7f65e4e00640 1 -- start start 2024-02-21T23:08:05.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.574+0000 7f65e4e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f65e006e8f0 0x7f65e006ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:05.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.574+0000 7f65e4e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65e00a7da0 con 0x7f65e014b750 2024-02-21T23:08:05.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.574+0000 7f65e4e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65e00c2790 con 0x7f65e0151e30 2024-02-21T23:08:05.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.574+0000 7f65e4e00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65e01522c0 con 0x7f65e006e8f0 2024-02-21T23:08:05.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.574+0000 7f65df400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f65e014b750 0x7f65e0155300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:46654/0 (socket says 172.21.15.28:46654) 2024-02-21T23:08:05.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.574+0000 7f65df400640 1 -- 172.21.15.28:0/1273553597 learned_addr learned my addr 172.21.15.28:0/1273553597 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:08:05.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.575+0000 7f65de000640 1 -- 172.21.15.28:0/1273553597 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2147878776 0 0) 0x7f65e00a7da0 con 0x7f65e014b750 2024-02-21T23:08:05.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.575+0000 7f65dfe00640 1 --2- 172.21.15.28:0/1273553597 >> v2:172.21.15.81:3300/0 conn(0x7f65e006e8f0 0x7f65e006ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:05.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.575+0000 7f65de000640 1 -- 172.21.15.28:0/1273553597 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65c4003520 con 0x7f65e014b750 2024-02-21T23:08:05.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.575+0000 7f65de000640 1 -- 172.21.15.28:0/1273553597 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3279851799 0 0) 0x7f65c4003520 con 0x7f65e014b750 2024-02-21T23:08:05.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.575+0000 7f65de000640 1 -- 172.21.15.28:0/1273553597 >> v2:172.21.15.81:3300/0 conn(0x7f65e006e8f0 msgr2=0x7f65e006ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:05.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.575+0000 7f65de000640 1 --2- 172.21.15.28:0/1273553597 >> v2:172.21.15.81:3300/0 conn(0x7f65e006e8f0 0x7f65e006ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:05.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.575+0000 7f65de000640 1 -- 172.21.15.28:0/1273553597 >> v1:172.21.15.81:6789/0 conn(0x7f65e0151e30 legacy=0x7f65e00a7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:05.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.575+0000 7f65de000640 1 -- 172.21.15.28:0/1273553597 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65e00c2a70 con 0x7f65e014b750 2024-02-21T23:08:05.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.575+0000 7f65de000640 1 -- 172.21.15.28:0/1273553597 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f65d0001670 con 0x7f65e014b750 2024-02-21T23:08:05.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.575+0000 7f65de000640 1 -- 172.21.15.28:0/1273553597 >> v1:172.21.15.28:6789/0 conn(0x7f65e014b750 legacy=0x7f65e0155300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:05.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.575+0000 7f65de000640 1 --2- 172.21.15.28:0/1273553597 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f65c4003a00 0x7f65c4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:05.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.575+0000 7f65de000640 1 --2- 172.21.15.28:0/1273553597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65c4004800 0x7f65c4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:05.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.575+0000 7f65de000640 1 --2- 172.21.15.28:0/1273553597 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65c40052e0 0x7f65c400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:05.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.575+0000 7f65de000640 1 -- 172.21.15.28:0/1273553597 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65c400c1d0 con 0x7f65c4004800 2024-02-21T23:08:05.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.575+0000 7f65de000640 1 -- 172.21.15.28:0/1273553597 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65c400c380 con 0x7f65c40052e0 2024-02-21T23:08:05.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.575+0000 7f65de000640 1 -- 172.21.15.28:0/1273553597 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65c400c580 con 0x7f65c4003a00 2024-02-21T23:08:05.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.575+0000 7f65dfe00640 1 --2- 172.21.15.28:0/1273553597 >> v2:172.21.15.81:3300/0 conn(0x7f65e006e8f0 0x7f65e006ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:08:05.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.576+0000 7f65dea00640 1 --2- 172.21.15.28:0/1273553597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65c4004800 0x7f65c4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:05.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.576+0000 7f65df400640 1 --2- 172.21.15.28:0/1273553597 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f65c4003a00 0x7f65c4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:05.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.576+0000 7f65dfe00640 1 --2- 172.21.15.28:0/1273553597 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65c40052e0 0x7f65c400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:05.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.576+0000 7f65dea00640 1 -- 172.21.15.28:0/1273553597 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f65c4003a00 msgr2=0x7f65c4003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:05.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.576+0000 7f65dea00640 1 --2- 172.21.15.28:0/1273553597 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f65c4003a00 0x7f65c4003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:05.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.576+0000 7f65dea00640 1 -- 172.21.15.28:0/1273553597 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65c40052e0 msgr2=0x7f65c400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:05.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.576+0000 7f65dea00640 1 --2- 172.21.15.28:0/1273553597 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65c40052e0 0x7f65c400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:05.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.576+0000 7f65dea00640 1 -- 172.21.15.28:0/1273553597 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65c4003520 con 0x7f65c4004800 2024-02-21T23:08:05.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.576+0000 7f65dfe00640 1 --2- 172.21.15.28:0/1273553597 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65c40052e0 0x7f65c400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:08:05.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.576+0000 7f65dfe00640 1 -- 172.21.15.28:0/1273553597 reap_dead start 2024-02-21T23:08:05.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.578+0000 7f65dea00640 1 --2- 172.21.15.28:0/1273553597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65c4004800 0x7f65c4004c80 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f65cc001750 tx=0x7f65c4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ccd0d44011169ea server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:05.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.578+0000 7f65de000640 1 -- 172.21.15.28:0/1273553597 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65cc001370 con 0x7f65c4004800 2024-02-21T23:08:05.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.578+0000 7f65de000640 1 -- 172.21.15.28:0/1273553597 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65cc0052d0 con 0x7f65c4004800 2024-02-21T23:08:05.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.578+0000 7f65de000640 1 -- 172.21.15.28:0/1273553597 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65cc003830 con 0x7f65c4004800 2024-02-21T23:08:05.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.579+0000 7f65e4e00640 1 -- 172.21.15.28:0/1273553597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65c4004800 msgr2=0x7f65c4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:05.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.579+0000 7f65e4e00640 1 --2- 172.21.15.28:0/1273553597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65c4004800 0x7f65c4004c80 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f65cc001750 tx=0x7f65c4005830 comp rx=0 tx=0).stop 2024-02-21T23:08:05.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.579+0000 7f65e4e00640 1 -- 172.21.15.28:0/1273553597 shutdown_connections 2024-02-21T23:08:05.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.579+0000 7f65e4e00640 1 --2- 172.21.15.28:0/1273553597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65c4004800 0x7f65c4004c80 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:05.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.579+0000 7f65e4e00640 1 -- 172.21.15.28:0/1273553597 >> 172.21.15.28:0/1273553597 conn(0x7f65e006b630 msgr2=0x7f65e00c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:05.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.579+0000 7f65e4e00640 1 -- 172.21.15.28:0/1273553597 shutdown_connections 2024-02-21T23:08:05.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.579+0000 7f65e4e00640 1 -- 172.21.15.28:0/1273553597 wait complete. 2024-02-21T23:08:05.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.581+0000 7f65e4e00640 1 Processor -- start 2024-02-21T23:08:05.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.581+0000 7f65e4e00640 1 -- start start 2024-02-21T23:08:05.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.582+0000 7f65e4e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65c4004800 0x7f65e0144cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:05.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.583+0000 7f65e4e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65e014ae00 0x7f65e00aa790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:05.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.583+0000 7f65e4e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f65e00aacd0 0x7f65e00a95b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:05.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.583+0000 7f65e4e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e01512e0 con 0x7f65c4004800 2024-02-21T23:08:05.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.583+0000 7f65e4e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e01519a0 con 0x7f65e014ae00 2024-02-21T23:08:05.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.583+0000 7f65e4e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65e00c09c0 con 0x7f65e00aacd0 2024-02-21T23:08:05.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.583+0000 7f65df400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65c4004800 0x7f65e0144cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:05.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.583+0000 7f65df400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65c4004800 0x7f65e0144cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54404/0 (socket says 172.21.15.28:54404) 2024-02-21T23:08:05.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.583+0000 7f65df400640 1 -- 172.21.15.28:0/238971464 learned_addr learned my addr 172.21.15.28:0/238971464 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:05.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.583+0000 7f65df400640 1 -- 172.21.15.28:0/238971464 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f65e00aacd0 msgr2=0x7f65e00a95b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:05.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.583+0000 7f65df400640 1 --2- 172.21.15.28:0/238971464 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f65e00aacd0 0x7f65e00a95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:05.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.583+0000 7f65df400640 1 -- 172.21.15.28:0/238971464 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65e014ae00 msgr2=0x7f65e00aa790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:05.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.583+0000 7f65df400640 1 --2- 172.21.15.28:0/238971464 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65e014ae00 0x7f65e00aa790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:05.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.583+0000 7f65df400640 1 -- 172.21.15.28:0/238971464 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65e0074990 con 0x7f65c4004800 2024-02-21T23:08:05.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.584+0000 7f65df400640 1 --2- 172.21.15.28:0/238971464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65c4004800 0x7f65e0144cf0 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f65d000e7b0 tx=0x7f65d000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:05.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.584+0000 7f65d7400640 1 -- 172.21.15.28:0/238971464 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d000c660 con 0x7f65c4004800 2024-02-21T23:08:05.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.584+0000 7f65d7400640 1 -- 172.21.15.28:0/238971464 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65d0018070 con 0x7f65c4004800 2024-02-21T23:08:05.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.584+0000 7f65d7400640 1 -- 172.21.15.28:0/238971464 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d000ca40 con 0x7f65c4004800 2024-02-21T23:08:05.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.584+0000 7f65e4e00640 1 -- 172.21.15.28:0/238971464 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65e0142570 con 0x7f65c4004800 2024-02-21T23:08:05.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.584+0000 7f65e4e00640 1 -- 172.21.15.28:0/238971464 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65e0142b30 con 0x7f65c4004800 2024-02-21T23:08:05.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.585+0000 7f65e4e00640 1 -- 172.21.15.28:0/238971464 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65e004f290 con 0x7f65c4004800 2024-02-21T23:08:05.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.586+0000 7f65d7400640 1 -- 172.21.15.28:0/238971464 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f65d000cbe0 con 0x7f65c4004800 2024-02-21T23:08:05.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.587+0000 7f65d7400640 1 --2- 172.21.15.28:0/238971464 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f65c006b500 0x7f65c006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:05.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.587+0000 7f65d7400640 1 -- 172.21.15.28:0/238971464 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f65d0089b40 con 0x7f65c4004800 2024-02-21T23:08:05.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.588+0000 7f65dea00640 1 --2- 172.21.15.28:0/238971464 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f65c006b500 0x7f65c006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:05.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.589+0000 7f65dea00640 1 --2- 172.21.15.28:0/238971464 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f65c006b500 0x7f65c006d9c0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f65cc033000 tx=0x7f65cc02f000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:05.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.592+0000 7f65d7400640 1 -- 172.21.15.28:0/238971464 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f65e0074990 con 0x7f65c4004800 2024-02-21T23:08:05.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.792+0000 7f65e4e00640 1 -- 172.21.15.28:0/238971464 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f65e0145230 con 0x7f65c4004800 2024-02-21T23:08:05.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.795+0000 7f65d7400640 1 -- 172.21.15.28:0/238971464 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v114) v1 ==== 74+0+13373 (secure 0 0 0) 0x7f65d0058ad0 con 0x7f65c4004800 2024-02-21T23:08:05.783 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:08:05.783 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":114,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:08:03.305208+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.81:6835/1754860807":"2024-02-22T23:08:00.240016+0000","172.21.15.81:6834/1754860807":"2024-02-22T23:08:00.240016+0000","172.21.15.28:6835/1259832430":"2024-02-22T23:07:41.139002+0000","172.21.15.28:6834/1259832430":"2024-02-22T23:07:41.139002+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:08:05.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.797+0000 7f65e4e00640 1 -- 172.21.15.28:0/238971464 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f65c006b500 msgr2=0x7f65c006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:05.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.797+0000 7f65e4e00640 1 --2- 172.21.15.28:0/238971464 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f65c006b500 0x7f65c006d9c0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f65cc033000 tx=0x7f65cc02f000 comp rx=0 tx=0).stop 2024-02-21T23:08:05.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.797+0000 7f65e4e00640 1 -- 172.21.15.28:0/238971464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65c4004800 msgr2=0x7f65e0144cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:05.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.797+0000 7f65e4e00640 1 --2- 172.21.15.28:0/238971464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65c4004800 0x7f65e0144cf0 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f65d000e7b0 tx=0x7f65d000ec70 comp rx=0 tx=0).stop 2024-02-21T23:08:05.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.797+0000 7f65e4e00640 1 -- 172.21.15.28:0/238971464 shutdown_connections 2024-02-21T23:08:05.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.797+0000 7f65e4e00640 1 --2- 172.21.15.28:0/238971464 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f65c006b500 0x7f65c006d9c0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:05.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.798+0000 7f65e4e00640 1 --2- 172.21.15.28:0/238971464 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f65e00aacd0 0x7f65e00a95b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:05.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.798+0000 7f65e4e00640 1 --2- 172.21.15.28:0/238971464 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65e014ae00 0x7f65e00aa790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:05.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.798+0000 7f65e4e00640 1 --2- 172.21.15.28:0/238971464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65c4004800 0x7f65e0144cf0 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:05.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.798+0000 7f65e4e00640 1 -- 172.21.15.28:0/238971464 >> 172.21.15.28:0/238971464 conn(0x7f65e006b630 msgr2=0x7f65e00c01c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:05.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.798+0000 7f65e4e00640 1 -- 172.21.15.28:0/238971464 shutdown_connections 2024-02-21T23:08:05.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:05.798+0000 7f65e4e00640 1 -- 172.21.15.28:0/238971464 wait complete. 2024-02-21T23:08:05.806 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-21T23:02:30.728329+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-21T23:08:05.806 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-21T23:08:06.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.046+0000 7effece00640 1 Processor -- start 2024-02-21T23:08:06.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.046+0000 7effece00640 1 -- start start 2024-02-21T23:08:06.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.047+0000 7effece00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7effe80c4580 0x7effe80c12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:06.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.048+0000 7effece00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effe806e220 con 0x7effe80c2c10 2024-02-21T23:08:06.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.048+0000 7effece00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effe806f780 con 0x7effe8150240 2024-02-21T23:08:06.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.048+0000 7effece00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effe8150620 con 0x7effe80c4580 2024-02-21T23:08:06.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.048+0000 7effe7e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7effe80c2c10 0x7effe806ae70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:46664/0 (socket says 172.21.15.28:46664) 2024-02-21T23:08:06.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.048+0000 7effe7e00640 1 -- 172.21.15.28:0/3722583146 learned_addr learned my addr 172.21.15.28:0/3722583146 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:08:06.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.048+0000 7effe6a00640 1 --2- 172.21.15.28:0/3722583146 >> v2:172.21.15.81:3300/0 conn(0x7effe80c4580 0x7effe80c12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:06.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.049+0000 7effe6000640 1 -- 172.21.15.28:0/3722583146 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2153259556 0 0) 0x7effe806e220 con 0x7effe80c2c10 2024-02-21T23:08:06.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.049+0000 7effe6000640 1 -- 172.21.15.28:0/3722583146 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effcc003520 con 0x7effe80c2c10 2024-02-21T23:08:06.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.049+0000 7effe6000640 1 -- 172.21.15.28:0/3722583146 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 587264432 0 0) 0x7effe806f780 con 0x7effe8150240 2024-02-21T23:08:06.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.049+0000 7effe6000640 1 -- 172.21.15.28:0/3722583146 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effe806e220 con 0x7effe8150240 2024-02-21T23:08:06.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.049+0000 7effe6000640 1 -- 172.21.15.28:0/3722583146 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1017192851 0 0) 0x7effcc003520 con 0x7effe80c2c10 2024-02-21T23:08:06.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.049+0000 7effe6000640 1 -- 172.21.15.28:0/3722583146 >> v2:172.21.15.81:3300/0 conn(0x7effe80c4580 msgr2=0x7effe80c12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:06.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.049+0000 7effe6000640 1 --2- 172.21.15.28:0/3722583146 >> v2:172.21.15.81:3300/0 conn(0x7effe80c4580 0x7effe80c12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:06.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.049+0000 7effe6000640 1 -- 172.21.15.28:0/3722583146 >> v1:172.21.15.81:6789/0 conn(0x7effe8150240 legacy=0x7effe806db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:06.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.049+0000 7effe6000640 1 -- 172.21.15.28:0/3722583146 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effe806f4a0 con 0x7effe80c2c10 2024-02-21T23:08:06.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.049+0000 7effe6000640 1 -- 172.21.15.28:0/3722583146 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7effe00019f0 con 0x7effe80c2c10 2024-02-21T23:08:06.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.049+0000 7effe6000640 1 -- 172.21.15.28:0/3722583146 >> v1:172.21.15.28:6789/0 conn(0x7effe80c2c10 legacy=0x7effe806ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:06.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.049+0000 7effe6000640 1 --2- 172.21.15.28:0/3722583146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effcc003b40 0x7effcc003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:06.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.049+0000 7effe6000640 1 --2- 172.21.15.28:0/3722583146 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7effcc004800 0x7effcc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:06.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.049+0000 7effe6000640 1 --2- 172.21.15.28:0/3722583146 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7effcc0051c0 0x7effcc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:06.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.049+0000 7effe6000640 1 -- 172.21.15.28:0/3722583146 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effcc00c0c0 con 0x7effcc003b40 2024-02-21T23:08:06.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.049+0000 7effe6000640 1 -- 172.21.15.28:0/3722583146 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effcc00c270 con 0x7effcc004800 2024-02-21T23:08:06.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.049+0000 7effe6000640 1 -- 172.21.15.28:0/3722583146 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effcc00c470 con 0x7effcc0051c0 2024-02-21T23:08:06.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.049+0000 7effe7400640 1 --2- 172.21.15.28:0/3722583146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effcc003b40 0x7effcc003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:06.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.050+0000 7effe7400640 1 -- 172.21.15.28:0/3722583146 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7effcc0051c0 msgr2=0x7effcc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:06.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.050+0000 7effe7400640 1 --2- 172.21.15.28:0/3722583146 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7effcc0051c0 0x7effcc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:06.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.050+0000 7effe6a00640 1 --2- 172.21.15.28:0/3722583146 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7effcc004800 0x7effcc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:06.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.050+0000 7effe7400640 1 -- 172.21.15.28:0/3722583146 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7effcc004800 msgr2=0x7effcc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:06.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.050+0000 7effe7400640 1 --2- 172.21.15.28:0/3722583146 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7effcc004800 0x7effcc004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:06.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.050+0000 7effe7400640 1 -- 172.21.15.28:0/3722583146 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effcc003520 con 0x7effcc003b40 2024-02-21T23:08:06.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.050+0000 7effe7e00640 1 -- 172.21.15.28:0/3722583146 reap_dead start 2024-02-21T23:08:06.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.051+0000 7effe7400640 1 --2- 172.21.15.28:0/3722583146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effcc003b40 0x7effcc003f40 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7effd80018d0 tx=0x7effcc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=608edb550389d29 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:06.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.051+0000 7effe6000640 1 -- 172.21.15.28:0/3722583146 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd8037070 con 0x7effcc003b40 2024-02-21T23:08:06.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.051+0000 7effe6000640 1 -- 172.21.15.28:0/3722583146 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effd80052d0 con 0x7effcc003b40 2024-02-21T23:08:06.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.051+0000 7effe6000640 1 -- 172.21.15.28:0/3722583146 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd80014f0 con 0x7effcc003b40 2024-02-21T23:08:06.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.052+0000 7effece00640 1 -- 172.21.15.28:0/3722583146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effcc003b40 msgr2=0x7effcc003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:06.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.052+0000 7effece00640 1 --2- 172.21.15.28:0/3722583146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effcc003b40 0x7effcc003f40 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7effd80018d0 tx=0x7effcc005830 comp rx=0 tx=0).stop 2024-02-21T23:08:06.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.052+0000 7effece00640 1 -- 172.21.15.28:0/3722583146 shutdown_connections 2024-02-21T23:08:06.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.052+0000 7effece00640 1 --2- 172.21.15.28:0/3722583146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effcc003b40 0x7effcc003f40 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:06.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.052+0000 7effece00640 1 -- 172.21.15.28:0/3722583146 >> 172.21.15.28:0/3722583146 conn(0x7effe80a8530 msgr2=0x7effe80a8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:06.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.052+0000 7effece00640 1 -- 172.21.15.28:0/3722583146 shutdown_connections 2024-02-21T23:08:06.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.052+0000 7effece00640 1 -- 172.21.15.28:0/3722583146 wait complete. 2024-02-21T23:08:06.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.054+0000 7effece00640 1 Processor -- start 2024-02-21T23:08:06.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.054+0000 7effece00640 1 -- start start 2024-02-21T23:08:06.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.055+0000 7effece00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7effcc003b40 0x7effe81de5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:06.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.055+0000 7effece00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7effe80a5d20 0x7effe8071f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:06.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.055+0000 7effece00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effe81eafc0 0x7effe81eb450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:06.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.055+0000 7effece00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe80c4a10 con 0x7effe81eafc0 2024-02-21T23:08:06.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.055+0000 7effece00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe8153a90 con 0x7effcc003b40 2024-02-21T23:08:06.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.055+0000 7effece00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effe80c4b90 con 0x7effe80a5d20 2024-02-21T23:08:06.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.055+0000 7effe7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effe81eafc0 0x7effe81eb450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:06.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.055+0000 7effe7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effe81eafc0 0x7effe81eb450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54424/0 (socket says 172.21.15.28:54424) 2024-02-21T23:08:06.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.055+0000 7effe7e00640 1 -- 172.21.15.28:0/4074846971 learned_addr learned my addr 172.21.15.28:0/4074846971 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:06.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.055+0000 7effe6a00640 1 --2- 172.21.15.28:0/4074846971 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7effe80a5d20 0x7effe8071f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:06.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.056+0000 7effe7400640 1 --2- 172.21.15.28:0/4074846971 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7effcc003b40 0x7effe81de5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:06.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.056+0000 7effe7e00640 1 -- 172.21.15.28:0/4074846971 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7effe80a5d20 msgr2=0x7effe8071f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:06.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.056+0000 7effe7e00640 1 --2- 172.21.15.28:0/4074846971 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7effe80a5d20 0x7effe8071f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:06.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.056+0000 7effe7e00640 1 -- 172.21.15.28:0/4074846971 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7effcc003b40 msgr2=0x7effe81de5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:06.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.056+0000 7effe7e00640 1 --2- 172.21.15.28:0/4074846971 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7effcc003b40 0x7effe81de5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:06.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.056+0000 7effe7e00640 1 -- 172.21.15.28:0/4074846971 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effe81eaa90 con 0x7effe81eafc0 2024-02-21T23:08:06.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.056+0000 7effe7e00640 1 --2- 172.21.15.28:0/4074846971 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effe81eafc0 0x7effe81eb450 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7effe00012e0 tx=0x7effe000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:06.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.056+0000 7effdf400640 1 -- 172.21.15.28:0/4074846971 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effe000a610 con 0x7effe81eafc0 2024-02-21T23:08:06.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.057+0000 7effdf400640 1 -- 172.21.15.28:0/4074846971 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effe000a7d0 con 0x7effe81eafc0 2024-02-21T23:08:06.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.057+0000 7effece00640 1 -- 172.21.15.28:0/4074846971 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effe81dd1f0 con 0x7effe81eafc0 2024-02-21T23:08:06.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.057+0000 7effdf400640 1 -- 172.21.15.28:0/4074846971 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effe000abc0 con 0x7effe81eafc0 2024-02-21T23:08:06.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.057+0000 7effece00640 1 -- 172.21.15.28:0/4074846971 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effe81dd7b0 con 0x7effe81eafc0 2024-02-21T23:08:06.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.058+0000 7effece00640 1 -- 172.21.15.28:0/4074846971 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effe806c440 con 0x7effe81eafc0 2024-02-21T23:08:06.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.059+0000 7effdf400640 1 -- 172.21.15.28:0/4074846971 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7effe0007110 con 0x7effe81eafc0 2024-02-21T23:08:06.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.060+0000 7effdf400640 1 --2- 172.21.15.28:0/4074846971 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7effc806b4d0 0x7effc806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:06.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.061+0000 7effdf400640 1 -- 172.21.15.28:0/4074846971 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5421+0+0 (secure 0 0 0) 0x7effe0089700 con 0x7effe81eafc0 2024-02-21T23:08:06.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.061+0000 7effe7400640 1 --2- 172.21.15.28:0/4074846971 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7effc806b4d0 0x7effc806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:06.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.062+0000 7effe7400640 1 --2- 172.21.15.28:0/4074846971 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7effc806b4d0 0x7effc806d990 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7effd8034000 tx=0x7effd8031000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:06.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.065+0000 7effdf400640 1 -- 172.21.15.28:0/4074846971 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7effe806c440 con 0x7effe81eafc0 2024-02-21T23:08:06.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.262+0000 7effece00640 1 -- 172.21.15.28:0/4074846971 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7effe8074360 con 0x7effe81eafc0 2024-02-21T23:08:06.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.263+0000 7effdf400640 1 -- 172.21.15.28:0/4074846971 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v114) v1 ==== 93+0+10 (secure 0 0 0) 0x7effe8074360 con 0x7effe81eafc0 2024-02-21T23:08:06.252 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-02-21T23:08:06.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.265+0000 7effece00640 1 -- 172.21.15.28:0/4074846971 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7effc806b4d0 msgr2=0x7effc806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:06.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.265+0000 7effece00640 1 --2- 172.21.15.28:0/4074846971 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7effc806b4d0 0x7effc806d990 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7effd8034000 tx=0x7effd8031000 comp rx=0 tx=0).stop 2024-02-21T23:08:06.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.266+0000 7effece00640 1 -- 172.21.15.28:0/4074846971 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effe81eafc0 msgr2=0x7effe81eb450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:06.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.266+0000 7effece00640 1 --2- 172.21.15.28:0/4074846971 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effe81eafc0 0x7effe81eb450 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7effe00012e0 tx=0x7effe000cc60 comp rx=0 tx=0).stop 2024-02-21T23:08:06.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.266+0000 7effece00640 1 -- 172.21.15.28:0/4074846971 shutdown_connections 2024-02-21T23:08:06.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.266+0000 7effece00640 1 --2- 172.21.15.28:0/4074846971 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7effc806b4d0 0x7effc806d990 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:06.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.266+0000 7effece00640 1 --2- 172.21.15.28:0/4074846971 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7effe80a5d20 0x7effe8071f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:06.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.266+0000 7effece00640 1 --2- 172.21.15.28:0/4074846971 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effe81eafc0 0x7effe81eb450 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:06.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.266+0000 7effece00640 1 --2- 172.21.15.28:0/4074846971 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7effcc003b40 0x7effe81de5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:06.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.266+0000 7effece00640 1 -- 172.21.15.28:0/4074846971 >> 172.21.15.28:0/4074846971 conn(0x7effe80a8530 msgr2=0x7effe80a8910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:06.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.266+0000 7effece00640 1 -- 172.21.15.28:0/4074846971 shutdown_connections 2024-02-21T23:08:06.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.266+0000 7effece00640 1 -- 172.21.15.28:0/4074846971 wait complete. 2024-02-21T23:08:06.273 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_steady' 2024-02-21T23:08:06.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.490+0000 7f05f5800640 1 Processor -- start 2024-02-21T23:08:06.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.490+0000 7f05f5800640 1 -- start start 2024-02-21T23:08:06.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.491+0000 7f05f5800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f05f0150450 0x7f05f00a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:06.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.491+0000 7f05f5800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f05f006ee40 0x7f05f01536f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:06.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.491+0000 7f05f5800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05f00c2760 con 0x7f05f0156af0 2024-02-21T23:08:06.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.491+0000 7f05f5800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05f00c22c0 con 0x7f05f0150450 2024-02-21T23:08:06.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.491+0000 7f05f5800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05f0156660 con 0x7f05f006ee40 2024-02-21T23:08:06.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.492+0000 7f05efe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f05f0150450 0x7f05f00a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:06.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.492+0000 7f05f4e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f05f0156af0 0x7f05f006dff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:46680/0 (socket says 172.21.15.28:46680) 2024-02-21T23:08:06.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.492+0000 7f05f4e00640 1 -- 172.21.15.28:0/2348454347 learned_addr learned my addr 172.21.15.28:0/2348454347 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:08:06.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.492+0000 7f05efe00640 1 -- 172.21.15.28:0/2348454347 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f05f006ee40 msgr2=0x7f05f01536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:06.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.492+0000 7f05efe00640 1 --2- 172.21.15.28:0/2348454347 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f05f006ee40 0x7f05f01536f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:06.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.492+0000 7f05efe00640 1 -- 172.21.15.28:0/2348454347 >> v1:172.21.15.28:6789/0 conn(0x7f05f0156af0 legacy=0x7f05f006dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:06.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.492+0000 7f05efe00640 1 -- 172.21.15.28:0/2348454347 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05f0153c30 con 0x7f05f0150450 2024-02-21T23:08:06.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.493+0000 7f05efe00640 1 --2- 172.21.15.28:0/2348454347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05f0150450 0x7f05f00a5fd0 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f05e00097a0 tx=0x7f05e002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dddcb05f900f8fd1 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:06.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.493+0000 7f05eea00640 1 -- 172.21.15.28:0/2348454347 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e0035020 con 0x7f05f0150450 2024-02-21T23:08:06.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.494+0000 7f05eea00640 1 -- 172.21.15.28:0/2348454347 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05e0031810 con 0x7f05f0150450 2024-02-21T23:08:06.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.494+0000 7f05eea00640 1 -- 172.21.15.28:0/2348454347 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e0031be0 con 0x7f05f0150450 2024-02-21T23:08:06.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.494+0000 7f05f5800640 1 -- 172.21.15.28:0/2348454347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05f0150450 msgr2=0x7f05f00a5fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:06.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.494+0000 7f05f5800640 1 --2- 172.21.15.28:0/2348454347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05f0150450 0x7f05f00a5fd0 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f05e00097a0 tx=0x7f05e002b8b0 comp rx=0 tx=0).stop 2024-02-21T23:08:06.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.494+0000 7f05f5800640 1 -- 172.21.15.28:0/2348454347 shutdown_connections 2024-02-21T23:08:06.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.494+0000 7f05f5800640 1 --2- 172.21.15.28:0/2348454347 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f05f006ee40 0x7f05f01536f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:06.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.494+0000 7f05f5800640 1 --2- 172.21.15.28:0/2348454347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05f0150450 0x7f05f00a5fd0 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:06.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.494+0000 7f05f5800640 1 -- 172.21.15.28:0/2348454347 >> 172.21.15.28:0/2348454347 conn(0x7f05f006b630 msgr2=0x7f05f00c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:06.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.494+0000 7f05f5800640 1 -- 172.21.15.28:0/2348454347 shutdown_connections 2024-02-21T23:08:06.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.494+0000 7f05f5800640 1 -- 172.21.15.28:0/2348454347 wait complete. 2024-02-21T23:08:06.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.496+0000 7f05f5800640 1 Processor -- start 2024-02-21T23:08:06.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.496+0000 7f05f5800640 1 -- start start 2024-02-21T23:08:06.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.497+0000 7f05f5800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f05f006ee40 0x7f05f0152540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:06.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.497+0000 7f05f5800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05f0150450 0x7f05f0150d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:06.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.497+0000 7f05f5800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05f0156af0 0x7f05f01512b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:06.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.497+0000 7f05f5800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05f00c2cb0 con 0x7f05f0156af0 2024-02-21T23:08:06.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.497+0000 7f05f5800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05f00c29d0 con 0x7f05f0150450 2024-02-21T23:08:06.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.497+0000 7f05f5800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05f006e700 con 0x7f05f006ee40 2024-02-21T23:08:06.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.497+0000 7f05f4e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05f0156af0 0x7f05f01512b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:06.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.497+0000 7f05f4e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05f0156af0 0x7f05f01512b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54440/0 (socket says 172.21.15.28:54440) 2024-02-21T23:08:06.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.497+0000 7f05f4e00640 1 -- 172.21.15.28:0/2129508002 learned_addr learned my addr 172.21.15.28:0/2129508002 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:06.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.498+0000 7f05efe00640 1 --2- 172.21.15.28:0/2129508002 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f05f006ee40 0x7f05f0152540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:06.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.498+0000 7f05f4e00640 1 -- 172.21.15.28:0/2129508002 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f05f006ee40 msgr2=0x7f05f0152540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:06.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.498+0000 7f05f4e00640 1 --2- 172.21.15.28:0/2129508002 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f05f006ee40 0x7f05f0152540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:06.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.498+0000 7f05f4e00640 1 -- 172.21.15.28:0/2129508002 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05f0150450 msgr2=0x7f05f0150d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:06.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.498+0000 7f05f4e00640 1 --2- 172.21.15.28:0/2129508002 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05f0150450 0x7f05f0150d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:06.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.498+0000 7f05f4e00640 1 -- 172.21.15.28:0/2129508002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05f0153e00 con 0x7f05f0156af0 2024-02-21T23:08:06.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.498+0000 7f05f4e00640 1 --2- 172.21.15.28:0/2129508002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05f0156af0 0x7f05f01512b0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f05e800ca80 tx=0x7f05e800cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:06.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.498+0000 7f05e7e00640 1 -- 172.21.15.28:0/2129508002 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e8017070 con 0x7f05f0156af0 2024-02-21T23:08:06.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.498+0000 7f05e7e00640 1 -- 172.21.15.28:0/2129508002 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05e800ac30 con 0x7f05f0156af0 2024-02-21T23:08:06.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.498+0000 7f05e7e00640 1 -- 172.21.15.28:0/2129508002 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e8012400 con 0x7f05f0156af0 2024-02-21T23:08:06.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.498+0000 7f05f5800640 1 -- 172.21.15.28:0/2129508002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05f01540e0 con 0x7f05f0156af0 2024-02-21T23:08:06.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.499+0000 7f05f5800640 1 -- 172.21.15.28:0/2129508002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05f01546a0 con 0x7f05f0156af0 2024-02-21T23:08:06.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.500+0000 7f05f5800640 1 -- 172.21.15.28:0/2129508002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05f004f290 con 0x7f05f0156af0 2024-02-21T23:08:06.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.501+0000 7f05e7e00640 1 -- 172.21.15.28:0/2129508002 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f05e801a070 con 0x7f05f0156af0 2024-02-21T23:08:06.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.501+0000 7f05e7e00640 1 --2- 172.21.15.28:0/2129508002 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f05d006b4d0 0x7f05d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:06.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.502+0000 7f05e7e00640 1 -- 172.21.15.28:0/2129508002 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f05e8089cb0 con 0x7f05f0156af0 2024-02-21T23:08:06.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.502+0000 7f05efe00640 1 --2- 172.21.15.28:0/2129508002 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f05d006b4d0 0x7f05d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:06.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.503+0000 7f05efe00640 1 --2- 172.21.15.28:0/2129508002 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f05d006b4d0 0x7f05d006d990 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f05e0036000 tx=0x7f05e0008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:06.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.506+0000 7f05e7e00640 1 -- 172.21.15.28:0/2129508002 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f05e8058600 con 0x7f05f0156af0 2024-02-21T23:08:06.638 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-21T23:08:06.645 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-21T23:08:06.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:06.776+0000 7f05f5800640 1 -- 172.21.15.28:0/2129508002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_steady"]} v 0) v1 -- 0x7f05f0151ba0 con 0x7f05f0156af0 2024-02-21T23:08:07.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.324+0000 7f05e7e00640 1 -- 172.21.15.28:0/2129508002 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_steady"]}]=0 v328) v1 ==== 148+0+0 (secure 0 0 0) 0x7f05e805c6a0 con 0x7f05f0156af0 2024-02-21T23:08:07.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.328+0000 7f05f5800640 1 -- 172.21.15.28:0/2129508002 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f05d006b4d0 msgr2=0x7f05d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:07.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.328+0000 7f05f5800640 1 --2- 172.21.15.28:0/2129508002 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f05d006b4d0 0x7f05d006d990 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f05e0036000 tx=0x7f05e0008030 comp rx=0 tx=0).stop 2024-02-21T23:08:07.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.328+0000 7f05f5800640 1 -- 172.21.15.28:0/2129508002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05f0156af0 msgr2=0x7f05f01512b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:07.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.328+0000 7f05f5800640 1 --2- 172.21.15.28:0/2129508002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05f0156af0 0x7f05f01512b0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f05e800ca80 tx=0x7f05e800cf40 comp rx=0 tx=0).stop 2024-02-21T23:08:07.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.328+0000 7f05f5800640 1 -- 172.21.15.28:0/2129508002 shutdown_connections 2024-02-21T23:08:07.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.328+0000 7f05f5800640 1 --2- 172.21.15.28:0/2129508002 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f05d006b4d0 0x7f05d006d990 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:07.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.328+0000 7f05f5800640 1 --2- 172.21.15.28:0/2129508002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05f0156af0 0x7f05f01512b0 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:07.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.328+0000 7f05f5800640 1 --2- 172.21.15.28:0/2129508002 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05f0150450 0x7f05f0150d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:07.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.328+0000 7f05f5800640 1 --2- 172.21.15.28:0/2129508002 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f05f006ee40 0x7f05f0152540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:07.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.328+0000 7f05f5800640 1 -- 172.21.15.28:0/2129508002 >> 172.21.15.28:0/2129508002 conn(0x7f05f006b630 msgr2=0x7f05f00c04b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:07.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.329+0000 7f05f5800640 1 -- 172.21.15.28:0/2129508002 shutdown_connections 2024-02-21T23:08:07.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.329+0000 7f05f5800640 1 -- 172.21.15.28:0/2129508002 wait complete. 2024-02-21T23:08:07.342 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-02-21T23:08:07.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.573+0000 7fb255e00640 1 Processor -- start 2024-02-21T23:08:07.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.573+0000 7fb255e00640 1 -- start start 2024-02-21T23:08:07.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.575+0000 7fb255e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb250158480 0x7fb250154f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:07.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.575+0000 7fb255e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fb25006e9d0 0x7fb25006ee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:07.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.575+0000 7fb255e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb25006f3a0 con 0x7fb250152560 2024-02-21T23:08:07.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.575+0000 7fb255e00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2501554a0 con 0x7fb25006e9d0 2024-02-21T23:08:07.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.575+0000 7fb255e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2500a6270 con 0x7fb250158480 2024-02-21T23:08:07.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.576+0000 7fb24fe00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb250158480 0x7fb250154f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:07.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.576+0000 7fb24fe00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb250158480 0x7fb250154f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3301/0 says I am v2:172.21.15.28:46408/0 (socket says 172.21.15.28:46408) 2024-02-21T23:08:07.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.576+0000 7fb24fe00640 1 -- 172.21.15.28:0/1922167031 learned_addr learned my addr 172.21.15.28:0/1922167031 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:07.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.576+0000 7fb255400640 1 --2- 172.21.15.28:0/1922167031 >> v2:172.21.15.81:3300/0 conn(0x7fb25006e9d0 0x7fb25006ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:07.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.576+0000 7fb24fe00640 1 -- 172.21.15.28:0/1922167031 >> v2:172.21.15.81:3300/0 conn(0x7fb25006e9d0 msgr2=0x7fb25006ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:07.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.576+0000 7fb24fe00640 1 --2- 172.21.15.28:0/1922167031 >> v2:172.21.15.81:3300/0 conn(0x7fb25006e9d0 0x7fb25006ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:07.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.576+0000 7fb24fe00640 1 -- 172.21.15.28:0/1922167031 >> v1:172.21.15.81:6789/0 conn(0x7fb250152560 legacy=0x7fb25006ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:07.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.576+0000 7fb24fe00640 1 -- 172.21.15.28:0/1922167031 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2501556d0 con 0x7fb250158480 2024-02-21T23:08:07.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.578+0000 7fb24fe00640 1 --2- 172.21.15.28:0/1922167031 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb250158480 0x7fb250154f60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb23c009870 tx=0x7fb23c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=42c79ee2f4032011 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:07.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.578+0000 7fb24f400640 1 -- 172.21.15.28:0/1922167031 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb23c034070 con 0x7fb250158480 2024-02-21T23:08:07.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.578+0000 7fb24f400640 1 -- 172.21.15.28:0/1922167031 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb23c030840 con 0x7fb250158480 2024-02-21T23:08:07.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.578+0000 7fb24f400640 1 -- 172.21.15.28:0/1922167031 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb23c030c10 con 0x7fb250158480 2024-02-21T23:08:07.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.579+0000 7fb255e00640 1 -- 172.21.15.28:0/1922167031 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb250158480 msgr2=0x7fb250154f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:07.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.579+0000 7fb255e00640 1 --2- 172.21.15.28:0/1922167031 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb250158480 0x7fb250154f60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb23c009870 tx=0x7fb23c02b980 comp rx=0 tx=0).stop 2024-02-21T23:08:07.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.579+0000 7fb255e00640 1 -- 172.21.15.28:0/1922167031 shutdown_connections 2024-02-21T23:08:07.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.579+0000 7fb255e00640 1 --2- 172.21.15.28:0/1922167031 >> v2:172.21.15.81:3300/0 conn(0x7fb25006e9d0 0x7fb25006ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:07.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.579+0000 7fb255e00640 1 --2- 172.21.15.28:0/1922167031 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb250158480 0x7fb250154f60 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:07.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.579+0000 7fb255e00640 1 -- 172.21.15.28:0/1922167031 >> 172.21.15.28:0/1922167031 conn(0x7fb2500a8530 msgr2=0x7fb2500a8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:07.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.579+0000 7fb255e00640 1 -- 172.21.15.28:0/1922167031 shutdown_connections 2024-02-21T23:08:07.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.579+0000 7fb255e00640 1 -- 172.21.15.28:0/1922167031 wait complete. 2024-02-21T23:08:07.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.581+0000 7fb255e00640 1 Processor -- start 2024-02-21T23:08:07.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.582+0000 7fb255e00640 1 -- start start 2024-02-21T23:08:07.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.583+0000 7fb255e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb25006e9d0 0x7fb250150880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:07.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.583+0000 7fb255e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb250152560 0x7fb2500710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:07.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.583+0000 7fb255e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb250158480 0x7fb2501e7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:07.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.583+0000 7fb255e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb25006e4b0 con 0x7fb25006e9d0 2024-02-21T23:08:07.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.583+0000 7fb255e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb250155c90 con 0x7fb250158480 2024-02-21T23:08:07.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.583+0000 7fb255e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb250152940 con 0x7fb250152560 2024-02-21T23:08:07.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.583+0000 7fb254a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb25006e9d0 0x7fb250150880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:07.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.583+0000 7fb254a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb25006e9d0 0x7fb250150880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54450/0 (socket says 172.21.15.28:54450) 2024-02-21T23:08:07.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.583+0000 7fb254a00640 1 -- 172.21.15.28:0/462508283 learned_addr learned my addr 172.21.15.28:0/462508283 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:07.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.583+0000 7fb24fe00640 1 --2- 172.21.15.28:0/462508283 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb250152560 0x7fb2500710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:07.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.583+0000 7fb255400640 1 --2- 172.21.15.28:0/462508283 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb250158480 0x7fb2501e7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:07.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.584+0000 7fb254a00640 1 -- 172.21.15.28:0/462508283 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb250152560 msgr2=0x7fb2500710f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:07.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.584+0000 7fb254a00640 1 --2- 172.21.15.28:0/462508283 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb250152560 0x7fb2500710f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:07.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.584+0000 7fb254a00640 1 -- 172.21.15.28:0/462508283 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb250158480 msgr2=0x7fb2501e7ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:07.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.584+0000 7fb254a00640 1 --2- 172.21.15.28:0/462508283 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb250158480 0x7fb2501e7ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:07.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.584+0000 7fb254a00640 1 -- 172.21.15.28:0/462508283 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb250071b70 con 0x7fb25006e9d0 2024-02-21T23:08:07.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.584+0000 7fb254a00640 1 --2- 172.21.15.28:0/462508283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb25006e9d0 0x7fb250150880 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7fb24000c400 tx=0x7fb24000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:07.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.584+0000 7fb24cc00640 1 -- 172.21.15.28:0/462508283 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2400052d0 con 0x7fb25006e9d0 2024-02-21T23:08:07.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.584+0000 7fb24cc00640 1 -- 172.21.15.28:0/462508283 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb240005490 con 0x7fb25006e9d0 2024-02-21T23:08:07.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.584+0000 7fb24cc00640 1 -- 172.21.15.28:0/462508283 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb240007260 con 0x7fb25006e9d0 2024-02-21T23:08:07.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.584+0000 7fb255e00640 1 -- 172.21.15.28:0/462508283 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb250071e50 con 0x7fb25006e9d0 2024-02-21T23:08:07.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.585+0000 7fb255e00640 1 -- 172.21.15.28:0/462508283 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb250154150 con 0x7fb25006e9d0 2024-02-21T23:08:07.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.586+0000 7fb255e00640 1 -- 172.21.15.28:0/462508283 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb25004f290 con 0x7fb25006e9d0 2024-02-21T23:08:07.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.587+0000 7fb24cc00640 1 -- 172.21.15.28:0/462508283 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb240007400 con 0x7fb25006e9d0 2024-02-21T23:08:07.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.588+0000 7fb24cc00640 1 --2- 172.21.15.28:0/462508283 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb23006b4d0 0x7fb23006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:07.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.588+0000 7fb24cc00640 1 -- 172.21.15.28:0/462508283 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5421+0+0 (secure 0 0 0) 0x7fb2400896d0 con 0x7fb25006e9d0 2024-02-21T23:08:07.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.588+0000 7fb24fe00640 1 --2- 172.21.15.28:0/462508283 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb23006b4d0 0x7fb23006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:07.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.590+0000 7fb24fe00640 1 --2- 172.21.15.28:0/462508283 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb23006b4d0 0x7fb23006d990 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fb23c036000 tx=0x7fb23c03b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:07.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.593+0000 7fb24cc00640 1 -- 172.21.15.28:0/462508283 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb2400585d0 con 0x7fb25006e9d0 2024-02-21T23:08:07.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.791+0000 7fb255e00640 1 -- 172.21.15.28:0/462508283 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fb25006e0f0 con 0x7fb25006e9d0 2024-02-21T23:08:07.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.795+0000 7fb24cc00640 1 -- 172.21.15.28:0/462508283 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v41)=0 v41) v1 ==== 122+0+0 (secure 0 0 0) 0x7fb240057ed0 con 0x7fb25006e9d0 2024-02-21T23:08:07.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.798+0000 7fb255e00640 1 -- 172.21.15.28:0/462508283 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb23006b4d0 msgr2=0x7fb23006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:07.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.798+0000 7fb255e00640 1 --2- 172.21.15.28:0/462508283 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb23006b4d0 0x7fb23006d990 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fb23c036000 tx=0x7fb23c03b040 comp rx=0 tx=0).stop 2024-02-21T23:08:07.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.798+0000 7fb255e00640 1 -- 172.21.15.28:0/462508283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb25006e9d0 msgr2=0x7fb250150880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:07.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.798+0000 7fb255e00640 1 --2- 172.21.15.28:0/462508283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb25006e9d0 0x7fb250150880 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7fb24000c400 tx=0x7fb24000c8c0 comp rx=0 tx=0).stop 2024-02-21T23:08:07.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.798+0000 7fb255e00640 1 -- 172.21.15.28:0/462508283 shutdown_connections 2024-02-21T23:08:07.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.798+0000 7fb255e00640 1 --2- 172.21.15.28:0/462508283 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb23006b4d0 0x7fb23006d990 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:07.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.798+0000 7fb255e00640 1 --2- 172.21.15.28:0/462508283 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb250152560 0x7fb2500710f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:07.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.798+0000 7fb255e00640 1 --2- 172.21.15.28:0/462508283 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb250158480 0x7fb2501e7ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:07.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.798+0000 7fb255e00640 1 --2- 172.21.15.28:0/462508283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb25006e9d0 0x7fb250150880 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:07.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.798+0000 7fb255e00640 1 -- 172.21.15.28:0/462508283 >> 172.21.15.28:0/462508283 conn(0x7fb2500a8530 msgr2=0x7fb2500a7810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:07.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.798+0000 7fb255e00640 1 -- 172.21.15.28:0/462508283 shutdown_connections 2024-02-21T23:08:07.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:07.798+0000 7fb255e00640 1 -- 172.21.15.28:0/462508283 wait complete. 2024-02-21T23:08:07.805 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-02-21T23:08:07.807 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-21T23:08:07.922 DEBUG:teuthology.orchestra.run.smithi081:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-21T23:08:08.043 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-02-21T23:08:08.043 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-21T23:08:08.043 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-21T23:08:08.103 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054608,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-02-21T23:08:08.103 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-02-21T23:08:08.103 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-21T23:08:08.159 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053833,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498812k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249404k,nr_inodes=812351,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498812k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-02-21T23:08:08.160 INFO:tasks.cephfs.cephfs_test_case:[, ] 2024-02-21T23:08:08.160 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:08:08.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.401+0000 7f18d6a00640 1 Processor -- start 2024-02-21T23:08:08.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.401+0000 7f18d6a00640 1 -- start start 2024-02-21T23:08:08.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.402+0000 7f18d6a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f18d0150450 0x7f18d00a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:08.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.403+0000 7f18d6a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f18d006ee40 0x7f18d01536f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:08.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.403+0000 7f18d6a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18d00c2760 con 0x7f18d0156af0 2024-02-21T23:08:08.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.403+0000 7f18d6a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18d00c22c0 con 0x7f18d0150450 2024-02-21T23:08:08.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.403+0000 7f18d6a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18d0156660 con 0x7f18d006ee40 2024-02-21T23:08:08.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.403+0000 7f18d5600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f18d0150450 0x7f18d00a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:08.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.403+0000 7f18d5600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f18d0150450 0x7f18d00a5fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54456/0 (socket says 172.21.15.28:54456) 2024-02-21T23:08:08.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.403+0000 7f18d5600640 1 -- 172.21.15.28:0/3536624884 learned_addr learned my addr 172.21.15.28:0/3536624884 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:08.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.403+0000 7f18d4c00640 1 --2- 172.21.15.28:0/3536624884 >> v2:172.21.15.81:3300/0 conn(0x7f18d006ee40 0x7f18d01536f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:08.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.404+0000 7f18d5600640 1 -- 172.21.15.28:0/3536624884 >> v2:172.21.15.81:3300/0 conn(0x7f18d006ee40 msgr2=0x7f18d01536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:08.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.404+0000 7f18d5600640 1 --2- 172.21.15.28:0/3536624884 >> v2:172.21.15.81:3300/0 conn(0x7f18d006ee40 0x7f18d01536f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:08.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.404+0000 7f18d5600640 1 -- 172.21.15.28:0/3536624884 >> v1:172.21.15.81:6789/0 conn(0x7f18d0156af0 legacy=0x7f18d006dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:08.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.404+0000 7f18d5600640 1 -- 172.21.15.28:0/3536624884 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18d0153c30 con 0x7f18d0150450 2024-02-21T23:08:08.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.405+0000 7f18d5600640 1 --2- 172.21.15.28:0/3536624884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18d0150450 0x7f18d00a5fd0 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f18d006bcc0 tx=0x7f18c402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c603c2722999e68e server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:08.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.405+0000 7f18cfe00640 1 -- 172.21.15.28:0/3536624884 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18c4035020 con 0x7f18d0150450 2024-02-21T23:08:08.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.405+0000 7f18cfe00640 1 -- 172.21.15.28:0/3536624884 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18c4031870 con 0x7f18d0150450 2024-02-21T23:08:08.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.406+0000 7f18cfe00640 1 -- 172.21.15.28:0/3536624884 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18c4031c40 con 0x7f18d0150450 2024-02-21T23:08:08.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.406+0000 7f18d6a00640 1 -- 172.21.15.28:0/3536624884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18d0150450 msgr2=0x7f18d00a5fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:08.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.406+0000 7f18d6a00640 1 --2- 172.21.15.28:0/3536624884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18d0150450 0x7f18d00a5fd0 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f18d006bcc0 tx=0x7f18c402b910 comp rx=0 tx=0).stop 2024-02-21T23:08:08.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.406+0000 7f18d6a00640 1 -- 172.21.15.28:0/3536624884 shutdown_connections 2024-02-21T23:08:08.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.406+0000 7f18d6a00640 1 --2- 172.21.15.28:0/3536624884 >> v2:172.21.15.81:3300/0 conn(0x7f18d006ee40 0x7f18d01536f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:08.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.406+0000 7f18d6a00640 1 --2- 172.21.15.28:0/3536624884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18d0150450 0x7f18d00a5fd0 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:08.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.406+0000 7f18d6a00640 1 -- 172.21.15.28:0/3536624884 >> 172.21.15.28:0/3536624884 conn(0x7f18d006b630 msgr2=0x7f18d00c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:08.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.406+0000 7f18d6a00640 1 -- 172.21.15.28:0/3536624884 shutdown_connections 2024-02-21T23:08:08.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.406+0000 7f18d6a00640 1 -- 172.21.15.28:0/3536624884 wait complete. 2024-02-21T23:08:08.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.408+0000 7f18d6a00640 1 Processor -- start 2024-02-21T23:08:08.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.408+0000 7f18d6a00640 1 -- start start 2024-02-21T23:08:08.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.409+0000 7f18d6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18d006ee40 0x7f18d00bfba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:08.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.409+0000 7f18d6a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f18d0150450 0x7f18d0149650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:08.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.409+0000 7f18d6a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f18d0156af0 0x7f18d0148990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:08.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.409+0000 7f18d6a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18d00c2cb0 con 0x7f18d006ee40 2024-02-21T23:08:08.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.409+0000 7f18d6a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18d00c29d0 con 0x7f18d0150450 2024-02-21T23:08:08.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.409+0000 7f18d6a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18d006e700 con 0x7f18d0156af0 2024-02-21T23:08:08.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.410+0000 7f18d5600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18d006ee40 0x7f18d00bfba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:08.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.410+0000 7f18d5600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18d006ee40 0x7f18d00bfba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54472/0 (socket says 172.21.15.28:54472) 2024-02-21T23:08:08.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.410+0000 7f18d5600640 1 -- 172.21.15.28:0/3583194062 learned_addr learned my addr 172.21.15.28:0/3583194062 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:08.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.410+0000 7f18d6000640 1 --2- 172.21.15.28:0/3583194062 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f18d0156af0 0x7f18d0148990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:08.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.410+0000 7f18d4c00640 1 --2- 172.21.15.28:0/3583194062 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f18d0150450 0x7f18d0149650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:08.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.410+0000 7f18d5600640 1 -- 172.21.15.28:0/3583194062 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f18d0156af0 msgr2=0x7f18d0148990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:08.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.410+0000 7f18d5600640 1 --2- 172.21.15.28:0/3583194062 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f18d0156af0 0x7f18d0148990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:08.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.410+0000 7f18d5600640 1 -- 172.21.15.28:0/3583194062 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f18d0150450 msgr2=0x7f18d0149650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:08.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.410+0000 7f18d5600640 1 --2- 172.21.15.28:0/3583194062 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f18d0150450 0x7f18d0149650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:08.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.410+0000 7f18d5600640 1 -- 172.21.15.28:0/3583194062 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18d0155420 con 0x7f18d006ee40 2024-02-21T23:08:08.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.411+0000 7f18d5600640 1 --2- 172.21.15.28:0/3583194062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18d006ee40 0x7f18d00bfba0 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f18c402b950 tx=0x7f18c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:08.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.411+0000 7f18cd600640 1 -- 172.21.15.28:0/3583194062 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18c4040070 con 0x7f18d006ee40 2024-02-21T23:08:08.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.411+0000 7f18cd600640 1 -- 172.21.15.28:0/3583194062 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18c4030d00 con 0x7f18d006ee40 2024-02-21T23:08:08.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.411+0000 7f18d6a00640 1 -- 172.21.15.28:0/3583194062 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18d0155700 con 0x7f18d006ee40 2024-02-21T23:08:08.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.411+0000 7f18cd600640 1 -- 172.21.15.28:0/3583194062 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18c4035040 con 0x7f18d006ee40 2024-02-21T23:08:08.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.411+0000 7f18d6a00640 1 -- 172.21.15.28:0/3583194062 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18d0155cc0 con 0x7f18d006ee40 2024-02-21T23:08:08.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.412+0000 7f18d6a00640 1 -- 172.21.15.28:0/3583194062 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18d004f290 con 0x7f18d006ee40 2024-02-21T23:08:08.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.414+0000 7f18cd600640 1 -- 172.21.15.28:0/3583194062 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f18c404b060 con 0x7f18d006ee40 2024-02-21T23:08:08.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.414+0000 7f18cd600640 1 --2- 172.21.15.28:0/3583194062 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f18b006b500 0x7f18b006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:08.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.415+0000 7f18cd600640 1 -- 172.21.15.28:0/3583194062 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f18c40abe90 con 0x7f18d006ee40 2024-02-21T23:08:08.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.415+0000 7f18d4c00640 1 --2- 172.21.15.28:0/3583194062 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f18b006b500 0x7f18b006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:08.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.416+0000 7f18d4c00640 1 --2- 172.21.15.28:0/3583194062 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f18b006b500 0x7f18b006d9c0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f18bc006fd0 tx=0x7f18bc008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:08.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.421+0000 7f18cd600640 1 -- 172.21.15.28:0/3583194062 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f18c407ae30 con 0x7f18d006ee40 2024-02-21T23:08:08.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.661+0000 7f18d6a00640 1 -- 172.21.15.28:0/3583194062 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f18d014c680 con 0x7f18d006ee40 2024-02-21T23:08:08.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.662+0000 7f18cd600640 1 -- 172.21.15.28:0/3583194062 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) v1 ==== 95+0+3703 (secure 0 0 0) 0x7f18c407a730 con 0x7f18d006ee40 2024-02-21T23:08:08.651 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:08:08.651 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":114,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5843,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3059988835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3059988835},{"type":"v1","addr":"172.21.15.28:6835","nonce":3059988835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":112},{"gid":6406,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/3795687424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":3795687424},{"type":"v1","addr":"172.21.15.81:6835","nonce":3795687424}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":114},{"gid":6412,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3147237867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3147237867},{"type":"v1","addr":"172.21.15.81:6837","nonce":3147237867}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":114},{"gid":7446,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/1119726035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1119726035},{"type":"v1","addr":"172.21.15.28:6833","nonce":1119726035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":96}],"filesystems":[]} 2024-02-21T23:08:08.651 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 114 2024-02-21T23:08:08.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.665+0000 7f18d6a00640 1 -- 172.21.15.28:0/3583194062 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f18b006b500 msgr2=0x7f18b006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:08.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.665+0000 7f18d6a00640 1 --2- 172.21.15.28:0/3583194062 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f18b006b500 0x7f18b006d9c0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f18bc006fd0 tx=0x7f18bc008040 comp rx=0 tx=0).stop 2024-02-21T23:08:08.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.665+0000 7f18d6a00640 1 -- 172.21.15.28:0/3583194062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18d006ee40 msgr2=0x7f18d00bfba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:08.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.665+0000 7f18d6a00640 1 --2- 172.21.15.28:0/3583194062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18d006ee40 0x7f18d00bfba0 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f18c402b950 tx=0x7f18c40046c0 comp rx=0 tx=0).stop 2024-02-21T23:08:08.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.665+0000 7f18d6a00640 1 -- 172.21.15.28:0/3583194062 shutdown_connections 2024-02-21T23:08:08.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.665+0000 7f18d6a00640 1 --2- 172.21.15.28:0/3583194062 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f18b006b500 0x7f18b006d9c0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:08.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.665+0000 7f18d6a00640 1 --2- 172.21.15.28:0/3583194062 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f18d0156af0 0x7f18d0148990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:08.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.665+0000 7f18d6a00640 1 --2- 172.21.15.28:0/3583194062 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f18d0150450 0x7f18d0149650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:08.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.665+0000 7f18d6a00640 1 --2- 172.21.15.28:0/3583194062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18d006ee40 0x7f18d00bfba0 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:08.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.665+0000 7f18d6a00640 1 -- 172.21.15.28:0/3583194062 >> 172.21.15.28:0/3583194062 conn(0x7f18d006b630 msgr2=0x7f18d00c2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:08.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.666+0000 7f18d6a00640 1 -- 172.21.15.28:0/3583194062 shutdown_connections 2024-02-21T23:08:08.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.666+0000 7f18d6a00640 1 -- 172.21.15.28:0/3583194062 wait complete. 2024-02-21T23:08:08.672 INFO:tasks.ceph.mds.a:Restarting daemon 2024-02-21T23:08:08.672 INFO:tasks.ceph.mds.a:Stopping old one... 2024-02-21T23:08:08.672 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-02-21T23:08:08.673 INFO:teuthology.orchestra.run:waiting for 300 2024-02-21T23:08:08.673 INFO:tasks.ceph.mds.c:Restarting daemon 2024-02-21T23:08:08.673 INFO:tasks.ceph.mds.c:Stopping old one... 2024-02-21T23:08:08.673 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-02-21T23:08:08.673 INFO:teuthology.orchestra.run:waiting for 300 2024-02-21T23:08:08.674 INFO:tasks.ceph.mds.b:Restarting daemon 2024-02-21T23:08:08.674 INFO:tasks.ceph.mds.b:Stopping old one... 2024-02-21T23:08:08.674 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-02-21T23:08:08.674 INFO:teuthology.orchestra.run:waiting for 300 2024-02-21T23:08:08.674 INFO:tasks.ceph.mds.d:Restarting daemon 2024-02-21T23:08:08.674 INFO:tasks.ceph.mds.d:Stopping old one... 2024-02-21T23:08:08.674 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-02-21T23:08:08.675 INFO:teuthology.orchestra.run:waiting for 300 2024-02-21T23:08:08.692 INFO:tasks.ceph.mds.c:Stopped 2024-02-21T23:08:08.692 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-02-21T23:08:08.694 INFO:tasks.ceph.mds.a:Stopped 2024-02-21T23:08:08.694 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-02-21T23:08:08.698 INFO:tasks.ceph.mds.b:Stopped 2024-02-21T23:08:08.698 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-02-21T23:08:08.700 INFO:tasks.ceph.mds.d:Stopped 2024-02-21T23:08:08.700 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-02-21T23:08:08.741 INFO:tasks.ceph.mds.c:Started 2024-02-21T23:08:08.741 DEBUG:teuthology.parallel:result is None 2024-02-21T23:08:08.747 INFO:tasks.ceph.mds.b:Started 2024-02-21T23:08:08.748 INFO:tasks.ceph.mds.a:Started 2024-02-21T23:08:08.748 DEBUG:teuthology.parallel:result is None 2024-02-21T23:08:08.748 DEBUG:teuthology.parallel:result is None 2024-02-21T23:08:08.754 INFO:tasks.ceph.mds.d:Started 2024-02-21T23:08:08.754 DEBUG:teuthology.parallel:result is None 2024-02-21T23:08:08.754 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-02-21T23:08:08.901 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-02-21T23:08:08.901 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-02-21T23:08:08.908 INFO:tasks.ceph.mds.b.smithi081.stdout:starting mds.b at 2024-02-21T23:08:08.909 INFO:tasks.ceph.mds.d.smithi081.stdout:starting mds.d at 2024-02-21T23:08:08.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.952+0000 7fbad7a00640 1 Processor -- start 2024-02-21T23:08:08.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.953+0000 7fbad7a00640 1 -- start start 2024-02-21T23:08:08.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.954+0000 7fbad7a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbad006ae70 0x7fbad0159050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:08.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.954+0000 7fbad7a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbad0072400 0x7fbad0072880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:08.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.954+0000 7fbad7a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbad006b6b0 con 0x7fbad006b250 2024-02-21T23:08:08.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.954+0000 7fbad7a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbad0155cc0 con 0x7fbad006ae70 2024-02-21T23:08:08.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.954+0000 7fbad7a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbad0155390 con 0x7fbad0072400 2024-02-21T23:08:08.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.954+0000 7fbad5000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbad006ae70 0x7fbad0159050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:08.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.954+0000 7fbacfe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fbad006b250 0x7fbad00c17f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:46692/0 (socket says 172.21.15.28:46692) 2024-02-21T23:08:08.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.954+0000 7fbacfe00640 1 -- 172.21.15.28:0/2542627096 learned_addr learned my addr 172.21.15.28:0/2542627096 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:08:08.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.954+0000 7fbad5a00640 1 --2- 172.21.15.28:0/2542627096 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbad0072400 0x7fbad0072880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:08.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.955+0000 7fbad5000640 1 -- 172.21.15.28:0/2542627096 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbad0072400 msgr2=0x7fbad0072880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:08.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.955+0000 7fbad5000640 1 --2- 172.21.15.28:0/2542627096 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbad0072400 0x7fbad0072880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:08.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.955+0000 7fbad5000640 1 -- 172.21.15.28:0/2542627096 >> v1:172.21.15.28:6789/0 conn(0x7fbad006b250 legacy=0x7fbad00c17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:08.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.955+0000 7fbad5000640 1 -- 172.21.15.28:0/2542627096 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbad0155670 con 0x7fbad006ae70 2024-02-21T23:08:08.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.955+0000 7fbacf400640 1 -- 172.21.15.28:0/2542627096 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3625363775 0 0) 0x7fbad006b6b0 con 0x7fbad006b250 2024-02-21T23:08:08.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.956+0000 7fbad5000640 1 --2- 172.21.15.28:0/2542627096 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbad006ae70 0x7fbad0159050 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7fbac40097a0 tx=0x7fbac402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=642eef3579fb34f9 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:08.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.956+0000 7fbacf400640 1 -- 172.21.15.28:0/2542627096 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbac40315a0 con 0x7fbad006ae70 2024-02-21T23:08:08.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.956+0000 7fbacf400640 1 -- 172.21.15.28:0/2542627096 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbac4037070 con 0x7fbad006ae70 2024-02-21T23:08:08.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.956+0000 7fbacf400640 1 -- 172.21.15.28:0/2542627096 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbac4031960 con 0x7fbad006ae70 2024-02-21T23:08:08.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.957+0000 7fbad7a00640 1 -- 172.21.15.28:0/2542627096 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbad006ae70 msgr2=0x7fbad0159050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:08.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.957+0000 7fbad7a00640 1 --2- 172.21.15.28:0/2542627096 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbad006ae70 0x7fbad0159050 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7fbac40097a0 tx=0x7fbac402b840 comp rx=0 tx=0).stop 2024-02-21T23:08:08.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.957+0000 7fbad7a00640 1 -- 172.21.15.28:0/2542627096 shutdown_connections 2024-02-21T23:08:08.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.957+0000 7fbad7a00640 1 --2- 172.21.15.28:0/2542627096 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbad0072400 0x7fbad0072880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:08.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.957+0000 7fbad7a00640 1 --2- 172.21.15.28:0/2542627096 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbad006ae70 0x7fbad0159050 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:08.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.957+0000 7fbad7a00640 1 -- 172.21.15.28:0/2542627096 >> 172.21.15.28:0/2542627096 conn(0x7fbad00a9690 msgr2=0x7fbad00a9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:08.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.957+0000 7fbad7a00640 1 -- 172.21.15.28:0/2542627096 shutdown_connections 2024-02-21T23:08:08.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.957+0000 7fbad7a00640 1 -- 172.21.15.28:0/2542627096 wait complete. 2024-02-21T23:08:08.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.959+0000 7fbad7a00640 1 Processor -- start 2024-02-21T23:08:08.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.959+0000 7fbad7a00640 1 -- start start 2024-02-21T23:08:08.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.960+0000 7fbad7a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbad006ae70 0x7fbad0157b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:08.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.960+0000 7fbad7a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbad006b250 0x7fbad01ebfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:08.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.960+0000 7fbad7a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbad0072400 0x7fbad01ec520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:08.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.960+0000 7fbad7a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbad0152960 con 0x7fbad0072400 2024-02-21T23:08:08.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.960+0000 7fbad7a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbad0155040 con 0x7fbad006ae70 2024-02-21T23:08:08.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.960+0000 7fbad7a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbad0152ae0 con 0x7fbad006b250 2024-02-21T23:08:08.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.960+0000 7fbad5a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbad0072400 0x7fbad01ec520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:08.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.960+0000 7fbad5a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbad0072400 0x7fbad01ec520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54528/0 (socket says 172.21.15.28:54528) 2024-02-21T23:08:08.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.960+0000 7fbad5a00640 1 -- 172.21.15.28:0/3400950613 learned_addr learned my addr 172.21.15.28:0/3400950613 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:08.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.960+0000 7fbad5000640 1 --2- 172.21.15.28:0/3400950613 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbad006ae70 0x7fbad0157b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:08.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.960+0000 7fbacfe00640 1 --2- 172.21.15.28:0/3400950613 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbad006b250 0x7fbad01ebfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:08.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.960+0000 7fbad5a00640 1 -- 172.21.15.28:0/3400950613 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbad006b250 msgr2=0x7fbad01ebfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:08.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.961+0000 7fbad5a00640 1 --2- 172.21.15.28:0/3400950613 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbad006b250 0x7fbad01ebfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:08.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.961+0000 7fbad5a00640 1 -- 172.21.15.28:0/3400950613 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbad006ae70 msgr2=0x7fbad0157b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:08.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.961+0000 7fbad5a00640 1 --2- 172.21.15.28:0/3400950613 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbad006ae70 0x7fbad0157b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:08.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.961+0000 7fbad5a00640 1 -- 172.21.15.28:0/3400950613 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbad01eca60 con 0x7fbad0072400 2024-02-21T23:08:08.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.961+0000 7fbad5a00640 1 --2- 172.21.15.28:0/3400950613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbad0072400 0x7fbad01ec520 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7fbac8007e40 tx=0x7fbac800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:08.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.961+0000 7fbaccc00640 1 -- 172.21.15.28:0/3400950613 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbac8014620 con 0x7fbad0072400 2024-02-21T23:08:08.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.961+0000 7fbaccc00640 1 -- 172.21.15.28:0/3400950613 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbac8019070 con 0x7fbad0072400 2024-02-21T23:08:08.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.961+0000 7fbaccc00640 1 -- 172.21.15.28:0/3400950613 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbac8014a20 con 0x7fbad0072400 2024-02-21T23:08:08.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.961+0000 7fbad7a00640 1 -- 172.21.15.28:0/3400950613 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbad01eef80 con 0x7fbad0072400 2024-02-21T23:08:08.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.961+0000 7fbad7a00640 1 -- 172.21.15.28:0/3400950613 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbad01eac80 con 0x7fbad0072400 2024-02-21T23:08:08.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.963+0000 7fbad7a00640 1 -- 172.21.15.28:0/3400950613 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbad00c0df0 con 0x7fbad0072400 2024-02-21T23:08:08.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.963+0000 7fbaccc00640 1 -- 172.21.15.28:0/3400950613 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbac8014bc0 con 0x7fbad0072400 2024-02-21T23:08:08.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.965+0000 7fbaccc00640 1 --2- 172.21.15.28:0/3400950613 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbab006b520 0x7fbab006d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:08.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.965+0000 7fbaccc00640 1 -- 172.21.15.28:0/3400950613 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5421+0+0 (secure 0 0 0) 0x7fbac808a810 con 0x7fbad0072400 2024-02-21T23:08:08.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.965+0000 7fbad5000640 1 --2- 172.21.15.28:0/3400950613 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbab006b520 0x7fbab006d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:08.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.966+0000 7fbad5000640 1 --2- 172.21.15.28:0/3400950613 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbab006b520 0x7fbab006d9e0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fbac4036000 tx=0x7fbac403d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:08.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:08.970+0000 7fbaccc00640 1 -- 172.21.15.28:0/3400950613 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fbac80591d0 con 0x7fbad0072400 2024-02-21T23:08:09.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.183+0000 7fbad7a00640 1 -- 172.21.15.28:0/3400950613 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fbad004f370 con 0x7fbad0072400 2024-02-21T23:08:09.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.348+0000 7fbaccc00640 1 -- 172.21.15.28:0/3400950613 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v116) v1 ==== 97+0+0 (secure 0 0 0) 0x7fbac805d270 con 0x7fbad0072400 2024-02-21T23:08:09.337 INFO:teuthology.orchestra.run.smithi028.stderr: removed all blocklist entries 2024-02-21T23:08:09.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.352+0000 7fbad7a00640 1 -- 172.21.15.28:0/3400950613 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbab006b520 msgr2=0x7fbab006d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:09.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.352+0000 7fbad7a00640 1 --2- 172.21.15.28:0/3400950613 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbab006b520 0x7fbab006d9e0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fbac4036000 tx=0x7fbac403d040 comp rx=0 tx=0).stop 2024-02-21T23:08:09.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.352+0000 7fbad7a00640 1 -- 172.21.15.28:0/3400950613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbad0072400 msgr2=0x7fbad01ec520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:09.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.352+0000 7fbad7a00640 1 --2- 172.21.15.28:0/3400950613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbad0072400 0x7fbad01ec520 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7fbac8007e40 tx=0x7fbac800e4b0 comp rx=0 tx=0).stop 2024-02-21T23:08:09.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.352+0000 7fbad7a00640 1 -- 172.21.15.28:0/3400950613 shutdown_connections 2024-02-21T23:08:09.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.352+0000 7fbad7a00640 1 --2- 172.21.15.28:0/3400950613 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbab006b520 0x7fbab006d9e0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:09.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.352+0000 7fbad7a00640 1 --2- 172.21.15.28:0/3400950613 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbad006b250 0x7fbad01ebfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:09.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.352+0000 7fbad7a00640 1 --2- 172.21.15.28:0/3400950613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbad0072400 0x7fbad01ec520 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:09.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.352+0000 7fbad7a00640 1 --2- 172.21.15.28:0/3400950613 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbad006ae70 0x7fbad0157b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:09.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.352+0000 7fbad7a00640 1 -- 172.21.15.28:0/3400950613 >> 172.21.15.28:0/3400950613 conn(0x7fbad00a9690 msgr2=0x7fbad0152eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:09.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.352+0000 7fbad7a00640 1 -- 172.21.15.28:0/3400950613 shutdown_connections 2024-02-21T23:08:09.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.352+0000 7fbad7a00640 1 -- 172.21.15.28:0/3400950613 wait complete. 2024-02-21T23:08:09.364 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-02-21T23:08:09.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.588+0000 7f8ff1200640 1 Processor -- start 2024-02-21T23:08:09.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.588+0000 7f8ff1200640 1 -- start start 2024-02-21T23:08:09.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.589+0000 7f8ff1200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8fec072430 0x7f8fec0c3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:09.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.589+0000 7f8ff1200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fec0c6560 con 0x7f8fec0c3280 2024-02-21T23:08:09.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.589+0000 7f8ff1200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fec0c5cc0 con 0x7f8fec155860 2024-02-21T23:08:09.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.589+0000 7f8ff1200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fec0c5fa0 con 0x7f8fec072430 2024-02-21T23:08:09.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.590+0000 7f8fe9e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f8fec0c3280 0x7f8fec1526e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:46704/0 (socket says 172.21.15.28:46704) 2024-02-21T23:08:09.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.590+0000 7f8fe9e00640 1 -- 172.21.15.28:0/2438350836 learned_addr learned my addr 172.21.15.28:0/2438350836 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:08:09.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.590+0000 7f8fe8a00640 1 -- 172.21.15.28:0/2438350836 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1021817467 0 0) 0x7f8fec0c6560 con 0x7f8fec0c3280 2024-02-21T23:08:09.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.590+0000 7f8fe8a00640 1 -- 172.21.15.28:0/2438350836 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fd0003520 con 0x7f8fec0c3280 2024-02-21T23:08:09.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.590+0000 7f8fe9400640 1 --2- 172.21.15.28:0/2438350836 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8fec072430 0x7f8fec0c3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:09.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.590+0000 7f8fe8a00640 1 -- 172.21.15.28:0/2438350836 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3905983478 0 0) 0x7f8fd0003520 con 0x7f8fec0c3280 2024-02-21T23:08:09.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.590+0000 7f8fe8a00640 1 -- 172.21.15.28:0/2438350836 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8fec072430 msgr2=0x7f8fec0c3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:09.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.590+0000 7f8fe8a00640 1 --2- 172.21.15.28:0/2438350836 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8fec072430 0x7f8fec0c3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:09.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.590+0000 7f8fe8a00640 1 -- 172.21.15.28:0/2438350836 >> v1:172.21.15.81:6789/0 conn(0x7f8fec155860 legacy=0x7f8fec15c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:09.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.590+0000 7f8fe8a00640 1 -- 172.21.15.28:0/2438350836 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fec0c6280 con 0x7f8fec0c3280 2024-02-21T23:08:09.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.591+0000 7f8fe8a00640 1 -- 172.21.15.28:0/2438350836 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f8fdc001670 con 0x7f8fec0c3280 2024-02-21T23:08:09.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.591+0000 7f8fe8a00640 1 -- 172.21.15.28:0/2438350836 >> v1:172.21.15.28:6789/0 conn(0x7f8fec0c3280 legacy=0x7f8fec1526e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:09.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.591+0000 7f8fe8a00640 1 --2- 172.21.15.28:0/2438350836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fd00039e0 0x7f8fd0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:09.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.591+0000 7f8fe8a00640 1 --2- 172.21.15.28:0/2438350836 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8fec072430 0x7f8fd0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:09.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.591+0000 7f8fe8a00640 1 --2- 172.21.15.28:0/2438350836 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8fd0004d40 0x7f8fd00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:09.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.591+0000 7f8fe8a00640 1 -- 172.21.15.28:0/2438350836 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fec0c5fa0 con 0x7f8fd00039e0 2024-02-21T23:08:09.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.591+0000 7f8fe8a00640 1 -- 172.21.15.28:0/2438350836 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd000ddb0 con 0x7f8fd0004d40 2024-02-21T23:08:09.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.591+0000 7f8fe8a00640 1 -- 172.21.15.28:0/2438350836 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd000dfb0 con 0x7f8fec072430 2024-02-21T23:08:09.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.591+0000 7f8fe9e00640 1 --2- 172.21.15.28:0/2438350836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fd00039e0 0x7f8fd0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:09.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.591+0000 7f8fea800640 1 --2- 172.21.15.28:0/2438350836 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8fd0004d40 0x7f8fd00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:09.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.591+0000 7f8fe9400640 1 --2- 172.21.15.28:0/2438350836 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8fec072430 0x7f8fd0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:09.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.591+0000 7f8fe9e00640 1 -- 172.21.15.28:0/2438350836 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8fec072430 msgr2=0x7f8fd0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:09.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.591+0000 7f8fe9e00640 1 --2- 172.21.15.28:0/2438350836 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8fec072430 0x7f8fd0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:09.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.591+0000 7f8fe9e00640 1 -- 172.21.15.28:0/2438350836 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8fd0004d40 msgr2=0x7f8fd00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:09.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.591+0000 7f8fe9e00640 1 --2- 172.21.15.28:0/2438350836 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8fd0004d40 0x7f8fd00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:09.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.591+0000 7f8fe9e00640 1 -- 172.21.15.28:0/2438350836 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fd0003520 con 0x7f8fd00039e0 2024-02-21T23:08:09.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.591+0000 7f8fea800640 1 --2- 172.21.15.28:0/2438350836 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8fd0004d40 0x7f8fd00051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:08:09.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.591+0000 7f8fe9400640 1 --2- 172.21.15.28:0/2438350836 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8fec072430 0x7f8fd0004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:08:09.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.592+0000 7f8fe9e00640 1 --2- 172.21.15.28:0/2438350836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fd00039e0 0x7f8fd0003e00 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc00c240 tx=0x7f8fd0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bcd3a07ce06a58c server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:09.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.593+0000 7f8fe8a00640 1 -- 172.21.15.28:0/2438350836 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fdc007cf0 con 0x7f8fd00039e0 2024-02-21T23:08:09.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.593+0000 7f8fe8a00640 1 -- 172.21.15.28:0/2438350836 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fdc030070 con 0x7f8fd00039e0 2024-02-21T23:08:09.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.593+0000 7f8fe8a00640 1 -- 172.21.15.28:0/2438350836 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fdc005d30 con 0x7f8fd00039e0 2024-02-21T23:08:09.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.593+0000 7f8ff1200640 1 -- 172.21.15.28:0/2438350836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fd00039e0 msgr2=0x7f8fd0003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:09.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.593+0000 7f8ff1200640 1 --2- 172.21.15.28:0/2438350836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fd00039e0 0x7f8fd0003e00 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc00c240 tx=0x7f8fd0005830 comp rx=0 tx=0).stop 2024-02-21T23:08:09.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.593+0000 7f8fea800640 1 -- 172.21.15.28:0/2438350836 reap_dead start 2024-02-21T23:08:09.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.593+0000 7f8ff1200640 1 -- 172.21.15.28:0/2438350836 shutdown_connections 2024-02-21T23:08:09.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.593+0000 7f8ff1200640 1 -- 172.21.15.28:0/2438350836 >> 172.21.15.28:0/2438350836 conn(0x7f8fec0c67f0 msgr2=0x7f8fec0c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:09.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.594+0000 7f8ff1200640 1 -- 172.21.15.28:0/2438350836 shutdown_connections 2024-02-21T23:08:09.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.594+0000 7f8ff1200640 1 -- 172.21.15.28:0/2438350836 wait complete. 2024-02-21T23:08:09.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.595+0000 7f8ff1200640 1 Processor -- start 2024-02-21T23:08:09.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.595+0000 7f8ff1200640 1 -- start start 2024-02-21T23:08:09.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.596+0000 7f8ff1200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8fd00039e0 0x7f8fec0a77d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:09.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.596+0000 7f8ff1200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fec154790 0x7f8fec14da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:09.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.596+0000 7f8ff1200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8fec14dfa0 0x7f8fec0abe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:09.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.596+0000 7f8ff1200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fec0c5aa0 con 0x7f8fec154790 2024-02-21T23:08:09.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.596+0000 7f8ff1200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fec0c5920 con 0x7f8fec14dfa0 2024-02-21T23:08:09.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.596+0000 7f8ff1200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fec0733e0 con 0x7f8fd00039e0 2024-02-21T23:08:09.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.596+0000 7f8fe9400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fec154790 0x7f8fec14da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:09.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.597+0000 7f8fe9400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fec154790 0x7f8fec14da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54540/0 (socket says 172.21.15.28:54540) 2024-02-21T23:08:09.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.597+0000 7f8fe9400640 1 -- 172.21.15.28:0/1131029583 learned_addr learned my addr 172.21.15.28:0/1131029583 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:09.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.597+0000 7f8fe9e00640 1 --2- 172.21.15.28:0/1131029583 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8fd00039e0 0x7f8fec0a77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:09.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.597+0000 7f8fea800640 1 --2- 172.21.15.28:0/1131029583 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8fec14dfa0 0x7f8fec0abe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:09.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.597+0000 7f8fe9400640 1 -- 172.21.15.28:0/1131029583 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8fd00039e0 msgr2=0x7f8fec0a77d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:09.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.597+0000 7f8fe9400640 1 --2- 172.21.15.28:0/1131029583 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8fd00039e0 0x7f8fec0a77d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:09.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.597+0000 7f8fe9400640 1 -- 172.21.15.28:0/1131029583 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8fec14dfa0 msgr2=0x7f8fec0abe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:09.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.597+0000 7f8fe9400640 1 --2- 172.21.15.28:0/1131029583 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8fec14dfa0 0x7f8fec0abe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:09.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.597+0000 7f8fe9400640 1 -- 172.21.15.28:0/1131029583 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fec0a8140 con 0x7f8fec154790 2024-02-21T23:08:09.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.597+0000 7f8fe9400640 1 --2- 172.21.15.28:0/1131029583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fec154790 0x7f8fec14da60 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f8fd400e810 tx=0x7f8fd400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:09.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.598+0000 7f8fe2000640 1 -- 172.21.15.28:0/1131029583 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fd4017070 con 0x7f8fec154790 2024-02-21T23:08:09.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.598+0000 7f8fe2000640 1 -- 172.21.15.28:0/1131029583 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fd400c670 con 0x7f8fec154790 2024-02-21T23:08:09.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.598+0000 7f8fe2000640 1 -- 172.21.15.28:0/1131029583 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fd400f040 con 0x7f8fec154790 2024-02-21T23:08:09.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.598+0000 7f8ff1200640 1 -- 172.21.15.28:0/1131029583 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fec1eeca0 con 0x7f8fec154790 2024-02-21T23:08:09.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.598+0000 7f8ff1200640 1 -- 172.21.15.28:0/1131029583 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fec0aa1d0 con 0x7f8fec154790 2024-02-21T23:08:09.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.599+0000 7f8ff1200640 1 -- 172.21.15.28:0/1131029583 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fec0aa3d0 con 0x7f8fec154790 2024-02-21T23:08:09.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.600+0000 7f8fe2000640 1 -- 172.21.15.28:0/1131029583 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8fd401a070 con 0x7f8fec154790 2024-02-21T23:08:09.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.601+0000 7f8fe2000640 1 --2- 172.21.15.28:0/1131029583 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8fc806b500 0x7f8fc806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:09.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.601+0000 7f8fe2000640 1 -- 172.21.15.28:0/1131029583 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f8fd4089a20 con 0x7f8fec154790 2024-02-21T23:08:09.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.601+0000 7f8fe9e00640 1 --2- 172.21.15.28:0/1131029583 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8fc806b500 0x7f8fc806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:09.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.602+0000 7f8fe9e00640 1 --2- 172.21.15.28:0/1131029583 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8fc806b500 0x7f8fc806d9c0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc034000 tx=0x7f8fdc007000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:09.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.605+0000 7f8fe2000640 1 -- 172.21.15.28:0/1131029583 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f8fec0aa3d0 con 0x7f8fec154790 2024-02-21T23:08:09.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.876+0000 7f8ff1200640 1 -- 172.21.15.28:0/1131029583 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f8fec067f30 con 0x7f8fec154790 2024-02-21T23:08:09.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.877+0000 7f8fe2000640 1 -- 172.21.15.28:0/1131029583 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v15) v1 ==== 80+0+4724 (secure 0 0 0) 0x7f8fd4058a50 con 0x7f8fec154790 2024-02-21T23:08:09.865 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:08:09.865 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-21T23:08:09.865 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_dump": [ 2024-02-21T23:08:09.865 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:08:09.865 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.a", 2024-02-21T23:08:09.865 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDSgNZl4FY1KRAAJSK5Ibb0ibKiKXYpTLgtmQ==", 2024-02-21T23:08:09.865 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:08:09.865 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:08:09.866 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-02-21T23:08:09.866 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-02-21T23:08:09.866 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:08:09.866 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:08:09.866 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:08:09.866 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:08:09.866 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.b", 2024-02-21T23:08:09.866 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDTgNZlwRcAERAAgsYmmkuobKelBEJVCRpXUg==", 2024-02-21T23:08:09.866 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:08:09.866 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:08:09.866 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-02-21T23:08:09.866 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-02-21T23:08:09.866 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:08:09.866 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:08:09.867 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:08:09.867 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:08:09.867 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.c", 2024-02-21T23:08:09.867 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDSgNZloFpQOxAAK2WBArXRVNTgqH9WWDEPDA==", 2024-02-21T23:08:09.867 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:08:09.867 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:08:09.867 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-02-21T23:08:09.867 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-02-21T23:08:09.867 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:08:09.867 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:08:09.867 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:08:09.867 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:08:09.867 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.d", 2024-02-21T23:08:09.867 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDTgNZlblOmIRAAxe2HfSKGSkkzJDN4aVdzgQ==", 2024-02-21T23:08:09.867 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:08:09.868 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:08:09.868 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-02-21T23:08:09.868 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-02-21T23:08:09.868 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:08:09.868 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:08:09.868 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:08:09.868 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:08:09.868 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.0", 2024-02-21T23:08:09.868 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDVgNZlxNJbLxAAXot7Q3chrLEBsFNO8kT6QQ==", 2024-02-21T23:08:09.868 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:08:09.869 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:08:09.869 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:08:09.869 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:08:09.869 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:08:09.869 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:08:09.869 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:08:09.869 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.1", 2024-02-21T23:08:09.869 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDYgNZlZZfWORAAr4XkOxLYLmumVbu362N2HQ==", 2024-02-21T23:08:09.869 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:08:09.869 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:08:09.869 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:08:09.869 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:08:09.869 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:08:09.869 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:08:09.870 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:08:09.870 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.2", 2024-02-21T23:08:09.870 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDcgNZl9mcDGBAAIZGOF8IUMenwo386x95mwg==", 2024-02-21T23:08:09.870 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:08:09.870 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:08:09.870 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:08:09.870 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:08:09.870 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:08:09.870 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:08:09.870 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:08:09.870 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.3", 2024-02-21T23:08:09.870 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDfgNZl8ALSMBAAag9cmGH0/4xALWSUmrZowg==", 2024-02-21T23:08:09.870 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:08:09.870 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:08:09.871 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:08:09.871 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:08:09.871 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:08:09.871 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:08:09.871 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:08:09.871 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.4", 2024-02-21T23:08:09.871 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDngNZlBgeKABAAeW99RVfCBvB1lp7w7mQBlQ==", 2024-02-21T23:08:09.871 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:08:09.871 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:08:09.871 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:08:09.871 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:08:09.871 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:08:09.871 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:08:09.871 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:08:09.871 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.5", 2024-02-21T23:08:09.872 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDqgNZljQG3ChAAY/IZEIcvcGNvlFGIWib3gg==", 2024-02-21T23:08:09.872 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:08:09.872 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:08:09.872 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:08:09.872 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:08:09.872 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:08:09.872 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:08:09.872 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:08:09.872 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.6", 2024-02-21T23:08:09.872 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDtgNZlKnakIxAAkW5xlkshzvEHJdmpgPxR+g==", 2024-02-21T23:08:09.872 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:08:09.872 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:08:09.872 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:08:09.873 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:08:09.873 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:08:09.873 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:08:09.873 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:08:09.873 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.7", 2024-02-21T23:08:09.873 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDxgNZlTWxYARAASX1/gmqO+8mQPP8uA4xjiQ==", 2024-02-21T23:08:09.873 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:08:09.873 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:08:09.873 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:08:09.873 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:08:09.873 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:08:09.873 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:08:09.874 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:08:09.874 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.0", 2024-02-21T23:08:09.874 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDTgNZlILvYLxAAoREWxKKT7YcXnhkgk8YvoQ==", 2024-02-21T23:08:09.874 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:08:09.874 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:08:09.874 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow r", 2024-02-21T23:08:09.874 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-02-21T23:08:09.874 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:08:09.874 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:08:09.874 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:08:09.874 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.1", 2024-02-21T23:08:09.874 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDTgNZlm7ImOxAA8iRcXlzFPZPaXQfsKhuO/g==", 2024-02-21T23:08:09.875 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:08:09.875 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:08:09.875 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-02-21T23:08:09.875 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-02-21T23:08:09.875 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-02-21T23:08:09.875 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:08:09.875 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:08:09.875 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:08:09.875 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.admin", 2024-02-21T23:08:09.875 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDRgNZlBYd1GhAAbK5qBCIkf0Ej5nEyPnTxIQ==", 2024-02-21T23:08:09.875 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:08:09.875 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-02-21T23:08:09.875 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-02-21T23:08:09.875 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-02-21T23:08:09.876 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:08:09.876 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:08:09.876 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:08:09.876 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:08:09.876 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.x", 2024-02-21T23:08:09.876 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDSgNZlJYKeHBAAYPskmLRr7Jw44TdKzM/JMg==", 2024-02-21T23:08:09.876 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:08:09.876 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-02-21T23:08:09.876 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-02-21T23:08:09.876 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:08:09.876 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:08:09.876 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:08:09.876 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:08:09.876 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.y", 2024-02-21T23:08:09.877 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDSgNZlSRkOEBAAOhRj9PsZDpjoV6WT4drrZA==", 2024-02-21T23:08:09.877 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:08:09.877 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-02-21T23:08:09.877 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-02-21T23:08:09.877 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:08:09.877 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:08:09.877 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:08:09.877 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-21T23:08:09.877 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-21T23:08:09.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.879+0000 7f8ff1200640 1 -- 172.21.15.28:0/1131029583 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8fc806b500 msgr2=0x7f8fc806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:09.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.879+0000 7f8ff1200640 1 --2- 172.21.15.28:0/1131029583 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8fc806b500 0x7f8fc806d9c0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc034000 tx=0x7f8fdc007000 comp rx=0 tx=0).stop 2024-02-21T23:08:09.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.879+0000 7f8ff1200640 1 -- 172.21.15.28:0/1131029583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fec154790 msgr2=0x7f8fec14da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:09.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.879+0000 7f8ff1200640 1 --2- 172.21.15.28:0/1131029583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fec154790 0x7f8fec14da60 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f8fd400e810 tx=0x7f8fd400ecd0 comp rx=0 tx=0).stop 2024-02-21T23:08:09.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.880+0000 7f8ff1200640 1 -- 172.21.15.28:0/1131029583 shutdown_connections 2024-02-21T23:08:09.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.880+0000 7f8ff1200640 1 --2- 172.21.15.28:0/1131029583 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8fc806b500 0x7f8fc806d9c0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:09.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.880+0000 7f8ff1200640 1 --2- 172.21.15.28:0/1131029583 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8fec14dfa0 0x7f8fec0abe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:09.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.880+0000 7f8ff1200640 1 --2- 172.21.15.28:0/1131029583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fec154790 0x7f8fec14da60 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:09.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.880+0000 7f8ff1200640 1 --2- 172.21.15.28:0/1131029583 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8fd00039e0 0x7f8fec0a77d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:09.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.880+0000 7f8ff1200640 1 -- 172.21.15.28:0/1131029583 >> 172.21.15.28:0/1131029583 conn(0x7f8fec0c67f0 msgr2=0x7f8fec155c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:09.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.880+0000 7f8ff1200640 1 -- 172.21.15.28:0/1131029583 shutdown_connections 2024-02-21T23:08:09.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:09.880+0000 7f8ff1200640 1 -- 172.21.15.28:0/1131029583 wait complete. 2024-02-21T23:08:09.887 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:08:10.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.118+0000 7fc488800640 1 Processor -- start 2024-02-21T23:08:10.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.119+0000 7fc488800640 1 -- start start 2024-02-21T23:08:10.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.120+0000 7fc488800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fc480071020 0x7fc480156550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:10.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.120+0000 7fc488800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc480071400 0x7fc4800c7840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:10.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.120+0000 7fc488800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc480156a90 con 0x7fc48006f410 2024-02-21T23:08:10.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.120+0000 7fc488800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4800c5130 con 0x7fc480071400 2024-02-21T23:08:10.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.120+0000 7fc488800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4800c4b70 con 0x7fc480071020 2024-02-21T23:08:10.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.120+0000 7fc486800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc480071400 0x7fc4800c7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:10.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.120+0000 7fc486800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc480071400 0x7fc4800c7840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54554/0 (socket says 172.21.15.28:54554) 2024-02-21T23:08:10.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.120+0000 7fc486800640 1 -- 172.21.15.28:0/3651593122 learned_addr learned my addr 172.21.15.28:0/3651593122 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:10.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.120+0000 7fc485e00640 1 --2- 172.21.15.28:0/3651593122 >> v2:172.21.15.81:3300/0 conn(0x7fc480071020 0x7fc480156550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:10.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.121+0000 7fc486800640 1 -- 172.21.15.28:0/3651593122 >> v2:172.21.15.81:3300/0 conn(0x7fc480071020 msgr2=0x7fc480156550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:10.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.121+0000 7fc486800640 1 --2- 172.21.15.28:0/3651593122 >> v2:172.21.15.81:3300/0 conn(0x7fc480071020 0x7fc480156550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:10.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.121+0000 7fc486800640 1 -- 172.21.15.28:0/3651593122 >> v1:172.21.15.81:6789/0 conn(0x7fc48006f410 legacy=0x7fc480153900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:10.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.121+0000 7fc486800640 1 -- 172.21.15.28:0/3651593122 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4800c4e50 con 0x7fc480071400 2024-02-21T23:08:10.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.122+0000 7fc486800640 1 --2- 172.21.15.28:0/3651593122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc480071400 0x7fc4800c7840 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7fc4800c52e0 tx=0x7fc47c02bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=389ffd793cf52f01 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:10.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.122+0000 7fc484a00640 1 -- 172.21.15.28:0/3651593122 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc47c02c960 con 0x7fc480071400 2024-02-21T23:08:10.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.122+0000 7fc484a00640 1 -- 172.21.15.28:0/3651593122 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc47c02cb00 con 0x7fc480071400 2024-02-21T23:08:10.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.122+0000 7fc484a00640 1 -- 172.21.15.28:0/3651593122 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc47c0343f0 con 0x7fc480071400 2024-02-21T23:08:10.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.123+0000 7fc488800640 1 -- 172.21.15.28:0/3651593122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc480071400 msgr2=0x7fc4800c7840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:10.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.123+0000 7fc488800640 1 --2- 172.21.15.28:0/3651593122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc480071400 0x7fc4800c7840 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7fc4800c52e0 tx=0x7fc47c02bdc0 comp rx=0 tx=0).stop 2024-02-21T23:08:10.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.123+0000 7fc488800640 1 -- 172.21.15.28:0/3651593122 shutdown_connections 2024-02-21T23:08:10.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.123+0000 7fc488800640 1 --2- 172.21.15.28:0/3651593122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc480071400 0x7fc4800c7840 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:10.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.123+0000 7fc488800640 1 --2- 172.21.15.28:0/3651593122 >> v2:172.21.15.81:3300/0 conn(0x7fc480071020 0x7fc480156550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:10.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.123+0000 7fc488800640 1 -- 172.21.15.28:0/3651593122 >> 172.21.15.28:0/3651593122 conn(0x7fc4800c8880 msgr2=0x7fc4800c8c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:10.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.123+0000 7fc488800640 1 -- 172.21.15.28:0/3651593122 shutdown_connections 2024-02-21T23:08:10.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.123+0000 7fc488800640 1 -- 172.21.15.28:0/3651593122 wait complete. 2024-02-21T23:08:10.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.125+0000 7fc488800640 1 Processor -- start 2024-02-21T23:08:10.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.125+0000 7fc488800640 1 -- start start 2024-02-21T23:08:10.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.126+0000 7fc488800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc48006f410 0x7fc48014bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:10.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.126+0000 7fc488800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc480071020 0x7fc480152340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:10.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.126+0000 7fc488800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc480071400 0x7fc4801ee020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:10.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.126+0000 7fc488800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4800c7d80 con 0x7fc48006f410 2024-02-21T23:08:10.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.126+0000 7fc488800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4801533e0 con 0x7fc480071020 2024-02-21T23:08:10.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.126+0000 7fc488800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4800c7f00 con 0x7fc480071400 2024-02-21T23:08:10.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.126+0000 7fc485e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc48006f410 0x7fc48014bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:10.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.126+0000 7fc485e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc48006f410 0x7fc48014bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54558/0 (socket says 172.21.15.28:54558) 2024-02-21T23:08:10.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.126+0000 7fc485e00640 1 -- 172.21.15.28:0/825261045 learned_addr learned my addr 172.21.15.28:0/825261045 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:10.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.126+0000 7fc485400640 1 --2- 172.21.15.28:0/825261045 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc480071020 0x7fc480152340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:10.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.126+0000 7fc486800640 1 --2- 172.21.15.28:0/825261045 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc480071400 0x7fc4801ee020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:10.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.126+0000 7fc485e00640 1 -- 172.21.15.28:0/825261045 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc480071400 msgr2=0x7fc4801ee020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:10.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.126+0000 7fc485e00640 1 --2- 172.21.15.28:0/825261045 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc480071400 0x7fc4801ee020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:10.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.126+0000 7fc485e00640 1 -- 172.21.15.28:0/825261045 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc480071020 msgr2=0x7fc480152340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:10.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.126+0000 7fc485e00640 1 --2- 172.21.15.28:0/825261045 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc480071020 0x7fc480152340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:10.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.126+0000 7fc485e00640 1 -- 172.21.15.28:0/825261045 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc48014c130 con 0x7fc48006f410 2024-02-21T23:08:10.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.126+0000 7fc485400640 1 --2- 172.21.15.28:0/825261045 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc480071020 0x7fc480152340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:08:10.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.127+0000 7fc485e00640 1 --2- 172.21.15.28:0/825261045 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc48006f410 0x7fc48014bb50 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7fc47400d800 tx=0x7fc47400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:10.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.127+0000 7fc47a000640 1 -- 172.21.15.28:0/825261045 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc47400b6c0 con 0x7fc48006f410 2024-02-21T23:08:10.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.127+0000 7fc47a000640 1 -- 172.21.15.28:0/825261045 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc474019070 con 0x7fc48006f410 2024-02-21T23:08:10.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.127+0000 7fc47a000640 1 -- 172.21.15.28:0/825261045 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc47400ba80 con 0x7fc48006f410 2024-02-21T23:08:10.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.127+0000 7fc488800640 1 -- 172.21.15.28:0/825261045 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc48014ab10 con 0x7fc48006f410 2024-02-21T23:08:10.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.127+0000 7fc488800640 1 -- 172.21.15.28:0/825261045 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc48014b0d0 con 0x7fc48006f410 2024-02-21T23:08:10.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.129+0000 7fc488800640 1 -- 172.21.15.28:0/825261045 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc480068640 con 0x7fc48006f410 2024-02-21T23:08:10.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.129+0000 7fc47a000640 1 -- 172.21.15.28:0/825261045 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc47400bc20 con 0x7fc48006f410 2024-02-21T23:08:10.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.130+0000 7fc47a000640 1 --2- 172.21.15.28:0/825261045 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc46406b4d0 0x7fc46406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:10.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.130+0000 7fc47a000640 1 -- 172.21.15.28:0/825261045 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fc47408a500 con 0x7fc48006f410 2024-02-21T23:08:10.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.131+0000 7fc485400640 1 --2- 172.21.15.28:0/825261045 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc46406b4d0 0x7fc46406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:10.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.132+0000 7fc485400640 1 --2- 172.21.15.28:0/825261045 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc46406b4d0 0x7fc46406d990 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fc4801ebbe0 tx=0x7fc470008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:10.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.135+0000 7fc47a000640 1 -- 172.21.15.28:0/825261045 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc474059530 con 0x7fc48006f410 2024-02-21T23:08:10.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.338+0000 7fc488800640 1 -- 172.21.15.28:0/825261045 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc48006c080 con 0x7fc48006f410 2024-02-21T23:08:10.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.340+0000 7fc47a000640 1 -- 172.21.15.28:0/825261045 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v118) v1 ==== 74+0+13114 (secure 0 0 0) 0x7fc474058e30 con 0x7fc48006f410 2024-02-21T23:08:10.329 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:08:10.329 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":118,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:08:09.365517+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:08:10.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.343+0000 7fc488800640 1 -- 172.21.15.28:0/825261045 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc46406b4d0 msgr2=0x7fc46406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:10.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.343+0000 7fc488800640 1 --2- 172.21.15.28:0/825261045 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc46406b4d0 0x7fc46406d990 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fc4801ebbe0 tx=0x7fc470008040 comp rx=0 tx=0).stop 2024-02-21T23:08:10.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.343+0000 7fc488800640 1 -- 172.21.15.28:0/825261045 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc48006f410 msgr2=0x7fc48014bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:10.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.343+0000 7fc488800640 1 --2- 172.21.15.28:0/825261045 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc48006f410 0x7fc48014bb50 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7fc47400d800 tx=0x7fc47400dcc0 comp rx=0 tx=0).stop 2024-02-21T23:08:10.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.344+0000 7fc488800640 1 -- 172.21.15.28:0/825261045 shutdown_connections 2024-02-21T23:08:10.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.344+0000 7fc488800640 1 --2- 172.21.15.28:0/825261045 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc46406b4d0 0x7fc46406d990 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:10.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.344+0000 7fc488800640 1 --2- 172.21.15.28:0/825261045 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc480071400 0x7fc4801ee020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:10.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.344+0000 7fc488800640 1 --2- 172.21.15.28:0/825261045 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc480071020 0x7fc480152340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:10.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.344+0000 7fc488800640 1 --2- 172.21.15.28:0/825261045 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc48006f410 0x7fc48014bb50 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:10.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.344+0000 7fc488800640 1 -- 172.21.15.28:0/825261045 >> 172.21.15.28:0/825261045 conn(0x7fc4800c8880 msgr2=0x7fc4801df390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:10.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.344+0000 7fc488800640 1 -- 172.21.15.28:0/825261045 shutdown_connections 2024-02-21T23:08:10.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.344+0000 7fc488800640 1 -- 172.21.15.28:0/825261045 wait complete. 2024-02-21T23:08:10.351 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-21T23:02:30.728329+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-21T23:08:10.352 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-21T23:08:10.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.578+0000 7f464fe00640 1 Processor -- start 2024-02-21T23:08:10.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.578+0000 7f464fe00640 1 -- start start 2024-02-21T23:08:10.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.579+0000 7f464fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f46500add20 0x7f4650153670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:10.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.579+0000 7f464fe00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f4650071110 0x7f46500c50b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:10.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.579+0000 7f464fe00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f46500c55f0 0x7f46500c69b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:10.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.579+0000 7f464fe00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46500c6520 con 0x7f46500add20 2024-02-21T23:08:10.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.579+0000 7f464fe00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f465006f130 con 0x7f4650071110 2024-02-21T23:08:10.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.579+0000 7f464fe00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f465006f3f0 con 0x7f46500c55f0 2024-02-21T23:08:10.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.580+0000 7f464ea00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f46500add20 0x7f4650153670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:10.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.580+0000 7f464ea00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f46500add20 0x7f4650153670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54566/0 (socket says 172.21.15.28:54566) 2024-02-21T23:08:10.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.580+0000 7f464ea00640 1 -- 172.21.15.28:0/2995139787 learned_addr learned my addr 172.21.15.28:0/2995139787 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:10.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.580+0000 7f464e000640 1 --2- 172.21.15.28:0/2995139787 >> v2:172.21.15.81:3300/0 conn(0x7f4650071110 0x7f46500c50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:10.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.580+0000 7f464ea00640 1 -- 172.21.15.28:0/2995139787 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f46500c55f0 msgr2=0x7f46500c69b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:10.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.580+0000 7f464f400640 1 --2- 172.21.15.28:0/2995139787 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f46500c55f0 0x7f46500c69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:10.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.580+0000 7f464ea00640 1 --2- 172.21.15.28:0/2995139787 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f46500c55f0 0x7f46500c69b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:10.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.580+0000 7f464ea00640 1 -- 172.21.15.28:0/2995139787 >> v2:172.21.15.81:3300/0 conn(0x7f4650071110 msgr2=0x7f46500c50b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:10.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.580+0000 7f464ea00640 1 --2- 172.21.15.28:0/2995139787 >> v2:172.21.15.81:3300/0 conn(0x7f4650071110 0x7f46500c50b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:10.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.580+0000 7f464ea00640 1 -- 172.21.15.28:0/2995139787 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f465006f6d0 con 0x7f46500add20 2024-02-21T23:08:10.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.580+0000 7f464e000640 1 --2- 172.21.15.28:0/2995139787 >> v2:172.21.15.81:3300/0 conn(0x7f4650071110 0x7f46500c50b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:08:10.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.582+0000 7f464ea00640 1 --2- 172.21.15.28:0/2995139787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46500add20 0x7f4650153670 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f4650070e70 tx=0x7f464002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32f2f8f98fa45c2e server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:10.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.582+0000 7f464d600640 1 -- 172.21.15.28:0/2995139787 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f464002e070 con 0x7f46500add20 2024-02-21T23:08:10.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.582+0000 7f464d600640 1 -- 172.21.15.28:0/2995139787 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4640037070 con 0x7f46500add20 2024-02-21T23:08:10.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.582+0000 7f464d600640 1 -- 172.21.15.28:0/2995139787 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4640031a00 con 0x7f46500add20 2024-02-21T23:08:10.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.583+0000 7f464fe00640 1 -- 172.21.15.28:0/2995139787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46500add20 msgr2=0x7f4650153670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:10.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.583+0000 7f464fe00640 1 --2- 172.21.15.28:0/2995139787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46500add20 0x7f4650153670 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f4650070e70 tx=0x7f464002b880 comp rx=0 tx=0).stop 2024-02-21T23:08:10.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.583+0000 7f464fe00640 1 -- 172.21.15.28:0/2995139787 shutdown_connections 2024-02-21T23:08:10.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.583+0000 7f464fe00640 1 --2- 172.21.15.28:0/2995139787 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f46500c55f0 0x7f46500c69b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:10.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.583+0000 7f464fe00640 1 --2- 172.21.15.28:0/2995139787 >> v2:172.21.15.81:3300/0 conn(0x7f4650071110 0x7f46500c50b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:10.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.583+0000 7f464fe00640 1 --2- 172.21.15.28:0/2995139787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46500add20 0x7f4650153670 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:10.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.583+0000 7f464fe00640 1 -- 172.21.15.28:0/2995139787 >> 172.21.15.28:0/2995139787 conn(0x7f46500c8620 msgr2=0x7f46500c8a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:10.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.583+0000 7f464fe00640 1 -- 172.21.15.28:0/2995139787 shutdown_connections 2024-02-21T23:08:10.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.583+0000 7f464fe00640 1 -- 172.21.15.28:0/2995139787 wait complete. 2024-02-21T23:08:10.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.585+0000 7f464fe00640 1 Processor -- start 2024-02-21T23:08:10.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.585+0000 7f464fe00640 1 -- start start 2024-02-21T23:08:10.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.586+0000 7f464fe00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4650071110 0x7f46501edbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:10.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.586+0000 7f464fe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46500add20 0x7f46501eb890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:10.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.586+0000 7f464fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46500c55f0 0x7f46501ebdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:10.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.587+0000 7f464fe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46500ae100 con 0x7f46500c55f0 2024-02-21T23:08:10.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.587+0000 7f464fe00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46500c2c30 con 0x7f46500add20 2024-02-21T23:08:10.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.587+0000 7f464fe00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46500ae280 con 0x7f4650071110 2024-02-21T23:08:10.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.587+0000 7f464f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46500c55f0 0x7f46501ebdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:10.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.587+0000 7f464f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46500c55f0 0x7f46501ebdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54578/0 (socket says 172.21.15.28:54578) 2024-02-21T23:08:10.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.587+0000 7f464f400640 1 -- 172.21.15.28:0/2411244765 learned_addr learned my addr 172.21.15.28:0/2411244765 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:10.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.587+0000 7f464e000640 1 --2- 172.21.15.28:0/2411244765 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46500add20 0x7f46501eb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:10.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.587+0000 7f464ea00640 1 --2- 172.21.15.28:0/2411244765 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4650071110 0x7f46501edbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:10.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.588+0000 7f464f400640 1 -- 172.21.15.28:0/2411244765 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4650071110 msgr2=0x7f46501edbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:10.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.588+0000 7f464f400640 1 --2- 172.21.15.28:0/2411244765 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4650071110 0x7f46501edbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:10.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.588+0000 7f464f400640 1 -- 172.21.15.28:0/2411244765 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46500add20 msgr2=0x7f46501eb890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:10.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.588+0000 7f464f400640 1 --2- 172.21.15.28:0/2411244765 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46500add20 0x7f46501eb890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:10.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.588+0000 7f464f400640 1 -- 172.21.15.28:0/2411244765 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f465014a660 con 0x7f46500c55f0 2024-02-21T23:08:10.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.588+0000 7f464f400640 1 --2- 172.21.15.28:0/2411244765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46500c55f0 0x7f46501ebdd0 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f464800cc30 tx=0x7f464800a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:10.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.588+0000 7f4646a00640 1 -- 172.21.15.28:0/2411244765 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f464800ce40 con 0x7f46500c55f0 2024-02-21T23:08:10.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.588+0000 7f4646a00640 1 -- 172.21.15.28:0/2411244765 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4648003990 con 0x7f46500c55f0 2024-02-21T23:08:10.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.588+0000 7f4646a00640 1 -- 172.21.15.28:0/2411244765 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4648003d40 con 0x7f46500c55f0 2024-02-21T23:08:10.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.588+0000 7f464fe00640 1 -- 172.21.15.28:0/2411244765 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46501ee680 con 0x7f46500c55f0 2024-02-21T23:08:10.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.588+0000 7f464fe00640 1 -- 172.21.15.28:0/2411244765 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f465014bc70 con 0x7f46500c55f0 2024-02-21T23:08:10.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.590+0000 7f464fe00640 1 -- 172.21.15.28:0/2411244765 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f465004f290 con 0x7f46500c55f0 2024-02-21T23:08:10.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.591+0000 7f4646a00640 1 -- 172.21.15.28:0/2411244765 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4648003ee0 con 0x7f46500c55f0 2024-02-21T23:08:10.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.592+0000 7f4646a00640 1 --2- 172.21.15.28:0/2411244765 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f462c06b4d0 0x7f462c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:10.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.592+0000 7f4646a00640 1 -- 172.21.15.28:0/2411244765 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f464808a4b0 con 0x7f46500c55f0 2024-02-21T23:08:10.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.593+0000 7f464ea00640 1 --2- 172.21.15.28:0/2411244765 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f462c06b4d0 0x7f462c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:10.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.594+0000 7f464ea00640 1 --2- 172.21.15.28:0/2411244765 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f462c06b4d0 0x7f462c06d990 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f4640036000 tx=0x7f464003d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:10.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.597+0000 7f4646a00640 1 -- 172.21.15.28:0/2411244765 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4648014030 con 0x7f46500c55f0 2024-02-21T23:08:10.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.796+0000 7f464fe00640 1 -- 172.21.15.28:0/2411244765 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f46500c80b0 con 0x7f46500c55f0 2024-02-21T23:08:10.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.797+0000 7f4646a00640 1 -- 172.21.15.28:0/2411244765 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v118) v1 ==== 93+0+10 (secure 0 0 0) 0x7f46480594e0 con 0x7f46500c55f0 2024-02-21T23:08:10.785 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-02-21T23:08:10.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.800+0000 7f464fe00640 1 -- 172.21.15.28:0/2411244765 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f462c06b4d0 msgr2=0x7f462c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:10.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.800+0000 7f464fe00640 1 --2- 172.21.15.28:0/2411244765 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f462c06b4d0 0x7f462c06d990 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f4640036000 tx=0x7f464003d040 comp rx=0 tx=0).stop 2024-02-21T23:08:10.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.800+0000 7f464fe00640 1 -- 172.21.15.28:0/2411244765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46500c55f0 msgr2=0x7f46501ebdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:10.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.800+0000 7f464fe00640 1 --2- 172.21.15.28:0/2411244765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46500c55f0 0x7f46501ebdd0 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f464800cc30 tx=0x7f464800a5c0 comp rx=0 tx=0).stop 2024-02-21T23:08:10.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.800+0000 7f464fe00640 1 -- 172.21.15.28:0/2411244765 shutdown_connections 2024-02-21T23:08:10.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.800+0000 7f464fe00640 1 --2- 172.21.15.28:0/2411244765 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f462c06b4d0 0x7f462c06d990 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:10.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.800+0000 7f464fe00640 1 --2- 172.21.15.28:0/2411244765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46500c55f0 0x7f46501ebdd0 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:10.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.800+0000 7f464fe00640 1 --2- 172.21.15.28:0/2411244765 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46500add20 0x7f46501eb890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:10.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.800+0000 7f464fe00640 1 --2- 172.21.15.28:0/2411244765 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4650071110 0x7f46501edbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:10.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.800+0000 7f464fe00640 1 -- 172.21.15.28:0/2411244765 >> 172.21.15.28:0/2411244765 conn(0x7f46500c8620 msgr2=0x7f46500ad020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:10.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.800+0000 7f464fe00640 1 -- 172.21.15.28:0/2411244765 shutdown_connections 2024-02-21T23:08:10.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:10.800+0000 7f464fe00640 1 -- 172.21.15.28:0/2411244765 wait complete. 2024-02-21T23:08:10.808 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-02-21T23:08:11.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.048+0000 7f4df6600640 1 Processor -- start 2024-02-21T23:08:11.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.049+0000 7f4df6600640 1 -- start start 2024-02-21T23:08:11.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.050+0000 7f4df6600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4df00b0130 0x7f4df00c8740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:11.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.050+0000 7f4df6600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4df01550b0 con 0x7f4df00c51c0 2024-02-21T23:08:11.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.050+0000 7f4df6600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4df0155350 con 0x7f4df0158c20 2024-02-21T23:08:11.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.050+0000 7f4df6600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4df0152a40 con 0x7f4df00b0130 2024-02-21T23:08:11.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.050+0000 7f4defe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f4df00c51c0 0x7f4df0158510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:46718/0 (socket says 172.21.15.28:46718) 2024-02-21T23:08:11.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.050+0000 7f4defe00640 1 -- 172.21.15.28:0/1941529778 learned_addr learned my addr 172.21.15.28:0/1941529778 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:08:11.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.051+0000 7f4df5200640 1 --2- 172.21.15.28:0/1941529778 >> v2:172.21.15.28:3300/0 conn(0x7f4df00b0130 0x7f4df00c8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:11.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.051+0000 7f4def400640 1 -- 172.21.15.28:0/1941529778 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3860392708 0 0) 0x7f4df01550b0 con 0x7f4df00c51c0 2024-02-21T23:08:11.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.051+0000 7f4def400640 1 -- 172.21.15.28:0/1941529778 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4dd4003520 con 0x7f4df00c51c0 2024-02-21T23:08:11.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.051+0000 7f4df5200640 1 -- 172.21.15.28:0/1941529778 >> v1:172.21.15.81:6789/0 conn(0x7f4df0158c20 legacy=0x7f4df015f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:11.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.051+0000 7f4df5200640 1 -- 172.21.15.28:0/1941529778 >> v1:172.21.15.28:6789/0 conn(0x7f4df00c51c0 legacy=0x7f4df0158510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:11.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.051+0000 7f4df5200640 1 -- 172.21.15.28:0/1941529778 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4df0155630 con 0x7f4df00b0130 2024-02-21T23:08:11.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.052+0000 7f4df5200640 1 --2- 172.21.15.28:0/1941529778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df00b0130 0x7f4df00c8740 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f4dd80097a0 tx=0x7f4dd802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b74258b5ec99aa3 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:11.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.053+0000 7f4def400640 1 -- 172.21.15.28:0/1941529778 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dd80315a0 con 0x7f4df00b0130 2024-02-21T23:08:11.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.053+0000 7f4def400640 1 -- 172.21.15.28:0/1941529778 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dd8037070 con 0x7f4df00b0130 2024-02-21T23:08:11.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.053+0000 7f4def400640 1 -- 172.21.15.28:0/1941529778 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dd8031940 con 0x7f4df00b0130 2024-02-21T23:08:11.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.053+0000 7f4df6600640 1 -- 172.21.15.28:0/1941529778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df00b0130 msgr2=0x7f4df00c8740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:11.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.053+0000 7f4df6600640 1 --2- 172.21.15.28:0/1941529778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df00b0130 0x7f4df00c8740 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f4dd80097a0 tx=0x7f4dd802b840 comp rx=0 tx=0).stop 2024-02-21T23:08:11.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.053+0000 7f4df6600640 1 -- 172.21.15.28:0/1941529778 shutdown_connections 2024-02-21T23:08:11.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.053+0000 7f4df6600640 1 --2- 172.21.15.28:0/1941529778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df00b0130 0x7f4df00c8740 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:11.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.053+0000 7f4df6600640 1 -- 172.21.15.28:0/1941529778 >> 172.21.15.28:0/1941529778 conn(0x7f4df00ac370 msgr2=0x7f4df00ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:11.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.053+0000 7f4df6600640 1 -- 172.21.15.28:0/1941529778 shutdown_connections 2024-02-21T23:08:11.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.053+0000 7f4df6600640 1 -- 172.21.15.28:0/1941529778 wait complete. 2024-02-21T23:08:11.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.055+0000 7f4df6600640 1 Processor -- start 2024-02-21T23:08:11.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.055+0000 7f4df6600640 1 -- start start 2024-02-21T23:08:11.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.056+0000 7f4df6600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df00b0130 0x7f4df00bedd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:11.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.056+0000 7f4df6600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4df00c51c0 0x7f4df00c0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:11.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.056+0000 7f4df6600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4df0158c20 0x7f4df0151fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:11.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.056+0000 7f4df6600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4df00c8dd0 con 0x7f4df00b0130 2024-02-21T23:08:11.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.056+0000 7f4df6600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4df0155e30 con 0x7f4df00c51c0 2024-02-21T23:08:11.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.056+0000 7f4df6600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4df00be060 con 0x7f4df0158c20 2024-02-21T23:08:11.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.057+0000 7f4df5200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df00b0130 0x7f4df00bedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:11.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.057+0000 7f4df5200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df00b0130 0x7f4df00bedd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54598/0 (socket says 172.21.15.28:54598) 2024-02-21T23:08:11.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.057+0000 7f4df5200640 1 -- 172.21.15.28:0/1409808917 learned_addr learned my addr 172.21.15.28:0/1409808917 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:11.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.057+0000 7f4defe00640 1 --2- 172.21.15.28:0/1409808917 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4df00c51c0 0x7f4df00c0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:11.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.057+0000 7f4df5200640 1 -- 172.21.15.28:0/1409808917 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4df0158c20 msgr2=0x7f4df0151fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:11.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.057+0000 7f4df5200640 1 --2- 172.21.15.28:0/1409808917 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4df0158c20 0x7f4df0151fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:11.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.057+0000 7f4df5200640 1 -- 172.21.15.28:0/1409808917 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4df00c51c0 msgr2=0x7f4df00c0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:11.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.057+0000 7f4df5200640 1 --2- 172.21.15.28:0/1409808917 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4df00c51c0 0x7f4df00c0eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:11.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.057+0000 7f4df5200640 1 -- 172.21.15.28:0/1409808917 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4df0152510 con 0x7f4df00b0130 2024-02-21T23:08:11.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.057+0000 7f4df5200640 1 --2- 172.21.15.28:0/1409808917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df00b0130 0x7f4df00bedd0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f4dd8000c00 tx=0x7f4dd80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:11.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.058+0000 7f4decc00640 1 -- 172.21.15.28:0/1409808917 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dd8032630 con 0x7f4df00b0130 2024-02-21T23:08:11.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.058+0000 7f4decc00640 1 -- 172.21.15.28:0/1409808917 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dd80327f0 con 0x7f4df00b0130 2024-02-21T23:08:11.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.058+0000 7f4decc00640 1 -- 172.21.15.28:0/1409808917 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dd8032be0 con 0x7f4df00b0130 2024-02-21T23:08:11.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.058+0000 7f4df6600640 1 -- 172.21.15.28:0/1409808917 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4df00c1520 con 0x7f4df00b0130 2024-02-21T23:08:11.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.058+0000 7f4df6600640 1 -- 172.21.15.28:0/1409808917 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4df00be7d0 con 0x7f4df00b0130 2024-02-21T23:08:11.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.059+0000 7f4df6600640 1 -- 172.21.15.28:0/1409808917 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4df004f290 con 0x7f4df00b0130 2024-02-21T23:08:11.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.060+0000 7f4decc00640 1 -- 172.21.15.28:0/1409808917 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4dd803b3f0 con 0x7f4df00b0130 2024-02-21T23:08:11.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.061+0000 7f4decc00640 1 --2- 172.21.15.28:0/1409808917 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4dcc06b4d0 0x7f4dcc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:11.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.061+0000 7f4decc00640 1 -- 172.21.15.28:0/1409808917 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f4dd802f030 con 0x7f4df00b0130 2024-02-21T23:08:11.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.061+0000 7f4defe00640 1 --2- 172.21.15.28:0/1409808917 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4dcc06b4d0 0x7f4dcc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:11.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.063+0000 7f4defe00640 1 --2- 172.21.15.28:0/1409808917 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4dcc06b4d0 0x7f4dcc06d990 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f4df00c3080 tx=0x7f4de0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:11.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.066+0000 7f4decc00640 1 -- 172.21.15.28:0/1409808917 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4df0152510 con 0x7f4df00b0130 2024-02-21T23:08:11.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.268+0000 7f4df6600640 1 -- 172.21.15.28:0/1409808917 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f4df006d9b0 con 0x7f4df00b0130 2024-02-21T23:08:11.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.268+0000 7f4decc00640 1 -- 172.21.15.28:0/1409808917 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v118) v1 ==== 84+0+65 (secure 0 0 0) 0x7f4dd8040070 con 0x7f4df00b0130 2024-02-21T23:08:11.256 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:08:11.256 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-02-21T23:08:11.257 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:08:11.257 INFO:teuthology.orchestra.run.smithi028.stdout: "poolnum": 1, 2024-02-21T23:08:11.257 INFO:teuthology.orchestra.run.smithi028.stdout: "poolname": ".mgr" 2024-02-21T23:08:11.257 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:08:11.257 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-02-21T23:08:11.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.270+0000 7f4df6600640 1 -- 172.21.15.28:0/1409808917 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4dcc06b4d0 msgr2=0x7f4dcc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:11.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.270+0000 7f4df6600640 1 --2- 172.21.15.28:0/1409808917 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4dcc06b4d0 0x7f4dcc06d990 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f4df00c3080 tx=0x7f4de0009040 comp rx=0 tx=0).stop 2024-02-21T23:08:11.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.270+0000 7f4df6600640 1 -- 172.21.15.28:0/1409808917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df00b0130 msgr2=0x7f4df00bedd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:11.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.270+0000 7f4df6600640 1 --2- 172.21.15.28:0/1409808917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df00b0130 0x7f4df00bedd0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f4dd8000c00 tx=0x7f4dd80046c0 comp rx=0 tx=0).stop 2024-02-21T23:08:11.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.271+0000 7f4df6600640 1 -- 172.21.15.28:0/1409808917 shutdown_connections 2024-02-21T23:08:11.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.271+0000 7f4df6600640 1 --2- 172.21.15.28:0/1409808917 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4dcc06b4d0 0x7f4dcc06d990 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:11.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.271+0000 7f4df6600640 1 --2- 172.21.15.28:0/1409808917 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4df0158c20 0x7f4df0151fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:11.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.271+0000 7f4df6600640 1 --2- 172.21.15.28:0/1409808917 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4df00c51c0 0x7f4df00c0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:11.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.271+0000 7f4df6600640 1 --2- 172.21.15.28:0/1409808917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df00b0130 0x7f4df00bedd0 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:11.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.271+0000 7f4df6600640 1 -- 172.21.15.28:0/1409808917 >> 172.21.15.28:0/1409808917 conn(0x7f4df00ac370 msgr2=0x7f4df00aad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:11.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.271+0000 7f4df6600640 1 -- 172.21.15.28:0/1409808917 shutdown_connections 2024-02-21T23:08:11.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.271+0000 7f4df6600640 1 -- 172.21.15.28:0/1409808917 wait complete. 2024-02-21T23:08:11.278 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-02-21T23:08:11.278 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-02-21T23:08:11.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.521+0000 7f293fe00640 1 Processor -- start 2024-02-21T23:08:11.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.521+0000 7f293fe00640 1 -- start start 2024-02-21T23:08:11.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.522+0000 7f293fe00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f29380c5a70 0x7f2938072310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:11.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.522+0000 7f293fe00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f2938156210 0x7f29380c29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:11.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.522+0000 7f293fe00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29380c2490 con 0x7f2938072850 2024-02-21T23:08:11.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.522+0000 7f293fe00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f293806f200 con 0x7f2938156210 2024-02-21T23:08:11.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.522+0000 7f293fe00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29380c1910 con 0x7f29380c5a70 2024-02-21T23:08:11.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.523+0000 7f293d400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f29380c5a70 0x7f2938072310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:11.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.523+0000 7f293d400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f29380c5a70 0x7f2938072310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3301/0 says I am v2:172.21.15.28:46566/0 (socket says 172.21.15.28:46566) 2024-02-21T23:08:11.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.523+0000 7f293d400640 1 -- 172.21.15.28:0/1622135189 learned_addr learned my addr 172.21.15.28:0/1622135189 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:11.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.523+0000 7f293de00640 1 --2- 172.21.15.28:0/1622135189 >> v2:172.21.15.81:3300/0 conn(0x7f2938156210 0x7f29380c29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:11.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.524+0000 7f293d400640 1 -- 172.21.15.28:0/1622135189 >> v2:172.21.15.81:3300/0 conn(0x7f2938156210 msgr2=0x7f29380c29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:11.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.524+0000 7f293d400640 1 --2- 172.21.15.28:0/1622135189 >> v2:172.21.15.81:3300/0 conn(0x7f2938156210 0x7f29380c29d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:11.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.524+0000 7f293d400640 1 -- 172.21.15.28:0/1622135189 >> v1:172.21.15.81:6789/0 conn(0x7f2938072850 legacy=0x7f2938155b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:11.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.524+0000 7f293d400640 1 -- 172.21.15.28:0/1622135189 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29380c1bf0 con 0x7f29380c5a70 2024-02-21T23:08:11.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.524+0000 7f2937e00640 1 -- 172.21.15.28:0/1622135189 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1279778830 0 0) 0x7f29380c2490 con 0x7f2938072850 2024-02-21T23:08:11.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.525+0000 7f293d400640 1 --2- 172.21.15.28:0/1622135189 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f29380c5a70 0x7f2938072310 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f292c009e50 tx=0x7f292c02be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1be11b1f3ff353db server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:11.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.526+0000 7f2937e00640 1 -- 172.21.15.28:0/1622135189 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f292c02c9e0 con 0x7f29380c5a70 2024-02-21T23:08:11.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.526+0000 7f2937e00640 1 -- 172.21.15.28:0/1622135189 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f292c02cb80 con 0x7f29380c5a70 2024-02-21T23:08:11.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.526+0000 7f2937e00640 1 -- 172.21.15.28:0/1622135189 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f292c0343f0 con 0x7f29380c5a70 2024-02-21T23:08:11.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.526+0000 7f293fe00640 1 -- 172.21.15.28:0/1622135189 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f29380c5a70 msgr2=0x7f2938072310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:11.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.526+0000 7f293fe00640 1 --2- 172.21.15.28:0/1622135189 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f29380c5a70 0x7f2938072310 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f292c009e50 tx=0x7f292c02be40 comp rx=0 tx=0).stop 2024-02-21T23:08:11.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.527+0000 7f293fe00640 1 -- 172.21.15.28:0/1622135189 shutdown_connections 2024-02-21T23:08:11.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.527+0000 7f293fe00640 1 --2- 172.21.15.28:0/1622135189 >> v2:172.21.15.81:3300/0 conn(0x7f2938156210 0x7f29380c29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:11.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.527+0000 7f293fe00640 1 --2- 172.21.15.28:0/1622135189 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f29380c5a70 0x7f2938072310 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:11.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.527+0000 7f293fe00640 1 -- 172.21.15.28:0/1622135189 >> 172.21.15.28:0/1622135189 conn(0x7f293806b830 msgr2=0x7f29380c67f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:11.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.527+0000 7f293fe00640 1 -- 172.21.15.28:0/1622135189 shutdown_connections 2024-02-21T23:08:11.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.527+0000 7f293fe00640 1 -- 172.21.15.28:0/1622135189 wait complete. 2024-02-21T23:08:11.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.528+0000 7f293fe00640 1 Processor -- start 2024-02-21T23:08:11.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.529+0000 7f293fe00640 1 -- start start 2024-02-21T23:08:11.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.529+0000 7f293fe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2938072850 0x7f293815f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:11.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.529+0000 7f293fe00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f29380c5a70 0x7f2938158c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:11.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.530+0000 7f293fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2938156210 0x7f29381427b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:11.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.530+0000 7f293fe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29380c5e50 con 0x7f2938156210 2024-02-21T23:08:11.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.530+0000 7f293fe00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29380c21b0 con 0x7f2938072850 2024-02-21T23:08:11.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.530+0000 7f293fe00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29380c5fd0 con 0x7f29380c5a70 2024-02-21T23:08:11.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.530+0000 7f293de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2938156210 0x7f29381427b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:11.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.530+0000 7f293de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2938156210 0x7f29381427b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54600/0 (socket says 172.21.15.28:54600) 2024-02-21T23:08:11.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.530+0000 7f293de00640 1 -- 172.21.15.28:0/3328109197 learned_addr learned my addr 172.21.15.28:0/3328109197 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:11.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.530+0000 7f293d400640 1 --2- 172.21.15.28:0/3328109197 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2938072850 0x7f293815f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:11.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.530+0000 7f293ca00640 1 --2- 172.21.15.28:0/3328109197 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f29380c5a70 0x7f2938158c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:11.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.531+0000 7f293de00640 1 -- 172.21.15.28:0/3328109197 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f29380c5a70 msgr2=0x7f2938158c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:11.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.531+0000 7f293de00640 1 --2- 172.21.15.28:0/3328109197 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f29380c5a70 0x7f2938158c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:11.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.531+0000 7f293de00640 1 -- 172.21.15.28:0/3328109197 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2938072850 msgr2=0x7f293815f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:11.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.531+0000 7f293de00640 1 --2- 172.21.15.28:0/3328109197 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2938072850 0x7f293815f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:11.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.531+0000 7f293de00640 1 -- 172.21.15.28:0/3328109197 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29381568d0 con 0x7f2938156210 2024-02-21T23:08:11.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.531+0000 7f293d400640 1 --2- 172.21.15.28:0/3328109197 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2938072850 0x7f293815f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:08:11.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.531+0000 7f293de00640 1 --2- 172.21.15.28:0/3328109197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2938156210 0x7f29381427b0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f293000e930 tx=0x7f293000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:11.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.531+0000 7f2935600640 1 -- 172.21.15.28:0/3328109197 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2930017070 con 0x7f2938156210 2024-02-21T23:08:11.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.531+0000 7f2935600640 1 -- 172.21.15.28:0/3328109197 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f293000c980 con 0x7f2938156210 2024-02-21T23:08:11.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.531+0000 7f293fe00640 1 -- 172.21.15.28:0/3328109197 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29381ebac0 con 0x7f2938156210 2024-02-21T23:08:11.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.531+0000 7f2935600640 1 -- 172.21.15.28:0/3328109197 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f293000cd70 con 0x7f2938156210 2024-02-21T23:08:11.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.531+0000 7f293fe00640 1 -- 172.21.15.28:0/3328109197 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29381576b0 con 0x7f2938156210 2024-02-21T23:08:11.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.533+0000 7f293fe00640 1 -- 172.21.15.28:0/3328109197 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2938156db0 con 0x7f2938156210 2024-02-21T23:08:11.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.534+0000 7f2935600640 1 -- 172.21.15.28:0/3328109197 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f293001a070 con 0x7f2938156210 2024-02-21T23:08:11.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.535+0000 7f2935600640 1 --2- 172.21.15.28:0/3328109197 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f291806b4d0 0x7f291806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:11.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.535+0000 7f2935600640 1 -- 172.21.15.28:0/3328109197 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f2930089f40 con 0x7f2938156210 2024-02-21T23:08:11.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.535+0000 7f293d400640 1 --2- 172.21.15.28:0/3328109197 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f291806b4d0 0x7f291806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:11.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.537+0000 7f293d400640 1 --2- 172.21.15.28:0/3328109197 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f291806b4d0 0x7f291806d990 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f292c035000 tx=0x7f292c030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:11.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.539+0000 7f2935600640 1 -- 172.21.15.28:0/3328109197 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f2938156db0 con 0x7f2938156210 2024-02-21T23:08:11.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:11.739+0000 7f293fe00640 1 -- 172.21.15.28:0/3328109197 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f2938067f30 con 0x7f2938156210 2024-02-21T23:08:12.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.383+0000 7f2935600640 1 -- 172.21.15.28:0/3328109197 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v119) v1 ==== 138+0+0 (secure 0 0 0) 0x7f29300589f0 con 0x7f2938156210 2024-02-21T23:08:12.372 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' created 2024-02-21T23:08:12.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.387+0000 7f293fe00640 1 -- 172.21.15.28:0/3328109197 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f291806b4d0 msgr2=0x7f291806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:12.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.387+0000 7f293fe00640 1 --2- 172.21.15.28:0/3328109197 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f291806b4d0 0x7f291806d990 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f292c035000 tx=0x7f292c030000 comp rx=0 tx=0).stop 2024-02-21T23:08:12.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.387+0000 7f293fe00640 1 -- 172.21.15.28:0/3328109197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2938156210 msgr2=0x7f29381427b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:12.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.387+0000 7f293fe00640 1 --2- 172.21.15.28:0/3328109197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2938156210 0x7f29381427b0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f293000e930 tx=0x7f293000edf0 comp rx=0 tx=0).stop 2024-02-21T23:08:12.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.387+0000 7f293fe00640 1 -- 172.21.15.28:0/3328109197 shutdown_connections 2024-02-21T23:08:12.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.387+0000 7f293fe00640 1 --2- 172.21.15.28:0/3328109197 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f291806b4d0 0x7f291806d990 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:12.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.387+0000 7f293fe00640 1 --2- 172.21.15.28:0/3328109197 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f29380c5a70 0x7f2938158c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:12.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.387+0000 7f293fe00640 1 --2- 172.21.15.28:0/3328109197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2938156210 0x7f29381427b0 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:12.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.387+0000 7f293fe00640 1 --2- 172.21.15.28:0/3328109197 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2938072850 0x7f293815f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:12.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.387+0000 7f293fe00640 1 -- 172.21.15.28:0/3328109197 >> 172.21.15.28:0/3328109197 conn(0x7f293806b830 msgr2=0x7f293806d980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:12.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.388+0000 7f293fe00640 1 -- 172.21.15.28:0/3328109197 shutdown_connections 2024-02-21T23:08:12.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.388+0000 7f293fe00640 1 -- 172.21.15.28:0/3328109197 wait complete. 2024-02-21T23:08:12.400 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-02-21T23:08:12.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.632+0000 7fb71f200640 1 Processor -- start 2024-02-21T23:08:12.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.633+0000 7fb71f200640 1 -- start start 2024-02-21T23:08:12.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.634+0000 7fb71f200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb720152370 0x7fb720155940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:12.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.634+0000 7fb71f200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fb7200bde20 0x7fb7200c6f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:12.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.634+0000 7fb71f200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb7200c74a0 0x7fb7200cab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:12.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.634+0000 7fb71f200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7201554b0 con 0x7fb7200c74a0 2024-02-21T23:08:12.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.634+0000 7fb71f200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb720154ea0 con 0x7fb7200bde20 2024-02-21T23:08:12.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.634+0000 7fb71f200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7200c5690 con 0x7fb720152370 2024-02-21T23:08:12.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.635+0000 7fb71e800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb7200c74a0 0x7fb7200cab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:12.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.635+0000 7fb71e800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb7200c74a0 0x7fb7200cab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54608/0 (socket says 172.21.15.28:54608) 2024-02-21T23:08:12.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.635+0000 7fb71e800640 1 -- 172.21.15.28:0/3688252158 learned_addr learned my addr 172.21.15.28:0/3688252158 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:12.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.635+0000 7fb71de00640 1 --2- 172.21.15.28:0/3688252158 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb720152370 0x7fb720155940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:12.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.635+0000 7fb71d400640 1 --2- 172.21.15.28:0/3688252158 >> v2:172.21.15.81:3300/0 conn(0x7fb7200bde20 0x7fb7200c6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:12.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.635+0000 7fb71e800640 1 -- 172.21.15.28:0/3688252158 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb720152370 msgr2=0x7fb720155940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:12.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.635+0000 7fb71e800640 1 --2- 172.21.15.28:0/3688252158 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb720152370 0x7fb720155940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:12.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.635+0000 7fb71e800640 1 -- 172.21.15.28:0/3688252158 >> v2:172.21.15.81:3300/0 conn(0x7fb7200bde20 msgr2=0x7fb7200c6f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:12.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.635+0000 7fb71e800640 1 --2- 172.21.15.28:0/3688252158 >> v2:172.21.15.81:3300/0 conn(0x7fb7200bde20 0x7fb7200c6f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:12.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.635+0000 7fb71e800640 1 -- 172.21.15.28:0/3688252158 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb720155140 con 0x7fb7200c74a0 2024-02-21T23:08:12.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.637+0000 7fb71e800640 1 --2- 172.21.15.28:0/3688252158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7200c74a0 0x7fb7200cab00 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7fb720067580 tx=0x7fb71802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85ac6d4adfc70bb1 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:12.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.637+0000 7fb71ca00640 1 -- 172.21.15.28:0/3688252158 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb71802ca70 con 0x7fb7200c74a0 2024-02-21T23:08:12.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.638+0000 7fb71ca00640 1 -- 172.21.15.28:0/3688252158 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb71802cc10 con 0x7fb7200c74a0 2024-02-21T23:08:12.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.638+0000 7fb71ca00640 1 -- 172.21.15.28:0/3688252158 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7180343f0 con 0x7fb7200c74a0 2024-02-21T23:08:12.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.638+0000 7fb71f200640 1 -- 172.21.15.28:0/3688252158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7200c74a0 msgr2=0x7fb7200cab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:12.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.638+0000 7fb71f200640 1 --2- 172.21.15.28:0/3688252158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7200c74a0 0x7fb7200cab00 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7fb720067580 tx=0x7fb71802bed0 comp rx=0 tx=0).stop 2024-02-21T23:08:12.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.638+0000 7fb71f200640 1 -- 172.21.15.28:0/3688252158 shutdown_connections 2024-02-21T23:08:12.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.638+0000 7fb71f200640 1 --2- 172.21.15.28:0/3688252158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7200c74a0 0x7fb7200cab00 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:12.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.638+0000 7fb71f200640 1 --2- 172.21.15.28:0/3688252158 >> v2:172.21.15.81:3300/0 conn(0x7fb7200bde20 0x7fb7200c6f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:12.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.638+0000 7fb71f200640 1 --2- 172.21.15.28:0/3688252158 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb720152370 0x7fb720155940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:12.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.638+0000 7fb71f200640 1 -- 172.21.15.28:0/3688252158 >> 172.21.15.28:0/3688252158 conn(0x7fb720072780 msgr2=0x7fb720072b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:12.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.639+0000 7fb71f200640 1 -- 172.21.15.28:0/3688252158 shutdown_connections 2024-02-21T23:08:12.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.639+0000 7fb71f200640 1 -- 172.21.15.28:0/3688252158 wait complete. 2024-02-21T23:08:12.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.640+0000 7fb71f200640 1 Processor -- start 2024-02-21T23:08:12.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.641+0000 7fb71f200640 1 -- start start 2024-02-21T23:08:12.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.641+0000 7fb71f200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7200bde20 0x7fb7200adc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:12.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.641+0000 7fb71f200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7200c74a0 0x7fb720149720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:12.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.641+0000 7fb71f200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb720152370 0x7fb720152e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:12.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.642+0000 7fb71f200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7200c5950 con 0x7fb7200c74a0 2024-02-21T23:08:12.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.642+0000 7fb71f200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb72006dd00 con 0x7fb7200bde20 2024-02-21T23:08:12.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.642+0000 7fb71f200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7200c5ad0 con 0x7fb720152370 2024-02-21T23:08:12.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.642+0000 7fb71d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7200c74a0 0x7fb720149720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:12.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.642+0000 7fb71d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7200c74a0 0x7fb720149720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54622/0 (socket says 172.21.15.28:54622) 2024-02-21T23:08:12.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.642+0000 7fb71d400640 1 -- 172.21.15.28:0/2442277374 learned_addr learned my addr 172.21.15.28:0/2442277374 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:12.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.642+0000 7fb71de00640 1 --2- 172.21.15.28:0/2442277374 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7200bde20 0x7fb7200adc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:12.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.642+0000 7fb71e800640 1 --2- 172.21.15.28:0/2442277374 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb720152370 0x7fb720152e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:12.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.643+0000 7fb71d400640 1 -- 172.21.15.28:0/2442277374 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb720152370 msgr2=0x7fb720152e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:12.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.643+0000 7fb71d400640 1 --2- 172.21.15.28:0/2442277374 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb720152370 0x7fb720152e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:12.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.643+0000 7fb71d400640 1 -- 172.21.15.28:0/2442277374 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7200bde20 msgr2=0x7fb7200adc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:12.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.643+0000 7fb71d400640 1 --2- 172.21.15.28:0/2442277374 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7200bde20 0x7fb7200adc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:12.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.643+0000 7fb71d400640 1 -- 172.21.15.28:0/2442277374 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7200af150 con 0x7fb7200c74a0 2024-02-21T23:08:12.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.643+0000 7fb71d400640 1 --2- 172.21.15.28:0/2442277374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7200c74a0 0x7fb720149720 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7fb70800e870 tx=0x7fb70800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:12.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.644+0000 7fb716000640 1 -- 172.21.15.28:0/2442277374 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb708017070 con 0x7fb7200c74a0 2024-02-21T23:08:12.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.644+0000 7fb716000640 1 -- 172.21.15.28:0/2442277374 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb70800ee60 con 0x7fb7200c74a0 2024-02-21T23:08:12.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.644+0000 7fb716000640 1 -- 172.21.15.28:0/2442277374 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb70800cb80 con 0x7fb7200c74a0 2024-02-21T23:08:12.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.644+0000 7fb71f200640 1 -- 172.21.15.28:0/2442277374 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7200af430 con 0x7fb7200c74a0 2024-02-21T23:08:12.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.644+0000 7fb71f200640 1 -- 172.21.15.28:0/2442277374 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7200af9f0 con 0x7fb7200c74a0 2024-02-21T23:08:12.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.645+0000 7fb71f200640 1 -- 172.21.15.28:0/2442277374 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb72004f290 con 0x7fb7200c74a0 2024-02-21T23:08:12.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.646+0000 7fb716000640 1 -- 172.21.15.28:0/2442277374 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb70801a070 con 0x7fb7200c74a0 2024-02-21T23:08:12.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.647+0000 7fb716000640 1 --2- 172.21.15.28:0/2442277374 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb6fc06b4d0 0x7fb6fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:12.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.647+0000 7fb716000640 1 -- 172.21.15.28:0/2442277374 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5619+0+0 (secure 0 0 0) 0x7fb708089f90 con 0x7fb7200c74a0 2024-02-21T23:08:12.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.648+0000 7fb71de00640 1 --2- 172.21.15.28:0/2442277374 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb6fc06b4d0 0x7fb6fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:12.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.649+0000 7fb71de00640 1 --2- 172.21.15.28:0/2442277374 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb6fc06b4d0 0x7fb6fc06d990 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fb7200c12e0 tx=0x7fb710009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:12.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.653+0000 7fb716000640 1 -- 172.21.15.28:0/2442277374 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb708058800 con 0x7fb7200c74a0 2024-02-21T23:08:12.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:12.862+0000 7fb71f200640 1 -- 172.21.15.28:0/2442277374 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fb72006da00 con 0x7fb7200c74a0 2024-02-21T23:08:13.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.389+0000 7fb716000640 1 -- 172.21.15.28:0/2442277374 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v120) v1 ==== 170+0+0 (secure 0 0 0) 0x7fb70805c8a0 con 0x7fb7200c74a0 2024-02-21T23:08:13.377 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' created 2024-02-21T23:08:13.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.393+0000 7fb71f200640 1 -- 172.21.15.28:0/2442277374 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb6fc06b4d0 msgr2=0x7fb6fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:13.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.393+0000 7fb71f200640 1 --2- 172.21.15.28:0/2442277374 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb6fc06b4d0 0x7fb6fc06d990 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fb7200c12e0 tx=0x7fb710009190 comp rx=0 tx=0).stop 2024-02-21T23:08:13.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.393+0000 7fb71f200640 1 -- 172.21.15.28:0/2442277374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7200c74a0 msgr2=0x7fb720149720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:13.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.393+0000 7fb71f200640 1 --2- 172.21.15.28:0/2442277374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7200c74a0 0x7fb720149720 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7fb70800e870 tx=0x7fb70800ed30 comp rx=0 tx=0).stop 2024-02-21T23:08:13.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.393+0000 7fb71f200640 1 -- 172.21.15.28:0/2442277374 shutdown_connections 2024-02-21T23:08:13.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.393+0000 7fb71f200640 1 --2- 172.21.15.28:0/2442277374 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb6fc06b4d0 0x7fb6fc06d990 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:13.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.393+0000 7fb71f200640 1 --2- 172.21.15.28:0/2442277374 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb720152370 0x7fb720152e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:13.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.393+0000 7fb71f200640 1 --2- 172.21.15.28:0/2442277374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7200c74a0 0x7fb720149720 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:13.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.393+0000 7fb71f200640 1 --2- 172.21.15.28:0/2442277374 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb7200bde20 0x7fb7200adc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:13.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.393+0000 7fb71f200640 1 -- 172.21.15.28:0/2442277374 >> 172.21.15.28:0/2442277374 conn(0x7fb720072780 msgr2=0x7fb7200c8570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:13.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.393+0000 7fb71f200640 1 -- 172.21.15.28:0/2442277374 shutdown_connections 2024-02-21T23:08:13.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.393+0000 7fb71f200640 1 -- 172.21.15.28:0/2442277374 wait complete. 2024-02-21T23:08:13.418 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-02-21T23:08:13.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.619+0000 7f0f3e400640 1 Processor -- start 2024-02-21T23:08:13.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.619+0000 7f0f3e400640 1 -- start start 2024-02-21T23:08:13.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.621+0000 7f0f3e400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0f380c5f50 0x7f0f3814fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:13.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.621+0000 7f0f3e400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0f38150530 0x7f0f38156500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:13.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.621+0000 7f0f3e400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f381509e0 con 0x7f0f3806b0e0 2024-02-21T23:08:13.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.621+0000 7f0f3e400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f38153bc0 con 0x7f0f38150530 2024-02-21T23:08:13.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.621+0000 7f0f3e400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f38152fb0 con 0x7f0f380c5f50 2024-02-21T23:08:13.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.621+0000 7f0f37e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0f38150530 0x7f0f38156500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:13.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.621+0000 7f0f2ea00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0f380c5f50 0x7f0f3814fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:13.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.621+0000 7f0f37e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0f38150530 0x7f0f38156500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54632/0 (socket says 172.21.15.28:54632) 2024-02-21T23:08:13.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.621+0000 7f0f37e00640 1 -- 172.21.15.28:0/2003554824 learned_addr learned my addr 172.21.15.28:0/2003554824 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:13.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.622+0000 7f0f37e00640 1 -- 172.21.15.28:0/2003554824 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0f380c5f50 msgr2=0x7f0f3814fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:13.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.622+0000 7f0f37e00640 1 --2- 172.21.15.28:0/2003554824 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0f380c5f50 0x7f0f3814fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:13.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.622+0000 7f0f37e00640 1 -- 172.21.15.28:0/2003554824 >> v1:172.21.15.81:6789/0 conn(0x7f0f3806b0e0 legacy=0x7f0f380c5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:13.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.622+0000 7f0f37e00640 1 -- 172.21.15.28:0/2003554824 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f38153290 con 0x7f0f38150530 2024-02-21T23:08:13.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.623+0000 7f0f37e00640 1 --2- 172.21.15.28:0/2003554824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f38150530 0x7f0f38156500 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f0f30009e50 tx=0x7f0f3002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f78c4c7959b417ae server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:13.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.623+0000 7f0f36a00640 1 -- 172.21.15.28:0/2003554824 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f3002ca70 con 0x7f0f38150530 2024-02-21T23:08:13.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.623+0000 7f0f36a00640 1 -- 172.21.15.28:0/2003554824 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f3002cc10 con 0x7f0f38150530 2024-02-21T23:08:13.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.623+0000 7f0f36a00640 1 -- 172.21.15.28:0/2003554824 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f300343f0 con 0x7f0f38150530 2024-02-21T23:08:13.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.624+0000 7f0f3e400640 1 -- 172.21.15.28:0/2003554824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f38150530 msgr2=0x7f0f38156500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:13.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.624+0000 7f0f3e400640 1 --2- 172.21.15.28:0/2003554824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f38150530 0x7f0f38156500 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f0f30009e50 tx=0x7f0f3002bed0 comp rx=0 tx=0).stop 2024-02-21T23:08:13.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.624+0000 7f0f3e400640 1 -- 172.21.15.28:0/2003554824 shutdown_connections 2024-02-21T23:08:13.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.624+0000 7f0f3e400640 1 --2- 172.21.15.28:0/2003554824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f38150530 0x7f0f38156500 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:13.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.624+0000 7f0f3e400640 1 --2- 172.21.15.28:0/2003554824 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0f380c5f50 0x7f0f3814fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:13.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.624+0000 7f0f3e400640 1 -- 172.21.15.28:0/2003554824 >> 172.21.15.28:0/2003554824 conn(0x7f0f380a85a0 msgr2=0x7f0f380a8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:13.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.624+0000 7f0f3e400640 1 -- 172.21.15.28:0/2003554824 shutdown_connections 2024-02-21T23:08:13.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.624+0000 7f0f3e400640 1 -- 172.21.15.28:0/2003554824 wait complete. 2024-02-21T23:08:13.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.626+0000 7f0f3e400640 1 Processor -- start 2024-02-21T23:08:13.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.626+0000 7f0f3e400640 1 -- start start 2024-02-21T23:08:13.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.627+0000 7f0f3e400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f3806b0e0 0x7f0f38149580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:13.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.627+0000 7f0f3e400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0f380c5f50 0x7f0f38150c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:13.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.627+0000 7f0f3e400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f38150530 0x7f0f381551a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:13.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.627+0000 7f0f3e400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f3806b4c0 con 0x7f0f3806b0e0 2024-02-21T23:08:13.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.627+0000 7f0f3e400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f38153850 con 0x7f0f38150530 2024-02-21T23:08:13.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.627+0000 7f0f3e400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f3806b640 con 0x7f0f380c5f50 2024-02-21T23:08:13.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.627+0000 7f0f37400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f3806b0e0 0x7f0f38149580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:13.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.627+0000 7f0f37400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f3806b0e0 0x7f0f38149580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54648/0 (socket says 172.21.15.28:54648) 2024-02-21T23:08:13.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.627+0000 7f0f37400640 1 -- 172.21.15.28:0/1649009477 learned_addr learned my addr 172.21.15.28:0/1649009477 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:13.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.628+0000 7f0f37e00640 1 --2- 172.21.15.28:0/1649009477 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f38150530 0x7f0f381551a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:13.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.628+0000 7f0f2ea00640 1 --2- 172.21.15.28:0/1649009477 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0f380c5f50 0x7f0f38150c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:13.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.628+0000 7f0f37400640 1 -- 172.21.15.28:0/1649009477 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0f380c5f50 msgr2=0x7f0f38150c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:13.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.628+0000 7f0f37400640 1 --2- 172.21.15.28:0/1649009477 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0f380c5f50 0x7f0f38150c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:13.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.628+0000 7f0f37400640 1 -- 172.21.15.28:0/1649009477 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f38150530 msgr2=0x7f0f381551a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:13.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.628+0000 7f0f37400640 1 --2- 172.21.15.28:0/1649009477 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f38150530 0x7f0f381551a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:13.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.628+0000 7f0f37400640 1 -- 172.21.15.28:0/1649009477 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f38151690 con 0x7f0f3806b0e0 2024-02-21T23:08:13.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.628+0000 7f0f37400640 1 --2- 172.21.15.28:0/1649009477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f3806b0e0 0x7f0f38149580 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f0f2800c3c0 tx=0x7f0f2800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:13.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.629+0000 7f0f2fe00640 1 -- 172.21.15.28:0/1649009477 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f280052d0 con 0x7f0f3806b0e0 2024-02-21T23:08:13.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.629+0000 7f0f2fe00640 1 -- 172.21.15.28:0/1649009477 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f28005490 con 0x7f0f3806b0e0 2024-02-21T23:08:13.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.629+0000 7f0f2fe00640 1 -- 172.21.15.28:0/1649009477 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f28007260 con 0x7f0f3806b0e0 2024-02-21T23:08:13.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.629+0000 7f0f3e400640 1 -- 172.21.15.28:0/1649009477 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f38151970 con 0x7f0f3806b0e0 2024-02-21T23:08:13.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.629+0000 7f0f3e400640 1 -- 172.21.15.28:0/1649009477 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f380c10c0 con 0x7f0f3806b0e0 2024-02-21T23:08:13.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.630+0000 7f0f3e400640 1 -- 172.21.15.28:0/1649009477 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f38067f30 con 0x7f0f3806b0e0 2024-02-21T23:08:13.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.631+0000 7f0f2fe00640 1 -- 172.21.15.28:0/1649009477 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0f28007400 con 0x7f0f3806b0e0 2024-02-21T23:08:13.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.632+0000 7f0f2fe00640 1 --2- 172.21.15.28:0/1649009477 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0f1406b4d0 0x7f0f1406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:13.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.632+0000 7f0f2fe00640 1 -- 172.21.15.28:0/1649009477 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6001+0+0 (secure 0 0 0) 0x7f0f28089810 con 0x7f0f3806b0e0 2024-02-21T23:08:13.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.633+0000 7f0f2ea00640 1 --2- 172.21.15.28:0/1649009477 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0f1406b4d0 0x7f0f1406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:13.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.634+0000 7f0f2ea00640 1 --2- 172.21.15.28:0/1649009477 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0f1406b4d0 0x7f0f1406d990 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f0f380c0b60 tx=0x7f0f20008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:13.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.637+0000 7f0f2fe00640 1 -- 172.21.15.28:0/1649009477 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f0f28058560 con 0x7f0f3806b0e0 2024-02-21T23:08:13.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:13.880+0000 7f0f3e400640 1 -- 172.21.15.28:0/1649009477 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f0f380abda0 con 0x7f0f3806b0e0 2024-02-21T23:08:14.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.416+0000 7f0f2fe00640 1 -- 172.21.15.28:0/1649009477 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 19 and data pool 20 v117) v1 ==== 174+0+0 (secure 0 0 0) 0x7f0f28057e60 con 0x7f0f3806b0e0 2024-02-21T23:08:14.405 INFO:teuthology.orchestra.run.smithi028.stderr:new fs with metadata pool 19 and data pool 20 2024-02-21T23:08:14.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.420+0000 7f0f3e400640 1 -- 172.21.15.28:0/1649009477 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0f1406b4d0 msgr2=0x7f0f1406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:14.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.420+0000 7f0f3e400640 1 --2- 172.21.15.28:0/1649009477 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0f1406b4d0 0x7f0f1406d990 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f0f380c0b60 tx=0x7f0f20008040 comp rx=0 tx=0).stop 2024-02-21T23:08:14.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.420+0000 7f0f3e400640 1 -- 172.21.15.28:0/1649009477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f3806b0e0 msgr2=0x7f0f38149580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:14.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.420+0000 7f0f3e400640 1 --2- 172.21.15.28:0/1649009477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f3806b0e0 0x7f0f38149580 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f0f2800c3c0 tx=0x7f0f2800c880 comp rx=0 tx=0).stop 2024-02-21T23:08:14.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.420+0000 7f0f3e400640 1 -- 172.21.15.28:0/1649009477 shutdown_connections 2024-02-21T23:08:14.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.420+0000 7f0f3e400640 1 --2- 172.21.15.28:0/1649009477 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0f1406b4d0 0x7f0f1406d990 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:14.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.420+0000 7f0f3e400640 1 --2- 172.21.15.28:0/1649009477 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0f380c5f50 0x7f0f38150c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:14.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.420+0000 7f0f3e400640 1 --2- 172.21.15.28:0/1649009477 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f38150530 0x7f0f381551a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:14.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.420+0000 7f0f3e400640 1 --2- 172.21.15.28:0/1649009477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f3806b0e0 0x7f0f38149580 secure :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f0f2800c3c0 tx=0x7f0f2800c880 comp rx=0 tx=0).stop 2024-02-21T23:08:14.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.420+0000 7f0f3e400640 1 -- 172.21.15.28:0/1649009477 >> 172.21.15.28:0/1649009477 conn(0x7f0f380a85a0 msgr2=0x7f0f380c50d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:14.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.421+0000 7f0f3e400640 1 -- 172.21.15.28:0/1649009477 shutdown_connections 2024-02-21T23:08:14.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.421+0000 7f0f3e400640 1 -- 172.21.15.28:0/1649009477 wait complete. 2024-02-21T23:08:14.444 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:08:14.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.642+0000 7fd5e0c00640 1 Processor -- start 2024-02-21T23:08:14.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.642+0000 7fd5e0c00640 1 -- start start 2024-02-21T23:08:14.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.643+0000 7fd5e0c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fd5dc0c11f0 0x7fd5dc153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:14.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.643+0000 7fd5e0c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5dc152e70 con 0x7fd5dc15b6d0 2024-02-21T23:08:14.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.643+0000 7fd5e0c00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5dc153150 con 0x7fd5dc0c15d0 2024-02-21T23:08:14.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.643+0000 7fd5e0c00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5dc0c1a60 con 0x7fd5dc0c11f0 2024-02-21T23:08:14.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.644+0000 7fd5da800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd5dc15b6d0 0x7fd5dc15dac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:46720/0 (socket says 172.21.15.28:46720) 2024-02-21T23:08:14.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.644+0000 7fd5da800640 1 -- 172.21.15.28:0/758672630 learned_addr learned my addr 172.21.15.28:0/758672630 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:08:14.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.644+0000 7fd5d9e00640 1 --2- 172.21.15.28:0/758672630 >> v2:172.21.15.81:3300/0 conn(0x7fd5dc0c11f0 0x7fd5dc153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:14.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.644+0000 7fd5d9400640 1 -- 172.21.15.28:0/758672630 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2443402712 0 0) 0x7fd5dc152e70 con 0x7fd5dc15b6d0 2024-02-21T23:08:14.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.644+0000 7fd5d9400640 1 -- 172.21.15.28:0/758672630 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5bc003520 con 0x7fd5dc15b6d0 2024-02-21T23:08:14.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.645+0000 7fd5d9400640 1 -- 172.21.15.28:0/758672630 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1358281990 0 0) 0x7fd5bc003520 con 0x7fd5dc15b6d0 2024-02-21T23:08:14.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.645+0000 7fd5d9400640 1 -- 172.21.15.28:0/758672630 >> v2:172.21.15.81:3300/0 conn(0x7fd5dc0c11f0 msgr2=0x7fd5dc153950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:14.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.645+0000 7fd5d9400640 1 --2- 172.21.15.28:0/758672630 >> v2:172.21.15.81:3300/0 conn(0x7fd5dc0c11f0 0x7fd5dc153950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:14.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.645+0000 7fd5d9400640 1 -- 172.21.15.28:0/758672630 >> v1:172.21.15.81:6789/0 conn(0x7fd5dc0c15d0 legacy=0x7fd5dc06dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:14.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.645+0000 7fd5d9400640 1 -- 172.21.15.28:0/758672630 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5dc153430 con 0x7fd5dc15b6d0 2024-02-21T23:08:14.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.645+0000 7fd5d9400640 1 -- 172.21.15.28:0/758672630 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fd5d40019f0 con 0x7fd5dc15b6d0 2024-02-21T23:08:14.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.645+0000 7fd5d9400640 1 -- 172.21.15.28:0/758672630 >> v1:172.21.15.28:6789/0 conn(0x7fd5dc15b6d0 legacy=0x7fd5dc15dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:14.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.645+0000 7fd5d9400640 1 --2- 172.21.15.28:0/758672630 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5bc003a00 0x7fd5bc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:14.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.645+0000 7fd5d9400640 1 --2- 172.21.15.28:0/758672630 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5bc004800 0x7fd5bc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:14.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.645+0000 7fd5d9400640 1 --2- 172.21.15.28:0/758672630 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd5bc0052e0 0x7fd5bc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:14.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.645+0000 7fd5d9400640 1 -- 172.21.15.28:0/758672630 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5dc0c1a60 con 0x7fd5bc003a00 2024-02-21T23:08:14.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.645+0000 7fd5d9400640 1 -- 172.21.15.28:0/758672630 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5bc00c1e0 con 0x7fd5bc004800 2024-02-21T23:08:14.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.645+0000 7fd5d9400640 1 -- 172.21.15.28:0/758672630 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5bc00c3e0 con 0x7fd5bc0052e0 2024-02-21T23:08:14.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.645+0000 7fd5d9e00640 1 --2- 172.21.15.28:0/758672630 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5bc003a00 0x7fd5bc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:14.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.645+0000 7fd5d3e00640 1 --2- 172.21.15.28:0/758672630 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5bc004800 0x7fd5bc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:14.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.646+0000 7fd5da800640 1 --2- 172.21.15.28:0/758672630 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd5bc0052e0 0x7fd5bc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:14.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.646+0000 7fd5d9e00640 1 -- 172.21.15.28:0/758672630 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd5bc0052e0 msgr2=0x7fd5bc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:14.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.646+0000 7fd5d9e00640 1 --2- 172.21.15.28:0/758672630 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd5bc0052e0 0x7fd5bc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:14.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.646+0000 7fd5d9e00640 1 -- 172.21.15.28:0/758672630 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5bc004800 msgr2=0x7fd5bc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:14.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.646+0000 7fd5d9e00640 1 --2- 172.21.15.28:0/758672630 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5bc004800 0x7fd5bc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:14.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.646+0000 7fd5d9e00640 1 -- 172.21.15.28:0/758672630 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5bc003520 con 0x7fd5bc003a00 2024-02-21T23:08:14.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.646+0000 7fd5da800640 1 -- 172.21.15.28:0/758672630 reap_dead start 2024-02-21T23:08:14.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.647+0000 7fd5d9e00640 1 --2- 172.21.15.28:0/758672630 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5bc003a00 0x7fd5bc003e20 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7fd5cc009ac0 tx=0x7fd5bc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28da1040e55aa2fe server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:14.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.648+0000 7fd5d9400640 1 -- 172.21.15.28:0/758672630 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5cc039070 con 0x7fd5bc003a00 2024-02-21T23:08:14.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.648+0000 7fd5d9400640 1 -- 172.21.15.28:0/758672630 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5cc02c6b0 con 0x7fd5bc003a00 2024-02-21T23:08:14.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.648+0000 7fd5d9400640 1 -- 172.21.15.28:0/758672630 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5cc009690 con 0x7fd5bc003a00 2024-02-21T23:08:14.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.648+0000 7fd5e0c00640 1 -- 172.21.15.28:0/758672630 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5bc003a00 msgr2=0x7fd5bc003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:14.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.648+0000 7fd5e0c00640 1 --2- 172.21.15.28:0/758672630 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5bc003a00 0x7fd5bc003e20 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7fd5cc009ac0 tx=0x7fd5bc005830 comp rx=0 tx=0).stop 2024-02-21T23:08:14.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.648+0000 7fd5e0c00640 1 -- 172.21.15.28:0/758672630 shutdown_connections 2024-02-21T23:08:14.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.648+0000 7fd5e0c00640 1 --2- 172.21.15.28:0/758672630 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5bc003a00 0x7fd5bc003e20 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:14.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.648+0000 7fd5e0c00640 1 -- 172.21.15.28:0/758672630 >> 172.21.15.28:0/758672630 conn(0x7fd5dc0c67f0 msgr2=0x7fd5dc0c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:14.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.648+0000 7fd5e0c00640 1 -- 172.21.15.28:0/758672630 shutdown_connections 2024-02-21T23:08:14.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.648+0000 7fd5e0c00640 1 -- 172.21.15.28:0/758672630 wait complete. 2024-02-21T23:08:14.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.650+0000 7fd5e0c00640 1 Processor -- start 2024-02-21T23:08:14.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.650+0000 7fd5e0c00640 1 -- start start 2024-02-21T23:08:14.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.651+0000 7fd5e0c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5bc003a00 0x7fd5dc14eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:14.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.651+0000 7fd5e0c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5dc1eee20 0x7fd5dc1eb680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:14.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.651+0000 7fd5e0c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd5dc1ebbc0 0x7fd5dc0bfc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:14.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.651+0000 7fd5e0c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5dc0c4c00 con 0x7fd5bc003a00 2024-02-21T23:08:14.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.651+0000 7fd5e0c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5dc0a5ec0 con 0x7fd5dc1eee20 2024-02-21T23:08:14.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.651+0000 7fd5e0c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5dc0a6110 con 0x7fd5dc1ebbc0 2024-02-21T23:08:14.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.652+0000 7fd5d9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5bc003a00 0x7fd5dc14eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:14.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.652+0000 7fd5d9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5bc003a00 0x7fd5dc14eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54664/0 (socket says 172.21.15.28:54664) 2024-02-21T23:08:14.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.652+0000 7fd5d9e00640 1 -- 172.21.15.28:0/3120784915 learned_addr learned my addr 172.21.15.28:0/3120784915 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:14.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.652+0000 7fd5d9e00640 1 -- 172.21.15.28:0/3120784915 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd5dc1ebbc0 msgr2=0x7fd5dc0bfc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:14.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.652+0000 7fd5da800640 1 --2- 172.21.15.28:0/3120784915 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd5dc1ebbc0 0x7fd5dc0bfc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:14.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.652+0000 7fd5d3e00640 1 --2- 172.21.15.28:0/3120784915 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5dc1eee20 0x7fd5dc1eb680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:14.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.652+0000 7fd5d9e00640 1 --2- 172.21.15.28:0/3120784915 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd5dc1ebbc0 0x7fd5dc0bfc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:14.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.652+0000 7fd5d9e00640 1 -- 172.21.15.28:0/3120784915 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5dc1eee20 msgr2=0x7fd5dc1eb680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:14.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.652+0000 7fd5d9e00640 1 --2- 172.21.15.28:0/3120784915 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5dc1eee20 0x7fd5dc1eb680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:14.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.652+0000 7fd5d9e00640 1 -- 172.21.15.28:0/3120784915 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5dc1eaea0 con 0x7fd5bc003a00 2024-02-21T23:08:14.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.652+0000 7fd5d3e00640 1 --2- 172.21.15.28:0/3120784915 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5dc1eee20 0x7fd5dc1eb680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:08:14.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.652+0000 7fd5d9e00640 1 --2- 172.21.15.28:0/3120784915 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5bc003a00 0x7fd5dc14eb80 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7fd5cc009ac0 tx=0x7fd5cc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:14.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.653+0000 7fd5d2000640 1 -- 172.21.15.28:0/3120784915 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5cc039050 con 0x7fd5bc003a00 2024-02-21T23:08:14.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.653+0000 7fd5d2000640 1 -- 172.21.15.28:0/3120784915 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5cc045070 con 0x7fd5bc003a00 2024-02-21T23:08:14.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.653+0000 7fd5d2000640 1 -- 172.21.15.28:0/3120784915 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5cc0044e0 con 0x7fd5bc003a00 2024-02-21T23:08:14.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.653+0000 7fd5e0c00640 1 -- 172.21.15.28:0/3120784915 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5dc072c60 con 0x7fd5bc003a00 2024-02-21T23:08:14.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.653+0000 7fd5e0c00640 1 -- 172.21.15.28:0/3120784915 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5dc073220 con 0x7fd5bc003a00 2024-02-21T23:08:14.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.654+0000 7fd5e0c00640 1 -- 172.21.15.28:0/3120784915 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5dc067f30 con 0x7fd5bc003a00 2024-02-21T23:08:14.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.655+0000 7fd5d2000640 1 -- 172.21.15.28:0/3120784915 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd5cc003a50 con 0x7fd5bc003a00 2024-02-21T23:08:14.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.656+0000 7fd5d2000640 1 --2- 172.21.15.28:0/3120784915 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd5b806b4d0 0x7fd5b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:14.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.657+0000 7fd5d2000640 1 -- 172.21.15.28:0/3120784915 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd5cc0abe10 con 0x7fd5bc003a00 2024-02-21T23:08:14.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.657+0000 7fd5d3e00640 1 --2- 172.21.15.28:0/3120784915 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd5b806b4d0 0x7fd5b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:14.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.658+0000 7fd5d3e00640 1 --2- 172.21.15.28:0/3120784915 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd5b806b4d0 0x7fd5b806d990 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fd5c4000c00 tx=0x7fd5c4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:14.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.662+0000 7fd5d2000640 1 -- 172.21.15.28:0/3120784915 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd5cc0af030 con 0x7fd5bc003a00 2024-02-21T23:08:14.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.856+0000 7fd5e0c00640 1 -- 172.21.15.28:0/3120784915 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd5dc0c0960 con 0x7fd5bc003a00 2024-02-21T23:08:14.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.859+0000 7fd5d2000640 1 -- 172.21.15.28:0/3120784915 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v121) v1 ==== 74+0+16853 (secure 0 0 0) 0x7fd5cc04e020 con 0x7fd5bc003a00 2024-02-21T23:08:14.850 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:08:14.850 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":121,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:08:13.882184+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:08:11.794800+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"121","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2024-02-21T23:08:12.918602+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"121","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:08:14.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.862+0000 7fd5e0c00640 1 -- 172.21.15.28:0/3120784915 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd5b806b4d0 msgr2=0x7fd5b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:14.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.862+0000 7fd5e0c00640 1 --2- 172.21.15.28:0/3120784915 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd5b806b4d0 0x7fd5b806d990 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fd5c4000c00 tx=0x7fd5c4009040 comp rx=0 tx=0).stop 2024-02-21T23:08:14.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.862+0000 7fd5e0c00640 1 -- 172.21.15.28:0/3120784915 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5bc003a00 msgr2=0x7fd5dc14eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:14.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.862+0000 7fd5e0c00640 1 --2- 172.21.15.28:0/3120784915 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5bc003a00 0x7fd5dc14eb80 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7fd5cc009ac0 tx=0x7fd5cc004730 comp rx=0 tx=0).stop 2024-02-21T23:08:14.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.862+0000 7fd5e0c00640 1 -- 172.21.15.28:0/3120784915 shutdown_connections 2024-02-21T23:08:14.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.862+0000 7fd5e0c00640 1 --2- 172.21.15.28:0/3120784915 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd5b806b4d0 0x7fd5b806d990 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:14.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.862+0000 7fd5e0c00640 1 --2- 172.21.15.28:0/3120784915 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd5dc1ebbc0 0x7fd5dc0bfc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:14.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.862+0000 7fd5e0c00640 1 --2- 172.21.15.28:0/3120784915 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5dc1eee20 0x7fd5dc1eb680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:14.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.862+0000 7fd5e0c00640 1 --2- 172.21.15.28:0/3120784915 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5bc003a00 0x7fd5dc14eb80 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:14.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.862+0000 7fd5e0c00640 1 -- 172.21.15.28:0/3120784915 >> 172.21.15.28:0/3120784915 conn(0x7fd5dc0c67f0 msgr2=0x7fd5dc152560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:14.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.863+0000 7fd5e0c00640 1 -- 172.21.15.28:0/3120784915 shutdown_connections 2024-02-21T23:08:14.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:14.863+0000 7fd5e0c00640 1 -- 172.21.15.28:0/3120784915 wait complete. 2024-02-21T23:08:14.868 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:08:15.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.091+0000 7fecede00640 1 Processor -- start 2024-02-21T23:08:15.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.091+0000 7fecede00640 1 -- start start 2024-02-21T23:08:15.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.092+0000 7fecede00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fece80aff90 0x7fece80c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:15.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.092+0000 7fecede00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fece80c5720 0x7fece815f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:15.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.093+0000 7fecede00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fece80c74c0 con 0x7fece806dbf0 2024-02-21T23:08:15.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.093+0000 7fecede00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fece80c7270 con 0x7fece80c5720 2024-02-21T23:08:15.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.093+0000 7fecede00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fece8158630 con 0x7fece80aff90 2024-02-21T23:08:15.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.093+0000 7feced400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fece80c5720 0x7fece815f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:15.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.093+0000 7feced400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fece80c5720 0x7fece815f890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.28:38024/0 (socket says 172.21.15.28:38024) 2024-02-21T23:08:15.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.093+0000 7feced400640 1 -- 172.21.15.28:0/1648454474 learned_addr learned my addr 172.21.15.28:0/1648454474 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:15.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.093+0000 7fececa00640 1 --2- 172.21.15.28:0/1648454474 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fece80aff90 0x7fece80c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:15.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.094+0000 7feced400640 1 -- 172.21.15.28:0/1648454474 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fece80aff90 msgr2=0x7fece80c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:15.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.094+0000 7feced400640 1 --2- 172.21.15.28:0/1648454474 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fece80aff90 0x7fece80c51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:15.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.094+0000 7feced400640 1 -- 172.21.15.28:0/1648454474 >> v1:172.21.15.81:6789/0 conn(0x7fece806dbf0 legacy=0x7fece8158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:15.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.094+0000 7feced400640 1 -- 172.21.15.28:0/1648454474 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fece80c77a0 con 0x7fece80c5720 2024-02-21T23:08:15.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.095+0000 7feced400640 1 --2- 172.21.15.28:0/1648454474 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fece80c5720 0x7fece815f890 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fece0009e50 tx=0x7fece002bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=45cb43bc306c1688 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:15.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.096+0000 7fece7400640 1 -- 172.21.15.28:0/1648454474 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece002cb00 con 0x7fece80c5720 2024-02-21T23:08:15.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.096+0000 7fece7400640 1 -- 172.21.15.28:0/1648454474 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fece002cca0 con 0x7fece80c5720 2024-02-21T23:08:15.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.096+0000 7fece7400640 1 -- 172.21.15.28:0/1648454474 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece0034610 con 0x7fece80c5720 2024-02-21T23:08:15.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.097+0000 7fecede00640 1 -- 172.21.15.28:0/1648454474 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fece80c5720 msgr2=0x7fece815f890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:15.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.097+0000 7fecede00640 1 --2- 172.21.15.28:0/1648454474 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fece80c5720 0x7fece815f890 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fece0009e50 tx=0x7fece002bf60 comp rx=0 tx=0).stop 2024-02-21T23:08:15.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.097+0000 7fecede00640 1 -- 172.21.15.28:0/1648454474 shutdown_connections 2024-02-21T23:08:15.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.097+0000 7fecede00640 1 --2- 172.21.15.28:0/1648454474 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fece80c5720 0x7fece815f890 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:15.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.097+0000 7fecede00640 1 --2- 172.21.15.28:0/1648454474 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fece80aff90 0x7fece80c51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:15.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.097+0000 7fecede00640 1 -- 172.21.15.28:0/1648454474 >> 172.21.15.28:0/1648454474 conn(0x7fece80ac370 msgr2=0x7fece80ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:15.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.097+0000 7fecede00640 1 -- 172.21.15.28:0/1648454474 shutdown_connections 2024-02-21T23:08:15.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.097+0000 7fecede00640 1 -- 172.21.15.28:0/1648454474 wait complete. 2024-02-21T23:08:15.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.098+0000 7fecede00640 1 Processor -- start 2024-02-21T23:08:15.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.099+0000 7fecede00640 1 -- start start 2024-02-21T23:08:15.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.099+0000 7fecede00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fece806dbf0 0x7fece80c21b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:15.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.099+0000 7fecede00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fece80aff90 0x7fece81eb250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:15.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.099+0000 7fecede00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fece80c5720 0x7fece80aabe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:15.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.099+0000 7fecede00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fece80c8760 con 0x7fece80c5720 2024-02-21T23:08:15.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.099+0000 7fecede00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fece80be0c0 con 0x7fece80aff90 2024-02-21T23:08:15.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.099+0000 7fecede00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fece80c88e0 con 0x7fece806dbf0 2024-02-21T23:08:15.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.100+0000 7feced400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fece80c5720 0x7fece80aabe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:15.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.100+0000 7feced400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fece80c5720 0x7fece80aabe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57524/0 (socket says 172.21.15.28:57524) 2024-02-21T23:08:15.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.100+0000 7feced400640 1 -- 172.21.15.28:0/2566261647 learned_addr learned my addr 172.21.15.28:0/2566261647 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:15.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.100+0000 7fece7e00640 1 --2- 172.21.15.28:0/2566261647 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fece80aff90 0x7fece81eb250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:15.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.100+0000 7fececa00640 1 --2- 172.21.15.28:0/2566261647 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fece806dbf0 0x7fece80c21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:15.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.100+0000 7feced400640 1 -- 172.21.15.28:0/2566261647 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fece806dbf0 msgr2=0x7fece80c21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:15.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.100+0000 7feced400640 1 --2- 172.21.15.28:0/2566261647 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fece806dbf0 0x7fece80c21b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:15.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.100+0000 7feced400640 1 -- 172.21.15.28:0/2566261647 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fece80aff90 msgr2=0x7fece81eb250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:15.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.100+0000 7feced400640 1 --2- 172.21.15.28:0/2566261647 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fece80aff90 0x7fece81eb250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:15.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.100+0000 7feced400640 1 -- 172.21.15.28:0/2566261647 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fece81ede80 con 0x7fece80c5720 2024-02-21T23:08:15.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.101+0000 7feced400640 1 --2- 172.21.15.28:0/2566261647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fece80c5720 0x7fece80aabe0 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7fece0036040 tx=0x7fece0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:15.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.101+0000 7fece4c00640 1 -- 172.21.15.28:0/2566261647 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece0044020 con 0x7fece80c5720 2024-02-21T23:08:15.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.101+0000 7fece4c00640 1 -- 172.21.15.28:0/2566261647 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fece002ce50 con 0x7fece80c5720 2024-02-21T23:08:15.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.101+0000 7fece4c00640 1 -- 172.21.15.28:0/2566261647 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece00408d0 con 0x7fece80c5720 2024-02-21T23:08:15.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.101+0000 7fecede00640 1 -- 172.21.15.28:0/2566261647 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fece81eb790 con 0x7fece80c5720 2024-02-21T23:08:15.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.101+0000 7fecede00640 1 -- 172.21.15.28:0/2566261647 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fece8154a40 con 0x7fece80c5720 2024-02-21T23:08:15.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.102+0000 7fecede00640 1 -- 172.21.15.28:0/2566261647 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fece804f290 con 0x7fece80c5720 2024-02-21T23:08:15.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.103+0000 7fece4c00640 1 -- 172.21.15.28:0/2566261647 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fece0047020 con 0x7fece80c5720 2024-02-21T23:08:15.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.104+0000 7fece4c00640 1 --2- 172.21.15.28:0/2566261647 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fecc806b4d0 0x7fecc806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:15.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.104+0000 7fece4c00640 1 -- 172.21.15.28:0/2566261647 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fece00acc50 con 0x7fece80c5720 2024-02-21T23:08:15.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.105+0000 7fececa00640 1 --2- 172.21.15.28:0/2566261647 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fecc806b4d0 0x7fecc806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:15.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.106+0000 7fececa00640 1 --2- 172.21.15.28:0/2566261647 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fecc806b4d0 0x7fecc806d990 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fecdc009a10 tx=0x7fecdc008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:15.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.108+0000 7fece4c00640 1 -- 172.21.15.28:0/2566261647 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fece004e030 con 0x7fece80c5720 2024-02-21T23:08:15.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.309+0000 7fecede00640 1 -- 172.21.15.28:0/2566261647 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fece8158140 con 0x7fece80c5720 2024-02-21T23:08:15.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.311+0000 7fece4c00640 1 -- 172.21.15.28:0/2566261647 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v121) v1 ==== 74+0+16853 (secure 0 0 0) 0x7fece007bb10 con 0x7fece80c5720 2024-02-21T23:08:15.299 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:08:15.299 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":121,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:08:13.882184+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:08:11.794800+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"121","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2024-02-21T23:08:12.918602+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"121","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:08:15.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.313+0000 7fecede00640 1 -- 172.21.15.28:0/2566261647 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fecc806b4d0 msgr2=0x7fecc806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:15.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.314+0000 7fecede00640 1 --2- 172.21.15.28:0/2566261647 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fecc806b4d0 0x7fecc806d990 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fecdc009a10 tx=0x7fecdc008040 comp rx=0 tx=0).stop 2024-02-21T23:08:15.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.314+0000 7fecede00640 1 -- 172.21.15.28:0/2566261647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fece80c5720 msgr2=0x7fece80aabe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:15.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.314+0000 7fecede00640 1 --2- 172.21.15.28:0/2566261647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fece80c5720 0x7fece80aabe0 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7fece0036040 tx=0x7fece0004a40 comp rx=0 tx=0).stop 2024-02-21T23:08:15.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.314+0000 7fecede00640 1 -- 172.21.15.28:0/2566261647 shutdown_connections 2024-02-21T23:08:15.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.314+0000 7fecede00640 1 --2- 172.21.15.28:0/2566261647 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fecc806b4d0 0x7fecc806d990 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:15.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.314+0000 7fecede00640 1 --2- 172.21.15.28:0/2566261647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fece80c5720 0x7fece80aabe0 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:15.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.314+0000 7fecede00640 1 --2- 172.21.15.28:0/2566261647 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fece80aff90 0x7fece81eb250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:15.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.314+0000 7fecede00640 1 --2- 172.21.15.28:0/2566261647 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fece806dbf0 0x7fece80c21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:15.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.314+0000 7fecede00640 1 -- 172.21.15.28:0/2566261647 >> 172.21.15.28:0/2566261647 conn(0x7fece80ac370 msgr2=0x7fece80ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:15.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.314+0000 7fecede00640 1 -- 172.21.15.28:0/2566261647 shutdown_connections 2024-02-21T23:08:15.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.314+0000 7fecede00640 1 -- 172.21.15.28:0/2566261647 wait complete. 2024-02-21T23:08:15.321 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-02-21T23:08:15.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.559+0000 7f68b7000640 1 Processor -- start 2024-02-21T23:08:15.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.559+0000 7f68b7000640 1 -- start start 2024-02-21T23:08:15.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.560+0000 7f68b7000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f68b00c6220 0x7f68b01586b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:15.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.561+0000 7f68b7000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68b00c20d0 con 0x7f68b0158bf0 2024-02-21T23:08:15.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.561+0000 7f68b7000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68b00c1290 con 0x7f68b00c4960 2024-02-21T23:08:15.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.561+0000 7f68b7000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68b00a62e0 con 0x7f68b00c6220 2024-02-21T23:08:15.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.561+0000 7f68b5000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f68b0158bf0 0x7f68b015f9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:57380/0 (socket says 172.21.15.28:57380) 2024-02-21T23:08:15.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.561+0000 7f68b5000640 1 -- 172.21.15.28:0/1677076379 learned_addr learned my addr 172.21.15.28:0/1677076379 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:08:15.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.561+0000 7f68af400640 1 --2- 172.21.15.28:0/1677076379 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f68b00c6220 0x7f68b01586b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:15.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.561+0000 7f68aea00640 1 -- 172.21.15.28:0/1677076379 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1544130210 0 0) 0x7f68b00c20d0 con 0x7f68b0158bf0 2024-02-21T23:08:15.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.561+0000 7f68aea00640 1 -- 172.21.15.28:0/1677076379 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6894003520 con 0x7f68b0158bf0 2024-02-21T23:08:15.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.562+0000 7f68aea00640 1 -- 172.21.15.28:0/1677076379 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1920549765 0 0) 0x7f6894003520 con 0x7f68b0158bf0 2024-02-21T23:08:15.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.562+0000 7f68aea00640 1 -- 172.21.15.28:0/1677076379 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f68b00c6220 msgr2=0x7f68b01586b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:15.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.562+0000 7f68aea00640 1 --2- 172.21.15.28:0/1677076379 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f68b00c6220 0x7f68b01586b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:15.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.562+0000 7f68aea00640 1 -- 172.21.15.28:0/1677076379 >> v1:172.21.15.81:6789/0 conn(0x7f68b00c4960 legacy=0x7f68b01524f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:15.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.562+0000 7f68aea00640 1 -- 172.21.15.28:0/1677076379 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68b00c1570 con 0x7f68b0158bf0 2024-02-21T23:08:15.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.562+0000 7f68aea00640 1 -- 172.21.15.28:0/1677076379 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f68a80019f0 con 0x7f68b0158bf0 2024-02-21T23:08:15.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.562+0000 7f68aea00640 1 -- 172.21.15.28:0/1677076379 >> v1:172.21.15.28:6789/0 conn(0x7f68b0158bf0 legacy=0x7f68b015f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:15.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.562+0000 7f68aea00640 1 --2- 172.21.15.28:0/1677076379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68940039e0 0x7f6894003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:15.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.562+0000 7f68aea00640 1 --2- 172.21.15.28:0/1677076379 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f68b00c6220 0x7f6894004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:15.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.562+0000 7f68aea00640 1 --2- 172.21.15.28:0/1677076379 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6894004d40 0x7f68940051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:15.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.562+0000 7f68aea00640 1 -- 172.21.15.28:0/1677076379 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68b00a62e0 con 0x7f68940039e0 2024-02-21T23:08:15.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.562+0000 7f68aea00640 1 -- 172.21.15.28:0/1677076379 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f689400ba70 con 0x7f6894004d40 2024-02-21T23:08:15.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.562+0000 7f68aea00640 1 -- 172.21.15.28:0/1677076379 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f689400bc70 con 0x7f68b00c6220 2024-02-21T23:08:15.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.562+0000 7f68afe00640 1 --2- 172.21.15.28:0/1677076379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68940039e0 0x7f6894003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:15.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.563+0000 7f68af400640 1 --2- 172.21.15.28:0/1677076379 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f68b00c6220 0x7f6894004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:15.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.563+0000 7f68afe00640 1 -- 172.21.15.28:0/1677076379 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f68b00c6220 msgr2=0x7f6894004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:15.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.563+0000 7f68afe00640 1 --2- 172.21.15.28:0/1677076379 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f68b00c6220 0x7f6894004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:15.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.563+0000 7f68b5000640 1 --2- 172.21.15.28:0/1677076379 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6894004d40 0x7f68940051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:15.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.563+0000 7f68afe00640 1 -- 172.21.15.28:0/1677076379 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6894004d40 msgr2=0x7f68940051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:15.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.563+0000 7f68afe00640 1 --2- 172.21.15.28:0/1677076379 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6894004d40 0x7f68940051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:15.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.563+0000 7f68afe00640 1 -- 172.21.15.28:0/1677076379 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6894003520 con 0x7f68940039e0 2024-02-21T23:08:15.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.564+0000 7f68afe00640 1 --2- 172.21.15.28:0/1677076379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68940039e0 0x7f6894003dc0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f68a0001860 tx=0x7f6894005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3aa185db9a310509 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:15.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.564+0000 7f68aea00640 1 -- 172.21.15.28:0/1677076379 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68a002f050 con 0x7f68940039e0 2024-02-21T23:08:15.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.564+0000 7f68aea00640 1 -- 172.21.15.28:0/1677076379 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68a0033650 con 0x7f68940039e0 2024-02-21T23:08:15.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.564+0000 7f68aea00640 1 -- 172.21.15.28:0/1677076379 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68a0001430 con 0x7f68940039e0 2024-02-21T23:08:15.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.564+0000 7f68b7000640 1 -- 172.21.15.28:0/1677076379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68940039e0 msgr2=0x7f6894003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:15.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.564+0000 7f68b7000640 1 --2- 172.21.15.28:0/1677076379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68940039e0 0x7f6894003dc0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f68a0001860 tx=0x7f6894005830 comp rx=0 tx=0).stop 2024-02-21T23:08:15.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.564+0000 7f68b5000640 1 -- 172.21.15.28:0/1677076379 reap_dead start 2024-02-21T23:08:15.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.565+0000 7f68b7000640 1 -- 172.21.15.28:0/1677076379 shutdown_connections 2024-02-21T23:08:15.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.565+0000 7f68b7000640 1 -- 172.21.15.28:0/1677076379 >> 172.21.15.28:0/1677076379 conn(0x7f68b00a85a0 msgr2=0x7f68b00a8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:15.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.565+0000 7f68b7000640 1 -- 172.21.15.28:0/1677076379 shutdown_connections 2024-02-21T23:08:15.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.565+0000 7f68b7000640 1 -- 172.21.15.28:0/1677076379 wait complete. 2024-02-21T23:08:15.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.566+0000 7f68b7000640 1 Processor -- start 2024-02-21T23:08:15.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.567+0000 7f68b7000640 1 -- start start 2024-02-21T23:08:15.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.567+0000 7f68b7000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f68940039e0 0x7f68b014df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:15.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.567+0000 7f68b7000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f68b00aaec0 0x7f68b01eb060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:15.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.567+0000 7f68b7000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68b00a6780 0x7f68b00aa190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:15.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.567+0000 7f68b7000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68b00a5f20 con 0x7f68b00a6780 2024-02-21T23:08:15.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.567+0000 7f68b7000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68b00c1960 con 0x7f68b00aaec0 2024-02-21T23:08:15.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.567+0000 7f68b7000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68b00a60a0 con 0x7f68940039e0 2024-02-21T23:08:15.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.568+0000 7f68b5000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68b00a6780 0x7f68b00aa190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:15.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.568+0000 7f68b5000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68b00a6780 0x7f68b00aa190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57548/0 (socket says 172.21.15.28:57548) 2024-02-21T23:08:15.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.568+0000 7f68b5000640 1 -- 172.21.15.28:0/3029535513 learned_addr learned my addr 172.21.15.28:0/3029535513 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:15.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.568+0000 7f68afe00640 1 --2- 172.21.15.28:0/3029535513 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f68940039e0 0x7f68b014df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:15.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.568+0000 7f68af400640 1 --2- 172.21.15.28:0/3029535513 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f68b00aaec0 0x7f68b01eb060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:15.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.568+0000 7f68b5000640 1 -- 172.21.15.28:0/3029535513 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f68940039e0 msgr2=0x7f68b014df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:15.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.568+0000 7f68b5000640 1 --2- 172.21.15.28:0/3029535513 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f68940039e0 0x7f68b014df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:15.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.568+0000 7f68b5000640 1 -- 172.21.15.28:0/3029535513 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f68b00aaec0 msgr2=0x7f68b01eb060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:15.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.568+0000 7f68b5000640 1 --2- 172.21.15.28:0/3029535513 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f68b00aaec0 0x7f68b01eb060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:15.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.568+0000 7f68b5000640 1 -- 172.21.15.28:0/3029535513 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68b014db70 con 0x7f68b00a6780 2024-02-21T23:08:15.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.568+0000 7f68afe00640 1 --2- 172.21.15.28:0/3029535513 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f68940039e0 0x7f68b014df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:08:15.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.569+0000 7f68b5000640 1 --2- 172.21.15.28:0/3029535513 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68b00a6780 0x7f68b00aa190 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f68a8007c40 tx=0x7f68a800b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:15.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.569+0000 7f68a7e00640 1 -- 172.21.15.28:0/3029535513 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68a800bb20 con 0x7f68b00a6780 2024-02-21T23:08:15.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.569+0000 7f68a7e00640 1 -- 172.21.15.28:0/3029535513 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68a800bce0 con 0x7f68b00a6780 2024-02-21T23:08:15.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.569+0000 7f68a7e00640 1 -- 172.21.15.28:0/3029535513 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68a8007350 con 0x7f68b00a6780 2024-02-21T23:08:15.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.569+0000 7f68b7000640 1 -- 172.21.15.28:0/3029535513 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68b00ab910 con 0x7f68b00a6780 2024-02-21T23:08:15.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.569+0000 7f68b7000640 1 -- 172.21.15.28:0/3029535513 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68b00ab350 con 0x7f68b00a6780 2024-02-21T23:08:15.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.570+0000 7f68b7000640 1 -- 172.21.15.28:0/3029535513 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68b0067f30 con 0x7f68b00a6780 2024-02-21T23:08:15.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.571+0000 7f68a7e00640 1 -- 172.21.15.28:0/3029535513 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f68a801b060 con 0x7f68b00a6780 2024-02-21T23:08:15.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.572+0000 7f68a7e00640 1 --2- 172.21.15.28:0/3029535513 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f689006b4d0 0x7f689006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:15.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.572+0000 7f68a7e00640 1 -- 172.21.15.28:0/3029535513 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f68a808a3e0 con 0x7f68b00a6780 2024-02-21T23:08:15.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.573+0000 7f68afe00640 1 --2- 172.21.15.28:0/3029535513 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f689006b4d0 0x7f689006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:15.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.574+0000 7f68afe00640 1 --2- 172.21.15.28:0/3029535513 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f689006b4d0 0x7f689006d990 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f68a0036000 tx=0x7f68a0031000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:15.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.576+0000 7f68a7e00640 1 -- 172.21.15.28:0/3029535513 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f68a80590c0 con 0x7f68b00a6780 2024-02-21T23:08:15.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:15.818+0000 7f68b7000640 1 -- 172.21.15.28:0/3029535513 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f68b00a8dc0 con 0x7f68b00a6780 2024-02-21T23:08:16.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:16.461+0000 7f68a7e00640 1 -- 172.21.15.28:0/3029535513 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v120) v1 ==== 118+0+0 (secure 0 0 0) 0x7f68a80589c0 con 0x7f68b00a6780 2024-02-21T23:08:16.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:16.465+0000 7f68b7000640 1 -- 172.21.15.28:0/3029535513 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f689006b4d0 msgr2=0x7f689006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:16.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:16.465+0000 7f68b7000640 1 --2- 172.21.15.28:0/3029535513 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f689006b4d0 0x7f689006d990 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f68a0036000 tx=0x7f68a0031000 comp rx=0 tx=0).stop 2024-02-21T23:08:16.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:16.465+0000 7f68b7000640 1 -- 172.21.15.28:0/3029535513 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68b00a6780 msgr2=0x7f68b00aa190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:16.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:16.465+0000 7f68b7000640 1 --2- 172.21.15.28:0/3029535513 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68b00a6780 0x7f68b00aa190 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f68a8007c40 tx=0x7f68a800b4a0 comp rx=0 tx=0).stop 2024-02-21T23:08:16.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:16.465+0000 7f68b7000640 1 -- 172.21.15.28:0/3029535513 shutdown_connections 2024-02-21T23:08:16.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:16.465+0000 7f68b7000640 1 --2- 172.21.15.28:0/3029535513 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f689006b4d0 0x7f689006d990 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:16.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:16.465+0000 7f68b7000640 1 --2- 172.21.15.28:0/3029535513 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68b00a6780 0x7f68b00aa190 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:16.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:16.465+0000 7f68b7000640 1 --2- 172.21.15.28:0/3029535513 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f68b00aaec0 0x7f68b01eb060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:16.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:16.465+0000 7f68b7000640 1 --2- 172.21.15.28:0/3029535513 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f68940039e0 0x7f68b014df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:16.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:16.465+0000 7f68b7000640 1 -- 172.21.15.28:0/3029535513 >> 172.21.15.28:0/3029535513 conn(0x7f68b00a85a0 msgr2=0x7f68b006ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:16.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:16.465+0000 7f68b7000640 1 -- 172.21.15.28:0/3029535513 shutdown_connections 2024-02-21T23:08:16.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:16.465+0000 7f68b7000640 1 -- 172.21.15.28:0/3029535513 wait complete. 2024-02-21T23:08:16.475 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-02-21T23:08:16.476 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-02-21T23:08:16.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:16.713+0000 7f4df9000640 1 Processor -- start 2024-02-21T23:08:16.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:16.714+0000 7f4df9000640 1 -- start start 2024-02-21T23:08:16.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:16.715+0000 7f4df9000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4df4150240 0x7f4df40c12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:16.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:16.715+0000 7f4df9000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4df406e220 con 0x7f4df40c2c10 2024-02-21T23:08:16.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:16.715+0000 7f4df9000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4df406f780 con 0x7f4df40c4580 2024-02-21T23:08:16.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:16.715+0000 7f4df9000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4df4150620 con 0x7f4df4150240 2024-02-21T23:08:16.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:16.715+0000 7f4df3400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4df4150240 0x7f4df40c12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:16.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:16.715+0000 7f4df3e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f4df40c2c10 0x7f4df406ae70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:57396/0 (socket says 172.21.15.28:57396) 2024-02-21T23:08:16.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:16.715+0000 7f4df3e00640 1 -- 172.21.15.28:0/1996404030 learned_addr learned my addr 172.21.15.28:0/1996404030 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:08:16.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:16.716+0000 7f4df3400640 1 -- 172.21.15.28:0/1996404030 >> v1:172.21.15.81:6789/0 conn(0x7f4df40c4580 legacy=0x7f4df406db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:16.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:16.716+0000 7f4df3400640 1 -- 172.21.15.28:0/1996404030 >> v1:172.21.15.28:6789/0 conn(0x7f4df40c2c10 legacy=0x7f4df406ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:16.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:16.716+0000 7f4df3400640 1 -- 172.21.15.28:0/1996404030 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4df406f4a0 con 0x7f4df4150240 2024-02-21T23:08:16.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:16.717+0000 7f4df3400640 1 --2- 172.21.15.28:0/1996404030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df4150240 0x7f4df40c12c0 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f4de40097a0 tx=0x7f4de402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fa3c29511ea38d4 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:16.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:16.717+0000 7f4df2000640 1 -- 172.21.15.28:0/1996404030 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4de4035020 con 0x7f4df4150240 2024-02-21T23:08:16.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:16.717+0000 7f4df2000640 1 -- 172.21.15.28:0/1996404030 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4de4031830 con 0x7f4df4150240 2024-02-21T23:08:16.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:16.717+0000 7f4df2000640 1 -- 172.21.15.28:0/1996404030 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4de4031c00 con 0x7f4df4150240 2024-02-21T23:08:16.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:16.718+0000 7f4df9000640 1 -- 172.21.15.28:0/1996404030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df4150240 msgr2=0x7f4df40c12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:16.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:16.718+0000 7f4df9000640 1 --2- 172.21.15.28:0/1996404030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df4150240 0x7f4df40c12c0 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f4de40097a0 tx=0x7f4de402b8d0 comp rx=0 tx=0).stop 2024-02-21T23:08:16.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:16.718+0000 7f4df9000640 1 -- 172.21.15.28:0/1996404030 shutdown_connections 2024-02-21T23:08:16.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:16.718+0000 7f4df9000640 1 --2- 172.21.15.28:0/1996404030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df4150240 0x7f4df40c12c0 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:16.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:16.718+0000 7f4df9000640 1 -- 172.21.15.28:0/1996404030 >> 172.21.15.28:0/1996404030 conn(0x7f4df40a8530 msgr2=0x7f4df40a8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:16.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:16.718+0000 7f4df9000640 1 -- 172.21.15.28:0/1996404030 shutdown_connections 2024-02-21T23:08:16.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:16.718+0000 7f4df9000640 1 -- 172.21.15.28:0/1996404030 wait complete. 2024-02-21T23:08:16.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:16.720+0000 7f4df9000640 1 Processor -- start 2024-02-21T23:08:16.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:16.721+0000 7f4df9000640 1 -- start start 2024-02-21T23:08:16.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:16.722+0000 7f4df9000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df40c2c10 0x7f4df4147d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:16.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:16.722+0000 7f4df9000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4df40c4580 0x7f4df40bfbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:16.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:16.722+0000 7f4df9000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4df4150240 0x7f4df41eac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:16.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:16.722+0000 7f4df9000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4df40c4a10 con 0x7f4df40c2c10 2024-02-21T23:08:16.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:16.722+0000 7f4df9000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4df4153a90 con 0x7f4df40c4580 2024-02-21T23:08:16.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:16.722+0000 7f4df9000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4df40c4b90 con 0x7f4df4150240 2024-02-21T23:08:16.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:16.722+0000 7f4df3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df40c2c10 0x7f4df4147d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:16.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:16.722+0000 7f4df3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df40c2c10 0x7f4df4147d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57560/0 (socket says 172.21.15.28:57560) 2024-02-21T23:08:16.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:16.722+0000 7f4df3400640 1 -- 172.21.15.28:0/1639731680 learned_addr learned my addr 172.21.15.28:0/1639731680 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:16.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:16.722+0000 7f4df3400640 1 -- 172.21.15.28:0/1639731680 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4df4150240 msgr2=0x7f4df41eac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:16.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:16.722+0000 7f4df3400640 1 --2- 172.21.15.28:0/1639731680 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4df4150240 0x7f4df41eac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:16.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:16.722+0000 7f4df3400640 1 -- 172.21.15.28:0/1639731680 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4df40c4580 msgr2=0x7f4df40bfbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:16.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:16.722+0000 7f4df2a00640 1 --2- 172.21.15.28:0/1639731680 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4df40c4580 0x7f4df40bfbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:16.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:16.723+0000 7f4df3400640 1 --2- 172.21.15.28:0/1639731680 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4df40c4580 0x7f4df40bfbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:16.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:16.723+0000 7f4df3400640 1 -- 172.21.15.28:0/1639731680 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4df4071760 con 0x7f4df40c2c10 2024-02-21T23:08:16.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:16.723+0000 7f4df2a00640 1 --2- 172.21.15.28:0/1639731680 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4df40c4580 0x7f4df40bfbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:08:16.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:16.723+0000 7f4df3400640 1 --2- 172.21.15.28:0/1639731680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df40c2c10 0x7f4df4147d70 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f4de402bee0 tx=0x7f4de40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:16.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:16.723+0000 7f4deb400640 1 -- 172.21.15.28:0/1639731680 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4de4041070 con 0x7f4df40c2c10 2024-02-21T23:08:16.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:16.723+0000 7f4deb400640 1 -- 172.21.15.28:0/1639731680 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4de403a950 con 0x7f4df40c2c10 2024-02-21T23:08:16.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:16.723+0000 7f4deb400640 1 -- 172.21.15.28:0/1639731680 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4de4035040 con 0x7f4df40c2c10 2024-02-21T23:08:16.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:16.723+0000 7f4df9000640 1 -- 172.21.15.28:0/1639731680 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4df41ee260 con 0x7f4df40c2c10 2024-02-21T23:08:16.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:16.723+0000 7f4df9000640 1 -- 172.21.15.28:0/1639731680 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4df41ee820 con 0x7f4df40c2c10 2024-02-21T23:08:16.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:16.725+0000 7f4df9000640 1 -- 172.21.15.28:0/1639731680 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4df404f290 con 0x7f4df40c2c10 2024-02-21T23:08:16.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:16.725+0000 7f4deb400640 1 -- 172.21.15.28:0/1639731680 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4de404c060 con 0x7f4df40c2c10 2024-02-21T23:08:16.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:16.726+0000 7f4deb400640 1 --2- 172.21.15.28:0/1639731680 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4dd406b500 0x7f4dd406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:16.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:16.726+0000 7f4deb400640 1 -- 172.21.15.28:0/1639731680 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4de40abcf0 con 0x7f4df40c2c10 2024-02-21T23:08:16.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:16.727+0000 7f4df2a00640 1 --2- 172.21.15.28:0/1639731680 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4dd406b500 0x7f4dd406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:16.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:16.728+0000 7f4df2a00640 1 --2- 172.21.15.28:0/1639731680 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4dd406b500 0x7f4dd406d9c0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f4df4150e20 tx=0x7f4de0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:16.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:16.731+0000 7f4deb400640 1 -- 172.21.15.28:0/1639731680 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4de40b0070 con 0x7f4df40c2c10 2024-02-21T23:08:16.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:16.978+0000 7f4df9000640 1 -- 172.21.15.28:0/1639731680 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f4df406f0e0 con 0x7f4df40c2c10 2024-02-21T23:08:17.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.464+0000 7f4deb400640 1 -- 172.21.15.28:0/1639731680 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v121) v1 ==== 122+0+0 (secure 0 0 0) 0x7f4de407aa70 con 0x7f4df40c2c10 2024-02-21T23:08:17.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.467+0000 7f4df9000640 1 -- 172.21.15.28:0/1639731680 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4dd406b500 msgr2=0x7f4dd406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:17.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.467+0000 7f4df9000640 1 --2- 172.21.15.28:0/1639731680 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4dd406b500 0x7f4dd406d9c0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f4df4150e20 tx=0x7f4de0009040 comp rx=0 tx=0).stop 2024-02-21T23:08:17.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.467+0000 7f4df9000640 1 -- 172.21.15.28:0/1639731680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df40c2c10 msgr2=0x7f4df4147d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:17.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.467+0000 7f4df9000640 1 --2- 172.21.15.28:0/1639731680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df40c2c10 0x7f4df4147d70 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f4de402bee0 tx=0x7f4de40046c0 comp rx=0 tx=0).stop 2024-02-21T23:08:17.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.468+0000 7f4df9000640 1 -- 172.21.15.28:0/1639731680 shutdown_connections 2024-02-21T23:08:17.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.468+0000 7f4df9000640 1 --2- 172.21.15.28:0/1639731680 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4dd406b500 0x7f4dd406d9c0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:17.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.468+0000 7f4df9000640 1 --2- 172.21.15.28:0/1639731680 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4df4150240 0x7f4df41eac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:17.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.468+0000 7f4df9000640 1 --2- 172.21.15.28:0/1639731680 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4df40c4580 0x7f4df40bfbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:17.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.468+0000 7f4df9000640 1 --2- 172.21.15.28:0/1639731680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df40c2c10 0x7f4df4147d70 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:17.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.468+0000 7f4df9000640 1 -- 172.21.15.28:0/1639731680 >> 172.21.15.28:0/1639731680 conn(0x7f4df40a8530 msgr2=0x7f4df414e400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:17.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.468+0000 7f4df9000640 1 -- 172.21.15.28:0/1639731680 shutdown_connections 2024-02-21T23:08:17.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.468+0000 7f4df9000640 1 -- 172.21.15.28:0/1639731680 wait complete. 2024-02-21T23:08:17.479 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-02-21T23:08:17.479 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:08:17.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.712+0000 7f0719200640 1 Processor -- start 2024-02-21T23:08:17.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.713+0000 7f0719200640 1 -- start start 2024-02-21T23:08:17.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.714+0000 7f0719200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f071406ee40 0x7f07140a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:17.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.714+0000 7f0719200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07140c2760 con 0x7f0714150450 2024-02-21T23:08:17.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.714+0000 7f0719200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0714156310 con 0x7f0714156af0 2024-02-21T23:08:17.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.714+0000 7f0719200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07140c22c0 con 0x7f071406ee40 2024-02-21T23:08:17.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.714+0000 7f0713400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f0714150450 0x7f07141536f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:57400/0 (socket says 172.21.15.28:57400) 2024-02-21T23:08:17.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.714+0000 7f0713400640 1 -- 172.21.15.28:0/25526807 learned_addr learned my addr 172.21.15.28:0/25526807 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:08:17.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.715+0000 7f0712000640 1 -- 172.21.15.28:0/25526807 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4257162127 0 0) 0x7f07140c2760 con 0x7f0714150450 2024-02-21T23:08:17.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.715+0000 7f0712a00640 1 --2- 172.21.15.28:0/25526807 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f071406ee40 0x7f07140a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:17.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.715+0000 7f0712000640 1 -- 172.21.15.28:0/25526807 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06f8003520 con 0x7f0714150450 2024-02-21T23:08:17.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.715+0000 7f0712000640 1 -- 172.21.15.28:0/25526807 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3699703053 0 0) 0x7f06f8003520 con 0x7f0714150450 2024-02-21T23:08:17.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.715+0000 7f0712000640 1 -- 172.21.15.28:0/25526807 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f071406ee40 msgr2=0x7f07140a5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:17.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.715+0000 7f0712000640 1 --2- 172.21.15.28:0/25526807 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f071406ee40 0x7f07140a5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:17.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.715+0000 7f0712000640 1 -- 172.21.15.28:0/25526807 >> v1:172.21.15.81:6789/0 conn(0x7f0714156af0 legacy=0x7f071406dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:17.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.715+0000 7f0712000640 1 -- 172.21.15.28:0/25526807 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07140c29d0 con 0x7f0714150450 2024-02-21T23:08:17.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.715+0000 7f0712000640 1 -- 172.21.15.28:0/25526807 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f0704001670 con 0x7f0714150450 2024-02-21T23:08:17.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.715+0000 7f0712000640 1 -- 172.21.15.28:0/25526807 >> v1:172.21.15.28:6789/0 conn(0x7f0714150450 legacy=0x7f07141536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:17.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.715+0000 7f0712000640 1 --2- 172.21.15.28:0/25526807 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f071406ee40 0x7f06f80039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:17.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.715+0000 7f0712000640 1 --2- 172.21.15.28:0/25526807 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06f8004010 0x7f06f8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:17.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.715+0000 7f0712000640 1 --2- 172.21.15.28:0/25526807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06f8004d40 0x7f06f80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:17.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.715+0000 7f0712000640 1 -- 172.21.15.28:0/25526807 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f800bb50 con 0x7f06f8004d40 2024-02-21T23:08:17.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.715+0000 7f0712000640 1 -- 172.21.15.28:0/25526807 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f800bd00 con 0x7f06f8004010 2024-02-21T23:08:17.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.715+0000 7f0712000640 1 -- 172.21.15.28:0/25526807 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06f800bf00 con 0x7f071406ee40 2024-02-21T23:08:17.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.716+0000 7f0713e00640 1 --2- 172.21.15.28:0/25526807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06f8004d40 0x7f06f80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:17.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.716+0000 7f0713400640 1 --2- 172.21.15.28:0/25526807 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f071406ee40 0x7f06f80039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:17.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.716+0000 7f0712a00640 1 --2- 172.21.15.28:0/25526807 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06f8004010 0x7f06f8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:17.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.716+0000 7f0713e00640 1 -- 172.21.15.28:0/25526807 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f071406ee40 msgr2=0x7f06f80039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:17.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.716+0000 7f0713e00640 1 --2- 172.21.15.28:0/25526807 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f071406ee40 0x7f06f80039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:17.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.716+0000 7f0713e00640 1 -- 172.21.15.28:0/25526807 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06f8004010 msgr2=0x7f06f8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:17.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.716+0000 7f0713e00640 1 --2- 172.21.15.28:0/25526807 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f06f8004010 0x7f06f8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:17.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.716+0000 7f0713e00640 1 -- 172.21.15.28:0/25526807 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f8003520 con 0x7f06f8004d40 2024-02-21T23:08:17.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.717+0000 7f0713e00640 1 --2- 172.21.15.28:0/25526807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06f8004d40 0x7f06f80051c0 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f070c001e50 tx=0x7f06f8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccfc078b38b9b1ee server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:17.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.718+0000 7f0712000640 1 -- 172.21.15.28:0/25526807 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f070c001a70 con 0x7f06f8004d40 2024-02-21T23:08:17.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.718+0000 7f0712000640 1 -- 172.21.15.28:0/25526807 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f070c003b70 con 0x7f06f8004d40 2024-02-21T23:08:17.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.718+0000 7f0712000640 1 -- 172.21.15.28:0/25526807 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f070c0050d0 con 0x7f06f8004d40 2024-02-21T23:08:17.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.718+0000 7f0719200640 1 -- 172.21.15.28:0/25526807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06f8004d40 msgr2=0x7f06f80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:17.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.718+0000 7f0719200640 1 --2- 172.21.15.28:0/25526807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06f8004d40 0x7f06f80051c0 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f070c001e50 tx=0x7f06f8005830 comp rx=0 tx=0).stop 2024-02-21T23:08:17.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.718+0000 7f0713e00640 1 -- 172.21.15.28:0/25526807 reap_dead start 2024-02-21T23:08:17.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.718+0000 7f0719200640 1 -- 172.21.15.28:0/25526807 shutdown_connections 2024-02-21T23:08:17.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.718+0000 7f0719200640 1 -- 172.21.15.28:0/25526807 >> 172.21.15.28:0/25526807 conn(0x7f071406b630 msgr2=0x7f07140c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:17.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.718+0000 7f0719200640 1 -- 172.21.15.28:0/25526807 shutdown_connections 2024-02-21T23:08:17.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.718+0000 7f0719200640 1 -- 172.21.15.28:0/25526807 wait complete. 2024-02-21T23:08:17.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.720+0000 7f0719200640 1 Processor -- start 2024-02-21T23:08:17.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.720+0000 7f0719200640 1 -- start start 2024-02-21T23:08:17.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.721+0000 7f0719200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f06f8004d40 0x7f0714155350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:17.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.721+0000 7f0719200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0714148040 0x7f07141e94b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:17.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.721+0000 7f0719200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f07141e99f0 0x7f0714143820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:17.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.721+0000 7f0719200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f071406e700 con 0x7f0714148040 2024-02-21T23:08:17.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.721+0000 7f0719200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0714156660 con 0x7f07141e99f0 2024-02-21T23:08:17.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.721+0000 7f0719200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f071406e880 con 0x7f06f8004d40 2024-02-21T23:08:17.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.721+0000 7f0712a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0714148040 0x7f07141e94b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:17.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.721+0000 7f0712a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0714148040 0x7f07141e94b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57576/0 (socket says 172.21.15.28:57576) 2024-02-21T23:08:17.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.721+0000 7f0712a00640 1 -- 172.21.15.28:0/1014127918 learned_addr learned my addr 172.21.15.28:0/1014127918 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:17.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.722+0000 7f0713e00640 1 --2- 172.21.15.28:0/1014127918 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f07141e99f0 0x7f0714143820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:17.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.722+0000 7f0713400640 1 --2- 172.21.15.28:0/1014127918 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f06f8004d40 0x7f0714155350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:17.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.722+0000 7f0712a00640 1 -- 172.21.15.28:0/1014127918 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f06f8004d40 msgr2=0x7f0714155350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:17.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.722+0000 7f0712a00640 1 --2- 172.21.15.28:0/1014127918 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f06f8004d40 0x7f0714155350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:17.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.722+0000 7f0712a00640 1 -- 172.21.15.28:0/1014127918 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f07141e99f0 msgr2=0x7f0714143820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:17.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.722+0000 7f0712a00640 1 --2- 172.21.15.28:0/1014127918 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f07141e99f0 0x7f0714143820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:17.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.722+0000 7f0712a00640 1 -- 172.21.15.28:0/1014127918 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0714072840 con 0x7f0714148040 2024-02-21T23:08:17.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.722+0000 7f0713e00640 1 --2- 172.21.15.28:0/1014127918 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f07141e99f0 0x7f0714143820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:08:17.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.722+0000 7f0712a00640 1 --2- 172.21.15.28:0/1014127918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0714148040 0x7f07141e94b0 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f070000f5e0 tx=0x7f070000faa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:17.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.723+0000 7f070b400640 1 -- 172.21.15.28:0/1014127918 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0700019020 con 0x7f0714148040 2024-02-21T23:08:17.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.723+0000 7f070b400640 1 -- 172.21.15.28:0/1014127918 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0700014870 con 0x7f0714148040 2024-02-21T23:08:17.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.723+0000 7f070b400640 1 -- 172.21.15.28:0/1014127918 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0700014c60 con 0x7f0714148040 2024-02-21T23:08:17.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.723+0000 7f0719200640 1 -- 172.21.15.28:0/1014127918 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f071414e780 con 0x7f0714148040 2024-02-21T23:08:17.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.723+0000 7f0719200640 1 -- 172.21.15.28:0/1014127918 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0714072dc0 con 0x7f0714148040 2024-02-21T23:08:17.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.724+0000 7f0719200640 1 -- 172.21.15.28:0/1014127918 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f071404f290 con 0x7f0714148040 2024-02-21T23:08:17.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.725+0000 7f070b400640 1 -- 172.21.15.28:0/1014127918 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f070001c020 con 0x7f0714148040 2024-02-21T23:08:17.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.726+0000 7f070b400640 1 --2- 172.21.15.28:0/1014127918 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f06f406b500 0x7f06f406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:17.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.726+0000 7f070b400640 1 -- 172.21.15.28:0/1014127918 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f070008ab30 con 0x7f0714148040 2024-02-21T23:08:17.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.726+0000 7f0713400640 1 --2- 172.21.15.28:0/1014127918 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f06f406b500 0x7f06f406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:17.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.728+0000 7f0713400640 1 --2- 172.21.15.28:0/1014127918 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f06f406b500 0x7f06f406d9c0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f0704007cb0 tx=0x7f0704009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:17.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.731+0000 7f070b400640 1 -- 172.21.15.28:0/1014127918 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f0700023020 con 0x7f0714148040 2024-02-21T23:08:17.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.970+0000 7f0719200640 1 -- 172.21.15.28:0/1014127918 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0714147610 con 0x7f0714148040 2024-02-21T23:08:17.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.971+0000 7f070b400640 1 -- 172.21.15.28:0/1014127918 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) v1 ==== 95+0+4946 (secure 0 0 0) 0x7f0700059810 con 0x7f0714148040 2024-02-21T23:08:17.959 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:08:17.960 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":121,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5948,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3556518453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3556518453},{"type":"v1","addr":"172.21.15.28:6835","nonce":3556518453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116},{"gid":5951,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/3293264689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":3293264689},{"type":"v1","addr":"172.21.15.81:6836","nonce":3293264689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116},{"gid":7905,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/1773560593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1773560593},{"type":"v1","addr":"172.21.15.28:6834","nonce":1773560593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":115}],"filesystems":[{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:08:13.883141+0000","modified":"2024-02-21T23:08:16.980266+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7914},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7914":{"gid":7914,"name":"d","rank":0,"incarnation":117,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6837/1038096496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":1038096496},{"type":"v1","addr":"172.21.15.81:6837","nonce":1038096496}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-21T23:08:17.960 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 121 2024-02-21T23:08:17.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.973+0000 7f0719200640 1 -- 172.21.15.28:0/1014127918 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f06f406b500 msgr2=0x7f06f406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:17.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.973+0000 7f0719200640 1 --2- 172.21.15.28:0/1014127918 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f06f406b500 0x7f06f406d9c0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f0704007cb0 tx=0x7f0704009040 comp rx=0 tx=0).stop 2024-02-21T23:08:17.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.974+0000 7f0719200640 1 -- 172.21.15.28:0/1014127918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0714148040 msgr2=0x7f07141e94b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:17.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.974+0000 7f0719200640 1 --2- 172.21.15.28:0/1014127918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0714148040 0x7f07141e94b0 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f070000f5e0 tx=0x7f070000faa0 comp rx=0 tx=0).stop 2024-02-21T23:08:17.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.974+0000 7f0719200640 1 -- 172.21.15.28:0/1014127918 shutdown_connections 2024-02-21T23:08:17.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.974+0000 7f0719200640 1 --2- 172.21.15.28:0/1014127918 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f06f406b500 0x7f06f406d9c0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:17.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.974+0000 7f0719200640 1 --2- 172.21.15.28:0/1014127918 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f07141e99f0 0x7f0714143820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:17.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.974+0000 7f0719200640 1 --2- 172.21.15.28:0/1014127918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0714148040 0x7f07141e94b0 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:17.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.974+0000 7f0719200640 1 --2- 172.21.15.28:0/1014127918 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f06f8004d40 0x7f0714155350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:17.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.974+0000 7f0719200640 1 -- 172.21.15.28:0/1014127918 >> 172.21.15.28:0/1014127918 conn(0x7f071406b630 msgr2=0x7f0714149fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:17.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.974+0000 7f0719200640 1 -- 172.21.15.28:0/1014127918 shutdown_connections 2024-02-21T23:08:17.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:17.974+0000 7f0719200640 1 -- 172.21.15.28:0/1014127918 wait complete. 2024-02-21T23:08:17.982 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:08:18.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.209+0000 7fcf9ca00640 1 Processor -- start 2024-02-21T23:08:18.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.210+0000 7fcf9ca00640 1 -- start start 2024-02-21T23:08:18.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.211+0000 7fcf9ca00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcf980c11f0 0x7fcf98153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:18.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.211+0000 7fcf9ca00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcf980c15d0 0x7fcf9806dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:18.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.211+0000 7fcf9ca00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf980c1a30 con 0x7fcf98156350 2024-02-21T23:08:18.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.211+0000 7fcf9ca00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf980a5ec0 con 0x7fcf980c15d0 2024-02-21T23:08:18.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.211+0000 7fcf9ca00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf98152e70 con 0x7fcf980c11f0 2024-02-21T23:08:18.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.211+0000 7fcf96800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fcf98156350 0x7fcf981567a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:57414/0 (socket says 172.21.15.28:57414) 2024-02-21T23:08:18.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.212+0000 7fcf96800640 1 -- 172.21.15.28:0/1368882009 learned_addr learned my addr 172.21.15.28:0/1368882009 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:08:18.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.212+0000 7fcf95400640 1 --2- 172.21.15.28:0/1368882009 >> v2:172.21.15.28:3300/0 conn(0x7fcf980c15d0 0x7fcf9806dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:18.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.212+0000 7fcf95e00640 1 --2- 172.21.15.28:0/1368882009 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcf980c11f0 0x7fcf98153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:18.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.212+0000 7fcf94a00640 1 -- 172.21.15.28:0/1368882009 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3689074031 0 0) 0x7fcf980c1a30 con 0x7fcf98156350 2024-02-21T23:08:18.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.212+0000 7fcf94a00640 1 -- 172.21.15.28:0/1368882009 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf7c003520 con 0x7fcf98156350 2024-02-21T23:08:18.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.212+0000 7fcf95400640 1 -- 172.21.15.28:0/1368882009 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcf980c11f0 msgr2=0x7fcf98153950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:18.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.212+0000 7fcf95400640 1 --2- 172.21.15.28:0/1368882009 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcf980c11f0 0x7fcf98153950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:18.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.212+0000 7fcf95400640 1 -- 172.21.15.28:0/1368882009 >> v1:172.21.15.28:6789/0 conn(0x7fcf98156350 legacy=0x7fcf981567a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:18.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.212+0000 7fcf95400640 1 -- 172.21.15.28:0/1368882009 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf98153150 con 0x7fcf980c15d0 2024-02-21T23:08:18.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.213+0000 7fcf95400640 1 --2- 172.21.15.28:0/1368882009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf980c15d0 0x7fcf9806dce0 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7fcf88009d90 tx=0x7fcf8802bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dee89a9cf65e7d85 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:18.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.214+0000 7fcf94a00640 1 -- 172.21.15.28:0/1368882009 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf88039070 con 0x7fcf980c15d0 2024-02-21T23:08:18.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.214+0000 7fcf94a00640 1 -- 172.21.15.28:0/1368882009 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf8802cb40 con 0x7fcf980c15d0 2024-02-21T23:08:18.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.214+0000 7fcf94a00640 1 -- 172.21.15.28:0/1368882009 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf88034440 con 0x7fcf980c15d0 2024-02-21T23:08:18.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.214+0000 7fcf9ca00640 1 -- 172.21.15.28:0/1368882009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf980c15d0 msgr2=0x7fcf9806dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:18.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.214+0000 7fcf9ca00640 1 --2- 172.21.15.28:0/1368882009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf980c15d0 0x7fcf9806dce0 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7fcf88009d90 tx=0x7fcf8802bd90 comp rx=0 tx=0).stop 2024-02-21T23:08:18.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.215+0000 7fcf9ca00640 1 -- 172.21.15.28:0/1368882009 shutdown_connections 2024-02-21T23:08:18.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.215+0000 7fcf9ca00640 1 --2- 172.21.15.28:0/1368882009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf980c15d0 0x7fcf9806dce0 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:18.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.215+0000 7fcf9ca00640 1 --2- 172.21.15.28:0/1368882009 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcf980c11f0 0x7fcf98153950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:18.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.215+0000 7fcf9ca00640 1 -- 172.21.15.28:0/1368882009 >> 172.21.15.28:0/1368882009 conn(0x7fcf980c67f0 msgr2=0x7fcf980c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:18.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.215+0000 7fcf9ca00640 1 -- 172.21.15.28:0/1368882009 shutdown_connections 2024-02-21T23:08:18.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.215+0000 7fcf9ca00640 1 -- 172.21.15.28:0/1368882009 wait complete. 2024-02-21T23:08:18.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.216+0000 7fcf9ca00640 1 Processor -- start 2024-02-21T23:08:18.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.217+0000 7fcf9ca00640 1 -- start start 2024-02-21T23:08:18.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.217+0000 7fcf9ca00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcf980c11f0 0x7fcf980a7a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:18.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.217+0000 7fcf9ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf980c15d0 0x7fcf98145390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:18.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.217+0000 7fcf9ca00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf98156350 0x7fcf981515b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:18.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.217+0000 7fcf9ca00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf980c4c00 con 0x7fcf980c15d0 2024-02-21T23:08:18.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.217+0000 7fcf9ca00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9814d1c0 con 0x7fcf98156350 2024-02-21T23:08:18.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.217+0000 7fcf9ca00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf980a6110 con 0x7fcf980c11f0 2024-02-21T23:08:18.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.218+0000 7fcf95400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf980c15d0 0x7fcf98145390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:18.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.218+0000 7fcf95400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf980c15d0 0x7fcf98145390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57592/0 (socket says 172.21.15.28:57592) 2024-02-21T23:08:18.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.218+0000 7fcf95400640 1 -- 172.21.15.28:0/1507155677 learned_addr learned my addr 172.21.15.28:0/1507155677 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:18.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.218+0000 7fcf95400640 1 -- 172.21.15.28:0/1507155677 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcf980c11f0 msgr2=0x7fcf980a7a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:18.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.218+0000 7fcf95e00640 1 --2- 172.21.15.28:0/1507155677 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcf980c11f0 0x7fcf980a7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:18.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.218+0000 7fcf96800640 1 --2- 172.21.15.28:0/1507155677 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf98156350 0x7fcf981515b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:18.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.218+0000 7fcf95400640 1 --2- 172.21.15.28:0/1507155677 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcf980c11f0 0x7fcf980a7a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:18.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.218+0000 7fcf95400640 1 -- 172.21.15.28:0/1507155677 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf98156350 msgr2=0x7fcf981515b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:18.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.218+0000 7fcf95400640 1 --2- 172.21.15.28:0/1507155677 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf98156350 0x7fcf981515b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:18.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.218+0000 7fcf95400640 1 -- 172.21.15.28:0/1507155677 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf98150d00 con 0x7fcf980c15d0 2024-02-21T23:08:18.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.219+0000 7fcf95400640 1 --2- 172.21.15.28:0/1507155677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf980c15d0 0x7fcf98145390 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7fcf88009d90 tx=0x7fcf88004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:18.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.219+0000 7fcf8e000640 1 -- 172.21.15.28:0/1507155677 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf88039050 con 0x7fcf980c15d0 2024-02-21T23:08:18.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.219+0000 7fcf8e000640 1 -- 172.21.15.28:0/1507155677 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf88045070 con 0x7fcf980c15d0 2024-02-21T23:08:18.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.219+0000 7fcf8e000640 1 -- 172.21.15.28:0/1507155677 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf8803e850 con 0x7fcf980c15d0 2024-02-21T23:08:18.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.219+0000 7fcf9ca00640 1 -- 172.21.15.28:0/1507155677 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf98150fe0 con 0x7fcf980c15d0 2024-02-21T23:08:18.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.219+0000 7fcf9ca00640 1 -- 172.21.15.28:0/1507155677 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf98141c20 con 0x7fcf980c15d0 2024-02-21T23:08:18.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.220+0000 7fcf9ca00640 1 -- 172.21.15.28:0/1507155677 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf980a7340 con 0x7fcf980c15d0 2024-02-21T23:08:18.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.221+0000 7fcf8e000640 1 -- 172.21.15.28:0/1507155677 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcf8803e9f0 con 0x7fcf980c15d0 2024-02-21T23:08:18.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.223+0000 7fcf8e000640 1 --2- 172.21.15.28:0/1507155677 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcf7406b4d0 0x7fcf7406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:18.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.223+0000 7fcf8e000640 1 -- 172.21.15.28:0/1507155677 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fcf880acff0 con 0x7fcf980c15d0 2024-02-21T23:08:18.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.223+0000 7fcf95e00640 1 --2- 172.21.15.28:0/1507155677 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcf7406b4d0 0x7fcf7406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:18.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.225+0000 7fcf95e00640 1 --2- 172.21.15.28:0/1507155677 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcf7406b4d0 0x7fcf7406d990 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fcf981449d0 tx=0x7fcf80008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:18.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.228+0000 7fcf8e000640 1 -- 172.21.15.28:0/1507155677 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fcf8807bed0 con 0x7fcf980c15d0 2024-02-21T23:08:18.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.426+0000 7fcf9ca00640 1 -- 172.21.15.28:0/1507155677 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcf98067f30 con 0x7fcf980c15d0 2024-02-21T23:08:18.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.430+0000 7fcf8e000640 1 -- 172.21.15.28:0/1507155677 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v122) v1 ==== 74+0+16840 (secure 0 0 0) 0x7fcf8807b7d0 con 0x7fcf980c15d0 2024-02-21T23:08:18.418 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:08:18.418 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":122,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:08:15.416721+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:08:11.794800+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"121","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2024-02-21T23:08:12.918602+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"122","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:08:18.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.432+0000 7fcf9ca00640 1 -- 172.21.15.28:0/1507155677 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcf7406b4d0 msgr2=0x7fcf7406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:18.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.432+0000 7fcf9ca00640 1 --2- 172.21.15.28:0/1507155677 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcf7406b4d0 0x7fcf7406d990 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fcf981449d0 tx=0x7fcf80008040 comp rx=0 tx=0).stop 2024-02-21T23:08:18.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.432+0000 7fcf9ca00640 1 -- 172.21.15.28:0/1507155677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf980c15d0 msgr2=0x7fcf98145390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:18.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.432+0000 7fcf9ca00640 1 --2- 172.21.15.28:0/1507155677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf980c15d0 0x7fcf98145390 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7fcf88009d90 tx=0x7fcf88004a40 comp rx=0 tx=0).stop 2024-02-21T23:08:18.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.433+0000 7fcf9ca00640 1 -- 172.21.15.28:0/1507155677 shutdown_connections 2024-02-21T23:08:18.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.433+0000 7fcf9ca00640 1 --2- 172.21.15.28:0/1507155677 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcf7406b4d0 0x7fcf7406d990 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:18.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.433+0000 7fcf9ca00640 1 --2- 172.21.15.28:0/1507155677 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf98156350 0x7fcf981515b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:18.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.433+0000 7fcf9ca00640 1 --2- 172.21.15.28:0/1507155677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf980c15d0 0x7fcf98145390 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:18.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.433+0000 7fcf9ca00640 1 --2- 172.21.15.28:0/1507155677 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcf980c11f0 0x7fcf980a7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:18.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.433+0000 7fcf9ca00640 1 -- 172.21.15.28:0/1507155677 >> 172.21.15.28:0/1507155677 conn(0x7fcf980c67f0 msgr2=0x7fcf98153710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:18.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.433+0000 7fcf9ca00640 1 -- 172.21.15.28:0/1507155677 shutdown_connections 2024-02-21T23:08:18.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.433+0000 7fcf9ca00640 1 -- 172.21.15.28:0/1507155677 wait complete. 2024-02-21T23:08:18.439 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-02-21T23:08:18.439 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-21T23:08:18.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.665+0000 7f95de800640 1 Processor -- start 2024-02-21T23:08:18.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.665+0000 7f95de800640 1 -- start start 2024-02-21T23:08:18.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.667+0000 7f95de800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f95d806ee40 0x7f95d80a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:18.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.667+0000 7f95de800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95d80c2760 con 0x7f95d8156af0 2024-02-21T23:08:18.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.667+0000 7f95de800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95d8156310 con 0x7f95d8150450 2024-02-21T23:08:18.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.667+0000 7f95de800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95d80c22c0 con 0x7f95d806ee40 2024-02-21T23:08:18.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.667+0000 7f95dde00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f95d8156af0 0x7f95d806dff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:57424/0 (socket says 172.21.15.28:57424) 2024-02-21T23:08:18.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.667+0000 7f95dde00640 1 -- 172.21.15.28:0/3536807649 learned_addr learned my addr 172.21.15.28:0/3536807649 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:08:18.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.667+0000 7f95dca00640 1 --2- 172.21.15.28:0/3536807649 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f95d806ee40 0x7f95d80a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:18.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.668+0000 7f95d7e00640 1 -- 172.21.15.28:0/3536807649 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1964378147 0 0) 0x7f95d80c2760 con 0x7f95d8156af0 2024-02-21T23:08:18.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.668+0000 7f95d7e00640 1 -- 172.21.15.28:0/3536807649 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95c0003520 con 0x7f95d8156af0 2024-02-21T23:08:18.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.668+0000 7f95d7e00640 1 -- 172.21.15.28:0/3536807649 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2495631957 0 0) 0x7f95c0003520 con 0x7f95d8156af0 2024-02-21T23:08:18.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.668+0000 7f95d7e00640 1 -- 172.21.15.28:0/3536807649 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f95d806ee40 msgr2=0x7f95d80a5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:18.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.668+0000 7f95d7e00640 1 --2- 172.21.15.28:0/3536807649 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f95d806ee40 0x7f95d80a5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:18.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.668+0000 7f95d7e00640 1 -- 172.21.15.28:0/3536807649 >> v1:172.21.15.81:6789/0 conn(0x7f95d8150450 legacy=0x7f95d81536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:18.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.668+0000 7f95d7e00640 1 -- 172.21.15.28:0/3536807649 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95d80c29d0 con 0x7f95d8156af0 2024-02-21T23:08:18.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.668+0000 7f95d7e00640 1 -- 172.21.15.28:0/3536807649 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f95d00019f0 con 0x7f95d8156af0 2024-02-21T23:08:18.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.668+0000 7f95d7e00640 1 -- 172.21.15.28:0/3536807649 >> v1:172.21.15.28:6789/0 conn(0x7f95d8156af0 legacy=0x7f95d806dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:18.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.668+0000 7f95d7e00640 1 --2- 172.21.15.28:0/3536807649 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95c00039e0 0x7f95c0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:18.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.668+0000 7f95d7e00640 1 --2- 172.21.15.28:0/3536807649 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95c0004800 0x7f95c0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:18.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.668+0000 7f95d7e00640 1 --2- 172.21.15.28:0/3536807649 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f95d806ee40 0x7f95c00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:18.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.668+0000 7f95d7e00640 1 -- 172.21.15.28:0/3536807649 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95c000ba70 con 0x7f95c00039e0 2024-02-21T23:08:18.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.668+0000 7f95d7e00640 1 -- 172.21.15.28:0/3536807649 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95c000bbf0 con 0x7f95c0004800 2024-02-21T23:08:18.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.668+0000 7f95d7e00640 1 -- 172.21.15.28:0/3536807649 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95c000bdf0 con 0x7f95d806ee40 2024-02-21T23:08:18.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.669+0000 7f95dd400640 1 --2- 172.21.15.28:0/3536807649 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95c00039e0 0x7f95c0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:18.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.669+0000 7f95dca00640 1 --2- 172.21.15.28:0/3536807649 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95c0004800 0x7f95c0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:18.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.669+0000 7f95dde00640 1 --2- 172.21.15.28:0/3536807649 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f95d806ee40 0x7f95c00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:18.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.669+0000 7f95dd400640 1 -- 172.21.15.28:0/3536807649 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f95d806ee40 msgr2=0x7f95c00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:18.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.669+0000 7f95dd400640 1 --2- 172.21.15.28:0/3536807649 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f95d806ee40 0x7f95c00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:18.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.669+0000 7f95dd400640 1 -- 172.21.15.28:0/3536807649 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95c0004800 msgr2=0x7f95c0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:18.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.669+0000 7f95dd400640 1 --2- 172.21.15.28:0/3536807649 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95c0004800 0x7f95c0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:18.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.669+0000 7f95dd400640 1 -- 172.21.15.28:0/3536807649 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95c0003520 con 0x7f95c00039e0 2024-02-21T23:08:18.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.670+0000 7f95dd400640 1 --2- 172.21.15.28:0/3536807649 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95c00039e0 0x7f95c0003e00 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f95cc001720 tx=0x7f95c0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ce454b8d15ce06b server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:18.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.671+0000 7f95d7e00640 1 -- 172.21.15.28:0/3536807649 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95cc001370 con 0x7f95c00039e0 2024-02-21T23:08:18.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.671+0000 7f95d7e00640 1 -- 172.21.15.28:0/3536807649 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95cc0052d0 con 0x7f95c00039e0 2024-02-21T23:08:18.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.671+0000 7f95d7e00640 1 -- 172.21.15.28:0/3536807649 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95cc003830 con 0x7f95c00039e0 2024-02-21T23:08:18.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.671+0000 7f95de800640 1 -- 172.21.15.28:0/3536807649 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95c00039e0 msgr2=0x7f95c0003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:18.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.671+0000 7f95de800640 1 --2- 172.21.15.28:0/3536807649 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95c00039e0 0x7f95c0003e00 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f95cc001720 tx=0x7f95c0005830 comp rx=0 tx=0).stop 2024-02-21T23:08:18.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.671+0000 7f95dde00640 1 -- 172.21.15.28:0/3536807649 reap_dead start 2024-02-21T23:08:18.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.671+0000 7f95de800640 1 -- 172.21.15.28:0/3536807649 shutdown_connections 2024-02-21T23:08:18.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.671+0000 7f95de800640 1 -- 172.21.15.28:0/3536807649 >> 172.21.15.28:0/3536807649 conn(0x7f95d806b630 msgr2=0x7f95d80c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:18.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.671+0000 7f95de800640 1 -- 172.21.15.28:0/3536807649 shutdown_connections 2024-02-21T23:08:18.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.672+0000 7f95de800640 1 -- 172.21.15.28:0/3536807649 wait complete. 2024-02-21T23:08:18.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.673+0000 7f95de800640 1 Processor -- start 2024-02-21T23:08:18.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.673+0000 7f95de800640 1 -- start start 2024-02-21T23:08:18.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.674+0000 7f95de800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95c00039e0 0x7f95d8155140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:18.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.674+0000 7f95de800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f95d81490a0 0x7f95d8153580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:18.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.674+0000 7f95de800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95d81ea7b0 0x7f95d81eac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:18.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.674+0000 7f95de800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95d806e700 con 0x7f95c00039e0 2024-02-21T23:08:18.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.674+0000 7f95de800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95d8156660 con 0x7f95d81ea7b0 2024-02-21T23:08:18.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.674+0000 7f95de800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95d806e880 con 0x7f95d81490a0 2024-02-21T23:08:18.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.675+0000 7f95dd400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95c00039e0 0x7f95d8155140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:18.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.675+0000 7f95dd400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95c00039e0 0x7f95d8155140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57610/0 (socket says 172.21.15.28:57610) 2024-02-21T23:08:18.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.675+0000 7f95dd400640 1 -- 172.21.15.28:0/1914875240 learned_addr learned my addr 172.21.15.28:0/1914875240 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:18.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.675+0000 7f95dde00640 1 --2- 172.21.15.28:0/1914875240 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95d81ea7b0 0x7f95d81eac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:18.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.675+0000 7f95dca00640 1 --2- 172.21.15.28:0/1914875240 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f95d81490a0 0x7f95d8153580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:18.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.675+0000 7f95dd400640 1 -- 172.21.15.28:0/1914875240 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f95d81490a0 msgr2=0x7f95d8153580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:18.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.675+0000 7f95dd400640 1 --2- 172.21.15.28:0/1914875240 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f95d81490a0 0x7f95d8153580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:18.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.675+0000 7f95dd400640 1 -- 172.21.15.28:0/1914875240 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95d81ea7b0 msgr2=0x7f95d81eac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:18.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.675+0000 7f95dd400640 1 --2- 172.21.15.28:0/1914875240 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95d81ea7b0 0x7f95d81eac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:18.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.675+0000 7f95dd400640 1 -- 172.21.15.28:0/1914875240 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95d814d660 con 0x7f95c00039e0 2024-02-21T23:08:18.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.675+0000 7f95dde00640 1 --2- 172.21.15.28:0/1914875240 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95d81ea7b0 0x7f95d81eac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:08:18.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.675+0000 7f95dd400640 1 --2- 172.21.15.28:0/1914875240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95c00039e0 0x7f95d8155140 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f95cc005040 tx=0x7f95cc001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:18.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.676+0000 7f95d5600640 1 -- 172.21.15.28:0/1914875240 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95cc041070 con 0x7f95c00039e0 2024-02-21T23:08:18.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.676+0000 7f95d5600640 1 -- 172.21.15.28:0/1914875240 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95cc005480 con 0x7f95c00039e0 2024-02-21T23:08:18.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.676+0000 7f95de800640 1 -- 172.21.15.28:0/1914875240 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95d8148dc0 con 0x7f95c00039e0 2024-02-21T23:08:18.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.676+0000 7f95d5600640 1 -- 172.21.15.28:0/1914875240 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95cc007a30 con 0x7f95c00039e0 2024-02-21T23:08:18.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.676+0000 7f95de800640 1 -- 172.21.15.28:0/1914875240 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95d8155680 con 0x7f95c00039e0 2024-02-21T23:08:18.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.677+0000 7f95de800640 1 -- 172.21.15.28:0/1914875240 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95d814d470 con 0x7f95c00039e0 2024-02-21T23:08:18.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.678+0000 7f95d5600640 1 -- 172.21.15.28:0/1914875240 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f95cc044070 con 0x7f95c00039e0 2024-02-21T23:08:18.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.679+0000 7f95d5600640 1 --2- 172.21.15.28:0/1914875240 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f95b806b4d0 0x7f95b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:18.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.679+0000 7f95d5600640 1 -- 172.21.15.28:0/1914875240 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f95cc0abfd0 con 0x7f95c00039e0 2024-02-21T23:08:18.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.679+0000 7f95dca00640 1 --2- 172.21.15.28:0/1914875240 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f95b806b4d0 0x7f95b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:18.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.680+0000 7f95dca00640 1 --2- 172.21.15.28:0/1914875240 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f95b806b4d0 0x7f95b806d990 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f95d8155940 tx=0x7f95c4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:18.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.683+0000 7f95d5600640 1 -- 172.21.15.28:0/1914875240 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f95cc07add0 con 0x7f95c00039e0 2024-02-21T23:08:18.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.879+0000 7f95de800640 1 -- 172.21.15.28:0/1914875240 --> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f95d8149530 con 0x7f95b806b4d0 2024-02-21T23:08:18.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.897+0000 7f95d5600640 1 -- 172.21.15.28:0/1914875240 <== mgr.4101 v2:172.21.15.81:6832/1848538577 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252559 (secure 0 0 0) 0x7f95c4009dc0 con 0x7f95b806b4d0 2024-02-21T23:08:18.889 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:08:18.891 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-21T23:08:18.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.903+0000 7f95de800640 1 -- 172.21.15.28:0/1914875240 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f95b806b4d0 msgr2=0x7f95b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:18.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.903+0000 7f95de800640 1 --2- 172.21.15.28:0/1914875240 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f95b806b4d0 0x7f95b806d990 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f95d8155940 tx=0x7f95c4008040 comp rx=0 tx=0).stop 2024-02-21T23:08:18.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.903+0000 7f95de800640 1 -- 172.21.15.28:0/1914875240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95c00039e0 msgr2=0x7f95d8155140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:18.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.903+0000 7f95de800640 1 --2- 172.21.15.28:0/1914875240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95c00039e0 0x7f95d8155140 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f95cc005040 tx=0x7f95cc001120 comp rx=0 tx=0).stop 2024-02-21T23:08:18.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.903+0000 7f95de800640 1 -- 172.21.15.28:0/1914875240 shutdown_connections 2024-02-21T23:08:18.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.903+0000 7f95de800640 1 --2- 172.21.15.28:0/1914875240 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f95b806b4d0 0x7f95b806d990 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:18.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.903+0000 7f95de800640 1 --2- 172.21.15.28:0/1914875240 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f95d81490a0 0x7f95d8153580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:18.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.903+0000 7f95de800640 1 --2- 172.21.15.28:0/1914875240 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95d81ea7b0 0x7f95d81eac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:18.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.903+0000 7f95de800640 1 --2- 172.21.15.28:0/1914875240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95c00039e0 0x7f95d8155140 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:18.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.903+0000 7f95de800640 1 -- 172.21.15.28:0/1914875240 >> 172.21.15.28:0/1914875240 conn(0x7f95d806b630 msgr2=0x7f95d814e290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:18.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.903+0000 7f95de800640 1 -- 172.21.15.28:0/1914875240 shutdown_connections 2024-02-21T23:08:18.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:18.903+0000 7f95de800640 1 -- 172.21.15.28:0/1914875240 wait complete. 2024-02-21T23:08:18.912 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":289,"stamp":"2024-02-21T23:08:18.785233+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":128,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":261520,"kb_used_data":12056,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754713200,"statfs":{"total":773094113280,"available":772826316800,"internally_reserved":0,"allocated":12345344,"data_stored":7093452,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002031"},"pg_stats":[{"pgid":"20.3f","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.467762+0000","last_change":"2024-02-21T23:08:14.449279+0000","last_active":"2024-02-21T23:08:15.467762+0000","last_peered":"2024-02-21T23:08:15.467762+0000","last_clean":"2024-02-21T23:08:15.467762+0000","last_became_active":"2024-02-21T23:08:14.448996+0000","last_became_peered":"2024-02-21T23:08:14.448996+0000","last_unstale":"2024-02-21T23:08:15.467762+0000","last_undegraded":"2024-02-21T23:08:15.467762+0000","last_fullsized":"2024-02-21T23:08:15.467762+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:37:10.848059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3e","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.429113+0000","last_change":"2024-02-21T23:08:14.441342+0000","last_active":"2024-02-21T23:08:15.429113+0000","last_peered":"2024-02-21T23:08:15.429113+0000","last_clean":"2024-02-21T23:08:15.429113+0000","last_became_active":"2024-02-21T23:08:14.441058+0000","last_became_peered":"2024-02-21T23:08:14.441058+0000","last_unstale":"2024-02-21T23:08:15.429113+0000","last_undegraded":"2024-02-21T23:08:15.429113+0000","last_fullsized":"2024-02-21T23:08:15.429113+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:54:54.757289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3d","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.466750+0000","last_change":"2024-02-21T23:08:14.459063+0000","last_active":"2024-02-21T23:08:15.466750+0000","last_peered":"2024-02-21T23:08:15.466750+0000","last_clean":"2024-02-21T23:08:15.466750+0000","last_became_active":"2024-02-21T23:08:14.458089+0000","last_became_peered":"2024-02-21T23:08:14.458089+0000","last_unstale":"2024-02-21T23:08:15.466750+0000","last_undegraded":"2024-02-21T23:08:15.466750+0000","last_fullsized":"2024-02-21T23:08:15.466750+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:20:49.830901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.3c","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.428625+0000","last_change":"2024-02-21T23:08:14.449246+0000","last_active":"2024-02-21T23:08:15.428625+0000","last_peered":"2024-02-21T23:08:15.428625+0000","last_clean":"2024-02-21T23:08:15.428625+0000","last_became_active":"2024-02-21T23:08:14.448193+0000","last_became_peered":"2024-02-21T23:08:14.448193+0000","last_unstale":"2024-02-21T23:08:15.428625+0000","last_undegraded":"2024-02-21T23:08:15.428625+0000","last_fullsized":"2024-02-21T23:08:15.428625+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:03:12.105566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3b","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.466240+0000","last_change":"2024-02-21T23:08:14.447763+0000","last_active":"2024-02-21T23:08:15.466240+0000","last_peered":"2024-02-21T23:08:15.466240+0000","last_clean":"2024-02-21T23:08:15.466240+0000","last_became_active":"2024-02-21T23:08:14.447343+0000","last_became_peered":"2024-02-21T23:08:14.447343+0000","last_unstale":"2024-02-21T23:08:15.466240+0000","last_undegraded":"2024-02-21T23:08:15.466240+0000","last_fullsized":"2024-02-21T23:08:15.466240+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:10:37.708703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.3a","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.424848+0000","last_change":"2024-02-21T23:08:14.463733+0000","last_active":"2024-02-21T23:08:15.424848+0000","last_peered":"2024-02-21T23:08:15.424848+0000","last_clean":"2024-02-21T23:08:15.424848+0000","last_became_active":"2024-02-21T23:08:14.462259+0000","last_became_peered":"2024-02-21T23:08:14.462259+0000","last_unstale":"2024-02-21T23:08:15.424848+0000","last_undegraded":"2024-02-21T23:08:15.424848+0000","last_fullsized":"2024-02-21T23:08:15.424848+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:01:34.090943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.39","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.465791+0000","last_change":"2024-02-21T23:08:14.452506+0000","last_active":"2024-02-21T23:08:15.465791+0000","last_peered":"2024-02-21T23:08:15.465791+0000","last_clean":"2024-02-21T23:08:15.465791+0000","last_became_active":"2024-02-21T23:08:14.452226+0000","last_became_peered":"2024-02-21T23:08:14.452226+0000","last_unstale":"2024-02-21T23:08:15.465791+0000","last_undegraded":"2024-02-21T23:08:15.465791+0000","last_fullsized":"2024-02-21T23:08:15.465791+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:44:15.626046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.38","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.426646+0000","last_change":"2024-02-21T23:08:14.442162+0000","last_active":"2024-02-21T23:08:15.426646+0000","last_peered":"2024-02-21T23:08:15.426646+0000","last_clean":"2024-02-21T23:08:15.426646+0000","last_became_active":"2024-02-21T23:08:14.441850+0000","last_became_peered":"2024-02-21T23:08:14.441850+0000","last_unstale":"2024-02-21T23:08:15.426646+0000","last_undegraded":"2024-02-21T23:08:15.426646+0000","last_fullsized":"2024-02-21T23:08:15.426646+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:13:06.500761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.37","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.425783+0000","last_change":"2024-02-21T23:08:14.455069+0000","last_active":"2024-02-21T23:08:15.425783+0000","last_peered":"2024-02-21T23:08:15.425783+0000","last_clean":"2024-02-21T23:08:15.425783+0000","last_became_active":"2024-02-21T23:08:14.454196+0000","last_became_peered":"2024-02-21T23:08:14.454196+0000","last_unstale":"2024-02-21T23:08:15.425783+0000","last_undegraded":"2024-02-21T23:08:15.425783+0000","last_fullsized":"2024-02-21T23:08:15.425783+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:01:18.703812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.36","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.468521+0000","last_change":"2024-02-21T23:08:14.432934+0000","last_active":"2024-02-21T23:08:15.468521+0000","last_peered":"2024-02-21T23:08:15.468521+0000","last_clean":"2024-02-21T23:08:15.468521+0000","last_became_active":"2024-02-21T23:08:14.432607+0000","last_became_peered":"2024-02-21T23:08:14.432607+0000","last_unstale":"2024-02-21T23:08:15.468521+0000","last_undegraded":"2024-02-21T23:08:15.468521+0000","last_fullsized":"2024-02-21T23:08:15.468521+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:40:43.618038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.35","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.428637+0000","last_change":"2024-02-21T23:08:14.451533+0000","last_active":"2024-02-21T23:08:15.428637+0000","last_peered":"2024-02-21T23:08:15.428637+0000","last_clean":"2024-02-21T23:08:15.428637+0000","last_became_active":"2024-02-21T23:08:14.450545+0000","last_became_peered":"2024-02-21T23:08:14.450545+0000","last_unstale":"2024-02-21T23:08:15.428637+0000","last_undegraded":"2024-02-21T23:08:15.428637+0000","last_fullsized":"2024-02-21T23:08:15.428637+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:54:23.719045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.34","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.468972+0000","last_change":"2024-02-21T23:08:14.450718+0000","last_active":"2024-02-21T23:08:15.468972+0000","last_peered":"2024-02-21T23:08:15.468972+0000","last_clean":"2024-02-21T23:08:15.468972+0000","last_became_active":"2024-02-21T23:08:14.450413+0000","last_became_peered":"2024-02-21T23:08:14.450413+0000","last_unstale":"2024-02-21T23:08:15.468972+0000","last_undegraded":"2024-02-21T23:08:15.468972+0000","last_fullsized":"2024-02-21T23:08:15.468972+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:43:19.437727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.33","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.425204+0000","last_change":"2024-02-21T23:08:14.466078+0000","last_active":"2024-02-21T23:08:15.425204+0000","last_peered":"2024-02-21T23:08:15.425204+0000","last_clean":"2024-02-21T23:08:15.425204+0000","last_became_active":"2024-02-21T23:08:14.463223+0000","last_became_peered":"2024-02-21T23:08:14.463223+0000","last_unstale":"2024-02-21T23:08:15.425204+0000","last_undegraded":"2024-02-21T23:08:15.425204+0000","last_fullsized":"2024-02-21T23:08:15.425204+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:03:12.105566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.32","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.467963+0000","last_change":"2024-02-21T23:08:14.455548+0000","last_active":"2024-02-21T23:08:15.467963+0000","last_peered":"2024-02-21T23:08:15.467963+0000","last_clean":"2024-02-21T23:08:15.467963+0000","last_became_active":"2024-02-21T23:08:14.455262+0000","last_became_peered":"2024-02-21T23:08:14.455262+0000","last_unstale":"2024-02-21T23:08:15.467963+0000","last_undegraded":"2024-02-21T23:08:15.467963+0000","last_fullsized":"2024-02-21T23:08:15.467963+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:29:47.674102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.31","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.467197+0000","last_change":"2024-02-21T23:08:14.462237+0000","last_active":"2024-02-21T23:08:15.467197+0000","last_peered":"2024-02-21T23:08:15.467197+0000","last_clean":"2024-02-21T23:08:15.467197+0000","last_became_active":"2024-02-21T23:08:14.461701+0000","last_became_peered":"2024-02-21T23:08:14.461701+0000","last_unstale":"2024-02-21T23:08:15.467197+0000","last_undegraded":"2024-02-21T23:08:15.467197+0000","last_fullsized":"2024-02-21T23:08:15.467197+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:00:49.516505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.30","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.424537+0000","last_change":"2024-02-21T23:08:14.445396+0000","last_active":"2024-02-21T23:08:15.424537+0000","last_peered":"2024-02-21T23:08:15.424537+0000","last_clean":"2024-02-21T23:08:15.424537+0000","last_became_active":"2024-02-21T23:08:14.445101+0000","last_became_peered":"2024-02-21T23:08:14.445101+0000","last_unstale":"2024-02-21T23:08:15.424537+0000","last_undegraded":"2024-02-21T23:08:15.424537+0000","last_fullsized":"2024-02-21T23:08:15.424537+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:16:07.331005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.2f","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.425812+0000","last_change":"2024-02-21T23:08:14.452042+0000","last_active":"2024-02-21T23:08:15.425812+0000","last_peered":"2024-02-21T23:08:15.425812+0000","last_clean":"2024-02-21T23:08:15.425812+0000","last_became_active":"2024-02-21T23:08:14.451684+0000","last_became_peered":"2024-02-21T23:08:14.451684+0000","last_unstale":"2024-02-21T23:08:15.425812+0000","last_undegraded":"2024-02-21T23:08:15.425812+0000","last_fullsized":"2024-02-21T23:08:15.425812+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:30:39.146916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.2e","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.466508+0000","last_change":"2024-02-21T23:08:14.440941+0000","last_active":"2024-02-21T23:08:15.466508+0000","last_peered":"2024-02-21T23:08:15.466508+0000","last_clean":"2024-02-21T23:08:15.466508+0000","last_became_active":"2024-02-21T23:08:14.440448+0000","last_became_peered":"2024-02-21T23:08:14.440448+0000","last_unstale":"2024-02-21T23:08:15.466508+0000","last_undegraded":"2024-02-21T23:08:15.466508+0000","last_fullsized":"2024-02-21T23:08:15.466508+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:59:31.892130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.2d","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.466601+0000","last_change":"2024-02-21T23:08:14.452885+0000","last_active":"2024-02-21T23:08:15.466601+0000","last_peered":"2024-02-21T23:08:15.466601+0000","last_clean":"2024-02-21T23:08:15.466601+0000","last_became_active":"2024-02-21T23:08:14.452551+0000","last_became_peered":"2024-02-21T23:08:14.452551+0000","last_unstale":"2024-02-21T23:08:15.466601+0000","last_undegraded":"2024-02-21T23:08:15.466601+0000","last_fullsized":"2024-02-21T23:08:15.466601+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:33:43.329787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.2c","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.466647+0000","last_change":"2024-02-21T23:08:14.459775+0000","last_active":"2024-02-21T23:08:15.466647+0000","last_peered":"2024-02-21T23:08:15.466647+0000","last_clean":"2024-02-21T23:08:15.466647+0000","last_became_active":"2024-02-21T23:08:14.459460+0000","last_became_peered":"2024-02-21T23:08:14.459460+0000","last_unstale":"2024-02-21T23:08:15.466647+0000","last_undegraded":"2024-02-21T23:08:15.466647+0000","last_fullsized":"2024-02-21T23:08:15.466647+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:46:54.184368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.2b","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.428226+0000","last_change":"2024-02-21T23:08:14.451956+0000","last_active":"2024-02-21T23:08:15.428226+0000","last_peered":"2024-02-21T23:08:15.428226+0000","last_clean":"2024-02-21T23:08:15.428226+0000","last_became_active":"2024-02-21T23:08:14.447631+0000","last_became_peered":"2024-02-21T23:08:14.447631+0000","last_unstale":"2024-02-21T23:08:15.428226+0000","last_undegraded":"2024-02-21T23:08:15.428226+0000","last_fullsized":"2024-02-21T23:08:15.428226+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:01:34.090943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2a","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.467775+0000","last_change":"2024-02-21T23:08:14.443331+0000","last_active":"2024-02-21T23:08:15.467775+0000","last_peered":"2024-02-21T23:08:15.467775+0000","last_clean":"2024-02-21T23:08:15.467775+0000","last_became_active":"2024-02-21T23:08:14.442067+0000","last_became_peered":"2024-02-21T23:08:14.442067+0000","last_unstale":"2024-02-21T23:08:15.467775+0000","last_undegraded":"2024-02-21T23:08:15.467775+0000","last_fullsized":"2024-02-21T23:08:15.467775+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:33:28.645253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.29","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.428471+0000","last_change":"2024-02-21T23:08:14.465223+0000","last_active":"2024-02-21T23:08:15.428471+0000","last_peered":"2024-02-21T23:08:15.428471+0000","last_clean":"2024-02-21T23:08:15.428471+0000","last_became_active":"2024-02-21T23:08:14.464452+0000","last_became_peered":"2024-02-21T23:08:14.464452+0000","last_unstale":"2024-02-21T23:08:15.428471+0000","last_undegraded":"2024-02-21T23:08:15.428471+0000","last_fullsized":"2024-02-21T23:08:15.428471+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:41:39.305102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.28","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.424604+0000","last_change":"2024-02-21T23:08:14.445139+0000","last_active":"2024-02-21T23:08:15.424604+0000","last_peered":"2024-02-21T23:08:15.424604+0000","last_clean":"2024-02-21T23:08:15.424604+0000","last_became_active":"2024-02-21T23:08:14.444872+0000","last_became_peered":"2024-02-21T23:08:14.444872+0000","last_unstale":"2024-02-21T23:08:15.424604+0000","last_undegraded":"2024-02-21T23:08:15.424604+0000","last_fullsized":"2024-02-21T23:08:15.424604+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:02:05.973126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.27","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.467155+0000","last_change":"2024-02-21T23:08:14.444418+0000","last_active":"2024-02-21T23:08:15.467155+0000","last_peered":"2024-02-21T23:08:15.467155+0000","last_clean":"2024-02-21T23:08:15.467155+0000","last_became_active":"2024-02-21T23:08:14.444123+0000","last_became_peered":"2024-02-21T23:08:14.444123+0000","last_unstale":"2024-02-21T23:08:15.467155+0000","last_undegraded":"2024-02-21T23:08:15.467155+0000","last_fullsized":"2024-02-21T23:08:15.467155+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:48:35.406735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.26","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.467559+0000","last_change":"2024-02-21T23:08:14.462662+0000","last_active":"2024-02-21T23:08:15.467559+0000","last_peered":"2024-02-21T23:08:15.467559+0000","last_clean":"2024-02-21T23:08:15.467559+0000","last_became_active":"2024-02-21T23:08:14.462381+0000","last_became_peered":"2024-02-21T23:08:14.462381+0000","last_unstale":"2024-02-21T23:08:15.467559+0000","last_undegraded":"2024-02-21T23:08:15.467559+0000","last_fullsized":"2024-02-21T23:08:15.467559+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:02:05.973126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.25","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.466316+0000","last_change":"2024-02-21T23:08:14.464323+0000","last_active":"2024-02-21T23:08:15.466316+0000","last_peered":"2024-02-21T23:08:15.466316+0000","last_clean":"2024-02-21T23:08:15.466316+0000","last_became_active":"2024-02-21T23:08:14.461888+0000","last_became_peered":"2024-02-21T23:08:14.461888+0000","last_unstale":"2024-02-21T23:08:15.466316+0000","last_undegraded":"2024-02-21T23:08:15.466316+0000","last_fullsized":"2024-02-21T23:08:15.466316+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:35:46.921047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.24","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.469091+0000","last_change":"2024-02-21T23:08:14.439704+0000","last_active":"2024-02-21T23:08:15.469091+0000","last_peered":"2024-02-21T23:08:15.469091+0000","last_clean":"2024-02-21T23:08:15.469091+0000","last_became_active":"2024-02-21T23:08:14.439397+0000","last_became_peered":"2024-02-21T23:08:14.439397+0000","last_unstale":"2024-02-21T23:08:15.469091+0000","last_undegraded":"2024-02-21T23:08:15.469091+0000","last_fullsized":"2024-02-21T23:08:15.469091+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:11:34.837356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.23","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.467717+0000","last_change":"2024-02-21T23:08:14.446999+0000","last_active":"2024-02-21T23:08:15.467717+0000","last_peered":"2024-02-21T23:08:15.467717+0000","last_clean":"2024-02-21T23:08:15.467717+0000","last_became_active":"2024-02-21T23:08:14.446705+0000","last_became_peered":"2024-02-21T23:08:14.446705+0000","last_unstale":"2024-02-21T23:08:15.467717+0000","last_undegraded":"2024-02-21T23:08:15.467717+0000","last_fullsized":"2024-02-21T23:08:15.467717+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:01:34.090943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.22","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.467207+0000","last_change":"2024-02-21T23:08:14.448941+0000","last_active":"2024-02-21T23:08:15.467207+0000","last_peered":"2024-02-21T23:08:15.467207+0000","last_clean":"2024-02-21T23:08:15.467207+0000","last_became_active":"2024-02-21T23:08:14.444323+0000","last_became_peered":"2024-02-21T23:08:14.444323+0000","last_unstale":"2024-02-21T23:08:15.467207+0000","last_undegraded":"2024-02-21T23:08:15.467207+0000","last_fullsized":"2024-02-21T23:08:15.467207+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:47:09.481691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.21","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.427094+0000","last_change":"2024-02-21T23:08:14.453634+0000","last_active":"2024-02-21T23:08:15.427094+0000","last_peered":"2024-02-21T23:08:15.427094+0000","last_clean":"2024-02-21T23:08:15.427094+0000","last_became_active":"2024-02-21T23:08:14.453352+0000","last_became_peered":"2024-02-21T23:08:14.453352+0000","last_unstale":"2024-02-21T23:08:15.427094+0000","last_undegraded":"2024-02-21T23:08:15.427094+0000","last_fullsized":"2024-02-21T23:08:15.427094+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:08:39.925583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.20","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.468840+0000","last_change":"2024-02-21T23:08:14.457744+0000","last_active":"2024-02-21T23:08:15.468840+0000","last_peered":"2024-02-21T23:08:15.468840+0000","last_clean":"2024-02-21T23:08:15.468840+0000","last_became_active":"2024-02-21T23:08:14.456937+0000","last_became_peered":"2024-02-21T23:08:14.456937+0000","last_unstale":"2024-02-21T23:08:15.468840+0000","last_undegraded":"2024-02-21T23:08:15.468840+0000","last_fullsized":"2024-02-21T23:08:15.468840+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:11:03.093612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.426713+0000","last_change":"2024-02-21T23:08:14.455515+0000","last_active":"2024-02-21T23:08:15.426713+0000","last_peered":"2024-02-21T23:08:15.426713+0000","last_clean":"2024-02-21T23:08:15.426713+0000","last_became_active":"2024-02-21T23:08:14.455243+0000","last_became_peered":"2024-02-21T23:08:14.455243+0000","last_unstale":"2024-02-21T23:08:15.426713+0000","last_undegraded":"2024-02-21T23:08:15.426713+0000","last_fullsized":"2024-02-21T23:08:15.426713+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:20:49.830901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1f","version":"121'4","reported_seq":18,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.467278+0000","last_change":"2024-02-21T23:08:13.411691+0000","last_active":"2024-02-21T23:08:15.467278+0000","last_peered":"2024-02-21T23:08:15.467278+0000","last_clean":"2024-02-21T23:08:15.467278+0000","last_became_active":"2024-02-21T23:08:13.411166+0000","last_became_peered":"2024-02-21T23:08:13.411166+0000","last_unstale":"2024-02-21T23:08:15.467278+0000","last_undegraded":"2024-02-21T23:08:15.467278+0000","last_fullsized":"2024-02-21T23:08:15.467278+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_clean_scrub_stamp":"2024-02-21T23:08:12.367049+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:17:52.454790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.467427+0000","last_change":"2024-02-21T23:08:14.442518+0000","last_active":"2024-02-21T23:08:15.467427+0000","last_peered":"2024-02-21T23:08:15.467427+0000","last_clean":"2024-02-21T23:08:15.467427+0000","last_became_active":"2024-02-21T23:08:14.441709+0000","last_became_peered":"2024-02-21T23:08:14.441709+0000","last_unstale":"2024-02-21T23:08:15.467427+0000","last_undegraded":"2024-02-21T23:08:15.467427+0000","last_fullsized":"2024-02-21T23:08:15.467427+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:48:49.955729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1e","version":"121'1","reported_seq":15,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.467047+0000","last_change":"2024-02-21T23:08:13.426961+0000","last_active":"2024-02-21T23:08:15.467047+0000","last_peered":"2024-02-21T23:08:15.467047+0000","last_clean":"2024-02-21T23:08:15.467047+0000","last_became_active":"2024-02-21T23:08:13.425923+0000","last_became_peered":"2024-02-21T23:08:13.425923+0000","last_unstale":"2024-02-21T23:08:15.467047+0000","last_undegraded":"2024-02-21T23:08:15.467047+0000","last_fullsized":"2024-02-21T23:08:15.467047+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_clean_scrub_stamp":"2024-02-21T23:08:12.367049+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:30:38.130535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1a","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.428188+0000","last_change":"2024-02-21T23:08:14.461921+0000","last_active":"2024-02-21T23:08:15.428188+0000","last_peered":"2024-02-21T23:08:15.428188+0000","last_clean":"2024-02-21T23:08:15.428188+0000","last_became_active":"2024-02-21T23:08:14.461643+0000","last_became_peered":"2024-02-21T23:08:14.461643+0000","last_unstale":"2024-02-21T23:08:15.428188+0000","last_undegraded":"2024-02-21T23:08:15.428188+0000","last_fullsized":"2024-02-21T23:08:15.428188+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:02:05.973126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1d","version":"121'2","reported_seq":16,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.466888+0000","last_change":"2024-02-21T23:08:13.430861+0000","last_active":"2024-02-21T23:08:15.466888+0000","last_peered":"2024-02-21T23:08:15.466888+0000","last_clean":"2024-02-21T23:08:15.466888+0000","last_became_active":"2024-02-21T23:08:13.429039+0000","last_became_peered":"2024-02-21T23:08:13.429039+0000","last_unstale":"2024-02-21T23:08:15.466888+0000","last_undegraded":"2024-02-21T23:08:15.466888+0000","last_fullsized":"2024-02-21T23:08:15.466888+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_clean_scrub_stamp":"2024-02-21T23:08:12.367049+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:47:08.465310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1b","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.467586+0000","last_change":"2024-02-21T23:08:14.455808+0000","last_active":"2024-02-21T23:08:15.467586+0000","last_peered":"2024-02-21T23:08:15.467586+0000","last_clean":"2024-02-21T23:08:15.467586+0000","last_became_active":"2024-02-21T23:08:14.455515+0000","last_became_peered":"2024-02-21T23:08:14.455515+0000","last_unstale":"2024-02-21T23:08:15.467586+0000","last_undegraded":"2024-02-21T23:08:15.467586+0000","last_fullsized":"2024-02-21T23:08:15.467586+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:09:02.474366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1c","version":"121'2","reported_seq":16,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.465697+0000","last_change":"2024-02-21T23:08:13.426772+0000","last_active":"2024-02-21T23:08:15.465697+0000","last_peered":"2024-02-21T23:08:15.465697+0000","last_clean":"2024-02-21T23:08:15.465697+0000","last_became_active":"2024-02-21T23:08:13.426462+0000","last_became_peered":"2024-02-21T23:08:13.426462+0000","last_unstale":"2024-02-21T23:08:15.465697+0000","last_undegraded":"2024-02-21T23:08:15.465697+0000","last_fullsized":"2024-02-21T23:08:15.465697+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_clean_scrub_stamp":"2024-02-21T23:08:12.367049+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:48:34.390354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.427692+0000","last_change":"2024-02-21T23:08:14.446601+0000","last_active":"2024-02-21T23:08:15.427692+0000","last_peered":"2024-02-21T23:08:15.427692+0000","last_clean":"2024-02-21T23:08:15.427692+0000","last_became_active":"2024-02-21T23:08:14.446282+0000","last_became_peered":"2024-02-21T23:08:14.446282+0000","last_unstale":"2024-02-21T23:08:15.427692+0000","last_undegraded":"2024-02-21T23:08:15.427692+0000","last_fullsized":"2024-02-21T23:08:15.427692+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:48:35.406735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1b","version":"121'1","reported_seq":15,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.427072+0000","last_change":"2024-02-21T23:08:13.423770+0000","last_active":"2024-02-21T23:08:15.427072+0000","last_peered":"2024-02-21T23:08:15.427072+0000","last_clean":"2024-02-21T23:08:15.427072+0000","last_became_active":"2024-02-21T23:08:13.423433+0000","last_became_peered":"2024-02-21T23:08:13.423433+0000","last_unstale":"2024-02-21T23:08:15.427072+0000","last_undegraded":"2024-02-21T23:08:15.427072+0000","last_fullsized":"2024-02-21T23:08:15.427072+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_clean_scrub_stamp":"2024-02-21T23:08:12.367049+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:27:41.020802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1d","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.428710+0000","last_change":"2024-02-21T23:08:14.441526+0000","last_active":"2024-02-21T23:08:15.428710+0000","last_peered":"2024-02-21T23:08:15.428710+0000","last_clean":"2024-02-21T23:08:15.428710+0000","last_became_active":"2024-02-21T23:08:14.441251+0000","last_became_peered":"2024-02-21T23:08:14.441251+0000","last_unstale":"2024-02-21T23:08:15.428710+0000","last_undegraded":"2024-02-21T23:08:15.428710+0000","last_fullsized":"2024-02-21T23:08:15.428710+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:47:09.481691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1a","version":"121'1","reported_seq":15,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.466102+0000","last_change":"2024-02-21T23:08:13.423864+0000","last_active":"2024-02-21T23:08:15.466102+0000","last_peered":"2024-02-21T23:08:15.466102+0000","last_clean":"2024-02-21T23:08:15.466102+0000","last_became_active":"2024-02-21T23:08:13.423576+0000","last_became_peered":"2024-02-21T23:08:13.423576+0000","last_unstale":"2024-02-21T23:08:15.466102+0000","last_undegraded":"2024-02-21T23:08:15.466102+0000","last_fullsized":"2024-02-21T23:08:15.466102+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_clean_scrub_stamp":"2024-02-21T23:08:12.367049+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:01:17.687431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.466841+0000","last_change":"2024-02-21T23:08:14.463245+0000","last_active":"2024-02-21T23:08:15.466841+0000","last_peered":"2024-02-21T23:08:15.466841+0000","last_clean":"2024-02-21T23:08:15.466841+0000","last_became_active":"2024-02-21T23:08:14.461701+0000","last_became_peered":"2024-02-21T23:08:14.461701+0000","last_unstale":"2024-02-21T23:08:15.466841+0000","last_undegraded":"2024-02-21T23:08:15.466841+0000","last_fullsized":"2024-02-21T23:08:15.466841+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:20:41.288649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":14,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.424867+0000","last_change":"2024-02-21T23:08:13.427214+0000","last_active":"2024-02-21T23:08:15.424867+0000","last_peered":"2024-02-21T23:08:15.424867+0000","last_clean":"2024-02-21T23:08:15.424867+0000","last_became_active":"2024-02-21T23:08:13.426923+0000","last_became_peered":"2024-02-21T23:08:13.426923+0000","last_unstale":"2024-02-21T23:08:15.424867+0000","last_undegraded":"2024-02-21T23:08:15.424867+0000","last_fullsized":"2024-02-21T23:08:15.424867+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_clean_scrub_stamp":"2024-02-21T23:08:12.367049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:57:40.070790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1f","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.467621+0000","last_change":"2024-02-21T23:08:14.459234+0000","last_active":"2024-02-21T23:08:15.467621+0000","last_peered":"2024-02-21T23:08:15.467621+0000","last_clean":"2024-02-21T23:08:15.467621+0000","last_became_active":"2024-02-21T23:08:14.458970+0000","last_became_peered":"2024-02-21T23:08:14.458970+0000","last_unstale":"2024-02-21T23:08:15.467621+0000","last_undegraded":"2024-02-21T23:08:15.467621+0000","last_fullsized":"2024-02-21T23:08:15.467621+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:03:18.697668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":14,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.466324+0000","last_change":"2024-02-21T23:08:13.417711+0000","last_active":"2024-02-21T23:08:15.466324+0000","last_peered":"2024-02-21T23:08:15.466324+0000","last_clean":"2024-02-21T23:08:15.466324+0000","last_became_active":"2024-02-21T23:08:13.412707+0000","last_became_peered":"2024-02-21T23:08:13.412707+0000","last_unstale":"2024-02-21T23:08:15.466324+0000","last_undegraded":"2024-02-21T23:08:15.466324+0000","last_fullsized":"2024-02-21T23:08:15.466324+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_clean_scrub_stamp":"2024-02-21T23:08:12.367049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:36:37.776608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.466786+0000","last_change":"2024-02-21T23:08:14.445447+0000","last_active":"2024-02-21T23:08:15.466786+0000","last_peered":"2024-02-21T23:08:15.466786+0000","last_clean":"2024-02-21T23:08:15.466786+0000","last_became_active":"2024-02-21T23:08:14.444905+0000","last_became_peered":"2024-02-21T23:08:14.444905+0000","last_unstale":"2024-02-21T23:08:15.466786+0000","last_undegraded":"2024-02-21T23:08:15.466786+0000","last_fullsized":"2024-02-21T23:08:15.466786+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:01:15.415412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":14,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.429439+0000","last_change":"2024-02-21T23:08:13.428573+0000","last_active":"2024-02-21T23:08:15.429439+0000","last_peered":"2024-02-21T23:08:15.429439+0000","last_clean":"2024-02-21T23:08:15.429439+0000","last_became_active":"2024-02-21T23:08:13.425871+0000","last_became_peered":"2024-02-21T23:08:13.425871+0000","last_unstale":"2024-02-21T23:08:15.429439+0000","last_undegraded":"2024-02-21T23:08:15.429439+0000","last_fullsized":"2024-02-21T23:08:15.429439+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_clean_scrub_stamp":"2024-02-21T23:08:12.367049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:08:17.593082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.425070+0000","last_change":"2024-02-21T23:08:14.457930+0000","last_active":"2024-02-21T23:08:15.425070+0000","last_peered":"2024-02-21T23:08:15.425070+0000","last_clean":"2024-02-21T23:08:15.425070+0000","last_became_active":"2024-02-21T23:08:14.457084+0000","last_became_peered":"2024-02-21T23:08:14.457084+0000","last_unstale":"2024-02-21T23:08:15.425070+0000","last_undegraded":"2024-02-21T23:08:15.425070+0000","last_fullsized":"2024-02-21T23:08:15.425070+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:41:39.305102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":14,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.468509+0000","last_change":"2024-02-21T23:08:13.409691+0000","last_active":"2024-02-21T23:08:15.468509+0000","last_peered":"2024-02-21T23:08:15.468509+0000","last_clean":"2024-02-21T23:08:15.468509+0000","last_became_active":"2024-02-21T23:08:13.409289+0000","last_became_peered":"2024-02-21T23:08:13.409289+0000","last_unstale":"2024-02-21T23:08:15.468509+0000","last_undegraded":"2024-02-21T23:08:15.468509+0000","last_fullsized":"2024-02-21T23:08:15.468509+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_clean_scrub_stamp":"2024-02-21T23:08:12.367049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:55:55.845158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.467327+0000","last_change":"2024-02-21T23:08:14.439759+0000","last_active":"2024-02-21T23:08:15.467327+0000","last_peered":"2024-02-21T23:08:15.467327+0000","last_clean":"2024-02-21T23:08:15.467327+0000","last_became_active":"2024-02-21T23:08:14.439471+0000","last_became_peered":"2024-02-21T23:08:14.439471+0000","last_unstale":"2024-02-21T23:08:15.467327+0000","last_undegraded":"2024-02-21T23:08:15.467327+0000","last_fullsized":"2024-02-21T23:08:15.467327+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:16:07.331005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.15","version":"121'3","reported_seq":17,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.467165+0000","last_change":"2024-02-21T23:08:13.431736+0000","last_active":"2024-02-21T23:08:15.467165+0000","last_peered":"2024-02-21T23:08:15.467165+0000","last_clean":"2024-02-21T23:08:15.467165+0000","last_became_active":"2024-02-21T23:08:13.431271+0000","last_became_peered":"2024-02-21T23:08:13.431271+0000","last_unstale":"2024-02-21T23:08:15.467165+0000","last_undegraded":"2024-02-21T23:08:15.467165+0000","last_fullsized":"2024-02-21T23:08:15.467165+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_clean_scrub_stamp":"2024-02-21T23:08:12.367049+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:16:06.314624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.13","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.467921+0000","last_change":"2024-02-21T23:08:14.450201+0000","last_active":"2024-02-21T23:08:15.467921+0000","last_peered":"2024-02-21T23:08:15.467921+0000","last_clean":"2024-02-21T23:08:15.467921+0000","last_became_active":"2024-02-21T23:08:14.449100+0000","last_became_peered":"2024-02-21T23:08:14.449100+0000","last_unstale":"2024-02-21T23:08:15.467921+0000","last_undegraded":"2024-02-21T23:08:15.467921+0000","last_fullsized":"2024-02-21T23:08:15.467921+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:41:39.305102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.14","version":"121'6","reported_seq":20,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.465808+0000","last_change":"2024-02-21T23:08:13.430036+0000","last_active":"2024-02-21T23:08:15.465808+0000","last_peered":"2024-02-21T23:08:15.465808+0000","last_clean":"2024-02-21T23:08:15.465808+0000","last_became_active":"2024-02-21T23:08:13.429720+0000","last_became_peered":"2024-02-21T23:08:13.429720+0000","last_unstale":"2024-02-21T23:08:15.465808+0000","last_undegraded":"2024-02-21T23:08:15.465808+0000","last_fullsized":"2024-02-21T23:08:15.465808+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_clean_scrub_stamp":"2024-02-21T23:08:12.367049+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:02:04.956745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.14","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.469387+0000","last_change":"2024-02-21T23:08:14.444654+0000","last_active":"2024-02-21T23:08:15.469387+0000","last_peered":"2024-02-21T23:08:15.469387+0000","last_clean":"2024-02-21T23:08:15.469387+0000","last_became_active":"2024-02-21T23:08:14.442399+0000","last_became_peered":"2024-02-21T23:08:14.442399+0000","last_unstale":"2024-02-21T23:08:15.469387+0000","last_undegraded":"2024-02-21T23:08:15.469387+0000","last_fullsized":"2024-02-21T23:08:15.469387+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:04:54.173602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.13","version":"121'2","reported_seq":16,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.467836+0000","last_change":"2024-02-21T23:08:13.408848+0000","last_active":"2024-02-21T23:08:15.467836+0000","last_peered":"2024-02-21T23:08:15.467836+0000","last_clean":"2024-02-21T23:08:15.467836+0000","last_became_active":"2024-02-21T23:08:13.408427+0000","last_became_peered":"2024-02-21T23:08:13.408427+0000","last_unstale":"2024-02-21T23:08:15.467836+0000","last_undegraded":"2024-02-21T23:08:15.467836+0000","last_fullsized":"2024-02-21T23:08:15.467836+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_clean_scrub_stamp":"2024-02-21T23:08:12.367049+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:30:41.304449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.467624+0000","last_change":"2024-02-21T23:08:14.457492+0000","last_active":"2024-02-21T23:08:15.467624+0000","last_peered":"2024-02-21T23:08:15.467624+0000","last_clean":"2024-02-21T23:08:15.467624+0000","last_became_active":"2024-02-21T23:08:14.457071+0000","last_became_peered":"2024-02-21T23:08:14.457071+0000","last_unstale":"2024-02-21T23:08:15.467624+0000","last_undegraded":"2024-02-21T23:08:15.467624+0000","last_fullsized":"2024-02-21T23:08:15.467624+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:55:15.574780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":14,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.466966+0000","last_change":"2024-02-21T23:08:13.425784+0000","last_active":"2024-02-21T23:08:15.466966+0000","last_peered":"2024-02-21T23:08:15.466966+0000","last_clean":"2024-02-21T23:08:15.466966+0000","last_became_active":"2024-02-21T23:08:13.425472+0000","last_became_peered":"2024-02-21T23:08:13.425472+0000","last_unstale":"2024-02-21T23:08:15.466966+0000","last_undegraded":"2024-02-21T23:08:15.466966+0000","last_fullsized":"2024-02-21T23:08:15.466966+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_clean_scrub_stamp":"2024-02-21T23:08:12.367049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:59:09.803895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.467544+0000","last_change":"2024-02-21T23:08:14.447350+0000","last_active":"2024-02-21T23:08:15.467544+0000","last_peered":"2024-02-21T23:08:15.467544+0000","last_clean":"2024-02-21T23:08:15.467544+0000","last_became_active":"2024-02-21T23:08:14.446246+0000","last_became_peered":"2024-02-21T23:08:14.446246+0000","last_unstale":"2024-02-21T23:08:15.467544+0000","last_undegraded":"2024-02-21T23:08:15.467544+0000","last_fullsized":"2024-02-21T23:08:15.467544+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:59:30.631819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.10","version":"121'8","reported_seq":22,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.424695+0000","last_change":"2024-02-21T23:08:13.429933+0000","last_active":"2024-02-21T23:08:15.424695+0000","last_peered":"2024-02-21T23:08:15.424695+0000","last_clean":"2024-02-21T23:08:15.424695+0000","last_became_active":"2024-02-21T23:08:13.428835+0000","last_became_peered":"2024-02-21T23:08:13.428835+0000","last_unstale":"2024-02-21T23:08:15.424695+0000","last_undegraded":"2024-02-21T23:08:15.424695+0000","last_fullsized":"2024-02-21T23:08:15.424695+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_clean_scrub_stamp":"2024-02-21T23:08:12.367049+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:08:17.593082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.466379+0000","last_change":"2024-02-21T23:08:14.447205+0000","last_active":"2024-02-21T23:08:15.466379+0000","last_peered":"2024-02-21T23:08:15.466379+0000","last_clean":"2024-02-21T23:08:15.466379+0000","last_became_active":"2024-02-21T23:08:14.446907+0000","last_became_peered":"2024-02-21T23:08:14.446907+0000","last_unstale":"2024-02-21T23:08:15.466379+0000","last_undegraded":"2024-02-21T23:08:15.466379+0000","last_fullsized":"2024-02-21T23:08:15.466379+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:33:33.871286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.f","version":"121'1","reported_seq":15,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.429511+0000","last_change":"2024-02-21T23:08:13.427895+0000","last_active":"2024-02-21T23:08:15.429511+0000","last_peered":"2024-02-21T23:08:15.429511+0000","last_clean":"2024-02-21T23:08:15.429511+0000","last_became_active":"2024-02-21T23:08:13.427555+0000","last_became_peered":"2024-02-21T23:08:13.427555+0000","last_unstale":"2024-02-21T23:08:15.429511+0000","last_undegraded":"2024-02-21T23:08:15.429511+0000","last_fullsized":"2024-02-21T23:08:15.429511+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_clean_scrub_stamp":"2024-02-21T23:08:12.367049+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:51:31.193688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.466134+0000","last_change":"2024-02-21T23:08:14.460421+0000","last_active":"2024-02-21T23:08:15.466134+0000","last_peered":"2024-02-21T23:08:15.466134+0000","last_clean":"2024-02-21T23:08:15.466134+0000","last_became_active":"2024-02-21T23:08:14.460109+0000","last_became_peered":"2024-02-21T23:08:14.460109+0000","last_unstale":"2024-02-21T23:08:15.466134+0000","last_undegraded":"2024-02-21T23:08:15.466134+0000","last_fullsized":"2024-02-21T23:08:15.466134+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:48:35.819688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.e","version":"121'2","reported_seq":16,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.467795+0000","last_change":"2024-02-21T23:08:13.408230+0000","last_active":"2024-02-21T23:08:15.467795+0000","last_peered":"2024-02-21T23:08:15.467795+0000","last_clean":"2024-02-21T23:08:15.467795+0000","last_became_active":"2024-02-21T23:08:13.407502+0000","last_became_peered":"2024-02-21T23:08:13.407502+0000","last_unstale":"2024-02-21T23:08:15.467795+0000","last_undegraded":"2024-02-21T23:08:15.467795+0000","last_fullsized":"2024-02-21T23:08:15.467795+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_clean_scrub_stamp":"2024-02-21T23:08:12.367049+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:41:29.201521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.465621+0000","last_change":"2024-02-21T23:08:14.464546+0000","last_active":"2024-02-21T23:08:15.465621+0000","last_peered":"2024-02-21T23:08:15.465621+0000","last_clean":"2024-02-21T23:08:15.465621+0000","last_became_active":"2024-02-21T23:08:14.463305+0000","last_became_peered":"2024-02-21T23:08:14.463305+0000","last_unstale":"2024-02-21T23:08:15.465621+0000","last_undegraded":"2024-02-21T23:08:15.465621+0000","last_fullsized":"2024-02-21T23:08:15.465621+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:23:21.375840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.d","version":"121'1","reported_seq":15,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.425747+0000","last_change":"2024-02-21T23:08:13.428145+0000","last_active":"2024-02-21T23:08:15.425747+0000","last_peered":"2024-02-21T23:08:15.425747+0000","last_clean":"2024-02-21T23:08:15.425747+0000","last_became_active":"2024-02-21T23:08:13.427845+0000","last_became_peered":"2024-02-21T23:08:13.427845+0000","last_unstale":"2024-02-21T23:08:15.425747+0000","last_undegraded":"2024-02-21T23:08:15.425747+0000","last_fullsized":"2024-02-21T23:08:15.425747+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_clean_scrub_stamp":"2024-02-21T23:08:12.367049+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:50:42.102752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.b","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.469824+0000","last_change":"2024-02-21T23:08:14.444063+0000","last_active":"2024-02-21T23:08:15.469824+0000","last_peered":"2024-02-21T23:08:15.469824+0000","last_clean":"2024-02-21T23:08:15.469824+0000","last_became_active":"2024-02-21T23:08:14.441607+0000","last_became_peered":"2024-02-21T23:08:14.441607+0000","last_unstale":"2024-02-21T23:08:15.469824+0000","last_undegraded":"2024-02-21T23:08:15.469824+0000","last_fullsized":"2024-02-21T23:08:15.469824+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:37:25.248467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":14,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.467667+0000","last_change":"2024-02-21T23:08:13.407579+0000","last_active":"2024-02-21T23:08:15.467667+0000","last_peered":"2024-02-21T23:08:15.467667+0000","last_clean":"2024-02-21T23:08:15.467667+0000","last_became_active":"2024-02-21T23:08:13.407110+0000","last_became_peered":"2024-02-21T23:08:13.407110+0000","last_unstale":"2024-02-21T23:08:15.467667+0000","last_undegraded":"2024-02-21T23:08:15.467667+0000","last_fullsized":"2024-02-21T23:08:15.467667+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_clean_scrub_stamp":"2024-02-21T23:08:12.367049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:34:34.521904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.468247+0000","last_change":"2024-02-21T23:08:14.440614+0000","last_active":"2024-02-21T23:08:15.468247+0000","last_peered":"2024-02-21T23:08:15.468247+0000","last_clean":"2024-02-21T23:08:15.468247+0000","last_became_active":"2024-02-21T23:08:14.440352+0000","last_became_peered":"2024-02-21T23:08:14.440352+0000","last_unstale":"2024-02-21T23:08:15.468247+0000","last_undegraded":"2024-02-21T23:08:15.468247+0000","last_fullsized":"2024-02-21T23:08:15.468247+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:21:07.830802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.b","version":"121'2","reported_seq":16,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.427344+0000","last_change":"2024-02-21T23:08:13.426149+0000","last_active":"2024-02-21T23:08:15.427344+0000","last_peered":"2024-02-21T23:08:15.427344+0000","last_clean":"2024-02-21T23:08:15.427344+0000","last_became_active":"2024-02-21T23:08:13.425844+0000","last_became_peered":"2024-02-21T23:08:13.425844+0000","last_unstale":"2024-02-21T23:08:15.427344+0000","last_undegraded":"2024-02-21T23:08:15.427344+0000","last_fullsized":"2024-02-21T23:08:15.427344+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_clean_scrub_stamp":"2024-02-21T23:08:12.367049+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:50:42.102752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.d","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.468516+0000","last_change":"2024-02-21T23:08:14.461236+0000","last_active":"2024-02-21T23:08:15.468516+0000","last_peered":"2024-02-21T23:08:15.468516+0000","last_clean":"2024-02-21T23:08:15.468516+0000","last_became_active":"2024-02-21T23:08:14.460948+0000","last_became_peered":"2024-02-21T23:08:14.460948+0000","last_unstale":"2024-02-21T23:08:15.468516+0000","last_undegraded":"2024-02-21T23:08:15.468516+0000","last_fullsized":"2024-02-21T23:08:15.468516+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:44:33.355216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":14,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.468083+0000","last_change":"2024-02-21T23:08:13.429851+0000","last_active":"2024-02-21T23:08:15.468083+0000","last_peered":"2024-02-21T23:08:15.468083+0000","last_clean":"2024-02-21T23:08:15.468083+0000","last_became_active":"2024-02-21T23:08:13.428515+0000","last_became_peered":"2024-02-21T23:08:13.428515+0000","last_unstale":"2024-02-21T23:08:15.468083+0000","last_undegraded":"2024-02-21T23:08:15.468083+0000","last_fullsized":"2024-02-21T23:08:15.468083+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_clean_scrub_stamp":"2024-02-21T23:08:12.367049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:19:12.785951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.467137+0000","last_change":"2024-02-21T23:08:14.447164+0000","last_active":"2024-02-21T23:08:15.467137+0000","last_peered":"2024-02-21T23:08:15.467137+0000","last_clean":"2024-02-21T23:08:15.467137+0000","last_became_active":"2024-02-21T23:08:14.446677+0000","last_became_peered":"2024-02-21T23:08:14.446677+0000","last_unstale":"2024-02-21T23:08:15.467137+0000","last_undegraded":"2024-02-21T23:08:15.467137+0000","last_fullsized":"2024-02-21T23:08:15.467137+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:04:25.009442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":14,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.466406+0000","last_change":"2024-02-21T23:08:13.433304+0000","last_active":"2024-02-21T23:08:15.466406+0000","last_peered":"2024-02-21T23:08:15.466406+0000","last_clean":"2024-02-21T23:08:15.466406+0000","last_became_active":"2024-02-21T23:08:13.432958+0000","last_became_peered":"2024-02-21T23:08:13.432958+0000","last_unstale":"2024-02-21T23:08:15.466406+0000","last_undegraded":"2024-02-21T23:08:15.466406+0000","last_fullsized":"2024-02-21T23:08:15.466406+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_clean_scrub_stamp":"2024-02-21T23:08:12.367049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:01:33.074562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.429342+0000","last_change":"2024-02-21T23:08:14.451366+0000","last_active":"2024-02-21T23:08:15.429342+0000","last_peered":"2024-02-21T23:08:15.429342+0000","last_clean":"2024-02-21T23:08:15.429342+0000","last_became_active":"2024-02-21T23:08:14.451043+0000","last_became_peered":"2024-02-21T23:08:14.451043+0000","last_unstale":"2024-02-21T23:08:15.429342+0000","last_undegraded":"2024-02-21T23:08:15.429342+0000","last_fullsized":"2024-02-21T23:08:15.429342+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:15:18.656888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":14,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.424745+0000","last_change":"2024-02-21T23:08:13.427040+0000","last_active":"2024-02-21T23:08:15.424745+0000","last_peered":"2024-02-21T23:08:15.424745+0000","last_clean":"2024-02-21T23:08:15.424745+0000","last_became_active":"2024-02-21T23:08:13.426611+0000","last_became_peered":"2024-02-21T23:08:13.426611+0000","last_unstale":"2024-02-21T23:08:15.424745+0000","last_undegraded":"2024-02-21T23:08:15.424745+0000","last_fullsized":"2024-02-21T23:08:15.424745+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_clean_scrub_stamp":"2024-02-21T23:08:12.367049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:15:31.378104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.466307+0000","last_change":"2024-02-21T23:08:14.461202+0000","last_active":"2024-02-21T23:08:15.466307+0000","last_peered":"2024-02-21T23:08:15.466307+0000","last_clean":"2024-02-21T23:08:15.466307+0000","last_became_active":"2024-02-21T23:08:14.458784+0000","last_became_peered":"2024-02-21T23:08:14.458784+0000","last_unstale":"2024-02-21T23:08:15.466307+0000","last_undegraded":"2024-02-21T23:08:15.466307+0000","last_fullsized":"2024-02-21T23:08:15.466307+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:55:20.800793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.7","version":"121'2","reported_seq":16,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.468244+0000","last_change":"2024-02-21T23:08:13.418201+0000","last_active":"2024-02-21T23:08:15.468244+0000","last_peered":"2024-02-21T23:08:15.468244+0000","last_clean":"2024-02-21T23:08:15.468244+0000","last_became_active":"2024-02-21T23:08:13.417467+0000","last_became_peered":"2024-02-21T23:08:13.417467+0000","last_unstale":"2024-02-21T23:08:15.468244+0000","last_undegraded":"2024-02-21T23:08:15.468244+0000","last_fullsized":"2024-02-21T23:08:15.468244+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_clean_scrub_stamp":"2024-02-21T23:08:12.367049+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:22:17.723754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.466674+0000","last_change":"2024-02-21T23:08:14.449840+0000","last_active":"2024-02-21T23:08:15.466674+0000","last_peered":"2024-02-21T23:08:15.466674+0000","last_clean":"2024-02-21T23:08:15.466674+0000","last_became_active":"2024-02-21T23:08:14.448908+0000","last_became_peered":"2024-02-21T23:08:14.448908+0000","last_unstale":"2024-02-21T23:08:15.466674+0000","last_undegraded":"2024-02-21T23:08:15.466674+0000","last_fullsized":"2024-02-21T23:08:15.466674+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:13:06.500761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":14,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.428812+0000","last_change":"2024-02-21T23:08:13.426738+0000","last_active":"2024-02-21T23:08:15.428812+0000","last_peered":"2024-02-21T23:08:15.428812+0000","last_clean":"2024-02-21T23:08:15.428812+0000","last_became_active":"2024-02-21T23:08:13.426454+0000","last_became_peered":"2024-02-21T23:08:13.426454+0000","last_unstale":"2024-02-21T23:08:15.428812+0000","last_undegraded":"2024-02-21T23:08:15.428812+0000","last_fullsized":"2024-02-21T23:08:15.428812+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_clean_scrub_stamp":"2024-02-21T23:08:12.367049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:22:55.245029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.465644+0000","last_change":"2024-02-21T23:08:14.450828+0000","last_active":"2024-02-21T23:08:15.465644+0000","last_peered":"2024-02-21T23:08:15.465644+0000","last_clean":"2024-02-21T23:08:15.465644+0000","last_became_active":"2024-02-21T23:08:14.450513+0000","last_became_peered":"2024-02-21T23:08:14.450513+0000","last_unstale":"2024-02-21T23:08:15.465644+0000","last_undegraded":"2024-02-21T23:08:15.465644+0000","last_fullsized":"2024-02-21T23:08:15.465644+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:59:04.556496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":14,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.427905+0000","last_change":"2024-02-21T23:08:13.429362+0000","last_active":"2024-02-21T23:08:15.427905+0000","last_peered":"2024-02-21T23:08:15.427905+0000","last_clean":"2024-02-21T23:08:15.427905+0000","last_became_active":"2024-02-21T23:08:13.429088+0000","last_became_peered":"2024-02-21T23:08:13.429088+0000","last_unstale":"2024-02-21T23:08:15.427905+0000","last_undegraded":"2024-02-21T23:08:15.427905+0000","last_fullsized":"2024-02-21T23:08:15.427905+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_clean_scrub_stamp":"2024-02-21T23:08:12.367049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:57:40.070790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.470096+0000","last_change":"2024-02-21T23:08:14.434316+0000","last_active":"2024-02-21T23:08:15.470096+0000","last_peered":"2024-02-21T23:08:15.470096+0000","last_clean":"2024-02-21T23:08:15.470096+0000","last_became_active":"2024-02-21T23:08:14.433537+0000","last_became_peered":"2024-02-21T23:08:14.433537+0000","last_unstale":"2024-02-21T23:08:15.470096+0000","last_undegraded":"2024-02-21T23:08:15.470096+0000","last_fullsized":"2024-02-21T23:08:15.470096+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:02:26.010423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":14,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.425217+0000","last_change":"2024-02-21T23:08:13.431471+0000","last_active":"2024-02-21T23:08:15.425217+0000","last_peered":"2024-02-21T23:08:15.425217+0000","last_clean":"2024-02-21T23:08:15.425217+0000","last_became_active":"2024-02-21T23:08:13.431206+0000","last_became_peered":"2024-02-21T23:08:13.431206+0000","last_unstale":"2024-02-21T23:08:15.425217+0000","last_undegraded":"2024-02-21T23:08:15.425217+0000","last_fullsized":"2024-02-21T23:08:15.425217+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_clean_scrub_stamp":"2024-02-21T23:08:12.367049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:27:41.020802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.425359+0000","last_change":"2024-02-21T23:08:14.455407+0000","last_active":"2024-02-21T23:08:15.425359+0000","last_peered":"2024-02-21T23:08:15.425359+0000","last_clean":"2024-02-21T23:08:15.425359+0000","last_became_active":"2024-02-21T23:08:14.455113+0000","last_became_peered":"2024-02-21T23:08:14.455113+0000","last_unstale":"2024-02-21T23:08:15.425359+0000","last_undegraded":"2024-02-21T23:08:15.425359+0000","last_fullsized":"2024-02-21T23:08:15.425359+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:54:54.757289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":14,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.428459+0000","last_change":"2024-02-21T23:08:13.423445+0000","last_active":"2024-02-21T23:08:15.428459+0000","last_peered":"2024-02-21T23:08:15.428459+0000","last_clean":"2024-02-21T23:08:15.428459+0000","last_became_active":"2024-02-21T23:08:13.423171+0000","last_became_peered":"2024-02-21T23:08:13.423171+0000","last_unstale":"2024-02-21T23:08:15.428459+0000","last_undegraded":"2024-02-21T23:08:15.428459+0000","last_fullsized":"2024-02-21T23:08:15.428459+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_clean_scrub_stamp":"2024-02-21T23:08:12.367049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:15:31.378104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.425449+0000","last_change":"2024-02-21T23:08:14.460611+0000","last_active":"2024-02-21T23:08:15.425449+0000","last_peered":"2024-02-21T23:08:15.425449+0000","last_clean":"2024-02-21T23:08:15.425449+0000","last_became_active":"2024-02-21T23:08:14.460196+0000","last_became_peered":"2024-02-21T23:08:14.460196+0000","last_unstale":"2024-02-21T23:08:15.425449+0000","last_undegraded":"2024-02-21T23:08:15.425449+0000","last_fullsized":"2024-02-21T23:08:15.425449+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:15:18.656888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":14,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.465758+0000","last_change":"2024-02-21T23:08:13.426964+0000","last_active":"2024-02-21T23:08:15.465758+0000","last_peered":"2024-02-21T23:08:15.465758+0000","last_clean":"2024-02-21T23:08:15.465758+0000","last_became_active":"2024-02-21T23:08:13.426252+0000","last_became_peered":"2024-02-21T23:08:13.426252+0000","last_unstale":"2024-02-21T23:08:15.465758+0000","last_undegraded":"2024-02-21T23:08:15.465758+0000","last_fullsized":"2024-02-21T23:08:15.465758+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_clean_scrub_stamp":"2024-02-21T23:08:12.367049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:08:38.909202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.425329+0000","last_change":"2024-02-21T23:08:14.460277+0000","last_active":"2024-02-21T23:08:15.425329+0000","last_peered":"2024-02-21T23:08:15.425329+0000","last_clean":"2024-02-21T23:08:15.425329+0000","last_became_active":"2024-02-21T23:08:14.459966+0000","last_became_peered":"2024-02-21T23:08:14.459966+0000","last_unstale":"2024-02-21T23:08:15.425329+0000","last_undegraded":"2024-02-21T23:08:15.425329+0000","last_fullsized":"2024-02-21T23:08:15.425329+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:54:23.719045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":14,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.425193+0000","last_change":"2024-02-21T23:08:13.431443+0000","last_active":"2024-02-21T23:08:15.425193+0000","last_peered":"2024-02-21T23:08:15.425193+0000","last_clean":"2024-02-21T23:08:15.425193+0000","last_became_active":"2024-02-21T23:08:13.431140+0000","last_became_peered":"2024-02-21T23:08:13.431140+0000","last_unstale":"2024-02-21T23:08:15.425193+0000","last_undegraded":"2024-02-21T23:08:15.425193+0000","last_fullsized":"2024-02-21T23:08:15.425193+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_clean_scrub_stamp":"2024-02-21T23:08:12.367049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:22:55.245029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.7","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.467896+0000","last_change":"2024-02-21T23:08:14.458665+0000","last_active":"2024-02-21T23:08:15.467896+0000","last_peered":"2024-02-21T23:08:15.467896+0000","last_clean":"2024-02-21T23:08:15.467896+0000","last_became_active":"2024-02-21T23:08:14.457434+0000","last_became_peered":"2024-02-21T23:08:14.457434+0000","last_unstale":"2024-02-21T23:08:15.467896+0000","last_undegraded":"2024-02-21T23:08:15.467896+0000","last_fullsized":"2024-02-21T23:08:15.467896+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:16:13.065511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":14,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.466480+0000","last_change":"2024-02-21T23:08:13.408639+0000","last_active":"2024-02-21T23:08:15.466480+0000","last_peered":"2024-02-21T23:08:15.466480+0000","last_clean":"2024-02-21T23:08:15.466480+0000","last_became_active":"2024-02-21T23:08:13.407970+0000","last_became_peered":"2024-02-21T23:08:13.407970+0000","last_unstale":"2024-02-21T23:08:15.466480+0000","last_undegraded":"2024-02-21T23:08:15.466480+0000","last_fullsized":"2024-02-21T23:08:15.466480+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_clean_scrub_stamp":"2024-02-21T23:08:12.367049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:32:22.922633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.15","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.466897+0000","last_change":"2024-02-21T23:08:14.459300+0000","last_active":"2024-02-21T23:08:15.466897+0000","last_peered":"2024-02-21T23:08:15.466897+0000","last_clean":"2024-02-21T23:08:15.466897+0000","last_became_active":"2024-02-21T23:08:14.458773+0000","last_became_peered":"2024-02-21T23:08:14.458773+0000","last_unstale":"2024-02-21T23:08:15.466897+0000","last_undegraded":"2024-02-21T23:08:15.466897+0000","last_fullsized":"2024-02-21T23:08:15.466897+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:08:39.925583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":14,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.465443+0000","last_change":"2024-02-21T23:08:13.431004+0000","last_active":"2024-02-21T23:08:15.465443+0000","last_peered":"2024-02-21T23:08:15.465443+0000","last_clean":"2024-02-21T23:08:15.465443+0000","last_became_active":"2024-02-21T23:08:13.430635+0000","last_became_peered":"2024-02-21T23:08:13.430635+0000","last_unstale":"2024-02-21T23:08:15.465443+0000","last_undegraded":"2024-02-21T23:08:15.465443+0000","last_fullsized":"2024-02-21T23:08:15.465443+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_clean_scrub_stamp":"2024-02-21T23:08:12.367049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:20:48.814520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":361,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.464970+0000","last_change":"2024-02-21T23:02:32.289335+0000","last_active":"2024-02-21T23:08:15.464970+0000","last_peered":"2024-02-21T23:08:15.464970+0000","last_clean":"2024-02-21T23:08:15.464970+0000","last_became_active":"2024-02-21T23:02:32.286913+0000","last_became_peered":"2024-02-21T23:02:32.286913+0000","last_unstale":"2024-02-21T23:08:15.464970+0000","last_undegraded":"2024-02-21T23:08:15.464970+0000","last_fullsized":"2024-02-21T23:08:15.464970+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_clean_scrub_stamp":"2024-02-21T23:02:31.172569+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:36:23.987374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":20,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":19,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574919,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32760,"kb_used_data":1572,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339080,"statfs":{"total":96636764160,"available":96603217920,"internally_reserved":0,"allocated":1609728,"data_stored":1335576,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574920,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32676,"kb_used_data":1488,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339164,"statfs":{"total":96636764160,"available":96603303936,"internally_reserved":0,"allocated":1523712,"data_stored":737486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574919,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32664,"kb_used_data":1484,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339176,"statfs":{"total":96636764160,"available":96603316224,"internally_reserved":0,"allocated":1519616,"data_stored":736836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574920,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32660,"kb_used_data":1480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339180,"statfs":{"total":96636764160,"available":96603320320,"internally_reserved":0,"allocated":1515520,"data_stored":736814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574920,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32676,"kb_used_data":1496,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339164,"statfs":{"total":96636764160,"available":96603303936,"internally_reserved":0,"allocated":1531904,"data_stored":738496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574921,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32668,"kb_used_data":1488,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339172,"statfs":{"total":96636764160,"available":96603312128,"internally_reserved":0,"allocated":1523712,"data_stored":737430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574919,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32680,"kb_used_data":1496,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339160,"statfs":{"total":96636764160,"available":96603299840,"internally_reserved":0,"allocated":1531904,"data_stored":743576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574919,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32736,"kb_used_data":1552,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339104,"statfs":{"total":96636764160,"available":96603242496,"internally_reserved":0,"allocated":1589248,"data_stored":1327238,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-21T23:08:18.920 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-21T23:08:19.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.191+0000 7f63e8c00640 1 Processor -- start 2024-02-21T23:08:19.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.191+0000 7f63e8c00640 1 -- start start 2024-02-21T23:08:19.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.193+0000 7f63e8c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f63e40a5f40 0x7f63e406dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:19.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.193+0000 7f63e8c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f63e40c0270 0x7f63e40c0700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:19.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.193+0000 7f63e8c00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63e40c0c40 con 0x7f63e40c3490 2024-02-21T23:08:19.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.193+0000 7f63e8c00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63e406eff0 con 0x7f63e40c0270 2024-02-21T23:08:19.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.193+0000 7f63e8c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e40bfde0 con 0x7f63e40a5f40 2024-02-21T23:08:19.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.193+0000 7f63e1e00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f63e40c3490 0x7f63e4152140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.28:57616/0 (socket says 172.21.15.28:57616) 2024-02-21T23:08:19.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.193+0000 7f63e1e00640 1 -- 172.21.15.28:0/1423984237 learned_addr learned my addr 172.21.15.28:0/1423984237 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:08:19.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.194+0000 7f63e2800640 1 --2- 172.21.15.28:0/1423984237 >> v2:172.21.15.81:3300/0 conn(0x7f63e40c0270 0x7f63e40c0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:19.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.194+0000 7f63dbe00640 1 --2- 172.21.15.28:0/1423984237 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f63e40a5f40 0x7f63e406dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:19.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.194+0000 7f63e1400640 1 -- 172.21.15.28:0/1423984237 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1462363321 0 0) 0x7f63e40c0c40 con 0x7f63e40c3490 2024-02-21T23:08:19.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.194+0000 7f63e1400640 1 -- 172.21.15.28:0/1423984237 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63c8003520 con 0x7f63e40c3490 2024-02-21T23:08:19.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.194+0000 7f63e2800640 1 -- 172.21.15.28:0/1423984237 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f63e40a5f40 msgr2=0x7f63e406dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:19.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.194+0000 7f63e2800640 1 --2- 172.21.15.28:0/1423984237 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f63e40a5f40 0x7f63e406dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:19.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.194+0000 7f63e2800640 1 -- 172.21.15.28:0/1423984237 >> v1:172.21.15.81:6789/0 conn(0x7f63e40c3490 legacy=0x7f63e4152140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:19.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.194+0000 7f63e2800640 1 -- 172.21.15.28:0/1423984237 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63e406e750 con 0x7f63e40c0270 2024-02-21T23:08:19.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.194+0000 7f63dbe00640 1 --2- 172.21.15.28:0/1423984237 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f63e40a5f40 0x7f63e406dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-21T23:08:19.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.196+0000 7f63e2800640 1 --2- 172.21.15.28:0/1423984237 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63e40c0270 0x7f63e40c0700 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f63dc009d90 tx=0x7f63dc02be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cafa99c96925f171 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:19.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.196+0000 7f63e1400640 1 -- 172.21.15.28:0/1423984237 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63dc039070 con 0x7f63e40c0270 2024-02-21T23:08:19.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.196+0000 7f63e1400640 1 -- 172.21.15.28:0/1423984237 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63dc02cbd0 con 0x7f63e40c0270 2024-02-21T23:08:19.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.196+0000 7f63e1400640 1 -- 172.21.15.28:0/1423984237 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63dc034440 con 0x7f63e40c0270 2024-02-21T23:08:19.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.197+0000 7f63e8c00640 1 -- 172.21.15.28:0/1423984237 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63e40c0270 msgr2=0x7f63e40c0700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:19.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.197+0000 7f63e8c00640 1 --2- 172.21.15.28:0/1423984237 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63e40c0270 0x7f63e40c0700 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f63dc009d90 tx=0x7f63dc02be20 comp rx=0 tx=0).stop 2024-02-21T23:08:19.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.197+0000 7f63e8c00640 1 -- 172.21.15.28:0/1423984237 shutdown_connections 2024-02-21T23:08:19.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.197+0000 7f63e8c00640 1 --2- 172.21.15.28:0/1423984237 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63e40c0270 0x7f63e40c0700 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:19.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.197+0000 7f63e8c00640 1 --2- 172.21.15.28:0/1423984237 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f63e40a5f40 0x7f63e406dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:19.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.197+0000 7f63e8c00640 1 -- 172.21.15.28:0/1423984237 >> 172.21.15.28:0/1423984237 conn(0x7f63e40c67f0 msgr2=0x7f63e40c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:19.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.197+0000 7f63e8c00640 1 -- 172.21.15.28:0/1423984237 shutdown_connections 2024-02-21T23:08:19.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.197+0000 7f63e8c00640 1 -- 172.21.15.28:0/1423984237 wait complete. 2024-02-21T23:08:19.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.199+0000 7f63e8c00640 1 Processor -- start 2024-02-21T23:08:19.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.199+0000 7f63e8c00640 1 -- start start 2024-02-21T23:08:19.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.200+0000 7f63e8c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63e40a5f40 0x7f63e4151270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:19.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.200+0000 7f63e8c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63e40c0270 0x7f63e41ec0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:19.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.200+0000 7f63e8c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f63e40c3490 0x7f63e41ec620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:19.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.200+0000 7f63e8c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e406ed10 con 0x7f63e40c0270 2024-02-21T23:08:19.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.200+0000 7f63e8c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e406e4f0 con 0x7f63e40a5f40 2024-02-21T23:08:19.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.200+0000 7f63e8c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e40c1c70 con 0x7f63e40c3490 2024-02-21T23:08:19.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.200+0000 7f63dbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63e40c0270 0x7f63e41ec0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:19.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.200+0000 7f63dbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63e40c0270 0x7f63e41ec0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57622/0 (socket says 172.21.15.28:57622) 2024-02-21T23:08:19.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.200+0000 7f63dbe00640 1 -- 172.21.15.28:0/904105530 learned_addr learned my addr 172.21.15.28:0/904105530 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:19.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.200+0000 7f63e1e00640 1 --2- 172.21.15.28:0/904105530 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63e40a5f40 0x7f63e4151270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:19.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.201+0000 7f63e2800640 1 --2- 172.21.15.28:0/904105530 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f63e40c3490 0x7f63e41ec620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:19.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.201+0000 7f63dbe00640 1 -- 172.21.15.28:0/904105530 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f63e40c3490 msgr2=0x7f63e41ec620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:19.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.201+0000 7f63dbe00640 1 --2- 172.21.15.28:0/904105530 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f63e40c3490 0x7f63e41ec620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:19.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.201+0000 7f63dbe00640 1 -- 172.21.15.28:0/904105530 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63e40a5f40 msgr2=0x7f63e4151270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:19.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.201+0000 7f63dbe00640 1 --2- 172.21.15.28:0/904105530 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63e40a5f40 0x7f63e4151270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:19.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.201+0000 7f63dbe00640 1 -- 172.21.15.28:0/904105530 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63e414d200 con 0x7f63e40c0270 2024-02-21T23:08:19.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.201+0000 7f63dbe00640 1 --2- 172.21.15.28:0/904105530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63e40c0270 0x7f63e41ec0e0 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f63cc00e990 tx=0x7f63cc00ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:19.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.201+0000 7f63da000640 1 -- 172.21.15.28:0/904105530 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63cc017070 con 0x7f63e40c0270 2024-02-21T23:08:19.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.201+0000 7f63da000640 1 -- 172.21.15.28:0/904105530 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63cc00c9e0 con 0x7f63e40c0270 2024-02-21T23:08:19.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.201+0000 7f63e8c00640 1 -- 172.21.15.28:0/904105530 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63e4156490 con 0x7f63e40c0270 2024-02-21T23:08:19.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.202+0000 7f63da000640 1 -- 172.21.15.28:0/904105530 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63cc00cdb0 con 0x7f63e40c0270 2024-02-21T23:08:19.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.202+0000 7f63e8c00640 1 -- 172.21.15.28:0/904105530 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63e41517b0 con 0x7f63e40c0270 2024-02-21T23:08:19.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.203+0000 7f63e8c00640 1 -- 172.21.15.28:0/904105530 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63e414d010 con 0x7f63e40c0270 2024-02-21T23:08:19.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.204+0000 7f63da000640 1 -- 172.21.15.28:0/904105530 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f63cc01a070 con 0x7f63e40c0270 2024-02-21T23:08:19.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.205+0000 7f63da000640 1 --2- 172.21.15.28:0/904105530 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f63c006b500 0x7f63c006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:19.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.205+0000 7f63da000640 1 -- 172.21.15.28:0/904105530 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f63cc08a240 con 0x7f63e40c0270 2024-02-21T23:08:19.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.205+0000 7f63e1e00640 1 --2- 172.21.15.28:0/904105530 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f63c006b500 0x7f63c006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:19.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.206+0000 7f63e1e00640 1 --2- 172.21.15.28:0/904105530 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f63c006b500 0x7f63c006d9c0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f63d4005880 tx=0x7f63d4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:19.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.209+0000 7f63da000640 1 -- 172.21.15.28:0/904105530 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f63cc058f20 con 0x7f63e40c0270 2024-02-21T23:08:19.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.404+0000 7f63e8c00640 1 -- 172.21.15.28:0/904105530 --> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f63e4067f30 con 0x7f63c006b500 2024-02-21T23:08:19.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.422+0000 7f63da000640 1 -- 172.21.15.28:0/904105530 <== mgr.4101 v2:172.21.15.81:6832/1848538577 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252559 (secure 0 0 0) 0x7f63e4067f30 con 0x7f63c006b500 2024-02-21T23:08:19.413 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:08:19.419 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-21T23:08:19.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.429+0000 7f63e8c00640 1 -- 172.21.15.28:0/904105530 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f63c006b500 msgr2=0x7f63c006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:19.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.429+0000 7f63e8c00640 1 --2- 172.21.15.28:0/904105530 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f63c006b500 0x7f63c006d9c0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f63d4005880 tx=0x7f63d4009040 comp rx=0 tx=0).stop 2024-02-21T23:08:19.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.429+0000 7f63e8c00640 1 -- 172.21.15.28:0/904105530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63e40c0270 msgr2=0x7f63e41ec0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:19.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.429+0000 7f63e8c00640 1 --2- 172.21.15.28:0/904105530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63e40c0270 0x7f63e41ec0e0 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f63cc00e990 tx=0x7f63cc00ee50 comp rx=0 tx=0).stop 2024-02-21T23:08:19.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.429+0000 7f63e8c00640 1 -- 172.21.15.28:0/904105530 shutdown_connections 2024-02-21T23:08:19.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.429+0000 7f63e8c00640 1 --2- 172.21.15.28:0/904105530 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f63c006b500 0x7f63c006d9c0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:19.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.429+0000 7f63e8c00640 1 --2- 172.21.15.28:0/904105530 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f63e40c3490 0x7f63e41ec620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:19.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.429+0000 7f63e8c00640 1 --2- 172.21.15.28:0/904105530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63e40c0270 0x7f63e41ec0e0 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:19.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.429+0000 7f63e8c00640 1 --2- 172.21.15.28:0/904105530 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63e40a5f40 0x7f63e4151270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:19.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.429+0000 7f63e8c00640 1 -- 172.21.15.28:0/904105530 >> 172.21.15.28:0/904105530 conn(0x7f63e40c67f0 msgr2=0x7f63e414bf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:19.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.429+0000 7f63e8c00640 1 -- 172.21.15.28:0/904105530 shutdown_connections 2024-02-21T23:08:19.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.429+0000 7f63e8c00640 1 -- 172.21.15.28:0/904105530 wait complete. 2024-02-21T23:08:19.437 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":289,"stamp":"2024-02-21T23:08:18.785233+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":128,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":261520,"kb_used_data":12056,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754713200,"statfs":{"total":773094113280,"available":772826316800,"internally_reserved":0,"allocated":12345344,"data_stored":7093452,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002031"},"pg_stats":[{"pgid":"20.3f","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.467762+0000","last_change":"2024-02-21T23:08:14.449279+0000","last_active":"2024-02-21T23:08:15.467762+0000","last_peered":"2024-02-21T23:08:15.467762+0000","last_clean":"2024-02-21T23:08:15.467762+0000","last_became_active":"2024-02-21T23:08:14.448996+0000","last_became_peered":"2024-02-21T23:08:14.448996+0000","last_unstale":"2024-02-21T23:08:15.467762+0000","last_undegraded":"2024-02-21T23:08:15.467762+0000","last_fullsized":"2024-02-21T23:08:15.467762+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:37:10.848059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3e","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.429113+0000","last_change":"2024-02-21T23:08:14.441342+0000","last_active":"2024-02-21T23:08:15.429113+0000","last_peered":"2024-02-21T23:08:15.429113+0000","last_clean":"2024-02-21T23:08:15.429113+0000","last_became_active":"2024-02-21T23:08:14.441058+0000","last_became_peered":"2024-02-21T23:08:14.441058+0000","last_unstale":"2024-02-21T23:08:15.429113+0000","last_undegraded":"2024-02-21T23:08:15.429113+0000","last_fullsized":"2024-02-21T23:08:15.429113+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:54:54.757289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3d","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.466750+0000","last_change":"2024-02-21T23:08:14.459063+0000","last_active":"2024-02-21T23:08:15.466750+0000","last_peered":"2024-02-21T23:08:15.466750+0000","last_clean":"2024-02-21T23:08:15.466750+0000","last_became_active":"2024-02-21T23:08:14.458089+0000","last_became_peered":"2024-02-21T23:08:14.458089+0000","last_unstale":"2024-02-21T23:08:15.466750+0000","last_undegraded":"2024-02-21T23:08:15.466750+0000","last_fullsized":"2024-02-21T23:08:15.466750+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:20:49.830901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.3c","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.428625+0000","last_change":"2024-02-21T23:08:14.449246+0000","last_active":"2024-02-21T23:08:15.428625+0000","last_peered":"2024-02-21T23:08:15.428625+0000","last_clean":"2024-02-21T23:08:15.428625+0000","last_became_active":"2024-02-21T23:08:14.448193+0000","last_became_peered":"2024-02-21T23:08:14.448193+0000","last_unstale":"2024-02-21T23:08:15.428625+0000","last_undegraded":"2024-02-21T23:08:15.428625+0000","last_fullsized":"2024-02-21T23:08:15.428625+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:03:12.105566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3b","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.466240+0000","last_change":"2024-02-21T23:08:14.447763+0000","last_active":"2024-02-21T23:08:15.466240+0000","last_peered":"2024-02-21T23:08:15.466240+0000","last_clean":"2024-02-21T23:08:15.466240+0000","last_became_active":"2024-02-21T23:08:14.447343+0000","last_became_peered":"2024-02-21T23:08:14.447343+0000","last_unstale":"2024-02-21T23:08:15.466240+0000","last_undegraded":"2024-02-21T23:08:15.466240+0000","last_fullsized":"2024-02-21T23:08:15.466240+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:10:37.708703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.3a","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.424848+0000","last_change":"2024-02-21T23:08:14.463733+0000","last_active":"2024-02-21T23:08:15.424848+0000","last_peered":"2024-02-21T23:08:15.424848+0000","last_clean":"2024-02-21T23:08:15.424848+0000","last_became_active":"2024-02-21T23:08:14.462259+0000","last_became_peered":"2024-02-21T23:08:14.462259+0000","last_unstale":"2024-02-21T23:08:15.424848+0000","last_undegraded":"2024-02-21T23:08:15.424848+0000","last_fullsized":"2024-02-21T23:08:15.424848+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:01:34.090943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.39","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.465791+0000","last_change":"2024-02-21T23:08:14.452506+0000","last_active":"2024-02-21T23:08:15.465791+0000","last_peered":"2024-02-21T23:08:15.465791+0000","last_clean":"2024-02-21T23:08:15.465791+0000","last_became_active":"2024-02-21T23:08:14.452226+0000","last_became_peered":"2024-02-21T23:08:14.452226+0000","last_unstale":"2024-02-21T23:08:15.465791+0000","last_undegraded":"2024-02-21T23:08:15.465791+0000","last_fullsized":"2024-02-21T23:08:15.465791+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:44:15.626046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.38","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.426646+0000","last_change":"2024-02-21T23:08:14.442162+0000","last_active":"2024-02-21T23:08:15.426646+0000","last_peered":"2024-02-21T23:08:15.426646+0000","last_clean":"2024-02-21T23:08:15.426646+0000","last_became_active":"2024-02-21T23:08:14.441850+0000","last_became_peered":"2024-02-21T23:08:14.441850+0000","last_unstale":"2024-02-21T23:08:15.426646+0000","last_undegraded":"2024-02-21T23:08:15.426646+0000","last_fullsized":"2024-02-21T23:08:15.426646+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:13:06.500761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.37","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.425783+0000","last_change":"2024-02-21T23:08:14.455069+0000","last_active":"2024-02-21T23:08:15.425783+0000","last_peered":"2024-02-21T23:08:15.425783+0000","last_clean":"2024-02-21T23:08:15.425783+0000","last_became_active":"2024-02-21T23:08:14.454196+0000","last_became_peered":"2024-02-21T23:08:14.454196+0000","last_unstale":"2024-02-21T23:08:15.425783+0000","last_undegraded":"2024-02-21T23:08:15.425783+0000","last_fullsized":"2024-02-21T23:08:15.425783+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:01:18.703812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.36","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.468521+0000","last_change":"2024-02-21T23:08:14.432934+0000","last_active":"2024-02-21T23:08:15.468521+0000","last_peered":"2024-02-21T23:08:15.468521+0000","last_clean":"2024-02-21T23:08:15.468521+0000","last_became_active":"2024-02-21T23:08:14.432607+0000","last_became_peered":"2024-02-21T23:08:14.432607+0000","last_unstale":"2024-02-21T23:08:15.468521+0000","last_undegraded":"2024-02-21T23:08:15.468521+0000","last_fullsized":"2024-02-21T23:08:15.468521+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:40:43.618038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.35","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.428637+0000","last_change":"2024-02-21T23:08:14.451533+0000","last_active":"2024-02-21T23:08:15.428637+0000","last_peered":"2024-02-21T23:08:15.428637+0000","last_clean":"2024-02-21T23:08:15.428637+0000","last_became_active":"2024-02-21T23:08:14.450545+0000","last_became_peered":"2024-02-21T23:08:14.450545+0000","last_unstale":"2024-02-21T23:08:15.428637+0000","last_undegraded":"2024-02-21T23:08:15.428637+0000","last_fullsized":"2024-02-21T23:08:15.428637+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:54:23.719045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.34","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.468972+0000","last_change":"2024-02-21T23:08:14.450718+0000","last_active":"2024-02-21T23:08:15.468972+0000","last_peered":"2024-02-21T23:08:15.468972+0000","last_clean":"2024-02-21T23:08:15.468972+0000","last_became_active":"2024-02-21T23:08:14.450413+0000","last_became_peered":"2024-02-21T23:08:14.450413+0000","last_unstale":"2024-02-21T23:08:15.468972+0000","last_undegraded":"2024-02-21T23:08:15.468972+0000","last_fullsized":"2024-02-21T23:08:15.468972+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:43:19.437727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.33","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.425204+0000","last_change":"2024-02-21T23:08:14.466078+0000","last_active":"2024-02-21T23:08:15.425204+0000","last_peered":"2024-02-21T23:08:15.425204+0000","last_clean":"2024-02-21T23:08:15.425204+0000","last_became_active":"2024-02-21T23:08:14.463223+0000","last_became_peered":"2024-02-21T23:08:14.463223+0000","last_unstale":"2024-02-21T23:08:15.425204+0000","last_undegraded":"2024-02-21T23:08:15.425204+0000","last_fullsized":"2024-02-21T23:08:15.425204+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:03:12.105566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.32","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.467963+0000","last_change":"2024-02-21T23:08:14.455548+0000","last_active":"2024-02-21T23:08:15.467963+0000","last_peered":"2024-02-21T23:08:15.467963+0000","last_clean":"2024-02-21T23:08:15.467963+0000","last_became_active":"2024-02-21T23:08:14.455262+0000","last_became_peered":"2024-02-21T23:08:14.455262+0000","last_unstale":"2024-02-21T23:08:15.467963+0000","last_undegraded":"2024-02-21T23:08:15.467963+0000","last_fullsized":"2024-02-21T23:08:15.467963+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:29:47.674102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.31","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.467197+0000","last_change":"2024-02-21T23:08:14.462237+0000","last_active":"2024-02-21T23:08:15.467197+0000","last_peered":"2024-02-21T23:08:15.467197+0000","last_clean":"2024-02-21T23:08:15.467197+0000","last_became_active":"2024-02-21T23:08:14.461701+0000","last_became_peered":"2024-02-21T23:08:14.461701+0000","last_unstale":"2024-02-21T23:08:15.467197+0000","last_undegraded":"2024-02-21T23:08:15.467197+0000","last_fullsized":"2024-02-21T23:08:15.467197+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:00:49.516505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.30","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.424537+0000","last_change":"2024-02-21T23:08:14.445396+0000","last_active":"2024-02-21T23:08:15.424537+0000","last_peered":"2024-02-21T23:08:15.424537+0000","last_clean":"2024-02-21T23:08:15.424537+0000","last_became_active":"2024-02-21T23:08:14.445101+0000","last_became_peered":"2024-02-21T23:08:14.445101+0000","last_unstale":"2024-02-21T23:08:15.424537+0000","last_undegraded":"2024-02-21T23:08:15.424537+0000","last_fullsized":"2024-02-21T23:08:15.424537+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:16:07.331005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.2f","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.425812+0000","last_change":"2024-02-21T23:08:14.452042+0000","last_active":"2024-02-21T23:08:15.425812+0000","last_peered":"2024-02-21T23:08:15.425812+0000","last_clean":"2024-02-21T23:08:15.425812+0000","last_became_active":"2024-02-21T23:08:14.451684+0000","last_became_peered":"2024-02-21T23:08:14.451684+0000","last_unstale":"2024-02-21T23:08:15.425812+0000","last_undegraded":"2024-02-21T23:08:15.425812+0000","last_fullsized":"2024-02-21T23:08:15.425812+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:30:39.146916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.2e","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.466508+0000","last_change":"2024-02-21T23:08:14.440941+0000","last_active":"2024-02-21T23:08:15.466508+0000","last_peered":"2024-02-21T23:08:15.466508+0000","last_clean":"2024-02-21T23:08:15.466508+0000","last_became_active":"2024-02-21T23:08:14.440448+0000","last_became_peered":"2024-02-21T23:08:14.440448+0000","last_unstale":"2024-02-21T23:08:15.466508+0000","last_undegraded":"2024-02-21T23:08:15.466508+0000","last_fullsized":"2024-02-21T23:08:15.466508+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:59:31.892130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.2d","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.466601+0000","last_change":"2024-02-21T23:08:14.452885+0000","last_active":"2024-02-21T23:08:15.466601+0000","last_peered":"2024-02-21T23:08:15.466601+0000","last_clean":"2024-02-21T23:08:15.466601+0000","last_became_active":"2024-02-21T23:08:14.452551+0000","last_became_peered":"2024-02-21T23:08:14.452551+0000","last_unstale":"2024-02-21T23:08:15.466601+0000","last_undegraded":"2024-02-21T23:08:15.466601+0000","last_fullsized":"2024-02-21T23:08:15.466601+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:33:43.329787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.2c","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.466647+0000","last_change":"2024-02-21T23:08:14.459775+0000","last_active":"2024-02-21T23:08:15.466647+0000","last_peered":"2024-02-21T23:08:15.466647+0000","last_clean":"2024-02-21T23:08:15.466647+0000","last_became_active":"2024-02-21T23:08:14.459460+0000","last_became_peered":"2024-02-21T23:08:14.459460+0000","last_unstale":"2024-02-21T23:08:15.466647+0000","last_undegraded":"2024-02-21T23:08:15.466647+0000","last_fullsized":"2024-02-21T23:08:15.466647+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:46:54.184368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.2b","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.428226+0000","last_change":"2024-02-21T23:08:14.451956+0000","last_active":"2024-02-21T23:08:15.428226+0000","last_peered":"2024-02-21T23:08:15.428226+0000","last_clean":"2024-02-21T23:08:15.428226+0000","last_became_active":"2024-02-21T23:08:14.447631+0000","last_became_peered":"2024-02-21T23:08:14.447631+0000","last_unstale":"2024-02-21T23:08:15.428226+0000","last_undegraded":"2024-02-21T23:08:15.428226+0000","last_fullsized":"2024-02-21T23:08:15.428226+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:01:34.090943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2a","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.467775+0000","last_change":"2024-02-21T23:08:14.443331+0000","last_active":"2024-02-21T23:08:15.467775+0000","last_peered":"2024-02-21T23:08:15.467775+0000","last_clean":"2024-02-21T23:08:15.467775+0000","last_became_active":"2024-02-21T23:08:14.442067+0000","last_became_peered":"2024-02-21T23:08:14.442067+0000","last_unstale":"2024-02-21T23:08:15.467775+0000","last_undegraded":"2024-02-21T23:08:15.467775+0000","last_fullsized":"2024-02-21T23:08:15.467775+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:33:28.645253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.29","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.428471+0000","last_change":"2024-02-21T23:08:14.465223+0000","last_active":"2024-02-21T23:08:15.428471+0000","last_peered":"2024-02-21T23:08:15.428471+0000","last_clean":"2024-02-21T23:08:15.428471+0000","last_became_active":"2024-02-21T23:08:14.464452+0000","last_became_peered":"2024-02-21T23:08:14.464452+0000","last_unstale":"2024-02-21T23:08:15.428471+0000","last_undegraded":"2024-02-21T23:08:15.428471+0000","last_fullsized":"2024-02-21T23:08:15.428471+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:41:39.305102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.28","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.424604+0000","last_change":"2024-02-21T23:08:14.445139+0000","last_active":"2024-02-21T23:08:15.424604+0000","last_peered":"2024-02-21T23:08:15.424604+0000","last_clean":"2024-02-21T23:08:15.424604+0000","last_became_active":"2024-02-21T23:08:14.444872+0000","last_became_peered":"2024-02-21T23:08:14.444872+0000","last_unstale":"2024-02-21T23:08:15.424604+0000","last_undegraded":"2024-02-21T23:08:15.424604+0000","last_fullsized":"2024-02-21T23:08:15.424604+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:02:05.973126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.27","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.467155+0000","last_change":"2024-02-21T23:08:14.444418+0000","last_active":"2024-02-21T23:08:15.467155+0000","last_peered":"2024-02-21T23:08:15.467155+0000","last_clean":"2024-02-21T23:08:15.467155+0000","last_became_active":"2024-02-21T23:08:14.444123+0000","last_became_peered":"2024-02-21T23:08:14.444123+0000","last_unstale":"2024-02-21T23:08:15.467155+0000","last_undegraded":"2024-02-21T23:08:15.467155+0000","last_fullsized":"2024-02-21T23:08:15.467155+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:48:35.406735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.26","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.467559+0000","last_change":"2024-02-21T23:08:14.462662+0000","last_active":"2024-02-21T23:08:15.467559+0000","last_peered":"2024-02-21T23:08:15.467559+0000","last_clean":"2024-02-21T23:08:15.467559+0000","last_became_active":"2024-02-21T23:08:14.462381+0000","last_became_peered":"2024-02-21T23:08:14.462381+0000","last_unstale":"2024-02-21T23:08:15.467559+0000","last_undegraded":"2024-02-21T23:08:15.467559+0000","last_fullsized":"2024-02-21T23:08:15.467559+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:02:05.973126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.25","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.466316+0000","last_change":"2024-02-21T23:08:14.464323+0000","last_active":"2024-02-21T23:08:15.466316+0000","last_peered":"2024-02-21T23:08:15.466316+0000","last_clean":"2024-02-21T23:08:15.466316+0000","last_became_active":"2024-02-21T23:08:14.461888+0000","last_became_peered":"2024-02-21T23:08:14.461888+0000","last_unstale":"2024-02-21T23:08:15.466316+0000","last_undegraded":"2024-02-21T23:08:15.466316+0000","last_fullsized":"2024-02-21T23:08:15.466316+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:35:46.921047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.24","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.469091+0000","last_change":"2024-02-21T23:08:14.439704+0000","last_active":"2024-02-21T23:08:15.469091+0000","last_peered":"2024-02-21T23:08:15.469091+0000","last_clean":"2024-02-21T23:08:15.469091+0000","last_became_active":"2024-02-21T23:08:14.439397+0000","last_became_peered":"2024-02-21T23:08:14.439397+0000","last_unstale":"2024-02-21T23:08:15.469091+0000","last_undegraded":"2024-02-21T23:08:15.469091+0000","last_fullsized":"2024-02-21T23:08:15.469091+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:11:34.837356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.23","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.467717+0000","last_change":"2024-02-21T23:08:14.446999+0000","last_active":"2024-02-21T23:08:15.467717+0000","last_peered":"2024-02-21T23:08:15.467717+0000","last_clean":"2024-02-21T23:08:15.467717+0000","last_became_active":"2024-02-21T23:08:14.446705+0000","last_became_peered":"2024-02-21T23:08:14.446705+0000","last_unstale":"2024-02-21T23:08:15.467717+0000","last_undegraded":"2024-02-21T23:08:15.467717+0000","last_fullsized":"2024-02-21T23:08:15.467717+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:01:34.090943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.22","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.467207+0000","last_change":"2024-02-21T23:08:14.448941+0000","last_active":"2024-02-21T23:08:15.467207+0000","last_peered":"2024-02-21T23:08:15.467207+0000","last_clean":"2024-02-21T23:08:15.467207+0000","last_became_active":"2024-02-21T23:08:14.444323+0000","last_became_peered":"2024-02-21T23:08:14.444323+0000","last_unstale":"2024-02-21T23:08:15.467207+0000","last_undegraded":"2024-02-21T23:08:15.467207+0000","last_fullsized":"2024-02-21T23:08:15.467207+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:47:09.481691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.21","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.427094+0000","last_change":"2024-02-21T23:08:14.453634+0000","last_active":"2024-02-21T23:08:15.427094+0000","last_peered":"2024-02-21T23:08:15.427094+0000","last_clean":"2024-02-21T23:08:15.427094+0000","last_became_active":"2024-02-21T23:08:14.453352+0000","last_became_peered":"2024-02-21T23:08:14.453352+0000","last_unstale":"2024-02-21T23:08:15.427094+0000","last_undegraded":"2024-02-21T23:08:15.427094+0000","last_fullsized":"2024-02-21T23:08:15.427094+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:08:39.925583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.20","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.468840+0000","last_change":"2024-02-21T23:08:14.457744+0000","last_active":"2024-02-21T23:08:15.468840+0000","last_peered":"2024-02-21T23:08:15.468840+0000","last_clean":"2024-02-21T23:08:15.468840+0000","last_became_active":"2024-02-21T23:08:14.456937+0000","last_became_peered":"2024-02-21T23:08:14.456937+0000","last_unstale":"2024-02-21T23:08:15.468840+0000","last_undegraded":"2024-02-21T23:08:15.468840+0000","last_fullsized":"2024-02-21T23:08:15.468840+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:11:03.093612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.426713+0000","last_change":"2024-02-21T23:08:14.455515+0000","last_active":"2024-02-21T23:08:15.426713+0000","last_peered":"2024-02-21T23:08:15.426713+0000","last_clean":"2024-02-21T23:08:15.426713+0000","last_became_active":"2024-02-21T23:08:14.455243+0000","last_became_peered":"2024-02-21T23:08:14.455243+0000","last_unstale":"2024-02-21T23:08:15.426713+0000","last_undegraded":"2024-02-21T23:08:15.426713+0000","last_fullsized":"2024-02-21T23:08:15.426713+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:20:49.830901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1f","version":"121'4","reported_seq":18,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.467278+0000","last_change":"2024-02-21T23:08:13.411691+0000","last_active":"2024-02-21T23:08:15.467278+0000","last_peered":"2024-02-21T23:08:15.467278+0000","last_clean":"2024-02-21T23:08:15.467278+0000","last_became_active":"2024-02-21T23:08:13.411166+0000","last_became_peered":"2024-02-21T23:08:13.411166+0000","last_unstale":"2024-02-21T23:08:15.467278+0000","last_undegraded":"2024-02-21T23:08:15.467278+0000","last_fullsized":"2024-02-21T23:08:15.467278+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_clean_scrub_stamp":"2024-02-21T23:08:12.367049+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:17:52.454790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.467427+0000","last_change":"2024-02-21T23:08:14.442518+0000","last_active":"2024-02-21T23:08:15.467427+0000","last_peered":"2024-02-21T23:08:15.467427+0000","last_clean":"2024-02-21T23:08:15.467427+0000","last_became_active":"2024-02-21T23:08:14.441709+0000","last_became_peered":"2024-02-21T23:08:14.441709+0000","last_unstale":"2024-02-21T23:08:15.467427+0000","last_undegraded":"2024-02-21T23:08:15.467427+0000","last_fullsized":"2024-02-21T23:08:15.467427+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:48:49.955729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1e","version":"121'1","reported_seq":15,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.467047+0000","last_change":"2024-02-21T23:08:13.426961+0000","last_active":"2024-02-21T23:08:15.467047+0000","last_peered":"2024-02-21T23:08:15.467047+0000","last_clean":"2024-02-21T23:08:15.467047+0000","last_became_active":"2024-02-21T23:08:13.425923+0000","last_became_peered":"2024-02-21T23:08:13.425923+0000","last_unstale":"2024-02-21T23:08:15.467047+0000","last_undegraded":"2024-02-21T23:08:15.467047+0000","last_fullsized":"2024-02-21T23:08:15.467047+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_clean_scrub_stamp":"2024-02-21T23:08:12.367049+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:30:38.130535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1a","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.428188+0000","last_change":"2024-02-21T23:08:14.461921+0000","last_active":"2024-02-21T23:08:15.428188+0000","last_peered":"2024-02-21T23:08:15.428188+0000","last_clean":"2024-02-21T23:08:15.428188+0000","last_became_active":"2024-02-21T23:08:14.461643+0000","last_became_peered":"2024-02-21T23:08:14.461643+0000","last_unstale":"2024-02-21T23:08:15.428188+0000","last_undegraded":"2024-02-21T23:08:15.428188+0000","last_fullsized":"2024-02-21T23:08:15.428188+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:02:05.973126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1d","version":"121'2","reported_seq":16,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.466888+0000","last_change":"2024-02-21T23:08:13.430861+0000","last_active":"2024-02-21T23:08:15.466888+0000","last_peered":"2024-02-21T23:08:15.466888+0000","last_clean":"2024-02-21T23:08:15.466888+0000","last_became_active":"2024-02-21T23:08:13.429039+0000","last_became_peered":"2024-02-21T23:08:13.429039+0000","last_unstale":"2024-02-21T23:08:15.466888+0000","last_undegraded":"2024-02-21T23:08:15.466888+0000","last_fullsized":"2024-02-21T23:08:15.466888+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_clean_scrub_stamp":"2024-02-21T23:08:12.367049+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:47:08.465310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1b","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.467586+0000","last_change":"2024-02-21T23:08:14.455808+0000","last_active":"2024-02-21T23:08:15.467586+0000","last_peered":"2024-02-21T23:08:15.467586+0000","last_clean":"2024-02-21T23:08:15.467586+0000","last_became_active":"2024-02-21T23:08:14.455515+0000","last_became_peered":"2024-02-21T23:08:14.455515+0000","last_unstale":"2024-02-21T23:08:15.467586+0000","last_undegraded":"2024-02-21T23:08:15.467586+0000","last_fullsized":"2024-02-21T23:08:15.467586+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:09:02.474366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1c","version":"121'2","reported_seq":16,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.465697+0000","last_change":"2024-02-21T23:08:13.426772+0000","last_active":"2024-02-21T23:08:15.465697+0000","last_peered":"2024-02-21T23:08:15.465697+0000","last_clean":"2024-02-21T23:08:15.465697+0000","last_became_active":"2024-02-21T23:08:13.426462+0000","last_became_peered":"2024-02-21T23:08:13.426462+0000","last_unstale":"2024-02-21T23:08:15.465697+0000","last_undegraded":"2024-02-21T23:08:15.465697+0000","last_fullsized":"2024-02-21T23:08:15.465697+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_clean_scrub_stamp":"2024-02-21T23:08:12.367049+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:48:34.390354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.427692+0000","last_change":"2024-02-21T23:08:14.446601+0000","last_active":"2024-02-21T23:08:15.427692+0000","last_peered":"2024-02-21T23:08:15.427692+0000","last_clean":"2024-02-21T23:08:15.427692+0000","last_became_active":"2024-02-21T23:08:14.446282+0000","last_became_peered":"2024-02-21T23:08:14.446282+0000","last_unstale":"2024-02-21T23:08:15.427692+0000","last_undegraded":"2024-02-21T23:08:15.427692+0000","last_fullsized":"2024-02-21T23:08:15.427692+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:48:35.406735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1b","version":"121'1","reported_seq":15,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.427072+0000","last_change":"2024-02-21T23:08:13.423770+0000","last_active":"2024-02-21T23:08:15.427072+0000","last_peered":"2024-02-21T23:08:15.427072+0000","last_clean":"2024-02-21T23:08:15.427072+0000","last_became_active":"2024-02-21T23:08:13.423433+0000","last_became_peered":"2024-02-21T23:08:13.423433+0000","last_unstale":"2024-02-21T23:08:15.427072+0000","last_undegraded":"2024-02-21T23:08:15.427072+0000","last_fullsized":"2024-02-21T23:08:15.427072+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_clean_scrub_stamp":"2024-02-21T23:08:12.367049+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:27:41.020802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1d","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.428710+0000","last_change":"2024-02-21T23:08:14.441526+0000","last_active":"2024-02-21T23:08:15.428710+0000","last_peered":"2024-02-21T23:08:15.428710+0000","last_clean":"2024-02-21T23:08:15.428710+0000","last_became_active":"2024-02-21T23:08:14.441251+0000","last_became_peered":"2024-02-21T23:08:14.441251+0000","last_unstale":"2024-02-21T23:08:15.428710+0000","last_undegraded":"2024-02-21T23:08:15.428710+0000","last_fullsized":"2024-02-21T23:08:15.428710+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:47:09.481691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1a","version":"121'1","reported_seq":15,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.466102+0000","last_change":"2024-02-21T23:08:13.423864+0000","last_active":"2024-02-21T23:08:15.466102+0000","last_peered":"2024-02-21T23:08:15.466102+0000","last_clean":"2024-02-21T23:08:15.466102+0000","last_became_active":"2024-02-21T23:08:13.423576+0000","last_became_peered":"2024-02-21T23:08:13.423576+0000","last_unstale":"2024-02-21T23:08:15.466102+0000","last_undegraded":"2024-02-21T23:08:15.466102+0000","last_fullsized":"2024-02-21T23:08:15.466102+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_clean_scrub_stamp":"2024-02-21T23:08:12.367049+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:01:17.687431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.466841+0000","last_change":"2024-02-21T23:08:14.463245+0000","last_active":"2024-02-21T23:08:15.466841+0000","last_peered":"2024-02-21T23:08:15.466841+0000","last_clean":"2024-02-21T23:08:15.466841+0000","last_became_active":"2024-02-21T23:08:14.461701+0000","last_became_peered":"2024-02-21T23:08:14.461701+0000","last_unstale":"2024-02-21T23:08:15.466841+0000","last_undegraded":"2024-02-21T23:08:15.466841+0000","last_fullsized":"2024-02-21T23:08:15.466841+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:20:41.288649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":14,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.424867+0000","last_change":"2024-02-21T23:08:13.427214+0000","last_active":"2024-02-21T23:08:15.424867+0000","last_peered":"2024-02-21T23:08:15.424867+0000","last_clean":"2024-02-21T23:08:15.424867+0000","last_became_active":"2024-02-21T23:08:13.426923+0000","last_became_peered":"2024-02-21T23:08:13.426923+0000","last_unstale":"2024-02-21T23:08:15.424867+0000","last_undegraded":"2024-02-21T23:08:15.424867+0000","last_fullsized":"2024-02-21T23:08:15.424867+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_clean_scrub_stamp":"2024-02-21T23:08:12.367049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:57:40.070790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1f","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.467621+0000","last_change":"2024-02-21T23:08:14.459234+0000","last_active":"2024-02-21T23:08:15.467621+0000","last_peered":"2024-02-21T23:08:15.467621+0000","last_clean":"2024-02-21T23:08:15.467621+0000","last_became_active":"2024-02-21T23:08:14.458970+0000","last_became_peered":"2024-02-21T23:08:14.458970+0000","last_unstale":"2024-02-21T23:08:15.467621+0000","last_undegraded":"2024-02-21T23:08:15.467621+0000","last_fullsized":"2024-02-21T23:08:15.467621+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:03:18.697668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":14,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.466324+0000","last_change":"2024-02-21T23:08:13.417711+0000","last_active":"2024-02-21T23:08:15.466324+0000","last_peered":"2024-02-21T23:08:15.466324+0000","last_clean":"2024-02-21T23:08:15.466324+0000","last_became_active":"2024-02-21T23:08:13.412707+0000","last_became_peered":"2024-02-21T23:08:13.412707+0000","last_unstale":"2024-02-21T23:08:15.466324+0000","last_undegraded":"2024-02-21T23:08:15.466324+0000","last_fullsized":"2024-02-21T23:08:15.466324+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_clean_scrub_stamp":"2024-02-21T23:08:12.367049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:36:37.776608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.466786+0000","last_change":"2024-02-21T23:08:14.445447+0000","last_active":"2024-02-21T23:08:15.466786+0000","last_peered":"2024-02-21T23:08:15.466786+0000","last_clean":"2024-02-21T23:08:15.466786+0000","last_became_active":"2024-02-21T23:08:14.444905+0000","last_became_peered":"2024-02-21T23:08:14.444905+0000","last_unstale":"2024-02-21T23:08:15.466786+0000","last_undegraded":"2024-02-21T23:08:15.466786+0000","last_fullsized":"2024-02-21T23:08:15.466786+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:01:15.415412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":14,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.429439+0000","last_change":"2024-02-21T23:08:13.428573+0000","last_active":"2024-02-21T23:08:15.429439+0000","last_peered":"2024-02-21T23:08:15.429439+0000","last_clean":"2024-02-21T23:08:15.429439+0000","last_became_active":"2024-02-21T23:08:13.425871+0000","last_became_peered":"2024-02-21T23:08:13.425871+0000","last_unstale":"2024-02-21T23:08:15.429439+0000","last_undegraded":"2024-02-21T23:08:15.429439+0000","last_fullsized":"2024-02-21T23:08:15.429439+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_clean_scrub_stamp":"2024-02-21T23:08:12.367049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:08:17.593082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.425070+0000","last_change":"2024-02-21T23:08:14.457930+0000","last_active":"2024-02-21T23:08:15.425070+0000","last_peered":"2024-02-21T23:08:15.425070+0000","last_clean":"2024-02-21T23:08:15.425070+0000","last_became_active":"2024-02-21T23:08:14.457084+0000","last_became_peered":"2024-02-21T23:08:14.457084+0000","last_unstale":"2024-02-21T23:08:15.425070+0000","last_undegraded":"2024-02-21T23:08:15.425070+0000","last_fullsized":"2024-02-21T23:08:15.425070+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:41:39.305102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":14,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.468509+0000","last_change":"2024-02-21T23:08:13.409691+0000","last_active":"2024-02-21T23:08:15.468509+0000","last_peered":"2024-02-21T23:08:15.468509+0000","last_clean":"2024-02-21T23:08:15.468509+0000","last_became_active":"2024-02-21T23:08:13.409289+0000","last_became_peered":"2024-02-21T23:08:13.409289+0000","last_unstale":"2024-02-21T23:08:15.468509+0000","last_undegraded":"2024-02-21T23:08:15.468509+0000","last_fullsized":"2024-02-21T23:08:15.468509+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_clean_scrub_stamp":"2024-02-21T23:08:12.367049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:55:55.845158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.467327+0000","last_change":"2024-02-21T23:08:14.439759+0000","last_active":"2024-02-21T23:08:15.467327+0000","last_peered":"2024-02-21T23:08:15.467327+0000","last_clean":"2024-02-21T23:08:15.467327+0000","last_became_active":"2024-02-21T23:08:14.439471+0000","last_became_peered":"2024-02-21T23:08:14.439471+0000","last_unstale":"2024-02-21T23:08:15.467327+0000","last_undegraded":"2024-02-21T23:08:15.467327+0000","last_fullsized":"2024-02-21T23:08:15.467327+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:16:07.331005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.15","version":"121'3","reported_seq":17,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.467165+0000","last_change":"2024-02-21T23:08:13.431736+0000","last_active":"2024-02-21T23:08:15.467165+0000","last_peered":"2024-02-21T23:08:15.467165+0000","last_clean":"2024-02-21T23:08:15.467165+0000","last_became_active":"2024-02-21T23:08:13.431271+0000","last_became_peered":"2024-02-21T23:08:13.431271+0000","last_unstale":"2024-02-21T23:08:15.467165+0000","last_undegraded":"2024-02-21T23:08:15.467165+0000","last_fullsized":"2024-02-21T23:08:15.467165+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_clean_scrub_stamp":"2024-02-21T23:08:12.367049+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:16:06.314624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.13","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.467921+0000","last_change":"2024-02-21T23:08:14.450201+0000","last_active":"2024-02-21T23:08:15.467921+0000","last_peered":"2024-02-21T23:08:15.467921+0000","last_clean":"2024-02-21T23:08:15.467921+0000","last_became_active":"2024-02-21T23:08:14.449100+0000","last_became_peered":"2024-02-21T23:08:14.449100+0000","last_unstale":"2024-02-21T23:08:15.467921+0000","last_undegraded":"2024-02-21T23:08:15.467921+0000","last_fullsized":"2024-02-21T23:08:15.467921+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:41:39.305102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.14","version":"121'6","reported_seq":20,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.465808+0000","last_change":"2024-02-21T23:08:13.430036+0000","last_active":"2024-02-21T23:08:15.465808+0000","last_peered":"2024-02-21T23:08:15.465808+0000","last_clean":"2024-02-21T23:08:15.465808+0000","last_became_active":"2024-02-21T23:08:13.429720+0000","last_became_peered":"2024-02-21T23:08:13.429720+0000","last_unstale":"2024-02-21T23:08:15.465808+0000","last_undegraded":"2024-02-21T23:08:15.465808+0000","last_fullsized":"2024-02-21T23:08:15.465808+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_clean_scrub_stamp":"2024-02-21T23:08:12.367049+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:02:04.956745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.14","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.469387+0000","last_change":"2024-02-21T23:08:14.444654+0000","last_active":"2024-02-21T23:08:15.469387+0000","last_peered":"2024-02-21T23:08:15.469387+0000","last_clean":"2024-02-21T23:08:15.469387+0000","last_became_active":"2024-02-21T23:08:14.442399+0000","last_became_peered":"2024-02-21T23:08:14.442399+0000","last_unstale":"2024-02-21T23:08:15.469387+0000","last_undegraded":"2024-02-21T23:08:15.469387+0000","last_fullsized":"2024-02-21T23:08:15.469387+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:04:54.173602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.13","version":"121'2","reported_seq":16,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.467836+0000","last_change":"2024-02-21T23:08:13.408848+0000","last_active":"2024-02-21T23:08:15.467836+0000","last_peered":"2024-02-21T23:08:15.467836+0000","last_clean":"2024-02-21T23:08:15.467836+0000","last_became_active":"2024-02-21T23:08:13.408427+0000","last_became_peered":"2024-02-21T23:08:13.408427+0000","last_unstale":"2024-02-21T23:08:15.467836+0000","last_undegraded":"2024-02-21T23:08:15.467836+0000","last_fullsized":"2024-02-21T23:08:15.467836+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_clean_scrub_stamp":"2024-02-21T23:08:12.367049+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:30:41.304449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.467624+0000","last_change":"2024-02-21T23:08:14.457492+0000","last_active":"2024-02-21T23:08:15.467624+0000","last_peered":"2024-02-21T23:08:15.467624+0000","last_clean":"2024-02-21T23:08:15.467624+0000","last_became_active":"2024-02-21T23:08:14.457071+0000","last_became_peered":"2024-02-21T23:08:14.457071+0000","last_unstale":"2024-02-21T23:08:15.467624+0000","last_undegraded":"2024-02-21T23:08:15.467624+0000","last_fullsized":"2024-02-21T23:08:15.467624+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:55:15.574780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":14,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.466966+0000","last_change":"2024-02-21T23:08:13.425784+0000","last_active":"2024-02-21T23:08:15.466966+0000","last_peered":"2024-02-21T23:08:15.466966+0000","last_clean":"2024-02-21T23:08:15.466966+0000","last_became_active":"2024-02-21T23:08:13.425472+0000","last_became_peered":"2024-02-21T23:08:13.425472+0000","last_unstale":"2024-02-21T23:08:15.466966+0000","last_undegraded":"2024-02-21T23:08:15.466966+0000","last_fullsized":"2024-02-21T23:08:15.466966+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_clean_scrub_stamp":"2024-02-21T23:08:12.367049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:59:09.803895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.467544+0000","last_change":"2024-02-21T23:08:14.447350+0000","last_active":"2024-02-21T23:08:15.467544+0000","last_peered":"2024-02-21T23:08:15.467544+0000","last_clean":"2024-02-21T23:08:15.467544+0000","last_became_active":"2024-02-21T23:08:14.446246+0000","last_became_peered":"2024-02-21T23:08:14.446246+0000","last_unstale":"2024-02-21T23:08:15.467544+0000","last_undegraded":"2024-02-21T23:08:15.467544+0000","last_fullsized":"2024-02-21T23:08:15.467544+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:59:30.631819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.10","version":"121'8","reported_seq":22,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.424695+0000","last_change":"2024-02-21T23:08:13.429933+0000","last_active":"2024-02-21T23:08:15.424695+0000","last_peered":"2024-02-21T23:08:15.424695+0000","last_clean":"2024-02-21T23:08:15.424695+0000","last_became_active":"2024-02-21T23:08:13.428835+0000","last_became_peered":"2024-02-21T23:08:13.428835+0000","last_unstale":"2024-02-21T23:08:15.424695+0000","last_undegraded":"2024-02-21T23:08:15.424695+0000","last_fullsized":"2024-02-21T23:08:15.424695+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_clean_scrub_stamp":"2024-02-21T23:08:12.367049+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:08:17.593082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.466379+0000","last_change":"2024-02-21T23:08:14.447205+0000","last_active":"2024-02-21T23:08:15.466379+0000","last_peered":"2024-02-21T23:08:15.466379+0000","last_clean":"2024-02-21T23:08:15.466379+0000","last_became_active":"2024-02-21T23:08:14.446907+0000","last_became_peered":"2024-02-21T23:08:14.446907+0000","last_unstale":"2024-02-21T23:08:15.466379+0000","last_undegraded":"2024-02-21T23:08:15.466379+0000","last_fullsized":"2024-02-21T23:08:15.466379+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:33:33.871286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.f","version":"121'1","reported_seq":15,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.429511+0000","last_change":"2024-02-21T23:08:13.427895+0000","last_active":"2024-02-21T23:08:15.429511+0000","last_peered":"2024-02-21T23:08:15.429511+0000","last_clean":"2024-02-21T23:08:15.429511+0000","last_became_active":"2024-02-21T23:08:13.427555+0000","last_became_peered":"2024-02-21T23:08:13.427555+0000","last_unstale":"2024-02-21T23:08:15.429511+0000","last_undegraded":"2024-02-21T23:08:15.429511+0000","last_fullsized":"2024-02-21T23:08:15.429511+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_clean_scrub_stamp":"2024-02-21T23:08:12.367049+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:51:31.193688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.466134+0000","last_change":"2024-02-21T23:08:14.460421+0000","last_active":"2024-02-21T23:08:15.466134+0000","last_peered":"2024-02-21T23:08:15.466134+0000","last_clean":"2024-02-21T23:08:15.466134+0000","last_became_active":"2024-02-21T23:08:14.460109+0000","last_became_peered":"2024-02-21T23:08:14.460109+0000","last_unstale":"2024-02-21T23:08:15.466134+0000","last_undegraded":"2024-02-21T23:08:15.466134+0000","last_fullsized":"2024-02-21T23:08:15.466134+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:48:35.819688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.e","version":"121'2","reported_seq":16,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.467795+0000","last_change":"2024-02-21T23:08:13.408230+0000","last_active":"2024-02-21T23:08:15.467795+0000","last_peered":"2024-02-21T23:08:15.467795+0000","last_clean":"2024-02-21T23:08:15.467795+0000","last_became_active":"2024-02-21T23:08:13.407502+0000","last_became_peered":"2024-02-21T23:08:13.407502+0000","last_unstale":"2024-02-21T23:08:15.467795+0000","last_undegraded":"2024-02-21T23:08:15.467795+0000","last_fullsized":"2024-02-21T23:08:15.467795+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_clean_scrub_stamp":"2024-02-21T23:08:12.367049+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:41:29.201521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.465621+0000","last_change":"2024-02-21T23:08:14.464546+0000","last_active":"2024-02-21T23:08:15.465621+0000","last_peered":"2024-02-21T23:08:15.465621+0000","last_clean":"2024-02-21T23:08:15.465621+0000","last_became_active":"2024-02-21T23:08:14.463305+0000","last_became_peered":"2024-02-21T23:08:14.463305+0000","last_unstale":"2024-02-21T23:08:15.465621+0000","last_undegraded":"2024-02-21T23:08:15.465621+0000","last_fullsized":"2024-02-21T23:08:15.465621+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:23:21.375840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.d","version":"121'1","reported_seq":15,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.425747+0000","last_change":"2024-02-21T23:08:13.428145+0000","last_active":"2024-02-21T23:08:15.425747+0000","last_peered":"2024-02-21T23:08:15.425747+0000","last_clean":"2024-02-21T23:08:15.425747+0000","last_became_active":"2024-02-21T23:08:13.427845+0000","last_became_peered":"2024-02-21T23:08:13.427845+0000","last_unstale":"2024-02-21T23:08:15.425747+0000","last_undegraded":"2024-02-21T23:08:15.425747+0000","last_fullsized":"2024-02-21T23:08:15.425747+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_clean_scrub_stamp":"2024-02-21T23:08:12.367049+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:50:42.102752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.b","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.469824+0000","last_change":"2024-02-21T23:08:14.444063+0000","last_active":"2024-02-21T23:08:15.469824+0000","last_peered":"2024-02-21T23:08:15.469824+0000","last_clean":"2024-02-21T23:08:15.469824+0000","last_became_active":"2024-02-21T23:08:14.441607+0000","last_became_peered":"2024-02-21T23:08:14.441607+0000","last_unstale":"2024-02-21T23:08:15.469824+0000","last_undegraded":"2024-02-21T23:08:15.469824+0000","last_fullsized":"2024-02-21T23:08:15.469824+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:37:25.248467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":14,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.467667+0000","last_change":"2024-02-21T23:08:13.407579+0000","last_active":"2024-02-21T23:08:15.467667+0000","last_peered":"2024-02-21T23:08:15.467667+0000","last_clean":"2024-02-21T23:08:15.467667+0000","last_became_active":"2024-02-21T23:08:13.407110+0000","last_became_peered":"2024-02-21T23:08:13.407110+0000","last_unstale":"2024-02-21T23:08:15.467667+0000","last_undegraded":"2024-02-21T23:08:15.467667+0000","last_fullsized":"2024-02-21T23:08:15.467667+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_clean_scrub_stamp":"2024-02-21T23:08:12.367049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:34:34.521904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.468247+0000","last_change":"2024-02-21T23:08:14.440614+0000","last_active":"2024-02-21T23:08:15.468247+0000","last_peered":"2024-02-21T23:08:15.468247+0000","last_clean":"2024-02-21T23:08:15.468247+0000","last_became_active":"2024-02-21T23:08:14.440352+0000","last_became_peered":"2024-02-21T23:08:14.440352+0000","last_unstale":"2024-02-21T23:08:15.468247+0000","last_undegraded":"2024-02-21T23:08:15.468247+0000","last_fullsized":"2024-02-21T23:08:15.468247+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:21:07.830802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.b","version":"121'2","reported_seq":16,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.427344+0000","last_change":"2024-02-21T23:08:13.426149+0000","last_active":"2024-02-21T23:08:15.427344+0000","last_peered":"2024-02-21T23:08:15.427344+0000","last_clean":"2024-02-21T23:08:15.427344+0000","last_became_active":"2024-02-21T23:08:13.425844+0000","last_became_peered":"2024-02-21T23:08:13.425844+0000","last_unstale":"2024-02-21T23:08:15.427344+0000","last_undegraded":"2024-02-21T23:08:15.427344+0000","last_fullsized":"2024-02-21T23:08:15.427344+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_clean_scrub_stamp":"2024-02-21T23:08:12.367049+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:50:42.102752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.d","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.468516+0000","last_change":"2024-02-21T23:08:14.461236+0000","last_active":"2024-02-21T23:08:15.468516+0000","last_peered":"2024-02-21T23:08:15.468516+0000","last_clean":"2024-02-21T23:08:15.468516+0000","last_became_active":"2024-02-21T23:08:14.460948+0000","last_became_peered":"2024-02-21T23:08:14.460948+0000","last_unstale":"2024-02-21T23:08:15.468516+0000","last_undegraded":"2024-02-21T23:08:15.468516+0000","last_fullsized":"2024-02-21T23:08:15.468516+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:44:33.355216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":14,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.468083+0000","last_change":"2024-02-21T23:08:13.429851+0000","last_active":"2024-02-21T23:08:15.468083+0000","last_peered":"2024-02-21T23:08:15.468083+0000","last_clean":"2024-02-21T23:08:15.468083+0000","last_became_active":"2024-02-21T23:08:13.428515+0000","last_became_peered":"2024-02-21T23:08:13.428515+0000","last_unstale":"2024-02-21T23:08:15.468083+0000","last_undegraded":"2024-02-21T23:08:15.468083+0000","last_fullsized":"2024-02-21T23:08:15.468083+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_clean_scrub_stamp":"2024-02-21T23:08:12.367049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:19:12.785951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.467137+0000","last_change":"2024-02-21T23:08:14.447164+0000","last_active":"2024-02-21T23:08:15.467137+0000","last_peered":"2024-02-21T23:08:15.467137+0000","last_clean":"2024-02-21T23:08:15.467137+0000","last_became_active":"2024-02-21T23:08:14.446677+0000","last_became_peered":"2024-02-21T23:08:14.446677+0000","last_unstale":"2024-02-21T23:08:15.467137+0000","last_undegraded":"2024-02-21T23:08:15.467137+0000","last_fullsized":"2024-02-21T23:08:15.467137+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:04:25.009442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":14,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.466406+0000","last_change":"2024-02-21T23:08:13.433304+0000","last_active":"2024-02-21T23:08:15.466406+0000","last_peered":"2024-02-21T23:08:15.466406+0000","last_clean":"2024-02-21T23:08:15.466406+0000","last_became_active":"2024-02-21T23:08:13.432958+0000","last_became_peered":"2024-02-21T23:08:13.432958+0000","last_unstale":"2024-02-21T23:08:15.466406+0000","last_undegraded":"2024-02-21T23:08:15.466406+0000","last_fullsized":"2024-02-21T23:08:15.466406+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_clean_scrub_stamp":"2024-02-21T23:08:12.367049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:01:33.074562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.429342+0000","last_change":"2024-02-21T23:08:14.451366+0000","last_active":"2024-02-21T23:08:15.429342+0000","last_peered":"2024-02-21T23:08:15.429342+0000","last_clean":"2024-02-21T23:08:15.429342+0000","last_became_active":"2024-02-21T23:08:14.451043+0000","last_became_peered":"2024-02-21T23:08:14.451043+0000","last_unstale":"2024-02-21T23:08:15.429342+0000","last_undegraded":"2024-02-21T23:08:15.429342+0000","last_fullsized":"2024-02-21T23:08:15.429342+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:15:18.656888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":14,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.424745+0000","last_change":"2024-02-21T23:08:13.427040+0000","last_active":"2024-02-21T23:08:15.424745+0000","last_peered":"2024-02-21T23:08:15.424745+0000","last_clean":"2024-02-21T23:08:15.424745+0000","last_became_active":"2024-02-21T23:08:13.426611+0000","last_became_peered":"2024-02-21T23:08:13.426611+0000","last_unstale":"2024-02-21T23:08:15.424745+0000","last_undegraded":"2024-02-21T23:08:15.424745+0000","last_fullsized":"2024-02-21T23:08:15.424745+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_clean_scrub_stamp":"2024-02-21T23:08:12.367049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:15:31.378104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.466307+0000","last_change":"2024-02-21T23:08:14.461202+0000","last_active":"2024-02-21T23:08:15.466307+0000","last_peered":"2024-02-21T23:08:15.466307+0000","last_clean":"2024-02-21T23:08:15.466307+0000","last_became_active":"2024-02-21T23:08:14.458784+0000","last_became_peered":"2024-02-21T23:08:14.458784+0000","last_unstale":"2024-02-21T23:08:15.466307+0000","last_undegraded":"2024-02-21T23:08:15.466307+0000","last_fullsized":"2024-02-21T23:08:15.466307+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:55:20.800793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.7","version":"121'2","reported_seq":16,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.468244+0000","last_change":"2024-02-21T23:08:13.418201+0000","last_active":"2024-02-21T23:08:15.468244+0000","last_peered":"2024-02-21T23:08:15.468244+0000","last_clean":"2024-02-21T23:08:15.468244+0000","last_became_active":"2024-02-21T23:08:13.417467+0000","last_became_peered":"2024-02-21T23:08:13.417467+0000","last_unstale":"2024-02-21T23:08:15.468244+0000","last_undegraded":"2024-02-21T23:08:15.468244+0000","last_fullsized":"2024-02-21T23:08:15.468244+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_clean_scrub_stamp":"2024-02-21T23:08:12.367049+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:22:17.723754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.466674+0000","last_change":"2024-02-21T23:08:14.449840+0000","last_active":"2024-02-21T23:08:15.466674+0000","last_peered":"2024-02-21T23:08:15.466674+0000","last_clean":"2024-02-21T23:08:15.466674+0000","last_became_active":"2024-02-21T23:08:14.448908+0000","last_became_peered":"2024-02-21T23:08:14.448908+0000","last_unstale":"2024-02-21T23:08:15.466674+0000","last_undegraded":"2024-02-21T23:08:15.466674+0000","last_fullsized":"2024-02-21T23:08:15.466674+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:13:06.500761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":14,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.428812+0000","last_change":"2024-02-21T23:08:13.426738+0000","last_active":"2024-02-21T23:08:15.428812+0000","last_peered":"2024-02-21T23:08:15.428812+0000","last_clean":"2024-02-21T23:08:15.428812+0000","last_became_active":"2024-02-21T23:08:13.426454+0000","last_became_peered":"2024-02-21T23:08:13.426454+0000","last_unstale":"2024-02-21T23:08:15.428812+0000","last_undegraded":"2024-02-21T23:08:15.428812+0000","last_fullsized":"2024-02-21T23:08:15.428812+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_clean_scrub_stamp":"2024-02-21T23:08:12.367049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:22:55.245029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.465644+0000","last_change":"2024-02-21T23:08:14.450828+0000","last_active":"2024-02-21T23:08:15.465644+0000","last_peered":"2024-02-21T23:08:15.465644+0000","last_clean":"2024-02-21T23:08:15.465644+0000","last_became_active":"2024-02-21T23:08:14.450513+0000","last_became_peered":"2024-02-21T23:08:14.450513+0000","last_unstale":"2024-02-21T23:08:15.465644+0000","last_undegraded":"2024-02-21T23:08:15.465644+0000","last_fullsized":"2024-02-21T23:08:15.465644+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:59:04.556496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":14,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.427905+0000","last_change":"2024-02-21T23:08:13.429362+0000","last_active":"2024-02-21T23:08:15.427905+0000","last_peered":"2024-02-21T23:08:15.427905+0000","last_clean":"2024-02-21T23:08:15.427905+0000","last_became_active":"2024-02-21T23:08:13.429088+0000","last_became_peered":"2024-02-21T23:08:13.429088+0000","last_unstale":"2024-02-21T23:08:15.427905+0000","last_undegraded":"2024-02-21T23:08:15.427905+0000","last_fullsized":"2024-02-21T23:08:15.427905+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_clean_scrub_stamp":"2024-02-21T23:08:12.367049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:57:40.070790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.470096+0000","last_change":"2024-02-21T23:08:14.434316+0000","last_active":"2024-02-21T23:08:15.470096+0000","last_peered":"2024-02-21T23:08:15.470096+0000","last_clean":"2024-02-21T23:08:15.470096+0000","last_became_active":"2024-02-21T23:08:14.433537+0000","last_became_peered":"2024-02-21T23:08:14.433537+0000","last_unstale":"2024-02-21T23:08:15.470096+0000","last_undegraded":"2024-02-21T23:08:15.470096+0000","last_fullsized":"2024-02-21T23:08:15.470096+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:02:26.010423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":14,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.425217+0000","last_change":"2024-02-21T23:08:13.431471+0000","last_active":"2024-02-21T23:08:15.425217+0000","last_peered":"2024-02-21T23:08:15.425217+0000","last_clean":"2024-02-21T23:08:15.425217+0000","last_became_active":"2024-02-21T23:08:13.431206+0000","last_became_peered":"2024-02-21T23:08:13.431206+0000","last_unstale":"2024-02-21T23:08:15.425217+0000","last_undegraded":"2024-02-21T23:08:15.425217+0000","last_fullsized":"2024-02-21T23:08:15.425217+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_clean_scrub_stamp":"2024-02-21T23:08:12.367049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:27:41.020802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.425359+0000","last_change":"2024-02-21T23:08:14.455407+0000","last_active":"2024-02-21T23:08:15.425359+0000","last_peered":"2024-02-21T23:08:15.425359+0000","last_clean":"2024-02-21T23:08:15.425359+0000","last_became_active":"2024-02-21T23:08:14.455113+0000","last_became_peered":"2024-02-21T23:08:14.455113+0000","last_unstale":"2024-02-21T23:08:15.425359+0000","last_undegraded":"2024-02-21T23:08:15.425359+0000","last_fullsized":"2024-02-21T23:08:15.425359+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:54:54.757289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":14,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.428459+0000","last_change":"2024-02-21T23:08:13.423445+0000","last_active":"2024-02-21T23:08:15.428459+0000","last_peered":"2024-02-21T23:08:15.428459+0000","last_clean":"2024-02-21T23:08:15.428459+0000","last_became_active":"2024-02-21T23:08:13.423171+0000","last_became_peered":"2024-02-21T23:08:13.423171+0000","last_unstale":"2024-02-21T23:08:15.428459+0000","last_undegraded":"2024-02-21T23:08:15.428459+0000","last_fullsized":"2024-02-21T23:08:15.428459+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_clean_scrub_stamp":"2024-02-21T23:08:12.367049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:15:31.378104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.425449+0000","last_change":"2024-02-21T23:08:14.460611+0000","last_active":"2024-02-21T23:08:15.425449+0000","last_peered":"2024-02-21T23:08:15.425449+0000","last_clean":"2024-02-21T23:08:15.425449+0000","last_became_active":"2024-02-21T23:08:14.460196+0000","last_became_peered":"2024-02-21T23:08:14.460196+0000","last_unstale":"2024-02-21T23:08:15.425449+0000","last_undegraded":"2024-02-21T23:08:15.425449+0000","last_fullsized":"2024-02-21T23:08:15.425449+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:15:18.656888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":14,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.465758+0000","last_change":"2024-02-21T23:08:13.426964+0000","last_active":"2024-02-21T23:08:15.465758+0000","last_peered":"2024-02-21T23:08:15.465758+0000","last_clean":"2024-02-21T23:08:15.465758+0000","last_became_active":"2024-02-21T23:08:13.426252+0000","last_became_peered":"2024-02-21T23:08:13.426252+0000","last_unstale":"2024-02-21T23:08:15.465758+0000","last_undegraded":"2024-02-21T23:08:15.465758+0000","last_fullsized":"2024-02-21T23:08:15.465758+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_clean_scrub_stamp":"2024-02-21T23:08:12.367049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:08:38.909202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.425329+0000","last_change":"2024-02-21T23:08:14.460277+0000","last_active":"2024-02-21T23:08:15.425329+0000","last_peered":"2024-02-21T23:08:15.425329+0000","last_clean":"2024-02-21T23:08:15.425329+0000","last_became_active":"2024-02-21T23:08:14.459966+0000","last_became_peered":"2024-02-21T23:08:14.459966+0000","last_unstale":"2024-02-21T23:08:15.425329+0000","last_undegraded":"2024-02-21T23:08:15.425329+0000","last_fullsized":"2024-02-21T23:08:15.425329+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:54:23.719045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":14,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.425193+0000","last_change":"2024-02-21T23:08:13.431443+0000","last_active":"2024-02-21T23:08:15.425193+0000","last_peered":"2024-02-21T23:08:15.425193+0000","last_clean":"2024-02-21T23:08:15.425193+0000","last_became_active":"2024-02-21T23:08:13.431140+0000","last_became_peered":"2024-02-21T23:08:13.431140+0000","last_unstale":"2024-02-21T23:08:15.425193+0000","last_undegraded":"2024-02-21T23:08:15.425193+0000","last_fullsized":"2024-02-21T23:08:15.425193+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_clean_scrub_stamp":"2024-02-21T23:08:12.367049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:22:55.245029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.7","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.467896+0000","last_change":"2024-02-21T23:08:14.458665+0000","last_active":"2024-02-21T23:08:15.467896+0000","last_peered":"2024-02-21T23:08:15.467896+0000","last_clean":"2024-02-21T23:08:15.467896+0000","last_became_active":"2024-02-21T23:08:14.457434+0000","last_became_peered":"2024-02-21T23:08:14.457434+0000","last_unstale":"2024-02-21T23:08:15.467896+0000","last_undegraded":"2024-02-21T23:08:15.467896+0000","last_fullsized":"2024-02-21T23:08:15.467896+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:16:13.065511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":14,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.466480+0000","last_change":"2024-02-21T23:08:13.408639+0000","last_active":"2024-02-21T23:08:15.466480+0000","last_peered":"2024-02-21T23:08:15.466480+0000","last_clean":"2024-02-21T23:08:15.466480+0000","last_became_active":"2024-02-21T23:08:13.407970+0000","last_became_peered":"2024-02-21T23:08:13.407970+0000","last_unstale":"2024-02-21T23:08:15.466480+0000","last_undegraded":"2024-02-21T23:08:15.466480+0000","last_fullsized":"2024-02-21T23:08:15.466480+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_clean_scrub_stamp":"2024-02-21T23:08:12.367049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:32:22.922633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.15","version":"0'0","reported_seq":12,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.466897+0000","last_change":"2024-02-21T23:08:14.459300+0000","last_active":"2024-02-21T23:08:15.466897+0000","last_peered":"2024-02-21T23:08:15.466897+0000","last_clean":"2024-02-21T23:08:15.466897+0000","last_became_active":"2024-02-21T23:08:14.458773+0000","last_became_peered":"2024-02-21T23:08:14.458773+0000","last_unstale":"2024-02-21T23:08:15.466897+0000","last_undegraded":"2024-02-21T23:08:15.466897+0000","last_fullsized":"2024-02-21T23:08:15.466897+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":120,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:13.383430+0000","last_clean_scrub_stamp":"2024-02-21T23:08:13.383430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:08:39.925583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":14,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.465443+0000","last_change":"2024-02-21T23:08:13.431004+0000","last_active":"2024-02-21T23:08:15.465443+0000","last_peered":"2024-02-21T23:08:15.465443+0000","last_clean":"2024-02-21T23:08:15.465443+0000","last_became_active":"2024-02-21T23:08:13.430635+0000","last_became_peered":"2024-02-21T23:08:13.430635+0000","last_unstale":"2024-02-21T23:08:15.465443+0000","last_undegraded":"2024-02-21T23:08:15.465443+0000","last_fullsized":"2024-02-21T23:08:15.465443+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:08:12.367049+0000","last_clean_scrub_stamp":"2024-02-21T23:08:12.367049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:20:48.814520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":361,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-21T23:08:15.464970+0000","last_change":"2024-02-21T23:02:32.289335+0000","last_active":"2024-02-21T23:08:15.464970+0000","last_peered":"2024-02-21T23:08:15.464970+0000","last_clean":"2024-02-21T23:08:15.464970+0000","last_became_active":"2024-02-21T23:02:32.286913+0000","last_became_peered":"2024-02-21T23:02:32.286913+0000","last_unstale":"2024-02-21T23:08:15.464970+0000","last_undegraded":"2024-02-21T23:08:15.464970+0000","last_fullsized":"2024-02-21T23:08:15.464970+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_clean_scrub_stamp":"2024-02-21T23:02:31.172569+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:36:23.987374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":20,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":19,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574919,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32760,"kb_used_data":1572,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339080,"statfs":{"total":96636764160,"available":96603217920,"internally_reserved":0,"allocated":1609728,"data_stored":1335576,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574920,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32676,"kb_used_data":1488,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339164,"statfs":{"total":96636764160,"available":96603303936,"internally_reserved":0,"allocated":1523712,"data_stored":737486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574919,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32664,"kb_used_data":1484,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339176,"statfs":{"total":96636764160,"available":96603316224,"internally_reserved":0,"allocated":1519616,"data_stored":736836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574920,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32660,"kb_used_data":1480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339180,"statfs":{"total":96636764160,"available":96603320320,"internally_reserved":0,"allocated":1515520,"data_stored":736814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574920,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32676,"kb_used_data":1496,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339164,"statfs":{"total":96636764160,"available":96603303936,"internally_reserved":0,"allocated":1531904,"data_stored":738496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574921,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32668,"kb_used_data":1488,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339172,"statfs":{"total":96636764160,"available":96603312128,"internally_reserved":0,"allocated":1523712,"data_stored":737430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574919,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32680,"kb_used_data":1496,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339160,"statfs":{"total":96636764160,"available":96603299840,"internally_reserved":0,"allocated":1531904,"data_stored":743576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574919,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32736,"kb_used_data":1552,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339104,"statfs":{"total":96636764160,"available":96603242496,"internally_reserved":0,"allocated":1589248,"data_stored":1327238,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-21T23:08:19.441 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-02-21T23:08:19.442 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-02-21T23:08:19.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.731+0000 7f2bf5e00640 1 Processor -- start 2024-02-21T23:08:19.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.731+0000 7f2bf5e00640 1 -- start start 2024-02-21T23:08:19.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.733+0000 7f2bf5e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2bf00c35c0 0x7f2bf0156bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:19.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.733+0000 7f2bf5e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bf006f1b0 con 0x7f2bf00a96c0 2024-02-21T23:08:19.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.733+0000 7f2bf5e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bf00c2ae0 con 0x7f2bf0150640 2024-02-21T23:08:19.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.733+0000 7f2bf5e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf006ea40 con 0x7f2bf00c35c0 2024-02-21T23:08:19.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.733+0000 7f2befe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f2bf00a96c0 0x7f2bf014ff30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:57430/0 (socket says 172.21.15.28:57430) 2024-02-21T23:08:19.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.733+0000 7f2befe00640 1 -- 172.21.15.28:0/2623819809 learned_addr learned my addr 172.21.15.28:0/2623819809 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:08:19.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.733+0000 7f2bf4a00640 1 --2- 172.21.15.28:0/2623819809 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2bf00c35c0 0x7f2bf0156bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:19.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.734+0000 7f2bef400640 1 -- 172.21.15.28:0/2623819809 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2210976989 0 0) 0x7f2bf006f1b0 con 0x7f2bf00a96c0 2024-02-21T23:08:19.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.734+0000 7f2bef400640 1 -- 172.21.15.28:0/2623819809 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bd8003520 con 0x7f2bf00a96c0 2024-02-21T23:08:19.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.734+0000 7f2bf4a00640 1 -- 172.21.15.28:0/2623819809 >> v1:172.21.15.81:6789/0 conn(0x7f2bf0150640 legacy=0x7f2bf0152c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:19.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.734+0000 7f2bf4a00640 1 -- 172.21.15.28:0/2623819809 >> v1:172.21.15.28:6789/0 conn(0x7f2bf00a96c0 legacy=0x7f2bf014ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:19.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.734+0000 7f2bf4a00640 1 -- 172.21.15.28:0/2623819809 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bf00c2dc0 con 0x7f2bf00c35c0 2024-02-21T23:08:19.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.735+0000 7f2bf4a00640 1 --2- 172.21.15.28:0/2623819809 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2bf00c35c0 0x7f2bf0156bf0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2be40097d0 tx=0x7f2be402b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=26512f5a1eb3689e server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:19.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.736+0000 7f2bef400640 1 -- 172.21.15.28:0/2623819809 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2be4035020 con 0x7f2bf00c35c0 2024-02-21T23:08:19.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.736+0000 7f2bef400640 1 -- 172.21.15.28:0/2623819809 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2be4031810 con 0x7f2bf00c35c0 2024-02-21T23:08:19.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.736+0000 7f2bef400640 1 -- 172.21.15.28:0/2623819809 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2be4031be0 con 0x7f2bf00c35c0 2024-02-21T23:08:19.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.737+0000 7f2bf5e00640 1 -- 172.21.15.28:0/2623819809 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2bf00c35c0 msgr2=0x7f2bf0156bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:19.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.737+0000 7f2bf5e00640 1 --2- 172.21.15.28:0/2623819809 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2bf00c35c0 0x7f2bf0156bf0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2be40097d0 tx=0x7f2be402b8b0 comp rx=0 tx=0).stop 2024-02-21T23:08:19.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.737+0000 7f2bf5e00640 1 -- 172.21.15.28:0/2623819809 shutdown_connections 2024-02-21T23:08:19.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.737+0000 7f2bf5e00640 1 --2- 172.21.15.28:0/2623819809 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2bf00c35c0 0x7f2bf0156bf0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:19.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.737+0000 7f2bf5e00640 1 -- 172.21.15.28:0/2623819809 >> 172.21.15.28:0/2623819809 conn(0x7f2bf00a8690 msgr2=0x7f2bf00a8a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:19.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.737+0000 7f2bf5e00640 1 -- 172.21.15.28:0/2623819809 shutdown_connections 2024-02-21T23:08:19.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.737+0000 7f2bf5e00640 1 -- 172.21.15.28:0/2623819809 wait complete. 2024-02-21T23:08:19.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.738+0000 7f2bf5e00640 1 Processor -- start 2024-02-21T23:08:19.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.739+0000 7f2bf5e00640 1 -- start start 2024-02-21T23:08:19.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.739+0000 7f2bf5e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bf00a96c0 0x7f2bf014af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:19.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.739+0000 7f2bf5e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2bf00c35c0 0x7f2bf00c5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:19.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.739+0000 7f2bf5e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2bf0150640 0x7f2bf0148580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:19.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.739+0000 7f2bf5e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf00c39a0 con 0x7f2bf00a96c0 2024-02-21T23:08:19.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.739+0000 7f2bf5e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf006e7b0 con 0x7f2bf0150640 2024-02-21T23:08:19.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.739+0000 7f2bf5e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf00c3b20 con 0x7f2bf00c35c0 2024-02-21T23:08:19.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.740+0000 7f2bf4a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bf00a96c0 0x7f2bf014af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:19.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.740+0000 7f2bf4a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bf00a96c0 0x7f2bf014af50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57636/0 (socket says 172.21.15.28:57636) 2024-02-21T23:08:19.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.740+0000 7f2bf4a00640 1 -- 172.21.15.28:0/1394821083 learned_addr learned my addr 172.21.15.28:0/1394821083 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:19.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.740+0000 7f2befe00640 1 --2- 172.21.15.28:0/1394821083 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2bf00c35c0 0x7f2bf00c5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:19.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.740+0000 7f2bf5400640 1 --2- 172.21.15.28:0/1394821083 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2bf0150640 0x7f2bf0148580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:19.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.740+0000 7f2bf4a00640 1 -- 172.21.15.28:0/1394821083 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2bf00c35c0 msgr2=0x7f2bf00c5bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:19.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.740+0000 7f2bf4a00640 1 --2- 172.21.15.28:0/1394821083 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2bf00c35c0 0x7f2bf00c5bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:19.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.740+0000 7f2bf4a00640 1 -- 172.21.15.28:0/1394821083 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2bf0150640 msgr2=0x7f2bf0148580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:19.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.740+0000 7f2bf4a00640 1 --2- 172.21.15.28:0/1394821083 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2bf0150640 0x7f2bf0148580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:19.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.740+0000 7f2bf4a00640 1 -- 172.21.15.28:0/1394821083 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bf0146320 con 0x7f2bf00a96c0 2024-02-21T23:08:19.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.740+0000 7f2bf5400640 1 --2- 172.21.15.28:0/1394821083 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2bf0150640 0x7f2bf0148580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:08:19.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.741+0000 7f2bf4a00640 1 --2- 172.21.15.28:0/1394821083 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bf00a96c0 0x7f2bf014af50 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f2be40098f0 tx=0x7f2be40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:19.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.741+0000 7f2becc00640 1 -- 172.21.15.28:0/1394821083 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2be4041070 con 0x7f2bf00a96c0 2024-02-21T23:08:19.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.741+0000 7f2becc00640 1 -- 172.21.15.28:0/1394821083 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2be403ad00 con 0x7f2bf00a96c0 2024-02-21T23:08:19.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.741+0000 7f2becc00640 1 -- 172.21.15.28:0/1394821083 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2be4035040 con 0x7f2bf00a96c0 2024-02-21T23:08:19.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.741+0000 7f2bf5e00640 1 -- 172.21.15.28:0/1394821083 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bf0146040 con 0x7f2bf00a96c0 2024-02-21T23:08:19.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.741+0000 7f2bf5e00640 1 -- 172.21.15.28:0/1394821083 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bf0146dc0 con 0x7f2bf00a96c0 2024-02-21T23:08:19.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.742+0000 7f2bf5e00640 1 -- 172.21.15.28:0/1394821083 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bf004f290 con 0x7f2bf00a96c0 2024-02-21T23:08:19.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.744+0000 7f2becc00640 1 -- 172.21.15.28:0/1394821083 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2be404c060 con 0x7f2bf00a96c0 2024-02-21T23:08:19.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.745+0000 7f2becc00640 1 --2- 172.21.15.28:0/1394821083 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2bd006b4d0 0x7f2bd006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:19.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.745+0000 7f2becc00640 1 -- 172.21.15.28:0/1394821083 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f2be40abf20 con 0x7f2bf00a96c0 2024-02-21T23:08:19.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.746+0000 7f2befe00640 1 --2- 172.21.15.28:0/1394821083 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2bd006b4d0 0x7f2bd006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:19.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.747+0000 7f2befe00640 1 --2- 172.21.15.28:0/1394821083 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2bd006b4d0 0x7f2bd006d990 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f2bf014f5f0 tx=0x7f2bdc009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:19.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:19.750+0000 7f2becc00640 1 -- 172.21.15.28:0/1394821083 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f2be404b2c0 con 0x7f2bf00a96c0 2024-02-21T23:08:20.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.019+0000 7f2bf5e00640 1 -- 172.21.15.28:0/1394821083 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f2bf0145dd0 con 0x7f2bf00a96c0 2024-02-21T23:08:20.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.023+0000 7f2becc00640 1 -- 172.21.15.28:0/1394821083 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v16) v1 ==== 190+0+147 (secure 0 0 0) 0x7f2be407ac00 con 0x7f2bf00a96c0 2024-02-21T23:08:20.012 INFO:teuthology.orchestra.run.smithi028.stdout:[client.0] 2024-02-21T23:08:20.012 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQDTgNZlILvYLxAAoREWxKKT7YcXnhkgk8YvoQ== 2024-02-21T23:08:20.012 INFO:teuthology.orchestra.run.smithi028.stdout: caps mds = "allow" 2024-02-21T23:08:20.013 INFO:teuthology.orchestra.run.smithi028.stdout: caps mon = "allow r" 2024-02-21T23:08:20.013 INFO:teuthology.orchestra.run.smithi028.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-02-21T23:08:20.013 INFO:teuthology.orchestra.run.smithi028.stderr:updated caps for client.0 2024-02-21T23:08:20.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.026+0000 7f2bf5e00640 1 -- 172.21.15.28:0/1394821083 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2bd006b4d0 msgr2=0x7f2bd006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:20.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.026+0000 7f2bf5e00640 1 --2- 172.21.15.28:0/1394821083 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2bd006b4d0 0x7f2bd006d990 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f2bf014f5f0 tx=0x7f2bdc009040 comp rx=0 tx=0).stop 2024-02-21T23:08:20.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.026+0000 7f2bf5e00640 1 -- 172.21.15.28:0/1394821083 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bf00a96c0 msgr2=0x7f2bf014af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:20.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.026+0000 7f2bf5e00640 1 --2- 172.21.15.28:0/1394821083 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bf00a96c0 0x7f2bf014af50 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f2be40098f0 tx=0x7f2be40046c0 comp rx=0 tx=0).stop 2024-02-21T23:08:20.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.026+0000 7f2bf5e00640 1 -- 172.21.15.28:0/1394821083 shutdown_connections 2024-02-21T23:08:20.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.026+0000 7f2bf5e00640 1 --2- 172.21.15.28:0/1394821083 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2bd006b4d0 0x7f2bd006d990 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:20.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.026+0000 7f2bf5e00640 1 --2- 172.21.15.28:0/1394821083 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2bf00c35c0 0x7f2bf00c5bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:20.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.026+0000 7f2bf5e00640 1 --2- 172.21.15.28:0/1394821083 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2bf0150640 0x7f2bf0148580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:20.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.026+0000 7f2bf5e00640 1 --2- 172.21.15.28:0/1394821083 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bf00a96c0 0x7f2bf014af50 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:20.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.026+0000 7f2bf5e00640 1 -- 172.21.15.28:0/1394821083 >> 172.21.15.28:0/1394821083 conn(0x7f2bf00a8690 msgr2=0x7f2bf00a8a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:20.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.027+0000 7f2bf5e00640 1 -- 172.21.15.28:0/1394821083 shutdown_connections 2024-02-21T23:08:20.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.027+0000 7f2bf5e00640 1 -- 172.21.15.28:0/1394821083 wait complete. 2024-02-21T23:08:20.034 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:08:20.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.262+0000 7f4e92400640 1 Processor -- start 2024-02-21T23:08:20.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.263+0000 7f4e92400640 1 -- start start 2024-02-21T23:08:20.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.264+0000 7f4e92400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4e8c0aff90 0x7f4e8c0c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:20.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.264+0000 7f4e92400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e8c0c74c0 con 0x7f4e8c0c5720 2024-02-21T23:08:20.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.264+0000 7f4e92400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e8c0c77a0 con 0x7f4e8c06dbf0 2024-02-21T23:08:20.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.264+0000 7f4e92400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e8c0c7270 con 0x7f4e8c0aff90 2024-02-21T23:08:20.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.265+0000 7f4e91000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4e8c0aff90 0x7f4e8c0c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:20.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.265+0000 7f4e91a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f4e8c0c5720 0x7f4e8c15f890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:57436/0 (socket says 172.21.15.28:57436) 2024-02-21T23:08:20.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.265+0000 7f4e91a00640 1 -- 172.21.15.28:0/1744076654 learned_addr learned my addr 172.21.15.28:0/1744076654 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:08:20.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.265+0000 7f4e8b400640 1 -- 172.21.15.28:0/1744076654 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4145289056 0 0) 0x7f4e8c0c74c0 con 0x7f4e8c0c5720 2024-02-21T23:08:20.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.265+0000 7f4e91000640 1 -- 172.21.15.28:0/1744076654 >> v1:172.21.15.81:6789/0 conn(0x7f4e8c06dbf0 legacy=0x7f4e8c158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:20.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.265+0000 7f4e91000640 1 -- 172.21.15.28:0/1744076654 >> v1:172.21.15.28:6789/0 conn(0x7f4e8c0c5720 legacy=0x7f4e8c15f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:20.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.265+0000 7f4e91000640 1 -- 172.21.15.28:0/1744076654 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e8c0c7a80 con 0x7f4e8c0aff90 2024-02-21T23:08:20.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.267+0000 7f4e91000640 1 --2- 172.21.15.28:0/1744076654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e8c0aff90 0x7f4e8c0c51e0 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f4e7c0097a0 tx=0x7f4e7c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b4561d843222ad6 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:20.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.267+0000 7f4e8b400640 1 -- 172.21.15.28:0/1744076654 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e7c034070 con 0x7f4e8c0aff90 2024-02-21T23:08:20.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.267+0000 7f4e8b400640 1 -- 172.21.15.28:0/1744076654 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e7c030610 con 0x7f4e8c0aff90 2024-02-21T23:08:20.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.267+0000 7f4e8b400640 1 -- 172.21.15.28:0/1744076654 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e7c0309e0 con 0x7f4e8c0aff90 2024-02-21T23:08:20.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.267+0000 7f4e92400640 1 -- 172.21.15.28:0/1744076654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e8c0aff90 msgr2=0x7f4e8c0c51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:20.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.268+0000 7f4e92400640 1 --2- 172.21.15.28:0/1744076654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e8c0aff90 0x7f4e8c0c51e0 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f4e7c0097a0 tx=0x7f4e7c02b960 comp rx=0 tx=0).stop 2024-02-21T23:08:20.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.268+0000 7f4e92400640 1 -- 172.21.15.28:0/1744076654 shutdown_connections 2024-02-21T23:08:20.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.268+0000 7f4e92400640 1 --2- 172.21.15.28:0/1744076654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e8c0aff90 0x7f4e8c0c51e0 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:20.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.268+0000 7f4e92400640 1 -- 172.21.15.28:0/1744076654 >> 172.21.15.28:0/1744076654 conn(0x7f4e8c0ac370 msgr2=0x7f4e8c0ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:20.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.268+0000 7f4e92400640 1 -- 172.21.15.28:0/1744076654 shutdown_connections 2024-02-21T23:08:20.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.268+0000 7f4e92400640 1 -- 172.21.15.28:0/1744076654 wait complete. 2024-02-21T23:08:20.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.269+0000 7f4e92400640 1 Processor -- start 2024-02-21T23:08:20.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.269+0000 7f4e92400640 1 -- start start 2024-02-21T23:08:20.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.270+0000 7f4e92400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e8c06dbf0 0x7f4e8c0c2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:20.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.270+0000 7f4e92400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4e8c0aff90 0x7f4e8c0be690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:20.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.270+0000 7f4e92400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4e8c0c5720 0x7f4e8c1ee230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:20.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.270+0000 7f4e92400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e8c0c8760 con 0x7f4e8c06dbf0 2024-02-21T23:08:20.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.270+0000 7f4e92400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e8c158630 con 0x7f4e8c0c5720 2024-02-21T23:08:20.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.270+0000 7f4e92400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e8c0c88e0 con 0x7f4e8c0aff90 2024-02-21T23:08:20.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.271+0000 7f4e91000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e8c06dbf0 0x7f4e8c0c2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:20.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.271+0000 7f4e91000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e8c06dbf0 0x7f4e8c0c2f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57658/0 (socket says 172.21.15.28:57658) 2024-02-21T23:08:20.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.271+0000 7f4e91000640 1 -- 172.21.15.28:0/1121957700 learned_addr learned my addr 172.21.15.28:0/1121957700 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:20.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.271+0000 7f4e8be00640 1 --2- 172.21.15.28:0/1121957700 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4e8c0aff90 0x7f4e8c0be690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:20.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.271+0000 7f4e91000640 1 -- 172.21.15.28:0/1121957700 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4e8c0aff90 msgr2=0x7f4e8c0be690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:20.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.271+0000 7f4e91000640 1 --2- 172.21.15.28:0/1121957700 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4e8c0aff90 0x7f4e8c0be690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:20.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.271+0000 7f4e91000640 1 -- 172.21.15.28:0/1121957700 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4e8c0c5720 msgr2=0x7f4e8c1ee230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:20.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.271+0000 7f4e91000640 1 --2- 172.21.15.28:0/1121957700 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4e8c0c5720 0x7f4e8c1ee230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:20.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.271+0000 7f4e91000640 1 -- 172.21.15.28:0/1121957700 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e8c0c3490 con 0x7f4e8c06dbf0 2024-02-21T23:08:20.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.271+0000 7f4e91000640 1 --2- 172.21.15.28:0/1121957700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e8c06dbf0 0x7f4e8c0c2f50 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f4e7c02bf70 tx=0x7f4e7c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:20.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.272+0000 7f4e88c00640 1 -- 172.21.15.28:0/1121957700 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e7c034040 con 0x7f4e8c06dbf0 2024-02-21T23:08:20.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.272+0000 7f4e88c00640 1 -- 172.21.15.28:0/1121957700 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e7c039a10 con 0x7f4e8c06dbf0 2024-02-21T23:08:20.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.272+0000 7f4e88c00640 1 -- 172.21.15.28:0/1121957700 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e7c039de0 con 0x7f4e8c06dbf0 2024-02-21T23:08:20.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.272+0000 7f4e92400640 1 -- 172.21.15.28:0/1121957700 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e8c155890 con 0x7f4e8c06dbf0 2024-02-21T23:08:20.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.272+0000 7f4e92400640 1 -- 172.21.15.28:0/1121957700 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e8c0c2470 con 0x7f4e8c06dbf0 2024-02-21T23:08:20.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.273+0000 7f4e92400640 1 -- 172.21.15.28:0/1121957700 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e8c04f290 con 0x7f4e8c06dbf0 2024-02-21T23:08:20.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.274+0000 7f4e88c00640 1 -- 172.21.15.28:0/1121957700 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4e7c037070 con 0x7f4e8c06dbf0 2024-02-21T23:08:20.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.275+0000 7f4e88c00640 1 --2- 172.21.15.28:0/1121957700 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4e6806b4d0 0x7f4e6806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:20.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.275+0000 7f4e88c00640 1 -- 172.21.15.28:0/1121957700 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4e7c0ac4a0 con 0x7f4e8c06dbf0 2024-02-21T23:08:20.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.276+0000 7f4e8be00640 1 --2- 172.21.15.28:0/1121957700 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4e6806b4d0 0x7f4e6806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:20.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.277+0000 7f4e8be00640 1 --2- 172.21.15.28:0/1121957700 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4e6806b4d0 0x7f4e6806d990 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f4e74007fd0 tx=0x7f4e74009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:20.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.279+0000 7f4e88c00640 1 -- 172.21.15.28:0/1121957700 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4e7c07b180 con 0x7f4e8c06dbf0 2024-02-21T23:08:20.359 INFO:tasks.ceph.mds.d.smithi081.stderr:2024-02-21T23:08:20.370+0000 7fb376e00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-21T23:08:20.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.518+0000 7f4e92400640 1 -- 172.21.15.28:0/1121957700 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4e8c06d9d0 con 0x7f4e8c06dbf0 2024-02-21T23:08:20.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.520+0000 7f4e88c00640 1 -- 172.21.15.28:0/1121957700 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) v1 ==== 95+0+4946 (secure 0 0 0) 0x7f4e7c07aa80 con 0x7f4e8c06dbf0 2024-02-21T23:08:20.508 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:08:20.508 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":121,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5948,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3556518453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3556518453},{"type":"v1","addr":"172.21.15.28:6835","nonce":3556518453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116},{"gid":5951,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/3293264689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":3293264689},{"type":"v1","addr":"172.21.15.81:6836","nonce":3293264689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116},{"gid":7905,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/1773560593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1773560593},{"type":"v1","addr":"172.21.15.28:6834","nonce":1773560593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":115}],"filesystems":[{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:08:13.883141+0000","modified":"2024-02-21T23:08:16.980266+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7914},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7914":{"gid":7914,"name":"d","rank":0,"incarnation":117,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6837/1038096496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":1038096496},{"type":"v1","addr":"172.21.15.81:6837","nonce":1038096496}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-21T23:08:20.508 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 121 2024-02-21T23:08:20.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.522+0000 7f4e92400640 1 -- 172.21.15.28:0/1121957700 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4e6806b4d0 msgr2=0x7f4e6806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:20.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.522+0000 7f4e92400640 1 --2- 172.21.15.28:0/1121957700 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4e6806b4d0 0x7f4e6806d990 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f4e74007fd0 tx=0x7f4e74009040 comp rx=0 tx=0).stop 2024-02-21T23:08:20.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.522+0000 7f4e92400640 1 -- 172.21.15.28:0/1121957700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e8c06dbf0 msgr2=0x7f4e8c0c2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:20.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.522+0000 7f4e92400640 1 --2- 172.21.15.28:0/1121957700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e8c06dbf0 0x7f4e8c0c2f50 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f4e7c02bf70 tx=0x7f4e7c0046c0 comp rx=0 tx=0).stop 2024-02-21T23:08:20.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.522+0000 7f4e92400640 1 -- 172.21.15.28:0/1121957700 shutdown_connections 2024-02-21T23:08:20.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.522+0000 7f4e92400640 1 --2- 172.21.15.28:0/1121957700 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4e6806b4d0 0x7f4e6806d990 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:20.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.522+0000 7f4e92400640 1 --2- 172.21.15.28:0/1121957700 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4e8c0aff90 0x7f4e8c0be690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:20.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.522+0000 7f4e92400640 1 --2- 172.21.15.28:0/1121957700 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4e8c0c5720 0x7f4e8c1ee230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:20.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.522+0000 7f4e92400640 1 --2- 172.21.15.28:0/1121957700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e8c06dbf0 0x7f4e8c0c2f50 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:20.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.522+0000 7f4e92400640 1 -- 172.21.15.28:0/1121957700 >> 172.21.15.28:0/1121957700 conn(0x7f4e8c0ac370 msgr2=0x7f4e8c157da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:20.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.523+0000 7f4e92400640 1 -- 172.21.15.28:0/1121957700 shutdown_connections 2024-02-21T23:08:20.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:20.523+0000 7f4e92400640 1 -- 172.21.15.28:0/1121957700 wait complete. 2024-02-21T23:08:20.930 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 121, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:08:13.883141+0000', 'modified': '2024-02-21T23:08:16.980266+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7914}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7914': {'gid': 7914, 'name': 'd', 'rank': 0, 'incarnation': 117, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.81:6837/1038096496', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6835', 'nonce': 1038096496}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 1038096496}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [20], 'metadata_pool': 19, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:08:20.931 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-21T23:08:20.931 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-02-21T23:08:21.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.179+0000 7f7b22400640 1 Processor -- start 2024-02-21T23:08:21.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.179+0000 7f7b22400640 1 -- start start 2024-02-21T23:08:21.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.180+0000 7f7b22400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f7b1c149830 0x7f7b1c0a8e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:21.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.180+0000 7f7b22400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7b1c06d410 0x7f7b1c0c0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:21.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.180+0000 7f7b22400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b1c149c10 con 0x7f7b1c0c1420 2024-02-21T23:08:21.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.180+0000 7f7b22400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b1c1537d0 con 0x7f7b1c06d410 2024-02-21T23:08:21.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.180+0000 7f7b22400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b1c153ab0 con 0x7f7b1c149830 2024-02-21T23:08:21.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.181+0000 7f7b21a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f7b1c0c1420 0x7f7b1c0c5a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:57442/0 (socket says 172.21.15.28:57442) 2024-02-21T23:08:21.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.181+0000 7f7b21a00640 1 -- 172.21.15.28:0/3666238229 learned_addr learned my addr 172.21.15.28:0/3666238229 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:08:21.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.181+0000 7f7b21000640 1 --2- 172.21.15.28:0/3666238229 >> v2:172.21.15.81:3300/0 conn(0x7f7b1c149830 0x7f7b1c0a8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:21.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.181+0000 7f7b1be00640 1 --2- 172.21.15.28:0/3666238229 >> v2:172.21.15.28:3300/0 conn(0x7f7b1c06d410 0x7f7b1c0c0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:21.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.181+0000 7f7b1b400640 1 -- 172.21.15.28:0/3666238229 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3368948641 0 0) 0x7f7b1c149c10 con 0x7f7b1c0c1420 2024-02-21T23:08:21.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.181+0000 7f7b1b400640 1 -- 172.21.15.28:0/3666238229 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b00003520 con 0x7f7b1c0c1420 2024-02-21T23:08:21.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.181+0000 7f7b1be00640 1 -- 172.21.15.28:0/3666238229 >> v2:172.21.15.81:3300/0 conn(0x7f7b1c149830 msgr2=0x7f7b1c0a8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:21.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.181+0000 7f7b1be00640 1 --2- 172.21.15.28:0/3666238229 >> v2:172.21.15.81:3300/0 conn(0x7f7b1c149830 0x7f7b1c0a8e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:21.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.181+0000 7f7b1be00640 1 -- 172.21.15.28:0/3666238229 >> v1:172.21.15.28:6789/0 conn(0x7f7b1c0c1420 legacy=0x7f7b1c0c5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:21.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.181+0000 7f7b1be00640 1 -- 172.21.15.28:0/3666238229 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b1c0c4fa0 con 0x7f7b1c06d410 2024-02-21T23:08:21.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.183+0000 7f7b1be00640 1 --2- 172.21.15.28:0/3666238229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b1c06d410 0x7f7b1c0c0ee0 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f7b1c06f7c0 tx=0x7f7b0802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78c9f08cc07973e7 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:21.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.183+0000 7f7b1b400640 1 -- 172.21.15.28:0/3666238229 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b08034070 con 0x7f7b1c06d410 2024-02-21T23:08:21.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.183+0000 7f7b1b400640 1 -- 172.21.15.28:0/3666238229 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b08030610 con 0x7f7b1c06d410 2024-02-21T23:08:21.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.183+0000 7f7b1b400640 1 -- 172.21.15.28:0/3666238229 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b080309e0 con 0x7f7b1c06d410 2024-02-21T23:08:21.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.183+0000 7f7b22400640 1 -- 172.21.15.28:0/3666238229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b1c06d410 msgr2=0x7f7b1c0c0ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:21.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.183+0000 7f7b22400640 1 --2- 172.21.15.28:0/3666238229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b1c06d410 0x7f7b1c0c0ee0 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f7b1c06f7c0 tx=0x7f7b0802b960 comp rx=0 tx=0).stop 2024-02-21T23:08:21.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.184+0000 7f7b22400640 1 -- 172.21.15.28:0/3666238229 shutdown_connections 2024-02-21T23:08:21.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.184+0000 7f7b22400640 1 --2- 172.21.15.28:0/3666238229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b1c06d410 0x7f7b1c0c0ee0 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:21.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.184+0000 7f7b22400640 1 --2- 172.21.15.28:0/3666238229 >> v2:172.21.15.81:3300/0 conn(0x7f7b1c149830 0x7f7b1c0a8e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:21.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.184+0000 7f7b22400640 1 -- 172.21.15.28:0/3666238229 >> 172.21.15.28:0/3666238229 conn(0x7f7b1c0a8320 msgr2=0x7f7b1c0a8730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:21.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.184+0000 7f7b22400640 1 -- 172.21.15.28:0/3666238229 shutdown_connections 2024-02-21T23:08:21.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.184+0000 7f7b22400640 1 -- 172.21.15.28:0/3666238229 wait complete. 2024-02-21T23:08:21.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.185+0000 7f7b22400640 1 Processor -- start 2024-02-21T23:08:21.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.186+0000 7f7b22400640 1 -- start start 2024-02-21T23:08:21.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.186+0000 7f7b22400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7b1c06d410 0x7f7b1c0ab460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:21.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.186+0000 7f7b22400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7b1c0c1420 0x7f7b1c143ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:21.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.186+0000 7f7b22400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b1c149830 0x7f7b1c143ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:21.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.186+0000 7f7b22400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b1c0c6190 con 0x7f7b1c149830 2024-02-21T23:08:21.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.186+0000 7f7b22400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b1c0c5560 con 0x7f7b1c06d410 2024-02-21T23:08:21.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.187+0000 7f7b22400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b1c0c6310 con 0x7f7b1c0c1420 2024-02-21T23:08:21.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.187+0000 7f7b21a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b1c149830 0x7f7b1c143ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:21.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.187+0000 7f7b21a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b1c149830 0x7f7b1c143ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57676/0 (socket says 172.21.15.28:57676) 2024-02-21T23:08:21.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.187+0000 7f7b21a00640 1 -- 172.21.15.28:0/3362974390 learned_addr learned my addr 172.21.15.28:0/3362974390 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:21.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.187+0000 7f7b21000640 1 --2- 172.21.15.28:0/3362974390 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7b1c06d410 0x7f7b1c0ab460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:21.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.187+0000 7f7b1be00640 1 --2- 172.21.15.28:0/3362974390 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7b1c0c1420 0x7f7b1c143ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:21.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.187+0000 7f7b21a00640 1 -- 172.21.15.28:0/3362974390 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7b1c0c1420 msgr2=0x7f7b1c143ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:21.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.187+0000 7f7b21a00640 1 --2- 172.21.15.28:0/3362974390 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7b1c0c1420 0x7f7b1c143ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:21.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.187+0000 7f7b21a00640 1 -- 172.21.15.28:0/3362974390 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7b1c06d410 msgr2=0x7f7b1c0ab460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:21.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.187+0000 7f7b21a00640 1 --2- 172.21.15.28:0/3362974390 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7b1c06d410 0x7f7b1c0ab460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:21.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.187+0000 7f7b21a00640 1 -- 172.21.15.28:0/3362974390 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b1c14c630 con 0x7f7b1c149830 2024-02-21T23:08:21.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.188+0000 7f7b21a00640 1 --2- 172.21.15.28:0/3362974390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b1c149830 0x7f7b1c143ff0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f7b1400cb40 tx=0x7f7b1400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:21.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.188+0000 7f7b18c00640 1 -- 172.21.15.28:0/3362974390 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b14017070 con 0x7f7b1c149830 2024-02-21T23:08:21.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.188+0000 7f7b18c00640 1 -- 172.21.15.28:0/3362974390 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b1400aca0 con 0x7f7b1c149830 2024-02-21T23:08:21.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.188+0000 7f7b18c00640 1 -- 172.21.15.28:0/3362974390 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b14012620 con 0x7f7b1c149830 2024-02-21T23:08:21.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.188+0000 7f7b22400640 1 -- 172.21.15.28:0/3362974390 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b1c14d0d0 con 0x7f7b1c149830 2024-02-21T23:08:21.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.188+0000 7f7b22400640 1 -- 172.21.15.28:0/3362974390 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b1c14c8d0 con 0x7f7b1c149830 2024-02-21T23:08:21.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.190+0000 7f7b18c00640 1 -- 172.21.15.28:0/3362974390 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7b1401a070 con 0x7f7b1c149830 2024-02-21T23:08:21.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.191+0000 7f7b18c00640 1 --2- 172.21.15.28:0/3362974390 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7afc06b4d0 0x7f7afc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:21.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.191+0000 7f7b18c00640 1 -- 172.21.15.28:0/3362974390 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f7b1408a200 con 0x7f7b1c149830 2024-02-21T23:08:21.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.192+0000 7f7b269bf740 1 Processor -- start 2024-02-21T23:08:21.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.192+0000 7f7b269bf740 1 -- start start 2024-02-21T23:08:21.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.192+0000 7f7b21000640 1 --2- 172.21.15.28:0/3362974390 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7afc06b4d0 0x7f7afc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:21.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.193+0000 7f7b269bf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a035a5430 0x562a035a5810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:21.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.193+0000 7f7b269bf740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562a035a6740 0x562a035e1580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:21.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.193+0000 7f7b269bf740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x562a035e1ac0 0x562a035dfb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:21.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.193+0000 7f7b269bf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a03505470 con 0x562a035a5430 2024-02-21T23:08:21.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.193+0000 7f7b269bf740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a034aeaf0 con 0x562a035a6740 2024-02-21T23:08:21.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.193+0000 7f7b269bf740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x562a03501510 con 0x562a035e1ac0 2024-02-21T23:08:21.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.193+0000 7f7b21000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a035a5430 0x562a035a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:21.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.193+0000 7f7b21000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a035a5430 0x562a035a5810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57690/0 (socket says 172.21.15.28:57690) 2024-02-21T23:08:21.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.193+0000 7f7b21000640 1 -- 172.21.15.28:0/2747214193 learned_addr learned my addr 172.21.15.28:0/2747214193 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:21.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.193+0000 7f7b21000640 1 --2- 172.21.15.28:0/3362974390 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7afc06b4d0 0x7f7afc06d990 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c009a30 tx=0x7f7b0c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:21.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.193+0000 7f7b1be00640 1 --2- 172.21.15.28:0/2747214193 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562a035a6740 0x562a035e1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:21.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.193+0000 7f7b21000640 1 -- 172.21.15.28:0/2747214193 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x562a035e1ac0 msgr2=0x562a035dfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:21.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.193+0000 7f7b21000640 1 --2- 172.21.15.28:0/2747214193 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x562a035e1ac0 0x562a035dfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:21.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.193+0000 7f7b21000640 1 -- 172.21.15.28:0/2747214193 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562a035a6740 msgr2=0x562a035e1580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:21.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.193+0000 7f7b21000640 1 --2- 172.21.15.28:0/2747214193 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562a035a6740 0x562a035e1580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:21.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.193+0000 7f7b21000640 1 -- 172.21.15.28:0/2747214193 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562a034b5d90 con 0x562a035a5430 2024-02-21T23:08:21.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.194+0000 7f7b21000640 1 --2- 172.21.15.28:0/2747214193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a035a5430 0x562a035a5810 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c011d60 tx=0x7f7b0c014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e3cb0900be6c501 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:21.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.194+0000 7f7b11600640 1 -- 172.21.15.28:0/2747214193 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b0c011d90 con 0x562a035a5430 2024-02-21T23:08:21.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.194+0000 7f7b11600640 1 -- 172.21.15.28:0/2747214193 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b0c026070 con 0x562a035a5430 2024-02-21T23:08:21.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.194+0000 7f7b11600640 1 -- 172.21.15.28:0/2747214193 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b0c021640 con 0x562a035a5430 2024-02-21T23:08:21.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.195+0000 7f7b269bf740 1 -- 172.21.15.28:0/2747214193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a035a5430 msgr2=0x562a035a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:21.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.195+0000 7f7b269bf740 1 --2- 172.21.15.28:0/2747214193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a035a5430 0x562a035a5810 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c011d60 tx=0x7f7b0c014000 comp rx=0 tx=0).stop 2024-02-21T23:08:21.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.195+0000 7f7b269bf740 1 -- 172.21.15.28:0/2747214193 shutdown_connections 2024-02-21T23:08:21.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.195+0000 7f7b269bf740 1 --2- 172.21.15.28:0/2747214193 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x562a035e1ac0 0x562a035dfb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:21.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.195+0000 7f7b269bf740 1 --2- 172.21.15.28:0/2747214193 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562a035a6740 0x562a035e1580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:21.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.195+0000 7f7b269bf740 1 --2- 172.21.15.28:0/2747214193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a035a5430 0x562a035a5810 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:21.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.195+0000 7f7b269bf740 1 -- 172.21.15.28:0/2747214193 >> 172.21.15.28:0/2747214193 conn(0x562a035a1d80 msgr2=0x562a035c6790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:21.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.195+0000 7f7b269bf740 1 -- 172.21.15.28:0/2747214193 shutdown_connections 2024-02-21T23:08:21.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.195+0000 7f7b269bf740 1 -- 172.21.15.28:0/2747214193 wait complete. 2024-02-21T23:08:21.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.196+0000 7f7b269bf740 1 -- start start 2024-02-21T23:08:21.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.196+0000 7f7b269bf740 1 Processor -- start 2024-02-21T23:08:21.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.197+0000 7f7b269bf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a035a6740 0x562a035da7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:21.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.197+0000 7f7b269bf740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x562a035e1ac0 0x562a035f1a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:21.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.197+0000 7f7b269bf740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562a035c6880 0x562a035ddca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:21.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.197+0000 7f7b269bf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a034a4040 con 0x562a035a6740 2024-02-21T23:08:21.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.197+0000 7f7b269bf740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a035c6ea0 con 0x562a035c6880 2024-02-21T23:08:21.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.197+0000 7f7b269bf740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x562a035f2740 con 0x562a035e1ac0 2024-02-21T23:08:21.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.197+0000 7f7b269bf740 10 client.0 _ll_register_callbacks cb 0x562a034c1920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:08:21.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.197+0000 7f7b21000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a035a6740 0x562a035da7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:21.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.197+0000 7f7b21000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a035a6740 0x562a035da7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57694/0 (socket says 172.21.15.28:57694) 2024-02-21T23:08:21.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.197+0000 7f7b21000640 1 -- 172.21.15.28:0/2402126679 learned_addr learned my addr 172.21.15.28:0/2402126679 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:21.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.197+0000 7f7b1be00640 1 --2- 172.21.15.28:0/2402126679 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x562a035e1ac0 0x562a035f1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:21.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.197+0000 7f7b21a00640 1 --2- 172.21.15.28:0/2402126679 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562a035c6880 0x562a035ddca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:21.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.198+0000 7f7b21000640 1 -- 172.21.15.28:0/2402126679 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x562a035e1ac0 msgr2=0x562a035f1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:21.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.198+0000 7f7b21000640 1 --2- 172.21.15.28:0/2402126679 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x562a035e1ac0 0x562a035f1a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:21.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.198+0000 7f7b21000640 1 -- 172.21.15.28:0/2402126679 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562a035c6880 msgr2=0x562a035ddca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:21.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.198+0000 7f7b21000640 1 --2- 172.21.15.28:0/2402126679 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562a035c6880 0x562a035ddca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:21.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.198+0000 7f7b21000640 1 -- 172.21.15.28:0/2402126679 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562a034c3aa0 con 0x562a035a6740 2024-02-21T23:08:21.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.198+0000 7f7b21000640 1 --2- 172.21.15.28:0/2402126679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a035a6740 0x562a035da7a0 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c023040 tx=0x7f7b0c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60dd2a122c770ace server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:21.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.198+0000 7f7b06000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:08:21.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.198+0000 7f7b06000640 1 -- 172.21.15.28:0/2402126679 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b0c0119a0 con 0x562a035a6740 2024-02-21T23:08:21.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.198+0000 7f7b06000640 1 -- 172.21.15.28:0/2402126679 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(122..122 src has 1..122) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f7b0c009200 con 0x562a035a6740 2024-02-21T23:08:21.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.198+0000 7f7b269bf740 1 -- 172.21.15.28:0/2402126679 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7b0c026070 con 0x562a035a6740 2024-02-21T23:08:21.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.198+0000 7f7b06000640 1 -- 172.21.15.28:0/2402126679 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=121) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7b0c03e050 con 0x562a035a6740 2024-02-21T23:08:21.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.199+0000 7f7b269bf740 10 client.8043 fetch_fsmap learned FSMap version 121 2024-02-21T23:08:21.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.199+0000 7f7b269bf740 1 -- 172.21.15.28:0/2402126679 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=121,monmap=2+}) v3 -- 0x562a0321a2d0 con 0x562a035a6740 2024-02-21T23:08:21.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.199+0000 7f7b06000640 1 -- 172.21.15.28:0/2402126679 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b0c026070 con 0x562a035a6740 2024-02-21T23:08:21.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.199+0000 7f7b06000640 1 -- 172.21.15.28:0/2402126679 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 121) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f7b0c041020 con 0x562a035a6740 2024-02-21T23:08:21.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.199+0000 7f7b269bf740 10 client.8043 fetch_fsmap finished waiting for FSMap version 121 2024-02-21T23:08:21.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.199+0000 7f7b269bf740 10 client.8043 resolve_mds: resolved name 'd' to daemon mds.d 2024-02-21T23:08:21.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.199+0000 7f7b269bf740 20 client.8043 populate_metadata read hostname 'smithi028' 2024-02-21T23:08:21.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.199+0000 7f7b269bf740 1 --2- 172.21.15.28:0/2402126679 >> [v2:172.21.15.81:6835/1038096496,v1:172.21.15.81:6837/1038096496] conn(0x562a035f5510 0x562a036003d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:21.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.199+0000 7f7b269bf740 4 client.8043 mds_command: new command op to 7914 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-21T23:08:21.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.199+0000 7f7b269bf740 1 -- 172.21.15.28:0/2402126679 --> [v2:172.21.15.81:6835/1038096496,v1:172.21.15.81:6837/1038096496] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562a035129b0 con 0x562a035f5510 2024-02-21T23:08:21.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.200+0000 7f7b1be00640 1 --2- 172.21.15.28:0/2402126679 >> [v2:172.21.15.81:6835/1038096496,v1:172.21.15.81:6837/1038096496] conn(0x562a035f5510 0x562a036003d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:21.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.201+0000 7f7b1be00640 1 --2- 172.21.15.28:0/2402126679 >> [v2:172.21.15.81:6835/1038096496,v1:172.21.15.81:6837/1038096496] conn(0x562a035f5510 0x562a036003d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=72a69bde65eda7aa server_cookie=1266a003d58ca1af in_seq=0 out_seq=0 2024-02-21T23:08:21.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.201+0000 7f7b06000640 10 client.8043 ms_handle_connect on v2:172.21.15.81:6835/1038096496 2024-02-21T23:08:21.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.203+0000 7f7b06000640 1 -- 172.21.15.28:0/2402126679 <== mds.0 v2:172.21.15.81:6835/1038096496 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x562a035129b0 con 0x562a035f5510 2024-02-21T23:08:21.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.203+0000 7f7b06000640 10 client.8043 handle_command_reply: tid=0 2024-02-21T23:08:21.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.203+0000 7f7b269bf740 1 client.8043 shutdown 2024-02-21T23:08:21.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.203+0000 7f7b269bf740 1 -- 172.21.15.28:0/2402126679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a035a6740 msgr2=0x562a035da7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:21.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.203+0000 7f7b269bf740 1 --2- 172.21.15.28:0/2402126679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a035a6740 0x562a035da7a0 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c023040 tx=0x7f7b0c0046c0 comp rx=0 tx=0).stop 2024-02-21T23:08:21.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.203+0000 7f7b269bf740 1 -- 172.21.15.28:0/2402126679 shutdown_connections 2024-02-21T23:08:21.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.203+0000 7f7b269bf740 1 --2- 172.21.15.28:0/2402126679 >> [v2:172.21.15.81:6835/1038096496,v1:172.21.15.81:6837/1038096496] conn(0x562a035f5510 0x562a036003d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:21.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.203+0000 7f7b269bf740 1 --2- 172.21.15.28:0/2402126679 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x562a035e1ac0 0x562a035f1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:21.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.203+0000 7f7b269bf740 1 --2- 172.21.15.28:0/2402126679 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562a035c6880 0x562a035ddca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:21.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.203+0000 7f7b06000640 0 client.8043 ms_handle_reset on v2:172.21.15.81:6835/1038096496 2024-02-21T23:08:21.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.203+0000 7f7b269bf740 1 --2- 172.21.15.28:0/2402126679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a035a6740 0x562a035da7a0 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:21.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.203+0000 7f7b269bf740 1 -- 172.21.15.28:0/2402126679 >> 172.21.15.28:0/2402126679 conn(0x562a035a5430 msgr2=0x562a035e4ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:21.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.203+0000 7f7b269bf740 1 -- 172.21.15.28:0/2402126679 shutdown_connections 2024-02-21T23:08:21.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.203+0000 7f7b269bf740 1 -- 172.21.15.28:0/2402126679 wait complete. 2024-02-21T23:08:21.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.203+0000 7f7b269bf740 20 client.8043 trim_cache size 0 max 16384 2024-02-21T23:08:21.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.221+0000 7f7b269bf740 1 Processor -- start 2024-02-21T23:08:21.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.221+0000 7f7b269bf740 1 -- start start 2024-02-21T23:08:21.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.222+0000 7f7b269bf740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562a035a6740 0x562a035e4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:21.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.222+0000 7f7b269bf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a035c6880 0x562a035c53f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:21.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.222+0000 7f7b269bf740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x562a035e1ac0 0x562a035c5930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:21.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.222+0000 7f7b269bf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b1c0729e0 con 0x562a035c6880 2024-02-21T23:08:21.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.222+0000 7f7b269bf740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b1c1ec690 con 0x562a035a6740 2024-02-21T23:08:21.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.222+0000 7f7b269bf740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b1c0723e0 con 0x562a035e1ac0 2024-02-21T23:08:21.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.223+0000 7f7b1be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a035c6880 0x562a035c53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:21.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.223+0000 7f7b21000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562a035a6740 0x562a035e4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:21.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.223+0000 7f7b1be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a035c6880 0x562a035c53f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57698/0 (socket says 172.21.15.28:57698) 2024-02-21T23:08:21.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.223+0000 7f7b1be00640 1 -- 172.21.15.28:0/2039772107 learned_addr learned my addr 172.21.15.28:0/2039772107 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:21.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.223+0000 7f7b21a00640 1 --2- 172.21.15.28:0/2039772107 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x562a035e1ac0 0x562a035c5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:21.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.223+0000 7f7b1be00640 1 -- 172.21.15.28:0/2039772107 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x562a035e1ac0 msgr2=0x562a035c5930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:21.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.223+0000 7f7b1be00640 1 --2- 172.21.15.28:0/2039772107 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x562a035e1ac0 0x562a035c5930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:21.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.223+0000 7f7b1be00640 1 -- 172.21.15.28:0/2039772107 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562a035a6740 msgr2=0x562a035e4cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:21.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.223+0000 7f7b1be00640 1 --2- 172.21.15.28:0/2039772107 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562a035a6740 0x562a035e4cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:21.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.223+0000 7f7b1be00640 1 -- 172.21.15.28:0/2039772107 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562a032e46b0 con 0x562a035c6880 2024-02-21T23:08:21.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.223+0000 7f7b1be00640 1 --2- 172.21.15.28:0/2039772107 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a035c6880 0x562a035c53f0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f7b0802bf70 tx=0x7f7b08030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf47f7a13e885ca5 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:21.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.224+0000 7f7b10c00640 1 -- 172.21.15.28:0/2039772107 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b08034040 con 0x562a035c6880 2024-02-21T23:08:21.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.224+0000 7f7b10c00640 1 -- 172.21.15.28:0/2039772107 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b08039870 con 0x562a035c6880 2024-02-21T23:08:21.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.224+0000 7f7b10c00640 1 -- 172.21.15.28:0/2039772107 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b08039c20 con 0x562a035c6880 2024-02-21T23:08:21.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.224+0000 7f7b269bf740 1 -- 172.21.15.28:0/2039772107 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a035c6880 msgr2=0x562a035c53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:21.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.224+0000 7f7b269bf740 1 --2- 172.21.15.28:0/2039772107 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a035c6880 0x562a035c53f0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f7b0802bf70 tx=0x7f7b08030000 comp rx=0 tx=0).stop 2024-02-21T23:08:21.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.224+0000 7f7b269bf740 1 -- 172.21.15.28:0/2039772107 shutdown_connections 2024-02-21T23:08:21.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.224+0000 7f7b269bf740 1 --2- 172.21.15.28:0/2039772107 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x562a035e1ac0 0x562a035c5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:21.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.224+0000 7f7b269bf740 1 --2- 172.21.15.28:0/2039772107 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a035c6880 0x562a035c53f0 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:21.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.224+0000 7f7b269bf740 1 --2- 172.21.15.28:0/2039772107 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562a035a6740 0x562a035e4cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:21.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.224+0000 7f7b269bf740 1 -- 172.21.15.28:0/2039772107 >> 172.21.15.28:0/2039772107 conn(0x562a035a5430 msgr2=0x562a035c6cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:21.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.224+0000 7f7b269bf740 1 -- 172.21.15.28:0/2039772107 shutdown_connections 2024-02-21T23:08:21.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.224+0000 7f7b269bf740 1 -- 172.21.15.28:0/2039772107 wait complete. 2024-02-21T23:08:21.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.225+0000 7f7b269bf740 1 -- start start 2024-02-21T23:08:21.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.225+0000 7f7b269bf740 1 Processor -- start 2024-02-21T23:08:21.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.226+0000 7f7b269bf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a035a6740 0x562a0359a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:21.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.226+0000 7f7b269bf740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562a035c6880 0x562a03653e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:21.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.226+0000 7f7b269bf740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x562a035e1ac0 0x562a0364db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:21.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.226+0000 7f7b269bf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a035a1110 con 0x562a035a6740 2024-02-21T23:08:21.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.226+0000 7f7b269bf740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a035db5e0 con 0x562a035c6880 2024-02-21T23:08:21.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.226+0000 7f7b269bf740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x562a035de2b0 con 0x562a035e1ac0 2024-02-21T23:08:21.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.226+0000 7f7b269bf740 10 client.0 _ll_register_callbacks cb 0x7f7b0c011c00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:08:21.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.226+0000 7f7b21000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a035a6740 0x562a0359a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:21.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.227+0000 7f7b21000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a035a6740 0x562a0359a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57710/0 (socket says 172.21.15.28:57710) 2024-02-21T23:08:21.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.227+0000 7f7b21000640 1 -- 172.21.15.28:0/1198561251 learned_addr learned my addr 172.21.15.28:0/1198561251 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:21.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.227+0000 7f7b21a00640 1 --2- 172.21.15.28:0/1198561251 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x562a035e1ac0 0x562a0364db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:21.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.227+0000 7f7b1be00640 1 --2- 172.21.15.28:0/1198561251 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562a035c6880 0x562a03653e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:21.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.227+0000 7f7b21000640 1 -- 172.21.15.28:0/1198561251 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x562a035e1ac0 msgr2=0x562a0364db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:21.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.227+0000 7f7b21000640 1 --2- 172.21.15.28:0/1198561251 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x562a035e1ac0 0x562a0364db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:21.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.227+0000 7f7b21000640 1 -- 172.21.15.28:0/1198561251 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562a035c6880 msgr2=0x562a03653e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:21.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.227+0000 7f7b21000640 1 --2- 172.21.15.28:0/1198561251 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562a035c6880 0x562a03653e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:21.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.227+0000 7f7b21000640 1 -- 172.21.15.28:0/1198561251 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562a035c25b0 con 0x562a035a6740 2024-02-21T23:08:21.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.228+0000 7f7b21000640 1 --2- 172.21.15.28:0/1198561251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a035a6740 0x562a0359a9f0 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c023040 tx=0x7f7b0c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cd889c42519ccbb server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:21.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.228+0000 7f7b06000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:08:21.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.228+0000 7f7b06000640 1 -- 172.21.15.28:0/1198561251 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b0c046070 con 0x562a035a6740 2024-02-21T23:08:21.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.228+0000 7f7b269bf740 1 -- 172.21.15.28:0/1198561251 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7b08039870 con 0x562a035a6740 2024-02-21T23:08:21.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.228+0000 7f7b06000640 1 -- 172.21.15.28:0/1198561251 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(122..122 src has 1..122) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f7b0c036690 con 0x562a035a6740 2024-02-21T23:08:21.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.228+0000 7f7b06000640 1 -- 172.21.15.28:0/1198561251 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=121) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7b0c051050 con 0x562a035a6740 2024-02-21T23:08:21.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.228+0000 7f7b269bf740 10 client.8049 fetch_fsmap learned FSMap version 121 2024-02-21T23:08:21.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.228+0000 7f7b269bf740 1 -- 172.21.15.28:0/1198561251 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=121,monmap=2+}) v3 -- 0x562a03605220 con 0x562a035a6740 2024-02-21T23:08:21.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.228+0000 7f7b06000640 1 -- 172.21.15.28:0/1198561251 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b08039870 con 0x562a035a6740 2024-02-21T23:08:21.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.228+0000 7f7b06000640 1 -- 172.21.15.28:0/1198561251 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 121) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f7b0c054020 con 0x562a035a6740 2024-02-21T23:08:21.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.228+0000 7f7b269bf740 10 client.8049 fetch_fsmap finished waiting for FSMap version 121 2024-02-21T23:08:21.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.229+0000 7f7b269bf740 10 client.8049 resolve_mds: resolved name 'd' to daemon mds.d 2024-02-21T23:08:21.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.229+0000 7f7b269bf740 20 client.8049 populate_metadata read hostname 'smithi028' 2024-02-21T23:08:21.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.229+0000 7f7b269bf740 1 --2- 172.21.15.28:0/1198561251 >> [v2:172.21.15.81:6835/1038096496,v1:172.21.15.81:6837/1038096496] conn(0x562a035edbf0 0x562a035edfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:21.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.229+0000 7f7b269bf740 4 client.8049 mds_command: new command op to 7914 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-21T23:08:21.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.229+0000 7f7b269bf740 1 -- 172.21.15.28:0/1198561251 --> [v2:172.21.15.81:6835/1038096496,v1:172.21.15.81:6837/1038096496] -- command(tid 0: {"prefix": "status"}) v1 -- 0x562a0350cd00 con 0x562a035edbf0 2024-02-21T23:08:21.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.229+0000 7f7b1be00640 1 --2- 172.21.15.28:0/1198561251 >> [v2:172.21.15.81:6835/1038096496,v1:172.21.15.81:6837/1038096496] conn(0x562a035edbf0 0x562a035edfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:21.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.230+0000 7f7b1be00640 1 --2- 172.21.15.28:0/1198561251 >> [v2:172.21.15.81:6835/1038096496,v1:172.21.15.81:6837/1038096496] conn(0x562a035edbf0 0x562a035edfd0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f2adde389efcf0e5 server_cookie=ad9f8960536cd6ae in_seq=0 out_seq=0 2024-02-21T23:08:21.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.230+0000 7f7b06000640 10 client.8049 ms_handle_connect on v2:172.21.15.81:6835/1038096496 2024-02-21T23:08:21.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.231+0000 7f7b06000640 1 -- 172.21.15.28:0/1198561251 <== mds.0 v2:172.21.15.81:6835/1038096496 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x562a0350cd00 con 0x562a035edbf0 2024-02-21T23:08:21.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.231+0000 7f7b06000640 10 client.8049 handle_command_reply: tid=0 2024-02-21T23:08:21.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.231+0000 7f7b269bf740 1 client.8049 shutdown 2024-02-21T23:08:21.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.231+0000 7f7b269bf740 1 -- 172.21.15.28:0/1198561251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a035a6740 msgr2=0x562a0359a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:21.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.231+0000 7f7b269bf740 1 --2- 172.21.15.28:0/1198561251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a035a6740 0x562a0359a9f0 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c023040 tx=0x7f7b0c016000 comp rx=0 tx=0).stop 2024-02-21T23:08:21.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.231+0000 7f7b269bf740 1 -- 172.21.15.28:0/1198561251 shutdown_connections 2024-02-21T23:08:21.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.231+0000 7f7b269bf740 1 --2- 172.21.15.28:0/1198561251 >> [v2:172.21.15.81:6835/1038096496,v1:172.21.15.81:6837/1038096496] conn(0x562a035edbf0 0x562a035edfd0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:21.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.231+0000 7f7b269bf740 1 --2- 172.21.15.28:0/1198561251 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x562a035e1ac0 0x562a0364db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:21.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.231+0000 7f7b269bf740 1 --2- 172.21.15.28:0/1198561251 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x562a035c6880 0x562a03653e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:21.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.231+0000 7f7b06000640 0 client.8049 ms_handle_reset on v2:172.21.15.81:6835/1038096496 2024-02-21T23:08:21.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.231+0000 7f7b269bf740 1 --2- 172.21.15.28:0/1198561251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a035a6740 0x562a0359a9f0 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:21.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.231+0000 7f7b269bf740 1 -- 172.21.15.28:0/1198561251 >> 172.21.15.28:0/1198561251 conn(0x562a035a5430 msgr2=0x562a035eee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:21.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.231+0000 7f7b269bf740 1 -- 172.21.15.28:0/1198561251 shutdown_connections 2024-02-21T23:08:21.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.231+0000 7f7b269bf740 1 -- 172.21.15.28:0/1198561251 wait complete. 2024-02-21T23:08:21.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.231+0000 7f7b269bf740 20 client.8049 trim_cache size 0 max 16384 2024-02-21T23:08:21.224 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-21T23:08:21.224 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac", 2024-02-21T23:08:21.224 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-02-21T23:08:21.224 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 7914, 2024-02-21T23:08:21.224 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-02-21T23:08:21.224 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-02-21T23:08:21.224 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-02-21T23:08:21.224 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 6.811820354, 2024-02-21T23:08:21.224 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 121, 2024-02-21T23:08:21.224 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 122, 2024-02-21T23:08:21.224 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 122, 2024-02-21T23:08:21.225 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 12.304645088999999 2024-02-21T23:08:21.225 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-21T23:08:21.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.232+0000 7f7b22400640 1 -- 172.21.15.28:0/3362974390 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7afc06b4d0 msgr2=0x7f7afc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:21.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.232+0000 7f7b22400640 1 --2- 172.21.15.28:0/3362974390 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7afc06b4d0 0x7f7afc06d990 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c009a30 tx=0x7f7b0c008040 comp rx=0 tx=0).stop 2024-02-21T23:08:21.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.232+0000 7f7b22400640 1 -- 172.21.15.28:0/3362974390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b1c149830 msgr2=0x7f7b1c143ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:21.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.232+0000 7f7b22400640 1 --2- 172.21.15.28:0/3362974390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b1c149830 0x7f7b1c143ff0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f7b1400cb40 tx=0x7f7b1400a060 comp rx=0 tx=0).stop 2024-02-21T23:08:21.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.232+0000 7f7b22400640 1 -- 172.21.15.28:0/3362974390 shutdown_connections 2024-02-21T23:08:21.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.232+0000 7f7b22400640 1 --2- 172.21.15.28:0/3362974390 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7afc06b4d0 0x7f7afc06d990 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:21.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.232+0000 7f7b22400640 1 --2- 172.21.15.28:0/3362974390 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7b1c0c1420 0x7f7b1c143ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:21.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.232+0000 7f7b22400640 1 --2- 172.21.15.28:0/3362974390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b1c149830 0x7f7b1c143ff0 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:21.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.232+0000 7f7b22400640 1 --2- 172.21.15.28:0/3362974390 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7b1c06d410 0x7f7b1c0ab460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:21.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.232+0000 7f7b22400640 1 -- 172.21.15.28:0/3362974390 >> 172.21.15.28:0/3362974390 conn(0x7f7b1c0a8320 msgr2=0x7f7b1c1523b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:21.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.232+0000 7f7b22400640 1 -- 172.21.15.28:0/3362974390 shutdown_connections 2024-02-21T23:08:21.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.232+0000 7f7b22400640 1 -- 172.21.15.28:0/3362974390 wait complete. 2024-02-21T23:08:21.239 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:08:21.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.475+0000 7f8a1b200640 1 Processor -- start 2024-02-21T23:08:21.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.475+0000 7f8a1b200640 1 -- start start 2024-02-21T23:08:21.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.476+0000 7f8a1b200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8a1c150450 0x7f8a1c0a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:21.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.477+0000 7f8a1b200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f8a1c06ee40 0x7f8a1c1536f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:21.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.477+0000 7f8a1b200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a1c0c2760 con 0x7f8a1c156af0 2024-02-21T23:08:21.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.477+0000 7f8a1b200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a1c0c22c0 con 0x7f8a1c06ee40 2024-02-21T23:08:21.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.477+0000 7f8a1b200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a1c156660 con 0x7f8a1c150450 2024-02-21T23:08:21.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.477+0000 7f8a1a800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f8a1c156af0 0x7f8a1c06dff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:57450/0 (socket says 172.21.15.28:57450) 2024-02-21T23:08:21.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.477+0000 7f8a1a800640 1 -- 172.21.15.28:0/806869476 learned_addr learned my addr 172.21.15.28:0/806869476 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:08:21.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.477+0000 7f8a19e00640 1 --2- 172.21.15.28:0/806869476 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8a1c150450 0x7f8a1c0a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:21.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.477+0000 7f8a19400640 1 --2- 172.21.15.28:0/806869476 >> v2:172.21.15.81:3300/0 conn(0x7f8a1c06ee40 0x7f8a1c1536f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:21.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.477+0000 7f8a18a00640 1 -- 172.21.15.28:0/806869476 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4148042941 0 0) 0x7f8a1c0c2760 con 0x7f8a1c156af0 2024-02-21T23:08:21.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.477+0000 7f8a18a00640 1 -- 172.21.15.28:0/806869476 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a00003520 con 0x7f8a1c156af0 2024-02-21T23:08:21.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.478+0000 7f8a18a00640 1 -- 172.21.15.28:0/806869476 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4222860574 0 0) 0x7f8a00003520 con 0x7f8a1c156af0 2024-02-21T23:08:21.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.478+0000 7f8a18a00640 1 -- 172.21.15.28:0/806869476 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8a1c150450 msgr2=0x7f8a1c0a5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:21.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.478+0000 7f8a18a00640 1 --2- 172.21.15.28:0/806869476 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8a1c150450 0x7f8a1c0a5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:21.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.478+0000 7f8a18a00640 1 -- 172.21.15.28:0/806869476 >> v2:172.21.15.81:3300/0 conn(0x7f8a1c06ee40 msgr2=0x7f8a1c1536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:21.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.478+0000 7f8a18a00640 1 --2- 172.21.15.28:0/806869476 >> v2:172.21.15.81:3300/0 conn(0x7f8a1c06ee40 0x7f8a1c1536f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:21.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.478+0000 7f8a18a00640 1 -- 172.21.15.28:0/806869476 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a1c153c30 con 0x7f8a1c156af0 2024-02-21T23:08:21.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.478+0000 7f8a18a00640 1 -- 172.21.15.28:0/806869476 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f8a140019f0 con 0x7f8a1c156af0 2024-02-21T23:08:21.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.478+0000 7f8a18a00640 1 -- 172.21.15.28:0/806869476 >> v1:172.21.15.28:6789/0 conn(0x7f8a1c156af0 legacy=0x7f8a1c06dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:21.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.478+0000 7f8a18a00640 1 --2- 172.21.15.28:0/806869476 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8a1c150450 0x7f8a000039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:21.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.478+0000 7f8a18a00640 1 --2- 172.21.15.28:0/806869476 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8a00003f80 0x7f8a00004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:21.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.478+0000 7f8a18a00640 1 --2- 172.21.15.28:0/806869476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a00004d40 0x7f8a000051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:21.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.478+0000 7f8a18a00640 1 -- 172.21.15.28:0/806869476 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a1c156660 con 0x7f8a00004d40 2024-02-21T23:08:21.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.478+0000 7f8a18a00640 1 -- 172.21.15.28:0/806869476 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a0000b9c0 con 0x7f8a00003f80 2024-02-21T23:08:21.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.478+0000 7f8a18a00640 1 -- 172.21.15.28:0/806869476 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a0000bb40 con 0x7f8a1c150450 2024-02-21T23:08:21.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.478+0000 7f8a1a800640 1 --2- 172.21.15.28:0/806869476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a00004d40 0x7f8a000051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:21.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.479+0000 7f8a19e00640 1 --2- 172.21.15.28:0/806869476 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8a1c150450 0x7f8a000039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:21.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.479+0000 7f8a1a800640 1 -- 172.21.15.28:0/806869476 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8a1c150450 msgr2=0x7f8a000039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:21.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.479+0000 7f8a1a800640 1 --2- 172.21.15.28:0/806869476 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8a1c150450 0x7f8a000039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:21.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.479+0000 7f8a19400640 1 --2- 172.21.15.28:0/806869476 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8a00003f80 0x7f8a00004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:21.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.479+0000 7f8a1a800640 1 -- 172.21.15.28:0/806869476 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8a00003f80 msgr2=0x7f8a00004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:21.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.479+0000 7f8a1a800640 1 --2- 172.21.15.28:0/806869476 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8a00003f80 0x7f8a00004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:21.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.479+0000 7f8a1a800640 1 -- 172.21.15.28:0/806869476 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a00003520 con 0x7f8a00004d40 2024-02-21T23:08:21.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.479+0000 7f8a19400640 1 --2- 172.21.15.28:0/806869476 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8a00003f80 0x7f8a00004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:08:21.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.479+0000 7f8a19e00640 1 --2- 172.21.15.28:0/806869476 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8a1c150450 0x7f8a000039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:08:21.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.480+0000 7f8a1a800640 1 --2- 172.21.15.28:0/806869476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a00004d40 0x7f8a000051c0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f8a1400a7d0 tx=0x7f8a00005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=740eb4d912cd1104 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:21.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.480+0000 7f8a18a00640 1 -- 172.21.15.28:0/806869476 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a1400a3f0 con 0x7f8a00004d40 2024-02-21T23:08:21.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.480+0000 7f8a18a00640 1 -- 172.21.15.28:0/806869476 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a14034630 con 0x7f8a00004d40 2024-02-21T23:08:21.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.480+0000 7f8a18a00640 1 -- 172.21.15.28:0/806869476 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a14034a80 con 0x7f8a00004d40 2024-02-21T23:08:21.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.481+0000 7f8a1b200640 1 -- 172.21.15.28:0/806869476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a00004d40 msgr2=0x7f8a000051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:21.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.481+0000 7f8a1b200640 1 --2- 172.21.15.28:0/806869476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a00004d40 0x7f8a000051c0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f8a1400a7d0 tx=0x7f8a00005830 comp rx=0 tx=0).stop 2024-02-21T23:08:21.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.481+0000 7f8a1a800640 1 -- 172.21.15.28:0/806869476 reap_dead start 2024-02-21T23:08:21.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.481+0000 7f8a1b200640 1 -- 172.21.15.28:0/806869476 shutdown_connections 2024-02-21T23:08:21.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.481+0000 7f8a1b200640 1 -- 172.21.15.28:0/806869476 >> 172.21.15.28:0/806869476 conn(0x7f8a1c06b630 msgr2=0x7f8a1c0c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:21.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.481+0000 7f8a1b200640 1 -- 172.21.15.28:0/806869476 shutdown_connections 2024-02-21T23:08:21.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.481+0000 7f8a1b200640 1 -- 172.21.15.28:0/806869476 wait complete. 2024-02-21T23:08:21.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.483+0000 7f8a1b200640 1 Processor -- start 2024-02-21T23:08:21.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.483+0000 7f8a1b200640 1 -- start start 2024-02-21T23:08:21.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.484+0000 7f8a1b200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8a00004d40 0x7f8a1c14d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:21.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.484+0000 7f8a1b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a1c149ca0 0x7f8a1c1ee990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:21.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.484+0000 7f8a1b200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8a1c149290 0x7f8a1c149740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:21.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.484+0000 7f8a1b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a1c0c2cb0 con 0x7f8a1c149ca0 2024-02-21T23:08:21.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.484+0000 7f8a1b200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a1c0c29d0 con 0x7f8a00004d40 2024-02-21T23:08:21.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.484+0000 7f8a1b200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a1c06e700 con 0x7f8a1c149290 2024-02-21T23:08:21.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.484+0000 7f8a19400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a1c149ca0 0x7f8a1c1ee990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:21.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.484+0000 7f8a19400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a1c149ca0 0x7f8a1c1ee990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57718/0 (socket says 172.21.15.28:57718) 2024-02-21T23:08:21.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.484+0000 7f8a19400640 1 -- 172.21.15.28:0/1345854443 learned_addr learned my addr 172.21.15.28:0/1345854443 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:21.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.485+0000 7f8a19400640 1 -- 172.21.15.28:0/1345854443 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8a1c149290 msgr2=0x7f8a1c149740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:21.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.485+0000 7f8a1a800640 1 --2- 172.21.15.28:0/1345854443 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8a1c149290 0x7f8a1c149740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:21.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.485+0000 7f8a19e00640 1 --2- 172.21.15.28:0/1345854443 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8a00004d40 0x7f8a1c14d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:21.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.485+0000 7f8a19400640 1 --2- 172.21.15.28:0/1345854443 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8a1c149290 0x7f8a1c149740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:21.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.485+0000 7f8a19400640 1 -- 172.21.15.28:0/1345854443 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8a00004d40 msgr2=0x7f8a1c14d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:21.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.485+0000 7f8a19400640 1 --2- 172.21.15.28:0/1345854443 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8a00004d40 0x7f8a1c14d5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:21.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.485+0000 7f8a19400640 1 -- 172.21.15.28:0/1345854443 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a1c14ebc0 con 0x7f8a1c149ca0 2024-02-21T23:08:21.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.485+0000 7f8a19400640 1 --2- 172.21.15.28:0/1345854443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a1c149ca0 0x7f8a1c1ee990 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f8a080109d0 tx=0x7f8a08010e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:21.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.485+0000 7f8a12000640 1 -- 172.21.15.28:0/1345854443 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a0800f040 con 0x7f8a1c149ca0 2024-02-21T23:08:21.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.485+0000 7f8a12000640 1 -- 172.21.15.28:0/1345854443 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a0800ebc0 con 0x7f8a1c149ca0 2024-02-21T23:08:21.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.485+0000 7f8a12000640 1 -- 172.21.15.28:0/1345854443 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a0801c400 con 0x7f8a1c149ca0 2024-02-21T23:08:21.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.485+0000 7f8a1b200640 1 -- 172.21.15.28:0/1345854443 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a1c1ee1d0 con 0x7f8a1c149ca0 2024-02-21T23:08:21.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.485+0000 7f8a1b200640 1 -- 172.21.15.28:0/1345854443 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a1c14fd50 con 0x7f8a1c149ca0 2024-02-21T23:08:21.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.487+0000 7f8a1b200640 1 -- 172.21.15.28:0/1345854443 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a1c04f290 con 0x7f8a1c149ca0 2024-02-21T23:08:21.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.488+0000 7f8a12000640 1 -- 172.21.15.28:0/1345854443 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8a0801c5a0 con 0x7f8a1c149ca0 2024-02-21T23:08:21.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.489+0000 7f8a12000640 1 --2- 172.21.15.28:0/1345854443 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f89fc06b500 0x7f89fc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:21.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.489+0000 7f8a12000640 1 -- 172.21.15.28:0/1345854443 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f8a0808aa60 con 0x7f8a1c149ca0 2024-02-21T23:08:21.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.489+0000 7f8a19e00640 1 --2- 172.21.15.28:0/1345854443 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f89fc06b500 0x7f89fc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:21.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.490+0000 7f8a19e00640 1 --2- 172.21.15.28:0/1345854443 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f89fc06b500 0x7f89fc06d9c0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f8a0c0099c0 tx=0x7f8a0c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:21.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.493+0000 7f8a12000640 1 -- 172.21.15.28:0/1345854443 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f8a08018030 con 0x7f8a1c149ca0 2024-02-21T23:08:21.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.732+0000 7f8a1b200640 1 -- 172.21.15.28:0/1345854443 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8a1c0ac890 con 0x7f8a1c149ca0 2024-02-21T23:08:21.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.733+0000 7f8a12000640 1 -- 172.21.15.28:0/1345854443 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) v1 ==== 95+0+4946 (secure 0 0 0) 0x7f8a08059190 con 0x7f8a1c149ca0 2024-02-21T23:08:21.722 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:08:21.722 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":121,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5948,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3556518453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3556518453},{"type":"v1","addr":"172.21.15.28:6835","nonce":3556518453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116},{"gid":5951,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/3293264689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":3293264689},{"type":"v1","addr":"172.21.15.81:6836","nonce":3293264689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116},{"gid":7905,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/1773560593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1773560593},{"type":"v1","addr":"172.21.15.28:6834","nonce":1773560593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":115}],"filesystems":[{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:08:13.883141+0000","modified":"2024-02-21T23:08:16.980266+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7914},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7914":{"gid":7914,"name":"d","rank":0,"incarnation":117,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6837/1038096496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":1038096496},{"type":"v1","addr":"172.21.15.81:6837","nonce":1038096496}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-21T23:08:21.722 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 121 2024-02-21T23:08:21.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.736+0000 7f8a1b200640 1 -- 172.21.15.28:0/1345854443 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f89fc06b500 msgr2=0x7f89fc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:21.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.736+0000 7f8a1b200640 1 --2- 172.21.15.28:0/1345854443 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f89fc06b500 0x7f89fc06d9c0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f8a0c0099c0 tx=0x7f8a0c008040 comp rx=0 tx=0).stop 2024-02-21T23:08:21.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.736+0000 7f8a1b200640 1 -- 172.21.15.28:0/1345854443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a1c149ca0 msgr2=0x7f8a1c1ee990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:21.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.736+0000 7f8a1b200640 1 --2- 172.21.15.28:0/1345854443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a1c149ca0 0x7f8a1c1ee990 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f8a080109d0 tx=0x7f8a08010e90 comp rx=0 tx=0).stop 2024-02-21T23:08:21.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.736+0000 7f8a1b200640 1 -- 172.21.15.28:0/1345854443 shutdown_connections 2024-02-21T23:08:21.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.736+0000 7f8a1b200640 1 --2- 172.21.15.28:0/1345854443 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f89fc06b500 0x7f89fc06d9c0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:21.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.736+0000 7f8a1b200640 1 --2- 172.21.15.28:0/1345854443 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8a1c149290 0x7f8a1c149740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:21.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.736+0000 7f8a1b200640 1 --2- 172.21.15.28:0/1345854443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a1c149ca0 0x7f8a1c1ee990 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:21.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.736+0000 7f8a1b200640 1 --2- 172.21.15.28:0/1345854443 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8a00004d40 0x7f8a1c14d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:21.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.736+0000 7f8a1b200640 1 -- 172.21.15.28:0/1345854443 >> 172.21.15.28:0/1345854443 conn(0x7f8a1c06b630 msgr2=0x7f8a1c153420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:21.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.737+0000 7f8a1b200640 1 -- 172.21.15.28:0/1345854443 shutdown_connections 2024-02-21T23:08:21.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.737+0000 7f8a1b200640 1 -- 172.21.15.28:0/1345854443 wait complete. 2024-02-21T23:08:21.745 INFO:tasks.cephfs.test_failover:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 9, "epoch": 121, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 9, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-02-21T23:08:13.883141+0000", "damaged": [], "data_pools": [ 20 ], "enabled": true, "epoch": 121, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7914": { "addr": "172.21.15.81:6837/1038096496", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6835", "nonce": 1038096496, "type": "v2" }, { "addr": "172.21.15.81:6837", "nonce": 1038096496, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7914, "incarnation": 117, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 19, "modified": "2024-02-21T23:08:16.980266+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7914 } } } ], "standbys": [ { "addr": "172.21.15.28:6835/3556518453", "addrs": { "addrvec": [ { "addr": "172.21.15.28:6833", "nonce": 3556518453, "type": "v2" }, { "addr": "172.21.15.28:6835", "nonce": 3556518453, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 116, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5948, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.81:6836/3293264689", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6834", "nonce": 3293264689, "type": "v2" }, { "addr": "172.21.15.81:6836", "nonce": 3293264689, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 116, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5951, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.28:6834/1773560593", "addrs": { "addrvec": [ { "addr": "172.21.15.28:6832", "nonce": 1773560593, "type": "v2" }, { "addr": "172.21.15.28:6834", "nonce": 1773560593, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 115, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7905, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-02-21T23:08:21.745 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-02-21T23:08:21.745 INFO:tasks.cephfs.test_failover:current = [{'gid': 7905, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.28:6834/1773560593', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 1773560593}, {'type': 'v1', 'addr': '172.21.15.28:6834', 'nonce': 1773560593}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 115}, {'gid': 5951, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.81:6836/3293264689', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6834', 'nonce': 3293264689}, {'type': 'v1', 'addr': '172.21.15.81:6836', 'nonce': 3293264689}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 116}, {'gid': 5948, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.28:6835/3556518453', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6833', 'nonce': 3556518453}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3556518453}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 116}, {'gid': 7914, 'name': 'd', 'rank': 0, 'incarnation': 117, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.81:6837/1038096496', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6835', 'nonce': 1038096496}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 1038096496}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}] 2024-02-21T23:08:21.746 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.d mds_join_fs cephfs 2024-02-21T23:08:21.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.988+0000 7f0d3a800640 1 Processor -- start 2024-02-21T23:08:21.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.988+0000 7f0d3a800640 1 -- start start 2024-02-21T23:08:21.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.990+0000 7f0d3a800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0d3406b0e0 0x7f0d340c5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:21.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.990+0000 7f0d3a800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0d34150700 0x7f0d34156500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:21.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.990+0000 7f0d3a800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d340c61e0 con 0x7f0d340c5d80 2024-02-21T23:08:21.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.990+0000 7f0d3a800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d34153bc0 con 0x7f0d3406b0e0 2024-02-21T23:08:21.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.990+0000 7f0d3a800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d34152fb0 con 0x7f0d34150700 2024-02-21T23:08:21.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.990+0000 7f0d33400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0d3406b0e0 0x7f0d340c5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:21.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.990+0000 7f0d2aa00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f0d340c5d80 0x7f0d3414fff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:57452/0 (socket says 172.21.15.28:57452) 2024-02-21T23:08:21.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.990+0000 7f0d2aa00640 1 -- 172.21.15.28:0/2609284076 learned_addr learned my addr 172.21.15.28:0/2609284076 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:08:21.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.991+0000 7f0d33e00640 1 --2- 172.21.15.28:0/2609284076 >> v2:172.21.15.81:3300/0 conn(0x7f0d34150700 0x7f0d34156500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:21.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.991+0000 7f0d33400640 1 -- 172.21.15.28:0/2609284076 >> v2:172.21.15.81:3300/0 conn(0x7f0d34150700 msgr2=0x7f0d34156500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:21.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.991+0000 7f0d33400640 1 --2- 172.21.15.28:0/2609284076 >> v2:172.21.15.81:3300/0 conn(0x7f0d34150700 0x7f0d34156500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:21.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.991+0000 7f0d33400640 1 -- 172.21.15.28:0/2609284076 >> v1:172.21.15.28:6789/0 conn(0x7f0d340c5d80 legacy=0x7f0d3414fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:21.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.991+0000 7f0d33400640 1 -- 172.21.15.28:0/2609284076 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d34153290 con 0x7f0d3406b0e0 2024-02-21T23:08:21.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.991+0000 7f0d32a00640 1 -- 172.21.15.28:0/2609284076 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1311847380 0 0) 0x7f0d340c61e0 con 0x7f0d340c5d80 2024-02-21T23:08:21.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.992+0000 7f0d33400640 1 --2- 172.21.15.28:0/2609284076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d3406b0e0 0x7f0d340c5840 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f0d340a64b0 tx=0x7f0d2402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=968ddfbc1e824651 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:21.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.993+0000 7f0d32a00640 1 -- 172.21.15.28:0/2609284076 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d24039070 con 0x7f0d3406b0e0 2024-02-21T23:08:21.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.993+0000 7f0d32a00640 1 -- 172.21.15.28:0/2609284076 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d2402cbf0 con 0x7f0d3406b0e0 2024-02-21T23:08:21.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.993+0000 7f0d32a00640 1 -- 172.21.15.28:0/2609284076 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d24034440 con 0x7f0d3406b0e0 2024-02-21T23:08:21.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.993+0000 7f0d3a800640 1 -- 172.21.15.28:0/2609284076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d3406b0e0 msgr2=0x7f0d340c5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:21.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.993+0000 7f0d3a800640 1 --2- 172.21.15.28:0/2609284076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d3406b0e0 0x7f0d340c5840 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f0d340a64b0 tx=0x7f0d2402be40 comp rx=0 tx=0).stop 2024-02-21T23:08:21.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.993+0000 7f0d3a800640 1 -- 172.21.15.28:0/2609284076 shutdown_connections 2024-02-21T23:08:21.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.993+0000 7f0d3a800640 1 --2- 172.21.15.28:0/2609284076 >> v2:172.21.15.81:3300/0 conn(0x7f0d34150700 0x7f0d34156500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:21.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.993+0000 7f0d3a800640 1 --2- 172.21.15.28:0/2609284076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d3406b0e0 0x7f0d340c5840 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:21.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.993+0000 7f0d3a800640 1 -- 172.21.15.28:0/2609284076 >> 172.21.15.28:0/2609284076 conn(0x7f0d340a85a0 msgr2=0x7f0d340a8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:21.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.993+0000 7f0d3a800640 1 -- 172.21.15.28:0/2609284076 shutdown_connections 2024-02-21T23:08:21.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.993+0000 7f0d3a800640 1 -- 172.21.15.28:0/2609284076 wait complete. 2024-02-21T23:08:21.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.995+0000 7f0d3a800640 1 Processor -- start 2024-02-21T23:08:21.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.995+0000 7f0d3a800640 1 -- start start 2024-02-21T23:08:21.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.996+0000 7f0d3a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d3406b0e0 0x7f0d340ab230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:21.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.996+0000 7f0d3a800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d340c5d80 0x7f0d341e8540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:21.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.996+0000 7f0d3a800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0d34150700 0x7f0d341e8a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:21.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.996+0000 7f0d3a800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d3406b4c0 con 0x7f0d3406b0e0 2024-02-21T23:08:21.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.996+0000 7f0d3a800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d34153850 con 0x7f0d340c5d80 2024-02-21T23:08:21.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.996+0000 7f0d3a800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d3406b640 con 0x7f0d34150700 2024-02-21T23:08:21.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.996+0000 7f0d33400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d3406b0e0 0x7f0d340ab230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:21.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.996+0000 7f0d33400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d3406b0e0 0x7f0d340ab230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57724/0 (socket says 172.21.15.28:57724) 2024-02-21T23:08:21.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.996+0000 7f0d33400640 1 -- 172.21.15.28:0/930158757 learned_addr learned my addr 172.21.15.28:0/930158757 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:21.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.997+0000 7f0d33e00640 1 --2- 172.21.15.28:0/930158757 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0d34150700 0x7f0d341e8a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:21.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.997+0000 7f0d2aa00640 1 --2- 172.21.15.28:0/930158757 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d340c5d80 0x7f0d341e8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:21.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.997+0000 7f0d33400640 1 -- 172.21.15.28:0/930158757 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0d34150700 msgr2=0x7f0d341e8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:21.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.997+0000 7f0d33400640 1 --2- 172.21.15.28:0/930158757 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0d34150700 0x7f0d341e8a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:21.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.997+0000 7f0d33400640 1 -- 172.21.15.28:0/930158757 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d340c5d80 msgr2=0x7f0d341e8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:21.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.997+0000 7f0d33400640 1 --2- 172.21.15.28:0/930158757 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d340c5d80 0x7f0d341e8540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:21.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.997+0000 7f0d33400640 1 -- 172.21.15.28:0/930158757 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d34152d40 con 0x7f0d3406b0e0 2024-02-21T23:08:21.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.997+0000 7f0d33400640 1 --2- 172.21.15.28:0/930158757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d3406b0e0 0x7f0d340ab230 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f0d24037040 tx=0x7f0d24004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:21.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.997+0000 7f0d2be00640 1 -- 172.21.15.28:0/930158757 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d24039050 con 0x7f0d3406b0e0 2024-02-21T23:08:21.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.998+0000 7f0d2be00640 1 -- 172.21.15.28:0/930158757 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d24045070 con 0x7f0d3406b0e0 2024-02-21T23:08:21.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.998+0000 7f0d2be00640 1 -- 172.21.15.28:0/930158757 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d2403e850 con 0x7f0d3406b0e0 2024-02-21T23:08:21.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.998+0000 7f0d3a800640 1 -- 172.21.15.28:0/930158757 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d341553f0 con 0x7f0d3406b0e0 2024-02-21T23:08:21.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.998+0000 7f0d3a800640 1 -- 172.21.15.28:0/930158757 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d341ebf70 con 0x7f0d3406b0e0 2024-02-21T23:08:21.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:21.999+0000 7f0d3a800640 1 -- 172.21.15.28:0/930158757 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d34067f30 con 0x7f0d3406b0e0 2024-02-21T23:08:21.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.000+0000 7f0d2be00640 1 -- 172.21.15.28:0/930158757 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0d2403e9f0 con 0x7f0d3406b0e0 2024-02-21T23:08:21.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.001+0000 7f0d2be00640 1 --2- 172.21.15.28:0/930158757 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0d1006b500 0x7f0d1006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:21.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.001+0000 7f0d2be00640 1 -- 172.21.15.28:0/930158757 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f0d240ad410 con 0x7f0d3406b0e0 2024-02-21T23:08:21.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.001+0000 7f0d2aa00640 1 --2- 172.21.15.28:0/930158757 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0d1006b500 0x7f0d1006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:21.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.003+0000 7f0d2aa00640 1 --2- 172.21.15.28:0/930158757 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0d1006b500 0x7f0d1006d9c0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f0d1c007fd0 tx=0x7f0d1c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:21.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.006+0000 7f0d2be00640 1 -- 172.21.15.28:0/930158757 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f0d2407c210 con 0x7f0d3406b0e0 2024-02-21T23:08:22.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.209+0000 7f0d3a800640 1 -- 172.21.15.28:0/930158757 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) v1 -- 0x7f0d3414b0d0 con 0x7f0d3406b0e0 2024-02-21T23:08:22.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.215+0000 7f0d2be00640 1 -- 172.21.15.28:0/930158757 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v42)=0 v42) v1 ==== 116+0+0 (secure 0 0 0) 0x7f0d2407bb10 con 0x7f0d3406b0e0 2024-02-21T23:08:22.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.218+0000 7f0d3a800640 1 -- 172.21.15.28:0/930158757 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0d1006b500 msgr2=0x7f0d1006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:22.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.218+0000 7f0d3a800640 1 --2- 172.21.15.28:0/930158757 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0d1006b500 0x7f0d1006d9c0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f0d1c007fd0 tx=0x7f0d1c009040 comp rx=0 tx=0).stop 2024-02-21T23:08:22.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.218+0000 7f0d3a800640 1 -- 172.21.15.28:0/930158757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d3406b0e0 msgr2=0x7f0d340ab230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:22.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.218+0000 7f0d3a800640 1 --2- 172.21.15.28:0/930158757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d3406b0e0 0x7f0d340ab230 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f0d24037040 tx=0x7f0d24004a40 comp rx=0 tx=0).stop 2024-02-21T23:08:22.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.218+0000 7f0d3a800640 1 -- 172.21.15.28:0/930158757 shutdown_connections 2024-02-21T23:08:22.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.218+0000 7f0d3a800640 1 --2- 172.21.15.28:0/930158757 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0d1006b500 0x7f0d1006d9c0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:22.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.218+0000 7f0d3a800640 1 --2- 172.21.15.28:0/930158757 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0d34150700 0x7f0d341e8a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:22.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.218+0000 7f0d3a800640 1 --2- 172.21.15.28:0/930158757 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d340c5d80 0x7f0d341e8540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:22.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.218+0000 7f0d3a800640 1 --2- 172.21.15.28:0/930158757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d3406b0e0 0x7f0d340ab230 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:22.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.218+0000 7f0d3a800640 1 -- 172.21.15.28:0/930158757 >> 172.21.15.28:0/930158757 conn(0x7f0d340a85a0 msgr2=0x7f0d340729d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:22.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.219+0000 7f0d3a800640 1 -- 172.21.15.28:0/930158757 shutdown_connections 2024-02-21T23:08:22.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.219+0000 7f0d3a800640 1 -- 172.21.15.28:0/930158757 wait complete. 2024-02-21T23:08:22.225 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:08:22.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.463+0000 7f6369200640 1 Processor -- start 2024-02-21T23:08:22.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.463+0000 7f6369200640 1 -- start start 2024-02-21T23:08:22.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.465+0000 7f6369200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f63641524a0 0x7f63640c7620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:22.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.465+0000 7f6369200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f636406ba00 con 0x7f63640c45d0 2024-02-21T23:08:22.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.465+0000 7f6369200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f636406f2d0 con 0x7f6364155c20 2024-02-21T23:08:22.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.465+0000 7f6369200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6364154ff0 con 0x7f63641524a0 2024-02-21T23:08:22.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.465+0000 7f6363400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f63641524a0 0x7f63640c7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:22.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.465+0000 7f6362a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f63640c45d0 0x7f6364155510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:57458/0 (socket says 172.21.15.28:57458) 2024-02-21T23:08:22.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.465+0000 7f6362a00640 1 -- 172.21.15.28:0/1633695546 learned_addr learned my addr 172.21.15.28:0/1633695546 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:08:22.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.466+0000 7f6362000640 1 -- 172.21.15.28:0/1633695546 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3966128281 0 0) 0x7f636406ba00 con 0x7f63640c45d0 2024-02-21T23:08:22.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.466+0000 7f6362000640 1 -- 172.21.15.28:0/1633695546 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6348003520 con 0x7f63640c45d0 2024-02-21T23:08:22.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.466+0000 7f6363400640 1 -- 172.21.15.28:0/1633695546 >> v1:172.21.15.81:6789/0 conn(0x7f6364155c20 legacy=0x7f636415f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:22.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.466+0000 7f6363400640 1 -- 172.21.15.28:0/1633695546 >> v1:172.21.15.28:6789/0 conn(0x7f63640c45d0 legacy=0x7f6364155510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:22.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.466+0000 7f6363400640 1 -- 172.21.15.28:0/1633695546 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f636406aee0 con 0x7f63641524a0 2024-02-21T23:08:22.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.467+0000 7f6363400640 1 --2- 172.21.15.28:0/1633695546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63641524a0 0x7f63640c7620 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f63540097a0 tx=0x7f635402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df05350da1aaf54d server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:22.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.468+0000 7f6362000640 1 -- 172.21.15.28:0/1633695546 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6354034070 con 0x7f63641524a0 2024-02-21T23:08:22.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.468+0000 7f6362000640 1 -- 172.21.15.28:0/1633695546 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6354030610 con 0x7f63641524a0 2024-02-21T23:08:22.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.468+0000 7f6362000640 1 -- 172.21.15.28:0/1633695546 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63540309e0 con 0x7f63641524a0 2024-02-21T23:08:22.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.468+0000 7f6369200640 1 -- 172.21.15.28:0/1633695546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63641524a0 msgr2=0x7f63640c7620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:22.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.468+0000 7f6369200640 1 --2- 172.21.15.28:0/1633695546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63641524a0 0x7f63640c7620 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f63540097a0 tx=0x7f635402b960 comp rx=0 tx=0).stop 2024-02-21T23:08:22.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.468+0000 7f6369200640 1 -- 172.21.15.28:0/1633695546 shutdown_connections 2024-02-21T23:08:22.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.468+0000 7f6369200640 1 --2- 172.21.15.28:0/1633695546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63641524a0 0x7f63640c7620 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:22.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.468+0000 7f6369200640 1 -- 172.21.15.28:0/1633695546 >> 172.21.15.28:0/1633695546 conn(0x7f6364067050 msgr2=0x7f63640a84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:22.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.469+0000 7f6369200640 1 -- 172.21.15.28:0/1633695546 shutdown_connections 2024-02-21T23:08:22.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.469+0000 7f6369200640 1 -- 172.21.15.28:0/1633695546 wait complete. 2024-02-21T23:08:22.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.470+0000 7f6369200640 1 Processor -- start 2024-02-21T23:08:22.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.470+0000 7f6369200640 1 -- start start 2024-02-21T23:08:22.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.471+0000 7f6369200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63640c45d0 0x7f63641ee060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:22.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.471+0000 7f6369200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f63641524a0 0x7f63640abd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:22.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.471+0000 7f6369200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63640ac260 0x7f636414e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:22.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.471+0000 7f6369200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63640c4fa0 con 0x7f63640ac260 2024-02-21T23:08:22.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.471+0000 7f6369200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f636406f5b0 con 0x7f63640c45d0 2024-02-21T23:08:22.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.471+0000 7f6369200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f636406b460 con 0x7f63641524a0 2024-02-21T23:08:22.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.472+0000 7f6363e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63640ac260 0x7f636414e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:22.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.472+0000 7f6363e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63640ac260 0x7f636414e150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57750/0 (socket says 172.21.15.28:57750) 2024-02-21T23:08:22.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.472+0000 7f6363e00640 1 -- 172.21.15.28:0/2297465827 learned_addr learned my addr 172.21.15.28:0/2297465827 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:22.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.472+0000 7f6362a00640 1 --2- 172.21.15.28:0/2297465827 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f63641524a0 0x7f63640abd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:22.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.472+0000 7f6363400640 1 --2- 172.21.15.28:0/2297465827 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63640c45d0 0x7f63641ee060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:22.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.472+0000 7f6363e00640 1 -- 172.21.15.28:0/2297465827 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f63641524a0 msgr2=0x7f63640abd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:22.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.472+0000 7f6363e00640 1 --2- 172.21.15.28:0/2297465827 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f63641524a0 0x7f63640abd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:22.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.472+0000 7f6363e00640 1 -- 172.21.15.28:0/2297465827 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63640c45d0 msgr2=0x7f63641ee060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:22.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.472+0000 7f6363e00640 1 --2- 172.21.15.28:0/2297465827 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63640c45d0 0x7f63641ee060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:22.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.472+0000 7f6363e00640 1 -- 172.21.15.28:0/2297465827 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63641de8c0 con 0x7f63640ac260 2024-02-21T23:08:22.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.472+0000 7f6363e00640 1 --2- 172.21.15.28:0/2297465827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63640ac260 0x7f636414e150 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f635c00c6b0 tx=0x7f635c00cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:22.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.473+0000 7f635b400640 1 -- 172.21.15.28:0/2297465827 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f635c003b70 con 0x7f63640ac260 2024-02-21T23:08:22.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.473+0000 7f635b400640 1 -- 172.21.15.28:0/2297465827 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f635c003d30 con 0x7f63640ac260 2024-02-21T23:08:22.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.473+0000 7f635b400640 1 -- 172.21.15.28:0/2297465827 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f635c005310 con 0x7f63640ac260 2024-02-21T23:08:22.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.473+0000 7f6369200640 1 -- 172.21.15.28:0/2297465827 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63641de5e0 con 0x7f63640ac260 2024-02-21T23:08:22.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.473+0000 7f6369200640 1 -- 172.21.15.28:0/2297465827 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63641ee5a0 con 0x7f63640ac260 2024-02-21T23:08:22.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.474+0000 7f6369200640 1 -- 172.21.15.28:0/2297465827 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f636404f290 con 0x7f63640ac260 2024-02-21T23:08:22.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.475+0000 7f635b400640 1 -- 172.21.15.28:0/2297465827 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f635c0054b0 con 0x7f63640ac260 2024-02-21T23:08:22.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.476+0000 7f635b400640 1 --2- 172.21.15.28:0/2297465827 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f634406b500 0x7f634406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:22.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.476+0000 7f635b400640 1 -- 172.21.15.28:0/2297465827 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f635c08a120 con 0x7f63640ac260 2024-02-21T23:08:22.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.477+0000 7f6363400640 1 --2- 172.21.15.28:0/2297465827 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f634406b500 0x7f634406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:22.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.478+0000 7f6363400640 1 --2- 172.21.15.28:0/2297465827 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f634406b500 0x7f634406d9c0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f6354036000 tx=0x7f635403b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:22.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.480+0000 7f635b400640 1 -- 172.21.15.28:0/2297465827 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f63641de8c0 con 0x7f63640ac260 2024-02-21T23:08:22.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.720+0000 7f6369200640 1 -- 172.21.15.28:0/2297465827 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f63640c4a70 con 0x7f63640ac260 2024-02-21T23:08:22.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.721+0000 7f635b400640 1 -- 172.21.15.28:0/2297465827 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) v1 ==== 95+0+4946 (secure 0 0 0) 0x7f635c058ea0 con 0x7f63640ac260 2024-02-21T23:08:22.710 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:08:22.711 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":121,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5948,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3556518453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3556518453},{"type":"v1","addr":"172.21.15.28:6835","nonce":3556518453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116},{"gid":5951,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/3293264689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":3293264689},{"type":"v1","addr":"172.21.15.81:6836","nonce":3293264689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116},{"gid":7905,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/1773560593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1773560593},{"type":"v1","addr":"172.21.15.28:6834","nonce":1773560593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":115}],"filesystems":[{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:08:13.883141+0000","modified":"2024-02-21T23:08:16.980266+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7914},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7914":{"gid":7914,"name":"d","rank":0,"incarnation":117,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6837/1038096496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":1038096496},{"type":"v1","addr":"172.21.15.81:6837","nonce":1038096496}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-21T23:08:22.711 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 121 2024-02-21T23:08:22.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.724+0000 7f6369200640 1 -- 172.21.15.28:0/2297465827 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f634406b500 msgr2=0x7f634406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:22.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.724+0000 7f6369200640 1 --2- 172.21.15.28:0/2297465827 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f634406b500 0x7f634406d9c0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f6354036000 tx=0x7f635403b040 comp rx=0 tx=0).stop 2024-02-21T23:08:22.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.724+0000 7f6369200640 1 -- 172.21.15.28:0/2297465827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63640ac260 msgr2=0x7f636414e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:22.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.724+0000 7f6369200640 1 --2- 172.21.15.28:0/2297465827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63640ac260 0x7f636414e150 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f635c00c6b0 tx=0x7f635c00cb70 comp rx=0 tx=0).stop 2024-02-21T23:08:22.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.724+0000 7f6369200640 1 -- 172.21.15.28:0/2297465827 shutdown_connections 2024-02-21T23:08:22.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.724+0000 7f6369200640 1 --2- 172.21.15.28:0/2297465827 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f634406b500 0x7f634406d9c0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:22.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.724+0000 7f6369200640 1 --2- 172.21.15.28:0/2297465827 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f63641524a0 0x7f63640abd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:22.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.724+0000 7f6369200640 1 --2- 172.21.15.28:0/2297465827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63640ac260 0x7f636414e150 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:22.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.724+0000 7f6369200640 1 --2- 172.21.15.28:0/2297465827 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63640c45d0 0x7f63641ee060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:22.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.724+0000 7f6369200640 1 -- 172.21.15.28:0/2297465827 >> 172.21.15.28:0/2297465827 conn(0x7f6364067050 msgr2=0x7f6364072c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:22.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.724+0000 7f6369200640 1 -- 172.21.15.28:0/2297465827 shutdown_connections 2024-02-21T23:08:22.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:22.724+0000 7f6369200640 1 -- 172.21.15.28:0/2297465827 wait complete. 2024-02-21T23:08:23.135 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 121, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:08:13.883141+0000', 'modified': '2024-02-21T23:08:16.980266+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7914}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7914': {'gid': 7914, 'name': 'd', 'rank': 0, 'incarnation': 117, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.81:6837/1038096496', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6835', 'nonce': 1038096496}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 1038096496}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [20], 'metadata_pool': 19, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:08:23.136 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-21T23:08:23.136 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-02-21T23:08:23.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.377+0000 7fcc0b200640 1 Processor -- start 2024-02-21T23:08:23.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.377+0000 7fcc0b200640 1 -- start start 2024-02-21T23:08:23.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.379+0000 7fcc0b200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcc0c1524c0 0x7fcc0c155a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:23.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.379+0000 7fcc0b200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcc0c0c85c0 0x7fcc0c0c70d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:23.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.379+0000 7fcc0b200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc0c06df50 con 0x7fcc0c0c5040 2024-02-21T23:08:23.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.379+0000 7fcc0b200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc0c0c5520 con 0x7fcc0c1524c0 2024-02-21T23:08:23.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.379+0000 7fcc0b200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0c0c5800 con 0x7fcc0c0c85c0 2024-02-21T23:08:23.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.379+0000 7fcc09e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcc0c1524c0 0x7fcc0c155a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:23.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.379+0000 7fcc09e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcc0c1524c0 0x7fcc0c155a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57760/0 (socket says 172.21.15.28:57760) 2024-02-21T23:08:23.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.379+0000 7fcc09e00640 1 -- 172.21.15.28:0/1579894549 learned_addr learned my addr 172.21.15.28:0/1579894549 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:23.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.379+0000 7fcc09e00640 1 -- 172.21.15.28:0/1579894549 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcc0c0c85c0 msgr2=0x7fcc0c0c70d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:23.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.379+0000 7fcc0a800640 1 --2- 172.21.15.28:0/1579894549 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcc0c0c85c0 0x7fcc0c0c70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:23.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.380+0000 7fcc09e00640 1 --2- 172.21.15.28:0/1579894549 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcc0c0c85c0 0x7fcc0c0c70d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:23.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.380+0000 7fcc09e00640 1 -- 172.21.15.28:0/1579894549 >> v1:172.21.15.81:6789/0 conn(0x7fcc0c0c5040 legacy=0x7fcc0c06ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:23.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.380+0000 7fcc09e00640 1 -- 172.21.15.28:0/1579894549 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc0c06c670 con 0x7fcc0c1524c0 2024-02-21T23:08:23.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.381+0000 7fcc09e00640 1 --2- 172.21.15.28:0/1579894549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc0c1524c0 0x7fcc0c155a10 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc009870 tx=0x7fcbfc02b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f525385fe89f3958 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:23.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.381+0000 7fcc09400640 1 -- 172.21.15.28:0/1579894549 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbfc038070 con 0x7fcc0c1524c0 2024-02-21T23:08:23.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.381+0000 7fcc09400640 1 -- 172.21.15.28:0/1579894549 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbfc031740 con 0x7fcc0c1524c0 2024-02-21T23:08:23.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.381+0000 7fcc09400640 1 -- 172.21.15.28:0/1579894549 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbfc02f050 con 0x7fcc0c1524c0 2024-02-21T23:08:23.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.382+0000 7fcc0b200640 1 -- 172.21.15.28:0/1579894549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc0c1524c0 msgr2=0x7fcc0c155a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:23.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.382+0000 7fcc0b200640 1 --2- 172.21.15.28:0/1579894549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc0c1524c0 0x7fcc0c155a10 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc009870 tx=0x7fcbfc02b7e0 comp rx=0 tx=0).stop 2024-02-21T23:08:23.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.382+0000 7fcc0b200640 1 -- 172.21.15.28:0/1579894549 shutdown_connections 2024-02-21T23:08:23.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.382+0000 7fcc0b200640 1 --2- 172.21.15.28:0/1579894549 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcc0c0c85c0 0x7fcc0c0c70d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:23.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.382+0000 7fcc0b200640 1 --2- 172.21.15.28:0/1579894549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc0c1524c0 0x7fcc0c155a10 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:23.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.382+0000 7fcc0b200640 1 -- 172.21.15.28:0/1579894549 >> 172.21.15.28:0/1579894549 conn(0x7fcc0c0ac2b0 msgr2=0x7fcc0c0ac6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:23.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.382+0000 7fcc0b200640 1 -- 172.21.15.28:0/1579894549 shutdown_connections 2024-02-21T23:08:23.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.382+0000 7fcc0b200640 1 -- 172.21.15.28:0/1579894549 wait complete. 2024-02-21T23:08:23.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.384+0000 7fcc0b200640 1 Processor -- start 2024-02-21T23:08:23.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.384+0000 7fcc0b200640 1 -- start start 2024-02-21T23:08:23.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.385+0000 7fcc0b200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcc0c0c5040 0x7fcc0c153f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:23.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.385+0000 7fcc0b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc0c0c85c0 0x7fcc0c0be1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:23.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.385+0000 7fcc0b200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc0c1524c0 0x7fcc0c0be730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:23.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.385+0000 7fcc0b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0c0c8a70 con 0x7fcc0c0c85c0 2024-02-21T23:08:23.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.385+0000 7fcc0b200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0c1528a0 con 0x7fcc0c1524c0 2024-02-21T23:08:23.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.385+0000 7fcc0b200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0c0c7610 con 0x7fcc0c0c5040 2024-02-21T23:08:23.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.385+0000 7fcc01400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc0c0c85c0 0x7fcc0c0be1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:23.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.385+0000 7fcc01400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc0c0c85c0 0x7fcc0c0be1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57774/0 (socket says 172.21.15.28:57774) 2024-02-21T23:08:23.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.385+0000 7fcc01400640 1 -- 172.21.15.28:0/2572019222 learned_addr learned my addr 172.21.15.28:0/2572019222 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:23.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.385+0000 7fcc0a800640 1 --2- 172.21.15.28:0/2572019222 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc0c1524c0 0x7fcc0c0be730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:23.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.385+0000 7fcc09e00640 1 --2- 172.21.15.28:0/2572019222 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcc0c0c5040 0x7fcc0c153f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:23.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.386+0000 7fcc01400640 1 -- 172.21.15.28:0/2572019222 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcc0c0c5040 msgr2=0x7fcc0c153f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:23.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.386+0000 7fcc01400640 1 --2- 172.21.15.28:0/2572019222 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcc0c0c5040 0x7fcc0c153f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:23.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.386+0000 7fcc01400640 1 -- 172.21.15.28:0/2572019222 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc0c1524c0 msgr2=0x7fcc0c0be730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:23.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.386+0000 7fcc01400640 1 --2- 172.21.15.28:0/2572019222 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc0c1524c0 0x7fcc0c0be730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:23.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.386+0000 7fcc01400640 1 -- 172.21.15.28:0/2572019222 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc0c1eab80 con 0x7fcc0c0c85c0 2024-02-21T23:08:23.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.386+0000 7fcc01400640 1 --2- 172.21.15.28:0/2572019222 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc0c0c85c0 0x7fcc0c0be1f0 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7fcbf400c7c0 tx=0x7fcbf400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:23.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.386+0000 7fcc02a00640 1 -- 172.21.15.28:0/2572019222 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf4016020 con 0x7fcc0c0c85c0 2024-02-21T23:08:23.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.386+0000 7fcc02a00640 1 -- 172.21.15.28:0/2572019222 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbf400aa50 con 0x7fcc0c0c85c0 2024-02-21T23:08:23.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.386+0000 7fcc0b200640 1 -- 172.21.15.28:0/2572019222 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc0c1eee70 con 0x7fcc0c0c85c0 2024-02-21T23:08:23.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.386+0000 7fcc02a00640 1 -- 172.21.15.28:0/2572019222 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf4012420 con 0x7fcc0c0c85c0 2024-02-21T23:08:23.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.387+0000 7fcc0b200640 1 -- 172.21.15.28:0/2572019222 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc0c1eb120 con 0x7fcc0c0c85c0 2024-02-21T23:08:23.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.389+0000 7fcc02a00640 1 -- 172.21.15.28:0/2572019222 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcbf4019020 con 0x7fcc0c0c85c0 2024-02-21T23:08:23.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.390+0000 7fcc02a00640 1 --2- 172.21.15.28:0/2572019222 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcbe806b4d0 0x7fcbe806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:23.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.390+0000 7fcc02a00640 1 -- 172.21.15.28:0/2572019222 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fcbf408a0b0 con 0x7fcc0c0c85c0 2024-02-21T23:08:23.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.391+0000 7fcc09e00640 1 --2- 172.21.15.28:0/2572019222 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcbe806b4d0 0x7fcbe806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:23.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.391+0000 7fcc13ba7740 1 Processor -- start 2024-02-21T23:08:23.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.391+0000 7fcc13ba7740 1 -- start start 2024-02-21T23:08:23.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.392+0000 7fcc13ba7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56052085a020 0x56052085a400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:23.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.392+0000 7fcc13ba7740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x56052085b330 0x560520896170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:23.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.392+0000 7fcc09e00640 1 --2- 172.21.15.28:0/2572019222 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcbe806b4d0 0x7fcbe806d990 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc035000 tx=0x7fcbfc008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:23.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.392+0000 7fcc13ba7740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5605208966b0 0x560520894720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:23.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.392+0000 7fcc13ba7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605207be470 con 0x56052085a020 2024-02-21T23:08:23.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.392+0000 7fcc13ba7740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x560520767af0 con 0x5605208966b0 2024-02-21T23:08:23.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.392+0000 7fcc13ba7740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x5605207ba510 con 0x56052085b330 2024-02-21T23:08:23.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.392+0000 7fcc09e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56052085a020 0x56052085a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:23.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.392+0000 7fcc09e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56052085a020 0x56052085a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57776/0 (socket says 172.21.15.28:57776) 2024-02-21T23:08:23.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.392+0000 7fcc09e00640 1 -- 172.21.15.28:0/1392041570 learned_addr learned my addr 172.21.15.28:0/1392041570 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:23.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.393+0000 7fcc01400640 1 --2- 172.21.15.28:0/1392041570 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x56052085b330 0x560520896170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:23.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.393+0000 7fcc09e00640 1 -- 172.21.15.28:0/1392041570 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x56052085b330 msgr2=0x560520896170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:23.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.393+0000 7fcc0a800640 1 --2- 172.21.15.28:0/1392041570 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5605208966b0 0x560520894720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:23.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.393+0000 7fcc09e00640 1 --2- 172.21.15.28:0/1392041570 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x56052085b330 0x560520896170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:23.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.393+0000 7fcc09e00640 1 -- 172.21.15.28:0/1392041570 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5605208966b0 msgr2=0x560520894720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:23.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.393+0000 7fcc09e00640 1 --2- 172.21.15.28:0/1392041570 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5605208966b0 0x560520894720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:23.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.393+0000 7fcc09e00640 1 -- 172.21.15.28:0/1392041570 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56052076ed90 con 0x56052085a020 2024-02-21T23:08:23.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.393+0000 7fcc09e00640 1 --2- 172.21.15.28:0/1392041570 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56052085a020 0x56052085a400 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc009960 tx=0x7fcbfc030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9928c35ea1c47f1c server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:23.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.393+0000 7fcbfa600640 1 -- 172.21.15.28:0/1392041570 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbfc04e070 con 0x56052085a020 2024-02-21T23:08:23.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.393+0000 7fcbfa600640 1 -- 172.21.15.28:0/1392041570 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbfc03d7c0 con 0x56052085a020 2024-02-21T23:08:23.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.393+0000 7fcbfa600640 1 -- 172.21.15.28:0/1392041570 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbfc038040 con 0x56052085a020 2024-02-21T23:08:23.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.394+0000 7fcc13ba7740 1 -- 172.21.15.28:0/1392041570 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56052085a020 msgr2=0x56052085a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:23.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.394+0000 7fcc13ba7740 1 --2- 172.21.15.28:0/1392041570 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56052085a020 0x56052085a400 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc009960 tx=0x7fcbfc030000 comp rx=0 tx=0).stop 2024-02-21T23:08:23.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.394+0000 7fcc13ba7740 1 -- 172.21.15.28:0/1392041570 shutdown_connections 2024-02-21T23:08:23.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.394+0000 7fcc13ba7740 1 --2- 172.21.15.28:0/1392041570 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x56052085b330 0x560520896170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:23.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.394+0000 7fcc13ba7740 1 --2- 172.21.15.28:0/1392041570 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5605208966b0 0x560520894720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:23.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.394+0000 7fcc13ba7740 1 --2- 172.21.15.28:0/1392041570 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56052085a020 0x56052085a400 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:23.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.394+0000 7fcc13ba7740 1 -- 172.21.15.28:0/1392041570 >> 172.21.15.28:0/1392041570 conn(0x560520856970 msgr2=0x56052087b380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:23.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.394+0000 7fcc13ba7740 1 -- 172.21.15.28:0/1392041570 shutdown_connections 2024-02-21T23:08:23.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.394+0000 7fcc13ba7740 1 -- 172.21.15.28:0/1392041570 wait complete. 2024-02-21T23:08:23.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.395+0000 7fcc13ba7740 1 -- start start 2024-02-21T23:08:23.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.395+0000 7fcc13ba7740 1 Processor -- start 2024-02-21T23:08:23.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.396+0000 7fcc13ba7740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x56052085b330 0x560520877e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:23.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.396+0000 7fcc13ba7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605208966b0 0x560520858680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:23.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.396+0000 7fcc13ba7740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5605208a6a80 0x5605208a9570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:23.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.396+0000 7fcc13ba7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x56052075d040 con 0x5605208966b0 2024-02-21T23:08:23.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.396+0000 7fcc13ba7740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x56052087ba90 con 0x5605208a6a80 2024-02-21T23:08:23.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.396+0000 7fcc13ba7740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x560520858bc0 con 0x56052085b330 2024-02-21T23:08:23.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.396+0000 7fcc13ba7740 10 client.0 _ll_register_callbacks cb 0x56052077a920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:08:23.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.396+0000 7fcc01400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605208966b0 0x560520858680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:23.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.397+0000 7fcc01400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605208966b0 0x560520858680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57786/0 (socket says 172.21.15.28:57786) 2024-02-21T23:08:23.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.397+0000 7fcc01400640 1 -- 172.21.15.28:0/2216329049 learned_addr learned my addr 172.21.15.28:0/2216329049 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:23.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.397+0000 7fcc09e00640 1 --2- 172.21.15.28:0/2216329049 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x56052085b330 0x560520877e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:23.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.397+0000 7fcc0a800640 1 --2- 172.21.15.28:0/2216329049 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5605208a6a80 0x5605208a9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:23.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.397+0000 7fcc01400640 1 -- 172.21.15.28:0/2216329049 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x56052085b330 msgr2=0x560520877e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:23.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.397+0000 7fcc01400640 1 --2- 172.21.15.28:0/2216329049 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x56052085b330 0x560520877e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:23.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.397+0000 7fcc01400640 1 -- 172.21.15.28:0/2216329049 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5605208a6a80 msgr2=0x5605208a9570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:23.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.397+0000 7fcc01400640 1 --2- 172.21.15.28:0/2216329049 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5605208a6a80 0x5605208a9570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:23.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.397+0000 7fcc01400640 1 -- 172.21.15.28:0/2216329049 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56052077caa0 con 0x5605208966b0 2024-02-21T23:08:23.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.397+0000 7fcc01400640 1 --2- 172.21.15.28:0/2216329049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605208966b0 0x560520858680 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7fcbf40883e0 tx=0x7fcbf4058e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3953fb148c009cd server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:23.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.397+0000 7fcbeea00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:08:23.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.397+0000 7fcbeea00640 1 -- 172.21.15.28:0/2216329049 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf4070030 con 0x5605208966b0 2024-02-21T23:08:23.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.397+0000 7fcbeea00640 1 -- 172.21.15.28:0/2216329049 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(122..122 src has 1..122) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fcbf4060910 con 0x5605208966b0 2024-02-21T23:08:23.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.398+0000 7fcc13ba7740 1 -- 172.21.15.28:0/2216329049 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcbfc03d7c0 con 0x5605208966b0 2024-02-21T23:08:23.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.398+0000 7fcbeea00640 1 -- 172.21.15.28:0/2216329049 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=122) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcbf4060da0 con 0x5605208966b0 2024-02-21T23:08:23.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.398+0000 7fcc13ba7740 10 client.8085 fetch_fsmap learned FSMap version 122 2024-02-21T23:08:23.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.398+0000 7fcc13ba7740 1 -- 172.21.15.28:0/2216329049 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=122,monmap=2+}) v3 -- 0x5605204d32d0 con 0x5605208966b0 2024-02-21T23:08:23.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.398+0000 7fcbeea00640 1 -- 172.21.15.28:0/2216329049 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbfc03d7c0 con 0x5605208966b0 2024-02-21T23:08:23.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.398+0000 7fcbeea00640 1 -- 172.21.15.28:0/2216329049 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 122) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fcbf4006b50 con 0x5605208966b0 2024-02-21T23:08:23.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.398+0000 7fcc13ba7740 10 client.8085 fetch_fsmap finished waiting for FSMap version 122 2024-02-21T23:08:23.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.398+0000 7fcc13ba7740 10 client.8085 resolve_mds: resolved name 'd' to daemon mds.d 2024-02-21T23:08:23.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.398+0000 7fcc13ba7740 20 client.8085 populate_metadata read hostname 'smithi028' 2024-02-21T23:08:23.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.398+0000 7fcc13ba7740 1 --2- 172.21.15.28:0/2216329049 >> [v2:172.21.15.81:6835/1038096496,v1:172.21.15.81:6837/1038096496] conn(0x560520876220 0x560520876600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:23.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.398+0000 7fcc13ba7740 4 client.8085 mds_command: new command op to 7914 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-21T23:08:23.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.398+0000 7fcc13ba7740 1 -- 172.21.15.28:0/2216329049 --> [v2:172.21.15.81:6835/1038096496,v1:172.21.15.81:6837/1038096496] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5605207cb9b0 con 0x560520876220 2024-02-21T23:08:23.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.399+0000 7fcc09e00640 1 --2- 172.21.15.28:0/2216329049 >> [v2:172.21.15.81:6835/1038096496,v1:172.21.15.81:6837/1038096496] conn(0x560520876220 0x560520876600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:23.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.400+0000 7fcc09e00640 1 --2- 172.21.15.28:0/2216329049 >> [v2:172.21.15.81:6835/1038096496,v1:172.21.15.81:6837/1038096496] conn(0x560520876220 0x560520876600 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd2468004dc2ec7 server_cookie=469347c5c44e1e94 in_seq=0 out_seq=0 2024-02-21T23:08:23.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.400+0000 7fcbeea00640 10 client.8085 ms_handle_connect on v2:172.21.15.81:6835/1038096496 2024-02-21T23:08:23.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.402+0000 7fcbeea00640 1 -- 172.21.15.28:0/2216329049 <== mds.0 v2:172.21.15.81:6835/1038096496 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5605207cb9b0 con 0x560520876220 2024-02-21T23:08:23.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.402+0000 7fcbeea00640 10 client.8085 handle_command_reply: tid=0 2024-02-21T23:08:23.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.402+0000 7fcc13ba7740 1 client.8085 shutdown 2024-02-21T23:08:23.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.402+0000 7fcc13ba7740 1 -- 172.21.15.28:0/2216329049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605208966b0 msgr2=0x560520858680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:23.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.402+0000 7fcc13ba7740 1 --2- 172.21.15.28:0/2216329049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605208966b0 0x560520858680 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7fcbf40883e0 tx=0x7fcbf4058e50 comp rx=0 tx=0).stop 2024-02-21T23:08:23.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.402+0000 7fcc13ba7740 1 -- 172.21.15.28:0/2216329049 shutdown_connections 2024-02-21T23:08:23.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.402+0000 7fcc13ba7740 1 --2- 172.21.15.28:0/2216329049 >> [v2:172.21.15.81:6835/1038096496,v1:172.21.15.81:6837/1038096496] conn(0x560520876220 0x560520876600 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:23.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.402+0000 7fcc13ba7740 1 --2- 172.21.15.28:0/2216329049 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5605208a6a80 0x5605208a9570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:23.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.402+0000 7fcc13ba7740 1 --2- 172.21.15.28:0/2216329049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605208966b0 0x560520858680 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:23.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.402+0000 7fcc13ba7740 1 --2- 172.21.15.28:0/2216329049 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x56052085b330 0x560520877e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:23.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.402+0000 7fcbeea00640 0 client.8085 ms_handle_reset on v2:172.21.15.81:6835/1038096496 2024-02-21T23:08:23.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.402+0000 7fcc13ba7740 1 -- 172.21.15.28:0/2216329049 >> 172.21.15.28:0/2216329049 conn(0x56052085a020 msgr2=0x560520894ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:23.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.402+0000 7fcc13ba7740 1 -- 172.21.15.28:0/2216329049 shutdown_connections 2024-02-21T23:08:23.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.402+0000 7fcc13ba7740 1 -- 172.21.15.28:0/2216329049 wait complete. 2024-02-21T23:08:23.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.403+0000 7fcc13ba7740 20 client.8085 trim_cache size 0 max 16384 2024-02-21T23:08:23.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.421+0000 7fcc13ba7740 1 Processor -- start 2024-02-21T23:08:23.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.421+0000 7fcc13ba7740 1 -- start start 2024-02-21T23:08:23.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.422+0000 7fcc13ba7740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x56052085b330 0x560520876ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:23.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.422+0000 7fcc13ba7740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5605208966b0 0x56052087a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:23.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.422+0000 7fcc13ba7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605208a6a80 0x56052087af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:23.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.422+0000 7fcc13ba7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0c1f6c50 con 0x5605208a6a80 2024-02-21T23:08:23.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.422+0000 7fcc13ba7740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0c0c0bd0 con 0x5605208966b0 2024-02-21T23:08:23.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.422+0000 7fcc13ba7740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0c1f6680 con 0x56052085b330 2024-02-21T23:08:23.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.422+0000 7fcc0a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605208a6a80 0x56052087af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:23.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.422+0000 7fcc0a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605208a6a80 0x56052087af20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57788/0 (socket says 172.21.15.28:57788) 2024-02-21T23:08:23.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.422+0000 7fcc0a800640 1 -- 172.21.15.28:0/1248508506 learned_addr learned my addr 172.21.15.28:0/1248508506 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:23.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.422+0000 7fcc09e00640 1 --2- 172.21.15.28:0/1248508506 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x56052085b330 0x560520876ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:23.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.422+0000 7fcc01400640 1 --2- 172.21.15.28:0/1248508506 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5605208966b0 0x56052087a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:23.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.423+0000 7fcc0a800640 1 -- 172.21.15.28:0/1248508506 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x56052085b330 msgr2=0x560520876ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:23.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.423+0000 7fcc0a800640 1 --2- 172.21.15.28:0/1248508506 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x56052085b330 0x560520876ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:23.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.423+0000 7fcc0a800640 1 -- 172.21.15.28:0/1248508506 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5605208966b0 msgr2=0x56052087a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:23.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.423+0000 7fcc0a800640 1 --2- 172.21.15.28:0/1248508506 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5605208966b0 0x56052087a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:23.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.423+0000 7fcc0a800640 1 -- 172.21.15.28:0/1248508506 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560520856710 con 0x5605208a6a80 2024-02-21T23:08:23.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.423+0000 7fcc09e00640 1 --2- 172.21.15.28:0/1248508506 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x56052085b330 0x560520876ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:08:23.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.423+0000 7fcc0a800640 1 --2- 172.21.15.28:0/1248508506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605208a6a80 0x56052087af20 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7fcc040107c0 tx=0x7fcc04010c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbb060ea8296d0ff server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:23.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.423+0000 7fcbf9c00640 1 -- 172.21.15.28:0/1248508506 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc04018070 con 0x5605208a6a80 2024-02-21T23:08:23.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.423+0000 7fcbf9c00640 1 -- 172.21.15.28:0/1248508506 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc0400ea40 con 0x5605208a6a80 2024-02-21T23:08:23.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.423+0000 7fcbf9c00640 1 -- 172.21.15.28:0/1248508506 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc0400ee10 con 0x5605208a6a80 2024-02-21T23:08:23.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.424+0000 7fcc13ba7740 1 -- 172.21.15.28:0/1248508506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605208a6a80 msgr2=0x56052087af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:23.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.424+0000 7fcc13ba7740 1 --2- 172.21.15.28:0/1248508506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605208a6a80 0x56052087af20 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7fcc040107c0 tx=0x7fcc04010c80 comp rx=0 tx=0).stop 2024-02-21T23:08:23.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.424+0000 7fcc13ba7740 1 -- 172.21.15.28:0/1248508506 shutdown_connections 2024-02-21T23:08:23.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.424+0000 7fcc13ba7740 1 --2- 172.21.15.28:0/1248508506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605208a6a80 0x56052087af20 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:23.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.424+0000 7fcc13ba7740 1 --2- 172.21.15.28:0/1248508506 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5605208966b0 0x56052087a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:23.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.424+0000 7fcc13ba7740 1 --2- 172.21.15.28:0/1248508506 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x56052085b330 0x560520876ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:23.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.424+0000 7fcc13ba7740 1 -- 172.21.15.28:0/1248508506 >> 172.21.15.28:0/1248508506 conn(0x56052085a020 msgr2=0x560520877c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:23.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.424+0000 7fcc13ba7740 1 -- 172.21.15.28:0/1248508506 shutdown_connections 2024-02-21T23:08:23.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.424+0000 7fcc13ba7740 1 -- 172.21.15.28:0/1248508506 wait complete. 2024-02-21T23:08:23.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.425+0000 7fcc13ba7740 1 -- start start 2024-02-21T23:08:23.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.425+0000 7fcc13ba7740 1 Processor -- start 2024-02-21T23:08:23.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.426+0000 7fcc13ba7740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56052085b330 0x560520877080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:23.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.426+0000 7fcc13ba7740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5605208966b0 0x5605208afe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:23.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.426+0000 7fcc13ba7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605208a6a80 0x560520902790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:23.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.426+0000 7fcc13ba7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x560520891d40 con 0x5605208a6a80 2024-02-21T23:08:23.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.426+0000 7fcc13ba7740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605208a6f00 con 0x56052085b330 2024-02-21T23:08:23.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.426+0000 7fcc13ba7740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x5605208a9f80 con 0x5605208966b0 2024-02-21T23:08:23.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.426+0000 7fcc13ba7740 10 client.0 _ll_register_callbacks cb 0x7fcbf4058240 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:08:23.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.426+0000 7fcc0a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605208a6a80 0x560520902790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:23.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.426+0000 7fcc0a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605208a6a80 0x560520902790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57798/0 (socket says 172.21.15.28:57798) 2024-02-21T23:08:23.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.426+0000 7fcc0a800640 1 -- 172.21.15.28:0/2445815683 learned_addr learned my addr 172.21.15.28:0/2445815683 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:23.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.426+0000 7fcc09e00640 1 --2- 172.21.15.28:0/2445815683 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56052085b330 0x560520877080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:23.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.426+0000 7fcc01400640 1 --2- 172.21.15.28:0/2445815683 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5605208966b0 0x5605208afe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:23.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.427+0000 7fcc0a800640 1 -- 172.21.15.28:0/2445815683 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5605208966b0 msgr2=0x5605208afe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:23.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.427+0000 7fcc0a800640 1 --2- 172.21.15.28:0/2445815683 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5605208966b0 0x5605208afe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:23.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.427+0000 7fcc0a800640 1 -- 172.21.15.28:0/2445815683 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56052085b330 msgr2=0x560520877080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:23.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.427+0000 7fcc0a800640 1 --2- 172.21.15.28:0/2445815683 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56052085b330 0x560520877080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:23.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.427+0000 7fcc0a800640 1 -- 172.21.15.28:0/2445815683 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56052059d6b0 con 0x5605208a6a80 2024-02-21T23:08:23.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.427+0000 7fcc0a800640 1 --2- 172.21.15.28:0/2445815683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605208a6a80 0x560520902790 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7fcc04000c00 tx=0x7fcc04004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d90675940a0c27e2 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:23.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.427+0000 7fcbeea00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:08:23.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.427+0000 7fcbeea00640 1 -- 172.21.15.28:0/2445815683 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc04018030 con 0x5605208a6a80 2024-02-21T23:08:23.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.427+0000 7fcbeea00640 1 -- 172.21.15.28:0/2445815683 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(122..122 src has 1..122) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fcc04028c80 con 0x5605208a6a80 2024-02-21T23:08:23.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.427+0000 7fcc13ba7740 1 -- 172.21.15.28:0/2445815683 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcc0400ea40 con 0x5605208a6a80 2024-02-21T23:08:23.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.428+0000 7fcbeea00640 1 -- 172.21.15.28:0/2445815683 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=122) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcc0402f050 con 0x5605208a6a80 2024-02-21T23:08:23.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.428+0000 7fcc13ba7740 10 client.8091 fetch_fsmap learned FSMap version 122 2024-02-21T23:08:23.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.428+0000 7fcc13ba7740 1 -- 172.21.15.28:0/2445815683 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=122,monmap=2+}) v3 -- 0x560520906330 con 0x5605208a6a80 2024-02-21T23:08:23.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.428+0000 7fcbeea00640 1 -- 172.21.15.28:0/2445815683 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc0400ea40 con 0x5605208a6a80 2024-02-21T23:08:23.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.428+0000 7fcbeea00640 1 -- 172.21.15.28:0/2445815683 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 122) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fcc04032020 con 0x5605208a6a80 2024-02-21T23:08:23.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.428+0000 7fcc13ba7740 10 client.8091 fetch_fsmap finished waiting for FSMap version 122 2024-02-21T23:08:23.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.428+0000 7fcc13ba7740 10 client.8091 resolve_mds: resolved name 'd' to daemon mds.d 2024-02-21T23:08:23.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.428+0000 7fcc13ba7740 20 client.8091 populate_metadata read hostname 'smithi028' 2024-02-21T23:08:23.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.428+0000 7fcc13ba7740 1 --2- 172.21.15.28:0/2445815683 >> [v2:172.21.15.81:6835/1038096496,v1:172.21.15.81:6837/1038096496] conn(0x560520902df0 0x560520872550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:23.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.428+0000 7fcc13ba7740 4 client.8091 mds_command: new command op to 7914 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-21T23:08:23.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.428+0000 7fcc13ba7740 1 -- 172.21.15.28:0/2445815683 --> [v2:172.21.15.81:6835/1038096496,v1:172.21.15.81:6837/1038096496] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5605207c5d00 con 0x560520902df0 2024-02-21T23:08:23.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.429+0000 7fcc09e00640 1 --2- 172.21.15.28:0/2445815683 >> [v2:172.21.15.81:6835/1038096496,v1:172.21.15.81:6837/1038096496] conn(0x560520902df0 0x560520872550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:23.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.429+0000 7fcc09e00640 1 --2- 172.21.15.28:0/2445815683 >> [v2:172.21.15.81:6835/1038096496,v1:172.21.15.81:6837/1038096496] conn(0x560520902df0 0x560520872550 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=381275a5278a6f28 server_cookie=5c021dccaddde93c in_seq=0 out_seq=0 2024-02-21T23:08:23.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.429+0000 7fcbeea00640 10 client.8091 ms_handle_connect on v2:172.21.15.81:6835/1038096496 2024-02-21T23:08:23.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.430+0000 7fcbeea00640 1 -- 172.21.15.28:0/2445815683 <== mds.0 v2:172.21.15.81:6835/1038096496 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+334 (crc 0 0 0) 0x5605207c5d00 con 0x560520902df0 2024-02-21T23:08:23.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.430+0000 7fcbeea00640 10 client.8091 handle_command_reply: tid=0 2024-02-21T23:08:23.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.430+0000 7fcc13ba7740 1 client.8091 shutdown 2024-02-21T23:08:23.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.430+0000 7fcc13ba7740 1 -- 172.21.15.28:0/2445815683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605208a6a80 msgr2=0x560520902790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:23.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.430+0000 7fcc13ba7740 1 --2- 172.21.15.28:0/2445815683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605208a6a80 0x560520902790 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7fcc04000c00 tx=0x7fcc04004730 comp rx=0 tx=0).stop 2024-02-21T23:08:23.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.430+0000 7fcc13ba7740 1 -- 172.21.15.28:0/2445815683 shutdown_connections 2024-02-21T23:08:23.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.430+0000 7fcc13ba7740 1 --2- 172.21.15.28:0/2445815683 >> [v2:172.21.15.81:6835/1038096496,v1:172.21.15.81:6837/1038096496] conn(0x560520902df0 0x560520872550 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:23.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.430+0000 7fcc13ba7740 1 --2- 172.21.15.28:0/2445815683 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5605208966b0 0x5605208afe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:23.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.430+0000 7fcc13ba7740 1 --2- 172.21.15.28:0/2445815683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5605208a6a80 0x560520902790 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:23.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.430+0000 7fcc13ba7740 1 --2- 172.21.15.28:0/2445815683 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56052085b330 0x560520877080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:23.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.430+0000 7fcbeea00640 0 client.8091 ms_handle_reset on v2:172.21.15.81:6835/1038096496 2024-02-21T23:08:23.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.430+0000 7fcc13ba7740 1 -- 172.21.15.28:0/2445815683 >> 172.21.15.28:0/2445815683 conn(0x56052085a020 msgr2=0x5605208583a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:23.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.430+0000 7fcc13ba7740 1 -- 172.21.15.28:0/2445815683 shutdown_connections 2024-02-21T23:08:23.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.430+0000 7fcc13ba7740 1 -- 172.21.15.28:0/2445815683 wait complete. 2024-02-21T23:08:23.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.431+0000 7fcc13ba7740 20 client.8091 trim_cache size 0 max 16384 2024-02-21T23:08:23.423 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-21T23:08:23.423 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac", 2024-02-21T23:08:23.423 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-02-21T23:08:23.423 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 7914, 2024-02-21T23:08:23.423 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-02-21T23:08:23.424 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-02-21T23:08:23.424 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-02-21T23:08:23.424 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 9.0120756560000004, 2024-02-21T23:08:23.424 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 122, 2024-02-21T23:08:23.424 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 122, 2024-02-21T23:08:23.424 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 122, 2024-02-21T23:08:23.424 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 14.504898600000001 2024-02-21T23:08:23.424 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-21T23:08:23.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.431+0000 7fcc0b200640 1 -- 172.21.15.28:0/2572019222 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcbe806b4d0 msgr2=0x7fcbe806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:23.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.431+0000 7fcc0b200640 1 --2- 172.21.15.28:0/2572019222 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcbe806b4d0 0x7fcbe806d990 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc035000 tx=0x7fcbfc008030 comp rx=0 tx=0).stop 2024-02-21T23:08:23.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.431+0000 7fcc0b200640 1 -- 172.21.15.28:0/2572019222 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc0c0c85c0 msgr2=0x7fcc0c0be1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:23.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.431+0000 7fcc0b200640 1 --2- 172.21.15.28:0/2572019222 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc0c0c85c0 0x7fcc0c0be1f0 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7fcbf400c7c0 tx=0x7fcbf400cc80 comp rx=0 tx=0).stop 2024-02-21T23:08:23.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.431+0000 7fcc0b200640 1 -- 172.21.15.28:0/2572019222 shutdown_connections 2024-02-21T23:08:23.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.431+0000 7fcc0b200640 1 --2- 172.21.15.28:0/2572019222 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcbe806b4d0 0x7fcbe806d990 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:23.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.431+0000 7fcc0b200640 1 --2- 172.21.15.28:0/2572019222 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc0c1524c0 0x7fcc0c0be730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:23.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.431+0000 7fcc0b200640 1 --2- 172.21.15.28:0/2572019222 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc0c0c85c0 0x7fcc0c0be1f0 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:23.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.431+0000 7fcc0b200640 1 --2- 172.21.15.28:0/2572019222 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcc0c0c5040 0x7fcc0c153f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:23.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.431+0000 7fcc0b200640 1 -- 172.21.15.28:0/2572019222 >> 172.21.15.28:0/2572019222 conn(0x7fcc0c0ac2b0 msgr2=0x7fcc0c0c7cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:23.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.431+0000 7fcc0b200640 1 -- 172.21.15.28:0/2572019222 shutdown_connections 2024-02-21T23:08:23.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:23.431+0000 7fcc0b200640 1 -- 172.21.15.28:0/2572019222 wait complete. 2024-02-21T23:08:23.437 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 9, "epoch": 121, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 9, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-02-21T23:08:13.883141+0000", "damaged": [], "data_pools": [ 20 ], "enabled": true, "epoch": 121, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7914": { "addr": "172.21.15.81:6837/1038096496", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6835", "nonce": 1038096496, "type": "v2" }, { "addr": "172.21.15.81:6837", "nonce": 1038096496, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7914, "incarnation": 117, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 19, "modified": "2024-02-21T23:08:16.980266+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7914 } } } ], "standbys": [ { "addr": "172.21.15.28:6835/3556518453", "addrs": { "addrvec": [ { "addr": "172.21.15.28:6833", "nonce": 3556518453, "type": "v2" }, { "addr": "172.21.15.28:6835", "nonce": 3556518453, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 116, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5948, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.81:6836/3293264689", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6834", "nonce": 3293264689, "type": "v2" }, { "addr": "172.21.15.81:6836", "nonce": 3293264689, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 116, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5951, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.28:6834/1773560593", "addrs": { "addrvec": [ { "addr": "172.21.15.28:6832", "nonce": 1773560593, "type": "v2" }, { "addr": "172.21.15.28:6834", "nonce": 1773560593, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 115, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7905, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-02-21T23:08:23.438 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': 9, 'name': 'd'}] 2024-02-21T23:08:23.438 INFO:tasks.cephfs.test_failover:current = [{'gid': 7905, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.28:6834/1773560593', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 1773560593}, {'type': 'v1', 'addr': '172.21.15.28:6834', 'nonce': 1773560593}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 115}, {'gid': 5951, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.81:6836/3293264689', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6834', 'nonce': 3293264689}, {'type': 'v1', 'addr': '172.21.15.81:6836', 'nonce': 3293264689}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 116}, {'gid': 5948, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.28:6835/3556518453', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6833', 'nonce': 3556518453}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3556518453}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 116}, {'gid': 7914, 'name': 'd', 'rank': 0, 'incarnation': 117, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.81:6837/1038096496', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6835', 'nonce': 1038096496}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 1038096496}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}] 2024-02-21T23:08:23.439 DEBUG:tasks.cephfs.test_failover:9 != -1 2024-02-21T23:08:23.439 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-02-21T23:08:28.440 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:08:28.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.682+0000 7fb937600640 1 Processor -- start 2024-02-21T23:08:28.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.682+0000 7fb937600640 1 -- start start 2024-02-21T23:08:28.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.684+0000 7fb937600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb9300c7180 0x7fb9300c75e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:28.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.684+0000 7fb937600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fb9301587f0 0x7fb930158c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:28.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.684+0000 7fb937600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb930152c80 con 0x7fb9301528a0 2024-02-21T23:08:28.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.684+0000 7fb937600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb930155300 con 0x7fb9300c7180 2024-02-21T23:08:28.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.684+0000 7fb937600640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9301555e0 con 0x7fb9301587f0 2024-02-21T23:08:28.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.684+0000 7fb92fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb9300c7180 0x7fb9300c75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:28.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.684+0000 7fb934c00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb9301528a0 0x7fb9300c6650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:56848/0 (socket says 172.21.15.28:56848) 2024-02-21T23:08:28.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.684+0000 7fb934c00640 1 -- 172.21.15.28:0/3953695167 learned_addr learned my addr 172.21.15.28:0/3953695167 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:08:28.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.685+0000 7fb935600640 1 --2- 172.21.15.28:0/3953695167 >> v2:172.21.15.81:3300/0 conn(0x7fb9301587f0 0x7fb930158c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:28.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.685+0000 7fb92fe00640 1 -- 172.21.15.28:0/3953695167 >> v2:172.21.15.81:3300/0 conn(0x7fb9301587f0 msgr2=0x7fb930158c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:28.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.685+0000 7fb92fe00640 1 --2- 172.21.15.28:0/3953695167 >> v2:172.21.15.81:3300/0 conn(0x7fb9301587f0 0x7fb930158c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:28.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.685+0000 7fb92fe00640 1 -- 172.21.15.28:0/3953695167 >> v1:172.21.15.28:6789/0 conn(0x7fb9301528a0 legacy=0x7fb9300c6650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:28.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.685+0000 7fb92fe00640 1 -- 172.21.15.28:0/3953695167 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9301558c0 con 0x7fb9300c7180 2024-02-21T23:08:28.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.686+0000 7fb92fe00640 1 --2- 172.21.15.28:0/3953695167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9300c7180 0x7fb9300c75e0 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7fb9300c96d0 tx=0x7fb91c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4f8b5b6b4e444c9 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:28.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.686+0000 7fb92f400640 1 -- 172.21.15.28:0/3953695167 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb91c0315a0 con 0x7fb9300c7180 2024-02-21T23:08:28.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.687+0000 7fb92f400640 1 -- 172.21.15.28:0/3953695167 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb91c037070 con 0x7fb9300c7180 2024-02-21T23:08:28.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.687+0000 7fb92f400640 1 -- 172.21.15.28:0/3953695167 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb91c031960 con 0x7fb9300c7180 2024-02-21T23:08:28.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.687+0000 7fb937600640 1 -- 172.21.15.28:0/3953695167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9300c7180 msgr2=0x7fb9300c75e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:28.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.687+0000 7fb937600640 1 --2- 172.21.15.28:0/3953695167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9300c7180 0x7fb9300c75e0 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7fb9300c96d0 tx=0x7fb91c02b840 comp rx=0 tx=0).stop 2024-02-21T23:08:28.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.687+0000 7fb937600640 1 -- 172.21.15.28:0/3953695167 shutdown_connections 2024-02-21T23:08:28.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.687+0000 7fb937600640 1 --2- 172.21.15.28:0/3953695167 >> v2:172.21.15.81:3300/0 conn(0x7fb9301587f0 0x7fb930158c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:28.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.687+0000 7fb937600640 1 --2- 172.21.15.28:0/3953695167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9300c7180 0x7fb9300c75e0 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:28.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.687+0000 7fb937600640 1 -- 172.21.15.28:0/3953695167 >> 172.21.15.28:0/3953695167 conn(0x7fb9300aa4c0 msgr2=0x7fb9300aa8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:28.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.687+0000 7fb937600640 1 -- 172.21.15.28:0/3953695167 shutdown_connections 2024-02-21T23:08:28.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.687+0000 7fb937600640 1 -- 172.21.15.28:0/3953695167 wait complete. 2024-02-21T23:08:28.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.689+0000 7fb937600640 1 Processor -- start 2024-02-21T23:08:28.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.689+0000 7fb937600640 1 -- start start 2024-02-21T23:08:28.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.690+0000 7fb937600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb9300c7180 0x7fb93006d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:28.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.690+0000 7fb937600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9301528a0 0x7fb930153ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:28.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.690+0000 7fb937600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9301587f0 0x7fb9301ee140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:28.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.690+0000 7fb937600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9300a8170 con 0x7fb9301528a0 2024-02-21T23:08:28.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.690+0000 7fb937600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb930155e80 con 0x7fb9301587f0 2024-02-21T23:08:28.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.690+0000 7fb937600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9300c6d60 con 0x7fb9300c7180 2024-02-21T23:08:28.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.691+0000 7fb92fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9301528a0 0x7fb930153ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:28.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.691+0000 7fb92fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9301528a0 0x7fb930153ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41332/0 (socket says 172.21.15.28:41332) 2024-02-21T23:08:28.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.691+0000 7fb92fe00640 1 -- 172.21.15.28:0/3220279333 learned_addr learned my addr 172.21.15.28:0/3220279333 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:28.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.691+0000 7fb935600640 1 --2- 172.21.15.28:0/3220279333 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9301587f0 0x7fb9301ee140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:28.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.691+0000 7fb934c00640 1 --2- 172.21.15.28:0/3220279333 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb9300c7180 0x7fb93006d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:28.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.691+0000 7fb92fe00640 1 -- 172.21.15.28:0/3220279333 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb9300c7180 msgr2=0x7fb93006d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:28.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.691+0000 7fb92fe00640 1 --2- 172.21.15.28:0/3220279333 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb9300c7180 0x7fb93006d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:28.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.691+0000 7fb92fe00640 1 -- 172.21.15.28:0/3220279333 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9301587f0 msgr2=0x7fb9301ee140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:28.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.691+0000 7fb92fe00640 1 --2- 172.21.15.28:0/3220279333 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9301587f0 0x7fb9301ee140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:28.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.691+0000 7fb92fe00640 1 -- 172.21.15.28:0/3220279333 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9300a8880 con 0x7fb9301528a0 2024-02-21T23:08:28.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.691+0000 7fb934c00640 1 --2- 172.21.15.28:0/3220279333 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb9300c7180 0x7fb93006d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:08:28.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.692+0000 7fb92fe00640 1 --2- 172.21.15.28:0/3220279333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9301528a0 0x7fb930153ab0 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7fb91c009890 tx=0x7fb91c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:28.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.692+0000 7fb92cc00640 1 -- 172.21.15.28:0/3220279333 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb91c032630 con 0x7fb9301528a0 2024-02-21T23:08:28.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.692+0000 7fb92cc00640 1 -- 172.21.15.28:0/3220279333 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb91c0327d0 con 0x7fb9301528a0 2024-02-21T23:08:28.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.692+0000 7fb92cc00640 1 -- 172.21.15.28:0/3220279333 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb91c032ba0 con 0x7fb9301528a0 2024-02-21T23:08:28.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.692+0000 7fb937600640 1 -- 172.21.15.28:0/3220279333 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9300a8390 con 0x7fb9301528a0 2024-02-21T23:08:28.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.692+0000 7fb937600640 1 -- 172.21.15.28:0/3220279333 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9301544e0 con 0x7fb9301528a0 2024-02-21T23:08:28.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.693+0000 7fb937600640 1 -- 172.21.15.28:0/3220279333 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb930067f30 con 0x7fb9301528a0 2024-02-21T23:08:28.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.694+0000 7fb92cc00640 1 -- 172.21.15.28:0/3220279333 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb91c03b3f0 con 0x7fb9301528a0 2024-02-21T23:08:28.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.695+0000 7fb92cc00640 1 --2- 172.21.15.28:0/3220279333 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb91006b4d0 0x7fb91006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:28.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.695+0000 7fb92cc00640 1 -- 172.21.15.28:0/3220279333 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb91c02f030 con 0x7fb9301528a0 2024-02-21T23:08:28.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.696+0000 7fb934c00640 1 --2- 172.21.15.28:0/3220279333 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb91006b4d0 0x7fb91006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:28.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.697+0000 7fb934c00640 1 --2- 172.21.15.28:0/3220279333 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb91006b4d0 0x7fb91006d990 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fb93014be70 tx=0x7fb924009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:28.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.700+0000 7fb92cc00640 1 -- 172.21.15.28:0/3220279333 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb91c040070 con 0x7fb9301528a0 2024-02-21T23:08:28.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.942+0000 7fb937600640 1 -- 172.21.15.28:0/3220279333 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb9300c34f0 con 0x7fb9301528a0 2024-02-21T23:08:28.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.943+0000 7fb92cc00640 1 -- 172.21.15.28:0/3220279333 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+4945 (secure 0 0 0) 0x7fb91c07af40 con 0x7fb9301528a0 2024-02-21T23:08:28.933 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:08:28.933 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":122,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5948,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3556518453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3556518453},{"type":"v1","addr":"172.21.15.28:6835","nonce":3556518453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116},{"gid":5951,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/3293264689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":3293264689},{"type":"v1","addr":"172.21.15.81:6836","nonce":3293264689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116},{"gid":7905,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/1773560593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1773560593},{"type":"v1","addr":"172.21.15.28:6834","nonce":1773560593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":115}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:08:13.883141+0000","modified":"2024-02-21T23:08:22.473107+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7914},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7914":{"gid":7914,"name":"d","rank":0,"incarnation":117,"state":"up:active","state_seq":5,"addr":"172.21.15.81:6837/1038096496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":1038096496},{"type":"v1","addr":"172.21.15.81:6837","nonce":1038096496}]},"join_fscid":9,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-21T23:08:28.933 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 122 2024-02-21T23:08:28.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.947+0000 7fb937600640 1 -- 172.21.15.28:0/3220279333 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb91006b4d0 msgr2=0x7fb91006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:28.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.947+0000 7fb937600640 1 --2- 172.21.15.28:0/3220279333 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb91006b4d0 0x7fb91006d990 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fb93014be70 tx=0x7fb924009040 comp rx=0 tx=0).stop 2024-02-21T23:08:28.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.947+0000 7fb937600640 1 -- 172.21.15.28:0/3220279333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9301528a0 msgr2=0x7fb930153ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:28.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.947+0000 7fb937600640 1 --2- 172.21.15.28:0/3220279333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9301528a0 0x7fb930153ab0 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7fb91c009890 tx=0x7fb91c0046c0 comp rx=0 tx=0).stop 2024-02-21T23:08:28.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.947+0000 7fb937600640 1 -- 172.21.15.28:0/3220279333 shutdown_connections 2024-02-21T23:08:28.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.947+0000 7fb937600640 1 --2- 172.21.15.28:0/3220279333 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb91006b4d0 0x7fb91006d990 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:28.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.947+0000 7fb937600640 1 --2- 172.21.15.28:0/3220279333 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9301587f0 0x7fb9301ee140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:28.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.947+0000 7fb937600640 1 --2- 172.21.15.28:0/3220279333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9301528a0 0x7fb930153ab0 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:28.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.947+0000 7fb937600640 1 --2- 172.21.15.28:0/3220279333 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb9300c7180 0x7fb93006d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:28.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.947+0000 7fb937600640 1 -- 172.21.15.28:0/3220279333 >> 172.21.15.28:0/3220279333 conn(0x7fb9300aa4c0 msgr2=0x7fb930153060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:28.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.947+0000 7fb937600640 1 -- 172.21.15.28:0/3220279333 shutdown_connections 2024-02-21T23:08:28.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:28.947+0000 7fb937600640 1 -- 172.21.15.28:0/3220279333 wait complete. 2024-02-21T23:08:29.363 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 122, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:08:13.883141+0000', 'modified': '2024-02-21T23:08:22.473107+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7914}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7914': {'gid': 7914, 'name': 'd', 'rank': 0, 'incarnation': 117, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.81:6837/1038096496', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6835', 'nonce': 1038096496}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 1038096496}]}, 'join_fscid': 9, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [20], 'metadata_pool': 19, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:08:29.363 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-21T23:08:29.363 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-02-21T23:08:29.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.604+0000 7f4259200640 1 Processor -- start 2024-02-21T23:08:29.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.605+0000 7f4259200640 1 -- start start 2024-02-21T23:08:29.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.606+0000 7f4259200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f42541527a0 0x7f4254158f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:29.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.606+0000 7f4259200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f425406d410 0x7f425406d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:29.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.606+0000 7f4259200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42540c2070 con 0x7f42540bf9a0 2024-02-21T23:08:29.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.606+0000 7f4259200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f425406e790 con 0x7f42541527a0 2024-02-21T23:08:29.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.606+0000 7f4259200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42540c29a0 con 0x7f425406d410 2024-02-21T23:08:29.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.607+0000 7f4253400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f42541527a0 0x7f4254158f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:29.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.607+0000 7f4253400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f42541527a0 0x7f4254158f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41334/0 (socket says 172.21.15.28:41334) 2024-02-21T23:08:29.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.607+0000 7f4253400640 1 -- 172.21.15.28:0/2387374730 learned_addr learned my addr 172.21.15.28:0/2387374730 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:29.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.607+0000 7f4253e00640 1 --2- 172.21.15.28:0/2387374730 >> v2:172.21.15.81:3300/0 conn(0x7f425406d410 0x7f425406d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:29.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.607+0000 7f4253400640 1 -- 172.21.15.28:0/2387374730 >> v2:172.21.15.81:3300/0 conn(0x7f425406d410 msgr2=0x7f425406d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:29.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.607+0000 7f4253400640 1 --2- 172.21.15.28:0/2387374730 >> v2:172.21.15.81:3300/0 conn(0x7f425406d410 0x7f425406d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:29.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.607+0000 7f4253400640 1 -- 172.21.15.28:0/2387374730 >> v1:172.21.15.81:6789/0 conn(0x7f42540bf9a0 legacy=0x7f42540bfdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:29.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.607+0000 7f4253400640 1 -- 172.21.15.28:0/2387374730 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42540c2350 con 0x7f42541527a0 2024-02-21T23:08:29.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.608+0000 7f4253400640 1 --2- 172.21.15.28:0/2387374730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42541527a0 0x7f4254158f70 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f425406cef0 tx=0x7f424402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51c42157ffc305e8 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:29.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.609+0000 7f4252000640 1 -- 172.21.15.28:0/2387374730 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4244034070 con 0x7f42541527a0 2024-02-21T23:08:29.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.609+0000 7f4252000640 1 -- 172.21.15.28:0/2387374730 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4244030860 con 0x7f42541527a0 2024-02-21T23:08:29.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.609+0000 7f4252000640 1 -- 172.21.15.28:0/2387374730 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4244030c30 con 0x7f42541527a0 2024-02-21T23:08:29.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.609+0000 7f4259200640 1 -- 172.21.15.28:0/2387374730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42541527a0 msgr2=0x7f4254158f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:29.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.609+0000 7f4259200640 1 --2- 172.21.15.28:0/2387374730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42541527a0 0x7f4254158f70 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f425406cef0 tx=0x7f424402b9a0 comp rx=0 tx=0).stop 2024-02-21T23:08:29.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.610+0000 7f4259200640 1 -- 172.21.15.28:0/2387374730 shutdown_connections 2024-02-21T23:08:29.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.610+0000 7f4259200640 1 --2- 172.21.15.28:0/2387374730 >> v2:172.21.15.81:3300/0 conn(0x7f425406d410 0x7f425406d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:29.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.610+0000 7f4259200640 1 --2- 172.21.15.28:0/2387374730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42541527a0 0x7f4254158f70 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:29.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.610+0000 7f4259200640 1 -- 172.21.15.28:0/2387374730 >> 172.21.15.28:0/2387374730 conn(0x7f42540a8320 msgr2=0x7f42540a8730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:29.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.610+0000 7f4259200640 1 -- 172.21.15.28:0/2387374730 shutdown_connections 2024-02-21T23:08:29.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.610+0000 7f4259200640 1 -- 172.21.15.28:0/2387374730 wait complete. 2024-02-21T23:08:29.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.612+0000 7f4259200640 1 Processor -- start 2024-02-21T23:08:29.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.612+0000 7f4259200640 1 -- start start 2024-02-21T23:08:29.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.613+0000 7f4259200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f425406d410 0x7f42541e1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:29.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.613+0000 7f4259200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42540bf9a0 0x7f425414d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:29.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.613+0000 7f4259200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f42541527a0 0x7f425414d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:29.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.613+0000 7f4259200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42540c1d20 con 0x7f42540bf9a0 2024-02-21T23:08:29.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.613+0000 7f4259200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42540c0500 con 0x7f42541527a0 2024-02-21T23:08:29.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.613+0000 7f4259200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f425406dd60 con 0x7f425406d410 2024-02-21T23:08:29.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.614+0000 7f4252a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42540bf9a0 0x7f425414d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:29.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.614+0000 7f4252a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42540bf9a0 0x7f425414d2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41342/0 (socket says 172.21.15.28:41342) 2024-02-21T23:08:29.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.614+0000 7f4252a00640 1 -- 172.21.15.28:0/1182661795 learned_addr learned my addr 172.21.15.28:0/1182661795 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:29.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.614+0000 7f4253e00640 1 --2- 172.21.15.28:0/1182661795 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f42541527a0 0x7f425414d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:29.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.614+0000 7f4253400640 1 --2- 172.21.15.28:0/1182661795 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f425406d410 0x7f42541e1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:29.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.614+0000 7f4252a00640 1 -- 172.21.15.28:0/1182661795 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f425406d410 msgr2=0x7f42541e1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:29.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.614+0000 7f4252a00640 1 --2- 172.21.15.28:0/1182661795 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f425406d410 0x7f42541e1ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:29.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.614+0000 7f4252a00640 1 -- 172.21.15.28:0/1182661795 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f42541527a0 msgr2=0x7f425414d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:29.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.614+0000 7f4252a00640 1 --2- 172.21.15.28:0/1182661795 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f42541527a0 0x7f425414d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:29.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.614+0000 7f4252a00640 1 -- 172.21.15.28:0/1182661795 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4254153a30 con 0x7f42540bf9a0 2024-02-21T23:08:29.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.614+0000 7f4253400640 1 --2- 172.21.15.28:0/1182661795 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f425406d410 0x7f42541e1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:08:29.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.615+0000 7f4252a00640 1 --2- 172.21.15.28:0/1182661795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42540bf9a0 0x7f425414d2b0 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f423c00c460 tx=0x7f423c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:29.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.615+0000 7f424b400640 1 -- 172.21.15.28:0/1182661795 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f423c016020 con 0x7f42540bf9a0 2024-02-21T23:08:29.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.615+0000 7f424b400640 1 -- 172.21.15.28:0/1182661795 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f423c012870 con 0x7f42540bf9a0 2024-02-21T23:08:29.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.615+0000 7f4259200640 1 -- 172.21.15.28:0/1182661795 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42541e0d30 con 0x7f42540bf9a0 2024-02-21T23:08:29.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.615+0000 7f424b400640 1 -- 172.21.15.28:0/1182661795 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f423c012c60 con 0x7f42540bf9a0 2024-02-21T23:08:29.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.615+0000 7f4259200640 1 -- 172.21.15.28:0/1182661795 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42541eea10 con 0x7f42540bf9a0 2024-02-21T23:08:29.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.617+0000 7f424b400640 1 -- 172.21.15.28:0/1182661795 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f423c019020 con 0x7f42540bf9a0 2024-02-21T23:08:29.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.618+0000 7f424b400640 1 --2- 172.21.15.28:0/1182661795 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f423006b4d0 0x7f423006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:29.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.618+0000 7f424b400640 1 -- 172.21.15.28:0/1182661795 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f423c08a0e0 con 0x7f42540bf9a0 2024-02-21T23:08:29.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.619+0000 7f425d7d7740 1 Processor -- start 2024-02-21T23:08:29.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.619+0000 7f4253400640 1 --2- 172.21.15.28:0/1182661795 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f423006b4d0 0x7f423006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:29.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.619+0000 7f425d7d7740 1 -- start start 2024-02-21T23:08:29.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.620+0000 7f425d7d7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5586accad020 0x5586accad400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:29.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.620+0000 7f425d7d7740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5586accae330 0x5586acce9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:29.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.620+0000 7f425d7d7740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5586acce96b0 0x5586acce7720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:29.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.620+0000 7f425d7d7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5586acc11470 con 0x5586accad020 2024-02-21T23:08:29.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.620+0000 7f425d7d7740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x5586acbbaaf0 con 0x5586acce96b0 2024-02-21T23:08:29.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.620+0000 7f425d7d7740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x5586acc0d510 con 0x5586accae330 2024-02-21T23:08:29.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.620+0000 7f4253400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5586accad020 0x5586accad400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:29.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.620+0000 7f4253400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5586accad020 0x5586accad400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41348/0 (socket says 172.21.15.28:41348) 2024-02-21T23:08:29.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.620+0000 7f4253400640 1 -- 172.21.15.28:0/794406082 learned_addr learned my addr 172.21.15.28:0/794406082 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:29.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.620+0000 7f4253400640 1 --2- 172.21.15.28:0/1182661795 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f423006b4d0 0x7f423006d990 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f4244036000 tx=0x7f424403b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:29.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.621+0000 7f4252a00640 1 --2- 172.21.15.28:0/794406082 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5586accae330 0x5586acce9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:29.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.621+0000 7f4253e00640 1 --2- 172.21.15.28:0/794406082 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5586acce96b0 0x5586acce7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:29.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.621+0000 7f4253400640 1 -- 172.21.15.28:0/794406082 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5586accae330 msgr2=0x5586acce9170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:29.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.621+0000 7f4253400640 1 --2- 172.21.15.28:0/794406082 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5586accae330 0x5586acce9170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:29.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.621+0000 7f4253400640 1 -- 172.21.15.28:0/794406082 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5586acce96b0 msgr2=0x5586acce7720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:29.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.621+0000 7f4253400640 1 --2- 172.21.15.28:0/794406082 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5586acce96b0 0x5586acce7720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:29.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.621+0000 7f4253400640 1 -- 172.21.15.28:0/794406082 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5586acbc1d90 con 0x5586accad020 2024-02-21T23:08:29.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.621+0000 7f4253400640 1 --2- 172.21.15.28:0/794406082 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5586accad020 0x5586accad400 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f424402bfb0 tx=0x7f4244008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28988d1d4c4f4e52 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:29.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.621+0000 7f4243e00640 1 -- 172.21.15.28:0/794406082 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4244034040 con 0x5586accad020 2024-02-21T23:08:29.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.621+0000 7f4243e00640 1 -- 172.21.15.28:0/794406082 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4244039b20 con 0x5586accad020 2024-02-21T23:08:29.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.621+0000 7f4243e00640 1 -- 172.21.15.28:0/794406082 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f424404e440 con 0x5586accad020 2024-02-21T23:08:29.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.622+0000 7f425d7d7740 1 -- 172.21.15.28:0/794406082 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5586accad020 msgr2=0x5586accad400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:29.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.622+0000 7f425d7d7740 1 --2- 172.21.15.28:0/794406082 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5586accad020 0x5586accad400 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f424402bfb0 tx=0x7f4244008030 comp rx=0 tx=0).stop 2024-02-21T23:08:29.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.622+0000 7f425d7d7740 1 -- 172.21.15.28:0/794406082 shutdown_connections 2024-02-21T23:08:29.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.622+0000 7f425d7d7740 1 --2- 172.21.15.28:0/794406082 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5586accae330 0x5586acce9170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:29.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.622+0000 7f425d7d7740 1 --2- 172.21.15.28:0/794406082 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5586acce96b0 0x5586acce7720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:29.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.622+0000 7f425d7d7740 1 --2- 172.21.15.28:0/794406082 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5586accad020 0x5586accad400 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:29.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.622+0000 7f425d7d7740 1 -- 172.21.15.28:0/794406082 >> 172.21.15.28:0/794406082 conn(0x5586acca9970 msgr2=0x5586accce380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:29.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.622+0000 7f425d7d7740 1 -- 172.21.15.28:0/794406082 shutdown_connections 2024-02-21T23:08:29.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.622+0000 7f425d7d7740 1 -- 172.21.15.28:0/794406082 wait complete. 2024-02-21T23:08:29.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.623+0000 7f425d7d7740 1 -- start start 2024-02-21T23:08:29.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.623+0000 7f425d7d7740 1 Processor -- start 2024-02-21T23:08:29.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.624+0000 7f425d7d7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5586accae330 0x5586accfc900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:29.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.624+0000 7f425d7d7740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5586acce96b0 0x5586acce0d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:29.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.624+0000 7f425d7d7740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5586accc91a0 0x5586accc7fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:29.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.624+0000 7f425d7d7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5586acbb0040 con 0x5586accae330 2024-02-21T23:08:29.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.624+0000 7f425d7d7740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x5586acccea90 con 0x5586accc91a0 2024-02-21T23:08:29.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.624+0000 7f425d7d7740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x5586acce1910 con 0x5586acce96b0 2024-02-21T23:08:29.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.624+0000 7f425d7d7740 10 client.0 _ll_register_callbacks cb 0x5586acbcd920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:08:29.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.624+0000 7f4253400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5586accae330 0x5586accfc900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:29.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.624+0000 7f4253400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5586accae330 0x5586accfc900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41352/0 (socket says 172.21.15.28:41352) 2024-02-21T23:08:29.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.624+0000 7f4253400640 1 -- 172.21.15.28:0/2071955527 learned_addr learned my addr 172.21.15.28:0/2071955527 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:29.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.625+0000 7f4253e00640 1 --2- 172.21.15.28:0/2071955527 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5586accc91a0 0x5586accc7fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:29.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.625+0000 7f4252a00640 1 --2- 172.21.15.28:0/2071955527 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5586acce96b0 0x5586acce0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:29.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.625+0000 7f4253400640 1 -- 172.21.15.28:0/2071955527 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5586acce96b0 msgr2=0x5586acce0d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:29.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.625+0000 7f4253400640 1 --2- 172.21.15.28:0/2071955527 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5586acce96b0 0x5586acce0d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:29.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.625+0000 7f4253400640 1 -- 172.21.15.28:0/2071955527 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5586accc91a0 msgr2=0x5586accc7fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:29.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.625+0000 7f4253400640 1 --2- 172.21.15.28:0/2071955527 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5586accc91a0 0x5586accc7fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:29.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.625+0000 7f4253400640 1 -- 172.21.15.28:0/2071955527 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5586acbcfaa0 con 0x5586accae330 2024-02-21T23:08:29.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.625+0000 7f4252a00640 1 --2- 172.21.15.28:0/2071955527 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5586acce96b0 0x5586acce0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:08:29.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.625+0000 7f4253400640 1 --2- 172.21.15.28:0/2071955527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5586accae330 0x5586accfc900 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f4244009930 tx=0x7f4244004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fb734a94ae94ddd server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:29.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.625+0000 7f4240c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:08:29.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.625+0000 7f4240c00640 1 -- 172.21.15.28:0/2071955527 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4244034040 con 0x5586accae330 2024-02-21T23:08:29.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.625+0000 7f4240c00640 1 -- 172.21.15.28:0/2071955527 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(122..122 src has 1..122) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4244046890 con 0x5586accae330 2024-02-21T23:08:29.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.626+0000 7f425d7d7740 1 -- 172.21.15.28:0/2071955527 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4244039b20 con 0x5586accae330 2024-02-21T23:08:29.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.626+0000 7f4240c00640 1 -- 172.21.15.28:0/2071955527 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=122) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4244046a90 con 0x5586accae330 2024-02-21T23:08:29.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.626+0000 7f425d7d7740 10 client.8112 fetch_fsmap learned FSMap version 122 2024-02-21T23:08:29.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.626+0000 7f425d7d7740 1 -- 172.21.15.28:0/2071955527 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=122,monmap=2+}) v3 -- 0x5586ac9262d0 con 0x5586accae330 2024-02-21T23:08:29.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.626+0000 7f4240c00640 1 -- 172.21.15.28:0/2071955527 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4244039b20 con 0x5586accae330 2024-02-21T23:08:29.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.626+0000 7f4240c00640 1 -- 172.21.15.28:0/2071955527 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 122) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f424404c070 con 0x5586accae330 2024-02-21T23:08:29.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.626+0000 7f425d7d7740 10 client.8112 fetch_fsmap finished waiting for FSMap version 122 2024-02-21T23:08:29.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.626+0000 7f425d7d7740 10 client.8112 resolve_mds: resolved name 'd' to daemon mds.d 2024-02-21T23:08:29.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.626+0000 7f425d7d7740 20 client.8112 populate_metadata read hostname 'smithi028' 2024-02-21T23:08:29.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.626+0000 7f425d7d7740 1 --2- 172.21.15.28:0/2071955527 >> [v2:172.21.15.81:6835/1038096496,v1:172.21.15.81:6837/1038096496] conn(0x5586acce1400 0x5586accf9630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:29.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.626+0000 7f425d7d7740 4 client.8112 mds_command: new command op to 7914 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-21T23:08:29.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.626+0000 7f425d7d7740 1 -- 172.21.15.28:0/2071955527 --> [v2:172.21.15.81:6835/1038096496,v1:172.21.15.81:6837/1038096496] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5586acc1e9b0 con 0x5586acce1400 2024-02-21T23:08:29.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.627+0000 7f4252a00640 1 --2- 172.21.15.28:0/2071955527 >> [v2:172.21.15.81:6835/1038096496,v1:172.21.15.81:6837/1038096496] conn(0x5586acce1400 0x5586accf9630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:29.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.628+0000 7f4252a00640 1 --2- 172.21.15.28:0/2071955527 >> [v2:172.21.15.81:6835/1038096496,v1:172.21.15.81:6837/1038096496] conn(0x5586acce1400 0x5586accf9630 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d8feea182b7da3a8 server_cookie=5e05f8a93bfa048a in_seq=0 out_seq=0 2024-02-21T23:08:29.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.628+0000 7f4240c00640 10 client.8112 ms_handle_connect on v2:172.21.15.81:6835/1038096496 2024-02-21T23:08:29.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.630+0000 7f4240c00640 1 -- 172.21.15.28:0/2071955527 <== mds.0 v2:172.21.15.81:6835/1038096496 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5586acc1e9b0 con 0x5586acce1400 2024-02-21T23:08:29.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.630+0000 7f4240c00640 10 client.8112 handle_command_reply: tid=0 2024-02-21T23:08:29.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.630+0000 7f425d7d7740 1 client.8112 shutdown 2024-02-21T23:08:29.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.630+0000 7f425d7d7740 1 -- 172.21.15.28:0/2071955527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5586accae330 msgr2=0x5586accfc900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:29.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.630+0000 7f425d7d7740 1 --2- 172.21.15.28:0/2071955527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5586accae330 0x5586accfc900 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f4244009930 tx=0x7f4244004730 comp rx=0 tx=0).stop 2024-02-21T23:08:29.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.630+0000 7f425d7d7740 1 -- 172.21.15.28:0/2071955527 shutdown_connections 2024-02-21T23:08:29.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.630+0000 7f425d7d7740 1 --2- 172.21.15.28:0/2071955527 >> [v2:172.21.15.81:6835/1038096496,v1:172.21.15.81:6837/1038096496] conn(0x5586acce1400 0x5586accf9630 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:29.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.630+0000 7f425d7d7740 1 --2- 172.21.15.28:0/2071955527 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5586acce96b0 0x5586acce0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:29.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.630+0000 7f425d7d7740 1 --2- 172.21.15.28:0/2071955527 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5586accc91a0 0x5586accc7fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:29.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.630+0000 7f425d7d7740 1 --2- 172.21.15.28:0/2071955527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5586accae330 0x5586accfc900 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:29.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.630+0000 7f425d7d7740 1 -- 172.21.15.28:0/2071955527 >> 172.21.15.28:0/2071955527 conn(0x5586accad020 msgr2=0x5586acca65f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:29.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.630+0000 7f4240c00640 0 client.8112 ms_handle_reset on v2:172.21.15.81:6835/1038096496 2024-02-21T23:08:29.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.630+0000 7f425d7d7740 1 -- 172.21.15.28:0/2071955527 shutdown_connections 2024-02-21T23:08:29.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.630+0000 7f425d7d7740 1 -- 172.21.15.28:0/2071955527 wait complete. 2024-02-21T23:08:29.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.630+0000 7f425d7d7740 20 client.8112 trim_cache size 0 max 16384 2024-02-21T23:08:29.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.649+0000 7f425d7d7740 1 Processor -- start 2024-02-21T23:08:29.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.649+0000 7f425d7d7740 1 -- start start 2024-02-21T23:08:29.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.649+0000 7f425d7d7740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5586accae330 0x5586accf9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:29.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.649+0000 7f425d7d7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5586accc91a0 0x5586acce8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:29.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.650+0000 7f425d7d7740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5586acce96b0 0x5586acceabc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:29.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.650+0000 7f425d7d7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42541f7930 con 0x5586accc91a0 2024-02-21T23:08:29.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.650+0000 7f425d7d7740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42541f6b40 con 0x5586acce96b0 2024-02-21T23:08:29.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.650+0000 7f425d7d7740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4254071ac0 con 0x5586accae330 2024-02-21T23:08:29.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.650+0000 7f4252a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5586accc91a0 0x5586acce8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:29.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.650+0000 7f4252a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5586accc91a0 0x5586acce8aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41358/0 (socket says 172.21.15.28:41358) 2024-02-21T23:08:29.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.650+0000 7f4252a00640 1 -- 172.21.15.28:0/565599120 learned_addr learned my addr 172.21.15.28:0/565599120 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:29.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.650+0000 7f4253400640 1 --2- 172.21.15.28:0/565599120 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5586accae330 0x5586accf9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:29.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.650+0000 7f4253e00640 1 --2- 172.21.15.28:0/565599120 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5586acce96b0 0x5586acceabc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:29.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.650+0000 7f4252a00640 1 -- 172.21.15.28:0/565599120 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5586accae330 msgr2=0x5586accf9ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:29.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.650+0000 7f4252a00640 1 --2- 172.21.15.28:0/565599120 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5586accae330 0x5586accf9ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:29.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.650+0000 7f4252a00640 1 -- 172.21.15.28:0/565599120 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5586acce96b0 msgr2=0x5586acceabc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:29.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.650+0000 7f4252a00640 1 --2- 172.21.15.28:0/565599120 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5586acce96b0 0x5586acceabc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:29.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.650+0000 7f4252a00640 1 -- 172.21.15.28:0/565599120 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5586ac9f06b0 con 0x5586accc91a0 2024-02-21T23:08:29.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.651+0000 7f4252a00640 1 --2- 172.21.15.28:0/565599120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5586accc91a0 0x5586acce8aa0 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f423c00c430 tx=0x7f423c088340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8681a9293e98d server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:29.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.651+0000 7f4243400640 1 -- 172.21.15.28:0/565599120 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f423c075070 con 0x5586accc91a0 2024-02-21T23:08:29.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.651+0000 7f4243400640 1 -- 172.21.15.28:0/565599120 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f423c056760 con 0x5586accc91a0 2024-02-21T23:08:29.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.651+0000 7f4243400640 1 -- 172.21.15.28:0/565599120 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f423c0569c0 con 0x5586accc91a0 2024-02-21T23:08:29.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.651+0000 7f425d7d7740 1 -- 172.21.15.28:0/565599120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5586accc91a0 msgr2=0x5586acce8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:29.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.651+0000 7f425d7d7740 1 --2- 172.21.15.28:0/565599120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5586accc91a0 0x5586acce8aa0 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f423c00c430 tx=0x7f423c088340 comp rx=0 tx=0).stop 2024-02-21T23:08:29.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.652+0000 7f425d7d7740 1 -- 172.21.15.28:0/565599120 shutdown_connections 2024-02-21T23:08:29.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.652+0000 7f425d7d7740 1 --2- 172.21.15.28:0/565599120 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5586acce96b0 0x5586acceabc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:29.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.652+0000 7f425d7d7740 1 --2- 172.21.15.28:0/565599120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5586accc91a0 0x5586acce8aa0 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:29.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.652+0000 7f425d7d7740 1 --2- 172.21.15.28:0/565599120 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5586accae330 0x5586accf9ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:29.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.652+0000 7f425d7d7740 1 -- 172.21.15.28:0/565599120 >> 172.21.15.28:0/565599120 conn(0x5586accad020 msgr2=0x5586acce1aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:29.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.652+0000 7f425d7d7740 1 -- 172.21.15.28:0/565599120 shutdown_connections 2024-02-21T23:08:29.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.652+0000 7f425d7d7740 1 -- 172.21.15.28:0/565599120 wait complete. 2024-02-21T23:08:29.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.652+0000 7f425d7d7740 1 -- start start 2024-02-21T23:08:29.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.652+0000 7f425d7d7740 1 Processor -- start 2024-02-21T23:08:29.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.653+0000 7f425d7d7740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5586accae330 0x5586acce5e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:29.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.653+0000 7f425d7d7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5586acce96b0 0x5586acd5c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:29.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.653+0000 7f425d7d7740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5586acd0c120 0x5586acd55840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:29.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.653+0000 7f425d7d7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5586accc9750 con 0x5586acce96b0 2024-02-21T23:08:29.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.653+0000 7f425d7d7740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42541f6540 con 0x5586acd0c120 2024-02-21T23:08:29.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.653+0000 7f425d7d7740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x5586accca610 con 0x5586accae330 2024-02-21T23:08:29.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.653+0000 7f425d7d7740 10 client.0 _ll_register_callbacks cb 0x7f42440308e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:08:29.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.654+0000 7f4252a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5586acce96b0 0x5586acd5c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:29.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.654+0000 7f4252a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5586acce96b0 0x5586acd5c4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41374/0 (socket says 172.21.15.28:41374) 2024-02-21T23:08:29.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.654+0000 7f4252a00640 1 -- 172.21.15.28:0/963255010 learned_addr learned my addr 172.21.15.28:0/963255010 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:29.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.654+0000 7f4252a00640 1 -- 172.21.15.28:0/963255010 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5586accae330 msgr2=0x5586acce5e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:29.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.654+0000 7f4253400640 1 --2- 172.21.15.28:0/963255010 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5586accae330 0x5586acce5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:29.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.654+0000 7f4253e00640 1 --2- 172.21.15.28:0/963255010 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5586acd0c120 0x5586acd55840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:29.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.654+0000 7f4252a00640 1 --2- 172.21.15.28:0/963255010 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5586accae330 0x5586acce5e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:29.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.654+0000 7f4252a00640 1 -- 172.21.15.28:0/963255010 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5586acd0c120 msgr2=0x5586acd55840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:29.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.654+0000 7f4252a00640 1 --2- 172.21.15.28:0/963255010 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5586acd0c120 0x5586acd55840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:29.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.654+0000 7f4252a00640 1 -- 172.21.15.28:0/963255010 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5586accfc780 con 0x5586acce96b0 2024-02-21T23:08:29.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.654+0000 7f4252a00640 1 --2- 172.21.15.28:0/963255010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5586acce96b0 0x5586acd5c4b0 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f423c054b20 tx=0x7f4244030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d82b3bc29c22c62 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:29.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.654+0000 7f4240c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:08:29.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.655+0000 7f4240c00640 1 -- 172.21.15.28:0/963255010 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f423c075070 con 0x5586acce96b0 2024-02-21T23:08:29.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.655+0000 7f4240c00640 1 -- 172.21.15.28:0/963255010 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(122..122 src has 1..122) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f423c0884f0 con 0x5586acce96b0 2024-02-21T23:08:29.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.655+0000 7f425d7d7740 1 -- 172.21.15.28:0/963255010 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f423c056760 con 0x5586acce96b0 2024-02-21T23:08:29.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.655+0000 7f4240c00640 1 -- 172.21.15.28:0/963255010 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=122) v2 ==== 24+0+0 (secure 0 0 0) 0x7f423c070130 con 0x5586acce96b0 2024-02-21T23:08:29.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.655+0000 7f425d7d7740 10 client.8118 fetch_fsmap learned FSMap version 122 2024-02-21T23:08:29.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.655+0000 7f425d7d7740 1 -- 172.21.15.28:0/963255010 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=122,monmap=2+}) v3 -- 0x5586acce8cf0 con 0x5586acce96b0 2024-02-21T23:08:29.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.655+0000 7f4240c00640 1 -- 172.21.15.28:0/963255010 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f423c056760 con 0x5586acce96b0 2024-02-21T23:08:29.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.655+0000 7f4240c00640 1 -- 172.21.15.28:0/963255010 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 122) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f423c054d30 con 0x5586acce96b0 2024-02-21T23:08:29.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.655+0000 7f425d7d7740 10 client.8118 fetch_fsmap finished waiting for FSMap version 122 2024-02-21T23:08:29.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.655+0000 7f425d7d7740 10 client.8118 resolve_mds: resolved name 'd' to daemon mds.d 2024-02-21T23:08:29.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.655+0000 7f425d7d7740 20 client.8118 populate_metadata read hostname 'smithi028' 2024-02-21T23:08:29.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.655+0000 7f425d7d7740 1 --2- 172.21.15.28:0/963255010 >> [v2:172.21.15.81:6835/1038096496,v1:172.21.15.81:6837/1038096496] conn(0x5586accdffc0 0x5586acce03a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:29.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.655+0000 7f425d7d7740 4 client.8118 mds_command: new command op to 7914 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-21T23:08:29.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.655+0000 7f425d7d7740 1 -- 172.21.15.28:0/963255010 --> [v2:172.21.15.81:6835/1038096496,v1:172.21.15.81:6837/1038096496] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5586acc18d00 con 0x5586accdffc0 2024-02-21T23:08:29.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.656+0000 7f4253400640 1 --2- 172.21.15.28:0/963255010 >> [v2:172.21.15.81:6835/1038096496,v1:172.21.15.81:6837/1038096496] conn(0x5586accdffc0 0x5586acce03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:29.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.657+0000 7f4253400640 1 --2- 172.21.15.28:0/963255010 >> [v2:172.21.15.81:6835/1038096496,v1:172.21.15.81:6837/1038096496] conn(0x5586accdffc0 0x5586acce03a0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5c832a705ca33b83 server_cookie=fc6e960d8ec6ae0 in_seq=0 out_seq=0 2024-02-21T23:08:29.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.657+0000 7f4240c00640 10 client.8118 ms_handle_connect on v2:172.21.15.81:6835/1038096496 2024-02-21T23:08:29.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.657+0000 7f4240c00640 1 -- 172.21.15.28:0/963255010 <== mds.0 v2:172.21.15.81:6835/1038096496 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+322 (crc 0 0 0) 0x5586acc18d00 con 0x5586accdffc0 2024-02-21T23:08:29.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.657+0000 7f4240c00640 10 client.8118 handle_command_reply: tid=0 2024-02-21T23:08:29.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.657+0000 7f425d7d7740 1 client.8118 shutdown 2024-02-21T23:08:29.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.658+0000 7f425d7d7740 1 -- 172.21.15.28:0/963255010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5586acce96b0 msgr2=0x5586acd5c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:29.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.658+0000 7f425d7d7740 1 --2- 172.21.15.28:0/963255010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5586acce96b0 0x5586acd5c4b0 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f423c054b20 tx=0x7f4244030000 comp rx=0 tx=0).stop 2024-02-21T23:08:29.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.658+0000 7f425d7d7740 1 -- 172.21.15.28:0/963255010 shutdown_connections 2024-02-21T23:08:29.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.658+0000 7f425d7d7740 1 --2- 172.21.15.28:0/963255010 >> [v2:172.21.15.81:6835/1038096496,v1:172.21.15.81:6837/1038096496] conn(0x5586accdffc0 0x5586acce03a0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:29.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.658+0000 7f425d7d7740 1 --2- 172.21.15.28:0/963255010 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5586acd0c120 0x5586acd55840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:29.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.658+0000 7f425d7d7740 1 --2- 172.21.15.28:0/963255010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5586acce96b0 0x5586acd5c4b0 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:29.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.658+0000 7f425d7d7740 1 --2- 172.21.15.28:0/963255010 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5586accae330 0x5586acce5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:29.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.658+0000 7f4240c00640 0 client.8118 ms_handle_reset on v2:172.21.15.81:6835/1038096496 2024-02-21T23:08:29.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.658+0000 7f425d7d7740 1 -- 172.21.15.28:0/963255010 >> 172.21.15.28:0/963255010 conn(0x5586accad020 msgr2=0x5586acced0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:29.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.658+0000 7f425d7d7740 1 -- 172.21.15.28:0/963255010 shutdown_connections 2024-02-21T23:08:29.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.658+0000 7f425d7d7740 1 -- 172.21.15.28:0/963255010 wait complete. 2024-02-21T23:08:29.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.658+0000 7f425d7d7740 20 client.8118 trim_cache size 0 max 16384 2024-02-21T23:08:29.651 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-21T23:08:29.652 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac", 2024-02-21T23:08:29.652 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-02-21T23:08:29.652 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 7914, 2024-02-21T23:08:29.652 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-02-21T23:08:29.652 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-02-21T23:08:29.652 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-02-21T23:08:29.652 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 15.239414472, 2024-02-21T23:08:29.652 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 122, 2024-02-21T23:08:29.652 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 122, 2024-02-21T23:08:29.652 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 122, 2024-02-21T23:08:29.652 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 20.732242656 2024-02-21T23:08:29.652 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-21T23:08:29.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.659+0000 7f4259200640 1 -- 172.21.15.28:0/1182661795 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f423006b4d0 msgr2=0x7f423006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:29.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.659+0000 7f4259200640 1 --2- 172.21.15.28:0/1182661795 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f423006b4d0 0x7f423006d990 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f4244036000 tx=0x7f424403b040 comp rx=0 tx=0).stop 2024-02-21T23:08:29.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.659+0000 7f4259200640 1 -- 172.21.15.28:0/1182661795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42540bf9a0 msgr2=0x7f425414d2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:29.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.659+0000 7f4259200640 1 --2- 172.21.15.28:0/1182661795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42540bf9a0 0x7f425414d2b0 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f423c00c460 tx=0x7f423c00c920 comp rx=0 tx=0).stop 2024-02-21T23:08:29.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.659+0000 7f4259200640 1 -- 172.21.15.28:0/1182661795 shutdown_connections 2024-02-21T23:08:29.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.659+0000 7f4259200640 1 --2- 172.21.15.28:0/1182661795 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f423006b4d0 0x7f423006d990 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:29.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.659+0000 7f4259200640 1 --2- 172.21.15.28:0/1182661795 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f42541527a0 0x7f425414d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:29.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.659+0000 7f4259200640 1 --2- 172.21.15.28:0/1182661795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42540bf9a0 0x7f425414d2b0 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:29.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.659+0000 7f4259200640 1 --2- 172.21.15.28:0/1182661795 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f425406d410 0x7f42541e1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:29.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.659+0000 7f4259200640 1 -- 172.21.15.28:0/1182661795 >> 172.21.15.28:0/1182661795 conn(0x7f42540a8320 msgr2=0x7f4254156340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:29.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.659+0000 7f4259200640 1 -- 172.21.15.28:0/1182661795 shutdown_connections 2024-02-21T23:08:29.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.659+0000 7f4259200640 1 -- 172.21.15.28:0/1182661795 wait complete. 2024-02-21T23:08:29.664 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 9, "epoch": 122, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 9, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-02-21T23:08:13.883141+0000", "damaged": [], "data_pools": [ 20 ], "enabled": true, "epoch": 122, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7914": { "addr": "172.21.15.81:6837/1038096496", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6835", "nonce": 1038096496, "type": "v2" }, { "addr": "172.21.15.81:6837", "nonce": 1038096496, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7914, "incarnation": 117, "join_fscid": 9, "name": "d", "rank": 0, "state": "up:active", "state_seq": 5 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 19, "modified": "2024-02-21T23:08:22.473107+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7914 } } } ], "standbys": [ { "addr": "172.21.15.28:6835/3556518453", "addrs": { "addrvec": [ { "addr": "172.21.15.28:6833", "nonce": 3556518453, "type": "v2" }, { "addr": "172.21.15.28:6835", "nonce": 3556518453, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 116, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5948, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.81:6836/3293264689", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6834", "nonce": 3293264689, "type": "v2" }, { "addr": "172.21.15.81:6836", "nonce": 3293264689, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 116, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5951, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.28:6834/1773560593", "addrs": { "addrvec": [ { "addr": "172.21.15.28:6832", "nonce": 1773560593, "type": "v2" }, { "addr": "172.21.15.28:6834", "nonce": 1773560593, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 115, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7905, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-02-21T23:08:29.666 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': 9, 'name': 'd'}] 2024-02-21T23:08:29.666 INFO:tasks.cephfs.test_failover:current = [{'gid': 7905, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.28:6834/1773560593', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 1773560593}, {'type': 'v1', 'addr': '172.21.15.28:6834', 'nonce': 1773560593}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 115}, {'gid': 5951, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.81:6836/3293264689', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6834', 'nonce': 3293264689}, {'type': 'v1', 'addr': '172.21.15.81:6836', 'nonce': 3293264689}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 116}, {'gid': 5948, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.28:6835/3556518453', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6833', 'nonce': 3556518453}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3556518453}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 116}, {'gid': 7914, 'name': 'd', 'rank': 0, 'incarnation': 117, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.81:6837/1038096496', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6835', 'nonce': 1038096496}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 1038096496}]}, 'join_fscid': 9, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}] 2024-02-21T23:08:29.666 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-02-21T23:08:29.666 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail 9:0 2024-02-21T23:08:29.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.890+0000 7f1abd200640 1 Processor -- start 2024-02-21T23:08:29.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.890+0000 7f1abd200640 1 -- start start 2024-02-21T23:08:29.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.891+0000 7f1abd200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f1ab8153950 0x7f1ab80c15d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:29.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.891+0000 7f1abd200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1ab815b6d0 0x7f1ab815dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:29.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.891+0000 7f1abd200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ab8152e70 con 0x7f1ab80c11f0 2024-02-21T23:08:29.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.891+0000 7f1abd200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ab80a5ec0 con 0x7f1ab8153950 2024-02-21T23:08:29.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.891+0000 7f1abd200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab8153150 con 0x7f1ab815b6d0 2024-02-21T23:08:29.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.892+0000 7f1ab5e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f1ab80c11f0 0x7f1ab806dce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:56858/0 (socket says 172.21.15.28:56858) 2024-02-21T23:08:29.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.892+0000 7f1ab5e00640 1 -- 172.21.15.28:0/1592168389 learned_addr learned my addr 172.21.15.28:0/1592168389 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:08:29.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.892+0000 7f1ab5400640 1 --2- 172.21.15.28:0/1592168389 >> v2:172.21.15.81:3300/0 conn(0x7f1ab8153950 0x7f1ab80c15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:29.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.892+0000 7f1ab6800640 1 --2- 172.21.15.28:0/1592168389 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1ab815b6d0 0x7f1ab815dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:29.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.892+0000 7f1ab4a00640 1 -- 172.21.15.28:0/1592168389 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3051684022 0 0) 0x7f1ab8152e70 con 0x7f1ab80c11f0 2024-02-21T23:08:29.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.892+0000 7f1ab4a00640 1 -- 172.21.15.28:0/1592168389 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a9c003520 con 0x7f1ab80c11f0 2024-02-21T23:08:29.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.893+0000 7f1ab4a00640 1 -- 172.21.15.28:0/1592168389 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2818104108 0 0) 0x7f1a9c003520 con 0x7f1ab80c11f0 2024-02-21T23:08:29.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.893+0000 7f1ab4a00640 1 -- 172.21.15.28:0/1592168389 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1ab815b6d0 msgr2=0x7f1ab815dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:29.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.893+0000 7f1ab4a00640 1 --2- 172.21.15.28:0/1592168389 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1ab815b6d0 0x7f1ab815dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:29.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.893+0000 7f1ab4a00640 1 -- 172.21.15.28:0/1592168389 >> v2:172.21.15.81:3300/0 conn(0x7f1ab8153950 msgr2=0x7f1ab80c15d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:29.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.893+0000 7f1ab4a00640 1 --2- 172.21.15.28:0/1592168389 >> v2:172.21.15.81:3300/0 conn(0x7f1ab8153950 0x7f1ab80c15d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:29.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.893+0000 7f1ab4a00640 1 -- 172.21.15.28:0/1592168389 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ab8153430 con 0x7f1ab80c11f0 2024-02-21T23:08:29.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.893+0000 7f1ab4a00640 1 -- 172.21.15.28:0/1592168389 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f1aa8001670 con 0x7f1ab80c11f0 2024-02-21T23:08:29.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.893+0000 7f1ab4a00640 1 -- 172.21.15.28:0/1592168389 >> v1:172.21.15.28:6789/0 conn(0x7f1ab80c11f0 legacy=0x7f1ab806dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:29.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.893+0000 7f1ab4a00640 1 --2- 172.21.15.28:0/1592168389 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1ab815b6d0 0x7f1a9c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:29.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.893+0000 7f1ab4a00640 1 --2- 172.21.15.28:0/1592168389 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a9c003f30 0x7f1a9c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:29.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.893+0000 7f1ab4a00640 1 --2- 172.21.15.28:0/1592168389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a9c004d40 0x7f1a9c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:29.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.893+0000 7f1ab4a00640 1 -- 172.21.15.28:0/1592168389 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab80a5ec0 con 0x7f1a9c004d40 2024-02-21T23:08:29.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.893+0000 7f1ab4a00640 1 -- 172.21.15.28:0/1592168389 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab8153150 con 0x7f1a9c003f30 2024-02-21T23:08:29.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.893+0000 7f1ab4a00640 1 -- 172.21.15.28:0/1592168389 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a9c00ba10 con 0x7f1ab815b6d0 2024-02-21T23:08:29.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.893+0000 7f1ab6800640 1 --2- 172.21.15.28:0/1592168389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a9c004d40 0x7f1a9c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:29.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.893+0000 7f1ab5e00640 1 --2- 172.21.15.28:0/1592168389 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1ab815b6d0 0x7f1a9c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:29.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.893+0000 7f1ab5400640 1 --2- 172.21.15.28:0/1592168389 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a9c003f30 0x7f1a9c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:29.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.894+0000 7f1ab6800640 1 -- 172.21.15.28:0/1592168389 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1ab815b6d0 msgr2=0x7f1a9c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:29.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.894+0000 7f1ab6800640 1 --2- 172.21.15.28:0/1592168389 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1ab815b6d0 0x7f1a9c0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:29.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.894+0000 7f1ab6800640 1 -- 172.21.15.28:0/1592168389 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a9c003f30 msgr2=0x7f1a9c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:29.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.894+0000 7f1ab6800640 1 --2- 172.21.15.28:0/1592168389 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a9c003f30 0x7f1a9c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:29.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.894+0000 7f1ab6800640 1 -- 172.21.15.28:0/1592168389 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a9c003520 con 0x7f1a9c004d40 2024-02-21T23:08:29.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.895+0000 7f1ab6800640 1 --2- 172.21.15.28:0/1592168389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a9c004d40 0x7f1a9c0051c0 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f1ab000e170 tx=0x7f1a9c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=561883dbee911081 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:29.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.895+0000 7f1ab4a00640 1 -- 172.21.15.28:0/1592168389 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab0039070 con 0x7f1a9c004d40 2024-02-21T23:08:29.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.895+0000 7f1ab4a00640 1 -- 172.21.15.28:0/1592168389 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ab002cbf0 con 0x7f1a9c004d40 2024-02-21T23:08:29.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.895+0000 7f1ab4a00640 1 -- 172.21.15.28:0/1592168389 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab0009c10 con 0x7f1a9c004d40 2024-02-21T23:08:29.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.896+0000 7f1abd200640 1 -- 172.21.15.28:0/1592168389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a9c004d40 msgr2=0x7f1a9c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:29.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.896+0000 7f1abd200640 1 --2- 172.21.15.28:0/1592168389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a9c004d40 0x7f1a9c0051c0 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f1ab000e170 tx=0x7f1a9c005830 comp rx=0 tx=0).stop 2024-02-21T23:08:29.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.896+0000 7f1ab6800640 1 -- 172.21.15.28:0/1592168389 reap_dead start 2024-02-21T23:08:29.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.896+0000 7f1abd200640 1 -- 172.21.15.28:0/1592168389 shutdown_connections 2024-02-21T23:08:29.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.896+0000 7f1abd200640 1 -- 172.21.15.28:0/1592168389 >> 172.21.15.28:0/1592168389 conn(0x7f1ab80c67f0 msgr2=0x7f1ab80c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:29.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.896+0000 7f1abd200640 1 -- 172.21.15.28:0/1592168389 shutdown_connections 2024-02-21T23:08:29.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.896+0000 7f1abd200640 1 -- 172.21.15.28:0/1592168389 wait complete. 2024-02-21T23:08:29.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.898+0000 7f1abd200640 1 Processor -- start 2024-02-21T23:08:29.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.898+0000 7f1abd200640 1 -- start start 2024-02-21T23:08:29.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.899+0000 7f1abd200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1a9c004d40 0x7f1ab8141610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:29.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.899+0000 7f1abd200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ab8141b50 0x7f1ab81448d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:29.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.899+0000 7f1abd200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ab8144e10 0x7f1ab81ee260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:29.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.899+0000 7f1abd200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab80a6110 con 0x7f1ab8144e10 2024-02-21T23:08:29.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.899+0000 7f1abd200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab80c4c00 con 0x7f1ab8141b50 2024-02-21T23:08:29.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.899+0000 7f1abd200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab80c0d90 con 0x7f1a9c004d40 2024-02-21T23:08:29.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.899+0000 7f1ab6800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ab8144e10 0x7f1ab81ee260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:29.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.899+0000 7f1ab6800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ab8144e10 0x7f1ab81ee260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41388/0 (socket says 172.21.15.28:41388) 2024-02-21T23:08:29.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.899+0000 7f1ab6800640 1 -- 172.21.15.28:0/238382360 learned_addr learned my addr 172.21.15.28:0/238382360 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:29.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.899+0000 7f1ab5e00640 1 --2- 172.21.15.28:0/238382360 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1a9c004d40 0x7f1ab8141610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:29.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.899+0000 7f1ab6800640 1 -- 172.21.15.28:0/238382360 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1a9c004d40 msgr2=0x7f1ab8141610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:29.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.899+0000 7f1ab6800640 1 --2- 172.21.15.28:0/238382360 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1a9c004d40 0x7f1ab8141610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:29.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.899+0000 7f1ab6800640 1 -- 172.21.15.28:0/238382360 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ab8141b50 msgr2=0x7f1ab81448d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:29.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.899+0000 7f1ab6800640 1 --2- 172.21.15.28:0/238382360 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ab8141b50 0x7f1ab81448d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:29.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.899+0000 7f1ab6800640 1 -- 172.21.15.28:0/238382360 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ab81e24d0 con 0x7f1ab8144e10 2024-02-21T23:08:29.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.900+0000 7f1ab6800640 1 --2- 172.21.15.28:0/238382360 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ab8144e10 0x7f1ab81ee260 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f1ab000e170 tx=0x7f1ab0009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:29.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.900+0000 7f1aae000640 1 -- 172.21.15.28:0/238382360 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab0039030 con 0x7f1ab8144e10 2024-02-21T23:08:29.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.900+0000 7f1aae000640 1 -- 172.21.15.28:0/238382360 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ab003a040 con 0x7f1ab8144e10 2024-02-21T23:08:29.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.900+0000 7f1aae000640 1 -- 172.21.15.28:0/238382360 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab0003990 con 0x7f1ab8144e10 2024-02-21T23:08:29.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.900+0000 7f1abd200640 1 -- 172.21.15.28:0/238382360 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ab814c810 con 0x7f1ab8144e10 2024-02-21T23:08:29.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.900+0000 7f1abd200640 1 -- 172.21.15.28:0/238382360 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ab8143df0 con 0x7f1ab8144e10 2024-02-21T23:08:29.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.902+0000 7f1abd200640 1 -- 172.21.15.28:0/238382360 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ab8067f30 con 0x7f1ab8144e10 2024-02-21T23:08:29.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.902+0000 7f1aae000640 1 -- 172.21.15.28:0/238382360 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1ab0045030 con 0x7f1ab8144e10 2024-02-21T23:08:29.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.903+0000 7f1aae000640 1 --2- 172.21.15.28:0/238382360 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1a9806b500 0x7f1a9806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:29.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.903+0000 7f1aae000640 1 -- 172.21.15.28:0/238382360 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1ab00ac410 con 0x7f1ab8144e10 2024-02-21T23:08:29.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.904+0000 7f1ab5e00640 1 --2- 172.21.15.28:0/238382360 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1a9806b500 0x7f1a9806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:29.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.905+0000 7f1ab5e00640 1 --2- 172.21.15.28:0/238382360 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1a9806b500 0x7f1a9806d9c0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f1ab8150600 tx=0x7f1aa8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:29.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:29.908+0000 7f1aae000640 1 -- 172.21.15.28:0/238382360 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1ab004c020 con 0x7f1ab8144e10 2024-02-21T23:08:30.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.182+0000 7f1abd200640 1 -- 172.21.15.28:0/238382360 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "9:0"} v 0) v1 -- 0x7f1ab80c0960 con 0x7f1ab8144e10 2024-02-21T23:08:30.172 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-02-21T23:08:30.183+0000 7f5ddc400640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-21T23:08:30.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.186+0000 7f1aae000640 1 -- 172.21.15.28:0/238382360 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "9:0"}]=0 failed mds gid 7914 v123) v1 ==== 97+0+0 (secure 0 0 0) 0x7f1ab007b0f0 con 0x7f1ab8144e10 2024-02-21T23:08:30.175 INFO:teuthology.orchestra.run.smithi028.stderr:failed mds gid 7914 2024-02-21T23:08:30.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.189+0000 7f1abd200640 1 -- 172.21.15.28:0/238382360 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1a9806b500 msgr2=0x7f1a9806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:30.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.190+0000 7f1abd200640 1 --2- 172.21.15.28:0/238382360 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1a9806b500 0x7f1a9806d9c0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f1ab8150600 tx=0x7f1aa8009040 comp rx=0 tx=0).stop 2024-02-21T23:08:30.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.190+0000 7f1abd200640 1 -- 172.21.15.28:0/238382360 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ab8144e10 msgr2=0x7f1ab81ee260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:30.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.190+0000 7f1abd200640 1 --2- 172.21.15.28:0/238382360 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ab8144e10 0x7f1ab81ee260 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f1ab000e170 tx=0x7f1ab0009750 comp rx=0 tx=0).stop 2024-02-21T23:08:30.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.190+0000 7f1abd200640 1 -- 172.21.15.28:0/238382360 shutdown_connections 2024-02-21T23:08:30.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.190+0000 7f1abd200640 1 --2- 172.21.15.28:0/238382360 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1a9806b500 0x7f1a9806d9c0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:30.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.190+0000 7f1abd200640 1 --2- 172.21.15.28:0/238382360 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ab8144e10 0x7f1ab81ee260 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:30.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.190+0000 7f1abd200640 1 --2- 172.21.15.28:0/238382360 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ab8141b50 0x7f1ab81448d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:30.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.190+0000 7f1abd200640 1 --2- 172.21.15.28:0/238382360 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1a9c004d40 0x7f1ab8141610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:30.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.190+0000 7f1abd200640 1 -- 172.21.15.28:0/238382360 >> 172.21.15.28:0/238382360 conn(0x7f1ab80c67f0 msgr2=0x7f1ab806dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:30.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.190+0000 7f1abd200640 1 -- 172.21.15.28:0/238382360 shutdown_connections 2024-02-21T23:08:30.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.190+0000 7f1abd200640 1 -- 172.21.15.28:0/238382360 wait complete. 2024-02-21T23:08:30.201 INFO:tasks.ceph.mds.d.smithi081.stderr: -442> 2024-02-21T23:08:20.370+0000 7fb376e00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-21T23:08:30.202 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:08:30.261 INFO:tasks.ceph.mds.d.smithi081.stdout:starting mds.d at 2024-02-21T23:08:30.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.405+0000 7fb011200640 1 Processor -- start 2024-02-21T23:08:30.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.405+0000 7fb011200640 1 -- start start 2024-02-21T23:08:30.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.407+0000 7fb011200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb00c0a5fd0 0x7fb00c06dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:30.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.407+0000 7fb011200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb00c0c2760 con 0x7fb00c150450 2024-02-21T23:08:30.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.407+0000 7fb011200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb00c156310 con 0x7fb00c06ee40 2024-02-21T23:08:30.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.407+0000 7fb011200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb00c0c22c0 con 0x7fb00c0a5fd0 2024-02-21T23:08:30.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.407+0000 7fb00b400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb00c150450 0x7fb00c1536f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:56866/0 (socket says 172.21.15.28:56866) 2024-02-21T23:08:30.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.407+0000 7fb00b400640 1 -- 172.21.15.28:0/346694808 learned_addr learned my addr 172.21.15.28:0/346694808 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:08:30.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.407+0000 7fb00be00640 1 --2- 172.21.15.28:0/346694808 >> v2:172.21.15.28:3300/0 conn(0x7fb00c0a5fd0 0x7fb00c06dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:30.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.407+0000 7fb00be00640 1 -- 172.21.15.28:0/346694808 >> v1:172.21.15.81:6789/0 conn(0x7fb00c06ee40 legacy=0x7fb00c156af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:30.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.407+0000 7fb00be00640 1 -- 172.21.15.28:0/346694808 >> v1:172.21.15.28:6789/0 conn(0x7fb00c150450 legacy=0x7fb00c1536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:30.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.407+0000 7fb00be00640 1 -- 172.21.15.28:0/346694808 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb00c0c29d0 con 0x7fb00c0a5fd0 2024-02-21T23:08:30.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.407+0000 7fb00aa00640 1 -- 172.21.15.28:0/346694808 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3087425682 0 0) 0x7fb00c0c2760 con 0x7fb00c150450 2024-02-21T23:08:30.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.409+0000 7fb00be00640 1 --2- 172.21.15.28:0/346694808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb00c0a5fd0 0x7fb00c06dff0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7fb0040097a0 tx=0x7fb00402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b572b3b4252b1de server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:30.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.409+0000 7fb00aa00640 1 -- 172.21.15.28:0/346694808 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb004035020 con 0x7fb00c0a5fd0 2024-02-21T23:08:30.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.409+0000 7fb00aa00640 1 -- 172.21.15.28:0/346694808 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb004031830 con 0x7fb00c0a5fd0 2024-02-21T23:08:30.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.409+0000 7fb00aa00640 1 -- 172.21.15.28:0/346694808 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb004031c00 con 0x7fb00c0a5fd0 2024-02-21T23:08:30.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.409+0000 7fb011200640 1 -- 172.21.15.28:0/346694808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb00c0a5fd0 msgr2=0x7fb00c06dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:30.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.409+0000 7fb011200640 1 --2- 172.21.15.28:0/346694808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb00c0a5fd0 0x7fb00c06dff0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7fb0040097a0 tx=0x7fb00402b8d0 comp rx=0 tx=0).stop 2024-02-21T23:08:30.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.409+0000 7fb011200640 1 -- 172.21.15.28:0/346694808 shutdown_connections 2024-02-21T23:08:30.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.409+0000 7fb011200640 1 --2- 172.21.15.28:0/346694808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb00c0a5fd0 0x7fb00c06dff0 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:30.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.409+0000 7fb011200640 1 -- 172.21.15.28:0/346694808 >> 172.21.15.28:0/346694808 conn(0x7fb00c06b630 msgr2=0x7fb00c0c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:30.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.410+0000 7fb011200640 1 -- 172.21.15.28:0/346694808 shutdown_connections 2024-02-21T23:08:30.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.410+0000 7fb011200640 1 -- 172.21.15.28:0/346694808 wait complete. 2024-02-21T23:08:30.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.411+0000 7fb011200640 1 Processor -- start 2024-02-21T23:08:30.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.412+0000 7fb011200640 1 -- start start 2024-02-21T23:08:30.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.413+0000 7fb011200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb00c06ee40 0x7fb00c0c53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:30.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.413+0000 7fb011200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb00c0a5fd0 0x7fb00c1484b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:30.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.413+0000 7fb011200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb00c150450 0x7fb00c1489f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:30.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.413+0000 7fb011200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb00c157200 con 0x7fb00c06ee40 2024-02-21T23:08:30.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.413+0000 7fb011200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb00c156660 con 0x7fb00c150450 2024-02-21T23:08:30.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.413+0000 7fb011200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb00c06e530 con 0x7fb00c0a5fd0 2024-02-21T23:08:30.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.413+0000 7fb00b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb00c06ee40 0x7fb00c0c53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:30.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.413+0000 7fb00b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb00c06ee40 0x7fb00c0c53b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41404/0 (socket says 172.21.15.28:41404) 2024-02-21T23:08:30.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.413+0000 7fb00b400640 1 -- 172.21.15.28:0/1182265313 learned_addr learned my addr 172.21.15.28:0/1182265313 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:30.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.413+0000 7fb00be00640 1 --2- 172.21.15.28:0/1182265313 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb00c150450 0x7fb00c1489f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:30.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.413+0000 7fb002a00640 1 --2- 172.21.15.28:0/1182265313 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb00c0a5fd0 0x7fb00c1484b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:30.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.413+0000 7fb00b400640 1 -- 172.21.15.28:0/1182265313 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb00c0a5fd0 msgr2=0x7fb00c1484b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:30.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.413+0000 7fb00b400640 1 --2- 172.21.15.28:0/1182265313 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb00c0a5fd0 0x7fb00c1484b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:30.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.413+0000 7fb00b400640 1 -- 172.21.15.28:0/1182265313 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb00c150450 msgr2=0x7fb00c1489f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:30.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.413+0000 7fb00b400640 1 --2- 172.21.15.28:0/1182265313 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb00c150450 0x7fb00c1489f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:30.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.413+0000 7fb00b400640 1 -- 172.21.15.28:0/1182265313 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb00c149050 con 0x7fb00c06ee40 2024-02-21T23:08:30.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.413+0000 7fb00be00640 1 --2- 172.21.15.28:0/1182265313 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb00c150450 0x7fb00c1489f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:08:30.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.414+0000 7fb00b400640 1 --2- 172.21.15.28:0/1182265313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb00c06ee40 0x7fb00c0c53b0 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7faffc00c7f0 tx=0x7faffc00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:30.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.414+0000 7fb003e00640 1 -- 172.21.15.28:0/1182265313 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faffc018070 con 0x7fb00c06ee40 2024-02-21T23:08:30.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.414+0000 7fb003e00640 1 -- 172.21.15.28:0/1182265313 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faffc00a640 con 0x7fb00c06ee40 2024-02-21T23:08:30.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.414+0000 7fb003e00640 1 -- 172.21.15.28:0/1182265313 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faffc00aa30 con 0x7fb00c06ee40 2024-02-21T23:08:30.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.414+0000 7fb011200640 1 -- 172.21.15.28:0/1182265313 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb00c150cd0 con 0x7fb00c06ee40 2024-02-21T23:08:30.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.414+0000 7fb011200640 1 -- 172.21.15.28:0/1182265313 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb00c151b40 con 0x7fb00c06ee40 2024-02-21T23:08:30.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.415+0000 7fb011200640 1 -- 172.21.15.28:0/1182265313 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb00c04f290 con 0x7fb00c06ee40 2024-02-21T23:08:30.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.416+0000 7fb003e00640 1 -- 172.21.15.28:0/1182265313 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faffc01b070 con 0x7fb00c06ee40 2024-02-21T23:08:30.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.417+0000 7fb003e00640 1 --2- 172.21.15.28:0/1182265313 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fafe806b4d0 0x7fafe806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:30.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.417+0000 7fb003e00640 1 -- 172.21.15.28:0/1182265313 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 6187+0+0 (secure 0 0 0) 0x7faffc0899d0 con 0x7fb00c06ee40 2024-02-21T23:08:30.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.418+0000 7fb002a00640 1 --2- 172.21.15.28:0/1182265313 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fafe806b4d0 0x7fafe806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:30.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.419+0000 7fb002a00640 1 --2- 172.21.15.28:0/1182265313 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fafe806b4d0 0x7fafe806d990 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fb00c1eb800 tx=0x7faff4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:30.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.422+0000 7fb003e00640 1 -- 172.21.15.28:0/1182265313 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7faffc058700 con 0x7fb00c06ee40 2024-02-21T23:08:30.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.659+0000 7fb011200640 1 -- 172.21.15.28:0/1182265313 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb00c0c1f10 con 0x7fb00c06ee40 2024-02-21T23:08:30.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.659+0000 7fb003e00640 1 -- 172.21.15.28:0/1182265313 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 124 v124) v1 ==== 95+0+4156 (secure 0 0 0) 0x7faffc058000 con 0x7fb00c06ee40 2024-02-21T23:08:30.649 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:08:30.649 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":124,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5948,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3556518453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3556518453},{"type":"v1","addr":"172.21.15.28:6835","nonce":3556518453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116},{"gid":5951,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/3293264689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":3293264689},{"type":"v1","addr":"172.21.15.81:6836","nonce":3293264689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:08:13.883141+0000","modified":"2024-02-21T23:08:30.190996+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":123,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7905},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7905":{"gid":7905,"name":"a","rank":0,"incarnation":124,"state":"up:replay","state_seq":1,"addr":"172.21.15.28:6834/1773560593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1773560593},{"type":"v1","addr":"172.21.15.28:6834","nonce":1773560593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-21T23:08:30.649 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 124 2024-02-21T23:08:30.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.662+0000 7fb011200640 1 -- 172.21.15.28:0/1182265313 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fafe806b4d0 msgr2=0x7fafe806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:30.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.662+0000 7fb011200640 1 --2- 172.21.15.28:0/1182265313 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fafe806b4d0 0x7fafe806d990 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fb00c1eb800 tx=0x7faff4009040 comp rx=0 tx=0).stop 2024-02-21T23:08:30.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.662+0000 7fb011200640 1 -- 172.21.15.28:0/1182265313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb00c06ee40 msgr2=0x7fb00c0c53b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:30.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.662+0000 7fb011200640 1 --2- 172.21.15.28:0/1182265313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb00c06ee40 0x7fb00c0c53b0 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7faffc00c7f0 tx=0x7faffc00ccb0 comp rx=0 tx=0).stop 2024-02-21T23:08:30.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.662+0000 7fb011200640 1 -- 172.21.15.28:0/1182265313 shutdown_connections 2024-02-21T23:08:30.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.662+0000 7fb011200640 1 --2- 172.21.15.28:0/1182265313 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fafe806b4d0 0x7fafe806d990 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:30.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.662+0000 7fb011200640 1 --2- 172.21.15.28:0/1182265313 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb00c0a5fd0 0x7fb00c1484b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:30.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.662+0000 7fb011200640 1 --2- 172.21.15.28:0/1182265313 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb00c150450 0x7fb00c1489f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:30.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.662+0000 7fb011200640 1 --2- 172.21.15.28:0/1182265313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb00c06ee40 0x7fb00c0c53b0 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:30.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.662+0000 7fb011200640 1 -- 172.21.15.28:0/1182265313 >> 172.21.15.28:0/1182265313 conn(0x7fb00c06b630 msgr2=0x7fb00c0a63b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:30.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.662+0000 7fb011200640 1 -- 172.21.15.28:0/1182265313 shutdown_connections 2024-02-21T23:08:30.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:30.662+0000 7fb011200640 1 -- 172.21.15.28:0/1182265313 wait complete. 2024-02-21T23:08:31.071 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 124, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:08:13.883141+0000', 'modified': '2024-02-21T23:08:30.190996+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 123, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7905}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7905': {'gid': 7905, 'name': 'a', 'rank': 0, 'incarnation': 124, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.28:6834/1773560593', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 1773560593}, {'type': 'v1', 'addr': '172.21.15.28:6834', 'nonce': 1773560593}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [20], 'metadata_pool': 19, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:08:31.072 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7905:up:replay 2024-02-21T23:08:32.073 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:08:32.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.317+0000 7f58e7200640 1 Processor -- start 2024-02-21T23:08:32.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.317+0000 7f58e7200640 1 -- start start 2024-02-21T23:08:32.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.318+0000 7f58e7200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f58e806dff0 0x7f58e806e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:32.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.319+0000 7f58e7200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f58e8158580 0x7f58e814b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:32.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.319+0000 7f58e7200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58e80a8b40 con 0x7f58e80c0fe0 2024-02-21T23:08:32.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.319+0000 7f58e7200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58e80a8e00 con 0x7f58e8158580 2024-02-21T23:08:32.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.319+0000 7f58e7200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58e80a90e0 con 0x7f58e806dff0 2024-02-21T23:08:32.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.319+0000 7f58e6800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f58e8158580 0x7f58e814b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:32.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.319+0000 7f58e6800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f58e8158580 0x7f58e814b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41410/0 (socket says 172.21.15.28:41410) 2024-02-21T23:08:32.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.319+0000 7f58e6800640 1 -- 172.21.15.28:0/1760415813 learned_addr learned my addr 172.21.15.28:0/1760415813 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:32.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.319+0000 7f58e5e00640 1 --2- 172.21.15.28:0/1760415813 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f58e806dff0 0x7f58e806e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:32.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.319+0000 7f58e6800640 1 -- 172.21.15.28:0/1760415813 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f58e806dff0 msgr2=0x7f58e806e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:32.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.319+0000 7f58e6800640 1 --2- 172.21.15.28:0/1760415813 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f58e806dff0 0x7f58e806e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:32.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.319+0000 7f58e6800640 1 -- 172.21.15.28:0/1760415813 >> v1:172.21.15.81:6789/0 conn(0x7f58e80c0fe0 legacy=0x7f58e8071350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:32.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.319+0000 7f58e6800640 1 -- 172.21.15.28:0/1760415813 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58e8072ed0 con 0x7f58e8158580 2024-02-21T23:08:32.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.321+0000 7f58e6800640 1 --2- 172.21.15.28:0/1760415813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58e8158580 0x7f58e814b2b0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f58e0009e50 tx=0x7f58e002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff8a3b251a3d2199 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:32.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.321+0000 7f58e4a00640 1 -- 172.21.15.28:0/1760415813 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e002ca70 con 0x7f58e8158580 2024-02-21T23:08:32.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.321+0000 7f58e4a00640 1 -- 172.21.15.28:0/1760415813 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58e002cc10 con 0x7f58e8158580 2024-02-21T23:08:32.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.321+0000 7f58e4a00640 1 -- 172.21.15.28:0/1760415813 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e00343f0 con 0x7f58e8158580 2024-02-21T23:08:32.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.322+0000 7f58e7200640 1 -- 172.21.15.28:0/1760415813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58e8158580 msgr2=0x7f58e814b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:32.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.322+0000 7f58e7200640 1 --2- 172.21.15.28:0/1760415813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58e8158580 0x7f58e814b2b0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f58e0009e50 tx=0x7f58e002bed0 comp rx=0 tx=0).stop 2024-02-21T23:08:32.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.322+0000 7f58e7200640 1 -- 172.21.15.28:0/1760415813 shutdown_connections 2024-02-21T23:08:32.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.322+0000 7f58e7200640 1 --2- 172.21.15.28:0/1760415813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58e8158580 0x7f58e814b2b0 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:32.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.322+0000 7f58e7200640 1 --2- 172.21.15.28:0/1760415813 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f58e806dff0 0x7f58e806e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:32.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.322+0000 7f58e7200640 1 -- 172.21.15.28:0/1760415813 >> 172.21.15.28:0/1760415813 conn(0x7f58e806b630 msgr2=0x7f58e80c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:32.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.322+0000 7f58e7200640 1 -- 172.21.15.28:0/1760415813 shutdown_connections 2024-02-21T23:08:32.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.322+0000 7f58e7200640 1 -- 172.21.15.28:0/1760415813 wait complete. 2024-02-21T23:08:32.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.324+0000 7f58e7200640 1 Processor -- start 2024-02-21T23:08:32.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.324+0000 7f58e7200640 1 -- start start 2024-02-21T23:08:32.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.325+0000 7f58e7200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f58e806dff0 0x7f58e80ab820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:32.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.325+0000 7f58e7200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58e80c0fe0 0x7f58e81eaf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:32.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.325+0000 7f58e7200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58e8158580 0x7f58e81eb4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:32.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.325+0000 7f58e7200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58e8158a10 con 0x7f58e8158580 2024-02-21T23:08:32.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.325+0000 7f58e7200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58e8073520 con 0x7f58e80c0fe0 2024-02-21T23:08:32.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.325+0000 7f58e7200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58e8158b90 con 0x7f58e806dff0 2024-02-21T23:08:32.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.325+0000 7f58e6800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58e8158580 0x7f58e81eb4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:32.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.325+0000 7f58e6800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58e8158580 0x7f58e81eb4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41414/0 (socket says 172.21.15.28:41414) 2024-02-21T23:08:32.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.325+0000 7f58e6800640 1 -- 172.21.15.28:0/1335419 learned_addr learned my addr 172.21.15.28:0/1335419 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:32.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.325+0000 7f58e5400640 1 --2- 172.21.15.28:0/1335419 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58e80c0fe0 0x7f58e81eaf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:32.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.325+0000 7f58e5e00640 1 --2- 172.21.15.28:0/1335419 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f58e806dff0 0x7f58e80ab820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:32.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.326+0000 7f58e6800640 1 -- 172.21.15.28:0/1335419 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f58e806dff0 msgr2=0x7f58e80ab820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:32.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.326+0000 7f58e6800640 1 --2- 172.21.15.28:0/1335419 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f58e806dff0 0x7f58e80ab820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:32.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.326+0000 7f58e6800640 1 -- 172.21.15.28:0/1335419 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58e80c0fe0 msgr2=0x7f58e81eaf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:32.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.326+0000 7f58e6800640 1 --2- 172.21.15.28:0/1335419 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58e80c0fe0 0x7f58e81eaf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:32.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.326+0000 7f58e6800640 1 -- 172.21.15.28:0/1335419 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58e8150a30 con 0x7f58e8158580 2024-02-21T23:08:32.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.326+0000 7f58e6800640 1 --2- 172.21.15.28:0/1335419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58e8158580 0x7f58e81eb4a0 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f58e0009e50 tx=0x7f58e0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:32.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.326+0000 7f58de000640 1 -- 172.21.15.28:0/1335419 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e0043020 con 0x7f58e8158580 2024-02-21T23:08:32.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.326+0000 7f58de000640 1 -- 172.21.15.28:0/1335419 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58e003e450 con 0x7f58e8158580 2024-02-21T23:08:32.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.326+0000 7f58de000640 1 -- 172.21.15.28:0/1335419 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e0009510 con 0x7f58e8158580 2024-02-21T23:08:32.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.326+0000 7f58e7200640 1 -- 172.21.15.28:0/1335419 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58e8156420 con 0x7f58e8158580 2024-02-21T23:08:32.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.327+0000 7f58e7200640 1 -- 172.21.15.28:0/1335419 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58e81569e0 con 0x7f58e8158580 2024-02-21T23:08:32.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.328+0000 7f58e7200640 1 -- 172.21.15.28:0/1335419 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58e804f290 con 0x7f58e8158580 2024-02-21T23:08:32.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.329+0000 7f58de000640 1 -- 172.21.15.28:0/1335419 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f58e0046020 con 0x7f58e8158580 2024-02-21T23:08:32.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.330+0000 7f58de000640 1 --2- 172.21.15.28:0/1335419 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f58c806b4d0 0x7f58c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:32.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.330+0000 7f58de000640 1 -- 172.21.15.28:0/1335419 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f58e00acc00 con 0x7f58e8158580 2024-02-21T23:08:32.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.330+0000 7f58e5e00640 1 --2- 172.21.15.28:0/1335419 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f58c806b4d0 0x7f58c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:32.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.331+0000 7f58e5e00640 1 --2- 172.21.15.28:0/1335419 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f58c806b4d0 0x7f58c806d990 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f58d8006fd0 tx=0x7f58d8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:32.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.334+0000 7f58de000640 1 -- 172.21.15.28:0/1335419 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f58e004d030 con 0x7f58e8158580 2024-02-21T23:08:32.561 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-02-21T23:08:32.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.583+0000 7f58e7200640 1 -- 172.21.15.28:0/1335419 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f58e80c7180 con 0x7f58e8158580 2024-02-21T23:08:32.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.583+0000 7f58de000640 1 -- 172.21.15.28:0/1335419 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) v1 ==== 95+0+4155 (secure 0 0 0) 0x7f58e8150a30 con 0x7f58e8158580 2024-02-21T23:08:32.572 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:08:32.573 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":127,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5948,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3556518453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3556518453},{"type":"v1","addr":"172.21.15.28:6835","nonce":3556518453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116},{"gid":5951,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/3293264689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":3293264689},{"type":"v1","addr":"172.21.15.81:6836","nonce":3293264689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116}],"filesystems":[{"mdsmap":{"epoch":127,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:08:13.883141+0000","modified":"2024-02-21T23:08:32.504798+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":124,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6634},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6634":{"gid":6634,"name":"d","rank":0,"incarnation":127,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6837/3687210533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":3687210533},{"type":"v1","addr":"172.21.15.81:6837","nonce":3687210533}]},"join_fscid":9,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-21T23:08:32.573 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 127 2024-02-21T23:08:32.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.585+0000 7f58e7200640 1 -- 172.21.15.28:0/1335419 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f58c806b4d0 msgr2=0x7f58c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:32.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.585+0000 7f58e7200640 1 --2- 172.21.15.28:0/1335419 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f58c806b4d0 0x7f58c806d990 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f58d8006fd0 tx=0x7f58d8008040 comp rx=0 tx=0).stop 2024-02-21T23:08:32.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.586+0000 7f58e7200640 1 -- 172.21.15.28:0/1335419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58e8158580 msgr2=0x7f58e81eb4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:32.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.586+0000 7f58e7200640 1 --2- 172.21.15.28:0/1335419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58e8158580 0x7f58e81eb4a0 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f58e0009e50 tx=0x7f58e0004a40 comp rx=0 tx=0).stop 2024-02-21T23:08:32.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.586+0000 7f58e7200640 1 -- 172.21.15.28:0/1335419 shutdown_connections 2024-02-21T23:08:32.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.586+0000 7f58e7200640 1 --2- 172.21.15.28:0/1335419 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f58c806b4d0 0x7f58c806d990 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:32.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.586+0000 7f58e7200640 1 --2- 172.21.15.28:0/1335419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58e8158580 0x7f58e81eb4a0 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:32.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.586+0000 7f58e7200640 1 --2- 172.21.15.28:0/1335419 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58e80c0fe0 0x7f58e81eaf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:32.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.586+0000 7f58e7200640 1 --2- 172.21.15.28:0/1335419 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f58e806dff0 0x7f58e80ab820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:32.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.586+0000 7f58e7200640 1 -- 172.21.15.28:0/1335419 >> 172.21.15.28:0/1335419 conn(0x7f58e806b630 msgr2=0x7f58e814cd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:32.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.586+0000 7f58e7200640 1 -- 172.21.15.28:0/1335419 shutdown_connections 2024-02-21T23:08:32.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:32.586+0000 7f58e7200640 1 -- 172.21.15.28:0/1335419 wait complete. 2024-02-21T23:08:32.995 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 127, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:08:13.883141+0000', 'modified': '2024-02-21T23:08:32.504798+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 124, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6634}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6634': {'gid': 6634, 'name': 'd', 'rank': 0, 'incarnation': 127, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.81:6837/3687210533', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6835', 'nonce': 3687210533}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3687210533}]}, 'join_fscid': 9, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [20], 'metadata_pool': 19, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:08:32.995 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6634:up:replay 2024-02-21T23:08:33.996 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:08:34.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.238+0000 7f4b5fc00640 1 Processor -- start 2024-02-21T23:08:34.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.238+0000 7f4b5fc00640 1 -- start start 2024-02-21T23:08:34.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.240+0000 7f4b5fc00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4b580bf990 0x7f4b580bfe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:34.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.240+0000 7f4b5fc00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b581551b0 con 0x7f4b5813e620 2024-02-21T23:08:34.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.240+0000 7f4b5fc00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b58155420 con 0x7f4b580c6460 2024-02-21T23:08:34.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.240+0000 7f4b5fc00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b580c71d0 con 0x7f4b580bf990 2024-02-21T23:08:34.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.240+0000 7f4b5d200640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f4b5813e620 0x7f4b58159070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:56882/0 (socket says 172.21.15.28:56882) 2024-02-21T23:08:34.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.240+0000 7f4b5d200640 1 -- 172.21.15.28:0/800625870 learned_addr learned my addr 172.21.15.28:0/800625870 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:08:34.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.241+0000 7f4b5dc00640 1 --2- 172.21.15.28:0/800625870 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4b580bf990 0x7f4b580bfe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:34.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.241+0000 7f4b57400640 1 -- 172.21.15.28:0/800625870 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3889104856 0 0) 0x7f4b581551b0 con 0x7f4b5813e620 2024-02-21T23:08:34.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.241+0000 7f4b57400640 1 -- 172.21.15.28:0/800625870 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b3c003520 con 0x7f4b5813e620 2024-02-21T23:08:34.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.241+0000 7f4b57400640 1 -- 172.21.15.28:0/800625870 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3728389966 0 0) 0x7f4b3c003520 con 0x7f4b5813e620 2024-02-21T23:08:34.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.241+0000 7f4b57400640 1 -- 172.21.15.28:0/800625870 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4b580bf990 msgr2=0x7f4b580bfe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:34.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.241+0000 7f4b57400640 1 --2- 172.21.15.28:0/800625870 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4b580bf990 0x7f4b580bfe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:34.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.241+0000 7f4b57400640 1 -- 172.21.15.28:0/800625870 >> v1:172.21.15.81:6789/0 conn(0x7f4b580c6460 legacy=0x7f4b580ca8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:34.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.241+0000 7f4b57400640 1 -- 172.21.15.28:0/800625870 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b58155700 con 0x7f4b5813e620 2024-02-21T23:08:34.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.241+0000 7f4b57400640 1 -- 172.21.15.28:0/800625870 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f4b48001670 con 0x7f4b5813e620 2024-02-21T23:08:34.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.241+0000 7f4b5dc00640 1 --2- 172.21.15.28:0/800625870 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4b580bf990 0x7f4b580bfe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:08:34.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.241+0000 7f4b57400640 1 -- 172.21.15.28:0/800625870 >> v1:172.21.15.28:6789/0 conn(0x7f4b5813e620 legacy=0x7f4b58159070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:34.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.241+0000 7f4b57400640 1 --2- 172.21.15.28:0/800625870 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4b580bf990 0x7f4b3c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:34.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.241+0000 7f4b57400640 1 --2- 172.21.15.28:0/800625870 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4b3c004020 0x7f4b3c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:34.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.241+0000 7f4b57400640 1 --2- 172.21.15.28:0/800625870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b3c004d40 0x7f4b3c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:34.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.241+0000 7f4b57400640 1 -- 172.21.15.28:0/800625870 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b580c71d0 con 0x7f4b3c004d40 2024-02-21T23:08:34.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.241+0000 7f4b57400640 1 -- 172.21.15.28:0/800625870 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b3c00bc00 con 0x7f4b3c004020 2024-02-21T23:08:34.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.241+0000 7f4b57400640 1 -- 172.21.15.28:0/800625870 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b3c00be00 con 0x7f4b580bf990 2024-02-21T23:08:34.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.242+0000 7f4b5dc00640 1 --2- 172.21.15.28:0/800625870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b3c004d40 0x7f4b3c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:34.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.242+0000 7f4b57e00640 1 --2- 172.21.15.28:0/800625870 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4b3c004020 0x7f4b3c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:34.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.242+0000 7f4b5d200640 1 --2- 172.21.15.28:0/800625870 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4b580bf990 0x7f4b3c0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:34.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.242+0000 7f4b5dc00640 1 -- 172.21.15.28:0/800625870 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4b580bf990 msgr2=0x7f4b3c0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:34.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.242+0000 7f4b5dc00640 1 --2- 172.21.15.28:0/800625870 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4b580bf990 0x7f4b3c0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:34.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.242+0000 7f4b5dc00640 1 -- 172.21.15.28:0/800625870 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4b3c004020 msgr2=0x7f4b3c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:34.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.242+0000 7f4b5dc00640 1 --2- 172.21.15.28:0/800625870 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4b3c004020 0x7f4b3c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:34.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.242+0000 7f4b5dc00640 1 -- 172.21.15.28:0/800625870 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b3c003520 con 0x7f4b3c004d40 2024-02-21T23:08:34.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.244+0000 7f4b5dc00640 1 --2- 172.21.15.28:0/800625870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b3c004d40 0x7f4b3c0051c0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f4b500073e0 tx=0x7f4b3c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f906ff7a10c518d7 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:34.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.244+0000 7f4b57400640 1 -- 172.21.15.28:0/800625870 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b50039070 con 0x7f4b3c004d40 2024-02-21T23:08:34.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.244+0000 7f4b57400640 1 -- 172.21.15.28:0/800625870 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b5002cd30 con 0x7f4b3c004d40 2024-02-21T23:08:34.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.244+0000 7f4b57400640 1 -- 172.21.15.28:0/800625870 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b50009c80 con 0x7f4b3c004d40 2024-02-21T23:08:34.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.245+0000 7f4b5fc00640 1 -- 172.21.15.28:0/800625870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b3c004d40 msgr2=0x7f4b3c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:34.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.245+0000 7f4b5fc00640 1 --2- 172.21.15.28:0/800625870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b3c004d40 0x7f4b3c0051c0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f4b500073e0 tx=0x7f4b3c005830 comp rx=0 tx=0).stop 2024-02-21T23:08:34.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.245+0000 7f4b5dc00640 1 -- 172.21.15.28:0/800625870 reap_dead start 2024-02-21T23:08:34.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.245+0000 7f4b5fc00640 1 -- 172.21.15.28:0/800625870 shutdown_connections 2024-02-21T23:08:34.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.245+0000 7f4b5fc00640 1 -- 172.21.15.28:0/800625870 >> 172.21.15.28:0/800625870 conn(0x7f4b5808b1c0 msgr2=0x7f4b580725a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:34.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.245+0000 7f4b5fc00640 1 -- 172.21.15.28:0/800625870 shutdown_connections 2024-02-21T23:08:34.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.245+0000 7f4b5fc00640 1 -- 172.21.15.28:0/800625870 wait complete. 2024-02-21T23:08:34.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.246+0000 7f4b5fc00640 1 Processor -- start 2024-02-21T23:08:34.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.247+0000 7f4b5fc00640 1 -- start start 2024-02-21T23:08:34.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.247+0000 7f4b5fc00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4b58156440 0x7f4b58156820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:34.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.248+0000 7f4b5fc00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4b58138c30 0x7f4b58137c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:34.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.248+0000 7f4b5fc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b58138180 0x7f4b581edef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:34.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.248+0000 7f4b5fc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b5813ea00 con 0x7f4b58138180 2024-02-21T23:08:34.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.248+0000 7f4b5fc00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b58155cc0 con 0x7f4b58156440 2024-02-21T23:08:34.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.248+0000 7f4b5fc00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b580cafc0 con 0x7f4b58138c30 2024-02-21T23:08:34.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.248+0000 7f4b5dc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b58138180 0x7f4b581edef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:34.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.248+0000 7f4b5dc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b58138180 0x7f4b581edef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41462/0 (socket says 172.21.15.28:41462) 2024-02-21T23:08:34.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.248+0000 7f4b5dc00640 1 -- 172.21.15.28:0/4182907621 learned_addr learned my addr 172.21.15.28:0/4182907621 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:34.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.248+0000 7f4b57e00640 1 --2- 172.21.15.28:0/4182907621 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4b58138c30 0x7f4b58137c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:34.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.248+0000 7f4b5d200640 1 --2- 172.21.15.28:0/4182907621 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4b58156440 0x7f4b58156820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:34.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.248+0000 7f4b5dc00640 1 -- 172.21.15.28:0/4182907621 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4b58138c30 msgr2=0x7f4b58137c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:34.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.248+0000 7f4b5dc00640 1 --2- 172.21.15.28:0/4182907621 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4b58138c30 0x7f4b58137c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:34.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.248+0000 7f4b5dc00640 1 -- 172.21.15.28:0/4182907621 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4b58156440 msgr2=0x7f4b58156820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:34.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.248+0000 7f4b5dc00640 1 --2- 172.21.15.28:0/4182907621 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4b58156440 0x7f4b58156820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:34.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.248+0000 7f4b5dc00640 1 -- 172.21.15.28:0/4182907621 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b581ea9d0 con 0x7f4b58138180 2024-02-21T23:08:34.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.248+0000 7f4b5d200640 1 --2- 172.21.15.28:0/4182907621 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4b58156440 0x7f4b58156820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:08:34.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.249+0000 7f4b5dc00640 1 --2- 172.21.15.28:0/4182907621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b58138180 0x7f4b581edef0 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f4b50037040 tx=0x7f4b50004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:34.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.249+0000 7f4b54c00640 1 -- 172.21.15.28:0/4182907621 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b50039030 con 0x7f4b58138180 2024-02-21T23:08:34.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.249+0000 7f4b54c00640 1 -- 172.21.15.28:0/4182907621 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b5003a040 con 0x7f4b58138180 2024-02-21T23:08:34.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.249+0000 7f4b54c00640 1 -- 172.21.15.28:0/4182907621 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b50009c80 con 0x7f4b58138180 2024-02-21T23:08:34.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.249+0000 7f4b5fc00640 1 -- 172.21.15.28:0/4182907621 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b58138880 con 0x7f4b58138180 2024-02-21T23:08:34.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.249+0000 7f4b5fc00640 1 -- 172.21.15.28:0/4182907621 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b581eaf30 con 0x7f4b58138180 2024-02-21T23:08:34.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.250+0000 7f4b5fc00640 1 -- 172.21.15.28:0/4182907621 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b58067f30 con 0x7f4b58138180 2024-02-21T23:08:34.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.251+0000 7f4b54c00640 1 -- 172.21.15.28:0/4182907621 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4b50045020 con 0x7f4b58138180 2024-02-21T23:08:34.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.252+0000 7f4b54c00640 1 --2- 172.21.15.28:0/4182907621 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4b3806b4d0 0x7f4b3806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:34.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.252+0000 7f4b54c00640 1 -- 172.21.15.28:0/4182907621 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f4b500abcd0 con 0x7f4b58138180 2024-02-21T23:08:34.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.253+0000 7f4b5d200640 1 --2- 172.21.15.28:0/4182907621 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4b3806b4d0 0x7f4b3806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:34.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.254+0000 7f4b5d200640 1 --2- 172.21.15.28:0/4182907621 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4b3806b4d0 0x7f4b3806d990 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f4b48001c00 tx=0x7f4b48000f90 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:34.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.257+0000 7f4b54c00640 1 -- 172.21.15.28:0/4182907621 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4b581ea9d0 con 0x7f4b58138180 2024-02-21T23:08:34.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.500+0000 7f4b5fc00640 1 -- 172.21.15.28:0/4182907621 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b580caa90 con 0x7f4b58138180 2024-02-21T23:08:34.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.501+0000 7f4b54c00640 1 -- 172.21.15.28:0/4182907621 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 128 v128) v1 ==== 95+0+4950 (secure 0 0 0) 0x7f4b5007a900 con 0x7f4b58138180 2024-02-21T23:08:34.490 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:08:34.490 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":128,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5948,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3556518453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3556518453},{"type":"v1","addr":"172.21.15.28:6835","nonce":3556518453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116},{"gid":5951,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/3293264689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":3293264689},{"type":"v1","addr":"172.21.15.81:6836","nonce":3293264689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116},{"gid":8157,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/1285325008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1285325008},{"type":"v1","addr":"172.21.15.28:6834","nonce":1285325008}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":128}],"filesystems":[{"mdsmap":{"epoch":128,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:08:13.883141+0000","modified":"2024-02-21T23:08:32.529536+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":124,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6634},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6634":{"gid":6634,"name":"d","rank":0,"incarnation":127,"state":"up:reconnect","state_seq":2,"addr":"172.21.15.81:6837/3687210533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":3687210533},{"type":"v1","addr":"172.21.15.81:6837","nonce":3687210533}]},"join_fscid":9,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-21T23:08:34.490 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 128 2024-02-21T23:08:34.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.503+0000 7f4b5fc00640 1 -- 172.21.15.28:0/4182907621 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4b3806b4d0 msgr2=0x7f4b3806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:34.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.503+0000 7f4b5fc00640 1 --2- 172.21.15.28:0/4182907621 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4b3806b4d0 0x7f4b3806d990 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f4b48001c00 tx=0x7f4b48000f90 comp rx=0 tx=0).stop 2024-02-21T23:08:34.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.503+0000 7f4b5fc00640 1 -- 172.21.15.28:0/4182907621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b58138180 msgr2=0x7f4b581edef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:34.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.503+0000 7f4b5fc00640 1 --2- 172.21.15.28:0/4182907621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b58138180 0x7f4b581edef0 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f4b50037040 tx=0x7f4b50004ab0 comp rx=0 tx=0).stop 2024-02-21T23:08:34.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.504+0000 7f4b5fc00640 1 -- 172.21.15.28:0/4182907621 shutdown_connections 2024-02-21T23:08:34.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.504+0000 7f4b5fc00640 1 --2- 172.21.15.28:0/4182907621 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4b3806b4d0 0x7f4b3806d990 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:34.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.504+0000 7f4b5fc00640 1 --2- 172.21.15.28:0/4182907621 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4b58138c30 0x7f4b58137c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:34.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.504+0000 7f4b5fc00640 1 --2- 172.21.15.28:0/4182907621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b58138180 0x7f4b581edef0 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:34.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.504+0000 7f4b5fc00640 1 --2- 172.21.15.28:0/4182907621 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4b58156440 0x7f4b58156820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:34.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.504+0000 7f4b5fc00640 1 -- 172.21.15.28:0/4182907621 >> 172.21.15.28:0/4182907621 conn(0x7f4b5808b1c0 msgr2=0x7f4b581533f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:34.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.504+0000 7f4b5fc00640 1 -- 172.21.15.28:0/4182907621 shutdown_connections 2024-02-21T23:08:34.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:34.504+0000 7f4b5fc00640 1 -- 172.21.15.28:0/4182907621 wait complete. 2024-02-21T23:08:34.923 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 128, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:08:13.883141+0000', 'modified': '2024-02-21T23:08:32.529536+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 124, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6634}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6634': {'gid': 6634, 'name': 'd', 'rank': 0, 'incarnation': 127, 'state': 'up:reconnect', 'state_seq': 2, 'addr': '172.21.15.81:6837/3687210533', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6835', 'nonce': 3687210533}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3687210533}]}, 'join_fscid': 9, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [20], 'metadata_pool': 19, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:08:34.924 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6634:up:reconnect 2024-02-21T23:08:35.925 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:08:36.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:36.167+0000 7f7de0400640 1 Processor -- start 2024-02-21T23:08:36.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:36.167+0000 7f7de0400640 1 -- start start 2024-02-21T23:08:36.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:36.168+0000 7f7de0400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7dd80c11f0 0x7f7dd8153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:36.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:36.168+0000 7f7de0400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dd8152e70 con 0x7f7dd80c15d0 2024-02-21T23:08:36.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:36.168+0000 7f7de0400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dd8153150 con 0x7f7dd815b6d0 2024-02-21T23:08:36.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:36.168+0000 7f7de0400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dd80c1a60 con 0x7f7dd80c11f0 2024-02-21T23:08:36.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:36.169+0000 7f7ddda00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7dd80c11f0 0x7f7dd8153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:36.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:36.169+0000 7f7ddd000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f7dd80c15d0 0x7f7dd806dce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:51936/0 (socket says 172.21.15.28:51936) 2024-02-21T23:08:36.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:36.169+0000 7f7ddd000640 1 -- 172.21.15.28:0/2725926646 learned_addr learned my addr 172.21.15.28:0/2725926646 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:08:36.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:36.169+0000 7f7ddda00640 1 -- 172.21.15.28:0/2725926646 >> v1:172.21.15.81:6789/0 conn(0x7f7dd815b6d0 legacy=0x7f7dd815dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:36.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:36.169+0000 7f7ddda00640 1 -- 172.21.15.28:0/2725926646 >> v1:172.21.15.28:6789/0 conn(0x7f7dd80c15d0 legacy=0x7f7dd806dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:36.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:36.169+0000 7f7ddda00640 1 -- 172.21.15.28:0/2725926646 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dd8153430 con 0x7f7dd80c11f0 2024-02-21T23:08:36.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:36.169+0000 7f7dd7e00640 1 -- 172.21.15.28:0/2725926646 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2477436329 0 0) 0x7f7dd8152e70 con 0x7f7dd80c15d0 2024-02-21T23:08:36.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:36.170+0000 7f7ddda00640 1 --2- 172.21.15.28:0/2725926646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dd80c11f0 0x7f7dd8153950 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f7dc80097a0 tx=0x7f7dc802b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=721b5cfcfc80156c server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:36.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:36.171+0000 7f7dd7e00640 1 -- 172.21.15.28:0/2725926646 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dc8038070 con 0x7f7dd80c11f0 2024-02-21T23:08:36.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:36.171+0000 7f7dd7e00640 1 -- 172.21.15.28:0/2725926646 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dc8031700 con 0x7f7dd80c11f0 2024-02-21T23:08:36.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:36.171+0000 7f7dd7e00640 1 -- 172.21.15.28:0/2725926646 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dc8031ad0 con 0x7f7dd80c11f0 2024-02-21T23:08:36.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:36.171+0000 7f7de0400640 1 -- 172.21.15.28:0/2725926646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dd80c11f0 msgr2=0x7f7dd8153950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:36.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:36.171+0000 7f7de0400640 1 --2- 172.21.15.28:0/2725926646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dd80c11f0 0x7f7dd8153950 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f7dc80097a0 tx=0x7f7dc802b7a0 comp rx=0 tx=0).stop 2024-02-21T23:08:36.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:36.171+0000 7f7de0400640 1 -- 172.21.15.28:0/2725926646 shutdown_connections 2024-02-21T23:08:36.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:36.171+0000 7f7de0400640 1 --2- 172.21.15.28:0/2725926646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dd80c11f0 0x7f7dd8153950 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:36.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:36.171+0000 7f7de0400640 1 -- 172.21.15.28:0/2725926646 >> 172.21.15.28:0/2725926646 conn(0x7f7dd80c67f0 msgr2=0x7f7dd80c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:36.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:36.171+0000 7f7de0400640 1 -- 172.21.15.28:0/2725926646 shutdown_connections 2024-02-21T23:08:36.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:36.171+0000 7f7de0400640 1 -- 172.21.15.28:0/2725926646 wait complete. 2024-02-21T23:08:36.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:36.173+0000 7f7de0400640 1 Processor -- start 2024-02-21T23:08:36.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:36.173+0000 7f7de0400640 1 -- start start 2024-02-21T23:08:36.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:36.174+0000 7f7de0400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dd80c11f0 0x7f7dd80bfe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:36.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:36.174+0000 7f7de0400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7dd80c15d0 0x7f7dd8071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:36.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:36.174+0000 7f7de0400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7dd815b6d0 0x7f7dd80718c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:36.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:36.174+0000 7f7de0400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd80c4c00 con 0x7f7dd80c11f0 2024-02-21T23:08:36.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:36.174+0000 7f7de0400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd80a5ec0 con 0x7f7dd815b6d0 2024-02-21T23:08:36.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:36.174+0000 7f7de0400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd80a6110 con 0x7f7dd80c15d0 2024-02-21T23:08:36.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:36.174+0000 7f7ddda00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dd80c11f0 0x7f7dd80bfe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:36.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:36.174+0000 7f7ddda00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dd80c11f0 0x7f7dd80bfe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35772/0 (socket says 172.21.15.28:35772) 2024-02-21T23:08:36.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:36.174+0000 7f7ddda00640 1 -- 172.21.15.28:0/637322574 learned_addr learned my addr 172.21.15.28:0/637322574 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:36.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:36.175+0000 7f7ddd000640 1 --2- 172.21.15.28:0/637322574 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7dd80c15d0 0x7f7dd8071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:36.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:36.175+0000 7f7dde400640 1 --2- 172.21.15.28:0/637322574 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7dd815b6d0 0x7f7dd80718c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:36.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:36.175+0000 7f7ddda00640 1 -- 172.21.15.28:0/637322574 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7dd80c15d0 msgr2=0x7f7dd8071380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:36.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:36.175+0000 7f7ddda00640 1 --2- 172.21.15.28:0/637322574 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7dd80c15d0 0x7f7dd8071380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:36.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:36.175+0000 7f7ddda00640 1 -- 172.21.15.28:0/637322574 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7dd815b6d0 msgr2=0x7f7dd80718c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:36.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:36.175+0000 7f7ddda00640 1 --2- 172.21.15.28:0/637322574 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7dd815b6d0 0x7f7dd80718c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:36.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:36.175+0000 7f7ddda00640 1 -- 172.21.15.28:0/637322574 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dd81ec060 con 0x7f7dd80c11f0 2024-02-21T23:08:36.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:36.175+0000 7f7ddd000640 1 --2- 172.21.15.28:0/637322574 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7dd80c15d0 0x7f7dd8071380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:08:36.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:36.175+0000 7f7ddda00640 1 --2- 172.21.15.28:0/637322574 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dd80c11f0 0x7f7dd80bfe00 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f7dc80097a0 tx=0x7f7dc80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:36.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:36.175+0000 7f7dd5600640 1 -- 172.21.15.28:0/637322574 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dc8043070 con 0x7f7dd80c11f0 2024-02-21T23:08:36.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:36.176+0000 7f7dd5600640 1 -- 172.21.15.28:0/637322574 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dc8033a30 con 0x7f7dd80c11f0 2024-02-21T23:08:36.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:36.176+0000 7f7dd5600640 1 -- 172.21.15.28:0/637322574 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dc8038040 con 0x7f7dd80c11f0 2024-02-21T23:08:36.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:36.176+0000 7f7de0400640 1 -- 172.21.15.28:0/637322574 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dd81eb2a0 con 0x7f7dd80c11f0 2024-02-21T23:08:36.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:36.176+0000 7f7de0400640 1 -- 172.21.15.28:0/637322574 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dd81eb860 con 0x7f7dd80c11f0 2024-02-21T23:08:36.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:36.177+0000 7f7de0400640 1 -- 172.21.15.28:0/637322574 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dd80c7100 con 0x7f7dd80c11f0 2024-02-21T23:08:36.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:36.178+0000 7f7dd5600640 1 -- 172.21.15.28:0/637322574 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7dc8046070 con 0x7f7dd80c11f0 2024-02-21T23:08:36.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:36.179+0000 7f7dd5600640 1 --2- 172.21.15.28:0/637322574 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7dbc06b4d0 0x7f7dbc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:36.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:36.179+0000 7f7dd5600640 1 -- 172.21.15.28:0/637322574 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f7dc80ac290 con 0x7f7dd80c11f0 2024-02-21T23:08:36.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:36.180+0000 7f7ddd000640 1 --2- 172.21.15.28:0/637322574 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7dbc06b4d0 0x7f7dbc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:36.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:36.181+0000 7f7ddd000640 1 --2- 172.21.15.28:0/637322574 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7dbc06b4d0 0x7f7dbc06d990 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc007fd0 tx=0x7f7dcc009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:36.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:36.183+0000 7f7dd5600640 1 -- 172.21.15.28:0/637322574 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7dd80c7100 con 0x7f7dd80c11f0 2024-02-21T23:08:36.204 INFO:tasks.ceph.mds.d.smithi081.stderr:2024-02-21T23:08:36.216+0000 7f8efe800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-21T23:08:36.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:36.423+0000 7f7de0400640 1 -- 172.21.15.28:0/637322574 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7dd81ebca0 con 0x7f7dd80c11f0 2024-02-21T23:08:36.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:36.424+0000 7f7dd5600640 1 -- 172.21.15.28:0/637322574 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 130 v130) v1 ==== 95+0+4947 (secure 0 0 0) 0x7f7dc8030070 con 0x7f7dd80c11f0 2024-02-21T23:08:36.412 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:08:36.412 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":130,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5948,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3556518453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3556518453},{"type":"v1","addr":"172.21.15.28:6835","nonce":3556518453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116},{"gid":5951,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/3293264689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":3293264689},{"type":"v1","addr":"172.21.15.81:6836","nonce":3293264689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116},{"gid":8157,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/1285325008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1285325008},{"type":"v1","addr":"172.21.15.28:6834","nonce":1285325008}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":128}],"filesystems":[{"mdsmap":{"epoch":130,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:08:13.883141+0000","modified":"2024-02-21T23:08:34.520926+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":124,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6634},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6634":{"gid":6634,"name":"d","rank":0,"incarnation":127,"state":"up:active","state_seq":4,"addr":"172.21.15.81:6837/3687210533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":3687210533},{"type":"v1","addr":"172.21.15.81:6837","nonce":3687210533}]},"join_fscid":9,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-21T23:08:36.412 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 130 2024-02-21T23:08:36.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:36.427+0000 7f7de0400640 1 -- 172.21.15.28:0/637322574 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7dbc06b4d0 msgr2=0x7f7dbc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:36.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:36.427+0000 7f7de0400640 1 --2- 172.21.15.28:0/637322574 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7dbc06b4d0 0x7f7dbc06d990 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc007fd0 tx=0x7f7dcc009040 comp rx=0 tx=0).stop 2024-02-21T23:08:36.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:36.427+0000 7f7de0400640 1 -- 172.21.15.28:0/637322574 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dd80c11f0 msgr2=0x7f7dd80bfe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:36.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:36.427+0000 7f7de0400640 1 --2- 172.21.15.28:0/637322574 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dd80c11f0 0x7f7dd80bfe00 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f7dc80097a0 tx=0x7f7dc80046c0 comp rx=0 tx=0).stop 2024-02-21T23:08:36.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:36.427+0000 7f7de0400640 1 -- 172.21.15.28:0/637322574 shutdown_connections 2024-02-21T23:08:36.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:36.427+0000 7f7de0400640 1 --2- 172.21.15.28:0/637322574 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7dbc06b4d0 0x7f7dbc06d990 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:36.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:36.427+0000 7f7de0400640 1 --2- 172.21.15.28:0/637322574 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7dd80c15d0 0x7f7dd8071380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:36.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:36.427+0000 7f7de0400640 1 --2- 172.21.15.28:0/637322574 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7dd815b6d0 0x7f7dd80718c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:36.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:36.427+0000 7f7de0400640 1 --2- 172.21.15.28:0/637322574 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dd80c11f0 0x7f7dd80bfe00 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:36.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:36.427+0000 7f7de0400640 1 -- 172.21.15.28:0/637322574 >> 172.21.15.28:0/637322574 conn(0x7f7dd80c67f0 msgr2=0x7f7dd8149e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:36.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:36.427+0000 7f7de0400640 1 -- 172.21.15.28:0/637322574 shutdown_connections 2024-02-21T23:08:36.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:36.427+0000 7f7de0400640 1 -- 172.21.15.28:0/637322574 wait complete. 2024-02-21T23:08:36.778 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-21T23:08:36.785 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-21T23:08:36.837 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 130, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:08:13.883141+0000', 'modified': '2024-02-21T23:08:34.520926+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 124, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6634}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6634': {'gid': 6634, 'name': 'd', 'rank': 0, 'incarnation': 127, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.81:6837/3687210533', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6835', 'nonce': 3687210533}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3687210533}]}, 'join_fscid': 9, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [20], 'metadata_pool': 19, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:08:36.837 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-21T23:08:36.837 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-02-21T23:08:37.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.067+0000 7f8bd9400640 1 Processor -- start 2024-02-21T23:08:37.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.068+0000 7f8bd9400640 1 -- start start 2024-02-21T23:08:37.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.069+0000 7f8bd9400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f8bd406d410 0x7f8bd40a8e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:37.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.069+0000 7f8bd9400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bd4149c10 con 0x7f8bd40c43e0 2024-02-21T23:08:37.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.069+0000 7f8bd9400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bd41537d0 con 0x7f8bd4149830 2024-02-21T23:08:37.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.069+0000 7f8bd9400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bd4153ab0 con 0x7f8bd406d410 2024-02-21T23:08:37.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.070+0000 7f8bd8a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f8bd40c43e0 0x7f8bd40c5a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:51944/0 (socket says 172.21.15.28:51944) 2024-02-21T23:08:37.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.070+0000 7f8bd8a00640 1 -- 172.21.15.28:0/4159587450 learned_addr learned my addr 172.21.15.28:0/4159587450 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:08:37.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.070+0000 7f8bcbe00640 1 --2- 172.21.15.28:0/4159587450 >> v2:172.21.15.81:3300/0 conn(0x7f8bd406d410 0x7f8bd40a8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:37.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.070+0000 7f8bd3400640 1 -- 172.21.15.28:0/4159587450 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3021902798 0 0) 0x7f8bd4149c10 con 0x7f8bd40c43e0 2024-02-21T23:08:37.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.070+0000 7f8bd3400640 1 -- 172.21.15.28:0/4159587450 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bb8003520 con 0x7f8bd40c43e0 2024-02-21T23:08:37.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.071+0000 7f8bd3400640 1 -- 172.21.15.28:0/4159587450 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4221950089 0 0) 0x7f8bb8003520 con 0x7f8bd40c43e0 2024-02-21T23:08:37.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.071+0000 7f8bd3400640 1 -- 172.21.15.28:0/4159587450 >> v2:172.21.15.81:3300/0 conn(0x7f8bd406d410 msgr2=0x7f8bd40a8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:37.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.071+0000 7f8bd3400640 1 --2- 172.21.15.28:0/4159587450 >> v2:172.21.15.81:3300/0 conn(0x7f8bd406d410 0x7f8bd40a8e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:37.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.071+0000 7f8bd3400640 1 -- 172.21.15.28:0/4159587450 >> v1:172.21.15.81:6789/0 conn(0x7f8bd4149830 legacy=0x7f8bd40c0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:37.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.071+0000 7f8bd3400640 1 -- 172.21.15.28:0/4159587450 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bd40c4fa0 con 0x7f8bd40c43e0 2024-02-21T23:08:37.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.071+0000 7f8bd3400640 1 -- 172.21.15.28:0/4159587450 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f8bcc0019f0 con 0x7f8bd40c43e0 2024-02-21T23:08:37.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.071+0000 7f8bd3400640 1 -- 172.21.15.28:0/4159587450 >> v1:172.21.15.28:6789/0 conn(0x7f8bd40c43e0 legacy=0x7f8bd40c5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:37.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.071+0000 7f8bd3400640 1 --2- 172.21.15.28:0/4159587450 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8bb8003a00 0x7f8bb8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:37.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.071+0000 7f8bd3400640 1 --2- 172.21.15.28:0/4159587450 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8bb8004800 0x7f8bb8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:37.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.071+0000 7f8bd3400640 1 --2- 172.21.15.28:0/4159587450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bb80051c0 0x7f8bb800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:37.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.071+0000 7f8bd3400640 1 -- 172.21.15.28:0/4159587450 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd4153ab0 con 0x7f8bb80051c0 2024-02-21T23:08:37.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.071+0000 7f8bd3400640 1 -- 172.21.15.28:0/4159587450 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bb800c0a0 con 0x7f8bb8004800 2024-02-21T23:08:37.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.071+0000 7f8bd3400640 1 -- 172.21.15.28:0/4159587450 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bb800c2a0 con 0x7f8bb8003a00 2024-02-21T23:08:37.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.071+0000 7f8bd8a00640 1 --2- 172.21.15.28:0/4159587450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bb80051c0 0x7f8bb800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:37.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.072+0000 7f8bcbe00640 1 --2- 172.21.15.28:0/4159587450 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8bb8004800 0x7f8bb8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:37.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.072+0000 7f8bd3e00640 1 --2- 172.21.15.28:0/4159587450 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8bb8003a00 0x7f8bb8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:37.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.072+0000 7f8bd8a00640 1 -- 172.21.15.28:0/4159587450 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8bb8003a00 msgr2=0x7f8bb8003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:37.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.072+0000 7f8bd8a00640 1 --2- 172.21.15.28:0/4159587450 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8bb8003a00 0x7f8bb8003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:37.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.072+0000 7f8bd8a00640 1 -- 172.21.15.28:0/4159587450 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8bb8004800 msgr2=0x7f8bb8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:37.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.072+0000 7f8bd8a00640 1 --2- 172.21.15.28:0/4159587450 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8bb8004800 0x7f8bb8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:37.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.072+0000 7f8bd8a00640 1 -- 172.21.15.28:0/4159587450 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bb8003520 con 0x7f8bb80051c0 2024-02-21T23:08:37.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.072+0000 7f8bd3e00640 1 --2- 172.21.15.28:0/4159587450 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8bb8003a00 0x7f8bb8003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:08:37.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.072+0000 7f8bcbe00640 1 --2- 172.21.15.28:0/4159587450 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8bb8004800 0x7f8bb8004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:08:37.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.073+0000 7f8bd8a00640 1 -- 172.21.15.28:0/4159587450 reap_dead start 2024-02-21T23:08:37.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.073+0000 7f8bd8a00640 1 --2- 172.21.15.28:0/4159587450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bb80051c0 0x7f8bb800b9c0 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc00a7b0 tx=0x7f8bb8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc853d584ceb25b6 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:37.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.073+0000 7f8bd3400640 1 -- 172.21.15.28:0/4159587450 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bcc00a3d0 con 0x7f8bb80051c0 2024-02-21T23:08:37.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.073+0000 7f8bd3400640 1 -- 172.21.15.28:0/4159587450 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bcc034650 con 0x7f8bb80051c0 2024-02-21T23:08:37.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.073+0000 7f8bd3400640 1 -- 172.21.15.28:0/4159587450 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bcc034a40 con 0x7f8bb80051c0 2024-02-21T23:08:37.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.074+0000 7f8bd9400640 1 -- 172.21.15.28:0/4159587450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bb80051c0 msgr2=0x7f8bb800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:37.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.074+0000 7f8bd9400640 1 --2- 172.21.15.28:0/4159587450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bb80051c0 0x7f8bb800b9c0 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc00a7b0 tx=0x7f8bb8005830 comp rx=0 tx=0).stop 2024-02-21T23:08:37.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.074+0000 7f8bd9400640 1 -- 172.21.15.28:0/4159587450 shutdown_connections 2024-02-21T23:08:37.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.074+0000 7f8bd9400640 1 --2- 172.21.15.28:0/4159587450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bb80051c0 0x7f8bb800b9c0 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:37.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.074+0000 7f8bd9400640 1 -- 172.21.15.28:0/4159587450 >> 172.21.15.28:0/4159587450 conn(0x7f8bd40a8320 msgr2=0x7f8bd40a8730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:37.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.074+0000 7f8bd9400640 1 -- 172.21.15.28:0/4159587450 shutdown_connections 2024-02-21T23:08:37.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.074+0000 7f8bd9400640 1 -- 172.21.15.28:0/4159587450 wait complete. 2024-02-21T23:08:37.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.076+0000 7f8bd9400640 1 Processor -- start 2024-02-21T23:08:37.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.076+0000 7f8bd9400640 1 -- start start 2024-02-21T23:08:37.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.077+0000 7f8bd9400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bb80051c0 0x7f8bd4072a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:37.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.077+0000 7f8bd9400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8bd41e04f0 0x7f8bd4144830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:37.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.077+0000 7f8bd9400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8bd4144d70 0x7f8bd41454c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:37.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.077+0000 7f8bd9400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd40a8e40 con 0x7f8bb80051c0 2024-02-21T23:08:37.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.077+0000 7f8bd9400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd40c5560 con 0x7f8bd4144d70 2024-02-21T23:08:37.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.077+0000 7f8bd9400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd40a8fc0 con 0x7f8bd41e04f0 2024-02-21T23:08:37.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.077+0000 7f8bd3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bb80051c0 0x7f8bd4072a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:37.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.077+0000 7f8bd3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bb80051c0 0x7f8bd4072a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35786/0 (socket says 172.21.15.28:35786) 2024-02-21T23:08:37.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.077+0000 7f8bd3e00640 1 -- 172.21.15.28:0/3503206408 learned_addr learned my addr 172.21.15.28:0/3503206408 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:37.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.077+0000 7f8bcbe00640 1 --2- 172.21.15.28:0/3503206408 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8bd41e04f0 0x7f8bd4144830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:37.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.077+0000 7f8bd8a00640 1 --2- 172.21.15.28:0/3503206408 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8bd4144d70 0x7f8bd41454c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:37.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.078+0000 7f8bd3e00640 1 -- 172.21.15.28:0/3503206408 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8bd41e04f0 msgr2=0x7f8bd4144830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:37.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.078+0000 7f8bd3e00640 1 --2- 172.21.15.28:0/3503206408 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8bd41e04f0 0x7f8bd4144830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:37.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.078+0000 7f8bd3e00640 1 -- 172.21.15.28:0/3503206408 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8bd4144d70 msgr2=0x7f8bd41454c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:37.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.078+0000 7f8bd3e00640 1 --2- 172.21.15.28:0/3503206408 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8bd4144d70 0x7f8bd41454c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:37.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.078+0000 7f8bd3e00640 1 -- 172.21.15.28:0/3503206408 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bd40aa6c0 con 0x7f8bb80051c0 2024-02-21T23:08:37.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.078+0000 7f8bcbe00640 1 --2- 172.21.15.28:0/3503206408 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8bd41e04f0 0x7f8bd4144830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:08:37.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.078+0000 7f8bd3e00640 1 --2- 172.21.15.28:0/3503206408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bb80051c0 0x7f8bd4072a80 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f8bc400d7a0 tx=0x7f8bc400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:37.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.078+0000 7f8bd0c00640 1 -- 172.21.15.28:0/3503206408 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bc4005270 con 0x7f8bb80051c0 2024-02-21T23:08:37.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.078+0000 7f8bd0c00640 1 -- 172.21.15.28:0/3503206408 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bc4017070 con 0x7f8bb80051c0 2024-02-21T23:08:37.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.078+0000 7f8bd0c00640 1 -- 172.21.15.28:0/3503206408 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bc400b6b0 con 0x7f8bb80051c0 2024-02-21T23:08:37.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.078+0000 7f8bd9400640 1 -- 172.21.15.28:0/3503206408 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bd40a9620 con 0x7f8bb80051c0 2024-02-21T23:08:37.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.078+0000 7f8bd9400640 1 -- 172.21.15.28:0/3503206408 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bd40a9be0 con 0x7f8bb80051c0 2024-02-21T23:08:37.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.080+0000 7f8bd0c00640 1 -- 172.21.15.28:0/3503206408 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8bc400b850 con 0x7f8bb80051c0 2024-02-21T23:08:37.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.081+0000 7f8bd0c00640 1 --2- 172.21.15.28:0/3503206408 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8bb006b500 0x7f8bb006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:37.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.082+0000 7f8bd0c00640 1 -- 172.21.15.28:0/3503206408 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f8bc408a2c0 con 0x7f8bb80051c0 2024-02-21T23:08:37.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.082+0000 7f8bcbe00640 1 --2- 172.21.15.28:0/3503206408 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8bb006b500 0x7f8bb006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:37.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.082+0000 7f8bdd963740 1 Processor -- start 2024-02-21T23:08:37.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.083+0000 7f8bdd963740 1 -- start start 2024-02-21T23:08:37.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.083+0000 7f8bcbe00640 1 --2- 172.21.15.28:0/3503206408 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8bb006b500 0x7f8bb006d9c0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f8bd41489f0 tx=0x7f8bbc008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:37.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.084+0000 7f8bdd963740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cdfc7e2020 0x55cdfc7e2400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:37.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.084+0000 7f8bdd963740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdfc7e3330 0x55cdfc81e170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:37.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.084+0000 7f8bdd963740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55cdfc81e6b0 0x55cdfc81c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:37.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.084+0000 7f8bdd963740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cdfc746470 con 0x55cdfc7e3330 2024-02-21T23:08:37.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.084+0000 7f8bdd963740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cdfc6efaf0 con 0x55cdfc7e2020 2024-02-21T23:08:37.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.084+0000 7f8bdd963740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cdfc742510 con 0x55cdfc81e6b0 2024-02-21T23:08:37.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.084+0000 7f8bcbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdfc7e3330 0x55cdfc81e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:37.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.084+0000 7f8bcbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdfc7e3330 0x55cdfc81e170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35792/0 (socket says 172.21.15.28:35792) 2024-02-21T23:08:37.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.084+0000 7f8bcbe00640 1 -- 172.21.15.28:0/4021758223 learned_addr learned my addr 172.21.15.28:0/4021758223 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:37.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.084+0000 7f8bd3e00640 1 --2- 172.21.15.28:0/4021758223 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cdfc7e2020 0x55cdfc7e2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:37.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.084+0000 7f8bd8a00640 1 --2- 172.21.15.28:0/4021758223 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55cdfc81e6b0 0x55cdfc81c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:37.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.084+0000 7f8bcbe00640 1 -- 172.21.15.28:0/4021758223 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55cdfc81e6b0 msgr2=0x55cdfc81c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:37.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.084+0000 7f8bcbe00640 1 --2- 172.21.15.28:0/4021758223 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55cdfc81e6b0 0x55cdfc81c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:37.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.084+0000 7f8bcbe00640 1 -- 172.21.15.28:0/4021758223 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cdfc7e2020 msgr2=0x55cdfc7e2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:37.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.084+0000 7f8bcbe00640 1 --2- 172.21.15.28:0/4021758223 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cdfc7e2020 0x55cdfc7e2400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:37.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.084+0000 7f8bcbe00640 1 -- 172.21.15.28:0/4021758223 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cdfc6f6d90 con 0x55cdfc7e3330 2024-02-21T23:08:37.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.084+0000 7f8bd3e00640 1 --2- 172.21.15.28:0/4021758223 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cdfc7e2020 0x55cdfc7e2400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:08:37.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.085+0000 7f8bcbe00640 1 --2- 172.21.15.28:0/4021758223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdfc7e3330 0x55cdfc81e170 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f8bbc011e70 tx=0x7f8bbc014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb3be668e63bb64f server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:37.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.085+0000 7f8bc3e00640 1 -- 172.21.15.28:0/4021758223 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bbc015e50 con 0x55cdfc7e3330 2024-02-21T23:08:37.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.085+0000 7f8bc3e00640 1 -- 172.21.15.28:0/4021758223 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bbc026070 con 0x55cdfc7e3330 2024-02-21T23:08:37.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.085+0000 7f8bc3e00640 1 -- 172.21.15.28:0/4021758223 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bbc021640 con 0x55cdfc7e3330 2024-02-21T23:08:37.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.086+0000 7f8bdd963740 1 -- 172.21.15.28:0/4021758223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdfc7e3330 msgr2=0x55cdfc81e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:37.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.086+0000 7f8bdd963740 1 --2- 172.21.15.28:0/4021758223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdfc7e3330 0x55cdfc81e170 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f8bbc011e70 tx=0x7f8bbc014000 comp rx=0 tx=0).stop 2024-02-21T23:08:37.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.086+0000 7f8bdd963740 1 -- 172.21.15.28:0/4021758223 shutdown_connections 2024-02-21T23:08:37.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.086+0000 7f8bdd963740 1 --2- 172.21.15.28:0/4021758223 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55cdfc81e6b0 0x55cdfc81c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:37.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.086+0000 7f8bdd963740 1 --2- 172.21.15.28:0/4021758223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdfc7e3330 0x55cdfc81e170 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:37.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.086+0000 7f8bdd963740 1 --2- 172.21.15.28:0/4021758223 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cdfc7e2020 0x55cdfc7e2400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:37.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.086+0000 7f8bdd963740 1 -- 172.21.15.28:0/4021758223 >> 172.21.15.28:0/4021758223 conn(0x55cdfc7de970 msgr2=0x55cdfc803380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:37.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.086+0000 7f8bdd963740 1 -- 172.21.15.28:0/4021758223 shutdown_connections 2024-02-21T23:08:37.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.086+0000 7f8bdd963740 1 -- 172.21.15.28:0/4021758223 wait complete. 2024-02-21T23:08:37.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.087+0000 7f8bdd963740 1 -- start start 2024-02-21T23:08:37.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.087+0000 7f8bdd963740 1 Processor -- start 2024-02-21T23:08:37.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.088+0000 7f8bdd963740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cdfc7e3330 0x55cdfc804f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:37.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.088+0000 7f8bdd963740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55cdfc81e6b0 0x55cdfc831a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:37.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.088+0000 7f8bdd963740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdfc7fe780 0x55cdfc824850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:37.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.088+0000 7f8bdd963740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cdfc6e5040 con 0x55cdfc7fe780 2024-02-21T23:08:37.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.088+0000 7f8bdd963740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cdfc803a90 con 0x55cdfc7e3330 2024-02-21T23:08:37.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.088+0000 7f8bdd963740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cdfc825320 con 0x55cdfc81e6b0 2024-02-21T23:08:37.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.088+0000 7f8bdd963740 10 client.0 _ll_register_callbacks cb 0x55cdfc702920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:08:37.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.088+0000 7f8bd8a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdfc7fe780 0x55cdfc824850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:37.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.088+0000 7f8bd8a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdfc7fe780 0x55cdfc824850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35800/0 (socket says 172.21.15.28:35800) 2024-02-21T23:08:37.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.088+0000 7f8bd8a00640 1 -- 172.21.15.28:0/597030831 learned_addr learned my addr 172.21.15.28:0/597030831 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:37.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.088+0000 7f8bd3e00640 1 --2- 172.21.15.28:0/597030831 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cdfc7e3330 0x55cdfc804f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:37.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.088+0000 7f8bcbe00640 1 --2- 172.21.15.28:0/597030831 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55cdfc81e6b0 0x55cdfc831a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:37.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.089+0000 7f8bd8a00640 1 -- 172.21.15.28:0/597030831 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55cdfc81e6b0 msgr2=0x55cdfc831a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:37.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.089+0000 7f8bd8a00640 1 --2- 172.21.15.28:0/597030831 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55cdfc81e6b0 0x55cdfc831a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:37.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.089+0000 7f8bd8a00640 1 -- 172.21.15.28:0/597030831 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cdfc7e3330 msgr2=0x55cdfc804f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:37.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.089+0000 7f8bd8a00640 1 --2- 172.21.15.28:0/597030831 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cdfc7e3330 0x55cdfc804f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:37.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.089+0000 7f8bd8a00640 1 -- 172.21.15.28:0/597030831 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cdfc704aa0 con 0x55cdfc7fe780 2024-02-21T23:08:37.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.089+0000 7f8bd8a00640 1 --2- 172.21.15.28:0/597030831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdfc7fe780 0x55cdfc824850 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc00a8d0 tx=0x7f8bcc02e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8863a2d3e921e4ba server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:37.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.089+0000 7f8bc0c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:08:37.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.089+0000 7f8bc0c00640 1 -- 172.21.15.28:0/597030831 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bcc046060 con 0x55cdfc7fe780 2024-02-21T23:08:37.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.089+0000 7f8bc0c00640 1 -- 172.21.15.28:0/597030831 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(124..124 src has 1..124) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f8bcc004b60 con 0x55cdfc7fe780 2024-02-21T23:08:37.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.089+0000 7f8bdd963740 1 -- 172.21.15.28:0/597030831 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8bbc026070 con 0x55cdfc7fe780 2024-02-21T23:08:37.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.090+0000 7f8bc0c00640 1 -- 172.21.15.28:0/597030831 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=130) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8bcc039040 con 0x55cdfc7fe780 2024-02-21T23:08:37.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.090+0000 7f8bdd963740 10 client.8184 fetch_fsmap learned FSMap version 130 2024-02-21T23:08:37.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.090+0000 7f8bdd963740 1 -- 172.21.15.28:0/597030831 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=130,monmap=2+}) v3 -- 0x55cdfc45b2d0 con 0x55cdfc7fe780 2024-02-21T23:08:37.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.090+0000 7f8bc0c00640 1 -- 172.21.15.28:0/597030831 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bbc026070 con 0x55cdfc7fe780 2024-02-21T23:08:37.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.090+0000 7f8bc0c00640 1 -- 172.21.15.28:0/597030831 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 130) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f8bcc04f080 con 0x55cdfc7fe780 2024-02-21T23:08:37.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.090+0000 7f8bdd963740 10 client.8184 fetch_fsmap finished waiting for FSMap version 130 2024-02-21T23:08:37.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.090+0000 7f8bdd963740 10 client.8184 resolve_mds: resolved name 'd' to daemon mds.d 2024-02-21T23:08:37.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.090+0000 7f8bdd963740 20 client.8184 populate_metadata read hostname 'smithi028' 2024-02-21T23:08:37.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.090+0000 7f8bdd963740 1 --2- 172.21.15.28:0/597030831 >> [v2:172.21.15.81:6835/3687210533,v1:172.21.15.81:6837/3687210533] conn(0x55cdfc7fd9f0 0x55cdfc82e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:37.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.090+0000 7f8bdd963740 4 client.8184 mds_command: new command op to 6634 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-21T23:08:37.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.090+0000 7f8bdd963740 1 -- 172.21.15.28:0/597030831 --> [v2:172.21.15.81:6835/3687210533,v1:172.21.15.81:6837/3687210533] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cdfc7539b0 con 0x55cdfc7fd9f0 2024-02-21T23:08:37.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.091+0000 7f8bd3e00640 1 --2- 172.21.15.28:0/597030831 >> [v2:172.21.15.81:6835/3687210533,v1:172.21.15.81:6837/3687210533] conn(0x55cdfc7fd9f0 0x55cdfc82e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:37.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.092+0000 7f8bd3e00640 1 --2- 172.21.15.28:0/597030831 >> [v2:172.21.15.81:6835/3687210533,v1:172.21.15.81:6837/3687210533] conn(0x55cdfc7fd9f0 0x55cdfc82e800 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c09d12348dbb76bd server_cookie=65c5157ac4f7958b in_seq=0 out_seq=0 2024-02-21T23:08:37.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.092+0000 7f8bc0c00640 10 client.8184 ms_handle_connect on v2:172.21.15.81:6835/3687210533 2024-02-21T23:08:37.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.094+0000 7f8bc0c00640 1 -- 172.21.15.28:0/597030831 <== mds.0 v2:172.21.15.81:6835/3687210533 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55cdfc7539b0 con 0x55cdfc7fd9f0 2024-02-21T23:08:37.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.094+0000 7f8bc0c00640 10 client.8184 handle_command_reply: tid=0 2024-02-21T23:08:37.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.094+0000 7f8bdd963740 1 client.8184 shutdown 2024-02-21T23:08:37.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.094+0000 7f8bdd963740 1 -- 172.21.15.28:0/597030831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdfc7fe780 msgr2=0x55cdfc824850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:37.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.094+0000 7f8bdd963740 1 --2- 172.21.15.28:0/597030831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdfc7fe780 0x55cdfc824850 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc00a8d0 tx=0x7f8bcc02e000 comp rx=0 tx=0).stop 2024-02-21T23:08:37.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.094+0000 7f8bdd963740 1 -- 172.21.15.28:0/597030831 shutdown_connections 2024-02-21T23:08:37.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.094+0000 7f8bdd963740 1 --2- 172.21.15.28:0/597030831 >> [v2:172.21.15.81:6835/3687210533,v1:172.21.15.81:6837/3687210533] conn(0x55cdfc7fd9f0 0x55cdfc82e800 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:37.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.094+0000 7f8bdd963740 1 --2- 172.21.15.28:0/597030831 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55cdfc81e6b0 0x55cdfc831a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:37.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.094+0000 7f8bc0c00640 0 client.8184 ms_handle_reset on v2:172.21.15.81:6835/3687210533 2024-02-21T23:08:37.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.094+0000 7f8bdd963740 1 --2- 172.21.15.28:0/597030831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdfc7fe780 0x55cdfc824850 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:37.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.094+0000 7f8bdd963740 1 --2- 172.21.15.28:0/597030831 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cdfc7e3330 0x55cdfc804f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:37.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.094+0000 7f8bdd963740 1 -- 172.21.15.28:0/597030831 >> 172.21.15.28:0/597030831 conn(0x55cdfc7e2020 msgr2=0x55cdfc81cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:37.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.094+0000 7f8bdd963740 1 -- 172.21.15.28:0/597030831 shutdown_connections 2024-02-21T23:08:37.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.094+0000 7f8bdd963740 1 -- 172.21.15.28:0/597030831 wait complete. 2024-02-21T23:08:37.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.095+0000 7f8bdd963740 20 client.8184 trim_cache size 0 max 16384 2024-02-21T23:08:37.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.112+0000 7f8bdd963740 1 Processor -- start 2024-02-21T23:08:37.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.112+0000 7f8bdd963740 1 -- start start 2024-02-21T23:08:37.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.113+0000 7f8bdd963740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55cdfc7e3330 0x55cdfc82efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:37.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.113+0000 7f8bdd963740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cdfc7fe780 0x55cdfc7f6da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:37.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.113+0000 7f8bdd963740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdfc81e6b0 0x55cdfc81f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:37.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.113+0000 7f8bdd963740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd41f73d0 con 0x55cdfc81e6b0 2024-02-21T23:08:37.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.113+0000 7f8bdd963740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd41f6850 con 0x55cdfc7fe780 2024-02-21T23:08:37.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.113+0000 7f8bdd963740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd41f6e50 con 0x55cdfc7e3330 2024-02-21T23:08:37.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.114+0000 7f8bd8a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdfc81e6b0 0x55cdfc81f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:37.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.114+0000 7f8bd8a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdfc81e6b0 0x55cdfc81f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35802/0 (socket says 172.21.15.28:35802) 2024-02-21T23:08:37.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.114+0000 7f8bd8a00640 1 -- 172.21.15.28:0/1326448114 learned_addr learned my addr 172.21.15.28:0/1326448114 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:37.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.114+0000 7f8bcbe00640 1 --2- 172.21.15.28:0/1326448114 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cdfc7fe780 0x55cdfc7f6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:37.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.114+0000 7f8bd3e00640 1 --2- 172.21.15.28:0/1326448114 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55cdfc7e3330 0x55cdfc82efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:37.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.114+0000 7f8bd8a00640 1 -- 172.21.15.28:0/1326448114 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55cdfc7e3330 msgr2=0x55cdfc82efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:37.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.114+0000 7f8bd8a00640 1 --2- 172.21.15.28:0/1326448114 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55cdfc7e3330 0x55cdfc82efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:37.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.114+0000 7f8bd8a00640 1 -- 172.21.15.28:0/1326448114 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cdfc7fe780 msgr2=0x55cdfc7f6da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:37.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.114+0000 7f8bd8a00640 1 --2- 172.21.15.28:0/1326448114 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cdfc7fe780 0x55cdfc7f6da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:37.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.114+0000 7f8bd8a00640 1 -- 172.21.15.28:0/1326448114 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cdfc5256b0 con 0x55cdfc81e6b0 2024-02-21T23:08:37.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.115+0000 7f8bd8a00640 1 --2- 172.21.15.28:0/1326448114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdfc81e6b0 0x55cdfc81f720 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc000f80 tx=0x7f8bcc041000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e76cd4be9656ba16 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:37.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.115+0000 7f8bc3400640 1 -- 172.21.15.28:0/1326448114 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bcc046020 con 0x55cdfc81e6b0 2024-02-21T23:08:37.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.115+0000 7f8bc3400640 1 -- 172.21.15.28:0/1326448114 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bcc001730 con 0x55cdfc81e6b0 2024-02-21T23:08:37.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.115+0000 7f8bc3400640 1 -- 172.21.15.28:0/1326448114 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bcc053040 con 0x55cdfc81e6b0 2024-02-21T23:08:37.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.115+0000 7f8bdd963740 1 -- 172.21.15.28:0/1326448114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdfc81e6b0 msgr2=0x55cdfc81f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:37.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.115+0000 7f8bdd963740 1 --2- 172.21.15.28:0/1326448114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdfc81e6b0 0x55cdfc81f720 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc000f80 tx=0x7f8bcc041000 comp rx=0 tx=0).stop 2024-02-21T23:08:37.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.115+0000 7f8bdd963740 1 -- 172.21.15.28:0/1326448114 shutdown_connections 2024-02-21T23:08:37.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.115+0000 7f8bdd963740 1 --2- 172.21.15.28:0/1326448114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdfc81e6b0 0x55cdfc81f720 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:37.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.115+0000 7f8bdd963740 1 --2- 172.21.15.28:0/1326448114 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cdfc7fe780 0x55cdfc7f6da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:37.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.115+0000 7f8bdd963740 1 --2- 172.21.15.28:0/1326448114 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55cdfc7e3330 0x55cdfc82efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:37.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.115+0000 7f8bdd963740 1 -- 172.21.15.28:0/1326448114 >> 172.21.15.28:0/1326448114 conn(0x55cdfc7e2020 msgr2=0x55cdfc7d7d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:37.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.116+0000 7f8bdd963740 1 -- 172.21.15.28:0/1326448114 shutdown_connections 2024-02-21T23:08:37.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.116+0000 7f8bdd963740 1 -- 172.21.15.28:0/1326448114 wait complete. 2024-02-21T23:08:37.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.116+0000 7f8bdd963740 1 -- start start 2024-02-21T23:08:37.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.116+0000 7f8bdd963740 1 Processor -- start 2024-02-21T23:08:37.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.117+0000 7f8bdd963740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55cdfc7e3330 0x55cdfc7f04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:37.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.117+0000 7f8bdd963740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cdfc7fe780 0x55cdfc7ffe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:37.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.117+0000 7f8bdd963740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdfc81e6b0 0x55cdfc7fa1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:37.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.117+0000 7f8bdd963740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cdfc7fd870 con 0x55cdfc81e6b0 2024-02-21T23:08:37.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.117+0000 7f8bdd963740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cdfc832020 con 0x55cdfc7fe780 2024-02-21T23:08:37.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.117+0000 7f8bdd963740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cdfc82e320 con 0x55cdfc7e3330 2024-02-21T23:08:37.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.117+0000 7f8bdd963740 10 client.0 _ll_register_callbacks cb 0x7f8bbc009060 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:08:37.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.117+0000 7f8bd8a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdfc81e6b0 0x55cdfc7fa1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:37.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.118+0000 7f8bd8a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdfc81e6b0 0x55cdfc7fa1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35812/0 (socket says 172.21.15.28:35812) 2024-02-21T23:08:37.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.118+0000 7f8bd8a00640 1 -- 172.21.15.28:0/2142997433 learned_addr learned my addr 172.21.15.28:0/2142997433 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:37.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.118+0000 7f8bcbe00640 1 --2- 172.21.15.28:0/2142997433 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cdfc7fe780 0x55cdfc7ffe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:37.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.118+0000 7f8bd3e00640 1 --2- 172.21.15.28:0/2142997433 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55cdfc7e3330 0x55cdfc7f04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:37.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.118+0000 7f8bd8a00640 1 -- 172.21.15.28:0/2142997433 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55cdfc7e3330 msgr2=0x55cdfc7f04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:37.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.118+0000 7f8bd8a00640 1 --2- 172.21.15.28:0/2142997433 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55cdfc7e3330 0x55cdfc7f04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:37.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.118+0000 7f8bd8a00640 1 -- 172.21.15.28:0/2142997433 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cdfc7fe780 msgr2=0x55cdfc7ffe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:37.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.118+0000 7f8bd8a00640 1 --2- 172.21.15.28:0/2142997433 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cdfc7fe780 0x55cdfc7ffe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:37.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.118+0000 7f8bd8a00640 1 -- 172.21.15.28:0/2142997433 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cdfc800500 con 0x55cdfc81e6b0 2024-02-21T23:08:37.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.118+0000 7f8bcbe00640 1 --2- 172.21.15.28:0/2142997433 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cdfc7fe780 0x55cdfc7ffe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:08:37.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.118+0000 7f8bd8a00640 1 --2- 172.21.15.28:0/2142997433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdfc81e6b0 0x55cdfc7fa1d0 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc036040 tx=0x7f8bcc002160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9e5382b08040fe5 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:37.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.118+0000 7f8bc0c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:08:37.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.119+0000 7f8bc0c00640 1 -- 172.21.15.28:0/2142997433 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bcc053040 con 0x55cdfc81e6b0 2024-02-21T23:08:37.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.119+0000 7f8bc0c00640 1 -- 172.21.15.28:0/2142997433 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(124..124 src has 1..124) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f8bcc058470 con 0x55cdfc81e6b0 2024-02-21T23:08:37.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.119+0000 7f8bdd963740 1 -- 172.21.15.28:0/2142997433 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8bcc001730 con 0x55cdfc81e6b0 2024-02-21T23:08:37.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.119+0000 7f8bc0c00640 1 -- 172.21.15.28:0/2142997433 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=130) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8bcc067050 con 0x55cdfc81e6b0 2024-02-21T23:08:37.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.119+0000 7f8bdd963740 10 client.8190 fetch_fsmap learned FSMap version 130 2024-02-21T23:08:37.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.119+0000 7f8bdd963740 1 -- 172.21.15.28:0/2142997433 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=130,monmap=2+}) v3 -- 0x55cdfc81cf90 con 0x55cdfc81e6b0 2024-02-21T23:08:37.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.119+0000 7f8bc0c00640 1 -- 172.21.15.28:0/2142997433 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bcc001730 con 0x55cdfc81e6b0 2024-02-21T23:08:37.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.119+0000 7f8bc0c00640 1 -- 172.21.15.28:0/2142997433 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 130) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f8bcc06a020 con 0x55cdfc81e6b0 2024-02-21T23:08:37.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.119+0000 7f8bdd963740 10 client.8190 fetch_fsmap finished waiting for FSMap version 130 2024-02-21T23:08:37.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.119+0000 7f8bdd963740 10 client.8190 resolve_mds: resolved name 'd' to daemon mds.d 2024-02-21T23:08:37.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.119+0000 7f8bdd963740 20 client.8190 populate_metadata read hostname 'smithi028' 2024-02-21T23:08:37.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.119+0000 7f8bdd963740 1 --2- 172.21.15.28:0/2142997433 >> [v2:172.21.15.81:6835/3687210533,v1:172.21.15.81:6837/3687210533] conn(0x55cdfc893070 0x55cdfc840c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:37.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.119+0000 7f8bdd963740 4 client.8190 mds_command: new command op to 6634 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-21T23:08:37.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.119+0000 7f8bdd963740 1 -- 172.21.15.28:0/2142997433 --> [v2:172.21.15.81:6835/3687210533,v1:172.21.15.81:6837/3687210533] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55cdfc74dd00 con 0x55cdfc893070 2024-02-21T23:08:37.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.120+0000 7f8bd3e00640 1 --2- 172.21.15.28:0/2142997433 >> [v2:172.21.15.81:6835/3687210533,v1:172.21.15.81:6837/3687210533] conn(0x55cdfc893070 0x55cdfc840c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:37.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.121+0000 7f8bd3e00640 1 --2- 172.21.15.28:0/2142997433 >> [v2:172.21.15.81:6835/3687210533,v1:172.21.15.81:6837/3687210533] conn(0x55cdfc893070 0x55cdfc840c90 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7607c710191e7bf0 server_cookie=37c00c43e3cb1424 in_seq=0 out_seq=0 2024-02-21T23:08:37.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.121+0000 7f8bc0c00640 10 client.8190 ms_handle_connect on v2:172.21.15.81:6835/3687210533 2024-02-21T23:08:37.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.121+0000 7f8bc0c00640 1 -- 172.21.15.28:0/2142997433 <== mds.0 v2:172.21.15.81:6835/3687210533 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+334 (crc 0 0 0) 0x55cdfc74dd00 con 0x55cdfc893070 2024-02-21T23:08:37.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.121+0000 7f8bc0c00640 10 client.8190 handle_command_reply: tid=0 2024-02-21T23:08:37.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.121+0000 7f8bdd963740 1 client.8190 shutdown 2024-02-21T23:08:37.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.121+0000 7f8bdd963740 1 -- 172.21.15.28:0/2142997433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdfc81e6b0 msgr2=0x55cdfc7fa1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:37.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.121+0000 7f8bdd963740 1 --2- 172.21.15.28:0/2142997433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdfc81e6b0 0x55cdfc7fa1d0 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc036040 tx=0x7f8bcc002160 comp rx=0 tx=0).stop 2024-02-21T23:08:37.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.122+0000 7f8bdd963740 1 -- 172.21.15.28:0/2142997433 shutdown_connections 2024-02-21T23:08:37.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.122+0000 7f8bdd963740 1 --2- 172.21.15.28:0/2142997433 >> [v2:172.21.15.81:6835/3687210533,v1:172.21.15.81:6837/3687210533] conn(0x55cdfc893070 0x55cdfc840c90 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:37.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.122+0000 7f8bdd963740 1 --2- 172.21.15.28:0/2142997433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdfc81e6b0 0x55cdfc7fa1d0 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:37.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.122+0000 7f8bdd963740 1 --2- 172.21.15.28:0/2142997433 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cdfc7fe780 0x55cdfc7ffe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:37.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.122+0000 7f8bdd963740 1 --2- 172.21.15.28:0/2142997433 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55cdfc7e3330 0x55cdfc7f04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:37.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.122+0000 7f8bc0c00640 0 client.8190 ms_handle_reset on v2:172.21.15.81:6835/3687210533 2024-02-21T23:08:37.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.122+0000 7f8bdd963740 1 -- 172.21.15.28:0/2142997433 >> 172.21.15.28:0/2142997433 conn(0x55cdfc7e2020 msgr2=0x55cdfc82e650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:37.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.122+0000 7f8bdd963740 1 -- 172.21.15.28:0/2142997433 shutdown_connections 2024-02-21T23:08:37.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.122+0000 7f8bdd963740 1 -- 172.21.15.28:0/2142997433 wait complete. 2024-02-21T23:08:37.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.122+0000 7f8bdd963740 20 client.8190 trim_cache size 0 max 16384 2024-02-21T23:08:37.113 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-21T23:08:37.113 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac", 2024-02-21T23:08:37.113 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-02-21T23:08:37.113 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 6634, 2024-02-21T23:08:37.114 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-02-21T23:08:37.114 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-02-21T23:08:37.114 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-02-21T23:08:37.114 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 4.6102580870000001, 2024-02-21T23:08:37.114 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 130, 2024-02-21T23:08:37.114 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 124, 2024-02-21T23:08:37.114 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 124, 2024-02-21T23:08:37.114 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 6.8445189610000003 2024-02-21T23:08:37.114 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-21T23:08:37.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.122+0000 7f8bd9400640 1 -- 172.21.15.28:0/3503206408 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8bb006b500 msgr2=0x7f8bb006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:37.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.122+0000 7f8bd9400640 1 --2- 172.21.15.28:0/3503206408 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8bb006b500 0x7f8bb006d9c0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f8bd41489f0 tx=0x7f8bbc008040 comp rx=0 tx=0).stop 2024-02-21T23:08:37.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.123+0000 7f8bd9400640 1 -- 172.21.15.28:0/3503206408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bb80051c0 msgr2=0x7f8bd4072a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:37.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.123+0000 7f8bd9400640 1 --2- 172.21.15.28:0/3503206408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bb80051c0 0x7f8bd4072a80 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f8bc400d7a0 tx=0x7f8bc400dc60 comp rx=0 tx=0).stop 2024-02-21T23:08:37.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.123+0000 7f8bd9400640 1 -- 172.21.15.28:0/3503206408 shutdown_connections 2024-02-21T23:08:37.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.123+0000 7f8bd9400640 1 --2- 172.21.15.28:0/3503206408 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8bb006b500 0x7f8bb006d9c0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:37.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.123+0000 7f8bd9400640 1 --2- 172.21.15.28:0/3503206408 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8bd41e04f0 0x7f8bd4144830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:37.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.123+0000 7f8bd9400640 1 --2- 172.21.15.28:0/3503206408 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8bd4144d70 0x7f8bd41454c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:37.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.123+0000 7f8bd9400640 1 --2- 172.21.15.28:0/3503206408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bb80051c0 0x7f8bd4072a80 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:37.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.123+0000 7f8bd9400640 1 -- 172.21.15.28:0/3503206408 >> 172.21.15.28:0/3503206408 conn(0x7f8bd40a8320 msgr2=0x7f8bd4150bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:37.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.123+0000 7f8bd9400640 1 -- 172.21.15.28:0/3503206408 shutdown_connections 2024-02-21T23:08:37.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.123+0000 7f8bd9400640 1 -- 172.21.15.28:0/3503206408 wait complete. 2024-02-21T23:08:37.129 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 9, "epoch": 130, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 9, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-02-21T23:08:13.883141+0000", "damaged": [], "data_pools": [ 20 ], "enabled": true, "epoch": 130, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6634": { "addr": "172.21.15.81:6837/3687210533", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6835", "nonce": 3687210533, "type": "v2" }, { "addr": "172.21.15.81:6837", "nonce": 3687210533, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6634, "incarnation": 127, "join_fscid": 9, "name": "d", "rank": 0, "state": "up:active", "state_seq": 4 } }, "last_failure": 0, "last_failure_osd_epoch": 124, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 19, "modified": "2024-02-21T23:08:34.520926+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6634 } } } ], "standbys": [ { "addr": "172.21.15.28:6835/3556518453", "addrs": { "addrvec": [ { "addr": "172.21.15.28:6833", "nonce": 3556518453, "type": "v2" }, { "addr": "172.21.15.28:6835", "nonce": 3556518453, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 116, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5948, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.81:6836/3293264689", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6834", "nonce": 3293264689, "type": "v2" }, { "addr": "172.21.15.81:6836", "nonce": 3293264689, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 116, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5951, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.28:6834/1285325008", "addrs": { "addrvec": [ { "addr": "172.21.15.28:6832", "nonce": 1285325008, "type": "v2" }, { "addr": "172.21.15.28:6834", "nonce": 1285325008, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 128, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8157, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-02-21T23:08:37.131 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': 9, 'name': 'd'}] 2024-02-21T23:08:37.131 INFO:tasks.cephfs.test_failover:current = [{'gid': 8157, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.28:6834/1285325008', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 1285325008}, {'type': 'v1', 'addr': '172.21.15.28:6834', 'nonce': 1285325008}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 128}, {'gid': 5951, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.81:6836/3293264689', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6834', 'nonce': 3293264689}, {'type': 'v1', 'addr': '172.21.15.81:6836', 'nonce': 3293264689}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 116}, {'gid': 5948, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.28:6835/3556518453', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6833', 'nonce': 3556518453}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3556518453}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 116}, {'gid': 6634, 'name': 'd', 'rank': 0, 'incarnation': 127, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.81:6837/3687210533', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6835', 'nonce': 3687210533}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3687210533}]}, 'join_fscid': 9, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}] 2024-02-21T23:08:37.131 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-02-21T23:08:37.131 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-02-21T23:08:37.131 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-21T23:08:37.247 DEBUG:teuthology.orchestra.run.smithi081:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-21T23:08:37.351 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-02-21T23:08:37.351 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-21T23:08:37.351 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-21T23:08:37.412 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054608,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-02-21T23:08:37.413 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-02-21T23:08:37.413 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-21T23:08:37.464 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053833,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498812k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249404k,nr_inodes=812351,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498812k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-02-21T23:08:37.464 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:08:37.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.707+0000 7fe55f400640 1 Processor -- start 2024-02-21T23:08:37.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.707+0000 7fe55f400640 1 -- start start 2024-02-21T23:08:37.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.709+0000 7fe55f400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe5581528a0 0x7fe5580c6650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:37.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.709+0000 7fe55f400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe5580c6b90 0x7fe5580c7180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:37.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.709+0000 7fe55f400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe558152c80 con 0x7fe5580c76c0 2024-02-21T23:08:37.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.709+0000 7fe55f400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe558155300 con 0x7fe5580c6b90 2024-02-21T23:08:37.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.709+0000 7fe55f400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5581555e0 con 0x7fe5581528a0 2024-02-21T23:08:37.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.709+0000 7fe557e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe5580c6b90 0x7fe5580c7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:37.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.709+0000 7fe557e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe5580c6b90 0x7fe5580c7180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35822/0 (socket says 172.21.15.28:35822) 2024-02-21T23:08:37.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.709+0000 7fe557e00640 1 -- 172.21.15.28:0/1308995628 learned_addr learned my addr 172.21.15.28:0/1308995628 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:37.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.709+0000 7fe55ca00640 1 --2- 172.21.15.28:0/1308995628 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe5581528a0 0x7fe5580c6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:37.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.710+0000 7fe557e00640 1 -- 172.21.15.28:0/1308995628 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe5581528a0 msgr2=0x7fe5580c6650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:37.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.710+0000 7fe557e00640 1 --2- 172.21.15.28:0/1308995628 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe5581528a0 0x7fe5580c6650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:37.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.710+0000 7fe557e00640 1 -- 172.21.15.28:0/1308995628 >> v1:172.21.15.81:6789/0 conn(0x7fe5580c76c0 legacy=0x7fe5581587f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:37.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.710+0000 7fe557e00640 1 -- 172.21.15.28:0/1308995628 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5581558c0 con 0x7fe5580c6b90 2024-02-21T23:08:37.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.711+0000 7fe557e00640 1 --2- 172.21.15.28:0/1308995628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5580c6b90 0x7fe5580c7180 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7fe544009870 tx=0x7fe54402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a7bc463fd475451 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:37.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.711+0000 7fe557400640 1 -- 172.21.15.28:0/1308995628 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe544035020 con 0x7fe5580c6b90 2024-02-21T23:08:37.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.711+0000 7fe557400640 1 -- 172.21.15.28:0/1308995628 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe544031640 con 0x7fe5580c6b90 2024-02-21T23:08:37.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.711+0000 7fe557400640 1 -- 172.21.15.28:0/1308995628 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe54402d040 con 0x7fe5580c6b90 2024-02-21T23:08:37.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.712+0000 7fe55f400640 1 -- 172.21.15.28:0/1308995628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5580c6b90 msgr2=0x7fe5580c7180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:37.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.712+0000 7fe55f400640 1 --2- 172.21.15.28:0/1308995628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5580c6b90 0x7fe5580c7180 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7fe544009870 tx=0x7fe54402b8f0 comp rx=0 tx=0).stop 2024-02-21T23:08:37.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.712+0000 7fe55f400640 1 -- 172.21.15.28:0/1308995628 shutdown_connections 2024-02-21T23:08:37.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.712+0000 7fe55f400640 1 --2- 172.21.15.28:0/1308995628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5580c6b90 0x7fe5580c7180 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:37.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.712+0000 7fe55f400640 1 --2- 172.21.15.28:0/1308995628 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe5581528a0 0x7fe5580c6650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:37.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.712+0000 7fe55f400640 1 -- 172.21.15.28:0/1308995628 >> 172.21.15.28:0/1308995628 conn(0x7fe5580aa4c0 msgr2=0x7fe5580aa8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:37.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.712+0000 7fe55f400640 1 -- 172.21.15.28:0/1308995628 shutdown_connections 2024-02-21T23:08:37.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.712+0000 7fe55f400640 1 -- 172.21.15.28:0/1308995628 wait complete. 2024-02-21T23:08:37.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.714+0000 7fe55f400640 1 Processor -- start 2024-02-21T23:08:37.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.714+0000 7fe55f400640 1 -- start start 2024-02-21T23:08:37.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.715+0000 7fe55f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5580c6b90 0x7fe55814f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:37.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.715+0000 7fe55f400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe5580c76c0 0x7fe55814df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:37.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.715+0000 7fe55f400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe5581528a0 0x7fe55814ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:37.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.715+0000 7fe55f400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5580a8170 con 0x7fe5580c6b90 2024-02-21T23:08:37.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.715+0000 7fe55f400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe558155e80 con 0x7fe5580c76c0 2024-02-21T23:08:37.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.715+0000 7fe55f400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe558158f00 con 0x7fe5581528a0 2024-02-21T23:08:37.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.715+0000 7fe55ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5580c6b90 0x7fe55814f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:37.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.715+0000 7fe55ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5580c6b90 0x7fe55814f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35834/0 (socket says 172.21.15.28:35834) 2024-02-21T23:08:37.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.715+0000 7fe55ca00640 1 -- 172.21.15.28:0/1483431531 learned_addr learned my addr 172.21.15.28:0/1483431531 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:37.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.716+0000 7fe557e00640 1 --2- 172.21.15.28:0/1483431531 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe5580c76c0 0x7fe55814df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:37.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.716+0000 7fe55d400640 1 --2- 172.21.15.28:0/1483431531 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe5581528a0 0x7fe55814ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:37.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.716+0000 7fe55ca00640 1 -- 172.21.15.28:0/1483431531 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe5581528a0 msgr2=0x7fe55814ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:37.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.716+0000 7fe55ca00640 1 --2- 172.21.15.28:0/1483431531 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe5581528a0 0x7fe55814ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:37.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.716+0000 7fe55ca00640 1 -- 172.21.15.28:0/1483431531 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe5580c76c0 msgr2=0x7fe55814df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:37.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.716+0000 7fe55ca00640 1 --2- 172.21.15.28:0/1483431531 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe5580c76c0 0x7fe55814df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:37.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.716+0000 7fe55ca00640 1 -- 172.21.15.28:0/1483431531 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe55814bad0 con 0x7fe5580c6b90 2024-02-21T23:08:37.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.716+0000 7fe55d400640 1 --2- 172.21.15.28:0/1483431531 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe5581528a0 0x7fe55814ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:08:37.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.716+0000 7fe55ca00640 1 --2- 172.21.15.28:0/1483431531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5580c6b90 0x7fe55814f0c0 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7fe54c00d620 tx=0x7fe54c00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:37.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.716+0000 7fe554c00640 1 -- 172.21.15.28:0/1483431531 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe54c00b620 con 0x7fe5580c6b90 2024-02-21T23:08:37.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.717+0000 7fe554c00640 1 -- 172.21.15.28:0/1483431531 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe54c019070 con 0x7fe5580c6b90 2024-02-21T23:08:37.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.717+0000 7fe554c00640 1 -- 172.21.15.28:0/1483431531 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe54c00ba00 con 0x7fe5580c6b90 2024-02-21T23:08:37.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.717+0000 7fe55f400640 1 -- 172.21.15.28:0/1483431531 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe558157990 con 0x7fe5580c6b90 2024-02-21T23:08:37.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.717+0000 7fe55f400640 1 -- 172.21.15.28:0/1483431531 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe558157f50 con 0x7fe5580c6b90 2024-02-21T23:08:37.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.718+0000 7fe55f400640 1 -- 172.21.15.28:0/1483431531 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe558067f30 con 0x7fe5580c6b90 2024-02-21T23:08:37.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.719+0000 7fe554c00640 1 -- 172.21.15.28:0/1483431531 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe54c00bba0 con 0x7fe5580c6b90 2024-02-21T23:08:37.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.720+0000 7fe554c00640 1 --2- 172.21.15.28:0/1483431531 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe53806b4d0 0x7fe53806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:37.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.720+0000 7fe554c00640 1 -- 172.21.15.28:0/1483431531 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 6273+0+0 (secure 0 0 0) 0x7fe54c08a8c0 con 0x7fe5580c6b90 2024-02-21T23:08:37.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.721+0000 7fe557e00640 1 --2- 172.21.15.28:0/1483431531 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe53806b4d0 0x7fe53806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:37.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.722+0000 7fe557e00640 1 --2- 172.21.15.28:0/1483431531 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe53806b4d0 0x7fe53806d990 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fe544036000 tx=0x7fe54403c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:37.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.725+0000 7fe554c00640 1 -- 172.21.15.28:0/1483431531 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe54c0594f0 con 0x7fe5580c6b90 2024-02-21T23:08:37.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.965+0000 7fe55f400640 1 -- 172.21.15.28:0/1483431531 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe5580707a0 con 0x7fe5580c6b90 2024-02-21T23:08:37.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.967+0000 7fe554c00640 1 -- 172.21.15.28:0/1483431531 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 130 v130) v1 ==== 95+0+4947 (secure 0 0 0) 0x7fe54c058df0 con 0x7fe5580c6b90 2024-02-21T23:08:37.955 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:08:37.955 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":130,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5948,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3556518453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3556518453},{"type":"v1","addr":"172.21.15.28:6835","nonce":3556518453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116},{"gid":5951,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/3293264689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":3293264689},{"type":"v1","addr":"172.21.15.81:6836","nonce":3293264689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116},{"gid":8157,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/1285325008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1285325008},{"type":"v1","addr":"172.21.15.28:6834","nonce":1285325008}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":128}],"filesystems":[{"mdsmap":{"epoch":130,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:08:13.883141+0000","modified":"2024-02-21T23:08:34.520926+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":124,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6634},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6634":{"gid":6634,"name":"d","rank":0,"incarnation":127,"state":"up:active","state_seq":4,"addr":"172.21.15.81:6837/3687210533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":3687210533},{"type":"v1","addr":"172.21.15.81:6837","nonce":3687210533}]},"join_fscid":9,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-21T23:08:37.955 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 130 2024-02-21T23:08:37.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.969+0000 7fe55f400640 1 -- 172.21.15.28:0/1483431531 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe53806b4d0 msgr2=0x7fe53806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:37.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.969+0000 7fe55f400640 1 --2- 172.21.15.28:0/1483431531 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe53806b4d0 0x7fe53806d990 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fe544036000 tx=0x7fe54403c040 comp rx=0 tx=0).stop 2024-02-21T23:08:37.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.969+0000 7fe55f400640 1 -- 172.21.15.28:0/1483431531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5580c6b90 msgr2=0x7fe55814f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:37.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.969+0000 7fe55f400640 1 --2- 172.21.15.28:0/1483431531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5580c6b90 0x7fe55814f0c0 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7fe54c00d620 tx=0x7fe54c00dae0 comp rx=0 tx=0).stop 2024-02-21T23:08:37.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.970+0000 7fe55f400640 1 -- 172.21.15.28:0/1483431531 shutdown_connections 2024-02-21T23:08:37.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.970+0000 7fe55f400640 1 --2- 172.21.15.28:0/1483431531 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe53806b4d0 0x7fe53806d990 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:37.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.970+0000 7fe55f400640 1 --2- 172.21.15.28:0/1483431531 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe5581528a0 0x7fe55814ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:37.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.970+0000 7fe55f400640 1 --2- 172.21.15.28:0/1483431531 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe5580c76c0 0x7fe55814df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:37.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.970+0000 7fe55f400640 1 --2- 172.21.15.28:0/1483431531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5580c6b90 0x7fe55814f0c0 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:37.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.970+0000 7fe55f400640 1 -- 172.21.15.28:0/1483431531 >> 172.21.15.28:0/1483431531 conn(0x7fe5580aa4c0 msgr2=0x7fe5580c4d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:37.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.970+0000 7fe55f400640 1 -- 172.21.15.28:0/1483431531 shutdown_connections 2024-02-21T23:08:37.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:37.970+0000 7fe55f400640 1 -- 172.21.15.28:0/1483431531 wait complete. 2024-02-21T23:08:37.977 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:08:38.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.203+0000 7f2e91000640 1 Processor -- start 2024-02-21T23:08:38.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.203+0000 7f2e91000640 1 -- start start 2024-02-21T23:08:38.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.205+0000 7f2e91000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2e8c150450 0x7f2e8c0a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:38.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.205+0000 7f2e91000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e8c0c2760 con 0x7f2e8c156af0 2024-02-21T23:08:38.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.205+0000 7f2e91000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e8c156310 con 0x7f2e8c06ee40 2024-02-21T23:08:38.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.205+0000 7f2e91000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e8c0c22c0 con 0x7f2e8c150450 2024-02-21T23:08:38.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.205+0000 7f2e8b400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2e8c150450 0x7f2e8c0a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:38.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.205+0000 7f2e8be00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f2e8c156af0 0x7f2e8c06dff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:51946/0 (socket says 172.21.15.28:51946) 2024-02-21T23:08:38.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.205+0000 7f2e8b400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2e8c150450 0x7f2e8c0a5fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35850/0 (socket says 172.21.15.28:35850) 2024-02-21T23:08:38.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.205+0000 7f2e8be00640 1 -- 172.21.15.28:0/3550533916 learned_addr learned my addr 172.21.15.28:0/3550533916 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:08:38.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.206+0000 7f2e8b400640 1 -- 172.21.15.28:0/3550533916 >> v1:172.21.15.81:6789/0 conn(0x7f2e8c06ee40 legacy=0x7f2e8c1536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:38.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.206+0000 7f2e8b400640 1 -- 172.21.15.28:0/3550533916 >> v1:172.21.15.28:6789/0 conn(0x7f2e8c156af0 legacy=0x7f2e8c06dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:38.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.206+0000 7f2e8b400640 1 -- 172.21.15.28:0/3550533916 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e8c0c29d0 con 0x7f2e8c150450 2024-02-21T23:08:38.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.206+0000 7f2e8a000640 1 -- 172.21.15.28:0/3550533916 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1347034108 0 0) 0x7f2e8c0c2760 con 0x7f2e8c156af0 2024-02-21T23:08:38.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.207+0000 7f2e8b400640 1 --2- 172.21.15.28:0/3550533916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e8c150450 0x7f2e8c0a5fd0 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f2e7c009d90 tx=0x7f2e7c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5e274d1753fc0d1 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:38.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.207+0000 7f2e8a000640 1 -- 172.21.15.28:0/3550533916 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e7c039070 con 0x7f2e8c150450 2024-02-21T23:08:38.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.207+0000 7f2e8a000640 1 -- 172.21.15.28:0/3550533916 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e7c02cc60 con 0x7f2e8c150450 2024-02-21T23:08:38.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.207+0000 7f2e8a000640 1 -- 172.21.15.28:0/3550533916 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e7c034660 con 0x7f2e8c150450 2024-02-21T23:08:38.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.208+0000 7f2e91000640 1 -- 172.21.15.28:0/3550533916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e8c150450 msgr2=0x7f2e8c0a5fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:38.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.208+0000 7f2e91000640 1 --2- 172.21.15.28:0/3550533916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e8c150450 0x7f2e8c0a5fd0 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f2e7c009d90 tx=0x7f2e7c02beb0 comp rx=0 tx=0).stop 2024-02-21T23:08:38.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.208+0000 7f2e91000640 1 -- 172.21.15.28:0/3550533916 shutdown_connections 2024-02-21T23:08:38.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.208+0000 7f2e91000640 1 --2- 172.21.15.28:0/3550533916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e8c150450 0x7f2e8c0a5fd0 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:38.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.208+0000 7f2e91000640 1 -- 172.21.15.28:0/3550533916 >> 172.21.15.28:0/3550533916 conn(0x7f2e8c06b630 msgr2=0x7f2e8c0c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:38.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.208+0000 7f2e91000640 1 -- 172.21.15.28:0/3550533916 shutdown_connections 2024-02-21T23:08:38.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.208+0000 7f2e91000640 1 -- 172.21.15.28:0/3550533916 wait complete. 2024-02-21T23:08:38.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.210+0000 7f2e91000640 1 Processor -- start 2024-02-21T23:08:38.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.210+0000 7f2e91000640 1 -- start start 2024-02-21T23:08:38.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.211+0000 7f2e91000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2e8c06ee40 0x7f2e8c072460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:38.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.211+0000 7f2e91000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e8c150450 0x7f2e8c1edcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:38.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.211+0000 7f2e91000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2e8c156af0 0x7f2e8c1ee1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:38.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.211+0000 7f2e91000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e8c06e700 con 0x7f2e8c150450 2024-02-21T23:08:38.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.211+0000 7f2e91000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e8c156660 con 0x7f2e8c156af0 2024-02-21T23:08:38.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.211+0000 7f2e91000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e8c06e880 con 0x7f2e8c06ee40 2024-02-21T23:08:38.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.211+0000 7f2e8aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e8c150450 0x7f2e8c1edcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:38.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.212+0000 7f2e8aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e8c150450 0x7f2e8c1edcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35856/0 (socket says 172.21.15.28:35856) 2024-02-21T23:08:38.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.212+0000 7f2e8aa00640 1 -- 172.21.15.28:0/2688465910 learned_addr learned my addr 172.21.15.28:0/2688465910 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:38.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.212+0000 7f2e8be00640 1 --2- 172.21.15.28:0/2688465910 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2e8c156af0 0x7f2e8c1ee1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:38.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.212+0000 7f2e8b400640 1 --2- 172.21.15.28:0/2688465910 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2e8c06ee40 0x7f2e8c072460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:38.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.212+0000 7f2e8aa00640 1 -- 172.21.15.28:0/2688465910 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2e8c06ee40 msgr2=0x7f2e8c072460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:38.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.212+0000 7f2e8aa00640 1 --2- 172.21.15.28:0/2688465910 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2e8c06ee40 0x7f2e8c072460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:38.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.212+0000 7f2e8aa00640 1 -- 172.21.15.28:0/2688465910 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2e8c156af0 msgr2=0x7f2e8c1ee1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:38.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.212+0000 7f2e8aa00640 1 --2- 172.21.15.28:0/2688465910 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2e8c156af0 0x7f2e8c1ee1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:38.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.212+0000 7f2e8aa00640 1 -- 172.21.15.28:0/2688465910 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e8c1ee730 con 0x7f2e8c150450 2024-02-21T23:08:38.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.212+0000 7f2e8b400640 1 --2- 172.21.15.28:0/2688465910 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2e8c06ee40 0x7f2e8c072460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:08:38.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.212+0000 7f2e8aa00640 1 --2- 172.21.15.28:0/2688465910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e8c150450 0x7f2e8c1edcb0 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f2e7800c460 tx=0x7f2e7800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:38.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.213+0000 7f2e83400640 1 -- 172.21.15.28:0/2688465910 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e78016020 con 0x7f2e8c150450 2024-02-21T23:08:38.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.213+0000 7f2e83400640 1 -- 172.21.15.28:0/2688465910 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e78012870 con 0x7f2e8c150450 2024-02-21T23:08:38.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.213+0000 7f2e83400640 1 -- 172.21.15.28:0/2688465910 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e78012c60 con 0x7f2e8c150450 2024-02-21T23:08:38.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.213+0000 7f2e91000640 1 -- 172.21.15.28:0/2688465910 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e8c1dcc80 con 0x7f2e8c150450 2024-02-21T23:08:38.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.213+0000 7f2e91000640 1 -- 172.21.15.28:0/2688465910 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e8c1dc480 con 0x7f2e8c150450 2024-02-21T23:08:38.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.214+0000 7f2e91000640 1 -- 172.21.15.28:0/2688465910 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e8c04f290 con 0x7f2e8c150450 2024-02-21T23:08:38.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.215+0000 7f2e83400640 1 -- 172.21.15.28:0/2688465910 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2e78019020 con 0x7f2e8c150450 2024-02-21T23:08:38.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.216+0000 7f2e83400640 1 --2- 172.21.15.28:0/2688465910 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2e6c06b4d0 0x7f2e6c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:38.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.216+0000 7f2e83400640 1 -- 172.21.15.28:0/2688465910 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f2e7808a170 con 0x7f2e8c150450 2024-02-21T23:08:38.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.216+0000 7f2e8b400640 1 --2- 172.21.15.28:0/2688465910 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2e6c06b4d0 0x7f2e6c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:38.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.218+0000 7f2e8b400640 1 --2- 172.21.15.28:0/2688465910 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2e6c06b4d0 0x7f2e6c06d990 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f2e7c036000 tx=0x7f2e7c030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:38.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.220+0000 7f2e83400640 1 -- 172.21.15.28:0/2688465910 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f2e78020030 con 0x7f2e8c150450 2024-02-21T23:08:38.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.424+0000 7f2e91000640 1 -- 172.21.15.28:0/2688465910 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2e8c0c1f10 con 0x7f2e8c150450 2024-02-21T23:08:38.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.425+0000 7f2e83400640 1 -- 172.21.15.28:0/2688465910 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v124) v1 ==== 74+0+17099 (secure 0 0 0) 0x7f2e8c1ee730 con 0x7f2e8c150450 2024-02-21T23:08:38.414 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:08:38.414 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":124,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:08:32.504818+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:08:11.794800+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"121","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2024-02-21T23:08:12.918602+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"122","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6834/1773560593":"2024-02-22T23:08:32.504770+0000","172.21.15.28:6832/1773560593":"2024-02-22T23:08:32.504770+0000","172.21.15.81:6837/1038096496":"2024-02-22T23:08:30.184500+0000","172.21.15.81:6835/1038096496":"2024-02-22T23:08:30.184500+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:08:38.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.428+0000 7f2e91000640 1 -- 172.21.15.28:0/2688465910 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2e6c06b4d0 msgr2=0x7f2e6c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:38.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.428+0000 7f2e91000640 1 --2- 172.21.15.28:0/2688465910 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2e6c06b4d0 0x7f2e6c06d990 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f2e7c036000 tx=0x7f2e7c030000 comp rx=0 tx=0).stop 2024-02-21T23:08:38.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.428+0000 7f2e91000640 1 -- 172.21.15.28:0/2688465910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e8c150450 msgr2=0x7f2e8c1edcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:38.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.428+0000 7f2e91000640 1 --2- 172.21.15.28:0/2688465910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e8c150450 0x7f2e8c1edcb0 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f2e7800c460 tx=0x7f2e7800c920 comp rx=0 tx=0).stop 2024-02-21T23:08:38.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.429+0000 7f2e91000640 1 -- 172.21.15.28:0/2688465910 shutdown_connections 2024-02-21T23:08:38.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.429+0000 7f2e91000640 1 --2- 172.21.15.28:0/2688465910 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2e6c06b4d0 0x7f2e6c06d990 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:38.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.429+0000 7f2e91000640 1 --2- 172.21.15.28:0/2688465910 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2e8c156af0 0x7f2e8c1ee1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:38.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.429+0000 7f2e91000640 1 --2- 172.21.15.28:0/2688465910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e8c150450 0x7f2e8c1edcb0 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:38.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.429+0000 7f2e91000640 1 --2- 172.21.15.28:0/2688465910 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2e8c06ee40 0x7f2e8c072460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:38.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.429+0000 7f2e91000640 1 -- 172.21.15.28:0/2688465910 >> 172.21.15.28:0/2688465910 conn(0x7f2e8c06b630 msgr2=0x7f2e8c06dff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:38.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.429+0000 7f2e91000640 1 -- 172.21.15.28:0/2688465910 shutdown_connections 2024-02-21T23:08:38.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.429+0000 7f2e91000640 1 -- 172.21.15.28:0/2688465910 wait complete. 2024-02-21T23:08:38.439 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-21T23:02:30.728329+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-21T23:08:11.794800+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '121', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_data', 'create_time': '2024-02-21T23:08:12.918602+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '122', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-21T23:08:38.440 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-21T23:08:38.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.677+0000 7f6c5b600640 1 Processor -- start 2024-02-21T23:08:38.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.677+0000 7f6c5b600640 1 -- start start 2024-02-21T23:08:38.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.679+0000 7f6c5b600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f6c540c5f50 0x7f6c5414fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:38.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.679+0000 7f6c5b600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c54152fb0 con 0x7f6c54150530 2024-02-21T23:08:38.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.679+0000 7f6c5b600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c54153290 con 0x7f6c5406b0e0 2024-02-21T23:08:38.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.679+0000 7f6c5b600640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c54153bc0 con 0x7f6c540c5f50 2024-02-21T23:08:38.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.679+0000 7f6c59600640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f6c54150530 0x7f6c54156500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:51956/0 (socket says 172.21.15.28:51956) 2024-02-21T23:08:38.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.679+0000 7f6c59600640 1 -- 172.21.15.28:0/90068633 learned_addr learned my addr 172.21.15.28:0/90068633 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:08:38.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.679+0000 7f6c53e00640 1 --2- 172.21.15.28:0/90068633 >> v2:172.21.15.81:3300/0 conn(0x7f6c540c5f50 0x7f6c5414fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:38.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.680+0000 7f6c53400640 1 -- 172.21.15.28:0/90068633 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 817145567 0 0) 0x7f6c54152fb0 con 0x7f6c54150530 2024-02-21T23:08:38.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.680+0000 7f6c53400640 1 -- 172.21.15.28:0/90068633 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c3c003520 con 0x7f6c54150530 2024-02-21T23:08:38.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.680+0000 7f6c53e00640 1 -- 172.21.15.28:0/90068633 >> v1:172.21.15.81:6789/0 conn(0x7f6c5406b0e0 legacy=0x7f6c540c5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:38.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.680+0000 7f6c53e00640 1 -- 172.21.15.28:0/90068633 >> v1:172.21.15.28:6789/0 conn(0x7f6c54150530 legacy=0x7f6c54156500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:38.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.680+0000 7f6c53e00640 1 -- 172.21.15.28:0/90068633 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c54153570 con 0x7f6c540c5f50 2024-02-21T23:08:38.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.682+0000 7f6c53e00640 1 --2- 172.21.15.28:0/90068633 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6c540c5f50 0x7f6c5414fff0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f6c400097a0 tx=0x7f6c4002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b1a3812d8460630e server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:38.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.683+0000 7f6c53400640 1 -- 172.21.15.28:0/90068633 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c40035020 con 0x7f6c540c5f50 2024-02-21T23:08:38.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.683+0000 7f6c53400640 1 -- 172.21.15.28:0/90068633 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c40031830 con 0x7f6c540c5f50 2024-02-21T23:08:38.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.683+0000 7f6c53400640 1 -- 172.21.15.28:0/90068633 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c40031be0 con 0x7f6c540c5f50 2024-02-21T23:08:38.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.684+0000 7f6c5b600640 1 -- 172.21.15.28:0/90068633 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6c540c5f50 msgr2=0x7f6c5414fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:38.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.684+0000 7f6c5b600640 1 --2- 172.21.15.28:0/90068633 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6c540c5f50 0x7f6c5414fff0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f6c400097a0 tx=0x7f6c4002b8d0 comp rx=0 tx=0).stop 2024-02-21T23:08:38.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.684+0000 7f6c5b600640 1 -- 172.21.15.28:0/90068633 shutdown_connections 2024-02-21T23:08:38.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.684+0000 7f6c5b600640 1 --2- 172.21.15.28:0/90068633 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6c540c5f50 0x7f6c5414fff0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:38.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.684+0000 7f6c5b600640 1 -- 172.21.15.28:0/90068633 >> 172.21.15.28:0/90068633 conn(0x7f6c540a85a0 msgr2=0x7f6c540a8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:38.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.684+0000 7f6c5b600640 1 -- 172.21.15.28:0/90068633 shutdown_connections 2024-02-21T23:08:38.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.684+0000 7f6c5b600640 1 -- 172.21.15.28:0/90068633 wait complete. 2024-02-21T23:08:38.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.685+0000 7f6c5b600640 1 Processor -- start 2024-02-21T23:08:38.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.686+0000 7f6c5b600640 1 -- start start 2024-02-21T23:08:38.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.686+0000 7f6c5b600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6c5406b0e0 0x7f6c54155c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:38.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.687+0000 7f6c5b600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6c540c5f50 0x7f6c54146dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:38.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.687+0000 7f6c5b600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c54150530 0x7f6c540c3470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:38.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.687+0000 7f6c5b600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c5406b4c0 con 0x7f6c54150530 2024-02-21T23:08:38.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.687+0000 7f6c5b600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c541509c0 con 0x7f6c5406b0e0 2024-02-21T23:08:38.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.687+0000 7f6c5b600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c5406b640 con 0x7f6c540c5f50 2024-02-21T23:08:38.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.687+0000 7f6c59600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c54150530 0x7f6c540c3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:38.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.687+0000 7f6c59600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c54150530 0x7f6c540c3470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35872/0 (socket says 172.21.15.28:35872) 2024-02-21T23:08:38.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.687+0000 7f6c59600640 1 -- 172.21.15.28:0/4035458280 learned_addr learned my addr 172.21.15.28:0/4035458280 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:38.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.687+0000 7f6c58c00640 1 --2- 172.21.15.28:0/4035458280 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6c5406b0e0 0x7f6c54155c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:38.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.687+0000 7f6c53e00640 1 --2- 172.21.15.28:0/4035458280 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6c540c5f50 0x7f6c54146dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:38.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.687+0000 7f6c59600640 1 -- 172.21.15.28:0/4035458280 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6c540c5f50 msgr2=0x7f6c54146dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:38.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.687+0000 7f6c59600640 1 --2- 172.21.15.28:0/4035458280 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6c540c5f50 0x7f6c54146dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:38.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.687+0000 7f6c59600640 1 -- 172.21.15.28:0/4035458280 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6c5406b0e0 msgr2=0x7f6c54155c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:38.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.687+0000 7f6c59600640 1 --2- 172.21.15.28:0/4035458280 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6c5406b0e0 0x7f6c54155c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:38.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.687+0000 7f6c59600640 1 -- 172.21.15.28:0/4035458280 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c540c3cb0 con 0x7f6c54150530 2024-02-21T23:08:38.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.687+0000 7f6c58c00640 1 --2- 172.21.15.28:0/4035458280 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6c5406b0e0 0x7f6c54155c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:08:38.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.688+0000 7f6c59600640 1 --2- 172.21.15.28:0/4035458280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c54150530 0x7f6c540c3470 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f6c4c00caa0 tx=0x7f6c4c00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:38.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.688+0000 7f6c50c00640 1 -- 172.21.15.28:0/4035458280 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c4c017070 con 0x7f6c54150530 2024-02-21T23:08:38.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.688+0000 7f6c50c00640 1 -- 172.21.15.28:0/4035458280 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c4c00ac00 con 0x7f6c54150530 2024-02-21T23:08:38.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.688+0000 7f6c50c00640 1 -- 172.21.15.28:0/4035458280 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c4c012400 con 0x7f6c54150530 2024-02-21T23:08:38.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.688+0000 7f6c5b600640 1 -- 172.21.15.28:0/4035458280 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c540c3f90 con 0x7f6c54150530 2024-02-21T23:08:38.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.688+0000 7f6c5b600640 1 -- 172.21.15.28:0/4035458280 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c540c2ce0 con 0x7f6c54150530 2024-02-21T23:08:38.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.689+0000 7f6c5b600640 1 -- 172.21.15.28:0/4035458280 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c54067f30 con 0x7f6c54150530 2024-02-21T23:08:38.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.691+0000 7f6c50c00640 1 -- 172.21.15.28:0/4035458280 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6c4c01a070 con 0x7f6c54150530 2024-02-21T23:08:38.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.691+0000 7f6c50c00640 1 --2- 172.21.15.28:0/4035458280 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6c3406b500 0x7f6c3406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:38.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.692+0000 7f6c50c00640 1 -- 172.21.15.28:0/4035458280 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f6c4c08a200 con 0x7f6c54150530 2024-02-21T23:08:38.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.692+0000 7f6c58c00640 1 --2- 172.21.15.28:0/4035458280 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6c3406b500 0x7f6c3406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:38.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.693+0000 7f6c58c00640 1 --2- 172.21.15.28:0/4035458280 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6c3406b500 0x7f6c3406d9c0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f6c5414fb10 tx=0x7f6c48009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:38.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.696+0000 7f6c50c00640 1 -- 172.21.15.28:0/4035458280 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f6c4c058e30 con 0x7f6c54150530 2024-02-21T23:08:38.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.898+0000 7f6c5b600640 1 -- 172.21.15.28:0/4035458280 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6c5414f860 con 0x7f6c54150530 2024-02-21T23:08:38.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.899+0000 7f6c50c00640 1 -- 172.21.15.28:0/4035458280 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v124) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6c4c058730 con 0x7f6c54150530 2024-02-21T23:08:38.889 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-02-21T23:08:38.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.902+0000 7f6c5b600640 1 -- 172.21.15.28:0/4035458280 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6c3406b500 msgr2=0x7f6c3406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:38.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.902+0000 7f6c5b600640 1 --2- 172.21.15.28:0/4035458280 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6c3406b500 0x7f6c3406d9c0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f6c5414fb10 tx=0x7f6c48009040 comp rx=0 tx=0).stop 2024-02-21T23:08:38.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.902+0000 7f6c5b600640 1 -- 172.21.15.28:0/4035458280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c54150530 msgr2=0x7f6c540c3470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:38.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.902+0000 7f6c5b600640 1 --2- 172.21.15.28:0/4035458280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c54150530 0x7f6c540c3470 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f6c4c00caa0 tx=0x7f6c4c00cf60 comp rx=0 tx=0).stop 2024-02-21T23:08:38.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.902+0000 7f6c5b600640 1 -- 172.21.15.28:0/4035458280 shutdown_connections 2024-02-21T23:08:38.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.902+0000 7f6c5b600640 1 --2- 172.21.15.28:0/4035458280 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6c3406b500 0x7f6c3406d9c0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:38.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.902+0000 7f6c5b600640 1 --2- 172.21.15.28:0/4035458280 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6c540c5f50 0x7f6c54146dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:38.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.902+0000 7f6c5b600640 1 --2- 172.21.15.28:0/4035458280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c54150530 0x7f6c540c3470 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:38.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.902+0000 7f6c5b600640 1 --2- 172.21.15.28:0/4035458280 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6c5406b0e0 0x7f6c54155c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:38.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.902+0000 7f6c5b600640 1 -- 172.21.15.28:0/4035458280 >> 172.21.15.28:0/4035458280 conn(0x7f6c540a85a0 msgr2=0x7f6c5414afa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:38.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.902+0000 7f6c5b600640 1 -- 172.21.15.28:0/4035458280 shutdown_connections 2024-02-21T23:08:38.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:38.902+0000 7f6c5b600640 1 -- 172.21.15.28:0/4035458280 wait complete. 2024-02-21T23:08:38.909 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-21T23:08:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.149+0000 7fe07d800640 1 Processor -- start 2024-02-21T23:08:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.149+0000 7fe07d800640 1 -- start start 2024-02-21T23:08:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.151+0000 7fe07d800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe07806b0e0 0x7fe0780c5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:39.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.151+0000 7fe07d800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fe0780c5d80 0x7fe07814fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:39.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.151+0000 7fe07d800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0780c61e0 con 0x7fe078150530 2024-02-21T23:08:39.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.151+0000 7fe07d800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe078153bc0 con 0x7fe0780c5d80 2024-02-21T23:08:39.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.151+0000 7fe07d800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe078152fb0 con 0x7fe07806b0e0 2024-02-21T23:08:39.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.151+0000 7fe076000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fe0780c5d80 0x7fe07814fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:39.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.151+0000 7fe076a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe07806b0e0 0x7fe0780c5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:39.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.151+0000 7fe076000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fe0780c5d80 0x7fe07814fff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.28:55350/0 (socket says 172.21.15.28:55350) 2024-02-21T23:08:39.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.151+0000 7fe076000640 1 -- 172.21.15.28:0/3242256896 learned_addr learned my addr 172.21.15.28:0/3242256896 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:39.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.152+0000 7fe076000640 1 -- 172.21.15.28:0/3242256896 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe07806b0e0 msgr2=0x7fe0780c5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:39.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.152+0000 7fe076000640 1 --2- 172.21.15.28:0/3242256896 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe07806b0e0 0x7fe0780c5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:39.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.152+0000 7fe076000640 1 -- 172.21.15.28:0/3242256896 >> v1:172.21.15.81:6789/0 conn(0x7fe078150530 legacy=0x7fe078156500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:39.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.152+0000 7fe076000640 1 -- 172.21.15.28:0/3242256896 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe078153290 con 0x7fe0780c5d80 2024-02-21T23:08:39.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.152+0000 7fe076a00640 1 --2- 172.21.15.28:0/3242256896 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe07806b0e0 0x7fe0780c5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-21T23:08:39.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.153+0000 7fe076000640 1 --2- 172.21.15.28:0/3242256896 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe0780c5d80 0x7fe07814fff0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fe064009870 tx=0x7fe06402b900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d19bcd29511f417 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:39.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.154+0000 7fe075600640 1 -- 172.21.15.28:0/3242256896 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe064035020 con 0x7fe0780c5d80 2024-02-21T23:08:39.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.154+0000 7fe075600640 1 -- 172.21.15.28:0/3242256896 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe06402d040 con 0x7fe0780c5d80 2024-02-21T23:08:39.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.154+0000 7fe075600640 1 -- 172.21.15.28:0/3242256896 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe064031a80 con 0x7fe0780c5d80 2024-02-21T23:08:39.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.155+0000 7fe07d800640 1 -- 172.21.15.28:0/3242256896 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe0780c5d80 msgr2=0x7fe07814fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:39.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.155+0000 7fe07d800640 1 --2- 172.21.15.28:0/3242256896 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe0780c5d80 0x7fe07814fff0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fe064009870 tx=0x7fe06402b900 comp rx=0 tx=0).stop 2024-02-21T23:08:39.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.155+0000 7fe07d800640 1 -- 172.21.15.28:0/3242256896 shutdown_connections 2024-02-21T23:08:39.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.155+0000 7fe07d800640 1 --2- 172.21.15.28:0/3242256896 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe0780c5d80 0x7fe07814fff0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:39.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.155+0000 7fe07d800640 1 --2- 172.21.15.28:0/3242256896 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe07806b0e0 0x7fe0780c5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:39.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.155+0000 7fe07d800640 1 -- 172.21.15.28:0/3242256896 >> 172.21.15.28:0/3242256896 conn(0x7fe0780a85a0 msgr2=0x7fe0780a8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:39.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.155+0000 7fe07d800640 1 -- 172.21.15.28:0/3242256896 shutdown_connections 2024-02-21T23:08:39.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.155+0000 7fe07d800640 1 -- 172.21.15.28:0/3242256896 wait complete. 2024-02-21T23:08:39.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.157+0000 7fe07d800640 1 Processor -- start 2024-02-21T23:08:39.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.157+0000 7fe07d800640 1 -- start start 2024-02-21T23:08:39.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.158+0000 7fe07d800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe07806b0e0 0x7fe07806dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:39.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.158+0000 7fe07d800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0780c5d80 0x7fe078149f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:39.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.158+0000 7fe07d800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe078150530 0x7fe07806e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:39.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.158+0000 7fe07d800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0781509c0 con 0x7fe0780c5d80 2024-02-21T23:08:39.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.158+0000 7fe07d800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe078153850 con 0x7fe078150530 2024-02-21T23:08:39.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.158+0000 7fe07d800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe07806b4c0 con 0x7fe07806b0e0 2024-02-21T23:08:39.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.158+0000 7fe076000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0780c5d80 0x7fe078149f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:39.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.158+0000 7fe076000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0780c5d80 0x7fe078149f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35888/0 (socket says 172.21.15.28:35888) 2024-02-21T23:08:39.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.158+0000 7fe076000640 1 -- 172.21.15.28:0/540974945 learned_addr learned my addr 172.21.15.28:0/540974945 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:39.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.159+0000 7fe077400640 1 --2- 172.21.15.28:0/540974945 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe078150530 0x7fe07806e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:39.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.159+0000 7fe076a00640 1 --2- 172.21.15.28:0/540974945 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe07806b0e0 0x7fe07806dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:39.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.159+0000 7fe076000640 1 -- 172.21.15.28:0/540974945 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe07806b0e0 msgr2=0x7fe07806dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:39.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.159+0000 7fe076000640 1 --2- 172.21.15.28:0/540974945 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe07806b0e0 0x7fe07806dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:39.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.159+0000 7fe076000640 1 -- 172.21.15.28:0/540974945 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe078150530 msgr2=0x7fe07806e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:39.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.159+0000 7fe076000640 1 --2- 172.21.15.28:0/540974945 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe078150530 0x7fe07806e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:39.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.159+0000 7fe076000640 1 -- 172.21.15.28:0/540974945 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe078155e30 con 0x7fe0780c5d80 2024-02-21T23:08:39.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.159+0000 7fe076000640 1 --2- 172.21.15.28:0/540974945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0780c5d80 0x7fe078149f00 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7fe064000c00 tx=0x7fe0640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:39.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.160+0000 7fe06ea00640 1 -- 172.21.15.28:0/540974945 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe064040070 con 0x7fe0780c5d80 2024-02-21T23:08:39.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.160+0000 7fe06ea00640 1 -- 172.21.15.28:0/540974945 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0640399f0 con 0x7fe0780c5d80 2024-02-21T23:08:39.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.160+0000 7fe06ea00640 1 -- 172.21.15.28:0/540974945 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe064035040 con 0x7fe0780c5d80 2024-02-21T23:08:39.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.160+0000 7fe07d800640 1 -- 172.21.15.28:0/540974945 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe078154ee0 con 0x7fe0780c5d80 2024-02-21T23:08:39.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.160+0000 7fe07d800640 1 -- 172.21.15.28:0/540974945 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe078154c00 con 0x7fe0780c5d80 2024-02-21T23:08:39.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.161+0000 7fe07d800640 1 -- 172.21.15.28:0/540974945 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0780c63d0 con 0x7fe0780c5d80 2024-02-21T23:08:39.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.162+0000 7fe06ea00640 1 -- 172.21.15.28:0/540974945 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe06404b060 con 0x7fe0780c5d80 2024-02-21T23:08:39.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.163+0000 7fe06ea00640 1 --2- 172.21.15.28:0/540974945 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe05806b500 0x7fe05806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:39.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.163+0000 7fe06ea00640 1 -- 172.21.15.28:0/540974945 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 6273+0+0 (secure 0 0 0) 0x7fe0640abe60 con 0x7fe0780c5d80 2024-02-21T23:08:39.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.163+0000 7fe076a00640 1 --2- 172.21.15.28:0/540974945 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe05806b500 0x7fe05806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:39.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.165+0000 7fe076a00640 1 --2- 172.21.15.28:0/540974945 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe05806b500 0x7fe05806d9c0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fe068009560 tx=0x7fe068008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:39.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.167+0000 7fe06ea00640 1 -- 172.21.15.28:0/540974945 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe0780c63d0 con 0x7fe0780c5d80 2024-02-21T23:08:39.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.367+0000 7fe07d800640 1 -- 172.21.15.28:0/540974945 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe07806b860 con 0x7fe0780c5d80 2024-02-21T23:08:39.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.368+0000 7fe06ea00640 1 -- 172.21.15.28:0/540974945 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v124) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe07806b860 con 0x7fe0780c5d80 2024-02-21T23:08:39.356 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-02-21T23:08:39.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.371+0000 7fe07d800640 1 -- 172.21.15.28:0/540974945 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe05806b500 msgr2=0x7fe05806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:39.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.371+0000 7fe07d800640 1 --2- 172.21.15.28:0/540974945 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe05806b500 0x7fe05806d9c0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fe068009560 tx=0x7fe068008040 comp rx=0 tx=0).stop 2024-02-21T23:08:39.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.371+0000 7fe07d800640 1 -- 172.21.15.28:0/540974945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0780c5d80 msgr2=0x7fe078149f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:39.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.371+0000 7fe07d800640 1 --2- 172.21.15.28:0/540974945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0780c5d80 0x7fe078149f00 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7fe064000c00 tx=0x7fe0640046c0 comp rx=0 tx=0).stop 2024-02-21T23:08:39.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.371+0000 7fe07d800640 1 -- 172.21.15.28:0/540974945 shutdown_connections 2024-02-21T23:08:39.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.371+0000 7fe07d800640 1 --2- 172.21.15.28:0/540974945 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe05806b500 0x7fe05806d9c0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:39.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.371+0000 7fe07d800640 1 --2- 172.21.15.28:0/540974945 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe078150530 0x7fe07806e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:39.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.371+0000 7fe07d800640 1 --2- 172.21.15.28:0/540974945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0780c5d80 0x7fe078149f00 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:39.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.371+0000 7fe07d800640 1 --2- 172.21.15.28:0/540974945 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe07806b0e0 0x7fe07806dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:39.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.371+0000 7fe07d800640 1 -- 172.21.15.28:0/540974945 >> 172.21.15.28:0/540974945 conn(0x7fe0780a85a0 msgr2=0x7fe07814b190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:39.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.371+0000 7fe07d800640 1 -- 172.21.15.28:0/540974945 shutdown_connections 2024-02-21T23:08:39.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.371+0000 7fe07d800640 1 -- 172.21.15.28:0/540974945 wait complete. 2024-02-21T23:08:39.379 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-21T23:08:39.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.620+0000 7f8dc4a00640 1 Processor -- start 2024-02-21T23:08:39.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.621+0000 7f8dc4a00640 1 -- start start 2024-02-21T23:08:39.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.622+0000 7f8dc4a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8dc0150240 0x7f8dc00c12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:39.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.622+0000 7f8dc4a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8dc00c2c10 0x7f8dc006ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:39.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.622+0000 7f8dc4a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dc006e220 con 0x7f8dc00c4580 2024-02-21T23:08:39.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.622+0000 7f8dc4a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8dc0150620 con 0x7f8dc00c2c10 2024-02-21T23:08:39.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.622+0000 7f8dc4a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc0153a90 con 0x7f8dc0150240 2024-02-21T23:08:39.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.622+0000 7f8dbf200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8dc00c2c10 0x7f8dc006ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:39.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.622+0000 7f8db5e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f8dc00c4580 0x7f8dc006db10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:51958/0 (socket says 172.21.15.28:51958) 2024-02-21T23:08:39.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.622+0000 7f8db5e00640 1 -- 172.21.15.28:0/991175282 learned_addr learned my addr 172.21.15.28:0/991175282 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:08:39.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.622+0000 7f8dbe800640 1 --2- 172.21.15.28:0/991175282 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8dc0150240 0x7f8dc00c12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:39.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.623+0000 7f8dbde00640 1 -- 172.21.15.28:0/991175282 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3837349084 0 0) 0x7f8dc006e220 con 0x7f8dc00c4580 2024-02-21T23:08:39.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.623+0000 7f8dbe800640 1 -- 172.21.15.28:0/991175282 >> v2:172.21.15.28:3300/0 conn(0x7f8dc00c2c10 msgr2=0x7f8dc006ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:39.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.623+0000 7f8dbe800640 1 --2- 172.21.15.28:0/991175282 >> v2:172.21.15.28:3300/0 conn(0x7f8dc00c2c10 0x7f8dc006ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:39.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.623+0000 7f8dbe800640 1 -- 172.21.15.28:0/991175282 >> v1:172.21.15.28:6789/0 conn(0x7f8dc00c4580 legacy=0x7f8dc006db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:39.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.623+0000 7f8dbe800640 1 -- 172.21.15.28:0/991175282 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dc006f780 con 0x7f8dc0150240 2024-02-21T23:08:39.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.624+0000 7f8dbe800640 1 --2- 172.21.15.28:0/991175282 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8dc0150240 0x7f8dc00c12c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8db00097a0 tx=0x7f8db002b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=293ad0f303ed6cd5 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:39.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.624+0000 7f8dbde00640 1 -- 172.21.15.28:0/991175282 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db0031780 con 0x7f8dc0150240 2024-02-21T23:08:39.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.624+0000 7f8dbde00640 1 -- 172.21.15.28:0/991175282 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8db0037070 con 0x7f8dc0150240 2024-02-21T23:08:39.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.624+0000 7f8dbde00640 1 -- 172.21.15.28:0/991175282 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db0031b40 con 0x7f8dc0150240 2024-02-21T23:08:39.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.625+0000 7f8dc4a00640 1 -- 172.21.15.28:0/991175282 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8dc0150240 msgr2=0x7f8dc00c12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:39.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.625+0000 7f8dc4a00640 1 --2- 172.21.15.28:0/991175282 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8dc0150240 0x7f8dc00c12c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8db00097a0 tx=0x7f8db002b820 comp rx=0 tx=0).stop 2024-02-21T23:08:39.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.625+0000 7f8dc4a00640 1 -- 172.21.15.28:0/991175282 shutdown_connections 2024-02-21T23:08:39.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.625+0000 7f8dc4a00640 1 --2- 172.21.15.28:0/991175282 >> v2:172.21.15.28:3300/0 conn(0x7f8dc00c2c10 0x7f8dc006ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:39.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.625+0000 7f8dc4a00640 1 --2- 172.21.15.28:0/991175282 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8dc0150240 0x7f8dc00c12c0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:39.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.625+0000 7f8dc4a00640 1 -- 172.21.15.28:0/991175282 >> 172.21.15.28:0/991175282 conn(0x7f8dc00a8530 msgr2=0x7f8dc00a8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:39.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.625+0000 7f8dc4a00640 1 -- 172.21.15.28:0/991175282 shutdown_connections 2024-02-21T23:08:39.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.625+0000 7f8dc4a00640 1 -- 172.21.15.28:0/991175282 wait complete. 2024-02-21T23:08:39.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.627+0000 7f8dc4a00640 1 Processor -- start 2024-02-21T23:08:39.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.627+0000 7f8dc4a00640 1 -- start start 2024-02-21T23:08:39.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.628+0000 7f8dc4a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8dc00c2c10 0x7f8dc00abfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:39.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.628+0000 7f8dc4a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8dc00c4580 0x7f8dc0154550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:39.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.628+0000 7f8dc4a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8dc0150240 0x7f8dc0154a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:39.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.628+0000 7f8dc4a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc00c4a10 con 0x7f8dc0150240 2024-02-21T23:08:39.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.628+0000 7f8dc4a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc0153720 con 0x7f8dc00c2c10 2024-02-21T23:08:39.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.628+0000 7f8dc4a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc00c4b90 con 0x7f8dc00c4580 2024-02-21T23:08:39.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.628+0000 7f8dbf200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8dc0150240 0x7f8dc0154a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:39.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.628+0000 7f8dbf200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8dc0150240 0x7f8dc0154a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35898/0 (socket says 172.21.15.28:35898) 2024-02-21T23:08:39.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.628+0000 7f8dbf200640 1 -- 172.21.15.28:0/2683496728 learned_addr learned my addr 172.21.15.28:0/2683496728 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:39.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.629+0000 7f8db5e00640 1 --2- 172.21.15.28:0/2683496728 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8dc00c4580 0x7f8dc0154550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:39.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.629+0000 7f8dbe800640 1 --2- 172.21.15.28:0/2683496728 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8dc00c2c10 0x7f8dc00abfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:39.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.629+0000 7f8dbf200640 1 -- 172.21.15.28:0/2683496728 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8dc00c4580 msgr2=0x7f8dc0154550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:39.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.629+0000 7f8dbf200640 1 --2- 172.21.15.28:0/2683496728 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8dc00c4580 0x7f8dc0154550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:39.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.629+0000 7f8dbf200640 1 -- 172.21.15.28:0/2683496728 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8dc00c2c10 msgr2=0x7f8dc00abfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:39.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.629+0000 7f8dbf200640 1 --2- 172.21.15.28:0/2683496728 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8dc00c2c10 0x7f8dc00abfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:39.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.629+0000 7f8dbf200640 1 -- 172.21.15.28:0/2683496728 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dc0071f70 con 0x7f8dc0150240 2024-02-21T23:08:39.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.629+0000 7f8dbf200640 1 --2- 172.21.15.28:0/2683496728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8dc0150240 0x7f8dc0154a90 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f8db8009d30 tx=0x7f8db800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:39.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.629+0000 7f8db7400640 1 -- 172.21.15.28:0/2683496728 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db8018020 con 0x7f8dc0150240 2024-02-21T23:08:39.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.630+0000 7f8db7400640 1 -- 172.21.15.28:0/2683496728 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8db8014480 con 0x7f8dc0150240 2024-02-21T23:08:39.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.630+0000 7f8dc4a00640 1 -- 172.21.15.28:0/2683496728 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dc00731e0 con 0x7f8dc0150240 2024-02-21T23:08:39.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.630+0000 7f8db7400640 1 -- 172.21.15.28:0/2683496728 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db8014870 con 0x7f8dc0150240 2024-02-21T23:08:39.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.630+0000 7f8dc4a00640 1 -- 172.21.15.28:0/2683496728 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dc0071c90 con 0x7f8dc0150240 2024-02-21T23:08:39.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.631+0000 7f8dc4a00640 1 -- 172.21.15.28:0/2683496728 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dc004f290 con 0x7f8dc0150240 2024-02-21T23:08:39.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.632+0000 7f8db7400640 1 -- 172.21.15.28:0/2683496728 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8db8005d10 con 0x7f8dc0150240 2024-02-21T23:08:39.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.633+0000 7f8db7400640 1 --2- 172.21.15.28:0/2683496728 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8d9c06b500 0x7f8d9c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:39.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.633+0000 7f8db7400640 1 -- 172.21.15.28:0/2683496728 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f8db808b230 con 0x7f8dc0150240 2024-02-21T23:08:39.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.634+0000 7f8dbe800640 1 --2- 172.21.15.28:0/2683496728 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8d9c06b500 0x7f8d9c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:39.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.635+0000 7f8dbe800640 1 --2- 172.21.15.28:0/2683496728 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8d9c06b500 0x7f8d9c06d9c0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f8db0036000 tx=0x7f8db003d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:39.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.638+0000 7f8db7400640 1 -- 172.21.15.28:0/2683496728 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f8db8022030 con 0x7f8dc0150240 2024-02-21T23:08:39.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.840+0000 7f8dc4a00640 1 -- 172.21.15.28:0/2683496728 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8dc006f0e0 con 0x7f8dc0150240 2024-02-21T23:08:39.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.841+0000 7f8db7400640 1 -- 172.21.15.28:0/2683496728 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v124) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8db8010040 con 0x7f8dc0150240 2024-02-21T23:08:39.829 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-02-21T23:08:39.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.844+0000 7f8dc4a00640 1 -- 172.21.15.28:0/2683496728 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8d9c06b500 msgr2=0x7f8d9c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:39.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.844+0000 7f8dc4a00640 1 --2- 172.21.15.28:0/2683496728 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8d9c06b500 0x7f8d9c06d9c0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f8db0036000 tx=0x7f8db003d040 comp rx=0 tx=0).stop 2024-02-21T23:08:39.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.844+0000 7f8dc4a00640 1 -- 172.21.15.28:0/2683496728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8dc0150240 msgr2=0x7f8dc0154a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:39.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.844+0000 7f8dc4a00640 1 --2- 172.21.15.28:0/2683496728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8dc0150240 0x7f8dc0154a90 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f8db8009d30 tx=0x7f8db800c490 comp rx=0 tx=0).stop 2024-02-21T23:08:39.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.844+0000 7f8dc4a00640 1 -- 172.21.15.28:0/2683496728 shutdown_connections 2024-02-21T23:08:39.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.844+0000 7f8dc4a00640 1 --2- 172.21.15.28:0/2683496728 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8d9c06b500 0x7f8d9c06d9c0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:39.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.844+0000 7f8dc4a00640 1 --2- 172.21.15.28:0/2683496728 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8dc00c4580 0x7f8dc0154550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:39.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.844+0000 7f8dc4a00640 1 --2- 172.21.15.28:0/2683496728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8dc0150240 0x7f8dc0154a90 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:39.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.844+0000 7f8dc4a00640 1 --2- 172.21.15.28:0/2683496728 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8dc00c2c10 0x7f8dc00abfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:39.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.844+0000 7f8dc4a00640 1 -- 172.21.15.28:0/2683496728 >> 172.21.15.28:0/2683496728 conn(0x7f8dc00a8530 msgr2=0x7f8dc006ae70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:39.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.844+0000 7f8dc4a00640 1 -- 172.21.15.28:0/2683496728 shutdown_connections 2024-02-21T23:08:39.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:39.844+0000 7f8dc4a00640 1 -- 172.21.15.28:0/2683496728 wait complete. 2024-02-21T23:08:39.851 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:08:40.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.080+0000 7f74a3400640 1 Processor -- start 2024-02-21T23:08:40.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.080+0000 7f74a3400640 1 -- start start 2024-02-21T23:08:40.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.081+0000 7f74a3400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f749c0c7be0 0x7f749c156f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:40.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.081+0000 7f74a3400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f749c070ad0 0x7f749c070f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:40.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.081+0000 7f74a3400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f749c1564a0 con 0x7f749c0c5a00 2024-02-21T23:08:40.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.081+0000 7f74a3400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f749c0c2f90 con 0x7f749c070ad0 2024-02-21T23:08:40.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.081+0000 7f74a3400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f749c0c5020 con 0x7f749c0c7be0 2024-02-21T23:08:40.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.082+0000 7f74a0a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f749c0c5a00 0x7f749c0c4910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:51960/0 (socket says 172.21.15.28:51960) 2024-02-21T23:08:40.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.082+0000 7f74a0a00640 1 -- 172.21.15.28:0/3208120580 learned_addr learned my addr 172.21.15.28:0/3208120580 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:08:40.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.082+0000 7f74a1400640 1 --2- 172.21.15.28:0/3208120580 >> v2:172.21.15.81:3300/0 conn(0x7f749c070ad0 0x7f749c070f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:40.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.082+0000 7f7493e00640 1 --2- 172.21.15.28:0/3208120580 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f749c0c7be0 0x7f749c156f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:40.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.082+0000 7f749be00640 1 -- 172.21.15.28:0/3208120580 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 716243102 0 0) 0x7f749c1564a0 con 0x7f749c0c5a00 2024-02-21T23:08:40.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.083+0000 7f749be00640 1 -- 172.21.15.28:0/3208120580 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7480003520 con 0x7f749c0c5a00 2024-02-21T23:08:40.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.083+0000 7f749be00640 1 -- 172.21.15.28:0/3208120580 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2372925397 0 0) 0x7f7480003520 con 0x7f749c0c5a00 2024-02-21T23:08:40.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.083+0000 7f749be00640 1 -- 172.21.15.28:0/3208120580 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f749c0c7be0 msgr2=0x7f749c156f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:40.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.083+0000 7f749be00640 1 --2- 172.21.15.28:0/3208120580 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f749c0c7be0 0x7f749c156f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:40.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.083+0000 7f749be00640 1 -- 172.21.15.28:0/3208120580 >> v2:172.21.15.81:3300/0 conn(0x7f749c070ad0 msgr2=0x7f749c070f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:40.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.083+0000 7f749be00640 1 --2- 172.21.15.28:0/3208120580 >> v2:172.21.15.81:3300/0 conn(0x7f749c070ad0 0x7f749c070f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:40.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.083+0000 7f749be00640 1 -- 172.21.15.28:0/3208120580 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f749c1539c0 con 0x7f749c0c5a00 2024-02-21T23:08:40.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.083+0000 7f749be00640 1 -- 172.21.15.28:0/3208120580 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f748c001670 con 0x7f749c0c5a00 2024-02-21T23:08:40.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.083+0000 7f749be00640 1 -- 172.21.15.28:0/3208120580 >> v1:172.21.15.28:6789/0 conn(0x7f749c0c5a00 legacy=0x7f749c0c4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:40.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.083+0000 7f749be00640 1 --2- 172.21.15.28:0/3208120580 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74800039b0 0x7f7480003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:40.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.083+0000 7f749be00640 1 --2- 172.21.15.28:0/3208120580 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f749c0c7be0 0x7f7480004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:40.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.083+0000 7f749be00640 1 --2- 172.21.15.28:0/3208120580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7480004d40 0x7f74800051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:40.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.083+0000 7f749be00640 1 -- 172.21.15.28:0/3208120580 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f749c0c2f90 con 0x7f7480004d40 2024-02-21T23:08:40.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.083+0000 7f749be00640 1 -- 172.21.15.28:0/3208120580 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f749c0c5020 con 0x7f74800039b0 2024-02-21T23:08:40.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.083+0000 7f749be00640 1 -- 172.21.15.28:0/3208120580 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f748000b9f0 con 0x7f749c0c7be0 2024-02-21T23:08:40.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.083+0000 7f74a1400640 1 --2- 172.21.15.28:0/3208120580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7480004d40 0x7f74800051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:40.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.084+0000 7f7493e00640 1 --2- 172.21.15.28:0/3208120580 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f749c0c7be0 0x7f7480004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:40.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.084+0000 7f74a0a00640 1 --2- 172.21.15.28:0/3208120580 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74800039b0 0x7f7480003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:40.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.084+0000 7f74a1400640 1 -- 172.21.15.28:0/3208120580 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f749c0c7be0 msgr2=0x7f7480004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:40.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.084+0000 7f74a1400640 1 --2- 172.21.15.28:0/3208120580 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f749c0c7be0 0x7f7480004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:40.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.084+0000 7f74a1400640 1 -- 172.21.15.28:0/3208120580 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74800039b0 msgr2=0x7f7480003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:40.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.084+0000 7f74a1400640 1 --2- 172.21.15.28:0/3208120580 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f74800039b0 0x7f7480003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:40.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.084+0000 7f74a1400640 1 -- 172.21.15.28:0/3208120580 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7480003520 con 0x7f7480004d40 2024-02-21T23:08:40.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.085+0000 7f74a1400640 1 --2- 172.21.15.28:0/3208120580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7480004d40 0x7f74800051c0 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f7494009fc0 tx=0x7f7480005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90b3edbde6b0a86 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:40.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.085+0000 7f749be00640 1 -- 172.21.15.28:0/3208120580 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7494039070 con 0x7f7480004d40 2024-02-21T23:08:40.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.085+0000 7f749be00640 1 -- 172.21.15.28:0/3208120580 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f749402cc60 con 0x7f7480004d40 2024-02-21T23:08:40.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.085+0000 7f749be00640 1 -- 172.21.15.28:0/3208120580 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7494009c10 con 0x7f7480004d40 2024-02-21T23:08:40.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.086+0000 7f74a3400640 1 -- 172.21.15.28:0/3208120580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7480004d40 msgr2=0x7f74800051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:40.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.086+0000 7f74a3400640 1 --2- 172.21.15.28:0/3208120580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7480004d40 0x7f74800051c0 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f7494009fc0 tx=0x7f7480005830 comp rx=0 tx=0).stop 2024-02-21T23:08:40.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.086+0000 7f74a1400640 1 -- 172.21.15.28:0/3208120580 reap_dead start 2024-02-21T23:08:40.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.086+0000 7f74a3400640 1 -- 172.21.15.28:0/3208120580 shutdown_connections 2024-02-21T23:08:40.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.086+0000 7f74a3400640 1 -- 172.21.15.28:0/3208120580 >> 172.21.15.28:0/3208120580 conn(0x7f749c0c8810 msgr2=0x7f749c0c8bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:40.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.086+0000 7f74a3400640 1 -- 172.21.15.28:0/3208120580 shutdown_connections 2024-02-21T23:08:40.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.086+0000 7f74a3400640 1 -- 172.21.15.28:0/3208120580 wait complete. 2024-02-21T23:08:40.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.088+0000 7f74a3400640 1 Processor -- start 2024-02-21T23:08:40.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.088+0000 7f74a3400640 1 -- start start 2024-02-21T23:08:40.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.089+0000 7f74a3400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7480004d40 0x7f749c0c1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:40.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.089+0000 7f74a3400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f749c1ead80 0x7f749c14cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:40.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.089+0000 7f74a3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f749c1dffd0 0x7f749c14d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:40.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.089+0000 7f74a3400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f749c0c5de0 con 0x7f749c1dffd0 2024-02-21T23:08:40.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.089+0000 7f74a3400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f749c156a20 con 0x7f7480004d40 2024-02-21T23:08:40.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.089+0000 7f74a3400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f749c153ca0 con 0x7f749c1ead80 2024-02-21T23:08:40.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.089+0000 7f74a1400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f749c1dffd0 0x7f749c14d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:40.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.089+0000 7f74a1400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f749c1dffd0 0x7f749c14d420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35914/0 (socket says 172.21.15.28:35914) 2024-02-21T23:08:40.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.089+0000 7f74a1400640 1 -- 172.21.15.28:0/482326770 learned_addr learned my addr 172.21.15.28:0/482326770 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:40.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.090+0000 7f74a0a00640 1 --2- 172.21.15.28:0/482326770 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7480004d40 0x7f749c0c1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:40.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.090+0000 7f7493e00640 1 --2- 172.21.15.28:0/482326770 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f749c1ead80 0x7f749c14cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:40.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.090+0000 7f74a1400640 1 -- 172.21.15.28:0/482326770 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f749c1ead80 msgr2=0x7f749c14cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:40.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.090+0000 7f74a1400640 1 --2- 172.21.15.28:0/482326770 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f749c1ead80 0x7f749c14cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:40.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.090+0000 7f74a1400640 1 -- 172.21.15.28:0/482326770 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7480004d40 msgr2=0x7f749c0c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:40.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.090+0000 7f74a1400640 1 --2- 172.21.15.28:0/482326770 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7480004d40 0x7f749c0c1eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:40.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.090+0000 7f74a1400640 1 -- 172.21.15.28:0/482326770 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f749c1eb280 con 0x7f749c1dffd0 2024-02-21T23:08:40.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.090+0000 7f74a0a00640 1 --2- 172.21.15.28:0/482326770 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7480004d40 0x7f749c0c1eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:08:40.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.090+0000 7f7493e00640 1 --2- 172.21.15.28:0/482326770 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f749c1ead80 0x7f749c14cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:08:40.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.090+0000 7f74a1400640 1 --2- 172.21.15.28:0/482326770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f749c1dffd0 0x7f749c14d420 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f7494002230 tx=0x7f7494009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:40.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.090+0000 7f7499600640 1 -- 172.21.15.28:0/482326770 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7494039070 con 0x7f749c1dffd0 2024-02-21T23:08:40.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.090+0000 7f7499600640 1 -- 172.21.15.28:0/482326770 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f749403a040 con 0x7f749c1dffd0 2024-02-21T23:08:40.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.090+0000 7f74a3400640 1 -- 172.21.15.28:0/482326770 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f749c1eb560 con 0x7f749c1dffd0 2024-02-21T23:08:40.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.091+0000 7f7499600640 1 -- 172.21.15.28:0/482326770 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7494003990 con 0x7f749c1dffd0 2024-02-21T23:08:40.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.091+0000 7f74a3400640 1 -- 172.21.15.28:0/482326770 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f749c1ebb20 con 0x7f749c1dffd0 2024-02-21T23:08:40.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.092+0000 7f74a3400640 1 -- 172.21.15.28:0/482326770 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f749c068640 con 0x7f749c1dffd0 2024-02-21T23:08:40.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.093+0000 7f7499600640 1 -- 172.21.15.28:0/482326770 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7494046030 con 0x7f749c1dffd0 2024-02-21T23:08:40.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.093+0000 7f7499600640 1 --2- 172.21.15.28:0/482326770 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f747806b500 0x7f747806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:40.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.094+0000 7f7499600640 1 -- 172.21.15.28:0/482326770 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f749402ca60 con 0x7f749c1dffd0 2024-02-21T23:08:40.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.094+0000 7f74a0a00640 1 --2- 172.21.15.28:0/482326770 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f747806b500 0x7f747806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:40.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.095+0000 7f74a0a00640 1 --2- 172.21.15.28:0/482326770 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f747806b500 0x7f747806d9c0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f749c14ddb0 tx=0x7f748c001010 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:40.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.098+0000 7f7499600640 1 -- 172.21.15.28:0/482326770 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f749404d020 con 0x7f749c1dffd0 2024-02-21T23:08:40.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.335+0000 7f74a3400640 1 -- 172.21.15.28:0/482326770 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f749c0c79c0 con 0x7f749c1dffd0 2024-02-21T23:08:40.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.337+0000 7f7499600640 1 -- 172.21.15.28:0/482326770 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 130 v130) v1 ==== 95+0+4947 (secure 0 0 0) 0x7f749407b070 con 0x7f749c1dffd0 2024-02-21T23:08:40.325 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:08:40.325 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":130,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5948,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3556518453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3556518453},{"type":"v1","addr":"172.21.15.28:6835","nonce":3556518453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116},{"gid":5951,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/3293264689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":3293264689},{"type":"v1","addr":"172.21.15.81:6836","nonce":3293264689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116},{"gid":8157,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/1285325008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1285325008},{"type":"v1","addr":"172.21.15.28:6834","nonce":1285325008}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":128}],"filesystems":[{"mdsmap":{"epoch":130,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:08:13.883141+0000","modified":"2024-02-21T23:08:34.520926+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":124,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6634},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6634":{"gid":6634,"name":"d","rank":0,"incarnation":127,"state":"up:active","state_seq":4,"addr":"172.21.15.81:6837/3687210533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":3687210533},{"type":"v1","addr":"172.21.15.81:6837","nonce":3687210533}]},"join_fscid":9,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-21T23:08:40.325 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 130 2024-02-21T23:08:40.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.339+0000 7f74a3400640 1 -- 172.21.15.28:0/482326770 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f747806b500 msgr2=0x7f747806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:40.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.339+0000 7f74a3400640 1 --2- 172.21.15.28:0/482326770 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f747806b500 0x7f747806d9c0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f749c14ddb0 tx=0x7f748c001010 comp rx=0 tx=0).stop 2024-02-21T23:08:40.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.339+0000 7f74a3400640 1 -- 172.21.15.28:0/482326770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f749c1dffd0 msgr2=0x7f749c14d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:40.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.339+0000 7f74a3400640 1 --2- 172.21.15.28:0/482326770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f749c1dffd0 0x7f749c14d420 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f7494002230 tx=0x7f7494009750 comp rx=0 tx=0).stop 2024-02-21T23:08:40.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.339+0000 7f74a3400640 1 -- 172.21.15.28:0/482326770 shutdown_connections 2024-02-21T23:08:40.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.339+0000 7f74a3400640 1 --2- 172.21.15.28:0/482326770 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f747806b500 0x7f747806d9c0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:40.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.339+0000 7f74a3400640 1 --2- 172.21.15.28:0/482326770 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f749c1ead80 0x7f749c14cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:40.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.339+0000 7f74a3400640 1 --2- 172.21.15.28:0/482326770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f749c1dffd0 0x7f749c14d420 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:40.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.339+0000 7f74a3400640 1 --2- 172.21.15.28:0/482326770 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7480004d40 0x7f749c0c1eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:40.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.340+0000 7f74a3400640 1 -- 172.21.15.28:0/482326770 >> 172.21.15.28:0/482326770 conn(0x7f749c0c8810 msgr2=0x7f749c0c4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:40.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.340+0000 7f74a3400640 1 -- 172.21.15.28:0/482326770 shutdown_connections 2024-02-21T23:08:40.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.340+0000 7f74a3400640 1 -- 172.21.15.28:0/482326770 wait complete. 2024-02-21T23:08:40.347 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:08:40.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.581+0000 7fdb9da00640 1 Processor -- start 2024-02-21T23:08:40.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.582+0000 7fdb9da00640 1 -- start start 2024-02-21T23:08:40.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.583+0000 7fdb9da00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdb98152500 0x7fdb981558c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:40.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.583+0000 7fdb9da00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdb980ca050 0x7fdb980bf6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:40.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.583+0000 7fdb9da00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb981528e0 con 0x7fdb980bfc00 2024-02-21T23:08:40.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.583+0000 7fdb9da00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb980c86e0 con 0x7fdb98152500 2024-02-21T23:08:40.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.583+0000 7fdb9da00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb980c8980 con 0x7fdb980ca050 2024-02-21T23:08:40.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.584+0000 7fdb97e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdb98152500 0x7fdb981558c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:40.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.584+0000 7fdb97e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdb98152500 0x7fdb981558c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35930/0 (socket says 172.21.15.28:35930) 2024-02-21T23:08:40.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.584+0000 7fdb9d000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fdb980bfc00 0x7fdb9806bbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:51974/0 (socket says 172.21.15.28:51974) 2024-02-21T23:08:40.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.584+0000 7fdb97e00640 1 -- 172.21.15.28:0/3149843710 learned_addr learned my addr 172.21.15.28:0/3149843710 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:40.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.584+0000 7fdb8fe00640 1 --2- 172.21.15.28:0/3149843710 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdb980ca050 0x7fdb980bf6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:40.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.584+0000 7fdb97e00640 1 -- 172.21.15.28:0/3149843710 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdb980ca050 msgr2=0x7fdb980bf6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:40.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.584+0000 7fdb97e00640 1 --2- 172.21.15.28:0/3149843710 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdb980ca050 0x7fdb980bf6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:40.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.584+0000 7fdb97e00640 1 -- 172.21.15.28:0/3149843710 >> v1:172.21.15.28:6789/0 conn(0x7fdb980bfc00 legacy=0x7fdb9806bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:40.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.584+0000 7fdb97e00640 1 -- 172.21.15.28:0/3149843710 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb980c8c60 con 0x7fdb98152500 2024-02-21T23:08:40.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.584+0000 7fdb97400640 1 -- 172.21.15.28:0/3149843710 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3441162097 0 0) 0x7fdb981528e0 con 0x7fdb980bfc00 2024-02-21T23:08:40.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.584+0000 7fdb8fe00640 1 --2- 172.21.15.28:0/3149843710 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdb980ca050 0x7fdb980bf6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:08:40.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.585+0000 7fdb97e00640 1 --2- 172.21.15.28:0/3149843710 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb98152500 0x7fdb981558c0 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7fdb88009870 tx=0x7fdb8802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76d8672497574033 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:40.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.586+0000 7fdb97400640 1 -- 172.21.15.28:0/3149843710 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb88035020 con 0x7fdb98152500 2024-02-21T23:08:40.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.586+0000 7fdb97400640 1 -- 172.21.15.28:0/3149843710 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb88031640 con 0x7fdb98152500 2024-02-21T23:08:40.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.586+0000 7fdb97400640 1 -- 172.21.15.28:0/3149843710 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb8802d040 con 0x7fdb98152500 2024-02-21T23:08:40.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.586+0000 7fdb9da00640 1 -- 172.21.15.28:0/3149843710 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb98152500 msgr2=0x7fdb981558c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:40.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.586+0000 7fdb9da00640 1 --2- 172.21.15.28:0/3149843710 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb98152500 0x7fdb981558c0 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7fdb88009870 tx=0x7fdb8802b8f0 comp rx=0 tx=0).stop 2024-02-21T23:08:40.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.586+0000 7fdb9da00640 1 -- 172.21.15.28:0/3149843710 shutdown_connections 2024-02-21T23:08:40.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.586+0000 7fdb9da00640 1 --2- 172.21.15.28:0/3149843710 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdb980ca050 0x7fdb980bf6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:40.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.586+0000 7fdb9da00640 1 --2- 172.21.15.28:0/3149843710 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb98152500 0x7fdb981558c0 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:40.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.586+0000 7fdb9da00640 1 -- 172.21.15.28:0/3149843710 >> 172.21.15.28:0/3149843710 conn(0x7fdb980ac450 msgr2=0x7fdb980ac860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:40.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.587+0000 7fdb9da00640 1 -- 172.21.15.28:0/3149843710 shutdown_connections 2024-02-21T23:08:40.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.587+0000 7fdb9da00640 1 -- 172.21.15.28:0/3149843710 wait complete. 2024-02-21T23:08:40.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.588+0000 7fdb9da00640 1 Processor -- start 2024-02-21T23:08:40.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.588+0000 7fdb9da00640 1 -- start start 2024-02-21T23:08:40.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.589+0000 7fdb9da00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdb980bfc00 0x7fdb980ab6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:40.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.589+0000 7fdb9da00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb980ca050 0x7fdb980ae280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:40.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.589+0000 7fdb9da00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdb98152500 0x7fdb980ae7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:40.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.589+0000 7fdb9da00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb9806c300 con 0x7fdb980ca050 2024-02-21T23:08:40.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.589+0000 7fdb9da00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb980c9220 con 0x7fdb98152500 2024-02-21T23:08:40.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.589+0000 7fdb9da00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb9806c480 con 0x7fdb980bfc00 2024-02-21T23:08:40.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.590+0000 7fdb8fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb980ca050 0x7fdb980ae280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:40.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.590+0000 7fdb8fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb980ca050 0x7fdb980ae280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35940/0 (socket says 172.21.15.28:35940) 2024-02-21T23:08:40.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.590+0000 7fdb8fe00640 1 -- 172.21.15.28:0/196994782 learned_addr learned my addr 172.21.15.28:0/196994782 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:40.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.590+0000 7fdb97e00640 1 --2- 172.21.15.28:0/196994782 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdb980bfc00 0x7fdb980ab6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:40.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.590+0000 7fdb9d000640 1 --2- 172.21.15.28:0/196994782 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdb98152500 0x7fdb980ae7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:40.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.590+0000 7fdb8fe00640 1 -- 172.21.15.28:0/196994782 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdb980bfc00 msgr2=0x7fdb980ab6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:40.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.590+0000 7fdb8fe00640 1 --2- 172.21.15.28:0/196994782 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdb980bfc00 0x7fdb980ab6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:40.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.590+0000 7fdb8fe00640 1 -- 172.21.15.28:0/196994782 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdb98152500 msgr2=0x7fdb980ae7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:40.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.590+0000 7fdb8fe00640 1 --2- 172.21.15.28:0/196994782 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdb98152500 0x7fdb980ae7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:40.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.590+0000 7fdb8fe00640 1 -- 172.21.15.28:0/196994782 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb9814f050 con 0x7fdb980ca050 2024-02-21T23:08:40.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.590+0000 7fdb8fe00640 1 --2- 172.21.15.28:0/196994782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb980ca050 0x7fdb980ae280 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7fdb8000c820 tx=0x7fdb8000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:40.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.591+0000 7fdb94c00640 1 -- 172.21.15.28:0/196994782 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb80015070 con 0x7fdb980ca050 2024-02-21T23:08:40.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.591+0000 7fdb94c00640 1 -- 172.21.15.28:0/196994782 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb80007a30 con 0x7fdb980ca050 2024-02-21T23:08:40.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.591+0000 7fdb9da00640 1 -- 172.21.15.28:0/196994782 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb9814c940 con 0x7fdb980ca050 2024-02-21T23:08:40.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.591+0000 7fdb94c00640 1 -- 172.21.15.28:0/196994782 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb80007e00 con 0x7fdb980ca050 2024-02-21T23:08:40.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.591+0000 7fdb9da00640 1 -- 172.21.15.28:0/196994782 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb980abbf0 con 0x7fdb980ca050 2024-02-21T23:08:40.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.592+0000 7fdb9da00640 1 -- 172.21.15.28:0/196994782 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb9804f290 con 0x7fdb980ca050 2024-02-21T23:08:40.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.593+0000 7fdb94c00640 1 -- 172.21.15.28:0/196994782 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdb80018070 con 0x7fdb980ca050 2024-02-21T23:08:40.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.595+0000 7fdb94c00640 1 --2- 172.21.15.28:0/196994782 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdb7406b4d0 0x7fdb7406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:40.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.595+0000 7fdb94c00640 1 -- 172.21.15.28:0/196994782 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 6273+0+0 (secure 0 0 0) 0x7fdb8008a430 con 0x7fdb980ca050 2024-02-21T23:08:40.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.595+0000 7fdb97e00640 1 --2- 172.21.15.28:0/196994782 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdb7406b4d0 0x7fdb7406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:40.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.597+0000 7fdb97e00640 1 --2- 172.21.15.28:0/196994782 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdb7406b4d0 0x7fdb7406d990 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fdb88036000 tx=0x7fdb8803c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:40.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.600+0000 7fdb94c00640 1 -- 172.21.15.28:0/196994782 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fdb80059060 con 0x7fdb980ca050 2024-02-21T23:08:40.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.803+0000 7fdb9da00640 1 -- 172.21.15.28:0/196994782 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdb980adfe0 con 0x7fdb980ca050 2024-02-21T23:08:40.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.804+0000 7fdb94c00640 1 -- 172.21.15.28:0/196994782 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v124) v1 ==== 74+0+17099 (secure 0 0 0) 0x7fdb80058960 con 0x7fdb980ca050 2024-02-21T23:08:40.792 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:08:40.793 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":124,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:08:32.504818+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:08:11.794800+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"121","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2024-02-21T23:08:12.918602+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"122","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6834/1773560593":"2024-02-22T23:08:32.504770+0000","172.21.15.28:6832/1773560593":"2024-02-22T23:08:32.504770+0000","172.21.15.81:6837/1038096496":"2024-02-22T23:08:30.184500+0000","172.21.15.81:6835/1038096496":"2024-02-22T23:08:30.184500+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:08:40.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.807+0000 7fdb9da00640 1 -- 172.21.15.28:0/196994782 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdb7406b4d0 msgr2=0x7fdb7406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:40.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.807+0000 7fdb9da00640 1 --2- 172.21.15.28:0/196994782 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdb7406b4d0 0x7fdb7406d990 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fdb88036000 tx=0x7fdb8803c040 comp rx=0 tx=0).stop 2024-02-21T23:08:40.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.807+0000 7fdb9da00640 1 -- 172.21.15.28:0/196994782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb980ca050 msgr2=0x7fdb980ae280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:40.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.807+0000 7fdb9da00640 1 --2- 172.21.15.28:0/196994782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb980ca050 0x7fdb980ae280 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7fdb8000c820 tx=0x7fdb8000cc20 comp rx=0 tx=0).stop 2024-02-21T23:08:40.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.807+0000 7fdb9da00640 1 -- 172.21.15.28:0/196994782 shutdown_connections 2024-02-21T23:08:40.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.807+0000 7fdb9da00640 1 --2- 172.21.15.28:0/196994782 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdb7406b4d0 0x7fdb7406d990 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:40.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.807+0000 7fdb9da00640 1 --2- 172.21.15.28:0/196994782 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdb98152500 0x7fdb980ae7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:40.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.807+0000 7fdb9da00640 1 --2- 172.21.15.28:0/196994782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb980ca050 0x7fdb980ae280 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:40.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.807+0000 7fdb9da00640 1 --2- 172.21.15.28:0/196994782 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdb980bfc00 0x7fdb980ab6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:40.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.807+0000 7fdb9da00640 1 -- 172.21.15.28:0/196994782 >> 172.21.15.28:0/196994782 conn(0x7fdb980ac450 msgr2=0x7fdb9806bbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:40.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.807+0000 7fdb9da00640 1 -- 172.21.15.28:0/196994782 shutdown_connections 2024-02-21T23:08:40.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:40.807+0000 7fdb9da00640 1 -- 172.21.15.28:0/196994782 wait complete. 2024-02-21T23:08:40.815 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-02-21T23:08:40.815 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:08:41.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.051+0000 7fde83600640 1 Processor -- start 2024-02-21T23:08:41.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.051+0000 7fde83600640 1 -- start start 2024-02-21T23:08:41.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.052+0000 7fde83600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fde7c1526b0 0x7fde7c1559c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:41.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.052+0000 7fde83600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fde7c0cadc0 0x7fde7c0c7e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:41.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.052+0000 7fde83600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde7c0c6f30 con 0x7fde7c0ca940 2024-02-21T23:08:41.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.052+0000 7fde83600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde7c0c8340 con 0x7fde7c0cadc0 2024-02-21T23:08:41.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.052+0000 7fde83600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde7c0c61b0 con 0x7fde7c1526b0 2024-02-21T23:08:41.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.053+0000 7fde81600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fde7c0cadc0 0x7fde7c0c7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:41.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.053+0000 7fde81600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fde7c0cadc0 0x7fde7c0c7e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35944/0 (socket says 172.21.15.28:35944) 2024-02-21T23:08:41.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.053+0000 7fde81600640 1 -- 172.21.15.28:0/835126527 learned_addr learned my addr 172.21.15.28:0/835126527 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:41.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.053+0000 7fde80c00640 1 --2- 172.21.15.28:0/835126527 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fde7c1526b0 0x7fde7c1559c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:41.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.053+0000 7fde81600640 1 -- 172.21.15.28:0/835126527 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fde7c1526b0 msgr2=0x7fde7c1559c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:41.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.053+0000 7fde81600640 1 --2- 172.21.15.28:0/835126527 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fde7c1526b0 0x7fde7c1559c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:41.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.053+0000 7fde81600640 1 -- 172.21.15.28:0/835126527 >> v1:172.21.15.28:6789/0 conn(0x7fde7c0ca940 legacy=0x7fde7c0c76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:41.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.053+0000 7fde81600640 1 -- 172.21.15.28:0/835126527 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde7c0c6450 con 0x7fde7c0cadc0 2024-02-21T23:08:41.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.053+0000 7fde7b400640 1 -- 172.21.15.28:0/835126527 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1880276424 0 0) 0x7fde7c0c6f30 con 0x7fde7c0ca940 2024-02-21T23:08:41.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.055+0000 7fde81600640 1 --2- 172.21.15.28:0/835126527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde7c0cadc0 0x7fde7c0c7e00 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7fde740098a0 tx=0x7fde7402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=930eb1a8872af698 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:41.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.055+0000 7fde7b400640 1 -- 172.21.15.28:0/835126527 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde74035020 con 0x7fde7c0cadc0 2024-02-21T23:08:41.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.055+0000 7fde7b400640 1 -- 172.21.15.28:0/835126527 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde74031640 con 0x7fde7c0cadc0 2024-02-21T23:08:41.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.055+0000 7fde7b400640 1 -- 172.21.15.28:0/835126527 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde7402d040 con 0x7fde7c0cadc0 2024-02-21T23:08:41.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.056+0000 7fde83600640 1 -- 172.21.15.28:0/835126527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde7c0cadc0 msgr2=0x7fde7c0c7e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:41.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.056+0000 7fde83600640 1 --2- 172.21.15.28:0/835126527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde7c0cadc0 0x7fde7c0c7e00 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7fde740098a0 tx=0x7fde7402b8f0 comp rx=0 tx=0).stop 2024-02-21T23:08:41.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.056+0000 7fde83600640 1 -- 172.21.15.28:0/835126527 shutdown_connections 2024-02-21T23:08:41.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.056+0000 7fde83600640 1 --2- 172.21.15.28:0/835126527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde7c0cadc0 0x7fde7c0c7e00 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:41.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.056+0000 7fde83600640 1 --2- 172.21.15.28:0/835126527 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fde7c1526b0 0x7fde7c1559c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:41.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.056+0000 7fde83600640 1 -- 172.21.15.28:0/835126527 >> 172.21.15.28:0/835126527 conn(0x7fde7c06d3d0 msgr2=0x7fde7c08afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:41.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.056+0000 7fde83600640 1 -- 172.21.15.28:0/835126527 shutdown_connections 2024-02-21T23:08:41.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.056+0000 7fde83600640 1 -- 172.21.15.28:0/835126527 wait complete. 2024-02-21T23:08:41.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.058+0000 7fde83600640 1 Processor -- start 2024-02-21T23:08:41.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.058+0000 7fde83600640 1 -- start start 2024-02-21T23:08:41.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.059+0000 7fde83600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fde7c0ca940 0x7fde7c1eea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:41.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.059+0000 7fde83600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde7c0cadc0 0x7fde7c150b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:41.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.059+0000 7fde83600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde7c1526b0 0x7fde7c14d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:41.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.059+0000 7fde83600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde7c0c71d0 con 0x7fde7c1526b0 2024-02-21T23:08:41.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.059+0000 7fde83600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde7c0c69f0 con 0x7fde7c0cadc0 2024-02-21T23:08:41.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.059+0000 7fde83600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde7c0c5c90 con 0x7fde7c0ca940 2024-02-21T23:08:41.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.059+0000 7fde81600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde7c1526b0 0x7fde7c14d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:41.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.059+0000 7fde81600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde7c1526b0 0x7fde7c14d6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35960/0 (socket says 172.21.15.28:35960) 2024-02-21T23:08:41.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.059+0000 7fde81600640 1 -- 172.21.15.28:0/4046941994 learned_addr learned my addr 172.21.15.28:0/4046941994 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:41.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.059+0000 7fde80c00640 1 --2- 172.21.15.28:0/4046941994 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fde7c0ca940 0x7fde7c1eea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:41.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.059+0000 7fde7be00640 1 --2- 172.21.15.28:0/4046941994 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde7c0cadc0 0x7fde7c150b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:41.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.060+0000 7fde81600640 1 -- 172.21.15.28:0/4046941994 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fde7c0ca940 msgr2=0x7fde7c1eea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:41.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.060+0000 7fde81600640 1 --2- 172.21.15.28:0/4046941994 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fde7c0ca940 0x7fde7c1eea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:41.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.060+0000 7fde81600640 1 -- 172.21.15.28:0/4046941994 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde7c0cadc0 msgr2=0x7fde7c150b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:41.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.060+0000 7fde81600640 1 --2- 172.21.15.28:0/4046941994 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde7c0cadc0 0x7fde7c150b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:41.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.060+0000 7fde81600640 1 -- 172.21.15.28:0/4046941994 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde7c155560 con 0x7fde7c1526b0 2024-02-21T23:08:41.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.060+0000 7fde81600640 1 --2- 172.21.15.28:0/4046941994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde7c1526b0 0x7fde7c14d6c0 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7fde740090b0 tx=0x7fde740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:41.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.060+0000 7fde78c00640 1 -- 172.21.15.28:0/4046941994 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde74040070 con 0x7fde7c1526b0 2024-02-21T23:08:41.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.060+0000 7fde78c00640 1 -- 172.21.15.28:0/4046941994 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde74039b90 con 0x7fde7c1526b0 2024-02-21T23:08:41.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.060+0000 7fde78c00640 1 -- 172.21.15.28:0/4046941994 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde74035040 con 0x7fde7c1526b0 2024-02-21T23:08:41.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.060+0000 7fde83600640 1 -- 172.21.15.28:0/4046941994 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde7c1df230 con 0x7fde7c1526b0 2024-02-21T23:08:41.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.061+0000 7fde83600640 1 -- 172.21.15.28:0/4046941994 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde7c1ee280 con 0x7fde7c1526b0 2024-02-21T23:08:41.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.062+0000 7fde83600640 1 -- 172.21.15.28:0/4046941994 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde7c067f30 con 0x7fde7c1526b0 2024-02-21T23:08:41.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.063+0000 7fde78c00640 1 -- 172.21.15.28:0/4046941994 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fde74038020 con 0x7fde7c1526b0 2024-02-21T23:08:41.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.064+0000 7fde78c00640 1 --2- 172.21.15.28:0/4046941994 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fde5806b4d0 0x7fde5806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:41.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.064+0000 7fde78c00640 1 -- 172.21.15.28:0/4046941994 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 6273+0+0 (secure 0 0 0) 0x7fde740abd70 con 0x7fde7c1526b0 2024-02-21T23:08:41.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.064+0000 7fde80c00640 1 --2- 172.21.15.28:0/4046941994 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fde5806b4d0 0x7fde5806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:41.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.066+0000 7fde80c00640 1 --2- 172.21.15.28:0/4046941994 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fde5806b4d0 0x7fde5806d990 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fde64006fd0 tx=0x7fde64008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:41.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.068+0000 7fde78c00640 1 -- 172.21.15.28:0/4046941994 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fde7407a9b0 con 0x7fde7c1526b0 2024-02-21T23:08:41.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.308+0000 7fde83600640 1 -- 172.21.15.28:0/4046941994 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde7c0c5f20 con 0x7fde7c1526b0 2024-02-21T23:08:41.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.309+0000 7fde78c00640 1 -- 172.21.15.28:0/4046941994 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 130 v130) v1 ==== 95+0+4947 (secure 0 0 0) 0x7fde7407a2b0 con 0x7fde7c1526b0 2024-02-21T23:08:41.297 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:08:41.298 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":130,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5948,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3556518453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3556518453},{"type":"v1","addr":"172.21.15.28:6835","nonce":3556518453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116},{"gid":5951,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/3293264689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":3293264689},{"type":"v1","addr":"172.21.15.81:6836","nonce":3293264689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116},{"gid":8157,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/1285325008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1285325008},{"type":"v1","addr":"172.21.15.28:6834","nonce":1285325008}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":128}],"filesystems":[{"mdsmap":{"epoch":130,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:08:13.883141+0000","modified":"2024-02-21T23:08:34.520926+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":124,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6634},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6634":{"gid":6634,"name":"d","rank":0,"incarnation":127,"state":"up:active","state_seq":4,"addr":"172.21.15.81:6837/3687210533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":3687210533},{"type":"v1","addr":"172.21.15.81:6837","nonce":3687210533}]},"join_fscid":9,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-21T23:08:41.298 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 130 2024-02-21T23:08:41.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.312+0000 7fde83600640 1 -- 172.21.15.28:0/4046941994 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fde5806b4d0 msgr2=0x7fde5806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:41.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.312+0000 7fde83600640 1 --2- 172.21.15.28:0/4046941994 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fde5806b4d0 0x7fde5806d990 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fde64006fd0 tx=0x7fde64008040 comp rx=0 tx=0).stop 2024-02-21T23:08:41.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.312+0000 7fde83600640 1 -- 172.21.15.28:0/4046941994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde7c1526b0 msgr2=0x7fde7c14d6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:41.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.312+0000 7fde83600640 1 --2- 172.21.15.28:0/4046941994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde7c1526b0 0x7fde7c14d6c0 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7fde740090b0 tx=0x7fde740046c0 comp rx=0 tx=0).stop 2024-02-21T23:08:41.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.313+0000 7fde83600640 1 -- 172.21.15.28:0/4046941994 shutdown_connections 2024-02-21T23:08:41.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.313+0000 7fde83600640 1 --2- 172.21.15.28:0/4046941994 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fde5806b4d0 0x7fde5806d990 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:41.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.313+0000 7fde83600640 1 --2- 172.21.15.28:0/4046941994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde7c1526b0 0x7fde7c14d6c0 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:41.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.313+0000 7fde83600640 1 --2- 172.21.15.28:0/4046941994 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde7c0cadc0 0x7fde7c150b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:41.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.313+0000 7fde83600640 1 --2- 172.21.15.28:0/4046941994 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fde7c0ca940 0x7fde7c1eea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:41.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.313+0000 7fde83600640 1 -- 172.21.15.28:0/4046941994 >> 172.21.15.28:0/4046941994 conn(0x7fde7c06d3d0 msgr2=0x7fde7c0c0470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:41.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.313+0000 7fde83600640 1 -- 172.21.15.28:0/4046941994 shutdown_connections 2024-02-21T23:08:41.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.313+0000 7fde83600640 1 -- 172.21.15.28:0/4046941994 wait complete. 2024-02-21T23:08:41.321 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:08:41.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.552+0000 7f7179a00640 1 Processor -- start 2024-02-21T23:08:41.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.552+0000 7f7179a00640 1 -- start start 2024-02-21T23:08:41.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.554+0000 7f7179a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f717414f0b0 0x7f7174155180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:41.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.554+0000 7f7179a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f71741521c0 0x7f7174152650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:41.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.554+0000 7f7179a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7174155760 con 0x7f71740a6160 2024-02-21T23:08:41.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.554+0000 7f7179a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7174151d30 con 0x7f717414f0b0 2024-02-21T23:08:41.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.554+0000 7f7179a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7174155a40 con 0x7f71741521c0 2024-02-21T23:08:41.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.554+0000 7f7173e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f717414f0b0 0x7f7174155180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:41.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.554+0000 7f7173400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f71740a6160 0x7f71740c2010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:51992/0 (socket says 172.21.15.28:51992) 2024-02-21T23:08:41.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.554+0000 7f7173400640 1 -- 172.21.15.28:0/271972619 learned_addr learned my addr 172.21.15.28:0/271972619 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:08:41.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.555+0000 7f7179000640 1 --2- 172.21.15.28:0/271972619 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f71741521c0 0x7f7174152650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:41.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.555+0000 7f7173e00640 1 -- 172.21.15.28:0/271972619 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f71741521c0 msgr2=0x7f7174152650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:41.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.555+0000 7f7173e00640 1 --2- 172.21.15.28:0/271972619 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f71741521c0 0x7f7174152650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:41.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.555+0000 7f7173e00640 1 -- 172.21.15.28:0/271972619 >> v1:172.21.15.28:6789/0 conn(0x7f71740a6160 legacy=0x7f71740c2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:41.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.555+0000 7f7173e00640 1 -- 172.21.15.28:0/271972619 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7174155d20 con 0x7f717414f0b0 2024-02-21T23:08:41.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.555+0000 7f7172a00640 1 -- 172.21.15.28:0/271972619 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3300301813 0 0) 0x7f7174155760 con 0x7f71740a6160 2024-02-21T23:08:41.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.557+0000 7f7173e00640 1 --2- 172.21.15.28:0/271972619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f717414f0b0 0x7f7174155180 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f7164009d90 tx=0x7f716402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=985e1be6b37dad9c server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:41.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.557+0000 7f7172a00640 1 -- 172.21.15.28:0/271972619 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7164039070 con 0x7f717414f0b0 2024-02-21T23:08:41.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.557+0000 7f7172a00640 1 -- 172.21.15.28:0/271972619 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f716402cc50 con 0x7f717414f0b0 2024-02-21T23:08:41.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.557+0000 7f7172a00640 1 -- 172.21.15.28:0/271972619 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7164034440 con 0x7f717414f0b0 2024-02-21T23:08:41.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.558+0000 7f7179a00640 1 -- 172.21.15.28:0/271972619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f717414f0b0 msgr2=0x7f7174155180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:41.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.558+0000 7f7179a00640 1 --2- 172.21.15.28:0/271972619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f717414f0b0 0x7f7174155180 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f7164009d90 tx=0x7f716402bea0 comp rx=0 tx=0).stop 2024-02-21T23:08:41.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.558+0000 7f7179a00640 1 -- 172.21.15.28:0/271972619 shutdown_connections 2024-02-21T23:08:41.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.558+0000 7f7179a00640 1 --2- 172.21.15.28:0/271972619 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f71741521c0 0x7f7174152650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:41.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.558+0000 7f7179a00640 1 --2- 172.21.15.28:0/271972619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f717414f0b0 0x7f7174155180 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:41.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.558+0000 7f7179a00640 1 -- 172.21.15.28:0/271972619 >> 172.21.15.28:0/271972619 conn(0x7f717406b630 msgr2=0x7f71740c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:41.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.558+0000 7f7179a00640 1 -- 172.21.15.28:0/271972619 shutdown_connections 2024-02-21T23:08:41.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.558+0000 7f7179a00640 1 -- 172.21.15.28:0/271972619 wait complete. 2024-02-21T23:08:41.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.560+0000 7f7179a00640 1 Processor -- start 2024-02-21T23:08:41.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.560+0000 7f7179a00640 1 -- start start 2024-02-21T23:08:41.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.561+0000 7f7179a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f71740a6160 0x7f71741ee830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:41.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.561+0000 7f7179a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f717414f0b0 0x7f717414c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:41.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.561+0000 7f7179a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71741521c0 0x7f717414ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:41.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.561+0000 7f7179a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71740c2720 con 0x7f717414f0b0 2024-02-21T23:08:41.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.561+0000 7f7179a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71740a68d0 con 0x7f71741521c0 2024-02-21T23:08:41.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.561+0000 7f7179a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71740c28a0 con 0x7f71740a6160 2024-02-21T23:08:41.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.561+0000 7f7173400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f717414f0b0 0x7f717414c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:41.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.561+0000 7f7173400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f717414f0b0 0x7f717414c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35974/0 (socket says 172.21.15.28:35974) 2024-02-21T23:08:41.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.561+0000 7f7173400640 1 -- 172.21.15.28:0/1877094159 learned_addr learned my addr 172.21.15.28:0/1877094159 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:41.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.562+0000 7f7173e00640 1 --2- 172.21.15.28:0/1877094159 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f71740a6160 0x7f71741ee830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:41.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.562+0000 7f7179000640 1 --2- 172.21.15.28:0/1877094159 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71741521c0 0x7f717414ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:41.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.562+0000 7f7173400640 1 -- 172.21.15.28:0/1877094159 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f71740a6160 msgr2=0x7f71741ee830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:41.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.562+0000 7f7173400640 1 --2- 172.21.15.28:0/1877094159 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f71740a6160 0x7f71741ee830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:41.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.562+0000 7f7173400640 1 -- 172.21.15.28:0/1877094159 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71741521c0 msgr2=0x7f717414ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:41.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.562+0000 7f7173400640 1 --2- 172.21.15.28:0/1877094159 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71741521c0 0x7f717414ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:41.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.562+0000 7f7173400640 1 -- 172.21.15.28:0/1877094159 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71741407f0 con 0x7f717414f0b0 2024-02-21T23:08:41.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.562+0000 7f7173e00640 1 --2- 172.21.15.28:0/1877094159 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f71740a6160 0x7f71741ee830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:08:41.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.562+0000 7f7179000640 1 --2- 172.21.15.28:0/1877094159 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71741521c0 0x7f717414ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:08:41.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.562+0000 7f7173400640 1 --2- 172.21.15.28:0/1877094159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f717414f0b0 0x7f717414c940 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f715c00c4d0 tx=0x7f715c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:41.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.562+0000 7f716be00640 1 -- 172.21.15.28:0/1877094159 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f715c018070 con 0x7f717414f0b0 2024-02-21T23:08:41.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.562+0000 7f716be00640 1 -- 172.21.15.28:0/1877094159 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f715c007500 con 0x7f717414f0b0 2024-02-21T23:08:41.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.562+0000 7f716be00640 1 -- 172.21.15.28:0/1877094159 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f715c004a20 con 0x7f717414f0b0 2024-02-21T23:08:41.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.562+0000 7f7179a00640 1 -- 172.21.15.28:0/1877094159 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71741ef220 con 0x7f717414f0b0 2024-02-21T23:08:41.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.563+0000 7f7179a00640 1 -- 172.21.15.28:0/1877094159 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71740a5e40 con 0x7f717414f0b0 2024-02-21T23:08:41.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.564+0000 7f7179a00640 1 -- 172.21.15.28:0/1877094159 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f717404f290 con 0x7f717414f0b0 2024-02-21T23:08:41.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.565+0000 7f716be00640 1 -- 172.21.15.28:0/1877094159 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f715c01b070 con 0x7f717414f0b0 2024-02-21T23:08:41.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.565+0000 7f716be00640 1 --2- 172.21.15.28:0/1877094159 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f715006b500 0x7f715006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:41.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.566+0000 7f716be00640 1 -- 172.21.15.28:0/1877094159 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f715c0896d0 con 0x7f717414f0b0 2024-02-21T23:08:41.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.566+0000 7f7173e00640 1 --2- 172.21.15.28:0/1877094159 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f715006b500 0x7f715006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:41.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.567+0000 7f7173e00640 1 --2- 172.21.15.28:0/1877094159 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f715006b500 0x7f715006d9c0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f7164036000 tx=0x7f7164030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:41.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.570+0000 7f716be00640 1 -- 172.21.15.28:0/1877094159 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f715c0583a0 con 0x7f717414f0b0 2024-02-21T23:08:41.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.811+0000 7f7179a00640 1 -- 172.21.15.28:0/1877094159 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7174068150 con 0x7f717414f0b0 2024-02-21T23:08:41.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.812+0000 7f716be00640 1 -- 172.21.15.28:0/1877094159 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 130 v130) v1 ==== 95+0+4947 (secure 0 0 0) 0x7f715c057ca0 con 0x7f717414f0b0 2024-02-21T23:08:41.800 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:08:41.800 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":130,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5948,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3556518453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3556518453},{"type":"v1","addr":"172.21.15.28:6835","nonce":3556518453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116},{"gid":5951,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/3293264689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":3293264689},{"type":"v1","addr":"172.21.15.81:6836","nonce":3293264689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116},{"gid":8157,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/1285325008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1285325008},{"type":"v1","addr":"172.21.15.28:6834","nonce":1285325008}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":128}],"filesystems":[{"mdsmap":{"epoch":130,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:08:13.883141+0000","modified":"2024-02-21T23:08:34.520926+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":124,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6634},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6634":{"gid":6634,"name":"d","rank":0,"incarnation":127,"state":"up:active","state_seq":4,"addr":"172.21.15.81:6837/3687210533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":3687210533},{"type":"v1","addr":"172.21.15.81:6837","nonce":3687210533}]},"join_fscid":9,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-02-21T23:08:41.800 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 130 2024-02-21T23:08:41.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.814+0000 7f7179a00640 1 -- 172.21.15.28:0/1877094159 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f715006b500 msgr2=0x7f715006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:41.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.814+0000 7f7179a00640 1 --2- 172.21.15.28:0/1877094159 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f715006b500 0x7f715006d9c0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f7164036000 tx=0x7f7164030000 comp rx=0 tx=0).stop 2024-02-21T23:08:41.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.814+0000 7f7179a00640 1 -- 172.21.15.28:0/1877094159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f717414f0b0 msgr2=0x7f717414c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:41.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.814+0000 7f7179a00640 1 --2- 172.21.15.28:0/1877094159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f717414f0b0 0x7f717414c940 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f715c00c4d0 tx=0x7f715c00c990 comp rx=0 tx=0).stop 2024-02-21T23:08:41.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.814+0000 7f7179a00640 1 -- 172.21.15.28:0/1877094159 shutdown_connections 2024-02-21T23:08:41.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.815+0000 7f7179a00640 1 --2- 172.21.15.28:0/1877094159 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f715006b500 0x7f715006d9c0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:41.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.815+0000 7f7179a00640 1 --2- 172.21.15.28:0/1877094159 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71741521c0 0x7f717414ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:41.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.815+0000 7f7179a00640 1 --2- 172.21.15.28:0/1877094159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f717414f0b0 0x7f717414c940 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:41.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.815+0000 7f7179a00640 1 --2- 172.21.15.28:0/1877094159 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f71740a6160 0x7f71741ee830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:41.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.815+0000 7f7179a00640 1 -- 172.21.15.28:0/1877094159 >> 172.21.15.28:0/1877094159 conn(0x7f717406b630 msgr2=0x7f7174070ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:41.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.815+0000 7f7179a00640 1 -- 172.21.15.28:0/1877094159 shutdown_connections 2024-02-21T23:08:41.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:41.815+0000 7f7179a00640 1 -- 172.21.15.28:0/1877094159 wait complete. 2024-02-21T23:08:41.823 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:08:42.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.051+0000 7f3d5e400640 1 Processor -- start 2024-02-21T23:08:42.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.051+0000 7f3d5e400640 1 -- start start 2024-02-21T23:08:42.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.053+0000 7f3d5e400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3d580aff90 0x7f3d580c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:42.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.053+0000 7f3d5e400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f3d580c5720 0x7f3d5815f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:42.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.053+0000 7f3d5e400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d580c74c0 con 0x7f3d5806dbf0 2024-02-21T23:08:42.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.053+0000 7f3d5e400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d580c7270 con 0x7f3d580c5720 2024-02-21T23:08:42.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.053+0000 7f3d5e400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d58158630 con 0x7f3d580aff90 2024-02-21T23:08:42.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.053+0000 7f3d5d000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3d580aff90 0x7f3d580c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:42.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.053+0000 7f3d5da00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f3d580c5720 0x7f3d5815f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:42.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.054+0000 7f3d5da00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f3d580c5720 0x7f3d5815f890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.28:55440/0 (socket says 172.21.15.28:55440) 2024-02-21T23:08:42.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.054+0000 7f3d5da00640 1 -- 172.21.15.28:0/1796769911 learned_addr learned my addr 172.21.15.28:0/1796769911 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:42.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.054+0000 7f3d57e00640 1 -- 172.21.15.28:0/1796769911 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3798636582 0 0) 0x7f3d580c74c0 con 0x7f3d5806dbf0 2024-02-21T23:08:42.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.054+0000 7f3d57e00640 1 -- 172.21.15.28:0/1796769911 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d3c003520 con 0x7f3d5806dbf0 2024-02-21T23:08:42.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.054+0000 7f3d5da00640 1 -- 172.21.15.28:0/1796769911 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3d580aff90 msgr2=0x7f3d580c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:42.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.054+0000 7f3d5da00640 1 --2- 172.21.15.28:0/1796769911 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3d580aff90 0x7f3d580c51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:42.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.054+0000 7f3d5da00640 1 -- 172.21.15.28:0/1796769911 >> v1:172.21.15.81:6789/0 conn(0x7f3d5806dbf0 legacy=0x7f3d58158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:42.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.054+0000 7f3d5da00640 1 -- 172.21.15.28:0/1796769911 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d580c77a0 con 0x7f3d580c5720 2024-02-21T23:08:42.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.054+0000 7f3d5d000640 1 --2- 172.21.15.28:0/1796769911 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3d580aff90 0x7f3d580c51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-21T23:08:42.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.056+0000 7f3d5da00640 1 --2- 172.21.15.28:0/1796769911 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d580c5720 0x7f3d5815f890 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f3d50009870 tx=0x7f3d5002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=91b0923777e19c1 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:42.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.056+0000 7f3d57e00640 1 -- 172.21.15.28:0/1796769911 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d50034070 con 0x7f3d580c5720 2024-02-21T23:08:42.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.056+0000 7f3d57e00640 1 -- 172.21.15.28:0/1796769911 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d50030840 con 0x7f3d580c5720 2024-02-21T23:08:42.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.056+0000 7f3d57e00640 1 -- 172.21.15.28:0/1796769911 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d50030c10 con 0x7f3d580c5720 2024-02-21T23:08:42.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.057+0000 7f3d5e400640 1 -- 172.21.15.28:0/1796769911 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d580c5720 msgr2=0x7f3d5815f890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:42.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.057+0000 7f3d5e400640 1 --2- 172.21.15.28:0/1796769911 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d580c5720 0x7f3d5815f890 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f3d50009870 tx=0x7f3d5002b980 comp rx=0 tx=0).stop 2024-02-21T23:08:42.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.057+0000 7f3d5e400640 1 -- 172.21.15.28:0/1796769911 shutdown_connections 2024-02-21T23:08:42.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.057+0000 7f3d5e400640 1 --2- 172.21.15.28:0/1796769911 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d580c5720 0x7f3d5815f890 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:42.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.057+0000 7f3d5e400640 1 --2- 172.21.15.28:0/1796769911 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3d580aff90 0x7f3d580c51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:42.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.057+0000 7f3d5e400640 1 -- 172.21.15.28:0/1796769911 >> 172.21.15.28:0/1796769911 conn(0x7f3d580ac370 msgr2=0x7f3d580ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:42.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.057+0000 7f3d5e400640 1 -- 172.21.15.28:0/1796769911 shutdown_connections 2024-02-21T23:08:42.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.058+0000 7f3d5e400640 1 -- 172.21.15.28:0/1796769911 wait complete. 2024-02-21T23:08:42.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.060+0000 7f3d5e400640 1 Processor -- start 2024-02-21T23:08:42.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.060+0000 7f3d5e400640 1 -- start start 2024-02-21T23:08:42.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.061+0000 7f3d5e400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3d5806dbf0 0x7f3d580aebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:42.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.061+0000 7f3d5e400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d580aff90 0x7f3d580bd250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:42.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.061+0000 7f3d5e400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d580c5720 0x7f3d580bd790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:42.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.061+0000 7f3d5e400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d580c8760 con 0x7f3d580c5720 2024-02-21T23:08:42.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.061+0000 7f3d5e400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d580be0c0 con 0x7f3d580aff90 2024-02-21T23:08:42.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.061+0000 7f3d5e400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d580c88e0 con 0x7f3d5806dbf0 2024-02-21T23:08:42.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.062+0000 7f3d5da00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d580c5720 0x7f3d580bd790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:42.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.062+0000 7f3d5da00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d580c5720 0x7f3d580bd790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35988/0 (socket says 172.21.15.28:35988) 2024-02-21T23:08:42.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.062+0000 7f3d5da00640 1 -- 172.21.15.28:0/2003648776 learned_addr learned my addr 172.21.15.28:0/2003648776 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:42.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.062+0000 7f3d5d000640 1 --2- 172.21.15.28:0/2003648776 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3d5806dbf0 0x7f3d580aebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:42.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.062+0000 7f3d4fe00640 1 --2- 172.21.15.28:0/2003648776 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d580aff90 0x7f3d580bd250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:42.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.062+0000 7f3d5da00640 1 -- 172.21.15.28:0/2003648776 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3d5806dbf0 msgr2=0x7f3d580aebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:42.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.062+0000 7f3d5da00640 1 --2- 172.21.15.28:0/2003648776 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3d5806dbf0 0x7f3d580aebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:42.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.062+0000 7f3d5da00640 1 -- 172.21.15.28:0/2003648776 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d580aff90 msgr2=0x7f3d580bd250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:42.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.062+0000 7f3d5da00640 1 --2- 172.21.15.28:0/2003648776 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d580aff90 0x7f3d580bd250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:42.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.062+0000 7f3d5da00640 1 -- 172.21.15.28:0/2003648776 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d581f3050 con 0x7f3d580c5720 2024-02-21T23:08:42.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.063+0000 7f3d5da00640 1 --2- 172.21.15.28:0/2003648776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d580c5720 0x7f3d580bd790 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f3d50009870 tx=0x7f3d500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:42.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.063+0000 7f3d55600640 1 -- 172.21.15.28:0/2003648776 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d50034040 con 0x7f3d580c5720 2024-02-21T23:08:42.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.063+0000 7f3d55600640 1 -- 172.21.15.28:0/2003648776 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d5003aca0 con 0x7f3d580c5720 2024-02-21T23:08:42.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.063+0000 7f3d5e400640 1 -- 172.21.15.28:0/2003648776 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d581f3330 con 0x7f3d580c5720 2024-02-21T23:08:42.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.063+0000 7f3d55600640 1 -- 172.21.15.28:0/2003648776 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d50044630 con 0x7f3d580c5720 2024-02-21T23:08:42.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.063+0000 7f3d5e400640 1 -- 172.21.15.28:0/2003648776 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d580af0f0 con 0x7f3d580c5720 2024-02-21T23:08:42.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.064+0000 7f3d5e400640 1 -- 172.21.15.28:0/2003648776 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d580ca530 con 0x7f3d580c5720 2024-02-21T23:08:42.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.065+0000 7f3d55600640 1 -- 172.21.15.28:0/2003648776 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3d50037070 con 0x7f3d580c5720 2024-02-21T23:08:42.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.067+0000 7f3d55600640 1 --2- 172.21.15.28:0/2003648776 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3d3406b500 0x7f3d3406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:42.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.067+0000 7f3d55600640 1 -- 172.21.15.28:0/2003648776 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f3d500ac390 con 0x7f3d580c5720 2024-02-21T23:08:42.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.067+0000 7f3d5d000640 1 --2- 172.21.15.28:0/2003648776 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3d3406b500 0x7f3d3406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:42.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.068+0000 7f3d5d000640 1 --2- 172.21.15.28:0/2003648776 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3d3406b500 0x7f3d3406d9c0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f3d48009bd0 tx=0x7f3d4800b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:42.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.072+0000 7f3d55600640 1 -- 172.21.15.28:0/2003648776 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3d580ca530 con 0x7f3d580c5720 2024-02-21T23:08:42.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.277+0000 7f3d5e400640 1 -- 172.21.15.28:0/2003648776 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3d5804f290 con 0x7f3d580c5720 2024-02-21T23:08:42.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.280+0000 7f3d55600640 1 -- 172.21.15.28:0/2003648776 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v124) v1 ==== 74+0+17099 (secure 0 0 0) 0x7f3d5003f070 con 0x7f3d580c5720 2024-02-21T23:08:42.269 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:08:42.269 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":124,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:08:32.504818+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:08:11.794800+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"121","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2024-02-21T23:08:12.918602+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"122","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6834/1773560593":"2024-02-22T23:08:32.504770+0000","172.21.15.28:6832/1773560593":"2024-02-22T23:08:32.504770+0000","172.21.15.81:6837/1038096496":"2024-02-22T23:08:30.184500+0000","172.21.15.81:6835/1038096496":"2024-02-22T23:08:30.184500+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:08:42.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.284+0000 7f3d5e400640 1 -- 172.21.15.28:0/2003648776 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3d3406b500 msgr2=0x7f3d3406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:42.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.284+0000 7f3d5e400640 1 --2- 172.21.15.28:0/2003648776 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3d3406b500 0x7f3d3406d9c0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f3d48009bd0 tx=0x7f3d4800b040 comp rx=0 tx=0).stop 2024-02-21T23:08:42.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.284+0000 7f3d5e400640 1 -- 172.21.15.28:0/2003648776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d580c5720 msgr2=0x7f3d580bd790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:42.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.284+0000 7f3d5e400640 1 --2- 172.21.15.28:0/2003648776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d580c5720 0x7f3d580bd790 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f3d50009870 tx=0x7f3d500046c0 comp rx=0 tx=0).stop 2024-02-21T23:08:42.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.284+0000 7f3d5e400640 1 -- 172.21.15.28:0/2003648776 shutdown_connections 2024-02-21T23:08:42.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.284+0000 7f3d5e400640 1 --2- 172.21.15.28:0/2003648776 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3d3406b500 0x7f3d3406d9c0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:42.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.284+0000 7f3d5e400640 1 --2- 172.21.15.28:0/2003648776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d580c5720 0x7f3d580bd790 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:42.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.284+0000 7f3d5e400640 1 --2- 172.21.15.28:0/2003648776 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d580aff90 0x7f3d580bd250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:42.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.284+0000 7f3d5e400640 1 --2- 172.21.15.28:0/2003648776 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3d5806dbf0 0x7f3d580aebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:42.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.284+0000 7f3d5e400640 1 -- 172.21.15.28:0/2003648776 >> 172.21.15.28:0/2003648776 conn(0x7f3d580ac370 msgr2=0x7f3d580ac750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:42.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.284+0000 7f3d5e400640 1 -- 172.21.15.28:0/2003648776 shutdown_connections 2024-02-21T23:08:42.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.284+0000 7f3d5e400640 1 -- 172.21.15.28:0/2003648776 wait complete. 2024-02-21T23:08:42.292 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-02-21T23:08:42.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.527+0000 7fbb35800640 1 Processor -- start 2024-02-21T23:08:42.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.527+0000 7fbb35800640 1 -- start start 2024-02-21T23:08:42.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.528+0000 7fbb35800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbb300c5040 0x7fbb30155a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:42.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.528+0000 7fbb35800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbb300c85c0 0x7fbb300c70d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:42.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.528+0000 7fbb35800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb3006df50 con 0x7fbb301524c0 2024-02-21T23:08:42.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.528+0000 7fbb35800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb300c5520 con 0x7fbb300c5040 2024-02-21T23:08:42.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.528+0000 7fbb35800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb300c5800 con 0x7fbb300c85c0 2024-02-21T23:08:42.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.529+0000 7fbb2f400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbb300c5040 0x7fbb30155a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:42.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.529+0000 7fbb2f400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbb300c5040 0x7fbb30155a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35992/0 (socket says 172.21.15.28:35992) 2024-02-21T23:08:42.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.529+0000 7fbb2f400640 1 -- 172.21.15.28:0/359479176 learned_addr learned my addr 172.21.15.28:0/359479176 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:42.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.529+0000 7fbb34e00640 1 --2- 172.21.15.28:0/359479176 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbb300c85c0 0x7fbb300c70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:42.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.529+0000 7fbb2f400640 1 -- 172.21.15.28:0/359479176 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbb300c85c0 msgr2=0x7fbb300c70d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:42.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.529+0000 7fbb2f400640 1 --2- 172.21.15.28:0/359479176 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbb300c85c0 0x7fbb300c70d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:42.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.529+0000 7fbb2f400640 1 -- 172.21.15.28:0/359479176 >> v1:172.21.15.81:6789/0 conn(0x7fbb301524c0 legacy=0x7fbb3006ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:42.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.529+0000 7fbb2f400640 1 -- 172.21.15.28:0/359479176 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb3006c670 con 0x7fbb300c5040 2024-02-21T23:08:42.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.530+0000 7fbb2f400640 1 --2- 172.21.15.28:0/359479176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb300c5040 0x7fbb30155a10 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7fbb18009870 tx=0x7fbb1802b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60b0ed667ca17ce8 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:42.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.531+0000 7fbb2ea00640 1 -- 172.21.15.28:0/359479176 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb18038070 con 0x7fbb300c5040 2024-02-21T23:08:42.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.531+0000 7fbb2ea00640 1 -- 172.21.15.28:0/359479176 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb18031740 con 0x7fbb300c5040 2024-02-21T23:08:42.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.531+0000 7fbb2ea00640 1 -- 172.21.15.28:0/359479176 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb1802f050 con 0x7fbb300c5040 2024-02-21T23:08:42.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.531+0000 7fbb35800640 1 -- 172.21.15.28:0/359479176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb300c5040 msgr2=0x7fbb30155a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:42.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.531+0000 7fbb35800640 1 --2- 172.21.15.28:0/359479176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb300c5040 0x7fbb30155a10 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7fbb18009870 tx=0x7fbb1802b7e0 comp rx=0 tx=0).stop 2024-02-21T23:08:42.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.531+0000 7fbb35800640 1 -- 172.21.15.28:0/359479176 shutdown_connections 2024-02-21T23:08:42.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.531+0000 7fbb35800640 1 --2- 172.21.15.28:0/359479176 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbb300c85c0 0x7fbb300c70d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:42.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.531+0000 7fbb35800640 1 --2- 172.21.15.28:0/359479176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb300c5040 0x7fbb30155a10 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:42.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.531+0000 7fbb35800640 1 -- 172.21.15.28:0/359479176 >> 172.21.15.28:0/359479176 conn(0x7fbb300ac2b0 msgr2=0x7fbb300ac6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:42.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.531+0000 7fbb35800640 1 -- 172.21.15.28:0/359479176 shutdown_connections 2024-02-21T23:08:42.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.531+0000 7fbb35800640 1 -- 172.21.15.28:0/359479176 wait complete. 2024-02-21T23:08:42.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.533+0000 7fbb35800640 1 Processor -- start 2024-02-21T23:08:42.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.533+0000 7fbb35800640 1 -- start start 2024-02-21T23:08:42.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.534+0000 7fbb35800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb300c5040 0x7fbb30149470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:42.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.534+0000 7fbb35800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbb300c85c0 0x7fbb300ada00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:42.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.534+0000 7fbb35800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb301524c0 0x7fbb300c3a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:42.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.534+0000 7fbb35800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb300c8a70 con 0x7fbb301524c0 2024-02-21T23:08:42.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.534+0000 7fbb35800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb301528a0 con 0x7fbb300c5040 2024-02-21T23:08:42.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.534+0000 7fbb35800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb300c7610 con 0x7fbb300c85c0 2024-02-21T23:08:42.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.534+0000 7fbb34e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb301524c0 0x7fbb300c3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:42.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.534+0000 7fbb34e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb301524c0 0x7fbb300c3a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36006/0 (socket says 172.21.15.28:36006) 2024-02-21T23:08:42.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.534+0000 7fbb34e00640 1 -- 172.21.15.28:0/1725368167 learned_addr learned my addr 172.21.15.28:0/1725368167 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:42.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.535+0000 7fbb2fe00640 1 --2- 172.21.15.28:0/1725368167 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb300c5040 0x7fbb30149470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:42.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.535+0000 7fbb2f400640 1 --2- 172.21.15.28:0/1725368167 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbb300c85c0 0x7fbb300ada00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:42.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.535+0000 7fbb34e00640 1 -- 172.21.15.28:0/1725368167 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbb300c85c0 msgr2=0x7fbb300ada00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:42.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.535+0000 7fbb34e00640 1 --2- 172.21.15.28:0/1725368167 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbb300c85c0 0x7fbb300ada00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:42.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.535+0000 7fbb34e00640 1 -- 172.21.15.28:0/1725368167 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb300c5040 msgr2=0x7fbb30149470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:42.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.535+0000 7fbb34e00640 1 --2- 172.21.15.28:0/1725368167 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb300c5040 0x7fbb30149470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:42.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.535+0000 7fbb34e00640 1 -- 172.21.15.28:0/1725368167 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb301eb930 con 0x7fbb301524c0 2024-02-21T23:08:42.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.535+0000 7fbb34e00640 1 --2- 172.21.15.28:0/1725368167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb301524c0 0x7fbb300c3a90 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7fbb2800cd70 tx=0x7fbb2800a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:42.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.535+0000 7fbb27e00640 1 -- 172.21.15.28:0/1725368167 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb28003b90 con 0x7fbb301524c0 2024-02-21T23:08:42.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.535+0000 7fbb27e00640 1 -- 172.21.15.28:0/1725368167 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb28003d30 con 0x7fbb301524c0 2024-02-21T23:08:42.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.535+0000 7fbb27e00640 1 -- 172.21.15.28:0/1725368167 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb28004240 con 0x7fbb301524c0 2024-02-21T23:08:42.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.536+0000 7fbb35800640 1 -- 172.21.15.28:0/1725368167 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb300c3290 con 0x7fbb301524c0 2024-02-21T23:08:42.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.536+0000 7fbb35800640 1 -- 172.21.15.28:0/1725368167 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb301eb600 con 0x7fbb301524c0 2024-02-21T23:08:42.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.537+0000 7fbb35800640 1 -- 172.21.15.28:0/1725368167 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb300c3080 con 0x7fbb301524c0 2024-02-21T23:08:42.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.538+0000 7fbb27e00640 1 -- 172.21.15.28:0/1725368167 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbb280043e0 con 0x7fbb301524c0 2024-02-21T23:08:42.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.539+0000 7fbb27e00640 1 --2- 172.21.15.28:0/1725368167 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbb0c06b4d0 0x7fbb0c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:42.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.539+0000 7fbb27e00640 1 -- 172.21.15.28:0/1725368167 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 6273+0+0 (secure 0 0 0) 0x7fbb2808afd0 con 0x7fbb301524c0 2024-02-21T23:08:42.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.539+0000 7fbb2fe00640 1 --2- 172.21.15.28:0/1725368167 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbb0c06b4d0 0x7fbb0c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:42.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.540+0000 7fbb2fe00640 1 --2- 172.21.15.28:0/1725368167 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbb0c06b4d0 0x7fbb0c06d990 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fbb20006fd0 tx=0x7fbb20008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:42.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.543+0000 7fbb27e00640 1 -- 172.21.15.28:0/1725368167 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fbb2801d340 con 0x7fbb301524c0 2024-02-21T23:08:42.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.785+0000 7fbb35800640 1 -- 172.21.15.28:0/1725368167 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fbb3004f290 con 0x7fbb301524c0 2024-02-21T23:08:42.773 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-02-21T23:08:42.786+0000 7f5ddc400640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-21T23:08:42.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.789+0000 7fbb27e00640 1 -- 172.21.15.28:0/1725368167 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v131) v1 ==== 161+0+0 (secure 0 0 0) 0x7fbb28059c00 con 0x7fbb301524c0 2024-02-21T23:08:42.777 INFO:teuthology.orchestra.run.smithi028.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-02-21T23:08:42.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.791+0000 7fbb35800640 1 -- 172.21.15.28:0/1725368167 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbb0c06b4d0 msgr2=0x7fbb0c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:42.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.791+0000 7fbb35800640 1 --2- 172.21.15.28:0/1725368167 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbb0c06b4d0 0x7fbb0c06d990 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fbb20006fd0 tx=0x7fbb20008040 comp rx=0 tx=0).stop 2024-02-21T23:08:42.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.791+0000 7fbb35800640 1 -- 172.21.15.28:0/1725368167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb301524c0 msgr2=0x7fbb300c3a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:42.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.791+0000 7fbb35800640 1 --2- 172.21.15.28:0/1725368167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb301524c0 0x7fbb300c3a90 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7fbb2800cd70 tx=0x7fbb2800a660 comp rx=0 tx=0).stop 2024-02-21T23:08:42.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.792+0000 7fbb35800640 1 -- 172.21.15.28:0/1725368167 shutdown_connections 2024-02-21T23:08:42.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.792+0000 7fbb35800640 1 --2- 172.21.15.28:0/1725368167 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbb0c06b4d0 0x7fbb0c06d990 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:42.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.792+0000 7fbb35800640 1 --2- 172.21.15.28:0/1725368167 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbb300c85c0 0x7fbb300ada00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:42.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.792+0000 7fbb35800640 1 --2- 172.21.15.28:0/1725368167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb301524c0 0x7fbb300c3a90 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:42.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.792+0000 7fbb35800640 1 --2- 172.21.15.28:0/1725368167 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb300c5040 0x7fbb30149470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:42.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.792+0000 7fbb35800640 1 -- 172.21.15.28:0/1725368167 >> 172.21.15.28:0/1725368167 conn(0x7fbb300ac2b0 msgr2=0x7fbb30154e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:42.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.792+0000 7fbb35800640 1 -- 172.21.15.28:0/1725368167 shutdown_connections 2024-02-21T23:08:42.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:42.792+0000 7fbb35800640 1 -- 172.21.15.28:0/1725368167 wait complete. 2024-02-21T23:08:42.797 INFO:tasks.ceph.mds.d.smithi081.stderr: -257> 2024-02-21T23:08:36.216+0000 7f8efe800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-21T23:08:42.800 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-02-21T23:08:42.862 INFO:tasks.ceph.mds.d.smithi081.stdout:starting mds.d at 2024-02-21T23:08:43.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.022+0000 7f90db800640 1 Processor -- start 2024-02-21T23:08:43.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.022+0000 7f90db800640 1 -- start start 2024-02-21T23:08:43.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.024+0000 7f90db800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f90d406b0e0 0x7f90d40c5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:43.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.024+0000 7f90db800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f90d4150700 0x7f90d4156500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:43.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.024+0000 7f90db800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90d40c61e0 con 0x7f90d40c5d80 2024-02-21T23:08:43.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.024+0000 7f90db800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90d4153bc0 con 0x7f90d406b0e0 2024-02-21T23:08:43.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.024+0000 7f90db800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90d4152fb0 con 0x7f90d4150700 2024-02-21T23:08:43.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.024+0000 7f90d8e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f90d406b0e0 0x7f90d40c5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:43.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.024+0000 7f90d8e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f90d406b0e0 0x7f90d40c5840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36016/0 (socket says 172.21.15.28:36016) 2024-02-21T23:08:43.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.024+0000 7f90d8e00640 1 -- 172.21.15.28:0/4091797635 learned_addr learned my addr 172.21.15.28:0/4091797635 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:43.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.024+0000 7f90d9800640 1 --2- 172.21.15.28:0/4091797635 >> v2:172.21.15.81:3300/0 conn(0x7f90d4150700 0x7f90d4156500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:43.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.025+0000 7f90d8e00640 1 -- 172.21.15.28:0/4091797635 >> v2:172.21.15.81:3300/0 conn(0x7f90d4150700 msgr2=0x7f90d4156500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:43.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.025+0000 7f90d8e00640 1 --2- 172.21.15.28:0/4091797635 >> v2:172.21.15.81:3300/0 conn(0x7f90d4150700 0x7f90d4156500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:43.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.025+0000 7f90d8e00640 1 -- 172.21.15.28:0/4091797635 >> v1:172.21.15.81:6789/0 conn(0x7f90d40c5d80 legacy=0x7f90d414fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:43.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.025+0000 7f90d8e00640 1 -- 172.21.15.28:0/4091797635 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90d4153290 con 0x7f90d406b0e0 2024-02-21T23:08:43.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.026+0000 7f90d8e00640 1 --2- 172.21.15.28:0/4091797635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90d406b0e0 0x7f90d40c5840 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f90d40a64b0 tx=0x7f90c402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72b153b1e5180558 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:43.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.026+0000 7f90d3400640 1 -- 172.21.15.28:0/4091797635 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c402e070 con 0x7f90d406b0e0 2024-02-21T23:08:43.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.027+0000 7f90d3400640 1 -- 172.21.15.28:0/4091797635 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90c4037070 con 0x7f90d406b0e0 2024-02-21T23:08:43.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.027+0000 7f90d3400640 1 -- 172.21.15.28:0/4091797635 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c4031a00 con 0x7f90d406b0e0 2024-02-21T23:08:43.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.027+0000 7f90db800640 1 -- 172.21.15.28:0/4091797635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90d406b0e0 msgr2=0x7f90d40c5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:43.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.027+0000 7f90db800640 1 --2- 172.21.15.28:0/4091797635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90d406b0e0 0x7f90d40c5840 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f90d40a64b0 tx=0x7f90c402b880 comp rx=0 tx=0).stop 2024-02-21T23:08:43.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.027+0000 7f90db800640 1 -- 172.21.15.28:0/4091797635 shutdown_connections 2024-02-21T23:08:43.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.027+0000 7f90db800640 1 --2- 172.21.15.28:0/4091797635 >> v2:172.21.15.81:3300/0 conn(0x7f90d4150700 0x7f90d4156500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:43.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.027+0000 7f90db800640 1 --2- 172.21.15.28:0/4091797635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90d406b0e0 0x7f90d40c5840 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:43.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.027+0000 7f90db800640 1 -- 172.21.15.28:0/4091797635 >> 172.21.15.28:0/4091797635 conn(0x7f90d40a85a0 msgr2=0x7f90d40a8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:43.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.028+0000 7f90db800640 1 -- 172.21.15.28:0/4091797635 shutdown_connections 2024-02-21T23:08:43.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.028+0000 7f90db800640 1 -- 172.21.15.28:0/4091797635 wait complete. 2024-02-21T23:08:43.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.029+0000 7f90db800640 1 Processor -- start 2024-02-21T23:08:43.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.029+0000 7f90db800640 1 -- start start 2024-02-21T23:08:43.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.030+0000 7f90db800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f90d406b0e0 0x7f90d40a98a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:43.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.030+0000 7f90db800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90d40c5d80 0x7f90d41523a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:43.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.030+0000 7f90db800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f90d4150700 0x7f90d406ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:43.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.030+0000 7f90db800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90d406b4c0 con 0x7f90d40c5d80 2024-02-21T23:08:43.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.030+0000 7f90db800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90d4153850 con 0x7f90d406b0e0 2024-02-21T23:08:43.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.030+0000 7f90db800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90d406b640 con 0x7f90d4150700 2024-02-21T23:08:43.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.031+0000 7f90d3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90d40c5d80 0x7f90d41523a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:43.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.031+0000 7f90d3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90d40c5d80 0x7f90d41523a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36032/0 (socket says 172.21.15.28:36032) 2024-02-21T23:08:43.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.031+0000 7f90d3e00640 1 -- 172.21.15.28:0/2183926749 learned_addr learned my addr 172.21.15.28:0/2183926749 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:43.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.031+0000 7f90d9800640 1 --2- 172.21.15.28:0/2183926749 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f90d4150700 0x7f90d406ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:43.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.031+0000 7f90d8e00640 1 --2- 172.21.15.28:0/2183926749 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f90d406b0e0 0x7f90d40a98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:43.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.031+0000 7f90d3e00640 1 -- 172.21.15.28:0/2183926749 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f90d4150700 msgr2=0x7f90d406ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:43.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.031+0000 7f90d3e00640 1 --2- 172.21.15.28:0/2183926749 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f90d4150700 0x7f90d406ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:43.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.031+0000 7f90d3e00640 1 -- 172.21.15.28:0/2183926749 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f90d406b0e0 msgr2=0x7f90d40a98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:43.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.031+0000 7f90d3e00640 1 --2- 172.21.15.28:0/2183926749 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f90d406b0e0 0x7f90d40a98a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:43.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.031+0000 7f90d3e00640 1 -- 172.21.15.28:0/2183926749 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90d41528e0 con 0x7f90d40c5d80 2024-02-21T23:08:43.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.031+0000 7f90d9800640 1 --2- 172.21.15.28:0/2183926749 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f90d4150700 0x7f90d406ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:08:43.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.031+0000 7f90d8e00640 1 --2- 172.21.15.28:0/2183926749 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f90d406b0e0 0x7f90d40a98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:08:43.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.031+0000 7f90d3e00640 1 --2- 172.21.15.28:0/2183926749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90d40c5d80 0x7f90d41523a0 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f90c800c430 tx=0x7f90c800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:43.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.032+0000 7f90d0c00640 1 -- 172.21.15.28:0/2183926749 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c8011790 con 0x7f90d40c5d80 2024-02-21T23:08:43.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.032+0000 7f90d0c00640 1 -- 172.21.15.28:0/2183926749 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90c8017070 con 0x7f90d40c5d80 2024-02-21T23:08:43.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.032+0000 7f90d0c00640 1 -- 172.21.15.28:0/2183926749 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c8011bb0 con 0x7f90d40c5d80 2024-02-21T23:08:43.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.032+0000 7f90db800640 1 -- 172.21.15.28:0/2183926749 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90d406f250 con 0x7f90d40c5d80 2024-02-21T23:08:43.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.032+0000 7f90db800640 1 -- 172.21.15.28:0/2183926749 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90d41eeb20 con 0x7f90d40c5d80 2024-02-21T23:08:43.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.033+0000 7f90db800640 1 -- 172.21.15.28:0/2183926749 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90d4067f30 con 0x7f90d40c5d80 2024-02-21T23:08:43.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.034+0000 7f90d0c00640 1 -- 172.21.15.28:0/2183926749 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f90c801c460 con 0x7f90d40c5d80 2024-02-21T23:08:43.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.035+0000 7f90d0c00640 1 --2- 172.21.15.28:0/2183926749 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f90b806b4d0 0x7f90b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:43.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.035+0000 7f90d0c00640 1 -- 172.21.15.28:0/2183926749 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 6359+0+0 (secure 0 0 0) 0x7f90c808a390 con 0x7f90d40c5d80 2024-02-21T23:08:43.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.036+0000 7f90d8e00640 1 --2- 172.21.15.28:0/2183926749 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f90b806b4d0 0x7f90b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:43.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.037+0000 7f90d8e00640 1 --2- 172.21.15.28:0/2183926749 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f90b806b4d0 0x7f90b806d990 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f90c4036000 tx=0x7f90c4008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:43.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.039+0000 7f90d0c00640 1 -- 172.21.15.28:0/2183926749 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f90d406f250 con 0x7f90d40c5d80 2024-02-21T23:08:43.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.279+0000 7f90db800640 1 -- 172.21.15.28:0/2183926749 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f90d40a6760 con 0x7f90d40c5d80 2024-02-21T23:08:43.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.799+0000 7f90d0c00640 1 -- 172.21.15.28:0/2183926749 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v132) v1 ==== 104+0+0 (secure 0 0 0) 0x7f90c8058f70 con 0x7f90d40c5d80 2024-02-21T23:08:43.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.803+0000 7f90db800640 1 -- 172.21.15.28:0/2183926749 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f90b806b4d0 msgr2=0x7f90b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:43.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.803+0000 7f90db800640 1 --2- 172.21.15.28:0/2183926749 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f90b806b4d0 0x7f90b806d990 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f90c4036000 tx=0x7f90c4008030 comp rx=0 tx=0).stop 2024-02-21T23:08:43.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.803+0000 7f90db800640 1 -- 172.21.15.28:0/2183926749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90d40c5d80 msgr2=0x7f90d41523a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:43.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.803+0000 7f90db800640 1 --2- 172.21.15.28:0/2183926749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90d40c5d80 0x7f90d41523a0 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f90c800c430 tx=0x7f90c800c8f0 comp rx=0 tx=0).stop 2024-02-21T23:08:43.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.803+0000 7f90db800640 1 -- 172.21.15.28:0/2183926749 shutdown_connections 2024-02-21T23:08:43.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.803+0000 7f90db800640 1 --2- 172.21.15.28:0/2183926749 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f90b806b4d0 0x7f90b806d990 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:43.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.803+0000 7f90db800640 1 --2- 172.21.15.28:0/2183926749 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f90d4150700 0x7f90d406ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:43.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.803+0000 7f90db800640 1 --2- 172.21.15.28:0/2183926749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90d40c5d80 0x7f90d41523a0 unknown :-1 s=CLOSED pgs=1463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:43.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.803+0000 7f90db800640 1 --2- 172.21.15.28:0/2183926749 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f90d406b0e0 0x7f90d40a98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:43.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.803+0000 7f90db800640 1 -- 172.21.15.28:0/2183926749 >> 172.21.15.28:0/2183926749 conn(0x7f90d40a85a0 msgr2=0x7f90d414bda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:43.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.804+0000 7f90db800640 1 -- 172.21.15.28:0/2183926749 shutdown_connections 2024-02-21T23:08:43.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:43.804+0000 7f90db800640 1 -- 172.21.15.28:0/2183926749 wait complete. 2024-02-21T23:08:43.813 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-02-21T23:08:44.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.054+0000 7fa2c7e00640 1 Processor -- start 2024-02-21T23:08:44.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.054+0000 7fa2c7e00640 1 -- start start 2024-02-21T23:08:44.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.056+0000 7fa2c7e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa2c006e780 0x7fa2c006eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:44.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.056+0000 7fa2c7e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa2c00a5fc0 0x7fa2c00c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:44.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.056+0000 7fa2c7e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2c0153cb0 con 0x7fa2c00c50c0 2024-02-21T23:08:44.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.056+0000 7fa2c7e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2c01543d0 con 0x7fa2c00a5fc0 2024-02-21T23:08:44.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.056+0000 7fa2c7e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2c00c5ad0 con 0x7fa2c006e780 2024-02-21T23:08:44.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.056+0000 7fa2c4a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa2c00a5fc0 0x7fa2c00c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:44.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.056+0000 7fa2c5e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa2c00c50c0 0x7fa2c01564d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:52002/0 (socket says 172.21.15.28:52002) 2024-02-21T23:08:44.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.056+0000 7fa2c5e00640 1 -- 172.21.15.28:0/3318294021 learned_addr learned my addr 172.21.15.28:0/3318294021 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:08:44.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.057+0000 7fa2c5400640 1 --2- 172.21.15.28:0/3318294021 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa2c006e780 0x7fa2c006eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:44.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.057+0000 7fa2c4a00640 1 -- 172.21.15.28:0/3318294021 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa2c006e780 msgr2=0x7fa2c006eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:44.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.057+0000 7fa2c4a00640 1 --2- 172.21.15.28:0/3318294021 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa2c006e780 0x7fa2c006eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:44.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.057+0000 7fa2c4a00640 1 -- 172.21.15.28:0/3318294021 >> v1:172.21.15.28:6789/0 conn(0x7fa2c00c50c0 legacy=0x7fa2c01564d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:44.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.057+0000 7fa2c4a00640 1 -- 172.21.15.28:0/3318294021 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2c00c5d20 con 0x7fa2c00a5fc0 2024-02-21T23:08:44.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.058+0000 7fa2c4a00640 1 --2- 172.21.15.28:0/3318294021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2c00a5fc0 0x7fa2c00c4b80 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7fa2ac009790 tx=0x7fa2ac02b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6491b0741f6a0cef server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:44.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.058+0000 7fa2bfe00640 1 -- 172.21.15.28:0/3318294021 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2ac031780 con 0x7fa2c00a5fc0 2024-02-21T23:08:44.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.058+0000 7fa2bfe00640 1 -- 172.21.15.28:0/3318294021 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2ac037070 con 0x7fa2c00a5fc0 2024-02-21T23:08:44.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.058+0000 7fa2bfe00640 1 -- 172.21.15.28:0/3318294021 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2ac031b40 con 0x7fa2c00a5fc0 2024-02-21T23:08:44.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.059+0000 7fa2c7e00640 1 -- 172.21.15.28:0/3318294021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2c00a5fc0 msgr2=0x7fa2c00c4b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:44.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.059+0000 7fa2c7e00640 1 --2- 172.21.15.28:0/3318294021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2c00a5fc0 0x7fa2c00c4b80 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7fa2ac009790 tx=0x7fa2ac02b820 comp rx=0 tx=0).stop 2024-02-21T23:08:44.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.059+0000 7fa2c7e00640 1 -- 172.21.15.28:0/3318294021 shutdown_connections 2024-02-21T23:08:44.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.059+0000 7fa2c7e00640 1 --2- 172.21.15.28:0/3318294021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2c00a5fc0 0x7fa2c00c4b80 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:44.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.059+0000 7fa2c7e00640 1 --2- 172.21.15.28:0/3318294021 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa2c006e780 0x7fa2c006eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:44.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.059+0000 7fa2c7e00640 1 -- 172.21.15.28:0/3318294021 >> 172.21.15.28:0/3318294021 conn(0x7fa2c006b830 msgr2=0x7fa2c00c67f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:44.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.059+0000 7fa2c7e00640 1 -- 172.21.15.28:0/3318294021 shutdown_connections 2024-02-21T23:08:44.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.059+0000 7fa2c7e00640 1 -- 172.21.15.28:0/3318294021 wait complete. 2024-02-21T23:08:44.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.061+0000 7fa2c7e00640 1 Processor -- start 2024-02-21T23:08:44.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.061+0000 7fa2c7e00640 1 -- start start 2024-02-21T23:08:44.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.062+0000 7fa2c7e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa2c006e780 0x7fa2c00bfe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:44.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.062+0000 7fa2c7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2c00a5fc0 0x7fa2c01ead00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:44.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.062+0000 7fa2c7e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa2c00c50c0 0x7fa2c01eb240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:44.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.062+0000 7fa2c7e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2c0153960 con 0x7fa2c00a5fc0 2024-02-21T23:08:44.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.062+0000 7fa2c7e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2c00c46f0 con 0x7fa2c006e780 2024-02-21T23:08:44.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.062+0000 7fa2c7e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2c006d110 con 0x7fa2c00c50c0 2024-02-21T23:08:44.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.062+0000 7fa2c4a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2c00a5fc0 0x7fa2c01ead00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:44.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.062+0000 7fa2c4a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2c00a5fc0 0x7fa2c01ead00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36048/0 (socket says 172.21.15.28:36048) 2024-02-21T23:08:44.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.062+0000 7fa2c4a00640 1 -- 172.21.15.28:0/1307262184 learned_addr learned my addr 172.21.15.28:0/1307262184 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:44.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.063+0000 7fa2c5400640 1 --2- 172.21.15.28:0/1307262184 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa2c006e780 0x7fa2c00bfe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:44.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.063+0000 7fa2c4a00640 1 -- 172.21.15.28:0/1307262184 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa2c00c50c0 msgr2=0x7fa2c01eb240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:44.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.063+0000 7fa2c5e00640 1 --2- 172.21.15.28:0/1307262184 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa2c00c50c0 0x7fa2c01eb240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:44.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.063+0000 7fa2c4a00640 1 --2- 172.21.15.28:0/1307262184 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa2c00c50c0 0x7fa2c01eb240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:44.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.063+0000 7fa2c4a00640 1 -- 172.21.15.28:0/1307262184 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa2c006e780 msgr2=0x7fa2c00bfe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:44.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.063+0000 7fa2c4a00640 1 --2- 172.21.15.28:0/1307262184 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa2c006e780 0x7fa2c00bfe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:44.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.063+0000 7fa2c4a00640 1 -- 172.21.15.28:0/1307262184 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2c0071470 con 0x7fa2c00a5fc0 2024-02-21T23:08:44.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.063+0000 7fa2c5400640 1 --2- 172.21.15.28:0/1307262184 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa2c006e780 0x7fa2c00bfe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:08:44.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.063+0000 7fa2c5e00640 1 --2- 172.21.15.28:0/1307262184 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa2c00c50c0 0x7fa2c01eb240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:08:44.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.063+0000 7fa2c4a00640 1 --2- 172.21.15.28:0/1307262184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2c00a5fc0 0x7fa2c01ead00 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7fa2ac000c00 tx=0x7fa2ac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:44.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.063+0000 7fa2bd600640 1 -- 172.21.15.28:0/1307262184 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2ac032610 con 0x7fa2c00a5fc0 2024-02-21T23:08:44.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.063+0000 7fa2bd600640 1 -- 172.21.15.28:0/1307262184 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2ac0327b0 con 0x7fa2c00a5fc0 2024-02-21T23:08:44.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.064+0000 7fa2bd600640 1 -- 172.21.15.28:0/1307262184 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2ac032b80 con 0x7fa2c00a5fc0 2024-02-21T23:08:44.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.064+0000 7fa2c7e00640 1 -- 172.21.15.28:0/1307262184 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2c01ee3f0 con 0x7fa2c00a5fc0 2024-02-21T23:08:44.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.064+0000 7fa2c7e00640 1 -- 172.21.15.28:0/1307262184 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2c00c03d0 con 0x7fa2c00a5fc0 2024-02-21T23:08:44.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.065+0000 7fa2c7e00640 1 -- 172.21.15.28:0/1307262184 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2c00c58b0 con 0x7fa2c00a5fc0 2024-02-21T23:08:44.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.066+0000 7fa2bd600640 1 -- 172.21.15.28:0/1307262184 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa2ac04c060 con 0x7fa2c00a5fc0 2024-02-21T23:08:44.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.067+0000 7fa2bd600640 1 --2- 172.21.15.28:0/1307262184 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa2a006b4d0 0x7fa2a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:44.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.067+0000 7fa2bd600640 1 -- 172.21.15.28:0/1307262184 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 6359+0+0 (secure 0 0 0) 0x7fa2ac0abff0 con 0x7fa2c00a5fc0 2024-02-21T23:08:44.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.068+0000 7fa2c5400640 1 --2- 172.21.15.28:0/1307262184 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa2a006b4d0 0x7fa2a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:44.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.069+0000 7fa2c5400640 1 --2- 172.21.15.28:0/1307262184 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa2a006b4d0 0x7fa2a006d990 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fa2b4006fd0 tx=0x7fa2b4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:44.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.072+0000 7fa2bd600640 1 -- 172.21.15.28:0/1307262184 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fa2ac040070 con 0x7fa2c00a5fc0 2024-02-21T23:08:44.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.279+0000 7fa2c7e00640 1 -- 172.21.15.28:0/1307262184 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fa2c00c06c0 con 0x7fa2c00a5fc0 2024-02-21T23:08:44.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.802+0000 7fa2bd600640 1 -- 172.21.15.28:0/1307262184 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v126) v1 ==== 181+0+0 (secure 0 0 0) 0x7fa2ac02f050 con 0x7fa2c00a5fc0 2024-02-21T23:08:44.791 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' removed 2024-02-21T23:08:44.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.807+0000 7fa2c7e00640 1 -- 172.21.15.28:0/1307262184 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa2a006b4d0 msgr2=0x7fa2a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:44.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.807+0000 7fa2c7e00640 1 --2- 172.21.15.28:0/1307262184 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa2a006b4d0 0x7fa2a006d990 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fa2b4006fd0 tx=0x7fa2b4008040 comp rx=0 tx=0).stop 2024-02-21T23:08:44.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.807+0000 7fa2c7e00640 1 -- 172.21.15.28:0/1307262184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2c00a5fc0 msgr2=0x7fa2c01ead00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:44.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.807+0000 7fa2c7e00640 1 --2- 172.21.15.28:0/1307262184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2c00a5fc0 0x7fa2c01ead00 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7fa2ac000c00 tx=0x7fa2ac0046c0 comp rx=0 tx=0).stop 2024-02-21T23:08:44.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.807+0000 7fa2c7e00640 1 -- 172.21.15.28:0/1307262184 shutdown_connections 2024-02-21T23:08:44.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.807+0000 7fa2c7e00640 1 --2- 172.21.15.28:0/1307262184 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa2a006b4d0 0x7fa2a006d990 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:44.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.807+0000 7fa2c7e00640 1 --2- 172.21.15.28:0/1307262184 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa2c00c50c0 0x7fa2c01eb240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:44.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.807+0000 7fa2c7e00640 1 --2- 172.21.15.28:0/1307262184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2c00a5fc0 0x7fa2c01ead00 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:44.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.807+0000 7fa2c7e00640 1 --2- 172.21.15.28:0/1307262184 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa2c006e780 0x7fa2c00bfe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:44.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.807+0000 7fa2c7e00640 1 -- 172.21.15.28:0/1307262184 >> 172.21.15.28:0/1307262184 conn(0x7fa2c006b830 msgr2=0x7fa2c00c67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:44.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.811+0000 7fa2c7e00640 1 -- 172.21.15.28:0/1307262184 shutdown_connections 2024-02-21T23:08:44.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:44.811+0000 7fa2c7e00640 1 -- 172.21.15.28:0/1307262184 wait complete. 2024-02-21T23:08:44.822 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-02-21T23:08:45.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.044+0000 7fb5ab200640 1 Processor -- start 2024-02-21T23:08:45.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.044+0000 7fb5ab200640 1 -- start start 2024-02-21T23:08:45.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.046+0000 7fb5ab200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb5ac072d90 0x7fb5ac154fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:45.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.046+0000 7fb5ab200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fb5ac06ae70 0x7fb5ac14e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:45.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.046+0000 7fb5ab200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5ac0c3880 con 0x7fb5ac154bd0 2024-02-21T23:08:45.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.046+0000 7fb5ab200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5ac1558a0 con 0x7fb5ac072d90 2024-02-21T23:08:45.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.046+0000 7fb5ab200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5ac06f6c0 con 0x7fb5ac06ae70 2024-02-21T23:08:45.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.046+0000 7fb5a9400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb5ac072d90 0x7fb5ac154fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:45.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.046+0000 7fb5a9400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb5ac072d90 0x7fb5ac154fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50260/0 (socket says 172.21.15.28:50260) 2024-02-21T23:08:45.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.046+0000 7fb5a9400640 1 -- 172.21.15.28:0/3675999180 learned_addr learned my addr 172.21.15.28:0/3675999180 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:45.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.047+0000 7fb5aa800640 1 --2- 172.21.15.28:0/3675999180 >> v2:172.21.15.81:3300/0 conn(0x7fb5ac06ae70 0x7fb5ac14e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:45.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.047+0000 7fb5a9400640 1 -- 172.21.15.28:0/3675999180 >> v2:172.21.15.81:3300/0 conn(0x7fb5ac06ae70 msgr2=0x7fb5ac14e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:45.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.047+0000 7fb5a9400640 1 --2- 172.21.15.28:0/3675999180 >> v2:172.21.15.81:3300/0 conn(0x7fb5ac06ae70 0x7fb5ac14e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:45.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.047+0000 7fb5a9400640 1 -- 172.21.15.28:0/3675999180 >> v1:172.21.15.81:6789/0 conn(0x7fb5ac154bd0 legacy=0x7fb5ac0c3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:45.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.047+0000 7fb5a9400640 1 -- 172.21.15.28:0/3675999180 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5ac1555c0 con 0x7fb5ac072d90 2024-02-21T23:08:45.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.048+0000 7fb5a9400640 1 --2- 172.21.15.28:0/3675999180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5ac072d90 0x7fb5ac154fb0 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac0731f0 tx=0x7fb59802b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ab9f42341afce5d server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:45.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.048+0000 7fb5a8a00640 1 -- 172.21.15.28:0/3675999180 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb598038070 con 0x7fb5ac072d90 2024-02-21T23:08:45.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.048+0000 7fb5a8a00640 1 -- 172.21.15.28:0/3675999180 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb59802f050 con 0x7fb5ac072d90 2024-02-21T23:08:45.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.048+0000 7fb5a8a00640 1 -- 172.21.15.28:0/3675999180 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb598031970 con 0x7fb5ac072d90 2024-02-21T23:08:45.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.049+0000 7fb5ab200640 1 -- 172.21.15.28:0/3675999180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5ac072d90 msgr2=0x7fb5ac154fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:45.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.049+0000 7fb5ab200640 1 --2- 172.21.15.28:0/3675999180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5ac072d90 0x7fb5ac154fb0 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac0731f0 tx=0x7fb59802b7f0 comp rx=0 tx=0).stop 2024-02-21T23:08:45.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.049+0000 7fb5ab200640 1 -- 172.21.15.28:0/3675999180 shutdown_connections 2024-02-21T23:08:45.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.049+0000 7fb5ab200640 1 --2- 172.21.15.28:0/3675999180 >> v2:172.21.15.81:3300/0 conn(0x7fb5ac06ae70 0x7fb5ac14e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:45.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.049+0000 7fb5ab200640 1 --2- 172.21.15.28:0/3675999180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5ac072d90 0x7fb5ac154fb0 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:45.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.049+0000 7fb5ab200640 1 -- 172.21.15.28:0/3675999180 >> 172.21.15.28:0/3675999180 conn(0x7fb5ac06ea30 msgr2=0x7fb5ac06ee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:45.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.049+0000 7fb5ab200640 1 -- 172.21.15.28:0/3675999180 shutdown_connections 2024-02-21T23:08:45.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.049+0000 7fb5ab200640 1 -- 172.21.15.28:0/3675999180 wait complete. 2024-02-21T23:08:45.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.051+0000 7fb5ab200640 1 Processor -- start 2024-02-21T23:08:45.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.052+0000 7fb5ab200640 1 -- start start 2024-02-21T23:08:45.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.053+0000 7fb5ab200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5ac06ae70 0x7fb5ac0c4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:45.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.053+0000 7fb5ab200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb5ac072d90 0x7fb5ac0c6870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:45.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.053+0000 7fb5ab200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5ac154bd0 0x7fb5ac13bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:45.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.053+0000 7fb5ab200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5ac0a86a0 con 0x7fb5ac154bd0 2024-02-21T23:08:45.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.053+0000 7fb5ab200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5ac0a8520 con 0x7fb5ac06ae70 2024-02-21T23:08:45.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.053+0000 7fb5ab200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5ac0c14e0 con 0x7fb5ac072d90 2024-02-21T23:08:45.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.053+0000 7fb5aa800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5ac154bd0 0x7fb5ac13bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:45.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.053+0000 7fb5aa800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5ac154bd0 0x7fb5ac13bba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50270/0 (socket says 172.21.15.28:50270) 2024-02-21T23:08:45.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.053+0000 7fb5aa800640 1 -- 172.21.15.28:0/984617196 learned_addr learned my addr 172.21.15.28:0/984617196 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:45.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.054+0000 7fb5a9e00640 1 --2- 172.21.15.28:0/984617196 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5ac06ae70 0x7fb5ac0c4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:45.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.054+0000 7fb5a9400640 1 --2- 172.21.15.28:0/984617196 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb5ac072d90 0x7fb5ac0c6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:45.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.054+0000 7fb5aa800640 1 -- 172.21.15.28:0/984617196 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb5ac072d90 msgr2=0x7fb5ac0c6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:45.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.054+0000 7fb5aa800640 1 --2- 172.21.15.28:0/984617196 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb5ac072d90 0x7fb5ac0c6870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:45.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.054+0000 7fb5aa800640 1 -- 172.21.15.28:0/984617196 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5ac06ae70 msgr2=0x7fb5ac0c4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:45.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.054+0000 7fb5aa800640 1 --2- 172.21.15.28:0/984617196 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5ac06ae70 0x7fb5ac0c4ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:45.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.054+0000 7fb5aa800640 1 -- 172.21.15.28:0/984617196 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5ac0acf40 con 0x7fb5ac154bd0 2024-02-21T23:08:45.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.054+0000 7fb5aa800640 1 --2- 172.21.15.28:0/984617196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5ac154bd0 0x7fb5ac13bba0 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7fb5a4007e70 tx=0x7fb5a400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:45.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.054+0000 7fb5a2000640 1 -- 172.21.15.28:0/984617196 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5a4014620 con 0x7fb5ac154bd0 2024-02-21T23:08:45.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.054+0000 7fb5a2000640 1 -- 172.21.15.28:0/984617196 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5a4019070 con 0x7fb5ac154bd0 2024-02-21T23:08:45.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.054+0000 7fb5a2000640 1 -- 172.21.15.28:0/984617196 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5a4014a20 con 0x7fb5ac154bd0 2024-02-21T23:08:45.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.055+0000 7fb5ab200640 1 -- 172.21.15.28:0/984617196 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5ac15f900 con 0x7fb5ac154bd0 2024-02-21T23:08:45.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.055+0000 7fb5ab200640 1 -- 172.21.15.28:0/984617196 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5ac1ee7c0 con 0x7fb5ac154bd0 2024-02-21T23:08:45.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.056+0000 7fb5ab200640 1 -- 172.21.15.28:0/984617196 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5ac04f290 con 0x7fb5ac154bd0 2024-02-21T23:08:45.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.057+0000 7fb5a2000640 1 -- 172.21.15.28:0/984617196 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb5a4014bc0 con 0x7fb5ac154bd0 2024-02-21T23:08:45.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.057+0000 7fb5a2000640 1 --2- 172.21.15.28:0/984617196 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb58c06b4d0 0x7fb58c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:45.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.058+0000 7fb5a2000640 1 -- 172.21.15.28:0/984617196 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5921+0+0 (secure 0 0 0) 0x7fb5a408abf0 con 0x7fb5ac154bd0 2024-02-21T23:08:45.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.058+0000 7fb5a9e00640 1 --2- 172.21.15.28:0/984617196 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb58c06b4d0 0x7fb58c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:45.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.059+0000 7fb5a9e00640 1 --2- 172.21.15.28:0/984617196 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb58c06b4d0 0x7fb58c06d990 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fb59c006fd0 tx=0x7fb59c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:45.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.062+0000 7fb5a2000640 1 -- 172.21.15.28:0/984617196 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb5a4059980 con 0x7fb5ac154bd0 2024-02-21T23:08:45.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.262+0000 7fb5ab200640 1 -- 172.21.15.28:0/984617196 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb5ac14ef00 con 0x7fb5ac154bd0 2024-02-21T23:08:45.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.817+0000 7fb5a2000640 1 -- 172.21.15.28:0/984617196 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v127) v1 ==== 169+0+0 (secure 0 0 0) 0x7fb5a4059280 con 0x7fb5ac154bd0 2024-02-21T23:08:45.806 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' removed 2024-02-21T23:08:45.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.822+0000 7fb5ab200640 1 -- 172.21.15.28:0/984617196 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb58c06b4d0 msgr2=0x7fb58c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:45.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.822+0000 7fb5ab200640 1 --2- 172.21.15.28:0/984617196 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb58c06b4d0 0x7fb58c06d990 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fb59c006fd0 tx=0x7fb59c008040 comp rx=0 tx=0).stop 2024-02-21T23:08:45.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.822+0000 7fb5ab200640 1 -- 172.21.15.28:0/984617196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5ac154bd0 msgr2=0x7fb5ac13bba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:45.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.822+0000 7fb5ab200640 1 --2- 172.21.15.28:0/984617196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5ac154bd0 0x7fb5ac13bba0 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7fb5a4007e70 tx=0x7fb5a400e4b0 comp rx=0 tx=0).stop 2024-02-21T23:08:45.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.823+0000 7fb5ab200640 1 -- 172.21.15.28:0/984617196 shutdown_connections 2024-02-21T23:08:45.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.823+0000 7fb5ab200640 1 --2- 172.21.15.28:0/984617196 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb58c06b4d0 0x7fb58c06d990 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:45.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.823+0000 7fb5ab200640 1 --2- 172.21.15.28:0/984617196 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb5ac072d90 0x7fb5ac0c6870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:45.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.823+0000 7fb5ab200640 1 --2- 172.21.15.28:0/984617196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5ac154bd0 0x7fb5ac13bba0 unknown :-1 s=CLOSED pgs=1468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:45.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.823+0000 7fb5ab200640 1 --2- 172.21.15.28:0/984617196 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5ac06ae70 0x7fb5ac0c4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:45.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.823+0000 7fb5ab200640 1 -- 172.21.15.28:0/984617196 >> 172.21.15.28:0/984617196 conn(0x7fb5ac06ea30 msgr2=0x7fb5ac074520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:45.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.823+0000 7fb5ab200640 1 -- 172.21.15.28:0/984617196 shutdown_connections 2024-02-21T23:08:45.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:45.823+0000 7fb5ab200640 1 -- 172.21.15.28:0/984617196 wait complete. 2024-02-21T23:08:45.837 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.d mds_join_fs 2024-02-21T23:08:46.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.053+0000 7f1eb8c00640 1 Processor -- start 2024-02-21T23:08:46.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.053+0000 7f1eb8c00640 1 -- start start 2024-02-21T23:08:46.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.055+0000 7f1eb8c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1eb41500f0 0x7f1eb406ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:46.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.055+0000 7f1eb8c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1eb406b3b0 0x7f1eb406e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:46.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.055+0000 7f1eb8c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eb406db10 con 0x7f1eb40c2c60 2024-02-21T23:08:46.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.055+0000 7f1eb8c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1eb40c1080 con 0x7f1eb406b3b0 2024-02-21T23:08:46.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.055+0000 7f1eb8c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb41539a0 con 0x7f1eb41500f0 2024-02-21T23:08:46.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.055+0000 7f1eb3e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1eb406b3b0 0x7f1eb406e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:46.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.055+0000 7f1eb3e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1eb406b3b0 0x7f1eb406e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50282/0 (socket says 172.21.15.28:50282) 2024-02-21T23:08:46.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.055+0000 7f1eb3e00640 1 -- 172.21.15.28:0/3825086890 learned_addr learned my addr 172.21.15.28:0/3825086890 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:46.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.056+0000 7f1eb3400640 1 --2- 172.21.15.28:0/3825086890 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1eb41500f0 0x7f1eb406ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:46.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.056+0000 7f1eb3e00640 1 -- 172.21.15.28:0/3825086890 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1eb41500f0 msgr2=0x7f1eb406ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:46.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.056+0000 7f1eb3e00640 1 --2- 172.21.15.28:0/3825086890 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1eb41500f0 0x7f1eb406ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:46.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.056+0000 7f1eb3e00640 1 -- 172.21.15.28:0/3825086890 >> v1:172.21.15.28:6789/0 conn(0x7f1eb40c2c60 legacy=0x7f1eb406dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:46.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.056+0000 7f1eb3e00640 1 -- 172.21.15.28:0/3825086890 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eb40c12d0 con 0x7f1eb406b3b0 2024-02-21T23:08:46.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.056+0000 7f1eb3400640 1 --2- 172.21.15.28:0/3825086890 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1eb41500f0 0x7f1eb406ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:08:46.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.057+0000 7f1eb3e00640 1 --2- 172.21.15.28:0/3825086890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1eb406b3b0 0x7f1eb406e700 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f1eac009e50 tx=0x7f1eac02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfb980b6587f6a1b server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:46.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.057+0000 7f1eb2000640 1 -- 172.21.15.28:0/3825086890 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eac02cb00 con 0x7f1eb406b3b0 2024-02-21T23:08:46.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.057+0000 7f1eb2000640 1 -- 172.21.15.28:0/3825086890 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eac02cca0 con 0x7f1eb406b3b0 2024-02-21T23:08:46.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.057+0000 7f1eb2000640 1 -- 172.21.15.28:0/3825086890 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eac034610 con 0x7f1eb406b3b0 2024-02-21T23:08:46.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.058+0000 7f1eb8c00640 1 -- 172.21.15.28:0/3825086890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1eb406b3b0 msgr2=0x7f1eb406e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:46.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.058+0000 7f1eb8c00640 1 --2- 172.21.15.28:0/3825086890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1eb406b3b0 0x7f1eb406e700 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f1eac009e50 tx=0x7f1eac02bf60 comp rx=0 tx=0).stop 2024-02-21T23:08:46.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.058+0000 7f1eb8c00640 1 -- 172.21.15.28:0/3825086890 shutdown_connections 2024-02-21T23:08:46.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.058+0000 7f1eb8c00640 1 --2- 172.21.15.28:0/3825086890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1eb406b3b0 0x7f1eb406e700 unknown :-1 s=CLOSED pgs=1470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:46.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.058+0000 7f1eb8c00640 1 --2- 172.21.15.28:0/3825086890 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1eb41500f0 0x7f1eb406ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:46.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.058+0000 7f1eb8c00640 1 -- 172.21.15.28:0/3825086890 >> 172.21.15.28:0/3825086890 conn(0x7f1eb40a8530 msgr2=0x7f1eb40a8910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:46.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.058+0000 7f1eb8c00640 1 -- 172.21.15.28:0/3825086890 shutdown_connections 2024-02-21T23:08:46.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.058+0000 7f1eb8c00640 1 -- 172.21.15.28:0/3825086890 wait complete. 2024-02-21T23:08:46.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.060+0000 7f1eb8c00640 1 Processor -- start 2024-02-21T23:08:46.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.060+0000 7f1eb8c00640 1 -- start start 2024-02-21T23:08:46.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.061+0000 7f1eb8c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1eb406b3b0 0x7f1eb4152360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:46.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.061+0000 7f1eb8c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1eb40c2c60 0x7f1eb41499f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:46.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.061+0000 7f1eb8c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1eb41500f0 0x7f1eb4148d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:46.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.061+0000 7f1eb8c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb41535e0 con 0x7f1eb406b3b0 2024-02-21T23:08:46.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.061+0000 7f1eb8c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb41504d0 con 0x7f1eb40c2c60 2024-02-21T23:08:46.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.061+0000 7f1eb8c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb4153760 con 0x7f1eb41500f0 2024-02-21T23:08:46.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.061+0000 7f1eb3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1eb406b3b0 0x7f1eb4152360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:46.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.061+0000 7f1eb3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1eb406b3b0 0x7f1eb4152360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50288/0 (socket says 172.21.15.28:50288) 2024-02-21T23:08:46.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.061+0000 7f1eb3400640 1 -- 172.21.15.28:0/829715530 learned_addr learned my addr 172.21.15.28:0/829715530 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:46.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.062+0000 7f1eb3400640 1 -- 172.21.15.28:0/829715530 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1eb41500f0 msgr2=0x7f1eb4148d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:46.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.062+0000 7f1eb3400640 1 --2- 172.21.15.28:0/829715530 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1eb41500f0 0x7f1eb4148d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:46.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.062+0000 7f1eb3400640 1 -- 172.21.15.28:0/829715530 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1eb40c2c60 msgr2=0x7f1eb41499f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:46.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.062+0000 7f1eb3400640 1 --2- 172.21.15.28:0/829715530 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1eb40c2c60 0x7f1eb41499f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:46.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.062+0000 7f1eb3400640 1 -- 172.21.15.28:0/829715530 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eb41e8880 con 0x7f1eb406b3b0 2024-02-21T23:08:46.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.062+0000 7f1eb3400640 1 --2- 172.21.15.28:0/829715530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1eb406b3b0 0x7f1eb4152360 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f1ea400c820 tx=0x7f1ea400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:46.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.062+0000 7f1eab400640 1 -- 172.21.15.28:0/829715530 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea4015070 con 0x7f1eb406b3b0 2024-02-21T23:08:46.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.063+0000 7f1eab400640 1 -- 172.21.15.28:0/829715530 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ea4007a30 con 0x7f1eb406b3b0 2024-02-21T23:08:46.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.063+0000 7f1eab400640 1 -- 172.21.15.28:0/829715530 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea4007e00 con 0x7f1eb406b3b0 2024-02-21T23:08:46.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.063+0000 7f1eb8c00640 1 -- 172.21.15.28:0/829715530 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1eb41e8b40 con 0x7f1eb406b3b0 2024-02-21T23:08:46.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.063+0000 7f1eb8c00640 1 -- 172.21.15.28:0/829715530 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1eb41e9100 con 0x7f1eb406b3b0 2024-02-21T23:08:46.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.064+0000 7f1eb8c00640 1 -- 172.21.15.28:0/829715530 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1eb404f290 con 0x7f1eb406b3b0 2024-02-21T23:08:46.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.065+0000 7f1eab400640 1 -- 172.21.15.28:0/829715530 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1ea4018070 con 0x7f1eb406b3b0 2024-02-21T23:08:46.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.066+0000 7f1eab400640 1 --2- 172.21.15.28:0/829715530 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1e9406b4d0 0x7f1e9406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:46.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.066+0000 7f1eab400640 1 -- 172.21.15.28:0/829715530 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f1ea408a030 con 0x7f1eb406b3b0 2024-02-21T23:08:46.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.066+0000 7f1eb2a00640 1 --2- 172.21.15.28:0/829715530 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1e9406b4d0 0x7f1e9406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:46.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.068+0000 7f1eb2a00640 1 --2- 172.21.15.28:0/829715530 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1e9406b4d0 0x7f1e9406d990 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f1ea0000f60 tx=0x7f1ea00079d0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:46.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.070+0000 7f1eab400640 1 -- 172.21.15.28:0/829715530 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1ea4058f60 con 0x7f1eb406b3b0 2024-02-21T23:08:46.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.271+0000 7f1eb8c00640 1 -- 172.21.15.28:0/829715530 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.d", "name": "mds_join_fs"} v 0) v1 -- 0x7f1eb40c4ff0 con 0x7f1eb406b3b0 2024-02-21T23:08:46.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.278+0000 7f1eab400640 1 -- 172.21.15.28:0/829715530 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.d", "name": "mds_join_fs"}]=0 v43) v1 ==== 96+0+0 (secure 0 0 0) 0x7f1ea4058860 con 0x7f1eb406b3b0 2024-02-21T23:08:46.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.281+0000 7f1eb8c00640 1 -- 172.21.15.28:0/829715530 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1e9406b4d0 msgr2=0x7f1e9406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:46.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.281+0000 7f1eb8c00640 1 --2- 172.21.15.28:0/829715530 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1e9406b4d0 0x7f1e9406d990 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f1ea0000f60 tx=0x7f1ea00079d0 comp rx=0 tx=0).stop 2024-02-21T23:08:46.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.281+0000 7f1eb8c00640 1 -- 172.21.15.28:0/829715530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1eb406b3b0 msgr2=0x7f1eb4152360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:46.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.281+0000 7f1eb8c00640 1 --2- 172.21.15.28:0/829715530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1eb406b3b0 0x7f1eb4152360 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f1ea400c820 tx=0x7f1ea400cce0 comp rx=0 tx=0).stop 2024-02-21T23:08:46.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.281+0000 7f1eb8c00640 1 -- 172.21.15.28:0/829715530 shutdown_connections 2024-02-21T23:08:46.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.281+0000 7f1eb8c00640 1 --2- 172.21.15.28:0/829715530 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1e9406b4d0 0x7f1e9406d990 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:46.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.281+0000 7f1eb8c00640 1 --2- 172.21.15.28:0/829715530 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1eb41500f0 0x7f1eb4148d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:46.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.281+0000 7f1eb8c00640 1 --2- 172.21.15.28:0/829715530 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1eb40c2c60 0x7f1eb41499f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:46.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.281+0000 7f1eb8c00640 1 --2- 172.21.15.28:0/829715530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1eb406b3b0 0x7f1eb4152360 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:46.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.281+0000 7f1eb8c00640 1 -- 172.21.15.28:0/829715530 >> 172.21.15.28:0/829715530 conn(0x7f1eb40a8530 msgr2=0x7f1eb414a520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:46.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.282+0000 7f1eb8c00640 1 -- 172.21.15.28:0/829715530 shutdown_connections 2024-02-21T23:08:46.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.282+0000 7f1eb8c00640 1 -- 172.21.15.28:0/829715530 wait complete. 2024-02-21T23:08:46.290 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-02-21T23:08:46.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.534+0000 7fab76a00640 1 Processor -- start 2024-02-21T23:08:46.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.534+0000 7fab76a00640 1 -- start start 2024-02-21T23:08:46.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.535+0000 7fab76a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fab7013e0c0 0x7fab700c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:46.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.535+0000 7fab76a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fab7006dbc0 0x7fab7006c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:46.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.535+0000 7fab76a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab70158550 con 0x7fab700c76f0 2024-02-21T23:08:46.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.535+0000 7fab76a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab700c89a0 con 0x7fab7006dbc0 2024-02-21T23:08:46.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.535+0000 7fab76a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab701587d0 con 0x7fab7013e0c0 2024-02-21T23:08:46.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.536+0000 7fab74c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fab7006dbc0 0x7fab7006c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:46.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.536+0000 7fab74c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fab7006dbc0 0x7fab7006c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50292/0 (socket says 172.21.15.28:50292) 2024-02-21T23:08:46.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.536+0000 7fab74c00640 1 -- 172.21.15.28:0/2063552463 learned_addr learned my addr 172.21.15.28:0/2063552463 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:46.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.536+0000 7fab75600640 1 --2- 172.21.15.28:0/2063552463 >> v2:172.21.15.81:3300/0 conn(0x7fab7013e0c0 0x7fab700c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:46.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.536+0000 7fab74c00640 1 -- 172.21.15.28:0/2063552463 >> v2:172.21.15.81:3300/0 conn(0x7fab7013e0c0 msgr2=0x7fab700c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:46.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.536+0000 7fab74c00640 1 --2- 172.21.15.28:0/2063552463 >> v2:172.21.15.81:3300/0 conn(0x7fab7013e0c0 0x7fab700c71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:46.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.536+0000 7fab74c00640 1 -- 172.21.15.28:0/2063552463 >> v1:172.21.15.28:6789/0 conn(0x7fab700c76f0 legacy=0x7fab7006c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:46.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.536+0000 7fab74c00640 1 -- 172.21.15.28:0/2063552463 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab70158ab0 con 0x7fab7006dbc0 2024-02-21T23:08:46.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.536+0000 7fab75600640 1 --2- 172.21.15.28:0/2063552463 >> v2:172.21.15.81:3300/0 conn(0x7fab7013e0c0 0x7fab700c71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:08:46.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.537+0000 7fab74c00640 1 --2- 172.21.15.28:0/2063552463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab7006dbc0 0x7fab7006c280 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7fab700ad180 tx=0x7fab5c02b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=367d97e385080d74 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:46.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.538+0000 7fab6fe00640 1 -- 172.21.15.28:0/2063552463 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab5c035020 con 0x7fab7006dbc0 2024-02-21T23:08:46.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.538+0000 7fab6fe00640 1 -- 172.21.15.28:0/2063552463 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab5c031880 con 0x7fab7006dbc0 2024-02-21T23:08:46.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.538+0000 7fab6fe00640 1 -- 172.21.15.28:0/2063552463 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab5c031c50 con 0x7fab7006dbc0 2024-02-21T23:08:46.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.538+0000 7fab76a00640 1 -- 172.21.15.28:0/2063552463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab7006dbc0 msgr2=0x7fab7006c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:46.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.538+0000 7fab76a00640 1 --2- 172.21.15.28:0/2063552463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab7006dbc0 0x7fab7006c280 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7fab700ad180 tx=0x7fab5c02b920 comp rx=0 tx=0).stop 2024-02-21T23:08:46.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.539+0000 7fab76a00640 1 -- 172.21.15.28:0/2063552463 shutdown_connections 2024-02-21T23:08:46.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.539+0000 7fab76a00640 1 --2- 172.21.15.28:0/2063552463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab7006dbc0 0x7fab7006c280 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:46.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.539+0000 7fab76a00640 1 --2- 172.21.15.28:0/2063552463 >> v2:172.21.15.81:3300/0 conn(0x7fab7013e0c0 0x7fab700c71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:46.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.539+0000 7fab76a00640 1 -- 172.21.15.28:0/2063552463 >> 172.21.15.28:0/2063552463 conn(0x7fab70073110 msgr2=0x7fab700ac490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:46.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.539+0000 7fab76a00640 1 -- 172.21.15.28:0/2063552463 shutdown_connections 2024-02-21T23:08:46.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.539+0000 7fab76a00640 1 -- 172.21.15.28:0/2063552463 wait complete. 2024-02-21T23:08:46.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.540+0000 7fab76a00640 1 Processor -- start 2024-02-21T23:08:46.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.541+0000 7fab76a00640 1 -- start start 2024-02-21T23:08:46.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.542+0000 7fab76a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fab7006dbc0 0x7fab701ee2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:46.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.542+0000 7fab76a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fab700c76f0 0x7fab70154200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:46.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.542+0000 7fab76a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab7013e0c0 0x7fab70154740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:46.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.542+0000 7fab76a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab7006ced0 con 0x7fab7013e0c0 2024-02-21T23:08:46.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.542+0000 7fab76a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab70159070 con 0x7fab7006dbc0 2024-02-21T23:08:46.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.542+0000 7fab76a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab7013e520 con 0x7fab700c76f0 2024-02-21T23:08:46.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.542+0000 7fab76000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab7013e0c0 0x7fab70154740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:46.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.542+0000 7fab76000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab7013e0c0 0x7fab70154740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50298/0 (socket says 172.21.15.28:50298) 2024-02-21T23:08:46.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.542+0000 7fab76000640 1 -- 172.21.15.28:0/2491636669 learned_addr learned my addr 172.21.15.28:0/2491636669 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:46.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.542+0000 7fab75600640 1 --2- 172.21.15.28:0/2491636669 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fab7006dbc0 0x7fab701ee2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:46.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.542+0000 7fab74c00640 1 --2- 172.21.15.28:0/2491636669 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fab700c76f0 0x7fab70154200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:46.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.542+0000 7fab76000640 1 -- 172.21.15.28:0/2491636669 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fab700c76f0 msgr2=0x7fab70154200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:46.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.542+0000 7fab76000640 1 --2- 172.21.15.28:0/2491636669 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fab700c76f0 0x7fab70154200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:46.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.542+0000 7fab76000640 1 -- 172.21.15.28:0/2491636669 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fab7006dbc0 msgr2=0x7fab701ee2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:46.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.542+0000 7fab76000640 1 --2- 172.21.15.28:0/2491636669 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fab7006dbc0 0x7fab701ee2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:46.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.542+0000 7fab76000640 1 -- 172.21.15.28:0/2491636669 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab7013cc70 con 0x7fab7013e0c0 2024-02-21T23:08:46.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.542+0000 7fab75600640 1 --2- 172.21.15.28:0/2491636669 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fab7006dbc0 0x7fab701ee2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:08:46.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.542+0000 7fab74c00640 1 --2- 172.21.15.28:0/2491636669 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fab700c76f0 0x7fab70154200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:08:46.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.543+0000 7fab76000640 1 --2- 172.21.15.28:0/2491636669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab7013e0c0 0x7fab70154740 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7fab6800c7a0 tx=0x7fab6800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:46.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.543+0000 7fab6d600640 1 -- 172.21.15.28:0/2491636669 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab68003b70 con 0x7fab7013e0c0 2024-02-21T23:08:46.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.543+0000 7fab6d600640 1 -- 172.21.15.28:0/2491636669 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab68003d30 con 0x7fab7013e0c0 2024-02-21T23:08:46.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.543+0000 7fab6d600640 1 -- 172.21.15.28:0/2491636669 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab6800aa90 con 0x7fab7013e0c0 2024-02-21T23:08:46.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.543+0000 7fab76a00640 1 -- 172.21.15.28:0/2491636669 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab701eece0 con 0x7fab7013e0c0 2024-02-21T23:08:46.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.543+0000 7fab76a00640 1 -- 172.21.15.28:0/2491636669 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab700af9f0 con 0x7fab7013e0c0 2024-02-21T23:08:46.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.545+0000 7fab76a00640 1 -- 172.21.15.28:0/2491636669 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab7013c740 con 0x7fab7013e0c0 2024-02-21T23:08:46.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.545+0000 7fab6d600640 1 -- 172.21.15.28:0/2491636669 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fab6800ac30 con 0x7fab7013e0c0 2024-02-21T23:08:46.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.546+0000 7fab6d600640 1 --2- 172.21.15.28:0/2491636669 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fab5006b4d0 0x7fab5006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:46.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.546+0000 7fab6d600640 1 -- 172.21.15.28:0/2491636669 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5507+0+0 (secure 0 0 0) 0x7fab6808a0a0 con 0x7fab7013e0c0 2024-02-21T23:08:46.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.547+0000 7fab75600640 1 --2- 172.21.15.28:0/2491636669 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fab5006b4d0 0x7fab5006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:46.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.548+0000 7fab75600640 1 --2- 172.21.15.28:0/2491636669 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fab5006b4d0 0x7fab5006d990 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fab64006fd0 tx=0x7fab64008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:46.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.551+0000 7fab6d600640 1 -- 172.21.15.28:0/2491636669 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fab7013c740 con 0x7fab7013e0c0 2024-02-21T23:08:46.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.753+0000 7fab76a00640 1 -- 172.21.15.28:0/2491636669 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fab7004f290 con 0x7fab7013e0c0 2024-02-21T23:08:46.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.757+0000 7fab6d600640 1 -- 172.21.15.28:0/2491636669 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v44) v1 ==== 104+0+0 (secure 0 0 0) 0x7fab70158550 con 0x7fab7013e0c0 2024-02-21T23:08:46.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.761+0000 7fab76a00640 1 -- 172.21.15.28:0/2491636669 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fab5006b4d0 msgr2=0x7fab5006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:46.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.761+0000 7fab76a00640 1 --2- 172.21.15.28:0/2491636669 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fab5006b4d0 0x7fab5006d990 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fab64006fd0 tx=0x7fab64008040 comp rx=0 tx=0).stop 2024-02-21T23:08:46.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.761+0000 7fab76a00640 1 -- 172.21.15.28:0/2491636669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab7013e0c0 msgr2=0x7fab70154740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:46.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.761+0000 7fab76a00640 1 --2- 172.21.15.28:0/2491636669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab7013e0c0 0x7fab70154740 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7fab6800c7a0 tx=0x7fab6800cc60 comp rx=0 tx=0).stop 2024-02-21T23:08:46.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.761+0000 7fab76a00640 1 -- 172.21.15.28:0/2491636669 shutdown_connections 2024-02-21T23:08:46.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.761+0000 7fab76a00640 1 --2- 172.21.15.28:0/2491636669 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fab5006b4d0 0x7fab5006d990 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:46.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.761+0000 7fab76a00640 1 --2- 172.21.15.28:0/2491636669 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fab700c76f0 0x7fab70154200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:46.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.761+0000 7fab76a00640 1 --2- 172.21.15.28:0/2491636669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab7013e0c0 0x7fab70154740 unknown :-1 s=CLOSED pgs=1474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:46.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.761+0000 7fab76a00640 1 --2- 172.21.15.28:0/2491636669 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fab7006dbc0 0x7fab701ee2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:46.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.761+0000 7fab76a00640 1 -- 172.21.15.28:0/2491636669 >> 172.21.15.28:0/2491636669 conn(0x7fab70073110 msgr2=0x7fab7013ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:46.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.761+0000 7fab76a00640 1 -- 172.21.15.28:0/2491636669 shutdown_connections 2024-02-21T23:08:46.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.761+0000 7fab76a00640 1 -- 172.21.15.28:0/2491636669 wait complete. 2024-02-21T23:08:46.769 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_steady' 2024-02-21T23:08:46.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.998+0000 7f0c9f200640 1 Processor -- start 2024-02-21T23:08:46.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:46.998+0000 7f0c9f200640 1 -- start start 2024-02-21T23:08:46.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:47.000+0000 7f0c9f200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0c98159db0 0x7f0c98072790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:46.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:47.000+0000 7f0c9f200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c980c1390 con 0x7f0c980c0f00 2024-02-21T23:08:46.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:47.000+0000 7f0c9f200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c980c1670 con 0x7f0c981569b0 2024-02-21T23:08:46.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:47.000+0000 7f0c9f200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c980c1fa0 con 0x7f0c98159db0 2024-02-21T23:08:46.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:47.000+0000 7f0c9e800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0c98159db0 0x7f0c98072790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:46.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:47.000+0000 7f0c9e800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0c98159db0 0x7f0c98072790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50300/0 (socket says 172.21.15.28:50300) 2024-02-21T23:08:46.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:47.000+0000 7f0c9e800640 1 -- 172.21.15.28:0/2501808295 learned_addr learned my addr 172.21.15.28:0/2501808295 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:46.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:47.001+0000 7f0c9e800640 1 -- 172.21.15.28:0/2501808295 >> v1:172.21.15.81:6789/0 conn(0x7f0c981569b0 legacy=0x7f0c9806dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:46.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:47.001+0000 7f0c9e800640 1 -- 172.21.15.28:0/2501808295 >> v1:172.21.15.28:6789/0 conn(0x7f0c980c0f00 legacy=0x7f0c98071070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:46.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:47.001+0000 7f0c9e800640 1 -- 172.21.15.28:0/2501808295 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c980c1950 con 0x7f0c98159db0 2024-02-21T23:08:46.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:47.002+0000 7f0c9e800640 1 --2- 172.21.15.28:0/2501808295 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c98159db0 0x7f0c98072790 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7f0c980c2120 tx=0x7f0c9402bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43e36ae27f701935 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:46.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:47.002+0000 7f0c9ca00640 1 -- 172.21.15.28:0/2501808295 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c9402ca60 con 0x7f0c98159db0 2024-02-21T23:08:46.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:47.002+0000 7f0c9ca00640 1 -- 172.21.15.28:0/2501808295 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c9402cc00 con 0x7f0c98159db0 2024-02-21T23:08:46.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:47.002+0000 7f0c9ca00640 1 -- 172.21.15.28:0/2501808295 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c940343f0 con 0x7f0c98159db0 2024-02-21T23:08:46.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:47.003+0000 7f0c9f200640 1 -- 172.21.15.28:0/2501808295 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c98159db0 msgr2=0x7f0c98072790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:46.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:47.003+0000 7f0c9f200640 1 --2- 172.21.15.28:0/2501808295 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c98159db0 0x7f0c98072790 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7f0c980c2120 tx=0x7f0c9402bec0 comp rx=0 tx=0).stop 2024-02-21T23:08:46.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:47.003+0000 7f0c9f200640 1 -- 172.21.15.28:0/2501808295 shutdown_connections 2024-02-21T23:08:46.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:47.003+0000 7f0c9f200640 1 --2- 172.21.15.28:0/2501808295 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c98159db0 0x7f0c98072790 unknown :-1 s=CLOSED pgs=1476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:46.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:47.003+0000 7f0c9f200640 1 -- 172.21.15.28:0/2501808295 >> 172.21.15.28:0/2501808295 conn(0x7f0c9806b630 msgr2=0x7f0c980c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:46.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:47.003+0000 7f0c9f200640 1 -- 172.21.15.28:0/2501808295 shutdown_connections 2024-02-21T23:08:46.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:47.003+0000 7f0c9f200640 1 -- 172.21.15.28:0/2501808295 wait complete. 2024-02-21T23:08:46.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:47.005+0000 7f0c9f200640 1 Processor -- start 2024-02-21T23:08:46.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:47.005+0000 7f0c9f200640 1 -- start start 2024-02-21T23:08:46.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:47.006+0000 7f0c9f200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c981569b0 0x7f0c9815c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:46.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:47.006+0000 7f0c9f200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0c98159db0 0x7f0c9815cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:46.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:47.006+0000 7f0c9f200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0c9815ba40 0x7f0c981e34d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:46.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:47.006+0000 7f0c9f200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c98072cd0 con 0x7f0c981569b0 2024-02-21T23:08:46.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:47.006+0000 7f0c9f200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c9815a200 con 0x7f0c98159db0 2024-02-21T23:08:46.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:47.006+0000 7f0c9f200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c980c2ba0 con 0x7f0c9815ba40 2024-02-21T23:08:46.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:47.006+0000 7f0c9de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c981569b0 0x7f0c9815c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:46.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:47.006+0000 7f0c9de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c981569b0 0x7f0c9815c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50308/0 (socket says 172.21.15.28:50308) 2024-02-21T23:08:46.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:47.006+0000 7f0c9de00640 1 -- 172.21.15.28:0/2748886034 learned_addr learned my addr 172.21.15.28:0/2748886034 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:46.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:47.007+0000 7f0c9e800640 1 --2- 172.21.15.28:0/2748886034 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0c9815ba40 0x7f0c981e34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:46.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:47.007+0000 7f0c9de00640 1 -- 172.21.15.28:0/2748886034 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0c9815ba40 msgr2=0x7f0c981e34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:46.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:47.007+0000 7f0c9de00640 1 --2- 172.21.15.28:0/2748886034 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0c9815ba40 0x7f0c981e34d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:46.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:47.007+0000 7f0c9de00640 1 -- 172.21.15.28:0/2748886034 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0c98159db0 msgr2=0x7f0c9815cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:46.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:47.007+0000 7f0c9de00640 1 --2- 172.21.15.28:0/2748886034 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0c98159db0 0x7f0c9815cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:46.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:47.007+0000 7f0c9de00640 1 -- 172.21.15.28:0/2748886034 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c9815a730 con 0x7f0c981569b0 2024-02-21T23:08:46.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:47.007+0000 7f0c9de00640 1 --2- 172.21.15.28:0/2748886034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c981569b0 0x7f0c9815c600 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c00c430 tx=0x7f0c8c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:46.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:47.007+0000 7f0c92000640 1 -- 172.21.15.28:0/2748886034 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c8c011790 con 0x7f0c981569b0 2024-02-21T23:08:46.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:47.007+0000 7f0c92000640 1 -- 172.21.15.28:0/2748886034 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c8c017070 con 0x7f0c981569b0 2024-02-21T23:08:46.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:47.007+0000 7f0c9f200640 1 -- 172.21.15.28:0/2748886034 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c981efc50 con 0x7f0c981569b0 2024-02-21T23:08:46.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:47.007+0000 7f0c92000640 1 -- 172.21.15.28:0/2748886034 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c8c011b70 con 0x7f0c981569b0 2024-02-21T23:08:46.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:47.007+0000 7f0c9f200640 1 -- 172.21.15.28:0/2748886034 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c981e55e0 con 0x7f0c981569b0 2024-02-21T23:08:46.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:47.009+0000 7f0c9f200640 1 -- 172.21.15.28:0/2748886034 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c9804f290 con 0x7f0c981569b0 2024-02-21T23:08:46.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:47.010+0000 7f0c92000640 1 -- 172.21.15.28:0/2748886034 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0c8c011d10 con 0x7f0c981569b0 2024-02-21T23:08:46.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:47.010+0000 7f0c92000640 1 --2- 172.21.15.28:0/2748886034 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0c7c06b4d0 0x7f0c7c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:46.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:47.011+0000 7f0c92000640 1 -- 172.21.15.28:0/2748886034 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f0c8c089ff0 con 0x7f0c981569b0 2024-02-21T23:08:46.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:47.011+0000 7f0c9d400640 1 --2- 172.21.15.28:0/2748886034 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0c7c06b4d0 0x7f0c7c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:47.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:47.012+0000 7f0c9d400640 1 --2- 172.21.15.28:0/2748886034 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0c7c06b4d0 0x7f0c7c06d990 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f0c88000f60 tx=0x7f0c880079d0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:47.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:47.015+0000 7f0c92000640 1 -- 172.21.15.28:0/2748886034 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f0c8c058f40 con 0x7f0c981569b0 2024-02-21T23:08:47.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:47.292+0000 7f0c9f200640 1 -- 172.21.15.28:0/2748886034 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_steady"]} v 0) v1 -- 0x7f0c98073540 con 0x7f0c981569b0 2024-02-21T23:08:47.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:47.822+0000 7f0c92000640 1 -- 172.21.15.28:0/2748886034 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_steady"]}]=0 v364) v1 ==== 145+0+0 (secure 0 0 0) 0x7f0c8c058840 con 0x7f0c981569b0 2024-02-21T23:08:47.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:47.826+0000 7f0c9f200640 1 -- 172.21.15.28:0/2748886034 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0c7c06b4d0 msgr2=0x7f0c7c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:47.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:47.826+0000 7f0c9f200640 1 --2- 172.21.15.28:0/2748886034 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0c7c06b4d0 0x7f0c7c06d990 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f0c88000f60 tx=0x7f0c880079d0 comp rx=0 tx=0).stop 2024-02-21T23:08:47.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:47.826+0000 7f0c9f200640 1 -- 172.21.15.28:0/2748886034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c981569b0 msgr2=0x7f0c9815c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:47.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:47.826+0000 7f0c9f200640 1 --2- 172.21.15.28:0/2748886034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c981569b0 0x7f0c9815c600 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c00c430 tx=0x7f0c8c00c8f0 comp rx=0 tx=0).stop 2024-02-21T23:08:47.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:47.826+0000 7f0c9f200640 1 -- 172.21.15.28:0/2748886034 shutdown_connections 2024-02-21T23:08:47.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:47.826+0000 7f0c9f200640 1 --2- 172.21.15.28:0/2748886034 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0c7c06b4d0 0x7f0c7c06d990 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:47.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:47.826+0000 7f0c9f200640 1 --2- 172.21.15.28:0/2748886034 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0c9815ba40 0x7f0c981e34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:47.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:47.826+0000 7f0c9f200640 1 --2- 172.21.15.28:0/2748886034 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0c98159db0 0x7f0c9815cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:47.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:47.826+0000 7f0c9f200640 1 --2- 172.21.15.28:0/2748886034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c981569b0 0x7f0c9815c600 unknown :-1 s=CLOSED pgs=1477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:47.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:47.826+0000 7f0c9f200640 1 -- 172.21.15.28:0/2748886034 >> 172.21.15.28:0/2748886034 conn(0x7f0c9806b630 msgr2=0x7f0c98158670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:47.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:47.826+0000 7f0c9f200640 1 -- 172.21.15.28:0/2748886034 shutdown_connections 2024-02-21T23:08:47.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:47.827+0000 7f0c9f200640 1 -- 172.21.15.28:0/2748886034 wait complete. 2024-02-21T23:08:47.837 INFO:tasks.cephfs_test_runner:That a sole MDS with mds_join_fs set will come back as active eventually even after failover. ... ok 2024-02-21T23:08:47.837 INFO:tasks.cephfs_test_runner:Starting test: test_join_fs_unset (tasks.cephfs.test_failover.TestClusterAffinity) That unsetting mds_join_fs will cause failover if another high-affinity standby exists. 2024-02-21T23:08:47.837 INFO:tasks.cephfs_test_runner:test_join_fs_unset (tasks.cephfs.test_failover.TestClusterAffinity) 2024-02-21T23:08:47.838 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:08:48.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.073+0000 7ff1bb200640 1 Processor -- start 2024-02-21T23:08:48.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.073+0000 7ff1bb200640 1 -- start start 2024-02-21T23:08:48.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.075+0000 7ff1bb200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff1bc0aff90 0x7ff1bc0c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:48.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.075+0000 7ff1bb200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff1bc0c5720 0x7ff1bc15f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:48.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.075+0000 7ff1bb200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1bc0c74c0 con 0x7ff1bc06dbf0 2024-02-21T23:08:48.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.075+0000 7ff1bb200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1bc0c7270 con 0x7ff1bc0c5720 2024-02-21T23:08:48.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.075+0000 7ff1bb200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1bc158630 con 0x7ff1bc0aff90 2024-02-21T23:08:48.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.075+0000 7ff1ba800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff1bc0c5720 0x7ff1bc15f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:48.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.075+0000 7ff1ba800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff1bc0c5720 0x7ff1bc15f890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50312/0 (socket says 172.21.15.28:50312) 2024-02-21T23:08:48.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.075+0000 7ff1ba800640 1 -- 172.21.15.28:0/3248789847 learned_addr learned my addr 172.21.15.28:0/3248789847 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:48.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.076+0000 7ff1b9e00640 1 --2- 172.21.15.28:0/3248789847 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff1bc0aff90 0x7ff1bc0c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:48.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.076+0000 7ff1ba800640 1 -- 172.21.15.28:0/3248789847 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff1bc0aff90 msgr2=0x7ff1bc0c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:48.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.076+0000 7ff1ba800640 1 --2- 172.21.15.28:0/3248789847 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff1bc0aff90 0x7ff1bc0c51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:48.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.076+0000 7ff1ba800640 1 -- 172.21.15.28:0/3248789847 >> v1:172.21.15.28:6789/0 conn(0x7ff1bc06dbf0 legacy=0x7ff1bc158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:48.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.076+0000 7ff1ba800640 1 -- 172.21.15.28:0/3248789847 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1bc0c77a0 con 0x7ff1bc0c5720 2024-02-21T23:08:48.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.077+0000 7ff1ba800640 1 --2- 172.21.15.28:0/3248789847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1bc0c5720 0x7ff1bc15f890 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7ff1b4009e50 tx=0x7ff1b402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ef29d0f7881ac8f server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:48.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.077+0000 7ff1b9400640 1 -- 172.21.15.28:0/3248789847 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1b402cb00 con 0x7ff1bc0c5720 2024-02-21T23:08:48.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.077+0000 7ff1b9400640 1 -- 172.21.15.28:0/3248789847 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1b402cca0 con 0x7ff1bc0c5720 2024-02-21T23:08:48.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.077+0000 7ff1b9400640 1 -- 172.21.15.28:0/3248789847 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1b4034610 con 0x7ff1bc0c5720 2024-02-21T23:08:48.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.078+0000 7ff1bb200640 1 -- 172.21.15.28:0/3248789847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1bc0c5720 msgr2=0x7ff1bc15f890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:48.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.078+0000 7ff1bb200640 1 --2- 172.21.15.28:0/3248789847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1bc0c5720 0x7ff1bc15f890 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7ff1b4009e50 tx=0x7ff1b402bf60 comp rx=0 tx=0).stop 2024-02-21T23:08:48.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.078+0000 7ff1bb200640 1 -- 172.21.15.28:0/3248789847 shutdown_connections 2024-02-21T23:08:48.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.078+0000 7ff1bb200640 1 --2- 172.21.15.28:0/3248789847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1bc0c5720 0x7ff1bc15f890 unknown :-1 s=CLOSED pgs=1479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:48.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.078+0000 7ff1bb200640 1 --2- 172.21.15.28:0/3248789847 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff1bc0aff90 0x7ff1bc0c51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:48.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.078+0000 7ff1bb200640 1 -- 172.21.15.28:0/3248789847 >> 172.21.15.28:0/3248789847 conn(0x7ff1bc0ac370 msgr2=0x7ff1bc0ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:48.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.078+0000 7ff1bb200640 1 -- 172.21.15.28:0/3248789847 shutdown_connections 2024-02-21T23:08:48.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.078+0000 7ff1bb200640 1 -- 172.21.15.28:0/3248789847 wait complete. 2024-02-21T23:08:48.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.080+0000 7ff1bb200640 1 Processor -- start 2024-02-21T23:08:48.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.081+0000 7ff1bb200640 1 -- start start 2024-02-21T23:08:48.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.082+0000 7ff1bb200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff1bc06dbf0 0x7ff1bc1eaeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:48.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.082+0000 7ff1bb200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff1bc0aff90 0x7ff1bc0ae740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:48.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.082+0000 7ff1bb200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1bc0c5720 0x7ff1bc1533d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:48.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.082+0000 7ff1bb200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1bc0c8760 con 0x7ff1bc0c5720 2024-02-21T23:08:48.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.082+0000 7ff1bb200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1bc0be0c0 con 0x7ff1bc0aff90 2024-02-21T23:08:48.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.082+0000 7ff1bb200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1bc0c88e0 con 0x7ff1bc06dbf0 2024-02-21T23:08:48.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.082+0000 7ff1ba800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1bc0c5720 0x7ff1bc1533d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:48.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.082+0000 7ff1ba800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1bc0c5720 0x7ff1bc1533d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50324/0 (socket says 172.21.15.28:50324) 2024-02-21T23:08:48.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.082+0000 7ff1ba800640 1 -- 172.21.15.28:0/1090735279 learned_addr learned my addr 172.21.15.28:0/1090735279 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:48.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.082+0000 7ff1b9e00640 1 --2- 172.21.15.28:0/1090735279 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff1bc06dbf0 0x7ff1bc1eaeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:48.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.083+0000 7ff1b3e00640 1 --2- 172.21.15.28:0/1090735279 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff1bc0aff90 0x7ff1bc0ae740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:48.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.083+0000 7ff1ba800640 1 -- 172.21.15.28:0/1090735279 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff1bc06dbf0 msgr2=0x7ff1bc1eaeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:48.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.083+0000 7ff1ba800640 1 --2- 172.21.15.28:0/1090735279 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff1bc06dbf0 0x7ff1bc1eaeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:48.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.083+0000 7ff1ba800640 1 -- 172.21.15.28:0/1090735279 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff1bc0aff90 msgr2=0x7ff1bc0ae740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:48.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.083+0000 7ff1ba800640 1 --2- 172.21.15.28:0/1090735279 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff1bc0aff90 0x7ff1bc0ae740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:48.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.083+0000 7ff1ba800640 1 -- 172.21.15.28:0/1090735279 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1bc154190 con 0x7ff1bc0c5720 2024-02-21T23:08:48.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.083+0000 7ff1ba800640 1 --2- 172.21.15.28:0/1090735279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1bc0c5720 0x7ff1bc1533d0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7ff1b4009e50 tx=0x7ff1b4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:48.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.083+0000 7ff1b2000640 1 -- 172.21.15.28:0/1090735279 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1b4044020 con 0x7ff1bc0c5720 2024-02-21T23:08:48.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.083+0000 7ff1b2000640 1 -- 172.21.15.28:0/1090735279 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1b402ce50 con 0x7ff1bc0c5720 2024-02-21T23:08:48.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.084+0000 7ff1bb200640 1 -- 172.21.15.28:0/1090735279 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1bc154470 con 0x7ff1bc0c5720 2024-02-21T23:08:48.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.084+0000 7ff1b2000640 1 -- 172.21.15.28:0/1090735279 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1b4009510 con 0x7ff1bc0c5720 2024-02-21T23:08:48.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.084+0000 7ff1bb200640 1 -- 172.21.15.28:0/1090735279 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1bc0aec80 con 0x7ff1bc0c5720 2024-02-21T23:08:48.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.085+0000 7ff1bb200640 1 -- 172.21.15.28:0/1090735279 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1bc04f290 con 0x7ff1bc0c5720 2024-02-21T23:08:48.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.086+0000 7ff1b2000640 1 -- 172.21.15.28:0/1090735279 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff1b4047020 con 0x7ff1bc0c5720 2024-02-21T23:08:48.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.087+0000 7ff1b2000640 1 --2- 172.21.15.28:0/1090735279 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff19806b4d0 0x7ff19806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:48.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.087+0000 7ff1b2000640 1 -- 172.21.15.28:0/1090735279 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5507+0+0 (secure 0 0 0) 0x7ff1b40acbd0 con 0x7ff1bc0c5720 2024-02-21T23:08:48.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.087+0000 7ff1b9e00640 1 --2- 172.21.15.28:0/1090735279 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff19806b4d0 0x7ff19806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:48.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.088+0000 7ff1b9e00640 1 --2- 172.21.15.28:0/1090735279 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff19806b4d0 0x7ff19806d990 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7ff1ac006fd0 tx=0x7ff1ac008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:48.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.091+0000 7ff1b2000640 1 -- 172.21.15.28:0/1090735279 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ff1b404e030 con 0x7ff1bc0c5720 2024-02-21T23:08:48.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.293+0000 7ff1bb200640 1 -- 172.21.15.28:0/1090735279 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff1bc1524f0 con 0x7ff1bc0c5720 2024-02-21T23:08:48.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.295+0000 7ff1b2000640 1 -- 172.21.15.28:0/1090735279 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v127) v1 ==== 74+0+13503 (secure 0 0 0) 0x7ff1b407bc20 con 0x7ff1bc0c5720 2024-02-21T23:08:48.283 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:08:48.284 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":127,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:08:45.803507+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.81:6835/3687210533":"2024-02-22T23:08:42.786271+0000","172.21.15.81:6837/3687210533":"2024-02-22T23:08:42.786271+0000","172.21.15.28:6834/1773560593":"2024-02-22T23:08:32.504770+0000","172.21.15.28:6832/1773560593":"2024-02-22T23:08:32.504770+0000","172.21.15.81:6837/1038096496":"2024-02-22T23:08:30.184500+0000","172.21.15.81:6835/1038096496":"2024-02-22T23:08:30.184500+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:08:48.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.298+0000 7ff1bb200640 1 -- 172.21.15.28:0/1090735279 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff19806b4d0 msgr2=0x7ff19806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:48.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.298+0000 7ff1bb200640 1 --2- 172.21.15.28:0/1090735279 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff19806b4d0 0x7ff19806d990 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7ff1ac006fd0 tx=0x7ff1ac008040 comp rx=0 tx=0).stop 2024-02-21T23:08:48.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.298+0000 7ff1bb200640 1 -- 172.21.15.28:0/1090735279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1bc0c5720 msgr2=0x7ff1bc1533d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:48.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.298+0000 7ff1bb200640 1 --2- 172.21.15.28:0/1090735279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1bc0c5720 0x7ff1bc1533d0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7ff1b4009e50 tx=0x7ff1b4004a40 comp rx=0 tx=0).stop 2024-02-21T23:08:48.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.298+0000 7ff1bb200640 1 -- 172.21.15.28:0/1090735279 shutdown_connections 2024-02-21T23:08:48.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.298+0000 7ff1bb200640 1 --2- 172.21.15.28:0/1090735279 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff19806b4d0 0x7ff19806d990 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:48.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.298+0000 7ff1bb200640 1 --2- 172.21.15.28:0/1090735279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1bc0c5720 0x7ff1bc1533d0 unknown :-1 s=CLOSED pgs=1480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:48.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.298+0000 7ff1bb200640 1 --2- 172.21.15.28:0/1090735279 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff1bc0aff90 0x7ff1bc0ae740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:48.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.298+0000 7ff1bb200640 1 --2- 172.21.15.28:0/1090735279 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff1bc06dbf0 0x7ff1bc1eaeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:48.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.298+0000 7ff1bb200640 1 -- 172.21.15.28:0/1090735279 >> 172.21.15.28:0/1090735279 conn(0x7ff1bc0ac370 msgr2=0x7ff1bc0ac750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:48.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.298+0000 7ff1bb200640 1 -- 172.21.15.28:0/1090735279 shutdown_connections 2024-02-21T23:08:48.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.298+0000 7ff1bb200640 1 -- 172.21.15.28:0/1090735279 wait complete. 2024-02-21T23:08:48.308 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-21T23:02:30.728329+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-21T23:08:48.308 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-21T23:08:48.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.547+0000 7f00a7200640 1 Processor -- start 2024-02-21T23:08:48.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.548+0000 7f00a7200640 1 -- start start 2024-02-21T23:08:48.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.549+0000 7f00a7200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f00a013e0c0 0x7f00a00c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:48.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.549+0000 7f00a7200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f00a006dbc0 0x7f00a006c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:48.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.549+0000 7f00a7200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00a0158550 con 0x7f00a00c76f0 2024-02-21T23:08:48.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.549+0000 7f00a7200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00a00c89a0 con 0x7f00a013e0c0 2024-02-21T23:08:48.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.549+0000 7f00a7200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00a01587d0 con 0x7f00a006dbc0 2024-02-21T23:08:48.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.550+0000 7f00a5e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f00a013e0c0 0x7f00a00c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:48.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.550+0000 7f00a5e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f00a013e0c0 0x7f00a00c71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50332/0 (socket says 172.21.15.28:50332) 2024-02-21T23:08:48.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.550+0000 7f00a5e00640 1 -- 172.21.15.28:0/1387276880 learned_addr learned my addr 172.21.15.28:0/1387276880 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:48.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.550+0000 7f00a5e00640 1 -- 172.21.15.28:0/1387276880 >> v2:172.21.15.81:3300/0 conn(0x7f00a006dbc0 msgr2=0x7f00a006c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:48.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.550+0000 7f00a5e00640 1 --2- 172.21.15.28:0/1387276880 >> v2:172.21.15.81:3300/0 conn(0x7f00a006dbc0 0x7f00a006c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:48.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.550+0000 7f00a5e00640 1 -- 172.21.15.28:0/1387276880 >> v1:172.21.15.81:6789/0 conn(0x7f00a00c76f0 legacy=0x7f00a006c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:48.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.550+0000 7f00a5e00640 1 -- 172.21.15.28:0/1387276880 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00a0158ab0 con 0x7f00a013e0c0 2024-02-21T23:08:48.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.551+0000 7f00a5e00640 1 --2- 172.21.15.28:0/1387276880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00a013e0c0 0x7f00a00c71b0 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f00a00c7100 tx=0x7f009402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8dd98edd552c99d server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:48.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.552+0000 7f00a4a00640 1 -- 172.21.15.28:0/1387276880 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f009402e070 con 0x7f00a013e0c0 2024-02-21T23:08:48.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.552+0000 7f00a4a00640 1 -- 172.21.15.28:0/1387276880 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0094037070 con 0x7f00a013e0c0 2024-02-21T23:08:48.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.552+0000 7f00a4a00640 1 -- 172.21.15.28:0/1387276880 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0094031a00 con 0x7f00a013e0c0 2024-02-21T23:08:48.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.552+0000 7f00a7200640 1 -- 172.21.15.28:0/1387276880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00a013e0c0 msgr2=0x7f00a00c71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:48.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.552+0000 7f00a7200640 1 --2- 172.21.15.28:0/1387276880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00a013e0c0 0x7f00a00c71b0 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f00a00c7100 tx=0x7f009402b880 comp rx=0 tx=0).stop 2024-02-21T23:08:48.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.553+0000 7f00a7200640 1 -- 172.21.15.28:0/1387276880 shutdown_connections 2024-02-21T23:08:48.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.553+0000 7f00a7200640 1 --2- 172.21.15.28:0/1387276880 >> v2:172.21.15.81:3300/0 conn(0x7f00a006dbc0 0x7f00a006c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:48.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.553+0000 7f00a7200640 1 --2- 172.21.15.28:0/1387276880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00a013e0c0 0x7f00a00c71b0 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:48.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.553+0000 7f00a7200640 1 -- 172.21.15.28:0/1387276880 >> 172.21.15.28:0/1387276880 conn(0x7f00a0073110 msgr2=0x7f00a00ac490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:48.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.553+0000 7f00a7200640 1 -- 172.21.15.28:0/1387276880 shutdown_connections 2024-02-21T23:08:48.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.553+0000 7f00a7200640 1 -- 172.21.15.28:0/1387276880 wait complete. 2024-02-21T23:08:48.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.555+0000 7f00a7200640 1 Processor -- start 2024-02-21T23:08:48.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.555+0000 7f00a7200640 1 -- start start 2024-02-21T23:08:48.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.557+0000 7f00a7200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f00a006dbc0 0x7f00a01ee680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:48.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.557+0000 7f00a7200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00a00c76f0 0x7f00a0137800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:48.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.557+0000 7f00a7200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f00a013e0c0 0x7f00a0151980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:48.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.557+0000 7f00a7200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00a006ced0 con 0x7f00a00c76f0 2024-02-21T23:08:48.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.557+0000 7f00a7200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00a0159070 con 0x7f00a013e0c0 2024-02-21T23:08:48.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.557+0000 7f00a7200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00a013e520 con 0x7f00a006dbc0 2024-02-21T23:08:48.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.557+0000 7f00a5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00a00c76f0 0x7f00a0137800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:48.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.557+0000 7f00a5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00a00c76f0 0x7f00a0137800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50334/0 (socket says 172.21.15.28:50334) 2024-02-21T23:08:48.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.557+0000 7f00a5400640 1 -- 172.21.15.28:0/2808148275 learned_addr learned my addr 172.21.15.28:0/2808148275 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:48.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.557+0000 7f00a5e00640 1 --2- 172.21.15.28:0/2808148275 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f00a006dbc0 0x7f00a01ee680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:48.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.557+0000 7f00a5400640 1 -- 172.21.15.28:0/2808148275 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f00a006dbc0 msgr2=0x7f00a01ee680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:48.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.557+0000 7f00a6800640 1 --2- 172.21.15.28:0/2808148275 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f00a013e0c0 0x7f00a0151980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:48.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.558+0000 7f00a5400640 1 --2- 172.21.15.28:0/2808148275 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f00a006dbc0 0x7f00a01ee680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:48.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.558+0000 7f00a5400640 1 -- 172.21.15.28:0/2808148275 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f00a013e0c0 msgr2=0x7f00a0151980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:48.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.558+0000 7f00a5400640 1 --2- 172.21.15.28:0/2808148275 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f00a013e0c0 0x7f00a0151980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:48.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.558+0000 7f00a5400640 1 -- 172.21.15.28:0/2808148275 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00a01de690 con 0x7f00a00c76f0 2024-02-21T23:08:48.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.558+0000 7f00a5e00640 1 --2- 172.21.15.28:0/2808148275 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f00a006dbc0 0x7f00a01ee680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:08:48.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.558+0000 7f00a5400640 1 --2- 172.21.15.28:0/2808148275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00a00c76f0 0x7f00a0137800 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7f009000b4d0 tx=0x7f009000b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:48.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.558+0000 7f009a000640 1 -- 172.21.15.28:0/2808148275 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0090014070 con 0x7f00a00c76f0 2024-02-21T23:08:48.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.558+0000 7f009a000640 1 -- 172.21.15.28:0/2808148275 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0090010870 con 0x7f00a00c76f0 2024-02-21T23:08:48.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.558+0000 7f00a7200640 1 -- 172.21.15.28:0/2808148275 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00a01de970 con 0x7f00a00c76f0 2024-02-21T23:08:48.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.558+0000 7f009a000640 1 -- 172.21.15.28:0/2808148275 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0090010c60 con 0x7f00a00c76f0 2024-02-21T23:08:48.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.558+0000 7f00a7200640 1 -- 172.21.15.28:0/2808148275 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00a01def30 con 0x7f00a00c76f0 2024-02-21T23:08:48.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.560+0000 7f00a7200640 1 -- 172.21.15.28:0/2808148275 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00a004f290 con 0x7f00a00c76f0 2024-02-21T23:08:48.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.561+0000 7f009a000640 1 -- 172.21.15.28:0/2808148275 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0090017070 con 0x7f00a00c76f0 2024-02-21T23:08:48.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.562+0000 7f009a000640 1 --2- 172.21.15.28:0/2808148275 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f008406b520 0x7f008406d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:48.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.562+0000 7f009a000640 1 -- 172.21.15.28:0/2808148275 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f0090089f70 con 0x7f00a00c76f0 2024-02-21T23:08:48.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.562+0000 7f00a5e00640 1 --2- 172.21.15.28:0/2808148275 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f008406b520 0x7f008406d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:48.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.563+0000 7f00a5e00640 1 --2- 172.21.15.28:0/2808148275 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f008406b520 0x7f008406d9e0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f0094036000 tx=0x7f0094008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:48.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.566+0000 7f009a000640 1 -- 172.21.15.28:0/2808148275 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f0090058e90 con 0x7f00a00c76f0 2024-02-21T23:08:48.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.766+0000 7f00a7200640 1 -- 172.21.15.28:0/2808148275 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f00a006d9a0 con 0x7f00a00c76f0 2024-02-21T23:08:48.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.768+0000 7f009a000640 1 -- 172.21.15.28:0/2808148275 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v127) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0090058790 con 0x7f00a00c76f0 2024-02-21T23:08:48.756 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-02-21T23:08:48.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.770+0000 7f00a7200640 1 -- 172.21.15.28:0/2808148275 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f008406b520 msgr2=0x7f008406d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:48.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.770+0000 7f00a7200640 1 --2- 172.21.15.28:0/2808148275 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f008406b520 0x7f008406d9e0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f0094036000 tx=0x7f0094008030 comp rx=0 tx=0).stop 2024-02-21T23:08:48.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.770+0000 7f00a7200640 1 -- 172.21.15.28:0/2808148275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00a00c76f0 msgr2=0x7f00a0137800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:48.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.770+0000 7f00a7200640 1 --2- 172.21.15.28:0/2808148275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00a00c76f0 0x7f00a0137800 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7f009000b4d0 tx=0x7f009000b990 comp rx=0 tx=0).stop 2024-02-21T23:08:48.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.771+0000 7f00a7200640 1 -- 172.21.15.28:0/2808148275 shutdown_connections 2024-02-21T23:08:48.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.771+0000 7f00a7200640 1 --2- 172.21.15.28:0/2808148275 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f008406b520 0x7f008406d9e0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:48.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.771+0000 7f00a7200640 1 --2- 172.21.15.28:0/2808148275 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f00a013e0c0 0x7f00a0151980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:48.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.771+0000 7f00a7200640 1 --2- 172.21.15.28:0/2808148275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00a00c76f0 0x7f00a0137800 unknown :-1 s=CLOSED pgs=1482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:48.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.771+0000 7f00a7200640 1 --2- 172.21.15.28:0/2808148275 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f00a006dbc0 0x7f00a01ee680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:48.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.771+0000 7f00a7200640 1 -- 172.21.15.28:0/2808148275 >> 172.21.15.28:0/2808148275 conn(0x7f00a0073110 msgr2=0x7f00a013d1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:48.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.771+0000 7f00a7200640 1 -- 172.21.15.28:0/2808148275 shutdown_connections 2024-02-21T23:08:48.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:48.771+0000 7f00a7200640 1 -- 172.21.15.28:0/2808148275 wait complete. 2024-02-21T23:08:48.779 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_unset' 2024-02-21T23:08:48.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:49.008+0000 7f05c2400640 1 Processor -- start 2024-02-21T23:08:48.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:49.008+0000 7f05c2400640 1 -- start start 2024-02-21T23:08:48.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:49.009+0000 7f05c2400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f05bc153950 0x7f05bc0c15d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:48.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:49.009+0000 7f05c2400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05bc152e70 con 0x7f05bc0c11f0 2024-02-21T23:08:48.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:49.009+0000 7f05c2400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05bc153150 con 0x7f05bc15b6d0 2024-02-21T23:08:48.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:49.009+0000 7f05c2400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05bc0a5ec0 con 0x7f05bc153950 2024-02-21T23:08:48.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:49.010+0000 7f05b3e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f05bc153950 0x7f05bc0c15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:48.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:49.010+0000 7f05bb400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f05bc0c11f0 0x7f05bc06dce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37876/0 (socket says 172.21.15.28:37876) 2024-02-21T23:08:48.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:49.010+0000 7f05bb400640 1 -- 172.21.15.28:0/3555461362 learned_addr learned my addr 172.21.15.28:0/3555461362 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:08:48.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:49.010+0000 7f05b3e00640 1 -- 172.21.15.28:0/3555461362 >> v1:172.21.15.81:6789/0 conn(0x7f05bc15b6d0 legacy=0x7f05bc15dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:48.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:49.010+0000 7f05b3e00640 1 -- 172.21.15.28:0/3555461362 >> v1:172.21.15.28:6789/0 conn(0x7f05bc0c11f0 legacy=0x7f05bc06dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:48.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:49.010+0000 7f05b3e00640 1 -- 172.21.15.28:0/3555461362 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05bc153430 con 0x7f05bc153950 2024-02-21T23:08:48.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:49.010+0000 7f05baa00640 1 -- 172.21.15.28:0/3555461362 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 172123363 0 0) 0x7f05bc152e70 con 0x7f05bc0c11f0 2024-02-21T23:08:49.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:49.011+0000 7f05b3e00640 1 --2- 172.21.15.28:0/3555461362 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05bc153950 0x7f05bc0c15d0 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f05a40097a0 tx=0x7f05a402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d89c3b7b8f9cd2e0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:49.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:49.012+0000 7f05baa00640 1 -- 172.21.15.28:0/3555461362 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a40315a0 con 0x7f05bc153950 2024-02-21T23:08:49.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:49.012+0000 7f05baa00640 1 -- 172.21.15.28:0/3555461362 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05a4037070 con 0x7f05bc153950 2024-02-21T23:08:49.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:49.012+0000 7f05baa00640 1 -- 172.21.15.28:0/3555461362 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a4031960 con 0x7f05bc153950 2024-02-21T23:08:49.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:49.012+0000 7f05c2400640 1 -- 172.21.15.28:0/3555461362 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05bc153950 msgr2=0x7f05bc0c15d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:49.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:49.012+0000 7f05c2400640 1 --2- 172.21.15.28:0/3555461362 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05bc153950 0x7f05bc0c15d0 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f05a40097a0 tx=0x7f05a402b840 comp rx=0 tx=0).stop 2024-02-21T23:08:49.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:49.012+0000 7f05c2400640 1 -- 172.21.15.28:0/3555461362 shutdown_connections 2024-02-21T23:08:49.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:49.012+0000 7f05c2400640 1 --2- 172.21.15.28:0/3555461362 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05bc153950 0x7f05bc0c15d0 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:49.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:49.012+0000 7f05c2400640 1 -- 172.21.15.28:0/3555461362 >> 172.21.15.28:0/3555461362 conn(0x7f05bc0c67f0 msgr2=0x7f05bc0c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:49.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:49.012+0000 7f05c2400640 1 -- 172.21.15.28:0/3555461362 shutdown_connections 2024-02-21T23:08:49.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:49.012+0000 7f05c2400640 1 -- 172.21.15.28:0/3555461362 wait complete. 2024-02-21T23:08:49.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:49.014+0000 7f05c2400640 1 Processor -- start 2024-02-21T23:08:49.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:49.014+0000 7f05c2400640 1 -- start start 2024-02-21T23:08:49.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:49.015+0000 7f05c2400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05bc0c11f0 0x7f05bc1502a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:49.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:49.015+0000 7f05c2400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f05bc153950 0x7f05bc0a7680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:49.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:49.015+0000 7f05c2400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05bc15b6d0 0x7f05bc141730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:49.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:49.015+0000 7f05c2400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05bc0a6110 con 0x7f05bc15b6d0 2024-02-21T23:08:49.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:49.015+0000 7f05c2400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05bc0c4c00 con 0x7f05bc0c11f0 2024-02-21T23:08:49.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:49.015+0000 7f05c2400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05bc0c0d90 con 0x7f05bc153950 2024-02-21T23:08:49.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:49.015+0000 7f05bbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05bc15b6d0 0x7f05bc141730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:49.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:49.015+0000 7f05bbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05bc15b6d0 0x7f05bc141730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50354/0 (socket says 172.21.15.28:50354) 2024-02-21T23:08:49.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:49.015+0000 7f05bbe00640 1 -- 172.21.15.28:0/2069273120 learned_addr learned my addr 172.21.15.28:0/2069273120 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:49.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:49.016+0000 7f05b3e00640 1 --2- 172.21.15.28:0/2069273120 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f05bc153950 0x7f05bc0a7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:49.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:49.016+0000 7f05bb400640 1 --2- 172.21.15.28:0/2069273120 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05bc0c11f0 0x7f05bc1502a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:49.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:49.016+0000 7f05bbe00640 1 -- 172.21.15.28:0/2069273120 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f05bc153950 msgr2=0x7f05bc0a7680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:49.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:49.016+0000 7f05bbe00640 1 --2- 172.21.15.28:0/2069273120 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f05bc153950 0x7f05bc0a7680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:49.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:49.016+0000 7f05bbe00640 1 -- 172.21.15.28:0/2069273120 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05bc0c11f0 msgr2=0x7f05bc1502a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:49.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:49.016+0000 7f05bbe00640 1 --2- 172.21.15.28:0/2069273120 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05bc0c11f0 0x7f05bc1502a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:49.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:49.016+0000 7f05bbe00640 1 -- 172.21.15.28:0/2069273120 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05bc1422d0 con 0x7f05bc15b6d0 2024-02-21T23:08:49.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:49.016+0000 7f05b3e00640 1 --2- 172.21.15.28:0/2069273120 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f05bc153950 0x7f05bc0a7680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:08:49.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:49.016+0000 7f05bbe00640 1 --2- 172.21.15.28:0/2069273120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05bc15b6d0 0x7f05bc141730 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f05b400c7b0 tx=0x7f05b400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:49.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:49.016+0000 7f05b3400640 1 -- 172.21.15.28:0/2069273120 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b4004b60 con 0x7f05bc15b6d0 2024-02-21T23:08:49.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:49.017+0000 7f05b3400640 1 -- 172.21.15.28:0/2069273120 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05b4004d00 con 0x7f05bc15b6d0 2024-02-21T23:08:49.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:49.017+0000 7f05b3400640 1 -- 172.21.15.28:0/2069273120 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b400aa90 con 0x7f05bc15b6d0 2024-02-21T23:08:49.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:49.017+0000 7f05c2400640 1 -- 172.21.15.28:0/2069273120 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05bc144a50 con 0x7f05bc15b6d0 2024-02-21T23:08:49.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:49.017+0000 7f05c2400640 1 -- 172.21.15.28:0/2069273120 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05bc0a7bc0 con 0x7f05bc15b6d0 2024-02-21T23:08:49.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:49.018+0000 7f05c2400640 1 -- 172.21.15.28:0/2069273120 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05bc0c43e0 con 0x7f05bc15b6d0 2024-02-21T23:08:49.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:49.019+0000 7f05b3400640 1 -- 172.21.15.28:0/2069273120 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f05b400ac30 con 0x7f05bc15b6d0 2024-02-21T23:08:49.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:49.019+0000 7f05b3400640 1 --2- 172.21.15.28:0/2069273120 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f059806b500 0x7f059806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:49.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:49.020+0000 7f05b3400640 1 -- 172.21.15.28:0/2069273120 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f05b408a3d0 con 0x7f05bc15b6d0 2024-02-21T23:08:49.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:49.020+0000 7f05bb400640 1 --2- 172.21.15.28:0/2069273120 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f059806b500 0x7f059806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:49.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:49.021+0000 7f05bb400640 1 --2- 172.21.15.28:0/2069273120 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f059806b500 0x7f059806d9c0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f05ac000f60 tx=0x7f05ac007c00 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:49.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:49.024+0000 7f05b3400640 1 -- 172.21.15.28:0/2069273120 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f05bc0c43e0 con 0x7f05bc15b6d0 2024-02-21T23:08:49.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:49.300+0000 7f05c2400640 1 -- 172.21.15.28:0/2069273120 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_unset"]} v 0) v1 -- 0x7f05bc067f30 con 0x7f05bc15b6d0 2024-02-21T23:08:49.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:49.836+0000 7f05b3400640 1 -- 172.21.15.28:0/2069273120 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_unset"]}]=0 v366) v1 ==== 147+0+0 (secure 0 0 0) 0x7f05b4020070 con 0x7f05bc15b6d0 2024-02-21T23:08:49.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:49.840+0000 7f05c2400640 1 -- 172.21.15.28:0/2069273120 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f059806b500 msgr2=0x7f059806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:49.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:49.840+0000 7f05c2400640 1 --2- 172.21.15.28:0/2069273120 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f059806b500 0x7f059806d9c0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f05ac000f60 tx=0x7f05ac007c00 comp rx=0 tx=0).stop 2024-02-21T23:08:49.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:49.840+0000 7f05c2400640 1 -- 172.21.15.28:0/2069273120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05bc15b6d0 msgr2=0x7f05bc141730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:49.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:49.840+0000 7f05c2400640 1 --2- 172.21.15.28:0/2069273120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05bc15b6d0 0x7f05bc141730 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f05b400c7b0 tx=0x7f05b400cc70 comp rx=0 tx=0).stop 2024-02-21T23:08:49.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:49.840+0000 7f05c2400640 1 -- 172.21.15.28:0/2069273120 shutdown_connections 2024-02-21T23:08:49.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:49.840+0000 7f05c2400640 1 --2- 172.21.15.28:0/2069273120 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f059806b500 0x7f059806d9c0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:49.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:49.840+0000 7f05c2400640 1 --2- 172.21.15.28:0/2069273120 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f05bc153950 0x7f05bc0a7680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:49.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:49.840+0000 7f05c2400640 1 --2- 172.21.15.28:0/2069273120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05bc15b6d0 0x7f05bc141730 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:49.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:49.840+0000 7f05c2400640 1 --2- 172.21.15.28:0/2069273120 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05bc0c11f0 0x7f05bc1502a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:49.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:49.840+0000 7f05c2400640 1 -- 172.21.15.28:0/2069273120 >> 172.21.15.28:0/2069273120 conn(0x7f05bc0c67f0 msgr2=0x7f05bc0c5370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:49.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:49.840+0000 7f05c2400640 1 -- 172.21.15.28:0/2069273120 shutdown_connections 2024-02-21T23:08:49.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:49.840+0000 7f05c2400640 1 -- 172.21.15.28:0/2069273120 wait complete. 2024-02-21T23:08:49.849 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-02-21T23:08:50.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.085+0000 7fdc81400640 1 Processor -- start 2024-02-21T23:08:50.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.085+0000 7fdc81400640 1 -- start start 2024-02-21T23:08:50.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.086+0000 7fdc81400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdc7c0c5a70 0x7fdc7c155a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:50.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.086+0000 7fdc81400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fdc7c0c22e0 0x7fdc7c155fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:50.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.087+0000 7fdc81400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdc7c0c2b30 0x7fdc7c06d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:50.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.087+0000 7fdc81400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc7c072a10 con 0x7fdc7c0c2b30 2024-02-21T23:08:50.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.087+0000 7fdc81400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc7c072580 con 0x7fdc7c0c22e0 2024-02-21T23:08:50.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.087+0000 7fdc81400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7c071970 con 0x7fdc7c0c5a70 2024-02-21T23:08:50.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.087+0000 7fdc7b200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdc7c0c2b30 0x7fdc7c06d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:50.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.087+0000 7fdc7b200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdc7c0c2b30 0x7fdc7c06d120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50370/0 (socket says 172.21.15.28:50370) 2024-02-21T23:08:50.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.087+0000 7fdc7b200640 1 -- 172.21.15.28:0/3838357104 learned_addr learned my addr 172.21.15.28:0/3838357104 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:50.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.087+0000 7fdc7a800640 1 --2- 172.21.15.28:0/3838357104 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdc7c0c5a70 0x7fdc7c155a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:50.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.087+0000 7fdc73e00640 1 --2- 172.21.15.28:0/3838357104 >> v2:172.21.15.81:3300/0 conn(0x7fdc7c0c22e0 0x7fdc7c155fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:50.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.087+0000 7fdc7b200640 1 -- 172.21.15.28:0/3838357104 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdc7c0c5a70 msgr2=0x7fdc7c155a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:50.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.087+0000 7fdc7b200640 1 --2- 172.21.15.28:0/3838357104 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdc7c0c5a70 0x7fdc7c155a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:50.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.088+0000 7fdc7b200640 1 -- 172.21.15.28:0/3838357104 >> v2:172.21.15.81:3300/0 conn(0x7fdc7c0c22e0 msgr2=0x7fdc7c155fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:50.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.088+0000 7fdc7b200640 1 --2- 172.21.15.28:0/3838357104 >> v2:172.21.15.81:3300/0 conn(0x7fdc7c0c22e0 0x7fdc7c155fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:50.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.088+0000 7fdc7b200640 1 -- 172.21.15.28:0/3838357104 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc7c071c50 con 0x7fdc7c0c2b30 2024-02-21T23:08:50.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.089+0000 7fdc7b200640 1 --2- 172.21.15.28:0/3838357104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc7c0c2b30 0x7fdc7c06d120 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c0c7890 tx=0x7fdc7402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4347374be3e2e9ee server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:50.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.089+0000 7fdc79e00640 1 -- 172.21.15.28:0/3838357104 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc7402c9e0 con 0x7fdc7c0c2b30 2024-02-21T23:08:50.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.089+0000 7fdc79e00640 1 -- 172.21.15.28:0/3838357104 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc7402cb80 con 0x7fdc7c0c2b30 2024-02-21T23:08:50.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.089+0000 7fdc79e00640 1 -- 172.21.15.28:0/3838357104 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc740343f0 con 0x7fdc7c0c2b30 2024-02-21T23:08:50.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.090+0000 7fdc81400640 1 -- 172.21.15.28:0/3838357104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc7c0c2b30 msgr2=0x7fdc7c06d120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:50.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.090+0000 7fdc81400640 1 --2- 172.21.15.28:0/3838357104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc7c0c2b30 0x7fdc7c06d120 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c0c7890 tx=0x7fdc7402be40 comp rx=0 tx=0).stop 2024-02-21T23:08:50.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.090+0000 7fdc81400640 1 -- 172.21.15.28:0/3838357104 shutdown_connections 2024-02-21T23:08:50.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.090+0000 7fdc81400640 1 --2- 172.21.15.28:0/3838357104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc7c0c2b30 0x7fdc7c06d120 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:50.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.090+0000 7fdc81400640 1 --2- 172.21.15.28:0/3838357104 >> v2:172.21.15.81:3300/0 conn(0x7fdc7c0c22e0 0x7fdc7c155fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:50.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.090+0000 7fdc81400640 1 --2- 172.21.15.28:0/3838357104 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdc7c0c5a70 0x7fdc7c155a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:50.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.090+0000 7fdc81400640 1 -- 172.21.15.28:0/3838357104 >> 172.21.15.28:0/3838357104 conn(0x7fdc7c0a85a0 msgr2=0x7fdc7c0a8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:50.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.090+0000 7fdc81400640 1 -- 172.21.15.28:0/3838357104 shutdown_connections 2024-02-21T23:08:50.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.090+0000 7fdc81400640 1 -- 172.21.15.28:0/3838357104 wait complete. 2024-02-21T23:08:50.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.092+0000 7fdc81400640 1 Processor -- start 2024-02-21T23:08:50.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.092+0000 7fdc81400640 1 -- start start 2024-02-21T23:08:50.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.093+0000 7fdc81400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc7c0c22e0 0x7fdc7c0a6f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:50.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.093+0000 7fdc81400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc7c0c2b30 0x7fdc7c1e07f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:50.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.093+0000 7fdc81400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdc7c0c5a70 0x7fdc7c1e0d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:50.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.093+0000 7fdc81400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7c0a6160 con 0x7fdc7c0c2b30 2024-02-21T23:08:50.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.093+0000 7fdc81400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7c072210 con 0x7fdc7c0c22e0 2024-02-21T23:08:50.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.093+0000 7fdc81400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7c0a62e0 con 0x7fdc7c0c5a70 2024-02-21T23:08:50.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.093+0000 7fdc73e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc7c0c2b30 0x7fdc7c1e07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:50.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.093+0000 7fdc73e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc7c0c2b30 0x7fdc7c1e07f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50380/0 (socket says 172.21.15.28:50380) 2024-02-21T23:08:50.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.093+0000 7fdc73e00640 1 -- 172.21.15.28:0/2529369619 learned_addr learned my addr 172.21.15.28:0/2529369619 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:50.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.093+0000 7fdc7b200640 1 --2- 172.21.15.28:0/2529369619 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdc7c0c5a70 0x7fdc7c1e0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:50.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.093+0000 7fdc73e00640 1 -- 172.21.15.28:0/2529369619 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdc7c0c5a70 msgr2=0x7fdc7c1e0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:50.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.093+0000 7fdc73e00640 1 --2- 172.21.15.28:0/2529369619 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdc7c0c5a70 0x7fdc7c1e0d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:50.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.093+0000 7fdc73e00640 1 -- 172.21.15.28:0/2529369619 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc7c0c22e0 msgr2=0x7fdc7c0a6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:50.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.093+0000 7fdc73e00640 1 --2- 172.21.15.28:0/2529369619 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc7c0c22e0 0x7fdc7c0a6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:50.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.093+0000 7fdc73e00640 1 -- 172.21.15.28:0/2529369619 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc7c1de150 con 0x7fdc7c0c2b30 2024-02-21T23:08:50.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.094+0000 7fdc73e00640 1 --2- 172.21.15.28:0/2529369619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc7c0c2b30 0x7fdc7c1e07f0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7fdc6400d5c0 tx=0x7fdc6400da80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:50.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.094+0000 7fdc72a00640 1 -- 172.21.15.28:0/2529369619 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc640125f0 con 0x7fdc7c0c2b30 2024-02-21T23:08:50.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.094+0000 7fdc72a00640 1 -- 172.21.15.28:0/2529369619 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc64019070 con 0x7fdc7c0c2b30 2024-02-21T23:08:50.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.094+0000 7fdc72a00640 1 -- 172.21.15.28:0/2529369619 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc640129d0 con 0x7fdc7c0c2b30 2024-02-21T23:08:50.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.094+0000 7fdc81400640 1 -- 172.21.15.28:0/2529369619 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc7c1de430 con 0x7fdc7c0c2b30 2024-02-21T23:08:50.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.094+0000 7fdc81400640 1 -- 172.21.15.28:0/2529369619 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc7c1df550 con 0x7fdc7c0c2b30 2024-02-21T23:08:50.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.095+0000 7fdc81400640 1 -- 172.21.15.28:0/2529369619 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc7c0c43e0 con 0x7fdc7c0c2b30 2024-02-21T23:08:50.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.096+0000 7fdc72a00640 1 -- 172.21.15.28:0/2529369619 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdc64012b70 con 0x7fdc7c0c2b30 2024-02-21T23:08:50.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.097+0000 7fdc72a00640 1 --2- 172.21.15.28:0/2529369619 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdc5806b4d0 0x7fdc5806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:50.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.097+0000 7fdc72a00640 1 -- 172.21.15.28:0/2529369619 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5507+0+0 (secure 0 0 0) 0x7fdc6408a480 con 0x7fdc7c0c2b30 2024-02-21T23:08:50.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.098+0000 7fdc7a800640 1 --2- 172.21.15.28:0/2529369619 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdc5806b4d0 0x7fdc5806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:50.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.099+0000 7fdc7a800640 1 --2- 172.21.15.28:0/2529369619 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdc5806b4d0 0x7fdc5806d990 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c155710 tx=0x7fdc6c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:50.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.101+0000 7fdc72a00640 1 -- 172.21.15.28:0/2529369619 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fdc7c0c43e0 con 0x7fdc7c0c2b30 2024-02-21T23:08:50.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.300+0000 7fdc81400640 1 -- 172.21.15.28:0/2529369619 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fdc7c0c5e50 con 0x7fdc7c0c2b30 2024-02-21T23:08:50.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.307+0000 7fdc72a00640 1 -- 172.21.15.28:0/2529369619 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v45)=0 v45) v1 ==== 122+0+0 (secure 0 0 0) 0x7fdc7c0c5e50 con 0x7fdc7c0c2b30 2024-02-21T23:08:50.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.309+0000 7fdc81400640 1 -- 172.21.15.28:0/2529369619 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdc5806b4d0 msgr2=0x7fdc5806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:50.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.309+0000 7fdc81400640 1 --2- 172.21.15.28:0/2529369619 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdc5806b4d0 0x7fdc5806d990 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c155710 tx=0x7fdc6c008040 comp rx=0 tx=0).stop 2024-02-21T23:08:50.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.309+0000 7fdc81400640 1 -- 172.21.15.28:0/2529369619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc7c0c2b30 msgr2=0x7fdc7c1e07f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:50.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.309+0000 7fdc81400640 1 --2- 172.21.15.28:0/2529369619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc7c0c2b30 0x7fdc7c1e07f0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7fdc6400d5c0 tx=0x7fdc6400da80 comp rx=0 tx=0).stop 2024-02-21T23:08:50.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.310+0000 7fdc81400640 1 -- 172.21.15.28:0/2529369619 shutdown_connections 2024-02-21T23:08:50.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.310+0000 7fdc81400640 1 --2- 172.21.15.28:0/2529369619 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdc5806b4d0 0x7fdc5806d990 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:50.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.310+0000 7fdc81400640 1 --2- 172.21.15.28:0/2529369619 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdc7c0c5a70 0x7fdc7c1e0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:50.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.310+0000 7fdc81400640 1 --2- 172.21.15.28:0/2529369619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc7c0c2b30 0x7fdc7c1e07f0 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:50.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.310+0000 7fdc81400640 1 --2- 172.21.15.28:0/2529369619 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc7c0c22e0 0x7fdc7c0a6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:50.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.310+0000 7fdc81400640 1 -- 172.21.15.28:0/2529369619 >> 172.21.15.28:0/2529369619 conn(0x7fdc7c0a85a0 msgr2=0x7fdc7c06e110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:50.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.310+0000 7fdc81400640 1 -- 172.21.15.28:0/2529369619 shutdown_connections 2024-02-21T23:08:50.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.310+0000 7fdc81400640 1 -- 172.21.15.28:0/2529369619 wait complete. 2024-02-21T23:08:50.314 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-02-21T23:08:50.315 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-21T23:08:50.428 DEBUG:teuthology.orchestra.run.smithi081:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-21T23:08:50.544 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-02-21T23:08:50.545 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-21T23:08:50.545 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-21T23:08:50.604 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054608,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-02-21T23:08:50.604 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-02-21T23:08:50.604 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-21T23:08:50.662 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053833,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498812k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249404k,nr_inodes=812351,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498812k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-02-21T23:08:50.662 INFO:tasks.cephfs.cephfs_test_case:[, ] 2024-02-21T23:08:50.662 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:08:50.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.909+0000 7f5608200640 1 Processor -- start 2024-02-21T23:08:50.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.909+0000 7f5608200640 1 -- start start 2024-02-21T23:08:50.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.911+0000 7f5608200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f56000c11f0 0x7f5600153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:50.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.911+0000 7f5608200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f56000c15d0 0x7f560006dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:50.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.911+0000 7f5608200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5600156350 0x7f5600156780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:50.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.911+0000 7f5608200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56000c1a60 con 0x7f56000c15d0 2024-02-21T23:08:50.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.911+0000 7f5608200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56000a5ec0 con 0x7f56000c11f0 2024-02-21T23:08:50.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.911+0000 7f5608200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5600152e70 con 0x7f5600156350 2024-02-21T23:08:50.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.911+0000 7f5604e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f56000c15d0 0x7f560006dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:50.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.911+0000 7f5604e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f56000c15d0 0x7f560006dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50386/0 (socket says 172.21.15.28:50386) 2024-02-21T23:08:50.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.911+0000 7f5604e00640 1 -- 172.21.15.28:0/351979086 learned_addr learned my addr 172.21.15.28:0/351979086 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:50.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.911+0000 7f5605800640 1 --2- 172.21.15.28:0/351979086 >> v2:172.21.15.81:3300/0 conn(0x7f56000c11f0 0x7f5600153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:50.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.911+0000 7f5606200640 1 --2- 172.21.15.28:0/351979086 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5600156350 0x7f5600156780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:50.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.912+0000 7f5604e00640 1 -- 172.21.15.28:0/351979086 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5600156350 msgr2=0x7f5600156780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:50.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.912+0000 7f5604e00640 1 --2- 172.21.15.28:0/351979086 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5600156350 0x7f5600156780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:50.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.912+0000 7f5604e00640 1 -- 172.21.15.28:0/351979086 >> v2:172.21.15.81:3300/0 conn(0x7f56000c11f0 msgr2=0x7f5600153950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:50.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.912+0000 7f5604e00640 1 --2- 172.21.15.28:0/351979086 >> v2:172.21.15.81:3300/0 conn(0x7f56000c11f0 0x7f5600153950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:50.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.912+0000 7f5604e00640 1 -- 172.21.15.28:0/351979086 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5600153150 con 0x7f56000c15d0 2024-02-21T23:08:50.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.913+0000 7f5604e00640 1 --2- 172.21.15.28:0/351979086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56000c15d0 0x7f560006dce0 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f56000c7590 tx=0x7f55ec02b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9d7b88c3bce5bff server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:50.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.913+0000 7f55ffe00640 1 -- 172.21.15.28:0/351979086 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55ec038070 con 0x7f56000c15d0 2024-02-21T23:08:50.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.913+0000 7f55ffe00640 1 -- 172.21.15.28:0/351979086 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55ec02f050 con 0x7f56000c15d0 2024-02-21T23:08:50.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.913+0000 7f55ffe00640 1 -- 172.21.15.28:0/351979086 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55ec031970 con 0x7f56000c15d0 2024-02-21T23:08:50.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.914+0000 7f5608200640 1 -- 172.21.15.28:0/351979086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56000c15d0 msgr2=0x7f560006dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:50.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.914+0000 7f5608200640 1 --2- 172.21.15.28:0/351979086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56000c15d0 0x7f560006dce0 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f56000c7590 tx=0x7f55ec02b7f0 comp rx=0 tx=0).stop 2024-02-21T23:08:50.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.914+0000 7f5608200640 1 -- 172.21.15.28:0/351979086 shutdown_connections 2024-02-21T23:08:50.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.914+0000 7f5608200640 1 --2- 172.21.15.28:0/351979086 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5600156350 0x7f5600156780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:50.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.914+0000 7f5608200640 1 --2- 172.21.15.28:0/351979086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56000c15d0 0x7f560006dce0 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:50.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.914+0000 7f5608200640 1 --2- 172.21.15.28:0/351979086 >> v2:172.21.15.81:3300/0 conn(0x7f56000c11f0 0x7f5600153950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:50.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.914+0000 7f5608200640 1 -- 172.21.15.28:0/351979086 >> 172.21.15.28:0/351979086 conn(0x7f56000c67f0 msgr2=0x7f56000c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:50.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.914+0000 7f5608200640 1 -- 172.21.15.28:0/351979086 shutdown_connections 2024-02-21T23:08:50.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.914+0000 7f5608200640 1 -- 172.21.15.28:0/351979086 wait complete. 2024-02-21T23:08:50.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.916+0000 7f5608200640 1 Processor -- start 2024-02-21T23:08:50.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.916+0000 7f5608200640 1 -- start start 2024-02-21T23:08:50.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.917+0000 7f5608200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56000c11f0 0x7f5600154520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:50.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.917+0000 7f5608200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f56000c15d0 0x7f5600154d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:50.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.917+0000 7f5608200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5600156350 0x7f56000a6e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:50.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.917+0000 7f5608200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56000c4c00 con 0x7f5600156350 2024-02-21T23:08:50.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.917+0000 7f5608200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f560014d1c0 con 0x7f56000c11f0 2024-02-21T23:08:50.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.917+0000 7f5608200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56000a6110 con 0x7f56000c15d0 2024-02-21T23:08:50.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.917+0000 7f5606200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5600156350 0x7f56000a6e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:50.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.918+0000 7f5606200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5600156350 0x7f56000a6e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50388/0 (socket says 172.21.15.28:50388) 2024-02-21T23:08:50.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.918+0000 7f5606200640 1 -- 172.21.15.28:0/2597479942 learned_addr learned my addr 172.21.15.28:0/2597479942 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:50.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.918+0000 7f5605800640 1 --2- 172.21.15.28:0/2597479942 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56000c11f0 0x7f5600154520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:50.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.918+0000 7f5604e00640 1 --2- 172.21.15.28:0/2597479942 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f56000c15d0 0x7f5600154d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:50.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.918+0000 7f5606200640 1 -- 172.21.15.28:0/2597479942 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f56000c15d0 msgr2=0x7f5600154d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:50.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.918+0000 7f5606200640 1 --2- 172.21.15.28:0/2597479942 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f56000c15d0 0x7f5600154d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:50.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.918+0000 7f5606200640 1 -- 172.21.15.28:0/2597479942 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56000c11f0 msgr2=0x7f5600154520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:50.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.918+0000 7f5606200640 1 --2- 172.21.15.28:0/2597479942 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56000c11f0 0x7f5600154520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:50.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.918+0000 7f5606200640 1 -- 172.21.15.28:0/2597479942 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56000a63d0 con 0x7f5600156350 2024-02-21T23:08:50.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.918+0000 7f5606200640 1 --2- 172.21.15.28:0/2597479942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5600156350 0x7f56000a6e90 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f55f8007e70 tx=0x7f55f800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:50.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.919+0000 7f55fd600640 1 -- 172.21.15.28:0/2597479942 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55f8014620 con 0x7f5600156350 2024-02-21T23:08:50.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.919+0000 7f55fd600640 1 -- 172.21.15.28:0/2597479942 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55f8019070 con 0x7f5600156350 2024-02-21T23:08:50.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.919+0000 7f5608200640 1 -- 172.21.15.28:0/2597479942 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5600072100 con 0x7f5600156350 2024-02-21T23:08:50.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.919+0000 7f55fd600640 1 -- 172.21.15.28:0/2597479942 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55f8014a20 con 0x7f5600156350 2024-02-21T23:08:50.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.919+0000 7f5608200640 1 -- 172.21.15.28:0/2597479942 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56000a6690 con 0x7f5600156350 2024-02-21T23:08:50.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.920+0000 7f5608200640 1 -- 172.21.15.28:0/2597479942 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5600072620 con 0x7f5600156350 2024-02-21T23:08:50.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.921+0000 7f55fd600640 1 -- 172.21.15.28:0/2597479942 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f55f8014bc0 con 0x7f5600156350 2024-02-21T23:08:50.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.922+0000 7f55fd600640 1 --2- 172.21.15.28:0/2597479942 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f55e006b500 0x7f55e006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:50.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.922+0000 7f55fd600640 1 -- 172.21.15.28:0/2597479942 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f55f808ab70 con 0x7f5600156350 2024-02-21T23:08:50.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.923+0000 7f5605800640 1 --2- 172.21.15.28:0/2597479942 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f55e006b500 0x7f55e006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:50.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.924+0000 7f5605800640 1 --2- 172.21.15.28:0/2597479942 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f55e006b500 0x7f55e006d9c0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f55f4006fd0 tx=0x7f55f4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:50.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:50.926+0000 7f55fd600640 1 -- 172.21.15.28:0/2597479942 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f55f8059980 con 0x7f5600156350 2024-02-21T23:08:51.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:51.169+0000 7f5608200640 1 -- 172.21.15.28:0/2597479942 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5600067f30 con 0x7f5600156350 2024-02-21T23:08:51.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:51.171+0000 7f55fd600640 1 -- 172.21.15.28:0/2597479942 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 132 v132) v1 ==== 95+0+3701 (secure 0 0 0) 0x7f55f8059280 con 0x7f5600156350 2024-02-21T23:08:51.159 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:08:51.159 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":132,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5948,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3556518453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3556518453},{"type":"v1","addr":"172.21.15.28:6835","nonce":3556518453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116},{"gid":5951,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/3293264689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":3293264689},{"type":"v1","addr":"172.21.15.81:6836","nonce":3293264689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":116},{"gid":6116,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/596588360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":596588360},{"type":"v1","addr":"172.21.15.81:6837","nonce":596588360}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":132},{"gid":8157,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/1285325008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1285325008},{"type":"v1","addr":"172.21.15.28:6834","nonce":1285325008}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":128}],"filesystems":[]} 2024-02-21T23:08:51.159 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 132 2024-02-21T23:08:51.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:51.174+0000 7f5608200640 1 -- 172.21.15.28:0/2597479942 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f55e006b500 msgr2=0x7f55e006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:51.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:51.174+0000 7f5608200640 1 --2- 172.21.15.28:0/2597479942 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f55e006b500 0x7f55e006d9c0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f55f4006fd0 tx=0x7f55f4008040 comp rx=0 tx=0).stop 2024-02-21T23:08:51.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:51.174+0000 7f5608200640 1 -- 172.21.15.28:0/2597479942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5600156350 msgr2=0x7f56000a6e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:51.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:51.174+0000 7f5608200640 1 --2- 172.21.15.28:0/2597479942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5600156350 0x7f56000a6e90 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f55f8007e70 tx=0x7f55f800e4b0 comp rx=0 tx=0).stop 2024-02-21T23:08:51.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:51.174+0000 7f5608200640 1 -- 172.21.15.28:0/2597479942 shutdown_connections 2024-02-21T23:08:51.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:51.174+0000 7f5608200640 1 --2- 172.21.15.28:0/2597479942 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f55e006b500 0x7f55e006d9c0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:51.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:51.174+0000 7f5608200640 1 --2- 172.21.15.28:0/2597479942 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f56000c15d0 0x7f5600154d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:51.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:51.174+0000 7f5608200640 1 --2- 172.21.15.28:0/2597479942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5600156350 0x7f56000a6e90 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:51.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:51.174+0000 7f5608200640 1 --2- 172.21.15.28:0/2597479942 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56000c11f0 0x7f5600154520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:51.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:51.174+0000 7f5608200640 1 -- 172.21.15.28:0/2597479942 >> 172.21.15.28:0/2597479942 conn(0x7f56000c67f0 msgr2=0x7f56000c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:51.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:51.174+0000 7f5608200640 1 -- 172.21.15.28:0/2597479942 shutdown_connections 2024-02-21T23:08:51.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:51.174+0000 7f5608200640 1 -- 172.21.15.28:0/2597479942 wait complete. 2024-02-21T23:08:51.183 INFO:tasks.ceph.mds.a:Restarting daemon 2024-02-21T23:08:51.184 INFO:tasks.ceph.mds.a:Stopping old one... 2024-02-21T23:08:51.184 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-02-21T23:08:51.184 INFO:teuthology.orchestra.run:waiting for 300 2024-02-21T23:08:51.184 INFO:tasks.ceph.mds.c:Restarting daemon 2024-02-21T23:08:51.184 INFO:tasks.ceph.mds.c:Stopping old one... 2024-02-21T23:08:51.184 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-02-21T23:08:51.185 INFO:teuthology.orchestra.run:waiting for 300 2024-02-21T23:08:51.185 INFO:tasks.ceph.mds.b:Restarting daemon 2024-02-21T23:08:51.185 INFO:tasks.ceph.mds.b:Stopping old one... 2024-02-21T23:08:51.185 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-02-21T23:08:51.185 INFO:teuthology.orchestra.run:waiting for 300 2024-02-21T23:08:51.185 INFO:tasks.ceph.mds.d:Restarting daemon 2024-02-21T23:08:51.185 INFO:tasks.ceph.mds.d:Stopping old one... 2024-02-21T23:08:51.186 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-02-21T23:08:51.186 INFO:teuthology.orchestra.run:waiting for 300 2024-02-21T23:08:51.204 INFO:tasks.ceph.mds.c:Stopped 2024-02-21T23:08:51.205 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-02-21T23:08:51.206 INFO:tasks.ceph.mds.b:Stopped 2024-02-21T23:08:51.206 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-02-21T23:08:51.208 INFO:tasks.ceph.mds.d:Stopped 2024-02-21T23:08:51.208 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-02-21T23:08:51.252 INFO:tasks.ceph.mds.c:Started 2024-02-21T23:08:51.253 DEBUG:teuthology.parallel:result is None 2024-02-21T23:08:51.255 INFO:tasks.ceph.mds.b:Started 2024-02-21T23:08:51.255 DEBUG:teuthology.parallel:result is None 2024-02-21T23:08:51.261 INFO:tasks.ceph.mds.d:Started 2024-02-21T23:08:51.261 DEBUG:teuthology.parallel:result is None 2024-02-21T23:08:51.422 INFO:tasks.ceph.mds.b.smithi081.stdout:starting mds.b at 2024-02-21T23:08:51.422 INFO:tasks.ceph.mds.d.smithi081.stdout:starting mds.d at 2024-02-21T23:08:51.444 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-02-21T23:08:52.731 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~0s 2024-02-21T23:08:57.205 INFO:tasks.ceph.mds.a:Stopped 2024-02-21T23:08:57.205 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-02-21T23:08:57.214 INFO:tasks.ceph.mds.a:Started 2024-02-21T23:08:57.214 DEBUG:teuthology.parallel:result is None 2024-02-21T23:08:57.214 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-02-21T23:08:57.382 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-02-21T23:08:57.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:57.425+0000 7f3194000640 1 Processor -- start 2024-02-21T23:08:57.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:57.425+0000 7f3194000640 1 -- start start 2024-02-21T23:08:57.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:57.426+0000 7f3194000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f318c0c2df0 0x7f318c156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:57.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:57.426+0000 7f3194000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f318c0c5f20 0x7f318c0c31d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:57.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:57.426+0000 7f3194000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f318c0c3710 0x7f318c153220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:57.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:57.426+0000 7f3194000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f318c06e6a0 con 0x7f318c0c2df0 2024-02-21T23:08:57.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:57.426+0000 7f3194000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f318c0c5c10 con 0x7f318c0c5f20 2024-02-21T23:08:57.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:57.426+0000 7f3194000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f318c06e8d0 con 0x7f318c0c3710 2024-02-21T23:08:57.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:57.427+0000 7f3191600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f318c0c2df0 0x7f318c156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:57.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:57.427+0000 7f3191600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f318c0c2df0 0x7f318c156ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41248/0 (socket says 172.21.15.28:41248) 2024-02-21T23:08:57.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:57.427+0000 7f3191600640 1 -- 172.21.15.28:0/606620612 learned_addr learned my addr 172.21.15.28:0/606620612 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:57.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:57.427+0000 7f3190c00640 1 --2- 172.21.15.28:0/606620612 >> v2:172.21.15.81:3300/0 conn(0x7f318c0c5f20 0x7f318c0c31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:57.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:57.427+0000 7f3192000640 1 --2- 172.21.15.28:0/606620612 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f318c0c3710 0x7f318c153220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:57.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:57.427+0000 7f3191600640 1 -- 172.21.15.28:0/606620612 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f318c0c3710 msgr2=0x7f318c153220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:57.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:57.427+0000 7f3191600640 1 --2- 172.21.15.28:0/606620612 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f318c0c3710 0x7f318c153220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:57.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:57.427+0000 7f3191600640 1 -- 172.21.15.28:0/606620612 >> v2:172.21.15.81:3300/0 conn(0x7f318c0c5f20 msgr2=0x7f318c0c31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:57.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:57.427+0000 7f3191600640 1 --2- 172.21.15.28:0/606620612 >> v2:172.21.15.81:3300/0 conn(0x7f318c0c5f20 0x7f318c0c31d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:57.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:57.427+0000 7f3191600640 1 -- 172.21.15.28:0/606620612 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f318c06ebb0 con 0x7f318c0c2df0 2024-02-21T23:08:57.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:57.428+0000 7f3191600640 1 --2- 172.21.15.28:0/606620612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f318c0c2df0 0x7f318c156ed0 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7f318c06f8b0 tx=0x7f318002b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75775af0880b1bc1 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:57.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:57.429+0000 7f318be00640 1 -- 172.21.15.28:0/606620612 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3180035020 con 0x7f318c0c2df0 2024-02-21T23:08:57.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:57.429+0000 7f318be00640 1 -- 172.21.15.28:0/606620612 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f318002d040 con 0x7f318c0c2df0 2024-02-21T23:08:57.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:57.429+0000 7f318be00640 1 -- 172.21.15.28:0/606620612 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3180031a80 con 0x7f318c0c2df0 2024-02-21T23:08:57.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:57.429+0000 7f3194000640 1 -- 172.21.15.28:0/606620612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f318c0c2df0 msgr2=0x7f318c156ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:57.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:57.429+0000 7f3194000640 1 --2- 172.21.15.28:0/606620612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f318c0c2df0 0x7f318c156ed0 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7f318c06f8b0 tx=0x7f318002b900 comp rx=0 tx=0).stop 2024-02-21T23:08:57.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:57.429+0000 7f3194000640 1 -- 172.21.15.28:0/606620612 shutdown_connections 2024-02-21T23:08:57.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:57.429+0000 7f3194000640 1 --2- 172.21.15.28:0/606620612 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f318c0c3710 0x7f318c153220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:57.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:57.429+0000 7f3194000640 1 --2- 172.21.15.28:0/606620612 >> v2:172.21.15.81:3300/0 conn(0x7f318c0c5f20 0x7f318c0c31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:57.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:57.429+0000 7f3194000640 1 --2- 172.21.15.28:0/606620612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f318c0c2df0 0x7f318c156ed0 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:57.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:57.429+0000 7f3194000640 1 -- 172.21.15.28:0/606620612 >> 172.21.15.28:0/606620612 conn(0x7f318c06b820 msgr2=0x7f318c0c67f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:57.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:57.429+0000 7f3194000640 1 -- 172.21.15.28:0/606620612 shutdown_connections 2024-02-21T23:08:57.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:57.430+0000 7f3194000640 1 -- 172.21.15.28:0/606620612 wait complete. 2024-02-21T23:08:57.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:57.431+0000 7f3194000640 1 Processor -- start 2024-02-21T23:08:57.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:57.431+0000 7f3194000640 1 -- start start 2024-02-21T23:08:57.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:57.432+0000 7f3194000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f318c0c2df0 0x7f318c0721e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:57.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:57.432+0000 7f3194000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f318c0c3710 0x7f318c0aa510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:57.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:57.432+0000 7f3194000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f318c0c5f20 0x7f318c0aaa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:57.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:57.432+0000 7f3194000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f318c153760 con 0x7f318c0c2df0 2024-02-21T23:08:57.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:57.432+0000 7f3194000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f318c06f170 con 0x7f318c0c3710 2024-02-21T23:08:57.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:57.432+0000 7f3194000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f318c1538e0 con 0x7f318c0c5f20 2024-02-21T23:08:57.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:57.432+0000 7f3191600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f318c0c2df0 0x7f318c0721e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:57.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:57.432+0000 7f3191600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f318c0c2df0 0x7f318c0721e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41256/0 (socket says 172.21.15.28:41256) 2024-02-21T23:08:57.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:57.432+0000 7f3191600640 1 -- 172.21.15.28:0/2477719534 learned_addr learned my addr 172.21.15.28:0/2477719534 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:57.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:57.433+0000 7f3191600640 1 -- 172.21.15.28:0/2477719534 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f318c0c5f20 msgr2=0x7f318c0aaa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:57.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:57.433+0000 7f3190c00640 1 --2- 172.21.15.28:0/2477719534 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f318c0c3710 0x7f318c0aa510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:57.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:57.433+0000 7f3191600640 1 --2- 172.21.15.28:0/2477719534 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f318c0c5f20 0x7f318c0aaa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:57.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:57.433+0000 7f3191600640 1 -- 172.21.15.28:0/2477719534 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f318c0c3710 msgr2=0x7f318c0aa510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:57.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:57.433+0000 7f3191600640 1 --2- 172.21.15.28:0/2477719534 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f318c0c3710 0x7f318c0aa510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:57.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:57.433+0000 7f3191600640 1 -- 172.21.15.28:0/2477719534 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f318c1eb740 con 0x7f318c0c2df0 2024-02-21T23:08:57.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:57.433+0000 7f3191600640 1 --2- 172.21.15.28:0/2477719534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f318c0c2df0 0x7f318c0721e0 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f318002b940 tx=0x7f31800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:57.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:57.434+0000 7f3189600640 1 -- 172.21.15.28:0/2477719534 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3180040070 con 0x7f318c0c2df0 2024-02-21T23:08:57.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:57.434+0000 7f3189600640 1 -- 172.21.15.28:0/2477719534 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3180039c10 con 0x7f318c0c2df0 2024-02-21T23:08:57.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:57.434+0000 7f3194000640 1 -- 172.21.15.28:0/2477719534 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f318c1eba20 con 0x7f318c0c2df0 2024-02-21T23:08:57.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:57.434+0000 7f3189600640 1 -- 172.21.15.28:0/2477719534 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3180035040 con 0x7f318c0c2df0 2024-02-21T23:08:57.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:57.434+0000 7f3194000640 1 -- 172.21.15.28:0/2477719534 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f318c072720 con 0x7f318c0c2df0 2024-02-21T23:08:57.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:57.435+0000 7f3194000640 1 -- 172.21.15.28:0/2477719534 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f318c0aa040 con 0x7f318c0c2df0 2024-02-21T23:08:57.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:57.436+0000 7f3189600640 1 -- 172.21.15.28:0/2477719534 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3180038020 con 0x7f318c0c2df0 2024-02-21T23:08:57.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:57.437+0000 7f3189600640 1 --2- 172.21.15.28:0/2477719534 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f316c06b4d0 0x7f316c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:57.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:57.437+0000 7f3189600640 1 -- 172.21.15.28:0/2477719534 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f31800abda0 con 0x7f318c0c2df0 2024-02-21T23:08:57.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:57.438+0000 7f3190c00640 1 --2- 172.21.15.28:0/2477719534 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f316c06b4d0 0x7f316c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:57.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:57.439+0000 7f3190c00640 1 --2- 172.21.15.28:0/2477719534 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f316c06b4d0 0x7f316c06d990 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f3178006fd0 tx=0x7f3178008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:57.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:57.442+0000 7f3189600640 1 -- 172.21.15.28:0/2477719534 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f318007acd0 con 0x7f318c0c2df0 2024-02-21T23:08:57.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:57.657+0000 7f3194000640 1 -- 172.21.15.28:0/2477719534 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f318c06f400 con 0x7f318c0c2df0 2024-02-21T23:08:59.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.329+0000 7f3189600640 1 -- 172.21.15.28:0/2477719534 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v132) v1 ==== 97+0+0 (secure 0 0 0) 0x7f318007a5d0 con 0x7f318c0c2df0 2024-02-21T23:08:59.319 INFO:teuthology.orchestra.run.smithi028.stderr: removed all blocklist entries 2024-02-21T23:08:59.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.333+0000 7f3194000640 1 -- 172.21.15.28:0/2477719534 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f316c06b4d0 msgr2=0x7f316c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:59.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.333+0000 7f3194000640 1 --2- 172.21.15.28:0/2477719534 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f316c06b4d0 0x7f316c06d990 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f3178006fd0 tx=0x7f3178008040 comp rx=0 tx=0).stop 2024-02-21T23:08:59.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.333+0000 7f3194000640 1 -- 172.21.15.28:0/2477719534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f318c0c2df0 msgr2=0x7f318c0721e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:59.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.333+0000 7f3194000640 1 --2- 172.21.15.28:0/2477719534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f318c0c2df0 0x7f318c0721e0 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f318002b940 tx=0x7f31800046c0 comp rx=0 tx=0).stop 2024-02-21T23:08:59.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.333+0000 7f3194000640 1 -- 172.21.15.28:0/2477719534 shutdown_connections 2024-02-21T23:08:59.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.333+0000 7f3194000640 1 --2- 172.21.15.28:0/2477719534 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f316c06b4d0 0x7f316c06d990 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:59.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.333+0000 7f3194000640 1 --2- 172.21.15.28:0/2477719534 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f318c0c5f20 0x7f318c0aaa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:59.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.333+0000 7f3194000640 1 --2- 172.21.15.28:0/2477719534 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f318c0c3710 0x7f318c0aa510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:59.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.333+0000 7f3194000640 1 --2- 172.21.15.28:0/2477719534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f318c0c2df0 0x7f318c0721e0 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:59.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.333+0000 7f3194000640 1 -- 172.21.15.28:0/2477719534 >> 172.21.15.28:0/2477719534 conn(0x7f318c06b820 msgr2=0x7f318c0c43e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:59.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.333+0000 7f3194000640 1 -- 172.21.15.28:0/2477719534 shutdown_connections 2024-02-21T23:08:59.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.334+0000 7f3194000640 1 -- 172.21.15.28:0/2477719534 wait complete. 2024-02-21T23:08:59.344 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-02-21T23:08:59.445 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a has been restored 2024-02-21T23:08:59.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.575+0000 7fcba7200640 1 Processor -- start 2024-02-21T23:08:59.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.575+0000 7fcba7200640 1 -- start start 2024-02-21T23:08:59.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.576+0000 7fcba7200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcba0072430 0x7fcba00c3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:59.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.576+0000 7fcba7200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcba0155860 0x7fcba0155d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:59.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.576+0000 7fcba7200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcba00c6560 con 0x7fcba00c3280 2024-02-21T23:08:59.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.576+0000 7fcba7200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcba00c5cc0 con 0x7fcba0072430 2024-02-21T23:08:59.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.576+0000 7fcba7200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcba00c5fa0 con 0x7fcba0155860 2024-02-21T23:08:59.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.576+0000 7fcb9f400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcba0072430 0x7fcba00c3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:59.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.577+0000 7fcb9f400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcba0072430 0x7fcba00c3660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41264/0 (socket says 172.21.15.28:41264) 2024-02-21T23:08:59.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.577+0000 7fcb9f400640 1 -- 172.21.15.28:0/1223675080 learned_addr learned my addr 172.21.15.28:0/1223675080 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:59.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.577+0000 7fcb9f400640 1 -- 172.21.15.28:0/1223675080 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcba0155860 msgr2=0x7fcba0155d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:59.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.577+0000 7fcb9f400640 1 --2- 172.21.15.28:0/1223675080 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcba0155860 0x7fcba0155d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:59.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.577+0000 7fcb9f400640 1 -- 172.21.15.28:0/1223675080 >> v1:172.21.15.28:6789/0 conn(0x7fcba00c3280 legacy=0x7fcba01526e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:08:59.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.577+0000 7fcb9f400640 1 -- 172.21.15.28:0/1223675080 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcba00c6280 con 0x7fcba0072430 2024-02-21T23:08:59.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.578+0000 7fcb9f400640 1 --2- 172.21.15.28:0/1223675080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcba0072430 0x7fcba00c3660 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7fcb88009870 tx=0x7fcb8802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c3a0565998a479 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:59.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.578+0000 7fcb9ea00640 1 -- 172.21.15.28:0/1223675080 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb880317c0 con 0x7fcba0072430 2024-02-21T23:08:59.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.579+0000 7fcb9ea00640 1 -- 172.21.15.28:0/1223675080 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb88037070 con 0x7fcba0072430 2024-02-21T23:08:59.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.579+0000 7fcb9ea00640 1 -- 172.21.15.28:0/1223675080 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb88031b80 con 0x7fcba0072430 2024-02-21T23:08:59.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.579+0000 7fcba7200640 1 -- 172.21.15.28:0/1223675080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcba0072430 msgr2=0x7fcba00c3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:59.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.579+0000 7fcba7200640 1 --2- 172.21.15.28:0/1223675080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcba0072430 0x7fcba00c3660 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7fcb88009870 tx=0x7fcb8802b860 comp rx=0 tx=0).stop 2024-02-21T23:08:59.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.579+0000 7fcba7200640 1 -- 172.21.15.28:0/1223675080 shutdown_connections 2024-02-21T23:08:59.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.579+0000 7fcba7200640 1 --2- 172.21.15.28:0/1223675080 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcba0155860 0x7fcba0155d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:59.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.579+0000 7fcba7200640 1 --2- 172.21.15.28:0/1223675080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcba0072430 0x7fcba00c3660 unknown :-1 s=CLOSED pgs=1497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:59.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.579+0000 7fcba7200640 1 -- 172.21.15.28:0/1223675080 >> 172.21.15.28:0/1223675080 conn(0x7fcba00c67f0 msgr2=0x7fcba00c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:59.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.579+0000 7fcba7200640 1 -- 172.21.15.28:0/1223675080 shutdown_connections 2024-02-21T23:08:59.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.579+0000 7fcba7200640 1 -- 172.21.15.28:0/1223675080 wait complete. 2024-02-21T23:08:59.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.581+0000 7fcba7200640 1 Processor -- start 2024-02-21T23:08:59.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.581+0000 7fcba7200640 1 -- start start 2024-02-21T23:08:59.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.582+0000 7fcba7200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcba0072430 0x7fcba00bd090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:59.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.582+0000 7fcba7200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcba00c3280 0x7fcba00bbc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:59.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.582+0000 7fcba7200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcba0155860 0x7fcba0151160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:59.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.582+0000 7fcba7200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcba00c5aa0 con 0x7fcba0072430 2024-02-21T23:08:59.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.582+0000 7fcba7200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcba00c5920 con 0x7fcba00c3280 2024-02-21T23:08:59.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.582+0000 7fcba7200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcba00733e0 con 0x7fcba0155860 2024-02-21T23:08:59.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.582+0000 7fcb9fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcba0072430 0x7fcba00bd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:59.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.583+0000 7fcb9fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcba0072430 0x7fcba00bd090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41266/0 (socket says 172.21.15.28:41266) 2024-02-21T23:08:59.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.583+0000 7fcb9fe00640 1 -- 172.21.15.28:0/1088685197 learned_addr learned my addr 172.21.15.28:0/1088685197 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:08:59.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.583+0000 7fcb9fe00640 1 -- 172.21.15.28:0/1088685197 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcba0155860 msgr2=0x7fcba0151160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:59.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.583+0000 7fcb9f400640 1 --2- 172.21.15.28:0/1088685197 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcba00c3280 0x7fcba00bbc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:59.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.583+0000 7fcba5200640 1 --2- 172.21.15.28:0/1088685197 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcba0155860 0x7fcba0151160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:59.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.583+0000 7fcb9fe00640 1 --2- 172.21.15.28:0/1088685197 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcba0155860 0x7fcba0151160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:59.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.583+0000 7fcb9fe00640 1 -- 172.21.15.28:0/1088685197 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcba00c3280 msgr2=0x7fcba00bbc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:59.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.583+0000 7fcb9fe00640 1 --2- 172.21.15.28:0/1088685197 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcba00c3280 0x7fcba00bbc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:59.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.583+0000 7fcb9fe00640 1 -- 172.21.15.28:0/1088685197 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcba00bd950 con 0x7fcba0072430 2024-02-21T23:08:59.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.583+0000 7fcb9f400640 1 --2- 172.21.15.28:0/1088685197 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcba00c3280 0x7fcba00bbc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:08:59.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.584+0000 7fcb9fe00640 1 --2- 172.21.15.28:0/1088685197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcba0072430 0x7fcba00bd090 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7fcb9000c430 tx=0x7fcb9000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:59.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.584+0000 7fcb97e00640 1 -- 172.21.15.28:0/1088685197 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb90011790 con 0x7fcba0072430 2024-02-21T23:08:59.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.584+0000 7fcb97e00640 1 -- 172.21.15.28:0/1088685197 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb90017070 con 0x7fcba0072430 2024-02-21T23:08:59.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.584+0000 7fcb97e00640 1 -- 172.21.15.28:0/1088685197 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb90011b70 con 0x7fcba0072430 2024-02-21T23:08:59.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.584+0000 7fcba7200640 1 -- 172.21.15.28:0/1088685197 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcba01526a0 con 0x7fcba0072430 2024-02-21T23:08:59.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.584+0000 7fcba7200640 1 -- 172.21.15.28:0/1088685197 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcba00bc1d0 con 0x7fcba0072430 2024-02-21T23:08:59.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.585+0000 7fcba7200640 1 -- 172.21.15.28:0/1088685197 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcba0067f30 con 0x7fcba0072430 2024-02-21T23:08:59.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.586+0000 7fcb97e00640 1 -- 172.21.15.28:0/1088685197 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcb90011d10 con 0x7fcba0072430 2024-02-21T23:08:59.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.587+0000 7fcb97e00640 1 --2- 172.21.15.28:0/1088685197 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcb7c06b4d0 0x7fcb7c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:08:59.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.587+0000 7fcb97e00640 1 -- 172.21.15.28:0/1088685197 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fcb90089e20 con 0x7fcba0072430 2024-02-21T23:08:59.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.588+0000 7fcb9f400640 1 --2- 172.21.15.28:0/1088685197 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcb7c06b4d0 0x7fcb7c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:08:59.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.589+0000 7fcb9f400640 1 --2- 172.21.15.28:0/1088685197 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcb7c06b4d0 0x7fcb7c06d990 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fcb88036000 tx=0x7fcb88008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:08:59.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.593+0000 7fcb97e00640 1 -- 172.21.15.28:0/1088685197 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fcb90058ef0 con 0x7fcba0072430 2024-02-21T23:08:59.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.867+0000 7fcba7200640 1 -- 172.21.15.28:0/1088685197 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fcba0152bc0 con 0x7fcba0072430 2024-02-21T23:08:59.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.868+0000 7fcb97e00640 1 -- 172.21.15.28:0/1088685197 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v16) v1 ==== 80+0+4724 (secure 0 0 0) 0x7fcb900587f0 con 0x7fcba0072430 2024-02-21T23:08:59.858 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:08:59.858 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-21T23:08:59.858 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_dump": [ 2024-02-21T23:08:59.858 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:08:59.858 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.a", 2024-02-21T23:08:59.859 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDSgNZl4FY1KRAAJSK5Ibb0ibKiKXYpTLgtmQ==", 2024-02-21T23:08:59.859 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:08:59.859 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:08:59.859 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-02-21T23:08:59.859 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-02-21T23:08:59.859 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:08:59.859 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:08:59.859 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:08:59.860 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:08:59.860 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.b", 2024-02-21T23:08:59.860 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDTgNZlwRcAERAAgsYmmkuobKelBEJVCRpXUg==", 2024-02-21T23:08:59.860 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:08:59.860 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:08:59.860 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-02-21T23:08:59.860 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-02-21T23:08:59.860 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:08:59.861 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:08:59.861 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:08:59.861 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:08:59.861 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.c", 2024-02-21T23:08:59.861 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDSgNZloFpQOxAAK2WBArXRVNTgqH9WWDEPDA==", 2024-02-21T23:08:59.861 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:08:59.861 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:08:59.861 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-02-21T23:08:59.862 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-02-21T23:08:59.862 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:08:59.862 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:08:59.862 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:08:59.862 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:08:59.862 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.d", 2024-02-21T23:08:59.862 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDTgNZlblOmIRAAxe2HfSKGSkkzJDN4aVdzgQ==", 2024-02-21T23:08:59.862 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:08:59.863 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:08:59.863 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-02-21T23:08:59.863 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-02-21T23:08:59.863 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:08:59.863 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:08:59.863 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:08:59.863 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:08:59.863 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.0", 2024-02-21T23:08:59.864 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDVgNZlxNJbLxAAXot7Q3chrLEBsFNO8kT6QQ==", 2024-02-21T23:08:59.864 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:08:59.864 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:08:59.864 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:08:59.864 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:08:59.864 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:08:59.864 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:08:59.865 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:08:59.865 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.1", 2024-02-21T23:08:59.865 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDYgNZlZZfWORAAr4XkOxLYLmumVbu362N2HQ==", 2024-02-21T23:08:59.865 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:08:59.865 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:08:59.865 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:08:59.865 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:08:59.865 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:08:59.866 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:08:59.866 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:08:59.866 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.2", 2024-02-21T23:08:59.866 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDcgNZl9mcDGBAAIZGOF8IUMenwo386x95mwg==", 2024-02-21T23:08:59.866 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:08:59.866 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:08:59.866 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:08:59.866 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:08:59.867 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:08:59.867 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:08:59.867 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:08:59.867 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.3", 2024-02-21T23:08:59.867 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDfgNZl8ALSMBAAag9cmGH0/4xALWSUmrZowg==", 2024-02-21T23:08:59.867 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:08:59.867 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:08:59.867 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:08:59.868 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:08:59.868 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:08:59.868 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:08:59.868 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:08:59.868 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.4", 2024-02-21T23:08:59.868 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDngNZlBgeKABAAeW99RVfCBvB1lp7w7mQBlQ==", 2024-02-21T23:08:59.868 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:08:59.869 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:08:59.869 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:08:59.869 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:08:59.869 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:08:59.869 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:08:59.869 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:08:59.869 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.5", 2024-02-21T23:08:59.869 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDqgNZljQG3ChAAY/IZEIcvcGNvlFGIWib3gg==", 2024-02-21T23:08:59.870 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:08:59.870 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:08:59.870 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:08:59.870 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:08:59.870 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:08:59.870 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:08:59.870 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:08:59.870 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.6", 2024-02-21T23:08:59.871 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDtgNZlKnakIxAAkW5xlkshzvEHJdmpgPxR+g==", 2024-02-21T23:08:59.871 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:08:59.871 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:08:59.871 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:08:59.871 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:08:59.871 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:08:59.871 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:08:59.871 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:08:59.872 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.7", 2024-02-21T23:08:59.872 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDxgNZlTWxYARAASX1/gmqO+8mQPP8uA4xjiQ==", 2024-02-21T23:08:59.872 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:08:59.872 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:08:59.872 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:08:59.872 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:08:59.872 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:08:59.873 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:08:59.873 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:08:59.873 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.0", 2024-02-21T23:08:59.873 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDTgNZlILvYLxAAoREWxKKT7YcXnhkgk8YvoQ==", 2024-02-21T23:08:59.873 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:08:59.873 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:08:59.873 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow r", 2024-02-21T23:08:59.873 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-02-21T23:08:59.873 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:08:59.874 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:08:59.874 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:08:59.874 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.1", 2024-02-21T23:08:59.874 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDTgNZlm7ImOxAA8iRcXlzFPZPaXQfsKhuO/g==", 2024-02-21T23:08:59.874 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:08:59.874 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:08:59.874 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-02-21T23:08:59.875 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-02-21T23:08:59.875 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-02-21T23:08:59.875 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:08:59.875 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:08:59.875 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:08:59.875 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.admin", 2024-02-21T23:08:59.875 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDRgNZlBYd1GhAAbK5qBCIkf0Ej5nEyPnTxIQ==", 2024-02-21T23:08:59.875 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:08:59.875 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-02-21T23:08:59.876 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-02-21T23:08:59.876 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-02-21T23:08:59.876 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:08:59.876 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:08:59.876 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:08:59.876 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:08:59.876 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.x", 2024-02-21T23:08:59.876 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDSgNZlJYKeHBAAYPskmLRr7Jw44TdKzM/JMg==", 2024-02-21T23:08:59.877 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:08:59.877 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-02-21T23:08:59.877 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-02-21T23:08:59.877 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:08:59.877 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:08:59.877 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:08:59.877 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:08:59.877 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.y", 2024-02-21T23:08:59.878 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDSgNZlSRkOEBAAOhRj9PsZDpjoV6WT4drrZA==", 2024-02-21T23:08:59.878 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:08:59.878 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-02-21T23:08:59.878 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-02-21T23:08:59.878 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:08:59.878 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:08:59.878 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:08:59.878 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-21T23:08:59.879 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-21T23:08:59.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.871+0000 7fcba7200640 1 -- 172.21.15.28:0/1088685197 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcb7c06b4d0 msgr2=0x7fcb7c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:59.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.871+0000 7fcba7200640 1 --2- 172.21.15.28:0/1088685197 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcb7c06b4d0 0x7fcb7c06d990 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fcb88036000 tx=0x7fcb88008030 comp rx=0 tx=0).stop 2024-02-21T23:08:59.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.871+0000 7fcba7200640 1 -- 172.21.15.28:0/1088685197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcba0072430 msgr2=0x7fcba00bd090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:08:59.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.871+0000 7fcba7200640 1 --2- 172.21.15.28:0/1088685197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcba0072430 0x7fcba00bd090 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7fcb9000c430 tx=0x7fcb9000c8f0 comp rx=0 tx=0).stop 2024-02-21T23:08:59.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.871+0000 7fcba7200640 1 -- 172.21.15.28:0/1088685197 shutdown_connections 2024-02-21T23:08:59.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.871+0000 7fcba7200640 1 --2- 172.21.15.28:0/1088685197 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcb7c06b4d0 0x7fcb7c06d990 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:59.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.871+0000 7fcba7200640 1 --2- 172.21.15.28:0/1088685197 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcba0155860 0x7fcba0151160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:59.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.871+0000 7fcba7200640 1 --2- 172.21.15.28:0/1088685197 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcba00c3280 0x7fcba00bbc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:59.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.871+0000 7fcba7200640 1 --2- 172.21.15.28:0/1088685197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcba0072430 0x7fcba00bd090 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:08:59.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.871+0000 7fcba7200640 1 -- 172.21.15.28:0/1088685197 >> 172.21.15.28:0/1088685197 conn(0x7fcba00c67f0 msgr2=0x7fcba01545d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:08:59.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.871+0000 7fcba7200640 1 -- 172.21.15.28:0/1088685197 shutdown_connections 2024-02-21T23:08:59.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:08:59.871+0000 7fcba7200640 1 -- 172.21.15.28:0/1088685197 wait complete. 2024-02-21T23:08:59.884 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:09:00.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.118+0000 7fd46b200640 1 Processor -- start 2024-02-21T23:09:00.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.118+0000 7fd46b200640 1 -- start start 2024-02-21T23:09:00.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.120+0000 7fd46b200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd46c14f0b0 0x7fd46c155180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:00.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.120+0000 7fd46b200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fd46c1521c0 0x7fd46c152610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:00.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.120+0000 7fd46b200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd46c155760 con 0x7fd46c0a6160 2024-02-21T23:09:00.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.120+0000 7fd46b200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd46c152b50 con 0x7fd46c14f0b0 2024-02-21T23:09:00.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.120+0000 7fd46b200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd46c151d30 con 0x7fd46c1521c0 2024-02-21T23:09:00.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.120+0000 7fd469e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd46c14f0b0 0x7fd46c155180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:00.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.120+0000 7fd461400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd46c0a6160 0x7fd46c0c2010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:49970/0 (socket says 172.21.15.28:49970) 2024-02-21T23:09:00.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.120+0000 7fd461400640 1 -- 172.21.15.28:0/3283202924 learned_addr learned my addr 172.21.15.28:0/3283202924 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:09:00.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.121+0000 7fd46a800640 1 --2- 172.21.15.28:0/3283202924 >> v2:172.21.15.81:3300/0 conn(0x7fd46c1521c0 0x7fd46c152610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:00.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.121+0000 7fd469e00640 1 -- 172.21.15.28:0/3283202924 >> v2:172.21.15.81:3300/0 conn(0x7fd46c1521c0 msgr2=0x7fd46c152610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:00.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.121+0000 7fd469e00640 1 --2- 172.21.15.28:0/3283202924 >> v2:172.21.15.81:3300/0 conn(0x7fd46c1521c0 0x7fd46c152610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:00.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.121+0000 7fd469e00640 1 -- 172.21.15.28:0/3283202924 >> v1:172.21.15.28:6789/0 conn(0x7fd46c0a6160 legacy=0x7fd46c0c2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:00.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.121+0000 7fd469e00640 1 -- 172.21.15.28:0/3283202924 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd46c155a40 con 0x7fd46c14f0b0 2024-02-21T23:09:00.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.121+0000 7fd469400640 1 -- 172.21.15.28:0/3283202924 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3301743732 0 0) 0x7fd46c155760 con 0x7fd46c0a6160 2024-02-21T23:09:00.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.122+0000 7fd469e00640 1 --2- 172.21.15.28:0/3283202924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd46c14f0b0 0x7fd46c155180 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7fd46c151eb0 tx=0x7fd45c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b440478e7f3560c6 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:00.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.123+0000 7fd469400640 1 -- 172.21.15.28:0/3283202924 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd45c035020 con 0x7fd46c14f0b0 2024-02-21T23:09:00.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.123+0000 7fd469400640 1 -- 172.21.15.28:0/3283202924 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd45c031830 con 0x7fd46c14f0b0 2024-02-21T23:09:00.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.123+0000 7fd469400640 1 -- 172.21.15.28:0/3283202924 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd45c031c00 con 0x7fd46c14f0b0 2024-02-21T23:09:00.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.123+0000 7fd46b200640 1 -- 172.21.15.28:0/3283202924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd46c14f0b0 msgr2=0x7fd46c155180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:00.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.123+0000 7fd46b200640 1 --2- 172.21.15.28:0/3283202924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd46c14f0b0 0x7fd46c155180 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7fd46c151eb0 tx=0x7fd45c02b8d0 comp rx=0 tx=0).stop 2024-02-21T23:09:00.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.124+0000 7fd46b200640 1 -- 172.21.15.28:0/3283202924 shutdown_connections 2024-02-21T23:09:00.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.124+0000 7fd46b200640 1 --2- 172.21.15.28:0/3283202924 >> v2:172.21.15.81:3300/0 conn(0x7fd46c1521c0 0x7fd46c152610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:00.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.124+0000 7fd46b200640 1 --2- 172.21.15.28:0/3283202924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd46c14f0b0 0x7fd46c155180 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:00.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.124+0000 7fd46b200640 1 -- 172.21.15.28:0/3283202924 >> 172.21.15.28:0/3283202924 conn(0x7fd46c06b630 msgr2=0x7fd46c0c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:00.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.124+0000 7fd46b200640 1 -- 172.21.15.28:0/3283202924 shutdown_connections 2024-02-21T23:09:00.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.124+0000 7fd46b200640 1 -- 172.21.15.28:0/3283202924 wait complete. 2024-02-21T23:09:00.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.125+0000 7fd46b200640 1 Processor -- start 2024-02-21T23:09:00.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.126+0000 7fd46b200640 1 -- start start 2024-02-21T23:09:00.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.127+0000 7fd46b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd46c0a6160 0x7fd46c153940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:00.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.127+0000 7fd46b200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd46c14f0b0 0x7fd46c153e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:00.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.127+0000 7fd46b200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd46c1521c0 0x7fd46c148280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:00.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.127+0000 7fd46b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd46c0a68d0 con 0x7fd46c0a6160 2024-02-21T23:09:00.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.127+0000 7fd46b200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd46c156000 con 0x7fd46c14f0b0 2024-02-21T23:09:00.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.127+0000 7fd46b200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd46c0c2720 con 0x7fd46c1521c0 2024-02-21T23:09:00.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.127+0000 7fd469e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd46c0a6160 0x7fd46c153940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:00.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.127+0000 7fd469e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd46c0a6160 0x7fd46c153940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41276/0 (socket says 172.21.15.28:41276) 2024-02-21T23:09:00.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.127+0000 7fd469e00640 1 -- 172.21.15.28:0/425789966 learned_addr learned my addr 172.21.15.28:0/425789966 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:00.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.127+0000 7fd469e00640 1 -- 172.21.15.28:0/425789966 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd46c1521c0 msgr2=0x7fd46c148280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:00.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.127+0000 7fd469e00640 1 --2- 172.21.15.28:0/425789966 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd46c1521c0 0x7fd46c148280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:00.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.127+0000 7fd461400640 1 --2- 172.21.15.28:0/425789966 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd46c14f0b0 0x7fd46c153e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:00.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.127+0000 7fd469e00640 1 -- 172.21.15.28:0/425789966 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd46c14f0b0 msgr2=0x7fd46c153e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:00.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.127+0000 7fd469e00640 1 --2- 172.21.15.28:0/425789966 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd46c14f0b0 0x7fd46c153e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:00.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.127+0000 7fd469e00640 1 -- 172.21.15.28:0/425789966 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd46c0c54b0 con 0x7fd46c0a6160 2024-02-21T23:09:00.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.127+0000 7fd461400640 1 --2- 172.21.15.28:0/425789966 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd46c14f0b0 0x7fd46c153e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:09:00.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.128+0000 7fd469e00640 1 --2- 172.21.15.28:0/425789966 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd46c0a6160 0x7fd46c153940 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7fd45c02bee0 tx=0x7fd45c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:00.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.128+0000 7fd462a00640 1 -- 172.21.15.28:0/425789966 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd45c041070 con 0x7fd46c0a6160 2024-02-21T23:09:00.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.128+0000 7fd462a00640 1 -- 172.21.15.28:0/425789966 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd45c03f070 con 0x7fd46c0a6160 2024-02-21T23:09:00.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.128+0000 7fd462a00640 1 -- 172.21.15.28:0/425789966 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd45c035040 con 0x7fd46c0a6160 2024-02-21T23:09:00.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.128+0000 7fd46b200640 1 -- 172.21.15.28:0/425789966 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd46c0c5790 con 0x7fd46c0a6160 2024-02-21T23:09:00.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.128+0000 7fd46b200640 1 -- 172.21.15.28:0/425789966 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd46c0c4280 con 0x7fd46c0a6160 2024-02-21T23:09:00.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.130+0000 7fd46b200640 1 -- 172.21.15.28:0/425789966 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd46c04f290 con 0x7fd46c0a6160 2024-02-21T23:09:00.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.131+0000 7fd462a00640 1 -- 172.21.15.28:0/425789966 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd45c038020 con 0x7fd46c0a6160 2024-02-21T23:09:00.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.132+0000 7fd462a00640 1 --2- 172.21.15.28:0/425789966 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd44806b4d0 0x7fd44806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:00.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.132+0000 7fd462a00640 1 -- 172.21.15.28:0/425789966 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fd45c0ab650 con 0x7fd46c0a6160 2024-02-21T23:09:00.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.133+0000 7fd461400640 1 --2- 172.21.15.28:0/425789966 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd44806b4d0 0x7fd44806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:00.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.134+0000 7fd461400640 1 --2- 172.21.15.28:0/425789966 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd44806b4d0 0x7fd44806d990 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fd46c0c3250 tx=0x7fd454009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:00.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.137+0000 7fd462a00640 1 -- 172.21.15.28:0/425789966 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd45c07a720 con 0x7fd46c0a6160 2024-02-21T23:09:00.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.339+0000 7fd46b200640 1 -- 172.21.15.28:0/425789966 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd46c0c0340 con 0x7fd46c0a6160 2024-02-21T23:09:00.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.340+0000 7fd462a00640 1 -- 172.21.15.28:0/425789966 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v132) v1 ==== 74+0+13114 (secure 0 0 0) 0x7fd45c07a020 con 0x7fd46c0a6160 2024-02-21T23:09:00.330 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:09:00.330 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":132,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:08:59.327236+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:09:00.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.344+0000 7fd46b200640 1 -- 172.21.15.28:0/425789966 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd44806b4d0 msgr2=0x7fd44806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:00.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.344+0000 7fd46b200640 1 --2- 172.21.15.28:0/425789966 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd44806b4d0 0x7fd44806d990 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fd46c0c3250 tx=0x7fd454009040 comp rx=0 tx=0).stop 2024-02-21T23:09:00.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.344+0000 7fd46b200640 1 -- 172.21.15.28:0/425789966 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd46c0a6160 msgr2=0x7fd46c153940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:00.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.344+0000 7fd46b200640 1 --2- 172.21.15.28:0/425789966 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd46c0a6160 0x7fd46c153940 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7fd45c02bee0 tx=0x7fd45c0046c0 comp rx=0 tx=0).stop 2024-02-21T23:09:00.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.344+0000 7fd46b200640 1 -- 172.21.15.28:0/425789966 shutdown_connections 2024-02-21T23:09:00.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.344+0000 7fd46b200640 1 --2- 172.21.15.28:0/425789966 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd44806b4d0 0x7fd44806d990 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:00.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.344+0000 7fd46b200640 1 --2- 172.21.15.28:0/425789966 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd46c1521c0 0x7fd46c148280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:00.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.344+0000 7fd46b200640 1 --2- 172.21.15.28:0/425789966 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd46c14f0b0 0x7fd46c153e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:00.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.344+0000 7fd46b200640 1 --2- 172.21.15.28:0/425789966 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd46c0a6160 0x7fd46c153940 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:00.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.344+0000 7fd46b200640 1 -- 172.21.15.28:0/425789966 >> 172.21.15.28:0/425789966 conn(0x7fd46c06b630 msgr2=0x7fd46c0c3560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:00.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.345+0000 7fd46b200640 1 -- 172.21.15.28:0/425789966 shutdown_connections 2024-02-21T23:09:00.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.345+0000 7fd46b200640 1 -- 172.21.15.28:0/425789966 wait complete. 2024-02-21T23:09:00.351 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-21T23:02:30.728329+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-21T23:09:00.352 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-21T23:09:00.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.577+0000 7f47a9a00640 1 Processor -- start 2024-02-21T23:09:00.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.577+0000 7f47a9a00640 1 -- start start 2024-02-21T23:09:00.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.578+0000 7f47a9a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f47a40c4580 0x7f47a40c12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:00.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.578+0000 7f47a9a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f47a40c2c10 0x7f47a406ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:00.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.579+0000 7f47a9a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47a406e220 con 0x7f47a4150240 2024-02-21T23:09:00.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.579+0000 7f47a9a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47a4150620 con 0x7f47a40c2c10 2024-02-21T23:09:00.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.579+0000 7f47a9a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47a4153a90 con 0x7f47a40c4580 2024-02-21T23:09:00.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.579+0000 7f47a9000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f47a40c2c10 0x7f47a406ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:00.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.579+0000 7f47a9000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f47a40c2c10 0x7f47a406ae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41280/0 (socket says 172.21.15.28:41280) 2024-02-21T23:09:00.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.579+0000 7f47a9000640 1 -- 172.21.15.28:0/827423149 learned_addr learned my addr 172.21.15.28:0/827423149 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:00.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.579+0000 7f47a3400640 1 --2- 172.21.15.28:0/827423149 >> v2:172.21.15.81:3300/0 conn(0x7f47a40c4580 0x7f47a40c12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:00.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.579+0000 7f47a9000640 1 -- 172.21.15.28:0/827423149 >> v2:172.21.15.81:3300/0 conn(0x7f47a40c4580 msgr2=0x7f47a40c12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:00.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.579+0000 7f47a9000640 1 --2- 172.21.15.28:0/827423149 >> v2:172.21.15.81:3300/0 conn(0x7f47a40c4580 0x7f47a40c12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:00.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.579+0000 7f47a9000640 1 -- 172.21.15.28:0/827423149 >> v1:172.21.15.81:6789/0 conn(0x7f47a4150240 legacy=0x7f47a406db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:00.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.579+0000 7f47a9000640 1 -- 172.21.15.28:0/827423149 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47a406f780 con 0x7f47a40c2c10 2024-02-21T23:09:00.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.581+0000 7f47a9000640 1 --2- 172.21.15.28:0/827423149 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47a40c2c10 0x7f47a406ae70 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f47a406f0b0 tx=0x7f479c02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3151f27530c2aab5 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:00.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.581+0000 7f47a2a00640 1 -- 172.21.15.28:0/827423149 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f479c02c9f0 con 0x7f47a40c2c10 2024-02-21T23:09:00.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.581+0000 7f47a2a00640 1 -- 172.21.15.28:0/827423149 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f479c02cb90 con 0x7f47a40c2c10 2024-02-21T23:09:00.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.581+0000 7f47a2a00640 1 -- 172.21.15.28:0/827423149 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f479c0343f0 con 0x7f47a40c2c10 2024-02-21T23:09:00.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.582+0000 7f47a9a00640 1 -- 172.21.15.28:0/827423149 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47a40c2c10 msgr2=0x7f47a406ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:00.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.582+0000 7f47a9a00640 1 --2- 172.21.15.28:0/827423149 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47a40c2c10 0x7f47a406ae70 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f47a406f0b0 tx=0x7f479c02be50 comp rx=0 tx=0).stop 2024-02-21T23:09:00.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.582+0000 7f47a9a00640 1 -- 172.21.15.28:0/827423149 shutdown_connections 2024-02-21T23:09:00.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.582+0000 7f47a9a00640 1 --2- 172.21.15.28:0/827423149 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47a40c2c10 0x7f47a406ae70 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:00.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.582+0000 7f47a9a00640 1 --2- 172.21.15.28:0/827423149 >> v2:172.21.15.81:3300/0 conn(0x7f47a40c4580 0x7f47a40c12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:00.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.582+0000 7f47a9a00640 1 -- 172.21.15.28:0/827423149 >> 172.21.15.28:0/827423149 conn(0x7f47a40a8530 msgr2=0x7f47a40a8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:00.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.582+0000 7f47a9a00640 1 -- 172.21.15.28:0/827423149 shutdown_connections 2024-02-21T23:09:00.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.582+0000 7f47a9a00640 1 -- 172.21.15.28:0/827423149 wait complete. 2024-02-21T23:09:00.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.584+0000 7f47a9a00640 1 Processor -- start 2024-02-21T23:09:00.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.584+0000 7f47a9a00640 1 -- start start 2024-02-21T23:09:00.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.585+0000 7f47a9a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f47a40c2c10 0x7f47a4156b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:00.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.585+0000 7f47a9a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47a40c4580 0x7f47a41e8610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:00.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.585+0000 7f47a9a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47a4150240 0x7f47a4147660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:00.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.585+0000 7f47a9a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a40c4a10 con 0x7f47a40c4580 2024-02-21T23:09:00.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.585+0000 7f47a9a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a4153720 con 0x7f47a4150240 2024-02-21T23:09:00.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.585+0000 7f47a9a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47a40c4b90 con 0x7f47a40c2c10 2024-02-21T23:09:00.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.585+0000 7f47a3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47a40c4580 0x7f47a41e8610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:00.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.585+0000 7f47a3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47a40c4580 0x7f47a41e8610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41296/0 (socket says 172.21.15.28:41296) 2024-02-21T23:09:00.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.585+0000 7f47a3400640 1 -- 172.21.15.28:0/3774619455 learned_addr learned my addr 172.21.15.28:0/3774619455 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:00.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.585+0000 7f47a9000640 1 --2- 172.21.15.28:0/3774619455 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47a4150240 0x7f47a4147660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:00.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.585+0000 7f47a3e00640 1 --2- 172.21.15.28:0/3774619455 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f47a40c2c10 0x7f47a4156b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:00.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.586+0000 7f47a3400640 1 -- 172.21.15.28:0/3774619455 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f47a40c2c10 msgr2=0x7f47a4156b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:00.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.586+0000 7f47a3400640 1 --2- 172.21.15.28:0/3774619455 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f47a40c2c10 0x7f47a4156b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:00.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.586+0000 7f47a3400640 1 -- 172.21.15.28:0/3774619455 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47a4150240 msgr2=0x7f47a4147660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:00.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.586+0000 7f47a3400640 1 --2- 172.21.15.28:0/3774619455 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47a4150240 0x7f47a4147660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:00.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.586+0000 7f47a3400640 1 -- 172.21.15.28:0/3774619455 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47a4153e60 con 0x7f47a40c4580 2024-02-21T23:09:00.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.586+0000 7f47a3400640 1 --2- 172.21.15.28:0/3774619455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47a40c4580 0x7f47a41e8610 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f479000d800 tx=0x7f479000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:00.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.586+0000 7f479be00640 1 -- 172.21.15.28:0/3774619455 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f479000b6c0 con 0x7f47a40c4580 2024-02-21T23:09:00.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.586+0000 7f479be00640 1 -- 172.21.15.28:0/3774619455 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4790019070 con 0x7f47a40c4580 2024-02-21T23:09:00.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.586+0000 7f479be00640 1 -- 172.21.15.28:0/3774619455 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f479000ba80 con 0x7f47a40c4580 2024-02-21T23:09:00.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.586+0000 7f47a9a00640 1 -- 172.21.15.28:0/3774619455 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47a41ec590 con 0x7f47a40c4580 2024-02-21T23:09:00.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.587+0000 7f47a9a00640 1 -- 172.21.15.28:0/3774619455 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47a4154120 con 0x7f47a40c4580 2024-02-21T23:09:00.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.588+0000 7f47a9a00640 1 -- 172.21.15.28:0/3774619455 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47a406c440 con 0x7f47a40c4580 2024-02-21T23:09:00.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.589+0000 7f479be00640 1 -- 172.21.15.28:0/3774619455 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f479000bc20 con 0x7f47a40c4580 2024-02-21T23:09:00.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.589+0000 7f479be00640 1 --2- 172.21.15.28:0/3774619455 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f478406b4d0 0x7f478406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:00.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.590+0000 7f479be00640 1 -- 172.21.15.28:0/3774619455 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f479008a460 con 0x7f47a40c4580 2024-02-21T23:09:00.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.590+0000 7f47a3e00640 1 --2- 172.21.15.28:0/3774619455 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f478406b4d0 0x7f478406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:00.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.591+0000 7f47a3e00640 1 --2- 172.21.15.28:0/3774619455 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f478406b4d0 0x7f478406d990 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f47a414e280 tx=0x7f4794009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:00.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.594+0000 7f479be00640 1 -- 172.21.15.28:0/3774619455 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f47a406c440 con 0x7f47a40c4580 2024-02-21T23:09:00.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.791+0000 7f47a9a00640 1 -- 172.21.15.28:0/3774619455 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f47a404f290 con 0x7f47a40c4580 2024-02-21T23:09:00.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.792+0000 7f479be00640 1 -- 172.21.15.28:0/3774619455 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v132) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4790059530 con 0x7f47a40c4580 2024-02-21T23:09:00.781 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-02-21T23:09:00.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.794+0000 7f47a9a00640 1 -- 172.21.15.28:0/3774619455 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f478406b4d0 msgr2=0x7f478406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:00.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.794+0000 7f47a9a00640 1 --2- 172.21.15.28:0/3774619455 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f478406b4d0 0x7f478406d990 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f47a414e280 tx=0x7f4794009040 comp rx=0 tx=0).stop 2024-02-21T23:09:00.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.794+0000 7f47a9a00640 1 -- 172.21.15.28:0/3774619455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47a40c4580 msgr2=0x7f47a41e8610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:00.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.794+0000 7f47a9a00640 1 --2- 172.21.15.28:0/3774619455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47a40c4580 0x7f47a41e8610 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f479000d800 tx=0x7f479000dcc0 comp rx=0 tx=0).stop 2024-02-21T23:09:00.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.794+0000 7f47a9a00640 1 -- 172.21.15.28:0/3774619455 shutdown_connections 2024-02-21T23:09:00.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.794+0000 7f47a9a00640 1 --2- 172.21.15.28:0/3774619455 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f478406b4d0 0x7f478406d990 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:00.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.794+0000 7f47a9a00640 1 --2- 172.21.15.28:0/3774619455 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47a4150240 0x7f47a4147660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:00.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.794+0000 7f47a9a00640 1 --2- 172.21.15.28:0/3774619455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47a40c4580 0x7f47a41e8610 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:00.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.794+0000 7f47a9a00640 1 --2- 172.21.15.28:0/3774619455 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f47a40c2c10 0x7f47a4156b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:00.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.794+0000 7f47a9a00640 1 -- 172.21.15.28:0/3774619455 >> 172.21.15.28:0/3774619455 conn(0x7f47a40a8530 msgr2=0x7f47a414da60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:00.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.795+0000 7f47a9a00640 1 -- 172.21.15.28:0/3774619455 shutdown_connections 2024-02-21T23:09:00.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:00.795+0000 7f47a9a00640 1 -- 172.21.15.28:0/3774619455 wait complete. 2024-02-21T23:09:00.804 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-02-21T23:09:01.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.037+0000 7f8c7a600640 1 Processor -- start 2024-02-21T23:09:01.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.037+0000 7f8c7a600640 1 -- start start 2024-02-21T23:09:01.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.039+0000 7f8c7a600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8c740c3280 0x7f8c740c3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:01.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.039+0000 7f8c7a600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c740c6560 con 0x7f8c74155860 2024-02-21T23:09:01.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.039+0000 7f8c7a600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c740c5cc0 con 0x7f8c74072430 2024-02-21T23:09:01.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.039+0000 7f8c7a600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c740c5fa0 con 0x7f8c740c3280 2024-02-21T23:09:01.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.039+0000 7f8c73e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f8c74155860 0x7f8c7415c5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:49982/0 (socket says 172.21.15.28:49982) 2024-02-21T23:09:01.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.039+0000 7f8c73e00640 1 -- 172.21.15.28:0/2187993487 learned_addr learned my addr 172.21.15.28:0/2187993487 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:09:01.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.040+0000 7f8c73400640 1 --2- 172.21.15.28:0/2187993487 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8c740c3280 0x7f8c740c3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:01.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.040+0000 7f8c72a00640 1 -- 172.21.15.28:0/2187993487 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3379800679 0 0) 0x7f8c740c6560 con 0x7f8c74155860 2024-02-21T23:09:01.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.040+0000 7f8c72a00640 1 -- 172.21.15.28:0/2187993487 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c58003520 con 0x7f8c74155860 2024-02-21T23:09:01.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.040+0000 7f8c72a00640 1 -- 172.21.15.28:0/2187993487 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 773566120 0 0) 0x7f8c58003520 con 0x7f8c74155860 2024-02-21T23:09:01.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.040+0000 7f8c72a00640 1 -- 172.21.15.28:0/2187993487 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8c740c3280 msgr2=0x7f8c740c3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:01.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.040+0000 7f8c72a00640 1 --2- 172.21.15.28:0/2187993487 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8c740c3280 0x7f8c740c3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:01.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.040+0000 7f8c72a00640 1 -- 172.21.15.28:0/2187993487 >> v1:172.21.15.81:6789/0 conn(0x7f8c74072430 legacy=0x7f8c741526e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:01.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.040+0000 7f8c72a00640 1 -- 172.21.15.28:0/2187993487 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c740c6280 con 0x7f8c74155860 2024-02-21T23:09:01.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.040+0000 7f8c72a00640 1 -- 172.21.15.28:0/2187993487 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f8c6c001670 con 0x7f8c74155860 2024-02-21T23:09:01.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.040+0000 7f8c72a00640 1 -- 172.21.15.28:0/2187993487 >> v1:172.21.15.28:6789/0 conn(0x7f8c74155860 legacy=0x7f8c7415c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:01.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.040+0000 7f8c72a00640 1 --2- 172.21.15.28:0/2187993487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c580039e0 0x7f8c58003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:01.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.040+0000 7f8c72a00640 1 --2- 172.21.15.28:0/2187993487 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c58004800 0x7f8c58004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:01.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.041+0000 7f8c72a00640 1 --2- 172.21.15.28:0/2187993487 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8c740c3280 0x7f8c58005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:01.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.041+0000 7f8c72a00640 1 -- 172.21.15.28:0/2187993487 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c740c5fa0 con 0x7f8c580039e0 2024-02-21T23:09:01.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.041+0000 7f8c72a00640 1 -- 172.21.15.28:0/2187993487 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c5800dd00 con 0x7f8c58004800 2024-02-21T23:09:01.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.041+0000 7f8c72a00640 1 -- 172.21.15.28:0/2187993487 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c5800df00 con 0x7f8c740c3280 2024-02-21T23:09:01.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.041+0000 7f8c73400640 1 --2- 172.21.15.28:0/2187993487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c580039e0 0x7f8c58003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:01.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.041+0000 7f8c6aa00640 1 --2- 172.21.15.28:0/2187993487 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c58004800 0x7f8c58004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:01.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.041+0000 7f8c73e00640 1 --2- 172.21.15.28:0/2187993487 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8c740c3280 0x7f8c58005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:01.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.041+0000 7f8c73400640 1 -- 172.21.15.28:0/2187993487 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8c740c3280 msgr2=0x7f8c58005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:01.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.041+0000 7f8c73400640 1 --2- 172.21.15.28:0/2187993487 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8c740c3280 0x7f8c58005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:01.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.041+0000 7f8c73400640 1 -- 172.21.15.28:0/2187993487 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c58004800 msgr2=0x7f8c58004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:01.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.041+0000 7f8c73400640 1 --2- 172.21.15.28:0/2187993487 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c58004800 0x7f8c58004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:01.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.041+0000 7f8c73400640 1 -- 172.21.15.28:0/2187993487 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c58003520 con 0x7f8c580039e0 2024-02-21T23:09:01.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.041+0000 7f8c73e00640 1 --2- 172.21.15.28:0/2187993487 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8c740c3280 0x7f8c58005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:09:01.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.042+0000 7f8c73400640 1 --2- 172.21.15.28:0/2187993487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c580039e0 0x7f8c58003e00 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f8c64009fc0 tx=0x7f8c58005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2eff31912c223a6b server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:01.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.043+0000 7f8c72a00640 1 -- 172.21.15.28:0/2187993487 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c64039070 con 0x7f8c580039e0 2024-02-21T23:09:01.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.043+0000 7f8c72a00640 1 -- 172.21.15.28:0/2187993487 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c6402cbf0 con 0x7f8c580039e0 2024-02-21T23:09:01.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.043+0000 7f8c72a00640 1 -- 172.21.15.28:0/2187993487 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c64009c10 con 0x7f8c580039e0 2024-02-21T23:09:01.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.043+0000 7f8c7a600640 1 -- 172.21.15.28:0/2187993487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c580039e0 msgr2=0x7f8c58003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:01.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.043+0000 7f8c7a600640 1 --2- 172.21.15.28:0/2187993487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c580039e0 0x7f8c58003e00 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f8c64009fc0 tx=0x7f8c58005830 comp rx=0 tx=0).stop 2024-02-21T23:09:01.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.043+0000 7f8c73e00640 1 -- 172.21.15.28:0/2187993487 reap_dead start 2024-02-21T23:09:01.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.043+0000 7f8c7a600640 1 -- 172.21.15.28:0/2187993487 shutdown_connections 2024-02-21T23:09:01.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.043+0000 7f8c7a600640 1 -- 172.21.15.28:0/2187993487 >> 172.21.15.28:0/2187993487 conn(0x7f8c740c67f0 msgr2=0x7f8c740c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:01.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.044+0000 7f8c7a600640 1 -- 172.21.15.28:0/2187993487 shutdown_connections 2024-02-21T23:09:01.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.044+0000 7f8c7a600640 1 -- 172.21.15.28:0/2187993487 wait complete. 2024-02-21T23:09:01.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.045+0000 7f8c7a600640 1 Processor -- start 2024-02-21T23:09:01.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.046+0000 7f8c7a600640 1 -- start start 2024-02-21T23:09:01.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.047+0000 7f8c7a600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c580039e0 0x7f8c740aa030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:01.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.047+0000 7f8c7a600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8c740bfec0 0x7f8c74149a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:01.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.047+0000 7f8c7a600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c74149fa0 0x7f8c741eafd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:01.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.047+0000 7f8c7a600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c740c5aa0 con 0x7f8c74149fa0 2024-02-21T23:09:01.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.047+0000 7f8c7a600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c740c5920 con 0x7f8c580039e0 2024-02-21T23:09:01.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.047+0000 7f8c7a600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c740733e0 con 0x7f8c740bfec0 2024-02-21T23:09:01.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.047+0000 7f8c73e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c74149fa0 0x7f8c741eafd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:01.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.047+0000 7f8c73e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c74149fa0 0x7f8c741eafd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41326/0 (socket says 172.21.15.28:41326) 2024-02-21T23:09:01.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.047+0000 7f8c73e00640 1 -- 172.21.15.28:0/3421891815 learned_addr learned my addr 172.21.15.28:0/3421891815 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:01.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.047+0000 7f8c73400640 1 --2- 172.21.15.28:0/3421891815 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c580039e0 0x7f8c740aa030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:01.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.047+0000 7f8c6aa00640 1 --2- 172.21.15.28:0/3421891815 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8c740bfec0 0x7f8c74149a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:01.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.047+0000 7f8c73e00640 1 -- 172.21.15.28:0/3421891815 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8c740bfec0 msgr2=0x7f8c74149a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:01.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.047+0000 7f8c73e00640 1 --2- 172.21.15.28:0/3421891815 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8c740bfec0 0x7f8c74149a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:01.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.047+0000 7f8c73e00640 1 -- 172.21.15.28:0/3421891815 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c580039e0 msgr2=0x7f8c740aa030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:01.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.047+0000 7f8c73e00640 1 --2- 172.21.15.28:0/3421891815 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c580039e0 0x7f8c740aa030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:01.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.048+0000 7f8c73e00640 1 -- 172.21.15.28:0/3421891815 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c740a9140 con 0x7f8c74149fa0 2024-02-21T23:09:01.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.048+0000 7f8c73e00640 1 --2- 172.21.15.28:0/3421891815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c74149fa0 0x7f8c741eafd0 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f8c7406fae0 tx=0x7f8c6c00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:01.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.048+0000 7f8c6be00640 1 -- 172.21.15.28:0/3421891815 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c6c0072d0 con 0x7f8c74149fa0 2024-02-21T23:09:01.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.048+0000 7f8c6be00640 1 -- 172.21.15.28:0/3421891815 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c6c007470 con 0x7f8c74149fa0 2024-02-21T23:09:01.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.048+0000 7f8c6be00640 1 -- 172.21.15.28:0/3421891815 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c6c004a20 con 0x7f8c74149fa0 2024-02-21T23:09:01.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.048+0000 7f8c7a600640 1 -- 172.21.15.28:0/3421891815 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c740ac510 con 0x7f8c74149fa0 2024-02-21T23:09:01.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.048+0000 7f8c7a600640 1 -- 172.21.15.28:0/3421891815 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c740a9c70 con 0x7f8c74149fa0 2024-02-21T23:09:01.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.049+0000 7f8c7a600640 1 -- 172.21.15.28:0/3421891815 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c74067f30 con 0x7f8c74149fa0 2024-02-21T23:09:01.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.051+0000 7f8c6be00640 1 -- 172.21.15.28:0/3421891815 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8c6c01b020 con 0x7f8c74149fa0 2024-02-21T23:09:01.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.051+0000 7f8c6be00640 1 --2- 172.21.15.28:0/3421891815 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8c5006b4d0 0x7f8c5006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:01.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.052+0000 7f8c6be00640 1 -- 172.21.15.28:0/3421891815 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f8c6c089c30 con 0x7f8c74149fa0 2024-02-21T23:09:01.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.052+0000 7f8c73400640 1 --2- 172.21.15.28:0/3421891815 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8c5006b4d0 0x7f8c5006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:01.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.053+0000 7f8c73400640 1 --2- 172.21.15.28:0/3421891815 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8c5006b4d0 0x7f8c5006d990 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f8c64036000 tx=0x7f8c6402e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:01.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.056+0000 7f8c6be00640 1 -- 172.21.15.28:0/3421891815 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f8c6c058610 con 0x7f8c74149fa0 2024-02-21T23:09:01.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.256+0000 7f8c7a600640 1 -- 172.21.15.28:0/3421891815 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f8c7414d6a0 con 0x7f8c74149fa0 2024-02-21T23:09:01.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.257+0000 7f8c6be00640 1 -- 172.21.15.28:0/3421891815 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v132) v1 ==== 84+0+65 (secure 0 0 0) 0x7f8c6c05c6b0 con 0x7f8c74149fa0 2024-02-21T23:09:01.246 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:09:01.246 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-02-21T23:09:01.246 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:09:01.246 INFO:teuthology.orchestra.run.smithi028.stdout: "poolnum": 1, 2024-02-21T23:09:01.246 INFO:teuthology.orchestra.run.smithi028.stdout: "poolname": ".mgr" 2024-02-21T23:09:01.246 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:09:01.246 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-02-21T23:09:01.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.259+0000 7f8c7a600640 1 -- 172.21.15.28:0/3421891815 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8c5006b4d0 msgr2=0x7f8c5006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:01.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.259+0000 7f8c7a600640 1 --2- 172.21.15.28:0/3421891815 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8c5006b4d0 0x7f8c5006d990 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f8c64036000 tx=0x7f8c6402e000 comp rx=0 tx=0).stop 2024-02-21T23:09:01.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.259+0000 7f8c7a600640 1 -- 172.21.15.28:0/3421891815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c74149fa0 msgr2=0x7f8c741eafd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:01.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.259+0000 7f8c7a600640 1 --2- 172.21.15.28:0/3421891815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c74149fa0 0x7f8c741eafd0 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f8c7406fae0 tx=0x7f8c6c00dc00 comp rx=0 tx=0).stop 2024-02-21T23:09:01.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.260+0000 7f8c7a600640 1 -- 172.21.15.28:0/3421891815 shutdown_connections 2024-02-21T23:09:01.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.260+0000 7f8c7a600640 1 --2- 172.21.15.28:0/3421891815 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8c5006b4d0 0x7f8c5006d990 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:01.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.260+0000 7f8c7a600640 1 --2- 172.21.15.28:0/3421891815 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8c740bfec0 0x7f8c74149a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:01.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.260+0000 7f8c7a600640 1 --2- 172.21.15.28:0/3421891815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c74149fa0 0x7f8c741eafd0 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:01.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.260+0000 7f8c7a600640 1 --2- 172.21.15.28:0/3421891815 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c580039e0 0x7f8c740aa030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:01.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.260+0000 7f8c7a600640 1 -- 172.21.15.28:0/3421891815 >> 172.21.15.28:0/3421891815 conn(0x7f8c740c67f0 msgr2=0x7f8c740c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:01.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.260+0000 7f8c7a600640 1 -- 172.21.15.28:0/3421891815 shutdown_connections 2024-02-21T23:09:01.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.260+0000 7f8c7a600640 1 -- 172.21.15.28:0/3421891815 wait complete. 2024-02-21T23:09:01.269 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-02-21T23:09:01.269 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-02-21T23:09:01.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.501+0000 7fbd1d600640 1 Processor -- start 2024-02-21T23:09:01.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.501+0000 7fbd1d600640 1 -- start start 2024-02-21T23:09:01.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.503+0000 7fbd1d600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbd180c46f0 0x7fbd18158820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:01.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.503+0000 7fbd1d600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd18152ad0 con 0x7fbd181526f0 2024-02-21T23:09:01.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.503+0000 7fbd1d600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd18155240 con 0x7fbd180c4dc0 2024-02-21T23:09:01.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.503+0000 7fbd1d600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd18155520 con 0x7fbd180c46f0 2024-02-21T23:09:01.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.503+0000 7fbd16a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fbd181526f0 0x7fbd180c1340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:49984/0 (socket says 172.21.15.28:49984) 2024-02-21T23:09:01.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.503+0000 7fbd16a00640 1 -- 172.21.15.28:0/3233686885 learned_addr learned my addr 172.21.15.28:0/3233686885 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:09:01.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.504+0000 7fbd0e000640 1 --2- 172.21.15.28:0/3233686885 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbd180c46f0 0x7fbd18158820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:01.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.504+0000 7fbd16000640 1 -- 172.21.15.28:0/3233686885 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 412911255 0 0) 0x7fbd18152ad0 con 0x7fbd181526f0 2024-02-21T23:09:01.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.504+0000 7fbd16000640 1 -- 172.21.15.28:0/3233686885 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbcfc003520 con 0x7fbd181526f0 2024-02-21T23:09:01.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.504+0000 7fbd16000640 1 -- 172.21.15.28:0/3233686885 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4229062644 0 0) 0x7fbcfc003520 con 0x7fbd181526f0 2024-02-21T23:09:01.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.504+0000 7fbd16000640 1 -- 172.21.15.28:0/3233686885 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbd180c46f0 msgr2=0x7fbd18158820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:01.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.504+0000 7fbd16000640 1 --2- 172.21.15.28:0/3233686885 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbd180c46f0 0x7fbd18158820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:01.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.504+0000 7fbd16000640 1 -- 172.21.15.28:0/3233686885 >> v1:172.21.15.81:6789/0 conn(0x7fbd180c4dc0 legacy=0x7fbd18158d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:01.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.504+0000 7fbd16000640 1 -- 172.21.15.28:0/3233686885 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd18155800 con 0x7fbd181526f0 2024-02-21T23:09:01.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.504+0000 7fbd16000640 1 -- 172.21.15.28:0/3233686885 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fbd080019f0 con 0x7fbd181526f0 2024-02-21T23:09:01.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.504+0000 7fbd16000640 1 -- 172.21.15.28:0/3233686885 >> v1:172.21.15.28:6789/0 conn(0x7fbd181526f0 legacy=0x7fbd180c1340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:01.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.504+0000 7fbd16000640 1 --2- 172.21.15.28:0/3233686885 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbcfc0039e0 0x7fbcfc003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:01.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.504+0000 7fbd16000640 1 --2- 172.21.15.28:0/3233686885 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcfc004800 0x7fbcfc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:01.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.504+0000 7fbd16000640 1 --2- 172.21.15.28:0/3233686885 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbd180c46f0 0x7fbcfc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:01.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.504+0000 7fbd16000640 1 -- 172.21.15.28:0/3233686885 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd18155520 con 0x7fbcfc004800 2024-02-21T23:09:01.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.504+0000 7fbd16000640 1 -- 172.21.15.28:0/3233686885 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcfc00baa0 con 0x7fbcfc0039e0 2024-02-21T23:09:01.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.504+0000 7fbd16000640 1 -- 172.21.15.28:0/3233686885 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcfc00bca0 con 0x7fbd180c46f0 2024-02-21T23:09:01.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.505+0000 7fbd0e000640 1 --2- 172.21.15.28:0/3233686885 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcfc004800 0x7fbcfc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:01.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.505+0000 7fbd0e000640 1 -- 172.21.15.28:0/3233686885 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbd180c46f0 msgr2=0x7fbcfc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:01.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.505+0000 7fbd0e000640 1 --2- 172.21.15.28:0/3233686885 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbd180c46f0 0x7fbcfc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:01.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.505+0000 7fbd0e000640 1 -- 172.21.15.28:0/3233686885 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbcfc0039e0 msgr2=0x7fbcfc003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:01.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.505+0000 7fbd16a00640 1 --2- 172.21.15.28:0/3233686885 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbcfc0039e0 0x7fbcfc003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:01.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.505+0000 7fbd0e000640 1 --2- 172.21.15.28:0/3233686885 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbcfc0039e0 0x7fbcfc003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:01.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.505+0000 7fbd0e000640 1 -- 172.21.15.28:0/3233686885 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcfc003520 con 0x7fbcfc004800 2024-02-21T23:09:01.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.505+0000 7fbd16a00640 1 --2- 172.21.15.28:0/3233686885 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbcfc0039e0 0x7fbcfc003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:09:01.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.506+0000 7fbd0e000640 1 --2- 172.21.15.28:0/3233686885 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcfc004800 0x7fbcfc004c80 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7fbd180c4b30 tx=0x7fbcfc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da7e1e4231f43755 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:01.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.507+0000 7fbd16000640 1 -- 172.21.15.28:0/3233686885 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd00009680 con 0x7fbcfc004800 2024-02-21T23:09:01.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.507+0000 7fbd16000640 1 -- 172.21.15.28:0/3233686885 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd00005990 con 0x7fbcfc004800 2024-02-21T23:09:01.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.507+0000 7fbd16000640 1 -- 172.21.15.28:0/3233686885 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd00005d80 con 0x7fbcfc004800 2024-02-21T23:09:01.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.507+0000 7fbd1d600640 1 -- 172.21.15.28:0/3233686885 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcfc004800 msgr2=0x7fbcfc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:01.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.507+0000 7fbd1d600640 1 --2- 172.21.15.28:0/3233686885 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcfc004800 0x7fbcfc004c80 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7fbd180c4b30 tx=0x7fbcfc005830 comp rx=0 tx=0).stop 2024-02-21T23:09:01.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.507+0000 7fbd17400640 1 -- 172.21.15.28:0/3233686885 reap_dead start 2024-02-21T23:09:01.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.507+0000 7fbd1d600640 1 -- 172.21.15.28:0/3233686885 shutdown_connections 2024-02-21T23:09:01.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.507+0000 7fbd1d600640 1 -- 172.21.15.28:0/3233686885 >> 172.21.15.28:0/3233686885 conn(0x7fbd1806b830 msgr2=0x7fbd180c67f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:01.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.507+0000 7fbd1d600640 1 -- 172.21.15.28:0/3233686885 shutdown_connections 2024-02-21T23:09:01.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.507+0000 7fbd1d600640 1 -- 172.21.15.28:0/3233686885 wait complete. 2024-02-21T23:09:01.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.509+0000 7fbd1d600640 1 Processor -- start 2024-02-21T23:09:01.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.509+0000 7fbd1d600640 1 -- start start 2024-02-21T23:09:01.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.510+0000 7fbd1d600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbcfc004800 0x7fbd180ab130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:01.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.510+0000 7fbd1d600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbd18072ed0 0x7fbd181eae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:01.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.510+0000 7fbd1d600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd180a9620 0x7fbd181eb390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:01.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.510+0000 7fbd1d600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd18159470 con 0x7fbd180a9620 2024-02-21T23:09:01.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.510+0000 7fbd1d600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd18155dc0 con 0x7fbd18072ed0 2024-02-21T23:09:01.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.510+0000 7fbd1d600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd180c3c30 con 0x7fbcfc004800 2024-02-21T23:09:01.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.510+0000 7fbd17400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd180a9620 0x7fbd181eb390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:01.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.510+0000 7fbd17400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd180a9620 0x7fbd181eb390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41344/0 (socket says 172.21.15.28:41344) 2024-02-21T23:09:01.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.510+0000 7fbd17400640 1 -- 172.21.15.28:0/610311689 learned_addr learned my addr 172.21.15.28:0/610311689 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:01.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.511+0000 7fbd16a00640 1 --2- 172.21.15.28:0/610311689 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbcfc004800 0x7fbd180ab130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:01.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.511+0000 7fbd0e000640 1 --2- 172.21.15.28:0/610311689 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbd18072ed0 0x7fbd181eae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:01.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.511+0000 7fbd17400640 1 -- 172.21.15.28:0/610311689 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbcfc004800 msgr2=0x7fbd180ab130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:01.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.511+0000 7fbd17400640 1 --2- 172.21.15.28:0/610311689 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbcfc004800 0x7fbd180ab130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:01.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.511+0000 7fbd17400640 1 -- 172.21.15.28:0/610311689 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbd18072ed0 msgr2=0x7fbd181eae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:01.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.511+0000 7fbd17400640 1 --2- 172.21.15.28:0/610311689 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbd18072ed0 0x7fbd181eae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:01.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.511+0000 7fbd17400640 1 -- 172.21.15.28:0/610311689 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd18158e90 con 0x7fbd180a9620 2024-02-21T23:09:01.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.511+0000 7fbd17400640 1 --2- 172.21.15.28:0/610311689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd180a9620 0x7fbd181eb390 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7fbd180c67c0 tx=0x7fbd1000c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:01.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.512+0000 7fbd0f400640 1 -- 172.21.15.28:0/610311689 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd100075e0 con 0x7fbd180a9620 2024-02-21T23:09:01.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.512+0000 7fbd0f400640 1 -- 172.21.15.28:0/610311689 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd100057c0 con 0x7fbd180a9620 2024-02-21T23:09:01.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.512+0000 7fbd0f400640 1 -- 172.21.15.28:0/610311689 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd10005bb0 con 0x7fbd180a9620 2024-02-21T23:09:01.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.512+0000 7fbd1d600640 1 -- 172.21.15.28:0/610311689 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd181ee3c0 con 0x7fbd180a9620 2024-02-21T23:09:01.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.512+0000 7fbd1d600640 1 -- 172.21.15.28:0/610311689 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd181ee680 con 0x7fbd180a9620 2024-02-21T23:09:01.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.513+0000 7fbd1d600640 1 -- 172.21.15.28:0/610311689 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd18067f30 con 0x7fbd180a9620 2024-02-21T23:09:01.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.514+0000 7fbd0f400640 1 -- 172.21.15.28:0/610311689 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbd10011070 con 0x7fbd180a9620 2024-02-21T23:09:01.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.515+0000 7fbd0f400640 1 --2- 172.21.15.28:0/610311689 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbcf406b500 0x7fbcf406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:01.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.515+0000 7fbd0f400640 1 -- 172.21.15.28:0/610311689 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fbd10089520 con 0x7fbd180a9620 2024-02-21T23:09:01.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.516+0000 7fbd16a00640 1 --2- 172.21.15.28:0/610311689 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbcf406b500 0x7fbcf406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:01.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.517+0000 7fbd16a00640 1 --2- 172.21.15.28:0/610311689 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbcf406b500 0x7fbcf406d9c0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fbd08001370 tx=0x7fbd08009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:01.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.519+0000 7fbd0f400640 1 -- 172.21.15.28:0/610311689 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fbd10019060 con 0x7fbd180a9620 2024-02-21T23:09:01.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:01.723+0000 7fbd1d600640 1 -- 172.21.15.28:0/610311689 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fbd180a6190 con 0x7fbd180a9620 2024-02-21T23:09:02.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.370+0000 7fbd0f400640 1 -- 172.21.15.28:0/610311689 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v133) v1 ==== 138+0+0 (secure 0 0 0) 0x7fbd1000f260 con 0x7fbd180a9620 2024-02-21T23:09:02.360 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' created 2024-02-21T23:09:02.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.374+0000 7fbd1d600640 1 -- 172.21.15.28:0/610311689 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbcf406b500 msgr2=0x7fbcf406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:02.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.374+0000 7fbd1d600640 1 --2- 172.21.15.28:0/610311689 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbcf406b500 0x7fbcf406d9c0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fbd08001370 tx=0x7fbd08009040 comp rx=0 tx=0).stop 2024-02-21T23:09:02.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.374+0000 7fbd1d600640 1 -- 172.21.15.28:0/610311689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd180a9620 msgr2=0x7fbd181eb390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:02.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.374+0000 7fbd1d600640 1 --2- 172.21.15.28:0/610311689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd180a9620 0x7fbd181eb390 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7fbd180c67c0 tx=0x7fbd1000c820 comp rx=0 tx=0).stop 2024-02-21T23:09:02.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.375+0000 7fbd1d600640 1 -- 172.21.15.28:0/610311689 shutdown_connections 2024-02-21T23:09:02.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.375+0000 7fbd1d600640 1 --2- 172.21.15.28:0/610311689 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbcf406b500 0x7fbcf406d9c0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:02.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.375+0000 7fbd1d600640 1 --2- 172.21.15.28:0/610311689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd180a9620 0x7fbd181eb390 unknown :-1 s=CLOSED pgs=1509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:02.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.375+0000 7fbd1d600640 1 --2- 172.21.15.28:0/610311689 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbd18072ed0 0x7fbd181eae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:02.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.375+0000 7fbd1d600640 1 --2- 172.21.15.28:0/610311689 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbcfc004800 0x7fbd180ab130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:02.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.375+0000 7fbd1d600640 1 -- 172.21.15.28:0/610311689 >> 172.21.15.28:0/610311689 conn(0x7fbd1806b830 msgr2=0x7fbd180c04b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:02.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.377+0000 7fbd1d600640 1 -- 172.21.15.28:0/610311689 shutdown_connections 2024-02-21T23:09:02.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.377+0000 7fbd1d600640 1 -- 172.21.15.28:0/610311689 wait complete. 2024-02-21T23:09:02.393 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-02-21T23:09:02.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.620+0000 7f47ea000640 1 Processor -- start 2024-02-21T23:09:02.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.620+0000 7f47ea000640 1 -- start start 2024-02-21T23:09:02.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.622+0000 7f47ea000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f47e4152370 0x7f47e4155940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:02.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.622+0000 7f47ea000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f47e40bde20 0x7f47e40c6f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:02.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.622+0000 7f47ea000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f47e40c74a0 0x7f47e40cab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:02.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.622+0000 7f47ea000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47e41554b0 con 0x7f47e4152370 2024-02-21T23:09:02.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.622+0000 7f47ea000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47e4154ea0 con 0x7f47e40c74a0 2024-02-21T23:09:02.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.622+0000 7f47ea000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47e40c5690 con 0x7f47e40bde20 2024-02-21T23:09:02.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.622+0000 7f47e8c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f47e4152370 0x7f47e4155940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:02.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.622+0000 7f47e8c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f47e4152370 0x7f47e4155940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41356/0 (socket says 172.21.15.28:41356) 2024-02-21T23:09:02.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.622+0000 7f47e8c00640 1 -- 172.21.15.28:0/3124241098 learned_addr learned my addr 172.21.15.28:0/3124241098 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:02.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.623+0000 7f47e8c00640 1 -- 172.21.15.28:0/3124241098 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f47e40bde20 msgr2=0x7f47e40c6f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:02.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.623+0000 7f47e9600640 1 --2- 172.21.15.28:0/3124241098 >> v2:172.21.15.81:3300/0 conn(0x7f47e40c74a0 0x7f47e40cab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:02.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.623+0000 7f47e3e00640 1 --2- 172.21.15.28:0/3124241098 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f47e40bde20 0x7f47e40c6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:02.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.623+0000 7f47e8c00640 1 --2- 172.21.15.28:0/3124241098 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f47e40bde20 0x7f47e40c6f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:02.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.623+0000 7f47e8c00640 1 -- 172.21.15.28:0/3124241098 >> v2:172.21.15.81:3300/0 conn(0x7f47e40c74a0 msgr2=0x7f47e40cab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:02.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.623+0000 7f47e8c00640 1 --2- 172.21.15.28:0/3124241098 >> v2:172.21.15.81:3300/0 conn(0x7f47e40c74a0 0x7f47e40cab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:02.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.623+0000 7f47e8c00640 1 -- 172.21.15.28:0/3124241098 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47e4155140 con 0x7f47e4152370 2024-02-21T23:09:02.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.624+0000 7f47e8c00640 1 --2- 172.21.15.28:0/3124241098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47e4152370 0x7f47e4155940 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7f47e406bb50 tx=0x7f47d802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6aa8cbf13563b5d server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:02.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.624+0000 7f47e3400640 1 -- 172.21.15.28:0/3124241098 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47d8035020 con 0x7f47e4152370 2024-02-21T23:09:02.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.624+0000 7f47e3400640 1 -- 172.21.15.28:0/3124241098 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47d8031870 con 0x7f47e4152370 2024-02-21T23:09:02.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.624+0000 7f47e3400640 1 -- 172.21.15.28:0/3124241098 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47d8031c40 con 0x7f47e4152370 2024-02-21T23:09:02.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.625+0000 7f47ea000640 1 -- 172.21.15.28:0/3124241098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47e4152370 msgr2=0x7f47e4155940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:02.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.625+0000 7f47ea000640 1 --2- 172.21.15.28:0/3124241098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47e4152370 0x7f47e4155940 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7f47e406bb50 tx=0x7f47d802b910 comp rx=0 tx=0).stop 2024-02-21T23:09:02.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.625+0000 7f47ea000640 1 -- 172.21.15.28:0/3124241098 shutdown_connections 2024-02-21T23:09:02.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.625+0000 7f47ea000640 1 --2- 172.21.15.28:0/3124241098 >> v2:172.21.15.81:3300/0 conn(0x7f47e40c74a0 0x7f47e40cab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:02.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.625+0000 7f47ea000640 1 --2- 172.21.15.28:0/3124241098 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f47e40bde20 0x7f47e40c6f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:02.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.625+0000 7f47ea000640 1 --2- 172.21.15.28:0/3124241098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47e4152370 0x7f47e4155940 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:02.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.625+0000 7f47ea000640 1 -- 172.21.15.28:0/3124241098 >> 172.21.15.28:0/3124241098 conn(0x7f47e4072780 msgr2=0x7f47e4072b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:02.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.625+0000 7f47ea000640 1 -- 172.21.15.28:0/3124241098 shutdown_connections 2024-02-21T23:09:02.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.625+0000 7f47ea000640 1 -- 172.21.15.28:0/3124241098 wait complete. 2024-02-21T23:09:02.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.627+0000 7f47ea000640 1 Processor -- start 2024-02-21T23:09:02.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.627+0000 7f47ea000640 1 -- start start 2024-02-21T23:09:02.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.628+0000 7f47ea000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47e40bde20 0x7f47e41440a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:02.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.628+0000 7f47ea000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47e40c74a0 0x7f47e41ee0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:02.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.628+0000 7f47ea000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f47e4152370 0x7f47e4143070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:02.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.628+0000 7f47ea000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47e40c5950 con 0x7f47e40c74a0 2024-02-21T23:09:02.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.628+0000 7f47ea000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47e406dd00 con 0x7f47e40bde20 2024-02-21T23:09:02.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.628+0000 7f47ea000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47e40c5ad0 con 0x7f47e4152370 2024-02-21T23:09:02.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.629+0000 7f47e3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47e40c74a0 0x7f47e41ee0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:02.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.629+0000 7f47e3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47e40c74a0 0x7f47e41ee0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41366/0 (socket says 172.21.15.28:41366) 2024-02-21T23:09:02.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.629+0000 7f47e3e00640 1 -- 172.21.15.28:0/338434481 learned_addr learned my addr 172.21.15.28:0/338434481 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:02.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.629+0000 7f47e8c00640 1 --2- 172.21.15.28:0/338434481 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47e40bde20 0x7f47e41440a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:02.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.629+0000 7f47e9600640 1 --2- 172.21.15.28:0/338434481 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f47e4152370 0x7f47e4143070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:02.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.629+0000 7f47e3e00640 1 -- 172.21.15.28:0/338434481 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f47e4152370 msgr2=0x7f47e4143070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:02.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.629+0000 7f47e3e00640 1 --2- 172.21.15.28:0/338434481 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f47e4152370 0x7f47e4143070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:02.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.629+0000 7f47e3e00640 1 -- 172.21.15.28:0/338434481 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47e40bde20 msgr2=0x7f47e41440a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:02.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.629+0000 7f47e3e00640 1 --2- 172.21.15.28:0/338434481 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47e40bde20 0x7f47e41440a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:02.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.629+0000 7f47e3e00640 1 -- 172.21.15.28:0/338434481 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47e40aa4e0 con 0x7f47e40c74a0 2024-02-21T23:09:02.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.629+0000 7f47e8c00640 1 --2- 172.21.15.28:0/338434481 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47e40bde20 0x7f47e41440a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:09:02.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.629+0000 7f47e3e00640 1 --2- 172.21.15.28:0/338434481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47e40c74a0 0x7f47e41ee0f0 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f47d000c5a0 tx=0x7f47d000ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:02.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.630+0000 7f47e0c00640 1 -- 172.21.15.28:0/338434481 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47d0015070 con 0x7f47e40c74a0 2024-02-21T23:09:02.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.630+0000 7f47e0c00640 1 -- 172.21.15.28:0/338434481 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47d0004220 con 0x7f47e40c74a0 2024-02-21T23:09:02.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.630+0000 7f47e0c00640 1 -- 172.21.15.28:0/338434481 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47d0003610 con 0x7f47e40c74a0 2024-02-21T23:09:02.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.630+0000 7f47ea000640 1 -- 172.21.15.28:0/338434481 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47e40abec0 con 0x7f47e40c74a0 2024-02-21T23:09:02.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.630+0000 7f47ea000640 1 -- 172.21.15.28:0/338434481 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47e40abb90 con 0x7f47e40c74a0 2024-02-21T23:09:02.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.631+0000 7f47ea000640 1 -- 172.21.15.28:0/338434481 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47e404f290 con 0x7f47e40c74a0 2024-02-21T23:09:02.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.632+0000 7f47e0c00640 1 -- 172.21.15.28:0/338434481 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f47d00037b0 con 0x7f47e40c74a0 2024-02-21T23:09:02.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.633+0000 7f47e0c00640 1 --2- 172.21.15.28:0/338434481 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f47c406b500 0x7f47c406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:02.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.634+0000 7f47e0c00640 1 -- 172.21.15.28:0/338434481 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 5619+0+0 (secure 0 0 0) 0x7f47d0089f50 con 0x7f47e40c74a0 2024-02-21T23:09:02.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.634+0000 7f47e8c00640 1 --2- 172.21.15.28:0/338434481 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f47c406b500 0x7f47c406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:02.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.635+0000 7f47e8c00640 1 --2- 172.21.15.28:0/338434481 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f47c406b500 0x7f47c406d9c0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f47d8036000 tx=0x7f47d803c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:02.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.639+0000 7f47e0c00640 1 -- 172.21.15.28:0/338434481 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f47d0058d80 con 0x7f47e40c74a0 2024-02-21T23:09:02.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:02.849+0000 7f47ea000640 1 -- 172.21.15.28:0/338434481 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f47e414a530 con 0x7f47e40c74a0 2024-02-21T23:09:03.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.375+0000 7f47e0c00640 1 -- 172.21.15.28:0/338434481 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v134) v1 ==== 170+0+0 (secure 0 0 0) 0x7f47d0058680 con 0x7f47e40c74a0 2024-02-21T23:09:03.365 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' created 2024-02-21T23:09:03.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.379+0000 7f47ea000640 1 -- 172.21.15.28:0/338434481 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f47c406b500 msgr2=0x7f47c406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:03.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.379+0000 7f47ea000640 1 --2- 172.21.15.28:0/338434481 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f47c406b500 0x7f47c406d9c0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f47d8036000 tx=0x7f47d803c040 comp rx=0 tx=0).stop 2024-02-21T23:09:03.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.379+0000 7f47ea000640 1 -- 172.21.15.28:0/338434481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47e40c74a0 msgr2=0x7f47e41ee0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:03.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.379+0000 7f47ea000640 1 --2- 172.21.15.28:0/338434481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47e40c74a0 0x7f47e41ee0f0 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f47d000c5a0 tx=0x7f47d000ca60 comp rx=0 tx=0).stop 2024-02-21T23:09:03.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.379+0000 7f47ea000640 1 -- 172.21.15.28:0/338434481 shutdown_connections 2024-02-21T23:09:03.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.379+0000 7f47ea000640 1 --2- 172.21.15.28:0/338434481 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f47c406b500 0x7f47c406d9c0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:03.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.379+0000 7f47ea000640 1 --2- 172.21.15.28:0/338434481 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f47e4152370 0x7f47e4143070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:03.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.379+0000 7f47ea000640 1 --2- 172.21.15.28:0/338434481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47e40c74a0 0x7f47e41ee0f0 unknown :-1 s=CLOSED pgs=1511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:03.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.380+0000 7f47ea000640 1 --2- 172.21.15.28:0/338434481 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f47e40bde20 0x7f47e41440a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:03.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.380+0000 7f47ea000640 1 -- 172.21.15.28:0/338434481 >> 172.21.15.28:0/338434481 conn(0x7f47e4072780 msgr2=0x7f47e414cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:03.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.381+0000 7f47ea000640 1 -- 172.21.15.28:0/338434481 shutdown_connections 2024-02-21T23:09:03.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.381+0000 7f47ea000640 1 -- 172.21.15.28:0/338434481 wait complete. 2024-02-21T23:09:03.407 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-02-21T23:09:03.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.610+0000 7f92c5400640 1 Processor -- start 2024-02-21T23:09:03.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.610+0000 7f92c5400640 1 -- start start 2024-02-21T23:09:03.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.611+0000 7f92c5400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f92c0150340 0x7f92c0070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:03.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.612+0000 7f92c5400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f92c00c4ac0 0x7f92c0152c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:03.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.612+0000 7f92c5400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f92c0156a50 0x7f92c00c5580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:03.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.612+0000 7f92c5400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92c0156290 con 0x7f92c0156a50 2024-02-21T23:09:03.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.612+0000 7f92c5400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92c00c2ca0 con 0x7f92c0150340 2024-02-21T23:09:03.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.612+0000 7f92c5400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92c00c4260 con 0x7f92c00c4ac0 2024-02-21T23:09:03.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.612+0000 7f92c4a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f92c0156a50 0x7f92c00c5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:03.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.612+0000 7f92c4a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f92c0156a50 0x7f92c00c5580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41378/0 (socket says 172.21.15.28:41378) 2024-02-21T23:09:03.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.612+0000 7f92c4a00640 1 -- 172.21.15.28:0/1256319292 learned_addr learned my addr 172.21.15.28:0/1256319292 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:03.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.612+0000 7f92bfe00640 1 --2- 172.21.15.28:0/1256319292 >> v2:172.21.15.81:3300/0 conn(0x7f92c0150340 0x7f92c0070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:03.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.612+0000 7f92b7e00640 1 --2- 172.21.15.28:0/1256319292 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f92c00c4ac0 0x7f92c0152c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:03.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.613+0000 7f92c4a00640 1 -- 172.21.15.28:0/1256319292 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f92c00c4ac0 msgr2=0x7f92c0152c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:03.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.613+0000 7f92c4a00640 1 --2- 172.21.15.28:0/1256319292 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f92c00c4ac0 0x7f92c0152c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:03.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.613+0000 7f92c4a00640 1 -- 172.21.15.28:0/1256319292 >> v2:172.21.15.81:3300/0 conn(0x7f92c0150340 msgr2=0x7f92c0070ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:03.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.613+0000 7f92c4a00640 1 --2- 172.21.15.28:0/1256319292 >> v2:172.21.15.81:3300/0 conn(0x7f92c0150340 0x7f92c0070ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:03.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.613+0000 7f92c4a00640 1 -- 172.21.15.28:0/1256319292 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92c00c4540 con 0x7f92c0156a50 2024-02-21T23:09:03.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.613+0000 7f92bfe00640 1 --2- 172.21.15.28:0/1256319292 >> v2:172.21.15.81:3300/0 conn(0x7f92c0150340 0x7f92c0070ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:09:03.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.614+0000 7f92c4a00640 1 --2- 172.21.15.28:0/1256319292 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92c0156a50 0x7f92c00c5580 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f92c00c8550 tx=0x7f92b802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c2ba6bcdaf344f5 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:03.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.614+0000 7f92bf400640 1 -- 172.21.15.28:0/1256319292 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b802c9f0 con 0x7f92c0156a50 2024-02-21T23:09:03.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.614+0000 7f92bf400640 1 -- 172.21.15.28:0/1256319292 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92b802cb90 con 0x7f92c0156a50 2024-02-21T23:09:03.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.614+0000 7f92bf400640 1 -- 172.21.15.28:0/1256319292 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b80343f0 con 0x7f92c0156a50 2024-02-21T23:09:03.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.615+0000 7f92c5400640 1 -- 172.21.15.28:0/1256319292 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92c0156a50 msgr2=0x7f92c00c5580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:03.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.615+0000 7f92c5400640 1 --2- 172.21.15.28:0/1256319292 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92c0156a50 0x7f92c00c5580 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f92c00c8550 tx=0x7f92b802be50 comp rx=0 tx=0).stop 2024-02-21T23:09:03.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.615+0000 7f92c5400640 1 -- 172.21.15.28:0/1256319292 shutdown_connections 2024-02-21T23:09:03.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.615+0000 7f92c5400640 1 --2- 172.21.15.28:0/1256319292 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92c0156a50 0x7f92c00c5580 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:03.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.615+0000 7f92c5400640 1 --2- 172.21.15.28:0/1256319292 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f92c00c4ac0 0x7f92c0152c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:03.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.615+0000 7f92c5400640 1 --2- 172.21.15.28:0/1256319292 >> v2:172.21.15.81:3300/0 conn(0x7f92c0150340 0x7f92c0070ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:03.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.615+0000 7f92c5400640 1 -- 172.21.15.28:0/1256319292 >> 172.21.15.28:0/1256319292 conn(0x7f92c006b290 msgr2=0x7f92c006b6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:03.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.615+0000 7f92c5400640 1 -- 172.21.15.28:0/1256319292 shutdown_connections 2024-02-21T23:09:03.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.615+0000 7f92c5400640 1 -- 172.21.15.28:0/1256319292 wait complete. 2024-02-21T23:09:03.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.617+0000 7f92c5400640 1 Processor -- start 2024-02-21T23:09:03.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.617+0000 7f92c5400640 1 -- start start 2024-02-21T23:09:03.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.618+0000 7f92c5400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f92c00c4ac0 0x7f92c01e8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:03.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.618+0000 7f92c5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92c0150340 0x7f92c00c63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:03.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.618+0000 7f92c5400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f92c0156a50 0x7f92c0143570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:03.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.618+0000 7f92c5400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92c0156ea0 con 0x7f92c0150340 2024-02-21T23:09:03.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.618+0000 7f92c5400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92c00c4820 con 0x7f92c00c4ac0 2024-02-21T23:09:03.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.618+0000 7f92c5400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92c0157020 con 0x7f92c0156a50 2024-02-21T23:09:03.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.618+0000 7f92b7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92c0150340 0x7f92c00c63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:03.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.618+0000 7f92b7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92c0150340 0x7f92c00c63d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41380/0 (socket says 172.21.15.28:41380) 2024-02-21T23:09:03.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.618+0000 7f92b7e00640 1 -- 172.21.15.28:0/1360365232 learned_addr learned my addr 172.21.15.28:0/1360365232 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:03.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.618+0000 7f92bfe00640 1 --2- 172.21.15.28:0/1360365232 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f92c00c4ac0 0x7f92c01e8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:03.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.618+0000 7f92c4a00640 1 --2- 172.21.15.28:0/1360365232 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f92c0156a50 0x7f92c0143570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:03.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.618+0000 7f92b7e00640 1 -- 172.21.15.28:0/1360365232 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f92c0156a50 msgr2=0x7f92c0143570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:03.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.618+0000 7f92b7e00640 1 --2- 172.21.15.28:0/1360365232 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f92c0156a50 0x7f92c0143570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:03.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.619+0000 7f92b7e00640 1 -- 172.21.15.28:0/1360365232 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f92c00c4ac0 msgr2=0x7f92c01e8b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:03.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.619+0000 7f92b7e00640 1 --2- 172.21.15.28:0/1360365232 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f92c00c4ac0 0x7f92c01e8b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:03.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.619+0000 7f92b7e00640 1 -- 172.21.15.28:0/1360365232 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92c01e93e0 con 0x7f92c0150340 2024-02-21T23:09:03.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.619+0000 7f92c4a00640 1 --2- 172.21.15.28:0/1360365232 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f92c0156a50 0x7f92c0143570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:09:03.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.619+0000 7f92bfe00640 1 --2- 172.21.15.28:0/1360365232 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f92c00c4ac0 0x7f92c01e8b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:09:03.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.619+0000 7f92b7e00640 1 --2- 172.21.15.28:0/1360365232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92c0150340 0x7f92c00c63d0 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f92a800d800 tx=0x7f92a800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:03.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.619+0000 7f92bcc00640 1 -- 172.21.15.28:0/1360365232 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92a8018030 con 0x7f92c0150340 2024-02-21T23:09:03.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.619+0000 7f92bcc00640 1 -- 172.21.15.28:0/1360365232 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92a80096a0 con 0x7f92c0150340 2024-02-21T23:09:03.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.619+0000 7f92c5400640 1 -- 172.21.15.28:0/1360365232 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92c00ad910 con 0x7f92c0150340 2024-02-21T23:09:03.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.619+0000 7f92bcc00640 1 -- 172.21.15.28:0/1360365232 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92a8009870 con 0x7f92c0150340 2024-02-21T23:09:03.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.619+0000 7f92c5400640 1 -- 172.21.15.28:0/1360365232 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92c01ec4f0 con 0x7f92c0150340 2024-02-21T23:09:03.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.621+0000 7f92c5400640 1 -- 172.21.15.28:0/1360365232 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92c004f290 con 0x7f92c0150340 2024-02-21T23:09:03.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.622+0000 7f92bcc00640 1 -- 172.21.15.28:0/1360365232 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f92a801b070 con 0x7f92c0150340 2024-02-21T23:09:03.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.623+0000 7f92bcc00640 1 --2- 172.21.15.28:0/1360365232 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f929c06b4d0 0x7f929c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:03.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.623+0000 7f92bcc00640 1 -- 172.21.15.28:0/1360365232 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 6001+0+0 (secure 0 0 0) 0x7f92a808a5a0 con 0x7f92c0150340 2024-02-21T23:09:03.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.623+0000 7f92bfe00640 1 --2- 172.21.15.28:0/1360365232 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f929c06b4d0 0x7f929c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:03.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.624+0000 7f92bfe00640 1 --2- 172.21.15.28:0/1360365232 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f929c06b4d0 0x7f929c06d990 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f92c00acaa0 tx=0x7f92b0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:03.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.628+0000 7f92bcc00640 1 -- 172.21.15.28:0/1360365232 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f92a80592e0 con 0x7f92c0150340 2024-02-21T23:09:03.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:03.866+0000 7f92c5400640 1 -- 172.21.15.28:0/1360365232 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f92c00c3fa0 con 0x7f92c0150340 2024-02-21T23:09:04.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.417+0000 7f92bcc00640 1 -- 172.21.15.28:0/1360365232 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 21 and data pool 22 v135) v1 ==== 174+0+0 (secure 0 0 0) 0x7f92a8058be0 con 0x7f92c0150340 2024-02-21T23:09:04.414 INFO:teuthology.orchestra.run.smithi028.stderr:new fs with metadata pool 21 and data pool 22 2024-02-21T23:09:04.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.428+0000 7f92c5400640 1 -- 172.21.15.28:0/1360365232 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f929c06b4d0 msgr2=0x7f929c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:04.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.428+0000 7f92c5400640 1 --2- 172.21.15.28:0/1360365232 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f929c06b4d0 0x7f929c06d990 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f92c00acaa0 tx=0x7f92b0008040 comp rx=0 tx=0).stop 2024-02-21T23:09:04.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.428+0000 7f92c5400640 1 -- 172.21.15.28:0/1360365232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92c0150340 msgr2=0x7f92c00c63d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:04.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.428+0000 7f92c5400640 1 --2- 172.21.15.28:0/1360365232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92c0150340 0x7f92c00c63d0 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f92a800d800 tx=0x7f92a800dcc0 comp rx=0 tx=0).stop 2024-02-21T23:09:04.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.432+0000 7f92c5400640 1 -- 172.21.15.28:0/1360365232 shutdown_connections 2024-02-21T23:09:04.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.432+0000 7f92c5400640 1 --2- 172.21.15.28:0/1360365232 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f929c06b4d0 0x7f929c06d990 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:04.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.432+0000 7f92c5400640 1 --2- 172.21.15.28:0/1360365232 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f92c0156a50 0x7f92c0143570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:04.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.432+0000 7f92c5400640 1 --2- 172.21.15.28:0/1360365232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92c0150340 0x7f92c00c63d0 secure :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f92a800d800 tx=0x7f92a800dcc0 comp rx=0 tx=0).stop 2024-02-21T23:09:04.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.432+0000 7f92c5400640 1 --2- 172.21.15.28:0/1360365232 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f92c00c4ac0 0x7f92c01e8b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:04.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.432+0000 7f92c5400640 1 -- 172.21.15.28:0/1360365232 >> 172.21.15.28:0/1360365232 conn(0x7f92c006b290 msgr2=0x7f92c00705e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:04.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.438+0000 7f92c5400640 1 -- 172.21.15.28:0/1360365232 shutdown_connections 2024-02-21T23:09:04.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.438+0000 7f92c5400640 1 -- 172.21.15.28:0/1360365232 wait complete. 2024-02-21T23:09:04.455 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:09:04.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.646+0000 7f637e200640 1 Processor -- start 2024-02-21T23:09:04.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.647+0000 7f637e200640 1 -- start start 2024-02-21T23:09:04.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.648+0000 7f637e200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f63780c54c0 0x7f63780a7f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:04.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.648+0000 7f637e200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6378158940 0x7f6378158dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:04.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.648+0000 7f637e200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63780c6820 con 0x7f63780c6ef0 2024-02-21T23:09:04.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.648+0000 7f637e200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63780c3de0 con 0x7f63780c54c0 2024-02-21T23:09:04.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.648+0000 7f637e200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63780c7fe0 con 0x7f6378158940 2024-02-21T23:09:04.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.649+0000 7f6377e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6378158940 0x7f6378158dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:04.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.649+0000 7f6377400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f63780c54c0 0x7f63780a7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:04.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.649+0000 7f6377e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6378158940 0x7f6378158dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3301/0 says I am v2:172.21.15.28:51354/0 (socket says 172.21.15.28:51354) 2024-02-21T23:09:04.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.649+0000 7f6377400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f63780c54c0 0x7f63780a7f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.28:45464/0 (socket says 172.21.15.28:45464) 2024-02-21T23:09:04.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.649+0000 7f6377e00640 1 -- 172.21.15.28:0/2130379875 learned_addr learned my addr 172.21.15.28:0/2130379875 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:04.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.649+0000 7f6377e00640 1 -- 172.21.15.28:0/2130379875 >> v2:172.21.15.81:3300/0 conn(0x7f63780c54c0 msgr2=0x7f63780a7f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:04.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.649+0000 7f6377e00640 1 --2- 172.21.15.28:0/2130379875 >> v2:172.21.15.81:3300/0 conn(0x7f63780c54c0 0x7f63780a7f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:04.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.649+0000 7f6377e00640 1 -- 172.21.15.28:0/2130379875 >> v1:172.21.15.81:6789/0 conn(0x7f63780c6ef0 legacy=0x7f6378152560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:04.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.650+0000 7f6377e00640 1 -- 172.21.15.28:0/2130379875 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63780c6b00 con 0x7f6378158940 2024-02-21T23:09:04.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.651+0000 7f6377e00640 1 --2- 172.21.15.28:0/2130379875 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6378158940 0x7f6378158dd0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6370009870 tx=0x7f637002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a8854e0aed971f9 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:04.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.651+0000 7f6376000640 1 -- 172.21.15.28:0/2130379875 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6370035020 con 0x7f6378158940 2024-02-21T23:09:04.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.651+0000 7f6376000640 1 -- 172.21.15.28:0/2130379875 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6370031640 con 0x7f6378158940 2024-02-21T23:09:04.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.651+0000 7f6376000640 1 -- 172.21.15.28:0/2130379875 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f637002d040 con 0x7f6378158940 2024-02-21T23:09:04.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.652+0000 7f637e200640 1 -- 172.21.15.28:0/2130379875 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6378158940 msgr2=0x7f6378158dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:04.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.652+0000 7f637e200640 1 --2- 172.21.15.28:0/2130379875 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6378158940 0x7f6378158dd0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6370009870 tx=0x7f637002b8f0 comp rx=0 tx=0).stop 2024-02-21T23:09:04.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.652+0000 7f637e200640 1 -- 172.21.15.28:0/2130379875 shutdown_connections 2024-02-21T23:09:04.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.652+0000 7f637e200640 1 --2- 172.21.15.28:0/2130379875 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6378158940 0x7f6378158dd0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:04.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.652+0000 7f637e200640 1 --2- 172.21.15.28:0/2130379875 >> v2:172.21.15.81:3300/0 conn(0x7f63780c54c0 0x7f63780a7f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:04.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.652+0000 7f637e200640 1 -- 172.21.15.28:0/2130379875 >> 172.21.15.28:0/2130379875 conn(0x7f63780c8810 msgr2=0x7f63780c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:04.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.652+0000 7f637e200640 1 -- 172.21.15.28:0/2130379875 shutdown_connections 2024-02-21T23:09:04.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.652+0000 7f637e200640 1 -- 172.21.15.28:0/2130379875 wait complete. 2024-02-21T23:09:04.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.654+0000 7f637e200640 1 Processor -- start 2024-02-21T23:09:04.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.654+0000 7f637e200640 1 -- start start 2024-02-21T23:09:04.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.655+0000 7f637e200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63780c54c0 0x7f6378149260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:04.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.655+0000 7f637e200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f63780c6ef0 0x7f6378148570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:04.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.655+0000 7f637e200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6378158940 0x7f6378070380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:04.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.655+0000 7f637e200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6378152c70 con 0x7f6378158940 2024-02-21T23:09:04.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.655+0000 7f637e200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63780c85a0 con 0x7f63780c54c0 2024-02-21T23:09:04.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.655+0000 7f637e200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6378152df0 con 0x7f63780c6ef0 2024-02-21T23:09:04.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.655+0000 7f6377e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6378158940 0x7f6378070380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:04.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.655+0000 7f6377e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6378158940 0x7f6378070380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41396/0 (socket says 172.21.15.28:41396) 2024-02-21T23:09:04.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.655+0000 7f6377e00640 1 -- 172.21.15.28:0/2820843642 learned_addr learned my addr 172.21.15.28:0/2820843642 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:04.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.656+0000 7f6377400640 1 --2- 172.21.15.28:0/2820843642 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63780c54c0 0x7f6378149260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:04.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.656+0000 7f6376a00640 1 --2- 172.21.15.28:0/2820843642 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f63780c6ef0 0x7f6378148570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:04.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.656+0000 7f6377e00640 1 -- 172.21.15.28:0/2820843642 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f63780c6ef0 msgr2=0x7f6378148570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:04.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.656+0000 7f6377e00640 1 --2- 172.21.15.28:0/2820843642 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f63780c6ef0 0x7f6378148570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:04.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.656+0000 7f6377e00640 1 -- 172.21.15.28:0/2820843642 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63780c54c0 msgr2=0x7f6378149260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:04.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.656+0000 7f6377e00640 1 --2- 172.21.15.28:0/2820843642 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63780c54c0 0x7f6378149260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:04.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.656+0000 7f6377e00640 1 -- 172.21.15.28:0/2820843642 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63780aa2f0 con 0x7f6378158940 2024-02-21T23:09:04.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.656+0000 7f6377400640 1 --2- 172.21.15.28:0/2820843642 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63780c54c0 0x7f6378149260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:09:04.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.656+0000 7f6376a00640 1 --2- 172.21.15.28:0/2820843642 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f63780c6ef0 0x7f6378148570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:09:04.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.656+0000 7f6377e00640 1 --2- 172.21.15.28:0/2820843642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6378158940 0x7f6378070380 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f6370009870 tx=0x7f63700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:04.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.657+0000 7f636f400640 1 -- 172.21.15.28:0/2820843642 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6370040070 con 0x7f6378158940 2024-02-21T23:09:04.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.657+0000 7f636f400640 1 -- 172.21.15.28:0/2820843642 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63700399f0 con 0x7f6378158940 2024-02-21T23:09:04.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.657+0000 7f636f400640 1 -- 172.21.15.28:0/2820843642 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6370035040 con 0x7f6378158940 2024-02-21T23:09:04.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.657+0000 7f637e200640 1 -- 172.21.15.28:0/2820843642 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6378148f80 con 0x7f6378158940 2024-02-21T23:09:04.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.657+0000 7f637e200640 1 -- 172.21.15.28:0/2820843642 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63780ad3d0 con 0x7f6378158940 2024-02-21T23:09:04.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.658+0000 7f637e200640 1 -- 172.21.15.28:0/2820843642 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6378068640 con 0x7f6378158940 2024-02-21T23:09:04.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.659+0000 7f636f400640 1 -- 172.21.15.28:0/2820843642 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f637004b060 con 0x7f6378158940 2024-02-21T23:09:04.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.660+0000 7f636f400640 1 --2- 172.21.15.28:0/2820843642 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f635406b4d0 0x7f635406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:04.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.660+0000 7f636f400640 1 -- 172.21.15.28:0/2820843642 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f63700abc30 con 0x7f6378158940 2024-02-21T23:09:04.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.660+0000 7f6377400640 1 --2- 172.21.15.28:0/2820843642 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f635406b4d0 0x7f635406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:04.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.661+0000 7f6377400640 1 --2- 172.21.15.28:0/2820843642 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f635406b4d0 0x7f635406d990 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f63780ad8e0 tx=0x7f6368008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:04.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.664+0000 7f636f400640 1 -- 172.21.15.28:0/2820843642 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f63780aa2f0 con 0x7f6378158940 2024-02-21T23:09:04.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.859+0000 7f637e200640 1 -- 172.21.15.28:0/2820843642 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f63780acec0 con 0x7f6378158940 2024-02-21T23:09:04.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.862+0000 7f636f400640 1 -- 172.21.15.28:0/2820843642 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v135) v1 ==== 74+0+16845 (secure 0 0 0) 0x7f637007a910 con 0x7f6378158940 2024-02-21T23:09:04.851 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:09:04.852 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":135,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:09:03.868484+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:09:01.777487+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"135","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2024-02-21T23:09:02.904570+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"135","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:09:04.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.864+0000 7f637e200640 1 -- 172.21.15.28:0/2820843642 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f635406b4d0 msgr2=0x7f635406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:04.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.864+0000 7f637e200640 1 --2- 172.21.15.28:0/2820843642 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f635406b4d0 0x7f635406d990 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f63780ad8e0 tx=0x7f6368008040 comp rx=0 tx=0).stop 2024-02-21T23:09:04.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.864+0000 7f637e200640 1 -- 172.21.15.28:0/2820843642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6378158940 msgr2=0x7f6378070380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:04.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.864+0000 7f637e200640 1 --2- 172.21.15.28:0/2820843642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6378158940 0x7f6378070380 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f6370009870 tx=0x7f63700046c0 comp rx=0 tx=0).stop 2024-02-21T23:09:04.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.865+0000 7f637e200640 1 -- 172.21.15.28:0/2820843642 shutdown_connections 2024-02-21T23:09:04.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.865+0000 7f637e200640 1 --2- 172.21.15.28:0/2820843642 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f635406b4d0 0x7f635406d990 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:04.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.865+0000 7f637e200640 1 --2- 172.21.15.28:0/2820843642 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f63780c6ef0 0x7f6378148570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:04.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.865+0000 7f637e200640 1 --2- 172.21.15.28:0/2820843642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6378158940 0x7f6378070380 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:04.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.865+0000 7f637e200640 1 --2- 172.21.15.28:0/2820843642 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f63780c54c0 0x7f6378149260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:04.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.865+0000 7f637e200640 1 -- 172.21.15.28:0/2820843642 >> 172.21.15.28:0/2820843642 conn(0x7f63780c8810 msgr2=0x7f637806c4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:04.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.865+0000 7f637e200640 1 -- 172.21.15.28:0/2820843642 shutdown_connections 2024-02-21T23:09:04.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:04.865+0000 7f637e200640 1 -- 172.21.15.28:0/2820843642 wait complete. 2024-02-21T23:09:04.873 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:09:05.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.094+0000 7fcd21600640 1 Processor -- start 2024-02-21T23:09:05.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.094+0000 7fcd21600640 1 -- start start 2024-02-21T23:09:05.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.096+0000 7fcd21600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcd1c150450 0x7fcd1c0a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:05.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.096+0000 7fcd21600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcd1c06ee40 0x7fcd1c1536f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:05.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.096+0000 7fcd21600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd1c0c2760 con 0x7fcd1c156af0 2024-02-21T23:09:05.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.096+0000 7fcd21600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd1c0c22c0 con 0x7fcd1c06ee40 2024-02-21T23:09:05.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.096+0000 7fcd21600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd1c156660 con 0x7fcd1c150450 2024-02-21T23:09:05.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.096+0000 7fcd1b400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcd1c06ee40 0x7fcd1c1536f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:05.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.096+0000 7fcd1b400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcd1c06ee40 0x7fcd1c1536f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56992/0 (socket says 172.21.15.28:56992) 2024-02-21T23:09:05.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.096+0000 7fcd1b400640 1 -- 172.21.15.28:0/2467306388 learned_addr learned my addr 172.21.15.28:0/2467306388 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:05.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.096+0000 7fcd1be00640 1 --2- 172.21.15.28:0/2467306388 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcd1c150450 0x7fcd1c0a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:05.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.097+0000 7fcd1b400640 1 -- 172.21.15.28:0/2467306388 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcd1c150450 msgr2=0x7fcd1c0a5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:05.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.097+0000 7fcd1b400640 1 --2- 172.21.15.28:0/2467306388 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcd1c150450 0x7fcd1c0a5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:05.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.097+0000 7fcd1b400640 1 -- 172.21.15.28:0/2467306388 >> v1:172.21.15.81:6789/0 conn(0x7fcd1c156af0 legacy=0x7fcd1c06dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:05.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.097+0000 7fcd1b400640 1 -- 172.21.15.28:0/2467306388 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd1c153c30 con 0x7fcd1c06ee40 2024-02-21T23:09:05.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.098+0000 7fcd1b400640 1 --2- 172.21.15.28:0/2467306388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd1c06ee40 0x7fcd1c1536f0 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7fcd08009870 tx=0x7fcd0802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b83a5d4d1ec59a2 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:05.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.099+0000 7fcd1aa00640 1 -- 172.21.15.28:0/2467306388 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd08034070 con 0x7fcd1c06ee40 2024-02-21T23:09:05.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.099+0000 7fcd1aa00640 1 -- 172.21.15.28:0/2467306388 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd08030840 con 0x7fcd1c06ee40 2024-02-21T23:09:05.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.099+0000 7fcd1aa00640 1 -- 172.21.15.28:0/2467306388 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd08030c10 con 0x7fcd1c06ee40 2024-02-21T23:09:05.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.099+0000 7fcd21600640 1 -- 172.21.15.28:0/2467306388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd1c06ee40 msgr2=0x7fcd1c1536f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:05.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.099+0000 7fcd21600640 1 --2- 172.21.15.28:0/2467306388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd1c06ee40 0x7fcd1c1536f0 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7fcd08009870 tx=0x7fcd0802b980 comp rx=0 tx=0).stop 2024-02-21T23:09:05.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.100+0000 7fcd21600640 1 -- 172.21.15.28:0/2467306388 shutdown_connections 2024-02-21T23:09:05.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.100+0000 7fcd21600640 1 --2- 172.21.15.28:0/2467306388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd1c06ee40 0x7fcd1c1536f0 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:05.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.100+0000 7fcd21600640 1 --2- 172.21.15.28:0/2467306388 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcd1c150450 0x7fcd1c0a5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:05.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.100+0000 7fcd21600640 1 -- 172.21.15.28:0/2467306388 >> 172.21.15.28:0/2467306388 conn(0x7fcd1c06b630 msgr2=0x7fcd1c0c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:05.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.100+0000 7fcd21600640 1 -- 172.21.15.28:0/2467306388 shutdown_connections 2024-02-21T23:09:05.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.100+0000 7fcd21600640 1 -- 172.21.15.28:0/2467306388 wait complete. 2024-02-21T23:09:05.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.102+0000 7fcd21600640 1 Processor -- start 2024-02-21T23:09:05.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.102+0000 7fcd21600640 1 -- start start 2024-02-21T23:09:05.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.103+0000 7fcd21600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcd1c06ee40 0x7fcd1c150b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:05.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.103+0000 7fcd21600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcd1c150450 0x7fcd1c0bfba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:05.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.103+0000 7fcd21600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd1c156af0 0x7fcd1c153e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:05.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.103+0000 7fcd21600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd1c0c2cb0 con 0x7fcd1c156af0 2024-02-21T23:09:05.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.103+0000 7fcd21600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd1c0c29d0 con 0x7fcd1c150450 2024-02-21T23:09:05.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.103+0000 7fcd21600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd1c06e700 con 0x7fcd1c06ee40 2024-02-21T23:09:05.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.103+0000 7fcd20c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd1c156af0 0x7fcd1c153e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:05.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.103+0000 7fcd20c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd1c156af0 0x7fcd1c153e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56994/0 (socket says 172.21.15.28:56994) 2024-02-21T23:09:05.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.103+0000 7fcd20c00640 1 -- 172.21.15.28:0/4181032816 learned_addr learned my addr 172.21.15.28:0/4181032816 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:05.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.103+0000 7fcd1be00640 1 --2- 172.21.15.28:0/4181032816 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcd1c06ee40 0x7fcd1c150b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:05.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.104+0000 7fcd1b400640 1 --2- 172.21.15.28:0/4181032816 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcd1c150450 0x7fcd1c0bfba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:05.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.104+0000 7fcd20c00640 1 -- 172.21.15.28:0/4181032816 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcd1c06ee40 msgr2=0x7fcd1c150b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:05.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.104+0000 7fcd20c00640 1 --2- 172.21.15.28:0/4181032816 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcd1c06ee40 0x7fcd1c150b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:05.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.104+0000 7fcd20c00640 1 -- 172.21.15.28:0/4181032816 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcd1c150450 msgr2=0x7fcd1c0bfba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:05.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.104+0000 7fcd20c00640 1 --2- 172.21.15.28:0/4181032816 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcd1c150450 0x7fcd1c0bfba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:05.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.104+0000 7fcd20c00640 1 -- 172.21.15.28:0/4181032816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd1c155a20 con 0x7fcd1c156af0 2024-02-21T23:09:05.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.104+0000 7fcd20c00640 1 --2- 172.21.15.28:0/4181032816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd1c156af0 0x7fcd1c153e00 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7fcd1400c6b0 tx=0x7fcd1400cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:05.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.104+0000 7fcd13e00640 1 -- 172.21.15.28:0/4181032816 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd14003b70 con 0x7fcd1c156af0 2024-02-21T23:09:05.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.104+0000 7fcd13e00640 1 -- 172.21.15.28:0/4181032816 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd14003d30 con 0x7fcd1c156af0 2024-02-21T23:09:05.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.104+0000 7fcd21600640 1 -- 172.21.15.28:0/4181032816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd1c1eae70 con 0x7fcd1c156af0 2024-02-21T23:09:05.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.104+0000 7fcd13e00640 1 -- 172.21.15.28:0/4181032816 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd14005310 con 0x7fcd1c156af0 2024-02-21T23:09:05.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.105+0000 7fcd21600640 1 -- 172.21.15.28:0/4181032816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd1c155fa0 con 0x7fcd1c156af0 2024-02-21T23:09:05.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.106+0000 7fcd21600640 1 -- 172.21.15.28:0/4181032816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd1c04f290 con 0x7fcd1c156af0 2024-02-21T23:09:05.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.107+0000 7fcd13e00640 1 -- 172.21.15.28:0/4181032816 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcd140054b0 con 0x7fcd1c156af0 2024-02-21T23:09:05.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.108+0000 7fcd13e00640 1 --2- 172.21.15.28:0/4181032816 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fccfc06b500 0x7fccfc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:05.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.108+0000 7fcd13e00640 1 -- 172.21.15.28:0/4181032816 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fcd1408a220 con 0x7fcd1c156af0 2024-02-21T23:09:05.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.108+0000 7fcd1be00640 1 --2- 172.21.15.28:0/4181032816 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fccfc06b500 0x7fccfc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:05.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.109+0000 7fcd1be00640 1 --2- 172.21.15.28:0/4181032816 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fccfc06b500 0x7fccfc06d9c0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fcd0c006fd0 tx=0x7fcd0c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:05.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.112+0000 7fcd13e00640 1 -- 172.21.15.28:0/4181032816 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fcd14058f00 con 0x7fcd1c156af0 2024-02-21T23:09:05.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.314+0000 7fcd21600640 1 -- 172.21.15.28:0/4181032816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcd1c0c1f10 con 0x7fcd1c156af0 2024-02-21T23:09:05.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.316+0000 7fcd13e00640 1 -- 172.21.15.28:0/4181032816 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v135) v1 ==== 74+0+16845 (secure 0 0 0) 0x7fcd14058800 con 0x7fcd1c156af0 2024-02-21T23:09:05.305 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:09:05.305 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":135,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:09:03.868484+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:09:01.777487+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"135","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2024-02-21T23:09:02.904570+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"135","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:09:05.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.318+0000 7fcd21600640 1 -- 172.21.15.28:0/4181032816 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fccfc06b500 msgr2=0x7fccfc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:05.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.318+0000 7fcd21600640 1 --2- 172.21.15.28:0/4181032816 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fccfc06b500 0x7fccfc06d9c0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fcd0c006fd0 tx=0x7fcd0c008040 comp rx=0 tx=0).stop 2024-02-21T23:09:05.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.318+0000 7fcd21600640 1 -- 172.21.15.28:0/4181032816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd1c156af0 msgr2=0x7fcd1c153e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:05.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.318+0000 7fcd21600640 1 --2- 172.21.15.28:0/4181032816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd1c156af0 0x7fcd1c153e00 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7fcd1400c6b0 tx=0x7fcd1400cb70 comp rx=0 tx=0).stop 2024-02-21T23:09:05.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.318+0000 7fcd21600640 1 -- 172.21.15.28:0/4181032816 shutdown_connections 2024-02-21T23:09:05.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.318+0000 7fcd21600640 1 --2- 172.21.15.28:0/4181032816 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fccfc06b500 0x7fccfc06d9c0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:05.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.318+0000 7fcd21600640 1 --2- 172.21.15.28:0/4181032816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd1c156af0 0x7fcd1c153e00 unknown :-1 s=CLOSED pgs=1516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:05.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.318+0000 7fcd21600640 1 --2- 172.21.15.28:0/4181032816 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcd1c150450 0x7fcd1c0bfba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:05.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.318+0000 7fcd21600640 1 --2- 172.21.15.28:0/4181032816 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcd1c06ee40 0x7fcd1c150b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:05.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.318+0000 7fcd21600640 1 -- 172.21.15.28:0/4181032816 >> 172.21.15.28:0/4181032816 conn(0x7fcd1c06b630 msgr2=0x7fcd1c0a7500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:05.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.318+0000 7fcd21600640 1 -- 172.21.15.28:0/4181032816 shutdown_connections 2024-02-21T23:09:05.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.318+0000 7fcd21600640 1 -- 172.21.15.28:0/4181032816 wait complete. 2024-02-21T23:09:05.327 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-02-21T23:09:05.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.566+0000 7f67a3000640 1 Processor -- start 2024-02-21T23:09:05.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.567+0000 7f67a3000640 1 -- start start 2024-02-21T23:09:05.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.568+0000 7f67a3000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f679c0c5a70 0x7f679c155a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:05.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.568+0000 7f67a3000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f679c0c22e0 0x7f679c155fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:05.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.568+0000 7f67a3000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f679c071970 con 0x7f679c0c2b30 2024-02-21T23:09:05.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.568+0000 7f67a3000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f679c072a10 con 0x7f679c0c22e0 2024-02-21T23:09:05.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.568+0000 7f67a3000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f679c072580 con 0x7f679c0c5a70 2024-02-21T23:09:05.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.568+0000 7f67a1000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f679c0c2b30 0x7f679c06d120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:56856/0 (socket says 172.21.15.28:56856) 2024-02-21T23:09:05.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.569+0000 7f67a1000640 1 -- 172.21.15.28:0/2849774230 learned_addr learned my addr 172.21.15.28:0/2849774230 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:09:05.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.569+0000 7f6793e00640 1 --2- 172.21.15.28:0/2849774230 >> v2:172.21.15.28:3300/0 conn(0x7f679c0c22e0 0x7f679c155fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:05.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.569+0000 7f679be00640 1 --2- 172.21.15.28:0/2849774230 >> v2:172.21.15.81:3300/0 conn(0x7f679c0c5a70 0x7f679c155a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:05.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.569+0000 7f679b400640 1 -- 172.21.15.28:0/2849774230 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 940262074 0 0) 0x7f679c071970 con 0x7f679c0c2b30 2024-02-21T23:09:05.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.569+0000 7f679b400640 1 -- 172.21.15.28:0/2849774230 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6780003520 con 0x7f679c0c2b30 2024-02-21T23:09:05.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.569+0000 7f6793e00640 1 -- 172.21.15.28:0/2849774230 >> v2:172.21.15.81:3300/0 conn(0x7f679c0c5a70 msgr2=0x7f679c155a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:05.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.569+0000 7f6793e00640 1 --2- 172.21.15.28:0/2849774230 >> v2:172.21.15.81:3300/0 conn(0x7f679c0c5a70 0x7f679c155a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:05.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.569+0000 7f6793e00640 1 -- 172.21.15.28:0/2849774230 >> v1:172.21.15.28:6789/0 conn(0x7f679c0c2b30 legacy=0x7f679c06d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:05.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.569+0000 7f6793e00640 1 -- 172.21.15.28:0/2849774230 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f679c071c50 con 0x7f679c0c22e0 2024-02-21T23:09:05.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.571+0000 7f6793e00640 1 --2- 172.21.15.28:0/2849774230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f679c0c22e0 0x7f679c155fd0 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f679c0c7810 tx=0x7f678402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5a6a4d048851d4a server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:05.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.571+0000 7f679b400640 1 -- 172.21.15.28:0/2849774230 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67840315a0 con 0x7f679c0c22e0 2024-02-21T23:09:05.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.571+0000 7f679b400640 1 -- 172.21.15.28:0/2849774230 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6784037070 con 0x7f679c0c22e0 2024-02-21T23:09:05.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.571+0000 7f679b400640 1 -- 172.21.15.28:0/2849774230 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6784031940 con 0x7f679c0c22e0 2024-02-21T23:09:05.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.572+0000 7f67a3000640 1 -- 172.21.15.28:0/2849774230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f679c0c22e0 msgr2=0x7f679c155fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:05.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.572+0000 7f67a3000640 1 --2- 172.21.15.28:0/2849774230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f679c0c22e0 0x7f679c155fd0 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f679c0c7810 tx=0x7f678402b840 comp rx=0 tx=0).stop 2024-02-21T23:09:05.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.572+0000 7f67a3000640 1 -- 172.21.15.28:0/2849774230 shutdown_connections 2024-02-21T23:09:05.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.572+0000 7f67a3000640 1 --2- 172.21.15.28:0/2849774230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f679c0c22e0 0x7f679c155fd0 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:05.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.572+0000 7f67a3000640 1 --2- 172.21.15.28:0/2849774230 >> v2:172.21.15.81:3300/0 conn(0x7f679c0c5a70 0x7f679c155a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:05.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.572+0000 7f67a3000640 1 -- 172.21.15.28:0/2849774230 >> 172.21.15.28:0/2849774230 conn(0x7f679c0a85a0 msgr2=0x7f679c0a8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:05.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.572+0000 7f67a3000640 1 -- 172.21.15.28:0/2849774230 shutdown_connections 2024-02-21T23:09:05.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.572+0000 7f67a3000640 1 -- 172.21.15.28:0/2849774230 wait complete. 2024-02-21T23:09:05.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.574+0000 7f67a3000640 1 Processor -- start 2024-02-21T23:09:05.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.574+0000 7f67a3000640 1 -- start start 2024-02-21T23:09:05.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.575+0000 7f67a3000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f679c0c22e0 0x7f679c0a6f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:05.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.575+0000 7f67a3000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f679c0c2b30 0x7f679c15ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:05.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.575+0000 7f67a3000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f679c0c5a70 0x7f679c15f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:05.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.575+0000 7f67a3000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f679c0a6160 con 0x7f679c0c2b30 2024-02-21T23:09:05.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.575+0000 7f67a3000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f679c072210 con 0x7f679c0c22e0 2024-02-21T23:09:05.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.575+0000 7f67a3000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f679c0a62e0 con 0x7f679c0c5a70 2024-02-21T23:09:05.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.575+0000 7f6793e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f679c0c2b30 0x7f679c15ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:05.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.575+0000 7f6793e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f679c0c2b30 0x7f679c15ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57018/0 (socket says 172.21.15.28:57018) 2024-02-21T23:09:05.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.575+0000 7f6793e00640 1 -- 172.21.15.28:0/4097770093 learned_addr learned my addr 172.21.15.28:0/4097770093 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:05.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.575+0000 7f679be00640 1 --2- 172.21.15.28:0/4097770093 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f679c0c22e0 0x7f679c0a6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:05.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.575+0000 7f67a1000640 1 --2- 172.21.15.28:0/4097770093 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f679c0c5a70 0x7f679c15f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:05.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.576+0000 7f6793e00640 1 -- 172.21.15.28:0/4097770093 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f679c0c5a70 msgr2=0x7f679c15f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:05.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.576+0000 7f6793e00640 1 --2- 172.21.15.28:0/4097770093 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f679c0c5a70 0x7f679c15f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:05.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.576+0000 7f6793e00640 1 -- 172.21.15.28:0/4097770093 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f679c0c22e0 msgr2=0x7f679c0a6f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:05.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.576+0000 7f6793e00640 1 --2- 172.21.15.28:0/4097770093 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f679c0c22e0 0x7f679c0a6f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:05.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.576+0000 7f6793e00640 1 -- 172.21.15.28:0/4097770093 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f679c0ab500 con 0x7f679c0c2b30 2024-02-21T23:09:05.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.576+0000 7f6793e00640 1 --2- 172.21.15.28:0/4097770093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f679c0c2b30 0x7f679c15ef40 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f678402b880 tx=0x7f67840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:05.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.576+0000 7f6798c00640 1 -- 172.21.15.28:0/4097770093 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67840317a0 con 0x7f679c0c2b30 2024-02-21T23:09:05.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.577+0000 7f6798c00640 1 -- 172.21.15.28:0/4097770093 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6784031940 con 0x7f679c0c2b30 2024-02-21T23:09:05.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.577+0000 7f6798c00640 1 -- 172.21.15.28:0/4097770093 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6784031d10 con 0x7f679c0c2b30 2024-02-21T23:09:05.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.577+0000 7f67a3000640 1 -- 172.21.15.28:0/4097770093 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f679c15d9d0 con 0x7f679c0c2b30 2024-02-21T23:09:05.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.577+0000 7f67a3000640 1 -- 172.21.15.28:0/4097770093 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f679c0abd70 con 0x7f679c0c2b30 2024-02-21T23:09:05.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.578+0000 7f67a3000640 1 -- 172.21.15.28:0/4097770093 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f679c15dba0 con 0x7f679c0c2b30 2024-02-21T23:09:05.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.580+0000 7f6798c00640 1 -- 172.21.15.28:0/4097770093 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6784032430 con 0x7f679c0c2b30 2024-02-21T23:09:05.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.581+0000 7f6798c00640 1 --2- 172.21.15.28:0/4097770093 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f677806b4d0 0x7f677806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:05.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.582+0000 7f6798c00640 1 -- 172.21.15.28:0/4097770093 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f678402f020 con 0x7f679c0c2b30 2024-02-21T23:09:05.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.582+0000 7f679be00640 1 --2- 172.21.15.28:0/4097770093 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f677806b4d0 0x7f677806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:05.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.583+0000 7f679be00640 1 --2- 172.21.15.28:0/4097770093 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f677806b4d0 0x7f677806d990 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f678c009a30 tx=0x7f678c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:05.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.586+0000 7f6798c00640 1 -- 172.21.15.28:0/4097770093 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f678407b390 con 0x7f679c0c2b30 2024-02-21T23:09:05.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:05.862+0000 7f67a3000640 1 -- 172.21.15.28:0/4097770093 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f679c067f30 con 0x7f679c0c2b30 2024-02-21T23:09:06.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.441+0000 7f6798c00640 1 -- 172.21.15.28:0/4097770093 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v138) v1 ==== 118+0+0 (secure 0 0 0) 0x7f678407ac90 con 0x7f679c0c2b30 2024-02-21T23:09:06.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.446+0000 7f67a3000640 1 -- 172.21.15.28:0/4097770093 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f677806b4d0 msgr2=0x7f677806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:06.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.446+0000 7f67a3000640 1 --2- 172.21.15.28:0/4097770093 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f677806b4d0 0x7f677806d990 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f678c009a30 tx=0x7f678c008040 comp rx=0 tx=0).stop 2024-02-21T23:09:06.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.446+0000 7f67a3000640 1 -- 172.21.15.28:0/4097770093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f679c0c2b30 msgr2=0x7f679c15ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:06.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.446+0000 7f67a3000640 1 --2- 172.21.15.28:0/4097770093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f679c0c2b30 0x7f679c15ef40 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f678402b880 tx=0x7f67840046c0 comp rx=0 tx=0).stop 2024-02-21T23:09:06.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.446+0000 7f67a3000640 1 -- 172.21.15.28:0/4097770093 shutdown_connections 2024-02-21T23:09:06.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.446+0000 7f67a3000640 1 --2- 172.21.15.28:0/4097770093 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f677806b4d0 0x7f677806d990 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:06.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.446+0000 7f67a3000640 1 --2- 172.21.15.28:0/4097770093 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f679c0c5a70 0x7f679c15f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:06.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.446+0000 7f67a3000640 1 --2- 172.21.15.28:0/4097770093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f679c0c2b30 0x7f679c15ef40 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:06.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.446+0000 7f67a3000640 1 --2- 172.21.15.28:0/4097770093 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f679c0c22e0 0x7f679c0a6f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:06.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.446+0000 7f67a3000640 1 -- 172.21.15.28:0/4097770093 >> 172.21.15.28:0/4097770093 conn(0x7f679c0a85a0 msgr2=0x7f679c06ded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:06.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.447+0000 7f67a3000640 1 -- 172.21.15.28:0/4097770093 shutdown_connections 2024-02-21T23:09:06.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.447+0000 7f67a3000640 1 -- 172.21.15.28:0/4097770093 wait complete. 2024-02-21T23:09:06.462 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-02-21T23:09:06.462 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-02-21T23:09:06.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.703+0000 7f1ac4c00640 1 Processor -- start 2024-02-21T23:09:06.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.703+0000 7f1ac4c00640 1 -- start start 2024-02-21T23:09:06.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.705+0000 7f1ac4c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1ac00c4580 0x7f1ac00c12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:06.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.705+0000 7f1ac4c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ac006e220 con 0x7f1ac0150240 2024-02-21T23:09:06.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.705+0000 7f1ac4c00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ac006f780 con 0x7f1ac00c2c10 2024-02-21T23:09:06.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.705+0000 7f1ac4c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac0150620 con 0x7f1ac00c4580 2024-02-21T23:09:06.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.705+0000 7f1abf400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f1ac0150240 0x7f1ac006db10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:56860/0 (socket says 172.21.15.28:56860) 2024-02-21T23:09:06.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.705+0000 7f1abf400640 1 -- 172.21.15.28:0/3062004815 learned_addr learned my addr 172.21.15.28:0/3062004815 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:09:06.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.705+0000 7f1abe000640 1 -- 172.21.15.28:0/3062004815 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 640802905 0 0) 0x7f1ac006e220 con 0x7f1ac0150240 2024-02-21T23:09:06.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.705+0000 7f1abea00640 1 --2- 172.21.15.28:0/3062004815 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1ac00c4580 0x7f1ac00c12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:06.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.705+0000 7f1abe000640 1 -- 172.21.15.28:0/3062004815 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1aa4003520 con 0x7f1ac0150240 2024-02-21T23:09:06.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.706+0000 7f1abe000640 1 -- 172.21.15.28:0/3062004815 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3679357191 0 0) 0x7f1aa4003520 con 0x7f1ac0150240 2024-02-21T23:09:06.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.706+0000 7f1abe000640 1 -- 172.21.15.28:0/3062004815 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1ac00c4580 msgr2=0x7f1ac00c12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:06.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.706+0000 7f1abe000640 1 --2- 172.21.15.28:0/3062004815 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1ac00c4580 0x7f1ac00c12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:06.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.706+0000 7f1abe000640 1 -- 172.21.15.28:0/3062004815 >> v1:172.21.15.81:6789/0 conn(0x7f1ac00c2c10 legacy=0x7f1ac006ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:06.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.706+0000 7f1abe000640 1 -- 172.21.15.28:0/3062004815 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ac006f4a0 con 0x7f1ac0150240 2024-02-21T23:09:06.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.706+0000 7f1abe000640 1 -- 172.21.15.28:0/3062004815 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f1ab0001670 con 0x7f1ac0150240 2024-02-21T23:09:06.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.706+0000 7f1abe000640 1 -- 172.21.15.28:0/3062004815 >> v1:172.21.15.28:6789/0 conn(0x7f1ac0150240 legacy=0x7f1ac006db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:06.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.706+0000 7f1abe000640 1 --2- 172.21.15.28:0/3062004815 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1ac00c4580 0x7f1aa40039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:06.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.706+0000 7f1abe000640 1 --2- 172.21.15.28:0/3062004815 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1aa4003fb0 0x7f1aa4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:06.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.706+0000 7f1abe000640 1 --2- 172.21.15.28:0/3062004815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aa4004d40 0x7f1aa40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:06.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.706+0000 7f1abe000640 1 -- 172.21.15.28:0/3062004815 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa400bb80 con 0x7f1aa4004d40 2024-02-21T23:09:06.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.706+0000 7f1abe000640 1 -- 172.21.15.28:0/3062004815 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa400bd30 con 0x7f1aa4003fb0 2024-02-21T23:09:06.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.706+0000 7f1abe000640 1 -- 172.21.15.28:0/3062004815 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa400bf30 con 0x7f1ac00c4580 2024-02-21T23:09:06.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.706+0000 7f1abfe00640 1 --2- 172.21.15.28:0/3062004815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aa4004d40 0x7f1aa40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:06.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.707+0000 7f1abea00640 1 --2- 172.21.15.28:0/3062004815 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1aa4003fb0 0x7f1aa4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:06.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.707+0000 7f1abf400640 1 --2- 172.21.15.28:0/3062004815 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1ac00c4580 0x7f1aa40039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:06.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.707+0000 7f1abfe00640 1 -- 172.21.15.28:0/3062004815 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1ac00c4580 msgr2=0x7f1aa40039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:06.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.707+0000 7f1abfe00640 1 --2- 172.21.15.28:0/3062004815 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1ac00c4580 0x7f1aa40039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:06.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.707+0000 7f1abfe00640 1 -- 172.21.15.28:0/3062004815 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1aa4003fb0 msgr2=0x7f1aa4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:06.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.707+0000 7f1abfe00640 1 --2- 172.21.15.28:0/3062004815 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1aa4003fb0 0x7f1aa4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:06.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.707+0000 7f1abfe00640 1 -- 172.21.15.28:0/3062004815 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aa4003520 con 0x7f1aa4004d40 2024-02-21T23:09:06.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.707+0000 7f1abea00640 1 --2- 172.21.15.28:0/3062004815 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1aa4003fb0 0x7f1aa4004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:09:06.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.708+0000 7f1abfe00640 1 --2- 172.21.15.28:0/3062004815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aa4004d40 0x7f1aa40051c0 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f1ab8001e50 tx=0x7f1aa4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=918f5cf37003d9bd server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:06.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.708+0000 7f1abe000640 1 -- 172.21.15.28:0/3062004815 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab8001a70 con 0x7f1aa4004d40 2024-02-21T23:09:06.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.708+0000 7f1abe000640 1 -- 172.21.15.28:0/3062004815 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ab8003b70 con 0x7f1aa4004d40 2024-02-21T23:09:06.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.708+0000 7f1abe000640 1 -- 172.21.15.28:0/3062004815 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab80050d0 con 0x7f1aa4004d40 2024-02-21T23:09:06.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.709+0000 7f1ac4c00640 1 -- 172.21.15.28:0/3062004815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aa4004d40 msgr2=0x7f1aa40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:06.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.709+0000 7f1ac4c00640 1 --2- 172.21.15.28:0/3062004815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aa4004d40 0x7f1aa40051c0 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f1ab8001e50 tx=0x7f1aa4005830 comp rx=0 tx=0).stop 2024-02-21T23:09:06.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.709+0000 7f1abfe00640 1 -- 172.21.15.28:0/3062004815 reap_dead start 2024-02-21T23:09:06.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.709+0000 7f1ac4c00640 1 -- 172.21.15.28:0/3062004815 shutdown_connections 2024-02-21T23:09:06.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.709+0000 7f1ac4c00640 1 -- 172.21.15.28:0/3062004815 >> 172.21.15.28:0/3062004815 conn(0x7f1ac00a8530 msgr2=0x7f1ac00a8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:06.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.709+0000 7f1ac4c00640 1 -- 172.21.15.28:0/3062004815 shutdown_connections 2024-02-21T23:09:06.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.709+0000 7f1ac4c00640 1 -- 172.21.15.28:0/3062004815 wait complete. 2024-02-21T23:09:06.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.711+0000 7f1ac4c00640 1 Processor -- start 2024-02-21T23:09:06.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.711+0000 7f1ac4c00640 1 -- start start 2024-02-21T23:09:06.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.712+0000 7f1ac4c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1aa4004d40 0x7f1ac00aba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:06.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.712+0000 7f1ac4c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ac01dc660 0x7f1ac01dd4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:06.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.712+0000 7f1ac4c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ac01dda30 0x7f1ac01ea140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:06.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.712+0000 7f1ac4c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac00c4a10 con 0x7f1ac01dc660 2024-02-21T23:09:06.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.712+0000 7f1ac4c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac0153a90 con 0x7f1ac01dda30 2024-02-21T23:09:06.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.712+0000 7f1ac4c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac00c4b90 con 0x7f1aa4004d40 2024-02-21T23:09:06.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.712+0000 7f1abea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ac01dc660 0x7f1ac01dd4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:06.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.712+0000 7f1abea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ac01dc660 0x7f1ac01dd4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57044/0 (socket says 172.21.15.28:57044) 2024-02-21T23:09:06.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.712+0000 7f1abea00640 1 -- 172.21.15.28:0/2721261601 learned_addr learned my addr 172.21.15.28:0/2721261601 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:06.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.712+0000 7f1abf400640 1 --2- 172.21.15.28:0/2721261601 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1aa4004d40 0x7f1ac00aba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:06.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.712+0000 7f1abfe00640 1 --2- 172.21.15.28:0/2721261601 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ac01dda30 0x7f1ac01ea140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:06.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.713+0000 7f1abea00640 1 -- 172.21.15.28:0/2721261601 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1aa4004d40 msgr2=0x7f1ac00aba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:06.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.713+0000 7f1abea00640 1 --2- 172.21.15.28:0/2721261601 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1aa4004d40 0x7f1ac00aba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:06.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.713+0000 7f1abea00640 1 -- 172.21.15.28:0/2721261601 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ac01dda30 msgr2=0x7f1ac01ea140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:06.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.713+0000 7f1abea00640 1 --2- 172.21.15.28:0/2721261601 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ac01dda30 0x7f1ac01ea140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:06.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.713+0000 7f1abea00640 1 -- 172.21.15.28:0/2721261601 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ac014b920 con 0x7f1ac01dc660 2024-02-21T23:09:06.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.713+0000 7f1abea00640 1 --2- 172.21.15.28:0/2721261601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ac01dc660 0x7f1ac01dd4f0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f1aac010880 tx=0x7f1aac010d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:06.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.713+0000 7f1ab7400640 1 -- 172.21.15.28:0/2721261601 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aac00f040 con 0x7f1ac01dc660 2024-02-21T23:09:06.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.713+0000 7f1ab7400640 1 -- 172.21.15.28:0/2721261601 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1aac00eb00 con 0x7f1ac01dc660 2024-02-21T23:09:06.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.713+0000 7f1ab7400640 1 -- 172.21.15.28:0/2721261601 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aac01c400 con 0x7f1ac01dc660 2024-02-21T23:09:06.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.713+0000 7f1ac4c00640 1 -- 172.21.15.28:0/2721261601 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ac0156bb0 con 0x7f1ac01dc660 2024-02-21T23:09:06.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.713+0000 7f1ac4c00640 1 -- 172.21.15.28:0/2721261601 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ac00abf80 con 0x7f1ac01dc660 2024-02-21T23:09:06.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.715+0000 7f1ac4c00640 1 -- 172.21.15.28:0/2721261601 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ac006c000 con 0x7f1ac01dc660 2024-02-21T23:09:06.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.715+0000 7f1ab7400640 1 -- 172.21.15.28:0/2721261601 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1aac01c5a0 con 0x7f1ac01dc660 2024-02-21T23:09:06.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.717+0000 7f1ab7400640 1 --2- 172.21.15.28:0/2721261601 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1aa006b4d0 0x7f1aa006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:06.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.717+0000 7f1ab7400640 1 -- 172.21.15.28:0/2721261601 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1aac08aab0 con 0x7f1ac01dc660 2024-02-21T23:09:06.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.718+0000 7f1abf400640 1 --2- 172.21.15.28:0/2721261601 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1aa006b4d0 0x7f1aa006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:06.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.719+0000 7f1abf400640 1 --2- 172.21.15.28:0/2721261601 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1aa006b4d0 0x7f1aa006d990 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f1ab0001050 tx=0x7f1ab0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:06.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:06.721+0000 7f1ab7400640 1 -- 172.21.15.28:0/2721261601 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1aac018030 con 0x7f1ac01dc660 2024-02-21T23:09:06.917 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-21T23:09:06.925 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-21T23:09:07.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.199+0000 7f1ac4c00640 1 -- 172.21.15.28:0/2721261601 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f1ac00acb10 con 0x7f1ac01dc660 2024-02-21T23:09:07.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.454+0000 7f1ab7400640 1 -- 172.21.15.28:0/2721261601 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v139) v1 ==== 122+0+0 (secure 0 0 0) 0x7f1aac059140 con 0x7f1ac01dc660 2024-02-21T23:09:07.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.458+0000 7f1ac4c00640 1 -- 172.21.15.28:0/2721261601 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1aa006b4d0 msgr2=0x7f1aa006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:07.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.458+0000 7f1ac4c00640 1 --2- 172.21.15.28:0/2721261601 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1aa006b4d0 0x7f1aa006d990 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f1ab0001050 tx=0x7f1ab0009040 comp rx=0 tx=0).stop 2024-02-21T23:09:07.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.458+0000 7f1ac4c00640 1 -- 172.21.15.28:0/2721261601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ac01dc660 msgr2=0x7f1ac01dd4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:07.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.458+0000 7f1ac4c00640 1 --2- 172.21.15.28:0/2721261601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ac01dc660 0x7f1ac01dd4f0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f1aac010880 tx=0x7f1aac010d40 comp rx=0 tx=0).stop 2024-02-21T23:09:07.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.458+0000 7f1ac4c00640 1 -- 172.21.15.28:0/2721261601 shutdown_connections 2024-02-21T23:09:07.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.458+0000 7f1ac4c00640 1 --2- 172.21.15.28:0/2721261601 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1aa006b4d0 0x7f1aa006d990 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:07.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.458+0000 7f1ac4c00640 1 --2- 172.21.15.28:0/2721261601 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ac01dda30 0x7f1ac01ea140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:07.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.458+0000 7f1ac4c00640 1 --2- 172.21.15.28:0/2721261601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ac01dc660 0x7f1ac01dd4f0 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:07.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.458+0000 7f1ac4c00640 1 --2- 172.21.15.28:0/2721261601 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1aa4004d40 0x7f1ac00aba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:07.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.458+0000 7f1ac4c00640 1 -- 172.21.15.28:0/2721261601 >> 172.21.15.28:0/2721261601 conn(0x7f1ac00a8530 msgr2=0x7f1ac0074360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:07.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.458+0000 7f1ac4c00640 1 -- 172.21.15.28:0/2721261601 shutdown_connections 2024-02-21T23:09:07.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.458+0000 7f1ac4c00640 1 -- 172.21.15.28:0/2721261601 wait complete. 2024-02-21T23:09:07.471 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-02-21T23:09:07.471 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:09:07.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.691+0000 7f6f2e800640 1 Processor -- start 2024-02-21T23:09:07.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.692+0000 7f6f2e800640 1 -- start start 2024-02-21T23:09:07.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.693+0000 7f6f2e800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f6f2806dff0 0x7f6f2806e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:07.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.693+0000 7f6f2e800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6f280c0fe0 0x7f6f28071350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:07.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.693+0000 7f6f2e800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6f28158580 0x7f6f2814b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:07.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.693+0000 7f6f2e800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f280a8b40 con 0x7f6f28158580 2024-02-21T23:09:07.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.693+0000 7f6f2e800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f280a8e00 con 0x7f6f2806dff0 2024-02-21T23:09:07.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.693+0000 7f6f2e800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f280a90e0 con 0x7f6f280c0fe0 2024-02-21T23:09:07.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.694+0000 7f6f2de00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6f28158580 0x7f6f2814b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:07.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.694+0000 7f6f2de00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6f28158580 0x7f6f2814b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57046/0 (socket says 172.21.15.28:57046) 2024-02-21T23:09:07.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.694+0000 7f6f2de00640 1 -- 172.21.15.28:0/1193974482 learned_addr learned my addr 172.21.15.28:0/1193974482 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:07.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.694+0000 7f6f2ca00640 1 --2- 172.21.15.28:0/1193974482 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6f280c0fe0 0x7f6f28071350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:07.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.694+0000 7f6f2d400640 1 --2- 172.21.15.28:0/1193974482 >> v2:172.21.15.81:3300/0 conn(0x7f6f2806dff0 0x7f6f2806e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:07.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.694+0000 7f6f2de00640 1 -- 172.21.15.28:0/1193974482 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6f280c0fe0 msgr2=0x7f6f28071350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:07.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.694+0000 7f6f2de00640 1 --2- 172.21.15.28:0/1193974482 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6f280c0fe0 0x7f6f28071350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:07.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.695+0000 7f6f2de00640 1 -- 172.21.15.28:0/1193974482 >> v2:172.21.15.81:3300/0 conn(0x7f6f2806dff0 msgr2=0x7f6f2806e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:07.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.695+0000 7f6f2de00640 1 --2- 172.21.15.28:0/1193974482 >> v2:172.21.15.81:3300/0 conn(0x7f6f2806dff0 0x7f6f2806e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:07.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.695+0000 7f6f2de00640 1 -- 172.21.15.28:0/1193974482 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f28072ed0 con 0x7f6f28158580 2024-02-21T23:09:07.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.696+0000 7f6f2de00640 1 --2- 172.21.15.28:0/1193974482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f28158580 0x7f6f2814b2b0 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f6f2806be50 tx=0x7f6f2002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed10d0692185bea8 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:07.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.696+0000 7f6f27e00640 1 -- 172.21.15.28:0/1193974482 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f2002ca70 con 0x7f6f28158580 2024-02-21T23:09:07.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.696+0000 7f6f27e00640 1 -- 172.21.15.28:0/1193974482 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f2002cc10 con 0x7f6f28158580 2024-02-21T23:09:07.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.696+0000 7f6f27e00640 1 -- 172.21.15.28:0/1193974482 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f200343f0 con 0x7f6f28158580 2024-02-21T23:09:07.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.697+0000 7f6f2e800640 1 -- 172.21.15.28:0/1193974482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f28158580 msgr2=0x7f6f2814b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:07.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.697+0000 7f6f2e800640 1 --2- 172.21.15.28:0/1193974482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f28158580 0x7f6f2814b2b0 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f6f2806be50 tx=0x7f6f2002bed0 comp rx=0 tx=0).stop 2024-02-21T23:09:07.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.697+0000 7f6f2e800640 1 -- 172.21.15.28:0/1193974482 shutdown_connections 2024-02-21T23:09:07.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.697+0000 7f6f2e800640 1 --2- 172.21.15.28:0/1193974482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f28158580 0x7f6f2814b2b0 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:07.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.697+0000 7f6f2e800640 1 --2- 172.21.15.28:0/1193974482 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6f280c0fe0 0x7f6f28071350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:07.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.697+0000 7f6f2e800640 1 --2- 172.21.15.28:0/1193974482 >> v2:172.21.15.81:3300/0 conn(0x7f6f2806dff0 0x7f6f2806e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:07.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.697+0000 7f6f2e800640 1 -- 172.21.15.28:0/1193974482 >> 172.21.15.28:0/1193974482 conn(0x7f6f2806b630 msgr2=0x7f6f280c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:07.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.697+0000 7f6f2e800640 1 -- 172.21.15.28:0/1193974482 shutdown_connections 2024-02-21T23:09:07.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.697+0000 7f6f2e800640 1 -- 172.21.15.28:0/1193974482 wait complete. 2024-02-21T23:09:07.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.699+0000 7f6f2e800640 1 Processor -- start 2024-02-21T23:09:07.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.699+0000 7f6f2e800640 1 -- start start 2024-02-21T23:09:07.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.700+0000 7f6f2e800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f2806dff0 0x7f6f28151f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:07.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.700+0000 7f6f2e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f280c0fe0 0x7f6f28152480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:07.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.700+0000 7f6f2e800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6f28158580 0x7f6f281eabe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:07.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.700+0000 7f6f2e800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f28071890 con 0x7f6f280c0fe0 2024-02-21T23:09:07.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.700+0000 7f6f2e800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f28073520 con 0x7f6f2806dff0 2024-02-21T23:09:07.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.700+0000 7f6f2e800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f28158a10 con 0x7f6f28158580 2024-02-21T23:09:07.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.700+0000 7f6f2ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f280c0fe0 0x7f6f28152480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:07.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.701+0000 7f6f2ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f280c0fe0 0x7f6f28152480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57050/0 (socket says 172.21.15.28:57050) 2024-02-21T23:09:07.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.701+0000 7f6f2ca00640 1 -- 172.21.15.28:0/2234149200 learned_addr learned my addr 172.21.15.28:0/2234149200 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:07.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.701+0000 7f6f2d400640 1 --2- 172.21.15.28:0/2234149200 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f2806dff0 0x7f6f28151f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:07.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.701+0000 7f6f2de00640 1 --2- 172.21.15.28:0/2234149200 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6f28158580 0x7f6f281eabe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:07.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.701+0000 7f6f2ca00640 1 -- 172.21.15.28:0/2234149200 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6f28158580 msgr2=0x7f6f281eabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:07.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.701+0000 7f6f2ca00640 1 --2- 172.21.15.28:0/2234149200 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6f28158580 0x7f6f281eabe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:07.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.701+0000 7f6f2ca00640 1 -- 172.21.15.28:0/2234149200 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f2806dff0 msgr2=0x7f6f28151f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:07.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.701+0000 7f6f2ca00640 1 --2- 172.21.15.28:0/2234149200 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f2806dff0 0x7f6f28151f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:07.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.701+0000 7f6f2ca00640 1 -- 172.21.15.28:0/2234149200 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f28158220 con 0x7f6f280c0fe0 2024-02-21T23:09:07.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.702+0000 7f6f2ca00640 1 --2- 172.21.15.28:0/2234149200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f280c0fe0 0x7f6f28152480 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f6f1400d830 tx=0x7f6f1400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:07.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.702+0000 7f6f25600640 1 -- 172.21.15.28:0/2234149200 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f1400b6c0 con 0x7f6f280c0fe0 2024-02-21T23:09:07.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.702+0000 7f6f25600640 1 -- 172.21.15.28:0/2234149200 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f14019070 con 0x7f6f280c0fe0 2024-02-21T23:09:07.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.702+0000 7f6f25600640 1 -- 172.21.15.28:0/2234149200 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f1400ba80 con 0x7f6f280c0fe0 2024-02-21T23:09:07.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.702+0000 7f6f2e800640 1 -- 172.21.15.28:0/2234149200 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f2814f5c0 con 0x7f6f280c0fe0 2024-02-21T23:09:07.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.702+0000 7f6f2e800640 1 -- 172.21.15.28:0/2234149200 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f280abfc0 con 0x7f6f280c0fe0 2024-02-21T23:09:07.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.703+0000 7f6f2e800640 1 -- 172.21.15.28:0/2234149200 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f280c5100 con 0x7f6f280c0fe0 2024-02-21T23:09:07.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.704+0000 7f6f25600640 1 -- 172.21.15.28:0/2234149200 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6f1401e460 con 0x7f6f280c0fe0 2024-02-21T23:09:07.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.705+0000 7f6f25600640 1 --2- 172.21.15.28:0/2234149200 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6f0806b500 0x7f6f0806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:07.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.705+0000 7f6f25600640 1 -- 172.21.15.28:0/2234149200 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6f1408aa40 con 0x7f6f280c0fe0 2024-02-21T23:09:07.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.706+0000 7f6f2d400640 1 --2- 172.21.15.28:0/2234149200 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6f0806b500 0x7f6f0806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:07.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.707+0000 7f6f2d400640 1 --2- 172.21.15.28:0/2234149200 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6f0806b500 0x7f6f0806d9c0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f6f1c006fd0 tx=0x7f6f1c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:07.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.710+0000 7f6f25600640 1 -- 172.21.15.28:0/2234149200 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f6f280c5100 con 0x7f6f280c0fe0 2024-02-21T23:09:07.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.949+0000 7f6f2e800640 1 -- 172.21.15.28:0/2234149200 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6f2804f290 con 0x7f6f280c0fe0 2024-02-21T23:09:07.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.950+0000 7f6f25600640 1 -- 172.21.15.28:0/2234149200 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 139 v139) v1 ==== 95+0+4948 (secure 0 0 0) 0x7f6f14059720 con 0x7f6f280c0fe0 2024-02-21T23:09:07.940 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:09:07.940 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":139,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6140,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/4241906475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":4241906475},{"type":"v1","addr":"172.21.15.81:6836","nonce":4241906475}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":133},{"gid":6143,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/2095908433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":2095908433},{"type":"v1","addr":"172.21.15.81:6837","nonce":2095908433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":133},{"gid":8370,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/3934548735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3934548735},{"type":"v1","addr":"172.21.15.28:6833","nonce":3934548735}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":133}],"filesystems":[{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:09:03.869294+0000","modified":"2024-02-21T23:09:07.201932+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8376},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8376":{"gid":8376,"name":"a","rank":0,"incarnation":135,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/4151254007","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4151254007},{"type":"v1","addr":"172.21.15.28:6835","nonce":4151254007}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-02-21T23:09:07.941 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 139 2024-02-21T23:09:07.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.952+0000 7f6f2e800640 1 -- 172.21.15.28:0/2234149200 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6f0806b500 msgr2=0x7f6f0806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:07.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.952+0000 7f6f2e800640 1 --2- 172.21.15.28:0/2234149200 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6f0806b500 0x7f6f0806d9c0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f6f1c006fd0 tx=0x7f6f1c008040 comp rx=0 tx=0).stop 2024-02-21T23:09:07.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.952+0000 7f6f2e800640 1 -- 172.21.15.28:0/2234149200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f280c0fe0 msgr2=0x7f6f28152480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:07.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.952+0000 7f6f2e800640 1 --2- 172.21.15.28:0/2234149200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f280c0fe0 0x7f6f28152480 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f6f1400d830 tx=0x7f6f1400dcf0 comp rx=0 tx=0).stop 2024-02-21T23:09:07.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.953+0000 7f6f2e800640 1 -- 172.21.15.28:0/2234149200 shutdown_connections 2024-02-21T23:09:07.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.953+0000 7f6f2e800640 1 --2- 172.21.15.28:0/2234149200 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6f0806b500 0x7f6f0806d9c0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:07.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.953+0000 7f6f2e800640 1 --2- 172.21.15.28:0/2234149200 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6f28158580 0x7f6f281eabe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:07.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.953+0000 7f6f2e800640 1 --2- 172.21.15.28:0/2234149200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f280c0fe0 0x7f6f28152480 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:07.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.953+0000 7f6f2e800640 1 --2- 172.21.15.28:0/2234149200 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f2806dff0 0x7f6f28151f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:07.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.953+0000 7f6f2e800640 1 -- 172.21.15.28:0/2234149200 >> 172.21.15.28:0/2234149200 conn(0x7f6f2806b630 msgr2=0x7f6f280c2600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:07.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.953+0000 7f6f2e800640 1 -- 172.21.15.28:0/2234149200 shutdown_connections 2024-02-21T23:09:07.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:07.953+0000 7f6f2e800640 1 -- 172.21.15.28:0/2234149200 wait complete. 2024-02-21T23:09:07.960 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:09:08.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.208+0000 7feddb200640 1 Processor -- start 2024-02-21T23:09:08.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.208+0000 7feddb200640 1 -- start start 2024-02-21T23:09:08.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.209+0000 7feddb200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7feddc0aff90 0x7feddc0c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:08.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.209+0000 7feddb200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feddc0c5720 0x7feddc15f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:08.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.209+0000 7feddb200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feddc0c74c0 con 0x7feddc06dbf0 2024-02-21T23:09:08.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.209+0000 7feddb200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feddc0c7270 con 0x7feddc0aff90 2024-02-21T23:09:08.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.209+0000 7feddb200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feddc158630 con 0x7feddc0c5720 2024-02-21T23:09:08.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.209+0000 7fedd9400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7feddc06dbf0 0x7feddc158ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:56868/0 (socket says 172.21.15.28:56868) 2024-02-21T23:09:08.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.209+0000 7fedd9400640 1 -- 172.21.15.28:0/4061715601 learned_addr learned my addr 172.21.15.28:0/4061715601 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:09:08.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.210+0000 7fedda800640 1 --2- 172.21.15.28:0/4061715601 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feddc0c5720 0x7feddc15f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:08.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.210+0000 7fedd9e00640 1 --2- 172.21.15.28:0/4061715601 >> v2:172.21.15.81:3300/0 conn(0x7feddc0aff90 0x7feddc0c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:08.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.210+0000 7fedd8a00640 1 -- 172.21.15.28:0/4061715601 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4173656764 0 0) 0x7feddc0c74c0 con 0x7feddc06dbf0 2024-02-21T23:09:08.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.210+0000 7fedd8a00640 1 -- 172.21.15.28:0/4061715601 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedc0003520 con 0x7feddc06dbf0 2024-02-21T23:09:08.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.210+0000 7fedd8a00640 1 -- 172.21.15.28:0/4061715601 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1027953874 0 0) 0x7fedc0003520 con 0x7feddc06dbf0 2024-02-21T23:09:08.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.210+0000 7fedd8a00640 1 -- 172.21.15.28:0/4061715601 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feddc0c5720 msgr2=0x7feddc15f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:08.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.210+0000 7fedd8a00640 1 --2- 172.21.15.28:0/4061715601 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feddc0c5720 0x7feddc15f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:08.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.210+0000 7fedd8a00640 1 -- 172.21.15.28:0/4061715601 >> v2:172.21.15.81:3300/0 conn(0x7feddc0aff90 msgr2=0x7feddc0c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:08.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.210+0000 7fedd8a00640 1 --2- 172.21.15.28:0/4061715601 >> v2:172.21.15.81:3300/0 conn(0x7feddc0aff90 0x7feddc0c51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:08.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.210+0000 7fedd8a00640 1 -- 172.21.15.28:0/4061715601 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feddc0c77a0 con 0x7feddc06dbf0 2024-02-21T23:09:08.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.210+0000 7fedd8a00640 1 -- 172.21.15.28:0/4061715601 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fedc8001670 con 0x7feddc06dbf0 2024-02-21T23:09:08.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.211+0000 7fedd8a00640 1 -- 172.21.15.28:0/4061715601 >> v1:172.21.15.28:6789/0 conn(0x7feddc06dbf0 legacy=0x7feddc158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:08.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.211+0000 7fedd8a00640 1 --2- 172.21.15.28:0/4061715601 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fedc00039b0 0x7fedc0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:08.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.211+0000 7fedd8a00640 1 --2- 172.21.15.28:0/4061715601 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feddc0c5720 0x7fedc0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:08.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.211+0000 7fedd8a00640 1 --2- 172.21.15.28:0/4061715601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fedc0004d40 0x7fedc00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:08.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.211+0000 7fedd8a00640 1 -- 172.21.15.28:0/4061715601 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feddc0c7270 con 0x7fedc0004d40 2024-02-21T23:09:08.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.211+0000 7fedd8a00640 1 -- 172.21.15.28:0/4061715601 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feddc158630 con 0x7fedc00039b0 2024-02-21T23:09:08.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.211+0000 7fedd8a00640 1 -- 172.21.15.28:0/4061715601 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedc000b9c0 con 0x7feddc0c5720 2024-02-21T23:09:08.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.211+0000 7fedda800640 1 --2- 172.21.15.28:0/4061715601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fedc0004d40 0x7fedc00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:08.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.211+0000 7fedd9400640 1 --2- 172.21.15.28:0/4061715601 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feddc0c5720 0x7fedc0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:08.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.211+0000 7fedd9e00640 1 --2- 172.21.15.28:0/4061715601 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fedc00039b0 0x7fedc0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:08.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.211+0000 7fedda800640 1 -- 172.21.15.28:0/4061715601 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feddc0c5720 msgr2=0x7fedc0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:08.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.212+0000 7fedda800640 1 --2- 172.21.15.28:0/4061715601 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feddc0c5720 0x7fedc0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:08.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.212+0000 7fedda800640 1 -- 172.21.15.28:0/4061715601 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fedc00039b0 msgr2=0x7fedc0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:08.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.212+0000 7fedda800640 1 --2- 172.21.15.28:0/4061715601 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fedc00039b0 0x7fedc0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:08.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.212+0000 7fedda800640 1 -- 172.21.15.28:0/4061715601 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedc0003520 con 0x7fedc0004d40 2024-02-21T23:09:08.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.212+0000 7fedd9400640 1 --2- 172.21.15.28:0/4061715601 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feddc0c5720 0x7fedc0004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:09:08.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.212+0000 7fedd9e00640 1 --2- 172.21.15.28:0/4061715601 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fedc00039b0 0x7fedc0003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:09:08.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.213+0000 7fedda800640 1 --2- 172.21.15.28:0/4061715601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fedc0004d40 0x7fedc00051c0 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7fedd4009ac0 tx=0x7fedc0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85e96229a2b3f166 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:08.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.214+0000 7fedd8a00640 1 -- 172.21.15.28:0/4061715601 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd4039070 con 0x7fedc0004d40 2024-02-21T23:09:08.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.214+0000 7fedd8a00640 1 -- 172.21.15.28:0/4061715601 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedd402c6b0 con 0x7fedc0004d40 2024-02-21T23:09:08.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.214+0000 7fedd8a00640 1 -- 172.21.15.28:0/4061715601 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd4009690 con 0x7fedc0004d40 2024-02-21T23:09:08.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.214+0000 7feddb200640 1 -- 172.21.15.28:0/4061715601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fedc0004d40 msgr2=0x7fedc00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:08.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.214+0000 7feddb200640 1 --2- 172.21.15.28:0/4061715601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fedc0004d40 0x7fedc00051c0 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7fedd4009ac0 tx=0x7fedc0005830 comp rx=0 tx=0).stop 2024-02-21T23:09:08.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.214+0000 7fedda800640 1 -- 172.21.15.28:0/4061715601 reap_dead start 2024-02-21T23:09:08.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.214+0000 7feddb200640 1 -- 172.21.15.28:0/4061715601 shutdown_connections 2024-02-21T23:09:08.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.214+0000 7feddb200640 1 -- 172.21.15.28:0/4061715601 >> 172.21.15.28:0/4061715601 conn(0x7feddc0ac370 msgr2=0x7feddc0ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:08.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.214+0000 7feddb200640 1 -- 172.21.15.28:0/4061715601 shutdown_connections 2024-02-21T23:09:08.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.214+0000 7feddb200640 1 -- 172.21.15.28:0/4061715601 wait complete. 2024-02-21T23:09:08.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.216+0000 7feddb200640 1 Processor -- start 2024-02-21T23:09:08.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.216+0000 7feddb200640 1 -- start start 2024-02-21T23:09:08.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.217+0000 7feddb200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fedc0004d40 0x7feddc0bd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:08.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.217+0000 7feddb200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feddc0c46f0 0x7feddc0c3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:08.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.217+0000 7feddb200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feddc0be240 0x7feddc0aab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:08.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.217+0000 7feddb200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feddc0c8760 con 0x7feddc0c46f0 2024-02-21T23:09:08.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.217+0000 7feddb200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feddc0be0c0 con 0x7fedc0004d40 2024-02-21T23:09:08.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.217+0000 7feddb200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feddc0c88e0 con 0x7feddc0be240 2024-02-21T23:09:08.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.218+0000 7fedd9400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feddc0c46f0 0x7feddc0c3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:08.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.218+0000 7fedd9400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feddc0c46f0 0x7feddc0c3ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57078/0 (socket says 172.21.15.28:57078) 2024-02-21T23:09:08.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.218+0000 7fedd9400640 1 -- 172.21.15.28:0/4095616453 learned_addr learned my addr 172.21.15.28:0/4095616453 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:08.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.218+0000 7fedd9e00640 1 --2- 172.21.15.28:0/4095616453 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fedc0004d40 0x7feddc0bd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:08.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.218+0000 7fedda800640 1 --2- 172.21.15.28:0/4095616453 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feddc0be240 0x7feddc0aab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:08.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.218+0000 7fedd9400640 1 -- 172.21.15.28:0/4095616453 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feddc0be240 msgr2=0x7feddc0aab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:08.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.218+0000 7fedd9400640 1 --2- 172.21.15.28:0/4095616453 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feddc0be240 0x7feddc0aab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:08.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.218+0000 7fedd9400640 1 -- 172.21.15.28:0/4095616453 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fedc0004d40 msgr2=0x7feddc0bd4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:08.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.218+0000 7fedd9400640 1 --2- 172.21.15.28:0/4095616453 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fedc0004d40 0x7feddc0bd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:08.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.218+0000 7fedd9400640 1 -- 172.21.15.28:0/4095616453 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feddc0c4010 con 0x7feddc0c46f0 2024-02-21T23:09:08.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.218+0000 7fedd9400640 1 --2- 172.21.15.28:0/4095616453 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feddc0c46f0 0x7feddc0c3ad0 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7fedc800e8f0 tx=0x7fedc800edb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:08.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.219+0000 7fedd2000640 1 -- 172.21.15.28:0/4095616453 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedc800c8d0 con 0x7feddc0c46f0 2024-02-21T23:09:08.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.219+0000 7fedd2000640 1 -- 172.21.15.28:0/4095616453 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedc8018070 con 0x7feddc0c46f0 2024-02-21T23:09:08.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.219+0000 7fedd2000640 1 -- 172.21.15.28:0/4095616453 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedc800cc70 con 0x7feddc0c46f0 2024-02-21T23:09:08.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.219+0000 7feddb200640 1 -- 172.21.15.28:0/4095616453 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feddc0bdac0 con 0x7feddc0c46f0 2024-02-21T23:09:08.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.219+0000 7feddb200640 1 -- 172.21.15.28:0/4095616453 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feddc0c12e0 con 0x7feddc0c46f0 2024-02-21T23:09:08.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.220+0000 7feddb200640 1 -- 172.21.15.28:0/4095616453 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feddc04f290 con 0x7feddc0c46f0 2024-02-21T23:09:08.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.222+0000 7fedd2000640 1 -- 172.21.15.28:0/4095616453 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fedc801d460 con 0x7feddc0c46f0 2024-02-21T23:09:08.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.223+0000 7fedd2000640 1 --2- 172.21.15.28:0/4095616453 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fedbc06b4d0 0x7fedbc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:08.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.223+0000 7fedd2000640 1 -- 172.21.15.28:0/4095616453 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fedc808a380 con 0x7feddc0c46f0 2024-02-21T23:09:08.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.223+0000 7fedd9e00640 1 --2- 172.21.15.28:0/4095616453 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fedbc06b4d0 0x7fedbc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:08.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.225+0000 7fedd9e00640 1 --2- 172.21.15.28:0/4095616453 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fedbc06b4d0 0x7fedbc06d990 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fedcc007440 tx=0x7fedcc00d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:08.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.228+0000 7fedd2000640 1 -- 172.21.15.28:0/4095616453 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fedc8059060 con 0x7feddc0c46f0 2024-02-21T23:09:08.341 INFO:tasks.ceph.mds.a.smithi028.stderr:2024-02-21T23:09:08.352+0000 7f8203000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-21T23:09:08.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.434+0000 7feddb200640 1 -- 172.21.15.28:0/4095616453 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feddc06b4a0 con 0x7feddc0c46f0 2024-02-21T23:09:08.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.437+0000 7fedd2000640 1 -- 172.21.15.28:0/4095616453 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v136) v1 ==== 74+0+16832 (secure 0 0 0) 0x7fedc8058960 con 0x7feddc0c46f0 2024-02-21T23:09:08.426 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:09:08.426 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":136,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:09:05.417507+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:09:01.777487+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"135","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2024-02-21T23:09:02.904570+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"136","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:09:08.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.440+0000 7feddb200640 1 -- 172.21.15.28:0/4095616453 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fedbc06b4d0 msgr2=0x7fedbc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:08.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.440+0000 7feddb200640 1 --2- 172.21.15.28:0/4095616453 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fedbc06b4d0 0x7fedbc06d990 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fedcc007440 tx=0x7fedcc00d040 comp rx=0 tx=0).stop 2024-02-21T23:09:08.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.440+0000 7feddb200640 1 -- 172.21.15.28:0/4095616453 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feddc0c46f0 msgr2=0x7feddc0c3ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:08.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.440+0000 7feddb200640 1 --2- 172.21.15.28:0/4095616453 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feddc0c46f0 0x7feddc0c3ad0 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7fedc800e8f0 tx=0x7fedc800edb0 comp rx=0 tx=0).stop 2024-02-21T23:09:08.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.440+0000 7feddb200640 1 -- 172.21.15.28:0/4095616453 shutdown_connections 2024-02-21T23:09:08.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.440+0000 7feddb200640 1 --2- 172.21.15.28:0/4095616453 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fedbc06b4d0 0x7fedbc06d990 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:08.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.440+0000 7feddb200640 1 --2- 172.21.15.28:0/4095616453 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feddc0be240 0x7feddc0aab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:08.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.440+0000 7feddb200640 1 --2- 172.21.15.28:0/4095616453 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feddc0c46f0 0x7feddc0c3ad0 unknown :-1 s=CLOSED pgs=1527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:08.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.440+0000 7feddb200640 1 --2- 172.21.15.28:0/4095616453 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fedc0004d40 0x7feddc0bd4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:08.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.440+0000 7feddb200640 1 -- 172.21.15.28:0/4095616453 >> 172.21.15.28:0/4095616453 conn(0x7feddc0ac370 msgr2=0x7feddc0c8a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:08.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.440+0000 7feddb200640 1 -- 172.21.15.28:0/4095616453 shutdown_connections 2024-02-21T23:09:08.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.440+0000 7feddb200640 1 -- 172.21.15.28:0/4095616453 wait complete. 2024-02-21T23:09:08.450 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-02-21T23:09:08.450 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-21T23:09:08.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.666+0000 7f5842200640 1 Processor -- start 2024-02-21T23:09:08.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.667+0000 7f5842200640 1 -- start start 2024-02-21T23:09:08.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.668+0000 7f5842200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f583c14f240 0x7f583c0c4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:08.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.668+0000 7f5842200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f583c0c36b0 con 0x7f583c155980 2024-02-21T23:09:08.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.668+0000 7f5842200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f583c0c3990 con 0x7f583c151cd0 2024-02-21T23:09:08.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.668+0000 7f5842200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f583c0c50e0 con 0x7f583c14f240 2024-02-21T23:09:08.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.669+0000 7f5841800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f583c155980 0x7f583c15c650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:56870/0 (socket says 172.21.15.28:56870) 2024-02-21T23:09:08.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.669+0000 7f5841800640 1 -- 172.21.15.28:0/2547232337 learned_addr learned my addr 172.21.15.28:0/2547232337 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:09:08.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.669+0000 7f5840e00640 1 --2- 172.21.15.28:0/2547232337 >> v2:172.21.15.81:3300/0 conn(0x7f583c14f240 0x7f583c0c4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:08.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.669+0000 7f583b400640 1 -- 172.21.15.28:0/2547232337 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2042096728 0 0) 0x7f583c0c36b0 con 0x7f583c155980 2024-02-21T23:09:08.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.669+0000 7f583b400640 1 -- 172.21.15.28:0/2547232337 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5824003520 con 0x7f583c155980 2024-02-21T23:09:08.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.669+0000 7f583b400640 1 -- 172.21.15.28:0/2547232337 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 121081548 0 0) 0x7f5824003520 con 0x7f583c155980 2024-02-21T23:09:08.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.670+0000 7f583b400640 1 -- 172.21.15.28:0/2547232337 >> v2:172.21.15.81:3300/0 conn(0x7f583c14f240 msgr2=0x7f583c0c4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:08.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.670+0000 7f583b400640 1 --2- 172.21.15.28:0/2547232337 >> v2:172.21.15.81:3300/0 conn(0x7f583c14f240 0x7f583c0c4ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:08.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.670+0000 7f583b400640 1 -- 172.21.15.28:0/2547232337 >> v1:172.21.15.81:6789/0 conn(0x7f583c151cd0 legacy=0x7f583c0c7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:08.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.670+0000 7f583b400640 1 -- 172.21.15.28:0/2547232337 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f583c0c3c70 con 0x7f583c155980 2024-02-21T23:09:08.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.670+0000 7f583b400640 1 -- 172.21.15.28:0/2547232337 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f58340019f0 con 0x7f583c155980 2024-02-21T23:09:08.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.670+0000 7f583b400640 1 -- 172.21.15.28:0/2547232337 >> v1:172.21.15.28:6789/0 conn(0x7f583c155980 legacy=0x7f583c15c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:08.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.670+0000 7f583b400640 1 --2- 172.21.15.28:0/2547232337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58240039b0 0x7f5824003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:08.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.670+0000 7f583b400640 1 --2- 172.21.15.28:0/2547232337 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5824004800 0x7f5824004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:08.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.670+0000 7f583b400640 1 --2- 172.21.15.28:0/2547232337 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58240051c0 0x7f582400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:08.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.670+0000 7f583b400640 1 -- 172.21.15.28:0/2547232337 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f583c0c50e0 con 0x7f58240039b0 2024-02-21T23:09:08.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.670+0000 7f583b400640 1 -- 172.21.15.28:0/2547232337 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f582400c060 con 0x7f58240051c0 2024-02-21T23:09:08.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.670+0000 7f583b400640 1 -- 172.21.15.28:0/2547232337 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f582400c260 con 0x7f5824004800 2024-02-21T23:09:08.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.670+0000 7f5840e00640 1 --2- 172.21.15.28:0/2547232337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58240039b0 0x7f5824003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:08.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.670+0000 7f5841800640 1 --2- 172.21.15.28:0/2547232337 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58240051c0 0x7f582400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:08.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.670+0000 7f583be00640 1 --2- 172.21.15.28:0/2547232337 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5824004800 0x7f5824004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:08.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.670+0000 7f5840e00640 1 -- 172.21.15.28:0/2547232337 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5824004800 msgr2=0x7f5824004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:08.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.671+0000 7f5840e00640 1 --2- 172.21.15.28:0/2547232337 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5824004800 0x7f5824004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:08.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.671+0000 7f5840e00640 1 -- 172.21.15.28:0/2547232337 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58240051c0 msgr2=0x7f582400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:08.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.671+0000 7f5840e00640 1 --2- 172.21.15.28:0/2547232337 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58240051c0 0x7f582400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:08.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.671+0000 7f5840e00640 1 -- 172.21.15.28:0/2547232337 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5824003520 con 0x7f58240039b0 2024-02-21T23:09:08.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.671+0000 7f583be00640 1 --2- 172.21.15.28:0/2547232337 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5824004800 0x7f5824004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:09:08.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.671+0000 7f5841800640 1 --2- 172.21.15.28:0/2547232337 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58240051c0 0x7f582400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:09:08.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.671+0000 7f5841800640 1 -- 172.21.15.28:0/2547232337 reap_dead start 2024-02-21T23:09:08.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.672+0000 7f5840e00640 1 --2- 172.21.15.28:0/2547232337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58240039b0 0x7f5824003dd0 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f5830009650 tx=0x7f5824005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e95feea48cdcea92 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:08.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.672+0000 7f583b400640 1 -- 172.21.15.28:0/2547232337 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f583003b070 con 0x7f58240039b0 2024-02-21T23:09:08.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.672+0000 7f583b400640 1 -- 172.21.15.28:0/2547232337 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f583002e8c0 con 0x7f58240039b0 2024-02-21T23:09:08.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.672+0000 7f583b400640 1 -- 172.21.15.28:0/2547232337 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5830009690 con 0x7f58240039b0 2024-02-21T23:09:08.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.673+0000 7f5842200640 1 -- 172.21.15.28:0/2547232337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58240039b0 msgr2=0x7f5824003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:08.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.673+0000 7f5842200640 1 --2- 172.21.15.28:0/2547232337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58240039b0 0x7f5824003dd0 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f5830009650 tx=0x7f5824005830 comp rx=0 tx=0).stop 2024-02-21T23:09:08.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.673+0000 7f5842200640 1 -- 172.21.15.28:0/2547232337 shutdown_connections 2024-02-21T23:09:08.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.673+0000 7f5842200640 1 --2- 172.21.15.28:0/2547232337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58240039b0 0x7f5824003dd0 unknown :-1 s=CLOSED pgs=1529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:08.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.673+0000 7f5842200640 1 -- 172.21.15.28:0/2547232337 >> 172.21.15.28:0/2547232337 conn(0x7f583c067050 msgr2=0x7f583c0a84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:08.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.673+0000 7f5842200640 1 -- 172.21.15.28:0/2547232337 shutdown_connections 2024-02-21T23:09:08.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.673+0000 7f5842200640 1 -- 172.21.15.28:0/2547232337 wait complete. 2024-02-21T23:09:08.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.675+0000 7f5842200640 1 Processor -- start 2024-02-21T23:09:08.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.675+0000 7f5842200640 1 -- start start 2024-02-21T23:09:08.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.676+0000 7f5842200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58240039b0 0x7f583c14a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:08.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.676+0000 7f5842200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f583c148530 0x7f583c1e64e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:08.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.676+0000 7f5842200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f583c1e6a20 0x7f583c1472f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:08.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.676+0000 7f5842200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f583c06f2e0 con 0x7f58240039b0 2024-02-21T23:09:08.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.676+0000 7f5842200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f583c14f620 con 0x7f583c1e6a20 2024-02-21T23:09:08.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.676+0000 7f5842200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f583c06f460 con 0x7f583c148530 2024-02-21T23:09:08.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.676+0000 7f5840e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58240039b0 0x7f583c14a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:08.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.676+0000 7f5840e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58240039b0 0x7f583c14a3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57102/0 (socket says 172.21.15.28:57102) 2024-02-21T23:09:08.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.676+0000 7f5840e00640 1 -- 172.21.15.28:0/794556119 learned_addr learned my addr 172.21.15.28:0/794556119 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:08.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.676+0000 7f5841800640 1 --2- 172.21.15.28:0/794556119 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f583c1e6a20 0x7f583c1472f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:08.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.676+0000 7f583be00640 1 --2- 172.21.15.28:0/794556119 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f583c148530 0x7f583c1e64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:08.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.676+0000 7f5840e00640 1 -- 172.21.15.28:0/794556119 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f583c148530 msgr2=0x7f583c1e64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:08.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.676+0000 7f5840e00640 1 --2- 172.21.15.28:0/794556119 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f583c148530 0x7f583c1e64e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:08.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.677+0000 7f5840e00640 1 -- 172.21.15.28:0/794556119 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f583c1e6a20 msgr2=0x7f583c1472f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:08.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.677+0000 7f5840e00640 1 --2- 172.21.15.28:0/794556119 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f583c1e6a20 0x7f583c1472f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:08.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.677+0000 7f5840e00640 1 -- 172.21.15.28:0/794556119 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f583c14fcc0 con 0x7f58240039b0 2024-02-21T23:09:08.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.677+0000 7f583be00640 1 --2- 172.21.15.28:0/794556119 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f583c148530 0x7f583c1e64e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:09:08.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.677+0000 7f5840e00640 1 --2- 172.21.15.28:0/794556119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58240039b0 0x7f583c14a3a0 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f5830009bb0 tx=0x7f5830004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:08.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.677+0000 7f5838c00640 1 -- 172.21.15.28:0/794556119 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f583003b050 con 0x7f58240039b0 2024-02-21T23:09:08.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.677+0000 7f5838c00640 1 -- 172.21.15.28:0/794556119 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5830047070 con 0x7f58240039b0 2024-02-21T23:09:08.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.677+0000 7f5838c00640 1 -- 172.21.15.28:0/794556119 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5830009690 con 0x7f58240039b0 2024-02-21T23:09:08.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.677+0000 7f5842200640 1 -- 172.21.15.28:0/794556119 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f583c1e6ea0 con 0x7f58240039b0 2024-02-21T23:09:08.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.678+0000 7f5842200640 1 -- 172.21.15.28:0/794556119 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f583c1e9a50 con 0x7f58240039b0 2024-02-21T23:09:08.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.679+0000 7f5842200640 1 -- 172.21.15.28:0/794556119 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f583c04f290 con 0x7f58240039b0 2024-02-21T23:09:08.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.681+0000 7f5838c00640 1 -- 172.21.15.28:0/794556119 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5830003a10 con 0x7f58240039b0 2024-02-21T23:09:08.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.682+0000 7f5838c00640 1 --2- 172.21.15.28:0/794556119 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f581c06b500 0x7f581c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:08.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.682+0000 7f5838c00640 1 -- 172.21.15.28:0/794556119 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f58300ae060 con 0x7f58240039b0 2024-02-21T23:09:08.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.683+0000 7f583be00640 1 --2- 172.21.15.28:0/794556119 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f581c06b500 0x7f581c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:08.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.684+0000 7f583be00640 1 --2- 172.21.15.28:0/794556119 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f581c06b500 0x7f581c06d9c0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f583c14ac70 tx=0x7f5828009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:08.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:08.687+0000 7f5838c00640 1 -- 172.21.15.28:0/794556119 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f5830050020 con 0x7f58240039b0 2024-02-21T23:09:08.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.009+0000 7f5842200640 1 -- 172.21.15.28:0/794556119 --> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f583c0ab5c0 con 0x7f581c06b500 2024-02-21T23:09:09.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.027+0000 7f5838c00640 1 -- 172.21.15.28:0/794556119 <== mgr.4101 v2:172.21.15.81:6832/1848538577 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252561 (secure 0 0 0) 0x7f583c0ab5c0 con 0x7f581c06b500 2024-02-21T23:09:09.020 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:09:09.022 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-21T23:09:09.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.032+0000 7f5842200640 1 -- 172.21.15.28:0/794556119 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f581c06b500 msgr2=0x7f581c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:09.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.032+0000 7f5842200640 1 --2- 172.21.15.28:0/794556119 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f581c06b500 0x7f581c06d9c0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f583c14ac70 tx=0x7f5828009040 comp rx=0 tx=0).stop 2024-02-21T23:09:09.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.032+0000 7f5842200640 1 -- 172.21.15.28:0/794556119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58240039b0 msgr2=0x7f583c14a3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:09.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.032+0000 7f5842200640 1 --2- 172.21.15.28:0/794556119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58240039b0 0x7f583c14a3a0 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f5830009bb0 tx=0x7f5830004730 comp rx=0 tx=0).stop 2024-02-21T23:09:09.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.032+0000 7f5842200640 1 -- 172.21.15.28:0/794556119 shutdown_connections 2024-02-21T23:09:09.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.032+0000 7f5842200640 1 --2- 172.21.15.28:0/794556119 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f581c06b500 0x7f581c06d9c0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:09.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.032+0000 7f5842200640 1 --2- 172.21.15.28:0/794556119 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f583c148530 0x7f583c1e64e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:09.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.032+0000 7f5842200640 1 --2- 172.21.15.28:0/794556119 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f583c1e6a20 0x7f583c1472f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:09.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.032+0000 7f5842200640 1 --2- 172.21.15.28:0/794556119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58240039b0 0x7f583c14a3a0 unknown :-1 s=CLOSED pgs=1530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:09.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.032+0000 7f5842200640 1 -- 172.21.15.28:0/794556119 >> 172.21.15.28:0/794556119 conn(0x7f583c067050 msgr2=0x7f583c0a84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:09.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.032+0000 7f5842200640 1 -- 172.21.15.28:0/794556119 shutdown_connections 2024-02-21T23:09:09.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.032+0000 7f5842200640 1 -- 172.21.15.28:0/794556119 wait complete. 2024-02-21T23:09:09.040 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":328,"stamp":"2024-02-21T23:09:08.802188+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":137,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":262948,"kb_used_data":13488,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754711772,"statfs":{"total":773094113280,"available":772824854528,"internally_reserved":0,"allocated":13811712,"data_stored":7807596,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002169"},"pg_stats":[{"pgid":"22.3f","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.449957+0000","last_change":"2024-02-21T23:09:04.447288+0000","last_active":"2024-02-21T23:09:05.449957+0000","last_peered":"2024-02-21T23:09:05.449957+0000","last_clean":"2024-02-21T23:09:05.449957+0000","last_became_active":"2024-02-21T23:09:04.446968+0000","last_became_peered":"2024-02-21T23:09:04.446968+0000","last_unstale":"2024-02-21T23:09:05.449957+0000","last_undegraded":"2024-02-21T23:09:05.449957+0000","last_fullsized":"2024-02-21T23:09:05.449957+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:35:17.242030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.3e","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.433652+0000","last_change":"2024-02-21T23:09:04.417638+0000","last_active":"2024-02-21T23:09:05.433652+0000","last_peered":"2024-02-21T23:09:05.433652+0000","last_clean":"2024-02-21T23:09:05.433652+0000","last_became_active":"2024-02-21T23:09:04.417041+0000","last_became_peered":"2024-02-21T23:09:04.417041+0000","last_unstale":"2024-02-21T23:09:05.433652+0000","last_undegraded":"2024-02-21T23:09:05.433652+0000","last_fullsized":"2024-02-21T23:09:05.433652+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:35:17.242030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3d","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.433825+0000","last_change":"2024-02-21T23:09:04.438506+0000","last_active":"2024-02-21T23:09:05.433825+0000","last_peered":"2024-02-21T23:09:05.433825+0000","last_clean":"2024-02-21T23:09:05.433825+0000","last_became_active":"2024-02-21T23:09:04.438203+0000","last_became_peered":"2024-02-21T23:09:04.438203+0000","last_unstale":"2024-02-21T23:09:05.433825+0000","last_undegraded":"2024-02-21T23:09:05.433825+0000","last_fullsized":"2024-02-21T23:09:05.433825+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:47:57.652729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3c","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.793882+0000","last_change":"2024-02-21T23:09:04.444562+0000","last_active":"2024-02-21T23:09:05.793882+0000","last_peered":"2024-02-21T23:09:05.793882+0000","last_clean":"2024-02-21T23:09:05.793882+0000","last_became_active":"2024-02-21T23:09:04.444227+0000","last_became_peered":"2024-02-21T23:09:04.444227+0000","last_unstale":"2024-02-21T23:09:05.793882+0000","last_undegraded":"2024-02-21T23:09:05.793882+0000","last_fullsized":"2024-02-21T23:09:05.793882+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:20:52.153827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.3b","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.427296+0000","last_change":"2024-02-21T23:09:04.437232+0000","last_active":"2024-02-21T23:09:05.427296+0000","last_peered":"2024-02-21T23:09:05.427296+0000","last_clean":"2024-02-21T23:09:05.427296+0000","last_became_active":"2024-02-21T23:09:04.436960+0000","last_became_peered":"2024-02-21T23:09:04.436960+0000","last_unstale":"2024-02-21T23:09:05.427296+0000","last_undegraded":"2024-02-21T23:09:05.427296+0000","last_fullsized":"2024-02-21T23:09:05.427296+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:46:39.489090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3a","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.426570+0000","last_change":"2024-02-21T23:09:04.437079+0000","last_active":"2024-02-21T23:09:05.426570+0000","last_peered":"2024-02-21T23:09:05.426570+0000","last_clean":"2024-02-21T23:09:05.426570+0000","last_became_active":"2024-02-21T23:09:04.436784+0000","last_became_peered":"2024-02-21T23:09:04.436784+0000","last_unstale":"2024-02-21T23:09:05.426570+0000","last_undegraded":"2024-02-21T23:09:05.426570+0000","last_fullsized":"2024-02-21T23:09:05.426570+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:14:37.913890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.39","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.449289+0000","last_change":"2024-02-21T23:09:04.447811+0000","last_active":"2024-02-21T23:09:05.449289+0000","last_peered":"2024-02-21T23:09:05.449289+0000","last_clean":"2024-02-21T23:09:05.449289+0000","last_became_active":"2024-02-21T23:09:04.444277+0000","last_became_peered":"2024-02-21T23:09:04.444277+0000","last_unstale":"2024-02-21T23:09:05.449289+0000","last_undegraded":"2024-02-21T23:09:05.449289+0000","last_fullsized":"2024-02-21T23:09:05.449289+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:47:47.107317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.38","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.428318+0000","last_change":"2024-02-21T23:09:04.444057+0000","last_active":"2024-02-21T23:09:05.428318+0000","last_peered":"2024-02-21T23:09:05.428318+0000","last_clean":"2024-02-21T23:09:05.428318+0000","last_became_active":"2024-02-21T23:09:04.438783+0000","last_became_peered":"2024-02-21T23:09:04.438783+0000","last_unstale":"2024-02-21T23:09:05.428318+0000","last_undegraded":"2024-02-21T23:09:05.428318+0000","last_fullsized":"2024-02-21T23:09:05.428318+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:41:33.605853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.37","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.426393+0000","last_change":"2024-02-21T23:09:04.437743+0000","last_active":"2024-02-21T23:09:05.426393+0000","last_peered":"2024-02-21T23:09:05.426393+0000","last_clean":"2024-02-21T23:09:05.426393+0000","last_became_active":"2024-02-21T23:09:04.436017+0000","last_became_peered":"2024-02-21T23:09:04.436017+0000","last_unstale":"2024-02-21T23:09:05.426393+0000","last_undegraded":"2024-02-21T23:09:05.426393+0000","last_fullsized":"2024-02-21T23:09:05.426393+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:07:42.091472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.36","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.427100+0000","last_change":"2024-02-21T23:09:04.434833+0000","last_active":"2024-02-21T23:09:05.427100+0000","last_peered":"2024-02-21T23:09:05.427100+0000","last_clean":"2024-02-21T23:09:05.427100+0000","last_became_active":"2024-02-21T23:09:04.434544+0000","last_became_peered":"2024-02-21T23:09:04.434544+0000","last_unstale":"2024-02-21T23:09:05.427100+0000","last_undegraded":"2024-02-21T23:09:05.427100+0000","last_fullsized":"2024-02-21T23:09:05.427100+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:36:53.116212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.35","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.791938+0000","last_change":"2024-02-21T23:09:04.437339+0000","last_active":"2024-02-21T23:09:05.791938+0000","last_peered":"2024-02-21T23:09:05.791938+0000","last_clean":"2024-02-21T23:09:05.791938+0000","last_became_active":"2024-02-21T23:09:04.437003+0000","last_became_peered":"2024-02-21T23:09:04.437003+0000","last_unstale":"2024-02-21T23:09:05.791938+0000","last_undegraded":"2024-02-21T23:09:05.791938+0000","last_fullsized":"2024-02-21T23:09:05.791938+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:16:57.318820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.34","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.794256+0000","last_change":"2024-02-21T23:09:04.429608+0000","last_active":"2024-02-21T23:09:05.794256+0000","last_peered":"2024-02-21T23:09:05.794256+0000","last_clean":"2024-02-21T23:09:05.794256+0000","last_became_active":"2024-02-21T23:09:04.429338+0000","last_became_peered":"2024-02-21T23:09:04.429338+0000","last_unstale":"2024-02-21T23:09:05.794256+0000","last_undegraded":"2024-02-21T23:09:05.794256+0000","last_fullsized":"2024-02-21T23:09:05.794256+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:48:25.691424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.33","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.426826+0000","last_change":"2024-02-21T23:09:04.434802+0000","last_active":"2024-02-21T23:09:05.426826+0000","last_peered":"2024-02-21T23:09:05.426826+0000","last_clean":"2024-02-21T23:09:05.426826+0000","last_became_active":"2024-02-21T23:09:04.434384+0000","last_became_peered":"2024-02-21T23:09:04.434384+0000","last_unstale":"2024-02-21T23:09:05.426826+0000","last_undegraded":"2024-02-21T23:09:05.426826+0000","last_fullsized":"2024-02-21T23:09:05.426826+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:40:12.326060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.32","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.793592+0000","last_change":"2024-02-21T23:09:04.436828+0000","last_active":"2024-02-21T23:09:05.793592+0000","last_peered":"2024-02-21T23:09:05.793592+0000","last_clean":"2024-02-21T23:09:05.793592+0000","last_became_active":"2024-02-21T23:09:04.435612+0000","last_became_peered":"2024-02-21T23:09:04.435612+0000","last_unstale":"2024-02-21T23:09:05.793592+0000","last_undegraded":"2024-02-21T23:09:05.793592+0000","last_fullsized":"2024-02-21T23:09:05.793592+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:03:15.998238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.31","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.432853+0000","last_change":"2024-02-21T23:09:04.417330+0000","last_active":"2024-02-21T23:09:05.432853+0000","last_peered":"2024-02-21T23:09:05.432853+0000","last_clean":"2024-02-21T23:09:05.432853+0000","last_became_active":"2024-02-21T23:09:04.416756+0000","last_became_peered":"2024-02-21T23:09:04.416756+0000","last_unstale":"2024-02-21T23:09:05.432853+0000","last_undegraded":"2024-02-21T23:09:05.432853+0000","last_fullsized":"2024-02-21T23:09:05.432853+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:38:43.748694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.30","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.426099+0000","last_change":"2024-02-21T23:09:04.443913+0000","last_active":"2024-02-21T23:09:05.426099+0000","last_peered":"2024-02-21T23:09:05.426099+0000","last_clean":"2024-02-21T23:09:05.426099+0000","last_became_active":"2024-02-21T23:09:04.443644+0000","last_became_peered":"2024-02-21T23:09:04.443644+0000","last_unstale":"2024-02-21T23:09:05.426099+0000","last_undegraded":"2024-02-21T23:09:05.426099+0000","last_fullsized":"2024-02-21T23:09:05.426099+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:03:00.364236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.2f","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.794201+0000","last_change":"2024-02-21T23:09:04.433260+0000","last_active":"2024-02-21T23:09:05.794201+0000","last_peered":"2024-02-21T23:09:05.794201+0000","last_clean":"2024-02-21T23:09:05.794201+0000","last_became_active":"2024-02-21T23:09:04.432980+0000","last_became_peered":"2024-02-21T23:09:04.432980+0000","last_unstale":"2024-02-21T23:09:05.794201+0000","last_undegraded":"2024-02-21T23:09:05.794201+0000","last_fullsized":"2024-02-21T23:09:05.794201+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:43:38.434476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.2e","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.791833+0000","last_change":"2024-02-21T23:09:04.450702+0000","last_active":"2024-02-21T23:09:05.791833+0000","last_peered":"2024-02-21T23:09:05.791833+0000","last_clean":"2024-02-21T23:09:05.791833+0000","last_became_active":"2024-02-21T23:09:04.449214+0000","last_became_peered":"2024-02-21T23:09:04.449214+0000","last_unstale":"2024-02-21T23:09:05.791833+0000","last_undegraded":"2024-02-21T23:09:05.791833+0000","last_fullsized":"2024-02-21T23:09:05.791833+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:49:25.394550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2d","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.456320+0000","last_change":"2024-02-21T23:09:04.443854+0000","last_active":"2024-02-21T23:09:05.456320+0000","last_peered":"2024-02-21T23:09:05.456320+0000","last_clean":"2024-02-21T23:09:05.456320+0000","last_became_active":"2024-02-21T23:09:04.443566+0000","last_became_peered":"2024-02-21T23:09:04.443566+0000","last_unstale":"2024-02-21T23:09:05.456320+0000","last_undegraded":"2024-02-21T23:09:05.456320+0000","last_fullsized":"2024-02-21T23:09:05.456320+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:41:19.469926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.2c","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.425454+0000","last_change":"2024-02-21T23:09:04.443122+0000","last_active":"2024-02-21T23:09:05.425454+0000","last_peered":"2024-02-21T23:09:05.425454+0000","last_clean":"2024-02-21T23:09:05.425454+0000","last_became_active":"2024-02-21T23:09:04.440509+0000","last_became_peered":"2024-02-21T23:09:04.440509+0000","last_unstale":"2024-02-21T23:09:05.425454+0000","last_undegraded":"2024-02-21T23:09:05.425454+0000","last_fullsized":"2024-02-21T23:09:05.425454+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:47:17.508603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.2b","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.427600+0000","last_change":"2024-02-21T23:09:04.442554+0000","last_active":"2024-02-21T23:09:05.427600+0000","last_peered":"2024-02-21T23:09:05.427600+0000","last_clean":"2024-02-21T23:09:05.427600+0000","last_became_active":"2024-02-21T23:09:04.442201+0000","last_became_peered":"2024-02-21T23:09:04.442201+0000","last_unstale":"2024-02-21T23:09:05.427600+0000","last_undegraded":"2024-02-21T23:09:05.427600+0000","last_fullsized":"2024-02-21T23:09:05.427600+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:51:53.596744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.2a","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.449694+0000","last_change":"2024-02-21T23:09:04.445475+0000","last_active":"2024-02-21T23:09:05.449694+0000","last_peered":"2024-02-21T23:09:05.449694+0000","last_clean":"2024-02-21T23:09:05.449694+0000","last_became_active":"2024-02-21T23:09:04.445131+0000","last_became_peered":"2024-02-21T23:09:04.445131+0000","last_unstale":"2024-02-21T23:09:05.449694+0000","last_undegraded":"2024-02-21T23:09:05.449694+0000","last_fullsized":"2024-02-21T23:09:05.449694+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:39:05.616018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.29","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.428292+0000","last_change":"2024-02-21T23:09:04.442615+0000","last_active":"2024-02-21T23:09:05.428292+0000","last_peered":"2024-02-21T23:09:05.428292+0000","last_clean":"2024-02-21T23:09:05.428292+0000","last_became_active":"2024-02-21T23:09:04.439182+0000","last_became_peered":"2024-02-21T23:09:04.439182+0000","last_unstale":"2024-02-21T23:09:05.428292+0000","last_undegraded":"2024-02-21T23:09:05.428292+0000","last_fullsized":"2024-02-21T23:09:05.428292+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:49:59.061685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.28","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.791848+0000","last_change":"2024-02-21T23:09:04.432498+0000","last_active":"2024-02-21T23:09:05.791848+0000","last_peered":"2024-02-21T23:09:05.791848+0000","last_clean":"2024-02-21T23:09:05.791848+0000","last_became_active":"2024-02-21T23:09:04.432182+0000","last_became_peered":"2024-02-21T23:09:04.432182+0000","last_unstale":"2024-02-21T23:09:05.791848+0000","last_undegraded":"2024-02-21T23:09:05.791848+0000","last_fullsized":"2024-02-21T23:09:05.791848+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:47:59.469506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.27","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.449907+0000","last_change":"2024-02-21T23:09:04.443245+0000","last_active":"2024-02-21T23:09:05.449907+0000","last_peered":"2024-02-21T23:09:05.449907+0000","last_clean":"2024-02-21T23:09:05.449907+0000","last_became_active":"2024-02-21T23:09:04.441112+0000","last_became_peered":"2024-02-21T23:09:04.441112+0000","last_unstale":"2024-02-21T23:09:05.449907+0000","last_undegraded":"2024-02-21T23:09:05.449907+0000","last_fullsized":"2024-02-21T23:09:05.449907+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:35:03.205357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.26","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.449685+0000","last_change":"2024-02-21T23:09:04.437460+0000","last_active":"2024-02-21T23:09:05.449685+0000","last_peered":"2024-02-21T23:09:05.449685+0000","last_clean":"2024-02-21T23:09:05.449685+0000","last_became_active":"2024-02-21T23:09:04.437181+0000","last_became_peered":"2024-02-21T23:09:04.437181+0000","last_unstale":"2024-02-21T23:09:05.449685+0000","last_undegraded":"2024-02-21T23:09:05.449685+0000","last_fullsized":"2024-02-21T23:09:05.449685+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:22:35.300138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.25","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.427310+0000","last_change":"2024-02-21T23:09:04.435176+0000","last_active":"2024-02-21T23:09:05.427310+0000","last_peered":"2024-02-21T23:09:05.427310+0000","last_clean":"2024-02-21T23:09:05.427310+0000","last_became_active":"2024-02-21T23:09:04.434898+0000","last_became_peered":"2024-02-21T23:09:04.434898+0000","last_unstale":"2024-02-21T23:09:05.427310+0000","last_undegraded":"2024-02-21T23:09:05.427310+0000","last_fullsized":"2024-02-21T23:09:05.427310+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:41:30.135576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.24","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.792073+0000","last_change":"2024-02-21T23:09:04.446378+0000","last_active":"2024-02-21T23:09:05.792073+0000","last_peered":"2024-02-21T23:09:05.792073+0000","last_clean":"2024-02-21T23:09:05.792073+0000","last_became_active":"2024-02-21T23:09:04.446067+0000","last_became_peered":"2024-02-21T23:09:04.446067+0000","last_unstale":"2024-02-21T23:09:05.792073+0000","last_undegraded":"2024-02-21T23:09:05.792073+0000","last_fullsized":"2024-02-21T23:09:05.792073+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:02:24.078758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.23","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.430320+0000","last_change":"2024-02-21T23:09:04.432716+0000","last_active":"2024-02-21T23:09:05.430320+0000","last_peered":"2024-02-21T23:09:05.430320+0000","last_clean":"2024-02-21T23:09:05.430320+0000","last_became_active":"2024-02-21T23:09:04.432035+0000","last_became_peered":"2024-02-21T23:09:04.432035+0000","last_unstale":"2024-02-21T23:09:05.430320+0000","last_undegraded":"2024-02-21T23:09:05.430320+0000","last_fullsized":"2024-02-21T23:09:05.430320+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:43:38.434476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.22","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.791701+0000","last_change":"2024-02-21T23:09:04.441919+0000","last_active":"2024-02-21T23:09:05.791701+0000","last_peered":"2024-02-21T23:09:05.791701+0000","last_clean":"2024-02-21T23:09:05.791701+0000","last_became_active":"2024-02-21T23:09:04.441641+0000","last_became_peered":"2024-02-21T23:09:04.441641+0000","last_unstale":"2024-02-21T23:09:05.791701+0000","last_undegraded":"2024-02-21T23:09:05.791701+0000","last_fullsized":"2024-02-21T23:09:05.791701+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:09:29.913398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.21","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.427517+0000","last_change":"2024-02-21T23:09:04.448460+0000","last_active":"2024-02-21T23:09:05.427517+0000","last_peered":"2024-02-21T23:09:05.427517+0000","last_clean":"2024-02-21T23:09:05.427517+0000","last_became_active":"2024-02-21T23:09:04.448065+0000","last_became_peered":"2024-02-21T23:09:04.448065+0000","last_unstale":"2024-02-21T23:09:05.427517+0000","last_undegraded":"2024-02-21T23:09:05.427517+0000","last_fullsized":"2024-02-21T23:09:05.427517+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:48:41.898794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.20","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.428465+0000","last_change":"2024-02-21T23:09:04.443803+0000","last_active":"2024-02-21T23:09:05.428465+0000","last_peered":"2024-02-21T23:09:05.428465+0000","last_clean":"2024-02-21T23:09:05.428465+0000","last_became_active":"2024-02-21T23:09:04.438165+0000","last_became_peered":"2024-02-21T23:09:04.438165+0000","last_unstale":"2024-02-21T23:09:05.428465+0000","last_undegraded":"2024-02-21T23:09:05.428465+0000","last_fullsized":"2024-02-21T23:09:05.428465+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:11:53.081427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.433613+0000","last_change":"2024-02-21T23:09:04.434296+0000","last_active":"2024-02-21T23:09:05.433613+0000","last_peered":"2024-02-21T23:09:05.433613+0000","last_clean":"2024-02-21T23:09:05.433613+0000","last_became_active":"2024-02-21T23:09:04.433998+0000","last_became_peered":"2024-02-21T23:09:04.433998+0000","last_unstale":"2024-02-21T23:09:05.433613+0000","last_undegraded":"2024-02-21T23:09:05.433613+0000","last_fullsized":"2024-02-21T23:09:05.433613+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:35:03.205357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1f","version":"135'4","reported_seq":18,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.426897+0000","last_change":"2024-02-21T23:09:03.399407+0000","last_active":"2024-02-21T23:09:05.426897+0000","last_peered":"2024-02-21T23:09:05.426897+0000","last_clean":"2024-02-21T23:09:05.426897+0000","last_became_active":"2024-02-21T23:09:03.399083+0000","last_became_peered":"2024-02-21T23:09:03.399083+0000","last_unstale":"2024-02-21T23:09:05.426897+0000","last_undegraded":"2024-02-21T23:09:05.426897+0000","last_fullsized":"2024-02-21T23:09:05.426897+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_clean_scrub_stamp":"2024-02-21T23:09:02.355513+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:38:42.732962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.433575+0000","last_change":"2024-02-21T23:09:04.422895+0000","last_active":"2024-02-21T23:09:05.433575+0000","last_peered":"2024-02-21T23:09:05.433575+0000","last_clean":"2024-02-21T23:09:05.433575+0000","last_became_active":"2024-02-21T23:09:04.422319+0000","last_became_peered":"2024-02-21T23:09:04.422319+0000","last_unstale":"2024-02-21T23:09:05.433575+0000","last_undegraded":"2024-02-21T23:09:05.433575+0000","last_fullsized":"2024-02-21T23:09:05.433575+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:19:48.874885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1e","version":"135'1","reported_seq":15,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.455580+0000","last_change":"2024-02-21T23:09:03.417943+0000","last_active":"2024-02-21T23:09:05.455580+0000","last_peered":"2024-02-21T23:09:05.455580+0000","last_clean":"2024-02-21T23:09:05.455580+0000","last_became_active":"2024-02-21T23:09:03.417648+0000","last_became_peered":"2024-02-21T23:09:03.417648+0000","last_unstale":"2024-02-21T23:09:05.455580+0000","last_undegraded":"2024-02-21T23:09:05.455580+0000","last_fullsized":"2024-02-21T23:09:05.455580+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_clean_scrub_stamp":"2024-02-21T23:09:02.355513+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:20:51.138095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.430247+0000","last_change":"2024-02-21T23:09:04.439036+0000","last_active":"2024-02-21T23:09:05.430247+0000","last_peered":"2024-02-21T23:09:05.430247+0000","last_clean":"2024-02-21T23:09:05.430247+0000","last_became_active":"2024-02-21T23:09:04.438681+0000","last_became_peered":"2024-02-21T23:09:04.438681+0000","last_unstale":"2024-02-21T23:09:05.430247+0000","last_undegraded":"2024-02-21T23:09:05.430247+0000","last_fullsized":"2024-02-21T23:09:05.430247+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:14:17.478899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"135'2","reported_seq":16,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.456346+0000","last_change":"2024-02-21T23:09:03.418343+0000","last_active":"2024-02-21T23:09:05.456346+0000","last_peered":"2024-02-21T23:09:05.456346+0000","last_clean":"2024-02-21T23:09:05.456346+0000","last_became_active":"2024-02-21T23:09:03.417877+0000","last_became_peered":"2024-02-21T23:09:03.417877+0000","last_unstale":"2024-02-21T23:09:05.456346+0000","last_undegraded":"2024-02-21T23:09:05.456346+0000","last_fullsized":"2024-02-21T23:09:05.456346+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_clean_scrub_stamp":"2024-02-21T23:09:02.355513+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:43:37.418744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.456039+0000","last_change":"2024-02-21T23:09:04.446487+0000","last_active":"2024-02-21T23:09:05.456039+0000","last_peered":"2024-02-21T23:09:05.456039+0000","last_clean":"2024-02-21T23:09:05.456039+0000","last_became_active":"2024-02-21T23:09:04.446168+0000","last_became_peered":"2024-02-21T23:09:04.446168+0000","last_unstale":"2024-02-21T23:09:05.456039+0000","last_undegraded":"2024-02-21T23:09:05.456039+0000","last_fullsized":"2024-02-21T23:09:05.456039+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:47:57.652729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1c","version":"135'2","reported_seq":16,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.433639+0000","last_change":"2024-02-21T23:09:03.414452+0000","last_active":"2024-02-21T23:09:05.433639+0000","last_peered":"2024-02-21T23:09:05.433639+0000","last_clean":"2024-02-21T23:09:05.433639+0000","last_became_active":"2024-02-21T23:09:03.414158+0000","last_became_peered":"2024-02-21T23:09:03.414158+0000","last_unstale":"2024-02-21T23:09:05.433639+0000","last_undegraded":"2024-02-21T23:09:05.433639+0000","last_fullsized":"2024-02-21T23:09:05.433639+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_clean_scrub_stamp":"2024-02-21T23:09:02.355513+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T11:03:16.672113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.426208+0000","last_change":"2024-02-21T23:09:04.444578+0000","last_active":"2024-02-21T23:09:05.426208+0000","last_peered":"2024-02-21T23:09:05.426208+0000","last_clean":"2024-02-21T23:09:05.426208+0000","last_became_active":"2024-02-21T23:09:04.444281+0000","last_became_peered":"2024-02-21T23:09:04.444281+0000","last_unstale":"2024-02-21T23:09:05.426208+0000","last_undegraded":"2024-02-21T23:09:05.426208+0000","last_fullsized":"2024-02-21T23:09:05.426208+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:36:53.116212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.1b","version":"135'1","reported_seq":15,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.430548+0000","last_change":"2024-02-21T23:09:03.418219+0000","last_active":"2024-02-21T23:09:05.430548+0000","last_peered":"2024-02-21T23:09:05.430548+0000","last_clean":"2024-02-21T23:09:05.430548+0000","last_became_active":"2024-02-21T23:09:03.416184+0000","last_became_peered":"2024-02-21T23:09:03.416184+0000","last_unstale":"2024-02-21T23:09:05.430548+0000","last_undegraded":"2024-02-21T23:09:05.430548+0000","last_fullsized":"2024-02-21T23:09:05.430548+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_clean_scrub_stamp":"2024-02-21T23:09:02.355513+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:21:30.260732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.793398+0000","last_change":"2024-02-21T23:09:04.434581+0000","last_active":"2024-02-21T23:09:05.793398+0000","last_peered":"2024-02-21T23:09:05.793398+0000","last_clean":"2024-02-21T23:09:05.793398+0000","last_became_active":"2024-02-21T23:09:04.432520+0000","last_became_peered":"2024-02-21T23:09:04.432520+0000","last_unstale":"2024-02-21T23:09:05.793398+0000","last_undegraded":"2024-02-21T23:09:05.793398+0000","last_fullsized":"2024-02-21T23:09:05.793398+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:05:44.161417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1a","version":"135'1","reported_seq":15,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.427205+0000","last_change":"2024-02-21T23:09:03.412440+0000","last_active":"2024-02-21T23:09:05.427205+0000","last_peered":"2024-02-21T23:09:05.427205+0000","last_clean":"2024-02-21T23:09:05.427205+0000","last_became_active":"2024-02-21T23:09:03.412105+0000","last_became_peered":"2024-02-21T23:09:03.412105+0000","last_unstale":"2024-02-21T23:09:05.427205+0000","last_undegraded":"2024-02-21T23:09:05.427205+0000","last_fullsized":"2024-02-21T23:09:05.427205+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_clean_scrub_stamp":"2024-02-21T23:09:02.355513+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:41:07.058236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.433777+0000","last_change":"2024-02-21T23:09:04.440651+0000","last_active":"2024-02-21T23:09:05.433777+0000","last_peered":"2024-02-21T23:09:05.433777+0000","last_clean":"2024-02-21T23:09:05.433777+0000","last_became_active":"2024-02-21T23:09:04.440281+0000","last_became_peered":"2024-02-21T23:09:04.440281+0000","last_unstale":"2024-02-21T23:09:05.433777+0000","last_undegraded":"2024-02-21T23:09:05.433777+0000","last_fullsized":"2024-02-21T23:09:05.433777+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:11:06.123295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":14,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.449102+0000","last_change":"2024-02-21T23:09:03.410236+0000","last_active":"2024-02-21T23:09:05.449102+0000","last_peered":"2024-02-21T23:09:05.449102+0000","last_clean":"2024-02-21T23:09:05.449102+0000","last_became_active":"2024-02-21T23:09:03.409955+0000","last_became_peered":"2024-02-21T23:09:03.409955+0000","last_unstale":"2024-02-21T23:09:05.449102+0000","last_undegraded":"2024-02-21T23:09:05.449102+0000","last_fullsized":"2024-02-21T23:09:05.449102+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_clean_scrub_stamp":"2024-02-21T23:09:02.355513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:41:32.590121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.449607+0000","last_change":"2024-02-21T23:09:04.443229+0000","last_active":"2024-02-21T23:09:05.449607+0000","last_peered":"2024-02-21T23:09:05.449607+0000","last_clean":"2024-02-21T23:09:05.449607+0000","last_became_active":"2024-02-21T23:09:04.440615+0000","last_became_peered":"2024-02-21T23:09:04.440615+0000","last_unstale":"2024-02-21T23:09:05.449607+0000","last_undegraded":"2024-02-21T23:09:05.449607+0000","last_fullsized":"2024-02-21T23:09:05.449607+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:41:08.073967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":14,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.426227+0000","last_change":"2024-02-21T23:09:03.413066+0000","last_active":"2024-02-21T23:09:05.426227+0000","last_peered":"2024-02-21T23:09:05.426227+0000","last_clean":"2024-02-21T23:09:05.426227+0000","last_became_active":"2024-02-21T23:09:03.412758+0000","last_became_peered":"2024-02-21T23:09:03.412758+0000","last_unstale":"2024-02-21T23:09:05.426227+0000","last_undegraded":"2024-02-21T23:09:05.426227+0000","last_fullsized":"2024-02-21T23:09:05.426227+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_clean_scrub_stamp":"2024-02-21T23:09:02.355513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:39:46.409525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.429236+0000","last_change":"2024-02-21T23:09:04.443589+0000","last_active":"2024-02-21T23:09:05.429236+0000","last_peered":"2024-02-21T23:09:05.429236+0000","last_clean":"2024-02-21T23:09:05.429236+0000","last_became_active":"2024-02-21T23:09:04.442663+0000","last_became_peered":"2024-02-21T23:09:04.442663+0000","last_unstale":"2024-02-21T23:09:05.429236+0000","last_undegraded":"2024-02-21T23:09:05.429236+0000","last_fullsized":"2024-02-21T23:09:05.429236+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:38:15.236282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":14,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.455918+0000","last_change":"2024-02-21T23:09:03.411990+0000","last_active":"2024-02-21T23:09:05.455918+0000","last_peered":"2024-02-21T23:09:05.455918+0000","last_clean":"2024-02-21T23:09:05.455918+0000","last_became_active":"2024-02-21T23:09:03.411029+0000","last_became_peered":"2024-02-21T23:09:03.411029+0000","last_unstale":"2024-02-21T23:09:05.455918+0000","last_undegraded":"2024-02-21T23:09:05.455918+0000","last_fullsized":"2024-02-21T23:09:05.455918+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_clean_scrub_stamp":"2024-02-21T23:09:02.355513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:48:24.675692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.792151+0000","last_change":"2024-02-21T23:09:04.437531+0000","last_active":"2024-02-21T23:09:05.792151+0000","last_peered":"2024-02-21T23:09:05.792151+0000","last_clean":"2024-02-21T23:09:05.792151+0000","last_became_active":"2024-02-21T23:09:04.435765+0000","last_became_peered":"2024-02-21T23:09:04.435765+0000","last_unstale":"2024-02-21T23:09:05.792151+0000","last_undegraded":"2024-02-21T23:09:05.792151+0000","last_fullsized":"2024-02-21T23:09:05.792151+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:42:29.292918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":14,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.450145+0000","last_change":"2024-02-21T23:09:03.416078+0000","last_active":"2024-02-21T23:09:05.450145+0000","last_peered":"2024-02-21T23:09:05.450145+0000","last_clean":"2024-02-21T23:09:05.450145+0000","last_became_active":"2024-02-21T23:09:03.415768+0000","last_became_peered":"2024-02-21T23:09:03.415768+0000","last_unstale":"2024-02-21T23:09:05.450145+0000","last_undegraded":"2024-02-21T23:09:05.450145+0000","last_fullsized":"2024-02-21T23:09:05.450145+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_clean_scrub_stamp":"2024-02-21T23:09:02.355513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:12:23.809440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.791979+0000","last_change":"2024-02-21T23:09:04.449452+0000","last_active":"2024-02-21T23:09:05.791979+0000","last_peered":"2024-02-21T23:09:05.791979+0000","last_clean":"2024-02-21T23:09:05.791979+0000","last_became_active":"2024-02-21T23:09:04.449109+0000","last_became_peered":"2024-02-21T23:09:04.449109+0000","last_unstale":"2024-02-21T23:09:05.791979+0000","last_undegraded":"2024-02-21T23:09:05.791979+0000","last_fullsized":"2024-02-21T23:09:05.791979+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:02:55.960941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.15","version":"135'3","reported_seq":17,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.792143+0000","last_change":"2024-02-21T23:09:03.399973+0000","last_active":"2024-02-21T23:09:05.792143+0000","last_peered":"2024-02-21T23:09:05.792143+0000","last_clean":"2024-02-21T23:09:05.792143+0000","last_became_active":"2024-02-21T23:09:03.399696+0000","last_became_peered":"2024-02-21T23:09:03.399696+0000","last_unstale":"2024-02-21T23:09:05.792143+0000","last_undegraded":"2024-02-21T23:09:05.792143+0000","last_fullsized":"2024-02-21T23:09:05.792143+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_clean_scrub_stamp":"2024-02-21T23:09:02.355513+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:28:31.009267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.428738+0000","last_change":"2024-02-21T23:09:04.448075+0000","last_active":"2024-02-21T23:09:05.428738+0000","last_peered":"2024-02-21T23:09:05.428738+0000","last_clean":"2024-02-21T23:09:05.428738+0000","last_became_active":"2024-02-21T23:09:04.443441+0000","last_became_peered":"2024-02-21T23:09:04.443441+0000","last_unstale":"2024-02-21T23:09:05.428738+0000","last_undegraded":"2024-02-21T23:09:05.428738+0000","last_fullsized":"2024-02-21T23:09:05.428738+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:12:24.825171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.13","version":"135'2","reported_seq":16,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.427470+0000","last_change":"2024-02-21T23:09:03.392609+0000","last_active":"2024-02-21T23:09:05.427470+0000","last_peered":"2024-02-21T23:09:05.427470+0000","last_clean":"2024-02-21T23:09:05.427470+0000","last_became_active":"2024-02-21T23:09:03.392119+0000","last_became_peered":"2024-02-21T23:09:03.392119+0000","last_unstale":"2024-02-21T23:09:05.427470+0000","last_undegraded":"2024-02-21T23:09:05.427470+0000","last_fullsized":"2024-02-21T23:09:05.427470+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_clean_scrub_stamp":"2024-02-21T23:09:02.355513+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:47:46.091586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.433083+0000","last_change":"2024-02-21T23:09:04.417763+0000","last_active":"2024-02-21T23:09:05.433083+0000","last_peered":"2024-02-21T23:09:05.433083+0000","last_clean":"2024-02-21T23:09:05.433083+0000","last_became_active":"2024-02-21T23:09:04.417047+0000","last_became_peered":"2024-02-21T23:09:04.417047+0000","last_unstale":"2024-02-21T23:09:05.433083+0000","last_undegraded":"2024-02-21T23:09:05.433083+0000","last_fullsized":"2024-02-21T23:09:05.433083+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:35:27.882774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":14,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.793369+0000","last_change":"2024-02-21T23:09:03.421629+0000","last_active":"2024-02-21T23:09:05.793369+0000","last_peered":"2024-02-21T23:09:05.793369+0000","last_clean":"2024-02-21T23:09:05.793369+0000","last_became_active":"2024-02-21T23:09:03.421246+0000","last_became_peered":"2024-02-21T23:09:03.421246+0000","last_unstale":"2024-02-21T23:09:05.793369+0000","last_undegraded":"2024-02-21T23:09:05.793369+0000","last_fullsized":"2024-02-21T23:09:05.793369+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_clean_scrub_stamp":"2024-02-21T23:09:02.355513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:39:28.594527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.427043+0000","last_change":"2024-02-21T23:09:04.442103+0000","last_active":"2024-02-21T23:09:05.427043+0000","last_peered":"2024-02-21T23:09:05.427043+0000","last_clean":"2024-02-21T23:09:05.427043+0000","last_became_active":"2024-02-21T23:09:04.441122+0000","last_became_peered":"2024-02-21T23:09:04.441122+0000","last_unstale":"2024-02-21T23:09:05.427043+0000","last_undegraded":"2024-02-21T23:09:05.427043+0000","last_fullsized":"2024-02-21T23:09:05.427043+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:47:17.508603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":14,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.793769+0000","last_change":"2024-02-21T23:09:03.411660+0000","last_active":"2024-02-21T23:09:05.793769+0000","last_peered":"2024-02-21T23:09:05.793769+0000","last_clean":"2024-02-21T23:09:05.793769+0000","last_became_active":"2024-02-21T23:09:03.411342+0000","last_became_peered":"2024-02-21T23:09:03.411342+0000","last_unstale":"2024-02-21T23:09:05.793769+0000","last_undegraded":"2024-02-21T23:09:05.793769+0000","last_fullsized":"2024-02-21T23:09:05.793769+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_clean_scrub_stamp":"2024-02-21T23:09:02.355513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:32:36.409698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.433341+0000","last_change":"2024-02-21T23:09:04.427876+0000","last_active":"2024-02-21T23:09:05.433341+0000","last_peered":"2024-02-21T23:09:05.433341+0000","last_clean":"2024-02-21T23:09:05.433341+0000","last_became_active":"2024-02-21T23:09:04.426571+0000","last_became_peered":"2024-02-21T23:09:04.426571+0000","last_unstale":"2024-02-21T23:09:05.433341+0000","last_undegraded":"2024-02-21T23:09:05.433341+0000","last_fullsized":"2024-02-21T23:09:05.433341+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:41:08.073967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.10","version":"135'8","reported_seq":22,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.448754+0000","last_change":"2024-02-21T23:09:03.418324+0000","last_active":"2024-02-21T23:09:05.448754+0000","last_peered":"2024-02-21T23:09:05.448754+0000","last_clean":"2024-02-21T23:09:05.448754+0000","last_became_active":"2024-02-21T23:09:03.417617+0000","last_became_peered":"2024-02-21T23:09:03.417617+0000","last_unstale":"2024-02-21T23:09:05.448754+0000","last_undegraded":"2024-02-21T23:09:05.448754+0000","last_fullsized":"2024-02-21T23:09:05.448754+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_clean_scrub_stamp":"2024-02-21T23:09:02.355513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:49:58.045953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.455503+0000","last_change":"2024-02-21T23:09:04.441228+0000","last_active":"2024-02-21T23:09:05.455503+0000","last_peered":"2024-02-21T23:09:05.455503+0000","last_clean":"2024-02-21T23:09:05.455503+0000","last_became_active":"2024-02-21T23:09:04.439293+0000","last_became_peered":"2024-02-21T23:09:04.439293+0000","last_unstale":"2024-02-21T23:09:05.455503+0000","last_undegraded":"2024-02-21T23:09:05.455503+0000","last_fullsized":"2024-02-21T23:09:05.455503+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:11:06.123295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.f","version":"135'1","reported_seq":15,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.791590+0000","last_change":"2024-02-21T23:09:03.410345+0000","last_active":"2024-02-21T23:09:05.791590+0000","last_peered":"2024-02-21T23:09:05.791590+0000","last_clean":"2024-02-21T23:09:05.791590+0000","last_became_active":"2024-02-21T23:09:03.410051+0000","last_became_peered":"2024-02-21T23:09:03.410051+0000","last_unstale":"2024-02-21T23:09:05.791590+0000","last_undegraded":"2024-02-21T23:09:05.791590+0000","last_fullsized":"2024-02-21T23:09:05.791590+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_clean_scrub_stamp":"2024-02-21T23:09:02.355513+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:58:30.059254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.456204+0000","last_change":"2024-02-21T23:09:04.450156+0000","last_active":"2024-02-21T23:09:05.456204+0000","last_peered":"2024-02-21T23:09:05.456204+0000","last_clean":"2024-02-21T23:09:05.456204+0000","last_became_active":"2024-02-21T23:09:04.447844+0000","last_became_peered":"2024-02-21T23:09:04.447844+0000","last_unstale":"2024-02-21T23:09:05.456204+0000","last_undegraded":"2024-02-21T23:09:05.456204+0000","last_fullsized":"2024-02-21T23:09:05.456204+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:11:50.891328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.e","version":"135'2","reported_seq":16,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.455728+0000","last_change":"2024-02-21T23:09:03.410582+0000","last_active":"2024-02-21T23:09:05.455728+0000","last_peered":"2024-02-21T23:09:05.455728+0000","last_clean":"2024-02-21T23:09:05.455728+0000","last_became_active":"2024-02-21T23:09:03.410277+0000","last_became_peered":"2024-02-21T23:09:03.410277+0000","last_unstale":"2024-02-21T23:09:05.455728+0000","last_undegraded":"2024-02-21T23:09:05.455728+0000","last_fullsized":"2024-02-21T23:09:05.455728+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_clean_scrub_stamp":"2024-02-21T23:09:02.355513+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:14:16.463167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.449862+0000","last_change":"2024-02-21T23:09:04.437586+0000","last_active":"2024-02-21T23:09:05.449862+0000","last_peered":"2024-02-21T23:09:05.449862+0000","last_clean":"2024-02-21T23:09:05.449862+0000","last_became_active":"2024-02-21T23:09:04.437302+0000","last_became_peered":"2024-02-21T23:09:04.437302+0000","last_unstale":"2024-02-21T23:09:05.449862+0000","last_undegraded":"2024-02-21T23:09:05.449862+0000","last_fullsized":"2024-02-21T23:09:05.449862+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:19:48.874885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.d","version":"135'1","reported_seq":15,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.433706+0000","last_change":"2024-02-21T23:09:03.412679+0000","last_active":"2024-02-21T23:09:05.433706+0000","last_peered":"2024-02-21T23:09:05.433706+0000","last_clean":"2024-02-21T23:09:05.433706+0000","last_became_active":"2024-02-21T23:09:03.412384+0000","last_became_peered":"2024-02-21T23:09:03.412384+0000","last_unstale":"2024-02-21T23:09:05.433706+0000","last_undegraded":"2024-02-21T23:09:05.433706+0000","last_fullsized":"2024-02-21T23:09:05.433706+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_clean_scrub_stamp":"2024-02-21T23:09:02.355513+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:34:27.316643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.450244+0000","last_change":"2024-02-21T23:09:04.441330+0000","last_active":"2024-02-21T23:09:05.450244+0000","last_peered":"2024-02-21T23:09:05.450244+0000","last_clean":"2024-02-21T23:09:05.450244+0000","last_became_active":"2024-02-21T23:09:04.438866+0000","last_became_peered":"2024-02-21T23:09:04.438866+0000","last_unstale":"2024-02-21T23:09:05.450244+0000","last_undegraded":"2024-02-21T23:09:05.450244+0000","last_fullsized":"2024-02-21T23:09:05.450244+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:47:57.652729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":14,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.794195+0000","last_change":"2024-02-21T23:09:03.422283+0000","last_active":"2024-02-21T23:09:05.794195+0000","last_peered":"2024-02-21T23:09:05.794195+0000","last_clean":"2024-02-21T23:09:05.794195+0000","last_became_active":"2024-02-21T23:09:03.421493+0000","last_became_peered":"2024-02-21T23:09:03.421493+0000","last_unstale":"2024-02-21T23:09:05.794195+0000","last_undegraded":"2024-02-21T23:09:05.794195+0000","last_fullsized":"2024-02-21T23:09:05.794195+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_clean_scrub_stamp":"2024-02-21T23:09:02.355513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:39:15.702164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.455254+0000","last_change":"2024-02-21T23:09:04.452380+0000","last_active":"2024-02-21T23:09:05.455254+0000","last_peered":"2024-02-21T23:09:05.455254+0000","last_clean":"2024-02-21T23:09:05.455254+0000","last_became_active":"2024-02-21T23:09:04.452104+0000","last_became_peered":"2024-02-21T23:09:04.452104+0000","last_unstale":"2024-02-21T23:09:05.455254+0000","last_undegraded":"2024-02-21T23:09:05.455254+0000","last_fullsized":"2024-02-21T23:09:05.455254+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:35:17.242030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.b","version":"135'2","reported_seq":16,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.425641+0000","last_change":"2024-02-21T23:09:03.421193+0000","last_active":"2024-02-21T23:09:05.425641+0000","last_peered":"2024-02-21T23:09:05.425641+0000","last_clean":"2024-02-21T23:09:05.425641+0000","last_became_active":"2024-02-21T23:09:03.420657+0000","last_became_peered":"2024-02-21T23:09:03.420657+0000","last_unstale":"2024-02-21T23:09:05.425641+0000","last_undegraded":"2024-02-21T23:09:05.425641+0000","last_fullsized":"2024-02-21T23:09:05.425641+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_clean_scrub_stamp":"2024-02-21T23:09:02.355513+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:14:36.898158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.433091+0000","last_change":"2024-02-21T23:09:04.437989+0000","last_active":"2024-02-21T23:09:05.433091+0000","last_peered":"2024-02-21T23:09:05.433091+0000","last_clean":"2024-02-21T23:09:05.433091+0000","last_became_active":"2024-02-21T23:09:04.437719+0000","last_became_peered":"2024-02-21T23:09:04.437719+0000","last_unstale":"2024-02-21T23:09:05.433091+0000","last_undegraded":"2024-02-21T23:09:05.433091+0000","last_fullsized":"2024-02-21T23:09:05.433091+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:47:47.107317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":14,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.429554+0000","last_change":"2024-02-21T23:09:03.417938+0000","last_active":"2024-02-21T23:09:05.429554+0000","last_peered":"2024-02-21T23:09:05.429554+0000","last_clean":"2024-02-21T23:09:05.429554+0000","last_became_active":"2024-02-21T23:09:03.413678+0000","last_became_peered":"2024-02-21T23:09:03.413678+0000","last_unstale":"2024-02-21T23:09:05.429554+0000","last_undegraded":"2024-02-21T23:09:05.429554+0000","last_fullsized":"2024-02-21T23:09:05.429554+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_clean_scrub_stamp":"2024-02-21T23:09:02.355513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:47:43.156451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.455082+0000","last_change":"2024-02-21T23:09:04.443806+0000","last_active":"2024-02-21T23:09:05.455082+0000","last_peered":"2024-02-21T23:09:05.455082+0000","last_clean":"2024-02-21T23:09:05.455082+0000","last_became_active":"2024-02-21T23:09:04.443476+0000","last_became_peered":"2024-02-21T23:09:04.443476+0000","last_unstale":"2024-02-21T23:09:05.455082+0000","last_undegraded":"2024-02-21T23:09:05.455082+0000","last_fullsized":"2024-02-21T23:09:05.455082+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:35:03.205357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":14,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.425096+0000","last_change":"2024-02-21T23:09:03.414331+0000","last_active":"2024-02-21T23:09:05.425096+0000","last_peered":"2024-02-21T23:09:05.425096+0000","last_clean":"2024-02-21T23:09:05.425096+0000","last_became_active":"2024-02-21T23:09:03.414026+0000","last_became_peered":"2024-02-21T23:09:03.414026+0000","last_unstale":"2024-02-21T23:09:05.425096+0000","last_undegraded":"2024-02-21T23:09:05.425096+0000","last_fullsized":"2024-02-21T23:09:05.425096+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_clean_scrub_stamp":"2024-02-21T23:09:02.355513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:02:59.348505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.433342+0000","last_change":"2024-02-21T23:09:04.421572+0000","last_active":"2024-02-21T23:09:05.433342+0000","last_peered":"2024-02-21T23:09:05.433342+0000","last_clean":"2024-02-21T23:09:05.433342+0000","last_became_active":"2024-02-21T23:09:04.421131+0000","last_became_peered":"2024-02-21T23:09:04.421131+0000","last_unstale":"2024-02-21T23:09:05.433342+0000","last_undegraded":"2024-02-21T23:09:05.433342+0000","last_fullsized":"2024-02-21T23:09:05.433342+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:22:35.300138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":14,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.429031+0000","last_change":"2024-02-21T23:09:03.408521+0000","last_active":"2024-02-21T23:09:05.429031+0000","last_peered":"2024-02-21T23:09:05.429031+0000","last_clean":"2024-02-21T23:09:05.429031+0000","last_became_active":"2024-02-21T23:09:03.408230+0000","last_became_peered":"2024-02-21T23:09:03.408230+0000","last_unstale":"2024-02-21T23:09:05.429031+0000","last_undegraded":"2024-02-21T23:09:05.429031+0000","last_fullsized":"2024-02-21T23:09:05.429031+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_clean_scrub_stamp":"2024-02-21T23:09:02.355513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:00:20.864214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.426837+0000","last_change":"2024-02-21T23:09:04.440123+0000","last_active":"2024-02-21T23:09:05.426837+0000","last_peered":"2024-02-21T23:09:05.426837+0000","last_clean":"2024-02-21T23:09:05.426837+0000","last_became_active":"2024-02-21T23:09:04.439731+0000","last_became_peered":"2024-02-21T23:09:04.439731+0000","last_unstale":"2024-02-21T23:09:05.426837+0000","last_undegraded":"2024-02-21T23:09:05.426837+0000","last_fullsized":"2024-02-21T23:09:05.426837+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:17:27.624052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.7","version":"135'2","reported_seq":16,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.794193+0000","last_change":"2024-02-21T23:09:03.425221+0000","last_active":"2024-02-21T23:09:05.794193+0000","last_peered":"2024-02-21T23:09:05.794193+0000","last_clean":"2024-02-21T23:09:05.794193+0000","last_became_active":"2024-02-21T23:09:03.424876+0000","last_became_peered":"2024-02-21T23:09:03.424876+0000","last_unstale":"2024-02-21T23:09:05.794193+0000","last_undegraded":"2024-02-21T23:09:05.794193+0000","last_fullsized":"2024-02-21T23:09:05.794193+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_clean_scrub_stamp":"2024-02-21T23:09:02.355513+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:29:50.750441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.794049+0000","last_change":"2024-02-21T23:09:04.435775+0000","last_active":"2024-02-21T23:09:05.794049+0000","last_peered":"2024-02-21T23:09:05.794049+0000","last_clean":"2024-02-21T23:09:05.794049+0000","last_became_active":"2024-02-21T23:09:04.435448+0000","last_became_peered":"2024-02-21T23:09:04.435448+0000","last_unstale":"2024-02-21T23:09:05.794049+0000","last_undegraded":"2024-02-21T23:09:05.794049+0000","last_fullsized":"2024-02-21T23:09:05.794049+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:14:17.478899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"136'1","reported_seq":15,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.453890+0000","last_change":"2024-02-21T23:09:03.417923+0000","last_active":"2024-02-21T23:09:05.453890+0000","last_peered":"2024-02-21T23:09:05.453890+0000","last_clean":"2024-02-21T23:09:05.453890+0000","last_became_active":"2024-02-21T23:09:03.417653+0000","last_became_peered":"2024-02-21T23:09:03.417653+0000","last_unstale":"2024-02-21T23:09:05.453890+0000","last_undegraded":"2024-02-21T23:09:05.453890+0000","last_fullsized":"2024-02-21T23:09:05.453890+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_clean_scrub_stamp":"2024-02-21T23:09:02.355513+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:05:43.145685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.450140+0000","last_change":"2024-02-21T23:09:04.446108+0000","last_active":"2024-02-21T23:09:05.450140+0000","last_peered":"2024-02-21T23:09:05.450140+0000","last_clean":"2024-02-21T23:09:05.450140+0000","last_became_active":"2024-02-21T23:09:04.445822+0000","last_became_peered":"2024-02-21T23:09:04.445822+0000","last_unstale":"2024-02-21T23:09:05.450140+0000","last_undegraded":"2024-02-21T23:09:05.450140+0000","last_fullsized":"2024-02-21T23:09:05.450140+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:11:06.123295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":14,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.433811+0000","last_change":"2024-02-21T23:09:03.421552+0000","last_active":"2024-02-21T23:09:05.433811+0000","last_peered":"2024-02-21T23:09:05.433811+0000","last_clean":"2024-02-21T23:09:05.433811+0000","last_became_active":"2024-02-21T23:09:03.421165+0000","last_became_peered":"2024-02-21T23:09:03.421165+0000","last_unstale":"2024-02-21T23:09:05.433811+0000","last_undegraded":"2024-02-21T23:09:05.433811+0000","last_fullsized":"2024-02-21T23:09:05.433811+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_clean_scrub_stamp":"2024-02-21T23:09:02.355513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:18:46.745293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.426703+0000","last_change":"2024-02-21T23:09:04.426453+0000","last_active":"2024-02-21T23:09:05.426703+0000","last_peered":"2024-02-21T23:09:05.426703+0000","last_clean":"2024-02-21T23:09:05.426703+0000","last_became_active":"2024-02-21T23:09:04.426113+0000","last_became_peered":"2024-02-21T23:09:04.426113+0000","last_unstale":"2024-02-21T23:09:05.426703+0000","last_undegraded":"2024-02-21T23:09:05.426703+0000","last_fullsized":"2024-02-21T23:09:05.426703+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:43:56.054676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":14,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.426969+0000","last_change":"2024-02-21T23:09:03.414797+0000","last_active":"2024-02-21T23:09:05.426969+0000","last_peered":"2024-02-21T23:09:05.426969+0000","last_clean":"2024-02-21T23:09:05.426969+0000","last_became_active":"2024-02-21T23:09:03.414480+0000","last_became_peered":"2024-02-21T23:09:03.414480+0000","last_unstale":"2024-02-21T23:09:05.426969+0000","last_undegraded":"2024-02-21T23:09:05.426969+0000","last_fullsized":"2024-02-21T23:09:05.426969+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_clean_scrub_stamp":"2024-02-21T23:09:02.355513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:35:26.867043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.428889+0000","last_change":"2024-02-21T23:09:04.448429+0000","last_active":"2024-02-21T23:09:05.428889+0000","last_peered":"2024-02-21T23:09:05.428889+0000","last_clean":"2024-02-21T23:09:05.428889+0000","last_became_active":"2024-02-21T23:09:04.445229+0000","last_became_peered":"2024-02-21T23:09:04.445229+0000","last_unstale":"2024-02-21T23:09:05.428889+0000","last_undegraded":"2024-02-21T23:09:05.428889+0000","last_fullsized":"2024-02-21T23:09:05.428889+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:05:44.161417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":14,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.449455+0000","last_change":"2024-02-21T23:09:03.413442+0000","last_active":"2024-02-21T23:09:05.449455+0000","last_peered":"2024-02-21T23:09:05.449455+0000","last_clean":"2024-02-21T23:09:05.449455+0000","last_became_active":"2024-02-21T23:09:03.412202+0000","last_became_peered":"2024-02-21T23:09:03.412202+0000","last_unstale":"2024-02-21T23:09:05.449455+0000","last_undegraded":"2024-02-21T23:09:05.449455+0000","last_fullsized":"2024-02-21T23:09:05.449455+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_clean_scrub_stamp":"2024-02-21T23:09:02.355513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:18:46.745293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.793444+0000","last_change":"2024-02-21T23:09:04.422659+0000","last_active":"2024-02-21T23:09:05.793444+0000","last_peered":"2024-02-21T23:09:05.793444+0000","last_clean":"2024-02-21T23:09:05.793444+0000","last_became_active":"2024-02-21T23:09:04.421925+0000","last_became_peered":"2024-02-21T23:09:04.421925+0000","last_unstale":"2024-02-21T23:09:05.793444+0000","last_undegraded":"2024-02-21T23:09:05.793444+0000","last_fullsized":"2024-02-21T23:09:05.793444+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:38:15.236282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":14,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.791734+0000","last_change":"2024-02-21T23:09:03.421966+0000","last_active":"2024-02-21T23:09:05.791734+0000","last_peered":"2024-02-21T23:09:05.791734+0000","last_clean":"2024-02-21T23:09:05.791734+0000","last_became_active":"2024-02-21T23:09:03.420981+0000","last_became_peered":"2024-02-21T23:09:03.420981+0000","last_unstale":"2024-02-21T23:09:05.791734+0000","last_undegraded":"2024-02-21T23:09:05.791734+0000","last_fullsized":"2024-02-21T23:09:05.791734+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_clean_scrub_stamp":"2024-02-21T23:09:02.355513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:23:45.233493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.433366+0000","last_change":"2024-02-21T23:09:04.435819+0000","last_active":"2024-02-21T23:09:05.433366+0000","last_peered":"2024-02-21T23:09:05.433366+0000","last_clean":"2024-02-21T23:09:05.433366+0000","last_became_active":"2024-02-21T23:09:04.435418+0000","last_became_peered":"2024-02-21T23:09:04.435418+0000","last_unstale":"2024-02-21T23:09:05.433366+0000","last_undegraded":"2024-02-21T23:09:05.433366+0000","last_fullsized":"2024-02-21T23:09:05.433366+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:39:05.616018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":14,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.449375+0000","last_change":"2024-02-21T23:09:03.414973+0000","last_active":"2024-02-21T23:09:05.449375+0000","last_peered":"2024-02-21T23:09:05.449375+0000","last_clean":"2024-02-21T23:09:05.449375+0000","last_became_active":"2024-02-21T23:09:03.412084+0000","last_became_peered":"2024-02-21T23:09:03.412084+0000","last_unstale":"2024-02-21T23:09:05.449375+0000","last_undegraded":"2024-02-21T23:09:05.449375+0000","last_fullsized":"2024-02-21T23:09:05.449375+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_clean_scrub_stamp":"2024-02-21T23:09:02.355513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:11:52.065696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.430121+0000","last_change":"2024-02-21T23:09:04.432964+0000","last_active":"2024-02-21T23:09:05.430121+0000","last_peered":"2024-02-21T23:09:05.430121+0000","last_clean":"2024-02-21T23:09:05.430121+0000","last_became_active":"2024-02-21T23:09:04.432409+0000","last_became_peered":"2024-02-21T23:09:04.432409+0000","last_unstale":"2024-02-21T23:09:05.430121+0000","last_undegraded":"2024-02-21T23:09:05.430121+0000","last_fullsized":"2024-02-21T23:09:05.430121+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:20:52.153827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":14,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.455511+0000","last_change":"2024-02-21T23:09:03.407108+0000","last_active":"2024-02-21T23:09:05.455511+0000","last_peered":"2024-02-21T23:09:05.455511+0000","last_clean":"2024-02-21T23:09:05.455511+0000","last_became_active":"2024-02-21T23:09:03.406087+0000","last_became_peered":"2024-02-21T23:09:03.406087+0000","last_unstale":"2024-02-21T23:09:05.455511+0000","last_undegraded":"2024-02-21T23:09:05.455511+0000","last_fullsized":"2024-02-21T23:09:05.455511+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_clean_scrub_stamp":"2024-02-21T23:09:02.355513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:39:46.409525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.429351+0000","last_change":"2024-02-21T23:09:04.437750+0000","last_active":"2024-02-21T23:09:05.429351+0000","last_peered":"2024-02-21T23:09:05.429351+0000","last_clean":"2024-02-21T23:09:05.429351+0000","last_became_active":"2024-02-21T23:09:04.435407+0000","last_became_peered":"2024-02-21T23:09:04.435407+0000","last_unstale":"2024-02-21T23:09:05.429351+0000","last_undegraded":"2024-02-21T23:09:05.429351+0000","last_fullsized":"2024-02-21T23:09:05.429351+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:03:15.998238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.14","version":"135'6","reported_seq":20,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.425493+0000","last_change":"2024-02-21T23:09:03.414900+0000","last_active":"2024-02-21T23:09:05.425493+0000","last_peered":"2024-02-21T23:09:05.425493+0000","last_clean":"2024-02-21T23:09:05.425493+0000","last_became_active":"2024-02-21T23:09:03.414618+0000","last_became_peered":"2024-02-21T23:09:03.414618+0000","last_unstale":"2024-02-21T23:09:05.425493+0000","last_undegraded":"2024-02-21T23:09:05.425493+0000","last_fullsized":"2024-02-21T23:09:05.425493+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_clean_scrub_stamp":"2024-02-21T23:09:02.355513+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:07:41.075740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":389,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.455700+0000","last_change":"2024-02-21T23:02:32.289335+0000","last_active":"2024-02-21T23:09:05.455700+0000","last_peered":"2024-02-21T23:09:05.455700+0000","last_clean":"2024-02-21T23:09:05.455700+0000","last_became_active":"2024-02-21T23:02:32.286913+0000","last_became_peered":"2024-02-21T23:02:32.286913+0000","last_unstale":"2024-02-21T23:09:05.455700+0000","last_undegraded":"2024-02-21T23:09:05.455700+0000","last_fullsized":"2024-02-21T23:09:05.455700+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_clean_scrub_stamp":"2024-02-21T23:02:31.172569+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:36:23.987374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":22,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":21,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574929,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32904,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338936,"statfs":{"total":96636764160,"available":96603070464,"internally_reserved":0,"allocated":1765376,"data_stored":1414799,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574930,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32844,"kb_used_data":1660,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338996,"statfs":{"total":96636764160,"available":96603131904,"internally_reserved":0,"allocated":1699840,"data_stored":824479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574929,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32840,"kb_used_data":1656,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339000,"statfs":{"total":96636764160,"available":96603136000,"internally_reserved":0,"allocated":1695744,"data_stored":824409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574930,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32860,"kb_used_data":1680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338980,"statfs":{"total":96636764160,"available":96603115520,"internally_reserved":0,"allocated":1720320,"data_stored":832713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574930,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32856,"kb_used_data":1676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338984,"statfs":{"total":96636764160,"available":96603119616,"internally_reserved":0,"allocated":1716224,"data_stored":831717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574931,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32852,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338988,"statfs":{"total":96636764160,"available":96603123712,"internally_reserved":0,"allocated":1703936,"data_stored":824521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574929,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32852,"kb_used_data":1672,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338988,"statfs":{"total":96636764160,"available":96603123712,"internally_reserved":0,"allocated":1712128,"data_stored":831171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574929,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32940,"kb_used_data":1756,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338900,"statfs":{"total":96636764160,"available":96603033600,"internally_reserved":0,"allocated":1798144,"data_stored":1423787,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-21T23:09:09.044 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-21T23:09:09.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.311+0000 7f39ab200640 1 Processor -- start 2024-02-21T23:09:09.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.311+0000 7f39ab200640 1 -- start start 2024-02-21T23:09:09.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.313+0000 7f39ab200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f39a40c0980 0x7f39a406dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:09.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.313+0000 7f39ab200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39a406e750 con 0x7f39a40a5f40 2024-02-21T23:09:09.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.313+0000 7f39ab200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39a406ea30 con 0x7f39a40c3490 2024-02-21T23:09:09.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.313+0000 7f39ab200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39a406eff0 con 0x7f39a40c0980 2024-02-21T23:09:09.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.313+0000 7f399b400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f39a40a5f40 0x7f39a40c0270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:56876/0 (socket says 172.21.15.28:56876) 2024-02-21T23:09:09.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.313+0000 7f39a9200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f39a40c0980 0x7f39a406dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:09.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.313+0000 7f399b400640 1 -- 172.21.15.28:0/737839260 learned_addr learned my addr 172.21.15.28:0/737839260 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:09:09.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.314+0000 7f39a3400640 1 -- 172.21.15.28:0/737839260 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1253886439 0 0) 0x7f39a406e750 con 0x7f39a40a5f40 2024-02-21T23:09:09.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.314+0000 7f39a3400640 1 -- 172.21.15.28:0/737839260 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3988003520 con 0x7f39a40a5f40 2024-02-21T23:09:09.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.314+0000 7f39a9200640 1 -- 172.21.15.28:0/737839260 >> v1:172.21.15.81:6789/0 conn(0x7f39a40c3490 legacy=0x7f39a4152140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:09.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.314+0000 7f39a9200640 1 -- 172.21.15.28:0/737839260 >> v1:172.21.15.28:6789/0 conn(0x7f39a40a5f40 legacy=0x7f39a40c0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:09.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.314+0000 7f39a9200640 1 -- 172.21.15.28:0/737839260 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39a406ed10 con 0x7f39a40c0980 2024-02-21T23:09:09.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.316+0000 7f39a9200640 1 --2- 172.21.15.28:0/737839260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39a40c0980 0x7f39a406dfb0 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f399c0097a0 tx=0x7f399c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2229f0ccd658e6f4 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:09.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.316+0000 7f39a3400640 1 -- 172.21.15.28:0/737839260 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f399c035020 con 0x7f39a40c0980 2024-02-21T23:09:09.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.316+0000 7f39a3400640 1 -- 172.21.15.28:0/737839260 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f399c031830 con 0x7f39a40c0980 2024-02-21T23:09:09.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.316+0000 7f39a3400640 1 -- 172.21.15.28:0/737839260 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f399c031c00 con 0x7f39a40c0980 2024-02-21T23:09:09.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.316+0000 7f39ab200640 1 -- 172.21.15.28:0/737839260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39a40c0980 msgr2=0x7f39a406dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:09.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.316+0000 7f39ab200640 1 --2- 172.21.15.28:0/737839260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39a40c0980 0x7f39a406dfb0 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f399c0097a0 tx=0x7f399c02b8d0 comp rx=0 tx=0).stop 2024-02-21T23:09:09.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.316+0000 7f39ab200640 1 -- 172.21.15.28:0/737839260 shutdown_connections 2024-02-21T23:09:09.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.316+0000 7f39ab200640 1 --2- 172.21.15.28:0/737839260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39a40c0980 0x7f39a406dfb0 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:09.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.316+0000 7f39ab200640 1 -- 172.21.15.28:0/737839260 >> 172.21.15.28:0/737839260 conn(0x7f39a40c67f0 msgr2=0x7f39a40c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:09.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.316+0000 7f39ab200640 1 -- 172.21.15.28:0/737839260 shutdown_connections 2024-02-21T23:09:09.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.316+0000 7f39ab200640 1 -- 172.21.15.28:0/737839260 wait complete. 2024-02-21T23:09:09.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.318+0000 7f39ab200640 1 Processor -- start 2024-02-21T23:09:09.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.318+0000 7f39ab200640 1 -- start start 2024-02-21T23:09:09.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.319+0000 7f39ab200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f39a40a5f40 0x7f39a41ee010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:09.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.319+0000 7f39ab200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39a40c0980 0x7f39a4072540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:09.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.319+0000 7f39ab200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39a40c3490 0x7f39a4156cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:09.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.319+0000 7f39ab200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39a406e4f0 con 0x7f39a40c0980 2024-02-21T23:09:09.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.319+0000 7f39ab200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39a40bfde0 con 0x7f39a40c3490 2024-02-21T23:09:09.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.319+0000 7f39ab200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39a4152850 con 0x7f39a40a5f40 2024-02-21T23:09:09.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.319+0000 7f399b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39a40c0980 0x7f39a4072540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:09.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.319+0000 7f399b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39a40c0980 0x7f39a4072540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57126/0 (socket says 172.21.15.28:57126) 2024-02-21T23:09:09.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.319+0000 7f399b400640 1 -- 172.21.15.28:0/3202344029 learned_addr learned my addr 172.21.15.28:0/3202344029 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:09.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.319+0000 7f39a9200640 1 --2- 172.21.15.28:0/3202344029 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39a40c3490 0x7f39a4156cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:09.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.319+0000 7f399b400640 1 -- 172.21.15.28:0/3202344029 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f39a40a5f40 msgr2=0x7f39a41ee010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:09.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.319+0000 7f399b400640 1 --2- 172.21.15.28:0/3202344029 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f39a40a5f40 0x7f39a41ee010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:09.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.320+0000 7f399b400640 1 -- 172.21.15.28:0/3202344029 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39a40c3490 msgr2=0x7f39a4156cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:09.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.320+0000 7f399b400640 1 --2- 172.21.15.28:0/3202344029 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39a40c3490 0x7f39a4156cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:09.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.320+0000 7f399b400640 1 -- 172.21.15.28:0/3202344029 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39a41ee550 con 0x7f39a40c0980 2024-02-21T23:09:09.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.320+0000 7f39a9200640 1 --2- 172.21.15.28:0/3202344029 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39a40c3490 0x7f39a4156cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:09:09.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.320+0000 7f399b400640 1 --2- 172.21.15.28:0/3202344029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39a40c0980 0x7f39a4072540 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f398c00cbf0 tx=0x7f398c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:09.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.320+0000 7f39a0c00640 1 -- 172.21.15.28:0/3202344029 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f398c017070 con 0x7f39a40c0980 2024-02-21T23:09:09.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.320+0000 7f39a0c00640 1 -- 172.21.15.28:0/3202344029 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f398c00ad50 con 0x7f39a40c0980 2024-02-21T23:09:09.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.320+0000 7f39ab200640 1 -- 172.21.15.28:0/3202344029 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39a4149c90 con 0x7f39a40c0980 2024-02-21T23:09:09.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.320+0000 7f39a0c00640 1 -- 172.21.15.28:0/3202344029 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f398c012620 con 0x7f39a40c0980 2024-02-21T23:09:09.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.320+0000 7f39ab200640 1 -- 172.21.15.28:0/3202344029 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39a4072a80 con 0x7f39a40c0980 2024-02-21T23:09:09.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.322+0000 7f39ab200640 1 -- 172.21.15.28:0/3202344029 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39a4155870 con 0x7f39a40c0980 2024-02-21T23:09:09.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.323+0000 7f39a0c00640 1 -- 172.21.15.28:0/3202344029 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f398c01a070 con 0x7f39a40c0980 2024-02-21T23:09:09.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.324+0000 7f39a0c00640 1 --2- 172.21.15.28:0/3202344029 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f398006b4d0 0x7f398006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:09.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.324+0000 7f39a0c00640 1 -- 172.21.15.28:0/3202344029 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f398c08a280 con 0x7f39a40c0980 2024-02-21T23:09:09.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.324+0000 7f39a3e00640 1 --2- 172.21.15.28:0/3202344029 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f398006b4d0 0x7f398006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:09.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.325+0000 7f39a3e00640 1 --2- 172.21.15.28:0/3202344029 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f398006b4d0 0x7f398006d990 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f3994000f60 tx=0x7f39940079d0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:09.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.328+0000 7f39a0c00640 1 -- 172.21.15.28:0/3202344029 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f398c058f60 con 0x7f39a40c0980 2024-02-21T23:09:09.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.530+0000 7f39ab200640 1 -- 172.21.15.28:0/3202344029 --> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f39a4067f30 con 0x7f398006b4d0 2024-02-21T23:09:09.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.548+0000 7f39a0c00640 1 -- 172.21.15.28:0/3202344029 <== mgr.4101 v2:172.21.15.81:6832/1848538577 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252561 (secure 0 0 0) 0x7f39a4067f30 con 0x7f398006b4d0 2024-02-21T23:09:09.541 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:09:09.542 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-21T23:09:09.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.554+0000 7f39ab200640 1 -- 172.21.15.28:0/3202344029 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f398006b4d0 msgr2=0x7f398006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:09.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.554+0000 7f39ab200640 1 --2- 172.21.15.28:0/3202344029 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f398006b4d0 0x7f398006d990 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f3994000f60 tx=0x7f39940079d0 comp rx=0 tx=0).stop 2024-02-21T23:09:09.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.554+0000 7f39ab200640 1 -- 172.21.15.28:0/3202344029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39a40c0980 msgr2=0x7f39a4072540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:09.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.554+0000 7f39ab200640 1 --2- 172.21.15.28:0/3202344029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39a40c0980 0x7f39a4072540 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f398c00cbf0 tx=0x7f398c00a430 comp rx=0 tx=0).stop 2024-02-21T23:09:09.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.554+0000 7f39ab200640 1 -- 172.21.15.28:0/3202344029 shutdown_connections 2024-02-21T23:09:09.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.554+0000 7f39ab200640 1 --2- 172.21.15.28:0/3202344029 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f398006b4d0 0x7f398006d990 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:09.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.554+0000 7f39ab200640 1 --2- 172.21.15.28:0/3202344029 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39a40c3490 0x7f39a4156cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:09.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.554+0000 7f39ab200640 1 --2- 172.21.15.28:0/3202344029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39a40c0980 0x7f39a4072540 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:09.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.554+0000 7f39ab200640 1 --2- 172.21.15.28:0/3202344029 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f39a40a5f40 0x7f39a41ee010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:09.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.554+0000 7f39ab200640 1 -- 172.21.15.28:0/3202344029 >> 172.21.15.28:0/3202344029 conn(0x7f39a40c67f0 msgr2=0x7f39a414bf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:09.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.554+0000 7f39ab200640 1 -- 172.21.15.28:0/3202344029 shutdown_connections 2024-02-21T23:09:09.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.554+0000 7f39ab200640 1 -- 172.21.15.28:0/3202344029 wait complete. 2024-02-21T23:09:09.562 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":328,"stamp":"2024-02-21T23:09:08.802188+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":137,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":262948,"kb_used_data":13488,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754711772,"statfs":{"total":773094113280,"available":772824854528,"internally_reserved":0,"allocated":13811712,"data_stored":7807596,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002169"},"pg_stats":[{"pgid":"22.3f","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.449957+0000","last_change":"2024-02-21T23:09:04.447288+0000","last_active":"2024-02-21T23:09:05.449957+0000","last_peered":"2024-02-21T23:09:05.449957+0000","last_clean":"2024-02-21T23:09:05.449957+0000","last_became_active":"2024-02-21T23:09:04.446968+0000","last_became_peered":"2024-02-21T23:09:04.446968+0000","last_unstale":"2024-02-21T23:09:05.449957+0000","last_undegraded":"2024-02-21T23:09:05.449957+0000","last_fullsized":"2024-02-21T23:09:05.449957+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:35:17.242030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.3e","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.433652+0000","last_change":"2024-02-21T23:09:04.417638+0000","last_active":"2024-02-21T23:09:05.433652+0000","last_peered":"2024-02-21T23:09:05.433652+0000","last_clean":"2024-02-21T23:09:05.433652+0000","last_became_active":"2024-02-21T23:09:04.417041+0000","last_became_peered":"2024-02-21T23:09:04.417041+0000","last_unstale":"2024-02-21T23:09:05.433652+0000","last_undegraded":"2024-02-21T23:09:05.433652+0000","last_fullsized":"2024-02-21T23:09:05.433652+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:35:17.242030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3d","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.433825+0000","last_change":"2024-02-21T23:09:04.438506+0000","last_active":"2024-02-21T23:09:05.433825+0000","last_peered":"2024-02-21T23:09:05.433825+0000","last_clean":"2024-02-21T23:09:05.433825+0000","last_became_active":"2024-02-21T23:09:04.438203+0000","last_became_peered":"2024-02-21T23:09:04.438203+0000","last_unstale":"2024-02-21T23:09:05.433825+0000","last_undegraded":"2024-02-21T23:09:05.433825+0000","last_fullsized":"2024-02-21T23:09:05.433825+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:47:57.652729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3c","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.793882+0000","last_change":"2024-02-21T23:09:04.444562+0000","last_active":"2024-02-21T23:09:05.793882+0000","last_peered":"2024-02-21T23:09:05.793882+0000","last_clean":"2024-02-21T23:09:05.793882+0000","last_became_active":"2024-02-21T23:09:04.444227+0000","last_became_peered":"2024-02-21T23:09:04.444227+0000","last_unstale":"2024-02-21T23:09:05.793882+0000","last_undegraded":"2024-02-21T23:09:05.793882+0000","last_fullsized":"2024-02-21T23:09:05.793882+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:20:52.153827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.3b","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.427296+0000","last_change":"2024-02-21T23:09:04.437232+0000","last_active":"2024-02-21T23:09:05.427296+0000","last_peered":"2024-02-21T23:09:05.427296+0000","last_clean":"2024-02-21T23:09:05.427296+0000","last_became_active":"2024-02-21T23:09:04.436960+0000","last_became_peered":"2024-02-21T23:09:04.436960+0000","last_unstale":"2024-02-21T23:09:05.427296+0000","last_undegraded":"2024-02-21T23:09:05.427296+0000","last_fullsized":"2024-02-21T23:09:05.427296+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:46:39.489090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3a","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.426570+0000","last_change":"2024-02-21T23:09:04.437079+0000","last_active":"2024-02-21T23:09:05.426570+0000","last_peered":"2024-02-21T23:09:05.426570+0000","last_clean":"2024-02-21T23:09:05.426570+0000","last_became_active":"2024-02-21T23:09:04.436784+0000","last_became_peered":"2024-02-21T23:09:04.436784+0000","last_unstale":"2024-02-21T23:09:05.426570+0000","last_undegraded":"2024-02-21T23:09:05.426570+0000","last_fullsized":"2024-02-21T23:09:05.426570+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:14:37.913890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.39","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.449289+0000","last_change":"2024-02-21T23:09:04.447811+0000","last_active":"2024-02-21T23:09:05.449289+0000","last_peered":"2024-02-21T23:09:05.449289+0000","last_clean":"2024-02-21T23:09:05.449289+0000","last_became_active":"2024-02-21T23:09:04.444277+0000","last_became_peered":"2024-02-21T23:09:04.444277+0000","last_unstale":"2024-02-21T23:09:05.449289+0000","last_undegraded":"2024-02-21T23:09:05.449289+0000","last_fullsized":"2024-02-21T23:09:05.449289+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:47:47.107317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.38","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.428318+0000","last_change":"2024-02-21T23:09:04.444057+0000","last_active":"2024-02-21T23:09:05.428318+0000","last_peered":"2024-02-21T23:09:05.428318+0000","last_clean":"2024-02-21T23:09:05.428318+0000","last_became_active":"2024-02-21T23:09:04.438783+0000","last_became_peered":"2024-02-21T23:09:04.438783+0000","last_unstale":"2024-02-21T23:09:05.428318+0000","last_undegraded":"2024-02-21T23:09:05.428318+0000","last_fullsized":"2024-02-21T23:09:05.428318+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:41:33.605853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.37","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.426393+0000","last_change":"2024-02-21T23:09:04.437743+0000","last_active":"2024-02-21T23:09:05.426393+0000","last_peered":"2024-02-21T23:09:05.426393+0000","last_clean":"2024-02-21T23:09:05.426393+0000","last_became_active":"2024-02-21T23:09:04.436017+0000","last_became_peered":"2024-02-21T23:09:04.436017+0000","last_unstale":"2024-02-21T23:09:05.426393+0000","last_undegraded":"2024-02-21T23:09:05.426393+0000","last_fullsized":"2024-02-21T23:09:05.426393+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:07:42.091472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.36","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.427100+0000","last_change":"2024-02-21T23:09:04.434833+0000","last_active":"2024-02-21T23:09:05.427100+0000","last_peered":"2024-02-21T23:09:05.427100+0000","last_clean":"2024-02-21T23:09:05.427100+0000","last_became_active":"2024-02-21T23:09:04.434544+0000","last_became_peered":"2024-02-21T23:09:04.434544+0000","last_unstale":"2024-02-21T23:09:05.427100+0000","last_undegraded":"2024-02-21T23:09:05.427100+0000","last_fullsized":"2024-02-21T23:09:05.427100+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:36:53.116212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.35","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.791938+0000","last_change":"2024-02-21T23:09:04.437339+0000","last_active":"2024-02-21T23:09:05.791938+0000","last_peered":"2024-02-21T23:09:05.791938+0000","last_clean":"2024-02-21T23:09:05.791938+0000","last_became_active":"2024-02-21T23:09:04.437003+0000","last_became_peered":"2024-02-21T23:09:04.437003+0000","last_unstale":"2024-02-21T23:09:05.791938+0000","last_undegraded":"2024-02-21T23:09:05.791938+0000","last_fullsized":"2024-02-21T23:09:05.791938+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:16:57.318820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.34","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.794256+0000","last_change":"2024-02-21T23:09:04.429608+0000","last_active":"2024-02-21T23:09:05.794256+0000","last_peered":"2024-02-21T23:09:05.794256+0000","last_clean":"2024-02-21T23:09:05.794256+0000","last_became_active":"2024-02-21T23:09:04.429338+0000","last_became_peered":"2024-02-21T23:09:04.429338+0000","last_unstale":"2024-02-21T23:09:05.794256+0000","last_undegraded":"2024-02-21T23:09:05.794256+0000","last_fullsized":"2024-02-21T23:09:05.794256+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:48:25.691424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.33","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.426826+0000","last_change":"2024-02-21T23:09:04.434802+0000","last_active":"2024-02-21T23:09:05.426826+0000","last_peered":"2024-02-21T23:09:05.426826+0000","last_clean":"2024-02-21T23:09:05.426826+0000","last_became_active":"2024-02-21T23:09:04.434384+0000","last_became_peered":"2024-02-21T23:09:04.434384+0000","last_unstale":"2024-02-21T23:09:05.426826+0000","last_undegraded":"2024-02-21T23:09:05.426826+0000","last_fullsized":"2024-02-21T23:09:05.426826+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:40:12.326060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.32","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.793592+0000","last_change":"2024-02-21T23:09:04.436828+0000","last_active":"2024-02-21T23:09:05.793592+0000","last_peered":"2024-02-21T23:09:05.793592+0000","last_clean":"2024-02-21T23:09:05.793592+0000","last_became_active":"2024-02-21T23:09:04.435612+0000","last_became_peered":"2024-02-21T23:09:04.435612+0000","last_unstale":"2024-02-21T23:09:05.793592+0000","last_undegraded":"2024-02-21T23:09:05.793592+0000","last_fullsized":"2024-02-21T23:09:05.793592+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:03:15.998238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.31","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.432853+0000","last_change":"2024-02-21T23:09:04.417330+0000","last_active":"2024-02-21T23:09:05.432853+0000","last_peered":"2024-02-21T23:09:05.432853+0000","last_clean":"2024-02-21T23:09:05.432853+0000","last_became_active":"2024-02-21T23:09:04.416756+0000","last_became_peered":"2024-02-21T23:09:04.416756+0000","last_unstale":"2024-02-21T23:09:05.432853+0000","last_undegraded":"2024-02-21T23:09:05.432853+0000","last_fullsized":"2024-02-21T23:09:05.432853+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:38:43.748694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.30","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.426099+0000","last_change":"2024-02-21T23:09:04.443913+0000","last_active":"2024-02-21T23:09:05.426099+0000","last_peered":"2024-02-21T23:09:05.426099+0000","last_clean":"2024-02-21T23:09:05.426099+0000","last_became_active":"2024-02-21T23:09:04.443644+0000","last_became_peered":"2024-02-21T23:09:04.443644+0000","last_unstale":"2024-02-21T23:09:05.426099+0000","last_undegraded":"2024-02-21T23:09:05.426099+0000","last_fullsized":"2024-02-21T23:09:05.426099+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:03:00.364236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.2f","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.794201+0000","last_change":"2024-02-21T23:09:04.433260+0000","last_active":"2024-02-21T23:09:05.794201+0000","last_peered":"2024-02-21T23:09:05.794201+0000","last_clean":"2024-02-21T23:09:05.794201+0000","last_became_active":"2024-02-21T23:09:04.432980+0000","last_became_peered":"2024-02-21T23:09:04.432980+0000","last_unstale":"2024-02-21T23:09:05.794201+0000","last_undegraded":"2024-02-21T23:09:05.794201+0000","last_fullsized":"2024-02-21T23:09:05.794201+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:43:38.434476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.2e","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.791833+0000","last_change":"2024-02-21T23:09:04.450702+0000","last_active":"2024-02-21T23:09:05.791833+0000","last_peered":"2024-02-21T23:09:05.791833+0000","last_clean":"2024-02-21T23:09:05.791833+0000","last_became_active":"2024-02-21T23:09:04.449214+0000","last_became_peered":"2024-02-21T23:09:04.449214+0000","last_unstale":"2024-02-21T23:09:05.791833+0000","last_undegraded":"2024-02-21T23:09:05.791833+0000","last_fullsized":"2024-02-21T23:09:05.791833+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:49:25.394550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2d","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.456320+0000","last_change":"2024-02-21T23:09:04.443854+0000","last_active":"2024-02-21T23:09:05.456320+0000","last_peered":"2024-02-21T23:09:05.456320+0000","last_clean":"2024-02-21T23:09:05.456320+0000","last_became_active":"2024-02-21T23:09:04.443566+0000","last_became_peered":"2024-02-21T23:09:04.443566+0000","last_unstale":"2024-02-21T23:09:05.456320+0000","last_undegraded":"2024-02-21T23:09:05.456320+0000","last_fullsized":"2024-02-21T23:09:05.456320+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:41:19.469926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.2c","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.425454+0000","last_change":"2024-02-21T23:09:04.443122+0000","last_active":"2024-02-21T23:09:05.425454+0000","last_peered":"2024-02-21T23:09:05.425454+0000","last_clean":"2024-02-21T23:09:05.425454+0000","last_became_active":"2024-02-21T23:09:04.440509+0000","last_became_peered":"2024-02-21T23:09:04.440509+0000","last_unstale":"2024-02-21T23:09:05.425454+0000","last_undegraded":"2024-02-21T23:09:05.425454+0000","last_fullsized":"2024-02-21T23:09:05.425454+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:47:17.508603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.2b","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.427600+0000","last_change":"2024-02-21T23:09:04.442554+0000","last_active":"2024-02-21T23:09:05.427600+0000","last_peered":"2024-02-21T23:09:05.427600+0000","last_clean":"2024-02-21T23:09:05.427600+0000","last_became_active":"2024-02-21T23:09:04.442201+0000","last_became_peered":"2024-02-21T23:09:04.442201+0000","last_unstale":"2024-02-21T23:09:05.427600+0000","last_undegraded":"2024-02-21T23:09:05.427600+0000","last_fullsized":"2024-02-21T23:09:05.427600+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:51:53.596744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.2a","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.449694+0000","last_change":"2024-02-21T23:09:04.445475+0000","last_active":"2024-02-21T23:09:05.449694+0000","last_peered":"2024-02-21T23:09:05.449694+0000","last_clean":"2024-02-21T23:09:05.449694+0000","last_became_active":"2024-02-21T23:09:04.445131+0000","last_became_peered":"2024-02-21T23:09:04.445131+0000","last_unstale":"2024-02-21T23:09:05.449694+0000","last_undegraded":"2024-02-21T23:09:05.449694+0000","last_fullsized":"2024-02-21T23:09:05.449694+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:39:05.616018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.29","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.428292+0000","last_change":"2024-02-21T23:09:04.442615+0000","last_active":"2024-02-21T23:09:05.428292+0000","last_peered":"2024-02-21T23:09:05.428292+0000","last_clean":"2024-02-21T23:09:05.428292+0000","last_became_active":"2024-02-21T23:09:04.439182+0000","last_became_peered":"2024-02-21T23:09:04.439182+0000","last_unstale":"2024-02-21T23:09:05.428292+0000","last_undegraded":"2024-02-21T23:09:05.428292+0000","last_fullsized":"2024-02-21T23:09:05.428292+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:49:59.061685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.28","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.791848+0000","last_change":"2024-02-21T23:09:04.432498+0000","last_active":"2024-02-21T23:09:05.791848+0000","last_peered":"2024-02-21T23:09:05.791848+0000","last_clean":"2024-02-21T23:09:05.791848+0000","last_became_active":"2024-02-21T23:09:04.432182+0000","last_became_peered":"2024-02-21T23:09:04.432182+0000","last_unstale":"2024-02-21T23:09:05.791848+0000","last_undegraded":"2024-02-21T23:09:05.791848+0000","last_fullsized":"2024-02-21T23:09:05.791848+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:47:59.469506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.27","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.449907+0000","last_change":"2024-02-21T23:09:04.443245+0000","last_active":"2024-02-21T23:09:05.449907+0000","last_peered":"2024-02-21T23:09:05.449907+0000","last_clean":"2024-02-21T23:09:05.449907+0000","last_became_active":"2024-02-21T23:09:04.441112+0000","last_became_peered":"2024-02-21T23:09:04.441112+0000","last_unstale":"2024-02-21T23:09:05.449907+0000","last_undegraded":"2024-02-21T23:09:05.449907+0000","last_fullsized":"2024-02-21T23:09:05.449907+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:35:03.205357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.26","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.449685+0000","last_change":"2024-02-21T23:09:04.437460+0000","last_active":"2024-02-21T23:09:05.449685+0000","last_peered":"2024-02-21T23:09:05.449685+0000","last_clean":"2024-02-21T23:09:05.449685+0000","last_became_active":"2024-02-21T23:09:04.437181+0000","last_became_peered":"2024-02-21T23:09:04.437181+0000","last_unstale":"2024-02-21T23:09:05.449685+0000","last_undegraded":"2024-02-21T23:09:05.449685+0000","last_fullsized":"2024-02-21T23:09:05.449685+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:22:35.300138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.25","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.427310+0000","last_change":"2024-02-21T23:09:04.435176+0000","last_active":"2024-02-21T23:09:05.427310+0000","last_peered":"2024-02-21T23:09:05.427310+0000","last_clean":"2024-02-21T23:09:05.427310+0000","last_became_active":"2024-02-21T23:09:04.434898+0000","last_became_peered":"2024-02-21T23:09:04.434898+0000","last_unstale":"2024-02-21T23:09:05.427310+0000","last_undegraded":"2024-02-21T23:09:05.427310+0000","last_fullsized":"2024-02-21T23:09:05.427310+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:41:30.135576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.24","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.792073+0000","last_change":"2024-02-21T23:09:04.446378+0000","last_active":"2024-02-21T23:09:05.792073+0000","last_peered":"2024-02-21T23:09:05.792073+0000","last_clean":"2024-02-21T23:09:05.792073+0000","last_became_active":"2024-02-21T23:09:04.446067+0000","last_became_peered":"2024-02-21T23:09:04.446067+0000","last_unstale":"2024-02-21T23:09:05.792073+0000","last_undegraded":"2024-02-21T23:09:05.792073+0000","last_fullsized":"2024-02-21T23:09:05.792073+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:02:24.078758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.23","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.430320+0000","last_change":"2024-02-21T23:09:04.432716+0000","last_active":"2024-02-21T23:09:05.430320+0000","last_peered":"2024-02-21T23:09:05.430320+0000","last_clean":"2024-02-21T23:09:05.430320+0000","last_became_active":"2024-02-21T23:09:04.432035+0000","last_became_peered":"2024-02-21T23:09:04.432035+0000","last_unstale":"2024-02-21T23:09:05.430320+0000","last_undegraded":"2024-02-21T23:09:05.430320+0000","last_fullsized":"2024-02-21T23:09:05.430320+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:43:38.434476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.22","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.791701+0000","last_change":"2024-02-21T23:09:04.441919+0000","last_active":"2024-02-21T23:09:05.791701+0000","last_peered":"2024-02-21T23:09:05.791701+0000","last_clean":"2024-02-21T23:09:05.791701+0000","last_became_active":"2024-02-21T23:09:04.441641+0000","last_became_peered":"2024-02-21T23:09:04.441641+0000","last_unstale":"2024-02-21T23:09:05.791701+0000","last_undegraded":"2024-02-21T23:09:05.791701+0000","last_fullsized":"2024-02-21T23:09:05.791701+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:09:29.913398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.21","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.427517+0000","last_change":"2024-02-21T23:09:04.448460+0000","last_active":"2024-02-21T23:09:05.427517+0000","last_peered":"2024-02-21T23:09:05.427517+0000","last_clean":"2024-02-21T23:09:05.427517+0000","last_became_active":"2024-02-21T23:09:04.448065+0000","last_became_peered":"2024-02-21T23:09:04.448065+0000","last_unstale":"2024-02-21T23:09:05.427517+0000","last_undegraded":"2024-02-21T23:09:05.427517+0000","last_fullsized":"2024-02-21T23:09:05.427517+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:48:41.898794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.20","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.428465+0000","last_change":"2024-02-21T23:09:04.443803+0000","last_active":"2024-02-21T23:09:05.428465+0000","last_peered":"2024-02-21T23:09:05.428465+0000","last_clean":"2024-02-21T23:09:05.428465+0000","last_became_active":"2024-02-21T23:09:04.438165+0000","last_became_peered":"2024-02-21T23:09:04.438165+0000","last_unstale":"2024-02-21T23:09:05.428465+0000","last_undegraded":"2024-02-21T23:09:05.428465+0000","last_fullsized":"2024-02-21T23:09:05.428465+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:11:53.081427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.433613+0000","last_change":"2024-02-21T23:09:04.434296+0000","last_active":"2024-02-21T23:09:05.433613+0000","last_peered":"2024-02-21T23:09:05.433613+0000","last_clean":"2024-02-21T23:09:05.433613+0000","last_became_active":"2024-02-21T23:09:04.433998+0000","last_became_peered":"2024-02-21T23:09:04.433998+0000","last_unstale":"2024-02-21T23:09:05.433613+0000","last_undegraded":"2024-02-21T23:09:05.433613+0000","last_fullsized":"2024-02-21T23:09:05.433613+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:35:03.205357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1f","version":"135'4","reported_seq":18,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.426897+0000","last_change":"2024-02-21T23:09:03.399407+0000","last_active":"2024-02-21T23:09:05.426897+0000","last_peered":"2024-02-21T23:09:05.426897+0000","last_clean":"2024-02-21T23:09:05.426897+0000","last_became_active":"2024-02-21T23:09:03.399083+0000","last_became_peered":"2024-02-21T23:09:03.399083+0000","last_unstale":"2024-02-21T23:09:05.426897+0000","last_undegraded":"2024-02-21T23:09:05.426897+0000","last_fullsized":"2024-02-21T23:09:05.426897+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_clean_scrub_stamp":"2024-02-21T23:09:02.355513+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:38:42.732962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.433575+0000","last_change":"2024-02-21T23:09:04.422895+0000","last_active":"2024-02-21T23:09:05.433575+0000","last_peered":"2024-02-21T23:09:05.433575+0000","last_clean":"2024-02-21T23:09:05.433575+0000","last_became_active":"2024-02-21T23:09:04.422319+0000","last_became_peered":"2024-02-21T23:09:04.422319+0000","last_unstale":"2024-02-21T23:09:05.433575+0000","last_undegraded":"2024-02-21T23:09:05.433575+0000","last_fullsized":"2024-02-21T23:09:05.433575+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:19:48.874885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1e","version":"135'1","reported_seq":15,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.455580+0000","last_change":"2024-02-21T23:09:03.417943+0000","last_active":"2024-02-21T23:09:05.455580+0000","last_peered":"2024-02-21T23:09:05.455580+0000","last_clean":"2024-02-21T23:09:05.455580+0000","last_became_active":"2024-02-21T23:09:03.417648+0000","last_became_peered":"2024-02-21T23:09:03.417648+0000","last_unstale":"2024-02-21T23:09:05.455580+0000","last_undegraded":"2024-02-21T23:09:05.455580+0000","last_fullsized":"2024-02-21T23:09:05.455580+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_clean_scrub_stamp":"2024-02-21T23:09:02.355513+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:20:51.138095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.430247+0000","last_change":"2024-02-21T23:09:04.439036+0000","last_active":"2024-02-21T23:09:05.430247+0000","last_peered":"2024-02-21T23:09:05.430247+0000","last_clean":"2024-02-21T23:09:05.430247+0000","last_became_active":"2024-02-21T23:09:04.438681+0000","last_became_peered":"2024-02-21T23:09:04.438681+0000","last_unstale":"2024-02-21T23:09:05.430247+0000","last_undegraded":"2024-02-21T23:09:05.430247+0000","last_fullsized":"2024-02-21T23:09:05.430247+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:14:17.478899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"135'2","reported_seq":16,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.456346+0000","last_change":"2024-02-21T23:09:03.418343+0000","last_active":"2024-02-21T23:09:05.456346+0000","last_peered":"2024-02-21T23:09:05.456346+0000","last_clean":"2024-02-21T23:09:05.456346+0000","last_became_active":"2024-02-21T23:09:03.417877+0000","last_became_peered":"2024-02-21T23:09:03.417877+0000","last_unstale":"2024-02-21T23:09:05.456346+0000","last_undegraded":"2024-02-21T23:09:05.456346+0000","last_fullsized":"2024-02-21T23:09:05.456346+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_clean_scrub_stamp":"2024-02-21T23:09:02.355513+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:43:37.418744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.456039+0000","last_change":"2024-02-21T23:09:04.446487+0000","last_active":"2024-02-21T23:09:05.456039+0000","last_peered":"2024-02-21T23:09:05.456039+0000","last_clean":"2024-02-21T23:09:05.456039+0000","last_became_active":"2024-02-21T23:09:04.446168+0000","last_became_peered":"2024-02-21T23:09:04.446168+0000","last_unstale":"2024-02-21T23:09:05.456039+0000","last_undegraded":"2024-02-21T23:09:05.456039+0000","last_fullsized":"2024-02-21T23:09:05.456039+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:47:57.652729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1c","version":"135'2","reported_seq":16,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.433639+0000","last_change":"2024-02-21T23:09:03.414452+0000","last_active":"2024-02-21T23:09:05.433639+0000","last_peered":"2024-02-21T23:09:05.433639+0000","last_clean":"2024-02-21T23:09:05.433639+0000","last_became_active":"2024-02-21T23:09:03.414158+0000","last_became_peered":"2024-02-21T23:09:03.414158+0000","last_unstale":"2024-02-21T23:09:05.433639+0000","last_undegraded":"2024-02-21T23:09:05.433639+0000","last_fullsized":"2024-02-21T23:09:05.433639+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_clean_scrub_stamp":"2024-02-21T23:09:02.355513+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T11:03:16.672113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.426208+0000","last_change":"2024-02-21T23:09:04.444578+0000","last_active":"2024-02-21T23:09:05.426208+0000","last_peered":"2024-02-21T23:09:05.426208+0000","last_clean":"2024-02-21T23:09:05.426208+0000","last_became_active":"2024-02-21T23:09:04.444281+0000","last_became_peered":"2024-02-21T23:09:04.444281+0000","last_unstale":"2024-02-21T23:09:05.426208+0000","last_undegraded":"2024-02-21T23:09:05.426208+0000","last_fullsized":"2024-02-21T23:09:05.426208+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:36:53.116212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.1b","version":"135'1","reported_seq":15,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.430548+0000","last_change":"2024-02-21T23:09:03.418219+0000","last_active":"2024-02-21T23:09:05.430548+0000","last_peered":"2024-02-21T23:09:05.430548+0000","last_clean":"2024-02-21T23:09:05.430548+0000","last_became_active":"2024-02-21T23:09:03.416184+0000","last_became_peered":"2024-02-21T23:09:03.416184+0000","last_unstale":"2024-02-21T23:09:05.430548+0000","last_undegraded":"2024-02-21T23:09:05.430548+0000","last_fullsized":"2024-02-21T23:09:05.430548+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_clean_scrub_stamp":"2024-02-21T23:09:02.355513+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:21:30.260732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.793398+0000","last_change":"2024-02-21T23:09:04.434581+0000","last_active":"2024-02-21T23:09:05.793398+0000","last_peered":"2024-02-21T23:09:05.793398+0000","last_clean":"2024-02-21T23:09:05.793398+0000","last_became_active":"2024-02-21T23:09:04.432520+0000","last_became_peered":"2024-02-21T23:09:04.432520+0000","last_unstale":"2024-02-21T23:09:05.793398+0000","last_undegraded":"2024-02-21T23:09:05.793398+0000","last_fullsized":"2024-02-21T23:09:05.793398+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:05:44.161417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1a","version":"135'1","reported_seq":15,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.427205+0000","last_change":"2024-02-21T23:09:03.412440+0000","last_active":"2024-02-21T23:09:05.427205+0000","last_peered":"2024-02-21T23:09:05.427205+0000","last_clean":"2024-02-21T23:09:05.427205+0000","last_became_active":"2024-02-21T23:09:03.412105+0000","last_became_peered":"2024-02-21T23:09:03.412105+0000","last_unstale":"2024-02-21T23:09:05.427205+0000","last_undegraded":"2024-02-21T23:09:05.427205+0000","last_fullsized":"2024-02-21T23:09:05.427205+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_clean_scrub_stamp":"2024-02-21T23:09:02.355513+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:41:07.058236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.433777+0000","last_change":"2024-02-21T23:09:04.440651+0000","last_active":"2024-02-21T23:09:05.433777+0000","last_peered":"2024-02-21T23:09:05.433777+0000","last_clean":"2024-02-21T23:09:05.433777+0000","last_became_active":"2024-02-21T23:09:04.440281+0000","last_became_peered":"2024-02-21T23:09:04.440281+0000","last_unstale":"2024-02-21T23:09:05.433777+0000","last_undegraded":"2024-02-21T23:09:05.433777+0000","last_fullsized":"2024-02-21T23:09:05.433777+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:11:06.123295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":14,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.449102+0000","last_change":"2024-02-21T23:09:03.410236+0000","last_active":"2024-02-21T23:09:05.449102+0000","last_peered":"2024-02-21T23:09:05.449102+0000","last_clean":"2024-02-21T23:09:05.449102+0000","last_became_active":"2024-02-21T23:09:03.409955+0000","last_became_peered":"2024-02-21T23:09:03.409955+0000","last_unstale":"2024-02-21T23:09:05.449102+0000","last_undegraded":"2024-02-21T23:09:05.449102+0000","last_fullsized":"2024-02-21T23:09:05.449102+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_clean_scrub_stamp":"2024-02-21T23:09:02.355513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:41:32.590121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.449607+0000","last_change":"2024-02-21T23:09:04.443229+0000","last_active":"2024-02-21T23:09:05.449607+0000","last_peered":"2024-02-21T23:09:05.449607+0000","last_clean":"2024-02-21T23:09:05.449607+0000","last_became_active":"2024-02-21T23:09:04.440615+0000","last_became_peered":"2024-02-21T23:09:04.440615+0000","last_unstale":"2024-02-21T23:09:05.449607+0000","last_undegraded":"2024-02-21T23:09:05.449607+0000","last_fullsized":"2024-02-21T23:09:05.449607+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:41:08.073967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":14,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.426227+0000","last_change":"2024-02-21T23:09:03.413066+0000","last_active":"2024-02-21T23:09:05.426227+0000","last_peered":"2024-02-21T23:09:05.426227+0000","last_clean":"2024-02-21T23:09:05.426227+0000","last_became_active":"2024-02-21T23:09:03.412758+0000","last_became_peered":"2024-02-21T23:09:03.412758+0000","last_unstale":"2024-02-21T23:09:05.426227+0000","last_undegraded":"2024-02-21T23:09:05.426227+0000","last_fullsized":"2024-02-21T23:09:05.426227+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_clean_scrub_stamp":"2024-02-21T23:09:02.355513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:39:46.409525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.429236+0000","last_change":"2024-02-21T23:09:04.443589+0000","last_active":"2024-02-21T23:09:05.429236+0000","last_peered":"2024-02-21T23:09:05.429236+0000","last_clean":"2024-02-21T23:09:05.429236+0000","last_became_active":"2024-02-21T23:09:04.442663+0000","last_became_peered":"2024-02-21T23:09:04.442663+0000","last_unstale":"2024-02-21T23:09:05.429236+0000","last_undegraded":"2024-02-21T23:09:05.429236+0000","last_fullsized":"2024-02-21T23:09:05.429236+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:38:15.236282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":14,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.455918+0000","last_change":"2024-02-21T23:09:03.411990+0000","last_active":"2024-02-21T23:09:05.455918+0000","last_peered":"2024-02-21T23:09:05.455918+0000","last_clean":"2024-02-21T23:09:05.455918+0000","last_became_active":"2024-02-21T23:09:03.411029+0000","last_became_peered":"2024-02-21T23:09:03.411029+0000","last_unstale":"2024-02-21T23:09:05.455918+0000","last_undegraded":"2024-02-21T23:09:05.455918+0000","last_fullsized":"2024-02-21T23:09:05.455918+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_clean_scrub_stamp":"2024-02-21T23:09:02.355513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:48:24.675692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.792151+0000","last_change":"2024-02-21T23:09:04.437531+0000","last_active":"2024-02-21T23:09:05.792151+0000","last_peered":"2024-02-21T23:09:05.792151+0000","last_clean":"2024-02-21T23:09:05.792151+0000","last_became_active":"2024-02-21T23:09:04.435765+0000","last_became_peered":"2024-02-21T23:09:04.435765+0000","last_unstale":"2024-02-21T23:09:05.792151+0000","last_undegraded":"2024-02-21T23:09:05.792151+0000","last_fullsized":"2024-02-21T23:09:05.792151+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:42:29.292918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":14,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.450145+0000","last_change":"2024-02-21T23:09:03.416078+0000","last_active":"2024-02-21T23:09:05.450145+0000","last_peered":"2024-02-21T23:09:05.450145+0000","last_clean":"2024-02-21T23:09:05.450145+0000","last_became_active":"2024-02-21T23:09:03.415768+0000","last_became_peered":"2024-02-21T23:09:03.415768+0000","last_unstale":"2024-02-21T23:09:05.450145+0000","last_undegraded":"2024-02-21T23:09:05.450145+0000","last_fullsized":"2024-02-21T23:09:05.450145+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_clean_scrub_stamp":"2024-02-21T23:09:02.355513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:12:23.809440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.791979+0000","last_change":"2024-02-21T23:09:04.449452+0000","last_active":"2024-02-21T23:09:05.791979+0000","last_peered":"2024-02-21T23:09:05.791979+0000","last_clean":"2024-02-21T23:09:05.791979+0000","last_became_active":"2024-02-21T23:09:04.449109+0000","last_became_peered":"2024-02-21T23:09:04.449109+0000","last_unstale":"2024-02-21T23:09:05.791979+0000","last_undegraded":"2024-02-21T23:09:05.791979+0000","last_fullsized":"2024-02-21T23:09:05.791979+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:02:55.960941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.15","version":"135'3","reported_seq":17,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.792143+0000","last_change":"2024-02-21T23:09:03.399973+0000","last_active":"2024-02-21T23:09:05.792143+0000","last_peered":"2024-02-21T23:09:05.792143+0000","last_clean":"2024-02-21T23:09:05.792143+0000","last_became_active":"2024-02-21T23:09:03.399696+0000","last_became_peered":"2024-02-21T23:09:03.399696+0000","last_unstale":"2024-02-21T23:09:05.792143+0000","last_undegraded":"2024-02-21T23:09:05.792143+0000","last_fullsized":"2024-02-21T23:09:05.792143+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_clean_scrub_stamp":"2024-02-21T23:09:02.355513+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:28:31.009267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.428738+0000","last_change":"2024-02-21T23:09:04.448075+0000","last_active":"2024-02-21T23:09:05.428738+0000","last_peered":"2024-02-21T23:09:05.428738+0000","last_clean":"2024-02-21T23:09:05.428738+0000","last_became_active":"2024-02-21T23:09:04.443441+0000","last_became_peered":"2024-02-21T23:09:04.443441+0000","last_unstale":"2024-02-21T23:09:05.428738+0000","last_undegraded":"2024-02-21T23:09:05.428738+0000","last_fullsized":"2024-02-21T23:09:05.428738+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:12:24.825171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.13","version":"135'2","reported_seq":16,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.427470+0000","last_change":"2024-02-21T23:09:03.392609+0000","last_active":"2024-02-21T23:09:05.427470+0000","last_peered":"2024-02-21T23:09:05.427470+0000","last_clean":"2024-02-21T23:09:05.427470+0000","last_became_active":"2024-02-21T23:09:03.392119+0000","last_became_peered":"2024-02-21T23:09:03.392119+0000","last_unstale":"2024-02-21T23:09:05.427470+0000","last_undegraded":"2024-02-21T23:09:05.427470+0000","last_fullsized":"2024-02-21T23:09:05.427470+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_clean_scrub_stamp":"2024-02-21T23:09:02.355513+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:47:46.091586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.433083+0000","last_change":"2024-02-21T23:09:04.417763+0000","last_active":"2024-02-21T23:09:05.433083+0000","last_peered":"2024-02-21T23:09:05.433083+0000","last_clean":"2024-02-21T23:09:05.433083+0000","last_became_active":"2024-02-21T23:09:04.417047+0000","last_became_peered":"2024-02-21T23:09:04.417047+0000","last_unstale":"2024-02-21T23:09:05.433083+0000","last_undegraded":"2024-02-21T23:09:05.433083+0000","last_fullsized":"2024-02-21T23:09:05.433083+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:35:27.882774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":14,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.793369+0000","last_change":"2024-02-21T23:09:03.421629+0000","last_active":"2024-02-21T23:09:05.793369+0000","last_peered":"2024-02-21T23:09:05.793369+0000","last_clean":"2024-02-21T23:09:05.793369+0000","last_became_active":"2024-02-21T23:09:03.421246+0000","last_became_peered":"2024-02-21T23:09:03.421246+0000","last_unstale":"2024-02-21T23:09:05.793369+0000","last_undegraded":"2024-02-21T23:09:05.793369+0000","last_fullsized":"2024-02-21T23:09:05.793369+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_clean_scrub_stamp":"2024-02-21T23:09:02.355513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:39:28.594527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.427043+0000","last_change":"2024-02-21T23:09:04.442103+0000","last_active":"2024-02-21T23:09:05.427043+0000","last_peered":"2024-02-21T23:09:05.427043+0000","last_clean":"2024-02-21T23:09:05.427043+0000","last_became_active":"2024-02-21T23:09:04.441122+0000","last_became_peered":"2024-02-21T23:09:04.441122+0000","last_unstale":"2024-02-21T23:09:05.427043+0000","last_undegraded":"2024-02-21T23:09:05.427043+0000","last_fullsized":"2024-02-21T23:09:05.427043+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:47:17.508603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":14,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.793769+0000","last_change":"2024-02-21T23:09:03.411660+0000","last_active":"2024-02-21T23:09:05.793769+0000","last_peered":"2024-02-21T23:09:05.793769+0000","last_clean":"2024-02-21T23:09:05.793769+0000","last_became_active":"2024-02-21T23:09:03.411342+0000","last_became_peered":"2024-02-21T23:09:03.411342+0000","last_unstale":"2024-02-21T23:09:05.793769+0000","last_undegraded":"2024-02-21T23:09:05.793769+0000","last_fullsized":"2024-02-21T23:09:05.793769+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_clean_scrub_stamp":"2024-02-21T23:09:02.355513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:32:36.409698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.433341+0000","last_change":"2024-02-21T23:09:04.427876+0000","last_active":"2024-02-21T23:09:05.433341+0000","last_peered":"2024-02-21T23:09:05.433341+0000","last_clean":"2024-02-21T23:09:05.433341+0000","last_became_active":"2024-02-21T23:09:04.426571+0000","last_became_peered":"2024-02-21T23:09:04.426571+0000","last_unstale":"2024-02-21T23:09:05.433341+0000","last_undegraded":"2024-02-21T23:09:05.433341+0000","last_fullsized":"2024-02-21T23:09:05.433341+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:41:08.073967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.10","version":"135'8","reported_seq":22,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.448754+0000","last_change":"2024-02-21T23:09:03.418324+0000","last_active":"2024-02-21T23:09:05.448754+0000","last_peered":"2024-02-21T23:09:05.448754+0000","last_clean":"2024-02-21T23:09:05.448754+0000","last_became_active":"2024-02-21T23:09:03.417617+0000","last_became_peered":"2024-02-21T23:09:03.417617+0000","last_unstale":"2024-02-21T23:09:05.448754+0000","last_undegraded":"2024-02-21T23:09:05.448754+0000","last_fullsized":"2024-02-21T23:09:05.448754+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_clean_scrub_stamp":"2024-02-21T23:09:02.355513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:49:58.045953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.455503+0000","last_change":"2024-02-21T23:09:04.441228+0000","last_active":"2024-02-21T23:09:05.455503+0000","last_peered":"2024-02-21T23:09:05.455503+0000","last_clean":"2024-02-21T23:09:05.455503+0000","last_became_active":"2024-02-21T23:09:04.439293+0000","last_became_peered":"2024-02-21T23:09:04.439293+0000","last_unstale":"2024-02-21T23:09:05.455503+0000","last_undegraded":"2024-02-21T23:09:05.455503+0000","last_fullsized":"2024-02-21T23:09:05.455503+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:11:06.123295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.f","version":"135'1","reported_seq":15,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.791590+0000","last_change":"2024-02-21T23:09:03.410345+0000","last_active":"2024-02-21T23:09:05.791590+0000","last_peered":"2024-02-21T23:09:05.791590+0000","last_clean":"2024-02-21T23:09:05.791590+0000","last_became_active":"2024-02-21T23:09:03.410051+0000","last_became_peered":"2024-02-21T23:09:03.410051+0000","last_unstale":"2024-02-21T23:09:05.791590+0000","last_undegraded":"2024-02-21T23:09:05.791590+0000","last_fullsized":"2024-02-21T23:09:05.791590+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_clean_scrub_stamp":"2024-02-21T23:09:02.355513+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:58:30.059254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.456204+0000","last_change":"2024-02-21T23:09:04.450156+0000","last_active":"2024-02-21T23:09:05.456204+0000","last_peered":"2024-02-21T23:09:05.456204+0000","last_clean":"2024-02-21T23:09:05.456204+0000","last_became_active":"2024-02-21T23:09:04.447844+0000","last_became_peered":"2024-02-21T23:09:04.447844+0000","last_unstale":"2024-02-21T23:09:05.456204+0000","last_undegraded":"2024-02-21T23:09:05.456204+0000","last_fullsized":"2024-02-21T23:09:05.456204+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:11:50.891328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.e","version":"135'2","reported_seq":16,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.455728+0000","last_change":"2024-02-21T23:09:03.410582+0000","last_active":"2024-02-21T23:09:05.455728+0000","last_peered":"2024-02-21T23:09:05.455728+0000","last_clean":"2024-02-21T23:09:05.455728+0000","last_became_active":"2024-02-21T23:09:03.410277+0000","last_became_peered":"2024-02-21T23:09:03.410277+0000","last_unstale":"2024-02-21T23:09:05.455728+0000","last_undegraded":"2024-02-21T23:09:05.455728+0000","last_fullsized":"2024-02-21T23:09:05.455728+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_clean_scrub_stamp":"2024-02-21T23:09:02.355513+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:14:16.463167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.449862+0000","last_change":"2024-02-21T23:09:04.437586+0000","last_active":"2024-02-21T23:09:05.449862+0000","last_peered":"2024-02-21T23:09:05.449862+0000","last_clean":"2024-02-21T23:09:05.449862+0000","last_became_active":"2024-02-21T23:09:04.437302+0000","last_became_peered":"2024-02-21T23:09:04.437302+0000","last_unstale":"2024-02-21T23:09:05.449862+0000","last_undegraded":"2024-02-21T23:09:05.449862+0000","last_fullsized":"2024-02-21T23:09:05.449862+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:19:48.874885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.d","version":"135'1","reported_seq":15,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.433706+0000","last_change":"2024-02-21T23:09:03.412679+0000","last_active":"2024-02-21T23:09:05.433706+0000","last_peered":"2024-02-21T23:09:05.433706+0000","last_clean":"2024-02-21T23:09:05.433706+0000","last_became_active":"2024-02-21T23:09:03.412384+0000","last_became_peered":"2024-02-21T23:09:03.412384+0000","last_unstale":"2024-02-21T23:09:05.433706+0000","last_undegraded":"2024-02-21T23:09:05.433706+0000","last_fullsized":"2024-02-21T23:09:05.433706+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_clean_scrub_stamp":"2024-02-21T23:09:02.355513+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:34:27.316643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.450244+0000","last_change":"2024-02-21T23:09:04.441330+0000","last_active":"2024-02-21T23:09:05.450244+0000","last_peered":"2024-02-21T23:09:05.450244+0000","last_clean":"2024-02-21T23:09:05.450244+0000","last_became_active":"2024-02-21T23:09:04.438866+0000","last_became_peered":"2024-02-21T23:09:04.438866+0000","last_unstale":"2024-02-21T23:09:05.450244+0000","last_undegraded":"2024-02-21T23:09:05.450244+0000","last_fullsized":"2024-02-21T23:09:05.450244+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:47:57.652729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":14,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.794195+0000","last_change":"2024-02-21T23:09:03.422283+0000","last_active":"2024-02-21T23:09:05.794195+0000","last_peered":"2024-02-21T23:09:05.794195+0000","last_clean":"2024-02-21T23:09:05.794195+0000","last_became_active":"2024-02-21T23:09:03.421493+0000","last_became_peered":"2024-02-21T23:09:03.421493+0000","last_unstale":"2024-02-21T23:09:05.794195+0000","last_undegraded":"2024-02-21T23:09:05.794195+0000","last_fullsized":"2024-02-21T23:09:05.794195+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_clean_scrub_stamp":"2024-02-21T23:09:02.355513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:39:15.702164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.455254+0000","last_change":"2024-02-21T23:09:04.452380+0000","last_active":"2024-02-21T23:09:05.455254+0000","last_peered":"2024-02-21T23:09:05.455254+0000","last_clean":"2024-02-21T23:09:05.455254+0000","last_became_active":"2024-02-21T23:09:04.452104+0000","last_became_peered":"2024-02-21T23:09:04.452104+0000","last_unstale":"2024-02-21T23:09:05.455254+0000","last_undegraded":"2024-02-21T23:09:05.455254+0000","last_fullsized":"2024-02-21T23:09:05.455254+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:35:17.242030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.b","version":"135'2","reported_seq":16,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.425641+0000","last_change":"2024-02-21T23:09:03.421193+0000","last_active":"2024-02-21T23:09:05.425641+0000","last_peered":"2024-02-21T23:09:05.425641+0000","last_clean":"2024-02-21T23:09:05.425641+0000","last_became_active":"2024-02-21T23:09:03.420657+0000","last_became_peered":"2024-02-21T23:09:03.420657+0000","last_unstale":"2024-02-21T23:09:05.425641+0000","last_undegraded":"2024-02-21T23:09:05.425641+0000","last_fullsized":"2024-02-21T23:09:05.425641+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_clean_scrub_stamp":"2024-02-21T23:09:02.355513+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:14:36.898158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.433091+0000","last_change":"2024-02-21T23:09:04.437989+0000","last_active":"2024-02-21T23:09:05.433091+0000","last_peered":"2024-02-21T23:09:05.433091+0000","last_clean":"2024-02-21T23:09:05.433091+0000","last_became_active":"2024-02-21T23:09:04.437719+0000","last_became_peered":"2024-02-21T23:09:04.437719+0000","last_unstale":"2024-02-21T23:09:05.433091+0000","last_undegraded":"2024-02-21T23:09:05.433091+0000","last_fullsized":"2024-02-21T23:09:05.433091+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:47:47.107317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":14,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.429554+0000","last_change":"2024-02-21T23:09:03.417938+0000","last_active":"2024-02-21T23:09:05.429554+0000","last_peered":"2024-02-21T23:09:05.429554+0000","last_clean":"2024-02-21T23:09:05.429554+0000","last_became_active":"2024-02-21T23:09:03.413678+0000","last_became_peered":"2024-02-21T23:09:03.413678+0000","last_unstale":"2024-02-21T23:09:05.429554+0000","last_undegraded":"2024-02-21T23:09:05.429554+0000","last_fullsized":"2024-02-21T23:09:05.429554+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_clean_scrub_stamp":"2024-02-21T23:09:02.355513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:47:43.156451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.455082+0000","last_change":"2024-02-21T23:09:04.443806+0000","last_active":"2024-02-21T23:09:05.455082+0000","last_peered":"2024-02-21T23:09:05.455082+0000","last_clean":"2024-02-21T23:09:05.455082+0000","last_became_active":"2024-02-21T23:09:04.443476+0000","last_became_peered":"2024-02-21T23:09:04.443476+0000","last_unstale":"2024-02-21T23:09:05.455082+0000","last_undegraded":"2024-02-21T23:09:05.455082+0000","last_fullsized":"2024-02-21T23:09:05.455082+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:35:03.205357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":14,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.425096+0000","last_change":"2024-02-21T23:09:03.414331+0000","last_active":"2024-02-21T23:09:05.425096+0000","last_peered":"2024-02-21T23:09:05.425096+0000","last_clean":"2024-02-21T23:09:05.425096+0000","last_became_active":"2024-02-21T23:09:03.414026+0000","last_became_peered":"2024-02-21T23:09:03.414026+0000","last_unstale":"2024-02-21T23:09:05.425096+0000","last_undegraded":"2024-02-21T23:09:05.425096+0000","last_fullsized":"2024-02-21T23:09:05.425096+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_clean_scrub_stamp":"2024-02-21T23:09:02.355513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:02:59.348505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.433342+0000","last_change":"2024-02-21T23:09:04.421572+0000","last_active":"2024-02-21T23:09:05.433342+0000","last_peered":"2024-02-21T23:09:05.433342+0000","last_clean":"2024-02-21T23:09:05.433342+0000","last_became_active":"2024-02-21T23:09:04.421131+0000","last_became_peered":"2024-02-21T23:09:04.421131+0000","last_unstale":"2024-02-21T23:09:05.433342+0000","last_undegraded":"2024-02-21T23:09:05.433342+0000","last_fullsized":"2024-02-21T23:09:05.433342+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:22:35.300138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":14,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.429031+0000","last_change":"2024-02-21T23:09:03.408521+0000","last_active":"2024-02-21T23:09:05.429031+0000","last_peered":"2024-02-21T23:09:05.429031+0000","last_clean":"2024-02-21T23:09:05.429031+0000","last_became_active":"2024-02-21T23:09:03.408230+0000","last_became_peered":"2024-02-21T23:09:03.408230+0000","last_unstale":"2024-02-21T23:09:05.429031+0000","last_undegraded":"2024-02-21T23:09:05.429031+0000","last_fullsized":"2024-02-21T23:09:05.429031+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_clean_scrub_stamp":"2024-02-21T23:09:02.355513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:00:20.864214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.426837+0000","last_change":"2024-02-21T23:09:04.440123+0000","last_active":"2024-02-21T23:09:05.426837+0000","last_peered":"2024-02-21T23:09:05.426837+0000","last_clean":"2024-02-21T23:09:05.426837+0000","last_became_active":"2024-02-21T23:09:04.439731+0000","last_became_peered":"2024-02-21T23:09:04.439731+0000","last_unstale":"2024-02-21T23:09:05.426837+0000","last_undegraded":"2024-02-21T23:09:05.426837+0000","last_fullsized":"2024-02-21T23:09:05.426837+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:17:27.624052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.7","version":"135'2","reported_seq":16,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.794193+0000","last_change":"2024-02-21T23:09:03.425221+0000","last_active":"2024-02-21T23:09:05.794193+0000","last_peered":"2024-02-21T23:09:05.794193+0000","last_clean":"2024-02-21T23:09:05.794193+0000","last_became_active":"2024-02-21T23:09:03.424876+0000","last_became_peered":"2024-02-21T23:09:03.424876+0000","last_unstale":"2024-02-21T23:09:05.794193+0000","last_undegraded":"2024-02-21T23:09:05.794193+0000","last_fullsized":"2024-02-21T23:09:05.794193+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_clean_scrub_stamp":"2024-02-21T23:09:02.355513+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:29:50.750441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.794049+0000","last_change":"2024-02-21T23:09:04.435775+0000","last_active":"2024-02-21T23:09:05.794049+0000","last_peered":"2024-02-21T23:09:05.794049+0000","last_clean":"2024-02-21T23:09:05.794049+0000","last_became_active":"2024-02-21T23:09:04.435448+0000","last_became_peered":"2024-02-21T23:09:04.435448+0000","last_unstale":"2024-02-21T23:09:05.794049+0000","last_undegraded":"2024-02-21T23:09:05.794049+0000","last_fullsized":"2024-02-21T23:09:05.794049+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:14:17.478899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"136'1","reported_seq":15,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.453890+0000","last_change":"2024-02-21T23:09:03.417923+0000","last_active":"2024-02-21T23:09:05.453890+0000","last_peered":"2024-02-21T23:09:05.453890+0000","last_clean":"2024-02-21T23:09:05.453890+0000","last_became_active":"2024-02-21T23:09:03.417653+0000","last_became_peered":"2024-02-21T23:09:03.417653+0000","last_unstale":"2024-02-21T23:09:05.453890+0000","last_undegraded":"2024-02-21T23:09:05.453890+0000","last_fullsized":"2024-02-21T23:09:05.453890+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_clean_scrub_stamp":"2024-02-21T23:09:02.355513+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:05:43.145685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.450140+0000","last_change":"2024-02-21T23:09:04.446108+0000","last_active":"2024-02-21T23:09:05.450140+0000","last_peered":"2024-02-21T23:09:05.450140+0000","last_clean":"2024-02-21T23:09:05.450140+0000","last_became_active":"2024-02-21T23:09:04.445822+0000","last_became_peered":"2024-02-21T23:09:04.445822+0000","last_unstale":"2024-02-21T23:09:05.450140+0000","last_undegraded":"2024-02-21T23:09:05.450140+0000","last_fullsized":"2024-02-21T23:09:05.450140+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:11:06.123295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":14,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.433811+0000","last_change":"2024-02-21T23:09:03.421552+0000","last_active":"2024-02-21T23:09:05.433811+0000","last_peered":"2024-02-21T23:09:05.433811+0000","last_clean":"2024-02-21T23:09:05.433811+0000","last_became_active":"2024-02-21T23:09:03.421165+0000","last_became_peered":"2024-02-21T23:09:03.421165+0000","last_unstale":"2024-02-21T23:09:05.433811+0000","last_undegraded":"2024-02-21T23:09:05.433811+0000","last_fullsized":"2024-02-21T23:09:05.433811+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_clean_scrub_stamp":"2024-02-21T23:09:02.355513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:18:46.745293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.426703+0000","last_change":"2024-02-21T23:09:04.426453+0000","last_active":"2024-02-21T23:09:05.426703+0000","last_peered":"2024-02-21T23:09:05.426703+0000","last_clean":"2024-02-21T23:09:05.426703+0000","last_became_active":"2024-02-21T23:09:04.426113+0000","last_became_peered":"2024-02-21T23:09:04.426113+0000","last_unstale":"2024-02-21T23:09:05.426703+0000","last_undegraded":"2024-02-21T23:09:05.426703+0000","last_fullsized":"2024-02-21T23:09:05.426703+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:43:56.054676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":14,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.426969+0000","last_change":"2024-02-21T23:09:03.414797+0000","last_active":"2024-02-21T23:09:05.426969+0000","last_peered":"2024-02-21T23:09:05.426969+0000","last_clean":"2024-02-21T23:09:05.426969+0000","last_became_active":"2024-02-21T23:09:03.414480+0000","last_became_peered":"2024-02-21T23:09:03.414480+0000","last_unstale":"2024-02-21T23:09:05.426969+0000","last_undegraded":"2024-02-21T23:09:05.426969+0000","last_fullsized":"2024-02-21T23:09:05.426969+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_clean_scrub_stamp":"2024-02-21T23:09:02.355513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:35:26.867043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.428889+0000","last_change":"2024-02-21T23:09:04.448429+0000","last_active":"2024-02-21T23:09:05.428889+0000","last_peered":"2024-02-21T23:09:05.428889+0000","last_clean":"2024-02-21T23:09:05.428889+0000","last_became_active":"2024-02-21T23:09:04.445229+0000","last_became_peered":"2024-02-21T23:09:04.445229+0000","last_unstale":"2024-02-21T23:09:05.428889+0000","last_undegraded":"2024-02-21T23:09:05.428889+0000","last_fullsized":"2024-02-21T23:09:05.428889+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:05:44.161417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":14,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.449455+0000","last_change":"2024-02-21T23:09:03.413442+0000","last_active":"2024-02-21T23:09:05.449455+0000","last_peered":"2024-02-21T23:09:05.449455+0000","last_clean":"2024-02-21T23:09:05.449455+0000","last_became_active":"2024-02-21T23:09:03.412202+0000","last_became_peered":"2024-02-21T23:09:03.412202+0000","last_unstale":"2024-02-21T23:09:05.449455+0000","last_undegraded":"2024-02-21T23:09:05.449455+0000","last_fullsized":"2024-02-21T23:09:05.449455+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_clean_scrub_stamp":"2024-02-21T23:09:02.355513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:18:46.745293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.793444+0000","last_change":"2024-02-21T23:09:04.422659+0000","last_active":"2024-02-21T23:09:05.793444+0000","last_peered":"2024-02-21T23:09:05.793444+0000","last_clean":"2024-02-21T23:09:05.793444+0000","last_became_active":"2024-02-21T23:09:04.421925+0000","last_became_peered":"2024-02-21T23:09:04.421925+0000","last_unstale":"2024-02-21T23:09:05.793444+0000","last_undegraded":"2024-02-21T23:09:05.793444+0000","last_fullsized":"2024-02-21T23:09:05.793444+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:38:15.236282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":14,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.791734+0000","last_change":"2024-02-21T23:09:03.421966+0000","last_active":"2024-02-21T23:09:05.791734+0000","last_peered":"2024-02-21T23:09:05.791734+0000","last_clean":"2024-02-21T23:09:05.791734+0000","last_became_active":"2024-02-21T23:09:03.420981+0000","last_became_peered":"2024-02-21T23:09:03.420981+0000","last_unstale":"2024-02-21T23:09:05.791734+0000","last_undegraded":"2024-02-21T23:09:05.791734+0000","last_fullsized":"2024-02-21T23:09:05.791734+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_clean_scrub_stamp":"2024-02-21T23:09:02.355513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:23:45.233493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.433366+0000","last_change":"2024-02-21T23:09:04.435819+0000","last_active":"2024-02-21T23:09:05.433366+0000","last_peered":"2024-02-21T23:09:05.433366+0000","last_clean":"2024-02-21T23:09:05.433366+0000","last_became_active":"2024-02-21T23:09:04.435418+0000","last_became_peered":"2024-02-21T23:09:04.435418+0000","last_unstale":"2024-02-21T23:09:05.433366+0000","last_undegraded":"2024-02-21T23:09:05.433366+0000","last_fullsized":"2024-02-21T23:09:05.433366+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:39:05.616018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":14,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.449375+0000","last_change":"2024-02-21T23:09:03.414973+0000","last_active":"2024-02-21T23:09:05.449375+0000","last_peered":"2024-02-21T23:09:05.449375+0000","last_clean":"2024-02-21T23:09:05.449375+0000","last_became_active":"2024-02-21T23:09:03.412084+0000","last_became_peered":"2024-02-21T23:09:03.412084+0000","last_unstale":"2024-02-21T23:09:05.449375+0000","last_undegraded":"2024-02-21T23:09:05.449375+0000","last_fullsized":"2024-02-21T23:09:05.449375+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_clean_scrub_stamp":"2024-02-21T23:09:02.355513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:11:52.065696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.430121+0000","last_change":"2024-02-21T23:09:04.432964+0000","last_active":"2024-02-21T23:09:05.430121+0000","last_peered":"2024-02-21T23:09:05.430121+0000","last_clean":"2024-02-21T23:09:05.430121+0000","last_became_active":"2024-02-21T23:09:04.432409+0000","last_became_peered":"2024-02-21T23:09:04.432409+0000","last_unstale":"2024-02-21T23:09:05.430121+0000","last_undegraded":"2024-02-21T23:09:05.430121+0000","last_fullsized":"2024-02-21T23:09:05.430121+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:20:52.153827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":14,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.455511+0000","last_change":"2024-02-21T23:09:03.407108+0000","last_active":"2024-02-21T23:09:05.455511+0000","last_peered":"2024-02-21T23:09:05.455511+0000","last_clean":"2024-02-21T23:09:05.455511+0000","last_became_active":"2024-02-21T23:09:03.406087+0000","last_became_peered":"2024-02-21T23:09:03.406087+0000","last_unstale":"2024-02-21T23:09:05.455511+0000","last_undegraded":"2024-02-21T23:09:05.455511+0000","last_fullsized":"2024-02-21T23:09:05.455511+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_clean_scrub_stamp":"2024-02-21T23:09:02.355513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:39:46.409525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":12,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.429351+0000","last_change":"2024-02-21T23:09:04.437750+0000","last_active":"2024-02-21T23:09:05.429351+0000","last_peered":"2024-02-21T23:09:05.429351+0000","last_clean":"2024-02-21T23:09:05.429351+0000","last_became_active":"2024-02-21T23:09:04.435407+0000","last_became_peered":"2024-02-21T23:09:04.435407+0000","last_unstale":"2024-02-21T23:09:05.429351+0000","last_undegraded":"2024-02-21T23:09:05.429351+0000","last_fullsized":"2024-02-21T23:09:05.429351+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":134,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:03.371245+0000","last_clean_scrub_stamp":"2024-02-21T23:09:03.371245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:03:15.998238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.14","version":"135'6","reported_seq":20,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.425493+0000","last_change":"2024-02-21T23:09:03.414900+0000","last_active":"2024-02-21T23:09:05.425493+0000","last_peered":"2024-02-21T23:09:05.425493+0000","last_clean":"2024-02-21T23:09:05.425493+0000","last_became_active":"2024-02-21T23:09:03.414618+0000","last_became_peered":"2024-02-21T23:09:03.414618+0000","last_unstale":"2024-02-21T23:09:05.425493+0000","last_undegraded":"2024-02-21T23:09:05.425493+0000","last_fullsized":"2024-02-21T23:09:05.425493+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":133,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:02.355513+0000","last_clean_scrub_stamp":"2024-02-21T23:09:02.355513+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:07:41.075740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":389,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-02-21T23:09:05.455700+0000","last_change":"2024-02-21T23:02:32.289335+0000","last_active":"2024-02-21T23:09:05.455700+0000","last_peered":"2024-02-21T23:09:05.455700+0000","last_clean":"2024-02-21T23:09:05.455700+0000","last_became_active":"2024-02-21T23:02:32.286913+0000","last_became_peered":"2024-02-21T23:02:32.286913+0000","last_unstale":"2024-02-21T23:09:05.455700+0000","last_undegraded":"2024-02-21T23:09:05.455700+0000","last_fullsized":"2024-02-21T23:09:05.455700+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_clean_scrub_stamp":"2024-02-21T23:02:31.172569+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:36:23.987374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":22,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":21,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574929,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32904,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338936,"statfs":{"total":96636764160,"available":96603070464,"internally_reserved":0,"allocated":1765376,"data_stored":1414799,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574930,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32844,"kb_used_data":1660,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338996,"statfs":{"total":96636764160,"available":96603131904,"internally_reserved":0,"allocated":1699840,"data_stored":824479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574929,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32840,"kb_used_data":1656,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339000,"statfs":{"total":96636764160,"available":96603136000,"internally_reserved":0,"allocated":1695744,"data_stored":824409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574930,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32860,"kb_used_data":1680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338980,"statfs":{"total":96636764160,"available":96603115520,"internally_reserved":0,"allocated":1720320,"data_stored":832713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574930,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32856,"kb_used_data":1676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338984,"statfs":{"total":96636764160,"available":96603119616,"internally_reserved":0,"allocated":1716224,"data_stored":831717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574931,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32852,"kb_used_data":1664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338988,"statfs":{"total":96636764160,"available":96603123712,"internally_reserved":0,"allocated":1703936,"data_stored":824521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574929,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32852,"kb_used_data":1672,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338988,"statfs":{"total":96636764160,"available":96603123712,"internally_reserved":0,"allocated":1712128,"data_stored":831171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574929,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32940,"kb_used_data":1756,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338900,"statfs":{"total":96636764160,"available":96603033600,"internally_reserved":0,"allocated":1798144,"data_stored":1423787,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-21T23:09:09.565 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-02-21T23:09:09.566 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-02-21T23:09:09.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.846+0000 7f53df200640 1 Processor -- start 2024-02-21T23:09:09.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.846+0000 7f53df200640 1 -- start start 2024-02-21T23:09:09.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.847+0000 7f53df200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f53e01500d0 0x7f53e0150560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:09.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.847+0000 7f53df200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f53e0153070 0x7f53e01534c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:09.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.847+0000 7f53df200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53e0070790 con 0x7f53e00c5700 2024-02-21T23:09:09.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.847+0000 7f53df200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53e0150aa0 con 0x7f53e01500d0 2024-02-21T23:09:09.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.847+0000 7f53df200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53e0070a30 con 0x7f53e0153070 2024-02-21T23:09:09.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.848+0000 7f53dde00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f53e00c5700 0x7f53e00c7080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:56880/0 (socket says 172.21.15.28:56880) 2024-02-21T23:09:09.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.848+0000 7f53dd400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f53e01500d0 0x7f53e0150560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:09.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.848+0000 7f53dde00640 1 -- 172.21.15.28:0/3080277763 learned_addr learned my addr 172.21.15.28:0/3080277763 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:09:09.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.848+0000 7f53de800640 1 --2- 172.21.15.28:0/3080277763 >> v2:172.21.15.81:3300/0 conn(0x7f53e0153070 0x7f53e01534c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:09.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.848+0000 7f53dd400640 1 -- 172.21.15.28:0/3080277763 >> v2:172.21.15.81:3300/0 conn(0x7f53e0153070 msgr2=0x7f53e01534c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:09.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.848+0000 7f53dd400640 1 --2- 172.21.15.28:0/3080277763 >> v2:172.21.15.81:3300/0 conn(0x7f53e0153070 0x7f53e01534c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:09.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.848+0000 7f53dca00640 1 -- 172.21.15.28:0/3080277763 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 789638422 0 0) 0x7f53e0070790 con 0x7f53e00c5700 2024-02-21T23:09:09.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.848+0000 7f53dd400640 1 -- 172.21.15.28:0/3080277763 >> v1:172.21.15.28:6789/0 conn(0x7f53e00c5700 legacy=0x7f53e00c7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:09.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.849+0000 7f53dd400640 1 -- 172.21.15.28:0/3080277763 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53e0070d10 con 0x7f53e01500d0 2024-02-21T23:09:09.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.850+0000 7f53dd400640 1 --2- 172.21.15.28:0/3080277763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53e01500d0 0x7f53e0150560 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f53e006ef80 tx=0x7f53cc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5535180a4ca7b089 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:09.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.850+0000 7f53dca00640 1 -- 172.21.15.28:0/3080277763 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53cc0315a0 con 0x7f53e01500d0 2024-02-21T23:09:09.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.850+0000 7f53dca00640 1 -- 172.21.15.28:0/3080277763 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53cc037070 con 0x7f53e01500d0 2024-02-21T23:09:09.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.850+0000 7f53dca00640 1 -- 172.21.15.28:0/3080277763 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53cc031960 con 0x7f53e01500d0 2024-02-21T23:09:09.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.851+0000 7f53df200640 1 -- 172.21.15.28:0/3080277763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53e01500d0 msgr2=0x7f53e0150560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:09.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.851+0000 7f53df200640 1 --2- 172.21.15.28:0/3080277763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53e01500d0 0x7f53e0150560 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f53e006ef80 tx=0x7f53cc02b840 comp rx=0 tx=0).stop 2024-02-21T23:09:09.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.851+0000 7f53df200640 1 -- 172.21.15.28:0/3080277763 shutdown_connections 2024-02-21T23:09:09.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.851+0000 7f53df200640 1 --2- 172.21.15.28:0/3080277763 >> v2:172.21.15.81:3300/0 conn(0x7f53e0153070 0x7f53e01534c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:09.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.851+0000 7f53df200640 1 --2- 172.21.15.28:0/3080277763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53e01500d0 0x7f53e0150560 unknown :-1 s=CLOSED pgs=1535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:09.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.851+0000 7f53df200640 1 -- 172.21.15.28:0/3080277763 >> 172.21.15.28:0/3080277763 conn(0x7f53e006b050 msgr2=0x7f53e006b460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:09.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.851+0000 7f53df200640 1 -- 172.21.15.28:0/3080277763 shutdown_connections 2024-02-21T23:09:09.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.851+0000 7f53df200640 1 -- 172.21.15.28:0/3080277763 wait complete. 2024-02-21T23:09:09.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.853+0000 7f53df200640 1 Processor -- start 2024-02-21T23:09:09.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.853+0000 7f53df200640 1 -- start start 2024-02-21T23:09:09.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.854+0000 7f53df200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53e00c5700 0x7f53e01526f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:09.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.854+0000 7f53df200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f53e01500d0 0x7f53e014f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:09.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.854+0000 7f53df200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f53e0153070 0x7f53e00ac850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:09.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.854+0000 7f53df200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53e00c7790 con 0x7f53e00c5700 2024-02-21T23:09:09.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.854+0000 7f53df200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53e00712d0 con 0x7f53e0153070 2024-02-21T23:09:09.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.854+0000 7f53df200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53e00c7df0 con 0x7f53e01500d0 2024-02-21T23:09:09.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.854+0000 7f53dde00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53e00c5700 0x7f53e01526f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:09.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.854+0000 7f53dde00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53e00c5700 0x7f53e01526f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57136/0 (socket says 172.21.15.28:57136) 2024-02-21T23:09:09.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.854+0000 7f53dde00640 1 -- 172.21.15.28:0/108663868 learned_addr learned my addr 172.21.15.28:0/108663868 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:09.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.854+0000 7f53de800640 1 --2- 172.21.15.28:0/108663868 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f53e0153070 0x7f53e00ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:09.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.854+0000 7f53dd400640 1 --2- 172.21.15.28:0/108663868 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f53e01500d0 0x7f53e014f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:09.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.854+0000 7f53dde00640 1 -- 172.21.15.28:0/108663868 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f53e01500d0 msgr2=0x7f53e014f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:09.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.855+0000 7f53dde00640 1 --2- 172.21.15.28:0/108663868 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f53e01500d0 0x7f53e014f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:09.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.855+0000 7f53dde00640 1 -- 172.21.15.28:0/108663868 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f53e0153070 msgr2=0x7f53e00ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:09.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.855+0000 7f53dde00640 1 --2- 172.21.15.28:0/108663868 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f53e0153070 0x7f53e00ac850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:09.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.855+0000 7f53dde00640 1 -- 172.21.15.28:0/108663868 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53e0146c80 con 0x7f53e00c5700 2024-02-21T23:09:09.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.855+0000 7f53de800640 1 --2- 172.21.15.28:0/108663868 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f53e0153070 0x7f53e00ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:09:09.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.855+0000 7f53dd400640 1 --2- 172.21.15.28:0/108663868 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f53e01500d0 0x7f53e014f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:09:09.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.855+0000 7f53dde00640 1 --2- 172.21.15.28:0/108663868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53e00c5700 0x7f53e01526f0 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f53d000c8c0 tx=0x7f53d000cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:09.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.855+0000 7f53d6000640 1 -- 172.21.15.28:0/108663868 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d0018070 con 0x7f53e00c5700 2024-02-21T23:09:09.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.855+0000 7f53d6000640 1 -- 172.21.15.28:0/108663868 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53d000a5e0 con 0x7f53e00c5700 2024-02-21T23:09:09.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.855+0000 7f53df200640 1 -- 172.21.15.28:0/108663868 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53e0151750 con 0x7f53e00c5700 2024-02-21T23:09:09.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.855+0000 7f53d6000640 1 -- 172.21.15.28:0/108663868 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d000a9d0 con 0x7f53e00c5700 2024-02-21T23:09:09.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.855+0000 7f53df200640 1 -- 172.21.15.28:0/108663868 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53e014fa10 con 0x7f53e00c5700 2024-02-21T23:09:09.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.857+0000 7f53df200640 1 -- 172.21.15.28:0/108663868 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53e0151540 con 0x7f53e00c5700 2024-02-21T23:09:09.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.858+0000 7f53d6000640 1 -- 172.21.15.28:0/108663868 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f53d001b070 con 0x7f53e00c5700 2024-02-21T23:09:09.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.859+0000 7f53d6000640 1 --2- 172.21.15.28:0/108663868 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f53c006b4d0 0x7f53c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:09.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.859+0000 7f53d6000640 1 -- 172.21.15.28:0/108663868 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f53d0089a70 con 0x7f53e00c5700 2024-02-21T23:09:09.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.859+0000 7f53dd400640 1 --2- 172.21.15.28:0/108663868 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f53c006b4d0 0x7f53c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:09.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.860+0000 7f53dd400640 1 --2- 172.21.15.28:0/108663868 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f53c006b4d0 0x7f53c006d990 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f53cc036000 tx=0x7f53cc008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:09.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:09.863+0000 7f53d6000640 1 -- 172.21.15.28:0/108663868 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f53d0058750 con 0x7f53e00c5700 2024-02-21T23:09:10.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.224+0000 7f53df200640 1 -- 172.21.15.28:0/108663868 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f53e004f290 con 0x7f53e00c5700 2024-02-21T23:09:10.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.229+0000 7f53d6000640 1 -- 172.21.15.28:0/108663868 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v17) v1 ==== 190+0+147 (secure 0 0 0) 0x7f53d0058050 con 0x7f53e00c5700 2024-02-21T23:09:10.217 INFO:teuthology.orchestra.run.smithi028.stdout:[client.0] 2024-02-21T23:09:10.217 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQDTgNZlILvYLxAAoREWxKKT7YcXnhkgk8YvoQ== 2024-02-21T23:09:10.217 INFO:teuthology.orchestra.run.smithi028.stdout: caps mds = "allow" 2024-02-21T23:09:10.217 INFO:teuthology.orchestra.run.smithi028.stdout: caps mon = "allow r" 2024-02-21T23:09:10.217 INFO:teuthology.orchestra.run.smithi028.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-02-21T23:09:10.217 INFO:teuthology.orchestra.run.smithi028.stderr:updated caps for client.0 2024-02-21T23:09:10.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.232+0000 7f53df200640 1 -- 172.21.15.28:0/108663868 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f53c006b4d0 msgr2=0x7f53c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:10.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.232+0000 7f53df200640 1 --2- 172.21.15.28:0/108663868 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f53c006b4d0 0x7f53c006d990 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f53cc036000 tx=0x7f53cc008030 comp rx=0 tx=0).stop 2024-02-21T23:09:10.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.232+0000 7f53df200640 1 -- 172.21.15.28:0/108663868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53e00c5700 msgr2=0x7f53e01526f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:10.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.232+0000 7f53df200640 1 --2- 172.21.15.28:0/108663868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53e00c5700 0x7f53e01526f0 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f53d000c8c0 tx=0x7f53d000cd80 comp rx=0 tx=0).stop 2024-02-21T23:09:10.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.232+0000 7f53df200640 1 -- 172.21.15.28:0/108663868 shutdown_connections 2024-02-21T23:09:10.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.232+0000 7f53df200640 1 --2- 172.21.15.28:0/108663868 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f53c006b4d0 0x7f53c006d990 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:10.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.232+0000 7f53df200640 1 --2- 172.21.15.28:0/108663868 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f53e01500d0 0x7f53e014f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:10.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.232+0000 7f53df200640 1 --2- 172.21.15.28:0/108663868 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f53e0153070 0x7f53e00ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:10.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.232+0000 7f53df200640 1 --2- 172.21.15.28:0/108663868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53e00c5700 0x7f53e01526f0 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:10.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.232+0000 7f53df200640 1 -- 172.21.15.28:0/108663868 >> 172.21.15.28:0/108663868 conn(0x7f53e006b050 msgr2=0x7f53e006b430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:10.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.232+0000 7f53df200640 1 -- 172.21.15.28:0/108663868 shutdown_connections 2024-02-21T23:09:10.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.232+0000 7f53df200640 1 -- 172.21.15.28:0/108663868 wait complete. 2024-02-21T23:09:10.249 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:09:10.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.479+0000 7faae5a00640 1 Processor -- start 2024-02-21T23:09:10.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.480+0000 7faae5a00640 1 -- start start 2024-02-21T23:09:10.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.481+0000 7faae5a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7faae013e620 0x7faae0159070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:10.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.481+0000 7faae5a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faae00bf990 0x7faae00bfe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:10.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.481+0000 7faae5a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faae01551b0 con 0x7faae00c6460 2024-02-21T23:09:10.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.481+0000 7faae5a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faae00c71d0 con 0x7faae013e620 2024-02-21T23:09:10.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.481+0000 7faae5a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faae01595b0 con 0x7faae00bf990 2024-02-21T23:09:10.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.481+0000 7faadea00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7faae013e620 0x7faae0159070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:10.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.482+0000 7faadea00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7faae013e620 0x7faae0159070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57146/0 (socket says 172.21.15.28:57146) 2024-02-21T23:09:10.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.482+0000 7faadea00640 1 -- 172.21.15.28:0/930539583 learned_addr learned my addr 172.21.15.28:0/930539583 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:10.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.482+0000 7faadea00640 1 -- 172.21.15.28:0/930539583 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faae00bf990 msgr2=0x7faae00bfe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:10.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.482+0000 7faadea00640 1 --2- 172.21.15.28:0/930539583 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faae00bf990 0x7faae00bfe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:10.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.482+0000 7faadea00640 1 -- 172.21.15.28:0/930539583 >> v1:172.21.15.81:6789/0 conn(0x7faae00c6460 legacy=0x7faae00ca8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:10.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.482+0000 7faadea00640 1 -- 172.21.15.28:0/930539583 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faae0155420 con 0x7faae013e620 2024-02-21T23:09:10.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.483+0000 7faadea00640 1 --2- 172.21.15.28:0/930539583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faae013e620 0x7faae0159070 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7faad0009870 tx=0x7faad002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adc641c3aef8cc73 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:10.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.483+0000 7faade000640 1 -- 172.21.15.28:0/930539583 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faad00317c0 con 0x7faae013e620 2024-02-21T23:09:10.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.484+0000 7faade000640 1 -- 172.21.15.28:0/930539583 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faad0037070 con 0x7faae013e620 2024-02-21T23:09:10.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.484+0000 7faade000640 1 -- 172.21.15.28:0/930539583 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faad0031b80 con 0x7faae013e620 2024-02-21T23:09:10.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.484+0000 7faae5a00640 1 -- 172.21.15.28:0/930539583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faae013e620 msgr2=0x7faae0159070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:10.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.484+0000 7faae5a00640 1 --2- 172.21.15.28:0/930539583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faae013e620 0x7faae0159070 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7faad0009870 tx=0x7faad002b860 comp rx=0 tx=0).stop 2024-02-21T23:09:10.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.484+0000 7faae5a00640 1 -- 172.21.15.28:0/930539583 shutdown_connections 2024-02-21T23:09:10.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.484+0000 7faae5a00640 1 --2- 172.21.15.28:0/930539583 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faae00bf990 0x7faae00bfe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:10.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.484+0000 7faae5a00640 1 --2- 172.21.15.28:0/930539583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faae013e620 0x7faae0159070 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:10.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.484+0000 7faae5a00640 1 -- 172.21.15.28:0/930539583 >> 172.21.15.28:0/930539583 conn(0x7faae008b1c0 msgr2=0x7faae00725a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:10.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.484+0000 7faae5a00640 1 -- 172.21.15.28:0/930539583 shutdown_connections 2024-02-21T23:09:10.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.484+0000 7faae5a00640 1 -- 172.21.15.28:0/930539583 wait complete. 2024-02-21T23:09:10.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.486+0000 7faae5a00640 1 Processor -- start 2024-02-21T23:09:10.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.486+0000 7faae5a00640 1 -- start start 2024-02-21T23:09:10.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.487+0000 7faae5a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faae00bf990 0x7faae0158830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:10.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.487+0000 7faae5a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faae00c6460 0x7faae01e0f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:10.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.487+0000 7faae5a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faae013e620 0x7faae0157360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:10.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.487+0000 7faae5a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae0155cc0 con 0x7faae00c6460 2024-02-21T23:09:10.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.487+0000 7faae5a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae01559e0 con 0x7faae00bf990 2024-02-21T23:09:10.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.487+0000 7faae5a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faae013ea00 con 0x7faae013e620 2024-02-21T23:09:10.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.487+0000 7faad7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faae00c6460 0x7faae01e0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:10.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.487+0000 7faad7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faae00c6460 0x7faae01e0f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57156/0 (socket says 172.21.15.28:57156) 2024-02-21T23:09:10.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.487+0000 7faad7e00640 1 -- 172.21.15.28:0/1968676047 learned_addr learned my addr 172.21.15.28:0/1968676047 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:10.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.488+0000 7faadea00640 1 --2- 172.21.15.28:0/1968676047 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faae00bf990 0x7faae0158830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:10.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.488+0000 7faadf400640 1 --2- 172.21.15.28:0/1968676047 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faae013e620 0x7faae0157360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:10.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.488+0000 7faad7e00640 1 -- 172.21.15.28:0/1968676047 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faae013e620 msgr2=0x7faae0157360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:10.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.488+0000 7faad7e00640 1 --2- 172.21.15.28:0/1968676047 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faae013e620 0x7faae0157360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:10.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.488+0000 7faad7e00640 1 -- 172.21.15.28:0/1968676047 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faae00bf990 msgr2=0x7faae0158830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:10.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.488+0000 7faad7e00640 1 --2- 172.21.15.28:0/1968676047 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faae00bf990 0x7faae0158830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:10.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.488+0000 7faad7e00640 1 -- 172.21.15.28:0/1968676047 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faae01de570 con 0x7faae00c6460 2024-02-21T23:09:10.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.488+0000 7faad7e00640 1 --2- 172.21.15.28:0/1968676047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faae00c6460 0x7faae01e0f80 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7faac800c410 tx=0x7faac800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:10.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.488+0000 7faad6a00640 1 -- 172.21.15.28:0/1968676047 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faac8017070 con 0x7faae00c6460 2024-02-21T23:09:10.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.488+0000 7faad6a00640 1 -- 172.21.15.28:0/1968676047 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faac80035f0 con 0x7faae00c6460 2024-02-21T23:09:10.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.489+0000 7faad6a00640 1 -- 172.21.15.28:0/1968676047 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faac80039e0 con 0x7faae00c6460 2024-02-21T23:09:10.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.489+0000 7faae5a00640 1 -- 172.21.15.28:0/1968676047 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faae0157fe0 con 0x7faae00c6460 2024-02-21T23:09:10.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.489+0000 7faae5a00640 1 -- 172.21.15.28:0/1968676047 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faae00b0290 con 0x7faae00c6460 2024-02-21T23:09:10.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.490+0000 7faae5a00640 1 -- 172.21.15.28:0/1968676047 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faae0067f30 con 0x7faae00c6460 2024-02-21T23:09:10.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.491+0000 7faad6a00640 1 -- 172.21.15.28:0/1968676047 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faac801a070 con 0x7faae00c6460 2024-02-21T23:09:10.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.492+0000 7faad6a00640 1 --2- 172.21.15.28:0/1968676047 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7faabc06b4d0 0x7faabc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:10.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.492+0000 7faad6a00640 1 -- 172.21.15.28:0/1968676047 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 6101+0+0 (secure 0 0 0) 0x7faac8089830 con 0x7faae00c6460 2024-02-21T23:09:10.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.493+0000 7faadea00640 1 --2- 172.21.15.28:0/1968676047 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7faabc06b4d0 0x7faabc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:10.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.494+0000 7faadea00640 1 --2- 172.21.15.28:0/1968676047 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7faabc06b4d0 0x7faabc06d990 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7faad0036000 tx=0x7faad003d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:10.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.496+0000 7faad6a00640 1 -- 172.21.15.28:0/1968676047 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7faac8058510 con 0x7faae00c6460 2024-02-21T23:09:10.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.734+0000 7faae5a00640 1 -- 172.21.15.28:0/1968676047 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faae0158590 con 0x7faae00c6460 2024-02-21T23:09:10.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.735+0000 7faad6a00640 1 -- 172.21.15.28:0/1968676047 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 139 v139) v1 ==== 95+0+4948 (secure 0 0 0) 0x7faac8057e10 con 0x7faae00c6460 2024-02-21T23:09:10.723 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:09:10.723 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":139,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6140,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/4241906475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":4241906475},{"type":"v1","addr":"172.21.15.81:6836","nonce":4241906475}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":133},{"gid":6143,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/2095908433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":2095908433},{"type":"v1","addr":"172.21.15.81:6837","nonce":2095908433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":133},{"gid":8370,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/3934548735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3934548735},{"type":"v1","addr":"172.21.15.28:6833","nonce":3934548735}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":133}],"filesystems":[{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:09:03.869294+0000","modified":"2024-02-21T23:09:07.201932+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8376},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8376":{"gid":8376,"name":"a","rank":0,"incarnation":135,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/4151254007","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4151254007},{"type":"v1","addr":"172.21.15.28:6835","nonce":4151254007}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-02-21T23:09:10.723 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 139 2024-02-21T23:09:10.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.738+0000 7faae5a00640 1 -- 172.21.15.28:0/1968676047 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7faabc06b4d0 msgr2=0x7faabc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:10.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.738+0000 7faae5a00640 1 --2- 172.21.15.28:0/1968676047 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7faabc06b4d0 0x7faabc06d990 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7faad0036000 tx=0x7faad003d040 comp rx=0 tx=0).stop 2024-02-21T23:09:10.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.738+0000 7faae5a00640 1 -- 172.21.15.28:0/1968676047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faae00c6460 msgr2=0x7faae01e0f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:10.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.738+0000 7faae5a00640 1 --2- 172.21.15.28:0/1968676047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faae00c6460 0x7faae01e0f80 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7faac800c410 tx=0x7faac800c8d0 comp rx=0 tx=0).stop 2024-02-21T23:09:10.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.739+0000 7faae5a00640 1 -- 172.21.15.28:0/1968676047 shutdown_connections 2024-02-21T23:09:10.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.739+0000 7faae5a00640 1 --2- 172.21.15.28:0/1968676047 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7faabc06b4d0 0x7faabc06d990 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:10.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.739+0000 7faae5a00640 1 --2- 172.21.15.28:0/1968676047 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faae013e620 0x7faae0157360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:10.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.739+0000 7faae5a00640 1 --2- 172.21.15.28:0/1968676047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faae00c6460 0x7faae01e0f80 unknown :-1 s=CLOSED pgs=1538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:10.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.739+0000 7faae5a00640 1 --2- 172.21.15.28:0/1968676047 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faae00bf990 0x7faae0158830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:10.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.739+0000 7faae5a00640 1 -- 172.21.15.28:0/1968676047 >> 172.21.15.28:0/1968676047 conn(0x7faae008b1c0 msgr2=0x7faae00c7350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:10.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.739+0000 7faae5a00640 1 -- 172.21.15.28:0/1968676047 shutdown_connections 2024-02-21T23:09:10.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:10.739+0000 7faae5a00640 1 -- 172.21.15.28:0/1968676047 wait complete. 2024-02-21T23:09:11.157 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 139, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:09:03.869294+0000', 'modified': '2024-02-21T23:09:07.201932+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8376}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8376': {'gid': 8376, 'name': 'a', 'rank': 0, 'incarnation': 135, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/4151254007', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 4151254007}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 4151254007}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:09:11.157 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-21T23:09:11.157 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-02-21T23:09:11.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.399+0000 7f7341a00640 1 Processor -- start 2024-02-21T23:09:11.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.399+0000 7f7341a00640 1 -- start start 2024-02-21T23:09:11.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.400+0000 7f7341a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f733c0c43e0 0x7f733c0a8e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:11.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.400+0000 7f7341a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f733c149c10 con 0x7f733c149830 2024-02-21T23:09:11.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.400+0000 7f7341a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f733c1537d0 con 0x7f733c06d410 2024-02-21T23:09:11.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.400+0000 7f7341a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f733c153ab0 con 0x7f733c0c43e0 2024-02-21T23:09:11.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.401+0000 7f733be00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f733c149830 0x7f733c0c0ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:56890/0 (socket says 172.21.15.28:56890) 2024-02-21T23:09:11.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.401+0000 7f733be00640 1 -- 172.21.15.28:0/59449915 learned_addr learned my addr 172.21.15.28:0/59449915 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:09:11.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.401+0000 7f7341000640 1 --2- 172.21.15.28:0/59449915 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f733c0c43e0 0x7f733c0a8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:11.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.401+0000 7f733aa00640 1 -- 172.21.15.28:0/59449915 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1304252965 0 0) 0x7f733c149c10 con 0x7f733c149830 2024-02-21T23:09:11.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.401+0000 7f733aa00640 1 -- 172.21.15.28:0/59449915 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7320003520 con 0x7f733c149830 2024-02-21T23:09:11.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.402+0000 7f733aa00640 1 -- 172.21.15.28:0/59449915 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2713502103 0 0) 0x7f733c1537d0 con 0x7f733c06d410 2024-02-21T23:09:11.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.402+0000 7f733aa00640 1 -- 172.21.15.28:0/59449915 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f733c149c10 con 0x7f733c06d410 2024-02-21T23:09:11.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.402+0000 7f733aa00640 1 -- 172.21.15.28:0/59449915 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 28599545 0 0) 0x7f7320003520 con 0x7f733c149830 2024-02-21T23:09:11.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.402+0000 7f733aa00640 1 -- 172.21.15.28:0/59449915 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f733c0c43e0 msgr2=0x7f733c0a8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:11.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.402+0000 7f733aa00640 1 --2- 172.21.15.28:0/59449915 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f733c0c43e0 0x7f733c0a8e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:11.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.402+0000 7f733aa00640 1 -- 172.21.15.28:0/59449915 >> v1:172.21.15.81:6789/0 conn(0x7f733c06d410 legacy=0x7f733c0c5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:11.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.402+0000 7f733aa00640 1 -- 172.21.15.28:0/59449915 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f733c0c4fa0 con 0x7f733c149830 2024-02-21T23:09:11.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.402+0000 7f733aa00640 1 -- 172.21.15.28:0/59449915 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f732c001670 con 0x7f733c149830 2024-02-21T23:09:11.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.402+0000 7f733aa00640 1 -- 172.21.15.28:0/59449915 >> v1:172.21.15.28:6789/0 conn(0x7f733c149830 legacy=0x7f733c0c0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:11.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.402+0000 7f733aa00640 1 --2- 172.21.15.28:0/59449915 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7320003b70 0x7f7320003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:11.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.402+0000 7f733aa00640 1 --2- 172.21.15.28:0/59449915 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7320004800 0x7f7320004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:11.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.402+0000 7f733aa00640 1 --2- 172.21.15.28:0/59449915 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f733c0c43e0 0x7f73200051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:11.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.402+0000 7f733aa00640 1 -- 172.21.15.28:0/59449915 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f733c153ab0 con 0x7f7320003b70 2024-02-21T23:09:11.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.402+0000 7f733aa00640 1 -- 172.21.15.28:0/59449915 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f732000ba70 con 0x7f7320004800 2024-02-21T23:09:11.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.402+0000 7f733aa00640 1 -- 172.21.15.28:0/59449915 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f732000bc40 con 0x7f733c0c43e0 2024-02-21T23:09:11.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.402+0000 7f733be00640 1 --2- 172.21.15.28:0/59449915 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7320003b70 0x7f7320003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:11.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.403+0000 7f7341000640 1 --2- 172.21.15.28:0/59449915 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f733c0c43e0 0x7f73200051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:11.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.403+0000 7f733be00640 1 -- 172.21.15.28:0/59449915 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f733c0c43e0 msgr2=0x7f73200051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:11.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.403+0000 7f733b400640 1 --2- 172.21.15.28:0/59449915 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7320004800 0x7f7320004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:11.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.403+0000 7f733be00640 1 --2- 172.21.15.28:0/59449915 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f733c0c43e0 0x7f73200051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:11.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.403+0000 7f733be00640 1 -- 172.21.15.28:0/59449915 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7320004800 msgr2=0x7f7320004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:11.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.403+0000 7f733be00640 1 --2- 172.21.15.28:0/59449915 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7320004800 0x7f7320004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:11.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.403+0000 7f733be00640 1 -- 172.21.15.28:0/59449915 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7320003520 con 0x7f7320003b70 2024-02-21T23:09:11.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.403+0000 7f733b400640 1 --2- 172.21.15.28:0/59449915 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7320004800 0x7f7320004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:09:11.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.404+0000 7f733be00640 1 --2- 172.21.15.28:0/59449915 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7320003b70 0x7f7320003f70 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7f732c00c190 tx=0x7f7320005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb237c8a4100b347 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:11.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.404+0000 7f733aa00640 1 -- 172.21.15.28:0/59449915 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f732c007ce0 con 0x7f7320003b70 2024-02-21T23:09:11.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.404+0000 7f733aa00640 1 -- 172.21.15.28:0/59449915 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f732c030070 con 0x7f7320003b70 2024-02-21T23:09:11.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.404+0000 7f733aa00640 1 -- 172.21.15.28:0/59449915 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f732c005dd0 con 0x7f7320003b70 2024-02-21T23:09:11.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.405+0000 7f7341a00640 1 -- 172.21.15.28:0/59449915 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7320003b70 msgr2=0x7f7320003f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:11.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.405+0000 7f7341a00640 1 --2- 172.21.15.28:0/59449915 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7320003b70 0x7f7320003f70 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7f732c00c190 tx=0x7f7320005830 comp rx=0 tx=0).stop 2024-02-21T23:09:11.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.405+0000 7f7341000640 1 -- 172.21.15.28:0/59449915 reap_dead start 2024-02-21T23:09:11.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.405+0000 7f7341a00640 1 -- 172.21.15.28:0/59449915 shutdown_connections 2024-02-21T23:09:11.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.405+0000 7f7341a00640 1 -- 172.21.15.28:0/59449915 >> 172.21.15.28:0/59449915 conn(0x7f733c0a8320 msgr2=0x7f733c0a8730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:11.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.405+0000 7f7341a00640 1 -- 172.21.15.28:0/59449915 shutdown_connections 2024-02-21T23:09:11.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.405+0000 7f7341a00640 1 -- 172.21.15.28:0/59449915 wait complete. 2024-02-21T23:09:11.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.407+0000 7f7341a00640 1 Processor -- start 2024-02-21T23:09:11.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.407+0000 7f7341a00640 1 -- start start 2024-02-21T23:09:11.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.408+0000 7f7341a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7320003b70 0x7f733c1ea640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:11.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.408+0000 7f7341a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f733c0a9620 0x7f733c0aabc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:11.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.408+0000 7f7341a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f733c1502a0 0x7f733c0abd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:11.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.408+0000 7f7341a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f733c0c15f0 con 0x7f733c0a9620 2024-02-21T23:09:11.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.408+0000 7f7341a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f733c0c5560 con 0x7f733c1502a0 2024-02-21T23:09:11.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.408+0000 7f7341a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f733c0c1770 con 0x7f7320003b70 2024-02-21T23:09:11.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.408+0000 7f733b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f733c0a9620 0x7f733c0aabc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:11.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.408+0000 7f733b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f733c0a9620 0x7f733c0aabc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57178/0 (socket says 172.21.15.28:57178) 2024-02-21T23:09:11.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.408+0000 7f733b400640 1 -- 172.21.15.28:0/1432010160 learned_addr learned my addr 172.21.15.28:0/1432010160 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:11.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.408+0000 7f733be00640 1 --2- 172.21.15.28:0/1432010160 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7320003b70 0x7f733c1ea640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:11.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.408+0000 7f7341000640 1 --2- 172.21.15.28:0/1432010160 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f733c1502a0 0x7f733c0abd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:11.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.409+0000 7f733b400640 1 -- 172.21.15.28:0/1432010160 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7320003b70 msgr2=0x7f733c1ea640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:11.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.409+0000 7f733b400640 1 --2- 172.21.15.28:0/1432010160 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7320003b70 0x7f733c1ea640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:11.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.409+0000 7f733b400640 1 -- 172.21.15.28:0/1432010160 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f733c1502a0 msgr2=0x7f733c0abd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:11.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.409+0000 7f733b400640 1 --2- 172.21.15.28:0/1432010160 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f733c1502a0 0x7f733c0abd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:11.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.409+0000 7f733b400640 1 -- 172.21.15.28:0/1432010160 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f733c0a9b20 con 0x7f733c0a9620 2024-02-21T23:09:11.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.409+0000 7f733b400640 1 --2- 172.21.15.28:0/1432010160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f733c0a9620 0x7f733c0aabc0 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7f732800d7a0 tx=0x7f732800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:11.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.409+0000 7f7333e00640 1 -- 172.21.15.28:0/1432010160 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7328016070 con 0x7f733c0a9620 2024-02-21T23:09:11.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.409+0000 7f7333e00640 1 -- 172.21.15.28:0/1432010160 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f732800ba60 con 0x7f733c0a9620 2024-02-21T23:09:11.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.409+0000 7f7333e00640 1 -- 172.21.15.28:0/1432010160 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f732800be50 con 0x7f733c0a9620 2024-02-21T23:09:11.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.410+0000 7f7341a00640 1 -- 172.21.15.28:0/1432010160 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f733c0a9e00 con 0x7f733c0a9620 2024-02-21T23:09:11.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.410+0000 7f7341a00640 1 -- 172.21.15.28:0/1432010160 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f733c0aa3c0 con 0x7f733c0a9620 2024-02-21T23:09:11.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.413+0000 7f7333e00640 1 -- 172.21.15.28:0/1432010160 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7328007020 con 0x7f733c0a9620 2024-02-21T23:09:11.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.415+0000 7f734601b740 1 Processor -- start 2024-02-21T23:09:11.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.415+0000 7f7333e00640 1 --2- 172.21.15.28:0/1432010160 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f731c06b6a0 0x7f731c06db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:11.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.415+0000 7f734601b740 1 -- start start 2024-02-21T23:09:11.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.415+0000 7f7333e00640 1 -- 172.21.15.28:0/1432010160 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f732808a290 con 0x7f733c0a9620 2024-02-21T23:09:11.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.415+0000 7f733be00640 1 --2- 172.21.15.28:0/1432010160 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f731c06b6a0 0x7f731c06db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:11.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.416+0000 7f734601b740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55aa6cb36020 0x55aa6cb36400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:11.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.416+0000 7f734601b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa6cb37330 0x55aa6cb72170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:11.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.416+0000 7f734601b740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55aa6cb726b0 0x55aa6cb70720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:11.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.416+0000 7f734601b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa6ca9a470 con 0x55aa6cb37330 2024-02-21T23:09:11.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.416+0000 7f734601b740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa6ca43af0 con 0x55aa6cb726b0 2024-02-21T23:09:11.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.416+0000 7f734601b740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x55aa6ca96510 con 0x55aa6cb36020 2024-02-21T23:09:11.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.416+0000 7f733b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa6cb37330 0x55aa6cb72170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:11.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.416+0000 7f733b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa6cb37330 0x55aa6cb72170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57192/0 (socket says 172.21.15.28:57192) 2024-02-21T23:09:11.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.416+0000 7f733b400640 1 -- 172.21.15.28:0/2809222229 learned_addr learned my addr 172.21.15.28:0/2809222229 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:11.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.416+0000 7f733be00640 1 --2- 172.21.15.28:0/2809222229 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55aa6cb36020 0x55aa6cb36400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:11.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.416+0000 7f7341000640 1 --2- 172.21.15.28:0/2809222229 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55aa6cb726b0 0x55aa6cb70720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:11.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.416+0000 7f733b400640 1 -- 172.21.15.28:0/2809222229 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55aa6cb36020 msgr2=0x55aa6cb36400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:11.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.416+0000 7f733b400640 1 --2- 172.21.15.28:0/2809222229 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55aa6cb36020 0x55aa6cb36400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:11.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.417+0000 7f733b400640 1 -- 172.21.15.28:0/2809222229 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55aa6cb726b0 msgr2=0x55aa6cb70720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:11.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.417+0000 7f733b400640 1 --2- 172.21.15.28:0/2809222229 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55aa6cb726b0 0x55aa6cb70720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:11.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.417+0000 7f733b400640 1 -- 172.21.15.28:0/2809222229 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55aa6ca4ad90 con 0x55aa6cb37330 2024-02-21T23:09:11.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.417+0000 7f733b400640 1 --2- 172.21.15.28:0/2809222229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa6cb37330 0x55aa6cb72170 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f7328058800 tx=0x7f73280590b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3f1cd4e3ff8508e server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:11.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.417+0000 7f7330c00640 1 -- 172.21.15.28:0/2809222229 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f732801f040 con 0x55aa6cb37330 2024-02-21T23:09:11.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.417+0000 7f7330c00640 1 -- 172.21.15.28:0/2809222229 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f732808a070 con 0x55aa6cb37330 2024-02-21T23:09:11.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.417+0000 7f7330c00640 1 -- 172.21.15.28:0/2809222229 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7328060f80 con 0x55aa6cb37330 2024-02-21T23:09:11.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.418+0000 7f734601b740 1 -- 172.21.15.28:0/2809222229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa6cb37330 msgr2=0x55aa6cb72170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:11.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.418+0000 7f733be00640 1 --2- 172.21.15.28:0/1432010160 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f731c06b6a0 0x7f731c06db60 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f732c034000 tx=0x7f732c02e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:11.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.418+0000 7f734601b740 1 --2- 172.21.15.28:0/2809222229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa6cb37330 0x55aa6cb72170 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f7328058800 tx=0x7f73280590b0 comp rx=0 tx=0).stop 2024-02-21T23:09:11.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.418+0000 7f734601b740 1 -- 172.21.15.28:0/2809222229 shutdown_connections 2024-02-21T23:09:11.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.418+0000 7f734601b740 1 --2- 172.21.15.28:0/2809222229 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55aa6cb726b0 0x55aa6cb70720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:11.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.418+0000 7f734601b740 1 --2- 172.21.15.28:0/2809222229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa6cb37330 0x55aa6cb72170 unknown :-1 s=CLOSED pgs=1542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:11.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.418+0000 7f734601b740 1 --2- 172.21.15.28:0/2809222229 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55aa6cb36020 0x55aa6cb36400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:11.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.418+0000 7f734601b740 1 -- 172.21.15.28:0/2809222229 >> 172.21.15.28:0/2809222229 conn(0x55aa6cb32970 msgr2=0x55aa6cb57380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:11.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.418+0000 7f734601b740 1 -- 172.21.15.28:0/2809222229 shutdown_connections 2024-02-21T23:09:11.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.418+0000 7f734601b740 1 -- 172.21.15.28:0/2809222229 wait complete. 2024-02-21T23:09:11.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.419+0000 7f734601b740 1 -- start start 2024-02-21T23:09:11.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.420+0000 7f734601b740 1 Processor -- start 2024-02-21T23:09:11.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.421+0000 7f734601b740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55aa6cb37330 0x55aa6cb85aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:11.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.421+0000 7f734601b740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55aa6cb726b0 0x55aa6cb78560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:11.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.421+0000 7f734601b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa6cb76af0 0x55aa6cb52130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:11.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.421+0000 7f734601b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa6ca39040 con 0x55aa6cb76af0 2024-02-21T23:09:11.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.421+0000 7f734601b740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa6cb57a90 con 0x55aa6cb37330 2024-02-21T23:09:11.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.421+0000 7f734601b740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x55aa6cb79280 con 0x55aa6cb726b0 2024-02-21T23:09:11.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.421+0000 7f734601b740 10 client.0 _ll_register_callbacks cb 0x55aa6ca56920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:09:11.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.421+0000 7f7341000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa6cb76af0 0x55aa6cb52130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:11.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.421+0000 7f7341000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa6cb76af0 0x55aa6cb52130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57202/0 (socket says 172.21.15.28:57202) 2024-02-21T23:09:11.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.421+0000 7f7341000640 1 -- 172.21.15.28:0/416995049 learned_addr learned my addr 172.21.15.28:0/416995049 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:11.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.421+0000 7f733b400640 1 --2- 172.21.15.28:0/416995049 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55aa6cb726b0 0x55aa6cb78560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:11.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.421+0000 7f733be00640 1 --2- 172.21.15.28:0/416995049 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55aa6cb37330 0x55aa6cb85aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:11.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.422+0000 7f7341000640 1 -- 172.21.15.28:0/416995049 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55aa6cb726b0 msgr2=0x55aa6cb78560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:11.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.422+0000 7f7341000640 1 --2- 172.21.15.28:0/416995049 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55aa6cb726b0 0x55aa6cb78560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:11.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.422+0000 7f7341000640 1 -- 172.21.15.28:0/416995049 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55aa6cb37330 msgr2=0x55aa6cb85aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:11.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.422+0000 7f7341000640 1 --2- 172.21.15.28:0/416995049 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55aa6cb37330 0x55aa6cb85aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:11.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.422+0000 7f7341000640 1 -- 172.21.15.28:0/416995049 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55aa6ca58aa0 con 0x55aa6cb76af0 2024-02-21T23:09:11.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.422+0000 7f733be00640 1 --2- 172.21.15.28:0/416995049 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55aa6cb37330 0x55aa6cb85aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:09:11.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.422+0000 7f7341000640 1 --2- 172.21.15.28:0/416995049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa6cb76af0 0x55aa6cb52130 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f733c0c0ca0 tx=0x7f7334007f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2de0f7bd61ce47a3 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:11.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.422+0000 7f7325600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:09:11.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.422+0000 7f7325600640 1 -- 172.21.15.28:0/416995049 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f733401c070 con 0x55aa6cb76af0 2024-02-21T23:09:11.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.422+0000 7f7325600640 1 -- 172.21.15.28:0/416995049 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(136..136 src has 1..136) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f7334014ba0 con 0x55aa6cb76af0 2024-02-21T23:09:11.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.422+0000 7f734601b740 1 -- 172.21.15.28:0/416995049 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55aa6caa79b0 con 0x55aa6cb76af0 2024-02-21T23:09:11.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.423+0000 7f7325600640 1 -- 172.21.15.28:0/416995049 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=139) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7334027050 con 0x55aa6cb76af0 2024-02-21T23:09:11.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.423+0000 7f734601b740 10 client.8505 fetch_fsmap learned FSMap version 139 2024-02-21T23:09:11.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.423+0000 7f734601b740 1 -- 172.21.15.28:0/416995049 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=139,monmap=2+}) v3 -- 0x55aa6c7af2d0 con 0x55aa6cb76af0 2024-02-21T23:09:11.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.423+0000 7f7325600640 1 -- 172.21.15.28:0/416995049 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55aa6caa79b0 con 0x55aa6cb76af0 2024-02-21T23:09:11.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.423+0000 7f7325600640 1 -- 172.21.15.28:0/416995049 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 139) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f733402a020 con 0x55aa6cb76af0 2024-02-21T23:09:11.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.423+0000 7f734601b740 10 client.8505 fetch_fsmap finished waiting for FSMap version 139 2024-02-21T23:09:11.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.423+0000 7f734601b740 10 client.8505 resolve_mds: resolved name 'a' to daemon mds.a 2024-02-21T23:09:11.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.423+0000 7f734601b740 20 client.8505 populate_metadata read hostname 'smithi028' 2024-02-21T23:09:11.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.423+0000 7f734601b740 1 --2- 172.21.15.28:0/416995049 >> [v2:172.21.15.28:6834/4151254007,v1:172.21.15.28:6835/4151254007] conn(0x55aa6cb559f0 0x55aa6cb90ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:11.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.423+0000 7f734601b740 4 client.8505 mds_command: new command op to 8376 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-21T23:09:11.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.423+0000 7f734601b740 1 -- 172.21.15.28:0/416995049 --> [v2:172.21.15.28:6834/4151254007,v1:172.21.15.28:6835/4151254007] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55aa6caa1d00 con 0x55aa6cb559f0 2024-02-21T23:09:11.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.423+0000 7f733be00640 1 --2- 172.21.15.28:0/416995049 >> [v2:172.21.15.28:6834/4151254007,v1:172.21.15.28:6835/4151254007] conn(0x55aa6cb559f0 0x55aa6cb90ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:11.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.424+0000 7f733be00640 1 --2- 172.21.15.28:0/416995049 >> [v2:172.21.15.28:6834/4151254007,v1:172.21.15.28:6835/4151254007] conn(0x55aa6cb559f0 0x55aa6cb90ce0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=36c96ee8e9ff19b server_cookie=8796f467f0b634b0 in_seq=0 out_seq=0 2024-02-21T23:09:11.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.424+0000 7f7325600640 10 client.8505 ms_handle_connect on v2:172.21.15.28:6834/4151254007 2024-02-21T23:09:11.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.425+0000 7f7325600640 1 -- 172.21.15.28:0/416995049 <== mds.0 v2:172.21.15.28:6834/4151254007 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55aa6caa1d00 con 0x55aa6cb559f0 2024-02-21T23:09:11.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.425+0000 7f7325600640 10 client.8505 handle_command_reply: tid=0 2024-02-21T23:09:11.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.425+0000 7f734601b740 1 client.8505 shutdown 2024-02-21T23:09:11.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.425+0000 7f734601b740 1 -- 172.21.15.28:0/416995049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa6cb76af0 msgr2=0x55aa6cb52130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:11.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.425+0000 7f734601b740 1 --2- 172.21.15.28:0/416995049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa6cb76af0 0x55aa6cb52130 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f733c0c0ca0 tx=0x7f7334007f10 comp rx=0 tx=0).stop 2024-02-21T23:09:11.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.425+0000 7f734601b740 1 -- 172.21.15.28:0/416995049 shutdown_connections 2024-02-21T23:09:11.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.425+0000 7f734601b740 1 --2- 172.21.15.28:0/416995049 >> [v2:172.21.15.28:6834/4151254007,v1:172.21.15.28:6835/4151254007] conn(0x55aa6cb559f0 0x55aa6cb90ce0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:11.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.425+0000 7f734601b740 1 --2- 172.21.15.28:0/416995049 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55aa6cb726b0 0x55aa6cb78560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:11.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.425+0000 7f734601b740 1 --2- 172.21.15.28:0/416995049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa6cb76af0 0x55aa6cb52130 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:11.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.425+0000 7f734601b740 1 --2- 172.21.15.28:0/416995049 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55aa6cb37330 0x55aa6cb85aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:11.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.425+0000 7f734601b740 1 -- 172.21.15.28:0/416995049 >> 172.21.15.28:0/416995049 conn(0x55aa6cb36020 msgr2=0x55aa6cb70ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:11.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.425+0000 7f7325600640 0 client.8505 ms_handle_reset on v2:172.21.15.28:6834/4151254007 2024-02-21T23:09:11.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.425+0000 7f734601b740 1 -- 172.21.15.28:0/416995049 shutdown_connections 2024-02-21T23:09:11.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.425+0000 7f734601b740 1 -- 172.21.15.28:0/416995049 wait complete. 2024-02-21T23:09:11.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.425+0000 7f734601b740 20 client.8505 trim_cache size 0 max 16384 2024-02-21T23:09:11.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.444+0000 7f734601b740 1 Processor -- start 2024-02-21T23:09:11.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.444+0000 7f734601b740 1 -- start start 2024-02-21T23:09:11.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.445+0000 7f734601b740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55aa6cb37330 0x55aa6cb2f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:11.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.445+0000 7f734601b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa6cb726b0 0x55aa6cb4bb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:11.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.445+0000 7f734601b740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55aa6cb76af0 0x55aa6cb4c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:11.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.445+0000 7f734601b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f733c1f73b0 con 0x55aa6cb726b0 2024-02-21T23:09:11.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.445+0000 7f734601b740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f733c1ee980 con 0x55aa6cb76af0 2024-02-21T23:09:11.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.445+0000 7f734601b740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f733c1f6de0 con 0x55aa6cb37330 2024-02-21T23:09:11.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.445+0000 7f733b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa6cb726b0 0x55aa6cb4bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:11.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.445+0000 7f733b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa6cb726b0 0x55aa6cb4bb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57214/0 (socket says 172.21.15.28:57214) 2024-02-21T23:09:11.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.445+0000 7f733b400640 1 -- 172.21.15.28:0/3907876359 learned_addr learned my addr 172.21.15.28:0/3907876359 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:11.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.445+0000 7f733be00640 1 --2- 172.21.15.28:0/3907876359 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55aa6cb37330 0x55aa6cb2f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:11.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.446+0000 7f7341000640 1 --2- 172.21.15.28:0/3907876359 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55aa6cb76af0 0x55aa6cb4c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:11.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.446+0000 7f733b400640 1 -- 172.21.15.28:0/3907876359 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55aa6cb37330 msgr2=0x55aa6cb2f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:11.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.446+0000 7f733b400640 1 --2- 172.21.15.28:0/3907876359 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55aa6cb37330 0x55aa6cb2f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:11.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.446+0000 7f733b400640 1 -- 172.21.15.28:0/3907876359 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55aa6cb76af0 msgr2=0x55aa6cb4c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:11.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.446+0000 7f733b400640 1 --2- 172.21.15.28:0/3907876359 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55aa6cb76af0 0x55aa6cb4c040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:11.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.446+0000 7f733b400640 1 -- 172.21.15.28:0/3907876359 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55aa6c8796b0 con 0x55aa6cb726b0 2024-02-21T23:09:11.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.446+0000 7f733b400640 1 --2- 172.21.15.28:0/3907876359 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa6cb726b0 0x55aa6cb4bb00 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f7328058a30 tx=0x7f732801c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20c37529fd75010c server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:11.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.446+0000 7f7327e00640 1 -- 172.21.15.28:0/3907876359 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f732801f040 con 0x55aa6cb726b0 2024-02-21T23:09:11.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.446+0000 7f7327e00640 1 -- 172.21.15.28:0/3907876359 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7328060f80 con 0x55aa6cb726b0 2024-02-21T23:09:11.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.446+0000 7f7327e00640 1 -- 172.21.15.28:0/3907876359 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7328054ee0 con 0x55aa6cb726b0 2024-02-21T23:09:11.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.447+0000 7f734601b740 1 -- 172.21.15.28:0/3907876359 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa6cb726b0 msgr2=0x55aa6cb4bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:11.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.447+0000 7f734601b740 1 --2- 172.21.15.28:0/3907876359 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa6cb726b0 0x55aa6cb4bb00 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f7328058a30 tx=0x7f732801c000 comp rx=0 tx=0).stop 2024-02-21T23:09:11.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.447+0000 7f734601b740 1 -- 172.21.15.28:0/3907876359 shutdown_connections 2024-02-21T23:09:11.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.447+0000 7f734601b740 1 --2- 172.21.15.28:0/3907876359 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55aa6cb76af0 0x55aa6cb4c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:11.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.447+0000 7f734601b740 1 --2- 172.21.15.28:0/3907876359 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa6cb726b0 0x55aa6cb4bb00 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:11.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.447+0000 7f734601b740 1 --2- 172.21.15.28:0/3907876359 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55aa6cb37330 0x55aa6cb2f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:11.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.447+0000 7f734601b740 1 -- 172.21.15.28:0/3907876359 >> 172.21.15.28:0/3907876359 conn(0x55aa6cb36020 msgr2=0x55aa6cb51cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:11.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.447+0000 7f734601b740 1 -- 172.21.15.28:0/3907876359 shutdown_connections 2024-02-21T23:09:11.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.447+0000 7f734601b740 1 -- 172.21.15.28:0/3907876359 wait complete. 2024-02-21T23:09:11.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.448+0000 7f734601b740 1 -- start start 2024-02-21T23:09:11.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.448+0000 7f734601b740 1 Processor -- start 2024-02-21T23:09:11.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.449+0000 7f734601b740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55aa6cb37330 0x55aa6cb73dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:11.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.449+0000 7f734601b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa6cb726b0 0x55aa6cb310f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:11.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.449+0000 7f734601b740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55aa6cb76af0 0x55aa6cbe56e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:11.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.449+0000 7f734601b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa6cb86590 con 0x55aa6cb726b0 2024-02-21T23:09:11.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.449+0000 7f734601b740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa6cb78fc0 con 0x55aa6cb76af0 2024-02-21T23:09:11.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.449+0000 7f734601b740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x55aa6cb52790 con 0x55aa6cb37330 2024-02-21T23:09:11.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.449+0000 7f734601b740 10 client.0 _ll_register_callbacks cb 0x7f7334007710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:09:11.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.449+0000 7f733b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa6cb726b0 0x55aa6cb310f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:11.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.449+0000 7f733b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa6cb726b0 0x55aa6cb310f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57222/0 (socket says 172.21.15.28:57222) 2024-02-21T23:09:11.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.449+0000 7f733b400640 1 -- 172.21.15.28:0/1721610773 learned_addr learned my addr 172.21.15.28:0/1721610773 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:11.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.449+0000 7f7341000640 1 --2- 172.21.15.28:0/1721610773 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55aa6cb76af0 0x55aa6cbe56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:11.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.449+0000 7f733be00640 1 --2- 172.21.15.28:0/1721610773 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55aa6cb37330 0x55aa6cb73dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:11.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.449+0000 7f733b400640 1 -- 172.21.15.28:0/1721610773 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55aa6cb37330 msgr2=0x55aa6cb73dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:11.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.449+0000 7f733b400640 1 --2- 172.21.15.28:0/1721610773 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55aa6cb37330 0x55aa6cb73dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:11.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.450+0000 7f733b400640 1 -- 172.21.15.28:0/1721610773 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55aa6cb76af0 msgr2=0x55aa6cbe56e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:11.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.450+0000 7f733b400640 1 --2- 172.21.15.28:0/1721610773 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55aa6cb76af0 0x55aa6cbe56e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:11.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.450+0000 7f733b400640 1 -- 172.21.15.28:0/1721610773 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55aa6cb36660 con 0x55aa6cb726b0 2024-02-21T23:09:11.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.450+0000 7f733b400640 1 --2- 172.21.15.28:0/1721610773 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa6cb726b0 0x55aa6cb310f0 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f7328056b20 tx=0x7f73280012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a8aa62a7f1ee67 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:11.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.450+0000 7f7325600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:09:11.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.450+0000 7f7325600640 1 -- 172.21.15.28:0/1721610773 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f732801f030 con 0x55aa6cb726b0 2024-02-21T23:09:11.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.450+0000 7f7325600640 1 -- 172.21.15.28:0/1721610773 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(136..136 src has 1..136) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f7328077250 con 0x55aa6cb726b0 2024-02-21T23:09:11.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.450+0000 7f734601b740 1 -- 172.21.15.28:0/1721610773 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7328060f80 con 0x55aa6cb726b0 2024-02-21T23:09:11.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.450+0000 7f7325600640 1 -- 172.21.15.28:0/1721610773 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=139) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7328077470 con 0x55aa6cb726b0 2024-02-21T23:09:11.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.450+0000 7f734601b740 10 client.8511 fetch_fsmap learned FSMap version 139 2024-02-21T23:09:11.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.450+0000 7f734601b740 1 -- 172.21.15.28:0/1721610773 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=139,monmap=2+}) v3 -- 0x55aa6cb72300 con 0x55aa6cb726b0 2024-02-21T23:09:11.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.451+0000 7f7325600640 1 -- 172.21.15.28:0/1721610773 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7328060f80 con 0x55aa6cb726b0 2024-02-21T23:09:11.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.451+0000 7f7325600640 1 -- 172.21.15.28:0/1721610773 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 139) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f7328065080 con 0x55aa6cb726b0 2024-02-21T23:09:11.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.451+0000 7f734601b740 10 client.8511 fetch_fsmap finished waiting for FSMap version 139 2024-02-21T23:09:11.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.451+0000 7f734601b740 10 client.8511 resolve_mds: resolved name 'a' to daemon mds.a 2024-02-21T23:09:11.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.451+0000 7f734601b740 20 client.8511 populate_metadata read hostname 'smithi028' 2024-02-21T23:09:11.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.451+0000 7f734601b740 1 --2- 172.21.15.28:0/1721610773 >> [v2:172.21.15.28:6834/4151254007,v1:172.21.15.28:6835/4151254007] conn(0x55aa6cbdec10 0x55aa6cbdeff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:11.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.451+0000 7f734601b740 4 client.8511 mds_command: new command op to 8376 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-21T23:09:11.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.451+0000 7f734601b740 1 -- 172.21.15.28:0/1721610773 --> [v2:172.21.15.28:6834/4151254007,v1:172.21.15.28:6835/4151254007] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55aa6ca20830 con 0x55aa6cbdec10 2024-02-21T23:09:11.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.451+0000 7f733be00640 1 --2- 172.21.15.28:0/1721610773 >> [v2:172.21.15.28:6834/4151254007,v1:172.21.15.28:6835/4151254007] conn(0x55aa6cbdec10 0x55aa6cbdeff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:11.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.452+0000 7f733be00640 1 --2- 172.21.15.28:0/1721610773 >> [v2:172.21.15.28:6834/4151254007,v1:172.21.15.28:6835/4151254007] conn(0x55aa6cbdec10 0x55aa6cbdeff0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e3e3dfa27b52885e server_cookie=746c525b9576609d in_seq=0 out_seq=0 2024-02-21T23:09:11.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.452+0000 7f7325600640 10 client.8511 ms_handle_connect on v2:172.21.15.28:6834/4151254007 2024-02-21T23:09:11.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.452+0000 7f7325600640 1 -- 172.21.15.28:0/1721610773 <== mds.0 v2:172.21.15.28:6834/4151254007 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+328 (crc 0 0 0) 0x55aa6ca20830 con 0x55aa6cbdec10 2024-02-21T23:09:11.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.452+0000 7f7325600640 10 client.8511 handle_command_reply: tid=0 2024-02-21T23:09:11.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.452+0000 7f734601b740 1 client.8511 shutdown 2024-02-21T23:09:11.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.452+0000 7f734601b740 1 -- 172.21.15.28:0/1721610773 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa6cb726b0 msgr2=0x55aa6cb310f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:11.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.452+0000 7f734601b740 1 --2- 172.21.15.28:0/1721610773 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa6cb726b0 0x55aa6cb310f0 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f7328056b20 tx=0x7f73280012d0 comp rx=0 tx=0).stop 2024-02-21T23:09:11.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.452+0000 7f734601b740 1 -- 172.21.15.28:0/1721610773 shutdown_connections 2024-02-21T23:09:11.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.452+0000 7f734601b740 1 --2- 172.21.15.28:0/1721610773 >> [v2:172.21.15.28:6834/4151254007,v1:172.21.15.28:6835/4151254007] conn(0x55aa6cbdec10 0x55aa6cbdeff0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:11.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.452+0000 7f734601b740 1 --2- 172.21.15.28:0/1721610773 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55aa6cb76af0 0x55aa6cbe56e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:11.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.452+0000 7f734601b740 1 --2- 172.21.15.28:0/1721610773 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aa6cb726b0 0x55aa6cb310f0 unknown :-1 s=CLOSED pgs=1545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:11.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.452+0000 7f734601b740 1 --2- 172.21.15.28:0/1721610773 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55aa6cb37330 0x55aa6cb73dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:11.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.452+0000 7f7325600640 0 client.8511 ms_handle_reset on v2:172.21.15.28:6834/4151254007 2024-02-21T23:09:11.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.452+0000 7f734601b740 1 -- 172.21.15.28:0/1721610773 >> 172.21.15.28:0/1721610773 conn(0x55aa6cb36020 msgr2=0x55aa6cb51cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:11.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.453+0000 7f734601b740 1 -- 172.21.15.28:0/1721610773 shutdown_connections 2024-02-21T23:09:11.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.453+0000 7f734601b740 1 -- 172.21.15.28:0/1721610773 wait complete. 2024-02-21T23:09:11.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.453+0000 7f734601b740 20 client.8511 trim_cache size 0 max 16384 2024-02-21T23:09:11.445 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-21T23:09:11.445 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac", 2024-02-21T23:09:11.445 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-02-21T23:09:11.445 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 8376, 2024-02-21T23:09:11.445 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-02-21T23:09:11.445 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-02-21T23:09:11.445 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-02-21T23:09:11.445 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 7.0318530580000003, 2024-02-21T23:09:11.445 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 139, 2024-02-21T23:09:11.445 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 136, 2024-02-21T23:09:11.445 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 136, 2024-02-21T23:09:11.445 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 14.053211635 2024-02-21T23:09:11.445 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-21T23:09:11.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.453+0000 7f7341a00640 1 -- 172.21.15.28:0/1432010160 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f731c06b6a0 msgr2=0x7f731c06db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:11.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.453+0000 7f7341a00640 1 --2- 172.21.15.28:0/1432010160 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f731c06b6a0 0x7f731c06db60 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f732c034000 tx=0x7f732c02e000 comp rx=0 tx=0).stop 2024-02-21T23:09:11.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.453+0000 7f7341a00640 1 -- 172.21.15.28:0/1432010160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f733c0a9620 msgr2=0x7f733c0aabc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:11.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.453+0000 7f7341a00640 1 --2- 172.21.15.28:0/1432010160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f733c0a9620 0x7f733c0aabc0 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7f732800d7a0 tx=0x7f732800dc60 comp rx=0 tx=0).stop 2024-02-21T23:09:11.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.454+0000 7f7341a00640 1 -- 172.21.15.28:0/1432010160 shutdown_connections 2024-02-21T23:09:11.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.454+0000 7f7341a00640 1 --2- 172.21.15.28:0/1432010160 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f731c06b6a0 0x7f731c06db60 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:11.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.454+0000 7f7341a00640 1 --2- 172.21.15.28:0/1432010160 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f733c1502a0 0x7f733c0abd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:11.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.454+0000 7f7341a00640 1 --2- 172.21.15.28:0/1432010160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f733c0a9620 0x7f733c0aabc0 unknown :-1 s=CLOSED pgs=1541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:11.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.454+0000 7f7341a00640 1 --2- 172.21.15.28:0/1432010160 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7320003b70 0x7f733c1ea640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:11.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.454+0000 7f7341a00640 1 -- 172.21.15.28:0/1432010160 >> 172.21.15.28:0/1432010160 conn(0x7f733c0a8320 msgr2=0x7f733c0bf9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:11.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.454+0000 7f7341a00640 1 -- 172.21.15.28:0/1432010160 shutdown_connections 2024-02-21T23:09:11.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.454+0000 7f7341a00640 1 -- 172.21.15.28:0/1432010160 wait complete. 2024-02-21T23:09:11.461 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:09:11.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.686+0000 7f7ac2000640 1 Processor -- start 2024-02-21T23:09:11.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.686+0000 7f7ac2000640 1 -- start start 2024-02-21T23:09:11.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.687+0000 7f7ac2000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7abc0c54c0 0x7f7abc0a7f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:11.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.688+0000 7f7ac2000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7abc0c6ef0 0x7f7abc152560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:11.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.688+0000 7f7ac2000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f7abc152aa0 0x7f7abc158940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:11.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.688+0000 7f7ac2000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7abc0c3de0 con 0x7f7abc0c54c0 2024-02-21T23:09:11.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.688+0000 7f7ac2000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7abc0c7fe0 con 0x7f7abc152aa0 2024-02-21T23:09:11.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.688+0000 7f7ac2000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7abc0c85a0 con 0x7f7abc0c6ef0 2024-02-21T23:09:11.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.688+0000 7f7abb400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7abc0c54c0 0x7f7abc0a7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:11.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.688+0000 7f7abb400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7abc0c54c0 0x7f7abc0a7f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57238/0 (socket says 172.21.15.28:57238) 2024-02-21T23:09:11.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.688+0000 7f7abb400640 1 -- 172.21.15.28:0/639071528 learned_addr learned my addr 172.21.15.28:0/639071528 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:11.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.688+0000 7f7abbe00640 1 --2- 172.21.15.28:0/639071528 >> v2:172.21.15.81:3300/0 conn(0x7f7abc152aa0 0x7f7abc158940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:11.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.688+0000 7f7abb400640 1 -- 172.21.15.28:0/639071528 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7abc0c6ef0 msgr2=0x7f7abc152560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:11.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.688+0000 7f7abaa00640 1 --2- 172.21.15.28:0/639071528 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7abc0c6ef0 0x7f7abc152560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:11.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.689+0000 7f7abb400640 1 --2- 172.21.15.28:0/639071528 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7abc0c6ef0 0x7f7abc152560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:11.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.689+0000 7f7abb400640 1 -- 172.21.15.28:0/639071528 >> v2:172.21.15.81:3300/0 conn(0x7f7abc152aa0 msgr2=0x7f7abc158940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:11.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.689+0000 7f7abb400640 1 --2- 172.21.15.28:0/639071528 >> v2:172.21.15.81:3300/0 conn(0x7f7abc152aa0 0x7f7abc158940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:11.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.689+0000 7f7abb400640 1 -- 172.21.15.28:0/639071528 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7abc0c6820 con 0x7f7abc0c54c0 2024-02-21T23:09:11.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.689+0000 7f7abbe00640 1 --2- 172.21.15.28:0/639071528 >> v2:172.21.15.81:3300/0 conn(0x7f7abc152aa0 0x7f7abc158940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:09:11.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.689+0000 7f7abaa00640 1 --2- 172.21.15.28:0/639071528 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7abc0c6ef0 0x7f7abc152560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:09:11.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.690+0000 7f7abb400640 1 --2- 172.21.15.28:0/639071528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7abc0c54c0 0x7f7abc0a7f40 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f7abc0c6300 tx=0x7f7aac02b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70b11631318d3900 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:11.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.690+0000 7f7aba000640 1 -- 172.21.15.28:0/639071528 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aac0317f0 con 0x7f7abc0c54c0 2024-02-21T23:09:11.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.690+0000 7f7aba000640 1 -- 172.21.15.28:0/639071528 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7aac037070 con 0x7f7abc0c54c0 2024-02-21T23:09:11.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.690+0000 7f7aba000640 1 -- 172.21.15.28:0/639071528 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aac031bb0 con 0x7f7abc0c54c0 2024-02-21T23:09:11.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.691+0000 7f7ac2000640 1 -- 172.21.15.28:0/639071528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7abc0c54c0 msgr2=0x7f7abc0a7f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:11.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.691+0000 7f7ac2000640 1 --2- 172.21.15.28:0/639071528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7abc0c54c0 0x7f7abc0a7f40 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f7abc0c6300 tx=0x7f7aac02b890 comp rx=0 tx=0).stop 2024-02-21T23:09:11.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.691+0000 7f7ac2000640 1 -- 172.21.15.28:0/639071528 shutdown_connections 2024-02-21T23:09:11.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.691+0000 7f7ac2000640 1 --2- 172.21.15.28:0/639071528 >> v2:172.21.15.81:3300/0 conn(0x7f7abc152aa0 0x7f7abc158940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:11.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.691+0000 7f7ac2000640 1 --2- 172.21.15.28:0/639071528 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7abc0c6ef0 0x7f7abc152560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:11.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.691+0000 7f7ac2000640 1 --2- 172.21.15.28:0/639071528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7abc0c54c0 0x7f7abc0a7f40 unknown :-1 s=CLOSED pgs=1546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:11.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.691+0000 7f7ac2000640 1 -- 172.21.15.28:0/639071528 >> 172.21.15.28:0/639071528 conn(0x7f7abc0c8810 msgr2=0x7f7abc0c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:11.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.691+0000 7f7ac2000640 1 -- 172.21.15.28:0/639071528 shutdown_connections 2024-02-21T23:09:11.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.691+0000 7f7ac2000640 1 -- 172.21.15.28:0/639071528 wait complete. 2024-02-21T23:09:11.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.693+0000 7f7ac2000640 1 Processor -- start 2024-02-21T23:09:11.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.693+0000 7f7ac2000640 1 -- start start 2024-02-21T23:09:11.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.694+0000 7f7ac2000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7abc0c54c0 0x7f7abc1ee320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:11.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.694+0000 7f7ac2000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7abc0c6ef0 0x7f7abc1ea2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:11.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.694+0000 7f7ac2000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7abc152aa0 0x7f7abc1551f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:11.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.694+0000 7f7ac2000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7abc0c58a0 con 0x7f7abc0c6ef0 2024-02-21T23:09:11.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.694+0000 7f7ac2000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7abc0c8230 con 0x7f7abc152aa0 2024-02-21T23:09:11.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.694+0000 7f7ac2000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7abc0c5a20 con 0x7f7abc0c54c0 2024-02-21T23:09:11.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.694+0000 7f7abaa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7abc0c6ef0 0x7f7abc1ea2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:11.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.694+0000 7f7abaa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7abc0c6ef0 0x7f7abc1ea2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57250/0 (socket says 172.21.15.28:57250) 2024-02-21T23:09:11.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.694+0000 7f7abaa00640 1 -- 172.21.15.28:0/1614828437 learned_addr learned my addr 172.21.15.28:0/1614828437 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:11.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.694+0000 7f7abb400640 1 --2- 172.21.15.28:0/1614828437 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7abc0c54c0 0x7f7abc1ee320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:11.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.694+0000 7f7abbe00640 1 --2- 172.21.15.28:0/1614828437 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7abc152aa0 0x7f7abc1551f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:11.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.695+0000 7f7abaa00640 1 -- 172.21.15.28:0/1614828437 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7abc0c54c0 msgr2=0x7f7abc1ee320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:11.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.695+0000 7f7abaa00640 1 --2- 172.21.15.28:0/1614828437 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7abc0c54c0 0x7f7abc1ee320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:11.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.695+0000 7f7abaa00640 1 -- 172.21.15.28:0/1614828437 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7abc152aa0 msgr2=0x7f7abc1551f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:11.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.695+0000 7f7abaa00640 1 --2- 172.21.15.28:0/1614828437 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7abc152aa0 0x7f7abc1551f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:11.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.695+0000 7f7abaa00640 1 -- 172.21.15.28:0/1614828437 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7abc0ab330 con 0x7f7abc0c6ef0 2024-02-21T23:09:11.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.695+0000 7f7abaa00640 1 --2- 172.21.15.28:0/1614828437 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7abc0c6ef0 0x7f7abc1ea2e0 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7f7aa800c7f0 tx=0x7f7aa800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:11.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.695+0000 7f7ab3400640 1 -- 172.21.15.28:0/1614828437 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa8015070 con 0x7f7abc0c6ef0 2024-02-21T23:09:11.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.695+0000 7f7ab3400640 1 -- 172.21.15.28:0/1614828437 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7aa8007a30 con 0x7f7abc0c6ef0 2024-02-21T23:09:11.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.695+0000 7f7ac2000640 1 -- 172.21.15.28:0/1614828437 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7abc1edb20 con 0x7f7abc0c6ef0 2024-02-21T23:09:11.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.695+0000 7f7ab3400640 1 -- 172.21.15.28:0/1614828437 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa8007e00 con 0x7f7abc0c6ef0 2024-02-21T23:09:11.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.696+0000 7f7ac2000640 1 -- 172.21.15.28:0/1614828437 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7abc0ab950 con 0x7f7abc0c6ef0 2024-02-21T23:09:11.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.697+0000 7f7ac2000640 1 -- 172.21.15.28:0/1614828437 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7abc068640 con 0x7f7abc0c6ef0 2024-02-21T23:09:11.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.698+0000 7f7ab3400640 1 -- 172.21.15.28:0/1614828437 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7aa8018070 con 0x7f7abc0c6ef0 2024-02-21T23:09:11.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.700+0000 7f7ab3400640 1 --2- 172.21.15.28:0/1614828437 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7a9c06b500 0x7f7a9c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:11.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.700+0000 7f7ab3400640 1 -- 172.21.15.28:0/1614828437 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f7aa808a1d0 con 0x7f7abc0c6ef0 2024-02-21T23:09:11.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.701+0000 7f7abb400640 1 --2- 172.21.15.28:0/1614828437 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7a9c06b500 0x7f7a9c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:11.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.702+0000 7f7abb400640 1 --2- 172.21.15.28:0/1614828437 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7a9c06b500 0x7f7a9c06d9c0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f7aac036000 tx=0x7f7aac03d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:11.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.703+0000 7f7ab3400640 1 -- 172.21.15.28:0/1614828437 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7aa8058860 con 0x7f7abc0c6ef0 2024-02-21T23:09:11.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.944+0000 7f7ac2000640 1 -- 172.21.15.28:0/1614828437 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7abc156060 con 0x7f7abc0c6ef0 2024-02-21T23:09:11.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.945+0000 7f7ab3400640 1 -- 172.21.15.28:0/1614828437 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 139 v139) v1 ==== 95+0+4948 (secure 0 0 0) 0x7f7aa805c900 con 0x7f7abc0c6ef0 2024-02-21T23:09:11.933 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:09:11.933 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":139,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6140,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/4241906475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":4241906475},{"type":"v1","addr":"172.21.15.81:6836","nonce":4241906475}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":133},{"gid":6143,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/2095908433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":2095908433},{"type":"v1","addr":"172.21.15.81:6837","nonce":2095908433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":133},{"gid":8370,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/3934548735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3934548735},{"type":"v1","addr":"172.21.15.28:6833","nonce":3934548735}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":133}],"filesystems":[{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:09:03.869294+0000","modified":"2024-02-21T23:09:07.201932+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8376},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8376":{"gid":8376,"name":"a","rank":0,"incarnation":135,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/4151254007","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4151254007},{"type":"v1","addr":"172.21.15.28:6835","nonce":4151254007}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-02-21T23:09:11.933 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 139 2024-02-21T23:09:11.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.948+0000 7f7ac2000640 1 -- 172.21.15.28:0/1614828437 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7a9c06b500 msgr2=0x7f7a9c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:11.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.948+0000 7f7ac2000640 1 --2- 172.21.15.28:0/1614828437 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7a9c06b500 0x7f7a9c06d9c0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f7aac036000 tx=0x7f7aac03d040 comp rx=0 tx=0).stop 2024-02-21T23:09:11.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.948+0000 7f7ac2000640 1 -- 172.21.15.28:0/1614828437 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7abc0c6ef0 msgr2=0x7f7abc1ea2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:11.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.948+0000 7f7ac2000640 1 --2- 172.21.15.28:0/1614828437 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7abc0c6ef0 0x7f7abc1ea2e0 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7f7aa800c7f0 tx=0x7f7aa800ccb0 comp rx=0 tx=0).stop 2024-02-21T23:09:11.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.948+0000 7f7ac2000640 1 -- 172.21.15.28:0/1614828437 shutdown_connections 2024-02-21T23:09:11.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.948+0000 7f7ac2000640 1 --2- 172.21.15.28:0/1614828437 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7a9c06b500 0x7f7a9c06d9c0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:11.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.948+0000 7f7ac2000640 1 --2- 172.21.15.28:0/1614828437 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7abc152aa0 0x7f7abc1551f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:11.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.948+0000 7f7ac2000640 1 --2- 172.21.15.28:0/1614828437 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7abc0c6ef0 0x7f7abc1ea2e0 unknown :-1 s=CLOSED pgs=1547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:11.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.948+0000 7f7ac2000640 1 --2- 172.21.15.28:0/1614828437 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7abc0c54c0 0x7f7abc1ee320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:11.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.948+0000 7f7ac2000640 1 -- 172.21.15.28:0/1614828437 >> 172.21.15.28:0/1614828437 conn(0x7f7abc0c8810 msgr2=0x7f7abc070ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:11.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.948+0000 7f7ac2000640 1 -- 172.21.15.28:0/1614828437 shutdown_connections 2024-02-21T23:09:11.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:11.948+0000 7f7ac2000640 1 -- 172.21.15.28:0/1614828437 wait complete. 2024-02-21T23:09:11.957 INFO:tasks.cephfs.test_failover:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 10, "epoch": 139, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 10, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-02-21T23:09:03.869294+0000", "damaged": [], "data_pools": [ 22 ], "enabled": true, "epoch": 139, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_8376": { "addr": "172.21.15.28:6835/4151254007", "addrs": { "addrvec": [ { "addr": "172.21.15.28:6834", "nonce": 4151254007, "type": "v2" }, { "addr": "172.21.15.28:6835", "nonce": 4151254007, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8376, "incarnation": 135, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 21, "modified": "2024-02-21T23:09:07.201932+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8376 } } } ], "standbys": [ { "addr": "172.21.15.81:6836/4241906475", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6834", "nonce": 4241906475, "type": "v2" }, { "addr": "172.21.15.81:6836", "nonce": 4241906475, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 133, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6140, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.81:6837/2095908433", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6835", "nonce": 2095908433, "type": "v2" }, { "addr": "172.21.15.81:6837", "nonce": 2095908433, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 133, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6143, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.28:6833/3934548735", "addrs": { "addrvec": [ { "addr": "172.21.15.28:6832", "nonce": 3934548735, "type": "v2" }, { "addr": "172.21.15.28:6833", "nonce": 3934548735, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 133, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8370, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-02-21T23:09:11.957 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-02-21T23:09:11.957 INFO:tasks.cephfs.test_failover:current = [{'gid': 8376, 'name': 'a', 'rank': 0, 'incarnation': 135, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/4151254007', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 4151254007}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 4151254007}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 6140, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.81:6836/4241906475', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6834', 'nonce': 4241906475}, {'type': 'v1', 'addr': '172.21.15.81:6836', 'nonce': 4241906475}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 133}, {'gid': 8370, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.28:6833/3934548735', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 3934548735}, {'type': 'v1', 'addr': '172.21.15.28:6833', 'nonce': 3934548735}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 133}, {'gid': 6143, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.81:6837/2095908433', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6835', 'nonce': 2095908433}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 2095908433}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 133}] 2024-02-21T23:09:11.958 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.b mds_join_fs cephfs 2024-02-21T23:09:12.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.197+0000 7fef68a00640 1 Processor -- start 2024-02-21T23:09:12.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.197+0000 7fef68a00640 1 -- start start 2024-02-21T23:09:12.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.198+0000 7fef68a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fef6406daa0 0x7fef640c8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:12.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.198+0000 7fef68a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fef641550e0 0x7fef64155530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:12.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.198+0000 7fef68a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef64158920 con 0x7fef64152650 2024-02-21T23:09:12.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.198+0000 7fef68a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef641586d0 con 0x7fef6406daa0 2024-02-21T23:09:12.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.199+0000 7fef68a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef64158c00 con 0x7fef641550e0 2024-02-21T23:09:12.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.199+0000 7fef63e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fef641550e0 0x7fef64155530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:12.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.199+0000 7fef5aa00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fef6406daa0 0x7fef640c8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:12.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.199+0000 7fef63e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fef641550e0 0x7fef64155530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3301/0 says I am v2:172.21.15.28:44914/0 (socket says 172.21.15.28:44914) 2024-02-21T23:09:12.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.199+0000 7fef63e00640 1 -- 172.21.15.28:0/3334686239 learned_addr learned my addr 172.21.15.28:0/3334686239 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:12.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.199+0000 7fef5aa00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fef6406daa0 0x7fef640c8ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.28:40504/0 (socket says 172.21.15.28:40504) 2024-02-21T23:09:12.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.200+0000 7fef5aa00640 1 -- 172.21.15.28:0/3334686239 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fef641550e0 msgr2=0x7fef64155530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:12.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.200+0000 7fef5aa00640 1 --2- 172.21.15.28:0/3334686239 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fef641550e0 0x7fef64155530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:12.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.200+0000 7fef5aa00640 1 -- 172.21.15.28:0/3334686239 >> v1:172.21.15.81:6789/0 conn(0x7fef64152650 legacy=0x7fef640c7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:12.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.200+0000 7fef5aa00640 1 -- 172.21.15.28:0/3334686239 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef64158ee0 con 0x7fef6406daa0 2024-02-21T23:09:12.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.201+0000 7fef5aa00640 1 --2- 172.21.15.28:0/3334686239 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fef6406daa0 0x7fef640c8ce0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fef4c009d90 tx=0x7fef4c02be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a8e18132365af80 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:12.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.202+0000 7fef62a00640 1 -- 172.21.15.28:0/3334686239 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef4c039070 con 0x7fef6406daa0 2024-02-21T23:09:12.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.202+0000 7fef62a00640 1 -- 172.21.15.28:0/3334686239 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef4c02cbc0 con 0x7fef6406daa0 2024-02-21T23:09:12.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.202+0000 7fef62a00640 1 -- 172.21.15.28:0/3334686239 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef4c034440 con 0x7fef6406daa0 2024-02-21T23:09:12.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.203+0000 7fef68a00640 1 -- 172.21.15.28:0/3334686239 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fef6406daa0 msgr2=0x7fef640c8ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:12.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.203+0000 7fef68a00640 1 --2- 172.21.15.28:0/3334686239 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fef6406daa0 0x7fef640c8ce0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fef4c009d90 tx=0x7fef4c02be10 comp rx=0 tx=0).stop 2024-02-21T23:09:12.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.203+0000 7fef68a00640 1 -- 172.21.15.28:0/3334686239 shutdown_connections 2024-02-21T23:09:12.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.203+0000 7fef68a00640 1 --2- 172.21.15.28:0/3334686239 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fef641550e0 0x7fef64155530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:12.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.203+0000 7fef68a00640 1 --2- 172.21.15.28:0/3334686239 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fef6406daa0 0x7fef640c8ce0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:12.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.203+0000 7fef68a00640 1 -- 172.21.15.28:0/3334686239 >> 172.21.15.28:0/3334686239 conn(0x7fef64072ce0 msgr2=0x7fef640ac340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:12.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.203+0000 7fef68a00640 1 -- 172.21.15.28:0/3334686239 shutdown_connections 2024-02-21T23:09:12.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.203+0000 7fef68a00640 1 -- 172.21.15.28:0/3334686239 wait complete. 2024-02-21T23:09:12.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.204+0000 7fef68a00640 1 Processor -- start 2024-02-21T23:09:12.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.205+0000 7fef68a00640 1 -- start start 2024-02-21T23:09:12.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.205+0000 7fef68a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef6406daa0 0x7fef64151260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:12.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.205+0000 7fef68a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fef64152650 0x7fef641ead20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:12.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.205+0000 7fef68a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fef641550e0 0x7fef641eb260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:12.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.205+0000 7fef68a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef64152a30 con 0x7fef6406daa0 2024-02-21T23:09:12.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.205+0000 7fef68a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef640bdfb0 con 0x7fef641550e0 2024-02-21T23:09:12.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.205+0000 7fef68a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef640c7870 con 0x7fef64152650 2024-02-21T23:09:12.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.206+0000 7fef63400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef6406daa0 0x7fef64151260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:12.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.206+0000 7fef63400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef6406daa0 0x7fef64151260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57256/0 (socket says 172.21.15.28:57256) 2024-02-21T23:09:12.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.206+0000 7fef63400640 1 -- 172.21.15.28:0/1561821988 learned_addr learned my addr 172.21.15.28:0/1561821988 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:12.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.206+0000 7fef5aa00640 1 --2- 172.21.15.28:0/1561821988 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fef64152650 0x7fef641ead20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:12.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.206+0000 7fef63e00640 1 --2- 172.21.15.28:0/1561821988 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fef641550e0 0x7fef641eb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:12.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.206+0000 7fef63400640 1 -- 172.21.15.28:0/1561821988 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fef64152650 msgr2=0x7fef641ead20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:12.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.206+0000 7fef63400640 1 --2- 172.21.15.28:0/1561821988 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fef64152650 0x7fef641ead20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:12.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.206+0000 7fef63400640 1 -- 172.21.15.28:0/1561821988 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fef641550e0 msgr2=0x7fef641eb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:12.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.206+0000 7fef63400640 1 --2- 172.21.15.28:0/1561821988 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fef641550e0 0x7fef641eb260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:12.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.206+0000 7fef63400640 1 -- 172.21.15.28:0/1561821988 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef640aa500 con 0x7fef6406daa0 2024-02-21T23:09:12.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.207+0000 7fef63400640 1 --2- 172.21.15.28:0/1561821988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef6406daa0 0x7fef64151260 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7fef5400c400 tx=0x7fef5400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:12.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.207+0000 7fef5be00640 1 -- 172.21.15.28:0/1561821988 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef540052d0 con 0x7fef6406daa0 2024-02-21T23:09:12.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.207+0000 7fef5be00640 1 -- 172.21.15.28:0/1561821988 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef54005490 con 0x7fef6406daa0 2024-02-21T23:09:12.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.207+0000 7fef5be00640 1 -- 172.21.15.28:0/1561821988 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef54007260 con 0x7fef6406daa0 2024-02-21T23:09:12.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.207+0000 7fef68a00640 1 -- 172.21.15.28:0/1561821988 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef641ee430 con 0x7fef6406daa0 2024-02-21T23:09:12.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.207+0000 7fef68a00640 1 -- 172.21.15.28:0/1561821988 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef641517a0 con 0x7fef6406daa0 2024-02-21T23:09:12.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.209+0000 7fef68a00640 1 -- 172.21.15.28:0/1561821988 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef640aa290 con 0x7fef6406daa0 2024-02-21T23:09:12.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.209+0000 7fef5be00640 1 -- 172.21.15.28:0/1561821988 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fef54007400 con 0x7fef6406daa0 2024-02-21T23:09:12.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.210+0000 7fef5be00640 1 --2- 172.21.15.28:0/1561821988 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fef4006b4d0 0x7fef4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:12.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.210+0000 7fef5be00640 1 -- 172.21.15.28:0/1561821988 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fef540898f0 con 0x7fef6406daa0 2024-02-21T23:09:12.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.211+0000 7fef5aa00640 1 --2- 172.21.15.28:0/1561821988 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fef4006b4d0 0x7fef4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:12.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.212+0000 7fef5aa00640 1 --2- 172.21.15.28:0/1561821988 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fef4006b4d0 0x7fef4006d990 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fef4c036000 tx=0x7fef4c030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:12.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.214+0000 7fef5be00640 1 -- 172.21.15.28:0/1561821988 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fef540585d0 con 0x7fef6406daa0 2024-02-21T23:09:12.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.496+0000 7fef68a00640 1 -- 172.21.15.28:0/1561821988 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) v1 -- 0x7fef6404f290 con 0x7fef6406daa0 2024-02-21T23:09:12.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.502+0000 7fef5be00640 1 -- 172.21.15.28:0/1561821988 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v46)=0 v46) v1 ==== 116+0+0 (secure 0 0 0) 0x7fef54057ed0 con 0x7fef6406daa0 2024-02-21T23:09:12.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.505+0000 7fef68a00640 1 -- 172.21.15.28:0/1561821988 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fef4006b4d0 msgr2=0x7fef4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:12.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.505+0000 7fef68a00640 1 --2- 172.21.15.28:0/1561821988 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fef4006b4d0 0x7fef4006d990 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fef4c036000 tx=0x7fef4c030000 comp rx=0 tx=0).stop 2024-02-21T23:09:12.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.505+0000 7fef68a00640 1 -- 172.21.15.28:0/1561821988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef6406daa0 msgr2=0x7fef64151260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:12.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.505+0000 7fef68a00640 1 --2- 172.21.15.28:0/1561821988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef6406daa0 0x7fef64151260 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7fef5400c400 tx=0x7fef5400c8c0 comp rx=0 tx=0).stop 2024-02-21T23:09:12.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.506+0000 7fef68a00640 1 -- 172.21.15.28:0/1561821988 shutdown_connections 2024-02-21T23:09:12.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.506+0000 7fef68a00640 1 --2- 172.21.15.28:0/1561821988 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fef4006b4d0 0x7fef4006d990 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:12.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.506+0000 7fef68a00640 1 --2- 172.21.15.28:0/1561821988 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fef64152650 0x7fef641ead20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:12.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.506+0000 7fef68a00640 1 --2- 172.21.15.28:0/1561821988 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fef641550e0 0x7fef641eb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:12.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.506+0000 7fef68a00640 1 --2- 172.21.15.28:0/1561821988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef6406daa0 0x7fef64151260 unknown :-1 s=CLOSED pgs=1548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:12.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.506+0000 7fef68a00640 1 -- 172.21.15.28:0/1561821988 >> 172.21.15.28:0/1561821988 conn(0x7fef64072ce0 msgr2=0x7fef6406cfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:12.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.506+0000 7fef68a00640 1 -- 172.21.15.28:0/1561821988 shutdown_connections 2024-02-21T23:09:12.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.506+0000 7fef68a00640 1 -- 172.21.15.28:0/1561821988 wait complete. 2024-02-21T23:09:12.512 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.d mds_join_fs cephfs 2024-02-21T23:09:12.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.739+0000 7f729d400640 1 Processor -- start 2024-02-21T23:09:12.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.739+0000 7f729d400640 1 -- start start 2024-02-21T23:09:12.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.741+0000 7f729d400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f729806daa0 0x7f72980c8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:12.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.741+0000 7f729d400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7298158920 con 0x7f72981550e0 2024-02-21T23:09:12.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.741+0000 7f729d400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7298158c00 con 0x7f7298152650 2024-02-21T23:09:12.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.741+0000 7f729d400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72981586d0 con 0x7f729806daa0 2024-02-21T23:09:12.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.741+0000 7f729ca00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f72981550e0 0x7f72981554f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:56900/0 (socket says 172.21.15.28:56900) 2024-02-21T23:09:12.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.741+0000 7f729ca00640 1 -- 172.21.15.28:0/3548202548 learned_addr learned my addr 172.21.15.28:0/3548202548 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:09:12.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.742+0000 7f728fe00640 1 --2- 172.21.15.28:0/3548202548 >> v2:172.21.15.81:3300/0 conn(0x7f729806daa0 0x7f72980c8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:12.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.742+0000 7f7297400640 1 -- 172.21.15.28:0/3548202548 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 247317008 0 0) 0x7f7298158920 con 0x7f72981550e0 2024-02-21T23:09:12.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.742+0000 7f7297400640 1 -- 172.21.15.28:0/3548202548 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f727c003520 con 0x7f72981550e0 2024-02-21T23:09:12.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.742+0000 7f7297400640 1 -- 172.21.15.28:0/3548202548 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1296946952 0 0) 0x7f727c003520 con 0x7f72981550e0 2024-02-21T23:09:12.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.742+0000 7f7297400640 1 -- 172.21.15.28:0/3548202548 >> v2:172.21.15.81:3300/0 conn(0x7f729806daa0 msgr2=0x7f72980c8ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:12.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.742+0000 7f7297400640 1 --2- 172.21.15.28:0/3548202548 >> v2:172.21.15.81:3300/0 conn(0x7f729806daa0 0x7f72980c8ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:12.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.742+0000 7f7297400640 1 -- 172.21.15.28:0/3548202548 >> v1:172.21.15.81:6789/0 conn(0x7f7298152650 legacy=0x7f72980c7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:12.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.742+0000 7f7297400640 1 -- 172.21.15.28:0/3548202548 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7298158ee0 con 0x7f72981550e0 2024-02-21T23:09:12.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.742+0000 7f7297400640 1 -- 172.21.15.28:0/3548202548 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f72900019f0 con 0x7f72981550e0 2024-02-21T23:09:12.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.743+0000 7f7297400640 1 -- 172.21.15.28:0/3548202548 >> v1:172.21.15.28:6789/0 conn(0x7f72981550e0 legacy=0x7f72981554f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:12.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.743+0000 7f7297400640 1 --2- 172.21.15.28:0/3548202548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f727c0039b0 0x7f727c003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:12.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.743+0000 7f7297400640 1 --2- 172.21.15.28:0/3548202548 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f727c004800 0x7f727c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:12.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.743+0000 7f7297400640 1 --2- 172.21.15.28:0/3548202548 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f727c005210 0x7f727c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:12.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.743+0000 7f7297400640 1 -- 172.21.15.28:0/3548202548 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72981586d0 con 0x7f727c0039b0 2024-02-21T23:09:12.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.743+0000 7f7297400640 1 -- 172.21.15.28:0/3548202548 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f727c00c140 con 0x7f727c005210 2024-02-21T23:09:12.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.743+0000 7f7297400640 1 -- 172.21.15.28:0/3548202548 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f727c00c340 con 0x7f727c004800 2024-02-21T23:09:12.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.743+0000 7f7297e00640 1 --2- 172.21.15.28:0/3548202548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f727c0039b0 0x7f727c003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:12.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.743+0000 7f7297e00640 1 -- 172.21.15.28:0/3548202548 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f727c004800 msgr2=0x7f727c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:12.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.743+0000 7f7297e00640 1 --2- 172.21.15.28:0/3548202548 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f727c004800 0x7f727c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:12.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.743+0000 7f7297e00640 1 -- 172.21.15.28:0/3548202548 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f727c005210 msgr2=0x7f727c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:12.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.743+0000 7f7297e00640 1 --2- 172.21.15.28:0/3548202548 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f727c005210 0x7f727c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:12.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.743+0000 7f7297e00640 1 -- 172.21.15.28:0/3548202548 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f727c003520 con 0x7f727c0039b0 2024-02-21T23:09:12.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.744+0000 7f729ca00640 1 -- 172.21.15.28:0/3548202548 reap_dead start 2024-02-21T23:09:12.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.745+0000 7f7297e00640 1 --2- 172.21.15.28:0/3548202548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f727c0039b0 0x7f727c003df0 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f7288001750 tx=0x7f727c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab2cd8936dccd1e9 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:12.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.745+0000 7f7297400640 1 -- 172.21.15.28:0/3548202548 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7288001370 con 0x7f727c0039b0 2024-02-21T23:09:12.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.745+0000 7f7297400640 1 -- 172.21.15.28:0/3548202548 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72880052d0 con 0x7f727c0039b0 2024-02-21T23:09:12.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.745+0000 7f7297400640 1 -- 172.21.15.28:0/3548202548 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7288003830 con 0x7f727c0039b0 2024-02-21T23:09:12.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.746+0000 7f729d400640 1 -- 172.21.15.28:0/3548202548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f727c0039b0 msgr2=0x7f727c003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:12.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.746+0000 7f729d400640 1 --2- 172.21.15.28:0/3548202548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f727c0039b0 0x7f727c003df0 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f7288001750 tx=0x7f727c005830 comp rx=0 tx=0).stop 2024-02-21T23:09:12.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.746+0000 7f729d400640 1 -- 172.21.15.28:0/3548202548 shutdown_connections 2024-02-21T23:09:12.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.746+0000 7f729d400640 1 --2- 172.21.15.28:0/3548202548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f727c0039b0 0x7f727c003df0 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:12.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.746+0000 7f729d400640 1 -- 172.21.15.28:0/3548202548 >> 172.21.15.28:0/3548202548 conn(0x7f7298072ce0 msgr2=0x7f72980ac340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:12.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.746+0000 7f729d400640 1 -- 172.21.15.28:0/3548202548 shutdown_connections 2024-02-21T23:09:12.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.746+0000 7f729d400640 1 -- 172.21.15.28:0/3548202548 wait complete. 2024-02-21T23:09:12.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.748+0000 7f729d400640 1 Processor -- start 2024-02-21T23:09:12.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.748+0000 7f729d400640 1 -- start start 2024-02-21T23:09:12.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.749+0000 7f729d400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f727c0039b0 0x7f72980aa2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:12.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.749+0000 7f729d400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f729814e6a0 0x7f7298156480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:12.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.749+0000 7f729d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72980aa7f0 0x7f72981569c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:12.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.749+0000 7f729d400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7298152a30 con 0x7f72980aa7f0 2024-02-21T23:09:12.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.749+0000 7f729d400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72980bdfb0 con 0x7f729814e6a0 2024-02-21T23:09:12.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.749+0000 7f729d400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72980c8ce0 con 0x7f727c0039b0 2024-02-21T23:09:12.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.750+0000 7f729ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72980aa7f0 0x7f72981569c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:12.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.750+0000 7f729ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72980aa7f0 0x7f72981569c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57272/0 (socket says 172.21.15.28:57272) 2024-02-21T23:09:12.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.750+0000 7f728fe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f729814e6a0 0x7f7298156480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:12.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.750+0000 7f729ca00640 1 -- 172.21.15.28:0/612615236 learned_addr learned my addr 172.21.15.28:0/612615236 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:12.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.750+0000 7f7297e00640 1 --2- 172.21.15.28:0/612615236 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f727c0039b0 0x7f72980aa2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:12.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.750+0000 7f729ca00640 1 -- 172.21.15.28:0/612615236 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f727c0039b0 msgr2=0x7f72980aa2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:12.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.750+0000 7f729ca00640 1 --2- 172.21.15.28:0/612615236 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f727c0039b0 0x7f72980aa2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:12.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.751+0000 7f729ca00640 1 -- 172.21.15.28:0/612615236 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f729814e6a0 msgr2=0x7f7298156480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:12.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.751+0000 7f729ca00640 1 --2- 172.21.15.28:0/612615236 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f729814e6a0 0x7f7298156480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:12.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.751+0000 7f729ca00640 1 -- 172.21.15.28:0/612615236 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72980c4490 con 0x7f72980aa7f0 2024-02-21T23:09:12.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.751+0000 7f729ca00640 1 --2- 172.21.15.28:0/612615236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72980aa7f0 0x7f72981569c0 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f7290001370 tx=0x7f729000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:12.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.751+0000 7f7294c00640 1 -- 172.21.15.28:0/612615236 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7290018070 con 0x7f72980aa7f0 2024-02-21T23:09:12.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.751+0000 7f7294c00640 1 -- 172.21.15.28:0/612615236 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f729000a5b0 con 0x7f72980aa7f0 2024-02-21T23:09:12.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.751+0000 7f7294c00640 1 -- 172.21.15.28:0/612615236 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f729000cea0 con 0x7f72980aa7f0 2024-02-21T23:09:12.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.751+0000 7f729d400640 1 -- 172.21.15.28:0/612615236 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72980ac200 con 0x7f72980aa7f0 2024-02-21T23:09:12.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.752+0000 7f729d400640 1 -- 172.21.15.28:0/612615236 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72981ebde0 con 0x7f72980aa7f0 2024-02-21T23:09:12.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.753+0000 7f729d400640 1 -- 172.21.15.28:0/612615236 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f729804f290 con 0x7f72980aa7f0 2024-02-21T23:09:12.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.755+0000 7f7294c00640 1 -- 172.21.15.28:0/612615236 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f729000a980 con 0x7f72980aa7f0 2024-02-21T23:09:12.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.756+0000 7f7294c00640 1 --2- 172.21.15.28:0/612615236 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f727406b500 0x7f727406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:12.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.756+0000 7f7294c00640 1 -- 172.21.15.28:0/612615236 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f7290089730 con 0x7f72980aa7f0 2024-02-21T23:09:12.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.757+0000 7f7297e00640 1 --2- 172.21.15.28:0/612615236 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f727406b500 0x7f727406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:12.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.758+0000 7f7297e00640 1 --2- 172.21.15.28:0/612615236 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f727406b500 0x7f727406d9c0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f7288033000 tx=0x7f7288009030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:12.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.761+0000 7f7294c00640 1 -- 172.21.15.28:0/612615236 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7290057cb0 con 0x7f72980aa7f0 2024-02-21T23:09:12.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.965+0000 7f729d400640 1 -- 172.21.15.28:0/612615236 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) v1 -- 0x7f7298156080 con 0x7f72980aa7f0 2024-02-21T23:09:12.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.971+0000 7f7294c00640 1 -- 172.21.15.28:0/612615236 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v47)=0 v47) v1 ==== 116+0+0 (secure 0 0 0) 0x7f729005bd50 con 0x7f72980aa7f0 2024-02-21T23:09:12.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.974+0000 7f729d400640 1 -- 172.21.15.28:0/612615236 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f727406b500 msgr2=0x7f727406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:12.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.974+0000 7f729d400640 1 --2- 172.21.15.28:0/612615236 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f727406b500 0x7f727406d9c0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f7288033000 tx=0x7f7288009030 comp rx=0 tx=0).stop 2024-02-21T23:09:12.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.974+0000 7f729d400640 1 -- 172.21.15.28:0/612615236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72980aa7f0 msgr2=0x7f72981569c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:12.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.974+0000 7f729d400640 1 --2- 172.21.15.28:0/612615236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72980aa7f0 0x7f72981569c0 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f7290001370 tx=0x7f729000cbf0 comp rx=0 tx=0).stop 2024-02-21T23:09:12.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.974+0000 7f729d400640 1 -- 172.21.15.28:0/612615236 shutdown_connections 2024-02-21T23:09:12.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.974+0000 7f729d400640 1 --2- 172.21.15.28:0/612615236 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f727406b500 0x7f727406d9c0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:12.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.974+0000 7f729d400640 1 --2- 172.21.15.28:0/612615236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72980aa7f0 0x7f72981569c0 unknown :-1 s=CLOSED pgs=1551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:12.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.974+0000 7f729d400640 1 --2- 172.21.15.28:0/612615236 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f729814e6a0 0x7f7298156480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:12.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.974+0000 7f729d400640 1 --2- 172.21.15.28:0/612615236 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f727c0039b0 0x7f72980aa2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:12.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.974+0000 7f729d400640 1 -- 172.21.15.28:0/612615236 >> 172.21.15.28:0/612615236 conn(0x7f7298072ce0 msgr2=0x7f72980c7160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:12.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.974+0000 7f729d400640 1 -- 172.21.15.28:0/612615236 shutdown_connections 2024-02-21T23:09:12.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:12.975+0000 7f729d400640 1 -- 172.21.15.28:0/612615236 wait complete. 2024-02-21T23:09:12.990 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:09:13.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.220+0000 7f0e6be00640 1 Processor -- start 2024-02-21T23:09:13.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.220+0000 7f0e6be00640 1 -- start start 2024-02-21T23:09:13.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.222+0000 7f0e6be00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0e6c150450 0x7f0e6c0a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:13.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.222+0000 7f0e6be00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0e6c156af0 0x7f0e6c06dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:13.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.222+0000 7f0e6be00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e6c0c2760 con 0x7f0e6c06ee40 2024-02-21T23:09:13.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.222+0000 7f0e6be00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e6c0c22c0 con 0x7f0e6c156af0 2024-02-21T23:09:13.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.222+0000 7f0e6be00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e6c156660 con 0x7f0e6c150450 2024-02-21T23:09:13.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.223+0000 7f0e6aa00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0e6c150450 0x7f0e6c0a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:13.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.223+0000 7f0e6b400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0e6c156af0 0x7f0e6c06dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:13.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.223+0000 7f0e6aa00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0e6c150450 0x7f0e6c0a5fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3301/0 says I am v2:172.21.15.28:44938/0 (socket says 172.21.15.28:44938) 2024-02-21T23:09:13.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.223+0000 7f0e6aa00640 1 -- 172.21.15.28:0/2563701668 learned_addr learned my addr 172.21.15.28:0/2563701668 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:13.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.223+0000 7f0e6b400640 1 -- 172.21.15.28:0/2563701668 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0e6c150450 msgr2=0x7f0e6c0a5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:13.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.223+0000 7f0e6b400640 1 --2- 172.21.15.28:0/2563701668 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0e6c150450 0x7f0e6c0a5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:13.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.223+0000 7f0e6b400640 1 -- 172.21.15.28:0/2563701668 >> v1:172.21.15.81:6789/0 conn(0x7f0e6c06ee40 legacy=0x7f0e6c1536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:13.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.223+0000 7f0e6b400640 1 -- 172.21.15.28:0/2563701668 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e6c156310 con 0x7f0e6c156af0 2024-02-21T23:09:13.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.225+0000 7f0e6b400640 1 --2- 172.21.15.28:0/2563701668 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0e6c156af0 0x7f0e6c06dff0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f0e64009d90 tx=0x7f0e6402bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b7c8499c9d990c6 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:13.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.225+0000 7f0e6a000640 1 -- 172.21.15.28:0/2563701668 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e64039070 con 0x7f0e6c156af0 2024-02-21T23:09:13.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.225+0000 7f0e6a000640 1 -- 172.21.15.28:0/2563701668 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e6402cc50 con 0x7f0e6c156af0 2024-02-21T23:09:13.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.225+0000 7f0e6a000640 1 -- 172.21.15.28:0/2563701668 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e64034440 con 0x7f0e6c156af0 2024-02-21T23:09:13.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.226+0000 7f0e6be00640 1 -- 172.21.15.28:0/2563701668 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0e6c156af0 msgr2=0x7f0e6c06dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:13.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.226+0000 7f0e6be00640 1 --2- 172.21.15.28:0/2563701668 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0e6c156af0 0x7f0e6c06dff0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f0e64009d90 tx=0x7f0e6402bea0 comp rx=0 tx=0).stop 2024-02-21T23:09:13.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.226+0000 7f0e6be00640 1 -- 172.21.15.28:0/2563701668 shutdown_connections 2024-02-21T23:09:13.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.226+0000 7f0e6be00640 1 --2- 172.21.15.28:0/2563701668 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0e6c156af0 0x7f0e6c06dff0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:13.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.226+0000 7f0e6be00640 1 --2- 172.21.15.28:0/2563701668 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0e6c150450 0x7f0e6c0a5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:13.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.226+0000 7f0e6be00640 1 -- 172.21.15.28:0/2563701668 >> 172.21.15.28:0/2563701668 conn(0x7f0e6c06b630 msgr2=0x7f0e6c0c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:13.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.226+0000 7f0e6be00640 1 -- 172.21.15.28:0/2563701668 shutdown_connections 2024-02-21T23:09:13.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.226+0000 7f0e6be00640 1 -- 172.21.15.28:0/2563701668 wait complete. 2024-02-21T23:09:13.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.228+0000 7f0e6be00640 1 Processor -- start 2024-02-21T23:09:13.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.228+0000 7f0e6be00640 1 -- start start 2024-02-21T23:09:13.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.229+0000 7f0e6be00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0e6c06ee40 0x7f0e6c149a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:13.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.229+0000 7f0e6be00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0e6c150450 0x7f0e6c0c5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:13.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.229+0000 7f0e6be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e6c156af0 0x7f0e6c150b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:13.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.229+0000 7f0e6be00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e6c156f80 con 0x7f0e6c156af0 2024-02-21T23:09:13.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.229+0000 7f0e6be00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e6c0c2cb0 con 0x7f0e6c150450 2024-02-21T23:09:13.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.229+0000 7f0e6be00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e6c157100 con 0x7f0e6c06ee40 2024-02-21T23:09:13.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.229+0000 7f0e6b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e6c156af0 0x7f0e6c150b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:13.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.229+0000 7f0e6b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e6c156af0 0x7f0e6c150b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57284/0 (socket says 172.21.15.28:57284) 2024-02-21T23:09:13.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.229+0000 7f0e6b400640 1 -- 172.21.15.28:0/703198188 learned_addr learned my addr 172.21.15.28:0/703198188 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:13.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.230+0000 7f0e6aa00640 1 --2- 172.21.15.28:0/703198188 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0e6c06ee40 0x7f0e6c149a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:13.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.230+0000 7f0e62000640 1 --2- 172.21.15.28:0/703198188 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0e6c150450 0x7f0e6c0c5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:13.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.230+0000 7f0e6b400640 1 -- 172.21.15.28:0/703198188 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0e6c06ee40 msgr2=0x7f0e6c149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:13.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.230+0000 7f0e6b400640 1 --2- 172.21.15.28:0/703198188 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0e6c06ee40 0x7f0e6c149a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:13.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.230+0000 7f0e6b400640 1 -- 172.21.15.28:0/703198188 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0e6c150450 msgr2=0x7f0e6c0c5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:13.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.230+0000 7f0e6b400640 1 --2- 172.21.15.28:0/703198188 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0e6c150450 0x7f0e6c0c5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:13.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.230+0000 7f0e6b400640 1 -- 172.21.15.28:0/703198188 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e6c14c6e0 con 0x7f0e6c156af0 2024-02-21T23:09:13.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.230+0000 7f0e6b400640 1 --2- 172.21.15.28:0/703198188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e6c156af0 0x7f0e6c150b30 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f0e64009eb0 tx=0x7f0e64004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:13.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.231+0000 7f0e63400640 1 -- 172.21.15.28:0/703198188 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e64039050 con 0x7f0e6c156af0 2024-02-21T23:09:13.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.231+0000 7f0e63400640 1 -- 172.21.15.28:0/703198188 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e64045070 con 0x7f0e6c156af0 2024-02-21T23:09:13.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.231+0000 7f0e63400640 1 -- 172.21.15.28:0/703198188 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e64009450 con 0x7f0e6c156af0 2024-02-21T23:09:13.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.231+0000 7f0e6be00640 1 -- 172.21.15.28:0/703198188 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e6c14b2a0 con 0x7f0e6c156af0 2024-02-21T23:09:13.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.231+0000 7f0e6be00640 1 -- 172.21.15.28:0/703198188 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e6c0c6420 con 0x7f0e6c156af0 2024-02-21T23:09:13.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.232+0000 7f0e6be00640 1 -- 172.21.15.28:0/703198188 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e6c04f290 con 0x7f0e6c156af0 2024-02-21T23:09:13.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.233+0000 7f0e63400640 1 -- 172.21.15.28:0/703198188 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0e6403e610 con 0x7f0e6c156af0 2024-02-21T23:09:13.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.234+0000 7f0e63400640 1 --2- 172.21.15.28:0/703198188 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0e4806b4d0 0x7f0e4806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:13.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.234+0000 7f0e63400640 1 -- 172.21.15.28:0/703198188 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f0e640acdc0 con 0x7f0e6c156af0 2024-02-21T23:09:13.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.235+0000 7f0e6aa00640 1 --2- 172.21.15.28:0/703198188 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0e4806b4d0 0x7f0e4806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:13.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.236+0000 7f0e6aa00640 1 --2- 172.21.15.28:0/703198188 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0e4806b4d0 0x7f0e4806d990 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f0e6c14bef0 tx=0x7f0e5c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:13.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.240+0000 7f0e63400640 1 -- 172.21.15.28:0/703198188 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f0e6407bb40 con 0x7f0e6c156af0 2024-02-21T23:09:13.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.481+0000 7f0e6be00640 1 -- 172.21.15.28:0/703198188 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0e6c0c1f10 con 0x7f0e6c156af0 2024-02-21T23:09:13.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.483+0000 7f0e63400640 1 -- 172.21.15.28:0/703198188 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 139 v139) v1 ==== 95+0+4948 (secure 0 0 0) 0x7f0e6407b440 con 0x7f0e6c156af0 2024-02-21T23:09:13.471 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:09:13.471 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":139,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6140,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/4241906475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":4241906475},{"type":"v1","addr":"172.21.15.81:6836","nonce":4241906475}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":133},{"gid":6143,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/2095908433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":2095908433},{"type":"v1","addr":"172.21.15.81:6837","nonce":2095908433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":133},{"gid":8370,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/3934548735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3934548735},{"type":"v1","addr":"172.21.15.28:6833","nonce":3934548735}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":133}],"filesystems":[{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:09:03.869294+0000","modified":"2024-02-21T23:09:07.201932+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8376},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8376":{"gid":8376,"name":"a","rank":0,"incarnation":135,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/4151254007","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4151254007},{"type":"v1","addr":"172.21.15.28:6835","nonce":4151254007}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-02-21T23:09:13.471 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 139 2024-02-21T23:09:13.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.485+0000 7f0e6be00640 1 -- 172.21.15.28:0/703198188 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0e4806b4d0 msgr2=0x7f0e4806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:13.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.485+0000 7f0e6be00640 1 --2- 172.21.15.28:0/703198188 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0e4806b4d0 0x7f0e4806d990 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f0e6c14bef0 tx=0x7f0e5c008040 comp rx=0 tx=0).stop 2024-02-21T23:09:13.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.485+0000 7f0e6be00640 1 -- 172.21.15.28:0/703198188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e6c156af0 msgr2=0x7f0e6c150b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:13.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.485+0000 7f0e6be00640 1 --2- 172.21.15.28:0/703198188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e6c156af0 0x7f0e6c150b30 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f0e64009eb0 tx=0x7f0e64004a40 comp rx=0 tx=0).stop 2024-02-21T23:09:13.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.486+0000 7f0e6be00640 1 -- 172.21.15.28:0/703198188 shutdown_connections 2024-02-21T23:09:13.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.486+0000 7f0e6be00640 1 --2- 172.21.15.28:0/703198188 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0e4806b4d0 0x7f0e4806d990 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:13.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.486+0000 7f0e6be00640 1 --2- 172.21.15.28:0/703198188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e6c156af0 0x7f0e6c150b30 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:13.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.486+0000 7f0e6be00640 1 --2- 172.21.15.28:0/703198188 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0e6c150450 0x7f0e6c0c5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:13.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.486+0000 7f0e6be00640 1 --2- 172.21.15.28:0/703198188 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0e6c06ee40 0x7f0e6c149a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:13.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.486+0000 7f0e6be00640 1 -- 172.21.15.28:0/703198188 >> 172.21.15.28:0/703198188 conn(0x7f0e6c06b630 msgr2=0x7f0e6c0a6fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:13.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.486+0000 7f0e6be00640 1 -- 172.21.15.28:0/703198188 shutdown_connections 2024-02-21T23:09:13.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:13.486+0000 7f0e6be00640 1 -- 172.21.15.28:0/703198188 wait complete. 2024-02-21T23:09:13.898 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 139, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:09:03.869294+0000', 'modified': '2024-02-21T23:09:07.201932+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8376}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8376': {'gid': 8376, 'name': 'a', 'rank': 0, 'incarnation': 135, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/4151254007', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 4151254007}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 4151254007}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:09:13.899 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-21T23:09:13.899 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-02-21T23:09:14.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.135+0000 7fdccd400640 1 Processor -- start 2024-02-21T23:09:14.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.135+0000 7fdccd400640 1 -- start start 2024-02-21T23:09:14.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.137+0000 7fdccd400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fdcc80c2df0 0x7fdcc8156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:14.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.137+0000 7fdccd400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdcc8153220 0x7fdcc8153630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:14.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.137+0000 7fdccd400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcc80c5c10 con 0x7fdcc80c5f20 2024-02-21T23:09:14.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.137+0000 7fdccd400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcc806e6a0 con 0x7fdcc8153220 2024-02-21T23:09:14.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.137+0000 7fdccd400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcc806e8d0 con 0x7fdcc80c2df0 2024-02-21T23:09:14.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.138+0000 7fdcc7200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdcc8153220 0x7fdcc8153630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:14.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.138+0000 7fdcc7200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdcc8153220 0x7fdcc8153630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57286/0 (socket says 172.21.15.28:57286) 2024-02-21T23:09:14.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.138+0000 7fdcc7200640 1 -- 172.21.15.28:0/3613026697 learned_addr learned my addr 172.21.15.28:0/3613026697 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:14.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.138+0000 7fdcc6800640 1 --2- 172.21.15.28:0/3613026697 >> v2:172.21.15.81:3300/0 conn(0x7fdcc80c2df0 0x7fdcc8156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:14.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.138+0000 7fdcc7200640 1 -- 172.21.15.28:0/3613026697 >> v2:172.21.15.81:3300/0 conn(0x7fdcc80c2df0 msgr2=0x7fdcc8156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:14.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.138+0000 7fdcc7200640 1 --2- 172.21.15.28:0/3613026697 >> v2:172.21.15.81:3300/0 conn(0x7fdcc80c2df0 0x7fdcc8156ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:14.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.138+0000 7fdcc7200640 1 -- 172.21.15.28:0/3613026697 >> v1:172.21.15.81:6789/0 conn(0x7fdcc80c5f20 legacy=0x7fdcc80c31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:14.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.138+0000 7fdcc7200640 1 -- 172.21.15.28:0/3613026697 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcc806ebb0 con 0x7fdcc8153220 2024-02-21T23:09:14.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.139+0000 7fdcc7200640 1 --2- 172.21.15.28:0/3613026697 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcc8153220 0x7fdcc8153630 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7fdcc806f760 tx=0x7fdcc002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba24d74c28d52dbe server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:14.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.140+0000 7fdcc5400640 1 -- 172.21.15.28:0/3613026697 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc002ca80 con 0x7fdcc8153220 2024-02-21T23:09:14.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.140+0000 7fdcc5400640 1 -- 172.21.15.28:0/3613026697 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcc002cc20 con 0x7fdcc8153220 2024-02-21T23:09:14.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.140+0000 7fdcc5400640 1 -- 172.21.15.28:0/3613026697 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc00343f0 con 0x7fdcc8153220 2024-02-21T23:09:14.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.140+0000 7fdccd400640 1 -- 172.21.15.28:0/3613026697 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcc8153220 msgr2=0x7fdcc8153630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:14.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.140+0000 7fdccd400640 1 --2- 172.21.15.28:0/3613026697 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcc8153220 0x7fdcc8153630 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7fdcc806f760 tx=0x7fdcc002bee0 comp rx=0 tx=0).stop 2024-02-21T23:09:14.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.140+0000 7fdccd400640 1 -- 172.21.15.28:0/3613026697 shutdown_connections 2024-02-21T23:09:14.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.140+0000 7fdccd400640 1 --2- 172.21.15.28:0/3613026697 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcc8153220 0x7fdcc8153630 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:14.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.140+0000 7fdccd400640 1 --2- 172.21.15.28:0/3613026697 >> v2:172.21.15.81:3300/0 conn(0x7fdcc80c2df0 0x7fdcc8156ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:14.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.140+0000 7fdccd400640 1 -- 172.21.15.28:0/3613026697 >> 172.21.15.28:0/3613026697 conn(0x7fdcc806b820 msgr2=0x7fdcc80c67f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:14.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.140+0000 7fdccd400640 1 -- 172.21.15.28:0/3613026697 shutdown_connections 2024-02-21T23:09:14.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.140+0000 7fdccd400640 1 -- 172.21.15.28:0/3613026697 wait complete. 2024-02-21T23:09:14.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.142+0000 7fdccd400640 1 Processor -- start 2024-02-21T23:09:14.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.142+0000 7fdccd400640 1 -- start start 2024-02-21T23:09:14.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.143+0000 7fdccd400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcc80c2df0 0x7fdcc80aa880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:14.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.143+0000 7fdccd400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdcc80c5f20 0x7fdcc80bfc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:14.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.143+0000 7fdccd400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdcc8153220 0x7fdcc81ee990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:14.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.143+0000 7fdccd400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc80c38e0 con 0x7fdcc80c2df0 2024-02-21T23:09:14.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.143+0000 7fdccd400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc806f170 con 0x7fdcc8153220 2024-02-21T23:09:14.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.143+0000 7fdccd400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc80c3a60 con 0x7fdcc80c5f20 2024-02-21T23:09:14.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.143+0000 7fdcc6800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcc80c2df0 0x7fdcc80aa880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:14.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.143+0000 7fdcc6800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcc80c2df0 0x7fdcc80aa880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57302/0 (socket says 172.21.15.28:57302) 2024-02-21T23:09:14.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.143+0000 7fdcc6800640 1 -- 172.21.15.28:0/850954437 learned_addr learned my addr 172.21.15.28:0/850954437 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:14.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.144+0000 7fdcc7200640 1 --2- 172.21.15.28:0/850954437 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdcc8153220 0x7fdcc81ee990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:14.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.144+0000 7fdcc5e00640 1 --2- 172.21.15.28:0/850954437 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdcc80c5f20 0x7fdcc80bfc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:14.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.144+0000 7fdcc6800640 1 -- 172.21.15.28:0/850954437 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdcc80c5f20 msgr2=0x7fdcc80bfc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:14.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.144+0000 7fdcc6800640 1 --2- 172.21.15.28:0/850954437 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdcc80c5f20 0x7fdcc80bfc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:14.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.144+0000 7fdcc6800640 1 -- 172.21.15.28:0/850954437 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdcc8153220 msgr2=0x7fdcc81ee990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:14.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.144+0000 7fdcc6800640 1 --2- 172.21.15.28:0/850954437 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdcc8153220 0x7fdcc81ee990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:14.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.144+0000 7fdcc6800640 1 -- 172.21.15.28:0/850954437 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcc80aae80 con 0x7fdcc80c2df0 2024-02-21T23:09:14.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.144+0000 7fdcc5e00640 1 --2- 172.21.15.28:0/850954437 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdcc80c5f20 0x7fdcc80bfc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:09:14.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.144+0000 7fdcc6800640 1 --2- 172.21.15.28:0/850954437 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcc80c2df0 0x7fdcc80aa880 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7fdcb800c7f0 tx=0x7fdcb800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:14.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.144+0000 7fdcbea00640 1 -- 172.21.15.28:0/850954437 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcb8015070 con 0x7fdcc80c2df0 2024-02-21T23:09:14.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.145+0000 7fdcbea00640 1 -- 172.21.15.28:0/850954437 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcb8004220 con 0x7fdcc80c2df0 2024-02-21T23:09:14.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.145+0000 7fdcbea00640 1 -- 172.21.15.28:0/850954437 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcb8007620 con 0x7fdcc80c2df0 2024-02-21T23:09:14.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.145+0000 7fdccd400640 1 -- 172.21.15.28:0/850954437 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcc80ab160 con 0x7fdcc80c2df0 2024-02-21T23:09:14.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.145+0000 7fdccd400640 1 -- 172.21.15.28:0/850954437 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcc81ee470 con 0x7fdcc80c2df0 2024-02-21T23:09:14.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.147+0000 7fdcbea00640 1 -- 172.21.15.28:0/850954437 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdcb80079d0 con 0x7fdcc80c2df0 2024-02-21T23:09:14.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.148+0000 7fdcbea00640 1 --2- 172.21.15.28:0/850954437 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdca806b4d0 0x7fdca806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:14.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.148+0000 7fdcbea00640 1 -- 172.21.15.28:0/850954437 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fdcb808a720 con 0x7fdcc80c2df0 2024-02-21T23:09:14.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.148+0000 7fdcc5e00640 1 --2- 172.21.15.28:0/850954437 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdca806b4d0 0x7fdca806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:14.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.149+0000 7fdcd01fa740 1 Processor -- start 2024-02-21T23:09:14.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.149+0000 7fdcd01fa740 1 -- start start 2024-02-21T23:09:14.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.149+0000 7fdcc5e00640 1 --2- 172.21.15.28:0/850954437 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdca806b4d0 0x7fdca806d990 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fdcc8068420 tx=0x7fdcb4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:14.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.150+0000 7fdcd01fa740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c417a90e40 0x55c417a91220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:14.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.150+0000 7fdcd01fa740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c417a92150 0x55c417accf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:14.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.150+0000 7fdcd01fa740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c417acd4d0 0x55c417acb540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:14.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.150+0000 7fdcd01fa740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4179f5470 con 0x55c417acd4d0 2024-02-21T23:09:14.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.150+0000 7fdcd01fa740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c41799eaf0 con 0x55c417a90e40 2024-02-21T23:09:14.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.150+0000 7fdcd01fa740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c4179f1510 con 0x55c417a92150 2024-02-21T23:09:14.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.150+0000 7fdcc7200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c417acd4d0 0x55c417acb540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:14.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.150+0000 7fdcc7200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c417acd4d0 0x55c417acb540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57310/0 (socket says 172.21.15.28:57310) 2024-02-21T23:09:14.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.150+0000 7fdcc7200640 1 -- 172.21.15.28:0/3322862767 learned_addr learned my addr 172.21.15.28:0/3322862767 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:14.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.150+0000 7fdcc6800640 1 --2- 172.21.15.28:0/3322862767 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c417a90e40 0x55c417a91220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:14.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.150+0000 7fdcc5e00640 1 --2- 172.21.15.28:0/3322862767 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c417a92150 0x55c417accf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:14.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.150+0000 7fdcc7200640 1 -- 172.21.15.28:0/3322862767 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c417a92150 msgr2=0x55c417accf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:14.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.150+0000 7fdcc7200640 1 --2- 172.21.15.28:0/3322862767 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c417a92150 0x55c417accf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:14.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.150+0000 7fdcc7200640 1 -- 172.21.15.28:0/3322862767 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c417a90e40 msgr2=0x55c417a91220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:14.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.150+0000 7fdcc7200640 1 --2- 172.21.15.28:0/3322862767 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c417a90e40 0x55c417a91220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:14.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.150+0000 7fdcc7200640 1 -- 172.21.15.28:0/3322862767 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c4179a5d90 con 0x55c417acd4d0 2024-02-21T23:09:14.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.151+0000 7fdcc7200640 1 --2- 172.21.15.28:0/3322862767 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c417acd4d0 0x55c417acb540 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7fdcc0009f10 tx=0x7fdcc002e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a07fcb764a10e3ee server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:14.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.151+0000 7fdcb3000640 1 -- 172.21.15.28:0/3322862767 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc0045070 con 0x55c417acd4d0 2024-02-21T23:09:14.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.151+0000 7fdcb3000640 1 -- 172.21.15.28:0/3322862767 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcc002cac0 con 0x55c417acd4d0 2024-02-21T23:09:14.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.151+0000 7fdcb3000640 1 -- 172.21.15.28:0/3322862767 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc00406b0 con 0x55c417acd4d0 2024-02-21T23:09:14.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.152+0000 7fdcd01fa740 1 -- 172.21.15.28:0/3322862767 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c417acd4d0 msgr2=0x55c417acb540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:14.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.152+0000 7fdcd01fa740 1 --2- 172.21.15.28:0/3322862767 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c417acd4d0 0x55c417acb540 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7fdcc0009f10 tx=0x7fdcc002e000 comp rx=0 tx=0).stop 2024-02-21T23:09:14.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.152+0000 7fdcd01fa740 1 -- 172.21.15.28:0/3322862767 shutdown_connections 2024-02-21T23:09:14.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.152+0000 7fdcd01fa740 1 --2- 172.21.15.28:0/3322862767 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c417a92150 0x55c417accf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:14.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.152+0000 7fdcd01fa740 1 --2- 172.21.15.28:0/3322862767 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c417acd4d0 0x55c417acb540 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:14.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.152+0000 7fdcd01fa740 1 --2- 172.21.15.28:0/3322862767 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c417a90e40 0x55c417a91220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:14.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.152+0000 7fdcd01fa740 1 -- 172.21.15.28:0/3322862767 >> 172.21.15.28:0/3322862767 conn(0x55c417a8d790 msgr2=0x55c417ab21a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:14.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.152+0000 7fdcd01fa740 1 -- 172.21.15.28:0/3322862767 shutdown_connections 2024-02-21T23:09:14.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.152+0000 7fdcd01fa740 1 -- 172.21.15.28:0/3322862767 wait complete. 2024-02-21T23:09:14.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.152+0000 7fdcd01fa740 1 -- start start 2024-02-21T23:09:14.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.153+0000 7fdcd01fa740 1 Processor -- start 2024-02-21T23:09:14.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.154+0000 7fdcd01fa740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c417a92150 0x55c417aca510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:14.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.154+0000 7fdcd01fa740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c417acd4d0 0x55c417adda20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:14.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.154+0000 7fdcd01fa740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c417ad1700 0x55c417aacf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:14.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.154+0000 7fdcd01fa740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c417994040 con 0x55c417a92150 2024-02-21T23:09:14.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.154+0000 7fdcd01fa740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c417ab28b0 con 0x55c417acd4d0 2024-02-21T23:09:14.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.154+0000 7fdcd01fa740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c417ad2ea0 con 0x55c417ad1700 2024-02-21T23:09:14.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.154+0000 7fdcd01fa740 10 client.0 _ll_register_callbacks cb 0x55c4179b1920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:09:14.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.154+0000 7fdcc6800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c417a92150 0x55c417aca510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:14.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.154+0000 7fdcc6800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c417a92150 0x55c417aca510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57312/0 (socket says 172.21.15.28:57312) 2024-02-21T23:09:14.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.154+0000 7fdcc6800640 1 -- 172.21.15.28:0/338666729 learned_addr learned my addr 172.21.15.28:0/338666729 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:14.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.154+0000 7fdcc7200640 1 --2- 172.21.15.28:0/338666729 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c417ad1700 0x55c417aacf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:14.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.154+0000 7fdcc6800640 1 -- 172.21.15.28:0/338666729 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c417ad1700 msgr2=0x55c417aacf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:14.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.154+0000 7fdcc5e00640 1 --2- 172.21.15.28:0/338666729 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c417acd4d0 0x55c417adda20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:14.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.154+0000 7fdcc6800640 1 --2- 172.21.15.28:0/338666729 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c417ad1700 0x55c417aacf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:14.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.154+0000 7fdcc6800640 1 -- 172.21.15.28:0/338666729 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c417acd4d0 msgr2=0x55c417adda20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:14.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.154+0000 7fdcc6800640 1 --2- 172.21.15.28:0/338666729 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c417acd4d0 0x55c417adda20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:14.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.154+0000 7fdcc6800640 1 -- 172.21.15.28:0/338666729 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c4179b3aa0 con 0x55c417a92150 2024-02-21T23:09:14.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.155+0000 7fdcc6800640 1 --2- 172.21.15.28:0/338666729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c417a92150 0x55c417aca510 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7fdcb8004630 tx=0x7fdcb805ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cc015f7fac69d9b server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:14.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.155+0000 7fdca7400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:09:14.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.155+0000 7fdca7400640 1 -- 172.21.15.28:0/338666729 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcb801e040 con 0x55c417a92150 2024-02-21T23:09:14.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.155+0000 7fdca7400640 1 -- 172.21.15.28:0/338666729 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(136..136 src has 1..136) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fdcb8007cc0 con 0x55c417a92150 2024-02-21T23:09:14.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.155+0000 7fdcd01fa740 1 -- 172.21.15.28:0/338666729 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdcc002cac0 con 0x55c417a92150 2024-02-21T23:09:14.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.155+0000 7fdca7400640 1 -- 172.21.15.28:0/338666729 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=139) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdcb8060f90 con 0x55c417a92150 2024-02-21T23:09:14.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.155+0000 7fdcd01fa740 10 client.8541 fetch_fsmap learned FSMap version 139 2024-02-21T23:09:14.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.155+0000 7fdcd01fa740 1 -- 172.21.15.28:0/338666729 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=139,monmap=2+}) v3 -- 0x55c41770a2d0 con 0x55c417a92150 2024-02-21T23:09:14.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.155+0000 7fdca7400640 1 -- 172.21.15.28:0/338666729 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcc002cac0 con 0x55c417a92150 2024-02-21T23:09:14.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.156+0000 7fdca7400640 1 -- 172.21.15.28:0/338666729 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 139) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fdcb8058160 con 0x55c417a92150 2024-02-21T23:09:14.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.156+0000 7fdcd01fa740 10 client.8541 fetch_fsmap finished waiting for FSMap version 139 2024-02-21T23:09:14.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.156+0000 7fdcd01fa740 10 client.8541 resolve_mds: resolved name 'a' to daemon mds.a 2024-02-21T23:09:14.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.156+0000 7fdcd01fa740 20 client.8541 populate_metadata read hostname 'smithi028' 2024-02-21T23:09:14.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.156+0000 7fdcd01fa740 1 --2- 172.21.15.28:0/338666729 >> [v2:172.21.15.28:6834/4151254007,v1:172.21.15.28:6835/4151254007] conn(0x55c417ae1230 0x55c417aec040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:14.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.156+0000 7fdcd01fa740 4 client.8541 mds_command: new command op to 8376 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-21T23:09:14.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.156+0000 7fdcd01fa740 1 -- 172.21.15.28:0/338666729 --> [v2:172.21.15.28:6834/4151254007,v1:172.21.15.28:6835/4151254007] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c417a029b0 con 0x55c417ae1230 2024-02-21T23:09:14.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.156+0000 7fdcc5e00640 1 --2- 172.21.15.28:0/338666729 >> [v2:172.21.15.28:6834/4151254007,v1:172.21.15.28:6835/4151254007] conn(0x55c417ae1230 0x55c417aec040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:14.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.157+0000 7fdca7400640 10 client.8541 ms_handle_connect on v2:172.21.15.28:6834/4151254007 2024-02-21T23:09:14.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.157+0000 7fdcc5e00640 1 --2- 172.21.15.28:0/338666729 >> [v2:172.21.15.28:6834/4151254007,v1:172.21.15.28:6835/4151254007] conn(0x55c417ae1230 0x55c417aec040 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b2aedb4320e2c53 server_cookie=f053478295ae3bdf in_seq=0 out_seq=0 2024-02-21T23:09:14.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.157+0000 7fdca7400640 1 -- 172.21.15.28:0/338666729 <== mds.0 v2:172.21.15.28:6834/4151254007 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c417a029b0 con 0x55c417ae1230 2024-02-21T23:09:14.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.157+0000 7fdca7400640 10 client.8541 handle_command_reply: tid=0 2024-02-21T23:09:14.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.158+0000 7fdcd01fa740 1 client.8541 shutdown 2024-02-21T23:09:14.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.158+0000 7fdcd01fa740 1 -- 172.21.15.28:0/338666729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c417a92150 msgr2=0x55c417aca510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:14.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.158+0000 7fdcd01fa740 1 --2- 172.21.15.28:0/338666729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c417a92150 0x55c417aca510 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7fdcb8004630 tx=0x7fdcb805ce60 comp rx=0 tx=0).stop 2024-02-21T23:09:14.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.158+0000 7fdcd01fa740 1 -- 172.21.15.28:0/338666729 shutdown_connections 2024-02-21T23:09:14.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.158+0000 7fdcd01fa740 1 --2- 172.21.15.28:0/338666729 >> [v2:172.21.15.28:6834/4151254007,v1:172.21.15.28:6835/4151254007] conn(0x55c417ae1230 0x55c417aec040 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:14.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.158+0000 7fdcd01fa740 1 --2- 172.21.15.28:0/338666729 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c417ad1700 0x55c417aacf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:14.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.158+0000 7fdcd01fa740 1 --2- 172.21.15.28:0/338666729 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c417acd4d0 0x55c417adda20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:14.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.158+0000 7fdcd01fa740 1 --2- 172.21.15.28:0/338666729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c417a92150 0x55c417aca510 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:14.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.158+0000 7fdca7400640 0 client.8541 ms_handle_reset on v2:172.21.15.28:6834/4151254007 2024-02-21T23:09:14.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.158+0000 7fdcd01fa740 1 -- 172.21.15.28:0/338666729 >> 172.21.15.28:0/338666729 conn(0x55c417a90e40 msgr2=0x55c417acbe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:14.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.158+0000 7fdcd01fa740 1 -- 172.21.15.28:0/338666729 shutdown_connections 2024-02-21T23:09:14.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.158+0000 7fdcd01fa740 1 -- 172.21.15.28:0/338666729 wait complete. 2024-02-21T23:09:14.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.158+0000 7fdcd01fa740 20 client.8541 trim_cache size 0 max 16384 2024-02-21T23:09:14.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.176+0000 7fdcd01fa740 1 Processor -- start 2024-02-21T23:09:14.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.177+0000 7fdcd01fa740 1 -- start start 2024-02-21T23:09:14.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.177+0000 7fdcd01fa740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c417a92150 0x55c417a8c610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:14.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.177+0000 7fdcd01fa740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c417acd4d0 0x55c417ab12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:14.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.177+0000 7fdcd01fa740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c417ad1700 0x55c417aacba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:14.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.177+0000 7fdcd01fa740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc81f6cf0 con 0x55c417acd4d0 2024-02-21T23:09:14.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.177+0000 7fdcd01fa740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc8071c30 con 0x55c417a92150 2024-02-21T23:09:14.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.177+0000 7fdcd01fa740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc81f6910 con 0x55c417ad1700 2024-02-21T23:09:14.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.178+0000 7fdcc5e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c417acd4d0 0x55c417ab12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:14.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.178+0000 7fdcc5e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c417acd4d0 0x55c417ab12c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57318/0 (socket says 172.21.15.28:57318) 2024-02-21T23:09:14.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.178+0000 7fdcc5e00640 1 -- 172.21.15.28:0/2800981174 learned_addr learned my addr 172.21.15.28:0/2800981174 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:14.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.178+0000 7fdcc7200640 1 --2- 172.21.15.28:0/2800981174 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c417ad1700 0x55c417aacba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:14.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.178+0000 7fdcc6800640 1 --2- 172.21.15.28:0/2800981174 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c417a92150 0x55c417a8c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:14.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.178+0000 7fdcc5e00640 1 -- 172.21.15.28:0/2800981174 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c417ad1700 msgr2=0x55c417aacba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:14.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.178+0000 7fdcc5e00640 1 --2- 172.21.15.28:0/2800981174 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c417ad1700 0x55c417aacba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:14.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.178+0000 7fdcc5e00640 1 -- 172.21.15.28:0/2800981174 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c417a92150 msgr2=0x55c417a8c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:14.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.178+0000 7fdcc5e00640 1 --2- 172.21.15.28:0/2800981174 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c417a92150 0x55c417a8c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:14.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.178+0000 7fdcc5e00640 1 -- 172.21.15.28:0/2800981174 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c4177d46b0 con 0x55c417acd4d0 2024-02-21T23:09:14.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.178+0000 7fdcc6800640 1 --2- 172.21.15.28:0/2800981174 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c417a92150 0x55c417a8c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:09:14.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.179+0000 7fdcc5e00640 1 --2- 172.21.15.28:0/2800981174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c417acd4d0 0x55c417ab12c0 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7fdcb400c8d0 tx=0x7fdcb4012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=812325ab5071b46f server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:14.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.179+0000 7fdcb2600640 1 -- 172.21.15.28:0/2800981174 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcb401c040 con 0x55c417acd4d0 2024-02-21T23:09:14.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.179+0000 7fdcb2600640 1 -- 172.21.15.28:0/2800981174 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcb400ce90 con 0x55c417acd4d0 2024-02-21T23:09:14.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.179+0000 7fdcb2600640 1 -- 172.21.15.28:0/2800981174 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcb40252e0 con 0x55c417acd4d0 2024-02-21T23:09:14.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.179+0000 7fdcd01fa740 1 -- 172.21.15.28:0/2800981174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c417acd4d0 msgr2=0x55c417ab12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:14.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.179+0000 7fdcd01fa740 1 --2- 172.21.15.28:0/2800981174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c417acd4d0 0x55c417ab12c0 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7fdcb400c8d0 tx=0x7fdcb4012000 comp rx=0 tx=0).stop 2024-02-21T23:09:14.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.179+0000 7fdcd01fa740 1 -- 172.21.15.28:0/2800981174 shutdown_connections 2024-02-21T23:09:14.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.179+0000 7fdcd01fa740 1 --2- 172.21.15.28:0/2800981174 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c417ad1700 0x55c417aacba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:14.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.179+0000 7fdcd01fa740 1 --2- 172.21.15.28:0/2800981174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c417acd4d0 0x55c417ab12c0 unknown :-1 s=CLOSED pgs=1557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:14.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.179+0000 7fdcd01fa740 1 --2- 172.21.15.28:0/2800981174 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c417a92150 0x55c417a8c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:14.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.179+0000 7fdcd01fa740 1 -- 172.21.15.28:0/2800981174 >> 172.21.15.28:0/2800981174 conn(0x55c417a90e40 msgr2=0x55c417ab2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:14.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.180+0000 7fdcd01fa740 1 -- 172.21.15.28:0/2800981174 shutdown_connections 2024-02-21T23:09:14.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.180+0000 7fdcd01fa740 1 -- 172.21.15.28:0/2800981174 wait complete. 2024-02-21T23:09:14.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.180+0000 7fdcd01fa740 1 -- start start 2024-02-21T23:09:14.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.180+0000 7fdcd01fa740 1 Processor -- start 2024-02-21T23:09:14.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.181+0000 7fdcd01fa740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c417a92150 0x55c417ae1a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:14.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.181+0000 7fdcd01fa740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c417acd4d0 0x55c417b3cb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:14.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.181+0000 7fdcd01fa740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c417ad1700 0x55c417b39340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:14.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.181+0000 7fdcd01fa740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c417ac7780 con 0x55c417acd4d0 2024-02-21T23:09:14.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.181+0000 7fdcd01fa740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c417add490 con 0x55c417ad1700 2024-02-21T23:09:14.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.181+0000 7fdcd01fa740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c417aad730 con 0x55c417a92150 2024-02-21T23:09:14.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.181+0000 7fdcd01fa740 10 client.0 _ll_register_callbacks cb 0x7fdcb8054fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:09:14.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.182+0000 7fdcc5e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c417acd4d0 0x55c417b3cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:14.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.182+0000 7fdcc5e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c417acd4d0 0x55c417b3cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57320/0 (socket says 172.21.15.28:57320) 2024-02-21T23:09:14.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.182+0000 7fdcc5e00640 1 -- 172.21.15.28:0/2905131493 learned_addr learned my addr 172.21.15.28:0/2905131493 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:14.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.182+0000 7fdcc6800640 1 --2- 172.21.15.28:0/2905131493 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c417a92150 0x55c417ae1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:14.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.182+0000 7fdcc5e00640 1 -- 172.21.15.28:0/2905131493 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c417a92150 msgr2=0x55c417ae1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:14.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.182+0000 7fdcc5e00640 1 --2- 172.21.15.28:0/2905131493 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c417a92150 0x55c417ae1a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:14.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.182+0000 7fdcc5e00640 1 -- 172.21.15.28:0/2905131493 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c417ad1700 msgr2=0x55c417b39340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:14.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.182+0000 7fdcc5e00640 1 --2- 172.21.15.28:0/2905131493 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c417ad1700 0x55c417b39340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:14.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.182+0000 7fdcc5e00640 1 -- 172.21.15.28:0/2905131493 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c417aca990 con 0x55c417acd4d0 2024-02-21T23:09:14.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.182+0000 7fdcc5e00640 1 --2- 172.21.15.28:0/2905131493 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c417acd4d0 0x55c417b3cb80 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x55c417a8d510 tx=0x7fdcb4004680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25e9a3f0846e1f35 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:14.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.182+0000 7fdca7400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:09:14.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.183+0000 7fdca7400640 1 -- 172.21.15.28:0/2905131493 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcb4034020 con 0x55c417acd4d0 2024-02-21T23:09:14.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.183+0000 7fdca7400640 1 -- 172.21.15.28:0/2905131493 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(136..136 src has 1..136) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fdcb4004040 con 0x55c417acd4d0 2024-02-21T23:09:14.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.183+0000 7fdcd01fa740 1 -- 172.21.15.28:0/2905131493 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdcb400ce90 con 0x55c417acd4d0 2024-02-21T23:09:14.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.183+0000 7fdca7400640 1 -- 172.21.15.28:0/2905131493 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=139) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdcb4004240 con 0x55c417acd4d0 2024-02-21T23:09:14.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.183+0000 7fdcd01fa740 10 client.8547 fetch_fsmap learned FSMap version 139 2024-02-21T23:09:14.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.183+0000 7fdcd01fa740 1 -- 172.21.15.28:0/2905131493 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=139,monmap=2+}) v3 -- 0x55c417aca450 con 0x55c417acd4d0 2024-02-21T23:09:14.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.183+0000 7fdca7400640 1 -- 172.21.15.28:0/2905131493 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcb400ce90 con 0x55c417acd4d0 2024-02-21T23:09:14.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.183+0000 7fdca7400640 1 -- 172.21.15.28:0/2905131493 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 139) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fdcb4041020 con 0x55c417acd4d0 2024-02-21T23:09:14.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.183+0000 7fdcd01fa740 10 client.8547 fetch_fsmap finished waiting for FSMap version 139 2024-02-21T23:09:14.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.183+0000 7fdcd01fa740 10 client.8547 resolve_mds: resolved name 'a' to daemon mds.a 2024-02-21T23:09:14.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.183+0000 7fdcd01fa740 20 client.8547 populate_metadata read hostname 'smithi028' 2024-02-21T23:09:14.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.183+0000 7fdcd01fa740 1 --2- 172.21.15.28:0/2905131493 >> [v2:172.21.15.28:6834/4151254007,v1:172.21.15.28:6835/4151254007] conn(0x55c417b3f740 0x55c417b3fb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:14.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.183+0000 7fdcd01fa740 4 client.8547 mds_command: new command op to 8376 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-21T23:09:14.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.183+0000 7fdcd01fa740 1 -- 172.21.15.28:0/2905131493 --> [v2:172.21.15.28:6834/4151254007,v1:172.21.15.28:6835/4151254007] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55c4179fcd00 con 0x55c417b3f740 2024-02-21T23:09:14.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.184+0000 7fdcc6800640 1 --2- 172.21.15.28:0/2905131493 >> [v2:172.21.15.28:6834/4151254007,v1:172.21.15.28:6835/4151254007] conn(0x55c417b3f740 0x55c417b3fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:14.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.184+0000 7fdcc6800640 1 --2- 172.21.15.28:0/2905131493 >> [v2:172.21.15.28:6834/4151254007,v1:172.21.15.28:6835/4151254007] conn(0x55c417b3f740 0x55c417b3fb20 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ceba255682ed730b server_cookie=2765386c2f23c8e in_seq=0 out_seq=0 2024-02-21T23:09:14.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.184+0000 7fdca7400640 10 client.8547 ms_handle_connect on v2:172.21.15.28:6834/4151254007 2024-02-21T23:09:14.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.184+0000 7fdca7400640 1 -- 172.21.15.28:0/2905131493 <== mds.0 v2:172.21.15.28:6834/4151254007 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55c4179fcd00 con 0x55c417b3f740 2024-02-21T23:09:14.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.184+0000 7fdca7400640 10 client.8547 handle_command_reply: tid=0 2024-02-21T23:09:14.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.184+0000 7fdcd01fa740 1 client.8547 shutdown 2024-02-21T23:09:14.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.185+0000 7fdcd01fa740 1 -- 172.21.15.28:0/2905131493 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c417acd4d0 msgr2=0x55c417b3cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:14.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.185+0000 7fdcd01fa740 1 --2- 172.21.15.28:0/2905131493 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c417acd4d0 0x55c417b3cb80 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x55c417a8d510 tx=0x7fdcb4004680 comp rx=0 tx=0).stop 2024-02-21T23:09:14.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.185+0000 7fdcd01fa740 1 -- 172.21.15.28:0/2905131493 shutdown_connections 2024-02-21T23:09:14.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.185+0000 7fdcd01fa740 1 --2- 172.21.15.28:0/2905131493 >> [v2:172.21.15.28:6834/4151254007,v1:172.21.15.28:6835/4151254007] conn(0x55c417b3f740 0x55c417b3fb20 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:14.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.185+0000 7fdcd01fa740 1 --2- 172.21.15.28:0/2905131493 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c417ad1700 0x55c417b39340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:14.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.185+0000 7fdcd01fa740 1 --2- 172.21.15.28:0/2905131493 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c417acd4d0 0x55c417b3cb80 unknown :-1 s=CLOSED pgs=1558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:14.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.185+0000 7fdcd01fa740 1 --2- 172.21.15.28:0/2905131493 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c417a92150 0x55c417ae1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:14.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.185+0000 7fdca7400640 0 client.8547 ms_handle_reset on v2:172.21.15.28:6834/4151254007 2024-02-21T23:09:14.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.185+0000 7fdcd01fa740 1 -- 172.21.15.28:0/2905131493 >> 172.21.15.28:0/2905131493 conn(0x55c417a90e40 msgr2=0x55c417a8c600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:14.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.185+0000 7fdcd01fa740 1 -- 172.21.15.28:0/2905131493 shutdown_connections 2024-02-21T23:09:14.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.185+0000 7fdcd01fa740 1 -- 172.21.15.28:0/2905131493 wait complete. 2024-02-21T23:09:14.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.185+0000 7fdcd01fa740 20 client.8547 trim_cache size 0 max 16384 2024-02-21T23:09:14.178 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-21T23:09:14.178 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac", 2024-02-21T23:09:14.178 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-02-21T23:09:14.178 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 8376, 2024-02-21T23:09:14.178 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-02-21T23:09:14.178 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-02-21T23:09:14.178 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-02-21T23:09:14.178 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 9.764303172, 2024-02-21T23:09:14.178 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 139, 2024-02-21T23:09:14.178 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 136, 2024-02-21T23:09:14.178 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 136, 2024-02-21T23:09:14.179 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 16.785661524999998 2024-02-21T23:09:14.179 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-21T23:09:14.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.186+0000 7fdccd400640 1 -- 172.21.15.28:0/850954437 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdca806b4d0 msgr2=0x7fdca806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:14.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.186+0000 7fdccd400640 1 --2- 172.21.15.28:0/850954437 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdca806b4d0 0x7fdca806d990 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fdcc8068420 tx=0x7fdcb4008040 comp rx=0 tx=0).stop 2024-02-21T23:09:14.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.186+0000 7fdccd400640 1 -- 172.21.15.28:0/850954437 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcc80c2df0 msgr2=0x7fdcc80aa880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:14.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.186+0000 7fdccd400640 1 --2- 172.21.15.28:0/850954437 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcc80c2df0 0x7fdcc80aa880 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7fdcb800c7f0 tx=0x7fdcb800ccb0 comp rx=0 tx=0).stop 2024-02-21T23:09:14.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.186+0000 7fdccd400640 1 -- 172.21.15.28:0/850954437 shutdown_connections 2024-02-21T23:09:14.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.186+0000 7fdccd400640 1 --2- 172.21.15.28:0/850954437 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdca806b4d0 0x7fdca806d990 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:14.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.186+0000 7fdccd400640 1 --2- 172.21.15.28:0/850954437 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdcc80c5f20 0x7fdcc80bfc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:14.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.186+0000 7fdccd400640 1 --2- 172.21.15.28:0/850954437 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdcc8153220 0x7fdcc81ee990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:14.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.186+0000 7fdccd400640 1 --2- 172.21.15.28:0/850954437 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcc80c2df0 0x7fdcc80aa880 unknown :-1 s=CLOSED pgs=1554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:14.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.186+0000 7fdccd400640 1 -- 172.21.15.28:0/850954437 >> 172.21.15.28:0/850954437 conn(0x7fdcc806b820 msgr2=0x7fdcc80c6790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:14.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.186+0000 7fdccd400640 1 -- 172.21.15.28:0/850954437 shutdown_connections 2024-02-21T23:09:14.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:14.186+0000 7fdccd400640 1 -- 172.21.15.28:0/850954437 wait complete. 2024-02-21T23:09:14.189 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 10, "epoch": 139, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 10, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-02-21T23:09:03.869294+0000", "damaged": [], "data_pools": [ 22 ], "enabled": true, "epoch": 139, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_8376": { "addr": "172.21.15.28:6835/4151254007", "addrs": { "addrvec": [ { "addr": "172.21.15.28:6834", "nonce": 4151254007, "type": "v2" }, { "addr": "172.21.15.28:6835", "nonce": 4151254007, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8376, "incarnation": 135, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 21, "modified": "2024-02-21T23:09:07.201932+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8376 } } } ], "standbys": [ { "addr": "172.21.15.81:6836/4241906475", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6834", "nonce": 4241906475, "type": "v2" }, { "addr": "172.21.15.81:6836", "nonce": 4241906475, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 133, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6140, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.81:6837/2095908433", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6835", "nonce": 2095908433, "type": "v2" }, { "addr": "172.21.15.81:6837", "nonce": 2095908433, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 133, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6143, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.28:6833/3934548735", "addrs": { "addrvec": [ { "addr": "172.21.15.28:6832", "nonce": 3934548735, "type": "v2" }, { "addr": "172.21.15.28:6833", "nonce": 3934548735, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 133, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8370, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-02-21T23:09:14.190 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': 10, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': 10, 'name': 'd'}] 2024-02-21T23:09:14.190 INFO:tasks.cephfs.test_failover:current = [{'gid': 8376, 'name': 'a', 'rank': 0, 'incarnation': 135, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/4151254007', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 4151254007}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 4151254007}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 6140, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.81:6836/4241906475', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6834', 'nonce': 4241906475}, {'type': 'v1', 'addr': '172.21.15.81:6836', 'nonce': 4241906475}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 133}, {'gid': 8370, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.28:6833/3934548735', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 3934548735}, {'type': 'v1', 'addr': '172.21.15.28:6833', 'nonce': 3934548735}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 133}, {'gid': 6143, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.81:6837/2095908433', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6835', 'nonce': 2095908433}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 2095908433}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 133}] 2024-02-21T23:09:14.191 DEBUG:tasks.cephfs.test_failover:10 != -1 2024-02-21T23:09:14.191 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-02-21T23:09:15.545 INFO:tasks.ceph.mds.a.smithi028.stderr: -319> 2024-02-21T23:09:08.352+0000 7f8203000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-21T23:09:15.599 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-02-21T23:09:19.192 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:09:19.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.436+0000 7f7a22c00640 1 Processor -- start 2024-02-21T23:09:19.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.436+0000 7f7a22c00640 1 -- start start 2024-02-21T23:09:19.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.437+0000 7f7a22c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7a1c0c11f0 0x7f7a1c153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:19.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.437+0000 7f7a22c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f7a1c0c15d0 0x7f7a1c06dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:19.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.437+0000 7f7a22c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7a1c156350 0x7f7a1c1567a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:19.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.437+0000 7f7a22c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a1c0a5ec0 con 0x7f7a1c156350 2024-02-21T23:09:19.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.437+0000 7f7a22c00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a1c0c1a30 con 0x7f7a1c0c15d0 2024-02-21T23:09:19.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.437+0000 7f7a22c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1c152e70 con 0x7f7a1c0c11f0 2024-02-21T23:09:19.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.437+0000 7f7a20c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7a1c156350 0x7f7a1c1567a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:19.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.437+0000 7f7a20c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7a1c156350 0x7f7a1c1567a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58776/0 (socket says 172.21.15.28:58776) 2024-02-21T23:09:19.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.437+0000 7f7a20c00640 1 -- 172.21.15.28:0/3692082629 learned_addr learned my addr 172.21.15.28:0/3692082629 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:19.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.438+0000 7f7a1be00640 1 --2- 172.21.15.28:0/3692082629 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7a1c0c11f0 0x7f7a1c153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:19.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.438+0000 7f7a1b400640 1 --2- 172.21.15.28:0/3692082629 >> v2:172.21.15.81:3300/0 conn(0x7f7a1c0c15d0 0x7f7a1c06dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:19.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.438+0000 7f7a20c00640 1 -- 172.21.15.28:0/3692082629 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7a1c0c11f0 msgr2=0x7f7a1c153950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:19.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.438+0000 7f7a20c00640 1 --2- 172.21.15.28:0/3692082629 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7a1c0c11f0 0x7f7a1c153950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:19.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.438+0000 7f7a20c00640 1 -- 172.21.15.28:0/3692082629 >> v2:172.21.15.81:3300/0 conn(0x7f7a1c0c15d0 msgr2=0x7f7a1c06dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:19.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.438+0000 7f7a20c00640 1 --2- 172.21.15.28:0/3692082629 >> v2:172.21.15.81:3300/0 conn(0x7f7a1c0c15d0 0x7f7a1c06dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:19.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.438+0000 7f7a20c00640 1 -- 172.21.15.28:0/3692082629 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a1c153150 con 0x7f7a1c156350 2024-02-21T23:09:19.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.439+0000 7f7a20c00640 1 --2- 172.21.15.28:0/3692082629 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a1c156350 0x7f7a1c1567a0 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c06fb00 tx=0x7f7a1402bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bfc794a490f1df server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:19.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.439+0000 7f7a1aa00640 1 -- 172.21.15.28:0/3692082629 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a1402c950 con 0x7f7a1c156350 2024-02-21T23:09:19.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.439+0000 7f7a1aa00640 1 -- 172.21.15.28:0/3692082629 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a1402caf0 con 0x7f7a1c156350 2024-02-21T23:09:19.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.439+0000 7f7a1aa00640 1 -- 172.21.15.28:0/3692082629 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a140343f0 con 0x7f7a1c156350 2024-02-21T23:09:19.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.440+0000 7f7a22c00640 1 -- 172.21.15.28:0/3692082629 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a1c156350 msgr2=0x7f7a1c1567a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:19.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.440+0000 7f7a22c00640 1 --2- 172.21.15.28:0/3692082629 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a1c156350 0x7f7a1c1567a0 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c06fb00 tx=0x7f7a1402bdb0 comp rx=0 tx=0).stop 2024-02-21T23:09:19.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.440+0000 7f7a22c00640 1 -- 172.21.15.28:0/3692082629 shutdown_connections 2024-02-21T23:09:19.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.440+0000 7f7a22c00640 1 --2- 172.21.15.28:0/3692082629 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a1c156350 0x7f7a1c1567a0 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:19.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.440+0000 7f7a22c00640 1 --2- 172.21.15.28:0/3692082629 >> v2:172.21.15.81:3300/0 conn(0x7f7a1c0c15d0 0x7f7a1c06dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:19.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.440+0000 7f7a22c00640 1 --2- 172.21.15.28:0/3692082629 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7a1c0c11f0 0x7f7a1c153950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:19.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.440+0000 7f7a22c00640 1 -- 172.21.15.28:0/3692082629 >> 172.21.15.28:0/3692082629 conn(0x7f7a1c0c67f0 msgr2=0x7f7a1c0c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:19.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.440+0000 7f7a22c00640 1 -- 172.21.15.28:0/3692082629 shutdown_connections 2024-02-21T23:09:19.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.440+0000 7f7a22c00640 1 -- 172.21.15.28:0/3692082629 wait complete. 2024-02-21T23:09:19.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.442+0000 7f7a22c00640 1 Processor -- start 2024-02-21T23:09:19.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.442+0000 7f7a22c00640 1 -- start start 2024-02-21T23:09:19.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.443+0000 7f7a22c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a1c0c11f0 0x7f7a1c1496d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:19.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.443+0000 7f7a22c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a1c0c15d0 0x7f7a1c142a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:19.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.443+0000 7f7a22c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7a1c156350 0x7f7a1c142fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:19.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.443+0000 7f7a22c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1c0c4c00 con 0x7f7a1c0c11f0 2024-02-21T23:09:19.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.443+0000 7f7a22c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1c14d1c0 con 0x7f7a1c0c15d0 2024-02-21T23:09:19.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.443+0000 7f7a22c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1c0a6110 con 0x7f7a1c156350 2024-02-21T23:09:19.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.443+0000 7f7a1be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a1c0c11f0 0x7f7a1c1496d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:19.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.443+0000 7f7a1be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a1c0c11f0 0x7f7a1c1496d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58788/0 (socket says 172.21.15.28:58788) 2024-02-21T23:09:19.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.443+0000 7f7a1be00640 1 -- 172.21.15.28:0/2506098546 learned_addr learned my addr 172.21.15.28:0/2506098546 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:19.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.443+0000 7f7a1b400640 1 --2- 172.21.15.28:0/2506098546 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a1c0c15d0 0x7f7a1c142a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:19.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.444+0000 7f7a20c00640 1 --2- 172.21.15.28:0/2506098546 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7a1c156350 0x7f7a1c142fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:19.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.444+0000 7f7a1be00640 1 -- 172.21.15.28:0/2506098546 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7a1c156350 msgr2=0x7f7a1c142fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:19.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.444+0000 7f7a1be00640 1 --2- 172.21.15.28:0/2506098546 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7a1c156350 0x7f7a1c142fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:19.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.444+0000 7f7a1be00640 1 -- 172.21.15.28:0/2506098546 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a1c0c15d0 msgr2=0x7f7a1c142a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:19.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.444+0000 7f7a1be00640 1 --2- 172.21.15.28:0/2506098546 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a1c0c15d0 0x7f7a1c142a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:19.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.444+0000 7f7a1be00640 1 -- 172.21.15.28:0/2506098546 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a1c1eafc0 con 0x7f7a1c0c11f0 2024-02-21T23:09:19.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.444+0000 7f7a1be00640 1 --2- 172.21.15.28:0/2506098546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a1c0c11f0 0x7f7a1c1496d0 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c00d800 tx=0x7f7a0c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:19.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.444+0000 7f7a13e00640 1 -- 172.21.15.28:0/2506098546 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a0c018030 con 0x7f7a1c0c11f0 2024-02-21T23:09:19.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.444+0000 7f7a13e00640 1 -- 172.21.15.28:0/2506098546 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a0c0096a0 con 0x7f7a1c0c11f0 2024-02-21T23:09:19.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.444+0000 7f7a22c00640 1 -- 172.21.15.28:0/2506098546 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a1c1eb280 con 0x7f7a1c0c11f0 2024-02-21T23:09:19.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.444+0000 7f7a13e00640 1 -- 172.21.15.28:0/2506098546 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a0c009870 con 0x7f7a1c0c11f0 2024-02-21T23:09:19.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.445+0000 7f7a22c00640 1 -- 172.21.15.28:0/2506098546 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a1c1eb840 con 0x7f7a1c0c11f0 2024-02-21T23:09:19.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.446+0000 7f7a22c00640 1 -- 172.21.15.28:0/2506098546 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a1c067f30 con 0x7f7a1c0c11f0 2024-02-21T23:09:19.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.447+0000 7f7a13e00640 1 -- 172.21.15.28:0/2506098546 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7a0c01b070 con 0x7f7a1c0c11f0 2024-02-21T23:09:19.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.448+0000 7f7a13e00640 1 --2- 172.21.15.28:0/2506098546 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f79fc06b4d0 0x7f79fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:19.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.448+0000 7f7a13e00640 1 -- 172.21.15.28:0/2506098546 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f7a0c08a6a0 con 0x7f7a1c0c11f0 2024-02-21T23:09:19.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.449+0000 7f7a1b400640 1 --2- 172.21.15.28:0/2506098546 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f79fc06b4d0 0x7f79fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:19.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.450+0000 7f7a1b400640 1 --2- 172.21.15.28:0/2506098546 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f79fc06b4d0 0x7f79fc06d990 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f7a08006fd0 tx=0x7f7a08008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:19.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.453+0000 7f7a13e00640 1 -- 172.21.15.28:0/2506098546 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7a0c059330 con 0x7f7a1c0c11f0 2024-02-21T23:09:19.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.693+0000 7f7a22c00640 1 -- 172.21.15.28:0/2506098546 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7a1c0c0960 con 0x7f7a1c0c11f0 2024-02-21T23:09:19.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.694+0000 7f7a13e00640 1 -- 172.21.15.28:0/2506098546 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) v1 ==== 95+0+4951 (secure 0 0 0) 0x7f7a0c058c30 con 0x7f7a1c0c11f0 2024-02-21T23:09:19.682 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:09:19.682 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":144,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6143,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":7,"addr":"172.21.15.81:6837/2095908433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":2095908433},{"type":"v1","addr":"172.21.15.81:6837","nonce":2095908433}]},"join_fscid":10,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":140},{"gid":8370,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/3934548735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3934548735},{"type":"v1","addr":"172.21.15.28:6833","nonce":3934548735}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":133},{"gid":8556,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2100132822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2100132822},{"type":"v1","addr":"172.21.15.28:6835","nonce":2100132822}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":142}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:09:03.869294+0000","modified":"2024-02-21T23:09:17.546876+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":137,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6140},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6140":{"gid":6140,"name":"b","rank":0,"incarnation":141,"state":"up:active","state_seq":10,"addr":"172.21.15.81:6836/4241906475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":4241906475},{"type":"v1","addr":"172.21.15.81:6836","nonce":4241906475}]},"join_fscid":10,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-02-21T23:09:19.682 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 144 2024-02-21T23:09:19.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.696+0000 7f7a22c00640 1 -- 172.21.15.28:0/2506098546 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f79fc06b4d0 msgr2=0x7f79fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:19.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.696+0000 7f7a22c00640 1 --2- 172.21.15.28:0/2506098546 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f79fc06b4d0 0x7f79fc06d990 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f7a08006fd0 tx=0x7f7a08008040 comp rx=0 tx=0).stop 2024-02-21T23:09:19.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.696+0000 7f7a22c00640 1 -- 172.21.15.28:0/2506098546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a1c0c11f0 msgr2=0x7f7a1c1496d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:19.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.696+0000 7f7a22c00640 1 --2- 172.21.15.28:0/2506098546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a1c0c11f0 0x7f7a1c1496d0 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c00d800 tx=0x7f7a0c00dcc0 comp rx=0 tx=0).stop 2024-02-21T23:09:19.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.696+0000 7f7a22c00640 1 -- 172.21.15.28:0/2506098546 shutdown_connections 2024-02-21T23:09:19.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.696+0000 7f7a22c00640 1 --2- 172.21.15.28:0/2506098546 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f79fc06b4d0 0x7f79fc06d990 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:19.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.696+0000 7f7a22c00640 1 --2- 172.21.15.28:0/2506098546 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7a1c156350 0x7f7a1c142fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:19.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.696+0000 7f7a22c00640 1 --2- 172.21.15.28:0/2506098546 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a1c0c15d0 0x7f7a1c142a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:19.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.696+0000 7f7a22c00640 1 --2- 172.21.15.28:0/2506098546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a1c0c11f0 0x7f7a1c1496d0 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:19.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.696+0000 7f7a22c00640 1 -- 172.21.15.28:0/2506098546 >> 172.21.15.28:0/2506098546 conn(0x7f7a1c0c67f0 msgr2=0x7f7a1c0bfc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:19.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.696+0000 7f7a22c00640 1 -- 172.21.15.28:0/2506098546 shutdown_connections 2024-02-21T23:09:19.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:19.696+0000 7f7a22c00640 1 -- 172.21.15.28:0/2506098546 wait complete. 2024-02-21T23:09:20.107 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 144, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:09:03.869294+0000', 'modified': '2024-02-21T23:09:17.546876+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 137, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6140}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6140': {'gid': 6140, 'name': 'b', 'rank': 0, 'incarnation': 141, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.81:6836/4241906475', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6834', 'nonce': 4241906475}, {'type': 'v1', 'addr': '172.21.15.81:6836', 'nonce': 4241906475}]}, 'join_fscid': 10, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:09:20.107 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-21T23:09:20.108 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-02-21T23:09:20.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.351+0000 7f1d82c00640 1 Processor -- start 2024-02-21T23:09:20.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.351+0000 7f1d82c00640 1 -- start start 2024-02-21T23:09:20.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.352+0000 7f1d82c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1d7c156c80 0x7f1d7c06e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:20.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.352+0000 7f1d82c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1d7c072e60 0x7f1d7c06ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:20.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.352+0000 7f1d82c00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d7c159fd0 con 0x7f1d7c0c11f0 2024-02-21T23:09:20.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.352+0000 7f1d82c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d7c0c5b60 con 0x7f1d7c072e60 2024-02-21T23:09:20.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.352+0000 7f1d82c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7c0c5db0 con 0x7f1d7c156c80 2024-02-21T23:09:20.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.353+0000 7f1d7be00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1d7c156c80 0x7f1d7c06e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:20.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.353+0000 7f1d7b400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1d7c072e60 0x7f1d7c06ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:20.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.353+0000 7f1d7b400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1d7c072e60 0x7f1d7c06ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58802/0 (socket says 172.21.15.28:58802) 2024-02-21T23:09:20.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.353+0000 7f1d7be00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1d7c156c80 0x7f1d7c06e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3301/0 says I am v2:172.21.15.28:47346/0 (socket says 172.21.15.28:47346) 2024-02-21T23:09:20.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.353+0000 7f1d7b400640 1 -- 172.21.15.28:0/993859512 learned_addr learned my addr 172.21.15.28:0/993859512 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:20.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.354+0000 7f1d7b400640 1 -- 172.21.15.28:0/993859512 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1d7c156c80 msgr2=0x7f1d7c06e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:20.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.354+0000 7f1d7b400640 1 --2- 172.21.15.28:0/993859512 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1d7c156c80 0x7f1d7c06e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:20.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.354+0000 7f1d7b400640 1 -- 172.21.15.28:0/993859512 >> v1:172.21.15.81:6789/0 conn(0x7f1d7c0c11f0 legacy=0x7f1d7c0c4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:20.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.354+0000 7f1d7b400640 1 -- 172.21.15.28:0/993859512 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d7c15a290 con 0x7f1d7c072e60 2024-02-21T23:09:20.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.355+0000 7f1d7b400640 1 --2- 172.21.15.28:0/993859512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d7c072e60 0x7f1d7c06ed80 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f1d68009870 tx=0x7f1d6802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8f2dd4e1d759d09 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:20.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.355+0000 7f1d7aa00640 1 -- 172.21.15.28:0/993859512 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d68034070 con 0x7f1d7c072e60 2024-02-21T23:09:20.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.355+0000 7f1d7aa00640 1 -- 172.21.15.28:0/993859512 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d68030840 con 0x7f1d7c072e60 2024-02-21T23:09:20.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.355+0000 7f1d7aa00640 1 -- 172.21.15.28:0/993859512 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d68030c10 con 0x7f1d7c072e60 2024-02-21T23:09:20.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.356+0000 7f1d82c00640 1 -- 172.21.15.28:0/993859512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d7c072e60 msgr2=0x7f1d7c06ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:20.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.356+0000 7f1d82c00640 1 --2- 172.21.15.28:0/993859512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d7c072e60 0x7f1d7c06ed80 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f1d68009870 tx=0x7f1d6802b980 comp rx=0 tx=0).stop 2024-02-21T23:09:20.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.356+0000 7f1d82c00640 1 -- 172.21.15.28:0/993859512 shutdown_connections 2024-02-21T23:09:20.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.356+0000 7f1d82c00640 1 --2- 172.21.15.28:0/993859512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d7c072e60 0x7f1d7c06ed80 unknown :-1 s=CLOSED pgs=1564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:20.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.356+0000 7f1d82c00640 1 --2- 172.21.15.28:0/993859512 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1d7c156c80 0x7f1d7c06e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:20.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.356+0000 7f1d82c00640 1 -- 172.21.15.28:0/993859512 >> 172.21.15.28:0/993859512 conn(0x7f1d7c06b820 msgr2=0x7f1d7c0c67f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:20.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.356+0000 7f1d82c00640 1 -- 172.21.15.28:0/993859512 shutdown_connections 2024-02-21T23:09:20.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.356+0000 7f1d82c00640 1 -- 172.21.15.28:0/993859512 wait complete. 2024-02-21T23:09:20.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.358+0000 7f1d82c00640 1 Processor -- start 2024-02-21T23:09:20.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.358+0000 7f1d82c00640 1 -- start start 2024-02-21T23:09:20.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.359+0000 7f1d82c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1d7c0c11f0 0x7f1d7c155970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:20.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.359+0000 7f1d82c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d7c156c80 0x7f1d7c155eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:20.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.359+0000 7f1d82c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d7c0ab4a0 0x7f1d7c1f2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:20.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.359+0000 7f1d82c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7c0c51e0 con 0x7f1d7c156c80 2024-02-21T23:09:20.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.359+0000 7f1d82c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7c0c0e10 con 0x7f1d7c0ab4a0 2024-02-21T23:09:20.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.359+0000 7f1d82c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7c0c5360 con 0x7f1d7c0c11f0 2024-02-21T23:09:20.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.360+0000 7f1d7b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d7c156c80 0x7f1d7c155eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:20.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.360+0000 7f1d7b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d7c156c80 0x7f1d7c155eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58808/0 (socket says 172.21.15.28:58808) 2024-02-21T23:09:20.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.360+0000 7f1d7b400640 1 -- 172.21.15.28:0/1553934277 learned_addr learned my addr 172.21.15.28:0/1553934277 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:20.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.360+0000 7f1d7be00640 1 --2- 172.21.15.28:0/1553934277 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1d7c0c11f0 0x7f1d7c155970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:20.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.360+0000 7f1d80c00640 1 --2- 172.21.15.28:0/1553934277 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d7c0ab4a0 0x7f1d7c1f2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:20.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.360+0000 7f1d7b400640 1 -- 172.21.15.28:0/1553934277 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1d7c0c11f0 msgr2=0x7f1d7c155970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:20.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.360+0000 7f1d7b400640 1 --2- 172.21.15.28:0/1553934277 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1d7c0c11f0 0x7f1d7c155970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:20.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.360+0000 7f1d7b400640 1 -- 172.21.15.28:0/1553934277 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d7c0ab4a0 msgr2=0x7f1d7c1f2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:20.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.360+0000 7f1d7b400640 1 --2- 172.21.15.28:0/1553934277 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d7c0ab4a0 0x7f1d7c1f2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:20.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.360+0000 7f1d7b400640 1 -- 172.21.15.28:0/1553934277 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d7c0a8b70 con 0x7f1d7c156c80 2024-02-21T23:09:20.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.361+0000 7f1d7b400640 1 --2- 172.21.15.28:0/1553934277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d7c156c80 0x7f1d7c155eb0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f1d68009870 tx=0x7f1d68004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:20.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.361+0000 7f1d73e00640 1 -- 172.21.15.28:0/1553934277 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d68048070 con 0x7f1d7c156c80 2024-02-21T23:09:20.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.361+0000 7f1d73e00640 1 -- 172.21.15.28:0/1553934277 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d68030640 con 0x7f1d7c156c80 2024-02-21T23:09:20.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.361+0000 7f1d73e00640 1 -- 172.21.15.28:0/1553934277 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d68034040 con 0x7f1d7c156c80 2024-02-21T23:09:20.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.361+0000 7f1d82c00640 1 -- 172.21.15.28:0/1553934277 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d7c15d390 con 0x7f1d7c156c80 2024-02-21T23:09:20.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.361+0000 7f1d82c00640 1 -- 172.21.15.28:0/1553934277 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d7c15d950 con 0x7f1d7c156c80 2024-02-21T23:09:20.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.363+0000 7f1d73e00640 1 -- 172.21.15.28:0/1553934277 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1d68037070 con 0x7f1d7c156c80 2024-02-21T23:09:20.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.364+0000 7f1d73e00640 1 --2- 172.21.15.28:0/1553934277 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1d5c06b500 0x7f1d5c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:20.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.364+0000 7f1d73e00640 1 -- 172.21.15.28:0/1553934277 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f1d680abfa0 con 0x7f1d7c156c80 2024-02-21T23:09:20.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.365+0000 7f1d85a64740 1 Processor -- start 2024-02-21T23:09:20.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.365+0000 7f1d7be00640 1 --2- 172.21.15.28:0/1553934277 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1d5c06b500 0x7f1d5c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:20.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.365+0000 7f1d85a64740 1 -- start start 2024-02-21T23:09:20.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.366+0000 7f1d85a64740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c84bc34e40 0x55c84bc35220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:20.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.366+0000 7f1d85a64740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c84bc36150 0x55c84bc70f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:20.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.366+0000 7f1d85a64740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84bc714d0 0x55c84bc6f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:20.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.366+0000 7f1d85a64740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c84bb99470 con 0x55c84bc714d0 2024-02-21T23:09:20.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.366+0000 7f1d85a64740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c84bb42af0 con 0x55c84bc34e40 2024-02-21T23:09:20.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.366+0000 7f1d85a64740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c84bb95510 con 0x55c84bc36150 2024-02-21T23:09:20.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.366+0000 7f1d7be00640 1 --2- 172.21.15.28:0/1553934277 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1d5c06b500 0x7f1d5c06d9c0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c009a30 tx=0x7f1d6c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:20.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.366+0000 7f1d80c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84bc714d0 0x55c84bc6f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:20.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.366+0000 7f1d80c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84bc714d0 0x55c84bc6f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58812/0 (socket says 172.21.15.28:58812) 2024-02-21T23:09:20.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.366+0000 7f1d80c00640 1 -- 172.21.15.28:0/1866930634 learned_addr learned my addr 172.21.15.28:0/1866930634 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:20.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.367+0000 7f1d7b400640 1 --2- 172.21.15.28:0/1866930634 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c84bc36150 0x55c84bc70f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:20.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.367+0000 7f1d7be00640 1 --2- 172.21.15.28:0/1866930634 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c84bc34e40 0x55c84bc35220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:20.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.367+0000 7f1d80c00640 1 -- 172.21.15.28:0/1866930634 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c84bc36150 msgr2=0x55c84bc70f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:20.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.367+0000 7f1d80c00640 1 --2- 172.21.15.28:0/1866930634 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c84bc36150 0x55c84bc70f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:20.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.367+0000 7f1d80c00640 1 -- 172.21.15.28:0/1866930634 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c84bc34e40 msgr2=0x55c84bc35220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:20.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.367+0000 7f1d80c00640 1 --2- 172.21.15.28:0/1866930634 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c84bc34e40 0x55c84bc35220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:20.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.367+0000 7f1d80c00640 1 -- 172.21.15.28:0/1866930634 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c84bb49d90 con 0x55c84bc714d0 2024-02-21T23:09:20.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.367+0000 7f1d7b400640 1 --2- 172.21.15.28:0/1866930634 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c84bc36150 0x55c84bc70f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:09:20.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.367+0000 7f1d7be00640 1 --2- 172.21.15.28:0/1866930634 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c84bc34e40 0x55c84bc35220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:09:20.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.367+0000 7f1d80c00640 1 --2- 172.21.15.28:0/1866930634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84bc714d0 0x55c84bc6f540 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f1d74001c70 tx=0x7f1d74002130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b56ef0fc3786373 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:20.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.367+0000 7f1d67e00640 1 -- 172.21.15.28:0/1866930634 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d74019070 con 0x55c84bc714d0 2024-02-21T23:09:20.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.367+0000 7f1d67e00640 1 -- 172.21.15.28:0/1866930634 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d74003b80 con 0x55c84bc714d0 2024-02-21T23:09:20.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.367+0000 7f1d67e00640 1 -- 172.21.15.28:0/1866930634 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d7400e800 con 0x55c84bc714d0 2024-02-21T23:09:20.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.368+0000 7f1d85a64740 1 -- 172.21.15.28:0/1866930634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84bc714d0 msgr2=0x55c84bc6f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:20.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.368+0000 7f1d85a64740 1 --2- 172.21.15.28:0/1866930634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84bc714d0 0x55c84bc6f540 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f1d74001c70 tx=0x7f1d74002130 comp rx=0 tx=0).stop 2024-02-21T23:09:20.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.368+0000 7f1d85a64740 1 -- 172.21.15.28:0/1866930634 shutdown_connections 2024-02-21T23:09:20.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.368+0000 7f1d85a64740 1 --2- 172.21.15.28:0/1866930634 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c84bc36150 0x55c84bc70f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:20.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.368+0000 7f1d85a64740 1 --2- 172.21.15.28:0/1866930634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84bc714d0 0x55c84bc6f540 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:20.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.368+0000 7f1d85a64740 1 --2- 172.21.15.28:0/1866930634 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c84bc34e40 0x55c84bc35220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:20.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.368+0000 7f1d85a64740 1 -- 172.21.15.28:0/1866930634 >> 172.21.15.28:0/1866930634 conn(0x55c84bc31790 msgr2=0x55c84bc561a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:20.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.368+0000 7f1d85a64740 1 -- 172.21.15.28:0/1866930634 shutdown_connections 2024-02-21T23:09:20.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.368+0000 7f1d85a64740 1 -- 172.21.15.28:0/1866930634 wait complete. 2024-02-21T23:09:20.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.369+0000 7f1d85a64740 1 -- start start 2024-02-21T23:09:20.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.369+0000 7f1d85a64740 1 Processor -- start 2024-02-21T23:09:20.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.370+0000 7f1d85a64740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c84bc36150 0x55c84bc77760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:20.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.370+0000 7f1d85a64740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c84bc714d0 0x55c84bc84930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:20.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.370+0000 7f1d85a64740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84bc75e50 0x55c84bc50a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:20.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.370+0000 7f1d85a64740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c84bb38040 con 0x55c84bc75e50 2024-02-21T23:09:20.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.370+0000 7f1d85a64740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c84bc568b0 con 0x55c84bc36150 2024-02-21T23:09:20.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.370+0000 7f1d85a64740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c84bc6c800 con 0x55c84bc714d0 2024-02-21T23:09:20.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.370+0000 7f1d85a64740 10 client.0 _ll_register_callbacks cb 0x55c84bb55920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:09:20.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.370+0000 7f1d80c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84bc75e50 0x55c84bc50a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:20.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.370+0000 7f1d80c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84bc75e50 0x55c84bc50a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58814/0 (socket says 172.21.15.28:58814) 2024-02-21T23:09:20.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.370+0000 7f1d80c00640 1 -- 172.21.15.28:0/1424413954 learned_addr learned my addr 172.21.15.28:0/1424413954 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:20.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.371+0000 7f1d7b400640 1 --2- 172.21.15.28:0/1424413954 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c84bc714d0 0x55c84bc84930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:20.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.371+0000 7f1d7be00640 1 --2- 172.21.15.28:0/1424413954 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c84bc36150 0x55c84bc77760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:20.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.371+0000 7f1d80c00640 1 -- 172.21.15.28:0/1424413954 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c84bc714d0 msgr2=0x55c84bc84930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:20.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.371+0000 7f1d80c00640 1 --2- 172.21.15.28:0/1424413954 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c84bc714d0 0x55c84bc84930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:20.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.371+0000 7f1d80c00640 1 -- 172.21.15.28:0/1424413954 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c84bc36150 msgr2=0x55c84bc77760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:20.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.371+0000 7f1d80c00640 1 --2- 172.21.15.28:0/1424413954 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c84bc36150 0x55c84bc77760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:20.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.371+0000 7f1d80c00640 1 -- 172.21.15.28:0/1424413954 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c84bb57aa0 con 0x55c84bc75e50 2024-02-21T23:09:20.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.371+0000 7f1d80c00640 1 --2- 172.21.15.28:0/1424413954 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84bc75e50 0x55c84bc50a00 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f1d74013000 tx=0x7f1d74001250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb3f9cdc8dabf0ae server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:20.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.371+0000 7f1d64c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:09:20.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.371+0000 7f1d64c00640 1 -- 172.21.15.28:0/1424413954 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d74019040 con 0x55c84bc75e50 2024-02-21T23:09:20.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.371+0000 7f1d64c00640 1 -- 172.21.15.28:0/1424413954 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(137..137 src has 1..137) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f1d7400fbc0 con 0x55c84bc75e50 2024-02-21T23:09:20.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.371+0000 7f1d85a64740 1 -- 172.21.15.28:0/1424413954 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1d74003b80 con 0x55c84bc75e50 2024-02-21T23:09:20.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.372+0000 7f1d64c00640 1 -- 172.21.15.28:0/1424413954 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=144) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1d7400fdc0 con 0x55c84bc75e50 2024-02-21T23:09:20.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.372+0000 7f1d85a64740 10 client.8574 fetch_fsmap learned FSMap version 144 2024-02-21T23:09:20.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.372+0000 7f1d85a64740 1 -- 172.21.15.28:0/1424413954 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=144,monmap=2+}) v3 -- 0x55c84b8ae2d0 con 0x55c84bc75e50 2024-02-21T23:09:20.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.372+0000 7f1d64c00640 1 -- 172.21.15.28:0/1424413954 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d74003b80 con 0x55c84bc75e50 2024-02-21T23:09:20.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.372+0000 7f1d64c00640 1 -- 172.21.15.28:0/1424413954 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 144) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f1d74006b50 con 0x55c84bc75e50 2024-02-21T23:09:20.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.372+0000 7f1d85a64740 10 client.8574 fetch_fsmap finished waiting for FSMap version 144 2024-02-21T23:09:20.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.372+0000 7f1d85a64740 10 client.8574 resolve_mds: resolved name 'b' to daemon mds.b 2024-02-21T23:09:20.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.372+0000 7f1d85a64740 20 client.8574 populate_metadata read hostname 'smithi028' 2024-02-21T23:09:20.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.372+0000 7f1d85a64740 1 --2- 172.21.15.28:0/1424413954 >> [v2:172.21.15.81:6834/4241906475,v1:172.21.15.81:6836/4241906475] conn(0x55c84bc81140 0x55c84bc81520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:20.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.372+0000 7f1d85a64740 4 client.8574 mds_command: new command op to 6140 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-21T23:09:20.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.372+0000 7f1d85a64740 1 -- 172.21.15.28:0/1424413954 --> [v2:172.21.15.81:6834/4241906475,v1:172.21.15.81:6836/4241906475] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c84bba69b0 con 0x55c84bc81140 2024-02-21T23:09:20.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.373+0000 7f1d7be00640 1 --2- 172.21.15.28:0/1424413954 >> [v2:172.21.15.81:6834/4241906475,v1:172.21.15.81:6836/4241906475] conn(0x55c84bc81140 0x55c84bc81520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:20.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.374+0000 7f1d7be00640 1 --2- 172.21.15.28:0/1424413954 >> [v2:172.21.15.81:6834/4241906475,v1:172.21.15.81:6836/4241906475] conn(0x55c84bc81140 0x55c84bc81520 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5df1cb2062d7bfdb server_cookie=dd45ddd836bb5d8b in_seq=0 out_seq=0 2024-02-21T23:09:20.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.374+0000 7f1d64c00640 10 client.8574 ms_handle_connect on v2:172.21.15.81:6834/4241906475 2024-02-21T23:09:20.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.376+0000 7f1d64c00640 1 -- 172.21.15.28:0/1424413954 <== mds.0 v2:172.21.15.81:6834/4241906475 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c84bba69b0 con 0x55c84bc81140 2024-02-21T23:09:20.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.377+0000 7f1d64c00640 10 client.8574 handle_command_reply: tid=0 2024-02-21T23:09:20.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.377+0000 7f1d85a64740 1 client.8574 shutdown 2024-02-21T23:09:20.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.377+0000 7f1d85a64740 1 -- 172.21.15.28:0/1424413954 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84bc75e50 msgr2=0x55c84bc50a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:20.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.377+0000 7f1d85a64740 1 --2- 172.21.15.28:0/1424413954 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84bc75e50 0x55c84bc50a00 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f1d74013000 tx=0x7f1d74001250 comp rx=0 tx=0).stop 2024-02-21T23:09:20.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.377+0000 7f1d85a64740 1 -- 172.21.15.28:0/1424413954 shutdown_connections 2024-02-21T23:09:20.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.377+0000 7f1d85a64740 1 --2- 172.21.15.28:0/1424413954 >> [v2:172.21.15.81:6834/4241906475,v1:172.21.15.81:6836/4241906475] conn(0x55c84bc81140 0x55c84bc81520 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:20.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.377+0000 7f1d85a64740 1 --2- 172.21.15.28:0/1424413954 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c84bc714d0 0x55c84bc84930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:20.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.377+0000 7f1d85a64740 1 --2- 172.21.15.28:0/1424413954 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84bc75e50 0x55c84bc50a00 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:20.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.377+0000 7f1d85a64740 1 --2- 172.21.15.28:0/1424413954 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c84bc36150 0x55c84bc77760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:20.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.377+0000 7f1d85a64740 1 -- 172.21.15.28:0/1424413954 >> 172.21.15.28:0/1424413954 conn(0x55c84bc34e40 msgr2=0x55c84bc6fea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:20.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.377+0000 7f1d64c00640 0 client.8574 ms_handle_reset on v2:172.21.15.81:6834/4241906475 2024-02-21T23:09:20.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.377+0000 7f1d85a64740 1 -- 172.21.15.28:0/1424413954 shutdown_connections 2024-02-21T23:09:20.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.377+0000 7f1d85a64740 1 -- 172.21.15.28:0/1424413954 wait complete. 2024-02-21T23:09:20.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.377+0000 7f1d85a64740 20 client.8574 trim_cache size 0 max 16384 2024-02-21T23:09:20.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.396+0000 7f1d85a64740 1 Processor -- start 2024-02-21T23:09:20.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.396+0000 7f1d85a64740 1 -- start start 2024-02-21T23:09:20.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.397+0000 7f1d85a64740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c84bc36150 0x55c84bc6dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:20.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.397+0000 7f1d85a64740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c84bc714d0 0x55c84bc6e170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:20.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.397+0000 7f1d85a64740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84bc75e50 0x55c84bc81520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:20.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.397+0000 7f1d85a64740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7c1fb310 con 0x55c84bc75e50 2024-02-21T23:09:20.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.397+0000 7f1d85a64740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7c15b220 con 0x55c84bc714d0 2024-02-21T23:09:20.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.397+0000 7f1d85a64740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7c1fade0 con 0x55c84bc36150 2024-02-21T23:09:20.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.397+0000 7f1d80c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84bc75e50 0x55c84bc81520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:20.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.397+0000 7f1d80c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84bc75e50 0x55c84bc81520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58826/0 (socket says 172.21.15.28:58826) 2024-02-21T23:09:20.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.397+0000 7f1d80c00640 1 -- 172.21.15.28:0/3166580663 learned_addr learned my addr 172.21.15.28:0/3166580663 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:20.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.397+0000 7f1d7be00640 1 --2- 172.21.15.28:0/3166580663 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c84bc36150 0x55c84bc6dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:20.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.397+0000 7f1d7b400640 1 --2- 172.21.15.28:0/3166580663 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c84bc714d0 0x55c84bc6e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:20.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.397+0000 7f1d80c00640 1 -- 172.21.15.28:0/3166580663 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c84bc36150 msgr2=0x55c84bc6dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:20.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.397+0000 7f1d80c00640 1 --2- 172.21.15.28:0/3166580663 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c84bc36150 0x55c84bc6dc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:20.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.397+0000 7f1d80c00640 1 -- 172.21.15.28:0/3166580663 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c84bc714d0 msgr2=0x55c84bc6e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:20.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.397+0000 7f1d80c00640 1 --2- 172.21.15.28:0/3166580663 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c84bc714d0 0x55c84bc6e170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:20.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.397+0000 7f1d80c00640 1 -- 172.21.15.28:0/3166580663 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c84bc35590 con 0x55c84bc75e50 2024-02-21T23:09:20.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.398+0000 7f1d80c00640 1 --2- 172.21.15.28:0/3166580663 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84bc75e50 0x55c84bc81520 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f1d74001a50 tx=0x7f1d7401c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c77f49cacd915128 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:20.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.398+0000 7f1d67400640 1 -- 172.21.15.28:0/3166580663 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d74035070 con 0x55c84bc75e50 2024-02-21T23:09:20.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.398+0000 7f1d67400640 1 -- 172.21.15.28:0/3166580663 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d74023070 con 0x55c84bc75e50 2024-02-21T23:09:20.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.398+0000 7f1d67400640 1 -- 172.21.15.28:0/3166580663 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d74019040 con 0x55c84bc75e50 2024-02-21T23:09:20.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.398+0000 7f1d85a64740 1 -- 172.21.15.28:0/3166580663 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84bc75e50 msgr2=0x55c84bc81520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:20.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.398+0000 7f1d85a64740 1 --2- 172.21.15.28:0/3166580663 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84bc75e50 0x55c84bc81520 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f1d74001a50 tx=0x7f1d7401c000 comp rx=0 tx=0).stop 2024-02-21T23:09:20.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.399+0000 7f1d85a64740 1 -- 172.21.15.28:0/3166580663 shutdown_connections 2024-02-21T23:09:20.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.399+0000 7f1d85a64740 1 --2- 172.21.15.28:0/3166580663 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84bc75e50 0x55c84bc81520 unknown :-1 s=CLOSED pgs=1568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:20.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.399+0000 7f1d85a64740 1 --2- 172.21.15.28:0/3166580663 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c84bc714d0 0x55c84bc6e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:20.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.399+0000 7f1d85a64740 1 --2- 172.21.15.28:0/3166580663 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c84bc36150 0x55c84bc6dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:20.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.399+0000 7f1d85a64740 1 -- 172.21.15.28:0/3166580663 >> 172.21.15.28:0/3166580663 conn(0x55c84bc34e40 msgr2=0x55c84bc84740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:20.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.399+0000 7f1d85a64740 1 -- 172.21.15.28:0/3166580663 shutdown_connections 2024-02-21T23:09:20.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.399+0000 7f1d85a64740 1 -- 172.21.15.28:0/3166580663 wait complete. 2024-02-21T23:09:20.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.399+0000 7f1d85a64740 1 -- start start 2024-02-21T23:09:20.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.399+0000 7f1d85a64740 1 Processor -- start 2024-02-21T23:09:20.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.400+0000 7f1d85a64740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c84bc36150 0x55c84bc6e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:20.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.400+0000 7f1d85a64740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c84bc714d0 0x55c84bce0a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:20.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.400+0000 7f1d85a64740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84bc75e50 0x55c84bcdde80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:20.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.400+0000 7f1d85a64740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c84bc782b0 con 0x55c84bc75e50 2024-02-21T23:09:20.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.400+0000 7f1d85a64740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c84bc80fc0 con 0x55c84bc714d0 2024-02-21T23:09:20.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.400+0000 7f1d85a64740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c84bc77cf0 con 0x55c84bc36150 2024-02-21T23:09:20.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.400+0000 7f1d85a64740 10 client.0 _ll_register_callbacks cb 0x7f1d74003d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:09:20.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.401+0000 7f1d80c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84bc75e50 0x55c84bcdde80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:20.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.401+0000 7f1d80c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84bc75e50 0x55c84bcdde80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58840/0 (socket says 172.21.15.28:58840) 2024-02-21T23:09:20.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.401+0000 7f1d80c00640 1 -- 172.21.15.28:0/1789798079 learned_addr learned my addr 172.21.15.28:0/1789798079 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:20.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.401+0000 7f1d7b400640 1 --2- 172.21.15.28:0/1789798079 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c84bc714d0 0x55c84bce0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:20.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.401+0000 7f1d80c00640 1 -- 172.21.15.28:0/1789798079 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c84bc36150 msgr2=0x55c84bc6e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:20.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.401+0000 7f1d80c00640 1 --2- 172.21.15.28:0/1789798079 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c84bc36150 0x55c84bc6e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:20.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.401+0000 7f1d80c00640 1 -- 172.21.15.28:0/1789798079 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c84bc714d0 msgr2=0x55c84bce0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:20.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.401+0000 7f1d80c00640 1 --2- 172.21.15.28:0/1789798079 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c84bc714d0 0x55c84bce0a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:20.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.401+0000 7f1d80c00640 1 -- 172.21.15.28:0/1789798079 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c84b9786b0 con 0x55c84bc75e50 2024-02-21T23:09:20.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.401+0000 7f1d80c00640 1 --2- 172.21.15.28:0/1789798079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84bc75e50 0x55c84bcdde80 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f1d74020040 tx=0x7f1d74009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cef86450ef2d056 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:20.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.401+0000 7f1d64c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:09:20.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.402+0000 7f1d64c00640 1 -- 172.21.15.28:0/1789798079 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d74019070 con 0x55c84bc75e50 2024-02-21T23:09:20.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.402+0000 7f1d64c00640 1 -- 172.21.15.28:0/1789798079 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(137..137 src has 1..137) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f1d7403ac90 con 0x55c84bc75e50 2024-02-21T23:09:20.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.402+0000 7f1d85a64740 1 -- 172.21.15.28:0/1789798079 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1d74023070 con 0x55c84bc75e50 2024-02-21T23:09:20.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.402+0000 7f1d64c00640 1 -- 172.21.15.28:0/1789798079 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=144) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1d74035070 con 0x55c84bc75e50 2024-02-21T23:09:20.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.402+0000 7f1d85a64740 10 client.8580 fetch_fsmap learned FSMap version 144 2024-02-21T23:09:20.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.402+0000 7f1d85a64740 1 -- 172.21.15.28:0/1789798079 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=144,monmap=2+}) v3 -- 0x55c84bc2ff20 con 0x55c84bc75e50 2024-02-21T23:09:20.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.402+0000 7f1d64c00640 1 -- 172.21.15.28:0/1789798079 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d74023070 con 0x55c84bc75e50 2024-02-21T23:09:20.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.402+0000 7f1d64c00640 1 -- 172.21.15.28:0/1789798079 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 144) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f1d74044080 con 0x55c84bc75e50 2024-02-21T23:09:20.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.402+0000 7f1d85a64740 10 client.8580 fetch_fsmap finished waiting for FSMap version 144 2024-02-21T23:09:20.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.402+0000 7f1d85a64740 10 client.8580 resolve_mds: resolved name 'b' to daemon mds.b 2024-02-21T23:09:20.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.402+0000 7f1d85a64740 20 client.8580 populate_metadata read hostname 'smithi028' 2024-02-21T23:09:20.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.402+0000 7f1d85a64740 1 --2- 172.21.15.28:0/1789798079 >> [v2:172.21.15.81:6834/4241906475,v1:172.21.15.81:6836/4241906475] conn(0x55c84bc93dd0 0x55c84bc941b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:20.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.403+0000 7f1d85a64740 4 client.8580 mds_command: new command op to 6140 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-21T23:09:20.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.403+0000 7f1d85a64740 1 -- 172.21.15.28:0/1789798079 --> [v2:172.21.15.81:6834/4241906475,v1:172.21.15.81:6836/4241906475] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55c84bba0d00 con 0x55c84bc93dd0 2024-02-21T23:09:20.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.403+0000 7f1d7be00640 1 --2- 172.21.15.28:0/1789798079 >> [v2:172.21.15.81:6834/4241906475,v1:172.21.15.81:6836/4241906475] conn(0x55c84bc93dd0 0x55c84bc941b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:20.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.404+0000 7f1d7be00640 1 --2- 172.21.15.28:0/1789798079 >> [v2:172.21.15.81:6834/4241906475,v1:172.21.15.81:6836/4241906475] conn(0x55c84bc93dd0 0x55c84bc941b0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8aa6422a27ecf7e7 server_cookie=51d24dd644a09e08 in_seq=0 out_seq=0 2024-02-21T23:09:20.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.404+0000 7f1d64c00640 10 client.8580 ms_handle_connect on v2:172.21.15.81:6834/4241906475 2024-02-21T23:09:20.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.405+0000 7f1d64c00640 1 -- 172.21.15.28:0/1789798079 <== mds.0 v2:172.21.15.81:6834/4241906475 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+334 (crc 0 0 0) 0x55c84bba0d00 con 0x55c84bc93dd0 2024-02-21T23:09:20.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.405+0000 7f1d64c00640 10 client.8580 handle_command_reply: tid=0 2024-02-21T23:09:20.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.405+0000 7f1d85a64740 1 client.8580 shutdown 2024-02-21T23:09:20.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.405+0000 7f1d85a64740 1 -- 172.21.15.28:0/1789798079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84bc75e50 msgr2=0x55c84bcdde80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:20.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.405+0000 7f1d85a64740 1 --2- 172.21.15.28:0/1789798079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84bc75e50 0x55c84bcdde80 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f1d74020040 tx=0x7f1d74009040 comp rx=0 tx=0).stop 2024-02-21T23:09:20.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.405+0000 7f1d85a64740 1 -- 172.21.15.28:0/1789798079 shutdown_connections 2024-02-21T23:09:20.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.405+0000 7f1d85a64740 1 --2- 172.21.15.28:0/1789798079 >> [v2:172.21.15.81:6834/4241906475,v1:172.21.15.81:6836/4241906475] conn(0x55c84bc93dd0 0x55c84bc941b0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:20.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.405+0000 7f1d85a64740 1 --2- 172.21.15.28:0/1789798079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c84bc75e50 0x55c84bcdde80 unknown :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:20.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.405+0000 7f1d85a64740 1 --2- 172.21.15.28:0/1789798079 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c84bc714d0 0x55c84bce0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:20.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.405+0000 7f1d85a64740 1 --2- 172.21.15.28:0/1789798079 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c84bc36150 0x55c84bc6e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:20.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.405+0000 7f1d85a64740 1 -- 172.21.15.28:0/1789798079 >> 172.21.15.28:0/1789798079 conn(0x55c84bc34e40 msgr2=0x55c84bc6fe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:20.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.405+0000 7f1d64c00640 0 client.8580 ms_handle_reset on v2:172.21.15.81:6834/4241906475 2024-02-21T23:09:20.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.405+0000 7f1d85a64740 1 -- 172.21.15.28:0/1789798079 shutdown_connections 2024-02-21T23:09:20.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.405+0000 7f1d85a64740 1 -- 172.21.15.28:0/1789798079 wait complete. 2024-02-21T23:09:20.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.405+0000 7f1d85a64740 20 client.8580 trim_cache size 0 max 16384 2024-02-21T23:09:20.396 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-21T23:09:20.396 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac", 2024-02-21T23:09:20.396 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-02-21T23:09:20.396 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 6140, 2024-02-21T23:09:20.396 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-02-21T23:09:20.396 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-02-21T23:09:20.396 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-02-21T23:09:20.397 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 4.8680960180000001, 2024-02-21T23:09:20.397 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 144, 2024-02-21T23:09:20.397 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 137, 2024-02-21T23:09:20.397 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 137, 2024-02-21T23:09:20.397 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 28.966959902999999 2024-02-21T23:09:20.397 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-21T23:09:20.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.406+0000 7f1d82c00640 1 -- 172.21.15.28:0/1553934277 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1d5c06b500 msgr2=0x7f1d5c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:20.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.406+0000 7f1d82c00640 1 --2- 172.21.15.28:0/1553934277 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1d5c06b500 0x7f1d5c06d9c0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c009a30 tx=0x7f1d6c008040 comp rx=0 tx=0).stop 2024-02-21T23:09:20.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.406+0000 7f1d82c00640 1 -- 172.21.15.28:0/1553934277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d7c156c80 msgr2=0x7f1d7c155eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:20.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.406+0000 7f1d82c00640 1 --2- 172.21.15.28:0/1553934277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d7c156c80 0x7f1d7c155eb0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f1d68009870 tx=0x7f1d68004730 comp rx=0 tx=0).stop 2024-02-21T23:09:20.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.406+0000 7f1d82c00640 1 -- 172.21.15.28:0/1553934277 shutdown_connections 2024-02-21T23:09:20.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.406+0000 7f1d82c00640 1 --2- 172.21.15.28:0/1553934277 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1d5c06b500 0x7f1d5c06d9c0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:20.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.406+0000 7f1d82c00640 1 --2- 172.21.15.28:0/1553934277 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1d7c0ab4a0 0x7f1d7c1f2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:20.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.406+0000 7f1d82c00640 1 --2- 172.21.15.28:0/1553934277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d7c156c80 0x7f1d7c155eb0 unknown :-1 s=CLOSED pgs=1565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:20.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.406+0000 7f1d82c00640 1 --2- 172.21.15.28:0/1553934277 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1d7c0c11f0 0x7f1d7c155970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:20.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.406+0000 7f1d82c00640 1 -- 172.21.15.28:0/1553934277 >> 172.21.15.28:0/1553934277 conn(0x7f1d7c06b820 msgr2=0x7f1d7c0c4ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:20.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.406+0000 7f1d82c00640 1 -- 172.21.15.28:0/1553934277 shutdown_connections 2024-02-21T23:09:20.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:20.406+0000 7f1d82c00640 1 -- 172.21.15.28:0/1553934277 wait complete. 2024-02-21T23:09:20.414 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 10, "epoch": 144, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 10, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-02-21T23:09:03.869294+0000", "damaged": [], "data_pools": [ 22 ], "enabled": true, "epoch": 144, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6140": { "addr": "172.21.15.81:6836/4241906475", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6834", "nonce": 4241906475, "type": "v2" }, { "addr": "172.21.15.81:6836", "nonce": 4241906475, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6140, "incarnation": 141, "join_fscid": 10, "name": "b", "rank": 0, "state": "up:active", "state_seq": 10 } }, "last_failure": 0, "last_failure_osd_epoch": 137, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 21, "modified": "2024-02-21T23:09:17.546876+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6140 } } } ], "standbys": [ { "addr": "172.21.15.81:6837/2095908433", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6835", "nonce": 2095908433, "type": "v2" }, { "addr": "172.21.15.81:6837", "nonce": 2095908433, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 140, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6143, "incarnation": 0, "join_fscid": 10, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 7 }, { "addr": "172.21.15.28:6833/3934548735", "addrs": { "addrvec": [ { "addr": "172.21.15.28:6832", "nonce": 3934548735, "type": "v2" }, { "addr": "172.21.15.28:6833", "nonce": 3934548735, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 133, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8370, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.28:6835/2100132822", "addrs": { "addrvec": [ { "addr": "172.21.15.28:6834", "nonce": 2100132822, "type": "v2" }, { "addr": "172.21.15.28:6835", "nonce": 2100132822, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 142, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8556, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-02-21T23:09:20.415 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': 10, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': 10, 'name': 'd'}] 2024-02-21T23:09:20.415 INFO:tasks.cephfs.test_failover:current = [{'gid': 8556, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.28:6835/2100132822', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 2100132822}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 2100132822}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 142}, {'gid': 6140, 'name': 'b', 'rank': 0, 'incarnation': 141, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.81:6836/4241906475', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6834', 'nonce': 4241906475}, {'type': 'v1', 'addr': '172.21.15.81:6836', 'nonce': 4241906475}]}, 'join_fscid': 10, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 8370, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.28:6833/3934548735', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 3934548735}, {'type': 'v1', 'addr': '172.21.15.28:6833', 'nonce': 3934548735}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 133}, {'gid': 6143, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 7, 'addr': '172.21.15.81:6837/2095908433', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6835', 'nonce': 2095908433}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 2095908433}]}, 'join_fscid': 10, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 140}] 2024-02-21T23:09:20.415 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-02-21T23:09:22.312 INFO:tasks.ceph.mds.b.smithi081.stderr:2024-02-21T23:09:22.323+0000 7febabc00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-21T23:09:25.416 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:09:25.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.666+0000 7ff2b3000640 1 Processor -- start 2024-02-21T23:09:25.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.666+0000 7ff2b3000640 1 -- start start 2024-02-21T23:09:25.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.667+0000 7ff2b3000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff2ac06ee40 0x7ff2ac0a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:25.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.668+0000 7ff2b3000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2ac0c2760 con 0x7ff2ac156af0 2024-02-21T23:09:25.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.668+0000 7ff2b3000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2ac156310 con 0x7ff2ac150450 2024-02-21T23:09:25.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.668+0000 7ff2b3000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2ac0c22c0 con 0x7ff2ac06ee40 2024-02-21T23:09:25.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.668+0000 7ff2b2600640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff2ac156af0 0x7ff2ac06dff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:44960/0 (socket says 172.21.15.28:44960) 2024-02-21T23:09:25.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.668+0000 7ff2b2600640 1 -- 172.21.15.28:0/1865978791 learned_addr learned my addr 172.21.15.28:0/1865978791 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:09:25.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.668+0000 7ff2b1200640 1 --2- 172.21.15.28:0/1865978791 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff2ac06ee40 0x7ff2ac0a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:25.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.669+0000 7ff2abe00640 1 -- 172.21.15.28:0/1865978791 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1685857784 0 0) 0x7ff2ac0c2760 con 0x7ff2ac156af0 2024-02-21T23:09:25.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.669+0000 7ff2abe00640 1 -- 172.21.15.28:0/1865978791 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff294003520 con 0x7ff2ac156af0 2024-02-21T23:09:25.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.669+0000 7ff2abe00640 1 -- 172.21.15.28:0/1865978791 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 981830796 0 0) 0x7ff294003520 con 0x7ff2ac156af0 2024-02-21T23:09:25.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.669+0000 7ff2abe00640 1 -- 172.21.15.28:0/1865978791 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff2ac06ee40 msgr2=0x7ff2ac0a5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:25.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.669+0000 7ff2abe00640 1 --2- 172.21.15.28:0/1865978791 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff2ac06ee40 0x7ff2ac0a5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:25.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.669+0000 7ff2abe00640 1 -- 172.21.15.28:0/1865978791 >> v1:172.21.15.81:6789/0 conn(0x7ff2ac150450 legacy=0x7ff2ac1536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:25.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.669+0000 7ff2abe00640 1 -- 172.21.15.28:0/1865978791 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2ac0c29d0 con 0x7ff2ac156af0 2024-02-21T23:09:25.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.669+0000 7ff2abe00640 1 -- 172.21.15.28:0/1865978791 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7ff2a40019f0 con 0x7ff2ac156af0 2024-02-21T23:09:25.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.669+0000 7ff2abe00640 1 -- 172.21.15.28:0/1865978791 >> v1:172.21.15.28:6789/0 conn(0x7ff2ac156af0 legacy=0x7ff2ac06dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:25.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.669+0000 7ff2abe00640 1 --2- 172.21.15.28:0/1865978791 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff2ac06ee40 0x7ff2940039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:25.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.669+0000 7ff2abe00640 1 --2- 172.21.15.28:0/1865978791 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff294003fb0 0x7ff294004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:25.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.669+0000 7ff2abe00640 1 --2- 172.21.15.28:0/1865978791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff294004d40 0x7ff2940051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:25.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.669+0000 7ff2abe00640 1 -- 172.21.15.28:0/1865978791 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff29400bad0 con 0x7ff294004d40 2024-02-21T23:09:25.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.669+0000 7ff2abe00640 1 -- 172.21.15.28:0/1865978791 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff29400bc80 con 0x7ff294003fb0 2024-02-21T23:09:25.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.669+0000 7ff2abe00640 1 -- 172.21.15.28:0/1865978791 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff29400be80 con 0x7ff2ac06ee40 2024-02-21T23:09:25.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.669+0000 7ff2b2600640 1 --2- 172.21.15.28:0/1865978791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff294004d40 0x7ff2940051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:25.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.670+0000 7ff2b1200640 1 --2- 172.21.15.28:0/1865978791 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff294003fb0 0x7ff294004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:25.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.670+0000 7ff2b1c00640 1 --2- 172.21.15.28:0/1865978791 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff2ac06ee40 0x7ff2940039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:25.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.670+0000 7ff2b2600640 1 -- 172.21.15.28:0/1865978791 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff2ac06ee40 msgr2=0x7ff2940039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:25.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.670+0000 7ff2b2600640 1 --2- 172.21.15.28:0/1865978791 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff2ac06ee40 0x7ff2940039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:25.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.670+0000 7ff2b2600640 1 -- 172.21.15.28:0/1865978791 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff294003fb0 msgr2=0x7ff294004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:25.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.670+0000 7ff2b2600640 1 --2- 172.21.15.28:0/1865978791 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff294003fb0 0x7ff294004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:25.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.670+0000 7ff2b2600640 1 -- 172.21.15.28:0/1865978791 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff294003520 con 0x7ff294004d40 2024-02-21T23:09:25.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.671+0000 7ff2b2600640 1 --2- 172.21.15.28:0/1865978791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff294004d40 0x7ff2940051c0 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7ff2a400a790 tx=0x7ff294005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a1471e680334320 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:25.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.671+0000 7ff2abe00640 1 -- 172.21.15.28:0/1865978791 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2a400a3b0 con 0x7ff294004d40 2024-02-21T23:09:25.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.671+0000 7ff2abe00640 1 -- 172.21.15.28:0/1865978791 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2a4034650 con 0x7ff294004d40 2024-02-21T23:09:25.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.671+0000 7ff2abe00640 1 -- 172.21.15.28:0/1865978791 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2a4034a80 con 0x7ff294004d40 2024-02-21T23:09:25.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.672+0000 7ff2b3000640 1 -- 172.21.15.28:0/1865978791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff294004d40 msgr2=0x7ff2940051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:25.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.672+0000 7ff2b3000640 1 --2- 172.21.15.28:0/1865978791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff294004d40 0x7ff2940051c0 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7ff2a400a790 tx=0x7ff294005830 comp rx=0 tx=0).stop 2024-02-21T23:09:25.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.672+0000 7ff2b2600640 1 -- 172.21.15.28:0/1865978791 reap_dead start 2024-02-21T23:09:25.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.672+0000 7ff2b3000640 1 -- 172.21.15.28:0/1865978791 shutdown_connections 2024-02-21T23:09:25.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.672+0000 7ff2b3000640 1 -- 172.21.15.28:0/1865978791 >> 172.21.15.28:0/1865978791 conn(0x7ff2ac06b630 msgr2=0x7ff2ac0c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:25.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.672+0000 7ff2b3000640 1 -- 172.21.15.28:0/1865978791 shutdown_connections 2024-02-21T23:09:25.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.672+0000 7ff2b3000640 1 -- 172.21.15.28:0/1865978791 wait complete. 2024-02-21T23:09:25.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.674+0000 7ff2b3000640 1 Processor -- start 2024-02-21T23:09:25.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.674+0000 7ff2b3000640 1 -- start start 2024-02-21T23:09:25.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.675+0000 7ff2b3000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff294004d40 0x7ff2ac1f23a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:25.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.675+0000 7ff2b3000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff2ac1f28e0 0x7ff2ac0aa7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:25.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.675+0000 7ff2b3000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff2ac0aad00 0x7ff2ac149670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:25.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.675+0000 7ff2b3000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2ac06e700 con 0x7ff294004d40 2024-02-21T23:09:25.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.675+0000 7ff2b3000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2ac156660 con 0x7ff2ac1f28e0 2024-02-21T23:09:25.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.675+0000 7ff2b3000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2ac06e880 con 0x7ff2ac0aad00 2024-02-21T23:09:25.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.675+0000 7ff2b1c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff294004d40 0x7ff2ac1f23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:25.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.675+0000 7ff2b1c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff294004d40 0x7ff2ac1f23a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57212/0 (socket says 172.21.15.28:57212) 2024-02-21T23:09:25.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.675+0000 7ff2b1c00640 1 -- 172.21.15.28:0/8974462 learned_addr learned my addr 172.21.15.28:0/8974462 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:25.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.676+0000 7ff2b1200640 1 --2- 172.21.15.28:0/8974462 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff2ac1f28e0 0x7ff2ac0aa7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:25.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.676+0000 7ff2b2600640 1 --2- 172.21.15.28:0/8974462 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff2ac0aad00 0x7ff2ac149670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:25.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.676+0000 7ff2b1c00640 1 -- 172.21.15.28:0/8974462 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff2ac0aad00 msgr2=0x7ff2ac149670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:25.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.676+0000 7ff2b1c00640 1 --2- 172.21.15.28:0/8974462 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff2ac0aad00 0x7ff2ac149670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:25.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.676+0000 7ff2b1c00640 1 -- 172.21.15.28:0/8974462 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff2ac1f28e0 msgr2=0x7ff2ac0aa7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:25.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.676+0000 7ff2b1c00640 1 --2- 172.21.15.28:0/8974462 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff2ac1f28e0 0x7ff2ac0aa7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:25.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.676+0000 7ff2b1c00640 1 -- 172.21.15.28:0/8974462 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2ac071b20 con 0x7ff294004d40 2024-02-21T23:09:25.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.676+0000 7ff2b2600640 1 --2- 172.21.15.28:0/8974462 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff2ac0aad00 0x7ff2ac149670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:09:25.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.676+0000 7ff2b1c00640 1 --2- 172.21.15.28:0/8974462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff294004d40 0x7ff2ac1f23a0 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7ff29c00e580 tx=0x7ff29c00ea40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:25.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.676+0000 7ff2a9600640 1 -- 172.21.15.28:0/8974462 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff29c0052e0 con 0x7ff294004d40 2024-02-21T23:09:25.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.676+0000 7ff2a9600640 1 -- 172.21.15.28:0/8974462 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff29c018070 con 0x7ff294004d40 2024-02-21T23:09:25.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.676+0000 7ff2a9600640 1 -- 172.21.15.28:0/8974462 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff29c003810 con 0x7ff294004d40 2024-02-21T23:09:25.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.676+0000 7ff2b3000640 1 -- 172.21.15.28:0/8974462 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2ac071e00 con 0x7ff294004d40 2024-02-21T23:09:25.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.677+0000 7ff2b3000640 1 -- 172.21.15.28:0/8974462 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2ac0c63b0 con 0x7ff294004d40 2024-02-21T23:09:25.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.678+0000 7ff2b3000640 1 -- 172.21.15.28:0/8974462 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2ac071fd0 con 0x7ff294004d40 2024-02-21T23:09:25.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.679+0000 7ff2a9600640 1 -- 172.21.15.28:0/8974462 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff29c0039b0 con 0x7ff294004d40 2024-02-21T23:09:25.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.680+0000 7ff2a9600640 1 --2- 172.21.15.28:0/8974462 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff29006b500 0x7ff29006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:25.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.680+0000 7ff2a9600640 1 -- 172.21.15.28:0/8974462 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 6245+0+0 (secure 0 0 0) 0x7ff29c089ff0 con 0x7ff294004d40 2024-02-21T23:09:25.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.680+0000 7ff2b1200640 1 --2- 172.21.15.28:0/8974462 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff29006b500 0x7ff29006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:25.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.682+0000 7ff2b1200640 1 --2- 172.21.15.28:0/8974462 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff29006b500 0x7ff29006d9c0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7ff2ac0c0890 tx=0x7ff2a0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:25.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.684+0000 7ff2a9600640 1 -- 172.21.15.28:0/8974462 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ff29c058c60 con 0x7ff294004d40 2024-02-21T23:09:25.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.926+0000 7ff2b3000640 1 -- 172.21.15.28:0/8974462 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff2ac04f290 con 0x7ff294004d40 2024-02-21T23:09:25.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.927+0000 7ff2a9600640 1 -- 172.21.15.28:0/8974462 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) v1 ==== 95+0+4951 (secure 0 0 0) 0x7ff29c058560 con 0x7ff294004d40 2024-02-21T23:09:25.915 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:09:25.916 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":144,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6143,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":7,"addr":"172.21.15.81:6837/2095908433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":2095908433},{"type":"v1","addr":"172.21.15.81:6837","nonce":2095908433}]},"join_fscid":10,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":140},{"gid":8370,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/3934548735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3934548735},{"type":"v1","addr":"172.21.15.28:6833","nonce":3934548735}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":133},{"gid":8556,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2100132822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2100132822},{"type":"v1","addr":"172.21.15.28:6835","nonce":2100132822}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":142}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:09:03.869294+0000","modified":"2024-02-21T23:09:17.546876+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":137,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6140},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6140":{"gid":6140,"name":"b","rank":0,"incarnation":141,"state":"up:active","state_seq":10,"addr":"172.21.15.81:6836/4241906475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":4241906475},{"type":"v1","addr":"172.21.15.81:6836","nonce":4241906475}]},"join_fscid":10,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-02-21T23:09:25.916 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 144 2024-02-21T23:09:25.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.930+0000 7ff2b3000640 1 -- 172.21.15.28:0/8974462 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff29006b500 msgr2=0x7ff29006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:25.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.930+0000 7ff2b3000640 1 --2- 172.21.15.28:0/8974462 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff29006b500 0x7ff29006d9c0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7ff2ac0c0890 tx=0x7ff2a0008040 comp rx=0 tx=0).stop 2024-02-21T23:09:25.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.930+0000 7ff2b3000640 1 -- 172.21.15.28:0/8974462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff294004d40 msgr2=0x7ff2ac1f23a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:25.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.930+0000 7ff2b3000640 1 --2- 172.21.15.28:0/8974462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff294004d40 0x7ff2ac1f23a0 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7ff29c00e580 tx=0x7ff29c00ea40 comp rx=0 tx=0).stop 2024-02-21T23:09:25.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.930+0000 7ff2b3000640 1 -- 172.21.15.28:0/8974462 shutdown_connections 2024-02-21T23:09:25.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.930+0000 7ff2b3000640 1 --2- 172.21.15.28:0/8974462 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff29006b500 0x7ff29006d9c0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:25.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.930+0000 7ff2b3000640 1 --2- 172.21.15.28:0/8974462 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff2ac0aad00 0x7ff2ac149670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:25.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.930+0000 7ff2b3000640 1 --2- 172.21.15.28:0/8974462 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff2ac1f28e0 0x7ff2ac0aa7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:25.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.930+0000 7ff2b3000640 1 --2- 172.21.15.28:0/8974462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff294004d40 0x7ff2ac1f23a0 unknown :-1 s=CLOSED pgs=1572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:25.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.930+0000 7ff2b3000640 1 -- 172.21.15.28:0/8974462 >> 172.21.15.28:0/8974462 conn(0x7ff2ac06b630 msgr2=0x7ff2ac14e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:25.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.930+0000 7ff2b3000640 1 -- 172.21.15.28:0/8974462 shutdown_connections 2024-02-21T23:09:25.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:25.930+0000 7ff2b3000640 1 -- 172.21.15.28:0/8974462 wait complete. 2024-02-21T23:09:26.340 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 144, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:09:03.869294+0000', 'modified': '2024-02-21T23:09:17.546876+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 137, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6140}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6140': {'gid': 6140, 'name': 'b', 'rank': 0, 'incarnation': 141, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.81:6836/4241906475', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6834', 'nonce': 4241906475}, {'type': 'v1', 'addr': '172.21.15.81:6836', 'nonce': 4241906475}]}, 'join_fscid': 10, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:09:26.340 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-21T23:09:26.341 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-02-21T23:09:26.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.584+0000 7f6b8b000640 1 Processor -- start 2024-02-21T23:09:26.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.585+0000 7f6b8b000640 1 -- start start 2024-02-21T23:09:26.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.587+0000 7f6b8b000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f6b84149830 0x7f6b840a8e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:26.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.587+0000 7f6b8b000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6b8406d410 0x7f6b840c0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:26.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.587+0000 7f6b8b000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b84149c10 con 0x7f6b840c1420 2024-02-21T23:09:26.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.587+0000 7f6b8b000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b841537d0 con 0x7f6b84149830 2024-02-21T23:09:26.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.587+0000 7f6b8b000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b84153ab0 con 0x7f6b8406d410 2024-02-21T23:09:26.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.588+0000 7f6b89200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6b8406d410 0x7f6b840c0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:26.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.588+0000 7f6b89200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6b8406d410 0x7f6b840c0ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3301/0 says I am v2:172.21.15.28:46976/0 (socket says 172.21.15.28:46976) 2024-02-21T23:09:26.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.588+0000 7f6b89200640 1 -- 172.21.15.28:0/2631168575 learned_addr learned my addr 172.21.15.28:0/2631168575 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:26.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.588+0000 7f6b89c00640 1 --2- 172.21.15.28:0/2631168575 >> v2:172.21.15.81:3300/0 conn(0x7f6b84149830 0x7f6b840a8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:26.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.589+0000 7f6b89200640 1 -- 172.21.15.28:0/2631168575 >> v2:172.21.15.81:3300/0 conn(0x7f6b84149830 msgr2=0x7f6b840a8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:26.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.589+0000 7f6b89200640 1 --2- 172.21.15.28:0/2631168575 >> v2:172.21.15.81:3300/0 conn(0x7f6b84149830 0x7f6b840a8e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:26.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.589+0000 7f6b89200640 1 -- 172.21.15.28:0/2631168575 >> v1:172.21.15.81:6789/0 conn(0x7f6b840c1420 legacy=0x7f6b840c5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:26.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.589+0000 7f6b89200640 1 -- 172.21.15.28:0/2631168575 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b840c4fa0 con 0x7f6b8406d410 2024-02-21T23:09:26.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.590+0000 7f6b89200640 1 --2- 172.21.15.28:0/2631168575 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6b8406d410 0x7f6b840c0ee0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6b70009870 tx=0x7f6b7002ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae01df5f41daee23 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:26.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.590+0000 7f6b83e00640 1 -- 172.21.15.28:0/2631168575 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b7002f840 con 0x7f6b8406d410 2024-02-21T23:09:26.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.591+0000 7f6b83e00640 1 -- 172.21.15.28:0/2631168575 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b7002f9e0 con 0x7f6b8406d410 2024-02-21T23:09:26.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.591+0000 7f6b83e00640 1 -- 172.21.15.28:0/2631168575 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b7002fdb0 con 0x7f6b8406d410 2024-02-21T23:09:26.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.591+0000 7f6b8b000640 1 -- 172.21.15.28:0/2631168575 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6b8406d410 msgr2=0x7f6b840c0ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:26.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.591+0000 7f6b8b000640 1 --2- 172.21.15.28:0/2631168575 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6b8406d410 0x7f6b840c0ee0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6b70009870 tx=0x7f6b7002ba20 comp rx=0 tx=0).stop 2024-02-21T23:09:26.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.591+0000 7f6b8b000640 1 -- 172.21.15.28:0/2631168575 shutdown_connections 2024-02-21T23:09:26.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.591+0000 7f6b8b000640 1 --2- 172.21.15.28:0/2631168575 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6b8406d410 0x7f6b840c0ee0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:26.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.591+0000 7f6b8b000640 1 --2- 172.21.15.28:0/2631168575 >> v2:172.21.15.81:3300/0 conn(0x7f6b84149830 0x7f6b840a8e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:26.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.591+0000 7f6b8b000640 1 -- 172.21.15.28:0/2631168575 >> 172.21.15.28:0/2631168575 conn(0x7f6b840a8320 msgr2=0x7f6b840a8730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:26.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.592+0000 7f6b8b000640 1 -- 172.21.15.28:0/2631168575 shutdown_connections 2024-02-21T23:09:26.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.592+0000 7f6b8b000640 1 -- 172.21.15.28:0/2631168575 wait complete. 2024-02-21T23:09:26.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.594+0000 7f6b8b000640 1 Processor -- start 2024-02-21T23:09:26.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.594+0000 7f6b8b000640 1 -- start start 2024-02-21T23:09:26.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.595+0000 7f6b8b000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b8406d410 0x7f6b840a9890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:26.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.595+0000 7f6b8b000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b840c1420 0x7f6b84144450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:26.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.595+0000 7f6b8b000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6b84149830 0x7f6b84144990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:26.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.595+0000 7f6b8b000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b840c6190 con 0x7f6b8406d410 2024-02-21T23:09:26.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.595+0000 7f6b8b000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b840c5560 con 0x7f6b840c1420 2024-02-21T23:09:26.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.595+0000 7f6b8b000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b840c6310 con 0x7f6b84149830 2024-02-21T23:09:26.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.595+0000 7f6b89c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b8406d410 0x7f6b840a9890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:26.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.595+0000 7f6b89c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b8406d410 0x7f6b840a9890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57216/0 (socket says 172.21.15.28:57216) 2024-02-21T23:09:26.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.595+0000 7f6b89c00640 1 -- 172.21.15.28:0/3426321946 learned_addr learned my addr 172.21.15.28:0/3426321946 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:26.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.596+0000 7f6b89200640 1 --2- 172.21.15.28:0/3426321946 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b840c1420 0x7f6b84144450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:26.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.596+0000 7f6b8a600640 1 --2- 172.21.15.28:0/3426321946 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6b84149830 0x7f6b84144990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:26.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.596+0000 7f6b89c00640 1 -- 172.21.15.28:0/3426321946 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6b84149830 msgr2=0x7f6b84144990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:26.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.596+0000 7f6b89c00640 1 --2- 172.21.15.28:0/3426321946 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6b84149830 0x7f6b84144990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:26.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.596+0000 7f6b89c00640 1 -- 172.21.15.28:0/3426321946 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b840c1420 msgr2=0x7f6b84144450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:26.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.596+0000 7f6b89c00640 1 --2- 172.21.15.28:0/3426321946 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b840c1420 0x7f6b84144450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:26.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.596+0000 7f6b89c00640 1 -- 172.21.15.28:0/3426321946 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b840aa620 con 0x7f6b8406d410 2024-02-21T23:09:26.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.596+0000 7f6b8a600640 1 --2- 172.21.15.28:0/3426321946 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6b84149830 0x7f6b84144990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:09:26.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.596+0000 7f6b89c00640 1 --2- 172.21.15.28:0/3426321946 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b8406d410 0x7f6b840a9890 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7f6b7800d620 tx=0x7f6b7800dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:26.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.596+0000 7f6b81600640 1 -- 172.21.15.28:0/3426321946 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b7800b620 con 0x7f6b8406d410 2024-02-21T23:09:26.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.596+0000 7f6b81600640 1 -- 172.21.15.28:0/3426321946 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b78019070 con 0x7f6b8406d410 2024-02-21T23:09:26.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.596+0000 7f6b8b000640 1 -- 172.21.15.28:0/3426321946 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b840aa900 con 0x7f6b8406d410 2024-02-21T23:09:26.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.596+0000 7f6b81600640 1 -- 172.21.15.28:0/3426321946 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b7800ba00 con 0x7f6b8406d410 2024-02-21T23:09:26.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.597+0000 7f6b8b000640 1 -- 172.21.15.28:0/3426321946 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b84143f30 con 0x7f6b8406d410 2024-02-21T23:09:26.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.599+0000 7f6b81600640 1 -- 172.21.15.28:0/3426321946 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6b7800bba0 con 0x7f6b8406d410 2024-02-21T23:09:26.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.600+0000 7f6b81600640 1 --2- 172.21.15.28:0/3426321946 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6b6406b4d0 0x7f6b6406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:26.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.600+0000 7f6b81600640 1 -- 172.21.15.28:0/3426321946 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f6b7808a8a0 con 0x7f6b8406d410 2024-02-21T23:09:26.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.600+0000 7f6b8f47d740 1 Processor -- start 2024-02-21T23:09:26.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.600+0000 7f6b89200640 1 --2- 172.21.15.28:0/3426321946 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6b6406b4d0 0x7f6b6406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:26.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.601+0000 7f6b8f47d740 1 -- start start 2024-02-21T23:09:26.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.602+0000 7f6b8f47d740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b76e1dc020 0x55b76e1dc400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:26.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.602+0000 7f6b8f47d740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55b76e1dd330 0x55b76e218170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:26.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.602+0000 7f6b8f47d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b76e2186b0 0x55b76e216720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:26.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.602+0000 7f6b8f47d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b76e140470 con 0x55b76e2186b0 2024-02-21T23:09:26.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.602+0000 7f6b8f47d740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b76e0e9af0 con 0x55b76e1dc020 2024-02-21T23:09:26.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.602+0000 7f6b8f47d740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b76e13c510 con 0x55b76e1dd330 2024-02-21T23:09:26.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.602+0000 7f6b8a600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b76e2186b0 0x55b76e216720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:26.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.602+0000 7f6b89200640 1 --2- 172.21.15.28:0/3426321946 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6b6406b4d0 0x7f6b6406d990 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f6b70036000 tx=0x7f6b70008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:26.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.602+0000 7f6b8a600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b76e2186b0 0x55b76e216720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57228/0 (socket says 172.21.15.28:57228) 2024-02-21T23:09:26.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.602+0000 7f6b8a600640 1 -- 172.21.15.28:0/2972880536 learned_addr learned my addr 172.21.15.28:0/2972880536 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:26.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.602+0000 7f6b89c00640 1 --2- 172.21.15.28:0/2972880536 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b76e1dc020 0x55b76e1dc400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:26.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.603+0000 7f6b8a600640 1 -- 172.21.15.28:0/2972880536 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55b76e1dd330 msgr2=0x55b76e218170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:26.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.603+0000 7f6b8a600640 1 --2- 172.21.15.28:0/2972880536 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55b76e1dd330 0x55b76e218170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:26.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.603+0000 7f6b8a600640 1 -- 172.21.15.28:0/2972880536 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b76e1dc020 msgr2=0x55b76e1dc400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:26.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.603+0000 7f6b8a600640 1 --2- 172.21.15.28:0/2972880536 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b76e1dc020 0x55b76e1dc400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:26.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.603+0000 7f6b8a600640 1 -- 172.21.15.28:0/2972880536 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b76e0f0d90 con 0x55b76e2186b0 2024-02-21T23:09:26.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.604+0000 7f6b8a600640 1 --2- 172.21.15.28:0/2972880536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b76e2186b0 0x55b76e216720 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c001ea0 tx=0x7f6b7c00d520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11e8567d116b4b27 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:26.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.604+0000 7f6b76000640 1 -- 172.21.15.28:0/2972880536 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b7c018070 con 0x55b76e2186b0 2024-02-21T23:09:26.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.604+0000 7f6b76000640 1 -- 172.21.15.28:0/2972880536 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b7c0020b0 con 0x55b76e2186b0 2024-02-21T23:09:26.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.604+0000 7f6b76000640 1 -- 172.21.15.28:0/2972880536 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b7c003bb0 con 0x55b76e2186b0 2024-02-21T23:09:26.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.605+0000 7f6b8f47d740 1 -- 172.21.15.28:0/2972880536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b76e2186b0 msgr2=0x55b76e216720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:26.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.605+0000 7f6b8f47d740 1 --2- 172.21.15.28:0/2972880536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b76e2186b0 0x55b76e216720 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c001ea0 tx=0x7f6b7c00d520 comp rx=0 tx=0).stop 2024-02-21T23:09:26.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.605+0000 7f6b8f47d740 1 -- 172.21.15.28:0/2972880536 shutdown_connections 2024-02-21T23:09:26.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.605+0000 7f6b8f47d740 1 --2- 172.21.15.28:0/2972880536 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55b76e1dd330 0x55b76e218170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:26.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.605+0000 7f6b8f47d740 1 --2- 172.21.15.28:0/2972880536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b76e2186b0 0x55b76e216720 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:26.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.605+0000 7f6b8f47d740 1 --2- 172.21.15.28:0/2972880536 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b76e1dc020 0x55b76e1dc400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:26.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.605+0000 7f6b8f47d740 1 -- 172.21.15.28:0/2972880536 >> 172.21.15.28:0/2972880536 conn(0x55b76e1d8970 msgr2=0x55b76e1fd380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:26.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.605+0000 7f6b8f47d740 1 -- 172.21.15.28:0/2972880536 shutdown_connections 2024-02-21T23:09:26.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.605+0000 7f6b8f47d740 1 -- 172.21.15.28:0/2972880536 wait complete. 2024-02-21T23:09:26.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.606+0000 7f6b8f47d740 1 -- start start 2024-02-21T23:09:26.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.606+0000 7f6b8f47d740 1 Processor -- start 2024-02-21T23:09:26.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.607+0000 7f6b8f47d740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55b76e1dd330 0x55b76e21cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:26.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.607+0000 7f6b8f47d740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b76e2186b0 0x55b76e21e400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:26.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.607+0000 7f6b8f47d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b76e1f8840 0x55b76e1f7c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:26.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.607+0000 7f6b8f47d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b76e0df040 con 0x55b76e1f8840 2024-02-21T23:09:26.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.607+0000 7f6b8f47d740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b76e1fda90 con 0x55b76e2186b0 2024-02-21T23:09:26.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.607+0000 7f6b8f47d740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b76e228c20 con 0x55b76e1dd330 2024-02-21T23:09:26.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.607+0000 7f6b8f47d740 10 client.0 _ll_register_callbacks cb 0x55b76e0fc920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:09:26.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.608+0000 7f6b8a600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b76e1f8840 0x55b76e1f7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:26.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.608+0000 7f6b8a600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b76e1f8840 0x55b76e1f7c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57240/0 (socket says 172.21.15.28:57240) 2024-02-21T23:09:26.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.608+0000 7f6b8a600640 1 -- 172.21.15.28:0/2591541398 learned_addr learned my addr 172.21.15.28:0/2591541398 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:26.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.608+0000 7f6b89c00640 1 --2- 172.21.15.28:0/2591541398 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55b76e1dd330 0x55b76e21cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:26.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.608+0000 7f6b89200640 1 --2- 172.21.15.28:0/2591541398 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b76e2186b0 0x55b76e21e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:26.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.608+0000 7f6b8a600640 1 -- 172.21.15.28:0/2591541398 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55b76e1dd330 msgr2=0x55b76e21cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:26.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.608+0000 7f6b8a600640 1 --2- 172.21.15.28:0/2591541398 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55b76e1dd330 0x55b76e21cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:26.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.608+0000 7f6b8a600640 1 -- 172.21.15.28:0/2591541398 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b76e2186b0 msgr2=0x55b76e21e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:26.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.608+0000 7f6b8a600640 1 --2- 172.21.15.28:0/2591541398 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b76e2186b0 0x55b76e21e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:26.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.608+0000 7f6b8a600640 1 -- 172.21.15.28:0/2591541398 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b76e0feaa0 con 0x55b76e1f8840 2024-02-21T23:09:26.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.608+0000 7f6b89200640 1 --2- 172.21.15.28:0/2591541398 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b76e2186b0 0x55b76e21e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:09:26.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.608+0000 7f6b8a600640 1 --2- 172.21.15.28:0/2591541398 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b76e1f8840 0x55b76e1f7c90 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c016040 tx=0x7f6b7c001250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c0739724c7e9e30 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:26.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.608+0000 7f6b6aa00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:09:26.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.609+0000 7f6b6aa00640 1 -- 172.21.15.28:0/2591541398 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b7c023030 con 0x55b76e1f8840 2024-02-21T23:09:26.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.609+0000 7f6b6aa00640 1 -- 172.21.15.28:0/2591541398 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(139..139 src has 1..139) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f6b7c0050d0 con 0x55b76e1f8840 2024-02-21T23:09:26.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.609+0000 7f6b8f47d740 1 -- 172.21.15.28:0/2591541398 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6b7c0020b0 con 0x55b76e1f8840 2024-02-21T23:09:26.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.609+0000 7f6b6aa00640 1 -- 172.21.15.28:0/2591541398 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=144) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6b7c02c050 con 0x55b76e1f8840 2024-02-21T23:09:26.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.609+0000 7f6b8f47d740 10 client.8595 fetch_fsmap learned FSMap version 144 2024-02-21T23:09:26.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.609+0000 7f6b8f47d740 1 -- 172.21.15.28:0/2591541398 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=144,monmap=2+}) v3 -- 0x55b76de552d0 con 0x55b76e1f8840 2024-02-21T23:09:26.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.609+0000 7f6b6aa00640 1 -- 172.21.15.28:0/2591541398 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b7c0020b0 con 0x55b76e1f8840 2024-02-21T23:09:26.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.609+0000 7f6b6aa00640 1 -- 172.21.15.28:0/2591541398 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 144) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f6b7c02f020 con 0x55b76e1f8840 2024-02-21T23:09:26.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.609+0000 7f6b8f47d740 10 client.8595 fetch_fsmap finished waiting for FSMap version 144 2024-02-21T23:09:26.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.610+0000 7f6b8f47d740 10 client.8595 resolve_mds: resolved name 'b' to daemon mds.b 2024-02-21T23:09:26.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.610+0000 7f6b8f47d740 20 client.8595 populate_metadata read hostname 'smithi028' 2024-02-21T23:09:26.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.610+0000 7f6b8f47d740 1 --2- 172.21.15.28:0/2591541398 >> [v2:172.21.15.81:6834/4241906475,v1:172.21.15.81:6836/4241906475] conn(0x55b76e22ba50 0x55b76e22be30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:26.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.610+0000 7f6b8f47d740 4 client.8595 mds_command: new command op to 6140 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-21T23:09:26.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.610+0000 7f6b8f47d740 1 -- 172.21.15.28:0/2591541398 --> [v2:172.21.15.81:6834/4241906475,v1:172.21.15.81:6836/4241906475] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b76e14d9b0 con 0x55b76e22ba50 2024-02-21T23:09:26.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.610+0000 7f6b89c00640 1 --2- 172.21.15.28:0/2591541398 >> [v2:172.21.15.81:6834/4241906475,v1:172.21.15.81:6836/4241906475] conn(0x55b76e22ba50 0x55b76e22be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:26.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.611+0000 7f6b89c00640 1 --2- 172.21.15.28:0/2591541398 >> [v2:172.21.15.81:6834/4241906475,v1:172.21.15.81:6836/4241906475] conn(0x55b76e22ba50 0x55b76e22be30 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=61abdd2f2de1fe8 server_cookie=7b28160c1ce08bb0 in_seq=0 out_seq=0 2024-02-21T23:09:26.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.611+0000 7f6b6aa00640 10 client.8595 ms_handle_connect on v2:172.21.15.81:6834/4241906475 2024-02-21T23:09:26.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.613+0000 7f6b6aa00640 1 -- 172.21.15.28:0/2591541398 <== mds.0 v2:172.21.15.81:6834/4241906475 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55b76e14d9b0 con 0x55b76e22ba50 2024-02-21T23:09:26.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.613+0000 7f6b6aa00640 10 client.8595 handle_command_reply: tid=0 2024-02-21T23:09:26.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.613+0000 7f6b8f47d740 1 client.8595 shutdown 2024-02-21T23:09:26.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.613+0000 7f6b8f47d740 1 -- 172.21.15.28:0/2591541398 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b76e1f8840 msgr2=0x55b76e1f7c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:26.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.613+0000 7f6b8f47d740 1 --2- 172.21.15.28:0/2591541398 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b76e1f8840 0x55b76e1f7c90 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c016040 tx=0x7f6b7c001250 comp rx=0 tx=0).stop 2024-02-21T23:09:26.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.613+0000 7f6b8f47d740 1 -- 172.21.15.28:0/2591541398 shutdown_connections 2024-02-21T23:09:26.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.613+0000 7f6b8f47d740 1 --2- 172.21.15.28:0/2591541398 >> [v2:172.21.15.81:6834/4241906475,v1:172.21.15.81:6836/4241906475] conn(0x55b76e22ba50 0x55b76e22be30 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:26.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.613+0000 7f6b8f47d740 1 --2- 172.21.15.28:0/2591541398 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b76e1f8840 0x55b76e1f7c90 unknown :-1 s=CLOSED pgs=1575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:26.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.613+0000 7f6b8f47d740 1 --2- 172.21.15.28:0/2591541398 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b76e2186b0 0x55b76e21e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:26.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.613+0000 7f6b8f47d740 1 --2- 172.21.15.28:0/2591541398 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55b76e1dd330 0x55b76e21cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:26.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.613+0000 7f6b8f47d740 1 -- 172.21.15.28:0/2591541398 >> 172.21.15.28:0/2591541398 conn(0x55b76e1dc020 msgr2=0x55b76e216ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:26.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.613+0000 7f6b6aa00640 0 client.8595 ms_handle_reset on v2:172.21.15.81:6834/4241906475 2024-02-21T23:09:26.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.613+0000 7f6b8f47d740 1 -- 172.21.15.28:0/2591541398 shutdown_connections 2024-02-21T23:09:26.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.613+0000 7f6b8f47d740 1 -- 172.21.15.28:0/2591541398 wait complete. 2024-02-21T23:09:26.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.614+0000 7f6b8f47d740 20 client.8595 trim_cache size 0 max 16384 2024-02-21T23:09:26.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.632+0000 7f6b8f47d740 1 Processor -- start 2024-02-21T23:09:26.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.632+0000 7f6b8f47d740 1 -- start start 2024-02-21T23:09:26.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.633+0000 7f6b8f47d740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55b76e1dd330 0x55b76e228210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:26.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.633+0000 7f6b8f47d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b76e1f8840 0x55b76e21cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:26.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.633+0000 7f6b8f47d740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b76e2186b0 0x55b76e21d220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:26.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.633+0000 7f6b8f47d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b841f6dc0 con 0x55b76e1f8840 2024-02-21T23:09:26.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.633+0000 7f6b8f47d740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b84140030 con 0x55b76e2186b0 2024-02-21T23:09:26.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.633+0000 7f6b8f47d740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b841ef080 con 0x55b76e1dd330 2024-02-21T23:09:26.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.633+0000 7f6b89200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b76e1f8840 0x55b76e21cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:26.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.634+0000 7f6b89200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b76e1f8840 0x55b76e21cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57248/0 (socket says 172.21.15.28:57248) 2024-02-21T23:09:26.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.634+0000 7f6b89200640 1 -- 172.21.15.28:0/991071471 learned_addr learned my addr 172.21.15.28:0/991071471 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:26.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.634+0000 7f6b89c00640 1 --2- 172.21.15.28:0/991071471 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55b76e1dd330 0x55b76e228210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:26.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.634+0000 7f6b8a600640 1 --2- 172.21.15.28:0/991071471 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b76e2186b0 0x55b76e21d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:26.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.634+0000 7f6b89200640 1 -- 172.21.15.28:0/991071471 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55b76e1dd330 msgr2=0x55b76e228210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:26.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.634+0000 7f6b89200640 1 --2- 172.21.15.28:0/991071471 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55b76e1dd330 0x55b76e228210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:26.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.634+0000 7f6b89200640 1 -- 172.21.15.28:0/991071471 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b76e2186b0 msgr2=0x55b76e21d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:26.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.634+0000 7f6b89200640 1 --2- 172.21.15.28:0/991071471 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b76e2186b0 0x55b76e21d220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:26.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.634+0000 7f6b89200640 1 -- 172.21.15.28:0/991071471 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b76df1f6b0 con 0x55b76e1f8840 2024-02-21T23:09:26.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.634+0000 7f6b89200640 1 --2- 172.21.15.28:0/991071471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b76e1f8840 0x55b76e21cce0 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7f6b7002c030 tx=0x7f6b70030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=801c8db324c5ddb2 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:26.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.635+0000 7f6b75600640 1 -- 172.21.15.28:0/991071471 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b700427c0 con 0x55b76e1f8840 2024-02-21T23:09:26.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.635+0000 7f6b75600640 1 -- 172.21.15.28:0/991071471 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b70052070 con 0x55b76e1f8840 2024-02-21T23:09:26.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.635+0000 7f6b75600640 1 -- 172.21.15.28:0/991071471 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b70042b60 con 0x55b76e1f8840 2024-02-21T23:09:26.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.635+0000 7f6b8f47d740 1 -- 172.21.15.28:0/991071471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b76e1f8840 msgr2=0x55b76e21cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:26.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.635+0000 7f6b8f47d740 1 --2- 172.21.15.28:0/991071471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b76e1f8840 0x55b76e21cce0 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7f6b7002c030 tx=0x7f6b70030000 comp rx=0 tx=0).stop 2024-02-21T23:09:26.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.635+0000 7f6b8f47d740 1 -- 172.21.15.28:0/991071471 shutdown_connections 2024-02-21T23:09:26.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.635+0000 7f6b8f47d740 1 --2- 172.21.15.28:0/991071471 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b76e2186b0 0x55b76e21d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:26.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.635+0000 7f6b8f47d740 1 --2- 172.21.15.28:0/991071471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b76e1f8840 0x55b76e21cce0 unknown :-1 s=CLOSED pgs=1576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:26.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.635+0000 7f6b8f47d740 1 --2- 172.21.15.28:0/991071471 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55b76e1dd330 0x55b76e228210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:26.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.635+0000 7f6b8f47d740 1 -- 172.21.15.28:0/991071471 >> 172.21.15.28:0/991071471 conn(0x55b76e1dc020 msgr2=0x55b76e1db8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:26.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.635+0000 7f6b8f47d740 1 -- 172.21.15.28:0/991071471 shutdown_connections 2024-02-21T23:09:26.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.635+0000 7f6b8f47d740 1 -- 172.21.15.28:0/991071471 wait complete. 2024-02-21T23:09:26.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.636+0000 7f6b8f47d740 1 -- start start 2024-02-21T23:09:26.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.636+0000 7f6b8f47d740 1 Processor -- start 2024-02-21T23:09:26.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.637+0000 7f6b8f47d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b76e1dd330 0x55b76e223ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:26.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.637+0000 7f6b8f47d740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55b76e2186b0 0x55b76e21d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:26.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.637+0000 7f6b8f47d740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b76e28adb0 0x55b76e284ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:26.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.637+0000 7f6b8f47d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b76e21f5f0 con 0x55b76e1dd330 2024-02-21T23:09:26.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.637+0000 7f6b8f47d740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b76e22b8d0 con 0x55b76e28adb0 2024-02-21T23:09:26.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.637+0000 7f6b8f47d740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b76e21ee60 con 0x55b76e2186b0 2024-02-21T23:09:26.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.637+0000 7f6b8f47d740 10 client.0 _ll_register_callbacks cb 0x7f6b7c001d10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:09:26.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.637+0000 7f6b89c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b76e1dd330 0x55b76e223ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:26.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.638+0000 7f6b89c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b76e1dd330 0x55b76e223ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57264/0 (socket says 172.21.15.28:57264) 2024-02-21T23:09:26.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.638+0000 7f6b89c00640 1 -- 172.21.15.28:0/4080717115 learned_addr learned my addr 172.21.15.28:0/4080717115 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:26.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.638+0000 7f6b89200640 1 --2- 172.21.15.28:0/4080717115 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55b76e2186b0 0x55b76e21d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:26.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.638+0000 7f6b89c00640 1 -- 172.21.15.28:0/4080717115 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55b76e2186b0 msgr2=0x55b76e21d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:26.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.638+0000 7f6b89c00640 1 --2- 172.21.15.28:0/4080717115 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55b76e2186b0 0x55b76e21d0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:26.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.638+0000 7f6b89c00640 1 -- 172.21.15.28:0/4080717115 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b76e28adb0 msgr2=0x55b76e284ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:26.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.638+0000 7f6b89c00640 1 --2- 172.21.15.28:0/4080717115 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b76e28adb0 0x55b76e284ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:26.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.638+0000 7f6b89c00640 1 -- 172.21.15.28:0/4080717115 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b76e21d9d0 con 0x55b76e1dd330 2024-02-21T23:09:26.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.638+0000 7f6b89c00640 1 --2- 172.21.15.28:0/4080717115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b76e1dd330 0x55b76e223ae0 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f6b78058af0 tx=0x7f6b7805f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e8b2b2f8b567a0c server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:26.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.638+0000 7f6b6aa00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:09:26.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.638+0000 7f6b6aa00640 1 -- 172.21.15.28:0/4080717115 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b7805de60 con 0x55b76e1dd330 2024-02-21T23:09:26.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.638+0000 7f6b6aa00640 1 -- 172.21.15.28:0/4080717115 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(139..139 src has 1..139) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f6b7805f0b0 con 0x55b76e1dd330 2024-02-21T23:09:26.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.638+0000 7f6b8f47d740 1 -- 172.21.15.28:0/4080717115 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6b70052070 con 0x55b76e1dd330 2024-02-21T23:09:26.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.639+0000 7f6b6aa00640 1 -- 172.21.15.28:0/4080717115 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=144) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6b7807e050 con 0x55b76e1dd330 2024-02-21T23:09:26.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.639+0000 7f6b8f47d740 10 client.8601 fetch_fsmap learned FSMap version 144 2024-02-21T23:09:26.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.639+0000 7f6b8f47d740 1 -- 172.21.15.28:0/4080717115 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=144,monmap=2+}) v3 -- 0x55b76e1fc3a0 con 0x55b76e1dd330 2024-02-21T23:09:26.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.639+0000 7f6b6aa00640 1 -- 172.21.15.28:0/4080717115 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b70052070 con 0x55b76e1dd330 2024-02-21T23:09:26.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.639+0000 7f6b6aa00640 1 -- 172.21.15.28:0/4080717115 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 144) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f6b78013070 con 0x55b76e1dd330 2024-02-21T23:09:26.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.639+0000 7f6b8f47d740 10 client.8601 fetch_fsmap finished waiting for FSMap version 144 2024-02-21T23:09:26.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.639+0000 7f6b8f47d740 10 client.8601 resolve_mds: resolved name 'b' to daemon mds.b 2024-02-21T23:09:26.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.639+0000 7f6b8f47d740 20 client.8601 populate_metadata read hostname 'smithi028' 2024-02-21T23:09:26.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.639+0000 7f6b8f47d740 1 --2- 172.21.15.28:0/4080717115 >> [v2:172.21.15.81:6834/4241906475,v1:172.21.15.81:6836/4241906475] conn(0x55b76e2244f0 0x55b76e2248d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:26.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.639+0000 7f6b8f47d740 4 client.8601 mds_command: new command op to 6140 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-21T23:09:26.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.639+0000 7f6b8f47d740 1 -- 172.21.15.28:0/4080717115 --> [v2:172.21.15.81:6834/4241906475,v1:172.21.15.81:6836/4241906475] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55b76e147d00 con 0x55b76e2244f0 2024-02-21T23:09:26.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.640+0000 7f6b89200640 1 --2- 172.21.15.28:0/4080717115 >> [v2:172.21.15.81:6834/4241906475,v1:172.21.15.81:6836/4241906475] conn(0x55b76e2244f0 0x55b76e2248d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:26.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.641+0000 7f6b89200640 1 --2- 172.21.15.28:0/4080717115 >> [v2:172.21.15.81:6834/4241906475,v1:172.21.15.81:6836/4241906475] conn(0x55b76e2244f0 0x55b76e2248d0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=25ad7dcd2b86ee34 server_cookie=289c0e3e186684e0 in_seq=0 out_seq=0 2024-02-21T23:09:26.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.641+0000 7f6b6aa00640 10 client.8601 ms_handle_connect on v2:172.21.15.81:6834/4241906475 2024-02-21T23:09:26.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.641+0000 7f6b6aa00640 1 -- 172.21.15.28:0/4080717115 <== mds.0 v2:172.21.15.81:6834/4241906475 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+328 (crc 0 0 0) 0x55b76e147d00 con 0x55b76e2244f0 2024-02-21T23:09:26.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.641+0000 7f6b6aa00640 10 client.8601 handle_command_reply: tid=0 2024-02-21T23:09:26.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.641+0000 7f6b8f47d740 1 client.8601 shutdown 2024-02-21T23:09:26.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.641+0000 7f6b8f47d740 1 -- 172.21.15.28:0/4080717115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b76e1dd330 msgr2=0x55b76e223ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:26.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.641+0000 7f6b8f47d740 1 --2- 172.21.15.28:0/4080717115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b76e1dd330 0x55b76e223ae0 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f6b78058af0 tx=0x7f6b7805f040 comp rx=0 tx=0).stop 2024-02-21T23:09:26.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.641+0000 7f6b8f47d740 1 -- 172.21.15.28:0/4080717115 shutdown_connections 2024-02-21T23:09:26.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.641+0000 7f6b8f47d740 1 --2- 172.21.15.28:0/4080717115 >> [v2:172.21.15.81:6834/4241906475,v1:172.21.15.81:6836/4241906475] conn(0x55b76e2244f0 0x55b76e2248d0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:26.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.641+0000 7f6b8f47d740 1 --2- 172.21.15.28:0/4080717115 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55b76e2186b0 0x55b76e21d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:26.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.642+0000 7f6b8f47d740 1 --2- 172.21.15.28:0/4080717115 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55b76e28adb0 0x55b76e284ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:26.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.642+0000 7f6b8f47d740 1 --2- 172.21.15.28:0/4080717115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b76e1dd330 0x55b76e223ae0 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:26.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.642+0000 7f6b8f47d740 1 -- 172.21.15.28:0/4080717115 >> 172.21.15.28:0/4080717115 conn(0x55b76e1dc020 msgr2=0x55b76e1d7600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:26.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.642+0000 7f6b6aa00640 0 client.8601 ms_handle_reset on v2:172.21.15.81:6834/4241906475 2024-02-21T23:09:26.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.642+0000 7f6b8f47d740 1 -- 172.21.15.28:0/4080717115 shutdown_connections 2024-02-21T23:09:26.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.642+0000 7f6b8f47d740 1 -- 172.21.15.28:0/4080717115 wait complete. 2024-02-21T23:09:26.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.642+0000 7f6b8f47d740 20 client.8601 trim_cache size 0 max 16384 2024-02-21T23:09:26.633 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-21T23:09:26.633 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac", 2024-02-21T23:09:26.633 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-02-21T23:09:26.633 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 6140, 2024-02-21T23:09:26.634 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-02-21T23:09:26.634 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-02-21T23:09:26.634 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-02-21T23:09:26.634 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 11.104440446, 2024-02-21T23:09:26.634 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 144, 2024-02-21T23:09:26.634 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 139, 2024-02-21T23:09:26.634 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 137, 2024-02-21T23:09:26.634 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 35.203307664999997 2024-02-21T23:09:26.634 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-21T23:09:26.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.642+0000 7f6b8b000640 1 -- 172.21.15.28:0/3426321946 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6b6406b4d0 msgr2=0x7f6b6406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:26.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.642+0000 7f6b8b000640 1 --2- 172.21.15.28:0/3426321946 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6b6406b4d0 0x7f6b6406d990 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f6b70036000 tx=0x7f6b70008030 comp rx=0 tx=0).stop 2024-02-21T23:09:26.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.642+0000 7f6b8b000640 1 -- 172.21.15.28:0/3426321946 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b8406d410 msgr2=0x7f6b840a9890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:26.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.642+0000 7f6b8b000640 1 --2- 172.21.15.28:0/3426321946 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b8406d410 0x7f6b840a9890 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7f6b7800d620 tx=0x7f6b7800dae0 comp rx=0 tx=0).stop 2024-02-21T23:09:26.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.643+0000 7f6b8b000640 1 -- 172.21.15.28:0/3426321946 shutdown_connections 2024-02-21T23:09:26.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.643+0000 7f6b8b000640 1 --2- 172.21.15.28:0/3426321946 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6b6406b4d0 0x7f6b6406d990 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:26.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.643+0000 7f6b8b000640 1 --2- 172.21.15.28:0/3426321946 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6b84149830 0x7f6b84144990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:26.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.643+0000 7f6b8b000640 1 --2- 172.21.15.28:0/3426321946 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6b840c1420 0x7f6b84144450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:26.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.643+0000 7f6b8b000640 1 --2- 172.21.15.28:0/3426321946 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b8406d410 0x7f6b840a9890 unknown :-1 s=CLOSED pgs=1573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:26.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.643+0000 7f6b8b000640 1 -- 172.21.15.28:0/3426321946 >> 172.21.15.28:0/3426321946 conn(0x7f6b840a8320 msgr2=0x7f6b84071030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:26.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.643+0000 7f6b8b000640 1 -- 172.21.15.28:0/3426321946 shutdown_connections 2024-02-21T23:09:26.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.643+0000 7f6b8b000640 1 -- 172.21.15.28:0/3426321946 wait complete. 2024-02-21T23:09:26.648 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.b mds_join_fs 2024-02-21T23:09:26.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.869+0000 7f8ebd200640 1 Processor -- start 2024-02-21T23:09:26.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.869+0000 7f8ebd200640 1 -- start start 2024-02-21T23:09:26.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.871+0000 7f8ebd200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f8eb813e0c0 0x7f8eb80c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:26.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.871+0000 7f8ebd200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8eb80c76f0 0x7f8eb806c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:26.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.871+0000 7f8ebd200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8eb8158550 con 0x7f8eb806dbc0 2024-02-21T23:09:26.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.871+0000 7f8ebd200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8eb80c89a0 con 0x7f8eb80c76f0 2024-02-21T23:09:26.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.871+0000 7f8ebd200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8eb81587d0 con 0x7f8eb813e0c0 2024-02-21T23:09:26.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.871+0000 7f8eb7e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8eb80c76f0 0x7f8eb806c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:26.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.872+0000 7f8eb7e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8eb80c76f0 0x7f8eb806c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57270/0 (socket says 172.21.15.28:57270) 2024-02-21T23:09:26.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.872+0000 7f8eb7e00640 1 -- 172.21.15.28:0/337450989 learned_addr learned my addr 172.21.15.28:0/337450989 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:26.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.872+0000 7f8eb7400640 1 --2- 172.21.15.28:0/337450989 >> v2:172.21.15.81:3300/0 conn(0x7f8eb813e0c0 0x7f8eb80c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:26.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.872+0000 7f8eb7e00640 1 -- 172.21.15.28:0/337450989 >> v2:172.21.15.81:3300/0 conn(0x7f8eb813e0c0 msgr2=0x7f8eb80c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:26.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.872+0000 7f8eb7e00640 1 --2- 172.21.15.28:0/337450989 >> v2:172.21.15.81:3300/0 conn(0x7f8eb813e0c0 0x7f8eb80c71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:26.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.872+0000 7f8eb7e00640 1 -- 172.21.15.28:0/337450989 >> v1:172.21.15.81:6789/0 conn(0x7f8eb806dbc0 legacy=0x7f8eb806c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:26.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.872+0000 7f8eb7e00640 1 -- 172.21.15.28:0/337450989 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eb8158ab0 con 0x7f8eb80c76f0 2024-02-21T23:09:26.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.873+0000 7f8eb7e00640 1 --2- 172.21.15.28:0/337450989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8eb80c76f0 0x7f8eb806c990 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f8eb80ad180 tx=0x7f8eb002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a24d78248ee21bb9 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:26.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.873+0000 7f8eb6000640 1 -- 172.21.15.28:0/337450989 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb0035020 con 0x7f8eb80c76f0 2024-02-21T23:09:26.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.874+0000 7f8eb6000640 1 -- 172.21.15.28:0/337450989 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8eb0031870 con 0x7f8eb80c76f0 2024-02-21T23:09:26.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.874+0000 7f8eb6000640 1 -- 172.21.15.28:0/337450989 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb0031c40 con 0x7f8eb80c76f0 2024-02-21T23:09:26.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.874+0000 7f8ebd200640 1 -- 172.21.15.28:0/337450989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8eb80c76f0 msgr2=0x7f8eb806c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:26.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.874+0000 7f8ebd200640 1 --2- 172.21.15.28:0/337450989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8eb80c76f0 0x7f8eb806c990 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f8eb80ad180 tx=0x7f8eb002b910 comp rx=0 tx=0).stop 2024-02-21T23:09:26.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.874+0000 7f8ebd200640 1 -- 172.21.15.28:0/337450989 shutdown_connections 2024-02-21T23:09:26.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.874+0000 7f8ebd200640 1 --2- 172.21.15.28:0/337450989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8eb80c76f0 0x7f8eb806c990 unknown :-1 s=CLOSED pgs=1578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:26.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.874+0000 7f8ebd200640 1 --2- 172.21.15.28:0/337450989 >> v2:172.21.15.81:3300/0 conn(0x7f8eb813e0c0 0x7f8eb80c71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:26.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.874+0000 7f8ebd200640 1 -- 172.21.15.28:0/337450989 >> 172.21.15.28:0/337450989 conn(0x7f8eb8073110 msgr2=0x7f8eb80ac490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:26.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.875+0000 7f8ebd200640 1 -- 172.21.15.28:0/337450989 shutdown_connections 2024-02-21T23:09:26.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.875+0000 7f8ebd200640 1 -- 172.21.15.28:0/337450989 wait complete. 2024-02-21T23:09:26.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.877+0000 7f8ebd200640 1 Processor -- start 2024-02-21T23:09:26.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.877+0000 7f8ebd200640 1 -- start start 2024-02-21T23:09:26.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.878+0000 7f8ebd200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8eb806dbc0 0x7f8eb8152470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:26.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.878+0000 7f8ebd200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8eb80c76f0 0x7f8eb813ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:26.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.878+0000 7f8ebd200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8eb813e0c0 0x7f8eb81df6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:26.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.878+0000 7f8ebd200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb806ced0 con 0x7f8eb813e0c0 2024-02-21T23:09:26.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.878+0000 7f8ebd200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb8159070 con 0x7f8eb80c76f0 2024-02-21T23:09:26.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.878+0000 7f8ebd200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb813e520 con 0x7f8eb806dbc0 2024-02-21T23:09:26.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.879+0000 7f8eb7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8eb813e0c0 0x7f8eb81df6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:26.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.879+0000 7f8eb7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8eb813e0c0 0x7f8eb81df6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57274/0 (socket says 172.21.15.28:57274) 2024-02-21T23:09:26.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.879+0000 7f8eb7e00640 1 -- 172.21.15.28:0/1090273406 learned_addr learned my addr 172.21.15.28:0/1090273406 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:26.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.879+0000 7f8eb6a00640 1 --2- 172.21.15.28:0/1090273406 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8eb80c76f0 0x7f8eb813ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:26.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.879+0000 7f8eb7400640 1 --2- 172.21.15.28:0/1090273406 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8eb806dbc0 0x7f8eb8152470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:26.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.879+0000 7f8eb7e00640 1 -- 172.21.15.28:0/1090273406 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8eb806dbc0 msgr2=0x7f8eb8152470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:26.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.879+0000 7f8eb7e00640 1 --2- 172.21.15.28:0/1090273406 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8eb806dbc0 0x7f8eb8152470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:26.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.879+0000 7f8eb7e00640 1 -- 172.21.15.28:0/1090273406 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8eb80c76f0 msgr2=0x7f8eb813ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:26.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.879+0000 7f8eb7e00640 1 --2- 172.21.15.28:0/1090273406 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8eb80c76f0 0x7f8eb813ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:26.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.879+0000 7f8eb7e00640 1 -- 172.21.15.28:0/1090273406 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eb81dea00 con 0x7f8eb813e0c0 2024-02-21T23:09:26.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.879+0000 7f8eb7400640 1 --2- 172.21.15.28:0/1090273406 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8eb806dbc0 0x7f8eb8152470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:09:26.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.879+0000 7f8eb7e00640 1 --2- 172.21.15.28:0/1090273406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8eb813e0c0 0x7f8eb81df6f0 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f8eb0009960 tx=0x7f8eb00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:26.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.880+0000 7f8eaf400640 1 -- 172.21.15.28:0/1090273406 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb0035040 con 0x7f8eb813e0c0 2024-02-21T23:09:26.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.880+0000 7f8eaf400640 1 -- 172.21.15.28:0/1090273406 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8eb0030d00 con 0x7f8eb813e0c0 2024-02-21T23:09:26.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.880+0000 7f8ebd200640 1 -- 172.21.15.28:0/1090273406 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8eb81e0890 con 0x7f8eb813e0c0 2024-02-21T23:09:26.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.880+0000 7f8eaf400640 1 -- 172.21.15.28:0/1090273406 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb00456a0 con 0x7f8eb813e0c0 2024-02-21T23:09:26.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.880+0000 7f8ebd200640 1 -- 172.21.15.28:0/1090273406 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8eb80ae300 con 0x7f8eb813e0c0 2024-02-21T23:09:26.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.881+0000 7f8ebd200640 1 -- 172.21.15.28:0/1090273406 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8eb81e06a0 con 0x7f8eb813e0c0 2024-02-21T23:09:26.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.882+0000 7f8eaf400640 1 -- 172.21.15.28:0/1090273406 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8eb0038020 con 0x7f8eb813e0c0 2024-02-21T23:09:26.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.883+0000 7f8eaf400640 1 --2- 172.21.15.28:0/1090273406 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8e9406b500 0x7f8e9406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:26.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.884+0000 7f8eaf400640 1 -- 172.21.15.28:0/1090273406 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f8eb00ac780 con 0x7f8eb813e0c0 2024-02-21T23:09:26.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.884+0000 7f8eb7400640 1 --2- 172.21.15.28:0/1090273406 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8e9406b500 0x7f8e9406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:26.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.885+0000 7f8eb7400640 1 --2- 172.21.15.28:0/1090273406 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8e9406b500 0x7f8e9406d9c0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f8eb81ecdd0 tx=0x7f8ea8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:26.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:26.888+0000 7f8eaf400640 1 -- 172.21.15.28:0/1090273406 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f8eb007b3d0 con 0x7f8eb813e0c0 2024-02-21T23:09:27.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.091+0000 7f8ebd200640 1 -- 172.21.15.28:0/1090273406 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.b", "name": "mds_join_fs"} v 0) v1 -- 0x7f8eb80c4b50 con 0x7f8eb813e0c0 2024-02-21T23:09:27.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.096+0000 7f8eaf400640 1 -- 172.21.15.28:0/1090273406 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.b", "name": "mds_join_fs"}]=0 v48) v1 ==== 96+0+0 (secure 0 0 0) 0x7f8eb007acd0 con 0x7f8eb813e0c0 2024-02-21T23:09:27.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.099+0000 7f8ebd200640 1 -- 172.21.15.28:0/1090273406 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8e9406b500 msgr2=0x7f8e9406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:27.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.099+0000 7f8ebd200640 1 --2- 172.21.15.28:0/1090273406 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8e9406b500 0x7f8e9406d9c0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f8eb81ecdd0 tx=0x7f8ea8008040 comp rx=0 tx=0).stop 2024-02-21T23:09:27.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.099+0000 7f8ebd200640 1 -- 172.21.15.28:0/1090273406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8eb813e0c0 msgr2=0x7f8eb81df6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:27.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.099+0000 7f8ebd200640 1 --2- 172.21.15.28:0/1090273406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8eb813e0c0 0x7f8eb81df6f0 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f8eb0009960 tx=0x7f8eb00046c0 comp rx=0 tx=0).stop 2024-02-21T23:09:27.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.099+0000 7f8ebd200640 1 -- 172.21.15.28:0/1090273406 shutdown_connections 2024-02-21T23:09:27.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.099+0000 7f8ebd200640 1 --2- 172.21.15.28:0/1090273406 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8e9406b500 0x7f8e9406d9c0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:27.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.099+0000 7f8ebd200640 1 --2- 172.21.15.28:0/1090273406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8eb813e0c0 0x7f8eb81df6f0 unknown :-1 s=CLOSED pgs=1579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:27.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.099+0000 7f8ebd200640 1 --2- 172.21.15.28:0/1090273406 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8eb80c76f0 0x7f8eb813ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:27.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.099+0000 7f8ebd200640 1 --2- 172.21.15.28:0/1090273406 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8eb806dbc0 0x7f8eb8152470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:27.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.099+0000 7f8ebd200640 1 -- 172.21.15.28:0/1090273406 >> 172.21.15.28:0/1090273406 conn(0x7f8eb8073110 msgr2=0x7f8eb80c5020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:27.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.099+0000 7f8ebd200640 1 -- 172.21.15.28:0/1090273406 shutdown_connections 2024-02-21T23:09:27.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.099+0000 7f8ebd200640 1 -- 172.21.15.28:0/1090273406 wait complete. 2024-02-21T23:09:27.107 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:09:27.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.338+0000 7f7690c00640 1 Processor -- start 2024-02-21T23:09:27.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.339+0000 7f7690c00640 1 -- start start 2024-02-21T23:09:27.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.340+0000 7f7690c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f768c06ee40 0x7f768c0a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:27.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.340+0000 7f7690c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f768c156af0 0x7f768c06dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:27.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.340+0000 7f7690c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f768c0c2760 con 0x7f768c150450 2024-02-21T23:09:27.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.340+0000 7f7690c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f768c0c22c0 con 0x7f768c156af0 2024-02-21T23:09:27.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.340+0000 7f7690c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f768c156660 con 0x7f768c06ee40 2024-02-21T23:09:27.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.341+0000 7f768b400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f768c150450 0x7f768c1536f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:44974/0 (socket says 172.21.15.28:44974) 2024-02-21T23:09:27.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.341+0000 7f768b400640 1 -- 172.21.15.28:0/2142539215 learned_addr learned my addr 172.21.15.28:0/2142539215 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:09:27.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.341+0000 7f768be00640 1 --2- 172.21.15.28:0/2142539215 >> v2:172.21.15.28:3300/0 conn(0x7f768c156af0 0x7f768c06dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:27.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.341+0000 7f768aa00640 1 --2- 172.21.15.28:0/2142539215 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f768c06ee40 0x7f768c0a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:27.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.341+0000 7f768a000640 1 -- 172.21.15.28:0/2142539215 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 120905699 0 0) 0x7f768c0c2760 con 0x7f768c150450 2024-02-21T23:09:27.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.341+0000 7f768a000640 1 -- 172.21.15.28:0/2142539215 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7670003520 con 0x7f768c150450 2024-02-21T23:09:27.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.341+0000 7f768be00640 1 -- 172.21.15.28:0/2142539215 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f768c06ee40 msgr2=0x7f768c0a5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:27.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.341+0000 7f768be00640 1 --2- 172.21.15.28:0/2142539215 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f768c06ee40 0x7f768c0a5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:27.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.341+0000 7f768be00640 1 -- 172.21.15.28:0/2142539215 >> v1:172.21.15.28:6789/0 conn(0x7f768c150450 legacy=0x7f768c1536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:27.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.341+0000 7f768be00640 1 -- 172.21.15.28:0/2142539215 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f768c156310 con 0x7f768c156af0 2024-02-21T23:09:27.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.342+0000 7f768be00640 1 --2- 172.21.15.28:0/2142539215 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f768c156af0 0x7f768c06dff0 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f76840097a0 tx=0x7f768402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bc1cef032ed4e55 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:27.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.343+0000 7f768a000640 1 -- 172.21.15.28:0/2142539215 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7684035020 con 0x7f768c156af0 2024-02-21T23:09:27.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.343+0000 7f768a000640 1 -- 172.21.15.28:0/2142539215 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7684031810 con 0x7f768c156af0 2024-02-21T23:09:27.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.343+0000 7f768a000640 1 -- 172.21.15.28:0/2142539215 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7684031bc0 con 0x7f768c156af0 2024-02-21T23:09:27.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.343+0000 7f7690c00640 1 -- 172.21.15.28:0/2142539215 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f768c156af0 msgr2=0x7f768c06dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:27.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.343+0000 7f7690c00640 1 --2- 172.21.15.28:0/2142539215 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f768c156af0 0x7f768c06dff0 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f76840097a0 tx=0x7f768402b8b0 comp rx=0 tx=0).stop 2024-02-21T23:09:27.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.344+0000 7f7690c00640 1 -- 172.21.15.28:0/2142539215 shutdown_connections 2024-02-21T23:09:27.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.344+0000 7f7690c00640 1 --2- 172.21.15.28:0/2142539215 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f768c156af0 0x7f768c06dff0 unknown :-1 s=CLOSED pgs=1581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:27.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.344+0000 7f7690c00640 1 --2- 172.21.15.28:0/2142539215 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f768c06ee40 0x7f768c0a5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:27.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.344+0000 7f7690c00640 1 -- 172.21.15.28:0/2142539215 >> 172.21.15.28:0/2142539215 conn(0x7f768c06b630 msgr2=0x7f768c0c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:27.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.344+0000 7f7690c00640 1 -- 172.21.15.28:0/2142539215 shutdown_connections 2024-02-21T23:09:27.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.344+0000 7f7690c00640 1 -- 172.21.15.28:0/2142539215 wait complete. 2024-02-21T23:09:27.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.345+0000 7f7690c00640 1 Processor -- start 2024-02-21T23:09:27.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.346+0000 7f7690c00640 1 -- start start 2024-02-21T23:09:27.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.346+0000 7f7690c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f768c06ee40 0x7f768c0bfba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:27.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.347+0000 7f7690c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f768c150450 0x7f768c145840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:27.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.347+0000 7f7690c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f768c156af0 0x7f768c145d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:27.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.347+0000 7f7690c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f768c156f80 con 0x7f768c156af0 2024-02-21T23:09:27.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.347+0000 7f7690c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f768c0c2cb0 con 0x7f768c06ee40 2024-02-21T23:09:27.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.347+0000 7f7690c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f768c157100 con 0x7f768c150450 2024-02-21T23:09:27.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.347+0000 7f768be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f768c156af0 0x7f768c145d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:27.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.347+0000 7f768be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f768c156af0 0x7f768c145d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57300/0 (socket says 172.21.15.28:57300) 2024-02-21T23:09:27.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.347+0000 7f768be00640 1 -- 172.21.15.28:0/664679154 learned_addr learned my addr 172.21.15.28:0/664679154 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:27.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.347+0000 7f768b400640 1 --2- 172.21.15.28:0/664679154 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f768c06ee40 0x7f768c0bfba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:27.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.347+0000 7f768aa00640 1 --2- 172.21.15.28:0/664679154 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f768c150450 0x7f768c145840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:27.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.347+0000 7f768be00640 1 -- 172.21.15.28:0/664679154 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f768c150450 msgr2=0x7f768c145840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:27.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.347+0000 7f768be00640 1 --2- 172.21.15.28:0/664679154 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f768c150450 0x7f768c145840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:27.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.347+0000 7f768be00640 1 -- 172.21.15.28:0/664679154 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f768c06ee40 msgr2=0x7f768c0bfba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:27.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.347+0000 7f768be00640 1 --2- 172.21.15.28:0/664679154 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f768c06ee40 0x7f768c0bfba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:27.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.347+0000 7f768be00640 1 -- 172.21.15.28:0/664679154 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f768c0ab830 con 0x7f768c156af0 2024-02-21T23:09:27.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.348+0000 7f768be00640 1 --2- 172.21.15.28:0/664679154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f768c156af0 0x7f768c145d80 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f76840098c0 tx=0x7f76840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:27.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.348+0000 7f7683400640 1 -- 172.21.15.28:0/664679154 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7684035070 con 0x7f768c156af0 2024-02-21T23:09:27.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.348+0000 7f7683400640 1 -- 172.21.15.28:0/664679154 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7684003610 con 0x7f768c156af0 2024-02-21T23:09:27.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.348+0000 7f7683400640 1 -- 172.21.15.28:0/664679154 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76840039e0 con 0x7f768c156af0 2024-02-21T23:09:27.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.348+0000 7f7690c00640 1 -- 172.21.15.28:0/664679154 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f768c0abb10 con 0x7f768c156af0 2024-02-21T23:09:27.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.348+0000 7f7690c00640 1 -- 172.21.15.28:0/664679154 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f768c0c00e0 con 0x7f768c156af0 2024-02-21T23:09:27.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.349+0000 7f7690c00640 1 -- 172.21.15.28:0/664679154 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f768c04f290 con 0x7f768c156af0 2024-02-21T23:09:27.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.350+0000 7f7683400640 1 -- 172.21.15.28:0/664679154 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7684038020 con 0x7f768c156af0 2024-02-21T23:09:27.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.351+0000 7f7683400640 1 --2- 172.21.15.28:0/664679154 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f766c06b4d0 0x7f766c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:27.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.351+0000 7f7683400640 1 -- 172.21.15.28:0/664679154 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f76840ac0e0 con 0x7f768c156af0 2024-02-21T23:09:27.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.352+0000 7f768b400640 1 --2- 172.21.15.28:0/664679154 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f766c06b4d0 0x7f766c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:27.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.353+0000 7f768b400640 1 --2- 172.21.15.28:0/664679154 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f766c06b4d0 0x7f766c06d990 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f767c005880 tx=0x7f767c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:27.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.356+0000 7f7683400640 1 -- 172.21.15.28:0/664679154 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f768407ad50 con 0x7f768c156af0 2024-02-21T23:09:27.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.601+0000 7f7690c00640 1 -- 172.21.15.28:0/664679154 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f768c0c1f10 con 0x7f768c156af0 2024-02-21T23:09:27.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.602+0000 7f7683400640 1 -- 172.21.15.28:0/664679154 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) v1 ==== 95+0+4951 (secure 0 0 0) 0x7f7684097e20 con 0x7f768c156af0 2024-02-21T23:09:27.590 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:09:27.590 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":144,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6143,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":7,"addr":"172.21.15.81:6837/2095908433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":2095908433},{"type":"v1","addr":"172.21.15.81:6837","nonce":2095908433}]},"join_fscid":10,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":140},{"gid":8370,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/3934548735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3934548735},{"type":"v1","addr":"172.21.15.28:6833","nonce":3934548735}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":133},{"gid":8556,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2100132822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2100132822},{"type":"v1","addr":"172.21.15.28:6835","nonce":2100132822}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":142}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:09:03.869294+0000","modified":"2024-02-21T23:09:17.546876+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":137,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6140},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6140":{"gid":6140,"name":"b","rank":0,"incarnation":141,"state":"up:active","state_seq":10,"addr":"172.21.15.81:6836/4241906475","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":4241906475},{"type":"v1","addr":"172.21.15.81:6836","nonce":4241906475}]},"join_fscid":10,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-02-21T23:09:27.590 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 144 2024-02-21T23:09:27.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.605+0000 7f7690c00640 1 -- 172.21.15.28:0/664679154 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f766c06b4d0 msgr2=0x7f766c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:27.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.605+0000 7f7690c00640 1 --2- 172.21.15.28:0/664679154 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f766c06b4d0 0x7f766c06d990 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f767c005880 tx=0x7f767c009040 comp rx=0 tx=0).stop 2024-02-21T23:09:27.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.605+0000 7f7690c00640 1 -- 172.21.15.28:0/664679154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f768c156af0 msgr2=0x7f768c145d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:27.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.605+0000 7f7690c00640 1 --2- 172.21.15.28:0/664679154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f768c156af0 0x7f768c145d80 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f76840098c0 tx=0x7f76840046c0 comp rx=0 tx=0).stop 2024-02-21T23:09:27.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.605+0000 7f7690c00640 1 -- 172.21.15.28:0/664679154 shutdown_connections 2024-02-21T23:09:27.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.605+0000 7f7690c00640 1 --2- 172.21.15.28:0/664679154 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f766c06b4d0 0x7f766c06d990 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:27.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.605+0000 7f7690c00640 1 --2- 172.21.15.28:0/664679154 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f768c150450 0x7f768c145840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:27.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.605+0000 7f7690c00640 1 --2- 172.21.15.28:0/664679154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f768c156af0 0x7f768c145d80 unknown :-1 s=CLOSED pgs=1582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:27.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.605+0000 7f7690c00640 1 --2- 172.21.15.28:0/664679154 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f768c06ee40 0x7f768c0bfba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:27.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.605+0000 7f7690c00640 1 -- 172.21.15.28:0/664679154 >> 172.21.15.28:0/664679154 conn(0x7f768c06b630 msgr2=0x7f768c14f2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:27.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.605+0000 7f7690c00640 1 -- 172.21.15.28:0/664679154 shutdown_connections 2024-02-21T23:09:27.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:27.605+0000 7f7690c00640 1 -- 172.21.15.28:0/664679154 wait complete. 2024-02-21T23:09:28.016 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 144, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:09:03.869294+0000', 'modified': '2024-02-21T23:09:17.546876+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 137, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6140}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6140': {'gid': 6140, 'name': 'b', 'rank': 0, 'incarnation': 141, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.81:6836/4241906475', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6834', 'nonce': 4241906475}, {'type': 'v1', 'addr': '172.21.15.81:6836', 'nonce': 4241906475}]}, 'join_fscid': 10, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:09:28.016 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-21T23:09:28.016 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-02-21T23:09:28.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.260+0000 7fc183200640 1 Processor -- start 2024-02-21T23:09:28.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.260+0000 7fc183200640 1 -- start start 2024-02-21T23:09:28.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.261+0000 7fc183200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc1841524c0 0x7fc184155a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:28.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.261+0000 7fc183200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fc1840c5040 0x7fc18406ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:28.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.261+0000 7fc183200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc1840c85c0 0x7fc1840c70d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:28.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.261+0000 7fc183200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc18406df50 con 0x7fc1841524c0 2024-02-21T23:09:28.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.261+0000 7fc183200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc18406d3b0 con 0x7fc1840c5040 2024-02-21T23:09:28.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.261+0000 7fc183200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1840c5520 con 0x7fc1840c85c0 2024-02-21T23:09:28.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.262+0000 7fc181e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc1841524c0 0x7fc184155a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:28.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.262+0000 7fc181e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc1841524c0 0x7fc184155a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57314/0 (socket says 172.21.15.28:57314) 2024-02-21T23:09:28.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.262+0000 7fc181e00640 1 -- 172.21.15.28:0/1130161265 learned_addr learned my addr 172.21.15.28:0/1130161265 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:28.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.262+0000 7fc181400640 1 --2- 172.21.15.28:0/1130161265 >> v2:172.21.15.81:3300/0 conn(0x7fc1840c5040 0x7fc18406ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:28.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.262+0000 7fc182800640 1 --2- 172.21.15.28:0/1130161265 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc1840c85c0 0x7fc1840c70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:28.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.262+0000 7fc181e00640 1 -- 172.21.15.28:0/1130161265 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc1840c85c0 msgr2=0x7fc1840c70d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:28.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.262+0000 7fc181e00640 1 --2- 172.21.15.28:0/1130161265 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc1840c85c0 0x7fc1840c70d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:28.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.262+0000 7fc181e00640 1 -- 172.21.15.28:0/1130161265 >> v2:172.21.15.81:3300/0 conn(0x7fc1840c5040 msgr2=0x7fc18406ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:28.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.262+0000 7fc181e00640 1 --2- 172.21.15.28:0/1130161265 >> v2:172.21.15.81:3300/0 conn(0x7fc1840c5040 0x7fc18406ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:28.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.262+0000 7fc181e00640 1 -- 172.21.15.28:0/1130161265 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1840c5800 con 0x7fc1841524c0 2024-02-21T23:09:28.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.264+0000 7fc181e00640 1 --2- 172.21.15.28:0/1130161265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1841524c0 0x7fc184155a10 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7fc1840cb760 tx=0x7fc17402b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=121e834d5a02f124 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:28.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.264+0000 7fc180a00640 1 -- 172.21.15.28:0/1130161265 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc174038070 con 0x7fc1841524c0 2024-02-21T23:09:28.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.264+0000 7fc180a00640 1 -- 172.21.15.28:0/1130161265 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc17402f050 con 0x7fc1841524c0 2024-02-21T23:09:28.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.264+0000 7fc180a00640 1 -- 172.21.15.28:0/1130161265 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc174031970 con 0x7fc1841524c0 2024-02-21T23:09:28.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.264+0000 7fc183200640 1 -- 172.21.15.28:0/1130161265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1841524c0 msgr2=0x7fc184155a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:28.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.264+0000 7fc183200640 1 --2- 172.21.15.28:0/1130161265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1841524c0 0x7fc184155a10 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7fc1840cb760 tx=0x7fc17402b7f0 comp rx=0 tx=0).stop 2024-02-21T23:09:28.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.265+0000 7fc183200640 1 -- 172.21.15.28:0/1130161265 shutdown_connections 2024-02-21T23:09:28.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.265+0000 7fc183200640 1 --2- 172.21.15.28:0/1130161265 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc1840c85c0 0x7fc1840c70d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:28.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.265+0000 7fc183200640 1 --2- 172.21.15.28:0/1130161265 >> v2:172.21.15.81:3300/0 conn(0x7fc1840c5040 0x7fc18406ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:28.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.265+0000 7fc183200640 1 --2- 172.21.15.28:0/1130161265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1841524c0 0x7fc184155a10 unknown :-1 s=CLOSED pgs=1583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:28.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.265+0000 7fc183200640 1 -- 172.21.15.28:0/1130161265 >> 172.21.15.28:0/1130161265 conn(0x7fc1840ac2b0 msgr2=0x7fc1840ac6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:28.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.265+0000 7fc183200640 1 -- 172.21.15.28:0/1130161265 shutdown_connections 2024-02-21T23:09:28.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.265+0000 7fc183200640 1 -- 172.21.15.28:0/1130161265 wait complete. 2024-02-21T23:09:28.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.266+0000 7fc183200640 1 Processor -- start 2024-02-21T23:09:28.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.266+0000 7fc183200640 1 -- start start 2024-02-21T23:09:28.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.267+0000 7fc183200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1840c5040 0x7fc1840aa090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:28.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.267+0000 7fc183200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc1840c85c0 0x7fc184158fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:28.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.267+0000 7fc183200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc1841524c0 0x7fc184153dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:28.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.267+0000 7fc183200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1841528a0 con 0x7fc1840c5040 2024-02-21T23:09:28.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.267+0000 7fc183200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc18406c950 con 0x7fc1840c85c0 2024-02-21T23:09:28.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.267+0000 7fc183200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1840c8a70 con 0x7fc1841524c0 2024-02-21T23:09:28.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.268+0000 7fc181e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1840c5040 0x7fc1840aa090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:28.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.268+0000 7fc181e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1840c5040 0x7fc1840aa090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57322/0 (socket says 172.21.15.28:57322) 2024-02-21T23:09:28.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.268+0000 7fc181e00640 1 -- 172.21.15.28:0/2564518153 learned_addr learned my addr 172.21.15.28:0/2564518153 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:28.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.268+0000 7fc182800640 1 --2- 172.21.15.28:0/2564518153 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc1841524c0 0x7fc184153dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:28.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.268+0000 7fc181400640 1 --2- 172.21.15.28:0/2564518153 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc1840c85c0 0x7fc184158fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:28.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.268+0000 7fc181e00640 1 -- 172.21.15.28:0/2564518153 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc1841524c0 msgr2=0x7fc184153dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:28.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.268+0000 7fc181e00640 1 --2- 172.21.15.28:0/2564518153 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc1841524c0 0x7fc184153dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:28.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.268+0000 7fc181e00640 1 -- 172.21.15.28:0/2564518153 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc1840c85c0 msgr2=0x7fc184158fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:28.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.268+0000 7fc181e00640 1 --2- 172.21.15.28:0/2564518153 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc1840c85c0 0x7fc184158fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:28.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.268+0000 7fc181e00640 1 -- 172.21.15.28:0/2564518153 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc184155360 con 0x7fc1840c5040 2024-02-21T23:09:28.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.269+0000 7fc181e00640 1 --2- 172.21.15.28:0/2564518153 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1840c5040 0x7fc1840aa090 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7fc174009960 tx=0x7fc1740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:28.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.269+0000 7fc17a000640 1 -- 172.21.15.28:0/2564518153 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc174042020 con 0x7fc1840c5040 2024-02-21T23:09:28.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.269+0000 7fc17a000640 1 -- 172.21.15.28:0/2564518153 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc17403d890 con 0x7fc1840c5040 2024-02-21T23:09:28.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.269+0000 7fc183200640 1 -- 172.21.15.28:0/2564518153 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc184155080 con 0x7fc1840c5040 2024-02-21T23:09:28.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.269+0000 7fc17a000640 1 -- 172.21.15.28:0/2564518153 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc174038040 con 0x7fc1840c5040 2024-02-21T23:09:28.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.269+0000 7fc183200640 1 -- 172.21.15.28:0/2564518153 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1841535c0 con 0x7fc1840c5040 2024-02-21T23:09:28.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.272+0000 7fc17a000640 1 -- 172.21.15.28:0/2564518153 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc174045020 con 0x7fc1840c5040 2024-02-21T23:09:28.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.273+0000 7fc17a000640 1 --2- 172.21.15.28:0/2564518153 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc16406b4d0 0x7fc16406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:28.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.273+0000 7fc17a000640 1 -- 172.21.15.28:0/2564518153 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 6245+0+0 (secure 0 0 0) 0x7fc1740abfe0 con 0x7fc1840c5040 2024-02-21T23:09:28.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.273+0000 7fc18b9ee740 1 Processor -- start 2024-02-21T23:09:28.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.274+0000 7fc181400640 1 --2- 172.21.15.28:0/2564518153 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc16406b4d0 0x7fc16406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:28.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.274+0000 7fc18b9ee740 1 -- start start 2024-02-21T23:09:28.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.275+0000 7fc18b9ee740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x563d520bf020 0x563d520bf400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:28.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.275+0000 7fc18b9ee740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563d520c0330 0x563d520fb170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:28.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.275+0000 7fc18b9ee740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d520fb6b0 0x563d520f9720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:28.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.275+0000 7fc18b9ee740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d52023470 con 0x563d520fb6b0 2024-02-21T23:09:28.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.275+0000 7fc18b9ee740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d51fccaf0 con 0x563d520c0330 2024-02-21T23:09:28.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.275+0000 7fc18b9ee740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x563d5201f510 con 0x563d520bf020 2024-02-21T23:09:28.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.275+0000 7fc182800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d520fb6b0 0x563d520f9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:28.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.275+0000 7fc182800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d520fb6b0 0x563d520f9720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57336/0 (socket says 172.21.15.28:57336) 2024-02-21T23:09:28.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.275+0000 7fc182800640 1 -- 172.21.15.28:0/1166724784 learned_addr learned my addr 172.21.15.28:0/1166724784 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:28.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.275+0000 7fc181e00640 1 --2- 172.21.15.28:0/1166724784 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x563d520bf020 0x563d520bf400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:28.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.275+0000 7fc181400640 1 --2- 172.21.15.28:0/2564518153 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc16406b4d0 0x7fc16406d990 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fc1840bdd50 tx=0x7fc170008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:28.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.275+0000 7fc182800640 1 -- 172.21.15.28:0/1166724784 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x563d520bf020 msgr2=0x563d520bf400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:28.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.275+0000 7fc182800640 1 --2- 172.21.15.28:0/1166724784 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x563d520bf020 0x563d520bf400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:28.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.275+0000 7fc182800640 1 -- 172.21.15.28:0/1166724784 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563d520c0330 msgr2=0x563d520fb170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:28.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.275+0000 7fc181400640 1 --2- 172.21.15.28:0/1166724784 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563d520c0330 0x563d520fb170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:28.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.275+0000 7fc182800640 1 --2- 172.21.15.28:0/1166724784 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563d520c0330 0x563d520fb170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:28.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.275+0000 7fc182800640 1 -- 172.21.15.28:0/1166724784 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563d51fd3d90 con 0x563d520fb6b0 2024-02-21T23:09:28.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.276+0000 7fc182800640 1 --2- 172.21.15.28:0/1166724784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d520fb6b0 0x563d520f9720 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7fc17c007410 tx=0x7fc17c0078d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1688bc95fc079e20 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:28.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.276+0000 7fc16e600640 1 -- 172.21.15.28:0/1166724784 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc17c0147e0 con 0x563d520fb6b0 2024-02-21T23:09:28.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.276+0000 7fc16e600640 1 -- 172.21.15.28:0/1166724784 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc17c01a070 con 0x563d520fb6b0 2024-02-21T23:09:28.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.276+0000 7fc16e600640 1 -- 172.21.15.28:0/1166724784 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc17c014b80 con 0x563d520fb6b0 2024-02-21T23:09:28.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.277+0000 7fc18b9ee740 1 -- 172.21.15.28:0/1166724784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d520fb6b0 msgr2=0x563d520f9720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:28.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.277+0000 7fc18b9ee740 1 --2- 172.21.15.28:0/1166724784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d520fb6b0 0x563d520f9720 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7fc17c007410 tx=0x7fc17c0078d0 comp rx=0 tx=0).stop 2024-02-21T23:09:28.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.277+0000 7fc18b9ee740 1 -- 172.21.15.28:0/1166724784 shutdown_connections 2024-02-21T23:09:28.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.277+0000 7fc18b9ee740 1 --2- 172.21.15.28:0/1166724784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d520fb6b0 0x563d520f9720 unknown :-1 s=CLOSED pgs=1585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:28.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.277+0000 7fc18b9ee740 1 --2- 172.21.15.28:0/1166724784 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563d520c0330 0x563d520fb170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:28.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.277+0000 7fc18b9ee740 1 --2- 172.21.15.28:0/1166724784 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x563d520bf020 0x563d520bf400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:28.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.277+0000 7fc18b9ee740 1 -- 172.21.15.28:0/1166724784 >> 172.21.15.28:0/1166724784 conn(0x563d520bb970 msgr2=0x563d520e0380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:28.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.277+0000 7fc18b9ee740 1 -- 172.21.15.28:0/1166724784 shutdown_connections 2024-02-21T23:09:28.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.277+0000 7fc18b9ee740 1 -- 172.21.15.28:0/1166724784 wait complete. 2024-02-21T23:09:28.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.278+0000 7fc18b9ee740 1 -- start start 2024-02-21T23:09:28.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.278+0000 7fc18b9ee740 1 Processor -- start 2024-02-21T23:09:28.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.279+0000 7fc18b9ee740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d520c0330 0x563d5210e890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:28.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.279+0000 7fc18b9ee740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x563d520fb6b0 0x563d520df0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:28.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.279+0000 7fc18b9ee740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563d520e0690 0x563d520da8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:28.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.279+0000 7fc18b9ee740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d51fc2040 con 0x563d520c0330 2024-02-21T23:09:28.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.279+0000 7fc18b9ee740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d520e0a90 con 0x563d520e0690 2024-02-21T23:09:28.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.279+0000 7fc18b9ee740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x563d520db7c0 con 0x563d520fb6b0 2024-02-21T23:09:28.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.279+0000 7fc18b9ee740 10 client.0 _ll_register_callbacks cb 0x563d51fdf920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:09:28.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.279+0000 7fc181e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d520c0330 0x563d5210e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:28.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.279+0000 7fc181e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d520c0330 0x563d5210e890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57346/0 (socket says 172.21.15.28:57346) 2024-02-21T23:09:28.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.279+0000 7fc181e00640 1 -- 172.21.15.28:0/1607984301 learned_addr learned my addr 172.21.15.28:0/1607984301 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:28.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.279+0000 7fc182800640 1 --2- 172.21.15.28:0/1607984301 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563d520e0690 0x563d520da8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:28.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.280+0000 7fc181400640 1 --2- 172.21.15.28:0/1607984301 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x563d520fb6b0 0x563d520df0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:28.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.280+0000 7fc181e00640 1 -- 172.21.15.28:0/1607984301 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x563d520fb6b0 msgr2=0x563d520df0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:28.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.280+0000 7fc181e00640 1 --2- 172.21.15.28:0/1607984301 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x563d520fb6b0 0x563d520df0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:28.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.280+0000 7fc181e00640 1 -- 172.21.15.28:0/1607984301 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563d520e0690 msgr2=0x563d520da8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:28.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.280+0000 7fc181e00640 1 --2- 172.21.15.28:0/1607984301 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563d520e0690 0x563d520da8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:28.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.280+0000 7fc181e00640 1 -- 172.21.15.28:0/1607984301 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563d51fe1aa0 con 0x563d520c0330 2024-02-21T23:09:28.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.280+0000 7fc182800640 1 --2- 172.21.15.28:0/1607984301 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563d520e0690 0x563d520da8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:09:28.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.280+0000 7fc181e00640 1 --2- 172.21.15.28:0/1607984301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d520c0330 0x563d5210e890 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7fc1740769f0 tx=0x7fc174079fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b709c3ece82277b server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:28.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.280+0000 7fc162a00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:09:28.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.280+0000 7fc162a00640 1 -- 172.21.15.28:0/1607984301 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc174082a10 con 0x563d520c0330 2024-02-21T23:09:28.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.280+0000 7fc162a00640 1 -- 172.21.15.28:0/1607984301 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(139..139 src has 1..139) v4 ==== 6245+0+0 (secure 0 0 0) 0x7fc1740799e0 con 0x563d520c0330 2024-02-21T23:09:28.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.280+0000 7fc18b9ee740 1 -- 172.21.15.28:0/1607984301 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc17c01a070 con 0x563d520c0330 2024-02-21T23:09:28.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.280+0000 7fc162a00640 1 -- 172.21.15.28:0/1607984301 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=144) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc1740aa050 con 0x563d520c0330 2024-02-21T23:09:28.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.280+0000 7fc18b9ee740 10 client.8628 fetch_fsmap learned FSMap version 144 2024-02-21T23:09:28.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.280+0000 7fc18b9ee740 1 -- 172.21.15.28:0/1607984301 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=144,monmap=2+}) v3 -- 0x563d51d382d0 con 0x563d520c0330 2024-02-21T23:09:28.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.281+0000 7fc162a00640 1 -- 172.21.15.28:0/1607984301 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc17c01a070 con 0x563d520c0330 2024-02-21T23:09:28.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.281+0000 7fc162a00640 1 -- 172.21.15.28:0/1607984301 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 144) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fc1740762e0 con 0x563d520c0330 2024-02-21T23:09:28.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.281+0000 7fc18b9ee740 10 client.8628 fetch_fsmap finished waiting for FSMap version 144 2024-02-21T23:09:28.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.281+0000 7fc18b9ee740 10 client.8628 resolve_mds: resolved name 'b' to daemon mds.b 2024-02-21T23:09:28.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.281+0000 7fc18b9ee740 20 client.8628 populate_metadata read hostname 'smithi028' 2024-02-21T23:09:28.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.281+0000 7fc18b9ee740 1 --2- 172.21.15.28:0/1607984301 >> [v2:172.21.15.81:6834/4241906475,v1:172.21.15.81:6836/4241906475] conn(0x563d5210b0a0 0x563d5210b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:28.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.281+0000 7fc18b9ee740 4 client.8628 mds_command: new command op to 6140 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-21T23:09:28.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.281+0000 7fc18b9ee740 1 -- 172.21.15.28:0/1607984301 --> [v2:172.21.15.81:6834/4241906475,v1:172.21.15.81:6836/4241906475] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563d520309b0 con 0x563d5210b0a0 2024-02-21T23:09:28.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.282+0000 7fc181400640 1 --2- 172.21.15.28:0/1607984301 >> [v2:172.21.15.81:6834/4241906475,v1:172.21.15.81:6836/4241906475] conn(0x563d5210b0a0 0x563d5210b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:28.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.282+0000 7fc181400640 1 --2- 172.21.15.28:0/1607984301 >> [v2:172.21.15.81:6834/4241906475,v1:172.21.15.81:6836/4241906475] conn(0x563d5210b0a0 0x563d5210b480 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6bf6245729c0bd22 server_cookie=b91ff27fdc566415 in_seq=0 out_seq=0 2024-02-21T23:09:28.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.282+0000 7fc162a00640 10 client.8628 ms_handle_connect on v2:172.21.15.81:6834/4241906475 2024-02-21T23:09:28.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.285+0000 7fc162a00640 1 -- 172.21.15.28:0/1607984301 <== mds.0 v2:172.21.15.81:6834/4241906475 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x563d520309b0 con 0x563d5210b0a0 2024-02-21T23:09:28.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.285+0000 7fc162a00640 10 client.8628 handle_command_reply: tid=0 2024-02-21T23:09:28.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.285+0000 7fc18b9ee740 1 client.8628 shutdown 2024-02-21T23:09:28.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.285+0000 7fc18b9ee740 1 -- 172.21.15.28:0/1607984301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d520c0330 msgr2=0x563d5210e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:28.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.285+0000 7fc18b9ee740 1 --2- 172.21.15.28:0/1607984301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d520c0330 0x563d5210e890 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7fc1740769f0 tx=0x7fc174079fe0 comp rx=0 tx=0).stop 2024-02-21T23:09:28.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.285+0000 7fc18b9ee740 1 -- 172.21.15.28:0/1607984301 shutdown_connections 2024-02-21T23:09:28.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.285+0000 7fc18b9ee740 1 --2- 172.21.15.28:0/1607984301 >> [v2:172.21.15.81:6834/4241906475,v1:172.21.15.81:6836/4241906475] conn(0x563d5210b0a0 0x563d5210b480 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:28.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.285+0000 7fc18b9ee740 1 --2- 172.21.15.28:0/1607984301 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x563d520fb6b0 0x563d520df0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:28.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.285+0000 7fc18b9ee740 1 --2- 172.21.15.28:0/1607984301 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563d520e0690 0x563d520da8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:28.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.285+0000 7fc18b9ee740 1 --2- 172.21.15.28:0/1607984301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d520c0330 0x563d5210e890 unknown :-1 s=CLOSED pgs=1586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:28.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.285+0000 7fc162a00640 0 client.8628 ms_handle_reset on v2:172.21.15.81:6834/4241906475 2024-02-21T23:09:28.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.285+0000 7fc18b9ee740 1 -- 172.21.15.28:0/1607984301 >> 172.21.15.28:0/1607984301 conn(0x563d520bf020 msgr2=0x563d520f9f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:28.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.285+0000 7fc18b9ee740 1 -- 172.21.15.28:0/1607984301 shutdown_connections 2024-02-21T23:09:28.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.285+0000 7fc18b9ee740 1 -- 172.21.15.28:0/1607984301 wait complete. 2024-02-21T23:09:28.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.285+0000 7fc18b9ee740 20 client.8628 trim_cache size 0 max 16384 2024-02-21T23:09:28.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.304+0000 7fc18b9ee740 1 Processor -- start 2024-02-21T23:09:28.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.304+0000 7fc18b9ee740 1 -- start start 2024-02-21T23:09:28.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.305+0000 7fc18b9ee740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x563d520c0330 0x563d520f5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:28.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.305+0000 7fc18b9ee740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d520e0690 0x563d520da460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:28.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.305+0000 7fc18b9ee740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563d520fb6b0 0x563d520de1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:28.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.305+0000 7fc18b9ee740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1841f6e40 con 0x563d520e0690 2024-02-21T23:09:28.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.305+0000 7fc18b9ee740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1841f6290 con 0x563d520fb6b0 2024-02-21T23:09:28.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.305+0000 7fc18b9ee740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1841f6840 con 0x563d520c0330 2024-02-21T23:09:28.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.305+0000 7fc181400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d520e0690 0x563d520da460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:28.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.305+0000 7fc181400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d520e0690 0x563d520da460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57360/0 (socket says 172.21.15.28:57360) 2024-02-21T23:09:28.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.305+0000 7fc181400640 1 -- 172.21.15.28:0/2808969519 learned_addr learned my addr 172.21.15.28:0/2808969519 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:28.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.305+0000 7fc181400640 1 -- 172.21.15.28:0/2808969519 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x563d520c0330 msgr2=0x563d520f5bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:28.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.305+0000 7fc181e00640 1 --2- 172.21.15.28:0/2808969519 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x563d520c0330 0x563d520f5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:28.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.305+0000 7fc181400640 1 --2- 172.21.15.28:0/2808969519 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x563d520c0330 0x563d520f5bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:28.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.305+0000 7fc181400640 1 -- 172.21.15.28:0/2808969519 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563d520fb6b0 msgr2=0x563d520de1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:28.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.305+0000 7fc181400640 1 --2- 172.21.15.28:0/2808969519 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563d520fb6b0 0x563d520de1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:28.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.305+0000 7fc181400640 1 -- 172.21.15.28:0/2808969519 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563d51e026b0 con 0x563d520e0690 2024-02-21T23:09:28.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.306+0000 7fc181400640 1 --2- 172.21.15.28:0/2808969519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d520e0690 0x563d520da460 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x563d520db3b0 tx=0x7fc170015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a5981b3eb3ffd4e server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:28.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.306+0000 7fc16dc00640 1 -- 172.21.15.28:0/2808969519 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc17002d070 con 0x563d520e0690 2024-02-21T23:09:28.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.306+0000 7fc16dc00640 1 -- 172.21.15.28:0/2808969519 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1700271d0 con 0x563d520e0690 2024-02-21T23:09:28.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.306+0000 7fc16dc00640 1 -- 172.21.15.28:0/2808969519 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc170014660 con 0x563d520e0690 2024-02-21T23:09:28.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.307+0000 7fc18b9ee740 1 -- 172.21.15.28:0/2808969519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d520e0690 msgr2=0x563d520da460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:28.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.307+0000 7fc18b9ee740 1 --2- 172.21.15.28:0/2808969519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d520e0690 0x563d520da460 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x563d520db3b0 tx=0x7fc170015000 comp rx=0 tx=0).stop 2024-02-21T23:09:28.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.307+0000 7fc18b9ee740 1 -- 172.21.15.28:0/2808969519 shutdown_connections 2024-02-21T23:09:28.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.307+0000 7fc18b9ee740 1 --2- 172.21.15.28:0/2808969519 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563d520fb6b0 0x563d520de1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:28.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.307+0000 7fc18b9ee740 1 --2- 172.21.15.28:0/2808969519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d520e0690 0x563d520da460 unknown :-1 s=CLOSED pgs=1587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:28.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.307+0000 7fc18b9ee740 1 --2- 172.21.15.28:0/2808969519 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x563d520c0330 0x563d520f5bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:28.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.307+0000 7fc18b9ee740 1 -- 172.21.15.28:0/2808969519 >> 172.21.15.28:0/2808969519 conn(0x563d520bf020 msgr2=0x563d520be710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:28.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.307+0000 7fc18b9ee740 1 -- 172.21.15.28:0/2808969519 shutdown_connections 2024-02-21T23:09:28.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.307+0000 7fc18b9ee740 1 -- 172.21.15.28:0/2808969519 wait complete. 2024-02-21T23:09:28.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.307+0000 7fc18b9ee740 1 -- start start 2024-02-21T23:09:28.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.308+0000 7fc18b9ee740 1 Processor -- start 2024-02-21T23:09:28.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.309+0000 7fc18b9ee740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x563d520c0330 0x563d520fac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:28.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.309+0000 7fc18b9ee740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563d520e0690 0x563d5216d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:28.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.309+0000 7fc18b9ee740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d520fb6b0 0x563d5211daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:28.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.309+0000 7fc18b9ee740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d51fc2040 con 0x563d520fb6b0 2024-02-21T23:09:28.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.309+0000 7fc18b9ee740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d5210af20 con 0x563d520e0690 2024-02-21T23:09:28.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.309+0000 7fc18b9ee740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x563d520ffe70 con 0x563d520c0330 2024-02-21T23:09:28.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.309+0000 7fc18b9ee740 10 client.0 _ll_register_callbacks cb 0x7fc17407a140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:09:28.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.309+0000 7fc182800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d520fb6b0 0x563d5211daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:28.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.309+0000 7fc182800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d520fb6b0 0x563d5211daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57374/0 (socket says 172.21.15.28:57374) 2024-02-21T23:09:28.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.309+0000 7fc182800640 1 -- 172.21.15.28:0/2477935131 learned_addr learned my addr 172.21.15.28:0/2477935131 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:28.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.309+0000 7fc181e00640 1 --2- 172.21.15.28:0/2477935131 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x563d520c0330 0x563d520fac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:28.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.309+0000 7fc181400640 1 --2- 172.21.15.28:0/2477935131 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563d520e0690 0x563d5216d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:28.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.309+0000 7fc182800640 1 -- 172.21.15.28:0/2477935131 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x563d520c0330 msgr2=0x563d520fac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:28.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.309+0000 7fc182800640 1 --2- 172.21.15.28:0/2477935131 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x563d520c0330 0x563d520fac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:28.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.309+0000 7fc182800640 1 -- 172.21.15.28:0/2477935131 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563d520e0690 msgr2=0x563d5216d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:28.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.309+0000 7fc182800640 1 --2- 172.21.15.28:0/2477935131 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563d520e0690 0x563d5216d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:28.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.309+0000 7fc182800640 1 -- 172.21.15.28:0/2477935131 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563d5216ad60 con 0x563d520fb6b0 2024-02-21T23:09:28.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.310+0000 7fc182800640 1 --2- 172.21.15.28:0/2477935131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d520fb6b0 0x563d5211daf0 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7fc17c007910 tx=0x7fc17c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71e19cbe5e6e87a0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:28.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.310+0000 7fc162a00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:09:28.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.310+0000 7fc162a00640 1 -- 172.21.15.28:0/2477935131 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc17c028470 con 0x563d520fb6b0 2024-02-21T23:09:28.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.310+0000 7fc162a00640 1 -- 172.21.15.28:0/2477935131 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(139..139 src has 1..139) v4 ==== 6245+0+0 (secure 0 0 0) 0x7fc17c028610 con 0x563d520fb6b0 2024-02-21T23:09:28.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.310+0000 7fc18b9ee740 1 -- 172.21.15.28:0/2477935131 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc1700271d0 con 0x563d520fb6b0 2024-02-21T23:09:28.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.310+0000 7fc162a00640 1 -- 172.21.15.28:0/2477935131 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=144) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc17c030050 con 0x563d520fb6b0 2024-02-21T23:09:28.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.310+0000 7fc18b9ee740 10 client.8634 fetch_fsmap learned FSMap version 144 2024-02-21T23:09:28.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.310+0000 7fc18b9ee740 1 -- 172.21.15.28:0/2477935131 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=144,monmap=2+}) v3 -- 0x563d5210d3a0 con 0x563d520fb6b0 2024-02-21T23:09:28.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.311+0000 7fc162a00640 1 -- 172.21.15.28:0/2477935131 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1700271d0 con 0x563d520fb6b0 2024-02-21T23:09:28.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.311+0000 7fc162a00640 1 -- 172.21.15.28:0/2477935131 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 144) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fc17c033020 con 0x563d520fb6b0 2024-02-21T23:09:28.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.311+0000 7fc18b9ee740 10 client.8634 fetch_fsmap finished waiting for FSMap version 144 2024-02-21T23:09:28.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.311+0000 7fc18b9ee740 10 client.8634 resolve_mds: resolved name 'b' to daemon mds.b 2024-02-21T23:09:28.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.311+0000 7fc18b9ee740 20 client.8634 populate_metadata read hostname 'smithi028' 2024-02-21T23:09:28.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.311+0000 7fc18b9ee740 1 --2- 172.21.15.28:0/2477935131 >> [v2:172.21.15.81:6834/4241906475,v1:172.21.15.81:6836/4241906475] conn(0x563d52167170 0x563d52167550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:28.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.311+0000 7fc18b9ee740 4 client.8634 mds_command: new command op to 6140 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-21T23:09:28.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.311+0000 7fc18b9ee740 1 -- 172.21.15.28:0/2477935131 --> [v2:172.21.15.81:6834/4241906475,v1:172.21.15.81:6836/4241906475] -- command(tid 0: {"prefix": "status"}) v1 -- 0x563d5202ad00 con 0x563d52167170 2024-02-21T23:09:28.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.311+0000 7fc181e00640 1 --2- 172.21.15.28:0/2477935131 >> [v2:172.21.15.81:6834/4241906475,v1:172.21.15.81:6836/4241906475] conn(0x563d52167170 0x563d52167550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:28.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.312+0000 7fc181e00640 1 --2- 172.21.15.28:0/2477935131 >> [v2:172.21.15.81:6834/4241906475,v1:172.21.15.81:6836/4241906475] conn(0x563d52167170 0x563d52167550 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa1932ae903aad9c server_cookie=4b42dd6ef8a522e2 in_seq=0 out_seq=0 2024-02-21T23:09:28.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.312+0000 7fc162a00640 10 client.8634 ms_handle_connect on v2:172.21.15.81:6834/4241906475 2024-02-21T23:09:28.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.313+0000 7fc162a00640 1 -- 172.21.15.28:0/2477935131 <== mds.0 v2:172.21.15.81:6834/4241906475 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+334 (crc 0 0 0) 0x563d5202ad00 con 0x563d52167170 2024-02-21T23:09:28.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.313+0000 7fc162a00640 10 client.8634 handle_command_reply: tid=0 2024-02-21T23:09:28.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.313+0000 7fc18b9ee740 1 client.8634 shutdown 2024-02-21T23:09:28.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.313+0000 7fc18b9ee740 1 -- 172.21.15.28:0/2477935131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d520fb6b0 msgr2=0x563d5211daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:28.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.313+0000 7fc18b9ee740 1 --2- 172.21.15.28:0/2477935131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d520fb6b0 0x563d5211daf0 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7fc17c007910 tx=0x7fc17c004a40 comp rx=0 tx=0).stop 2024-02-21T23:09:28.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.314+0000 7fc18b9ee740 1 -- 172.21.15.28:0/2477935131 shutdown_connections 2024-02-21T23:09:28.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.314+0000 7fc18b9ee740 1 --2- 172.21.15.28:0/2477935131 >> [v2:172.21.15.81:6834/4241906475,v1:172.21.15.81:6836/4241906475] conn(0x563d52167170 0x563d52167550 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:28.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.314+0000 7fc18b9ee740 1 --2- 172.21.15.28:0/2477935131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563d520fb6b0 0x563d5211daf0 unknown :-1 s=CLOSED pgs=1588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:28.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.314+0000 7fc18b9ee740 1 --2- 172.21.15.28:0/2477935131 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563d520e0690 0x563d5216d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:28.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.314+0000 7fc18b9ee740 1 --2- 172.21.15.28:0/2477935131 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x563d520c0330 0x563d520fac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:28.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.314+0000 7fc18b9ee740 1 -- 172.21.15.28:0/2477935131 >> 172.21.15.28:0/2477935131 conn(0x563d520bf020 msgr2=0x563d520fc850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:28.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.314+0000 7fc162a00640 0 client.8634 ms_handle_reset on v2:172.21.15.81:6834/4241906475 2024-02-21T23:09:28.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.314+0000 7fc18b9ee740 1 -- 172.21.15.28:0/2477935131 shutdown_connections 2024-02-21T23:09:28.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.314+0000 7fc18b9ee740 1 -- 172.21.15.28:0/2477935131 wait complete. 2024-02-21T23:09:28.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.314+0000 7fc18b9ee740 20 client.8634 trim_cache size 0 max 16384 2024-02-21T23:09:28.305 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-21T23:09:28.305 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac", 2024-02-21T23:09:28.305 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-02-21T23:09:28.305 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 6140, 2024-02-21T23:09:28.305 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-02-21T23:09:28.305 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-02-21T23:09:28.305 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-02-21T23:09:28.305 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 12.776285511999999, 2024-02-21T23:09:28.305 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 144, 2024-02-21T23:09:28.305 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 139, 2024-02-21T23:09:28.305 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 137, 2024-02-21T23:09:28.305 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 36.875152022000002 2024-02-21T23:09:28.306 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-21T23:09:28.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.315+0000 7fc183200640 1 -- 172.21.15.28:0/2564518153 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc16406b4d0 msgr2=0x7fc16406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:28.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.315+0000 7fc183200640 1 --2- 172.21.15.28:0/2564518153 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc16406b4d0 0x7fc16406d990 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fc1840bdd50 tx=0x7fc170008040 comp rx=0 tx=0).stop 2024-02-21T23:09:28.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.315+0000 7fc183200640 1 -- 172.21.15.28:0/2564518153 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1840c5040 msgr2=0x7fc1840aa090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:28.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.315+0000 7fc183200640 1 --2- 172.21.15.28:0/2564518153 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1840c5040 0x7fc1840aa090 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7fc174009960 tx=0x7fc1740046c0 comp rx=0 tx=0).stop 2024-02-21T23:09:28.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.315+0000 7fc183200640 1 -- 172.21.15.28:0/2564518153 shutdown_connections 2024-02-21T23:09:28.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.315+0000 7fc183200640 1 --2- 172.21.15.28:0/2564518153 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc16406b4d0 0x7fc16406d990 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:28.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.315+0000 7fc183200640 1 --2- 172.21.15.28:0/2564518153 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc1841524c0 0x7fc184153dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:28.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.315+0000 7fc183200640 1 --2- 172.21.15.28:0/2564518153 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc1840c85c0 0x7fc184158fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:28.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.315+0000 7fc183200640 1 --2- 172.21.15.28:0/2564518153 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1840c5040 0x7fc1840aa090 unknown :-1 s=CLOSED pgs=1584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:28.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.315+0000 7fc183200640 1 -- 172.21.15.28:0/2564518153 >> 172.21.15.28:0/2564518153 conn(0x7fc1840ac2b0 msgr2=0x7fc184150590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:28.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.315+0000 7fc183200640 1 -- 172.21.15.28:0/2564518153 shutdown_connections 2024-02-21T23:09:28.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:28.315+0000 7fc183200640 1 -- 172.21.15.28:0/2564518153 wait complete. 2024-02-21T23:09:28.320 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 10, "epoch": 144, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 10, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-02-21T23:09:03.869294+0000", "damaged": [], "data_pools": [ 22 ], "enabled": true, "epoch": 144, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6140": { "addr": "172.21.15.81:6836/4241906475", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6834", "nonce": 4241906475, "type": "v2" }, { "addr": "172.21.15.81:6836", "nonce": 4241906475, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6140, "incarnation": 141, "join_fscid": 10, "name": "b", "rank": 0, "state": "up:active", "state_seq": 10 } }, "last_failure": 0, "last_failure_osd_epoch": 137, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 21, "modified": "2024-02-21T23:09:17.546876+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6140 } } } ], "standbys": [ { "addr": "172.21.15.81:6837/2095908433", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6835", "nonce": 2095908433, "type": "v2" }, { "addr": "172.21.15.81:6837", "nonce": 2095908433, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 140, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6143, "incarnation": 0, "join_fscid": 10, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 7 }, { "addr": "172.21.15.28:6833/3934548735", "addrs": { "addrvec": [ { "addr": "172.21.15.28:6832", "nonce": 3934548735, "type": "v2" }, { "addr": "172.21.15.28:6833", "nonce": 3934548735, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 133, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8370, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.28:6835/2100132822", "addrs": { "addrvec": [ { "addr": "172.21.15.28:6834", "nonce": 2100132822, "type": "v2" }, { "addr": "172.21.15.28:6835", "nonce": 2100132822, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 142, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8556, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-02-21T23:09:28.321 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': 10, 'name': 'd', 'state': 'up:active'}] 2024-02-21T23:09:28.321 INFO:tasks.cephfs.test_failover:current = [{'gid': 8556, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.28:6835/2100132822', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 2100132822}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 2100132822}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 142}, {'gid': 6140, 'name': 'b', 'rank': 0, 'incarnation': 141, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.81:6836/4241906475', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6834', 'nonce': 4241906475}, {'type': 'v1', 'addr': '172.21.15.81:6836', 'nonce': 4241906475}]}, 'join_fscid': 10, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 8370, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.28:6833/3934548735', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 3934548735}, {'type': 'v1', 'addr': '172.21.15.28:6833', 'nonce': 3934548735}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 133}, {'gid': 6143, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 7, 'addr': '172.21.15.81:6837/2095908433', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6835', 'nonce': 2095908433}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 2095908433}]}, 'join_fscid': 10, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 140}] 2024-02-21T23:09:28.321 DEBUG:tasks.cephfs.test_failover:-1 != 10 2024-02-21T23:09:28.321 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-02-21T23:09:29.647 INFO:tasks.ceph.mds.b.smithi081.stderr: -360> 2024-02-21T23:09:22.323+0000 7febabc00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-21T23:09:29.696 INFO:tasks.ceph.mds.b.smithi081.stdout:starting mds.b at 2024-02-21T23:09:33.323 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:09:33.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.567+0000 7f0fb8600640 1 Processor -- start 2024-02-21T23:09:33.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.568+0000 7f0fb8600640 1 -- start start 2024-02-21T23:09:33.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.569+0000 7f0fb8600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0fb00c3280 0x7f0fb00c3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:33.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.569+0000 7f0fb8600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0fb0072430 0x7f0fb01526e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:33.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.569+0000 7f0fb8600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fb0155860 0x7f0fb0155d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:33.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.569+0000 7f0fb8600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fb0152c20 con 0x7f0fb0072430 2024-02-21T23:09:33.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.569+0000 7f0fb8600640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fb00c6560 con 0x7f0fb0155860 2024-02-21T23:09:33.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.569+0000 7f0fb8600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb00c5cc0 con 0x7f0fb00c3280 2024-02-21T23:09:33.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.569+0000 7f0fb5200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0fb0072430 0x7f0fb01526e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:33.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.570+0000 7f0fb5200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0fb0072430 0x7f0fb01526e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57384/0 (socket says 172.21.15.28:57384) 2024-02-21T23:09:33.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.570+0000 7f0fb5200640 1 -- 172.21.15.28:0/2708618319 learned_addr learned my addr 172.21.15.28:0/2708618319 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:33.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.570+0000 7f0fb6600640 1 --2- 172.21.15.28:0/2708618319 >> v2:172.21.15.81:3300/0 conn(0x7f0fb0155860 0x7f0fb0155d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:33.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.570+0000 7f0fb5c00640 1 --2- 172.21.15.28:0/2708618319 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0fb00c3280 0x7f0fb00c3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:33.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.570+0000 7f0fb5200640 1 -- 172.21.15.28:0/2708618319 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0fb00c3280 msgr2=0x7f0fb00c3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:33.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.570+0000 7f0fb5200640 1 --2- 172.21.15.28:0/2708618319 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0fb00c3280 0x7f0fb00c3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:33.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.570+0000 7f0fb5200640 1 -- 172.21.15.28:0/2708618319 >> v2:172.21.15.81:3300/0 conn(0x7f0fb0155860 msgr2=0x7f0fb0155d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:33.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.570+0000 7f0fb5200640 1 --2- 172.21.15.28:0/2708618319 >> v2:172.21.15.81:3300/0 conn(0x7f0fb0155860 0x7f0fb0155d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:33.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.570+0000 7f0fb5200640 1 -- 172.21.15.28:0/2708618319 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fb00c5fa0 con 0x7f0fb0072430 2024-02-21T23:09:33.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.571+0000 7f0fb5200640 1 --2- 172.21.15.28:0/2708618319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0fb0072430 0x7f0fb01526e0 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f0fb00c0c80 tx=0x7f0f9c02e050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc0fead4a1c800bb server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:33.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.572+0000 7f0fafe00640 1 -- 172.21.15.28:0/2708618319 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f9c03b070 con 0x7f0fb0072430 2024-02-21T23:09:33.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.572+0000 7f0fafe00640 1 -- 172.21.15.28:0/2708618319 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f9c02ee00 con 0x7f0fb0072430 2024-02-21T23:09:33.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.572+0000 7f0fafe00640 1 -- 172.21.15.28:0/2708618319 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f9c036660 con 0x7f0fb0072430 2024-02-21T23:09:33.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.572+0000 7f0fb8600640 1 -- 172.21.15.28:0/2708618319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0fb0072430 msgr2=0x7f0fb01526e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:33.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.572+0000 7f0fb8600640 1 --2- 172.21.15.28:0/2708618319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0fb0072430 0x7f0fb01526e0 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f0fb00c0c80 tx=0x7f0f9c02e050 comp rx=0 tx=0).stop 2024-02-21T23:09:33.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.572+0000 7f0fb8600640 1 -- 172.21.15.28:0/2708618319 shutdown_connections 2024-02-21T23:09:33.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.572+0000 7f0fb8600640 1 --2- 172.21.15.28:0/2708618319 >> v2:172.21.15.81:3300/0 conn(0x7f0fb0155860 0x7f0fb0155d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:33.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.572+0000 7f0fb8600640 1 --2- 172.21.15.28:0/2708618319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0fb0072430 0x7f0fb01526e0 unknown :-1 s=CLOSED pgs=1589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:33.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.572+0000 7f0fb8600640 1 --2- 172.21.15.28:0/2708618319 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0fb00c3280 0x7f0fb00c3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:33.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.572+0000 7f0fb8600640 1 -- 172.21.15.28:0/2708618319 >> 172.21.15.28:0/2708618319 conn(0x7f0fb00c67f0 msgr2=0x7f0fb00c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:33.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.573+0000 7f0fb8600640 1 -- 172.21.15.28:0/2708618319 shutdown_connections 2024-02-21T23:09:33.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.573+0000 7f0fb8600640 1 -- 172.21.15.28:0/2708618319 wait complete. 2024-02-21T23:09:33.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.574+0000 7f0fb8600640 1 Processor -- start 2024-02-21T23:09:33.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.575+0000 7f0fb8600640 1 -- start start 2024-02-21T23:09:33.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.575+0000 7f0fb8600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0fb0072430 0x7f0fb00a8950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:33.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.575+0000 7f0fb8600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0fb00c3280 0x7f0fb01e0df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:33.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.575+0000 7f0fb8600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0fb0155860 0x7f0fb01e1330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:33.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.575+0000 7f0fb8600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb00c5920 con 0x7f0fb00c3280 2024-02-21T23:09:33.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.576+0000 7f0fb8600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb00a6100 con 0x7f0fb0155860 2024-02-21T23:09:33.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.576+0000 7f0fb8600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb00c5aa0 con 0x7f0fb0072430 2024-02-21T23:09:33.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.576+0000 7f0fb5200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0fb00c3280 0x7f0fb01e0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:33.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.576+0000 7f0fb5200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0fb00c3280 0x7f0fb01e0df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57396/0 (socket says 172.21.15.28:57396) 2024-02-21T23:09:33.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.576+0000 7f0fb5200640 1 -- 172.21.15.28:0/734712218 learned_addr learned my addr 172.21.15.28:0/734712218 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:33.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.576+0000 7f0fb5c00640 1 --2- 172.21.15.28:0/734712218 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0fb0072430 0x7f0fb00a8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:33.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.576+0000 7f0fb6600640 1 --2- 172.21.15.28:0/734712218 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0fb0155860 0x7f0fb01e1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:33.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.576+0000 7f0fb5200640 1 -- 172.21.15.28:0/734712218 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0fb0072430 msgr2=0x7f0fb00a8950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:33.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.576+0000 7f0fb5200640 1 --2- 172.21.15.28:0/734712218 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0fb0072430 0x7f0fb00a8950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:33.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.576+0000 7f0fb5200640 1 -- 172.21.15.28:0/734712218 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0fb0155860 msgr2=0x7f0fb01e1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:33.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.576+0000 7f0fb5200640 1 --2- 172.21.15.28:0/734712218 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0fb0155860 0x7f0fb01e1330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:33.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.576+0000 7f0fb5200640 1 -- 172.21.15.28:0/734712218 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fb014ffe0 con 0x7f0fb00c3280 2024-02-21T23:09:33.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.576+0000 7f0fb6600640 1 --2- 172.21.15.28:0/734712218 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0fb0155860 0x7f0fb01e1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:09:33.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.576+0000 7f0fb5c00640 1 --2- 172.21.15.28:0/734712218 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0fb0072430 0x7f0fb00a8950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:09:33.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.577+0000 7f0fb5200640 1 --2- 172.21.15.28:0/734712218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0fb00c3280 0x7f0fb01e0df0 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7f0f9c02e090 tx=0x7f0f9c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:33.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.577+0000 7f0fad600640 1 -- 172.21.15.28:0/734712218 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f9c03b040 con 0x7f0fb00c3280 2024-02-21T23:09:33.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.577+0000 7f0fad600640 1 -- 172.21.15.28:0/734712218 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f9c02ec00 con 0x7f0fb00c3280 2024-02-21T23:09:33.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.577+0000 7f0fad600640 1 -- 172.21.15.28:0/734712218 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f9c041670 con 0x7f0fb00c3280 2024-02-21T23:09:33.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.577+0000 7f0fb8600640 1 -- 172.21.15.28:0/734712218 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fb01502c0 con 0x7f0fb00c3280 2024-02-21T23:09:33.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.577+0000 7f0fb8600640 1 -- 172.21.15.28:0/734712218 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fb0150880 con 0x7f0fb00c3280 2024-02-21T23:09:33.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.578+0000 7f0fb8600640 1 -- 172.21.15.28:0/734712218 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fb0067f30 con 0x7f0fb00c3280 2024-02-21T23:09:33.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.579+0000 7f0fad600640 1 -- 172.21.15.28:0/734712218 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0f9c048020 con 0x7f0fb00c3280 2024-02-21T23:09:33.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.580+0000 7f0fad600640 1 --2- 172.21.15.28:0/734712218 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0f9006b520 0x7f0f9006d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:33.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.580+0000 7f0fad600640 1 -- 172.21.15.28:0/734712218 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6331+0+0 (secure 0 0 0) 0x7f0f9c0ac3d0 con 0x7f0fb00c3280 2024-02-21T23:09:33.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.581+0000 7f0fb5c00640 1 --2- 172.21.15.28:0/734712218 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0f9006b520 0x7f0f9006d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:33.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.582+0000 7f0fb5c00640 1 --2- 172.21.15.28:0/734712218 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0f9006b520 0x7f0f9006d9e0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f0fa0009b90 tx=0x7f0fa0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:33.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.585+0000 7f0fad600640 1 -- 172.21.15.28:0/734712218 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f0f9c07b170 con 0x7f0fb00c3280 2024-02-21T23:09:33.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.825+0000 7f0fb8600640 1 -- 172.21.15.28:0/734712218 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0fb01e0b50 con 0x7f0fb00c3280 2024-02-21T23:09:33.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.827+0000 7f0fad600640 1 -- 172.21.15.28:0/734712218 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 149 v149) v1 ==== 95+0+4948 (secure 0 0 0) 0x7f0f9c07aa70 con 0x7f0fb00c3280 2024-02-21T23:09:33.816 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:09:33.816 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":149,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6934,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/878375431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":878375431},{"type":"v1","addr":"172.21.15.81:6836","nonce":878375431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":147},{"gid":8370,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/3934548735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3934548735},{"type":"v1","addr":"172.21.15.28:6833","nonce":3934548735}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":133},{"gid":8556,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2100132822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2100132822},{"type":"v1","addr":"172.21.15.28:6835","nonce":2100132822}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":142}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:09:03.869294+0000","modified":"2024-02-21T23:09:31.652165+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":140,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6143},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6143":{"gid":6143,"name":"d","rank":0,"incarnation":146,"state":"up:active","state_seq":13,"addr":"172.21.15.81:6837/2095908433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":2095908433},{"type":"v1","addr":"172.21.15.81:6837","nonce":2095908433}]},"join_fscid":10,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-02-21T23:09:33.817 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 149 2024-02-21T23:09:33.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.829+0000 7f0fb8600640 1 -- 172.21.15.28:0/734712218 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0f9006b520 msgr2=0x7f0f9006d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:33.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.829+0000 7f0fb8600640 1 --2- 172.21.15.28:0/734712218 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0f9006b520 0x7f0f9006d9e0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f0fa0009b90 tx=0x7f0fa0008040 comp rx=0 tx=0).stop 2024-02-21T23:09:33.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.829+0000 7f0fb8600640 1 -- 172.21.15.28:0/734712218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0fb00c3280 msgr2=0x7f0fb01e0df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:33.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.829+0000 7f0fb8600640 1 --2- 172.21.15.28:0/734712218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0fb00c3280 0x7f0fb01e0df0 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7f0f9c02e090 tx=0x7f0f9c004730 comp rx=0 tx=0).stop 2024-02-21T23:09:33.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.830+0000 7f0fb8600640 1 -- 172.21.15.28:0/734712218 shutdown_connections 2024-02-21T23:09:33.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.830+0000 7f0fb8600640 1 --2- 172.21.15.28:0/734712218 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0f9006b520 0x7f0f9006d9e0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:33.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.830+0000 7f0fb8600640 1 --2- 172.21.15.28:0/734712218 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0fb0155860 0x7f0fb01e1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:33.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.830+0000 7f0fb8600640 1 --2- 172.21.15.28:0/734712218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0fb00c3280 0x7f0fb01e0df0 unknown :-1 s=CLOSED pgs=1590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:33.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.830+0000 7f0fb8600640 1 --2- 172.21.15.28:0/734712218 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0fb0072430 0x7f0fb00a8950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:33.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.830+0000 7f0fb8600640 1 -- 172.21.15.28:0/734712218 >> 172.21.15.28:0/734712218 conn(0x7f0fb00c67f0 msgr2=0x7f0fb0155c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:33.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.830+0000 7f0fb8600640 1 -- 172.21.15.28:0/734712218 shutdown_connections 2024-02-21T23:09:33.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:33.830+0000 7f0fb8600640 1 -- 172.21.15.28:0/734712218 wait complete. 2024-02-21T23:09:34.242 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 149, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:09:03.869294+0000', 'modified': '2024-02-21T23:09:31.652165+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 140, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6143}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6143': {'gid': 6143, 'name': 'd', 'rank': 0, 'incarnation': 146, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.81:6837/2095908433', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6835', 'nonce': 2095908433}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 2095908433}]}, 'join_fscid': 10, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:09:34.242 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-21T23:09:34.243 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-02-21T23:09:34.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.484+0000 7f3d2a200640 1 Processor -- start 2024-02-21T23:09:34.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.485+0000 7f3d2a200640 1 -- start start 2024-02-21T23:09:34.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.486+0000 7f3d2a200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f3d241527a0 0x7f3d24158f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:34.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.486+0000 7f3d2a200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3d2406d410 0x7f3d2406d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:34.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.486+0000 7f3d2a200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d240c2070 con 0x7f3d240bf9a0 2024-02-21T23:09:34.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.486+0000 7f3d2a200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d2406e790 con 0x7f3d2406d410 2024-02-21T23:09:34.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.486+0000 7f3d2a200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d240c29a0 con 0x7f3d241527a0 2024-02-21T23:09:34.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.487+0000 7f3d29800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3d2406d410 0x7f3d2406d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:34.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.487+0000 7f3d29800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3d2406d410 0x7f3d2406d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57406/0 (socket says 172.21.15.28:57406) 2024-02-21T23:09:34.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.487+0000 7f3d29800640 1 -- 172.21.15.28:0/3424757538 learned_addr learned my addr 172.21.15.28:0/3424757538 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:34.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.487+0000 7f3d28e00640 1 --2- 172.21.15.28:0/3424757538 >> v2:172.21.15.81:3300/0 conn(0x7f3d241527a0 0x7f3d24158f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:34.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.487+0000 7f3d29800640 1 -- 172.21.15.28:0/3424757538 >> v2:172.21.15.81:3300/0 conn(0x7f3d241527a0 msgr2=0x7f3d24158f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:34.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.487+0000 7f3d29800640 1 --2- 172.21.15.28:0/3424757538 >> v2:172.21.15.81:3300/0 conn(0x7f3d241527a0 0x7f3d24158f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:34.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.487+0000 7f3d29800640 1 -- 172.21.15.28:0/3424757538 >> v1:172.21.15.81:6789/0 conn(0x7f3d240bf9a0 legacy=0x7f3d240bfdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:34.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.487+0000 7f3d29800640 1 -- 172.21.15.28:0/3424757538 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d240c2350 con 0x7f3d2406d410 2024-02-21T23:09:34.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.489+0000 7f3d29800640 1 --2- 172.21.15.28:0/3424757538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d2406d410 0x7f3d2406d820 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f3d24067ae0 tx=0x7f3d1c02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2943249dc925dba6 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:34.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.489+0000 7f3d23400640 1 -- 172.21.15.28:0/3424757538 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d1c02cb20 con 0x7f3d2406d410 2024-02-21T23:09:34.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.489+0000 7f3d23400640 1 -- 172.21.15.28:0/3424757538 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d1c02ccc0 con 0x7f3d2406d410 2024-02-21T23:09:34.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.489+0000 7f3d23400640 1 -- 172.21.15.28:0/3424757538 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d1c034610 con 0x7f3d2406d410 2024-02-21T23:09:34.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.489+0000 7f3d2a200640 1 -- 172.21.15.28:0/3424757538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d2406d410 msgr2=0x7f3d2406d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:34.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.489+0000 7f3d2a200640 1 --2- 172.21.15.28:0/3424757538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d2406d410 0x7f3d2406d820 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f3d24067ae0 tx=0x7f3d1c02bf80 comp rx=0 tx=0).stop 2024-02-21T23:09:34.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.490+0000 7f3d2a200640 1 -- 172.21.15.28:0/3424757538 shutdown_connections 2024-02-21T23:09:34.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.490+0000 7f3d2a200640 1 --2- 172.21.15.28:0/3424757538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d2406d410 0x7f3d2406d820 unknown :-1 s=CLOSED pgs=1591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:34.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.490+0000 7f3d2a200640 1 --2- 172.21.15.28:0/3424757538 >> v2:172.21.15.81:3300/0 conn(0x7f3d241527a0 0x7f3d24158f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:34.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.490+0000 7f3d2a200640 1 -- 172.21.15.28:0/3424757538 >> 172.21.15.28:0/3424757538 conn(0x7f3d240a8320 msgr2=0x7f3d240a8730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:34.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.490+0000 7f3d2a200640 1 -- 172.21.15.28:0/3424757538 shutdown_connections 2024-02-21T23:09:34.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.490+0000 7f3d2a200640 1 -- 172.21.15.28:0/3424757538 wait complete. 2024-02-21T23:09:34.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.491+0000 7f3d2a200640 1 Processor -- start 2024-02-21T23:09:34.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.491+0000 7f3d2a200640 1 -- start start 2024-02-21T23:09:34.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.492+0000 7f3d2a200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3d2406d410 0x7f3d241ee7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:34.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.492+0000 7f3d2a200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d240bf9a0 0x7f3d240a6960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:34.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.492+0000 7f3d2a200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d241527a0 0x7f3d2414d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:34.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.492+0000 7f3d2a200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d240c1d20 con 0x7f3d241527a0 2024-02-21T23:09:34.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.492+0000 7f3d2a200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d240c0500 con 0x7f3d240bf9a0 2024-02-21T23:09:34.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.492+0000 7f3d2a200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d2406dd60 con 0x7f3d2406d410 2024-02-21T23:09:34.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.493+0000 7f3d29800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d241527a0 0x7f3d2414d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:34.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.493+0000 7f3d29800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d241527a0 0x7f3d2414d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57420/0 (socket says 172.21.15.28:57420) 2024-02-21T23:09:34.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.493+0000 7f3d29800640 1 -- 172.21.15.28:0/1042361954 learned_addr learned my addr 172.21.15.28:0/1042361954 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:34.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.493+0000 7f3d23e00640 1 --2- 172.21.15.28:0/1042361954 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d240bf9a0 0x7f3d240a6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:34.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.493+0000 7f3d28e00640 1 --2- 172.21.15.28:0/1042361954 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3d2406d410 0x7f3d241ee7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:34.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.493+0000 7f3d29800640 1 -- 172.21.15.28:0/1042361954 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3d2406d410 msgr2=0x7f3d241ee7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:34.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.493+0000 7f3d29800640 1 --2- 172.21.15.28:0/1042361954 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3d2406d410 0x7f3d241ee7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:34.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.493+0000 7f3d29800640 1 -- 172.21.15.28:0/1042361954 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d240bf9a0 msgr2=0x7f3d240a6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:34.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.493+0000 7f3d29800640 1 --2- 172.21.15.28:0/1042361954 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d240bf9a0 0x7f3d240a6960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:34.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.493+0000 7f3d29800640 1 -- 172.21.15.28:0/1042361954 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d240a9b60 con 0x7f3d241527a0 2024-02-21T23:09:34.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.493+0000 7f3d29800640 1 --2- 172.21.15.28:0/1042361954 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d241527a0 0x7f3d2414d550 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c009f40 tx=0x7f3d1c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:34.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.494+0000 7f3d20c00640 1 -- 172.21.15.28:0/1042361954 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d1c043020 con 0x7f3d241527a0 2024-02-21T23:09:34.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.494+0000 7f3d20c00640 1 -- 172.21.15.28:0/1042361954 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d1c03e450 con 0x7f3d241527a0 2024-02-21T23:09:34.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.494+0000 7f3d20c00640 1 -- 172.21.15.28:0/1042361954 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d1c009510 con 0x7f3d241527a0 2024-02-21T23:09:34.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.494+0000 7f3d2a200640 1 -- 172.21.15.28:0/1042361954 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d24152e40 con 0x7f3d241527a0 2024-02-21T23:09:34.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.494+0000 7f3d2a200640 1 -- 172.21.15.28:0/1042361954 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d241533e0 con 0x7f3d241527a0 2024-02-21T23:09:34.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.496+0000 7f3d20c00640 1 -- 172.21.15.28:0/1042361954 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3d1c046020 con 0x7f3d241527a0 2024-02-21T23:09:34.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.497+0000 7f3d20c00640 1 --2- 172.21.15.28:0/1042361954 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3d0806b4d0 0x7f3d0806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:34.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.498+0000 7f3d20c00640 1 -- 172.21.15.28:0/1042361954 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6331+0+0 (secure 0 0 0) 0x7f3d1c0acd20 con 0x7f3d241527a0 2024-02-21T23:09:34.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.498+0000 7f3d2e634740 1 Processor -- start 2024-02-21T23:09:34.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.498+0000 7f3d28e00640 1 --2- 172.21.15.28:0/1042361954 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3d0806b4d0 0x7f3d0806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:34.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.498+0000 7f3d2e634740 1 -- start start 2024-02-21T23:09:34.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.499+0000 7f3d2e634740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634fd4fa020 0x5634fd4fa400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:34.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.499+0000 7f3d2e634740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5634fd4fb330 0x5634fd536170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:34.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.499+0000 7f3d2e634740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634fd5366b0 0x5634fd534720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:34.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.499+0000 7f3d2e634740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634fd45e470 con 0x5634fd5366b0 2024-02-21T23:09:34.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.499+0000 7f3d2e634740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634fd407af0 con 0x5634fd4fa020 2024-02-21T23:09:34.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.499+0000 7f3d2e634740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x5634fd45a510 con 0x5634fd4fb330 2024-02-21T23:09:34.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.499+0000 7f3d29800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634fd5366b0 0x5634fd534720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:34.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.499+0000 7f3d29800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634fd5366b0 0x5634fd534720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57424/0 (socket says 172.21.15.28:57424) 2024-02-21T23:09:34.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.499+0000 7f3d29800640 1 -- 172.21.15.28:0/3308039167 learned_addr learned my addr 172.21.15.28:0/3308039167 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:34.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.499+0000 7f3d28e00640 1 --2- 172.21.15.28:0/1042361954 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3d0806b4d0 0x7f3d0806d990 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f3d18006fd0 tx=0x7f3d18008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:34.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.500+0000 7f3d28e00640 1 --2- 172.21.15.28:0/3308039167 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634fd4fa020 0x5634fd4fa400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:34.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.500+0000 7f3d29800640 1 -- 172.21.15.28:0/3308039167 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5634fd4fb330 msgr2=0x5634fd536170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:34.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.500+0000 7f3d23e00640 1 --2- 172.21.15.28:0/3308039167 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5634fd4fb330 0x5634fd536170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:34.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.500+0000 7f3d29800640 1 --2- 172.21.15.28:0/3308039167 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5634fd4fb330 0x5634fd536170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:34.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.500+0000 7f3d29800640 1 -- 172.21.15.28:0/3308039167 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634fd4fa020 msgr2=0x5634fd4fa400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:34.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.500+0000 7f3d29800640 1 --2- 172.21.15.28:0/3308039167 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634fd4fa020 0x5634fd4fa400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:34.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.500+0000 7f3d29800640 1 -- 172.21.15.28:0/3308039167 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5634fd40ed90 con 0x5634fd5366b0 2024-02-21T23:09:34.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.500+0000 7f3d29800640 1 --2- 172.21.15.28:0/3308039167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634fd5366b0 0x5634fd534720 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c07b120 tx=0x7f3d1c07aaf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdc6b04287a4d12b server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:34.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.500+0000 7f3d11600640 1 -- 172.21.15.28:0/3308039167 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d1c07f2e0 con 0x5634fd5366b0 2024-02-21T23:09:34.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.500+0000 7f3d11600640 1 -- 172.21.15.28:0/3308039167 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d1c0838a0 con 0x5634fd5366b0 2024-02-21T23:09:34.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.500+0000 7f3d11600640 1 -- 172.21.15.28:0/3308039167 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d1c079490 con 0x5634fd5366b0 2024-02-21T23:09:34.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.501+0000 7f3d2e634740 1 -- 172.21.15.28:0/3308039167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634fd5366b0 msgr2=0x5634fd534720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:34.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.501+0000 7f3d2e634740 1 --2- 172.21.15.28:0/3308039167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634fd5366b0 0x5634fd534720 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c07b120 tx=0x7f3d1c07aaf0 comp rx=0 tx=0).stop 2024-02-21T23:09:34.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.501+0000 7f3d2e634740 1 -- 172.21.15.28:0/3308039167 shutdown_connections 2024-02-21T23:09:34.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.501+0000 7f3d2e634740 1 --2- 172.21.15.28:0/3308039167 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5634fd4fb330 0x5634fd536170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:34.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.501+0000 7f3d2e634740 1 --2- 172.21.15.28:0/3308039167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634fd5366b0 0x5634fd534720 unknown :-1 s=CLOSED pgs=1593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:34.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.501+0000 7f3d2e634740 1 --2- 172.21.15.28:0/3308039167 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634fd4fa020 0x5634fd4fa400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:34.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.501+0000 7f3d2e634740 1 -- 172.21.15.28:0/3308039167 >> 172.21.15.28:0/3308039167 conn(0x5634fd4f6970 msgr2=0x5634fd51b380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:34.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.501+0000 7f3d2e634740 1 -- 172.21.15.28:0/3308039167 shutdown_connections 2024-02-21T23:09:34.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.501+0000 7f3d2e634740 1 -- 172.21.15.28:0/3308039167 wait complete. 2024-02-21T23:09:34.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.502+0000 7f3d2e634740 1 -- start start 2024-02-21T23:09:34.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.502+0000 7f3d2e634740 1 Processor -- start 2024-02-21T23:09:34.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.503+0000 7f3d2e634740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5634fd4fb330 0x5634fd51d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:34.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.503+0000 7f3d2e634740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634fd5366b0 0x5634fd549b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:34.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.503+0000 7f3d2e634740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634fd51a910 0x5634fd5332e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:34.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.503+0000 7f3d2e634740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634fd45e470 con 0x5634fd51a910 2024-02-21T23:09:34.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.503+0000 7f3d2e634740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634fd3fd040 con 0x5634fd5366b0 2024-02-21T23:09:34.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.503+0000 7f3d2e634740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x5634fd51e290 con 0x5634fd4fb330 2024-02-21T23:09:34.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.503+0000 7f3d2e634740 10 client.0 _ll_register_callbacks cb 0x5634fd41a920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:09:34.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.503+0000 7f3d29800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634fd51a910 0x5634fd5332e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:34.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.503+0000 7f3d29800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634fd51a910 0x5634fd5332e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57440/0 (socket says 172.21.15.28:57440) 2024-02-21T23:09:34.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.503+0000 7f3d29800640 1 -- 172.21.15.28:0/2551874214 learned_addr learned my addr 172.21.15.28:0/2551874214 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:34.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.504+0000 7f3d29800640 1 -- 172.21.15.28:0/2551874214 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5634fd4fb330 msgr2=0x5634fd51d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:34.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.504+0000 7f3d29800640 1 --2- 172.21.15.28:0/2551874214 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5634fd4fb330 0x5634fd51d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:34.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.504+0000 7f3d29800640 1 -- 172.21.15.28:0/2551874214 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634fd5366b0 msgr2=0x5634fd549b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:34.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.504+0000 7f3d29800640 1 --2- 172.21.15.28:0/2551874214 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634fd5366b0 0x5634fd549b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:34.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.504+0000 7f3d29800640 1 -- 172.21.15.28:0/2551874214 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5634fd41caa0 con 0x5634fd51a910 2024-02-21T23:09:34.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.504+0000 7f3d29800640 1 --2- 172.21.15.28:0/2551874214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634fd51a910 0x5634fd5332e0 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c07b120 tx=0x7f3d1c009810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e9815e1cb29ac6c server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:34.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.504+0000 7f3d06000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:09:34.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.504+0000 7f3d06000640 1 -- 172.21.15.28:0/2551874214 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d1c03c350 con 0x5634fd51a910 2024-02-21T23:09:34.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.504+0000 7f3d06000640 1 -- 172.21.15.28:0/2551874214 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(140..140 src has 1..140) v4 ==== 6331+0+0 (secure 0 0 0) 0x7f3d1c03c4f0 con 0x5634fd51a910 2024-02-21T23:09:34.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.504+0000 7f3d2e634740 1 -- 172.21.15.28:0/2551874214 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3d1c0838a0 con 0x5634fd51a910 2024-02-21T23:09:34.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.504+0000 7f3d06000640 1 -- 172.21.15.28:0/2551874214 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=149) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3d1c09f050 con 0x5634fd51a910 2024-02-21T23:09:34.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.504+0000 7f3d2e634740 10 client.8658 fetch_fsmap learned FSMap version 149 2024-02-21T23:09:34.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.504+0000 7f3d2e634740 1 -- 172.21.15.28:0/2551874214 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=149,monmap=2+}) v3 -- 0x5634fd1732d0 con 0x5634fd51a910 2024-02-21T23:09:34.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.505+0000 7f3d06000640 1 -- 172.21.15.28:0/2551874214 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d1c0838a0 con 0x5634fd51a910 2024-02-21T23:09:34.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.505+0000 7f3d06000640 1 -- 172.21.15.28:0/2551874214 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 149) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f3d1c0a2020 con 0x5634fd51a910 2024-02-21T23:09:34.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.505+0000 7f3d2e634740 10 client.8658 fetch_fsmap finished waiting for FSMap version 149 2024-02-21T23:09:34.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.505+0000 7f3d2e634740 10 client.8658 resolve_mds: resolved name 'd' to daemon mds.d 2024-02-21T23:09:34.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.505+0000 7f3d2e634740 20 client.8658 populate_metadata read hostname 'smithi028' 2024-02-21T23:09:34.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.505+0000 7f3d2e634740 1 --2- 172.21.15.28:0/2551874214 >> [v2:172.21.15.81:6835/2095908433,v1:172.21.15.81:6837/2095908433] conn(0x5634fd546170 0x5634fd546590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:34.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.505+0000 7f3d2e634740 4 client.8658 mds_command: new command op to 6143 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-21T23:09:34.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.505+0000 7f3d2e634740 1 -- 172.21.15.28:0/2551874214 --> [v2:172.21.15.81:6835/2095908433,v1:172.21.15.81:6837/2095908433] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5634fd46b9b0 con 0x5634fd546170 2024-02-21T23:09:34.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.505+0000 7f3d28e00640 1 --2- 172.21.15.28:0/2551874214 >> [v2:172.21.15.81:6835/2095908433,v1:172.21.15.81:6837/2095908433] conn(0x5634fd546170 0x5634fd546590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:34.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.506+0000 7f3d28e00640 1 --2- 172.21.15.28:0/2551874214 >> [v2:172.21.15.81:6835/2095908433,v1:172.21.15.81:6837/2095908433] conn(0x5634fd546170 0x5634fd546590 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1dacc3357bd52e0e server_cookie=64a701c00e2d70fd in_seq=0 out_seq=0 2024-02-21T23:09:34.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.506+0000 7f3d06000640 10 client.8658 ms_handle_connect on v2:172.21.15.81:6835/2095908433 2024-02-21T23:09:34.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.508+0000 7f3d06000640 1 -- 172.21.15.28:0/2551874214 <== mds.0 v2:172.21.15.81:6835/2095908433 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5634fd46b9b0 con 0x5634fd546170 2024-02-21T23:09:34.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.508+0000 7f3d06000640 10 client.8658 handle_command_reply: tid=0 2024-02-21T23:09:34.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.508+0000 7f3d2e634740 1 client.8658 shutdown 2024-02-21T23:09:34.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.509+0000 7f3d2e634740 1 -- 172.21.15.28:0/2551874214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634fd51a910 msgr2=0x5634fd5332e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:34.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.509+0000 7f3d2e634740 1 --2- 172.21.15.28:0/2551874214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634fd51a910 0x5634fd5332e0 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c07b120 tx=0x7f3d1c009810 comp rx=0 tx=0).stop 2024-02-21T23:09:34.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.509+0000 7f3d2e634740 1 -- 172.21.15.28:0/2551874214 shutdown_connections 2024-02-21T23:09:34.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.509+0000 7f3d2e634740 1 --2- 172.21.15.28:0/2551874214 >> [v2:172.21.15.81:6835/2095908433,v1:172.21.15.81:6837/2095908433] conn(0x5634fd546170 0x5634fd546590 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:34.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.509+0000 7f3d2e634740 1 --2- 172.21.15.28:0/2551874214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634fd51a910 0x5634fd5332e0 unknown :-1 s=CLOSED pgs=1594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:34.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.509+0000 7f3d2e634740 1 --2- 172.21.15.28:0/2551874214 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634fd5366b0 0x5634fd549b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:34.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.509+0000 7f3d2e634740 1 --2- 172.21.15.28:0/2551874214 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5634fd4fb330 0x5634fd51d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:34.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.509+0000 7f3d06000640 0 client.8658 ms_handle_reset on v2:172.21.15.81:6835/2095908433 2024-02-21T23:09:34.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.509+0000 7f3d2e634740 1 -- 172.21.15.28:0/2551874214 >> 172.21.15.28:0/2551874214 conn(0x5634fd4fa020 msgr2=0x5634fd534fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:34.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.509+0000 7f3d2e634740 1 -- 172.21.15.28:0/2551874214 shutdown_connections 2024-02-21T23:09:34.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.509+0000 7f3d2e634740 1 -- 172.21.15.28:0/2551874214 wait complete. 2024-02-21T23:09:34.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.509+0000 7f3d2e634740 20 client.8658 trim_cache size 0 max 16384 2024-02-21T23:09:34.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.527+0000 7f3d2e634740 1 Processor -- start 2024-02-21T23:09:34.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.527+0000 7f3d2e634740 1 -- start start 2024-02-21T23:09:34.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.527+0000 7f3d2e634740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5634fd4fb330 0x5634fd4f37e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:34.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.527+0000 7f3d2e634740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634fd51a910 0x5634fd4f4230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:34.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.527+0000 7f3d2e634740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634fd5366b0 0x5634fd4f4770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:34.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.527+0000 7f3d2e634740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d241f6ec0 con 0x5634fd5366b0 2024-02-21T23:09:34.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.527+0000 7f3d2e634740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d241f6360 con 0x5634fd51a910 2024-02-21T23:09:34.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.527+0000 7f3d2e634740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d241f68c0 con 0x5634fd4fb330 2024-02-21T23:09:34.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.528+0000 7f3d29800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634fd5366b0 0x5634fd4f4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:34.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.528+0000 7f3d29800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634fd5366b0 0x5634fd4f4770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57450/0 (socket says 172.21.15.28:57450) 2024-02-21T23:09:34.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.528+0000 7f3d29800640 1 -- 172.21.15.28:0/2441764584 learned_addr learned my addr 172.21.15.28:0/2441764584 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:34.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.528+0000 7f3d28e00640 1 --2- 172.21.15.28:0/2441764584 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5634fd4fb330 0x5634fd4f37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:34.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.528+0000 7f3d23e00640 1 --2- 172.21.15.28:0/2441764584 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634fd51a910 0x5634fd4f4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:34.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.528+0000 7f3d29800640 1 -- 172.21.15.28:0/2441764584 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5634fd4fb330 msgr2=0x5634fd4f37e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:34.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.528+0000 7f3d29800640 1 --2- 172.21.15.28:0/2441764584 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5634fd4fb330 0x5634fd4f37e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:34.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.528+0000 7f3d29800640 1 -- 172.21.15.28:0/2441764584 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634fd51a910 msgr2=0x5634fd4f4230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:34.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.528+0000 7f3d29800640 1 --2- 172.21.15.28:0/2441764584 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634fd51a910 0x5634fd4f4230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:34.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.528+0000 7f3d29800640 1 -- 172.21.15.28:0/2441764584 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5634fd23d6b0 con 0x5634fd5366b0 2024-02-21T23:09:34.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.528+0000 7f3d28e00640 1 --2- 172.21.15.28:0/2441764584 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5634fd4fb330 0x5634fd4f37e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:09:34.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.529+0000 7f3d29800640 1 --2- 172.21.15.28:0/2441764584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634fd5366b0 0x5634fd4f4770 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c07ae90 tx=0x7f3d1c0a6040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c95b40a266773581 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:34.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.529+0000 7f3d10c00640 1 -- 172.21.15.28:0/2441764584 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d1c0838a0 con 0x5634fd5366b0 2024-02-21T23:09:34.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.529+0000 7f3d10c00640 1 -- 172.21.15.28:0/2441764584 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d1c097070 con 0x5634fd5366b0 2024-02-21T23:09:34.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.529+0000 7f3d10c00640 1 -- 172.21.15.28:0/2441764584 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d1c03c350 con 0x5634fd5366b0 2024-02-21T23:09:34.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.529+0000 7f3d2e634740 1 -- 172.21.15.28:0/2441764584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634fd5366b0 msgr2=0x5634fd4f4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:34.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.529+0000 7f3d2e634740 1 --2- 172.21.15.28:0/2441764584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634fd5366b0 0x5634fd4f4770 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c07ae90 tx=0x7f3d1c0a6040 comp rx=0 tx=0).stop 2024-02-21T23:09:34.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.529+0000 7f3d2e634740 1 -- 172.21.15.28:0/2441764584 shutdown_connections 2024-02-21T23:09:34.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.529+0000 7f3d2e634740 1 --2- 172.21.15.28:0/2441764584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634fd5366b0 0x5634fd4f4770 unknown :-1 s=CLOSED pgs=1595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:34.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.529+0000 7f3d2e634740 1 --2- 172.21.15.28:0/2441764584 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634fd51a910 0x5634fd4f4230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:34.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.529+0000 7f3d2e634740 1 --2- 172.21.15.28:0/2441764584 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5634fd4fb330 0x5634fd4f37e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:34.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.529+0000 7f3d2e634740 1 -- 172.21.15.28:0/2441764584 >> 172.21.15.28:0/2441764584 conn(0x5634fd4fa020 msgr2=0x5634fd549970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:34.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.529+0000 7f3d2e634740 1 -- 172.21.15.28:0/2441764584 shutdown_connections 2024-02-21T23:09:34.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.529+0000 7f3d2e634740 1 -- 172.21.15.28:0/2441764584 wait complete. 2024-02-21T23:09:34.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.530+0000 7f3d2e634740 1 -- start start 2024-02-21T23:09:34.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.530+0000 7f3d2e634740 1 Processor -- start 2024-02-21T23:09:34.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.531+0000 7f3d2e634740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634fd4fb330 0x5634fd538030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:34.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.531+0000 7f3d2e634740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634fd51a910 0x5634fd559190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:34.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.531+0000 7f3d2e634740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5634fd5366b0 0x5634fd5a1a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:34.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.531+0000 7f3d2e634740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634fd3fd040 con 0x5634fd4fb330 2024-02-21T23:09:34.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.531+0000 7f3d2e634740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634fd533940 con 0x5634fd51a910 2024-02-21T23:09:34.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.531+0000 7f3d2e634740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x5634fd45e470 con 0x5634fd5366b0 2024-02-21T23:09:34.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.531+0000 7f3d2e634740 10 client.0 _ll_register_callbacks cb 0x7f3d1c007310 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:09:34.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.531+0000 7f3d28e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634fd4fb330 0x5634fd538030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:34.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.532+0000 7f3d28e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634fd4fb330 0x5634fd538030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57466/0 (socket says 172.21.15.28:57466) 2024-02-21T23:09:34.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.532+0000 7f3d28e00640 1 -- 172.21.15.28:0/419446482 learned_addr learned my addr 172.21.15.28:0/419446482 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:34.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.532+0000 7f3d23e00640 1 --2- 172.21.15.28:0/419446482 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634fd51a910 0x5634fd559190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:34.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.532+0000 7f3d28e00640 1 -- 172.21.15.28:0/419446482 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5634fd5366b0 msgr2=0x5634fd5a1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:34.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.532+0000 7f3d28e00640 1 --2- 172.21.15.28:0/419446482 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5634fd5366b0 0x5634fd5a1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:34.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.532+0000 7f3d28e00640 1 -- 172.21.15.28:0/419446482 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634fd51a910 msgr2=0x5634fd559190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:34.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.532+0000 7f3d28e00640 1 --2- 172.21.15.28:0/419446482 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634fd51a910 0x5634fd559190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:34.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.532+0000 7f3d28e00640 1 -- 172.21.15.28:0/419446482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5634fd4f3d90 con 0x5634fd4fb330 2024-02-21T23:09:34.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.532+0000 7f3d28e00640 1 --2- 172.21.15.28:0/419446482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634fd4fb330 0x5634fd538030 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c07b350 tx=0x7f3d180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ef7dfb32b209a57 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:34.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.532+0000 7f3d06000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:09:34.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.532+0000 7f3d06000640 1 -- 172.21.15.28:0/419446482 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d1800f040 con 0x5634fd4fb330 2024-02-21T23:09:34.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.533+0000 7f3d06000640 1 -- 172.21.15.28:0/419446482 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(140..140 src has 1..140) v4 ==== 6331+0+0 (secure 0 0 0) 0x7f3d1800e790 con 0x5634fd4fb330 2024-02-21T23:09:34.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.533+0000 7f3d2e634740 1 -- 172.21.15.28:0/419446482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3d1c097070 con 0x5634fd4fb330 2024-02-21T23:09:34.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.533+0000 7f3d06000640 1 -- 172.21.15.28:0/419446482 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=149) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3d18036050 con 0x5634fd4fb330 2024-02-21T23:09:34.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.533+0000 7f3d2e634740 10 client.8664 fetch_fsmap learned FSMap version 149 2024-02-21T23:09:34.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.533+0000 7f3d2e634740 1 -- 172.21.15.28:0/419446482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=149,monmap=2+}) v3 -- 0x5634fd4f5130 con 0x5634fd4fb330 2024-02-21T23:09:34.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.533+0000 7f3d06000640 1 -- 172.21.15.28:0/419446482 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d1c097070 con 0x5634fd4fb330 2024-02-21T23:09:34.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.533+0000 7f3d06000640 1 -- 172.21.15.28:0/419446482 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 149) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f3d18039020 con 0x5634fd4fb330 2024-02-21T23:09:34.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.533+0000 7f3d2e634740 10 client.8664 fetch_fsmap finished waiting for FSMap version 149 2024-02-21T23:09:34.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.533+0000 7f3d2e634740 10 client.8664 resolve_mds: resolved name 'd' to daemon mds.d 2024-02-21T23:09:34.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.533+0000 7f3d2e634740 20 client.8664 populate_metadata read hostname 'smithi028' 2024-02-21T23:09:34.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.533+0000 7f3d2e634740 1 --2- 172.21.15.28:0/419446482 >> [v2:172.21.15.81:6835/2095908433,v1:172.21.15.81:6837/2095908433] conn(0x5634fd5a7de0 0x5634fd5a81c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:34.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.533+0000 7f3d2e634740 4 client.8664 mds_command: new command op to 6143 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-21T23:09:34.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.533+0000 7f3d2e634740 1 -- 172.21.15.28:0/419446482 --> [v2:172.21.15.81:6835/2095908433,v1:172.21.15.81:6837/2095908433] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5634fd465d00 con 0x5634fd5a7de0 2024-02-21T23:09:34.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.534+0000 7f3d23e00640 1 --2- 172.21.15.28:0/419446482 >> [v2:172.21.15.81:6835/2095908433,v1:172.21.15.81:6837/2095908433] conn(0x5634fd5a7de0 0x5634fd5a81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:34.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.535+0000 7f3d23e00640 1 --2- 172.21.15.28:0/419446482 >> [v2:172.21.15.81:6835/2095908433,v1:172.21.15.81:6837/2095908433] conn(0x5634fd5a7de0 0x5634fd5a81c0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3dd68adea9150431 server_cookie=4d65be016fa755d in_seq=0 out_seq=0 2024-02-21T23:09:34.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.535+0000 7f3d06000640 10 client.8664 ms_handle_connect on v2:172.21.15.81:6835/2095908433 2024-02-21T23:09:34.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.535+0000 7f3d06000640 1 -- 172.21.15.28:0/419446482 <== mds.0 v2:172.21.15.81:6835/2095908433 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+334 (crc 0 0 0) 0x5634fd465d00 con 0x5634fd5a7de0 2024-02-21T23:09:34.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.535+0000 7f3d06000640 10 client.8664 handle_command_reply: tid=0 2024-02-21T23:09:34.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.535+0000 7f3d2e634740 1 client.8664 shutdown 2024-02-21T23:09:34.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.535+0000 7f3d2e634740 1 -- 172.21.15.28:0/419446482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634fd4fb330 msgr2=0x5634fd538030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:34.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.535+0000 7f3d2e634740 1 --2- 172.21.15.28:0/419446482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634fd4fb330 0x5634fd538030 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c07b350 tx=0x7f3d180046c0 comp rx=0 tx=0).stop 2024-02-21T23:09:34.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.536+0000 7f3d2e634740 1 -- 172.21.15.28:0/419446482 shutdown_connections 2024-02-21T23:09:34.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.536+0000 7f3d2e634740 1 --2- 172.21.15.28:0/419446482 >> [v2:172.21.15.81:6835/2095908433,v1:172.21.15.81:6837/2095908433] conn(0x5634fd5a7de0 0x5634fd5a81c0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:34.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.536+0000 7f3d2e634740 1 --2- 172.21.15.28:0/419446482 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5634fd5366b0 0x5634fd5a1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:34.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.536+0000 7f3d2e634740 1 --2- 172.21.15.28:0/419446482 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5634fd51a910 0x5634fd559190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:34.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.536+0000 7f3d2e634740 1 --2- 172.21.15.28:0/419446482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634fd4fb330 0x5634fd538030 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:34.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.536+0000 7f3d06000640 0 client.8664 ms_handle_reset on v2:172.21.15.81:6835/2095908433 2024-02-21T23:09:34.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.536+0000 7f3d2e634740 1 -- 172.21.15.28:0/419446482 >> 172.21.15.28:0/419446482 conn(0x5634fd4fa020 msgr2=0x5634fd51b7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:34.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.536+0000 7f3d2e634740 1 -- 172.21.15.28:0/419446482 shutdown_connections 2024-02-21T23:09:34.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.536+0000 7f3d2e634740 1 -- 172.21.15.28:0/419446482 wait complete. 2024-02-21T23:09:34.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.536+0000 7f3d2e634740 20 client.8664 trim_cache size 0 max 16384 2024-02-21T23:09:34.527 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-21T23:09:34.527 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac", 2024-02-21T23:09:34.527 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-02-21T23:09:34.528 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 6143, 2024-02-21T23:09:34.528 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-02-21T23:09:34.528 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-02-21T23:09:34.528 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-02-21T23:09:34.528 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 4.9069608880000004, 2024-02-21T23:09:34.528 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 149, 2024-02-21T23:09:34.528 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 140, 2024-02-21T23:09:34.528 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 140, 2024-02-21T23:09:34.528 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 43.098276802999997 2024-02-21T23:09:34.528 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-21T23:09:34.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.536+0000 7f3d2a200640 1 -- 172.21.15.28:0/1042361954 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3d0806b4d0 msgr2=0x7f3d0806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:34.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.536+0000 7f3d2a200640 1 --2- 172.21.15.28:0/1042361954 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3d0806b4d0 0x7f3d0806d990 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f3d18006fd0 tx=0x7f3d18008040 comp rx=0 tx=0).stop 2024-02-21T23:09:34.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.536+0000 7f3d2a200640 1 -- 172.21.15.28:0/1042361954 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d241527a0 msgr2=0x7f3d2414d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:34.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.536+0000 7f3d2a200640 1 --2- 172.21.15.28:0/1042361954 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d241527a0 0x7f3d2414d550 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c009f40 tx=0x7f3d1c004a40 comp rx=0 tx=0).stop 2024-02-21T23:09:34.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.537+0000 7f3d2a200640 1 -- 172.21.15.28:0/1042361954 shutdown_connections 2024-02-21T23:09:34.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.537+0000 7f3d2a200640 1 --2- 172.21.15.28:0/1042361954 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3d0806b4d0 0x7f3d0806d990 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:34.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.537+0000 7f3d2a200640 1 --2- 172.21.15.28:0/1042361954 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d241527a0 0x7f3d2414d550 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:34.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.537+0000 7f3d2a200640 1 --2- 172.21.15.28:0/1042361954 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d240bf9a0 0x7f3d240a6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:34.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.537+0000 7f3d2a200640 1 --2- 172.21.15.28:0/1042361954 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3d2406d410 0x7f3d241ee7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:34.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.537+0000 7f3d2a200640 1 -- 172.21.15.28:0/1042361954 >> 172.21.15.28:0/1042361954 conn(0x7f3d240a8320 msgr2=0x7f3d240a8700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:34.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.537+0000 7f3d2a200640 1 -- 172.21.15.28:0/1042361954 shutdown_connections 2024-02-21T23:09:34.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:34.537+0000 7f3d2a200640 1 -- 172.21.15.28:0/1042361954 wait complete. 2024-02-21T23:09:34.541 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 10, "epoch": 149, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 10, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-02-21T23:09:03.869294+0000", "damaged": [], "data_pools": [ 22 ], "enabled": true, "epoch": 149, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6143": { "addr": "172.21.15.81:6837/2095908433", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6835", "nonce": 2095908433, "type": "v2" }, { "addr": "172.21.15.81:6837", "nonce": 2095908433, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6143, "incarnation": 146, "join_fscid": 10, "name": "d", "rank": 0, "state": "up:active", "state_seq": 13 } }, "last_failure": 0, "last_failure_osd_epoch": 140, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 21, "modified": "2024-02-21T23:09:31.652165+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6143 } } } ], "standbys": [ { "addr": "172.21.15.81:6836/878375431", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6834", "nonce": 878375431, "type": "v2" }, { "addr": "172.21.15.81:6836", "nonce": 878375431, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 147, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6934, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.28:6833/3934548735", "addrs": { "addrvec": [ { "addr": "172.21.15.28:6832", "nonce": 3934548735, "type": "v2" }, { "addr": "172.21.15.28:6833", "nonce": 3934548735, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 133, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8370, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.28:6835/2100132822", "addrs": { "addrvec": [ { "addr": "172.21.15.28:6834", "nonce": 2100132822, "type": "v2" }, { "addr": "172.21.15.28:6835", "nonce": 2100132822, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 142, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8556, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-02-21T23:09:34.542 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': 10, 'name': 'd', 'state': 'up:active'}] 2024-02-21T23:09:34.542 INFO:tasks.cephfs.test_failover:current = [{'gid': 8556, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.28:6835/2100132822', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 2100132822}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 2100132822}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 142}, {'gid': 6934, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.81:6836/878375431', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6834', 'nonce': 878375431}, {'type': 'v1', 'addr': '172.21.15.81:6836', 'nonce': 878375431}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 147}, {'gid': 8370, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.28:6833/3934548735', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 3934548735}, {'type': 'v1', 'addr': '172.21.15.28:6833', 'nonce': 3934548735}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 133}, {'gid': 6143, 'name': 'd', 'rank': 0, 'incarnation': 146, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.81:6837/2095908433', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6835', 'nonce': 2095908433}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 2095908433}]}, 'join_fscid': 10, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}] 2024-02-21T23:09:34.543 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-02-21T23:09:34.543 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-02-21T23:09:34.543 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-21T23:09:34.656 DEBUG:teuthology.orchestra.run.smithi081:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-21T23:09:34.784 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-02-21T23:09:34.784 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-21T23:09:34.784 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-21T23:09:34.843 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054608,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-02-21T23:09:34.843 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-02-21T23:09:34.844 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-21T23:09:34.896 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053833,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498812k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249404k,nr_inodes=812351,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498812k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-02-21T23:09:34.896 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:09:35.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.136+0000 7f4bed400640 1 Processor -- start 2024-02-21T23:09:35.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.137+0000 7f4bed400640 1 -- start start 2024-02-21T23:09:35.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.138+0000 7f4bed400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4be806ee40 0x7f4be80a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:35.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.138+0000 7f4bed400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f4be8156af0 0x7f4be806dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:35.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.138+0000 7f4bed400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4be80c2760 con 0x7f4be8150450 2024-02-21T23:09:35.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.138+0000 7f4bed400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4be80c22c0 con 0x7f4be8156af0 2024-02-21T23:09:35.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.138+0000 7f4bed400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4be8156660 con 0x7f4be806ee40 2024-02-21T23:09:35.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.138+0000 7f4beca00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f4be8156af0 0x7f4be806dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:35.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.139+0000 7f4beca00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f4be8156af0 0x7f4be806dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.28:43250/0 (socket says 172.21.15.28:43250) 2024-02-21T23:09:35.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.139+0000 7f4beca00640 1 -- 172.21.15.28:0/86724724 learned_addr learned my addr 172.21.15.28:0/86724724 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:35.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.139+0000 7f4be7400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4be806ee40 0x7f4be80a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:35.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.139+0000 7f4be7400640 1 -- 172.21.15.28:0/86724724 >> v2:172.21.15.81:3300/0 conn(0x7f4be8156af0 msgr2=0x7f4be806dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:35.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.139+0000 7f4be7400640 1 --2- 172.21.15.28:0/86724724 >> v2:172.21.15.81:3300/0 conn(0x7f4be8156af0 0x7f4be806dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:35.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.139+0000 7f4be7400640 1 -- 172.21.15.28:0/86724724 >> v1:172.21.15.81:6789/0 conn(0x7f4be8150450 legacy=0x7f4be81536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:35.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.139+0000 7f4be7400640 1 -- 172.21.15.28:0/86724724 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4be8156310 con 0x7f4be806ee40 2024-02-21T23:09:35.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.140+0000 7f4be7400640 1 --2- 172.21.15.28:0/86724724 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4be806ee40 0x7f4be80a5fd0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4bd4009d90 tx=0x7f4bd402bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b367debc03294f9 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:35.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.141+0000 7f4be6a00640 1 -- 172.21.15.28:0/86724724 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd4039070 con 0x7f4be806ee40 2024-02-21T23:09:35.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.141+0000 7f4be6a00640 1 -- 172.21.15.28:0/86724724 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bd402cc50 con 0x7f4be806ee40 2024-02-21T23:09:35.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.141+0000 7f4be6a00640 1 -- 172.21.15.28:0/86724724 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd4034440 con 0x7f4be806ee40 2024-02-21T23:09:35.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.141+0000 7f4bed400640 1 -- 172.21.15.28:0/86724724 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4be806ee40 msgr2=0x7f4be80a5fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:35.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.141+0000 7f4bed400640 1 --2- 172.21.15.28:0/86724724 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4be806ee40 0x7f4be80a5fd0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4bd4009d90 tx=0x7f4bd402bea0 comp rx=0 tx=0).stop 2024-02-21T23:09:35.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.141+0000 7f4bed400640 1 -- 172.21.15.28:0/86724724 shutdown_connections 2024-02-21T23:09:35.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.141+0000 7f4bed400640 1 --2- 172.21.15.28:0/86724724 >> v2:172.21.15.81:3300/0 conn(0x7f4be8156af0 0x7f4be806dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:35.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.141+0000 7f4bed400640 1 --2- 172.21.15.28:0/86724724 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4be806ee40 0x7f4be80a5fd0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:35.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.141+0000 7f4bed400640 1 -- 172.21.15.28:0/86724724 >> 172.21.15.28:0/86724724 conn(0x7f4be806b630 msgr2=0x7f4be80c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:35.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.142+0000 7f4bed400640 1 -- 172.21.15.28:0/86724724 shutdown_connections 2024-02-21T23:09:35.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.142+0000 7f4bed400640 1 -- 172.21.15.28:0/86724724 wait complete. 2024-02-21T23:09:35.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.143+0000 7f4bed400640 1 Processor -- start 2024-02-21T23:09:35.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.143+0000 7f4bed400640 1 -- start start 2024-02-21T23:09:35.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.144+0000 7f4bed400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4be806ee40 0x7f4be8155a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:35.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.144+0000 7f4bed400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4be8150450 0x7f4be8143dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:35.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.144+0000 7f4bed400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4be8156af0 0x7f4be80c4180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:35.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.144+0000 7f4bed400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4be8156f80 con 0x7f4be806ee40 2024-02-21T23:09:35.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.144+0000 7f4bed400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4be80c2cb0 con 0x7f4be8150450 2024-02-21T23:09:35.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.144+0000 7f4bed400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4be8157100 con 0x7f4be8156af0 2024-02-21T23:09:35.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.145+0000 7f4be7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4be806ee40 0x7f4be8155a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:35.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.145+0000 7f4be7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4be806ee40 0x7f4be8155a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60994/0 (socket says 172.21.15.28:60994) 2024-02-21T23:09:35.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.145+0000 7f4be7e00640 1 -- 172.21.15.28:0/3192585341 learned_addr learned my addr 172.21.15.28:0/3192585341 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:35.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.145+0000 7f4beca00640 1 --2- 172.21.15.28:0/3192585341 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4be8156af0 0x7f4be80c4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:35.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.145+0000 7f4be7400640 1 --2- 172.21.15.28:0/3192585341 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4be8150450 0x7f4be8143dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:35.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.145+0000 7f4be7e00640 1 -- 172.21.15.28:0/3192585341 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4be8156af0 msgr2=0x7f4be80c4180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:35.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.145+0000 7f4be7e00640 1 --2- 172.21.15.28:0/3192585341 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4be8156af0 0x7f4be80c4180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:35.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.145+0000 7f4be7e00640 1 -- 172.21.15.28:0/3192585341 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4be8150450 msgr2=0x7f4be8143dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:35.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.145+0000 7f4be7e00640 1 --2- 172.21.15.28:0/3192585341 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4be8150450 0x7f4be8143dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:35.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.145+0000 7f4be7e00640 1 -- 172.21.15.28:0/3192585341 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4be8144310 con 0x7f4be806ee40 2024-02-21T23:09:35.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.145+0000 7f4be7400640 1 --2- 172.21.15.28:0/3192585341 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4be8150450 0x7f4be8143dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:09:35.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.146+0000 7f4be7e00640 1 --2- 172.21.15.28:0/3192585341 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4be806ee40 0x7f4be8155a30 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f4bd800c450 tx=0x7f4bd800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:35.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.146+0000 7f4bdfe00640 1 -- 172.21.15.28:0/3192585341 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd80052d0 con 0x7f4be806ee40 2024-02-21T23:09:35.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.146+0000 7f4bdfe00640 1 -- 172.21.15.28:0/3192585341 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bd8005470 con 0x7f4be806ee40 2024-02-21T23:09:35.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.146+0000 7f4bdfe00640 1 -- 172.21.15.28:0/3192585341 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd8012870 con 0x7f4be806ee40 2024-02-21T23:09:35.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.146+0000 7f4bed400640 1 -- 172.21.15.28:0/3192585341 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4be8144600 con 0x7f4be806ee40 2024-02-21T23:09:35.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.146+0000 7f4bed400640 1 -- 172.21.15.28:0/3192585341 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4be8145070 con 0x7f4be806ee40 2024-02-21T23:09:35.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.147+0000 7f4bed400640 1 -- 172.21.15.28:0/3192585341 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4be804f290 con 0x7f4be806ee40 2024-02-21T23:09:35.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.148+0000 7f4bdfe00640 1 -- 172.21.15.28:0/3192585341 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4bd8012a10 con 0x7f4be806ee40 2024-02-21T23:09:35.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.149+0000 7f4bdfe00640 1 --2- 172.21.15.28:0/3192585341 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4bc806b4d0 0x7f4bc806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:35.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.149+0000 7f4bdfe00640 1 -- 172.21.15.28:0/3192585341 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6331+0+0 (secure 0 0 0) 0x7f4bd808a010 con 0x7f4be806ee40 2024-02-21T23:09:35.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.150+0000 7f4be7400640 1 --2- 172.21.15.28:0/3192585341 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4bc806b4d0 0x7f4bc806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:35.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.151+0000 7f4be7400640 1 --2- 172.21.15.28:0/3192585341 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4bc806b4d0 0x7f4bc806d990 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f4bd4036000 tx=0x7f4bd4030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:35.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.153+0000 7f4bdfe00640 1 -- 172.21.15.28:0/3192585341 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4bd8058c10 con 0x7f4be806ee40 2024-02-21T23:09:35.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.396+0000 7f4bed400640 1 -- 172.21.15.28:0/3192585341 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4be80ac890 con 0x7f4be806ee40 2024-02-21T23:09:35.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.397+0000 7f4bdfe00640 1 -- 172.21.15.28:0/3192585341 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 149 v149) v1 ==== 95+0+4948 (secure 0 0 0) 0x7f4bd8058510 con 0x7f4be806ee40 2024-02-21T23:09:35.386 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:09:35.386 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":149,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6934,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/878375431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":878375431},{"type":"v1","addr":"172.21.15.81:6836","nonce":878375431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":147},{"gid":8370,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/3934548735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3934548735},{"type":"v1","addr":"172.21.15.28:6833","nonce":3934548735}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":133},{"gid":8556,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2100132822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2100132822},{"type":"v1","addr":"172.21.15.28:6835","nonce":2100132822}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":142}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:09:03.869294+0000","modified":"2024-02-21T23:09:31.652165+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":140,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6143},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6143":{"gid":6143,"name":"d","rank":0,"incarnation":146,"state":"up:active","state_seq":13,"addr":"172.21.15.81:6837/2095908433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":2095908433},{"type":"v1","addr":"172.21.15.81:6837","nonce":2095908433}]},"join_fscid":10,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-02-21T23:09:35.386 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 149 2024-02-21T23:09:35.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.400+0000 7f4bed400640 1 -- 172.21.15.28:0/3192585341 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4bc806b4d0 msgr2=0x7f4bc806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:35.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.400+0000 7f4bed400640 1 --2- 172.21.15.28:0/3192585341 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4bc806b4d0 0x7f4bc806d990 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f4bd4036000 tx=0x7f4bd4030000 comp rx=0 tx=0).stop 2024-02-21T23:09:35.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.400+0000 7f4bed400640 1 -- 172.21.15.28:0/3192585341 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4be806ee40 msgr2=0x7f4be8155a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:35.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.400+0000 7f4bed400640 1 --2- 172.21.15.28:0/3192585341 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4be806ee40 0x7f4be8155a30 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f4bd800c450 tx=0x7f4bd800c910 comp rx=0 tx=0).stop 2024-02-21T23:09:35.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.400+0000 7f4bed400640 1 -- 172.21.15.28:0/3192585341 shutdown_connections 2024-02-21T23:09:35.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.400+0000 7f4bed400640 1 --2- 172.21.15.28:0/3192585341 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4bc806b4d0 0x7f4bc806d990 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:35.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.400+0000 7f4bed400640 1 --2- 172.21.15.28:0/3192585341 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4be8156af0 0x7f4be80c4180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:35.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.400+0000 7f4bed400640 1 --2- 172.21.15.28:0/3192585341 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4be8150450 0x7f4be8143dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:35.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.400+0000 7f4bed400640 1 --2- 172.21.15.28:0/3192585341 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4be806ee40 0x7f4be8155a30 unknown :-1 s=CLOSED pgs=1597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:35.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.400+0000 7f4bed400640 1 -- 172.21.15.28:0/3192585341 >> 172.21.15.28:0/3192585341 conn(0x7f4be806b630 msgr2=0x7f4be80a6600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:35.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.400+0000 7f4bed400640 1 -- 172.21.15.28:0/3192585341 shutdown_connections 2024-02-21T23:09:35.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.400+0000 7f4bed400640 1 -- 172.21.15.28:0/3192585341 wait complete. 2024-02-21T23:09:35.409 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:09:35.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.636+0000 7fbb97200640 1 Processor -- start 2024-02-21T23:09:35.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.636+0000 7fbb97200640 1 -- start start 2024-02-21T23:09:35.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.638+0000 7fbb97200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fbb9814f0b0 0x7fbb98155180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:35.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.638+0000 7fbb97200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbb981521c0 0x7fbb98152610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:35.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.638+0000 7fbb97200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb98155760 con 0x7fbb980a6160 2024-02-21T23:09:35.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.638+0000 7fbb97200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb98152b50 con 0x7fbb981521c0 2024-02-21T23:09:35.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.638+0000 7fbb97200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb98151d30 con 0x7fbb9814f0b0 2024-02-21T23:09:35.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.638+0000 7fbb96800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbb981521c0 0x7fbb98152610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:35.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.638+0000 7fbb96800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbb981521c0 0x7fbb98152610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32770/0 (socket says 172.21.15.28:32770) 2024-02-21T23:09:35.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.638+0000 7fbb96800640 1 -- 172.21.15.28:0/3737285333 learned_addr learned my addr 172.21.15.28:0/3737285333 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:35.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.638+0000 7fbb95e00640 1 --2- 172.21.15.28:0/3737285333 >> v2:172.21.15.81:3300/0 conn(0x7fbb9814f0b0 0x7fbb98155180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:35.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.639+0000 7fbb96800640 1 -- 172.21.15.28:0/3737285333 >> v2:172.21.15.81:3300/0 conn(0x7fbb9814f0b0 msgr2=0x7fbb98155180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:35.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.639+0000 7fbb96800640 1 --2- 172.21.15.28:0/3737285333 >> v2:172.21.15.81:3300/0 conn(0x7fbb9814f0b0 0x7fbb98155180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:35.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.639+0000 7fbb96800640 1 -- 172.21.15.28:0/3737285333 >> v1:172.21.15.81:6789/0 conn(0x7fbb980a6160 legacy=0x7fbb980c2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:35.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.639+0000 7fbb96800640 1 -- 172.21.15.28:0/3737285333 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb98155a40 con 0x7fbb981521c0 2024-02-21T23:09:35.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.640+0000 7fbb96800640 1 --2- 172.21.15.28:0/3737285333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb981521c0 0x7fbb98152610 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7fbb98151eb0 tx=0x7fbb9002e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38a00a9043abbca7 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:35.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.640+0000 7fbb95400640 1 -- 172.21.15.28:0/3737285333 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb9002ec80 con 0x7fbb981521c0 2024-02-21T23:09:35.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.640+0000 7fbb95400640 1 -- 172.21.15.28:0/3737285333 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb9002ee20 con 0x7fbb981521c0 2024-02-21T23:09:35.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.640+0000 7fbb95400640 1 -- 172.21.15.28:0/3737285333 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb90036610 con 0x7fbb981521c0 2024-02-21T23:09:35.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.641+0000 7fbb97200640 1 -- 172.21.15.28:0/3737285333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb981521c0 msgr2=0x7fbb98152610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:35.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.641+0000 7fbb97200640 1 --2- 172.21.15.28:0/3737285333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb981521c0 0x7fbb98152610 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7fbb98151eb0 tx=0x7fbb9002e0e0 comp rx=0 tx=0).stop 2024-02-21T23:09:35.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.641+0000 7fbb97200640 1 -- 172.21.15.28:0/3737285333 shutdown_connections 2024-02-21T23:09:35.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.641+0000 7fbb97200640 1 --2- 172.21.15.28:0/3737285333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb981521c0 0x7fbb98152610 unknown :-1 s=CLOSED pgs=1598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:35.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.641+0000 7fbb97200640 1 --2- 172.21.15.28:0/3737285333 >> v2:172.21.15.81:3300/0 conn(0x7fbb9814f0b0 0x7fbb98155180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:35.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.641+0000 7fbb97200640 1 -- 172.21.15.28:0/3737285333 >> 172.21.15.28:0/3737285333 conn(0x7fbb9806b630 msgr2=0x7fbb980c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:35.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.641+0000 7fbb97200640 1 -- 172.21.15.28:0/3737285333 shutdown_connections 2024-02-21T23:09:35.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.641+0000 7fbb97200640 1 -- 172.21.15.28:0/3737285333 wait complete. 2024-02-21T23:09:35.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.643+0000 7fbb97200640 1 Processor -- start 2024-02-21T23:09:35.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.643+0000 7fbb97200640 1 -- start start 2024-02-21T23:09:35.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.644+0000 7fbb97200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbb980a6160 0x7fbb981eb340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:35.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.644+0000 7fbb97200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb9814f0b0 0x7fbb9814d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:35.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.644+0000 7fbb97200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb981521c0 0x7fbb98153750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:35.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.644+0000 7fbb97200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb980a68d0 con 0x7fbb9814f0b0 2024-02-21T23:09:35.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.644+0000 7fbb97200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb98156000 con 0x7fbb981521c0 2024-02-21T23:09:35.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.644+0000 7fbb97200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb980c2720 con 0x7fbb980a6160 2024-02-21T23:09:35.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.644+0000 7fbb8d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb9814f0b0 0x7fbb9814d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:35.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.644+0000 7fbb8d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb9814f0b0 0x7fbb9814d950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32774/0 (socket says 172.21.15.28:32774) 2024-02-21T23:09:35.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.644+0000 7fbb8d400640 1 -- 172.21.15.28:0/1904817554 learned_addr learned my addr 172.21.15.28:0/1904817554 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:35.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.645+0000 7fbb96800640 1 --2- 172.21.15.28:0/1904817554 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb981521c0 0x7fbb98153750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:35.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.645+0000 7fbb95e00640 1 --2- 172.21.15.28:0/1904817554 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbb980a6160 0x7fbb981eb340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:35.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.645+0000 7fbb8d400640 1 -- 172.21.15.28:0/1904817554 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbb980a6160 msgr2=0x7fbb981eb340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:35.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.645+0000 7fbb8d400640 1 --2- 172.21.15.28:0/1904817554 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbb980a6160 0x7fbb981eb340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:35.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.645+0000 7fbb8d400640 1 -- 172.21.15.28:0/1904817554 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb981521c0 msgr2=0x7fbb98153750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:35.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.645+0000 7fbb8d400640 1 --2- 172.21.15.28:0/1904817554 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb981521c0 0x7fbb98153750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:35.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.645+0000 7fbb8d400640 1 -- 172.21.15.28:0/1904817554 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb9814e780 con 0x7fbb9814f0b0 2024-02-21T23:09:35.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.645+0000 7fbb95e00640 1 --2- 172.21.15.28:0/1904817554 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbb980a6160 0x7fbb981eb340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:09:35.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.645+0000 7fbb8d400640 1 --2- 172.21.15.28:0/1904817554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb9814f0b0 0x7fbb9814d950 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7fbb8000c460 tx=0x7fbb8000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:35.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.645+0000 7fbb8ea00640 1 -- 172.21.15.28:0/1904817554 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb80016020 con 0x7fbb9814f0b0 2024-02-21T23:09:35.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.645+0000 7fbb8ea00640 1 -- 172.21.15.28:0/1904817554 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb80012870 con 0x7fbb9814f0b0 2024-02-21T23:09:35.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.645+0000 7fbb8ea00640 1 -- 172.21.15.28:0/1904817554 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb80012c60 con 0x7fbb9814f0b0 2024-02-21T23:09:35.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.646+0000 7fbb97200640 1 -- 172.21.15.28:0/1904817554 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb981514e0 con 0x7fbb9814f0b0 2024-02-21T23:09:35.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.646+0000 7fbb97200640 1 -- 172.21.15.28:0/1904817554 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb981517c0 con 0x7fbb9814f0b0 2024-02-21T23:09:35.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.647+0000 7fbb97200640 1 -- 172.21.15.28:0/1904817554 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb9804f290 con 0x7fbb9814f0b0 2024-02-21T23:09:35.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.648+0000 7fbb8ea00640 1 -- 172.21.15.28:0/1904817554 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbb80019020 con 0x7fbb9814f0b0 2024-02-21T23:09:35.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.649+0000 7fbb8ea00640 1 --2- 172.21.15.28:0/1904817554 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbb7406b500 0x7fbb7406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:35.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.649+0000 7fbb8ea00640 1 -- 172.21.15.28:0/1904817554 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6331+0+0 (secure 0 0 0) 0x7fbb8008a130 con 0x7fbb9814f0b0 2024-02-21T23:09:35.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.649+0000 7fbb95e00640 1 --2- 172.21.15.28:0/1904817554 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbb7406b500 0x7fbb7406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:35.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.650+0000 7fbb95e00640 1 --2- 172.21.15.28:0/1904817554 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbb7406b500 0x7fbb7406d9c0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fbb88006fd0 tx=0x7fbb88008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:35.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.653+0000 7fbb8ea00640 1 -- 172.21.15.28:0/1904817554 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fbb80020030 con 0x7fbb9814f0b0 2024-02-21T23:09:35.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.854+0000 7fbb97200640 1 -- 172.21.15.28:0/1904817554 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbb980c29f0 con 0x7fbb9814f0b0 2024-02-21T23:09:35.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.857+0000 7fbb8ea00640 1 -- 172.21.15.28:0/1904817554 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v140) v1 ==== 74+0+17247 (secure 0 0 0) 0x7fbb80058d30 con 0x7fbb9814f0b0 2024-02-21T23:09:35.847 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:09:35.847 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":140,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:09:29.622962+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:09:01.777487+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"135","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2024-02-21T23:09:02.904570+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"136","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"22.7","mappings":[{"from":3,"to":0}]},{"pgid":"22.e","mappings":[{"from":3,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.81:6836/4241906475":"2024-02-22T23:09:29.622886+0000","172.21.15.81:6834/4241906475":"2024-02-22T23:09:29.622886+0000","172.21.15.28:6835/4151254007":"2024-02-22T23:09:15.528381+0000","172.21.15.28:6834/4151254007":"2024-02-22T23:09:15.528381+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:09:35.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.859+0000 7fbb97200640 1 -- 172.21.15.28:0/1904817554 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbb7406b500 msgr2=0x7fbb7406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:35.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.859+0000 7fbb97200640 1 --2- 172.21.15.28:0/1904817554 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbb7406b500 0x7fbb7406d9c0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fbb88006fd0 tx=0x7fbb88008040 comp rx=0 tx=0).stop 2024-02-21T23:09:35.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.859+0000 7fbb97200640 1 -- 172.21.15.28:0/1904817554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb9814f0b0 msgr2=0x7fbb9814d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:35.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.859+0000 7fbb97200640 1 --2- 172.21.15.28:0/1904817554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb9814f0b0 0x7fbb9814d950 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7fbb8000c460 tx=0x7fbb8000c920 comp rx=0 tx=0).stop 2024-02-21T23:09:35.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.859+0000 7fbb97200640 1 -- 172.21.15.28:0/1904817554 shutdown_connections 2024-02-21T23:09:35.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.859+0000 7fbb97200640 1 --2- 172.21.15.28:0/1904817554 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbb7406b500 0x7fbb7406d9c0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:35.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.859+0000 7fbb97200640 1 --2- 172.21.15.28:0/1904817554 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbb981521c0 0x7fbb98153750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:35.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.859+0000 7fbb97200640 1 --2- 172.21.15.28:0/1904817554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb9814f0b0 0x7fbb9814d950 unknown :-1 s=CLOSED pgs=1599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:35.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.859+0000 7fbb97200640 1 --2- 172.21.15.28:0/1904817554 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbb980a6160 0x7fbb981eb340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:35.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.859+0000 7fbb97200640 1 -- 172.21.15.28:0/1904817554 >> 172.21.15.28:0/1904817554 conn(0x7fbb9806b630 msgr2=0x7fbb980c2f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:35.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.860+0000 7fbb97200640 1 -- 172.21.15.28:0/1904817554 shutdown_connections 2024-02-21T23:09:35.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:35.860+0000 7fbb97200640 1 -- 172.21.15.28:0/1904817554 wait complete. 2024-02-21T23:09:35.868 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-21T23:02:30.728329+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 21, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-21T23:09:01.777487+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '135', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 22, 'pool_name': 'cephfs_data', 'create_time': '2024-02-21T23:09:02.904570+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '136', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-21T23:09:35.869 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-21T23:09:36.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.095+0000 7fa343600640 1 Processor -- start 2024-02-21T23:09:36.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.096+0000 7fa343600640 1 -- start start 2024-02-21T23:09:36.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.097+0000 7fa343600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa33c0c5f50 0x7fa33c14fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:36.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.097+0000 7fa343600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa33c1509e0 con 0x7fa33c06b0e0 2024-02-21T23:09:36.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.097+0000 7fa343600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa33c152fb0 con 0x7fa33c150530 2024-02-21T23:09:36.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.097+0000 7fa343600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa33c153bc0 con 0x7fa33c0c5f50 2024-02-21T23:09:36.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.097+0000 7fa340c00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa33c06b0e0 0x7fa33c0c5840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60100/0 (socket says 172.21.15.28:60100) 2024-02-21T23:09:36.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.097+0000 7fa340c00640 1 -- 172.21.15.28:0/280891313 learned_addr learned my addr 172.21.15.28:0/280891313 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:09:36.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.098+0000 7fa33be00640 1 --2- 172.21.15.28:0/280891313 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa33c0c5f50 0x7fa33c14fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:36.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.098+0000 7fa33b400640 1 -- 172.21.15.28:0/280891313 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 115509318 0 0) 0x7fa33c1509e0 con 0x7fa33c06b0e0 2024-02-21T23:09:36.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.098+0000 7fa33b400640 1 -- 172.21.15.28:0/280891313 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa324003520 con 0x7fa33c06b0e0 2024-02-21T23:09:36.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.098+0000 7fa33b400640 1 -- 172.21.15.28:0/280891313 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1923475634 0 0) 0x7fa324003520 con 0x7fa33c06b0e0 2024-02-21T23:09:36.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.098+0000 7fa33b400640 1 -- 172.21.15.28:0/280891313 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa33c0c5f50 msgr2=0x7fa33c14fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:36.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.098+0000 7fa33b400640 1 --2- 172.21.15.28:0/280891313 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa33c0c5f50 0x7fa33c14fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:36.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.099+0000 7fa33b400640 1 -- 172.21.15.28:0/280891313 >> v1:172.21.15.81:6789/0 conn(0x7fa33c150530 legacy=0x7fa33c156500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:36.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.099+0000 7fa33b400640 1 -- 172.21.15.28:0/280891313 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa33c153290 con 0x7fa33c06b0e0 2024-02-21T23:09:36.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.099+0000 7fa33b400640 1 -- 172.21.15.28:0/280891313 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fa330001670 con 0x7fa33c06b0e0 2024-02-21T23:09:36.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.099+0000 7fa33b400640 1 -- 172.21.15.28:0/280891313 >> v1:172.21.15.28:6789/0 conn(0x7fa33c06b0e0 legacy=0x7fa33c0c5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:36.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.099+0000 7fa33b400640 1 --2- 172.21.15.28:0/280891313 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa3240039e0 0x7fa324003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:36.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.099+0000 7fa33b400640 1 --2- 172.21.15.28:0/280891313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa324004800 0x7fa324004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:36.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.099+0000 7fa33b400640 1 --2- 172.21.15.28:0/280891313 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa33c0c5f50 0x7fa324005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:36.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.099+0000 7fa33b400640 1 -- 172.21.15.28:0/280891313 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa33c153bc0 con 0x7fa324004800 2024-02-21T23:09:36.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.099+0000 7fa33b400640 1 -- 172.21.15.28:0/280891313 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa32400baf0 con 0x7fa3240039e0 2024-02-21T23:09:36.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.099+0000 7fa33b400640 1 -- 172.21.15.28:0/280891313 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa32400bcf0 con 0x7fa33c0c5f50 2024-02-21T23:09:36.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.099+0000 7fa33be00640 1 --2- 172.21.15.28:0/280891313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa324004800 0x7fa324004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:36.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.099+0000 7fa340c00640 1 --2- 172.21.15.28:0/280891313 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa3240039e0 0x7fa324003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:36.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.100+0000 7fa33be00640 1 -- 172.21.15.28:0/280891313 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa33c0c5f50 msgr2=0x7fa324005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:36.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.100+0000 7fa341600640 1 --2- 172.21.15.28:0/280891313 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa33c0c5f50 0x7fa324005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:36.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.100+0000 7fa33be00640 1 --2- 172.21.15.28:0/280891313 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa33c0c5f50 0x7fa324005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:36.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.100+0000 7fa33be00640 1 -- 172.21.15.28:0/280891313 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa3240039e0 msgr2=0x7fa324003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:36.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.100+0000 7fa33be00640 1 --2- 172.21.15.28:0/280891313 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa3240039e0 0x7fa324003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:36.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.100+0000 7fa33be00640 1 -- 172.21.15.28:0/280891313 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa324003520 con 0x7fa324004800 2024-02-21T23:09:36.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.100+0000 7fa341600640 1 --2- 172.21.15.28:0/280891313 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa33c0c5f50 0x7fa324005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:09:36.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.101+0000 7fa33be00640 1 --2- 172.21.15.28:0/280891313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa324004800 0x7fa324004c80 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7fa328009ac0 tx=0x7fa324005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f1e9abc08084619 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:36.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.101+0000 7fa33b400640 1 -- 172.21.15.28:0/280891313 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa328039070 con 0x7fa324004800 2024-02-21T23:09:36.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.101+0000 7fa33b400640 1 -- 172.21.15.28:0/280891313 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa32802c6b0 con 0x7fa324004800 2024-02-21T23:09:36.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.101+0000 7fa33b400640 1 -- 172.21.15.28:0/280891313 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa328009690 con 0x7fa324004800 2024-02-21T23:09:36.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.102+0000 7fa343600640 1 -- 172.21.15.28:0/280891313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa324004800 msgr2=0x7fa324004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:36.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.102+0000 7fa343600640 1 --2- 172.21.15.28:0/280891313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa324004800 0x7fa324004c80 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7fa328009ac0 tx=0x7fa324005830 comp rx=0 tx=0).stop 2024-02-21T23:09:36.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.102+0000 7fa341600640 1 -- 172.21.15.28:0/280891313 reap_dead start 2024-02-21T23:09:36.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.102+0000 7fa343600640 1 -- 172.21.15.28:0/280891313 shutdown_connections 2024-02-21T23:09:36.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.102+0000 7fa343600640 1 -- 172.21.15.28:0/280891313 >> 172.21.15.28:0/280891313 conn(0x7fa33c0a85a0 msgr2=0x7fa33c0a8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:36.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.102+0000 7fa343600640 1 -- 172.21.15.28:0/280891313 shutdown_connections 2024-02-21T23:09:36.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.102+0000 7fa343600640 1 -- 172.21.15.28:0/280891313 wait complete. 2024-02-21T23:09:36.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.104+0000 7fa343600640 1 Processor -- start 2024-02-21T23:09:36.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.104+0000 7fa343600640 1 -- start start 2024-02-21T23:09:36.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.105+0000 7fa343600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa33c0bfe90 0x7fa33c1eb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:36.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.105+0000 7fa343600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa33c1eeda0 0x7fa33c150c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:36.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.105+0000 7fa343600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa33c149a40 0x7fa33c149ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:36.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.105+0000 7fa343600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa33c06b4c0 con 0x7fa33c1eeda0 2024-02-21T23:09:36.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.105+0000 7fa343600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa33c153850 con 0x7fa33c0bfe90 2024-02-21T23:09:36.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.105+0000 7fa343600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa33c06b640 con 0x7fa33c149a40 2024-02-21T23:09:36.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.105+0000 7fa33be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa33c1eeda0 0x7fa33c150c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:36.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.105+0000 7fa33be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa33c1eeda0 0x7fa33c150c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32794/0 (socket says 172.21.15.28:32794) 2024-02-21T23:09:36.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.105+0000 7fa33be00640 1 -- 172.21.15.28:0/559855485 learned_addr learned my addr 172.21.15.28:0/559855485 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:36.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.105+0000 7fa341600640 1 --2- 172.21.15.28:0/559855485 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa33c149a40 0x7fa33c149ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:36.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.105+0000 7fa340c00640 1 --2- 172.21.15.28:0/559855485 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa33c0bfe90 0x7fa33c1eb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:36.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.105+0000 7fa33be00640 1 -- 172.21.15.28:0/559855485 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa33c149a40 msgr2=0x7fa33c149ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:36.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.105+0000 7fa33be00640 1 --2- 172.21.15.28:0/559855485 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa33c149a40 0x7fa33c149ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:36.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.105+0000 7fa33be00640 1 -- 172.21.15.28:0/559855485 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa33c0bfe90 msgr2=0x7fa33c1eb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:36.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.105+0000 7fa33be00640 1 --2- 172.21.15.28:0/559855485 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa33c0bfe90 0x7fa33c1eb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:36.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.105+0000 7fa33be00640 1 -- 172.21.15.28:0/559855485 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa33c06e900 con 0x7fa33c1eeda0 2024-02-21T23:09:36.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.106+0000 7fa33be00640 1 --2- 172.21.15.28:0/559855485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa33c1eeda0 0x7fa33c150c10 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7fa32802c2a0 tx=0x7fa328004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:36.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.106+0000 7fa338c00640 1 -- 172.21.15.28:0/559855485 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa328039070 con 0x7fa33c1eeda0 2024-02-21T23:09:36.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.106+0000 7fa338c00640 1 -- 172.21.15.28:0/559855485 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa328043070 con 0x7fa33c1eeda0 2024-02-21T23:09:36.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.106+0000 7fa338c00640 1 -- 172.21.15.28:0/559855485 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3280044e0 con 0x7fa33c1eeda0 2024-02-21T23:09:36.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.106+0000 7fa343600640 1 -- 172.21.15.28:0/559855485 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa33c06e620 con 0x7fa33c1eeda0 2024-02-21T23:09:36.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.106+0000 7fa343600640 1 -- 172.21.15.28:0/559855485 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa33c1eb1e0 con 0x7fa33c1eeda0 2024-02-21T23:09:36.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.107+0000 7fa343600640 1 -- 172.21.15.28:0/559855485 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa33c0c53f0 con 0x7fa33c1eeda0 2024-02-21T23:09:36.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.108+0000 7fa338c00640 1 -- 172.21.15.28:0/559855485 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa328003a10 con 0x7fa33c1eeda0 2024-02-21T23:09:36.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.109+0000 7fa338c00640 1 --2- 172.21.15.28:0/559855485 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa31c06b520 0x7fa31c06d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:36.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.109+0000 7fa338c00640 1 -- 172.21.15.28:0/559855485 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6331+0+0 (secure 0 0 0) 0x7fa3280ab6c0 con 0x7fa33c1eeda0 2024-02-21T23:09:36.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.110+0000 7fa340c00640 1 --2- 172.21.15.28:0/559855485 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa31c06b520 0x7fa31c06d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:36.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.111+0000 7fa340c00640 1 --2- 172.21.15.28:0/559855485 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa31c06b520 0x7fa31c06d9e0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fa330005ab0 tx=0x7fa330009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:36.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.113+0000 7fa338c00640 1 -- 172.21.15.28:0/559855485 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fa33c0c53f0 con 0x7fa33c1eeda0 2024-02-21T23:09:36.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.307+0000 7fa343600640 1 -- 172.21.15.28:0/559855485 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa33c14be70 con 0x7fa33c1eeda0 2024-02-21T23:09:36.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.308+0000 7fa338c00640 1 -- 172.21.15.28:0/559855485 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v140) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa33c14be70 con 0x7fa33c1eeda0 2024-02-21T23:09:36.297 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-02-21T23:09:36.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.311+0000 7fa343600640 1 -- 172.21.15.28:0/559855485 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa31c06b520 msgr2=0x7fa31c06d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:36.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.311+0000 7fa343600640 1 --2- 172.21.15.28:0/559855485 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa31c06b520 0x7fa31c06d9e0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fa330005ab0 tx=0x7fa330009040 comp rx=0 tx=0).stop 2024-02-21T23:09:36.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.311+0000 7fa343600640 1 -- 172.21.15.28:0/559855485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa33c1eeda0 msgr2=0x7fa33c150c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:36.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.311+0000 7fa343600640 1 --2- 172.21.15.28:0/559855485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa33c1eeda0 0x7fa33c150c10 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7fa32802c2a0 tx=0x7fa328004730 comp rx=0 tx=0).stop 2024-02-21T23:09:36.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.311+0000 7fa343600640 1 -- 172.21.15.28:0/559855485 shutdown_connections 2024-02-21T23:09:36.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.311+0000 7fa343600640 1 --2- 172.21.15.28:0/559855485 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa31c06b520 0x7fa31c06d9e0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:36.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.311+0000 7fa343600640 1 --2- 172.21.15.28:0/559855485 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa33c149a40 0x7fa33c149ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:36.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.311+0000 7fa343600640 1 --2- 172.21.15.28:0/559855485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa33c1eeda0 0x7fa33c150c10 unknown :-1 s=CLOSED pgs=1602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:36.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.311+0000 7fa343600640 1 --2- 172.21.15.28:0/559855485 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa33c0bfe90 0x7fa33c1eb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:36.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.311+0000 7fa343600640 1 -- 172.21.15.28:0/559855485 >> 172.21.15.28:0/559855485 conn(0x7fa33c0a85a0 msgr2=0x7fa33c14e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:36.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.312+0000 7fa343600640 1 -- 172.21.15.28:0/559855485 shutdown_connections 2024-02-21T23:09:36.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.312+0000 7fa343600640 1 -- 172.21.15.28:0/559855485 wait complete. 2024-02-21T23:09:36.320 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-21T23:09:36.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.563+0000 7f4a8ac00640 1 Processor -- start 2024-02-21T23:09:36.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.563+0000 7f4a8ac00640 1 -- start start 2024-02-21T23:09:36.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.564+0000 7f4a8ac00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4a840c4580 0x7f4a840c12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:36.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.564+0000 7f4a8ac00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f4a840c2c10 0x7f4a8406ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:36.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.564+0000 7f4a8ac00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a8406e220 con 0x7f4a84150240 2024-02-21T23:09:36.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.564+0000 7f4a8ac00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a84150620 con 0x7f4a840c2c10 2024-02-21T23:09:36.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.564+0000 7f4a8ac00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a84153a90 con 0x7f4a840c4580 2024-02-21T23:09:36.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.565+0000 7f4a89800640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f4a84150240 0x7f4a8406db10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.28:46222/0 (socket says 172.21.15.28:46222) 2024-02-21T23:09:36.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.565+0000 7f4a89800640 1 -- 172.21.15.28:0/3565529937 learned_addr learned my addr 172.21.15.28:0/3565529937 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:09:36.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.565+0000 7f4a88e00640 1 --2- 172.21.15.28:0/3565529937 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4a840c4580 0x7f4a840c12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:36.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.565+0000 7f4a8a200640 1 --2- 172.21.15.28:0/3565529937 >> v2:172.21.15.81:3300/0 conn(0x7f4a840c2c10 0x7f4a8406ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:36.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.566+0000 7f4a83e00640 1 -- 172.21.15.28:0/3565529937 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2091518201 0 0) 0x7f4a8406e220 con 0x7f4a84150240 2024-02-21T23:09:36.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.566+0000 7f4a83e00640 1 -- 172.21.15.28:0/3565529937 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a68003520 con 0x7f4a84150240 2024-02-21T23:09:36.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.566+0000 7f4a8a200640 1 -- 172.21.15.28:0/3565529937 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4a840c4580 msgr2=0x7f4a840c12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:36.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.566+0000 7f4a8a200640 1 --2- 172.21.15.28:0/3565529937 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4a840c4580 0x7f4a840c12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:36.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.566+0000 7f4a8a200640 1 -- 172.21.15.28:0/3565529937 >> v1:172.21.15.81:6789/0 conn(0x7f4a84150240 legacy=0x7f4a8406db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:36.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.566+0000 7f4a8a200640 1 -- 172.21.15.28:0/3565529937 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a8406f780 con 0x7f4a840c2c10 2024-02-21T23:09:36.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.567+0000 7f4a8a200640 1 --2- 172.21.15.28:0/3565529937 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a840c2c10 0x7f4a8406ae70 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f4a7c009d90 tx=0x7f4a7c02be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a4f1677e35c01acd server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:36.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.568+0000 7f4a83e00640 1 -- 172.21.15.28:0/3565529937 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a7c039070 con 0x7f4a840c2c10 2024-02-21T23:09:36.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.568+0000 7f4a83e00640 1 -- 172.21.15.28:0/3565529937 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a7c02cbc0 con 0x7f4a840c2c10 2024-02-21T23:09:36.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.568+0000 7f4a83e00640 1 -- 172.21.15.28:0/3565529937 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a7c034440 con 0x7f4a840c2c10 2024-02-21T23:09:36.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.569+0000 7f4a8ac00640 1 -- 172.21.15.28:0/3565529937 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a840c2c10 msgr2=0x7f4a8406ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:36.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.569+0000 7f4a8ac00640 1 --2- 172.21.15.28:0/3565529937 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a840c2c10 0x7f4a8406ae70 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f4a7c009d90 tx=0x7f4a7c02be10 comp rx=0 tx=0).stop 2024-02-21T23:09:36.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.569+0000 7f4a8ac00640 1 -- 172.21.15.28:0/3565529937 shutdown_connections 2024-02-21T23:09:36.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.569+0000 7f4a8ac00640 1 --2- 172.21.15.28:0/3565529937 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a840c2c10 0x7f4a8406ae70 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:36.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.569+0000 7f4a8ac00640 1 --2- 172.21.15.28:0/3565529937 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4a840c4580 0x7f4a840c12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:36.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.569+0000 7f4a8ac00640 1 -- 172.21.15.28:0/3565529937 >> 172.21.15.28:0/3565529937 conn(0x7f4a840a8530 msgr2=0x7f4a840a8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:36.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.569+0000 7f4a8ac00640 1 -- 172.21.15.28:0/3565529937 shutdown_connections 2024-02-21T23:09:36.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.569+0000 7f4a8ac00640 1 -- 172.21.15.28:0/3565529937 wait complete. 2024-02-21T23:09:36.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.571+0000 7f4a8ac00640 1 Processor -- start 2024-02-21T23:09:36.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.571+0000 7f4a8ac00640 1 -- start start 2024-02-21T23:09:36.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.572+0000 7f4a8ac00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4a840c2c10 0x7f4a84154a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:36.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.572+0000 7f4a8ac00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a840c4580 0x7f4a841e18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:36.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.572+0000 7f4a8ac00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a84150240 0x7f4a84151de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:36.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.572+0000 7f4a8ac00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a840c4a10 con 0x7f4a840c4580 2024-02-21T23:09:36.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.572+0000 7f4a8ac00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a84153720 con 0x7f4a84150240 2024-02-21T23:09:36.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.572+0000 7f4a8ac00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a840c4b90 con 0x7f4a840c2c10 2024-02-21T23:09:36.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.572+0000 7f4a88e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a840c4580 0x7f4a841e18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:36.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.572+0000 7f4a88e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a840c4580 0x7f4a841e18a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32800/0 (socket says 172.21.15.28:32800) 2024-02-21T23:09:36.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.572+0000 7f4a88e00640 1 -- 172.21.15.28:0/3336511564 learned_addr learned my addr 172.21.15.28:0/3336511564 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:36.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.572+0000 7f4a8a200640 1 --2- 172.21.15.28:0/3336511564 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a84150240 0x7f4a84151de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:36.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.573+0000 7f4a89800640 1 --2- 172.21.15.28:0/3336511564 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4a840c2c10 0x7f4a84154a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:36.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.573+0000 7f4a88e00640 1 -- 172.21.15.28:0/3336511564 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4a840c2c10 msgr2=0x7f4a84154a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:36.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.573+0000 7f4a88e00640 1 --2- 172.21.15.28:0/3336511564 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4a840c2c10 0x7f4a84154a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:36.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.573+0000 7f4a88e00640 1 -- 172.21.15.28:0/3336511564 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a84150240 msgr2=0x7f4a84151de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:36.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.573+0000 7f4a88e00640 1 --2- 172.21.15.28:0/3336511564 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a84150240 0x7f4a84151de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:36.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.573+0000 7f4a88e00640 1 -- 172.21.15.28:0/3336511564 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a84157020 con 0x7f4a840c4580 2024-02-21T23:09:36.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.574+0000 7f4a88e00640 1 --2- 172.21.15.28:0/3336511564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a840c4580 0x7f4a841e18a0 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f4a7000e980 tx=0x7f4a7000ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:36.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.574+0000 7f4a81600640 1 -- 172.21.15.28:0/3336511564 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a70017070 con 0x7f4a840c4580 2024-02-21T23:09:36.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.574+0000 7f4a81600640 1 -- 172.21.15.28:0/3336511564 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a7000ca00 con 0x7f4a840c4580 2024-02-21T23:09:36.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.574+0000 7f4a81600640 1 -- 172.21.15.28:0/3336511564 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a7000cdd0 con 0x7f4a840c4580 2024-02-21T23:09:36.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.574+0000 7f4a8ac00640 1 -- 172.21.15.28:0/3336511564 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a840a9af0 con 0x7f4a840c4580 2024-02-21T23:09:36.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.574+0000 7f4a8ac00640 1 -- 172.21.15.28:0/3336511564 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a840a9d90 con 0x7f4a840c4580 2024-02-21T23:09:36.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.576+0000 7f4a8ac00640 1 -- 172.21.15.28:0/3336511564 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a8404f290 con 0x7f4a840c4580 2024-02-21T23:09:36.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.576+0000 7f4a81600640 1 -- 172.21.15.28:0/3336511564 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4a7001a070 con 0x7f4a840c4580 2024-02-21T23:09:36.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.578+0000 7f4a81600640 1 --2- 172.21.15.28:0/3336511564 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4a6406b4d0 0x7f4a6406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:36.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.578+0000 7f4a81600640 1 -- 172.21.15.28:0/3336511564 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6331+0+0 (secure 0 0 0) 0x7f4a7008a3d0 con 0x7f4a840c4580 2024-02-21T23:09:36.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.578+0000 7f4a89800640 1 --2- 172.21.15.28:0/3336511564 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4a6406b4d0 0x7f4a6406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:36.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.580+0000 7f4a89800640 1 --2- 172.21.15.28:0/3336511564 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4a6406b4d0 0x7f4a6406d990 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f4a740011e0 tx=0x7f4a74007d30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:36.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.582+0000 7f4a81600640 1 -- 172.21.15.28:0/3336511564 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4a70058980 con 0x7f4a840c4580 2024-02-21T23:09:36.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.783+0000 7f4a8ac00640 1 -- 172.21.15.28:0/3336511564 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4a8406f0e0 con 0x7f4a840c4580 2024-02-21T23:09:36.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.784+0000 7f4a81600640 1 -- 172.21.15.28:0/3336511564 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v140) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4a7005ca20 con 0x7f4a840c4580 2024-02-21T23:09:36.774 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-02-21T23:09:36.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.787+0000 7f4a8ac00640 1 -- 172.21.15.28:0/3336511564 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4a6406b4d0 msgr2=0x7f4a6406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:36.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.787+0000 7f4a8ac00640 1 --2- 172.21.15.28:0/3336511564 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4a6406b4d0 0x7f4a6406d990 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f4a740011e0 tx=0x7f4a74007d30 comp rx=0 tx=0).stop 2024-02-21T23:09:36.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.787+0000 7f4a8ac00640 1 -- 172.21.15.28:0/3336511564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a840c4580 msgr2=0x7f4a841e18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:36.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.787+0000 7f4a8ac00640 1 --2- 172.21.15.28:0/3336511564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a840c4580 0x7f4a841e18a0 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f4a7000e980 tx=0x7f4a7000ee40 comp rx=0 tx=0).stop 2024-02-21T23:09:36.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.787+0000 7f4a8ac00640 1 -- 172.21.15.28:0/3336511564 shutdown_connections 2024-02-21T23:09:36.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.787+0000 7f4a8ac00640 1 --2- 172.21.15.28:0/3336511564 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4a6406b4d0 0x7f4a6406d990 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:36.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.787+0000 7f4a8ac00640 1 --2- 172.21.15.28:0/3336511564 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a84150240 0x7f4a84151de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:36.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.787+0000 7f4a8ac00640 1 --2- 172.21.15.28:0/3336511564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a840c4580 0x7f4a841e18a0 unknown :-1 s=CLOSED pgs=1603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:36.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.787+0000 7f4a8ac00640 1 --2- 172.21.15.28:0/3336511564 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4a840c2c10 0x7f4a84154a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:36.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.787+0000 7f4a8ac00640 1 -- 172.21.15.28:0/3336511564 >> 172.21.15.28:0/3336511564 conn(0x7f4a840a8530 msgr2=0x7f4a840c57c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:36.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.787+0000 7f4a8ac00640 1 -- 172.21.15.28:0/3336511564 shutdown_connections 2024-02-21T23:09:36.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:36.787+0000 7f4a8ac00640 1 -- 172.21.15.28:0/3336511564 wait complete. 2024-02-21T23:09:36.797 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-21T23:09:37.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.031+0000 7fe97be00640 1 Processor -- start 2024-02-21T23:09:37.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.031+0000 7fe97be00640 1 -- start start 2024-02-21T23:09:37.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.033+0000 7fe97be00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fe97c13e0c0 0x7fe97c0c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:37.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.033+0000 7fe97be00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe97c06dbc0 0x7fe97c06c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:37.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.033+0000 7fe97be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe97c0c76f0 0x7fe97c06c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:37.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.033+0000 7fe97be00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe97c0c89a0 con 0x7fe97c0c76f0 2024-02-21T23:09:37.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.033+0000 7fe97be00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe97c158550 con 0x7fe97c13e0c0 2024-02-21T23:09:37.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.033+0000 7fe97be00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe97c1587d0 con 0x7fe97c06dbc0 2024-02-21T23:09:37.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.034+0000 7fe97b400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe97c0c76f0 0x7fe97c06c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:37.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.034+0000 7fe97b400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe97c0c76f0 0x7fe97c06c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32804/0 (socket says 172.21.15.28:32804) 2024-02-21T23:09:37.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.034+0000 7fe97b400640 1 -- 172.21.15.28:0/3509179327 learned_addr learned my addr 172.21.15.28:0/3509179327 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:37.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.034+0000 7fe97a000640 1 --2- 172.21.15.28:0/3509179327 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe97c06dbc0 0x7fe97c06c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:37.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.034+0000 7fe97aa00640 1 --2- 172.21.15.28:0/3509179327 >> v2:172.21.15.81:3300/0 conn(0x7fe97c13e0c0 0x7fe97c0c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:37.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.034+0000 7fe97b400640 1 -- 172.21.15.28:0/3509179327 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe97c06dbc0 msgr2=0x7fe97c06c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:37.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.034+0000 7fe97b400640 1 --2- 172.21.15.28:0/3509179327 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe97c06dbc0 0x7fe97c06c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:37.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.034+0000 7fe97b400640 1 -- 172.21.15.28:0/3509179327 >> v2:172.21.15.81:3300/0 conn(0x7fe97c13e0c0 msgr2=0x7fe97c0c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:37.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.034+0000 7fe97b400640 1 --2- 172.21.15.28:0/3509179327 >> v2:172.21.15.81:3300/0 conn(0x7fe97c13e0c0 0x7fe97c0c71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:37.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.034+0000 7fe97b400640 1 -- 172.21.15.28:0/3509179327 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe97c158ab0 con 0x7fe97c0c76f0 2024-02-21T23:09:37.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.035+0000 7fe97b400640 1 --2- 172.21.15.28:0/3509179327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe97c0c76f0 0x7fe97c06c7c0 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7fe97c0ad180 tx=0x7fe97402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df447f7a970c4010 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:37.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.036+0000 7fe979600640 1 -- 172.21.15.28:0/3509179327 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe97402c9f0 con 0x7fe97c0c76f0 2024-02-21T23:09:37.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.036+0000 7fe979600640 1 -- 172.21.15.28:0/3509179327 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe97402cb90 con 0x7fe97c0c76f0 2024-02-21T23:09:37.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.036+0000 7fe979600640 1 -- 172.21.15.28:0/3509179327 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9740343f0 con 0x7fe97c0c76f0 2024-02-21T23:09:37.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.036+0000 7fe97be00640 1 -- 172.21.15.28:0/3509179327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe97c0c76f0 msgr2=0x7fe97c06c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:37.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.036+0000 7fe97be00640 1 --2- 172.21.15.28:0/3509179327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe97c0c76f0 0x7fe97c06c7c0 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7fe97c0ad180 tx=0x7fe97402be50 comp rx=0 tx=0).stop 2024-02-21T23:09:37.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.036+0000 7fe97be00640 1 -- 172.21.15.28:0/3509179327 shutdown_connections 2024-02-21T23:09:37.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.036+0000 7fe97be00640 1 --2- 172.21.15.28:0/3509179327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe97c0c76f0 0x7fe97c06c7c0 unknown :-1 s=CLOSED pgs=1604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:37.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.036+0000 7fe97be00640 1 --2- 172.21.15.28:0/3509179327 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe97c06dbc0 0x7fe97c06c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:37.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.036+0000 7fe97be00640 1 --2- 172.21.15.28:0/3509179327 >> v2:172.21.15.81:3300/0 conn(0x7fe97c13e0c0 0x7fe97c0c71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:37.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.036+0000 7fe97be00640 1 -- 172.21.15.28:0/3509179327 >> 172.21.15.28:0/3509179327 conn(0x7fe97c073110 msgr2=0x7fe97c0ac490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:37.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.036+0000 7fe97be00640 1 -- 172.21.15.28:0/3509179327 shutdown_connections 2024-02-21T23:09:37.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.036+0000 7fe97be00640 1 -- 172.21.15.28:0/3509179327 wait complete. 2024-02-21T23:09:37.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.038+0000 7fe97be00640 1 Processor -- start 2024-02-21T23:09:37.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.038+0000 7fe97be00640 1 -- start start 2024-02-21T23:09:37.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.039+0000 7fe97be00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe97c06dbc0 0x7fe97c1eac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:37.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.039+0000 7fe97be00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe97c0c76f0 0x7fe97c0af9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:37.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.039+0000 7fe97be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe97c13e0c0 0x7fe97c0ab5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:37.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.039+0000 7fe97be00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe97c06cd00 con 0x7fe97c13e0c0 2024-02-21T23:09:37.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.039+0000 7fe97be00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe97c159070 con 0x7fe97c06dbc0 2024-02-21T23:09:37.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.039+0000 7fe97be00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe97c06ce80 con 0x7fe97c0c76f0 2024-02-21T23:09:37.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.039+0000 7fe97b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe97c13e0c0 0x7fe97c0ab5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:37.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.040+0000 7fe97b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe97c13e0c0 0x7fe97c0ab5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32816/0 (socket says 172.21.15.28:32816) 2024-02-21T23:09:37.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.040+0000 7fe97b400640 1 -- 172.21.15.28:0/4140992104 learned_addr learned my addr 172.21.15.28:0/4140992104 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:37.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.040+0000 7fe97aa00640 1 --2- 172.21.15.28:0/4140992104 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe97c06dbc0 0x7fe97c1eac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:37.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.040+0000 7fe97a000640 1 --2- 172.21.15.28:0/4140992104 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe97c0c76f0 0x7fe97c0af9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:37.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.040+0000 7fe97b400640 1 -- 172.21.15.28:0/4140992104 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe97c0c76f0 msgr2=0x7fe97c0af9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:37.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.040+0000 7fe97b400640 1 --2- 172.21.15.28:0/4140992104 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe97c0c76f0 0x7fe97c0af9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:37.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.040+0000 7fe97b400640 1 -- 172.21.15.28:0/4140992104 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe97c06dbc0 msgr2=0x7fe97c1eac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:37.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.040+0000 7fe97b400640 1 --2- 172.21.15.28:0/4140992104 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe97c06dbc0 0x7fe97c1eac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:37.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.040+0000 7fe97b400640 1 -- 172.21.15.28:0/4140992104 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe97c0ac330 con 0x7fe97c13e0c0 2024-02-21T23:09:37.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.040+0000 7fe97b400640 1 --2- 172.21.15.28:0/4140992104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe97c13e0c0 0x7fe97c0ab5c0 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7fe974036040 tx=0x7fe974004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:37.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.041+0000 7fe973400640 1 -- 172.21.15.28:0/4140992104 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe974042020 con 0x7fe97c13e0c0 2024-02-21T23:09:37.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.041+0000 7fe973400640 1 -- 172.21.15.28:0/4140992104 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe97403e450 con 0x7fe97c13e0c0 2024-02-21T23:09:37.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.041+0000 7fe97be00640 1 -- 172.21.15.28:0/4140992104 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe97c155820 con 0x7fe97c13e0c0 2024-02-21T23:09:37.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.041+0000 7fe973400640 1 -- 172.21.15.28:0/4140992104 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe974009510 con 0x7fe97c13e0c0 2024-02-21T23:09:37.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.041+0000 7fe97be00640 1 -- 172.21.15.28:0/4140992104 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe97c1ee250 con 0x7fe97c13e0c0 2024-02-21T23:09:37.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.042+0000 7fe97be00640 1 -- 172.21.15.28:0/4140992104 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe97c04f290 con 0x7fe97c13e0c0 2024-02-21T23:09:37.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.043+0000 7fe973400640 1 -- 172.21.15.28:0/4140992104 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe974045020 con 0x7fe97c13e0c0 2024-02-21T23:09:37.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.044+0000 7fe973400640 1 --2- 172.21.15.28:0/4140992104 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe95c06b4d0 0x7fe95c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:37.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.044+0000 7fe973400640 1 -- 172.21.15.28:0/4140992104 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6331+0+0 (secure 0 0 0) 0x7fe9740acc70 con 0x7fe97c13e0c0 2024-02-21T23:09:37.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.044+0000 7fe97aa00640 1 --2- 172.21.15.28:0/4140992104 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe95c06b4d0 0x7fe95c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:37.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.046+0000 7fe97aa00640 1 --2- 172.21.15.28:0/4140992104 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe95c06b4d0 0x7fe95c06d990 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fe96c006fd0 tx=0x7fe96c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:37.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.049+0000 7fe973400640 1 -- 172.21.15.28:0/4140992104 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe97404c030 con 0x7fe97c13e0c0 2024-02-21T23:09:37.057 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-21T23:09:37.063 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-21T23:09:37.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.252+0000 7fe97be00640 1 -- 172.21.15.28:0/4140992104 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe97c06d9a0 con 0x7fe97c13e0c0 2024-02-21T23:09:37.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.253+0000 7fe973400640 1 -- 172.21.15.28:0/4140992104 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v140) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe97c0ac330 con 0x7fe97c13e0c0 2024-02-21T23:09:37.242 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-02-21T23:09:37.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.256+0000 7fe97be00640 1 -- 172.21.15.28:0/4140992104 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe95c06b4d0 msgr2=0x7fe95c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:37.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.256+0000 7fe97be00640 1 --2- 172.21.15.28:0/4140992104 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe95c06b4d0 0x7fe95c06d990 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fe96c006fd0 tx=0x7fe96c008040 comp rx=0 tx=0).stop 2024-02-21T23:09:37.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.256+0000 7fe97be00640 1 -- 172.21.15.28:0/4140992104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe97c13e0c0 msgr2=0x7fe97c0ab5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:37.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.256+0000 7fe97be00640 1 --2- 172.21.15.28:0/4140992104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe97c13e0c0 0x7fe97c0ab5c0 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7fe974036040 tx=0x7fe974004a40 comp rx=0 tx=0).stop 2024-02-21T23:09:37.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.256+0000 7fe97be00640 1 -- 172.21.15.28:0/4140992104 shutdown_connections 2024-02-21T23:09:37.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.256+0000 7fe97be00640 1 --2- 172.21.15.28:0/4140992104 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe95c06b4d0 0x7fe95c06d990 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:37.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.256+0000 7fe97be00640 1 --2- 172.21.15.28:0/4140992104 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe97c0c76f0 0x7fe97c0af9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:37.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.256+0000 7fe97be00640 1 --2- 172.21.15.28:0/4140992104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe97c13e0c0 0x7fe97c0ab5c0 unknown :-1 s=CLOSED pgs=1605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:37.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.256+0000 7fe97be00640 1 --2- 172.21.15.28:0/4140992104 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe97c06dbc0 0x7fe97c1eac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:37.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.256+0000 7fe97be00640 1 -- 172.21.15.28:0/4140992104 >> 172.21.15.28:0/4140992104 conn(0x7fe97c073110 msgr2=0x7fe97c0c9ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:37.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.256+0000 7fe97be00640 1 -- 172.21.15.28:0/4140992104 shutdown_connections 2024-02-21T23:09:37.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.256+0000 7fe97be00640 1 -- 172.21.15.28:0/4140992104 wait complete. 2024-02-21T23:09:37.265 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:09:37.313 INFO:tasks.ceph.mds.d.smithi081.stderr:2024-02-21T23:09:37.324+0000 7f2284000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-21T23:09:37.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.482+0000 7fc24f200640 1 Processor -- start 2024-02-21T23:09:37.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.483+0000 7fc24f200640 1 -- start start 2024-02-21T23:09:37.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.484+0000 7fc24f200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fc24806ee40 0x7fc2480a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:37.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.484+0000 7fc24f200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2480c2760 con 0x7fc248156af0 2024-02-21T23:09:37.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.484+0000 7fc24f200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc248156310 con 0x7fc248150450 2024-02-21T23:09:37.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.484+0000 7fc24f200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2480c22c0 con 0x7fc24806ee40 2024-02-21T23:09:37.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.485+0000 7fc24e800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fc248156af0 0x7fc24806dff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60110/0 (socket says 172.21.15.28:60110) 2024-02-21T23:09:37.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.485+0000 7fc24e800640 1 -- 172.21.15.28:0/1194977159 learned_addr learned my addr 172.21.15.28:0/1194977159 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:09:37.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.485+0000 7fc24d400640 1 --2- 172.21.15.28:0/1194977159 >> v2:172.21.15.81:3300/0 conn(0x7fc24806ee40 0x7fc2480a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:37.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.485+0000 7fc24ca00640 1 -- 172.21.15.28:0/1194977159 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1541067052 0 0) 0x7fc2480c2760 con 0x7fc248156af0 2024-02-21T23:09:37.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.485+0000 7fc24ca00640 1 -- 172.21.15.28:0/1194977159 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc230003520 con 0x7fc248156af0 2024-02-21T23:09:37.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.486+0000 7fc24ca00640 1 -- 172.21.15.28:0/1194977159 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1180628250 0 0) 0x7fc230003520 con 0x7fc248156af0 2024-02-21T23:09:37.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.486+0000 7fc24ca00640 1 -- 172.21.15.28:0/1194977159 >> v2:172.21.15.81:3300/0 conn(0x7fc24806ee40 msgr2=0x7fc2480a5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:37.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.486+0000 7fc24ca00640 1 --2- 172.21.15.28:0/1194977159 >> v2:172.21.15.81:3300/0 conn(0x7fc24806ee40 0x7fc2480a5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:37.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.486+0000 7fc24ca00640 1 -- 172.21.15.28:0/1194977159 >> v1:172.21.15.81:6789/0 conn(0x7fc248150450 legacy=0x7fc2481536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:37.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.486+0000 7fc24ca00640 1 -- 172.21.15.28:0/1194977159 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2480c29d0 con 0x7fc248156af0 2024-02-21T23:09:37.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.486+0000 7fc24ca00640 1 -- 172.21.15.28:0/1194977159 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fc2440019f0 con 0x7fc248156af0 2024-02-21T23:09:37.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.486+0000 7fc24ca00640 1 -- 172.21.15.28:0/1194977159 >> v1:172.21.15.28:6789/0 conn(0x7fc248156af0 legacy=0x7fc24806dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:37.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.486+0000 7fc24ca00640 1 --2- 172.21.15.28:0/1194977159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2300039b0 0x7fc230003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:37.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.486+0000 7fc24ca00640 1 --2- 172.21.15.28:0/1194977159 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc230004800 0x7fc230004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:37.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.486+0000 7fc24ca00640 1 --2- 172.21.15.28:0/1194977159 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2300051c0 0x7fc23000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:37.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.486+0000 7fc24ca00640 1 -- 172.21.15.28:0/1194977159 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc23000c030 con 0x7fc2300039b0 2024-02-21T23:09:37.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.486+0000 7fc24ca00640 1 -- 172.21.15.28:0/1194977159 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc23000c1e0 con 0x7fc2300051c0 2024-02-21T23:09:37.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.486+0000 7fc24ca00640 1 -- 172.21.15.28:0/1194977159 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc23000c3e0 con 0x7fc230004800 2024-02-21T23:09:37.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.486+0000 7fc24d400640 1 --2- 172.21.15.28:0/1194977159 >> v2:172.21.15.81:3300/0 conn(0x7fc24806ee40 0x7fc2480a5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-21T23:09:37.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.486+0000 7fc24de00640 1 --2- 172.21.15.28:0/1194977159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2300039b0 0x7fc230003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:37.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.486+0000 7fc24e800640 1 --2- 172.21.15.28:0/1194977159 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2300051c0 0x7fc23000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:37.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.487+0000 7fc24de00640 1 -- 172.21.15.28:0/1194977159 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc230004800 msgr2=0x7fc230004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:37.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.487+0000 7fc24de00640 1 --2- 172.21.15.28:0/1194977159 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc230004800 0x7fc230004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:37.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.487+0000 7fc24de00640 1 -- 172.21.15.28:0/1194977159 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2300051c0 msgr2=0x7fc23000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:37.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.487+0000 7fc24de00640 1 --2- 172.21.15.28:0/1194977159 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2300051c0 0x7fc23000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:37.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.487+0000 7fc24e800640 1 -- 172.21.15.28:0/1194977159 reap_dead start 2024-02-21T23:09:37.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.487+0000 7fc24de00640 1 -- 172.21.15.28:0/1194977159 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc230003520 con 0x7fc2300039b0 2024-02-21T23:09:37.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.488+0000 7fc24de00640 1 --2- 172.21.15.28:0/1194977159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2300039b0 0x7fc230003dd0 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7fc23c001750 tx=0x7fc230005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f60557e3d998870 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:37.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.488+0000 7fc24ca00640 1 -- 172.21.15.28:0/1194977159 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc23c001370 con 0x7fc2300039b0 2024-02-21T23:09:37.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.488+0000 7fc24ca00640 1 -- 172.21.15.28:0/1194977159 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc23c0052d0 con 0x7fc2300039b0 2024-02-21T23:09:37.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.488+0000 7fc24ca00640 1 -- 172.21.15.28:0/1194977159 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc23c003830 con 0x7fc2300039b0 2024-02-21T23:09:37.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.489+0000 7fc24f200640 1 -- 172.21.15.28:0/1194977159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2300039b0 msgr2=0x7fc230003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:37.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.489+0000 7fc24f200640 1 --2- 172.21.15.28:0/1194977159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2300039b0 0x7fc230003dd0 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7fc23c001750 tx=0x7fc230005830 comp rx=0 tx=0).stop 2024-02-21T23:09:37.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.489+0000 7fc24f200640 1 -- 172.21.15.28:0/1194977159 shutdown_connections 2024-02-21T23:09:37.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.489+0000 7fc24f200640 1 --2- 172.21.15.28:0/1194977159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2300039b0 0x7fc230003dd0 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:37.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.489+0000 7fc24f200640 1 -- 172.21.15.28:0/1194977159 >> 172.21.15.28:0/1194977159 conn(0x7fc24806b630 msgr2=0x7fc2480c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:37.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.489+0000 7fc24f200640 1 -- 172.21.15.28:0/1194977159 shutdown_connections 2024-02-21T23:09:37.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.489+0000 7fc24f200640 1 -- 172.21.15.28:0/1194977159 wait complete. 2024-02-21T23:09:37.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.490+0000 7fc24f200640 1 Processor -- start 2024-02-21T23:09:37.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.491+0000 7fc24f200640 1 -- start start 2024-02-21T23:09:37.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.492+0000 7fc24f200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc2300039b0 0x7fc2481551b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:37.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.492+0000 7fc24f200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc248145840 0x7fc2481e8700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:37.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.492+0000 7fc24f200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc248151540 0x7fc2481e8c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:37.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.492+0000 7fc24f200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc248156f80 con 0x7fc248145840 2024-02-21T23:09:37.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.492+0000 7fc24f200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc248156660 con 0x7fc248151540 2024-02-21T23:09:37.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.492+0000 7fc24f200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc248157100 con 0x7fc2300039b0 2024-02-21T23:09:37.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.492+0000 7fc24d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc248145840 0x7fc2481e8700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:37.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.492+0000 7fc24d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc248145840 0x7fc2481e8700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32822/0 (socket says 172.21.15.28:32822) 2024-02-21T23:09:37.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.492+0000 7fc24d400640 1 -- 172.21.15.28:0/3191794492 learned_addr learned my addr 172.21.15.28:0/3191794492 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:37.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.492+0000 7fc24de00640 1 --2- 172.21.15.28:0/3191794492 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc2300039b0 0x7fc2481551b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:37.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.492+0000 7fc24e800640 1 --2- 172.21.15.28:0/3191794492 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc248151540 0x7fc2481e8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:37.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.492+0000 7fc24d400640 1 -- 172.21.15.28:0/3191794492 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc2300039b0 msgr2=0x7fc2481551b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:37.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.492+0000 7fc24d400640 1 --2- 172.21.15.28:0/3191794492 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc2300039b0 0x7fc2481551b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:37.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.492+0000 7fc24d400640 1 -- 172.21.15.28:0/3191794492 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc248151540 msgr2=0x7fc2481e8c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:37.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.492+0000 7fc24d400640 1 --2- 172.21.15.28:0/3191794492 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc248151540 0x7fc2481e8c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:37.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.492+0000 7fc24d400640 1 -- 172.21.15.28:0/3191794492 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc248145560 con 0x7fc248145840 2024-02-21T23:09:37.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.492+0000 7fc24de00640 1 --2- 172.21.15.28:0/3191794492 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc2300039b0 0x7fc2481551b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:09:37.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.493+0000 7fc24d400640 1 --2- 172.21.15.28:0/3191794492 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc248145840 0x7fc2481e8700 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7fc23800e910 tx=0x7fc23800edd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:37.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.493+0000 7fc242000640 1 -- 172.21.15.28:0/3191794492 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc238004030 con 0x7fc248145840 2024-02-21T23:09:37.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.493+0000 7fc242000640 1 -- 172.21.15.28:0/3191794492 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2380041f0 con 0x7fc248145840 2024-02-21T23:09:37.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.493+0000 7fc24f200640 1 -- 172.21.15.28:0/3191794492 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc24814d250 con 0x7fc248145840 2024-02-21T23:09:37.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.493+0000 7fc242000640 1 -- 172.21.15.28:0/3191794492 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc23800c940 con 0x7fc248145840 2024-02-21T23:09:37.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.493+0000 7fc24f200640 1 -- 172.21.15.28:0/3191794492 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2481e9180 con 0x7fc248145840 2024-02-21T23:09:37.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.494+0000 7fc24f200640 1 -- 172.21.15.28:0/3191794492 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc24814d040 con 0x7fc248145840 2024-02-21T23:09:37.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.495+0000 7fc242000640 1 -- 172.21.15.28:0/3191794492 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc23800cae0 con 0x7fc248145840 2024-02-21T23:09:37.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.496+0000 7fc242000640 1 --2- 172.21.15.28:0/3191794492 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc22c06b500 0x7fc22c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:37.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.497+0000 7fc242000640 1 -- 172.21.15.28:0/3191794492 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6331+0+0 (secure 0 0 0) 0x7fc23808a830 con 0x7fc248145840 2024-02-21T23:09:37.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.497+0000 7fc24de00640 1 --2- 172.21.15.28:0/3191794492 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc22c06b500 0x7fc22c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:37.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.498+0000 7fc24de00640 1 --2- 172.21.15.28:0/3191794492 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc22c06b500 0x7fc22c06d9c0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fc23c033000 tx=0x7fc23c009030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:37.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.501+0000 7fc242000640 1 -- 172.21.15.28:0/3191794492 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc238017020 con 0x7fc248145840 2024-02-21T23:09:37.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.740+0000 7fc24f200640 1 -- 172.21.15.28:0/3191794492 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc24804f290 con 0x7fc248145840 2024-02-21T23:09:37.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.741+0000 7fc242000640 1 -- 172.21.15.28:0/3191794492 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 149 v149) v1 ==== 95+0+4948 (secure 0 0 0) 0x7fc238059430 con 0x7fc248145840 2024-02-21T23:09:37.730 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:09:37.730 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":149,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6934,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/878375431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":878375431},{"type":"v1","addr":"172.21.15.81:6836","nonce":878375431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":147},{"gid":8370,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/3934548735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3934548735},{"type":"v1","addr":"172.21.15.28:6833","nonce":3934548735}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":133},{"gid":8556,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2100132822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2100132822},{"type":"v1","addr":"172.21.15.28:6835","nonce":2100132822}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":142}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:09:03.869294+0000","modified":"2024-02-21T23:09:31.652165+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":140,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6143},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6143":{"gid":6143,"name":"d","rank":0,"incarnation":146,"state":"up:active","state_seq":13,"addr":"172.21.15.81:6837/2095908433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":2095908433},{"type":"v1","addr":"172.21.15.81:6837","nonce":2095908433}]},"join_fscid":10,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-02-21T23:09:37.731 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 149 2024-02-21T23:09:37.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.744+0000 7fc24f200640 1 -- 172.21.15.28:0/3191794492 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc22c06b500 msgr2=0x7fc22c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:37.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.744+0000 7fc24f200640 1 --2- 172.21.15.28:0/3191794492 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc22c06b500 0x7fc22c06d9c0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fc23c033000 tx=0x7fc23c009030 comp rx=0 tx=0).stop 2024-02-21T23:09:37.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.744+0000 7fc24f200640 1 -- 172.21.15.28:0/3191794492 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc248145840 msgr2=0x7fc2481e8700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:37.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.744+0000 7fc24f200640 1 --2- 172.21.15.28:0/3191794492 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc248145840 0x7fc2481e8700 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7fc23800e910 tx=0x7fc23800edd0 comp rx=0 tx=0).stop 2024-02-21T23:09:37.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.744+0000 7fc24f200640 1 -- 172.21.15.28:0/3191794492 shutdown_connections 2024-02-21T23:09:37.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.744+0000 7fc24f200640 1 --2- 172.21.15.28:0/3191794492 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc22c06b500 0x7fc22c06d9c0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:37.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.744+0000 7fc24f200640 1 --2- 172.21.15.28:0/3191794492 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc248151540 0x7fc2481e8c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:37.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.744+0000 7fc24f200640 1 --2- 172.21.15.28:0/3191794492 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc248145840 0x7fc2481e8700 unknown :-1 s=CLOSED pgs=1608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:37.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.744+0000 7fc24f200640 1 --2- 172.21.15.28:0/3191794492 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc2300039b0 0x7fc2481551b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:37.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.744+0000 7fc24f200640 1 -- 172.21.15.28:0/3191794492 >> 172.21.15.28:0/3191794492 conn(0x7fc24806b630 msgr2=0x7fc2480a7260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:37.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.744+0000 7fc24f200640 1 -- 172.21.15.28:0/3191794492 shutdown_connections 2024-02-21T23:09:37.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.744+0000 7fc24f200640 1 -- 172.21.15.28:0/3191794492 wait complete. 2024-02-21T23:09:37.753 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:09:37.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.981+0000 7f146a800640 1 Processor -- start 2024-02-21T23:09:37.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.981+0000 7f146a800640 1 -- start start 2024-02-21T23:09:37.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.982+0000 7f146a800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f146406ee40 0x7f14640a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:37.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.982+0000 7f146a800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1464156af0 0x7f146406dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:37.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.983+0000 7f146a800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14640c2760 con 0x7f1464150450 2024-02-21T23:09:37.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.983+0000 7f146a800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14640c22c0 con 0x7f146406ee40 2024-02-21T23:09:37.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.983+0000 7f146a800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1464156660 con 0x7f1464156af0 2024-02-21T23:09:37.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.983+0000 7f1468a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f146406ee40 0x7f14640a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:37.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.983+0000 7f1468a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f146406ee40 0x7f14640a5fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.28:43364/0 (socket says 172.21.15.28:43364) 2024-02-21T23:09:37.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.983+0000 7f1468a00640 1 -- 172.21.15.28:0/1085789518 learned_addr learned my addr 172.21.15.28:0/1085789518 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:37.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.983+0000 7f1469e00640 1 --2- 172.21.15.28:0/1085789518 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1464156af0 0x7f146406dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:37.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.984+0000 7f1468a00640 1 -- 172.21.15.28:0/1085789518 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1464156af0 msgr2=0x7f146406dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:37.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.984+0000 7f1468a00640 1 --2- 172.21.15.28:0/1085789518 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1464156af0 0x7f146406dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:37.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.984+0000 7f1468a00640 1 -- 172.21.15.28:0/1085789518 >> v1:172.21.15.81:6789/0 conn(0x7f1464150450 legacy=0x7f14641536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:37.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.984+0000 7f1468a00640 1 -- 172.21.15.28:0/1085789518 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1464156310 con 0x7f146406ee40 2024-02-21T23:09:37.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.985+0000 7f1468a00640 1 --2- 172.21.15.28:0/1085789518 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f146406ee40 0x7f14640a5fd0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f1450009870 tx=0x7f145002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b2c4ee2a72e6b26 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:37.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.986+0000 7f1463e00640 1 -- 172.21.15.28:0/1085789518 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1450034070 con 0x7f146406ee40 2024-02-21T23:09:37.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.986+0000 7f1463e00640 1 -- 172.21.15.28:0/1085789518 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1450030840 con 0x7f146406ee40 2024-02-21T23:09:37.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.986+0000 7f1463e00640 1 -- 172.21.15.28:0/1085789518 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1450030c10 con 0x7f146406ee40 2024-02-21T23:09:37.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.986+0000 7f146a800640 1 -- 172.21.15.28:0/1085789518 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f146406ee40 msgr2=0x7f14640a5fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:37.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.986+0000 7f146a800640 1 --2- 172.21.15.28:0/1085789518 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f146406ee40 0x7f14640a5fd0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f1450009870 tx=0x7f145002b980 comp rx=0 tx=0).stop 2024-02-21T23:09:37.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.987+0000 7f146a800640 1 -- 172.21.15.28:0/1085789518 shutdown_connections 2024-02-21T23:09:37.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.987+0000 7f146a800640 1 --2- 172.21.15.28:0/1085789518 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1464156af0 0x7f146406dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:37.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.987+0000 7f146a800640 1 --2- 172.21.15.28:0/1085789518 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f146406ee40 0x7f14640a5fd0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:37.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.987+0000 7f146a800640 1 -- 172.21.15.28:0/1085789518 >> 172.21.15.28:0/1085789518 conn(0x7f146406b630 msgr2=0x7f14640c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:37.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.987+0000 7f146a800640 1 -- 172.21.15.28:0/1085789518 shutdown_connections 2024-02-21T23:09:37.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.987+0000 7f146a800640 1 -- 172.21.15.28:0/1085789518 wait complete. 2024-02-21T23:09:37.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.988+0000 7f146a800640 1 Processor -- start 2024-02-21T23:09:37.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.989+0000 7f146a800640 1 -- start start 2024-02-21T23:09:37.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.989+0000 7f146a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f146406ee40 0x7f1464155990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:37.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.989+0000 7f146a800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1464150450 0x7f1464153540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:37.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.990+0000 7f146a800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1464156af0 0x7f14640c0290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:37.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.990+0000 7f146a800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1464156f80 con 0x7f146406ee40 2024-02-21T23:09:37.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.990+0000 7f146a800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14640c2cb0 con 0x7f1464150450 2024-02-21T23:09:37.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.990+0000 7f146a800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1464157100 con 0x7f1464156af0 2024-02-21T23:09:37.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.990+0000 7f1469400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f146406ee40 0x7f1464155990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:37.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.990+0000 7f1469400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f146406ee40 0x7f1464155990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32836/0 (socket says 172.21.15.28:32836) 2024-02-21T23:09:37.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.990+0000 7f1469400640 1 -- 172.21.15.28:0/2781297515 learned_addr learned my addr 172.21.15.28:0/2781297515 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:37.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.990+0000 7f1468a00640 1 --2- 172.21.15.28:0/2781297515 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1464150450 0x7f1464153540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:37.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.990+0000 7f1469e00640 1 --2- 172.21.15.28:0/2781297515 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1464156af0 0x7f14640c0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:37.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.990+0000 7f1469400640 1 -- 172.21.15.28:0/2781297515 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1464156af0 msgr2=0x7f14640c0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:37.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.990+0000 7f1469400640 1 --2- 172.21.15.28:0/2781297515 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1464156af0 0x7f14640c0290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:37.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.990+0000 7f1469400640 1 -- 172.21.15.28:0/2781297515 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1464150450 msgr2=0x7f1464153540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:37.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.990+0000 7f1469400640 1 --2- 172.21.15.28:0/2781297515 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1464150450 0x7f1464153540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:37.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.991+0000 7f1469400640 1 -- 172.21.15.28:0/2781297515 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14640c5930 con 0x7f146406ee40 2024-02-21T23:09:37.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.991+0000 7f1469400640 1 --2- 172.21.15.28:0/2781297515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f146406ee40 0x7f1464155990 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f145800c480 tx=0x7f145800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:37.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.991+0000 7f1461600640 1 -- 172.21.15.28:0/2781297515 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14580052d0 con 0x7f146406ee40 2024-02-21T23:09:37.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.991+0000 7f1461600640 1 -- 172.21.15.28:0/2781297515 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1458005470 con 0x7f146406ee40 2024-02-21T23:09:37.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.991+0000 7f1461600640 1 -- 172.21.15.28:0/2781297515 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1458012870 con 0x7f146406ee40 2024-02-21T23:09:37.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.991+0000 7f146a800640 1 -- 172.21.15.28:0/2781297515 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14640c5c10 con 0x7f146406ee40 2024-02-21T23:09:37.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.991+0000 7f146a800640 1 -- 172.21.15.28:0/2781297515 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14640c61d0 con 0x7f146406ee40 2024-02-21T23:09:37.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.992+0000 7f146a800640 1 -- 172.21.15.28:0/2781297515 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f146404f290 con 0x7f146406ee40 2024-02-21T23:09:37.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.994+0000 7f1461600640 1 -- 172.21.15.28:0/2781297515 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1458012a10 con 0x7f146406ee40 2024-02-21T23:09:37.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.995+0000 7f1461600640 1 --2- 172.21.15.28:0/2781297515 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f144406b500 0x7f144406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:37.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.995+0000 7f1461600640 1 -- 172.21.15.28:0/2781297515 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6331+0+0 (secure 0 0 0) 0x7f145808a100 con 0x7f146406ee40 2024-02-21T23:09:37.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.996+0000 7f1468a00640 1 --2- 172.21.15.28:0/2781297515 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f144406b500 0x7f144406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:37.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:37.997+0000 7f1468a00640 1 --2- 172.21.15.28:0/2781297515 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f144406b500 0x7f144406d9c0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f1450036000 tx=0x7f145003b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:37.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.000+0000 7f1461600640 1 -- 172.21.15.28:0/2781297515 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1458058d10 con 0x7f146406ee40 2024-02-21T23:09:38.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.201+0000 7f146a800640 1 -- 172.21.15.28:0/2781297515 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f14640c0be0 con 0x7f146406ee40 2024-02-21T23:09:38.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.203+0000 7f1461600640 1 -- 172.21.15.28:0/2781297515 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v140) v1 ==== 74+0+17247 (secure 0 0 0) 0x7f1458058610 con 0x7f146406ee40 2024-02-21T23:09:38.192 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:09:38.193 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":140,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:09:29.622962+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:09:01.777487+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"135","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2024-02-21T23:09:02.904570+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"136","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"22.7","mappings":[{"from":3,"to":0}]},{"pgid":"22.e","mappings":[{"from":3,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.81:6836/4241906475":"2024-02-22T23:09:29.622886+0000","172.21.15.81:6834/4241906475":"2024-02-22T23:09:29.622886+0000","172.21.15.28:6835/4151254007":"2024-02-22T23:09:15.528381+0000","172.21.15.28:6834/4151254007":"2024-02-22T23:09:15.528381+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:09:38.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.206+0000 7f146a800640 1 -- 172.21.15.28:0/2781297515 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f144406b500 msgr2=0x7f144406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:38.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.206+0000 7f146a800640 1 --2- 172.21.15.28:0/2781297515 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f144406b500 0x7f144406d9c0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f1450036000 tx=0x7f145003b040 comp rx=0 tx=0).stop 2024-02-21T23:09:38.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.206+0000 7f146a800640 1 -- 172.21.15.28:0/2781297515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f146406ee40 msgr2=0x7f1464155990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:38.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.206+0000 7f146a800640 1 --2- 172.21.15.28:0/2781297515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f146406ee40 0x7f1464155990 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f145800c480 tx=0x7f145800c940 comp rx=0 tx=0).stop 2024-02-21T23:09:38.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.206+0000 7f146a800640 1 -- 172.21.15.28:0/2781297515 shutdown_connections 2024-02-21T23:09:38.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.206+0000 7f146a800640 1 --2- 172.21.15.28:0/2781297515 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f144406b500 0x7f144406d9c0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:38.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.206+0000 7f146a800640 1 --2- 172.21.15.28:0/2781297515 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1464156af0 0x7f14640c0290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:38.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.206+0000 7f146a800640 1 --2- 172.21.15.28:0/2781297515 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1464150450 0x7f1464153540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:38.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.206+0000 7f146a800640 1 --2- 172.21.15.28:0/2781297515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f146406ee40 0x7f1464155990 unknown :-1 s=CLOSED pgs=1609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:38.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.206+0000 7f146a800640 1 -- 172.21.15.28:0/2781297515 >> 172.21.15.28:0/2781297515 conn(0x7f146406b630 msgr2=0x7f1464149330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:38.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.206+0000 7f146a800640 1 -- 172.21.15.28:0/2781297515 shutdown_connections 2024-02-21T23:09:38.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.206+0000 7f146a800640 1 -- 172.21.15.28:0/2781297515 wait complete. 2024-02-21T23:09:38.215 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-02-21T23:09:38.216 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:09:38.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.450+0000 7f9492a00640 1 Processor -- start 2024-02-21T23:09:38.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.451+0000 7f9492a00640 1 -- start start 2024-02-21T23:09:38.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.452+0000 7f9492a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f948c150450 0x7f948c0a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:38.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.452+0000 7f9492a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f948c156af0 0x7f948c06dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:38.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.452+0000 7f9492a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f948c0c2760 con 0x7f948c06ee40 2024-02-21T23:09:38.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.452+0000 7f9492a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f948c0c22c0 con 0x7f948c156af0 2024-02-21T23:09:38.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.452+0000 7f9492a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f948c156660 con 0x7f948c150450 2024-02-21T23:09:38.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.453+0000 7f9492000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f948c156af0 0x7f948c06dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:38.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.453+0000 7f9491600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f948c150450 0x7f948c0a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:38.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.453+0000 7f9492000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f948c156af0 0x7f948c06dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.28:43384/0 (socket says 172.21.15.28:43384) 2024-02-21T23:09:38.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.453+0000 7f9492000640 1 -- 172.21.15.28:0/458965318 learned_addr learned my addr 172.21.15.28:0/458965318 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:38.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.454+0000 7f9492000640 1 -- 172.21.15.28:0/458965318 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f948c150450 msgr2=0x7f948c0a5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:38.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.454+0000 7f9492000640 1 --2- 172.21.15.28:0/458965318 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f948c150450 0x7f948c0a5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:38.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.454+0000 7f9492000640 1 -- 172.21.15.28:0/458965318 >> v1:172.21.15.81:6789/0 conn(0x7f948c06ee40 legacy=0x7f948c1536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:38.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.454+0000 7f9492000640 1 -- 172.21.15.28:0/458965318 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f948c156310 con 0x7f948c156af0 2024-02-21T23:09:38.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.455+0000 7f9492000640 1 --2- 172.21.15.28:0/458965318 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f948c156af0 0x7f948c06dff0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f9484009e50 tx=0x7f948402bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c2c25aedc0c2232 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:38.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.456+0000 7f948be00640 1 -- 172.21.15.28:0/458965318 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f948402cb00 con 0x7f948c156af0 2024-02-21T23:09:38.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.456+0000 7f948be00640 1 -- 172.21.15.28:0/458965318 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f948402cca0 con 0x7f948c156af0 2024-02-21T23:09:38.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.456+0000 7f948be00640 1 -- 172.21.15.28:0/458965318 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9484034610 con 0x7f948c156af0 2024-02-21T23:09:38.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.457+0000 7f9492a00640 1 -- 172.21.15.28:0/458965318 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f948c156af0 msgr2=0x7f948c06dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:38.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.457+0000 7f9492a00640 1 --2- 172.21.15.28:0/458965318 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f948c156af0 0x7f948c06dff0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f9484009e50 tx=0x7f948402bf60 comp rx=0 tx=0).stop 2024-02-21T23:09:38.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.457+0000 7f9492a00640 1 -- 172.21.15.28:0/458965318 shutdown_connections 2024-02-21T23:09:38.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.457+0000 7f9492a00640 1 --2- 172.21.15.28:0/458965318 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f948c156af0 0x7f948c06dff0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:38.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.457+0000 7f9492a00640 1 --2- 172.21.15.28:0/458965318 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f948c150450 0x7f948c0a5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:38.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.457+0000 7f9492a00640 1 -- 172.21.15.28:0/458965318 >> 172.21.15.28:0/458965318 conn(0x7f948c06b630 msgr2=0x7f948c0c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:38.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.457+0000 7f9492a00640 1 -- 172.21.15.28:0/458965318 shutdown_connections 2024-02-21T23:09:38.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.457+0000 7f9492a00640 1 -- 172.21.15.28:0/458965318 wait complete. 2024-02-21T23:09:38.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.458+0000 7f9492a00640 1 Processor -- start 2024-02-21T23:09:38.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.459+0000 7f9492a00640 1 -- start start 2024-02-21T23:09:38.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.459+0000 7f9492a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f948c06ee40 0x7f948c1eb520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:38.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.460+0000 7f9492a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f948c150450 0x7f948c14c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:38.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.460+0000 7f9492a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f948c156af0 0x7f948c14b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:38.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.460+0000 7f9492a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f948c156f80 con 0x7f948c06ee40 2024-02-21T23:09:38.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.460+0000 7f9492a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f948c0c2cb0 con 0x7f948c150450 2024-02-21T23:09:38.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.460+0000 7f9492a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f948c157100 con 0x7f948c156af0 2024-02-21T23:09:38.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.460+0000 7f9491600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f948c06ee40 0x7f948c1eb520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:38.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.460+0000 7f9491600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f948c06ee40 0x7f948c1eb520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32852/0 (socket says 172.21.15.28:32852) 2024-02-21T23:09:38.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.460+0000 7f9491600640 1 -- 172.21.15.28:0/2589240959 learned_addr learned my addr 172.21.15.28:0/2589240959 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:38.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.460+0000 7f9490c00640 1 --2- 172.21.15.28:0/2589240959 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f948c150450 0x7f948c14c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:38.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.460+0000 7f9492000640 1 --2- 172.21.15.28:0/2589240959 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f948c156af0 0x7f948c14b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:38.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.460+0000 7f9491600640 1 -- 172.21.15.28:0/2589240959 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f948c156af0 msgr2=0x7f948c14b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:38.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.460+0000 7f9491600640 1 --2- 172.21.15.28:0/2589240959 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f948c156af0 0x7f948c14b350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:38.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.460+0000 7f9491600640 1 -- 172.21.15.28:0/2589240959 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f948c150450 msgr2=0x7f948c14c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:38.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.460+0000 7f9491600640 1 --2- 172.21.15.28:0/2589240959 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f948c150450 0x7f948c14c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:38.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.460+0000 7f9491600640 1 -- 172.21.15.28:0/2589240959 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f948c1533a0 con 0x7f948c06ee40 2024-02-21T23:09:38.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.460+0000 7f9490c00640 1 --2- 172.21.15.28:0/2589240959 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f948c150450 0x7f948c14c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:09:38.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.461+0000 7f9491600640 1 --2- 172.21.15.28:0/2589240959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f948c06ee40 0x7f948c1eb520 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f948000e9b0 tx=0x7f948000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:38.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.461+0000 7f9489600640 1 -- 172.21.15.28:0/2589240959 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9480017070 con 0x7f948c06ee40 2024-02-21T23:09:38.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.461+0000 7f9492a00640 1 -- 172.21.15.28:0/2589240959 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f948c153680 con 0x7f948c06ee40 2024-02-21T23:09:38.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.461+0000 7f9489600640 1 -- 172.21.15.28:0/2589240959 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f948000c9a0 con 0x7f948c06ee40 2024-02-21T23:09:38.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.461+0000 7f9489600640 1 -- 172.21.15.28:0/2589240959 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f948000cd90 con 0x7f948c06ee40 2024-02-21T23:09:38.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.461+0000 7f9492a00640 1 -- 172.21.15.28:0/2589240959 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f948c14cb00 con 0x7f948c06ee40 2024-02-21T23:09:38.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.462+0000 7f9492a00640 1 -- 172.21.15.28:0/2589240959 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f948c04f290 con 0x7f948c06ee40 2024-02-21T23:09:38.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.464+0000 7f9489600640 1 -- 172.21.15.28:0/2589240959 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f948001a070 con 0x7f948c06ee40 2024-02-21T23:09:38.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.464+0000 7f9489600640 1 --2- 172.21.15.28:0/2589240959 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f946c06b4d0 0x7f946c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:38.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.465+0000 7f9489600640 1 -- 172.21.15.28:0/2589240959 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6331+0+0 (secure 0 0 0) 0x7f948008a830 con 0x7f948c06ee40 2024-02-21T23:09:38.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.465+0000 7f9490c00640 1 --2- 172.21.15.28:0/2589240959 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f946c06b4d0 0x7f946c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:38.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.466+0000 7f9490c00640 1 --2- 172.21.15.28:0/2589240959 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f946c06b4d0 0x7f946c06d990 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f948c14f4e0 tx=0x7f9478009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:38.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.469+0000 7f9489600640 1 -- 172.21.15.28:0/2589240959 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f9480059450 con 0x7f948c06ee40 2024-02-21T23:09:38.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.706+0000 7f9492a00640 1 -- 172.21.15.28:0/2589240959 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f948c0c1f10 con 0x7f948c06ee40 2024-02-21T23:09:38.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.707+0000 7f9489600640 1 -- 172.21.15.28:0/2589240959 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 149 v149) v1 ==== 95+0+4948 (secure 0 0 0) 0x7f9480058d50 con 0x7f948c06ee40 2024-02-21T23:09:38.696 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:09:38.696 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":149,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6934,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/878375431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":878375431},{"type":"v1","addr":"172.21.15.81:6836","nonce":878375431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":147},{"gid":8370,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/3934548735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3934548735},{"type":"v1","addr":"172.21.15.28:6833","nonce":3934548735}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":133},{"gid":8556,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2100132822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2100132822},{"type":"v1","addr":"172.21.15.28:6835","nonce":2100132822}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":142}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:09:03.869294+0000","modified":"2024-02-21T23:09:31.652165+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":140,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6143},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6143":{"gid":6143,"name":"d","rank":0,"incarnation":146,"state":"up:active","state_seq":13,"addr":"172.21.15.81:6837/2095908433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":2095908433},{"type":"v1","addr":"172.21.15.81:6837","nonce":2095908433}]},"join_fscid":10,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-02-21T23:09:38.697 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 149 2024-02-21T23:09:38.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.709+0000 7f9492a00640 1 -- 172.21.15.28:0/2589240959 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f946c06b4d0 msgr2=0x7f946c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:38.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.709+0000 7f9492a00640 1 --2- 172.21.15.28:0/2589240959 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f946c06b4d0 0x7f946c06d990 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f948c14f4e0 tx=0x7f9478009040 comp rx=0 tx=0).stop 2024-02-21T23:09:38.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.710+0000 7f9492a00640 1 -- 172.21.15.28:0/2589240959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f948c06ee40 msgr2=0x7f948c1eb520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:38.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.710+0000 7f9492a00640 1 --2- 172.21.15.28:0/2589240959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f948c06ee40 0x7f948c1eb520 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f948000e9b0 tx=0x7f948000ee70 comp rx=0 tx=0).stop 2024-02-21T23:09:38.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.710+0000 7f9492a00640 1 -- 172.21.15.28:0/2589240959 shutdown_connections 2024-02-21T23:09:38.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.710+0000 7f9492a00640 1 --2- 172.21.15.28:0/2589240959 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f946c06b4d0 0x7f946c06d990 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:38.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.710+0000 7f9492a00640 1 --2- 172.21.15.28:0/2589240959 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f948c156af0 0x7f948c14b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:38.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.710+0000 7f9492a00640 1 --2- 172.21.15.28:0/2589240959 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f948c150450 0x7f948c14c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:38.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.710+0000 7f9492a00640 1 --2- 172.21.15.28:0/2589240959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f948c06ee40 0x7f948c1eb520 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:38.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.710+0000 7f9492a00640 1 -- 172.21.15.28:0/2589240959 >> 172.21.15.28:0/2589240959 conn(0x7f948c06b630 msgr2=0x7f948c14e8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:38.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.710+0000 7f9492a00640 1 -- 172.21.15.28:0/2589240959 shutdown_connections 2024-02-21T23:09:38.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.710+0000 7f9492a00640 1 -- 172.21.15.28:0/2589240959 wait complete. 2024-02-21T23:09:38.719 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:09:38.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.945+0000 7fc6b8c00640 1 Processor -- start 2024-02-21T23:09:38.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.945+0000 7fc6b8c00640 1 -- start start 2024-02-21T23:09:38.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.946+0000 7fc6b8c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc6b40c11f0 0x7fc6b4153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:38.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.946+0000 7fc6b8c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fc6b40c15d0 0x7fc6b406dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:38.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.947+0000 7fc6b8c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc6b4156350 0x7fc6b41567a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:38.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.947+0000 7fc6b8c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6b40a5ec0 con 0x7fc6b4156350 2024-02-21T23:09:38.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.947+0000 7fc6b8c00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6b40c1a30 con 0x7fc6b40c15d0 2024-02-21T23:09:38.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.947+0000 7fc6b8c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b4152e70 con 0x7fc6b40c11f0 2024-02-21T23:09:38.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.947+0000 7fc6b2800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc6b4156350 0x7fc6b41567a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:38.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.947+0000 7fc6b2800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc6b4156350 0x7fc6b41567a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32866/0 (socket says 172.21.15.28:32866) 2024-02-21T23:09:38.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.947+0000 7fc6b2800640 1 -- 172.21.15.28:0/2986438787 learned_addr learned my addr 172.21.15.28:0/2986438787 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:38.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.947+0000 7fc6b1400640 1 --2- 172.21.15.28:0/2986438787 >> v2:172.21.15.81:3300/0 conn(0x7fc6b40c15d0 0x7fc6b406dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:38.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.947+0000 7fc6b1e00640 1 --2- 172.21.15.28:0/2986438787 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc6b40c11f0 0x7fc6b4153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:38.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.947+0000 7fc6b2800640 1 -- 172.21.15.28:0/2986438787 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc6b40c11f0 msgr2=0x7fc6b4153950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:38.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.947+0000 7fc6b2800640 1 --2- 172.21.15.28:0/2986438787 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc6b40c11f0 0x7fc6b4153950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:38.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.947+0000 7fc6b2800640 1 -- 172.21.15.28:0/2986438787 >> v2:172.21.15.81:3300/0 conn(0x7fc6b40c15d0 msgr2=0x7fc6b406dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:38.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.947+0000 7fc6b2800640 1 --2- 172.21.15.28:0/2986438787 >> v2:172.21.15.81:3300/0 conn(0x7fc6b40c15d0 0x7fc6b406dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:38.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.947+0000 7fc6b2800640 1 -- 172.21.15.28:0/2986438787 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6b4153150 con 0x7fc6b4156350 2024-02-21T23:09:38.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.947+0000 7fc6b1400640 1 --2- 172.21.15.28:0/2986438787 >> v2:172.21.15.81:3300/0 conn(0x7fc6b40c15d0 0x7fc6b406dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:09:38.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.948+0000 7fc6b1e00640 1 --2- 172.21.15.28:0/2986438787 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc6b40c11f0 0x7fc6b4153950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:09:38.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.949+0000 7fc6b2800640 1 --2- 172.21.15.28:0/2986438787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6b4156350 0x7fc6b41567a0 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7fc6b406fb00 tx=0x7fc6ac02bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc37501eb60264c server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:38.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.949+0000 7fc6b0a00640 1 -- 172.21.15.28:0/2986438787 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6ac02c960 con 0x7fc6b4156350 2024-02-21T23:09:38.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.949+0000 7fc6b0a00640 1 -- 172.21.15.28:0/2986438787 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6ac02cb00 con 0x7fc6b4156350 2024-02-21T23:09:38.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.949+0000 7fc6b0a00640 1 -- 172.21.15.28:0/2986438787 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6ac0343f0 con 0x7fc6b4156350 2024-02-21T23:09:38.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.950+0000 7fc6b8c00640 1 -- 172.21.15.28:0/2986438787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6b4156350 msgr2=0x7fc6b41567a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:38.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.950+0000 7fc6b8c00640 1 --2- 172.21.15.28:0/2986438787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6b4156350 0x7fc6b41567a0 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7fc6b406fb00 tx=0x7fc6ac02bdc0 comp rx=0 tx=0).stop 2024-02-21T23:09:38.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.950+0000 7fc6b8c00640 1 -- 172.21.15.28:0/2986438787 shutdown_connections 2024-02-21T23:09:38.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.950+0000 7fc6b8c00640 1 --2- 172.21.15.28:0/2986438787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6b4156350 0x7fc6b41567a0 unknown :-1 s=CLOSED pgs=1611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:38.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.950+0000 7fc6b8c00640 1 --2- 172.21.15.28:0/2986438787 >> v2:172.21.15.81:3300/0 conn(0x7fc6b40c15d0 0x7fc6b406dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:38.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.950+0000 7fc6b8c00640 1 --2- 172.21.15.28:0/2986438787 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc6b40c11f0 0x7fc6b4153950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:38.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.950+0000 7fc6b8c00640 1 -- 172.21.15.28:0/2986438787 >> 172.21.15.28:0/2986438787 conn(0x7fc6b40c67f0 msgr2=0x7fc6b40c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:38.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.950+0000 7fc6b8c00640 1 -- 172.21.15.28:0/2986438787 shutdown_connections 2024-02-21T23:09:38.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.950+0000 7fc6b8c00640 1 -- 172.21.15.28:0/2986438787 wait complete. 2024-02-21T23:09:38.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.952+0000 7fc6b8c00640 1 Processor -- start 2024-02-21T23:09:38.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.952+0000 7fc6b8c00640 1 -- start start 2024-02-21T23:09:38.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.953+0000 7fc6b8c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc6b40c11f0 0x7fc6b4072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:38.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.953+0000 7fc6b8c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6b40c15d0 0x7fc6b4155b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:38.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.953+0000 7fc6b8c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc6b4156350 0x7fc6b41437e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:38.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.953+0000 7fc6b8c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b40c4c00 con 0x7fc6b40c15d0 2024-02-21T23:09:38.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.953+0000 7fc6b8c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b414d1c0 con 0x7fc6b40c11f0 2024-02-21T23:09:38.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.953+0000 7fc6b8c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b40a6110 con 0x7fc6b4156350 2024-02-21T23:09:38.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.953+0000 7fc6b1400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6b40c15d0 0x7fc6b4155b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:38.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.953+0000 7fc6b1400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6b40c15d0 0x7fc6b4155b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32880/0 (socket says 172.21.15.28:32880) 2024-02-21T23:09:38.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.953+0000 7fc6b1400640 1 -- 172.21.15.28:0/1144177961 learned_addr learned my addr 172.21.15.28:0/1144177961 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:38.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.953+0000 7fc6b1e00640 1 --2- 172.21.15.28:0/1144177961 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc6b40c11f0 0x7fc6b4072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:38.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.953+0000 7fc6b2800640 1 --2- 172.21.15.28:0/1144177961 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc6b4156350 0x7fc6b41437e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:38.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.953+0000 7fc6b1400640 1 -- 172.21.15.28:0/1144177961 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc6b4156350 msgr2=0x7fc6b41437e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:38.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.953+0000 7fc6b1400640 1 --2- 172.21.15.28:0/1144177961 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc6b4156350 0x7fc6b41437e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:38.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.953+0000 7fc6b1400640 1 -- 172.21.15.28:0/1144177961 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc6b40c11f0 msgr2=0x7fc6b4072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:38.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.953+0000 7fc6b1400640 1 --2- 172.21.15.28:0/1144177961 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc6b40c11f0 0x7fc6b4072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:38.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.953+0000 7fc6b1400640 1 -- 172.21.15.28:0/1144177961 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6b40717f0 con 0x7fc6b40c15d0 2024-02-21T23:09:38.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.954+0000 7fc6b2800640 1 --2- 172.21.15.28:0/1144177961 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc6b4156350 0x7fc6b41437e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:09:38.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.954+0000 7fc6b1400640 1 --2- 172.21.15.28:0/1144177961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6b40c15d0 0x7fc6b4155b70 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7fc6a000c820 tx=0x7fc6a000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:38.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.954+0000 7fc6aa000640 1 -- 172.21.15.28:0/1144177961 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a0015070 con 0x7fc6b40c15d0 2024-02-21T23:09:38.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.954+0000 7fc6aa000640 1 -- 172.21.15.28:0/1144177961 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6a0007a30 con 0x7fc6b40c15d0 2024-02-21T23:09:38.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.954+0000 7fc6aa000640 1 -- 172.21.15.28:0/1144177961 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a0007de0 con 0x7fc6b40c15d0 2024-02-21T23:09:38.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.954+0000 7fc6b8c00640 1 -- 172.21.15.28:0/1144177961 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6b4150050 con 0x7fc6b40c15d0 2024-02-21T23:09:38.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.954+0000 7fc6b8c00640 1 -- 172.21.15.28:0/1144177961 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6b41560b0 con 0x7fc6b40c15d0 2024-02-21T23:09:38.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.956+0000 7fc6b8c00640 1 -- 172.21.15.28:0/1144177961 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6b40c3cb0 con 0x7fc6b40c15d0 2024-02-21T23:09:38.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.957+0000 7fc6aa000640 1 -- 172.21.15.28:0/1144177961 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc6a0018070 con 0x7fc6b40c15d0 2024-02-21T23:09:38.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.957+0000 7fc6aa000640 1 --2- 172.21.15.28:0/1144177961 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc69406b4d0 0x7fc69406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:38.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.958+0000 7fc6aa000640 1 -- 172.21.15.28:0/1144177961 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6331+0+0 (secure 0 0 0) 0x7fc6a008a2c0 con 0x7fc6b40c15d0 2024-02-21T23:09:38.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.958+0000 7fc6b1e00640 1 --2- 172.21.15.28:0/1144177961 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc69406b4d0 0x7fc69406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:38.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.959+0000 7fc6b1e00640 1 --2- 172.21.15.28:0/1144177961 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc69406b4d0 0x7fc69406d990 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fc6a400a980 tx=0x7fc6a4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:38.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:38.962+0000 7fc6aa000640 1 -- 172.21.15.28:0/1144177961 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc6b40c3cb0 con 0x7fc6b40c15d0 2024-02-21T23:09:39.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.200+0000 7fc6b8c00640 1 -- 172.21.15.28:0/1144177961 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc6b4067f30 con 0x7fc6b40c15d0 2024-02-21T23:09:39.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.201+0000 7fc6aa000640 1 -- 172.21.15.28:0/1144177961 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 149 v149) v1 ==== 95+0+4948 (secure 0 0 0) 0x7fc6a0058f60 con 0x7fc6b40c15d0 2024-02-21T23:09:39.190 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:09:39.190 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":149,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6934,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/878375431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":878375431},{"type":"v1","addr":"172.21.15.81:6836","nonce":878375431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":147},{"gid":8370,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/3934548735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3934548735},{"type":"v1","addr":"172.21.15.28:6833","nonce":3934548735}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":133},{"gid":8556,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2100132822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2100132822},{"type":"v1","addr":"172.21.15.28:6835","nonce":2100132822}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":142}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:09:03.869294+0000","modified":"2024-02-21T23:09:31.652165+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":140,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6143},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6143":{"gid":6143,"name":"d","rank":0,"incarnation":146,"state":"up:active","state_seq":13,"addr":"172.21.15.81:6837/2095908433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":2095908433},{"type":"v1","addr":"172.21.15.81:6837","nonce":2095908433}]},"join_fscid":10,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-02-21T23:09:39.191 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 149 2024-02-21T23:09:39.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.204+0000 7fc6b8c00640 1 -- 172.21.15.28:0/1144177961 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc69406b4d0 msgr2=0x7fc69406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:39.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.204+0000 7fc6b8c00640 1 --2- 172.21.15.28:0/1144177961 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc69406b4d0 0x7fc69406d990 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fc6a400a980 tx=0x7fc6a4008040 comp rx=0 tx=0).stop 2024-02-21T23:09:39.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.204+0000 7fc6b8c00640 1 -- 172.21.15.28:0/1144177961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6b40c15d0 msgr2=0x7fc6b4155b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:39.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.204+0000 7fc6b8c00640 1 --2- 172.21.15.28:0/1144177961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6b40c15d0 0x7fc6b4155b70 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7fc6a000c820 tx=0x7fc6a000cce0 comp rx=0 tx=0).stop 2024-02-21T23:09:39.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.204+0000 7fc6b8c00640 1 -- 172.21.15.28:0/1144177961 shutdown_connections 2024-02-21T23:09:39.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.204+0000 7fc6b8c00640 1 --2- 172.21.15.28:0/1144177961 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc69406b4d0 0x7fc69406d990 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:39.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.204+0000 7fc6b8c00640 1 --2- 172.21.15.28:0/1144177961 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc6b4156350 0x7fc6b41437e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:39.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.204+0000 7fc6b8c00640 1 --2- 172.21.15.28:0/1144177961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6b40c15d0 0x7fc6b4155b70 unknown :-1 s=CLOSED pgs=1612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:39.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.204+0000 7fc6b8c00640 1 --2- 172.21.15.28:0/1144177961 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc6b40c11f0 0x7fc6b4072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:39.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.204+0000 7fc6b8c00640 1 -- 172.21.15.28:0/1144177961 >> 172.21.15.28:0/1144177961 conn(0x7fc6b40c67f0 msgr2=0x7fc6b4148dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:39.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.204+0000 7fc6b8c00640 1 -- 172.21.15.28:0/1144177961 shutdown_connections 2024-02-21T23:09:39.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.204+0000 7fc6b8c00640 1 -- 172.21.15.28:0/1144177961 wait complete. 2024-02-21T23:09:39.211 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:09:39.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.435+0000 7fa44b800640 1 Processor -- start 2024-02-21T23:09:39.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.436+0000 7fa44b800640 1 -- start start 2024-02-21T23:09:39.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.437+0000 7fa44b800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa4440c3280 0x7fa4440c3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:39.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.437+0000 7fa44b800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa444072430 0x7fa4441526e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:39.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.437+0000 7fa44b800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa444152c20 con 0x7fa444155860 2024-02-21T23:09:39.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.437+0000 7fa44b800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4440c6560 con 0x7fa4440c3280 2024-02-21T23:09:39.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.437+0000 7fa44b800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4440c5cc0 con 0x7fa444072430 2024-02-21T23:09:39.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.437+0000 7fa449800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa444155860 0x7fa4440c46f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60114/0 (socket says 172.21.15.28:60114) 2024-02-21T23:09:39.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.437+0000 7fa449800640 1 -- 172.21.15.28:0/1139739259 learned_addr learned my addr 172.21.15.28:0/1139739259 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:09:39.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.437+0000 7fa448e00640 1 --2- 172.21.15.28:0/1139739259 >> v2:172.21.15.28:3300/0 conn(0x7fa4440c3280 0x7fa4440c3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:39.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.438+0000 7fa443e00640 1 --2- 172.21.15.28:0/1139739259 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa444072430 0x7fa4441526e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:39.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.438+0000 7fa443400640 1 -- 172.21.15.28:0/1139739259 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 724860572 0 0) 0x7fa43c001110 con 0x7fa444155860 2024-02-21T23:09:39.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.438+0000 7fa443400640 1 -- 172.21.15.28:0/1139739259 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa424003520 con 0x7fa444155860 2024-02-21T23:09:39.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.438+0000 7fa448e00640 1 -- 172.21.15.28:0/1139739259 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa444072430 msgr2=0x7fa4441526e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:39.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.438+0000 7fa448e00640 1 --2- 172.21.15.28:0/1139739259 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa444072430 0x7fa4441526e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:39.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.438+0000 7fa448e00640 1 -- 172.21.15.28:0/1139739259 >> v1:172.21.15.28:6789/0 conn(0x7fa444155860 legacy=0x7fa4440c46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:39.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.438+0000 7fa448e00640 1 -- 172.21.15.28:0/1139739259 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4440c5fa0 con 0x7fa4440c3280 2024-02-21T23:09:39.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.439+0000 7fa448e00640 1 --2- 172.21.15.28:0/1139739259 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4440c3280 0x7fa4440c3660 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7fa4340097a0 tx=0x7fa43402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41ec207662df0e53 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:39.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.439+0000 7fa443400640 1 -- 172.21.15.28:0/1139739259 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4340315a0 con 0x7fa4440c3280 2024-02-21T23:09:39.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.439+0000 7fa443400640 1 -- 172.21.15.28:0/1139739259 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa434037070 con 0x7fa4440c3280 2024-02-21T23:09:39.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.439+0000 7fa443400640 1 -- 172.21.15.28:0/1139739259 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa434031940 con 0x7fa4440c3280 2024-02-21T23:09:39.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.440+0000 7fa44b800640 1 -- 172.21.15.28:0/1139739259 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4440c3280 msgr2=0x7fa4440c3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:39.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.440+0000 7fa44b800640 1 --2- 172.21.15.28:0/1139739259 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4440c3280 0x7fa4440c3660 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7fa4340097a0 tx=0x7fa43402b840 comp rx=0 tx=0).stop 2024-02-21T23:09:39.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.440+0000 7fa44b800640 1 -- 172.21.15.28:0/1139739259 shutdown_connections 2024-02-21T23:09:39.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.440+0000 7fa44b800640 1 --2- 172.21.15.28:0/1139739259 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa444072430 0x7fa4441526e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:39.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.440+0000 7fa44b800640 1 --2- 172.21.15.28:0/1139739259 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4440c3280 0x7fa4440c3660 unknown :-1 s=CLOSED pgs=1614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:39.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.440+0000 7fa44b800640 1 -- 172.21.15.28:0/1139739259 >> 172.21.15.28:0/1139739259 conn(0x7fa4440c67f0 msgr2=0x7fa4440c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:39.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.440+0000 7fa44b800640 1 -- 172.21.15.28:0/1139739259 shutdown_connections 2024-02-21T23:09:39.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.440+0000 7fa44b800640 1 -- 172.21.15.28:0/1139739259 wait complete. 2024-02-21T23:09:39.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.442+0000 7fa44b800640 1 Processor -- start 2024-02-21T23:09:39.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.442+0000 7fa44b800640 1 -- start start 2024-02-21T23:09:39.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.443+0000 7fa44b800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa444072430 0x7fa444160010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:39.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.443+0000 7fa44b800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa4440c3280 0x7fa444149e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:39.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.443+0000 7fa44b800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa444155860 0x7fa4440bd3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:39.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.443+0000 7fa44b800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4440c5920 con 0x7fa444072430 2024-02-21T23:09:39.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.443+0000 7fa44b800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4440a6100 con 0x7fa4440c3280 2024-02-21T23:09:39.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.443+0000 7fa44b800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4440c5aa0 con 0x7fa444155860 2024-02-21T23:09:39.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.443+0000 7fa448e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa444072430 0x7fa444160010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:39.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.443+0000 7fa448e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa444072430 0x7fa444160010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32906/0 (socket says 172.21.15.28:32906) 2024-02-21T23:09:39.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.443+0000 7fa448e00640 1 -- 172.21.15.28:0/128742614 learned_addr learned my addr 172.21.15.28:0/128742614 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:39.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.444+0000 7fa449800640 1 --2- 172.21.15.28:0/128742614 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa444155860 0x7fa4440bd3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:39.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.444+0000 7fa443e00640 1 --2- 172.21.15.28:0/128742614 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa4440c3280 0x7fa444149e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:39.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.444+0000 7fa448e00640 1 -- 172.21.15.28:0/128742614 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa444155860 msgr2=0x7fa4440bd3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:39.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.444+0000 7fa448e00640 1 --2- 172.21.15.28:0/128742614 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa444155860 0x7fa4440bd3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:39.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.444+0000 7fa448e00640 1 -- 172.21.15.28:0/128742614 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa4440c3280 msgr2=0x7fa444149e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:39.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.444+0000 7fa448e00640 1 --2- 172.21.15.28:0/128742614 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa4440c3280 0x7fa444149e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:39.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.444+0000 7fa448e00640 1 -- 172.21.15.28:0/128742614 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4441eae90 con 0x7fa444072430 2024-02-21T23:09:39.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.444+0000 7fa448e00640 1 --2- 172.21.15.28:0/128742614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa444072430 0x7fa444160010 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7fa4340097a0 tx=0x7fa4340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:39.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.444+0000 7fa440c00640 1 -- 172.21.15.28:0/128742614 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa434032630 con 0x7fa444072430 2024-02-21T23:09:39.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.444+0000 7fa440c00640 1 -- 172.21.15.28:0/128742614 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4340327d0 con 0x7fa444072430 2024-02-21T23:09:39.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.444+0000 7fa440c00640 1 -- 172.21.15.28:0/128742614 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa434032bc0 con 0x7fa444072430 2024-02-21T23:09:39.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.444+0000 7fa44b800640 1 -- 172.21.15.28:0/128742614 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4441eb170 con 0x7fa444072430 2024-02-21T23:09:39.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.444+0000 7fa44b800640 1 -- 172.21.15.28:0/128742614 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4441eb730 con 0x7fa444072430 2024-02-21T23:09:39.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.445+0000 7fa44b800640 1 -- 172.21.15.28:0/128742614 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa444067f30 con 0x7fa444072430 2024-02-21T23:09:39.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.446+0000 7fa440c00640 1 -- 172.21.15.28:0/128742614 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa43404d3f0 con 0x7fa444072430 2024-02-21T23:09:39.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.447+0000 7fa440c00640 1 --2- 172.21.15.28:0/128742614 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa42006b4d0 0x7fa42006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:39.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.447+0000 7fa440c00640 1 -- 172.21.15.28:0/128742614 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6331+0+0 (secure 0 0 0) 0x7fa43402f030 con 0x7fa444072430 2024-02-21T23:09:39.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.448+0000 7fa443e00640 1 --2- 172.21.15.28:0/128742614 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa42006b4d0 0x7fa42006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:39.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.449+0000 7fa443e00640 1 --2- 172.21.15.28:0/128742614 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa42006b4d0 0x7fa42006d990 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fa4441e8980 tx=0x7fa42c00b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:39.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.452+0000 7fa440c00640 1 -- 172.21.15.28:0/128742614 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fa434040070 con 0x7fa444072430 2024-02-21T23:09:39.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.650+0000 7fa44b800640 1 -- 172.21.15.28:0/128742614 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa4440c5700 con 0x7fa444072430 2024-02-21T23:09:39.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.653+0000 7fa440c00640 1 -- 172.21.15.28:0/128742614 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v140) v1 ==== 74+0+17247 (secure 0 0 0) 0x7fa43407ad20 con 0x7fa444072430 2024-02-21T23:09:39.642 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:09:39.642 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":140,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:09:29.622962+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:09:01.777487+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"135","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2024-02-21T23:09:02.904570+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"136","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"22.7","mappings":[{"from":3,"to":0}]},{"pgid":"22.e","mappings":[{"from":3,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.81:6836/4241906475":"2024-02-22T23:09:29.622886+0000","172.21.15.81:6834/4241906475":"2024-02-22T23:09:29.622886+0000","172.21.15.28:6835/4151254007":"2024-02-22T23:09:15.528381+0000","172.21.15.28:6834/4151254007":"2024-02-22T23:09:15.528381+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:09:39.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.655+0000 7fa44b800640 1 -- 172.21.15.28:0/128742614 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa42006b4d0 msgr2=0x7fa42006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:39.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.655+0000 7fa44b800640 1 --2- 172.21.15.28:0/128742614 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa42006b4d0 0x7fa42006d990 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fa4441e8980 tx=0x7fa42c00b040 comp rx=0 tx=0).stop 2024-02-21T23:09:39.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.655+0000 7fa44b800640 1 -- 172.21.15.28:0/128742614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa444072430 msgr2=0x7fa444160010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:39.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.655+0000 7fa44b800640 1 --2- 172.21.15.28:0/128742614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa444072430 0x7fa444160010 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7fa4340097a0 tx=0x7fa4340046c0 comp rx=0 tx=0).stop 2024-02-21T23:09:39.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.656+0000 7fa44b800640 1 -- 172.21.15.28:0/128742614 shutdown_connections 2024-02-21T23:09:39.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.656+0000 7fa44b800640 1 --2- 172.21.15.28:0/128742614 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa42006b4d0 0x7fa42006d990 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:39.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.656+0000 7fa44b800640 1 --2- 172.21.15.28:0/128742614 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa444155860 0x7fa4440bd3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:39.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.656+0000 7fa44b800640 1 --2- 172.21.15.28:0/128742614 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa4440c3280 0x7fa444149e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:39.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.656+0000 7fa44b800640 1 --2- 172.21.15.28:0/128742614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa444072430 0x7fa444160010 unknown :-1 s=CLOSED pgs=1615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:39.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.656+0000 7fa44b800640 1 -- 172.21.15.28:0/128742614 >> 172.21.15.28:0/128742614 conn(0x7fa4440c67f0 msgr2=0x7fa444151ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:39.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.656+0000 7fa44b800640 1 -- 172.21.15.28:0/128742614 shutdown_connections 2024-02-21T23:09:39.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.656+0000 7fa44b800640 1 -- 172.21.15.28:0/128742614 wait complete. 2024-02-21T23:09:39.665 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-02-21T23:09:39.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.891+0000 7efc6be00640 1 Processor -- start 2024-02-21T23:09:39.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.892+0000 7efc6be00640 1 -- start start 2024-02-21T23:09:39.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.893+0000 7efc6be00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efc6c148700 0x7efc6c06d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:39.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.893+0000 7efc6be00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7efc6c0c13e0 0x7efc6c0c1800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:39.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.893+0000 7efc6be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7efc6c0c1d40 0x7efc6c155260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:39.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.893+0000 7efc6be00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc6c072e90 con 0x7efc6c0c1d40 2024-02-21T23:09:39.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.893+0000 7efc6be00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc6c0c0ee0 con 0x7efc6c0c13e0 2024-02-21T23:09:39.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.893+0000 7efc6be00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc6c073100 con 0x7efc6c148700 2024-02-21T23:09:39.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.894+0000 7efc6b400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7efc6c0c1d40 0x7efc6c155260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:39.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.894+0000 7efc6b400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7efc6c0c1d40 0x7efc6c155260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32916/0 (socket says 172.21.15.28:32916) 2024-02-21T23:09:39.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.894+0000 7efc6b400640 1 -- 172.21.15.28:0/3020234745 learned_addr learned my addr 172.21.15.28:0/3020234745 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:39.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.894+0000 7efc6aa00640 1 --2- 172.21.15.28:0/3020234745 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efc6c148700 0x7efc6c06d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:39.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.894+0000 7efc62000640 1 --2- 172.21.15.28:0/3020234745 >> v2:172.21.15.81:3300/0 conn(0x7efc6c0c13e0 0x7efc6c0c1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:39.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.894+0000 7efc6b400640 1 -- 172.21.15.28:0/3020234745 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efc6c148700 msgr2=0x7efc6c06d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:39.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.894+0000 7efc6b400640 1 --2- 172.21.15.28:0/3020234745 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efc6c148700 0x7efc6c06d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:39.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.894+0000 7efc6b400640 1 -- 172.21.15.28:0/3020234745 >> v2:172.21.15.81:3300/0 conn(0x7efc6c0c13e0 msgr2=0x7efc6c0c1800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:39.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.894+0000 7efc6b400640 1 --2- 172.21.15.28:0/3020234745 >> v2:172.21.15.81:3300/0 conn(0x7efc6c0c13e0 0x7efc6c0c1800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:39.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.894+0000 7efc6b400640 1 -- 172.21.15.28:0/3020234745 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc6c0733e0 con 0x7efc6c0c1d40 2024-02-21T23:09:39.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.894+0000 7efc62000640 1 --2- 172.21.15.28:0/3020234745 >> v2:172.21.15.81:3300/0 conn(0x7efc6c0c13e0 0x7efc6c0c1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:09:39.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.896+0000 7efc6b400640 1 --2- 172.21.15.28:0/3020234745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc6c0c1d40 0x7efc6c155260 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7efc6c06f7f0 tx=0x7efc6402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf2b8e432c66cd5c server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:39.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.896+0000 7efc6a000640 1 -- 172.21.15.28:0/3020234745 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc6402ca70 con 0x7efc6c0c1d40 2024-02-21T23:09:39.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.896+0000 7efc6a000640 1 -- 172.21.15.28:0/3020234745 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc6402cc10 con 0x7efc6c0c1d40 2024-02-21T23:09:39.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.896+0000 7efc6a000640 1 -- 172.21.15.28:0/3020234745 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc640343f0 con 0x7efc6c0c1d40 2024-02-21T23:09:39.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.897+0000 7efc6be00640 1 -- 172.21.15.28:0/3020234745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc6c0c1d40 msgr2=0x7efc6c155260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:39.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.897+0000 7efc6be00640 1 --2- 172.21.15.28:0/3020234745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc6c0c1d40 0x7efc6c155260 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7efc6c06f7f0 tx=0x7efc6402bed0 comp rx=0 tx=0).stop 2024-02-21T23:09:39.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.897+0000 7efc6be00640 1 -- 172.21.15.28:0/3020234745 shutdown_connections 2024-02-21T23:09:39.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.897+0000 7efc6be00640 1 --2- 172.21.15.28:0/3020234745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc6c0c1d40 0x7efc6c155260 unknown :-1 s=CLOSED pgs=1616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:39.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.897+0000 7efc6be00640 1 --2- 172.21.15.28:0/3020234745 >> v2:172.21.15.81:3300/0 conn(0x7efc6c0c13e0 0x7efc6c0c1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:39.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.897+0000 7efc6be00640 1 --2- 172.21.15.28:0/3020234745 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efc6c148700 0x7efc6c06d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:39.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.897+0000 7efc6be00640 1 -- 172.21.15.28:0/3020234745 >> 172.21.15.28:0/3020234745 conn(0x7efc6c0a8320 msgr2=0x7efc6c0a8730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:39.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.897+0000 7efc6be00640 1 -- 172.21.15.28:0/3020234745 shutdown_connections 2024-02-21T23:09:39.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.897+0000 7efc6be00640 1 -- 172.21.15.28:0/3020234745 wait complete. 2024-02-21T23:09:39.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.899+0000 7efc6be00640 1 Processor -- start 2024-02-21T23:09:39.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.899+0000 7efc6be00640 1 -- start start 2024-02-21T23:09:39.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.900+0000 7efc6be00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efc6c0c13e0 0x7efc6c151f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:39.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.900+0000 7efc6be00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efc6c0c1d40 0x7efc6c1edaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:39.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.901+0000 7efc6be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc6c148700 0x7efc6c1eb810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:39.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.901+0000 7efc6be00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc6c06d9f0 con 0x7efc6c148700 2024-02-21T23:09:39.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.901+0000 7efc6be00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc6c0a9080 con 0x7efc6c0c13e0 2024-02-21T23:09:39.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.901+0000 7efc6be00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc6c1557a0 con 0x7efc6c0c1d40 2024-02-21T23:09:39.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.901+0000 7efc6b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc6c148700 0x7efc6c1eb810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:39.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.901+0000 7efc6b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc6c148700 0x7efc6c1eb810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32926/0 (socket says 172.21.15.28:32926) 2024-02-21T23:09:39.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.901+0000 7efc6b400640 1 -- 172.21.15.28:0/2767463204 learned_addr learned my addr 172.21.15.28:0/2767463204 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:39.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.901+0000 7efc6aa00640 1 --2- 172.21.15.28:0/2767463204 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efc6c0c13e0 0x7efc6c151f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:39.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.901+0000 7efc62000640 1 --2- 172.21.15.28:0/2767463204 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efc6c0c1d40 0x7efc6c1edaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:39.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.901+0000 7efc6b400640 1 -- 172.21.15.28:0/2767463204 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efc6c0c1d40 msgr2=0x7efc6c1edaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:39.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.901+0000 7efc6b400640 1 --2- 172.21.15.28:0/2767463204 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efc6c0c1d40 0x7efc6c1edaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:39.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.901+0000 7efc6b400640 1 -- 172.21.15.28:0/2767463204 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efc6c0c13e0 msgr2=0x7efc6c151f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:39.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.901+0000 7efc6b400640 1 --2- 172.21.15.28:0/2767463204 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efc6c0c13e0 0x7efc6c151f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:39.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.901+0000 7efc6b400640 1 -- 172.21.15.28:0/2767463204 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc6c14a000 con 0x7efc6c148700 2024-02-21T23:09:39.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.902+0000 7efc6b400640 1 --2- 172.21.15.28:0/2767463204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc6c148700 0x7efc6c1eb810 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7efc64009f40 tx=0x7efc64004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:39.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.902+0000 7efc63400640 1 -- 172.21.15.28:0/2767463204 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc64043020 con 0x7efc6c148700 2024-02-21T23:09:39.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.902+0000 7efc63400640 1 -- 172.21.15.28:0/2767463204 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc6403e450 con 0x7efc6c148700 2024-02-21T23:09:39.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.902+0000 7efc63400640 1 -- 172.21.15.28:0/2767463204 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc64009510 con 0x7efc6c148700 2024-02-21T23:09:39.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.902+0000 7efc6be00640 1 -- 172.21.15.28:0/2767463204 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc6c1ee3c0 con 0x7efc6c148700 2024-02-21T23:09:39.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.902+0000 7efc6be00640 1 -- 172.21.15.28:0/2767463204 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc6c151790 con 0x7efc6c148700 2024-02-21T23:09:39.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.904+0000 7efc6be00640 1 -- 172.21.15.28:0/2767463204 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc6c0c73f0 con 0x7efc6c148700 2024-02-21T23:09:39.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.904+0000 7efc63400640 1 -- 172.21.15.28:0/2767463204 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efc64005d10 con 0x7efc6c148700 2024-02-21T23:09:39.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.905+0000 7efc63400640 1 --2- 172.21.15.28:0/2767463204 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7efc4806b4d0 0x7efc4806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:39.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.906+0000 7efc63400640 1 -- 172.21.15.28:0/2767463204 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6331+0+0 (secure 0 0 0) 0x7efc640acfc0 con 0x7efc6c148700 2024-02-21T23:09:39.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.906+0000 7efc6aa00640 1 --2- 172.21.15.28:0/2767463204 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7efc4806b4d0 0x7efc4806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:39.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.907+0000 7efc6aa00640 1 --2- 172.21.15.28:0/2767463204 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7efc4806b4d0 0x7efc4806d990 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7efc6c152c70 tx=0x7efc5c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:39.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:39.910+0000 7efc63400640 1 -- 172.21.15.28:0/2767463204 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7efc6c0c73f0 con 0x7efc6c148700 2024-02-21T23:09:40.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.148+0000 7efc6be00640 1 -- 172.21.15.28:0/2767463204 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7efc6c14ec30 con 0x7efc6c148700 2024-02-21T23:09:40.138 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-02-21T23:09:40.149+0000 7f5ddc400640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-21T23:09:40.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.153+0000 7efc63400640 1 -- 172.21.15.28:0/2767463204 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v150) v1 ==== 161+0+0 (secure 0 0 0) 0x7efc6407bc80 con 0x7efc6c148700 2024-02-21T23:09:40.142 INFO:teuthology.orchestra.run.smithi028.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-02-21T23:09:40.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.156+0000 7efc6be00640 1 -- 172.21.15.28:0/2767463204 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7efc4806b4d0 msgr2=0x7efc4806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:40.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.156+0000 7efc6be00640 1 --2- 172.21.15.28:0/2767463204 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7efc4806b4d0 0x7efc4806d990 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7efc6c152c70 tx=0x7efc5c008040 comp rx=0 tx=0).stop 2024-02-21T23:09:40.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.156+0000 7efc6be00640 1 -- 172.21.15.28:0/2767463204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc6c148700 msgr2=0x7efc6c1eb810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:40.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.156+0000 7efc6be00640 1 --2- 172.21.15.28:0/2767463204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc6c148700 0x7efc6c1eb810 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7efc64009f40 tx=0x7efc64004a40 comp rx=0 tx=0).stop 2024-02-21T23:09:40.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.156+0000 7efc6be00640 1 -- 172.21.15.28:0/2767463204 shutdown_connections 2024-02-21T23:09:40.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.156+0000 7efc6be00640 1 --2- 172.21.15.28:0/2767463204 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7efc4806b4d0 0x7efc4806d990 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:40.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.156+0000 7efc6be00640 1 --2- 172.21.15.28:0/2767463204 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efc6c0c1d40 0x7efc6c1edaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:40.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.156+0000 7efc6be00640 1 --2- 172.21.15.28:0/2767463204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc6c148700 0x7efc6c1eb810 unknown :-1 s=CLOSED pgs=1617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:40.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.156+0000 7efc6be00640 1 --2- 172.21.15.28:0/2767463204 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efc6c0c13e0 0x7efc6c151f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:40.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.156+0000 7efc6be00640 1 -- 172.21.15.28:0/2767463204 >> 172.21.15.28:0/2767463204 conn(0x7efc6c0a8320 msgr2=0x7efc6c0a6600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:40.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.156+0000 7efc6be00640 1 -- 172.21.15.28:0/2767463204 shutdown_connections 2024-02-21T23:09:40.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.156+0000 7efc6be00640 1 -- 172.21.15.28:0/2767463204 wait complete. 2024-02-21T23:09:40.162 INFO:tasks.ceph.mds.d.smithi081.stderr: -95> 2024-02-21T23:09:37.324+0000 7f2284000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-21T23:09:40.168 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-02-21T23:09:40.225 INFO:tasks.ceph.mds.d.smithi081.stdout:starting mds.d at 2024-02-21T23:09:40.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.405+0000 7f24cea00640 1 Processor -- start 2024-02-21T23:09:40.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.406+0000 7f24cea00640 1 -- start start 2024-02-21T23:09:40.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.407+0000 7f24cea00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f24c813e0c0 0x7f24c80c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:40.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.407+0000 7f24cea00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f24c806dbc0 0x7f24c806c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:40.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.407+0000 7f24cea00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24c8158550 con 0x7f24c80c76f0 2024-02-21T23:09:40.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.407+0000 7f24cea00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24c80c89a0 con 0x7f24c813e0c0 2024-02-21T23:09:40.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.408+0000 7f24cea00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24c81587d0 con 0x7f24c806dbc0 2024-02-21T23:09:40.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.408+0000 7f24cd600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f24c813e0c0 0x7f24c80c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:40.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.408+0000 7f24cd600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f24c813e0c0 0x7f24c80c71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32928/0 (socket says 172.21.15.28:32928) 2024-02-21T23:09:40.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.408+0000 7f24cd600640 1 -- 172.21.15.28:0/478975012 learned_addr learned my addr 172.21.15.28:0/478975012 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:40.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.408+0000 7f24ce000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f24c80c76f0 0x7f24c806c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60126/0 (socket says 172.21.15.28:60126) 2024-02-21T23:09:40.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.408+0000 7f24ccc00640 1 --2- 172.21.15.28:0/478975012 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f24c806dbc0 0x7f24c806c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:40.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.409+0000 7f24cd600640 1 -- 172.21.15.28:0/478975012 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f24c806dbc0 msgr2=0x7f24c806c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:40.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.409+0000 7f24cd600640 1 --2- 172.21.15.28:0/478975012 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f24c806dbc0 0x7f24c806c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:40.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.409+0000 7f24cd600640 1 -- 172.21.15.28:0/478975012 >> v1:172.21.15.28:6789/0 conn(0x7f24c80c76f0 legacy=0x7f24c806c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:40.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.409+0000 7f24cd600640 1 -- 172.21.15.28:0/478975012 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24c8158ab0 con 0x7f24c813e0c0 2024-02-21T23:09:40.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.409+0000 7f24ccc00640 1 --2- 172.21.15.28:0/478975012 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f24c806dbc0 0x7f24c806c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:09:40.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.410+0000 7f24cd600640 1 --2- 172.21.15.28:0/478975012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24c813e0c0 0x7f24c80c71b0 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7f24bc009e50 tx=0x7f24bc02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1778f634aef4a40d server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:40.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.410+0000 7f24c7e00640 1 -- 172.21.15.28:0/478975012 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24bc02ca80 con 0x7f24c813e0c0 2024-02-21T23:09:40.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.410+0000 7f24c7e00640 1 -- 172.21.15.28:0/478975012 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24bc02cc20 con 0x7f24c813e0c0 2024-02-21T23:09:40.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.410+0000 7f24c7e00640 1 -- 172.21.15.28:0/478975012 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24bc0343f0 con 0x7f24c813e0c0 2024-02-21T23:09:40.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.411+0000 7f24cea00640 1 -- 172.21.15.28:0/478975012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24c813e0c0 msgr2=0x7f24c80c71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:40.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.411+0000 7f24cea00640 1 --2- 172.21.15.28:0/478975012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24c813e0c0 0x7f24c80c71b0 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7f24bc009e50 tx=0x7f24bc02bee0 comp rx=0 tx=0).stop 2024-02-21T23:09:40.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.411+0000 7f24cea00640 1 -- 172.21.15.28:0/478975012 shutdown_connections 2024-02-21T23:09:40.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.411+0000 7f24cea00640 1 --2- 172.21.15.28:0/478975012 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f24c806dbc0 0x7f24c806c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:40.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.411+0000 7f24cea00640 1 --2- 172.21.15.28:0/478975012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24c813e0c0 0x7f24c80c71b0 unknown :-1 s=CLOSED pgs=1620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:40.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.411+0000 7f24cea00640 1 -- 172.21.15.28:0/478975012 >> 172.21.15.28:0/478975012 conn(0x7f24c8073110 msgr2=0x7f24c80ac490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:40.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.411+0000 7f24cea00640 1 -- 172.21.15.28:0/478975012 shutdown_connections 2024-02-21T23:09:40.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.411+0000 7f24cea00640 1 -- 172.21.15.28:0/478975012 wait complete. 2024-02-21T23:09:40.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.413+0000 7f24cea00640 1 Processor -- start 2024-02-21T23:09:40.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.413+0000 7f24cea00640 1 -- start start 2024-02-21T23:09:40.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.413+0000 7f24cea00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f24c806dbc0 0x7f24c80b0070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:40.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.414+0000 7f24cea00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f24c80c76f0 0x7f24c80ae4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:40.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.414+0000 7f24cea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24c813e0c0 0x7f24c8134070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:40.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.414+0000 7f24cea00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24c806ced0 con 0x7f24c813e0c0 2024-02-21T23:09:40.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.414+0000 7f24cea00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24c8159070 con 0x7f24c806dbc0 2024-02-21T23:09:40.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.414+0000 7f24cea00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24c813e520 con 0x7f24c80c76f0 2024-02-21T23:09:40.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.414+0000 7f24ce000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24c813e0c0 0x7f24c8134070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:40.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.414+0000 7f24ce000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24c813e0c0 0x7f24c8134070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32936/0 (socket says 172.21.15.28:32936) 2024-02-21T23:09:40.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.414+0000 7f24ce000640 1 -- 172.21.15.28:0/3146454402 learned_addr learned my addr 172.21.15.28:0/3146454402 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:40.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.414+0000 7f24cd600640 1 --2- 172.21.15.28:0/3146454402 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f24c806dbc0 0x7f24c80b0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:40.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.414+0000 7f24ccc00640 1 --2- 172.21.15.28:0/3146454402 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f24c80c76f0 0x7f24c80ae4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:40.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.414+0000 7f24ce000640 1 -- 172.21.15.28:0/3146454402 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f24c80c76f0 msgr2=0x7f24c80ae4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:40.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.414+0000 7f24ce000640 1 --2- 172.21.15.28:0/3146454402 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f24c80c76f0 0x7f24c80ae4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:40.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.414+0000 7f24ce000640 1 -- 172.21.15.28:0/3146454402 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f24c806dbc0 msgr2=0x7f24c80b0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:40.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.414+0000 7f24ce000640 1 --2- 172.21.15.28:0/3146454402 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f24c806dbc0 0x7f24c80b0070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:40.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.414+0000 7f24ce000640 1 -- 172.21.15.28:0/3146454402 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24c813b400 con 0x7f24c813e0c0 2024-02-21T23:09:40.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.414+0000 7f24ccc00640 1 --2- 172.21.15.28:0/3146454402 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f24c80c76f0 0x7f24c80ae4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:09:40.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.415+0000 7f24ce000640 1 --2- 172.21.15.28:0/3146454402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24c813e0c0 0x7f24c8134070 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7f24c000c400 tx=0x7f24c000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:40.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.415+0000 7f24c5600640 1 -- 172.21.15.28:0/3146454402 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24c00074e0 con 0x7f24c813e0c0 2024-02-21T23:09:40.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.415+0000 7f24c5600640 1 -- 172.21.15.28:0/3146454402 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24c0007680 con 0x7f24c813e0c0 2024-02-21T23:09:40.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.415+0000 7f24c5600640 1 -- 172.21.15.28:0/3146454402 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24c00049e0 con 0x7f24c813e0c0 2024-02-21T23:09:40.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.415+0000 7f24cea00640 1 -- 172.21.15.28:0/3146454402 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24c80ad6b0 con 0x7f24c813e0c0 2024-02-21T23:09:40.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.415+0000 7f24cea00640 1 -- 172.21.15.28:0/3146454402 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24c80adc70 con 0x7f24c813e0c0 2024-02-21T23:09:40.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.417+0000 7f24cea00640 1 -- 172.21.15.28:0/3146454402 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24c804f290 con 0x7f24c813e0c0 2024-02-21T23:09:40.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.418+0000 7f24c5600640 1 -- 172.21.15.28:0/3146454402 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f24c001a020 con 0x7f24c813e0c0 2024-02-21T23:09:40.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.418+0000 7f24c5600640 1 --2- 172.21.15.28:0/3146454402 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f24a806b4d0 0x7f24a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:40.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.419+0000 7f24c5600640 1 -- 172.21.15.28:0/3146454402 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f24c0089c50 con 0x7f24c813e0c0 2024-02-21T23:09:40.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.419+0000 7f24cd600640 1 --2- 172.21.15.28:0/3146454402 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f24a806b4d0 0x7f24a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:40.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.420+0000 7f24cd600640 1 --2- 172.21.15.28:0/3146454402 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f24a806b4d0 0x7f24a806d990 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f24bc035000 tx=0x7f24bc030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:40.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.423+0000 7f24c5600640 1 -- 172.21.15.28:0/3146454402 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f24c008f020 con 0x7f24c813e0c0 2024-02-21T23:09:40.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:40.666+0000 7f24cea00640 1 -- 172.21.15.28:0/3146454402 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f24c80ade40 con 0x7f24c813e0c0 2024-02-21T23:09:41.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.156+0000 7f24c5600640 1 -- 172.21.15.28:0/3146454402 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v151) v1 ==== 104+0+0 (secure 0 0 0) 0x7f24c8158550 con 0x7f24c813e0c0 2024-02-21T23:09:41.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.160+0000 7f24cea00640 1 -- 172.21.15.28:0/3146454402 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f24a806b4d0 msgr2=0x7f24a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:41.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.160+0000 7f24cea00640 1 --2- 172.21.15.28:0/3146454402 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f24a806b4d0 0x7f24a806d990 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f24bc035000 tx=0x7f24bc030000 comp rx=0 tx=0).stop 2024-02-21T23:09:41.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.160+0000 7f24cea00640 1 -- 172.21.15.28:0/3146454402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24c813e0c0 msgr2=0x7f24c8134070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:41.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.160+0000 7f24cea00640 1 --2- 172.21.15.28:0/3146454402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24c813e0c0 0x7f24c8134070 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7f24c000c400 tx=0x7f24c000c8c0 comp rx=0 tx=0).stop 2024-02-21T23:09:41.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.160+0000 7f24cea00640 1 -- 172.21.15.28:0/3146454402 shutdown_connections 2024-02-21T23:09:41.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.160+0000 7f24cea00640 1 --2- 172.21.15.28:0/3146454402 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f24a806b4d0 0x7f24a806d990 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:41.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.160+0000 7f24cea00640 1 --2- 172.21.15.28:0/3146454402 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f24c80c76f0 0x7f24c80ae4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:41.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.160+0000 7f24cea00640 1 --2- 172.21.15.28:0/3146454402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24c813e0c0 0x7f24c8134070 unknown :-1 s=CLOSED pgs=1621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:41.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.160+0000 7f24cea00640 1 --2- 172.21.15.28:0/3146454402 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f24c806dbc0 0x7f24c80b0070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:41.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.160+0000 7f24cea00640 1 -- 172.21.15.28:0/3146454402 >> 172.21.15.28:0/3146454402 conn(0x7f24c8073110 msgr2=0x7f24c806c280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:41.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.160+0000 7f24cea00640 1 -- 172.21.15.28:0/3146454402 shutdown_connections 2024-02-21T23:09:41.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.161+0000 7f24cea00640 1 -- 172.21.15.28:0/3146454402 wait complete. 2024-02-21T23:09:41.172 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-02-21T23:09:41.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.415+0000 7f44e7e00640 1 Processor -- start 2024-02-21T23:09:41.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.415+0000 7f44e7e00640 1 -- start start 2024-02-21T23:09:41.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.417+0000 7f44e7e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f44e80ca540 0x7f44e8158460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:41.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.417+0000 7f44e7e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f44e80c6290 0x7f44e81522c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:41.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.417+0000 7f44e7e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44e80caf40 con 0x7f44e806cfa0 2024-02-21T23:09:41.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.417+0000 7f44e7e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44e80c8ae0 con 0x7f44e80ca540 2024-02-21T23:09:41.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.417+0000 7f44e7e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44e80c87b0 con 0x7f44e80c6290 2024-02-21T23:09:41.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.417+0000 7f44e6000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f44e80ca540 0x7f44e8158460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:41.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.417+0000 7f44e6000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f44e80ca540 0x7f44e8158460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32942/0 (socket says 172.21.15.28:32942) 2024-02-21T23:09:41.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.417+0000 7f44e6000640 1 -- 172.21.15.28:0/3430460998 learned_addr learned my addr 172.21.15.28:0/3430460998 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:41.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.417+0000 7f44e7400640 1 --2- 172.21.15.28:0/3430460998 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f44e80c6290 0x7f44e81522c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:41.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.418+0000 7f44e6000640 1 -- 172.21.15.28:0/3430460998 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f44e80c6290 msgr2=0x7f44e81522c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:41.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.418+0000 7f44e6000640 1 --2- 172.21.15.28:0/3430460998 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f44e80c6290 0x7f44e81522c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:41.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.418+0000 7f44e6000640 1 -- 172.21.15.28:0/3430460998 >> v1:172.21.15.81:6789/0 conn(0x7f44e806cfa0 legacy=0x7f44e80c8f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:41.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.418+0000 7f44e6000640 1 -- 172.21.15.28:0/3430460998 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44e80c7f80 con 0x7f44e80ca540 2024-02-21T23:09:41.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.418+0000 7f44e7400640 1 --2- 172.21.15.28:0/3430460998 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f44e80c6290 0x7f44e81522c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:09:41.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.419+0000 7f44e6000640 1 --2- 172.21.15.28:0/3430460998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44e80ca540 0x7f44e8158460 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f44d0009870 tx=0x7f44d002b7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6daa92bffc9d798 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:41.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.419+0000 7f44e5600640 1 -- 172.21.15.28:0/3430460998 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d0038070 con 0x7f44e80ca540 2024-02-21T23:09:41.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.419+0000 7f44e5600640 1 -- 172.21.15.28:0/3430460998 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44d0033620 con 0x7f44e80ca540 2024-02-21T23:09:41.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.419+0000 7f44e5600640 1 -- 172.21.15.28:0/3430460998 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d00339f0 con 0x7f44e80ca540 2024-02-21T23:09:41.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.420+0000 7f44e7e00640 1 -- 172.21.15.28:0/3430460998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44e80ca540 msgr2=0x7f44e8158460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:41.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.420+0000 7f44e7e00640 1 --2- 172.21.15.28:0/3430460998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44e80ca540 0x7f44e8158460 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f44d0009870 tx=0x7f44d002b7d0 comp rx=0 tx=0).stop 2024-02-21T23:09:41.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.420+0000 7f44e7e00640 1 -- 172.21.15.28:0/3430460998 shutdown_connections 2024-02-21T23:09:41.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.420+0000 7f44e7e00640 1 --2- 172.21.15.28:0/3430460998 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f44e80c6290 0x7f44e81522c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:41.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.420+0000 7f44e7e00640 1 --2- 172.21.15.28:0/3430460998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44e80ca540 0x7f44e8158460 unknown :-1 s=CLOSED pgs=1622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:41.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.420+0000 7f44e7e00640 1 -- 172.21.15.28:0/3430460998 >> 172.21.15.28:0/3430460998 conn(0x7f44e80725a0 msgr2=0x7f44e80729b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:41.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.420+0000 7f44e7e00640 1 -- 172.21.15.28:0/3430460998 shutdown_connections 2024-02-21T23:09:41.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.420+0000 7f44e7e00640 1 -- 172.21.15.28:0/3430460998 wait complete. 2024-02-21T23:09:41.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.422+0000 7f44e7e00640 1 Processor -- start 2024-02-21T23:09:41.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.422+0000 7f44e7e00640 1 -- start start 2024-02-21T23:09:41.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.423+0000 7f44e7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44e806cfa0 0x7f44e814c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:41.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.423+0000 7f44e7e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44e80c6290 0x7f44e8146850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:41.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.423+0000 7f44e7e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f44e80ca540 0x7f44e8146d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:41.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.423+0000 7f44e7e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44e80c66a0 con 0x7f44e806cfa0 2024-02-21T23:09:41.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.423+0000 7f44e7e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44e80c6c90 con 0x7f44e80c6290 2024-02-21T23:09:41.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.423+0000 7f44e7e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44e806d380 con 0x7f44e80ca540 2024-02-21T23:09:41.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.423+0000 7f44e6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44e806cfa0 0x7f44e814c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:41.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.423+0000 7f44e6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44e806cfa0 0x7f44e814c4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32952/0 (socket says 172.21.15.28:32952) 2024-02-21T23:09:41.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.423+0000 7f44e6a00640 1 -- 172.21.15.28:0/1345275364 learned_addr learned my addr 172.21.15.28:0/1345275364 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:41.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.423+0000 7f44e7400640 1 --2- 172.21.15.28:0/1345275364 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f44e80ca540 0x7f44e8146d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:41.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.423+0000 7f44e6a00640 1 -- 172.21.15.28:0/1345275364 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f44e80ca540 msgr2=0x7f44e8146d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:41.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.423+0000 7f44e6000640 1 --2- 172.21.15.28:0/1345275364 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44e80c6290 0x7f44e8146850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:41.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.423+0000 7f44e6a00640 1 --2- 172.21.15.28:0/1345275364 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f44e80ca540 0x7f44e8146d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:41.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.423+0000 7f44e6a00640 1 -- 172.21.15.28:0/1345275364 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44e80c6290 msgr2=0x7f44e8146850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:41.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.423+0000 7f44e6a00640 1 --2- 172.21.15.28:0/1345275364 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44e80c6290 0x7f44e8146850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:41.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.423+0000 7f44e6a00640 1 -- 172.21.15.28:0/1345275364 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44e80729e0 con 0x7f44e806cfa0 2024-02-21T23:09:41.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.423+0000 7f44e7400640 1 --2- 172.21.15.28:0/1345275364 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f44e80ca540 0x7f44e8146d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:09:41.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.424+0000 7f44e6a00640 1 --2- 172.21.15.28:0/1345275364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44e806cfa0 0x7f44e814c4a0 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f44d800ba50 tx=0x7f44d800bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:41.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.424+0000 7f44dea00640 1 -- 172.21.15.28:0/1345275364 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d8019070 con 0x7f44e806cfa0 2024-02-21T23:09:41.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.424+0000 7f44dea00640 1 -- 172.21.15.28:0/1345275364 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44d800cc20 con 0x7f44e806cfa0 2024-02-21T23:09:41.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.424+0000 7f44dea00640 1 -- 172.21.15.28:0/1345275364 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d8014680 con 0x7f44e806cfa0 2024-02-21T23:09:41.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.424+0000 7f44e7e00640 1 -- 172.21.15.28:0/1345275364 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44e81474a0 con 0x7f44e806cfa0 2024-02-21T23:09:41.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.424+0000 7f44e7e00640 1 -- 172.21.15.28:0/1345275364 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44e8153b00 con 0x7f44e806cfa0 2024-02-21T23:09:41.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.426+0000 7f44e7e00640 1 -- 172.21.15.28:0/1345275364 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44e804f290 con 0x7f44e806cfa0 2024-02-21T23:09:41.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.427+0000 7f44dea00640 1 -- 172.21.15.28:0/1345275364 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f44d801c070 con 0x7f44e806cfa0 2024-02-21T23:09:41.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.428+0000 7f44dea00640 1 --2- 172.21.15.28:0/1345275364 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f44c406b4d0 0x7f44c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:41.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.428+0000 7f44dea00640 1 -- 172.21.15.28:0/1345275364 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6417+0+0 (secure 0 0 0) 0x7f44d808a090 con 0x7f44e806cfa0 2024-02-21T23:09:41.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.428+0000 7f44e6000640 1 --2- 172.21.15.28:0/1345275364 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f44c406b4d0 0x7f44c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:41.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.429+0000 7f44e6000640 1 --2- 172.21.15.28:0/1345275364 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f44c406b4d0 0x7f44c406d990 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f44d0035000 tx=0x7f44d0008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:41.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.432+0000 7f44dea00640 1 -- 172.21.15.28:0/1345275364 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f44d8058680 con 0x7f44e806cfa0 2024-02-21T23:09:41.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:41.630+0000 7f44e7e00640 1 -- 172.21.15.28:0/1345275364 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f44e80adbd0 con 0x7f44e806cfa0 2024-02-21T23:09:42.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.160+0000 7f44dea00640 1 -- 172.21.15.28:0/1345275364 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v142) v1 ==== 181+0+0 (secure 0 0 0) 0x7f44d805c720 con 0x7f44e806cfa0 2024-02-21T23:09:42.150 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' removed 2024-02-21T23:09:42.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.164+0000 7f44e7e00640 1 -- 172.21.15.28:0/1345275364 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f44c406b4d0 msgr2=0x7f44c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:42.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.164+0000 7f44e7e00640 1 --2- 172.21.15.28:0/1345275364 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f44c406b4d0 0x7f44c406d990 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f44d0035000 tx=0x7f44d0008030 comp rx=0 tx=0).stop 2024-02-21T23:09:42.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.164+0000 7f44e7e00640 1 -- 172.21.15.28:0/1345275364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44e806cfa0 msgr2=0x7f44e814c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:42.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.164+0000 7f44e7e00640 1 --2- 172.21.15.28:0/1345275364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44e806cfa0 0x7f44e814c4a0 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f44d800ba50 tx=0x7f44d800bf10 comp rx=0 tx=0).stop 2024-02-21T23:09:42.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.166+0000 7f44e7e00640 1 -- 172.21.15.28:0/1345275364 shutdown_connections 2024-02-21T23:09:42.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.166+0000 7f44e7e00640 1 --2- 172.21.15.28:0/1345275364 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f44c406b4d0 0x7f44c406d990 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:42.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.166+0000 7f44e7e00640 1 --2- 172.21.15.28:0/1345275364 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f44e80ca540 0x7f44e8146d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:42.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.166+0000 7f44e7e00640 1 --2- 172.21.15.28:0/1345275364 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44e80c6290 0x7f44e8146850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:42.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.166+0000 7f44e7e00640 1 --2- 172.21.15.28:0/1345275364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44e806cfa0 0x7f44e814c4a0 unknown :-1 s=CLOSED pgs=1623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:42.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.166+0000 7f44e7e00640 1 -- 172.21.15.28:0/1345275364 >> 172.21.15.28:0/1345275364 conn(0x7f44e80725a0 msgr2=0x7f44e80c8f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:42.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.167+0000 7f44e7e00640 1 -- 172.21.15.28:0/1345275364 shutdown_connections 2024-02-21T23:09:42.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.169+0000 7f44e7e00640 1 -- 172.21.15.28:0/1345275364 wait complete. 2024-02-21T23:09:42.181 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-02-21T23:09:42.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.408+0000 7f898ba00640 1 Processor -- start 2024-02-21T23:09:42.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.408+0000 7f898ba00640 1 -- start start 2024-02-21T23:09:42.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.409+0000 7f898ba00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f898406e780 0x7f898406eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:42.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.409+0000 7f898ba00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f89840a5fc0 0x7f89840c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:42.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.409+0000 7f898ba00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f89840c50c0 0x7f89841564d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:42.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.409+0000 7f898ba00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89841543d0 con 0x7f89840c50c0 2024-02-21T23:09:42.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.409+0000 7f898ba00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89840c5ad0 con 0x7f89840a5fc0 2024-02-21T23:09:42.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.409+0000 7f898ba00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89840c46f0 con 0x7f898406e780 2024-02-21T23:09:42.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.410+0000 7f8989a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f89840c50c0 0x7f89841564d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:42.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.410+0000 7f8989a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f89840c50c0 0x7f89841564d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32962/0 (socket says 172.21.15.28:32962) 2024-02-21T23:09:42.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.410+0000 7f8989a00640 1 -- 172.21.15.28:0/140117203 learned_addr learned my addr 172.21.15.28:0/140117203 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:42.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.410+0000 7f8989000640 1 --2- 172.21.15.28:0/140117203 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f898406e780 0x7f898406eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:42.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.410+0000 7f8983e00640 1 --2- 172.21.15.28:0/140117203 >> v2:172.21.15.81:3300/0 conn(0x7f89840a5fc0 0x7f89840c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:42.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.410+0000 7f8989a00640 1 -- 172.21.15.28:0/140117203 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f898406e780 msgr2=0x7f898406eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:42.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.410+0000 7f8989a00640 1 --2- 172.21.15.28:0/140117203 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f898406e780 0x7f898406eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:42.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.410+0000 7f8989a00640 1 -- 172.21.15.28:0/140117203 >> v2:172.21.15.81:3300/0 conn(0x7f89840a5fc0 msgr2=0x7f89840c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:42.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.410+0000 7f8989a00640 1 --2- 172.21.15.28:0/140117203 >> v2:172.21.15.81:3300/0 conn(0x7f89840a5fc0 0x7f89840c4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:42.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.410+0000 7f8989a00640 1 -- 172.21.15.28:0/140117203 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8984153cb0 con 0x7f89840c50c0 2024-02-21T23:09:42.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.412+0000 7f8989a00640 1 --2- 172.21.15.28:0/140117203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89840c50c0 0x7f89841564d0 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f898406f900 tx=0x7f897c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c190ba0d0321432 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:42.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.412+0000 7f8983400640 1 -- 172.21.15.28:0/140117203 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f897c02c9e0 con 0x7f89840c50c0 2024-02-21T23:09:42.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.412+0000 7f8983400640 1 -- 172.21.15.28:0/140117203 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f897c02cb80 con 0x7f89840c50c0 2024-02-21T23:09:42.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.412+0000 7f8983400640 1 -- 172.21.15.28:0/140117203 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f897c0343f0 con 0x7f89840c50c0 2024-02-21T23:09:42.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.413+0000 7f898ba00640 1 -- 172.21.15.28:0/140117203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89840c50c0 msgr2=0x7f89841564d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:42.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.413+0000 7f898ba00640 1 --2- 172.21.15.28:0/140117203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89840c50c0 0x7f89841564d0 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f898406f900 tx=0x7f897c02be40 comp rx=0 tx=0).stop 2024-02-21T23:09:42.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.413+0000 7f898ba00640 1 -- 172.21.15.28:0/140117203 shutdown_connections 2024-02-21T23:09:42.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.413+0000 7f898ba00640 1 --2- 172.21.15.28:0/140117203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89840c50c0 0x7f89841564d0 unknown :-1 s=CLOSED pgs=1624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:42.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.413+0000 7f898ba00640 1 --2- 172.21.15.28:0/140117203 >> v2:172.21.15.81:3300/0 conn(0x7f89840a5fc0 0x7f89840c4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:42.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.413+0000 7f898ba00640 1 --2- 172.21.15.28:0/140117203 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f898406e780 0x7f898406eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:42.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.413+0000 7f898ba00640 1 -- 172.21.15.28:0/140117203 >> 172.21.15.28:0/140117203 conn(0x7f898406b830 msgr2=0x7f89840c67f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:42.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.414+0000 7f898ba00640 1 -- 172.21.15.28:0/140117203 shutdown_connections 2024-02-21T23:09:42.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.414+0000 7f898ba00640 1 -- 172.21.15.28:0/140117203 wait complete. 2024-02-21T23:09:42.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.415+0000 7f898ba00640 1 Processor -- start 2024-02-21T23:09:42.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.416+0000 7f898ba00640 1 -- start start 2024-02-21T23:09:42.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.416+0000 7f898ba00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f898406e780 0x7f89840a2260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:42.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.416+0000 7f898ba00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f89840a5fc0 0x7f8984149320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:42.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.417+0000 7f898ba00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f89840c50c0 0x7f898414d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:42.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.417+0000 7f898ba00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8984153960 con 0x7f898406e780 2024-02-21T23:09:42.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.417+0000 7f898ba00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89840c6000 con 0x7f89840c50c0 2024-02-21T23:09:42.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.417+0000 7f898ba00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f898406d110 con 0x7f89840a5fc0 2024-02-21T23:09:42.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.417+0000 7f8989000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f898406e780 0x7f89840a2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:42.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.417+0000 7f8989000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f898406e780 0x7f89840a2260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32966/0 (socket says 172.21.15.28:32966) 2024-02-21T23:09:42.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.417+0000 7f8989000640 1 -- 172.21.15.28:0/1063031662 learned_addr learned my addr 172.21.15.28:0/1063031662 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:42.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.417+0000 7f8983e00640 1 --2- 172.21.15.28:0/1063031662 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f89840a5fc0 0x7f8984149320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:42.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.417+0000 7f8989a00640 1 --2- 172.21.15.28:0/1063031662 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f89840c50c0 0x7f898414d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:42.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.417+0000 7f8989000640 1 -- 172.21.15.28:0/1063031662 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f89840a5fc0 msgr2=0x7f8984149320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:42.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.417+0000 7f8989000640 1 --2- 172.21.15.28:0/1063031662 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f89840a5fc0 0x7f8984149320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:42.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.417+0000 7f8989000640 1 -- 172.21.15.28:0/1063031662 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f89840c50c0 msgr2=0x7f898414d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:42.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.417+0000 7f8989000640 1 --2- 172.21.15.28:0/1063031662 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f89840c50c0 0x7f898414d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:42.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.417+0000 7f8989000640 1 -- 172.21.15.28:0/1063031662 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89841f33c0 con 0x7f898406e780 2024-02-21T23:09:42.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.417+0000 7f8983e00640 1 --2- 172.21.15.28:0/1063031662 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f89840a5fc0 0x7f8984149320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:09:42.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.417+0000 7f8989a00640 1 --2- 172.21.15.28:0/1063031662 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f89840c50c0 0x7f898414d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:09:42.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.418+0000 7f8989000640 1 --2- 172.21.15.28:0/1063031662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f898406e780 0x7f89840a2260 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f897800d620 tx=0x7f897800dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:42.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.418+0000 7f8980c00640 1 -- 172.21.15.28:0/1063031662 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f897800b620 con 0x7f898406e780 2024-02-21T23:09:42.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.418+0000 7f8980c00640 1 -- 172.21.15.28:0/1063031662 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8978019070 con 0x7f898406e780 2024-02-21T23:09:42.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.418+0000 7f8980c00640 1 -- 172.21.15.28:0/1063031662 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f897800ba00 con 0x7f898406e780 2024-02-21T23:09:42.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.418+0000 7f898ba00640 1 -- 172.21.15.28:0/1063031662 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89841f2600 con 0x7f898406e780 2024-02-21T23:09:42.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.418+0000 7f898ba00640 1 -- 172.21.15.28:0/1063031662 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89841f2bc0 con 0x7f898406e780 2024-02-21T23:09:42.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.419+0000 7f898ba00640 1 -- 172.21.15.28:0/1063031662 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89840c5550 con 0x7f898406e780 2024-02-21T23:09:42.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.420+0000 7f8980c00640 1 -- 172.21.15.28:0/1063031662 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f897800bba0 con 0x7f898406e780 2024-02-21T23:09:42.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.421+0000 7f8980c00640 1 --2- 172.21.15.28:0/1063031662 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f896406b4d0 0x7f896406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:42.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.421+0000 7f8980c00640 1 -- 172.21.15.28:0/1063031662 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 5979+0+0 (secure 0 0 0) 0x7f897808a760 con 0x7f898406e780 2024-02-21T23:09:42.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.422+0000 7f8983e00640 1 --2- 172.21.15.28:0/1063031662 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f896406b4d0 0x7f896406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:42.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.423+0000 7f8983e00640 1 --2- 172.21.15.28:0/1063031662 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f896406b4d0 0x7f896406d990 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f89841eccf0 tx=0x7f8970008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:42.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.426+0000 7f8980c00640 1 -- 172.21.15.28:0/1063031662 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f89840c5550 con 0x7f898406e780 2024-02-21T23:09:42.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:42.631+0000 7f898ba00640 1 -- 172.21.15.28:0/1063031662 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f898406f400 con 0x7f898406e780 2024-02-21T23:09:43.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.164+0000 7f8980c00640 1 -- 172.21.15.28:0/1063031662 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v143) v1 ==== 169+0+0 (secure 0 0 0) 0x7f898406f400 con 0x7f898406e780 2024-02-21T23:09:43.153 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' removed 2024-02-21T23:09:43.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.169+0000 7f898ba00640 1 -- 172.21.15.28:0/1063031662 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f896406b4d0 msgr2=0x7f896406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:43.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.169+0000 7f898ba00640 1 --2- 172.21.15.28:0/1063031662 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f896406b4d0 0x7f896406d990 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f89841eccf0 tx=0x7f8970008040 comp rx=0 tx=0).stop 2024-02-21T23:09:43.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.169+0000 7f898ba00640 1 -- 172.21.15.28:0/1063031662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f898406e780 msgr2=0x7f89840a2260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:43.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.169+0000 7f898ba00640 1 --2- 172.21.15.28:0/1063031662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f898406e780 0x7f89840a2260 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f897800d620 tx=0x7f897800dae0 comp rx=0 tx=0).stop 2024-02-21T23:09:43.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.170+0000 7f898ba00640 1 -- 172.21.15.28:0/1063031662 shutdown_connections 2024-02-21T23:09:43.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.170+0000 7f898ba00640 1 --2- 172.21.15.28:0/1063031662 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f896406b4d0 0x7f896406d990 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:43.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.170+0000 7f898ba00640 1 --2- 172.21.15.28:0/1063031662 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f89840a5fc0 0x7f8984149320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:43.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.170+0000 7f898ba00640 1 --2- 172.21.15.28:0/1063031662 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f89840c50c0 0x7f898414d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:43.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.170+0000 7f898ba00640 1 --2- 172.21.15.28:0/1063031662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f898406e780 0x7f89840a2260 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:43.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.170+0000 7f898ba00640 1 -- 172.21.15.28:0/1063031662 >> 172.21.15.28:0/1063031662 conn(0x7f898406b830 msgr2=0x7f89841560f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:43.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.170+0000 7f898ba00640 1 -- 172.21.15.28:0/1063031662 shutdown_connections 2024-02-21T23:09:43.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.170+0000 7f898ba00640 1 -- 172.21.15.28:0/1063031662 wait complete. 2024-02-21T23:09:43.183 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.b mds_join_fs 2024-02-21T23:09:43.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.413+0000 7f90a1800640 1 Processor -- start 2024-02-21T23:09:43.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.413+0000 7f90a1800640 1 -- start start 2024-02-21T23:09:43.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.414+0000 7f90a1800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f909c0c2c60 0x7f909c06dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:43.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.414+0000 7f90a1800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f909c06f400 0x7f909c06e530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:43.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.414+0000 7f90a1800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f909c06db10 con 0x7f909c1500f0 2024-02-21T23:09:43.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.414+0000 7f90a1800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f909c0c1080 con 0x7f909c06f400 2024-02-21T23:09:43.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.414+0000 7f90a1800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f909c1539a0 con 0x7f909c0c2c60 2024-02-21T23:09:43.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.414+0000 7f90a0e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f909c06f400 0x7f909c06e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:43.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.414+0000 7f90a0e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f909c06f400 0x7f909c06e530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32976/0 (socket says 172.21.15.28:32976) 2024-02-21T23:09:43.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.414+0000 7f90a0e00640 1 -- 172.21.15.28:0/3824330530 learned_addr learned my addr 172.21.15.28:0/3824330530 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:43.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.415+0000 7f9093400640 1 --2- 172.21.15.28:0/3824330530 >> v2:172.21.15.81:3300/0 conn(0x7f909c0c2c60 0x7f909c06dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:43.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.415+0000 7f90a0e00640 1 -- 172.21.15.28:0/3824330530 >> v2:172.21.15.81:3300/0 conn(0x7f909c0c2c60 msgr2=0x7f909c06dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:43.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.415+0000 7f90a0e00640 1 --2- 172.21.15.28:0/3824330530 >> v2:172.21.15.81:3300/0 conn(0x7f909c0c2c60 0x7f909c06dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:43.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.415+0000 7f90a0e00640 1 -- 172.21.15.28:0/3824330530 >> v1:172.21.15.81:6789/0 conn(0x7f909c1500f0 legacy=0x7f909c06ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:43.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.415+0000 7f90a0e00640 1 -- 172.21.15.28:0/3824330530 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f909c0c12d0 con 0x7f909c06f400 2024-02-21T23:09:43.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.416+0000 7f90a0e00640 1 --2- 172.21.15.28:0/3824330530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f909c06f400 0x7f909c06e530 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f909c06dce0 tx=0x7f909402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5394e765db7571d1 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:43.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.417+0000 7f909b400640 1 -- 172.21.15.28:0/3824330530 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9094035020 con 0x7f909c06f400 2024-02-21T23:09:43.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.417+0000 7f909b400640 1 -- 172.21.15.28:0/3824330530 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9094031870 con 0x7f909c06f400 2024-02-21T23:09:43.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.417+0000 7f909b400640 1 -- 172.21.15.28:0/3824330530 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9094031c40 con 0x7f909c06f400 2024-02-21T23:09:43.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.417+0000 7f90a1800640 1 -- 172.21.15.28:0/3824330530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f909c06f400 msgr2=0x7f909c06e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:43.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.417+0000 7f90a1800640 1 --2- 172.21.15.28:0/3824330530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f909c06f400 0x7f909c06e530 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f909c06dce0 tx=0x7f909402b910 comp rx=0 tx=0).stop 2024-02-21T23:09:43.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.417+0000 7f90a1800640 1 -- 172.21.15.28:0/3824330530 shutdown_connections 2024-02-21T23:09:43.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.417+0000 7f90a1800640 1 --2- 172.21.15.28:0/3824330530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f909c06f400 0x7f909c06e530 unknown :-1 s=CLOSED pgs=1626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:43.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.417+0000 7f90a1800640 1 --2- 172.21.15.28:0/3824330530 >> v2:172.21.15.81:3300/0 conn(0x7f909c0c2c60 0x7f909c06dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:43.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.417+0000 7f90a1800640 1 -- 172.21.15.28:0/3824330530 >> 172.21.15.28:0/3824330530 conn(0x7f909c0a8530 msgr2=0x7f909c0a8910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:43.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.418+0000 7f90a1800640 1 -- 172.21.15.28:0/3824330530 shutdown_connections 2024-02-21T23:09:43.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.418+0000 7f90a1800640 1 -- 172.21.15.28:0/3824330530 wait complete. 2024-02-21T23:09:43.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.419+0000 7f90a1800640 1 Processor -- start 2024-02-21T23:09:43.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.419+0000 7f90a1800640 1 -- start start 2024-02-21T23:09:43.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.420+0000 7f90a1800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f909c06f400 0x7f909c0a9cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:43.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.420+0000 7f90a1800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f909c0c2c60 0x7f909c0c43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:43.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.420+0000 7f90a1800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f909c1500f0 0x7f909c1e9900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:43.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.420+0000 7f90a1800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f909c06b580 con 0x7f909c1500f0 2024-02-21T23:09:43.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.420+0000 7f90a1800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f909c1504d0 con 0x7f909c0c2c60 2024-02-21T23:09:43.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.420+0000 7f90a1800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f909c06b700 con 0x7f909c06f400 2024-02-21T23:09:43.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.421+0000 7f90a0e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f909c1500f0 0x7f909c1e9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:43.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.421+0000 7f90a0e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f909c1500f0 0x7f909c1e9900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32988/0 (socket says 172.21.15.28:32988) 2024-02-21T23:09:43.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.421+0000 7f90a0e00640 1 -- 172.21.15.28:0/598140560 learned_addr learned my addr 172.21.15.28:0/598140560 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:43.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.421+0000 7f909be00640 1 --2- 172.21.15.28:0/598140560 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f909c06f400 0x7f909c0a9cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:43.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.421+0000 7f9093400640 1 --2- 172.21.15.28:0/598140560 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f909c0c2c60 0x7f909c0c43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:43.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.421+0000 7f90a0e00640 1 -- 172.21.15.28:0/598140560 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f909c06f400 msgr2=0x7f909c0a9cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:43.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.421+0000 7f90a0e00640 1 --2- 172.21.15.28:0/598140560 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f909c06f400 0x7f909c0a9cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:43.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.421+0000 7f90a0e00640 1 -- 172.21.15.28:0/598140560 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f909c0c2c60 msgr2=0x7f909c0c43e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:43.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.421+0000 7f90a0e00640 1 --2- 172.21.15.28:0/598140560 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f909c0c2c60 0x7f909c0c43e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:43.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.421+0000 7f90a0e00640 1 -- 172.21.15.28:0/598140560 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f909c1df050 con 0x7f909c1500f0 2024-02-21T23:09:43.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.422+0000 7f90a0e00640 1 --2- 172.21.15.28:0/598140560 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f909c1500f0 0x7f909c1e9900 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f909402b950 tx=0x7f90940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:43.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.422+0000 7f9098c00640 1 -- 172.21.15.28:0/598140560 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9094040070 con 0x7f909c1500f0 2024-02-21T23:09:43.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.422+0000 7f9098c00640 1 -- 172.21.15.28:0/598140560 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9094030d00 con 0x7f909c1500f0 2024-02-21T23:09:43.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.422+0000 7f9098c00640 1 -- 172.21.15.28:0/598140560 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9094035040 con 0x7f909c1500f0 2024-02-21T23:09:43.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.422+0000 7f90a1800640 1 -- 172.21.15.28:0/598140560 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f909c0ac300 con 0x7f909c1500f0 2024-02-21T23:09:43.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.422+0000 7f90a1800640 1 -- 172.21.15.28:0/598140560 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f909c1ded70 con 0x7f909c1500f0 2024-02-21T23:09:43.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.423+0000 7f90a1800640 1 -- 172.21.15.28:0/598140560 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f909c04f290 con 0x7f909c1500f0 2024-02-21T23:09:43.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.424+0000 7f9098c00640 1 -- 172.21.15.28:0/598140560 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9094038020 con 0x7f909c1500f0 2024-02-21T23:09:43.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.427+0000 7f9098c00640 1 --2- 172.21.15.28:0/598140560 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f907806b4d0 0x7f907806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:43.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.427+0000 7f9098c00640 1 -- 172.21.15.28:0/598140560 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f90940abfe0 con 0x7f909c1500f0 2024-02-21T23:09:43.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.427+0000 7f909be00640 1 --2- 172.21.15.28:0/598140560 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f907806b4d0 0x7f907806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:43.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.428+0000 7f909be00640 1 --2- 172.21.15.28:0/598140560 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f907806b4d0 0x7f907806d990 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f908c009f20 tx=0x7f908c0094d0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:43.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.431+0000 7f9098c00640 1 -- 172.21.15.28:0/598140560 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f909407af10 con 0x7f909c1500f0 2024-02-21T23:09:43.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.632+0000 7f90a1800640 1 -- 172.21.15.28:0/598140560 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.b", "name": "mds_join_fs"} v 0) v1 -- 0x7f909c06f1e0 con 0x7f909c1500f0 2024-02-21T23:09:43.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.633+0000 7f9098c00640 1 -- 172.21.15.28:0/598140560 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.b", "name": "mds_join_fs"}]=0 v48) v1 ==== 96+0+0 (secure 0 0 0) 0x7f909407a810 con 0x7f909c1500f0 2024-02-21T23:09:43.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.636+0000 7f90a1800640 1 -- 172.21.15.28:0/598140560 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f907806b4d0 msgr2=0x7f907806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:43.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.636+0000 7f90a1800640 1 --2- 172.21.15.28:0/598140560 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f907806b4d0 0x7f907806d990 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f908c009f20 tx=0x7f908c0094d0 comp rx=0 tx=0).stop 2024-02-21T23:09:43.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.636+0000 7f90a1800640 1 -- 172.21.15.28:0/598140560 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f909c1500f0 msgr2=0x7f909c1e9900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:43.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.636+0000 7f90a1800640 1 --2- 172.21.15.28:0/598140560 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f909c1500f0 0x7f909c1e9900 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f909402b950 tx=0x7f90940046c0 comp rx=0 tx=0).stop 2024-02-21T23:09:43.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.636+0000 7f90a1800640 1 -- 172.21.15.28:0/598140560 shutdown_connections 2024-02-21T23:09:43.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.636+0000 7f90a1800640 1 --2- 172.21.15.28:0/598140560 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f907806b4d0 0x7f907806d990 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:43.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.636+0000 7f90a1800640 1 --2- 172.21.15.28:0/598140560 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f909c1500f0 0x7f909c1e9900 unknown :-1 s=CLOSED pgs=1627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:43.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.636+0000 7f90a1800640 1 --2- 172.21.15.28:0/598140560 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f909c0c2c60 0x7f909c0c43e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:43.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.636+0000 7f90a1800640 1 --2- 172.21.15.28:0/598140560 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f909c06f400 0x7f909c0a9cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:43.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.636+0000 7f90a1800640 1 -- 172.21.15.28:0/598140560 >> 172.21.15.28:0/598140560 conn(0x7f909c0a8530 msgr2=0x7f909c1523e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:43.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.636+0000 7f90a1800640 1 -- 172.21.15.28:0/598140560 shutdown_connections 2024-02-21T23:09:43.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.636+0000 7f90a1800640 1 -- 172.21.15.28:0/598140560 wait complete. 2024-02-21T23:09:43.645 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.d mds_join_fs 2024-02-21T23:09:43.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.882+0000 7fd52da00640 1 Processor -- start 2024-02-21T23:09:43.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.882+0000 7fd52da00640 1 -- start start 2024-02-21T23:09:43.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.883+0000 7fd52da00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd52806dbc0 0x7fd52806c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:43.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.883+0000 7fd52da00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fd52813e520 0x7fd52806c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:43.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.883+0000 7fd52da00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd528158550 con 0x7fd52813e0c0 2024-02-21T23:09:43.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.883+0000 7fd52da00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5280c89a0 con 0x7fd52806dbc0 2024-02-21T23:09:43.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.883+0000 7fd52da00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5281587d0 con 0x7fd52813e520 2024-02-21T23:09:43.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.883+0000 7fd527400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd52806dbc0 0x7fd52806c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:43.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.883+0000 7fd527400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd52806dbc0 0x7fd52806c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32992/0 (socket says 172.21.15.28:32992) 2024-02-21T23:09:43.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.883+0000 7fd527400640 1 -- 172.21.15.28:0/846129073 learned_addr learned my addr 172.21.15.28:0/846129073 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:43.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.884+0000 7fd52d000640 1 --2- 172.21.15.28:0/846129073 >> v2:172.21.15.81:3300/0 conn(0x7fd52813e520 0x7fd52806c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:43.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.884+0000 7fd527400640 1 -- 172.21.15.28:0/846129073 >> v2:172.21.15.81:3300/0 conn(0x7fd52813e520 msgr2=0x7fd52806c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:43.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.884+0000 7fd527400640 1 --2- 172.21.15.28:0/846129073 >> v2:172.21.15.81:3300/0 conn(0x7fd52813e520 0x7fd52806c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:43.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.884+0000 7fd527400640 1 -- 172.21.15.28:0/846129073 >> v1:172.21.15.81:6789/0 conn(0x7fd52813e0c0 legacy=0x7fd5280c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:43.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.884+0000 7fd527400640 1 -- 172.21.15.28:0/846129073 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd528158ab0 con 0x7fd52806dbc0 2024-02-21T23:09:43.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.885+0000 7fd527400640 1 --2- 172.21.15.28:0/846129073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd52806dbc0 0x7fd52806c280 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7fd5280c8260 tx=0x7fd51002bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6e85f9ba81b713 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:43.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.885+0000 7fd526a00640 1 -- 172.21.15.28:0/846129073 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd51002c960 con 0x7fd52806dbc0 2024-02-21T23:09:43.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.885+0000 7fd526a00640 1 -- 172.21.15.28:0/846129073 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd51002cb00 con 0x7fd52806dbc0 2024-02-21T23:09:43.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.885+0000 7fd526a00640 1 -- 172.21.15.28:0/846129073 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5100343f0 con 0x7fd52806dbc0 2024-02-21T23:09:43.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.886+0000 7fd52da00640 1 -- 172.21.15.28:0/846129073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd52806dbc0 msgr2=0x7fd52806c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:43.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.886+0000 7fd52da00640 1 --2- 172.21.15.28:0/846129073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd52806dbc0 0x7fd52806c280 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7fd5280c8260 tx=0x7fd51002bdc0 comp rx=0 tx=0).stop 2024-02-21T23:09:43.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.886+0000 7fd52da00640 1 -- 172.21.15.28:0/846129073 shutdown_connections 2024-02-21T23:09:43.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.886+0000 7fd52da00640 1 --2- 172.21.15.28:0/846129073 >> v2:172.21.15.81:3300/0 conn(0x7fd52813e520 0x7fd52806c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:43.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.886+0000 7fd52da00640 1 --2- 172.21.15.28:0/846129073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd52806dbc0 0x7fd52806c280 unknown :-1 s=CLOSED pgs=1628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:43.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.886+0000 7fd52da00640 1 -- 172.21.15.28:0/846129073 >> 172.21.15.28:0/846129073 conn(0x7fd528073110 msgr2=0x7fd5280ac490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:43.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.886+0000 7fd52da00640 1 -- 172.21.15.28:0/846129073 shutdown_connections 2024-02-21T23:09:43.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.886+0000 7fd52da00640 1 -- 172.21.15.28:0/846129073 wait complete. 2024-02-21T23:09:43.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.888+0000 7fd52da00640 1 Processor -- start 2024-02-21T23:09:43.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.888+0000 7fd52da00640 1 -- start start 2024-02-21T23:09:43.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.889+0000 7fd52da00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd52806dbc0 0x7fd52813be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:43.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.889+0000 7fd52da00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd52813e0c0 0x7fd528151da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:43.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.889+0000 7fd52da00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd52813e520 0x7fd528157500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:43.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.889+0000 7fd52da00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5280c78c0 con 0x7fd52806dbc0 2024-02-21T23:09:43.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.889+0000 7fd52da00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd528159070 con 0x7fd52813e0c0 2024-02-21T23:09:43.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.889+0000 7fd52da00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5280c7a40 con 0x7fd52813e520 2024-02-21T23:09:43.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.889+0000 7fd527e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd52806dbc0 0x7fd52813be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:43.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.889+0000 7fd527e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd52806dbc0 0x7fd52813be60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33004/0 (socket says 172.21.15.28:33004) 2024-02-21T23:09:43.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.889+0000 7fd527e00640 1 -- 172.21.15.28:0/2720210180 learned_addr learned my addr 172.21.15.28:0/2720210180 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:43.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.890+0000 7fd527400640 1 --2- 172.21.15.28:0/2720210180 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd52813e0c0 0x7fd528151da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:43.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.890+0000 7fd52d000640 1 --2- 172.21.15.28:0/2720210180 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd52813e520 0x7fd528157500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:43.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.890+0000 7fd527e00640 1 -- 172.21.15.28:0/2720210180 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd52813e520 msgr2=0x7fd528157500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:43.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.890+0000 7fd527e00640 1 --2- 172.21.15.28:0/2720210180 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd52813e520 0x7fd528157500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:43.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.890+0000 7fd527e00640 1 -- 172.21.15.28:0/2720210180 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd52813e0c0 msgr2=0x7fd528151da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:43.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.890+0000 7fd527e00640 1 --2- 172.21.15.28:0/2720210180 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd52813e0c0 0x7fd528151da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:43.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.890+0000 7fd527e00640 1 -- 172.21.15.28:0/2720210180 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5281eb020 con 0x7fd52806dbc0 2024-02-21T23:09:43.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.890+0000 7fd527e00640 1 --2- 172.21.15.28:0/2720210180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd52806dbc0 0x7fd52813be60 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7fd51800b3f0 tx=0x7fd51800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:43.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.891+0000 7fd51fe00640 1 -- 172.21.15.28:0/2720210180 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd518015020 con 0x7fd52806dbc0 2024-02-21T23:09:43.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.891+0000 7fd51fe00640 1 -- 172.21.15.28:0/2720210180 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd518011830 con 0x7fd52806dbc0 2024-02-21T23:09:43.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.891+0000 7fd51fe00640 1 -- 172.21.15.28:0/2720210180 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd518011c20 con 0x7fd52806dbc0 2024-02-21T23:09:43.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.891+0000 7fd52da00640 1 -- 172.21.15.28:0/2720210180 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5281eb300 con 0x7fd52806dbc0 2024-02-21T23:09:43.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.891+0000 7fd52da00640 1 -- 172.21.15.28:0/2720210180 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5281eb8c0 con 0x7fd52806dbc0 2024-02-21T23:09:43.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.892+0000 7fd52da00640 1 -- 172.21.15.28:0/2720210180 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5281534e0 con 0x7fd52806dbc0 2024-02-21T23:09:43.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.894+0000 7fd51fe00640 1 -- 172.21.15.28:0/2720210180 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd518018020 con 0x7fd52806dbc0 2024-02-21T23:09:43.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.895+0000 7fd51fe00640 1 --2- 172.21.15.28:0/2720210180 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd50406b4d0 0x7fd50406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:43.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.895+0000 7fd51fe00640 1 -- 172.21.15.28:0/2720210180 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5507+0+0 (secure 0 0 0) 0x7fd518089cd0 con 0x7fd52806dbc0 2024-02-21T23:09:43.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.895+0000 7fd527400640 1 --2- 172.21.15.28:0/2720210180 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd50406b4d0 0x7fd50406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:43.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.896+0000 7fd527400640 1 --2- 172.21.15.28:0/2720210180 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd50406b4d0 0x7fd50406d990 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fd510035000 tx=0x7fd510030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:43.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:43.898+0000 7fd51fe00640 1 -- 172.21.15.28:0/2720210180 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd51801f030 con 0x7fd52806dbc0 2024-02-21T23:09:44.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.107+0000 7fd52da00640 1 -- 172.21.15.28:0/2720210180 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.d", "name": "mds_join_fs"} v 0) v1 -- 0x7fd52804f290 con 0x7fd52806dbc0 2024-02-21T23:09:44.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.112+0000 7fd51fe00640 1 -- 172.21.15.28:0/2720210180 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.d", "name": "mds_join_fs"}]=0 v49) v1 ==== 96+0+0 (secure 0 0 0) 0x7fd5180172c0 con 0x7fd52806dbc0 2024-02-21T23:09:44.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.115+0000 7fd52da00640 1 -- 172.21.15.28:0/2720210180 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd50406b4d0 msgr2=0x7fd50406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:44.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.115+0000 7fd52da00640 1 --2- 172.21.15.28:0/2720210180 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd50406b4d0 0x7fd50406d990 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fd510035000 tx=0x7fd510030000 comp rx=0 tx=0).stop 2024-02-21T23:09:44.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.115+0000 7fd52da00640 1 -- 172.21.15.28:0/2720210180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd52806dbc0 msgr2=0x7fd52813be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:44.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.115+0000 7fd52da00640 1 --2- 172.21.15.28:0/2720210180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd52806dbc0 0x7fd52813be60 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7fd51800b3f0 tx=0x7fd51800b8b0 comp rx=0 tx=0).stop 2024-02-21T23:09:44.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.115+0000 7fd52da00640 1 -- 172.21.15.28:0/2720210180 shutdown_connections 2024-02-21T23:09:44.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.115+0000 7fd52da00640 1 --2- 172.21.15.28:0/2720210180 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd50406b4d0 0x7fd50406d990 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:44.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.115+0000 7fd52da00640 1 --2- 172.21.15.28:0/2720210180 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd52813e520 0x7fd528157500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:44.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.115+0000 7fd52da00640 1 --2- 172.21.15.28:0/2720210180 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd52813e0c0 0x7fd528151da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:44.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.115+0000 7fd52da00640 1 --2- 172.21.15.28:0/2720210180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd52806dbc0 0x7fd52813be60 unknown :-1 s=CLOSED pgs=1629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:44.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.115+0000 7fd52da00640 1 -- 172.21.15.28:0/2720210180 >> 172.21.15.28:0/2720210180 conn(0x7fd528073110 msgr2=0x7fd528139c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:44.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.115+0000 7fd52da00640 1 -- 172.21.15.28:0/2720210180 shutdown_connections 2024-02-21T23:09:44.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.115+0000 7fd52da00640 1 -- 172.21.15.28:0/2720210180 wait complete. 2024-02-21T23:09:44.123 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-02-21T23:09:44.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.346+0000 7fc168a00640 1 Processor -- start 2024-02-21T23:09:44.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.347+0000 7fc168a00640 1 -- start start 2024-02-21T23:09:44.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.348+0000 7fc168a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fc1640c22e0 0x7fc16406d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:44.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.348+0000 7fc168a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc164071970 con 0x7fc1641561a0 2024-02-21T23:09:44.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.348+0000 7fc168a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc164071c50 con 0x7fc1640c5a70 2024-02-21T23:09:44.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.348+0000 7fc168a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc164072a10 con 0x7fc1640c22e0 2024-02-21T23:09:44.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.349+0000 7fc162800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fc1641561a0 0x7fc16406d660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60130/0 (socket says 172.21.15.28:60130) 2024-02-21T23:09:44.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.349+0000 7fc162800640 1 -- 172.21.15.28:0/3121814700 learned_addr learned my addr 172.21.15.28:0/3121814700 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:09:44.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.349+0000 7fc161400640 1 -- 172.21.15.28:0/3121814700 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1106488885 0 0) 0x7fc164071970 con 0x7fc1641561a0 2024-02-21T23:09:44.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.349+0000 7fc15be00640 1 --2- 172.21.15.28:0/3121814700 >> v2:172.21.15.81:3300/0 conn(0x7fc1640c22e0 0x7fc16406d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:44.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.349+0000 7fc161400640 1 -- 172.21.15.28:0/3121814700 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc148003520 con 0x7fc1641561a0 2024-02-21T23:09:44.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.350+0000 7fc161400640 1 -- 172.21.15.28:0/3121814700 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1779119790 0 0) 0x7fc148003520 con 0x7fc1641561a0 2024-02-21T23:09:44.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.350+0000 7fc161400640 1 -- 172.21.15.28:0/3121814700 >> v2:172.21.15.81:3300/0 conn(0x7fc1640c22e0 msgr2=0x7fc16406d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:44.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.350+0000 7fc161400640 1 --2- 172.21.15.28:0/3121814700 >> v2:172.21.15.81:3300/0 conn(0x7fc1640c22e0 0x7fc16406d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:44.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.350+0000 7fc161400640 1 -- 172.21.15.28:0/3121814700 >> v1:172.21.15.81:6789/0 conn(0x7fc1640c5a70 legacy=0x7fc164155a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:44.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.350+0000 7fc161400640 1 -- 172.21.15.28:0/3121814700 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc164071f30 con 0x7fc1641561a0 2024-02-21T23:09:44.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.350+0000 7fc161400640 1 -- 172.21.15.28:0/3121814700 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fc15c0019f0 con 0x7fc1641561a0 2024-02-21T23:09:44.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.350+0000 7fc161400640 1 -- 172.21.15.28:0/3121814700 >> v1:172.21.15.28:6789/0 conn(0x7fc1641561a0 legacy=0x7fc16406d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:44.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.350+0000 7fc161400640 1 --2- 172.21.15.28:0/3121814700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1480039b0 0x7fc148003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:44.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.350+0000 7fc161400640 1 --2- 172.21.15.28:0/3121814700 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc148004800 0x7fc148004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:44.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.350+0000 7fc161400640 1 --2- 172.21.15.28:0/3121814700 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc1480052e0 0x7fc14800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:44.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.350+0000 7fc161400640 1 -- 172.21.15.28:0/3121814700 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc164072a10 con 0x7fc1480039b0 2024-02-21T23:09:44.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.350+0000 7fc161400640 1 -- 172.21.15.28:0/3121814700 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc14800c1b0 con 0x7fc1480052e0 2024-02-21T23:09:44.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.350+0000 7fc161400640 1 -- 172.21.15.28:0/3121814700 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc14800c3b0 con 0x7fc148004800 2024-02-21T23:09:44.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.350+0000 7fc161e00640 1 --2- 172.21.15.28:0/3121814700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1480039b0 0x7fc148003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:44.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.351+0000 7fc15be00640 1 --2- 172.21.15.28:0/3121814700 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc148004800 0x7fc148004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:44.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.351+0000 7fc162800640 1 --2- 172.21.15.28:0/3121814700 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc1480052e0 0x7fc14800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:44.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.351+0000 7fc161e00640 1 -- 172.21.15.28:0/3121814700 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc148004800 msgr2=0x7fc148004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:44.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.351+0000 7fc161e00640 1 --2- 172.21.15.28:0/3121814700 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc148004800 0x7fc148004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:44.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.351+0000 7fc161e00640 1 -- 172.21.15.28:0/3121814700 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc1480052e0 msgr2=0x7fc14800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:44.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.351+0000 7fc161e00640 1 --2- 172.21.15.28:0/3121814700 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc1480052e0 0x7fc14800b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:44.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.351+0000 7fc161e00640 1 -- 172.21.15.28:0/3121814700 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc148003520 con 0x7fc1480039b0 2024-02-21T23:09:44.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.351+0000 7fc162800640 1 -- 172.21.15.28:0/3121814700 reap_dead start 2024-02-21T23:09:44.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.352+0000 7fc161e00640 1 --2- 172.21.15.28:0/3121814700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1480039b0 0x7fc148003dd0 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7fc154001720 tx=0x7fc148005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b91537a35bbe9f8 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:44.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.352+0000 7fc161400640 1 -- 172.21.15.28:0/3121814700 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc154001370 con 0x7fc1480039b0 2024-02-21T23:09:44.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.352+0000 7fc161400640 1 -- 172.21.15.28:0/3121814700 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1540052d0 con 0x7fc1480039b0 2024-02-21T23:09:44.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.352+0000 7fc161400640 1 -- 172.21.15.28:0/3121814700 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc154003830 con 0x7fc1480039b0 2024-02-21T23:09:44.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.353+0000 7fc168a00640 1 -- 172.21.15.28:0/3121814700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1480039b0 msgr2=0x7fc148003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:44.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.353+0000 7fc168a00640 1 --2- 172.21.15.28:0/3121814700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1480039b0 0x7fc148003dd0 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7fc154001720 tx=0x7fc148005830 comp rx=0 tx=0).stop 2024-02-21T23:09:44.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.353+0000 7fc168a00640 1 -- 172.21.15.28:0/3121814700 shutdown_connections 2024-02-21T23:09:44.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.353+0000 7fc168a00640 1 --2- 172.21.15.28:0/3121814700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1480039b0 0x7fc148003dd0 unknown :-1 s=CLOSED pgs=1631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:44.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.353+0000 7fc168a00640 1 -- 172.21.15.28:0/3121814700 >> 172.21.15.28:0/3121814700 conn(0x7fc1640a85a0 msgr2=0x7fc1640a8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:44.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.353+0000 7fc168a00640 1 -- 172.21.15.28:0/3121814700 shutdown_connections 2024-02-21T23:09:44.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.353+0000 7fc168a00640 1 -- 172.21.15.28:0/3121814700 wait complete. 2024-02-21T23:09:44.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.355+0000 7fc168a00640 1 Processor -- start 2024-02-21T23:09:44.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.355+0000 7fc168a00640 1 -- start start 2024-02-21T23:09:44.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.356+0000 7fc168a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc1480039b0 0x7fc164152980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:44.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.356+0000 7fc168a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1640aaf10 0x7fc1641eb320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:44.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.356+0000 7fc168a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc16414cee0 0x7fc16414d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:44.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.356+0000 7fc168a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1640a6160 con 0x7fc1640aaf10 2024-02-21T23:09:44.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.356+0000 7fc168a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc164072580 con 0x7fc16414cee0 2024-02-21T23:09:44.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.356+0000 7fc168a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1640a62e0 con 0x7fc1480039b0 2024-02-21T23:09:44.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.356+0000 7fc15be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1640aaf10 0x7fc1641eb320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:44.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.356+0000 7fc15be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1640aaf10 0x7fc1641eb320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33010/0 (socket says 172.21.15.28:33010) 2024-02-21T23:09:44.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.356+0000 7fc15be00640 1 -- 172.21.15.28:0/4160650231 learned_addr learned my addr 172.21.15.28:0/4160650231 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:44.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.357+0000 7fc162800640 1 --2- 172.21.15.28:0/4160650231 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc16414cee0 0x7fc16414d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:44.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.357+0000 7fc161e00640 1 --2- 172.21.15.28:0/4160650231 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc1480039b0 0x7fc164152980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:44.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.357+0000 7fc15be00640 1 -- 172.21.15.28:0/4160650231 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc1480039b0 msgr2=0x7fc164152980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:44.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.357+0000 7fc15be00640 1 --2- 172.21.15.28:0/4160650231 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc1480039b0 0x7fc164152980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:44.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.357+0000 7fc15be00640 1 -- 172.21.15.28:0/4160650231 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc16414cee0 msgr2=0x7fc16414d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:44.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.357+0000 7fc15be00640 1 --2- 172.21.15.28:0/4160650231 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc16414cee0 0x7fc16414d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:44.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.357+0000 7fc15be00640 1 -- 172.21.15.28:0/4160650231 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc164152ec0 con 0x7fc1640aaf10 2024-02-21T23:09:44.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.357+0000 7fc15be00640 1 --2- 172.21.15.28:0/4160650231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1640aaf10 0x7fc1641eb320 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7fc14c00e540 tx=0x7fc14c00ea00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:44.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.358+0000 7fc15a000640 1 -- 172.21.15.28:0/4160650231 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc14c017070 con 0x7fc1640aaf10 2024-02-21T23:09:44.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.358+0000 7fc15a000640 1 -- 172.21.15.28:0/4160650231 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc14c013610 con 0x7fc1640aaf10 2024-02-21T23:09:44.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.358+0000 7fc15a000640 1 -- 172.21.15.28:0/4160650231 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc14c00f040 con 0x7fc1640aaf10 2024-02-21T23:09:44.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.358+0000 7fc168a00640 1 -- 172.21.15.28:0/4160650231 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc164153ca0 con 0x7fc1640aaf10 2024-02-21T23:09:44.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.358+0000 7fc168a00640 1 -- 172.21.15.28:0/4160650231 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1641eab60 con 0x7fc1640aaf10 2024-02-21T23:09:44.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.359+0000 7fc168a00640 1 -- 172.21.15.28:0/4160650231 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc164067f30 con 0x7fc1640aaf10 2024-02-21T23:09:44.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.360+0000 7fc15a000640 1 -- 172.21.15.28:0/4160650231 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc14c01a070 con 0x7fc1640aaf10 2024-02-21T23:09:44.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.361+0000 7fc15a000640 1 --2- 172.21.15.28:0/4160650231 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc14006b500 0x7fc14006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:44.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.361+0000 7fc15a000640 1 -- 172.21.15.28:0/4160650231 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5507+0+0 (secure 0 0 0) 0x7fc14c089ad0 con 0x7fc1640aaf10 2024-02-21T23:09:44.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.362+0000 7fc161e00640 1 --2- 172.21.15.28:0/4160650231 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc14006b500 0x7fc14006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:44.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.363+0000 7fc161e00640 1 --2- 172.21.15.28:0/4160650231 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc14006b500 0x7fc14006d9c0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fc154033000 tx=0x7fc154009030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:44.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.366+0000 7fc15a000640 1 -- 172.21.15.28:0/4160650231 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc164153ca0 con 0x7fc1640aaf10 2024-02-21T23:09:44.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.569+0000 7fc168a00640 1 -- 172.21.15.28:0/4160650231 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fc1640a5b10 con 0x7fc1640aaf10 2024-02-21T23:09:44.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.575+0000 7fc15a000640 1 -- 172.21.15.28:0/4160650231 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v50) v1 ==== 104+0+0 (secure 0 0 0) 0x7fc14c058a00 con 0x7fc1640aaf10 2024-02-21T23:09:44.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.578+0000 7fc168a00640 1 -- 172.21.15.28:0/4160650231 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc14006b500 msgr2=0x7fc14006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:44.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.578+0000 7fc168a00640 1 --2- 172.21.15.28:0/4160650231 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc14006b500 0x7fc14006d9c0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fc154033000 tx=0x7fc154009030 comp rx=0 tx=0).stop 2024-02-21T23:09:44.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.578+0000 7fc168a00640 1 -- 172.21.15.28:0/4160650231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1640aaf10 msgr2=0x7fc1641eb320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:44.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.578+0000 7fc168a00640 1 --2- 172.21.15.28:0/4160650231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1640aaf10 0x7fc1641eb320 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7fc14c00e540 tx=0x7fc14c00ea00 comp rx=0 tx=0).stop 2024-02-21T23:09:44.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.578+0000 7fc168a00640 1 -- 172.21.15.28:0/4160650231 shutdown_connections 2024-02-21T23:09:44.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.578+0000 7fc168a00640 1 --2- 172.21.15.28:0/4160650231 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc14006b500 0x7fc14006d9c0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:44.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.578+0000 7fc168a00640 1 --2- 172.21.15.28:0/4160650231 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc16414cee0 0x7fc16414d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:44.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.578+0000 7fc168a00640 1 --2- 172.21.15.28:0/4160650231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1640aaf10 0x7fc1641eb320 unknown :-1 s=CLOSED pgs=1632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:44.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.578+0000 7fc168a00640 1 --2- 172.21.15.28:0/4160650231 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc1480039b0 0x7fc164152980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:44.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.578+0000 7fc168a00640 1 -- 172.21.15.28:0/4160650231 >> 172.21.15.28:0/4160650231 conn(0x7fc1640a85a0 msgr2=0x7fc1640c5e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:44.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.579+0000 7fc168a00640 1 -- 172.21.15.28:0/4160650231 shutdown_connections 2024-02-21T23:09:44.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.579+0000 7fc168a00640 1 -- 172.21.15.28:0/4160650231 wait complete. 2024-02-21T23:09:44.585 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_unset' 2024-02-21T23:09:44.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.819+0000 7f7003200640 1 Processor -- start 2024-02-21T23:09:44.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.820+0000 7f7003200640 1 -- start start 2024-02-21T23:09:44.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.821+0000 7f7003200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f70041569b0 0x7f7004159db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:44.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.821+0000 7f7003200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f70040c0f00 0x7f700406dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:44.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.821+0000 7f7003200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f700406e530 0x7f700415c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:44.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.821+0000 7f7003200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70040c1fa0 con 0x7f700406e530 2024-02-21T23:09:44.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.821+0000 7f7003200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70040c1390 con 0x7f70041569b0 2024-02-21T23:09:44.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.821+0000 7f7003200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70040c1670 con 0x7f70040c0f00 2024-02-21T23:09:44.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.822+0000 7f7002800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f700406e530 0x7f700415c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:44.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.822+0000 7f7002800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f700406e530 0x7f700415c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33012/0 (socket says 172.21.15.28:33012) 2024-02-21T23:09:44.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.822+0000 7f7002800640 1 -- 172.21.15.28:0/1003616103 learned_addr learned my addr 172.21.15.28:0/1003616103 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:44.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.822+0000 7f7001e00640 1 --2- 172.21.15.28:0/1003616103 >> v2:172.21.15.81:3300/0 conn(0x7f70041569b0 0x7f7004159db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:44.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.822+0000 7f6ffbe00640 1 --2- 172.21.15.28:0/1003616103 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f70040c0f00 0x7f700406dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:44.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.822+0000 7f7002800640 1 -- 172.21.15.28:0/1003616103 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f70040c0f00 msgr2=0x7f700406dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:44.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.822+0000 7f7002800640 1 --2- 172.21.15.28:0/1003616103 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f70040c0f00 0x7f700406dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:44.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.822+0000 7f7002800640 1 -- 172.21.15.28:0/1003616103 >> v2:172.21.15.81:3300/0 conn(0x7f70041569b0 msgr2=0x7f7004159db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:44.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.822+0000 7f7002800640 1 --2- 172.21.15.28:0/1003616103 >> v2:172.21.15.81:3300/0 conn(0x7f70041569b0 0x7f7004159db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:44.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.822+0000 7f7002800640 1 -- 172.21.15.28:0/1003616103 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70040c1950 con 0x7f700406e530 2024-02-21T23:09:44.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.824+0000 7f7002800640 1 --2- 172.21.15.28:0/1003616103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f700406e530 0x7f700415c890 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f700406be20 tx=0x7f6ffc02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6efd5755b92416d server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:44.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.824+0000 7f7001400640 1 -- 172.21.15.28:0/1003616103 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ffc02c9e0 con 0x7f700406e530 2024-02-21T23:09:44.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.824+0000 7f7001400640 1 -- 172.21.15.28:0/1003616103 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ffc02cb80 con 0x7f700406e530 2024-02-21T23:09:44.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.824+0000 7f7001400640 1 -- 172.21.15.28:0/1003616103 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ffc0343f0 con 0x7f700406e530 2024-02-21T23:09:44.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.825+0000 7f7003200640 1 -- 172.21.15.28:0/1003616103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f700406e530 msgr2=0x7f700415c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:44.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.825+0000 7f7003200640 1 --2- 172.21.15.28:0/1003616103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f700406e530 0x7f700415c890 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f700406be20 tx=0x7f6ffc02be40 comp rx=0 tx=0).stop 2024-02-21T23:09:44.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.825+0000 7f7003200640 1 -- 172.21.15.28:0/1003616103 shutdown_connections 2024-02-21T23:09:44.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.825+0000 7f7003200640 1 --2- 172.21.15.28:0/1003616103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f700406e530 0x7f700415c890 unknown :-1 s=CLOSED pgs=1633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:44.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.825+0000 7f7003200640 1 --2- 172.21.15.28:0/1003616103 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f70040c0f00 0x7f700406dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:44.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.825+0000 7f7003200640 1 --2- 172.21.15.28:0/1003616103 >> v2:172.21.15.81:3300/0 conn(0x7f70041569b0 0x7f7004159db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:44.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.825+0000 7f7003200640 1 -- 172.21.15.28:0/1003616103 >> 172.21.15.28:0/1003616103 conn(0x7f700406b630 msgr2=0x7f70040c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:44.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.825+0000 7f7003200640 1 -- 172.21.15.28:0/1003616103 shutdown_connections 2024-02-21T23:09:44.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.825+0000 7f7003200640 1 -- 172.21.15.28:0/1003616103 wait complete. 2024-02-21T23:09:44.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.827+0000 7f7003200640 1 Processor -- start 2024-02-21T23:09:44.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.827+0000 7f7003200640 1 -- start start 2024-02-21T23:09:44.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.828+0000 7f7003200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f700406e530 0x7f70040a4b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:44.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.828+0000 7f7003200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70041569b0 0x7f7004153160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:44.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.828+0000 7f7003200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70041e43c0 0x7f7004158080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:44.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.828+0000 7f7003200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f700415cdd0 con 0x7f70041569b0 2024-02-21T23:09:44.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.828+0000 7f7003200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70040c2ba0 con 0x7f70041e43c0 2024-02-21T23:09:44.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.828+0000 7f7003200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f700415cf50 con 0x7f700406e530 2024-02-21T23:09:44.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.828+0000 7f6ffbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70041569b0 0x7f7004153160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:44.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.828+0000 7f6ffbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70041569b0 0x7f7004153160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33014/0 (socket says 172.21.15.28:33014) 2024-02-21T23:09:44.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.828+0000 7f6ffbe00640 1 -- 172.21.15.28:0/95712199 learned_addr learned my addr 172.21.15.28:0/95712199 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:44.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.828+0000 7f7001e00640 1 --2- 172.21.15.28:0/95712199 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f700406e530 0x7f70040a4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:44.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.829+0000 7f7002800640 1 --2- 172.21.15.28:0/95712199 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70041e43c0 0x7f7004158080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:44.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.829+0000 7f6ffbe00640 1 -- 172.21.15.28:0/95712199 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f700406e530 msgr2=0x7f70040a4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:44.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.829+0000 7f6ffbe00640 1 --2- 172.21.15.28:0/95712199 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f700406e530 0x7f70040a4b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:44.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.829+0000 7f6ffbe00640 1 -- 172.21.15.28:0/95712199 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70041e43c0 msgr2=0x7f7004158080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:44.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.829+0000 7f6ffbe00640 1 --2- 172.21.15.28:0/95712199 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70041e43c0 0x7f7004158080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:44.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.829+0000 7f6ffbe00640 1 -- 172.21.15.28:0/95712199 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70041f3230 con 0x7f70041569b0 2024-02-21T23:09:44.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.829+0000 7f6ffbe00640 1 --2- 172.21.15.28:0/95712199 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70041569b0 0x7f7004153160 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7f6fec00d830 tx=0x7f6fec00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:44.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.829+0000 7f6ffa000640 1 -- 172.21.15.28:0/95712199 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fec018020 con 0x7f70041569b0 2024-02-21T23:09:44.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.829+0000 7f6ffa000640 1 -- 172.21.15.28:0/95712199 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fec009640 con 0x7f70041569b0 2024-02-21T23:09:44.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.829+0000 7f6ffa000640 1 -- 172.21.15.28:0/95712199 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fec009a10 con 0x7f70041569b0 2024-02-21T23:09:44.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.829+0000 7f7003200640 1 -- 172.21.15.28:0/95712199 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70041f34d0 con 0x7f70041569b0 2024-02-21T23:09:44.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.830+0000 7f7003200640 1 -- 172.21.15.28:0/95712199 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70041e3840 con 0x7f70041569b0 2024-02-21T23:09:44.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.831+0000 7f7003200640 1 -- 172.21.15.28:0/95712199 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f700404f290 con 0x7f70041569b0 2024-02-21T23:09:44.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.832+0000 7f6ffa000640 1 -- 172.21.15.28:0/95712199 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6fec01b020 con 0x7f70041569b0 2024-02-21T23:09:44.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.833+0000 7f6ffa000640 1 --2- 172.21.15.28:0/95712199 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6fe006b4d0 0x7f6fe006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:44.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.833+0000 7f6ffa000640 1 -- 172.21.15.28:0/95712199 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f6fec08a4a0 con 0x7f70041569b0 2024-02-21T23:09:44.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.834+0000 7f7001e00640 1 --2- 172.21.15.28:0/95712199 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6fe006b4d0 0x7f6fe006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:44.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.835+0000 7f7001e00640 1 --2- 172.21.15.28:0/95712199 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6fe006b4d0 0x7f6fe006d990 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f6ff4006fd0 tx=0x7f6ff4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:44.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:44.838+0000 7f6ffa000640 1 -- 172.21.15.28:0/95712199 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f6fec022030 con 0x7f70041569b0 2024-02-21T23:09:45.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.113+0000 7f7003200640 1 -- 172.21.15.28:0/95712199 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_unset"]} v 0) v1 -- 0x7f70040abe60 con 0x7f70041569b0 2024-02-21T23:09:45.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.170+0000 7f6ffa000640 1 -- 172.21.15.28:0/95712199 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_unset"]}]=0 v413) v1 ==== 144+0+0 (secure 0 0 0) 0x7f6fec059470 con 0x7f70041569b0 2024-02-21T23:09:45.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.174+0000 7f7003200640 1 -- 172.21.15.28:0/95712199 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6fe006b4d0 msgr2=0x7f6fe006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:45.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.174+0000 7f7003200640 1 --2- 172.21.15.28:0/95712199 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6fe006b4d0 0x7f6fe006d990 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f6ff4006fd0 tx=0x7f6ff4008040 comp rx=0 tx=0).stop 2024-02-21T23:09:45.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.174+0000 7f7003200640 1 -- 172.21.15.28:0/95712199 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70041569b0 msgr2=0x7f7004153160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:45.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.174+0000 7f7003200640 1 --2- 172.21.15.28:0/95712199 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70041569b0 0x7f7004153160 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7f6fec00d830 tx=0x7f6fec00dcf0 comp rx=0 tx=0).stop 2024-02-21T23:09:45.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.174+0000 7f7003200640 1 -- 172.21.15.28:0/95712199 shutdown_connections 2024-02-21T23:09:45.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.174+0000 7f7003200640 1 --2- 172.21.15.28:0/95712199 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6fe006b4d0 0x7f6fe006d990 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:45.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.174+0000 7f7003200640 1 --2- 172.21.15.28:0/95712199 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70041e43c0 0x7f7004158080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:45.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.174+0000 7f7003200640 1 --2- 172.21.15.28:0/95712199 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70041569b0 0x7f7004153160 unknown :-1 s=CLOSED pgs=1634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:45.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.174+0000 7f7003200640 1 --2- 172.21.15.28:0/95712199 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f700406e530 0x7f70040a4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:45.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.174+0000 7f7003200640 1 -- 172.21.15.28:0/95712199 >> 172.21.15.28:0/95712199 conn(0x7f700406b630 msgr2=0x7f7004070ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:45.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.174+0000 7f7003200640 1 -- 172.21.15.28:0/95712199 shutdown_connections 2024-02-21T23:09:45.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.174+0000 7f7003200640 1 -- 172.21.15.28:0/95712199 wait complete. 2024-02-21T23:09:45.184 INFO:tasks.cephfs_test_runner:That unsetting mds_join_fs will cause failover if another high-affinity standby exists. ... ok 2024-02-21T23:09:45.184 INFO:tasks.cephfs_test_runner:Starting test: test_join_fs_vanilla (tasks.cephfs.test_failover.TestClusterAffinity) That a vanilla standby is preferred over others with mds_join_fs set to another fs. 2024-02-21T23:09:45.184 INFO:tasks.cephfs_test_runner:test_join_fs_vanilla (tasks.cephfs.test_failover.TestClusterAffinity) 2024-02-21T23:09:45.184 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:09:45.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.418+0000 7f001da00640 1 Processor -- start 2024-02-21T23:09:45.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.418+0000 7f001da00640 1 -- start start 2024-02-21T23:09:45.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.419+0000 7f001da00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0018071020 0x7f0018156550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:45.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.419+0000 7f001da00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0018071400 0x7f00180c7840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:45.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.419+0000 7f001da00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0018156a90 con 0x7f001806f410 2024-02-21T23:09:45.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.419+0000 7f001da00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00180c5130 con 0x7f0018071020 2024-02-21T23:09:45.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.419+0000 7f001da00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00180c4b70 con 0x7f0018071400 2024-02-21T23:09:45.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.420+0000 7f0016a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0018071020 0x7f0018156550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:45.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.420+0000 7f0016000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f001806f410 0x7f0018153900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36932/0 (socket says 172.21.15.28:36932) 2024-02-21T23:09:45.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.420+0000 7f0016000640 1 -- 172.21.15.28:0/1859241235 learned_addr learned my addr 172.21.15.28:0/1859241235 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:09:45.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.420+0000 7f0017400640 1 --2- 172.21.15.28:0/1859241235 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0018071400 0x7f00180c7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:45.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.420+0000 7f0016a00640 1 -- 172.21.15.28:0/1859241235 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0018071400 msgr2=0x7f00180c7840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:45.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.420+0000 7f0016a00640 1 --2- 172.21.15.28:0/1859241235 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0018071400 0x7f00180c7840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:45.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.420+0000 7f0016a00640 1 -- 172.21.15.28:0/1859241235 >> v1:172.21.15.28:6789/0 conn(0x7f001806f410 legacy=0x7f0018153900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:45.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.420+0000 7f0016a00640 1 -- 172.21.15.28:0/1859241235 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00180c4e50 con 0x7f0018071020 2024-02-21T23:09:45.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.420+0000 7f0015600640 1 -- 172.21.15.28:0/1859241235 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 276174051 0 0) 0x7f0018156a90 con 0x7f001806f410 2024-02-21T23:09:45.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.420+0000 7f0017400640 1 --2- 172.21.15.28:0/1859241235 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0018071400 0x7f00180c7840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:09:45.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.421+0000 7f0016a00640 1 --2- 172.21.15.28:0/1859241235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0018071020 0x7f0018156550 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f00000097a0 tx=0x7f000002b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c522a663b73973 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:45.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.422+0000 7f0015600640 1 -- 172.21.15.28:0/1859241235 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0000039070 con 0x7f0018071020 2024-02-21T23:09:45.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.422+0000 7f0015600640 1 -- 172.21.15.28:0/1859241235 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00000316f0 con 0x7f0018071020 2024-02-21T23:09:45.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.422+0000 7f0015600640 1 -- 172.21.15.28:0/1859241235 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0000031ac0 con 0x7f0018071020 2024-02-21T23:09:45.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.422+0000 7f001da00640 1 -- 172.21.15.28:0/1859241235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0018071020 msgr2=0x7f0018156550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:45.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.422+0000 7f001da00640 1 --2- 172.21.15.28:0/1859241235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0018071020 0x7f0018156550 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f00000097a0 tx=0x7f000002b790 comp rx=0 tx=0).stop 2024-02-21T23:09:45.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.422+0000 7f001da00640 1 -- 172.21.15.28:0/1859241235 shutdown_connections 2024-02-21T23:09:45.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.422+0000 7f001da00640 1 --2- 172.21.15.28:0/1859241235 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0018071400 0x7f00180c7840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:45.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.422+0000 7f001da00640 1 --2- 172.21.15.28:0/1859241235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0018071020 0x7f0018156550 unknown :-1 s=CLOSED pgs=1636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:45.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.422+0000 7f001da00640 1 -- 172.21.15.28:0/1859241235 >> 172.21.15.28:0/1859241235 conn(0x7f00180c8880 msgr2=0x7f00180c8c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:45.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.422+0000 7f001da00640 1 -- 172.21.15.28:0/1859241235 shutdown_connections 2024-02-21T23:09:45.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.423+0000 7f001da00640 1 -- 172.21.15.28:0/1859241235 wait complete. 2024-02-21T23:09:45.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.424+0000 7f001da00640 1 Processor -- start 2024-02-21T23:09:45.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.424+0000 7f001da00640 1 -- start start 2024-02-21T23:09:45.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.425+0000 7f001da00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f001806f410 0x7f00180ad760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:45.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.425+0000 7f001da00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0018071020 0x7f00181eacc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:45.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.425+0000 7f001da00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0018071400 0x7f001814b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:45.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.425+0000 7f001da00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00180c7d80 con 0x7f001806f410 2024-02-21T23:09:45.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.425+0000 7f001da00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00181533e0 con 0x7f0018071400 2024-02-21T23:09:45.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.425+0000 7f001da00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00180c7f00 con 0x7f0018071020 2024-02-21T23:09:45.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.426+0000 7f0016a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f001806f410 0x7f00180ad760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:45.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.426+0000 7f0016a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f001806f410 0x7f00180ad760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55886/0 (socket says 172.21.15.28:55886) 2024-02-21T23:09:45.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.426+0000 7f0016a00640 1 -- 172.21.15.28:0/3934172816 learned_addr learned my addr 172.21.15.28:0/3934172816 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:45.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.426+0000 7f0016000640 1 --2- 172.21.15.28:0/3934172816 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0018071020 0x7f00181eacc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:45.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.426+0000 7f0016a00640 1 -- 172.21.15.28:0/3934172816 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0018071020 msgr2=0x7f00181eacc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:45.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.426+0000 7f0016a00640 1 --2- 172.21.15.28:0/3934172816 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0018071020 0x7f00181eacc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:45.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.426+0000 7f0016a00640 1 -- 172.21.15.28:0/3934172816 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0018071400 msgr2=0x7f001814b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:45.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.426+0000 7f0016a00640 1 --2- 172.21.15.28:0/3934172816 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0018071400 0x7f001814b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:45.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.426+0000 7f0016a00640 1 -- 172.21.15.28:0/3934172816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f001806da40 con 0x7f001806f410 2024-02-21T23:09:45.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.426+0000 7f0016a00640 1 --2- 172.21.15.28:0/3934172816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f001806f410 0x7f00180ad760 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f0000000c00 tx=0x7f00000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:45.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.427+0000 7f000ea00640 1 -- 172.21.15.28:0/3934172816 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0000043070 con 0x7f001806f410 2024-02-21T23:09:45.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.427+0000 7f000ea00640 1 -- 172.21.15.28:0/3934172816 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00000346a0 con 0x7f001806f410 2024-02-21T23:09:45.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.427+0000 7f000ea00640 1 -- 172.21.15.28:0/3934172816 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0000039040 con 0x7f001806f410 2024-02-21T23:09:45.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.427+0000 7f001da00640 1 -- 172.21.15.28:0/3934172816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f001806dd20 con 0x7f001806f410 2024-02-21T23:09:45.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.427+0000 7f001da00640 1 -- 172.21.15.28:0/3934172816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00180adca0 con 0x7f001806f410 2024-02-21T23:09:45.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.428+0000 7f001da00640 1 -- 172.21.15.28:0/3934172816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00181ea5b0 con 0x7f001806f410 2024-02-21T23:09:45.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.429+0000 7f000ea00640 1 -- 172.21.15.28:0/3934172816 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0000046070 con 0x7f001806f410 2024-02-21T23:09:45.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.430+0000 7f000ea00640 1 --2- 172.21.15.28:0/3934172816 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7efff406b4d0 0x7efff406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:45.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.430+0000 7f000ea00640 1 -- 172.21.15.28:0/3934172816 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f00000abe20 con 0x7f001806f410 2024-02-21T23:09:45.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.431+0000 7f0016000640 1 --2- 172.21.15.28:0/3934172816 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7efff406b4d0 0x7efff406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:45.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.432+0000 7f0016000640 1 --2- 172.21.15.28:0/3934172816 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7efff406b4d0 0x7efff406d990 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f00181563c0 tx=0x7f0008009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:45.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.435+0000 7f000ea00640 1 -- 172.21.15.28:0/3934172816 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f000007adf0 con 0x7f001806f410 2024-02-21T23:09:45.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.641+0000 7f001da00640 1 -- 172.21.15.28:0/3934172816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0018068640 con 0x7f001806f410 2024-02-21T23:09:45.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.643+0000 7f000ea00640 1 -- 172.21.15.28:0/3934172816 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v143) v1 ==== 74+0+13503 (secure 0 0 0) 0x7f000007a6f0 con 0x7f001806f410 2024-02-21T23:09:45.631 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:09:45.631 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":143,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:09:43.161447+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.81:6837/2095908433":"2024-02-22T23:09:40.150762+0000","172.21.15.81:6836/4241906475":"2024-02-22T23:09:29.622886+0000","172.21.15.81:6834/4241906475":"2024-02-22T23:09:29.622886+0000","172.21.15.28:6835/4151254007":"2024-02-22T23:09:15.528381+0000","172.21.15.81:6835/2095908433":"2024-02-22T23:09:40.150762+0000","172.21.15.28:6834/4151254007":"2024-02-22T23:09:15.528381+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:09:45.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.646+0000 7f001da00640 1 -- 172.21.15.28:0/3934172816 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7efff406b4d0 msgr2=0x7efff406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:45.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.646+0000 7f001da00640 1 --2- 172.21.15.28:0/3934172816 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7efff406b4d0 0x7efff406d990 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f00181563c0 tx=0x7f0008009040 comp rx=0 tx=0).stop 2024-02-21T23:09:45.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.646+0000 7f001da00640 1 -- 172.21.15.28:0/3934172816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f001806f410 msgr2=0x7f00180ad760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:45.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.646+0000 7f001da00640 1 --2- 172.21.15.28:0/3934172816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f001806f410 0x7f00180ad760 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f0000000c00 tx=0x7f00000046c0 comp rx=0 tx=0).stop 2024-02-21T23:09:45.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.646+0000 7f001da00640 1 -- 172.21.15.28:0/3934172816 shutdown_connections 2024-02-21T23:09:45.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.646+0000 7f001da00640 1 --2- 172.21.15.28:0/3934172816 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7efff406b4d0 0x7efff406d990 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:45.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.646+0000 7f001da00640 1 --2- 172.21.15.28:0/3934172816 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0018071020 0x7f00181eacc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:45.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.646+0000 7f001da00640 1 --2- 172.21.15.28:0/3934172816 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0018071400 0x7f001814b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:45.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.646+0000 7f001da00640 1 --2- 172.21.15.28:0/3934172816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f001806f410 0x7f00180ad760 unknown :-1 s=CLOSED pgs=1637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:45.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.646+0000 7f001da00640 1 -- 172.21.15.28:0/3934172816 >> 172.21.15.28:0/3934172816 conn(0x7f00180c8880 msgr2=0x7f00180ac660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:45.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.646+0000 7f001da00640 1 -- 172.21.15.28:0/3934172816 shutdown_connections 2024-02-21T23:09:45.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.646+0000 7f001da00640 1 -- 172.21.15.28:0/3934172816 wait complete. 2024-02-21T23:09:45.652 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-21T23:02:30.728329+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-21T23:09:45.652 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-21T23:09:45.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.890+0000 7f29cbe00640 1 Processor -- start 2024-02-21T23:09:45.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.890+0000 7f29cbe00640 1 -- start start 2024-02-21T23:09:45.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.892+0000 7f29cbe00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f29cc13e0c0 0x7f29cc0c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:45.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.892+0000 7f29cbe00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f29cc06dbc0 0x7f29cc06c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:45.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.892+0000 7f29cbe00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29cc158550 con 0x7f29cc0c76f0 2024-02-21T23:09:45.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.892+0000 7f29cbe00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29cc0c89a0 con 0x7f29cc13e0c0 2024-02-21T23:09:45.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.892+0000 7f29cbe00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29cc1587d0 con 0x7f29cc06dbc0 2024-02-21T23:09:45.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.892+0000 7f29cb400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f29cc0c76f0 0x7f29cc06c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36934/0 (socket says 172.21.15.28:36934) 2024-02-21T23:09:45.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.892+0000 7f29cb400640 1 -- 172.21.15.28:0/356829241 learned_addr learned my addr 172.21.15.28:0/356829241 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:09:45.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.893+0000 7f29caa00640 1 --2- 172.21.15.28:0/356829241 >> v2:172.21.15.81:3300/0 conn(0x7f29cc13e0c0 0x7f29cc0c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:45.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.893+0000 7f29c2000640 1 --2- 172.21.15.28:0/356829241 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f29cc06dbc0 0x7f29cc06c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:45.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.893+0000 7f29ca000640 1 -- 172.21.15.28:0/356829241 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 141454058 0 0) 0x7f29c4001110 con 0x7f29cc0c76f0 2024-02-21T23:09:45.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.893+0000 7f29ca000640 1 -- 172.21.15.28:0/356829241 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29b0003520 con 0x7f29cc0c76f0 2024-02-21T23:09:45.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.893+0000 7f29ca000640 1 -- 172.21.15.28:0/356829241 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3897582925 0 0) 0x7f29b0003520 con 0x7f29cc0c76f0 2024-02-21T23:09:45.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.893+0000 7f29ca000640 1 -- 172.21.15.28:0/356829241 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f29cc06dbc0 msgr2=0x7f29cc06c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:45.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.893+0000 7f29ca000640 1 --2- 172.21.15.28:0/356829241 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f29cc06dbc0 0x7f29cc06c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:45.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.893+0000 7f29ca000640 1 -- 172.21.15.28:0/356829241 >> v2:172.21.15.81:3300/0 conn(0x7f29cc13e0c0 msgr2=0x7f29cc0c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:45.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.893+0000 7f29ca000640 1 --2- 172.21.15.28:0/356829241 >> v2:172.21.15.81:3300/0 conn(0x7f29cc13e0c0 0x7f29cc0c71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:45.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.893+0000 7f29ca000640 1 -- 172.21.15.28:0/356829241 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29cc158ab0 con 0x7f29cc0c76f0 2024-02-21T23:09:45.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.893+0000 7f29ca000640 1 -- 172.21.15.28:0/356829241 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f29c4001840 con 0x7f29cc0c76f0 2024-02-21T23:09:45.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.893+0000 7f29ca000640 1 -- 172.21.15.28:0/356829241 >> v1:172.21.15.28:6789/0 conn(0x7f29cc0c76f0 legacy=0x7f29cc06c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:45.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.893+0000 7f29ca000640 1 --2- 172.21.15.28:0/356829241 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29b00039b0 0x7f29b0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:45.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.893+0000 7f29ca000640 1 --2- 172.21.15.28:0/356829241 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f29cc06dbc0 0x7f29b0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:45.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.894+0000 7f29ca000640 1 --2- 172.21.15.28:0/356829241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29b0004d40 0x7f29b00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:45.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.894+0000 7f29ca000640 1 -- 172.21.15.28:0/356829241 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29cc1587d0 con 0x7f29b0004d40 2024-02-21T23:09:45.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.894+0000 7f29ca000640 1 -- 172.21.15.28:0/356829241 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29b000b9c0 con 0x7f29b00039b0 2024-02-21T23:09:45.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.894+0000 7f29ca000640 1 -- 172.21.15.28:0/356829241 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29b000bb40 con 0x7f29cc06dbc0 2024-02-21T23:09:45.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.894+0000 7f29cb400640 1 --2- 172.21.15.28:0/356829241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29b0004d40 0x7f29b00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:45.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.894+0000 7f29c2000640 1 --2- 172.21.15.28:0/356829241 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f29cc06dbc0 0x7f29b0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:45.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.894+0000 7f29caa00640 1 --2- 172.21.15.28:0/356829241 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29b00039b0 0x7f29b0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:45.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.894+0000 7f29cb400640 1 -- 172.21.15.28:0/356829241 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f29cc06dbc0 msgr2=0x7f29b0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:45.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.894+0000 7f29cb400640 1 --2- 172.21.15.28:0/356829241 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f29cc06dbc0 0x7f29b0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:45.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.894+0000 7f29cb400640 1 -- 172.21.15.28:0/356829241 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29b00039b0 msgr2=0x7f29b0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:45.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.894+0000 7f29cb400640 1 --2- 172.21.15.28:0/356829241 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29b00039b0 0x7f29b0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:45.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.894+0000 7f29cb400640 1 -- 172.21.15.28:0/356829241 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29b0003520 con 0x7f29b0004d40 2024-02-21T23:09:45.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.896+0000 7f29cb400640 1 --2- 172.21.15.28:0/356829241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29b0004d40 0x7f29b00051c0 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f29c400c3e0 tx=0x7f29b0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aafd0de46b44ae22 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:45.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.896+0000 7f29ca000640 1 -- 172.21.15.28:0/356829241 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29c400c0e0 con 0x7f29b0004d40 2024-02-21T23:09:45.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.896+0000 7f29ca000640 1 -- 172.21.15.28:0/356829241 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29c4030070 con 0x7f29b0004d40 2024-02-21T23:09:45.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.896+0000 7f29ca000640 1 -- 172.21.15.28:0/356829241 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29c40058d0 con 0x7f29b0004d40 2024-02-21T23:09:45.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.896+0000 7f29cbe00640 1 -- 172.21.15.28:0/356829241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29b0004d40 msgr2=0x7f29b00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:45.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.896+0000 7f29cbe00640 1 --2- 172.21.15.28:0/356829241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29b0004d40 0x7f29b00051c0 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f29c400c3e0 tx=0x7f29b0005830 comp rx=0 tx=0).stop 2024-02-21T23:09:45.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.896+0000 7f29cb400640 1 -- 172.21.15.28:0/356829241 reap_dead start 2024-02-21T23:09:45.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.896+0000 7f29cbe00640 1 -- 172.21.15.28:0/356829241 shutdown_connections 2024-02-21T23:09:45.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.897+0000 7f29cbe00640 1 -- 172.21.15.28:0/356829241 >> 172.21.15.28:0/356829241 conn(0x7f29cc073110 msgr2=0x7f29cc0ac490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:45.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.897+0000 7f29cbe00640 1 -- 172.21.15.28:0/356829241 shutdown_connections 2024-02-21T23:09:45.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.897+0000 7f29cbe00640 1 -- 172.21.15.28:0/356829241 wait complete. 2024-02-21T23:09:45.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.898+0000 7f29cbe00640 1 Processor -- start 2024-02-21T23:09:45.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.898+0000 7f29cbe00640 1 -- start start 2024-02-21T23:09:45.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.899+0000 7f29cbe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29b0004d40 0x7f29cc1570f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:45.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.899+0000 7f29cbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29cc155640 0x7f29cc0afd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:45.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.899+0000 7f29cbe00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f29cc155b90 0x7f29cc0adea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:45.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.899+0000 7f29cbe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29cc06ced0 con 0x7f29cc155640 2024-02-21T23:09:45.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.899+0000 7f29cbe00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29cc159070 con 0x7f29b0004d40 2024-02-21T23:09:45.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.899+0000 7f29cbe00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29cc13e520 con 0x7f29cc155b90 2024-02-21T23:09:45.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.900+0000 7f29c2000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29cc155640 0x7f29cc0afd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:45.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.900+0000 7f29c2000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29cc155640 0x7f29cc0afd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55898/0 (socket says 172.21.15.28:55898) 2024-02-21T23:09:45.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.900+0000 7f29c2000640 1 -- 172.21.15.28:0/3943612243 learned_addr learned my addr 172.21.15.28:0/3943612243 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:45.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.900+0000 7f29c2000640 1 -- 172.21.15.28:0/3943612243 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f29cc155b90 msgr2=0x7f29cc0adea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:45.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.900+0000 7f29caa00640 1 --2- 172.21.15.28:0/3943612243 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29b0004d40 0x7f29cc1570f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:45.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.900+0000 7f29cb400640 1 --2- 172.21.15.28:0/3943612243 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f29cc155b90 0x7f29cc0adea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:45.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.900+0000 7f29c2000640 1 --2- 172.21.15.28:0/3943612243 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f29cc155b90 0x7f29cc0adea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:45.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.900+0000 7f29c2000640 1 -- 172.21.15.28:0/3943612243 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29b0004d40 msgr2=0x7f29cc1570f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:45.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.900+0000 7f29c2000640 1 --2- 172.21.15.28:0/3943612243 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29b0004d40 0x7f29cc1570f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:45.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.900+0000 7f29c2000640 1 -- 172.21.15.28:0/3943612243 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29cc156590 con 0x7f29cc155640 2024-02-21T23:09:45.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.900+0000 7f29c2000640 1 --2- 172.21.15.28:0/3943612243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29cc155640 0x7f29cc0afd80 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7f29b4011600 tx=0x7f29b4011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:45.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.901+0000 7f29c3400640 1 -- 172.21.15.28:0/3943612243 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29b400f8c0 con 0x7f29cc155640 2024-02-21T23:09:45.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.901+0000 7f29c3400640 1 -- 172.21.15.28:0/3943612243 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29b400fa60 con 0x7f29cc155640 2024-02-21T23:09:45.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.901+0000 7f29cbe00640 1 -- 172.21.15.28:0/3943612243 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29cc06c490 con 0x7f29cc155640 2024-02-21T23:09:45.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.901+0000 7f29c3400640 1 -- 172.21.15.28:0/3943612243 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29b400fe10 con 0x7f29cc155640 2024-02-21T23:09:45.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.901+0000 7f29cbe00640 1 -- 172.21.15.28:0/3943612243 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29cc156830 con 0x7f29cc155640 2024-02-21T23:09:45.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.902+0000 7f29cbe00640 1 -- 172.21.15.28:0/3943612243 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29cc04f290 con 0x7f29cc155640 2024-02-21T23:09:45.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.903+0000 7f29c3400640 1 -- 172.21.15.28:0/3943612243 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f29b4015460 con 0x7f29cc155640 2024-02-21T23:09:45.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.904+0000 7f29c3400640 1 --2- 172.21.15.28:0/3943612243 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f29a806b4d0 0x7f29a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:45.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.904+0000 7f29c3400640 1 -- 172.21.15.28:0/3943612243 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f29b408aae0 con 0x7f29cc155640 2024-02-21T23:09:45.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.905+0000 7f29caa00640 1 --2- 172.21.15.28:0/3943612243 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f29a806b4d0 0x7f29a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:45.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.906+0000 7f29caa00640 1 --2- 172.21.15.28:0/3943612243 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f29a806b4d0 0x7f29a806d990 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f29bc007380 tx=0x7f29bc00b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:45.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:45.909+0000 7f29c3400640 1 -- 172.21.15.28:0/3943612243 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f29b4019030 con 0x7f29cc155640 2024-02-21T23:09:46.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.110+0000 7f29cbe00640 1 -- 172.21.15.28:0/3943612243 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f29cc152c60 con 0x7f29cc155640 2024-02-21T23:09:46.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.111+0000 7f29c3400640 1 -- 172.21.15.28:0/3943612243 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v143) v1 ==== 93+0+10 (secure 0 0 0) 0x7f29b4059a10 con 0x7f29cc155640 2024-02-21T23:09:46.099 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-02-21T23:09:46.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.114+0000 7f29cbe00640 1 -- 172.21.15.28:0/3943612243 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f29a806b4d0 msgr2=0x7f29a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:46.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.114+0000 7f29cbe00640 1 --2- 172.21.15.28:0/3943612243 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f29a806b4d0 0x7f29a806d990 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f29bc007380 tx=0x7f29bc00b040 comp rx=0 tx=0).stop 2024-02-21T23:09:46.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.114+0000 7f29cbe00640 1 -- 172.21.15.28:0/3943612243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29cc155640 msgr2=0x7f29cc0afd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:46.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.114+0000 7f29cbe00640 1 --2- 172.21.15.28:0/3943612243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29cc155640 0x7f29cc0afd80 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7f29b4011600 tx=0x7f29b4011ac0 comp rx=0 tx=0).stop 2024-02-21T23:09:46.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.114+0000 7f29cbe00640 1 -- 172.21.15.28:0/3943612243 shutdown_connections 2024-02-21T23:09:46.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.114+0000 7f29cbe00640 1 --2- 172.21.15.28:0/3943612243 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f29a806b4d0 0x7f29a806d990 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:46.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.114+0000 7f29cbe00640 1 --2- 172.21.15.28:0/3943612243 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f29cc155b90 0x7f29cc0adea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:46.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.114+0000 7f29cbe00640 1 --2- 172.21.15.28:0/3943612243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29cc155640 0x7f29cc0afd80 unknown :-1 s=CLOSED pgs=1640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:46.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.114+0000 7f29cbe00640 1 --2- 172.21.15.28:0/3943612243 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29b0004d40 0x7f29cc1570f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:46.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.114+0000 7f29cbe00640 1 -- 172.21.15.28:0/3943612243 >> 172.21.15.28:0/3943612243 conn(0x7f29cc073110 msgr2=0x7f29cc139ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:46.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.114+0000 7f29cbe00640 1 -- 172.21.15.28:0/3943612243 shutdown_connections 2024-02-21T23:09:46.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.114+0000 7f29cbe00640 1 -- 172.21.15.28:0/3943612243 wait complete. 2024-02-21T23:09:46.122 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_vanilla' 2024-02-21T23:09:46.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.341+0000 7f9436400640 1 Processor -- start 2024-02-21T23:09:46.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.341+0000 7f9436400640 1 -- start start 2024-02-21T23:09:46.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.342+0000 7f9436400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f94300aff90 0x7f94300c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:46.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.342+0000 7f9436400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f94300c5720 0x7f943015f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:46.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.342+0000 7f9436400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94300c74c0 con 0x7f943006dbf0 2024-02-21T23:09:46.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.342+0000 7f9436400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94300c7270 con 0x7f94300aff90 2024-02-21T23:09:46.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.342+0000 7f9436400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9430158630 con 0x7f94300c5720 2024-02-21T23:09:46.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.343+0000 7f9435000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f94300aff90 0x7f94300c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:46.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.343+0000 7f9435000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f94300aff90 0x7f94300c51e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55904/0 (socket says 172.21.15.28:55904) 2024-02-21T23:09:46.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.343+0000 7f9435000640 1 -- 172.21.15.28:0/2745113549 learned_addr learned my addr 172.21.15.28:0/2745113549 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:46.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.343+0000 7f9435a00640 1 --2- 172.21.15.28:0/2745113549 >> v2:172.21.15.81:3300/0 conn(0x7f94300c5720 0x7f943015f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:46.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.343+0000 7f9435000640 1 -- 172.21.15.28:0/2745113549 >> v2:172.21.15.81:3300/0 conn(0x7f94300c5720 msgr2=0x7f943015f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:46.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.343+0000 7f9435000640 1 --2- 172.21.15.28:0/2745113549 >> v2:172.21.15.81:3300/0 conn(0x7f94300c5720 0x7f943015f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:46.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.343+0000 7f9435000640 1 -- 172.21.15.28:0/2745113549 >> v1:172.21.15.81:6789/0 conn(0x7f943006dbf0 legacy=0x7f9430158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:46.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.343+0000 7f9435000640 1 -- 172.21.15.28:0/2745113549 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94300c77a0 con 0x7f94300aff90 2024-02-21T23:09:46.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.345+0000 7f9435000640 1 --2- 172.21.15.28:0/2745113549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94300aff90 0x7f94300c51e0 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7f94300c73f0 tx=0x7f942402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70edc6e151829dc4 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:46.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.345+0000 7f942f400640 1 -- 172.21.15.28:0/2745113549 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9424035020 con 0x7f94300aff90 2024-02-21T23:09:46.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.345+0000 7f942f400640 1 -- 172.21.15.28:0/2745113549 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9424031870 con 0x7f94300aff90 2024-02-21T23:09:46.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.345+0000 7f942f400640 1 -- 172.21.15.28:0/2745113549 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9424031c40 con 0x7f94300aff90 2024-02-21T23:09:46.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.346+0000 7f9436400640 1 -- 172.21.15.28:0/2745113549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94300aff90 msgr2=0x7f94300c51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:46.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.346+0000 7f9436400640 1 --2- 172.21.15.28:0/2745113549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94300aff90 0x7f94300c51e0 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7f94300c73f0 tx=0x7f942402b910 comp rx=0 tx=0).stop 2024-02-21T23:09:46.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.346+0000 7f9436400640 1 -- 172.21.15.28:0/2745113549 shutdown_connections 2024-02-21T23:09:46.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.346+0000 7f9436400640 1 --2- 172.21.15.28:0/2745113549 >> v2:172.21.15.81:3300/0 conn(0x7f94300c5720 0x7f943015f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:46.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.346+0000 7f9436400640 1 --2- 172.21.15.28:0/2745113549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94300aff90 0x7f94300c51e0 unknown :-1 s=CLOSED pgs=1641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:46.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.346+0000 7f9436400640 1 -- 172.21.15.28:0/2745113549 >> 172.21.15.28:0/2745113549 conn(0x7f94300ac370 msgr2=0x7f94300ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:46.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.346+0000 7f9436400640 1 -- 172.21.15.28:0/2745113549 shutdown_connections 2024-02-21T23:09:46.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.346+0000 7f9436400640 1 -- 172.21.15.28:0/2745113549 wait complete. 2024-02-21T23:09:46.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.348+0000 7f9436400640 1 Processor -- start 2024-02-21T23:09:46.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.348+0000 7f9436400640 1 -- start start 2024-02-21T23:09:46.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.349+0000 7f9436400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f943006dbf0 0x7f94300ae740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:46.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.349+0000 7f9436400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94300aff90 0x7f94300c31a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:46.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.349+0000 7f9436400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f94300c5720 0x7f94300c36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:46.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.349+0000 7f9436400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94300c8760 con 0x7f94300aff90 2024-02-21T23:09:46.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.349+0000 7f9436400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94300be0c0 con 0x7f94300c5720 2024-02-21T23:09:46.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.349+0000 7f9436400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94300c88e0 con 0x7f943006dbf0 2024-02-21T23:09:46.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.349+0000 7f942fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94300aff90 0x7f94300c31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:46.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.350+0000 7f942fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94300aff90 0x7f94300c31a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55914/0 (socket says 172.21.15.28:55914) 2024-02-21T23:09:46.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.350+0000 7f942fe00640 1 -- 172.21.15.28:0/3622465615 learned_addr learned my addr 172.21.15.28:0/3622465615 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:46.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.350+0000 7f9435a00640 1 --2- 172.21.15.28:0/3622465615 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f94300c5720 0x7f94300c36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:46.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.350+0000 7f9435000640 1 --2- 172.21.15.28:0/3622465615 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f943006dbf0 0x7f94300ae740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:46.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.350+0000 7f942fe00640 1 -- 172.21.15.28:0/3622465615 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f943006dbf0 msgr2=0x7f94300ae740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:46.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.350+0000 7f942fe00640 1 --2- 172.21.15.28:0/3622465615 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f943006dbf0 0x7f94300ae740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:46.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.350+0000 7f942fe00640 1 -- 172.21.15.28:0/3622465615 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f94300c5720 msgr2=0x7f94300c36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:46.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.350+0000 7f942fe00640 1 --2- 172.21.15.28:0/3622465615 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f94300c5720 0x7f94300c36e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:46.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.350+0000 7f942fe00640 1 -- 172.21.15.28:0/3622465615 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94301ea7b0 con 0x7f94300aff90 2024-02-21T23:09:46.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.350+0000 7f942fe00640 1 --2- 172.21.15.28:0/3622465615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94300aff90 0x7f94300c31a0 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f942000c430 tx=0x7f942000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:46.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.351+0000 7f942cc00640 1 -- 172.21.15.28:0/3622465615 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9420011790 con 0x7f94300aff90 2024-02-21T23:09:46.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.351+0000 7f942cc00640 1 -- 172.21.15.28:0/3622465615 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9420017070 con 0x7f94300aff90 2024-02-21T23:09:46.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.351+0000 7f942cc00640 1 -- 172.21.15.28:0/3622465615 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9420011b70 con 0x7f94300aff90 2024-02-21T23:09:46.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.351+0000 7f9436400640 1 -- 172.21.15.28:0/3622465615 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94301eaa90 con 0x7f94300aff90 2024-02-21T23:09:46.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.351+0000 7f9436400640 1 -- 172.21.15.28:0/3622465615 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94301eb050 con 0x7f94300aff90 2024-02-21T23:09:46.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.352+0000 7f9436400640 1 -- 172.21.15.28:0/3622465615 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f943004f290 con 0x7f94300aff90 2024-02-21T23:09:46.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.353+0000 7f942cc00640 1 -- 172.21.15.28:0/3622465615 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9420011d10 con 0x7f94300aff90 2024-02-21T23:09:46.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.354+0000 7f942cc00640 1 --2- 172.21.15.28:0/3622465615 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f941406b4d0 0x7f941406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:46.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.354+0000 7f942cc00640 1 -- 172.21.15.28:0/3622465615 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f9420089ff0 con 0x7f94300aff90 2024-02-21T23:09:46.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.355+0000 7f9435000640 1 --2- 172.21.15.28:0/3622465615 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f941406b4d0 0x7f941406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:46.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.356+0000 7f9435000640 1 --2- 172.21.15.28:0/3622465615 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f941406b4d0 0x7f941406d990 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f9424036000 tx=0x7f942403c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:46.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.359+0000 7f942cc00640 1 -- 172.21.15.28:0/3622465615 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f9420058f40 con 0x7f94300aff90 2024-02-21T23:09:46.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:46.631+0000 7f9436400640 1 -- 172.21.15.28:0/3622465615 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_vanilla"]} v 0) v1 -- 0x7f943006d9d0 con 0x7f94300aff90 2024-02-21T23:09:47.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.176+0000 7f942cc00640 1 -- 172.21.15.28:0/3622465615 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_vanilla"]}]=0 v415) v1 ==== 149+0+0 (secure 0 0 0) 0x7f9420058840 con 0x7f94300aff90 2024-02-21T23:09:47.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.179+0000 7f9436400640 1 -- 172.21.15.28:0/3622465615 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f941406b4d0 msgr2=0x7f941406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:47.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.179+0000 7f9436400640 1 --2- 172.21.15.28:0/3622465615 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f941406b4d0 0x7f941406d990 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f9424036000 tx=0x7f942403c040 comp rx=0 tx=0).stop 2024-02-21T23:09:47.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.179+0000 7f9436400640 1 -- 172.21.15.28:0/3622465615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94300aff90 msgr2=0x7f94300c31a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:47.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.179+0000 7f9436400640 1 --2- 172.21.15.28:0/3622465615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94300aff90 0x7f94300c31a0 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f942000c430 tx=0x7f942000c8f0 comp rx=0 tx=0).stop 2024-02-21T23:09:47.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.179+0000 7f9436400640 1 -- 172.21.15.28:0/3622465615 shutdown_connections 2024-02-21T23:09:47.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.179+0000 7f9436400640 1 --2- 172.21.15.28:0/3622465615 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f941406b4d0 0x7f941406d990 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:47.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.179+0000 7f9436400640 1 --2- 172.21.15.28:0/3622465615 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f94300c5720 0x7f94300c36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:47.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.179+0000 7f9436400640 1 --2- 172.21.15.28:0/3622465615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94300aff90 0x7f94300c31a0 unknown :-1 s=CLOSED pgs=1642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:47.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.179+0000 7f9436400640 1 --2- 172.21.15.28:0/3622465615 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f943006dbf0 0x7f94300ae740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:47.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.179+0000 7f9436400640 1 -- 172.21.15.28:0/3622465615 >> 172.21.15.28:0/3622465615 conn(0x7f94300ac370 msgr2=0x7f94300ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:47.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.179+0000 7f9436400640 1 -- 172.21.15.28:0/3622465615 shutdown_connections 2024-02-21T23:09:47.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.179+0000 7f9436400640 1 -- 172.21.15.28:0/3622465615 wait complete. 2024-02-21T23:09:47.191 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-02-21T23:09:47.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.432+0000 7faf9d200640 1 Processor -- start 2024-02-21T23:09:47.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.432+0000 7faf9d200640 1 -- start start 2024-02-21T23:09:47.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.433+0000 7faf9d200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7faf980c5a70 0x7faf98155a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:47.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.434+0000 7faf9d200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7faf980c22e0 0x7faf98155fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:47.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.434+0000 7faf9d200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf98071970 con 0x7faf980c2b30 2024-02-21T23:09:47.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.434+0000 7faf9d200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf98072a10 con 0x7faf980c22e0 2024-02-21T23:09:47.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.434+0000 7faf9d200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf98072580 con 0x7faf980c5a70 2024-02-21T23:09:47.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.434+0000 7faf95400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7faf980c22e0 0x7faf98155fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:47.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.434+0000 7faf96800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7faf980c2b30 0x7faf9806d120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36944/0 (socket says 172.21.15.28:36944) 2024-02-21T23:09:47.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.434+0000 7faf96800640 1 -- 172.21.15.28:0/3777804443 learned_addr learned my addr 172.21.15.28:0/3777804443 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:09:47.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.434+0000 7faf95e00640 1 --2- 172.21.15.28:0/3777804443 >> v2:172.21.15.81:3300/0 conn(0x7faf980c5a70 0x7faf98155a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:47.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.435+0000 7faf95400640 1 -- 172.21.15.28:0/3777804443 >> v2:172.21.15.81:3300/0 conn(0x7faf980c5a70 msgr2=0x7faf98155a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:47.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.435+0000 7faf95400640 1 --2- 172.21.15.28:0/3777804443 >> v2:172.21.15.81:3300/0 conn(0x7faf980c5a70 0x7faf98155a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:47.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.435+0000 7faf95400640 1 -- 172.21.15.28:0/3777804443 >> v1:172.21.15.28:6789/0 conn(0x7faf980c2b30 legacy=0x7faf9806d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:47.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.435+0000 7faf95400640 1 -- 172.21.15.28:0/3777804443 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf98071c50 con 0x7faf980c22e0 2024-02-21T23:09:47.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.436+0000 7faf95400640 1 --2- 172.21.15.28:0/3777804443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf980c22e0 0x7faf98155fd0 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7faf980c7810 tx=0x7faf8802e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3485401d5e813ab server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:47.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.436+0000 7faf94a00640 1 -- 172.21.15.28:0/3777804443 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf8803b070 con 0x7faf980c22e0 2024-02-21T23:09:47.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.436+0000 7faf94a00640 1 -- 172.21.15.28:0/3777804443 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf8802edf0 con 0x7faf980c22e0 2024-02-21T23:09:47.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.436+0000 7faf94a00640 1 -- 172.21.15.28:0/3777804443 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf88036660 con 0x7faf980c22e0 2024-02-21T23:09:47.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.437+0000 7faf9d200640 1 -- 172.21.15.28:0/3777804443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf980c22e0 msgr2=0x7faf98155fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:47.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.437+0000 7faf9d200640 1 --2- 172.21.15.28:0/3777804443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf980c22e0 0x7faf98155fd0 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7faf980c7810 tx=0x7faf8802e040 comp rx=0 tx=0).stop 2024-02-21T23:09:47.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.437+0000 7faf9d200640 1 -- 172.21.15.28:0/3777804443 shutdown_connections 2024-02-21T23:09:47.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.437+0000 7faf9d200640 1 --2- 172.21.15.28:0/3777804443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf980c22e0 0x7faf98155fd0 unknown :-1 s=CLOSED pgs=1644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:47.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.437+0000 7faf9d200640 1 --2- 172.21.15.28:0/3777804443 >> v2:172.21.15.81:3300/0 conn(0x7faf980c5a70 0x7faf98155a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:47.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.437+0000 7faf9d200640 1 -- 172.21.15.28:0/3777804443 >> 172.21.15.28:0/3777804443 conn(0x7faf980a85a0 msgr2=0x7faf980a8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:47.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.437+0000 7faf9d200640 1 -- 172.21.15.28:0/3777804443 shutdown_connections 2024-02-21T23:09:47.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.437+0000 7faf9d200640 1 -- 172.21.15.28:0/3777804443 wait complete. 2024-02-21T23:09:47.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.439+0000 7faf9d200640 1 Processor -- start 2024-02-21T23:09:47.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.439+0000 7faf9d200640 1 -- start start 2024-02-21T23:09:47.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.440+0000 7faf9d200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faf980c22e0 0x7faf981edf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:47.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.440+0000 7faf9d200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf980c2b30 0x7faf981e0d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:47.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.440+0000 7faf9d200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf980c5a70 0x7faf981ddbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:47.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.440+0000 7faf9d200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf980a6160 con 0x7faf980c5a70 2024-02-21T23:09:47.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.440+0000 7faf9d200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf98072210 con 0x7faf980c2b30 2024-02-21T23:09:47.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.440+0000 7faf9d200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf980a62e0 con 0x7faf980c22e0 2024-02-21T23:09:47.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.440+0000 7faf96800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf980c5a70 0x7faf981ddbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:47.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.440+0000 7faf96800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf980c5a70 0x7faf981ddbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55930/0 (socket says 172.21.15.28:55930) 2024-02-21T23:09:47.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.440+0000 7faf96800640 1 -- 172.21.15.28:0/163266902 learned_addr learned my addr 172.21.15.28:0/163266902 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:47.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.440+0000 7faf95400640 1 --2- 172.21.15.28:0/163266902 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf980c2b30 0x7faf981e0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:47.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.440+0000 7faf95e00640 1 --2- 172.21.15.28:0/163266902 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faf980c22e0 0x7faf981edf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:47.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.441+0000 7faf96800640 1 -- 172.21.15.28:0/163266902 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faf980c22e0 msgr2=0x7faf981edf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:47.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.441+0000 7faf96800640 1 --2- 172.21.15.28:0/163266902 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faf980c22e0 0x7faf981edf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:47.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.441+0000 7faf96800640 1 -- 172.21.15.28:0/163266902 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf980c2b30 msgr2=0x7faf981e0d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:47.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.441+0000 7faf96800640 1 --2- 172.21.15.28:0/163266902 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf980c2b30 0x7faf981e0d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:47.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.441+0000 7faf96800640 1 -- 172.21.15.28:0/163266902 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf98155150 con 0x7faf980c5a70 2024-02-21T23:09:47.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.441+0000 7faf95400640 1 --2- 172.21.15.28:0/163266902 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf980c2b30 0x7faf981e0d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:09:47.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.441+0000 7faf96800640 1 --2- 172.21.15.28:0/163266902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf980c5a70 0x7faf981ddbd0 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7faf9000c430 tx=0x7faf9000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:47.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.441+0000 7faf8e000640 1 -- 172.21.15.28:0/163266902 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf90017070 con 0x7faf980c5a70 2024-02-21T23:09:47.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.441+0000 7faf8e000640 1 -- 172.21.15.28:0/163266902 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf90005ce0 con 0x7faf980c5a70 2024-02-21T23:09:47.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.441+0000 7faf8e000640 1 -- 172.21.15.28:0/163266902 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf900072d0 con 0x7faf980c5a70 2024-02-21T23:09:47.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.441+0000 7faf9d200640 1 -- 172.21.15.28:0/163266902 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf98155430 con 0x7faf980c5a70 2024-02-21T23:09:47.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.442+0000 7faf9d200640 1 -- 172.21.15.28:0/163266902 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf981eea70 con 0x7faf980c5a70 2024-02-21T23:09:47.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.443+0000 7faf9d200640 1 -- 172.21.15.28:0/163266902 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf98067f30 con 0x7faf980c5a70 2024-02-21T23:09:47.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.444+0000 7faf8e000640 1 -- 172.21.15.28:0/163266902 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faf9001a070 con 0x7faf980c5a70 2024-02-21T23:09:47.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.445+0000 7faf8e000640 1 --2- 172.21.15.28:0/163266902 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7faf7406b4d0 0x7faf7406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:47.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.445+0000 7faf8e000640 1 -- 172.21.15.28:0/163266902 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5507+0+0 (secure 0 0 0) 0x7faf900893b0 con 0x7faf980c5a70 2024-02-21T23:09:47.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.445+0000 7faf95e00640 1 --2- 172.21.15.28:0/163266902 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7faf7406b4d0 0x7faf7406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:47.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.447+0000 7faf95e00640 1 --2- 172.21.15.28:0/163266902 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7faf7406b4d0 0x7faf7406d990 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7faf80006fd0 tx=0x7faf80008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:47.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.449+0000 7faf8e000640 1 -- 172.21.15.28:0/163266902 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7faf90057c90 con 0x7faf980c5a70 2024-02-21T23:09:47.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.655+0000 7faf9d200640 1 -- 172.21.15.28:0/163266902 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7faf980c16b0 con 0x7faf980c5a70 2024-02-21T23:09:47.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.662+0000 7faf8e000640 1 -- 172.21.15.28:0/163266902 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v51)=0 v51) v1 ==== 122+0+0 (secure 0 0 0) 0x7faf9005bd30 con 0x7faf980c5a70 2024-02-21T23:09:47.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.665+0000 7faf9d200640 1 -- 172.21.15.28:0/163266902 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7faf7406b4d0 msgr2=0x7faf7406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:47.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.665+0000 7faf9d200640 1 --2- 172.21.15.28:0/163266902 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7faf7406b4d0 0x7faf7406d990 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7faf80006fd0 tx=0x7faf80008040 comp rx=0 tx=0).stop 2024-02-21T23:09:47.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.665+0000 7faf9d200640 1 -- 172.21.15.28:0/163266902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf980c5a70 msgr2=0x7faf981ddbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:47.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.665+0000 7faf9d200640 1 --2- 172.21.15.28:0/163266902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf980c5a70 0x7faf981ddbd0 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7faf9000c430 tx=0x7faf9000c8f0 comp rx=0 tx=0).stop 2024-02-21T23:09:47.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.666+0000 7faf9d200640 1 -- 172.21.15.28:0/163266902 shutdown_connections 2024-02-21T23:09:47.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.666+0000 7faf9d200640 1 --2- 172.21.15.28:0/163266902 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7faf7406b4d0 0x7faf7406d990 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:47.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.666+0000 7faf9d200640 1 --2- 172.21.15.28:0/163266902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf980c5a70 0x7faf981ddbd0 unknown :-1 s=CLOSED pgs=1645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:47.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.666+0000 7faf9d200640 1 --2- 172.21.15.28:0/163266902 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf980c2b30 0x7faf981e0d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:47.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.666+0000 7faf9d200640 1 --2- 172.21.15.28:0/163266902 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faf980c22e0 0x7faf981edf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:47.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.666+0000 7faf9d200640 1 -- 172.21.15.28:0/163266902 >> 172.21.15.28:0/163266902 conn(0x7faf980a85a0 msgr2=0x7faf980ab620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:47.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.666+0000 7faf9d200640 1 -- 172.21.15.28:0/163266902 shutdown_connections 2024-02-21T23:09:47.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:47.666+0000 7faf9d200640 1 -- 172.21.15.28:0/163266902 wait complete. 2024-02-21T23:09:47.672 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-02-21T23:09:47.672 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-21T23:09:47.793 DEBUG:teuthology.orchestra.run.smithi081:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-21T23:09:47.915 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-02-21T23:09:47.916 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-21T23:09:47.916 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-21T23:09:47.976 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054608,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-02-21T23:09:47.976 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-02-21T23:09:47.976 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-21T23:09:48.033 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053833,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498812k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249404k,nr_inodes=812351,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498812k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-02-21T23:09:48.033 INFO:tasks.cephfs.cephfs_test_case:[, ] 2024-02-21T23:09:48.033 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:09:48.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.275+0000 7fa186c00640 1 Processor -- start 2024-02-21T23:09:48.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.275+0000 7fa186c00640 1 -- start start 2024-02-21T23:09:48.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.276+0000 7fa186c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa1800c8560 0x7fa1800c7ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:48.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.276+0000 7fa186c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fa1800c8940 0x7fa1800c54a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:48.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.276+0000 7fa186c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1800c2c20 con 0x7fa1800c64d0 2024-02-21T23:09:48.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.276+0000 7fa186c00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1800c8dd0 con 0x7fa1800c8940 2024-02-21T23:09:48.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.276+0000 7fa186c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1800c6910 con 0x7fa1800c8560 2024-02-21T23:09:48.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.277+0000 7fa184e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa1800c64d0 0x7fa1800c4d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36960/0 (socket says 172.21.15.28:36960) 2024-02-21T23:09:48.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.277+0000 7fa184e00640 1 -- 172.21.15.28:0/1587371985 learned_addr learned my addr 172.21.15.28:0/1587371985 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:09:48.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.277+0000 7fa185800640 1 --2- 172.21.15.28:0/1587371985 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa1800c8560 0x7fa1800c7ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:48.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.277+0000 7fa186200640 1 --2- 172.21.15.28:0/1587371985 >> v2:172.21.15.81:3300/0 conn(0x7fa1800c8940 0x7fa1800c54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:48.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.277+0000 7fa17fe00640 1 -- 172.21.15.28:0/1587371985 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1064809947 0 0) 0x7fa1800c2c20 con 0x7fa1800c64d0 2024-02-21T23:09:48.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.277+0000 7fa17fe00640 1 -- 172.21.15.28:0/1587371985 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa168003520 con 0x7fa1800c64d0 2024-02-21T23:09:48.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.278+0000 7fa185800640 1 -- 172.21.15.28:0/1587371985 >> v2:172.21.15.81:3300/0 conn(0x7fa1800c8940 msgr2=0x7fa1800c54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:48.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.278+0000 7fa185800640 1 --2- 172.21.15.28:0/1587371985 >> v2:172.21.15.81:3300/0 conn(0x7fa1800c8940 0x7fa1800c54a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:48.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.278+0000 7fa17fe00640 1 -- 172.21.15.28:0/1587371985 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3635081693 0 0) 0x7fa168003520 con 0x7fa1800c64d0 2024-02-21T23:09:48.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.278+0000 7fa185800640 1 -- 172.21.15.28:0/1587371985 >> v1:172.21.15.28:6789/0 conn(0x7fa1800c64d0 legacy=0x7fa1800c4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:48.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.278+0000 7fa185800640 1 -- 172.21.15.28:0/1587371985 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa180155100 con 0x7fa1800c8560 2024-02-21T23:09:48.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.279+0000 7fa185800640 1 --2- 172.21.15.28:0/1587371985 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa1800c8560 0x7fa1800c7ef0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa174009d90 tx=0x7fa17402bd70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a89e832bdb09b36 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:48.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.279+0000 7fa17fe00640 1 -- 172.21.15.28:0/1587371985 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa174039070 con 0x7fa1800c8560 2024-02-21T23:09:48.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.280+0000 7fa17fe00640 1 -- 172.21.15.28:0/1587371985 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa17402c8a0 con 0x7fa1800c8560 2024-02-21T23:09:48.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.280+0000 7fa17fe00640 1 -- 172.21.15.28:0/1587371985 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa16c001670 con 0x7fa1800c8560 2024-02-21T23:09:48.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.280+0000 7fa186c00640 1 -- 172.21.15.28:0/1587371985 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa1800c8560 msgr2=0x7fa1800c7ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:48.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.280+0000 7fa186c00640 1 --2- 172.21.15.28:0/1587371985 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa1800c8560 0x7fa1800c7ef0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa174009d90 tx=0x7fa17402bd70 comp rx=0 tx=0).stop 2024-02-21T23:09:48.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.280+0000 7fa186c00640 1 -- 172.21.15.28:0/1587371985 shutdown_connections 2024-02-21T23:09:48.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.280+0000 7fa186c00640 1 --2- 172.21.15.28:0/1587371985 >> v2:172.21.15.81:3300/0 conn(0x7fa1800c8940 0x7fa1800c54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:48.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.280+0000 7fa186c00640 1 --2- 172.21.15.28:0/1587371985 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa1800c8560 0x7fa1800c7ef0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:48.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.280+0000 7fa186c00640 1 -- 172.21.15.28:0/1587371985 >> 172.21.15.28:0/1587371985 conn(0x7fa18006ae70 msgr2=0x7fa18006b280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:48.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.280+0000 7fa186c00640 1 -- 172.21.15.28:0/1587371985 shutdown_connections 2024-02-21T23:09:48.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.280+0000 7fa186c00640 1 -- 172.21.15.28:0/1587371985 wait complete. 2024-02-21T23:09:48.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.282+0000 7fa186c00640 1 Processor -- start 2024-02-21T23:09:48.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.282+0000 7fa186c00640 1 -- start start 2024-02-21T23:09:48.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.283+0000 7fa186c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1800c64d0 0x7fa1800a9580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:48.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.283+0000 7fa186c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1800c8560 0x7fa180156db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:48.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.283+0000 7fa186c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa1800c8940 0x7fa1801ee240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:48.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.283+0000 7fa186c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1801559a0 con 0x7fa1800c8560 2024-02-21T23:09:48.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.283+0000 7fa186c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1801556c0 con 0x7fa1800c64d0 2024-02-21T23:09:48.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.283+0000 7fa186c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1800c59e0 con 0x7fa1800c8940 2024-02-21T23:09:48.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.283+0000 7fa184e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1800c8560 0x7fa180156db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:48.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.283+0000 7fa184e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1800c8560 0x7fa180156db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55942/0 (socket says 172.21.15.28:55942) 2024-02-21T23:09:48.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.283+0000 7fa184e00640 1 -- 172.21.15.28:0/3941093389 learned_addr learned my addr 172.21.15.28:0/3941093389 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:48.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.284+0000 7fa185800640 1 --2- 172.21.15.28:0/3941093389 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1800c64d0 0x7fa1800a9580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:48.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.284+0000 7fa186200640 1 --2- 172.21.15.28:0/3941093389 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa1800c8940 0x7fa1801ee240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:48.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.284+0000 7fa184e00640 1 -- 172.21.15.28:0/3941093389 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa1800c8940 msgr2=0x7fa1801ee240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:48.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.284+0000 7fa184e00640 1 --2- 172.21.15.28:0/3941093389 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa1800c8940 0x7fa1801ee240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:48.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.284+0000 7fa184e00640 1 -- 172.21.15.28:0/3941093389 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1800c64d0 msgr2=0x7fa1800a9580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:48.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.284+0000 7fa184e00640 1 --2- 172.21.15.28:0/3941093389 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1800c64d0 0x7fa1800a9580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:48.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.284+0000 7fa184e00640 1 -- 172.21.15.28:0/3941093389 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1800a9ac0 con 0x7fa1800c8560 2024-02-21T23:09:48.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.284+0000 7fa184e00640 1 --2- 172.21.15.28:0/3941093389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1800c8560 0x7fa180156db0 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7fa16c00c730 tx=0x7fa16c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:48.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.284+0000 7fa17d600640 1 -- 172.21.15.28:0/3941093389 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa16c017070 con 0x7fa1800c8560 2024-02-21T23:09:48.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.284+0000 7fa17d600640 1 -- 172.21.15.28:0/3941093389 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa16c005ad0 con 0x7fa1800c8560 2024-02-21T23:09:48.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.285+0000 7fa186c00640 1 -- 172.21.15.28:0/3941093389 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1801eec80 con 0x7fa1800c8560 2024-02-21T23:09:48.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.285+0000 7fa17d600640 1 -- 172.21.15.28:0/3941093389 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa16c007070 con 0x7fa1800c8560 2024-02-21T23:09:48.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.285+0000 7fa186c00640 1 -- 172.21.15.28:0/3941093389 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1801572f0 con 0x7fa1800c8560 2024-02-21T23:09:48.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.286+0000 7fa186c00640 1 -- 172.21.15.28:0/3941093389 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa180154780 con 0x7fa1800c8560 2024-02-21T23:09:48.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.287+0000 7fa17d600640 1 -- 172.21.15.28:0/3941093389 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa16c01a070 con 0x7fa1800c8560 2024-02-21T23:09:48.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.288+0000 7fa17d600640 1 --2- 172.21.15.28:0/3941093389 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa16006b4d0 0x7fa16006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:48.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.288+0000 7fa17d600640 1 -- 172.21.15.28:0/3941093389 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5507+0+0 (secure 0 0 0) 0x7fa16c089760 con 0x7fa1800c8560 2024-02-21T23:09:48.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.289+0000 7fa185800640 1 --2- 172.21.15.28:0/3941093389 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa16006b4d0 0x7fa16006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:48.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.290+0000 7fa185800640 1 --2- 172.21.15.28:0/3941093389 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa16006b4d0 0x7fa16006d990 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fa174036000 tx=0x7fa174030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:48.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.293+0000 7fa17d600640 1 -- 172.21.15.28:0/3941093389 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fa180154780 con 0x7fa1800c8560 2024-02-21T23:09:48.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.533+0000 7fa186c00640 1 -- 172.21.15.28:0/3941093389 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa18004f290 con 0x7fa1800c8560 2024-02-21T23:09:48.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.534+0000 7fa17d600640 1 -- 172.21.15.28:0/3941093389 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 151 v151) v1 ==== 95+0+3701 (secure 0 0 0) 0x7fa16c058610 con 0x7fa1800c8560 2024-02-21T23:09:48.522 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:09:48.522 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":151,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6344,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3620235199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":3620235199},{"type":"v1","addr":"172.21.15.81:6837","nonce":3620235199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":151},{"gid":6934,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/878375431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":878375431},{"type":"v1","addr":"172.21.15.81:6836","nonce":878375431}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":147},{"gid":8370,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/3934548735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3934548735},{"type":"v1","addr":"172.21.15.28:6833","nonce":3934548735}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":133},{"gid":8556,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2100132822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2100132822},{"type":"v1","addr":"172.21.15.28:6835","nonce":2100132822}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":142}],"filesystems":[]} 2024-02-21T23:09:48.522 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 151 2024-02-21T23:09:48.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.537+0000 7fa186c00640 1 -- 172.21.15.28:0/3941093389 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa16006b4d0 msgr2=0x7fa16006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:48.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.537+0000 7fa186c00640 1 --2- 172.21.15.28:0/3941093389 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa16006b4d0 0x7fa16006d990 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fa174036000 tx=0x7fa174030000 comp rx=0 tx=0).stop 2024-02-21T23:09:48.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.537+0000 7fa186c00640 1 -- 172.21.15.28:0/3941093389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1800c8560 msgr2=0x7fa180156db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:48.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.537+0000 7fa186c00640 1 --2- 172.21.15.28:0/3941093389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1800c8560 0x7fa180156db0 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7fa16c00c730 tx=0x7fa16c00cbf0 comp rx=0 tx=0).stop 2024-02-21T23:09:48.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.537+0000 7fa186c00640 1 -- 172.21.15.28:0/3941093389 shutdown_connections 2024-02-21T23:09:48.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.537+0000 7fa186c00640 1 --2- 172.21.15.28:0/3941093389 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa16006b4d0 0x7fa16006d990 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:48.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.537+0000 7fa186c00640 1 --2- 172.21.15.28:0/3941093389 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa1800c8940 0x7fa1801ee240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:48.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.537+0000 7fa186c00640 1 --2- 172.21.15.28:0/3941093389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1800c8560 0x7fa180156db0 unknown :-1 s=CLOSED pgs=1647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:48.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.537+0000 7fa186c00640 1 --2- 172.21.15.28:0/3941093389 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa1800c64d0 0x7fa1800a9580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:48.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.537+0000 7fa186c00640 1 -- 172.21.15.28:0/3941093389 >> 172.21.15.28:0/3941093389 conn(0x7fa18006ae70 msgr2=0x7fa180153f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:48.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.537+0000 7fa186c00640 1 -- 172.21.15.28:0/3941093389 shutdown_connections 2024-02-21T23:09:48.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.537+0000 7fa186c00640 1 -- 172.21.15.28:0/3941093389 wait complete. 2024-02-21T23:09:48.547 INFO:tasks.ceph.mds.a:Restarting daemon 2024-02-21T23:09:48.547 INFO:tasks.ceph.mds.a:Stopping old one... 2024-02-21T23:09:48.547 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-02-21T23:09:48.547 INFO:teuthology.orchestra.run:waiting for 300 2024-02-21T23:09:48.548 INFO:tasks.ceph.mds.c:Restarting daemon 2024-02-21T23:09:48.548 INFO:tasks.ceph.mds.c:Stopping old one... 2024-02-21T23:09:48.548 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-02-21T23:09:48.549 INFO:teuthology.orchestra.run:waiting for 300 2024-02-21T23:09:48.549 INFO:tasks.ceph.mds.b:Restarting daemon 2024-02-21T23:09:48.549 INFO:tasks.ceph.mds.b:Stopping old one... 2024-02-21T23:09:48.549 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-02-21T23:09:48.550 INFO:teuthology.orchestra.run:waiting for 300 2024-02-21T23:09:48.550 INFO:tasks.ceph.mds.d:Restarting daemon 2024-02-21T23:09:48.550 INFO:tasks.ceph.mds.d:Stopping old one... 2024-02-21T23:09:48.550 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-02-21T23:09:48.551 INFO:teuthology.orchestra.run:waiting for 300 2024-02-21T23:09:48.568 INFO:tasks.ceph.mds.a:Stopped 2024-02-21T23:09:48.568 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-02-21T23:09:48.570 INFO:tasks.ceph.mds.b:Stopped 2024-02-21T23:09:48.571 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-02-21T23:09:48.574 INFO:tasks.ceph.mds.c:Stopped 2024-02-21T23:09:48.574 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-02-21T23:09:48.575 INFO:tasks.ceph.mds.d:Stopped 2024-02-21T23:09:48.575 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-02-21T23:09:48.620 INFO:tasks.ceph.mds.a:Started 2024-02-21T23:09:48.621 DEBUG:teuthology.parallel:result is None 2024-02-21T23:09:48.622 INFO:tasks.ceph.mds.b:Started 2024-02-21T23:09:48.622 DEBUG:teuthology.parallel:result is None 2024-02-21T23:09:48.627 INFO:tasks.ceph.mds.c:Started 2024-02-21T23:09:48.627 DEBUG:teuthology.parallel:result is None 2024-02-21T23:09:48.628 INFO:tasks.ceph.mds.d:Started 2024-02-21T23:09:48.628 DEBUG:teuthology.parallel:result is None 2024-02-21T23:09:48.628 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-02-21T23:09:48.778 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-02-21T23:09:48.780 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-02-21T23:09:48.782 INFO:tasks.ceph.mds.b.smithi081.stdout:starting mds.b at 2024-02-21T23:09:48.790 INFO:tasks.ceph.mds.d.smithi081.stdout:starting mds.d at 2024-02-21T23:09:48.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.829+0000 7f3c37a00640 1 Processor -- start 2024-02-21T23:09:48.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.829+0000 7f3c37a00640 1 -- start start 2024-02-21T23:09:48.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.830+0000 7f3c37a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3c3006ae70 0x7f3c30159050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:48.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.830+0000 7f3c37a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f3c30072400 0x7f3c30072880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:48.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.830+0000 7f3c37a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c3006b6b0 con 0x7f3c3006b250 2024-02-21T23:09:48.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.830+0000 7f3c37a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c30155cc0 con 0x7f3c30072400 2024-02-21T23:09:48.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.830+0000 7f3c37a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c30155390 con 0x7f3c3006ae70 2024-02-21T23:09:48.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.831+0000 7f3c35000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3c3006ae70 0x7f3c30159050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:48.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.831+0000 7f3c35000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3c3006ae70 0x7f3c30159050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3301/0 says I am v2:172.21.15.28:43384/0 (socket says 172.21.15.28:43384) 2024-02-21T23:09:48.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.831+0000 7f3c35a00640 1 --2- 172.21.15.28:0/515692334 >> v2:172.21.15.81:3300/0 conn(0x7f3c30072400 0x7f3c30072880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:48.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.831+0000 7f3c35000640 1 -- 172.21.15.28:0/515692334 learned_addr learned my addr 172.21.15.28:0/515692334 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:48.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.831+0000 7f3c2f400640 1 -- 172.21.15.28:0/515692334 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1410292485 0 0) 0x7f3c3006b6b0 con 0x7f3c3006b250 2024-02-21T23:09:48.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.831+0000 7f3c35000640 1 -- 172.21.15.28:0/515692334 >> v2:172.21.15.81:3300/0 conn(0x7f3c30072400 msgr2=0x7f3c30072880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:48.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.831+0000 7f3c35000640 1 --2- 172.21.15.28:0/515692334 >> v2:172.21.15.81:3300/0 conn(0x7f3c30072400 0x7f3c30072880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:48.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.831+0000 7f3c35000640 1 -- 172.21.15.28:0/515692334 >> v1:172.21.15.81:6789/0 conn(0x7f3c3006b250 legacy=0x7f3c300c17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:48.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.831+0000 7f3c35000640 1 -- 172.21.15.28:0/515692334 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c30155670 con 0x7f3c3006ae70 2024-02-21T23:09:48.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.831+0000 7f3c35a00640 1 --2- 172.21.15.28:0/515692334 >> v2:172.21.15.81:3300/0 conn(0x7f3c30072400 0x7f3c30072880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-21T23:09:48.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.833+0000 7f3c35000640 1 --2- 172.21.15.28:0/515692334 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3c3006ae70 0x7f3c30159050 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f3c24009870 tx=0x7f3c2402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8efeee57c9aa4ba9 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:48.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.833+0000 7f3c2f400640 1 -- 172.21.15.28:0/515692334 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c24035020 con 0x7f3c3006ae70 2024-02-21T23:09:48.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.833+0000 7f3c2f400640 1 -- 172.21.15.28:0/515692334 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c24031640 con 0x7f3c3006ae70 2024-02-21T23:09:48.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.833+0000 7f3c2f400640 1 -- 172.21.15.28:0/515692334 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c2402d040 con 0x7f3c3006ae70 2024-02-21T23:09:48.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.834+0000 7f3c37a00640 1 -- 172.21.15.28:0/515692334 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3c3006ae70 msgr2=0x7f3c30159050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:48.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.834+0000 7f3c37a00640 1 --2- 172.21.15.28:0/515692334 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3c3006ae70 0x7f3c30159050 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f3c24009870 tx=0x7f3c2402b8f0 comp rx=0 tx=0).stop 2024-02-21T23:09:48.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.834+0000 7f3c37a00640 1 -- 172.21.15.28:0/515692334 shutdown_connections 2024-02-21T23:09:48.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.834+0000 7f3c37a00640 1 --2- 172.21.15.28:0/515692334 >> v2:172.21.15.81:3300/0 conn(0x7f3c30072400 0x7f3c30072880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:48.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.834+0000 7f3c37a00640 1 --2- 172.21.15.28:0/515692334 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3c3006ae70 0x7f3c30159050 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:48.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.834+0000 7f3c37a00640 1 -- 172.21.15.28:0/515692334 >> 172.21.15.28:0/515692334 conn(0x7f3c300a9690 msgr2=0x7f3c300a9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:48.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.834+0000 7f3c37a00640 1 -- 172.21.15.28:0/515692334 shutdown_connections 2024-02-21T23:09:48.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.834+0000 7f3c37a00640 1 -- 172.21.15.28:0/515692334 wait complete. 2024-02-21T23:09:48.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.835+0000 7f3c37a00640 1 Processor -- start 2024-02-21T23:09:48.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.836+0000 7f3c37a00640 1 -- start start 2024-02-21T23:09:48.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.836+0000 7f3c37a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c3006ae70 0x7f3c3014fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:48.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.836+0000 7f3c37a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c3006b250 0x7f3c30150250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:48.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.836+0000 7f3c37a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3c30072400 0x7f3c3014c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:48.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.836+0000 7f3c37a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c30152960 con 0x7f3c3006ae70 2024-02-21T23:09:48.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.836+0000 7f3c37a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c30155040 con 0x7f3c3006b250 2024-02-21T23:09:48.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.836+0000 7f3c37a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c30152ae0 con 0x7f3c30072400 2024-02-21T23:09:48.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.837+0000 7f3c35000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c3006ae70 0x7f3c3014fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:48.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.837+0000 7f3c35000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c3006ae70 0x7f3c3014fd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55968/0 (socket says 172.21.15.28:55968) 2024-02-21T23:09:48.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.837+0000 7f3c35000640 1 -- 172.21.15.28:0/3547528650 learned_addr learned my addr 172.21.15.28:0/3547528650 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:48.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.837+0000 7f3c35a00640 1 --2- 172.21.15.28:0/3547528650 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3c30072400 0x7f3c3014c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:48.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.837+0000 7f3c2fe00640 1 --2- 172.21.15.28:0/3547528650 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c3006b250 0x7f3c30150250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:48.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.837+0000 7f3c35000640 1 -- 172.21.15.28:0/3547528650 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3c30072400 msgr2=0x7f3c3014c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:48.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.837+0000 7f3c35000640 1 --2- 172.21.15.28:0/3547528650 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3c30072400 0x7f3c3014c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:48.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.837+0000 7f3c35000640 1 -- 172.21.15.28:0/3547528650 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c3006b250 msgr2=0x7f3c30150250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:48.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.837+0000 7f3c35000640 1 --2- 172.21.15.28:0/3547528650 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c3006b250 0x7f3c30150250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:48.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.837+0000 7f3c35000640 1 -- 172.21.15.28:0/3547528650 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c30150790 con 0x7f3c3006ae70 2024-02-21T23:09:48.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.837+0000 7f3c2fe00640 1 --2- 172.21.15.28:0/3547528650 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c3006b250 0x7f3c30150250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:09:48.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.837+0000 7f3c35a00640 1 --2- 172.21.15.28:0/3547528650 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3c30072400 0x7f3c3014c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:09:48.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.837+0000 7f3c35000640 1 --2- 172.21.15.28:0/3547528650 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c3006ae70 0x7f3c3014fd10 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7f3c24034040 tx=0x7f3c240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:48.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.838+0000 7f3c2cc00640 1 -- 172.21.15.28:0/3547528650 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c24040070 con 0x7f3c3006ae70 2024-02-21T23:09:48.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.838+0000 7f3c2cc00640 1 -- 172.21.15.28:0/3547528650 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c240399f0 con 0x7f3c3006ae70 2024-02-21T23:09:48.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.838+0000 7f3c37a00640 1 -- 172.21.15.28:0/3547528650 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c3006ebc0 con 0x7f3c3006ae70 2024-02-21T23:09:48.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.838+0000 7f3c2cc00640 1 -- 172.21.15.28:0/3547528650 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c24035040 con 0x7f3c3006ae70 2024-02-21T23:09:48.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.838+0000 7f3c37a00640 1 -- 172.21.15.28:0/3547528650 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c301579c0 con 0x7f3c3006ae70 2024-02-21T23:09:48.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.839+0000 7f3c37a00640 1 -- 172.21.15.28:0/3547528650 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c300a9470 con 0x7f3c3006ae70 2024-02-21T23:09:48.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.840+0000 7f3c2cc00640 1 -- 172.21.15.28:0/3547528650 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3c24038020 con 0x7f3c3006ae70 2024-02-21T23:09:48.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.841+0000 7f3c2cc00640 1 --2- 172.21.15.28:0/3547528650 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3c1006b500 0x7f3c1006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:48.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.841+0000 7f3c2cc00640 1 -- 172.21.15.28:0/3547528650 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5507+0+0 (secure 0 0 0) 0x7f3c240abae0 con 0x7f3c3006ae70 2024-02-21T23:09:48.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.841+0000 7f3c2fe00640 1 --2- 172.21.15.28:0/3547528650 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3c1006b500 0x7f3c1006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:48.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.842+0000 7f3c2fe00640 1 --2- 172.21.15.28:0/3547528650 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3c1006b500 0x7f3c1006d9c0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f3c3014e7f0 tx=0x7f3c1c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:48.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:48.845+0000 7f3c2cc00640 1 -- 172.21.15.28:0/3547528650 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3c30150790 con 0x7f3c3006ae70 2024-02-21T23:09:49.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.056+0000 7f3c37a00640 1 -- 172.21.15.28:0/3547528650 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f3c3004f370 con 0x7f3c3006ae70 2024-02-21T23:09:49.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.200+0000 7f3c2cc00640 1 -- 172.21.15.28:0/3547528650 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v145) v1 ==== 97+0+0 (secure 0 0 0) 0x7f3c2407a980 con 0x7f3c3006ae70 2024-02-21T23:09:49.189 INFO:teuthology.orchestra.run.smithi028.stderr: removed all blocklist entries 2024-02-21T23:09:49.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.204+0000 7f3c37a00640 1 -- 172.21.15.28:0/3547528650 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3c1006b500 msgr2=0x7f3c1006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:49.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.204+0000 7f3c37a00640 1 --2- 172.21.15.28:0/3547528650 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3c1006b500 0x7f3c1006d9c0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f3c3014e7f0 tx=0x7f3c1c009040 comp rx=0 tx=0).stop 2024-02-21T23:09:49.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.204+0000 7f3c37a00640 1 -- 172.21.15.28:0/3547528650 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c3006ae70 msgr2=0x7f3c3014fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:49.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.204+0000 7f3c37a00640 1 --2- 172.21.15.28:0/3547528650 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c3006ae70 0x7f3c3014fd10 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7f3c24034040 tx=0x7f3c240046c0 comp rx=0 tx=0).stop 2024-02-21T23:09:49.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.204+0000 7f3c37a00640 1 -- 172.21.15.28:0/3547528650 shutdown_connections 2024-02-21T23:09:49.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.204+0000 7f3c37a00640 1 --2- 172.21.15.28:0/3547528650 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3c1006b500 0x7f3c1006d9c0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:49.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.204+0000 7f3c37a00640 1 --2- 172.21.15.28:0/3547528650 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3c30072400 0x7f3c3014c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:49.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.204+0000 7f3c37a00640 1 --2- 172.21.15.28:0/3547528650 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c3006b250 0x7f3c30150250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:49.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.205+0000 7f3c37a00640 1 --2- 172.21.15.28:0/3547528650 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c3006ae70 0x7f3c3014fd10 unknown :-1 s=CLOSED pgs=1654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:49.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.205+0000 7f3c37a00640 1 -- 172.21.15.28:0/3547528650 >> 172.21.15.28:0/3547528650 conn(0x7f3c300a9690 msgr2=0x7f3c30154820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:49.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.205+0000 7f3c37a00640 1 -- 172.21.15.28:0/3547528650 shutdown_connections 2024-02-21T23:09:49.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.205+0000 7f3c37a00640 1 -- 172.21.15.28:0/3547528650 wait complete. 2024-02-21T23:09:49.212 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-02-21T23:09:49.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.437+0000 7fc3bea00640 1 Processor -- start 2024-02-21T23:09:49.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.438+0000 7fc3bea00640 1 -- start start 2024-02-21T23:09:49.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.439+0000 7fc3bea00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc3b8072430 0x7fc3b80c3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:49.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.439+0000 7fc3bea00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fc3b8155860 0x7fc3b8155cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:49.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.439+0000 7fc3bea00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3b80c6560 con 0x7fc3b80c3280 2024-02-21T23:09:49.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.439+0000 7fc3bea00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3b80c5cc0 con 0x7fc3b8072430 2024-02-21T23:09:49.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.439+0000 7fc3bea00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3b80c5fa0 con 0x7fc3b8155860 2024-02-21T23:09:49.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.439+0000 7fc3afe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc3b8072430 0x7fc3b80c3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:49.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.439+0000 7fc3afe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc3b8072430 0x7fc3b80c3660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55980/0 (socket says 172.21.15.28:55980) 2024-02-21T23:09:49.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.439+0000 7fc3afe00640 1 -- 172.21.15.28:0/2773123414 learned_addr learned my addr 172.21.15.28:0/2773123414 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:49.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.439+0000 7fc3bca00640 1 --2- 172.21.15.28:0/2773123414 >> v2:172.21.15.81:3300/0 conn(0x7fc3b8155860 0x7fc3b8155cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:49.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.440+0000 7fc3afe00640 1 -- 172.21.15.28:0/2773123414 >> v2:172.21.15.81:3300/0 conn(0x7fc3b8155860 msgr2=0x7fc3b8155cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:49.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.440+0000 7fc3afe00640 1 --2- 172.21.15.28:0/2773123414 >> v2:172.21.15.81:3300/0 conn(0x7fc3b8155860 0x7fc3b8155cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:49.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.440+0000 7fc3afe00640 1 -- 172.21.15.28:0/2773123414 >> v1:172.21.15.81:6789/0 conn(0x7fc3b80c3280 legacy=0x7fc3b81526e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:49.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.440+0000 7fc3afe00640 1 -- 172.21.15.28:0/2773123414 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3b80c6280 con 0x7fc3b8072430 2024-02-21T23:09:49.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.440+0000 7fc3bca00640 1 --2- 172.21.15.28:0/2773123414 >> v2:172.21.15.81:3300/0 conn(0x7fc3b8155860 0x7fc3b8155cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:09:49.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.441+0000 7fc3afe00640 1 --2- 172.21.15.28:0/2773123414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3b8072430 0x7fc3b80c3660 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7fc3b80c3190 tx=0x7fc3a002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e16796c02a198cd8 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:49.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.441+0000 7fc3b7400640 1 -- 172.21.15.28:0/2773123414 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a002e070 con 0x7fc3b8072430 2024-02-21T23:09:49.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.441+0000 7fc3b7400640 1 -- 172.21.15.28:0/2773123414 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3a0037070 con 0x7fc3b8072430 2024-02-21T23:09:49.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.441+0000 7fc3b7400640 1 -- 172.21.15.28:0/2773123414 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a0031a00 con 0x7fc3b8072430 2024-02-21T23:09:49.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.442+0000 7fc3bea00640 1 -- 172.21.15.28:0/2773123414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3b8072430 msgr2=0x7fc3b80c3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:49.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.442+0000 7fc3bea00640 1 --2- 172.21.15.28:0/2773123414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3b8072430 0x7fc3b80c3660 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7fc3b80c3190 tx=0x7fc3a002b880 comp rx=0 tx=0).stop 2024-02-21T23:09:49.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.442+0000 7fc3bea00640 1 -- 172.21.15.28:0/2773123414 shutdown_connections 2024-02-21T23:09:49.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.442+0000 7fc3bea00640 1 --2- 172.21.15.28:0/2773123414 >> v2:172.21.15.81:3300/0 conn(0x7fc3b8155860 0x7fc3b8155cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:49.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.442+0000 7fc3bea00640 1 --2- 172.21.15.28:0/2773123414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3b8072430 0x7fc3b80c3660 unknown :-1 s=CLOSED pgs=1655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:49.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.442+0000 7fc3bea00640 1 -- 172.21.15.28:0/2773123414 >> 172.21.15.28:0/2773123414 conn(0x7fc3b80c67f0 msgr2=0x7fc3b80c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:49.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.442+0000 7fc3bea00640 1 -- 172.21.15.28:0/2773123414 shutdown_connections 2024-02-21T23:09:49.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.442+0000 7fc3bea00640 1 -- 172.21.15.28:0/2773123414 wait complete. 2024-02-21T23:09:49.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.444+0000 7fc3bea00640 1 Processor -- start 2024-02-21T23:09:49.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.444+0000 7fc3bea00640 1 -- start start 2024-02-21T23:09:49.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.445+0000 7fc3bea00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc3b8072430 0x7fc3b80ac3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:49.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.445+0000 7fc3bea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3b80c3280 0x7fc3b814bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:49.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.445+0000 7fc3bea00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc3b8155860 0x7fc3b80a7d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:49.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.445+0000 7fc3bea00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b80c5aa0 con 0x7fc3b80c3280 2024-02-21T23:09:49.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.445+0000 7fc3bea00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b80c5920 con 0x7fc3b8072430 2024-02-21T23:09:49.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.445+0000 7fc3bea00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b80733e0 con 0x7fc3b8155860 2024-02-21T23:09:49.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.445+0000 7fc3afe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3b80c3280 0x7fc3b814bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:49.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.445+0000 7fc3afe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3b80c3280 0x7fc3b814bbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55984/0 (socket says 172.21.15.28:55984) 2024-02-21T23:09:49.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.445+0000 7fc3afe00640 1 -- 172.21.15.28:0/404647316 learned_addr learned my addr 172.21.15.28:0/404647316 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:49.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.445+0000 7fc3b7e00640 1 --2- 172.21.15.28:0/404647316 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc3b8072430 0x7fc3b80ac3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:49.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.446+0000 7fc3bca00640 1 --2- 172.21.15.28:0/404647316 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc3b8155860 0x7fc3b80a7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:49.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.446+0000 7fc3afe00640 1 -- 172.21.15.28:0/404647316 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc3b8155860 msgr2=0x7fc3b80a7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:49.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.446+0000 7fc3afe00640 1 --2- 172.21.15.28:0/404647316 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc3b8155860 0x7fc3b80a7d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:49.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.446+0000 7fc3afe00640 1 -- 172.21.15.28:0/404647316 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc3b8072430 msgr2=0x7fc3b80ac3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:49.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.446+0000 7fc3afe00640 1 --2- 172.21.15.28:0/404647316 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc3b8072430 0x7fc3b80ac3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:49.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.446+0000 7fc3afe00640 1 -- 172.21.15.28:0/404647316 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3b8071e70 con 0x7fc3b80c3280 2024-02-21T23:09:49.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.446+0000 7fc3afe00640 1 --2- 172.21.15.28:0/404647316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3b80c3280 0x7fc3b814bbd0 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7fc3a002b8c0 tx=0x7fc3a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:49.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.446+0000 7fc3b4c00640 1 -- 172.21.15.28:0/404647316 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a0032880 con 0x7fc3b80c3280 2024-02-21T23:09:49.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.446+0000 7fc3b4c00640 1 -- 172.21.15.28:0/404647316 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3a0032a20 con 0x7fc3b80c3280 2024-02-21T23:09:49.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.446+0000 7fc3bea00640 1 -- 172.21.15.28:0/404647316 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3b81ee070 con 0x7fc3b80c3280 2024-02-21T23:09:49.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.446+0000 7fc3b4c00640 1 -- 172.21.15.28:0/404647316 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a0032e10 con 0x7fc3b80c3280 2024-02-21T23:09:49.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.447+0000 7fc3bea00640 1 -- 172.21.15.28:0/404647316 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3b81ee630 con 0x7fc3b80c3280 2024-02-21T23:09:49.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.447+0000 7fc3bea00640 1 -- 172.21.15.28:0/404647316 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3b8067f30 con 0x7fc3b80c3280 2024-02-21T23:09:49.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.448+0000 7fc3b4c00640 1 -- 172.21.15.28:0/404647316 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc3a004c060 con 0x7fc3b80c3280 2024-02-21T23:09:49.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.449+0000 7fc3b4c00640 1 --2- 172.21.15.28:0/404647316 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc39406b4d0 0x7fc39406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:49.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.449+0000 7fc3b4c00640 1 -- 172.21.15.28:0/404647316 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fc3a00abae0 con 0x7fc3b80c3280 2024-02-21T23:09:49.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.450+0000 7fc3b7e00640 1 --2- 172.21.15.28:0/404647316 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc39406b4d0 0x7fc39406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:49.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.451+0000 7fc3b7e00640 1 --2- 172.21.15.28:0/404647316 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc39406b4d0 0x7fc39406d990 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fc3b80c0770 tx=0x7fc3a8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:49.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.454+0000 7fc3b4c00640 1 -- 172.21.15.28:0/404647316 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc3a0040070 con 0x7fc3b80c3280 2024-02-21T23:09:49.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.717+0000 7fc3bea00640 1 -- 172.21.15.28:0/404647316 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fc3b8152e30 con 0x7fc3b80c3280 2024-02-21T23:09:49.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.718+0000 7fc3b4c00640 1 -- 172.21.15.28:0/404647316 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v17) v1 ==== 80+0+4724 (secure 0 0 0) 0x7fc3a007ab10 con 0x7fc3b80c3280 2024-02-21T23:09:49.706 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:09:49.706 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-21T23:09:49.706 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_dump": [ 2024-02-21T23:09:49.706 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:09:49.706 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.a", 2024-02-21T23:09:49.706 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDSgNZl4FY1KRAAJSK5Ibb0ibKiKXYpTLgtmQ==", 2024-02-21T23:09:49.706 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:09:49.706 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:09:49.707 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-02-21T23:09:49.707 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-02-21T23:09:49.707 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:09:49.707 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:09:49.707 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:09:49.707 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:09:49.707 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.b", 2024-02-21T23:09:49.707 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDTgNZlwRcAERAAgsYmmkuobKelBEJVCRpXUg==", 2024-02-21T23:09:49.707 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:09:49.707 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:09:49.707 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-02-21T23:09:49.707 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-02-21T23:09:49.707 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:09:49.707 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:09:49.707 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:09:49.708 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:09:49.708 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.c", 2024-02-21T23:09:49.708 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDSgNZloFpQOxAAK2WBArXRVNTgqH9WWDEPDA==", 2024-02-21T23:09:49.708 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:09:49.708 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:09:49.708 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-02-21T23:09:49.708 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-02-21T23:09:49.708 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:09:49.708 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:09:49.708 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:09:49.708 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:09:49.709 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.d", 2024-02-21T23:09:49.709 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDTgNZlblOmIRAAxe2HfSKGSkkzJDN4aVdzgQ==", 2024-02-21T23:09:49.709 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:09:49.709 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:09:49.709 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-02-21T23:09:49.709 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-02-21T23:09:49.709 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:09:49.709 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:09:49.709 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:09:49.709 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:09:49.709 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.0", 2024-02-21T23:09:49.709 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDVgNZlxNJbLxAAXot7Q3chrLEBsFNO8kT6QQ==", 2024-02-21T23:09:49.709 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:09:49.709 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:09:49.709 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:09:49.710 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:09:49.710 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:09:49.710 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:09:49.710 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:09:49.710 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.1", 2024-02-21T23:09:49.710 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDYgNZlZZfWORAAr4XkOxLYLmumVbu362N2HQ==", 2024-02-21T23:09:49.710 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:09:49.710 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:09:49.710 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:09:49.710 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:09:49.710 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:09:49.711 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:09:49.711 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:09:49.711 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.2", 2024-02-21T23:09:49.711 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDcgNZl9mcDGBAAIZGOF8IUMenwo386x95mwg==", 2024-02-21T23:09:49.711 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:09:49.711 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:09:49.711 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:09:49.711 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:09:49.711 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:09:49.711 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:09:49.711 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:09:49.711 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.3", 2024-02-21T23:09:49.711 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDfgNZl8ALSMBAAag9cmGH0/4xALWSUmrZowg==", 2024-02-21T23:09:49.711 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:09:49.712 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:09:49.712 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:09:49.712 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:09:49.712 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:09:49.712 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:09:49.712 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:09:49.712 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.4", 2024-02-21T23:09:49.712 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDngNZlBgeKABAAeW99RVfCBvB1lp7w7mQBlQ==", 2024-02-21T23:09:49.712 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:09:49.712 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:09:49.713 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:09:49.713 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:09:49.713 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:09:49.713 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:09:49.713 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:09:49.713 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.5", 2024-02-21T23:09:49.713 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDqgNZljQG3ChAAY/IZEIcvcGNvlFGIWib3gg==", 2024-02-21T23:09:49.713 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:09:49.713 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:09:49.713 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:09:49.713 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:09:49.713 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:09:49.713 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:09:49.713 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:09:49.714 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.6", 2024-02-21T23:09:49.714 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDtgNZlKnakIxAAkW5xlkshzvEHJdmpgPxR+g==", 2024-02-21T23:09:49.714 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:09:49.714 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:09:49.714 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:09:49.714 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:09:49.714 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:09:49.714 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:09:49.714 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:09:49.714 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.7", 2024-02-21T23:09:49.714 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDxgNZlTWxYARAASX1/gmqO+8mQPP8uA4xjiQ==", 2024-02-21T23:09:49.714 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:09:49.714 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:09:49.714 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:09:49.714 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:09:49.715 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:09:49.715 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:09:49.715 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:09:49.715 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.0", 2024-02-21T23:09:49.715 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDTgNZlILvYLxAAoREWxKKT7YcXnhkgk8YvoQ==", 2024-02-21T23:09:49.715 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:09:49.715 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:09:49.715 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow r", 2024-02-21T23:09:49.715 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-02-21T23:09:49.715 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:09:49.715 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:09:49.715 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:09:49.715 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.1", 2024-02-21T23:09:49.715 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDTgNZlm7ImOxAA8iRcXlzFPZPaXQfsKhuO/g==", 2024-02-21T23:09:49.715 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:09:49.716 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:09:49.716 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-02-21T23:09:49.716 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-02-21T23:09:49.716 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-02-21T23:09:49.716 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:09:49.716 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:09:49.716 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:09:49.716 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.admin", 2024-02-21T23:09:49.716 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDRgNZlBYd1GhAAbK5qBCIkf0Ej5nEyPnTxIQ==", 2024-02-21T23:09:49.716 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:09:49.716 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-02-21T23:09:49.716 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-02-21T23:09:49.716 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-02-21T23:09:49.717 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:09:49.717 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:09:49.717 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:09:49.717 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:09:49.717 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.x", 2024-02-21T23:09:49.717 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDSgNZlJYKeHBAAYPskmLRr7Jw44TdKzM/JMg==", 2024-02-21T23:09:49.717 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:09:49.717 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-02-21T23:09:49.717 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-02-21T23:09:49.717 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:09:49.717 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:09:49.717 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:09:49.717 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:09:49.717 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.y", 2024-02-21T23:09:49.717 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDSgNZlSRkOEBAAOhRj9PsZDpjoV6WT4drrZA==", 2024-02-21T23:09:49.718 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:09:49.718 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-02-21T23:09:49.718 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-02-21T23:09:49.718 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:09:49.718 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:09:49.718 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:09:49.718 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-21T23:09:49.718 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-21T23:09:49.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.721+0000 7fc3bea00640 1 -- 172.21.15.28:0/404647316 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc39406b4d0 msgr2=0x7fc39406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:49.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.721+0000 7fc3bea00640 1 --2- 172.21.15.28:0/404647316 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc39406b4d0 0x7fc39406d990 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fc3b80c0770 tx=0x7fc3a8009040 comp rx=0 tx=0).stop 2024-02-21T23:09:49.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.721+0000 7fc3bea00640 1 -- 172.21.15.28:0/404647316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3b80c3280 msgr2=0x7fc3b814bbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:49.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.721+0000 7fc3bea00640 1 --2- 172.21.15.28:0/404647316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3b80c3280 0x7fc3b814bbd0 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7fc3a002b8c0 tx=0x7fc3a00046c0 comp rx=0 tx=0).stop 2024-02-21T23:09:49.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.721+0000 7fc3bea00640 1 -- 172.21.15.28:0/404647316 shutdown_connections 2024-02-21T23:09:49.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.721+0000 7fc3bea00640 1 --2- 172.21.15.28:0/404647316 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc39406b4d0 0x7fc39406d990 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:49.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.721+0000 7fc3bea00640 1 --2- 172.21.15.28:0/404647316 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc3b8155860 0x7fc3b80a7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:49.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.721+0000 7fc3bea00640 1 --2- 172.21.15.28:0/404647316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3b80c3280 0x7fc3b814bbd0 unknown :-1 s=CLOSED pgs=1656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:49.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.721+0000 7fc3bea00640 1 --2- 172.21.15.28:0/404647316 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc3b8072430 0x7fc3b80ac3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:49.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.721+0000 7fc3bea00640 1 -- 172.21.15.28:0/404647316 >> 172.21.15.28:0/404647316 conn(0x7fc3b80c67f0 msgr2=0x7fc3b80c26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:49.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.721+0000 7fc3bea00640 1 -- 172.21.15.28:0/404647316 shutdown_connections 2024-02-21T23:09:49.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.721+0000 7fc3bea00640 1 -- 172.21.15.28:0/404647316 wait complete. 2024-02-21T23:09:49.728 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:09:49.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.958+0000 7f3f76400640 1 Processor -- start 2024-02-21T23:09:49.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.958+0000 7f3f76400640 1 -- start start 2024-02-21T23:09:49.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.960+0000 7f3f76400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3f700c0fe0 0x7f3f7006e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:49.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.960+0000 7f3f76400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f3f70158580 0x7f3f7014b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:49.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.960+0000 7f3f76400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f700a8b40 con 0x7f3f7006dff0 2024-02-21T23:09:49.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.960+0000 7f3f76400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f700a8e00 con 0x7f3f700c0fe0 2024-02-21T23:09:49.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.960+0000 7f3f76400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f700a90e0 con 0x7f3f70158580 2024-02-21T23:09:49.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.960+0000 7f3f67e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3f700c0fe0 0x7f3f7006e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:49.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.960+0000 7f3f75000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f3f7006dff0 0x7f3f70071350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36972/0 (socket says 172.21.15.28:36972) 2024-02-21T23:09:49.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.960+0000 7f3f75000640 1 -- 172.21.15.28:0/2396257569 learned_addr learned my addr 172.21.15.28:0/2396257569 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:09:49.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.961+0000 7f3f75a00640 1 --2- 172.21.15.28:0/2396257569 >> v2:172.21.15.81:3300/0 conn(0x7f3f70158580 0x7f3f7014b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:49.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.961+0000 7f3f67e00640 1 -- 172.21.15.28:0/2396257569 >> v2:172.21.15.81:3300/0 conn(0x7f3f70158580 msgr2=0x7f3f7014b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:49.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.961+0000 7f3f67e00640 1 --2- 172.21.15.28:0/2396257569 >> v2:172.21.15.81:3300/0 conn(0x7f3f70158580 0x7f3f7014b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:49.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.961+0000 7f3f67e00640 1 -- 172.21.15.28:0/2396257569 >> v1:172.21.15.28:6789/0 conn(0x7f3f7006dff0 legacy=0x7f3f70071350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:49.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.961+0000 7f3f67e00640 1 -- 172.21.15.28:0/2396257569 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f70072ed0 con 0x7f3f700c0fe0 2024-02-21T23:09:49.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.962+0000 7f3f67e00640 1 --2- 172.21.15.28:0/2396257569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f700c0fe0 0x7f3f7006e3d0 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f3f7006f270 tx=0x7f3f6002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc65c5f189b9a157 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:49.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.962+0000 7f3f6fe00640 1 -- 172.21.15.28:0/2396257569 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f60035020 con 0x7f3f700c0fe0 2024-02-21T23:09:49.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.962+0000 7f3f6fe00640 1 -- 172.21.15.28:0/2396257569 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f60031830 con 0x7f3f700c0fe0 2024-02-21T23:09:49.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.962+0000 7f3f6fe00640 1 -- 172.21.15.28:0/2396257569 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f60031c00 con 0x7f3f700c0fe0 2024-02-21T23:09:49.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.963+0000 7f3f76400640 1 -- 172.21.15.28:0/2396257569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f700c0fe0 msgr2=0x7f3f7006e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:49.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.963+0000 7f3f76400640 1 --2- 172.21.15.28:0/2396257569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f700c0fe0 0x7f3f7006e3d0 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f3f7006f270 tx=0x7f3f6002b8d0 comp rx=0 tx=0).stop 2024-02-21T23:09:49.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.963+0000 7f3f76400640 1 -- 172.21.15.28:0/2396257569 shutdown_connections 2024-02-21T23:09:49.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.963+0000 7f3f76400640 1 --2- 172.21.15.28:0/2396257569 >> v2:172.21.15.81:3300/0 conn(0x7f3f70158580 0x7f3f7014b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:49.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.963+0000 7f3f76400640 1 --2- 172.21.15.28:0/2396257569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f700c0fe0 0x7f3f7006e3d0 unknown :-1 s=CLOSED pgs=1658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:49.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.963+0000 7f3f76400640 1 -- 172.21.15.28:0/2396257569 >> 172.21.15.28:0/2396257569 conn(0x7f3f7006b630 msgr2=0x7f3f700c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:49.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.963+0000 7f3f76400640 1 -- 172.21.15.28:0/2396257569 shutdown_connections 2024-02-21T23:09:49.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.963+0000 7f3f76400640 1 -- 172.21.15.28:0/2396257569 wait complete. 2024-02-21T23:09:49.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.965+0000 7f3f76400640 1 Processor -- start 2024-02-21T23:09:49.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.965+0000 7f3f76400640 1 -- start start 2024-02-21T23:09:49.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.966+0000 7f3f76400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f7006dff0 0x7f3f701eeac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:49.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.967+0000 7f3f76400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f700c0fe0 0x7f3f70154a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:49.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.967+0000 7f3f76400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3f70158580 0x7f3f70156db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:49.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.967+0000 7f3f76400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f70158980 con 0x7f3f7006dff0 2024-02-21T23:09:49.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.967+0000 7f3f76400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f70073520 con 0x7f3f700c0fe0 2024-02-21T23:09:49.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.967+0000 7f3f76400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f70158b00 con 0x7f3f70158580 2024-02-21T23:09:49.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.967+0000 7f3f75000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f7006dff0 0x7f3f701eeac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:49.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.967+0000 7f3f75000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f7006dff0 0x7f3f701eeac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56006/0 (socket says 172.21.15.28:56006) 2024-02-21T23:09:49.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.967+0000 7f3f75000640 1 -- 172.21.15.28:0/370702735 learned_addr learned my addr 172.21.15.28:0/370702735 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:49.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.967+0000 7f3f67e00640 1 --2- 172.21.15.28:0/370702735 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f700c0fe0 0x7f3f70154a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:49.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.967+0000 7f3f75000640 1 -- 172.21.15.28:0/370702735 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3f70158580 msgr2=0x7f3f70156db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:49.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.967+0000 7f3f75a00640 1 --2- 172.21.15.28:0/370702735 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3f70158580 0x7f3f70156db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:49.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.967+0000 7f3f75000640 1 --2- 172.21.15.28:0/370702735 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3f70158580 0x7f3f70156db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:49.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.967+0000 7f3f75000640 1 -- 172.21.15.28:0/370702735 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f700c0fe0 msgr2=0x7f3f70154a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:49.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.967+0000 7f3f75000640 1 --2- 172.21.15.28:0/370702735 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f700c0fe0 0x7f3f70154a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:49.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.967+0000 7f3f75000640 1 -- 172.21.15.28:0/370702735 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f701eac50 con 0x7f3f7006dff0 2024-02-21T23:09:49.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.968+0000 7f3f75000640 1 --2- 172.21.15.28:0/370702735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f7006dff0 0x7f3f701eeac0 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f3f5800c3f0 tx=0x7f3f5800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:49.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.968+0000 7f3f6d600640 1 -- 172.21.15.28:0/370702735 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f58017070 con 0x7f3f7006dff0 2024-02-21T23:09:49.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.968+0000 7f3f6d600640 1 -- 172.21.15.28:0/370702735 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f58005ce0 con 0x7f3f7006dff0 2024-02-21T23:09:49.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.968+0000 7f3f6d600640 1 -- 172.21.15.28:0/370702735 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f580072f0 con 0x7f3f7006dff0 2024-02-21T23:09:49.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.968+0000 7f3f76400640 1 -- 172.21.15.28:0/370702735 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f70152e50 con 0x7f3f7006dff0 2024-02-21T23:09:49.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.968+0000 7f3f76400640 1 -- 172.21.15.28:0/370702735 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f701eaef0 con 0x7f3f7006dff0 2024-02-21T23:09:49.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.969+0000 7f3f76400640 1 -- 172.21.15.28:0/370702735 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f700c2340 con 0x7f3f7006dff0 2024-02-21T23:09:49.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.970+0000 7f3f6d600640 1 -- 172.21.15.28:0/370702735 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3f5801a070 con 0x7f3f7006dff0 2024-02-21T23:09:49.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.971+0000 7f3f6d600640 1 --2- 172.21.15.28:0/370702735 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3f4c06b4d0 0x7f3f4c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:49.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.971+0000 7f3f6d600640 1 -- 172.21.15.28:0/370702735 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f3f580893f0 con 0x7f3f7006dff0 2024-02-21T23:09:49.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.972+0000 7f3f67e00640 1 --2- 172.21.15.28:0/370702735 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3f4c06b4d0 0x7f3f4c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:49.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.973+0000 7f3f67e00640 1 --2- 172.21.15.28:0/370702735 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3f4c06b4d0 0x7f3f4c06d990 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f3f60036000 tx=0x7f3f60008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:49.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:49.977+0000 7f3f6d600640 1 -- 172.21.15.28:0/370702735 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3f700c2340 con 0x7f3f7006dff0 2024-02-21T23:09:50.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.178+0000 7f3f76400640 1 -- 172.21.15.28:0/370702735 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3f7004f290 con 0x7f3f7006dff0 2024-02-21T23:09:50.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.180+0000 7f3f6d600640 1 -- 172.21.15.28:0/370702735 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v147) v1 ==== 74+0+13114 (secure 0 0 0) 0x7f3f58058420 con 0x7f3f7006dff0 2024-02-21T23:09:50.168 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:09:50.168 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":147,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:09:49.218067+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:09:50.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.183+0000 7f3f76400640 1 -- 172.21.15.28:0/370702735 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3f4c06b4d0 msgr2=0x7f3f4c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:50.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.183+0000 7f3f76400640 1 --2- 172.21.15.28:0/370702735 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3f4c06b4d0 0x7f3f4c06d990 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f3f60036000 tx=0x7f3f60008030 comp rx=0 tx=0).stop 2024-02-21T23:09:50.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.183+0000 7f3f76400640 1 -- 172.21.15.28:0/370702735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f7006dff0 msgr2=0x7f3f701eeac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:50.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.183+0000 7f3f76400640 1 --2- 172.21.15.28:0/370702735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f7006dff0 0x7f3f701eeac0 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f3f5800c3f0 tx=0x7f3f5800c8b0 comp rx=0 tx=0).stop 2024-02-21T23:09:50.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.183+0000 7f3f76400640 1 -- 172.21.15.28:0/370702735 shutdown_connections 2024-02-21T23:09:50.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.183+0000 7f3f76400640 1 --2- 172.21.15.28:0/370702735 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3f4c06b4d0 0x7f3f4c06d990 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:50.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.183+0000 7f3f76400640 1 --2- 172.21.15.28:0/370702735 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3f70158580 0x7f3f70156db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:50.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.183+0000 7f3f76400640 1 --2- 172.21.15.28:0/370702735 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f700c0fe0 0x7f3f70154a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:50.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.183+0000 7f3f76400640 1 --2- 172.21.15.28:0/370702735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f7006dff0 0x7f3f701eeac0 unknown :-1 s=CLOSED pgs=1659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:50.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.183+0000 7f3f76400640 1 -- 172.21.15.28:0/370702735 >> 172.21.15.28:0/370702735 conn(0x7f3f7006b630 msgr2=0x7f3f700745b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:50.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.184+0000 7f3f76400640 1 -- 172.21.15.28:0/370702735 shutdown_connections 2024-02-21T23:09:50.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.184+0000 7f3f76400640 1 -- 172.21.15.28:0/370702735 wait complete. 2024-02-21T23:09:50.194 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-21T23:02:30.728329+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-21T23:09:50.194 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-21T23:09:50.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.427+0000 7f4d23e00640 1 Processor -- start 2024-02-21T23:09:50.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.427+0000 7f4d23e00640 1 -- start start 2024-02-21T23:09:50.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.429+0000 7f4d23e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4d2413e0c0 0x7f4d240c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:50.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.429+0000 7f4d23e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f4d240c76f0 0x7f4d2406c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:50.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.429+0000 7f4d23e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d24158550 con 0x7f4d2406dbc0 2024-02-21T23:09:50.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.429+0000 7f4d23e00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d240c89a0 con 0x7f4d240c76f0 2024-02-21T23:09:50.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.429+0000 7f4d23e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d241587d0 con 0x7f4d2413e0c0 2024-02-21T23:09:50.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.430+0000 7f4d22a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4d2413e0c0 0x7f4d240c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:50.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.430+0000 7f4d23400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f4d240c76f0 0x7f4d2406c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:50.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.430+0000 7f4d22a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4d2413e0c0 0x7f4d240c71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3301/0 says I am v2:172.21.15.28:43430/0 (socket says 172.21.15.28:43430) 2024-02-21T23:09:50.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.430+0000 7f4d22a00640 1 -- 172.21.15.28:0/2601040695 learned_addr learned my addr 172.21.15.28:0/2601040695 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:50.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.430+0000 7f4d22a00640 1 -- 172.21.15.28:0/2601040695 >> v2:172.21.15.81:3300/0 conn(0x7f4d240c76f0 msgr2=0x7f4d2406c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:50.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.430+0000 7f4d22a00640 1 --2- 172.21.15.28:0/2601040695 >> v2:172.21.15.81:3300/0 conn(0x7f4d240c76f0 0x7f4d2406c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:50.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.430+0000 7f4d22a00640 1 -- 172.21.15.28:0/2601040695 >> v1:172.21.15.81:6789/0 conn(0x7f4d2406dbc0 legacy=0x7f4d2406c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:50.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.430+0000 7f4d22a00640 1 -- 172.21.15.28:0/2601040695 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d24158ab0 con 0x7f4d2413e0c0 2024-02-21T23:09:50.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.432+0000 7f4d22a00640 1 --2- 172.21.15.28:0/2601040695 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4d2413e0c0 0x7f4d240c71b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f4d14009870 tx=0x7f4d1402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=80aa0c87b5934530 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:50.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.432+0000 7f4d22000640 1 -- 172.21.15.28:0/2601040695 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d14035020 con 0x7f4d2413e0c0 2024-02-21T23:09:50.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.432+0000 7f4d22000640 1 -- 172.21.15.28:0/2601040695 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d14031640 con 0x7f4d2413e0c0 2024-02-21T23:09:50.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.432+0000 7f4d22000640 1 -- 172.21.15.28:0/2601040695 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d1402d040 con 0x7f4d2413e0c0 2024-02-21T23:09:50.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.433+0000 7f4d23e00640 1 -- 172.21.15.28:0/2601040695 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4d2413e0c0 msgr2=0x7f4d240c71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:50.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.433+0000 7f4d23e00640 1 --2- 172.21.15.28:0/2601040695 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4d2413e0c0 0x7f4d240c71b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f4d14009870 tx=0x7f4d1402b8f0 comp rx=0 tx=0).stop 2024-02-21T23:09:50.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.433+0000 7f4d23e00640 1 -- 172.21.15.28:0/2601040695 shutdown_connections 2024-02-21T23:09:50.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.433+0000 7f4d23e00640 1 --2- 172.21.15.28:0/2601040695 >> v2:172.21.15.81:3300/0 conn(0x7f4d240c76f0 0x7f4d2406c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:50.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.433+0000 7f4d23e00640 1 --2- 172.21.15.28:0/2601040695 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4d2413e0c0 0x7f4d240c71b0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:50.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.433+0000 7f4d23e00640 1 -- 172.21.15.28:0/2601040695 >> 172.21.15.28:0/2601040695 conn(0x7f4d24073110 msgr2=0x7f4d240ac490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:50.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.433+0000 7f4d23e00640 1 -- 172.21.15.28:0/2601040695 shutdown_connections 2024-02-21T23:09:50.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.433+0000 7f4d23e00640 1 -- 172.21.15.28:0/2601040695 wait complete. 2024-02-21T23:09:50.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.435+0000 7f4d23e00640 1 Processor -- start 2024-02-21T23:09:50.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.435+0000 7f4d23e00640 1 -- start start 2024-02-21T23:09:50.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.436+0000 7f4d23e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4d2406dbc0 0x7f4d241e1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:50.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.436+0000 7f4d23e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d240c76f0 0x7f4d240ad550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:50.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.436+0000 7f4d23e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d2413e0c0 0x7f4d241e0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:50.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.436+0000 7f4d23e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d2406ced0 con 0x7f4d2413e0c0 2024-02-21T23:09:50.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.436+0000 7f4d23e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d24159070 con 0x7f4d240c76f0 2024-02-21T23:09:50.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.436+0000 7f4d23e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d2413e520 con 0x7f4d2406dbc0 2024-02-21T23:09:50.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.436+0000 7f4d23400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d2413e0c0 0x7f4d241e0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:50.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.436+0000 7f4d23400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d2413e0c0 0x7f4d241e0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56016/0 (socket says 172.21.15.28:56016) 2024-02-21T23:09:50.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.436+0000 7f4d23400640 1 -- 172.21.15.28:0/211808249 learned_addr learned my addr 172.21.15.28:0/211808249 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:50.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.436+0000 7f4d22a00640 1 --2- 172.21.15.28:0/211808249 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4d2406dbc0 0x7f4d241e1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:50.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.436+0000 7f4d1a000640 1 --2- 172.21.15.28:0/211808249 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d240c76f0 0x7f4d240ad550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:50.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.437+0000 7f4d23400640 1 -- 172.21.15.28:0/211808249 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4d2406dbc0 msgr2=0x7f4d241e1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:50.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.437+0000 7f4d23400640 1 --2- 172.21.15.28:0/211808249 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4d2406dbc0 0x7f4d241e1ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:50.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.437+0000 7f4d23400640 1 -- 172.21.15.28:0/211808249 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d240c76f0 msgr2=0x7f4d240ad550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:50.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.437+0000 7f4d23400640 1 --2- 172.21.15.28:0/211808249 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d240c76f0 0x7f4d240ad550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:50.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.437+0000 7f4d23400640 1 -- 172.21.15.28:0/211808249 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d240caf30 con 0x7f4d2413e0c0 2024-02-21T23:09:50.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.437+0000 7f4d23400640 1 --2- 172.21.15.28:0/211808249 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d2413e0c0 0x7f4d241e0430 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f4d1c00efc0 tx=0x7f4d1c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:50.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.437+0000 7f4d1be00640 1 -- 172.21.15.28:0/211808249 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d1c018020 con 0x7f4d2413e0c0 2024-02-21T23:09:50.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.437+0000 7f4d1be00640 1 -- 172.21.15.28:0/211808249 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d1c014480 con 0x7f4d2413e0c0 2024-02-21T23:09:50.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.437+0000 7f4d23e00640 1 -- 172.21.15.28:0/211808249 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d240ca0f0 con 0x7f4d2413e0c0 2024-02-21T23:09:50.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.438+0000 7f4d1be00640 1 -- 172.21.15.28:0/211808249 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d1c014870 con 0x7f4d2413e0c0 2024-02-21T23:09:50.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.438+0000 7f4d23e00640 1 -- 172.21.15.28:0/211808249 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d240ca6b0 con 0x7f4d2413e0c0 2024-02-21T23:09:50.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.439+0000 7f4d23e00640 1 -- 172.21.15.28:0/211808249 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d241394d0 con 0x7f4d2413e0c0 2024-02-21T23:09:50.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.440+0000 7f4d1be00640 1 -- 172.21.15.28:0/211808249 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4d1c005d10 con 0x7f4d2413e0c0 2024-02-21T23:09:50.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.441+0000 7f4d1be00640 1 --2- 172.21.15.28:0/211808249 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4d0006b4d0 0x7f4d0006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:50.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.441+0000 7f4d1be00640 1 -- 172.21.15.28:0/211808249 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f4d1c08ad10 con 0x7f4d2413e0c0 2024-02-21T23:09:50.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.442+0000 7f4d22a00640 1 --2- 172.21.15.28:0/211808249 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4d0006b4d0 0x7f4d0006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:50.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.443+0000 7f4d22a00640 1 --2- 172.21.15.28:0/211808249 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4d0006b4d0 0x7f4d0006d990 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f4d14036000 tx=0x7f4d1403c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:50.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.445+0000 7f4d1be00640 1 -- 172.21.15.28:0/211808249 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4d241394d0 con 0x7f4d2413e0c0 2024-02-21T23:09:50.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.645+0000 7f4d23e00640 1 -- 172.21.15.28:0/211808249 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4d2404f290 con 0x7f4d2413e0c0 2024-02-21T23:09:50.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.645+0000 7f4d1be00640 1 -- 172.21.15.28:0/211808249 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v147) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4d1c090020 con 0x7f4d2413e0c0 2024-02-21T23:09:50.633 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-02-21T23:09:50.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.648+0000 7f4d23e00640 1 -- 172.21.15.28:0/211808249 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4d0006b4d0 msgr2=0x7f4d0006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:50.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.648+0000 7f4d23e00640 1 --2- 172.21.15.28:0/211808249 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4d0006b4d0 0x7f4d0006d990 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f4d14036000 tx=0x7f4d1403c040 comp rx=0 tx=0).stop 2024-02-21T23:09:50.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.648+0000 7f4d23e00640 1 -- 172.21.15.28:0/211808249 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d2413e0c0 msgr2=0x7f4d241e0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:50.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.648+0000 7f4d23e00640 1 --2- 172.21.15.28:0/211808249 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d2413e0c0 0x7f4d241e0430 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f4d1c00efc0 tx=0x7f4d1c00c490 comp rx=0 tx=0).stop 2024-02-21T23:09:50.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.648+0000 7f4d23e00640 1 -- 172.21.15.28:0/211808249 shutdown_connections 2024-02-21T23:09:50.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.648+0000 7f4d23e00640 1 --2- 172.21.15.28:0/211808249 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4d0006b4d0 0x7f4d0006d990 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:50.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.648+0000 7f4d23e00640 1 --2- 172.21.15.28:0/211808249 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d2413e0c0 0x7f4d241e0430 unknown :-1 s=CLOSED pgs=1660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:50.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.648+0000 7f4d23e00640 1 --2- 172.21.15.28:0/211808249 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d240c76f0 0x7f4d240ad550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:50.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.648+0000 7f4d23e00640 1 --2- 172.21.15.28:0/211808249 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4d2406dbc0 0x7f4d241e1ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:50.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.648+0000 7f4d23e00640 1 -- 172.21.15.28:0/211808249 >> 172.21.15.28:0/211808249 conn(0x7f4d24073110 msgr2=0x7f4d24139d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:50.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.649+0000 7f4d23e00640 1 -- 172.21.15.28:0/211808249 shutdown_connections 2024-02-21T23:09:50.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.649+0000 7f4d23e00640 1 -- 172.21.15.28:0/211808249 wait complete. 2024-02-21T23:09:50.656 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-02-21T23:09:50.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.892+0000 7fc3ab000640 1 Processor -- start 2024-02-21T23:09:50.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.893+0000 7fc3ab000640 1 -- start start 2024-02-21T23:09:50.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.894+0000 7fc3ab000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc3a40b0130 0x7fc3a40c8740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:50.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.894+0000 7fc3ab000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc3a40c51c0 0x7fc3a40c5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:50.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.894+0000 7fc3ab000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3a41550b0 con 0x7fc3a4158510 2024-02-21T23:09:50.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.894+0000 7fc3ab000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3a4152a40 con 0x7fc3a40c51c0 2024-02-21T23:09:50.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.894+0000 7fc3ab000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a4155350 con 0x7fc3a40b0130 2024-02-21T23:09:50.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.894+0000 7fc3aa600640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fc3a4158510 0x7fc3a4158960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36978/0 (socket says 172.21.15.28:36978) 2024-02-21T23:09:50.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.894+0000 7fc3aa600640 1 -- 172.21.15.28:0/2279132265 learned_addr learned my addr 172.21.15.28:0/2279132265 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:09:50.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.894+0000 7fc3a9200640 1 --2- 172.21.15.28:0/2279132265 >> v2:172.21.15.28:3300/0 conn(0x7fc3a40c51c0 0x7fc3a40c5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:50.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.895+0000 7fc3a9c00640 1 --2- 172.21.15.28:0/2279132265 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc3a40b0130 0x7fc3a40c8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:50.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.895+0000 7fc3a9200640 1 -- 172.21.15.28:0/2279132265 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc3a40b0130 msgr2=0x7fc3a40c8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:50.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.895+0000 7fc3a9200640 1 --2- 172.21.15.28:0/2279132265 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc3a40b0130 0x7fc3a40c8740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:50.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.895+0000 7fc3a9200640 1 -- 172.21.15.28:0/2279132265 >> v1:172.21.15.28:6789/0 conn(0x7fc3a4158510 legacy=0x7fc3a4158960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:50.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.895+0000 7fc3a9200640 1 -- 172.21.15.28:0/2279132265 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3a4155630 con 0x7fc3a40c51c0 2024-02-21T23:09:50.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.895+0000 7fc3a3e00640 1 -- 172.21.15.28:0/2279132265 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2478317895 0 0) 0x7fc3a41550b0 con 0x7fc3a4158510 2024-02-21T23:09:50.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.896+0000 7fc3a9200640 1 --2- 172.21.15.28:0/2279132265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3a40c51c0 0x7fc3a40c5640 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7fc398009d90 tx=0x7fc39802be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27ad4832a4096218 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:50.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.896+0000 7fc3a3e00640 1 -- 172.21.15.28:0/2279132265 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc398039070 con 0x7fc3a40c51c0 2024-02-21T23:09:50.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.897+0000 7fc3a3e00640 1 -- 172.21.15.28:0/2279132265 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc39802cbc0 con 0x7fc3a40c51c0 2024-02-21T23:09:50.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.897+0000 7fc3a3e00640 1 -- 172.21.15.28:0/2279132265 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc398034440 con 0x7fc3a40c51c0 2024-02-21T23:09:50.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.897+0000 7fc3ab000640 1 -- 172.21.15.28:0/2279132265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3a40c51c0 msgr2=0x7fc3a40c5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:50.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.897+0000 7fc3ab000640 1 --2- 172.21.15.28:0/2279132265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3a40c51c0 0x7fc3a40c5640 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7fc398009d90 tx=0x7fc39802be10 comp rx=0 tx=0).stop 2024-02-21T23:09:50.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.897+0000 7fc3ab000640 1 -- 172.21.15.28:0/2279132265 shutdown_connections 2024-02-21T23:09:50.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.897+0000 7fc3ab000640 1 --2- 172.21.15.28:0/2279132265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3a40c51c0 0x7fc3a40c5640 unknown :-1 s=CLOSED pgs=1662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:50.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.897+0000 7fc3ab000640 1 --2- 172.21.15.28:0/2279132265 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc3a40b0130 0x7fc3a40c8740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:50.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.897+0000 7fc3ab000640 1 -- 172.21.15.28:0/2279132265 >> 172.21.15.28:0/2279132265 conn(0x7fc3a40ac370 msgr2=0x7fc3a40ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:50.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.897+0000 7fc3ab000640 1 -- 172.21.15.28:0/2279132265 shutdown_connections 2024-02-21T23:09:50.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.897+0000 7fc3ab000640 1 -- 172.21.15.28:0/2279132265 wait complete. 2024-02-21T23:09:50.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.899+0000 7fc3ab000640 1 Processor -- start 2024-02-21T23:09:50.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.899+0000 7fc3ab000640 1 -- start start 2024-02-21T23:09:50.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.900+0000 7fc3ab000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc3a40b0130 0x7fc3a41520b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:50.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.900+0000 7fc3ab000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc3a40c51c0 0x7fc3a40c3760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:50.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.900+0000 7fc3ab000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3a4158510 0x7fc3a40c91c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:50.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.900+0000 7fc3ab000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a40c8dd0 con 0x7fc3a4158510 2024-02-21T23:09:50.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.900+0000 7fc3ab000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a4155e30 con 0x7fc3a40b0130 2024-02-21T23:09:50.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.900+0000 7fc3ab000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a40be060 con 0x7fc3a40c51c0 2024-02-21T23:09:50.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.900+0000 7fc3aa600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3a4158510 0x7fc3a40c91c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:50.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.901+0000 7fc3aa600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3a4158510 0x7fc3a40c91c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56036/0 (socket says 172.21.15.28:56036) 2024-02-21T23:09:50.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.901+0000 7fc3aa600640 1 -- 172.21.15.28:0/2477347635 learned_addr learned my addr 172.21.15.28:0/2477347635 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:50.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.901+0000 7fc3a9200640 1 --2- 172.21.15.28:0/2477347635 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc3a40c51c0 0x7fc3a40c3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:50.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.901+0000 7fc3a9c00640 1 --2- 172.21.15.28:0/2477347635 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc3a40b0130 0x7fc3a41520b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:50.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.901+0000 7fc3aa600640 1 -- 172.21.15.28:0/2477347635 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc3a40c51c0 msgr2=0x7fc3a40c3760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:50.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.901+0000 7fc3aa600640 1 --2- 172.21.15.28:0/2477347635 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc3a40c51c0 0x7fc3a40c3760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:50.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.901+0000 7fc3aa600640 1 -- 172.21.15.28:0/2477347635 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc3a40b0130 msgr2=0x7fc3a41520b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:50.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.901+0000 7fc3aa600640 1 --2- 172.21.15.28:0/2477347635 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc3a40b0130 0x7fc3a41520b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:50.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.901+0000 7fc3aa600640 1 -- 172.21.15.28:0/2477347635 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3a41eb5f0 con 0x7fc3a4158510 2024-02-21T23:09:50.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.901+0000 7fc3a9c00640 1 --2- 172.21.15.28:0/2477347635 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc3a40b0130 0x7fc3a41520b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:09:50.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.901+0000 7fc3aa600640 1 --2- 172.21.15.28:0/2477347635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3a4158510 0x7fc3a40c91c0 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7fc39c00c440 tx=0x7fc39c00c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:50.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.902+0000 7fc3a1600640 1 -- 172.21.15.28:0/2477347635 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc39c0074e0 con 0x7fc3a4158510 2024-02-21T23:09:50.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.902+0000 7fc3a1600640 1 -- 172.21.15.28:0/2477347635 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc39c0047e0 con 0x7fc3a4158510 2024-02-21T23:09:50.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.902+0000 7fc3ab000640 1 -- 172.21.15.28:0/2477347635 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3a40c1370 con 0x7fc3a4158510 2024-02-21T23:09:50.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.902+0000 7fc3a1600640 1 -- 172.21.15.28:0/2477347635 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc39c004bd0 con 0x7fc3a4158510 2024-02-21T23:09:50.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.902+0000 7fc3ab000640 1 -- 172.21.15.28:0/2477347635 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3a40c9700 con 0x7fc3a4158510 2024-02-21T23:09:50.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.903+0000 7fc3ab000640 1 -- 172.21.15.28:0/2477347635 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3a404f290 con 0x7fc3a4158510 2024-02-21T23:09:50.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.904+0000 7fc3a1600640 1 -- 172.21.15.28:0/2477347635 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc39c011070 con 0x7fc3a4158510 2024-02-21T23:09:50.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.905+0000 7fc3a1600640 1 --2- 172.21.15.28:0/2477347635 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc38806b4d0 0x7fc38806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:50.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.905+0000 7fc3a1600640 1 -- 172.21.15.28:0/2477347635 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fc39c089c00 con 0x7fc3a4158510 2024-02-21T23:09:50.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.906+0000 7fc3a9c00640 1 --2- 172.21.15.28:0/2477347635 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc38806b4d0 0x7fc38806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:50.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.907+0000 7fc3a9c00640 1 --2- 172.21.15.28:0/2477347635 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc38806b4d0 0x7fc38806d990 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fc394006fd0 tx=0x7fc394008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:50.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:50.909+0000 7fc3a1600640 1 -- 172.21.15.28:0/2477347635 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc39c058c30 con 0x7fc3a4158510 2024-02-21T23:09:51.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.108+0000 7fc3ab000640 1 -- 172.21.15.28:0/2477347635 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fc3a406d9b0 con 0x7fc3a4158510 2024-02-21T23:09:51.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.109+0000 7fc3a1600640 1 -- 172.21.15.28:0/2477347635 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v147) v1 ==== 84+0+65 (secure 0 0 0) 0x7fc39c058530 con 0x7fc3a4158510 2024-02-21T23:09:51.097 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:09:51.097 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-02-21T23:09:51.097 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:09:51.097 INFO:teuthology.orchestra.run.smithi028.stdout: "poolnum": 1, 2024-02-21T23:09:51.097 INFO:teuthology.orchestra.run.smithi028.stdout: "poolname": ".mgr" 2024-02-21T23:09:51.097 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:09:51.097 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-02-21T23:09:51.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.111+0000 7fc3ab000640 1 -- 172.21.15.28:0/2477347635 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc38806b4d0 msgr2=0x7fc38806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:51.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.111+0000 7fc3ab000640 1 --2- 172.21.15.28:0/2477347635 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc38806b4d0 0x7fc38806d990 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fc394006fd0 tx=0x7fc394008040 comp rx=0 tx=0).stop 2024-02-21T23:09:51.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.111+0000 7fc3ab000640 1 -- 172.21.15.28:0/2477347635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3a4158510 msgr2=0x7fc3a40c91c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:51.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.111+0000 7fc3ab000640 1 --2- 172.21.15.28:0/2477347635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3a4158510 0x7fc3a40c91c0 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7fc39c00c440 tx=0x7fc39c00c900 comp rx=0 tx=0).stop 2024-02-21T23:09:51.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.111+0000 7fc3ab000640 1 -- 172.21.15.28:0/2477347635 shutdown_connections 2024-02-21T23:09:51.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.111+0000 7fc3ab000640 1 --2- 172.21.15.28:0/2477347635 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc38806b4d0 0x7fc38806d990 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:51.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.111+0000 7fc3ab000640 1 --2- 172.21.15.28:0/2477347635 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc3a40c51c0 0x7fc3a40c3760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:51.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.111+0000 7fc3ab000640 1 --2- 172.21.15.28:0/2477347635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3a4158510 0x7fc3a40c91c0 unknown :-1 s=CLOSED pgs=1663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:51.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.111+0000 7fc3ab000640 1 --2- 172.21.15.28:0/2477347635 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc3a40b0130 0x7fc3a41520b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:51.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.111+0000 7fc3ab000640 1 -- 172.21.15.28:0/2477347635 >> 172.21.15.28:0/2477347635 conn(0x7fc3a40ac370 msgr2=0x7fc3a40ac750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:51.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.111+0000 7fc3ab000640 1 -- 172.21.15.28:0/2477347635 shutdown_connections 2024-02-21T23:09:51.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.112+0000 7fc3ab000640 1 -- 172.21.15.28:0/2477347635 wait complete. 2024-02-21T23:09:51.118 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-02-21T23:09:51.118 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-02-21T23:09:51.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.357+0000 7fbf49800640 1 Processor -- start 2024-02-21T23:09:51.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.358+0000 7fbf49800640 1 -- start start 2024-02-21T23:09:51.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.359+0000 7fbf49800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbf4406cfa0 0x7fbf440ca540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:51.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.359+0000 7fbf49800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fbf440c6290 0x7fbf441522c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:51.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.359+0000 7fbf49800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf440caf40 con 0x7fbf44158460 2024-02-21T23:09:51.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.359+0000 7fbf49800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf440c8ae0 con 0x7fbf440c6290 2024-02-21T23:09:51.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.359+0000 7fbf49800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf440c87b0 con 0x7fbf4406cfa0 2024-02-21T23:09:51.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.360+0000 7fbf43e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbf4406cfa0 0x7fbf440ca540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:51.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.360+0000 7fbf48e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fbf440c6290 0x7fbf441522c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:51.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.360+0000 7fbf43e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbf4406cfa0 0x7fbf440ca540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3301/0 says I am v2:172.21.15.28:43462/0 (socket says 172.21.15.28:43462) 2024-02-21T23:09:51.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.360+0000 7fbf43e00640 1 -- 172.21.15.28:0/3243778905 learned_addr learned my addr 172.21.15.28:0/3243778905 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:51.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.361+0000 7fbf48e00640 1 -- 172.21.15.28:0/3243778905 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbf4406cfa0 msgr2=0x7fbf440ca540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:51.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.361+0000 7fbf48e00640 1 --2- 172.21.15.28:0/3243778905 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbf4406cfa0 0x7fbf440ca540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:51.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.361+0000 7fbf48e00640 1 -- 172.21.15.28:0/3243778905 >> v1:172.21.15.81:6789/0 conn(0x7fbf44158460 legacy=0x7fbf440c8f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:51.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.361+0000 7fbf48e00640 1 -- 172.21.15.28:0/3243778905 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf441588c0 con 0x7fbf440c6290 2024-02-21T23:09:51.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.362+0000 7fbf48e00640 1 --2- 172.21.15.28:0/3243778905 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf440c6290 0x7fbf441522c0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fbf440c8c60 tx=0x7fbf3c02bd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=285368175660884f server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:51.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.363+0000 7fbf42a00640 1 -- 172.21.15.28:0/3243778905 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf3c039070 con 0x7fbf440c6290 2024-02-21T23:09:51.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.363+0000 7fbf42a00640 1 -- 172.21.15.28:0/3243778905 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf3c02cb20 con 0x7fbf440c6290 2024-02-21T23:09:51.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.363+0000 7fbf42a00640 1 -- 172.21.15.28:0/3243778905 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf3c034440 con 0x7fbf440c6290 2024-02-21T23:09:51.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.363+0000 7fbf49800640 1 -- 172.21.15.28:0/3243778905 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf440c6290 msgr2=0x7fbf441522c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:51.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.363+0000 7fbf49800640 1 --2- 172.21.15.28:0/3243778905 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf440c6290 0x7fbf441522c0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fbf440c8c60 tx=0x7fbf3c02bd70 comp rx=0 tx=0).stop 2024-02-21T23:09:51.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.363+0000 7fbf49800640 1 -- 172.21.15.28:0/3243778905 shutdown_connections 2024-02-21T23:09:51.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.363+0000 7fbf49800640 1 --2- 172.21.15.28:0/3243778905 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf440c6290 0x7fbf441522c0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:51.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.363+0000 7fbf49800640 1 --2- 172.21.15.28:0/3243778905 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbf4406cfa0 0x7fbf440ca540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:51.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.363+0000 7fbf49800640 1 -- 172.21.15.28:0/3243778905 >> 172.21.15.28:0/3243778905 conn(0x7fbf440725a0 msgr2=0x7fbf440729b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:51.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.363+0000 7fbf49800640 1 -- 172.21.15.28:0/3243778905 shutdown_connections 2024-02-21T23:09:51.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.364+0000 7fbf49800640 1 -- 172.21.15.28:0/3243778905 wait complete. 2024-02-21T23:09:51.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.365+0000 7fbf49800640 1 Processor -- start 2024-02-21T23:09:51.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.365+0000 7fbf49800640 1 -- start start 2024-02-21T23:09:51.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.366+0000 7fbf49800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf4406cfa0 0x7fbf4414cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:51.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.366+0000 7fbf49800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf440c6290 0x7fbf440aa920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:51.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.366+0000 7fbf49800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbf44158460 0x7fbf440aae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:51.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.366+0000 7fbf49800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf440c6c90 con 0x7fbf4406cfa0 2024-02-21T23:09:51.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.366+0000 7fbf49800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf440c69b0 con 0x7fbf440c6290 2024-02-21T23:09:51.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.366+0000 7fbf49800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf440c66c0 con 0x7fbf44158460 2024-02-21T23:09:51.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.366+0000 7fbf43e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf4406cfa0 0x7fbf4414cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:51.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.367+0000 7fbf43e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf4406cfa0 0x7fbf4414cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56038/0 (socket says 172.21.15.28:56038) 2024-02-21T23:09:51.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.367+0000 7fbf43e00640 1 -- 172.21.15.28:0/2808687957 learned_addr learned my addr 172.21.15.28:0/2808687957 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:51.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.367+0000 7fbf43e00640 1 -- 172.21.15.28:0/2808687957 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbf44158460 msgr2=0x7fbf440aae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:51.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.367+0000 7fbf43e00640 1 --2- 172.21.15.28:0/2808687957 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbf44158460 0x7fbf440aae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:51.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.367+0000 7fbf43e00640 1 -- 172.21.15.28:0/2808687957 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf440c6290 msgr2=0x7fbf440aa920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:51.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.367+0000 7fbf43400640 1 --2- 172.21.15.28:0/2808687957 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf440c6290 0x7fbf440aa920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:51.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.367+0000 7fbf43e00640 1 --2- 172.21.15.28:0/2808687957 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf440c6290 0x7fbf440aa920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:51.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.367+0000 7fbf43e00640 1 -- 172.21.15.28:0/2808687957 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf440acd80 con 0x7fbf4406cfa0 2024-02-21T23:09:51.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.367+0000 7fbf43400640 1 --2- 172.21.15.28:0/2808687957 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf440c6290 0x7fbf440aa920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:09:51.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.367+0000 7fbf43e00640 1 --2- 172.21.15.28:0/2808687957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf4406cfa0 0x7fbf4414cb60 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7fbf3400e8b0 tx=0x7fbf3400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:51.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.368+0000 7fbf3be00640 1 -- 172.21.15.28:0/2808687957 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf34017070 con 0x7fbf4406cfa0 2024-02-21T23:09:51.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.368+0000 7fbf3be00640 1 -- 172.21.15.28:0/2808687957 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf3400c940 con 0x7fbf4406cfa0 2024-02-21T23:09:51.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.368+0000 7fbf49800640 1 -- 172.21.15.28:0/2808687957 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf440ad060 con 0x7fbf4406cfa0 2024-02-21T23:09:51.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.368+0000 7fbf3be00640 1 -- 172.21.15.28:0/2808687957 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf3400cd10 con 0x7fbf4406cfa0 2024-02-21T23:09:51.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.368+0000 7fbf49800640 1 -- 172.21.15.28:0/2808687957 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf440af5d0 con 0x7fbf4406cfa0 2024-02-21T23:09:51.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.369+0000 7fbf49800640 1 -- 172.21.15.28:0/2808687957 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf440ad230 con 0x7fbf4406cfa0 2024-02-21T23:09:51.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.370+0000 7fbf3be00640 1 -- 172.21.15.28:0/2808687957 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbf3401a070 con 0x7fbf4406cfa0 2024-02-21T23:09:51.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.371+0000 7fbf3be00640 1 --2- 172.21.15.28:0/2808687957 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbf2006b500 0x7fbf2006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:51.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.371+0000 7fbf3be00640 1 -- 172.21.15.28:0/2808687957 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fbf3408a390 con 0x7fbf4406cfa0 2024-02-21T23:09:51.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.372+0000 7fbf43400640 1 --2- 172.21.15.28:0/2808687957 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbf2006b500 0x7fbf2006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:51.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.373+0000 7fbf43400640 1 --2- 172.21.15.28:0/2808687957 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbf2006b500 0x7fbf2006d9c0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fbf4414b770 tx=0x7fbf2c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:51.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.375+0000 7fbf3be00640 1 -- 172.21.15.28:0/2808687957 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fbf34059330 con 0x7fbf4406cfa0 2024-02-21T23:09:51.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:51.576+0000 7fbf49800640 1 -- 172.21.15.28:0/2808687957 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fbf4404f290 con 0x7fbf4406cfa0 2024-02-21T23:09:52.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.230+0000 7fbf3be00640 1 -- 172.21.15.28:0/2808687957 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v148) v1 ==== 138+0+0 (secure 0 0 0) 0x7fbf34058c30 con 0x7fbf4406cfa0 2024-02-21T23:09:52.219 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' created 2024-02-21T23:09:52.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.234+0000 7fbf49800640 1 -- 172.21.15.28:0/2808687957 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbf2006b500 msgr2=0x7fbf2006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:52.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.234+0000 7fbf49800640 1 --2- 172.21.15.28:0/2808687957 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbf2006b500 0x7fbf2006d9c0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fbf4414b770 tx=0x7fbf2c009040 comp rx=0 tx=0).stop 2024-02-21T23:09:52.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.234+0000 7fbf49800640 1 -- 172.21.15.28:0/2808687957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf4406cfa0 msgr2=0x7fbf4414cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:52.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.234+0000 7fbf49800640 1 --2- 172.21.15.28:0/2808687957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf4406cfa0 0x7fbf4414cb60 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7fbf3400e8b0 tx=0x7fbf3400ed70 comp rx=0 tx=0).stop 2024-02-21T23:09:52.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.234+0000 7fbf49800640 1 -- 172.21.15.28:0/2808687957 shutdown_connections 2024-02-21T23:09:52.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.234+0000 7fbf49800640 1 --2- 172.21.15.28:0/2808687957 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbf2006b500 0x7fbf2006d9c0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:52.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.234+0000 7fbf49800640 1 --2- 172.21.15.28:0/2808687957 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbf44158460 0x7fbf440aae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:52.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.234+0000 7fbf49800640 1 --2- 172.21.15.28:0/2808687957 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf440c6290 0x7fbf440aa920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:52.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.234+0000 7fbf49800640 1 --2- 172.21.15.28:0/2808687957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf4406cfa0 0x7fbf4414cb60 unknown :-1 s=CLOSED pgs=1664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:52.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.234+0000 7fbf49800640 1 -- 172.21.15.28:0/2808687957 >> 172.21.15.28:0/2808687957 conn(0x7fbf440725a0 msgr2=0x7fbf441503a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:52.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.234+0000 7fbf49800640 1 -- 172.21.15.28:0/2808687957 shutdown_connections 2024-02-21T23:09:52.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.234+0000 7fbf49800640 1 -- 172.21.15.28:0/2808687957 wait complete. 2024-02-21T23:09:52.250 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-02-21T23:09:52.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.469+0000 7f460cc00640 1 Processor -- start 2024-02-21T23:09:52.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.469+0000 7f460cc00640 1 -- start start 2024-02-21T23:09:52.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.471+0000 7f460cc00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f460806f6b0 0x7f46080c2b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:52.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.471+0000 7f460cc00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f46080c61c0 0x7f46080c46f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:52.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.471+0000 7f460cc00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46080c50a0 con 0x7f46080c4c30 2024-02-21T23:09:52.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.471+0000 7f460cc00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f460806b700 con 0x7f46080c61c0 2024-02-21T23:09:52.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.471+0000 7f460cc00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46080c3070 con 0x7f460806f6b0 2024-02-21T23:09:52.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.471+0000 7f4605400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f46080c61c0 0x7f46080c46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:52.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.472+0000 7f4605400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f46080c61c0 0x7f46080c46f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56044/0 (socket says 172.21.15.28:56044) 2024-02-21T23:09:52.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.472+0000 7f4605400640 1 -- 172.21.15.28:0/414399403 learned_addr learned my addr 172.21.15.28:0/414399403 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:52.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.472+0000 7f4605e00640 1 --2- 172.21.15.28:0/414399403 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f460806f6b0 0x7f46080c2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:52.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.472+0000 7f4605400640 1 -- 172.21.15.28:0/414399403 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f460806f6b0 msgr2=0x7f46080c2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:52.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.472+0000 7f4605400640 1 --2- 172.21.15.28:0/414399403 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f460806f6b0 0x7f46080c2b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:52.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.472+0000 7f4605400640 1 -- 172.21.15.28:0/414399403 >> v1:172.21.15.81:6789/0 conn(0x7f46080c4c30 legacy=0x7f4608154500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:52.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.472+0000 7f4605400640 1 -- 172.21.15.28:0/414399403 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f460806ef60 con 0x7f46080c61c0 2024-02-21T23:09:52.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.473+0000 7f4605400640 1 --2- 172.21.15.28:0/414399403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46080c61c0 0x7f46080c46f0 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7f45f400aa10 tx=0x7f45f402cdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=117c2c25187c0548 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:52.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.474+0000 7f4604a00640 1 -- 172.21.15.28:0/414399403 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45f403a070 con 0x7f46080c61c0 2024-02-21T23:09:52.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.474+0000 7f4604a00640 1 -- 172.21.15.28:0/414399403 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45f402db80 con 0x7f46080c61c0 2024-02-21T23:09:52.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.474+0000 7f4604a00640 1 -- 172.21.15.28:0/414399403 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45f400b040 con 0x7f46080c61c0 2024-02-21T23:09:52.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.474+0000 7f460cc00640 1 -- 172.21.15.28:0/414399403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46080c61c0 msgr2=0x7f46080c46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:52.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.474+0000 7f460cc00640 1 --2- 172.21.15.28:0/414399403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46080c61c0 0x7f46080c46f0 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7f45f400aa10 tx=0x7f45f402cdd0 comp rx=0 tx=0).stop 2024-02-21T23:09:52.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.474+0000 7f460cc00640 1 -- 172.21.15.28:0/414399403 shutdown_connections 2024-02-21T23:09:52.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.475+0000 7f460cc00640 1 --2- 172.21.15.28:0/414399403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46080c61c0 0x7f46080c46f0 unknown :-1 s=CLOSED pgs=1665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:52.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.475+0000 7f460cc00640 1 --2- 172.21.15.28:0/414399403 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f460806f6b0 0x7f46080c2b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:52.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.475+0000 7f460cc00640 1 -- 172.21.15.28:0/414399403 >> 172.21.15.28:0/414399403 conn(0x7f46080a85a0 msgr2=0x7f46080a8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:52.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.475+0000 7f460cc00640 1 -- 172.21.15.28:0/414399403 shutdown_connections 2024-02-21T23:09:52.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.475+0000 7f460cc00640 1 -- 172.21.15.28:0/414399403 wait complete. 2024-02-21T23:09:52.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.476+0000 7f460cc00640 1 Processor -- start 2024-02-21T23:09:52.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.476+0000 7f460cc00640 1 -- start start 2024-02-21T23:09:52.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.477+0000 7f460cc00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f460806f6b0 0x7f46081da720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:52.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.477+0000 7f460cc00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f46080c4c30 0x7f460814f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:52.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.477+0000 7f460cc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46080c61c0 0x7f46081db240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:52.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.477+0000 7f460cc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4608151e60 con 0x7f46080c61c0 2024-02-21T23:09:52.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.477+0000 7f460cc00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f460806d350 con 0x7f460806f6b0 2024-02-21T23:09:52.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.477+0000 7f460cc00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f460806d7e0 con 0x7f46080c4c30 2024-02-21T23:09:52.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.478+0000 7f4606800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46080c61c0 0x7f46081db240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:52.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.478+0000 7f4606800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46080c61c0 0x7f46081db240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56056/0 (socket says 172.21.15.28:56056) 2024-02-21T23:09:52.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.478+0000 7f4606800640 1 -- 172.21.15.28:0/1333150379 learned_addr learned my addr 172.21.15.28:0/1333150379 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:52.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.478+0000 7f4605400640 1 --2- 172.21.15.28:0/1333150379 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f46080c4c30 0x7f460814f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:52.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.478+0000 7f4605e00640 1 --2- 172.21.15.28:0/1333150379 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f460806f6b0 0x7f46081da720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:52.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.478+0000 7f4606800640 1 -- 172.21.15.28:0/1333150379 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f46080c4c30 msgr2=0x7f460814f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:52.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.478+0000 7f4606800640 1 --2- 172.21.15.28:0/1333150379 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f46080c4c30 0x7f460814f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:52.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.478+0000 7f4606800640 1 -- 172.21.15.28:0/1333150379 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f460806f6b0 msgr2=0x7f46081da720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:52.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.478+0000 7f4606800640 1 --2- 172.21.15.28:0/1333150379 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f460806f6b0 0x7f46081da720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:52.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.478+0000 7f4606800640 1 -- 172.21.15.28:0/1333150379 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f460814de90 con 0x7f46080c61c0 2024-02-21T23:09:52.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.479+0000 7f4606800640 1 --2- 172.21.15.28:0/1333150379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46080c61c0 0x7f46081db240 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f460000c7b0 tx=0x7f460000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:52.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.479+0000 7f45fe000640 1 -- 172.21.15.28:0/1333150379 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4600004b60 con 0x7f46080c61c0 2024-02-21T23:09:52.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.479+0000 7f45fe000640 1 -- 172.21.15.28:0/1333150379 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4600004d00 con 0x7f46080c61c0 2024-02-21T23:09:52.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.479+0000 7f45fe000640 1 -- 172.21.15.28:0/1333150379 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f460000aa90 con 0x7f46080c61c0 2024-02-21T23:09:52.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.479+0000 7f460cc00640 1 -- 172.21.15.28:0/1333150379 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46081ea780 con 0x7f46080c61c0 2024-02-21T23:09:52.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.479+0000 7f460cc00640 1 -- 172.21.15.28:0/1333150379 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46080c0a30 con 0x7f46080c61c0 2024-02-21T23:09:52.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.480+0000 7f460cc00640 1 -- 172.21.15.28:0/1333150379 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4608067f30 con 0x7f46080c61c0 2024-02-21T23:09:52.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.482+0000 7f45fe000640 1 -- 172.21.15.28:0/1333150379 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f460000ac30 con 0x7f46080c61c0 2024-02-21T23:09:52.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.483+0000 7f45fe000640 1 --2- 172.21.15.28:0/1333150379 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f45e806b4d0 0x7f45e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:52.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.483+0000 7f45fe000640 1 -- 172.21.15.28:0/1333150379 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 5619+0+0 (secure 0 0 0) 0x7f460008acc0 con 0x7f46080c61c0 2024-02-21T23:09:52.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.483+0000 7f4605e00640 1 --2- 172.21.15.28:0/1333150379 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f45e806b4d0 0x7f45e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:52.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.485+0000 7f4605e00640 1 --2- 172.21.15.28:0/1333150379 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f45e806b4d0 0x7f45e806d990 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f45f8006fd0 tx=0x7f45f8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:52.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.486+0000 7f45fe000640 1 -- 172.21.15.28:0/1333150379 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4600020070 con 0x7f46080c61c0 2024-02-21T23:09:52.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:52.687+0000 7f460cc00640 1 -- 172.21.15.28:0/1333150379 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f46080c5fa0 con 0x7f46080c61c0 2024-02-21T23:09:53.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.247+0000 7f45fe000640 1 -- 172.21.15.28:0/1333150379 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v149) v1 ==== 170+0+0 (secure 0 0 0) 0x7f4600059af0 con 0x7f46080c61c0 2024-02-21T23:09:53.235 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' created 2024-02-21T23:09:53.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.250+0000 7f460cc00640 1 -- 172.21.15.28:0/1333150379 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f45e806b4d0 msgr2=0x7f45e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:53.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.250+0000 7f460cc00640 1 --2- 172.21.15.28:0/1333150379 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f45e806b4d0 0x7f45e806d990 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f45f8006fd0 tx=0x7f45f8008040 comp rx=0 tx=0).stop 2024-02-21T23:09:53.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.250+0000 7f460cc00640 1 -- 172.21.15.28:0/1333150379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46080c61c0 msgr2=0x7f46081db240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:53.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.250+0000 7f460cc00640 1 --2- 172.21.15.28:0/1333150379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46080c61c0 0x7f46081db240 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f460000c7b0 tx=0x7f460000cc70 comp rx=0 tx=0).stop 2024-02-21T23:09:53.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.250+0000 7f460cc00640 1 -- 172.21.15.28:0/1333150379 shutdown_connections 2024-02-21T23:09:53.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.250+0000 7f460cc00640 1 --2- 172.21.15.28:0/1333150379 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f45e806b4d0 0x7f45e806d990 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:53.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.250+0000 7f460cc00640 1 --2- 172.21.15.28:0/1333150379 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f46080c4c30 0x7f460814f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:53.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.250+0000 7f460cc00640 1 --2- 172.21.15.28:0/1333150379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46080c61c0 0x7f46081db240 unknown :-1 s=CLOSED pgs=1666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:53.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.250+0000 7f460cc00640 1 --2- 172.21.15.28:0/1333150379 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f460806f6b0 0x7f46081da720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:53.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.250+0000 7f460cc00640 1 -- 172.21.15.28:0/1333150379 >> 172.21.15.28:0/1333150379 conn(0x7f46080a85a0 msgr2=0x7f46080ab630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:53.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.251+0000 7f460cc00640 1 -- 172.21.15.28:0/1333150379 shutdown_connections 2024-02-21T23:09:53.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.251+0000 7f460cc00640 1 -- 172.21.15.28:0/1333150379 wait complete. 2024-02-21T23:09:53.272 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-02-21T23:09:53.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.471+0000 7fcd7f400640 1 Processor -- start 2024-02-21T23:09:53.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.471+0000 7fcd7f400640 1 -- start start 2024-02-21T23:09:53.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.473+0000 7fcd7f400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcd78158dc0 0x7fcd781609f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:53.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.473+0000 7fcd7f400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd780c20d0 con 0x7fcd780c6220 2024-02-21T23:09:53.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.473+0000 7fcd7f400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd780c1290 con 0x7fcd780c4960 2024-02-21T23:09:53.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.473+0000 7fcd7f400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd780a62e0 con 0x7fcd78158dc0 2024-02-21T23:09:53.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.473+0000 7fcd77e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fcd780c6220 0x7fcd781586b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36992/0 (socket says 172.21.15.28:36992) 2024-02-21T23:09:53.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.473+0000 7fcd77e00640 1 -- 172.21.15.28:0/865484629 learned_addr learned my addr 172.21.15.28:0/865484629 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:09:53.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.474+0000 7fcd7d400640 1 --2- 172.21.15.28:0/865484629 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcd78158dc0 0x7fcd781609f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:53.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.474+0000 7fcd77400640 1 -- 172.21.15.28:0/865484629 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3051500048 0 0) 0x7fcd780c20d0 con 0x7fcd780c6220 2024-02-21T23:09:53.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.474+0000 7fcd77400640 1 -- 172.21.15.28:0/865484629 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd5c003520 con 0x7fcd780c6220 2024-02-21T23:09:53.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.474+0000 7fcd77400640 1 -- 172.21.15.28:0/865484629 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4286241750 0 0) 0x7fcd5c003520 con 0x7fcd780c6220 2024-02-21T23:09:53.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.474+0000 7fcd77400640 1 -- 172.21.15.28:0/865484629 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcd78158dc0 msgr2=0x7fcd781609f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:53.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.474+0000 7fcd77400640 1 --2- 172.21.15.28:0/865484629 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcd78158dc0 0x7fcd781609f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:53.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.474+0000 7fcd77400640 1 -- 172.21.15.28:0/865484629 >> v1:172.21.15.81:6789/0 conn(0x7fcd780c4960 legacy=0x7fcd781524f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:53.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.474+0000 7fcd77400640 1 -- 172.21.15.28:0/865484629 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd780c1570 con 0x7fcd780c6220 2024-02-21T23:09:53.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.474+0000 7fcd77400640 1 -- 172.21.15.28:0/865484629 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fcd64001670 con 0x7fcd780c6220 2024-02-21T23:09:53.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.474+0000 7fcd77400640 1 -- 172.21.15.28:0/865484629 >> v1:172.21.15.28:6789/0 conn(0x7fcd780c6220 legacy=0x7fcd781586b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:53.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.474+0000 7fcd77400640 1 --2- 172.21.15.28:0/865484629 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcd78158dc0 0x7fcd5c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:53.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.474+0000 7fcd77400640 1 --2- 172.21.15.28:0/865484629 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcd5c003f60 0x7fcd5c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:53.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.474+0000 7fcd77400640 1 --2- 172.21.15.28:0/865484629 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd5c004d40 0x7fcd5c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:53.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.474+0000 7fcd77400640 1 -- 172.21.15.28:0/865484629 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd780a62e0 con 0x7fcd5c004d40 2024-02-21T23:09:53.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.474+0000 7fcd77400640 1 -- 172.21.15.28:0/865484629 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd5c00bb50 con 0x7fcd5c003f60 2024-02-21T23:09:53.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.474+0000 7fcd77400640 1 -- 172.21.15.28:0/865484629 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd5c00bd50 con 0x7fcd78158dc0 2024-02-21T23:09:53.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.475+0000 7fcd7d400640 1 --2- 172.21.15.28:0/865484629 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd5c004d40 0x7fcd5c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:53.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.475+0000 7fcd77e00640 1 --2- 172.21.15.28:0/865484629 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcd5c003f60 0x7fcd5c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:53.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.475+0000 7fcd7ca00640 1 --2- 172.21.15.28:0/865484629 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcd78158dc0 0x7fcd5c0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:53.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.475+0000 7fcd7d400640 1 -- 172.21.15.28:0/865484629 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcd78158dc0 msgr2=0x7fcd5c0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:53.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.475+0000 7fcd7d400640 1 --2- 172.21.15.28:0/865484629 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcd78158dc0 0x7fcd5c0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:53.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.475+0000 7fcd7d400640 1 -- 172.21.15.28:0/865484629 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcd5c003f60 msgr2=0x7fcd5c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:53.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.475+0000 7fcd7d400640 1 --2- 172.21.15.28:0/865484629 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcd5c003f60 0x7fcd5c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:53.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.475+0000 7fcd7d400640 1 -- 172.21.15.28:0/865484629 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd5c003520 con 0x7fcd5c004d40 2024-02-21T23:09:53.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.476+0000 7fcd7d400640 1 --2- 172.21.15.28:0/865484629 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd5c004d40 0x7fcd5c0051c0 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7fcd70009fc0 tx=0x7fcd5c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cd2021ccbcad931 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:53.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.477+0000 7fcd77400640 1 -- 172.21.15.28:0/865484629 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd70039070 con 0x7fcd5c004d40 2024-02-21T23:09:53.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.477+0000 7fcd77400640 1 -- 172.21.15.28:0/865484629 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd7002cc20 con 0x7fcd5c004d40 2024-02-21T23:09:53.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.477+0000 7fcd77400640 1 -- 172.21.15.28:0/865484629 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd70009c10 con 0x7fcd5c004d40 2024-02-21T23:09:53.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.477+0000 7fcd7f400640 1 -- 172.21.15.28:0/865484629 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd5c004d40 msgr2=0x7fcd5c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:53.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.477+0000 7fcd7f400640 1 --2- 172.21.15.28:0/865484629 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd5c004d40 0x7fcd5c0051c0 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7fcd70009fc0 tx=0x7fcd5c005830 comp rx=0 tx=0).stop 2024-02-21T23:09:53.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.477+0000 7fcd7d400640 1 -- 172.21.15.28:0/865484629 reap_dead start 2024-02-21T23:09:53.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.477+0000 7fcd7f400640 1 -- 172.21.15.28:0/865484629 shutdown_connections 2024-02-21T23:09:53.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.477+0000 7fcd7f400640 1 -- 172.21.15.28:0/865484629 >> 172.21.15.28:0/865484629 conn(0x7fcd780a85a0 msgr2=0x7fcd780a8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:53.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.478+0000 7fcd7f400640 1 -- 172.21.15.28:0/865484629 shutdown_connections 2024-02-21T23:09:53.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.478+0000 7fcd7f400640 1 -- 172.21.15.28:0/865484629 wait complete. 2024-02-21T23:09:53.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.479+0000 7fcd7f400640 1 Processor -- start 2024-02-21T23:09:53.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.479+0000 7fcd7f400640 1 -- start start 2024-02-21T23:09:53.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.480+0000 7fcd7f400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcd5c004d40 0x7fcd781ee9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:53.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.480+0000 7fcd7f400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcd781eb450 0x7fcd780aa060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:53.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.480+0000 7fcd7f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd78072b70 0x7fcd78149560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:53.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.480+0000 7fcd7f400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd78159250 con 0x7fcd78072b70 2024-02-21T23:09:53.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.480+0000 7fcd7f400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd780c1960 con 0x7fcd781eb450 2024-02-21T23:09:53.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.480+0000 7fcd7f400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd780a5f20 con 0x7fcd5c004d40 2024-02-21T23:09:53.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.481+0000 7fcd7d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd78072b70 0x7fcd78149560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:53.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.481+0000 7fcd7d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd78072b70 0x7fcd78149560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56082/0 (socket says 172.21.15.28:56082) 2024-02-21T23:09:53.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.481+0000 7fcd7d400640 1 -- 172.21.15.28:0/1323524168 learned_addr learned my addr 172.21.15.28:0/1323524168 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:53.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.481+0000 7fcd77e00640 1 --2- 172.21.15.28:0/1323524168 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcd781eb450 0x7fcd780aa060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:53.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.481+0000 7fcd7ca00640 1 --2- 172.21.15.28:0/1323524168 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcd5c004d40 0x7fcd781ee9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:53.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.481+0000 7fcd7d400640 1 -- 172.21.15.28:0/1323524168 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcd5c004d40 msgr2=0x7fcd781ee9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:53.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.481+0000 7fcd7d400640 1 --2- 172.21.15.28:0/1323524168 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcd5c004d40 0x7fcd781ee9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:53.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.481+0000 7fcd7d400640 1 -- 172.21.15.28:0/1323524168 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcd781eb450 msgr2=0x7fcd780aa060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:53.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.481+0000 7fcd7d400640 1 --2- 172.21.15.28:0/1323524168 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcd781eb450 0x7fcd780aa060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:53.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.481+0000 7fcd7d400640 1 -- 172.21.15.28:0/1323524168 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd780aa5a0 con 0x7fcd78072b70 2024-02-21T23:09:53.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.481+0000 7fcd7d400640 1 --2- 172.21.15.28:0/1323524168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd78072b70 0x7fcd78149560 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7fcd70002230 tx=0x7fcd70009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:53.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.482+0000 7fcd74c00640 1 -- 172.21.15.28:0/1323524168 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd70039040 con 0x7fcd78072b70 2024-02-21T23:09:53.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.482+0000 7fcd74c00640 1 -- 172.21.15.28:0/1323524168 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd70046070 con 0x7fcd78072b70 2024-02-21T23:09:53.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.482+0000 7fcd7f400640 1 -- 172.21.15.28:0/1323524168 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd78157cb0 con 0x7fcd78072b70 2024-02-21T23:09:53.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.482+0000 7fcd74c00640 1 -- 172.21.15.28:0/1323524168 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd70035d50 con 0x7fcd78072b70 2024-02-21T23:09:53.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.482+0000 7fcd7f400640 1 -- 172.21.15.28:0/1323524168 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd78158270 con 0x7fcd78072b70 2024-02-21T23:09:53.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.483+0000 7fcd7f400640 1 -- 172.21.15.28:0/1323524168 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd78067f30 con 0x7fcd78072b70 2024-02-21T23:09:53.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.485+0000 7fcd74c00640 1 -- 172.21.15.28:0/1323524168 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcd7003f470 con 0x7fcd78072b70 2024-02-21T23:09:53.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.486+0000 7fcd74c00640 1 --2- 172.21.15.28:0/1323524168 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcd5806b500 0x7fcd5806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:53.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.486+0000 7fcd74c00640 1 -- 172.21.15.28:0/1323524168 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6001+0+0 (secure 0 0 0) 0x7fcd700acb20 con 0x7fcd78072b70 2024-02-21T23:09:53.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.487+0000 7fcd7ca00640 1 --2- 172.21.15.28:0/1323524168 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcd5806b500 0x7fcd5806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:53.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.488+0000 7fcd7ca00640 1 --2- 172.21.15.28:0/1323524168 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcd5806b500 0x7fcd5806d9c0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fcd68007fd0 tx=0x7fcd68009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:53.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.491+0000 7fcd74c00640 1 -- 172.21.15.28:0/1323524168 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fcd7007b980 con 0x7fcd78072b70 2024-02-21T23:09:53.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:53.728+0000 7fcd7f400640 1 -- 172.21.15.28:0/1323524168 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fcd781593d0 con 0x7fcd78072b70 2024-02-21T23:09:54.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.259+0000 7fcd74c00640 1 -- 172.21.15.28:0/1323524168 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 23 and data pool 24 v154) v1 ==== 174+0+0 (secure 0 0 0) 0x7fcd7007b280 con 0x7fcd78072b70 2024-02-21T23:09:54.249 INFO:teuthology.orchestra.run.smithi028.stderr:new fs with metadata pool 23 and data pool 24 2024-02-21T23:09:54.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.270+0000 7fcd7f400640 1 -- 172.21.15.28:0/1323524168 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcd5806b500 msgr2=0x7fcd5806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:54.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.270+0000 7fcd7f400640 1 --2- 172.21.15.28:0/1323524168 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcd5806b500 0x7fcd5806d9c0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fcd68007fd0 tx=0x7fcd68009040 comp rx=0 tx=0).stop 2024-02-21T23:09:54.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.270+0000 7fcd7f400640 1 -- 172.21.15.28:0/1323524168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd78072b70 msgr2=0x7fcd78149560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:54.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.270+0000 7fcd7f400640 1 --2- 172.21.15.28:0/1323524168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd78072b70 0x7fcd78149560 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7fcd70002230 tx=0x7fcd70009750 comp rx=0 tx=0).stop 2024-02-21T23:09:54.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.271+0000 7fcd7f400640 1 -- 172.21.15.28:0/1323524168 shutdown_connections 2024-02-21T23:09:54.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.271+0000 7fcd7f400640 1 --2- 172.21.15.28:0/1323524168 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcd5806b500 0x7fcd5806d9c0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:54.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.271+0000 7fcd7f400640 1 --2- 172.21.15.28:0/1323524168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd78072b70 0x7fcd78149560 unknown :-1 s=CLOSED pgs=1669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:54.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.271+0000 7fcd7f400640 1 --2- 172.21.15.28:0/1323524168 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcd781eb450 0x7fcd780aa060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:54.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.271+0000 7fcd7f400640 1 --2- 172.21.15.28:0/1323524168 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcd5c004d40 0x7fcd781ee9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:54.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.271+0000 7fcd7f400640 1 -- 172.21.15.28:0/1323524168 >> 172.21.15.28:0/1323524168 conn(0x7fcd780a85a0 msgr2=0x7fcd781522c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:54.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.271+0000 7fcd7f400640 1 -- 172.21.15.28:0/1323524168 shutdown_connections 2024-02-21T23:09:54.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.271+0000 7fcd7f400640 1 -- 172.21.15.28:0/1323524168 wait complete. 2024-02-21T23:09:54.297 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:09:54.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.489+0000 7fe897e00640 1 Processor -- start 2024-02-21T23:09:54.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.489+0000 7fe897e00640 1 -- start start 2024-02-21T23:09:54.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.490+0000 7fe897e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fe8980a5fd0 0x7fe89806dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:54.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.490+0000 7fe897e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8980c2760 con 0x7fe898150450 2024-02-21T23:09:54.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.491+0000 7fe897e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe898156310 con 0x7fe89806ee40 2024-02-21T23:09:54.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.491+0000 7fe897e00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8980c22c0 con 0x7fe8980a5fd0 2024-02-21T23:09:54.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.491+0000 7fe896a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe898150450 0x7fe8981536f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37008/0 (socket says 172.21.15.28:37008) 2024-02-21T23:09:54.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.491+0000 7fe896a00640 1 -- 172.21.15.28:0/1220202119 learned_addr learned my addr 172.21.15.28:0/1220202119 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:09:54.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.491+0000 7fe897400640 1 --2- 172.21.15.28:0/1220202119 >> v2:172.21.15.81:3300/0 conn(0x7fe8980a5fd0 0x7fe89806dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:54.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.492+0000 7fe896000640 1 -- 172.21.15.28:0/1220202119 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2164189856 0 0) 0x7fe8980c2760 con 0x7fe898150450 2024-02-21T23:09:54.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.492+0000 7fe896000640 1 -- 172.21.15.28:0/1220202119 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe87c003520 con 0x7fe898150450 2024-02-21T23:09:54.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.492+0000 7fe896000640 1 -- 172.21.15.28:0/1220202119 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 966449727 0 0) 0x7fe87c003520 con 0x7fe898150450 2024-02-21T23:09:54.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.492+0000 7fe896000640 1 -- 172.21.15.28:0/1220202119 >> v2:172.21.15.81:3300/0 conn(0x7fe8980a5fd0 msgr2=0x7fe89806dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:54.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.492+0000 7fe896000640 1 --2- 172.21.15.28:0/1220202119 >> v2:172.21.15.81:3300/0 conn(0x7fe8980a5fd0 0x7fe89806dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:54.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.492+0000 7fe896000640 1 -- 172.21.15.28:0/1220202119 >> v1:172.21.15.81:6789/0 conn(0x7fe89806ee40 legacy=0x7fe898156af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:54.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.492+0000 7fe896000640 1 -- 172.21.15.28:0/1220202119 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8980c29d0 con 0x7fe898150450 2024-02-21T23:09:54.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.492+0000 7fe896000640 1 -- 172.21.15.28:0/1220202119 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fe8880019f0 con 0x7fe898150450 2024-02-21T23:09:54.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.492+0000 7fe896000640 1 -- 172.21.15.28:0/1220202119 >> v1:172.21.15.28:6789/0 conn(0x7fe898150450 legacy=0x7fe8981536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:54.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.492+0000 7fe896000640 1 --2- 172.21.15.28:0/1220202119 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe87c003a00 0x7fe87c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:54.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.492+0000 7fe896000640 1 --2- 172.21.15.28:0/1220202119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe87c004800 0x7fe87c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:54.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.492+0000 7fe896000640 1 --2- 172.21.15.28:0/1220202119 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe87c0051c0 0x7fe87c00fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:54.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.492+0000 7fe896000640 1 -- 172.21.15.28:0/1220202119 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe87c0104a0 con 0x7fe87c004800 2024-02-21T23:09:54.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.492+0000 7fe896000640 1 -- 172.21.15.28:0/1220202119 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe87c010650 con 0x7fe87c0051c0 2024-02-21T23:09:54.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.492+0000 7fe896000640 1 -- 172.21.15.28:0/1220202119 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe87c010850 con 0x7fe87c003a00 2024-02-21T23:09:54.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.492+0000 7fe88e000640 1 --2- 172.21.15.28:0/1220202119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe87c004800 0x7fe87c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:54.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.493+0000 7fe896a00640 1 --2- 172.21.15.28:0/1220202119 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe87c003a00 0x7fe87c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:54.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.493+0000 7fe897400640 1 --2- 172.21.15.28:0/1220202119 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe87c0051c0 0x7fe87c00fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:54.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.493+0000 7fe88e000640 1 -- 172.21.15.28:0/1220202119 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe87c003a00 msgr2=0x7fe87c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:54.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.493+0000 7fe88e000640 1 --2- 172.21.15.28:0/1220202119 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe87c003a00 0x7fe87c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:54.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.493+0000 7fe88e000640 1 -- 172.21.15.28:0/1220202119 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe87c0051c0 msgr2=0x7fe87c00fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:54.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.493+0000 7fe88e000640 1 --2- 172.21.15.28:0/1220202119 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe87c0051c0 0x7fe87c00fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:54.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.493+0000 7fe88e000640 1 -- 172.21.15.28:0/1220202119 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe87c003520 con 0x7fe87c004800 2024-02-21T23:09:54.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.493+0000 7fe897400640 1 -- 172.21.15.28:0/1220202119 reap_dead start 2024-02-21T23:09:54.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.494+0000 7fe88e000640 1 --2- 172.21.15.28:0/1220202119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe87c004800 0x7fe87c004c80 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7fe880001750 tx=0x7fe87c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89137d2818affdb6 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:54.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.494+0000 7fe896000640 1 -- 172.21.15.28:0/1220202119 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe880001370 con 0x7fe87c004800 2024-02-21T23:09:54.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.494+0000 7fe896000640 1 -- 172.21.15.28:0/1220202119 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8800052d0 con 0x7fe87c004800 2024-02-21T23:09:54.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.494+0000 7fe896000640 1 -- 172.21.15.28:0/1220202119 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe880003830 con 0x7fe87c004800 2024-02-21T23:09:54.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.495+0000 7fe897e00640 1 -- 172.21.15.28:0/1220202119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe87c004800 msgr2=0x7fe87c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:54.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.495+0000 7fe897e00640 1 --2- 172.21.15.28:0/1220202119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe87c004800 0x7fe87c004c80 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7fe880001750 tx=0x7fe87c005830 comp rx=0 tx=0).stop 2024-02-21T23:09:54.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.495+0000 7fe897e00640 1 -- 172.21.15.28:0/1220202119 shutdown_connections 2024-02-21T23:09:54.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.495+0000 7fe897e00640 1 --2- 172.21.15.28:0/1220202119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe87c004800 0x7fe87c004c80 unknown :-1 s=CLOSED pgs=1671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:54.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.495+0000 7fe897e00640 1 -- 172.21.15.28:0/1220202119 >> 172.21.15.28:0/1220202119 conn(0x7fe89806b630 msgr2=0x7fe8980c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:54.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.495+0000 7fe897e00640 1 -- 172.21.15.28:0/1220202119 shutdown_connections 2024-02-21T23:09:54.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.495+0000 7fe897e00640 1 -- 172.21.15.28:0/1220202119 wait complete. 2024-02-21T23:09:54.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.497+0000 7fe897e00640 1 Processor -- start 2024-02-21T23:09:54.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.497+0000 7fe897e00640 1 -- start start 2024-02-21T23:09:54.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.498+0000 7fe897e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe87c004800 0x7fe8981e6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:54.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.498+0000 7fe897e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8980c60b0 0x7fe8980c4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:54.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.498+0000 7fe897e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe898151df0 0x7fe898155a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:54.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.498+0000 7fe897e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8981536f0 con 0x7fe8980c60b0 2024-02-21T23:09:54.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.498+0000 7fe897e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe898156660 con 0x7fe898151df0 2024-02-21T23:09:54.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.498+0000 7fe897e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe898153870 con 0x7fe87c004800 2024-02-21T23:09:54.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.498+0000 7fe88e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8980c60b0 0x7fe8980c4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:54.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.498+0000 7fe88e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8980c60b0 0x7fe8980c4d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56102/0 (socket says 172.21.15.28:56102) 2024-02-21T23:09:54.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.498+0000 7fe88e000640 1 -- 172.21.15.28:0/3680795687 learned_addr learned my addr 172.21.15.28:0/3680795687 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:54.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.498+0000 7fe896a00640 1 --2- 172.21.15.28:0/3680795687 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe87c004800 0x7fe8981e6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:54.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.498+0000 7fe897400640 1 --2- 172.21.15.28:0/3680795687 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe898151df0 0x7fe898155a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:54.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.499+0000 7fe88e000640 1 -- 172.21.15.28:0/3680795687 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe87c004800 msgr2=0x7fe8981e6380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:54.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.499+0000 7fe88e000640 1 --2- 172.21.15.28:0/3680795687 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe87c004800 0x7fe8981e6380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:54.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.499+0000 7fe88e000640 1 -- 172.21.15.28:0/3680795687 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe898151df0 msgr2=0x7fe898155a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:54.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.499+0000 7fe88e000640 1 --2- 172.21.15.28:0/3680795687 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe898151df0 0x7fe898155a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:54.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.499+0000 7fe88e000640 1 -- 172.21.15.28:0/3680795687 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe898144d30 con 0x7fe8980c60b0 2024-02-21T23:09:54.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.499+0000 7fe88e000640 1 --2- 172.21.15.28:0/3680795687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8980c60b0 0x7fe8980c4d50 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7fe880005040 tx=0x7fe880001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:54.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.499+0000 7fe88fe00640 1 -- 172.21.15.28:0/3680795687 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe880041070 con 0x7fe8980c60b0 2024-02-21T23:09:54.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.499+0000 7fe88fe00640 1 -- 172.21.15.28:0/3680795687 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe880005480 con 0x7fe8980c60b0 2024-02-21T23:09:54.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.499+0000 7fe88fe00640 1 -- 172.21.15.28:0/3680795687 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe88003d640 con 0x7fe8980c60b0 2024-02-21T23:09:54.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.499+0000 7fe897e00640 1 -- 172.21.15.28:0/3680795687 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe898144890 con 0x7fe8980c60b0 2024-02-21T23:09:54.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.500+0000 7fe897e00640 1 -- 172.21.15.28:0/3680795687 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe898154410 con 0x7fe8980c60b0 2024-02-21T23:09:54.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.501+0000 7fe897e00640 1 -- 172.21.15.28:0/3680795687 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe89804f290 con 0x7fe8980c60b0 2024-02-21T23:09:54.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.502+0000 7fe88fe00640 1 -- 172.21.15.28:0/3680795687 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe880044070 con 0x7fe8980c60b0 2024-02-21T23:09:54.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.503+0000 7fe88fe00640 1 --2- 172.21.15.28:0/3680795687 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe87406b4d0 0x7fe87406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:54.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.503+0000 7fe88fe00640 1 -- 172.21.15.28:0/3680795687 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe8800abdb0 con 0x7fe8980c60b0 2024-02-21T23:09:54.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.503+0000 7fe896a00640 1 --2- 172.21.15.28:0/3680795687 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe87406b4d0 0x7fe87406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:54.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.504+0000 7fe896a00640 1 --2- 172.21.15.28:0/3680795687 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe87406b4d0 0x7fe87406d990 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fe888002230 tx=0x7fe888002020 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:54.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.507+0000 7fe88fe00640 1 -- 172.21.15.28:0/3680795687 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe88007a460 con 0x7fe8980c60b0 2024-02-21T23:09:54.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.706+0000 7fe897e00640 1 -- 172.21.15.28:0/3680795687 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe8980c1f10 con 0x7fe8980c60b0 2024-02-21T23:09:54.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.709+0000 7fe88fe00640 1 -- 172.21.15.28:0/3680795687 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v150) v1 ==== 74+0+16837 (secure 0 0 0) 0x7fe88007e500 con 0x7fe8980c60b0 2024-02-21T23:09:54.697 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:09:54.697 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":150,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:09:53.729831+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:09:51.634291+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"150","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-02-21T23:09:52.736252+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"150","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:09:54.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.711+0000 7fe897e00640 1 -- 172.21.15.28:0/3680795687 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe87406b4d0 msgr2=0x7fe87406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:54.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.711+0000 7fe897e00640 1 --2- 172.21.15.28:0/3680795687 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe87406b4d0 0x7fe87406d990 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fe888002230 tx=0x7fe888002020 comp rx=0 tx=0).stop 2024-02-21T23:09:54.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.711+0000 7fe897e00640 1 -- 172.21.15.28:0/3680795687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8980c60b0 msgr2=0x7fe8980c4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:54.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.711+0000 7fe897e00640 1 --2- 172.21.15.28:0/3680795687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8980c60b0 0x7fe8980c4d50 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7fe880005040 tx=0x7fe880001120 comp rx=0 tx=0).stop 2024-02-21T23:09:54.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.711+0000 7fe897e00640 1 -- 172.21.15.28:0/3680795687 shutdown_connections 2024-02-21T23:09:54.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.712+0000 7fe897e00640 1 --2- 172.21.15.28:0/3680795687 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe87406b4d0 0x7fe87406d990 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:54.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.712+0000 7fe897e00640 1 --2- 172.21.15.28:0/3680795687 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe898151df0 0x7fe898155a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:54.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.712+0000 7fe897e00640 1 --2- 172.21.15.28:0/3680795687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8980c60b0 0x7fe8980c4d50 unknown :-1 s=CLOSED pgs=1672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:54.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.712+0000 7fe897e00640 1 --2- 172.21.15.28:0/3680795687 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe87c004800 0x7fe8981e6380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:54.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.712+0000 7fe897e00640 1 -- 172.21.15.28:0/3680795687 >> 172.21.15.28:0/3680795687 conn(0x7fe89806b630 msgr2=0x7fe8980c2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:54.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.712+0000 7fe897e00640 1 -- 172.21.15.28:0/3680795687 shutdown_connections 2024-02-21T23:09:54.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.712+0000 7fe897e00640 1 -- 172.21.15.28:0/3680795687 wait complete. 2024-02-21T23:09:54.718 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:09:54.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.951+0000 7febcba00640 1 Processor -- start 2024-02-21T23:09:54.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.952+0000 7febcba00640 1 -- start start 2024-02-21T23:09:54.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.953+0000 7febcba00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7febc41526b0 0x7febc41559c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:54.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.953+0000 7febcba00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7febc40ca940 0x7febc40cadc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:54.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.953+0000 7febcba00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febc40c6f30 con 0x7febc40c76f0 2024-02-21T23:09:54.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.953+0000 7febcba00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febc40c61b0 con 0x7febc40ca940 2024-02-21T23:09:54.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.953+0000 7febcba00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febc40c6450 con 0x7febc41526b0 2024-02-21T23:09:54.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.954+0000 7febc9a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7febc40c76f0 0x7febc40c7b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34302/0 (socket says 172.21.15.28:34302) 2024-02-21T23:09:54.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.954+0000 7febbbe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7febc40ca940 0x7febc40cadc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:54.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.954+0000 7febc9a00640 1 -- 172.21.15.28:0/2329946150 learned_addr learned my addr 172.21.15.28:0/2329946150 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:09:54.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.954+0000 7febc9000640 1 --2- 172.21.15.28:0/2329946150 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7febc41526b0 0x7febc41559c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:54.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.954+0000 7febbbe00640 1 -- 172.21.15.28:0/2329946150 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7febc41526b0 msgr2=0x7febc41559c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:54.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.954+0000 7febbbe00640 1 --2- 172.21.15.28:0/2329946150 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7febc41526b0 0x7febc41559c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:54.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.954+0000 7febbbe00640 1 -- 172.21.15.28:0/2329946150 >> v1:172.21.15.28:6789/0 conn(0x7febc40c76f0 legacy=0x7febc40c7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:54.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.954+0000 7febbbe00640 1 -- 172.21.15.28:0/2329946150 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febc40c8290 con 0x7febc40ca940 2024-02-21T23:09:54.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.954+0000 7febc3e00640 1 -- 172.21.15.28:0/2329946150 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2163745038 0 0) 0x7febc40c6f30 con 0x7febc40c76f0 2024-02-21T23:09:54.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.955+0000 7febbbe00640 1 --2- 172.21.15.28:0/2329946150 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febc40ca940 0x7febc40cadc0 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7febac0097a0 tx=0x7febac02b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f5f28106303c0c1 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:54.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.956+0000 7febc3e00640 1 -- 172.21.15.28:0/2329946150 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febac031780 con 0x7febc40ca940 2024-02-21T23:09:54.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.956+0000 7febc3e00640 1 -- 172.21.15.28:0/2329946150 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febac037070 con 0x7febc40ca940 2024-02-21T23:09:54.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.956+0000 7febc3e00640 1 -- 172.21.15.28:0/2329946150 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febac031b40 con 0x7febc40ca940 2024-02-21T23:09:54.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.956+0000 7febcba00640 1 -- 172.21.15.28:0/2329946150 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febc40ca940 msgr2=0x7febc40cadc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:54.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.956+0000 7febcba00640 1 --2- 172.21.15.28:0/2329946150 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febc40ca940 0x7febc40cadc0 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7febac0097a0 tx=0x7febac02b820 comp rx=0 tx=0).stop 2024-02-21T23:09:54.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.956+0000 7febcba00640 1 -- 172.21.15.28:0/2329946150 shutdown_connections 2024-02-21T23:09:54.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.956+0000 7febcba00640 1 --2- 172.21.15.28:0/2329946150 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febc40ca940 0x7febc40cadc0 unknown :-1 s=CLOSED pgs=1674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:54.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.956+0000 7febcba00640 1 --2- 172.21.15.28:0/2329946150 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7febc41526b0 0x7febc41559c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:54.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.956+0000 7febcba00640 1 -- 172.21.15.28:0/2329946150 >> 172.21.15.28:0/2329946150 conn(0x7febc406d3d0 msgr2=0x7febc408afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:54.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.956+0000 7febcba00640 1 -- 172.21.15.28:0/2329946150 shutdown_connections 2024-02-21T23:09:54.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.956+0000 7febcba00640 1 -- 172.21.15.28:0/2329946150 wait complete. 2024-02-21T23:09:54.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.958+0000 7febcba00640 1 Processor -- start 2024-02-21T23:09:54.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.958+0000 7febcba00640 1 -- start start 2024-02-21T23:09:54.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.959+0000 7febcba00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febc40c76f0 0x7febc4156330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:54.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.959+0000 7febcba00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7febc40ca940 0x7febc414d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:54.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.959+0000 7febcba00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7febc41526b0 0x7febc41ee530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:54.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.959+0000 7febcba00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febc40c71d0 con 0x7febc40c76f0 2024-02-21T23:09:54.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.959+0000 7febcba00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febc40c69f0 con 0x7febc40ca940 2024-02-21T23:09:54.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.959+0000 7febcba00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febc40c5c90 con 0x7febc41526b0 2024-02-21T23:09:54.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.959+0000 7febc9000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febc40c76f0 0x7febc4156330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:54.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.960+0000 7febc9000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febc40c76f0 0x7febc4156330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33714/0 (socket says 172.21.15.28:33714) 2024-02-21T23:09:54.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.960+0000 7febc9000640 1 -- 172.21.15.28:0/216920021 learned_addr learned my addr 172.21.15.28:0/216920021 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:54.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.960+0000 7febc9a00640 1 --2- 172.21.15.28:0/216920021 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7febc41526b0 0x7febc41ee530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:54.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.960+0000 7febbbe00640 1 --2- 172.21.15.28:0/216920021 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7febc40ca940 0x7febc414d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:54.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.960+0000 7febc9000640 1 -- 172.21.15.28:0/216920021 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7febc41526b0 msgr2=0x7febc41ee530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:54.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.960+0000 7febc9000640 1 --2- 172.21.15.28:0/216920021 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7febc41526b0 0x7febc41ee530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:54.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.960+0000 7febc9000640 1 -- 172.21.15.28:0/216920021 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7febc40ca940 msgr2=0x7febc414d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:54.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.960+0000 7febc9000640 1 --2- 172.21.15.28:0/216920021 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7febc40ca940 0x7febc414d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:54.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.960+0000 7febc9000640 1 -- 172.21.15.28:0/216920021 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febc40c9a00 con 0x7febc40c76f0 2024-02-21T23:09:54.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.960+0000 7febc9a00640 1 --2- 172.21.15.28:0/216920021 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7febc41526b0 0x7febc41ee530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:09:54.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.960+0000 7febbbe00640 1 --2- 172.21.15.28:0/216920021 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7febc40ca940 0x7febc414d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:09:54.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.961+0000 7febc9000640 1 --2- 172.21.15.28:0/216920021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febc40c76f0 0x7febc4156330 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7febb400d830 tx=0x7febb400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:54.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.961+0000 7febc1600640 1 -- 172.21.15.28:0/216920021 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febb400b6c0 con 0x7febc40c76f0 2024-02-21T23:09:54.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.961+0000 7febc1600640 1 -- 172.21.15.28:0/216920021 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febb4019070 con 0x7febc40c76f0 2024-02-21T23:09:54.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.961+0000 7febcba00640 1 -- 172.21.15.28:0/216920021 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febc40c9ce0 con 0x7febc40c76f0 2024-02-21T23:09:54.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.961+0000 7febc1600640 1 -- 172.21.15.28:0/216920021 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febb400ba80 con 0x7febc40c76f0 2024-02-21T23:09:54.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.961+0000 7febcba00640 1 -- 172.21.15.28:0/216920021 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febc41edd30 con 0x7febc40c76f0 2024-02-21T23:09:54.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.962+0000 7febcba00640 1 -- 172.21.15.28:0/216920021 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febc4067f30 con 0x7febc40c76f0 2024-02-21T23:09:54.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.963+0000 7febc1600640 1 -- 172.21.15.28:0/216920021 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7febb401e460 con 0x7febc40c76f0 2024-02-21T23:09:54.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.964+0000 7febc1600640 1 --2- 172.21.15.28:0/216920021 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7feba006b4d0 0x7feba006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:54.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.964+0000 7febc1600640 1 -- 172.21.15.28:0/216920021 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6101+0+0 (secure 0 0 0) 0x7febb408a9b0 con 0x7febc40c76f0 2024-02-21T23:09:54.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.965+0000 7febbbe00640 1 --2- 172.21.15.28:0/216920021 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7feba006b4d0 0x7feba006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:54.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.966+0000 7febbbe00640 1 --2- 172.21.15.28:0/216920021 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7feba006b4d0 0x7feba006d990 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7febac036000 tx=0x7febac008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:54.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:54.969+0000 7febc1600640 1 -- 172.21.15.28:0/216920021 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7febb4059730 con 0x7febc40c76f0 2024-02-21T23:09:55.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.169+0000 7febcba00640 1 -- 172.21.15.28:0/216920021 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7febc40c5f20 con 0x7febc40c76f0 2024-02-21T23:09:55.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.172+0000 7febc1600640 1 -- 172.21.15.28:0/216920021 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v150) v1 ==== 74+0+16837 (secure 0 0 0) 0x7febb4059030 con 0x7febc40c76f0 2024-02-21T23:09:55.161 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:09:55.161 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":150,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:09:53.729831+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:09:51.634291+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"150","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-02-21T23:09:52.736252+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"150","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:09:55.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.175+0000 7febcba00640 1 -- 172.21.15.28:0/216920021 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7feba006b4d0 msgr2=0x7feba006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:55.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.175+0000 7febcba00640 1 --2- 172.21.15.28:0/216920021 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7feba006b4d0 0x7feba006d990 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7febac036000 tx=0x7febac008030 comp rx=0 tx=0).stop 2024-02-21T23:09:55.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.175+0000 7febcba00640 1 -- 172.21.15.28:0/216920021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febc40c76f0 msgr2=0x7febc4156330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:55.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.175+0000 7febcba00640 1 --2- 172.21.15.28:0/216920021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febc40c76f0 0x7febc4156330 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7febb400d830 tx=0x7febb400dcf0 comp rx=0 tx=0).stop 2024-02-21T23:09:55.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.175+0000 7febcba00640 1 -- 172.21.15.28:0/216920021 shutdown_connections 2024-02-21T23:09:55.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.175+0000 7febcba00640 1 --2- 172.21.15.28:0/216920021 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7feba006b4d0 0x7feba006d990 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:55.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.175+0000 7febcba00640 1 --2- 172.21.15.28:0/216920021 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7febc41526b0 0x7febc41ee530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:55.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.175+0000 7febcba00640 1 --2- 172.21.15.28:0/216920021 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7febc40ca940 0x7febc414d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:55.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.175+0000 7febcba00640 1 --2- 172.21.15.28:0/216920021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febc40c76f0 0x7febc4156330 unknown :-1 s=CLOSED pgs=1675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:55.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.175+0000 7febcba00640 1 -- 172.21.15.28:0/216920021 >> 172.21.15.28:0/216920021 conn(0x7febc406d3d0 msgr2=0x7febc408afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:55.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.175+0000 7febcba00640 1 -- 172.21.15.28:0/216920021 shutdown_connections 2024-02-21T23:09:55.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.175+0000 7febcba00640 1 -- 172.21.15.28:0/216920021 wait complete. 2024-02-21T23:09:55.184 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-02-21T23:09:55.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.418+0000 7ff3b5200640 1 Processor -- start 2024-02-21T23:09:55.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.418+0000 7ff3b5200640 1 -- start start 2024-02-21T23:09:55.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.419+0000 7ff3b5200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7ff3b006daa0 0x7ff3b00c8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:55.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.419+0000 7ff3b5200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff3b01550e0 0x7ff3b0155530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:55.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.419+0000 7ff3b5200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3b0158920 con 0x7ff3b0152650 2024-02-21T23:09:55.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.419+0000 7ff3b5200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3b01586d0 con 0x7ff3b006daa0 2024-02-21T23:09:55.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.419+0000 7ff3b5200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b0158c00 con 0x7ff3b01550e0 2024-02-21T23:09:55.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.420+0000 7ff3af400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff3b0152650 0x7ff3b00c7160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34316/0 (socket says 172.21.15.28:34316) 2024-02-21T23:09:55.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.420+0000 7ff3af400640 1 -- 172.21.15.28:0/2524669858 learned_addr learned my addr 172.21.15.28:0/2524669858 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:09:55.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.420+0000 7ff3aea00640 1 --2- 172.21.15.28:0/2524669858 >> v2:172.21.15.81:3300/0 conn(0x7ff3b006daa0 0x7ff3b00c8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:55.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.420+0000 7ff3ae000640 1 -- 172.21.15.28:0/2524669858 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2961800758 0 0) 0x7ff3b0158920 con 0x7ff3b0152650 2024-02-21T23:09:55.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.420+0000 7ff3afe00640 1 --2- 172.21.15.28:0/2524669858 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff3b01550e0 0x7ff3b0155530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:55.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.420+0000 7ff3ae000640 1 -- 172.21.15.28:0/2524669858 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff394003520 con 0x7ff3b0152650 2024-02-21T23:09:55.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.421+0000 7ff3ae000640 1 -- 172.21.15.28:0/2524669858 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1286931828 0 0) 0x7ff394003520 con 0x7ff3b0152650 2024-02-21T23:09:55.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.421+0000 7ff3ae000640 1 -- 172.21.15.28:0/2524669858 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff3b01550e0 msgr2=0x7ff3b0155530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:55.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.421+0000 7ff3ae000640 1 --2- 172.21.15.28:0/2524669858 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff3b01550e0 0x7ff3b0155530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:55.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.421+0000 7ff3ae000640 1 -- 172.21.15.28:0/2524669858 >> v2:172.21.15.81:3300/0 conn(0x7ff3b006daa0 msgr2=0x7ff3b00c8ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:55.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.421+0000 7ff3ae000640 1 --2- 172.21.15.28:0/2524669858 >> v2:172.21.15.81:3300/0 conn(0x7ff3b006daa0 0x7ff3b00c8ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:55.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.421+0000 7ff3ae000640 1 -- 172.21.15.28:0/2524669858 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3b0158ee0 con 0x7ff3b0152650 2024-02-21T23:09:55.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.421+0000 7ff3ae000640 1 -- 172.21.15.28:0/2524669858 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7ff3a0001670 con 0x7ff3b0152650 2024-02-21T23:09:55.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.421+0000 7ff3ae000640 1 -- 172.21.15.28:0/2524669858 >> v1:172.21.15.28:6789/0 conn(0x7ff3b0152650 legacy=0x7ff3b00c7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:55.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.421+0000 7ff3ae000640 1 --2- 172.21.15.28:0/2524669858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3940039b0 0x7ff394003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:55.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.421+0000 7ff3ae000640 1 --2- 172.21.15.28:0/2524669858 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff3b01550e0 0x7ff394004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:55.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.421+0000 7ff3ae000640 1 --2- 172.21.15.28:0/2524669858 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff394004d40 0x7ff3940051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:55.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.421+0000 7ff3ae000640 1 -- 172.21.15.28:0/2524669858 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b01586d0 con 0x7ff3940039b0 2024-02-21T23:09:55.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.421+0000 7ff3ae000640 1 -- 172.21.15.28:0/2524669858 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b0158c00 con 0x7ff394004d40 2024-02-21T23:09:55.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.421+0000 7ff3ae000640 1 -- 172.21.15.28:0/2524669858 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff39400b9f0 con 0x7ff3b01550e0 2024-02-21T23:09:55.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.421+0000 7ff3af400640 1 --2- 172.21.15.28:0/2524669858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3940039b0 0x7ff394003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:55.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.422+0000 7ff3aea00640 1 --2- 172.21.15.28:0/2524669858 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff3b01550e0 0x7ff394004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:55.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.422+0000 7ff3af400640 1 -- 172.21.15.28:0/2524669858 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff3b01550e0 msgr2=0x7ff394004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:55.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.422+0000 7ff3af400640 1 --2- 172.21.15.28:0/2524669858 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff3b01550e0 0x7ff394004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:55.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.422+0000 7ff3af400640 1 -- 172.21.15.28:0/2524669858 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff394004d40 msgr2=0x7ff3940051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:55.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.422+0000 7ff3afe00640 1 --2- 172.21.15.28:0/2524669858 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff394004d40 0x7ff3940051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:55.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.422+0000 7ff3af400640 1 --2- 172.21.15.28:0/2524669858 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff394004d40 0x7ff3940051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:55.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.422+0000 7ff3af400640 1 -- 172.21.15.28:0/2524669858 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff394003520 con 0x7ff3940039b0 2024-02-21T23:09:55.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.422+0000 7ff3aea00640 1 --2- 172.21.15.28:0/2524669858 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff3b01550e0 0x7ff394004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:09:55.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.423+0000 7ff3af400640 1 --2- 172.21.15.28:0/2524669858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3940039b0 0x7ff394003dd0 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7ff3a000c190 tx=0x7ff394005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50dbc3035452830d server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:55.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.423+0000 7ff3ae000640 1 -- 172.21.15.28:0/2524669858 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a0007c60 con 0x7ff3940039b0 2024-02-21T23:09:55.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.423+0000 7ff3ae000640 1 -- 172.21.15.28:0/2524669858 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3a0030070 con 0x7ff3940039b0 2024-02-21T23:09:55.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.423+0000 7ff3ae000640 1 -- 172.21.15.28:0/2524669858 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a0005d30 con 0x7ff3940039b0 2024-02-21T23:09:55.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.423+0000 7ff3b5200640 1 -- 172.21.15.28:0/2524669858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3940039b0 msgr2=0x7ff394003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:55.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.423+0000 7ff3b5200640 1 --2- 172.21.15.28:0/2524669858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3940039b0 0x7ff394003dd0 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7ff3a000c190 tx=0x7ff394005830 comp rx=0 tx=0).stop 2024-02-21T23:09:55.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.424+0000 7ff3afe00640 1 -- 172.21.15.28:0/2524669858 reap_dead start 2024-02-21T23:09:55.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.424+0000 7ff3b5200640 1 -- 172.21.15.28:0/2524669858 shutdown_connections 2024-02-21T23:09:55.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.424+0000 7ff3b5200640 1 -- 172.21.15.28:0/2524669858 >> 172.21.15.28:0/2524669858 conn(0x7ff3b0072ce0 msgr2=0x7ff3b00ac340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:55.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.424+0000 7ff3b5200640 1 -- 172.21.15.28:0/2524669858 shutdown_connections 2024-02-21T23:09:55.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.424+0000 7ff3b5200640 1 -- 172.21.15.28:0/2524669858 wait complete. 2024-02-21T23:09:55.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.427+0000 7ff3b5200640 1 Processor -- start 2024-02-21T23:09:55.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.427+0000 7ff3b5200640 1 -- start start 2024-02-21T23:09:55.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.429+0000 7ff3b5200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3940039b0 0x7ff3b01516e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:55.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.429+0000 7ff3b5200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff3b00c72f0 0x7ff3b00abc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:55.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.429+0000 7ff3b5200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3b0148130 0x7ff3b00ac1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:55.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.429+0000 7ff3b5200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b0152a30 con 0x7ff3b0148130 2024-02-21T23:09:55.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.429+0000 7ff3b5200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b00bdfb0 con 0x7ff3940039b0 2024-02-21T23:09:55.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.429+0000 7ff3b5200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b00c7870 con 0x7ff3b00c72f0 2024-02-21T23:09:55.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.429+0000 7ff3afe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3b0148130 0x7ff3b00ac1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:55.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.429+0000 7ff3afe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3b0148130 0x7ff3b00ac1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33740/0 (socket says 172.21.15.28:33740) 2024-02-21T23:09:55.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.429+0000 7ff3afe00640 1 -- 172.21.15.28:0/3151985231 learned_addr learned my addr 172.21.15.28:0/3151985231 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:55.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.429+0000 7ff3aea00640 1 --2- 172.21.15.28:0/3151985231 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff3b00c72f0 0x7ff3b00abc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:55.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.429+0000 7ff3af400640 1 --2- 172.21.15.28:0/3151985231 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3940039b0 0x7ff3b01516e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:55.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.430+0000 7ff3afe00640 1 -- 172.21.15.28:0/3151985231 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff3b00c72f0 msgr2=0x7ff3b00abc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:55.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.430+0000 7ff3afe00640 1 --2- 172.21.15.28:0/3151985231 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff3b00c72f0 0x7ff3b00abc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:55.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.430+0000 7ff3afe00640 1 -- 172.21.15.28:0/3151985231 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3940039b0 msgr2=0x7ff3b01516e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:55.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.430+0000 7ff3afe00640 1 --2- 172.21.15.28:0/3151985231 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3940039b0 0x7ff3b01516e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:55.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.430+0000 7ff3afe00640 1 -- 172.21.15.28:0/3151985231 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3b014f990 con 0x7ff3b0148130 2024-02-21T23:09:55.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.430+0000 7ff3af400640 1 --2- 172.21.15.28:0/3151985231 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3940039b0 0x7ff3b01516e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:09:55.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.430+0000 7ff3afe00640 1 --2- 172.21.15.28:0/3151985231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3b0148130 0x7ff3b00ac1b0 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7ff3b006d820 tx=0x7ff3a800f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:55.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.430+0000 7ff3a7400640 1 -- 172.21.15.28:0/3151985231 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a801c070 con 0x7ff3b0148130 2024-02-21T23:09:55.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.430+0000 7ff3a7400640 1 -- 172.21.15.28:0/3151985231 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3a8017400 con 0x7ff3b0148130 2024-02-21T23:09:55.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.430+0000 7ff3a7400640 1 -- 172.21.15.28:0/3151985231 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a80177f0 con 0x7ff3b0148130 2024-02-21T23:09:55.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.430+0000 7ff3b5200640 1 -- 172.21.15.28:0/3151985231 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3b0147be0 con 0x7ff3b0148130 2024-02-21T23:09:55.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.431+0000 7ff3b5200640 1 -- 172.21.15.28:0/3151985231 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3b014fc50 con 0x7ff3b0148130 2024-02-21T23:09:55.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.432+0000 7ff3b5200640 1 -- 172.21.15.28:0/3151985231 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3b004f290 con 0x7ff3b0148130 2024-02-21T23:09:55.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.433+0000 7ff3a7400640 1 -- 172.21.15.28:0/3151985231 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff3a8005d10 con 0x7ff3b0148130 2024-02-21T23:09:55.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.434+0000 7ff3a7400640 1 --2- 172.21.15.28:0/3151985231 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff39006b4d0 0x7ff39006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:55.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.434+0000 7ff3a7400640 1 -- 172.21.15.28:0/3151985231 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff3a808a540 con 0x7ff3b0148130 2024-02-21T23:09:55.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.435+0000 7ff3af400640 1 --2- 172.21.15.28:0/3151985231 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff39006b4d0 0x7ff39006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:55.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.436+0000 7ff3af400640 1 --2- 172.21.15.28:0/3151985231 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff39006b4d0 0x7ff39006d990 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7ff3a0034000 tx=0x7ff3a0007000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:55.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.438+0000 7ff3a7400640 1 -- 172.21.15.28:0/3151985231 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ff3a8012070 con 0x7ff3b0148130 2024-02-21T23:09:55.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:55.720+0000 7ff3b5200640 1 -- 172.21.15.28:0/3151985231 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7ff3b014fe20 con 0x7ff3b0148130 2024-02-21T23:09:56.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.270+0000 7ff3a7400640 1 -- 172.21.15.28:0/3151985231 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v157) v1 ==== 118+0+0 (secure 0 0 0) 0x7ff3a8059230 con 0x7ff3b0148130 2024-02-21T23:09:56.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.274+0000 7ff3b5200640 1 -- 172.21.15.28:0/3151985231 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff39006b4d0 msgr2=0x7ff39006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:56.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.274+0000 7ff3b5200640 1 --2- 172.21.15.28:0/3151985231 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff39006b4d0 0x7ff39006d990 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7ff3a0034000 tx=0x7ff3a0007000 comp rx=0 tx=0).stop 2024-02-21T23:09:56.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.274+0000 7ff3b5200640 1 -- 172.21.15.28:0/3151985231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3b0148130 msgr2=0x7ff3b00ac1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:56.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.274+0000 7ff3b5200640 1 --2- 172.21.15.28:0/3151985231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3b0148130 0x7ff3b00ac1b0 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7ff3b006d820 tx=0x7ff3a800f4a0 comp rx=0 tx=0).stop 2024-02-21T23:09:56.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.274+0000 7ff3b5200640 1 -- 172.21.15.28:0/3151985231 shutdown_connections 2024-02-21T23:09:56.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.274+0000 7ff3b5200640 1 --2- 172.21.15.28:0/3151985231 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff39006b4d0 0x7ff39006d990 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:56.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.274+0000 7ff3b5200640 1 --2- 172.21.15.28:0/3151985231 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff3b00c72f0 0x7ff3b00abc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:56.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.274+0000 7ff3b5200640 1 --2- 172.21.15.28:0/3151985231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3b0148130 0x7ff3b00ac1b0 unknown :-1 s=CLOSED pgs=1678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:56.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.274+0000 7ff3b5200640 1 --2- 172.21.15.28:0/3151985231 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3940039b0 0x7ff3b01516e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:56.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.274+0000 7ff3b5200640 1 -- 172.21.15.28:0/3151985231 >> 172.21.15.28:0/3151985231 conn(0x7ff3b0072ce0 msgr2=0x7ff3b00c9bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:56.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.274+0000 7ff3b5200640 1 -- 172.21.15.28:0/3151985231 shutdown_connections 2024-02-21T23:09:56.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.274+0000 7ff3b5200640 1 -- 172.21.15.28:0/3151985231 wait complete. 2024-02-21T23:09:56.287 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-02-21T23:09:56.287 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-02-21T23:09:56.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.552+0000 7f70dfe00640 1 Processor -- start 2024-02-21T23:09:56.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.553+0000 7f70dfe00640 1 -- start start 2024-02-21T23:09:56.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.554+0000 7f70dfe00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f70e013e520 0x7f70e006c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:56.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.554+0000 7f70dfe00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70e0158550 con 0x7f70e013e0c0 2024-02-21T23:09:56.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.554+0000 7f70dfe00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70e01587d0 con 0x7f70e006dbc0 2024-02-21T23:09:56.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.554+0000 7f70dfe00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70e00c89a0 con 0x7f70e013e520 2024-02-21T23:09:56.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.555+0000 7f70dea00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f70e013e0c0 0x7f70e00c71b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34324/0 (socket says 172.21.15.28:34324) 2024-02-21T23:09:56.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.555+0000 7f70dea00640 1 -- 172.21.15.28:0/939832337 learned_addr learned my addr 172.21.15.28:0/939832337 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:09:56.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.555+0000 7f70df400640 1 --2- 172.21.15.28:0/939832337 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f70e013e520 0x7f70e006c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:56.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.555+0000 7f70de000640 1 -- 172.21.15.28:0/939832337 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 922371323 0 0) 0x7f70d0001110 con 0x7f70e013e0c0 2024-02-21T23:09:56.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.555+0000 7f70de000640 1 -- 172.21.15.28:0/939832337 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70c4003520 con 0x7f70e013e0c0 2024-02-21T23:09:56.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.556+0000 7f70de000640 1 -- 172.21.15.28:0/939832337 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1957670525 0 0) 0x7f70c4003520 con 0x7f70e013e0c0 2024-02-21T23:09:56.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.556+0000 7f70de000640 1 -- 172.21.15.28:0/939832337 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f70e013e520 msgr2=0x7f70e006c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:56.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.556+0000 7f70de000640 1 --2- 172.21.15.28:0/939832337 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f70e013e520 0x7f70e006c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:56.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.556+0000 7f70de000640 1 -- 172.21.15.28:0/939832337 >> v1:172.21.15.81:6789/0 conn(0x7f70e006dbc0 legacy=0x7f70e006c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:56.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.556+0000 7f70de000640 1 -- 172.21.15.28:0/939832337 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70e0158ab0 con 0x7f70e013e0c0 2024-02-21T23:09:56.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.556+0000 7f70de000640 1 -- 172.21.15.28:0/939832337 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f70d0001840 con 0x7f70e013e0c0 2024-02-21T23:09:56.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.556+0000 7f70de000640 1 -- 172.21.15.28:0/939832337 >> v1:172.21.15.28:6789/0 conn(0x7f70e013e0c0 legacy=0x7f70e00c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:56.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.556+0000 7f70de000640 1 --2- 172.21.15.28:0/939832337 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f70e013e520 0x7f70c40039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:56.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.556+0000 7f70de000640 1 --2- 172.21.15.28:0/939832337 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70c4003f20 0x7f70c4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:56.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.556+0000 7f70de000640 1 --2- 172.21.15.28:0/939832337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70c4004d40 0x7f70c40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:56.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.556+0000 7f70de000640 1 -- 172.21.15.28:0/939832337 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70c400bad0 con 0x7f70c4004d40 2024-02-21T23:09:56.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.556+0000 7f70de000640 1 -- 172.21.15.28:0/939832337 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70c400bc80 con 0x7f70c4003f20 2024-02-21T23:09:56.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.556+0000 7f70de000640 1 -- 172.21.15.28:0/939832337 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70c400be80 con 0x7f70e013e520 2024-02-21T23:09:56.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.557+0000 7f70df400640 1 --2- 172.21.15.28:0/939832337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70c4004d40 0x7f70c40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:56.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.557+0000 7f70dea00640 1 --2- 172.21.15.28:0/939832337 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f70e013e520 0x7f70c40039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:56.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.557+0000 7f70d7e00640 1 --2- 172.21.15.28:0/939832337 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70c4003f20 0x7f70c4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:56.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.557+0000 7f70df400640 1 -- 172.21.15.28:0/939832337 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f70e013e520 msgr2=0x7f70c40039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:56.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.557+0000 7f70df400640 1 --2- 172.21.15.28:0/939832337 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f70e013e520 0x7f70c40039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:56.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.557+0000 7f70df400640 1 -- 172.21.15.28:0/939832337 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70c4003f20 msgr2=0x7f70c4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:56.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.557+0000 7f70df400640 1 --2- 172.21.15.28:0/939832337 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70c4003f20 0x7f70c4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:56.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.557+0000 7f70df400640 1 -- 172.21.15.28:0/939832337 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70c4003520 con 0x7f70c4004d40 2024-02-21T23:09:56.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.557+0000 7f70dea00640 1 --2- 172.21.15.28:0/939832337 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f70e013e520 0x7f70c40039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:09:56.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.558+0000 7f70df400640 1 --2- 172.21.15.28:0/939832337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70c4004d40 0x7f70c40051c0 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7f70d8009fc0 tx=0x7f70c4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1a6e6460217c9c6 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:56.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.559+0000 7f70de000640 1 -- 172.21.15.28:0/939832337 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70d8039070 con 0x7f70c4004d40 2024-02-21T23:09:56.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.559+0000 7f70de000640 1 -- 172.21.15.28:0/939832337 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70d802cbf0 con 0x7f70c4004d40 2024-02-21T23:09:56.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.559+0000 7f70de000640 1 -- 172.21.15.28:0/939832337 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70d8009c10 con 0x7f70c4004d40 2024-02-21T23:09:56.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.559+0000 7f70dfe00640 1 -- 172.21.15.28:0/939832337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70c4004d40 msgr2=0x7f70c40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:56.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.559+0000 7f70dfe00640 1 --2- 172.21.15.28:0/939832337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70c4004d40 0x7f70c40051c0 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7f70d8009fc0 tx=0x7f70c4005830 comp rx=0 tx=0).stop 2024-02-21T23:09:56.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.559+0000 7f70df400640 1 -- 172.21.15.28:0/939832337 reap_dead start 2024-02-21T23:09:56.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.559+0000 7f70dfe00640 1 -- 172.21.15.28:0/939832337 shutdown_connections 2024-02-21T23:09:56.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.559+0000 7f70dfe00640 1 -- 172.21.15.28:0/939832337 >> 172.21.15.28:0/939832337 conn(0x7f70e0073110 msgr2=0x7f70e00ac490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:56.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.560+0000 7f70dfe00640 1 -- 172.21.15.28:0/939832337 shutdown_connections 2024-02-21T23:09:56.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.560+0000 7f70dfe00640 1 -- 172.21.15.28:0/939832337 wait complete. 2024-02-21T23:09:56.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.561+0000 7f70dfe00640 1 Processor -- start 2024-02-21T23:09:56.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.561+0000 7f70dfe00640 1 -- start start 2024-02-21T23:09:56.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.562+0000 7f70dfe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70c4004d40 0x7f70e01edd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:56.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.562+0000 7f70dfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70e00aa830 0x7f70e00ae240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:56.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.562+0000 7f70dfe00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f70e00ae780 0x7f70e0138f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:56.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.562+0000 7f70dfe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70e00c78c0 con 0x7f70e00aa830 2024-02-21T23:09:56.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.562+0000 7f70dfe00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70e0159070 con 0x7f70c4004d40 2024-02-21T23:09:56.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.562+0000 7f70dfe00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70e00c7a40 con 0x7f70e00ae780 2024-02-21T23:09:56.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.563+0000 7f70d7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70e00aa830 0x7f70e00ae240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:56.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.563+0000 7f70d7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70e00aa830 0x7f70e00ae240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33758/0 (socket says 172.21.15.28:33758) 2024-02-21T23:09:56.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.563+0000 7f70d7e00640 1 -- 172.21.15.28:0/1044622937 learned_addr learned my addr 172.21.15.28:0/1044622937 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:56.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.563+0000 7f70dea00640 1 --2- 172.21.15.28:0/1044622937 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70c4004d40 0x7f70e01edd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:56.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.563+0000 7f70df400640 1 --2- 172.21.15.28:0/1044622937 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f70e00ae780 0x7f70e0138f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:56.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.563+0000 7f70d7e00640 1 -- 172.21.15.28:0/1044622937 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f70e00ae780 msgr2=0x7f70e0138f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:56.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.563+0000 7f70d7e00640 1 --2- 172.21.15.28:0/1044622937 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f70e00ae780 0x7f70e0138f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:56.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.563+0000 7f70d7e00640 1 -- 172.21.15.28:0/1044622937 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70c4004d40 msgr2=0x7f70e01edd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:56.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.563+0000 7f70d7e00640 1 --2- 172.21.15.28:0/1044622937 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70c4004d40 0x7f70e01edd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:56.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.563+0000 7f70d7e00640 1 -- 172.21.15.28:0/1044622937 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70e00ad780 con 0x7f70e00aa830 2024-02-21T23:09:56.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.563+0000 7f70d7e00640 1 --2- 172.21.15.28:0/1044622937 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70e00aa830 0x7f70e00ae240 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f70c800e5d0 tx=0x7f70c800ea90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:56.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.564+0000 7f70d6a00640 1 -- 172.21.15.28:0/1044622937 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70c80053a0 con 0x7f70e00aa830 2024-02-21T23:09:56.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.564+0000 7f70d6a00640 1 -- 172.21.15.28:0/1044622937 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70c8018070 con 0x7f70e00aa830 2024-02-21T23:09:56.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.564+0000 7f70d6a00640 1 -- 172.21.15.28:0/1044622937 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70c8003830 con 0x7f70e00aa830 2024-02-21T23:09:56.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.564+0000 7f70dfe00640 1 -- 172.21.15.28:0/1044622937 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70e0138010 con 0x7f70e00aa830 2024-02-21T23:09:56.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.564+0000 7f70dfe00640 1 -- 172.21.15.28:0/1044622937 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70e01ee580 con 0x7f70e00aa830 2024-02-21T23:09:56.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.565+0000 7f70dfe00640 1 -- 172.21.15.28:0/1044622937 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70e004f290 con 0x7f70e00aa830 2024-02-21T23:09:56.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.566+0000 7f70d6a00640 1 -- 172.21.15.28:0/1044622937 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f70c80039d0 con 0x7f70e00aa830 2024-02-21T23:09:56.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.567+0000 7f70d6a00640 1 --2- 172.21.15.28:0/1044622937 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f70bc06b500 0x7f70bc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:56.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.567+0000 7f70d6a00640 1 -- 172.21.15.28:0/1044622937 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f70c808a200 con 0x7f70e00aa830 2024-02-21T23:09:56.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.568+0000 7f70dea00640 1 --2- 172.21.15.28:0/1044622937 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f70bc06b500 0x7f70bc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:56.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.569+0000 7f70dea00640 1 --2- 172.21.15.28:0/1044622937 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f70bc06b500 0x7f70bc06d9c0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f70d0000ff0 tx=0x7f70d0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:56.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.571+0000 7f70d6a00640 1 -- 172.21.15.28:0/1044622937 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f70c8058e50 con 0x7f70e00aa830 2024-02-21T23:09:56.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:56.809+0000 7f70dfe00640 1 -- 172.21.15.28:0/1044622937 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f70e006d9a0 con 0x7f70e00aa830 2024-02-21T23:09:57.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.280+0000 7f70d6a00640 1 -- 172.21.15.28:0/1044622937 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v158) v1 ==== 122+0+0 (secure 0 0 0) 0x7f70c8058750 con 0x7f70e00aa830 2024-02-21T23:09:57.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.283+0000 7f70dfe00640 1 -- 172.21.15.28:0/1044622937 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f70bc06b500 msgr2=0x7f70bc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:57.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.283+0000 7f70dfe00640 1 --2- 172.21.15.28:0/1044622937 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f70bc06b500 0x7f70bc06d9c0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f70d0000ff0 tx=0x7f70d0009040 comp rx=0 tx=0).stop 2024-02-21T23:09:57.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.283+0000 7f70dfe00640 1 -- 172.21.15.28:0/1044622937 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70e00aa830 msgr2=0x7f70e00ae240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:57.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.283+0000 7f70dfe00640 1 --2- 172.21.15.28:0/1044622937 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70e00aa830 0x7f70e00ae240 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f70c800e5d0 tx=0x7f70c800ea90 comp rx=0 tx=0).stop 2024-02-21T23:09:57.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.283+0000 7f70dfe00640 1 -- 172.21.15.28:0/1044622937 shutdown_connections 2024-02-21T23:09:57.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.283+0000 7f70dfe00640 1 --2- 172.21.15.28:0/1044622937 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f70bc06b500 0x7f70bc06d9c0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:57.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.283+0000 7f70dfe00640 1 --2- 172.21.15.28:0/1044622937 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f70e00ae780 0x7f70e0138f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:57.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.283+0000 7f70dfe00640 1 --2- 172.21.15.28:0/1044622937 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70e00aa830 0x7f70e00ae240 unknown :-1 s=CLOSED pgs=1681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:57.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.283+0000 7f70dfe00640 1 --2- 172.21.15.28:0/1044622937 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70c4004d40 0x7f70e01edd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:57.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.283+0000 7f70dfe00640 1 -- 172.21.15.28:0/1044622937 >> 172.21.15.28:0/1044622937 conn(0x7f70e0073110 msgr2=0x7f70e01519a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:57.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.284+0000 7f70dfe00640 1 -- 172.21.15.28:0/1044622937 shutdown_connections 2024-02-21T23:09:57.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.284+0000 7f70dfe00640 1 -- 172.21.15.28:0/1044622937 wait complete. 2024-02-21T23:09:57.288 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-02-21T23:09:57.288 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:09:57.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.502+0000 7f1f4f400640 1 Processor -- start 2024-02-21T23:09:57.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.502+0000 7f1f4f400640 1 -- start start 2024-02-21T23:09:57.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.504+0000 7f1f4f400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1f480c6460 0x7f1f480ca8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:57.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.504+0000 7f1f4f400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1f480cadf0 0x7f1f480bf990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:57.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.504+0000 7f1f4f400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f481551b0 con 0x7f1f4813e620 2024-02-21T23:09:57.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.504+0000 7f1f4f400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f480c71d0 con 0x7f1f480cadf0 2024-02-21T23:09:57.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.504+0000 7f1f4f400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f48155420 con 0x7f1f480c6460 2024-02-21T23:09:57.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.505+0000 7f1f4ca00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f1f4813e620 0x7f1f48159070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34326/0 (socket says 172.21.15.28:34326) 2024-02-21T23:09:57.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.505+0000 7f1f4ca00640 1 -- 172.21.15.28:0/812282337 learned_addr learned my addr 172.21.15.28:0/812282337 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:09:57.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.505+0000 7f1f4d400640 1 --2- 172.21.15.28:0/812282337 >> v2:172.21.15.28:3300/0 conn(0x7f1f480cadf0 0x7f1f480bf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:57.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.505+0000 7f1f47e00640 1 --2- 172.21.15.28:0/812282337 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1f480c6460 0x7f1f480ca8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:57.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.505+0000 7f1f47400640 1 -- 172.21.15.28:0/812282337 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 921283564 0 0) 0x7f1f481551b0 con 0x7f1f4813e620 2024-02-21T23:09:57.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.505+0000 7f1f47400640 1 -- 172.21.15.28:0/812282337 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f30003520 con 0x7f1f4813e620 2024-02-21T23:09:57.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.505+0000 7f1f4d400640 1 -- 172.21.15.28:0/812282337 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1f480c6460 msgr2=0x7f1f480ca8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:57.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.505+0000 7f1f4d400640 1 --2- 172.21.15.28:0/812282337 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1f480c6460 0x7f1f480ca8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:57.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.505+0000 7f1f4d400640 1 -- 172.21.15.28:0/812282337 >> v1:172.21.15.28:6789/0 conn(0x7f1f4813e620 legacy=0x7f1f48159070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:57.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.505+0000 7f1f4d400640 1 -- 172.21.15.28:0/812282337 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f48155700 con 0x7f1f480cadf0 2024-02-21T23:09:57.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.507+0000 7f1f4d400640 1 --2- 172.21.15.28:0/812282337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f480cadf0 0x7f1f480bf990 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7f1f40009d90 tx=0x7f1f4002be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d46bd87297085f2 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:57.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.507+0000 7f1f47400640 1 -- 172.21.15.28:0/812282337 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f40039070 con 0x7f1f480cadf0 2024-02-21T23:09:57.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.507+0000 7f1f47400640 1 -- 172.21.15.28:0/812282337 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f4002cbc0 con 0x7f1f480cadf0 2024-02-21T23:09:57.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.507+0000 7f1f47400640 1 -- 172.21.15.28:0/812282337 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f40034440 con 0x7f1f480cadf0 2024-02-21T23:09:57.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.507+0000 7f1f4f400640 1 -- 172.21.15.28:0/812282337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f480cadf0 msgr2=0x7f1f480bf990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:57.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.507+0000 7f1f4f400640 1 --2- 172.21.15.28:0/812282337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f480cadf0 0x7f1f480bf990 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7f1f40009d90 tx=0x7f1f4002be10 comp rx=0 tx=0).stop 2024-02-21T23:09:57.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.508+0000 7f1f4f400640 1 -- 172.21.15.28:0/812282337 shutdown_connections 2024-02-21T23:09:57.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.508+0000 7f1f4f400640 1 --2- 172.21.15.28:0/812282337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f480cadf0 0x7f1f480bf990 unknown :-1 s=CLOSED pgs=1683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:57.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.508+0000 7f1f4f400640 1 --2- 172.21.15.28:0/812282337 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1f480c6460 0x7f1f480ca8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:57.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.508+0000 7f1f4f400640 1 -- 172.21.15.28:0/812282337 >> 172.21.15.28:0/812282337 conn(0x7f1f4808b1c0 msgr2=0x7f1f480725a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:57.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.508+0000 7f1f4f400640 1 -- 172.21.15.28:0/812282337 shutdown_connections 2024-02-21T23:09:57.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.508+0000 7f1f4f400640 1 -- 172.21.15.28:0/812282337 wait complete. 2024-02-21T23:09:57.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.509+0000 7f1f4f400640 1 Processor -- start 2024-02-21T23:09:57.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.510+0000 7f1f4f400640 1 -- start start 2024-02-21T23:09:57.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.510+0000 7f1f4f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f480c6460 0x7f1f48156440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:57.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.510+0000 7f1f4f400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1f480cadf0 0x7f1f480c4030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:57.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.510+0000 7f1f4f400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1f4813e620 0x7f1f480c9410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:57.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.511+0000 7f1f4f400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f4813ea00 con 0x7f1f480c6460 2024-02-21T23:09:57.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.511+0000 7f1f4f400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f48155cc0 con 0x7f1f4813e620 2024-02-21T23:09:57.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.511+0000 7f1f4f400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f480bfed0 con 0x7f1f480cadf0 2024-02-21T23:09:57.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.511+0000 7f1f4ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f480c6460 0x7f1f48156440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:57.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.511+0000 7f1f4ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f480c6460 0x7f1f48156440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33766/0 (socket says 172.21.15.28:33766) 2024-02-21T23:09:57.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.511+0000 7f1f4ca00640 1 -- 172.21.15.28:0/920384925 learned_addr learned my addr 172.21.15.28:0/920384925 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:57.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.511+0000 7f1f4ca00640 1 -- 172.21.15.28:0/920384925 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1f480cadf0 msgr2=0x7f1f480c4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:57.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.511+0000 7f1f4ca00640 1 --2- 172.21.15.28:0/920384925 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1f480cadf0 0x7f1f480c4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:57.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.511+0000 7f1f4ca00640 1 -- 172.21.15.28:0/920384925 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1f4813e620 msgr2=0x7f1f480c9410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:57.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.511+0000 7f1f4ca00640 1 --2- 172.21.15.28:0/920384925 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1f4813e620 0x7f1f480c9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:57.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.511+0000 7f1f4ca00640 1 -- 172.21.15.28:0/920384925 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f481592f0 con 0x7f1f480c6460 2024-02-21T23:09:57.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.511+0000 7f1f4ca00640 1 --2- 172.21.15.28:0/920384925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f480c6460 0x7f1f48156440 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c00c3c0 tx=0x7f1f3c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:57.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.512+0000 7f1f44c00640 1 -- 172.21.15.28:0/920384925 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f3c017070 con 0x7f1f480c6460 2024-02-21T23:09:57.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.512+0000 7f1f44c00640 1 -- 172.21.15.28:0/920384925 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f3c005ce0 con 0x7f1f480c6460 2024-02-21T23:09:57.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.512+0000 7f1f44c00640 1 -- 172.21.15.28:0/920384925 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f3c0072f0 con 0x7f1f480c6460 2024-02-21T23:09:57.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.512+0000 7f1f4f400640 1 -- 172.21.15.28:0/920384925 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f480c8930 con 0x7f1f480c6460 2024-02-21T23:09:57.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.512+0000 7f1f4f400640 1 -- 172.21.15.28:0/920384925 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f480c8ef0 con 0x7f1f480c6460 2024-02-21T23:09:57.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.513+0000 7f1f4f400640 1 -- 172.21.15.28:0/920384925 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f480c2850 con 0x7f1f480c6460 2024-02-21T23:09:57.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.515+0000 7f1f44c00640 1 -- 172.21.15.28:0/920384925 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1f3c01a070 con 0x7f1f480c6460 2024-02-21T23:09:57.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.516+0000 7f1f44c00640 1 --2- 172.21.15.28:0/920384925 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1f2806b4d0 0x7f1f2806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:57.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.516+0000 7f1f44c00640 1 -- 172.21.15.28:0/920384925 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1f3c0897d0 con 0x7f1f480c6460 2024-02-21T23:09:57.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.516+0000 7f1f47e00640 1 --2- 172.21.15.28:0/920384925 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1f2806b4d0 0x7f1f2806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:57.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.518+0000 7f1f47e00640 1 --2- 172.21.15.28:0/920384925 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1f2806b4d0 0x7f1f2806d990 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f1f34006a90 tx=0x7f1f34006a20 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:57.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.521+0000 7f1f44c00640 1 -- 172.21.15.28:0/920384925 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1f480c2850 con 0x7f1f480c6460 2024-02-21T23:09:57.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.773+0000 7f1f4f400640 1 -- 172.21.15.28:0/920384925 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f480c90f0 con 0x7f1f480c6460 2024-02-21T23:09:57.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.775+0000 7f1f44c00640 1 -- 172.21.15.28:0/920384925 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 158 v158) v1 ==== 95+0+4945 (secure 0 0 0) 0x7f1f480c90f0 con 0x7f1f480c6460 2024-02-21T23:09:57.762 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:09:57.763 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":158,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6389,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1320495906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1320495906},{"type":"v1","addr":"172.21.15.81:6835","nonce":1320495906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":153},{"gid":6392,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/521625945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":521625945},{"type":"v1","addr":"172.21.15.81:6837","nonce":521625945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":153},{"gid":8817,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/2975378719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2975378719},{"type":"v1","addr":"172.21.15.28:6833","nonce":2975378719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":152}],"filesystems":[{"mdsmap":{"epoch":158,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:09:53.730296+0000","modified":"2024-02-21T23:09:56.809990+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8826},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8826":{"gid":8826,"name":"a","rank":0,"incarnation":154,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/2928639298","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2928639298},{"type":"v1","addr":"172.21.15.28:6835","nonce":2928639298}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2024-02-21T23:09:57.763 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 158 2024-02-21T23:09:57.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.778+0000 7f1f4f400640 1 -- 172.21.15.28:0/920384925 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1f2806b4d0 msgr2=0x7f1f2806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:57.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.778+0000 7f1f4f400640 1 --2- 172.21.15.28:0/920384925 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1f2806b4d0 0x7f1f2806d990 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f1f34006a90 tx=0x7f1f34006a20 comp rx=0 tx=0).stop 2024-02-21T23:09:57.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.778+0000 7f1f4f400640 1 -- 172.21.15.28:0/920384925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f480c6460 msgr2=0x7f1f48156440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:57.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.778+0000 7f1f4f400640 1 --2- 172.21.15.28:0/920384925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f480c6460 0x7f1f48156440 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c00c3c0 tx=0x7f1f3c00c880 comp rx=0 tx=0).stop 2024-02-21T23:09:57.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.778+0000 7f1f4f400640 1 -- 172.21.15.28:0/920384925 shutdown_connections 2024-02-21T23:09:57.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.778+0000 7f1f4f400640 1 --2- 172.21.15.28:0/920384925 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1f2806b4d0 0x7f1f2806d990 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:57.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.778+0000 7f1f4f400640 1 --2- 172.21.15.28:0/920384925 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1f480cadf0 0x7f1f480c4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:57.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.778+0000 7f1f4f400640 1 --2- 172.21.15.28:0/920384925 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1f4813e620 0x7f1f480c9410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:57.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.778+0000 7f1f4f400640 1 --2- 172.21.15.28:0/920384925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f480c6460 0x7f1f48156440 unknown :-1 s=CLOSED pgs=1684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:57.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.778+0000 7f1f4f400640 1 -- 172.21.15.28:0/920384925 >> 172.21.15.28:0/920384925 conn(0x7f1f4808b1c0 msgr2=0x7f1f480c7bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:57.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.778+0000 7f1f4f400640 1 -- 172.21.15.28:0/920384925 shutdown_connections 2024-02-21T23:09:57.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.778+0000 7f1f4f400640 1 -- 172.21.15.28:0/920384925 wait complete. 2024-02-21T23:09:57.783 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:09:57.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.995+0000 7f3933c00640 1 Processor -- start 2024-02-21T23:09:57.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.995+0000 7f3933c00640 1 -- start start 2024-02-21T23:09:57.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.997+0000 7f3933c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f392c13e620 0x7f392c159070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:57.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.997+0000 7f3933c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f392c0bf990 0x7f392c0bfe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:57.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.997+0000 7f3933c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f392c1551b0 con 0x7f392c0c6460 2024-02-21T23:09:57.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.997+0000 7f3933c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f392c0c71d0 con 0x7f392c0bf990 2024-02-21T23:09:57.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.997+0000 7f3933c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f392c1595b0 con 0x7f392c13e620 2024-02-21T23:09:57.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.997+0000 7f3923e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f392c0c6460 0x7f392c0ca8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34340/0 (socket says 172.21.15.28:34340) 2024-02-21T23:09:57.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.997+0000 7f3923e00640 1 -- 172.21.15.28:0/3287262936 learned_addr learned my addr 172.21.15.28:0/3287262936 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:09:57.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.997+0000 7f3931c00640 1 --2- 172.21.15.28:0/3287262936 >> v2:172.21.15.28:3300/0 conn(0x7f392c0bf990 0x7f392c0bfe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:57.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.998+0000 7f3931200640 1 --2- 172.21.15.28:0/3287262936 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f392c13e620 0x7f392c159070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:57.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.998+0000 7f392be00640 1 -- 172.21.15.28:0/3287262936 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 179223526 0 0) 0x7f392c1551b0 con 0x7f392c0c6460 2024-02-21T23:09:57.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.998+0000 7f392be00640 1 -- 172.21.15.28:0/3287262936 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3910003520 con 0x7f392c0c6460 2024-02-21T23:09:57.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.998+0000 7f3931c00640 1 -- 172.21.15.28:0/3287262936 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f392c13e620 msgr2=0x7f392c159070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:57.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.998+0000 7f3931c00640 1 --2- 172.21.15.28:0/3287262936 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f392c13e620 0x7f392c159070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:57.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.998+0000 7f3931c00640 1 -- 172.21.15.28:0/3287262936 >> v1:172.21.15.28:6789/0 conn(0x7f392c0c6460 legacy=0x7f392c0ca8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:57.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:57.998+0000 7f3931c00640 1 -- 172.21.15.28:0/3287262936 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f392c155420 con 0x7f392c0bf990 2024-02-21T23:09:57.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.000+0000 7f3931c00640 1 --2- 172.21.15.28:0/3287262936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f392c0bf990 0x7f392c0bfe00 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f3924009d90 tx=0x7f392402be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f66dd655707a3c3 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:57.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.000+0000 7f392be00640 1 -- 172.21.15.28:0/3287262936 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3924039070 con 0x7f392c0bf990 2024-02-21T23:09:57.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.000+0000 7f392be00640 1 -- 172.21.15.28:0/3287262936 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f392402cbd0 con 0x7f392c0bf990 2024-02-21T23:09:57.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.000+0000 7f392be00640 1 -- 172.21.15.28:0/3287262936 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3924034440 con 0x7f392c0bf990 2024-02-21T23:09:57.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.001+0000 7f3933c00640 1 -- 172.21.15.28:0/3287262936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f392c0bf990 msgr2=0x7f392c0bfe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:57.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.001+0000 7f3933c00640 1 --2- 172.21.15.28:0/3287262936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f392c0bf990 0x7f392c0bfe00 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f3924009d90 tx=0x7f392402be20 comp rx=0 tx=0).stop 2024-02-21T23:09:57.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.001+0000 7f3933c00640 1 -- 172.21.15.28:0/3287262936 shutdown_connections 2024-02-21T23:09:57.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.001+0000 7f3933c00640 1 --2- 172.21.15.28:0/3287262936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f392c0bf990 0x7f392c0bfe00 unknown :-1 s=CLOSED pgs=1686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:57.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.001+0000 7f3933c00640 1 --2- 172.21.15.28:0/3287262936 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f392c13e620 0x7f392c159070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:57.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.001+0000 7f3933c00640 1 -- 172.21.15.28:0/3287262936 >> 172.21.15.28:0/3287262936 conn(0x7f392c08b1c0 msgr2=0x7f392c0725a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:57.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.001+0000 7f3933c00640 1 -- 172.21.15.28:0/3287262936 shutdown_connections 2024-02-21T23:09:57.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.001+0000 7f3933c00640 1 -- 172.21.15.28:0/3287262936 wait complete. 2024-02-21T23:09:57.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.003+0000 7f3933c00640 1 Processor -- start 2024-02-21T23:09:57.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.003+0000 7f3933c00640 1 -- start start 2024-02-21T23:09:57.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.004+0000 7f3933c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f392c0bf990 0x7f392c0c9770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:57.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.004+0000 7f3933c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f392c0c6460 0x7f392c137c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:57.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.004+0000 7f3933c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f392c13e620 0x7f392c138180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:57.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.004+0000 7f3933c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f392c155cc0 con 0x7f392c13e620 2024-02-21T23:09:57.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.004+0000 7f3933c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f392c1559e0 con 0x7f392c0c6460 2024-02-21T23:09:57.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.004+0000 7f3933c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f392c13ea00 con 0x7f392c0bf990 2024-02-21T23:09:57.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.004+0000 7f3931c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f392c13e620 0x7f392c138180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:57.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.004+0000 7f3931c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f392c13e620 0x7f392c138180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33780/0 (socket says 172.21.15.28:33780) 2024-02-21T23:09:57.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.004+0000 7f3931c00640 1 -- 172.21.15.28:0/1206866622 learned_addr learned my addr 172.21.15.28:0/1206866622 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:57.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.004+0000 7f3931200640 1 --2- 172.21.15.28:0/1206866622 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f392c0bf990 0x7f392c0c9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:57.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.004+0000 7f3923e00640 1 --2- 172.21.15.28:0/1206866622 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f392c0c6460 0x7f392c137c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:57.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.005+0000 7f3931c00640 1 -- 172.21.15.28:0/1206866622 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f392c0bf990 msgr2=0x7f392c0c9770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:57.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.005+0000 7f3931c00640 1 --2- 172.21.15.28:0/1206866622 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f392c0bf990 0x7f392c0c9770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:57.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.005+0000 7f3931c00640 1 -- 172.21.15.28:0/1206866622 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f392c0c6460 msgr2=0x7f392c137c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:57.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.005+0000 7f3931c00640 1 --2- 172.21.15.28:0/1206866622 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f392c0c6460 0x7f392c137c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:57.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.005+0000 7f3931c00640 1 -- 172.21.15.28:0/1206866622 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f392c138a00 con 0x7f392c13e620 2024-02-21T23:09:57.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.005+0000 7f3923e00640 1 --2- 172.21.15.28:0/1206866622 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f392c0c6460 0x7f392c137c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:09:57.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.005+0000 7f3931c00640 1 --2- 172.21.15.28:0/1206866622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f392c13e620 0x7f392c138180 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7f3924009d90 tx=0x7f3924004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:57.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.005+0000 7f3929600640 1 -- 172.21.15.28:0/1206866622 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3924039050 con 0x7f392c13e620 2024-02-21T23:09:57.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.005+0000 7f3929600640 1 -- 172.21.15.28:0/1206866622 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3924045070 con 0x7f392c13e620 2024-02-21T23:09:57.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.005+0000 7f3929600640 1 -- 172.21.15.28:0/1206866622 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3924009450 con 0x7f392c13e620 2024-02-21T23:09:57.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.005+0000 7f3933c00640 1 -- 172.21.15.28:0/1206866622 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f392c0c4410 con 0x7f392c13e620 2024-02-21T23:09:57.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.005+0000 7f3933c00640 1 -- 172.21.15.28:0/1206866622 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f392c0ca5f0 con 0x7f392c13e620 2024-02-21T23:09:57.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.007+0000 7f3933c00640 1 -- 172.21.15.28:0/1206866622 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f392c067f30 con 0x7f392c13e620 2024-02-21T23:09:57.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.008+0000 7f3929600640 1 -- 172.21.15.28:0/1206866622 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f392403e850 con 0x7f392c13e620 2024-02-21T23:09:57.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.009+0000 7f3929600640 1 --2- 172.21.15.28:0/1206866622 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f390806b4d0 0x7f390806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:57.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.009+0000 7f3929600640 1 -- 172.21.15.28:0/1206866622 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f39240ad0d0 con 0x7f392c13e620 2024-02-21T23:09:57.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.010+0000 7f3931200640 1 --2- 172.21.15.28:0/1206866622 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f390806b4d0 0x7f390806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:57.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.011+0000 7f3931200640 1 --2- 172.21.15.28:0/1206866622 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f390806b4d0 0x7f390806d990 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f392c0c4150 tx=0x7f391c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:58.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.014+0000 7f3929600640 1 -- 172.21.15.28:0/1206866622 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3924032e20 con 0x7f392c13e620 2024-02-21T23:09:58.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.212+0000 7f3933c00640 1 -- 172.21.15.28:0/1206866622 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f392c0c8350 con 0x7f392c13e620 2024-02-21T23:09:58.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.214+0000 7f3929600640 1 -- 172.21.15.28:0/1206866622 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v151) v1 ==== 74+0+16824 (secure 0 0 0) 0x7f392407bed0 con 0x7f392c13e620 2024-02-21T23:09:58.202 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:09:58.202 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":151,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:09:55.258904+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:09:51.634291+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"150","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-02-21T23:09:52.736252+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"151","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:09:58.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.216+0000 7f3933c00640 1 -- 172.21.15.28:0/1206866622 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f390806b4d0 msgr2=0x7f390806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:58.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.216+0000 7f3933c00640 1 --2- 172.21.15.28:0/1206866622 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f390806b4d0 0x7f390806d990 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f392c0c4150 tx=0x7f391c008040 comp rx=0 tx=0).stop 2024-02-21T23:09:58.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.216+0000 7f3933c00640 1 -- 172.21.15.28:0/1206866622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f392c13e620 msgr2=0x7f392c138180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:58.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.216+0000 7f3933c00640 1 --2- 172.21.15.28:0/1206866622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f392c13e620 0x7f392c138180 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7f3924009d90 tx=0x7f3924004a40 comp rx=0 tx=0).stop 2024-02-21T23:09:58.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.217+0000 7f3933c00640 1 -- 172.21.15.28:0/1206866622 shutdown_connections 2024-02-21T23:09:58.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.217+0000 7f3933c00640 1 --2- 172.21.15.28:0/1206866622 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f390806b4d0 0x7f390806d990 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:58.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.217+0000 7f3933c00640 1 --2- 172.21.15.28:0/1206866622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f392c13e620 0x7f392c138180 unknown :-1 s=CLOSED pgs=1687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:58.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.217+0000 7f3933c00640 1 --2- 172.21.15.28:0/1206866622 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f392c0c6460 0x7f392c137c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:58.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.217+0000 7f3933c00640 1 --2- 172.21.15.28:0/1206866622 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f392c0bf990 0x7f392c0c9770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:58.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.217+0000 7f3933c00640 1 -- 172.21.15.28:0/1206866622 >> 172.21.15.28:0/1206866622 conn(0x7f392c08b1c0 msgr2=0x7f392c154810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:58.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.217+0000 7f3933c00640 1 -- 172.21.15.28:0/1206866622 shutdown_connections 2024-02-21T23:09:58.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.217+0000 7f3933c00640 1 -- 172.21.15.28:0/1206866622 wait complete. 2024-02-21T23:09:58.224 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-02-21T23:09:58.224 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-21T23:09:58.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.451+0000 7f312e400640 1 Processor -- start 2024-02-21T23:09:58.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.451+0000 7f312e400640 1 -- start start 2024-02-21T23:09:58.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.453+0000 7f312e400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f31281559c0 0x7f31280c76f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:58.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.453+0000 7f312e400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f31280c7c30 0x7f312815f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:58.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.453+0000 7f312e400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31280c6f30 con 0x7f31281526b0 2024-02-21T23:09:58.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.453+0000 7f312e400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31280c61b0 con 0x7f31280c7c30 2024-02-21T23:09:58.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.453+0000 7f312e400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31280c6450 con 0x7f31281559c0 2024-02-21T23:09:58.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.453+0000 7f3127e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f31280c7c30 0x7f312815f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:58.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.453+0000 7f3127e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f31280c7c30 0x7f312815f8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33784/0 (socket says 172.21.15.28:33784) 2024-02-21T23:09:58.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.453+0000 7f3127e00640 1 -- 172.21.15.28:0/753753779 learned_addr learned my addr 172.21.15.28:0/753753779 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:58.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.453+0000 7f3126a00640 1 --2- 172.21.15.28:0/753753779 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f31281559c0 0x7f31280c76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:58.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.454+0000 7f3127e00640 1 -- 172.21.15.28:0/753753779 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f31281559c0 msgr2=0x7f31280c76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:58.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.454+0000 7f3127e00640 1 --2- 172.21.15.28:0/753753779 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f31281559c0 0x7f31280c76f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:58.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.454+0000 7f3127e00640 1 -- 172.21.15.28:0/753753779 >> v1:172.21.15.81:6789/0 conn(0x7f31281526b0 legacy=0x7f31280ca940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:58.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.454+0000 7f3127e00640 1 -- 172.21.15.28:0/753753779 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31280c6710 con 0x7f31280c7c30 2024-02-21T23:09:58.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.455+0000 7f3127e00640 1 --2- 172.21.15.28:0/753753779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31280c7c30 0x7f312815f8c0 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7f3120009e80 tx=0x7f312002bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9acfa6de026c4083 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:58.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.455+0000 7f3126000640 1 -- 172.21.15.28:0/753753779 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f312002caf0 con 0x7f31280c7c30 2024-02-21T23:09:58.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.455+0000 7f3126000640 1 -- 172.21.15.28:0/753753779 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f312002cc90 con 0x7f31280c7c30 2024-02-21T23:09:58.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.455+0000 7f3126000640 1 -- 172.21.15.28:0/753753779 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3120034610 con 0x7f31280c7c30 2024-02-21T23:09:58.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.456+0000 7f312e400640 1 -- 172.21.15.28:0/753753779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31280c7c30 msgr2=0x7f312815f8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:58.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.456+0000 7f312e400640 1 --2- 172.21.15.28:0/753753779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31280c7c30 0x7f312815f8c0 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7f3120009e80 tx=0x7f312002bf50 comp rx=0 tx=0).stop 2024-02-21T23:09:58.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.456+0000 7f312e400640 1 -- 172.21.15.28:0/753753779 shutdown_connections 2024-02-21T23:09:58.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.456+0000 7f312e400640 1 --2- 172.21.15.28:0/753753779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31280c7c30 0x7f312815f8c0 unknown :-1 s=CLOSED pgs=1688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:58.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.456+0000 7f312e400640 1 --2- 172.21.15.28:0/753753779 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f31281559c0 0x7f31280c76f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:58.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.456+0000 7f312e400640 1 -- 172.21.15.28:0/753753779 >> 172.21.15.28:0/753753779 conn(0x7f312806d3d0 msgr2=0x7f312808afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:58.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.456+0000 7f312e400640 1 -- 172.21.15.28:0/753753779 shutdown_connections 2024-02-21T23:09:58.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.456+0000 7f312e400640 1 -- 172.21.15.28:0/753753779 wait complete. 2024-02-21T23:09:58.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.458+0000 7f312e400640 1 Processor -- start 2024-02-21T23:09:58.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.459+0000 7f312e400640 1 -- start start 2024-02-21T23:09:58.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.460+0000 7f312e400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31280c7c30 0x7f31281eb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:58.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.460+0000 7f312e400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f31281526b0 0x7f31281565a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:58.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.460+0000 7f312e400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f31281559c0 0x7f31280ca680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:58.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.460+0000 7f312e400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31280c5c90 con 0x7f31280c7c30 2024-02-21T23:09:58.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.460+0000 7f312e400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31280c71d0 con 0x7f31281559c0 2024-02-21T23:09:58.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.460+0000 7f312e400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31280c4e70 con 0x7f31281526b0 2024-02-21T23:09:58.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.461+0000 7f3127400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31280c7c30 0x7f31281eb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:58.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.461+0000 7f3127400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31280c7c30 0x7f31281eb7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33794/0 (socket says 172.21.15.28:33794) 2024-02-21T23:09:58.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.461+0000 7f3127400640 1 -- 172.21.15.28:0/1081015222 learned_addr learned my addr 172.21.15.28:0/1081015222 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:58.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.461+0000 7f3126a00640 1 --2- 172.21.15.28:0/1081015222 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f31281526b0 0x7f31281565a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:58.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.461+0000 7f3127e00640 1 --2- 172.21.15.28:0/1081015222 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f31281559c0 0x7f31280ca680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:58.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.461+0000 7f3127400640 1 -- 172.21.15.28:0/1081015222 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f31281526b0 msgr2=0x7f31281565a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:58.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.461+0000 7f3127400640 1 --2- 172.21.15.28:0/1081015222 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f31281526b0 0x7f31281565a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:58.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.461+0000 7f3127400640 1 -- 172.21.15.28:0/1081015222 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f31281559c0 msgr2=0x7f31280ca680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:58.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.461+0000 7f3127400640 1 --2- 172.21.15.28:0/1081015222 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f31281559c0 0x7f31280ca680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:58.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.461+0000 7f3127400640 1 -- 172.21.15.28:0/1081015222 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f312814d810 con 0x7f31280c7c30 2024-02-21T23:09:58.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.461+0000 7f3127400640 1 --2- 172.21.15.28:0/1081015222 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31280c7c30 0x7f31281eb7c0 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7f311800c430 tx=0x7f311800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:58.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.462+0000 7f311f400640 1 -- 172.21.15.28:0/1081015222 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31180052d0 con 0x7f31280c7c30 2024-02-21T23:09:58.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.462+0000 7f311f400640 1 -- 172.21.15.28:0/1081015222 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3118005490 con 0x7f31280c7c30 2024-02-21T23:09:58.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.462+0000 7f312e400640 1 -- 172.21.15.28:0/1081015222 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31280c8bd0 con 0x7f31280c7c30 2024-02-21T23:09:58.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.462+0000 7f311f400640 1 -- 172.21.15.28:0/1081015222 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3118007260 con 0x7f31280c7c30 2024-02-21T23:09:58.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.462+0000 7f312e400640 1 -- 172.21.15.28:0/1081015222 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31280c9190 con 0x7f31280c7c30 2024-02-21T23:09:58.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.463+0000 7f312e400640 1 -- 172.21.15.28:0/1081015222 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3128067f30 con 0x7f31280c7c30 2024-02-21T23:09:58.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.464+0000 7f311f400640 1 -- 172.21.15.28:0/1081015222 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3118007400 con 0x7f31280c7c30 2024-02-21T23:09:58.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.465+0000 7f311f400640 1 --2- 172.21.15.28:0/1081015222 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f310806b4d0 0x7f310806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:58.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.465+0000 7f311f400640 1 -- 172.21.15.28:0/1081015222 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f3118089890 con 0x7f31280c7c30 2024-02-21T23:09:58.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.466+0000 7f3126a00640 1 --2- 172.21.15.28:0/1081015222 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f310806b4d0 0x7f310806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:58.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.467+0000 7f3126a00640 1 --2- 172.21.15.28:0/1081015222 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f310806b4d0 0x7f310806d990 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f3114006fd0 tx=0x7f3114008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:58.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.470+0000 7f311f400640 1 -- 172.21.15.28:0/1081015222 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3118058570 con 0x7f31280c7c30 2024-02-21T23:09:58.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.697+0000 7f312e400640 1 -- 172.21.15.28:0/1081015222 --> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f31280c3630 con 0x7f310806b4d0 2024-02-21T23:09:58.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.717+0000 7f311f400640 1 -- 172.21.15.28:0/1081015222 <== mgr.4101 v2:172.21.15.81:6832/1848538577 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252526 (secure 0 0 0) 0x7f31280c3630 con 0x7f310806b4d0 2024-02-21T23:09:58.710 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:09:58.711 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-21T23:09:58.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.722+0000 7f312e400640 1 -- 172.21.15.28:0/1081015222 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f310806b4d0 msgr2=0x7f310806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:58.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.722+0000 7f312e400640 1 --2- 172.21.15.28:0/1081015222 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f310806b4d0 0x7f310806d990 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f3114006fd0 tx=0x7f3114008040 comp rx=0 tx=0).stop 2024-02-21T23:09:58.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.722+0000 7f312e400640 1 -- 172.21.15.28:0/1081015222 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31280c7c30 msgr2=0x7f31281eb7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:58.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.722+0000 7f312e400640 1 --2- 172.21.15.28:0/1081015222 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31280c7c30 0x7f31281eb7c0 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7f311800c430 tx=0x7f311800c8f0 comp rx=0 tx=0).stop 2024-02-21T23:09:58.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.722+0000 7f312e400640 1 -- 172.21.15.28:0/1081015222 shutdown_connections 2024-02-21T23:09:58.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.722+0000 7f312e400640 1 --2- 172.21.15.28:0/1081015222 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f310806b4d0 0x7f310806d990 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:58.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.722+0000 7f312e400640 1 --2- 172.21.15.28:0/1081015222 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f31281526b0 0x7f31281565a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:58.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.722+0000 7f312e400640 1 --2- 172.21.15.28:0/1081015222 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f31281559c0 0x7f31280ca680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:58.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.722+0000 7f312e400640 1 --2- 172.21.15.28:0/1081015222 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31280c7c30 0x7f31281eb7c0 unknown :-1 s=CLOSED pgs=1689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:58.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.722+0000 7f312e400640 1 -- 172.21.15.28:0/1081015222 >> 172.21.15.28:0/1081015222 conn(0x7f312806d3d0 msgr2=0x7f312808afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:58.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.722+0000 7f312e400640 1 -- 172.21.15.28:0/1081015222 shutdown_connections 2024-02-21T23:09:58.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.722+0000 7f312e400640 1 -- 172.21.15.28:0/1081015222 wait complete. 2024-02-21T23:09:58.729 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":367,"stamp":"2024-02-21T23:09:56.819178+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":98,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":264440,"kb_used_data":14980,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754710280,"statfs":{"total":773094113280,"available":772823326720,"internally_reserved":0,"allocated":15339520,"data_stored":8557192,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.586587"},"pg_stats":[{"pgid":"24.3f","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.276269+0000","last_change":"2024-02-21T23:09:54.279018+0000","last_active":"2024-02-21T23:09:55.276269+0000","last_peered":"2024-02-21T23:09:55.276269+0000","last_clean":"2024-02-21T23:09:55.276269+0000","last_became_active":"2024-02-21T23:09:54.278712+0000","last_became_peered":"2024-02-21T23:09:54.278712+0000","last_unstale":"2024-02-21T23:09:55.276269+0000","last_undegraded":"2024-02-21T23:09:55.276269+0000","last_fullsized":"2024-02-21T23:09:55.276269+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:55:18.591062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3e","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.287359+0000","last_change":"2024-02-21T23:09:54.285889+0000","last_active":"2024-02-21T23:09:54.287359+0000","last_peered":"2024-02-21T23:09:54.287359+0000","last_clean":"2024-02-21T23:09:54.287359+0000","last_became_active":"2024-02-21T23:09:54.284810+0000","last_became_peered":"2024-02-21T23:09:54.284810+0000","last_unstale":"2024-02-21T23:09:54.287359+0000","last_undegraded":"2024-02-21T23:09:54.287359+0000","last_fullsized":"2024-02-21T23:09:54.287359+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:13:44.635531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3d","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.275976+0000","last_change":"2024-02-21T23:09:54.263978+0000","last_active":"2024-02-21T23:09:55.275976+0000","last_peered":"2024-02-21T23:09:55.275976+0000","last_clean":"2024-02-21T23:09:55.275976+0000","last_became_active":"2024-02-21T23:09:54.263467+0000","last_became_peered":"2024-02-21T23:09:54.263467+0000","last_unstale":"2024-02-21T23:09:55.275976+0000","last_undegraded":"2024-02-21T23:09:55.275976+0000","last_fullsized":"2024-02-21T23:09:55.275976+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:46:35.436815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3c","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.276401+0000","last_change":"2024-02-21T23:09:54.279400+0000","last_active":"2024-02-21T23:09:55.276401+0000","last_peered":"2024-02-21T23:09:55.276401+0000","last_clean":"2024-02-21T23:09:55.276401+0000","last_became_active":"2024-02-21T23:09:54.279076+0000","last_became_peered":"2024-02-21T23:09:54.279076+0000","last_unstale":"2024-02-21T23:09:55.276401+0000","last_undegraded":"2024-02-21T23:09:55.276401+0000","last_fullsized":"2024-02-21T23:09:55.276401+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:02:11.247387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3b","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.274751+0000","last_change":"2024-02-21T23:09:54.271682+0000","last_active":"2024-02-21T23:09:54.274751+0000","last_peered":"2024-02-21T23:09:54.274751+0000","last_clean":"2024-02-21T23:09:54.274751+0000","last_became_active":"2024-02-21T23:09:54.271297+0000","last_became_peered":"2024-02-21T23:09:54.271297+0000","last_unstale":"2024-02-21T23:09:54.274751+0000","last_undegraded":"2024-02-21T23:09:54.274751+0000","last_fullsized":"2024-02-21T23:09:54.274751+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:51:09.560052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3a","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.286978+0000","last_change":"2024-02-21T23:09:54.281830+0000","last_active":"2024-02-21T23:09:54.286978+0000","last_peered":"2024-02-21T23:09:54.286978+0000","last_clean":"2024-02-21T23:09:54.286978+0000","last_became_active":"2024-02-21T23:09:54.281334+0000","last_became_peered":"2024-02-21T23:09:54.281334+0000","last_unstale":"2024-02-21T23:09:54.286978+0000","last_undegraded":"2024-02-21T23:09:54.286978+0000","last_fullsized":"2024-02-21T23:09:54.286978+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:29:25.643198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.39","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.290041+0000","last_change":"2024-02-21T23:09:54.277806+0000","last_active":"2024-02-21T23:09:55.290041+0000","last_peered":"2024-02-21T23:09:55.290041+0000","last_clean":"2024-02-21T23:09:55.290041+0000","last_became_active":"2024-02-21T23:09:54.277446+0000","last_became_peered":"2024-02-21T23:09:54.277446+0000","last_unstale":"2024-02-21T23:09:55.290041+0000","last_undegraded":"2024-02-21T23:09:55.290041+0000","last_fullsized":"2024-02-21T23:09:55.290041+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:59:34.646775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.38","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.289953+0000","last_change":"2024-02-21T23:09:54.286729+0000","last_active":"2024-02-21T23:09:55.289953+0000","last_peered":"2024-02-21T23:09:55.289953+0000","last_clean":"2024-02-21T23:09:55.289953+0000","last_became_active":"2024-02-21T23:09:54.286427+0000","last_became_peered":"2024-02-21T23:09:54.286427+0000","last_unstale":"2024-02-21T23:09:55.289953+0000","last_undegraded":"2024-02-21T23:09:55.289953+0000","last_fullsized":"2024-02-21T23:09:55.289953+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:13:27.311923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.37","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.290895+0000","last_change":"2024-02-21T23:09:54.286612+0000","last_active":"2024-02-21T23:09:55.290895+0000","last_peered":"2024-02-21T23:09:55.290895+0000","last_clean":"2024-02-21T23:09:55.290895+0000","last_became_active":"2024-02-21T23:09:54.286306+0000","last_became_peered":"2024-02-21T23:09:54.286306+0000","last_unstale":"2024-02-21T23:09:55.290895+0000","last_undegraded":"2024-02-21T23:09:55.290895+0000","last_fullsized":"2024-02-21T23:09:55.290895+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:04:16.317089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.36","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.291078+0000","last_change":"2024-02-21T23:09:54.288019+0000","last_active":"2024-02-21T23:09:55.291078+0000","last_peered":"2024-02-21T23:09:55.291078+0000","last_clean":"2024-02-21T23:09:55.291078+0000","last_became_active":"2024-02-21T23:09:54.287742+0000","last_became_peered":"2024-02-21T23:09:54.287742+0000","last_unstale":"2024-02-21T23:09:55.291078+0000","last_undegraded":"2024-02-21T23:09:55.291078+0000","last_fullsized":"2024-02-21T23:09:55.291078+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:13:33.442227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.35","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.286910+0000","last_change":"2024-02-21T23:09:54.283918+0000","last_active":"2024-02-21T23:09:54.286910+0000","last_peered":"2024-02-21T23:09:54.286910+0000","last_clean":"2024-02-21T23:09:54.286910+0000","last_became_active":"2024-02-21T23:09:54.280599+0000","last_became_peered":"2024-02-21T23:09:54.280599+0000","last_unstale":"2024-02-21T23:09:54.286910+0000","last_undegraded":"2024-02-21T23:09:54.286910+0000","last_fullsized":"2024-02-21T23:09:54.286910+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:13:44.635531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.34","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.287159+0000","last_change":"2024-02-21T23:09:54.282846+0000","last_active":"2024-02-21T23:09:54.287159+0000","last_peered":"2024-02-21T23:09:54.287159+0000","last_clean":"2024-02-21T23:09:54.287159+0000","last_became_active":"2024-02-21T23:09:54.282117+0000","last_became_peered":"2024-02-21T23:09:54.282117+0000","last_unstale":"2024-02-21T23:09:54.287159+0000","last_undegraded":"2024-02-21T23:09:54.287159+0000","last_fullsized":"2024-02-21T23:09:54.287159+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:58:27.362926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.33","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.273190+0000","last_change":"2024-02-21T23:09:54.284818+0000","last_active":"2024-02-21T23:09:55.273190+0000","last_peered":"2024-02-21T23:09:55.273190+0000","last_clean":"2024-02-21T23:09:55.273190+0000","last_became_active":"2024-02-21T23:09:54.284161+0000","last_became_peered":"2024-02-21T23:09:54.284161+0000","last_unstale":"2024-02-21T23:09:55.273190+0000","last_undegraded":"2024-02-21T23:09:55.273190+0000","last_fullsized":"2024-02-21T23:09:55.273190+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:02:12.340921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.32","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.273891+0000","last_change":"2024-02-21T23:09:54.272136+0000","last_active":"2024-02-21T23:09:54.273891+0000","last_peered":"2024-02-21T23:09:54.273891+0000","last_clean":"2024-02-21T23:09:54.273891+0000","last_became_active":"2024-02-21T23:09:54.271097+0000","last_became_peered":"2024-02-21T23:09:54.271097+0000","last_unstale":"2024-02-21T23:09:54.273891+0000","last_undegraded":"2024-02-21T23:09:54.273891+0000","last_fullsized":"2024-02-21T23:09:54.273891+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:52:57.889995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.31","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.286879+0000","last_change":"2024-02-21T23:09:54.281268+0000","last_active":"2024-02-21T23:09:54.286879+0000","last_peered":"2024-02-21T23:09:54.286879+0000","last_clean":"2024-02-21T23:09:54.286879+0000","last_became_active":"2024-02-21T23:09:54.280946+0000","last_became_peered":"2024-02-21T23:09:54.280946+0000","last_unstale":"2024-02-21T23:09:54.286879+0000","last_undegraded":"2024-02-21T23:09:54.286879+0000","last_fullsized":"2024-02-21T23:09:54.286879+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:31:28.395228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.30","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.275530+0000","last_change":"2024-02-21T23:09:54.278731+0000","last_active":"2024-02-21T23:09:55.275530+0000","last_peered":"2024-02-21T23:09:55.275530+0000","last_clean":"2024-02-21T23:09:55.275530+0000","last_became_active":"2024-02-21T23:09:54.278390+0000","last_became_peered":"2024-02-21T23:09:54.278390+0000","last_unstale":"2024-02-21T23:09:55.275530+0000","last_undegraded":"2024-02-21T23:09:55.275530+0000","last_fullsized":"2024-02-21T23:09:55.275530+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:28:54.079553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.2f","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.291090+0000","last_change":"2024-02-21T23:09:54.285947+0000","last_active":"2024-02-21T23:09:55.291090+0000","last_peered":"2024-02-21T23:09:55.291090+0000","last_clean":"2024-02-21T23:09:55.291090+0000","last_became_active":"2024-02-21T23:09:54.285658+0000","last_became_peered":"2024-02-21T23:09:54.285658+0000","last_unstale":"2024-02-21T23:09:55.291090+0000","last_undegraded":"2024-02-21T23:09:55.291090+0000","last_fullsized":"2024-02-21T23:09:55.291090+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:20:31.125684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2e","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.526134+0000","last_change":"2024-02-21T23:09:54.287286+0000","last_active":"2024-02-21T23:09:55.526134+0000","last_peered":"2024-02-21T23:09:55.526134+0000","last_clean":"2024-02-21T23:09:55.526134+0000","last_became_active":"2024-02-21T23:09:54.286924+0000","last_became_peered":"2024-02-21T23:09:54.286924+0000","last_unstale":"2024-02-21T23:09:55.526134+0000","last_undegraded":"2024-02-21T23:09:55.526134+0000","last_fullsized":"2024-02-21T23:09:55.526134+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:47:29.349669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.2d","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.286185+0000","last_change":"2024-02-21T23:09:54.279884+0000","last_active":"2024-02-21T23:09:54.286185+0000","last_peered":"2024-02-21T23:09:54.286185+0000","last_clean":"2024-02-21T23:09:54.286185+0000","last_became_active":"2024-02-21T23:09:54.276608+0000","last_became_peered":"2024-02-21T23:09:54.276608+0000","last_unstale":"2024-02-21T23:09:54.286185+0000","last_undegraded":"2024-02-21T23:09:54.286185+0000","last_fullsized":"2024-02-21T23:09:54.286185+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:34:50.354067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.2c","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.272941+0000","last_change":"2024-02-21T23:09:54.288944+0000","last_active":"2024-02-21T23:09:55.272941+0000","last_peered":"2024-02-21T23:09:55.272941+0000","last_clean":"2024-02-21T23:09:55.272941+0000","last_became_active":"2024-02-21T23:09:54.285913+0000","last_became_peered":"2024-02-21T23:09:54.285913+0000","last_unstale":"2024-02-21T23:09:55.272941+0000","last_undegraded":"2024-02-21T23:09:55.272941+0000","last_fullsized":"2024-02-21T23:09:55.272941+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:39:33.609273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.2b","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.273379+0000","last_change":"2024-02-21T23:09:54.285036+0000","last_active":"2024-02-21T23:09:55.273379+0000","last_peered":"2024-02-21T23:09:55.273379+0000","last_clean":"2024-02-21T23:09:55.273379+0000","last_became_active":"2024-02-21T23:09:54.284542+0000","last_became_peered":"2024-02-21T23:09:54.284542+0000","last_unstale":"2024-02-21T23:09:55.273379+0000","last_undegraded":"2024-02-21T23:09:55.273379+0000","last_fullsized":"2024-02-21T23:09:55.273379+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T11:05:01.241636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.2a","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.274942+0000","last_change":"2024-02-21T23:09:54.273350+0000","last_active":"2024-02-21T23:09:54.274942+0000","last_peered":"2024-02-21T23:09:54.274942+0000","last_clean":"2024-02-21T23:09:54.274942+0000","last_became_active":"2024-02-21T23:09:54.273022+0000","last_became_peered":"2024-02-21T23:09:54.273022+0000","last_unstale":"2024-02-21T23:09:54.274942+0000","last_undegraded":"2024-02-21T23:09:54.274942+0000","last_fullsized":"2024-02-21T23:09:54.274942+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:23:27.511315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.29","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.281903+0000","last_change":"2024-02-21T23:09:54.279718+0000","last_active":"2024-02-21T23:09:54.281903+0000","last_peered":"2024-02-21T23:09:54.281903+0000","last_clean":"2024-02-21T23:09:54.281903+0000","last_became_active":"2024-02-21T23:09:54.278989+0000","last_became_peered":"2024-02-21T23:09:54.278989+0000","last_unstale":"2024-02-21T23:09:54.281903+0000","last_undegraded":"2024-02-21T23:09:54.281903+0000","last_fullsized":"2024-02-21T23:09:54.281903+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:33:59.785551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.28","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.275774+0000","last_change":"2024-02-21T23:09:54.280624+0000","last_active":"2024-02-21T23:09:55.275774+0000","last_peered":"2024-02-21T23:09:55.275774+0000","last_clean":"2024-02-21T23:09:55.275774+0000","last_became_active":"2024-02-21T23:09:54.275644+0000","last_became_peered":"2024-02-21T23:09:54.275644+0000","last_unstale":"2024-02-21T23:09:55.275774+0000","last_undegraded":"2024-02-21T23:09:55.275774+0000","last_fullsized":"2024-02-21T23:09:55.275774+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:23:27.511315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.27","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.273927+0000","last_change":"2024-02-21T23:09:54.275341+0000","last_active":"2024-02-21T23:09:55.273927+0000","last_peered":"2024-02-21T23:09:55.273927+0000","last_clean":"2024-02-21T23:09:55.273927+0000","last_became_active":"2024-02-21T23:09:54.275010+0000","last_became_peered":"2024-02-21T23:09:54.275010+0000","last_unstale":"2024-02-21T23:09:55.273927+0000","last_undegraded":"2024-02-21T23:09:55.273927+0000","last_fullsized":"2024-02-21T23:09:55.273927+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:42:08.123188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.26","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.272151+0000","last_change":"2024-02-21T23:09:54.260902+0000","last_active":"2024-02-21T23:09:55.272151+0000","last_peered":"2024-02-21T23:09:55.272151+0000","last_clean":"2024-02-21T23:09:55.272151+0000","last_became_active":"2024-02-21T23:09:54.260084+0000","last_became_peered":"2024-02-21T23:09:54.260084+0000","last_unstale":"2024-02-21T23:09:55.272151+0000","last_undegraded":"2024-02-21T23:09:55.272151+0000","last_fullsized":"2024-02-21T23:09:55.272151+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:04:05.858817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.25","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.287285+0000","last_change":"2024-02-21T23:09:54.284116+0000","last_active":"2024-02-21T23:09:54.287285+0000","last_peered":"2024-02-21T23:09:54.287285+0000","last_clean":"2024-02-21T23:09:54.287285+0000","last_became_active":"2024-02-21T23:09:54.283802+0000","last_became_peered":"2024-02-21T23:09:54.283802+0000","last_unstale":"2024-02-21T23:09:54.287285+0000","last_undegraded":"2024-02-21T23:09:54.287285+0000","last_fullsized":"2024-02-21T23:09:54.287285+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:31:28.395228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.24","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.273666+0000","last_change":"2024-02-21T23:09:54.287481+0000","last_active":"2024-02-21T23:09:55.273666+0000","last_peered":"2024-02-21T23:09:55.273666+0000","last_clean":"2024-02-21T23:09:55.273666+0000","last_became_active":"2024-02-21T23:09:54.287137+0000","last_became_peered":"2024-02-21T23:09:54.287137+0000","last_unstale":"2024-02-21T23:09:55.273666+0000","last_undegraded":"2024-02-21T23:09:55.273666+0000","last_fullsized":"2024-02-21T23:09:55.273666+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:50:46.472002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.23","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.287067+0000","last_change":"2024-02-21T23:09:54.281613+0000","last_active":"2024-02-21T23:09:54.287067+0000","last_peered":"2024-02-21T23:09:54.287067+0000","last_clean":"2024-02-21T23:09:54.287067+0000","last_became_active":"2024-02-21T23:09:54.281316+0000","last_became_peered":"2024-02-21T23:09:54.281316+0000","last_unstale":"2024-02-21T23:09:54.287067+0000","last_undegraded":"2024-02-21T23:09:54.287067+0000","last_fullsized":"2024-02-21T23:09:54.287067+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:32:13.492141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.22","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.525855+0000","last_change":"2024-02-21T23:09:54.275665+0000","last_active":"2024-02-21T23:09:55.525855+0000","last_peered":"2024-02-21T23:09:55.525855+0000","last_clean":"2024-02-21T23:09:55.525855+0000","last_became_active":"2024-02-21T23:09:54.275143+0000","last_became_peered":"2024-02-21T23:09:54.275143+0000","last_unstale":"2024-02-21T23:09:55.525855+0000","last_undegraded":"2024-02-21T23:09:55.525855+0000","last_fullsized":"2024-02-21T23:09:55.525855+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:37:42.976792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.21","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.286230+0000","last_change":"2024-02-21T23:09:54.281200+0000","last_active":"2024-02-21T23:09:54.286230+0000","last_peered":"2024-02-21T23:09:54.286230+0000","last_clean":"2024-02-21T23:09:54.286230+0000","last_became_active":"2024-02-21T23:09:54.277867+0000","last_became_peered":"2024-02-21T23:09:54.277867+0000","last_unstale":"2024-02-21T23:09:54.286230+0000","last_undegraded":"2024-02-21T23:09:54.286230+0000","last_fullsized":"2024-02-21T23:09:54.286230+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:58:27.362926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.20","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.525712+0000","last_change":"2024-02-21T23:09:54.276078+0000","last_active":"2024-02-21T23:09:55.525712+0000","last_peered":"2024-02-21T23:09:55.525712+0000","last_clean":"2024-02-21T23:09:55.525712+0000","last_became_active":"2024-02-21T23:09:54.275724+0000","last_became_peered":"2024-02-21T23:09:54.275724+0000","last_unstale":"2024-02-21T23:09:55.525712+0000","last_undegraded":"2024-02-21T23:09:55.525712+0000","last_fullsized":"2024-02-21T23:09:55.525712+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:48:07.369182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.10","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.286630+0000","last_change":"2024-02-21T23:09:54.278972+0000","last_active":"2024-02-21T23:09:54.286630+0000","last_peered":"2024-02-21T23:09:54.286630+0000","last_clean":"2024-02-21T23:09:54.286630+0000","last_became_active":"2024-02-21T23:09:54.278146+0000","last_became_peered":"2024-02-21T23:09:54.278146+0000","last_unstale":"2024-02-21T23:09:54.286630+0000","last_undegraded":"2024-02-21T23:09:54.286630+0000","last_fullsized":"2024-02-21T23:09:54.286630+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:02:11.247387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"150'4","reported_seq":18,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.276587+0000","last_change":"2024-02-21T23:09:53.288506+0000","last_active":"2024-02-21T23:09:55.276587+0000","last_peered":"2024-02-21T23:09:55.276587+0000","last_clean":"2024-02-21T23:09:55.276587+0000","last_became_active":"2024-02-21T23:09:53.288199+0000","last_became_peered":"2024-02-21T23:09:53.288199+0000","last_unstale":"2024-02-21T23:09:55.276587+0000","last_undegraded":"2024-02-21T23:09:55.276587+0000","last_fullsized":"2024-02-21T23:09:55.276587+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:18:16.467715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.273863+0000","last_change":"2024-02-21T23:09:54.278032+0000","last_active":"2024-02-21T23:09:55.273863+0000","last_peered":"2024-02-21T23:09:55.273863+0000","last_clean":"2024-02-21T23:09:55.273863+0000","last_became_active":"2024-02-21T23:09:54.277741+0000","last_became_peered":"2024-02-21T23:09:54.277741+0000","last_unstale":"2024-02-21T23:09:55.273863+0000","last_undegraded":"2024-02-21T23:09:55.273863+0000","last_fullsized":"2024-02-21T23:09:55.273863+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:43:56.610613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"150'1","reported_seq":13,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.302587+0000","last_change":"2024-02-21T23:09:53.279374+0000","last_active":"2024-02-21T23:09:54.302587+0000","last_peered":"2024-02-21T23:09:54.302587+0000","last_clean":"2024-02-21T23:09:54.302587+0000","last_became_active":"2024-02-21T23:09:53.279053+0000","last_became_peered":"2024-02-21T23:09:53.279053+0000","last_unstale":"2024-02-21T23:09:54.302587+0000","last_undegraded":"2024-02-21T23:09:54.302587+0000","last_fullsized":"2024-02-21T23:09:54.302587+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:04:15.300172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.272323+0000","last_change":"2024-02-21T23:09:54.263915+0000","last_active":"2024-02-21T23:09:54.272323+0000","last_peered":"2024-02-21T23:09:54.272323+0000","last_clean":"2024-02-21T23:09:54.272323+0000","last_became_active":"2024-02-21T23:09:54.263634+0000","last_became_peered":"2024-02-21T23:09:54.263634+0000","last_unstale":"2024-02-21T23:09:54.272323+0000","last_undegraded":"2024-02-21T23:09:54.272323+0000","last_fullsized":"2024-02-21T23:09:54.272323+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:33:07.362294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.1d","version":"150'2","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.302666+0000","last_change":"2024-02-21T23:09:53.272110+0000","last_active":"2024-02-21T23:09:54.302666+0000","last_peered":"2024-02-21T23:09:54.302666+0000","last_clean":"2024-02-21T23:09:54.302666+0000","last_became_active":"2024-02-21T23:09:53.271641+0000","last_became_peered":"2024-02-21T23:09:53.271641+0000","last_unstale":"2024-02-21T23:09:54.302666+0000","last_undegraded":"2024-02-21T23:09:54.302666+0000","last_fullsized":"2024-02-21T23:09:54.302666+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:46:34.419898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.13","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.284893+0000","last_change":"2024-02-21T23:09:54.275867+0000","last_active":"2024-02-21T23:09:54.284893+0000","last_peered":"2024-02-21T23:09:54.284893+0000","last_clean":"2024-02-21T23:09:54.284893+0000","last_became_active":"2024-02-21T23:09:54.275591+0000","last_became_peered":"2024-02-21T23:09:54.275591+0000","last_unstale":"2024-02-21T23:09:54.284893+0000","last_undegraded":"2024-02-21T23:09:54.284893+0000","last_fullsized":"2024-02-21T23:09:54.284893+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:46:35.436815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1c","version":"150'2","reported_seq":16,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.273105+0000","last_change":"2024-02-21T23:09:53.274692+0000","last_active":"2024-02-21T23:09:55.273105+0000","last_peered":"2024-02-21T23:09:55.273105+0000","last_clean":"2024-02-21T23:09:55.273105+0000","last_became_active":"2024-02-21T23:09:53.274395+0000","last_became_peered":"2024-02-21T23:09:53.274395+0000","last_unstale":"2024-02-21T23:09:55.273105+0000","last_undegraded":"2024-02-21T23:09:55.273105+0000","last_fullsized":"2024-02-21T23:09:55.273105+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:40:18.453921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.14","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.290542+0000","last_change":"2024-02-21T23:09:54.289092+0000","last_active":"2024-02-21T23:09:55.290542+0000","last_peered":"2024-02-21T23:09:55.290542+0000","last_clean":"2024-02-21T23:09:55.290542+0000","last_became_active":"2024-02-21T23:09:54.288801+0000","last_became_peered":"2024-02-21T23:09:54.288801+0000","last_unstale":"2024-02-21T23:09:55.290542+0000","last_undegraded":"2024-02-21T23:09:55.290542+0000","last_fullsized":"2024-02-21T23:09:55.290542+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:05:09.189420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1b","version":"150'1","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.290367+0000","last_change":"2024-02-21T23:09:53.283422+0000","last_active":"2024-02-21T23:09:55.290367+0000","last_peered":"2024-02-21T23:09:55.290367+0000","last_clean":"2024-02-21T23:09:55.290367+0000","last_became_active":"2024-02-21T23:09:53.283140+0000","last_became_peered":"2024-02-21T23:09:53.283140+0000","last_unstale":"2024-02-21T23:09:55.290367+0000","last_undegraded":"2024-02-21T23:09:55.290367+0000","last_fullsized":"2024-02-21T23:09:55.290367+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:46:34.419898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.15","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.272579+0000","last_change":"2024-02-21T23:09:54.273728+0000","last_active":"2024-02-21T23:09:55.272579+0000","last_peered":"2024-02-21T23:09:55.272579+0000","last_clean":"2024-02-21T23:09:55.272579+0000","last_became_active":"2024-02-21T23:09:54.273343+0000","last_became_peered":"2024-02-21T23:09:54.273343+0000","last_unstale":"2024-02-21T23:09:55.272579+0000","last_undegraded":"2024-02-21T23:09:55.272579+0000","last_fullsized":"2024-02-21T23:09:55.272579+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:49:15.552003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.1a","version":"150'1","reported_seq":13,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.299542+0000","last_change":"2024-02-21T23:09:53.278176+0000","last_active":"2024-02-21T23:09:54.299542+0000","last_peered":"2024-02-21T23:09:54.299542+0000","last_clean":"2024-02-21T23:09:54.299542+0000","last_became_active":"2024-02-21T23:09:53.277880+0000","last_became_peered":"2024-02-21T23:09:53.277880+0000","last_unstale":"2024-02-21T23:09:54.299542+0000","last_undegraded":"2024-02-21T23:09:54.299542+0000","last_fullsized":"2024-02-21T23:09:54.299542+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:05:08.172503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.272250+0000","last_change":"2024-02-21T23:09:54.278094+0000","last_active":"2024-02-21T23:09:55.272250+0000","last_peered":"2024-02-21T23:09:55.272250+0000","last_clean":"2024-02-21T23:09:55.272250+0000","last_became_active":"2024-02-21T23:09:54.276160+0000","last_became_peered":"2024-02-21T23:09:54.276160+0000","last_unstale":"2024-02-21T23:09:55.272250+0000","last_undegraded":"2024-02-21T23:09:55.272250+0000","last_fullsized":"2024-02-21T23:09:55.272250+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:15:07.339478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":14,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.271567+0000","last_change":"2024-02-21T23:09:53.280005+0000","last_active":"2024-02-21T23:09:55.271567+0000","last_peered":"2024-02-21T23:09:55.271567+0000","last_clean":"2024-02-21T23:09:55.271567+0000","last_became_active":"2024-02-21T23:09:53.279689+0000","last_became_peered":"2024-02-21T23:09:53.279689+0000","last_unstale":"2024-02-21T23:09:55.271567+0000","last_undegraded":"2024-02-21T23:09:55.271567+0000","last_fullsized":"2024-02-21T23:09:55.271567+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:40:05.561558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.276563+0000","last_change":"2024-02-21T23:09:54.262565+0000","last_active":"2024-02-21T23:09:55.276563+0000","last_peered":"2024-02-21T23:09:55.276563+0000","last_clean":"2024-02-21T23:09:55.276563+0000","last_became_active":"2024-02-21T23:09:54.261897+0000","last_became_peered":"2024-02-21T23:09:54.261897+0000","last_unstale":"2024-02-21T23:09:55.276563+0000","last_undegraded":"2024-02-21T23:09:55.276563+0000","last_fullsized":"2024-02-21T23:09:55.276563+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:08:50.519955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":14,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.290103+0000","last_change":"2024-02-21T23:09:53.275695+0000","last_active":"2024-02-21T23:09:55.290103+0000","last_peered":"2024-02-21T23:09:55.290103+0000","last_clean":"2024-02-21T23:09:55.290103+0000","last_became_active":"2024-02-21T23:09:53.275343+0000","last_became_peered":"2024-02-21T23:09:53.275343+0000","last_unstale":"2024-02-21T23:09:55.290103+0000","last_undegraded":"2024-02-21T23:09:55.290103+0000","last_fullsized":"2024-02-21T23:09:55.290103+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:28:53.062636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.286660+0000","last_change":"2024-02-21T23:09:54.279220+0000","last_active":"2024-02-21T23:09:54.286660+0000","last_peered":"2024-02-21T23:09:54.286660+0000","last_clean":"2024-02-21T23:09:54.286660+0000","last_became_active":"2024-02-21T23:09:54.278653+0000","last_became_peered":"2024-02-21T23:09:54.278653+0000","last_unstale":"2024-02-21T23:09:54.286660+0000","last_undegraded":"2024-02-21T23:09:54.286660+0000","last_fullsized":"2024-02-21T23:09:54.286660+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:18:40.139538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":14,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.273576+0000","last_change":"2024-02-21T23:09:53.288439+0000","last_active":"2024-02-21T23:09:55.273576+0000","last_peered":"2024-02-21T23:09:55.273576+0000","last_clean":"2024-02-21T23:09:55.273576+0000","last_became_active":"2024-02-21T23:09:53.287601+0000","last_became_peered":"2024-02-21T23:09:53.287601+0000","last_unstale":"2024-02-21T23:09:55.273576+0000","last_undegraded":"2024-02-21T23:09:55.273576+0000","last_fullsized":"2024-02-21T23:09:55.273576+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:32:05.045159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.273172+0000","last_change":"2024-02-21T23:09:54.280274+0000","last_active":"2024-02-21T23:09:55.273172+0000","last_peered":"2024-02-21T23:09:55.273172+0000","last_clean":"2024-02-21T23:09:55.273172+0000","last_became_active":"2024-02-21T23:09:54.279951+0000","last_became_peered":"2024-02-21T23:09:54.279951+0000","last_unstale":"2024-02-21T23:09:55.273172+0000","last_undegraded":"2024-02-21T23:09:55.273172+0000","last_fullsized":"2024-02-21T23:09:55.273172+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:15:08.084683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.15","version":"150'3","reported_seq":17,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.275910+0000","last_change":"2024-02-21T23:09:53.279389+0000","last_active":"2024-02-21T23:09:55.275910+0000","last_peered":"2024-02-21T23:09:55.275910+0000","last_clean":"2024-02-21T23:09:55.275910+0000","last_became_active":"2024-02-21T23:09:53.279042+0000","last_became_peered":"2024-02-21T23:09:53.279042+0000","last_unstale":"2024-02-21T23:09:55.275910+0000","last_undegraded":"2024-02-21T23:09:55.275910+0000","last_fullsized":"2024-02-21T23:09:55.275910+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:44:44.898339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1b","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.272168+0000","last_change":"2024-02-21T23:09:54.285563+0000","last_active":"2024-02-21T23:09:55.272168+0000","last_peered":"2024-02-21T23:09:55.272168+0000","last_clean":"2024-02-21T23:09:55.272168+0000","last_became_active":"2024-02-21T23:09:54.285275+0000","last_became_peered":"2024-02-21T23:09:54.285275+0000","last_unstale":"2024-02-21T23:09:55.272168+0000","last_undegraded":"2024-02-21T23:09:55.272168+0000","last_fullsized":"2024-02-21T23:09:55.272168+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:21:42.014406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"150'6","reported_seq":20,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.273813+0000","last_change":"2024-02-21T23:09:53.288423+0000","last_active":"2024-02-21T23:09:55.273813+0000","last_peered":"2024-02-21T23:09:55.273813+0000","last_clean":"2024-02-21T23:09:55.273813+0000","last_became_active":"2024-02-21T23:09:53.287473+0000","last_became_peered":"2024-02-21T23:09:53.287473+0000","last_unstale":"2024-02-21T23:09:55.273813+0000","last_undegraded":"2024-02-21T23:09:55.273813+0000","last_fullsized":"2024-02-21T23:09:55.273813+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:13:32.425310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1c","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.526152+0000","last_change":"2024-02-21T23:09:54.278618+0000","last_active":"2024-02-21T23:09:55.526152+0000","last_peered":"2024-02-21T23:09:55.526152+0000","last_clean":"2024-02-21T23:09:55.526152+0000","last_became_active":"2024-02-21T23:09:54.278343+0000","last_became_peered":"2024-02-21T23:09:54.278343+0000","last_unstale":"2024-02-21T23:09:55.526152+0000","last_undegraded":"2024-02-21T23:09:55.526152+0000","last_fullsized":"2024-02-21T23:09:55.526152+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:49:31.759374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.13","version":"150'2","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.304290+0000","last_change":"2024-02-21T23:09:53.284172+0000","last_active":"2024-02-21T23:09:54.304290+0000","last_peered":"2024-02-21T23:09:54.304290+0000","last_clean":"2024-02-21T23:09:54.304290+0000","last_became_active":"2024-02-21T23:09:53.283507+0000","last_became_peered":"2024-02-21T23:09:53.283507+0000","last_unstale":"2024-02-21T23:09:54.304290+0000","last_undegraded":"2024-02-21T23:09:54.304290+0000","last_fullsized":"2024-02-21T23:09:54.304290+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:20:30.108767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1d","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.290794+0000","last_change":"2024-02-21T23:09:54.291717+0000","last_active":"2024-02-21T23:09:55.290794+0000","last_peered":"2024-02-21T23:09:55.290794+0000","last_clean":"2024-02-21T23:09:55.290794+0000","last_became_active":"2024-02-21T23:09:54.290842+0000","last_became_peered":"2024-02-21T23:09:54.290842+0000","last_unstale":"2024-02-21T23:09:55.290794+0000","last_undegraded":"2024-02-21T23:09:55.290794+0000","last_fullsized":"2024-02-21T23:09:55.290794+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:42:16.988326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":14,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.273465+0000","last_change":"2024-02-21T23:09:53.284733+0000","last_active":"2024-02-21T23:09:55.273465+0000","last_peered":"2024-02-21T23:09:55.273465+0000","last_clean":"2024-02-21T23:09:55.273465+0000","last_became_active":"2024-02-21T23:09:53.284399+0000","last_became_peered":"2024-02-21T23:09:53.284399+0000","last_unstale":"2024-02-21T23:09:55.273465+0000","last_undegraded":"2024-02-21T23:09:55.273465+0000","last_fullsized":"2024-02-21T23:09:55.273465+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:05:08.172503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1e","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.283361+0000","last_change":"2024-02-21T23:09:54.277339+0000","last_active":"2024-02-21T23:09:54.283361+0000","last_peered":"2024-02-21T23:09:54.283361+0000","last_clean":"2024-02-21T23:09:54.283361+0000","last_became_active":"2024-02-21T23:09:54.277039+0000","last_became_peered":"2024-02-21T23:09:54.277039+0000","last_unstale":"2024-02-21T23:09:54.283361+0000","last_undegraded":"2024-02-21T23:09:54.283361+0000","last_fullsized":"2024-02-21T23:09:54.283361+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:29:25.643198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":12,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.249482+0000","last_change":"2024-02-21T23:09:53.283915+0000","last_active":"2024-02-21T23:09:54.249482+0000","last_peered":"2024-02-21T23:09:54.249482+0000","last_clean":"2024-02-21T23:09:54.249482+0000","last_became_active":"2024-02-21T23:09:53.283532+0000","last_became_peered":"2024-02-21T23:09:53.283532+0000","last_unstale":"2024-02-21T23:09:54.249482+0000","last_undegraded":"2024-02-21T23:09:54.249482+0000","last_fullsized":"2024-02-21T23:09:54.249482+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:55:17.574145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1f","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.272553+0000","last_change":"2024-02-21T23:09:54.265384+0000","last_active":"2024-02-21T23:09:55.272553+0000","last_peered":"2024-02-21T23:09:55.272553+0000","last_clean":"2024-02-21T23:09:55.272553+0000","last_became_active":"2024-02-21T23:09:54.265073+0000","last_became_peered":"2024-02-21T23:09:54.265073+0000","last_unstale":"2024-02-21T23:09:55.272553+0000","last_undegraded":"2024-02-21T23:09:55.272553+0000","last_fullsized":"2024-02-21T23:09:55.272553+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:44:28.295055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.10","version":"150'8","reported_seq":22,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.290161+0000","last_change":"2024-02-21T23:09:53.285246+0000","last_active":"2024-02-21T23:09:55.290161+0000","last_peered":"2024-02-21T23:09:55.290161+0000","last_clean":"2024-02-21T23:09:55.290161+0000","last_became_active":"2024-02-21T23:09:53.282084+0000","last_became_peered":"2024-02-21T23:09:53.282084+0000","last_unstale":"2024-02-21T23:09:55.290161+0000","last_undegraded":"2024-02-21T23:09:55.290161+0000","last_fullsized":"2024-02-21T23:09:55.290161+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:23:26.494398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.289952+0000","last_change":"2024-02-21T23:09:54.289615+0000","last_active":"2024-02-21T23:09:55.289952+0000","last_peered":"2024-02-21T23:09:55.289952+0000","last_clean":"2024-02-21T23:09:55.289952+0000","last_became_active":"2024-02-21T23:09:54.289322+0000","last_became_peered":"2024-02-21T23:09:54.289322+0000","last_unstale":"2024-02-21T23:09:55.289952+0000","last_undegraded":"2024-02-21T23:09:55.289952+0000","last_fullsized":"2024-02-21T23:09:55.289952+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:34:28.642087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.f","version":"150'1","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.274043+0000","last_change":"2024-02-21T23:09:53.287210+0000","last_active":"2024-02-21T23:09:55.274043+0000","last_peered":"2024-02-21T23:09:55.274043+0000","last_clean":"2024-02-21T23:09:55.274043+0000","last_became_active":"2024-02-21T23:09:53.285372+0000","last_became_peered":"2024-02-21T23:09:53.285372+0000","last_unstale":"2024-02-21T23:09:55.274043+0000","last_undegraded":"2024-02-21T23:09:55.274043+0000","last_fullsized":"2024-02-21T23:09:55.274043+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:20:30.108767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.526125+0000","last_change":"2024-02-21T23:09:54.274363+0000","last_active":"2024-02-21T23:09:55.526125+0000","last_peered":"2024-02-21T23:09:55.526125+0000","last_clean":"2024-02-21T23:09:55.526125+0000","last_became_active":"2024-02-21T23:09:54.274084+0000","last_became_peered":"2024-02-21T23:09:54.274084+0000","last_unstale":"2024-02-21T23:09:55.526125+0000","last_undegraded":"2024-02-21T23:09:55.526125+0000","last_fullsized":"2024-02-21T23:09:55.526125+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:42:19.996155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.e","version":"150'2","reported_seq":16,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.291139+0000","last_change":"2024-02-21T23:09:53.275941+0000","last_active":"2024-02-21T23:09:55.291139+0000","last_peered":"2024-02-21T23:09:55.291139+0000","last_clean":"2024-02-21T23:09:55.291139+0000","last_became_active":"2024-02-21T23:09:53.275555+0000","last_became_peered":"2024-02-21T23:09:53.275555+0000","last_unstale":"2024-02-21T23:09:55.291139+0000","last_undegraded":"2024-02-21T23:09:55.291139+0000","last_fullsized":"2024-02-21T23:09:55.291139+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:02:10.230470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.262950+0000","last_change":"2024-02-21T23:09:54.259630+0000","last_active":"2024-02-21T23:09:54.262950+0000","last_peered":"2024-02-21T23:09:54.262950+0000","last_clean":"2024-02-21T23:09:54.262950+0000","last_became_active":"2024-02-21T23:09:54.258924+0000","last_became_peered":"2024-02-21T23:09:54.258924+0000","last_unstale":"2024-02-21T23:09:54.262950+0000","last_undegraded":"2024-02-21T23:09:54.262950+0000","last_fullsized":"2024-02-21T23:09:54.262950+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:18:40.139538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.d","version":"150'1","reported_seq":13,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.302851+0000","last_change":"2024-02-21T23:09:53.279385+0000","last_active":"2024-02-21T23:09:54.302851+0000","last_peered":"2024-02-21T23:09:54.302851+0000","last_clean":"2024-02-21T23:09:54.302851+0000","last_became_active":"2024-02-21T23:09:53.278074+0000","last_became_peered":"2024-02-21T23:09:53.278074+0000","last_unstale":"2024-02-21T23:09:54.302851+0000","last_undegraded":"2024-02-21T23:09:54.302851+0000","last_fullsized":"2024-02-21T23:09:54.302851+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:42:15.971409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.283275+0000","last_change":"2024-02-21T23:09:54.277296+0000","last_active":"2024-02-21T23:09:54.283275+0000","last_peered":"2024-02-21T23:09:54.283275+0000","last_clean":"2024-02-21T23:09:54.283275+0000","last_became_active":"2024-02-21T23:09:54.276973+0000","last_became_peered":"2024-02-21T23:09:54.276973+0000","last_unstale":"2024-02-21T23:09:54.283275+0000","last_undegraded":"2024-02-21T23:09:54.283275+0000","last_fullsized":"2024-02-21T23:09:54.283275+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:32:13.492141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":14,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.290715+0000","last_change":"2024-02-21T23:09:53.281804+0000","last_active":"2024-02-21T23:09:55.290715+0000","last_peered":"2024-02-21T23:09:55.290715+0000","last_clean":"2024-02-21T23:09:55.290715+0000","last_became_active":"2024-02-21T23:09:53.278643+0000","last_became_peered":"2024-02-21T23:09:53.278643+0000","last_unstale":"2024-02-21T23:09:55.290715+0000","last_undegraded":"2024-02-21T23:09:55.290715+0000","last_fullsized":"2024-02-21T23:09:55.290715+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:55:17.574145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.273827+0000","last_change":"2024-02-21T23:09:54.285141+0000","last_active":"2024-02-21T23:09:55.273827+0000","last_peered":"2024-02-21T23:09:55.273827+0000","last_clean":"2024-02-21T23:09:55.273827+0000","last_became_active":"2024-02-21T23:09:54.284498+0000","last_became_peered":"2024-02-21T23:09:54.284498+0000","last_unstale":"2024-02-21T23:09:55.273827+0000","last_undegraded":"2024-02-21T23:09:55.273827+0000","last_fullsized":"2024-02-21T23:09:55.273827+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:26:36.405020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.b","version":"150'2","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.301072+0000","last_change":"2024-02-21T23:09:53.284284+0000","last_active":"2024-02-21T23:09:54.301072+0000","last_peered":"2024-02-21T23:09:54.301072+0000","last_clean":"2024-02-21T23:09:54.301072+0000","last_became_active":"2024-02-21T23:09:53.283043+0000","last_became_peered":"2024-02-21T23:09:53.283043+0000","last_unstale":"2024-02-21T23:09:54.301072+0000","last_undegraded":"2024-02-21T23:09:54.301072+0000","last_fullsized":"2024-02-21T23:09:54.301072+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:32:05.045159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.287916+0000","last_change":"2024-02-21T23:09:54.275082+0000","last_active":"2024-02-21T23:09:54.287916+0000","last_peered":"2024-02-21T23:09:54.287916+0000","last_clean":"2024-02-21T23:09:54.287916+0000","last_became_active":"2024-02-21T23:09:54.274707+0000","last_became_peered":"2024-02-21T23:09:54.274707+0000","last_unstale":"2024-02-21T23:09:54.287916+0000","last_undegraded":"2024-02-21T23:09:54.287916+0000","last_fullsized":"2024-02-21T23:09:54.287916+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:11:44.589606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":12,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.247863+0000","last_change":"2024-02-21T23:09:53.277418+0000","last_active":"2024-02-21T23:09:54.247863+0000","last_peered":"2024-02-21T23:09:54.247863+0000","last_clean":"2024-02-21T23:09:54.247863+0000","last_became_active":"2024-02-21T23:09:53.277111+0000","last_became_peered":"2024-02-21T23:09:53.277111+0000","last_unstale":"2024-02-21T23:09:54.247863+0000","last_undegraded":"2024-02-21T23:09:54.247863+0000","last_fullsized":"2024-02-21T23:09:54.247863+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:23:26.494398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.281830+0000","last_change":"2024-02-21T23:09:54.280400+0000","last_active":"2024-02-21T23:09:54.281830+0000","last_peered":"2024-02-21T23:09:54.281830+0000","last_clean":"2024-02-21T23:09:54.281830+0000","last_became_active":"2024-02-21T23:09:54.280102+0000","last_became_peered":"2024-02-21T23:09:54.280102+0000","last_unstale":"2024-02-21T23:09:54.281830+0000","last_undegraded":"2024-02-21T23:09:54.281830+0000","last_fullsized":"2024-02-21T23:09:54.281830+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:59:59.717557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":14,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.272363+0000","last_change":"2024-02-21T23:09:53.282939+0000","last_active":"2024-02-21T23:09:55.272363+0000","last_peered":"2024-02-21T23:09:55.272363+0000","last_clean":"2024-02-21T23:09:55.272363+0000","last_became_active":"2024-02-21T23:09:53.282612+0000","last_became_peered":"2024-02-21T23:09:53.282612+0000","last_unstale":"2024-02-21T23:09:55.272363+0000","last_undegraded":"2024-02-21T23:09:55.272363+0000","last_fullsized":"2024-02-21T23:09:55.272363+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:30:40.609835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.287337+0000","last_change":"2024-02-21T23:09:54.282302+0000","last_active":"2024-02-21T23:09:54.287337+0000","last_peered":"2024-02-21T23:09:54.287337+0000","last_clean":"2024-02-21T23:09:54.287337+0000","last_became_active":"2024-02-21T23:09:54.280468+0000","last_became_peered":"2024-02-21T23:09:54.280468+0000","last_unstale":"2024-02-21T23:09:54.287337+0000","last_undegraded":"2024-02-21T23:09:54.287337+0000","last_fullsized":"2024-02-21T23:09:54.287337+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:34:50.354067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":12,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.268910+0000","last_change":"2024-02-21T23:09:53.283837+0000","last_active":"2024-02-21T23:09:54.268910+0000","last_peered":"2024-02-21T23:09:54.268910+0000","last_clean":"2024-02-21T23:09:54.268910+0000","last_became_active":"2024-02-21T23:09:53.283541+0000","last_became_peered":"2024-02-21T23:09:53.283541+0000","last_unstale":"2024-02-21T23:09:54.268910+0000","last_undegraded":"2024-02-21T23:09:54.268910+0000","last_fullsized":"2024-02-21T23:09:54.268910+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:28:53.062636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.283635+0000","last_change":"2024-02-21T23:09:54.274229+0000","last_active":"2024-02-21T23:09:54.283635+0000","last_peered":"2024-02-21T23:09:54.283635+0000","last_clean":"2024-02-21T23:09:54.283635+0000","last_became_active":"2024-02-21T23:09:54.271365+0000","last_became_peered":"2024-02-21T23:09:54.271365+0000","last_unstale":"2024-02-21T23:09:54.283635+0000","last_undegraded":"2024-02-21T23:09:54.283635+0000","last_fullsized":"2024-02-21T23:09:54.283635+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:08:50.519955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.7","version":"150'2","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.303030+0000","last_change":"2024-02-21T23:09:53.283873+0000","last_active":"2024-02-21T23:09:54.303030+0000","last_peered":"2024-02-21T23:09:54.303030+0000","last_clean":"2024-02-21T23:09:54.303030+0000","last_became_active":"2024-02-21T23:09:53.282829+0000","last_became_peered":"2024-02-21T23:09:53.282829+0000","last_unstale":"2024-02-21T23:09:54.303030+0000","last_undegraded":"2024-02-21T23:09:54.303030+0000","last_fullsized":"2024-02-21T23:09:54.303030+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:13:32.425310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.290237+0000","last_change":"2024-02-21T23:09:54.277479+0000","last_active":"2024-02-21T23:09:55.290237+0000","last_peered":"2024-02-21T23:09:55.290237+0000","last_clean":"2024-02-21T23:09:55.290237+0000","last_became_active":"2024-02-21T23:09:54.277065+0000","last_became_peered":"2024-02-21T23:09:54.277065+0000","last_unstale":"2024-02-21T23:09:55.290237+0000","last_undegraded":"2024-02-21T23:09:55.290237+0000","last_fullsized":"2024-02-21T23:09:55.290237+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:07:24.304915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.6","version":"151'1","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.289075+0000","last_change":"2024-02-21T23:09:53.285874+0000","last_active":"2024-02-21T23:09:55.289075+0000","last_peered":"2024-02-21T23:09:55.289075+0000","last_clean":"2024-02-21T23:09:55.289075+0000","last_became_active":"2024-02-21T23:09:53.285569+0000","last_became_peered":"2024-02-21T23:09:53.285569+0000","last_unstale":"2024-02-21T23:09:55.289075+0000","last_undegraded":"2024-02-21T23:09:55.289075+0000","last_fullsized":"2024-02-21T23:09:55.289075+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:52:56.873078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.272128+0000","last_change":"2024-02-21T23:09:54.286102+0000","last_active":"2024-02-21T23:09:55.272128+0000","last_peered":"2024-02-21T23:09:55.272128+0000","last_clean":"2024-02-21T23:09:55.272128+0000","last_became_active":"2024-02-21T23:09:54.283073+0000","last_became_peered":"2024-02-21T23:09:54.283073+0000","last_unstale":"2024-02-21T23:09:55.272128+0000","last_undegraded":"2024-02-21T23:09:55.272128+0000","last_fullsized":"2024-02-21T23:09:55.272128+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:39:05.096862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":14,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.273794+0000","last_change":"2024-02-21T23:09:53.285730+0000","last_active":"2024-02-21T23:09:55.273794+0000","last_peered":"2024-02-21T23:09:55.273794+0000","last_clean":"2024-02-21T23:09:55.273794+0000","last_became_active":"2024-02-21T23:09:53.285373+0000","last_became_peered":"2024-02-21T23:09:53.285373+0000","last_unstale":"2024-02-21T23:09:55.273794+0000","last_undegraded":"2024-02-21T23:09:55.273794+0000","last_fullsized":"2024-02-21T23:09:55.273794+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:04:15.300172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.b","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.290256+0000","last_change":"2024-02-21T23:09:54.277483+0000","last_active":"2024-02-21T23:09:55.290256+0000","last_peered":"2024-02-21T23:09:55.290256+0000","last_clean":"2024-02-21T23:09:55.290256+0000","last_became_active":"2024-02-21T23:09:54.277170+0000","last_became_peered":"2024-02-21T23:09:54.277170+0000","last_unstale":"2024-02-21T23:09:55.290256+0000","last_undegraded":"2024-02-21T23:09:55.290256+0000","last_fullsized":"2024-02-21T23:09:55.290256+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:33:07.362294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":14,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.273255+0000","last_change":"2024-02-21T23:09:53.267455+0000","last_active":"2024-02-21T23:09:55.273255+0000","last_peered":"2024-02-21T23:09:55.273255+0000","last_clean":"2024-02-21T23:09:55.273255+0000","last_became_active":"2024-02-21T23:09:53.266974+0000","last_became_peered":"2024-02-21T23:09:53.266974+0000","last_unstale":"2024-02-21T23:09:55.273255+0000","last_undegraded":"2024-02-21T23:09:55.273255+0000","last_fullsized":"2024-02-21T23:09:55.273255+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:33:26.269092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.276551+0000","last_change":"2024-02-21T23:09:54.275273+0000","last_active":"2024-02-21T23:09:55.276551+0000","last_peered":"2024-02-21T23:09:55.276551+0000","last_clean":"2024-02-21T23:09:55.276551+0000","last_became_active":"2024-02-21T23:09:54.274287+0000","last_became_peered":"2024-02-21T23:09:54.274287+0000","last_unstale":"2024-02-21T23:09:55.276551+0000","last_undegraded":"2024-02-21T23:09:55.276551+0000","last_fullsized":"2024-02-21T23:09:55.276551+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:18:40.139538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":14,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.275660+0000","last_change":"2024-02-21T23:09:53.279782+0000","last_active":"2024-02-21T23:09:55.275660+0000","last_peered":"2024-02-21T23:09:55.275660+0000","last_clean":"2024-02-21T23:09:55.275660+0000","last_became_active":"2024-02-21T23:09:53.279477+0000","last_became_peered":"2024-02-21T23:09:53.279477+0000","last_unstale":"2024-02-21T23:09:55.275660+0000","last_undegraded":"2024-02-21T23:09:55.275660+0000","last_fullsized":"2024-02-21T23:09:55.275660+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:41:01.169723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.d","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.286643+0000","last_change":"2024-02-21T23:09:54.279409+0000","last_active":"2024-02-21T23:09:54.286643+0000","last_peered":"2024-02-21T23:09:54.286643+0000","last_clean":"2024-02-21T23:09:54.286643+0000","last_became_active":"2024-02-21T23:09:54.279107+0000","last_became_peered":"2024-02-21T23:09:54.279107+0000","last_unstale":"2024-02-21T23:09:54.286643+0000","last_undegraded":"2024-02-21T23:09:54.286643+0000","last_fullsized":"2024-02-21T23:09:54.286643+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:08:50.519955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":14,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.273539+0000","last_change":"2024-02-21T23:09:53.278959+0000","last_active":"2024-02-21T23:09:55.273539+0000","last_peered":"2024-02-21T23:09:55.273539+0000","last_clean":"2024-02-21T23:09:55.273539+0000","last_became_active":"2024-02-21T23:09:53.278673+0000","last_became_peered":"2024-02-21T23:09:53.278673+0000","last_unstale":"2024-02-21T23:09:55.273539+0000","last_undegraded":"2024-02-21T23:09:55.273539+0000","last_fullsized":"2024-02-21T23:09:55.273539+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:42:15.971409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.e","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.273582+0000","last_change":"2024-02-21T23:09:54.282920+0000","last_active":"2024-02-21T23:09:55.273582+0000","last_peered":"2024-02-21T23:09:55.273582+0000","last_clean":"2024-02-21T23:09:55.273582+0000","last_became_active":"2024-02-21T23:09:54.282544+0000","last_became_peered":"2024-02-21T23:09:54.282544+0000","last_unstale":"2024-02-21T23:09:55.273582+0000","last_undegraded":"2024-02-21T23:09:55.273582+0000","last_fullsized":"2024-02-21T23:09:55.273582+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:40:05.206906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":12,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.249891+0000","last_change":"2024-02-21T23:09:53.275586+0000","last_active":"2024-02-21T23:09:54.249891+0000","last_peered":"2024-02-21T23:09:54.249891+0000","last_clean":"2024-02-21T23:09:54.249891+0000","last_became_active":"2024-02-21T23:09:53.274828+0000","last_became_peered":"2024-02-21T23:09:53.274828+0000","last_unstale":"2024-02-21T23:09:54.249891+0000","last_undegraded":"2024-02-21T23:09:54.249891+0000","last_fullsized":"2024-02-21T23:09:54.249891+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:02:10.230470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.f","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.272618+0000","last_change":"2024-02-21T23:09:54.277320+0000","last_active":"2024-02-21T23:09:55.272618+0000","last_peered":"2024-02-21T23:09:55.272618+0000","last_clean":"2024-02-21T23:09:55.272618+0000","last_became_active":"2024-02-21T23:09:54.276368+0000","last_became_peered":"2024-02-21T23:09:54.276368+0000","last_unstale":"2024-02-21T23:09:55.272618+0000","last_undegraded":"2024-02-21T23:09:55.272618+0000","last_fullsized":"2024-02-21T23:09:55.272618+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:40:37.285836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":12,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.260467+0000","last_change":"2024-02-21T23:09:53.278027+0000","last_active":"2024-02-21T23:09:54.260467+0000","last_peered":"2024-02-21T23:09:54.260467+0000","last_clean":"2024-02-21T23:09:54.260467+0000","last_became_active":"2024-02-21T23:09:53.277660+0000","last_became_peered":"2024-02-21T23:09:53.277660+0000","last_unstale":"2024-02-21T23:09:54.260467+0000","last_undegraded":"2024-02-21T23:09:54.260467+0000","last_fullsized":"2024-02-21T23:09:54.260467+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:07:23.287998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.284794+0000","last_change":"2024-02-21T23:09:54.277356+0000","last_active":"2024-02-21T23:09:54.284794+0000","last_peered":"2024-02-21T23:09:54.284794+0000","last_clean":"2024-02-21T23:09:54.284794+0000","last_became_active":"2024-02-21T23:09:54.277087+0000","last_became_peered":"2024-02-21T23:09:54.277087+0000","last_unstale":"2024-02-21T23:09:54.284794+0000","last_undegraded":"2024-02-21T23:09:54.284794+0000","last_fullsized":"2024-02-21T23:09:54.284794+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:55:18.591062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":14,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.273779+0000","last_change":"2024-02-21T23:09:53.286181+0000","last_active":"2024-02-21T23:09:55.273779+0000","last_peered":"2024-02-21T23:09:55.273779+0000","last_clean":"2024-02-21T23:09:55.273779+0000","last_became_active":"2024-02-21T23:09:53.285849+0000","last_became_peered":"2024-02-21T23:09:53.285849+0000","last_unstale":"2024-02-21T23:09:55.273779+0000","last_undegraded":"2024-02-21T23:09:55.273779+0000","last_fullsized":"2024-02-21T23:09:55.273779+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:51:08.543135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":415,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.268984+0000","last_change":"2024-02-21T23:02:32.289335+0000","last_active":"2024-02-21T23:09:54.268984+0000","last_peered":"2024-02-21T23:09:54.268984+0000","last_clean":"2024-02-21T23:09:54.268984+0000","last_became_active":"2024-02-21T23:02:32.286913+0000","last_became_peered":"2024-02-21T23:02:32.286913+0000","last_unstale":"2024-02-21T23:09:54.268984+0000","last_undegraded":"2024-02-21T23:09:54.268984+0000","last_fullsized":"2024-02-21T23:09:54.268984+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_clean_scrub_stamp":"2024-02-21T23:02:31.172569+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:36:23.987374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":24,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":23,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574939,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33092,"kb_used_data":1908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338748,"statfs":{"total":96636764160,"available":96602877952,"internally_reserved":0,"allocated":1953792,"data_stored":1509313,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574940,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33036,"kb_used_data":1856,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338804,"statfs":{"total":96636764160,"available":96602935296,"internally_reserved":0,"allocated":1900544,"data_stored":925637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574939,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33032,"kb_used_data":1852,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338808,"statfs":{"total":96636764160,"available":96602939392,"internally_reserved":0,"allocated":1896448,"data_stored":919069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574940,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33040,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338800,"statfs":{"total":96636764160,"available":96602931200,"internally_reserved":0,"allocated":1904640,"data_stored":919709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574940,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33060,"kb_used_data":1876,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338780,"statfs":{"total":96636764160,"available":96602910720,"internally_reserved":0,"allocated":1921024,"data_stored":927853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574941,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33040,"kb_used_data":1856,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338800,"statfs":{"total":96636764160,"available":96602931200,"internally_reserved":0,"allocated":1900544,"data_stored":925637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574939,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33040,"kb_used_data":1856,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338800,"statfs":{"total":96636764160,"available":96602931200,"internally_reserved":0,"allocated":1900544,"data_stored":920557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574939,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33100,"kb_used_data":1916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338740,"statfs":{"total":96636764160,"available":96602869760,"internally_reserved":0,"allocated":1961984,"data_stored":1509417,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-21T23:09:58.734 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-21T23:09:58.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.999+0000 7fb4cbe00640 1 Processor -- start 2024-02-21T23:09:58.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:58.999+0000 7fb4cbe00640 1 -- start start 2024-02-21T23:09:58.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.001+0000 7fb4cbe00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fb4cc06e3d0 0x7fb4cc14b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:58.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.001+0000 7fb4cbe00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4cc0a8b40 con 0x7fb4cc0c0fe0 2024-02-21T23:09:58.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.001+0000 7fb4cbe00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4cc06e7d0 con 0x7fb4cc06dff0 2024-02-21T23:09:58.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.001+0000 7fb4cbe00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4cc0a8e00 con 0x7fb4cc06e3d0 2024-02-21T23:09:58.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.001+0000 7fb4ca000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb4cc0c0fe0 0x7fb4cc158580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34352/0 (socket says 172.21.15.28:34352) 2024-02-21T23:09:58.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.001+0000 7fb4ca000640 1 -- 172.21.15.28:0/910906527 learned_addr learned my addr 172.21.15.28:0/910906527 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:09:58.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.002+0000 7fb4cb400640 1 --2- 172.21.15.28:0/910906527 >> v2:172.21.15.81:3300/0 conn(0x7fb4cc06e3d0 0x7fb4cc14b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:58.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.002+0000 7fb4c9600640 1 -- 172.21.15.28:0/910906527 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1270288013 0 0) 0x7fb4cc0a8b40 con 0x7fb4cc0c0fe0 2024-02-21T23:09:58.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.002+0000 7fb4c9600640 1 -- 172.21.15.28:0/910906527 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4b0003520 con 0x7fb4cc0c0fe0 2024-02-21T23:09:58.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.002+0000 7fb4c9600640 1 -- 172.21.15.28:0/910906527 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2506797621 0 0) 0x7fb4b0003520 con 0x7fb4cc0c0fe0 2024-02-21T23:09:58.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.002+0000 7fb4c9600640 1 -- 172.21.15.28:0/910906527 >> v2:172.21.15.81:3300/0 conn(0x7fb4cc06e3d0 msgr2=0x7fb4cc14b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:58.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.002+0000 7fb4c9600640 1 --2- 172.21.15.28:0/910906527 >> v2:172.21.15.81:3300/0 conn(0x7fb4cc06e3d0 0x7fb4cc14b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:58.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.002+0000 7fb4c9600640 1 -- 172.21.15.28:0/910906527 >> v1:172.21.15.81:6789/0 conn(0x7fb4cc06dff0 legacy=0x7fb4cc071350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:58.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.002+0000 7fb4c9600640 1 -- 172.21.15.28:0/910906527 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4cc0a90e0 con 0x7fb4cc0c0fe0 2024-02-21T23:09:58.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.002+0000 7fb4c9600640 1 -- 172.21.15.28:0/910906527 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fb4b8001670 con 0x7fb4cc0c0fe0 2024-02-21T23:09:58.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.002+0000 7fb4c9600640 1 -- 172.21.15.28:0/910906527 >> v1:172.21.15.28:6789/0 conn(0x7fb4cc0c0fe0 legacy=0x7fb4cc158580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:58.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.002+0000 7fb4c9600640 1 --2- 172.21.15.28:0/910906527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4b00039b0 0x7fb4b0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:58.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.002+0000 7fb4c9600640 1 --2- 172.21.15.28:0/910906527 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb4b0004800 0x7fb4b0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:58.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.002+0000 7fb4c9600640 1 --2- 172.21.15.28:0/910906527 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb4b00051c0 0x7fb4b000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:58.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.002+0000 7fb4c9600640 1 -- 172.21.15.28:0/910906527 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4cc0a8e00 con 0x7fb4b00039b0 2024-02-21T23:09:58.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.002+0000 7fb4c9600640 1 -- 172.21.15.28:0/910906527 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b000c060 con 0x7fb4b00051c0 2024-02-21T23:09:58.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.002+0000 7fb4c9600640 1 -- 172.21.15.28:0/910906527 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b000c260 con 0x7fb4b0004800 2024-02-21T23:09:58.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.003+0000 7fb4caa00640 1 --2- 172.21.15.28:0/910906527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4b00039b0 0x7fb4b0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:58.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.003+0000 7fb4cb400640 1 --2- 172.21.15.28:0/910906527 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb4b00051c0 0x7fb4b000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:58.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.003+0000 7fb4caa00640 1 -- 172.21.15.28:0/910906527 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb4b0004800 msgr2=0x7fb4b0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:58.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.003+0000 7fb4ca000640 1 --2- 172.21.15.28:0/910906527 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb4b0004800 0x7fb4b0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:58.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.003+0000 7fb4caa00640 1 --2- 172.21.15.28:0/910906527 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb4b0004800 0x7fb4b0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:58.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.003+0000 7fb4caa00640 1 -- 172.21.15.28:0/910906527 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb4b00051c0 msgr2=0x7fb4b000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:58.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.003+0000 7fb4caa00640 1 --2- 172.21.15.28:0/910906527 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb4b00051c0 0x7fb4b000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:58.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.003+0000 7fb4caa00640 1 -- 172.21.15.28:0/910906527 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4b0003520 con 0x7fb4b00039b0 2024-02-21T23:09:58.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.003+0000 7fb4cb400640 1 --2- 172.21.15.28:0/910906527 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb4b00051c0 0x7fb4b000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:09:58.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.003+0000 7fb4cb400640 1 -- 172.21.15.28:0/910906527 reap_dead start 2024-02-21T23:09:58.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.003+0000 7fb4ca000640 1 --2- 172.21.15.28:0/910906527 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb4b0004800 0x7fb4b0004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:09:58.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.004+0000 7fb4caa00640 1 --2- 172.21.15.28:0/910906527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4b00039b0 0x7fb4b0003dd0 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc001750 tx=0x7fb4b0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55e22a8b8f2855d server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:58.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.005+0000 7fb4c9600640 1 -- 172.21.15.28:0/910906527 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4bc001370 con 0x7fb4b00039b0 2024-02-21T23:09:58.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.005+0000 7fb4c9600640 1 -- 172.21.15.28:0/910906527 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4bc0052d0 con 0x7fb4b00039b0 2024-02-21T23:09:58.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.005+0000 7fb4c9600640 1 -- 172.21.15.28:0/910906527 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4bc003830 con 0x7fb4b00039b0 2024-02-21T23:09:58.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.005+0000 7fb4cbe00640 1 -- 172.21.15.28:0/910906527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4b00039b0 msgr2=0x7fb4b0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:58.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.005+0000 7fb4cbe00640 1 --2- 172.21.15.28:0/910906527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4b00039b0 0x7fb4b0003dd0 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc001750 tx=0x7fb4b0005830 comp rx=0 tx=0).stop 2024-02-21T23:09:58.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.005+0000 7fb4cbe00640 1 -- 172.21.15.28:0/910906527 shutdown_connections 2024-02-21T23:09:58.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.005+0000 7fb4cbe00640 1 --2- 172.21.15.28:0/910906527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4b00039b0 0x7fb4b0003dd0 unknown :-1 s=CLOSED pgs=1691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:58.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.005+0000 7fb4cbe00640 1 -- 172.21.15.28:0/910906527 >> 172.21.15.28:0/910906527 conn(0x7fb4cc06b630 msgr2=0x7fb4cc0c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:58.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.006+0000 7fb4cbe00640 1 -- 172.21.15.28:0/910906527 shutdown_connections 2024-02-21T23:09:58.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.006+0000 7fb4cbe00640 1 -- 172.21.15.28:0/910906527 wait complete. 2024-02-21T23:09:58.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.007+0000 7fb4cbe00640 1 Processor -- start 2024-02-21T23:09:58.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.007+0000 7fb4cbe00640 1 -- start start 2024-02-21T23:09:58.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.008+0000 7fb4cbe00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb4b00039b0 0x7fb4cc151790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:58.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.008+0000 7fb4cbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4cc151cd0 0x7fb4cc153b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:58.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.008+0000 7fb4cbe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb4cc0a7fd0 0x7fb4cc1540c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:58.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.008+0000 7fb4cbe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4cc073520 con 0x7fb4cc151cd0 2024-02-21T23:09:58.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.008+0000 7fb4cbe00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4cc0731b0 con 0x7fb4cc0a7fd0 2024-02-21T23:09:58.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.008+0000 7fb4cbe00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4cc071350 con 0x7fb4b00039b0 2024-02-21T23:09:58.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.009+0000 7fb4ca000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4cc151cd0 0x7fb4cc153b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:58.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.009+0000 7fb4ca000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4cc151cd0 0x7fb4cc153b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33802/0 (socket says 172.21.15.28:33802) 2024-02-21T23:09:58.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.009+0000 7fb4ca000640 1 -- 172.21.15.28:0/574109384 learned_addr learned my addr 172.21.15.28:0/574109384 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:58.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.009+0000 7fb4caa00640 1 --2- 172.21.15.28:0/574109384 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb4b00039b0 0x7fb4cc151790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:58.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.009+0000 7fb4cb400640 1 --2- 172.21.15.28:0/574109384 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb4cc0a7fd0 0x7fb4cc1540c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:58.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.009+0000 7fb4ca000640 1 -- 172.21.15.28:0/574109384 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb4b00039b0 msgr2=0x7fb4cc151790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:58.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.009+0000 7fb4ca000640 1 --2- 172.21.15.28:0/574109384 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb4b00039b0 0x7fb4cc151790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:58.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.009+0000 7fb4ca000640 1 -- 172.21.15.28:0/574109384 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb4cc0a7fd0 msgr2=0x7fb4cc1540c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:58.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.009+0000 7fb4ca000640 1 --2- 172.21.15.28:0/574109384 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb4cc0a7fd0 0x7fb4cc1540c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:58.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.009+0000 7fb4ca000640 1 -- 172.21.15.28:0/574109384 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4cc14f5c0 con 0x7fb4cc151cd0 2024-02-21T23:09:58.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.009+0000 7fb4caa00640 1 --2- 172.21.15.28:0/574109384 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb4b00039b0 0x7fb4cc151790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:09:58.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.010+0000 7fb4ca000640 1 --2- 172.21.15.28:0/574109384 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4cc151cd0 0x7fb4cc153b80 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7fb4b800d770 tx=0x7fb4b800db70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:58.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.010+0000 7fb4c2a00640 1 -- 172.21.15.28:0/574109384 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4b8019070 con 0x7fb4cc151cd0 2024-02-21T23:09:58.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.010+0000 7fb4c2a00640 1 -- 172.21.15.28:0/574109384 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4b80072d0 con 0x7fb4cc151cd0 2024-02-21T23:09:58.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.010+0000 7fb4c2a00640 1 -- 172.21.15.28:0/574109384 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4b80047e0 con 0x7fb4cc151cd0 2024-02-21T23:09:58.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.010+0000 7fb4cbe00640 1 -- 172.21.15.28:0/574109384 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4cc1559e0 con 0x7fb4cc151cd0 2024-02-21T23:09:59.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.010+0000 7fb4cbe00640 1 -- 172.21.15.28:0/574109384 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4cc14f860 con 0x7fb4cc151cd0 2024-02-21T23:09:59.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.011+0000 7fb4cbe00640 1 -- 172.21.15.28:0/574109384 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4cc04f290 con 0x7fb4cc151cd0 2024-02-21T23:09:59.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.012+0000 7fb4c2a00640 1 -- 172.21.15.28:0/574109384 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb4b801c070 con 0x7fb4cc151cd0 2024-02-21T23:09:59.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.013+0000 7fb4c2a00640 1 --2- 172.21.15.28:0/574109384 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb4ac06b500 0x7fb4ac06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:59.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.013+0000 7fb4c2a00640 1 -- 172.21.15.28:0/574109384 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb4b8089a90 con 0x7fb4cc151cd0 2024-02-21T23:09:59.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.014+0000 7fb4caa00640 1 --2- 172.21.15.28:0/574109384 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb4ac06b500 0x7fb4ac06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:59.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.015+0000 7fb4caa00640 1 --2- 172.21.15.28:0/574109384 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb4ac06b500 0x7fb4ac06d9c0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc033000 tx=0x7fb4bc009030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:59.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.022+0000 7fb4c2a00640 1 -- 172.21.15.28:0/574109384 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb4b8058120 con 0x7fb4cc151cd0 2024-02-21T23:09:59.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.222+0000 7fb4cbe00640 1 -- 172.21.15.28:0/574109384 --> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb4cc06ecb0 con 0x7fb4ac06b500 2024-02-21T23:09:59.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.240+0000 7fb4c2a00640 1 -- 172.21.15.28:0/574109384 <== mgr.4101 v2:172.21.15.81:6832/1848538577 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252526 (secure 0 0 0) 0x7fb4cc06ecb0 con 0x7fb4ac06b500 2024-02-21T23:09:59.232 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:09:59.233 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-21T23:09:59.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.245+0000 7fb4cbe00640 1 -- 172.21.15.28:0/574109384 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb4ac06b500 msgr2=0x7fb4ac06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:59.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.245+0000 7fb4cbe00640 1 --2- 172.21.15.28:0/574109384 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb4ac06b500 0x7fb4ac06d9c0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc033000 tx=0x7fb4bc009030 comp rx=0 tx=0).stop 2024-02-21T23:09:59.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.245+0000 7fb4cbe00640 1 -- 172.21.15.28:0/574109384 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4cc151cd0 msgr2=0x7fb4cc153b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:59.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.245+0000 7fb4cbe00640 1 --2- 172.21.15.28:0/574109384 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4cc151cd0 0x7fb4cc153b80 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7fb4b800d770 tx=0x7fb4b800db70 comp rx=0 tx=0).stop 2024-02-21T23:09:59.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.245+0000 7fb4cbe00640 1 -- 172.21.15.28:0/574109384 shutdown_connections 2024-02-21T23:09:59.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.245+0000 7fb4cbe00640 1 --2- 172.21.15.28:0/574109384 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb4ac06b500 0x7fb4ac06d9c0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:59.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.245+0000 7fb4cbe00640 1 --2- 172.21.15.28:0/574109384 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb4cc0a7fd0 0x7fb4cc1540c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:59.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.245+0000 7fb4cbe00640 1 --2- 172.21.15.28:0/574109384 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4cc151cd0 0x7fb4cc153b80 unknown :-1 s=CLOSED pgs=1692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:59.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.245+0000 7fb4cbe00640 1 --2- 172.21.15.28:0/574109384 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb4b00039b0 0x7fb4cc151790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:59.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.245+0000 7fb4cbe00640 1 -- 172.21.15.28:0/574109384 >> 172.21.15.28:0/574109384 conn(0x7fb4cc06b630 msgr2=0x7fb4cc157680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:59.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.245+0000 7fb4cbe00640 1 -- 172.21.15.28:0/574109384 shutdown_connections 2024-02-21T23:09:59.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.245+0000 7fb4cbe00640 1 -- 172.21.15.28:0/574109384 wait complete. 2024-02-21T23:09:59.258 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":368,"stamp":"2024-02-21T23:09:58.820072+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":98,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":264440,"kb_used_data":14980,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754710280,"statfs":{"total":773094113280,"available":772823326720,"internally_reserved":0,"allocated":15339520,"data_stored":8557192,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002305"},"pg_stats":[{"pgid":"24.3f","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.276269+0000","last_change":"2024-02-21T23:09:54.279018+0000","last_active":"2024-02-21T23:09:55.276269+0000","last_peered":"2024-02-21T23:09:55.276269+0000","last_clean":"2024-02-21T23:09:55.276269+0000","last_became_active":"2024-02-21T23:09:54.278712+0000","last_became_peered":"2024-02-21T23:09:54.278712+0000","last_unstale":"2024-02-21T23:09:55.276269+0000","last_undegraded":"2024-02-21T23:09:55.276269+0000","last_fullsized":"2024-02-21T23:09:55.276269+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:55:18.591062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3e","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.287359+0000","last_change":"2024-02-21T23:09:54.285889+0000","last_active":"2024-02-21T23:09:54.287359+0000","last_peered":"2024-02-21T23:09:54.287359+0000","last_clean":"2024-02-21T23:09:54.287359+0000","last_became_active":"2024-02-21T23:09:54.284810+0000","last_became_peered":"2024-02-21T23:09:54.284810+0000","last_unstale":"2024-02-21T23:09:54.287359+0000","last_undegraded":"2024-02-21T23:09:54.287359+0000","last_fullsized":"2024-02-21T23:09:54.287359+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:13:44.635531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3d","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.275976+0000","last_change":"2024-02-21T23:09:54.263978+0000","last_active":"2024-02-21T23:09:55.275976+0000","last_peered":"2024-02-21T23:09:55.275976+0000","last_clean":"2024-02-21T23:09:55.275976+0000","last_became_active":"2024-02-21T23:09:54.263467+0000","last_became_peered":"2024-02-21T23:09:54.263467+0000","last_unstale":"2024-02-21T23:09:55.275976+0000","last_undegraded":"2024-02-21T23:09:55.275976+0000","last_fullsized":"2024-02-21T23:09:55.275976+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:46:35.436815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3c","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.276401+0000","last_change":"2024-02-21T23:09:54.279400+0000","last_active":"2024-02-21T23:09:55.276401+0000","last_peered":"2024-02-21T23:09:55.276401+0000","last_clean":"2024-02-21T23:09:55.276401+0000","last_became_active":"2024-02-21T23:09:54.279076+0000","last_became_peered":"2024-02-21T23:09:54.279076+0000","last_unstale":"2024-02-21T23:09:55.276401+0000","last_undegraded":"2024-02-21T23:09:55.276401+0000","last_fullsized":"2024-02-21T23:09:55.276401+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:02:11.247387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.3b","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.274751+0000","last_change":"2024-02-21T23:09:54.271682+0000","last_active":"2024-02-21T23:09:54.274751+0000","last_peered":"2024-02-21T23:09:54.274751+0000","last_clean":"2024-02-21T23:09:54.274751+0000","last_became_active":"2024-02-21T23:09:54.271297+0000","last_became_peered":"2024-02-21T23:09:54.271297+0000","last_unstale":"2024-02-21T23:09:54.274751+0000","last_undegraded":"2024-02-21T23:09:54.274751+0000","last_fullsized":"2024-02-21T23:09:54.274751+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:51:09.560052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3a","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.286978+0000","last_change":"2024-02-21T23:09:54.281830+0000","last_active":"2024-02-21T23:09:54.286978+0000","last_peered":"2024-02-21T23:09:54.286978+0000","last_clean":"2024-02-21T23:09:54.286978+0000","last_became_active":"2024-02-21T23:09:54.281334+0000","last_became_peered":"2024-02-21T23:09:54.281334+0000","last_unstale":"2024-02-21T23:09:54.286978+0000","last_undegraded":"2024-02-21T23:09:54.286978+0000","last_fullsized":"2024-02-21T23:09:54.286978+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:29:25.643198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.39","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.290041+0000","last_change":"2024-02-21T23:09:54.277806+0000","last_active":"2024-02-21T23:09:55.290041+0000","last_peered":"2024-02-21T23:09:55.290041+0000","last_clean":"2024-02-21T23:09:55.290041+0000","last_became_active":"2024-02-21T23:09:54.277446+0000","last_became_peered":"2024-02-21T23:09:54.277446+0000","last_unstale":"2024-02-21T23:09:55.290041+0000","last_undegraded":"2024-02-21T23:09:55.290041+0000","last_fullsized":"2024-02-21T23:09:55.290041+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:59:34.646775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.38","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.289953+0000","last_change":"2024-02-21T23:09:54.286729+0000","last_active":"2024-02-21T23:09:55.289953+0000","last_peered":"2024-02-21T23:09:55.289953+0000","last_clean":"2024-02-21T23:09:55.289953+0000","last_became_active":"2024-02-21T23:09:54.286427+0000","last_became_peered":"2024-02-21T23:09:54.286427+0000","last_unstale":"2024-02-21T23:09:55.289953+0000","last_undegraded":"2024-02-21T23:09:55.289953+0000","last_fullsized":"2024-02-21T23:09:55.289953+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:13:27.311923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.37","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.290895+0000","last_change":"2024-02-21T23:09:54.286612+0000","last_active":"2024-02-21T23:09:55.290895+0000","last_peered":"2024-02-21T23:09:55.290895+0000","last_clean":"2024-02-21T23:09:55.290895+0000","last_became_active":"2024-02-21T23:09:54.286306+0000","last_became_peered":"2024-02-21T23:09:54.286306+0000","last_unstale":"2024-02-21T23:09:55.290895+0000","last_undegraded":"2024-02-21T23:09:55.290895+0000","last_fullsized":"2024-02-21T23:09:55.290895+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:04:16.317089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.36","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.291078+0000","last_change":"2024-02-21T23:09:54.288019+0000","last_active":"2024-02-21T23:09:55.291078+0000","last_peered":"2024-02-21T23:09:55.291078+0000","last_clean":"2024-02-21T23:09:55.291078+0000","last_became_active":"2024-02-21T23:09:54.287742+0000","last_became_peered":"2024-02-21T23:09:54.287742+0000","last_unstale":"2024-02-21T23:09:55.291078+0000","last_undegraded":"2024-02-21T23:09:55.291078+0000","last_fullsized":"2024-02-21T23:09:55.291078+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:13:33.442227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.35","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.286910+0000","last_change":"2024-02-21T23:09:54.283918+0000","last_active":"2024-02-21T23:09:54.286910+0000","last_peered":"2024-02-21T23:09:54.286910+0000","last_clean":"2024-02-21T23:09:54.286910+0000","last_became_active":"2024-02-21T23:09:54.280599+0000","last_became_peered":"2024-02-21T23:09:54.280599+0000","last_unstale":"2024-02-21T23:09:54.286910+0000","last_undegraded":"2024-02-21T23:09:54.286910+0000","last_fullsized":"2024-02-21T23:09:54.286910+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:13:44.635531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.34","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.287159+0000","last_change":"2024-02-21T23:09:54.282846+0000","last_active":"2024-02-21T23:09:54.287159+0000","last_peered":"2024-02-21T23:09:54.287159+0000","last_clean":"2024-02-21T23:09:54.287159+0000","last_became_active":"2024-02-21T23:09:54.282117+0000","last_became_peered":"2024-02-21T23:09:54.282117+0000","last_unstale":"2024-02-21T23:09:54.287159+0000","last_undegraded":"2024-02-21T23:09:54.287159+0000","last_fullsized":"2024-02-21T23:09:54.287159+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:58:27.362926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.33","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.273190+0000","last_change":"2024-02-21T23:09:54.284818+0000","last_active":"2024-02-21T23:09:55.273190+0000","last_peered":"2024-02-21T23:09:55.273190+0000","last_clean":"2024-02-21T23:09:55.273190+0000","last_became_active":"2024-02-21T23:09:54.284161+0000","last_became_peered":"2024-02-21T23:09:54.284161+0000","last_unstale":"2024-02-21T23:09:55.273190+0000","last_undegraded":"2024-02-21T23:09:55.273190+0000","last_fullsized":"2024-02-21T23:09:55.273190+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:02:12.340921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.32","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.273891+0000","last_change":"2024-02-21T23:09:54.272136+0000","last_active":"2024-02-21T23:09:54.273891+0000","last_peered":"2024-02-21T23:09:54.273891+0000","last_clean":"2024-02-21T23:09:54.273891+0000","last_became_active":"2024-02-21T23:09:54.271097+0000","last_became_peered":"2024-02-21T23:09:54.271097+0000","last_unstale":"2024-02-21T23:09:54.273891+0000","last_undegraded":"2024-02-21T23:09:54.273891+0000","last_fullsized":"2024-02-21T23:09:54.273891+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:52:57.889995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.31","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.286879+0000","last_change":"2024-02-21T23:09:54.281268+0000","last_active":"2024-02-21T23:09:54.286879+0000","last_peered":"2024-02-21T23:09:54.286879+0000","last_clean":"2024-02-21T23:09:54.286879+0000","last_became_active":"2024-02-21T23:09:54.280946+0000","last_became_peered":"2024-02-21T23:09:54.280946+0000","last_unstale":"2024-02-21T23:09:54.286879+0000","last_undegraded":"2024-02-21T23:09:54.286879+0000","last_fullsized":"2024-02-21T23:09:54.286879+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:31:28.395228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.30","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.275530+0000","last_change":"2024-02-21T23:09:54.278731+0000","last_active":"2024-02-21T23:09:55.275530+0000","last_peered":"2024-02-21T23:09:55.275530+0000","last_clean":"2024-02-21T23:09:55.275530+0000","last_became_active":"2024-02-21T23:09:54.278390+0000","last_became_peered":"2024-02-21T23:09:54.278390+0000","last_unstale":"2024-02-21T23:09:55.275530+0000","last_undegraded":"2024-02-21T23:09:55.275530+0000","last_fullsized":"2024-02-21T23:09:55.275530+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:28:54.079553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.2f","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.291090+0000","last_change":"2024-02-21T23:09:54.285947+0000","last_active":"2024-02-21T23:09:55.291090+0000","last_peered":"2024-02-21T23:09:55.291090+0000","last_clean":"2024-02-21T23:09:55.291090+0000","last_became_active":"2024-02-21T23:09:54.285658+0000","last_became_peered":"2024-02-21T23:09:54.285658+0000","last_unstale":"2024-02-21T23:09:55.291090+0000","last_undegraded":"2024-02-21T23:09:55.291090+0000","last_fullsized":"2024-02-21T23:09:55.291090+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:20:31.125684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2e","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.526134+0000","last_change":"2024-02-21T23:09:54.287286+0000","last_active":"2024-02-21T23:09:55.526134+0000","last_peered":"2024-02-21T23:09:55.526134+0000","last_clean":"2024-02-21T23:09:55.526134+0000","last_became_active":"2024-02-21T23:09:54.286924+0000","last_became_peered":"2024-02-21T23:09:54.286924+0000","last_unstale":"2024-02-21T23:09:55.526134+0000","last_undegraded":"2024-02-21T23:09:55.526134+0000","last_fullsized":"2024-02-21T23:09:55.526134+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:47:29.349669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.2d","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.286185+0000","last_change":"2024-02-21T23:09:54.279884+0000","last_active":"2024-02-21T23:09:54.286185+0000","last_peered":"2024-02-21T23:09:54.286185+0000","last_clean":"2024-02-21T23:09:54.286185+0000","last_became_active":"2024-02-21T23:09:54.276608+0000","last_became_peered":"2024-02-21T23:09:54.276608+0000","last_unstale":"2024-02-21T23:09:54.286185+0000","last_undegraded":"2024-02-21T23:09:54.286185+0000","last_fullsized":"2024-02-21T23:09:54.286185+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:34:50.354067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.2c","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.272941+0000","last_change":"2024-02-21T23:09:54.288944+0000","last_active":"2024-02-21T23:09:55.272941+0000","last_peered":"2024-02-21T23:09:55.272941+0000","last_clean":"2024-02-21T23:09:55.272941+0000","last_became_active":"2024-02-21T23:09:54.285913+0000","last_became_peered":"2024-02-21T23:09:54.285913+0000","last_unstale":"2024-02-21T23:09:55.272941+0000","last_undegraded":"2024-02-21T23:09:55.272941+0000","last_fullsized":"2024-02-21T23:09:55.272941+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:39:33.609273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.2b","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.273379+0000","last_change":"2024-02-21T23:09:54.285036+0000","last_active":"2024-02-21T23:09:55.273379+0000","last_peered":"2024-02-21T23:09:55.273379+0000","last_clean":"2024-02-21T23:09:55.273379+0000","last_became_active":"2024-02-21T23:09:54.284542+0000","last_became_peered":"2024-02-21T23:09:54.284542+0000","last_unstale":"2024-02-21T23:09:55.273379+0000","last_undegraded":"2024-02-21T23:09:55.273379+0000","last_fullsized":"2024-02-21T23:09:55.273379+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T11:05:01.241636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.2a","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.274942+0000","last_change":"2024-02-21T23:09:54.273350+0000","last_active":"2024-02-21T23:09:54.274942+0000","last_peered":"2024-02-21T23:09:54.274942+0000","last_clean":"2024-02-21T23:09:54.274942+0000","last_became_active":"2024-02-21T23:09:54.273022+0000","last_became_peered":"2024-02-21T23:09:54.273022+0000","last_unstale":"2024-02-21T23:09:54.274942+0000","last_undegraded":"2024-02-21T23:09:54.274942+0000","last_fullsized":"2024-02-21T23:09:54.274942+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:23:27.511315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.29","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.281903+0000","last_change":"2024-02-21T23:09:54.279718+0000","last_active":"2024-02-21T23:09:54.281903+0000","last_peered":"2024-02-21T23:09:54.281903+0000","last_clean":"2024-02-21T23:09:54.281903+0000","last_became_active":"2024-02-21T23:09:54.278989+0000","last_became_peered":"2024-02-21T23:09:54.278989+0000","last_unstale":"2024-02-21T23:09:54.281903+0000","last_undegraded":"2024-02-21T23:09:54.281903+0000","last_fullsized":"2024-02-21T23:09:54.281903+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:33:59.785551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.28","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.275774+0000","last_change":"2024-02-21T23:09:54.280624+0000","last_active":"2024-02-21T23:09:55.275774+0000","last_peered":"2024-02-21T23:09:55.275774+0000","last_clean":"2024-02-21T23:09:55.275774+0000","last_became_active":"2024-02-21T23:09:54.275644+0000","last_became_peered":"2024-02-21T23:09:54.275644+0000","last_unstale":"2024-02-21T23:09:55.275774+0000","last_undegraded":"2024-02-21T23:09:55.275774+0000","last_fullsized":"2024-02-21T23:09:55.275774+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:23:27.511315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.27","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.273927+0000","last_change":"2024-02-21T23:09:54.275341+0000","last_active":"2024-02-21T23:09:55.273927+0000","last_peered":"2024-02-21T23:09:55.273927+0000","last_clean":"2024-02-21T23:09:55.273927+0000","last_became_active":"2024-02-21T23:09:54.275010+0000","last_became_peered":"2024-02-21T23:09:54.275010+0000","last_unstale":"2024-02-21T23:09:55.273927+0000","last_undegraded":"2024-02-21T23:09:55.273927+0000","last_fullsized":"2024-02-21T23:09:55.273927+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:42:08.123188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.26","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.272151+0000","last_change":"2024-02-21T23:09:54.260902+0000","last_active":"2024-02-21T23:09:55.272151+0000","last_peered":"2024-02-21T23:09:55.272151+0000","last_clean":"2024-02-21T23:09:55.272151+0000","last_became_active":"2024-02-21T23:09:54.260084+0000","last_became_peered":"2024-02-21T23:09:54.260084+0000","last_unstale":"2024-02-21T23:09:55.272151+0000","last_undegraded":"2024-02-21T23:09:55.272151+0000","last_fullsized":"2024-02-21T23:09:55.272151+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:04:05.858817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.25","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.287285+0000","last_change":"2024-02-21T23:09:54.284116+0000","last_active":"2024-02-21T23:09:54.287285+0000","last_peered":"2024-02-21T23:09:54.287285+0000","last_clean":"2024-02-21T23:09:54.287285+0000","last_became_active":"2024-02-21T23:09:54.283802+0000","last_became_peered":"2024-02-21T23:09:54.283802+0000","last_unstale":"2024-02-21T23:09:54.287285+0000","last_undegraded":"2024-02-21T23:09:54.287285+0000","last_fullsized":"2024-02-21T23:09:54.287285+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:31:28.395228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.24","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.273666+0000","last_change":"2024-02-21T23:09:54.287481+0000","last_active":"2024-02-21T23:09:55.273666+0000","last_peered":"2024-02-21T23:09:55.273666+0000","last_clean":"2024-02-21T23:09:55.273666+0000","last_became_active":"2024-02-21T23:09:54.287137+0000","last_became_peered":"2024-02-21T23:09:54.287137+0000","last_unstale":"2024-02-21T23:09:55.273666+0000","last_undegraded":"2024-02-21T23:09:55.273666+0000","last_fullsized":"2024-02-21T23:09:55.273666+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:50:46.472002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.23","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.287067+0000","last_change":"2024-02-21T23:09:54.281613+0000","last_active":"2024-02-21T23:09:54.287067+0000","last_peered":"2024-02-21T23:09:54.287067+0000","last_clean":"2024-02-21T23:09:54.287067+0000","last_became_active":"2024-02-21T23:09:54.281316+0000","last_became_peered":"2024-02-21T23:09:54.281316+0000","last_unstale":"2024-02-21T23:09:54.287067+0000","last_undegraded":"2024-02-21T23:09:54.287067+0000","last_fullsized":"2024-02-21T23:09:54.287067+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:32:13.492141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.22","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.525855+0000","last_change":"2024-02-21T23:09:54.275665+0000","last_active":"2024-02-21T23:09:55.525855+0000","last_peered":"2024-02-21T23:09:55.525855+0000","last_clean":"2024-02-21T23:09:55.525855+0000","last_became_active":"2024-02-21T23:09:54.275143+0000","last_became_peered":"2024-02-21T23:09:54.275143+0000","last_unstale":"2024-02-21T23:09:55.525855+0000","last_undegraded":"2024-02-21T23:09:55.525855+0000","last_fullsized":"2024-02-21T23:09:55.525855+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:37:42.976792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.21","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.286230+0000","last_change":"2024-02-21T23:09:54.281200+0000","last_active":"2024-02-21T23:09:54.286230+0000","last_peered":"2024-02-21T23:09:54.286230+0000","last_clean":"2024-02-21T23:09:54.286230+0000","last_became_active":"2024-02-21T23:09:54.277867+0000","last_became_peered":"2024-02-21T23:09:54.277867+0000","last_unstale":"2024-02-21T23:09:54.286230+0000","last_undegraded":"2024-02-21T23:09:54.286230+0000","last_fullsized":"2024-02-21T23:09:54.286230+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:58:27.362926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.20","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.525712+0000","last_change":"2024-02-21T23:09:54.276078+0000","last_active":"2024-02-21T23:09:55.525712+0000","last_peered":"2024-02-21T23:09:55.525712+0000","last_clean":"2024-02-21T23:09:55.525712+0000","last_became_active":"2024-02-21T23:09:54.275724+0000","last_became_peered":"2024-02-21T23:09:54.275724+0000","last_unstale":"2024-02-21T23:09:55.525712+0000","last_undegraded":"2024-02-21T23:09:55.525712+0000","last_fullsized":"2024-02-21T23:09:55.525712+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:48:07.369182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.10","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.286630+0000","last_change":"2024-02-21T23:09:54.278972+0000","last_active":"2024-02-21T23:09:54.286630+0000","last_peered":"2024-02-21T23:09:54.286630+0000","last_clean":"2024-02-21T23:09:54.286630+0000","last_became_active":"2024-02-21T23:09:54.278146+0000","last_became_peered":"2024-02-21T23:09:54.278146+0000","last_unstale":"2024-02-21T23:09:54.286630+0000","last_undegraded":"2024-02-21T23:09:54.286630+0000","last_fullsized":"2024-02-21T23:09:54.286630+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:02:11.247387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"150'4","reported_seq":18,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.276587+0000","last_change":"2024-02-21T23:09:53.288506+0000","last_active":"2024-02-21T23:09:55.276587+0000","last_peered":"2024-02-21T23:09:55.276587+0000","last_clean":"2024-02-21T23:09:55.276587+0000","last_became_active":"2024-02-21T23:09:53.288199+0000","last_became_peered":"2024-02-21T23:09:53.288199+0000","last_unstale":"2024-02-21T23:09:55.276587+0000","last_undegraded":"2024-02-21T23:09:55.276587+0000","last_fullsized":"2024-02-21T23:09:55.276587+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:18:16.467715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.273863+0000","last_change":"2024-02-21T23:09:54.278032+0000","last_active":"2024-02-21T23:09:55.273863+0000","last_peered":"2024-02-21T23:09:55.273863+0000","last_clean":"2024-02-21T23:09:55.273863+0000","last_became_active":"2024-02-21T23:09:54.277741+0000","last_became_peered":"2024-02-21T23:09:54.277741+0000","last_unstale":"2024-02-21T23:09:55.273863+0000","last_undegraded":"2024-02-21T23:09:55.273863+0000","last_fullsized":"2024-02-21T23:09:55.273863+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:43:56.610613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"150'1","reported_seq":13,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.302587+0000","last_change":"2024-02-21T23:09:53.279374+0000","last_active":"2024-02-21T23:09:54.302587+0000","last_peered":"2024-02-21T23:09:54.302587+0000","last_clean":"2024-02-21T23:09:54.302587+0000","last_became_active":"2024-02-21T23:09:53.279053+0000","last_became_peered":"2024-02-21T23:09:53.279053+0000","last_unstale":"2024-02-21T23:09:54.302587+0000","last_undegraded":"2024-02-21T23:09:54.302587+0000","last_fullsized":"2024-02-21T23:09:54.302587+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:04:15.300172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.272323+0000","last_change":"2024-02-21T23:09:54.263915+0000","last_active":"2024-02-21T23:09:54.272323+0000","last_peered":"2024-02-21T23:09:54.272323+0000","last_clean":"2024-02-21T23:09:54.272323+0000","last_became_active":"2024-02-21T23:09:54.263634+0000","last_became_peered":"2024-02-21T23:09:54.263634+0000","last_unstale":"2024-02-21T23:09:54.272323+0000","last_undegraded":"2024-02-21T23:09:54.272323+0000","last_fullsized":"2024-02-21T23:09:54.272323+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:33:07.362294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.1d","version":"150'2","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.302666+0000","last_change":"2024-02-21T23:09:53.272110+0000","last_active":"2024-02-21T23:09:54.302666+0000","last_peered":"2024-02-21T23:09:54.302666+0000","last_clean":"2024-02-21T23:09:54.302666+0000","last_became_active":"2024-02-21T23:09:53.271641+0000","last_became_peered":"2024-02-21T23:09:53.271641+0000","last_unstale":"2024-02-21T23:09:54.302666+0000","last_undegraded":"2024-02-21T23:09:54.302666+0000","last_fullsized":"2024-02-21T23:09:54.302666+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:46:34.419898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.13","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.284893+0000","last_change":"2024-02-21T23:09:54.275867+0000","last_active":"2024-02-21T23:09:54.284893+0000","last_peered":"2024-02-21T23:09:54.284893+0000","last_clean":"2024-02-21T23:09:54.284893+0000","last_became_active":"2024-02-21T23:09:54.275591+0000","last_became_peered":"2024-02-21T23:09:54.275591+0000","last_unstale":"2024-02-21T23:09:54.284893+0000","last_undegraded":"2024-02-21T23:09:54.284893+0000","last_fullsized":"2024-02-21T23:09:54.284893+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:46:35.436815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1c","version":"150'2","reported_seq":16,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.273105+0000","last_change":"2024-02-21T23:09:53.274692+0000","last_active":"2024-02-21T23:09:55.273105+0000","last_peered":"2024-02-21T23:09:55.273105+0000","last_clean":"2024-02-21T23:09:55.273105+0000","last_became_active":"2024-02-21T23:09:53.274395+0000","last_became_peered":"2024-02-21T23:09:53.274395+0000","last_unstale":"2024-02-21T23:09:55.273105+0000","last_undegraded":"2024-02-21T23:09:55.273105+0000","last_fullsized":"2024-02-21T23:09:55.273105+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:40:18.453921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.14","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.290542+0000","last_change":"2024-02-21T23:09:54.289092+0000","last_active":"2024-02-21T23:09:55.290542+0000","last_peered":"2024-02-21T23:09:55.290542+0000","last_clean":"2024-02-21T23:09:55.290542+0000","last_became_active":"2024-02-21T23:09:54.288801+0000","last_became_peered":"2024-02-21T23:09:54.288801+0000","last_unstale":"2024-02-21T23:09:55.290542+0000","last_undegraded":"2024-02-21T23:09:55.290542+0000","last_fullsized":"2024-02-21T23:09:55.290542+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:05:09.189420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1b","version":"150'1","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.290367+0000","last_change":"2024-02-21T23:09:53.283422+0000","last_active":"2024-02-21T23:09:55.290367+0000","last_peered":"2024-02-21T23:09:55.290367+0000","last_clean":"2024-02-21T23:09:55.290367+0000","last_became_active":"2024-02-21T23:09:53.283140+0000","last_became_peered":"2024-02-21T23:09:53.283140+0000","last_unstale":"2024-02-21T23:09:55.290367+0000","last_undegraded":"2024-02-21T23:09:55.290367+0000","last_fullsized":"2024-02-21T23:09:55.290367+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:46:34.419898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.15","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.272579+0000","last_change":"2024-02-21T23:09:54.273728+0000","last_active":"2024-02-21T23:09:55.272579+0000","last_peered":"2024-02-21T23:09:55.272579+0000","last_clean":"2024-02-21T23:09:55.272579+0000","last_became_active":"2024-02-21T23:09:54.273343+0000","last_became_peered":"2024-02-21T23:09:54.273343+0000","last_unstale":"2024-02-21T23:09:55.272579+0000","last_undegraded":"2024-02-21T23:09:55.272579+0000","last_fullsized":"2024-02-21T23:09:55.272579+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:49:15.552003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.1a","version":"150'1","reported_seq":13,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.299542+0000","last_change":"2024-02-21T23:09:53.278176+0000","last_active":"2024-02-21T23:09:54.299542+0000","last_peered":"2024-02-21T23:09:54.299542+0000","last_clean":"2024-02-21T23:09:54.299542+0000","last_became_active":"2024-02-21T23:09:53.277880+0000","last_became_peered":"2024-02-21T23:09:53.277880+0000","last_unstale":"2024-02-21T23:09:54.299542+0000","last_undegraded":"2024-02-21T23:09:54.299542+0000","last_fullsized":"2024-02-21T23:09:54.299542+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:05:08.172503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.272250+0000","last_change":"2024-02-21T23:09:54.278094+0000","last_active":"2024-02-21T23:09:55.272250+0000","last_peered":"2024-02-21T23:09:55.272250+0000","last_clean":"2024-02-21T23:09:55.272250+0000","last_became_active":"2024-02-21T23:09:54.276160+0000","last_became_peered":"2024-02-21T23:09:54.276160+0000","last_unstale":"2024-02-21T23:09:55.272250+0000","last_undegraded":"2024-02-21T23:09:55.272250+0000","last_fullsized":"2024-02-21T23:09:55.272250+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:15:07.339478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":14,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.271567+0000","last_change":"2024-02-21T23:09:53.280005+0000","last_active":"2024-02-21T23:09:55.271567+0000","last_peered":"2024-02-21T23:09:55.271567+0000","last_clean":"2024-02-21T23:09:55.271567+0000","last_became_active":"2024-02-21T23:09:53.279689+0000","last_became_peered":"2024-02-21T23:09:53.279689+0000","last_unstale":"2024-02-21T23:09:55.271567+0000","last_undegraded":"2024-02-21T23:09:55.271567+0000","last_fullsized":"2024-02-21T23:09:55.271567+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:40:05.561558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.276563+0000","last_change":"2024-02-21T23:09:54.262565+0000","last_active":"2024-02-21T23:09:55.276563+0000","last_peered":"2024-02-21T23:09:55.276563+0000","last_clean":"2024-02-21T23:09:55.276563+0000","last_became_active":"2024-02-21T23:09:54.261897+0000","last_became_peered":"2024-02-21T23:09:54.261897+0000","last_unstale":"2024-02-21T23:09:55.276563+0000","last_undegraded":"2024-02-21T23:09:55.276563+0000","last_fullsized":"2024-02-21T23:09:55.276563+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:08:50.519955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":14,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.290103+0000","last_change":"2024-02-21T23:09:53.275695+0000","last_active":"2024-02-21T23:09:55.290103+0000","last_peered":"2024-02-21T23:09:55.290103+0000","last_clean":"2024-02-21T23:09:55.290103+0000","last_became_active":"2024-02-21T23:09:53.275343+0000","last_became_peered":"2024-02-21T23:09:53.275343+0000","last_unstale":"2024-02-21T23:09:55.290103+0000","last_undegraded":"2024-02-21T23:09:55.290103+0000","last_fullsized":"2024-02-21T23:09:55.290103+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:28:53.062636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.286660+0000","last_change":"2024-02-21T23:09:54.279220+0000","last_active":"2024-02-21T23:09:54.286660+0000","last_peered":"2024-02-21T23:09:54.286660+0000","last_clean":"2024-02-21T23:09:54.286660+0000","last_became_active":"2024-02-21T23:09:54.278653+0000","last_became_peered":"2024-02-21T23:09:54.278653+0000","last_unstale":"2024-02-21T23:09:54.286660+0000","last_undegraded":"2024-02-21T23:09:54.286660+0000","last_fullsized":"2024-02-21T23:09:54.286660+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:18:40.139538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":14,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.273576+0000","last_change":"2024-02-21T23:09:53.288439+0000","last_active":"2024-02-21T23:09:55.273576+0000","last_peered":"2024-02-21T23:09:55.273576+0000","last_clean":"2024-02-21T23:09:55.273576+0000","last_became_active":"2024-02-21T23:09:53.287601+0000","last_became_peered":"2024-02-21T23:09:53.287601+0000","last_unstale":"2024-02-21T23:09:55.273576+0000","last_undegraded":"2024-02-21T23:09:55.273576+0000","last_fullsized":"2024-02-21T23:09:55.273576+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:32:05.045159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.273172+0000","last_change":"2024-02-21T23:09:54.280274+0000","last_active":"2024-02-21T23:09:55.273172+0000","last_peered":"2024-02-21T23:09:55.273172+0000","last_clean":"2024-02-21T23:09:55.273172+0000","last_became_active":"2024-02-21T23:09:54.279951+0000","last_became_peered":"2024-02-21T23:09:54.279951+0000","last_unstale":"2024-02-21T23:09:55.273172+0000","last_undegraded":"2024-02-21T23:09:55.273172+0000","last_fullsized":"2024-02-21T23:09:55.273172+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:15:08.084683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.15","version":"150'3","reported_seq":17,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.275910+0000","last_change":"2024-02-21T23:09:53.279389+0000","last_active":"2024-02-21T23:09:55.275910+0000","last_peered":"2024-02-21T23:09:55.275910+0000","last_clean":"2024-02-21T23:09:55.275910+0000","last_became_active":"2024-02-21T23:09:53.279042+0000","last_became_peered":"2024-02-21T23:09:53.279042+0000","last_unstale":"2024-02-21T23:09:55.275910+0000","last_undegraded":"2024-02-21T23:09:55.275910+0000","last_fullsized":"2024-02-21T23:09:55.275910+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:44:44.898339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1b","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.272168+0000","last_change":"2024-02-21T23:09:54.285563+0000","last_active":"2024-02-21T23:09:55.272168+0000","last_peered":"2024-02-21T23:09:55.272168+0000","last_clean":"2024-02-21T23:09:55.272168+0000","last_became_active":"2024-02-21T23:09:54.285275+0000","last_became_peered":"2024-02-21T23:09:54.285275+0000","last_unstale":"2024-02-21T23:09:55.272168+0000","last_undegraded":"2024-02-21T23:09:55.272168+0000","last_fullsized":"2024-02-21T23:09:55.272168+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:21:42.014406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"150'6","reported_seq":20,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.273813+0000","last_change":"2024-02-21T23:09:53.288423+0000","last_active":"2024-02-21T23:09:55.273813+0000","last_peered":"2024-02-21T23:09:55.273813+0000","last_clean":"2024-02-21T23:09:55.273813+0000","last_became_active":"2024-02-21T23:09:53.287473+0000","last_became_peered":"2024-02-21T23:09:53.287473+0000","last_unstale":"2024-02-21T23:09:55.273813+0000","last_undegraded":"2024-02-21T23:09:55.273813+0000","last_fullsized":"2024-02-21T23:09:55.273813+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:13:32.425310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1c","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.526152+0000","last_change":"2024-02-21T23:09:54.278618+0000","last_active":"2024-02-21T23:09:55.526152+0000","last_peered":"2024-02-21T23:09:55.526152+0000","last_clean":"2024-02-21T23:09:55.526152+0000","last_became_active":"2024-02-21T23:09:54.278343+0000","last_became_peered":"2024-02-21T23:09:54.278343+0000","last_unstale":"2024-02-21T23:09:55.526152+0000","last_undegraded":"2024-02-21T23:09:55.526152+0000","last_fullsized":"2024-02-21T23:09:55.526152+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:49:31.759374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.13","version":"150'2","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.304290+0000","last_change":"2024-02-21T23:09:53.284172+0000","last_active":"2024-02-21T23:09:54.304290+0000","last_peered":"2024-02-21T23:09:54.304290+0000","last_clean":"2024-02-21T23:09:54.304290+0000","last_became_active":"2024-02-21T23:09:53.283507+0000","last_became_peered":"2024-02-21T23:09:53.283507+0000","last_unstale":"2024-02-21T23:09:54.304290+0000","last_undegraded":"2024-02-21T23:09:54.304290+0000","last_fullsized":"2024-02-21T23:09:54.304290+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:20:30.108767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1d","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.290794+0000","last_change":"2024-02-21T23:09:54.291717+0000","last_active":"2024-02-21T23:09:55.290794+0000","last_peered":"2024-02-21T23:09:55.290794+0000","last_clean":"2024-02-21T23:09:55.290794+0000","last_became_active":"2024-02-21T23:09:54.290842+0000","last_became_peered":"2024-02-21T23:09:54.290842+0000","last_unstale":"2024-02-21T23:09:55.290794+0000","last_undegraded":"2024-02-21T23:09:55.290794+0000","last_fullsized":"2024-02-21T23:09:55.290794+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:42:16.988326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":14,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.273465+0000","last_change":"2024-02-21T23:09:53.284733+0000","last_active":"2024-02-21T23:09:55.273465+0000","last_peered":"2024-02-21T23:09:55.273465+0000","last_clean":"2024-02-21T23:09:55.273465+0000","last_became_active":"2024-02-21T23:09:53.284399+0000","last_became_peered":"2024-02-21T23:09:53.284399+0000","last_unstale":"2024-02-21T23:09:55.273465+0000","last_undegraded":"2024-02-21T23:09:55.273465+0000","last_fullsized":"2024-02-21T23:09:55.273465+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:05:08.172503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1e","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.283361+0000","last_change":"2024-02-21T23:09:54.277339+0000","last_active":"2024-02-21T23:09:54.283361+0000","last_peered":"2024-02-21T23:09:54.283361+0000","last_clean":"2024-02-21T23:09:54.283361+0000","last_became_active":"2024-02-21T23:09:54.277039+0000","last_became_peered":"2024-02-21T23:09:54.277039+0000","last_unstale":"2024-02-21T23:09:54.283361+0000","last_undegraded":"2024-02-21T23:09:54.283361+0000","last_fullsized":"2024-02-21T23:09:54.283361+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:29:25.643198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":12,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.249482+0000","last_change":"2024-02-21T23:09:53.283915+0000","last_active":"2024-02-21T23:09:54.249482+0000","last_peered":"2024-02-21T23:09:54.249482+0000","last_clean":"2024-02-21T23:09:54.249482+0000","last_became_active":"2024-02-21T23:09:53.283532+0000","last_became_peered":"2024-02-21T23:09:53.283532+0000","last_unstale":"2024-02-21T23:09:54.249482+0000","last_undegraded":"2024-02-21T23:09:54.249482+0000","last_fullsized":"2024-02-21T23:09:54.249482+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:55:17.574145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1f","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.272553+0000","last_change":"2024-02-21T23:09:54.265384+0000","last_active":"2024-02-21T23:09:55.272553+0000","last_peered":"2024-02-21T23:09:55.272553+0000","last_clean":"2024-02-21T23:09:55.272553+0000","last_became_active":"2024-02-21T23:09:54.265073+0000","last_became_peered":"2024-02-21T23:09:54.265073+0000","last_unstale":"2024-02-21T23:09:55.272553+0000","last_undegraded":"2024-02-21T23:09:55.272553+0000","last_fullsized":"2024-02-21T23:09:55.272553+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:44:28.295055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.10","version":"150'8","reported_seq":22,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.290161+0000","last_change":"2024-02-21T23:09:53.285246+0000","last_active":"2024-02-21T23:09:55.290161+0000","last_peered":"2024-02-21T23:09:55.290161+0000","last_clean":"2024-02-21T23:09:55.290161+0000","last_became_active":"2024-02-21T23:09:53.282084+0000","last_became_peered":"2024-02-21T23:09:53.282084+0000","last_unstale":"2024-02-21T23:09:55.290161+0000","last_undegraded":"2024-02-21T23:09:55.290161+0000","last_fullsized":"2024-02-21T23:09:55.290161+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:23:26.494398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.289952+0000","last_change":"2024-02-21T23:09:54.289615+0000","last_active":"2024-02-21T23:09:55.289952+0000","last_peered":"2024-02-21T23:09:55.289952+0000","last_clean":"2024-02-21T23:09:55.289952+0000","last_became_active":"2024-02-21T23:09:54.289322+0000","last_became_peered":"2024-02-21T23:09:54.289322+0000","last_unstale":"2024-02-21T23:09:55.289952+0000","last_undegraded":"2024-02-21T23:09:55.289952+0000","last_fullsized":"2024-02-21T23:09:55.289952+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:34:28.642087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.f","version":"150'1","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.274043+0000","last_change":"2024-02-21T23:09:53.287210+0000","last_active":"2024-02-21T23:09:55.274043+0000","last_peered":"2024-02-21T23:09:55.274043+0000","last_clean":"2024-02-21T23:09:55.274043+0000","last_became_active":"2024-02-21T23:09:53.285372+0000","last_became_peered":"2024-02-21T23:09:53.285372+0000","last_unstale":"2024-02-21T23:09:55.274043+0000","last_undegraded":"2024-02-21T23:09:55.274043+0000","last_fullsized":"2024-02-21T23:09:55.274043+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:20:30.108767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.526125+0000","last_change":"2024-02-21T23:09:54.274363+0000","last_active":"2024-02-21T23:09:55.526125+0000","last_peered":"2024-02-21T23:09:55.526125+0000","last_clean":"2024-02-21T23:09:55.526125+0000","last_became_active":"2024-02-21T23:09:54.274084+0000","last_became_peered":"2024-02-21T23:09:54.274084+0000","last_unstale":"2024-02-21T23:09:55.526125+0000","last_undegraded":"2024-02-21T23:09:55.526125+0000","last_fullsized":"2024-02-21T23:09:55.526125+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:42:19.996155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.e","version":"150'2","reported_seq":16,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.291139+0000","last_change":"2024-02-21T23:09:53.275941+0000","last_active":"2024-02-21T23:09:55.291139+0000","last_peered":"2024-02-21T23:09:55.291139+0000","last_clean":"2024-02-21T23:09:55.291139+0000","last_became_active":"2024-02-21T23:09:53.275555+0000","last_became_peered":"2024-02-21T23:09:53.275555+0000","last_unstale":"2024-02-21T23:09:55.291139+0000","last_undegraded":"2024-02-21T23:09:55.291139+0000","last_fullsized":"2024-02-21T23:09:55.291139+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:02:10.230470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.262950+0000","last_change":"2024-02-21T23:09:54.259630+0000","last_active":"2024-02-21T23:09:54.262950+0000","last_peered":"2024-02-21T23:09:54.262950+0000","last_clean":"2024-02-21T23:09:54.262950+0000","last_became_active":"2024-02-21T23:09:54.258924+0000","last_became_peered":"2024-02-21T23:09:54.258924+0000","last_unstale":"2024-02-21T23:09:54.262950+0000","last_undegraded":"2024-02-21T23:09:54.262950+0000","last_fullsized":"2024-02-21T23:09:54.262950+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:18:40.139538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.d","version":"150'1","reported_seq":13,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.302851+0000","last_change":"2024-02-21T23:09:53.279385+0000","last_active":"2024-02-21T23:09:54.302851+0000","last_peered":"2024-02-21T23:09:54.302851+0000","last_clean":"2024-02-21T23:09:54.302851+0000","last_became_active":"2024-02-21T23:09:53.278074+0000","last_became_peered":"2024-02-21T23:09:53.278074+0000","last_unstale":"2024-02-21T23:09:54.302851+0000","last_undegraded":"2024-02-21T23:09:54.302851+0000","last_fullsized":"2024-02-21T23:09:54.302851+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:42:15.971409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.283275+0000","last_change":"2024-02-21T23:09:54.277296+0000","last_active":"2024-02-21T23:09:54.283275+0000","last_peered":"2024-02-21T23:09:54.283275+0000","last_clean":"2024-02-21T23:09:54.283275+0000","last_became_active":"2024-02-21T23:09:54.276973+0000","last_became_peered":"2024-02-21T23:09:54.276973+0000","last_unstale":"2024-02-21T23:09:54.283275+0000","last_undegraded":"2024-02-21T23:09:54.283275+0000","last_fullsized":"2024-02-21T23:09:54.283275+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:32:13.492141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":14,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.290715+0000","last_change":"2024-02-21T23:09:53.281804+0000","last_active":"2024-02-21T23:09:55.290715+0000","last_peered":"2024-02-21T23:09:55.290715+0000","last_clean":"2024-02-21T23:09:55.290715+0000","last_became_active":"2024-02-21T23:09:53.278643+0000","last_became_peered":"2024-02-21T23:09:53.278643+0000","last_unstale":"2024-02-21T23:09:55.290715+0000","last_undegraded":"2024-02-21T23:09:55.290715+0000","last_fullsized":"2024-02-21T23:09:55.290715+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:55:17.574145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.273827+0000","last_change":"2024-02-21T23:09:54.285141+0000","last_active":"2024-02-21T23:09:55.273827+0000","last_peered":"2024-02-21T23:09:55.273827+0000","last_clean":"2024-02-21T23:09:55.273827+0000","last_became_active":"2024-02-21T23:09:54.284498+0000","last_became_peered":"2024-02-21T23:09:54.284498+0000","last_unstale":"2024-02-21T23:09:55.273827+0000","last_undegraded":"2024-02-21T23:09:55.273827+0000","last_fullsized":"2024-02-21T23:09:55.273827+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:26:36.405020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.b","version":"150'2","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.301072+0000","last_change":"2024-02-21T23:09:53.284284+0000","last_active":"2024-02-21T23:09:54.301072+0000","last_peered":"2024-02-21T23:09:54.301072+0000","last_clean":"2024-02-21T23:09:54.301072+0000","last_became_active":"2024-02-21T23:09:53.283043+0000","last_became_peered":"2024-02-21T23:09:53.283043+0000","last_unstale":"2024-02-21T23:09:54.301072+0000","last_undegraded":"2024-02-21T23:09:54.301072+0000","last_fullsized":"2024-02-21T23:09:54.301072+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:32:05.045159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.287916+0000","last_change":"2024-02-21T23:09:54.275082+0000","last_active":"2024-02-21T23:09:54.287916+0000","last_peered":"2024-02-21T23:09:54.287916+0000","last_clean":"2024-02-21T23:09:54.287916+0000","last_became_active":"2024-02-21T23:09:54.274707+0000","last_became_peered":"2024-02-21T23:09:54.274707+0000","last_unstale":"2024-02-21T23:09:54.287916+0000","last_undegraded":"2024-02-21T23:09:54.287916+0000","last_fullsized":"2024-02-21T23:09:54.287916+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:11:44.589606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":12,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.247863+0000","last_change":"2024-02-21T23:09:53.277418+0000","last_active":"2024-02-21T23:09:54.247863+0000","last_peered":"2024-02-21T23:09:54.247863+0000","last_clean":"2024-02-21T23:09:54.247863+0000","last_became_active":"2024-02-21T23:09:53.277111+0000","last_became_peered":"2024-02-21T23:09:53.277111+0000","last_unstale":"2024-02-21T23:09:54.247863+0000","last_undegraded":"2024-02-21T23:09:54.247863+0000","last_fullsized":"2024-02-21T23:09:54.247863+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:23:26.494398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.281830+0000","last_change":"2024-02-21T23:09:54.280400+0000","last_active":"2024-02-21T23:09:54.281830+0000","last_peered":"2024-02-21T23:09:54.281830+0000","last_clean":"2024-02-21T23:09:54.281830+0000","last_became_active":"2024-02-21T23:09:54.280102+0000","last_became_peered":"2024-02-21T23:09:54.280102+0000","last_unstale":"2024-02-21T23:09:54.281830+0000","last_undegraded":"2024-02-21T23:09:54.281830+0000","last_fullsized":"2024-02-21T23:09:54.281830+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:59:59.717557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":14,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.272363+0000","last_change":"2024-02-21T23:09:53.282939+0000","last_active":"2024-02-21T23:09:55.272363+0000","last_peered":"2024-02-21T23:09:55.272363+0000","last_clean":"2024-02-21T23:09:55.272363+0000","last_became_active":"2024-02-21T23:09:53.282612+0000","last_became_peered":"2024-02-21T23:09:53.282612+0000","last_unstale":"2024-02-21T23:09:55.272363+0000","last_undegraded":"2024-02-21T23:09:55.272363+0000","last_fullsized":"2024-02-21T23:09:55.272363+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:30:40.609835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.287337+0000","last_change":"2024-02-21T23:09:54.282302+0000","last_active":"2024-02-21T23:09:54.287337+0000","last_peered":"2024-02-21T23:09:54.287337+0000","last_clean":"2024-02-21T23:09:54.287337+0000","last_became_active":"2024-02-21T23:09:54.280468+0000","last_became_peered":"2024-02-21T23:09:54.280468+0000","last_unstale":"2024-02-21T23:09:54.287337+0000","last_undegraded":"2024-02-21T23:09:54.287337+0000","last_fullsized":"2024-02-21T23:09:54.287337+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:34:50.354067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":12,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.268910+0000","last_change":"2024-02-21T23:09:53.283837+0000","last_active":"2024-02-21T23:09:54.268910+0000","last_peered":"2024-02-21T23:09:54.268910+0000","last_clean":"2024-02-21T23:09:54.268910+0000","last_became_active":"2024-02-21T23:09:53.283541+0000","last_became_peered":"2024-02-21T23:09:53.283541+0000","last_unstale":"2024-02-21T23:09:54.268910+0000","last_undegraded":"2024-02-21T23:09:54.268910+0000","last_fullsized":"2024-02-21T23:09:54.268910+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:28:53.062636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.283635+0000","last_change":"2024-02-21T23:09:54.274229+0000","last_active":"2024-02-21T23:09:54.283635+0000","last_peered":"2024-02-21T23:09:54.283635+0000","last_clean":"2024-02-21T23:09:54.283635+0000","last_became_active":"2024-02-21T23:09:54.271365+0000","last_became_peered":"2024-02-21T23:09:54.271365+0000","last_unstale":"2024-02-21T23:09:54.283635+0000","last_undegraded":"2024-02-21T23:09:54.283635+0000","last_fullsized":"2024-02-21T23:09:54.283635+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:08:50.519955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.7","version":"150'2","reported_seq":14,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.303030+0000","last_change":"2024-02-21T23:09:53.283873+0000","last_active":"2024-02-21T23:09:54.303030+0000","last_peered":"2024-02-21T23:09:54.303030+0000","last_clean":"2024-02-21T23:09:54.303030+0000","last_became_active":"2024-02-21T23:09:53.282829+0000","last_became_peered":"2024-02-21T23:09:53.282829+0000","last_unstale":"2024-02-21T23:09:54.303030+0000","last_undegraded":"2024-02-21T23:09:54.303030+0000","last_fullsized":"2024-02-21T23:09:54.303030+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:13:32.425310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.290237+0000","last_change":"2024-02-21T23:09:54.277479+0000","last_active":"2024-02-21T23:09:55.290237+0000","last_peered":"2024-02-21T23:09:55.290237+0000","last_clean":"2024-02-21T23:09:55.290237+0000","last_became_active":"2024-02-21T23:09:54.277065+0000","last_became_peered":"2024-02-21T23:09:54.277065+0000","last_unstale":"2024-02-21T23:09:55.290237+0000","last_undegraded":"2024-02-21T23:09:55.290237+0000","last_fullsized":"2024-02-21T23:09:55.290237+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:07:24.304915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.6","version":"151'1","reported_seq":15,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.289075+0000","last_change":"2024-02-21T23:09:53.285874+0000","last_active":"2024-02-21T23:09:55.289075+0000","last_peered":"2024-02-21T23:09:55.289075+0000","last_clean":"2024-02-21T23:09:55.289075+0000","last_became_active":"2024-02-21T23:09:53.285569+0000","last_became_peered":"2024-02-21T23:09:53.285569+0000","last_unstale":"2024-02-21T23:09:55.289075+0000","last_undegraded":"2024-02-21T23:09:55.289075+0000","last_fullsized":"2024-02-21T23:09:55.289075+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:52:56.873078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.272128+0000","last_change":"2024-02-21T23:09:54.286102+0000","last_active":"2024-02-21T23:09:55.272128+0000","last_peered":"2024-02-21T23:09:55.272128+0000","last_clean":"2024-02-21T23:09:55.272128+0000","last_became_active":"2024-02-21T23:09:54.283073+0000","last_became_peered":"2024-02-21T23:09:54.283073+0000","last_unstale":"2024-02-21T23:09:55.272128+0000","last_undegraded":"2024-02-21T23:09:55.272128+0000","last_fullsized":"2024-02-21T23:09:55.272128+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:39:05.096862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":14,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.273794+0000","last_change":"2024-02-21T23:09:53.285730+0000","last_active":"2024-02-21T23:09:55.273794+0000","last_peered":"2024-02-21T23:09:55.273794+0000","last_clean":"2024-02-21T23:09:55.273794+0000","last_became_active":"2024-02-21T23:09:53.285373+0000","last_became_peered":"2024-02-21T23:09:53.285373+0000","last_unstale":"2024-02-21T23:09:55.273794+0000","last_undegraded":"2024-02-21T23:09:55.273794+0000","last_fullsized":"2024-02-21T23:09:55.273794+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:04:15.300172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.b","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.290256+0000","last_change":"2024-02-21T23:09:54.277483+0000","last_active":"2024-02-21T23:09:55.290256+0000","last_peered":"2024-02-21T23:09:55.290256+0000","last_clean":"2024-02-21T23:09:55.290256+0000","last_became_active":"2024-02-21T23:09:54.277170+0000","last_became_peered":"2024-02-21T23:09:54.277170+0000","last_unstale":"2024-02-21T23:09:55.290256+0000","last_undegraded":"2024-02-21T23:09:55.290256+0000","last_fullsized":"2024-02-21T23:09:55.290256+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:33:07.362294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":14,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.273255+0000","last_change":"2024-02-21T23:09:53.267455+0000","last_active":"2024-02-21T23:09:55.273255+0000","last_peered":"2024-02-21T23:09:55.273255+0000","last_clean":"2024-02-21T23:09:55.273255+0000","last_became_active":"2024-02-21T23:09:53.266974+0000","last_became_peered":"2024-02-21T23:09:53.266974+0000","last_unstale":"2024-02-21T23:09:55.273255+0000","last_undegraded":"2024-02-21T23:09:55.273255+0000","last_fullsized":"2024-02-21T23:09:55.273255+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:33:26.269092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.276551+0000","last_change":"2024-02-21T23:09:54.275273+0000","last_active":"2024-02-21T23:09:55.276551+0000","last_peered":"2024-02-21T23:09:55.276551+0000","last_clean":"2024-02-21T23:09:55.276551+0000","last_became_active":"2024-02-21T23:09:54.274287+0000","last_became_peered":"2024-02-21T23:09:54.274287+0000","last_unstale":"2024-02-21T23:09:55.276551+0000","last_undegraded":"2024-02-21T23:09:55.276551+0000","last_fullsized":"2024-02-21T23:09:55.276551+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:18:40.139538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":14,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.275660+0000","last_change":"2024-02-21T23:09:53.279782+0000","last_active":"2024-02-21T23:09:55.275660+0000","last_peered":"2024-02-21T23:09:55.275660+0000","last_clean":"2024-02-21T23:09:55.275660+0000","last_became_active":"2024-02-21T23:09:53.279477+0000","last_became_peered":"2024-02-21T23:09:53.279477+0000","last_unstale":"2024-02-21T23:09:55.275660+0000","last_undegraded":"2024-02-21T23:09:55.275660+0000","last_fullsized":"2024-02-21T23:09:55.275660+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:41:01.169723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.d","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.286643+0000","last_change":"2024-02-21T23:09:54.279409+0000","last_active":"2024-02-21T23:09:54.286643+0000","last_peered":"2024-02-21T23:09:54.286643+0000","last_clean":"2024-02-21T23:09:54.286643+0000","last_became_active":"2024-02-21T23:09:54.279107+0000","last_became_peered":"2024-02-21T23:09:54.279107+0000","last_unstale":"2024-02-21T23:09:54.286643+0000","last_undegraded":"2024-02-21T23:09:54.286643+0000","last_fullsized":"2024-02-21T23:09:54.286643+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:08:50.519955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":14,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.273539+0000","last_change":"2024-02-21T23:09:53.278959+0000","last_active":"2024-02-21T23:09:55.273539+0000","last_peered":"2024-02-21T23:09:55.273539+0000","last_clean":"2024-02-21T23:09:55.273539+0000","last_became_active":"2024-02-21T23:09:53.278673+0000","last_became_peered":"2024-02-21T23:09:53.278673+0000","last_unstale":"2024-02-21T23:09:55.273539+0000","last_undegraded":"2024-02-21T23:09:55.273539+0000","last_fullsized":"2024-02-21T23:09:55.273539+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:42:15.971409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.e","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.273582+0000","last_change":"2024-02-21T23:09:54.282920+0000","last_active":"2024-02-21T23:09:55.273582+0000","last_peered":"2024-02-21T23:09:55.273582+0000","last_clean":"2024-02-21T23:09:55.273582+0000","last_became_active":"2024-02-21T23:09:54.282544+0000","last_became_peered":"2024-02-21T23:09:54.282544+0000","last_unstale":"2024-02-21T23:09:55.273582+0000","last_undegraded":"2024-02-21T23:09:55.273582+0000","last_fullsized":"2024-02-21T23:09:55.273582+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:40:05.206906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":12,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.249891+0000","last_change":"2024-02-21T23:09:53.275586+0000","last_active":"2024-02-21T23:09:54.249891+0000","last_peered":"2024-02-21T23:09:54.249891+0000","last_clean":"2024-02-21T23:09:54.249891+0000","last_became_active":"2024-02-21T23:09:53.274828+0000","last_became_peered":"2024-02-21T23:09:53.274828+0000","last_unstale":"2024-02-21T23:09:54.249891+0000","last_undegraded":"2024-02-21T23:09:54.249891+0000","last_fullsized":"2024-02-21T23:09:54.249891+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:02:10.230470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.f","version":"0'0","reported_seq":12,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.272618+0000","last_change":"2024-02-21T23:09:54.277320+0000","last_active":"2024-02-21T23:09:55.272618+0000","last_peered":"2024-02-21T23:09:55.272618+0000","last_clean":"2024-02-21T23:09:55.272618+0000","last_became_active":"2024-02-21T23:09:54.276368+0000","last_became_peered":"2024-02-21T23:09:54.276368+0000","last_unstale":"2024-02-21T23:09:55.272618+0000","last_undegraded":"2024-02-21T23:09:55.272618+0000","last_fullsized":"2024-02-21T23:09:55.272618+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:40:37.285836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":12,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.260467+0000","last_change":"2024-02-21T23:09:53.278027+0000","last_active":"2024-02-21T23:09:54.260467+0000","last_peered":"2024-02-21T23:09:54.260467+0000","last_clean":"2024-02-21T23:09:54.260467+0000","last_became_active":"2024-02-21T23:09:53.277660+0000","last_became_peered":"2024-02-21T23:09:53.277660+0000","last_unstale":"2024-02-21T23:09:54.260467+0000","last_undegraded":"2024-02-21T23:09:54.260467+0000","last_fullsized":"2024-02-21T23:09:54.260467+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:07:23.287998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":9,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.284794+0000","last_change":"2024-02-21T23:09:54.277356+0000","last_active":"2024-02-21T23:09:54.284794+0000","last_peered":"2024-02-21T23:09:54.284794+0000","last_clean":"2024-02-21T23:09:54.284794+0000","last_became_active":"2024-02-21T23:09:54.277087+0000","last_became_peered":"2024-02-21T23:09:54.277087+0000","last_unstale":"2024-02-21T23:09:54.284794+0000","last_undegraded":"2024-02-21T23:09:54.284794+0000","last_fullsized":"2024-02-21T23:09:54.284794+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:55:18.591062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":14,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-21T23:09:55.273779+0000","last_change":"2024-02-21T23:09:53.286181+0000","last_active":"2024-02-21T23:09:55.273779+0000","last_peered":"2024-02-21T23:09:55.273779+0000","last_clean":"2024-02-21T23:09:55.273779+0000","last_became_active":"2024-02-21T23:09:53.285849+0000","last_became_peered":"2024-02-21T23:09:53.285849+0000","last_unstale":"2024-02-21T23:09:55.273779+0000","last_undegraded":"2024-02-21T23:09:55.273779+0000","last_fullsized":"2024-02-21T23:09:55.273779+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:51:08.543135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":415,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-02-21T23:09:54.268984+0000","last_change":"2024-02-21T23:02:32.289335+0000","last_active":"2024-02-21T23:09:54.268984+0000","last_peered":"2024-02-21T23:09:54.268984+0000","last_clean":"2024-02-21T23:09:54.268984+0000","last_became_active":"2024-02-21T23:02:32.286913+0000","last_became_peered":"2024-02-21T23:02:32.286913+0000","last_unstale":"2024-02-21T23:09:54.268984+0000","last_undegraded":"2024-02-21T23:09:54.268984+0000","last_fullsized":"2024-02-21T23:09:54.268984+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_clean_scrub_stamp":"2024-02-21T23:02:31.172569+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:36:23.987374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":24,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":23,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574939,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33092,"kb_used_data":1908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338748,"statfs":{"total":96636764160,"available":96602877952,"internally_reserved":0,"allocated":1953792,"data_stored":1509313,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574940,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33036,"kb_used_data":1856,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338804,"statfs":{"total":96636764160,"available":96602935296,"internally_reserved":0,"allocated":1900544,"data_stored":925637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574939,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33032,"kb_used_data":1852,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338808,"statfs":{"total":96636764160,"available":96602939392,"internally_reserved":0,"allocated":1896448,"data_stored":919069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574940,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33040,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338800,"statfs":{"total":96636764160,"available":96602931200,"internally_reserved":0,"allocated":1904640,"data_stored":919709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574940,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33060,"kb_used_data":1876,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338780,"statfs":{"total":96636764160,"available":96602910720,"internally_reserved":0,"allocated":1921024,"data_stored":927853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574941,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33040,"kb_used_data":1856,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338800,"statfs":{"total":96636764160,"available":96602931200,"internally_reserved":0,"allocated":1900544,"data_stored":925637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574939,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33040,"kb_used_data":1856,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338800,"statfs":{"total":96636764160,"available":96602931200,"internally_reserved":0,"allocated":1900544,"data_stored":920557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574939,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33100,"kb_used_data":1916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338740,"statfs":{"total":96636764160,"available":96602869760,"internally_reserved":0,"allocated":1961984,"data_stored":1509417,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-21T23:09:59.262 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-02-21T23:09:59.262 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-02-21T23:09:59.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.539+0000 7f382da00640 1 Processor -- start 2024-02-21T23:09:59.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.540+0000 7f382da00640 1 -- start start 2024-02-21T23:09:59.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.541+0000 7f382da00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f38280c8b20 0x7f38281558f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:59.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.541+0000 7f382da00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38280c8fb0 con 0x7f38281527e0 2024-02-21T23:09:59.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.541+0000 7f382da00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38280c5010 con 0x7f38280c86c0 2024-02-21T23:09:59.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.541+0000 7f382da00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3828152bc0 con 0x7f38280c8b20 2024-02-21T23:09:59.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.541+0000 7f3826a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f38281527e0 0x7f38280c4590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34354/0 (socket says 172.21.15.28:34354) 2024-02-21T23:09:59.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.541+0000 7f3826a00640 1 -- 172.21.15.28:0/1990805845 learned_addr learned my addr 172.21.15.28:0/1990805845 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:09:59.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.542+0000 7f3827400640 1 --2- 172.21.15.28:0/1990805845 >> v2:172.21.15.81:3300/0 conn(0x7f38280c8b20 0x7f38281558f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:59.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.542+0000 7f3826000640 1 -- 172.21.15.28:0/1990805845 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3159140607 0 0) 0x7f38280c8fb0 con 0x7f38281527e0 2024-02-21T23:09:59.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.542+0000 7f3826000640 1 -- 172.21.15.28:0/1990805845 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f380c003520 con 0x7f38281527e0 2024-02-21T23:09:59.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.542+0000 7f3826000640 1 -- 172.21.15.28:0/1990805845 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1868173104 0 0) 0x7f380c003520 con 0x7f38281527e0 2024-02-21T23:09:59.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.542+0000 7f3826000640 1 -- 172.21.15.28:0/1990805845 >> v2:172.21.15.81:3300/0 conn(0x7f38280c8b20 msgr2=0x7f38281558f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:59.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.542+0000 7f3826000640 1 --2- 172.21.15.28:0/1990805845 >> v2:172.21.15.81:3300/0 conn(0x7f38280c8b20 0x7f38281558f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:59.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.542+0000 7f3826000640 1 -- 172.21.15.28:0/1990805845 >> v1:172.21.15.81:6789/0 conn(0x7f38280c86c0 legacy=0x7f38281551e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:59.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.542+0000 7f3826000640 1 -- 172.21.15.28:0/1990805845 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38280c6760 con 0x7f38281527e0 2024-02-21T23:09:59.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.542+0000 7f3826000640 1 -- 172.21.15.28:0/1990805845 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f3818001670 con 0x7f38281527e0 2024-02-21T23:09:59.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.542+0000 7f3826000640 1 -- 172.21.15.28:0/1990805845 >> v1:172.21.15.28:6789/0 conn(0x7f38281527e0 legacy=0x7f38280c4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:59.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.542+0000 7f3826000640 1 --2- 172.21.15.28:0/1990805845 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f380c0039b0 0x7f380c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:59.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.542+0000 7f3826000640 1 --2- 172.21.15.28:0/1990805845 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f380c004800 0x7f380c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:59.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.542+0000 7f3826000640 1 --2- 172.21.15.28:0/1990805845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f380c0051c0 0x7f380c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:59.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.542+0000 7f3826000640 1 -- 172.21.15.28:0/1990805845 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3828152bc0 con 0x7f380c0051c0 2024-02-21T23:09:59.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.542+0000 7f3826000640 1 -- 172.21.15.28:0/1990805845 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f380c00c070 con 0x7f380c0039b0 2024-02-21T23:09:59.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.542+0000 7f3826000640 1 -- 172.21.15.28:0/1990805845 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f380c00c270 con 0x7f380c004800 2024-02-21T23:09:59.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.543+0000 7f3827400640 1 --2- 172.21.15.28:0/1990805845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f380c0051c0 0x7f380c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:59.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.543+0000 7f3826a00640 1 --2- 172.21.15.28:0/1990805845 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f380c0039b0 0x7f380c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:59.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.543+0000 7f381fe00640 1 --2- 172.21.15.28:0/1990805845 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f380c004800 0x7f380c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:59.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.543+0000 7f3827400640 1 -- 172.21.15.28:0/1990805845 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f380c004800 msgr2=0x7f380c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:59.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.543+0000 7f3827400640 1 --2- 172.21.15.28:0/1990805845 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f380c004800 0x7f380c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:59.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.543+0000 7f3827400640 1 -- 172.21.15.28:0/1990805845 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f380c0039b0 msgr2=0x7f380c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:09:59.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.543+0000 7f3827400640 1 --2- 172.21.15.28:0/1990805845 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f380c0039b0 0x7f380c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:59.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.543+0000 7f3827400640 1 -- 172.21.15.28:0/1990805845 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f380c003520 con 0x7f380c0051c0 2024-02-21T23:09:59.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.544+0000 7f3827400640 1 -- 172.21.15.28:0/1990805845 reap_dead start 2024-02-21T23:09:59.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.544+0000 7f3827400640 1 --2- 172.21.15.28:0/1990805845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f380c0051c0 0x7f380c00b9c0 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f3820009fc0 tx=0x7f380c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b37405043e1d8611 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:59.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.545+0000 7f3826000640 1 -- 172.21.15.28:0/1990805845 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3820009c10 con 0x7f380c0051c0 2024-02-21T23:09:59.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.545+0000 7f3826000640 1 -- 172.21.15.28:0/1990805845 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f382002cca0 con 0x7f380c0051c0 2024-02-21T23:09:59.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.545+0000 7f3826000640 1 -- 172.21.15.28:0/1990805845 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3820003bd0 con 0x7f380c0051c0 2024-02-21T23:09:59.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.546+0000 7f382da00640 1 -- 172.21.15.28:0/1990805845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f380c0051c0 msgr2=0x7f380c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:59.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.546+0000 7f382da00640 1 --2- 172.21.15.28:0/1990805845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f380c0051c0 0x7f380c00b9c0 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f3820009fc0 tx=0x7f380c005830 comp rx=0 tx=0).stop 2024-02-21T23:09:59.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.546+0000 7f382da00640 1 -- 172.21.15.28:0/1990805845 shutdown_connections 2024-02-21T23:09:59.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.546+0000 7f382da00640 1 --2- 172.21.15.28:0/1990805845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f380c0051c0 0x7f380c00b9c0 unknown :-1 s=CLOSED pgs=1694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:59.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.546+0000 7f382da00640 1 -- 172.21.15.28:0/1990805845 >> 172.21.15.28:0/1990805845 conn(0x7f382806b4a0 msgr2=0x7f38280aa550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:59.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.546+0000 7f382da00640 1 -- 172.21.15.28:0/1990805845 shutdown_connections 2024-02-21T23:09:59.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.546+0000 7f382da00640 1 -- 172.21.15.28:0/1990805845 wait complete. 2024-02-21T23:09:59.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.549+0000 7f382da00640 1 Processor -- start 2024-02-21T23:09:59.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.549+0000 7f382da00640 1 -- start start 2024-02-21T23:09:59.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.550+0000 7f382da00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f380c0051c0 0x7f38280ab800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:59.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.550+0000 7f382da00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f38280abd40 0x7f382814adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:59.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.550+0000 7f382da00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f382814b2f0 0x7f38280a94c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:59.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.550+0000 7f382da00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38280c6d20 con 0x7f380c0051c0 2024-02-21T23:09:59.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.550+0000 7f382da00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38280c7090 con 0x7f38280abd40 2024-02-21T23:09:59.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.550+0000 7f382da00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38280ab320 con 0x7f382814b2f0 2024-02-21T23:09:59.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.551+0000 7f3826a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f380c0051c0 0x7f38280ab800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:59.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.551+0000 7f3826a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f380c0051c0 0x7f38280ab800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33818/0 (socket says 172.21.15.28:33818) 2024-02-21T23:09:59.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.551+0000 7f3826a00640 1 -- 172.21.15.28:0/1443462093 learned_addr learned my addr 172.21.15.28:0/1443462093 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:09:59.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.551+0000 7f3827400640 1 --2- 172.21.15.28:0/1443462093 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f382814b2f0 0x7f38280a94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:59.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.551+0000 7f381fe00640 1 --2- 172.21.15.28:0/1443462093 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f38280abd40 0x7f382814adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:59.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.551+0000 7f3826a00640 1 -- 172.21.15.28:0/1443462093 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f382814b2f0 msgr2=0x7f38280a94c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:59.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.551+0000 7f3826a00640 1 --2- 172.21.15.28:0/1443462093 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f382814b2f0 0x7f38280a94c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:59.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.551+0000 7f3826a00640 1 -- 172.21.15.28:0/1443462093 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f38280abd40 msgr2=0x7f382814adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:59.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.551+0000 7f3826a00640 1 --2- 172.21.15.28:0/1443462093 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f38280abd40 0x7f382814adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:59.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.551+0000 7f3826a00640 1 -- 172.21.15.28:0/1443462093 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38280ad3d0 con 0x7f380c0051c0 2024-02-21T23:09:59.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.551+0000 7f3827400640 1 --2- 172.21.15.28:0/1443462093 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f382814b2f0 0x7f38280a94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:09:59.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.551+0000 7f381fe00640 1 --2- 172.21.15.28:0/1443462093 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f38280abd40 0x7f382814adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:09:59.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.552+0000 7f3826a00640 1 --2- 172.21.15.28:0/1443462093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f380c0051c0 0x7f38280ab800 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f381800e830 tx=0x7f381800ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:59.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.552+0000 7f381ea00640 1 -- 172.21.15.28:0/1443462093 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f381800c6c0 con 0x7f380c0051c0 2024-02-21T23:09:59.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.552+0000 7f381ea00640 1 -- 172.21.15.28:0/1443462093 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3818018070 con 0x7f380c0051c0 2024-02-21T23:09:59.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.552+0000 7f381ea00640 1 -- 172.21.15.28:0/1443462093 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f381800caa0 con 0x7f380c0051c0 2024-02-21T23:09:59.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.552+0000 7f382da00640 1 -- 172.21.15.28:0/1443462093 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f382814bc10 con 0x7f380c0051c0 2024-02-21T23:09:59.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.552+0000 7f382da00640 1 -- 172.21.15.28:0/1443462093 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38280ac1c0 con 0x7f380c0051c0 2024-02-21T23:09:59.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.553+0000 7f382da00640 1 -- 172.21.15.28:0/1443462093 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38280aa600 con 0x7f380c0051c0 2024-02-21T23:09:59.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.555+0000 7f381ea00640 1 -- 172.21.15.28:0/1443462093 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3818013420 con 0x7f380c0051c0 2024-02-21T23:09:59.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.555+0000 7f381ea00640 1 --2- 172.21.15.28:0/1443462093 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f380406b500 0x7f380406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:09:59.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.556+0000 7f381ea00640 1 -- 172.21.15.28:0/1443462093 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f381808a080 con 0x7f380c0051c0 2024-02-21T23:09:59.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.556+0000 7f381fe00640 1 --2- 172.21.15.28:0/1443462093 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f380406b500 0x7f380406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:09:59.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.557+0000 7f381fe00640 1 --2- 172.21.15.28:0/1443462093 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f380406b500 0x7f380406d9c0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f3810000c00 tx=0x7f3810009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:09:59.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.560+0000 7f381ea00640 1 -- 172.21.15.28:0/1443462093 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3818058d70 con 0x7f380c0051c0 2024-02-21T23:09:59.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.867+0000 7f382da00640 1 -- 172.21.15.28:0/1443462093 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f3828068640 con 0x7f380c0051c0 2024-02-21T23:09:59.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.871+0000 7f381ea00640 1 -- 172.21.15.28:0/1443462093 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v18) v1 ==== 190+0+147 (secure 0 0 0) 0x7f3818058670 con 0x7f380c0051c0 2024-02-21T23:09:59.859 INFO:teuthology.orchestra.run.smithi028.stdout:[client.0] 2024-02-21T23:09:59.859 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQDTgNZlILvYLxAAoREWxKKT7YcXnhkgk8YvoQ== 2024-02-21T23:09:59.859 INFO:teuthology.orchestra.run.smithi028.stdout: caps mds = "allow" 2024-02-21T23:09:59.859 INFO:teuthology.orchestra.run.smithi028.stdout: caps mon = "allow r" 2024-02-21T23:09:59.859 INFO:teuthology.orchestra.run.smithi028.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-02-21T23:09:59.860 INFO:teuthology.orchestra.run.smithi028.stderr:updated caps for client.0 2024-02-21T23:09:59.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.874+0000 7f382da00640 1 -- 172.21.15.28:0/1443462093 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f380406b500 msgr2=0x7f380406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:59.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.874+0000 7f382da00640 1 --2- 172.21.15.28:0/1443462093 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f380406b500 0x7f380406d9c0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f3810000c00 tx=0x7f3810009040 comp rx=0 tx=0).stop 2024-02-21T23:09:59.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.874+0000 7f382da00640 1 -- 172.21.15.28:0/1443462093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f380c0051c0 msgr2=0x7f38280ab800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:09:59.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.874+0000 7f382da00640 1 --2- 172.21.15.28:0/1443462093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f380c0051c0 0x7f38280ab800 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f381800e830 tx=0x7f381800ecf0 comp rx=0 tx=0).stop 2024-02-21T23:09:59.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.874+0000 7f382da00640 1 -- 172.21.15.28:0/1443462093 shutdown_connections 2024-02-21T23:09:59.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.874+0000 7f382da00640 1 --2- 172.21.15.28:0/1443462093 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f380406b500 0x7f380406d9c0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:59.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.874+0000 7f382da00640 1 --2- 172.21.15.28:0/1443462093 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f382814b2f0 0x7f38280a94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:59.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.874+0000 7f382da00640 1 --2- 172.21.15.28:0/1443462093 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f38280abd40 0x7f382814adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:59.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.874+0000 7f382da00640 1 --2- 172.21.15.28:0/1443462093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f380c0051c0 0x7f38280ab800 unknown :-1 s=CLOSED pgs=1695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:09:59.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.874+0000 7f382da00640 1 -- 172.21.15.28:0/1443462093 >> 172.21.15.28:0/1443462093 conn(0x7f382806b4a0 msgr2=0x7f3828150a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:09:59.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.874+0000 7f382da00640 1 -- 172.21.15.28:0/1443462093 shutdown_connections 2024-02-21T23:09:59.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:09:59.874+0000 7f382da00640 1 -- 172.21.15.28:0/1443462093 wait complete. 2024-02-21T23:09:59.881 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:10:00.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.102+0000 7fc79a200640 1 Processor -- start 2024-02-21T23:10:00.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.102+0000 7fc79a200640 1 -- start start 2024-02-21T23:10:00.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.104+0000 7fc79a200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc7940aff90 0x7fc7940c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:00.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.104+0000 7fc79a200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fc79406dbf0 0x7fc794158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:00.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.104+0000 7fc79a200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7940c74c0 con 0x7fc7940c5720 2024-02-21T23:10:00.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.104+0000 7fc79a200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7940c7270 con 0x7fc7940aff90 2024-02-21T23:10:00.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.104+0000 7fc79a200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc794158630 con 0x7fc79406dbf0 2024-02-21T23:10:00.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.104+0000 7fc798e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc7940aff90 0x7fc7940c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:00.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.104+0000 7fc798e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc7940aff90 0x7fc7940c51e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33820/0 (socket says 172.21.15.28:33820) 2024-02-21T23:10:00.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.104+0000 7fc798e00640 1 -- 172.21.15.28:0/1837005066 learned_addr learned my addr 172.21.15.28:0/1837005066 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:00.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.105+0000 7fc793e00640 1 --2- 172.21.15.28:0/1837005066 >> v2:172.21.15.81:3300/0 conn(0x7fc79406dbf0 0x7fc794158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:00.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.105+0000 7fc798e00640 1 -- 172.21.15.28:0/1837005066 >> v2:172.21.15.81:3300/0 conn(0x7fc79406dbf0 msgr2=0x7fc794158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:00.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.105+0000 7fc798e00640 1 --2- 172.21.15.28:0/1837005066 >> v2:172.21.15.81:3300/0 conn(0x7fc79406dbf0 0x7fc794158ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:00.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.105+0000 7fc798e00640 1 -- 172.21.15.28:0/1837005066 >> v1:172.21.15.81:6789/0 conn(0x7fc7940c5720 legacy=0x7fc79406c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:00.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.105+0000 7fc798e00640 1 -- 172.21.15.28:0/1837005066 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7940c77a0 con 0x7fc7940aff90 2024-02-21T23:10:00.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.106+0000 7fc798e00640 1 --2- 172.21.15.28:0/1837005066 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7940aff90 0x7fc7940c51e0 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7fc7940c73f0 tx=0x7fc78802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94d72a6fa0ebb6f2 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:00.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.106+0000 7fc793400640 1 -- 172.21.15.28:0/1837005066 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc788035020 con 0x7fc7940aff90 2024-02-21T23:10:00.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.106+0000 7fc793400640 1 -- 172.21.15.28:0/1837005066 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc788031870 con 0x7fc7940aff90 2024-02-21T23:10:00.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.106+0000 7fc793400640 1 -- 172.21.15.28:0/1837005066 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc788031c40 con 0x7fc7940aff90 2024-02-21T23:10:00.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.107+0000 7fc79a200640 1 -- 172.21.15.28:0/1837005066 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7940aff90 msgr2=0x7fc7940c51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:00.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.107+0000 7fc79a200640 1 --2- 172.21.15.28:0/1837005066 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7940aff90 0x7fc7940c51e0 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7fc7940c73f0 tx=0x7fc78802b910 comp rx=0 tx=0).stop 2024-02-21T23:10:00.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.107+0000 7fc79a200640 1 -- 172.21.15.28:0/1837005066 shutdown_connections 2024-02-21T23:10:00.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.107+0000 7fc79a200640 1 --2- 172.21.15.28:0/1837005066 >> v2:172.21.15.81:3300/0 conn(0x7fc79406dbf0 0x7fc794158ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:00.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.107+0000 7fc79a200640 1 --2- 172.21.15.28:0/1837005066 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7940aff90 0x7fc7940c51e0 unknown :-1 s=CLOSED pgs=1696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:00.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.107+0000 7fc79a200640 1 -- 172.21.15.28:0/1837005066 >> 172.21.15.28:0/1837005066 conn(0x7fc7940ac370 msgr2=0x7fc7940ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:00.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.107+0000 7fc79a200640 1 -- 172.21.15.28:0/1837005066 shutdown_connections 2024-02-21T23:10:00.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.107+0000 7fc79a200640 1 -- 172.21.15.28:0/1837005066 wait complete. 2024-02-21T23:10:00.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.110+0000 7fc79a200640 1 Processor -- start 2024-02-21T23:10:00.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.110+0000 7fc79a200640 1 -- start start 2024-02-21T23:10:00.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.112+0000 7fc79a200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc79406dbf0 0x7fc7941eb4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:00.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.112+0000 7fc79a200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc7940aff90 0x7fc7941515e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:00.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.112+0000 7fc79a200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7940c5720 0x7fc7940c1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:00.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.112+0000 7fc79a200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7940c8760 con 0x7fc7940c5720 2024-02-21T23:10:00.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.112+0000 7fc79a200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7940be0c0 con 0x7fc79406dbf0 2024-02-21T23:10:00.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.112+0000 7fc79a200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7940c88e0 con 0x7fc7940aff90 2024-02-21T23:10:00.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.112+0000 7fc799800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7940c5720 0x7fc7940c1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:00.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.112+0000 7fc799800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7940c5720 0x7fc7940c1930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33834/0 (socket says 172.21.15.28:33834) 2024-02-21T23:10:00.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.112+0000 7fc799800640 1 -- 172.21.15.28:0/2215707195 learned_addr learned my addr 172.21.15.28:0/2215707195 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:00.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.112+0000 7fc798e00640 1 --2- 172.21.15.28:0/2215707195 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc79406dbf0 0x7fc7941eb4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:00.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.112+0000 7fc793e00640 1 --2- 172.21.15.28:0/2215707195 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc7940aff90 0x7fc7941515e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:00.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.113+0000 7fc799800640 1 -- 172.21.15.28:0/2215707195 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc7940aff90 msgr2=0x7fc7941515e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:00.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.113+0000 7fc799800640 1 --2- 172.21.15.28:0/2215707195 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc7940aff90 0x7fc7941515e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:00.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.113+0000 7fc799800640 1 -- 172.21.15.28:0/2215707195 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc79406dbf0 msgr2=0x7fc7941eb4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:00.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.113+0000 7fc799800640 1 --2- 172.21.15.28:0/2215707195 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc79406dbf0 0x7fc7941eb4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:00.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.113+0000 7fc799800640 1 -- 172.21.15.28:0/2215707195 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc794154a00 con 0x7fc7940c5720 2024-02-21T23:10:00.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.113+0000 7fc798e00640 1 --2- 172.21.15.28:0/2215707195 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc79406dbf0 0x7fc7941eb4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:10:00.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.113+0000 7fc799800640 1 --2- 172.21.15.28:0/2215707195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7940c5720 0x7fc7940c1930 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7fc78c00b970 tx=0x7fc78c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:00.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.113+0000 7fc790c00640 1 -- 172.21.15.28:0/2215707195 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc78c019070 con 0x7fc7940c5720 2024-02-21T23:10:00.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.113+0000 7fc790c00640 1 -- 172.21.15.28:0/2215707195 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc78c00cae0 con 0x7fc7940c5720 2024-02-21T23:10:00.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.113+0000 7fc79a200640 1 -- 172.21.15.28:0/2215707195 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc794152af0 con 0x7fc7940c5720 2024-02-21T23:10:00.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.113+0000 7fc790c00640 1 -- 172.21.15.28:0/2215707195 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc78c014440 con 0x7fc7940c5720 2024-02-21T23:10:00.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.113+0000 7fc79a200640 1 -- 172.21.15.28:0/2215707195 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7941530b0 con 0x7fc7940c5720 2024-02-21T23:10:00.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.115+0000 7fc79a200640 1 -- 172.21.15.28:0/2215707195 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc79404f290 con 0x7fc7940c5720 2024-02-21T23:10:00.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.117+0000 7fc790c00640 1 -- 172.21.15.28:0/2215707195 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc78c01c070 con 0x7fc7940c5720 2024-02-21T23:10:00.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.117+0000 7fc790c00640 1 --2- 172.21.15.28:0/2215707195 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc77406b500 0x7fc77406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:00.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.118+0000 7fc790c00640 1 -- 172.21.15.28:0/2215707195 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc78c089e90 con 0x7fc7940c5720 2024-02-21T23:10:00.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.118+0000 7fc798e00640 1 --2- 172.21.15.28:0/2215707195 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc77406b500 0x7fc77406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:00.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.119+0000 7fc798e00640 1 --2- 172.21.15.28:0/2215707195 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc77406b500 0x7fc77406d9c0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fc788036000 tx=0x7fc78803c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:00.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.122+0000 7fc790c00640 1 -- 172.21.15.28:0/2215707195 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc78c058bf0 con 0x7fc7940c5720 2024-02-21T23:10:00.199 INFO:tasks.ceph.mds.a.smithi028.stderr:2024-02-21T23:10:00.211+0000 7f3824800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-21T23:10:00.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.363+0000 7fc79a200640 1 -- 172.21.15.28:0/2215707195 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc794153280 con 0x7fc7940c5720 2024-02-21T23:10:00.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.364+0000 7fc790c00640 1 -- 172.21.15.28:0/2215707195 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 158 v158) v1 ==== 95+0+4945 (secure 0 0 0) 0x7fc78c0584f0 con 0x7fc7940c5720 2024-02-21T23:10:00.352 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:10:00.352 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":158,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6389,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1320495906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1320495906},{"type":"v1","addr":"172.21.15.81:6835","nonce":1320495906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":153},{"gid":6392,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/521625945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":521625945},{"type":"v1","addr":"172.21.15.81:6837","nonce":521625945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":153},{"gid":8817,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/2975378719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2975378719},{"type":"v1","addr":"172.21.15.28:6833","nonce":2975378719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":152}],"filesystems":[{"mdsmap":{"epoch":158,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:09:53.730296+0000","modified":"2024-02-21T23:09:56.809990+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8826},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8826":{"gid":8826,"name":"a","rank":0,"incarnation":154,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/2928639298","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2928639298},{"type":"v1","addr":"172.21.15.28:6835","nonce":2928639298}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2024-02-21T23:10:00.353 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 158 2024-02-21T23:10:00.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.367+0000 7fc79a200640 1 -- 172.21.15.28:0/2215707195 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc77406b500 msgr2=0x7fc77406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:00.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.367+0000 7fc79a200640 1 --2- 172.21.15.28:0/2215707195 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc77406b500 0x7fc77406d9c0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fc788036000 tx=0x7fc78803c040 comp rx=0 tx=0).stop 2024-02-21T23:10:00.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.367+0000 7fc79a200640 1 -- 172.21.15.28:0/2215707195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7940c5720 msgr2=0x7fc7940c1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:00.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.367+0000 7fc79a200640 1 --2- 172.21.15.28:0/2215707195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7940c5720 0x7fc7940c1930 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7fc78c00b970 tx=0x7fc78c00be30 comp rx=0 tx=0).stop 2024-02-21T23:10:00.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.368+0000 7fc79a200640 1 -- 172.21.15.28:0/2215707195 shutdown_connections 2024-02-21T23:10:00.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.368+0000 7fc79a200640 1 --2- 172.21.15.28:0/2215707195 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc77406b500 0x7fc77406d9c0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:00.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.368+0000 7fc79a200640 1 --2- 172.21.15.28:0/2215707195 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc7940aff90 0x7fc7941515e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:00.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.368+0000 7fc79a200640 1 --2- 172.21.15.28:0/2215707195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7940c5720 0x7fc7940c1930 unknown :-1 s=CLOSED pgs=1697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:00.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.368+0000 7fc79a200640 1 --2- 172.21.15.28:0/2215707195 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc79406dbf0 0x7fc7941eb4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:00.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.368+0000 7fc79a200640 1 -- 172.21.15.28:0/2215707195 >> 172.21.15.28:0/2215707195 conn(0x7fc7940ac370 msgr2=0x7fc7940ac750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:00.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.368+0000 7fc79a200640 1 -- 172.21.15.28:0/2215707195 shutdown_connections 2024-02-21T23:10:00.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:00.368+0000 7fc79a200640 1 -- 172.21.15.28:0/2215707195 wait complete. 2024-02-21T23:10:00.777 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 158, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:09:53.730296+0000', 'modified': '2024-02-21T23:09:56.809990+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8826}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8826': {'gid': 8826, 'name': 'a', 'rank': 0, 'incarnation': 154, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/2928639298', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 2928639298}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 2928639298}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [24], 'metadata_pool': 23, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:10:00.777 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-21T23:10:00.778 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-02-21T23:10:01.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.023+0000 7f5bbbc00640 1 Processor -- start 2024-02-21T23:10:01.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.023+0000 7f5bbbc00640 1 -- start start 2024-02-21T23:10:01.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.024+0000 7f5bbbc00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5bb406ed50 0x7f5bb40c3280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:01.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.024+0000 7f5bbbc00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f5bb40c37c0 0x7f5bb4158ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:01.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.024+0000 7f5bbbc00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bb40c5af0 con 0x7f5bb40c1c00 2024-02-21T23:10:01.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.024+0000 7f5bbbc00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bb40c5660 con 0x7f5bb406ed50 2024-02-21T23:10:01.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.024+0000 7f5bbbc00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bb40c4d30 con 0x7f5bb40c37c0 2024-02-21T23:10:01.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.024+0000 7f5bb9200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5bb406ed50 0x7f5bb40c3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:01.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.025+0000 7f5bb9200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5bb406ed50 0x7f5bb40c3280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33838/0 (socket says 172.21.15.28:33838) 2024-02-21T23:10:01.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.025+0000 7f5bb9200640 1 -- 172.21.15.28:0/895950388 learned_addr learned my addr 172.21.15.28:0/895950388 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:01.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.025+0000 7f5bb9200640 1 -- 172.21.15.28:0/895950388 >> v2:172.21.15.81:3300/0 conn(0x7f5bb40c37c0 msgr2=0x7f5bb4158ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:01.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.025+0000 7f5bb9200640 1 --2- 172.21.15.28:0/895950388 >> v2:172.21.15.81:3300/0 conn(0x7f5bb40c37c0 0x7f5bb4158ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:01.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.025+0000 7f5bb9200640 1 -- 172.21.15.28:0/895950388 >> v1:172.21.15.81:6789/0 conn(0x7f5bb40c1c00 legacy=0x7f5bb41587c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:01.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.025+0000 7f5bb9200640 1 -- 172.21.15.28:0/895950388 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bb40c5010 con 0x7f5bb406ed50 2024-02-21T23:10:01.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.026+0000 7f5bb9200640 1 --2- 172.21.15.28:0/895950388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bb406ed50 0x7f5bb40c3280 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7f5bb40c2820 tx=0x7f5ba802b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48d920cd2a8db394 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:01.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.027+0000 7f5bb3400640 1 -- 172.21.15.28:0/895950388 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ba8038070 con 0x7f5bb406ed50 2024-02-21T23:10:01.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.027+0000 7f5bb3400640 1 -- 172.21.15.28:0/895950388 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ba802f050 con 0x7f5bb406ed50 2024-02-21T23:10:01.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.027+0000 7f5bb3400640 1 -- 172.21.15.28:0/895950388 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ba8031970 con 0x7f5bb406ed50 2024-02-21T23:10:01.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.027+0000 7f5bbbc00640 1 -- 172.21.15.28:0/895950388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bb406ed50 msgr2=0x7f5bb40c3280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:01.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.027+0000 7f5bbbc00640 1 --2- 172.21.15.28:0/895950388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bb406ed50 0x7f5bb40c3280 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7f5bb40c2820 tx=0x7f5ba802b7f0 comp rx=0 tx=0).stop 2024-02-21T23:10:01.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.027+0000 7f5bbbc00640 1 -- 172.21.15.28:0/895950388 shutdown_connections 2024-02-21T23:10:01.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.027+0000 7f5bbbc00640 1 --2- 172.21.15.28:0/895950388 >> v2:172.21.15.81:3300/0 conn(0x7f5bb40c37c0 0x7f5bb4158ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:01.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.027+0000 7f5bbbc00640 1 --2- 172.21.15.28:0/895950388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bb406ed50 0x7f5bb40c3280 unknown :-1 s=CLOSED pgs=1698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:01.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.027+0000 7f5bbbc00640 1 -- 172.21.15.28:0/895950388 >> 172.21.15.28:0/895950388 conn(0x7f5bb406b820 msgr2=0x7f5bb40c67f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:01.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.028+0000 7f5bbbc00640 1 -- 172.21.15.28:0/895950388 shutdown_connections 2024-02-21T23:10:01.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.028+0000 7f5bbbc00640 1 -- 172.21.15.28:0/895950388 wait complete. 2024-02-21T23:10:01.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.029+0000 7f5bbbc00640 1 Processor -- start 2024-02-21T23:10:01.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.030+0000 7f5bbbc00640 1 -- start start 2024-02-21T23:10:01.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.030+0000 7f5bbbc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bb406ed50 0x7f5bb41dee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:01.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.030+0000 7f5bbbc00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5bb40c1c00 0x7f5bb41eb5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:01.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.030+0000 7f5bbbc00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5bb40c37c0 0x7f5bb41e0e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:01.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.030+0000 7f5bbbc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb40c6070 con 0x7f5bb406ed50 2024-02-21T23:10:01.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.030+0000 7f5bbbc00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb406f130 con 0x7f5bb40c1c00 2024-02-21T23:10:01.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.030+0000 7f5bbbc00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb40c4880 con 0x7f5bb40c37c0 2024-02-21T23:10:01.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.031+0000 7f5bb9200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bb406ed50 0x7f5bb41dee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:01.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.031+0000 7f5bb9200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bb406ed50 0x7f5bb41dee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33840/0 (socket says 172.21.15.28:33840) 2024-02-21T23:10:01.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.031+0000 7f5bb9200640 1 -- 172.21.15.28:0/2708890002 learned_addr learned my addr 172.21.15.28:0/2708890002 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:01.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.031+0000 7f5bb3e00640 1 --2- 172.21.15.28:0/2708890002 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5bb40c1c00 0x7f5bb41eb5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:01.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.031+0000 7f5bb9200640 1 -- 172.21.15.28:0/2708890002 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5bb40c37c0 msgr2=0x7f5bb41e0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:01.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.031+0000 7f5bb9200640 1 --2- 172.21.15.28:0/2708890002 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5bb40c37c0 0x7f5bb41e0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:01.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.031+0000 7f5bb9200640 1 -- 172.21.15.28:0/2708890002 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5bb40c1c00 msgr2=0x7f5bb41eb5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:01.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.031+0000 7f5bb9200640 1 --2- 172.21.15.28:0/2708890002 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5bb40c1c00 0x7f5bb41eb5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:01.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.031+0000 7f5bb9200640 1 -- 172.21.15.28:0/2708890002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bb41eb0f0 con 0x7f5bb406ed50 2024-02-21T23:10:01.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.031+0000 7f5bb3e00640 1 --2- 172.21.15.28:0/2708890002 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5bb40c1c00 0x7f5bb41eb5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:10:01.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.032+0000 7f5bb9200640 1 --2- 172.21.15.28:0/2708890002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bb406ed50 0x7f5bb41dee10 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7f5ba8000c00 tx=0x7f5ba80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:01.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.032+0000 7f5bb0c00640 1 -- 172.21.15.28:0/2708890002 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ba8043070 con 0x7f5bb406ed50 2024-02-21T23:10:01.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.032+0000 7f5bb0c00640 1 -- 172.21.15.28:0/2708890002 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ba803f890 con 0x7f5bb406ed50 2024-02-21T23:10:01.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.032+0000 7f5bbbc00640 1 -- 172.21.15.28:0/2708890002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bb4153170 con 0x7f5bb406ed50 2024-02-21T23:10:01.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.032+0000 7f5bb0c00640 1 -- 172.21.15.28:0/2708890002 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ba8038040 con 0x7f5bb406ed50 2024-02-21T23:10:01.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.032+0000 7f5bbbc00640 1 -- 172.21.15.28:0/2708890002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bb41df350 con 0x7f5bb406ed50 2024-02-21T23:10:01.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.036+0000 7f5bb0c00640 1 -- 172.21.15.28:0/2708890002 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5ba8046070 con 0x7f5bb406ed50 2024-02-21T23:10:01.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.036+0000 7f5bbebec740 1 Processor -- start 2024-02-21T23:10:01.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.036+0000 7f5bbebec740 1 -- start start 2024-02-21T23:10:01.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.036+0000 7f5bb0c00640 1 --2- 172.21.15.28:0/2708890002 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5b9406b4d0 0x7f5b9406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:01.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.037+0000 7f5bb0c00640 1 -- 172.21.15.28:0/2708890002 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5ba80ac3a0 con 0x7f5bb406ed50 2024-02-21T23:10:01.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.037+0000 7f5bb9c00640 1 --2- 172.21.15.28:0/2708890002 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5b9406b4d0 0x7f5b9406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:01.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.037+0000 7f5bbebec740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5565dbef0e40 0x5565dbef1220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:01.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.037+0000 7f5bbebec740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565dbef2150 0x5565dbf2cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:01.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.037+0000 7f5bbebec740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5565dbf2d4d0 0x5565dbf2b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:01.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.037+0000 7f5bbebec740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565dbe55470 con 0x5565dbef2150 2024-02-21T23:10:01.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.037+0000 7f5bbebec740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565dbdfeaf0 con 0x5565dbef0e40 2024-02-21T23:10:01.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.037+0000 7f5bbebec740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x5565dbe51510 con 0x5565dbf2d4d0 2024-02-21T23:10:01.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.038+0000 7f5bb3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565dbef2150 0x5565dbf2cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:01.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.038+0000 7f5bb3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565dbef2150 0x5565dbf2cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33852/0 (socket says 172.21.15.28:33852) 2024-02-21T23:10:01.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.038+0000 7f5bb3e00640 1 -- 172.21.15.28:0/2471690655 learned_addr learned my addr 172.21.15.28:0/2471690655 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:01.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.038+0000 7f5bb9200640 1 --2- 172.21.15.28:0/2471690655 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5565dbef0e40 0x5565dbef1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:01.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.038+0000 7f5bb9c00640 1 --2- 172.21.15.28:0/2471690655 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5565dbf2d4d0 0x5565dbf2b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:01.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.038+0000 7f5bb3e00640 1 -- 172.21.15.28:0/2471690655 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5565dbf2d4d0 msgr2=0x5565dbf2b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:01.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.038+0000 7f5bb3e00640 1 --2- 172.21.15.28:0/2471690655 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5565dbf2d4d0 0x5565dbf2b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:01.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.038+0000 7f5bb3e00640 1 -- 172.21.15.28:0/2471690655 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5565dbef0e40 msgr2=0x5565dbef1220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:01.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.038+0000 7f5bb3e00640 1 --2- 172.21.15.28:0/2471690655 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5565dbef0e40 0x5565dbef1220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:01.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.038+0000 7f5bb3e00640 1 -- 172.21.15.28:0/2471690655 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5565dbe05d90 con 0x5565dbef2150 2024-02-21T23:10:01.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.038+0000 7f5bb9c00640 1 --2- 172.21.15.28:0/2471690655 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5565dbf2d4d0 0x5565dbf2b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:10:01.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.039+0000 7f5bb3e00640 1 --2- 172.21.15.28:0/2471690655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565dbef2150 0x5565dbf2cf90 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7f5ba000a8c0 tx=0x7f5ba000ad80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4f76af66b44dc2a server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:01.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.039+0000 7f5bb9c00640 1 --2- 172.21.15.28:0/2708890002 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5b9406b4d0 0x7f5b9406d990 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f5bac00abc0 tx=0x7f5bac00c3b0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:01.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.039+0000 7f5ba5600640 1 -- 172.21.15.28:0/2471690655 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ba0015070 con 0x5565dbef2150 2024-02-21T23:10:01.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.039+0000 7f5ba5600640 1 -- 172.21.15.28:0/2471690655 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ba000cb20 con 0x5565dbef2150 2024-02-21T23:10:01.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.039+0000 7f5ba5600640 1 -- 172.21.15.28:0/2471690655 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ba00113f0 con 0x5565dbef2150 2024-02-21T23:10:01.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.039+0000 7f5bbebec740 1 -- 172.21.15.28:0/2471690655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565dbef2150 msgr2=0x5565dbf2cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:01.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.039+0000 7f5bbebec740 1 --2- 172.21.15.28:0/2471690655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565dbef2150 0x5565dbf2cf90 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7f5ba000a8c0 tx=0x7f5ba000ad80 comp rx=0 tx=0).stop 2024-02-21T23:10:01.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.040+0000 7f5bbebec740 1 -- 172.21.15.28:0/2471690655 shutdown_connections 2024-02-21T23:10:01.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.040+0000 7f5bbebec740 1 --2- 172.21.15.28:0/2471690655 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5565dbf2d4d0 0x5565dbf2b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:01.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.040+0000 7f5bbebec740 1 --2- 172.21.15.28:0/2471690655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565dbef2150 0x5565dbf2cf90 unknown :-1 s=CLOSED pgs=1700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:01.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.040+0000 7f5bbebec740 1 --2- 172.21.15.28:0/2471690655 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5565dbef0e40 0x5565dbef1220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:01.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.040+0000 7f5bbebec740 1 -- 172.21.15.28:0/2471690655 >> 172.21.15.28:0/2471690655 conn(0x5565dbeed790 msgr2=0x5565dbf121a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:01.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.040+0000 7f5bbebec740 1 -- 172.21.15.28:0/2471690655 shutdown_connections 2024-02-21T23:10:01.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.040+0000 7f5bbebec740 1 -- 172.21.15.28:0/2471690655 wait complete. 2024-02-21T23:10:01.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.040+0000 7f5bbebec740 1 -- start start 2024-02-21T23:10:01.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.041+0000 7f5bbebec740 1 Processor -- start 2024-02-21T23:10:01.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.041+0000 7f5bbebec740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5565dbef2150 0x5565dbf32150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:01.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.042+0000 7f5bbebec740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565dbf2d4d0 0x5565dbf3d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:01.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.042+0000 7f5bbebec740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5565dbf275f0 0x5565dbf0cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:01.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.042+0000 7f5bbebec740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565dbdf4040 con 0x5565dbf2d4d0 2024-02-21T23:10:01.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.042+0000 7f5bbebec740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565dbf128b0 con 0x5565dbf275f0 2024-02-21T23:10:01.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.042+0000 7f5bbebec740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x5565dbf33090 con 0x5565dbef2150 2024-02-21T23:10:01.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.042+0000 7f5bbebec740 10 client.0 _ll_register_callbacks cb 0x5565dbe11920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:10:01.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.042+0000 7f5bb3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565dbf2d4d0 0x5565dbf3d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:01.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.042+0000 7f5bb3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565dbf2d4d0 0x5565dbf3d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33866/0 (socket says 172.21.15.28:33866) 2024-02-21T23:10:01.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.042+0000 7f5bb3e00640 1 -- 172.21.15.28:0/963125939 learned_addr learned my addr 172.21.15.28:0/963125939 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:01.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.042+0000 7f5bb9200640 1 --2- 172.21.15.28:0/963125939 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5565dbef2150 0x5565dbf32150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:01.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.042+0000 7f5bb9c00640 1 --2- 172.21.15.28:0/963125939 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5565dbf275f0 0x5565dbf0cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:01.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.042+0000 7f5bb3e00640 1 -- 172.21.15.28:0/963125939 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5565dbef2150 msgr2=0x5565dbf32150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:01.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.042+0000 7f5bb3e00640 1 --2- 172.21.15.28:0/963125939 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5565dbef2150 0x5565dbf32150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:01.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.042+0000 7f5bb3e00640 1 -- 172.21.15.28:0/963125939 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5565dbf275f0 msgr2=0x5565dbf0cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:01.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.042+0000 7f5bb3e00640 1 --2- 172.21.15.28:0/963125939 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5565dbf275f0 0x5565dbf0cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:01.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.042+0000 7f5bb3e00640 1 -- 172.21.15.28:0/963125939 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5565dbe13aa0 con 0x5565dbf2d4d0 2024-02-21T23:10:01.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.042+0000 7f5bb9c00640 1 --2- 172.21.15.28:0/963125939 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5565dbf275f0 0x5565dbf0cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:10:01.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.042+0000 7f5bb9200640 1 --2- 172.21.15.28:0/963125939 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5565dbef2150 0x5565dbf32150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:10:01.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.043+0000 7f5bb3e00640 1 --2- 172.21.15.28:0/963125939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565dbf2d4d0 0x5565dbf3d8b0 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7f5ba001b690 tx=0x7f5ba0004640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e7f087b52121452 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:01.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.043+0000 7f5b9a000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:10:01.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.043+0000 7f5b9a000640 1 -- 172.21.15.28:0/963125939 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ba002b060 con 0x5565dbf2d4d0 2024-02-21T23:10:01.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.043+0000 7f5bbebec740 1 -- 172.21.15.28:0/963125939 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5ba000cb20 con 0x5565dbf2d4d0 2024-02-21T23:10:01.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.043+0000 7f5b9a000640 1 -- 172.21.15.28:0/963125939 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(151..151 src has 1..151) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5ba000c8e0 con 0x5565dbf2d4d0 2024-02-21T23:10:01.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.043+0000 7f5b9a000640 1 -- 172.21.15.28:0/963125939 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=158) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5ba0031050 con 0x5565dbf2d4d0 2024-02-21T23:10:01.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.043+0000 7f5bbebec740 10 client.8958 fetch_fsmap learned FSMap version 158 2024-02-21T23:10:01.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.043+0000 7f5bbebec740 1 -- 172.21.15.28:0/963125939 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=158,monmap=2+}) v3 -- 0x5565dbb6a2d0 con 0x5565dbf2d4d0 2024-02-21T23:10:01.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.044+0000 7f5b9a000640 1 -- 172.21.15.28:0/963125939 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ba000cb20 con 0x5565dbf2d4d0 2024-02-21T23:10:01.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.044+0000 7f5b9a000640 1 -- 172.21.15.28:0/963125939 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 158) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f5ba0034020 con 0x5565dbf2d4d0 2024-02-21T23:10:01.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.044+0000 7f5bbebec740 10 client.8958 fetch_fsmap finished waiting for FSMap version 158 2024-02-21T23:10:01.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.044+0000 7f5bbebec740 10 client.8958 resolve_mds: resolved name 'a' to daemon mds.a 2024-02-21T23:10:01.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.044+0000 7f5bbebec740 20 client.8958 populate_metadata read hostname 'smithi028' 2024-02-21T23:10:01.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.044+0000 7f5bbebec740 1 --2- 172.21.15.28:0/963125939 >> [v2:172.21.15.28:6834/2928639298,v1:172.21.15.28:6835/2928639298] conn(0x5565dbf40fa0 0x5565dbf41380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:01.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.044+0000 7f5bbebec740 4 client.8958 mds_command: new command op to 8826 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-21T23:10:01.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.044+0000 7f5bbebec740 1 -- 172.21.15.28:0/963125939 --> [v2:172.21.15.28:6834/2928639298,v1:172.21.15.28:6835/2928639298] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5565dbe629b0 con 0x5565dbf40fa0 2024-02-21T23:10:01.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.044+0000 7f5bb9200640 1 --2- 172.21.15.28:0/963125939 >> [v2:172.21.15.28:6834/2928639298,v1:172.21.15.28:6835/2928639298] conn(0x5565dbf40fa0 0x5565dbf41380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:01.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.045+0000 7f5bb9200640 1 --2- 172.21.15.28:0/963125939 >> [v2:172.21.15.28:6834/2928639298,v1:172.21.15.28:6835/2928639298] conn(0x5565dbf40fa0 0x5565dbf41380 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=885f4d92304079a1 server_cookie=54cb27acf8085c95 in_seq=0 out_seq=0 2024-02-21T23:10:01.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.045+0000 7f5b9a000640 10 client.8958 ms_handle_connect on v2:172.21.15.28:6834/2928639298 2024-02-21T23:10:01.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.046+0000 7f5b9a000640 1 -- 172.21.15.28:0/963125939 <== mds.0 v2:172.21.15.28:6834/2928639298 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5565dbe629b0 con 0x5565dbf40fa0 2024-02-21T23:10:01.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.046+0000 7f5b9a000640 10 client.8958 handle_command_reply: tid=0 2024-02-21T23:10:01.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.046+0000 7f5bbebec740 1 client.8958 shutdown 2024-02-21T23:10:01.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.047+0000 7f5bbebec740 1 -- 172.21.15.28:0/963125939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565dbf2d4d0 msgr2=0x5565dbf3d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:01.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.047+0000 7f5bbebec740 1 --2- 172.21.15.28:0/963125939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565dbf2d4d0 0x5565dbf3d8b0 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7f5ba001b690 tx=0x7f5ba0004640 comp rx=0 tx=0).stop 2024-02-21T23:10:01.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.047+0000 7f5bbebec740 1 -- 172.21.15.28:0/963125939 shutdown_connections 2024-02-21T23:10:01.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.047+0000 7f5bbebec740 1 --2- 172.21.15.28:0/963125939 >> [v2:172.21.15.28:6834/2928639298,v1:172.21.15.28:6835/2928639298] conn(0x5565dbf40fa0 0x5565dbf41380 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:01.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.047+0000 7f5bbebec740 1 --2- 172.21.15.28:0/963125939 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5565dbf275f0 0x5565dbf0cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:01.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.047+0000 7f5bbebec740 1 --2- 172.21.15.28:0/963125939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565dbf2d4d0 0x5565dbf3d8b0 unknown :-1 s=CLOSED pgs=1701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:01.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.047+0000 7f5bbebec740 1 --2- 172.21.15.28:0/963125939 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5565dbef2150 0x5565dbf32150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:01.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.047+0000 7f5bbebec740 1 -- 172.21.15.28:0/963125939 >> 172.21.15.28:0/963125939 conn(0x5565dbef0e40 msgr2=0x5565dbeea5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:01.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.047+0000 7f5b9a000640 0 client.8958 ms_handle_reset on v2:172.21.15.28:6834/2928639298 2024-02-21T23:10:01.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.047+0000 7f5bbebec740 1 -- 172.21.15.28:0/963125939 shutdown_connections 2024-02-21T23:10:01.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.047+0000 7f5bbebec740 1 -- 172.21.15.28:0/963125939 wait complete. 2024-02-21T23:10:01.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.047+0000 7f5bbebec740 20 client.8958 trim_cache size 0 max 16384 2024-02-21T23:10:01.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.066+0000 7f5bbebec740 1 Processor -- start 2024-02-21T23:10:01.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.066+0000 7f5bbebec740 1 -- start start 2024-02-21T23:10:01.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.066+0000 7f5bbebec740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5565dbef2150 0x5565dbf0dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:01.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.066+0000 7f5bbebec740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5565dbf275f0 0x5565dbf4ff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:01.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.067+0000 7f5bbebec740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565dbf2d4d0 0x5565dbf504c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:01.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.067+0000 7f5bbebec740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb41f7160 con 0x5565dbf2d4d0 2024-02-21T23:10:01.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.067+0000 7f5bbebec740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb41554e0 con 0x5565dbef2150 2024-02-21T23:10:01.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.067+0000 7f5bbebec740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb41f6b90 con 0x5565dbf275f0 2024-02-21T23:10:01.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.067+0000 7f5bb9c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565dbf2d4d0 0x5565dbf504c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:01.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.067+0000 7f5bb9c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565dbf2d4d0 0x5565dbf504c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33874/0 (socket says 172.21.15.28:33874) 2024-02-21T23:10:01.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.067+0000 7f5bb9c00640 1 -- 172.21.15.28:0/302430930 learned_addr learned my addr 172.21.15.28:0/302430930 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:01.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.067+0000 7f5bb9200640 1 --2- 172.21.15.28:0/302430930 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5565dbef2150 0x5565dbf0dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:01.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.067+0000 7f5bb3e00640 1 --2- 172.21.15.28:0/302430930 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5565dbf275f0 0x5565dbf4ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:01.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.067+0000 7f5bb9c00640 1 -- 172.21.15.28:0/302430930 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5565dbf275f0 msgr2=0x5565dbf4ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:01.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.067+0000 7f5bb9c00640 1 --2- 172.21.15.28:0/302430930 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5565dbf275f0 0x5565dbf4ff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:01.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.067+0000 7f5bb9c00640 1 -- 172.21.15.28:0/302430930 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5565dbef2150 msgr2=0x5565dbf0dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:01.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.067+0000 7f5bb9c00640 1 --2- 172.21.15.28:0/302430930 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5565dbef2150 0x5565dbf0dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:01.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.067+0000 7f5bb9c00640 1 -- 172.21.15.28:0/302430930 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5565dbef1590 con 0x5565dbf2d4d0 2024-02-21T23:10:01.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.068+0000 7f5bb9c00640 1 --2- 172.21.15.28:0/302430930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565dbf2d4d0 0x5565dbf504c0 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7f5bac008310 tx=0x7f5bac017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e921045948ce73f server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:01.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.068+0000 7f5ba4c00640 1 -- 172.21.15.28:0/302430930 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bac01f070 con 0x5565dbf2d4d0 2024-02-21T23:10:01.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.068+0000 7f5ba4c00640 1 -- 172.21.15.28:0/302430930 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bac00de30 con 0x5565dbf2d4d0 2024-02-21T23:10:01.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.068+0000 7f5ba4c00640 1 -- 172.21.15.28:0/302430930 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bac016050 con 0x5565dbf2d4d0 2024-02-21T23:10:01.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.069+0000 7f5bbebec740 1 -- 172.21.15.28:0/302430930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565dbf2d4d0 msgr2=0x5565dbf504c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:01.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.069+0000 7f5bbebec740 1 --2- 172.21.15.28:0/302430930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565dbf2d4d0 0x5565dbf504c0 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7f5bac008310 tx=0x7f5bac017000 comp rx=0 tx=0).stop 2024-02-21T23:10:01.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.069+0000 7f5bbebec740 1 -- 172.21.15.28:0/302430930 shutdown_connections 2024-02-21T23:10:01.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.069+0000 7f5bbebec740 1 --2- 172.21.15.28:0/302430930 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5565dbf275f0 0x5565dbf4ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:01.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.069+0000 7f5bbebec740 1 --2- 172.21.15.28:0/302430930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565dbf2d4d0 0x5565dbf504c0 unknown :-1 s=CLOSED pgs=1702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:01.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.069+0000 7f5bbebec740 1 --2- 172.21.15.28:0/302430930 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5565dbef2150 0x5565dbf0dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:01.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.069+0000 7f5bbebec740 1 -- 172.21.15.28:0/302430930 >> 172.21.15.28:0/302430930 conn(0x5565dbef0e40 msgr2=0x5565dbf31f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:01.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.069+0000 7f5bbebec740 1 -- 172.21.15.28:0/302430930 shutdown_connections 2024-02-21T23:10:01.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.069+0000 7f5bbebec740 1 -- 172.21.15.28:0/302430930 wait complete. 2024-02-21T23:10:01.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.070+0000 7f5bbebec740 1 -- start start 2024-02-21T23:10:01.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.070+0000 7f5bbebec740 1 Processor -- start 2024-02-21T23:10:01.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.072+0000 7f5bbebec740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5565dbef2150 0x5565dbf2bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:01.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.072+0000 7f5bbebec740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5565dbf275f0 0x5565dbf9f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:01.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.072+0000 7f5bbebec740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565dbf2d4d0 0x5565dbf99020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:01.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.072+0000 7f5bbebec740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565dbf28950 con 0x5565dbf2d4d0 2024-02-21T23:10:01.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.072+0000 7f5bbebec740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565dbf11e90 con 0x5565dbef2150 2024-02-21T23:10:01.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.072+0000 7f5bbebec740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x5565dbf0d470 con 0x5565dbf275f0 2024-02-21T23:10:01.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.072+0000 7f5bbebec740 10 client.0 _ll_register_callbacks cb 0x7f5ba000a490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:10:01.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.072+0000 7f5bb9c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565dbf2d4d0 0x5565dbf99020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:01.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.072+0000 7f5bb9c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565dbf2d4d0 0x5565dbf99020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33882/0 (socket says 172.21.15.28:33882) 2024-02-21T23:10:01.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.072+0000 7f5bb9c00640 1 -- 172.21.15.28:0/3277848032 learned_addr learned my addr 172.21.15.28:0/3277848032 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:01.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.073+0000 7f5bb9200640 1 --2- 172.21.15.28:0/3277848032 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5565dbef2150 0x5565dbf2bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:01.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.073+0000 7f5bb3e00640 1 --2- 172.21.15.28:0/3277848032 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5565dbf275f0 0x5565dbf9f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:01.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.073+0000 7f5bb9c00640 1 -- 172.21.15.28:0/3277848032 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5565dbf275f0 msgr2=0x5565dbf9f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:01.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.073+0000 7f5bb9c00640 1 --2- 172.21.15.28:0/3277848032 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5565dbf275f0 0x5565dbf9f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:01.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.073+0000 7f5bb9c00640 1 -- 172.21.15.28:0/3277848032 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5565dbef2150 msgr2=0x5565dbf2bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:01.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.073+0000 7f5bb9c00640 1 --2- 172.21.15.28:0/3277848032 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5565dbef2150 0x5565dbf2bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:01.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.073+0000 7f5bb9c00640 1 -- 172.21.15.28:0/3277848032 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5565dbc346b0 con 0x5565dbf2d4d0 2024-02-21T23:10:01.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.073+0000 7f5bb3e00640 1 --2- 172.21.15.28:0/3277848032 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5565dbf275f0 0x5565dbf9f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:10:01.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.073+0000 7f5bb9c00640 1 --2- 172.21.15.28:0/3277848032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565dbf2d4d0 0x5565dbf99020 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7f5bac00c8b0 tx=0x7f5bac004b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=625156b98ae5ab37 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:01.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.073+0000 7f5b9a000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:10:01.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.074+0000 7f5b9a000640 1 -- 172.21.15.28:0/3277848032 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bac02a070 con 0x5565dbf2d4d0 2024-02-21T23:10:01.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.074+0000 7f5b9a000640 1 -- 172.21.15.28:0/3277848032 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(151..151 src has 1..151) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5bac024870 con 0x5565dbf2d4d0 2024-02-21T23:10:01.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.074+0000 7f5bbebec740 1 -- 172.21.15.28:0/3277848032 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5bac00de30 con 0x5565dbf2d4d0 2024-02-21T23:10:01.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.074+0000 7f5b9a000640 1 -- 172.21.15.28:0/3277848032 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=158) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5bac01f040 con 0x5565dbf2d4d0 2024-02-21T23:10:01.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.074+0000 7f5bbebec740 10 client.8964 fetch_fsmap learned FSMap version 158 2024-02-21T23:10:01.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.074+0000 7f5bbebec740 1 -- 172.21.15.28:0/3277848032 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=158,monmap=2+}) v3 -- 0x5565dbeea810 con 0x5565dbf2d4d0 2024-02-21T23:10:01.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.074+0000 7f5b9a000640 1 -- 172.21.15.28:0/3277848032 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bac00de30 con 0x5565dbf2d4d0 2024-02-21T23:10:01.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.074+0000 7f5b9a000640 1 -- 172.21.15.28:0/3277848032 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 158) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f5bac03c080 con 0x5565dbf2d4d0 2024-02-21T23:10:01.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.074+0000 7f5bbebec740 10 client.8964 fetch_fsmap finished waiting for FSMap version 158 2024-02-21T23:10:01.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.074+0000 7f5bbebec740 10 client.8964 resolve_mds: resolved name 'a' to daemon mds.a 2024-02-21T23:10:01.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.074+0000 7f5bbebec740 20 client.8964 populate_metadata read hostname 'smithi028' 2024-02-21T23:10:01.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.074+0000 7f5bbebec740 1 --2- 172.21.15.28:0/3277848032 >> [v2:172.21.15.28:6834/2928639298,v1:172.21.15.28:6835/2928639298] conn(0x5565dbf0e870 0x5565dbf0ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:01.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.074+0000 7f5bbebec740 4 client.8964 mds_command: new command op to 8826 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-21T23:10:01.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.074+0000 7f5bbebec740 1 -- 172.21.15.28:0/3277848032 --> [v2:172.21.15.28:6834/2928639298,v1:172.21.15.28:6835/2928639298] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5565dbe5cd00 con 0x5565dbf0e870 2024-02-21T23:10:01.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.075+0000 7f5bb9200640 1 --2- 172.21.15.28:0/3277848032 >> [v2:172.21.15.28:6834/2928639298,v1:172.21.15.28:6835/2928639298] conn(0x5565dbf0e870 0x5565dbf0ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:01.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.075+0000 7f5bb9200640 1 --2- 172.21.15.28:0/3277848032 >> [v2:172.21.15.28:6834/2928639298,v1:172.21.15.28:6835/2928639298] conn(0x5565dbf0e870 0x5565dbf0ec50 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=79205d9e1344ae29 server_cookie=1d09420d217a5fd7 in_seq=0 out_seq=0 2024-02-21T23:10:01.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.075+0000 7f5b9a000640 10 client.8964 ms_handle_connect on v2:172.21.15.28:6834/2928639298 2024-02-21T23:10:01.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.076+0000 7f5b9a000640 1 -- 172.21.15.28:0/3277848032 <== mds.0 v2:172.21.15.28:6834/2928639298 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+321 (crc 0 0 0) 0x5565dbe5cd00 con 0x5565dbf0e870 2024-02-21T23:10:01.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.076+0000 7f5b9a000640 10 client.8964 handle_command_reply: tid=0 2024-02-21T23:10:01.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.076+0000 7f5bbebec740 1 client.8964 shutdown 2024-02-21T23:10:01.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.076+0000 7f5bbebec740 1 -- 172.21.15.28:0/3277848032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565dbf2d4d0 msgr2=0x5565dbf99020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:01.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.076+0000 7f5bbebec740 1 --2- 172.21.15.28:0/3277848032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565dbf2d4d0 0x5565dbf99020 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7f5bac00c8b0 tx=0x7f5bac004b20 comp rx=0 tx=0).stop 2024-02-21T23:10:01.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.076+0000 7f5bbebec740 1 -- 172.21.15.28:0/3277848032 shutdown_connections 2024-02-21T23:10:01.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.076+0000 7f5bbebec740 1 --2- 172.21.15.28:0/3277848032 >> [v2:172.21.15.28:6834/2928639298,v1:172.21.15.28:6835/2928639298] conn(0x5565dbf0e870 0x5565dbf0ec50 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:01.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.076+0000 7f5bbebec740 1 --2- 172.21.15.28:0/3277848032 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5565dbf275f0 0x5565dbf9f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:01.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.076+0000 7f5bbebec740 1 --2- 172.21.15.28:0/3277848032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565dbf2d4d0 0x5565dbf99020 unknown :-1 s=CLOSED pgs=1703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:01.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.076+0000 7f5b9a000640 0 client.8964 ms_handle_reset on v2:172.21.15.28:6834/2928639298 2024-02-21T23:10:01.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.076+0000 7f5bbebec740 1 --2- 172.21.15.28:0/3277848032 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5565dbef2150 0x5565dbf2bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:01.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.076+0000 7f5bbebec740 1 -- 172.21.15.28:0/3277848032 >> 172.21.15.28:0/3277848032 conn(0x5565dbef0e40 msgr2=0x5565dbef0530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:01.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.076+0000 7f5bbebec740 1 -- 172.21.15.28:0/3277848032 shutdown_connections 2024-02-21T23:10:01.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.076+0000 7f5bbebec740 1 -- 172.21.15.28:0/3277848032 wait complete. 2024-02-21T23:10:01.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.076+0000 7f5bbebec740 20 client.8964 trim_cache size 0 max 16384 2024-02-21T23:10:01.068 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-21T23:10:01.068 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac", 2024-02-21T23:10:01.068 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-02-21T23:10:01.068 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 8826, 2024-02-21T23:10:01.068 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-02-21T23:10:01.068 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-02-21T23:10:01.068 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-02-21T23:10:01.068 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 6.812098432, 2024-02-21T23:10:01.068 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 158, 2024-02-21T23:10:01.068 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 151, 2024-02-21T23:10:01.068 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 151, 2024-02-21T23:10:01.068 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 12.276724026 2024-02-21T23:10:01.068 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-21T23:10:01.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.077+0000 7f5bbbc00640 1 -- 172.21.15.28:0/2708890002 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5b9406b4d0 msgr2=0x7f5b9406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:01.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.077+0000 7f5bbbc00640 1 --2- 172.21.15.28:0/2708890002 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5b9406b4d0 0x7f5b9406d990 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f5bac00abc0 tx=0x7f5bac00c3b0 comp rx=0 tx=0).stop 2024-02-21T23:10:01.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.077+0000 7f5bbbc00640 1 -- 172.21.15.28:0/2708890002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bb406ed50 msgr2=0x7f5bb41dee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:01.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.077+0000 7f5bbbc00640 1 --2- 172.21.15.28:0/2708890002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bb406ed50 0x7f5bb41dee10 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7f5ba8000c00 tx=0x7f5ba80046c0 comp rx=0 tx=0).stop 2024-02-21T23:10:01.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.077+0000 7f5bbbc00640 1 -- 172.21.15.28:0/2708890002 shutdown_connections 2024-02-21T23:10:01.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.077+0000 7f5bbbc00640 1 --2- 172.21.15.28:0/2708890002 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5b9406b4d0 0x7f5b9406d990 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:01.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.077+0000 7f5bbbc00640 1 --2- 172.21.15.28:0/2708890002 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5bb40c37c0 0x7f5bb41e0e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:01.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.077+0000 7f5bbbc00640 1 --2- 172.21.15.28:0/2708890002 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5bb40c1c00 0x7f5bb41eb5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:01.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.077+0000 7f5bbbc00640 1 --2- 172.21.15.28:0/2708890002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5bb406ed50 0x7f5bb41dee10 unknown :-1 s=CLOSED pgs=1699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:01.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.077+0000 7f5bbbc00640 1 -- 172.21.15.28:0/2708890002 >> 172.21.15.28:0/2708890002 conn(0x7f5bb406b820 msgr2=0x7f5bb40aa450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:01.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.077+0000 7f5bbbc00640 1 -- 172.21.15.28:0/2708890002 shutdown_connections 2024-02-21T23:10:01.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.077+0000 7f5bbbc00640 1 -- 172.21.15.28:0/2708890002 wait complete. 2024-02-21T23:10:01.083 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:10:01.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.306+0000 7f6db1a00640 1 Processor -- start 2024-02-21T23:10:01.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.306+0000 7f6db1a00640 1 -- start start 2024-02-21T23:10:01.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.308+0000 7f6db1a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f6dac06f330 0x7f6dac0c7290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:01.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.308+0000 7f6db1a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6dac0c7be0 0x7f6dac14c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:01.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.308+0000 7f6db1a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dac0c5020 con 0x7f6dac070d40 2024-02-21T23:10:01.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.308+0000 7f6db1a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6dac153a10 con 0x7f6dac06f330 2024-02-21T23:10:01.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.308+0000 7f6db1a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dac153cd0 con 0x7f6dac0c7be0 2024-02-21T23:10:01.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.309+0000 7f6daaa00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f6dac070d40 0x7f6dac0c4910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34358/0 (socket says 172.21.15.28:34358) 2024-02-21T23:10:01.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.309+0000 7f6daaa00640 1 -- 172.21.15.28:0/2634145689 learned_addr learned my addr 172.21.15.28:0/2634145689 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:10:01.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.309+0000 7f6daa000640 1 --2- 172.21.15.28:0/2634145689 >> v2:172.21.15.81:3300/0 conn(0x7f6dac06f330 0x7f6dac0c7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:01.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.309+0000 7f6dab400640 1 --2- 172.21.15.28:0/2634145689 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6dac0c7be0 0x7f6dac14c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:01.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.310+0000 7f6daa000640 1 -- 172.21.15.28:0/2634145689 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6dac0c7be0 msgr2=0x7f6dac14c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:01.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.310+0000 7f6daa000640 1 --2- 172.21.15.28:0/2634145689 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6dac0c7be0 0x7f6dac14c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:01.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.310+0000 7f6daa000640 1 -- 172.21.15.28:0/2634145689 >> v1:172.21.15.28:6789/0 conn(0x7f6dac070d40 legacy=0x7f6dac0c4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:01.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.310+0000 7f6daa000640 1 -- 172.21.15.28:0/2634145689 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dac0c67b0 con 0x7f6dac06f330 2024-02-21T23:10:01.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.312+0000 7f6daa000640 1 --2- 172.21.15.28:0/2634145689 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6dac06f330 0x7f6dac0c7290 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f6d980097a0 tx=0x7f6d9802b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c7592a1cc0561eb5 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:01.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.315+0000 7f6da9600640 1 -- 172.21.15.28:0/2634145689 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d98035020 con 0x7f6dac06f330 2024-02-21T23:10:01.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.315+0000 7f6da9600640 1 -- 172.21.15.28:0/2634145689 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d98031810 con 0x7f6dac06f330 2024-02-21T23:10:01.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.315+0000 7f6da9600640 1 -- 172.21.15.28:0/2634145689 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d98031be0 con 0x7f6dac06f330 2024-02-21T23:10:01.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.315+0000 7f6db1a00640 1 -- 172.21.15.28:0/2634145689 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6dac06f330 msgr2=0x7f6dac0c7290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:01.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.315+0000 7f6db1a00640 1 --2- 172.21.15.28:0/2634145689 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6dac06f330 0x7f6dac0c7290 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f6d980097a0 tx=0x7f6d9802b8b0 comp rx=0 tx=0).stop 2024-02-21T23:10:01.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.315+0000 7f6db1a00640 1 -- 172.21.15.28:0/2634145689 shutdown_connections 2024-02-21T23:10:01.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.315+0000 7f6db1a00640 1 --2- 172.21.15.28:0/2634145689 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6dac0c7be0 0x7f6dac14c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:01.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.315+0000 7f6db1a00640 1 --2- 172.21.15.28:0/2634145689 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6dac06f330 0x7f6dac0c7290 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:01.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.315+0000 7f6db1a00640 1 -- 172.21.15.28:0/2634145689 >> 172.21.15.28:0/2634145689 conn(0x7f6dac0c8810 msgr2=0x7f6dac0c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:01.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.316+0000 7f6db1a00640 1 -- 172.21.15.28:0/2634145689 shutdown_connections 2024-02-21T23:10:01.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.316+0000 7f6db1a00640 1 -- 172.21.15.28:0/2634145689 wait complete. 2024-02-21T23:10:01.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.318+0000 7f6db1a00640 1 Processor -- start 2024-02-21T23:10:01.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.318+0000 7f6db1a00640 1 -- start start 2024-02-21T23:10:01.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.319+0000 7f6db1a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6dac06f330 0x7f6dac147ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:01.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.319+0000 7f6db1a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dac070d40 0x7f6dac147ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:01.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.320+0000 7f6db1a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6dac0c7be0 0x7f6dac1ee8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:01.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.320+0000 7f6db1a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dac071550 con 0x7f6dac070d40 2024-02-21T23:10:01.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.320+0000 7f6db1a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dac0c6d70 con 0x7f6dac06f330 2024-02-21T23:10:01.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.320+0000 7f6db1a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dac071120 con 0x7f6dac0c7be0 2024-02-21T23:10:01.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.320+0000 7f6daa000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dac070d40 0x7f6dac147ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:01.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.320+0000 7f6daa000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dac070d40 0x7f6dac147ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33894/0 (socket says 172.21.15.28:33894) 2024-02-21T23:10:01.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.320+0000 7f6daa000640 1 -- 172.21.15.28:0/1156712270 learned_addr learned my addr 172.21.15.28:0/1156712270 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:01.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.320+0000 7f6dab400640 1 --2- 172.21.15.28:0/1156712270 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6dac0c7be0 0x7f6dac1ee8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:01.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.320+0000 7f6daaa00640 1 --2- 172.21.15.28:0/1156712270 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6dac06f330 0x7f6dac147ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:01.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.320+0000 7f6daa000640 1 -- 172.21.15.28:0/1156712270 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6dac0c7be0 msgr2=0x7f6dac1ee8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:01.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.320+0000 7f6daa000640 1 --2- 172.21.15.28:0/1156712270 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6dac0c7be0 0x7f6dac1ee8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:01.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.320+0000 7f6daa000640 1 -- 172.21.15.28:0/1156712270 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6dac06f330 msgr2=0x7f6dac147ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:01.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.320+0000 7f6daa000640 1 --2- 172.21.15.28:0/1156712270 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6dac06f330 0x7f6dac147ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:01.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.320+0000 7f6daa000640 1 -- 172.21.15.28:0/1156712270 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dac152d60 con 0x7f6dac070d40 2024-02-21T23:10:01.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.320+0000 7f6daaa00640 1 --2- 172.21.15.28:0/1156712270 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6dac06f330 0x7f6dac147ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:10:01.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.321+0000 7f6daa000640 1 --2- 172.21.15.28:0/1156712270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dac070d40 0x7f6dac147ff0 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f6d98009890 tx=0x7f6d980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:01.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.321+0000 7f6da2a00640 1 -- 172.21.15.28:0/1156712270 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d98035040 con 0x7f6dac070d40 2024-02-21T23:10:01.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.321+0000 7f6da2a00640 1 -- 172.21.15.28:0/1156712270 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d9803ad30 con 0x7f6dac070d40 2024-02-21T23:10:01.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.321+0000 7f6da2a00640 1 -- 172.21.15.28:0/1156712270 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d980466a0 con 0x7f6dac070d40 2024-02-21T23:10:01.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.321+0000 7f6db1a00640 1 -- 172.21.15.28:0/1156712270 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dac0ac070 con 0x7f6dac070d40 2024-02-21T23:10:01.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.321+0000 7f6db1a00640 1 -- 172.21.15.28:0/1156712270 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dac0ac630 con 0x7f6dac070d40 2024-02-21T23:10:01.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.322+0000 7f6db1a00640 1 -- 172.21.15.28:0/1156712270 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dac068640 con 0x7f6dac070d40 2024-02-21T23:10:01.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.323+0000 7f6da2a00640 1 -- 172.21.15.28:0/1156712270 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6d98038020 con 0x7f6dac070d40 2024-02-21T23:10:01.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.324+0000 7f6da2a00640 1 --2- 172.21.15.28:0/1156712270 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6d8c06b4d0 0x7f6d8c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:01.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.325+0000 7f6da2a00640 1 -- 172.21.15.28:0/1156712270 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6d980ac490 con 0x7f6dac070d40 2024-02-21T23:10:01.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.325+0000 7f6daaa00640 1 --2- 172.21.15.28:0/1156712270 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6d8c06b4d0 0x7f6d8c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:01.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.326+0000 7f6daaa00640 1 --2- 172.21.15.28:0/1156712270 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6d8c06b4d0 0x7f6d8c06d990 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f6dac1ee5d0 tx=0x7f6d9c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:01.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.329+0000 7f6da2a00640 1 -- 172.21.15.28:0/1156712270 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f6d9807b210 con 0x7f6dac070d40 2024-02-21T23:10:01.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.531+0000 7f6db1a00640 1 -- 172.21.15.28:0/1156712270 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6dac0a7d40 con 0x7f6dac070d40 2024-02-21T23:10:01.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.534+0000 7f6da2a00640 1 -- 172.21.15.28:0/1156712270 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v151) v1 ==== 74+0+16824 (secure 0 0 0) 0x7f6d9807ab10 con 0x7f6dac070d40 2024-02-21T23:10:01.523 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:10:01.523 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":151,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:09:55.258904+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:09:51.634291+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"150","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-02-21T23:09:52.736252+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"151","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":149,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:10:01.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.537+0000 7f6db1a00640 1 -- 172.21.15.28:0/1156712270 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6d8c06b4d0 msgr2=0x7f6d8c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:01.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.537+0000 7f6db1a00640 1 --2- 172.21.15.28:0/1156712270 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6d8c06b4d0 0x7f6d8c06d990 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f6dac1ee5d0 tx=0x7f6d9c009040 comp rx=0 tx=0).stop 2024-02-21T23:10:01.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.537+0000 7f6db1a00640 1 -- 172.21.15.28:0/1156712270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dac070d40 msgr2=0x7f6dac147ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:01.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.537+0000 7f6db1a00640 1 --2- 172.21.15.28:0/1156712270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dac070d40 0x7f6dac147ff0 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f6d98009890 tx=0x7f6d980046c0 comp rx=0 tx=0).stop 2024-02-21T23:10:01.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.538+0000 7f6db1a00640 1 -- 172.21.15.28:0/1156712270 shutdown_connections 2024-02-21T23:10:01.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.538+0000 7f6db1a00640 1 --2- 172.21.15.28:0/1156712270 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6d8c06b4d0 0x7f6d8c06d990 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:01.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.538+0000 7f6db1a00640 1 --2- 172.21.15.28:0/1156712270 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6dac0c7be0 0x7f6dac1ee8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:01.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.538+0000 7f6db1a00640 1 --2- 172.21.15.28:0/1156712270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dac070d40 0x7f6dac147ff0 unknown :-1 s=CLOSED pgs=1705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:01.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.538+0000 7f6db1a00640 1 --2- 172.21.15.28:0/1156712270 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6dac06f330 0x7f6dac147ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:01.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.538+0000 7f6db1a00640 1 -- 172.21.15.28:0/1156712270 >> 172.21.15.28:0/1156712270 conn(0x7f6dac0c8810 msgr2=0x7f6dac0a8530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:01.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.538+0000 7f6db1a00640 1 -- 172.21.15.28:0/1156712270 shutdown_connections 2024-02-21T23:10:01.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.538+0000 7f6db1a00640 1 -- 172.21.15.28:0/1156712270 wait complete. 2024-02-21T23:10:01.553 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-21T23:02:30.728329+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 23, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-21T23:09:51.634291+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '150', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 24, 'pool_name': 'cephfs_data', 'create_time': '2024-02-21T23:09:52.736252+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '151', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-21T23:10:01.554 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-21T23:10:01.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.781+0000 7fd1b5e00640 1 Processor -- start 2024-02-21T23:10:01.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.781+0000 7fd1b5e00640 1 -- start start 2024-02-21T23:10:01.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.783+0000 7fd1b5e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fd1b013e0c0 0x7fd1b00c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:01.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.783+0000 7fd1b5e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd1b006dbc0 0x7fd1b006c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:01.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.783+0000 7fd1b5e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd1b00c76f0 0x7fd1b006c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:01.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.783+0000 7fd1b5e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1b00c89a0 con 0x7fd1b006dbc0 2024-02-21T23:10:01.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.783+0000 7fd1b5e00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1b0158550 con 0x7fd1b013e0c0 2024-02-21T23:10:01.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.783+0000 7fd1b5e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b01587d0 con 0x7fd1b00c76f0 2024-02-21T23:10:01.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.783+0000 7fd1afe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd1b006dbc0 0x7fd1b006c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:01.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.783+0000 7fd1afe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd1b006dbc0 0x7fd1b006c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33896/0 (socket says 172.21.15.28:33896) 2024-02-21T23:10:01.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.783+0000 7fd1afe00640 1 -- 172.21.15.28:0/1085837497 learned_addr learned my addr 172.21.15.28:0/1085837497 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:01.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.784+0000 7fd1b4a00640 1 --2- 172.21.15.28:0/1085837497 >> v2:172.21.15.81:3300/0 conn(0x7fd1b013e0c0 0x7fd1b00c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:01.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.784+0000 7fd1afe00640 1 -- 172.21.15.28:0/1085837497 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd1b00c76f0 msgr2=0x7fd1b006c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:01.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.784+0000 7fd1b5400640 1 --2- 172.21.15.28:0/1085837497 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd1b00c76f0 0x7fd1b006c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:01.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.784+0000 7fd1afe00640 1 --2- 172.21.15.28:0/1085837497 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd1b00c76f0 0x7fd1b006c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:01.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.784+0000 7fd1afe00640 1 -- 172.21.15.28:0/1085837497 >> v2:172.21.15.81:3300/0 conn(0x7fd1b013e0c0 msgr2=0x7fd1b00c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:01.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.784+0000 7fd1afe00640 1 --2- 172.21.15.28:0/1085837497 >> v2:172.21.15.81:3300/0 conn(0x7fd1b013e0c0 0x7fd1b00c71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:01.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.784+0000 7fd1afe00640 1 -- 172.21.15.28:0/1085837497 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1b0158ab0 con 0x7fd1b006dbc0 2024-02-21T23:10:01.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.784+0000 7fd1b5400640 1 --2- 172.21.15.28:0/1085837497 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd1b00c76f0 0x7fd1b006c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:10:01.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.785+0000 7fd1afe00640 1 --2- 172.21.15.28:0/1085837497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1b006dbc0 0x7fd1b006c280 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7fd1b00ad180 tx=0x7fd19c02b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c6ac82edb4f4ffb server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:01.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.785+0000 7fd1af400640 1 -- 172.21.15.28:0/1085837497 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd19c0317d0 con 0x7fd1b006dbc0 2024-02-21T23:10:01.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.785+0000 7fd1af400640 1 -- 172.21.15.28:0/1085837497 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd19c037070 con 0x7fd1b006dbc0 2024-02-21T23:10:01.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.785+0000 7fd1af400640 1 -- 172.21.15.28:0/1085837497 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd19c031b90 con 0x7fd1b006dbc0 2024-02-21T23:10:01.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.786+0000 7fd1b5e00640 1 -- 172.21.15.28:0/1085837497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1b006dbc0 msgr2=0x7fd1b006c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:01.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.786+0000 7fd1b5e00640 1 --2- 172.21.15.28:0/1085837497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1b006dbc0 0x7fd1b006c280 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7fd1b00ad180 tx=0x7fd19c02b870 comp rx=0 tx=0).stop 2024-02-21T23:10:01.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.786+0000 7fd1b5e00640 1 -- 172.21.15.28:0/1085837497 shutdown_connections 2024-02-21T23:10:01.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.786+0000 7fd1b5e00640 1 --2- 172.21.15.28:0/1085837497 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd1b00c76f0 0x7fd1b006c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:01.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.786+0000 7fd1b5e00640 1 --2- 172.21.15.28:0/1085837497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1b006dbc0 0x7fd1b006c280 unknown :-1 s=CLOSED pgs=1706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:01.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.786+0000 7fd1b5e00640 1 --2- 172.21.15.28:0/1085837497 >> v2:172.21.15.81:3300/0 conn(0x7fd1b013e0c0 0x7fd1b00c71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:01.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.786+0000 7fd1b5e00640 1 -- 172.21.15.28:0/1085837497 >> 172.21.15.28:0/1085837497 conn(0x7fd1b0073110 msgr2=0x7fd1b00ac490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:01.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.786+0000 7fd1b5e00640 1 -- 172.21.15.28:0/1085837497 shutdown_connections 2024-02-21T23:10:01.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.786+0000 7fd1b5e00640 1 -- 172.21.15.28:0/1085837497 wait complete. 2024-02-21T23:10:01.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.788+0000 7fd1b5e00640 1 Processor -- start 2024-02-21T23:10:01.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.788+0000 7fd1b5e00640 1 -- start start 2024-02-21T23:10:01.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.789+0000 7fd1b5e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1b006dbc0 0x7fd1b01ee420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:01.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.789+0000 7fd1b5e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1b00c76f0 0x7fd1b01e1b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:01.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.789+0000 7fd1b5e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd1b013e0c0 0x7fd1b01e2040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:01.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.789+0000 7fd1b5e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b006cd00 con 0x7fd1b00c76f0 2024-02-21T23:10:01.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.789+0000 7fd1b5e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b0159070 con 0x7fd1b006dbc0 2024-02-21T23:10:01.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.789+0000 7fd1b5e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b006ce80 con 0x7fd1b013e0c0 2024-02-21T23:10:01.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.790+0000 7fd1afe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1b00c76f0 0x7fd1b01e1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:01.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.790+0000 7fd1afe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1b00c76f0 0x7fd1b01e1b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33910/0 (socket says 172.21.15.28:33910) 2024-02-21T23:10:01.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.790+0000 7fd1afe00640 1 -- 172.21.15.28:0/940712928 learned_addr learned my addr 172.21.15.28:0/940712928 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:01.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.790+0000 7fd1b5400640 1 --2- 172.21.15.28:0/940712928 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd1b013e0c0 0x7fd1b01e2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:01.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.790+0000 7fd1b4a00640 1 --2- 172.21.15.28:0/940712928 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1b006dbc0 0x7fd1b01ee420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:01.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.790+0000 7fd1afe00640 1 -- 172.21.15.28:0/940712928 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd1b013e0c0 msgr2=0x7fd1b01e2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:01.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.790+0000 7fd1afe00640 1 --2- 172.21.15.28:0/940712928 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd1b013e0c0 0x7fd1b01e2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:01.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.790+0000 7fd1afe00640 1 -- 172.21.15.28:0/940712928 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1b006dbc0 msgr2=0x7fd1b01ee420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:01.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.791+0000 7fd1afe00640 1 --2- 172.21.15.28:0/940712928 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1b006dbc0 0x7fd1b01ee420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:01.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.791+0000 7fd1afe00640 1 -- 172.21.15.28:0/940712928 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1b0152dd0 con 0x7fd1b00c76f0 2024-02-21T23:10:01.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.791+0000 7fd1b4a00640 1 --2- 172.21.15.28:0/940712928 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1b006dbc0 0x7fd1b01ee420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:10:01.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.791+0000 7fd1afe00640 1 --2- 172.21.15.28:0/940712928 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1b00c76f0 0x7fd1b01e1b00 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7fd19c000c00 tx=0x7fd19c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:01.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.792+0000 7fd1acc00640 1 -- 172.21.15.28:0/940712928 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd19c032660 con 0x7fd1b00c76f0 2024-02-21T23:10:01.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.792+0000 7fd1acc00640 1 -- 172.21.15.28:0/940712928 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd19c032840 con 0x7fd1b00c76f0 2024-02-21T23:10:01.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.792+0000 7fd1acc00640 1 -- 172.21.15.28:0/940712928 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd19c032c30 con 0x7fd1b00c76f0 2024-02-21T23:10:01.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.792+0000 7fd1b5e00640 1 -- 172.21.15.28:0/940712928 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1b01ef040 con 0x7fd1b00c76f0 2024-02-21T23:10:01.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.792+0000 7fd1b5e00640 1 -- 172.21.15.28:0/940712928 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1b00af7e0 con 0x7fd1b00c76f0 2024-02-21T23:10:01.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.793+0000 7fd1b5e00640 1 -- 172.21.15.28:0/940712928 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1b004f290 con 0x7fd1b00c76f0 2024-02-21T23:10:01.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.794+0000 7fd1acc00640 1 -- 172.21.15.28:0/940712928 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd19c0453f0 con 0x7fd1b00c76f0 2024-02-21T23:10:01.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.795+0000 7fd1acc00640 1 --2- 172.21.15.28:0/940712928 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd19006b4d0 0x7fd19006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:01.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.795+0000 7fd1acc00640 1 -- 172.21.15.28:0/940712928 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd19c0ac4c0 con 0x7fd1b00c76f0 2024-02-21T23:10:01.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.796+0000 7fd1b4a00640 1 --2- 172.21.15.28:0/940712928 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd19006b4d0 0x7fd19006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:01.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.797+0000 7fd1b4a00640 1 --2- 172.21.15.28:0/940712928 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd19006b4d0 0x7fd19006d990 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fd1a4006fd0 tx=0x7fd1a4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:01.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:01.799+0000 7fd1acc00640 1 -- 172.21.15.28:0/940712928 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd19c040070 con 0x7fd1b00c76f0 2024-02-21T23:10:01.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.004+0000 7fd1b5e00640 1 -- 172.21.15.28:0/940712928 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd1b006bec0 con 0x7fd1b00c76f0 2024-02-21T23:10:01.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.005+0000 7fd1acc00640 1 -- 172.21.15.28:0/940712928 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v151) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd19c07b110 con 0x7fd1b00c76f0 2024-02-21T23:10:01.993 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-02-21T23:10:01.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.008+0000 7fd1b5e00640 1 -- 172.21.15.28:0/940712928 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd19006b4d0 msgr2=0x7fd19006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:01.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.008+0000 7fd1b5e00640 1 --2- 172.21.15.28:0/940712928 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd19006b4d0 0x7fd19006d990 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fd1a4006fd0 tx=0x7fd1a4008040 comp rx=0 tx=0).stop 2024-02-21T23:10:01.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.008+0000 7fd1b5e00640 1 -- 172.21.15.28:0/940712928 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1b00c76f0 msgr2=0x7fd1b01e1b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:01.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.008+0000 7fd1b5e00640 1 --2- 172.21.15.28:0/940712928 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1b00c76f0 0x7fd1b01e1b00 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7fd19c000c00 tx=0x7fd19c0046c0 comp rx=0 tx=0).stop 2024-02-21T23:10:01.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.008+0000 7fd1b5e00640 1 -- 172.21.15.28:0/940712928 shutdown_connections 2024-02-21T23:10:01.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.008+0000 7fd1b5e00640 1 --2- 172.21.15.28:0/940712928 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd19006b4d0 0x7fd19006d990 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:01.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.008+0000 7fd1b5e00640 1 --2- 172.21.15.28:0/940712928 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd1b013e0c0 0x7fd1b01e2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:01.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.008+0000 7fd1b5e00640 1 --2- 172.21.15.28:0/940712928 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1b00c76f0 0x7fd1b01e1b00 unknown :-1 s=CLOSED pgs=1707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:01.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.008+0000 7fd1b5e00640 1 --2- 172.21.15.28:0/940712928 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd1b006dbc0 0x7fd1b01ee420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:01.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.008+0000 7fd1b5e00640 1 -- 172.21.15.28:0/940712928 >> 172.21.15.28:0/940712928 conn(0x7fd1b0073110 msgr2=0x7fd1b0138460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:01.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.008+0000 7fd1b5e00640 1 -- 172.21.15.28:0/940712928 shutdown_connections 2024-02-21T23:10:01.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.008+0000 7fd1b5e00640 1 -- 172.21.15.28:0/940712928 wait complete. 2024-02-21T23:10:02.016 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-21T23:10:02.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.232+0000 7f0f87200640 1 Processor -- start 2024-02-21T23:10:02.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.232+0000 7f0f87200640 1 -- start start 2024-02-21T23:10:02.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.235+0000 7f0f87200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0f8813e0c0 0x7f0f880c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:02.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.235+0000 7f0f87200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0f880c76f0 0x7f0f8806c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:02.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.235+0000 7f0f87200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f88158550 con 0x7f0f8806dbc0 2024-02-21T23:10:02.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.235+0000 7f0f87200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f880c89a0 con 0x7f0f880c76f0 2024-02-21T23:10:02.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.235+0000 7f0f87200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f881587d0 con 0x7f0f8813e0c0 2024-02-21T23:10:02.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.235+0000 7f0f86800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0f880c76f0 0x7f0f8806c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:02.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.235+0000 7f0f86800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0f880c76f0 0x7f0f8806c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33920/0 (socket says 172.21.15.28:33920) 2024-02-21T23:10:02.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.235+0000 7f0f86800640 1 -- 172.21.15.28:0/553958536 learned_addr learned my addr 172.21.15.28:0/553958536 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:02.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.235+0000 7f0f85e00640 1 --2- 172.21.15.28:0/553958536 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0f8813e0c0 0x7f0f880c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:02.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.236+0000 7f0f86800640 1 -- 172.21.15.28:0/553958536 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0f8813e0c0 msgr2=0x7f0f880c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:02.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.236+0000 7f0f86800640 1 --2- 172.21.15.28:0/553958536 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0f8813e0c0 0x7f0f880c71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:02.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.236+0000 7f0f86800640 1 -- 172.21.15.28:0/553958536 >> v1:172.21.15.81:6789/0 conn(0x7f0f8806dbc0 legacy=0x7f0f8806c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:02.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.236+0000 7f0f86800640 1 -- 172.21.15.28:0/553958536 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f88158ab0 con 0x7f0f880c76f0 2024-02-21T23:10:02.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.237+0000 7f0f86800640 1 --2- 172.21.15.28:0/553958536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f880c76f0 0x7f0f8806c990 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f0f80009870 tx=0x7f0f8002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b27e8631d74124bc server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:02.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.237+0000 7f0f84a00640 1 -- 172.21.15.28:0/553958536 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f800317c0 con 0x7f0f880c76f0 2024-02-21T23:10:02.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.237+0000 7f0f84a00640 1 -- 172.21.15.28:0/553958536 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f80037070 con 0x7f0f880c76f0 2024-02-21T23:10:02.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.237+0000 7f0f84a00640 1 -- 172.21.15.28:0/553958536 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f80031b80 con 0x7f0f880c76f0 2024-02-21T23:10:02.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.238+0000 7f0f87200640 1 -- 172.21.15.28:0/553958536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f880c76f0 msgr2=0x7f0f8806c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:02.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.238+0000 7f0f87200640 1 --2- 172.21.15.28:0/553958536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f880c76f0 0x7f0f8806c990 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f0f80009870 tx=0x7f0f8002b860 comp rx=0 tx=0).stop 2024-02-21T23:10:02.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.238+0000 7f0f87200640 1 -- 172.21.15.28:0/553958536 shutdown_connections 2024-02-21T23:10:02.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.238+0000 7f0f87200640 1 --2- 172.21.15.28:0/553958536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f880c76f0 0x7f0f8806c990 unknown :-1 s=CLOSED pgs=1708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:02.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.238+0000 7f0f87200640 1 --2- 172.21.15.28:0/553958536 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0f8813e0c0 0x7f0f880c71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:02.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.238+0000 7f0f87200640 1 -- 172.21.15.28:0/553958536 >> 172.21.15.28:0/553958536 conn(0x7f0f88073110 msgr2=0x7f0f880ac490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:02.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.238+0000 7f0f87200640 1 -- 172.21.15.28:0/553958536 shutdown_connections 2024-02-21T23:10:02.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.238+0000 7f0f87200640 1 -- 172.21.15.28:0/553958536 wait complete. 2024-02-21T23:10:02.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.240+0000 7f0f87200640 1 Processor -- start 2024-02-21T23:10:02.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.240+0000 7f0f87200640 1 -- start start 2024-02-21T23:10:02.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.241+0000 7f0f87200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f8806dbc0 0x7f0f880adf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:02.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.241+0000 7f0f87200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f880c76f0 0x7f0f88136e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:02.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.241+0000 7f0f87200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0f8813e0c0 0x7f0f881373d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:02.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.241+0000 7f0f87200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f8806ced0 con 0x7f0f8806dbc0 2024-02-21T23:10:02.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.241+0000 7f0f87200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f88159070 con 0x7f0f880c76f0 2024-02-21T23:10:02.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.241+0000 7f0f87200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f8813e520 con 0x7f0f8813e0c0 2024-02-21T23:10:02.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.241+0000 7f0f85e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f8806dbc0 0x7f0f880adf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:02.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.241+0000 7f0f85e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f8806dbc0 0x7f0f880adf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33932/0 (socket says 172.21.15.28:33932) 2024-02-21T23:10:02.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.241+0000 7f0f85e00640 1 -- 172.21.15.28:0/3362440292 learned_addr learned my addr 172.21.15.28:0/3362440292 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:02.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.241+0000 7f0f86800640 1 --2- 172.21.15.28:0/3362440292 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0f8813e0c0 0x7f0f881373d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:02.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.241+0000 7f0f85400640 1 --2- 172.21.15.28:0/3362440292 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f880c76f0 0x7f0f88136e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:02.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.241+0000 7f0f85e00640 1 -- 172.21.15.28:0/3362440292 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0f8813e0c0 msgr2=0x7f0f881373d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:02.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.242+0000 7f0f85e00640 1 --2- 172.21.15.28:0/3362440292 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0f8813e0c0 0x7f0f881373d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:02.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.242+0000 7f0f85e00640 1 -- 172.21.15.28:0/3362440292 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f880c76f0 msgr2=0x7f0f88136e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:02.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.242+0000 7f0f85e00640 1 --2- 172.21.15.28:0/3362440292 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f880c76f0 0x7f0f88136e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:02.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.242+0000 7f0f85e00640 1 -- 172.21.15.28:0/3362440292 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f880afed0 con 0x7f0f8806dbc0 2024-02-21T23:10:02.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.242+0000 7f0f86800640 1 --2- 172.21.15.28:0/3362440292 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0f8813e0c0 0x7f0f881373d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:10:02.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.242+0000 7f0f85e00640 1 --2- 172.21.15.28:0/3362440292 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f8806dbc0 0x7f0f880adf90 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7f0f78007e70 tx=0x7f0f7800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:02.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.242+0000 7f0f7e000640 1 -- 172.21.15.28:0/3362440292 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f78014620 con 0x7f0f8806dbc0 2024-02-21T23:10:02.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.242+0000 7f0f7e000640 1 -- 172.21.15.28:0/3362440292 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f78019070 con 0x7f0f8806dbc0 2024-02-21T23:10:02.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.242+0000 7f0f7e000640 1 -- 172.21.15.28:0/3362440292 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f78014a00 con 0x7f0f8806dbc0 2024-02-21T23:10:02.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.242+0000 7f0f87200640 1 -- 172.21.15.28:0/3362440292 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f880b01b0 con 0x7f0f8806dbc0 2024-02-21T23:10:02.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.242+0000 7f0f87200640 1 -- 172.21.15.28:0/3362440292 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f880af580 con 0x7f0f8806dbc0 2024-02-21T23:10:02.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.244+0000 7f0f87200640 1 -- 172.21.15.28:0/3362440292 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f8804f290 con 0x7f0f8806dbc0 2024-02-21T23:10:02.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.245+0000 7f0f7e000640 1 -- 172.21.15.28:0/3362440292 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0f78014ba0 con 0x7f0f8806dbc0 2024-02-21T23:10:02.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.246+0000 7f0f7e000640 1 --2- 172.21.15.28:0/3362440292 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0f6406b500 0x7f0f6406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:02.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.246+0000 7f0f7e000640 1 -- 172.21.15.28:0/3362440292 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f0f7808ab40 con 0x7f0f8806dbc0 2024-02-21T23:10:02.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.246+0000 7f0f85400640 1 --2- 172.21.15.28:0/3362440292 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0f6406b500 0x7f0f6406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:02.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.247+0000 7f0f85400640 1 --2- 172.21.15.28:0/3362440292 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0f6406b500 0x7f0f6406d9c0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f0f880ac820 tx=0x7f0f70009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:02.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.255+0000 7f0f7e000640 1 -- 172.21.15.28:0/3362440292 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f0f780598c0 con 0x7f0f8806dbc0 2024-02-21T23:10:02.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.620+0000 7f0f87200640 1 -- 172.21.15.28:0/3362440292 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0f8806d9a0 con 0x7f0f8806dbc0 2024-02-21T23:10:02.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.621+0000 7f0f7e000640 1 -- 172.21.15.28:0/3362440292 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v151) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0f780591c0 con 0x7f0f8806dbc0 2024-02-21T23:10:02.609 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-02-21T23:10:02.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.624+0000 7f0f87200640 1 -- 172.21.15.28:0/3362440292 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0f6406b500 msgr2=0x7f0f6406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:02.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.624+0000 7f0f87200640 1 --2- 172.21.15.28:0/3362440292 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0f6406b500 0x7f0f6406d9c0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f0f880ac820 tx=0x7f0f70009040 comp rx=0 tx=0).stop 2024-02-21T23:10:02.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.624+0000 7f0f87200640 1 -- 172.21.15.28:0/3362440292 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f8806dbc0 msgr2=0x7f0f880adf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:02.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.624+0000 7f0f87200640 1 --2- 172.21.15.28:0/3362440292 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f8806dbc0 0x7f0f880adf90 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7f0f78007e70 tx=0x7f0f7800e4b0 comp rx=0 tx=0).stop 2024-02-21T23:10:02.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.624+0000 7f0f87200640 1 -- 172.21.15.28:0/3362440292 shutdown_connections 2024-02-21T23:10:02.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.624+0000 7f0f87200640 1 --2- 172.21.15.28:0/3362440292 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0f6406b500 0x7f0f6406d9c0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:02.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.624+0000 7f0f87200640 1 --2- 172.21.15.28:0/3362440292 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0f8813e0c0 0x7f0f881373d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:02.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.624+0000 7f0f87200640 1 --2- 172.21.15.28:0/3362440292 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f880c76f0 0x7f0f88136e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:02.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.624+0000 7f0f87200640 1 --2- 172.21.15.28:0/3362440292 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f8806dbc0 0x7f0f880adf90 unknown :-1 s=CLOSED pgs=1709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:02.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.624+0000 7f0f87200640 1 -- 172.21.15.28:0/3362440292 >> 172.21.15.28:0/3362440292 conn(0x7f0f88073110 msgr2=0x7f0f8813ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:02.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.624+0000 7f0f87200640 1 -- 172.21.15.28:0/3362440292 shutdown_connections 2024-02-21T23:10:02.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.624+0000 7f0f87200640 1 -- 172.21.15.28:0/3362440292 wait complete. 2024-02-21T23:10:02.632 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-21T23:10:02.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.862+0000 7f73ed800640 1 Processor -- start 2024-02-21T23:10:02.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.862+0000 7f73ed800640 1 -- start start 2024-02-21T23:10:02.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.864+0000 7f73ed800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f73e813e520 0x7f73e806c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:02.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.864+0000 7f73ed800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73e8158550 con 0x7f73e806dbc0 2024-02-21T23:10:02.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.864+0000 7f73ed800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73e81587d0 con 0x7f73e813e0c0 2024-02-21T23:10:02.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.864+0000 7f73ed800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73e80c89a0 con 0x7f73e813e520 2024-02-21T23:10:02.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.864+0000 7f73dfe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f73e806dbc0 0x7f73e806c280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34366/0 (socket says 172.21.15.28:34366) 2024-02-21T23:10:02.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.864+0000 7f73dfe00640 1 -- 172.21.15.28:0/2598519261 learned_addr learned my addr 172.21.15.28:0/2598519261 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:10:02.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.865+0000 7f73ece00640 1 --2- 172.21.15.28:0/2598519261 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f73e813e520 0x7f73e806c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:02.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.865+0000 7f73e7400640 1 -- 172.21.15.28:0/2598519261 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1630047975 0 0) 0x7f73d0001110 con 0x7f73e806dbc0 2024-02-21T23:10:02.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.865+0000 7f73e7400640 1 -- 172.21.15.28:0/2598519261 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73cc003520 con 0x7f73e806dbc0 2024-02-21T23:10:02.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.865+0000 7f73e7400640 1 -- 172.21.15.28:0/2598519261 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2867555319 0 0) 0x7f73cc003520 con 0x7f73e806dbc0 2024-02-21T23:10:02.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.865+0000 7f73e7400640 1 -- 172.21.15.28:0/2598519261 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f73e813e520 msgr2=0x7f73e806c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:02.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.865+0000 7f73e7400640 1 --2- 172.21.15.28:0/2598519261 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f73e813e520 0x7f73e806c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:02.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.865+0000 7f73e7400640 1 -- 172.21.15.28:0/2598519261 >> v1:172.21.15.81:6789/0 conn(0x7f73e813e0c0 legacy=0x7f73e80c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:02.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.866+0000 7f73e7400640 1 -- 172.21.15.28:0/2598519261 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73e8158ab0 con 0x7f73e806dbc0 2024-02-21T23:10:02.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.866+0000 7f73e7400640 1 -- 172.21.15.28:0/2598519261 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f73d0001840 con 0x7f73e806dbc0 2024-02-21T23:10:02.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.866+0000 7f73e7400640 1 -- 172.21.15.28:0/2598519261 >> v1:172.21.15.28:6789/0 conn(0x7f73e806dbc0 legacy=0x7f73e806c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:02.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.866+0000 7f73e7400640 1 --2- 172.21.15.28:0/2598519261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f73cc0039e0 0x7f73cc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:02.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.866+0000 7f73e7400640 1 --2- 172.21.15.28:0/2598519261 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73cc004800 0x7f73cc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:02.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.866+0000 7f73e7400640 1 --2- 172.21.15.28:0/2598519261 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f73e813e520 0x7f73cc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:02.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.866+0000 7f73e7400640 1 -- 172.21.15.28:0/2598519261 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73cc00ba90 con 0x7f73cc0039e0 2024-02-21T23:10:02.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.866+0000 7f73e7400640 1 -- 172.21.15.28:0/2598519261 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73cc00bc40 con 0x7f73cc004800 2024-02-21T23:10:02.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.866+0000 7f73e7400640 1 -- 172.21.15.28:0/2598519261 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73cc00be40 con 0x7f73e813e520 2024-02-21T23:10:02.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.866+0000 7f73e7e00640 1 --2- 172.21.15.28:0/2598519261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f73cc0039e0 0x7f73cc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:02.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.866+0000 7f73ece00640 1 --2- 172.21.15.28:0/2598519261 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f73e813e520 0x7f73cc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:02.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.866+0000 7f73dfe00640 1 --2- 172.21.15.28:0/2598519261 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73cc004800 0x7f73cc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:02.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.867+0000 7f73e7e00640 1 -- 172.21.15.28:0/2598519261 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f73e813e520 msgr2=0x7f73cc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:02.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.867+0000 7f73e7e00640 1 --2- 172.21.15.28:0/2598519261 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f73e813e520 0x7f73cc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:02.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.867+0000 7f73e7e00640 1 -- 172.21.15.28:0/2598519261 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73cc004800 msgr2=0x7f73cc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:02.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.867+0000 7f73e7e00640 1 --2- 172.21.15.28:0/2598519261 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73cc004800 0x7f73cc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:02.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.867+0000 7f73e7e00640 1 -- 172.21.15.28:0/2598519261 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73cc003520 con 0x7f73cc0039e0 2024-02-21T23:10:02.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.868+0000 7f73e7e00640 1 --2- 172.21.15.28:0/2598519261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f73cc0039e0 0x7f73cc003dc0 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f73d8001860 tx=0x7f73cc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f63c106de9e2bf0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:02.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.868+0000 7f73e7400640 1 -- 172.21.15.28:0/2598519261 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d802f050 con 0x7f73cc0039e0 2024-02-21T23:10:02.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.868+0000 7f73e7400640 1 -- 172.21.15.28:0/2598519261 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73d8033650 con 0x7f73cc0039e0 2024-02-21T23:10:02.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.868+0000 7f73e7400640 1 -- 172.21.15.28:0/2598519261 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d8001430 con 0x7f73cc0039e0 2024-02-21T23:10:02.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.869+0000 7f73ed800640 1 -- 172.21.15.28:0/2598519261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f73cc0039e0 msgr2=0x7f73cc003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:02.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.869+0000 7f73ed800640 1 --2- 172.21.15.28:0/2598519261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f73cc0039e0 0x7f73cc003dc0 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f73d8001860 tx=0x7f73cc005830 comp rx=0 tx=0).stop 2024-02-21T23:10:02.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.869+0000 7f73ece00640 1 -- 172.21.15.28:0/2598519261 reap_dead start 2024-02-21T23:10:02.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.869+0000 7f73ed800640 1 -- 172.21.15.28:0/2598519261 shutdown_connections 2024-02-21T23:10:02.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.869+0000 7f73ed800640 1 -- 172.21.15.28:0/2598519261 >> 172.21.15.28:0/2598519261 conn(0x7f73e8073110 msgr2=0x7f73e80ac490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:02.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.869+0000 7f73ed800640 1 -- 172.21.15.28:0/2598519261 shutdown_connections 2024-02-21T23:10:02.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.869+0000 7f73ed800640 1 -- 172.21.15.28:0/2598519261 wait complete. 2024-02-21T23:10:02.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.872+0000 7f73ed800640 1 Processor -- start 2024-02-21T23:10:02.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.872+0000 7f73ed800640 1 -- start start 2024-02-21T23:10:02.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.874+0000 7f73ed800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73cc0039e0 0x7f73e81ee210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:02.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.874+0000 7f73ed800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f73e81386d0 0x7f73e80afb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:02.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.874+0000 7f73ed800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f73e81ee750 0x7f73e80b00c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:02.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.874+0000 7f73ed800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e80c78c0 con 0x7f73e81ee750 2024-02-21T23:10:02.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.874+0000 7f73ed800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e8159070 con 0x7f73cc0039e0 2024-02-21T23:10:02.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.874+0000 7f73ed800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73e80c7a40 con 0x7f73e81386d0 2024-02-21T23:10:02.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.874+0000 7f73ece00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f73e81ee750 0x7f73e80b00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:02.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.874+0000 7f73ece00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f73e81ee750 0x7f73e80b00c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33956/0 (socket says 172.21.15.28:33956) 2024-02-21T23:10:02.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.874+0000 7f73ece00640 1 -- 172.21.15.28:0/4089394507 learned_addr learned my addr 172.21.15.28:0/4089394507 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:02.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.874+0000 7f73dfe00640 1 --2- 172.21.15.28:0/4089394507 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f73e81386d0 0x7f73e80afb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:02.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.874+0000 7f73e7e00640 1 --2- 172.21.15.28:0/4089394507 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73cc0039e0 0x7f73e81ee210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:02.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.874+0000 7f73ece00640 1 -- 172.21.15.28:0/4089394507 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f73e81386d0 msgr2=0x7f73e80afb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:02.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.874+0000 7f73ece00640 1 --2- 172.21.15.28:0/4089394507 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f73e81386d0 0x7f73e80afb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:02.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.874+0000 7f73ece00640 1 -- 172.21.15.28:0/4089394507 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73cc0039e0 msgr2=0x7f73e81ee210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:02.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.874+0000 7f73ece00640 1 --2- 172.21.15.28:0/4089394507 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73cc0039e0 0x7f73e81ee210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:02.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.874+0000 7f73ece00640 1 -- 172.21.15.28:0/4089394507 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73e80ab410 con 0x7f73e81ee750 2024-02-21T23:10:02.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.874+0000 7f73e7e00640 1 --2- 172.21.15.28:0/4089394507 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73cc0039e0 0x7f73e81ee210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:10:02.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.875+0000 7f73ece00640 1 --2- 172.21.15.28:0/4089394507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f73e81ee750 0x7f73e80b00c0 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7f73e80ca6e0 tx=0x7f73e00115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:02.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.875+0000 7f73e4c00640 1 -- 172.21.15.28:0/4089394507 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73e0003990 con 0x7f73e81ee750 2024-02-21T23:10:02.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.875+0000 7f73e4c00640 1 -- 172.21.15.28:0/4089394507 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73e001b070 con 0x7f73e81ee750 2024-02-21T23:10:02.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.875+0000 7f73e4c00640 1 -- 172.21.15.28:0/4089394507 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73e0003b70 con 0x7f73e81ee750 2024-02-21T23:10:02.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.875+0000 7f73ed800640 1 -- 172.21.15.28:0/4089394507 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73e80ae060 con 0x7f73e81ee750 2024-02-21T23:10:02.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.875+0000 7f73ed800640 1 -- 172.21.15.28:0/4089394507 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73e813a650 con 0x7f73e81ee750 2024-02-21T23:10:02.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.877+0000 7f73ed800640 1 -- 172.21.15.28:0/4089394507 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73e80ae260 con 0x7f73e81ee750 2024-02-21T23:10:02.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.878+0000 7f73e4c00640 1 -- 172.21.15.28:0/4089394507 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f73e0005d10 con 0x7f73e81ee750 2024-02-21T23:10:02.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.878+0000 7f73e4c00640 1 --2- 172.21.15.28:0/4089394507 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f73c406b500 0x7f73c406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:02.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.879+0000 7f73e4c00640 1 -- 172.21.15.28:0/4089394507 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f73e008a710 con 0x7f73e81ee750 2024-02-21T23:10:02.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.880+0000 7f73e7e00640 1 --2- 172.21.15.28:0/4089394507 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f73c406b500 0x7f73c406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:02.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.881+0000 7f73e7e00640 1 --2- 172.21.15.28:0/4089394507 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f73c406b500 0x7f73c406d9c0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f73d8036000 tx=0x7f73d8031000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:02.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:02.883+0000 7f73e4c00640 1 -- 172.21.15.28:0/4089394507 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f73e80ae260 con 0x7f73e81ee750 2024-02-21T23:10:03.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.090+0000 7f73ed800640 1 -- 172.21.15.28:0/4089394507 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f73e804f290 con 0x7f73e81ee750 2024-02-21T23:10:03.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.091+0000 7f73e4c00640 1 -- 172.21.15.28:0/4089394507 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v151) v1 ==== 100+0+11 (secure 0 0 0) 0x7f73e0014350 con 0x7f73e81ee750 2024-02-21T23:10:03.079 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-02-21T23:10:03.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.094+0000 7f73ed800640 1 -- 172.21.15.28:0/4089394507 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f73c406b500 msgr2=0x7f73c406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:03.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.094+0000 7f73ed800640 1 --2- 172.21.15.28:0/4089394507 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f73c406b500 0x7f73c406d9c0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f73d8036000 tx=0x7f73d8031000 comp rx=0 tx=0).stop 2024-02-21T23:10:03.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.094+0000 7f73ed800640 1 -- 172.21.15.28:0/4089394507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f73e81ee750 msgr2=0x7f73e80b00c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:03.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.094+0000 7f73ed800640 1 --2- 172.21.15.28:0/4089394507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f73e81ee750 0x7f73e80b00c0 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7f73e80ca6e0 tx=0x7f73e00115a0 comp rx=0 tx=0).stop 2024-02-21T23:10:03.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.094+0000 7f73ed800640 1 -- 172.21.15.28:0/4089394507 shutdown_connections 2024-02-21T23:10:03.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.094+0000 7f73ed800640 1 --2- 172.21.15.28:0/4089394507 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f73c406b500 0x7f73c406d9c0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:03.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.094+0000 7f73ed800640 1 --2- 172.21.15.28:0/4089394507 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f73e81386d0 0x7f73e80afb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:03.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.094+0000 7f73ed800640 1 --2- 172.21.15.28:0/4089394507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f73e81ee750 0x7f73e80b00c0 unknown :-1 s=CLOSED pgs=1712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:03.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.094+0000 7f73ed800640 1 --2- 172.21.15.28:0/4089394507 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73cc0039e0 0x7f73e81ee210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:03.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.094+0000 7f73ed800640 1 -- 172.21.15.28:0/4089394507 >> 172.21.15.28:0/4089394507 conn(0x7f73e8073110 msgr2=0x7f73e813b320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:03.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.094+0000 7f73ed800640 1 -- 172.21.15.28:0/4089394507 shutdown_connections 2024-02-21T23:10:03.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.094+0000 7f73ed800640 1 -- 172.21.15.28:0/4089394507 wait complete. 2024-02-21T23:10:03.109 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-02-21T23:10:03.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.347+0000 7f92de400640 1 Processor -- start 2024-02-21T23:10:03.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.347+0000 7f92de400640 1 -- start start 2024-02-21T23:10:03.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.350+0000 7f92de400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f92d8158c20 0x7f92d8160890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:03.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.350+0000 7f92de400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92d81550b0 con 0x7f92d80c8740 2024-02-21T23:10:03.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.350+0000 7f92de400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92d8155350 con 0x7f92d80b0130 2024-02-21T23:10:03.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.350+0000 7f92de400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92d8152a40 con 0x7f92d8158c20 2024-02-21T23:10:03.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.350+0000 7f92cfe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f92d80c8740 0x7f92d8158510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34378/0 (socket says 172.21.15.28:34378) 2024-02-21T23:10:03.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.350+0000 7f92cfe00640 1 -- 172.21.15.28:0/3392211297 learned_addr learned my addr 172.21.15.28:0/3392211297 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:10:03.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.350+0000 7f92dda00640 1 --2- 172.21.15.28:0/3392211297 >> v2:172.21.15.81:3300/0 conn(0x7f92d8158c20 0x7f92d8160890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:03.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.351+0000 7f92d7e00640 1 -- 172.21.15.28:0/3392211297 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 731004247 0 0) 0x7f92d81550b0 con 0x7f92d80c8740 2024-02-21T23:10:03.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.351+0000 7f92d7e00640 1 -- 172.21.15.28:0/3392211297 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92b8003520 con 0x7f92d80c8740 2024-02-21T23:10:03.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.351+0000 7f92d7e00640 1 -- 172.21.15.28:0/3392211297 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 803801882 0 0) 0x7f92b8003520 con 0x7f92d80c8740 2024-02-21T23:10:03.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.351+0000 7f92d7e00640 1 -- 172.21.15.28:0/3392211297 >> v2:172.21.15.81:3300/0 conn(0x7f92d8158c20 msgr2=0x7f92d8160890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:03.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.351+0000 7f92d7e00640 1 --2- 172.21.15.28:0/3392211297 >> v2:172.21.15.81:3300/0 conn(0x7f92d8158c20 0x7f92d8160890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:03.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.351+0000 7f92d7e00640 1 -- 172.21.15.28:0/3392211297 >> v1:172.21.15.81:6789/0 conn(0x7f92d80b0130 legacy=0x7f92d80c51c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:03.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.351+0000 7f92d7e00640 1 -- 172.21.15.28:0/3392211297 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92d8155630 con 0x7f92d80c8740 2024-02-21T23:10:03.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.351+0000 7f92d7e00640 1 -- 172.21.15.28:0/3392211297 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f92c0001690 con 0x7f92d80c8740 2024-02-21T23:10:03.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.351+0000 7f92d7e00640 1 -- 172.21.15.28:0/3392211297 >> v1:172.21.15.28:6789/0 conn(0x7f92d80c8740 legacy=0x7f92d8158510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:03.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.351+0000 7f92d7e00640 1 --2- 172.21.15.28:0/3392211297 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f92b8003a00 0x7f92b8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:03.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.351+0000 7f92d7e00640 1 --2- 172.21.15.28:0/3392211297 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f92b8004800 0x7f92b8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:03.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.351+0000 7f92d7e00640 1 --2- 172.21.15.28:0/3392211297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92b8005210 0x7f92b800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:03.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.351+0000 7f92d7e00640 1 -- 172.21.15.28:0/3392211297 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92d8152a40 con 0x7f92b8005210 2024-02-21T23:10:03.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.351+0000 7f92d7e00640 1 -- 172.21.15.28:0/3392211297 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92b800c160 con 0x7f92b8003a00 2024-02-21T23:10:03.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.351+0000 7f92d7e00640 1 -- 172.21.15.28:0/3392211297 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92b800c360 con 0x7f92b8004800 2024-02-21T23:10:03.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.352+0000 7f92dda00640 1 --2- 172.21.15.28:0/3392211297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92b8005210 0x7f92b800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:03.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.352+0000 7f92dd000640 1 --2- 172.21.15.28:0/3392211297 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f92b8003a00 0x7f92b8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:03.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.352+0000 7f92cfe00640 1 --2- 172.21.15.28:0/3392211297 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f92b8004800 0x7f92b8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:03.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.352+0000 7f92dda00640 1 -- 172.21.15.28:0/3392211297 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f92b8004800 msgr2=0x7f92b8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:03.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.352+0000 7f92dda00640 1 --2- 172.21.15.28:0/3392211297 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f92b8004800 0x7f92b8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:03.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.352+0000 7f92dda00640 1 -- 172.21.15.28:0/3392211297 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f92b8003a00 msgr2=0x7f92b8003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:03.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.352+0000 7f92dda00640 1 --2- 172.21.15.28:0/3392211297 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f92b8003a00 0x7f92b8003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:03.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.352+0000 7f92dda00640 1 -- 172.21.15.28:0/3392211297 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92b8003520 con 0x7f92b8005210 2024-02-21T23:10:03.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.352+0000 7f92cfe00640 1 --2- 172.21.15.28:0/3392211297 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f92b8004800 0x7f92b8004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:10:03.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.353+0000 7f92dda00640 1 -- 172.21.15.28:0/3392211297 reap_dead start 2024-02-21T23:10:03.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.353+0000 7f92dda00640 1 --2- 172.21.15.28:0/3392211297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92b8005210 0x7f92b800b9c0 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f92d0009f90 tx=0x7f92b8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97c17d3fe1085bb5 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:03.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.354+0000 7f92d7e00640 1 -- 172.21.15.28:0/3392211297 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92d0009c10 con 0x7f92b8005210 2024-02-21T23:10:03.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.354+0000 7f92d7e00640 1 -- 172.21.15.28:0/3392211297 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92d002cbf0 con 0x7f92b8005210 2024-02-21T23:10:03.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.354+0000 7f92d7e00640 1 -- 172.21.15.28:0/3392211297 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92d0003990 con 0x7f92b8005210 2024-02-21T23:10:03.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.354+0000 7f92de400640 1 -- 172.21.15.28:0/3392211297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92b8005210 msgr2=0x7f92b800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:03.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.354+0000 7f92de400640 1 --2- 172.21.15.28:0/3392211297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92b8005210 0x7f92b800b9c0 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f92d0009f90 tx=0x7f92b8005830 comp rx=0 tx=0).stop 2024-02-21T23:10:03.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.355+0000 7f92de400640 1 -- 172.21.15.28:0/3392211297 shutdown_connections 2024-02-21T23:10:03.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.355+0000 7f92de400640 1 --2- 172.21.15.28:0/3392211297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92b8005210 0x7f92b800b9c0 unknown :-1 s=CLOSED pgs=1714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:03.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.355+0000 7f92de400640 1 -- 172.21.15.28:0/3392211297 >> 172.21.15.28:0/3392211297 conn(0x7f92d80ac370 msgr2=0x7f92d80ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:03.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.355+0000 7f92de400640 1 -- 172.21.15.28:0/3392211297 shutdown_connections 2024-02-21T23:10:03.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.355+0000 7f92de400640 1 -- 172.21.15.28:0/3392211297 wait complete. 2024-02-21T23:10:03.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.357+0000 7f92de400640 1 Processor -- start 2024-02-21T23:10:03.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.357+0000 7f92de400640 1 -- start start 2024-02-21T23:10:03.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.358+0000 7f92de400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92b8005210 0x7f92d80c0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:03.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.358+0000 7f92de400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f92d80be2d0 0x7f92d8151ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:03.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.358+0000 7f92de400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f92d80aeff0 0x7f92d80c41c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:03.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.358+0000 7f92de400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92d80c8dd0 con 0x7f92b8005210 2024-02-21T23:10:03.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.358+0000 7f92de400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92d8155e30 con 0x7f92d80aeff0 2024-02-21T23:10:03.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.358+0000 7f92de400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92d80be060 con 0x7f92d80be2d0 2024-02-21T23:10:03.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.358+0000 7f92dd000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92b8005210 0x7f92d80c0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:03.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.358+0000 7f92dd000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92b8005210 0x7f92d80c0fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33974/0 (socket says 172.21.15.28:33974) 2024-02-21T23:10:03.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.358+0000 7f92dd000640 1 -- 172.21.15.28:0/46622008 learned_addr learned my addr 172.21.15.28:0/46622008 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:03.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.358+0000 7f92cfe00640 1 --2- 172.21.15.28:0/46622008 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f92d80be2d0 0x7f92d8151ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:03.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.358+0000 7f92dda00640 1 --2- 172.21.15.28:0/46622008 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f92d80aeff0 0x7f92d80c41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:03.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.358+0000 7f92dd000640 1 -- 172.21.15.28:0/46622008 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f92d80be2d0 msgr2=0x7f92d8151ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:03.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.358+0000 7f92dd000640 1 --2- 172.21.15.28:0/46622008 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f92d80be2d0 0x7f92d8151ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:03.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.358+0000 7f92dd000640 1 -- 172.21.15.28:0/46622008 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f92d80aeff0 msgr2=0x7f92d80c41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:03.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.358+0000 7f92dd000640 1 --2- 172.21.15.28:0/46622008 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f92d80aeff0 0x7f92d80c41c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:03.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.358+0000 7f92dd000640 1 -- 172.21.15.28:0/46622008 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92d80bfa80 con 0x7f92b8005210 2024-02-21T23:10:03.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.359+0000 7f92dd000640 1 --2- 172.21.15.28:0/46622008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92b8005210 0x7f92d80c0fc0 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f92c800e440 tx=0x7f92c800e900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:03.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.359+0000 7f92d5600640 1 -- 172.21.15.28:0/46622008 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92c8019070 con 0x7f92b8005210 2024-02-21T23:10:03.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.359+0000 7f92d5600640 1 -- 172.21.15.28:0/46622008 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92c80052e0 con 0x7f92b8005210 2024-02-21T23:10:03.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.359+0000 7f92de400640 1 -- 172.21.15.28:0/46622008 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92d8152220 con 0x7f92b8005210 2024-02-21T23:10:03.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.359+0000 7f92d5600640 1 -- 172.21.15.28:0/46622008 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92c8005490 con 0x7f92b8005210 2024-02-21T23:10:03.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.359+0000 7f92de400640 1 -- 172.21.15.28:0/46622008 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92d80bf150 con 0x7f92b8005210 2024-02-21T23:10:03.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.360+0000 7f92de400640 1 -- 172.21.15.28:0/46622008 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92d804f290 con 0x7f92b8005210 2024-02-21T23:10:03.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.363+0000 7f92d5600640 1 -- 172.21.15.28:0/46622008 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f92c801c070 con 0x7f92b8005210 2024-02-21T23:10:03.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.363+0000 7f92d5600640 1 --2- 172.21.15.28:0/46622008 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f92b406b500 0x7f92b406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:03.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.364+0000 7f92d5600640 1 -- 172.21.15.28:0/46622008 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f92c8089890 con 0x7f92b8005210 2024-02-21T23:10:03.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.364+0000 7f92cfe00640 1 --2- 172.21.15.28:0/46622008 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f92b406b500 0x7f92b406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:03.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.365+0000 7f92cfe00640 1 --2- 172.21.15.28:0/46622008 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f92b406b500 0x7f92b406d9c0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f92c0001eb0 tx=0x7f92c0001490 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:03.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.368+0000 7f92d5600640 1 -- 172.21.15.28:0/46622008 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f92d80bfa80 con 0x7f92b8005210 2024-02-21T23:10:03.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.571+0000 7f92de400640 1 -- 172.21.15.28:0/46622008 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f92d80be6b0 con 0x7f92b8005210 2024-02-21T23:10:03.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.572+0000 7f92d5600640 1 -- 172.21.15.28:0/46622008 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v151) v1 ==== 84+0+209 (secure 0 0 0) 0x7f92c8057f20 con 0x7f92b8005210 2024-02-21T23:10:03.560 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:10:03.560 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-02-21T23:10:03.560 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:10:03.561 INFO:teuthology.orchestra.run.smithi028.stdout: "poolnum": 1, 2024-02-21T23:10:03.561 INFO:teuthology.orchestra.run.smithi028.stdout: "poolname": ".mgr" 2024-02-21T23:10:03.561 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:10:03.561 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:10:03.561 INFO:teuthology.orchestra.run.smithi028.stdout: "poolnum": 23, 2024-02-21T23:10:03.561 INFO:teuthology.orchestra.run.smithi028.stdout: "poolname": "cephfs_metadata" 2024-02-21T23:10:03.561 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:10:03.561 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:10:03.561 INFO:teuthology.orchestra.run.smithi028.stdout: "poolnum": 24, 2024-02-21T23:10:03.561 INFO:teuthology.orchestra.run.smithi028.stdout: "poolname": "cephfs_data" 2024-02-21T23:10:03.561 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:10:03.561 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-02-21T23:10:03.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.575+0000 7f92de400640 1 -- 172.21.15.28:0/46622008 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f92b406b500 msgr2=0x7f92b406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:03.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.575+0000 7f92de400640 1 --2- 172.21.15.28:0/46622008 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f92b406b500 0x7f92b406d9c0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f92c0001eb0 tx=0x7f92c0001490 comp rx=0 tx=0).stop 2024-02-21T23:10:03.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.575+0000 7f92de400640 1 -- 172.21.15.28:0/46622008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92b8005210 msgr2=0x7f92d80c0fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:03.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.575+0000 7f92de400640 1 --2- 172.21.15.28:0/46622008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92b8005210 0x7f92d80c0fc0 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f92c800e440 tx=0x7f92c800e900 comp rx=0 tx=0).stop 2024-02-21T23:10:03.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.575+0000 7f92de400640 1 -- 172.21.15.28:0/46622008 shutdown_connections 2024-02-21T23:10:03.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.575+0000 7f92de400640 1 --2- 172.21.15.28:0/46622008 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f92b406b500 0x7f92b406d9c0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:03.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.575+0000 7f92de400640 1 --2- 172.21.15.28:0/46622008 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f92d80be2d0 0x7f92d8151ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:03.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.575+0000 7f92de400640 1 --2- 172.21.15.28:0/46622008 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f92d80aeff0 0x7f92d80c41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:03.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.575+0000 7f92de400640 1 --2- 172.21.15.28:0/46622008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92b8005210 0x7f92d80c0fc0 unknown :-1 s=CLOSED pgs=1715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:03.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.575+0000 7f92de400640 1 -- 172.21.15.28:0/46622008 >> 172.21.15.28:0/46622008 conn(0x7f92d80ac370 msgr2=0x7f92d80c97e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:03.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.575+0000 7f92de400640 1 -- 172.21.15.28:0/46622008 shutdown_connections 2024-02-21T23:10:03.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.575+0000 7f92de400640 1 -- 172.21.15.28:0/46622008 wait complete. 2024-02-21T23:10:03.585 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs2' 2024-02-21T23:10:03.585 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs2_metadata --pg_num_min 64 2024-02-21T23:10:03.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.813+0000 7f98aa200640 1 Processor -- start 2024-02-21T23:10:03.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.814+0000 7f98aa200640 1 -- start start 2024-02-21T23:10:03.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.815+0000 7f98aa200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f98a41524a0 0x7f98a40a9d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:03.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.815+0000 7f98aa200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98a4152880 con 0x7f98a4155800 2024-02-21T23:10:03.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.815+0000 7f98aa200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98a40c4b80 con 0x7f98a406b8f0 2024-02-21T23:10:03.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.815+0000 7f98aa200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98a40c4e60 con 0x7f98a41524a0 2024-02-21T23:10:03.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.815+0000 7f98a9800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f98a4155800 0x7f98a40c7620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34382/0 (socket says 172.21.15.28:34382) 2024-02-21T23:10:03.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.815+0000 7f98a9800640 1 -- 172.21.15.28:0/1425168562 learned_addr learned my addr 172.21.15.28:0/1425168562 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:10:03.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.816+0000 7f98a8e00640 1 --2- 172.21.15.28:0/1425168562 >> v2:172.21.15.81:3300/0 conn(0x7f98a41524a0 0x7f98a40a9d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:03.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.816+0000 7f98a3400640 1 -- 172.21.15.28:0/1425168562 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2081551685 0 0) 0x7f98a4152880 con 0x7f98a4155800 2024-02-21T23:10:03.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.816+0000 7f98a3400640 1 -- 172.21.15.28:0/1425168562 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f988c003520 con 0x7f98a4155800 2024-02-21T23:10:03.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.816+0000 7f98a3400640 1 -- 172.21.15.28:0/1425168562 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2039870925 0 0) 0x7f988c003520 con 0x7f98a4155800 2024-02-21T23:10:03.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.816+0000 7f98a3400640 1 -- 172.21.15.28:0/1425168562 >> v2:172.21.15.81:3300/0 conn(0x7f98a41524a0 msgr2=0x7f98a40a9d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:03.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.816+0000 7f98a3400640 1 --2- 172.21.15.28:0/1425168562 >> v2:172.21.15.81:3300/0 conn(0x7f98a41524a0 0x7f98a40a9d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:03.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.816+0000 7f98a3400640 1 -- 172.21.15.28:0/1425168562 >> v1:172.21.15.81:6789/0 conn(0x7f98a406b8f0 legacy=0x7f98a4072380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:03.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.816+0000 7f98a3400640 1 -- 172.21.15.28:0/1425168562 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98a4155000 con 0x7f98a4155800 2024-02-21T23:10:03.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.816+0000 7f98a3400640 1 -- 172.21.15.28:0/1425168562 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f989c0019f0 con 0x7f98a4155800 2024-02-21T23:10:03.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.816+0000 7f98a3400640 1 -- 172.21.15.28:0/1425168562 >> v1:172.21.15.28:6789/0 conn(0x7f98a4155800 legacy=0x7f98a40c7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:03.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.816+0000 7f98a3400640 1 --2- 172.21.15.28:0/1425168562 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f988c003a00 0x7f988c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:03.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.816+0000 7f98a3400640 1 --2- 172.21.15.28:0/1425168562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f988c004800 0x7f988c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:03.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.816+0000 7f98a3400640 1 --2- 172.21.15.28:0/1425168562 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f988c0053c0 0x7f988c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:03.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.816+0000 7f98a3400640 1 -- 172.21.15.28:0/1425168562 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98a40c4e60 con 0x7f988c004800 2024-02-21T23:10:03.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.816+0000 7f98a3400640 1 -- 172.21.15.28:0/1425168562 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f988c00c2a0 con 0x7f988c0053c0 2024-02-21T23:10:03.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.816+0000 7f98a3400640 1 -- 172.21.15.28:0/1425168562 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f988c00c4a0 con 0x7f988c003a00 2024-02-21T23:10:03.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.817+0000 7f98a3e00640 1 --2- 172.21.15.28:0/1425168562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f988c004800 0x7f988c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:03.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.817+0000 7f98a8e00640 1 --2- 172.21.15.28:0/1425168562 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f988c003a00 0x7f988c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:03.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.817+0000 7f98a9800640 1 --2- 172.21.15.28:0/1425168562 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f988c0053c0 0x7f988c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:03.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.817+0000 7f98a3e00640 1 -- 172.21.15.28:0/1425168562 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f988c003a00 msgr2=0x7f988c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:03.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.817+0000 7f98a3e00640 1 --2- 172.21.15.28:0/1425168562 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f988c003a00 0x7f988c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:03.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.817+0000 7f98a3e00640 1 -- 172.21.15.28:0/1425168562 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f988c0053c0 msgr2=0x7f988c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:03.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.817+0000 7f98a3e00640 1 --2- 172.21.15.28:0/1425168562 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f988c0053c0 0x7f988c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:03.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.817+0000 7f98a3e00640 1 -- 172.21.15.28:0/1425168562 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f988c003520 con 0x7f988c004800 2024-02-21T23:10:03.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.817+0000 7f98a8e00640 1 --2- 172.21.15.28:0/1425168562 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f988c003a00 0x7f988c003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:10:03.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.817+0000 7f98a9800640 1 --2- 172.21.15.28:0/1425168562 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f988c0053c0 0x7f988c00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:10:03.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.817+0000 7f98a9800640 1 -- 172.21.15.28:0/1425168562 reap_dead start 2024-02-21T23:10:03.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.818+0000 7f98a3e00640 1 --2- 172.21.15.28:0/1425168562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f988c004800 0x7f988c004c80 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f9890001750 tx=0x7f988c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d01a6f372e90b0d9 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:03.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.818+0000 7f98a3400640 1 -- 172.21.15.28:0/1425168562 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9890001370 con 0x7f988c004800 2024-02-21T23:10:03.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.819+0000 7f98a3400640 1 -- 172.21.15.28:0/1425168562 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98900052d0 con 0x7f988c004800 2024-02-21T23:10:03.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.819+0000 7f98a3400640 1 -- 172.21.15.28:0/1425168562 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9890003830 con 0x7f988c004800 2024-02-21T23:10:03.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.819+0000 7f98aa200640 1 -- 172.21.15.28:0/1425168562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f988c004800 msgr2=0x7f988c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:03.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.819+0000 7f98aa200640 1 --2- 172.21.15.28:0/1425168562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f988c004800 0x7f988c004c80 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f9890001750 tx=0x7f988c005830 comp rx=0 tx=0).stop 2024-02-21T23:10:03.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.819+0000 7f98aa200640 1 -- 172.21.15.28:0/1425168562 shutdown_connections 2024-02-21T23:10:03.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.819+0000 7f98aa200640 1 --2- 172.21.15.28:0/1425168562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f988c004800 0x7f988c004c80 unknown :-1 s=CLOSED pgs=1717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:03.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.819+0000 7f98aa200640 1 -- 172.21.15.28:0/1425168562 >> 172.21.15.28:0/1425168562 conn(0x7f98a4066e40 msgr2=0x7f98a40a84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:03.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.819+0000 7f98aa200640 1 -- 172.21.15.28:0/1425168562 shutdown_connections 2024-02-21T23:10:03.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.819+0000 7f98aa200640 1 -- 172.21.15.28:0/1425168562 wait complete. 2024-02-21T23:10:03.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.821+0000 7f98aa200640 1 Processor -- start 2024-02-21T23:10:03.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.821+0000 7f98aa200640 1 -- start start 2024-02-21T23:10:03.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.822+0000 7f98aa200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f988c004800 0x7f98a406f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:03.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.822+0000 7f98aa200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f98a41e0fc0 0x7f98a41ee890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:03.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.822+0000 7f98aa200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98a41debb0 0x7f98a41eedd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:03.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.822+0000 7f98aa200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98a40a9d40 con 0x7f98a41debb0 2024-02-21T23:10:03.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.822+0000 7f98aa200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98a40a9ec0 con 0x7f98a41e0fc0 2024-02-21T23:10:03.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.822+0000 7f98aa200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98a40aa040 con 0x7f988c004800 2024-02-21T23:10:03.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.822+0000 7f98a9800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98a41debb0 0x7f98a41eedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:03.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.822+0000 7f98a9800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98a41debb0 0x7f98a41eedd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33994/0 (socket says 172.21.15.28:33994) 2024-02-21T23:10:03.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.822+0000 7f98a9800640 1 -- 172.21.15.28:0/3345520407 learned_addr learned my addr 172.21.15.28:0/3345520407 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:03.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.823+0000 7f98a3e00640 1 --2- 172.21.15.28:0/3345520407 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f98a41e0fc0 0x7f98a41ee890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:03.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.823+0000 7f98a8e00640 1 --2- 172.21.15.28:0/3345520407 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f988c004800 0x7f98a406f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:03.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.823+0000 7f98a9800640 1 -- 172.21.15.28:0/3345520407 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f988c004800 msgr2=0x7f98a406f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:03.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.823+0000 7f98a9800640 1 --2- 172.21.15.28:0/3345520407 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f988c004800 0x7f98a406f3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:03.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.823+0000 7f98a9800640 1 -- 172.21.15.28:0/3345520407 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f98a41e0fc0 msgr2=0x7f98a41ee890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:03.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.823+0000 7f98a9800640 1 --2- 172.21.15.28:0/3345520407 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f98a41e0fc0 0x7f98a41ee890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:03.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.823+0000 7f98a9800640 1 -- 172.21.15.28:0/3345520407 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98a40c58a0 con 0x7f98a41debb0 2024-02-21T23:10:03.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.823+0000 7f98a9800640 1 --2- 172.21.15.28:0/3345520407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98a41debb0 0x7f98a41eedd0 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f98a40c44d0 tx=0x7f989c00b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:03.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.824+0000 7f98a0c00640 1 -- 172.21.15.28:0/3345520407 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f989c004d60 con 0x7f98a41debb0 2024-02-21T23:10:03.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.824+0000 7f98a0c00640 1 -- 172.21.15.28:0/3345520407 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f989c00bb50 con 0x7f98a41debb0 2024-02-21T23:10:03.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.824+0000 7f98a0c00640 1 -- 172.21.15.28:0/3345520407 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f989c007110 con 0x7f98a41debb0 2024-02-21T23:10:03.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.824+0000 7f98aa200640 1 -- 172.21.15.28:0/3345520407 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98a40c7190 con 0x7f98a41debb0 2024-02-21T23:10:03.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.824+0000 7f98aa200640 1 -- 172.21.15.28:0/3345520407 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98a40c7730 con 0x7f98a41debb0 2024-02-21T23:10:03.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.825+0000 7f98aa200640 1 -- 172.21.15.28:0/3345520407 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98a406ce20 con 0x7f98a41debb0 2024-02-21T23:10:03.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.826+0000 7f98a0c00640 1 -- 172.21.15.28:0/3345520407 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f989c01b020 con 0x7f98a41debb0 2024-02-21T23:10:03.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.827+0000 7f98a0c00640 1 --2- 172.21.15.28:0/3345520407 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f988406b500 0x7f988406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:03.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.827+0000 7f98a0c00640 1 -- 172.21.15.28:0/3345520407 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f989c08a410 con 0x7f98a41debb0 2024-02-21T23:10:03.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.827+0000 7f98a8e00640 1 --2- 172.21.15.28:0/3345520407 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f988406b500 0x7f988406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:03.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.828+0000 7f98a8e00640 1 --2- 172.21.15.28:0/3345520407 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f988406b500 0x7f988406d9c0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f98a4050b90 tx=0x7f9898008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:03.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:03.835+0000 7f98a0c00640 1 -- 172.21.15.28:0/3345520407 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f98a406ce20 con 0x7f98a41debb0 2024-02-21T23:10:04.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.041+0000 7f98aa200640 1 -- 172.21.15.28:0/3345520407 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs2_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f98a4068a50 con 0x7f98a41debb0 2024-02-21T23:10:04.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.334+0000 7f98a0c00640 1 -- 172.21.15.28:0/3345520407 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs2_metadata", "pg_num_min": 64}]=0 pool 'cephfs2_metadata' created v152) v1 ==== 140+0+0 (secure 0 0 0) 0x7f98a4068a50 con 0x7f98a41debb0 2024-02-21T23:10:04.322 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs2_metadata' created 2024-02-21T23:10:04.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.339+0000 7f98aa200640 1 -- 172.21.15.28:0/3345520407 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f988406b500 msgr2=0x7f988406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:04.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.339+0000 7f98aa200640 1 --2- 172.21.15.28:0/3345520407 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f988406b500 0x7f988406d9c0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f98a4050b90 tx=0x7f9898008040 comp rx=0 tx=0).stop 2024-02-21T23:10:04.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.339+0000 7f98aa200640 1 -- 172.21.15.28:0/3345520407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98a41debb0 msgr2=0x7f98a41eedd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:04.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.339+0000 7f98aa200640 1 --2- 172.21.15.28:0/3345520407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98a41debb0 0x7f98a41eedd0 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f98a40c44d0 tx=0x7f989c00b4a0 comp rx=0 tx=0).stop 2024-02-21T23:10:04.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.339+0000 7f98aa200640 1 -- 172.21.15.28:0/3345520407 shutdown_connections 2024-02-21T23:10:04.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.339+0000 7f98aa200640 1 --2- 172.21.15.28:0/3345520407 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f988406b500 0x7f988406d9c0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:04.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.339+0000 7f98aa200640 1 --2- 172.21.15.28:0/3345520407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98a41debb0 0x7f98a41eedd0 unknown :-1 s=CLOSED pgs=1718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:04.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.339+0000 7f98aa200640 1 --2- 172.21.15.28:0/3345520407 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f98a41e0fc0 0x7f98a41ee890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:04.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.339+0000 7f98aa200640 1 --2- 172.21.15.28:0/3345520407 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f988c004800 0x7f98a406f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:04.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.339+0000 7f98aa200640 1 -- 172.21.15.28:0/3345520407 >> 172.21.15.28:0/3345520407 conn(0x7f98a4066e40 msgr2=0x7f98a40a84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:04.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.339+0000 7f98aa200640 1 -- 172.21.15.28:0/3345520407 shutdown_connections 2024-02-21T23:10:04.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.339+0000 7f98aa200640 1 -- 172.21.15.28:0/3345520407 wait complete. 2024-02-21T23:10:04.351 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs2_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-02-21T23:10:04.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.558+0000 7ff495400640 1 Processor -- start 2024-02-21T23:10:04.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.558+0000 7ff495400640 1 -- start start 2024-02-21T23:10:04.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.560+0000 7ff495400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff490156fb0 0x7ff4900c5520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:04.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.560+0000 7ff495400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff49006f680 con 0x7ff49006f150 2024-02-21T23:10:04.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.560+0000 7ff495400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4901564d0 con 0x7ff49014ccd0 2024-02-21T23:10:04.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.560+0000 7ff495400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff490156270 con 0x7ff490156fb0 2024-02-21T23:10:04.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.560+0000 7ff494a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff49006f150 0x7ff490072240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34398/0 (socket says 172.21.15.28:34398) 2024-02-21T23:10:04.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.560+0000 7ff494a00640 1 -- 172.21.15.28:0/100572915 learned_addr learned my addr 172.21.15.28:0/100572915 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:10:04.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.561+0000 7ff48f400640 1 --2- 172.21.15.28:0/100572915 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff490156fb0 0x7ff4900c5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:04.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.561+0000 7ff48ea00640 1 -- 172.21.15.28:0/100572915 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4183903938 0 0) 0x7ff49006f680 con 0x7ff49006f150 2024-02-21T23:10:04.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.561+0000 7ff48ea00640 1 -- 172.21.15.28:0/100572915 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff474003520 con 0x7ff49006f150 2024-02-21T23:10:04.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.561+0000 7ff48ea00640 1 -- 172.21.15.28:0/100572915 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3971897991 0 0) 0x7ff474003520 con 0x7ff49006f150 2024-02-21T23:10:04.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.561+0000 7ff48ea00640 1 -- 172.21.15.28:0/100572915 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff490156fb0 msgr2=0x7ff4900c5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:04.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.561+0000 7ff48ea00640 1 --2- 172.21.15.28:0/100572915 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff490156fb0 0x7ff4900c5520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:04.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.562+0000 7ff48ea00640 1 -- 172.21.15.28:0/100572915 >> v1:172.21.15.81:6789/0 conn(0x7ff49014ccd0 legacy=0x7ff490153700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:04.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.562+0000 7ff48ea00640 1 -- 172.21.15.28:0/100572915 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4901567b0 con 0x7ff49006f150 2024-02-21T23:10:04.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.562+0000 7ff48ea00640 1 -- 172.21.15.28:0/100572915 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7ff4880019f0 con 0x7ff49006f150 2024-02-21T23:10:04.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.562+0000 7ff48ea00640 1 -- 172.21.15.28:0/100572915 >> v1:172.21.15.28:6789/0 conn(0x7ff49006f150 legacy=0x7ff490072240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:04.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.562+0000 7ff48ea00640 1 --2- 172.21.15.28:0/100572915 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff4740039e0 0x7ff474003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:04.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.562+0000 7ff48ea00640 1 --2- 172.21.15.28:0/100572915 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff490156fb0 0x7ff474004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:04.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.562+0000 7ff48ea00640 1 --2- 172.21.15.28:0/100572915 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff474004d40 0x7ff4740051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:04.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.562+0000 7ff48ea00640 1 -- 172.21.15.28:0/100572915 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff490156270 con 0x7ff474004d40 2024-02-21T23:10:04.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.562+0000 7ff48ea00640 1 -- 172.21.15.28:0/100572915 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff47400bb80 con 0x7ff4740039e0 2024-02-21T23:10:04.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.562+0000 7ff48ea00640 1 -- 172.21.15.28:0/100572915 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff47400bd80 con 0x7ff490156fb0 2024-02-21T23:10:04.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.562+0000 7ff494a00640 1 --2- 172.21.15.28:0/100572915 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff474004d40 0x7ff4740051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:04.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.562+0000 7ff48fe00640 1 --2- 172.21.15.28:0/100572915 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff4740039e0 0x7ff474003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:04.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.562+0000 7ff48f400640 1 --2- 172.21.15.28:0/100572915 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff490156fb0 0x7ff474004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:04.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.563+0000 7ff494a00640 1 -- 172.21.15.28:0/100572915 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff490156fb0 msgr2=0x7ff474004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:04.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.563+0000 7ff494a00640 1 --2- 172.21.15.28:0/100572915 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff490156fb0 0x7ff474004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:04.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.563+0000 7ff494a00640 1 -- 172.21.15.28:0/100572915 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff4740039e0 msgr2=0x7ff474003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:04.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.563+0000 7ff494a00640 1 --2- 172.21.15.28:0/100572915 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff4740039e0 0x7ff474003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:04.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.563+0000 7ff494a00640 1 -- 172.21.15.28:0/100572915 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff474003520 con 0x7ff474004d40 2024-02-21T23:10:04.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.564+0000 7ff494a00640 1 --2- 172.21.15.28:0/100572915 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff474004d40 0x7ff4740051c0 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7ff48800a830 tx=0x7ff474005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74c6e8c18a9c0c76 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:04.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.564+0000 7ff48ea00640 1 -- 172.21.15.28:0/100572915 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff48800a480 con 0x7ff474004d40 2024-02-21T23:10:04.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.565+0000 7ff48ea00640 1 -- 172.21.15.28:0/100572915 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff488034630 con 0x7ff474004d40 2024-02-21T23:10:04.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.565+0000 7ff48ea00640 1 -- 172.21.15.28:0/100572915 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff488034a40 con 0x7ff474004d40 2024-02-21T23:10:04.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.565+0000 7ff495400640 1 -- 172.21.15.28:0/100572915 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff474004d40 msgr2=0x7ff4740051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:04.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.565+0000 7ff495400640 1 --2- 172.21.15.28:0/100572915 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff474004d40 0x7ff4740051c0 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7ff48800a830 tx=0x7ff474005830 comp rx=0 tx=0).stop 2024-02-21T23:10:04.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.565+0000 7ff494a00640 1 -- 172.21.15.28:0/100572915 reap_dead start 2024-02-21T23:10:04.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.565+0000 7ff495400640 1 -- 172.21.15.28:0/100572915 shutdown_connections 2024-02-21T23:10:04.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.565+0000 7ff495400640 1 -- 172.21.15.28:0/100572915 >> 172.21.15.28:0/100572915 conn(0x7ff490066e40 msgr2=0x7ff4900c7620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:04.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.565+0000 7ff495400640 1 -- 172.21.15.28:0/100572915 shutdown_connections 2024-02-21T23:10:04.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.565+0000 7ff495400640 1 -- 172.21.15.28:0/100572915 wait complete. 2024-02-21T23:10:04.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.567+0000 7ff495400640 1 Processor -- start 2024-02-21T23:10:04.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.567+0000 7ff495400640 1 -- start start 2024-02-21T23:10:04.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.568+0000 7ff495400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4900c3310 0x7ff4901ec390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:04.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.568+0000 7ff495400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff4901e8dd0 0x7ff49014d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:04.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.568+0000 7ff495400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff490140df0 0x7ff490141280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:04.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.568+0000 7ff495400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff490072ad0 con 0x7ff4900c3310 2024-02-21T23:10:04.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.568+0000 7ff495400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff490072950 con 0x7ff4901e8dd0 2024-02-21T23:10:04.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.568+0000 7ff495400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4900c5520 con 0x7ff490140df0 2024-02-21T23:10:04.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.568+0000 7ff48fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4900c3310 0x7ff4901ec390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:04.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.568+0000 7ff48fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4900c3310 0x7ff4901ec390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34004/0 (socket says 172.21.15.28:34004) 2024-02-21T23:10:04.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.568+0000 7ff48fe00640 1 -- 172.21.15.28:0/2714836329 learned_addr learned my addr 172.21.15.28:0/2714836329 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:04.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.569+0000 7ff494a00640 1 --2- 172.21.15.28:0/2714836329 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff490140df0 0x7ff490141280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:04.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.569+0000 7ff48fe00640 1 -- 172.21.15.28:0/2714836329 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff490140df0 msgr2=0x7ff490141280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:04.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.569+0000 7ff48fe00640 1 --2- 172.21.15.28:0/2714836329 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff490140df0 0x7ff490141280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:04.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.569+0000 7ff48f400640 1 --2- 172.21.15.28:0/2714836329 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff4901e8dd0 0x7ff49014d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:04.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.569+0000 7ff48fe00640 1 -- 172.21.15.28:0/2714836329 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff4901e8dd0 msgr2=0x7ff49014d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:04.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.569+0000 7ff48fe00640 1 --2- 172.21.15.28:0/2714836329 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff4901e8dd0 0x7ff49014d630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:04.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.569+0000 7ff48fe00640 1 -- 172.21.15.28:0/2714836329 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff49014ffd0 con 0x7ff4900c3310 2024-02-21T23:10:04.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.569+0000 7ff48fe00640 1 --2- 172.21.15.28:0/2714836329 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4900c3310 0x7ff4901ec390 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7ff48000f850 tx=0x7ff48000fd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:04.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.569+0000 7ff487e00640 1 -- 172.21.15.28:0/2714836329 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff48000fe40 con 0x7ff4900c3310 2024-02-21T23:10:04.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.569+0000 7ff487e00640 1 -- 172.21.15.28:0/2714836329 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff480018070 con 0x7ff4900c3310 2024-02-21T23:10:04.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.569+0000 7ff487e00640 1 -- 172.21.15.28:0/2714836329 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff48000d730 con 0x7ff4900c3310 2024-02-21T23:10:04.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.570+0000 7ff495400640 1 -- 172.21.15.28:0/2714836329 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4901502b0 con 0x7ff4900c3310 2024-02-21T23:10:04.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.570+0000 7ff495400640 1 -- 172.21.15.28:0/2714836329 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4901ebe70 con 0x7ff4900c3310 2024-02-21T23:10:04.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.571+0000 7ff495400640 1 -- 172.21.15.28:0/2714836329 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff490150480 con 0x7ff4900c3310 2024-02-21T23:10:04.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.572+0000 7ff487e00640 1 -- 172.21.15.28:0/2714836329 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff48000d8d0 con 0x7ff4900c3310 2024-02-21T23:10:04.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.573+0000 7ff487e00640 1 --2- 172.21.15.28:0/2714836329 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff47006b4d0 0x7ff47006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:04.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.573+0000 7ff487e00640 1 -- 172.21.15.28:0/2714836329 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 6472+0+0 (secure 0 0 0) 0x7ff48008ae90 con 0x7ff4900c3310 2024-02-21T23:10:04.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.574+0000 7ff48f400640 1 --2- 172.21.15.28:0/2714836329 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff47006b4d0 0x7ff47006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:04.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.575+0000 7ff48f400640 1 --2- 172.21.15.28:0/2714836329 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff47006b4d0 0x7ff47006d990 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7ff47c0099e0 tx=0x7ff47c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:04.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.579+0000 7ff487e00640 1 -- 172.21.15.28:0/2714836329 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ff4800599f0 con 0x7ff4900c3310 2024-02-21T23:10:04.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:04.781+0000 7ff495400640 1 -- 172.21.15.28:0/2714836329 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs2_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7ff49004f290 con 0x7ff4900c3310 2024-02-21T23:10:05.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.352+0000 7ff487e00640 1 -- 172.21.15.28:0/2714836329 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs2_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs2_data' created v153) v1 ==== 172+0+0 (secure 0 0 0) 0x7ff48008a020 con 0x7ff4900c3310 2024-02-21T23:10:05.340 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs2_data' created 2024-02-21T23:10:05.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.356+0000 7ff495400640 1 -- 172.21.15.28:0/2714836329 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff47006b4d0 msgr2=0x7ff47006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:05.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.356+0000 7ff495400640 1 --2- 172.21.15.28:0/2714836329 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff47006b4d0 0x7ff47006d990 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7ff47c0099e0 tx=0x7ff47c008040 comp rx=0 tx=0).stop 2024-02-21T23:10:05.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.356+0000 7ff495400640 1 -- 172.21.15.28:0/2714836329 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4900c3310 msgr2=0x7ff4901ec390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:05.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.356+0000 7ff495400640 1 --2- 172.21.15.28:0/2714836329 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4900c3310 0x7ff4901ec390 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7ff48000f850 tx=0x7ff48000fd10 comp rx=0 tx=0).stop 2024-02-21T23:10:05.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.359+0000 7ff495400640 1 -- 172.21.15.28:0/2714836329 shutdown_connections 2024-02-21T23:10:05.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.359+0000 7ff495400640 1 --2- 172.21.15.28:0/2714836329 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff47006b4d0 0x7ff47006d990 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:05.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.359+0000 7ff495400640 1 --2- 172.21.15.28:0/2714836329 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff490140df0 0x7ff490141280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:05.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.359+0000 7ff495400640 1 --2- 172.21.15.28:0/2714836329 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff4901e8dd0 0x7ff49014d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:05.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.359+0000 7ff495400640 1 --2- 172.21.15.28:0/2714836329 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4900c3310 0x7ff4901ec390 unknown :-1 s=CLOSED pgs=1721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:05.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.359+0000 7ff495400640 1 -- 172.21.15.28:0/2714836329 >> 172.21.15.28:0/2714836329 conn(0x7ff490066e40 msgr2=0x7ff4900c4d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:05.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.360+0000 7ff495400640 1 -- 172.21.15.28:0/2714836329 shutdown_connections 2024-02-21T23:10:05.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.360+0000 7ff495400640 1 -- 172.21.15.28:0/2714836329 wait complete. 2024-02-21T23:10:05.386 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs2 cephfs2_metadata cephfs2_data 2024-02-21T23:10:05.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.583+0000 7f35f1c00640 1 Processor -- start 2024-02-21T23:10:05.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.583+0000 7f35f1c00640 1 -- start start 2024-02-21T23:10:05.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.584+0000 7f35f1c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f35ec06b0e0 0x7f35ec0c5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:05.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.584+0000 7f35f1c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f35ec0c5d80 0x7f35ec14fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:05.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.584+0000 7f35f1c00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35ec0c61e0 con 0x7f35ec150530 2024-02-21T23:10:05.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.584+0000 7f35f1c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35ec153bc0 con 0x7f35ec06b0e0 2024-02-21T23:10:05.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.584+0000 7f35f1c00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35ec152fb0 con 0x7f35ec0c5d80 2024-02-21T23:10:05.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.585+0000 7f35eaa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f35ec06b0e0 0x7f35ec0c5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:05.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.585+0000 7f35eaa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f35ec06b0e0 0x7f35ec0c5840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35420/0 (socket says 172.21.15.28:35420) 2024-02-21T23:10:05.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.585+0000 7f35eaa00640 1 -- 172.21.15.28:0/2683571642 learned_addr learned my addr 172.21.15.28:0/2683571642 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:05.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.585+0000 7f35ea000640 1 --2- 172.21.15.28:0/2683571642 >> v2:172.21.15.81:3300/0 conn(0x7f35ec0c5d80 0x7f35ec14fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:05.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.585+0000 7f35eaa00640 1 -- 172.21.15.28:0/2683571642 >> v2:172.21.15.81:3300/0 conn(0x7f35ec0c5d80 msgr2=0x7f35ec14fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:05.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.585+0000 7f35eaa00640 1 --2- 172.21.15.28:0/2683571642 >> v2:172.21.15.81:3300/0 conn(0x7f35ec0c5d80 0x7f35ec14fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:05.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.585+0000 7f35eaa00640 1 -- 172.21.15.28:0/2683571642 >> v1:172.21.15.81:6789/0 conn(0x7f35ec150530 legacy=0x7f35ec156500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:05.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.585+0000 7f35eaa00640 1 -- 172.21.15.28:0/2683571642 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35ec153290 con 0x7f35ec06b0e0 2024-02-21T23:10:05.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.587+0000 7f35eaa00640 1 --2- 172.21.15.28:0/2683571642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35ec06b0e0 0x7f35ec0c5840 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7f35ec0c65f0 tx=0x7f35dc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef652d85798e40c9 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:05.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.587+0000 7f35e9600640 1 -- 172.21.15.28:0/2683571642 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35dc02e070 con 0x7f35ec06b0e0 2024-02-21T23:10:05.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.587+0000 7f35e9600640 1 -- 172.21.15.28:0/2683571642 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35dc037070 con 0x7f35ec06b0e0 2024-02-21T23:10:05.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.587+0000 7f35e9600640 1 -- 172.21.15.28:0/2683571642 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35dc031a00 con 0x7f35ec06b0e0 2024-02-21T23:10:05.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.588+0000 7f35f1c00640 1 -- 172.21.15.28:0/2683571642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35ec06b0e0 msgr2=0x7f35ec0c5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:05.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.588+0000 7f35f1c00640 1 --2- 172.21.15.28:0/2683571642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35ec06b0e0 0x7f35ec0c5840 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7f35ec0c65f0 tx=0x7f35dc02b880 comp rx=0 tx=0).stop 2024-02-21T23:10:05.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.588+0000 7f35f1c00640 1 -- 172.21.15.28:0/2683571642 shutdown_connections 2024-02-21T23:10:05.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.588+0000 7f35f1c00640 1 --2- 172.21.15.28:0/2683571642 >> v2:172.21.15.81:3300/0 conn(0x7f35ec0c5d80 0x7f35ec14fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:05.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.588+0000 7f35f1c00640 1 --2- 172.21.15.28:0/2683571642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35ec06b0e0 0x7f35ec0c5840 unknown :-1 s=CLOSED pgs=1722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:05.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.588+0000 7f35f1c00640 1 -- 172.21.15.28:0/2683571642 >> 172.21.15.28:0/2683571642 conn(0x7f35ec0a85a0 msgr2=0x7f35ec0a8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:05.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.588+0000 7f35f1c00640 1 -- 172.21.15.28:0/2683571642 shutdown_connections 2024-02-21T23:10:05.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.588+0000 7f35f1c00640 1 -- 172.21.15.28:0/2683571642 wait complete. 2024-02-21T23:10:05.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.590+0000 7f35f1c00640 1 Processor -- start 2024-02-21T23:10:05.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.590+0000 7f35f1c00640 1 -- start start 2024-02-21T23:10:05.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.591+0000 7f35f1c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f35ec06b0e0 0x7f35ec149580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:05.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.591+0000 7f35f1c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35ec0c5d80 0x7f35ec14f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:05.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.591+0000 7f35f1c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f35ec150530 0x7f35ec0aa8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:05.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.591+0000 7f35f1c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35ec150970 con 0x7f35ec0c5d80 2024-02-21T23:10:05.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.591+0000 7f35f1c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35ec153850 con 0x7f35ec06b0e0 2024-02-21T23:10:05.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.591+0000 7f35f1c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35ec06b4c0 con 0x7f35ec150530 2024-02-21T23:10:05.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.591+0000 7f35ea000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35ec0c5d80 0x7f35ec14f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:05.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.591+0000 7f35ea000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35ec0c5d80 0x7f35ec14f190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35430/0 (socket says 172.21.15.28:35430) 2024-02-21T23:10:05.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.591+0000 7f35ea000640 1 -- 172.21.15.28:0/304989473 learned_addr learned my addr 172.21.15.28:0/304989473 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:05.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.592+0000 7f35eaa00640 1 --2- 172.21.15.28:0/304989473 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f35ec06b0e0 0x7f35ec149580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:05.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.592+0000 7f35ea000640 1 -- 172.21.15.28:0/304989473 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f35ec150530 msgr2=0x7f35ec0aa8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:05.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.592+0000 7f35eb400640 1 --2- 172.21.15.28:0/304989473 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f35ec150530 0x7f35ec0aa8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:05.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.592+0000 7f35ea000640 1 --2- 172.21.15.28:0/304989473 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f35ec150530 0x7f35ec0aa8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:05.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.592+0000 7f35ea000640 1 -- 172.21.15.28:0/304989473 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f35ec06b0e0 msgr2=0x7f35ec149580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:05.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.592+0000 7f35ea000640 1 --2- 172.21.15.28:0/304989473 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f35ec06b0e0 0x7f35ec149580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:05.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.592+0000 7f35ea000640 1 -- 172.21.15.28:0/304989473 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35ec151b90 con 0x7f35ec0c5d80 2024-02-21T23:10:05.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.592+0000 7f35eb400640 1 --2- 172.21.15.28:0/304989473 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f35ec150530 0x7f35ec0aa8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:10:05.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.592+0000 7f35ea000640 1 --2- 172.21.15.28:0/304989473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35ec0c5d80 0x7f35ec14f190 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7f35d800d800 tx=0x7f35d800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:05.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.592+0000 7f35e2a00640 1 -- 172.21.15.28:0/304989473 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35d800b660 con 0x7f35ec0c5d80 2024-02-21T23:10:05.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.592+0000 7f35e2a00640 1 -- 172.21.15.28:0/304989473 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35d8019070 con 0x7f35ec0c5d80 2024-02-21T23:10:05.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.592+0000 7f35e2a00640 1 -- 172.21.15.28:0/304989473 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35d800ba20 con 0x7f35ec0c5d80 2024-02-21T23:10:05.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.593+0000 7f35f1c00640 1 -- 172.21.15.28:0/304989473 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35ec151e70 con 0x7f35ec0c5d80 2024-02-21T23:10:05.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.593+0000 7f35f1c00640 1 -- 172.21.15.28:0/304989473 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35ec149ac0 con 0x7f35ec0c5d80 2024-02-21T23:10:05.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.594+0000 7f35f1c00640 1 -- 172.21.15.28:0/304989473 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35ec067f30 con 0x7f35ec0c5d80 2024-02-21T23:10:05.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.595+0000 7f35e2a00640 1 -- 172.21.15.28:0/304989473 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f35d800bbc0 con 0x7f35ec0c5d80 2024-02-21T23:10:05.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.596+0000 7f35e2a00640 1 --2- 172.21.15.28:0/304989473 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f35cc06b4d0 0x7f35cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:05.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.596+0000 7f35e2a00640 1 -- 172.21.15.28:0/304989473 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 6855+0+0 (secure 0 0 0) 0x7f35d808aa80 con 0x7f35ec0c5d80 2024-02-21T23:10:05.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.596+0000 7f35eaa00640 1 --2- 172.21.15.28:0/304989473 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f35cc06b4d0 0x7f35cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:05.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.598+0000 7f35eaa00640 1 --2- 172.21.15.28:0/304989473 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f35cc06b4d0 0x7f35cc06d990 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f35dc036000 tx=0x7f35dc03d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:05.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.601+0000 7f35e2a00640 1 -- 172.21.15.28:0/304989473 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f35d801c360 con 0x7f35ec0c5d80 2024-02-21T23:10:05.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:05.841+0000 7f35f1c00640 1 -- 172.21.15.28:0/304989473 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs2", "metadata": "cephfs2_metadata", "data": "cephfs2_data"} v 0) v1 -- 0x7f35ec14ead0 con 0x7f35ec0c5d80 2024-02-21T23:10:06.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.359+0000 7f35e2a00640 1 -- 172.21.15.28:0/304989473 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs2", "metadata": "cephfs2_metadata", "data": "cephfs2_data"}]=0 new fs with metadata pool 25 and data pool 26 v159) v1 ==== 177+0+0 (secure 0 0 0) 0x7f35d8059510 con 0x7f35ec0c5d80 2024-02-21T23:10:06.348 INFO:teuthology.orchestra.run.smithi028.stderr:new fs with metadata pool 25 and data pool 26 2024-02-21T23:10:06.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.367+0000 7f35f1c00640 1 -- 172.21.15.28:0/304989473 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f35cc06b4d0 msgr2=0x7f35cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:06.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.367+0000 7f35f1c00640 1 --2- 172.21.15.28:0/304989473 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f35cc06b4d0 0x7f35cc06d990 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f35dc036000 tx=0x7f35dc03d040 comp rx=0 tx=0).stop 2024-02-21T23:10:06.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.368+0000 7f35f1c00640 1 -- 172.21.15.28:0/304989473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35ec0c5d80 msgr2=0x7f35ec14f190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:06.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.368+0000 7f35f1c00640 1 --2- 172.21.15.28:0/304989473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35ec0c5d80 0x7f35ec14f190 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7f35d800d800 tx=0x7f35d800dcc0 comp rx=0 tx=0).stop 2024-02-21T23:10:06.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.368+0000 7f35f1c00640 1 -- 172.21.15.28:0/304989473 shutdown_connections 2024-02-21T23:10:06.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.368+0000 7f35f1c00640 1 --2- 172.21.15.28:0/304989473 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f35cc06b4d0 0x7f35cc06d990 secure :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f35dc036000 tx=0x7f35dc03d040 comp rx=0 tx=0).stop 2024-02-21T23:10:06.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.368+0000 7f35f1c00640 1 --2- 172.21.15.28:0/304989473 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f35ec150530 0x7f35ec0aa8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:06.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.368+0000 7f35f1c00640 1 --2- 172.21.15.28:0/304989473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35ec0c5d80 0x7f35ec14f190 secure :-1 s=CLOSED pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7f35d800d800 tx=0x7f35d800dcc0 comp rx=0 tx=0).stop 2024-02-21T23:10:06.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.368+0000 7f35f1c00640 1 --2- 172.21.15.28:0/304989473 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f35ec06b0e0 0x7f35ec149580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:06.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.368+0000 7f35f1c00640 1 -- 172.21.15.28:0/304989473 >> 172.21.15.28:0/304989473 conn(0x7f35ec0a85a0 msgr2=0x7f35ec0c4550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:06.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.372+0000 7f35f1c00640 1 -- 172.21.15.28:0/304989473 shutdown_connections 2024-02-21T23:10:06.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.373+0000 7f35f1c00640 1 -- 172.21.15.28:0/304989473 wait complete. 2024-02-21T23:10:06.397 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:10:06.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.594+0000 7f28c1a00640 1 Processor -- start 2024-02-21T23:10:06.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.594+0000 7f28c1a00640 1 -- start start 2024-02-21T23:10:06.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.596+0000 7f28c1a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f28bc0aff90 0x7f28bc0c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:06.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.596+0000 7f28c1a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28bc0c74c0 con 0x7f28bc0c5720 2024-02-21T23:10:06.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.596+0000 7f28c1a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28bc0c77a0 con 0x7f28bc06dbf0 2024-02-21T23:10:06.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.596+0000 7f28c1a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28bc0c7270 con 0x7f28bc0aff90 2024-02-21T23:10:06.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.596+0000 7f28bbe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f28bc0aff90 0x7f28bc0c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:06.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.596+0000 7f28c1000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f28bc0c5720 0x7f28bc15f890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60866/0 (socket says 172.21.15.28:60866) 2024-02-21T23:10:06.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.596+0000 7f28c1000640 1 -- 172.21.15.28:0/1467819899 learned_addr learned my addr 172.21.15.28:0/1467819899 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:10:06.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.597+0000 7f28bbe00640 1 -- 172.21.15.28:0/1467819899 >> v1:172.21.15.81:6789/0 conn(0x7f28bc06dbf0 legacy=0x7f28bc158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:06.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.597+0000 7f28bbe00640 1 -- 172.21.15.28:0/1467819899 >> v1:172.21.15.28:6789/0 conn(0x7f28bc0c5720 legacy=0x7f28bc15f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:06.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.597+0000 7f28bbe00640 1 -- 172.21.15.28:0/1467819899 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28bc0c7a80 con 0x7f28bc0aff90 2024-02-21T23:10:06.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.598+0000 7f28bbe00640 1 --2- 172.21.15.28:0/1467819899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28bc0aff90 0x7f28bc0c51e0 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7f28ac0097a0 tx=0x7f28ac02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9e96ee1913a2d29 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:06.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.598+0000 7f28bb400640 1 -- 172.21.15.28:0/1467819899 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28ac035020 con 0x7f28bc0aff90 2024-02-21T23:10:06.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.598+0000 7f28bb400640 1 -- 172.21.15.28:0/1467819899 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28ac031830 con 0x7f28bc0aff90 2024-02-21T23:10:06.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.598+0000 7f28bb400640 1 -- 172.21.15.28:0/1467819899 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28ac031c00 con 0x7f28bc0aff90 2024-02-21T23:10:06.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.599+0000 7f28c1a00640 1 -- 172.21.15.28:0/1467819899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28bc0aff90 msgr2=0x7f28bc0c51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:06.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.599+0000 7f28c1a00640 1 --2- 172.21.15.28:0/1467819899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28bc0aff90 0x7f28bc0c51e0 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7f28ac0097a0 tx=0x7f28ac02b8d0 comp rx=0 tx=0).stop 2024-02-21T23:10:06.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.599+0000 7f28c1a00640 1 -- 172.21.15.28:0/1467819899 shutdown_connections 2024-02-21T23:10:06.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.599+0000 7f28c1a00640 1 --2- 172.21.15.28:0/1467819899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28bc0aff90 0x7f28bc0c51e0 unknown :-1 s=CLOSED pgs=1725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:06.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.599+0000 7f28c1a00640 1 -- 172.21.15.28:0/1467819899 >> 172.21.15.28:0/1467819899 conn(0x7f28bc0ac370 msgr2=0x7f28bc0ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:06.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.599+0000 7f28c1a00640 1 -- 172.21.15.28:0/1467819899 shutdown_connections 2024-02-21T23:10:06.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.599+0000 7f28c1a00640 1 -- 172.21.15.28:0/1467819899 wait complete. 2024-02-21T23:10:06.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.601+0000 7f28c1a00640 1 Processor -- start 2024-02-21T23:10:06.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.601+0000 7f28c1a00640 1 -- start start 2024-02-21T23:10:06.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.602+0000 7f28c1a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f28bc06dbf0 0x7f28bc0ae740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:06.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.602+0000 7f28c1a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28bc0aff90 0x7f28bc0c1f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:06.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.602+0000 7f28c1a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f28bc0c5720 0x7f28bc14e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:06.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.602+0000 7f28c1a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28bc0c8760 con 0x7f28bc0aff90 2024-02-21T23:10:06.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.602+0000 7f28c1a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28bc158630 con 0x7f28bc0c5720 2024-02-21T23:10:06.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.602+0000 7f28c1a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28bc0c88e0 con 0x7f28bc06dbf0 2024-02-21T23:10:06.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.602+0000 7f28b3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28bc0aff90 0x7f28bc0c1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:06.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.602+0000 7f28b3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28bc0aff90 0x7f28bc0c1f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35448/0 (socket says 172.21.15.28:35448) 2024-02-21T23:10:06.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.602+0000 7f28b3400640 1 -- 172.21.15.28:0/803365694 learned_addr learned my addr 172.21.15.28:0/803365694 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:06.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.602+0000 7f28c1000640 1 --2- 172.21.15.28:0/803365694 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f28bc0c5720 0x7f28bc14e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:06.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.602+0000 7f28bbe00640 1 --2- 172.21.15.28:0/803365694 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f28bc06dbf0 0x7f28bc0ae740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:06.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.602+0000 7f28b3400640 1 -- 172.21.15.28:0/803365694 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f28bc06dbf0 msgr2=0x7f28bc0ae740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:06.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.602+0000 7f28b3400640 1 --2- 172.21.15.28:0/803365694 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f28bc06dbf0 0x7f28bc0ae740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:06.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.602+0000 7f28b3400640 1 -- 172.21.15.28:0/803365694 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f28bc0c5720 msgr2=0x7f28bc14e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:06.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.602+0000 7f28b3400640 1 --2- 172.21.15.28:0/803365694 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f28bc0c5720 0x7f28bc14e290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:06.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.602+0000 7f28b3400640 1 -- 172.21.15.28:0/803365694 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28bc0be610 con 0x7f28bc0aff90 2024-02-21T23:10:06.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.602+0000 7f28bbe00640 1 --2- 172.21.15.28:0/803365694 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f28bc06dbf0 0x7f28bc0ae740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:10:06.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.603+0000 7f28c1000640 1 --2- 172.21.15.28:0/803365694 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f28bc0c5720 0x7f28bc14e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:10:06.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.603+0000 7f28b3400640 1 --2- 172.21.15.28:0/803365694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28bc0aff90 0x7f28bc0c1f90 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7f28a400c400 tx=0x7f28a400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:06.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.603+0000 7f28b8c00640 1 -- 172.21.15.28:0/803365694 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28a4011790 con 0x7f28bc0aff90 2024-02-21T23:10:06.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.603+0000 7f28b8c00640 1 -- 172.21.15.28:0/803365694 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28a4017070 con 0x7f28bc0aff90 2024-02-21T23:10:06.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.603+0000 7f28b8c00640 1 -- 172.21.15.28:0/803365694 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28a4011bb0 con 0x7f28bc0aff90 2024-02-21T23:10:06.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.603+0000 7f28c1a00640 1 -- 172.21.15.28:0/803365694 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28bc152e20 con 0x7f28bc0aff90 2024-02-21T23:10:06.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.603+0000 7f28c1a00640 1 -- 172.21.15.28:0/803365694 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28bc1533e0 con 0x7f28bc0aff90 2024-02-21T23:10:06.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.604+0000 7f28c1a00640 1 -- 172.21.15.28:0/803365694 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28bc04f290 con 0x7f28bc0aff90 2024-02-21T23:10:06.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.605+0000 7f28b8c00640 1 -- 172.21.15.28:0/803365694 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f28a401c460 con 0x7f28bc0aff90 2024-02-21T23:10:06.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.606+0000 7f28b8c00640 1 --2- 172.21.15.28:0/803365694 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f289806b4d0 0x7f289806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:06.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.606+0000 7f28b8c00640 1 -- 172.21.15.28:0/803365694 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f28a408a5c0 con 0x7f28bc0aff90 2024-02-21T23:10:06.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.607+0000 7f28bbe00640 1 --2- 172.21.15.28:0/803365694 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f289806b4d0 0x7f289806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:06.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.608+0000 7f28bbe00640 1 --2- 172.21.15.28:0/803365694 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f289806b4d0 0x7f289806d990 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f28ac036000 tx=0x7f28ac008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:06.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.611+0000 7f28b8c00640 1 -- 172.21.15.28:0/803365694 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f28a4058ec0 con 0x7f28bc0aff90 2024-02-21T23:10:06.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.810+0000 7f28c1a00640 1 -- 172.21.15.28:0/803365694 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f28bc06d9d0 con 0x7f28bc0aff90 2024-02-21T23:10:06.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.814+0000 7f28b8c00640 1 -- 172.21.15.28:0/803365694 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v154) v1 ==== 74+0+20555 (secure 0 0 0) 0x7f28a40587c0 con 0x7f28bc0aff90 2024-02-21T23:10:06.803 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:10:06.803 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":154,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:10:05.843379+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:09:51.634291+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"150","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-02-21T23:09:52.736252+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"151","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs2_metadata","create_time":"2024-02-21T23:10:04.095451+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"154","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs2_data","create_time":"2024-02-21T23:10:04.836982+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"154","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:10:06.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.816+0000 7f28c1a00640 1 -- 172.21.15.28:0/803365694 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f289806b4d0 msgr2=0x7f289806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:06.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.816+0000 7f28c1a00640 1 --2- 172.21.15.28:0/803365694 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f289806b4d0 0x7f289806d990 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f28ac036000 tx=0x7f28ac008030 comp rx=0 tx=0).stop 2024-02-21T23:10:06.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.816+0000 7f28c1a00640 1 -- 172.21.15.28:0/803365694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28bc0aff90 msgr2=0x7f28bc0c1f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:06.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.816+0000 7f28c1a00640 1 --2- 172.21.15.28:0/803365694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28bc0aff90 0x7f28bc0c1f90 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7f28a400c400 tx=0x7f28a400c8c0 comp rx=0 tx=0).stop 2024-02-21T23:10:06.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.816+0000 7f28c1a00640 1 -- 172.21.15.28:0/803365694 shutdown_connections 2024-02-21T23:10:06.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.816+0000 7f28c1a00640 1 --2- 172.21.15.28:0/803365694 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f289806b4d0 0x7f289806d990 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:06.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.816+0000 7f28c1a00640 1 --2- 172.21.15.28:0/803365694 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f28bc0c5720 0x7f28bc14e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:06.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.816+0000 7f28c1a00640 1 --2- 172.21.15.28:0/803365694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28bc0aff90 0x7f28bc0c1f90 unknown :-1 s=CLOSED pgs=1726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:06.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.816+0000 7f28c1a00640 1 --2- 172.21.15.28:0/803365694 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f28bc06dbf0 0x7f28bc0ae740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:06.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.816+0000 7f28c1a00640 1 -- 172.21.15.28:0/803365694 >> 172.21.15.28:0/803365694 conn(0x7f28bc0ac370 msgr2=0x7f28bc0c8ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:06.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.816+0000 7f28c1a00640 1 -- 172.21.15.28:0/803365694 shutdown_connections 2024-02-21T23:10:06.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:06.816+0000 7f28c1a00640 1 -- 172.21.15.28:0/803365694 wait complete. 2024-02-21T23:10:06.824 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:10:07.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.050+0000 7fd941e00640 1 Processor -- start 2024-02-21T23:10:07.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.050+0000 7fd941e00640 1 -- start start 2024-02-21T23:10:07.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.051+0000 7fd941e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd93c150450 0x7fd93c0a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:07.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.052+0000 7fd941e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd93c0c2760 con 0x7fd93c156af0 2024-02-21T23:10:07.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.052+0000 7fd941e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd93c156310 con 0x7fd93c06ee40 2024-02-21T23:10:07.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.052+0000 7fd941e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd93c0c22c0 con 0x7fd93c150450 2024-02-21T23:10:07.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.052+0000 7fd940a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd93c150450 0x7fd93c0a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:07.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.052+0000 7fd941400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd93c156af0 0x7fd93c06dff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60870/0 (socket says 172.21.15.28:60870) 2024-02-21T23:10:07.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.052+0000 7fd941400640 1 -- 172.21.15.28:0/2712271271 learned_addr learned my addr 172.21.15.28:0/2712271271 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:10:07.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.052+0000 7fd940a00640 1 -- 172.21.15.28:0/2712271271 >> v1:172.21.15.81:6789/0 conn(0x7fd93c06ee40 legacy=0x7fd93c1536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:07.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.052+0000 7fd940a00640 1 -- 172.21.15.28:0/2712271271 >> v1:172.21.15.28:6789/0 conn(0x7fd93c156af0 legacy=0x7fd93c06dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:07.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.052+0000 7fd940a00640 1 -- 172.21.15.28:0/2712271271 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd93c0c29d0 con 0x7fd93c150450 2024-02-21T23:10:07.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.054+0000 7fd940a00640 1 --2- 172.21.15.28:0/2712271271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd93c150450 0x7fd93c0a5fd0 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7fd9300097a0 tx=0x7fd93002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0d17c74cf41a1fa server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:07.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.054+0000 7fd93b400640 1 -- 172.21.15.28:0/2712271271 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd930035020 con 0x7fd93c150450 2024-02-21T23:10:07.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.054+0000 7fd93b400640 1 -- 172.21.15.28:0/2712271271 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd930031830 con 0x7fd93c150450 2024-02-21T23:10:07.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.054+0000 7fd93b400640 1 -- 172.21.15.28:0/2712271271 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd930031c00 con 0x7fd93c150450 2024-02-21T23:10:07.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.055+0000 7fd941e00640 1 -- 172.21.15.28:0/2712271271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd93c150450 msgr2=0x7fd93c0a5fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:07.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.055+0000 7fd941e00640 1 --2- 172.21.15.28:0/2712271271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd93c150450 0x7fd93c0a5fd0 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7fd9300097a0 tx=0x7fd93002b8d0 comp rx=0 tx=0).stop 2024-02-21T23:10:07.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.055+0000 7fd941e00640 1 -- 172.21.15.28:0/2712271271 shutdown_connections 2024-02-21T23:10:07.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.055+0000 7fd941e00640 1 --2- 172.21.15.28:0/2712271271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd93c150450 0x7fd93c0a5fd0 unknown :-1 s=CLOSED pgs=1728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:07.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.055+0000 7fd941e00640 1 -- 172.21.15.28:0/2712271271 >> 172.21.15.28:0/2712271271 conn(0x7fd93c06b630 msgr2=0x7fd93c0c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:07.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.055+0000 7fd941e00640 1 -- 172.21.15.28:0/2712271271 shutdown_connections 2024-02-21T23:10:07.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.055+0000 7fd941e00640 1 -- 172.21.15.28:0/2712271271 wait complete. 2024-02-21T23:10:07.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.057+0000 7fd941e00640 1 Processor -- start 2024-02-21T23:10:07.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.057+0000 7fd941e00640 1 -- start start 2024-02-21T23:10:07.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.058+0000 7fd941e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd93c06ee40 0x7fd93c0c4b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:07.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.058+0000 7fd941e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd93c150450 0x7fd93c153e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:07.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.058+0000 7fd941e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd93c156af0 0x7fd93c1e1a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:07.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.058+0000 7fd941e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd93c06e700 con 0x7fd93c150450 2024-02-21T23:10:07.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.058+0000 7fd941e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd93c156660 con 0x7fd93c06ee40 2024-02-21T23:10:07.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.058+0000 7fd941e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd93c06e880 con 0x7fd93c156af0 2024-02-21T23:10:07.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.058+0000 7fd93be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd93c150450 0x7fd93c153e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:07.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.058+0000 7fd93be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd93c150450 0x7fd93c153e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35466/0 (socket says 172.21.15.28:35466) 2024-02-21T23:10:07.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.058+0000 7fd93be00640 1 -- 172.21.15.28:0/2202916160 learned_addr learned my addr 172.21.15.28:0/2202916160 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:07.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.058+0000 7fd940a00640 1 --2- 172.21.15.28:0/2202916160 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd93c06ee40 0x7fd93c0c4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:07.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.058+0000 7fd941400640 1 --2- 172.21.15.28:0/2202916160 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd93c156af0 0x7fd93c1e1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:07.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.058+0000 7fd93be00640 1 -- 172.21.15.28:0/2202916160 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd93c156af0 msgr2=0x7fd93c1e1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:07.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.058+0000 7fd93be00640 1 --2- 172.21.15.28:0/2202916160 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd93c156af0 0x7fd93c1e1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:07.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.058+0000 7fd93be00640 1 -- 172.21.15.28:0/2202916160 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd93c06ee40 msgr2=0x7fd93c0c4b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:07.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.058+0000 7fd93be00640 1 --2- 172.21.15.28:0/2202916160 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd93c06ee40 0x7fd93c0c4b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:07.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.058+0000 7fd93be00640 1 -- 172.21.15.28:0/2202916160 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd93c1ee860 con 0x7fd93c150450 2024-02-21T23:10:07.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.058+0000 7fd940a00640 1 --2- 172.21.15.28:0/2202916160 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd93c06ee40 0x7fd93c0c4b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:10:07.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.058+0000 7fd941400640 1 --2- 172.21.15.28:0/2202916160 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd93c156af0 0x7fd93c1e1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:10:07.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.059+0000 7fd93be00640 1 --2- 172.21.15.28:0/2202916160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd93c150450 0x7fd93c153e40 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7fd92800c430 tx=0x7fd92800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:07.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.059+0000 7fd938c00640 1 -- 172.21.15.28:0/2202916160 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd928011790 con 0x7fd93c150450 2024-02-21T23:10:07.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.059+0000 7fd938c00640 1 -- 172.21.15.28:0/2202916160 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd928017070 con 0x7fd93c150450 2024-02-21T23:10:07.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.059+0000 7fd941e00640 1 -- 172.21.15.28:0/2202916160 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd93c1eeb40 con 0x7fd93c150450 2024-02-21T23:10:07.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.059+0000 7fd938c00640 1 -- 172.21.15.28:0/2202916160 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd928011b70 con 0x7fd93c150450 2024-02-21T23:10:07.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.059+0000 7fd941e00640 1 -- 172.21.15.28:0/2202916160 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd93c072230 con 0x7fd93c150450 2024-02-21T23:10:07.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.060+0000 7fd941e00640 1 -- 172.21.15.28:0/2202916160 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd93c04f290 con 0x7fd93c150450 2024-02-21T23:10:07.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.061+0000 7fd938c00640 1 -- 172.21.15.28:0/2202916160 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd928011d10 con 0x7fd93c150450 2024-02-21T23:10:07.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.062+0000 7fd938c00640 1 --2- 172.21.15.28:0/2202916160 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd91c06b4d0 0x7fd91c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:07.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.062+0000 7fd938c00640 1 -- 172.21.15.28:0/2202916160 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 6957+0+0 (secure 0 0 0) 0x7fd92808a5a0 con 0x7fd93c150450 2024-02-21T23:10:07.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.063+0000 7fd940a00640 1 --2- 172.21.15.28:0/2202916160 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd91c06b4d0 0x7fd91c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:07.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.064+0000 7fd940a00640 1 --2- 172.21.15.28:0/2202916160 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd91c06b4d0 0x7fd91c06d990 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fd930036000 tx=0x7fd930008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:07.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.067+0000 7fd938c00640 1 -- 172.21.15.28:0/2202916160 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd928058f40 con 0x7fd93c150450 2024-02-21T23:10:07.192 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-21T23:10:07.200 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-21T23:10:07.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.332+0000 7fd941e00640 1 -- 172.21.15.28:0/2202916160 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd93c0c1f10 con 0x7fd93c150450 2024-02-21T23:10:07.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.334+0000 7fd938c00640 1 -- 172.21.15.28:0/2202916160 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v154) v1 ==== 74+0+20555 (secure 0 0 0) 0x7fd928058840 con 0x7fd93c150450 2024-02-21T23:10:07.324 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:10:07.324 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":154,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:10:05.843379+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:09:51.634291+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"150","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-02-21T23:09:52.736252+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"151","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs2_metadata","create_time":"2024-02-21T23:10:04.095451+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"154","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs2_data","create_time":"2024-02-21T23:10:04.836982+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"154","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:10:07.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.337+0000 7fd941e00640 1 -- 172.21.15.28:0/2202916160 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd91c06b4d0 msgr2=0x7fd91c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:07.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.337+0000 7fd941e00640 1 --2- 172.21.15.28:0/2202916160 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd91c06b4d0 0x7fd91c06d990 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fd930036000 tx=0x7fd930008030 comp rx=0 tx=0).stop 2024-02-21T23:10:07.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.337+0000 7fd941e00640 1 -- 172.21.15.28:0/2202916160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd93c150450 msgr2=0x7fd93c153e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:07.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.337+0000 7fd941e00640 1 --2- 172.21.15.28:0/2202916160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd93c150450 0x7fd93c153e40 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7fd92800c430 tx=0x7fd92800c8f0 comp rx=0 tx=0).stop 2024-02-21T23:10:07.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.337+0000 7fd941e00640 1 -- 172.21.15.28:0/2202916160 shutdown_connections 2024-02-21T23:10:07.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.337+0000 7fd941e00640 1 --2- 172.21.15.28:0/2202916160 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd91c06b4d0 0x7fd91c06d990 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:07.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.337+0000 7fd941e00640 1 --2- 172.21.15.28:0/2202916160 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd93c156af0 0x7fd93c1e1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:07.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.337+0000 7fd941e00640 1 --2- 172.21.15.28:0/2202916160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd93c150450 0x7fd93c153e40 unknown :-1 s=CLOSED pgs=1729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:07.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.337+0000 7fd941e00640 1 --2- 172.21.15.28:0/2202916160 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd93c06ee40 0x7fd93c0c4b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:07.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.337+0000 7fd941e00640 1 -- 172.21.15.28:0/2202916160 >> 172.21.15.28:0/2202916160 conn(0x7fd93c06b630 msgr2=0x7fd93c14e020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:07.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.337+0000 7fd941e00640 1 -- 172.21.15.28:0/2202916160 shutdown_connections 2024-02-21T23:10:07.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.337+0000 7fd941e00640 1 -- 172.21.15.28:0/2202916160 wait complete. 2024-02-21T23:10:07.344 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs2 standby_count_wanted 0 2024-02-21T23:10:07.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.589+0000 7f799da00640 1 Processor -- start 2024-02-21T23:10:07.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.589+0000 7f799da00640 1 -- start start 2024-02-21T23:10:07.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.591+0000 7f799da00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f799806dbc0 0x7f799806c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:07.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.591+0000 7f799da00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f799813e520 0x7f799806c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:07.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.591+0000 7f799da00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7998158550 con 0x7f799813e0c0 2024-02-21T23:10:07.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.591+0000 7f799da00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79980c89a0 con 0x7f799806dbc0 2024-02-21T23:10:07.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.591+0000 7f799da00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79981587d0 con 0x7f799813e520 2024-02-21T23:10:07.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.591+0000 7f798f400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f799806dbc0 0x7f799806c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:07.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.591+0000 7f798f400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f799806dbc0 0x7f799806c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35474/0 (socket says 172.21.15.28:35474) 2024-02-21T23:10:07.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.591+0000 7f798f400640 1 -- 172.21.15.28:0/3323166879 learned_addr learned my addr 172.21.15.28:0/3323166879 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:07.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.591+0000 7f799d000640 1 --2- 172.21.15.28:0/3323166879 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f799813e520 0x7f799806c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:07.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.592+0000 7f798f400640 1 -- 172.21.15.28:0/3323166879 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f799813e520 msgr2=0x7f799806c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:07.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.592+0000 7f798f400640 1 --2- 172.21.15.28:0/3323166879 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f799813e520 0x7f799806c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:07.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.592+0000 7f798f400640 1 -- 172.21.15.28:0/3323166879 >> v1:172.21.15.81:6789/0 conn(0x7f799813e0c0 legacy=0x7f79980c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:07.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.592+0000 7f798f400640 1 -- 172.21.15.28:0/3323166879 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7998158ab0 con 0x7f799806dbc0 2024-02-21T23:10:07.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.594+0000 7f798f400640 1 --2- 172.21.15.28:0/3323166879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f799806dbc0 0x7f799806c280 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7f7980009870 tx=0x7f798002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e57db638aa99535e server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:07.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.595+0000 7f7997400640 1 -- 172.21.15.28:0/3323166879 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7980035020 con 0x7f799806dbc0 2024-02-21T23:10:07.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.595+0000 7f7997400640 1 -- 172.21.15.28:0/3323166879 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7980031640 con 0x7f799806dbc0 2024-02-21T23:10:07.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.595+0000 7f7997400640 1 -- 172.21.15.28:0/3323166879 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f798002d040 con 0x7f799806dbc0 2024-02-21T23:10:07.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.595+0000 7f799da00640 1 -- 172.21.15.28:0/3323166879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f799806dbc0 msgr2=0x7f799806c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:07.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.595+0000 7f799da00640 1 --2- 172.21.15.28:0/3323166879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f799806dbc0 0x7f799806c280 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7f7980009870 tx=0x7f798002b8f0 comp rx=0 tx=0).stop 2024-02-21T23:10:07.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.595+0000 7f799da00640 1 -- 172.21.15.28:0/3323166879 shutdown_connections 2024-02-21T23:10:07.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.595+0000 7f799da00640 1 --2- 172.21.15.28:0/3323166879 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f799813e520 0x7f799806c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:07.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.595+0000 7f799da00640 1 --2- 172.21.15.28:0/3323166879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f799806dbc0 0x7f799806c280 unknown :-1 s=CLOSED pgs=1730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:07.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.595+0000 7f799da00640 1 -- 172.21.15.28:0/3323166879 >> 172.21.15.28:0/3323166879 conn(0x7f7998073110 msgr2=0x7f79980ac490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:07.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.596+0000 7f799da00640 1 -- 172.21.15.28:0/3323166879 shutdown_connections 2024-02-21T23:10:07.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.596+0000 7f799da00640 1 -- 172.21.15.28:0/3323166879 wait complete. 2024-02-21T23:10:07.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.598+0000 7f799da00640 1 Processor -- start 2024-02-21T23:10:07.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.598+0000 7f799da00640 1 -- start start 2024-02-21T23:10:07.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.599+0000 7f799da00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f799806dbc0 0x7f79980abca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:07.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.599+0000 7f799da00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f799813e0c0 0x7f79980ad8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:07.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.599+0000 7f799da00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f799813e520 0x7f7998152e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:07.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.599+0000 7f799da00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79980c78c0 con 0x7f799813e520 2024-02-21T23:10:07.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.599+0000 7f799da00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7998159070 con 0x7f799813e0c0 2024-02-21T23:10:07.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.599+0000 7f799da00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79980c7a40 con 0x7f799806dbc0 2024-02-21T23:10:07.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.599+0000 7f799d000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f799813e520 0x7f7998152e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:07.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.600+0000 7f799d000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f799813e520 0x7f7998152e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35480/0 (socket says 172.21.15.28:35480) 2024-02-21T23:10:07.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.600+0000 7f799d000640 1 -- 172.21.15.28:0/54590637 learned_addr learned my addr 172.21.15.28:0/54590637 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:07.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.600+0000 7f7997e00640 1 --2- 172.21.15.28:0/54590637 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f799806dbc0 0x7f79980abca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:07.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.600+0000 7f798f400640 1 --2- 172.21.15.28:0/54590637 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f799813e0c0 0x7f79980ad8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:07.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.600+0000 7f799d000640 1 -- 172.21.15.28:0/54590637 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f799806dbc0 msgr2=0x7f79980abca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:07.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.600+0000 7f799d000640 1 --2- 172.21.15.28:0/54590637 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f799806dbc0 0x7f79980abca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:07.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.600+0000 7f799d000640 1 -- 172.21.15.28:0/54590637 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f799813e0c0 msgr2=0x7f79980ad8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:07.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.600+0000 7f799d000640 1 --2- 172.21.15.28:0/54590637 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f799813e0c0 0x7f79980ad8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:07.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.600+0000 7f799d000640 1 -- 172.21.15.28:0/54590637 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7998155bf0 con 0x7f799813e520 2024-02-21T23:10:07.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.600+0000 7f798f400640 1 --2- 172.21.15.28:0/54590637 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f799813e0c0 0x7f79980ad8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:10:07.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.601+0000 7f799d000640 1 --2- 172.21.15.28:0/54590637 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f799813e520 0x7f7998152e80 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7f7990007e40 tx=0x7f799000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:07.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.601+0000 7f7994c00640 1 -- 172.21.15.28:0/54590637 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7990014620 con 0x7f799813e520 2024-02-21T23:10:07.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.601+0000 7f7994c00640 1 -- 172.21.15.28:0/54590637 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7990019070 con 0x7f799813e520 2024-02-21T23:10:07.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.601+0000 7f799da00640 1 -- 172.21.15.28:0/54590637 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79981ee820 con 0x7f799813e520 2024-02-21T23:10:07.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.601+0000 7f7994c00640 1 -- 172.21.15.28:0/54590637 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7990014a20 con 0x7f799813e520 2024-02-21T23:10:07.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.601+0000 7f799da00640 1 -- 172.21.15.28:0/54590637 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79980b0350 con 0x7f799813e520 2024-02-21T23:10:07.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.603+0000 7f799da00640 1 -- 172.21.15.28:0/54590637 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f799804f290 con 0x7f799813e520 2024-02-21T23:10:07.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.604+0000 7f7994c00640 1 -- 172.21.15.28:0/54590637 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7990014bc0 con 0x7f799813e520 2024-02-21T23:10:07.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.606+0000 7f7994c00640 1 --2- 172.21.15.28:0/54590637 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f797406b4d0 0x7f797406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:07.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.606+0000 7f7994c00640 1 -- 172.21.15.28:0/54590637 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f799008ae90 con 0x7f799813e520 2024-02-21T23:10:07.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.608+0000 7f7997e00640 1 --2- 172.21.15.28:0/54590637 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f797406b4d0 0x7f797406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:07.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.610+0000 7f7997e00640 1 --2- 172.21.15.28:0/54590637 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f797406b4d0 0x7f797406d990 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f7998152790 tx=0x7f7988009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:07.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.610+0000 7f7994c00640 1 -- 172.21.15.28:0/54590637 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f79900598c0 con 0x7f799813e520 2024-02-21T23:10:07.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:07.855+0000 7f799da00640 1 -- 172.21.15.28:0/54590637 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs2", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f799806d9a0 con 0x7f799813e520 2024-02-21T23:10:08.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.383+0000 7f7994c00640 1 -- 172.21.15.28:0/54590637 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs2", "var": "standby_count_wanted", "val": "0"}]=0 v162) v1 ==== 119+0+0 (secure 0 0 0) 0x7f79900591c0 con 0x7f799813e520 2024-02-21T23:10:08.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.387+0000 7f799da00640 1 -- 172.21.15.28:0/54590637 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f797406b4d0 msgr2=0x7f797406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:08.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.387+0000 7f799da00640 1 --2- 172.21.15.28:0/54590637 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f797406b4d0 0x7f797406d990 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f7998152790 tx=0x7f7988009040 comp rx=0 tx=0).stop 2024-02-21T23:10:08.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.387+0000 7f799da00640 1 -- 172.21.15.28:0/54590637 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f799813e520 msgr2=0x7f7998152e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:08.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.387+0000 7f799da00640 1 --2- 172.21.15.28:0/54590637 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f799813e520 0x7f7998152e80 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7f7990007e40 tx=0x7f799000e4b0 comp rx=0 tx=0).stop 2024-02-21T23:10:08.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.387+0000 7f799da00640 1 -- 172.21.15.28:0/54590637 shutdown_connections 2024-02-21T23:10:08.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.387+0000 7f799da00640 1 --2- 172.21.15.28:0/54590637 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f797406b4d0 0x7f797406d990 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:08.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.387+0000 7f799da00640 1 --2- 172.21.15.28:0/54590637 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f799813e520 0x7f7998152e80 unknown :-1 s=CLOSED pgs=1731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:08.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.387+0000 7f799da00640 1 --2- 172.21.15.28:0/54590637 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f799813e0c0 0x7f79980ad8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:08.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.387+0000 7f799da00640 1 --2- 172.21.15.28:0/54590637 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f799806dbc0 0x7f79980abca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:08.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.387+0000 7f799da00640 1 -- 172.21.15.28:0/54590637 >> 172.21.15.28:0/54590637 conn(0x7f7998073110 msgr2=0x7f799806c750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:08.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.387+0000 7f799da00640 1 -- 172.21.15.28:0/54590637 shutdown_connections 2024-02-21T23:10:08.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.387+0000 7f799da00640 1 -- 172.21.15.28:0/54590637 wait complete. 2024-02-21T23:10:08.403 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-02-21T23:10:08.403 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs2 allow_standby_replay false 2024-02-21T23:10:08.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.629+0000 7fb3e2400640 1 Processor -- start 2024-02-21T23:10:08.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.629+0000 7fb3e2400640 1 -- start start 2024-02-21T23:10:08.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.632+0000 7fb3e2400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fb3dc150780 0x7fb3dc153b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:08.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.632+0000 7fb3e2400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb3dc0a9e90 0x7fb3dc0ac4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:08.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.632+0000 7fb3e2400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3dc06f750 con 0x7fb3dc072620 2024-02-21T23:10:08.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.632+0000 7fb3e2400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3dc06f500 con 0x7fb3dc0a9e90 2024-02-21T23:10:08.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.632+0000 7fb3e2400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3dc06c500 con 0x7fb3dc150780 2024-02-21T23:10:08.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.633+0000 7fb3daa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb3dc0a9e90 0x7fb3dc0ac4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:08.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.633+0000 7fb3daa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb3dc0a9e90 0x7fb3dc0ac4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35496/0 (socket says 172.21.15.28:35496) 2024-02-21T23:10:08.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.633+0000 7fb3daa00640 1 -- 172.21.15.28:0/2665355347 learned_addr learned my addr 172.21.15.28:0/2665355347 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:08.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.633+0000 7fb3db400640 1 --2- 172.21.15.28:0/2665355347 >> v2:172.21.15.81:3300/0 conn(0x7fb3dc150780 0x7fb3dc153b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:08.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.633+0000 7fb3daa00640 1 -- 172.21.15.28:0/2665355347 >> v2:172.21.15.81:3300/0 conn(0x7fb3dc150780 msgr2=0x7fb3dc153b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:08.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.633+0000 7fb3daa00640 1 --2- 172.21.15.28:0/2665355347 >> v2:172.21.15.81:3300/0 conn(0x7fb3dc150780 0x7fb3dc153b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:08.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.633+0000 7fb3daa00640 1 -- 172.21.15.28:0/2665355347 >> v1:172.21.15.81:6789/0 conn(0x7fb3dc072620 legacy=0x7fb3dc1565d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:08.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.634+0000 7fb3daa00640 1 -- 172.21.15.28:0/2665355347 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3dc0c3410 con 0x7fb3dc0a9e90 2024-02-21T23:10:08.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.635+0000 7fb3daa00640 1 --2- 172.21.15.28:0/2665355347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3dc0a9e90 0x7fb3dc0ac4e0 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7fb3dc067c00 tx=0x7fb3c802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=769a96d2e3df53e5 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:08.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.635+0000 7fb3da000640 1 -- 172.21.15.28:0/2665355347 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c802e070 con 0x7fb3dc0a9e90 2024-02-21T23:10:08.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.635+0000 7fb3da000640 1 -- 172.21.15.28:0/2665355347 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3c8037070 con 0x7fb3dc0a9e90 2024-02-21T23:10:08.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.635+0000 7fb3da000640 1 -- 172.21.15.28:0/2665355347 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c8031a00 con 0x7fb3dc0a9e90 2024-02-21T23:10:08.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.636+0000 7fb3e2400640 1 -- 172.21.15.28:0/2665355347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3dc0a9e90 msgr2=0x7fb3dc0ac4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:08.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.636+0000 7fb3e2400640 1 --2- 172.21.15.28:0/2665355347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3dc0a9e90 0x7fb3dc0ac4e0 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7fb3dc067c00 tx=0x7fb3c802b880 comp rx=0 tx=0).stop 2024-02-21T23:10:08.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.636+0000 7fb3e2400640 1 -- 172.21.15.28:0/2665355347 shutdown_connections 2024-02-21T23:10:08.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.636+0000 7fb3e2400640 1 --2- 172.21.15.28:0/2665355347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3dc0a9e90 0x7fb3dc0ac4e0 unknown :-1 s=CLOSED pgs=1732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:08.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.636+0000 7fb3e2400640 1 --2- 172.21.15.28:0/2665355347 >> v2:172.21.15.81:3300/0 conn(0x7fb3dc150780 0x7fb3dc153b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:08.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.636+0000 7fb3e2400640 1 -- 172.21.15.28:0/2665355347 >> 172.21.15.28:0/2665355347 conn(0x7fb3dc071b00 msgr2=0x7fb3dc071f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:08.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.636+0000 7fb3e2400640 1 -- 172.21.15.28:0/2665355347 shutdown_connections 2024-02-21T23:10:08.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.636+0000 7fb3e2400640 1 -- 172.21.15.28:0/2665355347 wait complete. 2024-02-21T23:10:08.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.638+0000 7fb3e2400640 1 Processor -- start 2024-02-21T23:10:08.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.638+0000 7fb3e2400640 1 -- start start 2024-02-21T23:10:08.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.639+0000 7fb3e2400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3dc072620 0x7fb3dc1ee190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:08.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.639+0000 7fb3e2400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb3dc0a9e90 0x7fb3dc144b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:08.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.639+0000 7fb3e2400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb3dc150780 0x7fb3dc1450a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:08.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.639+0000 7fb3e2400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3dc0aca20 con 0x7fb3dc072620 2024-02-21T23:10:08.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.639+0000 7fb3e2400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3dc0c1650 con 0x7fb3dc0a9e90 2024-02-21T23:10:08.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.639+0000 7fb3e2400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3dc0a8ec0 con 0x7fb3dc150780 2024-02-21T23:10:08.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.639+0000 7fb3db400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3dc072620 0x7fb3dc1ee190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:08.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.639+0000 7fb3db400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3dc072620 0x7fb3dc1ee190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35500/0 (socket says 172.21.15.28:35500) 2024-02-21T23:10:08.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.639+0000 7fb3db400640 1 -- 172.21.15.28:0/1321285058 learned_addr learned my addr 172.21.15.28:0/1321285058 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:08.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.640+0000 7fb3daa00640 1 --2- 172.21.15.28:0/1321285058 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb3dc0a9e90 0x7fb3dc144b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:08.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.640+0000 7fb3dbe00640 1 --2- 172.21.15.28:0/1321285058 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb3dc150780 0x7fb3dc1450a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:08.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.640+0000 7fb3db400640 1 -- 172.21.15.28:0/1321285058 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb3dc150780 msgr2=0x7fb3dc1450a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:08.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.640+0000 7fb3db400640 1 --2- 172.21.15.28:0/1321285058 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb3dc150780 0x7fb3dc1450a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:08.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.640+0000 7fb3db400640 1 -- 172.21.15.28:0/1321285058 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb3dc0a9e90 msgr2=0x7fb3dc144b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:08.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.640+0000 7fb3db400640 1 --2- 172.21.15.28:0/1321285058 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb3dc0a9e90 0x7fb3dc144b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:08.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.640+0000 7fb3db400640 1 -- 172.21.15.28:0/1321285058 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3dc1455e0 con 0x7fb3dc072620 2024-02-21T23:10:08.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.640+0000 7fb3db400640 1 --2- 172.21.15.28:0/1321285058 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3dc072620 0x7fb3dc1ee190 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc00d620 tx=0x7fb3cc00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:08.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.641+0000 7fb3d3400640 1 -- 172.21.15.28:0/1321285058 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3cc00b620 con 0x7fb3dc072620 2024-02-21T23:10:08.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.641+0000 7fb3d3400640 1 -- 172.21.15.28:0/1321285058 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3cc019070 con 0x7fb3dc072620 2024-02-21T23:10:08.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.641+0000 7fb3e2400640 1 -- 172.21.15.28:0/1321285058 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3dc145970 con 0x7fb3dc072620 2024-02-21T23:10:08.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.641+0000 7fb3d3400640 1 -- 172.21.15.28:0/1321285058 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3cc00ba00 con 0x7fb3dc072620 2024-02-21T23:10:08.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.641+0000 7fb3e2400640 1 -- 172.21.15.28:0/1321285058 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3dc14f410 con 0x7fb3dc072620 2024-02-21T23:10:08.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.642+0000 7fb3e2400640 1 -- 172.21.15.28:0/1321285058 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3dc068640 con 0x7fb3dc072620 2024-02-21T23:10:08.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.643+0000 7fb3d3400640 1 -- 172.21.15.28:0/1321285058 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb3cc00bba0 con 0x7fb3dc072620 2024-02-21T23:10:08.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.645+0000 7fb3d3400640 1 --2- 172.21.15.28:0/1321285058 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb3bc06b4d0 0x7fb3bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:08.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.645+0000 7fb3d3400640 1 -- 172.21.15.28:0/1321285058 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 6957+0+0 (secure 0 0 0) 0x7fb3cc08ab30 con 0x7fb3dc072620 2024-02-21T23:10:08.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.646+0000 7fb3daa00640 1 --2- 172.21.15.28:0/1321285058 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb3bc06b4d0 0x7fb3bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:08.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.647+0000 7fb3daa00640 1 --2- 172.21.15.28:0/1321285058 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb3bc06b4d0 0x7fb3bc06d990 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fb3c8036000 tx=0x7fb3c803d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:08.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.649+0000 7fb3d3400640 1 -- 172.21.15.28:0/1321285058 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb3cc0594c0 con 0x7fb3dc072620 2024-02-21T23:10:08.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:08.925+0000 7fb3e2400640 1 -- 172.21.15.28:0/1321285058 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs2", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fb3dc067130 con 0x7fb3dc072620 2024-02-21T23:10:09.199 INFO:tasks.ceph.mds.c.smithi028.stderr:2024-02-21T23:10:09.210+0000 7f81e2a00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-21T23:10:09.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.395+0000 7fb3d3400640 1 -- 172.21.15.28:0/1321285058 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs2", "var": "allow_standby_replay", "val": "false"}]=0 v163) v1 ==== 123+0+0 (secure 0 0 0) 0x7fb3cc058dc0 con 0x7fb3dc072620 2024-02-21T23:10:09.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.399+0000 7fb3e2400640 1 -- 172.21.15.28:0/1321285058 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb3bc06b4d0 msgr2=0x7fb3bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:09.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.399+0000 7fb3e2400640 1 --2- 172.21.15.28:0/1321285058 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb3bc06b4d0 0x7fb3bc06d990 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fb3c8036000 tx=0x7fb3c803d040 comp rx=0 tx=0).stop 2024-02-21T23:10:09.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.399+0000 7fb3e2400640 1 -- 172.21.15.28:0/1321285058 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3dc072620 msgr2=0x7fb3dc1ee190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:09.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.399+0000 7fb3e2400640 1 --2- 172.21.15.28:0/1321285058 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3dc072620 0x7fb3dc1ee190 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc00d620 tx=0x7fb3cc00dae0 comp rx=0 tx=0).stop 2024-02-21T23:10:09.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.400+0000 7fb3e2400640 1 -- 172.21.15.28:0/1321285058 shutdown_connections 2024-02-21T23:10:09.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.400+0000 7fb3e2400640 1 --2- 172.21.15.28:0/1321285058 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb3bc06b4d0 0x7fb3bc06d990 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:09.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.400+0000 7fb3e2400640 1 --2- 172.21.15.28:0/1321285058 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb3dc150780 0x7fb3dc1450a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:09.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.400+0000 7fb3e2400640 1 --2- 172.21.15.28:0/1321285058 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb3dc0a9e90 0x7fb3dc144b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:09.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.400+0000 7fb3e2400640 1 --2- 172.21.15.28:0/1321285058 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3dc072620 0x7fb3dc1ee190 unknown :-1 s=CLOSED pgs=1733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:09.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.400+0000 7fb3e2400640 1 -- 172.21.15.28:0/1321285058 >> 172.21.15.28:0/1321285058 conn(0x7fb3dc071b00 msgr2=0x7fb3dc14dfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:09.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.400+0000 7fb3e2400640 1 -- 172.21.15.28:0/1321285058 shutdown_connections 2024-02-21T23:10:09.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.400+0000 7fb3e2400640 1 -- 172.21.15.28:0/1321285058 wait complete. 2024-02-21T23:10:09.409 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs2' 2024-02-21T23:10:09.409 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:10:09.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.637+0000 7f4331c00640 1 Processor -- start 2024-02-21T23:10:09.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.637+0000 7f4331c00640 1 -- start start 2024-02-21T23:10:09.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.639+0000 7f4331c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f432c06ee40 0x7f432c0a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:09.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.639+0000 7f4331c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f432c156af0 0x7f432c06dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:09.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.639+0000 7f4331c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f432c0c2760 con 0x7f432c150450 2024-02-21T23:10:09.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.639+0000 7f4331c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f432c0c22c0 con 0x7f432c156af0 2024-02-21T23:10:09.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.639+0000 7f4331c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f432c156660 con 0x7f432c06ee40 2024-02-21T23:10:09.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.640+0000 7f432be00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f432c150450 0x7f432c1536f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60878/0 (socket says 172.21.15.28:60878) 2024-02-21T23:10:09.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.640+0000 7f432be00640 1 -- 172.21.15.28:0/2664206799 learned_addr learned my addr 172.21.15.28:0/2664206799 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:10:09.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.640+0000 7f4331200640 1 --2- 172.21.15.28:0/2664206799 >> v2:172.21.15.28:3300/0 conn(0x7f432c156af0 0x7f432c06dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:09.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.640+0000 7f432b400640 1 --2- 172.21.15.28:0/2664206799 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f432c06ee40 0x7f432c0a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:09.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.640+0000 7f432aa00640 1 -- 172.21.15.28:0/2664206799 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2623387531 0 0) 0x7f432c0c2760 con 0x7f432c150450 2024-02-21T23:10:09.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.640+0000 7f432aa00640 1 -- 172.21.15.28:0/2664206799 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4310003520 con 0x7f432c150450 2024-02-21T23:10:09.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.641+0000 7f4331200640 1 -- 172.21.15.28:0/2664206799 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f432c06ee40 msgr2=0x7f432c0a5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:09.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.641+0000 7f4331200640 1 --2- 172.21.15.28:0/2664206799 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f432c06ee40 0x7f432c0a5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:09.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.641+0000 7f4331200640 1 -- 172.21.15.28:0/2664206799 >> v1:172.21.15.28:6789/0 conn(0x7f432c150450 legacy=0x7f432c1536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:09.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.641+0000 7f4331200640 1 -- 172.21.15.28:0/2664206799 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f432c156310 con 0x7f432c156af0 2024-02-21T23:10:09.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.642+0000 7f4331200640 1 --2- 172.21.15.28:0/2664206799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f432c156af0 0x7f432c06dff0 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7f4324009d90 tx=0x7f432402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=115af94891dc7be9 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:09.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.643+0000 7f432aa00640 1 -- 172.21.15.28:0/2664206799 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4324039070 con 0x7f432c156af0 2024-02-21T23:10:09.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.643+0000 7f432aa00640 1 -- 172.21.15.28:0/2664206799 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f432402cc50 con 0x7f432c156af0 2024-02-21T23:10:09.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.643+0000 7f432aa00640 1 -- 172.21.15.28:0/2664206799 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4324034440 con 0x7f432c156af0 2024-02-21T23:10:09.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.643+0000 7f4331c00640 1 -- 172.21.15.28:0/2664206799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f432c156af0 msgr2=0x7f432c06dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:09.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.643+0000 7f4331c00640 1 --2- 172.21.15.28:0/2664206799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f432c156af0 0x7f432c06dff0 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7f4324009d90 tx=0x7f432402bea0 comp rx=0 tx=0).stop 2024-02-21T23:10:09.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.643+0000 7f4331c00640 1 -- 172.21.15.28:0/2664206799 shutdown_connections 2024-02-21T23:10:09.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.643+0000 7f4331c00640 1 --2- 172.21.15.28:0/2664206799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f432c156af0 0x7f432c06dff0 unknown :-1 s=CLOSED pgs=1735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:09.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.643+0000 7f4331c00640 1 --2- 172.21.15.28:0/2664206799 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f432c06ee40 0x7f432c0a5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:09.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.643+0000 7f4331c00640 1 -- 172.21.15.28:0/2664206799 >> 172.21.15.28:0/2664206799 conn(0x7f432c06b630 msgr2=0x7f432c0c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:09.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.644+0000 7f4331c00640 1 -- 172.21.15.28:0/2664206799 shutdown_connections 2024-02-21T23:10:09.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.644+0000 7f4331c00640 1 -- 172.21.15.28:0/2664206799 wait complete. 2024-02-21T23:10:09.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.645+0000 7f4331c00640 1 Processor -- start 2024-02-21T23:10:09.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.645+0000 7f4331c00640 1 -- start start 2024-02-21T23:10:09.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.646+0000 7f4331c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f432c06ee40 0x7f432c155520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:09.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.646+0000 7f4331c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f432c150450 0x7f432c1533b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:09.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.646+0000 7f4331c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f432c156af0 0x7f432c0c4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:09.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.646+0000 7f4331c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f432c156f80 con 0x7f432c06ee40 2024-02-21T23:10:09.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.646+0000 7f4331c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f432c0c2cb0 con 0x7f432c156af0 2024-02-21T23:10:09.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.646+0000 7f4331c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f432c157100 con 0x7f432c150450 2024-02-21T23:10:09.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.647+0000 7f432be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f432c06ee40 0x7f432c155520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:09.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.647+0000 7f432be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f432c06ee40 0x7f432c155520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35522/0 (socket says 172.21.15.28:35522) 2024-02-21T23:10:09.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.647+0000 7f432be00640 1 -- 172.21.15.28:0/995049164 learned_addr learned my addr 172.21.15.28:0/995049164 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:09.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.647+0000 7f432b400640 1 --2- 172.21.15.28:0/995049164 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f432c150450 0x7f432c1533b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:09.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.647+0000 7f432be00640 1 -- 172.21.15.28:0/995049164 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f432c150450 msgr2=0x7f432c1533b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:09.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.647+0000 7f432be00640 1 --2- 172.21.15.28:0/995049164 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f432c150450 0x7f432c1533b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:09.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.647+0000 7f432be00640 1 -- 172.21.15.28:0/995049164 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f432c156af0 msgr2=0x7f432c0c4ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:09.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.647+0000 7f432be00640 1 --2- 172.21.15.28:0/995049164 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f432c156af0 0x7f432c0c4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:09.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.647+0000 7f432be00640 1 -- 172.21.15.28:0/995049164 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f432c155e40 con 0x7f432c06ee40 2024-02-21T23:10:09.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.647+0000 7f432be00640 1 --2- 172.21.15.28:0/995049164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f432c06ee40 0x7f432c155520 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7f431c00c430 tx=0x7f431c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:09.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.648+0000 7f4323e00640 1 -- 172.21.15.28:0/995049164 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f431c017070 con 0x7f432c06ee40 2024-02-21T23:10:09.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.648+0000 7f4323e00640 1 -- 172.21.15.28:0/995049164 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f431c005ce0 con 0x7f432c06ee40 2024-02-21T23:10:09.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.648+0000 7f4323e00640 1 -- 172.21.15.28:0/995049164 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f431c0072f0 con 0x7f432c06ee40 2024-02-21T23:10:09.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.648+0000 7f4331c00640 1 -- 172.21.15.28:0/995049164 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f432c153ad0 con 0x7f432c06ee40 2024-02-21T23:10:09.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.648+0000 7f4331c00640 1 -- 172.21.15.28:0/995049164 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f432c151b00 con 0x7f432c06ee40 2024-02-21T23:10:09.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.650+0000 7f4331c00640 1 -- 172.21.15.28:0/995049164 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f432c04f290 con 0x7f432c06ee40 2024-02-21T23:10:09.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.650+0000 7f4323e00640 1 -- 172.21.15.28:0/995049164 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f431c01a070 con 0x7f432c06ee40 2024-02-21T23:10:09.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.652+0000 7f4323e00640 1 --2- 172.21.15.28:0/995049164 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f430c06b4d0 0x7f430c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:09.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.652+0000 7f4323e00640 1 -- 172.21.15.28:0/995049164 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f431c089bc0 con 0x7f432c06ee40 2024-02-21T23:10:09.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.652+0000 7f432b400640 1 --2- 172.21.15.28:0/995049164 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f430c06b4d0 0x7f430c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:09.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.653+0000 7f432b400640 1 --2- 172.21.15.28:0/995049164 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f430c06b4d0 0x7f430c06d990 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f432c0c6140 tx=0x7f4318008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:09.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.655+0000 7f4323e00640 1 -- 172.21.15.28:0/995049164 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f431c058560 con 0x7f432c06ee40 2024-02-21T23:10:09.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.919+0000 7f4331c00640 1 -- 172.21.15.28:0/995049164 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f432c0c6290 con 0x7f432c06ee40 2024-02-21T23:10:09.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.920+0000 7f4323e00640 1 -- 172.21.15.28:0/995049164 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 163 v163) v1 ==== 95+0+6191 (secure 0 0 0) 0x7f431c057e60 con 0x7f432c06ee40 2024-02-21T23:10:09.909 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:10:09.909 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":163,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6389,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1320495906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1320495906},{"type":"v1","addr":"172.21.15.81:6835","nonce":1320495906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":153},{"gid":6392,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/521625945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":521625945},{"type":"v1","addr":"172.21.15.81:6837","nonce":521625945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":153}],"filesystems":[{"mdsmap":{"epoch":158,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:09:53.730296+0000","modified":"2024-02-21T23:09:56.809990+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8826},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8826":{"gid":8826,"name":"a","rank":0,"incarnation":154,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/2928639298","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2928639298},{"type":"v1","addr":"172.21.15.28:6835","nonce":2928639298}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11},{"mdsmap":{"epoch":163,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:10:05.843850+0000","modified":"2024-02-21T23:10:08.927829+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8817},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8817":{"gid":8817,"name":"c","rank":0,"incarnation":159,"state":"up:active","state_seq":6,"addr":"172.21.15.28:6833/2975378719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2975378719},{"type":"v1","addr":"172.21.15.28:6833","nonce":2975378719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2024-02-21T23:10:09.909 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 163 2024-02-21T23:10:09.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.923+0000 7f4331c00640 1 -- 172.21.15.28:0/995049164 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f430c06b4d0 msgr2=0x7f430c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:09.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.923+0000 7f4331c00640 1 --2- 172.21.15.28:0/995049164 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f430c06b4d0 0x7f430c06d990 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f432c0c6140 tx=0x7f4318008040 comp rx=0 tx=0).stop 2024-02-21T23:10:09.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.923+0000 7f4331c00640 1 -- 172.21.15.28:0/995049164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f432c06ee40 msgr2=0x7f432c155520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:09.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.923+0000 7f4331c00640 1 --2- 172.21.15.28:0/995049164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f432c06ee40 0x7f432c155520 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7f431c00c430 tx=0x7f431c00c8f0 comp rx=0 tx=0).stop 2024-02-21T23:10:09.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.923+0000 7f4331c00640 1 -- 172.21.15.28:0/995049164 shutdown_connections 2024-02-21T23:10:09.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.923+0000 7f4331c00640 1 --2- 172.21.15.28:0/995049164 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f430c06b4d0 0x7f430c06d990 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:09.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.923+0000 7f4331c00640 1 --2- 172.21.15.28:0/995049164 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f432c150450 0x7f432c1533b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:09.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.923+0000 7f4331c00640 1 --2- 172.21.15.28:0/995049164 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f432c156af0 0x7f432c0c4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:09.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.923+0000 7f4331c00640 1 --2- 172.21.15.28:0/995049164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f432c06ee40 0x7f432c155520 unknown :-1 s=CLOSED pgs=1736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:09.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.923+0000 7f4331c00640 1 -- 172.21.15.28:0/995049164 >> 172.21.15.28:0/995049164 conn(0x7f432c06b630 msgr2=0x7f432c0c2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:09.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.923+0000 7f4331c00640 1 -- 172.21.15.28:0/995049164 shutdown_connections 2024-02-21T23:10:09.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:09.923+0000 7f4331c00640 1 -- 172.21.15.28:0/995049164 wait complete. 2024-02-21T23:10:09.932 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:10:10.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.147+0000 7f788de00640 1 Processor -- start 2024-02-21T23:10:10.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.147+0000 7f788de00640 1 -- start start 2024-02-21T23:10:10.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.149+0000 7f788de00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7888152500 0x7f78881558c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:10.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.149+0000 7f788de00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f78880ca050 0x7f78880bf6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:10.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.149+0000 7f788de00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78881528e0 con 0x7f78880bfc00 2024-02-21T23:10:10.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.149+0000 7f788de00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78880c86e0 con 0x7f78880ca050 2024-02-21T23:10:10.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.149+0000 7f788de00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78880c8980 con 0x7f7888152500 2024-02-21T23:10:10.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.150+0000 7f788ca00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7888152500 0x7f78881558c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:10.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.150+0000 7f7887e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f78880ca050 0x7f78880bf6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:10.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.150+0000 7f788ca00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7888152500 0x7f78881558c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3301/0 says I am v2:172.21.15.28:47554/0 (socket says 172.21.15.28:47554) 2024-02-21T23:10:10.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.150+0000 7f788ca00640 1 -- 172.21.15.28:0/994002947 learned_addr learned my addr 172.21.15.28:0/994002947 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:10.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.151+0000 7f788ca00640 1 -- 172.21.15.28:0/994002947 >> v2:172.21.15.81:3300/0 conn(0x7f78880ca050 msgr2=0x7f78880bf6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:10.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.151+0000 7f788ca00640 1 --2- 172.21.15.28:0/994002947 >> v2:172.21.15.81:3300/0 conn(0x7f78880ca050 0x7f78880bf6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:10.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.151+0000 7f788ca00640 1 -- 172.21.15.28:0/994002947 >> v1:172.21.15.81:6789/0 conn(0x7f78880bfc00 legacy=0x7f788806bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:10.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.151+0000 7f788ca00640 1 -- 172.21.15.28:0/994002947 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78880c8c60 con 0x7f7888152500 2024-02-21T23:10:10.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.152+0000 7f788ca00640 1 --2- 172.21.15.28:0/994002947 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7888152500 0x7f78881558c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f787c009870 tx=0x7f787c02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=127abaeaa5205be4 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:10.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.152+0000 7f7887400640 1 -- 172.21.15.28:0/994002947 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f787c035020 con 0x7f7888152500 2024-02-21T23:10:10.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.152+0000 7f7887400640 1 -- 172.21.15.28:0/994002947 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f787c031640 con 0x7f7888152500 2024-02-21T23:10:10.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.152+0000 7f7887400640 1 -- 172.21.15.28:0/994002947 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f787c02d040 con 0x7f7888152500 2024-02-21T23:10:10.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.153+0000 7f788de00640 1 -- 172.21.15.28:0/994002947 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7888152500 msgr2=0x7f78881558c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:10.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.153+0000 7f788de00640 1 --2- 172.21.15.28:0/994002947 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7888152500 0x7f78881558c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f787c009870 tx=0x7f787c02b8f0 comp rx=0 tx=0).stop 2024-02-21T23:10:10.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.153+0000 7f788de00640 1 -- 172.21.15.28:0/994002947 shutdown_connections 2024-02-21T23:10:10.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.153+0000 7f788de00640 1 --2- 172.21.15.28:0/994002947 >> v2:172.21.15.81:3300/0 conn(0x7f78880ca050 0x7f78880bf6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:10.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.153+0000 7f788de00640 1 --2- 172.21.15.28:0/994002947 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7888152500 0x7f78881558c0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:10.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.153+0000 7f788de00640 1 -- 172.21.15.28:0/994002947 >> 172.21.15.28:0/994002947 conn(0x7f78880ac450 msgr2=0x7f78880ac860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:10.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.153+0000 7f788de00640 1 -- 172.21.15.28:0/994002947 shutdown_connections 2024-02-21T23:10:10.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.153+0000 7f788de00640 1 -- 172.21.15.28:0/994002947 wait complete. 2024-02-21T23:10:10.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.155+0000 7f788de00640 1 Processor -- start 2024-02-21T23:10:10.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.155+0000 7f788de00640 1 -- start start 2024-02-21T23:10:10.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.156+0000 7f788de00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f78880bfc00 0x7f78881eb6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:10.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.156+0000 7f788de00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f78880ca050 0x7f78880ab9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:10.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.156+0000 7f788de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7888152500 0x7f7888157410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:10.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.156+0000 7f788de00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f788806c300 con 0x7f7888152500 2024-02-21T23:10:10.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.156+0000 7f788de00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78880c9220 con 0x7f78880bfc00 2024-02-21T23:10:10.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.156+0000 7f788de00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f788806c480 con 0x7f78880ca050 2024-02-21T23:10:10.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.157+0000 7f788d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7888152500 0x7f7888157410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:10.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.157+0000 7f788d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7888152500 0x7f7888157410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35534/0 (socket says 172.21.15.28:35534) 2024-02-21T23:10:10.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.157+0000 7f788d400640 1 -- 172.21.15.28:0/3691715511 learned_addr learned my addr 172.21.15.28:0/3691715511 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:10.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.157+0000 7f7887e00640 1 --2- 172.21.15.28:0/3691715511 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f78880ca050 0x7f78880ab9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:10.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.157+0000 7f788ca00640 1 --2- 172.21.15.28:0/3691715511 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f78880bfc00 0x7f78881eb6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:10.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.157+0000 7f788d400640 1 -- 172.21.15.28:0/3691715511 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f78880ca050 msgr2=0x7f78880ab9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:10.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.157+0000 7f788d400640 1 --2- 172.21.15.28:0/3691715511 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f78880ca050 0x7f78880ab9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:10.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.157+0000 7f788d400640 1 -- 172.21.15.28:0/3691715511 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f78880bfc00 msgr2=0x7f78881eb6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:10.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.157+0000 7f788d400640 1 --2- 172.21.15.28:0/3691715511 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f78880bfc00 0x7f78881eb6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:10.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.157+0000 7f788d400640 1 -- 172.21.15.28:0/3691715511 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78881eac10 con 0x7f7888152500 2024-02-21T23:10:10.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.157+0000 7f788ca00640 1 --2- 172.21.15.28:0/3691715511 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f78880bfc00 0x7f78881eb6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:10:10.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.158+0000 7f788d400640 1 --2- 172.21.15.28:0/3691715511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7888152500 0x7f7888157410 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f788000c7d0 tx=0x7f788000cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:10.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.158+0000 7f7884c00640 1 -- 172.21.15.28:0/3691715511 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7880003b70 con 0x7f7888152500 2024-02-21T23:10:10.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.158+0000 7f7884c00640 1 -- 172.21.15.28:0/3691715511 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7880003d30 con 0x7f7888152500 2024-02-21T23:10:10.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.158+0000 7f7884c00640 1 -- 172.21.15.28:0/3691715511 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f788000aa90 con 0x7f7888152500 2024-02-21T23:10:10.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.158+0000 7f788de00640 1 -- 172.21.15.28:0/3691715511 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78881eec00 con 0x7f7888152500 2024-02-21T23:10:10.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.158+0000 7f788de00640 1 -- 172.21.15.28:0/3691715511 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78880abf00 con 0x7f7888152500 2024-02-21T23:10:10.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.159+0000 7f788de00640 1 -- 172.21.15.28:0/3691715511 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78880c0ed0 con 0x7f7888152500 2024-02-21T23:10:10.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.162+0000 7f7884c00640 1 -- 172.21.15.28:0/3691715511 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f788000ac30 con 0x7f7888152500 2024-02-21T23:10:10.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.163+0000 7f7884c00640 1 --2- 172.21.15.28:0/3691715511 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f786c06b500 0x7f786c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:10.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.163+0000 7f7884c00640 1 -- 172.21.15.28:0/3691715511 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f788008a610 con 0x7f7888152500 2024-02-21T23:10:10.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.163+0000 7f788ca00640 1 --2- 172.21.15.28:0/3691715511 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f786c06b500 0x7f786c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:10.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.164+0000 7f788ca00640 1 --2- 172.21.15.28:0/3691715511 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f786c06b500 0x7f786c06d9c0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f787c036000 tx=0x7f787c03c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:10.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.167+0000 7f7884c00640 1 -- 172.21.15.28:0/3691715511 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f78880c0ed0 con 0x7f7888152500 2024-02-21T23:10:10.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.391+0000 7f788de00640 1 -- 172.21.15.28:0/3691715511 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f788804f290 con 0x7f7888152500 2024-02-21T23:10:10.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.394+0000 7f7884c00640 1 -- 172.21.15.28:0/3691715511 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v155) v1 ==== 74+0+20542 (secure 0 0 0) 0x7f7880058fb0 con 0x7f7888152500 2024-02-21T23:10:10.383 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:10:10.383 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":155,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:10:07.361092+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:09:51.634291+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"150","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-02-21T23:09:52.736252+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"151","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs2_metadata","create_time":"2024-02-21T23:10:04.095451+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"154","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs2_data","create_time":"2024-02-21T23:10:04.836982+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:10:10.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.397+0000 7f788de00640 1 -- 172.21.15.28:0/3691715511 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f786c06b500 msgr2=0x7f786c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:10.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.397+0000 7f788de00640 1 --2- 172.21.15.28:0/3691715511 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f786c06b500 0x7f786c06d9c0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f787c036000 tx=0x7f787c03c040 comp rx=0 tx=0).stop 2024-02-21T23:10:10.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.397+0000 7f788de00640 1 -- 172.21.15.28:0/3691715511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7888152500 msgr2=0x7f7888157410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:10.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.397+0000 7f788de00640 1 --2- 172.21.15.28:0/3691715511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7888152500 0x7f7888157410 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f788000c7d0 tx=0x7f788000cc90 comp rx=0 tx=0).stop 2024-02-21T23:10:10.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.397+0000 7f788de00640 1 -- 172.21.15.28:0/3691715511 shutdown_connections 2024-02-21T23:10:10.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.397+0000 7f788de00640 1 --2- 172.21.15.28:0/3691715511 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f786c06b500 0x7f786c06d9c0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:10.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.397+0000 7f788de00640 1 --2- 172.21.15.28:0/3691715511 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f78880ca050 0x7f78880ab9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:10.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.397+0000 7f788de00640 1 --2- 172.21.15.28:0/3691715511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7888152500 0x7f7888157410 unknown :-1 s=CLOSED pgs=1737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:10.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.397+0000 7f788de00640 1 --2- 172.21.15.28:0/3691715511 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f78880bfc00 0x7f78881eb6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:10.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.397+0000 7f788de00640 1 -- 172.21.15.28:0/3691715511 >> 172.21.15.28:0/3691715511 conn(0x7f78880ac450 msgr2=0x7f7888153cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:10.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.397+0000 7f788de00640 1 -- 172.21.15.28:0/3691715511 shutdown_connections 2024-02-21T23:10:10.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.397+0000 7f788de00640 1 -- 172.21.15.28:0/3691715511 wait complete. 2024-02-21T23:10:10.407 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-02-21T23:10:10.407 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-21T23:10:10.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.617+0000 7f9719e00640 1 Processor -- start 2024-02-21T23:10:10.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.618+0000 7f9719e00640 1 -- start start 2024-02-21T23:10:10.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.620+0000 7f9719e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9714153950 0x7f97140c15d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:10.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.620+0000 7f9719e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9714152e70 con 0x7f97140c11f0 2024-02-21T23:10:10.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.620+0000 7f9719e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9714153150 con 0x7f971415b6d0 2024-02-21T23:10:10.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.620+0000 7f9719e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97140a5ec0 con 0x7f9714153950 2024-02-21T23:10:10.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.620+0000 7f9712a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f97140c11f0 0x7f971406dce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60880/0 (socket says 172.21.15.28:60880) 2024-02-21T23:10:10.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.620+0000 7f970be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9714153950 0x7f97140c15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:10.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.620+0000 7f9712a00640 1 -- 172.21.15.28:0/2482578980 learned_addr learned my addr 172.21.15.28:0/2482578980 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:10:10.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.621+0000 7f970be00640 1 -- 172.21.15.28:0/2482578980 >> v1:172.21.15.81:6789/0 conn(0x7f971415b6d0 legacy=0x7f971415dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:10.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.621+0000 7f970be00640 1 -- 172.21.15.28:0/2482578980 >> v1:172.21.15.28:6789/0 conn(0x7f97140c11f0 legacy=0x7f971406dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:10.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.621+0000 7f970be00640 1 -- 172.21.15.28:0/2482578980 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9714153430 con 0x7f9714153950 2024-02-21T23:10:10.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.621+0000 7f9712000640 1 -- 172.21.15.28:0/2482578980 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3746838817 0 0) 0x7f9714152e70 con 0x7f97140c11f0 2024-02-21T23:10:10.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.622+0000 7f970be00640 1 --2- 172.21.15.28:0/2482578980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9714153950 0x7f97140c15d0 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7f96fc0097a0 tx=0x7f96fc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67ce995d3a349e9b server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:10.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.622+0000 7f9712000640 1 -- 172.21.15.28:0/2482578980 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96fc0315a0 con 0x7f9714153950 2024-02-21T23:10:10.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.622+0000 7f9712000640 1 -- 172.21.15.28:0/2482578980 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96fc037070 con 0x7f9714153950 2024-02-21T23:10:10.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.622+0000 7f9712000640 1 -- 172.21.15.28:0/2482578980 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96fc031960 con 0x7f9714153950 2024-02-21T23:10:10.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.623+0000 7f9719e00640 1 -- 172.21.15.28:0/2482578980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9714153950 msgr2=0x7f97140c15d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:10.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.623+0000 7f9719e00640 1 --2- 172.21.15.28:0/2482578980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9714153950 0x7f97140c15d0 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7f96fc0097a0 tx=0x7f96fc02b840 comp rx=0 tx=0).stop 2024-02-21T23:10:10.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.623+0000 7f9719e00640 1 -- 172.21.15.28:0/2482578980 shutdown_connections 2024-02-21T23:10:10.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.623+0000 7f9719e00640 1 --2- 172.21.15.28:0/2482578980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9714153950 0x7f97140c15d0 unknown :-1 s=CLOSED pgs=1739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:10.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.623+0000 7f9719e00640 1 -- 172.21.15.28:0/2482578980 >> 172.21.15.28:0/2482578980 conn(0x7f97140c67f0 msgr2=0x7f97140c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:10.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.623+0000 7f9719e00640 1 -- 172.21.15.28:0/2482578980 shutdown_connections 2024-02-21T23:10:10.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.623+0000 7f9719e00640 1 -- 172.21.15.28:0/2482578980 wait complete. 2024-02-21T23:10:10.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.625+0000 7f9719e00640 1 Processor -- start 2024-02-21T23:10:10.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.625+0000 7f9719e00640 1 -- start start 2024-02-21T23:10:10.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.626+0000 7f9719e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97140c11f0 0x7f971414c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:10.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.626+0000 7f9719e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9714153950 0x7f97140aa6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:10.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.626+0000 7f9719e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f971415b6d0 0x7f97140aac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:10.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.626+0000 7f9719e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97140a6110 con 0x7f97140c11f0 2024-02-21T23:10:10.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.626+0000 7f9719e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97140c4c00 con 0x7f9714153950 2024-02-21T23:10:10.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.626+0000 7f9719e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97140c0d90 con 0x7f971415b6d0 2024-02-21T23:10:10.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.626+0000 7f9712a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97140c11f0 0x7f971414c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:10.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.626+0000 7f9712a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97140c11f0 0x7f971414c040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35554/0 (socket says 172.21.15.28:35554) 2024-02-21T23:10:10.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.626+0000 7f9712a00640 1 -- 172.21.15.28:0/3224368810 learned_addr learned my addr 172.21.15.28:0/3224368810 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:10.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.627+0000 7f970be00640 1 --2- 172.21.15.28:0/3224368810 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9714153950 0x7f97140aa6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:10.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.627+0000 7f9713400640 1 --2- 172.21.15.28:0/3224368810 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f971415b6d0 0x7f97140aac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:10.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.627+0000 7f9712a00640 1 -- 172.21.15.28:0/3224368810 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f971415b6d0 msgr2=0x7f97140aac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:10.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.627+0000 7f9712a00640 1 --2- 172.21.15.28:0/3224368810 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f971415b6d0 0x7f97140aac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:10.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.627+0000 7f9712a00640 1 -- 172.21.15.28:0/3224368810 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9714153950 msgr2=0x7f97140aa6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:10.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.627+0000 7f9712a00640 1 --2- 172.21.15.28:0/3224368810 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9714153950 0x7f97140aa6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:10.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.627+0000 7f9712a00640 1 -- 172.21.15.28:0/3224368810 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9714147bd0 con 0x7f97140c11f0 2024-02-21T23:10:10.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.627+0000 7f970be00640 1 --2- 172.21.15.28:0/3224368810 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9714153950 0x7f97140aa6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:10:10.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.627+0000 7f9712a00640 1 --2- 172.21.15.28:0/3224368810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97140c11f0 0x7f971414c040 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7f970400c460 tx=0x7f970400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:10.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.627+0000 7f970b400640 1 -- 172.21.15.28:0/3224368810 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9704018070 con 0x7f97140c11f0 2024-02-21T23:10:10.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.627+0000 7f970b400640 1 -- 172.21.15.28:0/3224368810 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9704007500 con 0x7f97140c11f0 2024-02-21T23:10:10.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.627+0000 7f9719e00640 1 -- 172.21.15.28:0/3224368810 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9714147e90 con 0x7f97140c11f0 2024-02-21T23:10:10.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.627+0000 7f970b400640 1 -- 172.21.15.28:0/3224368810 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9704004a20 con 0x7f97140c11f0 2024-02-21T23:10:10.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.628+0000 7f9719e00640 1 -- 172.21.15.28:0/3224368810 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97140a8d10 con 0x7f97140c11f0 2024-02-21T23:10:10.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.629+0000 7f9719e00640 1 -- 172.21.15.28:0/3224368810 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9714074ab0 con 0x7f97140c11f0 2024-02-21T23:10:10.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.630+0000 7f970b400640 1 -- 172.21.15.28:0/3224368810 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f970401b070 con 0x7f97140c11f0 2024-02-21T23:10:10.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.631+0000 7f970b400640 1 --2- 172.21.15.28:0/3224368810 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f96f006b4d0 0x7f96f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:10.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.631+0000 7f970b400640 1 -- 172.21.15.28:0/3224368810 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f97040898b0 con 0x7f97140c11f0 2024-02-21T23:10:10.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.632+0000 7f970be00640 1 --2- 172.21.15.28:0/3224368810 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f96f006b4d0 0x7f96f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:10.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.633+0000 7f970be00640 1 --2- 172.21.15.28:0/3224368810 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f96f006b4d0 0x7f96f006d990 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f96fc036000 tx=0x7f96fc008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:10.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.635+0000 7f970b400640 1 -- 172.21.15.28:0/3224368810 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f9714074ab0 con 0x7f97140c11f0 2024-02-21T23:10:10.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.912+0000 7f9719e00640 1 -- 172.21.15.28:0/3224368810 --> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9714067f30 con 0x7f96f006b4d0 2024-02-21T23:10:10.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.952+0000 7f970b400640 1 -- 172.21.15.28:0/3224368810 <== mgr.4101 v2:172.21.15.81:6832/1848538577 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+489418 (secure 0 0 0) 0x7f9714067f30 con 0x7f96f006b4d0 2024-02-21T23:10:10.948 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:10:10.951 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-21T23:10:10.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.961+0000 7f9719e00640 1 -- 172.21.15.28:0/3224368810 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f96f006b4d0 msgr2=0x7f96f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:10.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.961+0000 7f9719e00640 1 --2- 172.21.15.28:0/3224368810 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f96f006b4d0 0x7f96f006d990 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f96fc036000 tx=0x7f96fc008030 comp rx=0 tx=0).stop 2024-02-21T23:10:10.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.961+0000 7f9719e00640 1 -- 172.21.15.28:0/3224368810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97140c11f0 msgr2=0x7f971414c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:10.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.961+0000 7f9719e00640 1 --2- 172.21.15.28:0/3224368810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97140c11f0 0x7f971414c040 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7f970400c460 tx=0x7f970400c920 comp rx=0 tx=0).stop 2024-02-21T23:10:10.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.961+0000 7f9719e00640 1 -- 172.21.15.28:0/3224368810 shutdown_connections 2024-02-21T23:10:10.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.961+0000 7f9719e00640 1 --2- 172.21.15.28:0/3224368810 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f96f006b4d0 0x7f96f006d990 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:10.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.961+0000 7f9719e00640 1 --2- 172.21.15.28:0/3224368810 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f971415b6d0 0x7f97140aac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:10.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.961+0000 7f9719e00640 1 --2- 172.21.15.28:0/3224368810 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9714153950 0x7f97140aa6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:10.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.961+0000 7f9719e00640 1 --2- 172.21.15.28:0/3224368810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97140c11f0 0x7f971414c040 unknown :-1 s=CLOSED pgs=1740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:10.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.961+0000 7f9719e00640 1 -- 172.21.15.28:0/3224368810 >> 172.21.15.28:0/3224368810 conn(0x7f97140c67f0 msgr2=0x7f9714148c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:10.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.961+0000 7f9719e00640 1 -- 172.21.15.28:0/3224368810 shutdown_connections 2024-02-21T23:10:10.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:10.961+0000 7f9719e00640 1 -- 172.21.15.28:0/3224368810 wait complete. 2024-02-21T23:10:10.968 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":378,"stamp":"2024-02-21T23:10:10.824304+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":593450,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":219,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":19,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":171,"ondisk_log_size":171,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":309,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":264816,"kb_used_data":15344,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754709904,"statfs":{"total":773094113280,"available":772822941696,"internally_reserved":0,"allocated":15712256,"data_stored":8763576,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":672,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002075"},"pg_stats":[{"pgid":"26.3d","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.386132+0000","last_change":"2024-02-21T23:10:05.386132+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.386132+0000","last_undegraded":"2024-02-21T23:10:05.386132+0000","last_fullsized":"2024-02-21T23:10:05.386132+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.3f","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.394150+0000","last_change":"2024-02-21T23:09:54.279018+0000","last_active":"2024-02-21T23:10:05.394150+0000","last_peered":"2024-02-21T23:10:05.394150+0000","last_clean":"2024-02-21T23:10:05.394150+0000","last_became_active":"2024-02-21T23:09:54.278712+0000","last_became_peered":"2024-02-21T23:09:54.278712+0000","last_unstale":"2024-02-21T23:10:05.394150+0000","last_undegraded":"2024-02-21T23:10:05.394150+0000","last_fullsized":"2024-02-21T23:10:05.394150+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:55:18.591062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.3c","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232651+0000","last_change":"2024-02-21T23:10:06.385434+0000","last_active":"2024-02-21T23:10:08.232651+0000","last_peered":"2024-02-21T23:10:08.232651+0000","last_clean":"2024-02-21T23:10:08.232651+0000","last_became_active":"2024-02-21T23:10:06.384878+0000","last_became_peered":"2024-02-21T23:10:06.384878+0000","last_unstale":"2024-02-21T23:10:08.232651+0000","last_undegraded":"2024-02-21T23:10:08.232651+0000","last_fullsized":"2024-02-21T23:10:08.232651+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:52:21.069235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3e","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.373373+0000","last_change":"2024-02-21T23:09:54.285889+0000","last_active":"2024-02-21T23:10:05.373373+0000","last_peered":"2024-02-21T23:10:05.373373+0000","last_clean":"2024-02-21T23:10:05.373373+0000","last_became_active":"2024-02-21T23:09:54.284810+0000","last_became_peered":"2024-02-21T23:09:54.284810+0000","last_unstale":"2024-02-21T23:10:05.373373+0000","last_undegraded":"2024-02-21T23:10:05.373373+0000","last_fullsized":"2024-02-21T23:10:05.373373+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:04:16.317089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.3f","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236866+0000","last_change":"2024-02-21T23:10:06.389219+0000","last_active":"2024-02-21T23:10:08.236866+0000","last_peered":"2024-02-21T23:10:08.236866+0000","last_clean":"2024-02-21T23:10:08.236866+0000","last_became_active":"2024-02-21T23:10:06.388931+0000","last_became_peered":"2024-02-21T23:10:06.388931+0000","last_unstale":"2024-02-21T23:10:08.236866+0000","last_undegraded":"2024-02-21T23:10:08.236866+0000","last_fullsized":"2024-02-21T23:10:08.236866+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:46:45.683446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.3d","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.392968+0000","last_change":"2024-02-21T23:09:54.263978+0000","last_active":"2024-02-21T23:10:05.392968+0000","last_peered":"2024-02-21T23:10:05.392968+0000","last_clean":"2024-02-21T23:10:05.392968+0000","last_became_active":"2024-02-21T23:09:54.263467+0000","last_became_peered":"2024-02-21T23:09:54.263467+0000","last_unstale":"2024-02-21T23:10:05.392968+0000","last_undegraded":"2024-02-21T23:10:05.392968+0000","last_fullsized":"2024-02-21T23:10:05.392968+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:46:35.436815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.3e","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236400+0000","last_change":"2024-02-21T23:10:06.406362+0000","last_active":"2024-02-21T23:10:08.236400+0000","last_peered":"2024-02-21T23:10:08.236400+0000","last_clean":"2024-02-21T23:10:08.236400+0000","last_became_active":"2024-02-21T23:10:06.405859+0000","last_became_peered":"2024-02-21T23:10:06.405859+0000","last_unstale":"2024-02-21T23:10:08.236400+0000","last_undegraded":"2024-02-21T23:10:08.236400+0000","last_fullsized":"2024-02-21T23:10:08.236400+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:09:17.354064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.3c","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.390902+0000","last_change":"2024-02-21T23:09:54.279400+0000","last_active":"2024-02-21T23:10:05.390902+0000","last_peered":"2024-02-21T23:10:05.390902+0000","last_clean":"2024-02-21T23:10:05.390902+0000","last_became_active":"2024-02-21T23:09:54.279076+0000","last_became_peered":"2024-02-21T23:09:54.279076+0000","last_unstale":"2024-02-21T23:10:05.390902+0000","last_undegraded":"2024-02-21T23:10:05.390902+0000","last_fullsized":"2024-02-21T23:10:05.390902+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:02:11.247387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.39","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.375184+0000","last_change":"2024-02-21T23:10:06.375038+0000","last_active":"2024-02-21T23:10:07.375184+0000","last_peered":"2024-02-21T23:10:07.375184+0000","last_clean":"2024-02-21T23:10:07.375184+0000","last_became_active":"2024-02-21T23:10:06.374714+0000","last_became_peered":"2024-02-21T23:10:06.374714+0000","last_unstale":"2024-02-21T23:10:07.375184+0000","last_undegraded":"2024-02-21T23:10:07.375184+0000","last_fullsized":"2024-02-21T23:10:07.375184+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:48:19.473212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.3b","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.233882+0000","last_change":"2024-02-21T23:09:54.271682+0000","last_active":"2024-02-21T23:10:08.233882+0000","last_peered":"2024-02-21T23:10:08.233882+0000","last_clean":"2024-02-21T23:10:08.233882+0000","last_became_active":"2024-02-21T23:09:54.271297+0000","last_became_peered":"2024-02-21T23:09:54.271297+0000","last_unstale":"2024-02-21T23:10:08.233882+0000","last_undegraded":"2024-02-21T23:10:08.233882+0000","last_fullsized":"2024-02-21T23:10:08.233882+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:16:10.796552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.38","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.362732+0000","last_change":"2024-02-21T23:10:05.362732+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.362732+0000","last_undegraded":"2024-02-21T23:10:05.362732+0000","last_fullsized":"2024-02-21T23:10:05.362732+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3a","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.369709+0000","last_change":"2024-02-21T23:09:54.281830+0000","last_active":"2024-02-21T23:10:05.369709+0000","last_peered":"2024-02-21T23:10:05.369709+0000","last_clean":"2024-02-21T23:10:05.369709+0000","last_became_active":"2024-02-21T23:09:54.281334+0000","last_became_peered":"2024-02-21T23:09:54.281334+0000","last_unstale":"2024-02-21T23:10:05.369709+0000","last_undegraded":"2024-02-21T23:10:05.369709+0000","last_fullsized":"2024-02-21T23:10:05.369709+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:07:24.304915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.3b","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.233350+0000","last_change":"2024-02-21T23:10:06.385935+0000","last_active":"2024-02-21T23:10:08.233350+0000","last_peered":"2024-02-21T23:10:08.233350+0000","last_clean":"2024-02-21T23:10:08.233350+0000","last_became_active":"2024-02-21T23:10:06.385657+0000","last_became_peered":"2024-02-21T23:10:06.385657+0000","last_unstale":"2024-02-21T23:10:08.233350+0000","last_undegraded":"2024-02-21T23:10:08.233350+0000","last_fullsized":"2024-02-21T23:10:08.233350+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:27:47.158633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.39","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.233971+0000","last_change":"2024-02-21T23:09:54.277806+0000","last_active":"2024-02-21T23:10:08.233971+0000","last_peered":"2024-02-21T23:10:08.233971+0000","last_clean":"2024-02-21T23:10:08.233971+0000","last_became_active":"2024-02-21T23:09:54.277446+0000","last_became_peered":"2024-02-21T23:09:54.277446+0000","last_unstale":"2024-02-21T23:10:08.233971+0000","last_undegraded":"2024-02-21T23:10:08.233971+0000","last_fullsized":"2024-02-21T23:10:08.233971+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:59:34.646775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.3a","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.231962+0000","last_change":"2024-02-21T23:10:06.394386+0000","last_active":"2024-02-21T23:10:08.231962+0000","last_peered":"2024-02-21T23:10:08.231962+0000","last_clean":"2024-02-21T23:10:08.231962+0000","last_became_active":"2024-02-21T23:10:06.391655+0000","last_became_peered":"2024-02-21T23:10:06.391655+0000","last_unstale":"2024-02-21T23:10:08.231962+0000","last_undegraded":"2024-02-21T23:10:08.231962+0000","last_fullsized":"2024-02-21T23:10:08.231962+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:15:42.134184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.38","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232805+0000","last_change":"2024-02-21T23:09:54.286729+0000","last_active":"2024-02-21T23:10:08.232805+0000","last_peered":"2024-02-21T23:10:08.232805+0000","last_clean":"2024-02-21T23:10:08.232805+0000","last_became_active":"2024-02-21T23:09:54.286427+0000","last_became_peered":"2024-02-21T23:09:54.286427+0000","last_unstale":"2024-02-21T23:10:08.232805+0000","last_undegraded":"2024-02-21T23:10:08.232805+0000","last_fullsized":"2024-02-21T23:10:08.232805+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:13:27.311923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.35","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.373594+0000","last_change":"2024-02-21T23:10:05.373594+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.373594+0000","last_undegraded":"2024-02-21T23:10:05.373594+0000","last_fullsized":"2024-02-21T23:10:05.373594+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.37","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.237533+0000","last_change":"2024-02-21T23:09:54.286612+0000","last_active":"2024-02-21T23:10:08.237533+0000","last_peered":"2024-02-21T23:10:08.237533+0000","last_clean":"2024-02-21T23:10:08.237533+0000","last_became_active":"2024-02-21T23:09:54.286306+0000","last_became_peered":"2024-02-21T23:09:54.286306+0000","last_unstale":"2024-02-21T23:10:08.237533+0000","last_undegraded":"2024-02-21T23:10:08.237533+0000","last_fullsized":"2024-02-21T23:10:08.237533+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:04:16.317089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.34","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232897+0000","last_change":"2024-02-21T23:10:06.385190+0000","last_active":"2024-02-21T23:10:08.232897+0000","last_peered":"2024-02-21T23:10:08.232897+0000","last_clean":"2024-02-21T23:10:08.232897+0000","last_became_active":"2024-02-21T23:10:06.384496+0000","last_became_peered":"2024-02-21T23:10:06.384496+0000","last_unstale":"2024-02-21T23:10:08.232897+0000","last_undegraded":"2024-02-21T23:10:08.232897+0000","last_fullsized":"2024-02-21T23:10:08.232897+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:28:15.352760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.36","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.237062+0000","last_change":"2024-02-21T23:09:54.288019+0000","last_active":"2024-02-21T23:10:08.237062+0000","last_peered":"2024-02-21T23:10:08.237062+0000","last_clean":"2024-02-21T23:10:08.237062+0000","last_became_active":"2024-02-21T23:09:54.287742+0000","last_became_peered":"2024-02-21T23:09:54.287742+0000","last_unstale":"2024-02-21T23:10:08.237062+0000","last_undegraded":"2024-02-21T23:10:08.237062+0000","last_fullsized":"2024-02-21T23:10:08.237062+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:13:33.442227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.37","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236970+0000","last_change":"2024-02-21T23:10:06.398025+0000","last_active":"2024-02-21T23:10:08.236970+0000","last_peered":"2024-02-21T23:10:08.236970+0000","last_clean":"2024-02-21T23:10:08.236970+0000","last_became_active":"2024-02-21T23:10:06.395723+0000","last_became_peered":"2024-02-21T23:10:06.395723+0000","last_unstale":"2024-02-21T23:10:08.236970+0000","last_undegraded":"2024-02-21T23:10:08.236970+0000","last_fullsized":"2024-02-21T23:10:08.236970+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:50:33.682312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.35","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.378855+0000","last_change":"2024-02-21T23:09:54.283918+0000","last_active":"2024-02-21T23:10:05.378855+0000","last_peered":"2024-02-21T23:10:05.378855+0000","last_clean":"2024-02-21T23:10:05.378855+0000","last_became_active":"2024-02-21T23:09:54.280599+0000","last_became_peered":"2024-02-21T23:09:54.280599+0000","last_unstale":"2024-02-21T23:10:05.378855+0000","last_undegraded":"2024-02-21T23:10:05.378855+0000","last_fullsized":"2024-02-21T23:10:05.378855+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:33:07.362294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.36","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234837+0000","last_change":"2024-02-21T23:10:06.377423+0000","last_active":"2024-02-21T23:10:08.234837+0000","last_peered":"2024-02-21T23:10:08.234837+0000","last_clean":"2024-02-21T23:10:08.234837+0000","last_became_active":"2024-02-21T23:10:06.376998+0000","last_became_peered":"2024-02-21T23:10:06.376998+0000","last_unstale":"2024-02-21T23:10:08.234837+0000","last_undegraded":"2024-02-21T23:10:08.234837+0000","last_fullsized":"2024-02-21T23:10:08.234837+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:26:32.419695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.34","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.373186+0000","last_change":"2024-02-21T23:09:54.282846+0000","last_active":"2024-02-21T23:10:05.373186+0000","last_peered":"2024-02-21T23:10:05.373186+0000","last_clean":"2024-02-21T23:10:05.373186+0000","last_became_active":"2024-02-21T23:09:54.282117+0000","last_became_peered":"2024-02-21T23:09:54.282117+0000","last_unstale":"2024-02-21T23:10:05.373186+0000","last_undegraded":"2024-02-21T23:10:05.373186+0000","last_fullsized":"2024-02-21T23:10:05.373186+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:33:07.362294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.31","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234391+0000","last_change":"2024-02-21T23:10:06.382496+0000","last_active":"2024-02-21T23:10:08.234391+0000","last_peered":"2024-02-21T23:10:08.234391+0000","last_clean":"2024-02-21T23:10:08.234391+0000","last_became_active":"2024-02-21T23:10:06.379727+0000","last_became_peered":"2024-02-21T23:10:06.379727+0000","last_unstale":"2024-02-21T23:10:08.234391+0000","last_undegraded":"2024-02-21T23:10:08.234391+0000","last_fullsized":"2024-02-21T23:10:08.234391+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:10:21.186085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.33","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234540+0000","last_change":"2024-02-21T23:09:54.284818+0000","last_active":"2024-02-21T23:10:08.234540+0000","last_peered":"2024-02-21T23:10:08.234540+0000","last_clean":"2024-02-21T23:10:08.234540+0000","last_became_active":"2024-02-21T23:09:54.284161+0000","last_became_peered":"2024-02-21T23:09:54.284161+0000","last_unstale":"2024-02-21T23:10:08.234540+0000","last_undegraded":"2024-02-21T23:10:08.234540+0000","last_fullsized":"2024-02-21T23:10:08.234540+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:02:12.340921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.30","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.231818+0000","last_change":"2024-02-21T23:10:06.388372+0000","last_active":"2024-02-21T23:10:08.231818+0000","last_peered":"2024-02-21T23:10:08.231818+0000","last_clean":"2024-02-21T23:10:08.231818+0000","last_became_active":"2024-02-21T23:10:06.388099+0000","last_became_peered":"2024-02-21T23:10:06.388099+0000","last_unstale":"2024-02-21T23:10:08.231818+0000","last_undegraded":"2024-02-21T23:10:08.231818+0000","last_fullsized":"2024-02-21T23:10:08.231818+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:55:32.267269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.32","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.231579+0000","last_change":"2024-02-21T23:09:54.272136+0000","last_active":"2024-02-21T23:10:08.231579+0000","last_peered":"2024-02-21T23:10:08.231579+0000","last_clean":"2024-02-21T23:10:08.231579+0000","last_became_active":"2024-02-21T23:09:54.271097+0000","last_became_peered":"2024-02-21T23:09:54.271097+0000","last_unstale":"2024-02-21T23:10:08.231579+0000","last_undegraded":"2024-02-21T23:10:08.231579+0000","last_fullsized":"2024-02-21T23:10:08.231579+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:30:45.437334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.33","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.381970+0000","last_change":"2024-02-21T23:10:05.381970+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.381970+0000","last_undegraded":"2024-02-21T23:10:05.381970+0000","last_fullsized":"2024-02-21T23:10:05.381970+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.31","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.377899+0000","last_change":"2024-02-21T23:09:54.281268+0000","last_active":"2024-02-21T23:10:05.377899+0000","last_peered":"2024-02-21T23:10:05.377899+0000","last_clean":"2024-02-21T23:10:05.377899+0000","last_became_active":"2024-02-21T23:09:54.280946+0000","last_became_peered":"2024-02-21T23:09:54.280946+0000","last_unstale":"2024-02-21T23:10:05.377899+0000","last_undegraded":"2024-02-21T23:10:05.377899+0000","last_fullsized":"2024-02-21T23:10:05.377899+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:03:44.493909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.32","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.374763+0000","last_change":"2024-02-21T23:10:06.381260+0000","last_active":"2024-02-21T23:10:07.374763+0000","last_peered":"2024-02-21T23:10:07.374763+0000","last_clean":"2024-02-21T23:10:07.374763+0000","last_became_active":"2024-02-21T23:10:06.380936+0000","last_became_peered":"2024-02-21T23:10:06.380936+0000","last_unstale":"2024-02-21T23:10:07.374763+0000","last_undegraded":"2024-02-21T23:10:07.374763+0000","last_fullsized":"2024-02-21T23:10:07.374763+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:41:14.290670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.30","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.387425+0000","last_change":"2024-02-21T23:09:54.278731+0000","last_active":"2024-02-21T23:10:05.387425+0000","last_peered":"2024-02-21T23:10:05.387425+0000","last_clean":"2024-02-21T23:10:05.387425+0000","last_became_active":"2024-02-21T23:09:54.278390+0000","last_became_peered":"2024-02-21T23:09:54.278390+0000","last_unstale":"2024-02-21T23:10:05.387425+0000","last_undegraded":"2024-02-21T23:10:05.387425+0000","last_fullsized":"2024-02-21T23:10:05.387425+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:28:54.079553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.2d","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.382164+0000","last_change":"2024-02-21T23:10:05.382164+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.382164+0000","last_undegraded":"2024-02-21T23:10:05.382164+0000","last_fullsized":"2024-02-21T23:10:05.382164+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.2f","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.237800+0000","last_change":"2024-02-21T23:09:54.285947+0000","last_active":"2024-02-21T23:10:08.237800+0000","last_peered":"2024-02-21T23:10:08.237800+0000","last_clean":"2024-02-21T23:10:08.237800+0000","last_became_active":"2024-02-21T23:09:54.285658+0000","last_became_peered":"2024-02-21T23:09:54.285658+0000","last_unstale":"2024-02-21T23:10:08.237800+0000","last_undegraded":"2024-02-21T23:10:08.237800+0000","last_fullsized":"2024-02-21T23:10:08.237800+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:20:31.125684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2c","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236540+0000","last_change":"2024-02-21T23:10:06.376864+0000","last_active":"2024-02-21T23:10:08.236540+0000","last_peered":"2024-02-21T23:10:08.236540+0000","last_clean":"2024-02-21T23:10:08.236540+0000","last_became_active":"2024-02-21T23:10:06.376486+0000","last_became_peered":"2024-02-21T23:10:06.376486+0000","last_unstale":"2024-02-21T23:10:08.236540+0000","last_undegraded":"2024-02-21T23:10:08.236540+0000","last_fullsized":"2024-02-21T23:10:08.236540+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:22:04.227521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.2e","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.391407+0000","last_change":"2024-02-21T23:09:54.287286+0000","last_active":"2024-02-21T23:10:05.391407+0000","last_peered":"2024-02-21T23:10:05.391407+0000","last_clean":"2024-02-21T23:10:05.391407+0000","last_became_active":"2024-02-21T23:09:54.286924+0000","last_became_peered":"2024-02-21T23:09:54.286924+0000","last_unstale":"2024-02-21T23:10:05.391407+0000","last_undegraded":"2024-02-21T23:10:05.391407+0000","last_fullsized":"2024-02-21T23:10:05.391407+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:47:29.349669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.2f","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.235953+0000","last_change":"2024-02-21T23:10:06.377113+0000","last_active":"2024-02-21T23:10:08.235953+0000","last_peered":"2024-02-21T23:10:08.235953+0000","last_clean":"2024-02-21T23:10:08.235953+0000","last_became_active":"2024-02-21T23:10:06.376790+0000","last_became_peered":"2024-02-21T23:10:06.376790+0000","last_unstale":"2024-02-21T23:10:08.235953+0000","last_undegraded":"2024-02-21T23:10:08.235953+0000","last_fullsized":"2024-02-21T23:10:08.235953+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:38:03.009039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.2d","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.379032+0000","last_change":"2024-02-21T23:09:54.279884+0000","last_active":"2024-02-21T23:10:05.379032+0000","last_peered":"2024-02-21T23:10:05.379032+0000","last_clean":"2024-02-21T23:10:05.379032+0000","last_became_active":"2024-02-21T23:09:54.276608+0000","last_became_peered":"2024-02-21T23:09:54.276608+0000","last_unstale":"2024-02-21T23:10:05.379032+0000","last_undegraded":"2024-02-21T23:10:05.379032+0000","last_fullsized":"2024-02-21T23:10:05.379032+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:05:09.189420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.2e","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.383479+0000","last_change":"2024-02-21T23:10:05.383479+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.383479+0000","last_undegraded":"2024-02-21T23:10:05.383479+0000","last_fullsized":"2024-02-21T23:10:05.383479+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.2c","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.375678+0000","last_change":"2024-02-21T23:09:54.288944+0000","last_active":"2024-02-21T23:10:07.375678+0000","last_peered":"2024-02-21T23:10:07.375678+0000","last_clean":"2024-02-21T23:10:07.375678+0000","last_became_active":"2024-02-21T23:09:54.285913+0000","last_became_peered":"2024-02-21T23:09:54.285913+0000","last_unstale":"2024-02-21T23:10:07.375678+0000","last_undegraded":"2024-02-21T23:10:07.375678+0000","last_fullsized":"2024-02-21T23:10:07.375678+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:39:33.609273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.29","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.363048+0000","last_change":"2024-02-21T23:10:05.363048+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.363048+0000","last_undegraded":"2024-02-21T23:10:05.363048+0000","last_fullsized":"2024-02-21T23:10:05.363048+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.2b","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234769+0000","last_change":"2024-02-21T23:09:54.285036+0000","last_active":"2024-02-21T23:10:08.234769+0000","last_peered":"2024-02-21T23:10:08.234769+0000","last_clean":"2024-02-21T23:10:08.234769+0000","last_became_active":"2024-02-21T23:09:54.284542+0000","last_became_peered":"2024-02-21T23:09:54.284542+0000","last_unstale":"2024-02-21T23:10:08.234769+0000","last_undegraded":"2024-02-21T23:10:08.234769+0000","last_fullsized":"2024-02-21T23:10:08.234769+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T11:05:01.241636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.28","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234521+0000","last_change":"2024-02-21T23:10:06.396769+0000","last_active":"2024-02-21T23:10:08.234521+0000","last_peered":"2024-02-21T23:10:08.234521+0000","last_clean":"2024-02-21T23:10:08.234521+0000","last_became_active":"2024-02-21T23:10:06.384723+0000","last_became_peered":"2024-02-21T23:10:06.384723+0000","last_unstale":"2024-02-21T23:10:08.234521+0000","last_undegraded":"2024-02-21T23:10:08.234521+0000","last_fullsized":"2024-02-21T23:10:08.234521+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:45:13.867812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2a","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.370120+0000","last_change":"2024-02-21T23:09:54.273350+0000","last_active":"2024-02-21T23:10:05.370120+0000","last_peered":"2024-02-21T23:10:05.370120+0000","last_clean":"2024-02-21T23:10:05.370120+0000","last_became_active":"2024-02-21T23:09:54.273022+0000","last_became_peered":"2024-02-21T23:09:54.273022+0000","last_unstale":"2024-02-21T23:10:05.370120+0000","last_undegraded":"2024-02-21T23:10:05.370120+0000","last_fullsized":"2024-02-21T23:10:05.370120+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:34:28.642087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.2b","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.235284+0000","last_change":"2024-02-21T23:10:06.394830+0000","last_active":"2024-02-21T23:10:08.235284+0000","last_peered":"2024-02-21T23:10:08.235284+0000","last_clean":"2024-02-21T23:10:08.235284+0000","last_became_active":"2024-02-21T23:10:06.386513+0000","last_became_peered":"2024-02-21T23:10:06.386513+0000","last_unstale":"2024-02-21T23:10:08.235284+0000","last_undegraded":"2024-02-21T23:10:08.235284+0000","last_fullsized":"2024-02-21T23:10:08.235284+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:30:09.767949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.29","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.230852+0000","last_change":"2024-02-21T23:09:54.279718+0000","last_active":"2024-02-21T23:10:08.230852+0000","last_peered":"2024-02-21T23:10:08.230852+0000","last_clean":"2024-02-21T23:10:08.230852+0000","last_became_active":"2024-02-21T23:09:54.278989+0000","last_became_peered":"2024-02-21T23:09:54.278989+0000","last_unstale":"2024-02-21T23:10:08.230852+0000","last_undegraded":"2024-02-21T23:10:08.230852+0000","last_fullsized":"2024-02-21T23:10:08.230852+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:36:41.922548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.2a","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.235326+0000","last_change":"2024-02-21T23:10:06.397414+0000","last_active":"2024-02-21T23:10:08.235326+0000","last_peered":"2024-02-21T23:10:08.235326+0000","last_clean":"2024-02-21T23:10:08.235326+0000","last_became_active":"2024-02-21T23:10:06.396982+0000","last_became_peered":"2024-02-21T23:10:06.396982+0000","last_unstale":"2024-02-21T23:10:08.235326+0000","last_undegraded":"2024-02-21T23:10:08.235326+0000","last_fullsized":"2024-02-21T23:10:08.235326+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:05:37.143681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.28","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.387543+0000","last_change":"2024-02-21T23:09:54.280624+0000","last_active":"2024-02-21T23:10:05.387543+0000","last_peered":"2024-02-21T23:10:05.387543+0000","last_clean":"2024-02-21T23:10:05.387543+0000","last_became_active":"2024-02-21T23:09:54.275644+0000","last_became_peered":"2024-02-21T23:09:54.275644+0000","last_unstale":"2024-02-21T23:10:05.387543+0000","last_undegraded":"2024-02-21T23:10:05.387543+0000","last_fullsized":"2024-02-21T23:10:05.387543+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:23:27.511315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.25","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.365921+0000","last_change":"2024-02-21T23:10:05.365921+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.365921+0000","last_undegraded":"2024-02-21T23:10:05.365921+0000","last_fullsized":"2024-02-21T23:10:05.365921+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.27","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236738+0000","last_change":"2024-02-21T23:09:54.275341+0000","last_active":"2024-02-21T23:10:08.236738+0000","last_peered":"2024-02-21T23:10:08.236738+0000","last_clean":"2024-02-21T23:10:08.236738+0000","last_became_active":"2024-02-21T23:09:54.275010+0000","last_became_peered":"2024-02-21T23:09:54.275010+0000","last_unstale":"2024-02-21T23:10:08.236738+0000","last_undegraded":"2024-02-21T23:10:08.236738+0000","last_fullsized":"2024-02-21T23:10:08.236738+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:42:08.123188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.24","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236163+0000","last_change":"2024-02-21T23:10:06.386616+0000","last_active":"2024-02-21T23:10:08.236163+0000","last_peered":"2024-02-21T23:10:08.236163+0000","last_clean":"2024-02-21T23:10:08.236163+0000","last_became_active":"2024-02-21T23:10:06.386339+0000","last_became_peered":"2024-02-21T23:10:06.386339+0000","last_unstale":"2024-02-21T23:10:08.236163+0000","last_undegraded":"2024-02-21T23:10:08.236163+0000","last_fullsized":"2024-02-21T23:10:08.236163+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:44:41.892333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.26","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.376034+0000","last_change":"2024-02-21T23:09:54.260902+0000","last_active":"2024-02-21T23:10:07.376034+0000","last_peered":"2024-02-21T23:10:07.376034+0000","last_clean":"2024-02-21T23:10:07.376034+0000","last_became_active":"2024-02-21T23:09:54.260084+0000","last_became_peered":"2024-02-21T23:09:54.260084+0000","last_unstale":"2024-02-21T23:10:07.376034+0000","last_undegraded":"2024-02-21T23:10:07.376034+0000","last_fullsized":"2024-02-21T23:10:07.376034+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:04:05.858817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.27","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.375938+0000","last_change":"2024-02-21T23:10:06.374537+0000","last_active":"2024-02-21T23:10:07.375938+0000","last_peered":"2024-02-21T23:10:07.375938+0000","last_clean":"2024-02-21T23:10:07.375938+0000","last_became_active":"2024-02-21T23:10:06.374201+0000","last_became_peered":"2024-02-21T23:10:06.374201+0000","last_unstale":"2024-02-21T23:10:07.375938+0000","last_undegraded":"2024-02-21T23:10:07.375938+0000","last_fullsized":"2024-02-21T23:10:07.375938+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:52:55.561353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.25","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.373077+0000","last_change":"2024-02-21T23:09:54.284116+0000","last_active":"2024-02-21T23:10:05.373077+0000","last_peered":"2024-02-21T23:10:05.373077+0000","last_clean":"2024-02-21T23:10:05.373077+0000","last_became_active":"2024-02-21T23:09:54.283802+0000","last_became_peered":"2024-02-21T23:09:54.283802+0000","last_unstale":"2024-02-21T23:10:05.373077+0000","last_undegraded":"2024-02-21T23:10:05.373077+0000","last_fullsized":"2024-02-21T23:10:05.373077+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:05:09.189420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.26","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.374040+0000","last_change":"2024-02-21T23:10:05.374040+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.374040+0000","last_undegraded":"2024-02-21T23:10:05.374040+0000","last_fullsized":"2024-02-21T23:10:05.374040+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.24","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236346+0000","last_change":"2024-02-21T23:09:54.287481+0000","last_active":"2024-02-21T23:10:08.236346+0000","last_peered":"2024-02-21T23:10:08.236346+0000","last_clean":"2024-02-21T23:10:08.236346+0000","last_became_active":"2024-02-21T23:09:54.287137+0000","last_became_peered":"2024-02-21T23:09:54.287137+0000","last_unstale":"2024-02-21T23:10:08.236346+0000","last_undegraded":"2024-02-21T23:10:08.236346+0000","last_fullsized":"2024-02-21T23:10:08.236346+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:50:46.472002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.21","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.374988+0000","last_change":"2024-02-21T23:10:06.370867+0000","last_active":"2024-02-21T23:10:07.374988+0000","last_peered":"2024-02-21T23:10:07.374988+0000","last_clean":"2024-02-21T23:10:07.374988+0000","last_became_active":"2024-02-21T23:10:06.370583+0000","last_became_peered":"2024-02-21T23:10:06.370583+0000","last_unstale":"2024-02-21T23:10:07.374988+0000","last_undegraded":"2024-02-21T23:10:07.374988+0000","last_fullsized":"2024-02-21T23:10:07.374988+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:18:29.588662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.23","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.372190+0000","last_change":"2024-02-21T23:09:54.281613+0000","last_active":"2024-02-21T23:10:05.372190+0000","last_peered":"2024-02-21T23:10:05.372190+0000","last_clean":"2024-02-21T23:10:05.372190+0000","last_became_active":"2024-02-21T23:09:54.281316+0000","last_became_peered":"2024-02-21T23:09:54.281316+0000","last_unstale":"2024-02-21T23:10:05.372190+0000","last_undegraded":"2024-02-21T23:10:05.372190+0000","last_fullsized":"2024-02-21T23:10:05.372190+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:13:33.442227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.20","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232111+0000","last_change":"2024-02-21T23:10:06.382019+0000","last_active":"2024-02-21T23:10:08.232111+0000","last_peered":"2024-02-21T23:10:08.232111+0000","last_clean":"2024-02-21T23:10:08.232111+0000","last_became_active":"2024-02-21T23:10:06.381739+0000","last_became_peered":"2024-02-21T23:10:06.381739+0000","last_unstale":"2024-02-21T23:10:08.232111+0000","last_undegraded":"2024-02-21T23:10:08.232111+0000","last_fullsized":"2024-02-21T23:10:08.232111+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:25:17.018194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.22","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.390353+0000","last_change":"2024-02-21T23:09:54.275665+0000","last_active":"2024-02-21T23:10:05.390353+0000","last_peered":"2024-02-21T23:10:05.390353+0000","last_clean":"2024-02-21T23:10:05.390353+0000","last_became_active":"2024-02-21T23:09:54.275143+0000","last_became_peered":"2024-02-21T23:09:54.275143+0000","last_unstale":"2024-02-21T23:10:05.390353+0000","last_undegraded":"2024-02-21T23:10:05.390353+0000","last_fullsized":"2024-02-21T23:10:05.390353+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:37:42.976792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.23","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.371720+0000","last_change":"2024-02-21T23:10:05.371720+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.371720+0000","last_undegraded":"2024-02-21T23:10:05.371720+0000","last_fullsized":"2024-02-21T23:10:05.371720+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.21","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.378116+0000","last_change":"2024-02-21T23:09:54.281200+0000","last_active":"2024-02-21T23:10:05.378116+0000","last_peered":"2024-02-21T23:10:05.378116+0000","last_clean":"2024-02-21T23:10:05.378116+0000","last_became_active":"2024-02-21T23:09:54.277867+0000","last_became_peered":"2024-02-21T23:09:54.277867+0000","last_unstale":"2024-02-21T23:10:05.378116+0000","last_undegraded":"2024-02-21T23:10:05.378116+0000","last_fullsized":"2024-02-21T23:10:05.378116+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:59:34.646775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.22","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.375181+0000","last_change":"2024-02-21T23:10:05.375181+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.375181+0000","last_undegraded":"2024-02-21T23:10:05.375181+0000","last_fullsized":"2024-02-21T23:10:05.375181+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.20","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.390496+0000","last_change":"2024-02-21T23:09:54.276078+0000","last_active":"2024-02-21T23:10:05.390496+0000","last_peered":"2024-02-21T23:10:05.390496+0000","last_clean":"2024-02-21T23:10:05.390496+0000","last_became_active":"2024-02-21T23:09:54.275724+0000","last_became_peered":"2024-02-21T23:09:54.275724+0000","last_unstale":"2024-02-21T23:10:05.390496+0000","last_undegraded":"2024-02-21T23:10:05.390496+0000","last_fullsized":"2024-02-21T23:10:05.390496+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:48:07.369182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.377674+0000","last_change":"2024-02-21T23:10:05.377674+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.377674+0000","last_undegraded":"2024-02-21T23:10:05.377674+0000","last_fullsized":"2024-02-21T23:10:05.377674+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.230111+0000","last_change":"2024-02-21T23:10:05.385230+0000","last_active":"2024-02-21T23:10:08.230111+0000","last_peered":"2024-02-21T23:10:08.230111+0000","last_clean":"2024-02-21T23:10:08.230111+0000","last_became_active":"2024-02-21T23:10:05.384017+0000","last_became_peered":"2024-02-21T23:10:05.384017+0000","last_unstale":"2024-02-21T23:10:08.230111+0000","last_undegraded":"2024-02-21T23:10:08.230111+0000","last_fullsized":"2024-02-21T23:10:08.230111+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:45:12.852122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.10","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.368345+0000","last_change":"2024-02-21T23:09:54.278972+0000","last_active":"2024-02-21T23:10:05.368345+0000","last_peered":"2024-02-21T23:10:05.368345+0000","last_clean":"2024-02-21T23:10:05.368345+0000","last_became_active":"2024-02-21T23:09:54.278146+0000","last_became_peered":"2024-02-21T23:09:54.278146+0000","last_unstale":"2024-02-21T23:10:05.368345+0000","last_undegraded":"2024-02-21T23:10:05.368345+0000","last_fullsized":"2024-02-21T23:10:05.368345+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:59:34.646775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"150'4","reported_seq":22,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.394358+0000","last_change":"2024-02-21T23:09:53.288506+0000","last_active":"2024-02-21T23:10:05.394358+0000","last_peered":"2024-02-21T23:10:05.394358+0000","last_clean":"2024-02-21T23:10:05.394358+0000","last_became_active":"2024-02-21T23:09:53.288199+0000","last_became_peered":"2024-02-21T23:09:53.288199+0000","last_unstale":"2024-02-21T23:10:05.394358+0000","last_undegraded":"2024-02-21T23:10:05.394358+0000","last_fullsized":"2024-02-21T23:10:05.394358+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:18:16.467715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.13","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.378353+0000","last_change":"2024-02-21T23:10:05.378353+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.378353+0000","last_undegraded":"2024-02-21T23:10:05.378353+0000","last_fullsized":"2024-02-21T23:10:05.378353+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.10","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.386340+0000","last_change":"2024-02-21T23:10:05.382619+0000","last_active":"2024-02-21T23:10:05.386340+0000","last_peered":"2024-02-21T23:10:05.386340+0000","last_clean":"2024-02-21T23:10:05.386340+0000","last_became_active":"2024-02-21T23:10:05.382336+0000","last_became_peered":"2024-02-21T23:10:05.382336+0000","last_unstale":"2024-02-21T23:10:05.386340+0000","last_undegraded":"2024-02-21T23:10:05.386340+0000","last_fullsized":"2024-02-21T23:10:05.386340+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:11:55.677946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.231615+0000","last_change":"2024-02-21T23:09:54.278032+0000","last_active":"2024-02-21T23:10:08.231615+0000","last_peered":"2024-02-21T23:10:08.231615+0000","last_clean":"2024-02-21T23:10:08.231615+0000","last_became_active":"2024-02-21T23:09:54.277741+0000","last_became_peered":"2024-02-21T23:09:54.277741+0000","last_unstale":"2024-02-21T23:10:08.231615+0000","last_undegraded":"2024-02-21T23:10:08.231615+0000","last_fullsized":"2024-02-21T23:10:08.231615+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:43:56.610613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"150'1","reported_seq":23,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232900+0000","last_change":"2024-02-21T23:09:53.279374+0000","last_active":"2024-02-21T23:10:08.232900+0000","last_peered":"2024-02-21T23:10:08.232900+0000","last_clean":"2024-02-21T23:10:08.232900+0000","last_became_active":"2024-02-21T23:09:53.279053+0000","last_became_peered":"2024-02-21T23:09:53.279053+0000","last_unstale":"2024-02-21T23:10:08.232900+0000","last_undegraded":"2024-02-21T23:10:08.232900+0000","last_fullsized":"2024-02-21T23:10:08.232900+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:04:15.300172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.10","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.375571+0000","last_change":"2024-02-21T23:10:06.375103+0000","last_active":"2024-02-21T23:10:07.375571+0000","last_peered":"2024-02-21T23:10:07.375571+0000","last_clean":"2024-02-21T23:10:07.375571+0000","last_became_active":"2024-02-21T23:10:06.374814+0000","last_became_peered":"2024-02-21T23:10:06.374814+0000","last_unstale":"2024-02-21T23:10:07.375571+0000","last_undegraded":"2024-02-21T23:10:07.375571+0000","last_fullsized":"2024-02-21T23:10:07.375571+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:42:32.100185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.13","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.395209+0000","last_change":"2024-02-21T23:10:05.392497+0000","last_active":"2024-02-21T23:10:05.395209+0000","last_peered":"2024-02-21T23:10:05.395209+0000","last_clean":"2024-02-21T23:10:05.395209+0000","last_became_active":"2024-02-21T23:10:05.392183+0000","last_became_peered":"2024-02-21T23:10:05.392183+0000","last_unstale":"2024-02-21T23:10:05.395209+0000","last_undegraded":"2024-02-21T23:10:05.395209+0000","last_fullsized":"2024-02-21T23:10:05.395209+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:13:55.723872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.231985+0000","last_change":"2024-02-21T23:09:54.263915+0000","last_active":"2024-02-21T23:10:08.231985+0000","last_peered":"2024-02-21T23:10:08.231985+0000","last_clean":"2024-02-21T23:10:08.231985+0000","last_became_active":"2024-02-21T23:09:54.263634+0000","last_became_peered":"2024-02-21T23:09:54.263634+0000","last_unstale":"2024-02-21T23:10:08.231985+0000","last_undegraded":"2024-02-21T23:10:08.231985+0000","last_fullsized":"2024-02-21T23:10:08.231985+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:47:34.065041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.1d","version":"150'2","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.379180+0000","last_change":"2024-02-21T23:09:53.272110+0000","last_active":"2024-02-21T23:10:05.379180+0000","last_peered":"2024-02-21T23:10:05.379180+0000","last_clean":"2024-02-21T23:10:05.379180+0000","last_became_active":"2024-02-21T23:09:53.271641+0000","last_became_peered":"2024-02-21T23:09:53.271641+0000","last_unstale":"2024-02-21T23:10:05.379180+0000","last_undegraded":"2024-02-21T23:10:05.379180+0000","last_fullsized":"2024-02-21T23:10:05.379180+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:46:34.419898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.231438+0000","last_change":"2024-02-21T23:10:06.380458+0000","last_active":"2024-02-21T23:10:08.231438+0000","last_peered":"2024-02-21T23:10:08.231438+0000","last_clean":"2024-02-21T23:10:08.231438+0000","last_became_active":"2024-02-21T23:10:06.380137+0000","last_became_peered":"2024-02-21T23:10:06.380137+0000","last_unstale":"2024-02-21T23:10:08.231438+0000","last_undegraded":"2024-02-21T23:10:08.231438+0000","last_fullsized":"2024-02-21T23:10:08.231438+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:51:13.639406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236172+0000","last_change":"2024-02-21T23:10:05.393823+0000","last_active":"2024-02-21T23:10:08.236172+0000","last_peered":"2024-02-21T23:10:08.236172+0000","last_clean":"2024-02-21T23:10:08.236172+0000","last_became_active":"2024-02-21T23:10:05.390445+0000","last_became_peered":"2024-02-21T23:10:05.390445+0000","last_unstale":"2024-02-21T23:10:08.236172+0000","last_undegraded":"2024-02-21T23:10:08.236172+0000","last_fullsized":"2024-02-21T23:10:08.236172+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:30:56.525674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.13","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.372349+0000","last_change":"2024-02-21T23:09:54.275867+0000","last_active":"2024-02-21T23:10:05.372349+0000","last_peered":"2024-02-21T23:10:05.372349+0000","last_clean":"2024-02-21T23:10:05.372349+0000","last_became_active":"2024-02-21T23:09:54.275591+0000","last_became_peered":"2024-02-21T23:09:54.275591+0000","last_unstale":"2024-02-21T23:10:05.372349+0000","last_undegraded":"2024-02-21T23:10:05.372349+0000","last_fullsized":"2024-02-21T23:10:05.372349+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:42:16.988326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1c","version":"150'2","reported_seq":24,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.375729+0000","last_change":"2024-02-21T23:09:53.274692+0000","last_active":"2024-02-21T23:10:07.375729+0000","last_peered":"2024-02-21T23:10:07.375729+0000","last_clean":"2024-02-21T23:10:07.375729+0000","last_became_active":"2024-02-21T23:09:53.274395+0000","last_became_peered":"2024-02-21T23:09:53.274395+0000","last_unstale":"2024-02-21T23:10:07.375729+0000","last_undegraded":"2024-02-21T23:10:07.375729+0000","last_fullsized":"2024-02-21T23:10:07.375729+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:40:18.453921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.382764+0000","last_change":"2024-02-21T23:10:05.382764+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.382764+0000","last_undegraded":"2024-02-21T23:10:05.382764+0000","last_fullsized":"2024-02-21T23:10:05.382764+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.15","version":"154'3","reported_seq":17,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.233612+0000","last_change":"2024-02-21T23:10:05.374604+0000","last_active":"2024-02-21T23:10:08.233612+0000","last_peered":"2024-02-21T23:10:08.233612+0000","last_clean":"2024-02-21T23:10:08.233612+0000","last_became_active":"2024-02-21T23:10:05.374208+0000","last_became_peered":"2024-02-21T23:10:05.374208+0000","last_unstale":"2024-02-21T23:10:08.233612+0000","last_undegraded":"2024-02-21T23:10:08.233612+0000","last_fullsized":"2024-02-21T23:10:08.233612+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:30:08.752259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.14","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.235541+0000","last_change":"2024-02-21T23:09:54.289092+0000","last_active":"2024-02-21T23:10:08.235541+0000","last_peered":"2024-02-21T23:10:08.235541+0000","last_clean":"2024-02-21T23:10:08.235541+0000","last_became_active":"2024-02-21T23:09:54.288801+0000","last_became_peered":"2024-02-21T23:09:54.288801+0000","last_unstale":"2024-02-21T23:10:08.235541+0000","last_undegraded":"2024-02-21T23:10:08.235541+0000","last_fullsized":"2024-02-21T23:10:08.235541+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:05:09.189420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1b","version":"150'1","reported_seq":23,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236011+0000","last_change":"2024-02-21T23:09:53.283422+0000","last_active":"2024-02-21T23:10:08.236011+0000","last_peered":"2024-02-21T23:10:08.236011+0000","last_clean":"2024-02-21T23:10:08.236011+0000","last_became_active":"2024-02-21T23:09:53.283140+0000","last_became_peered":"2024-02-21T23:09:53.283140+0000","last_unstale":"2024-02-21T23:10:08.236011+0000","last_undegraded":"2024-02-21T23:10:08.236011+0000","last_fullsized":"2024-02-21T23:10:08.236011+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:46:34.419898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.237280+0000","last_change":"2024-02-21T23:10:06.403933+0000","last_active":"2024-02-21T23:10:08.237280+0000","last_peered":"2024-02-21T23:10:08.237280+0000","last_clean":"2024-02-21T23:10:08.237280+0000","last_became_active":"2024-02-21T23:10:06.403548+0000","last_became_peered":"2024-02-21T23:10:06.403548+0000","last_unstale":"2024-02-21T23:10:08.237280+0000","last_undegraded":"2024-02-21T23:10:08.237280+0000","last_fullsized":"2024-02-21T23:10:08.237280+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:11:20.176324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.14","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.394841+0000","last_change":"2024-02-21T23:10:05.388142+0000","last_active":"2024-02-21T23:10:05.394841+0000","last_peered":"2024-02-21T23:10:05.394841+0000","last_clean":"2024-02-21T23:10:05.394841+0000","last_became_active":"2024-02-21T23:10:05.387764+0000","last_became_peered":"2024-02-21T23:10:05.387764+0000","last_unstale":"2024-02-21T23:10:05.394841+0000","last_undegraded":"2024-02-21T23:10:05.394841+0000","last_fullsized":"2024-02-21T23:10:05.394841+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:58:38.451266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.15","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.375532+0000","last_change":"2024-02-21T23:09:54.273728+0000","last_active":"2024-02-21T23:10:07.375532+0000","last_peered":"2024-02-21T23:10:07.375532+0000","last_clean":"2024-02-21T23:10:07.375532+0000","last_became_active":"2024-02-21T23:09:54.273343+0000","last_became_peered":"2024-02-21T23:09:54.273343+0000","last_unstale":"2024-02-21T23:10:07.375532+0000","last_undegraded":"2024-02-21T23:10:07.375532+0000","last_fullsized":"2024-02-21T23:10:07.375532+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:49:15.552003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.1a","version":"150'1","reported_seq":23,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232153+0000","last_change":"2024-02-21T23:09:53.278176+0000","last_active":"2024-02-21T23:10:08.232153+0000","last_peered":"2024-02-21T23:10:08.232153+0000","last_clean":"2024-02-21T23:10:08.232153+0000","last_became_active":"2024-02-21T23:09:53.277880+0000","last_became_peered":"2024-02-21T23:09:53.277880+0000","last_unstale":"2024-02-21T23:10:08.232153+0000","last_undegraded":"2024-02-21T23:10:08.232153+0000","last_fullsized":"2024-02-21T23:10:08.232153+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:05:08.172503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.14","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.364119+0000","last_change":"2024-02-21T23:10:05.364119+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.364119+0000","last_undegraded":"2024-02-21T23:10:05.364119+0000","last_fullsized":"2024-02-21T23:10:05.364119+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236121+0000","last_change":"2024-02-21T23:10:05.375277+0000","last_active":"2024-02-21T23:10:08.236121+0000","last_peered":"2024-02-21T23:10:08.236121+0000","last_clean":"2024-02-21T23:10:08.236121+0000","last_became_active":"2024-02-21T23:10:05.374983+0000","last_became_peered":"2024-02-21T23:10:05.374983+0000","last_unstale":"2024-02-21T23:10:08.236121+0000","last_undegraded":"2024-02-21T23:10:08.236121+0000","last_fullsized":"2024-02-21T23:10:08.236121+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:11:19.160634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.376102+0000","last_change":"2024-02-21T23:09:54.278094+0000","last_active":"2024-02-21T23:10:07.376102+0000","last_peered":"2024-02-21T23:10:07.376102+0000","last_clean":"2024-02-21T23:10:07.376102+0000","last_became_active":"2024-02-21T23:09:54.276160+0000","last_became_peered":"2024-02-21T23:09:54.276160+0000","last_unstale":"2024-02-21T23:10:07.376102+0000","last_undegraded":"2024-02-21T23:10:07.376102+0000","last_fullsized":"2024-02-21T23:10:07.376102+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:15:07.339478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.375244+0000","last_change":"2024-02-21T23:09:53.280005+0000","last_active":"2024-02-21T23:10:07.375244+0000","last_peered":"2024-02-21T23:10:07.375244+0000","last_clean":"2024-02-21T23:10:07.375244+0000","last_became_active":"2024-02-21T23:09:53.279689+0000","last_became_peered":"2024-02-21T23:09:53.279689+0000","last_unstale":"2024-02-21T23:10:07.375244+0000","last_undegraded":"2024-02-21T23:10:07.375244+0000","last_fullsized":"2024-02-21T23:10:07.375244+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:40:05.561558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.15","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234855+0000","last_change":"2024-02-21T23:10:06.384066+0000","last_active":"2024-02-21T23:10:08.234855+0000","last_peered":"2024-02-21T23:10:08.234855+0000","last_clean":"2024-02-21T23:10:08.234855+0000","last_became_active":"2024-02-21T23:10:06.383780+0000","last_became_peered":"2024-02-21T23:10:06.383780+0000","last_unstale":"2024-02-21T23:10:08.234855+0000","last_undegraded":"2024-02-21T23:10:08.234855+0000","last_fullsized":"2024-02-21T23:10:08.234855+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:24:37.460238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236590+0000","last_change":"2024-02-21T23:10:05.393045+0000","last_active":"2024-02-21T23:10:08.236590+0000","last_peered":"2024-02-21T23:10:08.236590+0000","last_clean":"2024-02-21T23:10:08.236590+0000","last_became_active":"2024-02-21T23:10:05.392683+0000","last_became_peered":"2024-02-21T23:10:05.392683+0000","last_unstale":"2024-02-21T23:10:08.236590+0000","last_undegraded":"2024-02-21T23:10:08.236590+0000","last_fullsized":"2024-02-21T23:10:08.236590+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:45:39.918602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.394254+0000","last_change":"2024-02-21T23:09:54.262565+0000","last_active":"2024-02-21T23:10:05.394254+0000","last_peered":"2024-02-21T23:10:05.394254+0000","last_clean":"2024-02-21T23:10:05.394254+0000","last_became_active":"2024-02-21T23:09:54.261897+0000","last_became_peered":"2024-02-21T23:09:54.261897+0000","last_unstale":"2024-02-21T23:10:05.394254+0000","last_undegraded":"2024-02-21T23:10:05.394254+0000","last_fullsized":"2024-02-21T23:10:05.394254+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:08:50.519955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234939+0000","last_change":"2024-02-21T23:09:53.275695+0000","last_active":"2024-02-21T23:10:08.234939+0000","last_peered":"2024-02-21T23:10:08.234939+0000","last_clean":"2024-02-21T23:10:08.234939+0000","last_became_active":"2024-02-21T23:09:53.275343+0000","last_became_peered":"2024-02-21T23:09:53.275343+0000","last_unstale":"2024-02-21T23:10:08.234939+0000","last_undegraded":"2024-02-21T23:10:08.234939+0000","last_fullsized":"2024-02-21T23:10:08.234939+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:28:53.062636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1a","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.235787+0000","last_change":"2024-02-21T23:10:06.403716+0000","last_active":"2024-02-21T23:10:08.235787+0000","last_peered":"2024-02-21T23:10:08.235787+0000","last_clean":"2024-02-21T23:10:08.235787+0000","last_became_active":"2024-02-21T23:10:06.403226+0000","last_became_peered":"2024-02-21T23:10:06.403226+0000","last_unstale":"2024-02-21T23:10:08.235787+0000","last_undegraded":"2024-02-21T23:10:08.235787+0000","last_fullsized":"2024-02-21T23:10:08.235787+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:28:15.518153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.394797+0000","last_change":"2024-02-21T23:10:05.390846+0000","last_active":"2024-02-21T23:10:05.394797+0000","last_peered":"2024-02-21T23:10:05.394797+0000","last_clean":"2024-02-21T23:10:05.394797+0000","last_became_active":"2024-02-21T23:10:05.388088+0000","last_became_peered":"2024-02-21T23:10:05.388088+0000","last_unstale":"2024-02-21T23:10:05.394797+0000","last_undegraded":"2024-02-21T23:10:05.394797+0000","last_fullsized":"2024-02-21T23:10:05.394797+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:31:39.483569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.368060+0000","last_change":"2024-02-21T23:09:54.279220+0000","last_active":"2024-02-21T23:10:05.368060+0000","last_peered":"2024-02-21T23:10:05.368060+0000","last_clean":"2024-02-21T23:10:05.368060+0000","last_became_active":"2024-02-21T23:09:54.278653+0000","last_became_peered":"2024-02-21T23:09:54.278653+0000","last_unstale":"2024-02-21T23:10:05.368060+0000","last_undegraded":"2024-02-21T23:10:05.368060+0000","last_fullsized":"2024-02-21T23:10:05.368060+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:13:27.311923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236554+0000","last_change":"2024-02-21T23:09:53.288439+0000","last_active":"2024-02-21T23:10:08.236554+0000","last_peered":"2024-02-21T23:10:08.236554+0000","last_clean":"2024-02-21T23:10:08.236554+0000","last_became_active":"2024-02-21T23:09:53.287601+0000","last_became_peered":"2024-02-21T23:09:53.287601+0000","last_unstale":"2024-02-21T23:10:08.236554+0000","last_undegraded":"2024-02-21T23:10:08.236554+0000","last_fullsized":"2024-02-21T23:10:08.236554+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:32:05.045159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.376263+0000","last_change":"2024-02-21T23:10:05.376263+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.376263+0000","last_undegraded":"2024-02-21T23:10:05.376263+0000","last_fullsized":"2024-02-21T23:10:05.376263+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.1b","version":"154'1","reported_seq":15,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.233920+0000","last_change":"2024-02-21T23:10:05.387544+0000","last_active":"2024-02-21T23:10:08.233920+0000","last_peered":"2024-02-21T23:10:08.233920+0000","last_clean":"2024-02-21T23:10:08.233920+0000","last_became_active":"2024-02-21T23:10:05.385992+0000","last_became_peered":"2024-02-21T23:10:05.385992+0000","last_unstale":"2024-02-21T23:10:08.233920+0000","last_undegraded":"2024-02-21T23:10:08.233920+0000","last_fullsized":"2024-02-21T23:10:08.233920+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:28:14.502463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232148+0000","last_change":"2024-02-21T23:09:54.280274+0000","last_active":"2024-02-21T23:10:08.232148+0000","last_peered":"2024-02-21T23:10:08.232148+0000","last_clean":"2024-02-21T23:10:08.232148+0000","last_became_active":"2024-02-21T23:09:54.279951+0000","last_became_peered":"2024-02-21T23:09:54.279951+0000","last_unstale":"2024-02-21T23:10:08.232148+0000","last_undegraded":"2024-02-21T23:10:08.232148+0000","last_fullsized":"2024-02-21T23:10:08.232148+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:15:08.084683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.15","version":"150'3","reported_seq":21,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.393092+0000","last_change":"2024-02-21T23:09:53.279389+0000","last_active":"2024-02-21T23:10:05.393092+0000","last_peered":"2024-02-21T23:10:05.393092+0000","last_clean":"2024-02-21T23:10:05.393092+0000","last_became_active":"2024-02-21T23:09:53.279042+0000","last_became_peered":"2024-02-21T23:09:53.279042+0000","last_unstale":"2024-02-21T23:10:05.393092+0000","last_undegraded":"2024-02-21T23:10:05.393092+0000","last_fullsized":"2024-02-21T23:10:05.393092+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:44:44.898339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.233266+0000","last_change":"2024-02-21T23:10:06.379758+0000","last_active":"2024-02-21T23:10:08.233266+0000","last_peered":"2024-02-21T23:10:08.233266+0000","last_clean":"2024-02-21T23:10:08.233266+0000","last_became_active":"2024-02-21T23:10:06.379416+0000","last_became_peered":"2024-02-21T23:10:06.379416+0000","last_unstale":"2024-02-21T23:10:08.233266+0000","last_undegraded":"2024-02-21T23:10:08.233266+0000","last_fullsized":"2024-02-21T23:10:08.233266+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:06:55.637480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1a","version":"154'1","reported_seq":15,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.231363+0000","last_change":"2024-02-21T23:10:05.386289+0000","last_active":"2024-02-21T23:10:08.231363+0000","last_peered":"2024-02-21T23:10:08.231363+0000","last_clean":"2024-02-21T23:10:08.231363+0000","last_became_active":"2024-02-21T23:10:05.384200+0000","last_became_peered":"2024-02-21T23:10:05.384200+0000","last_unstale":"2024-02-21T23:10:08.231363+0000","last_undegraded":"2024-02-21T23:10:08.231363+0000","last_fullsized":"2024-02-21T23:10:08.231363+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:24:31.837524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1b","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.375019+0000","last_change":"2024-02-21T23:09:54.285563+0000","last_active":"2024-02-21T23:10:07.375019+0000","last_peered":"2024-02-21T23:10:07.375019+0000","last_clean":"2024-02-21T23:10:07.375019+0000","last_became_active":"2024-02-21T23:09:54.285275+0000","last_became_peered":"2024-02-21T23:09:54.285275+0000","last_unstale":"2024-02-21T23:10:07.375019+0000","last_undegraded":"2024-02-21T23:10:07.375019+0000","last_fullsized":"2024-02-21T23:10:07.375019+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:21:42.014406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"150'6","reported_seq":28,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236789+0000","last_change":"2024-02-21T23:09:53.288423+0000","last_active":"2024-02-21T23:10:08.236789+0000","last_peered":"2024-02-21T23:10:08.236789+0000","last_clean":"2024-02-21T23:10:08.236789+0000","last_became_active":"2024-02-21T23:09:53.287473+0000","last_became_peered":"2024-02-21T23:09:53.287473+0000","last_unstale":"2024-02-21T23:10:08.236789+0000","last_undegraded":"2024-02-21T23:10:08.236789+0000","last_fullsized":"2024-02-21T23:10:08.236789+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:13:32.425310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1e","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.373032+0000","last_change":"2024-02-21T23:10:05.373032+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.373032+0000","last_undegraded":"2024-02-21T23:10:05.373032+0000","last_fullsized":"2024-02-21T23:10:05.373032+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1d","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.391672+0000","last_change":"2024-02-21T23:10:05.388219+0000","last_active":"2024-02-21T23:10:05.391672+0000","last_peered":"2024-02-21T23:10:05.391672+0000","last_clean":"2024-02-21T23:10:05.391672+0000","last_became_active":"2024-02-21T23:10:05.387863+0000","last_became_peered":"2024-02-21T23:10:05.387863+0000","last_unstale":"2024-02-21T23:10:05.391672+0000","last_undegraded":"2024-02-21T23:10:05.391672+0000","last_fullsized":"2024-02-21T23:10:05.391672+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:34:10.873891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1c","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.390781+0000","last_change":"2024-02-21T23:09:54.278618+0000","last_active":"2024-02-21T23:10:05.390781+0000","last_peered":"2024-02-21T23:10:05.390781+0000","last_clean":"2024-02-21T23:10:05.390781+0000","last_became_active":"2024-02-21T23:09:54.278343+0000","last_became_peered":"2024-02-21T23:09:54.278343+0000","last_unstale":"2024-02-21T23:10:05.390781+0000","last_undegraded":"2024-02-21T23:10:05.390781+0000","last_fullsized":"2024-02-21T23:10:05.390781+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:49:31.759374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.13","version":"150'2","reported_seq":24,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234083+0000","last_change":"2024-02-21T23:09:53.284172+0000","last_active":"2024-02-21T23:10:08.234083+0000","last_peered":"2024-02-21T23:10:08.234083+0000","last_clean":"2024-02-21T23:10:08.234083+0000","last_became_active":"2024-02-21T23:09:53.283507+0000","last_became_peered":"2024-02-21T23:09:53.283507+0000","last_unstale":"2024-02-21T23:10:08.234083+0000","last_undegraded":"2024-02-21T23:10:08.234083+0000","last_fullsized":"2024-02-21T23:10:08.234083+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:20:30.108767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1f","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.382382+0000","last_change":"2024-02-21T23:10:05.382382+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.382382+0000","last_undegraded":"2024-02-21T23:10:05.382382+0000","last_fullsized":"2024-02-21T23:10:05.382382+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1c","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.394701+0000","last_change":"2024-02-21T23:10:05.387736+0000","last_active":"2024-02-21T23:10:05.394701+0000","last_peered":"2024-02-21T23:10:05.394701+0000","last_clean":"2024-02-21T23:10:05.394701+0000","last_became_active":"2024-02-21T23:10:05.387416+0000","last_became_peered":"2024-02-21T23:10:05.387416+0000","last_unstale":"2024-02-21T23:10:05.394701+0000","last_undegraded":"2024-02-21T23:10:05.394701+0000","last_fullsized":"2024-02-21T23:10:05.394701+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:35:01.442407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1d","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236368+0000","last_change":"2024-02-21T23:09:54.291717+0000","last_active":"2024-02-21T23:10:08.236368+0000","last_peered":"2024-02-21T23:10:08.236368+0000","last_clean":"2024-02-21T23:10:08.236368+0000","last_became_active":"2024-02-21T23:09:54.290842+0000","last_became_peered":"2024-02-21T23:09:54.290842+0000","last_unstale":"2024-02-21T23:10:08.236368+0000","last_undegraded":"2024-02-21T23:10:08.236368+0000","last_fullsized":"2024-02-21T23:10:08.236368+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:42:16.988326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232712+0000","last_change":"2024-02-21T23:09:53.284733+0000","last_active":"2024-02-21T23:10:08.232712+0000","last_peered":"2024-02-21T23:10:08.232712+0000","last_clean":"2024-02-21T23:10:08.232712+0000","last_became_active":"2024-02-21T23:09:53.284399+0000","last_became_peered":"2024-02-21T23:09:53.284399+0000","last_unstale":"2024-02-21T23:10:08.232712+0000","last_undegraded":"2024-02-21T23:10:08.232712+0000","last_fullsized":"2024-02-21T23:10:08.232712+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:05:08.172503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1c","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.365861+0000","last_change":"2024-02-21T23:10:05.365861+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.365861+0000","last_undegraded":"2024-02-21T23:10:05.365861+0000","last_fullsized":"2024-02-21T23:10:05.365861+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.1f","version":"154'4","reported_seq":18,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232862+0000","last_change":"2024-02-21T23:10:05.381058+0000","last_active":"2024-02-21T23:10:08.232862+0000","last_peered":"2024-02-21T23:10:08.232862+0000","last_clean":"2024-02-21T23:10:08.232862+0000","last_became_active":"2024-02-21T23:10:05.380737+0000","last_became_peered":"2024-02-21T23:10:05.380737+0000","last_unstale":"2024-02-21T23:10:08.232862+0000","last_undegraded":"2024-02-21T23:10:08.232862+0000","last_fullsized":"2024-02-21T23:10:08.232862+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:09:16.338374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1e","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.381340+0000","last_change":"2024-02-21T23:09:54.277339+0000","last_active":"2024-02-21T23:10:05.381340+0000","last_peered":"2024-02-21T23:10:05.381340+0000","last_clean":"2024-02-21T23:10:05.381340+0000","last_became_active":"2024-02-21T23:09:54.277039+0000","last_became_peered":"2024-02-21T23:09:54.277039+0000","last_unstale":"2024-02-21T23:10:05.381340+0000","last_undegraded":"2024-02-21T23:10:05.381340+0000","last_fullsized":"2024-02-21T23:10:05.381340+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:42:16.988326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":18,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.378558+0000","last_change":"2024-02-21T23:09:53.283915+0000","last_active":"2024-02-21T23:10:05.378558+0000","last_peered":"2024-02-21T23:10:05.378558+0000","last_clean":"2024-02-21T23:10:05.378558+0000","last_became_active":"2024-02-21T23:09:53.283532+0000","last_became_peered":"2024-02-21T23:09:53.283532+0000","last_unstale":"2024-02-21T23:10:05.378558+0000","last_undegraded":"2024-02-21T23:10:05.378558+0000","last_fullsized":"2024-02-21T23:10:05.378558+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:55:17.574145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1d","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.375317+0000","last_change":"2024-02-21T23:10:06.373957+0000","last_active":"2024-02-21T23:10:07.375317+0000","last_peered":"2024-02-21T23:10:07.375317+0000","last_clean":"2024-02-21T23:10:07.375317+0000","last_became_active":"2024-02-21T23:10:06.373657+0000","last_became_peered":"2024-02-21T23:10:06.373657+0000","last_unstale":"2024-02-21T23:10:07.375317+0000","last_undegraded":"2024-02-21T23:10:07.375317+0000","last_fullsized":"2024-02-21T23:10:07.375317+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:37:55.080822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1e","version":"154'1","reported_seq":15,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.235331+0000","last_change":"2024-02-21T23:10:05.388527+0000","last_active":"2024-02-21T23:10:08.235331+0000","last_peered":"2024-02-21T23:10:08.235331+0000","last_clean":"2024-02-21T23:10:08.235331+0000","last_became_active":"2024-02-21T23:10:05.388214+0000","last_became_peered":"2024-02-21T23:10:05.388214+0000","last_unstale":"2024-02-21T23:10:08.235331+0000","last_undegraded":"2024-02-21T23:10:08.235331+0000","last_fullsized":"2024-02-21T23:10:08.235331+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:50:32.666622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1f","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.376181+0000","last_change":"2024-02-21T23:09:54.265384+0000","last_active":"2024-02-21T23:10:07.376181+0000","last_peered":"2024-02-21T23:10:07.376181+0000","last_clean":"2024-02-21T23:10:07.376181+0000","last_became_active":"2024-02-21T23:09:54.265073+0000","last_became_peered":"2024-02-21T23:09:54.265073+0000","last_unstale":"2024-02-21T23:10:07.376181+0000","last_undegraded":"2024-02-21T23:10:07.376181+0000","last_fullsized":"2024-02-21T23:10:07.376181+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:44:28.295055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.10","version":"150'8","reported_seq":30,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.235156+0000","last_change":"2024-02-21T23:09:53.285246+0000","last_active":"2024-02-21T23:10:08.235156+0000","last_peered":"2024-02-21T23:10:08.235156+0000","last_clean":"2024-02-21T23:10:08.235156+0000","last_became_active":"2024-02-21T23:09:53.282084+0000","last_became_peered":"2024-02-21T23:09:53.282084+0000","last_unstale":"2024-02-21T23:10:08.235156+0000","last_undegraded":"2024-02-21T23:10:08.235156+0000","last_fullsized":"2024-02-21T23:10:08.235156+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:23:26.494398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.235925+0000","last_change":"2024-02-21T23:10:06.390439+0000","last_active":"2024-02-21T23:10:08.235925+0000","last_peered":"2024-02-21T23:10:08.235925+0000","last_clean":"2024-02-21T23:10:08.235925+0000","last_became_active":"2024-02-21T23:10:06.390012+0000","last_became_peered":"2024-02-21T23:10:06.390012+0000","last_unstale":"2024-02-21T23:10:08.235925+0000","last_undegraded":"2024-02-21T23:10:08.235925+0000","last_fullsized":"2024-02-21T23:10:08.235925+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:17:37.624314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.374798+0000","last_change":"2024-02-21T23:10:05.393586+0000","last_active":"2024-02-21T23:10:07.374798+0000","last_peered":"2024-02-21T23:10:07.374798+0000","last_clean":"2024-02-21T23:10:07.374798+0000","last_became_active":"2024-02-21T23:10:05.392606+0000","last_became_peered":"2024-02-21T23:10:05.392606+0000","last_unstale":"2024-02-21T23:10:07.374798+0000","last_undegraded":"2024-02-21T23:10:07.374798+0000","last_fullsized":"2024-02-21T23:10:07.374798+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:12:07.072214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234736+0000","last_change":"2024-02-21T23:09:54.289615+0000","last_active":"2024-02-21T23:10:08.234736+0000","last_peered":"2024-02-21T23:10:08.234736+0000","last_clean":"2024-02-21T23:10:08.234736+0000","last_became_active":"2024-02-21T23:09:54.289322+0000","last_became_peered":"2024-02-21T23:09:54.289322+0000","last_unstale":"2024-02-21T23:10:08.234736+0000","last_undegraded":"2024-02-21T23:10:08.234736+0000","last_fullsized":"2024-02-21T23:10:08.234736+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:34:28.642087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.f","version":"150'1","reported_seq":23,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236888+0000","last_change":"2024-02-21T23:09:53.287210+0000","last_active":"2024-02-21T23:10:08.236888+0000","last_peered":"2024-02-21T23:10:08.236888+0000","last_clean":"2024-02-21T23:10:08.236888+0000","last_became_active":"2024-02-21T23:09:53.285372+0000","last_became_peered":"2024-02-21T23:09:53.285372+0000","last_unstale":"2024-02-21T23:10:08.236888+0000","last_undegraded":"2024-02-21T23:10:08.236888+0000","last_fullsized":"2024-02-21T23:10:08.236888+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:20:30.108767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.363690+0000","last_change":"2024-02-21T23:10:05.363690+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.363690+0000","last_undegraded":"2024-02-21T23:10:05.363690+0000","last_fullsized":"2024-02-21T23:10:05.363690+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.395633+0000","last_change":"2024-02-21T23:10:05.395057+0000","last_active":"2024-02-21T23:10:05.395633+0000","last_peered":"2024-02-21T23:10:05.395633+0000","last_clean":"2024-02-21T23:10:05.395633+0000","last_became_active":"2024-02-21T23:10:05.394783+0000","last_became_peered":"2024-02-21T23:10:05.394783+0000","last_unstale":"2024-02-21T23:10:05.395633+0000","last_undegraded":"2024-02-21T23:10:05.395633+0000","last_fullsized":"2024-02-21T23:10:05.395633+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:55:29.679403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.390638+0000","last_change":"2024-02-21T23:09:54.274363+0000","last_active":"2024-02-21T23:10:05.390638+0000","last_peered":"2024-02-21T23:10:05.390638+0000","last_clean":"2024-02-21T23:10:05.390638+0000","last_became_active":"2024-02-21T23:09:54.274084+0000","last_became_peered":"2024-02-21T23:09:54.274084+0000","last_unstale":"2024-02-21T23:10:05.390638+0000","last_undegraded":"2024-02-21T23:10:05.390638+0000","last_fullsized":"2024-02-21T23:10:05.390638+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:42:19.996155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.e","version":"150'2","reported_seq":24,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.237049+0000","last_change":"2024-02-21T23:09:53.275941+0000","last_active":"2024-02-21T23:10:08.237049+0000","last_peered":"2024-02-21T23:10:08.237049+0000","last_clean":"2024-02-21T23:10:08.237049+0000","last_became_active":"2024-02-21T23:09:53.275555+0000","last_became_peered":"2024-02-21T23:09:53.275555+0000","last_unstale":"2024-02-21T23:10:08.237049+0000","last_undegraded":"2024-02-21T23:10:08.237049+0000","last_fullsized":"2024-02-21T23:10:08.237049+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:02:10.230470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.386121+0000","last_change":"2024-02-21T23:10:05.386121+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.386121+0000","last_undegraded":"2024-02-21T23:10:05.386121+0000","last_fullsized":"2024-02-21T23:10:05.386121+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.396519+0000","last_change":"2024-02-21T23:10:05.395810+0000","last_active":"2024-02-21T23:10:05.396519+0000","last_peered":"2024-02-21T23:10:05.396519+0000","last_clean":"2024-02-21T23:10:05.396519+0000","last_became_active":"2024-02-21T23:10:05.395273+0000","last_became_peered":"2024-02-21T23:10:05.395273+0000","last_unstale":"2024-02-21T23:10:05.396519+0000","last_undegraded":"2024-02-21T23:10:05.396519+0000","last_fullsized":"2024-02-21T23:10:05.396519+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:34:10.873891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.378450+0000","last_change":"2024-02-21T23:09:54.259630+0000","last_active":"2024-02-21T23:10:05.378450+0000","last_peered":"2024-02-21T23:10:05.378450+0000","last_clean":"2024-02-21T23:10:05.378450+0000","last_became_active":"2024-02-21T23:09:54.258924+0000","last_became_peered":"2024-02-21T23:09:54.258924+0000","last_unstale":"2024-02-21T23:10:05.378450+0000","last_undegraded":"2024-02-21T23:10:05.378450+0000","last_fullsized":"2024-02-21T23:10:05.378450+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:34:28.642087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.d","version":"150'1","reported_seq":23,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234011+0000","last_change":"2024-02-21T23:09:53.279385+0000","last_active":"2024-02-21T23:10:08.234011+0000","last_peered":"2024-02-21T23:10:08.234011+0000","last_clean":"2024-02-21T23:10:08.234011+0000","last_became_active":"2024-02-21T23:09:53.278074+0000","last_became_peered":"2024-02-21T23:09:53.278074+0000","last_unstale":"2024-02-21T23:10:08.234011+0000","last_undegraded":"2024-02-21T23:10:08.234011+0000","last_fullsized":"2024-02-21T23:10:08.234011+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:42:15.971409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.380894+0000","last_change":"2024-02-21T23:10:05.380894+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.380894+0000","last_undegraded":"2024-02-21T23:10:05.380894+0000","last_fullsized":"2024-02-21T23:10:05.380894+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.387108+0000","last_change":"2024-02-21T23:10:05.385621+0000","last_active":"2024-02-21T23:10:05.387108+0000","last_peered":"2024-02-21T23:10:05.387108+0000","last_clean":"2024-02-21T23:10:05.387108+0000","last_became_active":"2024-02-21T23:10:05.384654+0000","last_became_peered":"2024-02-21T23:10:05.384654+0000","last_unstale":"2024-02-21T23:10:05.387108+0000","last_undegraded":"2024-02-21T23:10:05.387108+0000","last_fullsized":"2024-02-21T23:10:05.387108+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:11:55.677946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.380169+0000","last_change":"2024-02-21T23:09:54.277296+0000","last_active":"2024-02-21T23:10:05.380169+0000","last_peered":"2024-02-21T23:10:05.380169+0000","last_clean":"2024-02-21T23:10:05.380169+0000","last_became_active":"2024-02-21T23:09:54.276973+0000","last_became_peered":"2024-02-21T23:09:54.276973+0000","last_unstale":"2024-02-21T23:10:05.380169+0000","last_undegraded":"2024-02-21T23:10:05.380169+0000","last_fullsized":"2024-02-21T23:10:05.380169+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:07:24.304915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.235861+0000","last_change":"2024-02-21T23:09:53.281804+0000","last_active":"2024-02-21T23:10:08.235861+0000","last_peered":"2024-02-21T23:10:08.235861+0000","last_clean":"2024-02-21T23:10:08.235861+0000","last_became_active":"2024-02-21T23:09:53.278643+0000","last_became_peered":"2024-02-21T23:09:53.278643+0000","last_unstale":"2024-02-21T23:10:08.235861+0000","last_undegraded":"2024-02-21T23:10:08.235861+0000","last_fullsized":"2024-02-21T23:10:08.235861+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:55:17.574145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.6","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.363355+0000","last_change":"2024-02-21T23:10:05.363355+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.363355+0000","last_undegraded":"2024-02-21T23:10:05.363355+0000","last_fullsized":"2024-02-21T23:10:05.363355+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234955+0000","last_change":"2024-02-21T23:10:05.389031+0000","last_active":"2024-02-21T23:10:08.234955+0000","last_peered":"2024-02-21T23:10:08.234955+0000","last_clean":"2024-02-21T23:10:08.234955+0000","last_became_active":"2024-02-21T23:10:05.388789+0000","last_became_peered":"2024-02-21T23:10:05.388789+0000","last_unstale":"2024-02-21T23:10:08.234955+0000","last_undegraded":"2024-02-21T23:10:08.234955+0000","last_fullsized":"2024-02-21T23:10:08.234955+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:46:44.667756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236537+0000","last_change":"2024-02-21T23:09:54.285141+0000","last_active":"2024-02-21T23:10:08.236537+0000","last_peered":"2024-02-21T23:10:08.236537+0000","last_clean":"2024-02-21T23:10:08.236537+0000","last_became_active":"2024-02-21T23:09:54.284498+0000","last_became_peered":"2024-02-21T23:09:54.284498+0000","last_unstale":"2024-02-21T23:10:08.236537+0000","last_undegraded":"2024-02-21T23:10:08.236537+0000","last_fullsized":"2024-02-21T23:10:08.236537+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:26:36.405020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.b","version":"150'2","reported_seq":24,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234201+0000","last_change":"2024-02-21T23:09:53.284284+0000","last_active":"2024-02-21T23:10:08.234201+0000","last_peered":"2024-02-21T23:10:08.234201+0000","last_clean":"2024-02-21T23:10:08.234201+0000","last_became_active":"2024-02-21T23:09:53.283043+0000","last_became_peered":"2024-02-21T23:09:53.283043+0000","last_unstale":"2024-02-21T23:10:08.234201+0000","last_undegraded":"2024-02-21T23:10:08.234201+0000","last_fullsized":"2024-02-21T23:10:08.234201+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:32:05.045159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.7","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.376118+0000","last_change":"2024-02-21T23:10:06.392968+0000","last_active":"2024-02-21T23:10:07.376118+0000","last_peered":"2024-02-21T23:10:07.376118+0000","last_clean":"2024-02-21T23:10:07.376118+0000","last_became_active":"2024-02-21T23:10:06.390031+0000","last_became_peered":"2024-02-21T23:10:06.390031+0000","last_unstale":"2024-02-21T23:10:07.376118+0000","last_undegraded":"2024-02-21T23:10:07.376118+0000","last_fullsized":"2024-02-21T23:10:07.376118+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:17:07.163396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236747+0000","last_change":"2024-02-21T23:10:05.392092+0000","last_active":"2024-02-21T23:10:08.236747+0000","last_peered":"2024-02-21T23:10:08.236747+0000","last_clean":"2024-02-21T23:10:08.236747+0000","last_became_active":"2024-02-21T23:10:05.391793+0000","last_became_peered":"2024-02-21T23:10:05.391793+0000","last_unstale":"2024-02-21T23:10:08.236747+0000","last_undegraded":"2024-02-21T23:10:08.236747+0000","last_fullsized":"2024-02-21T23:10:08.236747+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:09:16.338374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.233964+0000","last_change":"2024-02-21T23:09:54.275082+0000","last_active":"2024-02-21T23:10:08.233964+0000","last_peered":"2024-02-21T23:10:08.233964+0000","last_clean":"2024-02-21T23:10:08.233964+0000","last_became_active":"2024-02-21T23:09:54.274707+0000","last_became_peered":"2024-02-21T23:09:54.274707+0000","last_unstale":"2024-02-21T23:10:08.233964+0000","last_undegraded":"2024-02-21T23:10:08.233964+0000","last_fullsized":"2024-02-21T23:10:08.233964+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:39:52.080584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":18,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.378606+0000","last_change":"2024-02-21T23:09:53.277418+0000","last_active":"2024-02-21T23:10:05.378606+0000","last_peered":"2024-02-21T23:10:05.378606+0000","last_clean":"2024-02-21T23:10:05.378606+0000","last_became_active":"2024-02-21T23:09:53.277111+0000","last_became_peered":"2024-02-21T23:09:53.277111+0000","last_unstale":"2024-02-21T23:10:05.378606+0000","last_undegraded":"2024-02-21T23:10:05.378606+0000","last_fullsized":"2024-02-21T23:10:05.378606+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:23:26.494398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.375599+0000","last_change":"2024-02-21T23:10:06.391104+0000","last_active":"2024-02-21T23:10:07.375599+0000","last_peered":"2024-02-21T23:10:07.375599+0000","last_clean":"2024-02-21T23:10:07.375599+0000","last_became_active":"2024-02-21T23:10:06.390816+0000","last_became_peered":"2024-02-21T23:10:06.390816+0000","last_unstale":"2024-02-21T23:10:07.375599+0000","last_undegraded":"2024-02-21T23:10:07.375599+0000","last_fullsized":"2024-02-21T23:10:07.375599+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:49:43.863404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.7","version":"154'2","reported_seq":16,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234306+0000","last_change":"2024-02-21T23:10:05.378389+0000","last_active":"2024-02-21T23:10:08.234306+0000","last_peered":"2024-02-21T23:10:08.234306+0000","last_clean":"2024-02-21T23:10:08.234306+0000","last_became_active":"2024-02-21T23:10:05.378083+0000","last_became_peered":"2024-02-21T23:10:05.378083+0000","last_unstale":"2024-02-21T23:10:08.234306+0000","last_undegraded":"2024-02-21T23:10:08.234306+0000","last_fullsized":"2024-02-21T23:10:08.234306+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:28:14.502463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232579+0000","last_change":"2024-02-21T23:09:54.280400+0000","last_active":"2024-02-21T23:10:08.232579+0000","last_peered":"2024-02-21T23:10:08.232579+0000","last_clean":"2024-02-21T23:10:08.232579+0000","last_became_active":"2024-02-21T23:09:54.280102+0000","last_became_peered":"2024-02-21T23:09:54.280102+0000","last_unstale":"2024-02-21T23:10:08.232579+0000","last_undegraded":"2024-02-21T23:10:08.232579+0000","last_fullsized":"2024-02-21T23:10:08.232579+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:45:28.830261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.375349+0000","last_change":"2024-02-21T23:09:53.282939+0000","last_active":"2024-02-21T23:10:07.375349+0000","last_peered":"2024-02-21T23:10:07.375349+0000","last_clean":"2024-02-21T23:10:07.375349+0000","last_became_active":"2024-02-21T23:09:53.282612+0000","last_became_peered":"2024-02-21T23:09:53.282612+0000","last_unstale":"2024-02-21T23:10:07.375349+0000","last_undegraded":"2024-02-21T23:10:07.375349+0000","last_fullsized":"2024-02-21T23:10:07.375349+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:30:40.609835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236333+0000","last_change":"2024-02-21T23:10:06.396738+0000","last_active":"2024-02-21T23:10:08.236333+0000","last_peered":"2024-02-21T23:10:08.236333+0000","last_clean":"2024-02-21T23:10:08.236333+0000","last_became_active":"2024-02-21T23:10:06.396390+0000","last_became_peered":"2024-02-21T23:10:06.396390+0000","last_unstale":"2024-02-21T23:10:08.236333+0000","last_undegraded":"2024-02-21T23:10:08.236333+0000","last_fullsized":"2024-02-21T23:10:08.236333+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:24:32.853214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.6","version":"155'1","reported_seq":15,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.375969+0000","last_change":"2024-02-21T23:10:05.395962+0000","last_active":"2024-02-21T23:10:07.375969+0000","last_peered":"2024-02-21T23:10:07.375969+0000","last_clean":"2024-02-21T23:10:07.375969+0000","last_became_active":"2024-02-21T23:10:05.394863+0000","last_became_peered":"2024-02-21T23:10:05.394863+0000","last_unstale":"2024-02-21T23:10:07.375969+0000","last_undegraded":"2024-02-21T23:10:07.375969+0000","last_fullsized":"2024-02-21T23:10:07.375969+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:48:58.601649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.373315+0000","last_change":"2024-02-21T23:09:54.282302+0000","last_active":"2024-02-21T23:10:05.373315+0000","last_peered":"2024-02-21T23:10:05.373315+0000","last_clean":"2024-02-21T23:10:05.373315+0000","last_became_active":"2024-02-21T23:09:54.280468+0000","last_became_peered":"2024-02-21T23:09:54.280468+0000","last_unstale":"2024-02-21T23:10:05.373315+0000","last_undegraded":"2024-02-21T23:10:05.373315+0000","last_fullsized":"2024-02-21T23:10:05.373315+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:32:06.062076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":18,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.368703+0000","last_change":"2024-02-21T23:09:53.283837+0000","last_active":"2024-02-21T23:10:05.368703+0000","last_peered":"2024-02-21T23:10:05.368703+0000","last_clean":"2024-02-21T23:10:05.368703+0000","last_became_active":"2024-02-21T23:09:53.283541+0000","last_became_peered":"2024-02-21T23:09:53.283541+0000","last_unstale":"2024-02-21T23:10:05.368703+0000","last_undegraded":"2024-02-21T23:10:05.368703+0000","last_fullsized":"2024-02-21T23:10:05.368703+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:28:53.062636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.231400+0000","last_change":"2024-02-21T23:10:06.392422+0000","last_active":"2024-02-21T23:10:08.231400+0000","last_peered":"2024-02-21T23:10:08.231400+0000","last_clean":"2024-02-21T23:10:08.231400+0000","last_became_active":"2024-02-21T23:10:06.392121+0000","last_became_peered":"2024-02-21T23:10:06.392121+0000","last_unstale":"2024-02-21T23:10:08.231400+0000","last_undegraded":"2024-02-21T23:10:08.231400+0000","last_fullsized":"2024-02-21T23:10:08.231400+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:09:29.246717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.374605+0000","last_change":"2024-02-21T23:10:05.398114+0000","last_active":"2024-02-21T23:10:07.374605+0000","last_peered":"2024-02-21T23:10:07.374605+0000","last_clean":"2024-02-21T23:10:07.374605+0000","last_became_active":"2024-02-21T23:10:05.397811+0000","last_became_peered":"2024-02-21T23:10:05.397811+0000","last_unstale":"2024-02-21T23:10:07.374605+0000","last_undegraded":"2024-02-21T23:10:07.374605+0000","last_fullsized":"2024-02-21T23:10:07.374605+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:36:18.190950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.377721+0000","last_change":"2024-02-21T23:09:54.274229+0000","last_active":"2024-02-21T23:10:05.377721+0000","last_peered":"2024-02-21T23:10:05.377721+0000","last_clean":"2024-02-21T23:10:05.377721+0000","last_became_active":"2024-02-21T23:09:54.271365+0000","last_became_peered":"2024-02-21T23:09:54.271365+0000","last_unstale":"2024-02-21T23:10:05.377721+0000","last_undegraded":"2024-02-21T23:10:05.377721+0000","last_fullsized":"2024-02-21T23:10:05.377721+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:13:27.311923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.7","version":"150'2","reported_seq":24,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.233419+0000","last_change":"2024-02-21T23:09:53.283873+0000","last_active":"2024-02-21T23:10:08.233419+0000","last_peered":"2024-02-21T23:10:08.233419+0000","last_clean":"2024-02-21T23:10:08.233419+0000","last_became_active":"2024-02-21T23:09:53.282829+0000","last_became_peered":"2024-02-21T23:09:53.282829+0000","last_unstale":"2024-02-21T23:10:08.233419+0000","last_undegraded":"2024-02-21T23:10:08.233419+0000","last_fullsized":"2024-02-21T23:10:08.233419+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:13:32.425310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.b","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234395+0000","last_change":"2024-02-21T23:10:06.367171+0000","last_active":"2024-02-21T23:10:08.234395+0000","last_peered":"2024-02-21T23:10:08.234395+0000","last_clean":"2024-02-21T23:10:08.234395+0000","last_became_active":"2024-02-21T23:10:06.366866+0000","last_became_peered":"2024-02-21T23:10:06.366866+0000","last_unstale":"2024-02-21T23:10:08.234395+0000","last_undegraded":"2024-02-21T23:10:08.234395+0000","last_fullsized":"2024-02-21T23:10:08.234395+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:26:16.569464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.393848+0000","last_change":"2024-02-21T23:10:05.389876+0000","last_active":"2024-02-21T23:10:05.393848+0000","last_peered":"2024-02-21T23:10:05.393848+0000","last_clean":"2024-02-21T23:10:05.393848+0000","last_became_active":"2024-02-21T23:10:05.389568+0000","last_became_peered":"2024-02-21T23:10:05.389568+0000","last_unstale":"2024-02-21T23:10:05.393848+0000","last_undegraded":"2024-02-21T23:10:05.393848+0000","last_fullsized":"2024-02-21T23:10:05.393848+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:23:38.599655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234481+0000","last_change":"2024-02-21T23:09:54.277479+0000","last_active":"2024-02-21T23:10:08.234481+0000","last_peered":"2024-02-21T23:10:08.234481+0000","last_clean":"2024-02-21T23:10:08.234481+0000","last_became_active":"2024-02-21T23:09:54.277065+0000","last_became_peered":"2024-02-21T23:09:54.277065+0000","last_unstale":"2024-02-21T23:10:08.234481+0000","last_undegraded":"2024-02-21T23:10:08.234481+0000","last_fullsized":"2024-02-21T23:10:08.234481+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:07:24.304915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.6","version":"151'1","reported_seq":23,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.235714+0000","last_change":"2024-02-21T23:09:53.285874+0000","last_active":"2024-02-21T23:10:08.235714+0000","last_peered":"2024-02-21T23:10:08.235714+0000","last_clean":"2024-02-21T23:10:08.235714+0000","last_became_active":"2024-02-21T23:09:53.285569+0000","last_became_peered":"2024-02-21T23:09:53.285569+0000","last_unstale":"2024-02-21T23:10:08.235714+0000","last_undegraded":"2024-02-21T23:10:08.235714+0000","last_fullsized":"2024-02-21T23:10:08.235714+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:52:56.873078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.233184+0000","last_change":"2024-02-21T23:10:06.385851+0000","last_active":"2024-02-21T23:10:08.233184+0000","last_peered":"2024-02-21T23:10:08.233184+0000","last_clean":"2024-02-21T23:10:08.233184+0000","last_became_active":"2024-02-21T23:10:06.385482+0000","last_became_peered":"2024-02-21T23:10:06.385482+0000","last_unstale":"2024-02-21T23:10:08.233184+0000","last_undegraded":"2024-02-21T23:10:08.233184+0000","last_fullsized":"2024-02-21T23:10:08.233184+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:47:42.794742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.b","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.395230+0000","last_change":"2024-02-21T23:10:05.394114+0000","last_active":"2024-02-21T23:10:05.395230+0000","last_peered":"2024-02-21T23:10:05.395230+0000","last_clean":"2024-02-21T23:10:05.395230+0000","last_became_active":"2024-02-21T23:10:05.393152+0000","last_became_peered":"2024-02-21T23:10:05.393152+0000","last_unstale":"2024-02-21T23:10:05.395230+0000","last_undegraded":"2024-02-21T23:10:05.395230+0000","last_fullsized":"2024-02-21T23:10:05.395230+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:01:25.971350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.374932+0000","last_change":"2024-02-21T23:09:54.286102+0000","last_active":"2024-02-21T23:10:07.374932+0000","last_peered":"2024-02-21T23:10:07.374932+0000","last_clean":"2024-02-21T23:10:07.374932+0000","last_became_active":"2024-02-21T23:09:54.283073+0000","last_became_peered":"2024-02-21T23:09:54.283073+0000","last_unstale":"2024-02-21T23:10:07.374932+0000","last_undegraded":"2024-02-21T23:10:07.374932+0000","last_fullsized":"2024-02-21T23:10:07.374932+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:39:05.096862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.235327+0000","last_change":"2024-02-21T23:09:53.285730+0000","last_active":"2024-02-21T23:10:08.235327+0000","last_peered":"2024-02-21T23:10:08.235327+0000","last_clean":"2024-02-21T23:10:08.235327+0000","last_became_active":"2024-02-21T23:09:53.285373+0000","last_became_peered":"2024-02-21T23:09:53.285373+0000","last_unstale":"2024-02-21T23:10:08.235327+0000","last_undegraded":"2024-02-21T23:10:08.235327+0000","last_fullsized":"2024-02-21T23:10:08.235327+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:04:15.300172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.230302+0000","last_change":"2024-02-21T23:10:06.380586+0000","last_active":"2024-02-21T23:10:08.230302+0000","last_peered":"2024-02-21T23:10:08.230302+0000","last_clean":"2024-02-21T23:10:08.230302+0000","last_became_active":"2024-02-21T23:10:06.380293+0000","last_became_peered":"2024-02-21T23:10:06.380293+0000","last_unstale":"2024-02-21T23:10:08.230302+0000","last_undegraded":"2024-02-21T23:10:08.230302+0000","last_fullsized":"2024-02-21T23:10:08.230302+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:01:26.662036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.231095+0000","last_change":"2024-02-21T23:10:05.380512+0000","last_active":"2024-02-21T23:10:08.231095+0000","last_peered":"2024-02-21T23:10:08.231095+0000","last_clean":"2024-02-21T23:10:08.231095+0000","last_became_active":"2024-02-21T23:10:05.380207+0000","last_became_peered":"2024-02-21T23:10:05.380207+0000","last_unstale":"2024-02-21T23:10:08.231095+0000","last_undegraded":"2024-02-21T23:10:08.231095+0000","last_fullsized":"2024-02-21T23:10:08.231095+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:17:36.608624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.b","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.235603+0000","last_change":"2024-02-21T23:09:54.277483+0000","last_active":"2024-02-21T23:10:08.235603+0000","last_peered":"2024-02-21T23:10:08.235603+0000","last_clean":"2024-02-21T23:10:08.235603+0000","last_became_active":"2024-02-21T23:09:54.277170+0000","last_became_peered":"2024-02-21T23:09:54.277170+0000","last_unstale":"2024-02-21T23:10:08.235603+0000","last_undegraded":"2024-02-21T23:10:08.235603+0000","last_fullsized":"2024-02-21T23:10:08.235603+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:33:07.362294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.375793+0000","last_change":"2024-02-21T23:09:53.267455+0000","last_active":"2024-02-21T23:10:07.375793+0000","last_peered":"2024-02-21T23:10:07.375793+0000","last_clean":"2024-02-21T23:10:07.375793+0000","last_became_active":"2024-02-21T23:09:53.266974+0000","last_became_peered":"2024-02-21T23:09:53.266974+0000","last_unstale":"2024-02-21T23:10:07.375793+0000","last_undegraded":"2024-02-21T23:10:07.375793+0000","last_fullsized":"2024-02-21T23:10:07.375793+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:33:26.269092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.e","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.371905+0000","last_change":"2024-02-21T23:10:05.371905+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.371905+0000","last_undegraded":"2024-02-21T23:10:05.371905+0000","last_fullsized":"2024-02-21T23:10:05.371905+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.d","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.395287+0000","last_change":"2024-02-21T23:10:05.393632+0000","last_active":"2024-02-21T23:10:05.395287+0000","last_peered":"2024-02-21T23:10:05.395287+0000","last_clean":"2024-02-21T23:10:05.395287+0000","last_became_active":"2024-02-21T23:10:05.393070+0000","last_became_peered":"2024-02-21T23:10:05.393070+0000","last_unstale":"2024-02-21T23:10:05.395287+0000","last_undegraded":"2024-02-21T23:10:05.395287+0000","last_fullsized":"2024-02-21T23:10:05.395287+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:46:46.525155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.391035+0000","last_change":"2024-02-21T23:09:54.275273+0000","last_active":"2024-02-21T23:10:05.391035+0000","last_peered":"2024-02-21T23:10:05.391035+0000","last_clean":"2024-02-21T23:10:05.391035+0000","last_became_active":"2024-02-21T23:09:54.274287+0000","last_became_peered":"2024-02-21T23:09:54.274287+0000","last_unstale":"2024-02-21T23:10:05.391035+0000","last_undegraded":"2024-02-21T23:10:05.391035+0000","last_fullsized":"2024-02-21T23:10:05.391035+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:18:40.139538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":18,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.391026+0000","last_change":"2024-02-21T23:09:53.279782+0000","last_active":"2024-02-21T23:10:05.391026+0000","last_peered":"2024-02-21T23:10:05.391026+0000","last_clean":"2024-02-21T23:10:05.391026+0000","last_became_active":"2024-02-21T23:09:53.279477+0000","last_became_peered":"2024-02-21T23:09:53.279477+0000","last_unstale":"2024-02-21T23:10:05.391026+0000","last_undegraded":"2024-02-21T23:10:05.391026+0000","last_fullsized":"2024-02-21T23:10:05.391026+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:41:01.169723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.f","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.237523+0000","last_change":"2024-02-21T23:10:06.399721+0000","last_active":"2024-02-21T23:10:08.237523+0000","last_peered":"2024-02-21T23:10:08.237523+0000","last_clean":"2024-02-21T23:10:08.237523+0000","last_became_active":"2024-02-21T23:10:06.398869+0000","last_became_peered":"2024-02-21T23:10:06.398869+0000","last_unstale":"2024-02-21T23:10:08.237523+0000","last_undegraded":"2024-02-21T23:10:08.237523+0000","last_fullsized":"2024-02-21T23:10:08.237523+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:48:37.041227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.382546+0000","last_change":"2024-02-21T23:10:05.380230+0000","last_active":"2024-02-21T23:10:05.382546+0000","last_peered":"2024-02-21T23:10:05.382546+0000","last_clean":"2024-02-21T23:10:05.382546+0000","last_became_active":"2024-02-21T23:10:05.378612+0000","last_became_peered":"2024-02-21T23:10:05.378612+0000","last_unstale":"2024-02-21T23:10:05.382546+0000","last_undegraded":"2024-02-21T23:10:05.382546+0000","last_fullsized":"2024-02-21T23:10:05.382546+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:51:20.648392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.d","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.373520+0000","last_change":"2024-02-21T23:09:54.279409+0000","last_active":"2024-02-21T23:10:05.373520+0000","last_peered":"2024-02-21T23:10:05.373520+0000","last_clean":"2024-02-21T23:10:05.373520+0000","last_became_active":"2024-02-21T23:09:54.279107+0000","last_became_peered":"2024-02-21T23:09:54.279107+0000","last_unstale":"2024-02-21T23:10:05.373520+0000","last_undegraded":"2024-02-21T23:10:05.373520+0000","last_fullsized":"2024-02-21T23:10:05.373520+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:20:31.125684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232898+0000","last_change":"2024-02-21T23:09:53.278959+0000","last_active":"2024-02-21T23:10:08.232898+0000","last_peered":"2024-02-21T23:10:08.232898+0000","last_clean":"2024-02-21T23:10:08.232898+0000","last_became_active":"2024-02-21T23:09:53.278673+0000","last_became_peered":"2024-02-21T23:09:53.278673+0000","last_unstale":"2024-02-21T23:10:08.232898+0000","last_undegraded":"2024-02-21T23:10:08.232898+0000","last_fullsized":"2024-02-21T23:10:08.232898+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:42:15.971409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.375427+0000","last_change":"2024-02-21T23:10:06.387361+0000","last_active":"2024-02-21T23:10:07.375427+0000","last_peered":"2024-02-21T23:10:07.375427+0000","last_clean":"2024-02-21T23:10:07.375427+0000","last_became_active":"2024-02-21T23:10:06.387063+0000","last_became_peered":"2024-02-21T23:10:06.387063+0000","last_unstale":"2024-02-21T23:10:07.375427+0000","last_undegraded":"2024-02-21T23:10:07.375427+0000","last_fullsized":"2024-02-21T23:10:07.375427+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:47:41.453699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.385563+0000","last_change":"2024-02-21T23:10:05.381912+0000","last_active":"2024-02-21T23:10:05.385563+0000","last_peered":"2024-02-21T23:10:05.385563+0000","last_clean":"2024-02-21T23:10:05.385563+0000","last_became_active":"2024-02-21T23:10:05.381618+0000","last_became_peered":"2024-02-21T23:10:05.381618+0000","last_unstale":"2024-02-21T23:10:05.385563+0000","last_undegraded":"2024-02-21T23:10:05.385563+0000","last_fullsized":"2024-02-21T23:10:05.385563+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:53:08.978335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.e","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.235442+0000","last_change":"2024-02-21T23:09:54.282920+0000","last_active":"2024-02-21T23:10:08.235442+0000","last_peered":"2024-02-21T23:10:08.235442+0000","last_clean":"2024-02-21T23:10:08.235442+0000","last_became_active":"2024-02-21T23:09:54.282544+0000","last_became_peered":"2024-02-21T23:09:54.282544+0000","last_unstale":"2024-02-21T23:10:08.235442+0000","last_undegraded":"2024-02-21T23:10:08.235442+0000","last_fullsized":"2024-02-21T23:10:08.235442+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:40:05.206906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":18,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.378866+0000","last_change":"2024-02-21T23:09:53.275586+0000","last_active":"2024-02-21T23:10:05.378866+0000","last_peered":"2024-02-21T23:10:05.378866+0000","last_clean":"2024-02-21T23:10:05.378866+0000","last_became_active":"2024-02-21T23:09:53.274828+0000","last_became_peered":"2024-02-21T23:09:53.274828+0000","last_unstale":"2024-02-21T23:10:05.378866+0000","last_undegraded":"2024-02-21T23:10:05.378866+0000","last_fullsized":"2024-02-21T23:10:05.378866+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:02:10.230470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.d","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234506+0000","last_change":"2024-02-21T23:10:06.377243+0000","last_active":"2024-02-21T23:10:08.234506+0000","last_peered":"2024-02-21T23:10:08.234506+0000","last_clean":"2024-02-21T23:10:08.234506+0000","last_became_active":"2024-02-21T23:10:06.376521+0000","last_became_peered":"2024-02-21T23:10:06.376521+0000","last_unstale":"2024-02-21T23:10:08.234506+0000","last_undegraded":"2024-02-21T23:10:08.234506+0000","last_fullsized":"2024-02-21T23:10:08.234506+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:02:54.477082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.e","version":"154'2","reported_seq":16,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232515+0000","last_change":"2024-02-21T23:10:05.388215+0000","last_active":"2024-02-21T23:10:08.232515+0000","last_peered":"2024-02-21T23:10:08.232515+0000","last_clean":"2024-02-21T23:10:08.232515+0000","last_became_active":"2024-02-21T23:10:05.387907+0000","last_became_peered":"2024-02-21T23:10:05.387907+0000","last_unstale":"2024-02-21T23:10:08.232515+0000","last_undegraded":"2024-02-21T23:10:08.232515+0000","last_fullsized":"2024-02-21T23:10:08.232515+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:05:36.127991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.f","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.376215+0000","last_change":"2024-02-21T23:09:54.277320+0000","last_active":"2024-02-21T23:10:07.376215+0000","last_peered":"2024-02-21T23:10:07.376215+0000","last_clean":"2024-02-21T23:10:07.376215+0000","last_became_active":"2024-02-21T23:09:54.276368+0000","last_became_peered":"2024-02-21T23:09:54.276368+0000","last_unstale":"2024-02-21T23:10:07.376215+0000","last_undegraded":"2024-02-21T23:10:07.376215+0000","last_fullsized":"2024-02-21T23:10:07.376215+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:40:37.285836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.230247+0000","last_change":"2024-02-21T23:09:53.278027+0000","last_active":"2024-02-21T23:10:08.230247+0000","last_peered":"2024-02-21T23:10:08.230247+0000","last_clean":"2024-02-21T23:10:08.230247+0000","last_became_active":"2024-02-21T23:09:53.277660+0000","last_became_peered":"2024-02-21T23:09:53.277660+0000","last_unstale":"2024-02-21T23:10:08.230247+0000","last_undegraded":"2024-02-21T23:10:08.230247+0000","last_fullsized":"2024-02-21T23:10:08.230247+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:07:23.287998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1b","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.385924+0000","last_change":"2024-02-21T23:10:05.385924+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.385924+0000","last_undegraded":"2024-02-21T23:10:05.385924+0000","last_fullsized":"2024-02-21T23:10:05.385924+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.230256+0000","last_change":"2024-02-21T23:10:05.390395+0000","last_active":"2024-02-21T23:10:08.230256+0000","last_peered":"2024-02-21T23:10:08.230256+0000","last_clean":"2024-02-21T23:10:08.230256+0000","last_became_active":"2024-02-21T23:10:05.390115+0000","last_became_peered":"2024-02-21T23:10:05.390115+0000","last_unstale":"2024-02-21T23:10:08.230256+0000","last_undegraded":"2024-02-21T23:10:08.230256+0000","last_fullsized":"2024-02-21T23:10:08.230256+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:10:20.170395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.369300+0000","last_change":"2024-02-21T23:09:54.277356+0000","last_active":"2024-02-21T23:10:05.369300+0000","last_peered":"2024-02-21T23:10:05.369300+0000","last_clean":"2024-02-21T23:10:05.369300+0000","last_became_active":"2024-02-21T23:09:54.277087+0000","last_became_peered":"2024-02-21T23:09:54.277087+0000","last_unstale":"2024-02-21T23:10:05.369300+0000","last_undegraded":"2024-02-21T23:10:05.369300+0000","last_fullsized":"2024-02-21T23:10:05.369300+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:03:44.493909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.235691+0000","last_change":"2024-02-21T23:09:53.286181+0000","last_active":"2024-02-21T23:10:08.235691+0000","last_peered":"2024-02-21T23:10:08.235691+0000","last_clean":"2024-02-21T23:10:08.235691+0000","last_became_active":"2024-02-21T23:09:53.285849+0000","last_became_peered":"2024-02-21T23:09:53.285849+0000","last_unstale":"2024-02-21T23:10:08.235691+0000","last_undegraded":"2024-02-21T23:10:08.235691+0000","last_fullsized":"2024-02-21T23:10:08.235691+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:51:08.543135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":421,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.368921+0000","last_change":"2024-02-21T23:02:32.289335+0000","last_active":"2024-02-21T23:10:05.368921+0000","last_peered":"2024-02-21T23:10:05.368921+0000","last_clean":"2024-02-21T23:10:05.368921+0000","last_became_active":"2024-02-21T23:02:32.286913+0000","last_became_peered":"2024-02-21T23:02:32.286913+0000","last_unstale":"2024-02-21T23:10:05.368921+0000","last_undegraded":"2024-02-21T23:10:05.368921+0000","last_fullsized":"2024-02-21T23:10:05.368921+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_clean_scrub_stamp":"2024-02-21T23:02:31.172569+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:36:23.987374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":26,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":25,"num_pg":32,"stat_sum":{"num_bytes":672,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15,"ondisk_log_size":15,"up":64,"acting":64,"num_store_stats":8},{"poolid":24,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":23,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574941,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33124,"kb_used_data":1944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338716,"statfs":{"total":96636764160,"available":96602845184,"internally_reserved":0,"allocated":1990656,"data_stored":1530590,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574942,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33064,"kb_used_data":1880,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338776,"statfs":{"total":96636764160,"available":96602906624,"internally_reserved":0,"allocated":1925120,"data_stored":940222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574941,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33056,"kb_used_data":1876,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338784,"statfs":{"total":96636764160,"available":96602914816,"internally_reserved":0,"allocated":1921024,"data_stored":932824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574942,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33080,"kb_used_data":1896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338760,"statfs":{"total":96636764160,"available":96602890240,"internally_reserved":0,"allocated":1941504,"data_stored":940986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574943,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33120,"kb_used_data":1932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338720,"statfs":{"total":96636764160,"available":96602849280,"internally_reserved":0,"allocated":1978368,"data_stored":958172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574944,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33096,"kb_used_data":1908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338744,"statfs":{"total":96636764160,"available":96602873856,"internally_reserved":0,"allocated":1953792,"data_stored":956378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574942,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33104,"kb_used_data":1916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338736,"statfs":{"total":96636764160,"available":96602865664,"internally_reserved":0,"allocated":1961984,"data_stored":957408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574942,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33172,"kb_used_data":1992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338668,"statfs":{"total":96636764160,"available":96602796032,"internally_reserved":0,"allocated":2039808,"data_stored":1546996,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-21T23:10:10.976 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-21T23:10:11.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.246+0000 7f7d00000640 1 Processor -- start 2024-02-21T23:10:11.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.246+0000 7f7d00000640 1 -- start start 2024-02-21T23:10:11.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.249+0000 7f7d00000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f7cf813e620 0x7f7cf8159070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:11.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.249+0000 7f7d00000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7cf80bf990 0x7f7cf80bfdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:11.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.249+0000 7f7d00000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cf81551b0 con 0x7f7cf80c6460 2024-02-21T23:10:11.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.249+0000 7f7d00000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cf80c71d0 con 0x7f7cf80bf990 2024-02-21T23:10:11.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.249+0000 7f7d00000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cf81595b0 con 0x7f7cf813e620 2024-02-21T23:10:11.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.249+0000 7f7cfe000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7cf80bf990 0x7f7cf80bfdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:11.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.249+0000 7f7cfe000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7cf80bf990 0x7f7cf80bfdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35558/0 (socket says 172.21.15.28:35558) 2024-02-21T23:10:11.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.249+0000 7f7cfe000640 1 -- 172.21.15.28:0/2472348855 learned_addr learned my addr 172.21.15.28:0/2472348855 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:11.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.250+0000 7f7cfd600640 1 --2- 172.21.15.28:0/2472348855 >> v2:172.21.15.81:3300/0 conn(0x7f7cf813e620 0x7f7cf8159070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:11.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.250+0000 7f7cfe000640 1 -- 172.21.15.28:0/2472348855 >> v2:172.21.15.81:3300/0 conn(0x7f7cf813e620 msgr2=0x7f7cf8159070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:11.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.250+0000 7f7cfe000640 1 --2- 172.21.15.28:0/2472348855 >> v2:172.21.15.81:3300/0 conn(0x7f7cf813e620 0x7f7cf8159070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:11.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.250+0000 7f7cfe000640 1 -- 172.21.15.28:0/2472348855 >> v1:172.21.15.81:6789/0 conn(0x7f7cf80c6460 legacy=0x7f7cf80ca8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:11.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.250+0000 7f7cfe000640 1 -- 172.21.15.28:0/2472348855 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cf8155420 con 0x7f7cf80bf990 2024-02-21T23:10:11.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.251+0000 7f7cfe000640 1 --2- 172.21.15.28:0/2472348855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cf80bf990 0x7f7cf80bfdc0 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7f7cf808c100 tx=0x7f7cf002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f7cd0e4b8fb5c36 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:11.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.251+0000 7f7cf7e00640 1 -- 172.21.15.28:0/2472348855 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cf002c9f0 con 0x7f7cf80bf990 2024-02-21T23:10:11.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.251+0000 7f7cf7e00640 1 -- 172.21.15.28:0/2472348855 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cf002cb90 con 0x7f7cf80bf990 2024-02-21T23:10:11.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.251+0000 7f7cf7e00640 1 -- 172.21.15.28:0/2472348855 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cf00343f0 con 0x7f7cf80bf990 2024-02-21T23:10:11.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.252+0000 7f7d00000640 1 -- 172.21.15.28:0/2472348855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cf80bf990 msgr2=0x7f7cf80bfdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:11.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.252+0000 7f7d00000640 1 --2- 172.21.15.28:0/2472348855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cf80bf990 0x7f7cf80bfdc0 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7f7cf808c100 tx=0x7f7cf002be50 comp rx=0 tx=0).stop 2024-02-21T23:10:11.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.252+0000 7f7d00000640 1 -- 172.21.15.28:0/2472348855 shutdown_connections 2024-02-21T23:10:11.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.252+0000 7f7d00000640 1 --2- 172.21.15.28:0/2472348855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cf80bf990 0x7f7cf80bfdc0 unknown :-1 s=CLOSED pgs=1741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:11.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.252+0000 7f7d00000640 1 --2- 172.21.15.28:0/2472348855 >> v2:172.21.15.81:3300/0 conn(0x7f7cf813e620 0x7f7cf8159070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:11.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.252+0000 7f7d00000640 1 -- 172.21.15.28:0/2472348855 >> 172.21.15.28:0/2472348855 conn(0x7f7cf808b1c0 msgr2=0x7f7cf80725a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:11.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.252+0000 7f7d00000640 1 -- 172.21.15.28:0/2472348855 shutdown_connections 2024-02-21T23:10:11.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.252+0000 7f7d00000640 1 -- 172.21.15.28:0/2472348855 wait complete. 2024-02-21T23:10:11.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.254+0000 7f7d00000640 1 Processor -- start 2024-02-21T23:10:11.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.254+0000 7f7d00000640 1 -- start start 2024-02-21T23:10:11.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.255+0000 7f7d00000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cf80bf990 0x7f7cf81e2010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:11.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.255+0000 7f7d00000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7cf80c6460 0x7f7cf81ebc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:11.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.255+0000 7f7d00000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7cf813e620 0x7f7cf8157750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:11.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.255+0000 7f7d00000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf8155cc0 con 0x7f7cf80bf990 2024-02-21T23:10:11.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.255+0000 7f7d00000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf81559e0 con 0x7f7cf80c6460 2024-02-21T23:10:11.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.255+0000 7f7d00000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf813ea00 con 0x7f7cf813e620 2024-02-21T23:10:11.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.255+0000 7f7cfd600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cf80bf990 0x7f7cf81e2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:11.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.255+0000 7f7cfd600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cf80bf990 0x7f7cf81e2010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35574/0 (socket says 172.21.15.28:35574) 2024-02-21T23:10:11.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.255+0000 7f7cfd600640 1 -- 172.21.15.28:0/1250526920 learned_addr learned my addr 172.21.15.28:0/1250526920 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:11.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.256+0000 7f7cfcc00640 1 --2- 172.21.15.28:0/1250526920 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7cf80c6460 0x7f7cf81ebc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:11.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.256+0000 7f7cfe000640 1 --2- 172.21.15.28:0/1250526920 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7cf813e620 0x7f7cf8157750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:11.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.256+0000 7f7cfd600640 1 -- 172.21.15.28:0/1250526920 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7cf813e620 msgr2=0x7f7cf8157750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:11.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.256+0000 7f7cfd600640 1 --2- 172.21.15.28:0/1250526920 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7cf813e620 0x7f7cf8157750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:11.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.256+0000 7f7cfd600640 1 -- 172.21.15.28:0/1250526920 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7cf80c6460 msgr2=0x7f7cf81ebc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:11.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.256+0000 7f7cfd600640 1 --2- 172.21.15.28:0/1250526920 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7cf80c6460 0x7f7cf81ebc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:11.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.256+0000 7f7cfd600640 1 -- 172.21.15.28:0/1250526920 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cf813b540 con 0x7f7cf80bf990 2024-02-21T23:10:11.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.256+0000 7f7cfcc00640 1 --2- 172.21.15.28:0/1250526920 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7cf80c6460 0x7f7cf81ebc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:10:11.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.256+0000 7f7cfd600640 1 --2- 172.21.15.28:0/1250526920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cf80bf990 0x7f7cf81e2010 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7f7cec00d600 tx=0x7f7cec00dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:11.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.256+0000 7f7cf5600640 1 -- 172.21.15.28:0/1250526920 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cec0125f0 con 0x7f7cf80bf990 2024-02-21T23:10:11.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.256+0000 7f7cf5600640 1 -- 172.21.15.28:0/1250526920 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cec019070 con 0x7f7cf80bf990 2024-02-21T23:10:11.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.256+0000 7f7cf5600640 1 -- 172.21.15.28:0/1250526920 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cec0129f0 con 0x7f7cf80bf990 2024-02-21T23:10:11.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.257+0000 7f7d00000640 1 -- 172.21.15.28:0/1250526920 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cf8158300 con 0x7f7cf80bf990 2024-02-21T23:10:11.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.257+0000 7f7d00000640 1 -- 172.21.15.28:0/1250526920 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cf81ede40 con 0x7f7cf80bf990 2024-02-21T23:10:11.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.258+0000 7f7d00000640 1 -- 172.21.15.28:0/1250526920 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cf813e160 con 0x7f7cf80bf990 2024-02-21T23:10:11.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.259+0000 7f7cf5600640 1 -- 172.21.15.28:0/1250526920 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7cec012b90 con 0x7f7cf80bf990 2024-02-21T23:10:11.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.260+0000 7f7cf5600640 1 --2- 172.21.15.28:0/1250526920 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7cdc06b4d0 0x7f7cdc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:11.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.260+0000 7f7cf5600640 1 -- 172.21.15.28:0/1250526920 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f7cec08aab0 con 0x7f7cf80bf990 2024-02-21T23:10:11.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.261+0000 7f7cfcc00640 1 --2- 172.21.15.28:0/1250526920 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7cdc06b4d0 0x7f7cdc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:11.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.262+0000 7f7cfcc00640 1 --2- 172.21.15.28:0/1250526920 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7cdc06b4d0 0x7f7cdc06d990 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f7cf813d700 tx=0x7f7ce8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:11.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.269+0000 7f7cf5600640 1 -- 172.21.15.28:0/1250526920 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7cec0594e0 con 0x7f7cf80bf990 2024-02-21T23:10:11.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.479+0000 7f7d00000640 1 -- 172.21.15.28:0/1250526920 --> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7cf8067f30 con 0x7f7cdc06b4d0 2024-02-21T23:10:11.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.514+0000 7f7cf5600640 1 -- 172.21.15.28:0/1250526920 <== mgr.4101 v2:172.21.15.81:6832/1848538577 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+489418 (secure 0 0 0) 0x7f7cf8067f30 con 0x7f7cdc06b4d0 2024-02-21T23:10:11.513 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:10:11.518 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-21T23:10:11.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.522+0000 7f7d00000640 1 -- 172.21.15.28:0/1250526920 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7cdc06b4d0 msgr2=0x7f7cdc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:11.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.522+0000 7f7d00000640 1 --2- 172.21.15.28:0/1250526920 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7cdc06b4d0 0x7f7cdc06d990 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f7cf813d700 tx=0x7f7ce8008040 comp rx=0 tx=0).stop 2024-02-21T23:10:11.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.522+0000 7f7d00000640 1 -- 172.21.15.28:0/1250526920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cf80bf990 msgr2=0x7f7cf81e2010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:11.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.522+0000 7f7d00000640 1 --2- 172.21.15.28:0/1250526920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cf80bf990 0x7f7cf81e2010 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7f7cec00d600 tx=0x7f7cec00dac0 comp rx=0 tx=0).stop 2024-02-21T23:10:11.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.523+0000 7f7d00000640 1 -- 172.21.15.28:0/1250526920 shutdown_connections 2024-02-21T23:10:11.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.523+0000 7f7d00000640 1 --2- 172.21.15.28:0/1250526920 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7cdc06b4d0 0x7f7cdc06d990 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:11.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.523+0000 7f7d00000640 1 --2- 172.21.15.28:0/1250526920 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7cf813e620 0x7f7cf8157750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:11.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.523+0000 7f7d00000640 1 --2- 172.21.15.28:0/1250526920 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7cf80c6460 0x7f7cf81ebc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:11.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.523+0000 7f7d00000640 1 --2- 172.21.15.28:0/1250526920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cf80bf990 0x7f7cf81e2010 unknown :-1 s=CLOSED pgs=1742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:11.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.523+0000 7f7d00000640 1 -- 172.21.15.28:0/1250526920 >> 172.21.15.28:0/1250526920 conn(0x7f7cf808b1c0 msgr2=0x7f7cf8153ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:11.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.523+0000 7f7d00000640 1 -- 172.21.15.28:0/1250526920 shutdown_connections 2024-02-21T23:10:11.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.523+0000 7f7d00000640 1 -- 172.21.15.28:0/1250526920 wait complete. 2024-02-21T23:10:11.540 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":378,"stamp":"2024-02-21T23:10:10.824304+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":593450,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":219,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":19,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":171,"ondisk_log_size":171,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":309,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":264816,"kb_used_data":15344,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754709904,"statfs":{"total":773094113280,"available":772822941696,"internally_reserved":0,"allocated":15712256,"data_stored":8763576,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":672,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002075"},"pg_stats":[{"pgid":"26.3d","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.386132+0000","last_change":"2024-02-21T23:10:05.386132+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.386132+0000","last_undegraded":"2024-02-21T23:10:05.386132+0000","last_fullsized":"2024-02-21T23:10:05.386132+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.3f","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.394150+0000","last_change":"2024-02-21T23:09:54.279018+0000","last_active":"2024-02-21T23:10:05.394150+0000","last_peered":"2024-02-21T23:10:05.394150+0000","last_clean":"2024-02-21T23:10:05.394150+0000","last_became_active":"2024-02-21T23:09:54.278712+0000","last_became_peered":"2024-02-21T23:09:54.278712+0000","last_unstale":"2024-02-21T23:10:05.394150+0000","last_undegraded":"2024-02-21T23:10:05.394150+0000","last_fullsized":"2024-02-21T23:10:05.394150+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:55:18.591062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.3c","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232651+0000","last_change":"2024-02-21T23:10:06.385434+0000","last_active":"2024-02-21T23:10:08.232651+0000","last_peered":"2024-02-21T23:10:08.232651+0000","last_clean":"2024-02-21T23:10:08.232651+0000","last_became_active":"2024-02-21T23:10:06.384878+0000","last_became_peered":"2024-02-21T23:10:06.384878+0000","last_unstale":"2024-02-21T23:10:08.232651+0000","last_undegraded":"2024-02-21T23:10:08.232651+0000","last_fullsized":"2024-02-21T23:10:08.232651+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:52:21.069235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3e","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.373373+0000","last_change":"2024-02-21T23:09:54.285889+0000","last_active":"2024-02-21T23:10:05.373373+0000","last_peered":"2024-02-21T23:10:05.373373+0000","last_clean":"2024-02-21T23:10:05.373373+0000","last_became_active":"2024-02-21T23:09:54.284810+0000","last_became_peered":"2024-02-21T23:09:54.284810+0000","last_unstale":"2024-02-21T23:10:05.373373+0000","last_undegraded":"2024-02-21T23:10:05.373373+0000","last_fullsized":"2024-02-21T23:10:05.373373+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:04:16.317089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.3f","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236866+0000","last_change":"2024-02-21T23:10:06.389219+0000","last_active":"2024-02-21T23:10:08.236866+0000","last_peered":"2024-02-21T23:10:08.236866+0000","last_clean":"2024-02-21T23:10:08.236866+0000","last_became_active":"2024-02-21T23:10:06.388931+0000","last_became_peered":"2024-02-21T23:10:06.388931+0000","last_unstale":"2024-02-21T23:10:08.236866+0000","last_undegraded":"2024-02-21T23:10:08.236866+0000","last_fullsized":"2024-02-21T23:10:08.236866+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:46:45.683446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.3d","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.392968+0000","last_change":"2024-02-21T23:09:54.263978+0000","last_active":"2024-02-21T23:10:05.392968+0000","last_peered":"2024-02-21T23:10:05.392968+0000","last_clean":"2024-02-21T23:10:05.392968+0000","last_became_active":"2024-02-21T23:09:54.263467+0000","last_became_peered":"2024-02-21T23:09:54.263467+0000","last_unstale":"2024-02-21T23:10:05.392968+0000","last_undegraded":"2024-02-21T23:10:05.392968+0000","last_fullsized":"2024-02-21T23:10:05.392968+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:46:35.436815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.3e","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236400+0000","last_change":"2024-02-21T23:10:06.406362+0000","last_active":"2024-02-21T23:10:08.236400+0000","last_peered":"2024-02-21T23:10:08.236400+0000","last_clean":"2024-02-21T23:10:08.236400+0000","last_became_active":"2024-02-21T23:10:06.405859+0000","last_became_peered":"2024-02-21T23:10:06.405859+0000","last_unstale":"2024-02-21T23:10:08.236400+0000","last_undegraded":"2024-02-21T23:10:08.236400+0000","last_fullsized":"2024-02-21T23:10:08.236400+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:09:17.354064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.3c","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.390902+0000","last_change":"2024-02-21T23:09:54.279400+0000","last_active":"2024-02-21T23:10:05.390902+0000","last_peered":"2024-02-21T23:10:05.390902+0000","last_clean":"2024-02-21T23:10:05.390902+0000","last_became_active":"2024-02-21T23:09:54.279076+0000","last_became_peered":"2024-02-21T23:09:54.279076+0000","last_unstale":"2024-02-21T23:10:05.390902+0000","last_undegraded":"2024-02-21T23:10:05.390902+0000","last_fullsized":"2024-02-21T23:10:05.390902+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:02:11.247387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.39","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.375184+0000","last_change":"2024-02-21T23:10:06.375038+0000","last_active":"2024-02-21T23:10:07.375184+0000","last_peered":"2024-02-21T23:10:07.375184+0000","last_clean":"2024-02-21T23:10:07.375184+0000","last_became_active":"2024-02-21T23:10:06.374714+0000","last_became_peered":"2024-02-21T23:10:06.374714+0000","last_unstale":"2024-02-21T23:10:07.375184+0000","last_undegraded":"2024-02-21T23:10:07.375184+0000","last_fullsized":"2024-02-21T23:10:07.375184+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:48:19.473212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.3b","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.233882+0000","last_change":"2024-02-21T23:09:54.271682+0000","last_active":"2024-02-21T23:10:08.233882+0000","last_peered":"2024-02-21T23:10:08.233882+0000","last_clean":"2024-02-21T23:10:08.233882+0000","last_became_active":"2024-02-21T23:09:54.271297+0000","last_became_peered":"2024-02-21T23:09:54.271297+0000","last_unstale":"2024-02-21T23:10:08.233882+0000","last_undegraded":"2024-02-21T23:10:08.233882+0000","last_fullsized":"2024-02-21T23:10:08.233882+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:16:10.796552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.38","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.362732+0000","last_change":"2024-02-21T23:10:05.362732+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.362732+0000","last_undegraded":"2024-02-21T23:10:05.362732+0000","last_fullsized":"2024-02-21T23:10:05.362732+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3a","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.369709+0000","last_change":"2024-02-21T23:09:54.281830+0000","last_active":"2024-02-21T23:10:05.369709+0000","last_peered":"2024-02-21T23:10:05.369709+0000","last_clean":"2024-02-21T23:10:05.369709+0000","last_became_active":"2024-02-21T23:09:54.281334+0000","last_became_peered":"2024-02-21T23:09:54.281334+0000","last_unstale":"2024-02-21T23:10:05.369709+0000","last_undegraded":"2024-02-21T23:10:05.369709+0000","last_fullsized":"2024-02-21T23:10:05.369709+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:07:24.304915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.3b","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.233350+0000","last_change":"2024-02-21T23:10:06.385935+0000","last_active":"2024-02-21T23:10:08.233350+0000","last_peered":"2024-02-21T23:10:08.233350+0000","last_clean":"2024-02-21T23:10:08.233350+0000","last_became_active":"2024-02-21T23:10:06.385657+0000","last_became_peered":"2024-02-21T23:10:06.385657+0000","last_unstale":"2024-02-21T23:10:08.233350+0000","last_undegraded":"2024-02-21T23:10:08.233350+0000","last_fullsized":"2024-02-21T23:10:08.233350+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:27:47.158633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.39","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.233971+0000","last_change":"2024-02-21T23:09:54.277806+0000","last_active":"2024-02-21T23:10:08.233971+0000","last_peered":"2024-02-21T23:10:08.233971+0000","last_clean":"2024-02-21T23:10:08.233971+0000","last_became_active":"2024-02-21T23:09:54.277446+0000","last_became_peered":"2024-02-21T23:09:54.277446+0000","last_unstale":"2024-02-21T23:10:08.233971+0000","last_undegraded":"2024-02-21T23:10:08.233971+0000","last_fullsized":"2024-02-21T23:10:08.233971+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:59:34.646775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.3a","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.231962+0000","last_change":"2024-02-21T23:10:06.394386+0000","last_active":"2024-02-21T23:10:08.231962+0000","last_peered":"2024-02-21T23:10:08.231962+0000","last_clean":"2024-02-21T23:10:08.231962+0000","last_became_active":"2024-02-21T23:10:06.391655+0000","last_became_peered":"2024-02-21T23:10:06.391655+0000","last_unstale":"2024-02-21T23:10:08.231962+0000","last_undegraded":"2024-02-21T23:10:08.231962+0000","last_fullsized":"2024-02-21T23:10:08.231962+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:15:42.134184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.38","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232805+0000","last_change":"2024-02-21T23:09:54.286729+0000","last_active":"2024-02-21T23:10:08.232805+0000","last_peered":"2024-02-21T23:10:08.232805+0000","last_clean":"2024-02-21T23:10:08.232805+0000","last_became_active":"2024-02-21T23:09:54.286427+0000","last_became_peered":"2024-02-21T23:09:54.286427+0000","last_unstale":"2024-02-21T23:10:08.232805+0000","last_undegraded":"2024-02-21T23:10:08.232805+0000","last_fullsized":"2024-02-21T23:10:08.232805+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:13:27.311923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.35","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.373594+0000","last_change":"2024-02-21T23:10:05.373594+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.373594+0000","last_undegraded":"2024-02-21T23:10:05.373594+0000","last_fullsized":"2024-02-21T23:10:05.373594+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.37","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.237533+0000","last_change":"2024-02-21T23:09:54.286612+0000","last_active":"2024-02-21T23:10:08.237533+0000","last_peered":"2024-02-21T23:10:08.237533+0000","last_clean":"2024-02-21T23:10:08.237533+0000","last_became_active":"2024-02-21T23:09:54.286306+0000","last_became_peered":"2024-02-21T23:09:54.286306+0000","last_unstale":"2024-02-21T23:10:08.237533+0000","last_undegraded":"2024-02-21T23:10:08.237533+0000","last_fullsized":"2024-02-21T23:10:08.237533+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:04:16.317089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.34","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232897+0000","last_change":"2024-02-21T23:10:06.385190+0000","last_active":"2024-02-21T23:10:08.232897+0000","last_peered":"2024-02-21T23:10:08.232897+0000","last_clean":"2024-02-21T23:10:08.232897+0000","last_became_active":"2024-02-21T23:10:06.384496+0000","last_became_peered":"2024-02-21T23:10:06.384496+0000","last_unstale":"2024-02-21T23:10:08.232897+0000","last_undegraded":"2024-02-21T23:10:08.232897+0000","last_fullsized":"2024-02-21T23:10:08.232897+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:28:15.352760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.36","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.237062+0000","last_change":"2024-02-21T23:09:54.288019+0000","last_active":"2024-02-21T23:10:08.237062+0000","last_peered":"2024-02-21T23:10:08.237062+0000","last_clean":"2024-02-21T23:10:08.237062+0000","last_became_active":"2024-02-21T23:09:54.287742+0000","last_became_peered":"2024-02-21T23:09:54.287742+0000","last_unstale":"2024-02-21T23:10:08.237062+0000","last_undegraded":"2024-02-21T23:10:08.237062+0000","last_fullsized":"2024-02-21T23:10:08.237062+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:13:33.442227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.37","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236970+0000","last_change":"2024-02-21T23:10:06.398025+0000","last_active":"2024-02-21T23:10:08.236970+0000","last_peered":"2024-02-21T23:10:08.236970+0000","last_clean":"2024-02-21T23:10:08.236970+0000","last_became_active":"2024-02-21T23:10:06.395723+0000","last_became_peered":"2024-02-21T23:10:06.395723+0000","last_unstale":"2024-02-21T23:10:08.236970+0000","last_undegraded":"2024-02-21T23:10:08.236970+0000","last_fullsized":"2024-02-21T23:10:08.236970+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:50:33.682312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.35","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.378855+0000","last_change":"2024-02-21T23:09:54.283918+0000","last_active":"2024-02-21T23:10:05.378855+0000","last_peered":"2024-02-21T23:10:05.378855+0000","last_clean":"2024-02-21T23:10:05.378855+0000","last_became_active":"2024-02-21T23:09:54.280599+0000","last_became_peered":"2024-02-21T23:09:54.280599+0000","last_unstale":"2024-02-21T23:10:05.378855+0000","last_undegraded":"2024-02-21T23:10:05.378855+0000","last_fullsized":"2024-02-21T23:10:05.378855+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:33:07.362294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.36","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234837+0000","last_change":"2024-02-21T23:10:06.377423+0000","last_active":"2024-02-21T23:10:08.234837+0000","last_peered":"2024-02-21T23:10:08.234837+0000","last_clean":"2024-02-21T23:10:08.234837+0000","last_became_active":"2024-02-21T23:10:06.376998+0000","last_became_peered":"2024-02-21T23:10:06.376998+0000","last_unstale":"2024-02-21T23:10:08.234837+0000","last_undegraded":"2024-02-21T23:10:08.234837+0000","last_fullsized":"2024-02-21T23:10:08.234837+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:26:32.419695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.34","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.373186+0000","last_change":"2024-02-21T23:09:54.282846+0000","last_active":"2024-02-21T23:10:05.373186+0000","last_peered":"2024-02-21T23:10:05.373186+0000","last_clean":"2024-02-21T23:10:05.373186+0000","last_became_active":"2024-02-21T23:09:54.282117+0000","last_became_peered":"2024-02-21T23:09:54.282117+0000","last_unstale":"2024-02-21T23:10:05.373186+0000","last_undegraded":"2024-02-21T23:10:05.373186+0000","last_fullsized":"2024-02-21T23:10:05.373186+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:33:07.362294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.31","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234391+0000","last_change":"2024-02-21T23:10:06.382496+0000","last_active":"2024-02-21T23:10:08.234391+0000","last_peered":"2024-02-21T23:10:08.234391+0000","last_clean":"2024-02-21T23:10:08.234391+0000","last_became_active":"2024-02-21T23:10:06.379727+0000","last_became_peered":"2024-02-21T23:10:06.379727+0000","last_unstale":"2024-02-21T23:10:08.234391+0000","last_undegraded":"2024-02-21T23:10:08.234391+0000","last_fullsized":"2024-02-21T23:10:08.234391+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:10:21.186085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.33","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234540+0000","last_change":"2024-02-21T23:09:54.284818+0000","last_active":"2024-02-21T23:10:08.234540+0000","last_peered":"2024-02-21T23:10:08.234540+0000","last_clean":"2024-02-21T23:10:08.234540+0000","last_became_active":"2024-02-21T23:09:54.284161+0000","last_became_peered":"2024-02-21T23:09:54.284161+0000","last_unstale":"2024-02-21T23:10:08.234540+0000","last_undegraded":"2024-02-21T23:10:08.234540+0000","last_fullsized":"2024-02-21T23:10:08.234540+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:02:12.340921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.30","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.231818+0000","last_change":"2024-02-21T23:10:06.388372+0000","last_active":"2024-02-21T23:10:08.231818+0000","last_peered":"2024-02-21T23:10:08.231818+0000","last_clean":"2024-02-21T23:10:08.231818+0000","last_became_active":"2024-02-21T23:10:06.388099+0000","last_became_peered":"2024-02-21T23:10:06.388099+0000","last_unstale":"2024-02-21T23:10:08.231818+0000","last_undegraded":"2024-02-21T23:10:08.231818+0000","last_fullsized":"2024-02-21T23:10:08.231818+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:55:32.267269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.32","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.231579+0000","last_change":"2024-02-21T23:09:54.272136+0000","last_active":"2024-02-21T23:10:08.231579+0000","last_peered":"2024-02-21T23:10:08.231579+0000","last_clean":"2024-02-21T23:10:08.231579+0000","last_became_active":"2024-02-21T23:09:54.271097+0000","last_became_peered":"2024-02-21T23:09:54.271097+0000","last_unstale":"2024-02-21T23:10:08.231579+0000","last_undegraded":"2024-02-21T23:10:08.231579+0000","last_fullsized":"2024-02-21T23:10:08.231579+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:30:45.437334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.33","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.381970+0000","last_change":"2024-02-21T23:10:05.381970+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.381970+0000","last_undegraded":"2024-02-21T23:10:05.381970+0000","last_fullsized":"2024-02-21T23:10:05.381970+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.31","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.377899+0000","last_change":"2024-02-21T23:09:54.281268+0000","last_active":"2024-02-21T23:10:05.377899+0000","last_peered":"2024-02-21T23:10:05.377899+0000","last_clean":"2024-02-21T23:10:05.377899+0000","last_became_active":"2024-02-21T23:09:54.280946+0000","last_became_peered":"2024-02-21T23:09:54.280946+0000","last_unstale":"2024-02-21T23:10:05.377899+0000","last_undegraded":"2024-02-21T23:10:05.377899+0000","last_fullsized":"2024-02-21T23:10:05.377899+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:03:44.493909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.32","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.374763+0000","last_change":"2024-02-21T23:10:06.381260+0000","last_active":"2024-02-21T23:10:07.374763+0000","last_peered":"2024-02-21T23:10:07.374763+0000","last_clean":"2024-02-21T23:10:07.374763+0000","last_became_active":"2024-02-21T23:10:06.380936+0000","last_became_peered":"2024-02-21T23:10:06.380936+0000","last_unstale":"2024-02-21T23:10:07.374763+0000","last_undegraded":"2024-02-21T23:10:07.374763+0000","last_fullsized":"2024-02-21T23:10:07.374763+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:41:14.290670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.30","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.387425+0000","last_change":"2024-02-21T23:09:54.278731+0000","last_active":"2024-02-21T23:10:05.387425+0000","last_peered":"2024-02-21T23:10:05.387425+0000","last_clean":"2024-02-21T23:10:05.387425+0000","last_became_active":"2024-02-21T23:09:54.278390+0000","last_became_peered":"2024-02-21T23:09:54.278390+0000","last_unstale":"2024-02-21T23:10:05.387425+0000","last_undegraded":"2024-02-21T23:10:05.387425+0000","last_fullsized":"2024-02-21T23:10:05.387425+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:28:54.079553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.2d","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.382164+0000","last_change":"2024-02-21T23:10:05.382164+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.382164+0000","last_undegraded":"2024-02-21T23:10:05.382164+0000","last_fullsized":"2024-02-21T23:10:05.382164+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.2f","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.237800+0000","last_change":"2024-02-21T23:09:54.285947+0000","last_active":"2024-02-21T23:10:08.237800+0000","last_peered":"2024-02-21T23:10:08.237800+0000","last_clean":"2024-02-21T23:10:08.237800+0000","last_became_active":"2024-02-21T23:09:54.285658+0000","last_became_peered":"2024-02-21T23:09:54.285658+0000","last_unstale":"2024-02-21T23:10:08.237800+0000","last_undegraded":"2024-02-21T23:10:08.237800+0000","last_fullsized":"2024-02-21T23:10:08.237800+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:20:31.125684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2c","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236540+0000","last_change":"2024-02-21T23:10:06.376864+0000","last_active":"2024-02-21T23:10:08.236540+0000","last_peered":"2024-02-21T23:10:08.236540+0000","last_clean":"2024-02-21T23:10:08.236540+0000","last_became_active":"2024-02-21T23:10:06.376486+0000","last_became_peered":"2024-02-21T23:10:06.376486+0000","last_unstale":"2024-02-21T23:10:08.236540+0000","last_undegraded":"2024-02-21T23:10:08.236540+0000","last_fullsized":"2024-02-21T23:10:08.236540+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:22:04.227521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.2e","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.391407+0000","last_change":"2024-02-21T23:09:54.287286+0000","last_active":"2024-02-21T23:10:05.391407+0000","last_peered":"2024-02-21T23:10:05.391407+0000","last_clean":"2024-02-21T23:10:05.391407+0000","last_became_active":"2024-02-21T23:09:54.286924+0000","last_became_peered":"2024-02-21T23:09:54.286924+0000","last_unstale":"2024-02-21T23:10:05.391407+0000","last_undegraded":"2024-02-21T23:10:05.391407+0000","last_fullsized":"2024-02-21T23:10:05.391407+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:47:29.349669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.2f","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.235953+0000","last_change":"2024-02-21T23:10:06.377113+0000","last_active":"2024-02-21T23:10:08.235953+0000","last_peered":"2024-02-21T23:10:08.235953+0000","last_clean":"2024-02-21T23:10:08.235953+0000","last_became_active":"2024-02-21T23:10:06.376790+0000","last_became_peered":"2024-02-21T23:10:06.376790+0000","last_unstale":"2024-02-21T23:10:08.235953+0000","last_undegraded":"2024-02-21T23:10:08.235953+0000","last_fullsized":"2024-02-21T23:10:08.235953+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:38:03.009039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.2d","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.379032+0000","last_change":"2024-02-21T23:09:54.279884+0000","last_active":"2024-02-21T23:10:05.379032+0000","last_peered":"2024-02-21T23:10:05.379032+0000","last_clean":"2024-02-21T23:10:05.379032+0000","last_became_active":"2024-02-21T23:09:54.276608+0000","last_became_peered":"2024-02-21T23:09:54.276608+0000","last_unstale":"2024-02-21T23:10:05.379032+0000","last_undegraded":"2024-02-21T23:10:05.379032+0000","last_fullsized":"2024-02-21T23:10:05.379032+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:05:09.189420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.2e","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.383479+0000","last_change":"2024-02-21T23:10:05.383479+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.383479+0000","last_undegraded":"2024-02-21T23:10:05.383479+0000","last_fullsized":"2024-02-21T23:10:05.383479+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.2c","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.375678+0000","last_change":"2024-02-21T23:09:54.288944+0000","last_active":"2024-02-21T23:10:07.375678+0000","last_peered":"2024-02-21T23:10:07.375678+0000","last_clean":"2024-02-21T23:10:07.375678+0000","last_became_active":"2024-02-21T23:09:54.285913+0000","last_became_peered":"2024-02-21T23:09:54.285913+0000","last_unstale":"2024-02-21T23:10:07.375678+0000","last_undegraded":"2024-02-21T23:10:07.375678+0000","last_fullsized":"2024-02-21T23:10:07.375678+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:39:33.609273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.29","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.363048+0000","last_change":"2024-02-21T23:10:05.363048+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.363048+0000","last_undegraded":"2024-02-21T23:10:05.363048+0000","last_fullsized":"2024-02-21T23:10:05.363048+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.2b","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234769+0000","last_change":"2024-02-21T23:09:54.285036+0000","last_active":"2024-02-21T23:10:08.234769+0000","last_peered":"2024-02-21T23:10:08.234769+0000","last_clean":"2024-02-21T23:10:08.234769+0000","last_became_active":"2024-02-21T23:09:54.284542+0000","last_became_peered":"2024-02-21T23:09:54.284542+0000","last_unstale":"2024-02-21T23:10:08.234769+0000","last_undegraded":"2024-02-21T23:10:08.234769+0000","last_fullsized":"2024-02-21T23:10:08.234769+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T11:05:01.241636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.28","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234521+0000","last_change":"2024-02-21T23:10:06.396769+0000","last_active":"2024-02-21T23:10:08.234521+0000","last_peered":"2024-02-21T23:10:08.234521+0000","last_clean":"2024-02-21T23:10:08.234521+0000","last_became_active":"2024-02-21T23:10:06.384723+0000","last_became_peered":"2024-02-21T23:10:06.384723+0000","last_unstale":"2024-02-21T23:10:08.234521+0000","last_undegraded":"2024-02-21T23:10:08.234521+0000","last_fullsized":"2024-02-21T23:10:08.234521+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:45:13.867812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2a","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.370120+0000","last_change":"2024-02-21T23:09:54.273350+0000","last_active":"2024-02-21T23:10:05.370120+0000","last_peered":"2024-02-21T23:10:05.370120+0000","last_clean":"2024-02-21T23:10:05.370120+0000","last_became_active":"2024-02-21T23:09:54.273022+0000","last_became_peered":"2024-02-21T23:09:54.273022+0000","last_unstale":"2024-02-21T23:10:05.370120+0000","last_undegraded":"2024-02-21T23:10:05.370120+0000","last_fullsized":"2024-02-21T23:10:05.370120+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:34:28.642087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.2b","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.235284+0000","last_change":"2024-02-21T23:10:06.394830+0000","last_active":"2024-02-21T23:10:08.235284+0000","last_peered":"2024-02-21T23:10:08.235284+0000","last_clean":"2024-02-21T23:10:08.235284+0000","last_became_active":"2024-02-21T23:10:06.386513+0000","last_became_peered":"2024-02-21T23:10:06.386513+0000","last_unstale":"2024-02-21T23:10:08.235284+0000","last_undegraded":"2024-02-21T23:10:08.235284+0000","last_fullsized":"2024-02-21T23:10:08.235284+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:30:09.767949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.29","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.230852+0000","last_change":"2024-02-21T23:09:54.279718+0000","last_active":"2024-02-21T23:10:08.230852+0000","last_peered":"2024-02-21T23:10:08.230852+0000","last_clean":"2024-02-21T23:10:08.230852+0000","last_became_active":"2024-02-21T23:09:54.278989+0000","last_became_peered":"2024-02-21T23:09:54.278989+0000","last_unstale":"2024-02-21T23:10:08.230852+0000","last_undegraded":"2024-02-21T23:10:08.230852+0000","last_fullsized":"2024-02-21T23:10:08.230852+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:36:41.922548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.2a","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.235326+0000","last_change":"2024-02-21T23:10:06.397414+0000","last_active":"2024-02-21T23:10:08.235326+0000","last_peered":"2024-02-21T23:10:08.235326+0000","last_clean":"2024-02-21T23:10:08.235326+0000","last_became_active":"2024-02-21T23:10:06.396982+0000","last_became_peered":"2024-02-21T23:10:06.396982+0000","last_unstale":"2024-02-21T23:10:08.235326+0000","last_undegraded":"2024-02-21T23:10:08.235326+0000","last_fullsized":"2024-02-21T23:10:08.235326+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:05:37.143681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.28","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.387543+0000","last_change":"2024-02-21T23:09:54.280624+0000","last_active":"2024-02-21T23:10:05.387543+0000","last_peered":"2024-02-21T23:10:05.387543+0000","last_clean":"2024-02-21T23:10:05.387543+0000","last_became_active":"2024-02-21T23:09:54.275644+0000","last_became_peered":"2024-02-21T23:09:54.275644+0000","last_unstale":"2024-02-21T23:10:05.387543+0000","last_undegraded":"2024-02-21T23:10:05.387543+0000","last_fullsized":"2024-02-21T23:10:05.387543+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:23:27.511315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.25","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.365921+0000","last_change":"2024-02-21T23:10:05.365921+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.365921+0000","last_undegraded":"2024-02-21T23:10:05.365921+0000","last_fullsized":"2024-02-21T23:10:05.365921+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.27","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236738+0000","last_change":"2024-02-21T23:09:54.275341+0000","last_active":"2024-02-21T23:10:08.236738+0000","last_peered":"2024-02-21T23:10:08.236738+0000","last_clean":"2024-02-21T23:10:08.236738+0000","last_became_active":"2024-02-21T23:09:54.275010+0000","last_became_peered":"2024-02-21T23:09:54.275010+0000","last_unstale":"2024-02-21T23:10:08.236738+0000","last_undegraded":"2024-02-21T23:10:08.236738+0000","last_fullsized":"2024-02-21T23:10:08.236738+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:42:08.123188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.24","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236163+0000","last_change":"2024-02-21T23:10:06.386616+0000","last_active":"2024-02-21T23:10:08.236163+0000","last_peered":"2024-02-21T23:10:08.236163+0000","last_clean":"2024-02-21T23:10:08.236163+0000","last_became_active":"2024-02-21T23:10:06.386339+0000","last_became_peered":"2024-02-21T23:10:06.386339+0000","last_unstale":"2024-02-21T23:10:08.236163+0000","last_undegraded":"2024-02-21T23:10:08.236163+0000","last_fullsized":"2024-02-21T23:10:08.236163+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:44:41.892333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.26","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.376034+0000","last_change":"2024-02-21T23:09:54.260902+0000","last_active":"2024-02-21T23:10:07.376034+0000","last_peered":"2024-02-21T23:10:07.376034+0000","last_clean":"2024-02-21T23:10:07.376034+0000","last_became_active":"2024-02-21T23:09:54.260084+0000","last_became_peered":"2024-02-21T23:09:54.260084+0000","last_unstale":"2024-02-21T23:10:07.376034+0000","last_undegraded":"2024-02-21T23:10:07.376034+0000","last_fullsized":"2024-02-21T23:10:07.376034+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:04:05.858817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.27","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.375938+0000","last_change":"2024-02-21T23:10:06.374537+0000","last_active":"2024-02-21T23:10:07.375938+0000","last_peered":"2024-02-21T23:10:07.375938+0000","last_clean":"2024-02-21T23:10:07.375938+0000","last_became_active":"2024-02-21T23:10:06.374201+0000","last_became_peered":"2024-02-21T23:10:06.374201+0000","last_unstale":"2024-02-21T23:10:07.375938+0000","last_undegraded":"2024-02-21T23:10:07.375938+0000","last_fullsized":"2024-02-21T23:10:07.375938+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:52:55.561353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.25","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.373077+0000","last_change":"2024-02-21T23:09:54.284116+0000","last_active":"2024-02-21T23:10:05.373077+0000","last_peered":"2024-02-21T23:10:05.373077+0000","last_clean":"2024-02-21T23:10:05.373077+0000","last_became_active":"2024-02-21T23:09:54.283802+0000","last_became_peered":"2024-02-21T23:09:54.283802+0000","last_unstale":"2024-02-21T23:10:05.373077+0000","last_undegraded":"2024-02-21T23:10:05.373077+0000","last_fullsized":"2024-02-21T23:10:05.373077+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:05:09.189420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.26","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.374040+0000","last_change":"2024-02-21T23:10:05.374040+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.374040+0000","last_undegraded":"2024-02-21T23:10:05.374040+0000","last_fullsized":"2024-02-21T23:10:05.374040+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.24","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236346+0000","last_change":"2024-02-21T23:09:54.287481+0000","last_active":"2024-02-21T23:10:08.236346+0000","last_peered":"2024-02-21T23:10:08.236346+0000","last_clean":"2024-02-21T23:10:08.236346+0000","last_became_active":"2024-02-21T23:09:54.287137+0000","last_became_peered":"2024-02-21T23:09:54.287137+0000","last_unstale":"2024-02-21T23:10:08.236346+0000","last_undegraded":"2024-02-21T23:10:08.236346+0000","last_fullsized":"2024-02-21T23:10:08.236346+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:50:46.472002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.21","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.374988+0000","last_change":"2024-02-21T23:10:06.370867+0000","last_active":"2024-02-21T23:10:07.374988+0000","last_peered":"2024-02-21T23:10:07.374988+0000","last_clean":"2024-02-21T23:10:07.374988+0000","last_became_active":"2024-02-21T23:10:06.370583+0000","last_became_peered":"2024-02-21T23:10:06.370583+0000","last_unstale":"2024-02-21T23:10:07.374988+0000","last_undegraded":"2024-02-21T23:10:07.374988+0000","last_fullsized":"2024-02-21T23:10:07.374988+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:18:29.588662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.23","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.372190+0000","last_change":"2024-02-21T23:09:54.281613+0000","last_active":"2024-02-21T23:10:05.372190+0000","last_peered":"2024-02-21T23:10:05.372190+0000","last_clean":"2024-02-21T23:10:05.372190+0000","last_became_active":"2024-02-21T23:09:54.281316+0000","last_became_peered":"2024-02-21T23:09:54.281316+0000","last_unstale":"2024-02-21T23:10:05.372190+0000","last_undegraded":"2024-02-21T23:10:05.372190+0000","last_fullsized":"2024-02-21T23:10:05.372190+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:13:33.442227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.20","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232111+0000","last_change":"2024-02-21T23:10:06.382019+0000","last_active":"2024-02-21T23:10:08.232111+0000","last_peered":"2024-02-21T23:10:08.232111+0000","last_clean":"2024-02-21T23:10:08.232111+0000","last_became_active":"2024-02-21T23:10:06.381739+0000","last_became_peered":"2024-02-21T23:10:06.381739+0000","last_unstale":"2024-02-21T23:10:08.232111+0000","last_undegraded":"2024-02-21T23:10:08.232111+0000","last_fullsized":"2024-02-21T23:10:08.232111+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:25:17.018194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.22","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.390353+0000","last_change":"2024-02-21T23:09:54.275665+0000","last_active":"2024-02-21T23:10:05.390353+0000","last_peered":"2024-02-21T23:10:05.390353+0000","last_clean":"2024-02-21T23:10:05.390353+0000","last_became_active":"2024-02-21T23:09:54.275143+0000","last_became_peered":"2024-02-21T23:09:54.275143+0000","last_unstale":"2024-02-21T23:10:05.390353+0000","last_undegraded":"2024-02-21T23:10:05.390353+0000","last_fullsized":"2024-02-21T23:10:05.390353+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:37:42.976792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.23","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.371720+0000","last_change":"2024-02-21T23:10:05.371720+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.371720+0000","last_undegraded":"2024-02-21T23:10:05.371720+0000","last_fullsized":"2024-02-21T23:10:05.371720+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.21","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.378116+0000","last_change":"2024-02-21T23:09:54.281200+0000","last_active":"2024-02-21T23:10:05.378116+0000","last_peered":"2024-02-21T23:10:05.378116+0000","last_clean":"2024-02-21T23:10:05.378116+0000","last_became_active":"2024-02-21T23:09:54.277867+0000","last_became_peered":"2024-02-21T23:09:54.277867+0000","last_unstale":"2024-02-21T23:10:05.378116+0000","last_undegraded":"2024-02-21T23:10:05.378116+0000","last_fullsized":"2024-02-21T23:10:05.378116+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:59:34.646775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.22","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.375181+0000","last_change":"2024-02-21T23:10:05.375181+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.375181+0000","last_undegraded":"2024-02-21T23:10:05.375181+0000","last_fullsized":"2024-02-21T23:10:05.375181+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.20","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.390496+0000","last_change":"2024-02-21T23:09:54.276078+0000","last_active":"2024-02-21T23:10:05.390496+0000","last_peered":"2024-02-21T23:10:05.390496+0000","last_clean":"2024-02-21T23:10:05.390496+0000","last_became_active":"2024-02-21T23:09:54.275724+0000","last_became_peered":"2024-02-21T23:09:54.275724+0000","last_unstale":"2024-02-21T23:10:05.390496+0000","last_undegraded":"2024-02-21T23:10:05.390496+0000","last_fullsized":"2024-02-21T23:10:05.390496+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:48:07.369182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.377674+0000","last_change":"2024-02-21T23:10:05.377674+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.377674+0000","last_undegraded":"2024-02-21T23:10:05.377674+0000","last_fullsized":"2024-02-21T23:10:05.377674+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.230111+0000","last_change":"2024-02-21T23:10:05.385230+0000","last_active":"2024-02-21T23:10:08.230111+0000","last_peered":"2024-02-21T23:10:08.230111+0000","last_clean":"2024-02-21T23:10:08.230111+0000","last_became_active":"2024-02-21T23:10:05.384017+0000","last_became_peered":"2024-02-21T23:10:05.384017+0000","last_unstale":"2024-02-21T23:10:08.230111+0000","last_undegraded":"2024-02-21T23:10:08.230111+0000","last_fullsized":"2024-02-21T23:10:08.230111+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:45:12.852122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.10","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.368345+0000","last_change":"2024-02-21T23:09:54.278972+0000","last_active":"2024-02-21T23:10:05.368345+0000","last_peered":"2024-02-21T23:10:05.368345+0000","last_clean":"2024-02-21T23:10:05.368345+0000","last_became_active":"2024-02-21T23:09:54.278146+0000","last_became_peered":"2024-02-21T23:09:54.278146+0000","last_unstale":"2024-02-21T23:10:05.368345+0000","last_undegraded":"2024-02-21T23:10:05.368345+0000","last_fullsized":"2024-02-21T23:10:05.368345+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:59:34.646775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"150'4","reported_seq":22,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.394358+0000","last_change":"2024-02-21T23:09:53.288506+0000","last_active":"2024-02-21T23:10:05.394358+0000","last_peered":"2024-02-21T23:10:05.394358+0000","last_clean":"2024-02-21T23:10:05.394358+0000","last_became_active":"2024-02-21T23:09:53.288199+0000","last_became_peered":"2024-02-21T23:09:53.288199+0000","last_unstale":"2024-02-21T23:10:05.394358+0000","last_undegraded":"2024-02-21T23:10:05.394358+0000","last_fullsized":"2024-02-21T23:10:05.394358+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:18:16.467715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.13","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.378353+0000","last_change":"2024-02-21T23:10:05.378353+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.378353+0000","last_undegraded":"2024-02-21T23:10:05.378353+0000","last_fullsized":"2024-02-21T23:10:05.378353+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.10","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.386340+0000","last_change":"2024-02-21T23:10:05.382619+0000","last_active":"2024-02-21T23:10:05.386340+0000","last_peered":"2024-02-21T23:10:05.386340+0000","last_clean":"2024-02-21T23:10:05.386340+0000","last_became_active":"2024-02-21T23:10:05.382336+0000","last_became_peered":"2024-02-21T23:10:05.382336+0000","last_unstale":"2024-02-21T23:10:05.386340+0000","last_undegraded":"2024-02-21T23:10:05.386340+0000","last_fullsized":"2024-02-21T23:10:05.386340+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:11:55.677946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.231615+0000","last_change":"2024-02-21T23:09:54.278032+0000","last_active":"2024-02-21T23:10:08.231615+0000","last_peered":"2024-02-21T23:10:08.231615+0000","last_clean":"2024-02-21T23:10:08.231615+0000","last_became_active":"2024-02-21T23:09:54.277741+0000","last_became_peered":"2024-02-21T23:09:54.277741+0000","last_unstale":"2024-02-21T23:10:08.231615+0000","last_undegraded":"2024-02-21T23:10:08.231615+0000","last_fullsized":"2024-02-21T23:10:08.231615+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:43:56.610613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"150'1","reported_seq":23,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232900+0000","last_change":"2024-02-21T23:09:53.279374+0000","last_active":"2024-02-21T23:10:08.232900+0000","last_peered":"2024-02-21T23:10:08.232900+0000","last_clean":"2024-02-21T23:10:08.232900+0000","last_became_active":"2024-02-21T23:09:53.279053+0000","last_became_peered":"2024-02-21T23:09:53.279053+0000","last_unstale":"2024-02-21T23:10:08.232900+0000","last_undegraded":"2024-02-21T23:10:08.232900+0000","last_fullsized":"2024-02-21T23:10:08.232900+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:04:15.300172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.10","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.375571+0000","last_change":"2024-02-21T23:10:06.375103+0000","last_active":"2024-02-21T23:10:07.375571+0000","last_peered":"2024-02-21T23:10:07.375571+0000","last_clean":"2024-02-21T23:10:07.375571+0000","last_became_active":"2024-02-21T23:10:06.374814+0000","last_became_peered":"2024-02-21T23:10:06.374814+0000","last_unstale":"2024-02-21T23:10:07.375571+0000","last_undegraded":"2024-02-21T23:10:07.375571+0000","last_fullsized":"2024-02-21T23:10:07.375571+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:42:32.100185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.13","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.395209+0000","last_change":"2024-02-21T23:10:05.392497+0000","last_active":"2024-02-21T23:10:05.395209+0000","last_peered":"2024-02-21T23:10:05.395209+0000","last_clean":"2024-02-21T23:10:05.395209+0000","last_became_active":"2024-02-21T23:10:05.392183+0000","last_became_peered":"2024-02-21T23:10:05.392183+0000","last_unstale":"2024-02-21T23:10:05.395209+0000","last_undegraded":"2024-02-21T23:10:05.395209+0000","last_fullsized":"2024-02-21T23:10:05.395209+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:13:55.723872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.231985+0000","last_change":"2024-02-21T23:09:54.263915+0000","last_active":"2024-02-21T23:10:08.231985+0000","last_peered":"2024-02-21T23:10:08.231985+0000","last_clean":"2024-02-21T23:10:08.231985+0000","last_became_active":"2024-02-21T23:09:54.263634+0000","last_became_peered":"2024-02-21T23:09:54.263634+0000","last_unstale":"2024-02-21T23:10:08.231985+0000","last_undegraded":"2024-02-21T23:10:08.231985+0000","last_fullsized":"2024-02-21T23:10:08.231985+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:47:34.065041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.1d","version":"150'2","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.379180+0000","last_change":"2024-02-21T23:09:53.272110+0000","last_active":"2024-02-21T23:10:05.379180+0000","last_peered":"2024-02-21T23:10:05.379180+0000","last_clean":"2024-02-21T23:10:05.379180+0000","last_became_active":"2024-02-21T23:09:53.271641+0000","last_became_peered":"2024-02-21T23:09:53.271641+0000","last_unstale":"2024-02-21T23:10:05.379180+0000","last_undegraded":"2024-02-21T23:10:05.379180+0000","last_fullsized":"2024-02-21T23:10:05.379180+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:46:34.419898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.231438+0000","last_change":"2024-02-21T23:10:06.380458+0000","last_active":"2024-02-21T23:10:08.231438+0000","last_peered":"2024-02-21T23:10:08.231438+0000","last_clean":"2024-02-21T23:10:08.231438+0000","last_became_active":"2024-02-21T23:10:06.380137+0000","last_became_peered":"2024-02-21T23:10:06.380137+0000","last_unstale":"2024-02-21T23:10:08.231438+0000","last_undegraded":"2024-02-21T23:10:08.231438+0000","last_fullsized":"2024-02-21T23:10:08.231438+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:51:13.639406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236172+0000","last_change":"2024-02-21T23:10:05.393823+0000","last_active":"2024-02-21T23:10:08.236172+0000","last_peered":"2024-02-21T23:10:08.236172+0000","last_clean":"2024-02-21T23:10:08.236172+0000","last_became_active":"2024-02-21T23:10:05.390445+0000","last_became_peered":"2024-02-21T23:10:05.390445+0000","last_unstale":"2024-02-21T23:10:08.236172+0000","last_undegraded":"2024-02-21T23:10:08.236172+0000","last_fullsized":"2024-02-21T23:10:08.236172+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:30:56.525674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.13","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.372349+0000","last_change":"2024-02-21T23:09:54.275867+0000","last_active":"2024-02-21T23:10:05.372349+0000","last_peered":"2024-02-21T23:10:05.372349+0000","last_clean":"2024-02-21T23:10:05.372349+0000","last_became_active":"2024-02-21T23:09:54.275591+0000","last_became_peered":"2024-02-21T23:09:54.275591+0000","last_unstale":"2024-02-21T23:10:05.372349+0000","last_undegraded":"2024-02-21T23:10:05.372349+0000","last_fullsized":"2024-02-21T23:10:05.372349+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:42:16.988326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1c","version":"150'2","reported_seq":24,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.375729+0000","last_change":"2024-02-21T23:09:53.274692+0000","last_active":"2024-02-21T23:10:07.375729+0000","last_peered":"2024-02-21T23:10:07.375729+0000","last_clean":"2024-02-21T23:10:07.375729+0000","last_became_active":"2024-02-21T23:09:53.274395+0000","last_became_peered":"2024-02-21T23:09:53.274395+0000","last_unstale":"2024-02-21T23:10:07.375729+0000","last_undegraded":"2024-02-21T23:10:07.375729+0000","last_fullsized":"2024-02-21T23:10:07.375729+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:40:18.453921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.382764+0000","last_change":"2024-02-21T23:10:05.382764+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.382764+0000","last_undegraded":"2024-02-21T23:10:05.382764+0000","last_fullsized":"2024-02-21T23:10:05.382764+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.15","version":"154'3","reported_seq":17,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.233612+0000","last_change":"2024-02-21T23:10:05.374604+0000","last_active":"2024-02-21T23:10:08.233612+0000","last_peered":"2024-02-21T23:10:08.233612+0000","last_clean":"2024-02-21T23:10:08.233612+0000","last_became_active":"2024-02-21T23:10:05.374208+0000","last_became_peered":"2024-02-21T23:10:05.374208+0000","last_unstale":"2024-02-21T23:10:08.233612+0000","last_undegraded":"2024-02-21T23:10:08.233612+0000","last_fullsized":"2024-02-21T23:10:08.233612+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:30:08.752259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.14","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.235541+0000","last_change":"2024-02-21T23:09:54.289092+0000","last_active":"2024-02-21T23:10:08.235541+0000","last_peered":"2024-02-21T23:10:08.235541+0000","last_clean":"2024-02-21T23:10:08.235541+0000","last_became_active":"2024-02-21T23:09:54.288801+0000","last_became_peered":"2024-02-21T23:09:54.288801+0000","last_unstale":"2024-02-21T23:10:08.235541+0000","last_undegraded":"2024-02-21T23:10:08.235541+0000","last_fullsized":"2024-02-21T23:10:08.235541+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:05:09.189420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1b","version":"150'1","reported_seq":23,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236011+0000","last_change":"2024-02-21T23:09:53.283422+0000","last_active":"2024-02-21T23:10:08.236011+0000","last_peered":"2024-02-21T23:10:08.236011+0000","last_clean":"2024-02-21T23:10:08.236011+0000","last_became_active":"2024-02-21T23:09:53.283140+0000","last_became_peered":"2024-02-21T23:09:53.283140+0000","last_unstale":"2024-02-21T23:10:08.236011+0000","last_undegraded":"2024-02-21T23:10:08.236011+0000","last_fullsized":"2024-02-21T23:10:08.236011+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:46:34.419898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.237280+0000","last_change":"2024-02-21T23:10:06.403933+0000","last_active":"2024-02-21T23:10:08.237280+0000","last_peered":"2024-02-21T23:10:08.237280+0000","last_clean":"2024-02-21T23:10:08.237280+0000","last_became_active":"2024-02-21T23:10:06.403548+0000","last_became_peered":"2024-02-21T23:10:06.403548+0000","last_unstale":"2024-02-21T23:10:08.237280+0000","last_undegraded":"2024-02-21T23:10:08.237280+0000","last_fullsized":"2024-02-21T23:10:08.237280+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:11:20.176324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.14","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.394841+0000","last_change":"2024-02-21T23:10:05.388142+0000","last_active":"2024-02-21T23:10:05.394841+0000","last_peered":"2024-02-21T23:10:05.394841+0000","last_clean":"2024-02-21T23:10:05.394841+0000","last_became_active":"2024-02-21T23:10:05.387764+0000","last_became_peered":"2024-02-21T23:10:05.387764+0000","last_unstale":"2024-02-21T23:10:05.394841+0000","last_undegraded":"2024-02-21T23:10:05.394841+0000","last_fullsized":"2024-02-21T23:10:05.394841+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:58:38.451266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.15","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.375532+0000","last_change":"2024-02-21T23:09:54.273728+0000","last_active":"2024-02-21T23:10:07.375532+0000","last_peered":"2024-02-21T23:10:07.375532+0000","last_clean":"2024-02-21T23:10:07.375532+0000","last_became_active":"2024-02-21T23:09:54.273343+0000","last_became_peered":"2024-02-21T23:09:54.273343+0000","last_unstale":"2024-02-21T23:10:07.375532+0000","last_undegraded":"2024-02-21T23:10:07.375532+0000","last_fullsized":"2024-02-21T23:10:07.375532+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:49:15.552003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.1a","version":"150'1","reported_seq":23,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232153+0000","last_change":"2024-02-21T23:09:53.278176+0000","last_active":"2024-02-21T23:10:08.232153+0000","last_peered":"2024-02-21T23:10:08.232153+0000","last_clean":"2024-02-21T23:10:08.232153+0000","last_became_active":"2024-02-21T23:09:53.277880+0000","last_became_peered":"2024-02-21T23:09:53.277880+0000","last_unstale":"2024-02-21T23:10:08.232153+0000","last_undegraded":"2024-02-21T23:10:08.232153+0000","last_fullsized":"2024-02-21T23:10:08.232153+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:05:08.172503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.14","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.364119+0000","last_change":"2024-02-21T23:10:05.364119+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.364119+0000","last_undegraded":"2024-02-21T23:10:05.364119+0000","last_fullsized":"2024-02-21T23:10:05.364119+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236121+0000","last_change":"2024-02-21T23:10:05.375277+0000","last_active":"2024-02-21T23:10:08.236121+0000","last_peered":"2024-02-21T23:10:08.236121+0000","last_clean":"2024-02-21T23:10:08.236121+0000","last_became_active":"2024-02-21T23:10:05.374983+0000","last_became_peered":"2024-02-21T23:10:05.374983+0000","last_unstale":"2024-02-21T23:10:08.236121+0000","last_undegraded":"2024-02-21T23:10:08.236121+0000","last_fullsized":"2024-02-21T23:10:08.236121+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:11:19.160634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.376102+0000","last_change":"2024-02-21T23:09:54.278094+0000","last_active":"2024-02-21T23:10:07.376102+0000","last_peered":"2024-02-21T23:10:07.376102+0000","last_clean":"2024-02-21T23:10:07.376102+0000","last_became_active":"2024-02-21T23:09:54.276160+0000","last_became_peered":"2024-02-21T23:09:54.276160+0000","last_unstale":"2024-02-21T23:10:07.376102+0000","last_undegraded":"2024-02-21T23:10:07.376102+0000","last_fullsized":"2024-02-21T23:10:07.376102+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:15:07.339478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.375244+0000","last_change":"2024-02-21T23:09:53.280005+0000","last_active":"2024-02-21T23:10:07.375244+0000","last_peered":"2024-02-21T23:10:07.375244+0000","last_clean":"2024-02-21T23:10:07.375244+0000","last_became_active":"2024-02-21T23:09:53.279689+0000","last_became_peered":"2024-02-21T23:09:53.279689+0000","last_unstale":"2024-02-21T23:10:07.375244+0000","last_undegraded":"2024-02-21T23:10:07.375244+0000","last_fullsized":"2024-02-21T23:10:07.375244+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:40:05.561558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.15","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234855+0000","last_change":"2024-02-21T23:10:06.384066+0000","last_active":"2024-02-21T23:10:08.234855+0000","last_peered":"2024-02-21T23:10:08.234855+0000","last_clean":"2024-02-21T23:10:08.234855+0000","last_became_active":"2024-02-21T23:10:06.383780+0000","last_became_peered":"2024-02-21T23:10:06.383780+0000","last_unstale":"2024-02-21T23:10:08.234855+0000","last_undegraded":"2024-02-21T23:10:08.234855+0000","last_fullsized":"2024-02-21T23:10:08.234855+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:24:37.460238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236590+0000","last_change":"2024-02-21T23:10:05.393045+0000","last_active":"2024-02-21T23:10:08.236590+0000","last_peered":"2024-02-21T23:10:08.236590+0000","last_clean":"2024-02-21T23:10:08.236590+0000","last_became_active":"2024-02-21T23:10:05.392683+0000","last_became_peered":"2024-02-21T23:10:05.392683+0000","last_unstale":"2024-02-21T23:10:08.236590+0000","last_undegraded":"2024-02-21T23:10:08.236590+0000","last_fullsized":"2024-02-21T23:10:08.236590+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:45:39.918602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.394254+0000","last_change":"2024-02-21T23:09:54.262565+0000","last_active":"2024-02-21T23:10:05.394254+0000","last_peered":"2024-02-21T23:10:05.394254+0000","last_clean":"2024-02-21T23:10:05.394254+0000","last_became_active":"2024-02-21T23:09:54.261897+0000","last_became_peered":"2024-02-21T23:09:54.261897+0000","last_unstale":"2024-02-21T23:10:05.394254+0000","last_undegraded":"2024-02-21T23:10:05.394254+0000","last_fullsized":"2024-02-21T23:10:05.394254+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:08:50.519955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234939+0000","last_change":"2024-02-21T23:09:53.275695+0000","last_active":"2024-02-21T23:10:08.234939+0000","last_peered":"2024-02-21T23:10:08.234939+0000","last_clean":"2024-02-21T23:10:08.234939+0000","last_became_active":"2024-02-21T23:09:53.275343+0000","last_became_peered":"2024-02-21T23:09:53.275343+0000","last_unstale":"2024-02-21T23:10:08.234939+0000","last_undegraded":"2024-02-21T23:10:08.234939+0000","last_fullsized":"2024-02-21T23:10:08.234939+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:28:53.062636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1a","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.235787+0000","last_change":"2024-02-21T23:10:06.403716+0000","last_active":"2024-02-21T23:10:08.235787+0000","last_peered":"2024-02-21T23:10:08.235787+0000","last_clean":"2024-02-21T23:10:08.235787+0000","last_became_active":"2024-02-21T23:10:06.403226+0000","last_became_peered":"2024-02-21T23:10:06.403226+0000","last_unstale":"2024-02-21T23:10:08.235787+0000","last_undegraded":"2024-02-21T23:10:08.235787+0000","last_fullsized":"2024-02-21T23:10:08.235787+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:28:15.518153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.394797+0000","last_change":"2024-02-21T23:10:05.390846+0000","last_active":"2024-02-21T23:10:05.394797+0000","last_peered":"2024-02-21T23:10:05.394797+0000","last_clean":"2024-02-21T23:10:05.394797+0000","last_became_active":"2024-02-21T23:10:05.388088+0000","last_became_peered":"2024-02-21T23:10:05.388088+0000","last_unstale":"2024-02-21T23:10:05.394797+0000","last_undegraded":"2024-02-21T23:10:05.394797+0000","last_fullsized":"2024-02-21T23:10:05.394797+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:31:39.483569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.368060+0000","last_change":"2024-02-21T23:09:54.279220+0000","last_active":"2024-02-21T23:10:05.368060+0000","last_peered":"2024-02-21T23:10:05.368060+0000","last_clean":"2024-02-21T23:10:05.368060+0000","last_became_active":"2024-02-21T23:09:54.278653+0000","last_became_peered":"2024-02-21T23:09:54.278653+0000","last_unstale":"2024-02-21T23:10:05.368060+0000","last_undegraded":"2024-02-21T23:10:05.368060+0000","last_fullsized":"2024-02-21T23:10:05.368060+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:13:27.311923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236554+0000","last_change":"2024-02-21T23:09:53.288439+0000","last_active":"2024-02-21T23:10:08.236554+0000","last_peered":"2024-02-21T23:10:08.236554+0000","last_clean":"2024-02-21T23:10:08.236554+0000","last_became_active":"2024-02-21T23:09:53.287601+0000","last_became_peered":"2024-02-21T23:09:53.287601+0000","last_unstale":"2024-02-21T23:10:08.236554+0000","last_undegraded":"2024-02-21T23:10:08.236554+0000","last_fullsized":"2024-02-21T23:10:08.236554+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:32:05.045159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.376263+0000","last_change":"2024-02-21T23:10:05.376263+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.376263+0000","last_undegraded":"2024-02-21T23:10:05.376263+0000","last_fullsized":"2024-02-21T23:10:05.376263+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.1b","version":"154'1","reported_seq":15,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.233920+0000","last_change":"2024-02-21T23:10:05.387544+0000","last_active":"2024-02-21T23:10:08.233920+0000","last_peered":"2024-02-21T23:10:08.233920+0000","last_clean":"2024-02-21T23:10:08.233920+0000","last_became_active":"2024-02-21T23:10:05.385992+0000","last_became_peered":"2024-02-21T23:10:05.385992+0000","last_unstale":"2024-02-21T23:10:08.233920+0000","last_undegraded":"2024-02-21T23:10:08.233920+0000","last_fullsized":"2024-02-21T23:10:08.233920+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:28:14.502463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232148+0000","last_change":"2024-02-21T23:09:54.280274+0000","last_active":"2024-02-21T23:10:08.232148+0000","last_peered":"2024-02-21T23:10:08.232148+0000","last_clean":"2024-02-21T23:10:08.232148+0000","last_became_active":"2024-02-21T23:09:54.279951+0000","last_became_peered":"2024-02-21T23:09:54.279951+0000","last_unstale":"2024-02-21T23:10:08.232148+0000","last_undegraded":"2024-02-21T23:10:08.232148+0000","last_fullsized":"2024-02-21T23:10:08.232148+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:15:08.084683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.15","version":"150'3","reported_seq":21,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.393092+0000","last_change":"2024-02-21T23:09:53.279389+0000","last_active":"2024-02-21T23:10:05.393092+0000","last_peered":"2024-02-21T23:10:05.393092+0000","last_clean":"2024-02-21T23:10:05.393092+0000","last_became_active":"2024-02-21T23:09:53.279042+0000","last_became_peered":"2024-02-21T23:09:53.279042+0000","last_unstale":"2024-02-21T23:10:05.393092+0000","last_undegraded":"2024-02-21T23:10:05.393092+0000","last_fullsized":"2024-02-21T23:10:05.393092+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:44:44.898339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.233266+0000","last_change":"2024-02-21T23:10:06.379758+0000","last_active":"2024-02-21T23:10:08.233266+0000","last_peered":"2024-02-21T23:10:08.233266+0000","last_clean":"2024-02-21T23:10:08.233266+0000","last_became_active":"2024-02-21T23:10:06.379416+0000","last_became_peered":"2024-02-21T23:10:06.379416+0000","last_unstale":"2024-02-21T23:10:08.233266+0000","last_undegraded":"2024-02-21T23:10:08.233266+0000","last_fullsized":"2024-02-21T23:10:08.233266+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:06:55.637480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1a","version":"154'1","reported_seq":15,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.231363+0000","last_change":"2024-02-21T23:10:05.386289+0000","last_active":"2024-02-21T23:10:08.231363+0000","last_peered":"2024-02-21T23:10:08.231363+0000","last_clean":"2024-02-21T23:10:08.231363+0000","last_became_active":"2024-02-21T23:10:05.384200+0000","last_became_peered":"2024-02-21T23:10:05.384200+0000","last_unstale":"2024-02-21T23:10:08.231363+0000","last_undegraded":"2024-02-21T23:10:08.231363+0000","last_fullsized":"2024-02-21T23:10:08.231363+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:24:31.837524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1b","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.375019+0000","last_change":"2024-02-21T23:09:54.285563+0000","last_active":"2024-02-21T23:10:07.375019+0000","last_peered":"2024-02-21T23:10:07.375019+0000","last_clean":"2024-02-21T23:10:07.375019+0000","last_became_active":"2024-02-21T23:09:54.285275+0000","last_became_peered":"2024-02-21T23:09:54.285275+0000","last_unstale":"2024-02-21T23:10:07.375019+0000","last_undegraded":"2024-02-21T23:10:07.375019+0000","last_fullsized":"2024-02-21T23:10:07.375019+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:21:42.014406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"150'6","reported_seq":28,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236789+0000","last_change":"2024-02-21T23:09:53.288423+0000","last_active":"2024-02-21T23:10:08.236789+0000","last_peered":"2024-02-21T23:10:08.236789+0000","last_clean":"2024-02-21T23:10:08.236789+0000","last_became_active":"2024-02-21T23:09:53.287473+0000","last_became_peered":"2024-02-21T23:09:53.287473+0000","last_unstale":"2024-02-21T23:10:08.236789+0000","last_undegraded":"2024-02-21T23:10:08.236789+0000","last_fullsized":"2024-02-21T23:10:08.236789+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:13:32.425310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1e","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.373032+0000","last_change":"2024-02-21T23:10:05.373032+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.373032+0000","last_undegraded":"2024-02-21T23:10:05.373032+0000","last_fullsized":"2024-02-21T23:10:05.373032+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1d","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.391672+0000","last_change":"2024-02-21T23:10:05.388219+0000","last_active":"2024-02-21T23:10:05.391672+0000","last_peered":"2024-02-21T23:10:05.391672+0000","last_clean":"2024-02-21T23:10:05.391672+0000","last_became_active":"2024-02-21T23:10:05.387863+0000","last_became_peered":"2024-02-21T23:10:05.387863+0000","last_unstale":"2024-02-21T23:10:05.391672+0000","last_undegraded":"2024-02-21T23:10:05.391672+0000","last_fullsized":"2024-02-21T23:10:05.391672+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:34:10.873891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1c","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.390781+0000","last_change":"2024-02-21T23:09:54.278618+0000","last_active":"2024-02-21T23:10:05.390781+0000","last_peered":"2024-02-21T23:10:05.390781+0000","last_clean":"2024-02-21T23:10:05.390781+0000","last_became_active":"2024-02-21T23:09:54.278343+0000","last_became_peered":"2024-02-21T23:09:54.278343+0000","last_unstale":"2024-02-21T23:10:05.390781+0000","last_undegraded":"2024-02-21T23:10:05.390781+0000","last_fullsized":"2024-02-21T23:10:05.390781+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:49:31.759374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.13","version":"150'2","reported_seq":24,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234083+0000","last_change":"2024-02-21T23:09:53.284172+0000","last_active":"2024-02-21T23:10:08.234083+0000","last_peered":"2024-02-21T23:10:08.234083+0000","last_clean":"2024-02-21T23:10:08.234083+0000","last_became_active":"2024-02-21T23:09:53.283507+0000","last_became_peered":"2024-02-21T23:09:53.283507+0000","last_unstale":"2024-02-21T23:10:08.234083+0000","last_undegraded":"2024-02-21T23:10:08.234083+0000","last_fullsized":"2024-02-21T23:10:08.234083+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:20:30.108767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1f","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.382382+0000","last_change":"2024-02-21T23:10:05.382382+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.382382+0000","last_undegraded":"2024-02-21T23:10:05.382382+0000","last_fullsized":"2024-02-21T23:10:05.382382+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1c","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.394701+0000","last_change":"2024-02-21T23:10:05.387736+0000","last_active":"2024-02-21T23:10:05.394701+0000","last_peered":"2024-02-21T23:10:05.394701+0000","last_clean":"2024-02-21T23:10:05.394701+0000","last_became_active":"2024-02-21T23:10:05.387416+0000","last_became_peered":"2024-02-21T23:10:05.387416+0000","last_unstale":"2024-02-21T23:10:05.394701+0000","last_undegraded":"2024-02-21T23:10:05.394701+0000","last_fullsized":"2024-02-21T23:10:05.394701+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:35:01.442407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1d","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236368+0000","last_change":"2024-02-21T23:09:54.291717+0000","last_active":"2024-02-21T23:10:08.236368+0000","last_peered":"2024-02-21T23:10:08.236368+0000","last_clean":"2024-02-21T23:10:08.236368+0000","last_became_active":"2024-02-21T23:09:54.290842+0000","last_became_peered":"2024-02-21T23:09:54.290842+0000","last_unstale":"2024-02-21T23:10:08.236368+0000","last_undegraded":"2024-02-21T23:10:08.236368+0000","last_fullsized":"2024-02-21T23:10:08.236368+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:42:16.988326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232712+0000","last_change":"2024-02-21T23:09:53.284733+0000","last_active":"2024-02-21T23:10:08.232712+0000","last_peered":"2024-02-21T23:10:08.232712+0000","last_clean":"2024-02-21T23:10:08.232712+0000","last_became_active":"2024-02-21T23:09:53.284399+0000","last_became_peered":"2024-02-21T23:09:53.284399+0000","last_unstale":"2024-02-21T23:10:08.232712+0000","last_undegraded":"2024-02-21T23:10:08.232712+0000","last_fullsized":"2024-02-21T23:10:08.232712+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:05:08.172503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1c","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.365861+0000","last_change":"2024-02-21T23:10:05.365861+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.365861+0000","last_undegraded":"2024-02-21T23:10:05.365861+0000","last_fullsized":"2024-02-21T23:10:05.365861+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.1f","version":"154'4","reported_seq":18,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232862+0000","last_change":"2024-02-21T23:10:05.381058+0000","last_active":"2024-02-21T23:10:08.232862+0000","last_peered":"2024-02-21T23:10:08.232862+0000","last_clean":"2024-02-21T23:10:08.232862+0000","last_became_active":"2024-02-21T23:10:05.380737+0000","last_became_peered":"2024-02-21T23:10:05.380737+0000","last_unstale":"2024-02-21T23:10:08.232862+0000","last_undegraded":"2024-02-21T23:10:08.232862+0000","last_fullsized":"2024-02-21T23:10:08.232862+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:09:16.338374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1e","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.381340+0000","last_change":"2024-02-21T23:09:54.277339+0000","last_active":"2024-02-21T23:10:05.381340+0000","last_peered":"2024-02-21T23:10:05.381340+0000","last_clean":"2024-02-21T23:10:05.381340+0000","last_became_active":"2024-02-21T23:09:54.277039+0000","last_became_peered":"2024-02-21T23:09:54.277039+0000","last_unstale":"2024-02-21T23:10:05.381340+0000","last_undegraded":"2024-02-21T23:10:05.381340+0000","last_fullsized":"2024-02-21T23:10:05.381340+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:42:16.988326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":18,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.378558+0000","last_change":"2024-02-21T23:09:53.283915+0000","last_active":"2024-02-21T23:10:05.378558+0000","last_peered":"2024-02-21T23:10:05.378558+0000","last_clean":"2024-02-21T23:10:05.378558+0000","last_became_active":"2024-02-21T23:09:53.283532+0000","last_became_peered":"2024-02-21T23:09:53.283532+0000","last_unstale":"2024-02-21T23:10:05.378558+0000","last_undegraded":"2024-02-21T23:10:05.378558+0000","last_fullsized":"2024-02-21T23:10:05.378558+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:55:17.574145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1d","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.375317+0000","last_change":"2024-02-21T23:10:06.373957+0000","last_active":"2024-02-21T23:10:07.375317+0000","last_peered":"2024-02-21T23:10:07.375317+0000","last_clean":"2024-02-21T23:10:07.375317+0000","last_became_active":"2024-02-21T23:10:06.373657+0000","last_became_peered":"2024-02-21T23:10:06.373657+0000","last_unstale":"2024-02-21T23:10:07.375317+0000","last_undegraded":"2024-02-21T23:10:07.375317+0000","last_fullsized":"2024-02-21T23:10:07.375317+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:37:55.080822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1e","version":"154'1","reported_seq":15,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.235331+0000","last_change":"2024-02-21T23:10:05.388527+0000","last_active":"2024-02-21T23:10:08.235331+0000","last_peered":"2024-02-21T23:10:08.235331+0000","last_clean":"2024-02-21T23:10:08.235331+0000","last_became_active":"2024-02-21T23:10:05.388214+0000","last_became_peered":"2024-02-21T23:10:05.388214+0000","last_unstale":"2024-02-21T23:10:08.235331+0000","last_undegraded":"2024-02-21T23:10:08.235331+0000","last_fullsized":"2024-02-21T23:10:08.235331+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:50:32.666622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1f","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.376181+0000","last_change":"2024-02-21T23:09:54.265384+0000","last_active":"2024-02-21T23:10:07.376181+0000","last_peered":"2024-02-21T23:10:07.376181+0000","last_clean":"2024-02-21T23:10:07.376181+0000","last_became_active":"2024-02-21T23:09:54.265073+0000","last_became_peered":"2024-02-21T23:09:54.265073+0000","last_unstale":"2024-02-21T23:10:07.376181+0000","last_undegraded":"2024-02-21T23:10:07.376181+0000","last_fullsized":"2024-02-21T23:10:07.376181+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:44:28.295055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.10","version":"150'8","reported_seq":30,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.235156+0000","last_change":"2024-02-21T23:09:53.285246+0000","last_active":"2024-02-21T23:10:08.235156+0000","last_peered":"2024-02-21T23:10:08.235156+0000","last_clean":"2024-02-21T23:10:08.235156+0000","last_became_active":"2024-02-21T23:09:53.282084+0000","last_became_peered":"2024-02-21T23:09:53.282084+0000","last_unstale":"2024-02-21T23:10:08.235156+0000","last_undegraded":"2024-02-21T23:10:08.235156+0000","last_fullsized":"2024-02-21T23:10:08.235156+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:23:26.494398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.235925+0000","last_change":"2024-02-21T23:10:06.390439+0000","last_active":"2024-02-21T23:10:08.235925+0000","last_peered":"2024-02-21T23:10:08.235925+0000","last_clean":"2024-02-21T23:10:08.235925+0000","last_became_active":"2024-02-21T23:10:06.390012+0000","last_became_peered":"2024-02-21T23:10:06.390012+0000","last_unstale":"2024-02-21T23:10:08.235925+0000","last_undegraded":"2024-02-21T23:10:08.235925+0000","last_fullsized":"2024-02-21T23:10:08.235925+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:17:37.624314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.374798+0000","last_change":"2024-02-21T23:10:05.393586+0000","last_active":"2024-02-21T23:10:07.374798+0000","last_peered":"2024-02-21T23:10:07.374798+0000","last_clean":"2024-02-21T23:10:07.374798+0000","last_became_active":"2024-02-21T23:10:05.392606+0000","last_became_peered":"2024-02-21T23:10:05.392606+0000","last_unstale":"2024-02-21T23:10:07.374798+0000","last_undegraded":"2024-02-21T23:10:07.374798+0000","last_fullsized":"2024-02-21T23:10:07.374798+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:12:07.072214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234736+0000","last_change":"2024-02-21T23:09:54.289615+0000","last_active":"2024-02-21T23:10:08.234736+0000","last_peered":"2024-02-21T23:10:08.234736+0000","last_clean":"2024-02-21T23:10:08.234736+0000","last_became_active":"2024-02-21T23:09:54.289322+0000","last_became_peered":"2024-02-21T23:09:54.289322+0000","last_unstale":"2024-02-21T23:10:08.234736+0000","last_undegraded":"2024-02-21T23:10:08.234736+0000","last_fullsized":"2024-02-21T23:10:08.234736+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:34:28.642087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.f","version":"150'1","reported_seq":23,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236888+0000","last_change":"2024-02-21T23:09:53.287210+0000","last_active":"2024-02-21T23:10:08.236888+0000","last_peered":"2024-02-21T23:10:08.236888+0000","last_clean":"2024-02-21T23:10:08.236888+0000","last_became_active":"2024-02-21T23:09:53.285372+0000","last_became_peered":"2024-02-21T23:09:53.285372+0000","last_unstale":"2024-02-21T23:10:08.236888+0000","last_undegraded":"2024-02-21T23:10:08.236888+0000","last_fullsized":"2024-02-21T23:10:08.236888+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:20:30.108767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.363690+0000","last_change":"2024-02-21T23:10:05.363690+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.363690+0000","last_undegraded":"2024-02-21T23:10:05.363690+0000","last_fullsized":"2024-02-21T23:10:05.363690+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.395633+0000","last_change":"2024-02-21T23:10:05.395057+0000","last_active":"2024-02-21T23:10:05.395633+0000","last_peered":"2024-02-21T23:10:05.395633+0000","last_clean":"2024-02-21T23:10:05.395633+0000","last_became_active":"2024-02-21T23:10:05.394783+0000","last_became_peered":"2024-02-21T23:10:05.394783+0000","last_unstale":"2024-02-21T23:10:05.395633+0000","last_undegraded":"2024-02-21T23:10:05.395633+0000","last_fullsized":"2024-02-21T23:10:05.395633+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:55:29.679403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.390638+0000","last_change":"2024-02-21T23:09:54.274363+0000","last_active":"2024-02-21T23:10:05.390638+0000","last_peered":"2024-02-21T23:10:05.390638+0000","last_clean":"2024-02-21T23:10:05.390638+0000","last_became_active":"2024-02-21T23:09:54.274084+0000","last_became_peered":"2024-02-21T23:09:54.274084+0000","last_unstale":"2024-02-21T23:10:05.390638+0000","last_undegraded":"2024-02-21T23:10:05.390638+0000","last_fullsized":"2024-02-21T23:10:05.390638+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:42:19.996155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.e","version":"150'2","reported_seq":24,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.237049+0000","last_change":"2024-02-21T23:09:53.275941+0000","last_active":"2024-02-21T23:10:08.237049+0000","last_peered":"2024-02-21T23:10:08.237049+0000","last_clean":"2024-02-21T23:10:08.237049+0000","last_became_active":"2024-02-21T23:09:53.275555+0000","last_became_peered":"2024-02-21T23:09:53.275555+0000","last_unstale":"2024-02-21T23:10:08.237049+0000","last_undegraded":"2024-02-21T23:10:08.237049+0000","last_fullsized":"2024-02-21T23:10:08.237049+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:02:10.230470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.386121+0000","last_change":"2024-02-21T23:10:05.386121+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.386121+0000","last_undegraded":"2024-02-21T23:10:05.386121+0000","last_fullsized":"2024-02-21T23:10:05.386121+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.396519+0000","last_change":"2024-02-21T23:10:05.395810+0000","last_active":"2024-02-21T23:10:05.396519+0000","last_peered":"2024-02-21T23:10:05.396519+0000","last_clean":"2024-02-21T23:10:05.396519+0000","last_became_active":"2024-02-21T23:10:05.395273+0000","last_became_peered":"2024-02-21T23:10:05.395273+0000","last_unstale":"2024-02-21T23:10:05.396519+0000","last_undegraded":"2024-02-21T23:10:05.396519+0000","last_fullsized":"2024-02-21T23:10:05.396519+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:34:10.873891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.378450+0000","last_change":"2024-02-21T23:09:54.259630+0000","last_active":"2024-02-21T23:10:05.378450+0000","last_peered":"2024-02-21T23:10:05.378450+0000","last_clean":"2024-02-21T23:10:05.378450+0000","last_became_active":"2024-02-21T23:09:54.258924+0000","last_became_peered":"2024-02-21T23:09:54.258924+0000","last_unstale":"2024-02-21T23:10:05.378450+0000","last_undegraded":"2024-02-21T23:10:05.378450+0000","last_fullsized":"2024-02-21T23:10:05.378450+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:34:28.642087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.d","version":"150'1","reported_seq":23,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234011+0000","last_change":"2024-02-21T23:09:53.279385+0000","last_active":"2024-02-21T23:10:08.234011+0000","last_peered":"2024-02-21T23:10:08.234011+0000","last_clean":"2024-02-21T23:10:08.234011+0000","last_became_active":"2024-02-21T23:09:53.278074+0000","last_became_peered":"2024-02-21T23:09:53.278074+0000","last_unstale":"2024-02-21T23:10:08.234011+0000","last_undegraded":"2024-02-21T23:10:08.234011+0000","last_fullsized":"2024-02-21T23:10:08.234011+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:42:15.971409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.380894+0000","last_change":"2024-02-21T23:10:05.380894+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.380894+0000","last_undegraded":"2024-02-21T23:10:05.380894+0000","last_fullsized":"2024-02-21T23:10:05.380894+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.387108+0000","last_change":"2024-02-21T23:10:05.385621+0000","last_active":"2024-02-21T23:10:05.387108+0000","last_peered":"2024-02-21T23:10:05.387108+0000","last_clean":"2024-02-21T23:10:05.387108+0000","last_became_active":"2024-02-21T23:10:05.384654+0000","last_became_peered":"2024-02-21T23:10:05.384654+0000","last_unstale":"2024-02-21T23:10:05.387108+0000","last_undegraded":"2024-02-21T23:10:05.387108+0000","last_fullsized":"2024-02-21T23:10:05.387108+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:11:55.677946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.380169+0000","last_change":"2024-02-21T23:09:54.277296+0000","last_active":"2024-02-21T23:10:05.380169+0000","last_peered":"2024-02-21T23:10:05.380169+0000","last_clean":"2024-02-21T23:10:05.380169+0000","last_became_active":"2024-02-21T23:09:54.276973+0000","last_became_peered":"2024-02-21T23:09:54.276973+0000","last_unstale":"2024-02-21T23:10:05.380169+0000","last_undegraded":"2024-02-21T23:10:05.380169+0000","last_fullsized":"2024-02-21T23:10:05.380169+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:07:24.304915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.235861+0000","last_change":"2024-02-21T23:09:53.281804+0000","last_active":"2024-02-21T23:10:08.235861+0000","last_peered":"2024-02-21T23:10:08.235861+0000","last_clean":"2024-02-21T23:10:08.235861+0000","last_became_active":"2024-02-21T23:09:53.278643+0000","last_became_peered":"2024-02-21T23:09:53.278643+0000","last_unstale":"2024-02-21T23:10:08.235861+0000","last_undegraded":"2024-02-21T23:10:08.235861+0000","last_fullsized":"2024-02-21T23:10:08.235861+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:55:17.574145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.6","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.363355+0000","last_change":"2024-02-21T23:10:05.363355+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.363355+0000","last_undegraded":"2024-02-21T23:10:05.363355+0000","last_fullsized":"2024-02-21T23:10:05.363355+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234955+0000","last_change":"2024-02-21T23:10:05.389031+0000","last_active":"2024-02-21T23:10:08.234955+0000","last_peered":"2024-02-21T23:10:08.234955+0000","last_clean":"2024-02-21T23:10:08.234955+0000","last_became_active":"2024-02-21T23:10:05.388789+0000","last_became_peered":"2024-02-21T23:10:05.388789+0000","last_unstale":"2024-02-21T23:10:08.234955+0000","last_undegraded":"2024-02-21T23:10:08.234955+0000","last_fullsized":"2024-02-21T23:10:08.234955+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:46:44.667756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236537+0000","last_change":"2024-02-21T23:09:54.285141+0000","last_active":"2024-02-21T23:10:08.236537+0000","last_peered":"2024-02-21T23:10:08.236537+0000","last_clean":"2024-02-21T23:10:08.236537+0000","last_became_active":"2024-02-21T23:09:54.284498+0000","last_became_peered":"2024-02-21T23:09:54.284498+0000","last_unstale":"2024-02-21T23:10:08.236537+0000","last_undegraded":"2024-02-21T23:10:08.236537+0000","last_fullsized":"2024-02-21T23:10:08.236537+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:26:36.405020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.b","version":"150'2","reported_seq":24,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234201+0000","last_change":"2024-02-21T23:09:53.284284+0000","last_active":"2024-02-21T23:10:08.234201+0000","last_peered":"2024-02-21T23:10:08.234201+0000","last_clean":"2024-02-21T23:10:08.234201+0000","last_became_active":"2024-02-21T23:09:53.283043+0000","last_became_peered":"2024-02-21T23:09:53.283043+0000","last_unstale":"2024-02-21T23:10:08.234201+0000","last_undegraded":"2024-02-21T23:10:08.234201+0000","last_fullsized":"2024-02-21T23:10:08.234201+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:32:05.045159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.7","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.376118+0000","last_change":"2024-02-21T23:10:06.392968+0000","last_active":"2024-02-21T23:10:07.376118+0000","last_peered":"2024-02-21T23:10:07.376118+0000","last_clean":"2024-02-21T23:10:07.376118+0000","last_became_active":"2024-02-21T23:10:06.390031+0000","last_became_peered":"2024-02-21T23:10:06.390031+0000","last_unstale":"2024-02-21T23:10:07.376118+0000","last_undegraded":"2024-02-21T23:10:07.376118+0000","last_fullsized":"2024-02-21T23:10:07.376118+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:17:07.163396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236747+0000","last_change":"2024-02-21T23:10:05.392092+0000","last_active":"2024-02-21T23:10:08.236747+0000","last_peered":"2024-02-21T23:10:08.236747+0000","last_clean":"2024-02-21T23:10:08.236747+0000","last_became_active":"2024-02-21T23:10:05.391793+0000","last_became_peered":"2024-02-21T23:10:05.391793+0000","last_unstale":"2024-02-21T23:10:08.236747+0000","last_undegraded":"2024-02-21T23:10:08.236747+0000","last_fullsized":"2024-02-21T23:10:08.236747+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:09:16.338374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.233964+0000","last_change":"2024-02-21T23:09:54.275082+0000","last_active":"2024-02-21T23:10:08.233964+0000","last_peered":"2024-02-21T23:10:08.233964+0000","last_clean":"2024-02-21T23:10:08.233964+0000","last_became_active":"2024-02-21T23:09:54.274707+0000","last_became_peered":"2024-02-21T23:09:54.274707+0000","last_unstale":"2024-02-21T23:10:08.233964+0000","last_undegraded":"2024-02-21T23:10:08.233964+0000","last_fullsized":"2024-02-21T23:10:08.233964+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:39:52.080584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":18,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.378606+0000","last_change":"2024-02-21T23:09:53.277418+0000","last_active":"2024-02-21T23:10:05.378606+0000","last_peered":"2024-02-21T23:10:05.378606+0000","last_clean":"2024-02-21T23:10:05.378606+0000","last_became_active":"2024-02-21T23:09:53.277111+0000","last_became_peered":"2024-02-21T23:09:53.277111+0000","last_unstale":"2024-02-21T23:10:05.378606+0000","last_undegraded":"2024-02-21T23:10:05.378606+0000","last_fullsized":"2024-02-21T23:10:05.378606+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:23:26.494398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.375599+0000","last_change":"2024-02-21T23:10:06.391104+0000","last_active":"2024-02-21T23:10:07.375599+0000","last_peered":"2024-02-21T23:10:07.375599+0000","last_clean":"2024-02-21T23:10:07.375599+0000","last_became_active":"2024-02-21T23:10:06.390816+0000","last_became_peered":"2024-02-21T23:10:06.390816+0000","last_unstale":"2024-02-21T23:10:07.375599+0000","last_undegraded":"2024-02-21T23:10:07.375599+0000","last_fullsized":"2024-02-21T23:10:07.375599+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:49:43.863404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.7","version":"154'2","reported_seq":16,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234306+0000","last_change":"2024-02-21T23:10:05.378389+0000","last_active":"2024-02-21T23:10:08.234306+0000","last_peered":"2024-02-21T23:10:08.234306+0000","last_clean":"2024-02-21T23:10:08.234306+0000","last_became_active":"2024-02-21T23:10:05.378083+0000","last_became_peered":"2024-02-21T23:10:05.378083+0000","last_unstale":"2024-02-21T23:10:08.234306+0000","last_undegraded":"2024-02-21T23:10:08.234306+0000","last_fullsized":"2024-02-21T23:10:08.234306+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:28:14.502463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232579+0000","last_change":"2024-02-21T23:09:54.280400+0000","last_active":"2024-02-21T23:10:08.232579+0000","last_peered":"2024-02-21T23:10:08.232579+0000","last_clean":"2024-02-21T23:10:08.232579+0000","last_became_active":"2024-02-21T23:09:54.280102+0000","last_became_peered":"2024-02-21T23:09:54.280102+0000","last_unstale":"2024-02-21T23:10:08.232579+0000","last_undegraded":"2024-02-21T23:10:08.232579+0000","last_fullsized":"2024-02-21T23:10:08.232579+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:45:28.830261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.375349+0000","last_change":"2024-02-21T23:09:53.282939+0000","last_active":"2024-02-21T23:10:07.375349+0000","last_peered":"2024-02-21T23:10:07.375349+0000","last_clean":"2024-02-21T23:10:07.375349+0000","last_became_active":"2024-02-21T23:09:53.282612+0000","last_became_peered":"2024-02-21T23:09:53.282612+0000","last_unstale":"2024-02-21T23:10:07.375349+0000","last_undegraded":"2024-02-21T23:10:07.375349+0000","last_fullsized":"2024-02-21T23:10:07.375349+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:30:40.609835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236333+0000","last_change":"2024-02-21T23:10:06.396738+0000","last_active":"2024-02-21T23:10:08.236333+0000","last_peered":"2024-02-21T23:10:08.236333+0000","last_clean":"2024-02-21T23:10:08.236333+0000","last_became_active":"2024-02-21T23:10:06.396390+0000","last_became_peered":"2024-02-21T23:10:06.396390+0000","last_unstale":"2024-02-21T23:10:08.236333+0000","last_undegraded":"2024-02-21T23:10:08.236333+0000","last_fullsized":"2024-02-21T23:10:08.236333+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:24:32.853214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.6","version":"155'1","reported_seq":15,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.375969+0000","last_change":"2024-02-21T23:10:05.395962+0000","last_active":"2024-02-21T23:10:07.375969+0000","last_peered":"2024-02-21T23:10:07.375969+0000","last_clean":"2024-02-21T23:10:07.375969+0000","last_became_active":"2024-02-21T23:10:05.394863+0000","last_became_peered":"2024-02-21T23:10:05.394863+0000","last_unstale":"2024-02-21T23:10:07.375969+0000","last_undegraded":"2024-02-21T23:10:07.375969+0000","last_fullsized":"2024-02-21T23:10:07.375969+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:48:58.601649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.373315+0000","last_change":"2024-02-21T23:09:54.282302+0000","last_active":"2024-02-21T23:10:05.373315+0000","last_peered":"2024-02-21T23:10:05.373315+0000","last_clean":"2024-02-21T23:10:05.373315+0000","last_became_active":"2024-02-21T23:09:54.280468+0000","last_became_peered":"2024-02-21T23:09:54.280468+0000","last_unstale":"2024-02-21T23:10:05.373315+0000","last_undegraded":"2024-02-21T23:10:05.373315+0000","last_fullsized":"2024-02-21T23:10:05.373315+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:32:06.062076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":18,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.368703+0000","last_change":"2024-02-21T23:09:53.283837+0000","last_active":"2024-02-21T23:10:05.368703+0000","last_peered":"2024-02-21T23:10:05.368703+0000","last_clean":"2024-02-21T23:10:05.368703+0000","last_became_active":"2024-02-21T23:09:53.283541+0000","last_became_peered":"2024-02-21T23:09:53.283541+0000","last_unstale":"2024-02-21T23:10:05.368703+0000","last_undegraded":"2024-02-21T23:10:05.368703+0000","last_fullsized":"2024-02-21T23:10:05.368703+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:28:53.062636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.231400+0000","last_change":"2024-02-21T23:10:06.392422+0000","last_active":"2024-02-21T23:10:08.231400+0000","last_peered":"2024-02-21T23:10:08.231400+0000","last_clean":"2024-02-21T23:10:08.231400+0000","last_became_active":"2024-02-21T23:10:06.392121+0000","last_became_peered":"2024-02-21T23:10:06.392121+0000","last_unstale":"2024-02-21T23:10:08.231400+0000","last_undegraded":"2024-02-21T23:10:08.231400+0000","last_fullsized":"2024-02-21T23:10:08.231400+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:09:29.246717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.374605+0000","last_change":"2024-02-21T23:10:05.398114+0000","last_active":"2024-02-21T23:10:07.374605+0000","last_peered":"2024-02-21T23:10:07.374605+0000","last_clean":"2024-02-21T23:10:07.374605+0000","last_became_active":"2024-02-21T23:10:05.397811+0000","last_became_peered":"2024-02-21T23:10:05.397811+0000","last_unstale":"2024-02-21T23:10:07.374605+0000","last_undegraded":"2024-02-21T23:10:07.374605+0000","last_fullsized":"2024-02-21T23:10:07.374605+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:36:18.190950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.377721+0000","last_change":"2024-02-21T23:09:54.274229+0000","last_active":"2024-02-21T23:10:05.377721+0000","last_peered":"2024-02-21T23:10:05.377721+0000","last_clean":"2024-02-21T23:10:05.377721+0000","last_became_active":"2024-02-21T23:09:54.271365+0000","last_became_peered":"2024-02-21T23:09:54.271365+0000","last_unstale":"2024-02-21T23:10:05.377721+0000","last_undegraded":"2024-02-21T23:10:05.377721+0000","last_fullsized":"2024-02-21T23:10:05.377721+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:13:27.311923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.7","version":"150'2","reported_seq":24,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.233419+0000","last_change":"2024-02-21T23:09:53.283873+0000","last_active":"2024-02-21T23:10:08.233419+0000","last_peered":"2024-02-21T23:10:08.233419+0000","last_clean":"2024-02-21T23:10:08.233419+0000","last_became_active":"2024-02-21T23:09:53.282829+0000","last_became_peered":"2024-02-21T23:09:53.282829+0000","last_unstale":"2024-02-21T23:10:08.233419+0000","last_undegraded":"2024-02-21T23:10:08.233419+0000","last_fullsized":"2024-02-21T23:10:08.233419+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:13:32.425310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.b","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234395+0000","last_change":"2024-02-21T23:10:06.367171+0000","last_active":"2024-02-21T23:10:08.234395+0000","last_peered":"2024-02-21T23:10:08.234395+0000","last_clean":"2024-02-21T23:10:08.234395+0000","last_became_active":"2024-02-21T23:10:06.366866+0000","last_became_peered":"2024-02-21T23:10:06.366866+0000","last_unstale":"2024-02-21T23:10:08.234395+0000","last_undegraded":"2024-02-21T23:10:08.234395+0000","last_fullsized":"2024-02-21T23:10:08.234395+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:26:16.569464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.393848+0000","last_change":"2024-02-21T23:10:05.389876+0000","last_active":"2024-02-21T23:10:05.393848+0000","last_peered":"2024-02-21T23:10:05.393848+0000","last_clean":"2024-02-21T23:10:05.393848+0000","last_became_active":"2024-02-21T23:10:05.389568+0000","last_became_peered":"2024-02-21T23:10:05.389568+0000","last_unstale":"2024-02-21T23:10:05.393848+0000","last_undegraded":"2024-02-21T23:10:05.393848+0000","last_fullsized":"2024-02-21T23:10:05.393848+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:23:38.599655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234481+0000","last_change":"2024-02-21T23:09:54.277479+0000","last_active":"2024-02-21T23:10:08.234481+0000","last_peered":"2024-02-21T23:10:08.234481+0000","last_clean":"2024-02-21T23:10:08.234481+0000","last_became_active":"2024-02-21T23:09:54.277065+0000","last_became_peered":"2024-02-21T23:09:54.277065+0000","last_unstale":"2024-02-21T23:10:08.234481+0000","last_undegraded":"2024-02-21T23:10:08.234481+0000","last_fullsized":"2024-02-21T23:10:08.234481+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:07:24.304915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.6","version":"151'1","reported_seq":23,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.235714+0000","last_change":"2024-02-21T23:09:53.285874+0000","last_active":"2024-02-21T23:10:08.235714+0000","last_peered":"2024-02-21T23:10:08.235714+0000","last_clean":"2024-02-21T23:10:08.235714+0000","last_became_active":"2024-02-21T23:09:53.285569+0000","last_became_peered":"2024-02-21T23:09:53.285569+0000","last_unstale":"2024-02-21T23:10:08.235714+0000","last_undegraded":"2024-02-21T23:10:08.235714+0000","last_fullsized":"2024-02-21T23:10:08.235714+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:52:56.873078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.233184+0000","last_change":"2024-02-21T23:10:06.385851+0000","last_active":"2024-02-21T23:10:08.233184+0000","last_peered":"2024-02-21T23:10:08.233184+0000","last_clean":"2024-02-21T23:10:08.233184+0000","last_became_active":"2024-02-21T23:10:06.385482+0000","last_became_peered":"2024-02-21T23:10:06.385482+0000","last_unstale":"2024-02-21T23:10:08.233184+0000","last_undegraded":"2024-02-21T23:10:08.233184+0000","last_fullsized":"2024-02-21T23:10:08.233184+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:47:42.794742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.b","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.395230+0000","last_change":"2024-02-21T23:10:05.394114+0000","last_active":"2024-02-21T23:10:05.395230+0000","last_peered":"2024-02-21T23:10:05.395230+0000","last_clean":"2024-02-21T23:10:05.395230+0000","last_became_active":"2024-02-21T23:10:05.393152+0000","last_became_peered":"2024-02-21T23:10:05.393152+0000","last_unstale":"2024-02-21T23:10:05.395230+0000","last_undegraded":"2024-02-21T23:10:05.395230+0000","last_fullsized":"2024-02-21T23:10:05.395230+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:01:25.971350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.374932+0000","last_change":"2024-02-21T23:09:54.286102+0000","last_active":"2024-02-21T23:10:07.374932+0000","last_peered":"2024-02-21T23:10:07.374932+0000","last_clean":"2024-02-21T23:10:07.374932+0000","last_became_active":"2024-02-21T23:09:54.283073+0000","last_became_peered":"2024-02-21T23:09:54.283073+0000","last_unstale":"2024-02-21T23:10:07.374932+0000","last_undegraded":"2024-02-21T23:10:07.374932+0000","last_fullsized":"2024-02-21T23:10:07.374932+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:39:05.096862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.235327+0000","last_change":"2024-02-21T23:09:53.285730+0000","last_active":"2024-02-21T23:10:08.235327+0000","last_peered":"2024-02-21T23:10:08.235327+0000","last_clean":"2024-02-21T23:10:08.235327+0000","last_became_active":"2024-02-21T23:09:53.285373+0000","last_became_peered":"2024-02-21T23:09:53.285373+0000","last_unstale":"2024-02-21T23:10:08.235327+0000","last_undegraded":"2024-02-21T23:10:08.235327+0000","last_fullsized":"2024-02-21T23:10:08.235327+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:04:15.300172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.230302+0000","last_change":"2024-02-21T23:10:06.380586+0000","last_active":"2024-02-21T23:10:08.230302+0000","last_peered":"2024-02-21T23:10:08.230302+0000","last_clean":"2024-02-21T23:10:08.230302+0000","last_became_active":"2024-02-21T23:10:06.380293+0000","last_became_peered":"2024-02-21T23:10:06.380293+0000","last_unstale":"2024-02-21T23:10:08.230302+0000","last_undegraded":"2024-02-21T23:10:08.230302+0000","last_fullsized":"2024-02-21T23:10:08.230302+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:01:26.662036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.231095+0000","last_change":"2024-02-21T23:10:05.380512+0000","last_active":"2024-02-21T23:10:08.231095+0000","last_peered":"2024-02-21T23:10:08.231095+0000","last_clean":"2024-02-21T23:10:08.231095+0000","last_became_active":"2024-02-21T23:10:05.380207+0000","last_became_peered":"2024-02-21T23:10:05.380207+0000","last_unstale":"2024-02-21T23:10:08.231095+0000","last_undegraded":"2024-02-21T23:10:08.231095+0000","last_fullsized":"2024-02-21T23:10:08.231095+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:17:36.608624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.b","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.235603+0000","last_change":"2024-02-21T23:09:54.277483+0000","last_active":"2024-02-21T23:10:08.235603+0000","last_peered":"2024-02-21T23:10:08.235603+0000","last_clean":"2024-02-21T23:10:08.235603+0000","last_became_active":"2024-02-21T23:09:54.277170+0000","last_became_peered":"2024-02-21T23:09:54.277170+0000","last_unstale":"2024-02-21T23:10:08.235603+0000","last_undegraded":"2024-02-21T23:10:08.235603+0000","last_fullsized":"2024-02-21T23:10:08.235603+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:33:07.362294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.375793+0000","last_change":"2024-02-21T23:09:53.267455+0000","last_active":"2024-02-21T23:10:07.375793+0000","last_peered":"2024-02-21T23:10:07.375793+0000","last_clean":"2024-02-21T23:10:07.375793+0000","last_became_active":"2024-02-21T23:09:53.266974+0000","last_became_peered":"2024-02-21T23:09:53.266974+0000","last_unstale":"2024-02-21T23:10:07.375793+0000","last_undegraded":"2024-02-21T23:10:07.375793+0000","last_fullsized":"2024-02-21T23:10:07.375793+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:33:26.269092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.e","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.371905+0000","last_change":"2024-02-21T23:10:05.371905+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.371905+0000","last_undegraded":"2024-02-21T23:10:05.371905+0000","last_fullsized":"2024-02-21T23:10:05.371905+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.d","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.395287+0000","last_change":"2024-02-21T23:10:05.393632+0000","last_active":"2024-02-21T23:10:05.395287+0000","last_peered":"2024-02-21T23:10:05.395287+0000","last_clean":"2024-02-21T23:10:05.395287+0000","last_became_active":"2024-02-21T23:10:05.393070+0000","last_became_peered":"2024-02-21T23:10:05.393070+0000","last_unstale":"2024-02-21T23:10:05.395287+0000","last_undegraded":"2024-02-21T23:10:05.395287+0000","last_fullsized":"2024-02-21T23:10:05.395287+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:46:46.525155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.391035+0000","last_change":"2024-02-21T23:09:54.275273+0000","last_active":"2024-02-21T23:10:05.391035+0000","last_peered":"2024-02-21T23:10:05.391035+0000","last_clean":"2024-02-21T23:10:05.391035+0000","last_became_active":"2024-02-21T23:09:54.274287+0000","last_became_peered":"2024-02-21T23:09:54.274287+0000","last_unstale":"2024-02-21T23:10:05.391035+0000","last_undegraded":"2024-02-21T23:10:05.391035+0000","last_fullsized":"2024-02-21T23:10:05.391035+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:18:40.139538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":18,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.391026+0000","last_change":"2024-02-21T23:09:53.279782+0000","last_active":"2024-02-21T23:10:05.391026+0000","last_peered":"2024-02-21T23:10:05.391026+0000","last_clean":"2024-02-21T23:10:05.391026+0000","last_became_active":"2024-02-21T23:09:53.279477+0000","last_became_peered":"2024-02-21T23:09:53.279477+0000","last_unstale":"2024-02-21T23:10:05.391026+0000","last_undegraded":"2024-02-21T23:10:05.391026+0000","last_fullsized":"2024-02-21T23:10:05.391026+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:41:01.169723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.f","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.237523+0000","last_change":"2024-02-21T23:10:06.399721+0000","last_active":"2024-02-21T23:10:08.237523+0000","last_peered":"2024-02-21T23:10:08.237523+0000","last_clean":"2024-02-21T23:10:08.237523+0000","last_became_active":"2024-02-21T23:10:06.398869+0000","last_became_peered":"2024-02-21T23:10:06.398869+0000","last_unstale":"2024-02-21T23:10:08.237523+0000","last_undegraded":"2024-02-21T23:10:08.237523+0000","last_fullsized":"2024-02-21T23:10:08.237523+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:48:37.041227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.382546+0000","last_change":"2024-02-21T23:10:05.380230+0000","last_active":"2024-02-21T23:10:05.382546+0000","last_peered":"2024-02-21T23:10:05.382546+0000","last_clean":"2024-02-21T23:10:05.382546+0000","last_became_active":"2024-02-21T23:10:05.378612+0000","last_became_peered":"2024-02-21T23:10:05.378612+0000","last_unstale":"2024-02-21T23:10:05.382546+0000","last_undegraded":"2024-02-21T23:10:05.382546+0000","last_fullsized":"2024-02-21T23:10:05.382546+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:51:20.648392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.d","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.373520+0000","last_change":"2024-02-21T23:09:54.279409+0000","last_active":"2024-02-21T23:10:05.373520+0000","last_peered":"2024-02-21T23:10:05.373520+0000","last_clean":"2024-02-21T23:10:05.373520+0000","last_became_active":"2024-02-21T23:09:54.279107+0000","last_became_peered":"2024-02-21T23:09:54.279107+0000","last_unstale":"2024-02-21T23:10:05.373520+0000","last_undegraded":"2024-02-21T23:10:05.373520+0000","last_fullsized":"2024-02-21T23:10:05.373520+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:20:31.125684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232898+0000","last_change":"2024-02-21T23:09:53.278959+0000","last_active":"2024-02-21T23:10:08.232898+0000","last_peered":"2024-02-21T23:10:08.232898+0000","last_clean":"2024-02-21T23:10:08.232898+0000","last_became_active":"2024-02-21T23:09:53.278673+0000","last_became_peered":"2024-02-21T23:09:53.278673+0000","last_unstale":"2024-02-21T23:10:08.232898+0000","last_undegraded":"2024-02-21T23:10:08.232898+0000","last_fullsized":"2024-02-21T23:10:08.232898+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:42:15.971409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.375427+0000","last_change":"2024-02-21T23:10:06.387361+0000","last_active":"2024-02-21T23:10:07.375427+0000","last_peered":"2024-02-21T23:10:07.375427+0000","last_clean":"2024-02-21T23:10:07.375427+0000","last_became_active":"2024-02-21T23:10:06.387063+0000","last_became_peered":"2024-02-21T23:10:06.387063+0000","last_unstale":"2024-02-21T23:10:07.375427+0000","last_undegraded":"2024-02-21T23:10:07.375427+0000","last_fullsized":"2024-02-21T23:10:07.375427+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:47:41.453699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.385563+0000","last_change":"2024-02-21T23:10:05.381912+0000","last_active":"2024-02-21T23:10:05.385563+0000","last_peered":"2024-02-21T23:10:05.385563+0000","last_clean":"2024-02-21T23:10:05.385563+0000","last_became_active":"2024-02-21T23:10:05.381618+0000","last_became_peered":"2024-02-21T23:10:05.381618+0000","last_unstale":"2024-02-21T23:10:05.385563+0000","last_undegraded":"2024-02-21T23:10:05.385563+0000","last_fullsized":"2024-02-21T23:10:05.385563+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:53:08.978335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.e","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.235442+0000","last_change":"2024-02-21T23:09:54.282920+0000","last_active":"2024-02-21T23:10:08.235442+0000","last_peered":"2024-02-21T23:10:08.235442+0000","last_clean":"2024-02-21T23:10:08.235442+0000","last_became_active":"2024-02-21T23:09:54.282544+0000","last_became_peered":"2024-02-21T23:09:54.282544+0000","last_unstale":"2024-02-21T23:10:08.235442+0000","last_undegraded":"2024-02-21T23:10:08.235442+0000","last_fullsized":"2024-02-21T23:10:08.235442+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:40:05.206906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":18,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.378866+0000","last_change":"2024-02-21T23:09:53.275586+0000","last_active":"2024-02-21T23:10:05.378866+0000","last_peered":"2024-02-21T23:10:05.378866+0000","last_clean":"2024-02-21T23:10:05.378866+0000","last_became_active":"2024-02-21T23:09:53.274828+0000","last_became_peered":"2024-02-21T23:09:53.274828+0000","last_unstale":"2024-02-21T23:10:05.378866+0000","last_undegraded":"2024-02-21T23:10:05.378866+0000","last_fullsized":"2024-02-21T23:10:05.378866+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:02:10.230470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.d","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234506+0000","last_change":"2024-02-21T23:10:06.377243+0000","last_active":"2024-02-21T23:10:08.234506+0000","last_peered":"2024-02-21T23:10:08.234506+0000","last_clean":"2024-02-21T23:10:08.234506+0000","last_became_active":"2024-02-21T23:10:06.376521+0000","last_became_peered":"2024-02-21T23:10:06.376521+0000","last_unstale":"2024-02-21T23:10:08.234506+0000","last_undegraded":"2024-02-21T23:10:08.234506+0000","last_fullsized":"2024-02-21T23:10:08.234506+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:02:54.477082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.e","version":"154'2","reported_seq":16,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232515+0000","last_change":"2024-02-21T23:10:05.388215+0000","last_active":"2024-02-21T23:10:08.232515+0000","last_peered":"2024-02-21T23:10:08.232515+0000","last_clean":"2024-02-21T23:10:08.232515+0000","last_became_active":"2024-02-21T23:10:05.387907+0000","last_became_peered":"2024-02-21T23:10:05.387907+0000","last_unstale":"2024-02-21T23:10:08.232515+0000","last_undegraded":"2024-02-21T23:10:08.232515+0000","last_fullsized":"2024-02-21T23:10:08.232515+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:05:36.127991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.f","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.376215+0000","last_change":"2024-02-21T23:09:54.277320+0000","last_active":"2024-02-21T23:10:07.376215+0000","last_peered":"2024-02-21T23:10:07.376215+0000","last_clean":"2024-02-21T23:10:07.376215+0000","last_became_active":"2024-02-21T23:09:54.276368+0000","last_became_peered":"2024-02-21T23:09:54.276368+0000","last_unstale":"2024-02-21T23:10:07.376215+0000","last_undegraded":"2024-02-21T23:10:07.376215+0000","last_fullsized":"2024-02-21T23:10:07.376215+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:40:37.285836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.230247+0000","last_change":"2024-02-21T23:09:53.278027+0000","last_active":"2024-02-21T23:10:08.230247+0000","last_peered":"2024-02-21T23:10:08.230247+0000","last_clean":"2024-02-21T23:10:08.230247+0000","last_became_active":"2024-02-21T23:09:53.277660+0000","last_became_peered":"2024-02-21T23:09:53.277660+0000","last_unstale":"2024-02-21T23:10:08.230247+0000","last_undegraded":"2024-02-21T23:10:08.230247+0000","last_fullsized":"2024-02-21T23:10:08.230247+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:07:23.287998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1b","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.385924+0000","last_change":"2024-02-21T23:10:05.385924+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.385924+0000","last_undegraded":"2024-02-21T23:10:05.385924+0000","last_fullsized":"2024-02-21T23:10:05.385924+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.230256+0000","last_change":"2024-02-21T23:10:05.390395+0000","last_active":"2024-02-21T23:10:08.230256+0000","last_peered":"2024-02-21T23:10:08.230256+0000","last_clean":"2024-02-21T23:10:08.230256+0000","last_became_active":"2024-02-21T23:10:05.390115+0000","last_became_peered":"2024-02-21T23:10:05.390115+0000","last_unstale":"2024-02-21T23:10:08.230256+0000","last_undegraded":"2024-02-21T23:10:08.230256+0000","last_fullsized":"2024-02-21T23:10:08.230256+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:10:20.170395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.369300+0000","last_change":"2024-02-21T23:09:54.277356+0000","last_active":"2024-02-21T23:10:05.369300+0000","last_peered":"2024-02-21T23:10:05.369300+0000","last_clean":"2024-02-21T23:10:05.369300+0000","last_became_active":"2024-02-21T23:09:54.277087+0000","last_became_peered":"2024-02-21T23:09:54.277087+0000","last_unstale":"2024-02-21T23:10:05.369300+0000","last_undegraded":"2024-02-21T23:10:05.369300+0000","last_fullsized":"2024-02-21T23:10:05.369300+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:03:44.493909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.235691+0000","last_change":"2024-02-21T23:09:53.286181+0000","last_active":"2024-02-21T23:10:08.235691+0000","last_peered":"2024-02-21T23:10:08.235691+0000","last_clean":"2024-02-21T23:10:08.235691+0000","last_became_active":"2024-02-21T23:09:53.285849+0000","last_became_peered":"2024-02-21T23:09:53.285849+0000","last_unstale":"2024-02-21T23:10:08.235691+0000","last_undegraded":"2024-02-21T23:10:08.235691+0000","last_fullsized":"2024-02-21T23:10:08.235691+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:51:08.543135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":421,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.368921+0000","last_change":"2024-02-21T23:02:32.289335+0000","last_active":"2024-02-21T23:10:05.368921+0000","last_peered":"2024-02-21T23:10:05.368921+0000","last_clean":"2024-02-21T23:10:05.368921+0000","last_became_active":"2024-02-21T23:02:32.286913+0000","last_became_peered":"2024-02-21T23:02:32.286913+0000","last_unstale":"2024-02-21T23:10:05.368921+0000","last_undegraded":"2024-02-21T23:10:05.368921+0000","last_fullsized":"2024-02-21T23:10:05.368921+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_clean_scrub_stamp":"2024-02-21T23:02:31.172569+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:36:23.987374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":26,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":25,"num_pg":32,"stat_sum":{"num_bytes":672,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15,"ondisk_log_size":15,"up":64,"acting":64,"num_store_stats":8},{"poolid":24,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":23,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574941,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33124,"kb_used_data":1944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338716,"statfs":{"total":96636764160,"available":96602845184,"internally_reserved":0,"allocated":1990656,"data_stored":1530590,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574942,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33064,"kb_used_data":1880,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338776,"statfs":{"total":96636764160,"available":96602906624,"internally_reserved":0,"allocated":1925120,"data_stored":940222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574941,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33056,"kb_used_data":1876,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338784,"statfs":{"total":96636764160,"available":96602914816,"internally_reserved":0,"allocated":1921024,"data_stored":932824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574942,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33080,"kb_used_data":1896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338760,"statfs":{"total":96636764160,"available":96602890240,"internally_reserved":0,"allocated":1941504,"data_stored":940986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574943,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33120,"kb_used_data":1932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338720,"statfs":{"total":96636764160,"available":96602849280,"internally_reserved":0,"allocated":1978368,"data_stored":958172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574944,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33096,"kb_used_data":1908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338744,"statfs":{"total":96636764160,"available":96602873856,"internally_reserved":0,"allocated":1953792,"data_stored":956378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574942,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33104,"kb_used_data":1916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338736,"statfs":{"total":96636764160,"available":96602865664,"internally_reserved":0,"allocated":1961984,"data_stored":957408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574942,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33172,"kb_used_data":1992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338668,"statfs":{"total":96636764160,"available":96602796032,"internally_reserved":0,"allocated":2039808,"data_stored":1546996,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-21T23:10:11.549 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-21T23:10:11.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.786+0000 7f993cc00640 1 Processor -- start 2024-02-21T23:10:11.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.786+0000 7f993cc00640 1 -- start start 2024-02-21T23:10:11.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.787+0000 7f993cc00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f99380c8560 0x7f99380c7ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:11.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.788+0000 7f993cc00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99380c2c20 con 0x7f99380c64d0 2024-02-21T23:10:11.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.788+0000 7f993cc00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9938155100 con 0x7f99380c8940 2024-02-21T23:10:11.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.788+0000 7f993cc00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99380c8dd0 con 0x7f99380c8560 2024-02-21T23:10:11.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.788+0000 7f9936a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f99380c64d0 0x7f99380c4d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60882/0 (socket says 172.21.15.28:60882) 2024-02-21T23:10:11.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.788+0000 7f9936a00640 1 -- 172.21.15.28:0/1891347651 learned_addr learned my addr 172.21.15.28:0/1891347651 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:10:11.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.788+0000 7f9937400640 1 --2- 172.21.15.28:0/1891347651 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f99380c8560 0x7f99380c7ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:11.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.788+0000 7f9936000640 1 -- 172.21.15.28:0/1891347651 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1934111381 0 0) 0x7f99380c2c20 con 0x7f99380c64d0 2024-02-21T23:10:11.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.788+0000 7f9936000640 1 -- 172.21.15.28:0/1891347651 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f991c003520 con 0x7f99380c64d0 2024-02-21T23:10:11.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.789+0000 7f9936000640 1 -- 172.21.15.28:0/1891347651 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4136531386 0 0) 0x7f991c003520 con 0x7f99380c64d0 2024-02-21T23:10:11.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.789+0000 7f9936000640 1 -- 172.21.15.28:0/1891347651 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f99380c8560 msgr2=0x7f99380c7ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:11.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.789+0000 7f9936000640 1 --2- 172.21.15.28:0/1891347651 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f99380c8560 0x7f99380c7ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:11.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.789+0000 7f9936000640 1 -- 172.21.15.28:0/1891347651 >> v1:172.21.15.81:6789/0 conn(0x7f99380c8940 legacy=0x7f99380c54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:11.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.789+0000 7f9936000640 1 -- 172.21.15.28:0/1891347651 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99381553e0 con 0x7f99380c64d0 2024-02-21T23:10:11.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.789+0000 7f9936000640 1 -- 172.21.15.28:0/1891347651 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f9924001670 con 0x7f99380c64d0 2024-02-21T23:10:11.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.789+0000 7f9936000640 1 -- 172.21.15.28:0/1891347651 >> v1:172.21.15.28:6789/0 conn(0x7f99380c64d0 legacy=0x7f99380c4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:11.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.789+0000 7f9936000640 1 --2- 172.21.15.28:0/1891347651 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f991c0039e0 0x7f991c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:11.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.789+0000 7f9936000640 1 --2- 172.21.15.28:0/1891347651 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f99380c8560 0x7f991c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:11.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.789+0000 7f9936000640 1 --2- 172.21.15.28:0/1891347651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f991c004d40 0x7f991c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:11.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.789+0000 7f9936000640 1 -- 172.21.15.28:0/1891347651 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99380c8dd0 con 0x7f991c004d40 2024-02-21T23:10:11.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.789+0000 7f9936000640 1 -- 172.21.15.28:0/1891347651 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f991c00baf0 con 0x7f991c0039e0 2024-02-21T23:10:11.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.789+0000 7f9936000640 1 -- 172.21.15.28:0/1891347651 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f991c00bcf0 con 0x7f99380c8560 2024-02-21T23:10:11.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.790+0000 7f9937e00640 1 --2- 172.21.15.28:0/1891347651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f991c004d40 0x7f991c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:11.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.790+0000 7f9937400640 1 --2- 172.21.15.28:0/1891347651 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f991c0039e0 0x7f991c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:11.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.790+0000 7f9936a00640 1 --2- 172.21.15.28:0/1891347651 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f99380c8560 0x7f991c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:11.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.790+0000 7f9937e00640 1 -- 172.21.15.28:0/1891347651 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f99380c8560 msgr2=0x7f991c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:11.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.790+0000 7f9937e00640 1 --2- 172.21.15.28:0/1891347651 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f99380c8560 0x7f991c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:11.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.790+0000 7f9937e00640 1 -- 172.21.15.28:0/1891347651 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f991c0039e0 msgr2=0x7f991c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:11.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.790+0000 7f9937e00640 1 --2- 172.21.15.28:0/1891347651 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f991c0039e0 0x7f991c003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:11.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.790+0000 7f9937e00640 1 -- 172.21.15.28:0/1891347651 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f991c003520 con 0x7f991c004d40 2024-02-21T23:10:11.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.791+0000 7f9937e00640 1 --2- 172.21.15.28:0/1891347651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f991c004d40 0x7f991c0051c0 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7f9930001a30 tx=0x7f991c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15cc8952f7658301 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:11.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.791+0000 7f9936000640 1 -- 172.21.15.28:0/1891347651 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9930001a70 con 0x7f991c004d40 2024-02-21T23:10:11.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.791+0000 7f9936000640 1 -- 172.21.15.28:0/1891347651 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9930034e00 con 0x7f991c004d40 2024-02-21T23:10:11.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.791+0000 7f9936000640 1 -- 172.21.15.28:0/1891347651 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9930003bb0 con 0x7f991c004d40 2024-02-21T23:10:11.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.792+0000 7f993cc00640 1 -- 172.21.15.28:0/1891347651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f991c004d40 msgr2=0x7f991c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:11.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.792+0000 7f993cc00640 1 --2- 172.21.15.28:0/1891347651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f991c004d40 0x7f991c0051c0 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7f9930001a30 tx=0x7f991c005830 comp rx=0 tx=0).stop 2024-02-21T23:10:11.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.792+0000 7f9937e00640 1 -- 172.21.15.28:0/1891347651 reap_dead start 2024-02-21T23:10:11.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.792+0000 7f993cc00640 1 -- 172.21.15.28:0/1891347651 shutdown_connections 2024-02-21T23:10:11.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.792+0000 7f993cc00640 1 -- 172.21.15.28:0/1891347651 >> 172.21.15.28:0/1891347651 conn(0x7f993806ae70 msgr2=0x7f993806b280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:11.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.792+0000 7f993cc00640 1 -- 172.21.15.28:0/1891347651 shutdown_connections 2024-02-21T23:10:11.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.792+0000 7f993cc00640 1 -- 172.21.15.28:0/1891347651 wait complete. 2024-02-21T23:10:11.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.794+0000 7f993cc00640 1 Processor -- start 2024-02-21T23:10:11.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.794+0000 7f993cc00640 1 -- start start 2024-02-21T23:10:11.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.795+0000 7f993cc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99380a85e0 0x7f99380abfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:11.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.795+0000 7f993cc00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f99380aad40 0x7f99380ac520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:11.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.795+0000 7f993cc00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f99380a9950 0x7f99381e9990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:11.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.795+0000 7f993cc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99381559a0 con 0x7f99380a85e0 2024-02-21T23:10:11.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.795+0000 7f993cc00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99380c6910 con 0x7f99380aad40 2024-02-21T23:10:11.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.795+0000 7f993cc00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9938154db0 con 0x7f99380a9950 2024-02-21T23:10:11.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.795+0000 7f9937400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99380a85e0 0x7f99380abfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:11.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.795+0000 7f9937400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99380a85e0 0x7f99380abfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35592/0 (socket says 172.21.15.28:35592) 2024-02-21T23:10:11.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.795+0000 7f9937400640 1 -- 172.21.15.28:0/1466822234 learned_addr learned my addr 172.21.15.28:0/1466822234 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:11.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.795+0000 7f9936a00640 1 --2- 172.21.15.28:0/1466822234 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f99380aad40 0x7f99380ac520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:11.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.795+0000 7f9937400640 1 -- 172.21.15.28:0/1466822234 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f99380a9950 msgr2=0x7f99381e9990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:11.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.795+0000 7f9937400640 1 --2- 172.21.15.28:0/1466822234 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f99380a9950 0x7f99381e9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:11.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.795+0000 7f9937400640 1 -- 172.21.15.28:0/1466822234 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f99380aad40 msgr2=0x7f99380ac520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:11.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.795+0000 7f9937400640 1 --2- 172.21.15.28:0/1466822234 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f99380aad40 0x7f99380ac520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:11.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.795+0000 7f9937400640 1 -- 172.21.15.28:0/1466822234 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99380ad2a0 con 0x7f99380a85e0 2024-02-21T23:10:11.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.796+0000 7f9937400640 1 --2- 172.21.15.28:0/1466822234 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99380a85e0 0x7f99380abfe0 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7f99280115e0 tx=0x7f9928011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:11.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.796+0000 7f992f400640 1 -- 172.21.15.28:0/1466822234 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f992800f8f0 con 0x7f99380a85e0 2024-02-21T23:10:11.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.796+0000 7f992f400640 1 -- 172.21.15.28:0/1466822234 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f992800fa90 con 0x7f99380a85e0 2024-02-21T23:10:11.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.796+0000 7f993cc00640 1 -- 172.21.15.28:0/1466822234 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99380ad580 con 0x7f99380a85e0 2024-02-21T23:10:11.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.796+0000 7f992f400640 1 -- 172.21.15.28:0/1466822234 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f992800fe40 con 0x7f99380a85e0 2024-02-21T23:10:11.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.796+0000 7f993cc00640 1 -- 172.21.15.28:0/1466822234 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99380aca60 con 0x7f99380a85e0 2024-02-21T23:10:11.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.798+0000 7f993cc00640 1 -- 172.21.15.28:0/1466822234 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f993804f290 con 0x7f99380a85e0 2024-02-21T23:10:11.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.798+0000 7f992f400640 1 -- 172.21.15.28:0/1466822234 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9928015460 con 0x7f99380a85e0 2024-02-21T23:10:11.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.800+0000 7f992f400640 1 --2- 172.21.15.28:0/1466822234 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f991806b4d0 0x7f991806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:11.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.800+0000 7f992f400640 1 -- 172.21.15.28:0/1466822234 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f992808aff0 con 0x7f99380a85e0 2024-02-21T23:10:11.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.800+0000 7f9936a00640 1 --2- 172.21.15.28:0/1466822234 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f991806b4d0 0x7f991806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:11.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.801+0000 7f9936a00640 1 --2- 172.21.15.28:0/1466822234 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f991806b4d0 0x7f991806d990 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f9924001050 tx=0x7f9924001190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:11.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:11.804+0000 7f992f400640 1 -- 172.21.15.28:0/1466822234 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f9928019030 con 0x7f99380a85e0 2024-02-21T23:10:11.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.007+0000 7f993cc00640 1 -- 172.21.15.28:0/1466822234 --> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f99380c1f40 con 0x7f991806b4d0 2024-02-21T23:10:12.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.041+0000 7f992f400640 1 -- 172.21.15.28:0/1466822234 <== mgr.4101 v2:172.21.15.81:6832/1848538577 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+489418 (secure 0 0 0) 0x7f99380c1f40 con 0x7f991806b4d0 2024-02-21T23:10:12.036 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:10:12.039 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-21T23:10:12.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.048+0000 7f993cc00640 1 -- 172.21.15.28:0/1466822234 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f991806b4d0 msgr2=0x7f991806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:12.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.048+0000 7f993cc00640 1 --2- 172.21.15.28:0/1466822234 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f991806b4d0 0x7f991806d990 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f9924001050 tx=0x7f9924001190 comp rx=0 tx=0).stop 2024-02-21T23:10:12.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.048+0000 7f993cc00640 1 -- 172.21.15.28:0/1466822234 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99380a85e0 msgr2=0x7f99380abfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:12.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.048+0000 7f993cc00640 1 --2- 172.21.15.28:0/1466822234 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99380a85e0 0x7f99380abfe0 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7f99280115e0 tx=0x7f9928011aa0 comp rx=0 tx=0).stop 2024-02-21T23:10:12.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.048+0000 7f993cc00640 1 -- 172.21.15.28:0/1466822234 shutdown_connections 2024-02-21T23:10:12.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.048+0000 7f993cc00640 1 --2- 172.21.15.28:0/1466822234 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f991806b4d0 0x7f991806d990 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:12.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.048+0000 7f993cc00640 1 --2- 172.21.15.28:0/1466822234 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f99380a9950 0x7f99381e9990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:12.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.048+0000 7f993cc00640 1 --2- 172.21.15.28:0/1466822234 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f99380aad40 0x7f99380ac520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:12.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.048+0000 7f993cc00640 1 --2- 172.21.15.28:0/1466822234 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99380a85e0 0x7f99380abfe0 unknown :-1 s=CLOSED pgs=1745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:12.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.048+0000 7f993cc00640 1 -- 172.21.15.28:0/1466822234 >> 172.21.15.28:0/1466822234 conn(0x7f993806ae70 msgr2=0x7f9938153730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:12.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.048+0000 7f993cc00640 1 -- 172.21.15.28:0/1466822234 shutdown_connections 2024-02-21T23:10:12.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.048+0000 7f993cc00640 1 -- 172.21.15.28:0/1466822234 wait complete. 2024-02-21T23:10:12.056 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":378,"stamp":"2024-02-21T23:10:10.824304+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":593450,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":219,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":19,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":171,"ondisk_log_size":171,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":309,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":264816,"kb_used_data":15344,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754709904,"statfs":{"total":773094113280,"available":772822941696,"internally_reserved":0,"allocated":15712256,"data_stored":8763576,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":672,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002075"},"pg_stats":[{"pgid":"26.3d","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.386132+0000","last_change":"2024-02-21T23:10:05.386132+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.386132+0000","last_undegraded":"2024-02-21T23:10:05.386132+0000","last_fullsized":"2024-02-21T23:10:05.386132+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.3f","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.394150+0000","last_change":"2024-02-21T23:09:54.279018+0000","last_active":"2024-02-21T23:10:05.394150+0000","last_peered":"2024-02-21T23:10:05.394150+0000","last_clean":"2024-02-21T23:10:05.394150+0000","last_became_active":"2024-02-21T23:09:54.278712+0000","last_became_peered":"2024-02-21T23:09:54.278712+0000","last_unstale":"2024-02-21T23:10:05.394150+0000","last_undegraded":"2024-02-21T23:10:05.394150+0000","last_fullsized":"2024-02-21T23:10:05.394150+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:55:18.591062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.3c","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232651+0000","last_change":"2024-02-21T23:10:06.385434+0000","last_active":"2024-02-21T23:10:08.232651+0000","last_peered":"2024-02-21T23:10:08.232651+0000","last_clean":"2024-02-21T23:10:08.232651+0000","last_became_active":"2024-02-21T23:10:06.384878+0000","last_became_peered":"2024-02-21T23:10:06.384878+0000","last_unstale":"2024-02-21T23:10:08.232651+0000","last_undegraded":"2024-02-21T23:10:08.232651+0000","last_fullsized":"2024-02-21T23:10:08.232651+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:52:21.069235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3e","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.373373+0000","last_change":"2024-02-21T23:09:54.285889+0000","last_active":"2024-02-21T23:10:05.373373+0000","last_peered":"2024-02-21T23:10:05.373373+0000","last_clean":"2024-02-21T23:10:05.373373+0000","last_became_active":"2024-02-21T23:09:54.284810+0000","last_became_peered":"2024-02-21T23:09:54.284810+0000","last_unstale":"2024-02-21T23:10:05.373373+0000","last_undegraded":"2024-02-21T23:10:05.373373+0000","last_fullsized":"2024-02-21T23:10:05.373373+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:04:16.317089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.3f","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236866+0000","last_change":"2024-02-21T23:10:06.389219+0000","last_active":"2024-02-21T23:10:08.236866+0000","last_peered":"2024-02-21T23:10:08.236866+0000","last_clean":"2024-02-21T23:10:08.236866+0000","last_became_active":"2024-02-21T23:10:06.388931+0000","last_became_peered":"2024-02-21T23:10:06.388931+0000","last_unstale":"2024-02-21T23:10:08.236866+0000","last_undegraded":"2024-02-21T23:10:08.236866+0000","last_fullsized":"2024-02-21T23:10:08.236866+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:46:45.683446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.3d","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.392968+0000","last_change":"2024-02-21T23:09:54.263978+0000","last_active":"2024-02-21T23:10:05.392968+0000","last_peered":"2024-02-21T23:10:05.392968+0000","last_clean":"2024-02-21T23:10:05.392968+0000","last_became_active":"2024-02-21T23:09:54.263467+0000","last_became_peered":"2024-02-21T23:09:54.263467+0000","last_unstale":"2024-02-21T23:10:05.392968+0000","last_undegraded":"2024-02-21T23:10:05.392968+0000","last_fullsized":"2024-02-21T23:10:05.392968+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:46:35.436815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.3e","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236400+0000","last_change":"2024-02-21T23:10:06.406362+0000","last_active":"2024-02-21T23:10:08.236400+0000","last_peered":"2024-02-21T23:10:08.236400+0000","last_clean":"2024-02-21T23:10:08.236400+0000","last_became_active":"2024-02-21T23:10:06.405859+0000","last_became_peered":"2024-02-21T23:10:06.405859+0000","last_unstale":"2024-02-21T23:10:08.236400+0000","last_undegraded":"2024-02-21T23:10:08.236400+0000","last_fullsized":"2024-02-21T23:10:08.236400+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:09:17.354064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.3c","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.390902+0000","last_change":"2024-02-21T23:09:54.279400+0000","last_active":"2024-02-21T23:10:05.390902+0000","last_peered":"2024-02-21T23:10:05.390902+0000","last_clean":"2024-02-21T23:10:05.390902+0000","last_became_active":"2024-02-21T23:09:54.279076+0000","last_became_peered":"2024-02-21T23:09:54.279076+0000","last_unstale":"2024-02-21T23:10:05.390902+0000","last_undegraded":"2024-02-21T23:10:05.390902+0000","last_fullsized":"2024-02-21T23:10:05.390902+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:02:11.247387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.39","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.375184+0000","last_change":"2024-02-21T23:10:06.375038+0000","last_active":"2024-02-21T23:10:07.375184+0000","last_peered":"2024-02-21T23:10:07.375184+0000","last_clean":"2024-02-21T23:10:07.375184+0000","last_became_active":"2024-02-21T23:10:06.374714+0000","last_became_peered":"2024-02-21T23:10:06.374714+0000","last_unstale":"2024-02-21T23:10:07.375184+0000","last_undegraded":"2024-02-21T23:10:07.375184+0000","last_fullsized":"2024-02-21T23:10:07.375184+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:48:19.473212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.3b","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.233882+0000","last_change":"2024-02-21T23:09:54.271682+0000","last_active":"2024-02-21T23:10:08.233882+0000","last_peered":"2024-02-21T23:10:08.233882+0000","last_clean":"2024-02-21T23:10:08.233882+0000","last_became_active":"2024-02-21T23:09:54.271297+0000","last_became_peered":"2024-02-21T23:09:54.271297+0000","last_unstale":"2024-02-21T23:10:08.233882+0000","last_undegraded":"2024-02-21T23:10:08.233882+0000","last_fullsized":"2024-02-21T23:10:08.233882+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:16:10.796552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.38","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.362732+0000","last_change":"2024-02-21T23:10:05.362732+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.362732+0000","last_undegraded":"2024-02-21T23:10:05.362732+0000","last_fullsized":"2024-02-21T23:10:05.362732+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3a","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.369709+0000","last_change":"2024-02-21T23:09:54.281830+0000","last_active":"2024-02-21T23:10:05.369709+0000","last_peered":"2024-02-21T23:10:05.369709+0000","last_clean":"2024-02-21T23:10:05.369709+0000","last_became_active":"2024-02-21T23:09:54.281334+0000","last_became_peered":"2024-02-21T23:09:54.281334+0000","last_unstale":"2024-02-21T23:10:05.369709+0000","last_undegraded":"2024-02-21T23:10:05.369709+0000","last_fullsized":"2024-02-21T23:10:05.369709+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:07:24.304915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.3b","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.233350+0000","last_change":"2024-02-21T23:10:06.385935+0000","last_active":"2024-02-21T23:10:08.233350+0000","last_peered":"2024-02-21T23:10:08.233350+0000","last_clean":"2024-02-21T23:10:08.233350+0000","last_became_active":"2024-02-21T23:10:06.385657+0000","last_became_peered":"2024-02-21T23:10:06.385657+0000","last_unstale":"2024-02-21T23:10:08.233350+0000","last_undegraded":"2024-02-21T23:10:08.233350+0000","last_fullsized":"2024-02-21T23:10:08.233350+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:27:47.158633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.39","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.233971+0000","last_change":"2024-02-21T23:09:54.277806+0000","last_active":"2024-02-21T23:10:08.233971+0000","last_peered":"2024-02-21T23:10:08.233971+0000","last_clean":"2024-02-21T23:10:08.233971+0000","last_became_active":"2024-02-21T23:09:54.277446+0000","last_became_peered":"2024-02-21T23:09:54.277446+0000","last_unstale":"2024-02-21T23:10:08.233971+0000","last_undegraded":"2024-02-21T23:10:08.233971+0000","last_fullsized":"2024-02-21T23:10:08.233971+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:59:34.646775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.3a","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.231962+0000","last_change":"2024-02-21T23:10:06.394386+0000","last_active":"2024-02-21T23:10:08.231962+0000","last_peered":"2024-02-21T23:10:08.231962+0000","last_clean":"2024-02-21T23:10:08.231962+0000","last_became_active":"2024-02-21T23:10:06.391655+0000","last_became_peered":"2024-02-21T23:10:06.391655+0000","last_unstale":"2024-02-21T23:10:08.231962+0000","last_undegraded":"2024-02-21T23:10:08.231962+0000","last_fullsized":"2024-02-21T23:10:08.231962+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:15:42.134184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.38","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232805+0000","last_change":"2024-02-21T23:09:54.286729+0000","last_active":"2024-02-21T23:10:08.232805+0000","last_peered":"2024-02-21T23:10:08.232805+0000","last_clean":"2024-02-21T23:10:08.232805+0000","last_became_active":"2024-02-21T23:09:54.286427+0000","last_became_peered":"2024-02-21T23:09:54.286427+0000","last_unstale":"2024-02-21T23:10:08.232805+0000","last_undegraded":"2024-02-21T23:10:08.232805+0000","last_fullsized":"2024-02-21T23:10:08.232805+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:13:27.311923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.35","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.373594+0000","last_change":"2024-02-21T23:10:05.373594+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.373594+0000","last_undegraded":"2024-02-21T23:10:05.373594+0000","last_fullsized":"2024-02-21T23:10:05.373594+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.37","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.237533+0000","last_change":"2024-02-21T23:09:54.286612+0000","last_active":"2024-02-21T23:10:08.237533+0000","last_peered":"2024-02-21T23:10:08.237533+0000","last_clean":"2024-02-21T23:10:08.237533+0000","last_became_active":"2024-02-21T23:09:54.286306+0000","last_became_peered":"2024-02-21T23:09:54.286306+0000","last_unstale":"2024-02-21T23:10:08.237533+0000","last_undegraded":"2024-02-21T23:10:08.237533+0000","last_fullsized":"2024-02-21T23:10:08.237533+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:04:16.317089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.34","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232897+0000","last_change":"2024-02-21T23:10:06.385190+0000","last_active":"2024-02-21T23:10:08.232897+0000","last_peered":"2024-02-21T23:10:08.232897+0000","last_clean":"2024-02-21T23:10:08.232897+0000","last_became_active":"2024-02-21T23:10:06.384496+0000","last_became_peered":"2024-02-21T23:10:06.384496+0000","last_unstale":"2024-02-21T23:10:08.232897+0000","last_undegraded":"2024-02-21T23:10:08.232897+0000","last_fullsized":"2024-02-21T23:10:08.232897+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:28:15.352760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.36","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.237062+0000","last_change":"2024-02-21T23:09:54.288019+0000","last_active":"2024-02-21T23:10:08.237062+0000","last_peered":"2024-02-21T23:10:08.237062+0000","last_clean":"2024-02-21T23:10:08.237062+0000","last_became_active":"2024-02-21T23:09:54.287742+0000","last_became_peered":"2024-02-21T23:09:54.287742+0000","last_unstale":"2024-02-21T23:10:08.237062+0000","last_undegraded":"2024-02-21T23:10:08.237062+0000","last_fullsized":"2024-02-21T23:10:08.237062+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:13:33.442227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.37","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236970+0000","last_change":"2024-02-21T23:10:06.398025+0000","last_active":"2024-02-21T23:10:08.236970+0000","last_peered":"2024-02-21T23:10:08.236970+0000","last_clean":"2024-02-21T23:10:08.236970+0000","last_became_active":"2024-02-21T23:10:06.395723+0000","last_became_peered":"2024-02-21T23:10:06.395723+0000","last_unstale":"2024-02-21T23:10:08.236970+0000","last_undegraded":"2024-02-21T23:10:08.236970+0000","last_fullsized":"2024-02-21T23:10:08.236970+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:50:33.682312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.35","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.378855+0000","last_change":"2024-02-21T23:09:54.283918+0000","last_active":"2024-02-21T23:10:05.378855+0000","last_peered":"2024-02-21T23:10:05.378855+0000","last_clean":"2024-02-21T23:10:05.378855+0000","last_became_active":"2024-02-21T23:09:54.280599+0000","last_became_peered":"2024-02-21T23:09:54.280599+0000","last_unstale":"2024-02-21T23:10:05.378855+0000","last_undegraded":"2024-02-21T23:10:05.378855+0000","last_fullsized":"2024-02-21T23:10:05.378855+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:33:07.362294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.36","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234837+0000","last_change":"2024-02-21T23:10:06.377423+0000","last_active":"2024-02-21T23:10:08.234837+0000","last_peered":"2024-02-21T23:10:08.234837+0000","last_clean":"2024-02-21T23:10:08.234837+0000","last_became_active":"2024-02-21T23:10:06.376998+0000","last_became_peered":"2024-02-21T23:10:06.376998+0000","last_unstale":"2024-02-21T23:10:08.234837+0000","last_undegraded":"2024-02-21T23:10:08.234837+0000","last_fullsized":"2024-02-21T23:10:08.234837+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:26:32.419695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.34","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.373186+0000","last_change":"2024-02-21T23:09:54.282846+0000","last_active":"2024-02-21T23:10:05.373186+0000","last_peered":"2024-02-21T23:10:05.373186+0000","last_clean":"2024-02-21T23:10:05.373186+0000","last_became_active":"2024-02-21T23:09:54.282117+0000","last_became_peered":"2024-02-21T23:09:54.282117+0000","last_unstale":"2024-02-21T23:10:05.373186+0000","last_undegraded":"2024-02-21T23:10:05.373186+0000","last_fullsized":"2024-02-21T23:10:05.373186+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:33:07.362294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.31","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234391+0000","last_change":"2024-02-21T23:10:06.382496+0000","last_active":"2024-02-21T23:10:08.234391+0000","last_peered":"2024-02-21T23:10:08.234391+0000","last_clean":"2024-02-21T23:10:08.234391+0000","last_became_active":"2024-02-21T23:10:06.379727+0000","last_became_peered":"2024-02-21T23:10:06.379727+0000","last_unstale":"2024-02-21T23:10:08.234391+0000","last_undegraded":"2024-02-21T23:10:08.234391+0000","last_fullsized":"2024-02-21T23:10:08.234391+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:10:21.186085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.33","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234540+0000","last_change":"2024-02-21T23:09:54.284818+0000","last_active":"2024-02-21T23:10:08.234540+0000","last_peered":"2024-02-21T23:10:08.234540+0000","last_clean":"2024-02-21T23:10:08.234540+0000","last_became_active":"2024-02-21T23:09:54.284161+0000","last_became_peered":"2024-02-21T23:09:54.284161+0000","last_unstale":"2024-02-21T23:10:08.234540+0000","last_undegraded":"2024-02-21T23:10:08.234540+0000","last_fullsized":"2024-02-21T23:10:08.234540+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:02:12.340921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.30","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.231818+0000","last_change":"2024-02-21T23:10:06.388372+0000","last_active":"2024-02-21T23:10:08.231818+0000","last_peered":"2024-02-21T23:10:08.231818+0000","last_clean":"2024-02-21T23:10:08.231818+0000","last_became_active":"2024-02-21T23:10:06.388099+0000","last_became_peered":"2024-02-21T23:10:06.388099+0000","last_unstale":"2024-02-21T23:10:08.231818+0000","last_undegraded":"2024-02-21T23:10:08.231818+0000","last_fullsized":"2024-02-21T23:10:08.231818+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:55:32.267269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.32","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.231579+0000","last_change":"2024-02-21T23:09:54.272136+0000","last_active":"2024-02-21T23:10:08.231579+0000","last_peered":"2024-02-21T23:10:08.231579+0000","last_clean":"2024-02-21T23:10:08.231579+0000","last_became_active":"2024-02-21T23:09:54.271097+0000","last_became_peered":"2024-02-21T23:09:54.271097+0000","last_unstale":"2024-02-21T23:10:08.231579+0000","last_undegraded":"2024-02-21T23:10:08.231579+0000","last_fullsized":"2024-02-21T23:10:08.231579+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:30:45.437334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.33","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.381970+0000","last_change":"2024-02-21T23:10:05.381970+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.381970+0000","last_undegraded":"2024-02-21T23:10:05.381970+0000","last_fullsized":"2024-02-21T23:10:05.381970+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.31","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.377899+0000","last_change":"2024-02-21T23:09:54.281268+0000","last_active":"2024-02-21T23:10:05.377899+0000","last_peered":"2024-02-21T23:10:05.377899+0000","last_clean":"2024-02-21T23:10:05.377899+0000","last_became_active":"2024-02-21T23:09:54.280946+0000","last_became_peered":"2024-02-21T23:09:54.280946+0000","last_unstale":"2024-02-21T23:10:05.377899+0000","last_undegraded":"2024-02-21T23:10:05.377899+0000","last_fullsized":"2024-02-21T23:10:05.377899+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:03:44.493909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.32","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.374763+0000","last_change":"2024-02-21T23:10:06.381260+0000","last_active":"2024-02-21T23:10:07.374763+0000","last_peered":"2024-02-21T23:10:07.374763+0000","last_clean":"2024-02-21T23:10:07.374763+0000","last_became_active":"2024-02-21T23:10:06.380936+0000","last_became_peered":"2024-02-21T23:10:06.380936+0000","last_unstale":"2024-02-21T23:10:07.374763+0000","last_undegraded":"2024-02-21T23:10:07.374763+0000","last_fullsized":"2024-02-21T23:10:07.374763+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:41:14.290670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.30","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.387425+0000","last_change":"2024-02-21T23:09:54.278731+0000","last_active":"2024-02-21T23:10:05.387425+0000","last_peered":"2024-02-21T23:10:05.387425+0000","last_clean":"2024-02-21T23:10:05.387425+0000","last_became_active":"2024-02-21T23:09:54.278390+0000","last_became_peered":"2024-02-21T23:09:54.278390+0000","last_unstale":"2024-02-21T23:10:05.387425+0000","last_undegraded":"2024-02-21T23:10:05.387425+0000","last_fullsized":"2024-02-21T23:10:05.387425+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:28:54.079553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.2d","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.382164+0000","last_change":"2024-02-21T23:10:05.382164+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.382164+0000","last_undegraded":"2024-02-21T23:10:05.382164+0000","last_fullsized":"2024-02-21T23:10:05.382164+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.2f","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.237800+0000","last_change":"2024-02-21T23:09:54.285947+0000","last_active":"2024-02-21T23:10:08.237800+0000","last_peered":"2024-02-21T23:10:08.237800+0000","last_clean":"2024-02-21T23:10:08.237800+0000","last_became_active":"2024-02-21T23:09:54.285658+0000","last_became_peered":"2024-02-21T23:09:54.285658+0000","last_unstale":"2024-02-21T23:10:08.237800+0000","last_undegraded":"2024-02-21T23:10:08.237800+0000","last_fullsized":"2024-02-21T23:10:08.237800+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:20:31.125684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2c","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236540+0000","last_change":"2024-02-21T23:10:06.376864+0000","last_active":"2024-02-21T23:10:08.236540+0000","last_peered":"2024-02-21T23:10:08.236540+0000","last_clean":"2024-02-21T23:10:08.236540+0000","last_became_active":"2024-02-21T23:10:06.376486+0000","last_became_peered":"2024-02-21T23:10:06.376486+0000","last_unstale":"2024-02-21T23:10:08.236540+0000","last_undegraded":"2024-02-21T23:10:08.236540+0000","last_fullsized":"2024-02-21T23:10:08.236540+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:22:04.227521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.2e","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.391407+0000","last_change":"2024-02-21T23:09:54.287286+0000","last_active":"2024-02-21T23:10:05.391407+0000","last_peered":"2024-02-21T23:10:05.391407+0000","last_clean":"2024-02-21T23:10:05.391407+0000","last_became_active":"2024-02-21T23:09:54.286924+0000","last_became_peered":"2024-02-21T23:09:54.286924+0000","last_unstale":"2024-02-21T23:10:05.391407+0000","last_undegraded":"2024-02-21T23:10:05.391407+0000","last_fullsized":"2024-02-21T23:10:05.391407+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:47:29.349669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.2f","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.235953+0000","last_change":"2024-02-21T23:10:06.377113+0000","last_active":"2024-02-21T23:10:08.235953+0000","last_peered":"2024-02-21T23:10:08.235953+0000","last_clean":"2024-02-21T23:10:08.235953+0000","last_became_active":"2024-02-21T23:10:06.376790+0000","last_became_peered":"2024-02-21T23:10:06.376790+0000","last_unstale":"2024-02-21T23:10:08.235953+0000","last_undegraded":"2024-02-21T23:10:08.235953+0000","last_fullsized":"2024-02-21T23:10:08.235953+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:38:03.009039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.2d","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.379032+0000","last_change":"2024-02-21T23:09:54.279884+0000","last_active":"2024-02-21T23:10:05.379032+0000","last_peered":"2024-02-21T23:10:05.379032+0000","last_clean":"2024-02-21T23:10:05.379032+0000","last_became_active":"2024-02-21T23:09:54.276608+0000","last_became_peered":"2024-02-21T23:09:54.276608+0000","last_unstale":"2024-02-21T23:10:05.379032+0000","last_undegraded":"2024-02-21T23:10:05.379032+0000","last_fullsized":"2024-02-21T23:10:05.379032+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:05:09.189420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.2e","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.383479+0000","last_change":"2024-02-21T23:10:05.383479+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.383479+0000","last_undegraded":"2024-02-21T23:10:05.383479+0000","last_fullsized":"2024-02-21T23:10:05.383479+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.2c","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.375678+0000","last_change":"2024-02-21T23:09:54.288944+0000","last_active":"2024-02-21T23:10:07.375678+0000","last_peered":"2024-02-21T23:10:07.375678+0000","last_clean":"2024-02-21T23:10:07.375678+0000","last_became_active":"2024-02-21T23:09:54.285913+0000","last_became_peered":"2024-02-21T23:09:54.285913+0000","last_unstale":"2024-02-21T23:10:07.375678+0000","last_undegraded":"2024-02-21T23:10:07.375678+0000","last_fullsized":"2024-02-21T23:10:07.375678+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:39:33.609273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.29","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.363048+0000","last_change":"2024-02-21T23:10:05.363048+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.363048+0000","last_undegraded":"2024-02-21T23:10:05.363048+0000","last_fullsized":"2024-02-21T23:10:05.363048+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.2b","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234769+0000","last_change":"2024-02-21T23:09:54.285036+0000","last_active":"2024-02-21T23:10:08.234769+0000","last_peered":"2024-02-21T23:10:08.234769+0000","last_clean":"2024-02-21T23:10:08.234769+0000","last_became_active":"2024-02-21T23:09:54.284542+0000","last_became_peered":"2024-02-21T23:09:54.284542+0000","last_unstale":"2024-02-21T23:10:08.234769+0000","last_undegraded":"2024-02-21T23:10:08.234769+0000","last_fullsized":"2024-02-21T23:10:08.234769+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T11:05:01.241636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.28","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234521+0000","last_change":"2024-02-21T23:10:06.396769+0000","last_active":"2024-02-21T23:10:08.234521+0000","last_peered":"2024-02-21T23:10:08.234521+0000","last_clean":"2024-02-21T23:10:08.234521+0000","last_became_active":"2024-02-21T23:10:06.384723+0000","last_became_peered":"2024-02-21T23:10:06.384723+0000","last_unstale":"2024-02-21T23:10:08.234521+0000","last_undegraded":"2024-02-21T23:10:08.234521+0000","last_fullsized":"2024-02-21T23:10:08.234521+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:45:13.867812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2a","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.370120+0000","last_change":"2024-02-21T23:09:54.273350+0000","last_active":"2024-02-21T23:10:05.370120+0000","last_peered":"2024-02-21T23:10:05.370120+0000","last_clean":"2024-02-21T23:10:05.370120+0000","last_became_active":"2024-02-21T23:09:54.273022+0000","last_became_peered":"2024-02-21T23:09:54.273022+0000","last_unstale":"2024-02-21T23:10:05.370120+0000","last_undegraded":"2024-02-21T23:10:05.370120+0000","last_fullsized":"2024-02-21T23:10:05.370120+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:34:28.642087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.2b","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.235284+0000","last_change":"2024-02-21T23:10:06.394830+0000","last_active":"2024-02-21T23:10:08.235284+0000","last_peered":"2024-02-21T23:10:08.235284+0000","last_clean":"2024-02-21T23:10:08.235284+0000","last_became_active":"2024-02-21T23:10:06.386513+0000","last_became_peered":"2024-02-21T23:10:06.386513+0000","last_unstale":"2024-02-21T23:10:08.235284+0000","last_undegraded":"2024-02-21T23:10:08.235284+0000","last_fullsized":"2024-02-21T23:10:08.235284+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:30:09.767949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.29","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.230852+0000","last_change":"2024-02-21T23:09:54.279718+0000","last_active":"2024-02-21T23:10:08.230852+0000","last_peered":"2024-02-21T23:10:08.230852+0000","last_clean":"2024-02-21T23:10:08.230852+0000","last_became_active":"2024-02-21T23:09:54.278989+0000","last_became_peered":"2024-02-21T23:09:54.278989+0000","last_unstale":"2024-02-21T23:10:08.230852+0000","last_undegraded":"2024-02-21T23:10:08.230852+0000","last_fullsized":"2024-02-21T23:10:08.230852+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:36:41.922548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.2a","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.235326+0000","last_change":"2024-02-21T23:10:06.397414+0000","last_active":"2024-02-21T23:10:08.235326+0000","last_peered":"2024-02-21T23:10:08.235326+0000","last_clean":"2024-02-21T23:10:08.235326+0000","last_became_active":"2024-02-21T23:10:06.396982+0000","last_became_peered":"2024-02-21T23:10:06.396982+0000","last_unstale":"2024-02-21T23:10:08.235326+0000","last_undegraded":"2024-02-21T23:10:08.235326+0000","last_fullsized":"2024-02-21T23:10:08.235326+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:05:37.143681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.28","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.387543+0000","last_change":"2024-02-21T23:09:54.280624+0000","last_active":"2024-02-21T23:10:05.387543+0000","last_peered":"2024-02-21T23:10:05.387543+0000","last_clean":"2024-02-21T23:10:05.387543+0000","last_became_active":"2024-02-21T23:09:54.275644+0000","last_became_peered":"2024-02-21T23:09:54.275644+0000","last_unstale":"2024-02-21T23:10:05.387543+0000","last_undegraded":"2024-02-21T23:10:05.387543+0000","last_fullsized":"2024-02-21T23:10:05.387543+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:23:27.511315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.25","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.365921+0000","last_change":"2024-02-21T23:10:05.365921+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.365921+0000","last_undegraded":"2024-02-21T23:10:05.365921+0000","last_fullsized":"2024-02-21T23:10:05.365921+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.27","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236738+0000","last_change":"2024-02-21T23:09:54.275341+0000","last_active":"2024-02-21T23:10:08.236738+0000","last_peered":"2024-02-21T23:10:08.236738+0000","last_clean":"2024-02-21T23:10:08.236738+0000","last_became_active":"2024-02-21T23:09:54.275010+0000","last_became_peered":"2024-02-21T23:09:54.275010+0000","last_unstale":"2024-02-21T23:10:08.236738+0000","last_undegraded":"2024-02-21T23:10:08.236738+0000","last_fullsized":"2024-02-21T23:10:08.236738+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:42:08.123188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.24","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236163+0000","last_change":"2024-02-21T23:10:06.386616+0000","last_active":"2024-02-21T23:10:08.236163+0000","last_peered":"2024-02-21T23:10:08.236163+0000","last_clean":"2024-02-21T23:10:08.236163+0000","last_became_active":"2024-02-21T23:10:06.386339+0000","last_became_peered":"2024-02-21T23:10:06.386339+0000","last_unstale":"2024-02-21T23:10:08.236163+0000","last_undegraded":"2024-02-21T23:10:08.236163+0000","last_fullsized":"2024-02-21T23:10:08.236163+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:44:41.892333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.26","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.376034+0000","last_change":"2024-02-21T23:09:54.260902+0000","last_active":"2024-02-21T23:10:07.376034+0000","last_peered":"2024-02-21T23:10:07.376034+0000","last_clean":"2024-02-21T23:10:07.376034+0000","last_became_active":"2024-02-21T23:09:54.260084+0000","last_became_peered":"2024-02-21T23:09:54.260084+0000","last_unstale":"2024-02-21T23:10:07.376034+0000","last_undegraded":"2024-02-21T23:10:07.376034+0000","last_fullsized":"2024-02-21T23:10:07.376034+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:04:05.858817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.27","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.375938+0000","last_change":"2024-02-21T23:10:06.374537+0000","last_active":"2024-02-21T23:10:07.375938+0000","last_peered":"2024-02-21T23:10:07.375938+0000","last_clean":"2024-02-21T23:10:07.375938+0000","last_became_active":"2024-02-21T23:10:06.374201+0000","last_became_peered":"2024-02-21T23:10:06.374201+0000","last_unstale":"2024-02-21T23:10:07.375938+0000","last_undegraded":"2024-02-21T23:10:07.375938+0000","last_fullsized":"2024-02-21T23:10:07.375938+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:52:55.561353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.25","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.373077+0000","last_change":"2024-02-21T23:09:54.284116+0000","last_active":"2024-02-21T23:10:05.373077+0000","last_peered":"2024-02-21T23:10:05.373077+0000","last_clean":"2024-02-21T23:10:05.373077+0000","last_became_active":"2024-02-21T23:09:54.283802+0000","last_became_peered":"2024-02-21T23:09:54.283802+0000","last_unstale":"2024-02-21T23:10:05.373077+0000","last_undegraded":"2024-02-21T23:10:05.373077+0000","last_fullsized":"2024-02-21T23:10:05.373077+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:05:09.189420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.26","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.374040+0000","last_change":"2024-02-21T23:10:05.374040+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.374040+0000","last_undegraded":"2024-02-21T23:10:05.374040+0000","last_fullsized":"2024-02-21T23:10:05.374040+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.24","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236346+0000","last_change":"2024-02-21T23:09:54.287481+0000","last_active":"2024-02-21T23:10:08.236346+0000","last_peered":"2024-02-21T23:10:08.236346+0000","last_clean":"2024-02-21T23:10:08.236346+0000","last_became_active":"2024-02-21T23:09:54.287137+0000","last_became_peered":"2024-02-21T23:09:54.287137+0000","last_unstale":"2024-02-21T23:10:08.236346+0000","last_undegraded":"2024-02-21T23:10:08.236346+0000","last_fullsized":"2024-02-21T23:10:08.236346+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:50:46.472002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.21","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.374988+0000","last_change":"2024-02-21T23:10:06.370867+0000","last_active":"2024-02-21T23:10:07.374988+0000","last_peered":"2024-02-21T23:10:07.374988+0000","last_clean":"2024-02-21T23:10:07.374988+0000","last_became_active":"2024-02-21T23:10:06.370583+0000","last_became_peered":"2024-02-21T23:10:06.370583+0000","last_unstale":"2024-02-21T23:10:07.374988+0000","last_undegraded":"2024-02-21T23:10:07.374988+0000","last_fullsized":"2024-02-21T23:10:07.374988+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:18:29.588662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.23","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.372190+0000","last_change":"2024-02-21T23:09:54.281613+0000","last_active":"2024-02-21T23:10:05.372190+0000","last_peered":"2024-02-21T23:10:05.372190+0000","last_clean":"2024-02-21T23:10:05.372190+0000","last_became_active":"2024-02-21T23:09:54.281316+0000","last_became_peered":"2024-02-21T23:09:54.281316+0000","last_unstale":"2024-02-21T23:10:05.372190+0000","last_undegraded":"2024-02-21T23:10:05.372190+0000","last_fullsized":"2024-02-21T23:10:05.372190+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:13:33.442227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.20","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232111+0000","last_change":"2024-02-21T23:10:06.382019+0000","last_active":"2024-02-21T23:10:08.232111+0000","last_peered":"2024-02-21T23:10:08.232111+0000","last_clean":"2024-02-21T23:10:08.232111+0000","last_became_active":"2024-02-21T23:10:06.381739+0000","last_became_peered":"2024-02-21T23:10:06.381739+0000","last_unstale":"2024-02-21T23:10:08.232111+0000","last_undegraded":"2024-02-21T23:10:08.232111+0000","last_fullsized":"2024-02-21T23:10:08.232111+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:25:17.018194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.22","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.390353+0000","last_change":"2024-02-21T23:09:54.275665+0000","last_active":"2024-02-21T23:10:05.390353+0000","last_peered":"2024-02-21T23:10:05.390353+0000","last_clean":"2024-02-21T23:10:05.390353+0000","last_became_active":"2024-02-21T23:09:54.275143+0000","last_became_peered":"2024-02-21T23:09:54.275143+0000","last_unstale":"2024-02-21T23:10:05.390353+0000","last_undegraded":"2024-02-21T23:10:05.390353+0000","last_fullsized":"2024-02-21T23:10:05.390353+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:37:42.976792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.23","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.371720+0000","last_change":"2024-02-21T23:10:05.371720+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.371720+0000","last_undegraded":"2024-02-21T23:10:05.371720+0000","last_fullsized":"2024-02-21T23:10:05.371720+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.21","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.378116+0000","last_change":"2024-02-21T23:09:54.281200+0000","last_active":"2024-02-21T23:10:05.378116+0000","last_peered":"2024-02-21T23:10:05.378116+0000","last_clean":"2024-02-21T23:10:05.378116+0000","last_became_active":"2024-02-21T23:09:54.277867+0000","last_became_peered":"2024-02-21T23:09:54.277867+0000","last_unstale":"2024-02-21T23:10:05.378116+0000","last_undegraded":"2024-02-21T23:10:05.378116+0000","last_fullsized":"2024-02-21T23:10:05.378116+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:59:34.646775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.22","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.375181+0000","last_change":"2024-02-21T23:10:05.375181+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.375181+0000","last_undegraded":"2024-02-21T23:10:05.375181+0000","last_fullsized":"2024-02-21T23:10:05.375181+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.20","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.390496+0000","last_change":"2024-02-21T23:09:54.276078+0000","last_active":"2024-02-21T23:10:05.390496+0000","last_peered":"2024-02-21T23:10:05.390496+0000","last_clean":"2024-02-21T23:10:05.390496+0000","last_became_active":"2024-02-21T23:09:54.275724+0000","last_became_peered":"2024-02-21T23:09:54.275724+0000","last_unstale":"2024-02-21T23:10:05.390496+0000","last_undegraded":"2024-02-21T23:10:05.390496+0000","last_fullsized":"2024-02-21T23:10:05.390496+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:48:07.369182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.377674+0000","last_change":"2024-02-21T23:10:05.377674+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.377674+0000","last_undegraded":"2024-02-21T23:10:05.377674+0000","last_fullsized":"2024-02-21T23:10:05.377674+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.230111+0000","last_change":"2024-02-21T23:10:05.385230+0000","last_active":"2024-02-21T23:10:08.230111+0000","last_peered":"2024-02-21T23:10:08.230111+0000","last_clean":"2024-02-21T23:10:08.230111+0000","last_became_active":"2024-02-21T23:10:05.384017+0000","last_became_peered":"2024-02-21T23:10:05.384017+0000","last_unstale":"2024-02-21T23:10:08.230111+0000","last_undegraded":"2024-02-21T23:10:08.230111+0000","last_fullsized":"2024-02-21T23:10:08.230111+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:45:12.852122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.10","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.368345+0000","last_change":"2024-02-21T23:09:54.278972+0000","last_active":"2024-02-21T23:10:05.368345+0000","last_peered":"2024-02-21T23:10:05.368345+0000","last_clean":"2024-02-21T23:10:05.368345+0000","last_became_active":"2024-02-21T23:09:54.278146+0000","last_became_peered":"2024-02-21T23:09:54.278146+0000","last_unstale":"2024-02-21T23:10:05.368345+0000","last_undegraded":"2024-02-21T23:10:05.368345+0000","last_fullsized":"2024-02-21T23:10:05.368345+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:59:34.646775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"150'4","reported_seq":22,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.394358+0000","last_change":"2024-02-21T23:09:53.288506+0000","last_active":"2024-02-21T23:10:05.394358+0000","last_peered":"2024-02-21T23:10:05.394358+0000","last_clean":"2024-02-21T23:10:05.394358+0000","last_became_active":"2024-02-21T23:09:53.288199+0000","last_became_peered":"2024-02-21T23:09:53.288199+0000","last_unstale":"2024-02-21T23:10:05.394358+0000","last_undegraded":"2024-02-21T23:10:05.394358+0000","last_fullsized":"2024-02-21T23:10:05.394358+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:18:16.467715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.13","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.378353+0000","last_change":"2024-02-21T23:10:05.378353+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.378353+0000","last_undegraded":"2024-02-21T23:10:05.378353+0000","last_fullsized":"2024-02-21T23:10:05.378353+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.10","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.386340+0000","last_change":"2024-02-21T23:10:05.382619+0000","last_active":"2024-02-21T23:10:05.386340+0000","last_peered":"2024-02-21T23:10:05.386340+0000","last_clean":"2024-02-21T23:10:05.386340+0000","last_became_active":"2024-02-21T23:10:05.382336+0000","last_became_peered":"2024-02-21T23:10:05.382336+0000","last_unstale":"2024-02-21T23:10:05.386340+0000","last_undegraded":"2024-02-21T23:10:05.386340+0000","last_fullsized":"2024-02-21T23:10:05.386340+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:11:55.677946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.231615+0000","last_change":"2024-02-21T23:09:54.278032+0000","last_active":"2024-02-21T23:10:08.231615+0000","last_peered":"2024-02-21T23:10:08.231615+0000","last_clean":"2024-02-21T23:10:08.231615+0000","last_became_active":"2024-02-21T23:09:54.277741+0000","last_became_peered":"2024-02-21T23:09:54.277741+0000","last_unstale":"2024-02-21T23:10:08.231615+0000","last_undegraded":"2024-02-21T23:10:08.231615+0000","last_fullsized":"2024-02-21T23:10:08.231615+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:43:56.610613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"150'1","reported_seq":23,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232900+0000","last_change":"2024-02-21T23:09:53.279374+0000","last_active":"2024-02-21T23:10:08.232900+0000","last_peered":"2024-02-21T23:10:08.232900+0000","last_clean":"2024-02-21T23:10:08.232900+0000","last_became_active":"2024-02-21T23:09:53.279053+0000","last_became_peered":"2024-02-21T23:09:53.279053+0000","last_unstale":"2024-02-21T23:10:08.232900+0000","last_undegraded":"2024-02-21T23:10:08.232900+0000","last_fullsized":"2024-02-21T23:10:08.232900+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:04:15.300172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.10","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.375571+0000","last_change":"2024-02-21T23:10:06.375103+0000","last_active":"2024-02-21T23:10:07.375571+0000","last_peered":"2024-02-21T23:10:07.375571+0000","last_clean":"2024-02-21T23:10:07.375571+0000","last_became_active":"2024-02-21T23:10:06.374814+0000","last_became_peered":"2024-02-21T23:10:06.374814+0000","last_unstale":"2024-02-21T23:10:07.375571+0000","last_undegraded":"2024-02-21T23:10:07.375571+0000","last_fullsized":"2024-02-21T23:10:07.375571+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:42:32.100185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.13","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.395209+0000","last_change":"2024-02-21T23:10:05.392497+0000","last_active":"2024-02-21T23:10:05.395209+0000","last_peered":"2024-02-21T23:10:05.395209+0000","last_clean":"2024-02-21T23:10:05.395209+0000","last_became_active":"2024-02-21T23:10:05.392183+0000","last_became_peered":"2024-02-21T23:10:05.392183+0000","last_unstale":"2024-02-21T23:10:05.395209+0000","last_undegraded":"2024-02-21T23:10:05.395209+0000","last_fullsized":"2024-02-21T23:10:05.395209+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:13:55.723872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.231985+0000","last_change":"2024-02-21T23:09:54.263915+0000","last_active":"2024-02-21T23:10:08.231985+0000","last_peered":"2024-02-21T23:10:08.231985+0000","last_clean":"2024-02-21T23:10:08.231985+0000","last_became_active":"2024-02-21T23:09:54.263634+0000","last_became_peered":"2024-02-21T23:09:54.263634+0000","last_unstale":"2024-02-21T23:10:08.231985+0000","last_undegraded":"2024-02-21T23:10:08.231985+0000","last_fullsized":"2024-02-21T23:10:08.231985+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:47:34.065041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.1d","version":"150'2","reported_seq":20,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.379180+0000","last_change":"2024-02-21T23:09:53.272110+0000","last_active":"2024-02-21T23:10:05.379180+0000","last_peered":"2024-02-21T23:10:05.379180+0000","last_clean":"2024-02-21T23:10:05.379180+0000","last_became_active":"2024-02-21T23:09:53.271641+0000","last_became_peered":"2024-02-21T23:09:53.271641+0000","last_unstale":"2024-02-21T23:10:05.379180+0000","last_undegraded":"2024-02-21T23:10:05.379180+0000","last_fullsized":"2024-02-21T23:10:05.379180+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:46:34.419898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.231438+0000","last_change":"2024-02-21T23:10:06.380458+0000","last_active":"2024-02-21T23:10:08.231438+0000","last_peered":"2024-02-21T23:10:08.231438+0000","last_clean":"2024-02-21T23:10:08.231438+0000","last_became_active":"2024-02-21T23:10:06.380137+0000","last_became_peered":"2024-02-21T23:10:06.380137+0000","last_unstale":"2024-02-21T23:10:08.231438+0000","last_undegraded":"2024-02-21T23:10:08.231438+0000","last_fullsized":"2024-02-21T23:10:08.231438+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:51:13.639406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236172+0000","last_change":"2024-02-21T23:10:05.393823+0000","last_active":"2024-02-21T23:10:08.236172+0000","last_peered":"2024-02-21T23:10:08.236172+0000","last_clean":"2024-02-21T23:10:08.236172+0000","last_became_active":"2024-02-21T23:10:05.390445+0000","last_became_peered":"2024-02-21T23:10:05.390445+0000","last_unstale":"2024-02-21T23:10:08.236172+0000","last_undegraded":"2024-02-21T23:10:08.236172+0000","last_fullsized":"2024-02-21T23:10:08.236172+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:30:56.525674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.13","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.372349+0000","last_change":"2024-02-21T23:09:54.275867+0000","last_active":"2024-02-21T23:10:05.372349+0000","last_peered":"2024-02-21T23:10:05.372349+0000","last_clean":"2024-02-21T23:10:05.372349+0000","last_became_active":"2024-02-21T23:09:54.275591+0000","last_became_peered":"2024-02-21T23:09:54.275591+0000","last_unstale":"2024-02-21T23:10:05.372349+0000","last_undegraded":"2024-02-21T23:10:05.372349+0000","last_fullsized":"2024-02-21T23:10:05.372349+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:42:16.988326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1c","version":"150'2","reported_seq":24,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.375729+0000","last_change":"2024-02-21T23:09:53.274692+0000","last_active":"2024-02-21T23:10:07.375729+0000","last_peered":"2024-02-21T23:10:07.375729+0000","last_clean":"2024-02-21T23:10:07.375729+0000","last_became_active":"2024-02-21T23:09:53.274395+0000","last_became_peered":"2024-02-21T23:09:53.274395+0000","last_unstale":"2024-02-21T23:10:07.375729+0000","last_undegraded":"2024-02-21T23:10:07.375729+0000","last_fullsized":"2024-02-21T23:10:07.375729+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:40:18.453921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.382764+0000","last_change":"2024-02-21T23:10:05.382764+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.382764+0000","last_undegraded":"2024-02-21T23:10:05.382764+0000","last_fullsized":"2024-02-21T23:10:05.382764+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.15","version":"154'3","reported_seq":17,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.233612+0000","last_change":"2024-02-21T23:10:05.374604+0000","last_active":"2024-02-21T23:10:08.233612+0000","last_peered":"2024-02-21T23:10:08.233612+0000","last_clean":"2024-02-21T23:10:08.233612+0000","last_became_active":"2024-02-21T23:10:05.374208+0000","last_became_peered":"2024-02-21T23:10:05.374208+0000","last_unstale":"2024-02-21T23:10:08.233612+0000","last_undegraded":"2024-02-21T23:10:08.233612+0000","last_fullsized":"2024-02-21T23:10:08.233612+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:30:08.752259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.14","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.235541+0000","last_change":"2024-02-21T23:09:54.289092+0000","last_active":"2024-02-21T23:10:08.235541+0000","last_peered":"2024-02-21T23:10:08.235541+0000","last_clean":"2024-02-21T23:10:08.235541+0000","last_became_active":"2024-02-21T23:09:54.288801+0000","last_became_peered":"2024-02-21T23:09:54.288801+0000","last_unstale":"2024-02-21T23:10:08.235541+0000","last_undegraded":"2024-02-21T23:10:08.235541+0000","last_fullsized":"2024-02-21T23:10:08.235541+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:05:09.189420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1b","version":"150'1","reported_seq":23,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236011+0000","last_change":"2024-02-21T23:09:53.283422+0000","last_active":"2024-02-21T23:10:08.236011+0000","last_peered":"2024-02-21T23:10:08.236011+0000","last_clean":"2024-02-21T23:10:08.236011+0000","last_became_active":"2024-02-21T23:09:53.283140+0000","last_became_peered":"2024-02-21T23:09:53.283140+0000","last_unstale":"2024-02-21T23:10:08.236011+0000","last_undegraded":"2024-02-21T23:10:08.236011+0000","last_fullsized":"2024-02-21T23:10:08.236011+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:46:34.419898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.237280+0000","last_change":"2024-02-21T23:10:06.403933+0000","last_active":"2024-02-21T23:10:08.237280+0000","last_peered":"2024-02-21T23:10:08.237280+0000","last_clean":"2024-02-21T23:10:08.237280+0000","last_became_active":"2024-02-21T23:10:06.403548+0000","last_became_peered":"2024-02-21T23:10:06.403548+0000","last_unstale":"2024-02-21T23:10:08.237280+0000","last_undegraded":"2024-02-21T23:10:08.237280+0000","last_fullsized":"2024-02-21T23:10:08.237280+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:11:20.176324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.14","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.394841+0000","last_change":"2024-02-21T23:10:05.388142+0000","last_active":"2024-02-21T23:10:05.394841+0000","last_peered":"2024-02-21T23:10:05.394841+0000","last_clean":"2024-02-21T23:10:05.394841+0000","last_became_active":"2024-02-21T23:10:05.387764+0000","last_became_peered":"2024-02-21T23:10:05.387764+0000","last_unstale":"2024-02-21T23:10:05.394841+0000","last_undegraded":"2024-02-21T23:10:05.394841+0000","last_fullsized":"2024-02-21T23:10:05.394841+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:58:38.451266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.15","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.375532+0000","last_change":"2024-02-21T23:09:54.273728+0000","last_active":"2024-02-21T23:10:07.375532+0000","last_peered":"2024-02-21T23:10:07.375532+0000","last_clean":"2024-02-21T23:10:07.375532+0000","last_became_active":"2024-02-21T23:09:54.273343+0000","last_became_peered":"2024-02-21T23:09:54.273343+0000","last_unstale":"2024-02-21T23:10:07.375532+0000","last_undegraded":"2024-02-21T23:10:07.375532+0000","last_fullsized":"2024-02-21T23:10:07.375532+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:49:15.552003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.1a","version":"150'1","reported_seq":23,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232153+0000","last_change":"2024-02-21T23:09:53.278176+0000","last_active":"2024-02-21T23:10:08.232153+0000","last_peered":"2024-02-21T23:10:08.232153+0000","last_clean":"2024-02-21T23:10:08.232153+0000","last_became_active":"2024-02-21T23:09:53.277880+0000","last_became_peered":"2024-02-21T23:09:53.277880+0000","last_unstale":"2024-02-21T23:10:08.232153+0000","last_undegraded":"2024-02-21T23:10:08.232153+0000","last_fullsized":"2024-02-21T23:10:08.232153+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:05:08.172503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.14","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.364119+0000","last_change":"2024-02-21T23:10:05.364119+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.364119+0000","last_undegraded":"2024-02-21T23:10:05.364119+0000","last_fullsized":"2024-02-21T23:10:05.364119+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236121+0000","last_change":"2024-02-21T23:10:05.375277+0000","last_active":"2024-02-21T23:10:08.236121+0000","last_peered":"2024-02-21T23:10:08.236121+0000","last_clean":"2024-02-21T23:10:08.236121+0000","last_became_active":"2024-02-21T23:10:05.374983+0000","last_became_peered":"2024-02-21T23:10:05.374983+0000","last_unstale":"2024-02-21T23:10:08.236121+0000","last_undegraded":"2024-02-21T23:10:08.236121+0000","last_fullsized":"2024-02-21T23:10:08.236121+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:11:19.160634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.376102+0000","last_change":"2024-02-21T23:09:54.278094+0000","last_active":"2024-02-21T23:10:07.376102+0000","last_peered":"2024-02-21T23:10:07.376102+0000","last_clean":"2024-02-21T23:10:07.376102+0000","last_became_active":"2024-02-21T23:09:54.276160+0000","last_became_peered":"2024-02-21T23:09:54.276160+0000","last_unstale":"2024-02-21T23:10:07.376102+0000","last_undegraded":"2024-02-21T23:10:07.376102+0000","last_fullsized":"2024-02-21T23:10:07.376102+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:15:07.339478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.375244+0000","last_change":"2024-02-21T23:09:53.280005+0000","last_active":"2024-02-21T23:10:07.375244+0000","last_peered":"2024-02-21T23:10:07.375244+0000","last_clean":"2024-02-21T23:10:07.375244+0000","last_became_active":"2024-02-21T23:09:53.279689+0000","last_became_peered":"2024-02-21T23:09:53.279689+0000","last_unstale":"2024-02-21T23:10:07.375244+0000","last_undegraded":"2024-02-21T23:10:07.375244+0000","last_fullsized":"2024-02-21T23:10:07.375244+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:40:05.561558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.15","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234855+0000","last_change":"2024-02-21T23:10:06.384066+0000","last_active":"2024-02-21T23:10:08.234855+0000","last_peered":"2024-02-21T23:10:08.234855+0000","last_clean":"2024-02-21T23:10:08.234855+0000","last_became_active":"2024-02-21T23:10:06.383780+0000","last_became_peered":"2024-02-21T23:10:06.383780+0000","last_unstale":"2024-02-21T23:10:08.234855+0000","last_undegraded":"2024-02-21T23:10:08.234855+0000","last_fullsized":"2024-02-21T23:10:08.234855+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:24:37.460238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236590+0000","last_change":"2024-02-21T23:10:05.393045+0000","last_active":"2024-02-21T23:10:08.236590+0000","last_peered":"2024-02-21T23:10:08.236590+0000","last_clean":"2024-02-21T23:10:08.236590+0000","last_became_active":"2024-02-21T23:10:05.392683+0000","last_became_peered":"2024-02-21T23:10:05.392683+0000","last_unstale":"2024-02-21T23:10:08.236590+0000","last_undegraded":"2024-02-21T23:10:08.236590+0000","last_fullsized":"2024-02-21T23:10:08.236590+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:45:39.918602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.394254+0000","last_change":"2024-02-21T23:09:54.262565+0000","last_active":"2024-02-21T23:10:05.394254+0000","last_peered":"2024-02-21T23:10:05.394254+0000","last_clean":"2024-02-21T23:10:05.394254+0000","last_became_active":"2024-02-21T23:09:54.261897+0000","last_became_peered":"2024-02-21T23:09:54.261897+0000","last_unstale":"2024-02-21T23:10:05.394254+0000","last_undegraded":"2024-02-21T23:10:05.394254+0000","last_fullsized":"2024-02-21T23:10:05.394254+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:08:50.519955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234939+0000","last_change":"2024-02-21T23:09:53.275695+0000","last_active":"2024-02-21T23:10:08.234939+0000","last_peered":"2024-02-21T23:10:08.234939+0000","last_clean":"2024-02-21T23:10:08.234939+0000","last_became_active":"2024-02-21T23:09:53.275343+0000","last_became_peered":"2024-02-21T23:09:53.275343+0000","last_unstale":"2024-02-21T23:10:08.234939+0000","last_undegraded":"2024-02-21T23:10:08.234939+0000","last_fullsized":"2024-02-21T23:10:08.234939+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:28:53.062636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1a","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.235787+0000","last_change":"2024-02-21T23:10:06.403716+0000","last_active":"2024-02-21T23:10:08.235787+0000","last_peered":"2024-02-21T23:10:08.235787+0000","last_clean":"2024-02-21T23:10:08.235787+0000","last_became_active":"2024-02-21T23:10:06.403226+0000","last_became_peered":"2024-02-21T23:10:06.403226+0000","last_unstale":"2024-02-21T23:10:08.235787+0000","last_undegraded":"2024-02-21T23:10:08.235787+0000","last_fullsized":"2024-02-21T23:10:08.235787+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:28:15.518153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.394797+0000","last_change":"2024-02-21T23:10:05.390846+0000","last_active":"2024-02-21T23:10:05.394797+0000","last_peered":"2024-02-21T23:10:05.394797+0000","last_clean":"2024-02-21T23:10:05.394797+0000","last_became_active":"2024-02-21T23:10:05.388088+0000","last_became_peered":"2024-02-21T23:10:05.388088+0000","last_unstale":"2024-02-21T23:10:05.394797+0000","last_undegraded":"2024-02-21T23:10:05.394797+0000","last_fullsized":"2024-02-21T23:10:05.394797+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:31:39.483569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.368060+0000","last_change":"2024-02-21T23:09:54.279220+0000","last_active":"2024-02-21T23:10:05.368060+0000","last_peered":"2024-02-21T23:10:05.368060+0000","last_clean":"2024-02-21T23:10:05.368060+0000","last_became_active":"2024-02-21T23:09:54.278653+0000","last_became_peered":"2024-02-21T23:09:54.278653+0000","last_unstale":"2024-02-21T23:10:05.368060+0000","last_undegraded":"2024-02-21T23:10:05.368060+0000","last_fullsized":"2024-02-21T23:10:05.368060+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:13:27.311923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236554+0000","last_change":"2024-02-21T23:09:53.288439+0000","last_active":"2024-02-21T23:10:08.236554+0000","last_peered":"2024-02-21T23:10:08.236554+0000","last_clean":"2024-02-21T23:10:08.236554+0000","last_became_active":"2024-02-21T23:09:53.287601+0000","last_became_peered":"2024-02-21T23:09:53.287601+0000","last_unstale":"2024-02-21T23:10:08.236554+0000","last_undegraded":"2024-02-21T23:10:08.236554+0000","last_fullsized":"2024-02-21T23:10:08.236554+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:32:05.045159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.376263+0000","last_change":"2024-02-21T23:10:05.376263+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.376263+0000","last_undegraded":"2024-02-21T23:10:05.376263+0000","last_fullsized":"2024-02-21T23:10:05.376263+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.1b","version":"154'1","reported_seq":15,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.233920+0000","last_change":"2024-02-21T23:10:05.387544+0000","last_active":"2024-02-21T23:10:08.233920+0000","last_peered":"2024-02-21T23:10:08.233920+0000","last_clean":"2024-02-21T23:10:08.233920+0000","last_became_active":"2024-02-21T23:10:05.385992+0000","last_became_peered":"2024-02-21T23:10:05.385992+0000","last_unstale":"2024-02-21T23:10:08.233920+0000","last_undegraded":"2024-02-21T23:10:08.233920+0000","last_fullsized":"2024-02-21T23:10:08.233920+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:28:14.502463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232148+0000","last_change":"2024-02-21T23:09:54.280274+0000","last_active":"2024-02-21T23:10:08.232148+0000","last_peered":"2024-02-21T23:10:08.232148+0000","last_clean":"2024-02-21T23:10:08.232148+0000","last_became_active":"2024-02-21T23:09:54.279951+0000","last_became_peered":"2024-02-21T23:09:54.279951+0000","last_unstale":"2024-02-21T23:10:08.232148+0000","last_undegraded":"2024-02-21T23:10:08.232148+0000","last_fullsized":"2024-02-21T23:10:08.232148+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:15:08.084683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.15","version":"150'3","reported_seq":21,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.393092+0000","last_change":"2024-02-21T23:09:53.279389+0000","last_active":"2024-02-21T23:10:05.393092+0000","last_peered":"2024-02-21T23:10:05.393092+0000","last_clean":"2024-02-21T23:10:05.393092+0000","last_became_active":"2024-02-21T23:09:53.279042+0000","last_became_peered":"2024-02-21T23:09:53.279042+0000","last_unstale":"2024-02-21T23:10:05.393092+0000","last_undegraded":"2024-02-21T23:10:05.393092+0000","last_fullsized":"2024-02-21T23:10:05.393092+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:44:44.898339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.233266+0000","last_change":"2024-02-21T23:10:06.379758+0000","last_active":"2024-02-21T23:10:08.233266+0000","last_peered":"2024-02-21T23:10:08.233266+0000","last_clean":"2024-02-21T23:10:08.233266+0000","last_became_active":"2024-02-21T23:10:06.379416+0000","last_became_peered":"2024-02-21T23:10:06.379416+0000","last_unstale":"2024-02-21T23:10:08.233266+0000","last_undegraded":"2024-02-21T23:10:08.233266+0000","last_fullsized":"2024-02-21T23:10:08.233266+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:06:55.637480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1a","version":"154'1","reported_seq":15,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.231363+0000","last_change":"2024-02-21T23:10:05.386289+0000","last_active":"2024-02-21T23:10:08.231363+0000","last_peered":"2024-02-21T23:10:08.231363+0000","last_clean":"2024-02-21T23:10:08.231363+0000","last_became_active":"2024-02-21T23:10:05.384200+0000","last_became_peered":"2024-02-21T23:10:05.384200+0000","last_unstale":"2024-02-21T23:10:08.231363+0000","last_undegraded":"2024-02-21T23:10:08.231363+0000","last_fullsized":"2024-02-21T23:10:08.231363+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:24:31.837524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1b","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.375019+0000","last_change":"2024-02-21T23:09:54.285563+0000","last_active":"2024-02-21T23:10:07.375019+0000","last_peered":"2024-02-21T23:10:07.375019+0000","last_clean":"2024-02-21T23:10:07.375019+0000","last_became_active":"2024-02-21T23:09:54.285275+0000","last_became_peered":"2024-02-21T23:09:54.285275+0000","last_unstale":"2024-02-21T23:10:07.375019+0000","last_undegraded":"2024-02-21T23:10:07.375019+0000","last_fullsized":"2024-02-21T23:10:07.375019+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:21:42.014406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"150'6","reported_seq":28,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236789+0000","last_change":"2024-02-21T23:09:53.288423+0000","last_active":"2024-02-21T23:10:08.236789+0000","last_peered":"2024-02-21T23:10:08.236789+0000","last_clean":"2024-02-21T23:10:08.236789+0000","last_became_active":"2024-02-21T23:09:53.287473+0000","last_became_peered":"2024-02-21T23:09:53.287473+0000","last_unstale":"2024-02-21T23:10:08.236789+0000","last_undegraded":"2024-02-21T23:10:08.236789+0000","last_fullsized":"2024-02-21T23:10:08.236789+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:13:32.425310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1e","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.373032+0000","last_change":"2024-02-21T23:10:05.373032+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.373032+0000","last_undegraded":"2024-02-21T23:10:05.373032+0000","last_fullsized":"2024-02-21T23:10:05.373032+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1d","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.391672+0000","last_change":"2024-02-21T23:10:05.388219+0000","last_active":"2024-02-21T23:10:05.391672+0000","last_peered":"2024-02-21T23:10:05.391672+0000","last_clean":"2024-02-21T23:10:05.391672+0000","last_became_active":"2024-02-21T23:10:05.387863+0000","last_became_peered":"2024-02-21T23:10:05.387863+0000","last_unstale":"2024-02-21T23:10:05.391672+0000","last_undegraded":"2024-02-21T23:10:05.391672+0000","last_fullsized":"2024-02-21T23:10:05.391672+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:34:10.873891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1c","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.390781+0000","last_change":"2024-02-21T23:09:54.278618+0000","last_active":"2024-02-21T23:10:05.390781+0000","last_peered":"2024-02-21T23:10:05.390781+0000","last_clean":"2024-02-21T23:10:05.390781+0000","last_became_active":"2024-02-21T23:09:54.278343+0000","last_became_peered":"2024-02-21T23:09:54.278343+0000","last_unstale":"2024-02-21T23:10:05.390781+0000","last_undegraded":"2024-02-21T23:10:05.390781+0000","last_fullsized":"2024-02-21T23:10:05.390781+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:49:31.759374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.13","version":"150'2","reported_seq":24,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234083+0000","last_change":"2024-02-21T23:09:53.284172+0000","last_active":"2024-02-21T23:10:08.234083+0000","last_peered":"2024-02-21T23:10:08.234083+0000","last_clean":"2024-02-21T23:10:08.234083+0000","last_became_active":"2024-02-21T23:09:53.283507+0000","last_became_peered":"2024-02-21T23:09:53.283507+0000","last_unstale":"2024-02-21T23:10:08.234083+0000","last_undegraded":"2024-02-21T23:10:08.234083+0000","last_fullsized":"2024-02-21T23:10:08.234083+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:20:30.108767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1f","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.382382+0000","last_change":"2024-02-21T23:10:05.382382+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.382382+0000","last_undegraded":"2024-02-21T23:10:05.382382+0000","last_fullsized":"2024-02-21T23:10:05.382382+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1c","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.394701+0000","last_change":"2024-02-21T23:10:05.387736+0000","last_active":"2024-02-21T23:10:05.394701+0000","last_peered":"2024-02-21T23:10:05.394701+0000","last_clean":"2024-02-21T23:10:05.394701+0000","last_became_active":"2024-02-21T23:10:05.387416+0000","last_became_peered":"2024-02-21T23:10:05.387416+0000","last_unstale":"2024-02-21T23:10:05.394701+0000","last_undegraded":"2024-02-21T23:10:05.394701+0000","last_fullsized":"2024-02-21T23:10:05.394701+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:35:01.442407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1d","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236368+0000","last_change":"2024-02-21T23:09:54.291717+0000","last_active":"2024-02-21T23:10:08.236368+0000","last_peered":"2024-02-21T23:10:08.236368+0000","last_clean":"2024-02-21T23:10:08.236368+0000","last_became_active":"2024-02-21T23:09:54.290842+0000","last_became_peered":"2024-02-21T23:09:54.290842+0000","last_unstale":"2024-02-21T23:10:08.236368+0000","last_undegraded":"2024-02-21T23:10:08.236368+0000","last_fullsized":"2024-02-21T23:10:08.236368+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:42:16.988326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232712+0000","last_change":"2024-02-21T23:09:53.284733+0000","last_active":"2024-02-21T23:10:08.232712+0000","last_peered":"2024-02-21T23:10:08.232712+0000","last_clean":"2024-02-21T23:10:08.232712+0000","last_became_active":"2024-02-21T23:09:53.284399+0000","last_became_peered":"2024-02-21T23:09:53.284399+0000","last_unstale":"2024-02-21T23:10:08.232712+0000","last_undegraded":"2024-02-21T23:10:08.232712+0000","last_fullsized":"2024-02-21T23:10:08.232712+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:05:08.172503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1c","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.365861+0000","last_change":"2024-02-21T23:10:05.365861+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.365861+0000","last_undegraded":"2024-02-21T23:10:05.365861+0000","last_fullsized":"2024-02-21T23:10:05.365861+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.1f","version":"154'4","reported_seq":18,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232862+0000","last_change":"2024-02-21T23:10:05.381058+0000","last_active":"2024-02-21T23:10:08.232862+0000","last_peered":"2024-02-21T23:10:08.232862+0000","last_clean":"2024-02-21T23:10:08.232862+0000","last_became_active":"2024-02-21T23:10:05.380737+0000","last_became_peered":"2024-02-21T23:10:05.380737+0000","last_unstale":"2024-02-21T23:10:08.232862+0000","last_undegraded":"2024-02-21T23:10:08.232862+0000","last_fullsized":"2024-02-21T23:10:08.232862+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:09:16.338374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1e","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.381340+0000","last_change":"2024-02-21T23:09:54.277339+0000","last_active":"2024-02-21T23:10:05.381340+0000","last_peered":"2024-02-21T23:10:05.381340+0000","last_clean":"2024-02-21T23:10:05.381340+0000","last_became_active":"2024-02-21T23:09:54.277039+0000","last_became_peered":"2024-02-21T23:09:54.277039+0000","last_unstale":"2024-02-21T23:10:05.381340+0000","last_undegraded":"2024-02-21T23:10:05.381340+0000","last_fullsized":"2024-02-21T23:10:05.381340+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:42:16.988326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":18,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.378558+0000","last_change":"2024-02-21T23:09:53.283915+0000","last_active":"2024-02-21T23:10:05.378558+0000","last_peered":"2024-02-21T23:10:05.378558+0000","last_clean":"2024-02-21T23:10:05.378558+0000","last_became_active":"2024-02-21T23:09:53.283532+0000","last_became_peered":"2024-02-21T23:09:53.283532+0000","last_unstale":"2024-02-21T23:10:05.378558+0000","last_undegraded":"2024-02-21T23:10:05.378558+0000","last_fullsized":"2024-02-21T23:10:05.378558+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:55:17.574145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1d","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.375317+0000","last_change":"2024-02-21T23:10:06.373957+0000","last_active":"2024-02-21T23:10:07.375317+0000","last_peered":"2024-02-21T23:10:07.375317+0000","last_clean":"2024-02-21T23:10:07.375317+0000","last_became_active":"2024-02-21T23:10:06.373657+0000","last_became_peered":"2024-02-21T23:10:06.373657+0000","last_unstale":"2024-02-21T23:10:07.375317+0000","last_undegraded":"2024-02-21T23:10:07.375317+0000","last_fullsized":"2024-02-21T23:10:07.375317+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:37:55.080822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1e","version":"154'1","reported_seq":15,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.235331+0000","last_change":"2024-02-21T23:10:05.388527+0000","last_active":"2024-02-21T23:10:08.235331+0000","last_peered":"2024-02-21T23:10:08.235331+0000","last_clean":"2024-02-21T23:10:08.235331+0000","last_became_active":"2024-02-21T23:10:05.388214+0000","last_became_peered":"2024-02-21T23:10:05.388214+0000","last_unstale":"2024-02-21T23:10:08.235331+0000","last_undegraded":"2024-02-21T23:10:08.235331+0000","last_fullsized":"2024-02-21T23:10:08.235331+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:50:32.666622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1f","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.376181+0000","last_change":"2024-02-21T23:09:54.265384+0000","last_active":"2024-02-21T23:10:07.376181+0000","last_peered":"2024-02-21T23:10:07.376181+0000","last_clean":"2024-02-21T23:10:07.376181+0000","last_became_active":"2024-02-21T23:09:54.265073+0000","last_became_peered":"2024-02-21T23:09:54.265073+0000","last_unstale":"2024-02-21T23:10:07.376181+0000","last_undegraded":"2024-02-21T23:10:07.376181+0000","last_fullsized":"2024-02-21T23:10:07.376181+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:44:28.295055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.10","version":"150'8","reported_seq":30,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.235156+0000","last_change":"2024-02-21T23:09:53.285246+0000","last_active":"2024-02-21T23:10:08.235156+0000","last_peered":"2024-02-21T23:10:08.235156+0000","last_clean":"2024-02-21T23:10:08.235156+0000","last_became_active":"2024-02-21T23:09:53.282084+0000","last_became_peered":"2024-02-21T23:09:53.282084+0000","last_unstale":"2024-02-21T23:10:08.235156+0000","last_undegraded":"2024-02-21T23:10:08.235156+0000","last_fullsized":"2024-02-21T23:10:08.235156+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:23:26.494398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.235925+0000","last_change":"2024-02-21T23:10:06.390439+0000","last_active":"2024-02-21T23:10:08.235925+0000","last_peered":"2024-02-21T23:10:08.235925+0000","last_clean":"2024-02-21T23:10:08.235925+0000","last_became_active":"2024-02-21T23:10:06.390012+0000","last_became_peered":"2024-02-21T23:10:06.390012+0000","last_unstale":"2024-02-21T23:10:08.235925+0000","last_undegraded":"2024-02-21T23:10:08.235925+0000","last_fullsized":"2024-02-21T23:10:08.235925+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:17:37.624314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.374798+0000","last_change":"2024-02-21T23:10:05.393586+0000","last_active":"2024-02-21T23:10:07.374798+0000","last_peered":"2024-02-21T23:10:07.374798+0000","last_clean":"2024-02-21T23:10:07.374798+0000","last_became_active":"2024-02-21T23:10:05.392606+0000","last_became_peered":"2024-02-21T23:10:05.392606+0000","last_unstale":"2024-02-21T23:10:07.374798+0000","last_undegraded":"2024-02-21T23:10:07.374798+0000","last_fullsized":"2024-02-21T23:10:07.374798+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:12:07.072214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234736+0000","last_change":"2024-02-21T23:09:54.289615+0000","last_active":"2024-02-21T23:10:08.234736+0000","last_peered":"2024-02-21T23:10:08.234736+0000","last_clean":"2024-02-21T23:10:08.234736+0000","last_became_active":"2024-02-21T23:09:54.289322+0000","last_became_peered":"2024-02-21T23:09:54.289322+0000","last_unstale":"2024-02-21T23:10:08.234736+0000","last_undegraded":"2024-02-21T23:10:08.234736+0000","last_fullsized":"2024-02-21T23:10:08.234736+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:34:28.642087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.f","version":"150'1","reported_seq":23,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236888+0000","last_change":"2024-02-21T23:09:53.287210+0000","last_active":"2024-02-21T23:10:08.236888+0000","last_peered":"2024-02-21T23:10:08.236888+0000","last_clean":"2024-02-21T23:10:08.236888+0000","last_became_active":"2024-02-21T23:09:53.285372+0000","last_became_peered":"2024-02-21T23:09:53.285372+0000","last_unstale":"2024-02-21T23:10:08.236888+0000","last_undegraded":"2024-02-21T23:10:08.236888+0000","last_fullsized":"2024-02-21T23:10:08.236888+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:20:30.108767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.363690+0000","last_change":"2024-02-21T23:10:05.363690+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.363690+0000","last_undegraded":"2024-02-21T23:10:05.363690+0000","last_fullsized":"2024-02-21T23:10:05.363690+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.395633+0000","last_change":"2024-02-21T23:10:05.395057+0000","last_active":"2024-02-21T23:10:05.395633+0000","last_peered":"2024-02-21T23:10:05.395633+0000","last_clean":"2024-02-21T23:10:05.395633+0000","last_became_active":"2024-02-21T23:10:05.394783+0000","last_became_peered":"2024-02-21T23:10:05.394783+0000","last_unstale":"2024-02-21T23:10:05.395633+0000","last_undegraded":"2024-02-21T23:10:05.395633+0000","last_fullsized":"2024-02-21T23:10:05.395633+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:55:29.679403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.390638+0000","last_change":"2024-02-21T23:09:54.274363+0000","last_active":"2024-02-21T23:10:05.390638+0000","last_peered":"2024-02-21T23:10:05.390638+0000","last_clean":"2024-02-21T23:10:05.390638+0000","last_became_active":"2024-02-21T23:09:54.274084+0000","last_became_peered":"2024-02-21T23:09:54.274084+0000","last_unstale":"2024-02-21T23:10:05.390638+0000","last_undegraded":"2024-02-21T23:10:05.390638+0000","last_fullsized":"2024-02-21T23:10:05.390638+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:42:19.996155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.e","version":"150'2","reported_seq":24,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.237049+0000","last_change":"2024-02-21T23:09:53.275941+0000","last_active":"2024-02-21T23:10:08.237049+0000","last_peered":"2024-02-21T23:10:08.237049+0000","last_clean":"2024-02-21T23:10:08.237049+0000","last_became_active":"2024-02-21T23:09:53.275555+0000","last_became_peered":"2024-02-21T23:09:53.275555+0000","last_unstale":"2024-02-21T23:10:08.237049+0000","last_undegraded":"2024-02-21T23:10:08.237049+0000","last_fullsized":"2024-02-21T23:10:08.237049+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:02:10.230470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.386121+0000","last_change":"2024-02-21T23:10:05.386121+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.386121+0000","last_undegraded":"2024-02-21T23:10:05.386121+0000","last_fullsized":"2024-02-21T23:10:05.386121+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.396519+0000","last_change":"2024-02-21T23:10:05.395810+0000","last_active":"2024-02-21T23:10:05.396519+0000","last_peered":"2024-02-21T23:10:05.396519+0000","last_clean":"2024-02-21T23:10:05.396519+0000","last_became_active":"2024-02-21T23:10:05.395273+0000","last_became_peered":"2024-02-21T23:10:05.395273+0000","last_unstale":"2024-02-21T23:10:05.396519+0000","last_undegraded":"2024-02-21T23:10:05.396519+0000","last_fullsized":"2024-02-21T23:10:05.396519+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:34:10.873891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.378450+0000","last_change":"2024-02-21T23:09:54.259630+0000","last_active":"2024-02-21T23:10:05.378450+0000","last_peered":"2024-02-21T23:10:05.378450+0000","last_clean":"2024-02-21T23:10:05.378450+0000","last_became_active":"2024-02-21T23:09:54.258924+0000","last_became_peered":"2024-02-21T23:09:54.258924+0000","last_unstale":"2024-02-21T23:10:05.378450+0000","last_undegraded":"2024-02-21T23:10:05.378450+0000","last_fullsized":"2024-02-21T23:10:05.378450+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:34:28.642087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.d","version":"150'1","reported_seq":23,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234011+0000","last_change":"2024-02-21T23:09:53.279385+0000","last_active":"2024-02-21T23:10:08.234011+0000","last_peered":"2024-02-21T23:10:08.234011+0000","last_clean":"2024-02-21T23:10:08.234011+0000","last_became_active":"2024-02-21T23:09:53.278074+0000","last_became_peered":"2024-02-21T23:09:53.278074+0000","last_unstale":"2024-02-21T23:10:08.234011+0000","last_undegraded":"2024-02-21T23:10:08.234011+0000","last_fullsized":"2024-02-21T23:10:08.234011+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:42:15.971409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.380894+0000","last_change":"2024-02-21T23:10:05.380894+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.380894+0000","last_undegraded":"2024-02-21T23:10:05.380894+0000","last_fullsized":"2024-02-21T23:10:05.380894+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.387108+0000","last_change":"2024-02-21T23:10:05.385621+0000","last_active":"2024-02-21T23:10:05.387108+0000","last_peered":"2024-02-21T23:10:05.387108+0000","last_clean":"2024-02-21T23:10:05.387108+0000","last_became_active":"2024-02-21T23:10:05.384654+0000","last_became_peered":"2024-02-21T23:10:05.384654+0000","last_unstale":"2024-02-21T23:10:05.387108+0000","last_undegraded":"2024-02-21T23:10:05.387108+0000","last_fullsized":"2024-02-21T23:10:05.387108+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:11:55.677946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.380169+0000","last_change":"2024-02-21T23:09:54.277296+0000","last_active":"2024-02-21T23:10:05.380169+0000","last_peered":"2024-02-21T23:10:05.380169+0000","last_clean":"2024-02-21T23:10:05.380169+0000","last_became_active":"2024-02-21T23:09:54.276973+0000","last_became_peered":"2024-02-21T23:09:54.276973+0000","last_unstale":"2024-02-21T23:10:05.380169+0000","last_undegraded":"2024-02-21T23:10:05.380169+0000","last_fullsized":"2024-02-21T23:10:05.380169+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:07:24.304915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.235861+0000","last_change":"2024-02-21T23:09:53.281804+0000","last_active":"2024-02-21T23:10:08.235861+0000","last_peered":"2024-02-21T23:10:08.235861+0000","last_clean":"2024-02-21T23:10:08.235861+0000","last_became_active":"2024-02-21T23:09:53.278643+0000","last_became_peered":"2024-02-21T23:09:53.278643+0000","last_unstale":"2024-02-21T23:10:08.235861+0000","last_undegraded":"2024-02-21T23:10:08.235861+0000","last_fullsized":"2024-02-21T23:10:08.235861+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:55:17.574145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.6","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.363355+0000","last_change":"2024-02-21T23:10:05.363355+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.363355+0000","last_undegraded":"2024-02-21T23:10:05.363355+0000","last_fullsized":"2024-02-21T23:10:05.363355+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234955+0000","last_change":"2024-02-21T23:10:05.389031+0000","last_active":"2024-02-21T23:10:08.234955+0000","last_peered":"2024-02-21T23:10:08.234955+0000","last_clean":"2024-02-21T23:10:08.234955+0000","last_became_active":"2024-02-21T23:10:05.388789+0000","last_became_peered":"2024-02-21T23:10:05.388789+0000","last_unstale":"2024-02-21T23:10:08.234955+0000","last_undegraded":"2024-02-21T23:10:08.234955+0000","last_fullsized":"2024-02-21T23:10:08.234955+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:46:44.667756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236537+0000","last_change":"2024-02-21T23:09:54.285141+0000","last_active":"2024-02-21T23:10:08.236537+0000","last_peered":"2024-02-21T23:10:08.236537+0000","last_clean":"2024-02-21T23:10:08.236537+0000","last_became_active":"2024-02-21T23:09:54.284498+0000","last_became_peered":"2024-02-21T23:09:54.284498+0000","last_unstale":"2024-02-21T23:10:08.236537+0000","last_undegraded":"2024-02-21T23:10:08.236537+0000","last_fullsized":"2024-02-21T23:10:08.236537+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:26:36.405020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.b","version":"150'2","reported_seq":24,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234201+0000","last_change":"2024-02-21T23:09:53.284284+0000","last_active":"2024-02-21T23:10:08.234201+0000","last_peered":"2024-02-21T23:10:08.234201+0000","last_clean":"2024-02-21T23:10:08.234201+0000","last_became_active":"2024-02-21T23:09:53.283043+0000","last_became_peered":"2024-02-21T23:09:53.283043+0000","last_unstale":"2024-02-21T23:10:08.234201+0000","last_undegraded":"2024-02-21T23:10:08.234201+0000","last_fullsized":"2024-02-21T23:10:08.234201+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:32:05.045159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.7","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.376118+0000","last_change":"2024-02-21T23:10:06.392968+0000","last_active":"2024-02-21T23:10:07.376118+0000","last_peered":"2024-02-21T23:10:07.376118+0000","last_clean":"2024-02-21T23:10:07.376118+0000","last_became_active":"2024-02-21T23:10:06.390031+0000","last_became_peered":"2024-02-21T23:10:06.390031+0000","last_unstale":"2024-02-21T23:10:07.376118+0000","last_undegraded":"2024-02-21T23:10:07.376118+0000","last_fullsized":"2024-02-21T23:10:07.376118+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:17:07.163396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236747+0000","last_change":"2024-02-21T23:10:05.392092+0000","last_active":"2024-02-21T23:10:08.236747+0000","last_peered":"2024-02-21T23:10:08.236747+0000","last_clean":"2024-02-21T23:10:08.236747+0000","last_became_active":"2024-02-21T23:10:05.391793+0000","last_became_peered":"2024-02-21T23:10:05.391793+0000","last_unstale":"2024-02-21T23:10:08.236747+0000","last_undegraded":"2024-02-21T23:10:08.236747+0000","last_fullsized":"2024-02-21T23:10:08.236747+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:09:16.338374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.233964+0000","last_change":"2024-02-21T23:09:54.275082+0000","last_active":"2024-02-21T23:10:08.233964+0000","last_peered":"2024-02-21T23:10:08.233964+0000","last_clean":"2024-02-21T23:10:08.233964+0000","last_became_active":"2024-02-21T23:09:54.274707+0000","last_became_peered":"2024-02-21T23:09:54.274707+0000","last_unstale":"2024-02-21T23:10:08.233964+0000","last_undegraded":"2024-02-21T23:10:08.233964+0000","last_fullsized":"2024-02-21T23:10:08.233964+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:39:52.080584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":18,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.378606+0000","last_change":"2024-02-21T23:09:53.277418+0000","last_active":"2024-02-21T23:10:05.378606+0000","last_peered":"2024-02-21T23:10:05.378606+0000","last_clean":"2024-02-21T23:10:05.378606+0000","last_became_active":"2024-02-21T23:09:53.277111+0000","last_became_peered":"2024-02-21T23:09:53.277111+0000","last_unstale":"2024-02-21T23:10:05.378606+0000","last_undegraded":"2024-02-21T23:10:05.378606+0000","last_fullsized":"2024-02-21T23:10:05.378606+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:23:26.494398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.375599+0000","last_change":"2024-02-21T23:10:06.391104+0000","last_active":"2024-02-21T23:10:07.375599+0000","last_peered":"2024-02-21T23:10:07.375599+0000","last_clean":"2024-02-21T23:10:07.375599+0000","last_became_active":"2024-02-21T23:10:06.390816+0000","last_became_peered":"2024-02-21T23:10:06.390816+0000","last_unstale":"2024-02-21T23:10:07.375599+0000","last_undegraded":"2024-02-21T23:10:07.375599+0000","last_fullsized":"2024-02-21T23:10:07.375599+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:49:43.863404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.7","version":"154'2","reported_seq":16,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234306+0000","last_change":"2024-02-21T23:10:05.378389+0000","last_active":"2024-02-21T23:10:08.234306+0000","last_peered":"2024-02-21T23:10:08.234306+0000","last_clean":"2024-02-21T23:10:08.234306+0000","last_became_active":"2024-02-21T23:10:05.378083+0000","last_became_peered":"2024-02-21T23:10:05.378083+0000","last_unstale":"2024-02-21T23:10:08.234306+0000","last_undegraded":"2024-02-21T23:10:08.234306+0000","last_fullsized":"2024-02-21T23:10:08.234306+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:28:14.502463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232579+0000","last_change":"2024-02-21T23:09:54.280400+0000","last_active":"2024-02-21T23:10:08.232579+0000","last_peered":"2024-02-21T23:10:08.232579+0000","last_clean":"2024-02-21T23:10:08.232579+0000","last_became_active":"2024-02-21T23:09:54.280102+0000","last_became_peered":"2024-02-21T23:09:54.280102+0000","last_unstale":"2024-02-21T23:10:08.232579+0000","last_undegraded":"2024-02-21T23:10:08.232579+0000","last_fullsized":"2024-02-21T23:10:08.232579+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:45:28.830261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.375349+0000","last_change":"2024-02-21T23:09:53.282939+0000","last_active":"2024-02-21T23:10:07.375349+0000","last_peered":"2024-02-21T23:10:07.375349+0000","last_clean":"2024-02-21T23:10:07.375349+0000","last_became_active":"2024-02-21T23:09:53.282612+0000","last_became_peered":"2024-02-21T23:09:53.282612+0000","last_unstale":"2024-02-21T23:10:07.375349+0000","last_undegraded":"2024-02-21T23:10:07.375349+0000","last_fullsized":"2024-02-21T23:10:07.375349+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:30:40.609835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236333+0000","last_change":"2024-02-21T23:10:06.396738+0000","last_active":"2024-02-21T23:10:08.236333+0000","last_peered":"2024-02-21T23:10:08.236333+0000","last_clean":"2024-02-21T23:10:08.236333+0000","last_became_active":"2024-02-21T23:10:06.396390+0000","last_became_peered":"2024-02-21T23:10:06.396390+0000","last_unstale":"2024-02-21T23:10:08.236333+0000","last_undegraded":"2024-02-21T23:10:08.236333+0000","last_fullsized":"2024-02-21T23:10:08.236333+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:24:32.853214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.6","version":"155'1","reported_seq":15,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.375969+0000","last_change":"2024-02-21T23:10:05.395962+0000","last_active":"2024-02-21T23:10:07.375969+0000","last_peered":"2024-02-21T23:10:07.375969+0000","last_clean":"2024-02-21T23:10:07.375969+0000","last_became_active":"2024-02-21T23:10:05.394863+0000","last_became_peered":"2024-02-21T23:10:05.394863+0000","last_unstale":"2024-02-21T23:10:07.375969+0000","last_undegraded":"2024-02-21T23:10:07.375969+0000","last_fullsized":"2024-02-21T23:10:07.375969+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:48:58.601649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.373315+0000","last_change":"2024-02-21T23:09:54.282302+0000","last_active":"2024-02-21T23:10:05.373315+0000","last_peered":"2024-02-21T23:10:05.373315+0000","last_clean":"2024-02-21T23:10:05.373315+0000","last_became_active":"2024-02-21T23:09:54.280468+0000","last_became_peered":"2024-02-21T23:09:54.280468+0000","last_unstale":"2024-02-21T23:10:05.373315+0000","last_undegraded":"2024-02-21T23:10:05.373315+0000","last_fullsized":"2024-02-21T23:10:05.373315+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:32:06.062076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":18,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.368703+0000","last_change":"2024-02-21T23:09:53.283837+0000","last_active":"2024-02-21T23:10:05.368703+0000","last_peered":"2024-02-21T23:10:05.368703+0000","last_clean":"2024-02-21T23:10:05.368703+0000","last_became_active":"2024-02-21T23:09:53.283541+0000","last_became_peered":"2024-02-21T23:09:53.283541+0000","last_unstale":"2024-02-21T23:10:05.368703+0000","last_undegraded":"2024-02-21T23:10:05.368703+0000","last_fullsized":"2024-02-21T23:10:05.368703+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:28:53.062636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.231400+0000","last_change":"2024-02-21T23:10:06.392422+0000","last_active":"2024-02-21T23:10:08.231400+0000","last_peered":"2024-02-21T23:10:08.231400+0000","last_clean":"2024-02-21T23:10:08.231400+0000","last_became_active":"2024-02-21T23:10:06.392121+0000","last_became_peered":"2024-02-21T23:10:06.392121+0000","last_unstale":"2024-02-21T23:10:08.231400+0000","last_undegraded":"2024-02-21T23:10:08.231400+0000","last_fullsized":"2024-02-21T23:10:08.231400+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:09:29.246717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.374605+0000","last_change":"2024-02-21T23:10:05.398114+0000","last_active":"2024-02-21T23:10:07.374605+0000","last_peered":"2024-02-21T23:10:07.374605+0000","last_clean":"2024-02-21T23:10:07.374605+0000","last_became_active":"2024-02-21T23:10:05.397811+0000","last_became_peered":"2024-02-21T23:10:05.397811+0000","last_unstale":"2024-02-21T23:10:07.374605+0000","last_undegraded":"2024-02-21T23:10:07.374605+0000","last_fullsized":"2024-02-21T23:10:07.374605+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:36:18.190950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.377721+0000","last_change":"2024-02-21T23:09:54.274229+0000","last_active":"2024-02-21T23:10:05.377721+0000","last_peered":"2024-02-21T23:10:05.377721+0000","last_clean":"2024-02-21T23:10:05.377721+0000","last_became_active":"2024-02-21T23:09:54.271365+0000","last_became_peered":"2024-02-21T23:09:54.271365+0000","last_unstale":"2024-02-21T23:10:05.377721+0000","last_undegraded":"2024-02-21T23:10:05.377721+0000","last_fullsized":"2024-02-21T23:10:05.377721+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:13:27.311923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.7","version":"150'2","reported_seq":24,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.233419+0000","last_change":"2024-02-21T23:09:53.283873+0000","last_active":"2024-02-21T23:10:08.233419+0000","last_peered":"2024-02-21T23:10:08.233419+0000","last_clean":"2024-02-21T23:10:08.233419+0000","last_became_active":"2024-02-21T23:09:53.282829+0000","last_became_peered":"2024-02-21T23:09:53.282829+0000","last_unstale":"2024-02-21T23:10:08.233419+0000","last_undegraded":"2024-02-21T23:10:08.233419+0000","last_fullsized":"2024-02-21T23:10:08.233419+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:13:32.425310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.b","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234395+0000","last_change":"2024-02-21T23:10:06.367171+0000","last_active":"2024-02-21T23:10:08.234395+0000","last_peered":"2024-02-21T23:10:08.234395+0000","last_clean":"2024-02-21T23:10:08.234395+0000","last_became_active":"2024-02-21T23:10:06.366866+0000","last_became_peered":"2024-02-21T23:10:06.366866+0000","last_unstale":"2024-02-21T23:10:08.234395+0000","last_undegraded":"2024-02-21T23:10:08.234395+0000","last_fullsized":"2024-02-21T23:10:08.234395+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:26:16.569464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.393848+0000","last_change":"2024-02-21T23:10:05.389876+0000","last_active":"2024-02-21T23:10:05.393848+0000","last_peered":"2024-02-21T23:10:05.393848+0000","last_clean":"2024-02-21T23:10:05.393848+0000","last_became_active":"2024-02-21T23:10:05.389568+0000","last_became_peered":"2024-02-21T23:10:05.389568+0000","last_unstale":"2024-02-21T23:10:05.393848+0000","last_undegraded":"2024-02-21T23:10:05.393848+0000","last_fullsized":"2024-02-21T23:10:05.393848+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:23:38.599655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234481+0000","last_change":"2024-02-21T23:09:54.277479+0000","last_active":"2024-02-21T23:10:08.234481+0000","last_peered":"2024-02-21T23:10:08.234481+0000","last_clean":"2024-02-21T23:10:08.234481+0000","last_became_active":"2024-02-21T23:09:54.277065+0000","last_became_peered":"2024-02-21T23:09:54.277065+0000","last_unstale":"2024-02-21T23:10:08.234481+0000","last_undegraded":"2024-02-21T23:10:08.234481+0000","last_fullsized":"2024-02-21T23:10:08.234481+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:07:24.304915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.6","version":"151'1","reported_seq":23,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.235714+0000","last_change":"2024-02-21T23:09:53.285874+0000","last_active":"2024-02-21T23:10:08.235714+0000","last_peered":"2024-02-21T23:10:08.235714+0000","last_clean":"2024-02-21T23:10:08.235714+0000","last_became_active":"2024-02-21T23:09:53.285569+0000","last_became_peered":"2024-02-21T23:09:53.285569+0000","last_unstale":"2024-02-21T23:10:08.235714+0000","last_undegraded":"2024-02-21T23:10:08.235714+0000","last_fullsized":"2024-02-21T23:10:08.235714+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:52:56.873078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.233184+0000","last_change":"2024-02-21T23:10:06.385851+0000","last_active":"2024-02-21T23:10:08.233184+0000","last_peered":"2024-02-21T23:10:08.233184+0000","last_clean":"2024-02-21T23:10:08.233184+0000","last_became_active":"2024-02-21T23:10:06.385482+0000","last_became_peered":"2024-02-21T23:10:06.385482+0000","last_unstale":"2024-02-21T23:10:08.233184+0000","last_undegraded":"2024-02-21T23:10:08.233184+0000","last_fullsized":"2024-02-21T23:10:08.233184+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:47:42.794742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.b","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.395230+0000","last_change":"2024-02-21T23:10:05.394114+0000","last_active":"2024-02-21T23:10:05.395230+0000","last_peered":"2024-02-21T23:10:05.395230+0000","last_clean":"2024-02-21T23:10:05.395230+0000","last_became_active":"2024-02-21T23:10:05.393152+0000","last_became_peered":"2024-02-21T23:10:05.393152+0000","last_unstale":"2024-02-21T23:10:05.395230+0000","last_undegraded":"2024-02-21T23:10:05.395230+0000","last_fullsized":"2024-02-21T23:10:05.395230+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:01:25.971350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.374932+0000","last_change":"2024-02-21T23:09:54.286102+0000","last_active":"2024-02-21T23:10:07.374932+0000","last_peered":"2024-02-21T23:10:07.374932+0000","last_clean":"2024-02-21T23:10:07.374932+0000","last_became_active":"2024-02-21T23:09:54.283073+0000","last_became_peered":"2024-02-21T23:09:54.283073+0000","last_unstale":"2024-02-21T23:10:07.374932+0000","last_undegraded":"2024-02-21T23:10:07.374932+0000","last_fullsized":"2024-02-21T23:10:07.374932+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:39:05.096862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.235327+0000","last_change":"2024-02-21T23:09:53.285730+0000","last_active":"2024-02-21T23:10:08.235327+0000","last_peered":"2024-02-21T23:10:08.235327+0000","last_clean":"2024-02-21T23:10:08.235327+0000","last_became_active":"2024-02-21T23:09:53.285373+0000","last_became_peered":"2024-02-21T23:09:53.285373+0000","last_unstale":"2024-02-21T23:10:08.235327+0000","last_undegraded":"2024-02-21T23:10:08.235327+0000","last_fullsized":"2024-02-21T23:10:08.235327+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:04:15.300172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.230302+0000","last_change":"2024-02-21T23:10:06.380586+0000","last_active":"2024-02-21T23:10:08.230302+0000","last_peered":"2024-02-21T23:10:08.230302+0000","last_clean":"2024-02-21T23:10:08.230302+0000","last_became_active":"2024-02-21T23:10:06.380293+0000","last_became_peered":"2024-02-21T23:10:06.380293+0000","last_unstale":"2024-02-21T23:10:08.230302+0000","last_undegraded":"2024-02-21T23:10:08.230302+0000","last_fullsized":"2024-02-21T23:10:08.230302+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:01:26.662036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.231095+0000","last_change":"2024-02-21T23:10:05.380512+0000","last_active":"2024-02-21T23:10:08.231095+0000","last_peered":"2024-02-21T23:10:08.231095+0000","last_clean":"2024-02-21T23:10:08.231095+0000","last_became_active":"2024-02-21T23:10:05.380207+0000","last_became_peered":"2024-02-21T23:10:05.380207+0000","last_unstale":"2024-02-21T23:10:08.231095+0000","last_undegraded":"2024-02-21T23:10:08.231095+0000","last_fullsized":"2024-02-21T23:10:08.231095+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:17:36.608624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.b","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.235603+0000","last_change":"2024-02-21T23:09:54.277483+0000","last_active":"2024-02-21T23:10:08.235603+0000","last_peered":"2024-02-21T23:10:08.235603+0000","last_clean":"2024-02-21T23:10:08.235603+0000","last_became_active":"2024-02-21T23:09:54.277170+0000","last_became_peered":"2024-02-21T23:09:54.277170+0000","last_unstale":"2024-02-21T23:10:08.235603+0000","last_undegraded":"2024-02-21T23:10:08.235603+0000","last_fullsized":"2024-02-21T23:10:08.235603+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:33:07.362294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.375793+0000","last_change":"2024-02-21T23:09:53.267455+0000","last_active":"2024-02-21T23:10:07.375793+0000","last_peered":"2024-02-21T23:10:07.375793+0000","last_clean":"2024-02-21T23:10:07.375793+0000","last_became_active":"2024-02-21T23:09:53.266974+0000","last_became_peered":"2024-02-21T23:09:53.266974+0000","last_unstale":"2024-02-21T23:10:07.375793+0000","last_undegraded":"2024-02-21T23:10:07.375793+0000","last_fullsized":"2024-02-21T23:10:07.375793+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:33:26.269092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.e","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.371905+0000","last_change":"2024-02-21T23:10:05.371905+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.371905+0000","last_undegraded":"2024-02-21T23:10:05.371905+0000","last_fullsized":"2024-02-21T23:10:05.371905+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.d","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.395287+0000","last_change":"2024-02-21T23:10:05.393632+0000","last_active":"2024-02-21T23:10:05.395287+0000","last_peered":"2024-02-21T23:10:05.395287+0000","last_clean":"2024-02-21T23:10:05.395287+0000","last_became_active":"2024-02-21T23:10:05.393070+0000","last_became_peered":"2024-02-21T23:10:05.393070+0000","last_unstale":"2024-02-21T23:10:05.395287+0000","last_undegraded":"2024-02-21T23:10:05.395287+0000","last_fullsized":"2024-02-21T23:10:05.395287+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:46:46.525155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.391035+0000","last_change":"2024-02-21T23:09:54.275273+0000","last_active":"2024-02-21T23:10:05.391035+0000","last_peered":"2024-02-21T23:10:05.391035+0000","last_clean":"2024-02-21T23:10:05.391035+0000","last_became_active":"2024-02-21T23:09:54.274287+0000","last_became_peered":"2024-02-21T23:09:54.274287+0000","last_unstale":"2024-02-21T23:10:05.391035+0000","last_undegraded":"2024-02-21T23:10:05.391035+0000","last_fullsized":"2024-02-21T23:10:05.391035+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:18:40.139538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":18,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.391026+0000","last_change":"2024-02-21T23:09:53.279782+0000","last_active":"2024-02-21T23:10:05.391026+0000","last_peered":"2024-02-21T23:10:05.391026+0000","last_clean":"2024-02-21T23:10:05.391026+0000","last_became_active":"2024-02-21T23:09:53.279477+0000","last_became_peered":"2024-02-21T23:09:53.279477+0000","last_unstale":"2024-02-21T23:10:05.391026+0000","last_undegraded":"2024-02-21T23:10:05.391026+0000","last_fullsized":"2024-02-21T23:10:05.391026+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:41:01.169723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.f","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.237523+0000","last_change":"2024-02-21T23:10:06.399721+0000","last_active":"2024-02-21T23:10:08.237523+0000","last_peered":"2024-02-21T23:10:08.237523+0000","last_clean":"2024-02-21T23:10:08.237523+0000","last_became_active":"2024-02-21T23:10:06.398869+0000","last_became_peered":"2024-02-21T23:10:06.398869+0000","last_unstale":"2024-02-21T23:10:08.237523+0000","last_undegraded":"2024-02-21T23:10:08.237523+0000","last_fullsized":"2024-02-21T23:10:08.237523+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:48:37.041227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.382546+0000","last_change":"2024-02-21T23:10:05.380230+0000","last_active":"2024-02-21T23:10:05.382546+0000","last_peered":"2024-02-21T23:10:05.382546+0000","last_clean":"2024-02-21T23:10:05.382546+0000","last_became_active":"2024-02-21T23:10:05.378612+0000","last_became_peered":"2024-02-21T23:10:05.378612+0000","last_unstale":"2024-02-21T23:10:05.382546+0000","last_undegraded":"2024-02-21T23:10:05.382546+0000","last_fullsized":"2024-02-21T23:10:05.382546+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:51:20.648392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.d","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.373520+0000","last_change":"2024-02-21T23:09:54.279409+0000","last_active":"2024-02-21T23:10:05.373520+0000","last_peered":"2024-02-21T23:10:05.373520+0000","last_clean":"2024-02-21T23:10:05.373520+0000","last_became_active":"2024-02-21T23:09:54.279107+0000","last_became_peered":"2024-02-21T23:09:54.279107+0000","last_unstale":"2024-02-21T23:10:05.373520+0000","last_undegraded":"2024-02-21T23:10:05.373520+0000","last_fullsized":"2024-02-21T23:10:05.373520+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:20:31.125684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232898+0000","last_change":"2024-02-21T23:09:53.278959+0000","last_active":"2024-02-21T23:10:08.232898+0000","last_peered":"2024-02-21T23:10:08.232898+0000","last_clean":"2024-02-21T23:10:08.232898+0000","last_became_active":"2024-02-21T23:09:53.278673+0000","last_became_peered":"2024-02-21T23:09:53.278673+0000","last_unstale":"2024-02-21T23:10:08.232898+0000","last_undegraded":"2024-02-21T23:10:08.232898+0000","last_fullsized":"2024-02-21T23:10:08.232898+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:42:15.971409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.375427+0000","last_change":"2024-02-21T23:10:06.387361+0000","last_active":"2024-02-21T23:10:07.375427+0000","last_peered":"2024-02-21T23:10:07.375427+0000","last_clean":"2024-02-21T23:10:07.375427+0000","last_became_active":"2024-02-21T23:10:06.387063+0000","last_became_peered":"2024-02-21T23:10:06.387063+0000","last_unstale":"2024-02-21T23:10:07.375427+0000","last_undegraded":"2024-02-21T23:10:07.375427+0000","last_fullsized":"2024-02-21T23:10:07.375427+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:47:41.453699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"0'0","reported_seq":9,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.385563+0000","last_change":"2024-02-21T23:10:05.381912+0000","last_active":"2024-02-21T23:10:05.385563+0000","last_peered":"2024-02-21T23:10:05.385563+0000","last_clean":"2024-02-21T23:10:05.385563+0000","last_became_active":"2024-02-21T23:10:05.381618+0000","last_became_peered":"2024-02-21T23:10:05.381618+0000","last_unstale":"2024-02-21T23:10:05.385563+0000","last_undegraded":"2024-02-21T23:10:05.385563+0000","last_fullsized":"2024-02-21T23:10:05.385563+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:53:08.978335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.e","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.235442+0000","last_change":"2024-02-21T23:09:54.282920+0000","last_active":"2024-02-21T23:10:08.235442+0000","last_peered":"2024-02-21T23:10:08.235442+0000","last_clean":"2024-02-21T23:10:08.235442+0000","last_became_active":"2024-02-21T23:09:54.282544+0000","last_became_peered":"2024-02-21T23:09:54.282544+0000","last_unstale":"2024-02-21T23:10:08.235442+0000","last_undegraded":"2024-02-21T23:10:08.235442+0000","last_fullsized":"2024-02-21T23:10:08.235442+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:40:05.206906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":18,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.378866+0000","last_change":"2024-02-21T23:09:53.275586+0000","last_active":"2024-02-21T23:10:05.378866+0000","last_peered":"2024-02-21T23:10:05.378866+0000","last_clean":"2024-02-21T23:10:05.378866+0000","last_became_active":"2024-02-21T23:09:53.274828+0000","last_became_peered":"2024-02-21T23:09:53.274828+0000","last_unstale":"2024-02-21T23:10:05.378866+0000","last_undegraded":"2024-02-21T23:10:05.378866+0000","last_fullsized":"2024-02-21T23:10:05.378866+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:02:10.230470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.d","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234506+0000","last_change":"2024-02-21T23:10:06.377243+0000","last_active":"2024-02-21T23:10:08.234506+0000","last_peered":"2024-02-21T23:10:08.234506+0000","last_clean":"2024-02-21T23:10:08.234506+0000","last_became_active":"2024-02-21T23:10:06.376521+0000","last_became_peered":"2024-02-21T23:10:06.376521+0000","last_unstale":"2024-02-21T23:10:08.234506+0000","last_undegraded":"2024-02-21T23:10:08.234506+0000","last_fullsized":"2024-02-21T23:10:08.234506+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:02:54.477082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.e","version":"154'2","reported_seq":16,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232515+0000","last_change":"2024-02-21T23:10:05.388215+0000","last_active":"2024-02-21T23:10:08.232515+0000","last_peered":"2024-02-21T23:10:08.232515+0000","last_clean":"2024-02-21T23:10:08.232515+0000","last_became_active":"2024-02-21T23:10:05.387907+0000","last_became_peered":"2024-02-21T23:10:05.387907+0000","last_unstale":"2024-02-21T23:10:08.232515+0000","last_undegraded":"2024-02-21T23:10:08.232515+0000","last_fullsized":"2024-02-21T23:10:08.232515+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:05:36.127991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.f","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.376215+0000","last_change":"2024-02-21T23:09:54.277320+0000","last_active":"2024-02-21T23:10:07.376215+0000","last_peered":"2024-02-21T23:10:07.376215+0000","last_clean":"2024-02-21T23:10:07.376215+0000","last_became_active":"2024-02-21T23:09:54.276368+0000","last_became_peered":"2024-02-21T23:09:54.276368+0000","last_unstale":"2024-02-21T23:10:07.376215+0000","last_undegraded":"2024-02-21T23:10:07.376215+0000","last_fullsized":"2024-02-21T23:10:07.376215+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:40:37.285836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.230247+0000","last_change":"2024-02-21T23:09:53.278027+0000","last_active":"2024-02-21T23:10:08.230247+0000","last_peered":"2024-02-21T23:10:08.230247+0000","last_clean":"2024-02-21T23:10:08.230247+0000","last_became_active":"2024-02-21T23:09:53.277660+0000","last_became_peered":"2024-02-21T23:09:53.277660+0000","last_unstale":"2024-02-21T23:10:08.230247+0000","last_undegraded":"2024-02-21T23:10:08.230247+0000","last_fullsized":"2024-02-21T23:10:08.230247+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:07:23.287998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1b","version":"0'0","reported_seq":1,"reported_epoch":153,"state":"creating+peering","last_fresh":"2024-02-21T23:10:05.385924+0000","last_change":"2024-02-21T23:10:05.385924+0000","last_active":"2024-02-21T23:10:05.335854+0000","last_peered":"2024-02-21T23:10:05.335854+0000","last_clean":"2024-02-21T23:10:05.335854+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-21T23:10:05.385924+0000","last_undegraded":"2024-02-21T23:10:05.385924+0000","last_fullsized":"2024-02-21T23:10:05.385924+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.230256+0000","last_change":"2024-02-21T23:10:05.390395+0000","last_active":"2024-02-21T23:10:08.230256+0000","last_peered":"2024-02-21T23:10:08.230256+0000","last_clean":"2024-02-21T23:10:08.230256+0000","last_became_active":"2024-02-21T23:10:05.390115+0000","last_became_peered":"2024-02-21T23:10:05.390115+0000","last_unstale":"2024-02-21T23:10:08.230256+0000","last_undegraded":"2024-02-21T23:10:08.230256+0000","last_fullsized":"2024-02-21T23:10:08.230256+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:10:20.170395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":16,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.369300+0000","last_change":"2024-02-21T23:09:54.277356+0000","last_active":"2024-02-21T23:10:05.369300+0000","last_peered":"2024-02-21T23:10:05.369300+0000","last_clean":"2024-02-21T23:10:05.369300+0000","last_became_active":"2024-02-21T23:09:54.277087+0000","last_became_peered":"2024-02-21T23:09:54.277087+0000","last_unstale":"2024-02-21T23:10:05.369300+0000","last_undegraded":"2024-02-21T23:10:05.369300+0000","last_fullsized":"2024-02-21T23:10:05.369300+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:03:44.493909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.235691+0000","last_change":"2024-02-21T23:09:53.286181+0000","last_active":"2024-02-21T23:10:08.235691+0000","last_peered":"2024-02-21T23:10:08.235691+0000","last_clean":"2024-02-21T23:10:08.235691+0000","last_became_active":"2024-02-21T23:09:53.285849+0000","last_became_peered":"2024-02-21T23:09:53.285849+0000","last_unstale":"2024-02-21T23:10:08.235691+0000","last_undegraded":"2024-02-21T23:10:08.235691+0000","last_fullsized":"2024-02-21T23:10:08.235691+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:51:08.543135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":421,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-02-21T23:10:05.368921+0000","last_change":"2024-02-21T23:02:32.289335+0000","last_active":"2024-02-21T23:10:05.368921+0000","last_peered":"2024-02-21T23:10:05.368921+0000","last_clean":"2024-02-21T23:10:05.368921+0000","last_became_active":"2024-02-21T23:02:32.286913+0000","last_became_peered":"2024-02-21T23:02:32.286913+0000","last_unstale":"2024-02-21T23:10:05.368921+0000","last_undegraded":"2024-02-21T23:10:05.368921+0000","last_fullsized":"2024-02-21T23:10:05.368921+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_clean_scrub_stamp":"2024-02-21T23:02:31.172569+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:36:23.987374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":26,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":25,"num_pg":32,"stat_sum":{"num_bytes":672,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15,"ondisk_log_size":15,"up":64,"acting":64,"num_store_stats":8},{"poolid":24,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":23,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574941,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33124,"kb_used_data":1944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338716,"statfs":{"total":96636764160,"available":96602845184,"internally_reserved":0,"allocated":1990656,"data_stored":1530590,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574942,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33064,"kb_used_data":1880,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338776,"statfs":{"total":96636764160,"available":96602906624,"internally_reserved":0,"allocated":1925120,"data_stored":940222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574941,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33056,"kb_used_data":1876,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338784,"statfs":{"total":96636764160,"available":96602914816,"internally_reserved":0,"allocated":1921024,"data_stored":932824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574942,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33080,"kb_used_data":1896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338760,"statfs":{"total":96636764160,"available":96602890240,"internally_reserved":0,"allocated":1941504,"data_stored":940986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574943,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33120,"kb_used_data":1932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338720,"statfs":{"total":96636764160,"available":96602849280,"internally_reserved":0,"allocated":1978368,"data_stored":958172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574944,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33096,"kb_used_data":1908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338744,"statfs":{"total":96636764160,"available":96602873856,"internally_reserved":0,"allocated":1953792,"data_stored":956378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574942,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33104,"kb_used_data":1916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338736,"statfs":{"total":96636764160,"available":96602865664,"internally_reserved":0,"allocated":1961984,"data_stored":957408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574942,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33172,"kb_used_data":1992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338668,"statfs":{"total":96636764160,"available":96602796032,"internally_reserved":0,"allocated":2039808,"data_stored":1546996,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-21T23:10:12.064 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.3d is not active+clean 2024-02-21T23:10:12.064 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 153, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:10:05.386132+0000', 'last_change': '2024-02-21T23:10:05.386132+0000', 'last_active': '2024-02-21T23:10:05.335854+0000', 'last_peered': '2024-02-21T23:10:05.335854+0000', 'last_clean': '2024-02-21T23:10:05.335854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:10:05.386132+0000', 'last_undegraded': '2024-02-21T23:10:05.386132+0000', 'last_fullsized': '2024-02-21T23:10:05.386132+0000', 'mapping_epoch': 153, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 153, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'last_clean_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-02-21T23:10:12.064 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.38 is not active+clean 2024-02-21T23:10:12.064 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 153, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:10:05.362732+0000', 'last_change': '2024-02-21T23:10:05.362732+0000', 'last_active': '2024-02-21T23:10:05.335854+0000', 'last_peered': '2024-02-21T23:10:05.335854+0000', 'last_clean': '2024-02-21T23:10:05.335854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:10:05.362732+0000', 'last_undegraded': '2024-02-21T23:10:05.362732+0000', 'last_fullsized': '2024-02-21T23:10:05.362732+0000', 'mapping_epoch': 153, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 153, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'last_clean_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-02-21T23:10:12.064 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.35 is not active+clean 2024-02-21T23:10:12.065 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 153, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:10:05.373594+0000', 'last_change': '2024-02-21T23:10:05.373594+0000', 'last_active': '2024-02-21T23:10:05.335854+0000', 'last_peered': '2024-02-21T23:10:05.335854+0000', 'last_clean': '2024-02-21T23:10:05.335854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:10:05.373594+0000', 'last_undegraded': '2024-02-21T23:10:05.373594+0000', 'last_fullsized': '2024-02-21T23:10:05.373594+0000', 'mapping_epoch': 153, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 153, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'last_clean_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-02-21T23:10:12.065 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.33 is not active+clean 2024-02-21T23:10:12.065 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 153, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:10:05.381970+0000', 'last_change': '2024-02-21T23:10:05.381970+0000', 'last_active': '2024-02-21T23:10:05.335854+0000', 'last_peered': '2024-02-21T23:10:05.335854+0000', 'last_clean': '2024-02-21T23:10:05.335854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:10:05.381970+0000', 'last_undegraded': '2024-02-21T23:10:05.381970+0000', 'last_fullsized': '2024-02-21T23:10:05.381970+0000', 'mapping_epoch': 153, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 153, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'last_clean_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-02-21T23:10:12.065 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.2d is not active+clean 2024-02-21T23:10:12.065 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 153, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:10:05.382164+0000', 'last_change': '2024-02-21T23:10:05.382164+0000', 'last_active': '2024-02-21T23:10:05.335854+0000', 'last_peered': '2024-02-21T23:10:05.335854+0000', 'last_clean': '2024-02-21T23:10:05.335854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:10:05.382164+0000', 'last_undegraded': '2024-02-21T23:10:05.382164+0000', 'last_fullsized': '2024-02-21T23:10:05.382164+0000', 'mapping_epoch': 153, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 153, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'last_clean_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-02-21T23:10:12.065 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.2e is not active+clean 2024-02-21T23:10:12.065 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 153, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:10:05.383479+0000', 'last_change': '2024-02-21T23:10:05.383479+0000', 'last_active': '2024-02-21T23:10:05.335854+0000', 'last_peered': '2024-02-21T23:10:05.335854+0000', 'last_clean': '2024-02-21T23:10:05.335854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:10:05.383479+0000', 'last_undegraded': '2024-02-21T23:10:05.383479+0000', 'last_fullsized': '2024-02-21T23:10:05.383479+0000', 'mapping_epoch': 153, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 153, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'last_clean_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-02-21T23:10:12.065 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.29 is not active+clean 2024-02-21T23:10:12.065 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 153, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:10:05.363048+0000', 'last_change': '2024-02-21T23:10:05.363048+0000', 'last_active': '2024-02-21T23:10:05.335854+0000', 'last_peered': '2024-02-21T23:10:05.335854+0000', 'last_clean': '2024-02-21T23:10:05.335854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:10:05.363048+0000', 'last_undegraded': '2024-02-21T23:10:05.363048+0000', 'last_fullsized': '2024-02-21T23:10:05.363048+0000', 'mapping_epoch': 153, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 153, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'last_clean_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-02-21T23:10:12.065 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.25 is not active+clean 2024-02-21T23:10:12.065 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 153, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:10:05.365921+0000', 'last_change': '2024-02-21T23:10:05.365921+0000', 'last_active': '2024-02-21T23:10:05.335854+0000', 'last_peered': '2024-02-21T23:10:05.335854+0000', 'last_clean': '2024-02-21T23:10:05.335854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:10:05.365921+0000', 'last_undegraded': '2024-02-21T23:10:05.365921+0000', 'last_fullsized': '2024-02-21T23:10:05.365921+0000', 'mapping_epoch': 153, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 153, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'last_clean_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-02-21T23:10:12.066 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.26 is not active+clean 2024-02-21T23:10:12.066 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 153, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:10:05.374040+0000', 'last_change': '2024-02-21T23:10:05.374040+0000', 'last_active': '2024-02-21T23:10:05.335854+0000', 'last_peered': '2024-02-21T23:10:05.335854+0000', 'last_clean': '2024-02-21T23:10:05.335854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:10:05.374040+0000', 'last_undegraded': '2024-02-21T23:10:05.374040+0000', 'last_fullsized': '2024-02-21T23:10:05.374040+0000', 'mapping_epoch': 153, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 153, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'last_clean_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-02-21T23:10:12.066 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.23 is not active+clean 2024-02-21T23:10:12.066 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 153, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:10:05.371720+0000', 'last_change': '2024-02-21T23:10:05.371720+0000', 'last_active': '2024-02-21T23:10:05.335854+0000', 'last_peered': '2024-02-21T23:10:05.335854+0000', 'last_clean': '2024-02-21T23:10:05.335854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:10:05.371720+0000', 'last_undegraded': '2024-02-21T23:10:05.371720+0000', 'last_fullsized': '2024-02-21T23:10:05.371720+0000', 'mapping_epoch': 153, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 153, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'last_clean_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-02-21T23:10:12.066 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.22 is not active+clean 2024-02-21T23:10:12.066 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 153, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:10:05.375181+0000', 'last_change': '2024-02-21T23:10:05.375181+0000', 'last_active': '2024-02-21T23:10:05.335854+0000', 'last_peered': '2024-02-21T23:10:05.335854+0000', 'last_clean': '2024-02-21T23:10:05.335854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:10:05.375181+0000', 'last_undegraded': '2024-02-21T23:10:05.375181+0000', 'last_fullsized': '2024-02-21T23:10:05.375181+0000', 'mapping_epoch': 153, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 153, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'last_clean_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-02-21T23:10:12.066 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.12 is not active+clean 2024-02-21T23:10:12.066 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 153, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:10:05.377674+0000', 'last_change': '2024-02-21T23:10:05.377674+0000', 'last_active': '2024-02-21T23:10:05.335854+0000', 'last_peered': '2024-02-21T23:10:05.335854+0000', 'last_clean': '2024-02-21T23:10:05.335854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:10:05.377674+0000', 'last_undegraded': '2024-02-21T23:10:05.377674+0000', 'last_fullsized': '2024-02-21T23:10:05.377674+0000', 'mapping_epoch': 153, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 153, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'last_clean_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-02-21T23:10:12.066 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.13 is not active+clean 2024-02-21T23:10:12.066 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 153, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:10:05.378353+0000', 'last_change': '2024-02-21T23:10:05.378353+0000', 'last_active': '2024-02-21T23:10:05.335854+0000', 'last_peered': '2024-02-21T23:10:05.335854+0000', 'last_clean': '2024-02-21T23:10:05.335854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:10:05.378353+0000', 'last_undegraded': '2024-02-21T23:10:05.378353+0000', 'last_fullsized': '2024-02-21T23:10:05.378353+0000', 'mapping_epoch': 153, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 153, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'last_clean_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-02-21T23:10:12.066 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.16 is not active+clean 2024-02-21T23:10:12.067 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 153, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:10:05.382764+0000', 'last_change': '2024-02-21T23:10:05.382764+0000', 'last_active': '2024-02-21T23:10:05.335854+0000', 'last_peered': '2024-02-21T23:10:05.335854+0000', 'last_clean': '2024-02-21T23:10:05.335854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:10:05.382764+0000', 'last_undegraded': '2024-02-21T23:10:05.382764+0000', 'last_fullsized': '2024-02-21T23:10:05.382764+0000', 'mapping_epoch': 153, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 153, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'last_clean_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-02-21T23:10:12.067 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.14 is not active+clean 2024-02-21T23:10:12.067 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 153, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:10:05.364119+0000', 'last_change': '2024-02-21T23:10:05.364119+0000', 'last_active': '2024-02-21T23:10:05.335854+0000', 'last_peered': '2024-02-21T23:10:05.335854+0000', 'last_clean': '2024-02-21T23:10:05.335854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:10:05.364119+0000', 'last_undegraded': '2024-02-21T23:10:05.364119+0000', 'last_fullsized': '2024-02-21T23:10:05.364119+0000', 'mapping_epoch': 153, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 153, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'last_clean_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-02-21T23:10:12.067 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.18 is not active+clean 2024-02-21T23:10:12.067 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 153, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:10:05.376263+0000', 'last_change': '2024-02-21T23:10:05.376263+0000', 'last_active': '2024-02-21T23:10:05.335854+0000', 'last_peered': '2024-02-21T23:10:05.335854+0000', 'last_clean': '2024-02-21T23:10:05.335854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:10:05.376263+0000', 'last_undegraded': '2024-02-21T23:10:05.376263+0000', 'last_fullsized': '2024-02-21T23:10:05.376263+0000', 'mapping_epoch': 153, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 153, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'last_clean_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-02-21T23:10:12.067 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.1e is not active+clean 2024-02-21T23:10:12.067 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 153, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:10:05.373032+0000', 'last_change': '2024-02-21T23:10:05.373032+0000', 'last_active': '2024-02-21T23:10:05.335854+0000', 'last_peered': '2024-02-21T23:10:05.335854+0000', 'last_clean': '2024-02-21T23:10:05.335854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:10:05.373032+0000', 'last_undegraded': '2024-02-21T23:10:05.373032+0000', 'last_fullsized': '2024-02-21T23:10:05.373032+0000', 'mapping_epoch': 153, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 153, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'last_clean_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-02-21T23:10:12.067 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.1f is not active+clean 2024-02-21T23:10:12.067 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 153, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:10:05.382382+0000', 'last_change': '2024-02-21T23:10:05.382382+0000', 'last_active': '2024-02-21T23:10:05.335854+0000', 'last_peered': '2024-02-21T23:10:05.335854+0000', 'last_clean': '2024-02-21T23:10:05.335854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:10:05.382382+0000', 'last_undegraded': '2024-02-21T23:10:05.382382+0000', 'last_fullsized': '2024-02-21T23:10:05.382382+0000', 'mapping_epoch': 153, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 153, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'last_clean_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-02-21T23:10:12.067 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.1c is not active+clean 2024-02-21T23:10:12.067 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 153, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:10:05.365861+0000', 'last_change': '2024-02-21T23:10:05.365861+0000', 'last_active': '2024-02-21T23:10:05.335854+0000', 'last_peered': '2024-02-21T23:10:05.335854+0000', 'last_clean': '2024-02-21T23:10:05.335854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:10:05.365861+0000', 'last_undegraded': '2024-02-21T23:10:05.365861+0000', 'last_fullsized': '2024-02-21T23:10:05.365861+0000', 'mapping_epoch': 153, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 153, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'last_clean_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-02-21T23:10:12.068 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.3 is not active+clean 2024-02-21T23:10:12.068 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 153, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:10:05.363690+0000', 'last_change': '2024-02-21T23:10:05.363690+0000', 'last_active': '2024-02-21T23:10:05.335854+0000', 'last_peered': '2024-02-21T23:10:05.335854+0000', 'last_clean': '2024-02-21T23:10:05.335854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:10:05.363690+0000', 'last_undegraded': '2024-02-21T23:10:05.363690+0000', 'last_fullsized': '2024-02-21T23:10:05.363690+0000', 'mapping_epoch': 153, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 153, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'last_clean_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-02-21T23:10:12.068 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.0 is not active+clean 2024-02-21T23:10:12.068 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 153, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:10:05.386121+0000', 'last_change': '2024-02-21T23:10:05.386121+0000', 'last_active': '2024-02-21T23:10:05.335854+0000', 'last_peered': '2024-02-21T23:10:05.335854+0000', 'last_clean': '2024-02-21T23:10:05.335854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:10:05.386121+0000', 'last_undegraded': '2024-02-21T23:10:05.386121+0000', 'last_fullsized': '2024-02-21T23:10:05.386121+0000', 'mapping_epoch': 153, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 153, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'last_clean_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-02-21T23:10:12.068 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.1 is not active+clean 2024-02-21T23:10:12.068 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 153, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:10:05.380894+0000', 'last_change': '2024-02-21T23:10:05.380894+0000', 'last_active': '2024-02-21T23:10:05.335854+0000', 'last_peered': '2024-02-21T23:10:05.335854+0000', 'last_clean': '2024-02-21T23:10:05.335854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:10:05.380894+0000', 'last_undegraded': '2024-02-21T23:10:05.380894+0000', 'last_fullsized': '2024-02-21T23:10:05.380894+0000', 'mapping_epoch': 153, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 153, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'last_clean_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-02-21T23:10:12.068 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.6 is not active+clean 2024-02-21T23:10:12.068 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 153, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:10:05.363355+0000', 'last_change': '2024-02-21T23:10:05.363355+0000', 'last_active': '2024-02-21T23:10:05.335854+0000', 'last_peered': '2024-02-21T23:10:05.335854+0000', 'last_clean': '2024-02-21T23:10:05.335854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:10:05.363355+0000', 'last_undegraded': '2024-02-21T23:10:05.363355+0000', 'last_fullsized': '2024-02-21T23:10:05.363355+0000', 'mapping_epoch': 153, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 153, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'last_clean_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-02-21T23:10:12.068 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.e is not active+clean 2024-02-21T23:10:12.068 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 153, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:10:05.371905+0000', 'last_change': '2024-02-21T23:10:05.371905+0000', 'last_active': '2024-02-21T23:10:05.335854+0000', 'last_peered': '2024-02-21T23:10:05.335854+0000', 'last_clean': '2024-02-21T23:10:05.335854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:10:05.371905+0000', 'last_undegraded': '2024-02-21T23:10:05.371905+0000', 'last_fullsized': '2024-02-21T23:10:05.371905+0000', 'mapping_epoch': 153, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 153, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'last_clean_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-02-21T23:10:12.068 INFO:tasks.cephfs.filesystem.ceph_manager:PG 26.1b is not active+clean 2024-02-21T23:10:12.069 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '26.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 153, 'state': 'creating+peering', 'last_fresh': '2024-02-21T23:10:05.385924+0000', 'last_change': '2024-02-21T23:10:05.385924+0000', 'last_active': '2024-02-21T23:10:05.335854+0000', 'last_peered': '2024-02-21T23:10:05.335854+0000', 'last_clean': '2024-02-21T23:10:05.335854+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-02-21T23:10:05.385924+0000', 'last_undegraded': '2024-02-21T23:10:05.385924+0000', 'last_fullsized': '2024-02-21T23:10:05.385924+0000', 'mapping_epoch': 153, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 153, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'last_clean_scrub_stamp': '2024-02-21T23:10:05.335854+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-02-21T23:10:12.069 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-02-21T23:10:12.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.325+0000 7fe7dde00640 1 Processor -- start 2024-02-21T23:10:12.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.326+0000 7fe7dde00640 1 -- start start 2024-02-21T23:10:12.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.327+0000 7fe7dde00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fe7d80aff90 0x7fe7d80c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:12.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.327+0000 7fe7dde00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe7d806dbf0 0x7fe7d8158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:12.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.327+0000 7fe7dde00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe7d80c5720 0x7fe7d8159000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:12.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.327+0000 7fe7dde00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7d80c7270 con 0x7fe7d806dbf0 2024-02-21T23:10:12.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.327+0000 7fe7dde00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7d8158630 con 0x7fe7d80aff90 2024-02-21T23:10:12.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.327+0000 7fe7dde00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d80c74c0 con 0x7fe7d80c5720 2024-02-21T23:10:12.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.327+0000 7fe7d7e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe7d806dbf0 0x7fe7d8158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:12.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.328+0000 7fe7d7e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe7d806dbf0 0x7fe7d8158ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35608/0 (socket says 172.21.15.28:35608) 2024-02-21T23:10:12.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.328+0000 7fe7d7e00640 1 -- 172.21.15.28:0/2735215869 learned_addr learned my addr 172.21.15.28:0/2735215869 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:12.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.328+0000 7fe7dca00640 1 --2- 172.21.15.28:0/2735215869 >> v2:172.21.15.81:3300/0 conn(0x7fe7d80aff90 0x7fe7d80c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:12.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.328+0000 7fe7dd400640 1 --2- 172.21.15.28:0/2735215869 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe7d80c5720 0x7fe7d8159000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:12.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.328+0000 7fe7d7e00640 1 -- 172.21.15.28:0/2735215869 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe7d80c5720 msgr2=0x7fe7d8159000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:12.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.328+0000 7fe7d7e00640 1 --2- 172.21.15.28:0/2735215869 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe7d80c5720 0x7fe7d8159000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:12.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.328+0000 7fe7d7e00640 1 -- 172.21.15.28:0/2735215869 >> v2:172.21.15.81:3300/0 conn(0x7fe7d80aff90 msgr2=0x7fe7d80c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:12.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.328+0000 7fe7d7e00640 1 --2- 172.21.15.28:0/2735215869 >> v2:172.21.15.81:3300/0 conn(0x7fe7d80aff90 0x7fe7d80c51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:12.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.328+0000 7fe7d7e00640 1 -- 172.21.15.28:0/2735215869 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7d80c77a0 con 0x7fe7d806dbf0 2024-02-21T23:10:12.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.329+0000 7fe7d7e00640 1 --2- 172.21.15.28:0/2735215869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7d806dbf0 0x7fe7d8158ac0 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7fe7d80aa1b0 tx=0x7fe7c402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ac0dc522c7fad82 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:12.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.330+0000 7fe7d7400640 1 -- 172.21.15.28:0/2735215869 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7c4035020 con 0x7fe7d806dbf0 2024-02-21T23:10:12.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.330+0000 7fe7d7400640 1 -- 172.21.15.28:0/2735215869 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7c4031870 con 0x7fe7d806dbf0 2024-02-21T23:10:12.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.330+0000 7fe7d7400640 1 -- 172.21.15.28:0/2735215869 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7c4031c40 con 0x7fe7d806dbf0 2024-02-21T23:10:12.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.330+0000 7fe7dde00640 1 -- 172.21.15.28:0/2735215869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7d806dbf0 msgr2=0x7fe7d8158ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:12.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.330+0000 7fe7dde00640 1 --2- 172.21.15.28:0/2735215869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7d806dbf0 0x7fe7d8158ac0 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7fe7d80aa1b0 tx=0x7fe7c402b910 comp rx=0 tx=0).stop 2024-02-21T23:10:12.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.330+0000 7fe7dde00640 1 -- 172.21.15.28:0/2735215869 shutdown_connections 2024-02-21T23:10:12.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.330+0000 7fe7dde00640 1 --2- 172.21.15.28:0/2735215869 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe7d80c5720 0x7fe7d8159000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:12.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.330+0000 7fe7dde00640 1 --2- 172.21.15.28:0/2735215869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7d806dbf0 0x7fe7d8158ac0 unknown :-1 s=CLOSED pgs=1746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:12.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.330+0000 7fe7dde00640 1 --2- 172.21.15.28:0/2735215869 >> v2:172.21.15.81:3300/0 conn(0x7fe7d80aff90 0x7fe7d80c51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:12.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.330+0000 7fe7dde00640 1 -- 172.21.15.28:0/2735215869 >> 172.21.15.28:0/2735215869 conn(0x7fe7d80ac370 msgr2=0x7fe7d80ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:12.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.331+0000 7fe7dde00640 1 -- 172.21.15.28:0/2735215869 shutdown_connections 2024-02-21T23:10:12.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.331+0000 7fe7dde00640 1 -- 172.21.15.28:0/2735215869 wait complete. 2024-02-21T23:10:12.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.332+0000 7fe7dde00640 1 Processor -- start 2024-02-21T23:10:12.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.332+0000 7fe7dde00640 1 -- start start 2024-02-21T23:10:12.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.333+0000 7fe7dde00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7d806dbf0 0x7fe7d80c1ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:12.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.333+0000 7fe7dde00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe7d80aff90 0x7fe7d814f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:12.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.333+0000 7fe7dde00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe7d80c5720 0x7fe7d814f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:12.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.333+0000 7fe7dde00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d80c8760 con 0x7fe7d806dbf0 2024-02-21T23:10:12.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.333+0000 7fe7dde00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d80be0c0 con 0x7fe7d80aff90 2024-02-21T23:10:12.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.333+0000 7fe7dde00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7d80c88e0 con 0x7fe7d80c5720 2024-02-21T23:10:12.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.334+0000 7fe7dca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7d806dbf0 0x7fe7d80c1ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:12.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.334+0000 7fe7dca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7d806dbf0 0x7fe7d80c1ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35610/0 (socket says 172.21.15.28:35610) 2024-02-21T23:10:12.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.334+0000 7fe7dca00640 1 -- 172.21.15.28:0/1983996192 learned_addr learned my addr 172.21.15.28:0/1983996192 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:12.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.334+0000 7fe7dd400640 1 --2- 172.21.15.28:0/1983996192 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe7d80c5720 0x7fe7d814f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:12.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.334+0000 7fe7d7e00640 1 --2- 172.21.15.28:0/1983996192 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe7d80aff90 0x7fe7d814f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:12.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.334+0000 7fe7dca00640 1 -- 172.21.15.28:0/1983996192 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe7d80c5720 msgr2=0x7fe7d814f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:12.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.334+0000 7fe7dca00640 1 --2- 172.21.15.28:0/1983996192 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe7d80c5720 0x7fe7d814f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:12.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.334+0000 7fe7dca00640 1 -- 172.21.15.28:0/1983996192 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe7d80aff90 msgr2=0x7fe7d814f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:12.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.334+0000 7fe7dca00640 1 --2- 172.21.15.28:0/1983996192 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe7d80aff90 0x7fe7d814f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:12.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.334+0000 7fe7dca00640 1 -- 172.21.15.28:0/1983996192 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7d81ee510 con 0x7fe7d806dbf0 2024-02-21T23:10:12.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.335+0000 7fe7dca00640 1 --2- 172.21.15.28:0/1983996192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7d806dbf0 0x7fe7d80c1ba0 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc00d830 tx=0x7fe7cc00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:12.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.335+0000 7fe7d4c00640 1 -- 172.21.15.28:0/1983996192 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc018020 con 0x7fe7d806dbf0 2024-02-21T23:10:12.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.335+0000 7fe7d4c00640 1 -- 172.21.15.28:0/1983996192 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7cc0096a0 con 0x7fe7d806dbf0 2024-02-21T23:10:12.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.335+0000 7fe7dde00640 1 -- 172.21.15.28:0/1983996192 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7d81ee7f0 con 0x7fe7d806dbf0 2024-02-21T23:10:12.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.335+0000 7fe7d4c00640 1 -- 172.21.15.28:0/1983996192 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc009a70 con 0x7fe7d806dbf0 2024-02-21T23:10:12.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.335+0000 7fe7dde00640 1 -- 172.21.15.28:0/1983996192 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7d81eedb0 con 0x7fe7d806dbf0 2024-02-21T23:10:12.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.336+0000 7fe7dde00640 1 -- 172.21.15.28:0/1983996192 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7d804f290 con 0x7fe7d806dbf0 2024-02-21T23:10:12.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.337+0000 7fe7d4c00640 1 -- 172.21.15.28:0/1983996192 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe7cc01b020 con 0x7fe7d806dbf0 2024-02-21T23:10:12.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.339+0000 7fe7d4c00640 1 --2- 172.21.15.28:0/1983996192 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe7b806b4d0 0x7fe7b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:12.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.339+0000 7fe7d4c00640 1 -- 172.21.15.28:0/1983996192 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 6957+0+0 (secure 0 0 0) 0x7fe7cc08abc0 con 0x7fe7d806dbf0 2024-02-21T23:10:12.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.339+0000 7fe7d7e00640 1 --2- 172.21.15.28:0/1983996192 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe7b806b4d0 0x7fe7b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:12.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.341+0000 7fe7d7e00640 1 --2- 172.21.15.28:0/1983996192 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe7b806b4d0 0x7fe7b806d990 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fe7c4036000 tx=0x7fe7c403c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:12.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.343+0000 7fe7d4c00640 1 -- 172.21.15.28:0/1983996192 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe7cc022030 con 0x7fe7d806dbf0 2024-02-21T23:10:12.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.705+0000 7fe7dde00640 1 -- 172.21.15.28:0/1983996192 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fe7d81559a0 con 0x7fe7d806dbf0 2024-02-21T23:10:12.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.706+0000 7fe7d4c00640 1 -- 172.21.15.28:0/1983996192 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1627 (secure 0 0 0) 0x7fe7cc0102e0 con 0x7fe7d806dbf0 2024-02-21T23:10:12.695 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:10:12.695 INFO:teuthology.orchestra.run.smithi028.stdout:{"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":475,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":155,"num_osds":8,"num_up_osds":8,"osd_up_since":1708556550,"num_in_osds":8,"osd_in_since":1708556541,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":168},{"state_name":"creating+peering","count":25}],"num_pgs":193,"num_pools":5,"num_objects":31,"data_bytes":593450,"bytes_used":271171584,"bytes_avail":772822941696,"bytes_total":773094113280,"inactive_pgs_ratio":0.12953367829322815,"write_bytes_sec":341,"read_op_per_sec":0,"write_op_per_sec":2},"fsmap":{"epoch":163,"id":11,"up":1,"in":1,"max":1,"id":12,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":12,"rank":0,"name":"c","status":"up:active","gid":8817},{"filesystem_id":11,"rank":0,"name":"a","status":"up:active","gid":8826}],"up:standby":2},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":18,"modified":"2024-02-21T23:09:40.814548+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-02-21T23:10:12.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.708+0000 7fe7dde00640 1 -- 172.21.15.28:0/1983996192 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe7b806b4d0 msgr2=0x7fe7b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:12.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.708+0000 7fe7dde00640 1 --2- 172.21.15.28:0/1983996192 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe7b806b4d0 0x7fe7b806d990 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fe7c4036000 tx=0x7fe7c403c040 comp rx=0 tx=0).stop 2024-02-21T23:10:12.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.708+0000 7fe7dde00640 1 -- 172.21.15.28:0/1983996192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7d806dbf0 msgr2=0x7fe7d80c1ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:12.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.708+0000 7fe7dde00640 1 --2- 172.21.15.28:0/1983996192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7d806dbf0 0x7fe7d80c1ba0 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc00d830 tx=0x7fe7cc00dcf0 comp rx=0 tx=0).stop 2024-02-21T23:10:12.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.709+0000 7fe7dde00640 1 -- 172.21.15.28:0/1983996192 shutdown_connections 2024-02-21T23:10:12.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.709+0000 7fe7dde00640 1 --2- 172.21.15.28:0/1983996192 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe7b806b4d0 0x7fe7b806d990 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:12.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.709+0000 7fe7dde00640 1 --2- 172.21.15.28:0/1983996192 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe7d80c5720 0x7fe7d814f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:12.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.709+0000 7fe7dde00640 1 --2- 172.21.15.28:0/1983996192 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe7d80aff90 0x7fe7d814f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:12.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.709+0000 7fe7dde00640 1 --2- 172.21.15.28:0/1983996192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7d806dbf0 0x7fe7d80c1ba0 unknown :-1 s=CLOSED pgs=1747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:12.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.709+0000 7fe7dde00640 1 -- 172.21.15.28:0/1983996192 >> 172.21.15.28:0/1983996192 conn(0x7fe7d80ac370 msgr2=0x7fe7d80ac750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:12.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.709+0000 7fe7dde00640 1 -- 172.21.15.28:0/1983996192 shutdown_connections 2024-02-21T23:10:12.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.709+0000 7fe7dde00640 1 -- 172.21.15.28:0/1983996192 wait complete. 2024-02-21T23:10:12.720 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-02-21T23:10:12.720 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-21T23:10:12.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.949+0000 7ff29ee00640 1 Processor -- start 2024-02-21T23:10:12.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.949+0000 7ff29ee00640 1 -- start start 2024-02-21T23:10:12.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.950+0000 7ff29ee00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff29806dbf0 0x7ff298158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:12.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.950+0000 7ff29ee00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff298155960 0x7ff298159000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:12.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.950+0000 7ff29ee00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2980c74c0 con 0x7ff2980aff90 2024-02-21T23:10:12.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.950+0000 7ff29ee00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2980c7270 con 0x7ff29806dbf0 2024-02-21T23:10:12.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.950+0000 7ff29ee00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff298158630 con 0x7ff298155960 2024-02-21T23:10:12.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.950+0000 7ff29da00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff2980aff90 0x7ff2980c51e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60890/0 (socket says 172.21.15.28:60890) 2024-02-21T23:10:12.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.951+0000 7ff29da00640 1 -- 172.21.15.28:0/4056642502 learned_addr learned my addr 172.21.15.28:0/4056642502 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:10:12.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.951+0000 7ff29d000640 1 --2- 172.21.15.28:0/4056642502 >> v2:172.21.15.28:3300/0 conn(0x7ff29806dbf0 0x7ff298158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:12.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.951+0000 7ff29e400640 1 --2- 172.21.15.28:0/4056642502 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff298155960 0x7ff298159000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:12.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.951+0000 7ff297e00640 1 -- 172.21.15.28:0/4056642502 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3790216306 0 0) 0x7ff2980c74c0 con 0x7ff2980aff90 2024-02-21T23:10:12.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.951+0000 7ff297e00640 1 -- 172.21.15.28:0/4056642502 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff280003520 con 0x7ff2980aff90 2024-02-21T23:10:12.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.951+0000 7ff29d000640 1 -- 172.21.15.28:0/4056642502 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff298155960 msgr2=0x7ff298159000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:12.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.951+0000 7ff29d000640 1 --2- 172.21.15.28:0/4056642502 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff298155960 0x7ff298159000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:12.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.951+0000 7ff29d000640 1 -- 172.21.15.28:0/4056642502 >> v1:172.21.15.28:6789/0 conn(0x7ff2980aff90 legacy=0x7ff2980c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:12.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.951+0000 7ff29d000640 1 -- 172.21.15.28:0/4056642502 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2980c77a0 con 0x7ff29806dbf0 2024-02-21T23:10:12.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.953+0000 7ff29d000640 1 --2- 172.21.15.28:0/4056642502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff29806dbf0 0x7ff298158ac0 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7ff2840097a0 tx=0x7ff28402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f2588a9b5c9e756 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:12.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.953+0000 7ff297e00640 1 -- 172.21.15.28:0/4056642502 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff284035020 con 0x7ff29806dbf0 2024-02-21T23:10:12.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.953+0000 7ff297e00640 1 -- 172.21.15.28:0/4056642502 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff284031810 con 0x7ff29806dbf0 2024-02-21T23:10:12.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.953+0000 7ff297e00640 1 -- 172.21.15.28:0/4056642502 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff284031bc0 con 0x7ff29806dbf0 2024-02-21T23:10:12.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.953+0000 7ff29ee00640 1 -- 172.21.15.28:0/4056642502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff29806dbf0 msgr2=0x7ff298158ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:12.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.953+0000 7ff29ee00640 1 --2- 172.21.15.28:0/4056642502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff29806dbf0 0x7ff298158ac0 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7ff2840097a0 tx=0x7ff28402b8b0 comp rx=0 tx=0).stop 2024-02-21T23:10:12.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.954+0000 7ff29ee00640 1 -- 172.21.15.28:0/4056642502 shutdown_connections 2024-02-21T23:10:12.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.954+0000 7ff29ee00640 1 --2- 172.21.15.28:0/4056642502 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff298155960 0x7ff298159000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:12.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.954+0000 7ff29ee00640 1 --2- 172.21.15.28:0/4056642502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff29806dbf0 0x7ff298158ac0 unknown :-1 s=CLOSED pgs=1749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:12.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.954+0000 7ff29ee00640 1 -- 172.21.15.28:0/4056642502 >> 172.21.15.28:0/4056642502 conn(0x7ff2980ac370 msgr2=0x7ff2980ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:12.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.954+0000 7ff29ee00640 1 -- 172.21.15.28:0/4056642502 shutdown_connections 2024-02-21T23:10:12.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.954+0000 7ff29ee00640 1 -- 172.21.15.28:0/4056642502 wait complete. 2024-02-21T23:10:12.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.956+0000 7ff29ee00640 1 Processor -- start 2024-02-21T23:10:12.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.956+0000 7ff29ee00640 1 -- start start 2024-02-21T23:10:12.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.957+0000 7ff29ee00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff29806dbf0 0x7ff298151f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:12.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.957+0000 7ff29ee00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff2980aff90 0x7ff2980c0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:12.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.957+0000 7ff29ee00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff298155960 0x7ff2980c2660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:12.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.957+0000 7ff29ee00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2980c58f0 con 0x7ff29806dbf0 2024-02-21T23:10:12.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.957+0000 7ff29ee00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2980be0c0 con 0x7ff2980aff90 2024-02-21T23:10:12.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.957+0000 7ff29ee00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2980c5a70 con 0x7ff298155960 2024-02-21T23:10:12.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.957+0000 7ff29da00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff29806dbf0 0x7ff298151f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:12.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.957+0000 7ff29da00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff29806dbf0 0x7ff298151f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35630/0 (socket says 172.21.15.28:35630) 2024-02-21T23:10:12.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.957+0000 7ff29da00640 1 -- 172.21.15.28:0/2572089498 learned_addr learned my addr 172.21.15.28:0/2572089498 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:12.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.957+0000 7ff29d000640 1 --2- 172.21.15.28:0/2572089498 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff2980aff90 0x7ff2980c0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:12.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.957+0000 7ff29e400640 1 --2- 172.21.15.28:0/2572089498 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff298155960 0x7ff2980c2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:12.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.957+0000 7ff29da00640 1 -- 172.21.15.28:0/2572089498 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff298155960 msgr2=0x7ff2980c2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:12.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.958+0000 7ff29da00640 1 --2- 172.21.15.28:0/2572089498 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff298155960 0x7ff2980c2660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:12.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.958+0000 7ff29da00640 1 -- 172.21.15.28:0/2572089498 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff2980aff90 msgr2=0x7ff2980c0840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:12.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.958+0000 7ff29da00640 1 --2- 172.21.15.28:0/2572089498 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff2980aff90 0x7ff2980c0840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:12.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.958+0000 7ff29da00640 1 -- 172.21.15.28:0/2572089498 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2980be240 con 0x7ff29806dbf0 2024-02-21T23:10:12.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.958+0000 7ff29da00640 1 --2- 172.21.15.28:0/2572089498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff29806dbf0 0x7ff298151f80 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7ff28c00c460 tx=0x7ff28c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:12.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.958+0000 7ff295600640 1 -- 172.21.15.28:0/2572089498 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff28c017070 con 0x7ff29806dbf0 2024-02-21T23:10:12.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.958+0000 7ff295600640 1 -- 172.21.15.28:0/2572089498 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff28c005ce0 con 0x7ff29806dbf0 2024-02-21T23:10:12.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.958+0000 7ff295600640 1 -- 172.21.15.28:0/2572089498 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff28c007290 con 0x7ff29806dbf0 2024-02-21T23:10:12.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.958+0000 7ff29ee00640 1 -- 172.21.15.28:0/2572089498 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2981edb30 con 0x7ff29806dbf0 2024-02-21T23:10:12.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.958+0000 7ff29ee00640 1 -- 172.21.15.28:0/2572089498 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2980c1e60 con 0x7ff29806dbf0 2024-02-21T23:10:12.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.960+0000 7ff29ee00640 1 -- 172.21.15.28:0/2572089498 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff29804f290 con 0x7ff29806dbf0 2024-02-21T23:10:12.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.961+0000 7ff295600640 1 -- 172.21.15.28:0/2572089498 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff28c01a070 con 0x7ff29806dbf0 2024-02-21T23:10:12.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.961+0000 7ff295600640 1 --2- 172.21.15.28:0/2572089498 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff27806b4d0 0x7ff27806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:12.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.962+0000 7ff295600640 1 -- 172.21.15.28:0/2572089498 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 6957+0+0 (secure 0 0 0) 0x7ff28c089aa0 con 0x7ff29806dbf0 2024-02-21T23:10:12.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.962+0000 7ff29d000640 1 --2- 172.21.15.28:0/2572089498 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff27806b4d0 0x7ff27806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:12.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.963+0000 7ff29d000640 1 --2- 172.21.15.28:0/2572089498 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff27806b4d0 0x7ff27806d990 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7ff284036000 tx=0x7ff284008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:12.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:12.967+0000 7ff295600640 1 -- 172.21.15.28:0/2572089498 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ff28c058430 con 0x7ff29806dbf0 2024-02-21T23:10:13.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:13.241+0000 7ff29ee00640 1 -- 172.21.15.28:0/2572089498 --> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff2981552b0 con 0x7ff27806b4d0 2024-02-21T23:10:13.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:13.275+0000 7ff295600640 1 -- 172.21.15.28:0/2572089498 <== mgr.4101 v2:172.21.15.81:6832/1848538577 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+491533 (secure 0 0 0) 0x7ff2981552b0 con 0x7ff27806b4d0 2024-02-21T23:10:13.273 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:10:13.276 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-21T23:10:13.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:13.282+0000 7ff29ee00640 1 -- 172.21.15.28:0/2572089498 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff27806b4d0 msgr2=0x7ff27806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:13.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:13.282+0000 7ff29ee00640 1 --2- 172.21.15.28:0/2572089498 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff27806b4d0 0x7ff27806d990 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7ff284036000 tx=0x7ff284008030 comp rx=0 tx=0).stop 2024-02-21T23:10:13.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:13.282+0000 7ff29ee00640 1 -- 172.21.15.28:0/2572089498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff29806dbf0 msgr2=0x7ff298151f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:13.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:13.282+0000 7ff29ee00640 1 --2- 172.21.15.28:0/2572089498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff29806dbf0 0x7ff298151f80 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7ff28c00c460 tx=0x7ff28c00c920 comp rx=0 tx=0).stop 2024-02-21T23:10:13.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:13.282+0000 7ff29ee00640 1 -- 172.21.15.28:0/2572089498 shutdown_connections 2024-02-21T23:10:13.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:13.282+0000 7ff29ee00640 1 --2- 172.21.15.28:0/2572089498 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff27806b4d0 0x7ff27806d990 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:13.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:13.282+0000 7ff29ee00640 1 --2- 172.21.15.28:0/2572089498 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff298155960 0x7ff2980c2660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:13.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:13.282+0000 7ff29ee00640 1 --2- 172.21.15.28:0/2572089498 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff2980aff90 0x7ff2980c0840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:13.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:13.282+0000 7ff29ee00640 1 --2- 172.21.15.28:0/2572089498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff29806dbf0 0x7ff298151f80 unknown :-1 s=CLOSED pgs=1750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:13.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:13.282+0000 7ff29ee00640 1 -- 172.21.15.28:0/2572089498 >> 172.21.15.28:0/2572089498 conn(0x7ff2980ac370 msgr2=0x7ff2980cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:13.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:13.282+0000 7ff29ee00640 1 -- 172.21.15.28:0/2572089498 shutdown_connections 2024-02-21T23:10:13.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:13.283+0000 7ff29ee00640 1 -- 172.21.15.28:0/2572089498 wait complete. 2024-02-21T23:10:13.290 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":379,"stamp":"2024-02-21T23:10:12.825586+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595188,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":249,"num_write_kb":2444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":195,"ondisk_log_size":195,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":386,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":264960,"kb_used_data":15488,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754709760,"statfs":{"total":773094113280,"available":772822794240,"internally_reserved":0,"allocated":15859712,"data_stored":8836744,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.471094"},"pg_stats":[{"pgid":"26.3d","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.380162+0000","last_change":"2024-02-21T23:10:06.382729+0000","last_active":"2024-02-21T23:10:07.380162+0000","last_peered":"2024-02-21T23:10:07.380162+0000","last_clean":"2024-02-21T23:10:07.380162+0000","last_became_active":"2024-02-21T23:10:06.381756+0000","last_became_peered":"2024-02-21T23:10:06.381756+0000","last_unstale":"2024-02-21T23:10:07.380162+0000","last_undegraded":"2024-02-21T23:10:07.380162+0000","last_fullsized":"2024-02-21T23:10:07.380162+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:03:56.597939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.3f","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232348+0000","last_change":"2024-02-21T23:09:54.279018+0000","last_active":"2024-02-21T23:10:08.232348+0000","last_peered":"2024-02-21T23:10:08.232348+0000","last_clean":"2024-02-21T23:10:08.232348+0000","last_became_active":"2024-02-21T23:09:54.278712+0000","last_became_peered":"2024-02-21T23:09:54.278712+0000","last_unstale":"2024-02-21T23:10:08.232348+0000","last_undegraded":"2024-02-21T23:10:08.232348+0000","last_fullsized":"2024-02-21T23:10:08.232348+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:55:18.591062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.3c","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232651+0000","last_change":"2024-02-21T23:10:06.385434+0000","last_active":"2024-02-21T23:10:08.232651+0000","last_peered":"2024-02-21T23:10:08.232651+0000","last_clean":"2024-02-21T23:10:08.232651+0000","last_became_active":"2024-02-21T23:10:06.384878+0000","last_became_peered":"2024-02-21T23:10:06.384878+0000","last_unstale":"2024-02-21T23:10:08.232651+0000","last_undegraded":"2024-02-21T23:10:08.232651+0000","last_fullsized":"2024-02-21T23:10:08.232651+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:52:21.069235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3e","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.609956+0000","last_change":"2024-02-21T23:09:54.285889+0000","last_active":"2024-02-21T23:10:07.609956+0000","last_peered":"2024-02-21T23:10:07.609956+0000","last_clean":"2024-02-21T23:10:07.609956+0000","last_became_active":"2024-02-21T23:09:54.284810+0000","last_became_peered":"2024-02-21T23:09:54.284810+0000","last_unstale":"2024-02-21T23:10:07.609956+0000","last_undegraded":"2024-02-21T23:10:07.609956+0000","last_fullsized":"2024-02-21T23:10:07.609956+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:04:16.317089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.3f","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236866+0000","last_change":"2024-02-21T23:10:06.389219+0000","last_active":"2024-02-21T23:10:08.236866+0000","last_peered":"2024-02-21T23:10:08.236866+0000","last_clean":"2024-02-21T23:10:08.236866+0000","last_became_active":"2024-02-21T23:10:06.388931+0000","last_became_peered":"2024-02-21T23:10:06.388931+0000","last_unstale":"2024-02-21T23:10:08.236866+0000","last_undegraded":"2024-02-21T23:10:08.236866+0000","last_fullsized":"2024-02-21T23:10:08.236866+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:46:45.683446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.3d","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.231699+0000","last_change":"2024-02-21T23:09:54.263978+0000","last_active":"2024-02-21T23:10:08.231699+0000","last_peered":"2024-02-21T23:10:08.231699+0000","last_clean":"2024-02-21T23:10:08.231699+0000","last_became_active":"2024-02-21T23:09:54.263467+0000","last_became_peered":"2024-02-21T23:09:54.263467+0000","last_unstale":"2024-02-21T23:10:08.231699+0000","last_undegraded":"2024-02-21T23:10:08.231699+0000","last_fullsized":"2024-02-21T23:10:08.231699+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:46:35.436815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.3e","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236400+0000","last_change":"2024-02-21T23:10:06.406362+0000","last_active":"2024-02-21T23:10:08.236400+0000","last_peered":"2024-02-21T23:10:08.236400+0000","last_clean":"2024-02-21T23:10:08.236400+0000","last_became_active":"2024-02-21T23:10:06.405859+0000","last_became_peered":"2024-02-21T23:10:06.405859+0000","last_unstale":"2024-02-21T23:10:08.236400+0000","last_undegraded":"2024-02-21T23:10:08.236400+0000","last_fullsized":"2024-02-21T23:10:08.236400+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:09:17.354064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.3c","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.231855+0000","last_change":"2024-02-21T23:09:54.279400+0000","last_active":"2024-02-21T23:10:08.231855+0000","last_peered":"2024-02-21T23:10:08.231855+0000","last_clean":"2024-02-21T23:10:08.231855+0000","last_became_active":"2024-02-21T23:09:54.279076+0000","last_became_peered":"2024-02-21T23:09:54.279076+0000","last_unstale":"2024-02-21T23:10:08.231855+0000","last_undegraded":"2024-02-21T23:10:08.231855+0000","last_fullsized":"2024-02-21T23:10:08.231855+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:02:11.247387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.39","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.375184+0000","last_change":"2024-02-21T23:10:06.375038+0000","last_active":"2024-02-21T23:10:07.375184+0000","last_peered":"2024-02-21T23:10:07.375184+0000","last_clean":"2024-02-21T23:10:07.375184+0000","last_became_active":"2024-02-21T23:10:06.374714+0000","last_became_peered":"2024-02-21T23:10:06.374714+0000","last_unstale":"2024-02-21T23:10:07.375184+0000","last_undegraded":"2024-02-21T23:10:07.375184+0000","last_fullsized":"2024-02-21T23:10:07.375184+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:48:19.473212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.3b","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.233882+0000","last_change":"2024-02-21T23:09:54.271682+0000","last_active":"2024-02-21T23:10:08.233882+0000","last_peered":"2024-02-21T23:10:08.233882+0000","last_clean":"2024-02-21T23:10:08.233882+0000","last_became_active":"2024-02-21T23:09:54.271297+0000","last_became_peered":"2024-02-21T23:09:54.271297+0000","last_unstale":"2024-02-21T23:10:08.233882+0000","last_undegraded":"2024-02-21T23:10:08.233882+0000","last_fullsized":"2024-02-21T23:10:08.233882+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:16:10.796552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.38","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.609207+0000","last_change":"2024-02-21T23:10:06.393732+0000","last_active":"2024-02-21T23:10:07.609207+0000","last_peered":"2024-02-21T23:10:07.609207+0000","last_clean":"2024-02-21T23:10:07.609207+0000","last_became_active":"2024-02-21T23:10:06.393448+0000","last_became_peered":"2024-02-21T23:10:06.393448+0000","last_unstale":"2024-02-21T23:10:07.609207+0000","last_undegraded":"2024-02-21T23:10:07.609207+0000","last_fullsized":"2024-02-21T23:10:07.609207+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:42:20.227218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3a","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.608838+0000","last_change":"2024-02-21T23:09:54.281830+0000","last_active":"2024-02-21T23:10:07.608838+0000","last_peered":"2024-02-21T23:10:07.608838+0000","last_clean":"2024-02-21T23:10:07.608838+0000","last_became_active":"2024-02-21T23:09:54.281334+0000","last_became_peered":"2024-02-21T23:09:54.281334+0000","last_unstale":"2024-02-21T23:10:07.608838+0000","last_undegraded":"2024-02-21T23:10:07.608838+0000","last_fullsized":"2024-02-21T23:10:07.608838+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:07:24.304915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.3b","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.233350+0000","last_change":"2024-02-21T23:10:06.385935+0000","last_active":"2024-02-21T23:10:08.233350+0000","last_peered":"2024-02-21T23:10:08.233350+0000","last_clean":"2024-02-21T23:10:08.233350+0000","last_became_active":"2024-02-21T23:10:06.385657+0000","last_became_peered":"2024-02-21T23:10:06.385657+0000","last_unstale":"2024-02-21T23:10:08.233350+0000","last_undegraded":"2024-02-21T23:10:08.233350+0000","last_fullsized":"2024-02-21T23:10:08.233350+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:27:47.158633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.39","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.233971+0000","last_change":"2024-02-21T23:09:54.277806+0000","last_active":"2024-02-21T23:10:08.233971+0000","last_peered":"2024-02-21T23:10:08.233971+0000","last_clean":"2024-02-21T23:10:08.233971+0000","last_became_active":"2024-02-21T23:09:54.277446+0000","last_became_peered":"2024-02-21T23:09:54.277446+0000","last_unstale":"2024-02-21T23:10:08.233971+0000","last_undegraded":"2024-02-21T23:10:08.233971+0000","last_fullsized":"2024-02-21T23:10:08.233971+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:59:34.646775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.3a","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.231962+0000","last_change":"2024-02-21T23:10:06.394386+0000","last_active":"2024-02-21T23:10:08.231962+0000","last_peered":"2024-02-21T23:10:08.231962+0000","last_clean":"2024-02-21T23:10:08.231962+0000","last_became_active":"2024-02-21T23:10:06.391655+0000","last_became_peered":"2024-02-21T23:10:06.391655+0000","last_unstale":"2024-02-21T23:10:08.231962+0000","last_undegraded":"2024-02-21T23:10:08.231962+0000","last_fullsized":"2024-02-21T23:10:08.231962+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:15:42.134184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.38","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232805+0000","last_change":"2024-02-21T23:09:54.286729+0000","last_active":"2024-02-21T23:10:08.232805+0000","last_peered":"2024-02-21T23:10:08.232805+0000","last_clean":"2024-02-21T23:10:08.232805+0000","last_became_active":"2024-02-21T23:09:54.286427+0000","last_became_peered":"2024-02-21T23:09:54.286427+0000","last_unstale":"2024-02-21T23:10:08.232805+0000","last_undegraded":"2024-02-21T23:10:08.232805+0000","last_fullsized":"2024-02-21T23:10:08.232805+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:13:27.311923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.35","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.380954+0000","last_change":"2024-02-21T23:10:06.390905+0000","last_active":"2024-02-21T23:10:07.380954+0000","last_peered":"2024-02-21T23:10:07.380954+0000","last_clean":"2024-02-21T23:10:07.380954+0000","last_became_active":"2024-02-21T23:10:06.390622+0000","last_became_peered":"2024-02-21T23:10:06.390622+0000","last_unstale":"2024-02-21T23:10:07.380954+0000","last_undegraded":"2024-02-21T23:10:07.380954+0000","last_fullsized":"2024-02-21T23:10:07.380954+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:20:39.771135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.37","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.237533+0000","last_change":"2024-02-21T23:09:54.286612+0000","last_active":"2024-02-21T23:10:08.237533+0000","last_peered":"2024-02-21T23:10:08.237533+0000","last_clean":"2024-02-21T23:10:08.237533+0000","last_became_active":"2024-02-21T23:09:54.286306+0000","last_became_peered":"2024-02-21T23:09:54.286306+0000","last_unstale":"2024-02-21T23:10:08.237533+0000","last_undegraded":"2024-02-21T23:10:08.237533+0000","last_fullsized":"2024-02-21T23:10:08.237533+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:04:16.317089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.34","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232897+0000","last_change":"2024-02-21T23:10:06.385190+0000","last_active":"2024-02-21T23:10:08.232897+0000","last_peered":"2024-02-21T23:10:08.232897+0000","last_clean":"2024-02-21T23:10:08.232897+0000","last_became_active":"2024-02-21T23:10:06.384496+0000","last_became_peered":"2024-02-21T23:10:06.384496+0000","last_unstale":"2024-02-21T23:10:08.232897+0000","last_undegraded":"2024-02-21T23:10:08.232897+0000","last_fullsized":"2024-02-21T23:10:08.232897+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:28:15.352760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.36","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.237062+0000","last_change":"2024-02-21T23:09:54.288019+0000","last_active":"2024-02-21T23:10:08.237062+0000","last_peered":"2024-02-21T23:10:08.237062+0000","last_clean":"2024-02-21T23:10:08.237062+0000","last_became_active":"2024-02-21T23:09:54.287742+0000","last_became_peered":"2024-02-21T23:09:54.287742+0000","last_unstale":"2024-02-21T23:10:08.237062+0000","last_undegraded":"2024-02-21T23:10:08.237062+0000","last_fullsized":"2024-02-21T23:10:08.237062+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:13:33.442227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.37","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236970+0000","last_change":"2024-02-21T23:10:06.398025+0000","last_active":"2024-02-21T23:10:08.236970+0000","last_peered":"2024-02-21T23:10:08.236970+0000","last_clean":"2024-02-21T23:10:08.236970+0000","last_became_active":"2024-02-21T23:10:06.395723+0000","last_became_peered":"2024-02-21T23:10:06.395723+0000","last_unstale":"2024-02-21T23:10:08.236970+0000","last_undegraded":"2024-02-21T23:10:08.236970+0000","last_fullsized":"2024-02-21T23:10:08.236970+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:50:33.682312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.35","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.381341+0000","last_change":"2024-02-21T23:09:54.283918+0000","last_active":"2024-02-21T23:10:07.381341+0000","last_peered":"2024-02-21T23:10:07.381341+0000","last_clean":"2024-02-21T23:10:07.381341+0000","last_became_active":"2024-02-21T23:09:54.280599+0000","last_became_peered":"2024-02-21T23:09:54.280599+0000","last_unstale":"2024-02-21T23:10:07.381341+0000","last_undegraded":"2024-02-21T23:10:07.381341+0000","last_fullsized":"2024-02-21T23:10:07.381341+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:33:07.362294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.36","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234837+0000","last_change":"2024-02-21T23:10:06.377423+0000","last_active":"2024-02-21T23:10:08.234837+0000","last_peered":"2024-02-21T23:10:08.234837+0000","last_clean":"2024-02-21T23:10:08.234837+0000","last_became_active":"2024-02-21T23:10:06.376998+0000","last_became_peered":"2024-02-21T23:10:06.376998+0000","last_unstale":"2024-02-21T23:10:08.234837+0000","last_undegraded":"2024-02-21T23:10:08.234837+0000","last_fullsized":"2024-02-21T23:10:08.234837+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:26:32.419695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.34","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.610543+0000","last_change":"2024-02-21T23:09:54.282846+0000","last_active":"2024-02-21T23:10:07.610543+0000","last_peered":"2024-02-21T23:10:07.610543+0000","last_clean":"2024-02-21T23:10:07.610543+0000","last_became_active":"2024-02-21T23:09:54.282117+0000","last_became_peered":"2024-02-21T23:09:54.282117+0000","last_unstale":"2024-02-21T23:10:07.610543+0000","last_undegraded":"2024-02-21T23:10:07.610543+0000","last_fullsized":"2024-02-21T23:10:07.610543+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:33:07.362294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.31","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234391+0000","last_change":"2024-02-21T23:10:06.382496+0000","last_active":"2024-02-21T23:10:08.234391+0000","last_peered":"2024-02-21T23:10:08.234391+0000","last_clean":"2024-02-21T23:10:08.234391+0000","last_became_active":"2024-02-21T23:10:06.379727+0000","last_became_peered":"2024-02-21T23:10:06.379727+0000","last_unstale":"2024-02-21T23:10:08.234391+0000","last_undegraded":"2024-02-21T23:10:08.234391+0000","last_fullsized":"2024-02-21T23:10:08.234391+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:10:21.186085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.33","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234540+0000","last_change":"2024-02-21T23:09:54.284818+0000","last_active":"2024-02-21T23:10:08.234540+0000","last_peered":"2024-02-21T23:10:08.234540+0000","last_clean":"2024-02-21T23:10:08.234540+0000","last_became_active":"2024-02-21T23:09:54.284161+0000","last_became_peered":"2024-02-21T23:09:54.284161+0000","last_unstale":"2024-02-21T23:10:08.234540+0000","last_undegraded":"2024-02-21T23:10:08.234540+0000","last_fullsized":"2024-02-21T23:10:08.234540+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:02:12.340921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.30","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.231818+0000","last_change":"2024-02-21T23:10:06.388372+0000","last_active":"2024-02-21T23:10:08.231818+0000","last_peered":"2024-02-21T23:10:08.231818+0000","last_clean":"2024-02-21T23:10:08.231818+0000","last_became_active":"2024-02-21T23:10:06.388099+0000","last_became_peered":"2024-02-21T23:10:06.388099+0000","last_unstale":"2024-02-21T23:10:08.231818+0000","last_undegraded":"2024-02-21T23:10:08.231818+0000","last_fullsized":"2024-02-21T23:10:08.231818+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:55:32.267269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.32","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.231579+0000","last_change":"2024-02-21T23:09:54.272136+0000","last_active":"2024-02-21T23:10:08.231579+0000","last_peered":"2024-02-21T23:10:08.231579+0000","last_clean":"2024-02-21T23:10:08.231579+0000","last_became_active":"2024-02-21T23:09:54.271097+0000","last_became_peered":"2024-02-21T23:09:54.271097+0000","last_unstale":"2024-02-21T23:10:08.231579+0000","last_undegraded":"2024-02-21T23:10:08.231579+0000","last_fullsized":"2024-02-21T23:10:08.231579+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:30:45.437334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.33","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.230766+0000","last_change":"2024-02-21T23:10:06.383867+0000","last_active":"2024-02-21T23:10:08.230766+0000","last_peered":"2024-02-21T23:10:08.230766+0000","last_clean":"2024-02-21T23:10:08.230766+0000","last_became_active":"2024-02-21T23:10:06.382862+0000","last_became_peered":"2024-02-21T23:10:06.382862+0000","last_unstale":"2024-02-21T23:10:08.230766+0000","last_undegraded":"2024-02-21T23:10:08.230766+0000","last_fullsized":"2024-02-21T23:10:08.230766+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:30:57.541364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.31","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.380179+0000","last_change":"2024-02-21T23:09:54.281268+0000","last_active":"2024-02-21T23:10:07.380179+0000","last_peered":"2024-02-21T23:10:07.380179+0000","last_clean":"2024-02-21T23:10:07.380179+0000","last_became_active":"2024-02-21T23:09:54.280946+0000","last_became_peered":"2024-02-21T23:09:54.280946+0000","last_unstale":"2024-02-21T23:10:07.380179+0000","last_undegraded":"2024-02-21T23:10:07.380179+0000","last_fullsized":"2024-02-21T23:10:07.380179+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:03:44.493909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.32","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.374763+0000","last_change":"2024-02-21T23:10:06.381260+0000","last_active":"2024-02-21T23:10:07.374763+0000","last_peered":"2024-02-21T23:10:07.374763+0000","last_clean":"2024-02-21T23:10:07.374763+0000","last_became_active":"2024-02-21T23:10:06.380936+0000","last_became_peered":"2024-02-21T23:10:06.380936+0000","last_unstale":"2024-02-21T23:10:07.374763+0000","last_undegraded":"2024-02-21T23:10:07.374763+0000","last_fullsized":"2024-02-21T23:10:07.374763+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:41:14.290670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.30","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.230478+0000","last_change":"2024-02-21T23:09:54.278731+0000","last_active":"2024-02-21T23:10:08.230478+0000","last_peered":"2024-02-21T23:10:08.230478+0000","last_clean":"2024-02-21T23:10:08.230478+0000","last_became_active":"2024-02-21T23:09:54.278390+0000","last_became_peered":"2024-02-21T23:09:54.278390+0000","last_unstale":"2024-02-21T23:10:08.230478+0000","last_undegraded":"2024-02-21T23:10:08.230478+0000","last_fullsized":"2024-02-21T23:10:08.230478+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:28:54.079553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.2d","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.231262+0000","last_change":"2024-02-21T23:10:06.367099+0000","last_active":"2024-02-21T23:10:08.231262+0000","last_peered":"2024-02-21T23:10:08.231262+0000","last_clean":"2024-02-21T23:10:08.231262+0000","last_became_active":"2024-02-21T23:10:06.366296+0000","last_became_peered":"2024-02-21T23:10:06.366296+0000","last_unstale":"2024-02-21T23:10:08.231262+0000","last_undegraded":"2024-02-21T23:10:08.231262+0000","last_fullsized":"2024-02-21T23:10:08.231262+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:47:46.169071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.2f","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.237800+0000","last_change":"2024-02-21T23:09:54.285947+0000","last_active":"2024-02-21T23:10:08.237800+0000","last_peered":"2024-02-21T23:10:08.237800+0000","last_clean":"2024-02-21T23:10:08.237800+0000","last_became_active":"2024-02-21T23:09:54.285658+0000","last_became_peered":"2024-02-21T23:09:54.285658+0000","last_unstale":"2024-02-21T23:10:08.237800+0000","last_undegraded":"2024-02-21T23:10:08.237800+0000","last_fullsized":"2024-02-21T23:10:08.237800+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:20:31.125684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2c","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236540+0000","last_change":"2024-02-21T23:10:06.376864+0000","last_active":"2024-02-21T23:10:08.236540+0000","last_peered":"2024-02-21T23:10:08.236540+0000","last_clean":"2024-02-21T23:10:08.236540+0000","last_became_active":"2024-02-21T23:10:06.376486+0000","last_became_peered":"2024-02-21T23:10:06.376486+0000","last_unstale":"2024-02-21T23:10:08.236540+0000","last_undegraded":"2024-02-21T23:10:08.236540+0000","last_fullsized":"2024-02-21T23:10:08.236540+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:22:04.227521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.2e","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.379953+0000","last_change":"2024-02-21T23:09:54.287286+0000","last_active":"2024-02-21T23:10:07.379953+0000","last_peered":"2024-02-21T23:10:07.379953+0000","last_clean":"2024-02-21T23:10:07.379953+0000","last_became_active":"2024-02-21T23:09:54.286924+0000","last_became_peered":"2024-02-21T23:09:54.286924+0000","last_unstale":"2024-02-21T23:10:07.379953+0000","last_undegraded":"2024-02-21T23:10:07.379953+0000","last_fullsized":"2024-02-21T23:10:07.379953+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:47:29.349669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.2f","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.235953+0000","last_change":"2024-02-21T23:10:06.377113+0000","last_active":"2024-02-21T23:10:08.235953+0000","last_peered":"2024-02-21T23:10:08.235953+0000","last_clean":"2024-02-21T23:10:08.235953+0000","last_became_active":"2024-02-21T23:10:06.376790+0000","last_became_peered":"2024-02-21T23:10:06.376790+0000","last_unstale":"2024-02-21T23:10:08.235953+0000","last_undegraded":"2024-02-21T23:10:08.235953+0000","last_fullsized":"2024-02-21T23:10:08.235953+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:38:03.009039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.2d","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.381461+0000","last_change":"2024-02-21T23:09:54.279884+0000","last_active":"2024-02-21T23:10:07.381461+0000","last_peered":"2024-02-21T23:10:07.381461+0000","last_clean":"2024-02-21T23:10:07.381461+0000","last_became_active":"2024-02-21T23:09:54.276608+0000","last_became_peered":"2024-02-21T23:09:54.276608+0000","last_unstale":"2024-02-21T23:10:07.381461+0000","last_undegraded":"2024-02-21T23:10:07.381461+0000","last_fullsized":"2024-02-21T23:10:07.381461+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:05:09.189420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.2e","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.231704+0000","last_change":"2024-02-21T23:10:06.378194+0000","last_active":"2024-02-21T23:10:08.231704+0000","last_peered":"2024-02-21T23:10:08.231704+0000","last_clean":"2024-02-21T23:10:08.231704+0000","last_became_active":"2024-02-21T23:10:06.377905+0000","last_became_peered":"2024-02-21T23:10:06.377905+0000","last_unstale":"2024-02-21T23:10:08.231704+0000","last_undegraded":"2024-02-21T23:10:08.231704+0000","last_fullsized":"2024-02-21T23:10:08.231704+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:36:54.026578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.2c","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.375678+0000","last_change":"2024-02-21T23:09:54.288944+0000","last_active":"2024-02-21T23:10:07.375678+0000","last_peered":"2024-02-21T23:10:07.375678+0000","last_clean":"2024-02-21T23:10:07.375678+0000","last_became_active":"2024-02-21T23:09:54.285913+0000","last_became_peered":"2024-02-21T23:09:54.285913+0000","last_unstale":"2024-02-21T23:10:07.375678+0000","last_undegraded":"2024-02-21T23:10:07.375678+0000","last_fullsized":"2024-02-21T23:10:07.375678+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:39:33.609273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.29","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.610265+0000","last_change":"2024-02-21T23:10:06.385773+0000","last_active":"2024-02-21T23:10:07.610265+0000","last_peered":"2024-02-21T23:10:07.610265+0000","last_clean":"2024-02-21T23:10:07.610265+0000","last_became_active":"2024-02-21T23:10:06.385462+0000","last_became_peered":"2024-02-21T23:10:06.385462+0000","last_unstale":"2024-02-21T23:10:07.610265+0000","last_undegraded":"2024-02-21T23:10:07.610265+0000","last_fullsized":"2024-02-21T23:10:07.610265+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:47:42.794742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.2b","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234769+0000","last_change":"2024-02-21T23:09:54.285036+0000","last_active":"2024-02-21T23:10:08.234769+0000","last_peered":"2024-02-21T23:10:08.234769+0000","last_clean":"2024-02-21T23:10:08.234769+0000","last_became_active":"2024-02-21T23:09:54.284542+0000","last_became_peered":"2024-02-21T23:09:54.284542+0000","last_unstale":"2024-02-21T23:10:08.234769+0000","last_undegraded":"2024-02-21T23:10:08.234769+0000","last_fullsized":"2024-02-21T23:10:08.234769+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T11:05:01.241636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.28","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234521+0000","last_change":"2024-02-21T23:10:06.396769+0000","last_active":"2024-02-21T23:10:08.234521+0000","last_peered":"2024-02-21T23:10:08.234521+0000","last_clean":"2024-02-21T23:10:08.234521+0000","last_became_active":"2024-02-21T23:10:06.384723+0000","last_became_peered":"2024-02-21T23:10:06.384723+0000","last_unstale":"2024-02-21T23:10:08.234521+0000","last_undegraded":"2024-02-21T23:10:08.234521+0000","last_fullsized":"2024-02-21T23:10:08.234521+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:45:13.867812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2a","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.609154+0000","last_change":"2024-02-21T23:09:54.273350+0000","last_active":"2024-02-21T23:10:07.609154+0000","last_peered":"2024-02-21T23:10:07.609154+0000","last_clean":"2024-02-21T23:10:07.609154+0000","last_became_active":"2024-02-21T23:09:54.273022+0000","last_became_peered":"2024-02-21T23:09:54.273022+0000","last_unstale":"2024-02-21T23:10:07.609154+0000","last_undegraded":"2024-02-21T23:10:07.609154+0000","last_fullsized":"2024-02-21T23:10:07.609154+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:34:28.642087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.2b","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.235284+0000","last_change":"2024-02-21T23:10:06.394830+0000","last_active":"2024-02-21T23:10:08.235284+0000","last_peered":"2024-02-21T23:10:08.235284+0000","last_clean":"2024-02-21T23:10:08.235284+0000","last_became_active":"2024-02-21T23:10:06.386513+0000","last_became_peered":"2024-02-21T23:10:06.386513+0000","last_unstale":"2024-02-21T23:10:08.235284+0000","last_undegraded":"2024-02-21T23:10:08.235284+0000","last_fullsized":"2024-02-21T23:10:08.235284+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:30:09.767949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.29","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.230852+0000","last_change":"2024-02-21T23:09:54.279718+0000","last_active":"2024-02-21T23:10:08.230852+0000","last_peered":"2024-02-21T23:10:08.230852+0000","last_clean":"2024-02-21T23:10:08.230852+0000","last_became_active":"2024-02-21T23:09:54.278989+0000","last_became_peered":"2024-02-21T23:09:54.278989+0000","last_unstale":"2024-02-21T23:10:08.230852+0000","last_undegraded":"2024-02-21T23:10:08.230852+0000","last_fullsized":"2024-02-21T23:10:08.230852+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:36:41.922548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.2a","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.235326+0000","last_change":"2024-02-21T23:10:06.397414+0000","last_active":"2024-02-21T23:10:08.235326+0000","last_peered":"2024-02-21T23:10:08.235326+0000","last_clean":"2024-02-21T23:10:08.235326+0000","last_became_active":"2024-02-21T23:10:06.396982+0000","last_became_peered":"2024-02-21T23:10:06.396982+0000","last_unstale":"2024-02-21T23:10:08.235326+0000","last_undegraded":"2024-02-21T23:10:08.235326+0000","last_fullsized":"2024-02-21T23:10:08.235326+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:05:37.143681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.28","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.230583+0000","last_change":"2024-02-21T23:09:54.280624+0000","last_active":"2024-02-21T23:10:08.230583+0000","last_peered":"2024-02-21T23:10:08.230583+0000","last_clean":"2024-02-21T23:10:08.230583+0000","last_became_active":"2024-02-21T23:09:54.275644+0000","last_became_peered":"2024-02-21T23:09:54.275644+0000","last_unstale":"2024-02-21T23:10:08.230583+0000","last_undegraded":"2024-02-21T23:10:08.230583+0000","last_fullsized":"2024-02-21T23:10:08.230583+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:23:27.511315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.25","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.609295+0000","last_change":"2024-02-21T23:10:06.389786+0000","last_active":"2024-02-21T23:10:07.609295+0000","last_peered":"2024-02-21T23:10:07.609295+0000","last_clean":"2024-02-21T23:10:07.609295+0000","last_became_active":"2024-02-21T23:10:06.389447+0000","last_became_peered":"2024-02-21T23:10:06.389447+0000","last_unstale":"2024-02-21T23:10:07.609295+0000","last_undegraded":"2024-02-21T23:10:07.609295+0000","last_fullsized":"2024-02-21T23:10:07.609295+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:50:58.576033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.27","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236738+0000","last_change":"2024-02-21T23:09:54.275341+0000","last_active":"2024-02-21T23:10:08.236738+0000","last_peered":"2024-02-21T23:10:08.236738+0000","last_clean":"2024-02-21T23:10:08.236738+0000","last_became_active":"2024-02-21T23:09:54.275010+0000","last_became_peered":"2024-02-21T23:09:54.275010+0000","last_unstale":"2024-02-21T23:10:08.236738+0000","last_undegraded":"2024-02-21T23:10:08.236738+0000","last_fullsized":"2024-02-21T23:10:08.236738+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:42:08.123188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.24","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236163+0000","last_change":"2024-02-21T23:10:06.386616+0000","last_active":"2024-02-21T23:10:08.236163+0000","last_peered":"2024-02-21T23:10:08.236163+0000","last_clean":"2024-02-21T23:10:08.236163+0000","last_became_active":"2024-02-21T23:10:06.386339+0000","last_became_peered":"2024-02-21T23:10:06.386339+0000","last_unstale":"2024-02-21T23:10:08.236163+0000","last_undegraded":"2024-02-21T23:10:08.236163+0000","last_fullsized":"2024-02-21T23:10:08.236163+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:44:41.892333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.26","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.376034+0000","last_change":"2024-02-21T23:09:54.260902+0000","last_active":"2024-02-21T23:10:07.376034+0000","last_peered":"2024-02-21T23:10:07.376034+0000","last_clean":"2024-02-21T23:10:07.376034+0000","last_became_active":"2024-02-21T23:09:54.260084+0000","last_became_peered":"2024-02-21T23:09:54.260084+0000","last_unstale":"2024-02-21T23:10:07.376034+0000","last_undegraded":"2024-02-21T23:10:07.376034+0000","last_fullsized":"2024-02-21T23:10:07.376034+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:04:05.858817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.27","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.375938+0000","last_change":"2024-02-21T23:10:06.374537+0000","last_active":"2024-02-21T23:10:07.375938+0000","last_peered":"2024-02-21T23:10:07.375938+0000","last_clean":"2024-02-21T23:10:07.375938+0000","last_became_active":"2024-02-21T23:10:06.374201+0000","last_became_peered":"2024-02-21T23:10:06.374201+0000","last_unstale":"2024-02-21T23:10:07.375938+0000","last_undegraded":"2024-02-21T23:10:07.375938+0000","last_fullsized":"2024-02-21T23:10:07.375938+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:52:55.561353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.25","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.609643+0000","last_change":"2024-02-21T23:09:54.284116+0000","last_active":"2024-02-21T23:10:07.609643+0000","last_peered":"2024-02-21T23:10:07.609643+0000","last_clean":"2024-02-21T23:10:07.609643+0000","last_became_active":"2024-02-21T23:09:54.283802+0000","last_became_peered":"2024-02-21T23:09:54.283802+0000","last_unstale":"2024-02-21T23:10:07.609643+0000","last_undegraded":"2024-02-21T23:10:07.609643+0000","last_fullsized":"2024-02-21T23:10:07.609643+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:05:09.189420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.26","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.380096+0000","last_change":"2024-02-21T23:10:06.393845+0000","last_active":"2024-02-21T23:10:07.380096+0000","last_peered":"2024-02-21T23:10:07.380096+0000","last_clean":"2024-02-21T23:10:07.380096+0000","last_became_active":"2024-02-21T23:10:06.391431+0000","last_became_peered":"2024-02-21T23:10:06.391431+0000","last_unstale":"2024-02-21T23:10:07.380096+0000","last_undegraded":"2024-02-21T23:10:07.380096+0000","last_fullsized":"2024-02-21T23:10:07.380096+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:26:48.509050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.24","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236346+0000","last_change":"2024-02-21T23:09:54.287481+0000","last_active":"2024-02-21T23:10:08.236346+0000","last_peered":"2024-02-21T23:10:08.236346+0000","last_clean":"2024-02-21T23:10:08.236346+0000","last_became_active":"2024-02-21T23:09:54.287137+0000","last_became_peered":"2024-02-21T23:09:54.287137+0000","last_unstale":"2024-02-21T23:10:08.236346+0000","last_undegraded":"2024-02-21T23:10:08.236346+0000","last_fullsized":"2024-02-21T23:10:08.236346+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:50:46.472002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.21","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.374988+0000","last_change":"2024-02-21T23:10:06.370867+0000","last_active":"2024-02-21T23:10:07.374988+0000","last_peered":"2024-02-21T23:10:07.374988+0000","last_clean":"2024-02-21T23:10:07.374988+0000","last_became_active":"2024-02-21T23:10:06.370583+0000","last_became_peered":"2024-02-21T23:10:06.370583+0000","last_unstale":"2024-02-21T23:10:07.374988+0000","last_undegraded":"2024-02-21T23:10:07.374988+0000","last_fullsized":"2024-02-21T23:10:07.374988+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:18:29.588662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.23","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.610263+0000","last_change":"2024-02-21T23:09:54.281613+0000","last_active":"2024-02-21T23:10:07.610263+0000","last_peered":"2024-02-21T23:10:07.610263+0000","last_clean":"2024-02-21T23:10:07.610263+0000","last_became_active":"2024-02-21T23:09:54.281316+0000","last_became_peered":"2024-02-21T23:09:54.281316+0000","last_unstale":"2024-02-21T23:10:07.610263+0000","last_undegraded":"2024-02-21T23:10:07.610263+0000","last_fullsized":"2024-02-21T23:10:07.610263+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:13:33.442227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.20","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232111+0000","last_change":"2024-02-21T23:10:06.382019+0000","last_active":"2024-02-21T23:10:08.232111+0000","last_peered":"2024-02-21T23:10:08.232111+0000","last_clean":"2024-02-21T23:10:08.232111+0000","last_became_active":"2024-02-21T23:10:06.381739+0000","last_became_peered":"2024-02-21T23:10:06.381739+0000","last_unstale":"2024-02-21T23:10:08.232111+0000","last_undegraded":"2024-02-21T23:10:08.232111+0000","last_fullsized":"2024-02-21T23:10:08.232111+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:25:17.018194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.22","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.379703+0000","last_change":"2024-02-21T23:09:54.275665+0000","last_active":"2024-02-21T23:10:07.379703+0000","last_peered":"2024-02-21T23:10:07.379703+0000","last_clean":"2024-02-21T23:10:07.379703+0000","last_became_active":"2024-02-21T23:09:54.275143+0000","last_became_peered":"2024-02-21T23:09:54.275143+0000","last_unstale":"2024-02-21T23:10:07.379703+0000","last_undegraded":"2024-02-21T23:10:07.379703+0000","last_fullsized":"2024-02-21T23:10:07.379703+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:37:42.976792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.23","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.380797+0000","last_change":"2024-02-21T23:10:06.398551+0000","last_active":"2024-02-21T23:10:07.380797+0000","last_peered":"2024-02-21T23:10:07.380797+0000","last_clean":"2024-02-21T23:10:07.380797+0000","last_became_active":"2024-02-21T23:10:06.397827+0000","last_became_peered":"2024-02-21T23:10:06.397827+0000","last_unstale":"2024-02-21T23:10:07.380797+0000","last_undegraded":"2024-02-21T23:10:07.380797+0000","last_fullsized":"2024-02-21T23:10:07.380797+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:42:20.227218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.21","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.380298+0000","last_change":"2024-02-21T23:09:54.281200+0000","last_active":"2024-02-21T23:10:07.380298+0000","last_peered":"2024-02-21T23:10:07.380298+0000","last_clean":"2024-02-21T23:10:07.380298+0000","last_became_active":"2024-02-21T23:09:54.277867+0000","last_became_peered":"2024-02-21T23:09:54.277867+0000","last_unstale":"2024-02-21T23:10:07.380298+0000","last_undegraded":"2024-02-21T23:10:07.380298+0000","last_fullsized":"2024-02-21T23:10:07.380298+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:59:34.646775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.22","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.381831+0000","last_change":"2024-02-21T23:10:06.398306+0000","last_active":"2024-02-21T23:10:07.381831+0000","last_peered":"2024-02-21T23:10:07.381831+0000","last_clean":"2024-02-21T23:10:07.381831+0000","last_became_active":"2024-02-21T23:10:06.397841+0000","last_became_peered":"2024-02-21T23:10:06.397841+0000","last_unstale":"2024-02-21T23:10:07.381831+0000","last_undegraded":"2024-02-21T23:10:07.381831+0000","last_fullsized":"2024-02-21T23:10:07.381831+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:06:55.637480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.20","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.379915+0000","last_change":"2024-02-21T23:09:54.276078+0000","last_active":"2024-02-21T23:10:07.379915+0000","last_peered":"2024-02-21T23:10:07.379915+0000","last_clean":"2024-02-21T23:10:07.379915+0000","last_became_active":"2024-02-21T23:09:54.275724+0000","last_became_peered":"2024-02-21T23:09:54.275724+0000","last_unstale":"2024-02-21T23:10:07.379915+0000","last_undegraded":"2024-02-21T23:10:07.379915+0000","last_fullsized":"2024-02-21T23:10:07.379915+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:48:07.369182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.230483+0000","last_change":"2024-02-21T23:10:06.380827+0000","last_active":"2024-02-21T23:10:08.230483+0000","last_peered":"2024-02-21T23:10:08.230483+0000","last_clean":"2024-02-21T23:10:08.230483+0000","last_became_active":"2024-02-21T23:10:06.380551+0000","last_became_peered":"2024-02-21T23:10:06.380551+0000","last_unstale":"2024-02-21T23:10:08.230483+0000","last_undegraded":"2024-02-21T23:10:08.230483+0000","last_fullsized":"2024-02-21T23:10:08.230483+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:34:11.889581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.230111+0000","last_change":"2024-02-21T23:10:05.385230+0000","last_active":"2024-02-21T23:10:08.230111+0000","last_peered":"2024-02-21T23:10:08.230111+0000","last_clean":"2024-02-21T23:10:08.230111+0000","last_became_active":"2024-02-21T23:10:05.384017+0000","last_became_peered":"2024-02-21T23:10:05.384017+0000","last_unstale":"2024-02-21T23:10:08.230111+0000","last_undegraded":"2024-02-21T23:10:08.230111+0000","last_fullsized":"2024-02-21T23:10:08.230111+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:45:12.852122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.10","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.609579+0000","last_change":"2024-02-21T23:09:54.278972+0000","last_active":"2024-02-21T23:10:07.609579+0000","last_peered":"2024-02-21T23:10:07.609579+0000","last_clean":"2024-02-21T23:10:07.609579+0000","last_became_active":"2024-02-21T23:09:54.278146+0000","last_became_peered":"2024-02-21T23:09:54.278146+0000","last_unstale":"2024-02-21T23:10:07.609579+0000","last_undegraded":"2024-02-21T23:10:07.609579+0000","last_fullsized":"2024-02-21T23:10:07.609579+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:59:34.646775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"150'4","reported_seq":26,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232604+0000","last_change":"2024-02-21T23:09:53.288506+0000","last_active":"2024-02-21T23:10:08.232604+0000","last_peered":"2024-02-21T23:10:08.232604+0000","last_clean":"2024-02-21T23:10:08.232604+0000","last_became_active":"2024-02-21T23:09:53.288199+0000","last_became_peered":"2024-02-21T23:09:53.288199+0000","last_unstale":"2024-02-21T23:10:08.232604+0000","last_undegraded":"2024-02-21T23:10:08.232604+0000","last_fullsized":"2024-02-21T23:10:08.232604+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:18:16.467715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.13","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232017+0000","last_change":"2024-02-21T23:10:06.396696+0000","last_active":"2024-02-21T23:10:08.232017+0000","last_peered":"2024-02-21T23:10:08.232017+0000","last_clean":"2024-02-21T23:10:08.232017+0000","last_became_active":"2024-02-21T23:10:06.396394+0000","last_became_peered":"2024-02-21T23:10:06.396394+0000","last_unstale":"2024-02-21T23:10:08.232017+0000","last_undegraded":"2024-02-21T23:10:08.232017+0000","last_fullsized":"2024-02-21T23:10:08.232017+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:40:04.184614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.10","version":"154'8","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.380094+0000","last_change":"2024-02-21T23:10:05.382619+0000","last_active":"2024-02-21T23:10:07.380094+0000","last_peered":"2024-02-21T23:10:07.380094+0000","last_clean":"2024-02-21T23:10:07.380094+0000","last_became_active":"2024-02-21T23:10:05.382336+0000","last_became_peered":"2024-02-21T23:10:05.382336+0000","last_unstale":"2024-02-21T23:10:07.380094+0000","last_undegraded":"2024-02-21T23:10:07.380094+0000","last_fullsized":"2024-02-21T23:10:07.380094+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:30:56.525674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.231615+0000","last_change":"2024-02-21T23:09:54.278032+0000","last_active":"2024-02-21T23:10:08.231615+0000","last_peered":"2024-02-21T23:10:08.231615+0000","last_clean":"2024-02-21T23:10:08.231615+0000","last_became_active":"2024-02-21T23:09:54.277741+0000","last_became_peered":"2024-02-21T23:09:54.277741+0000","last_unstale":"2024-02-21T23:10:08.231615+0000","last_undegraded":"2024-02-21T23:10:08.231615+0000","last_fullsized":"2024-02-21T23:10:08.231615+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:43:56.610613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"150'1","reported_seq":23,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232900+0000","last_change":"2024-02-21T23:09:53.279374+0000","last_active":"2024-02-21T23:10:08.232900+0000","last_peered":"2024-02-21T23:10:08.232900+0000","last_clean":"2024-02-21T23:10:08.232900+0000","last_became_active":"2024-02-21T23:09:53.279053+0000","last_became_peered":"2024-02-21T23:09:53.279053+0000","last_unstale":"2024-02-21T23:10:08.232900+0000","last_undegraded":"2024-02-21T23:10:08.232900+0000","last_fullsized":"2024-02-21T23:10:08.232900+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:04:15.300172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.10","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.375571+0000","last_change":"2024-02-21T23:10:06.375103+0000","last_active":"2024-02-21T23:10:07.375571+0000","last_peered":"2024-02-21T23:10:07.375571+0000","last_clean":"2024-02-21T23:10:07.375571+0000","last_became_active":"2024-02-21T23:10:06.374814+0000","last_became_peered":"2024-02-21T23:10:06.374814+0000","last_unstale":"2024-02-21T23:10:07.375571+0000","last_undegraded":"2024-02-21T23:10:07.375571+0000","last_fullsized":"2024-02-21T23:10:07.375571+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:42:32.100185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.13","version":"154'2","reported_seq":16,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.231271+0000","last_change":"2024-02-21T23:10:05.392497+0000","last_active":"2024-02-21T23:10:08.231271+0000","last_peered":"2024-02-21T23:10:08.231271+0000","last_clean":"2024-02-21T23:10:08.231271+0000","last_became_active":"2024-02-21T23:10:05.392183+0000","last_became_peered":"2024-02-21T23:10:05.392183+0000","last_unstale":"2024-02-21T23:10:08.231271+0000","last_undegraded":"2024-02-21T23:10:08.231271+0000","last_fullsized":"2024-02-21T23:10:08.231271+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:13:38.400263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.231985+0000","last_change":"2024-02-21T23:09:54.263915+0000","last_active":"2024-02-21T23:10:08.231985+0000","last_peered":"2024-02-21T23:10:08.231985+0000","last_clean":"2024-02-21T23:10:08.231985+0000","last_became_active":"2024-02-21T23:09:54.263634+0000","last_became_peered":"2024-02-21T23:09:54.263634+0000","last_unstale":"2024-02-21T23:10:08.231985+0000","last_undegraded":"2024-02-21T23:10:08.231985+0000","last_fullsized":"2024-02-21T23:10:08.231985+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:47:34.065041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.1d","version":"150'2","reported_seq":24,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.381587+0000","last_change":"2024-02-21T23:09:53.272110+0000","last_active":"2024-02-21T23:10:07.381587+0000","last_peered":"2024-02-21T23:10:07.381587+0000","last_clean":"2024-02-21T23:10:07.381587+0000","last_became_active":"2024-02-21T23:09:53.271641+0000","last_became_peered":"2024-02-21T23:09:53.271641+0000","last_unstale":"2024-02-21T23:10:07.381587+0000","last_undegraded":"2024-02-21T23:10:07.381587+0000","last_fullsized":"2024-02-21T23:10:07.381587+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:46:34.419898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.231438+0000","last_change":"2024-02-21T23:10:06.380458+0000","last_active":"2024-02-21T23:10:08.231438+0000","last_peered":"2024-02-21T23:10:08.231438+0000","last_clean":"2024-02-21T23:10:08.231438+0000","last_became_active":"2024-02-21T23:10:06.380137+0000","last_became_peered":"2024-02-21T23:10:06.380137+0000","last_unstale":"2024-02-21T23:10:08.231438+0000","last_undegraded":"2024-02-21T23:10:08.231438+0000","last_fullsized":"2024-02-21T23:10:08.231438+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:51:13.639406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236172+0000","last_change":"2024-02-21T23:10:05.393823+0000","last_active":"2024-02-21T23:10:08.236172+0000","last_peered":"2024-02-21T23:10:08.236172+0000","last_clean":"2024-02-21T23:10:08.236172+0000","last_became_active":"2024-02-21T23:10:05.390445+0000","last_became_peered":"2024-02-21T23:10:05.390445+0000","last_unstale":"2024-02-21T23:10:08.236172+0000","last_undegraded":"2024-02-21T23:10:08.236172+0000","last_fullsized":"2024-02-21T23:10:08.236172+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:30:56.525674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.13","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.610343+0000","last_change":"2024-02-21T23:09:54.275867+0000","last_active":"2024-02-21T23:10:07.610343+0000","last_peered":"2024-02-21T23:10:07.610343+0000","last_clean":"2024-02-21T23:10:07.610343+0000","last_became_active":"2024-02-21T23:09:54.275591+0000","last_became_peered":"2024-02-21T23:09:54.275591+0000","last_unstale":"2024-02-21T23:10:07.610343+0000","last_undegraded":"2024-02-21T23:10:07.610343+0000","last_fullsized":"2024-02-21T23:10:07.610343+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:42:16.988326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1c","version":"150'2","reported_seq":24,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.375729+0000","last_change":"2024-02-21T23:09:53.274692+0000","last_active":"2024-02-21T23:10:07.375729+0000","last_peered":"2024-02-21T23:10:07.375729+0000","last_clean":"2024-02-21T23:10:07.375729+0000","last_became_active":"2024-02-21T23:09:53.274395+0000","last_became_peered":"2024-02-21T23:09:53.274395+0000","last_unstale":"2024-02-21T23:10:07.375729+0000","last_undegraded":"2024-02-21T23:10:07.375729+0000","last_fullsized":"2024-02-21T23:10:07.375729+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:40:18.453921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.231853+0000","last_change":"2024-02-21T23:10:06.384222+0000","last_active":"2024-02-21T23:10:08.231853+0000","last_peered":"2024-02-21T23:10:08.231853+0000","last_clean":"2024-02-21T23:10:08.231853+0000","last_became_active":"2024-02-21T23:10:06.383621+0000","last_became_peered":"2024-02-21T23:10:06.383621+0000","last_unstale":"2024-02-21T23:10:08.231853+0000","last_undegraded":"2024-02-21T23:10:08.231853+0000","last_fullsized":"2024-02-21T23:10:08.231853+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:16:22.900582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.15","version":"154'3","reported_seq":17,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.233612+0000","last_change":"2024-02-21T23:10:05.374604+0000","last_active":"2024-02-21T23:10:08.233612+0000","last_peered":"2024-02-21T23:10:08.233612+0000","last_clean":"2024-02-21T23:10:08.233612+0000","last_became_active":"2024-02-21T23:10:05.374208+0000","last_became_peered":"2024-02-21T23:10:05.374208+0000","last_unstale":"2024-02-21T23:10:08.233612+0000","last_undegraded":"2024-02-21T23:10:08.233612+0000","last_fullsized":"2024-02-21T23:10:08.233612+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:30:08.752259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.14","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.235541+0000","last_change":"2024-02-21T23:09:54.289092+0000","last_active":"2024-02-21T23:10:08.235541+0000","last_peered":"2024-02-21T23:10:08.235541+0000","last_clean":"2024-02-21T23:10:08.235541+0000","last_became_active":"2024-02-21T23:09:54.288801+0000","last_became_peered":"2024-02-21T23:09:54.288801+0000","last_unstale":"2024-02-21T23:10:08.235541+0000","last_undegraded":"2024-02-21T23:10:08.235541+0000","last_fullsized":"2024-02-21T23:10:08.235541+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:05:09.189420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1b","version":"150'1","reported_seq":23,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236011+0000","last_change":"2024-02-21T23:09:53.283422+0000","last_active":"2024-02-21T23:10:08.236011+0000","last_peered":"2024-02-21T23:10:08.236011+0000","last_clean":"2024-02-21T23:10:08.236011+0000","last_became_active":"2024-02-21T23:09:53.283140+0000","last_became_peered":"2024-02-21T23:09:53.283140+0000","last_unstale":"2024-02-21T23:10:08.236011+0000","last_undegraded":"2024-02-21T23:10:08.236011+0000","last_fullsized":"2024-02-21T23:10:08.236011+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:46:34.419898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.237280+0000","last_change":"2024-02-21T23:10:06.403933+0000","last_active":"2024-02-21T23:10:08.237280+0000","last_peered":"2024-02-21T23:10:08.237280+0000","last_clean":"2024-02-21T23:10:08.237280+0000","last_became_active":"2024-02-21T23:10:06.403548+0000","last_became_peered":"2024-02-21T23:10:06.403548+0000","last_unstale":"2024-02-21T23:10:08.237280+0000","last_undegraded":"2024-02-21T23:10:08.237280+0000","last_fullsized":"2024-02-21T23:10:08.237280+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:11:20.176324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.14","version":"154'6","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.230908+0000","last_change":"2024-02-21T23:10:05.388142+0000","last_active":"2024-02-21T23:10:08.230908+0000","last_peered":"2024-02-21T23:10:08.230908+0000","last_clean":"2024-02-21T23:10:08.230908+0000","last_became_active":"2024-02-21T23:10:05.387764+0000","last_became_peered":"2024-02-21T23:10:05.387764+0000","last_unstale":"2024-02-21T23:10:08.230908+0000","last_undegraded":"2024-02-21T23:10:08.230908+0000","last_fullsized":"2024-02-21T23:10:08.230908+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:03:55.582249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.15","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.375532+0000","last_change":"2024-02-21T23:09:54.273728+0000","last_active":"2024-02-21T23:10:07.375532+0000","last_peered":"2024-02-21T23:10:07.375532+0000","last_clean":"2024-02-21T23:10:07.375532+0000","last_became_active":"2024-02-21T23:09:54.273343+0000","last_became_peered":"2024-02-21T23:09:54.273343+0000","last_unstale":"2024-02-21T23:10:07.375532+0000","last_undegraded":"2024-02-21T23:10:07.375532+0000","last_fullsized":"2024-02-21T23:10:07.375532+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:49:15.552003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.1a","version":"150'1","reported_seq":23,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232153+0000","last_change":"2024-02-21T23:09:53.278176+0000","last_active":"2024-02-21T23:10:08.232153+0000","last_peered":"2024-02-21T23:10:08.232153+0000","last_clean":"2024-02-21T23:10:08.232153+0000","last_became_active":"2024-02-21T23:09:53.277880+0000","last_became_peered":"2024-02-21T23:09:53.277880+0000","last_unstale":"2024-02-21T23:10:08.232153+0000","last_undegraded":"2024-02-21T23:10:08.232153+0000","last_fullsized":"2024-02-21T23:10:08.232153+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:05:08.172503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.14","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.610388+0000","last_change":"2024-02-21T23:10:06.397559+0000","last_active":"2024-02-21T23:10:07.610388+0000","last_peered":"2024-02-21T23:10:07.610388+0000","last_clean":"2024-02-21T23:10:07.610388+0000","last_became_active":"2024-02-21T23:10:06.396499+0000","last_became_peered":"2024-02-21T23:10:06.396499+0000","last_unstale":"2024-02-21T23:10:07.610388+0000","last_undegraded":"2024-02-21T23:10:07.610388+0000","last_fullsized":"2024-02-21T23:10:07.610388+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:20:39.771135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236121+0000","last_change":"2024-02-21T23:10:05.375277+0000","last_active":"2024-02-21T23:10:08.236121+0000","last_peered":"2024-02-21T23:10:08.236121+0000","last_clean":"2024-02-21T23:10:08.236121+0000","last_became_active":"2024-02-21T23:10:05.374983+0000","last_became_peered":"2024-02-21T23:10:05.374983+0000","last_unstale":"2024-02-21T23:10:08.236121+0000","last_undegraded":"2024-02-21T23:10:08.236121+0000","last_fullsized":"2024-02-21T23:10:08.236121+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:11:19.160634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.376102+0000","last_change":"2024-02-21T23:09:54.278094+0000","last_active":"2024-02-21T23:10:07.376102+0000","last_peered":"2024-02-21T23:10:07.376102+0000","last_clean":"2024-02-21T23:10:07.376102+0000","last_became_active":"2024-02-21T23:09:54.276160+0000","last_became_peered":"2024-02-21T23:09:54.276160+0000","last_unstale":"2024-02-21T23:10:07.376102+0000","last_undegraded":"2024-02-21T23:10:07.376102+0000","last_fullsized":"2024-02-21T23:10:07.376102+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:15:07.339478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.375244+0000","last_change":"2024-02-21T23:09:53.280005+0000","last_active":"2024-02-21T23:10:07.375244+0000","last_peered":"2024-02-21T23:10:07.375244+0000","last_clean":"2024-02-21T23:10:07.375244+0000","last_became_active":"2024-02-21T23:09:53.279689+0000","last_became_peered":"2024-02-21T23:09:53.279689+0000","last_unstale":"2024-02-21T23:10:07.375244+0000","last_undegraded":"2024-02-21T23:10:07.375244+0000","last_fullsized":"2024-02-21T23:10:07.375244+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:40:05.561558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.15","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234855+0000","last_change":"2024-02-21T23:10:06.384066+0000","last_active":"2024-02-21T23:10:08.234855+0000","last_peered":"2024-02-21T23:10:08.234855+0000","last_clean":"2024-02-21T23:10:08.234855+0000","last_became_active":"2024-02-21T23:10:06.383780+0000","last_became_peered":"2024-02-21T23:10:06.383780+0000","last_unstale":"2024-02-21T23:10:08.234855+0000","last_undegraded":"2024-02-21T23:10:08.234855+0000","last_fullsized":"2024-02-21T23:10:08.234855+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:24:37.460238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236590+0000","last_change":"2024-02-21T23:10:05.393045+0000","last_active":"2024-02-21T23:10:08.236590+0000","last_peered":"2024-02-21T23:10:08.236590+0000","last_clean":"2024-02-21T23:10:08.236590+0000","last_became_active":"2024-02-21T23:10:05.392683+0000","last_became_peered":"2024-02-21T23:10:05.392683+0000","last_unstale":"2024-02-21T23:10:08.236590+0000","last_undegraded":"2024-02-21T23:10:08.236590+0000","last_fullsized":"2024-02-21T23:10:08.236590+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:45:39.918602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232441+0000","last_change":"2024-02-21T23:09:54.262565+0000","last_active":"2024-02-21T23:10:08.232441+0000","last_peered":"2024-02-21T23:10:08.232441+0000","last_clean":"2024-02-21T23:10:08.232441+0000","last_became_active":"2024-02-21T23:09:54.261897+0000","last_became_peered":"2024-02-21T23:09:54.261897+0000","last_unstale":"2024-02-21T23:10:08.232441+0000","last_undegraded":"2024-02-21T23:10:08.232441+0000","last_fullsized":"2024-02-21T23:10:08.232441+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:08:50.519955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234939+0000","last_change":"2024-02-21T23:09:53.275695+0000","last_active":"2024-02-21T23:10:08.234939+0000","last_peered":"2024-02-21T23:10:08.234939+0000","last_clean":"2024-02-21T23:10:08.234939+0000","last_became_active":"2024-02-21T23:09:53.275343+0000","last_became_peered":"2024-02-21T23:09:53.275343+0000","last_unstale":"2024-02-21T23:10:08.234939+0000","last_undegraded":"2024-02-21T23:10:08.234939+0000","last_fullsized":"2024-02-21T23:10:08.234939+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:28:53.062636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1a","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.235787+0000","last_change":"2024-02-21T23:10:06.403716+0000","last_active":"2024-02-21T23:10:08.235787+0000","last_peered":"2024-02-21T23:10:08.235787+0000","last_clean":"2024-02-21T23:10:08.235787+0000","last_became_active":"2024-02-21T23:10:06.403226+0000","last_became_peered":"2024-02-21T23:10:06.403226+0000","last_unstale":"2024-02-21T23:10:08.235787+0000","last_undegraded":"2024-02-21T23:10:08.235787+0000","last_fullsized":"2024-02-21T23:10:08.235787+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:28:15.518153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.230984+0000","last_change":"2024-02-21T23:10:05.390846+0000","last_active":"2024-02-21T23:10:08.230984+0000","last_peered":"2024-02-21T23:10:08.230984+0000","last_clean":"2024-02-21T23:10:08.230984+0000","last_became_active":"2024-02-21T23:10:05.388088+0000","last_became_peered":"2024-02-21T23:10:05.388088+0000","last_unstale":"2024-02-21T23:10:08.230984+0000","last_undegraded":"2024-02-21T23:10:08.230984+0000","last_fullsized":"2024-02-21T23:10:08.230984+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:59:45.735115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.608719+0000","last_change":"2024-02-21T23:09:54.279220+0000","last_active":"2024-02-21T23:10:07.608719+0000","last_peered":"2024-02-21T23:10:07.608719+0000","last_clean":"2024-02-21T23:10:07.608719+0000","last_became_active":"2024-02-21T23:09:54.278653+0000","last_became_peered":"2024-02-21T23:09:54.278653+0000","last_unstale":"2024-02-21T23:10:07.608719+0000","last_undegraded":"2024-02-21T23:10:07.608719+0000","last_fullsized":"2024-02-21T23:10:07.608719+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:13:27.311923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236554+0000","last_change":"2024-02-21T23:09:53.288439+0000","last_active":"2024-02-21T23:10:08.236554+0000","last_peered":"2024-02-21T23:10:08.236554+0000","last_clean":"2024-02-21T23:10:08.236554+0000","last_became_active":"2024-02-21T23:09:53.287601+0000","last_became_peered":"2024-02-21T23:09:53.287601+0000","last_unstale":"2024-02-21T23:10:08.236554+0000","last_undegraded":"2024-02-21T23:10:08.236554+0000","last_fullsized":"2024-02-21T23:10:08.236554+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:32:05.045159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.230304+0000","last_change":"2024-02-21T23:10:06.373866+0000","last_active":"2024-02-21T23:10:08.230304+0000","last_peered":"2024-02-21T23:10:08.230304+0000","last_clean":"2024-02-21T23:10:08.230304+0000","last_became_active":"2024-02-21T23:10:06.373575+0000","last_became_peered":"2024-02-21T23:10:06.373575+0000","last_unstale":"2024-02-21T23:10:08.230304+0000","last_undegraded":"2024-02-21T23:10:08.230304+0000","last_fullsized":"2024-02-21T23:10:08.230304+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:11:56.693636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.1b","version":"154'1","reported_seq":15,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.233920+0000","last_change":"2024-02-21T23:10:05.387544+0000","last_active":"2024-02-21T23:10:08.233920+0000","last_peered":"2024-02-21T23:10:08.233920+0000","last_clean":"2024-02-21T23:10:08.233920+0000","last_became_active":"2024-02-21T23:10:05.385992+0000","last_became_peered":"2024-02-21T23:10:05.385992+0000","last_unstale":"2024-02-21T23:10:08.233920+0000","last_undegraded":"2024-02-21T23:10:08.233920+0000","last_fullsized":"2024-02-21T23:10:08.233920+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:28:14.502463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232148+0000","last_change":"2024-02-21T23:09:54.280274+0000","last_active":"2024-02-21T23:10:08.232148+0000","last_peered":"2024-02-21T23:10:08.232148+0000","last_clean":"2024-02-21T23:10:08.232148+0000","last_became_active":"2024-02-21T23:09:54.279951+0000","last_became_peered":"2024-02-21T23:09:54.279951+0000","last_unstale":"2024-02-21T23:10:08.232148+0000","last_undegraded":"2024-02-21T23:10:08.232148+0000","last_fullsized":"2024-02-21T23:10:08.232148+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:15:08.084683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.15","version":"150'3","reported_seq":25,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232015+0000","last_change":"2024-02-21T23:09:53.279389+0000","last_active":"2024-02-21T23:10:08.232015+0000","last_peered":"2024-02-21T23:10:08.232015+0000","last_clean":"2024-02-21T23:10:08.232015+0000","last_became_active":"2024-02-21T23:09:53.279042+0000","last_became_peered":"2024-02-21T23:09:53.279042+0000","last_unstale":"2024-02-21T23:10:08.232015+0000","last_undegraded":"2024-02-21T23:10:08.232015+0000","last_fullsized":"2024-02-21T23:10:08.232015+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:44:44.898339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.233266+0000","last_change":"2024-02-21T23:10:06.379758+0000","last_active":"2024-02-21T23:10:08.233266+0000","last_peered":"2024-02-21T23:10:08.233266+0000","last_clean":"2024-02-21T23:10:08.233266+0000","last_became_active":"2024-02-21T23:10:06.379416+0000","last_became_peered":"2024-02-21T23:10:06.379416+0000","last_unstale":"2024-02-21T23:10:08.233266+0000","last_undegraded":"2024-02-21T23:10:08.233266+0000","last_fullsized":"2024-02-21T23:10:08.233266+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:06:55.637480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1a","version":"154'1","reported_seq":15,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.231363+0000","last_change":"2024-02-21T23:10:05.386289+0000","last_active":"2024-02-21T23:10:08.231363+0000","last_peered":"2024-02-21T23:10:08.231363+0000","last_clean":"2024-02-21T23:10:08.231363+0000","last_became_active":"2024-02-21T23:10:05.384200+0000","last_became_peered":"2024-02-21T23:10:05.384200+0000","last_unstale":"2024-02-21T23:10:08.231363+0000","last_undegraded":"2024-02-21T23:10:08.231363+0000","last_fullsized":"2024-02-21T23:10:08.231363+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:24:31.837524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1b","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.375019+0000","last_change":"2024-02-21T23:09:54.285563+0000","last_active":"2024-02-21T23:10:07.375019+0000","last_peered":"2024-02-21T23:10:07.375019+0000","last_clean":"2024-02-21T23:10:07.375019+0000","last_became_active":"2024-02-21T23:09:54.285275+0000","last_became_peered":"2024-02-21T23:09:54.285275+0000","last_unstale":"2024-02-21T23:10:07.375019+0000","last_undegraded":"2024-02-21T23:10:07.375019+0000","last_fullsized":"2024-02-21T23:10:07.375019+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:21:42.014406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"150'6","reported_seq":28,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236789+0000","last_change":"2024-02-21T23:09:53.288423+0000","last_active":"2024-02-21T23:10:08.236789+0000","last_peered":"2024-02-21T23:10:08.236789+0000","last_clean":"2024-02-21T23:10:08.236789+0000","last_became_active":"2024-02-21T23:09:53.287473+0000","last_became_peered":"2024-02-21T23:09:53.287473+0000","last_unstale":"2024-02-21T23:10:08.236789+0000","last_undegraded":"2024-02-21T23:10:08.236789+0000","last_fullsized":"2024-02-21T23:10:08.236789+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:13:32.425310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1e","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.380496+0000","last_change":"2024-02-21T23:10:06.389938+0000","last_active":"2024-02-21T23:10:07.380496+0000","last_peered":"2024-02-21T23:10:07.380496+0000","last_clean":"2024-02-21T23:10:07.380496+0000","last_became_active":"2024-02-21T23:10:06.387554+0000","last_became_peered":"2024-02-21T23:10:06.387554+0000","last_unstale":"2024-02-21T23:10:07.380496+0000","last_undegraded":"2024-02-21T23:10:07.380496+0000","last_fullsized":"2024-02-21T23:10:07.380496+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:44:08.714643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1d","version":"154'2","reported_seq":16,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.381075+0000","last_change":"2024-02-21T23:10:05.388219+0000","last_active":"2024-02-21T23:10:07.381075+0000","last_peered":"2024-02-21T23:10:07.381075+0000","last_clean":"2024-02-21T23:10:07.381075+0000","last_became_active":"2024-02-21T23:10:05.387863+0000","last_became_peered":"2024-02-21T23:10:05.387863+0000","last_unstale":"2024-02-21T23:10:07.381075+0000","last_undegraded":"2024-02-21T23:10:07.381075+0000","last_fullsized":"2024-02-21T23:10:07.381075+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:16:21.884892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1c","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.379732+0000","last_change":"2024-02-21T23:09:54.278618+0000","last_active":"2024-02-21T23:10:07.379732+0000","last_peered":"2024-02-21T23:10:07.379732+0000","last_clean":"2024-02-21T23:10:07.379732+0000","last_became_active":"2024-02-21T23:09:54.278343+0000","last_became_peered":"2024-02-21T23:09:54.278343+0000","last_unstale":"2024-02-21T23:10:07.379732+0000","last_undegraded":"2024-02-21T23:10:07.379732+0000","last_fullsized":"2024-02-21T23:10:07.379732+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:49:31.759374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.13","version":"150'2","reported_seq":24,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234083+0000","last_change":"2024-02-21T23:09:53.284172+0000","last_active":"2024-02-21T23:10:08.234083+0000","last_peered":"2024-02-21T23:10:08.234083+0000","last_clean":"2024-02-21T23:10:08.234083+0000","last_became_active":"2024-02-21T23:09:53.283507+0000","last_became_peered":"2024-02-21T23:09:53.283507+0000","last_unstale":"2024-02-21T23:10:08.234083+0000","last_undegraded":"2024-02-21T23:10:08.234083+0000","last_fullsized":"2024-02-21T23:10:08.234083+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:20:30.108767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1f","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.381418+0000","last_change":"2024-02-21T23:10:06.388266+0000","last_active":"2024-02-21T23:10:07.381418+0000","last_peered":"2024-02-21T23:10:07.381418+0000","last_clean":"2024-02-21T23:10:07.381418+0000","last_became_active":"2024-02-21T23:10:06.387844+0000","last_became_peered":"2024-02-21T23:10:06.387844+0000","last_unstale":"2024-02-21T23:10:07.381418+0000","last_undegraded":"2024-02-21T23:10:07.381418+0000","last_fullsized":"2024-02-21T23:10:07.381418+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:13:39.415953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1c","version":"154'2","reported_seq":16,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.231548+0000","last_change":"2024-02-21T23:10:05.387736+0000","last_active":"2024-02-21T23:10:08.231548+0000","last_peered":"2024-02-21T23:10:08.231548+0000","last_clean":"2024-02-21T23:10:08.231548+0000","last_became_active":"2024-02-21T23:10:05.387416+0000","last_became_peered":"2024-02-21T23:10:05.387416+0000","last_unstale":"2024-02-21T23:10:08.231548+0000","last_undegraded":"2024-02-21T23:10:08.231548+0000","last_fullsized":"2024-02-21T23:10:08.231548+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:34:39.730427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1d","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236368+0000","last_change":"2024-02-21T23:09:54.291717+0000","last_active":"2024-02-21T23:10:08.236368+0000","last_peered":"2024-02-21T23:10:08.236368+0000","last_clean":"2024-02-21T23:10:08.236368+0000","last_became_active":"2024-02-21T23:09:54.290842+0000","last_became_peered":"2024-02-21T23:09:54.290842+0000","last_unstale":"2024-02-21T23:10:08.236368+0000","last_undegraded":"2024-02-21T23:10:08.236368+0000","last_fullsized":"2024-02-21T23:10:08.236368+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:42:16.988326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232712+0000","last_change":"2024-02-21T23:09:53.284733+0000","last_active":"2024-02-21T23:10:08.232712+0000","last_peered":"2024-02-21T23:10:08.232712+0000","last_clean":"2024-02-21T23:10:08.232712+0000","last_became_active":"2024-02-21T23:09:53.284399+0000","last_became_peered":"2024-02-21T23:09:53.284399+0000","last_unstale":"2024-02-21T23:10:08.232712+0000","last_undegraded":"2024-02-21T23:10:08.232712+0000","last_fullsized":"2024-02-21T23:10:08.232712+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:05:08.172503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1c","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.609880+0000","last_change":"2024-02-21T23:10:06.402186+0000","last_active":"2024-02-21T23:10:07.609880+0000","last_peered":"2024-02-21T23:10:07.609880+0000","last_clean":"2024-02-21T23:10:07.609880+0000","last_became_active":"2024-02-21T23:10:06.401920+0000","last_became_peered":"2024-02-21T23:10:06.401920+0000","last_unstale":"2024-02-21T23:10:07.609880+0000","last_undegraded":"2024-02-21T23:10:07.609880+0000","last_fullsized":"2024-02-21T23:10:07.609880+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:26:48.509050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.1f","version":"154'4","reported_seq":18,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232862+0000","last_change":"2024-02-21T23:10:05.381058+0000","last_active":"2024-02-21T23:10:08.232862+0000","last_peered":"2024-02-21T23:10:08.232862+0000","last_clean":"2024-02-21T23:10:08.232862+0000","last_became_active":"2024-02-21T23:10:05.380737+0000","last_became_peered":"2024-02-21T23:10:05.380737+0000","last_unstale":"2024-02-21T23:10:08.232862+0000","last_undegraded":"2024-02-21T23:10:08.232862+0000","last_fullsized":"2024-02-21T23:10:08.232862+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:09:16.338374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1e","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.381415+0000","last_change":"2024-02-21T23:09:54.277339+0000","last_active":"2024-02-21T23:10:07.381415+0000","last_peered":"2024-02-21T23:10:07.381415+0000","last_clean":"2024-02-21T23:10:07.381415+0000","last_became_active":"2024-02-21T23:09:54.277039+0000","last_became_peered":"2024-02-21T23:09:54.277039+0000","last_unstale":"2024-02-21T23:10:07.381415+0000","last_undegraded":"2024-02-21T23:10:07.381415+0000","last_fullsized":"2024-02-21T23:10:07.381415+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:42:16.988326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.380471+0000","last_change":"2024-02-21T23:09:53.283915+0000","last_active":"2024-02-21T23:10:07.380471+0000","last_peered":"2024-02-21T23:10:07.380471+0000","last_clean":"2024-02-21T23:10:07.380471+0000","last_became_active":"2024-02-21T23:09:53.283532+0000","last_became_peered":"2024-02-21T23:09:53.283532+0000","last_unstale":"2024-02-21T23:10:07.380471+0000","last_undegraded":"2024-02-21T23:10:07.380471+0000","last_fullsized":"2024-02-21T23:10:07.380471+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:55:17.574145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1d","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.375317+0000","last_change":"2024-02-21T23:10:06.373957+0000","last_active":"2024-02-21T23:10:07.375317+0000","last_peered":"2024-02-21T23:10:07.375317+0000","last_clean":"2024-02-21T23:10:07.375317+0000","last_became_active":"2024-02-21T23:10:06.373657+0000","last_became_peered":"2024-02-21T23:10:06.373657+0000","last_unstale":"2024-02-21T23:10:07.375317+0000","last_undegraded":"2024-02-21T23:10:07.375317+0000","last_fullsized":"2024-02-21T23:10:07.375317+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:37:55.080822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1e","version":"154'1","reported_seq":15,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.235331+0000","last_change":"2024-02-21T23:10:05.388527+0000","last_active":"2024-02-21T23:10:08.235331+0000","last_peered":"2024-02-21T23:10:08.235331+0000","last_clean":"2024-02-21T23:10:08.235331+0000","last_became_active":"2024-02-21T23:10:05.388214+0000","last_became_peered":"2024-02-21T23:10:05.388214+0000","last_unstale":"2024-02-21T23:10:08.235331+0000","last_undegraded":"2024-02-21T23:10:08.235331+0000","last_fullsized":"2024-02-21T23:10:08.235331+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:50:32.666622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1f","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.376181+0000","last_change":"2024-02-21T23:09:54.265384+0000","last_active":"2024-02-21T23:10:07.376181+0000","last_peered":"2024-02-21T23:10:07.376181+0000","last_clean":"2024-02-21T23:10:07.376181+0000","last_became_active":"2024-02-21T23:09:54.265073+0000","last_became_peered":"2024-02-21T23:09:54.265073+0000","last_unstale":"2024-02-21T23:10:07.376181+0000","last_undegraded":"2024-02-21T23:10:07.376181+0000","last_fullsized":"2024-02-21T23:10:07.376181+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:44:28.295055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.10","version":"150'8","reported_seq":30,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.235156+0000","last_change":"2024-02-21T23:09:53.285246+0000","last_active":"2024-02-21T23:10:08.235156+0000","last_peered":"2024-02-21T23:10:08.235156+0000","last_clean":"2024-02-21T23:10:08.235156+0000","last_became_active":"2024-02-21T23:09:53.282084+0000","last_became_peered":"2024-02-21T23:09:53.282084+0000","last_unstale":"2024-02-21T23:10:08.235156+0000","last_undegraded":"2024-02-21T23:10:08.235156+0000","last_fullsized":"2024-02-21T23:10:08.235156+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:23:26.494398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.235925+0000","last_change":"2024-02-21T23:10:06.390439+0000","last_active":"2024-02-21T23:10:08.235925+0000","last_peered":"2024-02-21T23:10:08.235925+0000","last_clean":"2024-02-21T23:10:08.235925+0000","last_became_active":"2024-02-21T23:10:06.390012+0000","last_became_peered":"2024-02-21T23:10:06.390012+0000","last_unstale":"2024-02-21T23:10:08.235925+0000","last_undegraded":"2024-02-21T23:10:08.235925+0000","last_fullsized":"2024-02-21T23:10:08.235925+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:17:37.624314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.374798+0000","last_change":"2024-02-21T23:10:05.393586+0000","last_active":"2024-02-21T23:10:07.374798+0000","last_peered":"2024-02-21T23:10:07.374798+0000","last_clean":"2024-02-21T23:10:07.374798+0000","last_became_active":"2024-02-21T23:10:05.392606+0000","last_became_peered":"2024-02-21T23:10:05.392606+0000","last_unstale":"2024-02-21T23:10:07.374798+0000","last_undegraded":"2024-02-21T23:10:07.374798+0000","last_fullsized":"2024-02-21T23:10:07.374798+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:12:07.072214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234736+0000","last_change":"2024-02-21T23:09:54.289615+0000","last_active":"2024-02-21T23:10:08.234736+0000","last_peered":"2024-02-21T23:10:08.234736+0000","last_clean":"2024-02-21T23:10:08.234736+0000","last_became_active":"2024-02-21T23:09:54.289322+0000","last_became_peered":"2024-02-21T23:09:54.289322+0000","last_unstale":"2024-02-21T23:10:08.234736+0000","last_undegraded":"2024-02-21T23:10:08.234736+0000","last_fullsized":"2024-02-21T23:10:08.234736+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:34:28.642087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.f","version":"150'1","reported_seq":23,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236888+0000","last_change":"2024-02-21T23:09:53.287210+0000","last_active":"2024-02-21T23:10:08.236888+0000","last_peered":"2024-02-21T23:10:08.236888+0000","last_clean":"2024-02-21T23:10:08.236888+0000","last_became_active":"2024-02-21T23:09:53.285372+0000","last_became_peered":"2024-02-21T23:09:53.285372+0000","last_unstale":"2024-02-21T23:10:08.236888+0000","last_undegraded":"2024-02-21T23:10:08.236888+0000","last_fullsized":"2024-02-21T23:10:08.236888+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:20:30.108767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.609645+0000","last_change":"2024-02-21T23:10:06.393455+0000","last_active":"2024-02-21T23:10:07.609645+0000","last_peered":"2024-02-21T23:10:07.609645+0000","last_clean":"2024-02-21T23:10:07.609645+0000","last_became_active":"2024-02-21T23:10:06.393088+0000","last_became_peered":"2024-02-21T23:10:06.393088+0000","last_unstale":"2024-02-21T23:10:07.609645+0000","last_undegraded":"2024-02-21T23:10:07.609645+0000","last_fullsized":"2024-02-21T23:10:07.609645+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:44:08.714643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.379203+0000","last_change":"2024-02-21T23:10:05.395057+0000","last_active":"2024-02-21T23:10:07.379203+0000","last_peered":"2024-02-21T23:10:07.379203+0000","last_clean":"2024-02-21T23:10:07.379203+0000","last_became_active":"2024-02-21T23:10:05.394783+0000","last_became_peered":"2024-02-21T23:10:05.394783+0000","last_unstale":"2024-02-21T23:10:07.379203+0000","last_undegraded":"2024-02-21T23:10:07.379203+0000","last_fullsized":"2024-02-21T23:10:07.379203+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:02:22.335727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.379499+0000","last_change":"2024-02-21T23:09:54.274363+0000","last_active":"2024-02-21T23:10:07.379499+0000","last_peered":"2024-02-21T23:10:07.379499+0000","last_clean":"2024-02-21T23:10:07.379499+0000","last_became_active":"2024-02-21T23:09:54.274084+0000","last_became_peered":"2024-02-21T23:09:54.274084+0000","last_unstale":"2024-02-21T23:10:07.379499+0000","last_undegraded":"2024-02-21T23:10:07.379499+0000","last_fullsized":"2024-02-21T23:10:07.379499+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:42:19.996155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.e","version":"150'2","reported_seq":24,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.237049+0000","last_change":"2024-02-21T23:09:53.275941+0000","last_active":"2024-02-21T23:10:08.237049+0000","last_peered":"2024-02-21T23:10:08.237049+0000","last_clean":"2024-02-21T23:10:08.237049+0000","last_became_active":"2024-02-21T23:09:53.275555+0000","last_became_peered":"2024-02-21T23:09:53.275555+0000","last_unstale":"2024-02-21T23:10:08.237049+0000","last_undegraded":"2024-02-21T23:10:08.237049+0000","last_fullsized":"2024-02-21T23:10:08.237049+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:02:10.230470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.379162+0000","last_change":"2024-02-21T23:10:06.377153+0000","last_active":"2024-02-21T23:10:07.379162+0000","last_peered":"2024-02-21T23:10:07.379162+0000","last_clean":"2024-02-21T23:10:07.379162+0000","last_became_active":"2024-02-21T23:10:06.376854+0000","last_became_peered":"2024-02-21T23:10:06.376854+0000","last_unstale":"2024-02-21T23:10:07.379162+0000","last_undegraded":"2024-02-21T23:10:07.379162+0000","last_fullsized":"2024-02-21T23:10:07.379162+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:13:56.739562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.608871+0000","last_change":"2024-02-21T23:10:05.395810+0000","last_active":"2024-02-21T23:10:07.608871+0000","last_peered":"2024-02-21T23:10:07.608871+0000","last_clean":"2024-02-21T23:10:07.608871+0000","last_became_active":"2024-02-21T23:10:05.395273+0000","last_became_peered":"2024-02-21T23:10:05.395273+0000","last_unstale":"2024-02-21T23:10:07.608871+0000","last_undegraded":"2024-02-21T23:10:07.608871+0000","last_fullsized":"2024-02-21T23:10:07.608871+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:00:10.805897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.380745+0000","last_change":"2024-02-21T23:09:54.259630+0000","last_active":"2024-02-21T23:10:07.380745+0000","last_peered":"2024-02-21T23:10:07.380745+0000","last_clean":"2024-02-21T23:10:07.380745+0000","last_became_active":"2024-02-21T23:09:54.258924+0000","last_became_peered":"2024-02-21T23:09:54.258924+0000","last_unstale":"2024-02-21T23:10:07.380745+0000","last_undegraded":"2024-02-21T23:10:07.380745+0000","last_fullsized":"2024-02-21T23:10:07.380745+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:34:28.642087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.d","version":"150'1","reported_seq":23,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234011+0000","last_change":"2024-02-21T23:09:53.279385+0000","last_active":"2024-02-21T23:10:08.234011+0000","last_peered":"2024-02-21T23:10:08.234011+0000","last_clean":"2024-02-21T23:10:08.234011+0000","last_became_active":"2024-02-21T23:09:53.278074+0000","last_became_peered":"2024-02-21T23:09:53.278074+0000","last_unstale":"2024-02-21T23:10:08.234011+0000","last_undegraded":"2024-02-21T23:10:08.234011+0000","last_fullsized":"2024-02-21T23:10:08.234011+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:42:15.971409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.230685+0000","last_change":"2024-02-21T23:10:06.383577+0000","last_active":"2024-02-21T23:10:08.230685+0000","last_peered":"2024-02-21T23:10:08.230685+0000","last_clean":"2024-02-21T23:10:08.230685+0000","last_became_active":"2024-02-21T23:10:06.383253+0000","last_became_peered":"2024-02-21T23:10:06.383253+0000","last_unstale":"2024-02-21T23:10:08.230685+0000","last_undegraded":"2024-02-21T23:10:08.230685+0000","last_fullsized":"2024-02-21T23:10:08.230685+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:00:11.821587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.608567+0000","last_change":"2024-02-21T23:10:05.385621+0000","last_active":"2024-02-21T23:10:07.608567+0000","last_peered":"2024-02-21T23:10:07.608567+0000","last_clean":"2024-02-21T23:10:07.608567+0000","last_became_active":"2024-02-21T23:10:05.384654+0000","last_became_peered":"2024-02-21T23:10:05.384654+0000","last_unstale":"2024-02-21T23:10:07.608567+0000","last_undegraded":"2024-02-21T23:10:07.608567+0000","last_fullsized":"2024-02-21T23:10:07.608567+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:30:56.525674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.381032+0000","last_change":"2024-02-21T23:09:54.277296+0000","last_active":"2024-02-21T23:10:07.381032+0000","last_peered":"2024-02-21T23:10:07.381032+0000","last_clean":"2024-02-21T23:10:07.381032+0000","last_became_active":"2024-02-21T23:09:54.276973+0000","last_became_peered":"2024-02-21T23:09:54.276973+0000","last_unstale":"2024-02-21T23:10:07.381032+0000","last_undegraded":"2024-02-21T23:10:07.381032+0000","last_fullsized":"2024-02-21T23:10:07.381032+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:07:24.304915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.235861+0000","last_change":"2024-02-21T23:09:53.281804+0000","last_active":"2024-02-21T23:10:08.235861+0000","last_peered":"2024-02-21T23:10:08.235861+0000","last_clean":"2024-02-21T23:10:08.235861+0000","last_became_active":"2024-02-21T23:09:53.278643+0000","last_became_peered":"2024-02-21T23:09:53.278643+0000","last_unstale":"2024-02-21T23:10:08.235861+0000","last_undegraded":"2024-02-21T23:10:08.235861+0000","last_fullsized":"2024-02-21T23:10:08.235861+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:55:17.574145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.6","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.609698+0000","last_change":"2024-02-21T23:10:06.389855+0000","last_active":"2024-02-21T23:10:07.609698+0000","last_peered":"2024-02-21T23:10:07.609698+0000","last_clean":"2024-02-21T23:10:07.609698+0000","last_became_active":"2024-02-21T23:10:06.389147+0000","last_became_peered":"2024-02-21T23:10:06.389147+0000","last_unstale":"2024-02-21T23:10:07.609698+0000","last_undegraded":"2024-02-21T23:10:07.609698+0000","last_fullsized":"2024-02-21T23:10:07.609698+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:40:17.310936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234955+0000","last_change":"2024-02-21T23:10:05.389031+0000","last_active":"2024-02-21T23:10:08.234955+0000","last_peered":"2024-02-21T23:10:08.234955+0000","last_clean":"2024-02-21T23:10:08.234955+0000","last_became_active":"2024-02-21T23:10:05.388789+0000","last_became_peered":"2024-02-21T23:10:05.388789+0000","last_unstale":"2024-02-21T23:10:08.234955+0000","last_undegraded":"2024-02-21T23:10:08.234955+0000","last_fullsized":"2024-02-21T23:10:08.234955+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:46:44.667756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236537+0000","last_change":"2024-02-21T23:09:54.285141+0000","last_active":"2024-02-21T23:10:08.236537+0000","last_peered":"2024-02-21T23:10:08.236537+0000","last_clean":"2024-02-21T23:10:08.236537+0000","last_became_active":"2024-02-21T23:09:54.284498+0000","last_became_peered":"2024-02-21T23:09:54.284498+0000","last_unstale":"2024-02-21T23:10:08.236537+0000","last_undegraded":"2024-02-21T23:10:08.236537+0000","last_fullsized":"2024-02-21T23:10:08.236537+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:26:36.405020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.b","version":"150'2","reported_seq":24,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234201+0000","last_change":"2024-02-21T23:09:53.284284+0000","last_active":"2024-02-21T23:10:08.234201+0000","last_peered":"2024-02-21T23:10:08.234201+0000","last_clean":"2024-02-21T23:10:08.234201+0000","last_became_active":"2024-02-21T23:09:53.283043+0000","last_became_peered":"2024-02-21T23:09:53.283043+0000","last_unstale":"2024-02-21T23:10:08.234201+0000","last_undegraded":"2024-02-21T23:10:08.234201+0000","last_fullsized":"2024-02-21T23:10:08.234201+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:32:05.045159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.7","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.376118+0000","last_change":"2024-02-21T23:10:06.392968+0000","last_active":"2024-02-21T23:10:07.376118+0000","last_peered":"2024-02-21T23:10:07.376118+0000","last_clean":"2024-02-21T23:10:07.376118+0000","last_became_active":"2024-02-21T23:10:06.390031+0000","last_became_peered":"2024-02-21T23:10:06.390031+0000","last_unstale":"2024-02-21T23:10:07.376118+0000","last_undegraded":"2024-02-21T23:10:07.376118+0000","last_fullsized":"2024-02-21T23:10:07.376118+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:17:07.163396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236747+0000","last_change":"2024-02-21T23:10:05.392092+0000","last_active":"2024-02-21T23:10:08.236747+0000","last_peered":"2024-02-21T23:10:08.236747+0000","last_clean":"2024-02-21T23:10:08.236747+0000","last_became_active":"2024-02-21T23:10:05.391793+0000","last_became_peered":"2024-02-21T23:10:05.391793+0000","last_unstale":"2024-02-21T23:10:08.236747+0000","last_undegraded":"2024-02-21T23:10:08.236747+0000","last_fullsized":"2024-02-21T23:10:08.236747+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:09:16.338374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.233964+0000","last_change":"2024-02-21T23:09:54.275082+0000","last_active":"2024-02-21T23:10:08.233964+0000","last_peered":"2024-02-21T23:10:08.233964+0000","last_clean":"2024-02-21T23:10:08.233964+0000","last_became_active":"2024-02-21T23:09:54.274707+0000","last_became_peered":"2024-02-21T23:09:54.274707+0000","last_unstale":"2024-02-21T23:10:08.233964+0000","last_undegraded":"2024-02-21T23:10:08.233964+0000","last_fullsized":"2024-02-21T23:10:08.233964+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:39:52.080584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.381286+0000","last_change":"2024-02-21T23:09:53.277418+0000","last_active":"2024-02-21T23:10:07.381286+0000","last_peered":"2024-02-21T23:10:07.381286+0000","last_clean":"2024-02-21T23:10:07.381286+0000","last_became_active":"2024-02-21T23:09:53.277111+0000","last_became_peered":"2024-02-21T23:09:53.277111+0000","last_unstale":"2024-02-21T23:10:07.381286+0000","last_undegraded":"2024-02-21T23:10:07.381286+0000","last_fullsized":"2024-02-21T23:10:07.381286+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:23:26.494398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.375599+0000","last_change":"2024-02-21T23:10:06.391104+0000","last_active":"2024-02-21T23:10:07.375599+0000","last_peered":"2024-02-21T23:10:07.375599+0000","last_clean":"2024-02-21T23:10:07.375599+0000","last_became_active":"2024-02-21T23:10:06.390816+0000","last_became_peered":"2024-02-21T23:10:06.390816+0000","last_unstale":"2024-02-21T23:10:07.375599+0000","last_undegraded":"2024-02-21T23:10:07.375599+0000","last_fullsized":"2024-02-21T23:10:07.375599+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:49:43.863404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.7","version":"154'2","reported_seq":16,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234306+0000","last_change":"2024-02-21T23:10:05.378389+0000","last_active":"2024-02-21T23:10:08.234306+0000","last_peered":"2024-02-21T23:10:08.234306+0000","last_clean":"2024-02-21T23:10:08.234306+0000","last_became_active":"2024-02-21T23:10:05.378083+0000","last_became_peered":"2024-02-21T23:10:05.378083+0000","last_unstale":"2024-02-21T23:10:08.234306+0000","last_undegraded":"2024-02-21T23:10:08.234306+0000","last_fullsized":"2024-02-21T23:10:08.234306+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:28:14.502463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232579+0000","last_change":"2024-02-21T23:09:54.280400+0000","last_active":"2024-02-21T23:10:08.232579+0000","last_peered":"2024-02-21T23:10:08.232579+0000","last_clean":"2024-02-21T23:10:08.232579+0000","last_became_active":"2024-02-21T23:09:54.280102+0000","last_became_peered":"2024-02-21T23:09:54.280102+0000","last_unstale":"2024-02-21T23:10:08.232579+0000","last_undegraded":"2024-02-21T23:10:08.232579+0000","last_fullsized":"2024-02-21T23:10:08.232579+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:45:28.830261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.375349+0000","last_change":"2024-02-21T23:09:53.282939+0000","last_active":"2024-02-21T23:10:07.375349+0000","last_peered":"2024-02-21T23:10:07.375349+0000","last_clean":"2024-02-21T23:10:07.375349+0000","last_became_active":"2024-02-21T23:09:53.282612+0000","last_became_peered":"2024-02-21T23:09:53.282612+0000","last_unstale":"2024-02-21T23:10:07.375349+0000","last_undegraded":"2024-02-21T23:10:07.375349+0000","last_fullsized":"2024-02-21T23:10:07.375349+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:30:40.609835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236333+0000","last_change":"2024-02-21T23:10:06.396738+0000","last_active":"2024-02-21T23:10:08.236333+0000","last_peered":"2024-02-21T23:10:08.236333+0000","last_clean":"2024-02-21T23:10:08.236333+0000","last_became_active":"2024-02-21T23:10:06.396390+0000","last_became_peered":"2024-02-21T23:10:06.396390+0000","last_unstale":"2024-02-21T23:10:08.236333+0000","last_undegraded":"2024-02-21T23:10:08.236333+0000","last_fullsized":"2024-02-21T23:10:08.236333+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:24:32.853214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.6","version":"155'1","reported_seq":15,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.375969+0000","last_change":"2024-02-21T23:10:05.395962+0000","last_active":"2024-02-21T23:10:07.375969+0000","last_peered":"2024-02-21T23:10:07.375969+0000","last_clean":"2024-02-21T23:10:07.375969+0000","last_became_active":"2024-02-21T23:10:05.394863+0000","last_became_peered":"2024-02-21T23:10:05.394863+0000","last_unstale":"2024-02-21T23:10:07.375969+0000","last_undegraded":"2024-02-21T23:10:07.375969+0000","last_fullsized":"2024-02-21T23:10:07.375969+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:48:58.601649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.610043+0000","last_change":"2024-02-21T23:09:54.282302+0000","last_active":"2024-02-21T23:10:07.610043+0000","last_peered":"2024-02-21T23:10:07.610043+0000","last_clean":"2024-02-21T23:10:07.610043+0000","last_became_active":"2024-02-21T23:09:54.280468+0000","last_became_peered":"2024-02-21T23:09:54.280468+0000","last_unstale":"2024-02-21T23:10:07.610043+0000","last_undegraded":"2024-02-21T23:10:07.610043+0000","last_fullsized":"2024-02-21T23:10:07.610043+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:32:06.062076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.610683+0000","last_change":"2024-02-21T23:09:53.283837+0000","last_active":"2024-02-21T23:10:07.610683+0000","last_peered":"2024-02-21T23:10:07.610683+0000","last_clean":"2024-02-21T23:10:07.610683+0000","last_became_active":"2024-02-21T23:09:53.283541+0000","last_became_peered":"2024-02-21T23:09:53.283541+0000","last_unstale":"2024-02-21T23:10:07.610683+0000","last_undegraded":"2024-02-21T23:10:07.610683+0000","last_fullsized":"2024-02-21T23:10:07.610683+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:28:53.062636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.231400+0000","last_change":"2024-02-21T23:10:06.392422+0000","last_active":"2024-02-21T23:10:08.231400+0000","last_peered":"2024-02-21T23:10:08.231400+0000","last_clean":"2024-02-21T23:10:08.231400+0000","last_became_active":"2024-02-21T23:10:06.392121+0000","last_became_peered":"2024-02-21T23:10:06.392121+0000","last_unstale":"2024-02-21T23:10:08.231400+0000","last_undegraded":"2024-02-21T23:10:08.231400+0000","last_fullsized":"2024-02-21T23:10:08.231400+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:09:29.246717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.374605+0000","last_change":"2024-02-21T23:10:05.398114+0000","last_active":"2024-02-21T23:10:07.374605+0000","last_peered":"2024-02-21T23:10:07.374605+0000","last_clean":"2024-02-21T23:10:07.374605+0000","last_became_active":"2024-02-21T23:10:05.397811+0000","last_became_peered":"2024-02-21T23:10:05.397811+0000","last_unstale":"2024-02-21T23:10:07.374605+0000","last_undegraded":"2024-02-21T23:10:07.374605+0000","last_fullsized":"2024-02-21T23:10:07.374605+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:36:18.190950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.380388+0000","last_change":"2024-02-21T23:09:54.274229+0000","last_active":"2024-02-21T23:10:07.380388+0000","last_peered":"2024-02-21T23:10:07.380388+0000","last_clean":"2024-02-21T23:10:07.380388+0000","last_became_active":"2024-02-21T23:09:54.271365+0000","last_became_peered":"2024-02-21T23:09:54.271365+0000","last_unstale":"2024-02-21T23:10:07.380388+0000","last_undegraded":"2024-02-21T23:10:07.380388+0000","last_fullsized":"2024-02-21T23:10:07.380388+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:13:27.311923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.7","version":"150'2","reported_seq":24,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.233419+0000","last_change":"2024-02-21T23:09:53.283873+0000","last_active":"2024-02-21T23:10:08.233419+0000","last_peered":"2024-02-21T23:10:08.233419+0000","last_clean":"2024-02-21T23:10:08.233419+0000","last_became_active":"2024-02-21T23:09:53.282829+0000","last_became_peered":"2024-02-21T23:09:53.282829+0000","last_unstale":"2024-02-21T23:10:08.233419+0000","last_undegraded":"2024-02-21T23:10:08.233419+0000","last_fullsized":"2024-02-21T23:10:08.233419+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:13:32.425310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.b","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234395+0000","last_change":"2024-02-21T23:10:06.367171+0000","last_active":"2024-02-21T23:10:08.234395+0000","last_peered":"2024-02-21T23:10:08.234395+0000","last_clean":"2024-02-21T23:10:08.234395+0000","last_became_active":"2024-02-21T23:10:06.366866+0000","last_became_peered":"2024-02-21T23:10:06.366866+0000","last_unstale":"2024-02-21T23:10:08.234395+0000","last_undegraded":"2024-02-21T23:10:08.234395+0000","last_fullsized":"2024-02-21T23:10:08.234395+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:26:16.569464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.379666+0000","last_change":"2024-02-21T23:10:05.389876+0000","last_active":"2024-02-21T23:10:07.379666+0000","last_peered":"2024-02-21T23:10:07.379666+0000","last_clean":"2024-02-21T23:10:07.379666+0000","last_became_active":"2024-02-21T23:10:05.389568+0000","last_became_peered":"2024-02-21T23:10:05.389568+0000","last_unstale":"2024-02-21T23:10:07.379666+0000","last_undegraded":"2024-02-21T23:10:07.379666+0000","last_fullsized":"2024-02-21T23:10:07.379666+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:18:51.227878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234481+0000","last_change":"2024-02-21T23:09:54.277479+0000","last_active":"2024-02-21T23:10:08.234481+0000","last_peered":"2024-02-21T23:10:08.234481+0000","last_clean":"2024-02-21T23:10:08.234481+0000","last_became_active":"2024-02-21T23:09:54.277065+0000","last_became_peered":"2024-02-21T23:09:54.277065+0000","last_unstale":"2024-02-21T23:10:08.234481+0000","last_undegraded":"2024-02-21T23:10:08.234481+0000","last_fullsized":"2024-02-21T23:10:08.234481+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:07:24.304915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.6","version":"151'1","reported_seq":23,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.235714+0000","last_change":"2024-02-21T23:09:53.285874+0000","last_active":"2024-02-21T23:10:08.235714+0000","last_peered":"2024-02-21T23:10:08.235714+0000","last_clean":"2024-02-21T23:10:08.235714+0000","last_became_active":"2024-02-21T23:09:53.285569+0000","last_became_peered":"2024-02-21T23:09:53.285569+0000","last_unstale":"2024-02-21T23:10:08.235714+0000","last_undegraded":"2024-02-21T23:10:08.235714+0000","last_fullsized":"2024-02-21T23:10:08.235714+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:52:56.873078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.233184+0000","last_change":"2024-02-21T23:10:06.385851+0000","last_active":"2024-02-21T23:10:08.233184+0000","last_peered":"2024-02-21T23:10:08.233184+0000","last_clean":"2024-02-21T23:10:08.233184+0000","last_became_active":"2024-02-21T23:10:06.385482+0000","last_became_peered":"2024-02-21T23:10:06.385482+0000","last_unstale":"2024-02-21T23:10:08.233184+0000","last_undegraded":"2024-02-21T23:10:08.233184+0000","last_fullsized":"2024-02-21T23:10:08.233184+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:47:42.794742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.b","version":"154'2","reported_seq":16,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.231292+0000","last_change":"2024-02-21T23:10:05.394114+0000","last_active":"2024-02-21T23:10:08.231292+0000","last_peered":"2024-02-21T23:10:08.231292+0000","last_clean":"2024-02-21T23:10:08.231292+0000","last_became_active":"2024-02-21T23:10:05.393152+0000","last_became_peered":"2024-02-21T23:10:05.393152+0000","last_unstale":"2024-02-21T23:10:08.231292+0000","last_undegraded":"2024-02-21T23:10:08.231292+0000","last_fullsized":"2024-02-21T23:10:08.231292+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:07:35.393255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.374932+0000","last_change":"2024-02-21T23:09:54.286102+0000","last_active":"2024-02-21T23:10:07.374932+0000","last_peered":"2024-02-21T23:10:07.374932+0000","last_clean":"2024-02-21T23:10:07.374932+0000","last_became_active":"2024-02-21T23:09:54.283073+0000","last_became_peered":"2024-02-21T23:09:54.283073+0000","last_unstale":"2024-02-21T23:10:07.374932+0000","last_undegraded":"2024-02-21T23:10:07.374932+0000","last_fullsized":"2024-02-21T23:10:07.374932+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:39:05.096862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.235327+0000","last_change":"2024-02-21T23:09:53.285730+0000","last_active":"2024-02-21T23:10:08.235327+0000","last_peered":"2024-02-21T23:10:08.235327+0000","last_clean":"2024-02-21T23:10:08.235327+0000","last_became_active":"2024-02-21T23:09:53.285373+0000","last_became_peered":"2024-02-21T23:09:53.285373+0000","last_unstale":"2024-02-21T23:10:08.235327+0000","last_undegraded":"2024-02-21T23:10:08.235327+0000","last_fullsized":"2024-02-21T23:10:08.235327+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:04:15.300172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.230302+0000","last_change":"2024-02-21T23:10:06.380586+0000","last_active":"2024-02-21T23:10:08.230302+0000","last_peered":"2024-02-21T23:10:08.230302+0000","last_clean":"2024-02-21T23:10:08.230302+0000","last_became_active":"2024-02-21T23:10:06.380293+0000","last_became_peered":"2024-02-21T23:10:06.380293+0000","last_unstale":"2024-02-21T23:10:08.230302+0000","last_undegraded":"2024-02-21T23:10:08.230302+0000","last_fullsized":"2024-02-21T23:10:08.230302+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:01:26.662036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.231095+0000","last_change":"2024-02-21T23:10:05.380512+0000","last_active":"2024-02-21T23:10:08.231095+0000","last_peered":"2024-02-21T23:10:08.231095+0000","last_clean":"2024-02-21T23:10:08.231095+0000","last_became_active":"2024-02-21T23:10:05.380207+0000","last_became_peered":"2024-02-21T23:10:05.380207+0000","last_unstale":"2024-02-21T23:10:08.231095+0000","last_undegraded":"2024-02-21T23:10:08.231095+0000","last_fullsized":"2024-02-21T23:10:08.231095+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:17:36.608624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.b","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.235603+0000","last_change":"2024-02-21T23:09:54.277483+0000","last_active":"2024-02-21T23:10:08.235603+0000","last_peered":"2024-02-21T23:10:08.235603+0000","last_clean":"2024-02-21T23:10:08.235603+0000","last_became_active":"2024-02-21T23:09:54.277170+0000","last_became_peered":"2024-02-21T23:09:54.277170+0000","last_unstale":"2024-02-21T23:10:08.235603+0000","last_undegraded":"2024-02-21T23:10:08.235603+0000","last_fullsized":"2024-02-21T23:10:08.235603+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:33:07.362294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.375793+0000","last_change":"2024-02-21T23:09:53.267455+0000","last_active":"2024-02-21T23:10:07.375793+0000","last_peered":"2024-02-21T23:10:07.375793+0000","last_clean":"2024-02-21T23:10:07.375793+0000","last_became_active":"2024-02-21T23:09:53.266974+0000","last_became_peered":"2024-02-21T23:09:53.266974+0000","last_unstale":"2024-02-21T23:10:07.375793+0000","last_undegraded":"2024-02-21T23:10:07.375793+0000","last_fullsized":"2024-02-21T23:10:07.375793+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:33:26.269092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.e","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.380986+0000","last_change":"2024-02-21T23:10:06.392129+0000","last_active":"2024-02-21T23:10:07.380986+0000","last_peered":"2024-02-21T23:10:07.380986+0000","last_clean":"2024-02-21T23:10:07.380986+0000","last_became_active":"2024-02-21T23:10:06.391056+0000","last_became_peered":"2024-02-21T23:10:06.391056+0000","last_unstale":"2024-02-21T23:10:07.380986+0000","last_undegraded":"2024-02-21T23:10:07.380986+0000","last_fullsized":"2024-02-21T23:10:07.380986+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:47:42.794742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.d","version":"154'1","reported_seq":15,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.380360+0000","last_change":"2024-02-21T23:10:05.393632+0000","last_active":"2024-02-21T23:10:07.380360+0000","last_peered":"2024-02-21T23:10:07.380360+0000","last_clean":"2024-02-21T23:10:07.380360+0000","last_became_active":"2024-02-21T23:10:05.393070+0000","last_became_peered":"2024-02-21T23:10:05.393070+0000","last_unstale":"2024-02-21T23:10:07.380360+0000","last_undegraded":"2024-02-21T23:10:07.380360+0000","last_fullsized":"2024-02-21T23:10:07.380360+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:09:01.608295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232424+0000","last_change":"2024-02-21T23:09:54.275273+0000","last_active":"2024-02-21T23:10:08.232424+0000","last_peered":"2024-02-21T23:10:08.232424+0000","last_clean":"2024-02-21T23:10:08.232424+0000","last_became_active":"2024-02-21T23:09:54.274287+0000","last_became_peered":"2024-02-21T23:09:54.274287+0000","last_unstale":"2024-02-21T23:10:08.232424+0000","last_undegraded":"2024-02-21T23:10:08.232424+0000","last_fullsized":"2024-02-21T23:10:08.232424+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:18:40.139538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232148+0000","last_change":"2024-02-21T23:09:53.279782+0000","last_active":"2024-02-21T23:10:08.232148+0000","last_peered":"2024-02-21T23:10:08.232148+0000","last_clean":"2024-02-21T23:10:08.232148+0000","last_became_active":"2024-02-21T23:09:53.279477+0000","last_became_peered":"2024-02-21T23:09:53.279477+0000","last_unstale":"2024-02-21T23:10:08.232148+0000","last_undegraded":"2024-02-21T23:10:08.232148+0000","last_fullsized":"2024-02-21T23:10:08.232148+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:41:01.169723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.f","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.237523+0000","last_change":"2024-02-21T23:10:06.399721+0000","last_active":"2024-02-21T23:10:08.237523+0000","last_peered":"2024-02-21T23:10:08.237523+0000","last_clean":"2024-02-21T23:10:08.237523+0000","last_became_active":"2024-02-21T23:10:06.398869+0000","last_became_peered":"2024-02-21T23:10:06.398869+0000","last_unstale":"2024-02-21T23:10:08.237523+0000","last_undegraded":"2024-02-21T23:10:08.237523+0000","last_fullsized":"2024-02-21T23:10:08.237523+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:48:37.041227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.380907+0000","last_change":"2024-02-21T23:10:05.380230+0000","last_active":"2024-02-21T23:10:07.380907+0000","last_peered":"2024-02-21T23:10:07.380907+0000","last_clean":"2024-02-21T23:10:07.380907+0000","last_became_active":"2024-02-21T23:10:05.378612+0000","last_became_peered":"2024-02-21T23:10:05.378612+0000","last_unstale":"2024-02-21T23:10:07.380907+0000","last_undegraded":"2024-02-21T23:10:07.380907+0000","last_fullsized":"2024-02-21T23:10:07.380907+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:36:53.010888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.d","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.610179+0000","last_change":"2024-02-21T23:09:54.279409+0000","last_active":"2024-02-21T23:10:07.610179+0000","last_peered":"2024-02-21T23:10:07.610179+0000","last_clean":"2024-02-21T23:10:07.610179+0000","last_became_active":"2024-02-21T23:09:54.279107+0000","last_became_peered":"2024-02-21T23:09:54.279107+0000","last_unstale":"2024-02-21T23:10:07.610179+0000","last_undegraded":"2024-02-21T23:10:07.610179+0000","last_fullsized":"2024-02-21T23:10:07.610179+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:20:31.125684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232898+0000","last_change":"2024-02-21T23:09:53.278959+0000","last_active":"2024-02-21T23:10:08.232898+0000","last_peered":"2024-02-21T23:10:08.232898+0000","last_clean":"2024-02-21T23:10:08.232898+0000","last_became_active":"2024-02-21T23:09:53.278673+0000","last_became_peered":"2024-02-21T23:09:53.278673+0000","last_unstale":"2024-02-21T23:10:08.232898+0000","last_undegraded":"2024-02-21T23:10:08.232898+0000","last_fullsized":"2024-02-21T23:10:08.232898+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:42:15.971409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.375427+0000","last_change":"2024-02-21T23:10:06.387361+0000","last_active":"2024-02-21T23:10:07.375427+0000","last_peered":"2024-02-21T23:10:07.375427+0000","last_clean":"2024-02-21T23:10:07.375427+0000","last_became_active":"2024-02-21T23:10:06.387063+0000","last_became_peered":"2024-02-21T23:10:06.387063+0000","last_unstale":"2024-02-21T23:10:07.375427+0000","last_undegraded":"2024-02-21T23:10:07.375427+0000","last_fullsized":"2024-02-21T23:10:07.375427+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:47:41.453699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"154'1","reported_seq":15,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.381702+0000","last_change":"2024-02-21T23:10:05.381912+0000","last_active":"2024-02-21T23:10:07.381702+0000","last_peered":"2024-02-21T23:10:07.381702+0000","last_clean":"2024-02-21T23:10:07.381702+0000","last_became_active":"2024-02-21T23:10:05.381618+0000","last_became_peered":"2024-02-21T23:10:05.381618+0000","last_unstale":"2024-02-21T23:10:07.381702+0000","last_undegraded":"2024-02-21T23:10:07.381702+0000","last_fullsized":"2024-02-21T23:10:07.381702+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:47:45.153381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.e","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.235442+0000","last_change":"2024-02-21T23:09:54.282920+0000","last_active":"2024-02-21T23:10:08.235442+0000","last_peered":"2024-02-21T23:10:08.235442+0000","last_clean":"2024-02-21T23:10:08.235442+0000","last_became_active":"2024-02-21T23:09:54.282544+0000","last_became_peered":"2024-02-21T23:09:54.282544+0000","last_unstale":"2024-02-21T23:10:08.235442+0000","last_undegraded":"2024-02-21T23:10:08.235442+0000","last_fullsized":"2024-02-21T23:10:08.235442+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:40:05.206906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.380542+0000","last_change":"2024-02-21T23:09:53.275586+0000","last_active":"2024-02-21T23:10:07.380542+0000","last_peered":"2024-02-21T23:10:07.380542+0000","last_clean":"2024-02-21T23:10:07.380542+0000","last_became_active":"2024-02-21T23:09:53.274828+0000","last_became_peered":"2024-02-21T23:09:53.274828+0000","last_unstale":"2024-02-21T23:10:07.380542+0000","last_undegraded":"2024-02-21T23:10:07.380542+0000","last_fullsized":"2024-02-21T23:10:07.380542+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:02:10.230470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.d","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234506+0000","last_change":"2024-02-21T23:10:06.377243+0000","last_active":"2024-02-21T23:10:08.234506+0000","last_peered":"2024-02-21T23:10:08.234506+0000","last_clean":"2024-02-21T23:10:08.234506+0000","last_became_active":"2024-02-21T23:10:06.376521+0000","last_became_peered":"2024-02-21T23:10:06.376521+0000","last_unstale":"2024-02-21T23:10:08.234506+0000","last_undegraded":"2024-02-21T23:10:08.234506+0000","last_fullsized":"2024-02-21T23:10:08.234506+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:02:54.477082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.e","version":"154'2","reported_seq":16,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232515+0000","last_change":"2024-02-21T23:10:05.388215+0000","last_active":"2024-02-21T23:10:08.232515+0000","last_peered":"2024-02-21T23:10:08.232515+0000","last_clean":"2024-02-21T23:10:08.232515+0000","last_became_active":"2024-02-21T23:10:05.387907+0000","last_became_peered":"2024-02-21T23:10:05.387907+0000","last_unstale":"2024-02-21T23:10:08.232515+0000","last_undegraded":"2024-02-21T23:10:08.232515+0000","last_fullsized":"2024-02-21T23:10:08.232515+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:05:36.127991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.f","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.376215+0000","last_change":"2024-02-21T23:09:54.277320+0000","last_active":"2024-02-21T23:10:07.376215+0000","last_peered":"2024-02-21T23:10:07.376215+0000","last_clean":"2024-02-21T23:10:07.376215+0000","last_became_active":"2024-02-21T23:09:54.276368+0000","last_became_peered":"2024-02-21T23:09:54.276368+0000","last_unstale":"2024-02-21T23:10:07.376215+0000","last_undegraded":"2024-02-21T23:10:07.376215+0000","last_fullsized":"2024-02-21T23:10:07.376215+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:40:37.285836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.230247+0000","last_change":"2024-02-21T23:09:53.278027+0000","last_active":"2024-02-21T23:10:08.230247+0000","last_peered":"2024-02-21T23:10:08.230247+0000","last_clean":"2024-02-21T23:10:08.230247+0000","last_became_active":"2024-02-21T23:09:53.277660+0000","last_became_peered":"2024-02-21T23:09:53.277660+0000","last_unstale":"2024-02-21T23:10:08.230247+0000","last_undegraded":"2024-02-21T23:10:08.230247+0000","last_fullsized":"2024-02-21T23:10:08.230247+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:07:23.287998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1b","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.379594+0000","last_change":"2024-02-21T23:10:06.386342+0000","last_active":"2024-02-21T23:10:07.379594+0000","last_peered":"2024-02-21T23:10:07.379594+0000","last_clean":"2024-02-21T23:10:07.379594+0000","last_became_active":"2024-02-21T23:10:06.385507+0000","last_became_peered":"2024-02-21T23:10:06.385507+0000","last_unstale":"2024-02-21T23:10:07.379594+0000","last_undegraded":"2024-02-21T23:10:07.379594+0000","last_fullsized":"2024-02-21T23:10:07.379594+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:01:26.987040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.230256+0000","last_change":"2024-02-21T23:10:05.390395+0000","last_active":"2024-02-21T23:10:08.230256+0000","last_peered":"2024-02-21T23:10:08.230256+0000","last_clean":"2024-02-21T23:10:08.230256+0000","last_became_active":"2024-02-21T23:10:05.390115+0000","last_became_peered":"2024-02-21T23:10:05.390115+0000","last_unstale":"2024-02-21T23:10:08.230256+0000","last_undegraded":"2024-02-21T23:10:08.230256+0000","last_fullsized":"2024-02-21T23:10:08.230256+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:10:20.170395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.608818+0000","last_change":"2024-02-21T23:09:54.277356+0000","last_active":"2024-02-21T23:10:07.608818+0000","last_peered":"2024-02-21T23:10:07.608818+0000","last_clean":"2024-02-21T23:10:07.608818+0000","last_became_active":"2024-02-21T23:09:54.277087+0000","last_became_peered":"2024-02-21T23:09:54.277087+0000","last_unstale":"2024-02-21T23:10:07.608818+0000","last_undegraded":"2024-02-21T23:10:07.608818+0000","last_fullsized":"2024-02-21T23:10:07.608818+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:03:44.493909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.235691+0000","last_change":"2024-02-21T23:09:53.286181+0000","last_active":"2024-02-21T23:10:08.235691+0000","last_peered":"2024-02-21T23:10:08.235691+0000","last_clean":"2024-02-21T23:10:08.235691+0000","last_became_active":"2024-02-21T23:09:53.285849+0000","last_became_peered":"2024-02-21T23:09:53.285849+0000","last_unstale":"2024-02-21T23:10:08.235691+0000","last_undegraded":"2024-02-21T23:10:08.235691+0000","last_fullsized":"2024-02-21T23:10:08.235691+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:51:08.543135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":425,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.610870+0000","last_change":"2024-02-21T23:02:32.289335+0000","last_active":"2024-02-21T23:10:07.610870+0000","last_peered":"2024-02-21T23:10:07.610870+0000","last_clean":"2024-02-21T23:10:07.610870+0000","last_became_active":"2024-02-21T23:02:32.286913+0000","last_became_peered":"2024-02-21T23:02:32.286913+0000","last_unstale":"2024-02-21T23:10:07.610870+0000","last_undegraded":"2024-02-21T23:10:07.610870+0000","last_fullsized":"2024-02-21T23:10:07.610870+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_clean_scrub_stamp":"2024-02-21T23:02:31.172569+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:36:23.987374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":26,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":25,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":24,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":23,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574942,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33148,"kb_used_data":1968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338692,"statfs":{"total":96636764160,"available":96602820608,"internally_reserved":0,"allocated":2015232,"data_stored":1546164,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574943,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33088,"kb_used_data":1908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338752,"statfs":{"total":96636764160,"available":96602882048,"internally_reserved":0,"allocated":1953792,"data_stored":955866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574942,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33112,"kb_used_data":1928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338728,"statfs":{"total":96636764160,"available":96602857472,"internally_reserved":0,"allocated":1974272,"data_stored":957554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574943,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33120,"kb_used_data":1936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338720,"statfs":{"total":96636764160,"available":96602849280,"internally_reserved":0,"allocated":1982464,"data_stored":958206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574943,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33120,"kb_used_data":1932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338720,"statfs":{"total":96636764160,"available":96602849280,"internally_reserved":0,"allocated":1978368,"data_stored":958172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574944,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33096,"kb_used_data":1908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338744,"statfs":{"total":96636764160,"available":96602873856,"internally_reserved":0,"allocated":1953792,"data_stored":956378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574942,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33104,"kb_used_data":1916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338736,"statfs":{"total":96636764160,"available":96602865664,"internally_reserved":0,"allocated":1961984,"data_stored":957408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574942,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33172,"kb_used_data":1992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338668,"statfs":{"total":96636764160,"available":96602796032,"internally_reserved":0,"allocated":2039808,"data_stored":1546996,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-21T23:10:16.299 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-21T23:10:16.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.527+0000 7f6e16e00640 1 Processor -- start 2024-02-21T23:10:16.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.527+0000 7f6e16e00640 1 -- start start 2024-02-21T23:10:16.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.529+0000 7f6e16e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6e100aff90 0x7f6e100c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:16.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.529+0000 7f6e16e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6e1006dbf0 0x7f6e10158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:16.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.529+0000 7f6e16e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e100c74c0 con 0x7f6e100c5720 2024-02-21T23:10:16.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.529+0000 7f6e16e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e100c7270 con 0x7f6e1006dbf0 2024-02-21T23:10:16.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.529+0000 7f6e16e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e10158630 con 0x7f6e100aff90 2024-02-21T23:10:16.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.529+0000 7f6e15000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6e1006dbf0 0x7f6e10158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:16.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.529+0000 7f6e15000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6e1006dbf0 0x7f6e10158ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49738/0 (socket says 172.21.15.28:49738) 2024-02-21T23:10:16.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.529+0000 7f6e15000640 1 -- 172.21.15.28:0/1707675379 learned_addr learned my addr 172.21.15.28:0/1707675379 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:16.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.530+0000 7f6e15a00640 1 --2- 172.21.15.28:0/1707675379 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6e100aff90 0x7f6e100c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:16.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.530+0000 7f6e15000640 1 -- 172.21.15.28:0/1707675379 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6e100aff90 msgr2=0x7f6e100c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:16.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.530+0000 7f6e15000640 1 --2- 172.21.15.28:0/1707675379 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6e100aff90 0x7f6e100c51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:16.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.530+0000 7f6e15000640 1 -- 172.21.15.28:0/1707675379 >> v1:172.21.15.81:6789/0 conn(0x7f6e100c5720 legacy=0x7f6e1006c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:16.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.530+0000 7f6e15000640 1 -- 172.21.15.28:0/1707675379 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e100c77a0 con 0x7f6e1006dbf0 2024-02-21T23:10:16.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.531+0000 7f6e15000640 1 --2- 172.21.15.28:0/1707675379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e1006dbf0 0x7f6e10158ac0 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc009870 tx=0x7f6dfc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66599d88f18655d5 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:16.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.531+0000 7f6e0fe00640 1 -- 172.21.15.28:0/1707675379 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dfc035020 con 0x7f6e1006dbf0 2024-02-21T23:10:16.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.531+0000 7f6e0fe00640 1 -- 172.21.15.28:0/1707675379 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dfc031640 con 0x7f6e1006dbf0 2024-02-21T23:10:16.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.531+0000 7f6e0fe00640 1 -- 172.21.15.28:0/1707675379 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dfc02d040 con 0x7f6e1006dbf0 2024-02-21T23:10:16.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.532+0000 7f6e16e00640 1 -- 172.21.15.28:0/1707675379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e1006dbf0 msgr2=0x7f6e10158ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:16.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.532+0000 7f6e16e00640 1 --2- 172.21.15.28:0/1707675379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e1006dbf0 0x7f6e10158ac0 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc009870 tx=0x7f6dfc02b8f0 comp rx=0 tx=0).stop 2024-02-21T23:10:16.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.532+0000 7f6e16e00640 1 -- 172.21.15.28:0/1707675379 shutdown_connections 2024-02-21T23:10:16.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.532+0000 7f6e16e00640 1 --2- 172.21.15.28:0/1707675379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e1006dbf0 0x7f6e10158ac0 unknown :-1 s=CLOSED pgs=1751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:16.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.532+0000 7f6e16e00640 1 --2- 172.21.15.28:0/1707675379 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6e100aff90 0x7f6e100c51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:16.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.532+0000 7f6e16e00640 1 -- 172.21.15.28:0/1707675379 >> 172.21.15.28:0/1707675379 conn(0x7f6e100ac370 msgr2=0x7f6e100ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:16.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.532+0000 7f6e16e00640 1 -- 172.21.15.28:0/1707675379 shutdown_connections 2024-02-21T23:10:16.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.532+0000 7f6e16e00640 1 -- 172.21.15.28:0/1707675379 wait complete. 2024-02-21T23:10:16.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.534+0000 7f6e16e00640 1 Processor -- start 2024-02-21T23:10:16.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.534+0000 7f6e16e00640 1 -- start start 2024-02-21T23:10:16.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.535+0000 7f6e16e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6e1006dbf0 0x7f6e10153ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:16.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.535+0000 7f6e16e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e100aff90 0x7f6e1014b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:16.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.535+0000 7f6e16e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e100c5720 0x7f6e1014bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:16.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.535+0000 7f6e16e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e100c8760 con 0x7f6e100aff90 2024-02-21T23:10:16.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.535+0000 7f6e16e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e100be0c0 con 0x7f6e100c5720 2024-02-21T23:10:16.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.535+0000 7f6e16e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e100c88e0 con 0x7f6e1006dbf0 2024-02-21T23:10:16.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.535+0000 7f6e15000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e100aff90 0x7f6e1014b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:16.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.535+0000 7f6e15000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e100aff90 0x7f6e1014b860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49754/0 (socket says 172.21.15.28:49754) 2024-02-21T23:10:16.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.535+0000 7f6e15000640 1 -- 172.21.15.28:0/950497381 learned_addr learned my addr 172.21.15.28:0/950497381 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:16.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.536+0000 7f6e16400640 1 --2- 172.21.15.28:0/950497381 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e100c5720 0x7f6e1014bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:16.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.536+0000 7f6e15a00640 1 --2- 172.21.15.28:0/950497381 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6e1006dbf0 0x7f6e10153ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:16.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.536+0000 7f6e15000640 1 -- 172.21.15.28:0/950497381 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6e1006dbf0 msgr2=0x7f6e10153ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:16.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.536+0000 7f6e15000640 1 --2- 172.21.15.28:0/950497381 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6e1006dbf0 0x7f6e10153ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:16.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.536+0000 7f6e15000640 1 -- 172.21.15.28:0/950497381 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e100c5720 msgr2=0x7f6e1014bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:16.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.536+0000 7f6e15000640 1 --2- 172.21.15.28:0/950497381 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e100c5720 0x7f6e1014bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:16.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.536+0000 7f6e15000640 1 -- 172.21.15.28:0/950497381 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e100c1b80 con 0x7f6e100aff90 2024-02-21T23:10:16.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.536+0000 7f6e16400640 1 --2- 172.21.15.28:0/950497381 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e100c5720 0x7f6e1014bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:10:16.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.536+0000 7f6e15000640 1 --2- 172.21.15.28:0/950497381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e100aff90 0x7f6e1014b860 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc02b930 tx=0x7f6dfc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:16.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.537+0000 7f6e0d600640 1 -- 172.21.15.28:0/950497381 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dfc03aa60 con 0x7f6e100aff90 2024-02-21T23:10:16.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.537+0000 7f6e0d600640 1 -- 172.21.15.28:0/950497381 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dfc03ac00 con 0x7f6e100aff90 2024-02-21T23:10:16.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.537+0000 7f6e0d600640 1 -- 172.21.15.28:0/950497381 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dfc0443f0 con 0x7f6e100aff90 2024-02-21T23:10:16.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.537+0000 7f6e16e00640 1 -- 172.21.15.28:0/950497381 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e10153830 con 0x7f6e100aff90 2024-02-21T23:10:16.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.537+0000 7f6e16e00640 1 -- 172.21.15.28:0/950497381 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e10153ad0 con 0x7f6e100aff90 2024-02-21T23:10:16.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.538+0000 7f6e16e00640 1 -- 172.21.15.28:0/950497381 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e1004f290 con 0x7f6e100aff90 2024-02-21T23:10:16.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.539+0000 7f6e0d600640 1 -- 172.21.15.28:0/950497381 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6dfc038020 con 0x7f6e100aff90 2024-02-21T23:10:16.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.540+0000 7f6e0d600640 1 --2- 172.21.15.28:0/950497381 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6df006b4d0 0x7f6df006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:16.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.540+0000 7f6e0d600640 1 -- 172.21.15.28:0/950497381 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f6dfc0ac480 con 0x7f6e100aff90 2024-02-21T23:10:16.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.541+0000 7f6e15a00640 1 --2- 172.21.15.28:0/950497381 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6df006b4d0 0x7f6df006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:16.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.542+0000 7f6e15a00640 1 --2- 172.21.15.28:0/950497381 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6df006b4d0 0x7f6df006d990 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f6e04006fd0 tx=0x7f6e04008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:16.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.544+0000 7f6e0d600640 1 -- 172.21.15.28:0/950497381 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f6dfc03f070 con 0x7f6e100aff90 2024-02-21T23:10:16.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.750+0000 7f6e16e00640 1 -- 172.21.15.28:0/950497381 --> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6e10156dc0 con 0x7f6df006b4d0 2024-02-21T23:10:16.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.784+0000 7f6e0d600640 1 -- 172.21.15.28:0/950497381 <== mgr.4101 v2:172.21.15.81:6832/1848538577 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+491533 (secure 0 0 0) 0x7f6e10156dc0 con 0x7f6df006b4d0 2024-02-21T23:10:16.780 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:10:16.783 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-21T23:10:16.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.791+0000 7f6e16e00640 1 -- 172.21.15.28:0/950497381 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6df006b4d0 msgr2=0x7f6df006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:16.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.791+0000 7f6e16e00640 1 --2- 172.21.15.28:0/950497381 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6df006b4d0 0x7f6df006d990 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f6e04006fd0 tx=0x7f6e04008040 comp rx=0 tx=0).stop 2024-02-21T23:10:16.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.792+0000 7f6e16e00640 1 -- 172.21.15.28:0/950497381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e100aff90 msgr2=0x7f6e1014b860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:16.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.792+0000 7f6e16e00640 1 --2- 172.21.15.28:0/950497381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e100aff90 0x7f6e1014b860 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc02b930 tx=0x7f6dfc0046c0 comp rx=0 tx=0).stop 2024-02-21T23:10:16.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.792+0000 7f6e16e00640 1 -- 172.21.15.28:0/950497381 shutdown_connections 2024-02-21T23:10:16.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.792+0000 7f6e16e00640 1 --2- 172.21.15.28:0/950497381 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6df006b4d0 0x7f6df006d990 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:16.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.792+0000 7f6e16e00640 1 --2- 172.21.15.28:0/950497381 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6e100c5720 0x7f6e1014bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:16.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.792+0000 7f6e16e00640 1 --2- 172.21.15.28:0/950497381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e100aff90 0x7f6e1014b860 unknown :-1 s=CLOSED pgs=1752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:16.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.792+0000 7f6e16e00640 1 --2- 172.21.15.28:0/950497381 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6e1006dbf0 0x7f6e10153ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:16.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.792+0000 7f6e16e00640 1 -- 172.21.15.28:0/950497381 >> 172.21.15.28:0/950497381 conn(0x7f6e100ac370 msgr2=0x7f6e100ac750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:16.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.792+0000 7f6e16e00640 1 -- 172.21.15.28:0/950497381 shutdown_connections 2024-02-21T23:10:16.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:16.792+0000 7f6e16e00640 1 -- 172.21.15.28:0/950497381 wait complete. 2024-02-21T23:10:16.799 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":380,"stamp":"2024-02-21T23:10:14.826627+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595188,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":249,"num_write_kb":2444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":195,"ondisk_log_size":195,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":386,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":264960,"kb_used_data":15488,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754709760,"statfs":{"total":773094113280,"available":772822794240,"internally_reserved":0,"allocated":15859712,"data_stored":8836744,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.481842"},"pg_stats":[{"pgid":"26.3d","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.380162+0000","last_change":"2024-02-21T23:10:06.382729+0000","last_active":"2024-02-21T23:10:07.380162+0000","last_peered":"2024-02-21T23:10:07.380162+0000","last_clean":"2024-02-21T23:10:07.380162+0000","last_became_active":"2024-02-21T23:10:06.381756+0000","last_became_peered":"2024-02-21T23:10:06.381756+0000","last_unstale":"2024-02-21T23:10:07.380162+0000","last_undegraded":"2024-02-21T23:10:07.380162+0000","last_fullsized":"2024-02-21T23:10:07.380162+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:03:56.597939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.3f","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232348+0000","last_change":"2024-02-21T23:09:54.279018+0000","last_active":"2024-02-21T23:10:08.232348+0000","last_peered":"2024-02-21T23:10:08.232348+0000","last_clean":"2024-02-21T23:10:08.232348+0000","last_became_active":"2024-02-21T23:09:54.278712+0000","last_became_peered":"2024-02-21T23:09:54.278712+0000","last_unstale":"2024-02-21T23:10:08.232348+0000","last_undegraded":"2024-02-21T23:10:08.232348+0000","last_fullsized":"2024-02-21T23:10:08.232348+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:55:18.591062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.3c","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232651+0000","last_change":"2024-02-21T23:10:06.385434+0000","last_active":"2024-02-21T23:10:08.232651+0000","last_peered":"2024-02-21T23:10:08.232651+0000","last_clean":"2024-02-21T23:10:08.232651+0000","last_became_active":"2024-02-21T23:10:06.384878+0000","last_became_peered":"2024-02-21T23:10:06.384878+0000","last_unstale":"2024-02-21T23:10:08.232651+0000","last_undegraded":"2024-02-21T23:10:08.232651+0000","last_fullsized":"2024-02-21T23:10:08.232651+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:52:21.069235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.3e","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.609956+0000","last_change":"2024-02-21T23:09:54.285889+0000","last_active":"2024-02-21T23:10:07.609956+0000","last_peered":"2024-02-21T23:10:07.609956+0000","last_clean":"2024-02-21T23:10:07.609956+0000","last_became_active":"2024-02-21T23:09:54.284810+0000","last_became_peered":"2024-02-21T23:09:54.284810+0000","last_unstale":"2024-02-21T23:10:07.609956+0000","last_undegraded":"2024-02-21T23:10:07.609956+0000","last_fullsized":"2024-02-21T23:10:07.609956+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:04:16.317089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.3f","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236866+0000","last_change":"2024-02-21T23:10:06.389219+0000","last_active":"2024-02-21T23:10:08.236866+0000","last_peered":"2024-02-21T23:10:08.236866+0000","last_clean":"2024-02-21T23:10:08.236866+0000","last_became_active":"2024-02-21T23:10:06.388931+0000","last_became_peered":"2024-02-21T23:10:06.388931+0000","last_unstale":"2024-02-21T23:10:08.236866+0000","last_undegraded":"2024-02-21T23:10:08.236866+0000","last_fullsized":"2024-02-21T23:10:08.236866+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:46:45.683446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.3d","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.231699+0000","last_change":"2024-02-21T23:09:54.263978+0000","last_active":"2024-02-21T23:10:08.231699+0000","last_peered":"2024-02-21T23:10:08.231699+0000","last_clean":"2024-02-21T23:10:08.231699+0000","last_became_active":"2024-02-21T23:09:54.263467+0000","last_became_peered":"2024-02-21T23:09:54.263467+0000","last_unstale":"2024-02-21T23:10:08.231699+0000","last_undegraded":"2024-02-21T23:10:08.231699+0000","last_fullsized":"2024-02-21T23:10:08.231699+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:46:35.436815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.3e","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236400+0000","last_change":"2024-02-21T23:10:06.406362+0000","last_active":"2024-02-21T23:10:08.236400+0000","last_peered":"2024-02-21T23:10:08.236400+0000","last_clean":"2024-02-21T23:10:08.236400+0000","last_became_active":"2024-02-21T23:10:06.405859+0000","last_became_peered":"2024-02-21T23:10:06.405859+0000","last_unstale":"2024-02-21T23:10:08.236400+0000","last_undegraded":"2024-02-21T23:10:08.236400+0000","last_fullsized":"2024-02-21T23:10:08.236400+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:09:17.354064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.3c","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.231855+0000","last_change":"2024-02-21T23:09:54.279400+0000","last_active":"2024-02-21T23:10:08.231855+0000","last_peered":"2024-02-21T23:10:08.231855+0000","last_clean":"2024-02-21T23:10:08.231855+0000","last_became_active":"2024-02-21T23:09:54.279076+0000","last_became_peered":"2024-02-21T23:09:54.279076+0000","last_unstale":"2024-02-21T23:10:08.231855+0000","last_undegraded":"2024-02-21T23:10:08.231855+0000","last_fullsized":"2024-02-21T23:10:08.231855+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:02:11.247387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.39","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.375184+0000","last_change":"2024-02-21T23:10:06.375038+0000","last_active":"2024-02-21T23:10:07.375184+0000","last_peered":"2024-02-21T23:10:07.375184+0000","last_clean":"2024-02-21T23:10:07.375184+0000","last_became_active":"2024-02-21T23:10:06.374714+0000","last_became_peered":"2024-02-21T23:10:06.374714+0000","last_unstale":"2024-02-21T23:10:07.375184+0000","last_undegraded":"2024-02-21T23:10:07.375184+0000","last_fullsized":"2024-02-21T23:10:07.375184+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:48:19.473212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.3b","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.233882+0000","last_change":"2024-02-21T23:09:54.271682+0000","last_active":"2024-02-21T23:10:08.233882+0000","last_peered":"2024-02-21T23:10:08.233882+0000","last_clean":"2024-02-21T23:10:08.233882+0000","last_became_active":"2024-02-21T23:09:54.271297+0000","last_became_peered":"2024-02-21T23:09:54.271297+0000","last_unstale":"2024-02-21T23:10:08.233882+0000","last_undegraded":"2024-02-21T23:10:08.233882+0000","last_fullsized":"2024-02-21T23:10:08.233882+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:16:10.796552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.38","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.609207+0000","last_change":"2024-02-21T23:10:06.393732+0000","last_active":"2024-02-21T23:10:07.609207+0000","last_peered":"2024-02-21T23:10:07.609207+0000","last_clean":"2024-02-21T23:10:07.609207+0000","last_became_active":"2024-02-21T23:10:06.393448+0000","last_became_peered":"2024-02-21T23:10:06.393448+0000","last_unstale":"2024-02-21T23:10:07.609207+0000","last_undegraded":"2024-02-21T23:10:07.609207+0000","last_fullsized":"2024-02-21T23:10:07.609207+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:42:20.227218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3a","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.608838+0000","last_change":"2024-02-21T23:09:54.281830+0000","last_active":"2024-02-21T23:10:07.608838+0000","last_peered":"2024-02-21T23:10:07.608838+0000","last_clean":"2024-02-21T23:10:07.608838+0000","last_became_active":"2024-02-21T23:09:54.281334+0000","last_became_peered":"2024-02-21T23:09:54.281334+0000","last_unstale":"2024-02-21T23:10:07.608838+0000","last_undegraded":"2024-02-21T23:10:07.608838+0000","last_fullsized":"2024-02-21T23:10:07.608838+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:07:24.304915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.3b","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.233350+0000","last_change":"2024-02-21T23:10:06.385935+0000","last_active":"2024-02-21T23:10:08.233350+0000","last_peered":"2024-02-21T23:10:08.233350+0000","last_clean":"2024-02-21T23:10:08.233350+0000","last_became_active":"2024-02-21T23:10:06.385657+0000","last_became_peered":"2024-02-21T23:10:06.385657+0000","last_unstale":"2024-02-21T23:10:08.233350+0000","last_undegraded":"2024-02-21T23:10:08.233350+0000","last_fullsized":"2024-02-21T23:10:08.233350+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:27:47.158633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.39","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.233971+0000","last_change":"2024-02-21T23:09:54.277806+0000","last_active":"2024-02-21T23:10:08.233971+0000","last_peered":"2024-02-21T23:10:08.233971+0000","last_clean":"2024-02-21T23:10:08.233971+0000","last_became_active":"2024-02-21T23:09:54.277446+0000","last_became_peered":"2024-02-21T23:09:54.277446+0000","last_unstale":"2024-02-21T23:10:08.233971+0000","last_undegraded":"2024-02-21T23:10:08.233971+0000","last_fullsized":"2024-02-21T23:10:08.233971+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:59:34.646775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.3a","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.231962+0000","last_change":"2024-02-21T23:10:06.394386+0000","last_active":"2024-02-21T23:10:08.231962+0000","last_peered":"2024-02-21T23:10:08.231962+0000","last_clean":"2024-02-21T23:10:08.231962+0000","last_became_active":"2024-02-21T23:10:06.391655+0000","last_became_peered":"2024-02-21T23:10:06.391655+0000","last_unstale":"2024-02-21T23:10:08.231962+0000","last_undegraded":"2024-02-21T23:10:08.231962+0000","last_fullsized":"2024-02-21T23:10:08.231962+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:15:42.134184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.38","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232805+0000","last_change":"2024-02-21T23:09:54.286729+0000","last_active":"2024-02-21T23:10:08.232805+0000","last_peered":"2024-02-21T23:10:08.232805+0000","last_clean":"2024-02-21T23:10:08.232805+0000","last_became_active":"2024-02-21T23:09:54.286427+0000","last_became_peered":"2024-02-21T23:09:54.286427+0000","last_unstale":"2024-02-21T23:10:08.232805+0000","last_undegraded":"2024-02-21T23:10:08.232805+0000","last_fullsized":"2024-02-21T23:10:08.232805+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:13:27.311923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.35","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.380954+0000","last_change":"2024-02-21T23:10:06.390905+0000","last_active":"2024-02-21T23:10:07.380954+0000","last_peered":"2024-02-21T23:10:07.380954+0000","last_clean":"2024-02-21T23:10:07.380954+0000","last_became_active":"2024-02-21T23:10:06.390622+0000","last_became_peered":"2024-02-21T23:10:06.390622+0000","last_unstale":"2024-02-21T23:10:07.380954+0000","last_undegraded":"2024-02-21T23:10:07.380954+0000","last_fullsized":"2024-02-21T23:10:07.380954+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:20:39.771135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.37","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.237533+0000","last_change":"2024-02-21T23:09:54.286612+0000","last_active":"2024-02-21T23:10:08.237533+0000","last_peered":"2024-02-21T23:10:08.237533+0000","last_clean":"2024-02-21T23:10:08.237533+0000","last_became_active":"2024-02-21T23:09:54.286306+0000","last_became_peered":"2024-02-21T23:09:54.286306+0000","last_unstale":"2024-02-21T23:10:08.237533+0000","last_undegraded":"2024-02-21T23:10:08.237533+0000","last_fullsized":"2024-02-21T23:10:08.237533+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:04:16.317089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.34","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232897+0000","last_change":"2024-02-21T23:10:06.385190+0000","last_active":"2024-02-21T23:10:08.232897+0000","last_peered":"2024-02-21T23:10:08.232897+0000","last_clean":"2024-02-21T23:10:08.232897+0000","last_became_active":"2024-02-21T23:10:06.384496+0000","last_became_peered":"2024-02-21T23:10:06.384496+0000","last_unstale":"2024-02-21T23:10:08.232897+0000","last_undegraded":"2024-02-21T23:10:08.232897+0000","last_fullsized":"2024-02-21T23:10:08.232897+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:28:15.352760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.36","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.237062+0000","last_change":"2024-02-21T23:09:54.288019+0000","last_active":"2024-02-21T23:10:08.237062+0000","last_peered":"2024-02-21T23:10:08.237062+0000","last_clean":"2024-02-21T23:10:08.237062+0000","last_became_active":"2024-02-21T23:09:54.287742+0000","last_became_peered":"2024-02-21T23:09:54.287742+0000","last_unstale":"2024-02-21T23:10:08.237062+0000","last_undegraded":"2024-02-21T23:10:08.237062+0000","last_fullsized":"2024-02-21T23:10:08.237062+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:13:33.442227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.37","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236970+0000","last_change":"2024-02-21T23:10:06.398025+0000","last_active":"2024-02-21T23:10:08.236970+0000","last_peered":"2024-02-21T23:10:08.236970+0000","last_clean":"2024-02-21T23:10:08.236970+0000","last_became_active":"2024-02-21T23:10:06.395723+0000","last_became_peered":"2024-02-21T23:10:06.395723+0000","last_unstale":"2024-02-21T23:10:08.236970+0000","last_undegraded":"2024-02-21T23:10:08.236970+0000","last_fullsized":"2024-02-21T23:10:08.236970+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:50:33.682312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.35","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.381341+0000","last_change":"2024-02-21T23:09:54.283918+0000","last_active":"2024-02-21T23:10:07.381341+0000","last_peered":"2024-02-21T23:10:07.381341+0000","last_clean":"2024-02-21T23:10:07.381341+0000","last_became_active":"2024-02-21T23:09:54.280599+0000","last_became_peered":"2024-02-21T23:09:54.280599+0000","last_unstale":"2024-02-21T23:10:07.381341+0000","last_undegraded":"2024-02-21T23:10:07.381341+0000","last_fullsized":"2024-02-21T23:10:07.381341+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:33:07.362294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.36","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234837+0000","last_change":"2024-02-21T23:10:06.377423+0000","last_active":"2024-02-21T23:10:08.234837+0000","last_peered":"2024-02-21T23:10:08.234837+0000","last_clean":"2024-02-21T23:10:08.234837+0000","last_became_active":"2024-02-21T23:10:06.376998+0000","last_became_peered":"2024-02-21T23:10:06.376998+0000","last_unstale":"2024-02-21T23:10:08.234837+0000","last_undegraded":"2024-02-21T23:10:08.234837+0000","last_fullsized":"2024-02-21T23:10:08.234837+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:26:32.419695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.34","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.610543+0000","last_change":"2024-02-21T23:09:54.282846+0000","last_active":"2024-02-21T23:10:07.610543+0000","last_peered":"2024-02-21T23:10:07.610543+0000","last_clean":"2024-02-21T23:10:07.610543+0000","last_became_active":"2024-02-21T23:09:54.282117+0000","last_became_peered":"2024-02-21T23:09:54.282117+0000","last_unstale":"2024-02-21T23:10:07.610543+0000","last_undegraded":"2024-02-21T23:10:07.610543+0000","last_fullsized":"2024-02-21T23:10:07.610543+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:33:07.362294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.31","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234391+0000","last_change":"2024-02-21T23:10:06.382496+0000","last_active":"2024-02-21T23:10:08.234391+0000","last_peered":"2024-02-21T23:10:08.234391+0000","last_clean":"2024-02-21T23:10:08.234391+0000","last_became_active":"2024-02-21T23:10:06.379727+0000","last_became_peered":"2024-02-21T23:10:06.379727+0000","last_unstale":"2024-02-21T23:10:08.234391+0000","last_undegraded":"2024-02-21T23:10:08.234391+0000","last_fullsized":"2024-02-21T23:10:08.234391+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:10:21.186085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.33","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234540+0000","last_change":"2024-02-21T23:09:54.284818+0000","last_active":"2024-02-21T23:10:08.234540+0000","last_peered":"2024-02-21T23:10:08.234540+0000","last_clean":"2024-02-21T23:10:08.234540+0000","last_became_active":"2024-02-21T23:09:54.284161+0000","last_became_peered":"2024-02-21T23:09:54.284161+0000","last_unstale":"2024-02-21T23:10:08.234540+0000","last_undegraded":"2024-02-21T23:10:08.234540+0000","last_fullsized":"2024-02-21T23:10:08.234540+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:02:12.340921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.30","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.231818+0000","last_change":"2024-02-21T23:10:06.388372+0000","last_active":"2024-02-21T23:10:08.231818+0000","last_peered":"2024-02-21T23:10:08.231818+0000","last_clean":"2024-02-21T23:10:08.231818+0000","last_became_active":"2024-02-21T23:10:06.388099+0000","last_became_peered":"2024-02-21T23:10:06.388099+0000","last_unstale":"2024-02-21T23:10:08.231818+0000","last_undegraded":"2024-02-21T23:10:08.231818+0000","last_fullsized":"2024-02-21T23:10:08.231818+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:55:32.267269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.32","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.231579+0000","last_change":"2024-02-21T23:09:54.272136+0000","last_active":"2024-02-21T23:10:08.231579+0000","last_peered":"2024-02-21T23:10:08.231579+0000","last_clean":"2024-02-21T23:10:08.231579+0000","last_became_active":"2024-02-21T23:09:54.271097+0000","last_became_peered":"2024-02-21T23:09:54.271097+0000","last_unstale":"2024-02-21T23:10:08.231579+0000","last_undegraded":"2024-02-21T23:10:08.231579+0000","last_fullsized":"2024-02-21T23:10:08.231579+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:30:45.437334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.33","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.230766+0000","last_change":"2024-02-21T23:10:06.383867+0000","last_active":"2024-02-21T23:10:08.230766+0000","last_peered":"2024-02-21T23:10:08.230766+0000","last_clean":"2024-02-21T23:10:08.230766+0000","last_became_active":"2024-02-21T23:10:06.382862+0000","last_became_peered":"2024-02-21T23:10:06.382862+0000","last_unstale":"2024-02-21T23:10:08.230766+0000","last_undegraded":"2024-02-21T23:10:08.230766+0000","last_fullsized":"2024-02-21T23:10:08.230766+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:30:57.541364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.31","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.380179+0000","last_change":"2024-02-21T23:09:54.281268+0000","last_active":"2024-02-21T23:10:07.380179+0000","last_peered":"2024-02-21T23:10:07.380179+0000","last_clean":"2024-02-21T23:10:07.380179+0000","last_became_active":"2024-02-21T23:09:54.280946+0000","last_became_peered":"2024-02-21T23:09:54.280946+0000","last_unstale":"2024-02-21T23:10:07.380179+0000","last_undegraded":"2024-02-21T23:10:07.380179+0000","last_fullsized":"2024-02-21T23:10:07.380179+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:03:44.493909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.32","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.374763+0000","last_change":"2024-02-21T23:10:06.381260+0000","last_active":"2024-02-21T23:10:07.374763+0000","last_peered":"2024-02-21T23:10:07.374763+0000","last_clean":"2024-02-21T23:10:07.374763+0000","last_became_active":"2024-02-21T23:10:06.380936+0000","last_became_peered":"2024-02-21T23:10:06.380936+0000","last_unstale":"2024-02-21T23:10:07.374763+0000","last_undegraded":"2024-02-21T23:10:07.374763+0000","last_fullsized":"2024-02-21T23:10:07.374763+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:41:14.290670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.30","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.230478+0000","last_change":"2024-02-21T23:09:54.278731+0000","last_active":"2024-02-21T23:10:08.230478+0000","last_peered":"2024-02-21T23:10:08.230478+0000","last_clean":"2024-02-21T23:10:08.230478+0000","last_became_active":"2024-02-21T23:09:54.278390+0000","last_became_peered":"2024-02-21T23:09:54.278390+0000","last_unstale":"2024-02-21T23:10:08.230478+0000","last_undegraded":"2024-02-21T23:10:08.230478+0000","last_fullsized":"2024-02-21T23:10:08.230478+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:28:54.079553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.2d","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.231262+0000","last_change":"2024-02-21T23:10:06.367099+0000","last_active":"2024-02-21T23:10:08.231262+0000","last_peered":"2024-02-21T23:10:08.231262+0000","last_clean":"2024-02-21T23:10:08.231262+0000","last_became_active":"2024-02-21T23:10:06.366296+0000","last_became_peered":"2024-02-21T23:10:06.366296+0000","last_unstale":"2024-02-21T23:10:08.231262+0000","last_undegraded":"2024-02-21T23:10:08.231262+0000","last_fullsized":"2024-02-21T23:10:08.231262+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:47:46.169071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.2f","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.237800+0000","last_change":"2024-02-21T23:09:54.285947+0000","last_active":"2024-02-21T23:10:08.237800+0000","last_peered":"2024-02-21T23:10:08.237800+0000","last_clean":"2024-02-21T23:10:08.237800+0000","last_became_active":"2024-02-21T23:09:54.285658+0000","last_became_peered":"2024-02-21T23:09:54.285658+0000","last_unstale":"2024-02-21T23:10:08.237800+0000","last_undegraded":"2024-02-21T23:10:08.237800+0000","last_fullsized":"2024-02-21T23:10:08.237800+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:20:31.125684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2c","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236540+0000","last_change":"2024-02-21T23:10:06.376864+0000","last_active":"2024-02-21T23:10:08.236540+0000","last_peered":"2024-02-21T23:10:08.236540+0000","last_clean":"2024-02-21T23:10:08.236540+0000","last_became_active":"2024-02-21T23:10:06.376486+0000","last_became_peered":"2024-02-21T23:10:06.376486+0000","last_unstale":"2024-02-21T23:10:08.236540+0000","last_undegraded":"2024-02-21T23:10:08.236540+0000","last_fullsized":"2024-02-21T23:10:08.236540+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:22:04.227521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.2e","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.379953+0000","last_change":"2024-02-21T23:09:54.287286+0000","last_active":"2024-02-21T23:10:07.379953+0000","last_peered":"2024-02-21T23:10:07.379953+0000","last_clean":"2024-02-21T23:10:07.379953+0000","last_became_active":"2024-02-21T23:09:54.286924+0000","last_became_peered":"2024-02-21T23:09:54.286924+0000","last_unstale":"2024-02-21T23:10:07.379953+0000","last_undegraded":"2024-02-21T23:10:07.379953+0000","last_fullsized":"2024-02-21T23:10:07.379953+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:47:29.349669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.2f","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.235953+0000","last_change":"2024-02-21T23:10:06.377113+0000","last_active":"2024-02-21T23:10:08.235953+0000","last_peered":"2024-02-21T23:10:08.235953+0000","last_clean":"2024-02-21T23:10:08.235953+0000","last_became_active":"2024-02-21T23:10:06.376790+0000","last_became_peered":"2024-02-21T23:10:06.376790+0000","last_unstale":"2024-02-21T23:10:08.235953+0000","last_undegraded":"2024-02-21T23:10:08.235953+0000","last_fullsized":"2024-02-21T23:10:08.235953+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:38:03.009039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.2d","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.381461+0000","last_change":"2024-02-21T23:09:54.279884+0000","last_active":"2024-02-21T23:10:07.381461+0000","last_peered":"2024-02-21T23:10:07.381461+0000","last_clean":"2024-02-21T23:10:07.381461+0000","last_became_active":"2024-02-21T23:09:54.276608+0000","last_became_peered":"2024-02-21T23:09:54.276608+0000","last_unstale":"2024-02-21T23:10:07.381461+0000","last_undegraded":"2024-02-21T23:10:07.381461+0000","last_fullsized":"2024-02-21T23:10:07.381461+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:05:09.189420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.2e","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.231704+0000","last_change":"2024-02-21T23:10:06.378194+0000","last_active":"2024-02-21T23:10:08.231704+0000","last_peered":"2024-02-21T23:10:08.231704+0000","last_clean":"2024-02-21T23:10:08.231704+0000","last_became_active":"2024-02-21T23:10:06.377905+0000","last_became_peered":"2024-02-21T23:10:06.377905+0000","last_unstale":"2024-02-21T23:10:08.231704+0000","last_undegraded":"2024-02-21T23:10:08.231704+0000","last_fullsized":"2024-02-21T23:10:08.231704+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:36:54.026578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.2c","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.375678+0000","last_change":"2024-02-21T23:09:54.288944+0000","last_active":"2024-02-21T23:10:07.375678+0000","last_peered":"2024-02-21T23:10:07.375678+0000","last_clean":"2024-02-21T23:10:07.375678+0000","last_became_active":"2024-02-21T23:09:54.285913+0000","last_became_peered":"2024-02-21T23:09:54.285913+0000","last_unstale":"2024-02-21T23:10:07.375678+0000","last_undegraded":"2024-02-21T23:10:07.375678+0000","last_fullsized":"2024-02-21T23:10:07.375678+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:39:33.609273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.29","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.610265+0000","last_change":"2024-02-21T23:10:06.385773+0000","last_active":"2024-02-21T23:10:07.610265+0000","last_peered":"2024-02-21T23:10:07.610265+0000","last_clean":"2024-02-21T23:10:07.610265+0000","last_became_active":"2024-02-21T23:10:06.385462+0000","last_became_peered":"2024-02-21T23:10:06.385462+0000","last_unstale":"2024-02-21T23:10:07.610265+0000","last_undegraded":"2024-02-21T23:10:07.610265+0000","last_fullsized":"2024-02-21T23:10:07.610265+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:47:42.794742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.2b","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234769+0000","last_change":"2024-02-21T23:09:54.285036+0000","last_active":"2024-02-21T23:10:08.234769+0000","last_peered":"2024-02-21T23:10:08.234769+0000","last_clean":"2024-02-21T23:10:08.234769+0000","last_became_active":"2024-02-21T23:09:54.284542+0000","last_became_peered":"2024-02-21T23:09:54.284542+0000","last_unstale":"2024-02-21T23:10:08.234769+0000","last_undegraded":"2024-02-21T23:10:08.234769+0000","last_fullsized":"2024-02-21T23:10:08.234769+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T11:05:01.241636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.28","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234521+0000","last_change":"2024-02-21T23:10:06.396769+0000","last_active":"2024-02-21T23:10:08.234521+0000","last_peered":"2024-02-21T23:10:08.234521+0000","last_clean":"2024-02-21T23:10:08.234521+0000","last_became_active":"2024-02-21T23:10:06.384723+0000","last_became_peered":"2024-02-21T23:10:06.384723+0000","last_unstale":"2024-02-21T23:10:08.234521+0000","last_undegraded":"2024-02-21T23:10:08.234521+0000","last_fullsized":"2024-02-21T23:10:08.234521+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:45:13.867812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.2a","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.609154+0000","last_change":"2024-02-21T23:09:54.273350+0000","last_active":"2024-02-21T23:10:07.609154+0000","last_peered":"2024-02-21T23:10:07.609154+0000","last_clean":"2024-02-21T23:10:07.609154+0000","last_became_active":"2024-02-21T23:09:54.273022+0000","last_became_peered":"2024-02-21T23:09:54.273022+0000","last_unstale":"2024-02-21T23:10:07.609154+0000","last_undegraded":"2024-02-21T23:10:07.609154+0000","last_fullsized":"2024-02-21T23:10:07.609154+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:34:28.642087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.2b","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.235284+0000","last_change":"2024-02-21T23:10:06.394830+0000","last_active":"2024-02-21T23:10:08.235284+0000","last_peered":"2024-02-21T23:10:08.235284+0000","last_clean":"2024-02-21T23:10:08.235284+0000","last_became_active":"2024-02-21T23:10:06.386513+0000","last_became_peered":"2024-02-21T23:10:06.386513+0000","last_unstale":"2024-02-21T23:10:08.235284+0000","last_undegraded":"2024-02-21T23:10:08.235284+0000","last_fullsized":"2024-02-21T23:10:08.235284+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:30:09.767949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.29","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.230852+0000","last_change":"2024-02-21T23:09:54.279718+0000","last_active":"2024-02-21T23:10:08.230852+0000","last_peered":"2024-02-21T23:10:08.230852+0000","last_clean":"2024-02-21T23:10:08.230852+0000","last_became_active":"2024-02-21T23:09:54.278989+0000","last_became_peered":"2024-02-21T23:09:54.278989+0000","last_unstale":"2024-02-21T23:10:08.230852+0000","last_undegraded":"2024-02-21T23:10:08.230852+0000","last_fullsized":"2024-02-21T23:10:08.230852+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:36:41.922548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.2a","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.235326+0000","last_change":"2024-02-21T23:10:06.397414+0000","last_active":"2024-02-21T23:10:08.235326+0000","last_peered":"2024-02-21T23:10:08.235326+0000","last_clean":"2024-02-21T23:10:08.235326+0000","last_became_active":"2024-02-21T23:10:06.396982+0000","last_became_peered":"2024-02-21T23:10:06.396982+0000","last_unstale":"2024-02-21T23:10:08.235326+0000","last_undegraded":"2024-02-21T23:10:08.235326+0000","last_fullsized":"2024-02-21T23:10:08.235326+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:05:37.143681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.28","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.230583+0000","last_change":"2024-02-21T23:09:54.280624+0000","last_active":"2024-02-21T23:10:08.230583+0000","last_peered":"2024-02-21T23:10:08.230583+0000","last_clean":"2024-02-21T23:10:08.230583+0000","last_became_active":"2024-02-21T23:09:54.275644+0000","last_became_peered":"2024-02-21T23:09:54.275644+0000","last_unstale":"2024-02-21T23:10:08.230583+0000","last_undegraded":"2024-02-21T23:10:08.230583+0000","last_fullsized":"2024-02-21T23:10:08.230583+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:23:27.511315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.25","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.609295+0000","last_change":"2024-02-21T23:10:06.389786+0000","last_active":"2024-02-21T23:10:07.609295+0000","last_peered":"2024-02-21T23:10:07.609295+0000","last_clean":"2024-02-21T23:10:07.609295+0000","last_became_active":"2024-02-21T23:10:06.389447+0000","last_became_peered":"2024-02-21T23:10:06.389447+0000","last_unstale":"2024-02-21T23:10:07.609295+0000","last_undegraded":"2024-02-21T23:10:07.609295+0000","last_fullsized":"2024-02-21T23:10:07.609295+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:50:58.576033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.27","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236738+0000","last_change":"2024-02-21T23:09:54.275341+0000","last_active":"2024-02-21T23:10:08.236738+0000","last_peered":"2024-02-21T23:10:08.236738+0000","last_clean":"2024-02-21T23:10:08.236738+0000","last_became_active":"2024-02-21T23:09:54.275010+0000","last_became_peered":"2024-02-21T23:09:54.275010+0000","last_unstale":"2024-02-21T23:10:08.236738+0000","last_undegraded":"2024-02-21T23:10:08.236738+0000","last_fullsized":"2024-02-21T23:10:08.236738+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:42:08.123188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.24","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236163+0000","last_change":"2024-02-21T23:10:06.386616+0000","last_active":"2024-02-21T23:10:08.236163+0000","last_peered":"2024-02-21T23:10:08.236163+0000","last_clean":"2024-02-21T23:10:08.236163+0000","last_became_active":"2024-02-21T23:10:06.386339+0000","last_became_peered":"2024-02-21T23:10:06.386339+0000","last_unstale":"2024-02-21T23:10:08.236163+0000","last_undegraded":"2024-02-21T23:10:08.236163+0000","last_fullsized":"2024-02-21T23:10:08.236163+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:44:41.892333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.26","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.376034+0000","last_change":"2024-02-21T23:09:54.260902+0000","last_active":"2024-02-21T23:10:07.376034+0000","last_peered":"2024-02-21T23:10:07.376034+0000","last_clean":"2024-02-21T23:10:07.376034+0000","last_became_active":"2024-02-21T23:09:54.260084+0000","last_became_peered":"2024-02-21T23:09:54.260084+0000","last_unstale":"2024-02-21T23:10:07.376034+0000","last_undegraded":"2024-02-21T23:10:07.376034+0000","last_fullsized":"2024-02-21T23:10:07.376034+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:04:05.858817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.27","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.375938+0000","last_change":"2024-02-21T23:10:06.374537+0000","last_active":"2024-02-21T23:10:07.375938+0000","last_peered":"2024-02-21T23:10:07.375938+0000","last_clean":"2024-02-21T23:10:07.375938+0000","last_became_active":"2024-02-21T23:10:06.374201+0000","last_became_peered":"2024-02-21T23:10:06.374201+0000","last_unstale":"2024-02-21T23:10:07.375938+0000","last_undegraded":"2024-02-21T23:10:07.375938+0000","last_fullsized":"2024-02-21T23:10:07.375938+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:52:55.561353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.25","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.609643+0000","last_change":"2024-02-21T23:09:54.284116+0000","last_active":"2024-02-21T23:10:07.609643+0000","last_peered":"2024-02-21T23:10:07.609643+0000","last_clean":"2024-02-21T23:10:07.609643+0000","last_became_active":"2024-02-21T23:09:54.283802+0000","last_became_peered":"2024-02-21T23:09:54.283802+0000","last_unstale":"2024-02-21T23:10:07.609643+0000","last_undegraded":"2024-02-21T23:10:07.609643+0000","last_fullsized":"2024-02-21T23:10:07.609643+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:05:09.189420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.26","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.380096+0000","last_change":"2024-02-21T23:10:06.393845+0000","last_active":"2024-02-21T23:10:07.380096+0000","last_peered":"2024-02-21T23:10:07.380096+0000","last_clean":"2024-02-21T23:10:07.380096+0000","last_became_active":"2024-02-21T23:10:06.391431+0000","last_became_peered":"2024-02-21T23:10:06.391431+0000","last_unstale":"2024-02-21T23:10:07.380096+0000","last_undegraded":"2024-02-21T23:10:07.380096+0000","last_fullsized":"2024-02-21T23:10:07.380096+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:26:48.509050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.24","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236346+0000","last_change":"2024-02-21T23:09:54.287481+0000","last_active":"2024-02-21T23:10:08.236346+0000","last_peered":"2024-02-21T23:10:08.236346+0000","last_clean":"2024-02-21T23:10:08.236346+0000","last_became_active":"2024-02-21T23:09:54.287137+0000","last_became_peered":"2024-02-21T23:09:54.287137+0000","last_unstale":"2024-02-21T23:10:08.236346+0000","last_undegraded":"2024-02-21T23:10:08.236346+0000","last_fullsized":"2024-02-21T23:10:08.236346+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:50:46.472002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.21","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.374988+0000","last_change":"2024-02-21T23:10:06.370867+0000","last_active":"2024-02-21T23:10:07.374988+0000","last_peered":"2024-02-21T23:10:07.374988+0000","last_clean":"2024-02-21T23:10:07.374988+0000","last_became_active":"2024-02-21T23:10:06.370583+0000","last_became_peered":"2024-02-21T23:10:06.370583+0000","last_unstale":"2024-02-21T23:10:07.374988+0000","last_undegraded":"2024-02-21T23:10:07.374988+0000","last_fullsized":"2024-02-21T23:10:07.374988+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:18:29.588662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.23","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.610263+0000","last_change":"2024-02-21T23:09:54.281613+0000","last_active":"2024-02-21T23:10:07.610263+0000","last_peered":"2024-02-21T23:10:07.610263+0000","last_clean":"2024-02-21T23:10:07.610263+0000","last_became_active":"2024-02-21T23:09:54.281316+0000","last_became_peered":"2024-02-21T23:09:54.281316+0000","last_unstale":"2024-02-21T23:10:07.610263+0000","last_undegraded":"2024-02-21T23:10:07.610263+0000","last_fullsized":"2024-02-21T23:10:07.610263+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:13:33.442227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.20","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232111+0000","last_change":"2024-02-21T23:10:06.382019+0000","last_active":"2024-02-21T23:10:08.232111+0000","last_peered":"2024-02-21T23:10:08.232111+0000","last_clean":"2024-02-21T23:10:08.232111+0000","last_became_active":"2024-02-21T23:10:06.381739+0000","last_became_peered":"2024-02-21T23:10:06.381739+0000","last_unstale":"2024-02-21T23:10:08.232111+0000","last_undegraded":"2024-02-21T23:10:08.232111+0000","last_fullsized":"2024-02-21T23:10:08.232111+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:25:17.018194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.22","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.379703+0000","last_change":"2024-02-21T23:09:54.275665+0000","last_active":"2024-02-21T23:10:07.379703+0000","last_peered":"2024-02-21T23:10:07.379703+0000","last_clean":"2024-02-21T23:10:07.379703+0000","last_became_active":"2024-02-21T23:09:54.275143+0000","last_became_peered":"2024-02-21T23:09:54.275143+0000","last_unstale":"2024-02-21T23:10:07.379703+0000","last_undegraded":"2024-02-21T23:10:07.379703+0000","last_fullsized":"2024-02-21T23:10:07.379703+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:37:42.976792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.23","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.380797+0000","last_change":"2024-02-21T23:10:06.398551+0000","last_active":"2024-02-21T23:10:07.380797+0000","last_peered":"2024-02-21T23:10:07.380797+0000","last_clean":"2024-02-21T23:10:07.380797+0000","last_became_active":"2024-02-21T23:10:06.397827+0000","last_became_peered":"2024-02-21T23:10:06.397827+0000","last_unstale":"2024-02-21T23:10:07.380797+0000","last_undegraded":"2024-02-21T23:10:07.380797+0000","last_fullsized":"2024-02-21T23:10:07.380797+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:42:20.227218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.21","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.380298+0000","last_change":"2024-02-21T23:09:54.281200+0000","last_active":"2024-02-21T23:10:07.380298+0000","last_peered":"2024-02-21T23:10:07.380298+0000","last_clean":"2024-02-21T23:10:07.380298+0000","last_became_active":"2024-02-21T23:09:54.277867+0000","last_became_peered":"2024-02-21T23:09:54.277867+0000","last_unstale":"2024-02-21T23:10:07.380298+0000","last_undegraded":"2024-02-21T23:10:07.380298+0000","last_fullsized":"2024-02-21T23:10:07.380298+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:59:34.646775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.22","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.381831+0000","last_change":"2024-02-21T23:10:06.398306+0000","last_active":"2024-02-21T23:10:07.381831+0000","last_peered":"2024-02-21T23:10:07.381831+0000","last_clean":"2024-02-21T23:10:07.381831+0000","last_became_active":"2024-02-21T23:10:06.397841+0000","last_became_peered":"2024-02-21T23:10:06.397841+0000","last_unstale":"2024-02-21T23:10:07.381831+0000","last_undegraded":"2024-02-21T23:10:07.381831+0000","last_fullsized":"2024-02-21T23:10:07.381831+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:06:55.637480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.20","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.379915+0000","last_change":"2024-02-21T23:09:54.276078+0000","last_active":"2024-02-21T23:10:07.379915+0000","last_peered":"2024-02-21T23:10:07.379915+0000","last_clean":"2024-02-21T23:10:07.379915+0000","last_became_active":"2024-02-21T23:09:54.275724+0000","last_became_peered":"2024-02-21T23:09:54.275724+0000","last_unstale":"2024-02-21T23:10:07.379915+0000","last_undegraded":"2024-02-21T23:10:07.379915+0000","last_fullsized":"2024-02-21T23:10:07.379915+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:48:07.369182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.230483+0000","last_change":"2024-02-21T23:10:06.380827+0000","last_active":"2024-02-21T23:10:08.230483+0000","last_peered":"2024-02-21T23:10:08.230483+0000","last_clean":"2024-02-21T23:10:08.230483+0000","last_became_active":"2024-02-21T23:10:06.380551+0000","last_became_peered":"2024-02-21T23:10:06.380551+0000","last_unstale":"2024-02-21T23:10:08.230483+0000","last_undegraded":"2024-02-21T23:10:08.230483+0000","last_fullsized":"2024-02-21T23:10:08.230483+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:34:11.889581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.230111+0000","last_change":"2024-02-21T23:10:05.385230+0000","last_active":"2024-02-21T23:10:08.230111+0000","last_peered":"2024-02-21T23:10:08.230111+0000","last_clean":"2024-02-21T23:10:08.230111+0000","last_became_active":"2024-02-21T23:10:05.384017+0000","last_became_peered":"2024-02-21T23:10:05.384017+0000","last_unstale":"2024-02-21T23:10:08.230111+0000","last_undegraded":"2024-02-21T23:10:08.230111+0000","last_fullsized":"2024-02-21T23:10:08.230111+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:45:12.852122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.10","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.609579+0000","last_change":"2024-02-21T23:09:54.278972+0000","last_active":"2024-02-21T23:10:07.609579+0000","last_peered":"2024-02-21T23:10:07.609579+0000","last_clean":"2024-02-21T23:10:07.609579+0000","last_became_active":"2024-02-21T23:09:54.278146+0000","last_became_peered":"2024-02-21T23:09:54.278146+0000","last_unstale":"2024-02-21T23:10:07.609579+0000","last_undegraded":"2024-02-21T23:10:07.609579+0000","last_fullsized":"2024-02-21T23:10:07.609579+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:59:34.646775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"150'4","reported_seq":26,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232604+0000","last_change":"2024-02-21T23:09:53.288506+0000","last_active":"2024-02-21T23:10:08.232604+0000","last_peered":"2024-02-21T23:10:08.232604+0000","last_clean":"2024-02-21T23:10:08.232604+0000","last_became_active":"2024-02-21T23:09:53.288199+0000","last_became_peered":"2024-02-21T23:09:53.288199+0000","last_unstale":"2024-02-21T23:10:08.232604+0000","last_undegraded":"2024-02-21T23:10:08.232604+0000","last_fullsized":"2024-02-21T23:10:08.232604+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:18:16.467715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.13","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232017+0000","last_change":"2024-02-21T23:10:06.396696+0000","last_active":"2024-02-21T23:10:08.232017+0000","last_peered":"2024-02-21T23:10:08.232017+0000","last_clean":"2024-02-21T23:10:08.232017+0000","last_became_active":"2024-02-21T23:10:06.396394+0000","last_became_peered":"2024-02-21T23:10:06.396394+0000","last_unstale":"2024-02-21T23:10:08.232017+0000","last_undegraded":"2024-02-21T23:10:08.232017+0000","last_fullsized":"2024-02-21T23:10:08.232017+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:40:04.184614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.10","version":"154'8","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.380094+0000","last_change":"2024-02-21T23:10:05.382619+0000","last_active":"2024-02-21T23:10:07.380094+0000","last_peered":"2024-02-21T23:10:07.380094+0000","last_clean":"2024-02-21T23:10:07.380094+0000","last_became_active":"2024-02-21T23:10:05.382336+0000","last_became_peered":"2024-02-21T23:10:05.382336+0000","last_unstale":"2024-02-21T23:10:07.380094+0000","last_undegraded":"2024-02-21T23:10:07.380094+0000","last_fullsized":"2024-02-21T23:10:07.380094+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:30:56.525674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.231615+0000","last_change":"2024-02-21T23:09:54.278032+0000","last_active":"2024-02-21T23:10:08.231615+0000","last_peered":"2024-02-21T23:10:08.231615+0000","last_clean":"2024-02-21T23:10:08.231615+0000","last_became_active":"2024-02-21T23:09:54.277741+0000","last_became_peered":"2024-02-21T23:09:54.277741+0000","last_unstale":"2024-02-21T23:10:08.231615+0000","last_undegraded":"2024-02-21T23:10:08.231615+0000","last_fullsized":"2024-02-21T23:10:08.231615+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:43:56.610613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"150'1","reported_seq":23,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232900+0000","last_change":"2024-02-21T23:09:53.279374+0000","last_active":"2024-02-21T23:10:08.232900+0000","last_peered":"2024-02-21T23:10:08.232900+0000","last_clean":"2024-02-21T23:10:08.232900+0000","last_became_active":"2024-02-21T23:09:53.279053+0000","last_became_peered":"2024-02-21T23:09:53.279053+0000","last_unstale":"2024-02-21T23:10:08.232900+0000","last_undegraded":"2024-02-21T23:10:08.232900+0000","last_fullsized":"2024-02-21T23:10:08.232900+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:04:15.300172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.10","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.375571+0000","last_change":"2024-02-21T23:10:06.375103+0000","last_active":"2024-02-21T23:10:07.375571+0000","last_peered":"2024-02-21T23:10:07.375571+0000","last_clean":"2024-02-21T23:10:07.375571+0000","last_became_active":"2024-02-21T23:10:06.374814+0000","last_became_peered":"2024-02-21T23:10:06.374814+0000","last_unstale":"2024-02-21T23:10:07.375571+0000","last_undegraded":"2024-02-21T23:10:07.375571+0000","last_fullsized":"2024-02-21T23:10:07.375571+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:42:32.100185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.13","version":"154'2","reported_seq":16,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.231271+0000","last_change":"2024-02-21T23:10:05.392497+0000","last_active":"2024-02-21T23:10:08.231271+0000","last_peered":"2024-02-21T23:10:08.231271+0000","last_clean":"2024-02-21T23:10:08.231271+0000","last_became_active":"2024-02-21T23:10:05.392183+0000","last_became_peered":"2024-02-21T23:10:05.392183+0000","last_unstale":"2024-02-21T23:10:08.231271+0000","last_undegraded":"2024-02-21T23:10:08.231271+0000","last_fullsized":"2024-02-21T23:10:08.231271+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:13:38.400263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.231985+0000","last_change":"2024-02-21T23:09:54.263915+0000","last_active":"2024-02-21T23:10:08.231985+0000","last_peered":"2024-02-21T23:10:08.231985+0000","last_clean":"2024-02-21T23:10:08.231985+0000","last_became_active":"2024-02-21T23:09:54.263634+0000","last_became_peered":"2024-02-21T23:09:54.263634+0000","last_unstale":"2024-02-21T23:10:08.231985+0000","last_undegraded":"2024-02-21T23:10:08.231985+0000","last_fullsized":"2024-02-21T23:10:08.231985+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:47:34.065041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.1d","version":"150'2","reported_seq":24,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.381587+0000","last_change":"2024-02-21T23:09:53.272110+0000","last_active":"2024-02-21T23:10:07.381587+0000","last_peered":"2024-02-21T23:10:07.381587+0000","last_clean":"2024-02-21T23:10:07.381587+0000","last_became_active":"2024-02-21T23:09:53.271641+0000","last_became_peered":"2024-02-21T23:09:53.271641+0000","last_unstale":"2024-02-21T23:10:07.381587+0000","last_undegraded":"2024-02-21T23:10:07.381587+0000","last_fullsized":"2024-02-21T23:10:07.381587+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:46:34.419898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.231438+0000","last_change":"2024-02-21T23:10:06.380458+0000","last_active":"2024-02-21T23:10:08.231438+0000","last_peered":"2024-02-21T23:10:08.231438+0000","last_clean":"2024-02-21T23:10:08.231438+0000","last_became_active":"2024-02-21T23:10:06.380137+0000","last_became_peered":"2024-02-21T23:10:06.380137+0000","last_unstale":"2024-02-21T23:10:08.231438+0000","last_undegraded":"2024-02-21T23:10:08.231438+0000","last_fullsized":"2024-02-21T23:10:08.231438+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:51:13.639406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236172+0000","last_change":"2024-02-21T23:10:05.393823+0000","last_active":"2024-02-21T23:10:08.236172+0000","last_peered":"2024-02-21T23:10:08.236172+0000","last_clean":"2024-02-21T23:10:08.236172+0000","last_became_active":"2024-02-21T23:10:05.390445+0000","last_became_peered":"2024-02-21T23:10:05.390445+0000","last_unstale":"2024-02-21T23:10:08.236172+0000","last_undegraded":"2024-02-21T23:10:08.236172+0000","last_fullsized":"2024-02-21T23:10:08.236172+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:30:56.525674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.13","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.610343+0000","last_change":"2024-02-21T23:09:54.275867+0000","last_active":"2024-02-21T23:10:07.610343+0000","last_peered":"2024-02-21T23:10:07.610343+0000","last_clean":"2024-02-21T23:10:07.610343+0000","last_became_active":"2024-02-21T23:09:54.275591+0000","last_became_peered":"2024-02-21T23:09:54.275591+0000","last_unstale":"2024-02-21T23:10:07.610343+0000","last_undegraded":"2024-02-21T23:10:07.610343+0000","last_fullsized":"2024-02-21T23:10:07.610343+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:42:16.988326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1c","version":"150'2","reported_seq":24,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.375729+0000","last_change":"2024-02-21T23:09:53.274692+0000","last_active":"2024-02-21T23:10:07.375729+0000","last_peered":"2024-02-21T23:10:07.375729+0000","last_clean":"2024-02-21T23:10:07.375729+0000","last_became_active":"2024-02-21T23:09:53.274395+0000","last_became_peered":"2024-02-21T23:09:53.274395+0000","last_unstale":"2024-02-21T23:10:07.375729+0000","last_undegraded":"2024-02-21T23:10:07.375729+0000","last_fullsized":"2024-02-21T23:10:07.375729+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:40:18.453921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.231853+0000","last_change":"2024-02-21T23:10:06.384222+0000","last_active":"2024-02-21T23:10:08.231853+0000","last_peered":"2024-02-21T23:10:08.231853+0000","last_clean":"2024-02-21T23:10:08.231853+0000","last_became_active":"2024-02-21T23:10:06.383621+0000","last_became_peered":"2024-02-21T23:10:06.383621+0000","last_unstale":"2024-02-21T23:10:08.231853+0000","last_undegraded":"2024-02-21T23:10:08.231853+0000","last_fullsized":"2024-02-21T23:10:08.231853+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:16:22.900582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.15","version":"154'3","reported_seq":17,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.233612+0000","last_change":"2024-02-21T23:10:05.374604+0000","last_active":"2024-02-21T23:10:08.233612+0000","last_peered":"2024-02-21T23:10:08.233612+0000","last_clean":"2024-02-21T23:10:08.233612+0000","last_became_active":"2024-02-21T23:10:05.374208+0000","last_became_peered":"2024-02-21T23:10:05.374208+0000","last_unstale":"2024-02-21T23:10:08.233612+0000","last_undegraded":"2024-02-21T23:10:08.233612+0000","last_fullsized":"2024-02-21T23:10:08.233612+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:30:08.752259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.14","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.235541+0000","last_change":"2024-02-21T23:09:54.289092+0000","last_active":"2024-02-21T23:10:08.235541+0000","last_peered":"2024-02-21T23:10:08.235541+0000","last_clean":"2024-02-21T23:10:08.235541+0000","last_became_active":"2024-02-21T23:09:54.288801+0000","last_became_peered":"2024-02-21T23:09:54.288801+0000","last_unstale":"2024-02-21T23:10:08.235541+0000","last_undegraded":"2024-02-21T23:10:08.235541+0000","last_fullsized":"2024-02-21T23:10:08.235541+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:05:09.189420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1b","version":"150'1","reported_seq":23,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236011+0000","last_change":"2024-02-21T23:09:53.283422+0000","last_active":"2024-02-21T23:10:08.236011+0000","last_peered":"2024-02-21T23:10:08.236011+0000","last_clean":"2024-02-21T23:10:08.236011+0000","last_became_active":"2024-02-21T23:09:53.283140+0000","last_became_peered":"2024-02-21T23:09:53.283140+0000","last_unstale":"2024-02-21T23:10:08.236011+0000","last_undegraded":"2024-02-21T23:10:08.236011+0000","last_fullsized":"2024-02-21T23:10:08.236011+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:46:34.419898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.237280+0000","last_change":"2024-02-21T23:10:06.403933+0000","last_active":"2024-02-21T23:10:08.237280+0000","last_peered":"2024-02-21T23:10:08.237280+0000","last_clean":"2024-02-21T23:10:08.237280+0000","last_became_active":"2024-02-21T23:10:06.403548+0000","last_became_peered":"2024-02-21T23:10:06.403548+0000","last_unstale":"2024-02-21T23:10:08.237280+0000","last_undegraded":"2024-02-21T23:10:08.237280+0000","last_fullsized":"2024-02-21T23:10:08.237280+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:11:20.176324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.14","version":"154'6","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.230908+0000","last_change":"2024-02-21T23:10:05.388142+0000","last_active":"2024-02-21T23:10:08.230908+0000","last_peered":"2024-02-21T23:10:08.230908+0000","last_clean":"2024-02-21T23:10:08.230908+0000","last_became_active":"2024-02-21T23:10:05.387764+0000","last_became_peered":"2024-02-21T23:10:05.387764+0000","last_unstale":"2024-02-21T23:10:08.230908+0000","last_undegraded":"2024-02-21T23:10:08.230908+0000","last_fullsized":"2024-02-21T23:10:08.230908+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:03:55.582249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.15","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.375532+0000","last_change":"2024-02-21T23:09:54.273728+0000","last_active":"2024-02-21T23:10:07.375532+0000","last_peered":"2024-02-21T23:10:07.375532+0000","last_clean":"2024-02-21T23:10:07.375532+0000","last_became_active":"2024-02-21T23:09:54.273343+0000","last_became_peered":"2024-02-21T23:09:54.273343+0000","last_unstale":"2024-02-21T23:10:07.375532+0000","last_undegraded":"2024-02-21T23:10:07.375532+0000","last_fullsized":"2024-02-21T23:10:07.375532+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:49:15.552003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.1a","version":"150'1","reported_seq":23,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232153+0000","last_change":"2024-02-21T23:09:53.278176+0000","last_active":"2024-02-21T23:10:08.232153+0000","last_peered":"2024-02-21T23:10:08.232153+0000","last_clean":"2024-02-21T23:10:08.232153+0000","last_became_active":"2024-02-21T23:09:53.277880+0000","last_became_peered":"2024-02-21T23:09:53.277880+0000","last_unstale":"2024-02-21T23:10:08.232153+0000","last_undegraded":"2024-02-21T23:10:08.232153+0000","last_fullsized":"2024-02-21T23:10:08.232153+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:05:08.172503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.14","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.610388+0000","last_change":"2024-02-21T23:10:06.397559+0000","last_active":"2024-02-21T23:10:07.610388+0000","last_peered":"2024-02-21T23:10:07.610388+0000","last_clean":"2024-02-21T23:10:07.610388+0000","last_became_active":"2024-02-21T23:10:06.396499+0000","last_became_peered":"2024-02-21T23:10:06.396499+0000","last_unstale":"2024-02-21T23:10:07.610388+0000","last_undegraded":"2024-02-21T23:10:07.610388+0000","last_fullsized":"2024-02-21T23:10:07.610388+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:20:39.771135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236121+0000","last_change":"2024-02-21T23:10:05.375277+0000","last_active":"2024-02-21T23:10:08.236121+0000","last_peered":"2024-02-21T23:10:08.236121+0000","last_clean":"2024-02-21T23:10:08.236121+0000","last_became_active":"2024-02-21T23:10:05.374983+0000","last_became_peered":"2024-02-21T23:10:05.374983+0000","last_unstale":"2024-02-21T23:10:08.236121+0000","last_undegraded":"2024-02-21T23:10:08.236121+0000","last_fullsized":"2024-02-21T23:10:08.236121+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:11:19.160634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.376102+0000","last_change":"2024-02-21T23:09:54.278094+0000","last_active":"2024-02-21T23:10:07.376102+0000","last_peered":"2024-02-21T23:10:07.376102+0000","last_clean":"2024-02-21T23:10:07.376102+0000","last_became_active":"2024-02-21T23:09:54.276160+0000","last_became_peered":"2024-02-21T23:09:54.276160+0000","last_unstale":"2024-02-21T23:10:07.376102+0000","last_undegraded":"2024-02-21T23:10:07.376102+0000","last_fullsized":"2024-02-21T23:10:07.376102+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:15:07.339478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.375244+0000","last_change":"2024-02-21T23:09:53.280005+0000","last_active":"2024-02-21T23:10:07.375244+0000","last_peered":"2024-02-21T23:10:07.375244+0000","last_clean":"2024-02-21T23:10:07.375244+0000","last_became_active":"2024-02-21T23:09:53.279689+0000","last_became_peered":"2024-02-21T23:09:53.279689+0000","last_unstale":"2024-02-21T23:10:07.375244+0000","last_undegraded":"2024-02-21T23:10:07.375244+0000","last_fullsized":"2024-02-21T23:10:07.375244+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:40:05.561558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.15","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234855+0000","last_change":"2024-02-21T23:10:06.384066+0000","last_active":"2024-02-21T23:10:08.234855+0000","last_peered":"2024-02-21T23:10:08.234855+0000","last_clean":"2024-02-21T23:10:08.234855+0000","last_became_active":"2024-02-21T23:10:06.383780+0000","last_became_peered":"2024-02-21T23:10:06.383780+0000","last_unstale":"2024-02-21T23:10:08.234855+0000","last_undegraded":"2024-02-21T23:10:08.234855+0000","last_fullsized":"2024-02-21T23:10:08.234855+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:24:37.460238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236590+0000","last_change":"2024-02-21T23:10:05.393045+0000","last_active":"2024-02-21T23:10:08.236590+0000","last_peered":"2024-02-21T23:10:08.236590+0000","last_clean":"2024-02-21T23:10:08.236590+0000","last_became_active":"2024-02-21T23:10:05.392683+0000","last_became_peered":"2024-02-21T23:10:05.392683+0000","last_unstale":"2024-02-21T23:10:08.236590+0000","last_undegraded":"2024-02-21T23:10:08.236590+0000","last_fullsized":"2024-02-21T23:10:08.236590+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:45:39.918602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232441+0000","last_change":"2024-02-21T23:09:54.262565+0000","last_active":"2024-02-21T23:10:08.232441+0000","last_peered":"2024-02-21T23:10:08.232441+0000","last_clean":"2024-02-21T23:10:08.232441+0000","last_became_active":"2024-02-21T23:09:54.261897+0000","last_became_peered":"2024-02-21T23:09:54.261897+0000","last_unstale":"2024-02-21T23:10:08.232441+0000","last_undegraded":"2024-02-21T23:10:08.232441+0000","last_fullsized":"2024-02-21T23:10:08.232441+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:08:50.519955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234939+0000","last_change":"2024-02-21T23:09:53.275695+0000","last_active":"2024-02-21T23:10:08.234939+0000","last_peered":"2024-02-21T23:10:08.234939+0000","last_clean":"2024-02-21T23:10:08.234939+0000","last_became_active":"2024-02-21T23:09:53.275343+0000","last_became_peered":"2024-02-21T23:09:53.275343+0000","last_unstale":"2024-02-21T23:10:08.234939+0000","last_undegraded":"2024-02-21T23:10:08.234939+0000","last_fullsized":"2024-02-21T23:10:08.234939+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:28:53.062636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1a","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.235787+0000","last_change":"2024-02-21T23:10:06.403716+0000","last_active":"2024-02-21T23:10:08.235787+0000","last_peered":"2024-02-21T23:10:08.235787+0000","last_clean":"2024-02-21T23:10:08.235787+0000","last_became_active":"2024-02-21T23:10:06.403226+0000","last_became_peered":"2024-02-21T23:10:06.403226+0000","last_unstale":"2024-02-21T23:10:08.235787+0000","last_undegraded":"2024-02-21T23:10:08.235787+0000","last_fullsized":"2024-02-21T23:10:08.235787+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:28:15.518153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.230984+0000","last_change":"2024-02-21T23:10:05.390846+0000","last_active":"2024-02-21T23:10:08.230984+0000","last_peered":"2024-02-21T23:10:08.230984+0000","last_clean":"2024-02-21T23:10:08.230984+0000","last_became_active":"2024-02-21T23:10:05.388088+0000","last_became_peered":"2024-02-21T23:10:05.388088+0000","last_unstale":"2024-02-21T23:10:08.230984+0000","last_undegraded":"2024-02-21T23:10:08.230984+0000","last_fullsized":"2024-02-21T23:10:08.230984+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:59:45.735115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.608719+0000","last_change":"2024-02-21T23:09:54.279220+0000","last_active":"2024-02-21T23:10:07.608719+0000","last_peered":"2024-02-21T23:10:07.608719+0000","last_clean":"2024-02-21T23:10:07.608719+0000","last_became_active":"2024-02-21T23:09:54.278653+0000","last_became_peered":"2024-02-21T23:09:54.278653+0000","last_unstale":"2024-02-21T23:10:07.608719+0000","last_undegraded":"2024-02-21T23:10:07.608719+0000","last_fullsized":"2024-02-21T23:10:07.608719+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:13:27.311923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236554+0000","last_change":"2024-02-21T23:09:53.288439+0000","last_active":"2024-02-21T23:10:08.236554+0000","last_peered":"2024-02-21T23:10:08.236554+0000","last_clean":"2024-02-21T23:10:08.236554+0000","last_became_active":"2024-02-21T23:09:53.287601+0000","last_became_peered":"2024-02-21T23:09:53.287601+0000","last_unstale":"2024-02-21T23:10:08.236554+0000","last_undegraded":"2024-02-21T23:10:08.236554+0000","last_fullsized":"2024-02-21T23:10:08.236554+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:32:05.045159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.230304+0000","last_change":"2024-02-21T23:10:06.373866+0000","last_active":"2024-02-21T23:10:08.230304+0000","last_peered":"2024-02-21T23:10:08.230304+0000","last_clean":"2024-02-21T23:10:08.230304+0000","last_became_active":"2024-02-21T23:10:06.373575+0000","last_became_peered":"2024-02-21T23:10:06.373575+0000","last_unstale":"2024-02-21T23:10:08.230304+0000","last_undegraded":"2024-02-21T23:10:08.230304+0000","last_fullsized":"2024-02-21T23:10:08.230304+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:11:56.693636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.1b","version":"154'1","reported_seq":15,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.233920+0000","last_change":"2024-02-21T23:10:05.387544+0000","last_active":"2024-02-21T23:10:08.233920+0000","last_peered":"2024-02-21T23:10:08.233920+0000","last_clean":"2024-02-21T23:10:08.233920+0000","last_became_active":"2024-02-21T23:10:05.385992+0000","last_became_peered":"2024-02-21T23:10:05.385992+0000","last_unstale":"2024-02-21T23:10:08.233920+0000","last_undegraded":"2024-02-21T23:10:08.233920+0000","last_fullsized":"2024-02-21T23:10:08.233920+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:28:14.502463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232148+0000","last_change":"2024-02-21T23:09:54.280274+0000","last_active":"2024-02-21T23:10:08.232148+0000","last_peered":"2024-02-21T23:10:08.232148+0000","last_clean":"2024-02-21T23:10:08.232148+0000","last_became_active":"2024-02-21T23:09:54.279951+0000","last_became_peered":"2024-02-21T23:09:54.279951+0000","last_unstale":"2024-02-21T23:10:08.232148+0000","last_undegraded":"2024-02-21T23:10:08.232148+0000","last_fullsized":"2024-02-21T23:10:08.232148+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:15:08.084683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.15","version":"150'3","reported_seq":25,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232015+0000","last_change":"2024-02-21T23:09:53.279389+0000","last_active":"2024-02-21T23:10:08.232015+0000","last_peered":"2024-02-21T23:10:08.232015+0000","last_clean":"2024-02-21T23:10:08.232015+0000","last_became_active":"2024-02-21T23:09:53.279042+0000","last_became_peered":"2024-02-21T23:09:53.279042+0000","last_unstale":"2024-02-21T23:10:08.232015+0000","last_undegraded":"2024-02-21T23:10:08.232015+0000","last_fullsized":"2024-02-21T23:10:08.232015+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:44:44.898339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.233266+0000","last_change":"2024-02-21T23:10:06.379758+0000","last_active":"2024-02-21T23:10:08.233266+0000","last_peered":"2024-02-21T23:10:08.233266+0000","last_clean":"2024-02-21T23:10:08.233266+0000","last_became_active":"2024-02-21T23:10:06.379416+0000","last_became_peered":"2024-02-21T23:10:06.379416+0000","last_unstale":"2024-02-21T23:10:08.233266+0000","last_undegraded":"2024-02-21T23:10:08.233266+0000","last_fullsized":"2024-02-21T23:10:08.233266+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:06:55.637480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1a","version":"154'1","reported_seq":15,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.231363+0000","last_change":"2024-02-21T23:10:05.386289+0000","last_active":"2024-02-21T23:10:08.231363+0000","last_peered":"2024-02-21T23:10:08.231363+0000","last_clean":"2024-02-21T23:10:08.231363+0000","last_became_active":"2024-02-21T23:10:05.384200+0000","last_became_peered":"2024-02-21T23:10:05.384200+0000","last_unstale":"2024-02-21T23:10:08.231363+0000","last_undegraded":"2024-02-21T23:10:08.231363+0000","last_fullsized":"2024-02-21T23:10:08.231363+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:24:31.837524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1b","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.375019+0000","last_change":"2024-02-21T23:09:54.285563+0000","last_active":"2024-02-21T23:10:07.375019+0000","last_peered":"2024-02-21T23:10:07.375019+0000","last_clean":"2024-02-21T23:10:07.375019+0000","last_became_active":"2024-02-21T23:09:54.285275+0000","last_became_peered":"2024-02-21T23:09:54.285275+0000","last_unstale":"2024-02-21T23:10:07.375019+0000","last_undegraded":"2024-02-21T23:10:07.375019+0000","last_fullsized":"2024-02-21T23:10:07.375019+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:21:42.014406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"150'6","reported_seq":28,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236789+0000","last_change":"2024-02-21T23:09:53.288423+0000","last_active":"2024-02-21T23:10:08.236789+0000","last_peered":"2024-02-21T23:10:08.236789+0000","last_clean":"2024-02-21T23:10:08.236789+0000","last_became_active":"2024-02-21T23:09:53.287473+0000","last_became_peered":"2024-02-21T23:09:53.287473+0000","last_unstale":"2024-02-21T23:10:08.236789+0000","last_undegraded":"2024-02-21T23:10:08.236789+0000","last_fullsized":"2024-02-21T23:10:08.236789+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:13:32.425310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1e","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.380496+0000","last_change":"2024-02-21T23:10:06.389938+0000","last_active":"2024-02-21T23:10:07.380496+0000","last_peered":"2024-02-21T23:10:07.380496+0000","last_clean":"2024-02-21T23:10:07.380496+0000","last_became_active":"2024-02-21T23:10:06.387554+0000","last_became_peered":"2024-02-21T23:10:06.387554+0000","last_unstale":"2024-02-21T23:10:07.380496+0000","last_undegraded":"2024-02-21T23:10:07.380496+0000","last_fullsized":"2024-02-21T23:10:07.380496+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:44:08.714643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1d","version":"154'2","reported_seq":16,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.381075+0000","last_change":"2024-02-21T23:10:05.388219+0000","last_active":"2024-02-21T23:10:07.381075+0000","last_peered":"2024-02-21T23:10:07.381075+0000","last_clean":"2024-02-21T23:10:07.381075+0000","last_became_active":"2024-02-21T23:10:05.387863+0000","last_became_peered":"2024-02-21T23:10:05.387863+0000","last_unstale":"2024-02-21T23:10:07.381075+0000","last_undegraded":"2024-02-21T23:10:07.381075+0000","last_fullsized":"2024-02-21T23:10:07.381075+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:16:21.884892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1c","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.379732+0000","last_change":"2024-02-21T23:09:54.278618+0000","last_active":"2024-02-21T23:10:07.379732+0000","last_peered":"2024-02-21T23:10:07.379732+0000","last_clean":"2024-02-21T23:10:07.379732+0000","last_became_active":"2024-02-21T23:09:54.278343+0000","last_became_peered":"2024-02-21T23:09:54.278343+0000","last_unstale":"2024-02-21T23:10:07.379732+0000","last_undegraded":"2024-02-21T23:10:07.379732+0000","last_fullsized":"2024-02-21T23:10:07.379732+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:49:31.759374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.13","version":"150'2","reported_seq":24,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234083+0000","last_change":"2024-02-21T23:09:53.284172+0000","last_active":"2024-02-21T23:10:08.234083+0000","last_peered":"2024-02-21T23:10:08.234083+0000","last_clean":"2024-02-21T23:10:08.234083+0000","last_became_active":"2024-02-21T23:09:53.283507+0000","last_became_peered":"2024-02-21T23:09:53.283507+0000","last_unstale":"2024-02-21T23:10:08.234083+0000","last_undegraded":"2024-02-21T23:10:08.234083+0000","last_fullsized":"2024-02-21T23:10:08.234083+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:20:30.108767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1f","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.381418+0000","last_change":"2024-02-21T23:10:06.388266+0000","last_active":"2024-02-21T23:10:07.381418+0000","last_peered":"2024-02-21T23:10:07.381418+0000","last_clean":"2024-02-21T23:10:07.381418+0000","last_became_active":"2024-02-21T23:10:06.387844+0000","last_became_peered":"2024-02-21T23:10:06.387844+0000","last_unstale":"2024-02-21T23:10:07.381418+0000","last_undegraded":"2024-02-21T23:10:07.381418+0000","last_fullsized":"2024-02-21T23:10:07.381418+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:13:39.415953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1c","version":"154'2","reported_seq":16,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.231548+0000","last_change":"2024-02-21T23:10:05.387736+0000","last_active":"2024-02-21T23:10:08.231548+0000","last_peered":"2024-02-21T23:10:08.231548+0000","last_clean":"2024-02-21T23:10:08.231548+0000","last_became_active":"2024-02-21T23:10:05.387416+0000","last_became_peered":"2024-02-21T23:10:05.387416+0000","last_unstale":"2024-02-21T23:10:08.231548+0000","last_undegraded":"2024-02-21T23:10:08.231548+0000","last_fullsized":"2024-02-21T23:10:08.231548+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:34:39.730427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1d","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236368+0000","last_change":"2024-02-21T23:09:54.291717+0000","last_active":"2024-02-21T23:10:08.236368+0000","last_peered":"2024-02-21T23:10:08.236368+0000","last_clean":"2024-02-21T23:10:08.236368+0000","last_became_active":"2024-02-21T23:09:54.290842+0000","last_became_peered":"2024-02-21T23:09:54.290842+0000","last_unstale":"2024-02-21T23:10:08.236368+0000","last_undegraded":"2024-02-21T23:10:08.236368+0000","last_fullsized":"2024-02-21T23:10:08.236368+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:42:16.988326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232712+0000","last_change":"2024-02-21T23:09:53.284733+0000","last_active":"2024-02-21T23:10:08.232712+0000","last_peered":"2024-02-21T23:10:08.232712+0000","last_clean":"2024-02-21T23:10:08.232712+0000","last_became_active":"2024-02-21T23:09:53.284399+0000","last_became_peered":"2024-02-21T23:09:53.284399+0000","last_unstale":"2024-02-21T23:10:08.232712+0000","last_undegraded":"2024-02-21T23:10:08.232712+0000","last_fullsized":"2024-02-21T23:10:08.232712+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:05:08.172503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1c","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.609880+0000","last_change":"2024-02-21T23:10:06.402186+0000","last_active":"2024-02-21T23:10:07.609880+0000","last_peered":"2024-02-21T23:10:07.609880+0000","last_clean":"2024-02-21T23:10:07.609880+0000","last_became_active":"2024-02-21T23:10:06.401920+0000","last_became_peered":"2024-02-21T23:10:06.401920+0000","last_unstale":"2024-02-21T23:10:07.609880+0000","last_undegraded":"2024-02-21T23:10:07.609880+0000","last_fullsized":"2024-02-21T23:10:07.609880+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:26:48.509050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.1f","version":"154'4","reported_seq":18,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232862+0000","last_change":"2024-02-21T23:10:05.381058+0000","last_active":"2024-02-21T23:10:08.232862+0000","last_peered":"2024-02-21T23:10:08.232862+0000","last_clean":"2024-02-21T23:10:08.232862+0000","last_became_active":"2024-02-21T23:10:05.380737+0000","last_became_peered":"2024-02-21T23:10:05.380737+0000","last_unstale":"2024-02-21T23:10:08.232862+0000","last_undegraded":"2024-02-21T23:10:08.232862+0000","last_fullsized":"2024-02-21T23:10:08.232862+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:09:16.338374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1e","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.381415+0000","last_change":"2024-02-21T23:09:54.277339+0000","last_active":"2024-02-21T23:10:07.381415+0000","last_peered":"2024-02-21T23:10:07.381415+0000","last_clean":"2024-02-21T23:10:07.381415+0000","last_became_active":"2024-02-21T23:09:54.277039+0000","last_became_peered":"2024-02-21T23:09:54.277039+0000","last_unstale":"2024-02-21T23:10:07.381415+0000","last_undegraded":"2024-02-21T23:10:07.381415+0000","last_fullsized":"2024-02-21T23:10:07.381415+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:42:16.988326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.380471+0000","last_change":"2024-02-21T23:09:53.283915+0000","last_active":"2024-02-21T23:10:07.380471+0000","last_peered":"2024-02-21T23:10:07.380471+0000","last_clean":"2024-02-21T23:10:07.380471+0000","last_became_active":"2024-02-21T23:09:53.283532+0000","last_became_peered":"2024-02-21T23:09:53.283532+0000","last_unstale":"2024-02-21T23:10:07.380471+0000","last_undegraded":"2024-02-21T23:10:07.380471+0000","last_fullsized":"2024-02-21T23:10:07.380471+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:55:17.574145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1d","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.375317+0000","last_change":"2024-02-21T23:10:06.373957+0000","last_active":"2024-02-21T23:10:07.375317+0000","last_peered":"2024-02-21T23:10:07.375317+0000","last_clean":"2024-02-21T23:10:07.375317+0000","last_became_active":"2024-02-21T23:10:06.373657+0000","last_became_peered":"2024-02-21T23:10:06.373657+0000","last_unstale":"2024-02-21T23:10:07.375317+0000","last_undegraded":"2024-02-21T23:10:07.375317+0000","last_fullsized":"2024-02-21T23:10:07.375317+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:37:55.080822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1e","version":"154'1","reported_seq":15,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.235331+0000","last_change":"2024-02-21T23:10:05.388527+0000","last_active":"2024-02-21T23:10:08.235331+0000","last_peered":"2024-02-21T23:10:08.235331+0000","last_clean":"2024-02-21T23:10:08.235331+0000","last_became_active":"2024-02-21T23:10:05.388214+0000","last_became_peered":"2024-02-21T23:10:05.388214+0000","last_unstale":"2024-02-21T23:10:08.235331+0000","last_undegraded":"2024-02-21T23:10:08.235331+0000","last_fullsized":"2024-02-21T23:10:08.235331+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:50:32.666622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1f","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.376181+0000","last_change":"2024-02-21T23:09:54.265384+0000","last_active":"2024-02-21T23:10:07.376181+0000","last_peered":"2024-02-21T23:10:07.376181+0000","last_clean":"2024-02-21T23:10:07.376181+0000","last_became_active":"2024-02-21T23:09:54.265073+0000","last_became_peered":"2024-02-21T23:09:54.265073+0000","last_unstale":"2024-02-21T23:10:07.376181+0000","last_undegraded":"2024-02-21T23:10:07.376181+0000","last_fullsized":"2024-02-21T23:10:07.376181+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:44:28.295055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.10","version":"150'8","reported_seq":30,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.235156+0000","last_change":"2024-02-21T23:09:53.285246+0000","last_active":"2024-02-21T23:10:08.235156+0000","last_peered":"2024-02-21T23:10:08.235156+0000","last_clean":"2024-02-21T23:10:08.235156+0000","last_became_active":"2024-02-21T23:09:53.282084+0000","last_became_peered":"2024-02-21T23:09:53.282084+0000","last_unstale":"2024-02-21T23:10:08.235156+0000","last_undegraded":"2024-02-21T23:10:08.235156+0000","last_fullsized":"2024-02-21T23:10:08.235156+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:23:26.494398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.235925+0000","last_change":"2024-02-21T23:10:06.390439+0000","last_active":"2024-02-21T23:10:08.235925+0000","last_peered":"2024-02-21T23:10:08.235925+0000","last_clean":"2024-02-21T23:10:08.235925+0000","last_became_active":"2024-02-21T23:10:06.390012+0000","last_became_peered":"2024-02-21T23:10:06.390012+0000","last_unstale":"2024-02-21T23:10:08.235925+0000","last_undegraded":"2024-02-21T23:10:08.235925+0000","last_fullsized":"2024-02-21T23:10:08.235925+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:17:37.624314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.374798+0000","last_change":"2024-02-21T23:10:05.393586+0000","last_active":"2024-02-21T23:10:07.374798+0000","last_peered":"2024-02-21T23:10:07.374798+0000","last_clean":"2024-02-21T23:10:07.374798+0000","last_became_active":"2024-02-21T23:10:05.392606+0000","last_became_peered":"2024-02-21T23:10:05.392606+0000","last_unstale":"2024-02-21T23:10:07.374798+0000","last_undegraded":"2024-02-21T23:10:07.374798+0000","last_fullsized":"2024-02-21T23:10:07.374798+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:12:07.072214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234736+0000","last_change":"2024-02-21T23:09:54.289615+0000","last_active":"2024-02-21T23:10:08.234736+0000","last_peered":"2024-02-21T23:10:08.234736+0000","last_clean":"2024-02-21T23:10:08.234736+0000","last_became_active":"2024-02-21T23:09:54.289322+0000","last_became_peered":"2024-02-21T23:09:54.289322+0000","last_unstale":"2024-02-21T23:10:08.234736+0000","last_undegraded":"2024-02-21T23:10:08.234736+0000","last_fullsized":"2024-02-21T23:10:08.234736+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:34:28.642087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.f","version":"150'1","reported_seq":23,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236888+0000","last_change":"2024-02-21T23:09:53.287210+0000","last_active":"2024-02-21T23:10:08.236888+0000","last_peered":"2024-02-21T23:10:08.236888+0000","last_clean":"2024-02-21T23:10:08.236888+0000","last_became_active":"2024-02-21T23:09:53.285372+0000","last_became_peered":"2024-02-21T23:09:53.285372+0000","last_unstale":"2024-02-21T23:10:08.236888+0000","last_undegraded":"2024-02-21T23:10:08.236888+0000","last_fullsized":"2024-02-21T23:10:08.236888+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:20:30.108767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.609645+0000","last_change":"2024-02-21T23:10:06.393455+0000","last_active":"2024-02-21T23:10:07.609645+0000","last_peered":"2024-02-21T23:10:07.609645+0000","last_clean":"2024-02-21T23:10:07.609645+0000","last_became_active":"2024-02-21T23:10:06.393088+0000","last_became_peered":"2024-02-21T23:10:06.393088+0000","last_unstale":"2024-02-21T23:10:07.609645+0000","last_undegraded":"2024-02-21T23:10:07.609645+0000","last_fullsized":"2024-02-21T23:10:07.609645+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:44:08.714643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.379203+0000","last_change":"2024-02-21T23:10:05.395057+0000","last_active":"2024-02-21T23:10:07.379203+0000","last_peered":"2024-02-21T23:10:07.379203+0000","last_clean":"2024-02-21T23:10:07.379203+0000","last_became_active":"2024-02-21T23:10:05.394783+0000","last_became_peered":"2024-02-21T23:10:05.394783+0000","last_unstale":"2024-02-21T23:10:07.379203+0000","last_undegraded":"2024-02-21T23:10:07.379203+0000","last_fullsized":"2024-02-21T23:10:07.379203+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:02:22.335727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.379499+0000","last_change":"2024-02-21T23:09:54.274363+0000","last_active":"2024-02-21T23:10:07.379499+0000","last_peered":"2024-02-21T23:10:07.379499+0000","last_clean":"2024-02-21T23:10:07.379499+0000","last_became_active":"2024-02-21T23:09:54.274084+0000","last_became_peered":"2024-02-21T23:09:54.274084+0000","last_unstale":"2024-02-21T23:10:07.379499+0000","last_undegraded":"2024-02-21T23:10:07.379499+0000","last_fullsized":"2024-02-21T23:10:07.379499+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:42:19.996155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.e","version":"150'2","reported_seq":24,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.237049+0000","last_change":"2024-02-21T23:09:53.275941+0000","last_active":"2024-02-21T23:10:08.237049+0000","last_peered":"2024-02-21T23:10:08.237049+0000","last_clean":"2024-02-21T23:10:08.237049+0000","last_became_active":"2024-02-21T23:09:53.275555+0000","last_became_peered":"2024-02-21T23:09:53.275555+0000","last_unstale":"2024-02-21T23:10:08.237049+0000","last_undegraded":"2024-02-21T23:10:08.237049+0000","last_fullsized":"2024-02-21T23:10:08.237049+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:02:10.230470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.379162+0000","last_change":"2024-02-21T23:10:06.377153+0000","last_active":"2024-02-21T23:10:07.379162+0000","last_peered":"2024-02-21T23:10:07.379162+0000","last_clean":"2024-02-21T23:10:07.379162+0000","last_became_active":"2024-02-21T23:10:06.376854+0000","last_became_peered":"2024-02-21T23:10:06.376854+0000","last_unstale":"2024-02-21T23:10:07.379162+0000","last_undegraded":"2024-02-21T23:10:07.379162+0000","last_fullsized":"2024-02-21T23:10:07.379162+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:13:56.739562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.608871+0000","last_change":"2024-02-21T23:10:05.395810+0000","last_active":"2024-02-21T23:10:07.608871+0000","last_peered":"2024-02-21T23:10:07.608871+0000","last_clean":"2024-02-21T23:10:07.608871+0000","last_became_active":"2024-02-21T23:10:05.395273+0000","last_became_peered":"2024-02-21T23:10:05.395273+0000","last_unstale":"2024-02-21T23:10:07.608871+0000","last_undegraded":"2024-02-21T23:10:07.608871+0000","last_fullsized":"2024-02-21T23:10:07.608871+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:00:10.805897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.380745+0000","last_change":"2024-02-21T23:09:54.259630+0000","last_active":"2024-02-21T23:10:07.380745+0000","last_peered":"2024-02-21T23:10:07.380745+0000","last_clean":"2024-02-21T23:10:07.380745+0000","last_became_active":"2024-02-21T23:09:54.258924+0000","last_became_peered":"2024-02-21T23:09:54.258924+0000","last_unstale":"2024-02-21T23:10:07.380745+0000","last_undegraded":"2024-02-21T23:10:07.380745+0000","last_fullsized":"2024-02-21T23:10:07.380745+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:34:28.642087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.d","version":"150'1","reported_seq":23,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234011+0000","last_change":"2024-02-21T23:09:53.279385+0000","last_active":"2024-02-21T23:10:08.234011+0000","last_peered":"2024-02-21T23:10:08.234011+0000","last_clean":"2024-02-21T23:10:08.234011+0000","last_became_active":"2024-02-21T23:09:53.278074+0000","last_became_peered":"2024-02-21T23:09:53.278074+0000","last_unstale":"2024-02-21T23:10:08.234011+0000","last_undegraded":"2024-02-21T23:10:08.234011+0000","last_fullsized":"2024-02-21T23:10:08.234011+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:42:15.971409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.230685+0000","last_change":"2024-02-21T23:10:06.383577+0000","last_active":"2024-02-21T23:10:08.230685+0000","last_peered":"2024-02-21T23:10:08.230685+0000","last_clean":"2024-02-21T23:10:08.230685+0000","last_became_active":"2024-02-21T23:10:06.383253+0000","last_became_peered":"2024-02-21T23:10:06.383253+0000","last_unstale":"2024-02-21T23:10:08.230685+0000","last_undegraded":"2024-02-21T23:10:08.230685+0000","last_fullsized":"2024-02-21T23:10:08.230685+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:00:11.821587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.608567+0000","last_change":"2024-02-21T23:10:05.385621+0000","last_active":"2024-02-21T23:10:07.608567+0000","last_peered":"2024-02-21T23:10:07.608567+0000","last_clean":"2024-02-21T23:10:07.608567+0000","last_became_active":"2024-02-21T23:10:05.384654+0000","last_became_peered":"2024-02-21T23:10:05.384654+0000","last_unstale":"2024-02-21T23:10:07.608567+0000","last_undegraded":"2024-02-21T23:10:07.608567+0000","last_fullsized":"2024-02-21T23:10:07.608567+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:30:56.525674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.381032+0000","last_change":"2024-02-21T23:09:54.277296+0000","last_active":"2024-02-21T23:10:07.381032+0000","last_peered":"2024-02-21T23:10:07.381032+0000","last_clean":"2024-02-21T23:10:07.381032+0000","last_became_active":"2024-02-21T23:09:54.276973+0000","last_became_peered":"2024-02-21T23:09:54.276973+0000","last_unstale":"2024-02-21T23:10:07.381032+0000","last_undegraded":"2024-02-21T23:10:07.381032+0000","last_fullsized":"2024-02-21T23:10:07.381032+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:07:24.304915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.235861+0000","last_change":"2024-02-21T23:09:53.281804+0000","last_active":"2024-02-21T23:10:08.235861+0000","last_peered":"2024-02-21T23:10:08.235861+0000","last_clean":"2024-02-21T23:10:08.235861+0000","last_became_active":"2024-02-21T23:09:53.278643+0000","last_became_peered":"2024-02-21T23:09:53.278643+0000","last_unstale":"2024-02-21T23:10:08.235861+0000","last_undegraded":"2024-02-21T23:10:08.235861+0000","last_fullsized":"2024-02-21T23:10:08.235861+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:55:17.574145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.6","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.609698+0000","last_change":"2024-02-21T23:10:06.389855+0000","last_active":"2024-02-21T23:10:07.609698+0000","last_peered":"2024-02-21T23:10:07.609698+0000","last_clean":"2024-02-21T23:10:07.609698+0000","last_became_active":"2024-02-21T23:10:06.389147+0000","last_became_peered":"2024-02-21T23:10:06.389147+0000","last_unstale":"2024-02-21T23:10:07.609698+0000","last_undegraded":"2024-02-21T23:10:07.609698+0000","last_fullsized":"2024-02-21T23:10:07.609698+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:40:17.310936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234955+0000","last_change":"2024-02-21T23:10:05.389031+0000","last_active":"2024-02-21T23:10:08.234955+0000","last_peered":"2024-02-21T23:10:08.234955+0000","last_clean":"2024-02-21T23:10:08.234955+0000","last_became_active":"2024-02-21T23:10:05.388789+0000","last_became_peered":"2024-02-21T23:10:05.388789+0000","last_unstale":"2024-02-21T23:10:08.234955+0000","last_undegraded":"2024-02-21T23:10:08.234955+0000","last_fullsized":"2024-02-21T23:10:08.234955+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:46:44.667756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236537+0000","last_change":"2024-02-21T23:09:54.285141+0000","last_active":"2024-02-21T23:10:08.236537+0000","last_peered":"2024-02-21T23:10:08.236537+0000","last_clean":"2024-02-21T23:10:08.236537+0000","last_became_active":"2024-02-21T23:09:54.284498+0000","last_became_peered":"2024-02-21T23:09:54.284498+0000","last_unstale":"2024-02-21T23:10:08.236537+0000","last_undegraded":"2024-02-21T23:10:08.236537+0000","last_fullsized":"2024-02-21T23:10:08.236537+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:26:36.405020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.b","version":"150'2","reported_seq":24,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234201+0000","last_change":"2024-02-21T23:09:53.284284+0000","last_active":"2024-02-21T23:10:08.234201+0000","last_peered":"2024-02-21T23:10:08.234201+0000","last_clean":"2024-02-21T23:10:08.234201+0000","last_became_active":"2024-02-21T23:09:53.283043+0000","last_became_peered":"2024-02-21T23:09:53.283043+0000","last_unstale":"2024-02-21T23:10:08.234201+0000","last_undegraded":"2024-02-21T23:10:08.234201+0000","last_fullsized":"2024-02-21T23:10:08.234201+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:32:05.045159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.7","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.376118+0000","last_change":"2024-02-21T23:10:06.392968+0000","last_active":"2024-02-21T23:10:07.376118+0000","last_peered":"2024-02-21T23:10:07.376118+0000","last_clean":"2024-02-21T23:10:07.376118+0000","last_became_active":"2024-02-21T23:10:06.390031+0000","last_became_peered":"2024-02-21T23:10:06.390031+0000","last_unstale":"2024-02-21T23:10:07.376118+0000","last_undegraded":"2024-02-21T23:10:07.376118+0000","last_fullsized":"2024-02-21T23:10:07.376118+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:17:07.163396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236747+0000","last_change":"2024-02-21T23:10:05.392092+0000","last_active":"2024-02-21T23:10:08.236747+0000","last_peered":"2024-02-21T23:10:08.236747+0000","last_clean":"2024-02-21T23:10:08.236747+0000","last_became_active":"2024-02-21T23:10:05.391793+0000","last_became_peered":"2024-02-21T23:10:05.391793+0000","last_unstale":"2024-02-21T23:10:08.236747+0000","last_undegraded":"2024-02-21T23:10:08.236747+0000","last_fullsized":"2024-02-21T23:10:08.236747+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:09:16.338374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.233964+0000","last_change":"2024-02-21T23:09:54.275082+0000","last_active":"2024-02-21T23:10:08.233964+0000","last_peered":"2024-02-21T23:10:08.233964+0000","last_clean":"2024-02-21T23:10:08.233964+0000","last_became_active":"2024-02-21T23:09:54.274707+0000","last_became_peered":"2024-02-21T23:09:54.274707+0000","last_unstale":"2024-02-21T23:10:08.233964+0000","last_undegraded":"2024-02-21T23:10:08.233964+0000","last_fullsized":"2024-02-21T23:10:08.233964+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:39:52.080584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.381286+0000","last_change":"2024-02-21T23:09:53.277418+0000","last_active":"2024-02-21T23:10:07.381286+0000","last_peered":"2024-02-21T23:10:07.381286+0000","last_clean":"2024-02-21T23:10:07.381286+0000","last_became_active":"2024-02-21T23:09:53.277111+0000","last_became_peered":"2024-02-21T23:09:53.277111+0000","last_unstale":"2024-02-21T23:10:07.381286+0000","last_undegraded":"2024-02-21T23:10:07.381286+0000","last_fullsized":"2024-02-21T23:10:07.381286+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:23:26.494398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.375599+0000","last_change":"2024-02-21T23:10:06.391104+0000","last_active":"2024-02-21T23:10:07.375599+0000","last_peered":"2024-02-21T23:10:07.375599+0000","last_clean":"2024-02-21T23:10:07.375599+0000","last_became_active":"2024-02-21T23:10:06.390816+0000","last_became_peered":"2024-02-21T23:10:06.390816+0000","last_unstale":"2024-02-21T23:10:07.375599+0000","last_undegraded":"2024-02-21T23:10:07.375599+0000","last_fullsized":"2024-02-21T23:10:07.375599+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:49:43.863404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.7","version":"154'2","reported_seq":16,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234306+0000","last_change":"2024-02-21T23:10:05.378389+0000","last_active":"2024-02-21T23:10:08.234306+0000","last_peered":"2024-02-21T23:10:08.234306+0000","last_clean":"2024-02-21T23:10:08.234306+0000","last_became_active":"2024-02-21T23:10:05.378083+0000","last_became_peered":"2024-02-21T23:10:05.378083+0000","last_unstale":"2024-02-21T23:10:08.234306+0000","last_undegraded":"2024-02-21T23:10:08.234306+0000","last_fullsized":"2024-02-21T23:10:08.234306+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:28:14.502463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232579+0000","last_change":"2024-02-21T23:09:54.280400+0000","last_active":"2024-02-21T23:10:08.232579+0000","last_peered":"2024-02-21T23:10:08.232579+0000","last_clean":"2024-02-21T23:10:08.232579+0000","last_became_active":"2024-02-21T23:09:54.280102+0000","last_became_peered":"2024-02-21T23:09:54.280102+0000","last_unstale":"2024-02-21T23:10:08.232579+0000","last_undegraded":"2024-02-21T23:10:08.232579+0000","last_fullsized":"2024-02-21T23:10:08.232579+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:45:28.830261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.375349+0000","last_change":"2024-02-21T23:09:53.282939+0000","last_active":"2024-02-21T23:10:07.375349+0000","last_peered":"2024-02-21T23:10:07.375349+0000","last_clean":"2024-02-21T23:10:07.375349+0000","last_became_active":"2024-02-21T23:09:53.282612+0000","last_became_peered":"2024-02-21T23:09:53.282612+0000","last_unstale":"2024-02-21T23:10:07.375349+0000","last_undegraded":"2024-02-21T23:10:07.375349+0000","last_fullsized":"2024-02-21T23:10:07.375349+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:30:40.609835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.236333+0000","last_change":"2024-02-21T23:10:06.396738+0000","last_active":"2024-02-21T23:10:08.236333+0000","last_peered":"2024-02-21T23:10:08.236333+0000","last_clean":"2024-02-21T23:10:08.236333+0000","last_became_active":"2024-02-21T23:10:06.396390+0000","last_became_peered":"2024-02-21T23:10:06.396390+0000","last_unstale":"2024-02-21T23:10:08.236333+0000","last_undegraded":"2024-02-21T23:10:08.236333+0000","last_fullsized":"2024-02-21T23:10:08.236333+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:24:32.853214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.6","version":"155'1","reported_seq":15,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.375969+0000","last_change":"2024-02-21T23:10:05.395962+0000","last_active":"2024-02-21T23:10:07.375969+0000","last_peered":"2024-02-21T23:10:07.375969+0000","last_clean":"2024-02-21T23:10:07.375969+0000","last_became_active":"2024-02-21T23:10:05.394863+0000","last_became_peered":"2024-02-21T23:10:05.394863+0000","last_unstale":"2024-02-21T23:10:07.375969+0000","last_undegraded":"2024-02-21T23:10:07.375969+0000","last_fullsized":"2024-02-21T23:10:07.375969+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:48:58.601649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.610043+0000","last_change":"2024-02-21T23:09:54.282302+0000","last_active":"2024-02-21T23:10:07.610043+0000","last_peered":"2024-02-21T23:10:07.610043+0000","last_clean":"2024-02-21T23:10:07.610043+0000","last_became_active":"2024-02-21T23:09:54.280468+0000","last_became_peered":"2024-02-21T23:09:54.280468+0000","last_unstale":"2024-02-21T23:10:07.610043+0000","last_undegraded":"2024-02-21T23:10:07.610043+0000","last_fullsized":"2024-02-21T23:10:07.610043+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:32:06.062076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.610683+0000","last_change":"2024-02-21T23:09:53.283837+0000","last_active":"2024-02-21T23:10:07.610683+0000","last_peered":"2024-02-21T23:10:07.610683+0000","last_clean":"2024-02-21T23:10:07.610683+0000","last_became_active":"2024-02-21T23:09:53.283541+0000","last_became_peered":"2024-02-21T23:09:53.283541+0000","last_unstale":"2024-02-21T23:10:07.610683+0000","last_undegraded":"2024-02-21T23:10:07.610683+0000","last_fullsized":"2024-02-21T23:10:07.610683+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:28:53.062636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.231400+0000","last_change":"2024-02-21T23:10:06.392422+0000","last_active":"2024-02-21T23:10:08.231400+0000","last_peered":"2024-02-21T23:10:08.231400+0000","last_clean":"2024-02-21T23:10:08.231400+0000","last_became_active":"2024-02-21T23:10:06.392121+0000","last_became_peered":"2024-02-21T23:10:06.392121+0000","last_unstale":"2024-02-21T23:10:08.231400+0000","last_undegraded":"2024-02-21T23:10:08.231400+0000","last_fullsized":"2024-02-21T23:10:08.231400+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:09:29.246717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.374605+0000","last_change":"2024-02-21T23:10:05.398114+0000","last_active":"2024-02-21T23:10:07.374605+0000","last_peered":"2024-02-21T23:10:07.374605+0000","last_clean":"2024-02-21T23:10:07.374605+0000","last_became_active":"2024-02-21T23:10:05.397811+0000","last_became_peered":"2024-02-21T23:10:05.397811+0000","last_unstale":"2024-02-21T23:10:07.374605+0000","last_undegraded":"2024-02-21T23:10:07.374605+0000","last_fullsized":"2024-02-21T23:10:07.374605+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:36:18.190950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.380388+0000","last_change":"2024-02-21T23:09:54.274229+0000","last_active":"2024-02-21T23:10:07.380388+0000","last_peered":"2024-02-21T23:10:07.380388+0000","last_clean":"2024-02-21T23:10:07.380388+0000","last_became_active":"2024-02-21T23:09:54.271365+0000","last_became_peered":"2024-02-21T23:09:54.271365+0000","last_unstale":"2024-02-21T23:10:07.380388+0000","last_undegraded":"2024-02-21T23:10:07.380388+0000","last_fullsized":"2024-02-21T23:10:07.380388+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:13:27.311923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.7","version":"150'2","reported_seq":24,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.233419+0000","last_change":"2024-02-21T23:09:53.283873+0000","last_active":"2024-02-21T23:10:08.233419+0000","last_peered":"2024-02-21T23:10:08.233419+0000","last_clean":"2024-02-21T23:10:08.233419+0000","last_became_active":"2024-02-21T23:09:53.282829+0000","last_became_peered":"2024-02-21T23:09:53.282829+0000","last_unstale":"2024-02-21T23:10:08.233419+0000","last_undegraded":"2024-02-21T23:10:08.233419+0000","last_fullsized":"2024-02-21T23:10:08.233419+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:13:32.425310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.b","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234395+0000","last_change":"2024-02-21T23:10:06.367171+0000","last_active":"2024-02-21T23:10:08.234395+0000","last_peered":"2024-02-21T23:10:08.234395+0000","last_clean":"2024-02-21T23:10:08.234395+0000","last_became_active":"2024-02-21T23:10:06.366866+0000","last_became_peered":"2024-02-21T23:10:06.366866+0000","last_unstale":"2024-02-21T23:10:08.234395+0000","last_undegraded":"2024-02-21T23:10:08.234395+0000","last_fullsized":"2024-02-21T23:10:08.234395+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:26:16.569464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.379666+0000","last_change":"2024-02-21T23:10:05.389876+0000","last_active":"2024-02-21T23:10:07.379666+0000","last_peered":"2024-02-21T23:10:07.379666+0000","last_clean":"2024-02-21T23:10:07.379666+0000","last_became_active":"2024-02-21T23:10:05.389568+0000","last_became_peered":"2024-02-21T23:10:05.389568+0000","last_unstale":"2024-02-21T23:10:07.379666+0000","last_undegraded":"2024-02-21T23:10:07.379666+0000","last_fullsized":"2024-02-21T23:10:07.379666+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:18:51.227878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234481+0000","last_change":"2024-02-21T23:09:54.277479+0000","last_active":"2024-02-21T23:10:08.234481+0000","last_peered":"2024-02-21T23:10:08.234481+0000","last_clean":"2024-02-21T23:10:08.234481+0000","last_became_active":"2024-02-21T23:09:54.277065+0000","last_became_peered":"2024-02-21T23:09:54.277065+0000","last_unstale":"2024-02-21T23:10:08.234481+0000","last_undegraded":"2024-02-21T23:10:08.234481+0000","last_fullsized":"2024-02-21T23:10:08.234481+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:07:24.304915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.6","version":"151'1","reported_seq":23,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.235714+0000","last_change":"2024-02-21T23:09:53.285874+0000","last_active":"2024-02-21T23:10:08.235714+0000","last_peered":"2024-02-21T23:10:08.235714+0000","last_clean":"2024-02-21T23:10:08.235714+0000","last_became_active":"2024-02-21T23:09:53.285569+0000","last_became_peered":"2024-02-21T23:09:53.285569+0000","last_unstale":"2024-02-21T23:10:08.235714+0000","last_undegraded":"2024-02-21T23:10:08.235714+0000","last_fullsized":"2024-02-21T23:10:08.235714+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:52:56.873078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.233184+0000","last_change":"2024-02-21T23:10:06.385851+0000","last_active":"2024-02-21T23:10:08.233184+0000","last_peered":"2024-02-21T23:10:08.233184+0000","last_clean":"2024-02-21T23:10:08.233184+0000","last_became_active":"2024-02-21T23:10:06.385482+0000","last_became_peered":"2024-02-21T23:10:06.385482+0000","last_unstale":"2024-02-21T23:10:08.233184+0000","last_undegraded":"2024-02-21T23:10:08.233184+0000","last_fullsized":"2024-02-21T23:10:08.233184+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:47:42.794742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.b","version":"154'2","reported_seq":16,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.231292+0000","last_change":"2024-02-21T23:10:05.394114+0000","last_active":"2024-02-21T23:10:08.231292+0000","last_peered":"2024-02-21T23:10:08.231292+0000","last_clean":"2024-02-21T23:10:08.231292+0000","last_became_active":"2024-02-21T23:10:05.393152+0000","last_became_peered":"2024-02-21T23:10:05.393152+0000","last_unstale":"2024-02-21T23:10:08.231292+0000","last_undegraded":"2024-02-21T23:10:08.231292+0000","last_fullsized":"2024-02-21T23:10:08.231292+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:07:35.393255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.374932+0000","last_change":"2024-02-21T23:09:54.286102+0000","last_active":"2024-02-21T23:10:07.374932+0000","last_peered":"2024-02-21T23:10:07.374932+0000","last_clean":"2024-02-21T23:10:07.374932+0000","last_became_active":"2024-02-21T23:09:54.283073+0000","last_became_peered":"2024-02-21T23:09:54.283073+0000","last_unstale":"2024-02-21T23:10:07.374932+0000","last_undegraded":"2024-02-21T23:10:07.374932+0000","last_fullsized":"2024-02-21T23:10:07.374932+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:39:05.096862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.235327+0000","last_change":"2024-02-21T23:09:53.285730+0000","last_active":"2024-02-21T23:10:08.235327+0000","last_peered":"2024-02-21T23:10:08.235327+0000","last_clean":"2024-02-21T23:10:08.235327+0000","last_became_active":"2024-02-21T23:09:53.285373+0000","last_became_peered":"2024-02-21T23:09:53.285373+0000","last_unstale":"2024-02-21T23:10:08.235327+0000","last_undegraded":"2024-02-21T23:10:08.235327+0000","last_fullsized":"2024-02-21T23:10:08.235327+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:04:15.300172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.230302+0000","last_change":"2024-02-21T23:10:06.380586+0000","last_active":"2024-02-21T23:10:08.230302+0000","last_peered":"2024-02-21T23:10:08.230302+0000","last_clean":"2024-02-21T23:10:08.230302+0000","last_became_active":"2024-02-21T23:10:06.380293+0000","last_became_peered":"2024-02-21T23:10:06.380293+0000","last_unstale":"2024-02-21T23:10:08.230302+0000","last_undegraded":"2024-02-21T23:10:08.230302+0000","last_fullsized":"2024-02-21T23:10:08.230302+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:01:26.662036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.231095+0000","last_change":"2024-02-21T23:10:05.380512+0000","last_active":"2024-02-21T23:10:08.231095+0000","last_peered":"2024-02-21T23:10:08.231095+0000","last_clean":"2024-02-21T23:10:08.231095+0000","last_became_active":"2024-02-21T23:10:05.380207+0000","last_became_peered":"2024-02-21T23:10:05.380207+0000","last_unstale":"2024-02-21T23:10:08.231095+0000","last_undegraded":"2024-02-21T23:10:08.231095+0000","last_fullsized":"2024-02-21T23:10:08.231095+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:17:36.608624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.b","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.235603+0000","last_change":"2024-02-21T23:09:54.277483+0000","last_active":"2024-02-21T23:10:08.235603+0000","last_peered":"2024-02-21T23:10:08.235603+0000","last_clean":"2024-02-21T23:10:08.235603+0000","last_became_active":"2024-02-21T23:09:54.277170+0000","last_became_peered":"2024-02-21T23:09:54.277170+0000","last_unstale":"2024-02-21T23:10:08.235603+0000","last_undegraded":"2024-02-21T23:10:08.235603+0000","last_fullsized":"2024-02-21T23:10:08.235603+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:33:07.362294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.375793+0000","last_change":"2024-02-21T23:09:53.267455+0000","last_active":"2024-02-21T23:10:07.375793+0000","last_peered":"2024-02-21T23:10:07.375793+0000","last_clean":"2024-02-21T23:10:07.375793+0000","last_became_active":"2024-02-21T23:09:53.266974+0000","last_became_peered":"2024-02-21T23:09:53.266974+0000","last_unstale":"2024-02-21T23:10:07.375793+0000","last_undegraded":"2024-02-21T23:10:07.375793+0000","last_fullsized":"2024-02-21T23:10:07.375793+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:33:26.269092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.e","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.380986+0000","last_change":"2024-02-21T23:10:06.392129+0000","last_active":"2024-02-21T23:10:07.380986+0000","last_peered":"2024-02-21T23:10:07.380986+0000","last_clean":"2024-02-21T23:10:07.380986+0000","last_became_active":"2024-02-21T23:10:06.391056+0000","last_became_peered":"2024-02-21T23:10:06.391056+0000","last_unstale":"2024-02-21T23:10:07.380986+0000","last_undegraded":"2024-02-21T23:10:07.380986+0000","last_fullsized":"2024-02-21T23:10:07.380986+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:47:42.794742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.d","version":"154'1","reported_seq":15,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.380360+0000","last_change":"2024-02-21T23:10:05.393632+0000","last_active":"2024-02-21T23:10:07.380360+0000","last_peered":"2024-02-21T23:10:07.380360+0000","last_clean":"2024-02-21T23:10:07.380360+0000","last_became_active":"2024-02-21T23:10:05.393070+0000","last_became_peered":"2024-02-21T23:10:05.393070+0000","last_unstale":"2024-02-21T23:10:07.380360+0000","last_undegraded":"2024-02-21T23:10:07.380360+0000","last_fullsized":"2024-02-21T23:10:07.380360+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:09:01.608295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232424+0000","last_change":"2024-02-21T23:09:54.275273+0000","last_active":"2024-02-21T23:10:08.232424+0000","last_peered":"2024-02-21T23:10:08.232424+0000","last_clean":"2024-02-21T23:10:08.232424+0000","last_became_active":"2024-02-21T23:09:54.274287+0000","last_became_peered":"2024-02-21T23:09:54.274287+0000","last_unstale":"2024-02-21T23:10:08.232424+0000","last_undegraded":"2024-02-21T23:10:08.232424+0000","last_fullsized":"2024-02-21T23:10:08.232424+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:18:40.139538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232148+0000","last_change":"2024-02-21T23:09:53.279782+0000","last_active":"2024-02-21T23:10:08.232148+0000","last_peered":"2024-02-21T23:10:08.232148+0000","last_clean":"2024-02-21T23:10:08.232148+0000","last_became_active":"2024-02-21T23:09:53.279477+0000","last_became_peered":"2024-02-21T23:09:53.279477+0000","last_unstale":"2024-02-21T23:10:08.232148+0000","last_undegraded":"2024-02-21T23:10:08.232148+0000","last_fullsized":"2024-02-21T23:10:08.232148+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:41:01.169723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.f","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.237523+0000","last_change":"2024-02-21T23:10:06.399721+0000","last_active":"2024-02-21T23:10:08.237523+0000","last_peered":"2024-02-21T23:10:08.237523+0000","last_clean":"2024-02-21T23:10:08.237523+0000","last_became_active":"2024-02-21T23:10:06.398869+0000","last_became_peered":"2024-02-21T23:10:06.398869+0000","last_unstale":"2024-02-21T23:10:08.237523+0000","last_undegraded":"2024-02-21T23:10:08.237523+0000","last_fullsized":"2024-02-21T23:10:08.237523+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:48:37.041227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.380907+0000","last_change":"2024-02-21T23:10:05.380230+0000","last_active":"2024-02-21T23:10:07.380907+0000","last_peered":"2024-02-21T23:10:07.380907+0000","last_clean":"2024-02-21T23:10:07.380907+0000","last_became_active":"2024-02-21T23:10:05.378612+0000","last_became_peered":"2024-02-21T23:10:05.378612+0000","last_unstale":"2024-02-21T23:10:07.380907+0000","last_undegraded":"2024-02-21T23:10:07.380907+0000","last_fullsized":"2024-02-21T23:10:07.380907+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:36:53.010888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.d","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.610179+0000","last_change":"2024-02-21T23:09:54.279409+0000","last_active":"2024-02-21T23:10:07.610179+0000","last_peered":"2024-02-21T23:10:07.610179+0000","last_clean":"2024-02-21T23:10:07.610179+0000","last_became_active":"2024-02-21T23:09:54.279107+0000","last_became_peered":"2024-02-21T23:09:54.279107+0000","last_unstale":"2024-02-21T23:10:07.610179+0000","last_undegraded":"2024-02-21T23:10:07.610179+0000","last_fullsized":"2024-02-21T23:10:07.610179+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:20:31.125684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232898+0000","last_change":"2024-02-21T23:09:53.278959+0000","last_active":"2024-02-21T23:10:08.232898+0000","last_peered":"2024-02-21T23:10:08.232898+0000","last_clean":"2024-02-21T23:10:08.232898+0000","last_became_active":"2024-02-21T23:09:53.278673+0000","last_became_peered":"2024-02-21T23:09:53.278673+0000","last_unstale":"2024-02-21T23:10:08.232898+0000","last_undegraded":"2024-02-21T23:10:08.232898+0000","last_fullsized":"2024-02-21T23:10:08.232898+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:42:15.971409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.375427+0000","last_change":"2024-02-21T23:10:06.387361+0000","last_active":"2024-02-21T23:10:07.375427+0000","last_peered":"2024-02-21T23:10:07.375427+0000","last_clean":"2024-02-21T23:10:07.375427+0000","last_became_active":"2024-02-21T23:10:06.387063+0000","last_became_peered":"2024-02-21T23:10:06.387063+0000","last_unstale":"2024-02-21T23:10:07.375427+0000","last_undegraded":"2024-02-21T23:10:07.375427+0000","last_fullsized":"2024-02-21T23:10:07.375427+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:47:41.453699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"154'1","reported_seq":15,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.381702+0000","last_change":"2024-02-21T23:10:05.381912+0000","last_active":"2024-02-21T23:10:07.381702+0000","last_peered":"2024-02-21T23:10:07.381702+0000","last_clean":"2024-02-21T23:10:07.381702+0000","last_became_active":"2024-02-21T23:10:05.381618+0000","last_became_peered":"2024-02-21T23:10:05.381618+0000","last_unstale":"2024-02-21T23:10:07.381702+0000","last_undegraded":"2024-02-21T23:10:07.381702+0000","last_fullsized":"2024-02-21T23:10:07.381702+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:47:45.153381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.e","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.235442+0000","last_change":"2024-02-21T23:09:54.282920+0000","last_active":"2024-02-21T23:10:08.235442+0000","last_peered":"2024-02-21T23:10:08.235442+0000","last_clean":"2024-02-21T23:10:08.235442+0000","last_became_active":"2024-02-21T23:09:54.282544+0000","last_became_peered":"2024-02-21T23:09:54.282544+0000","last_unstale":"2024-02-21T23:10:08.235442+0000","last_undegraded":"2024-02-21T23:10:08.235442+0000","last_fullsized":"2024-02-21T23:10:08.235442+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:40:05.206906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.380542+0000","last_change":"2024-02-21T23:09:53.275586+0000","last_active":"2024-02-21T23:10:07.380542+0000","last_peered":"2024-02-21T23:10:07.380542+0000","last_clean":"2024-02-21T23:10:07.380542+0000","last_became_active":"2024-02-21T23:09:53.274828+0000","last_became_peered":"2024-02-21T23:09:53.274828+0000","last_unstale":"2024-02-21T23:10:07.380542+0000","last_undegraded":"2024-02-21T23:10:07.380542+0000","last_fullsized":"2024-02-21T23:10:07.380542+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:02:10.230470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.d","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.234506+0000","last_change":"2024-02-21T23:10:06.377243+0000","last_active":"2024-02-21T23:10:08.234506+0000","last_peered":"2024-02-21T23:10:08.234506+0000","last_clean":"2024-02-21T23:10:08.234506+0000","last_became_active":"2024-02-21T23:10:06.376521+0000","last_became_peered":"2024-02-21T23:10:06.376521+0000","last_unstale":"2024-02-21T23:10:08.234506+0000","last_undegraded":"2024-02-21T23:10:08.234506+0000","last_fullsized":"2024-02-21T23:10:08.234506+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:02:54.477082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.e","version":"154'2","reported_seq":16,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.232515+0000","last_change":"2024-02-21T23:10:05.388215+0000","last_active":"2024-02-21T23:10:08.232515+0000","last_peered":"2024-02-21T23:10:08.232515+0000","last_clean":"2024-02-21T23:10:08.232515+0000","last_became_active":"2024-02-21T23:10:05.387907+0000","last_became_peered":"2024-02-21T23:10:05.387907+0000","last_unstale":"2024-02-21T23:10:08.232515+0000","last_undegraded":"2024-02-21T23:10:08.232515+0000","last_fullsized":"2024-02-21T23:10:08.232515+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:05:36.127991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.f","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.376215+0000","last_change":"2024-02-21T23:09:54.277320+0000","last_active":"2024-02-21T23:10:07.376215+0000","last_peered":"2024-02-21T23:10:07.376215+0000","last_clean":"2024-02-21T23:10:07.376215+0000","last_became_active":"2024-02-21T23:09:54.276368+0000","last_became_peered":"2024-02-21T23:09:54.276368+0000","last_unstale":"2024-02-21T23:10:07.376215+0000","last_undegraded":"2024-02-21T23:10:07.376215+0000","last_fullsized":"2024-02-21T23:10:07.376215+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:40:37.285836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.230247+0000","last_change":"2024-02-21T23:09:53.278027+0000","last_active":"2024-02-21T23:10:08.230247+0000","last_peered":"2024-02-21T23:10:08.230247+0000","last_clean":"2024-02-21T23:10:08.230247+0000","last_became_active":"2024-02-21T23:09:53.277660+0000","last_became_peered":"2024-02-21T23:09:53.277660+0000","last_unstale":"2024-02-21T23:10:08.230247+0000","last_undegraded":"2024-02-21T23:10:08.230247+0000","last_fullsized":"2024-02-21T23:10:08.230247+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:07:23.287998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1b","version":"0'0","reported_seq":12,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.379594+0000","last_change":"2024-02-21T23:10:06.386342+0000","last_active":"2024-02-21T23:10:07.379594+0000","last_peered":"2024-02-21T23:10:07.379594+0000","last_clean":"2024-02-21T23:10:07.379594+0000","last_became_active":"2024-02-21T23:10:06.385507+0000","last_became_peered":"2024-02-21T23:10:06.385507+0000","last_unstale":"2024-02-21T23:10:07.379594+0000","last_undegraded":"2024-02-21T23:10:07.379594+0000","last_fullsized":"2024-02-21T23:10:07.379594+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":153,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:05.335854+0000","last_clean_scrub_stamp":"2024-02-21T23:10:05.335854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:01:26.987040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":14,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.230256+0000","last_change":"2024-02-21T23:10:05.390395+0000","last_active":"2024-02-21T23:10:08.230256+0000","last_peered":"2024-02-21T23:10:08.230256+0000","last_clean":"2024-02-21T23:10:08.230256+0000","last_became_active":"2024-02-21T23:10:05.390115+0000","last_became_peered":"2024-02-21T23:10:05.390115+0000","last_unstale":"2024-02-21T23:10:08.230256+0000","last_undegraded":"2024-02-21T23:10:08.230256+0000","last_fullsized":"2024-02-21T23:10:08.230256+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:04.320164+0000","last_clean_scrub_stamp":"2024-02-21T23:10:04.320164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:10:20.170395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":20,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.608818+0000","last_change":"2024-02-21T23:09:54.277356+0000","last_active":"2024-02-21T23:10:07.608818+0000","last_peered":"2024-02-21T23:10:07.608818+0000","last_clean":"2024-02-21T23:10:07.608818+0000","last_became_active":"2024-02-21T23:09:54.277087+0000","last_became_peered":"2024-02-21T23:09:54.277087+0000","last_unstale":"2024-02-21T23:10:07.608818+0000","last_undegraded":"2024-02-21T23:10:07.608818+0000","last_fullsized":"2024-02-21T23:10:07.608818+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:53.231824+0000","last_clean_scrub_stamp":"2024-02-21T23:09:53.231824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:03:44.493909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":22,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:08.235691+0000","last_change":"2024-02-21T23:09:53.286181+0000","last_active":"2024-02-21T23:10:08.235691+0000","last_peered":"2024-02-21T23:10:08.235691+0000","last_clean":"2024-02-21T23:10:08.235691+0000","last_became_active":"2024-02-21T23:09:53.285849+0000","last_became_peered":"2024-02-21T23:09:53.285849+0000","last_unstale":"2024-02-21T23:10:08.235691+0000","last_undegraded":"2024-02-21T23:10:08.235691+0000","last_fullsized":"2024-02-21T23:10:08.235691+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:09:52.214907+0000","last_clean_scrub_stamp":"2024-02-21T23:09:52.214907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:51:08.543135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":425,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-02-21T23:10:07.610870+0000","last_change":"2024-02-21T23:02:32.289335+0000","last_active":"2024-02-21T23:10:07.610870+0000","last_peered":"2024-02-21T23:10:07.610870+0000","last_clean":"2024-02-21T23:10:07.610870+0000","last_became_active":"2024-02-21T23:02:32.286913+0000","last_became_peered":"2024-02-21T23:02:32.286913+0000","last_unstale":"2024-02-21T23:10:07.610870+0000","last_undegraded":"2024-02-21T23:10:07.610870+0000","last_fullsized":"2024-02-21T23:10:07.610870+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_clean_scrub_stamp":"2024-02-21T23:02:31.172569+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:36:23.987374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":26,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":25,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":24,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":23,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574942,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33148,"kb_used_data":1968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338692,"statfs":{"total":96636764160,"available":96602820608,"internally_reserved":0,"allocated":2015232,"data_stored":1546164,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574943,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33088,"kb_used_data":1908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338752,"statfs":{"total":96636764160,"available":96602882048,"internally_reserved":0,"allocated":1953792,"data_stored":955866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574942,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33112,"kb_used_data":1928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338728,"statfs":{"total":96636764160,"available":96602857472,"internally_reserved":0,"allocated":1974272,"data_stored":957554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574943,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33120,"kb_used_data":1936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338720,"statfs":{"total":96636764160,"available":96602849280,"internally_reserved":0,"allocated":1982464,"data_stored":958206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574943,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33120,"kb_used_data":1932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338720,"statfs":{"total":96636764160,"available":96602849280,"internally_reserved":0,"allocated":1978368,"data_stored":958172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574944,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33096,"kb_used_data":1908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338744,"statfs":{"total":96636764160,"available":96602873856,"internally_reserved":0,"allocated":1953792,"data_stored":956378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574942,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33104,"kb_used_data":1916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338736,"statfs":{"total":96636764160,"available":96602865664,"internally_reserved":0,"allocated":1961984,"data_stored":957408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574942,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33172,"kb_used_data":1992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338668,"statfs":{"total":96636764160,"available":96602796032,"internally_reserved":0,"allocated":2039808,"data_stored":1546996,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-21T23:10:16.807 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-02-21T23:10:16.807 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:10:17.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.079+0000 7fb82e600640 1 Processor -- start 2024-02-21T23:10:17.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.079+0000 7fb82e600640 1 -- start start 2024-02-21T23:10:17.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.081+0000 7fb82e600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb8280a5fd0 0x7fb82806dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:17.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.081+0000 7fb82e600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8280c2760 con 0x7fb82806ee40 2024-02-21T23:10:17.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.081+0000 7fb82e600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb828156310 con 0x7fb828150450 2024-02-21T23:10:17.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.081+0000 7fb82e600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8280c22c0 con 0x7fb8280a5fd0 2024-02-21T23:10:17.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.081+0000 7fb827e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb82806ee40 0x7fb828156af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39298/0 (socket says 172.21.15.28:39298) 2024-02-21T23:10:17.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.081+0000 7fb827e00640 1 -- 172.21.15.28:0/161936289 learned_addr learned my addr 172.21.15.28:0/161936289 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:10:17.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.082+0000 7fb82dc00640 1 --2- 172.21.15.28:0/161936289 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb8280a5fd0 0x7fb82806dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:17.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.082+0000 7fb827400640 1 -- 172.21.15.28:0/161936289 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 879672236 0 0) 0x7fb8280c2760 con 0x7fb82806ee40 2024-02-21T23:10:17.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.082+0000 7fb827400640 1 -- 172.21.15.28:0/161936289 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb810003520 con 0x7fb82806ee40 2024-02-21T23:10:17.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.082+0000 7fb827400640 1 -- 172.21.15.28:0/161936289 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2081124110 0 0) 0x7fb828156310 con 0x7fb828150450 2024-02-21T23:10:17.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.083+0000 7fb827400640 1 -- 172.21.15.28:0/161936289 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8280c2760 con 0x7fb828150450 2024-02-21T23:10:17.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.083+0000 7fb827400640 1 -- 172.21.15.28:0/161936289 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 822035446 0 0) 0x7fb810003520 con 0x7fb82806ee40 2024-02-21T23:10:17.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.083+0000 7fb827400640 1 -- 172.21.15.28:0/161936289 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb8280a5fd0 msgr2=0x7fb82806dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:17.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.083+0000 7fb827400640 1 --2- 172.21.15.28:0/161936289 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb8280a5fd0 0x7fb82806dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:17.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.083+0000 7fb827400640 1 -- 172.21.15.28:0/161936289 >> v1:172.21.15.81:6789/0 conn(0x7fb828150450 legacy=0x7fb8281536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:17.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.083+0000 7fb827400640 1 -- 172.21.15.28:0/161936289 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8280c29d0 con 0x7fb82806ee40 2024-02-21T23:10:17.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.083+0000 7fb827400640 1 -- 172.21.15.28:0/161936289 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fb814001670 con 0x7fb82806ee40 2024-02-21T23:10:17.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.083+0000 7fb827400640 1 -- 172.21.15.28:0/161936289 >> v1:172.21.15.28:6789/0 conn(0x7fb82806ee40 legacy=0x7fb828156af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:17.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.083+0000 7fb827400640 1 --2- 172.21.15.28:0/161936289 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb810003b70 0x7fb810003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:17.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.083+0000 7fb827400640 1 --2- 172.21.15.28:0/161936289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb810004800 0x7fb810004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:17.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.083+0000 7fb827400640 1 --2- 172.21.15.28:0/161936289 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb8280a5fd0 0x7fb8100051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:17.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.083+0000 7fb827400640 1 -- 172.21.15.28:0/161936289 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb81000ba70 con 0x7fb810004800 2024-02-21T23:10:17.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.083+0000 7fb827400640 1 -- 172.21.15.28:0/161936289 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb81000bc20 con 0x7fb810003b70 2024-02-21T23:10:17.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.083+0000 7fb827400640 1 -- 172.21.15.28:0/161936289 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb81000be20 con 0x7fb8280a5fd0 2024-02-21T23:10:17.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.083+0000 7fb827e00640 1 --2- 172.21.15.28:0/161936289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb810004800 0x7fb810004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:17.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.083+0000 7fb82d200640 1 --2- 172.21.15.28:0/161936289 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb810003b70 0x7fb810003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:17.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.083+0000 7fb82dc00640 1 --2- 172.21.15.28:0/161936289 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb8280a5fd0 0x7fb8100051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:17.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.084+0000 7fb827e00640 1 -- 172.21.15.28:0/161936289 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb8280a5fd0 msgr2=0x7fb8100051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:17.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.084+0000 7fb827e00640 1 --2- 172.21.15.28:0/161936289 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb8280a5fd0 0x7fb8100051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:17.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.084+0000 7fb827e00640 1 -- 172.21.15.28:0/161936289 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb810003b70 msgr2=0x7fb810003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:17.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.084+0000 7fb827e00640 1 --2- 172.21.15.28:0/161936289 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb810003b70 0x7fb810003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:17.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.084+0000 7fb827e00640 1 -- 172.21.15.28:0/161936289 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb810003520 con 0x7fb810004800 2024-02-21T23:10:17.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.085+0000 7fb827e00640 1 --2- 172.21.15.28:0/161936289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb810004800 0x7fb810004c80 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7fb81400c0e0 tx=0x7fb810005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=770ffebf9a00781 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:17.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.085+0000 7fb827400640 1 -- 172.21.15.28:0/161936289 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb814007c00 con 0x7fb810004800 2024-02-21T23:10:17.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.085+0000 7fb827400640 1 -- 172.21.15.28:0/161936289 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb814030070 con 0x7fb810004800 2024-02-21T23:10:17.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.085+0000 7fb827400640 1 -- 172.21.15.28:0/161936289 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb814005d10 con 0x7fb810004800 2024-02-21T23:10:17.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.086+0000 7fb82e600640 1 -- 172.21.15.28:0/161936289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb810004800 msgr2=0x7fb810004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:17.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.086+0000 7fb82e600640 1 --2- 172.21.15.28:0/161936289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb810004800 0x7fb810004c80 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7fb81400c0e0 tx=0x7fb810005830 comp rx=0 tx=0).stop 2024-02-21T23:10:17.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.086+0000 7fb82dc00640 1 -- 172.21.15.28:0/161936289 reap_dead start 2024-02-21T23:10:17.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.086+0000 7fb82e600640 1 -- 172.21.15.28:0/161936289 shutdown_connections 2024-02-21T23:10:17.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.086+0000 7fb82e600640 1 -- 172.21.15.28:0/161936289 >> 172.21.15.28:0/161936289 conn(0x7fb82806b630 msgr2=0x7fb8280c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:17.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.086+0000 7fb82e600640 1 -- 172.21.15.28:0/161936289 shutdown_connections 2024-02-21T23:10:17.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.086+0000 7fb82e600640 1 -- 172.21.15.28:0/161936289 wait complete. 2024-02-21T23:10:17.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.089+0000 7fb82e600640 1 Processor -- start 2024-02-21T23:10:17.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.089+0000 7fb82e600640 1 -- start start 2024-02-21T23:10:17.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.090+0000 7fb82e600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb810004800 0x7fb828154050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:17.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.090+0000 7fb82e600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb828151500 0x7fb8280c5b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:17.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.091+0000 7fb82e600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb8280c6040 0x7fb8281eb9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:17.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.091+0000 7fb82e600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb828157200 con 0x7fb828151500 2024-02-21T23:10:17.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.091+0000 7fb82e600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb828156660 con 0x7fb8280c6040 2024-02-21T23:10:17.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.091+0000 7fb82e600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb82806dff0 con 0x7fb810004800 2024-02-21T23:10:17.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.091+0000 7fb827e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb828151500 0x7fb8280c5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:17.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.091+0000 7fb827e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb828151500 0x7fb8280c5b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49768/0 (socket says 172.21.15.28:49768) 2024-02-21T23:10:17.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.091+0000 7fb827e00640 1 -- 172.21.15.28:0/100687791 learned_addr learned my addr 172.21.15.28:0/100687791 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:17.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.091+0000 7fb82d200640 1 --2- 172.21.15.28:0/100687791 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb810004800 0x7fb828154050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:17.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.091+0000 7fb82dc00640 1 --2- 172.21.15.28:0/100687791 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb8280c6040 0x7fb8281eb9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:17.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.092+0000 7fb827e00640 1 -- 172.21.15.28:0/100687791 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb810004800 msgr2=0x7fb828154050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:17.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.092+0000 7fb827e00640 1 --2- 172.21.15.28:0/100687791 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb810004800 0x7fb828154050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:17.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.092+0000 7fb827e00640 1 -- 172.21.15.28:0/100687791 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb8280c6040 msgr2=0x7fb8281eb9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:17.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.092+0000 7fb827e00640 1 --2- 172.21.15.28:0/100687791 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb8280c6040 0x7fb8281eb9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:17.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.092+0000 7fb827e00640 1 -- 172.21.15.28:0/100687791 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8281534b0 con 0x7fb828151500 2024-02-21T23:10:17.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.092+0000 7fb827e00640 1 --2- 172.21.15.28:0/100687791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb828151500 0x7fb8280c5b00 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7fb81400c200 tx=0x7fb828070ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:17.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.093+0000 7fb824c00640 1 -- 172.21.15.28:0/100687791 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb814045070 con 0x7fb828151500 2024-02-21T23:10:17.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.093+0000 7fb824c00640 1 -- 172.21.15.28:0/100687791 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb81402cba0 con 0x7fb828151500 2024-02-21T23:10:17.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.093+0000 7fb824c00640 1 -- 172.21.15.28:0/100687791 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb814007c00 con 0x7fb828151500 2024-02-21T23:10:17.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.093+0000 7fb82e600640 1 -- 172.21.15.28:0/100687791 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb828153790 con 0x7fb828151500 2024-02-21T23:10:17.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.093+0000 7fb82e600640 1 -- 172.21.15.28:0/100687791 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8280c55e0 con 0x7fb828151500 2024-02-21T23:10:17.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.095+0000 7fb82e600640 1 -- 172.21.15.28:0/100687791 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb82804f290 con 0x7fb828151500 2024-02-21T23:10:17.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.095+0000 7fb824c00640 1 -- 172.21.15.28:0/100687791 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb814007070 con 0x7fb828151500 2024-02-21T23:10:17.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.096+0000 7fb824c00640 1 --2- 172.21.15.28:0/100687791 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb80806b500 0x7fb80806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:17.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.097+0000 7fb824c00640 1 -- 172.21.15.28:0/100687791 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 6957+0+0 (secure 0 0 0) 0x7fb8140ab6a0 con 0x7fb828151500 2024-02-21T23:10:17.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.097+0000 7fb82d200640 1 --2- 172.21.15.28:0/100687791 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb80806b500 0x7fb80806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:17.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.098+0000 7fb82d200640 1 --2- 172.21.15.28:0/100687791 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb80806b500 0x7fb80806d9c0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fb81c007fd0 tx=0x7fb81c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:17.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.101+0000 7fb824c00640 1 -- 172.21.15.28:0/100687791 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb8140799e0 con 0x7fb828151500 2024-02-21T23:10:17.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.391+0000 7fb82e600640 1 -- 172.21.15.28:0/100687791 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb8281495f0 con 0x7fb828151500 2024-02-21T23:10:17.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.392+0000 7fb824c00640 1 -- 172.21.15.28:0/100687791 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 163 v163) v1 ==== 95+0+6191 (secure 0 0 0) 0x7fb81407da80 con 0x7fb828151500 2024-02-21T23:10:17.383 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:10:17.384 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":163,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6389,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1320495906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1320495906},{"type":"v1","addr":"172.21.15.81:6835","nonce":1320495906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":153},{"gid":6392,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/521625945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":521625945},{"type":"v1","addr":"172.21.15.81:6837","nonce":521625945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":153}],"filesystems":[{"mdsmap":{"epoch":158,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:09:53.730296+0000","modified":"2024-02-21T23:09:56.809990+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8826},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8826":{"gid":8826,"name":"a","rank":0,"incarnation":154,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/2928639298","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2928639298},{"type":"v1","addr":"172.21.15.28:6835","nonce":2928639298}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11},{"mdsmap":{"epoch":163,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:10:05.843850+0000","modified":"2024-02-21T23:10:08.927829+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8817},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8817":{"gid":8817,"name":"c","rank":0,"incarnation":159,"state":"up:active","state_seq":6,"addr":"172.21.15.28:6833/2975378719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2975378719},{"type":"v1","addr":"172.21.15.28:6833","nonce":2975378719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2024-02-21T23:10:17.385 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 163 2024-02-21T23:10:17.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.395+0000 7fb82e600640 1 -- 172.21.15.28:0/100687791 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb80806b500 msgr2=0x7fb80806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:17.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.395+0000 7fb82e600640 1 --2- 172.21.15.28:0/100687791 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb80806b500 0x7fb80806d9c0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fb81c007fd0 tx=0x7fb81c009040 comp rx=0 tx=0).stop 2024-02-21T23:10:17.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.395+0000 7fb82e600640 1 -- 172.21.15.28:0/100687791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb828151500 msgr2=0x7fb8280c5b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:17.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.395+0000 7fb82e600640 1 --2- 172.21.15.28:0/100687791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb828151500 0x7fb8280c5b00 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7fb81400c200 tx=0x7fb828070ec0 comp rx=0 tx=0).stop 2024-02-21T23:10:17.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.395+0000 7fb82e600640 1 -- 172.21.15.28:0/100687791 shutdown_connections 2024-02-21T23:10:17.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.395+0000 7fb82e600640 1 --2- 172.21.15.28:0/100687791 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb80806b500 0x7fb80806d9c0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:17.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.395+0000 7fb82e600640 1 --2- 172.21.15.28:0/100687791 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb8280c6040 0x7fb8281eb9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:17.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.395+0000 7fb82e600640 1 --2- 172.21.15.28:0/100687791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb828151500 0x7fb8280c5b00 unknown :-1 s=CLOSED pgs=1755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:17.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.395+0000 7fb82e600640 1 --2- 172.21.15.28:0/100687791 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb810004800 0x7fb828154050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:17.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.395+0000 7fb82e600640 1 -- 172.21.15.28:0/100687791 >> 172.21.15.28:0/100687791 conn(0x7fb82806b630 msgr2=0x7fb828074540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:17.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.395+0000 7fb82e600640 1 -- 172.21.15.28:0/100687791 shutdown_connections 2024-02-21T23:10:17.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.395+0000 7fb82e600640 1 -- 172.21.15.28:0/100687791 wait complete. 2024-02-21T23:10:17.409 INFO:tasks.cephfs.test_failover:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 11, "epoch": 163, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 11, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-02-21T23:09:53.730296+0000", "damaged": [], "data_pools": [ 24 ], "enabled": true, "epoch": 158, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_8826": { "addr": "172.21.15.28:6835/2928639298", "addrs": { "addrvec": [ { "addr": "172.21.15.28:6834", "nonce": 2928639298, "type": "v2" }, { "addr": "172.21.15.28:6835", "nonce": 2928639298, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8826, "incarnation": 154, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 23, "modified": "2024-02-21T23:09:56.809990+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8826 } } }, { "id": 12, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-02-21T23:10:05.843850+0000", "damaged": [], "data_pools": [ 26 ], "enabled": true, "epoch": 163, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs2", "in": [ 0 ], "info": { "gid_8817": { "addr": "172.21.15.28:6833/2975378719", "addrs": { "addrvec": [ { "addr": "172.21.15.28:6832", "nonce": 2975378719, "type": "v2" }, { "addr": "172.21.15.28:6833", "nonce": 2975378719, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8817, "incarnation": 159, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 6 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 25, "modified": "2024-02-21T23:10:08.927829+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8817 } } } ], "standbys": [ { "addr": "172.21.15.81:6835/1320495906", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6834", "nonce": 1320495906, "type": "v2" }, { "addr": "172.21.15.81:6835", "nonce": 1320495906, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 153, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6389, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.81:6837/521625945", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6836", "nonce": 521625945, "type": "v2" }, { "addr": "172.21.15.81:6837", "nonce": 521625945, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 153, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6392, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-02-21T23:10:17.409 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-02-21T23:10:17.409 INFO:tasks.cephfs.test_failover:current = [{'gid': 8826, 'name': 'a', 'rank': 0, 'incarnation': 154, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/2928639298', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 2928639298}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 2928639298}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 6389, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.81:6835/1320495906', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6834', 'nonce': 1320495906}, {'type': 'v1', 'addr': '172.21.15.81:6835', 'nonce': 1320495906}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 153}, {'gid': 8817, 'name': 'c', 'rank': 0, 'incarnation': 159, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.28:6833/2975378719', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 2975378719}, {'type': 'v1', 'addr': '172.21.15.28:6833', 'nonce': 2975378719}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 6392, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.81:6837/521625945', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 521625945}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 521625945}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 153}] 2024-02-21T23:10:17.409 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:10:17.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.634+0000 7ff35ca00640 1 Processor -- start 2024-02-21T23:10:17.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.634+0000 7ff35ca00640 1 -- start start 2024-02-21T23:10:17.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.636+0000 7ff35ca00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff358155960 0x7ff3581608a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:17.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.636+0000 7ff35ca00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3580c74c0 con 0x7ff3580aff90 2024-02-21T23:10:17.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.636+0000 7ff35ca00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3580c77a0 con 0x7ff35806dbf0 2024-02-21T23:10:17.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.636+0000 7ff35ca00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3580c7270 con 0x7ff358155960 2024-02-21T23:10:17.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.636+0000 7ff356800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff3580aff90 0x7ff3580c51e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39302/0 (socket says 172.21.15.28:39302) 2024-02-21T23:10:17.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.636+0000 7ff356800640 1 -- 172.21.15.28:0/3778198151 learned_addr learned my addr 172.21.15.28:0/3778198151 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:10:17.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.636+0000 7ff357200640 1 --2- 172.21.15.28:0/3778198151 >> v2:172.21.15.28:3300/0 conn(0x7ff358155960 0x7ff3581608a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:17.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.637+0000 7ff355400640 1 -- 172.21.15.28:0/3778198151 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2849233506 0 0) 0x7ff3580c74c0 con 0x7ff3580aff90 2024-02-21T23:10:17.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.637+0000 7ff355400640 1 -- 172.21.15.28:0/3778198151 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff33c003520 con 0x7ff3580aff90 2024-02-21T23:10:17.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.637+0000 7ff357200640 1 -- 172.21.15.28:0/3778198151 >> v1:172.21.15.81:6789/0 conn(0x7ff35806dbf0 legacy=0x7ff358158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:17.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.637+0000 7ff357200640 1 -- 172.21.15.28:0/3778198151 >> v1:172.21.15.28:6789/0 conn(0x7ff3580aff90 legacy=0x7ff3580c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:17.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.637+0000 7ff357200640 1 -- 172.21.15.28:0/3778198151 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3580c7a80 con 0x7ff358155960 2024-02-21T23:10:17.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.638+0000 7ff357200640 1 --2- 172.21.15.28:0/3778198151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff358155960 0x7ff3581608a0 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7ff350009d90 tx=0x7ff35002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b243a08722b44011 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:17.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.638+0000 7ff355400640 1 -- 172.21.15.28:0/3778198151 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff350039070 con 0x7ff358155960 2024-02-21T23:10:17.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.638+0000 7ff355400640 1 -- 172.21.15.28:0/3778198151 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff35002cc60 con 0x7ff358155960 2024-02-21T23:10:17.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.638+0000 7ff355400640 1 -- 172.21.15.28:0/3778198151 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff350034660 con 0x7ff358155960 2024-02-21T23:10:17.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.639+0000 7ff35ca00640 1 -- 172.21.15.28:0/3778198151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff358155960 msgr2=0x7ff3581608a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:17.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.639+0000 7ff35ca00640 1 --2- 172.21.15.28:0/3778198151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff358155960 0x7ff3581608a0 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7ff350009d90 tx=0x7ff35002beb0 comp rx=0 tx=0).stop 2024-02-21T23:10:17.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.639+0000 7ff35ca00640 1 -- 172.21.15.28:0/3778198151 shutdown_connections 2024-02-21T23:10:17.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.639+0000 7ff35ca00640 1 --2- 172.21.15.28:0/3778198151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff358155960 0x7ff3581608a0 unknown :-1 s=CLOSED pgs=1757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:17.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.639+0000 7ff35ca00640 1 -- 172.21.15.28:0/3778198151 >> 172.21.15.28:0/3778198151 conn(0x7ff3580ac370 msgr2=0x7ff3580ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:17.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.639+0000 7ff35ca00640 1 -- 172.21.15.28:0/3778198151 shutdown_connections 2024-02-21T23:10:17.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.639+0000 7ff35ca00640 1 -- 172.21.15.28:0/3778198151 wait complete. 2024-02-21T23:10:17.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.642+0000 7ff35ca00640 1 Processor -- start 2024-02-21T23:10:17.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.642+0000 7ff35ca00640 1 -- start start 2024-02-21T23:10:17.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.644+0000 7ff35ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff35806dbf0 0x7ff3581505a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:17.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.644+0000 7ff35ca00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff3580aff90 0x7ff3581ee4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:17.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.644+0000 7ff35ca00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff358155960 0x7ff358154040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:17.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.644+0000 7ff35ca00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3580c58f0 con 0x7ff35806dbf0 2024-02-21T23:10:17.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.644+0000 7ff35ca00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff358158630 con 0x7ff358155960 2024-02-21T23:10:17.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.644+0000 7ff35ca00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3580c5a70 con 0x7ff3580aff90 2024-02-21T23:10:17.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.644+0000 7ff356800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff35806dbf0 0x7ff3581505a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:17.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.644+0000 7ff356800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff35806dbf0 0x7ff3581505a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49786/0 (socket says 172.21.15.28:49786) 2024-02-21T23:10:17.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.644+0000 7ff356800640 1 -- 172.21.15.28:0/1962363151 learned_addr learned my addr 172.21.15.28:0/1962363151 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:17.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.644+0000 7ff357200640 1 --2- 172.21.15.28:0/1962363151 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff358155960 0x7ff358154040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:17.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.644+0000 7ff356800640 1 -- 172.21.15.28:0/1962363151 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff3580aff90 msgr2=0x7ff3581ee4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:17.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.644+0000 7ff356800640 1 --2- 172.21.15.28:0/1962363151 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff3580aff90 0x7ff3581ee4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:17.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.644+0000 7ff356800640 1 -- 172.21.15.28:0/1962363151 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff358155960 msgr2=0x7ff358154040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:17.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.644+0000 7ff356800640 1 --2- 172.21.15.28:0/1962363151 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff358155960 0x7ff358154040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:17.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.644+0000 7ff356800640 1 -- 172.21.15.28:0/1962363151 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3581514b0 con 0x7ff35806dbf0 2024-02-21T23:10:17.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.644+0000 7ff357200640 1 --2- 172.21.15.28:0/1962363151 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff358155960 0x7ff358154040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:10:17.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.645+0000 7ff356800640 1 --2- 172.21.15.28:0/1962363151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff35806dbf0 0x7ff3581505a0 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7ff34800c430 tx=0x7ff34800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:17.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.645+0000 7ff34ea00640 1 -- 172.21.15.28:0/1962363151 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff348017070 con 0x7ff35806dbf0 2024-02-21T23:10:17.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.645+0000 7ff34ea00640 1 -- 172.21.15.28:0/1962363151 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff348005ce0 con 0x7ff35806dbf0 2024-02-21T23:10:17.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.645+0000 7ff34ea00640 1 -- 172.21.15.28:0/1962363151 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3480072f0 con 0x7ff35806dbf0 2024-02-21T23:10:17.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.645+0000 7ff35ca00640 1 -- 172.21.15.28:0/1962363151 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff358151790 con 0x7ff35806dbf0 2024-02-21T23:10:17.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.645+0000 7ff35ca00640 1 -- 172.21.15.28:0/1962363151 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff35814fda0 con 0x7ff35806dbf0 2024-02-21T23:10:17.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.647+0000 7ff35ca00640 1 -- 172.21.15.28:0/1962363151 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff35804f290 con 0x7ff35806dbf0 2024-02-21T23:10:17.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.647+0000 7ff34ea00640 1 -- 172.21.15.28:0/1962363151 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff34801a070 con 0x7ff35806dbf0 2024-02-21T23:10:17.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.649+0000 7ff34ea00640 1 --2- 172.21.15.28:0/1962363151 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff33806b4d0 0x7ff33806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:17.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.649+0000 7ff34ea00640 1 -- 172.21.15.28:0/1962363151 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 6957+0+0 (secure 0 0 0) 0x7ff348089b10 con 0x7ff35806dbf0 2024-02-21T23:10:17.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.650+0000 7ff355e00640 1 --2- 172.21.15.28:0/1962363151 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff33806b4d0 0x7ff33806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:17.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.651+0000 7ff355e00640 1 --2- 172.21.15.28:0/1962363151 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff33806b4d0 0x7ff33806d990 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7ff3581575e0 tx=0x7ff3440079b0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:17.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.653+0000 7ff34ea00640 1 -- 172.21.15.28:0/1962363151 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ff348058600 con 0x7ff35806dbf0 2024-02-21T23:10:17.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.985+0000 7ff35ca00640 1 -- 172.21.15.28:0/1962363151 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff35806d9d0 con 0x7ff35806dbf0 2024-02-21T23:10:17.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.986+0000 7ff34ea00640 1 -- 172.21.15.28:0/1962363151 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 163 v163) v1 ==== 95+0+6191 (secure 0 0 0) 0x7ff348057f00 con 0x7ff35806dbf0 2024-02-21T23:10:17.976 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:10:17.976 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":163,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6389,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1320495906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1320495906},{"type":"v1","addr":"172.21.15.81:6835","nonce":1320495906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":153},{"gid":6392,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/521625945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":521625945},{"type":"v1","addr":"172.21.15.81:6837","nonce":521625945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":153}],"filesystems":[{"mdsmap":{"epoch":158,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:09:53.730296+0000","modified":"2024-02-21T23:09:56.809990+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8826},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8826":{"gid":8826,"name":"a","rank":0,"incarnation":154,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/2928639298","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2928639298},{"type":"v1","addr":"172.21.15.28:6835","nonce":2928639298}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11},{"mdsmap":{"epoch":163,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:10:05.843850+0000","modified":"2024-02-21T23:10:08.927829+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8817},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8817":{"gid":8817,"name":"c","rank":0,"incarnation":159,"state":"up:active","state_seq":6,"addr":"172.21.15.28:6833/2975378719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2975378719},{"type":"v1","addr":"172.21.15.28:6833","nonce":2975378719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2024-02-21T23:10:17.977 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 163 2024-02-21T23:10:17.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.988+0000 7ff35ca00640 1 -- 172.21.15.28:0/1962363151 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff33806b4d0 msgr2=0x7ff33806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:17.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.988+0000 7ff35ca00640 1 --2- 172.21.15.28:0/1962363151 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff33806b4d0 0x7ff33806d990 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7ff3581575e0 tx=0x7ff3440079b0 comp rx=0 tx=0).stop 2024-02-21T23:10:17.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.988+0000 7ff35ca00640 1 -- 172.21.15.28:0/1962363151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff35806dbf0 msgr2=0x7ff3581505a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:17.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.988+0000 7ff35ca00640 1 --2- 172.21.15.28:0/1962363151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff35806dbf0 0x7ff3581505a0 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7ff34800c430 tx=0x7ff34800c8f0 comp rx=0 tx=0).stop 2024-02-21T23:10:17.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.988+0000 7ff35ca00640 1 -- 172.21.15.28:0/1962363151 shutdown_connections 2024-02-21T23:10:17.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.988+0000 7ff35ca00640 1 --2- 172.21.15.28:0/1962363151 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff33806b4d0 0x7ff33806d990 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:17.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.988+0000 7ff35ca00640 1 --2- 172.21.15.28:0/1962363151 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff3580aff90 0x7ff3581ee4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:17.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.988+0000 7ff35ca00640 1 --2- 172.21.15.28:0/1962363151 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff358155960 0x7ff358154040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:17.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.988+0000 7ff35ca00640 1 --2- 172.21.15.28:0/1962363151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff35806dbf0 0x7ff3581505a0 unknown :-1 s=CLOSED pgs=1758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:17.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.988+0000 7ff35ca00640 1 -- 172.21.15.28:0/1962363151 >> 172.21.15.28:0/1962363151 conn(0x7ff3580ac370 msgr2=0x7ff3580c50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:17.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.988+0000 7ff35ca00640 1 -- 172.21.15.28:0/1962363151 shutdown_connections 2024-02-21T23:10:17.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:17.988+0000 7ff35ca00640 1 -- 172.21.15.28:0/1962363151 wait complete. 2024-02-21T23:10:17.995 INFO:tasks.cephfs.test_failover:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 11, "epoch": 163, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 11, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-02-21T23:09:53.730296+0000", "damaged": [], "data_pools": [ 24 ], "enabled": true, "epoch": 158, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_8826": { "addr": "172.21.15.28:6835/2928639298", "addrs": { "addrvec": [ { "addr": "172.21.15.28:6834", "nonce": 2928639298, "type": "v2" }, { "addr": "172.21.15.28:6835", "nonce": 2928639298, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8826, "incarnation": 154, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 23, "modified": "2024-02-21T23:09:56.809990+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8826 } } }, { "id": 12, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-02-21T23:10:05.843850+0000", "damaged": [], "data_pools": [ 26 ], "enabled": true, "epoch": 163, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs2", "in": [ 0 ], "info": { "gid_8817": { "addr": "172.21.15.28:6833/2975378719", "addrs": { "addrvec": [ { "addr": "172.21.15.28:6832", "nonce": 2975378719, "type": "v2" }, { "addr": "172.21.15.28:6833", "nonce": 2975378719, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8817, "incarnation": 159, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 6 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 25, "modified": "2024-02-21T23:10:08.927829+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8817 } } } ], "standbys": [ { "addr": "172.21.15.81:6835/1320495906", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6834", "nonce": 1320495906, "type": "v2" }, { "addr": "172.21.15.81:6835", "nonce": 1320495906, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 153, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6389, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.81:6837/521625945", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6836", "nonce": 521625945, "type": "v2" }, { "addr": "172.21.15.81:6837", "nonce": 521625945, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 153, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6392, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-02-21T23:10:17.995 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': -1, 'name': 'b'}, {'join_fscid': -1, 'name': 'c'}, {'join_fscid': -1, 'name': 'd'}] 2024-02-21T23:10:17.995 INFO:tasks.cephfs.test_failover:current = [{'gid': 8826, 'name': 'a', 'rank': 0, 'incarnation': 154, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/2928639298', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 2928639298}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 2928639298}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 6389, 'name': 'b', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.81:6835/1320495906', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6834', 'nonce': 1320495906}, {'type': 'v1', 'addr': '172.21.15.81:6835', 'nonce': 1320495906}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 153}, {'gid': 8817, 'name': 'c', 'rank': 0, 'incarnation': 159, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.28:6833/2975378719', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 2975378719}, {'type': 'v1', 'addr': '172.21.15.28:6833', 'nonce': 2975378719}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 6392, 'name': 'd', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.81:6837/521625945', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 521625945}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 521625945}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 153}] 2024-02-21T23:10:17.996 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.b mds_join_fs cephfs2 2024-02-21T23:10:18.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.213+0000 7fc94b000640 1 Processor -- start 2024-02-21T23:10:18.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.213+0000 7fc94b000640 1 -- start start 2024-02-21T23:10:18.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.217+0000 7fc94b000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc94413e0c0 0x7fc9440c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:18.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.217+0000 7fc94b000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fc94406dbc0 0x7fc94406c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:18.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.217+0000 7fc94b000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc944158550 con 0x7fc9440c76f0 2024-02-21T23:10:18.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.217+0000 7fc94b000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9440c89a0 con 0x7fc94413e0c0 2024-02-21T23:10:18.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.217+0000 7fc94b000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9441587d0 con 0x7fc94406dbc0 2024-02-21T23:10:18.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.217+0000 7fc949c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc94413e0c0 0x7fc9440c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:18.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.217+0000 7fc949c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc94413e0c0 0x7fc9440c71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49800/0 (socket says 172.21.15.28:49800) 2024-02-21T23:10:18.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.217+0000 7fc949c00640 1 -- 172.21.15.28:0/1360393283 learned_addr learned my addr 172.21.15.28:0/1360393283 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:18.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.217+0000 7fc949200640 1 --2- 172.21.15.28:0/1360393283 >> v2:172.21.15.81:3300/0 conn(0x7fc94406dbc0 0x7fc94406c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:18.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.217+0000 7fc949c00640 1 -- 172.21.15.28:0/1360393283 >> v2:172.21.15.81:3300/0 conn(0x7fc94406dbc0 msgr2=0x7fc94406c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:18.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.217+0000 7fc949c00640 1 --2- 172.21.15.28:0/1360393283 >> v2:172.21.15.81:3300/0 conn(0x7fc94406dbc0 0x7fc94406c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:18.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.217+0000 7fc949c00640 1 -- 172.21.15.28:0/1360393283 >> v1:172.21.15.28:6789/0 conn(0x7fc9440c76f0 legacy=0x7fc94406c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:18.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.217+0000 7fc949c00640 1 -- 172.21.15.28:0/1360393283 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc944158ab0 con 0x7fc94413e0c0 2024-02-21T23:10:18.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.219+0000 7fc949c00640 1 --2- 172.21.15.28:0/1360393283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc94413e0c0 0x7fc9440c71b0 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7fc9440c7100 tx=0x7fc93802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=213d008252e550f8 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:18.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.219+0000 7fc943e00640 1 -- 172.21.15.28:0/1360393283 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc938035020 con 0x7fc94413e0c0 2024-02-21T23:10:18.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.219+0000 7fc943e00640 1 -- 172.21.15.28:0/1360393283 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc938031870 con 0x7fc94413e0c0 2024-02-21T23:10:18.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.219+0000 7fc943e00640 1 -- 172.21.15.28:0/1360393283 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc938031c40 con 0x7fc94413e0c0 2024-02-21T23:10:18.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.220+0000 7fc94b000640 1 -- 172.21.15.28:0/1360393283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc94413e0c0 msgr2=0x7fc9440c71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:18.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.220+0000 7fc94b000640 1 --2- 172.21.15.28:0/1360393283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc94413e0c0 0x7fc9440c71b0 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7fc9440c7100 tx=0x7fc93802b910 comp rx=0 tx=0).stop 2024-02-21T23:10:18.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.220+0000 7fc94b000640 1 -- 172.21.15.28:0/1360393283 shutdown_connections 2024-02-21T23:10:18.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.220+0000 7fc94b000640 1 --2- 172.21.15.28:0/1360393283 >> v2:172.21.15.81:3300/0 conn(0x7fc94406dbc0 0x7fc94406c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:18.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.220+0000 7fc94b000640 1 --2- 172.21.15.28:0/1360393283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc94413e0c0 0x7fc9440c71b0 unknown :-1 s=CLOSED pgs=1760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:18.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.220+0000 7fc94b000640 1 -- 172.21.15.28:0/1360393283 >> 172.21.15.28:0/1360393283 conn(0x7fc944073110 msgr2=0x7fc9440ac490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:18.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.220+0000 7fc94b000640 1 -- 172.21.15.28:0/1360393283 shutdown_connections 2024-02-21T23:10:18.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.220+0000 7fc94b000640 1 -- 172.21.15.28:0/1360393283 wait complete. 2024-02-21T23:10:18.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.221+0000 7fc94b000640 1 Processor -- start 2024-02-21T23:10:18.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.222+0000 7fc94b000640 1 -- start start 2024-02-21T23:10:18.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.222+0000 7fc94b000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc94406dbc0 0x7fc9440aade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:18.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.222+0000 7fc94b000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9440c76f0 0x7fc9441eb090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:18.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.223+0000 7fc94b000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc94413e0c0 0x7fc944157640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:18.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.223+0000 7fc94b000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc94406ced0 con 0x7fc9440c76f0 2024-02-21T23:10:18.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.223+0000 7fc94b000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc944159070 con 0x7fc94406dbc0 2024-02-21T23:10:18.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.223+0000 7fc94b000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc94413e520 con 0x7fc94413e0c0 2024-02-21T23:10:18.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.223+0000 7fc949200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9440c76f0 0x7fc9441eb090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:18.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.223+0000 7fc949200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9440c76f0 0x7fc9441eb090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49814/0 (socket says 172.21.15.28:49814) 2024-02-21T23:10:18.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.223+0000 7fc949200640 1 -- 172.21.15.28:0/766911084 learned_addr learned my addr 172.21.15.28:0/766911084 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:18.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.223+0000 7fc949c00640 1 --2- 172.21.15.28:0/766911084 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc94406dbc0 0x7fc9440aade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:18.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.223+0000 7fc949200640 1 -- 172.21.15.28:0/766911084 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc94413e0c0 msgr2=0x7fc944157640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:18.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.223+0000 7fc949200640 1 --2- 172.21.15.28:0/766911084 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc94413e0c0 0x7fc944157640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:18.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.223+0000 7fc949200640 1 -- 172.21.15.28:0/766911084 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc94406dbc0 msgr2=0x7fc9440aade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:18.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.223+0000 7fc949200640 1 --2- 172.21.15.28:0/766911084 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc94406dbc0 0x7fc9440aade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:18.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.223+0000 7fc949200640 1 -- 172.21.15.28:0/766911084 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9440ac100 con 0x7fc9440c76f0 2024-02-21T23:10:18.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.223+0000 7fc949c00640 1 --2- 172.21.15.28:0/766911084 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc94406dbc0 0x7fc9440aade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:10:18.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.224+0000 7fc949200640 1 --2- 172.21.15.28:0/766911084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9440c76f0 0x7fc9441eb090 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7fc93000d800 tx=0x7fc93000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:18.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.224+0000 7fc941600640 1 -- 172.21.15.28:0/766911084 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc93000b6c0 con 0x7fc9440c76f0 2024-02-21T23:10:18.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.224+0000 7fc941600640 1 -- 172.21.15.28:0/766911084 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc930019070 con 0x7fc9440c76f0 2024-02-21T23:10:18.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.224+0000 7fc941600640 1 -- 172.21.15.28:0/766911084 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc93000ba80 con 0x7fc9440c76f0 2024-02-21T23:10:18.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.224+0000 7fc94b000640 1 -- 172.21.15.28:0/766911084 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9440ac3e0 con 0x7fc9440c76f0 2024-02-21T23:10:18.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.224+0000 7fc94b000640 1 -- 172.21.15.28:0/766911084 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9440aa5e0 con 0x7fc9440c76f0 2024-02-21T23:10:18.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.226+0000 7fc94b000640 1 -- 172.21.15.28:0/766911084 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc94404f290 con 0x7fc9440c76f0 2024-02-21T23:10:18.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.226+0000 7fc941600640 1 -- 172.21.15.28:0/766911084 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc93000bc20 con 0x7fc9440c76f0 2024-02-21T23:10:18.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.227+0000 7fc941600640 1 --2- 172.21.15.28:0/766911084 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc92406b4d0 0x7fc92406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:18.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.227+0000 7fc941600640 1 -- 172.21.15.28:0/766911084 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 6957+0+0 (secure 0 0 0) 0x7fc93008abd0 con 0x7fc9440c76f0 2024-02-21T23:10:18.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.228+0000 7fc949c00640 1 --2- 172.21.15.28:0/766911084 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc92406b4d0 0x7fc92406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:18.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.229+0000 7fc949c00640 1 --2- 172.21.15.28:0/766911084 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc92406b4d0 0x7fc92406d990 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fc938036000 tx=0x7fc938008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:18.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.232+0000 7fc941600640 1 -- 172.21.15.28:0/766911084 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc930059560 con 0x7fc9440c76f0 2024-02-21T23:10:18.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.498+0000 7fc94b000640 1 -- 172.21.15.28:0/766911084 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) v1 -- 0x7fc9440c5870 con 0x7fc9440c76f0 2024-02-21T23:10:18.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.503+0000 7fc941600640 1 -- 172.21.15.28:0/766911084 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v52)=0 v52) v1 ==== 117+0+0 (secure 0 0 0) 0x7fc930058e60 con 0x7fc9440c76f0 2024-02-21T23:10:18.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.505+0000 7fc94b000640 1 -- 172.21.15.28:0/766911084 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc92406b4d0 msgr2=0x7fc92406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:18.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.505+0000 7fc94b000640 1 --2- 172.21.15.28:0/766911084 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc92406b4d0 0x7fc92406d990 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fc938036000 tx=0x7fc938008030 comp rx=0 tx=0).stop 2024-02-21T23:10:18.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.506+0000 7fc94b000640 1 -- 172.21.15.28:0/766911084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9440c76f0 msgr2=0x7fc9441eb090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:18.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.506+0000 7fc94b000640 1 --2- 172.21.15.28:0/766911084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9440c76f0 0x7fc9441eb090 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7fc93000d800 tx=0x7fc93000dcc0 comp rx=0 tx=0).stop 2024-02-21T23:10:18.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.506+0000 7fc94b000640 1 -- 172.21.15.28:0/766911084 shutdown_connections 2024-02-21T23:10:18.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.506+0000 7fc94b000640 1 --2- 172.21.15.28:0/766911084 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc92406b4d0 0x7fc92406d990 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:18.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.506+0000 7fc94b000640 1 --2- 172.21.15.28:0/766911084 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc94413e0c0 0x7fc944157640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:18.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.506+0000 7fc94b000640 1 --2- 172.21.15.28:0/766911084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9440c76f0 0x7fc9441eb090 unknown :-1 s=CLOSED pgs=1761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:18.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.506+0000 7fc94b000640 1 --2- 172.21.15.28:0/766911084 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc94406dbc0 0x7fc9440aade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:18.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.506+0000 7fc94b000640 1 -- 172.21.15.28:0/766911084 >> 172.21.15.28:0/766911084 conn(0x7fc944073110 msgr2=0x7fc9440c8fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:18.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.506+0000 7fc94b000640 1 -- 172.21.15.28:0/766911084 shutdown_connections 2024-02-21T23:10:18.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.506+0000 7fc94b000640 1 -- 172.21.15.28:0/766911084 wait complete. 2024-02-21T23:10:18.514 INFO:tasks.cephfs.test_failover:assigning affinity to cephfs2 for active mds (mds.c) 2024-02-21T23:10:18.514 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds.c mds_join_fs cephfs2 2024-02-21T23:10:18.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.741+0000 7f0923200640 1 Processor -- start 2024-02-21T23:10:18.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.742+0000 7f0923200640 1 -- start start 2024-02-21T23:10:18.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.743+0000 7f0923200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f092413e0c0 0x7f09240c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:18.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.743+0000 7f0923200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f092406dbc0 0x7f092406c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:18.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.743+0000 7f0923200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f09240c76f0 0x7f092406c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:18.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.743+0000 7f0923200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09240c89a0 con 0x7f092406dbc0 2024-02-21T23:10:18.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.743+0000 7f0923200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0924158550 con 0x7f09240c76f0 2024-02-21T23:10:18.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.743+0000 7f0923200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09241587d0 con 0x7f092413e0c0 2024-02-21T23:10:18.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.744+0000 7f0921400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f092406dbc0 0x7f092406c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:18.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.744+0000 7f0921400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f092406dbc0 0x7f092406c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49816/0 (socket says 172.21.15.28:49816) 2024-02-21T23:10:18.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.744+0000 7f0921400640 1 -- 172.21.15.28:0/2527266329 learned_addr learned my addr 172.21.15.28:0/2527266329 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:18.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.744+0000 7f0921e00640 1 --2- 172.21.15.28:0/2527266329 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f092413e0c0 0x7f09240c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:18.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.744+0000 7f0922800640 1 --2- 172.21.15.28:0/2527266329 >> v2:172.21.15.81:3300/0 conn(0x7f09240c76f0 0x7f092406c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:18.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.744+0000 7f0921400640 1 -- 172.21.15.28:0/2527266329 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f092413e0c0 msgr2=0x7f09240c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:18.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.744+0000 7f0921400640 1 --2- 172.21.15.28:0/2527266329 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f092413e0c0 0x7f09240c71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:18.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.744+0000 7f0921400640 1 -- 172.21.15.28:0/2527266329 >> v2:172.21.15.81:3300/0 conn(0x7f09240c76f0 msgr2=0x7f092406c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:18.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.744+0000 7f0921400640 1 --2- 172.21.15.28:0/2527266329 >> v2:172.21.15.81:3300/0 conn(0x7f09240c76f0 0x7f092406c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:18.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.744+0000 7f0921400640 1 -- 172.21.15.28:0/2527266329 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0924158ab0 con 0x7f092406dbc0 2024-02-21T23:10:18.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.746+0000 7f0921400640 1 --2- 172.21.15.28:0/2527266329 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f092406dbc0 0x7f092406c280 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7f092406bd10 tx=0x7f091002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=306b587b6d4f7b2a server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:18.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.746+0000 7f0920a00640 1 -- 172.21.15.28:0/2527266329 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f091002e070 con 0x7f092406dbc0 2024-02-21T23:10:18.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.746+0000 7f0920a00640 1 -- 172.21.15.28:0/2527266329 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0910037070 con 0x7f092406dbc0 2024-02-21T23:10:18.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.746+0000 7f0920a00640 1 -- 172.21.15.28:0/2527266329 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0910031a00 con 0x7f092406dbc0 2024-02-21T23:10:18.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.747+0000 7f0923200640 1 -- 172.21.15.28:0/2527266329 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f092406dbc0 msgr2=0x7f092406c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:18.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.747+0000 7f0923200640 1 --2- 172.21.15.28:0/2527266329 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f092406dbc0 0x7f092406c280 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7f092406bd10 tx=0x7f091002b880 comp rx=0 tx=0).stop 2024-02-21T23:10:18.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.747+0000 7f0923200640 1 -- 172.21.15.28:0/2527266329 shutdown_connections 2024-02-21T23:10:18.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.747+0000 7f0923200640 1 --2- 172.21.15.28:0/2527266329 >> v2:172.21.15.81:3300/0 conn(0x7f09240c76f0 0x7f092406c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:18.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.747+0000 7f0923200640 1 --2- 172.21.15.28:0/2527266329 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f092406dbc0 0x7f092406c280 unknown :-1 s=CLOSED pgs=1762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:18.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.747+0000 7f0923200640 1 --2- 172.21.15.28:0/2527266329 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f092413e0c0 0x7f09240c71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:18.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.747+0000 7f0923200640 1 -- 172.21.15.28:0/2527266329 >> 172.21.15.28:0/2527266329 conn(0x7f0924073110 msgr2=0x7f09240ac490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:18.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.747+0000 7f0923200640 1 -- 172.21.15.28:0/2527266329 shutdown_connections 2024-02-21T23:10:18.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.747+0000 7f0923200640 1 -- 172.21.15.28:0/2527266329 wait complete. 2024-02-21T23:10:18.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.750+0000 7f0923200640 1 Processor -- start 2024-02-21T23:10:18.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.750+0000 7f0923200640 1 -- start start 2024-02-21T23:10:18.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.751+0000 7f0923200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f092406dbc0 0x7f09241ee210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:18.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.751+0000 7f0923200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f09240c76f0 0x7f0924136220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:18.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.751+0000 7f0923200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f092413e0c0 0x7f0924136760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:18.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.752+0000 7f0923200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f092406cd00 con 0x7f092406dbc0 2024-02-21T23:10:18.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.752+0000 7f0923200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0924159070 con 0x7f092413e0c0 2024-02-21T23:10:18.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.752+0000 7f0923200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f092406ce80 con 0x7f09240c76f0 2024-02-21T23:10:18.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.752+0000 7f0921e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f092406dbc0 0x7f09241ee210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:18.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.752+0000 7f0921e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f092406dbc0 0x7f09241ee210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49828/0 (socket says 172.21.15.28:49828) 2024-02-21T23:10:18.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.752+0000 7f0921e00640 1 -- 172.21.15.28:0/675632298 learned_addr learned my addr 172.21.15.28:0/675632298 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:18.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.752+0000 7f0921e00640 1 -- 172.21.15.28:0/675632298 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f09240c76f0 msgr2=0x7f0924136220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:18.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.752+0000 7f0921e00640 1 --2- 172.21.15.28:0/675632298 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f09240c76f0 0x7f0924136220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:18.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.752+0000 7f0921e00640 1 -- 172.21.15.28:0/675632298 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f092413e0c0 msgr2=0x7f0924136760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:18.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.752+0000 7f0921e00640 1 --2- 172.21.15.28:0/675632298 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f092413e0c0 0x7f0924136760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:18.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.752+0000 7f0921e00640 1 -- 172.21.15.28:0/675632298 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0924153b60 con 0x7f092406dbc0 2024-02-21T23:10:18.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.753+0000 7f0921e00640 1 --2- 172.21.15.28:0/675632298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f092406dbc0 0x7f09241ee210 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7f091400d800 tx=0x7f091400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:18.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.753+0000 7f091a000640 1 -- 172.21.15.28:0/675632298 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f091400b6c0 con 0x7f092406dbc0 2024-02-21T23:10:18.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.753+0000 7f091a000640 1 -- 172.21.15.28:0/675632298 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0914019070 con 0x7f092406dbc0 2024-02-21T23:10:18.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.753+0000 7f091a000640 1 -- 172.21.15.28:0/675632298 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f091400ba80 con 0x7f092406dbc0 2024-02-21T23:10:18.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.753+0000 7f0923200640 1 -- 172.21.15.28:0/675632298 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0924136ca0 con 0x7f092406dbc0 2024-02-21T23:10:18.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.753+0000 7f0923200640 1 -- 172.21.15.28:0/675632298 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09241eeb50 con 0x7f092406dbc0 2024-02-21T23:10:18.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.755+0000 7f0923200640 1 -- 172.21.15.28:0/675632298 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f092413c360 con 0x7f092406dbc0 2024-02-21T23:10:18.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.755+0000 7f091a000640 1 -- 172.21.15.28:0/675632298 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f091400bc20 con 0x7f092406dbc0 2024-02-21T23:10:18.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.757+0000 7f091a000640 1 --2- 172.21.15.28:0/675632298 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f090406b500 0x7f090406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:18.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.757+0000 7f091a000640 1 -- 172.21.15.28:0/675632298 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f091408ab10 con 0x7f092406dbc0 2024-02-21T23:10:18.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.758+0000 7f0921400640 1 --2- 172.21.15.28:0/675632298 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f090406b500 0x7f090406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:18.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.759+0000 7f0921400640 1 --2- 172.21.15.28:0/675632298 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f090406b500 0x7f090406d9c0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f0910036000 tx=0x7f0910030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:18.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.761+0000 7f091a000640 1 -- 172.21.15.28:0/675632298 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f092413c360 con 0x7f092406dbc0 2024-02-21T23:10:18.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.968+0000 7f0923200640 1 -- 172.21.15.28:0/675632298 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mds_join_fs}] v 0) v1 -- 0x7f092404f290 con 0x7f092406dbc0 2024-02-21T23:10:18.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.974+0000 7f091a000640 1 -- 172.21.15.28:0/675632298 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_join_fs}]=0 v53)=0 v53) v1 ==== 117+0+0 (secure 0 0 0) 0x7f091401c360 con 0x7f092406dbc0 2024-02-21T23:10:18.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.978+0000 7f0923200640 1 -- 172.21.15.28:0/675632298 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f090406b500 msgr2=0x7f090406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:18.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.978+0000 7f0923200640 1 --2- 172.21.15.28:0/675632298 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f090406b500 0x7f090406d9c0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f0910036000 tx=0x7f0910030000 comp rx=0 tx=0).stop 2024-02-21T23:10:18.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.978+0000 7f0923200640 1 -- 172.21.15.28:0/675632298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f092406dbc0 msgr2=0x7f09241ee210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:18.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.978+0000 7f0923200640 1 --2- 172.21.15.28:0/675632298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f092406dbc0 0x7f09241ee210 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7f091400d800 tx=0x7f091400dcc0 comp rx=0 tx=0).stop 2024-02-21T23:10:18.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.978+0000 7f0923200640 1 -- 172.21.15.28:0/675632298 shutdown_connections 2024-02-21T23:10:18.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.978+0000 7f0923200640 1 --2- 172.21.15.28:0/675632298 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f090406b500 0x7f090406d9c0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:18.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.978+0000 7f0923200640 1 --2- 172.21.15.28:0/675632298 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f09240c76f0 0x7f0924136220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:18.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.978+0000 7f0923200640 1 --2- 172.21.15.28:0/675632298 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f092413e0c0 0x7f0924136760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:18.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.979+0000 7f0923200640 1 --2- 172.21.15.28:0/675632298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f092406dbc0 0x7f09241ee210 unknown :-1 s=CLOSED pgs=1763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:18.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.979+0000 7f0923200640 1 -- 172.21.15.28:0/675632298 >> 172.21.15.28:0/675632298 conn(0x7f0924073110 msgr2=0x7f09240c5020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:18.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.979+0000 7f0923200640 1 -- 172.21.15.28:0/675632298 shutdown_connections 2024-02-21T23:10:18.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:18.979+0000 7f0923200640 1 -- 172.21.15.28:0/675632298 wait complete. 2024-02-21T23:10:18.993 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail 11:0 2024-02-21T23:10:19.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.236+0000 7f9daa400640 1 Processor -- start 2024-02-21T23:10:19.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.237+0000 7f9daa400640 1 -- start start 2024-02-21T23:10:19.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.239+0000 7f9daa400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f9da4152600 0x7f9da40c0f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:19.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.239+0000 7f9daa400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9da40c6b60 0x7f9da4158930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:19.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.240+0000 7f9daa400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9da41529e0 con 0x7f9da40c6190 2024-02-21T23:10:19.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.240+0000 7f9daa400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9da40c7200 con 0x7f9da40c6b60 2024-02-21T23:10:19.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.240+0000 7f9daa400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9da40c74e0 con 0x7f9da4152600 2024-02-21T23:10:19.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.240+0000 7f9da9a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f9da40c6190 0x7f9da4158e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39320/0 (socket says 172.21.15.28:39320) 2024-02-21T23:10:19.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.240+0000 7f9da9a00640 1 -- 172.21.15.28:0/1233140440 learned_addr learned my addr 172.21.15.28:0/1233140440 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:10:19.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.240+0000 7f9da3e00640 1 --2- 172.21.15.28:0/1233140440 >> v2:172.21.15.28:3300/0 conn(0x7f9da40c6b60 0x7f9da4158930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:19.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.240+0000 7f9da9000640 1 --2- 172.21.15.28:0/1233140440 >> v2:172.21.15.81:3300/0 conn(0x7f9da4152600 0x7f9da40c0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:19.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.241+0000 7f9da3e00640 1 -- 172.21.15.28:0/1233140440 >> v2:172.21.15.81:3300/0 conn(0x7f9da4152600 msgr2=0x7f9da40c0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:19.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.240+0000 7f9da3400640 1 -- 172.21.15.28:0/1233140440 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2782451068 0 0) 0x7f9da41529e0 con 0x7f9da40c6190 2024-02-21T23:10:19.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.241+0000 7f9da3e00640 1 --2- 172.21.15.28:0/1233140440 >> v2:172.21.15.81:3300/0 conn(0x7f9da4152600 0x7f9da40c0f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:19.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.241+0000 7f9da3e00640 1 -- 172.21.15.28:0/1233140440 >> v1:172.21.15.28:6789/0 conn(0x7f9da40c6190 legacy=0x7f9da4158e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:19.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.241+0000 7f9da3e00640 1 -- 172.21.15.28:0/1233140440 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9da40c77c0 con 0x7f9da40c6b60 2024-02-21T23:10:19.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.242+0000 7f9da3e00640 1 --2- 172.21.15.28:0/1233140440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9da40c6b60 0x7f9da4158930 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7f9da4072f70 tx=0x7f9d9002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9653c84253b45077 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:19.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.242+0000 7f9da3400640 1 -- 172.21.15.28:0/1233140440 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d90035020 con 0x7f9da40c6b60 2024-02-21T23:10:19.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.242+0000 7f9da3400640 1 -- 172.21.15.28:0/1233140440 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d90031830 con 0x7f9da40c6b60 2024-02-21T23:10:19.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.242+0000 7f9da3400640 1 -- 172.21.15.28:0/1233140440 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d90031c00 con 0x7f9da40c6b60 2024-02-21T23:10:19.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.243+0000 7f9daa400640 1 -- 172.21.15.28:0/1233140440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9da40c6b60 msgr2=0x7f9da4158930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:19.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.243+0000 7f9daa400640 1 --2- 172.21.15.28:0/1233140440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9da40c6b60 0x7f9da4158930 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7f9da4072f70 tx=0x7f9d9002b8d0 comp rx=0 tx=0).stop 2024-02-21T23:10:19.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.243+0000 7f9daa400640 1 -- 172.21.15.28:0/1233140440 shutdown_connections 2024-02-21T23:10:19.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.243+0000 7f9daa400640 1 --2- 172.21.15.28:0/1233140440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9da40c6b60 0x7f9da4158930 unknown :-1 s=CLOSED pgs=1765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:19.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.243+0000 7f9daa400640 1 --2- 172.21.15.28:0/1233140440 >> v2:172.21.15.81:3300/0 conn(0x7f9da4152600 0x7f9da40c0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:19.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.243+0000 7f9daa400640 1 -- 172.21.15.28:0/1233140440 >> 172.21.15.28:0/1233140440 conn(0x7f9da406d1a0 msgr2=0x7f9da40725a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:19.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.243+0000 7f9daa400640 1 -- 172.21.15.28:0/1233140440 shutdown_connections 2024-02-21T23:10:19.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.243+0000 7f9daa400640 1 -- 172.21.15.28:0/1233140440 wait complete. 2024-02-21T23:10:19.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.245+0000 7f9daa400640 1 Processor -- start 2024-02-21T23:10:19.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.245+0000 7f9daa400640 1 -- start start 2024-02-21T23:10:19.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.246+0000 7f9daa400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9da40c6190 0x7f9da40bfa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:19.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.246+0000 7f9daa400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9da40c6b60 0x7f9da41575c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:19.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.246+0000 7f9daa400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9da4152600 0x7f9da4157b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:19.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.246+0000 7f9daa400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9da4155ae0 con 0x7f9da4152600 2024-02-21T23:10:19.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.246+0000 7f9daa400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9da40c7aa0 con 0x7f9da40c6b60 2024-02-21T23:10:19.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.246+0000 7f9daa400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9da4155c60 con 0x7f9da40c6190 2024-02-21T23:10:19.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.246+0000 7f9da9a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9da4152600 0x7f9da4157b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:19.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.246+0000 7f9da9a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9da4152600 0x7f9da4157b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49846/0 (socket says 172.21.15.28:49846) 2024-02-21T23:10:19.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.246+0000 7f9da9a00640 1 -- 172.21.15.28:0/1931555304 learned_addr learned my addr 172.21.15.28:0/1931555304 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:19.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.246+0000 7f9da3e00640 1 --2- 172.21.15.28:0/1931555304 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9da40c6b60 0x7f9da41575c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:19.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.246+0000 7f9da9000640 1 --2- 172.21.15.28:0/1931555304 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9da40c6190 0x7f9da40bfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:19.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.247+0000 7f9da9a00640 1 -- 172.21.15.28:0/1931555304 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9da40c6190 msgr2=0x7f9da40bfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:19.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.247+0000 7f9da9a00640 1 --2- 172.21.15.28:0/1931555304 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9da40c6190 0x7f9da40bfa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:19.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.247+0000 7f9da9a00640 1 -- 172.21.15.28:0/1931555304 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9da40c6b60 msgr2=0x7f9da41575c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:19.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.247+0000 7f9da9a00640 1 --2- 172.21.15.28:0/1931555304 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9da40c6b60 0x7f9da41575c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:19.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.247+0000 7f9da9a00640 1 -- 172.21.15.28:0/1931555304 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9da41edba0 con 0x7f9da4152600 2024-02-21T23:10:19.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.247+0000 7f9da9000640 1 --2- 172.21.15.28:0/1931555304 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9da40c6190 0x7f9da40bfa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:10:19.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.247+0000 7f9da9a00640 1 --2- 172.21.15.28:0/1931555304 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9da4152600 0x7f9da4157b00 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7f9d9c00c730 tx=0x7f9d9c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:19.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.248+0000 7f9da0c00640 1 -- 172.21.15.28:0/1931555304 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d9c018070 con 0x7f9da4152600 2024-02-21T23:10:19.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.248+0000 7f9da0c00640 1 -- 172.21.15.28:0/1931555304 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d9c00ce40 con 0x7f9da4152600 2024-02-21T23:10:19.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.248+0000 7f9daa400640 1 -- 172.21.15.28:0/1931555304 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9da41ee920 con 0x7f9da4152600 2024-02-21T23:10:19.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.248+0000 7f9da0c00640 1 -- 172.21.15.28:0/1931555304 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d9c00a860 con 0x7f9da4152600 2024-02-21T23:10:19.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.248+0000 7f9daa400640 1 -- 172.21.15.28:0/1931555304 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9da41ee120 con 0x7f9da4152600 2024-02-21T23:10:19.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.249+0000 7f9daa400640 1 -- 172.21.15.28:0/1931555304 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9da404f290 con 0x7f9da4152600 2024-02-21T23:10:19.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.250+0000 7f9da0c00640 1 -- 172.21.15.28:0/1931555304 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9d9c01b070 con 0x7f9da4152600 2024-02-21T23:10:19.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.252+0000 7f9da0c00640 1 --2- 172.21.15.28:0/1931555304 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9d8406b4d0 0x7f9d8406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:19.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.252+0000 7f9da0c00640 1 -- 172.21.15.28:0/1931555304 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 6957+0+0 (secure 0 0 0) 0x7f9d9c089c40 con 0x7f9da4152600 2024-02-21T23:10:19.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.252+0000 7f9da9000640 1 --2- 172.21.15.28:0/1931555304 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9d8406b4d0 0x7f9d8406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:19.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.253+0000 7f9da9000640 1 --2- 172.21.15.28:0/1931555304 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9d8406b4d0 0x7f9d8406d990 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f9d98009b90 tx=0x7f9d98008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:19.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.256+0000 7f9da0c00640 1 -- 172.21.15.28:0/1931555304 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f9d9c057f80 con 0x7f9da4152600 2024-02-21T23:10:19.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.549+0000 7f9daa400640 1 -- 172.21.15.28:0/1931555304 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "11:0"} v 0) v1 -- 0x7f9da40c6630 con 0x7f9da4152600 2024-02-21T23:10:19.538 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-02-21T23:10:19.550+0000 7f5ddc400640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-21T23:10:19.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.553+0000 7f9da0c00640 1 -- 172.21.15.28:0/1931555304 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "11:0"}]=0 failed mds gid 8826 v164) v1 ==== 98+0+0 (secure 0 0 0) 0x7f9d9c05c020 con 0x7f9da4152600 2024-02-21T23:10:19.541 INFO:teuthology.orchestra.run.smithi028.stderr:failed mds gid 8826 2024-02-21T23:10:19.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.556+0000 7f9daa400640 1 -- 172.21.15.28:0/1931555304 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9d8406b4d0 msgr2=0x7f9d8406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:19.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.556+0000 7f9daa400640 1 --2- 172.21.15.28:0/1931555304 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9d8406b4d0 0x7f9d8406d990 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f9d98009b90 tx=0x7f9d98008040 comp rx=0 tx=0).stop 2024-02-21T23:10:19.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.556+0000 7f9daa400640 1 -- 172.21.15.28:0/1931555304 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9da4152600 msgr2=0x7f9da4157b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:19.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.556+0000 7f9daa400640 1 --2- 172.21.15.28:0/1931555304 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9da4152600 0x7f9da4157b00 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7f9d9c00c730 tx=0x7f9d9c00cbf0 comp rx=0 tx=0).stop 2024-02-21T23:10:19.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.557+0000 7f9daa400640 1 -- 172.21.15.28:0/1931555304 shutdown_connections 2024-02-21T23:10:19.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.557+0000 7f9daa400640 1 --2- 172.21.15.28:0/1931555304 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9d8406b4d0 0x7f9d8406d990 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:19.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.557+0000 7f9daa400640 1 --2- 172.21.15.28:0/1931555304 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9da4152600 0x7f9da4157b00 unknown :-1 s=CLOSED pgs=1766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:19.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.557+0000 7f9daa400640 1 --2- 172.21.15.28:0/1931555304 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9da40c6b60 0x7f9da41575c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:19.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.557+0000 7f9daa400640 1 --2- 172.21.15.28:0/1931555304 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9da40c6190 0x7f9da40bfa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:19.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.557+0000 7f9daa400640 1 -- 172.21.15.28:0/1931555304 >> 172.21.15.28:0/1931555304 conn(0x7f9da406d1a0 msgr2=0x7f9da40c2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:19.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.558+0000 7f9daa400640 1 -- 172.21.15.28:0/1931555304 shutdown_connections 2024-02-21T23:10:19.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.558+0000 7f9daa400640 1 -- 172.21.15.28:0/1931555304 wait complete. 2024-02-21T23:10:19.561 INFO:tasks.ceph.mds.a.smithi028.stderr: -666> 2024-02-21T23:10:00.211+0000 7f3824800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-21T23:10:19.568 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:10:19.632 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-02-21T23:10:19.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.771+0000 7f6946400640 1 Processor -- start 2024-02-21T23:10:19.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.771+0000 7f6946400640 1 -- start start 2024-02-21T23:10:19.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.773+0000 7f6946400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f69400c66c0 0x7f69400c3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:19.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.773+0000 7f6946400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f69400a8090 0x7f69400c4360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:19.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.773+0000 7f6946400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6940152f50 con 0x7f6940070d20 2024-02-21T23:10:19.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.773+0000 7f6946400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69400c3340 con 0x7f69400c66c0 2024-02-21T23:10:19.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.773+0000 7f6946400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69400c3620 con 0x7f69400a8090 2024-02-21T23:10:19.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.773+0000 7f6937e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f69400c66c0 0x7f69400c3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:19.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.773+0000 7f6937e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f69400c66c0 0x7f69400c3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49882/0 (socket says 172.21.15.28:49882) 2024-02-21T23:10:19.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.773+0000 7f6937e00640 1 -- 172.21.15.28:0/164829070 learned_addr learned my addr 172.21.15.28:0/164829070 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:19.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.773+0000 7f6937e00640 1 -- 172.21.15.28:0/164829070 >> v2:172.21.15.81:3300/0 conn(0x7f69400a8090 msgr2=0x7f69400c4360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:19.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.773+0000 7f6937e00640 1 --2- 172.21.15.28:0/164829070 >> v2:172.21.15.81:3300/0 conn(0x7f69400a8090 0x7f69400c4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:19.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.773+0000 7f6937e00640 1 -- 172.21.15.28:0/164829070 >> v1:172.21.15.81:6789/0 conn(0x7f6940070d20 legacy=0x7f6940150230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:19.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.773+0000 7f6937e00640 1 -- 172.21.15.28:0/164829070 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69400c3900 con 0x7f69400c66c0 2024-02-21T23:10:19.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.775+0000 7f6937e00640 1 --2- 172.21.15.28:0/164829070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69400c66c0 0x7f69400c3e20 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7f69400c2b70 tx=0x7f692802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=299b737c130883d2 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:19.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.775+0000 7f693ea00640 1 -- 172.21.15.28:0/164829070 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f692802e070 con 0x7f69400c66c0 2024-02-21T23:10:19.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.775+0000 7f693ea00640 1 -- 172.21.15.28:0/164829070 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6928037070 con 0x7f69400c66c0 2024-02-21T23:10:19.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.775+0000 7f693ea00640 1 -- 172.21.15.28:0/164829070 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6928031a00 con 0x7f69400c66c0 2024-02-21T23:10:19.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.776+0000 7f6946400640 1 -- 172.21.15.28:0/164829070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69400c66c0 msgr2=0x7f69400c3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:19.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.776+0000 7f6946400640 1 --2- 172.21.15.28:0/164829070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69400c66c0 0x7f69400c3e20 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7f69400c2b70 tx=0x7f692802b880 comp rx=0 tx=0).stop 2024-02-21T23:10:19.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.776+0000 7f6946400640 1 -- 172.21.15.28:0/164829070 shutdown_connections 2024-02-21T23:10:19.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.776+0000 7f6946400640 1 --2- 172.21.15.28:0/164829070 >> v2:172.21.15.81:3300/0 conn(0x7f69400a8090 0x7f69400c4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:19.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.776+0000 7f6946400640 1 --2- 172.21.15.28:0/164829070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69400c66c0 0x7f69400c3e20 unknown :-1 s=CLOSED pgs=1770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:19.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.776+0000 7f6946400640 1 -- 172.21.15.28:0/164829070 >> 172.21.15.28:0/164829070 conn(0x7f69400aa550 msgr2=0x7f69400aa930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:19.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.776+0000 7f6946400640 1 -- 172.21.15.28:0/164829070 shutdown_connections 2024-02-21T23:10:19.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.776+0000 7f6946400640 1 -- 172.21.15.28:0/164829070 wait complete. 2024-02-21T23:10:19.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.778+0000 7f6946400640 1 Processor -- start 2024-02-21T23:10:19.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.778+0000 7f6946400640 1 -- start start 2024-02-21T23:10:19.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.779+0000 7f6946400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6940070d20 0x7f6940156bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:19.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.779+0000 7f6946400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69400a8090 0x7f69400ac260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:19.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.779+0000 7f6946400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f69400c66c0 0x7f69401ebd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:19.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.779+0000 7f6946400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6940071280 con 0x7f69400a8090 2024-02-21T23:10:19.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.779+0000 7f6946400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6940071100 con 0x7f69400c66c0 2024-02-21T23:10:19.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.779+0000 7f6946400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69400c48a0 con 0x7f6940070d20 2024-02-21T23:10:19.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.779+0000 7f6937e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69400a8090 0x7f69400ac260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:19.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.780+0000 7f6937e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69400a8090 0x7f69400ac260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49898/0 (socket says 172.21.15.28:49898) 2024-02-21T23:10:19.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.780+0000 7f6937e00640 1 -- 172.21.15.28:0/3447314433 learned_addr learned my addr 172.21.15.28:0/3447314433 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:19.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.780+0000 7f693f400640 1 --2- 172.21.15.28:0/3447314433 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6940070d20 0x7f6940156bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:19.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.780+0000 7f6937e00640 1 -- 172.21.15.28:0/3447314433 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6940070d20 msgr2=0x7f6940156bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:19.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.780+0000 7f6937e00640 1 --2- 172.21.15.28:0/3447314433 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6940070d20 0x7f6940156bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:19.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.780+0000 7f6937e00640 1 -- 172.21.15.28:0/3447314433 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f69400c66c0 msgr2=0x7f69401ebd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:19.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.780+0000 7f693fe00640 1 --2- 172.21.15.28:0/3447314433 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f69400c66c0 0x7f69401ebd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:19.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.780+0000 7f6937e00640 1 --2- 172.21.15.28:0/3447314433 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f69400c66c0 0x7f69401ebd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:19.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.780+0000 7f6937e00640 1 -- 172.21.15.28:0/3447314433 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f694014f600 con 0x7f69400a8090 2024-02-21T23:10:19.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.780+0000 7f693f400640 1 --2- 172.21.15.28:0/3447314433 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6940070d20 0x7f6940156bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:10:19.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.780+0000 7f693fe00640 1 --2- 172.21.15.28:0/3447314433 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f69400c66c0 0x7f69401ebd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:10:19.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.780+0000 7f6937e00640 1 --2- 172.21.15.28:0/3447314433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69400a8090 0x7f69400ac260 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7f6928000c00 tx=0x7f69280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:19.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.781+0000 7f6937400640 1 -- 172.21.15.28:0/3447314433 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6928032880 con 0x7f69400a8090 2024-02-21T23:10:19.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.781+0000 7f6937400640 1 -- 172.21.15.28:0/3447314433 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6928032a20 con 0x7f69400a8090 2024-02-21T23:10:19.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.781+0000 7f6937400640 1 -- 172.21.15.28:0/3447314433 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6928032e10 con 0x7f69400a8090 2024-02-21T23:10:19.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.781+0000 7f6946400640 1 -- 172.21.15.28:0/3447314433 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69401ec910 con 0x7f69400a8090 2024-02-21T23:10:19.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.781+0000 7f6946400640 1 -- 172.21.15.28:0/3447314433 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f694014fba0 con 0x7f69400a8090 2024-02-21T23:10:19.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.782+0000 7f6946400640 1 -- 172.21.15.28:0/3447314433 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6940068640 con 0x7f69400a8090 2024-02-21T23:10:19.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.783+0000 7f6937400640 1 -- 172.21.15.28:0/3447314433 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f69280453f0 con 0x7f69400a8090 2024-02-21T23:10:19.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.784+0000 7f6937400640 1 --2- 172.21.15.28:0/3447314433 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f691c06b4d0 0x7f691c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:19.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.785+0000 7f6937400640 1 -- 172.21.15.28:0/3447314433 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 7043+0+0 (secure 0 0 0) 0x7f69280ac9c0 con 0x7f69400a8090 2024-02-21T23:10:19.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.785+0000 7f693f400640 1 --2- 172.21.15.28:0/3447314433 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f691c06b4d0 0x7f691c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:19.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.786+0000 7f693f400640 1 --2- 172.21.15.28:0/3447314433 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f691c06b4d0 0x7f691c06d990 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f6930003eb0 tx=0x7f6930009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:19.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:19.789+0000 7f6937400640 1 -- 172.21.15.28:0/3447314433 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f6928040070 con 0x7f69400a8090 2024-02-21T23:10:20.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:20.031+0000 7f6946400640 1 -- 172.21.15.28:0/3447314433 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f69400c2b70 con 0x7f69400a8090 2024-02-21T23:10:20.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:20.032+0000 7f6937400640 1 -- 172.21.15.28:0/3447314433 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 165 v165) v1 ==== 95+0+5401 (secure 0 0 0) 0x7f692807aca0 con 0x7f69400a8090 2024-02-21T23:10:20.020 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:10:20.020 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":165,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6389,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1320495906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1320495906},{"type":"v1","addr":"172.21.15.81:6835","nonce":1320495906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":153}],"filesystems":[{"mdsmap":{"epoch":165,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:09:53.730296+0000","modified":"2024-02-21T23:10:19.558708+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":156,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6392},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6392":{"gid":6392,"name":"d","rank":0,"incarnation":165,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6837/521625945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":521625945},{"type":"v1","addr":"172.21.15.81:6837","nonce":521625945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11},{"mdsmap":{"epoch":163,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:10:05.843850+0000","modified":"2024-02-21T23:10:08.927829+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8817},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8817":{"gid":8817,"name":"c","rank":0,"incarnation":159,"state":"up:active","state_seq":6,"addr":"172.21.15.28:6833/2975378719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2975378719},{"type":"v1","addr":"172.21.15.28:6833","nonce":2975378719}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2024-02-21T23:10:20.020 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 165 2024-02-21T23:10:20.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:20.034+0000 7f6946400640 1 -- 172.21.15.28:0/3447314433 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f691c06b4d0 msgr2=0x7f691c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:20.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:20.034+0000 7f6946400640 1 --2- 172.21.15.28:0/3447314433 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f691c06b4d0 0x7f691c06d990 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f6930003eb0 tx=0x7f6930009040 comp rx=0 tx=0).stop 2024-02-21T23:10:20.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:20.034+0000 7f6946400640 1 -- 172.21.15.28:0/3447314433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69400a8090 msgr2=0x7f69400ac260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:20.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:20.034+0000 7f6946400640 1 --2- 172.21.15.28:0/3447314433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69400a8090 0x7f69400ac260 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7f6928000c00 tx=0x7f69280046c0 comp rx=0 tx=0).stop 2024-02-21T23:10:20.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:20.034+0000 7f6946400640 1 -- 172.21.15.28:0/3447314433 shutdown_connections 2024-02-21T23:10:20.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:20.034+0000 7f6946400640 1 --2- 172.21.15.28:0/3447314433 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f691c06b4d0 0x7f691c06d990 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:20.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:20.034+0000 7f6946400640 1 --2- 172.21.15.28:0/3447314433 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f69400c66c0 0x7f69401ebd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:20.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:20.034+0000 7f6946400640 1 --2- 172.21.15.28:0/3447314433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69400a8090 0x7f69400ac260 unknown :-1 s=CLOSED pgs=1771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:20.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:20.034+0000 7f6946400640 1 --2- 172.21.15.28:0/3447314433 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6940070d20 0x7f6940156bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:20.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:20.034+0000 7f6946400640 1 -- 172.21.15.28:0/3447314433 >> 172.21.15.28:0/3447314433 conn(0x7f69400aa550 msgr2=0x7f694006fa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:20.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:20.035+0000 7f6946400640 1 -- 172.21.15.28:0/3447314433 shutdown_connections 2024-02-21T23:10:20.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:20.035+0000 7f6946400640 1 -- 172.21.15.28:0/3447314433 wait complete. 2024-02-21T23:10:20.445 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 165, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:09:53.730296+0000', 'modified': '2024-02-21T23:10:19.558708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 156, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6392}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6392': {'gid': 6392, 'name': 'd', 'rank': 0, 'incarnation': 165, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.81:6837/521625945', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 521625945}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 521625945}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [24], 'metadata_pool': 23, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:10:20.446 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6392:up:replay 2024-02-21T23:10:21.446 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:10:21.585 INFO:tasks.ceph.mds.c.smithi028.stderr: -419> 2024-02-21T23:10:09.210+0000 7f81e2a00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-21T23:10:21.657 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-02-21T23:10:21.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.715+0000 7fb3a1000640 1 Processor -- start 2024-02-21T23:10:21.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.715+0000 7fb3a1000640 1 -- start start 2024-02-21T23:10:21.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.716+0000 7fb3a1000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb39c152500 0x7fb39c1558c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:21.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.716+0000 7fb3a1000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb39c0ca050 0x7fb39c0bf6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:21.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.716+0000 7fb3a1000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb39c1528e0 con 0x7fb39c0bfc00 2024-02-21T23:10:21.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.716+0000 7fb3a1000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb39c0c86e0 con 0x7fb39c152500 2024-02-21T23:10:21.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.716+0000 7fb3a1000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb39c0c8980 con 0x7fb39c0ca050 2024-02-21T23:10:21.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.716+0000 7fb39b400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb39c152500 0x7fb39c1558c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:21.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.716+0000 7fb39b400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb39c152500 0x7fb39c1558c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49936/0 (socket says 172.21.15.28:49936) 2024-02-21T23:10:21.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.716+0000 7fb39b400640 1 -- 172.21.15.28:0/2290182534 learned_addr learned my addr 172.21.15.28:0/2290182534 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:21.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.717+0000 7fb39aa00640 1 --2- 172.21.15.28:0/2290182534 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb39c0ca050 0x7fb39c0bf6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:21.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.717+0000 7fb39b400640 1 -- 172.21.15.28:0/2290182534 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb39c0ca050 msgr2=0x7fb39c0bf6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:21.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.717+0000 7fb39b400640 1 --2- 172.21.15.28:0/2290182534 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb39c0ca050 0x7fb39c0bf6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:21.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.717+0000 7fb39b400640 1 -- 172.21.15.28:0/2290182534 >> v1:172.21.15.81:6789/0 conn(0x7fb39c0bfc00 legacy=0x7fb39c06bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:21.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.717+0000 7fb39b400640 1 -- 172.21.15.28:0/2290182534 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb39c0c8c60 con 0x7fb39c152500 2024-02-21T23:10:21.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.717+0000 7fb39aa00640 1 --2- 172.21.15.28:0/2290182534 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb39c0ca050 0x7fb39c0bf6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:10:21.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.718+0000 7fb39b400640 1 --2- 172.21.15.28:0/2290182534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb39c152500 0x7fb39c1558c0 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7fb38c009870 tx=0x7fb38c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f3b9f08699fb06c server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:21.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.718+0000 7fb39a000640 1 -- 172.21.15.28:0/2290182534 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb38c0317c0 con 0x7fb39c152500 2024-02-21T23:10:21.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.718+0000 7fb39a000640 1 -- 172.21.15.28:0/2290182534 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb38c037070 con 0x7fb39c152500 2024-02-21T23:10:21.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.718+0000 7fb39a000640 1 -- 172.21.15.28:0/2290182534 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb38c031b80 con 0x7fb39c152500 2024-02-21T23:10:21.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.719+0000 7fb3a1000640 1 -- 172.21.15.28:0/2290182534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb39c152500 msgr2=0x7fb39c1558c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:21.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.719+0000 7fb3a1000640 1 --2- 172.21.15.28:0/2290182534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb39c152500 0x7fb39c1558c0 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7fb38c009870 tx=0x7fb38c02b860 comp rx=0 tx=0).stop 2024-02-21T23:10:21.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.719+0000 7fb3a1000640 1 -- 172.21.15.28:0/2290182534 shutdown_connections 2024-02-21T23:10:21.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.719+0000 7fb3a1000640 1 --2- 172.21.15.28:0/2290182534 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb39c0ca050 0x7fb39c0bf6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:21.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.719+0000 7fb3a1000640 1 --2- 172.21.15.28:0/2290182534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb39c152500 0x7fb39c1558c0 unknown :-1 s=CLOSED pgs=1774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:21.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.719+0000 7fb3a1000640 1 -- 172.21.15.28:0/2290182534 >> 172.21.15.28:0/2290182534 conn(0x7fb39c0ac450 msgr2=0x7fb39c0ac860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:21.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.719+0000 7fb3a1000640 1 -- 172.21.15.28:0/2290182534 shutdown_connections 2024-02-21T23:10:21.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.719+0000 7fb3a1000640 1 -- 172.21.15.28:0/2290182534 wait complete. 2024-02-21T23:10:21.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.721+0000 7fb3a1000640 1 Processor -- start 2024-02-21T23:10:21.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.721+0000 7fb3a1000640 1 -- start start 2024-02-21T23:10:21.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.722+0000 7fb3a1000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb39c0bfc00 0x7fb39c156970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:21.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.722+0000 7fb3a1000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb39c0ca050 0x7fb39c147620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:21.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.722+0000 7fb3a1000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb39c152500 0x7fb39c147b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:21.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.722+0000 7fb3a1000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb39c06c300 con 0x7fb39c0bfc00 2024-02-21T23:10:21.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.722+0000 7fb3a1000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb39c0c9220 con 0x7fb39c0ca050 2024-02-21T23:10:21.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.722+0000 7fb3a1000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb39c06c480 con 0x7fb39c152500 2024-02-21T23:10:21.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.722+0000 7fb39b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb39c0bfc00 0x7fb39c156970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:21.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.722+0000 7fb39b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb39c0bfc00 0x7fb39c156970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49952/0 (socket says 172.21.15.28:49952) 2024-02-21T23:10:21.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.722+0000 7fb39b400640 1 -- 172.21.15.28:0/1954281274 learned_addr learned my addr 172.21.15.28:0/1954281274 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:21.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.723+0000 7fb39aa00640 1 --2- 172.21.15.28:0/1954281274 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb39c0ca050 0x7fb39c147620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:21.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.723+0000 7fb39b400640 1 -- 172.21.15.28:0/1954281274 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb39c152500 msgr2=0x7fb39c147b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:21.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.723+0000 7fb39be00640 1 --2- 172.21.15.28:0/1954281274 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb39c152500 0x7fb39c147b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:21.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.723+0000 7fb39b400640 1 --2- 172.21.15.28:0/1954281274 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb39c152500 0x7fb39c147b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:21.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.723+0000 7fb39b400640 1 -- 172.21.15.28:0/1954281274 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb39c0ca050 msgr2=0x7fb39c147620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:21.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.723+0000 7fb39b400640 1 --2- 172.21.15.28:0/1954281274 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb39c0ca050 0x7fb39c147620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:21.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.723+0000 7fb39b400640 1 -- 172.21.15.28:0/1954281274 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb39c0aebf0 con 0x7fb39c0bfc00 2024-02-21T23:10:21.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.723+0000 7fb39aa00640 1 --2- 172.21.15.28:0/1954281274 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb39c0ca050 0x7fb39c147620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:10:21.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.723+0000 7fb39b400640 1 --2- 172.21.15.28:0/1954281274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb39c0bfc00 0x7fb39c156970 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7fb38c000c00 tx=0x7fb38c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:21.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.723+0000 7fb393400640 1 -- 172.21.15.28:0/1954281274 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb38c032860 con 0x7fb39c0bfc00 2024-02-21T23:10:21.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.723+0000 7fb393400640 1 -- 172.21.15.28:0/1954281274 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb38c032a20 con 0x7fb39c0bfc00 2024-02-21T23:10:21.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.723+0000 7fb393400640 1 -- 172.21.15.28:0/1954281274 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb38c032e10 con 0x7fb39c0bfc00 2024-02-21T23:10:21.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.723+0000 7fb3a1000640 1 -- 172.21.15.28:0/1954281274 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb39c14c2e0 con 0x7fb39c0bfc00 2024-02-21T23:10:21.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.724+0000 7fb3a1000640 1 -- 172.21.15.28:0/1954281274 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb39c14a7f0 con 0x7fb39c0bfc00 2024-02-21T23:10:21.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.725+0000 7fb3a1000640 1 -- 172.21.15.28:0/1954281274 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb39c04f290 con 0x7fb39c0bfc00 2024-02-21T23:10:21.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.726+0000 7fb393400640 1 -- 172.21.15.28:0/1954281274 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb38c04c060 con 0x7fb39c0bfc00 2024-02-21T23:10:21.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.726+0000 7fb393400640 1 --2- 172.21.15.28:0/1954281274 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb37c06b500 0x7fb37c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:21.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.727+0000 7fb393400640 1 -- 172.21.15.28:0/1954281274 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 7129+0+0 (secure 0 0 0) 0x7fb38c0ac3a0 con 0x7fb39c0bfc00 2024-02-21T23:10:21.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.727+0000 7fb39aa00640 1 --2- 172.21.15.28:0/1954281274 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb37c06b500 0x7fb37c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:21.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.728+0000 7fb39aa00640 1 --2- 172.21.15.28:0/1954281274 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb37c06b500 0x7fb37c06d9c0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fb388006fd0 tx=0x7fb388008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:21.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.731+0000 7fb393400640 1 -- 172.21.15.28:0/1954281274 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb39c0aebf0 con 0x7fb39c0bfc00 2024-02-21T23:10:21.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.972+0000 7fb3a1000640 1 -- 172.21.15.28:0/1954281274 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb39c0c5160 con 0x7fb39c0bfc00 2024-02-21T23:10:21.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.973+0000 7fb393400640 1 -- 172.21.15.28:0/1954281274 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 168 v168) v1 ==== 95+0+5404 (secure 0 0 0) 0x7fb38c040070 con 0x7fb39c0bfc00 2024-02-21T23:10:21.961 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:10:21.961 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":168,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9141,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2895215997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2895215997},{"type":"v1","addr":"172.21.15.28:6835","nonce":2895215997}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":166}],"filesystems":[{"mdsmap":{"epoch":167,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:09:53.730296+0000","modified":"2024-02-21T23:10:20.573740+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":156,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6392},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6392":{"gid":6392,"name":"d","rank":0,"incarnation":165,"state":"up:rejoin","state_seq":10,"addr":"172.21.15.81:6837/521625945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":521625945},{"type":"v1","addr":"172.21.15.81:6837","nonce":521625945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11},{"mdsmap":{"epoch":168,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:10:05.843850+0000","modified":"2024-02-21T23:10:21.581969+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":157,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6389},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6389":{"gid":6389,"name":"b","rank":0,"incarnation":168,"state":"up:replay","state_seq":9,"addr":"172.21.15.81:6835/1320495906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1320495906},{"type":"v1","addr":"172.21.15.81:6835","nonce":1320495906}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2024-02-21T23:10:21.961 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 168 2024-02-21T23:10:21.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.975+0000 7fb3a1000640 1 -- 172.21.15.28:0/1954281274 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb37c06b500 msgr2=0x7fb37c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:21.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.975+0000 7fb3a1000640 1 --2- 172.21.15.28:0/1954281274 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb37c06b500 0x7fb37c06d9c0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fb388006fd0 tx=0x7fb388008040 comp rx=0 tx=0).stop 2024-02-21T23:10:21.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.975+0000 7fb3a1000640 1 -- 172.21.15.28:0/1954281274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb39c0bfc00 msgr2=0x7fb39c156970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:21.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.975+0000 7fb3a1000640 1 --2- 172.21.15.28:0/1954281274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb39c0bfc00 0x7fb39c156970 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7fb38c000c00 tx=0x7fb38c0046c0 comp rx=0 tx=0).stop 2024-02-21T23:10:21.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.976+0000 7fb3a1000640 1 -- 172.21.15.28:0/1954281274 shutdown_connections 2024-02-21T23:10:21.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.976+0000 7fb3a1000640 1 --2- 172.21.15.28:0/1954281274 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb37c06b500 0x7fb37c06d9c0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:21.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.976+0000 7fb3a1000640 1 --2- 172.21.15.28:0/1954281274 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb39c152500 0x7fb39c147b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:21.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.976+0000 7fb3a1000640 1 --2- 172.21.15.28:0/1954281274 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb39c0ca050 0x7fb39c147620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:21.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.976+0000 7fb3a1000640 1 --2- 172.21.15.28:0/1954281274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb39c0bfc00 0x7fb39c156970 unknown :-1 s=CLOSED pgs=1775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:21.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.976+0000 7fb3a1000640 1 -- 172.21.15.28:0/1954281274 >> 172.21.15.28:0/1954281274 conn(0x7fb39c0ac450 msgr2=0x7fb39c0caf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:21.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.976+0000 7fb3a1000640 1 -- 172.21.15.28:0/1954281274 shutdown_connections 2024-02-21T23:10:21.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:21.976+0000 7fb3a1000640 1 -- 172.21.15.28:0/1954281274 wait complete. 2024-02-21T23:10:22.388 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 167, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:09:53.730296+0000', 'modified': '2024-02-21T23:10:20.573740+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 156, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6392}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6392': {'gid': 6392, 'name': 'd', 'rank': 0, 'incarnation': 165, 'state': 'up:rejoin', 'state_seq': 10, 'addr': '172.21.15.81:6837/521625945', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 521625945}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 521625945}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [24], 'metadata_pool': 23, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:10:22.388 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6392:up:rejoin 2024-02-21T23:10:23.389 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:10:23.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.634+0000 7ff689c00640 1 Processor -- start 2024-02-21T23:10:23.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.634+0000 7ff689c00640 1 -- start start 2024-02-21T23:10:23.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.635+0000 7ff689c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff6840c3280 0x7ff6840c3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:23.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.635+0000 7ff689c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff684155860 0x7ff684155d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:23.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.635+0000 7ff689c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6840c6560 con 0x7ff684072430 2024-02-21T23:10:23.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.635+0000 7ff689c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6840c5cc0 con 0x7ff684155860 2024-02-21T23:10:23.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.635+0000 7ff689c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6840c5fa0 con 0x7ff6840c3280 2024-02-21T23:10:23.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.636+0000 7ff682000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff684072430 0x7ff6841526e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39340/0 (socket says 172.21.15.28:39340) 2024-02-21T23:10:23.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.636+0000 7ff682000640 1 -- 172.21.15.28:0/1888060970 learned_addr learned my addr 172.21.15.28:0/1888060970 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:10:23.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.636+0000 7ff683400640 1 --2- 172.21.15.28:0/1888060970 >> v2:172.21.15.28:3300/0 conn(0x7ff684155860 0x7ff684155d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:23.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.636+0000 7ff682a00640 1 --2- 172.21.15.28:0/1888060970 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff6840c3280 0x7ff6840c3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:23.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.636+0000 7ff681600640 1 -- 172.21.15.28:0/1888060970 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1932213010 0 0) 0x7ff6840c6560 con 0x7ff684072430 2024-02-21T23:10:23.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.636+0000 7ff681600640 1 -- 172.21.15.28:0/1888060970 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff668003520 con 0x7ff684072430 2024-02-21T23:10:23.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.636+0000 7ff683400640 1 -- 172.21.15.28:0/1888060970 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff6840c3280 msgr2=0x7ff6840c3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:23.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.636+0000 7ff683400640 1 --2- 172.21.15.28:0/1888060970 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff6840c3280 0x7ff6840c3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:23.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.636+0000 7ff683400640 1 -- 172.21.15.28:0/1888060970 >> v1:172.21.15.28:6789/0 conn(0x7ff684072430 legacy=0x7ff6841526e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:23.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.636+0000 7ff683400640 1 -- 172.21.15.28:0/1888060970 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6840c6280 con 0x7ff684155860 2024-02-21T23:10:23.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.638+0000 7ff683400640 1 --2- 172.21.15.28:0/1888060970 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff684155860 0x7ff684155d10 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7ff67c009d90 tx=0x7ff67c02be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c4a0f37133331cf server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:23.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.638+0000 7ff681600640 1 -- 172.21.15.28:0/1888060970 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff67c039070 con 0x7ff684155860 2024-02-21T23:10:23.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.638+0000 7ff681600640 1 -- 172.21.15.28:0/1888060970 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff67c02cbc0 con 0x7ff684155860 2024-02-21T23:10:23.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.638+0000 7ff681600640 1 -- 172.21.15.28:0/1888060970 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff67c034440 con 0x7ff684155860 2024-02-21T23:10:23.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.639+0000 7ff689c00640 1 -- 172.21.15.28:0/1888060970 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff684155860 msgr2=0x7ff684155d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:23.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.639+0000 7ff689c00640 1 --2- 172.21.15.28:0/1888060970 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff684155860 0x7ff684155d10 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7ff67c009d90 tx=0x7ff67c02be10 comp rx=0 tx=0).stop 2024-02-21T23:10:23.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.639+0000 7ff689c00640 1 -- 172.21.15.28:0/1888060970 shutdown_connections 2024-02-21T23:10:23.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.639+0000 7ff689c00640 1 --2- 172.21.15.28:0/1888060970 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff684155860 0x7ff684155d10 unknown :-1 s=CLOSED pgs=1777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:23.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.639+0000 7ff689c00640 1 --2- 172.21.15.28:0/1888060970 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff6840c3280 0x7ff6840c3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:23.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.639+0000 7ff689c00640 1 -- 172.21.15.28:0/1888060970 >> 172.21.15.28:0/1888060970 conn(0x7ff6840c67f0 msgr2=0x7ff6840c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:23.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.639+0000 7ff689c00640 1 -- 172.21.15.28:0/1888060970 shutdown_connections 2024-02-21T23:10:23.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.639+0000 7ff689c00640 1 -- 172.21.15.28:0/1888060970 wait complete. 2024-02-21T23:10:23.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.641+0000 7ff689c00640 1 Processor -- start 2024-02-21T23:10:23.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.641+0000 7ff689c00640 1 -- start start 2024-02-21T23:10:23.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.642+0000 7ff689c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff684072430 0x7ff6840ac370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:23.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.642+0000 7ff689c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6840c3280 0x7ff68414e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:23.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.642+0000 7ff689c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff684155860 0x7ff68414e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:23.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.642+0000 7ff689c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6840c5aa0 con 0x7ff6840c3280 2024-02-21T23:10:23.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.642+0000 7ff689c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6840c5920 con 0x7ff684072430 2024-02-21T23:10:23.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.642+0000 7ff689c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6840733e0 con 0x7ff684155860 2024-02-21T23:10:23.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.642+0000 7ff682000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6840c3280 0x7ff68414e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:23.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.642+0000 7ff682000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6840c3280 0x7ff68414e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49972/0 (socket says 172.21.15.28:49972) 2024-02-21T23:10:23.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.642+0000 7ff682000640 1 -- 172.21.15.28:0/3985809301 learned_addr learned my addr 172.21.15.28:0/3985809301 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:23.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.642+0000 7ff682a00640 1 --2- 172.21.15.28:0/3985809301 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff684072430 0x7ff6840ac370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:23.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.642+0000 7ff683400640 1 --2- 172.21.15.28:0/3985809301 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff684155860 0x7ff68414e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:23.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.642+0000 7ff682000640 1 -- 172.21.15.28:0/3985809301 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff684155860 msgr2=0x7ff68414e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:23.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.643+0000 7ff682000640 1 --2- 172.21.15.28:0/3985809301 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff684155860 0x7ff68414e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:23.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.643+0000 7ff682000640 1 -- 172.21.15.28:0/3985809301 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff684072430 msgr2=0x7ff6840ac370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:23.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.643+0000 7ff682000640 1 --2- 172.21.15.28:0/3985809301 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff684072430 0x7ff6840ac370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:23.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.643+0000 7ff682000640 1 -- 172.21.15.28:0/3985809301 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6841db440 con 0x7ff6840c3280 2024-02-21T23:10:23.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.643+0000 7ff683400640 1 --2- 172.21.15.28:0/3985809301 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff684155860 0x7ff68414e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:10:23.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.643+0000 7ff682000640 1 --2- 172.21.15.28:0/3985809301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6840c3280 0x7ff68414e060 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7ff67000c390 tx=0x7ff67000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:23.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.643+0000 7ff67aa00640 1 -- 172.21.15.28:0/3985809301 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff670017070 con 0x7ff6840c3280 2024-02-21T23:10:23.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.643+0000 7ff67aa00640 1 -- 172.21.15.28:0/3985809301 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff670005d00 con 0x7ff6840c3280 2024-02-21T23:10:23.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.643+0000 7ff689c00640 1 -- 172.21.15.28:0/3985809301 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6841db700 con 0x7ff6840c3280 2024-02-21T23:10:23.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.643+0000 7ff67aa00640 1 -- 172.21.15.28:0/3985809301 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff670007310 con 0x7ff6840c3280 2024-02-21T23:10:23.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.643+0000 7ff689c00640 1 -- 172.21.15.28:0/3985809301 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6841db9e0 con 0x7ff6840c3280 2024-02-21T23:10:23.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.645+0000 7ff689c00640 1 -- 172.21.15.28:0/3985809301 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff684067f30 con 0x7ff6840c3280 2024-02-21T23:10:23.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.646+0000 7ff67aa00640 1 -- 172.21.15.28:0/3985809301 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff67001a070 con 0x7ff6840c3280 2024-02-21T23:10:23.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.647+0000 7ff67aa00640 1 --2- 172.21.15.28:0/3985809301 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff66406b4d0 0x7ff66406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:23.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.647+0000 7ff67aa00640 1 -- 172.21.15.28:0/3985809301 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 7129+0+0 (secure 0 0 0) 0x7ff670089b70 con 0x7ff6840c3280 2024-02-21T23:10:23.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.648+0000 7ff682a00640 1 --2- 172.21.15.28:0/3985809301 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff66406b4d0 0x7ff66406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:23.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.649+0000 7ff682a00640 1 --2- 172.21.15.28:0/3985809301 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff66406b4d0 0x7ff66406d990 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7ff67400bfd0 tx=0x7ff67400b750 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:23.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.652+0000 7ff67aa00640 1 -- 172.21.15.28:0/3985809301 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ff670058450 con 0x7ff6840c3280 2024-02-21T23:10:23.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.890+0000 7ff689c00640 1 -- 172.21.15.28:0/3985809301 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff68406f2d0 con 0x7ff6840c3280 2024-02-21T23:10:23.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.892+0000 7ff67aa00640 1 -- 172.21.15.28:0/3985809301 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 170 v170) v1 ==== 95+0+6197 (secure 0 0 0) 0x7ff670057d50 con 0x7ff6840c3280 2024-02-21T23:10:23.880 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:10:23.880 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":170,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9141,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2895215997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2895215997},{"type":"v1","addr":"172.21.15.28:6835","nonce":2895215997}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":166},{"gid":9153,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/2178509171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2178509171},{"type":"v1","addr":"172.21.15.28:6833","nonce":2178509171}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":169}],"filesystems":[{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:09:53.730296+0000","modified":"2024-02-21T23:10:21.591594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":156,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6392},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6392":{"gid":6392,"name":"d","rank":0,"incarnation":165,"state":"up:active","state_seq":11,"addr":"172.21.15.81:6837/521625945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":521625945},{"type":"v1","addr":"172.21.15.81:6837","nonce":521625945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11},{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:10:05.843850+0000","modified":"2024-02-21T23:10:22.596481+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":157,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6389},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6389":{"gid":6389,"name":"b","rank":0,"incarnation":168,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.81:6835/1320495906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1320495906},{"type":"v1","addr":"172.21.15.81:6835","nonce":1320495906}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2024-02-21T23:10:23.880 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 170 2024-02-21T23:10:23.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.894+0000 7ff689c00640 1 -- 172.21.15.28:0/3985809301 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff66406b4d0 msgr2=0x7ff66406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:23.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.895+0000 7ff689c00640 1 --2- 172.21.15.28:0/3985809301 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff66406b4d0 0x7ff66406d990 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7ff67400bfd0 tx=0x7ff67400b750 comp rx=0 tx=0).stop 2024-02-21T23:10:23.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.895+0000 7ff689c00640 1 -- 172.21.15.28:0/3985809301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6840c3280 msgr2=0x7ff68414e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:23.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.895+0000 7ff689c00640 1 --2- 172.21.15.28:0/3985809301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6840c3280 0x7ff68414e060 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7ff67000c390 tx=0x7ff67000c850 comp rx=0 tx=0).stop 2024-02-21T23:10:23.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.895+0000 7ff689c00640 1 -- 172.21.15.28:0/3985809301 shutdown_connections 2024-02-21T23:10:23.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.895+0000 7ff689c00640 1 --2- 172.21.15.28:0/3985809301 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff66406b4d0 0x7ff66406d990 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:23.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.895+0000 7ff689c00640 1 --2- 172.21.15.28:0/3985809301 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff684155860 0x7ff68414e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:23.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.895+0000 7ff689c00640 1 --2- 172.21.15.28:0/3985809301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6840c3280 0x7ff68414e060 unknown :-1 s=CLOSED pgs=1778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:23.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.895+0000 7ff689c00640 1 --2- 172.21.15.28:0/3985809301 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff684072430 0x7ff6840ac370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:23.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.895+0000 7ff689c00640 1 -- 172.21.15.28:0/3985809301 >> 172.21.15.28:0/3985809301 conn(0x7ff6840c67f0 msgr2=0x7ff6840c4d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:23.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.895+0000 7ff689c00640 1 -- 172.21.15.28:0/3985809301 shutdown_connections 2024-02-21T23:10:23.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:23.895+0000 7ff689c00640 1 -- 172.21.15.28:0/3985809301 wait complete. 2024-02-21T23:10:24.307 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 169, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:09:53.730296+0000', 'modified': '2024-02-21T23:10:21.591594+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 156, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6392}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6392': {'gid': 6392, 'name': 'd', 'rank': 0, 'incarnation': 165, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.81:6837/521625945', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 521625945}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 521625945}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [24], 'metadata_pool': 23, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:10:24.307 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-21T23:10:24.308 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-02-21T23:10:24.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.555+0000 7f35fda00640 1 Processor -- start 2024-02-21T23:10:24.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.555+0000 7f35fda00640 1 -- start start 2024-02-21T23:10:24.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.556+0000 7f35fda00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f35f806ed50 0x7f35f80c3280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:24.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.556+0000 7f35fda00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f35f80c37c0 0x7f35f8158ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:24.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.556+0000 7f35fda00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35f80c5af0 con 0x7f35f80c1c00 2024-02-21T23:10:24.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.556+0000 7f35fda00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35f80c5660 con 0x7f35f80c37c0 2024-02-21T23:10:24.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.556+0000 7f35fda00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35f80c4d30 con 0x7f35f806ed50 2024-02-21T23:10:24.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.557+0000 7f35f6a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f35f806ed50 0x7f35f80c3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:24.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.557+0000 7f35f7400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f35f80c37c0 0x7f35f8158ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:24.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.557+0000 7f35f7400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f35f80c37c0 0x7f35f8158ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49976/0 (socket says 172.21.15.28:49976) 2024-02-21T23:10:24.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.557+0000 7f35f6a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f35f806ed50 0x7f35f80c3280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3301/0 says I am v2:172.21.15.28:37758/0 (socket says 172.21.15.28:37758) 2024-02-21T23:10:24.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.557+0000 7f35f7400640 1 -- 172.21.15.28:0/1259196168 learned_addr learned my addr 172.21.15.28:0/1259196168 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:24.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.557+0000 7f35f7400640 1 -- 172.21.15.28:0/1259196168 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f35f806ed50 msgr2=0x7f35f80c3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:24.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.557+0000 7f35f7400640 1 --2- 172.21.15.28:0/1259196168 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f35f806ed50 0x7f35f80c3280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:24.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.557+0000 7f35f7400640 1 -- 172.21.15.28:0/1259196168 >> v1:172.21.15.81:6789/0 conn(0x7f35f80c1c00 legacy=0x7f35f81587c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:24.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.558+0000 7f35f7400640 1 -- 172.21.15.28:0/1259196168 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35f80c5010 con 0x7f35f80c37c0 2024-02-21T23:10:24.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.559+0000 7f35f7400640 1 --2- 172.21.15.28:0/1259196168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35f80c37c0 0x7f35f8158ed0 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7f35f0009e50 tx=0x7f35f002bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9ec1c2390a918a9 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:24.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.559+0000 7f35f5600640 1 -- 172.21.15.28:0/1259196168 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35f002c950 con 0x7f35f80c37c0 2024-02-21T23:10:24.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.559+0000 7f35f5600640 1 -- 172.21.15.28:0/1259196168 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35f002caf0 con 0x7f35f80c37c0 2024-02-21T23:10:24.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.559+0000 7f35f5600640 1 -- 172.21.15.28:0/1259196168 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35f00343f0 con 0x7f35f80c37c0 2024-02-21T23:10:24.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.560+0000 7f35fda00640 1 -- 172.21.15.28:0/1259196168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35f80c37c0 msgr2=0x7f35f8158ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:24.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.560+0000 7f35fda00640 1 --2- 172.21.15.28:0/1259196168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35f80c37c0 0x7f35f8158ed0 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7f35f0009e50 tx=0x7f35f002bdb0 comp rx=0 tx=0).stop 2024-02-21T23:10:24.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.560+0000 7f35fda00640 1 -- 172.21.15.28:0/1259196168 shutdown_connections 2024-02-21T23:10:24.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.560+0000 7f35fda00640 1 --2- 172.21.15.28:0/1259196168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35f80c37c0 0x7f35f8158ed0 unknown :-1 s=CLOSED pgs=1779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:24.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.560+0000 7f35fda00640 1 --2- 172.21.15.28:0/1259196168 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f35f806ed50 0x7f35f80c3280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:24.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.560+0000 7f35fda00640 1 -- 172.21.15.28:0/1259196168 >> 172.21.15.28:0/1259196168 conn(0x7f35f806b820 msgr2=0x7f35f80c67f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:24.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.560+0000 7f35fda00640 1 -- 172.21.15.28:0/1259196168 shutdown_connections 2024-02-21T23:10:24.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.560+0000 7f35fda00640 1 -- 172.21.15.28:0/1259196168 wait complete. 2024-02-21T23:10:24.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.562+0000 7f35fda00640 1 Processor -- start 2024-02-21T23:10:24.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.562+0000 7f35fda00640 1 -- start start 2024-02-21T23:10:24.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.563+0000 7f35fda00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f35f806ed50 0x7f35f8157a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:24.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.563+0000 7f35fda00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35f80c1c00 0x7f35f81478f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:24.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.563+0000 7f35fda00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f35f80c37c0 0x7f35f8149d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:24.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.563+0000 7f35fda00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35f80c6070 con 0x7f35f80c1c00 2024-02-21T23:10:24.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.563+0000 7f35fda00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35f806f130 con 0x7f35f80c37c0 2024-02-21T23:10:24.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.563+0000 7f35fda00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35f80c4880 con 0x7f35f806ed50 2024-02-21T23:10:24.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.563+0000 7f35f6000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35f80c1c00 0x7f35f81478f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:24.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.563+0000 7f35f6000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35f80c1c00 0x7f35f81478f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49980/0 (socket says 172.21.15.28:49980) 2024-02-21T23:10:24.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.563+0000 7f35f6000640 1 -- 172.21.15.28:0/3162783579 learned_addr learned my addr 172.21.15.28:0/3162783579 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:24.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.563+0000 7f35f6a00640 1 --2- 172.21.15.28:0/3162783579 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f35f806ed50 0x7f35f8157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:24.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.563+0000 7f35f7400640 1 --2- 172.21.15.28:0/3162783579 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f35f80c37c0 0x7f35f8149d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:24.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.563+0000 7f35f6000640 1 -- 172.21.15.28:0/3162783579 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f35f806ed50 msgr2=0x7f35f8157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:24.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.563+0000 7f35f6000640 1 --2- 172.21.15.28:0/3162783579 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f35f806ed50 0x7f35f8157a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:24.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.563+0000 7f35f6000640 1 -- 172.21.15.28:0/3162783579 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f35f80c37c0 msgr2=0x7f35f8149d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:24.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.563+0000 7f35f6000640 1 --2- 172.21.15.28:0/3162783579 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f35f80c37c0 0x7f35f8149d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:24.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.563+0000 7f35f6000640 1 -- 172.21.15.28:0/3162783579 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35f814b190 con 0x7f35f80c1c00 2024-02-21T23:10:24.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.563+0000 7f35f7400640 1 --2- 172.21.15.28:0/3162783579 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f35f80c37c0 0x7f35f8149d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:10:24.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.564+0000 7f35f6000640 1 --2- 172.21.15.28:0/3162783579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35f80c1c00 0x7f35f81478f0 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f35e800b420 tx=0x7f35e800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:24.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.564+0000 7f35eea00640 1 -- 172.21.15.28:0/3162783579 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35e8011830 con 0x7f35f80c1c00 2024-02-21T23:10:24.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.564+0000 7f35eea00640 1 -- 172.21.15.28:0/3162783579 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35e8016070 con 0x7f35f80c1c00 2024-02-21T23:10:24.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.564+0000 7f35eea00640 1 -- 172.21.15.28:0/3162783579 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35e8011c10 con 0x7f35f80c1c00 2024-02-21T23:10:24.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.564+0000 7f35fda00640 1 -- 172.21.15.28:0/3162783579 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35f814b430 con 0x7f35f80c1c00 2024-02-21T23:10:24.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.564+0000 7f35fda00640 1 -- 172.21.15.28:0/3162783579 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35f81473d0 con 0x7f35f80c1c00 2024-02-21T23:10:24.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.567+0000 7f35eea00640 1 -- 172.21.15.28:0/3162783579 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f35e801b460 con 0x7f35f80c1c00 2024-02-21T23:10:24.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.567+0000 7f35eea00640 1 --2- 172.21.15.28:0/3162783579 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f35d406b4d0 0x7f35d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:24.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.568+0000 7f35eea00640 1 -- 172.21.15.28:0/3162783579 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 7129+0+0 (secure 0 0 0) 0x7f35e808a320 con 0x7f35f80c1c00 2024-02-21T23:10:24.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.568+0000 7f3600a3a740 1 Processor -- start 2024-02-21T23:10:24.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.568+0000 7f35f6a00640 1 --2- 172.21.15.28:0/3162783579 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f35d406b4d0 0x7f35d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:24.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.568+0000 7f3600a3a740 1 -- start start 2024-02-21T23:10:24.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.569+0000 7f3600a3a740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55736a7a7e40 0x55736a7a8220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:24.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.569+0000 7f3600a3a740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55736a7a9150 0x55736a7e3f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:24.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.569+0000 7f3600a3a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55736a7e44d0 0x55736a7e2540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:24.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.569+0000 7f3600a3a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55736a70c470 con 0x55736a7e44d0 2024-02-21T23:10:24.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.569+0000 7f3600a3a740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55736a6b5af0 con 0x55736a7a7e40 2024-02-21T23:10:24.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.569+0000 7f3600a3a740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x55736a708510 con 0x55736a7a9150 2024-02-21T23:10:24.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.569+0000 7f35f6a00640 1 --2- 172.21.15.28:0/3162783579 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f35d406b4d0 0x7f35d406d990 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f35e0009a30 tx=0x7f35e0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:24.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.569+0000 7f35f7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55736a7e44d0 0x55736a7e2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:24.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.570+0000 7f35f7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55736a7e44d0 0x55736a7e2540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49992/0 (socket says 172.21.15.28:49992) 2024-02-21T23:10:24.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.570+0000 7f35f7400640 1 -- 172.21.15.28:0/3492208643 learned_addr learned my addr 172.21.15.28:0/3492208643 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:24.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.570+0000 7f35f6a00640 1 --2- 172.21.15.28:0/3492208643 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55736a7a7e40 0x55736a7a8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:24.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.570+0000 7f35f6000640 1 --2- 172.21.15.28:0/3492208643 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55736a7a9150 0x55736a7e3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:24.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.570+0000 7f35f7400640 1 -- 172.21.15.28:0/3492208643 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55736a7a9150 msgr2=0x55736a7e3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:24.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.570+0000 7f35f7400640 1 --2- 172.21.15.28:0/3492208643 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55736a7a9150 0x55736a7e3f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:24.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.570+0000 7f35f7400640 1 -- 172.21.15.28:0/3492208643 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55736a7a7e40 msgr2=0x55736a7a8220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:24.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.570+0000 7f35f7400640 1 --2- 172.21.15.28:0/3492208643 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55736a7a7e40 0x55736a7a8220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:24.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.570+0000 7f35f7400640 1 -- 172.21.15.28:0/3492208643 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55736a6bcd90 con 0x55736a7e44d0 2024-02-21T23:10:24.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.570+0000 7f35f7400640 1 --2- 172.21.15.28:0/3492208643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55736a7e44d0 0x55736a7e2540 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7f35f002c3c0 tx=0x7f35f002e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77ce24871e496728 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:24.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.571+0000 7f35e7400640 1 -- 172.21.15.28:0/3492208643 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35f0045070 con 0x55736a7e44d0 2024-02-21T23:10:24.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.571+0000 7f35e7400640 1 -- 172.21.15.28:0/3492208643 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35f002c990 con 0x55736a7e44d0 2024-02-21T23:10:24.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.571+0000 7f35e7400640 1 -- 172.21.15.28:0/3492208643 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35f00406b0 con 0x55736a7e44d0 2024-02-21T23:10:24.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.571+0000 7f3600a3a740 1 -- 172.21.15.28:0/3492208643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55736a7e44d0 msgr2=0x55736a7e2540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:24.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.571+0000 7f3600a3a740 1 --2- 172.21.15.28:0/3492208643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55736a7e44d0 0x55736a7e2540 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7f35f002c3c0 tx=0x7f35f002e000 comp rx=0 tx=0).stop 2024-02-21T23:10:24.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.571+0000 7f3600a3a740 1 -- 172.21.15.28:0/3492208643 shutdown_connections 2024-02-21T23:10:24.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.571+0000 7f3600a3a740 1 --2- 172.21.15.28:0/3492208643 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55736a7a9150 0x55736a7e3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:24.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.571+0000 7f3600a3a740 1 --2- 172.21.15.28:0/3492208643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55736a7e44d0 0x55736a7e2540 unknown :-1 s=CLOSED pgs=1781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:24.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.571+0000 7f3600a3a740 1 --2- 172.21.15.28:0/3492208643 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55736a7a7e40 0x55736a7a8220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:24.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.571+0000 7f3600a3a740 1 -- 172.21.15.28:0/3492208643 >> 172.21.15.28:0/3492208643 conn(0x55736a7a4790 msgr2=0x55736a7c91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:24.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.572+0000 7f3600a3a740 1 -- 172.21.15.28:0/3492208643 shutdown_connections 2024-02-21T23:10:24.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.572+0000 7f3600a3a740 1 -- 172.21.15.28:0/3492208643 wait complete. 2024-02-21T23:10:24.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.572+0000 7f3600a3a740 1 -- start start 2024-02-21T23:10:24.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.573+0000 7f3600a3a740 1 Processor -- start 2024-02-21T23:10:24.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.574+0000 7f3600a3a740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55736a7a9150 0x55736a7f79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:24.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.574+0000 7f3600a3a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55736a7e44d0 0x55736a7ea8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:24.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.574+0000 7f3600a3a740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55736a7c46d0 0x55736a7b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:24.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.574+0000 7f3600a3a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55736a6ab040 con 0x55736a7e44d0 2024-02-21T23:10:24.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.574+0000 7f3600a3a740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55736a7c98b0 con 0x55736a7c46d0 2024-02-21T23:10:24.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.574+0000 7f3600a3a740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x55736a7eb610 con 0x55736a7a9150 2024-02-21T23:10:24.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.574+0000 7f3600a3a740 10 client.0 _ll_register_callbacks cb 0x55736a6c8920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:10:24.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.574+0000 7f35f6000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55736a7e44d0 0x55736a7ea8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:24.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.574+0000 7f35f6000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55736a7e44d0 0x55736a7ea8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50002/0 (socket says 172.21.15.28:50002) 2024-02-21T23:10:24.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.574+0000 7f35f6000640 1 -- 172.21.15.28:0/1287370924 learned_addr learned my addr 172.21.15.28:0/1287370924 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:24.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.574+0000 7f35f7400640 1 --2- 172.21.15.28:0/1287370924 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55736a7c46d0 0x55736a7b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:24.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.574+0000 7f35f6a00640 1 --2- 172.21.15.28:0/1287370924 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55736a7a9150 0x55736a7f79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:24.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.574+0000 7f35f6000640 1 -- 172.21.15.28:0/1287370924 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55736a7a9150 msgr2=0x55736a7f79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:24.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.574+0000 7f35f6000640 1 --2- 172.21.15.28:0/1287370924 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55736a7a9150 0x55736a7f79a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:24.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.574+0000 7f35f6000640 1 -- 172.21.15.28:0/1287370924 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55736a7c46d0 msgr2=0x55736a7b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:24.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.574+0000 7f35f6000640 1 --2- 172.21.15.28:0/1287370924 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55736a7c46d0 0x55736a7b5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:24.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.574+0000 7f35f6000640 1 -- 172.21.15.28:0/1287370924 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55736a6caaa0 con 0x55736a7e44d0 2024-02-21T23:10:24.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.574+0000 7f35f7400640 1 --2- 172.21.15.28:0/1287370924 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55736a7c46d0 0x55736a7b5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:10:24.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.575+0000 7f35f6000640 1 --2- 172.21.15.28:0/1287370924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55736a7e44d0 0x55736a7ea8f0 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7f35e8058400 tx=0x7f35e80585b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40de8f8da3cc308a server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:24.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.575+0000 7f35dbe00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:10:24.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.575+0000 7f35dbe00640 1 -- 172.21.15.28:0/1287370924 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35e805c190 con 0x55736a7e44d0 2024-02-21T23:10:24.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.575+0000 7f35dbe00640 1 -- 172.21.15.28:0/1287370924 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(157..157 src has 1..157) v4 ==== 7129+0+0 (secure 0 0 0) 0x7f35e805c330 con 0x55736a7e44d0 2024-02-21T23:10:24.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.575+0000 7f3600a3a740 1 -- 172.21.15.28:0/1287370924 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f35f002c990 con 0x55736a7e44d0 2024-02-21T23:10:24.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.575+0000 7f35dbe00640 1 -- 172.21.15.28:0/1287370924 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=170) v2 ==== 24+0+0 (secure 0 0 0) 0x7f35e8081b80 con 0x55736a7e44d0 2024-02-21T23:10:24.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.575+0000 7f3600a3a740 10 client.9180 fetch_fsmap learned FSMap version 170 2024-02-21T23:10:24.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.575+0000 7f3600a3a740 1 -- 172.21.15.28:0/1287370924 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=170,monmap=2+}) v3 -- 0x55736a4212d0 con 0x55736a7e44d0 2024-02-21T23:10:24.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.575+0000 7f35dbe00640 1 -- 172.21.15.28:0/1287370924 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35f002c990 con 0x55736a7e44d0 2024-02-21T23:10:24.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.575+0000 7f35dbe00640 1 -- 172.21.15.28:0/1287370924 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 170) v1 ==== 3889+0+0 (secure 0 0 0) 0x7f35e80578c0 con 0x55736a7e44d0 2024-02-21T23:10:24.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.576+0000 7f3600a3a740 10 client.9180 fetch_fsmap finished waiting for FSMap version 170 2024-02-21T23:10:24.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.576+0000 7f3600a3a740 10 client.9180 resolve_mds: resolved name 'd' to daemon mds.d 2024-02-21T23:10:24.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.576+0000 7f3600a3a740 20 client.9180 populate_metadata read hostname 'smithi028' 2024-02-21T23:10:24.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.576+0000 7f3600a3a740 1 --2- 172.21.15.28:0/1287370924 >> [v2:172.21.15.81:6836/521625945,v1:172.21.15.81:6837/521625945] conn(0x55736a7e0e40 0x55736a7e1220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:24.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.576+0000 7f3600a3a740 4 client.9180 mds_command: new command op to 6392 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-21T23:10:24.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.576+0000 7f3600a3a740 1 -- 172.21.15.28:0/1287370924 --> [v2:172.21.15.81:6836/521625945,v1:172.21.15.81:6837/521625945] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55736a7199b0 con 0x55736a7e0e40 2024-02-21T23:10:24.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.576+0000 7f35f6a00640 1 --2- 172.21.15.28:0/1287370924 >> [v2:172.21.15.81:6836/521625945,v1:172.21.15.81:6837/521625945] conn(0x55736a7e0e40 0x55736a7e1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:24.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.577+0000 7f35f6a00640 1 --2- 172.21.15.28:0/1287370924 >> [v2:172.21.15.81:6836/521625945,v1:172.21.15.81:6837/521625945] conn(0x55736a7e0e40 0x55736a7e1220 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=44f4ebfcc38fb9d1 server_cookie=e5bf01cd27262a0e in_seq=0 out_seq=0 2024-02-21T23:10:24.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.577+0000 7f35dbe00640 10 client.9180 ms_handle_connect on v2:172.21.15.81:6836/521625945 2024-02-21T23:10:24.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.579+0000 7f35dbe00640 1 -- 172.21.15.28:0/1287370924 <== mds.0 v2:172.21.15.81:6836/521625945 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55736a7199b0 con 0x55736a7e0e40 2024-02-21T23:10:24.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.579+0000 7f35dbe00640 10 client.9180 handle_command_reply: tid=0 2024-02-21T23:10:24.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.579+0000 7f3600a3a740 1 client.9180 shutdown 2024-02-21T23:10:24.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.580+0000 7f3600a3a740 1 -- 172.21.15.28:0/1287370924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55736a7e44d0 msgr2=0x55736a7ea8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:24.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.580+0000 7f3600a3a740 1 --2- 172.21.15.28:0/1287370924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55736a7e44d0 0x55736a7ea8f0 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7f35e8058400 tx=0x7f35e80585b0 comp rx=0 tx=0).stop 2024-02-21T23:10:24.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.580+0000 7f3600a3a740 1 -- 172.21.15.28:0/1287370924 shutdown_connections 2024-02-21T23:10:24.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.580+0000 7f3600a3a740 1 --2- 172.21.15.28:0/1287370924 >> [v2:172.21.15.81:6836/521625945,v1:172.21.15.81:6837/521625945] conn(0x55736a7e0e40 0x55736a7e1220 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:24.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.580+0000 7f3600a3a740 1 --2- 172.21.15.28:0/1287370924 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55736a7c46d0 0x55736a7b5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:24.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.580+0000 7f3600a3a740 1 --2- 172.21.15.28:0/1287370924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55736a7e44d0 0x55736a7ea8f0 unknown :-1 s=CLOSED pgs=1782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:24.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.580+0000 7f35dbe00640 0 client.9180 ms_handle_reset on v2:172.21.15.81:6836/521625945 2024-02-21T23:10:24.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.580+0000 7f3600a3a740 1 --2- 172.21.15.28:0/1287370924 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55736a7a9150 0x55736a7f79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:24.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.580+0000 7f3600a3a740 1 -- 172.21.15.28:0/1287370924 >> 172.21.15.28:0/1287370924 conn(0x55736a7a7e40 msgr2=0x55736a7a15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:24.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.580+0000 7f3600a3a740 1 -- 172.21.15.28:0/1287370924 shutdown_connections 2024-02-21T23:10:24.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.580+0000 7f3600a3a740 1 -- 172.21.15.28:0/1287370924 wait complete. 2024-02-21T23:10:24.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.580+0000 7f3600a3a740 20 client.9180 trim_cache size 0 max 16384 2024-02-21T23:10:24.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.599+0000 7f3600a3a740 1 Processor -- start 2024-02-21T23:10:24.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.600+0000 7f3600a3a740 1 -- start start 2024-02-21T23:10:24.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.601+0000 7f3600a3a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55736a7a9150 0x55736a7e3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:24.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.601+0000 7f3600a3a740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55736a7c46d0 0x55736a7e5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:24.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.601+0000 7f3600a3a740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55736a7e44d0 0x55736a7a15f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:24.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.601+0000 7f3600a3a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35f81f9dd0 con 0x55736a7a9150 2024-02-21T23:10:24.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.601+0000 7f3600a3a740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35f81facb0 con 0x55736a7e44d0 2024-02-21T23:10:24.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.601+0000 7f3600a3a740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35f81f8ad0 con 0x55736a7c46d0 2024-02-21T23:10:24.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.601+0000 7f35f6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55736a7a9150 0x55736a7e3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:24.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.601+0000 7f35f6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55736a7a9150 0x55736a7e3bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50004/0 (socket says 172.21.15.28:50004) 2024-02-21T23:10:24.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.601+0000 7f35f6a00640 1 -- 172.21.15.28:0/2796725916 learned_addr learned my addr 172.21.15.28:0/2796725916 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:24.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.601+0000 7f35f6000640 1 --2- 172.21.15.28:0/2796725916 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55736a7c46d0 0x55736a7e5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:24.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.601+0000 7f35f7400640 1 --2- 172.21.15.28:0/2796725916 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55736a7e44d0 0x55736a7a15f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:24.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.601+0000 7f35f6a00640 1 -- 172.21.15.28:0/2796725916 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55736a7c46d0 msgr2=0x55736a7e5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:24.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.601+0000 7f35f6a00640 1 --2- 172.21.15.28:0/2796725916 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55736a7c46d0 0x55736a7e5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:24.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.601+0000 7f35f6a00640 1 -- 172.21.15.28:0/2796725916 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55736a7e44d0 msgr2=0x55736a7a15f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:24.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.601+0000 7f35f6a00640 1 --2- 172.21.15.28:0/2796725916 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55736a7e44d0 0x55736a7a15f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:24.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.601+0000 7f35f6a00640 1 -- 172.21.15.28:0/2796725916 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55736a7a8590 con 0x55736a7a9150 2024-02-21T23:10:24.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.601+0000 7f35f7400640 1 --2- 172.21.15.28:0/2796725916 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55736a7e44d0 0x55736a7a15f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:10:24.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.601+0000 7f35f6000640 1 --2- 172.21.15.28:0/2796725916 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55736a7c46d0 0x55736a7e5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:10:24.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.602+0000 7f35f6a00640 1 --2- 172.21.15.28:0/2796725916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55736a7a9150 0x55736a7e3bb0 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7f35e0011630 tx=0x7f35e0014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d5f5ccaca0d1376 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:24.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.602+0000 7f35e6a00640 1 -- 172.21.15.28:0/2796725916 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35e0030070 con 0x55736a7a9150 2024-02-21T23:10:24.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.602+0000 7f35e6a00640 1 -- 172.21.15.28:0/2796725916 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35e001a040 con 0x55736a7a9150 2024-02-21T23:10:24.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.602+0000 7f35e6a00640 1 -- 172.21.15.28:0/2796725916 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35e002ad70 con 0x55736a7a9150 2024-02-21T23:10:24.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.603+0000 7f3600a3a740 1 -- 172.21.15.28:0/2796725916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55736a7a9150 msgr2=0x55736a7e3bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:24.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.603+0000 7f3600a3a740 1 --2- 172.21.15.28:0/2796725916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55736a7a9150 0x55736a7e3bb0 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7f35e0011630 tx=0x7f35e0014000 comp rx=0 tx=0).stop 2024-02-21T23:10:24.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.603+0000 7f3600a3a740 1 -- 172.21.15.28:0/2796725916 shutdown_connections 2024-02-21T23:10:24.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.603+0000 7f3600a3a740 1 --2- 172.21.15.28:0/2796725916 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55736a7c46d0 0x55736a7e5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:24.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.603+0000 7f3600a3a740 1 --2- 172.21.15.28:0/2796725916 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55736a7e44d0 0x55736a7a15f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:24.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.603+0000 7f3600a3a740 1 --2- 172.21.15.28:0/2796725916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55736a7a9150 0x55736a7e3bb0 unknown :-1 s=CLOSED pgs=1783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:24.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.603+0000 7f3600a3a740 1 -- 172.21.15.28:0/2796725916 >> 172.21.15.28:0/2796725916 conn(0x55736a7a7e40 msgr2=0x55736a7c8fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:24.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.603+0000 7f3600a3a740 1 -- 172.21.15.28:0/2796725916 shutdown_connections 2024-02-21T23:10:24.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.603+0000 7f3600a3a740 1 -- 172.21.15.28:0/2796725916 wait complete. 2024-02-21T23:10:24.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.604+0000 7f3600a3a740 1 -- start start 2024-02-21T23:10:24.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.604+0000 7f3600a3a740 1 Processor -- start 2024-02-21T23:10:24.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.605+0000 7f3600a3a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55736a7a9150 0x55736a7e9c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:24.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.605+0000 7f3600a3a740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55736a7c46d0 0x55736a7a19f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:24.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.605+0000 7f3600a3a740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55736a7e44d0 0x55736a850300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:24.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.605+0000 7f3600a3a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55736a7e0cc0 con 0x55736a7a9150 2024-02-21T23:10:24.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.605+0000 7f3600a3a740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35f81f9040 con 0x55736a7e44d0 2024-02-21T23:10:24.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.605+0000 7f3600a3a740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x55736a7b6370 con 0x55736a7c46d0 2024-02-21T23:10:24.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.605+0000 7f3600a3a740 10 client.0 _ll_register_callbacks cb 0x7f35e8058b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:10:24.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.605+0000 7f35f6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55736a7a9150 0x55736a7e9c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:24.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.605+0000 7f35f6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55736a7a9150 0x55736a7e9c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50006/0 (socket says 172.21.15.28:50006) 2024-02-21T23:10:24.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.605+0000 7f35f6a00640 1 -- 172.21.15.28:0/1253193014 learned_addr learned my addr 172.21.15.28:0/1253193014 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:24.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.605+0000 7f35f6000640 1 --2- 172.21.15.28:0/1253193014 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55736a7c46d0 0x55736a7a19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:24.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.605+0000 7f35f7400640 1 --2- 172.21.15.28:0/1253193014 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55736a7e44d0 0x55736a850300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:24.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.606+0000 7f35f6a00640 1 -- 172.21.15.28:0/1253193014 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55736a7c46d0 msgr2=0x55736a7a19f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:24.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.606+0000 7f35f6a00640 1 --2- 172.21.15.28:0/1253193014 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55736a7c46d0 0x55736a7a19f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:24.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.606+0000 7f35f6a00640 1 -- 172.21.15.28:0/1253193014 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55736a7e44d0 msgr2=0x55736a850300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:24.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.606+0000 7f35f6a00640 1 --2- 172.21.15.28:0/1253193014 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55736a7e44d0 0x55736a850300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:24.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.606+0000 7f35f6a00640 1 -- 172.21.15.28:0/1253193014 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55736a4eb6b0 con 0x55736a7a9150 2024-02-21T23:10:24.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.606+0000 7f35f6a00640 1 --2- 172.21.15.28:0/1253193014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55736a7a9150 0x55736a7e9c20 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7f35e0012010 tx=0x7f35e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31db5aa7a8cca15b server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:24.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.606+0000 7f35dbe00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:10:24.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.606+0000 7f35dbe00640 1 -- 172.21.15.28:0/1253193014 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35e003b060 con 0x55736a7a9150 2024-02-21T23:10:24.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.606+0000 7f35dbe00640 1 -- 172.21.15.28:0/1253193014 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(157..157 src has 1..157) v4 ==== 7129+0+0 (secure 0 0 0) 0x7f35e002ad80 con 0x55736a7a9150 2024-02-21T23:10:24.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.606+0000 7f3600a3a740 1 -- 172.21.15.28:0/1253193014 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f35e001a040 con 0x55736a7a9150 2024-02-21T23:10:24.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.607+0000 7f35dbe00640 1 -- 172.21.15.28:0/1253193014 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=171) v2 ==== 24+0+0 (secure 0 0 0) 0x7f35e0041050 con 0x55736a7a9150 2024-02-21T23:10:24.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.607+0000 7f3600a3a740 10 client.9186 fetch_fsmap learned FSMap version 171 2024-02-21T23:10:24.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.607+0000 7f3600a3a740 1 -- 172.21.15.28:0/1253193014 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=171,monmap=2+}) v3 -- 0x55736a7e2d50 con 0x55736a7a9150 2024-02-21T23:10:24.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.607+0000 7f35dbe00640 1 -- 172.21.15.28:0/1253193014 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35e001a040 con 0x55736a7a9150 2024-02-21T23:10:24.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.607+0000 7f35dbe00640 1 -- 172.21.15.28:0/1253193014 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 171) v1 ==== 3889+0+0 (secure 0 0 0) 0x7f35e0044020 con 0x55736a7a9150 2024-02-21T23:10:24.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.607+0000 7f3600a3a740 10 client.9186 fetch_fsmap finished waiting for FSMap version 171 2024-02-21T23:10:24.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.607+0000 7f3600a3a740 10 client.9186 resolve_mds: resolved name 'd' to daemon mds.d 2024-02-21T23:10:24.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.607+0000 7f3600a3a740 20 client.9186 populate_metadata read hostname 'smithi028' 2024-02-21T23:10:24.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.607+0000 7f3600a3a740 1 --2- 172.21.15.28:0/1253193014 >> [v2:172.21.15.81:6836/521625945,v1:172.21.15.81:6837/521625945] conn(0x55736a850950 0x55736a7a22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:24.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.607+0000 7f3600a3a740 4 client.9186 mds_command: new command op to 6392 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-21T23:10:24.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.607+0000 7f3600a3a740 1 -- 172.21.15.28:0/1253193014 --> [v2:172.21.15.81:6836/521625945,v1:172.21.15.81:6837/521625945] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55736a7df7a0 con 0x55736a850950 2024-02-21T23:10:24.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.608+0000 7f35f6000640 1 --2- 172.21.15.28:0/1253193014 >> [v2:172.21.15.81:6836/521625945,v1:172.21.15.81:6837/521625945] conn(0x55736a850950 0x55736a7a22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:24.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.608+0000 7f35f6000640 1 --2- 172.21.15.28:0/1253193014 >> [v2:172.21.15.81:6836/521625945,v1:172.21.15.81:6837/521625945] conn(0x55736a850950 0x55736a7a22e0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=11b183d1f3c4e481 server_cookie=bf34681a966eba20 in_seq=0 out_seq=0 2024-02-21T23:10:24.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.608+0000 7f35dbe00640 10 client.9186 ms_handle_connect on v2:172.21.15.81:6836/521625945 2024-02-21T23:10:24.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.609+0000 7f35dbe00640 1 -- 172.21.15.28:0/1253193014 <== mds.0 v2:172.21.15.81:6836/521625945 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+334 (crc 0 0 0) 0x55736a7df7a0 con 0x55736a850950 2024-02-21T23:10:24.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.609+0000 7f35dbe00640 10 client.9186 handle_command_reply: tid=0 2024-02-21T23:10:24.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.609+0000 7f3600a3a740 1 client.9186 shutdown 2024-02-21T23:10:24.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.609+0000 7f3600a3a740 1 -- 172.21.15.28:0/1253193014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55736a7a9150 msgr2=0x55736a7e9c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:24.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.609+0000 7f3600a3a740 1 --2- 172.21.15.28:0/1253193014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55736a7a9150 0x55736a7e9c20 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7f35e0012010 tx=0x7f35e00046c0 comp rx=0 tx=0).stop 2024-02-21T23:10:24.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.609+0000 7f3600a3a740 1 -- 172.21.15.28:0/1253193014 shutdown_connections 2024-02-21T23:10:24.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.609+0000 7f3600a3a740 1 --2- 172.21.15.28:0/1253193014 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55736a7c46d0 0x55736a7a19f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:24.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.609+0000 7f3600a3a740 1 --2- 172.21.15.28:0/1253193014 >> [v2:172.21.15.81:6836/521625945,v1:172.21.15.81:6837/521625945] conn(0x55736a850950 0x55736a7a22e0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:24.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.609+0000 7f3600a3a740 1 --2- 172.21.15.28:0/1253193014 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55736a7e44d0 0x55736a850300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:24.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.609+0000 7f3600a3a740 1 --2- 172.21.15.28:0/1253193014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55736a7a9150 0x55736a7e9c20 unknown :-1 s=CLOSED pgs=1784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:24.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.609+0000 7f3600a3a740 1 -- 172.21.15.28:0/1253193014 >> 172.21.15.28:0/1253193014 conn(0x55736a7a7e40 msgr2=0x55736a7e1220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:24.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.609+0000 7f35dbe00640 0 client.9186 ms_handle_reset on v2:172.21.15.81:6836/521625945 2024-02-21T23:10:24.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.609+0000 7f3600a3a740 1 -- 172.21.15.28:0/1253193014 shutdown_connections 2024-02-21T23:10:24.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.609+0000 7f3600a3a740 1 -- 172.21.15.28:0/1253193014 wait complete. 2024-02-21T23:10:24.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.610+0000 7f3600a3a740 20 client.9186 trim_cache size 0 max 16384 2024-02-21T23:10:24.604 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-21T23:10:24.604 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac", 2024-02-21T23:10:24.605 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-02-21T23:10:24.605 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 6392, 2024-02-21T23:10:24.605 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-02-21T23:10:24.605 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-02-21T23:10:24.605 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-02-21T23:10:24.605 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 5.0426413940000003, 2024-02-21T23:10:24.605 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 169, 2024-02-21T23:10:24.605 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 157, 2024-02-21T23:10:24.605 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 157, 2024-02-21T23:10:24.605 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 35.802592257999997 2024-02-21T23:10:24.605 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-21T23:10:24.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.610+0000 7f35fda00640 1 -- 172.21.15.28:0/3162783579 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f35d406b4d0 msgr2=0x7f35d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:24.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.610+0000 7f35fda00640 1 --2- 172.21.15.28:0/3162783579 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f35d406b4d0 0x7f35d406d990 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f35e0009a30 tx=0x7f35e0008040 comp rx=0 tx=0).stop 2024-02-21T23:10:24.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.610+0000 7f35fda00640 1 -- 172.21.15.28:0/3162783579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35f80c1c00 msgr2=0x7f35f81478f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:24.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.610+0000 7f35fda00640 1 --2- 172.21.15.28:0/3162783579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35f80c1c00 0x7f35f81478f0 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f35e800b420 tx=0x7f35e800b8e0 comp rx=0 tx=0).stop 2024-02-21T23:10:24.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.610+0000 7f35fda00640 1 -- 172.21.15.28:0/3162783579 shutdown_connections 2024-02-21T23:10:24.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.610+0000 7f35fda00640 1 --2- 172.21.15.28:0/3162783579 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f35d406b4d0 0x7f35d406d990 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:24.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.610+0000 7f35fda00640 1 --2- 172.21.15.28:0/3162783579 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f35f80c37c0 0x7f35f8149d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:24.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.610+0000 7f35fda00640 1 --2- 172.21.15.28:0/3162783579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35f80c1c00 0x7f35f81478f0 unknown :-1 s=CLOSED pgs=1780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:24.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.610+0000 7f35fda00640 1 --2- 172.21.15.28:0/3162783579 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f35f806ed50 0x7f35f8157a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:24.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.610+0000 7f35fda00640 1 -- 172.21.15.28:0/3162783579 >> 172.21.15.28:0/3162783579 conn(0x7f35f806b820 msgr2=0x7f35f814dc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:24.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.610+0000 7f35fda00640 1 -- 172.21.15.28:0/3162783579 shutdown_connections 2024-02-21T23:10:24.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.611+0000 7f35fda00640 1 -- 172.21.15.28:0/3162783579 wait complete. 2024-02-21T23:10:24.616 DEBUG:tasks.cephfs.test_failover:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 11, "epoch": 170, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 11, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-02-21T23:09:53.730296+0000", "damaged": [], "data_pools": [ 24 ], "enabled": true, "epoch": 169, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6392": { "addr": "172.21.15.81:6837/521625945", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6836", "nonce": 521625945, "type": "v2" }, { "addr": "172.21.15.81:6837", "nonce": 521625945, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6392, "incarnation": 165, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 11 } }, "last_failure": 0, "last_failure_osd_epoch": 156, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 23, "modified": "2024-02-21T23:10:21.591594+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6392 } } }, { "id": 12, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-02-21T23:10:05.843850+0000", "damaged": [], "data_pools": [ 26 ], "enabled": true, "epoch": 170, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs2", "in": [ 0 ], "info": { "gid_6389": { "addr": "172.21.15.81:6835/1320495906", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6834", "nonce": 1320495906, "type": "v2" }, { "addr": "172.21.15.81:6835", "nonce": 1320495906, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6389, "incarnation": 168, "join_fscid": 12, "name": "b", "rank": 0, "state": "up:rejoin", "state_seq": 11 } }, "last_failure": 0, "last_failure_osd_epoch": 157, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 25, "modified": "2024-02-21T23:10:22.596481+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6389 } } } ], "standbys": [ { "addr": "172.21.15.28:6835/2895215997", "addrs": { "addrvec": [ { "addr": "172.21.15.28:6834", "nonce": 2895215997, "type": "v2" }, { "addr": "172.21.15.28:6835", "nonce": 2895215997, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 166, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 9141, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.28:6833/2178509171", "addrs": { "addrvec": [ { "addr": "172.21.15.28:6832", "nonce": 2178509171, "type": "v2" }, { "addr": "172.21.15.28:6833", "nonce": 2178509171, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 169, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 9153, "incarnation": 0, "join_fscid": 12, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-02-21T23:10:24.618 INFO:tasks.cephfs.test_failover:target = [{'join_fscid': -1, 'name': 'a'}, {'join_fscid': 12, 'name': 'b'}, {'join_fscid': 12, 'name': 'c'}, {'join_fscid': -1, 'name': 'd', 'state': 'up:active'}] 2024-02-21T23:10:24.618 INFO:tasks.cephfs.test_failover:current = [{'gid': 9141, 'name': 'a', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.28:6835/2895215997', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 2895215997}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 2895215997}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 166}, {'gid': 6389, 'name': 'b', 'rank': 0, 'incarnation': 168, 'state': 'up:rejoin', 'state_seq': 11, 'addr': '172.21.15.81:6835/1320495906', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6834', 'nonce': 1320495906}, {'type': 'v1', 'addr': '172.21.15.81:6835', 'nonce': 1320495906}]}, 'join_fscid': 12, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, {'gid': 9153, 'name': 'c', 'rank': -1, 'incarnation': 0, 'state': 'up:standby', 'state_seq': 1, 'addr': '172.21.15.28:6833/2178509171', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 2178509171}, {'type': 'v1', 'addr': '172.21.15.28:6833', 'nonce': 2178509171}]}, 'join_fscid': 12, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'epoch': 169}, {'gid': 6392, 'name': 'd', 'rank': 0, 'incarnation': 165, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.81:6837/521625945', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 521625945}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 521625945}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}] 2024-02-21T23:10:24.619 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-02-21T23:10:24.619 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:10:24.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.836+0000 7f8db2600640 1 Processor -- start 2024-02-21T23:10:24.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.836+0000 7f8db2600640 1 -- start start 2024-02-21T23:10:24.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.838+0000 7f8db2600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8dac0c11f0 0x7f8dac153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:24.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.838+0000 7f8db2600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8dac15b6d0 0x7f8dac15dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:24.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.838+0000 7f8db2600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dac152e70 con 0x7f8dac0c15d0 2024-02-21T23:10:24.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.838+0000 7f8db2600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8dac0c1a60 con 0x7f8dac0c11f0 2024-02-21T23:10:24.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.838+0000 7f8db2600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dac0a5ec0 con 0x7f8dac15b6d0 2024-02-21T23:10:24.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.838+0000 7f8dab400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8dac0c11f0 0x7f8dac153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:24.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.838+0000 7f8dab400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8dac0c11f0 0x7f8dac153950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50014/0 (socket says 172.21.15.28:50014) 2024-02-21T23:10:24.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.838+0000 7f8dab400640 1 -- 172.21.15.28:0/482686501 learned_addr learned my addr 172.21.15.28:0/482686501 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:24.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.838+0000 7f8dabe00640 1 --2- 172.21.15.28:0/482686501 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8dac15b6d0 0x7f8dac15dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:24.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.838+0000 7f8dab400640 1 -- 172.21.15.28:0/482686501 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8dac15b6d0 msgr2=0x7f8dac15dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:24.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.838+0000 7f8dab400640 1 --2- 172.21.15.28:0/482686501 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8dac15b6d0 0x7f8dac15dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:24.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.838+0000 7f8dab400640 1 -- 172.21.15.28:0/482686501 >> v1:172.21.15.81:6789/0 conn(0x7f8dac0c15d0 legacy=0x7f8dac06dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:24.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.838+0000 7f8dab400640 1 -- 172.21.15.28:0/482686501 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dac153150 con 0x7f8dac0c11f0 2024-02-21T23:10:24.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.838+0000 7f8dabe00640 1 --2- 172.21.15.28:0/482686501 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8dac15b6d0 0x7f8dac15dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:10:24.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.840+0000 7f8dab400640 1 --2- 172.21.15.28:0/482686501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8dac0c11f0 0x7f8dac153950 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7f8d9c009870 tx=0x7f8d9c02b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b11b4bca29af093f server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:24.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.840+0000 7f8daa000640 1 -- 172.21.15.28:0/482686501 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d9c038070 con 0x7f8dac0c11f0 2024-02-21T23:10:24.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.840+0000 7f8daa000640 1 -- 172.21.15.28:0/482686501 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d9c031740 con 0x7f8dac0c11f0 2024-02-21T23:10:24.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.840+0000 7f8daa000640 1 -- 172.21.15.28:0/482686501 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d9c02f050 con 0x7f8dac0c11f0 2024-02-21T23:10:24.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.841+0000 7f8db2600640 1 -- 172.21.15.28:0/482686501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8dac0c11f0 msgr2=0x7f8dac153950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:24.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.841+0000 7f8db2600640 1 --2- 172.21.15.28:0/482686501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8dac0c11f0 0x7f8dac153950 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7f8d9c009870 tx=0x7f8d9c02b7e0 comp rx=0 tx=0).stop 2024-02-21T23:10:24.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.841+0000 7f8db2600640 1 -- 172.21.15.28:0/482686501 shutdown_connections 2024-02-21T23:10:24.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.841+0000 7f8db2600640 1 --2- 172.21.15.28:0/482686501 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8dac15b6d0 0x7f8dac15dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:24.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.841+0000 7f8db2600640 1 --2- 172.21.15.28:0/482686501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8dac0c11f0 0x7f8dac153950 unknown :-1 s=CLOSED pgs=1785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:24.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.841+0000 7f8db2600640 1 -- 172.21.15.28:0/482686501 >> 172.21.15.28:0/482686501 conn(0x7f8dac0c67f0 msgr2=0x7f8dac0c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:24.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.841+0000 7f8db2600640 1 -- 172.21.15.28:0/482686501 shutdown_connections 2024-02-21T23:10:24.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.841+0000 7f8db2600640 1 -- 172.21.15.28:0/482686501 wait complete. 2024-02-21T23:10:24.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.843+0000 7f8db2600640 1 Processor -- start 2024-02-21T23:10:24.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.843+0000 7f8db2600640 1 -- start start 2024-02-21T23:10:24.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.844+0000 7f8db2600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8dac0c11f0 0x7f8dac1ef070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:24.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.844+0000 7f8db2600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8dac0c15d0 0x7f8dac1497d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:24.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.844+0000 7f8db2600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8dac15b6d0 0x7f8dac0a63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:24.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.844+0000 7f8db2600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dac0c4c00 con 0x7f8dac15b6d0 2024-02-21T23:10:24.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.844+0000 7f8db2600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dac14d1c0 con 0x7f8dac0c11f0 2024-02-21T23:10:24.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.844+0000 7f8db2600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dac0a6110 con 0x7f8dac0c15d0 2024-02-21T23:10:24.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.844+0000 7f8dabe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8dac15b6d0 0x7f8dac0a63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:24.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.844+0000 7f8dabe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8dac15b6d0 0x7f8dac0a63d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50028/0 (socket says 172.21.15.28:50028) 2024-02-21T23:10:24.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.844+0000 7f8dabe00640 1 -- 172.21.15.28:0/3427572001 learned_addr learned my addr 172.21.15.28:0/3427572001 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:24.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.844+0000 7f8dab400640 1 --2- 172.21.15.28:0/3427572001 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8dac0c11f0 0x7f8dac1ef070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:24.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.844+0000 7f8daaa00640 1 --2- 172.21.15.28:0/3427572001 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8dac0c15d0 0x7f8dac1497d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:24.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.845+0000 7f8dabe00640 1 -- 172.21.15.28:0/3427572001 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8dac0c15d0 msgr2=0x7f8dac1497d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:24.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.845+0000 7f8dabe00640 1 --2- 172.21.15.28:0/3427572001 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8dac0c15d0 0x7f8dac1497d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:24.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.845+0000 7f8dabe00640 1 -- 172.21.15.28:0/3427572001 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8dac0c11f0 msgr2=0x7f8dac1ef070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:24.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.845+0000 7f8dabe00640 1 --2- 172.21.15.28:0/3427572001 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8dac0c11f0 0x7f8dac1ef070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:24.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.845+0000 7f8dabe00640 1 -- 172.21.15.28:0/3427572001 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dac1ee5b0 con 0x7f8dac15b6d0 2024-02-21T23:10:24.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.845+0000 7f8dabe00640 1 --2- 172.21.15.28:0/3427572001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8dac15b6d0 0x7f8dac0a63d0 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7f8da400cd80 tx=0x7f8da400a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:24.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.845+0000 7f8da3400640 1 -- 172.21.15.28:0/3427572001 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8da400f680 con 0x7f8dac15b6d0 2024-02-21T23:10:24.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.846+0000 7f8da3400640 1 -- 172.21.15.28:0/3427572001 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8da400f840 con 0x7f8dac15b6d0 2024-02-21T23:10:24.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.846+0000 7f8db2600640 1 -- 172.21.15.28:0/3427572001 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dac13ded0 con 0x7f8dac15b6d0 2024-02-21T23:10:24.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.846+0000 7f8da3400640 1 -- 172.21.15.28:0/3427572001 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8da400fc30 con 0x7f8dac15b6d0 2024-02-21T23:10:24.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.846+0000 7f8db2600640 1 -- 172.21.15.28:0/3427572001 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dac1eeb50 con 0x7f8dac15b6d0 2024-02-21T23:10:24.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.847+0000 7f8db2600640 1 -- 172.21.15.28:0/3427572001 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dac1eed50 con 0x7f8dac15b6d0 2024-02-21T23:10:24.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.848+0000 7f8da3400640 1 -- 172.21.15.28:0/3427572001 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8da40213f0 con 0x7f8dac15b6d0 2024-02-21T23:10:24.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.849+0000 7f8da3400640 1 --2- 172.21.15.28:0/3427572001 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8d8c06b4d0 0x7f8d8c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:24.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.849+0000 7f8da3400640 1 -- 172.21.15.28:0/3427572001 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 7129+0+0 (secure 0 0 0) 0x7f8da408b4d0 con 0x7f8dac15b6d0 2024-02-21T23:10:24.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.849+0000 7f8dab400640 1 --2- 172.21.15.28:0/3427572001 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8d8c06b4d0 0x7f8d8c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:24.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.851+0000 7f8dab400640 1 --2- 172.21.15.28:0/3427572001 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8d8c06b4d0 0x7f8d8c06d990 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f8d9c035000 tx=0x7f8d9c008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:24.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:24.854+0000 7f8da3400640 1 -- 172.21.15.28:0/3427572001 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f8dac1eed50 con 0x7f8dac15b6d0 2024-02-21T23:10:25.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.097+0000 7f8db2600640 1 -- 172.21.15.28:0/3427572001 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8dac067f30 con 0x7f8dac15b6d0 2024-02-21T23:10:25.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.098+0000 7f8da3400640 1 -- 172.21.15.28:0/3427572001 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 171 v171) v1 ==== 95+0+6197 (secure 0 0 0) 0x7f8da4014030 con 0x7f8dac15b6d0 2024-02-21T23:10:25.086 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:10:25.087 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":171,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9141,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2895215997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2895215997},{"type":"v1","addr":"172.21.15.28:6835","nonce":2895215997}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":166},{"gid":9153,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/2178509171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2178509171},{"type":"v1","addr":"172.21.15.28:6833","nonce":2178509171}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":169}],"filesystems":[{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:09:53.730296+0000","modified":"2024-02-21T23:10:21.591594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":156,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6392},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6392":{"gid":6392,"name":"d","rank":0,"incarnation":165,"state":"up:active","state_seq":11,"addr":"172.21.15.81:6837/521625945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":521625945},{"type":"v1","addr":"172.21.15.81:6837","nonce":521625945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11},{"mdsmap":{"epoch":171,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:10:05.843850+0000","modified":"2024-02-21T23:10:23.599068+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":157,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6389},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6389":{"gid":6389,"name":"b","rank":0,"incarnation":168,"state":"up:active","state_seq":12,"addr":"172.21.15.81:6835/1320495906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1320495906},{"type":"v1","addr":"172.21.15.81:6835","nonce":1320495906}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2024-02-21T23:10:25.087 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 171 2024-02-21T23:10:25.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.101+0000 7f8db2600640 1 -- 172.21.15.28:0/3427572001 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8d8c06b4d0 msgr2=0x7f8d8c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:25.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.101+0000 7f8db2600640 1 --2- 172.21.15.28:0/3427572001 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8d8c06b4d0 0x7f8d8c06d990 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f8d9c035000 tx=0x7f8d9c008030 comp rx=0 tx=0).stop 2024-02-21T23:10:25.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.101+0000 7f8db2600640 1 -- 172.21.15.28:0/3427572001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8dac15b6d0 msgr2=0x7f8dac0a63d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:25.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.101+0000 7f8db2600640 1 --2- 172.21.15.28:0/3427572001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8dac15b6d0 0x7f8dac0a63d0 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7f8da400cd80 tx=0x7f8da400a650 comp rx=0 tx=0).stop 2024-02-21T23:10:25.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.101+0000 7f8db2600640 1 -- 172.21.15.28:0/3427572001 shutdown_connections 2024-02-21T23:10:25.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.101+0000 7f8db2600640 1 --2- 172.21.15.28:0/3427572001 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8d8c06b4d0 0x7f8d8c06d990 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:25.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.101+0000 7f8db2600640 1 --2- 172.21.15.28:0/3427572001 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8dac0c15d0 0x7f8dac1497d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:25.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.101+0000 7f8db2600640 1 --2- 172.21.15.28:0/3427572001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8dac15b6d0 0x7f8dac0a63d0 unknown :-1 s=CLOSED pgs=1786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:25.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.101+0000 7f8db2600640 1 --2- 172.21.15.28:0/3427572001 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8dac0c11f0 0x7f8dac1ef070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:25.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.101+0000 7f8db2600640 1 -- 172.21.15.28:0/3427572001 >> 172.21.15.28:0/3427572001 conn(0x7f8dac0c67f0 msgr2=0x7f8dac0c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:25.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.101+0000 7f8db2600640 1 -- 172.21.15.28:0/3427572001 shutdown_connections 2024-02-21T23:10:25.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.101+0000 7f8db2600640 1 -- 172.21.15.28:0/3427572001 wait complete. 2024-02-21T23:10:25.108 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-02-21T23:10:25.108 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-21T23:10:25.204 INFO:tasks.ceph.mds.b.smithi081.stderr:2024-02-21T23:10:25.216+0000 7f4689e00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-21T23:10:25.204 INFO:tasks.ceph.mds.d.smithi081.stderr:2024-02-21T23:10:25.216+0000 7f56b0800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-21T23:10:25.225 DEBUG:teuthology.orchestra.run.smithi081:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-21T23:10:25.359 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-02-21T23:10:25.359 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-21T23:10:25.359 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-21T23:10:25.417 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054608,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-02-21T23:10:25.417 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-02-21T23:10:25.417 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-21T23:10:25.463 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053833,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498812k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249404k,nr_inodes=812351,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498812k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-02-21T23:10:25.464 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:10:25.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.707+0000 7f5779000640 1 Processor -- start 2024-02-21T23:10:25.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.707+0000 7f5779000640 1 -- start start 2024-02-21T23:10:25.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.709+0000 7f5779000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f57740aff90 0x7f57740c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:25.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.709+0000 7f5779000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f577406dbf0 0x7f5774158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:25.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.709+0000 7f5779000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f57740c5720 0x7f5774159000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:25.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.709+0000 7f5779000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57740c7270 con 0x7f57740aff90 2024-02-21T23:10:25.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.709+0000 7f5779000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5774158630 con 0x7f577406dbf0 2024-02-21T23:10:25.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.709+0000 7f5779000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57740c74c0 con 0x7f57740c5720 2024-02-21T23:10:25.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.709+0000 7f5773400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f57740aff90 0x7f57740c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:25.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.709+0000 7f5773400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f57740aff90 0x7f57740c51e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39264/0 (socket says 172.21.15.28:39264) 2024-02-21T23:10:25.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.709+0000 7f5773400640 1 -- 172.21.15.28:0/958080766 learned_addr learned my addr 172.21.15.28:0/958080766 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:25.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.709+0000 7f5772a00640 1 --2- 172.21.15.28:0/958080766 >> v2:172.21.15.81:3300/0 conn(0x7f577406dbf0 0x7f5774158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:25.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.709+0000 7f5773e00640 1 --2- 172.21.15.28:0/958080766 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f57740c5720 0x7f5774159000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:25.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.710+0000 7f5773400640 1 -- 172.21.15.28:0/958080766 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f57740c5720 msgr2=0x7f5774159000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:25.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.710+0000 7f5773400640 1 --2- 172.21.15.28:0/958080766 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f57740c5720 0x7f5774159000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:25.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.710+0000 7f5773400640 1 -- 172.21.15.28:0/958080766 >> v2:172.21.15.81:3300/0 conn(0x7f577406dbf0 msgr2=0x7f5774158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:25.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.710+0000 7f5773400640 1 --2- 172.21.15.28:0/958080766 >> v2:172.21.15.81:3300/0 conn(0x7f577406dbf0 0x7f5774158ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:25.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.710+0000 7f5773400640 1 -- 172.21.15.28:0/958080766 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57740c77a0 con 0x7f57740aff90 2024-02-21T23:10:25.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.711+0000 7f5773400640 1 --2- 172.21.15.28:0/958080766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57740aff90 0x7f57740c51e0 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f57740cb8c0 tx=0x7f576402b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=251bdf727d66d7f6 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:25.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.711+0000 7f5772000640 1 -- 172.21.15.28:0/958080766 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5764035020 con 0x7f57740aff90 2024-02-21T23:10:25.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.711+0000 7f5772000640 1 -- 172.21.15.28:0/958080766 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5764031880 con 0x7f57740aff90 2024-02-21T23:10:25.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.711+0000 7f5772000640 1 -- 172.21.15.28:0/958080766 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5764031c50 con 0x7f57740aff90 2024-02-21T23:10:25.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.712+0000 7f5779000640 1 -- 172.21.15.28:0/958080766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57740aff90 msgr2=0x7f57740c51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:25.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.712+0000 7f5779000640 1 --2- 172.21.15.28:0/958080766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57740aff90 0x7f57740c51e0 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f57740cb8c0 tx=0x7f576402b920 comp rx=0 tx=0).stop 2024-02-21T23:10:25.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.712+0000 7f5779000640 1 -- 172.21.15.28:0/958080766 shutdown_connections 2024-02-21T23:10:25.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.712+0000 7f5779000640 1 --2- 172.21.15.28:0/958080766 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f57740c5720 0x7f5774159000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:25.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.712+0000 7f5779000640 1 --2- 172.21.15.28:0/958080766 >> v2:172.21.15.81:3300/0 conn(0x7f577406dbf0 0x7f5774158ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:25.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.712+0000 7f5779000640 1 --2- 172.21.15.28:0/958080766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57740aff90 0x7f57740c51e0 unknown :-1 s=CLOSED pgs=1787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:25.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.712+0000 7f5779000640 1 -- 172.21.15.28:0/958080766 >> 172.21.15.28:0/958080766 conn(0x7f57740ac370 msgr2=0x7f57740ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:25.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.712+0000 7f5779000640 1 -- 172.21.15.28:0/958080766 shutdown_connections 2024-02-21T23:10:25.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.712+0000 7f5779000640 1 -- 172.21.15.28:0/958080766 wait complete. 2024-02-21T23:10:25.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.714+0000 7f5779000640 1 Processor -- start 2024-02-21T23:10:25.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.714+0000 7f5779000640 1 -- start start 2024-02-21T23:10:25.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.715+0000 7f5779000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f577406dbf0 0x7f57740c3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:25.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.715+0000 7f5779000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57740aff90 0x7f57740bcac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:25.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.715+0000 7f5779000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f57740c5720 0x7f57740bd000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:25.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.715+0000 7f5779000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57740c8760 con 0x7f57740aff90 2024-02-21T23:10:25.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.715+0000 7f5779000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57740be0c0 con 0x7f577406dbf0 2024-02-21T23:10:25.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.715+0000 7f5779000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57740c88e0 con 0x7f57740c5720 2024-02-21T23:10:25.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.715+0000 7f5772a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57740aff90 0x7f57740bcac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:25.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.715+0000 7f5772a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57740aff90 0x7f57740bcac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39266/0 (socket says 172.21.15.28:39266) 2024-02-21T23:10:25.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.715+0000 7f5772a00640 1 -- 172.21.15.28:0/1321729521 learned_addr learned my addr 172.21.15.28:0/1321729521 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:25.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.716+0000 7f5773e00640 1 --2- 172.21.15.28:0/1321729521 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f57740c5720 0x7f57740bd000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:25.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.716+0000 7f5773400640 1 --2- 172.21.15.28:0/1321729521 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f577406dbf0 0x7f57740c3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:25.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.716+0000 7f5772a00640 1 -- 172.21.15.28:0/1321729521 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f57740c5720 msgr2=0x7f57740bd000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:25.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.716+0000 7f5772a00640 1 --2- 172.21.15.28:0/1321729521 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f57740c5720 0x7f57740bd000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:25.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.716+0000 7f5772a00640 1 -- 172.21.15.28:0/1321729521 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f577406dbf0 msgr2=0x7f57740c3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:25.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.716+0000 7f5772a00640 1 --2- 172.21.15.28:0/1321729521 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f577406dbf0 0x7f57740c3ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:25.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.716+0000 7f5772a00640 1 -- 172.21.15.28:0/1321729521 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57740bdba0 con 0x7f57740aff90 2024-02-21T23:10:25.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.716+0000 7f5772a00640 1 --2- 172.21.15.28:0/1321729521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57740aff90 0x7f57740bcac0 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f576000e890 tx=0x7f576000ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:25.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.716+0000 7f576b400640 1 -- 172.21.15.28:0/1321729521 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5760017070 con 0x7f57740aff90 2024-02-21T23:10:25.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.717+0000 7f576b400640 1 -- 172.21.15.28:0/1321729521 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f576000c940 con 0x7f57740aff90 2024-02-21T23:10:25.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.717+0000 7f576b400640 1 -- 172.21.15.28:0/1321729521 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f576000cd10 con 0x7f57740aff90 2024-02-21T23:10:25.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.717+0000 7f5779000640 1 -- 172.21.15.28:0/1321729521 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57741eb090 con 0x7f57740aff90 2024-02-21T23:10:25.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.717+0000 7f5779000640 1 -- 172.21.15.28:0/1321729521 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57740c19d0 con 0x7f57740aff90 2024-02-21T23:10:25.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.718+0000 7f5779000640 1 -- 172.21.15.28:0/1321729521 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f577404f290 con 0x7f57740aff90 2024-02-21T23:10:25.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.719+0000 7f576b400640 1 -- 172.21.15.28:0/1321729521 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f576001a070 con 0x7f57740aff90 2024-02-21T23:10:25.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.720+0000 7f576b400640 1 --2- 172.21.15.28:0/1321729521 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f575406b4d0 0x7f575406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:25.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.720+0000 7f576b400640 1 -- 172.21.15.28:0/1321729521 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 7129+0+0 (secure 0 0 0) 0x7f576008aaf0 con 0x7f57740aff90 2024-02-21T23:10:25.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.720+0000 7f5773400640 1 --2- 172.21.15.28:0/1321729521 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f575406b4d0 0x7f575406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:25.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.721+0000 7f5773400640 1 --2- 172.21.15.28:0/1321729521 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f575406b4d0 0x7f575406d990 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f5764036000 tx=0x7f576403c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:25.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.724+0000 7f576b400640 1 -- 172.21.15.28:0/1321729521 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f57600593d0 con 0x7f57740aff90 2024-02-21T23:10:25.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.965+0000 7f5779000640 1 -- 172.21.15.28:0/1321729521 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f577406d9d0 con 0x7f57740aff90 2024-02-21T23:10:25.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.967+0000 7f576b400640 1 -- 172.21.15.28:0/1321729521 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 171 v171) v1 ==== 95+0+6197 (secure 0 0 0) 0x7f576005f0d0 con 0x7f57740aff90 2024-02-21T23:10:25.955 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:10:25.955 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":171,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9141,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2895215997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2895215997},{"type":"v1","addr":"172.21.15.28:6835","nonce":2895215997}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":166},{"gid":9153,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/2178509171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2178509171},{"type":"v1","addr":"172.21.15.28:6833","nonce":2178509171}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":169}],"filesystems":[{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:09:53.730296+0000","modified":"2024-02-21T23:10:21.591594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":156,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6392},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6392":{"gid":6392,"name":"d","rank":0,"incarnation":165,"state":"up:active","state_seq":11,"addr":"172.21.15.81:6837/521625945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":521625945},{"type":"v1","addr":"172.21.15.81:6837","nonce":521625945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11},{"mdsmap":{"epoch":171,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:10:05.843850+0000","modified":"2024-02-21T23:10:23.599068+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":157,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6389},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6389":{"gid":6389,"name":"b","rank":0,"incarnation":168,"state":"up:active","state_seq":12,"addr":"172.21.15.81:6835/1320495906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1320495906},{"type":"v1","addr":"172.21.15.81:6835","nonce":1320495906}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2024-02-21T23:10:25.955 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 171 2024-02-21T23:10:25.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.969+0000 7f5779000640 1 -- 172.21.15.28:0/1321729521 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f575406b4d0 msgr2=0x7f575406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:25.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.969+0000 7f5779000640 1 --2- 172.21.15.28:0/1321729521 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f575406b4d0 0x7f575406d990 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f5764036000 tx=0x7f576403c040 comp rx=0 tx=0).stop 2024-02-21T23:10:25.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.970+0000 7f5779000640 1 -- 172.21.15.28:0/1321729521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57740aff90 msgr2=0x7f57740bcac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:25.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.970+0000 7f5779000640 1 --2- 172.21.15.28:0/1321729521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57740aff90 0x7f57740bcac0 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f576000e890 tx=0x7f576000ed50 comp rx=0 tx=0).stop 2024-02-21T23:10:25.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.970+0000 7f5779000640 1 -- 172.21.15.28:0/1321729521 shutdown_connections 2024-02-21T23:10:25.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.970+0000 7f5779000640 1 --2- 172.21.15.28:0/1321729521 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f575406b4d0 0x7f575406d990 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:25.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.970+0000 7f5779000640 1 --2- 172.21.15.28:0/1321729521 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f57740c5720 0x7f57740bd000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:25.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.970+0000 7f5779000640 1 --2- 172.21.15.28:0/1321729521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57740aff90 0x7f57740bcac0 unknown :-1 s=CLOSED pgs=1788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:25.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.970+0000 7f5779000640 1 --2- 172.21.15.28:0/1321729521 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f577406dbf0 0x7f57740c3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:25.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.970+0000 7f5779000640 1 -- 172.21.15.28:0/1321729521 >> 172.21.15.28:0/1321729521 conn(0x7f57740ac370 msgr2=0x7f57740ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:25.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.970+0000 7f5779000640 1 -- 172.21.15.28:0/1321729521 shutdown_connections 2024-02-21T23:10:25.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:25.970+0000 7f5779000640 1 -- 172.21.15.28:0/1321729521 wait complete. 2024-02-21T23:10:25.978 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:10:26.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.209+0000 7f62bc800640 1 Processor -- start 2024-02-21T23:10:26.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.209+0000 7f62bc800640 1 -- start start 2024-02-21T23:10:26.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.211+0000 7f62bc800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f62b40c11f0 0x7f62b4153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:26.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.211+0000 7f62bc800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f62b415b6d0 0x7f62b415dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:26.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.211+0000 7f62bc800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62b4152e70 con 0x7f62b40c15d0 2024-02-21T23:10:26.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.211+0000 7f62bc800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62b40c1a60 con 0x7f62b415b6d0 2024-02-21T23:10:26.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.211+0000 7f62bc800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62b40a5ec0 con 0x7f62b40c11f0 2024-02-21T23:10:26.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.211+0000 7f62ba800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f62b415b6d0 0x7f62b415dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:26.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.211+0000 7f62ba800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f62b415b6d0 0x7f62b415dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39276/0 (socket says 172.21.15.28:39276) 2024-02-21T23:10:26.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.211+0000 7f62ba800640 1 -- 172.21.15.28:0/2802830975 learned_addr learned my addr 172.21.15.28:0/2802830975 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:26.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.212+0000 7f62b9e00640 1 --2- 172.21.15.28:0/2802830975 >> v2:172.21.15.81:3300/0 conn(0x7f62b40c11f0 0x7f62b4153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:26.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.212+0000 7f62ba800640 1 -- 172.21.15.28:0/2802830975 >> v2:172.21.15.81:3300/0 conn(0x7f62b40c11f0 msgr2=0x7f62b4153950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:26.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.212+0000 7f62ba800640 1 --2- 172.21.15.28:0/2802830975 >> v2:172.21.15.81:3300/0 conn(0x7f62b40c11f0 0x7f62b4153950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:26.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.212+0000 7f62ba800640 1 -- 172.21.15.28:0/2802830975 >> v1:172.21.15.81:6789/0 conn(0x7f62b40c15d0 legacy=0x7f62b406dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:26.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.212+0000 7f62ba800640 1 -- 172.21.15.28:0/2802830975 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62b4153150 con 0x7f62b415b6d0 2024-02-21T23:10:26.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.213+0000 7f62ba800640 1 --2- 172.21.15.28:0/2802830975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62b415b6d0 0x7f62b415dac0 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7f62b40c7590 tx=0x7f62b002b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=150817c63881ca1b server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:26.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.213+0000 7f62b8a00640 1 -- 172.21.15.28:0/2802830975 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b0038070 con 0x7f62b415b6d0 2024-02-21T23:10:26.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.213+0000 7f62b8a00640 1 -- 172.21.15.28:0/2802830975 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62b002f050 con 0x7f62b415b6d0 2024-02-21T23:10:26.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.213+0000 7f62b8a00640 1 -- 172.21.15.28:0/2802830975 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b0031970 con 0x7f62b415b6d0 2024-02-21T23:10:26.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.214+0000 7f62bc800640 1 -- 172.21.15.28:0/2802830975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62b415b6d0 msgr2=0x7f62b415dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:26.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.214+0000 7f62bc800640 1 --2- 172.21.15.28:0/2802830975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62b415b6d0 0x7f62b415dac0 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7f62b40c7590 tx=0x7f62b002b7f0 comp rx=0 tx=0).stop 2024-02-21T23:10:26.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.214+0000 7f62bc800640 1 -- 172.21.15.28:0/2802830975 shutdown_connections 2024-02-21T23:10:26.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.214+0000 7f62bc800640 1 --2- 172.21.15.28:0/2802830975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62b415b6d0 0x7f62b415dac0 unknown :-1 s=CLOSED pgs=1789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:26.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.214+0000 7f62bc800640 1 --2- 172.21.15.28:0/2802830975 >> v2:172.21.15.81:3300/0 conn(0x7f62b40c11f0 0x7f62b4153950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:26.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.214+0000 7f62bc800640 1 -- 172.21.15.28:0/2802830975 >> 172.21.15.28:0/2802830975 conn(0x7f62b40c67f0 msgr2=0x7f62b40c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:26.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.214+0000 7f62bc800640 1 -- 172.21.15.28:0/2802830975 shutdown_connections 2024-02-21T23:10:26.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.214+0000 7f62bc800640 1 -- 172.21.15.28:0/2802830975 wait complete. 2024-02-21T23:10:26.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.216+0000 7f62bc800640 1 Processor -- start 2024-02-21T23:10:26.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.216+0000 7f62bc800640 1 -- start start 2024-02-21T23:10:26.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.217+0000 7f62bc800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62b40c11f0 0x7f62b414d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:26.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.217+0000 7f62bc800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f62b40c15d0 0x7f62b414e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:26.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.217+0000 7f62bc800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62b415b6d0 0x7f62b4143520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:26.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.217+0000 7f62bc800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62b40c4c00 con 0x7f62b415b6d0 2024-02-21T23:10:26.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.217+0000 7f62bc800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62b414d1c0 con 0x7f62b40c11f0 2024-02-21T23:10:26.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.217+0000 7f62bc800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62b40a6110 con 0x7f62b40c15d0 2024-02-21T23:10:26.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.217+0000 7f62ba800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62b415b6d0 0x7f62b4143520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:26.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.217+0000 7f62ba800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62b415b6d0 0x7f62b4143520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39282/0 (socket says 172.21.15.28:39282) 2024-02-21T23:10:26.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.217+0000 7f62ba800640 1 -- 172.21.15.28:0/3508680914 learned_addr learned my addr 172.21.15.28:0/3508680914 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:26.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.218+0000 7f62b9e00640 1 --2- 172.21.15.28:0/3508680914 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62b40c11f0 0x7f62b414d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:26.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.218+0000 7f62b9400640 1 --2- 172.21.15.28:0/3508680914 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f62b40c15d0 0x7f62b414e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:26.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.218+0000 7f62ba800640 1 -- 172.21.15.28:0/3508680914 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f62b40c15d0 msgr2=0x7f62b414e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:26.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.218+0000 7f62ba800640 1 --2- 172.21.15.28:0/3508680914 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f62b40c15d0 0x7f62b414e8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:26.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.218+0000 7f62ba800640 1 -- 172.21.15.28:0/3508680914 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62b40c11f0 msgr2=0x7f62b414d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:26.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.218+0000 7f62ba800640 1 --2- 172.21.15.28:0/3508680914 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62b40c11f0 0x7f62b414d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:26.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.218+0000 7f62ba800640 1 -- 172.21.15.28:0/3508680914 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62b4146080 con 0x7f62b415b6d0 2024-02-21T23:10:26.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.218+0000 7f62ba800640 1 --2- 172.21.15.28:0/3508680914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62b415b6d0 0x7f62b4143520 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f62b0009960 tx=0x7f62b00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:26.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.219+0000 7f62aa000640 1 -- 172.21.15.28:0/3508680914 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b0042070 con 0x7f62b415b6d0 2024-02-21T23:10:26.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.219+0000 7f62aa000640 1 -- 172.21.15.28:0/3508680914 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62b0033850 con 0x7f62b415b6d0 2024-02-21T23:10:26.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.219+0000 7f62aa000640 1 -- 172.21.15.28:0/3508680914 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b0038040 con 0x7f62b415b6d0 2024-02-21T23:10:26.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.219+0000 7f62bc800640 1 -- 172.21.15.28:0/3508680914 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62b414dc50 con 0x7f62b415b6d0 2024-02-21T23:10:26.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.219+0000 7f62bc800640 1 -- 172.21.15.28:0/3508680914 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62b41450c0 con 0x7f62b415b6d0 2024-02-21T23:10:26.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.220+0000 7f62bc800640 1 -- 172.21.15.28:0/3508680914 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62b40c65d0 con 0x7f62b415b6d0 2024-02-21T23:10:26.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.221+0000 7f62aa000640 1 -- 172.21.15.28:0/3508680914 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f62b0045070 con 0x7f62b415b6d0 2024-02-21T23:10:26.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.222+0000 7f62aa000640 1 --2- 172.21.15.28:0/3508680914 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f629806b4d0 0x7f629806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:26.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.222+0000 7f62aa000640 1 -- 172.21.15.28:0/3508680914 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 7129+0+0 (secure 0 0 0) 0x7f62b00ac3a0 con 0x7f62b415b6d0 2024-02-21T23:10:26.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.222+0000 7f62b9e00640 1 --2- 172.21.15.28:0/3508680914 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f629806b4d0 0x7f629806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:26.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.224+0000 7f62b9e00640 1 --2- 172.21.15.28:0/3508680914 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f629806b4d0 0x7f629806d990 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f62b4150970 tx=0x7f62ac0039b0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:26.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.226+0000 7f62aa000640 1 -- 172.21.15.28:0/3508680914 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f62b40c65d0 con 0x7f62b415b6d0 2024-02-21T23:10:26.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.431+0000 7f62bc800640 1 -- 172.21.15.28:0/3508680914 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f62b4067f30 con 0x7f62b415b6d0 2024-02-21T23:10:26.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.434+0000 7f62aa000640 1 -- 172.21.15.28:0/3508680914 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v157) v1 ==== 74+0+20801 (secure 0 0 0) 0x7f62b0031e90 con 0x7f62b415b6d0 2024-02-21T23:10:26.422 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:10:26.422 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":157,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:10:21.581993+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:09:51.634291+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"150","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-02-21T23:09:52.736252+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"151","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs2_metadata","create_time":"2024-02-21T23:10:04.095451+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"154","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs2_data","create_time":"2024-02-21T23:10:04.836982+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6833/2975378719":"2024-02-22T23:10:21.581948+0000","172.21.15.28:6832/2975378719":"2024-02-22T23:10:21.581948+0000","172.21.15.28:6835/2928639298":"2024-02-22T23:10:19.550812+0000","172.21.15.28:6834/2928639298":"2024-02-22T23:10:19.550812+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:10:26.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.437+0000 7f62bc800640 1 -- 172.21.15.28:0/3508680914 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f629806b4d0 msgr2=0x7f629806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:26.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.437+0000 7f62bc800640 1 --2- 172.21.15.28:0/3508680914 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f629806b4d0 0x7f629806d990 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f62b4150970 tx=0x7f62ac0039b0 comp rx=0 tx=0).stop 2024-02-21T23:10:26.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.437+0000 7f62bc800640 1 -- 172.21.15.28:0/3508680914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62b415b6d0 msgr2=0x7f62b4143520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:26.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.437+0000 7f62bc800640 1 --2- 172.21.15.28:0/3508680914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62b415b6d0 0x7f62b4143520 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f62b0009960 tx=0x7f62b00046c0 comp rx=0 tx=0).stop 2024-02-21T23:10:26.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.437+0000 7f62bc800640 1 -- 172.21.15.28:0/3508680914 shutdown_connections 2024-02-21T23:10:26.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.437+0000 7f62bc800640 1 --2- 172.21.15.28:0/3508680914 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f629806b4d0 0x7f629806d990 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:26.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.437+0000 7f62bc800640 1 --2- 172.21.15.28:0/3508680914 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f62b40c15d0 0x7f62b414e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:26.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.437+0000 7f62bc800640 1 --2- 172.21.15.28:0/3508680914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62b415b6d0 0x7f62b4143520 unknown :-1 s=CLOSED pgs=1790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:26.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.437+0000 7f62bc800640 1 --2- 172.21.15.28:0/3508680914 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62b40c11f0 0x7f62b414d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:26.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.437+0000 7f62bc800640 1 -- 172.21.15.28:0/3508680914 >> 172.21.15.28:0/3508680914 conn(0x7f62b40c67f0 msgr2=0x7f62b40a63b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:26.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.437+0000 7f62bc800640 1 -- 172.21.15.28:0/3508680914 shutdown_connections 2024-02-21T23:10:26.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.437+0000 7f62bc800640 1 -- 172.21.15.28:0/3508680914 wait complete. 2024-02-21T23:10:26.446 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-21T23:02:30.728329+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 23, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-21T23:09:51.634291+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '150', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 24, 'pool_name': 'cephfs_data', 'create_time': '2024-02-21T23:09:52.736252+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '151', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 25, 'pool_name': 'cephfs2_metadata', 'create_time': '2024-02-21T23:10:04.095451+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '154', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs2'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 26, 'pool_name': 'cephfs2_data', 'create_time': '2024-02-21T23:10:04.836982+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '155', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs2'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-21T23:10:26.447 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-21T23:10:26.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.691+0000 7fd9eb000640 1 Processor -- start 2024-02-21T23:10:26.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.691+0000 7fd9eb000640 1 -- start start 2024-02-21T23:10:26.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.692+0000 7fd9eb000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd9e40c5f50 0x7fd9e414fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:26.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.693+0000 7fd9eb000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fd9e4150530 0x7fd9e4156500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:26.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.693+0000 7fd9eb000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9e41509e0 con 0x7fd9e406b0e0 2024-02-21T23:10:26.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.693+0000 7fd9eb000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9e4153bc0 con 0x7fd9e4150530 2024-02-21T23:10:26.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.693+0000 7fd9eb000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e4152fb0 con 0x7fd9e40c5f50 2024-02-21T23:10:26.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.693+0000 7fd9e3e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd9e406b0e0 0x7fd9e40c5840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:33368/0 (socket says 172.21.15.28:33368) 2024-02-21T23:10:26.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.693+0000 7fd9e3e00640 1 -- 172.21.15.28:0/2757041809 learned_addr learned my addr 172.21.15.28:0/2757041809 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:10:26.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.693+0000 7fd9e3400640 1 --2- 172.21.15.28:0/2757041809 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd9e40c5f50 0x7fd9e414fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:26.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.693+0000 7fd9e2a00640 1 -- 172.21.15.28:0/2757041809 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 104735782 0 0) 0x7fd9e41509e0 con 0x7fd9e406b0e0 2024-02-21T23:10:26.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.693+0000 7fd9e9000640 1 --2- 172.21.15.28:0/2757041809 >> v2:172.21.15.81:3300/0 conn(0x7fd9e4150530 0x7fd9e4156500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:26.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.693+0000 7fd9e2a00640 1 -- 172.21.15.28:0/2757041809 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9c8003520 con 0x7fd9e406b0e0 2024-02-21T23:10:26.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.694+0000 7fd9e3400640 1 -- 172.21.15.28:0/2757041809 >> v2:172.21.15.81:3300/0 conn(0x7fd9e4150530 msgr2=0x7fd9e4156500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:26.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.694+0000 7fd9e2a00640 1 -- 172.21.15.28:0/2757041809 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 540530012 0 0) 0x7fd9c8003520 con 0x7fd9e406b0e0 2024-02-21T23:10:26.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.694+0000 7fd9e3400640 1 --2- 172.21.15.28:0/2757041809 >> v2:172.21.15.81:3300/0 conn(0x7fd9e4150530 0x7fd9e4156500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:26.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.694+0000 7fd9e3400640 1 -- 172.21.15.28:0/2757041809 >> v1:172.21.15.28:6789/0 conn(0x7fd9e406b0e0 legacy=0x7fd9e40c5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:26.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.694+0000 7fd9e3400640 1 -- 172.21.15.28:0/2757041809 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9e4153290 con 0x7fd9e40c5f50 2024-02-21T23:10:26.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.695+0000 7fd9e3400640 1 --2- 172.21.15.28:0/2757041809 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd9e40c5f50 0x7fd9e414fff0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd9d00097a0 tx=0x7fd9d002b710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b81e8eb7dbe0333d server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:26.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.695+0000 7fd9e2a00640 1 -- 172.21.15.28:0/2757041809 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d0037070 con 0x7fd9e40c5f50 2024-02-21T23:10:26.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.696+0000 7fd9e2a00640 1 -- 172.21.15.28:0/2757041809 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9d0030510 con 0x7fd9e40c5f50 2024-02-21T23:10:26.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.696+0000 7fd9e2a00640 1 -- 172.21.15.28:0/2757041809 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d4001670 con 0x7fd9e40c5f50 2024-02-21T23:10:26.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.696+0000 7fd9eb000640 1 -- 172.21.15.28:0/2757041809 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd9e40c5f50 msgr2=0x7fd9e414fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:26.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.696+0000 7fd9eb000640 1 --2- 172.21.15.28:0/2757041809 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd9e40c5f50 0x7fd9e414fff0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd9d00097a0 tx=0x7fd9d002b710 comp rx=0 tx=0).stop 2024-02-21T23:10:26.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.696+0000 7fd9eb000640 1 -- 172.21.15.28:0/2757041809 shutdown_connections 2024-02-21T23:10:26.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.696+0000 7fd9eb000640 1 --2- 172.21.15.28:0/2757041809 >> v2:172.21.15.81:3300/0 conn(0x7fd9e4150530 0x7fd9e4156500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:26.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.696+0000 7fd9eb000640 1 --2- 172.21.15.28:0/2757041809 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd9e40c5f50 0x7fd9e414fff0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:26.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.696+0000 7fd9eb000640 1 -- 172.21.15.28:0/2757041809 >> 172.21.15.28:0/2757041809 conn(0x7fd9e40a85a0 msgr2=0x7fd9e40a8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:26.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.696+0000 7fd9eb000640 1 -- 172.21.15.28:0/2757041809 shutdown_connections 2024-02-21T23:10:26.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.696+0000 7fd9eb000640 1 -- 172.21.15.28:0/2757041809 wait complete. 2024-02-21T23:10:26.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.698+0000 7fd9eb000640 1 Processor -- start 2024-02-21T23:10:26.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.698+0000 7fd9eb000640 1 -- start start 2024-02-21T23:10:26.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.699+0000 7fd9eb000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9e406b0e0 0x7fd9e40c51a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:26.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.699+0000 7fd9eb000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd9e40c5f50 0x7fd9e4147ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:26.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.699+0000 7fd9eb000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd9e4150530 0x7fd9e4147ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:26.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.699+0000 7fd9eb000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e406b4c0 con 0x7fd9e406b0e0 2024-02-21T23:10:26.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.699+0000 7fd9eb000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e4153850 con 0x7fd9e4150530 2024-02-21T23:10:26.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.699+0000 7fd9eb000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e406b640 con 0x7fd9e40c5f50 2024-02-21T23:10:26.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.699+0000 7fd9e3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9e406b0e0 0x7fd9e40c51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:26.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.699+0000 7fd9e3400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd9e40c5f50 0x7fd9e4147ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:26.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.699+0000 7fd9e3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9e406b0e0 0x7fd9e40c51a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39294/0 (socket says 172.21.15.28:39294) 2024-02-21T23:10:26.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.699+0000 7fd9e3e00640 1 -- 172.21.15.28:0/2277497534 learned_addr learned my addr 172.21.15.28:0/2277497534 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:26.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.700+0000 7fd9e9000640 1 --2- 172.21.15.28:0/2277497534 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd9e4150530 0x7fd9e4147ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:26.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.700+0000 7fd9e3e00640 1 -- 172.21.15.28:0/2277497534 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd9e40c5f50 msgr2=0x7fd9e4147ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:26.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.700+0000 7fd9e3e00640 1 --2- 172.21.15.28:0/2277497534 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd9e40c5f50 0x7fd9e4147ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:26.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.700+0000 7fd9e3e00640 1 -- 172.21.15.28:0/2277497534 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd9e4150530 msgr2=0x7fd9e4147ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:26.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.700+0000 7fd9e3e00640 1 --2- 172.21.15.28:0/2277497534 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd9e4150530 0x7fd9e4147ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:26.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.700+0000 7fd9e3e00640 1 -- 172.21.15.28:0/2277497534 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9e4148530 con 0x7fd9e406b0e0 2024-02-21T23:10:26.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.700+0000 7fd9e3e00640 1 --2- 172.21.15.28:0/2277497534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9e406b0e0 0x7fd9e40c51a0 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7fd9d400c730 tx=0x7fd9d400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:26.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.700+0000 7fd9dbe00640 1 -- 172.21.15.28:0/2277497534 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d4017070 con 0x7fd9e406b0e0 2024-02-21T23:10:26.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.701+0000 7fd9dbe00640 1 -- 172.21.15.28:0/2277497534 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9d4005ad0 con 0x7fd9e406b0e0 2024-02-21T23:10:26.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.701+0000 7fd9dbe00640 1 -- 172.21.15.28:0/2277497534 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d4007070 con 0x7fd9e406b0e0 2024-02-21T23:10:26.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.701+0000 7fd9eb000640 1 -- 172.21.15.28:0/2277497534 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9e40ab540 con 0x7fd9e406b0e0 2024-02-21T23:10:26.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.701+0000 7fd9eb000640 1 -- 172.21.15.28:0/2277497534 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9e40aa780 con 0x7fd9e406b0e0 2024-02-21T23:10:26.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.702+0000 7fd9eb000640 1 -- 172.21.15.28:0/2277497534 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9e406c560 con 0x7fd9e406b0e0 2024-02-21T23:10:26.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.703+0000 7fd9dbe00640 1 -- 172.21.15.28:0/2277497534 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd9d401a070 con 0x7fd9e406b0e0 2024-02-21T23:10:26.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.704+0000 7fd9dbe00640 1 --2- 172.21.15.28:0/2277497534 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd9c406b500 0x7fd9c406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:26.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.704+0000 7fd9dbe00640 1 -- 172.21.15.28:0/2277497534 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 7216+0+0 (secure 0 0 0) 0x7fd9d4089c30 con 0x7fd9e406b0e0 2024-02-21T23:10:26.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.704+0000 7fd9e3400640 1 --2- 172.21.15.28:0/2277497534 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd9c406b500 0x7fd9c406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:26.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.705+0000 7fd9e3400640 1 --2- 172.21.15.28:0/2277497534 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd9c406b500 0x7fd9c406d9c0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fd9d0034000 tx=0x7fd9d0008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:26.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.708+0000 7fd9dbe00640 1 -- 172.21.15.28:0/2277497534 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd9e406c560 con 0x7fd9e406b0e0 2024-02-21T23:10:26.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.905+0000 7fd9eb000640 1 -- 172.21.15.28:0/2277497534 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd9e4067f30 con 0x7fd9e406b0e0 2024-02-21T23:10:26.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.906+0000 7fd9dbe00640 1 -- 172.21.15.28:0/2277497534 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v158) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd9d4057e90 con 0x7fd9e406b0e0 2024-02-21T23:10:26.894 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-02-21T23:10:26.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.908+0000 7fd9eb000640 1 -- 172.21.15.28:0/2277497534 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd9c406b500 msgr2=0x7fd9c406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:26.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.909+0000 7fd9eb000640 1 --2- 172.21.15.28:0/2277497534 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd9c406b500 0x7fd9c406d9c0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fd9d0034000 tx=0x7fd9d0008030 comp rx=0 tx=0).stop 2024-02-21T23:10:26.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.909+0000 7fd9eb000640 1 -- 172.21.15.28:0/2277497534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9e406b0e0 msgr2=0x7fd9e40c51a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:26.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.909+0000 7fd9eb000640 1 --2- 172.21.15.28:0/2277497534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9e406b0e0 0x7fd9e40c51a0 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7fd9d400c730 tx=0x7fd9d400cbf0 comp rx=0 tx=0).stop 2024-02-21T23:10:26.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.909+0000 7fd9eb000640 1 -- 172.21.15.28:0/2277497534 shutdown_connections 2024-02-21T23:10:26.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.909+0000 7fd9eb000640 1 --2- 172.21.15.28:0/2277497534 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd9c406b500 0x7fd9c406d9c0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:26.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.909+0000 7fd9eb000640 1 --2- 172.21.15.28:0/2277497534 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd9e40c5f50 0x7fd9e4147ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:26.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.909+0000 7fd9eb000640 1 --2- 172.21.15.28:0/2277497534 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd9e4150530 0x7fd9e4147ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:26.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.909+0000 7fd9eb000640 1 --2- 172.21.15.28:0/2277497534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9e406b0e0 0x7fd9e40c51a0 unknown :-1 s=CLOSED pgs=1792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:26.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.909+0000 7fd9eb000640 1 -- 172.21.15.28:0/2277497534 >> 172.21.15.28:0/2277497534 conn(0x7fd9e40a85a0 msgr2=0x7fd9e40a8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:26.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.909+0000 7fd9eb000640 1 -- 172.21.15.28:0/2277497534 shutdown_connections 2024-02-21T23:10:26.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:26.909+0000 7fd9eb000640 1 -- 172.21.15.28:0/2277497534 wait complete. 2024-02-21T23:10:26.917 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-21T23:10:27.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.159+0000 7ffbd5600640 1 Processor -- start 2024-02-21T23:10:27.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.160+0000 7ffbd5600640 1 -- start start 2024-02-21T23:10:27.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.161+0000 7ffbd5600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ffbd006dbc0 0x7ffbd006c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:27.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.161+0000 7ffbd5600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7ffbd013e520 0x7ffbd006c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:27.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.161+0000 7ffbd5600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbd0158550 con 0x7ffbd013e0c0 2024-02-21T23:10:27.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.161+0000 7ffbd5600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbd00c89a0 con 0x7ffbd006dbc0 2024-02-21T23:10:27.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.161+0000 7ffbd5600640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbd01587d0 con 0x7ffbd013e520 2024-02-21T23:10:27.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.161+0000 7ffbc7e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ffbd006dbc0 0x7ffbd006c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:27.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.161+0000 7ffbcfe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ffbd013e0c0 0x7ffbd00c71b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:33384/0 (socket says 172.21.15.28:33384) 2024-02-21T23:10:27.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.162+0000 7ffbcfe00640 1 -- 172.21.15.28:0/3396392381 learned_addr learned my addr 172.21.15.28:0/3396392381 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:10:27.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.162+0000 7ffbd4c00640 1 --2- 172.21.15.28:0/3396392381 >> v2:172.21.15.81:3300/0 conn(0x7ffbd013e520 0x7ffbd006c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:27.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.162+0000 7ffbc7e00640 1 -- 172.21.15.28:0/3396392381 >> v2:172.21.15.81:3300/0 conn(0x7ffbd013e520 msgr2=0x7ffbd006c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:27.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.162+0000 7ffbc7e00640 1 --2- 172.21.15.28:0/3396392381 >> v2:172.21.15.81:3300/0 conn(0x7ffbd013e520 0x7ffbd006c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:27.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.162+0000 7ffbc7e00640 1 -- 172.21.15.28:0/3396392381 >> v1:172.21.15.28:6789/0 conn(0x7ffbd013e0c0 legacy=0x7ffbd00c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:27.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.162+0000 7ffbc7e00640 1 -- 172.21.15.28:0/3396392381 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbd0158ab0 con 0x7ffbd006dbc0 2024-02-21T23:10:27.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.163+0000 7ffbc7e00640 1 --2- 172.21.15.28:0/3396392381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbd006dbc0 0x7ffbd006c280 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7ffbd00c8260 tx=0x7ffbb802b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ab852e7abb62321 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:27.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.163+0000 7ffbcf400640 1 -- 172.21.15.28:0/3396392381 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb8038070 con 0x7ffbd006dbc0 2024-02-21T23:10:27.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.164+0000 7ffbcf400640 1 -- 172.21.15.28:0/3396392381 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbb8031520 con 0x7ffbd006dbc0 2024-02-21T23:10:27.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.164+0000 7ffbcf400640 1 -- 172.21.15.28:0/3396392381 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb80318f0 con 0x7ffbd006dbc0 2024-02-21T23:10:27.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.164+0000 7ffbd5600640 1 -- 172.21.15.28:0/3396392381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbd006dbc0 msgr2=0x7ffbd006c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:27.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.164+0000 7ffbd5600640 1 --2- 172.21.15.28:0/3396392381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbd006dbc0 0x7ffbd006c280 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7ffbd00c8260 tx=0x7ffbb802b7c0 comp rx=0 tx=0).stop 2024-02-21T23:10:27.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.164+0000 7ffbd5600640 1 -- 172.21.15.28:0/3396392381 shutdown_connections 2024-02-21T23:10:27.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.164+0000 7ffbd5600640 1 --2- 172.21.15.28:0/3396392381 >> v2:172.21.15.81:3300/0 conn(0x7ffbd013e520 0x7ffbd006c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:27.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.164+0000 7ffbd5600640 1 --2- 172.21.15.28:0/3396392381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbd006dbc0 0x7ffbd006c280 unknown :-1 s=CLOSED pgs=1794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:27.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.164+0000 7ffbd5600640 1 -- 172.21.15.28:0/3396392381 >> 172.21.15.28:0/3396392381 conn(0x7ffbd0073110 msgr2=0x7ffbd00ac490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:27.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.164+0000 7ffbd5600640 1 -- 172.21.15.28:0/3396392381 shutdown_connections 2024-02-21T23:10:27.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.164+0000 7ffbd5600640 1 -- 172.21.15.28:0/3396392381 wait complete. 2024-02-21T23:10:27.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.166+0000 7ffbd5600640 1 Processor -- start 2024-02-21T23:10:27.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.166+0000 7ffbd5600640 1 -- start start 2024-02-21T23:10:27.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.167+0000 7ffbd5600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbd006dbc0 0x7ffbd01ebb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:27.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.167+0000 7ffbd5600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffbd013e0c0 0x7ffbd0133a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:27.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.167+0000 7ffbd5600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffbd013e520 0x7ffbd00ab5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:27.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.167+0000 7ffbd5600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd00c78c0 con 0x7ffbd006dbc0 2024-02-21T23:10:27.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.167+0000 7ffbd5600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd0159070 con 0x7ffbd013e0c0 2024-02-21T23:10:27.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.167+0000 7ffbd5600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd00c7a40 con 0x7ffbd013e520 2024-02-21T23:10:27.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.167+0000 7ffbcfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbd006dbc0 0x7ffbd01ebb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:27.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.167+0000 7ffbcfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbd006dbc0 0x7ffbd01ebb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39322/0 (socket says 172.21.15.28:39322) 2024-02-21T23:10:27.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.167+0000 7ffbcfe00640 1 -- 172.21.15.28:0/2221791962 learned_addr learned my addr 172.21.15.28:0/2221791962 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:27.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.168+0000 7ffbc7e00640 1 --2- 172.21.15.28:0/2221791962 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffbd013e0c0 0x7ffbd0133a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:27.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.168+0000 7ffbd4c00640 1 --2- 172.21.15.28:0/2221791962 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffbd013e520 0x7ffbd00ab5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:27.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.168+0000 7ffbcfe00640 1 -- 172.21.15.28:0/2221791962 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffbd013e520 msgr2=0x7ffbd00ab5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:27.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.168+0000 7ffbcfe00640 1 --2- 172.21.15.28:0/2221791962 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffbd013e520 0x7ffbd00ab5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:27.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.168+0000 7ffbcfe00640 1 -- 172.21.15.28:0/2221791962 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffbd013e0c0 msgr2=0x7ffbd0133a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:27.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.168+0000 7ffbcfe00640 1 --2- 172.21.15.28:0/2221791962 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffbd013e0c0 0x7ffbd0133a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:27.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.168+0000 7ffbcfe00640 1 -- 172.21.15.28:0/2221791962 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbd0137f10 con 0x7ffbd006dbc0 2024-02-21T23:10:27.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.168+0000 7ffbcfe00640 1 --2- 172.21.15.28:0/2221791962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbd006dbc0 0x7ffbd01ebb20 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7ffbc000c3f0 tx=0x7ffbc000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:27.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.168+0000 7ffbccc00640 1 -- 172.21.15.28:0/2221791962 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbc0017070 con 0x7ffbd006dbc0 2024-02-21T23:10:27.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.169+0000 7ffbccc00640 1 -- 172.21.15.28:0/2221791962 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbc0005ce0 con 0x7ffbd006dbc0 2024-02-21T23:10:27.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.169+0000 7ffbccc00640 1 -- 172.21.15.28:0/2221791962 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbc00072f0 con 0x7ffbd006dbc0 2024-02-21T23:10:27.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.169+0000 7ffbd5600640 1 -- 172.21.15.28:0/2221791962 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbd00ad580 con 0x7ffbd006dbc0 2024-02-21T23:10:27.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.169+0000 7ffbd5600640 1 -- 172.21.15.28:0/2221791962 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbd00adb00 con 0x7ffbd006dbc0 2024-02-21T23:10:27.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.170+0000 7ffbd5600640 1 -- 172.21.15.28:0/2221791962 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbd004f290 con 0x7ffbd006dbc0 2024-02-21T23:10:27.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.171+0000 7ffbccc00640 1 -- 172.21.15.28:0/2221791962 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffbc001a070 con 0x7ffbd006dbc0 2024-02-21T23:10:27.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.172+0000 7ffbccc00640 1 --2- 172.21.15.28:0/2221791962 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ffbac06b4d0 0x7ffbac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:27.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.172+0000 7ffbccc00640 1 -- 172.21.15.28:0/2221791962 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 7216+0+0 (secure 0 0 0) 0x7ffbc0089b40 con 0x7ffbd006dbc0 2024-02-21T23:10:27.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.172+0000 7ffbc7e00640 1 --2- 172.21.15.28:0/2221791962 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ffbac06b4d0 0x7ffbac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:27.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.173+0000 7ffbc7e00640 1 --2- 172.21.15.28:0/2221791962 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ffbac06b4d0 0x7ffbac06d990 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7ffbb8035000 tx=0x7ffbb8030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:27.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.176+0000 7ffbccc00640 1 -- 172.21.15.28:0/2221791962 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ffbd0158550 con 0x7ffbd006dbc0 2024-02-21T23:10:27.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.374+0000 7ffbd5600640 1 -- 172.21.15.28:0/2221791962 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ffbd00c76a0 con 0x7ffbd006dbc0 2024-02-21T23:10:27.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.375+0000 7ffbccc00640 1 -- 172.21.15.28:0/2221791962 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v158) v1 ==== 104+0+11 (secure 0 0 0) 0x7ffbc0057d10 con 0x7ffbd006dbc0 2024-02-21T23:10:27.363 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-02-21T23:10:27.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.378+0000 7ffbd5600640 1 -- 172.21.15.28:0/2221791962 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ffbac06b4d0 msgr2=0x7ffbac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:27.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.378+0000 7ffbd5600640 1 --2- 172.21.15.28:0/2221791962 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ffbac06b4d0 0x7ffbac06d990 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7ffbb8035000 tx=0x7ffbb8030000 comp rx=0 tx=0).stop 2024-02-21T23:10:27.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.378+0000 7ffbd5600640 1 -- 172.21.15.28:0/2221791962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbd006dbc0 msgr2=0x7ffbd01ebb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:27.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.378+0000 7ffbd5600640 1 --2- 172.21.15.28:0/2221791962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbd006dbc0 0x7ffbd01ebb20 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7ffbc000c3f0 tx=0x7ffbc000c8b0 comp rx=0 tx=0).stop 2024-02-21T23:10:27.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.378+0000 7ffbd5600640 1 -- 172.21.15.28:0/2221791962 shutdown_connections 2024-02-21T23:10:27.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.378+0000 7ffbd5600640 1 --2- 172.21.15.28:0/2221791962 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ffbac06b4d0 0x7ffbac06d990 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:27.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.378+0000 7ffbd5600640 1 --2- 172.21.15.28:0/2221791962 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffbd013e520 0x7ffbd00ab5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:27.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.378+0000 7ffbd5600640 1 --2- 172.21.15.28:0/2221791962 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffbd013e0c0 0x7ffbd0133a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:27.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.378+0000 7ffbd5600640 1 --2- 172.21.15.28:0/2221791962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbd006dbc0 0x7ffbd01ebb20 unknown :-1 s=CLOSED pgs=1795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:27.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.378+0000 7ffbd5600640 1 -- 172.21.15.28:0/2221791962 >> 172.21.15.28:0/2221791962 conn(0x7ffbd0073110 msgr2=0x7ffbd0151430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:27.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.378+0000 7ffbd5600640 1 -- 172.21.15.28:0/2221791962 shutdown_connections 2024-02-21T23:10:27.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.378+0000 7ffbd5600640 1 -- 172.21.15.28:0/2221791962 wait complete. 2024-02-21T23:10:27.387 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-21T23:10:27.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.619+0000 7f5d32800640 1 Processor -- start 2024-02-21T23:10:27.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.620+0000 7f5d32800640 1 -- start start 2024-02-21T23:10:27.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.621+0000 7f5d32800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f5d2c1505e0 0x7f5d2c0c2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:27.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.621+0000 7f5d32800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5d2c156ec0 0x7f5d2c071540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:27.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.621+0000 7f5d32800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d2c1509c0 con 0x7f5d2c0c4a30 2024-02-21T23:10:27.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.621+0000 7f5d32800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d2c0c64d0 con 0x7f5d2c156ec0 2024-02-21T23:10:27.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.621+0000 7f5d32800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d2c0c67b0 con 0x7f5d2c1505e0 2024-02-21T23:10:27.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.622+0000 7f5d30a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5d2c156ec0 0x7f5d2c071540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:27.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.622+0000 7f5d30a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5d2c156ec0 0x7f5d2c071540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39334/0 (socket says 172.21.15.28:39334) 2024-02-21T23:10:27.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.622+0000 7f5d30a00640 1 -- 172.21.15.28:0/3794055126 learned_addr learned my addr 172.21.15.28:0/3794055126 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:27.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.622+0000 7f5d31400640 1 --2- 172.21.15.28:0/3794055126 >> v2:172.21.15.81:3300/0 conn(0x7f5d2c1505e0 0x7f5d2c0c2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:27.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.622+0000 7f5d30a00640 1 -- 172.21.15.28:0/3794055126 >> v2:172.21.15.81:3300/0 conn(0x7f5d2c1505e0 msgr2=0x7f5d2c0c2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:27.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.622+0000 7f5d30a00640 1 --2- 172.21.15.28:0/3794055126 >> v2:172.21.15.81:3300/0 conn(0x7f5d2c1505e0 0x7f5d2c0c2ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:27.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.622+0000 7f5d30a00640 1 -- 172.21.15.28:0/3794055126 >> v1:172.21.15.81:6789/0 conn(0x7f5d2c0c4a30 legacy=0x7f5d2c0c4ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:27.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.622+0000 7f5d30a00640 1 -- 172.21.15.28:0/3794055126 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d2c0bd690 con 0x7f5d2c156ec0 2024-02-21T23:10:27.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.623+0000 7f5d30a00640 1 --2- 172.21.15.28:0/3794055126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d2c156ec0 0x7f5d2c071540 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7f5d18009870 tx=0x7f5d1802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ae8135d490037d8 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:27.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.623+0000 7f5d2be00640 1 -- 172.21.15.28:0/3794055126 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d1802e070 con 0x7f5d2c156ec0 2024-02-21T23:10:27.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.623+0000 7f5d2be00640 1 -- 172.21.15.28:0/3794055126 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d18037070 con 0x7f5d2c156ec0 2024-02-21T23:10:27.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.623+0000 7f5d2be00640 1 -- 172.21.15.28:0/3794055126 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d18031a00 con 0x7f5d2c156ec0 2024-02-21T23:10:27.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.624+0000 7f5d32800640 1 -- 172.21.15.28:0/3794055126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d2c156ec0 msgr2=0x7f5d2c071540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:27.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.624+0000 7f5d32800640 1 --2- 172.21.15.28:0/3794055126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d2c156ec0 0x7f5d2c071540 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7f5d18009870 tx=0x7f5d1802b880 comp rx=0 tx=0).stop 2024-02-21T23:10:27.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.624+0000 7f5d32800640 1 -- 172.21.15.28:0/3794055126 shutdown_connections 2024-02-21T23:10:27.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.624+0000 7f5d32800640 1 --2- 172.21.15.28:0/3794055126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d2c156ec0 0x7f5d2c071540 unknown :-1 s=CLOSED pgs=1796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:27.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.624+0000 7f5d32800640 1 --2- 172.21.15.28:0/3794055126 >> v2:172.21.15.81:3300/0 conn(0x7f5d2c1505e0 0x7f5d2c0c2ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:27.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.624+0000 7f5d32800640 1 -- 172.21.15.28:0/3794055126 >> 172.21.15.28:0/3794055126 conn(0x7f5d2c06ba30 msgr2=0x7f5d2c0aa280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:27.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.624+0000 7f5d32800640 1 -- 172.21.15.28:0/3794055126 shutdown_connections 2024-02-21T23:10:27.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.624+0000 7f5d32800640 1 -- 172.21.15.28:0/3794055126 wait complete. 2024-02-21T23:10:27.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.626+0000 7f5d32800640 1 Processor -- start 2024-02-21T23:10:27.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.626+0000 7f5d32800640 1 -- start start 2024-02-21T23:10:27.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.627+0000 7f5d32800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d2c0c4a30 0x7f5d2c0a94b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:27.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.627+0000 7f5d32800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d2c1505e0 0x7f5d2c0ad230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:27.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.627+0000 7f5d32800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5d2c156ec0 0x7f5d2c14a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:27.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.627+0000 7f5d32800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d2c0c55d0 con 0x7f5d2c0c4a30 2024-02-21T23:10:27.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.627+0000 7f5d32800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d2c157330 con 0x7f5d2c1505e0 2024-02-21T23:10:27.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.627+0000 7f5d32800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d2c0c5750 con 0x7f5d2c156ec0 2024-02-21T23:10:27.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.627+0000 7f5d31400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d2c0c4a30 0x7f5d2c0a94b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:27.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.628+0000 7f5d31400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d2c0c4a30 0x7f5d2c0a94b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39336/0 (socket says 172.21.15.28:39336) 2024-02-21T23:10:27.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.628+0000 7f5d31400640 1 -- 172.21.15.28:0/1374671764 learned_addr learned my addr 172.21.15.28:0/1374671764 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:27.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.628+0000 7f5d31e00640 1 --2- 172.21.15.28:0/1374671764 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5d2c156ec0 0x7f5d2c14a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:27.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.628+0000 7f5d30a00640 1 --2- 172.21.15.28:0/1374671764 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d2c1505e0 0x7f5d2c0ad230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:27.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.628+0000 7f5d31400640 1 -- 172.21.15.28:0/1374671764 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5d2c156ec0 msgr2=0x7f5d2c14a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:27.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.628+0000 7f5d31400640 1 --2- 172.21.15.28:0/1374671764 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5d2c156ec0 0x7f5d2c14a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:27.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.628+0000 7f5d31400640 1 -- 172.21.15.28:0/1374671764 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d2c1505e0 msgr2=0x7f5d2c0ad230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:27.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.628+0000 7f5d31400640 1 --2- 172.21.15.28:0/1374671764 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d2c1505e0 0x7f5d2c0ad230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:27.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.628+0000 7f5d31400640 1 -- 172.21.15.28:0/1374671764 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d2c1e9760 con 0x7f5d2c0c4a30 2024-02-21T23:10:27.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.628+0000 7f5d31400640 1 --2- 172.21.15.28:0/1374671764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d2c0c4a30 0x7f5d2c0a94b0 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7f5d2000d800 tx=0x7f5d2000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:27.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.630+0000 7f5d29600640 1 -- 172.21.15.28:0/1374671764 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d2000b6c0 con 0x7f5d2c0c4a30 2024-02-21T23:10:27.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.630+0000 7f5d29600640 1 -- 172.21.15.28:0/1374671764 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d20019070 con 0x7f5d2c0c4a30 2024-02-21T23:10:27.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.630+0000 7f5d29600640 1 -- 172.21.15.28:0/1374671764 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d2000ba80 con 0x7f5d2c0c4a30 2024-02-21T23:10:27.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.630+0000 7f5d32800640 1 -- 172.21.15.28:0/1374671764 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d2c0ab370 con 0x7f5d2c0c4a30 2024-02-21T23:10:27.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.630+0000 7f5d32800640 1 -- 172.21.15.28:0/1374671764 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d2c1e9480 con 0x7f5d2c0c4a30 2024-02-21T23:10:27.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.631+0000 7f5d32800640 1 -- 172.21.15.28:0/1374671764 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d2c1e9270 con 0x7f5d2c0c4a30 2024-02-21T23:10:27.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.632+0000 7f5d29600640 1 -- 172.21.15.28:0/1374671764 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5d2000bc20 con 0x7f5d2c0c4a30 2024-02-21T23:10:27.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.633+0000 7f5d29600640 1 --2- 172.21.15.28:0/1374671764 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5d0c06b4d0 0x7f5d0c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:27.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.633+0000 7f5d29600640 1 -- 172.21.15.28:0/1374671764 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 7216+0+0 (secure 0 0 0) 0x7f5d2008ac00 con 0x7f5d2c0c4a30 2024-02-21T23:10:27.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.633+0000 7f5d30a00640 1 --2- 172.21.15.28:0/1374671764 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5d0c06b4d0 0x7f5d0c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:27.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.635+0000 7f5d30a00640 1 --2- 172.21.15.28:0/1374671764 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5d0c06b4d0 0x7f5d0c06d990 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f5d18036000 tx=0x7f5d18008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:27.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.637+0000 7f5d29600640 1 -- 172.21.15.28:0/1374671764 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f5d20091080 con 0x7f5d2c0c4a30 2024-02-21T23:10:27.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.838+0000 7f5d32800640 1 -- 172.21.15.28:0/1374671764 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5d2c04f290 con 0x7f5d2c0c4a30 2024-02-21T23:10:27.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.839+0000 7f5d29600640 1 -- 172.21.15.28:0/1374671764 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v159) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5d20059530 con 0x7f5d2c0c4a30 2024-02-21T23:10:27.828 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-02-21T23:10:27.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.842+0000 7f5d32800640 1 -- 172.21.15.28:0/1374671764 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5d0c06b4d0 msgr2=0x7f5d0c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:27.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.842+0000 7f5d32800640 1 --2- 172.21.15.28:0/1374671764 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5d0c06b4d0 0x7f5d0c06d990 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f5d18036000 tx=0x7f5d18008030 comp rx=0 tx=0).stop 2024-02-21T23:10:27.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.842+0000 7f5d32800640 1 -- 172.21.15.28:0/1374671764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d2c0c4a30 msgr2=0x7f5d2c0a94b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:27.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.842+0000 7f5d32800640 1 --2- 172.21.15.28:0/1374671764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d2c0c4a30 0x7f5d2c0a94b0 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7f5d2000d800 tx=0x7f5d2000dcc0 comp rx=0 tx=0).stop 2024-02-21T23:10:27.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.842+0000 7f5d32800640 1 -- 172.21.15.28:0/1374671764 shutdown_connections 2024-02-21T23:10:27.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.842+0000 7f5d32800640 1 --2- 172.21.15.28:0/1374671764 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5d0c06b4d0 0x7f5d0c06d990 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:27.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.842+0000 7f5d32800640 1 --2- 172.21.15.28:0/1374671764 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5d2c156ec0 0x7f5d2c14a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:27.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.842+0000 7f5d32800640 1 --2- 172.21.15.28:0/1374671764 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d2c1505e0 0x7f5d2c0ad230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:27.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.842+0000 7f5d32800640 1 --2- 172.21.15.28:0/1374671764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d2c0c4a30 0x7f5d2c0a94b0 unknown :-1 s=CLOSED pgs=1797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:27.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.842+0000 7f5d32800640 1 -- 172.21.15.28:0/1374671764 >> 172.21.15.28:0/1374671764 conn(0x7f5d2c06ba30 msgr2=0x7f5d2c06f810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:27.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.842+0000 7f5d32800640 1 -- 172.21.15.28:0/1374671764 shutdown_connections 2024-02-21T23:10:27.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:27.842+0000 7f5d32800640 1 -- 172.21.15.28:0/1374671764 wait complete. 2024-02-21T23:10:27.849 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs2_metadata pg_num 2024-02-21T23:10:28.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.086+0000 7f80e3200640 1 Processor -- start 2024-02-21T23:10:28.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.086+0000 7f80e3200640 1 -- start start 2024-02-21T23:10:28.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.087+0000 7f80e3200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f80e40add20 0x7f80e4153670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:28.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.087+0000 7f80e3200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f80e40c69b0 0x7f80e40c6e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:28.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.087+0000 7f80e3200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80e406f130 con 0x7f80e4071110 2024-02-21T23:10:28.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.087+0000 7f80e3200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80e40c6520 con 0x7f80e40c69b0 2024-02-21T23:10:28.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.087+0000 7f80e3200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80e406f3f0 con 0x7f80e40add20 2024-02-21T23:10:28.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.088+0000 7f80e1400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f80e4071110 0x7f80e40c50b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:33398/0 (socket says 172.21.15.28:33398) 2024-02-21T23:10:28.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.088+0000 7f80e1400640 1 -- 172.21.15.28:0/739578184 learned_addr learned my addr 172.21.15.28:0/739578184 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:10:28.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.088+0000 7f80e2800640 1 --2- 172.21.15.28:0/739578184 >> v2:172.21.15.28:3300/0 conn(0x7f80e40c69b0 0x7f80e40c6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:28.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.088+0000 7f80e1e00640 1 --2- 172.21.15.28:0/739578184 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f80e40add20 0x7f80e4153670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:28.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.088+0000 7f80e0a00640 1 -- 172.21.15.28:0/739578184 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2344590452 0 0) 0x7f80e406f130 con 0x7f80e4071110 2024-02-21T23:10:28.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.088+0000 7f80e0a00640 1 -- 172.21.15.28:0/739578184 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80c8003520 con 0x7f80e4071110 2024-02-21T23:10:28.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.089+0000 7f80e0a00640 1 -- 172.21.15.28:0/739578184 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1268800455 0 0) 0x7f80c8003520 con 0x7f80e4071110 2024-02-21T23:10:28.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.089+0000 7f80e2800640 1 -- 172.21.15.28:0/739578184 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f80e40add20 msgr2=0x7f80e4153670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:28.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.089+0000 7f80e2800640 1 --2- 172.21.15.28:0/739578184 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f80e40add20 0x7f80e4153670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:28.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.089+0000 7f80e2800640 1 -- 172.21.15.28:0/739578184 >> v1:172.21.15.28:6789/0 conn(0x7f80e4071110 legacy=0x7f80e40c50b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:28.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.089+0000 7f80e2800640 1 -- 172.21.15.28:0/739578184 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80e406f6d0 con 0x7f80e40c69b0 2024-02-21T23:10:28.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.089+0000 7f80e1e00640 1 --2- 172.21.15.28:0/739578184 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f80e40add20 0x7f80e4153670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-21T23:10:28.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.090+0000 7f80e2800640 1 --2- 172.21.15.28:0/739578184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80e40c69b0 0x7f80e40c6e00 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7f80dc009d90 tx=0x7f80dc02bce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a09db3b72091da6 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:28.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.090+0000 7f80e0a00640 1 -- 172.21.15.28:0/739578184 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80dc039070 con 0x7f80e40c69b0 2024-02-21T23:10:28.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.090+0000 7f80e0a00640 1 -- 172.21.15.28:0/739578184 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80dc02c810 con 0x7f80e40c69b0 2024-02-21T23:10:28.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.090+0000 7f80e0a00640 1 -- 172.21.15.28:0/739578184 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80cc001670 con 0x7f80e40c69b0 2024-02-21T23:10:28.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.091+0000 7f80e3200640 1 -- 172.21.15.28:0/739578184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80e40c69b0 msgr2=0x7f80e40c6e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:28.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.091+0000 7f80e3200640 1 --2- 172.21.15.28:0/739578184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80e40c69b0 0x7f80e40c6e00 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7f80dc009d90 tx=0x7f80dc02bce0 comp rx=0 tx=0).stop 2024-02-21T23:10:28.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.091+0000 7f80e3200640 1 -- 172.21.15.28:0/739578184 shutdown_connections 2024-02-21T23:10:28.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.091+0000 7f80e3200640 1 --2- 172.21.15.28:0/739578184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80e40c69b0 0x7f80e40c6e00 unknown :-1 s=CLOSED pgs=1799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:28.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.091+0000 7f80e3200640 1 --2- 172.21.15.28:0/739578184 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f80e40add20 0x7f80e4153670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:28.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.091+0000 7f80e3200640 1 -- 172.21.15.28:0/739578184 >> 172.21.15.28:0/739578184 conn(0x7f80e40c8620 msgr2=0x7f80e40c8a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:28.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.091+0000 7f80e3200640 1 -- 172.21.15.28:0/739578184 shutdown_connections 2024-02-21T23:10:28.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.091+0000 7f80e3200640 1 -- 172.21.15.28:0/739578184 wait complete. 2024-02-21T23:10:28.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.093+0000 7f80e3200640 1 Processor -- start 2024-02-21T23:10:28.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.093+0000 7f80e3200640 1 -- start start 2024-02-21T23:10:28.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.094+0000 7f80e3200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f80e4071110 0x7f80e41e1d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:28.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.094+0000 7f80e3200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80e40add20 0x7f80e40aabf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:28.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.094+0000 7f80e3200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f80e40c69b0 0x7f80e41eddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:28.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.094+0000 7f80e3200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80e40c57c0 con 0x7f80e40add20 2024-02-21T23:10:28.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.094+0000 7f80e3200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80e40c2c30 con 0x7f80e40c69b0 2024-02-21T23:10:28.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.094+0000 7f80e3200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80e40c5940 con 0x7f80e4071110 2024-02-21T23:10:28.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.094+0000 7f80e1400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80e40add20 0x7f80e40aabf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:28.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.094+0000 7f80e1400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80e40add20 0x7f80e40aabf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39360/0 (socket says 172.21.15.28:39360) 2024-02-21T23:10:28.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.094+0000 7f80e1400640 1 -- 172.21.15.28:0/1595411997 learned_addr learned my addr 172.21.15.28:0/1595411997 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:28.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.094+0000 7f80e1e00640 1 --2- 172.21.15.28:0/1595411997 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f80e4071110 0x7f80e41e1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:28.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.095+0000 7f80e2800640 1 --2- 172.21.15.28:0/1595411997 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f80e40c69b0 0x7f80e41eddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:28.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.095+0000 7f80e1400640 1 -- 172.21.15.28:0/1595411997 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f80e4071110 msgr2=0x7f80e41e1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:28.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.095+0000 7f80e1400640 1 --2- 172.21.15.28:0/1595411997 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f80e4071110 0x7f80e41e1d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:28.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.095+0000 7f80e1400640 1 -- 172.21.15.28:0/1595411997 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f80e40c69b0 msgr2=0x7f80e41eddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:28.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.095+0000 7f80e1400640 1 --2- 172.21.15.28:0/1595411997 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f80e40c69b0 0x7f80e41eddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:28.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.095+0000 7f80e1400640 1 -- 172.21.15.28:0/1595411997 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80e414f240 con 0x7f80e40add20 2024-02-21T23:10:28.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.095+0000 7f80e1400640 1 --2- 172.21.15.28:0/1595411997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80e40add20 0x7f80e40aabf0 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7f80cc0017e0 tx=0x7f80cc00cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:28.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.095+0000 7f80da000640 1 -- 172.21.15.28:0/1595411997 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80cc005ad0 con 0x7f80e40add20 2024-02-21T23:10:28.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.095+0000 7f80da000640 1 -- 172.21.15.28:0/1595411997 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80cc005c70 con 0x7f80e40add20 2024-02-21T23:10:28.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.095+0000 7f80da000640 1 -- 172.21.15.28:0/1595411997 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80cc007290 con 0x7f80e40add20 2024-02-21T23:10:28.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.095+0000 7f80e3200640 1 -- 172.21.15.28:0/1595411997 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80e41de4d0 con 0x7f80e40add20 2024-02-21T23:10:28.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.096+0000 7f80e3200640 1 -- 172.21.15.28:0/1595411997 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80e41ddcd0 con 0x7f80e40add20 2024-02-21T23:10:28.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.097+0000 7f80e3200640 1 -- 172.21.15.28:0/1595411997 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80e404f290 con 0x7f80e40add20 2024-02-21T23:10:28.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.098+0000 7f80da000640 1 -- 172.21.15.28:0/1595411997 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f80cc007430 con 0x7f80e40add20 2024-02-21T23:10:28.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.099+0000 7f80da000640 1 --2- 172.21.15.28:0/1595411997 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f80c006b520 0x7f80c006d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:28.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.099+0000 7f80da000640 1 -- 172.21.15.28:0/1595411997 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 7216+0+0 (secure 0 0 0) 0x7f80cc08a570 con 0x7f80e40add20 2024-02-21T23:10:28.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.100+0000 7f80e1e00640 1 --2- 172.21.15.28:0/1595411997 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f80c006b520 0x7f80c006d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:28.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.101+0000 7f80e1e00640 1 --2- 172.21.15.28:0/1595411997 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f80c006b520 0x7f80c006d9e0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f80d4009560 tx=0x7f80d4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:28.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.104+0000 7f80da000640 1 -- 172.21.15.28:0/1595411997 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f80cc058ea0 con 0x7f80e40add20 2024-02-21T23:10:28.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.303+0000 7f80e3200640 1 -- 172.21.15.28:0/1595411997 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs2_metadata", "var": "pg_num"} v 0) v1 -- 0x7f80e40c80b0 con 0x7f80e40add20 2024-02-21T23:10:28.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.305+0000 7f80da000640 1 -- 172.21.15.28:0/1595411997 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs2_metadata", "var": "pg_num"}]=0 v159) v1 ==== 105+0+11 (secure 0 0 0) 0x7f80cc0587a0 con 0x7f80e40add20 2024-02-21T23:10:28.293 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-02-21T23:10:28.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.308+0000 7f80e3200640 1 -- 172.21.15.28:0/1595411997 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f80c006b520 msgr2=0x7f80c006d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:28.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.308+0000 7f80e3200640 1 --2- 172.21.15.28:0/1595411997 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f80c006b520 0x7f80c006d9e0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f80d4009560 tx=0x7f80d4008040 comp rx=0 tx=0).stop 2024-02-21T23:10:28.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.308+0000 7f80e3200640 1 -- 172.21.15.28:0/1595411997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80e40add20 msgr2=0x7f80e40aabf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:28.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.308+0000 7f80e3200640 1 --2- 172.21.15.28:0/1595411997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80e40add20 0x7f80e40aabf0 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7f80cc0017e0 tx=0x7f80cc00cbe0 comp rx=0 tx=0).stop 2024-02-21T23:10:28.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.308+0000 7f80e3200640 1 -- 172.21.15.28:0/1595411997 shutdown_connections 2024-02-21T23:10:28.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.308+0000 7f80e3200640 1 --2- 172.21.15.28:0/1595411997 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f80c006b520 0x7f80c006d9e0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:28.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.308+0000 7f80e3200640 1 --2- 172.21.15.28:0/1595411997 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f80e40c69b0 0x7f80e41eddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:28.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.308+0000 7f80e3200640 1 --2- 172.21.15.28:0/1595411997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80e40add20 0x7f80e40aabf0 unknown :-1 s=CLOSED pgs=1800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:28.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.308+0000 7f80e3200640 1 --2- 172.21.15.28:0/1595411997 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f80e4071110 0x7f80e41e1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:28.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.308+0000 7f80e3200640 1 -- 172.21.15.28:0/1595411997 >> 172.21.15.28:0/1595411997 conn(0x7f80e40c8620 msgr2=0x7f80e40c8a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:28.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.308+0000 7f80e3200640 1 -- 172.21.15.28:0/1595411997 shutdown_connections 2024-02-21T23:10:28.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.308+0000 7f80e3200640 1 -- 172.21.15.28:0/1595411997 wait complete. 2024-02-21T23:10:28.316 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs2_data pg_num 2024-02-21T23:10:28.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.555+0000 7f889d000640 1 Processor -- start 2024-02-21T23:10:28.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.555+0000 7f889d000640 1 -- start start 2024-02-21T23:10:28.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.556+0000 7f889d000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8898158480 0x7f8898154f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:28.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.556+0000 7f889d000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f889806e9d0 0x7f889806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:28.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.556+0000 7f889d000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f889806f380 con 0x7f8898152560 2024-02-21T23:10:28.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.556+0000 7f889d000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88981554a0 con 0x7f8898158480 2024-02-21T23:10:28.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.556+0000 7f889d000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88980a6270 con 0x7f889806e9d0 2024-02-21T23:10:28.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.557+0000 7f888fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8898158480 0x7f8898154f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:28.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.557+0000 7f8897400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f8898152560 0x7f889806ae70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:33406/0 (socket says 172.21.15.28:33406) 2024-02-21T23:10:28.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.557+0000 7f8897400640 1 -- 172.21.15.28:0/3451408379 learned_addr learned my addr 172.21.15.28:0/3451408379 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:10:28.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.557+0000 7f8897e00640 1 --2- 172.21.15.28:0/3451408379 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f889806e9d0 0x7f889806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:28.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.557+0000 7f8896a00640 1 -- 172.21.15.28:0/3451408379 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 197541408 0 0) 0x7f889806f380 con 0x7f8898152560 2024-02-21T23:10:28.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.557+0000 7f8896a00640 1 -- 172.21.15.28:0/3451408379 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8878003520 con 0x7f8898152560 2024-02-21T23:10:28.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.558+0000 7f888fe00640 1 -- 172.21.15.28:0/3451408379 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f889806e9d0 msgr2=0x7f889806ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:28.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.558+0000 7f888fe00640 1 --2- 172.21.15.28:0/3451408379 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f889806e9d0 0x7f889806ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:28.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.558+0000 7f888fe00640 1 -- 172.21.15.28:0/3451408379 >> v1:172.21.15.28:6789/0 conn(0x7f8898152560 legacy=0x7f889806ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:28.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.558+0000 7f888fe00640 1 -- 172.21.15.28:0/3451408379 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88981556d0 con 0x7f8898158480 2024-02-21T23:10:28.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.559+0000 7f888fe00640 1 --2- 172.21.15.28:0/3451408379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8898158480 0x7f8898154f60 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7f88800097a0 tx=0x7f888002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4fec6ae11543553 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:28.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.559+0000 7f8896a00640 1 -- 172.21.15.28:0/3451408379 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8880035020 con 0x7f8898158480 2024-02-21T23:10:28.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.559+0000 7f8896a00640 1 -- 172.21.15.28:0/3451408379 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8880031810 con 0x7f8898158480 2024-02-21T23:10:28.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.559+0000 7f8896a00640 1 -- 172.21.15.28:0/3451408379 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8880031be0 con 0x7f8898158480 2024-02-21T23:10:28.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.560+0000 7f889d000640 1 -- 172.21.15.28:0/3451408379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8898158480 msgr2=0x7f8898154f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:28.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.560+0000 7f889d000640 1 --2- 172.21.15.28:0/3451408379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8898158480 0x7f8898154f60 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7f88800097a0 tx=0x7f888002b8b0 comp rx=0 tx=0).stop 2024-02-21T23:10:28.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.560+0000 7f889d000640 1 -- 172.21.15.28:0/3451408379 shutdown_connections 2024-02-21T23:10:28.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.560+0000 7f889d000640 1 --2- 172.21.15.28:0/3451408379 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f889806e9d0 0x7f889806ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:28.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.560+0000 7f889d000640 1 --2- 172.21.15.28:0/3451408379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8898158480 0x7f8898154f60 unknown :-1 s=CLOSED pgs=1802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:28.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.560+0000 7f889d000640 1 -- 172.21.15.28:0/3451408379 >> 172.21.15.28:0/3451408379 conn(0x7f88980a8530 msgr2=0x7f88980a8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:28.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.560+0000 7f889d000640 1 -- 172.21.15.28:0/3451408379 shutdown_connections 2024-02-21T23:10:28.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.560+0000 7f889d000640 1 -- 172.21.15.28:0/3451408379 wait complete. 2024-02-21T23:10:28.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.562+0000 7f889d000640 1 Processor -- start 2024-02-21T23:10:28.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.562+0000 7f889d000640 1 -- start start 2024-02-21T23:10:28.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.563+0000 7f889d000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f889806e9d0 0x7f8898156540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:28.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.563+0000 7f889d000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8898152560 0x7f8898144df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:28.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.563+0000 7f889d000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8898158480 0x7f8898145330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:28.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.563+0000 7f889d000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f889806e4b0 con 0x7f8898152560 2024-02-21T23:10:28.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.563+0000 7f889d000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8898155c90 con 0x7f8898158480 2024-02-21T23:10:28.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.563+0000 7f889d000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8898152940 con 0x7f889806e9d0 2024-02-21T23:10:28.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.564+0000 7f888fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8898152560 0x7f8898144df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:28.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.564+0000 7f888fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8898152560 0x7f8898144df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39380/0 (socket says 172.21.15.28:39380) 2024-02-21T23:10:28.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.564+0000 7f888fe00640 1 -- 172.21.15.28:0/1947450971 learned_addr learned my addr 172.21.15.28:0/1947450971 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:28.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.564+0000 7f8897400640 1 --2- 172.21.15.28:0/1947450971 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f889806e9d0 0x7f8898156540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:28.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.564+0000 7f8897e00640 1 --2- 172.21.15.28:0/1947450971 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8898158480 0x7f8898145330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:28.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.564+0000 7f888fe00640 1 -- 172.21.15.28:0/1947450971 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f889806e9d0 msgr2=0x7f8898156540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:28.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.564+0000 7f888fe00640 1 --2- 172.21.15.28:0/1947450971 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f889806e9d0 0x7f8898156540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:28.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.564+0000 7f888fe00640 1 -- 172.21.15.28:0/1947450971 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8898158480 msgr2=0x7f8898145330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:28.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.564+0000 7f888fe00640 1 --2- 172.21.15.28:0/1947450971 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8898158480 0x7f8898145330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:28.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.565+0000 7f888fe00640 1 -- 172.21.15.28:0/1947450971 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8898145be0 con 0x7f8898152560 2024-02-21T23:10:28.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.565+0000 7f888fe00640 1 --2- 172.21.15.28:0/1947450971 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8898152560 0x7f8898144df0 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7f88800098c0 tx=0x7f88800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:28.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.565+0000 7f888f400640 1 -- 172.21.15.28:0/1947450971 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8880041070 con 0x7f8898152560 2024-02-21T23:10:28.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.565+0000 7f888f400640 1 -- 172.21.15.28:0/1947450971 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8880031e40 con 0x7f8898152560 2024-02-21T23:10:28.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.565+0000 7f888f400640 1 -- 172.21.15.28:0/1947450971 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8880035040 con 0x7f8898152560 2024-02-21T23:10:28.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.565+0000 7f889d000640 1 -- 172.21.15.28:0/1947450971 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88981509a0 con 0x7f8898152560 2024-02-21T23:10:28.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.565+0000 7f889d000640 1 -- 172.21.15.28:0/1947450971 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8898150f60 con 0x7f8898152560 2024-02-21T23:10:28.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.567+0000 7f889d000640 1 -- 172.21.15.28:0/1947450971 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f889804f290 con 0x7f8898152560 2024-02-21T23:10:28.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.568+0000 7f888f400640 1 -- 172.21.15.28:0/1947450971 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f888004c060 con 0x7f8898152560 2024-02-21T23:10:28.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.569+0000 7f888f400640 1 --2- 172.21.15.28:0/1947450971 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f887406b4d0 0x7f887406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:28.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.569+0000 7f888f400640 1 -- 172.21.15.28:0/1947450971 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 7216+0+0 (secure 0 0 0) 0x7f88800ac1c0 con 0x7f8898152560 2024-02-21T23:10:28.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.569+0000 7f8897400640 1 --2- 172.21.15.28:0/1947450971 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f887406b4d0 0x7f887406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:28.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.571+0000 7f8897400640 1 --2- 172.21.15.28:0/1947450971 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f887406b4d0 0x7f887406d990 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f8888005880 tx=0x7f8888009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:28.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.574+0000 7f888f400640 1 -- 172.21.15.28:0/1947450971 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f888007a420 con 0x7f8898152560 2024-02-21T23:10:28.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.776+0000 7f889d000640 1 -- 172.21.15.28:0/1947450971 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs2_data", "var": "pg_num"} v 0) v1 -- 0x7f88981ea7e0 con 0x7f8898152560 2024-02-21T23:10:28.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.777+0000 7f888f400640 1 -- 172.21.15.28:0/1947450971 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs2_data", "var": "pg_num"}]=0 v159) v1 ==== 101+0+11 (secure 0 0 0) 0x7f88981ea7e0 con 0x7f8898152560 2024-02-21T23:10:28.765 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-02-21T23:10:28.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.780+0000 7f889d000640 1 -- 172.21.15.28:0/1947450971 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f887406b4d0 msgr2=0x7f887406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:28.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.780+0000 7f889d000640 1 --2- 172.21.15.28:0/1947450971 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f887406b4d0 0x7f887406d990 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f8888005880 tx=0x7f8888009040 comp rx=0 tx=0).stop 2024-02-21T23:10:28.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.780+0000 7f889d000640 1 -- 172.21.15.28:0/1947450971 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8898152560 msgr2=0x7f8898144df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:28.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.780+0000 7f889d000640 1 --2- 172.21.15.28:0/1947450971 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8898152560 0x7f8898144df0 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7f88800098c0 tx=0x7f88800046c0 comp rx=0 tx=0).stop 2024-02-21T23:10:28.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.780+0000 7f889d000640 1 -- 172.21.15.28:0/1947450971 shutdown_connections 2024-02-21T23:10:28.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.780+0000 7f889d000640 1 --2- 172.21.15.28:0/1947450971 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f887406b4d0 0x7f887406d990 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:28.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.780+0000 7f889d000640 1 --2- 172.21.15.28:0/1947450971 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8898158480 0x7f8898145330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:28.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.780+0000 7f889d000640 1 --2- 172.21.15.28:0/1947450971 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8898152560 0x7f8898144df0 unknown :-1 s=CLOSED pgs=1803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:28.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.780+0000 7f889d000640 1 --2- 172.21.15.28:0/1947450971 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f889806e9d0 0x7f8898156540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:28.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.780+0000 7f889d000640 1 -- 172.21.15.28:0/1947450971 >> 172.21.15.28:0/1947450971 conn(0x7f88980a8530 msgr2=0x7f88980c55b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:28.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.780+0000 7f889d000640 1 -- 172.21.15.28:0/1947450971 shutdown_connections 2024-02-21T23:10:28.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:28.780+0000 7f889d000640 1 -- 172.21.15.28:0/1947450971 wait complete. 2024-02-21T23:10:28.789 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:10:29.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.025+0000 7fd59d600640 1 Processor -- start 2024-02-21T23:10:29.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.025+0000 7fd59d600640 1 -- start start 2024-02-21T23:10:29.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.027+0000 7fd59d600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd5980a5fd0 0x7fd59806dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:29.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.027+0000 7fd59d600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5980c2760 con 0x7fd59806ee40 2024-02-21T23:10:29.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.027+0000 7fd59d600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd598156310 con 0x7fd598150450 2024-02-21T23:10:29.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.027+0000 7fd59d600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5980c22c0 con 0x7fd5980a5fd0 2024-02-21T23:10:29.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.028+0000 7fd597400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd59806ee40 0x7fd598156af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:33418/0 (socket says 172.21.15.28:33418) 2024-02-21T23:10:29.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.028+0000 7fd597400640 1 -- 172.21.15.28:0/2706503268 learned_addr learned my addr 172.21.15.28:0/2706503268 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:10:29.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.028+0000 7fd59cc00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd5980a5fd0 0x7fd59806dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:29.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.028+0000 7fd59cc00640 1 -- 172.21.15.28:0/2706503268 >> v1:172.21.15.81:6789/0 conn(0x7fd598150450 legacy=0x7fd5981536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:29.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.028+0000 7fd59cc00640 1 -- 172.21.15.28:0/2706503268 >> v1:172.21.15.28:6789/0 conn(0x7fd59806ee40 legacy=0x7fd598156af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:29.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.028+0000 7fd596a00640 1 -- 172.21.15.28:0/2706503268 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2811897773 0 0) 0x7fd5980c2760 con 0x7fd59806ee40 2024-02-21T23:10:29.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.028+0000 7fd59cc00640 1 -- 172.21.15.28:0/2706503268 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5980c29d0 con 0x7fd5980a5fd0 2024-02-21T23:10:29.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.029+0000 7fd59cc00640 1 --2- 172.21.15.28:0/2706503268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5980a5fd0 0x7fd59806dff0 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7fd590009d90 tx=0x7fd59002bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de9c5354d7d80a67 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:29.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.030+0000 7fd596a00640 1 -- 172.21.15.28:0/2706503268 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd590039070 con 0x7fd5980a5fd0 2024-02-21T23:10:29.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.030+0000 7fd596a00640 1 -- 172.21.15.28:0/2706503268 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd59002cd00 con 0x7fd5980a5fd0 2024-02-21T23:10:29.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.030+0000 7fd596a00640 1 -- 172.21.15.28:0/2706503268 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd590034660 con 0x7fd5980a5fd0 2024-02-21T23:10:29.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.030+0000 7fd59d600640 1 -- 172.21.15.28:0/2706503268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5980a5fd0 msgr2=0x7fd59806dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:29.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.030+0000 7fd59d600640 1 --2- 172.21.15.28:0/2706503268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5980a5fd0 0x7fd59806dff0 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7fd590009d90 tx=0x7fd59002bf50 comp rx=0 tx=0).stop 2024-02-21T23:10:29.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.031+0000 7fd59d600640 1 -- 172.21.15.28:0/2706503268 shutdown_connections 2024-02-21T23:10:29.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.031+0000 7fd59d600640 1 --2- 172.21.15.28:0/2706503268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5980a5fd0 0x7fd59806dff0 unknown :-1 s=CLOSED pgs=1805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:29.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.031+0000 7fd59d600640 1 -- 172.21.15.28:0/2706503268 >> 172.21.15.28:0/2706503268 conn(0x7fd59806b630 msgr2=0x7fd5980c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:29.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.031+0000 7fd59d600640 1 -- 172.21.15.28:0/2706503268 shutdown_connections 2024-02-21T23:10:29.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.031+0000 7fd59d600640 1 -- 172.21.15.28:0/2706503268 wait complete. 2024-02-21T23:10:29.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.032+0000 7fd59d600640 1 Processor -- start 2024-02-21T23:10:29.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.032+0000 7fd59d600640 1 -- start start 2024-02-21T23:10:29.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.033+0000 7fd59d600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd59806ee40 0x7fd598145880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:29.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.033+0000 7fd59d600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd5980a5fd0 0x7fd598154cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:29.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.033+0000 7fd59d600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd598150450 0x7fd5981e8d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:29.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.033+0000 7fd59d600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd598157200 con 0x7fd598150450 2024-02-21T23:10:29.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.033+0000 7fd59d600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd598156660 con 0x7fd59806ee40 2024-02-21T23:10:29.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.033+0000 7fd59d600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd59806e530 con 0x7fd5980a5fd0 2024-02-21T23:10:29.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.034+0000 7fd59cc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd598150450 0x7fd5981e8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:29.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.034+0000 7fd59cc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd598150450 0x7fd5981e8d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39408/0 (socket says 172.21.15.28:39408) 2024-02-21T23:10:29.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.034+0000 7fd59cc00640 1 -- 172.21.15.28:0/317525432 learned_addr learned my addr 172.21.15.28:0/317525432 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:29.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.034+0000 7fd59cc00640 1 -- 172.21.15.28:0/317525432 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd5980a5fd0 msgr2=0x7fd598154cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:29.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.034+0000 7fd59cc00640 1 --2- 172.21.15.28:0/317525432 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd5980a5fd0 0x7fd598154cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:29.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.034+0000 7fd59cc00640 1 -- 172.21.15.28:0/317525432 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd59806ee40 msgr2=0x7fd598145880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:29.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.034+0000 7fd59cc00640 1 --2- 172.21.15.28:0/317525432 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd59806ee40 0x7fd598145880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:29.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.034+0000 7fd59cc00640 1 -- 172.21.15.28:0/317525432 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd598155b50 con 0x7fd598150450 2024-02-21T23:10:29.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.034+0000 7fd59cc00640 1 --2- 172.21.15.28:0/317525432 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd598150450 0x7fd5981e8d70 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7fd590037040 tx=0x7fd590004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:29.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.035+0000 7fd58fe00640 1 -- 172.21.15.28:0/317525432 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd590044030 con 0x7fd598150450 2024-02-21T23:10:29.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.035+0000 7fd58fe00640 1 -- 172.21.15.28:0/317525432 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd59003f470 con 0x7fd598150450 2024-02-21T23:10:29.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.035+0000 7fd58fe00640 1 -- 172.21.15.28:0/317525432 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd590039040 con 0x7fd598150450 2024-02-21T23:10:29.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.035+0000 7fd59d600640 1 -- 172.21.15.28:0/317525432 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd598155e30 con 0x7fd598150450 2024-02-21T23:10:29.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.035+0000 7fd59d600640 1 -- 172.21.15.28:0/317525432 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd598155210 con 0x7fd598150450 2024-02-21T23:10:29.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.036+0000 7fd59d600640 1 -- 172.21.15.28:0/317525432 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd59814e6b0 con 0x7fd598150450 2024-02-21T23:10:29.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.037+0000 7fd58fe00640 1 -- 172.21.15.28:0/317525432 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd590047070 con 0x7fd598150450 2024-02-21T23:10:29.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.038+0000 7fd58fe00640 1 --2- 172.21.15.28:0/317525432 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd57806b4d0 0x7fd57806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:29.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.038+0000 7fd58fe00640 1 -- 172.21.15.28:0/317525432 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 7216+0+0 (secure 0 0 0) 0x7fd5900ad050 con 0x7fd598150450 2024-02-21T23:10:29.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.039+0000 7fd597e00640 1 --2- 172.21.15.28:0/317525432 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd57806b4d0 0x7fd57806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:29.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.040+0000 7fd597e00640 1 --2- 172.21.15.28:0/317525432 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd57806b4d0 0x7fd57806d990 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fd588000f60 tx=0x7fd5880079b0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:29.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.044+0000 7fd58fe00640 1 -- 172.21.15.28:0/317525432 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd59814e6b0 con 0x7fd598150450 2024-02-21T23:10:29.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.283+0000 7fd59d600640 1 -- 172.21.15.28:0/317525432 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd59804f290 con 0x7fd598150450 2024-02-21T23:10:29.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.284+0000 7fd58fe00640 1 -- 172.21.15.28:0/317525432 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 171 v171) v1 ==== 95+0+6197 (secure 0 0 0) 0x7fd59004e070 con 0x7fd598150450 2024-02-21T23:10:29.272 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:10:29.273 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":171,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9141,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2895215997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2895215997},{"type":"v1","addr":"172.21.15.28:6835","nonce":2895215997}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":166},{"gid":9153,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/2178509171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2178509171},{"type":"v1","addr":"172.21.15.28:6833","nonce":2178509171}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":169}],"filesystems":[{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:09:53.730296+0000","modified":"2024-02-21T23:10:21.591594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":156,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6392},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6392":{"gid":6392,"name":"d","rank":0,"incarnation":165,"state":"up:active","state_seq":11,"addr":"172.21.15.81:6837/521625945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":521625945},{"type":"v1","addr":"172.21.15.81:6837","nonce":521625945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11},{"mdsmap":{"epoch":171,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:10:05.843850+0000","modified":"2024-02-21T23:10:23.599068+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":157,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6389},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6389":{"gid":6389,"name":"b","rank":0,"incarnation":168,"state":"up:active","state_seq":12,"addr":"172.21.15.81:6835/1320495906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1320495906},{"type":"v1","addr":"172.21.15.81:6835","nonce":1320495906}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2024-02-21T23:10:29.273 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 171 2024-02-21T23:10:29.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.287+0000 7fd59d600640 1 -- 172.21.15.28:0/317525432 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd57806b4d0 msgr2=0x7fd57806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:29.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.287+0000 7fd59d600640 1 --2- 172.21.15.28:0/317525432 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd57806b4d0 0x7fd57806d990 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fd588000f60 tx=0x7fd5880079b0 comp rx=0 tx=0).stop 2024-02-21T23:10:29.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.287+0000 7fd59d600640 1 -- 172.21.15.28:0/317525432 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd598150450 msgr2=0x7fd5981e8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:29.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.287+0000 7fd59d600640 1 --2- 172.21.15.28:0/317525432 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd598150450 0x7fd5981e8d70 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7fd590037040 tx=0x7fd590004a40 comp rx=0 tx=0).stop 2024-02-21T23:10:29.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.287+0000 7fd59d600640 1 -- 172.21.15.28:0/317525432 shutdown_connections 2024-02-21T23:10:29.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.287+0000 7fd59d600640 1 --2- 172.21.15.28:0/317525432 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd57806b4d0 0x7fd57806d990 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:29.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.287+0000 7fd59d600640 1 --2- 172.21.15.28:0/317525432 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd5980a5fd0 0x7fd598154cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:29.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.287+0000 7fd59d600640 1 --2- 172.21.15.28:0/317525432 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd598150450 0x7fd5981e8d70 unknown :-1 s=CLOSED pgs=1806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:29.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.287+0000 7fd59d600640 1 --2- 172.21.15.28:0/317525432 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd59806ee40 0x7fd598145880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:29.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.287+0000 7fd59d600640 1 -- 172.21.15.28:0/317525432 >> 172.21.15.28:0/317525432 conn(0x7fd59806b630 msgr2=0x7fd5980c3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:29.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.287+0000 7fd59d600640 1 -- 172.21.15.28:0/317525432 shutdown_connections 2024-02-21T23:10:29.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.287+0000 7fd59d600640 1 -- 172.21.15.28:0/317525432 wait complete. 2024-02-21T23:10:29.296 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:10:29.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.531+0000 7fe307800640 1 Processor -- start 2024-02-21T23:10:29.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.531+0000 7fe307800640 1 -- start start 2024-02-21T23:10:29.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.532+0000 7fe307800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe300153950 0x7fe3000c15d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:29.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.532+0000 7fe307800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe30015b6d0 0x7fe30015dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:29.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.532+0000 7fe307800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe300152e70 con 0x7fe3000c11f0 2024-02-21T23:10:29.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.532+0000 7fe307800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3000a5ec0 con 0x7fe30015b6d0 2024-02-21T23:10:29.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.532+0000 7fe307800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe300153150 con 0x7fe300153950 2024-02-21T23:10:29.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.533+0000 7fe305800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe30015b6d0 0x7fe30015dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:29.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.533+0000 7fe305800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe30015b6d0 0x7fe30015dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39418/0 (socket says 172.21.15.28:39418) 2024-02-21T23:10:29.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.533+0000 7fe305800640 1 -- 172.21.15.28:0/3544872262 learned_addr learned my addr 172.21.15.28:0/3544872262 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:29.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.533+0000 7fe2ffe00640 1 --2- 172.21.15.28:0/3544872262 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe300153950 0x7fe3000c15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:29.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.533+0000 7fe305800640 1 -- 172.21.15.28:0/3544872262 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe300153950 msgr2=0x7fe3000c15d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:29.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.533+0000 7fe305800640 1 --2- 172.21.15.28:0/3544872262 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe300153950 0x7fe3000c15d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:29.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.533+0000 7fe305800640 1 -- 172.21.15.28:0/3544872262 >> v1:172.21.15.81:6789/0 conn(0x7fe3000c11f0 legacy=0x7fe30006dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:29.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.533+0000 7fe305800640 1 -- 172.21.15.28:0/3544872262 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe300153430 con 0x7fe30015b6d0 2024-02-21T23:10:29.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.535+0000 7fe305800640 1 --2- 172.21.15.28:0/3544872262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe30015b6d0 0x7fe30015dac0 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7fe2f8009870 tx=0x7fe2f802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=582128327128d718 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:29.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.535+0000 7fe2ff400640 1 -- 172.21.15.28:0/3544872262 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2f80317c0 con 0x7fe30015b6d0 2024-02-21T23:10:29.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.535+0000 7fe2ff400640 1 -- 172.21.15.28:0/3544872262 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2f8037070 con 0x7fe30015b6d0 2024-02-21T23:10:29.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.535+0000 7fe2ff400640 1 -- 172.21.15.28:0/3544872262 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2f8031b80 con 0x7fe30015b6d0 2024-02-21T23:10:29.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.536+0000 7fe307800640 1 -- 172.21.15.28:0/3544872262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe30015b6d0 msgr2=0x7fe30015dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:29.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.536+0000 7fe307800640 1 --2- 172.21.15.28:0/3544872262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe30015b6d0 0x7fe30015dac0 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7fe2f8009870 tx=0x7fe2f802b860 comp rx=0 tx=0).stop 2024-02-21T23:10:29.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.536+0000 7fe307800640 1 -- 172.21.15.28:0/3544872262 shutdown_connections 2024-02-21T23:10:29.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.536+0000 7fe307800640 1 --2- 172.21.15.28:0/3544872262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe30015b6d0 0x7fe30015dac0 unknown :-1 s=CLOSED pgs=1807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:29.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.536+0000 7fe307800640 1 --2- 172.21.15.28:0/3544872262 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe300153950 0x7fe3000c15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:29.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.536+0000 7fe307800640 1 -- 172.21.15.28:0/3544872262 >> 172.21.15.28:0/3544872262 conn(0x7fe3000c67f0 msgr2=0x7fe3000c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:29.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.536+0000 7fe307800640 1 -- 172.21.15.28:0/3544872262 shutdown_connections 2024-02-21T23:10:29.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.536+0000 7fe307800640 1 -- 172.21.15.28:0/3544872262 wait complete. 2024-02-21T23:10:29.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.537+0000 7fe307800640 1 Processor -- start 2024-02-21T23:10:29.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.538+0000 7fe307800640 1 -- start start 2024-02-21T23:10:29.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.538+0000 7fe307800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3000c11f0 0x7fe300142740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:29.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.538+0000 7fe307800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe300153950 0x7fe300143c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:29.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.538+0000 7fe307800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe30015b6d0 0x7fe3001ebfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:29.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.538+0000 7fe307800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3000a6110 con 0x7fe300153950 2024-02-21T23:10:29.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.538+0000 7fe307800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3000c4c00 con 0x7fe3000c11f0 2024-02-21T23:10:29.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.538+0000 7fe307800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3000c0d90 con 0x7fe30015b6d0 2024-02-21T23:10:29.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.539+0000 7fe2ffe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe300153950 0x7fe300143c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:29.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.539+0000 7fe2ffe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe300153950 0x7fe300143c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39420/0 (socket says 172.21.15.28:39420) 2024-02-21T23:10:29.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.539+0000 7fe2ffe00640 1 -- 172.21.15.28:0/766839999 learned_addr learned my addr 172.21.15.28:0/766839999 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:29.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.539+0000 7fe2ffe00640 1 -- 172.21.15.28:0/766839999 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe30015b6d0 msgr2=0x7fe3001ebfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:29.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.539+0000 7fe2ffe00640 1 --2- 172.21.15.28:0/766839999 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe30015b6d0 0x7fe3001ebfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:29.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.539+0000 7fe2ffe00640 1 -- 172.21.15.28:0/766839999 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3000c11f0 msgr2=0x7fe300142740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:29.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.539+0000 7fe304e00640 1 --2- 172.21.15.28:0/766839999 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3000c11f0 0x7fe300142740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:29.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.539+0000 7fe2ffe00640 1 --2- 172.21.15.28:0/766839999 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3000c11f0 0x7fe300142740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:29.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.539+0000 7fe2ffe00640 1 -- 172.21.15.28:0/766839999 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3000a7020 con 0x7fe300153950 2024-02-21T23:10:29.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.539+0000 7fe2ffe00640 1 --2- 172.21.15.28:0/766839999 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe300153950 0x7fe300143c60 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec00d830 tx=0x7fe2ec00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:29.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.540+0000 7fe2fcc00640 1 -- 172.21.15.28:0/766839999 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2ec00b6c0 con 0x7fe300153950 2024-02-21T23:10:29.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.540+0000 7fe2fcc00640 1 -- 172.21.15.28:0/766839999 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2ec019070 con 0x7fe300153950 2024-02-21T23:10:29.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.540+0000 7fe307800640 1 -- 172.21.15.28:0/766839999 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe300144500 con 0x7fe300153950 2024-02-21T23:10:29.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.540+0000 7fe2fcc00640 1 -- 172.21.15.28:0/766839999 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2ec00ba80 con 0x7fe300153950 2024-02-21T23:10:29.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.540+0000 7fe307800640 1 -- 172.21.15.28:0/766839999 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3001ecb70 con 0x7fe300153950 2024-02-21T23:10:29.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.541+0000 7fe307800640 1 -- 172.21.15.28:0/766839999 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe300067f30 con 0x7fe300153950 2024-02-21T23:10:29.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.542+0000 7fe2fcc00640 1 -- 172.21.15.28:0/766839999 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe2ec01e460 con 0x7fe300153950 2024-02-21T23:10:29.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.543+0000 7fe2fcc00640 1 --2- 172.21.15.28:0/766839999 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe2e006b4d0 0x7fe2e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:29.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.543+0000 7fe2fcc00640 1 -- 172.21.15.28:0/766839999 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 7216+0+0 (secure 0 0 0) 0x7fe2ec08af10 con 0x7fe300153950 2024-02-21T23:10:29.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.544+0000 7fe304e00640 1 --2- 172.21.15.28:0/766839999 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe2e006b4d0 0x7fe2e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:29.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.545+0000 7fe304e00640 1 --2- 172.21.15.28:0/766839999 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe2e006b4d0 0x7fe2e006d990 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fe2f0007dc0 tx=0x7fe2f00072f0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:29.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.547+0000 7fe2fcc00640 1 -- 172.21.15.28:0/766839999 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe2ec059730 con 0x7fe300153950 2024-02-21T23:10:29.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.753+0000 7fe307800640 1 -- 172.21.15.28:0/766839999 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe3000c0960 con 0x7fe300153950 2024-02-21T23:10:29.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.755+0000 7fe2fcc00640 1 -- 172.21.15.28:0/766839999 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v159) v1 ==== 74+0+21003 (secure 0 0 0) 0x7fe2ec059030 con 0x7fe300153950 2024-02-21T23:10:29.743 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:10:29.743 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":159,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:10:27.626041+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:09:51.634291+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"150","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-02-21T23:09:52.736252+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"151","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs2_metadata","create_time":"2024-02-21T23:10:04.095451+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"154","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs2_data","create_time":"2024-02-21T23:10:04.836982+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"26.29","mappings":[{"from":1,"to":0}]},{"pgid":"26.36","mappings":[{"from":3,"to":0}]},{"pgid":"26.3f","mappings":[{"from":1,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6833/2975378719":"2024-02-22T23:10:21.581948+0000","172.21.15.28:6832/2975378719":"2024-02-22T23:10:21.581948+0000","172.21.15.28:6835/2928639298":"2024-02-22T23:10:19.550812+0000","172.21.15.28:6834/2928639298":"2024-02-22T23:10:19.550812+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:10:29.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.758+0000 7fe307800640 1 -- 172.21.15.28:0/766839999 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe2e006b4d0 msgr2=0x7fe2e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:29.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.758+0000 7fe307800640 1 --2- 172.21.15.28:0/766839999 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe2e006b4d0 0x7fe2e006d990 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fe2f0007dc0 tx=0x7fe2f00072f0 comp rx=0 tx=0).stop 2024-02-21T23:10:29.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.758+0000 7fe307800640 1 -- 172.21.15.28:0/766839999 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe300153950 msgr2=0x7fe300143c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:29.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.758+0000 7fe307800640 1 --2- 172.21.15.28:0/766839999 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe300153950 0x7fe300143c60 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec00d830 tx=0x7fe2ec00dcf0 comp rx=0 tx=0).stop 2024-02-21T23:10:29.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.758+0000 7fe307800640 1 -- 172.21.15.28:0/766839999 shutdown_connections 2024-02-21T23:10:29.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.758+0000 7fe307800640 1 --2- 172.21.15.28:0/766839999 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe2e006b4d0 0x7fe2e006d990 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:29.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.758+0000 7fe307800640 1 --2- 172.21.15.28:0/766839999 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe30015b6d0 0x7fe3001ebfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:29.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.758+0000 7fe307800640 1 --2- 172.21.15.28:0/766839999 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe300153950 0x7fe300143c60 unknown :-1 s=CLOSED pgs=1808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:29.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.758+0000 7fe307800640 1 --2- 172.21.15.28:0/766839999 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3000c11f0 0x7fe300142740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:29.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.758+0000 7fe307800640 1 -- 172.21.15.28:0/766839999 >> 172.21.15.28:0/766839999 conn(0x7fe3000c67f0 msgr2=0x7fe3000c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:29.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.758+0000 7fe307800640 1 -- 172.21.15.28:0/766839999 shutdown_connections 2024-02-21T23:10:29.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.758+0000 7fe307800640 1 -- 172.21.15.28:0/766839999 wait complete. 2024-02-21T23:10:29.765 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-02-21T23:10:29.766 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:10:29.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.994+0000 7f7c6f400640 1 Processor -- start 2024-02-21T23:10:29.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.994+0000 7f7c6f400640 1 -- start start 2024-02-21T23:10:29.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.996+0000 7f7c6f400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f7c680c4fe0 0x7f7c6806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:29.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.996+0000 7f7c6f400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7c68159400 0x7f7c6806e600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:29.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.996+0000 7f7c6f400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c68159900 con 0x7f7c681569d0 2024-02-21T23:10:29.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.996+0000 7f7c6f400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c68159be0 con 0x7f7c680c4fe0 2024-02-21T23:10:29.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.996+0000 7f7c6f400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c68159ec0 con 0x7f7c68159400 2024-02-21T23:10:29.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.996+0000 7f7c67e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f7c680c4fe0 0x7f7c6806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:29.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.997+0000 7f7c6d400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7c68159400 0x7f7c6806e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:29.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.997+0000 7f7c67e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f7c680c4fe0 0x7f7c6806e0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.28:35762/0 (socket says 172.21.15.28:35762) 2024-02-21T23:10:29.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.997+0000 7f7c67e00640 1 -- 172.21.15.28:0/426655277 learned_addr learned my addr 172.21.15.28:0/426655277 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:29.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.997+0000 7f7c6ca00640 1 --1- 172.21.15.28:0/426655277 >> v1:172.21.15.81:6789/0 conn(0x7f7c681569d0 0x7f7c680c3280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.28:46776/0 (socket says 172.21.15.28:46776) 2024-02-21T23:10:29.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.997+0000 7f7c67400640 1 -- 172.21.15.28:0/426655277 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4198847941 0 0) 0x7f7c68159900 con 0x7f7c681569d0 2024-02-21T23:10:29.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.997+0000 7f7c67400640 1 -- 172.21.15.28:0/426655277 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c50003520 con 0x7f7c681569d0 2024-02-21T23:10:29.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.997+0000 7f7c6d400640 1 -- 172.21.15.28:0/426655277 >> v2:172.21.15.81:3300/0 conn(0x7f7c680c4fe0 msgr2=0x7f7c6806e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:29.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.997+0000 7f7c6d400640 1 --2- 172.21.15.28:0/426655277 >> v2:172.21.15.81:3300/0 conn(0x7f7c680c4fe0 0x7f7c6806e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:29.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.997+0000 7f7c6d400640 1 -- 172.21.15.28:0/426655277 >> v1:172.21.15.81:6789/0 conn(0x7f7c681569d0 legacy=0x7f7c680c3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:29.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.998+0000 7f7c6d400640 1 -- 172.21.15.28:0/426655277 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c6815a1a0 con 0x7f7c68159400 2024-02-21T23:10:29.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:29.999+0000 7f7c6d400640 1 --2- 172.21.15.28:0/426655277 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7c68159400 0x7f7c6806e600 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7c60009d90 tx=0x7f7c6002be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d9a95f5296e66ac server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:29.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.000+0000 7f7c67400640 1 -- 172.21.15.28:0/426655277 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c60039070 con 0x7f7c68159400 2024-02-21T23:10:29.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.000+0000 7f7c67400640 1 -- 172.21.15.28:0/426655277 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c6002cbc0 con 0x7f7c68159400 2024-02-21T23:10:29.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.000+0000 7f7c67400640 1 -- 172.21.15.28:0/426655277 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c60034440 con 0x7f7c68159400 2024-02-21T23:10:29.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.000+0000 7f7c6f400640 1 -- 172.21.15.28:0/426655277 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7c68159400 msgr2=0x7f7c6806e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:29.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.000+0000 7f7c6f400640 1 --2- 172.21.15.28:0/426655277 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7c68159400 0x7f7c6806e600 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7c60009d90 tx=0x7f7c6002be10 comp rx=0 tx=0).stop 2024-02-21T23:10:29.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.000+0000 7f7c6f400640 1 -- 172.21.15.28:0/426655277 shutdown_connections 2024-02-21T23:10:29.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.000+0000 7f7c6f400640 1 --2- 172.21.15.28:0/426655277 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7c68159400 0x7f7c6806e600 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:29.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.000+0000 7f7c6f400640 1 --2- 172.21.15.28:0/426655277 >> v2:172.21.15.81:3300/0 conn(0x7f7c680c4fe0 0x7f7c6806e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:29.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.000+0000 7f7c6f400640 1 -- 172.21.15.28:0/426655277 >> 172.21.15.28:0/426655277 conn(0x7f7c680c67f0 msgr2=0x7f7c680c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:29.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.001+0000 7f7c6f400640 1 -- 172.21.15.28:0/426655277 shutdown_connections 2024-02-21T23:10:29.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.001+0000 7f7c6f400640 1 -- 172.21.15.28:0/426655277 wait complete. 2024-02-21T23:10:29.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.002+0000 7f7c6f400640 1 Processor -- start 2024-02-21T23:10:29.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.002+0000 7f7c6f400640 1 -- start start 2024-02-21T23:10:29.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.003+0000 7f7c6f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c680c4fe0 0x7f7c681512d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:29.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.003+0000 7f7c6f400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7c681569d0 0x7f7c68153560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:29.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.003+0000 7f7c6f400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7c68159400 0x7f7c68153aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:29.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.003+0000 7f7c6f400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c680c3b10 con 0x7f7c680c4fe0 2024-02-21T23:10:29.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.003+0000 7f7c6f400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c680c3990 con 0x7f7c681569d0 2024-02-21T23:10:29.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.003+0000 7f7c6f400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c6806eb40 con 0x7f7c68159400 2024-02-21T23:10:29.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.004+0000 7f7c6ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c680c4fe0 0x7f7c681512d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:29.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.004+0000 7f7c6ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c680c4fe0 0x7f7c681512d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39430/0 (socket says 172.21.15.28:39430) 2024-02-21T23:10:29.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.004+0000 7f7c6ca00640 1 -- 172.21.15.28:0/522586396 learned_addr learned my addr 172.21.15.28:0/522586396 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:29.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.004+0000 7f7c6d400640 1 --2- 172.21.15.28:0/522586396 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7c68159400 0x7f7c68153aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:29.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.004+0000 7f7c67e00640 1 --2- 172.21.15.28:0/522586396 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7c681569d0 0x7f7c68153560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:29.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.004+0000 7f7c6ca00640 1 -- 172.21.15.28:0/522586396 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7c68159400 msgr2=0x7f7c68153aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:29.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.004+0000 7f7c6ca00640 1 --2- 172.21.15.28:0/522586396 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7c68159400 0x7f7c68153aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:29.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.004+0000 7f7c6ca00640 1 -- 172.21.15.28:0/522586396 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7c681569d0 msgr2=0x7f7c68153560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:29.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.004+0000 7f7c6ca00640 1 --2- 172.21.15.28:0/522586396 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7c681569d0 0x7f7c68153560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:29.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.004+0000 7f7c6ca00640 1 -- 172.21.15.28:0/522586396 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c68072ec0 con 0x7f7c680c4fe0 2024-02-21T23:10:29.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.005+0000 7f7c6ca00640 1 --2- 172.21.15.28:0/522586396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c680c4fe0 0x7f7c681512d0 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c00c500 tx=0x7f7c5c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:29.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.005+0000 7f7c64c00640 1 -- 172.21.15.28:0/522586396 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c5c018070 con 0x7f7c680c4fe0 2024-02-21T23:10:29.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.005+0000 7f7c64c00640 1 -- 172.21.15.28:0/522586396 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c5c007500 con 0x7f7c680c4fe0 2024-02-21T23:10:29.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.005+0000 7f7c64c00640 1 -- 172.21.15.28:0/522586396 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c5c004a20 con 0x7f7c680c4fe0 2024-02-21T23:10:29.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.005+0000 7f7c6f400640 1 -- 172.21.15.28:0/522586396 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c680abea0 con 0x7f7c680c4fe0 2024-02-21T23:10:29.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.005+0000 7f7c6f400640 1 -- 172.21.15.28:0/522586396 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c680ac460 con 0x7f7c680c4fe0 2024-02-21T23:10:29.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.007+0000 7f7c6f400640 1 -- 172.21.15.28:0/522586396 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c68067f30 con 0x7f7c680c4fe0 2024-02-21T23:10:29.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.008+0000 7f7c64c00640 1 -- 172.21.15.28:0/522586396 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7c5c004bc0 con 0x7f7c680c4fe0 2024-02-21T23:10:29.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.008+0000 7f7c64c00640 1 --2- 172.21.15.28:0/522586396 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7c4806b4d0 0x7f7c4806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:29.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.009+0000 7f7c64c00640 1 -- 172.21.15.28:0/522586396 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 7216+0+0 (secure 0 0 0) 0x7f7c5c089b10 con 0x7f7c680c4fe0 2024-02-21T23:10:29.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.009+0000 7f7c67e00640 1 --2- 172.21.15.28:0/522586396 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7c4806b4d0 0x7f7c4806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:29.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.010+0000 7f7c67e00640 1 --2- 172.21.15.28:0/522586396 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7c4806b4d0 0x7f7c4806d990 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f7c68151ea0 tx=0x7f7c54008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:30.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.014+0000 7f7c64c00640 1 -- 172.21.15.28:0/522586396 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7c5c057d70 con 0x7f7c680c4fe0 2024-02-21T23:10:30.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.250+0000 7f7c6f400640 1 -- 172.21.15.28:0/522586396 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7c6806f1d0 con 0x7f7c680c4fe0 2024-02-21T23:10:30.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.251+0000 7f7c64c00640 1 -- 172.21.15.28:0/522586396 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 171 v171) v1 ==== 95+0+6197 (secure 0 0 0) 0x7f7c5c05be10 con 0x7f7c680c4fe0 2024-02-21T23:10:30.239 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:10:30.240 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":171,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9141,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2895215997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2895215997},{"type":"v1","addr":"172.21.15.28:6835","nonce":2895215997}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":166},{"gid":9153,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/2178509171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2178509171},{"type":"v1","addr":"172.21.15.28:6833","nonce":2178509171}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":169}],"filesystems":[{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:09:53.730296+0000","modified":"2024-02-21T23:10:21.591594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":156,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6392},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6392":{"gid":6392,"name":"d","rank":0,"incarnation":165,"state":"up:active","state_seq":11,"addr":"172.21.15.81:6837/521625945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":521625945},{"type":"v1","addr":"172.21.15.81:6837","nonce":521625945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11},{"mdsmap":{"epoch":171,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:10:05.843850+0000","modified":"2024-02-21T23:10:23.599068+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":157,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6389},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6389":{"gid":6389,"name":"b","rank":0,"incarnation":168,"state":"up:active","state_seq":12,"addr":"172.21.15.81:6835/1320495906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1320495906},{"type":"v1","addr":"172.21.15.81:6835","nonce":1320495906}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2024-02-21T23:10:30.240 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 171 2024-02-21T23:10:30.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.254+0000 7f7c6f400640 1 -- 172.21.15.28:0/522586396 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7c4806b4d0 msgr2=0x7f7c4806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:30.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.254+0000 7f7c6f400640 1 --2- 172.21.15.28:0/522586396 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7c4806b4d0 0x7f7c4806d990 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f7c68151ea0 tx=0x7f7c54008040 comp rx=0 tx=0).stop 2024-02-21T23:10:30.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.254+0000 7f7c6f400640 1 -- 172.21.15.28:0/522586396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c680c4fe0 msgr2=0x7f7c681512d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:30.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.254+0000 7f7c6f400640 1 --2- 172.21.15.28:0/522586396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c680c4fe0 0x7f7c681512d0 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c00c500 tx=0x7f7c5c00c9c0 comp rx=0 tx=0).stop 2024-02-21T23:10:30.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.254+0000 7f7c6f400640 1 -- 172.21.15.28:0/522586396 shutdown_connections 2024-02-21T23:10:30.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.254+0000 7f7c6f400640 1 --2- 172.21.15.28:0/522586396 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7c4806b4d0 0x7f7c4806d990 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:30.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.254+0000 7f7c6f400640 1 --2- 172.21.15.28:0/522586396 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7c68159400 0x7f7c68153aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:30.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.254+0000 7f7c6f400640 1 --2- 172.21.15.28:0/522586396 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7c681569d0 0x7f7c68153560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:30.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.254+0000 7f7c6f400640 1 --2- 172.21.15.28:0/522586396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c680c4fe0 0x7f7c681512d0 unknown :-1 s=CLOSED pgs=1809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:30.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.254+0000 7f7c6f400640 1 -- 172.21.15.28:0/522586396 >> 172.21.15.28:0/522586396 conn(0x7f7c680c67f0 msgr2=0x7f7c68155be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:30.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.254+0000 7f7c6f400640 1 -- 172.21.15.28:0/522586396 shutdown_connections 2024-02-21T23:10:30.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.254+0000 7f7c6f400640 1 -- 172.21.15.28:0/522586396 wait complete. 2024-02-21T23:10:30.261 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:10:30.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.493+0000 7fdc90c00640 1 Processor -- start 2024-02-21T23:10:30.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.493+0000 7fdc90c00640 1 -- start start 2024-02-21T23:10:30.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.494+0000 7fdc90c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fdc8c0c3490 0x7fdc8c06dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:30.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.495+0000 7fdc90c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc8c06e750 con 0x7fdc8c0a5f40 2024-02-21T23:10:30.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.495+0000 7fdc90c00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc8c06ea30 con 0x7fdc8c0c0270 2024-02-21T23:10:30.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.495+0000 7fdc90c00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc8c06eff0 con 0x7fdc8c0c3490 2024-02-21T23:10:30.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.495+0000 7fdc89400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fdc8c0a5f40 0x7fdc8c152140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:33426/0 (socket says 172.21.15.28:33426) 2024-02-21T23:10:30.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.495+0000 7fdc89400640 1 -- 172.21.15.28:0/505788772 learned_addr learned my addr 172.21.15.28:0/505788772 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:10:30.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.495+0000 7fdc89e00640 1 --2- 172.21.15.28:0/505788772 >> v2:172.21.15.81:3300/0 conn(0x7fdc8c0c3490 0x7fdc8c06dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:30.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.495+0000 7fdc88a00640 1 -- 172.21.15.28:0/505788772 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 60809538 0 0) 0x7fdc8c06e750 con 0x7fdc8c0a5f40 2024-02-21T23:10:30.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.495+0000 7fdc88a00640 1 -- 172.21.15.28:0/505788772 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc70003520 con 0x7fdc8c0a5f40 2024-02-21T23:10:30.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.496+0000 7fdc88a00640 1 -- 172.21.15.28:0/505788772 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3734386019 0 0) 0x7fdc70003520 con 0x7fdc8c0a5f40 2024-02-21T23:10:30.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.496+0000 7fdc88a00640 1 -- 172.21.15.28:0/505788772 >> v2:172.21.15.81:3300/0 conn(0x7fdc8c0c3490 msgr2=0x7fdc8c06dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:30.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.496+0000 7fdc88a00640 1 --2- 172.21.15.28:0/505788772 >> v2:172.21.15.81:3300/0 conn(0x7fdc8c0c3490 0x7fdc8c06dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:30.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.496+0000 7fdc88a00640 1 -- 172.21.15.28:0/505788772 >> v1:172.21.15.81:6789/0 conn(0x7fdc8c0c0270 legacy=0x7fdc8c0c06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:30.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.496+0000 7fdc88a00640 1 -- 172.21.15.28:0/505788772 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc8c06ed10 con 0x7fdc8c0a5f40 2024-02-21T23:10:30.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.496+0000 7fdc88a00640 1 -- 172.21.15.28:0/505788772 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fdc78001670 con 0x7fdc8c0a5f40 2024-02-21T23:10:30.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.496+0000 7fdc88a00640 1 -- 172.21.15.28:0/505788772 >> v1:172.21.15.28:6789/0 conn(0x7fdc8c0a5f40 legacy=0x7fdc8c152140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:30.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.496+0000 7fdc88a00640 1 --2- 172.21.15.28:0/505788772 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc70003a00 0x7fdc70003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:30.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.496+0000 7fdc88a00640 1 --2- 172.21.15.28:0/505788772 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc70004800 0x7fdc70004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:30.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.496+0000 7fdc88a00640 1 --2- 172.21.15.28:0/505788772 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdc70005210 0x7fdc7000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:30.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.496+0000 7fdc88a00640 1 -- 172.21.15.28:0/505788772 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc8c06eff0 con 0x7fdc70003a00 2024-02-21T23:10:30.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.496+0000 7fdc88a00640 1 -- 172.21.15.28:0/505788772 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7000c160 con 0x7fdc70004800 2024-02-21T23:10:30.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.496+0000 7fdc88a00640 1 -- 172.21.15.28:0/505788772 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7000c360 con 0x7fdc70005210 2024-02-21T23:10:30.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.496+0000 7fdc89e00640 1 --2- 172.21.15.28:0/505788772 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc70003a00 0x7fdc70003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:30.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.497+0000 7fdc89400640 1 --2- 172.21.15.28:0/505788772 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc70004800 0x7fdc70004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:30.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.497+0000 7fdc8a800640 1 --2- 172.21.15.28:0/505788772 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdc70005210 0x7fdc7000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:30.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.497+0000 7fdc89e00640 1 -- 172.21.15.28:0/505788772 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdc70005210 msgr2=0x7fdc7000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:30.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.497+0000 7fdc89e00640 1 --2- 172.21.15.28:0/505788772 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdc70005210 0x7fdc7000b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:30.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.497+0000 7fdc89e00640 1 -- 172.21.15.28:0/505788772 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc70004800 msgr2=0x7fdc70004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:30.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.497+0000 7fdc89e00640 1 --2- 172.21.15.28:0/505788772 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc70004800 0x7fdc70004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:30.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.497+0000 7fdc89e00640 1 -- 172.21.15.28:0/505788772 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc70003520 con 0x7fdc70003a00 2024-02-21T23:10:30.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.497+0000 7fdc89400640 1 --2- 172.21.15.28:0/505788772 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc70004800 0x7fdc70004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:10:30.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.497+0000 7fdc8a800640 1 -- 172.21.15.28:0/505788772 reap_dead start 2024-02-21T23:10:30.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.498+0000 7fdc89e00640 1 --2- 172.21.15.28:0/505788772 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc70003a00 0x7fdc70003e20 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c009a90 tx=0x7fdc70005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ce3ef6c5d6a16ed server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:30.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.498+0000 7fdc88a00640 1 -- 172.21.15.28:0/505788772 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc7c039070 con 0x7fdc70003a00 2024-02-21T23:10:30.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.498+0000 7fdc88a00640 1 -- 172.21.15.28:0/505788772 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc7c02c6b0 con 0x7fdc70003a00 2024-02-21T23:10:30.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.498+0000 7fdc88a00640 1 -- 172.21.15.28:0/505788772 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc7c009690 con 0x7fdc70003a00 2024-02-21T23:10:30.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.499+0000 7fdc90c00640 1 -- 172.21.15.28:0/505788772 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc70003a00 msgr2=0x7fdc70003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:30.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.499+0000 7fdc90c00640 1 --2- 172.21.15.28:0/505788772 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc70003a00 0x7fdc70003e20 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c009a90 tx=0x7fdc70005830 comp rx=0 tx=0).stop 2024-02-21T23:10:30.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.499+0000 7fdc90c00640 1 -- 172.21.15.28:0/505788772 shutdown_connections 2024-02-21T23:10:30.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.499+0000 7fdc90c00640 1 --2- 172.21.15.28:0/505788772 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc70003a00 0x7fdc70003e20 unknown :-1 s=CLOSED pgs=1811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:30.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.499+0000 7fdc90c00640 1 -- 172.21.15.28:0/505788772 >> 172.21.15.28:0/505788772 conn(0x7fdc8c0c67f0 msgr2=0x7fdc8c0c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:30.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.499+0000 7fdc90c00640 1 -- 172.21.15.28:0/505788772 shutdown_connections 2024-02-21T23:10:30.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.499+0000 7fdc90c00640 1 -- 172.21.15.28:0/505788772 wait complete. 2024-02-21T23:10:30.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.501+0000 7fdc90c00640 1 Processor -- start 2024-02-21T23:10:30.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.501+0000 7fdc90c00640 1 -- start start 2024-02-21T23:10:30.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.502+0000 7fdc90c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc70003a00 0x7fdc8c0a9a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:30.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.502+0000 7fdc90c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdc8c147b90 0x7fdc8c146a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:30.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.502+0000 7fdc90c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc8c147660 0x7fdc8c146f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:30.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.502+0000 7fdc90c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc8c0c3870 con 0x7fdc8c147660 2024-02-21T23:10:30.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.502+0000 7fdc90c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc8c0bfde0 con 0x7fdc70003a00 2024-02-21T23:10:30.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.502+0000 7fdc90c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc8c0c39f0 con 0x7fdc8c147b90 2024-02-21T23:10:30.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.502+0000 7fdc8a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc8c147660 0x7fdc8c146f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:30.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.502+0000 7fdc8a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc8c147660 0x7fdc8c146f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39442/0 (socket says 172.21.15.28:39442) 2024-02-21T23:10:30.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.502+0000 7fdc8a800640 1 -- 172.21.15.28:0/1236800481 learned_addr learned my addr 172.21.15.28:0/1236800481 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:30.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.502+0000 7fdc89e00640 1 --2- 172.21.15.28:0/1236800481 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc70003a00 0x7fdc8c0a9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:30.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.502+0000 7fdc89400640 1 --2- 172.21.15.28:0/1236800481 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdc8c147b90 0x7fdc8c146a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:30.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.502+0000 7fdc8a800640 1 -- 172.21.15.28:0/1236800481 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdc8c147b90 msgr2=0x7fdc8c146a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:30.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.502+0000 7fdc8a800640 1 --2- 172.21.15.28:0/1236800481 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdc8c147b90 0x7fdc8c146a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:30.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.502+0000 7fdc8a800640 1 -- 172.21.15.28:0/1236800481 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc70003a00 msgr2=0x7fdc8c0a9a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:30.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.502+0000 7fdc8a800640 1 --2- 172.21.15.28:0/1236800481 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc70003a00 0x7fdc8c0a9a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:30.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.502+0000 7fdc8a800640 1 -- 172.21.15.28:0/1236800481 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc8c14d5a0 con 0x7fdc8c147660 2024-02-21T23:10:30.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.502+0000 7fdc89400640 1 --2- 172.21.15.28:0/1236800481 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdc8c147b90 0x7fdc8c146a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:10:30.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.503+0000 7fdc8a800640 1 --2- 172.21.15.28:0/1236800481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc8c147660 0x7fdc8c146f70 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c0c4330 tx=0x7fdc8400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:30.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.503+0000 7fdc82000640 1 -- 172.21.15.28:0/1236800481 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc84018070 con 0x7fdc8c147660 2024-02-21T23:10:30.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.503+0000 7fdc82000640 1 -- 172.21.15.28:0/1236800481 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc84003b70 con 0x7fdc8c147660 2024-02-21T23:10:30.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.503+0000 7fdc82000640 1 -- 172.21.15.28:0/1236800481 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc84005070 con 0x7fdc8c147660 2024-02-21T23:10:30.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.503+0000 7fdc90c00640 1 -- 172.21.15.28:0/1236800481 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc8c1eb940 con 0x7fdc8c147660 2024-02-21T23:10:30.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.503+0000 7fdc90c00640 1 -- 172.21.15.28:0/1236800481 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc8c1ebbe0 con 0x7fdc8c147660 2024-02-21T23:10:30.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.504+0000 7fdc90c00640 1 -- 172.21.15.28:0/1236800481 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc8c067f30 con 0x7fdc8c147660 2024-02-21T23:10:30.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.505+0000 7fdc82000640 1 -- 172.21.15.28:0/1236800481 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdc8401b070 con 0x7fdc8c147660 2024-02-21T23:10:30.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.506+0000 7fdc82000640 1 --2- 172.21.15.28:0/1236800481 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdc6c06b500 0x7fdc6c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:30.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.506+0000 7fdc82000640 1 -- 172.21.15.28:0/1236800481 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 7216+0+0 (secure 0 0 0) 0x7fdc84089eb0 con 0x7fdc8c147660 2024-02-21T23:10:30.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.507+0000 7fdc89e00640 1 --2- 172.21.15.28:0/1236800481 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdc6c06b500 0x7fdc6c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:30.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.508+0000 7fdc89e00640 1 --2- 172.21.15.28:0/1236800481 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdc6c06b500 0x7fdc6c06d9c0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c036000 tx=0x7fdc7c030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:30.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.511+0000 7fdc82000640 1 -- 172.21.15.28:0/1236800481 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fdc84058760 con 0x7fdc8c147660 2024-02-21T23:10:30.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.752+0000 7fdc90c00640 1 -- 172.21.15.28:0/1236800481 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdc8c0732a0 con 0x7fdc8c147660 2024-02-21T23:10:30.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.753+0000 7fdc82000640 1 -- 172.21.15.28:0/1236800481 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 171 v171) v1 ==== 95+0+6197 (secure 0 0 0) 0x7fdc84058060 con 0x7fdc8c147660 2024-02-21T23:10:30.741 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:10:30.742 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":171,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9141,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2895215997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2895215997},{"type":"v1","addr":"172.21.15.28:6835","nonce":2895215997}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":166},{"gid":9153,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/2178509171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2178509171},{"type":"v1","addr":"172.21.15.28:6833","nonce":2178509171}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":169}],"filesystems":[{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:09:53.730296+0000","modified":"2024-02-21T23:10:21.591594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":156,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6392},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6392":{"gid":6392,"name":"d","rank":0,"incarnation":165,"state":"up:active","state_seq":11,"addr":"172.21.15.81:6837/521625945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":521625945},{"type":"v1","addr":"172.21.15.81:6837","nonce":521625945}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[24],"metadata_pool":23,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11},{"mdsmap":{"epoch":171,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:10:05.843850+0000","modified":"2024-02-21T23:10:23.599068+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":157,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6389},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6389":{"gid":6389,"name":"b","rank":0,"incarnation":168,"state":"up:active","state_seq":12,"addr":"172.21.15.81:6835/1320495906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1320495906},{"type":"v1","addr":"172.21.15.81:6835","nonce":1320495906}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2024-02-21T23:10:30.742 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 171 2024-02-21T23:10:30.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.756+0000 7fdc90c00640 1 -- 172.21.15.28:0/1236800481 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdc6c06b500 msgr2=0x7fdc6c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:30.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.756+0000 7fdc90c00640 1 --2- 172.21.15.28:0/1236800481 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdc6c06b500 0x7fdc6c06d9c0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c036000 tx=0x7fdc7c030000 comp rx=0 tx=0).stop 2024-02-21T23:10:30.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.756+0000 7fdc90c00640 1 -- 172.21.15.28:0/1236800481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc8c147660 msgr2=0x7fdc8c146f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:30.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.756+0000 7fdc90c00640 1 --2- 172.21.15.28:0/1236800481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc8c147660 0x7fdc8c146f70 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c0c4330 tx=0x7fdc8400dc00 comp rx=0 tx=0).stop 2024-02-21T23:10:30.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.756+0000 7fdc90c00640 1 -- 172.21.15.28:0/1236800481 shutdown_connections 2024-02-21T23:10:30.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.756+0000 7fdc90c00640 1 --2- 172.21.15.28:0/1236800481 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdc6c06b500 0x7fdc6c06d9c0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:30.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.756+0000 7fdc90c00640 1 --2- 172.21.15.28:0/1236800481 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdc8c147b90 0x7fdc8c146a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:30.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.756+0000 7fdc90c00640 1 --2- 172.21.15.28:0/1236800481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc8c147660 0x7fdc8c146f70 unknown :-1 s=CLOSED pgs=1812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:30.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.756+0000 7fdc90c00640 1 --2- 172.21.15.28:0/1236800481 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc70003a00 0x7fdc8c0a9a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:30.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.756+0000 7fdc90c00640 1 -- 172.21.15.28:0/1236800481 >> 172.21.15.28:0/1236800481 conn(0x7fdc8c0c67f0 msgr2=0x7fdc8c0c43e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:30.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.756+0000 7fdc90c00640 1 -- 172.21.15.28:0/1236800481 shutdown_connections 2024-02-21T23:10:30.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.757+0000 7fdc90c00640 1 -- 172.21.15.28:0/1236800481 wait complete. 2024-02-21T23:10:30.763 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:10:30.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.984+0000 7f203a000640 1 Processor -- start 2024-02-21T23:10:30.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.985+0000 7f203a000640 1 -- start start 2024-02-21T23:10:30.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.986+0000 7f203a000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f20340c75e0 0x7f2034158f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:30.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.986+0000 7f203a000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2034152c80 con 0x7f20340c7180 2024-02-21T23:10:30.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.986+0000 7f203a000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2034159440 con 0x7f20341528a0 2024-02-21T23:10:30.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.986+0000 7f203a000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20340c7a10 con 0x7f20340c75e0 2024-02-21T23:10:30.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.987+0000 7f2032a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f20340c7180 0x7f20341587f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:33436/0 (socket says 172.21.15.28:33436) 2024-02-21T23:10:30.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.987+0000 7f2032a00640 1 -- 172.21.15.28:0/2146711546 learned_addr learned my addr 172.21.15.28:0/2146711546 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:10:30.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.987+0000 7f2033e00640 1 --2- 172.21.15.28:0/2146711546 >> v2:172.21.15.81:3300/0 conn(0x7f20340c75e0 0x7f2034158f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:30.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.987+0000 7f2032000640 1 -- 172.21.15.28:0/2146711546 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2567673834 0 0) 0x7f2034152c80 con 0x7f20340c7180 2024-02-21T23:10:30.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.987+0000 7f2032000640 1 -- 172.21.15.28:0/2146711546 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2018003520 con 0x7f20340c7180 2024-02-21T23:10:30.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.987+0000 7f2032000640 1 -- 172.21.15.28:0/2146711546 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 179565009 0 0) 0x7f2018003520 con 0x7f20340c7180 2024-02-21T23:10:30.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.987+0000 7f2032000640 1 -- 172.21.15.28:0/2146711546 >> v2:172.21.15.81:3300/0 conn(0x7f20340c75e0 msgr2=0x7f2034158f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:30.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.987+0000 7f2032000640 1 --2- 172.21.15.28:0/2146711546 >> v2:172.21.15.81:3300/0 conn(0x7f20340c75e0 0x7f2034158f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:30.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.987+0000 7f2032000640 1 -- 172.21.15.28:0/2146711546 >> v1:172.21.15.81:6789/0 conn(0x7f20341528a0 legacy=0x7f20340c6650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:30.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.987+0000 7f2032000640 1 -- 172.21.15.28:0/2146711546 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2034155300 con 0x7f20340c7180 2024-02-21T23:10:30.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.987+0000 7f2032000640 1 -- 172.21.15.28:0/2146711546 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f20200019f0 con 0x7f20340c7180 2024-02-21T23:10:30.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.988+0000 7f2032000640 1 -- 172.21.15.28:0/2146711546 >> v1:172.21.15.28:6789/0 conn(0x7f20340c7180 legacy=0x7f20341587f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:30.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.988+0000 7f2032000640 1 --2- 172.21.15.28:0/2146711546 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20180039b0 0x7f2018003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:30.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.988+0000 7f2032000640 1 --2- 172.21.15.28:0/2146711546 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2018004800 0x7f2018004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:30.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.988+0000 7f2032000640 1 --2- 172.21.15.28:0/2146711546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20180051c0 0x7f201800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:30.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.988+0000 7f2032000640 1 -- 172.21.15.28:0/2146711546 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20340c7a10 con 0x7f20180051c0 2024-02-21T23:10:30.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.988+0000 7f2032000640 1 -- 172.21.15.28:0/2146711546 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f201800c160 con 0x7f20180039b0 2024-02-21T23:10:30.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.988+0000 7f2032000640 1 -- 172.21.15.28:0/2146711546 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f201800c360 con 0x7f2018004800 2024-02-21T23:10:30.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.988+0000 7f2033e00640 1 --2- 172.21.15.28:0/2146711546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20180051c0 0x7f201800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:30.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.988+0000 7f2032a00640 1 --2- 172.21.15.28:0/2146711546 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2018004800 0x7f2018004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:30.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.988+0000 7f2033400640 1 --2- 172.21.15.28:0/2146711546 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20180039b0 0x7f2018003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:30.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.988+0000 7f2033e00640 1 -- 172.21.15.28:0/2146711546 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2018004800 msgr2=0x7f2018004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:30.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.988+0000 7f2033e00640 1 --2- 172.21.15.28:0/2146711546 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2018004800 0x7f2018004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:30.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.988+0000 7f2033e00640 1 -- 172.21.15.28:0/2146711546 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20180039b0 msgr2=0x7f2018003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:30.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.988+0000 7f2033e00640 1 --2- 172.21.15.28:0/2146711546 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20180039b0 0x7f2018003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:30.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.988+0000 7f2033e00640 1 -- 172.21.15.28:0/2146711546 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2018003520 con 0x7f20180051c0 2024-02-21T23:10:30.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.989+0000 7f2033400640 1 --2- 172.21.15.28:0/2146711546 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f20180039b0 0x7f2018003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:10:30.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.989+0000 7f2033e00640 1 -- 172.21.15.28:0/2146711546 reap_dead start 2024-02-21T23:10:30.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.990+0000 7f2033e00640 1 --2- 172.21.15.28:0/2146711546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20180051c0 0x7f201800b9c0 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7f202c009ac0 tx=0x7f2018005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1b5d1adc8dd17b7 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:30.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.990+0000 7f2032000640 1 -- 172.21.15.28:0/2146711546 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f202c009690 con 0x7f20180051c0 2024-02-21T23:10:30.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.990+0000 7f2032000640 1 -- 172.21.15.28:0/2146711546 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f202c004020 con 0x7f20180051c0 2024-02-21T23:10:30.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.990+0000 7f2032000640 1 -- 172.21.15.28:0/2146711546 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f202c004410 con 0x7f20180051c0 2024-02-21T23:10:30.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.990+0000 7f203a000640 1 -- 172.21.15.28:0/2146711546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20180051c0 msgr2=0x7f201800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:30.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.990+0000 7f203a000640 1 --2- 172.21.15.28:0/2146711546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20180051c0 0x7f201800b9c0 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7f202c009ac0 tx=0x7f2018005830 comp rx=0 tx=0).stop 2024-02-21T23:10:30.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.990+0000 7f203a000640 1 -- 172.21.15.28:0/2146711546 shutdown_connections 2024-02-21T23:10:30.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.990+0000 7f203a000640 1 --2- 172.21.15.28:0/2146711546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20180051c0 0x7f201800b9c0 unknown :-1 s=CLOSED pgs=1814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:30.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.990+0000 7f203a000640 1 -- 172.21.15.28:0/2146711546 >> 172.21.15.28:0/2146711546 conn(0x7f20340aa4c0 msgr2=0x7f20340aa8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:30.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.991+0000 7f203a000640 1 -- 172.21.15.28:0/2146711546 shutdown_connections 2024-02-21T23:10:30.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.991+0000 7f203a000640 1 -- 172.21.15.28:0/2146711546 wait complete. 2024-02-21T23:10:30.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.992+0000 7f203a000640 1 Processor -- start 2024-02-21T23:10:30.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.992+0000 7f203a000640 1 -- start start 2024-02-21T23:10:30.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.993+0000 7f203a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20180051c0 0x7f203414bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:30.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.993+0000 7f203a000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2034154a20 0x7f2034153000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:30.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.993+0000 7f203a000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2034153540 0x7f20341ee0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:30.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.993+0000 7f203a000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2034155ba0 con 0x7f20180051c0 2024-02-21T23:10:30.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.993+0000 7f203a000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20341558c0 con 0x7f2034154a20 2024-02-21T23:10:30.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.993+0000 7f203a000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2034155e80 con 0x7f2034153540 2024-02-21T23:10:30.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.994+0000 7f2033400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20180051c0 0x7f203414bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:30.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.994+0000 7f2033400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20180051c0 0x7f203414bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39462/0 (socket says 172.21.15.28:39462) 2024-02-21T23:10:30.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.994+0000 7f2033400640 1 -- 172.21.15.28:0/1855248165 learned_addr learned my addr 172.21.15.28:0/1855248165 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:30.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.994+0000 7f2033e00640 1 --2- 172.21.15.28:0/1855248165 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2034153540 0x7f20341ee0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:30.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.994+0000 7f2032a00640 1 --2- 172.21.15.28:0/1855248165 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2034154a20 0x7f2034153000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:30.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.994+0000 7f2033400640 1 -- 172.21.15.28:0/1855248165 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2034153540 msgr2=0x7f20341ee0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:30.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.994+0000 7f2033400640 1 --2- 172.21.15.28:0/1855248165 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2034153540 0x7f20341ee0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:30.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.994+0000 7f2033400640 1 -- 172.21.15.28:0/1855248165 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2034154a20 msgr2=0x7f2034153000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:30.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.994+0000 7f2033400640 1 --2- 172.21.15.28:0/1855248165 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2034154a20 0x7f2034153000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:30.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.994+0000 7f2033400640 1 -- 172.21.15.28:0/1855248165 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20341539c0 con 0x7f20180051c0 2024-02-21T23:10:30.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.994+0000 7f2033400640 1 --2- 172.21.15.28:0/1855248165 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20180051c0 0x7f203414bb50 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f202400d7a0 tx=0x7f202400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:30.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.995+0000 7f202b400640 1 -- 172.21.15.28:0/1855248165 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2024005270 con 0x7f20180051c0 2024-02-21T23:10:30.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.995+0000 7f202b400640 1 -- 172.21.15.28:0/1855248165 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2024017070 con 0x7f20180051c0 2024-02-21T23:10:30.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.995+0000 7f202b400640 1 -- 172.21.15.28:0/1855248165 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f202400b6b0 con 0x7f20180051c0 2024-02-21T23:10:30.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.995+0000 7f203a000640 1 -- 172.21.15.28:0/1855248165 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f203414b870 con 0x7f20180051c0 2024-02-21T23:10:30.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.995+0000 7f203a000640 1 -- 172.21.15.28:0/1855248165 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20341eb9f0 con 0x7f20180051c0 2024-02-21T23:10:30.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.996+0000 7f203a000640 1 -- 172.21.15.28:0/1855248165 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2034067f30 con 0x7f20180051c0 2024-02-21T23:10:30.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.997+0000 7f202b400640 1 -- 172.21.15.28:0/1855248165 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f202400b850 con 0x7f20180051c0 2024-02-21T23:10:30.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.998+0000 7f202b400640 1 --2- 172.21.15.28:0/1855248165 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f201406b4d0 0x7f201406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:30.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.998+0000 7f202b400640 1 -- 172.21.15.28:0/1855248165 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 7216+0+0 (secure 0 0 0) 0x7f202408a5c0 con 0x7f20180051c0 2024-02-21T23:10:30.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:30.998+0000 7f2032a00640 1 --2- 172.21.15.28:0/1855248165 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f201406b4d0 0x7f201406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:30.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.000+0000 7f2032a00640 1 --2- 172.21.15.28:0/1855248165 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f201406b4d0 0x7f201406d990 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f20200013d0 tx=0x7f2020009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:30.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.003+0000 7f202b400640 1 -- 172.21.15.28:0/1855248165 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f20341539c0 con 0x7f20180051c0 2024-02-21T23:10:31.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.203+0000 7f203a000640 1 -- 172.21.15.28:0/1855248165 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f20340707a0 con 0x7f20180051c0 2024-02-21T23:10:31.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.207+0000 7f202b400640 1 -- 172.21.15.28:0/1855248165 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v159) v1 ==== 74+0+21003 (secure 0 0 0) 0x7f2024058ef0 con 0x7f20180051c0 2024-02-21T23:10:31.195 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:10:31.196 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":159,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:10:27.626041+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:09:51.634291+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"150","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_data","create_time":"2024-02-21T23:09:52.736252+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"151","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs2_metadata","create_time":"2024-02-21T23:10:04.095451+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"154","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs2_data","create_time":"2024-02-21T23:10:04.836982+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"26.29","mappings":[{"from":1,"to":0}]},{"pgid":"26.36","mappings":[{"from":3,"to":0}]},{"pgid":"26.3f","mappings":[{"from":1,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6833/2975378719":"2024-02-22T23:10:21.581948+0000","172.21.15.28:6832/2975378719":"2024-02-22T23:10:21.581948+0000","172.21.15.28:6835/2928639298":"2024-02-22T23:10:19.550812+0000","172.21.15.28:6834/2928639298":"2024-02-22T23:10:19.550812+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:10:31.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.210+0000 7f203a000640 1 -- 172.21.15.28:0/1855248165 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f201406b4d0 msgr2=0x7f201406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:31.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.210+0000 7f203a000640 1 --2- 172.21.15.28:0/1855248165 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f201406b4d0 0x7f201406d990 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f20200013d0 tx=0x7f2020009040 comp rx=0 tx=0).stop 2024-02-21T23:10:31.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.210+0000 7f203a000640 1 -- 172.21.15.28:0/1855248165 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20180051c0 msgr2=0x7f203414bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:31.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.210+0000 7f203a000640 1 --2- 172.21.15.28:0/1855248165 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20180051c0 0x7f203414bb50 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f202400d7a0 tx=0x7f202400dc60 comp rx=0 tx=0).stop 2024-02-21T23:10:31.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.210+0000 7f203a000640 1 -- 172.21.15.28:0/1855248165 shutdown_connections 2024-02-21T23:10:31.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.210+0000 7f203a000640 1 --2- 172.21.15.28:0/1855248165 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f201406b4d0 0x7f201406d990 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:31.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.210+0000 7f203a000640 1 --2- 172.21.15.28:0/1855248165 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2034153540 0x7f20341ee0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:31.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.210+0000 7f203a000640 1 --2- 172.21.15.28:0/1855248165 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2034154a20 0x7f2034153000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:31.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.210+0000 7f203a000640 1 --2- 172.21.15.28:0/1855248165 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20180051c0 0x7f203414bb50 unknown :-1 s=CLOSED pgs=1815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:31.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.210+0000 7f203a000640 1 -- 172.21.15.28:0/1855248165 >> 172.21.15.28:0/1855248165 conn(0x7f20340aa4c0 msgr2=0x7f203414f560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:31.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.210+0000 7f203a000640 1 -- 172.21.15.28:0/1855248165 shutdown_connections 2024-02-21T23:10:31.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.210+0000 7f203a000640 1 -- 172.21.15.28:0/1855248165 wait complete. 2024-02-21T23:10:31.218 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-02-21T23:10:31.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.454+0000 7fc792c00640 1 Processor -- start 2024-02-21T23:10:31.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.454+0000 7fc792c00640 1 -- start start 2024-02-21T23:10:31.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.455+0000 7fc792c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc78c06d410 0x7fc78c0a8e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:31.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.455+0000 7fc792c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc78c0c43e0 0x7fc78c0c5a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:31.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.455+0000 7fc792c00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc78c149c10 con 0x7fc78c149830 2024-02-21T23:10:31.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.455+0000 7fc792c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc78c1537d0 con 0x7fc78c0c43e0 2024-02-21T23:10:31.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.455+0000 7fc792c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc78c153ab0 con 0x7fc78c06d410 2024-02-21T23:10:31.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.455+0000 7fc792200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc78c0c43e0 0x7fc78c0c5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:31.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.456+0000 7fc792200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc78c0c43e0 0x7fc78c0c5a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39474/0 (socket says 172.21.15.28:39474) 2024-02-21T23:10:31.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.456+0000 7fc792200640 1 -- 172.21.15.28:0/470783798 learned_addr learned my addr 172.21.15.28:0/470783798 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:31.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.456+0000 7fc790e00640 1 --2- 172.21.15.28:0/470783798 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc78c06d410 0x7fc78c0a8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:31.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.456+0000 7fc792200640 1 -- 172.21.15.28:0/470783798 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc78c06d410 msgr2=0x7fc78c0a8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:31.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.456+0000 7fc792200640 1 --2- 172.21.15.28:0/470783798 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc78c06d410 0x7fc78c0a8e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:31.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.456+0000 7fc792200640 1 -- 172.21.15.28:0/470783798 >> v1:172.21.15.81:6789/0 conn(0x7fc78c149830 legacy=0x7fc78c0c0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:31.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.456+0000 7fc792200640 1 -- 172.21.15.28:0/470783798 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc78c0c4fa0 con 0x7fc78c0c43e0 2024-02-21T23:10:31.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.457+0000 7fc792200640 1 --2- 172.21.15.28:0/470783798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc78c0c43e0 0x7fc78c0c5a80 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7fc784009e50 tx=0x7fc78402bff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=780eb6e13bd80514 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:31.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.458+0000 7fc78be00640 1 -- 172.21.15.28:0/470783798 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc78402cb90 con 0x7fc78c0c43e0 2024-02-21T23:10:31.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.458+0000 7fc78be00640 1 -- 172.21.15.28:0/470783798 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc78402cd30 con 0x7fc78c0c43e0 2024-02-21T23:10:31.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.458+0000 7fc78be00640 1 -- 172.21.15.28:0/470783798 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc784034610 con 0x7fc78c0c43e0 2024-02-21T23:10:31.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.458+0000 7fc792c00640 1 -- 172.21.15.28:0/470783798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc78c0c43e0 msgr2=0x7fc78c0c5a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:31.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.458+0000 7fc792c00640 1 --2- 172.21.15.28:0/470783798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc78c0c43e0 0x7fc78c0c5a80 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7fc784009e50 tx=0x7fc78402bff0 comp rx=0 tx=0).stop 2024-02-21T23:10:31.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.458+0000 7fc792c00640 1 -- 172.21.15.28:0/470783798 shutdown_connections 2024-02-21T23:10:31.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.458+0000 7fc792c00640 1 --2- 172.21.15.28:0/470783798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc78c0c43e0 0x7fc78c0c5a80 unknown :-1 s=CLOSED pgs=1816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:31.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.458+0000 7fc792c00640 1 --2- 172.21.15.28:0/470783798 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc78c06d410 0x7fc78c0a8e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:31.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.458+0000 7fc792c00640 1 -- 172.21.15.28:0/470783798 >> 172.21.15.28:0/470783798 conn(0x7fc78c0a8320 msgr2=0x7fc78c0a8730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:31.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.458+0000 7fc792c00640 1 -- 172.21.15.28:0/470783798 shutdown_connections 2024-02-21T23:10:31.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.458+0000 7fc792c00640 1 -- 172.21.15.28:0/470783798 wait complete. 2024-02-21T23:10:31.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.460+0000 7fc792c00640 1 Processor -- start 2024-02-21T23:10:31.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.460+0000 7fc792c00640 1 -- start start 2024-02-21T23:10:31.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.461+0000 7fc792c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc78c06d410 0x7fc78c1eb620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:31.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.461+0000 7fc792c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc78c0c43e0 0x7fc78c071f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:31.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.461+0000 7fc792c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc78c149830 0x7fc78c1dfcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:31.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.461+0000 7fc792c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc78c0c15f0 con 0x7fc78c06d410 2024-02-21T23:10:31.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.461+0000 7fc792c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc78c0c5560 con 0x7fc78c0c43e0 2024-02-21T23:10:31.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.461+0000 7fc792c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc78c0c1770 con 0x7fc78c149830 2024-02-21T23:10:31.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.461+0000 7fc791800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc78c06d410 0x7fc78c1eb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:31.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.461+0000 7fc791800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc78c06d410 0x7fc78c1eb620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39484/0 (socket says 172.21.15.28:39484) 2024-02-21T23:10:31.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.461+0000 7fc791800640 1 -- 172.21.15.28:0/1275529762 learned_addr learned my addr 172.21.15.28:0/1275529762 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:31.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.462+0000 7fc792200640 1 --2- 172.21.15.28:0/1275529762 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc78c149830 0x7fc78c1dfcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:31.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.462+0000 7fc790e00640 1 --2- 172.21.15.28:0/1275529762 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc78c0c43e0 0x7fc78c071f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:31.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.462+0000 7fc791800640 1 -- 172.21.15.28:0/1275529762 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc78c149830 msgr2=0x7fc78c1dfcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:31.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.462+0000 7fc791800640 1 --2- 172.21.15.28:0/1275529762 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc78c149830 0x7fc78c1dfcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:31.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.462+0000 7fc791800640 1 -- 172.21.15.28:0/1275529762 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc78c0c43e0 msgr2=0x7fc78c071f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:31.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.462+0000 7fc791800640 1 --2- 172.21.15.28:0/1275529762 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc78c0c43e0 0x7fc78c071f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:31.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.462+0000 7fc791800640 1 -- 172.21.15.28:0/1275529762 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc78c153f00 con 0x7fc78c06d410 2024-02-21T23:10:31.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.462+0000 7fc791800640 1 --2- 172.21.15.28:0/1275529762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc78c06d410 0x7fc78c1eb620 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7fc78000c430 tx=0x7fc78000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:31.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.462+0000 7fc789600640 1 -- 172.21.15.28:0/1275529762 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc780017070 con 0x7fc78c06d410 2024-02-21T23:10:31.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.463+0000 7fc789600640 1 -- 172.21.15.28:0/1275529762 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7800052d0 con 0x7fc78c06d410 2024-02-21T23:10:31.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.463+0000 7fc789600640 1 -- 172.21.15.28:0/1275529762 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc780003830 con 0x7fc78c06d410 2024-02-21T23:10:31.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.463+0000 7fc792c00640 1 -- 172.21.15.28:0/1275529762 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc78c1541e0 con 0x7fc78c06d410 2024-02-21T23:10:31.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.463+0000 7fc792c00640 1 -- 172.21.15.28:0/1275529762 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc78c1547a0 con 0x7fc78c06d410 2024-02-21T23:10:31.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.464+0000 7fc792c00640 1 -- 172.21.15.28:0/1275529762 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc78c0c73f0 con 0x7fc78c06d410 2024-02-21T23:10:31.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.465+0000 7fc789600640 1 -- 172.21.15.28:0/1275529762 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc78001a070 con 0x7fc78c06d410 2024-02-21T23:10:31.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.466+0000 7fc789600640 1 --2- 172.21.15.28:0/1275529762 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc76c06b4d0 0x7fc76c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:31.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.466+0000 7fc789600640 1 -- 172.21.15.28:0/1275529762 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(159..159 src has 1..159) v4 ==== 7216+0+0 (secure 0 0 0) 0x7fc780089b10 con 0x7fc78c06d410 2024-02-21T23:10:31.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.466+0000 7fc790e00640 1 --2- 172.21.15.28:0/1275529762 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc76c06b4d0 0x7fc76c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:31.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.468+0000 7fc790e00640 1 --2- 172.21.15.28:0/1275529762 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc76c06b4d0 0x7fc76c06d990 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fc778006fd0 tx=0x7fc778008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:31.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.470+0000 7fc789600640 1 -- 172.21.15.28:0/1275529762 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc78c0c73f0 con 0x7fc78c06d410 2024-02-21T23:10:31.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.710+0000 7fc792c00640 1 -- 172.21.15.28:0/1275529762 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fc78c04f290 con 0x7fc78c06d410 2024-02-21T23:10:31.699 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-02-21T23:10:31.712+0000 7f5ddc400640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-21T23:10:31.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.715+0000 7fc789600640 1 -- 172.21.15.28:0/1275529762 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v172) v1 ==== 161+0+0 (secure 0 0 0) 0x7fc7800583c0 con 0x7fc78c06d410 2024-02-21T23:10:31.703 INFO:teuthology.orchestra.run.smithi028.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-02-21T23:10:31.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.719+0000 7fc792c00640 1 -- 172.21.15.28:0/1275529762 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc76c06b4d0 msgr2=0x7fc76c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:31.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.719+0000 7fc792c00640 1 --2- 172.21.15.28:0/1275529762 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc76c06b4d0 0x7fc76c06d990 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fc778006fd0 tx=0x7fc778008040 comp rx=0 tx=0).stop 2024-02-21T23:10:31.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.719+0000 7fc792c00640 1 -- 172.21.15.28:0/1275529762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc78c06d410 msgr2=0x7fc78c1eb620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:31.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.719+0000 7fc792c00640 1 --2- 172.21.15.28:0/1275529762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc78c06d410 0x7fc78c1eb620 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7fc78000c430 tx=0x7fc78000c8f0 comp rx=0 tx=0).stop 2024-02-21T23:10:31.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.720+0000 7fc792c00640 1 -- 172.21.15.28:0/1275529762 shutdown_connections 2024-02-21T23:10:31.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.720+0000 7fc792c00640 1 --2- 172.21.15.28:0/1275529762 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc76c06b4d0 0x7fc76c06d990 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:31.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.720+0000 7fc792c00640 1 --2- 172.21.15.28:0/1275529762 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc78c149830 0x7fc78c1dfcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:31.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.720+0000 7fc792c00640 1 --2- 172.21.15.28:0/1275529762 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc78c0c43e0 0x7fc78c071f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:31.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.720+0000 7fc792c00640 1 --2- 172.21.15.28:0/1275529762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc78c06d410 0x7fc78c1eb620 unknown :-1 s=CLOSED pgs=1817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:31.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.720+0000 7fc792c00640 1 -- 172.21.15.28:0/1275529762 >> 172.21.15.28:0/1275529762 conn(0x7fc78c0a8320 msgr2=0x7fc78c1522e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:31.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.721+0000 7fc792c00640 1 -- 172.21.15.28:0/1275529762 shutdown_connections 2024-02-21T23:10:31.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.721+0000 7fc792c00640 1 -- 172.21.15.28:0/1275529762 wait complete. 2024-02-21T23:10:31.721 INFO:tasks.ceph.mds.d.smithi081.stderr: -223> 2024-02-21T23:10:25.216+0000 7f56b0800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-21T23:10:31.730 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-02-21T23:10:31.768 INFO:tasks.ceph.mds.d.smithi081.stdout:starting mds.d at 2024-02-21T23:10:31.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.952+0000 7fdef3e00640 1 Processor -- start 2024-02-21T23:10:31.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.952+0000 7fdef3e00640 1 -- start start 2024-02-21T23:10:31.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.954+0000 7fdef3e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdef40c1080 0x7fdef4159c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:31.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.954+0000 7fdef3e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdef415a1a0 con 0x7fdef415d170 2024-02-21T23:10:31.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.954+0000 7fdef3e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdef40c1590 con 0x7fdef4153470 2024-02-21T23:10:31.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.954+0000 7fdef3e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdef415cce0 con 0x7fdef40c1080 2024-02-21T23:10:31.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.954+0000 7fdef3400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fdef415d170 0x7fdef406ae70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:33438/0 (socket says 172.21.15.28:33438) 2024-02-21T23:10:31.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.954+0000 7fdef3400640 1 -- 172.21.15.28:0/541666610 learned_addr learned my addr 172.21.15.28:0/541666610 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:10:31.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.954+0000 7fdef2000640 1 --2- 172.21.15.28:0/541666610 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdef40c1080 0x7fdef4159c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:31.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.955+0000 7fdef1600640 1 -- 172.21.15.28:0/541666610 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3720627910 0 0) 0x7fdeec001490 con 0x7fdef415d170 2024-02-21T23:10:31.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.955+0000 7fdef1600640 1 -- 172.21.15.28:0/541666610 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fded8003520 con 0x7fdef415d170 2024-02-21T23:10:31.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.955+0000 7fdef1600640 1 -- 172.21.15.28:0/541666610 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3055001214 0 0) 0x7fdef40c1590 con 0x7fdef4153470 2024-02-21T23:10:31.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.955+0000 7fdef1600640 1 -- 172.21.15.28:0/541666610 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdeec001490 con 0x7fdef4153470 2024-02-21T23:10:31.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.955+0000 7fdef1600640 1 -- 172.21.15.28:0/541666610 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3864500071 0 0) 0x7fded8003520 con 0x7fdef415d170 2024-02-21T23:10:31.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.955+0000 7fdef1600640 1 -- 172.21.15.28:0/541666610 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdef40c1080 msgr2=0x7fdef4159c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:31.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.955+0000 7fdef1600640 1 --2- 172.21.15.28:0/541666610 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdef40c1080 0x7fdef4159c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:31.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.955+0000 7fdef1600640 1 -- 172.21.15.28:0/541666610 >> v1:172.21.15.81:6789/0 conn(0x7fdef4153470 legacy=0x7fdef40c4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:31.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.955+0000 7fdef1600640 1 -- 172.21.15.28:0/541666610 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdef40c1870 con 0x7fdef415d170 2024-02-21T23:10:31.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.955+0000 7fdef1600640 1 -- 172.21.15.28:0/541666610 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fdeec001bc0 con 0x7fdef415d170 2024-02-21T23:10:31.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.955+0000 7fdef1600640 1 -- 172.21.15.28:0/541666610 >> v1:172.21.15.28:6789/0 conn(0x7fdef415d170 legacy=0x7fdef406ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:31.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.955+0000 7fdef1600640 1 --2- 172.21.15.28:0/541666610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fded8003b70 0x7fded8003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:31.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.955+0000 7fdef1600640 1 --2- 172.21.15.28:0/541666610 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fded8004800 0x7fded8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:31.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.955+0000 7fdef1600640 1 --2- 172.21.15.28:0/541666610 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdef40c1080 0x7fded80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:31.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.955+0000 7fdef1600640 1 -- 172.21.15.28:0/541666610 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef415cce0 con 0x7fded8003b70 2024-02-21T23:10:31.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.955+0000 7fdef1600640 1 -- 172.21.15.28:0/541666610 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fded800ba70 con 0x7fded8004800 2024-02-21T23:10:31.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.955+0000 7fdef1600640 1 -- 172.21.15.28:0/541666610 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fded800bc70 con 0x7fdef40c1080 2024-02-21T23:10:31.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.956+0000 7fdef2a00640 1 --2- 172.21.15.28:0/541666610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fded8003b70 0x7fded8003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:31.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.956+0000 7fdef3400640 1 --2- 172.21.15.28:0/541666610 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdef40c1080 0x7fded80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:31.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.956+0000 7fdef2000640 1 --2- 172.21.15.28:0/541666610 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fded8004800 0x7fded8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:31.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.956+0000 7fdef2a00640 1 -- 172.21.15.28:0/541666610 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdef40c1080 msgr2=0x7fded80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:31.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.956+0000 7fdef2a00640 1 --2- 172.21.15.28:0/541666610 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdef40c1080 0x7fded80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:31.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.956+0000 7fdef2a00640 1 -- 172.21.15.28:0/541666610 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fded8004800 msgr2=0x7fded8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:31.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.956+0000 7fdef2a00640 1 --2- 172.21.15.28:0/541666610 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fded8004800 0x7fded8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:31.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.956+0000 7fdef2a00640 1 -- 172.21.15.28:0/541666610 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fded8003520 con 0x7fded8003b70 2024-02-21T23:10:31.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.956+0000 7fdef3400640 1 --2- 172.21.15.28:0/541666610 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdef40c1080 0x7fded80051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:10:31.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.957+0000 7fdef2a00640 1 --2- 172.21.15.28:0/541666610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fded8003b70 0x7fded8003f70 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7fdee40018d0 tx=0x7fded8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ee8b13f81f92966 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:31.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.958+0000 7fdef1600640 1 -- 172.21.15.28:0/541666610 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee4037070 con 0x7fded8003b70 2024-02-21T23:10:31.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.958+0000 7fdef1600640 1 -- 172.21.15.28:0/541666610 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdee40052d0 con 0x7fded8003b70 2024-02-21T23:10:31.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.958+0000 7fdef1600640 1 -- 172.21.15.28:0/541666610 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee40014f0 con 0x7fded8003b70 2024-02-21T23:10:31.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.958+0000 7fdef3e00640 1 -- 172.21.15.28:0/541666610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fded8003b70 msgr2=0x7fded8003f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:31.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.958+0000 7fdef3e00640 1 --2- 172.21.15.28:0/541666610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fded8003b70 0x7fded8003f70 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7fdee40018d0 tx=0x7fded8005830 comp rx=0 tx=0).stop 2024-02-21T23:10:31.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.958+0000 7fdef3400640 1 -- 172.21.15.28:0/541666610 reap_dead start 2024-02-21T23:10:31.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.958+0000 7fdef3e00640 1 -- 172.21.15.28:0/541666610 shutdown_connections 2024-02-21T23:10:31.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.958+0000 7fdef3e00640 1 -- 172.21.15.28:0/541666610 >> 172.21.15.28:0/541666610 conn(0x7fdef40a8530 msgr2=0x7fdef40a8910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:31.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.958+0000 7fdef3e00640 1 -- 172.21.15.28:0/541666610 shutdown_connections 2024-02-21T23:10:31.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.958+0000 7fdef3e00640 1 -- 172.21.15.28:0/541666610 wait complete. 2024-02-21T23:10:31.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.960+0000 7fdef3e00640 1 Processor -- start 2024-02-21T23:10:31.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.960+0000 7fdef3e00640 1 -- start start 2024-02-21T23:10:31.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.961+0000 7fdef3e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fded8003b70 0x7fdef41585a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:31.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.961+0000 7fdef3e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdef41f2d00 0x7fdef40aaee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:31.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.961+0000 7fdef3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdef4158ae0 0x7fdef40ab420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:31.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.961+0000 7fdef3e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef406b580 con 0x7fdef4158ae0 2024-02-21T23:10:31.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.961+0000 7fdef3e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef415d620 con 0x7fded8003b70 2024-02-21T23:10:31.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.961+0000 7fdef3e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdef406b700 con 0x7fdef41f2d00 2024-02-21T23:10:31.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.962+0000 7fdef2a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fded8003b70 0x7fdef41585a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:31.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.962+0000 7fdef2000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdef41f2d00 0x7fdef40aaee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:31.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.962+0000 7fdef3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdef4158ae0 0x7fdef40ab420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:31.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.962+0000 7fdef2a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fded8003b70 0x7fdef41585a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.28:35826/0 (socket says 172.21.15.28:35826) 2024-02-21T23:10:31.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.962+0000 7fdef2a00640 1 -- 172.21.15.28:0/3994600406 learned_addr learned my addr 172.21.15.28:0/3994600406 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:31.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.962+0000 7fdef3400640 1 -- 172.21.15.28:0/3994600406 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdef41f2d00 msgr2=0x7fdef40aaee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:31.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.962+0000 7fdef3400640 1 --2- 172.21.15.28:0/3994600406 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdef41f2d00 0x7fdef40aaee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:31.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.962+0000 7fdef3400640 1 -- 172.21.15.28:0/3994600406 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fded8003b70 msgr2=0x7fdef41585a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:31.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.962+0000 7fdef3400640 1 --2- 172.21.15.28:0/3994600406 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fded8003b70 0x7fdef41585a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:31.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.962+0000 7fdef3400640 1 -- 172.21.15.28:0/3994600406 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdef40a9840 con 0x7fdef4158ae0 2024-02-21T23:10:31.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.963+0000 7fdef3400640 1 --2- 172.21.15.28:0/3994600406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdef4158ae0 0x7fdef40ab420 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7fdef406ba80 tx=0x7fdeec00e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:31.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.963+0000 7fdeeaa00640 1 -- 172.21.15.28:0/3994600406 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeec00eda0 con 0x7fdef4158ae0 2024-02-21T23:10:31.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.963+0000 7fdeeaa00640 1 -- 172.21.15.28:0/3994600406 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdeec014430 con 0x7fdef4158ae0 2024-02-21T23:10:31.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.963+0000 7fdef3e00640 1 -- 172.21.15.28:0/3994600406 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdef4152940 con 0x7fdef4158ae0 2024-02-21T23:10:31.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.963+0000 7fdeeaa00640 1 -- 172.21.15.28:0/3994600406 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeec014820 con 0x7fdef4158ae0 2024-02-21T23:10:31.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.963+0000 7fdef3e00640 1 -- 172.21.15.28:0/3994600406 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdef4073130 con 0x7fdef4158ae0 2024-02-21T23:10:31.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.964+0000 7fdef3e00640 1 -- 172.21.15.28:0/3994600406 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdef404f290 con 0x7fdef4158ae0 2024-02-21T23:10:31.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.965+0000 7fdeeaa00640 1 -- 172.21.15.28:0/3994600406 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdeec0149c0 con 0x7fdef4158ae0 2024-02-21T23:10:31.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.966+0000 7fdeeaa00640 1 --2- 172.21.15.28:0/3994600406 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fded406b4d0 0x7fded406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:31.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.967+0000 7fdeeaa00640 1 -- 172.21.15.28:0/3994600406 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(160..160 src has 1..160) v4 ==== 7302+0+0 (secure 0 0 0) 0x7fdeec08a630 con 0x7fdef4158ae0 2024-02-21T23:10:31.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.967+0000 7fdef2a00640 1 --2- 172.21.15.28:0/3994600406 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fded406b4d0 0x7fded406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:31.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.968+0000 7fdef2a00640 1 --2- 172.21.15.28:0/3994600406 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fded406b4d0 0x7fded406d990 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fdee4000f60 tx=0x7fdee4031000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:31.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:31.971+0000 7fdeeaa00640 1 -- 172.21.15.28:0/3994600406 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fdef415a1a0 con 0x7fdef4158ae0 2024-02-21T23:10:32.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.212+0000 7fdef3e00640 1 -- 172.21.15.28:0/3994600406 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fdef4156cd0 con 0x7fdef4158ae0 2024-02-21T23:10:32.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.542+0000 7fdeeaa00640 1 -- 172.21.15.28:0/3994600406 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v173) v1 ==== 104+0+0 (secure 0 0 0) 0x7fdeec0588b0 con 0x7fdef4158ae0 2024-02-21T23:10:32.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.546+0000 7fdef3e00640 1 -- 172.21.15.28:0/3994600406 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fded406b4d0 msgr2=0x7fded406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:32.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.546+0000 7fdef3e00640 1 --2- 172.21.15.28:0/3994600406 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fded406b4d0 0x7fded406d990 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fdee4000f60 tx=0x7fdee4031000 comp rx=0 tx=0).stop 2024-02-21T23:10:32.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.546+0000 7fdef3e00640 1 -- 172.21.15.28:0/3994600406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdef4158ae0 msgr2=0x7fdef40ab420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:32.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.546+0000 7fdef3e00640 1 --2- 172.21.15.28:0/3994600406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdef4158ae0 0x7fdef40ab420 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7fdef406ba80 tx=0x7fdeec00e530 comp rx=0 tx=0).stop 2024-02-21T23:10:32.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.546+0000 7fdef3e00640 1 -- 172.21.15.28:0/3994600406 shutdown_connections 2024-02-21T23:10:32.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.546+0000 7fdef3e00640 1 --2- 172.21.15.28:0/3994600406 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fded406b4d0 0x7fded406d990 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:32.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.546+0000 7fdef3e00640 1 --2- 172.21.15.28:0/3994600406 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdef41f2d00 0x7fdef40aaee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:32.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.546+0000 7fdef3e00640 1 --2- 172.21.15.28:0/3994600406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdef4158ae0 0x7fdef40ab420 unknown :-1 s=CLOSED pgs=1821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:32.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.546+0000 7fdef3e00640 1 --2- 172.21.15.28:0/3994600406 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fded8003b70 0x7fdef41585a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:32.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.546+0000 7fdef3e00640 1 -- 172.21.15.28:0/3994600406 >> 172.21.15.28:0/3994600406 conn(0x7fdef40a8530 msgr2=0x7fdef40c5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:32.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.546+0000 7fdef3e00640 1 -- 172.21.15.28:0/3994600406 shutdown_connections 2024-02-21T23:10:32.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.546+0000 7fdef3e00640 1 -- 172.21.15.28:0/3994600406 wait complete. 2024-02-21T23:10:32.555 INFO:tasks.ceph.mds.b.smithi081.stderr: -244> 2024-02-21T23:10:25.216+0000 7f4689e00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-21T23:10:32.561 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-02-21T23:10:32.603 INFO:tasks.ceph.mds.b.smithi081.stdout:starting mds.b at 2024-02-21T23:10:32.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.774+0000 7fe303200640 1 Processor -- start 2024-02-21T23:10:32.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.774+0000 7fe303200640 1 -- start start 2024-02-21T23:10:32.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.776+0000 7fe303200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe2fc06e780 0x7fe2fc06eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:32.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.776+0000 7fe303200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe2fc0c5290 0x7fe2fc15d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:32.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.776+0000 7fe303200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2fc153cb0 con 0x7fe2fc0a5fc0 2024-02-21T23:10:32.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.776+0000 7fe303200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2fc1543d0 con 0x7fe2fc0c5290 2024-02-21T23:10:32.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.776+0000 7fe303200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2fc0c5ad0 con 0x7fe2fc06e780 2024-02-21T23:10:32.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.776+0000 7fe301200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe2fc0c5290 0x7fe2fc15d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:32.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.776+0000 7fe301200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe2fc0c5290 0x7fe2fc15d860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39514/0 (socket says 172.21.15.28:39514) 2024-02-21T23:10:32.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.776+0000 7fe301200640 1 -- 172.21.15.28:0/2814275335 learned_addr learned my addr 172.21.15.28:0/2814275335 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:32.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.776+0000 7fe2fbe00640 1 --2- 172.21.15.28:0/2814275335 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe2fc06e780 0x7fe2fc06eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:32.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.776+0000 7fe301200640 1 -- 172.21.15.28:0/2814275335 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe2fc06e780 msgr2=0x7fe2fc06eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:32.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.776+0000 7fe301200640 1 --2- 172.21.15.28:0/2814275335 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe2fc06e780 0x7fe2fc06eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:32.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.776+0000 7fe301200640 1 -- 172.21.15.28:0/2814275335 >> v1:172.21.15.81:6789/0 conn(0x7fe2fc0a5fc0 legacy=0x7fe2fc0c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:32.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.777+0000 7fe301200640 1 -- 172.21.15.28:0/2814275335 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2fc0c5d20 con 0x7fe2fc0c5290 2024-02-21T23:10:32.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.778+0000 7fe301200640 1 --2- 172.21.15.28:0/2814275335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2fc0c5290 0x7fe2fc15d860 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7fe2f4009e50 tx=0x7fe2f402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc422b2a073a6a8f server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:32.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.778+0000 7fe2fb400640 1 -- 172.21.15.28:0/2814275335 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2f402c9e0 con 0x7fe2fc0c5290 2024-02-21T23:10:32.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.778+0000 7fe2fb400640 1 -- 172.21.15.28:0/2814275335 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2f402cb80 con 0x7fe2fc0c5290 2024-02-21T23:10:32.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.778+0000 7fe2fb400640 1 -- 172.21.15.28:0/2814275335 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2f40343f0 con 0x7fe2fc0c5290 2024-02-21T23:10:32.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.779+0000 7fe303200640 1 -- 172.21.15.28:0/2814275335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2fc0c5290 msgr2=0x7fe2fc15d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:32.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.779+0000 7fe303200640 1 --2- 172.21.15.28:0/2814275335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2fc0c5290 0x7fe2fc15d860 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7fe2f4009e50 tx=0x7fe2f402be40 comp rx=0 tx=0).stop 2024-02-21T23:10:32.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.779+0000 7fe303200640 1 -- 172.21.15.28:0/2814275335 shutdown_connections 2024-02-21T23:10:32.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.779+0000 7fe303200640 1 --2- 172.21.15.28:0/2814275335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2fc0c5290 0x7fe2fc15d860 unknown :-1 s=CLOSED pgs=1823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:32.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.779+0000 7fe303200640 1 --2- 172.21.15.28:0/2814275335 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe2fc06e780 0x7fe2fc06eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:32.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.779+0000 7fe303200640 1 -- 172.21.15.28:0/2814275335 >> 172.21.15.28:0/2814275335 conn(0x7fe2fc06b830 msgr2=0x7fe2fc0c67f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:32.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.779+0000 7fe303200640 1 -- 172.21.15.28:0/2814275335 shutdown_connections 2024-02-21T23:10:32.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.779+0000 7fe303200640 1 -- 172.21.15.28:0/2814275335 wait complete. 2024-02-21T23:10:32.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.780+0000 7fe303200640 1 Processor -- start 2024-02-21T23:10:32.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.781+0000 7fe303200640 1 -- start start 2024-02-21T23:10:32.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.782+0000 7fe303200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2fc06e780 0x7fe2fc0c2700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:32.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.782+0000 7fe303200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2fc0a5fc0 0x7fe2fc1dd920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:32.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.782+0000 7fe303200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe2fc0c5290 0x7fe2fc1ec040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:32.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.782+0000 7fe303200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2fc153960 con 0x7fe2fc0a5fc0 2024-02-21T23:10:32.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.782+0000 7fe303200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2fc0c46f0 con 0x7fe2fc06e780 2024-02-21T23:10:32.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.782+0000 7fe303200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2fc06d110 con 0x7fe2fc0c5290 2024-02-21T23:10:32.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.782+0000 7fe2f3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2fc0a5fc0 0x7fe2fc1dd920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:32.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.782+0000 7fe2f3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2fc0a5fc0 0x7fe2fc1dd920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39520/0 (socket says 172.21.15.28:39520) 2024-02-21T23:10:32.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.782+0000 7fe2f3e00640 1 -- 172.21.15.28:0/2473794262 learned_addr learned my addr 172.21.15.28:0/2473794262 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:32.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.782+0000 7fe2f3e00640 1 -- 172.21.15.28:0/2473794262 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe2fc0c5290 msgr2=0x7fe2fc1ec040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:32.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.782+0000 7fe2f3e00640 1 --2- 172.21.15.28:0/2473794262 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe2fc0c5290 0x7fe2fc1ec040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:32.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.782+0000 7fe2f3e00640 1 -- 172.21.15.28:0/2473794262 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2fc06e780 msgr2=0x7fe2fc0c2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:32.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.782+0000 7fe2f3e00640 1 --2- 172.21.15.28:0/2473794262 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2fc06e780 0x7fe2fc0c2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:32.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.782+0000 7fe2f3e00640 1 -- 172.21.15.28:0/2473794262 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2fc14ddc0 con 0x7fe2fc0a5fc0 2024-02-21T23:10:32.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.783+0000 7fe2f3e00640 1 --2- 172.21.15.28:0/2473794262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2fc0a5fc0 0x7fe2fc1dd920 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7fe2e400b410 tx=0x7fe2e400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:32.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.783+0000 7fe2f8c00640 1 -- 172.21.15.28:0/2473794262 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e4015020 con 0x7fe2fc0a5fc0 2024-02-21T23:10:32.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.783+0000 7fe2f8c00640 1 -- 172.21.15.28:0/2473794262 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2e4011830 con 0x7fe2fc0a5fc0 2024-02-21T23:10:32.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.783+0000 7fe2f8c00640 1 -- 172.21.15.28:0/2473794262 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e4011c20 con 0x7fe2fc0a5fc0 2024-02-21T23:10:32.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.783+0000 7fe303200640 1 -- 172.21.15.28:0/2473794262 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2fc1dc540 con 0x7fe2fc0a5fc0 2024-02-21T23:10:32.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.783+0000 7fe303200640 1 -- 172.21.15.28:0/2473794262 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2fc0a8ed0 con 0x7fe2fc0a5fc0 2024-02-21T23:10:32.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.784+0000 7fe303200640 1 -- 172.21.15.28:0/2473794262 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2fc067f30 con 0x7fe2fc0a5fc0 2024-02-21T23:10:32.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.785+0000 7fe2f8c00640 1 -- 172.21.15.28:0/2473794262 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe2e4018020 con 0x7fe2fc0a5fc0 2024-02-21T23:10:32.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.786+0000 7fe2f8c00640 1 --2- 172.21.15.28:0/2473794262 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe2d806b500 0x7fe2d806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:32.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.786+0000 7fe2f8c00640 1 -- 172.21.15.28:0/2473794262 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(161..161 src has 1..161) v4 ==== 7388+0+0 (secure 0 0 0) 0x7fe2e408a5e0 con 0x7fe2fc0a5fc0 2024-02-21T23:10:32.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.787+0000 7fe2fbe00640 1 --2- 172.21.15.28:0/2473794262 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe2d806b500 0x7fe2d806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:32.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.788+0000 7fe2fbe00640 1 --2- 172.21.15.28:0/2473794262 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe2d806b500 0x7fe2d806d9c0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec006fd0 tx=0x7fe2ec008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:32.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.791+0000 7fe2f8c00640 1 -- 172.21.15.28:0/2473794262 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe2fc14ddc0 con 0x7fe2fc0a5fc0 2024-02-21T23:10:32.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:32.990+0000 7fe303200640 1 -- 172.21.15.28:0/2473794262 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe2fc152910 con 0x7fe2fc0a5fc0 2024-02-21T23:10:33.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.680+0000 7fe2f8c00640 1 -- 172.21.15.28:0/2473794262 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v162) v1 ==== 181+0+0 (secure 0 0 0) 0x7fe2e40202a0 con 0x7fe2fc0a5fc0 2024-02-21T23:10:33.668 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' removed 2024-02-21T23:10:33.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.684+0000 7fe303200640 1 -- 172.21.15.28:0/2473794262 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe2d806b500 msgr2=0x7fe2d806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:33.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.684+0000 7fe303200640 1 --2- 172.21.15.28:0/2473794262 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe2d806b500 0x7fe2d806d9c0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec006fd0 tx=0x7fe2ec008040 comp rx=0 tx=0).stop 2024-02-21T23:10:33.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.684+0000 7fe303200640 1 -- 172.21.15.28:0/2473794262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2fc0a5fc0 msgr2=0x7fe2fc1dd920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:33.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.684+0000 7fe303200640 1 --2- 172.21.15.28:0/2473794262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2fc0a5fc0 0x7fe2fc1dd920 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7fe2e400b410 tx=0x7fe2e400b8d0 comp rx=0 tx=0).stop 2024-02-21T23:10:33.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.684+0000 7fe303200640 1 -- 172.21.15.28:0/2473794262 shutdown_connections 2024-02-21T23:10:33.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.684+0000 7fe303200640 1 --2- 172.21.15.28:0/2473794262 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe2d806b500 0x7fe2d806d9c0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:33.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.684+0000 7fe303200640 1 --2- 172.21.15.28:0/2473794262 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe2fc0c5290 0x7fe2fc1ec040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:33.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.684+0000 7fe303200640 1 --2- 172.21.15.28:0/2473794262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2fc0a5fc0 0x7fe2fc1dd920 unknown :-1 s=CLOSED pgs=1824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:33.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.684+0000 7fe303200640 1 --2- 172.21.15.28:0/2473794262 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2fc06e780 0x7fe2fc0c2700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:33.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.684+0000 7fe303200640 1 -- 172.21.15.28:0/2473794262 >> 172.21.15.28:0/2473794262 conn(0x7fe2fc06b830 msgr2=0x7fe2fc156980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:33.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.685+0000 7fe303200640 1 -- 172.21.15.28:0/2473794262 shutdown_connections 2024-02-21T23:10:33.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.685+0000 7fe303200640 1 -- 172.21.15.28:0/2473794262 wait complete. 2024-02-21T23:10:33.701 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-02-21T23:10:33.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.933+0000 7feb80000640 1 Processor -- start 2024-02-21T23:10:33.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.933+0000 7feb80000640 1 -- start start 2024-02-21T23:10:33.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.934+0000 7feb80000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7feb781526f0 0x7feb780c1340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:33.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.934+0000 7feb80000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feb780c46f0 0x7feb78158820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:33.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.934+0000 7feb80000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7feb780c4dc0 0x7feb78158d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:33.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.934+0000 7feb80000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb78152ad0 con 0x7feb781526f0 2024-02-21T23:10:33.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.934+0000 7feb80000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb78155240 con 0x7feb780c4dc0 2024-02-21T23:10:33.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.934+0000 7feb80000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb78155520 con 0x7feb780c46f0 2024-02-21T23:10:33.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.935+0000 7feb7d600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7feb781526f0 0x7feb780c1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:33.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.935+0000 7feb7d600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7feb781526f0 0x7feb780c1340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39532/0 (socket says 172.21.15.28:39532) 2024-02-21T23:10:33.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.935+0000 7feb7d600640 1 -- 172.21.15.28:0/3894415192 learned_addr learned my addr 172.21.15.28:0/3894415192 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:33.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.935+0000 7feb7e000640 1 --2- 172.21.15.28:0/3894415192 >> v2:172.21.15.81:3300/0 conn(0x7feb780c4dc0 0x7feb78158d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:33.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.935+0000 7feb7cc00640 1 --2- 172.21.15.28:0/3894415192 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feb780c46f0 0x7feb78158820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:33.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.935+0000 7feb7d600640 1 -- 172.21.15.28:0/3894415192 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feb780c46f0 msgr2=0x7feb78158820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:33.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.935+0000 7feb7d600640 1 --2- 172.21.15.28:0/3894415192 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feb780c46f0 0x7feb78158820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:33.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.935+0000 7feb7d600640 1 -- 172.21.15.28:0/3894415192 >> v2:172.21.15.81:3300/0 conn(0x7feb780c4dc0 msgr2=0x7feb78158d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:33.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.935+0000 7feb7d600640 1 --2- 172.21.15.28:0/3894415192 >> v2:172.21.15.81:3300/0 conn(0x7feb780c4dc0 0x7feb78158d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:33.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.936+0000 7feb7d600640 1 -- 172.21.15.28:0/3894415192 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb78155800 con 0x7feb781526f0 2024-02-21T23:10:33.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.936+0000 7feb7cc00640 1 --2- 172.21.15.28:0/3894415192 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feb780c46f0 0x7feb78158820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:10:33.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.937+0000 7feb7d600640 1 --2- 172.21.15.28:0/3894415192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb781526f0 0x7feb780c1340 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7feb780c7880 tx=0x7feb6c02b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14fc03d306d0ba1 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:33.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.937+0000 7feb77e00640 1 -- 172.21.15.28:0/3894415192 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb6c035020 con 0x7feb781526f0 2024-02-21T23:10:33.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.937+0000 7feb77e00640 1 -- 172.21.15.28:0/3894415192 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb6c02d040 con 0x7feb781526f0 2024-02-21T23:10:33.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.937+0000 7feb77e00640 1 -- 172.21.15.28:0/3894415192 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb6c031a80 con 0x7feb781526f0 2024-02-21T23:10:33.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.938+0000 7feb80000640 1 -- 172.21.15.28:0/3894415192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb781526f0 msgr2=0x7feb780c1340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:33.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.938+0000 7feb80000640 1 --2- 172.21.15.28:0/3894415192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb781526f0 0x7feb780c1340 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7feb780c7880 tx=0x7feb6c02b900 comp rx=0 tx=0).stop 2024-02-21T23:10:33.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.938+0000 7feb80000640 1 -- 172.21.15.28:0/3894415192 shutdown_connections 2024-02-21T23:10:33.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.938+0000 7feb80000640 1 --2- 172.21.15.28:0/3894415192 >> v2:172.21.15.81:3300/0 conn(0x7feb780c4dc0 0x7feb78158d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:33.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.938+0000 7feb80000640 1 --2- 172.21.15.28:0/3894415192 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feb780c46f0 0x7feb78158820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:33.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.938+0000 7feb80000640 1 --2- 172.21.15.28:0/3894415192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb781526f0 0x7feb780c1340 unknown :-1 s=CLOSED pgs=1825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:33.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.938+0000 7feb80000640 1 -- 172.21.15.28:0/3894415192 >> 172.21.15.28:0/3894415192 conn(0x7feb7806b830 msgr2=0x7feb780c67f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:33.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.938+0000 7feb80000640 1 -- 172.21.15.28:0/3894415192 shutdown_connections 2024-02-21T23:10:33.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.938+0000 7feb80000640 1 -- 172.21.15.28:0/3894415192 wait complete. 2024-02-21T23:10:33.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.940+0000 7feb80000640 1 Processor -- start 2024-02-21T23:10:33.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.940+0000 7feb80000640 1 -- start start 2024-02-21T23:10:33.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.942+0000 7feb80000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb780c46f0 0x7feb7814f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:33.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.942+0000 7feb80000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feb780c4dc0 0x7feb78071340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:33.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.942+0000 7feb80000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feb78072f00 0x7feb78154260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:33.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.942+0000 7feb80000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb780c3c30 con 0x7feb780c46f0 2024-02-21T23:10:33.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.942+0000 7feb80000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb78155dc0 con 0x7feb780c4dc0 2024-02-21T23:10:33.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.942+0000 7feb80000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb781592a0 con 0x7feb78072f00 2024-02-21T23:10:33.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.942+0000 7feb7d600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb780c46f0 0x7feb7814f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:33.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.942+0000 7feb7d600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb780c46f0 0x7feb7814f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39544/0 (socket says 172.21.15.28:39544) 2024-02-21T23:10:33.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.942+0000 7feb7d600640 1 -- 172.21.15.28:0/3533848033 learned_addr learned my addr 172.21.15.28:0/3533848033 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:33.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.942+0000 7feb7cc00640 1 --2- 172.21.15.28:0/3533848033 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feb780c4dc0 0x7feb78071340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:33.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.942+0000 7feb7d600640 1 -- 172.21.15.28:0/3533848033 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feb78072f00 msgr2=0x7feb78154260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:33.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.942+0000 7feb7d600640 1 --2- 172.21.15.28:0/3533848033 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feb78072f00 0x7feb78154260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:33.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.942+0000 7feb7d600640 1 -- 172.21.15.28:0/3533848033 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feb780c4dc0 msgr2=0x7feb78071340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:33.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.942+0000 7feb7d600640 1 --2- 172.21.15.28:0/3533848033 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feb780c4dc0 0x7feb78071340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:33.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.942+0000 7feb7d600640 1 -- 172.21.15.28:0/3533848033 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb78071880 con 0x7feb780c46f0 2024-02-21T23:10:33.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.943+0000 7feb7d600640 1 --2- 172.21.15.28:0/3533848033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb780c46f0 0x7feb7814f040 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7feb6c02be90 tx=0x7feb6c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:33.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.943+0000 7feb75600640 1 -- 172.21.15.28:0/3533848033 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb6c040070 con 0x7feb780c46f0 2024-02-21T23:10:33.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.943+0000 7feb75600640 1 -- 172.21.15.28:0/3533848033 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb6c039bf0 con 0x7feb780c46f0 2024-02-21T23:10:33.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.943+0000 7feb75600640 1 -- 172.21.15.28:0/3533848033 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb6c035040 con 0x7feb780c46f0 2024-02-21T23:10:33.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.943+0000 7feb80000640 1 -- 172.21.15.28:0/3533848033 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb7814caa0 con 0x7feb780c46f0 2024-02-21T23:10:33.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.944+0000 7feb80000640 1 -- 172.21.15.28:0/3533848033 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb781eb400 con 0x7feb780c46f0 2024-02-21T23:10:33.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.945+0000 7feb80000640 1 -- 172.21.15.28:0/3533848033 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb78067f30 con 0x7feb780c46f0 2024-02-21T23:10:33.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.946+0000 7feb75600640 1 -- 172.21.15.28:0/3533848033 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feb6c04b060 con 0x7feb780c46f0 2024-02-21T23:10:33.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.947+0000 7feb75600640 1 --2- 172.21.15.28:0/3533848033 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7feb5806b500 0x7feb5806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:33.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.948+0000 7feb75600640 1 -- 172.21.15.28:0/3533848033 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6950+0+0 (secure 0 0 0) 0x7feb6c0ac120 con 0x7feb780c46f0 2024-02-21T23:10:33.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.948+0000 7feb7cc00640 1 --2- 172.21.15.28:0/3533848033 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7feb5806b500 0x7feb5806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:33.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.949+0000 7feb7cc00640 1 --2- 172.21.15.28:0/3533848033 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7feb5806b500 0x7feb5806d9c0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7feb64006fd0 tx=0x7feb64008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:33.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:33.952+0000 7feb75600640 1 -- 172.21.15.28:0/3533848033 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7feb6c02f310 con 0x7feb780c46f0 2024-02-21T23:10:34.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.152+0000 7feb80000640 1 -- 172.21.15.28:0/3533848033 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7feb780a6190 con 0x7feb780c46f0 2024-02-21T23:10:34.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.692+0000 7feb75600640 1 -- 172.21.15.28:0/3533848033 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v163) v1 ==== 169+0+0 (secure 0 0 0) 0x7feb6c07ab50 con 0x7feb780c46f0 2024-02-21T23:10:34.681 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' removed 2024-02-21T23:10:34.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.699+0000 7feb80000640 1 -- 172.21.15.28:0/3533848033 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7feb5806b500 msgr2=0x7feb5806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:34.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.699+0000 7feb80000640 1 --2- 172.21.15.28:0/3533848033 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7feb5806b500 0x7feb5806d9c0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7feb64006fd0 tx=0x7feb64008040 comp rx=0 tx=0).stop 2024-02-21T23:10:34.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.699+0000 7feb80000640 1 -- 172.21.15.28:0/3533848033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb780c46f0 msgr2=0x7feb7814f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:34.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.699+0000 7feb80000640 1 --2- 172.21.15.28:0/3533848033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb780c46f0 0x7feb7814f040 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7feb6c02be90 tx=0x7feb6c0046c0 comp rx=0 tx=0).stop 2024-02-21T23:10:34.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.703+0000 7feb80000640 1 -- 172.21.15.28:0/3533848033 shutdown_connections 2024-02-21T23:10:34.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.703+0000 7feb80000640 1 --2- 172.21.15.28:0/3533848033 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7feb5806b500 0x7feb5806d9c0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:34.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.703+0000 7feb80000640 1 --2- 172.21.15.28:0/3533848033 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feb78072f00 0x7feb78154260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:34.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.703+0000 7feb80000640 1 --2- 172.21.15.28:0/3533848033 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feb780c4dc0 0x7feb78071340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:34.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.703+0000 7feb80000640 1 --2- 172.21.15.28:0/3533848033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb780c46f0 0x7feb7814f040 unknown :-1 s=CLOSED pgs=1826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:34.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.703+0000 7feb80000640 1 -- 172.21.15.28:0/3533848033 >> 172.21.15.28:0/3533848033 conn(0x7feb7806b830 msgr2=0x7feb78150900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:34.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.706+0000 7feb80000640 1 -- 172.21.15.28:0/3533848033 shutdown_connections 2024-02-21T23:10:34.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.706+0000 7feb80000640 1 -- 172.21.15.28:0/3533848033 wait complete. 2024-02-21T23:10:34.717 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:10:34.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.925+0000 7f613da00640 1 Processor -- start 2024-02-21T23:10:34.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.926+0000 7f613da00640 1 -- start start 2024-02-21T23:10:34.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.927+0000 7f613da00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6138150450 0x7f61380a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:34.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.927+0000 7f613da00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61380c2760 con 0x7f613806ee40 2024-02-21T23:10:34.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.927+0000 7f613da00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6138156310 con 0x7f6138156af0 2024-02-21T23:10:34.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.927+0000 7f613da00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61380c22c0 con 0x7f6138150450 2024-02-21T23:10:34.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.927+0000 7f612f400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f613806ee40 0x7f61381536f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:47776/0 (socket says 172.21.15.28:47776) 2024-02-21T23:10:34.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.927+0000 7f612f400640 1 -- 172.21.15.28:0/2028262184 learned_addr learned my addr 172.21.15.28:0/2028262184 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:10:34.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.927+0000 7f6137e00640 1 --2- 172.21.15.28:0/2028262184 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6138150450 0x7f61380a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:34.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.928+0000 7f6137400640 1 -- 172.21.15.28:0/2028262184 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1157134847 0 0) 0x7f61380c2760 con 0x7f613806ee40 2024-02-21T23:10:34.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.928+0000 7f6137400640 1 -- 172.21.15.28:0/2028262184 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f611c003520 con 0x7f613806ee40 2024-02-21T23:10:34.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.928+0000 7f6137400640 1 -- 172.21.15.28:0/2028262184 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3714129531 0 0) 0x7f611c003520 con 0x7f613806ee40 2024-02-21T23:10:34.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.928+0000 7f6137400640 1 -- 172.21.15.28:0/2028262184 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6138150450 msgr2=0x7f61380a5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:34.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.928+0000 7f6137400640 1 --2- 172.21.15.28:0/2028262184 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6138150450 0x7f61380a5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:34.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.928+0000 7f6137400640 1 -- 172.21.15.28:0/2028262184 >> v1:172.21.15.81:6789/0 conn(0x7f6138156af0 legacy=0x7f613806dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:34.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.928+0000 7f6137400640 1 -- 172.21.15.28:0/2028262184 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61380c29d0 con 0x7f613806ee40 2024-02-21T23:10:34.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.928+0000 7f6137400640 1 -- 172.21.15.28:0/2028262184 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f6120001670 con 0x7f613806ee40 2024-02-21T23:10:34.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.928+0000 7f6137400640 1 -- 172.21.15.28:0/2028262184 >> v1:172.21.15.28:6789/0 conn(0x7f613806ee40 legacy=0x7f61381536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:34.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.928+0000 7f6137400640 1 --2- 172.21.15.28:0/2028262184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f611c0039e0 0x7f611c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:34.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.928+0000 7f6137400640 1 --2- 172.21.15.28:0/2028262184 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6138150450 0x7f611c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:34.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.928+0000 7f6137400640 1 --2- 172.21.15.28:0/2028262184 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f611c004d40 0x7f611c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:34.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.928+0000 7f6137400640 1 -- 172.21.15.28:0/2028262184 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f611c00ba70 con 0x7f611c0039e0 2024-02-21T23:10:34.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.928+0000 7f6137400640 1 -- 172.21.15.28:0/2028262184 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f611c00bbf0 con 0x7f611c004d40 2024-02-21T23:10:34.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.928+0000 7f6137400640 1 -- 172.21.15.28:0/2028262184 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f611c00bdf0 con 0x7f6138150450 2024-02-21T23:10:34.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.929+0000 7f6137e00640 1 --2- 172.21.15.28:0/2028262184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f611c0039e0 0x7f611c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:34.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.929+0000 7f613d000640 1 --2- 172.21.15.28:0/2028262184 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f611c004d40 0x7f611c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:34.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.929+0000 7f612f400640 1 --2- 172.21.15.28:0/2028262184 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6138150450 0x7f611c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:34.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.929+0000 7f6137e00640 1 -- 172.21.15.28:0/2028262184 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6138150450 msgr2=0x7f611c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:34.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.929+0000 7f6137e00640 1 --2- 172.21.15.28:0/2028262184 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6138150450 0x7f611c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:34.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.929+0000 7f6137e00640 1 -- 172.21.15.28:0/2028262184 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f611c004d40 msgr2=0x7f611c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:34.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.929+0000 7f6137e00640 1 --2- 172.21.15.28:0/2028262184 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f611c004d40 0x7f611c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:34.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.929+0000 7f6137e00640 1 -- 172.21.15.28:0/2028262184 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f611c003520 con 0x7f611c0039e0 2024-02-21T23:10:34.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.929+0000 7f612f400640 1 --2- 172.21.15.28:0/2028262184 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6138150450 0x7f611c004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:10:34.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.929+0000 7f613d000640 1 --2- 172.21.15.28:0/2028262184 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f611c004d40 0x7f611c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:10:34.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.930+0000 7f6137e00640 1 --2- 172.21.15.28:0/2028262184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f611c0039e0 0x7f611c003e00 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7f6128009a90 tx=0x7f611c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c6676a36455bd9c server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:34.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.931+0000 7f6137400640 1 -- 172.21.15.28:0/2028262184 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6128039070 con 0x7f611c0039e0 2024-02-21T23:10:34.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.931+0000 7f6137400640 1 -- 172.21.15.28:0/2028262184 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f612802c680 con 0x7f611c0039e0 2024-02-21T23:10:34.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.931+0000 7f6137400640 1 -- 172.21.15.28:0/2028262184 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6128009690 con 0x7f611c0039e0 2024-02-21T23:10:34.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.931+0000 7f613da00640 1 -- 172.21.15.28:0/2028262184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f611c0039e0 msgr2=0x7f611c003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:34.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.931+0000 7f613da00640 1 --2- 172.21.15.28:0/2028262184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f611c0039e0 0x7f611c003e00 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7f6128009a90 tx=0x7f611c005830 comp rx=0 tx=0).stop 2024-02-21T23:10:34.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.931+0000 7f613d000640 1 -- 172.21.15.28:0/2028262184 reap_dead start 2024-02-21T23:10:34.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.931+0000 7f613da00640 1 -- 172.21.15.28:0/2028262184 shutdown_connections 2024-02-21T23:10:34.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.931+0000 7f613da00640 1 -- 172.21.15.28:0/2028262184 >> 172.21.15.28:0/2028262184 conn(0x7f613806b630 msgr2=0x7f61380c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:34.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.932+0000 7f613da00640 1 -- 172.21.15.28:0/2028262184 shutdown_connections 2024-02-21T23:10:34.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.932+0000 7f613da00640 1 -- 172.21.15.28:0/2028262184 wait complete. 2024-02-21T23:10:34.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.933+0000 7f613da00640 1 Processor -- start 2024-02-21T23:10:34.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.934+0000 7f613da00640 1 -- start start 2024-02-21T23:10:34.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.935+0000 7f613da00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f611c0039e0 0x7f613814d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:34.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.935+0000 7f613da00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61381edbb0 0x7f6138072080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:34.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.935+0000 7f613da00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6138072da0 0x7f61380725c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:34.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.935+0000 7f613da00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f613806e700 con 0x7f61381edbb0 2024-02-21T23:10:34.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.935+0000 7f613da00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6138156660 con 0x7f6138072da0 2024-02-21T23:10:34.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.935+0000 7f613da00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f613806e880 con 0x7f611c0039e0 2024-02-21T23:10:34.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.935+0000 7f612f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61381edbb0 0x7f6138072080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:34.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.935+0000 7f612f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61381edbb0 0x7f6138072080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53334/0 (socket says 172.21.15.28:53334) 2024-02-21T23:10:34.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.935+0000 7f612f400640 1 -- 172.21.15.28:0/3117541477 learned_addr learned my addr 172.21.15.28:0/3117541477 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:34.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.935+0000 7f6137e00640 1 --2- 172.21.15.28:0/3117541477 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f611c0039e0 0x7f613814d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:34.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.935+0000 7f613d000640 1 --2- 172.21.15.28:0/3117541477 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6138072da0 0x7f61380725c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:34.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.935+0000 7f612f400640 1 -- 172.21.15.28:0/3117541477 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f611c0039e0 msgr2=0x7f613814d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:34.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.936+0000 7f612f400640 1 --2- 172.21.15.28:0/3117541477 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f611c0039e0 0x7f613814d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:34.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.936+0000 7f612f400640 1 -- 172.21.15.28:0/3117541477 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6138072da0 msgr2=0x7f61380725c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:34.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.936+0000 7f612f400640 1 --2- 172.21.15.28:0/3117541477 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6138072da0 0x7f61380725c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:34.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.936+0000 7f612f400640 1 -- 172.21.15.28:0/3117541477 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6138153900 con 0x7f61381edbb0 2024-02-21T23:10:34.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.936+0000 7f6137e00640 1 --2- 172.21.15.28:0/3117541477 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f611c0039e0 0x7f613814d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:10:34.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.936+0000 7f613d000640 1 --2- 172.21.15.28:0/3117541477 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6138072da0 0x7f61380725c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:10:34.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.936+0000 7f612f400640 1 --2- 172.21.15.28:0/3117541477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61381edbb0 0x7f6138072080 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7f612000d8e0 tx=0x7f612000dda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:34.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.936+0000 7f6134c00640 1 -- 172.21.15.28:0/3117541477 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6120019070 con 0x7f61381edbb0 2024-02-21T23:10:34.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.936+0000 7f6134c00640 1 -- 172.21.15.28:0/3117541477 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61200074e0 con 0x7f61381edbb0 2024-02-21T23:10:34.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.936+0000 7f6134c00640 1 -- 172.21.15.28:0/3117541477 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61200049e0 con 0x7f61381edbb0 2024-02-21T23:10:34.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.936+0000 7f613da00640 1 -- 172.21.15.28:0/3117541477 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f613814d7a0 con 0x7f61381edbb0 2024-02-21T23:10:34.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.936+0000 7f613da00640 1 -- 172.21.15.28:0/3117541477 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61380715a0 con 0x7f61381edbb0 2024-02-21T23:10:34.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.937+0000 7f613da00640 1 -- 172.21.15.28:0/3117541477 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f613804f290 con 0x7f61381edbb0 2024-02-21T23:10:34.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.938+0000 7f6134c00640 1 -- 172.21.15.28:0/3117541477 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f612001c070 con 0x7f61381edbb0 2024-02-21T23:10:34.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.939+0000 7f6134c00640 1 --2- 172.21.15.28:0/3117541477 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f611406b500 0x7f611406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:34.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.939+0000 7f6134c00640 1 -- 172.21.15.28:0/3117541477 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 6536+0+0 (secure 0 0 0) 0x7f612008a0b0 con 0x7f61381edbb0 2024-02-21T23:10:34.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.940+0000 7f6137e00640 1 --2- 172.21.15.28:0/3117541477 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f611406b500 0x7f611406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:34.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.941+0000 7f6137e00640 1 --2- 172.21.15.28:0/3117541477 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f611406b500 0x7f611406d9c0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f6128036000 tx=0x7f6128030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:34.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:34.944+0000 7f6134c00640 1 -- 172.21.15.28:0/3117541477 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f6120058be0 con 0x7f61381edbb0 2024-02-21T23:10:35.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.147+0000 7f613da00640 1 -- 172.21.15.28:0/3117541477 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f61380c1f10 con 0x7f61381edbb0 2024-02-21T23:10:35.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.150+0000 7f6134c00640 1 -- 172.21.15.28:0/3117541477 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v163) v1 ==== 74+0+17551 (secure 0 0 0) 0x7f61200584e0 con 0x7f61381edbb0 2024-02-21T23:10:35.138 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:10:35.139 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":163,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:10:34.681492+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs2_metadata","create_time":"2024-02-21T23:10:04.095451+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"154","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs2_data","create_time":"2024-02-21T23:10:04.836982+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"26.29","mappings":[{"from":1,"to":0}]},{"pgid":"26.36","mappings":[{"from":3,"to":0}]},{"pgid":"26.3f","mappings":[{"from":1,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.81:6835/1320495906":"2024-02-22T23:10:32.538190+0000","172.21.15.81:6834/1320495906":"2024-02-22T23:10:32.538190+0000","172.21.15.81:6837/521625945":"2024-02-22T23:10:31.712025+0000","172.21.15.28:6833/2975378719":"2024-02-22T23:10:21.581948+0000","172.21.15.28:6832/2975378719":"2024-02-22T23:10:21.581948+0000","172.21.15.81:6836/521625945":"2024-02-22T23:10:31.712025+0000","172.21.15.28:6835/2928639298":"2024-02-22T23:10:19.550812+0000","172.21.15.28:6834/2928639298":"2024-02-22T23:10:19.550812+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:10:35.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.153+0000 7f613da00640 1 -- 172.21.15.28:0/3117541477 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f611406b500 msgr2=0x7f611406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:35.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.153+0000 7f613da00640 1 --2- 172.21.15.28:0/3117541477 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f611406b500 0x7f611406d9c0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f6128036000 tx=0x7f6128030000 comp rx=0 tx=0).stop 2024-02-21T23:10:35.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.153+0000 7f613da00640 1 -- 172.21.15.28:0/3117541477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61381edbb0 msgr2=0x7f6138072080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:35.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.153+0000 7f613da00640 1 --2- 172.21.15.28:0/3117541477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61381edbb0 0x7f6138072080 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7f612000d8e0 tx=0x7f612000dda0 comp rx=0 tx=0).stop 2024-02-21T23:10:35.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.153+0000 7f613da00640 1 -- 172.21.15.28:0/3117541477 shutdown_connections 2024-02-21T23:10:35.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.153+0000 7f613da00640 1 --2- 172.21.15.28:0/3117541477 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f611406b500 0x7f611406d9c0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:35.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.153+0000 7f613da00640 1 --2- 172.21.15.28:0/3117541477 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6138072da0 0x7f61380725c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:35.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.153+0000 7f613da00640 1 --2- 172.21.15.28:0/3117541477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61381edbb0 0x7f6138072080 unknown :-1 s=CLOSED pgs=1829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:35.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.153+0000 7f613da00640 1 --2- 172.21.15.28:0/3117541477 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f611c0039e0 0x7f613814d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:35.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.153+0000 7f613da00640 1 -- 172.21.15.28:0/3117541477 >> 172.21.15.28:0/3117541477 conn(0x7f613806b630 msgr2=0x7f613814e260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:35.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.153+0000 7f613da00640 1 -- 172.21.15.28:0/3117541477 shutdown_connections 2024-02-21T23:10:35.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.153+0000 7f613da00640 1 -- 172.21.15.28:0/3117541477 wait complete. 2024-02-21T23:10:35.159 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-21T23:02:30.728329+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 25, 'pool_name': 'cephfs2_metadata', 'create_time': '2024-02-21T23:10:04.095451+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '154', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs2'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 26, 'pool_name': 'cephfs2_data', 'create_time': '2024-02-21T23:10:04.836982+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '155', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs2'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-21T23:10:35.160 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-21T23:10:35.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.390+0000 7f5f4f000640 1 Processor -- start 2024-02-21T23:10:35.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.390+0000 7f5f4f000640 1 -- start start 2024-02-21T23:10:35.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.391+0000 7f5f4f000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5f4806b0e0 0x7f5f480c5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:35.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.391+0000 7f5f4f000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f480c61e0 con 0x7f5f48150700 2024-02-21T23:10:35.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.391+0000 7f5f4f000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f48152fb0 con 0x7f5f480c5d80 2024-02-21T23:10:35.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.391+0000 7f5f4f000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f48153bc0 con 0x7f5f4806b0e0 2024-02-21T23:10:35.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.392+0000 7f5f4d000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f5f48150700 0x7f5f48156500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:47792/0 (socket says 172.21.15.28:47792) 2024-02-21T23:10:35.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.392+0000 7f5f4d000640 1 -- 172.21.15.28:0/3326176738 learned_addr learned my addr 172.21.15.28:0/3326176738 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:10:35.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.392+0000 7f5f47e00640 1 --2- 172.21.15.28:0/3326176738 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5f4806b0e0 0x7f5f480c5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:35.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.392+0000 7f5f46a00640 1 -- 172.21.15.28:0/3326176738 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2855291116 0 0) 0x7f5f480c61e0 con 0x7f5f48150700 2024-02-21T23:10:35.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.392+0000 7f5f46a00640 1 -- 172.21.15.28:0/3326176738 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f2c003520 con 0x7f5f48150700 2024-02-21T23:10:35.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.393+0000 7f5f46a00640 1 -- 172.21.15.28:0/3326176738 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2821884434 0 0) 0x7f5f2c003520 con 0x7f5f48150700 2024-02-21T23:10:35.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.393+0000 7f5f46a00640 1 -- 172.21.15.28:0/3326176738 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5f4806b0e0 msgr2=0x7f5f480c5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:35.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.393+0000 7f5f46a00640 1 --2- 172.21.15.28:0/3326176738 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5f4806b0e0 0x7f5f480c5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:35.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.393+0000 7f5f46a00640 1 -- 172.21.15.28:0/3326176738 >> v1:172.21.15.81:6789/0 conn(0x7f5f480c5d80 legacy=0x7f5f4814fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:35.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.393+0000 7f5f46a00640 1 -- 172.21.15.28:0/3326176738 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f48153290 con 0x7f5f48150700 2024-02-21T23:10:35.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.393+0000 7f5f46a00640 1 -- 172.21.15.28:0/3326176738 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f5f400019f0 con 0x7f5f48150700 2024-02-21T23:10:35.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.393+0000 7f5f46a00640 1 -- 172.21.15.28:0/3326176738 >> v1:172.21.15.28:6789/0 conn(0x7f5f48150700 legacy=0x7f5f48156500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:35.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.393+0000 7f5f46a00640 1 --2- 172.21.15.28:0/3326176738 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5f4806b0e0 0x7f5f2c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:35.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.393+0000 7f5f46a00640 1 --2- 172.21.15.28:0/3326176738 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f2c003f60 0x7f5f2c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:35.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.393+0000 7f5f46a00640 1 --2- 172.21.15.28:0/3326176738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f2c004d40 0x7f5f2c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:35.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.393+0000 7f5f46a00640 1 -- 172.21.15.28:0/3326176738 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f48153bc0 con 0x7f5f2c004d40 2024-02-21T23:10:35.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.393+0000 7f5f46a00640 1 -- 172.21.15.28:0/3326176738 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f2c00bb50 con 0x7f5f2c003f60 2024-02-21T23:10:35.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.393+0000 7f5f46a00640 1 -- 172.21.15.28:0/3326176738 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f2c00bd50 con 0x7f5f4806b0e0 2024-02-21T23:10:35.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.393+0000 7f5f4d000640 1 --2- 172.21.15.28:0/3326176738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f2c004d40 0x7f5f2c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:35.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.394+0000 7f5f47e00640 1 --2- 172.21.15.28:0/3326176738 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5f4806b0e0 0x7f5f2c0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:35.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.394+0000 7f5f4d000640 1 -- 172.21.15.28:0/3326176738 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5f4806b0e0 msgr2=0x7f5f2c0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:35.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.394+0000 7f5f4d000640 1 --2- 172.21.15.28:0/3326176738 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5f4806b0e0 0x7f5f2c0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:35.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.394+0000 7f5f4d000640 1 -- 172.21.15.28:0/3326176738 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f2c003f60 msgr2=0x7f5f2c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:35.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.394+0000 7f5f4d000640 1 --2- 172.21.15.28:0/3326176738 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f2c003f60 0x7f5f2c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:35.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.394+0000 7f5f4d000640 1 -- 172.21.15.28:0/3326176738 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f2c003520 con 0x7f5f2c004d40 2024-02-21T23:10:35.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.395+0000 7f5f4d000640 1 --2- 172.21.15.28:0/3326176738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f2c004d40 0x7f5f2c0051c0 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7f5f4000a860 tx=0x7f5f2c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fd169699b7b81ff server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:35.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.395+0000 7f5f46a00640 1 -- 172.21.15.28:0/3326176738 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f4000a480 con 0x7f5f2c004d40 2024-02-21T23:10:35.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.395+0000 7f5f46a00640 1 -- 172.21.15.28:0/3326176738 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f40034630 con 0x7f5f2c004d40 2024-02-21T23:10:35.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.395+0000 7f5f46a00640 1 -- 172.21.15.28:0/3326176738 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f40034a60 con 0x7f5f2c004d40 2024-02-21T23:10:35.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.396+0000 7f5f4f000640 1 -- 172.21.15.28:0/3326176738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f2c004d40 msgr2=0x7f5f2c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:35.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.396+0000 7f5f4f000640 1 --2- 172.21.15.28:0/3326176738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f2c004d40 0x7f5f2c0051c0 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7f5f4000a860 tx=0x7f5f2c005830 comp rx=0 tx=0).stop 2024-02-21T23:10:35.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.396+0000 7f5f4d000640 1 -- 172.21.15.28:0/3326176738 reap_dead start 2024-02-21T23:10:35.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.396+0000 7f5f4f000640 1 -- 172.21.15.28:0/3326176738 shutdown_connections 2024-02-21T23:10:35.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.396+0000 7f5f4f000640 1 -- 172.21.15.28:0/3326176738 >> 172.21.15.28:0/3326176738 conn(0x7f5f480a85a0 msgr2=0x7f5f480a8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:35.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.396+0000 7f5f4f000640 1 -- 172.21.15.28:0/3326176738 shutdown_connections 2024-02-21T23:10:35.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.396+0000 7f5f4f000640 1 -- 172.21.15.28:0/3326176738 wait complete. 2024-02-21T23:10:35.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.398+0000 7f5f4f000640 1 Processor -- start 2024-02-21T23:10:35.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.398+0000 7f5f4f000640 1 -- start start 2024-02-21T23:10:35.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.399+0000 7f5f4f000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f2c004d40 0x7f5f48143400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:35.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.399+0000 7f5f4f000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5f48140a30 0x7f5f481ee970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:35.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.399+0000 7f5f4f000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f480ab620 0x7f5f481ea9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:35.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.399+0000 7f5f4f000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f4806b4c0 con 0x7f5f480ab620 2024-02-21T23:10:35.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.399+0000 7f5f4f000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f48153850 con 0x7f5f2c004d40 2024-02-21T23:10:35.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.399+0000 7f5f4f000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f4806b640 con 0x7f5f48140a30 2024-02-21T23:10:35.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.399+0000 7f5f4d000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f480ab620 0x7f5f481ea9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:35.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.399+0000 7f5f4d000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f480ab620 0x7f5f481ea9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53360/0 (socket says 172.21.15.28:53360) 2024-02-21T23:10:35.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.399+0000 7f5f4d000640 1 -- 172.21.15.28:0/3183550183 learned_addr learned my addr 172.21.15.28:0/3183550183 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:35.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.400+0000 7f5f47400640 1 --2- 172.21.15.28:0/3183550183 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5f48140a30 0x7f5f481ee970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:35.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.400+0000 7f5f47e00640 1 --2- 172.21.15.28:0/3183550183 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f2c004d40 0x7f5f48143400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:35.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.400+0000 7f5f4d000640 1 -- 172.21.15.28:0/3183550183 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5f48140a30 msgr2=0x7f5f481ee970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:35.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.400+0000 7f5f4d000640 1 --2- 172.21.15.28:0/3183550183 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5f48140a30 0x7f5f481ee970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:35.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.400+0000 7f5f4d000640 1 -- 172.21.15.28:0/3183550183 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f2c004d40 msgr2=0x7f5f48143400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:35.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.400+0000 7f5f4d000640 1 --2- 172.21.15.28:0/3183550183 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f2c004d40 0x7f5f48143400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:35.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.400+0000 7f5f4d000640 1 -- 172.21.15.28:0/3183550183 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f480a9ae0 con 0x7f5f480ab620 2024-02-21T23:10:35.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.400+0000 7f5f4d000640 1 --2- 172.21.15.28:0/3183550183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f480ab620 0x7f5f481ea9b0 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7f5f4000a980 tx=0x7f5f48153b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:35.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.401+0000 7f5f3fe00640 1 -- 172.21.15.28:0/3183550183 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f4000a480 con 0x7f5f480ab620 2024-02-21T23:10:35.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.401+0000 7f5f3fe00640 1 -- 172.21.15.28:0/3183550183 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f40044070 con 0x7f5f480ab620 2024-02-21T23:10:35.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.401+0000 7f5f3fe00640 1 -- 172.21.15.28:0/3183550183 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f40001250 con 0x7f5f480ab620 2024-02-21T23:10:35.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.401+0000 7f5f4f000640 1 -- 172.21.15.28:0/3183550183 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f48140fa0 con 0x7f5f480ab620 2024-02-21T23:10:35.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.401+0000 7f5f4f000640 1 -- 172.21.15.28:0/3183550183 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f481ee170 con 0x7f5f480ab620 2024-02-21T23:10:35.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.402+0000 7f5f4f000640 1 -- 172.21.15.28:0/3183550183 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f48067f30 con 0x7f5f480ab620 2024-02-21T23:10:35.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.403+0000 7f5f3fe00640 1 -- 172.21.15.28:0/3183550183 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5f400349f0 con 0x7f5f480ab620 2024-02-21T23:10:35.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.404+0000 7f5f3fe00640 1 --2- 172.21.15.28:0/3183550183 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5f2806b670 0x7f5f2806db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:35.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.404+0000 7f5f3fe00640 1 -- 172.21.15.28:0/3183550183 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 6536+0+0 (secure 0 0 0) 0x7f5f400ac8c0 con 0x7f5f480ab620 2024-02-21T23:10:35.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.404+0000 7f5f47e00640 1 --2- 172.21.15.28:0/3183550183 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5f2806b670 0x7f5f2806db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:35.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.406+0000 7f5f47e00640 1 --2- 172.21.15.28:0/3183550183 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5f2806b670 0x7f5f2806db30 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f5f38009a10 tx=0x7f5f38008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:35.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.409+0000 7f5f3fe00640 1 -- 172.21.15.28:0/3183550183 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f5f4003e350 con 0x7f5f480ab620 2024-02-21T23:10:35.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.618+0000 7f5f4f000640 1 -- 172.21.15.28:0/3183550183 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5f480c4fe0 con 0x7f5f480ab620 2024-02-21T23:10:35.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.619+0000 7f5f3fe00640 1 -- 172.21.15.28:0/3183550183 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v163) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5f4007b3f0 con 0x7f5f480ab620 2024-02-21T23:10:35.607 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-02-21T23:10:35.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.623+0000 7f5f4f000640 1 -- 172.21.15.28:0/3183550183 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5f2806b670 msgr2=0x7f5f2806db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:35.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.623+0000 7f5f4f000640 1 --2- 172.21.15.28:0/3183550183 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5f2806b670 0x7f5f2806db30 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f5f38009a10 tx=0x7f5f38008040 comp rx=0 tx=0).stop 2024-02-21T23:10:35.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.623+0000 7f5f4f000640 1 -- 172.21.15.28:0/3183550183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f480ab620 msgr2=0x7f5f481ea9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:35.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.623+0000 7f5f4f000640 1 --2- 172.21.15.28:0/3183550183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f480ab620 0x7f5f481ea9b0 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7f5f4000a980 tx=0x7f5f48153b30 comp rx=0 tx=0).stop 2024-02-21T23:10:35.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.623+0000 7f5f4f000640 1 -- 172.21.15.28:0/3183550183 shutdown_connections 2024-02-21T23:10:35.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.623+0000 7f5f4f000640 1 --2- 172.21.15.28:0/3183550183 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5f2806b670 0x7f5f2806db30 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:35.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.623+0000 7f5f4f000640 1 --2- 172.21.15.28:0/3183550183 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5f48140a30 0x7f5f481ee970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:35.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.623+0000 7f5f4f000640 1 --2- 172.21.15.28:0/3183550183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f480ab620 0x7f5f481ea9b0 unknown :-1 s=CLOSED pgs=1832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:35.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.623+0000 7f5f4f000640 1 --2- 172.21.15.28:0/3183550183 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5f2c004d40 0x7f5f48143400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:35.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.623+0000 7f5f4f000640 1 -- 172.21.15.28:0/3183550183 >> 172.21.15.28:0/3183550183 conn(0x7f5f480a85a0 msgr2=0x7f5f4814da80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:35.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.623+0000 7f5f4f000640 1 -- 172.21.15.28:0/3183550183 shutdown_connections 2024-02-21T23:10:35.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.623+0000 7f5f4f000640 1 -- 172.21.15.28:0/3183550183 wait complete. 2024-02-21T23:10:35.631 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs2_metadata pg_num 2024-02-21T23:10:35.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.861+0000 7facade00640 1 Processor -- start 2024-02-21T23:10:35.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.861+0000 7facade00640 1 -- start start 2024-02-21T23:10:35.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.862+0000 7facade00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faca8152650 0x7faca80c8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:35.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.862+0000 7facade00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7faca806daa0 0x7faca80c7160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:35.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.862+0000 7facade00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faca8158920 con 0x7faca80c76a0 2024-02-21T23:10:35.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.862+0000 7facade00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faca81586d0 con 0x7faca806daa0 2024-02-21T23:10:35.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.862+0000 7facade00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faca8158c00 con 0x7faca8152650 2024-02-21T23:10:35.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.863+0000 7faca7e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7faca806daa0 0x7faca80c7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:35.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.863+0000 7faca7e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7faca806daa0 0x7faca80c7160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53368/0 (socket says 172.21.15.28:53368) 2024-02-21T23:10:35.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.863+0000 7faca7e00640 1 -- 172.21.15.28:0/3713534830 learned_addr learned my addr 172.21.15.28:0/3713534830 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:35.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.863+0000 7facaca00640 1 --2- 172.21.15.28:0/3713534830 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faca8152650 0x7faca80c8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:35.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.863+0000 7faca7e00640 1 -- 172.21.15.28:0/3713534830 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faca8152650 msgr2=0x7faca80c8ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:35.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.863+0000 7faca7e00640 1 --2- 172.21.15.28:0/3713534830 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faca8152650 0x7faca80c8ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:35.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.863+0000 7faca7e00640 1 -- 172.21.15.28:0/3713534830 >> v1:172.21.15.81:6789/0 conn(0x7faca80c76a0 legacy=0x7faca81550e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:35.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.863+0000 7faca7e00640 1 -- 172.21.15.28:0/3713534830 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faca8158ee0 con 0x7faca806daa0 2024-02-21T23:10:35.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.864+0000 7faca7e00640 1 --2- 172.21.15.28:0/3713534830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faca806daa0 0x7faca80c7160 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7fac90009870 tx=0x7fac9002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3fa125fddb3ea16 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:35.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.865+0000 7faca7400640 1 -- 172.21.15.28:0/3713534830 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac900317c0 con 0x7faca806daa0 2024-02-21T23:10:35.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.865+0000 7faca7400640 1 -- 172.21.15.28:0/3713534830 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac90037070 con 0x7faca806daa0 2024-02-21T23:10:35.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.865+0000 7faca7400640 1 -- 172.21.15.28:0/3713534830 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac90031b80 con 0x7faca806daa0 2024-02-21T23:10:35.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.865+0000 7facade00640 1 -- 172.21.15.28:0/3713534830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faca806daa0 msgr2=0x7faca80c7160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:35.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.865+0000 7facade00640 1 --2- 172.21.15.28:0/3713534830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faca806daa0 0x7faca80c7160 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7fac90009870 tx=0x7fac9002b860 comp rx=0 tx=0).stop 2024-02-21T23:10:35.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.866+0000 7facade00640 1 -- 172.21.15.28:0/3713534830 shutdown_connections 2024-02-21T23:10:35.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.866+0000 7facade00640 1 --2- 172.21.15.28:0/3713534830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faca806daa0 0x7faca80c7160 unknown :-1 s=CLOSED pgs=1833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:35.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.866+0000 7facade00640 1 --2- 172.21.15.28:0/3713534830 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faca8152650 0x7faca80c8ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:35.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.866+0000 7facade00640 1 -- 172.21.15.28:0/3713534830 >> 172.21.15.28:0/3713534830 conn(0x7faca8072ce0 msgr2=0x7faca80ac340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:35.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.866+0000 7facade00640 1 -- 172.21.15.28:0/3713534830 shutdown_connections 2024-02-21T23:10:35.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.866+0000 7facade00640 1 -- 172.21.15.28:0/3713534830 wait complete. 2024-02-21T23:10:35.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.867+0000 7facade00640 1 Processor -- start 2024-02-21T23:10:35.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.868+0000 7facade00640 1 -- start start 2024-02-21T23:10:35.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.868+0000 7facade00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faca806daa0 0x7faca8153450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:35.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.868+0000 7facade00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faca80c76a0 0x7faca81e1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:35.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.868+0000 7facade00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faca8152650 0x7faca8157710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:35.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.868+0000 7facade00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faca8152a30 con 0x7faca806daa0 2024-02-21T23:10:35.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.868+0000 7facade00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faca80bdfb0 con 0x7faca8152650 2024-02-21T23:10:35.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.868+0000 7facade00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faca81557f0 con 0x7faca80c76a0 2024-02-21T23:10:35.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.869+0000 7facaca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faca806daa0 0x7faca8153450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:35.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.869+0000 7facaca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faca806daa0 0x7faca8153450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53374/0 (socket says 172.21.15.28:53374) 2024-02-21T23:10:35.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.869+0000 7facaca00640 1 -- 172.21.15.28:0/795746006 learned_addr learned my addr 172.21.15.28:0/795746006 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:35.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.869+0000 7faca7e00640 1 --2- 172.21.15.28:0/795746006 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faca80c76a0 0x7faca81e1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:35.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.869+0000 7facad400640 1 --2- 172.21.15.28:0/795746006 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faca8152650 0x7faca8157710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:35.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.869+0000 7facaca00640 1 -- 172.21.15.28:0/795746006 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faca80c76a0 msgr2=0x7faca81e1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:35.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.869+0000 7facaca00640 1 --2- 172.21.15.28:0/795746006 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faca80c76a0 0x7faca81e1dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:35.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.869+0000 7facaca00640 1 -- 172.21.15.28:0/795746006 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faca8152650 msgr2=0x7faca8157710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:35.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.869+0000 7facaca00640 1 --2- 172.21.15.28:0/795746006 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faca8152650 0x7faca8157710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:35.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.869+0000 7facaca00640 1 -- 172.21.15.28:0/795746006 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faca814ba60 con 0x7faca806daa0 2024-02-21T23:10:35.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.869+0000 7faca7e00640 1 --2- 172.21.15.28:0/795746006 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faca80c76a0 0x7faca81e1dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:10:35.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.870+0000 7facaca00640 1 --2- 172.21.15.28:0/795746006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faca806daa0 0x7faca8153450 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7fac9800d5f0 tx=0x7fac9800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:35.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.870+0000 7faca4c00640 1 -- 172.21.15.28:0/795746006 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac980125f0 con 0x7faca806daa0 2024-02-21T23:10:35.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.870+0000 7faca4c00640 1 -- 172.21.15.28:0/795746006 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac98019070 con 0x7faca806daa0 2024-02-21T23:10:35.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.870+0000 7faca4c00640 1 -- 172.21.15.28:0/795746006 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac980129d0 con 0x7faca806daa0 2024-02-21T23:10:35.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.870+0000 7facade00640 1 -- 172.21.15.28:0/795746006 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faca814bd40 con 0x7faca806daa0 2024-02-21T23:10:35.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.870+0000 7facade00640 1 -- 172.21.15.28:0/795746006 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faca80acf10 con 0x7faca806daa0 2024-02-21T23:10:35.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.871+0000 7facade00640 1 -- 172.21.15.28:0/795746006 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faca804f290 con 0x7faca806daa0 2024-02-21T23:10:35.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.873+0000 7faca4c00640 1 -- 172.21.15.28:0/795746006 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fac98012b70 con 0x7faca806daa0 2024-02-21T23:10:35.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.874+0000 7faca4c00640 1 --2- 172.21.15.28:0/795746006 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fac8406b4d0 0x7fac8406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:35.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.874+0000 7faca4c00640 1 -- 172.21.15.28:0/795746006 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 6536+0+0 (secure 0 0 0) 0x7fac9808a940 con 0x7faca806daa0 2024-02-21T23:10:35.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.875+0000 7faca7e00640 1 --2- 172.21.15.28:0/795746006 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fac8406b4d0 0x7fac8406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:35.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.876+0000 7faca7e00640 1 --2- 172.21.15.28:0/795746006 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fac8406b4d0 0x7fac8406d990 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fac90036000 tx=0x7fac90008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:35.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:35.879+0000 7faca4c00640 1 -- 172.21.15.28:0/795746006 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fac98059470 con 0x7faca806daa0 2024-02-21T23:10:36.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.078+0000 7facade00640 1 -- 172.21.15.28:0/795746006 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs2_metadata", "var": "pg_num"} v 0) v1 -- 0x7faca80c1160 con 0x7faca806daa0 2024-02-21T23:10:36.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.078+0000 7faca4c00640 1 -- 172.21.15.28:0/795746006 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs2_metadata", "var": "pg_num"}]=0 v163) v1 ==== 105+0+11 (secure 0 0 0) 0x7fac98058d70 con 0x7faca806daa0 2024-02-21T23:10:36.066 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-02-21T23:10:36.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.081+0000 7facade00640 1 -- 172.21.15.28:0/795746006 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fac8406b4d0 msgr2=0x7fac8406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:36.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.081+0000 7facade00640 1 --2- 172.21.15.28:0/795746006 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fac8406b4d0 0x7fac8406d990 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fac90036000 tx=0x7fac90008030 comp rx=0 tx=0).stop 2024-02-21T23:10:36.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.081+0000 7facade00640 1 -- 172.21.15.28:0/795746006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faca806daa0 msgr2=0x7faca8153450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:36.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.081+0000 7facade00640 1 --2- 172.21.15.28:0/795746006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faca806daa0 0x7faca8153450 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7fac9800d5f0 tx=0x7fac9800dab0 comp rx=0 tx=0).stop 2024-02-21T23:10:36.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.081+0000 7facade00640 1 -- 172.21.15.28:0/795746006 shutdown_connections 2024-02-21T23:10:36.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.081+0000 7facade00640 1 --2- 172.21.15.28:0/795746006 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fac8406b4d0 0x7fac8406d990 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:36.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.081+0000 7facade00640 1 --2- 172.21.15.28:0/795746006 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faca80c76a0 0x7faca81e1dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:36.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.081+0000 7facade00640 1 --2- 172.21.15.28:0/795746006 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faca8152650 0x7faca8157710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:36.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.081+0000 7facade00640 1 --2- 172.21.15.28:0/795746006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faca806daa0 0x7faca8153450 unknown :-1 s=CLOSED pgs=1834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:36.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.081+0000 7facade00640 1 -- 172.21.15.28:0/795746006 >> 172.21.15.28:0/795746006 conn(0x7faca8072ce0 msgr2=0x7faca81de740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:36.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.082+0000 7facade00640 1 -- 172.21.15.28:0/795746006 shutdown_connections 2024-02-21T23:10:36.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.082+0000 7facade00640 1 -- 172.21.15.28:0/795746006 wait complete. 2024-02-21T23:10:36.090 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs2_data pg_num 2024-02-21T23:10:36.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.330+0000 7f4a56200640 1 Processor -- start 2024-02-21T23:10:36.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.331+0000 7f4a56200640 1 -- start start 2024-02-21T23:10:36.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.332+0000 7f4a56200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4a500c5f50 0x7f4a5014fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:36.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.332+0000 7f4a56200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f4a50150530 0x7f4a50156500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:36.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.332+0000 7f4a56200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a50152fb0 con 0x7f4a5006b0e0 2024-02-21T23:10:36.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.332+0000 7f4a56200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a50153bc0 con 0x7f4a500c5f50 2024-02-21T23:10:36.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.332+0000 7f4a56200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a50153290 con 0x7f4a50150530 2024-02-21T23:10:36.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.333+0000 7f4a4ea00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4a500c5f50 0x7f4a5014fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:36.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.333+0000 7f4a4f400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f4a5006b0e0 0x7f4a500c5840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:47806/0 (socket says 172.21.15.28:47806) 2024-02-21T23:10:36.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.333+0000 7f4a4f400640 1 -- 172.21.15.28:0/3612865446 learned_addr learned my addr 172.21.15.28:0/3612865446 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:10:36.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.333+0000 7f4a4fe00640 1 --2- 172.21.15.28:0/3612865446 >> v2:172.21.15.81:3300/0 conn(0x7f4a50150530 0x7f4a50156500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:36.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.333+0000 7f4a4ea00640 1 -- 172.21.15.28:0/3612865446 >> v2:172.21.15.81:3300/0 conn(0x7f4a50150530 msgr2=0x7f4a50156500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:36.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.333+0000 7f4a4ea00640 1 --2- 172.21.15.28:0/3612865446 >> v2:172.21.15.81:3300/0 conn(0x7f4a50150530 0x7f4a50156500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:36.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.333+0000 7f4a4ea00640 1 -- 172.21.15.28:0/3612865446 >> v1:172.21.15.28:6789/0 conn(0x7f4a5006b0e0 legacy=0x7f4a500c5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:36.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.333+0000 7f4a4ea00640 1 -- 172.21.15.28:0/3612865446 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a50153570 con 0x7f4a500c5f50 2024-02-21T23:10:36.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.333+0000 7f4a4e000640 1 -- 172.21.15.28:0/3612865446 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1055285078 0 0) 0x7f4a50152fb0 con 0x7f4a5006b0e0 2024-02-21T23:10:36.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.335+0000 7f4a4ea00640 1 --2- 172.21.15.28:0/3612865446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a500c5f50 0x7f4a5014fff0 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7f4a500a9570 tx=0x7f4a3c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a11dd82cd858444 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:36.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.335+0000 7f4a4e000640 1 -- 172.21.15.28:0/3612865446 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a3c0315a0 con 0x7f4a500c5f50 2024-02-21T23:10:36.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.335+0000 7f4a4e000640 1 -- 172.21.15.28:0/3612865446 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a3c037070 con 0x7f4a500c5f50 2024-02-21T23:10:36.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.335+0000 7f4a4e000640 1 -- 172.21.15.28:0/3612865446 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a3c031960 con 0x7f4a500c5f50 2024-02-21T23:10:36.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.336+0000 7f4a56200640 1 -- 172.21.15.28:0/3612865446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a500c5f50 msgr2=0x7f4a5014fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:36.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.336+0000 7f4a56200640 1 --2- 172.21.15.28:0/3612865446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a500c5f50 0x7f4a5014fff0 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7f4a500a9570 tx=0x7f4a3c02b840 comp rx=0 tx=0).stop 2024-02-21T23:10:36.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.336+0000 7f4a56200640 1 -- 172.21.15.28:0/3612865446 shutdown_connections 2024-02-21T23:10:36.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.336+0000 7f4a56200640 1 --2- 172.21.15.28:0/3612865446 >> v2:172.21.15.81:3300/0 conn(0x7f4a50150530 0x7f4a50156500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:36.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.336+0000 7f4a56200640 1 --2- 172.21.15.28:0/3612865446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a500c5f50 0x7f4a5014fff0 unknown :-1 s=CLOSED pgs=1836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:36.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.336+0000 7f4a56200640 1 -- 172.21.15.28:0/3612865446 >> 172.21.15.28:0/3612865446 conn(0x7f4a500a85a0 msgr2=0x7f4a500a8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:36.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.336+0000 7f4a56200640 1 -- 172.21.15.28:0/3612865446 shutdown_connections 2024-02-21T23:10:36.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.336+0000 7f4a56200640 1 -- 172.21.15.28:0/3612865446 wait complete. 2024-02-21T23:10:36.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.338+0000 7f4a56200640 1 Processor -- start 2024-02-21T23:10:36.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.338+0000 7f4a56200640 1 -- start start 2024-02-21T23:10:36.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.339+0000 7f4a56200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4a5006b0e0 0x7f4a5006e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:36.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.339+0000 7f4a56200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a500c5f50 0x7f4a50149ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:36.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.339+0000 7f4a56200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a50150530 0x7f4a5014a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:36.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.339+0000 7f4a56200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a5006b4c0 con 0x7f4a50150530 2024-02-21T23:10:36.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.339+0000 7f4a56200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a501509c0 con 0x7f4a500c5f50 2024-02-21T23:10:36.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.339+0000 7f4a56200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a5006b640 con 0x7f4a5006b0e0 2024-02-21T23:10:36.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.339+0000 7f4a4fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a50150530 0x7f4a5014a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:36.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.339+0000 7f4a4fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a50150530 0x7f4a5014a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53384/0 (socket says 172.21.15.28:53384) 2024-02-21T23:10:36.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.339+0000 7f4a4fe00640 1 -- 172.21.15.28:0/2754034531 learned_addr learned my addr 172.21.15.28:0/2754034531 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:36.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.340+0000 7f4a4ea00640 1 --2- 172.21.15.28:0/2754034531 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a500c5f50 0x7f4a50149ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:36.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.340+0000 7f4a4f400640 1 --2- 172.21.15.28:0/2754034531 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4a5006b0e0 0x7f4a5006e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:36.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.340+0000 7f4a4fe00640 1 -- 172.21.15.28:0/2754034531 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4a5006b0e0 msgr2=0x7f4a5006e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:36.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.340+0000 7f4a4fe00640 1 --2- 172.21.15.28:0/2754034531 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4a5006b0e0 0x7f4a5006e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:36.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.340+0000 7f4a4fe00640 1 -- 172.21.15.28:0/2754034531 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a500c5f50 msgr2=0x7f4a50149ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:36.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.340+0000 7f4a4fe00640 1 --2- 172.21.15.28:0/2754034531 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a500c5f50 0x7f4a50149ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:36.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.340+0000 7f4a4fe00640 1 -- 172.21.15.28:0/2754034531 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a500a8390 con 0x7f4a50150530 2024-02-21T23:10:36.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.340+0000 7f4a4fe00640 1 --2- 172.21.15.28:0/2754034531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a50150530 0x7f4a5014a010 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7f4a48007e60 tx=0x7f4a4800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:36.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.341+0000 7f4a47400640 1 -- 172.21.15.28:0/2754034531 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a48014420 con 0x7f4a50150530 2024-02-21T23:10:36.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.341+0000 7f4a47400640 1 -- 172.21.15.28:0/2754034531 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a48019070 con 0x7f4a50150530 2024-02-21T23:10:36.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.341+0000 7f4a47400640 1 -- 172.21.15.28:0/2754034531 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a48014820 con 0x7f4a50150530 2024-02-21T23:10:36.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.341+0000 7f4a56200640 1 -- 172.21.15.28:0/2754034531 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a50154c60 con 0x7f4a50150530 2024-02-21T23:10:36.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.341+0000 7f4a56200640 1 -- 172.21.15.28:0/2754034531 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a50155220 con 0x7f4a50150530 2024-02-21T23:10:36.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.342+0000 7f4a56200640 1 -- 172.21.15.28:0/2754034531 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a50067f30 con 0x7f4a50150530 2024-02-21T23:10:36.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.343+0000 7f4a47400640 1 -- 172.21.15.28:0/2754034531 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4a480149c0 con 0x7f4a50150530 2024-02-21T23:10:36.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.344+0000 7f4a47400640 1 --2- 172.21.15.28:0/2754034531 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4a3006b500 0x7f4a3006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:36.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.344+0000 7f4a47400640 1 -- 172.21.15.28:0/2754034531 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 6536+0+0 (secure 0 0 0) 0x7f4a4808ad00 con 0x7f4a50150530 2024-02-21T23:10:36.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.345+0000 7f4a4f400640 1 --2- 172.21.15.28:0/2754034531 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4a3006b500 0x7f4a3006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:36.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.346+0000 7f4a4f400640 1 --2- 172.21.15.28:0/2754034531 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4a3006b500 0x7f4a3006d9c0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f4a40007fd0 tx=0x7f4a40009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:36.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.353+0000 7f4a47400640 1 -- 172.21.15.28:0/2754034531 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4a500a8390 con 0x7f4a50150530 2024-02-21T23:10:36.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.554+0000 7f4a56200640 1 -- 172.21.15.28:0/2754034531 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs2_data", "var": "pg_num"} v 0) v1 -- 0x7f4a5014adb0 con 0x7f4a50150530 2024-02-21T23:10:36.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.554+0000 7f4a47400640 1 -- 172.21.15.28:0/2754034531 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs2_data", "var": "pg_num"}]=0 v163) v1 ==== 101+0+11 (secure 0 0 0) 0x7f4a48059830 con 0x7f4a50150530 2024-02-21T23:10:36.542 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-02-21T23:10:36.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.557+0000 7f4a56200640 1 -- 172.21.15.28:0/2754034531 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4a3006b500 msgr2=0x7f4a3006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:36.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.557+0000 7f4a56200640 1 --2- 172.21.15.28:0/2754034531 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4a3006b500 0x7f4a3006d9c0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f4a40007fd0 tx=0x7f4a40009040 comp rx=0 tx=0).stop 2024-02-21T23:10:36.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.557+0000 7f4a56200640 1 -- 172.21.15.28:0/2754034531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a50150530 msgr2=0x7f4a5014a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:36.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.557+0000 7f4a56200640 1 --2- 172.21.15.28:0/2754034531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a50150530 0x7f4a5014a010 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7f4a48007e60 tx=0x7f4a4800e4b0 comp rx=0 tx=0).stop 2024-02-21T23:10:36.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.557+0000 7f4a56200640 1 -- 172.21.15.28:0/2754034531 shutdown_connections 2024-02-21T23:10:36.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.557+0000 7f4a56200640 1 --2- 172.21.15.28:0/2754034531 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4a3006b500 0x7f4a3006d9c0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:36.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.557+0000 7f4a56200640 1 --2- 172.21.15.28:0/2754034531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a50150530 0x7f4a5014a010 unknown :-1 s=CLOSED pgs=1837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:36.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.557+0000 7f4a56200640 1 --2- 172.21.15.28:0/2754034531 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a500c5f50 0x7f4a50149ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:36.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.557+0000 7f4a56200640 1 --2- 172.21.15.28:0/2754034531 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4a5006b0e0 0x7f4a5006e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:36.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.557+0000 7f4a56200640 1 -- 172.21.15.28:0/2754034531 >> 172.21.15.28:0/2754034531 conn(0x7f4a500a85a0 msgr2=0x7f4a500c5080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:36.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.558+0000 7f4a56200640 1 -- 172.21.15.28:0/2754034531 shutdown_connections 2024-02-21T23:10:36.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.558+0000 7f4a56200640 1 -- 172.21.15.28:0/2754034531 wait complete. 2024-02-21T23:10:36.565 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:10:36.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.854+0000 7f8b7aa00640 1 Processor -- start 2024-02-21T23:10:36.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.854+0000 7f8b7aa00640 1 -- start start 2024-02-21T23:10:36.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.856+0000 7f8b7aa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8b740aff90 0x7f8b740c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:36.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.856+0000 7f8b7aa00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f8b740c5720 0x7f8b7415f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:36.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.856+0000 7f8b7aa00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b740c74c0 con 0x7f8b7406dbf0 2024-02-21T23:10:36.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.856+0000 7f8b7aa00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b740c7270 con 0x7f8b740aff90 2024-02-21T23:10:36.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.856+0000 7f8b7aa00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b74158630 con 0x7f8b740c5720 2024-02-21T23:10:36.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.856+0000 7f8b78c00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f8b7406dbf0 0x7f8b74158ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:47812/0 (socket says 172.21.15.28:47812) 2024-02-21T23:10:36.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.856+0000 7f8b78c00640 1 -- 172.21.15.28:0/2701731340 learned_addr learned my addr 172.21.15.28:0/2701731340 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:10:36.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.856+0000 7f8b79600640 1 --2- 172.21.15.28:0/2701731340 >> v2:172.21.15.28:3300/0 conn(0x7f8b740aff90 0x7f8b740c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:36.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.857+0000 7f8b7a000640 1 --2- 172.21.15.28:0/2701731340 >> v2:172.21.15.81:3300/0 conn(0x7f8b740c5720 0x7f8b7415f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:36.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.857+0000 7f8b73e00640 1 -- 172.21.15.28:0/2701731340 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2522500201 0 0) 0x7f8b740c74c0 con 0x7f8b7406dbf0 2024-02-21T23:10:36.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.857+0000 7f8b79600640 1 -- 172.21.15.28:0/2701731340 >> v2:172.21.15.81:3300/0 conn(0x7f8b740c5720 msgr2=0x7f8b7415f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:36.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.857+0000 7f8b79600640 1 --2- 172.21.15.28:0/2701731340 >> v2:172.21.15.81:3300/0 conn(0x7f8b740c5720 0x7f8b7415f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:36.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.857+0000 7f8b79600640 1 -- 172.21.15.28:0/2701731340 >> v1:172.21.15.28:6789/0 conn(0x7f8b7406dbf0 legacy=0x7f8b74158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:36.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.857+0000 7f8b79600640 1 -- 172.21.15.28:0/2701731340 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b740c77a0 con 0x7f8b740aff90 2024-02-21T23:10:36.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.858+0000 7f8b79600640 1 --2- 172.21.15.28:0/2701731340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b740aff90 0x7f8b740c51e0 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7f8b740c73f0 tx=0x7f8b6802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e60d5733537dca4 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:36.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.858+0000 7f8b73e00640 1 -- 172.21.15.28:0/2701731340 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b68035020 con 0x7f8b740aff90 2024-02-21T23:10:36.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.858+0000 7f8b73e00640 1 -- 172.21.15.28:0/2701731340 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b68031830 con 0x7f8b740aff90 2024-02-21T23:10:36.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.858+0000 7f8b73e00640 1 -- 172.21.15.28:0/2701731340 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b68031c00 con 0x7f8b740aff90 2024-02-21T23:10:36.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.859+0000 7f8b7aa00640 1 -- 172.21.15.28:0/2701731340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b740aff90 msgr2=0x7f8b740c51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:36.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.859+0000 7f8b7aa00640 1 --2- 172.21.15.28:0/2701731340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b740aff90 0x7f8b740c51e0 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7f8b740c73f0 tx=0x7f8b6802b8d0 comp rx=0 tx=0).stop 2024-02-21T23:10:36.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.859+0000 7f8b7aa00640 1 -- 172.21.15.28:0/2701731340 shutdown_connections 2024-02-21T23:10:36.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.859+0000 7f8b7aa00640 1 --2- 172.21.15.28:0/2701731340 >> v2:172.21.15.81:3300/0 conn(0x7f8b740c5720 0x7f8b7415f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:36.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.859+0000 7f8b7aa00640 1 --2- 172.21.15.28:0/2701731340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b740aff90 0x7f8b740c51e0 unknown :-1 s=CLOSED pgs=1839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:36.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.859+0000 7f8b7aa00640 1 -- 172.21.15.28:0/2701731340 >> 172.21.15.28:0/2701731340 conn(0x7f8b740ac370 msgr2=0x7f8b740ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:36.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.859+0000 7f8b7aa00640 1 -- 172.21.15.28:0/2701731340 shutdown_connections 2024-02-21T23:10:36.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.859+0000 7f8b7aa00640 1 -- 172.21.15.28:0/2701731340 wait complete. 2024-02-21T23:10:36.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.861+0000 7f8b7aa00640 1 Processor -- start 2024-02-21T23:10:36.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.861+0000 7f8b7aa00640 1 -- start start 2024-02-21T23:10:36.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.862+0000 7f8b7aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b7406dbf0 0x7f8b740c8d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:36.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.862+0000 7f8b7aa00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8b740aff90 0x7f8b740c35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:36.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.862+0000 7f8b7aa00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b740c5720 0x7f8b74151c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:36.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.862+0000 7f8b7aa00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b740c8760 con 0x7f8b7406dbf0 2024-02-21T23:10:36.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.862+0000 7f8b7aa00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b740be0c0 con 0x7f8b740c5720 2024-02-21T23:10:36.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.862+0000 7f8b7aa00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b740c88e0 con 0x7f8b740aff90 2024-02-21T23:10:36.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.862+0000 7f8b79600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b7406dbf0 0x7f8b740c8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:36.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.863+0000 7f8b79600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b7406dbf0 0x7f8b740c8d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53406/0 (socket says 172.21.15.28:53406) 2024-02-21T23:10:36.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.863+0000 7f8b79600640 1 -- 172.21.15.28:0/2290601316 learned_addr learned my addr 172.21.15.28:0/2290601316 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:36.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.863+0000 7f8b79600640 1 -- 172.21.15.28:0/2290601316 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8b740aff90 msgr2=0x7f8b740c35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:36.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.863+0000 7f8b79600640 1 --2- 172.21.15.28:0/2290601316 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8b740aff90 0x7f8b740c35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:36.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.863+0000 7f8b79600640 1 -- 172.21.15.28:0/2290601316 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b740c5720 msgr2=0x7f8b74151c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:36.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.863+0000 7f8b79600640 1 --2- 172.21.15.28:0/2290601316 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b740c5720 0x7f8b74151c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:36.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.863+0000 7f8b79600640 1 -- 172.21.15.28:0/2290601316 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b740bd910 con 0x7f8b7406dbf0 2024-02-21T23:10:36.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.863+0000 7f8b79600640 1 --2- 172.21.15.28:0/2290601316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b7406dbf0 0x7f8b740c8d30 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7f8b68009860 tx=0x7f8b680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:36.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.864+0000 7f8b71600640 1 -- 172.21.15.28:0/2290601316 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b68041070 con 0x7f8b7406dbf0 2024-02-21T23:10:36.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.864+0000 7f8b71600640 1 -- 172.21.15.28:0/2290601316 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b6803a950 con 0x7f8b7406dbf0 2024-02-21T23:10:36.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.864+0000 7f8b71600640 1 -- 172.21.15.28:0/2290601316 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b68035040 con 0x7f8b7406dbf0 2024-02-21T23:10:36.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.864+0000 7f8b7aa00640 1 -- 172.21.15.28:0/2290601316 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b740bd630 con 0x7f8b7406dbf0 2024-02-21T23:10:36.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.864+0000 7f8b7aa00640 1 -- 172.21.15.28:0/2290601316 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b740c1300 con 0x7f8b7406dbf0 2024-02-21T23:10:36.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.866+0000 7f8b7aa00640 1 -- 172.21.15.28:0/2290601316 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b7404f290 con 0x7f8b7406dbf0 2024-02-21T23:10:36.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.866+0000 7f8b71600640 1 -- 172.21.15.28:0/2290601316 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8b68038020 con 0x7f8b7406dbf0 2024-02-21T23:10:36.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.868+0000 7f8b71600640 1 --2- 172.21.15.28:0/2290601316 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8b5406b4d0 0x7f8b5406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:36.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.868+0000 7f8b71600640 1 -- 172.21.15.28:0/2290601316 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 6536+0+0 (secure 0 0 0) 0x7f8b680abe40 con 0x7f8b7406dbf0 2024-02-21T23:10:36.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.869+0000 7f8b78c00640 1 --2- 172.21.15.28:0/2290601316 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8b5406b4d0 0x7f8b5406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:36.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.870+0000 7f8b78c00640 1 --2- 172.21.15.28:0/2290601316 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8b5406b4d0 0x7f8b5406d990 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f8b60007fd0 tx=0x7f8b60007920 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:36.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:36.872+0000 7f8b71600640 1 -- 172.21.15.28:0/2290601316 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f8b6807aa10 con 0x7f8b7406dbf0 2024-02-21T23:10:37.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.113+0000 7f8b7aa00640 1 -- 172.21.15.28:0/2290601316 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b7406d9d0 con 0x7f8b7406dbf0 2024-02-21T23:10:37.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.114+0000 7f8b71600640 1 -- 172.21.15.28:0/2290601316 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 176 v176) v1 ==== 95+0+4948 (secure 0 0 0) 0x7f8b6807a310 con 0x7f8b7406dbf0 2024-02-21T23:10:37.102 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:10:37.102 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":176,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6599,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/704098282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":704098282},{"type":"v1","addr":"172.21.15.81:6837","nonce":704098282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":173},{"gid":7372,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1804589714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1804589714},{"type":"v1","addr":"172.21.15.81:6835","nonce":1804589714}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":174},{"gid":9141,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2895215997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2895215997},{"type":"v1","addr":"172.21.15.28:6835","nonce":2895215997}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":166}],"filesystems":[{"mdsmap":{"epoch":176,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:10:05.843850+0000","modified":"2024-02-21T23:10:34.554243+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":161,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9153},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9153":{"gid":9153,"name":"c","rank":0,"incarnation":173,"state":"up:active","state_seq":6,"addr":"172.21.15.28:6833/2178509171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2178509171},{"type":"v1","addr":"172.21.15.28:6833","nonce":2178509171}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2024-02-21T23:10:37.102 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 176 2024-02-21T23:10:37.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.116+0000 7f8b7aa00640 1 -- 172.21.15.28:0/2290601316 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8b5406b4d0 msgr2=0x7f8b5406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:37.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.116+0000 7f8b7aa00640 1 --2- 172.21.15.28:0/2290601316 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8b5406b4d0 0x7f8b5406d990 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f8b60007fd0 tx=0x7f8b60007920 comp rx=0 tx=0).stop 2024-02-21T23:10:37.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.116+0000 7f8b7aa00640 1 -- 172.21.15.28:0/2290601316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b7406dbf0 msgr2=0x7f8b740c8d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:37.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.116+0000 7f8b7aa00640 1 --2- 172.21.15.28:0/2290601316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b7406dbf0 0x7f8b740c8d30 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7f8b68009860 tx=0x7f8b680046c0 comp rx=0 tx=0).stop 2024-02-21T23:10:37.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.116+0000 7f8b7aa00640 1 -- 172.21.15.28:0/2290601316 shutdown_connections 2024-02-21T23:10:37.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.116+0000 7f8b7aa00640 1 --2- 172.21.15.28:0/2290601316 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8b5406b4d0 0x7f8b5406d990 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:37.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.116+0000 7f8b7aa00640 1 --2- 172.21.15.28:0/2290601316 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8b740aff90 0x7f8b740c35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:37.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.116+0000 7f8b7aa00640 1 --2- 172.21.15.28:0/2290601316 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b740c5720 0x7f8b74151c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:37.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.116+0000 7f8b7aa00640 1 --2- 172.21.15.28:0/2290601316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b7406dbf0 0x7f8b740c8d30 unknown :-1 s=CLOSED pgs=1840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:37.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.116+0000 7f8b7aa00640 1 -- 172.21.15.28:0/2290601316 >> 172.21.15.28:0/2290601316 conn(0x7f8b740ac370 msgr2=0x7f8b740ac750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:37.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.117+0000 7f8b7aa00640 1 -- 172.21.15.28:0/2290601316 shutdown_connections 2024-02-21T23:10:37.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.117+0000 7f8b7aa00640 1 -- 172.21.15.28:0/2290601316 wait complete. 2024-02-21T23:10:37.125 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:10:37.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.346+0000 7f9f07200640 1 Processor -- start 2024-02-21T23:10:37.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.346+0000 7f9f07200640 1 -- start start 2024-02-21T23:10:37.337 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-21T23:10:37.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.349+0000 7f9f07200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f9f080c4d20 0x7f9f08159600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:37.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.349+0000 7f9f07200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9f080c1910 0x7f9f08159b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:37.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.349+0000 7f9f07200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f0806af10 con 0x7f9f08156a90 2024-02-21T23:10:37.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.349+0000 7f9f07200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f0806fbd0 con 0x7f9f080c1910 2024-02-21T23:10:37.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.349+0000 7f9f07200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f0806b1b0 con 0x7f9f080c4d20 2024-02-21T23:10:37.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.349+0000 7f9f05e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f9f08156a90 0x7f9f0806f130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:47824/0 (socket says 172.21.15.28:47824) 2024-02-21T23:10:37.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.350+0000 7f9f05e00640 1 -- 172.21.15.28:0/3120738035 learned_addr learned my addr 172.21.15.28:0/3120738035 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:10:37.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.350+0000 7f9f06800640 1 --2- 172.21.15.28:0/3120738035 >> v2:172.21.15.28:3300/0 conn(0x7f9f080c1910 0x7f9f08159b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:37.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.350+0000 7f9f05400640 1 --2- 172.21.15.28:0/3120738035 >> v2:172.21.15.81:3300/0 conn(0x7f9f080c4d20 0x7f9f08159600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:37.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.350+0000 7f9f04a00640 1 -- 172.21.15.28:0/3120738035 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3913881260 0 0) 0x7f9f0806af10 con 0x7f9f08156a90 2024-02-21T23:10:37.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.350+0000 7f9f06800640 1 -- 172.21.15.28:0/3120738035 >> v2:172.21.15.81:3300/0 conn(0x7f9f080c4d20 msgr2=0x7f9f08159600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:37.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.350+0000 7f9f06800640 1 --2- 172.21.15.28:0/3120738035 >> v2:172.21.15.81:3300/0 conn(0x7f9f080c4d20 0x7f9f08159600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:37.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.350+0000 7f9f06800640 1 -- 172.21.15.28:0/3120738035 >> v1:172.21.15.28:6789/0 conn(0x7f9f08156a90 legacy=0x7f9f0806f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:37.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.350+0000 7f9f06800640 1 -- 172.21.15.28:0/3120738035 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f0806fe20 con 0x7f9f080c1910 2024-02-21T23:10:37.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.351+0000 7f9f06800640 1 --2- 172.21.15.28:0/3120738035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f080c1910 0x7f9f08159b40 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7f9f0806fd50 tx=0x7f9f0002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68deca7a5204a914 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:37.342 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-21T23:10:37.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.352+0000 7f9f04a00640 1 -- 172.21.15.28:0/3120738035 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f00039070 con 0x7f9f080c1910 2024-02-21T23:10:37.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.352+0000 7f9f04a00640 1 -- 172.21.15.28:0/3120738035 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f0002cbd0 con 0x7f9f080c1910 2024-02-21T23:10:37.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.352+0000 7f9f04a00640 1 -- 172.21.15.28:0/3120738035 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f00034440 con 0x7f9f080c1910 2024-02-21T23:10:37.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.352+0000 7f9f07200640 1 -- 172.21.15.28:0/3120738035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f080c1910 msgr2=0x7f9f08159b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:37.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.352+0000 7f9f07200640 1 --2- 172.21.15.28:0/3120738035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f080c1910 0x7f9f08159b40 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7f9f0806fd50 tx=0x7f9f0002be20 comp rx=0 tx=0).stop 2024-02-21T23:10:37.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.353+0000 7f9f07200640 1 -- 172.21.15.28:0/3120738035 shutdown_connections 2024-02-21T23:10:37.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.353+0000 7f9f07200640 1 --2- 172.21.15.28:0/3120738035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f080c1910 0x7f9f08159b40 unknown :-1 s=CLOSED pgs=1842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:37.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.353+0000 7f9f07200640 1 --2- 172.21.15.28:0/3120738035 >> v2:172.21.15.81:3300/0 conn(0x7f9f080c4d20 0x7f9f08159600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:37.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.353+0000 7f9f07200640 1 -- 172.21.15.28:0/3120738035 >> 172.21.15.28:0/3120738035 conn(0x7f9f08067050 msgr2=0x7f9f080a84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:37.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.353+0000 7f9f07200640 1 -- 172.21.15.28:0/3120738035 shutdown_connections 2024-02-21T23:10:37.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.353+0000 7f9f07200640 1 -- 172.21.15.28:0/3120738035 wait complete. 2024-02-21T23:10:37.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.354+0000 7f9f07200640 1 Processor -- start 2024-02-21T23:10:37.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.355+0000 7f9f07200640 1 -- start start 2024-02-21T23:10:37.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.355+0000 7f9f07200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f080c1910 0x7f9f080ac1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:37.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.355+0000 7f9f07200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9f080c4d20 0x7f9f0815bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:37.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.356+0000 7f9f07200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f08156a90 0x7f9f081548d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:37.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.356+0000 7f9f07200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f0806f9c0 con 0x7f9f080c1910 2024-02-21T23:10:37.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.356+0000 7f9f07200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f0806f840 con 0x7f9f08156a90 2024-02-21T23:10:37.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.356+0000 7f9f07200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f0806bc60 con 0x7f9f080c4d20 2024-02-21T23:10:37.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.356+0000 7f9f05e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f080c1910 0x7f9f080ac1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:37.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.356+0000 7f9f05e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f080c1910 0x7f9f080ac1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53420/0 (socket says 172.21.15.28:53420) 2024-02-21T23:10:37.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.356+0000 7f9f05e00640 1 -- 172.21.15.28:0/334264247 learned_addr learned my addr 172.21.15.28:0/334264247 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:37.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.356+0000 7f9f05400640 1 --2- 172.21.15.28:0/334264247 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9f080c4d20 0x7f9f0815bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:37.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.356+0000 7f9f06800640 1 --2- 172.21.15.28:0/334264247 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f08156a90 0x7f9f081548d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:37.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.356+0000 7f9f05e00640 1 -- 172.21.15.28:0/334264247 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9f080c4d20 msgr2=0x7f9f0815bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:37.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.356+0000 7f9f05e00640 1 --2- 172.21.15.28:0/334264247 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9f080c4d20 0x7f9f0815bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:37.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.356+0000 7f9f05e00640 1 -- 172.21.15.28:0/334264247 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f08156a90 msgr2=0x7f9f081548d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:37.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.356+0000 7f9f05e00640 1 --2- 172.21.15.28:0/334264247 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f08156a90 0x7f9f081548d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:37.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.356+0000 7f9f05e00640 1 -- 172.21.15.28:0/334264247 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f080aaa90 con 0x7f9f080c1910 2024-02-21T23:10:37.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.357+0000 7f9f05e00640 1 --2- 172.21.15.28:0/334264247 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f080c1910 0x7f9f080ac1d0 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7f9ef800c460 tx=0x7f9ef800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:37.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.357+0000 7f9efe000640 1 -- 172.21.15.28:0/334264247 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ef8018070 con 0x7f9f080c1910 2024-02-21T23:10:37.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.357+0000 7f9efe000640 1 -- 172.21.15.28:0/334264247 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ef8007500 con 0x7f9f080c1910 2024-02-21T23:10:37.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.357+0000 7f9f07200640 1 -- 172.21.15.28:0/334264247 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f080ab6e0 con 0x7f9f080c1910 2024-02-21T23:10:37.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.357+0000 7f9efe000640 1 -- 172.21.15.28:0/334264247 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ef8004a20 con 0x7f9f080c1910 2024-02-21T23:10:37.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.357+0000 7f9f07200640 1 -- 172.21.15.28:0/334264247 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f080ab980 con 0x7f9f080c1910 2024-02-21T23:10:37.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.359+0000 7f9f07200640 1 -- 172.21.15.28:0/334264247 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f0806f130 con 0x7f9f080c1910 2024-02-21T23:10:37.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.360+0000 7f9efe000640 1 -- 172.21.15.28:0/334264247 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9ef801b070 con 0x7f9f080c1910 2024-02-21T23:10:37.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.361+0000 7f9efe000640 1 --2- 172.21.15.28:0/334264247 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9ee806b4d0 0x7f9ee806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:37.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.361+0000 7f9efe000640 1 -- 172.21.15.28:0/334264247 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 6536+0+0 (secure 0 0 0) 0x7f9ef8089840 con 0x7f9f080c1910 2024-02-21T23:10:37.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.362+0000 7f9f05400640 1 --2- 172.21.15.28:0/334264247 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9ee806b4d0 0x7f9ee806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:37.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.363+0000 7f9f05400640 1 --2- 172.21.15.28:0/334264247 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9ee806b4d0 0x7f9ee806d990 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f9f0815a680 tx=0x7f9ef4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:37.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.366+0000 7f9efe000640 1 -- 172.21.15.28:0/334264247 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f9f0806f130 con 0x7f9f080c1910 2024-02-21T23:10:37.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.574+0000 7f9f07200640 1 -- 172.21.15.28:0/334264247 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9f0804f290 con 0x7f9f080c1910 2024-02-21T23:10:37.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.577+0000 7f9efe000640 1 -- 172.21.15.28:0/334264247 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v163) v1 ==== 74+0+17551 (secure 0 0 0) 0x7f9ef80582e0 con 0x7f9f080c1910 2024-02-21T23:10:37.566 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:10:37.566 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":163,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:10:34.681492+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs2_metadata","create_time":"2024-02-21T23:10:04.095451+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"154","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs2_data","create_time":"2024-02-21T23:10:04.836982+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"26.29","mappings":[{"from":1,"to":0}]},{"pgid":"26.36","mappings":[{"from":3,"to":0}]},{"pgid":"26.3f","mappings":[{"from":1,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.81:6835/1320495906":"2024-02-22T23:10:32.538190+0000","172.21.15.81:6834/1320495906":"2024-02-22T23:10:32.538190+0000","172.21.15.81:6837/521625945":"2024-02-22T23:10:31.712025+0000","172.21.15.28:6833/2975378719":"2024-02-22T23:10:21.581948+0000","172.21.15.28:6832/2975378719":"2024-02-22T23:10:21.581948+0000","172.21.15.81:6836/521625945":"2024-02-22T23:10:31.712025+0000","172.21.15.28:6835/2928639298":"2024-02-22T23:10:19.550812+0000","172.21.15.28:6834/2928639298":"2024-02-22T23:10:19.550812+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:10:37.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.580+0000 7f9f07200640 1 -- 172.21.15.28:0/334264247 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9ee806b4d0 msgr2=0x7f9ee806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:37.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.580+0000 7f9f07200640 1 --2- 172.21.15.28:0/334264247 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9ee806b4d0 0x7f9ee806d990 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f9f0815a680 tx=0x7f9ef4008040 comp rx=0 tx=0).stop 2024-02-21T23:10:37.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.580+0000 7f9f07200640 1 -- 172.21.15.28:0/334264247 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f080c1910 msgr2=0x7f9f080ac1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:37.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.580+0000 7f9f07200640 1 --2- 172.21.15.28:0/334264247 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f080c1910 0x7f9f080ac1d0 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7f9ef800c460 tx=0x7f9ef800c920 comp rx=0 tx=0).stop 2024-02-21T23:10:37.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.580+0000 7f9f07200640 1 -- 172.21.15.28:0/334264247 shutdown_connections 2024-02-21T23:10:37.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.580+0000 7f9f07200640 1 --2- 172.21.15.28:0/334264247 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9ee806b4d0 0x7f9ee806d990 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:37.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.580+0000 7f9f07200640 1 --2- 172.21.15.28:0/334264247 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9f080c4d20 0x7f9f0815bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:37.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.580+0000 7f9f07200640 1 --2- 172.21.15.28:0/334264247 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f08156a90 0x7f9f081548d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:37.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.580+0000 7f9f07200640 1 --2- 172.21.15.28:0/334264247 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f080c1910 0x7f9f080ac1d0 unknown :-1 s=CLOSED pgs=1843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:37.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.580+0000 7f9f07200640 1 -- 172.21.15.28:0/334264247 >> 172.21.15.28:0/334264247 conn(0x7f9f08067050 msgr2=0x7f9f080a9bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:37.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.581+0000 7f9f07200640 1 -- 172.21.15.28:0/334264247 shutdown_connections 2024-02-21T23:10:37.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.581+0000 7f9f07200640 1 -- 172.21.15.28:0/334264247 wait complete. 2024-02-21T23:10:37.588 INFO:tasks.cephfs.filesystem:Destroying file system cephfs2 and related pools 2024-02-21T23:10:37.588 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:10:37.596 INFO:tasks.ceph.mds.c.smithi028.stderr:2024-02-21T23:10:37.605+0000 7fe517e00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-21T23:10:37.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.823+0000 7f139f200640 1 Processor -- start 2024-02-21T23:10:37.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.823+0000 7f139f200640 1 -- start start 2024-02-21T23:10:37.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.826+0000 7f139f200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f13a00c40b0 0x7f13a00c4510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:37.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.826+0000 7f139f200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f13a0159230 0x7f13a01596a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:37.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.826+0000 7f139f200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13a015c8c0 con 0x7f13a01567b0 2024-02-21T23:10:37.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.826+0000 7f139f200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13a0070240 con 0x7f13a00c40b0 2024-02-21T23:10:37.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.826+0000 7f139f200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13a015d490 con 0x7f13a0159230 2024-02-21T23:10:37.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.827+0000 7f1395400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f13a00c40b0 0x7f13a00c4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:37.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.827+0000 7f139de00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f13a01567b0 0x7f13a006fb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:47836/0 (socket says 172.21.15.28:47836) 2024-02-21T23:10:37.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.827+0000 7f139de00640 1 -- 172.21.15.28:0/2952071938 learned_addr learned my addr 172.21.15.28:0/2952071938 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:10:37.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.827+0000 7f139e800640 1 --2- 172.21.15.28:0/2952071938 >> v2:172.21.15.81:3300/0 conn(0x7f13a0159230 0x7f13a01596a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:37.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.827+0000 7f1395400640 1 -- 172.21.15.28:0/2952071938 >> v2:172.21.15.81:3300/0 conn(0x7f13a0159230 msgr2=0x7f13a01596a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:37.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.827+0000 7f1395400640 1 --2- 172.21.15.28:0/2952071938 >> v2:172.21.15.81:3300/0 conn(0x7f13a0159230 0x7f13a01596a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:37.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.827+0000 7f1395400640 1 -- 172.21.15.28:0/2952071938 >> v1:172.21.15.28:6789/0 conn(0x7f13a01567b0 legacy=0x7f13a006fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:37.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.827+0000 7f1395400640 1 -- 172.21.15.28:0/2952071938 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13a015cb60 con 0x7f13a00c40b0 2024-02-21T23:10:37.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.827+0000 7f139d400640 1 -- 172.21.15.28:0/2952071938 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 627322098 0 0) 0x7f13a015c8c0 con 0x7f13a01567b0 2024-02-21T23:10:37.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.829+0000 7f1395400640 1 --2- 172.21.15.28:0/2952071938 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13a00c40b0 0x7f13a00c4510 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7f13a006ddc0 tx=0x7f138802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78d8c761dcb28c88 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:37.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.829+0000 7f139d400640 1 -- 172.21.15.28:0/2952071938 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1388034070 con 0x7f13a00c40b0 2024-02-21T23:10:37.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.829+0000 7f139d400640 1 -- 172.21.15.28:0/2952071938 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1388030610 con 0x7f13a00c40b0 2024-02-21T23:10:37.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.829+0000 7f139d400640 1 -- 172.21.15.28:0/2952071938 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13880309e0 con 0x7f13a00c40b0 2024-02-21T23:10:37.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.830+0000 7f139f200640 1 -- 172.21.15.28:0/2952071938 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13a00c40b0 msgr2=0x7f13a00c4510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:37.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.830+0000 7f139f200640 1 --2- 172.21.15.28:0/2952071938 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13a00c40b0 0x7f13a00c4510 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7f13a006ddc0 tx=0x7f138802b960 comp rx=0 tx=0).stop 2024-02-21T23:10:37.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.830+0000 7f139f200640 1 -- 172.21.15.28:0/2952071938 shutdown_connections 2024-02-21T23:10:37.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.830+0000 7f139f200640 1 --2- 172.21.15.28:0/2952071938 >> v2:172.21.15.81:3300/0 conn(0x7f13a0159230 0x7f13a01596a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:37.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.830+0000 7f139f200640 1 --2- 172.21.15.28:0/2952071938 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13a00c40b0 0x7f13a00c4510 unknown :-1 s=CLOSED pgs=1845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:37.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.830+0000 7f139f200640 1 -- 172.21.15.28:0/2952071938 >> 172.21.15.28:0/2952071938 conn(0x7f13a0067050 msgr2=0x7f13a006bf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:37.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.830+0000 7f139f200640 1 -- 172.21.15.28:0/2952071938 shutdown_connections 2024-02-21T23:10:37.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.830+0000 7f139f200640 1 -- 172.21.15.28:0/2952071938 wait complete. 2024-02-21T23:10:37.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.832+0000 7f139f200640 1 Processor -- start 2024-02-21T23:10:37.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.832+0000 7f139f200640 1 -- start start 2024-02-21T23:10:37.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.833+0000 7f139f200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f13a00c40b0 0x7f13a01f2720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:37.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.833+0000 7f139f200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13a01567b0 0x7f13a01ef150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:37.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.833+0000 7f139f200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f13a0159230 0x7f13a0152c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:37.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.833+0000 7f139f200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13a0156b90 con 0x7f13a01567b0 2024-02-21T23:10:37.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.833+0000 7f139f200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13a015d120 con 0x7f13a0159230 2024-02-21T23:10:37.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.833+0000 7f139f200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13a00c5f30 con 0x7f13a00c40b0 2024-02-21T23:10:37.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.833+0000 7f1395400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13a01567b0 0x7f13a01ef150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:37.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.833+0000 7f1395400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13a01567b0 0x7f13a01ef150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53442/0 (socket says 172.21.15.28:53442) 2024-02-21T23:10:37.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.833+0000 7f1395400640 1 -- 172.21.15.28:0/3936305028 learned_addr learned my addr 172.21.15.28:0/3936305028 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:37.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.833+0000 7f139e800640 1 --2- 172.21.15.28:0/3936305028 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f13a0159230 0x7f13a0152c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:37.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.833+0000 7f139de00640 1 --2- 172.21.15.28:0/3936305028 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f13a00c40b0 0x7f13a01f2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:37.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.834+0000 7f1395400640 1 -- 172.21.15.28:0/3936305028 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f13a00c40b0 msgr2=0x7f13a01f2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:37.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.834+0000 7f1395400640 1 --2- 172.21.15.28:0/3936305028 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f13a00c40b0 0x7f13a01f2720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:37.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.834+0000 7f1395400640 1 -- 172.21.15.28:0/3936305028 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f13a0159230 msgr2=0x7f13a0152c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:37.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.834+0000 7f1395400640 1 --2- 172.21.15.28:0/3936305028 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f13a0159230 0x7f13a0152c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:37.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.834+0000 7f1395400640 1 -- 172.21.15.28:0/3936305028 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13a00a8ad0 con 0x7f13a01567b0 2024-02-21T23:10:37.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.834+0000 7f139e800640 1 --2- 172.21.15.28:0/3936305028 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f13a0159230 0x7f13a0152c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:10:37.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.834+0000 7f1395400640 1 --2- 172.21.15.28:0/3936305028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13a01567b0 0x7f13a01ef150 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7f138802bf70 tx=0x7f13880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:37.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.834+0000 7f1396a00640 1 -- 172.21.15.28:0/3936305028 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1388046020 con 0x7f13a01567b0 2024-02-21T23:10:37.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.834+0000 7f1396a00640 1 -- 172.21.15.28:0/3936305028 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1388039a10 con 0x7f13a01567b0 2024-02-21T23:10:37.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.834+0000 7f1396a00640 1 -- 172.21.15.28:0/3936305028 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1388034040 con 0x7f13a01567b0 2024-02-21T23:10:37.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.834+0000 7f139f200640 1 -- 172.21.15.28:0/3936305028 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13a00a8db0 con 0x7f13a01567b0 2024-02-21T23:10:37.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.835+0000 7f139f200640 1 -- 172.21.15.28:0/3936305028 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13a01f1f20 con 0x7f13a01567b0 2024-02-21T23:10:37.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.836+0000 7f139f200640 1 -- 172.21.15.28:0/3936305028 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13a004f290 con 0x7f13a01567b0 2024-02-21T23:10:37.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.837+0000 7f1396a00640 1 -- 172.21.15.28:0/3936305028 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1388037070 con 0x7f13a01567b0 2024-02-21T23:10:37.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.837+0000 7f1396a00640 1 --2- 172.21.15.28:0/3936305028 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f137c06b4d0 0x7f137c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:37.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.838+0000 7f1396a00640 1 -- 172.21.15.28:0/3936305028 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 6536+0+0 (secure 0 0 0) 0x7f13880ac2e0 con 0x7f13a01567b0 2024-02-21T23:10:37.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.838+0000 7f139de00640 1 --2- 172.21.15.28:0/3936305028 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f137c06b4d0 0x7f137c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:37.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.839+0000 7f139de00640 1 --2- 172.21.15.28:0/3936305028 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f137c06b4d0 0x7f137c06d990 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f13900012e0 tx=0x7f1390007ef0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:37.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:37.842+0000 7f1396a00640 1 -- 172.21.15.28:0/3936305028 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1388050020 con 0x7f13a01567b0 2024-02-21T23:10:38.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.082+0000 7f139f200640 1 -- 172.21.15.28:0/3936305028 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f13a006f780 con 0x7f13a01567b0 2024-02-21T23:10:38.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.084+0000 7f1396a00640 1 -- 172.21.15.28:0/3936305028 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 176 v176) v1 ==== 95+0+4948 (secure 0 0 0) 0x7f1388039e20 con 0x7f13a01567b0 2024-02-21T23:10:38.072 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:10:38.072 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":176,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6599,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/704098282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":704098282},{"type":"v1","addr":"172.21.15.81:6837","nonce":704098282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":173},{"gid":7372,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1804589714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1804589714},{"type":"v1","addr":"172.21.15.81:6835","nonce":1804589714}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":174},{"gid":9141,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2895215997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2895215997},{"type":"v1","addr":"172.21.15.28:6835","nonce":2895215997}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":166}],"filesystems":[{"mdsmap":{"epoch":176,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:10:05.843850+0000","modified":"2024-02-21T23:10:34.554243+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":161,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9153},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9153":{"gid":9153,"name":"c","rank":0,"incarnation":173,"state":"up:active","state_seq":6,"addr":"172.21.15.28:6833/2178509171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2178509171},{"type":"v1","addr":"172.21.15.28:6833","nonce":2178509171}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2024-02-21T23:10:38.072 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 176 2024-02-21T23:10:38.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.087+0000 7f139f200640 1 -- 172.21.15.28:0/3936305028 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f137c06b4d0 msgr2=0x7f137c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:38.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.087+0000 7f139f200640 1 --2- 172.21.15.28:0/3936305028 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f137c06b4d0 0x7f137c06d990 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f13900012e0 tx=0x7f1390007ef0 comp rx=0 tx=0).stop 2024-02-21T23:10:38.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.087+0000 7f139f200640 1 -- 172.21.15.28:0/3936305028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13a01567b0 msgr2=0x7f13a01ef150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:38.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.087+0000 7f139f200640 1 --2- 172.21.15.28:0/3936305028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13a01567b0 0x7f13a01ef150 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7f138802bf70 tx=0x7f13880046c0 comp rx=0 tx=0).stop 2024-02-21T23:10:38.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.087+0000 7f139f200640 1 -- 172.21.15.28:0/3936305028 shutdown_connections 2024-02-21T23:10:38.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.087+0000 7f139f200640 1 --2- 172.21.15.28:0/3936305028 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f137c06b4d0 0x7f137c06d990 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:38.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.087+0000 7f139f200640 1 --2- 172.21.15.28:0/3936305028 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f13a0159230 0x7f13a0152c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:38.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.087+0000 7f139f200640 1 --2- 172.21.15.28:0/3936305028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13a01567b0 0x7f13a01ef150 unknown :-1 s=CLOSED pgs=1846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:38.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.087+0000 7f139f200640 1 --2- 172.21.15.28:0/3936305028 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f13a00c40b0 0x7f13a01f2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:38.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.087+0000 7f139f200640 1 -- 172.21.15.28:0/3936305028 >> 172.21.15.28:0/3936305028 conn(0x7f13a0067050 msgr2=0x7f13a0072d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:38.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.087+0000 7f139f200640 1 -- 172.21.15.28:0/3936305028 shutdown_connections 2024-02-21T23:10:38.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.087+0000 7f139f200640 1 -- 172.21.15.28:0/3936305028 wait complete. 2024-02-21T23:10:38.093 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:10:38.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.318+0000 7f5d14a00640 1 Processor -- start 2024-02-21T23:10:38.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.318+0000 7f5d14a00640 1 -- start start 2024-02-21T23:10:38.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.320+0000 7f5d14a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f5d100c7ef0 0x7f5d100c64d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:38.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.320+0000 7f5d14a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d100c2c20 con 0x7f5d100c8940 2024-02-21T23:10:38.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.320+0000 7f5d14a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d10155100 con 0x7f5d100c8560 2024-02-21T23:10:38.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.320+0000 7f5d14a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d100c8db0 con 0x7f5d100c7ef0 2024-02-21T23:10:38.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.320+0000 7f5d0f200640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f5d100c8940 0x7f5d100c54a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:47842/0 (socket says 172.21.15.28:47842) 2024-02-21T23:10:38.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.320+0000 7f5d0f200640 1 -- 172.21.15.28:0/611585732 learned_addr learned my addr 172.21.15.28:0/611585732 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:10:38.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.320+0000 7f5d0de00640 1 --2- 172.21.15.28:0/611585732 >> v2:172.21.15.81:3300/0 conn(0x7f5d100c7ef0 0x7f5d100c64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:38.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.321+0000 7f5d0d400640 1 -- 172.21.15.28:0/611585732 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 719397062 0 0) 0x7f5d100c2c20 con 0x7f5d100c8940 2024-02-21T23:10:38.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.321+0000 7f5d0d400640 1 -- 172.21.15.28:0/611585732 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cf4003520 con 0x7f5d100c8940 2024-02-21T23:10:38.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.321+0000 7f5d0d400640 1 -- 172.21.15.28:0/611585732 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2779717285 0 0) 0x7f5cf4003520 con 0x7f5d100c8940 2024-02-21T23:10:38.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.321+0000 7f5d0d400640 1 -- 172.21.15.28:0/611585732 >> v2:172.21.15.81:3300/0 conn(0x7f5d100c7ef0 msgr2=0x7f5d100c64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:38.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.321+0000 7f5d0d400640 1 --2- 172.21.15.28:0/611585732 >> v2:172.21.15.81:3300/0 conn(0x7f5d100c7ef0 0x7f5d100c64d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:38.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.321+0000 7f5d0d400640 1 -- 172.21.15.28:0/611585732 >> v1:172.21.15.81:6789/0 conn(0x7f5d100c8560 legacy=0x7f5d100c4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:38.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.321+0000 7f5d0d400640 1 -- 172.21.15.28:0/611585732 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d101553e0 con 0x7f5d100c8940 2024-02-21T23:10:38.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.321+0000 7f5d0d400640 1 -- 172.21.15.28:0/611585732 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f5d080019f0 con 0x7f5d100c8940 2024-02-21T23:10:38.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.321+0000 7f5d0d400640 1 -- 172.21.15.28:0/611585732 >> v1:172.21.15.28:6789/0 conn(0x7f5d100c8940 legacy=0x7f5d100c54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:38.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.321+0000 7f5d0d400640 1 --2- 172.21.15.28:0/611585732 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5cf4003a00 0x7f5cf4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:38.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.321+0000 7f5d0d400640 1 --2- 172.21.15.28:0/611585732 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5cf4004800 0x7f5cf4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:38.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.321+0000 7f5d0d400640 1 --2- 172.21.15.28:0/611585732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cf4005210 0x7f5cf400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:38.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.321+0000 7f5d0d400640 1 -- 172.21.15.28:0/611585732 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d100c8db0 con 0x7f5cf4005210 2024-02-21T23:10:38.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.321+0000 7f5d0d400640 1 -- 172.21.15.28:0/611585732 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cf400c160 con 0x7f5cf4004800 2024-02-21T23:10:38.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.321+0000 7f5d0d400640 1 -- 172.21.15.28:0/611585732 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cf400c360 con 0x7f5cf4003a00 2024-02-21T23:10:38.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.322+0000 7f5d0f200640 1 --2- 172.21.15.28:0/611585732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cf4005210 0x7f5cf400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:38.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.322+0000 7f5d0de00640 1 --2- 172.21.15.28:0/611585732 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5cf4004800 0x7f5cf4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:38.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.322+0000 7f5d0e800640 1 --2- 172.21.15.28:0/611585732 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5cf4003a00 0x7f5cf4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:38.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.322+0000 7f5d0f200640 1 -- 172.21.15.28:0/611585732 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5cf4003a00 msgr2=0x7f5cf4003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:38.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.322+0000 7f5d0f200640 1 --2- 172.21.15.28:0/611585732 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5cf4003a00 0x7f5cf4003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:38.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.322+0000 7f5d0f200640 1 -- 172.21.15.28:0/611585732 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5cf4004800 msgr2=0x7f5cf4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:38.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.322+0000 7f5d0f200640 1 --2- 172.21.15.28:0/611585732 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5cf4004800 0x7f5cf4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:38.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.322+0000 7f5d0f200640 1 -- 172.21.15.28:0/611585732 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cf4003520 con 0x7f5cf4005210 2024-02-21T23:10:38.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.322+0000 7f5d0de00640 1 --2- 172.21.15.28:0/611585732 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5cf4004800 0x7f5cf4004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:10:38.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.323+0000 7f5d0f200640 1 -- 172.21.15.28:0/611585732 reap_dead start 2024-02-21T23:10:38.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.323+0000 7f5d0f200640 1 --2- 172.21.15.28:0/611585732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cf4005210 0x7f5cf400b9c0 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f5d0800a780 tx=0x7f5cf4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d93153c98625c405 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:38.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.323+0000 7f5d0d400640 1 -- 172.21.15.28:0/611585732 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d0800a3d0 con 0x7f5cf4005210 2024-02-21T23:10:38.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.323+0000 7f5d0d400640 1 -- 172.21.15.28:0/611585732 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d08034650 con 0x7f5cf4005210 2024-02-21T23:10:38.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.323+0000 7f5d0d400640 1 -- 172.21.15.28:0/611585732 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d08034a40 con 0x7f5cf4005210 2024-02-21T23:10:38.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.324+0000 7f5d14a00640 1 -- 172.21.15.28:0/611585732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cf4005210 msgr2=0x7f5cf400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:38.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.324+0000 7f5d14a00640 1 --2- 172.21.15.28:0/611585732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cf4005210 0x7f5cf400b9c0 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f5d0800a780 tx=0x7f5cf4005830 comp rx=0 tx=0).stop 2024-02-21T23:10:38.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.324+0000 7f5d14a00640 1 -- 172.21.15.28:0/611585732 shutdown_connections 2024-02-21T23:10:38.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.324+0000 7f5d14a00640 1 --2- 172.21.15.28:0/611585732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cf4005210 0x7f5cf400b9c0 unknown :-1 s=CLOSED pgs=1848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:38.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.324+0000 7f5d14a00640 1 -- 172.21.15.28:0/611585732 >> 172.21.15.28:0/611585732 conn(0x7f5d1006ae70 msgr2=0x7f5d1006b280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:38.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.324+0000 7f5d14a00640 1 -- 172.21.15.28:0/611585732 shutdown_connections 2024-02-21T23:10:38.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.324+0000 7f5d14a00640 1 -- 172.21.15.28:0/611585732 wait complete. 2024-02-21T23:10:38.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.326+0000 7f5d14a00640 1 Processor -- start 2024-02-21T23:10:38.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.326+0000 7f5d14a00640 1 -- start start 2024-02-21T23:10:38.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.327+0000 7f5d14a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cf4005210 0x7f5d101506b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:38.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.327+0000 7f5d14a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5d10150bf0 0x7f5d100a91b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:38.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.327+0000 7f5d14a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d100aad20 0x7f5d100a96f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:38.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.327+0000 7f5d14a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d10154db0 con 0x7f5cf4005210 2024-02-21T23:10:38.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.327+0000 7f5d14a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d101559a0 con 0x7f5d100aad20 2024-02-21T23:10:38.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.327+0000 7f5d14a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d10071a90 con 0x7f5d10150bf0 2024-02-21T23:10:38.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.327+0000 7f5d0e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cf4005210 0x7f5d101506b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:38.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.327+0000 7f5d0e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cf4005210 0x7f5d101506b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53456/0 (socket says 172.21.15.28:53456) 2024-02-21T23:10:38.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.327+0000 7f5d0e800640 1 -- 172.21.15.28:0/1745485080 learned_addr learned my addr 172.21.15.28:0/1745485080 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:38.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.328+0000 7f5d0de00640 1 --2- 172.21.15.28:0/1745485080 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5d10150bf0 0x7f5d100a91b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:38.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.328+0000 7f5d0f200640 1 --2- 172.21.15.28:0/1745485080 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d100aad20 0x7f5d100a96f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:38.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.328+0000 7f5d0e800640 1 -- 172.21.15.28:0/1745485080 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5d10150bf0 msgr2=0x7f5d100a91b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:38.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.328+0000 7f5d0e800640 1 --2- 172.21.15.28:0/1745485080 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5d10150bf0 0x7f5d100a91b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:38.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.328+0000 7f5d0e800640 1 -- 172.21.15.28:0/1745485080 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d100aad20 msgr2=0x7f5d100a96f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:38.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.328+0000 7f5d0e800640 1 --2- 172.21.15.28:0/1745485080 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d100aad20 0x7f5d100a96f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:38.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.328+0000 7f5d0e800640 1 -- 172.21.15.28:0/1745485080 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d10156ac0 con 0x7f5cf4005210 2024-02-21T23:10:38.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.328+0000 7f5d0e800640 1 --2- 172.21.15.28:0/1745485080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cf4005210 0x7f5d101506b0 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f5d0000e4d0 tx=0x7f5d0000e990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:38.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.328+0000 7f5d06a00640 1 -- 172.21.15.28:0/1745485080 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d000052e0 con 0x7f5cf4005210 2024-02-21T23:10:38.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.328+0000 7f5d06a00640 1 -- 172.21.15.28:0/1745485080 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d00018070 con 0x7f5cf4005210 2024-02-21T23:10:38.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.328+0000 7f5d06a00640 1 -- 172.21.15.28:0/1745485080 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d00003810 con 0x7f5cf4005210 2024-02-21T23:10:38.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.328+0000 7f5d14a00640 1 -- 172.21.15.28:0/1745485080 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d100a9e10 con 0x7f5cf4005210 2024-02-21T23:10:38.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.329+0000 7f5d14a00640 1 -- 172.21.15.28:0/1745485080 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d10156d80 con 0x7f5cf4005210 2024-02-21T23:10:38.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.330+0000 7f5d14a00640 1 -- 172.21.15.28:0/1745485080 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d1004f290 con 0x7f5cf4005210 2024-02-21T23:10:38.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.332+0000 7f5d06a00640 1 -- 172.21.15.28:0/1745485080 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5d000039b0 con 0x7f5cf4005210 2024-02-21T23:10:38.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.333+0000 7f5d06a00640 1 --2- 172.21.15.28:0/1745485080 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5cf006b500 0x7f5cf006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:38.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.333+0000 7f5d06a00640 1 -- 172.21.15.28:0/1745485080 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 6536+0+0 (secure 0 0 0) 0x7f5d0008a1f0 con 0x7f5cf4005210 2024-02-21T23:10:38.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.333+0000 7f5d0de00640 1 --2- 172.21.15.28:0/1745485080 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5cf006b500 0x7f5cf006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:38.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.335+0000 7f5d0de00640 1 --2- 172.21.15.28:0/1745485080 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5cf006b500 0x7f5cf006d9c0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f5d100ad460 tx=0x7f5cfc008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:38.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.342+0000 7f5d06a00640 1 -- 172.21.15.28:0/1745485080 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f5d00058c90 con 0x7f5cf4005210 2024-02-21T23:10:38.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.598+0000 7f5d14a00640 1 -- 172.21.15.28:0/1745485080 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5d100c1f40 con 0x7f5cf4005210 2024-02-21T23:10:38.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.599+0000 7f5d06a00640 1 -- 172.21.15.28:0/1745485080 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 176 v176) v1 ==== 95+0+4948 (secure 0 0 0) 0x7f5d00058590 con 0x7f5cf4005210 2024-02-21T23:10:38.588 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:10:38.589 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":176,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6599,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/704098282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":704098282},{"type":"v1","addr":"172.21.15.81:6837","nonce":704098282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":173},{"gid":7372,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1804589714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1804589714},{"type":"v1","addr":"172.21.15.81:6835","nonce":1804589714}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":174},{"gid":9141,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2895215997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2895215997},{"type":"v1","addr":"172.21.15.28:6835","nonce":2895215997}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":166}],"filesystems":[{"mdsmap":{"epoch":176,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:10:05.843850+0000","modified":"2024-02-21T23:10:34.554243+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":161,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9153},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9153":{"gid":9153,"name":"c","rank":0,"incarnation":173,"state":"up:active","state_seq":6,"addr":"172.21.15.28:6833/2178509171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2178509171},{"type":"v1","addr":"172.21.15.28:6833","nonce":2178509171}]},"join_fscid":12,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs2","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2024-02-21T23:10:38.589 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 176 2024-02-21T23:10:38.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.602+0000 7f5d14a00640 1 -- 172.21.15.28:0/1745485080 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5cf006b500 msgr2=0x7f5cf006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:38.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.602+0000 7f5d14a00640 1 --2- 172.21.15.28:0/1745485080 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5cf006b500 0x7f5cf006d9c0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f5d100ad460 tx=0x7f5cfc008040 comp rx=0 tx=0).stop 2024-02-21T23:10:38.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.602+0000 7f5d14a00640 1 -- 172.21.15.28:0/1745485080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cf4005210 msgr2=0x7f5d101506b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:38.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.602+0000 7f5d14a00640 1 --2- 172.21.15.28:0/1745485080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cf4005210 0x7f5d101506b0 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f5d0000e4d0 tx=0x7f5d0000e990 comp rx=0 tx=0).stop 2024-02-21T23:10:38.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.602+0000 7f5d14a00640 1 -- 172.21.15.28:0/1745485080 shutdown_connections 2024-02-21T23:10:38.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.602+0000 7f5d14a00640 1 --2- 172.21.15.28:0/1745485080 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5cf006b500 0x7f5cf006d9c0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:38.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.602+0000 7f5d14a00640 1 --2- 172.21.15.28:0/1745485080 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5d10150bf0 0x7f5d100a91b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:38.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.602+0000 7f5d14a00640 1 --2- 172.21.15.28:0/1745485080 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5d100aad20 0x7f5d100a96f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:38.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.602+0000 7f5d14a00640 1 --2- 172.21.15.28:0/1745485080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cf4005210 0x7f5d101506b0 unknown :-1 s=CLOSED pgs=1849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:38.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.602+0000 7f5d14a00640 1 -- 172.21.15.28:0/1745485080 >> 172.21.15.28:0/1745485080 conn(0x7f5d1006ae70 msgr2=0x7f5d1006b250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:38.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.602+0000 7f5d14a00640 1 -- 172.21.15.28:0/1745485080 shutdown_connections 2024-02-21T23:10:38.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.602+0000 7f5d14a00640 1 -- 172.21.15.28:0/1745485080 wait complete. 2024-02-21T23:10:38.612 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:10:38.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.840+0000 7fb9af200640 1 Processor -- start 2024-02-21T23:10:38.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.840+0000 7fb9af200640 1 -- start start 2024-02-21T23:10:38.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.841+0000 7fb9af200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fb9b0150450 0x7fb9b00a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:38.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.841+0000 7fb9af200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb9b006ee40 0x7fb9b01536f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:38.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.841+0000 7fb9af200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9b00c2760 con 0x7fb9b0156af0 2024-02-21T23:10:38.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.842+0000 7fb9af200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9b00c22c0 con 0x7fb9b0150450 2024-02-21T23:10:38.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.842+0000 7fb9af200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9b0156660 con 0x7fb9b006ee40 2024-02-21T23:10:38.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.842+0000 7fb9ae800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb9b0156af0 0x7fb9b006dff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:47848/0 (socket says 172.21.15.28:47848) 2024-02-21T23:10:38.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.842+0000 7fb9ae800640 1 -- 172.21.15.28:0/349073036 learned_addr learned my addr 172.21.15.28:0/349073036 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:10:38.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.842+0000 7fb9a5400640 1 --2- 172.21.15.28:0/349073036 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb9b006ee40 0x7fb9b01536f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:38.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.842+0000 7fb9ade00640 1 --2- 172.21.15.28:0/349073036 >> v2:172.21.15.81:3300/0 conn(0x7fb9b0150450 0x7fb9b00a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:38.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.842+0000 7fb9ad400640 1 -- 172.21.15.28:0/349073036 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2630670016 0 0) 0x7fb9b00c2760 con 0x7fb9b0156af0 2024-02-21T23:10:38.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.843+0000 7fb9ad400640 1 -- 172.21.15.28:0/349073036 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb994003520 con 0x7fb9b0156af0 2024-02-21T23:10:38.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.843+0000 7fb9ad400640 1 -- 172.21.15.28:0/349073036 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2166070740 0 0) 0x7fb994003520 con 0x7fb9b0156af0 2024-02-21T23:10:38.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.843+0000 7fb9ad400640 1 -- 172.21.15.28:0/349073036 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb9b006ee40 msgr2=0x7fb9b01536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:38.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.843+0000 7fb9ad400640 1 --2- 172.21.15.28:0/349073036 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb9b006ee40 0x7fb9b01536f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:38.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.843+0000 7fb9ad400640 1 -- 172.21.15.28:0/349073036 >> v2:172.21.15.81:3300/0 conn(0x7fb9b0150450 msgr2=0x7fb9b00a5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:38.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.843+0000 7fb9ad400640 1 --2- 172.21.15.28:0/349073036 >> v2:172.21.15.81:3300/0 conn(0x7fb9b0150450 0x7fb9b00a5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:38.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.843+0000 7fb9ad400640 1 -- 172.21.15.28:0/349073036 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9b0153c30 con 0x7fb9b0156af0 2024-02-21T23:10:38.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.843+0000 7fb9ad400640 1 -- 172.21.15.28:0/349073036 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fb9a80019f0 con 0x7fb9b0156af0 2024-02-21T23:10:38.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.843+0000 7fb9ad400640 1 -- 172.21.15.28:0/349073036 >> v1:172.21.15.28:6789/0 conn(0x7fb9b0156af0 legacy=0x7fb9b006dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:38.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.843+0000 7fb9ad400640 1 --2- 172.21.15.28:0/349073036 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9940039b0 0x7fb994003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:38.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.843+0000 7fb9ad400640 1 --2- 172.21.15.28:0/349073036 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb994004800 0x7fb994004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:38.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.843+0000 7fb9ad400640 1 --2- 172.21.15.28:0/349073036 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb9b006ee40 0x7fb9940051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:38.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.843+0000 7fb9ad400640 1 -- 172.21.15.28:0/349073036 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9b0156660 con 0x7fb994004800 2024-02-21T23:10:38.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.843+0000 7fb9ad400640 1 -- 172.21.15.28:0/349073036 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb99400b9c0 con 0x7fb9940039b0 2024-02-21T23:10:38.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.843+0000 7fb9ad400640 1 -- 172.21.15.28:0/349073036 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb99400bb40 con 0x7fb9b006ee40 2024-02-21T23:10:38.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.843+0000 7fb9a5400640 1 --2- 172.21.15.28:0/349073036 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb994004800 0x7fb994004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:38.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.844+0000 7fb9ade00640 1 --2- 172.21.15.28:0/349073036 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9940039b0 0x7fb994003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:38.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.844+0000 7fb9a5400640 1 -- 172.21.15.28:0/349073036 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb9b006ee40 msgr2=0x7fb9940051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:38.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.844+0000 7fb9a5400640 1 --2- 172.21.15.28:0/349073036 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb9b006ee40 0x7fb9940051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:38.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.844+0000 7fb9a5400640 1 -- 172.21.15.28:0/349073036 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9940039b0 msgr2=0x7fb994003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:38.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.844+0000 7fb9a5400640 1 --2- 172.21.15.28:0/349073036 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9940039b0 0x7fb994003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:38.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.844+0000 7fb9a5400640 1 -- 172.21.15.28:0/349073036 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb994003520 con 0x7fb994004800 2024-02-21T23:10:38.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.845+0000 7fb9a5400640 1 --2- 172.21.15.28:0/349073036 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb994004800 0x7fb994004c80 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7fb998009ac0 tx=0x7fb994005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21ef5b8f4b0b4a71 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:38.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.845+0000 7fb9ad400640 1 -- 172.21.15.28:0/349073036 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb998039070 con 0x7fb994004800 2024-02-21T23:10:38.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.845+0000 7fb9ad400640 1 -- 172.21.15.28:0/349073036 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb99802c6d0 con 0x7fb994004800 2024-02-21T23:10:38.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.845+0000 7fb9ad400640 1 -- 172.21.15.28:0/349073036 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb998009690 con 0x7fb994004800 2024-02-21T23:10:38.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.846+0000 7fb9af200640 1 -- 172.21.15.28:0/349073036 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb994004800 msgr2=0x7fb994004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:38.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.846+0000 7fb9af200640 1 --2- 172.21.15.28:0/349073036 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb994004800 0x7fb994004c80 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7fb998009ac0 tx=0x7fb994005830 comp rx=0 tx=0).stop 2024-02-21T23:10:38.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.846+0000 7fb9ae800640 1 -- 172.21.15.28:0/349073036 reap_dead start 2024-02-21T23:10:38.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.846+0000 7fb9af200640 1 -- 172.21.15.28:0/349073036 shutdown_connections 2024-02-21T23:10:38.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.846+0000 7fb9af200640 1 -- 172.21.15.28:0/349073036 >> 172.21.15.28:0/349073036 conn(0x7fb9b006b630 msgr2=0x7fb9b00c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:38.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.846+0000 7fb9af200640 1 -- 172.21.15.28:0/349073036 shutdown_connections 2024-02-21T23:10:38.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.846+0000 7fb9af200640 1 -- 172.21.15.28:0/349073036 wait complete. 2024-02-21T23:10:38.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.848+0000 7fb9af200640 1 Processor -- start 2024-02-21T23:10:38.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.848+0000 7fb9af200640 1 -- start start 2024-02-21T23:10:38.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.849+0000 7fb9af200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb994004800 0x7fb9b014f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:38.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.849+0000 7fb9af200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9b0152720 0x7fb9b0147650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:38.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.849+0000 7fb9af200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9b0155af0 0x7fb9b0147b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:38.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.849+0000 7fb9af200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9b00c2cb0 con 0x7fb9b0155af0 2024-02-21T23:10:38.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.849+0000 7fb9af200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9b00c29d0 con 0x7fb9b0152720 2024-02-21T23:10:38.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.849+0000 7fb9af200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9b006e700 con 0x7fb994004800 2024-02-21T23:10:38.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.849+0000 7fb9ae800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9b0155af0 0x7fb9b0147b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:38.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.849+0000 7fb9ae800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9b0155af0 0x7fb9b0147b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53482/0 (socket says 172.21.15.28:53482) 2024-02-21T23:10:38.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.849+0000 7fb9ae800640 1 -- 172.21.15.28:0/650143166 learned_addr learned my addr 172.21.15.28:0/650143166 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:38.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.850+0000 7fb9ade00640 1 --2- 172.21.15.28:0/650143166 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb994004800 0x7fb9b014f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:38.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.850+0000 7fb9a5400640 1 --2- 172.21.15.28:0/650143166 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9b0152720 0x7fb9b0147650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:38.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.850+0000 7fb9ae800640 1 -- 172.21.15.28:0/650143166 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb994004800 msgr2=0x7fb9b014f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:38.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.850+0000 7fb9ae800640 1 --2- 172.21.15.28:0/650143166 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb994004800 0x7fb9b014f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:38.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.850+0000 7fb9ae800640 1 -- 172.21.15.28:0/650143166 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9b0152720 msgr2=0x7fb9b0147650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:38.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.850+0000 7fb9ae800640 1 --2- 172.21.15.28:0/650143166 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9b0152720 0x7fb9b0147650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:38.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.850+0000 7fb9ae800640 1 -- 172.21.15.28:0/650143166 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9b00ac370 con 0x7fb9b0155af0 2024-02-21T23:10:38.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.850+0000 7fb9a5400640 1 --2- 172.21.15.28:0/650143166 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9b0152720 0x7fb9b0147650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:10:38.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.850+0000 7fb9ae800640 1 --2- 172.21.15.28:0/650143166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9b0155af0 0x7fb9b0147b90 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7fb9b00c64b0 tx=0x7fb9a800ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:38.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.850+0000 7fb9a6a00640 1 -- 172.21.15.28:0/650143166 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9a800a640 con 0x7fb9b0155af0 2024-02-21T23:10:38.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.851+0000 7fb9a6a00640 1 -- 172.21.15.28:0/650143166 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9a800a7e0 con 0x7fb9b0155af0 2024-02-21T23:10:38.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.851+0000 7fb9a6a00640 1 -- 172.21.15.28:0/650143166 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9a800abd0 con 0x7fb9b0155af0 2024-02-21T23:10:38.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.851+0000 7fb9af200640 1 -- 172.21.15.28:0/650143166 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9b00c5960 con 0x7fb9b0155af0 2024-02-21T23:10:38.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.851+0000 7fb9af200640 1 -- 172.21.15.28:0/650143166 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9b00c5310 con 0x7fb9b0155af0 2024-02-21T23:10:38.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.852+0000 7fb9af200640 1 -- 172.21.15.28:0/650143166 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9b004f290 con 0x7fb9b0155af0 2024-02-21T23:10:38.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.854+0000 7fb9a6a00640 1 -- 172.21.15.28:0/650143166 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb9a8007110 con 0x7fb9b0155af0 2024-02-21T23:10:38.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.855+0000 7fb9a6a00640 1 --2- 172.21.15.28:0/650143166 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb98c06b4d0 0x7fb98c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:38.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.855+0000 7fb9a6a00640 1 -- 172.21.15.28:0/650143166 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 6536+0+0 (secure 0 0 0) 0x7fb9a8089fb0 con 0x7fb9b0155af0 2024-02-21T23:10:38.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.856+0000 7fb9ade00640 1 --2- 172.21.15.28:0/650143166 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb98c06b4d0 0x7fb98c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:38.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.857+0000 7fb9ade00640 1 --2- 172.21.15.28:0/650143166 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb98c06b4d0 0x7fb98c06d990 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fb9a0009a90 tx=0x7fb9a0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:38.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:38.860+0000 7fb9a6a00640 1 -- 172.21.15.28:0/650143166 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb9a801b020 con 0x7fb9b0155af0 2024-02-21T23:10:39.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.062+0000 7fb9af200640 1 -- 172.21.15.28:0/650143166 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb9b00c1f10 con 0x7fb9b0155af0 2024-02-21T23:10:39.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.065+0000 7fb9a6a00640 1 -- 172.21.15.28:0/650143166 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v163) v1 ==== 74+0+17551 (secure 0 0 0) 0x7fb9a8058ae0 con 0x7fb9b0155af0 2024-02-21T23:10:39.054 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:10:39.054 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":163,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:10:34.681492+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs2_metadata","create_time":"2024-02-21T23:10:04.095451+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"154","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs2_data","create_time":"2024-02-21T23:10:04.836982+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"155","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs2"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"26.29","mappings":[{"from":1,"to":0}]},{"pgid":"26.36","mappings":[{"from":3,"to":0}]},{"pgid":"26.3f","mappings":[{"from":1,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.81:6835/1320495906":"2024-02-22T23:10:32.538190+0000","172.21.15.81:6834/1320495906":"2024-02-22T23:10:32.538190+0000","172.21.15.81:6837/521625945":"2024-02-22T23:10:31.712025+0000","172.21.15.28:6833/2975378719":"2024-02-22T23:10:21.581948+0000","172.21.15.28:6832/2975378719":"2024-02-22T23:10:21.581948+0000","172.21.15.81:6836/521625945":"2024-02-22T23:10:31.712025+0000","172.21.15.28:6835/2928639298":"2024-02-22T23:10:19.550812+0000","172.21.15.28:6834/2928639298":"2024-02-22T23:10:19.550812+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:10:39.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.068+0000 7fb9af200640 1 -- 172.21.15.28:0/650143166 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb98c06b4d0 msgr2=0x7fb98c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:39.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.068+0000 7fb9af200640 1 --2- 172.21.15.28:0/650143166 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb98c06b4d0 0x7fb98c06d990 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fb9a0009a90 tx=0x7fb9a0008040 comp rx=0 tx=0).stop 2024-02-21T23:10:39.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.068+0000 7fb9af200640 1 -- 172.21.15.28:0/650143166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9b0155af0 msgr2=0x7fb9b0147b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:39.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.068+0000 7fb9af200640 1 --2- 172.21.15.28:0/650143166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9b0155af0 0x7fb9b0147b90 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7fb9b00c64b0 tx=0x7fb9a800ccd0 comp rx=0 tx=0).stop 2024-02-21T23:10:39.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.068+0000 7fb9af200640 1 -- 172.21.15.28:0/650143166 shutdown_connections 2024-02-21T23:10:39.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.068+0000 7fb9af200640 1 --2- 172.21.15.28:0/650143166 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb98c06b4d0 0x7fb98c06d990 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:39.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.068+0000 7fb9af200640 1 --2- 172.21.15.28:0/650143166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9b0155af0 0x7fb9b0147b90 unknown :-1 s=CLOSED pgs=1852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:39.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.068+0000 7fb9af200640 1 --2- 172.21.15.28:0/650143166 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9b0152720 0x7fb9b0147650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:39.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.068+0000 7fb9af200640 1 --2- 172.21.15.28:0/650143166 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb994004800 0x7fb9b014f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:39.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.068+0000 7fb9af200640 1 -- 172.21.15.28:0/650143166 >> 172.21.15.28:0/650143166 conn(0x7fb9b006b630 msgr2=0x7fb9b006dff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:39.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.068+0000 7fb9af200640 1 -- 172.21.15.28:0/650143166 shutdown_connections 2024-02-21T23:10:39.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.068+0000 7fb9af200640 1 -- 172.21.15.28:0/650143166 wait complete. 2024-02-21T23:10:39.079 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs2 2024-02-21T23:10:39.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.304+0000 7f58c5c00640 1 Processor -- start 2024-02-21T23:10:39.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.305+0000 7f58c5c00640 1 -- start start 2024-02-21T23:10:39.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.306+0000 7f58c5c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f58c00c5040 0x7f58c0155a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:39.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.306+0000 7f58c5c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f58c00c85c0 0x7f58c00c70d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:39.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.306+0000 7f58c5c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58c006df50 con 0x7f58c01524c0 2024-02-21T23:10:39.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.306+0000 7f58c5c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58c00c5520 con 0x7f58c00c5040 2024-02-21T23:10:39.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.306+0000 7f58c5c00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58c00c5800 con 0x7f58c00c85c0 2024-02-21T23:10:39.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.307+0000 7f58bfe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f58c01524c0 0x7f58c006ce70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:47858/0 (socket says 172.21.15.28:47858) 2024-02-21T23:10:39.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.307+0000 7f58bfe00640 1 -- 172.21.15.28:0/1368436766 learned_addr learned my addr 172.21.15.28:0/1368436766 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:10:39.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.307+0000 7f58bf400640 1 --2- 172.21.15.28:0/1368436766 >> v2:172.21.15.28:3300/0 conn(0x7f58c00c5040 0x7f58c0155a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:39.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.307+0000 7f58c5200640 1 --2- 172.21.15.28:0/1368436766 >> v2:172.21.15.81:3300/0 conn(0x7f58c00c85c0 0x7f58c00c70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:39.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.307+0000 7f58bf400640 1 -- 172.21.15.28:0/1368436766 >> v2:172.21.15.81:3300/0 conn(0x7f58c00c85c0 msgr2=0x7f58c00c70d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:39.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.307+0000 7f58bea00640 1 -- 172.21.15.28:0/1368436766 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3180413012 0 0) 0x7f58c006df50 con 0x7f58c01524c0 2024-02-21T23:10:39.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.307+0000 7f58bf400640 1 --2- 172.21.15.28:0/1368436766 >> v2:172.21.15.81:3300/0 conn(0x7f58c00c85c0 0x7f58c00c70d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:39.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.307+0000 7f58bf400640 1 -- 172.21.15.28:0/1368436766 >> v1:172.21.15.28:6789/0 conn(0x7f58c01524c0 legacy=0x7f58c006ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:39.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.307+0000 7f58bf400640 1 -- 172.21.15.28:0/1368436766 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58c006c670 con 0x7f58c00c5040 2024-02-21T23:10:39.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.307+0000 7f58c5200640 1 --2- 172.21.15.28:0/1368436766 >> v2:172.21.15.81:3300/0 conn(0x7f58c00c85c0 0x7f58c00c70d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:10:39.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.309+0000 7f58bf400640 1 --2- 172.21.15.28:0/1368436766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58c00c5040 0x7f58c0155a10 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7f58c00c4af0 tx=0x7f58ac02b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5693225a5156ccb server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:39.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.309+0000 7f58bea00640 1 -- 172.21.15.28:0/1368436766 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58ac038070 con 0x7f58c00c5040 2024-02-21T23:10:39.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.309+0000 7f58bea00640 1 -- 172.21.15.28:0/1368436766 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58ac031710 con 0x7f58c00c5040 2024-02-21T23:10:39.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.309+0000 7f58bea00640 1 -- 172.21.15.28:0/1368436766 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58ac031ae0 con 0x7f58c00c5040 2024-02-21T23:10:39.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.309+0000 7f58c5c00640 1 -- 172.21.15.28:0/1368436766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58c00c5040 msgr2=0x7f58c0155a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:39.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.309+0000 7f58c5c00640 1 --2- 172.21.15.28:0/1368436766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58c00c5040 0x7f58c0155a10 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7f58c00c4af0 tx=0x7f58ac02b7b0 comp rx=0 tx=0).stop 2024-02-21T23:10:39.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.310+0000 7f58c5c00640 1 -- 172.21.15.28:0/1368436766 shutdown_connections 2024-02-21T23:10:39.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.310+0000 7f58c5c00640 1 --2- 172.21.15.28:0/1368436766 >> v2:172.21.15.81:3300/0 conn(0x7f58c00c85c0 0x7f58c00c70d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:39.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.310+0000 7f58c5c00640 1 --2- 172.21.15.28:0/1368436766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58c00c5040 0x7f58c0155a10 unknown :-1 s=CLOSED pgs=1854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:39.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.310+0000 7f58c5c00640 1 -- 172.21.15.28:0/1368436766 >> 172.21.15.28:0/1368436766 conn(0x7f58c00ac2b0 msgr2=0x7f58c00ac6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:39.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.310+0000 7f58c5c00640 1 -- 172.21.15.28:0/1368436766 shutdown_connections 2024-02-21T23:10:39.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.310+0000 7f58c5c00640 1 -- 172.21.15.28:0/1368436766 wait complete. 2024-02-21T23:10:39.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.311+0000 7f58c5c00640 1 Processor -- start 2024-02-21T23:10:39.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.311+0000 7f58c5c00640 1 -- start start 2024-02-21T23:10:39.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.312+0000 7f58c5c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f58c00c5040 0x7f58c00c3f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:39.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.312+0000 7f58c5c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58c00c85c0 0x7f58c0153d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:39.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.312+0000 7f58c5c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58c01524c0 0x7f58c01ebab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:39.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.312+0000 7f58c5c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58c00c8a70 con 0x7f58c00c85c0 2024-02-21T23:10:39.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.312+0000 7f58c5c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58c01528a0 con 0x7f58c01524c0 2024-02-21T23:10:39.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.312+0000 7f58c5c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58c00c7610 con 0x7f58c00c5040 2024-02-21T23:10:39.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.313+0000 7f58bf400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58c00c85c0 0x7f58c0153d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:39.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.313+0000 7f58bf400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58c00c85c0 0x7f58c0153d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53500/0 (socket says 172.21.15.28:53500) 2024-02-21T23:10:39.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.313+0000 7f58bf400640 1 -- 172.21.15.28:0/306831301 learned_addr learned my addr 172.21.15.28:0/306831301 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:39.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.313+0000 7f58bfe00640 1 --2- 172.21.15.28:0/306831301 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f58c00c5040 0x7f58c00c3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:39.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.313+0000 7f58c5200640 1 --2- 172.21.15.28:0/306831301 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58c01524c0 0x7f58c01ebab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:39.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.313+0000 7f58bf400640 1 -- 172.21.15.28:0/306831301 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f58c00c5040 msgr2=0x7f58c00c3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:39.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.313+0000 7f58bf400640 1 --2- 172.21.15.28:0/306831301 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f58c00c5040 0x7f58c00c3f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:39.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.313+0000 7f58bf400640 1 -- 172.21.15.28:0/306831301 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58c01524c0 msgr2=0x7f58c01ebab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:39.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.313+0000 7f58bf400640 1 --2- 172.21.15.28:0/306831301 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58c01524c0 0x7f58c01ebab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:39.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.313+0000 7f58bf400640 1 -- 172.21.15.28:0/306831301 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58c00c0970 con 0x7f58c00c85c0 2024-02-21T23:10:39.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.313+0000 7f58bfe00640 1 --2- 172.21.15.28:0/306831301 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f58c00c5040 0x7f58c00c3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:10:39.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.313+0000 7f58c5200640 1 --2- 172.21.15.28:0/306831301 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58c01524c0 0x7f58c01ebab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:10:39.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.314+0000 7f58bf400640 1 --2- 172.21.15.28:0/306831301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58c00c85c0 0x7f58c0153d50 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7f58ac009890 tx=0x7f58ac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:39.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.314+0000 7f58b7e00640 1 -- 172.21.15.28:0/306831301 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58ac042020 con 0x7f58c00c85c0 2024-02-21T23:10:39.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.314+0000 7f58b7e00640 1 -- 172.21.15.28:0/306831301 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58ac033a40 con 0x7f58c00c85c0 2024-02-21T23:10:39.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.314+0000 7f58c5c00640 1 -- 172.21.15.28:0/306831301 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58c00c0c10 con 0x7f58c00c85c0 2024-02-21T23:10:39.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.314+0000 7f58b7e00640 1 -- 172.21.15.28:0/306831301 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58ac038040 con 0x7f58c00c85c0 2024-02-21T23:10:39.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.314+0000 7f58c5c00640 1 -- 172.21.15.28:0/306831301 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58c0154290 con 0x7f58c00c85c0 2024-02-21T23:10:39.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.316+0000 7f58c5c00640 1 -- 172.21.15.28:0/306831301 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58c014f480 con 0x7f58c00c85c0 2024-02-21T23:10:39.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.316+0000 7f58b7e00640 1 -- 172.21.15.28:0/306831301 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f58ac045020 con 0x7f58c00c85c0 2024-02-21T23:10:39.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.318+0000 7f58b7e00640 1 --2- 172.21.15.28:0/306831301 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f58a006b4d0 0x7f58a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:39.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.318+0000 7f58b7e00640 1 -- 172.21.15.28:0/306831301 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 6536+0+0 (secure 0 0 0) 0x7f58ac0ac260 con 0x7f58c00c85c0 2024-02-21T23:10:39.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.318+0000 7f58bfe00640 1 --2- 172.21.15.28:0/306831301 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f58a006b4d0 0x7f58a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:39.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.319+0000 7f58bfe00640 1 --2- 172.21.15.28:0/306831301 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f58a006b4d0 0x7f58a006d990 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f58b0000f60 tx=0x7f58b0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:39.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.323+0000 7f58b7e00640 1 -- 172.21.15.28:0/306831301 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f58ac04c030 con 0x7f58c00c85c0 2024-02-21T23:10:39.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.575+0000 7f58c5c00640 1 -- 172.21.15.28:0/306831301 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs2"} v 0) v1 -- 0x7f58c004f290 con 0x7f58c00c85c0 2024-02-21T23:10:39.565 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-02-21T23:10:39.576+0000 7f5ddc400640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-21T23:10:39.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.579+0000 7f58b7e00640 1 -- 172.21.15.28:0/306831301 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs2"}]=0 cephfs2 marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v177) v1 ==== 163+0+0 (secure 0 0 0) 0x7f58ac07ad90 con 0x7f58c00c85c0 2024-02-21T23:10:39.567 INFO:teuthology.orchestra.run.smithi028.stderr:cephfs2 marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-02-21T23:10:39.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.584+0000 7f58c5c00640 1 -- 172.21.15.28:0/306831301 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f58a006b4d0 msgr2=0x7f58a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:39.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.584+0000 7f58c5c00640 1 --2- 172.21.15.28:0/306831301 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f58a006b4d0 0x7f58a006d990 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f58b0000f60 tx=0x7f58b0009040 comp rx=0 tx=0).stop 2024-02-21T23:10:39.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.584+0000 7f58c5c00640 1 -- 172.21.15.28:0/306831301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58c00c85c0 msgr2=0x7f58c0153d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:39.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.584+0000 7f58c5c00640 1 --2- 172.21.15.28:0/306831301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58c00c85c0 0x7f58c0153d50 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7f58ac009890 tx=0x7f58ac0046c0 comp rx=0 tx=0).stop 2024-02-21T23:10:39.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.584+0000 7f58c5c00640 1 -- 172.21.15.28:0/306831301 shutdown_connections 2024-02-21T23:10:39.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.584+0000 7f58c5c00640 1 --2- 172.21.15.28:0/306831301 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f58a006b4d0 0x7f58a006d990 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:39.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.584+0000 7f58c5c00640 1 --2- 172.21.15.28:0/306831301 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58c01524c0 0x7f58c01ebab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:39.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.584+0000 7f58c5c00640 1 --2- 172.21.15.28:0/306831301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58c00c85c0 0x7f58c0153d50 unknown :-1 s=CLOSED pgs=1855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:39.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.584+0000 7f58c5c00640 1 --2- 172.21.15.28:0/306831301 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f58c00c5040 0x7f58c00c3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:39.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.584+0000 7f58c5c00640 1 -- 172.21.15.28:0/306831301 >> 172.21.15.28:0/306831301 conn(0x7f58c00ac2b0 msgr2=0x7f58c00c7cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:39.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.584+0000 7f58c5c00640 1 -- 172.21.15.28:0/306831301 shutdown_connections 2024-02-21T23:10:39.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.584+0000 7f58c5c00640 1 -- 172.21.15.28:0/306831301 wait complete. 2024-02-21T23:10:39.579 INFO:tasks.ceph.mds.c.smithi028.stderr: -65> 2024-02-21T23:10:37.605+0000 7fe517e00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-21T23:10:39.595 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs2 --yes-i-really-mean-it 2024-02-21T23:10:39.627 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-02-21T23:10:39.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.801+0000 7f4560400640 1 Processor -- start 2024-02-21T23:10:39.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.801+0000 7f4560400640 1 -- start start 2024-02-21T23:10:39.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.803+0000 7f4560400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f45580c5f50 0x7f455814fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:39.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.803+0000 7f4560400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4558150530 0x7f4558156500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:39.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.803+0000 7f4560400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45581509e0 con 0x7f455806b0e0 2024-02-21T23:10:39.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.803+0000 7f4560400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4558153bc0 con 0x7f45580c5f50 2024-02-21T23:10:39.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.803+0000 7f4560400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4558152fb0 con 0x7f4558150530 2024-02-21T23:10:39.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.803+0000 7f455e400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4558150530 0x7f4558156500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:39.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.804+0000 7f455d000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f45580c5f50 0x7f455814fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:39.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.804+0000 7f455e400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4558150530 0x7f4558156500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3301/0 says I am v2:172.21.15.28:39304/0 (socket says 172.21.15.28:39304) 2024-02-21T23:10:39.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.804+0000 7f455d000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f45580c5f50 0x7f455814fff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.28:54314/0 (socket says 172.21.15.28:54314) 2024-02-21T23:10:39.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.804+0000 7f455e400640 1 -- 172.21.15.28:0/286253785 learned_addr learned my addr 172.21.15.28:0/286253785 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:39.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.804+0000 7f455d000640 1 -- 172.21.15.28:0/286253785 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4558150530 msgr2=0x7f4558156500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:39.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.804+0000 7f455d000640 1 --2- 172.21.15.28:0/286253785 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4558150530 0x7f4558156500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:39.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.804+0000 7f455d000640 1 -- 172.21.15.28:0/286253785 >> v1:172.21.15.81:6789/0 conn(0x7f455806b0e0 legacy=0x7f45580c5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:39.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.804+0000 7f455d000640 1 -- 172.21.15.28:0/286253785 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4558153290 con 0x7f45580c5f50 2024-02-21T23:10:39.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.806+0000 7f455d000640 1 --2- 172.21.15.28:0/286253785 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f45580c5f50 0x7f455814fff0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f45440097a0 tx=0x7f454402b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b5e737b1e658f73b server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:39.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.807+0000 7f4557e00640 1 -- 172.21.15.28:0/286253785 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4544031780 con 0x7f45580c5f50 2024-02-21T23:10:39.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.807+0000 7f4557e00640 1 -- 172.21.15.28:0/286253785 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4544037070 con 0x7f45580c5f50 2024-02-21T23:10:39.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.807+0000 7f4557e00640 1 -- 172.21.15.28:0/286253785 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4544031b40 con 0x7f45580c5f50 2024-02-21T23:10:39.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.808+0000 7f4560400640 1 -- 172.21.15.28:0/286253785 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f45580c5f50 msgr2=0x7f455814fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:39.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.808+0000 7f4560400640 1 --2- 172.21.15.28:0/286253785 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f45580c5f50 0x7f455814fff0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f45440097a0 tx=0x7f454402b820 comp rx=0 tx=0).stop 2024-02-21T23:10:39.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.808+0000 7f4560400640 1 -- 172.21.15.28:0/286253785 shutdown_connections 2024-02-21T23:10:39.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.808+0000 7f4560400640 1 --2- 172.21.15.28:0/286253785 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4558150530 0x7f4558156500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:39.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.808+0000 7f4560400640 1 --2- 172.21.15.28:0/286253785 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f45580c5f50 0x7f455814fff0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:39.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.808+0000 7f4560400640 1 -- 172.21.15.28:0/286253785 >> 172.21.15.28:0/286253785 conn(0x7f45580a85a0 msgr2=0x7f45580a8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:39.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.808+0000 7f4560400640 1 -- 172.21.15.28:0/286253785 shutdown_connections 2024-02-21T23:10:39.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.808+0000 7f4560400640 1 -- 172.21.15.28:0/286253785 wait complete. 2024-02-21T23:10:39.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.810+0000 7f4560400640 1 Processor -- start 2024-02-21T23:10:39.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.810+0000 7f4560400640 1 -- start start 2024-02-21T23:10:39.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.811+0000 7f4560400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f455806b0e0 0x7f45581eb3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:39.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.811+0000 7f4560400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f45580c5f50 0x7f455814d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:39.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.811+0000 7f4560400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4558150530 0x7f45580bfe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:39.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.811+0000 7f4560400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f455806b4c0 con 0x7f4558150530 2024-02-21T23:10:39.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.811+0000 7f4560400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4558153850 con 0x7f455806b0e0 2024-02-21T23:10:39.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.811+0000 7f4560400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f455806b640 con 0x7f45580c5f50 2024-02-21T23:10:39.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.811+0000 7f455e400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4558150530 0x7f45580bfe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:39.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.811+0000 7f455e400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4558150530 0x7f45580bfe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53528/0 (socket says 172.21.15.28:53528) 2024-02-21T23:10:39.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.811+0000 7f455e400640 1 -- 172.21.15.28:0/4103499463 learned_addr learned my addr 172.21.15.28:0/4103499463 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:39.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.811+0000 7f455da00640 1 --2- 172.21.15.28:0/4103499463 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f455806b0e0 0x7f45581eb3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:39.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.811+0000 7f455d000640 1 --2- 172.21.15.28:0/4103499463 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f45580c5f50 0x7f455814d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:39.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.812+0000 7f455e400640 1 -- 172.21.15.28:0/4103499463 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f45580c5f50 msgr2=0x7f455814d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:39.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.812+0000 7f455e400640 1 --2- 172.21.15.28:0/4103499463 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f45580c5f50 0x7f455814d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:39.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.812+0000 7f455e400640 1 -- 172.21.15.28:0/4103499463 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f455806b0e0 msgr2=0x7f45581eb3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:39.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.812+0000 7f455e400640 1 --2- 172.21.15.28:0/4103499463 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f455806b0e0 0x7f45581eb3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:39.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.812+0000 7f455e400640 1 -- 172.21.15.28:0/4103499463 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f455806df50 con 0x7f4558150530 2024-02-21T23:10:39.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.812+0000 7f455da00640 1 --2- 172.21.15.28:0/4103499463 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f455806b0e0 0x7f45581eb3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:10:39.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.812+0000 7f455e400640 1 --2- 172.21.15.28:0/4103499463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4558150530 0x7f45580bfe90 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7f455000e870 tx=0x7f455000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:39.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.812+0000 7f4555600640 1 -- 172.21.15.28:0/4103499463 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4550017070 con 0x7f4558150530 2024-02-21T23:10:39.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.812+0000 7f4555600640 1 -- 172.21.15.28:0/4103499463 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f455000ee60 con 0x7f4558150530 2024-02-21T23:10:39.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.812+0000 7f4555600640 1 -- 172.21.15.28:0/4103499463 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f455000cb60 con 0x7f4558150530 2024-02-21T23:10:39.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.812+0000 7f4560400640 1 -- 172.21.15.28:0/4103499463 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f455806e230 con 0x7f4558150530 2024-02-21T23:10:39.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.813+0000 7f4560400640 1 -- 172.21.15.28:0/4103499463 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45581edfa0 con 0x7f4558150530 2024-02-21T23:10:39.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.814+0000 7f4560400640 1 -- 172.21.15.28:0/4103499463 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f455814cef0 con 0x7f4558150530 2024-02-21T23:10:39.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.815+0000 7f4555600640 1 -- 172.21.15.28:0/4103499463 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f455001a070 con 0x7f4558150530 2024-02-21T23:10:39.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.816+0000 7f4555600640 1 --2- 172.21.15.28:0/4103499463 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f453806b4d0 0x7f453806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:39.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.816+0000 7f4555600640 1 -- 172.21.15.28:0/4103499463 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 6622+0+0 (secure 0 0 0) 0x7f455008a7f0 con 0x7f4558150530 2024-02-21T23:10:39.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.816+0000 7f455da00640 1 --2- 172.21.15.28:0/4103499463 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f453806b4d0 0x7f453806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:39.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.817+0000 7f455da00640 1 --2- 172.21.15.28:0/4103499463 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f453806b4d0 0x7f453806d990 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f454c001e00 tx=0x7f454c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:39.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:39.820+0000 7f4555600640 1 -- 172.21.15.28:0/4103499463 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f455814cef0 con 0x7f4558150530 2024-02-21T23:10:40.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.060+0000 7f4560400640 1 -- 172.21.15.28:0/4103499463 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs2", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f4558067f30 con 0x7f4558150530 2024-02-21T23:10:40.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.582+0000 7f4555600640 1 -- 172.21.15.28:0/4103499463 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs2", "yes_i_really_mean_it": true}]=0 v178) v1 ==== 105+0+0 (secure 0 0 0) 0x7f45500592d0 con 0x7f4558150530 2024-02-21T23:10:40.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.586+0000 7f4560400640 1 -- 172.21.15.28:0/4103499463 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f453806b4d0 msgr2=0x7f453806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:40.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.586+0000 7f4560400640 1 --2- 172.21.15.28:0/4103499463 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f453806b4d0 0x7f453806d990 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f454c001e00 tx=0x7f454c009040 comp rx=0 tx=0).stop 2024-02-21T23:10:40.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.586+0000 7f4560400640 1 -- 172.21.15.28:0/4103499463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4558150530 msgr2=0x7f45580bfe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:40.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.586+0000 7f4560400640 1 --2- 172.21.15.28:0/4103499463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4558150530 0x7f45580bfe90 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7f455000e870 tx=0x7f455000ed30 comp rx=0 tx=0).stop 2024-02-21T23:10:40.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.586+0000 7f4560400640 1 -- 172.21.15.28:0/4103499463 shutdown_connections 2024-02-21T23:10:40.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.586+0000 7f4560400640 1 --2- 172.21.15.28:0/4103499463 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f453806b4d0 0x7f453806d990 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:40.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.586+0000 7f4560400640 1 --2- 172.21.15.28:0/4103499463 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f45580c5f50 0x7f455814d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:40.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.586+0000 7f4560400640 1 --2- 172.21.15.28:0/4103499463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4558150530 0x7f45580bfe90 unknown :-1 s=CLOSED pgs=1859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:40.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.586+0000 7f4560400640 1 --2- 172.21.15.28:0/4103499463 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f455806b0e0 0x7f45581eb3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:40.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.586+0000 7f4560400640 1 -- 172.21.15.28:0/4103499463 >> 172.21.15.28:0/4103499463 conn(0x7f45580a85a0 msgr2=0x7f455814acf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:40.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.586+0000 7f4560400640 1 -- 172.21.15.28:0/4103499463 shutdown_connections 2024-02-21T23:10:40.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.586+0000 7f4560400640 1 -- 172.21.15.28:0/4103499463 wait complete. 2024-02-21T23:10:40.595 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs2_metadata cephfs2_metadata --yes-i-really-really-mean-it 2024-02-21T23:10:40.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.875+0000 7ff17fe00640 1 Processor -- start 2024-02-21T23:10:40.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.875+0000 7ff17fe00640 1 -- start start 2024-02-21T23:10:40.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.877+0000 7ff17fe00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff1800c06b0 0x7ff18006b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:40.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.877+0000 7ff17fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff180071ce0 0x7ff180072150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:40.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.877+0000 7ff17fe00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1800c0d00 con 0x7ff18006ae70 2024-02-21T23:10:40.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.877+0000 7ff17fe00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff18006ef40 con 0x7ff180071ce0 2024-02-21T23:10:40.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.877+0000 7ff17fe00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff18006e180 con 0x7ff1800c06b0 2024-02-21T23:10:40.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.877+0000 7ff17f400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff180071ce0 0x7ff180072150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:40.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.877+0000 7ff17f400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff180071ce0 0x7ff180072150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53534/0 (socket says 172.21.15.28:53534) 2024-02-21T23:10:40.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.877+0000 7ff17f400640 1 -- 172.21.15.28:0/2704753219 learned_addr learned my addr 172.21.15.28:0/2704753219 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:40.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.878+0000 7ff17e000640 1 --2- 172.21.15.28:0/2704753219 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff1800c06b0 0x7ff18006b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:40.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.878+0000 7ff17f400640 1 -- 172.21.15.28:0/2704753219 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff1800c06b0 msgr2=0x7ff18006b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:40.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.878+0000 7ff17f400640 1 --2- 172.21.15.28:0/2704753219 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff1800c06b0 0x7ff18006b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:40.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.878+0000 7ff17f400640 1 -- 172.21.15.28:0/2704753219 >> v1:172.21.15.81:6789/0 conn(0x7ff18006ae70 legacy=0x7ff180155960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:40.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.878+0000 7ff17f400640 1 -- 172.21.15.28:0/2704753219 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff18006e460 con 0x7ff180071ce0 2024-02-21T23:10:40.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.878+0000 7ff17e000640 1 --2- 172.21.15.28:0/2704753219 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff1800c06b0 0x7ff18006b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:10:40.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.879+0000 7ff17f400640 1 --2- 172.21.15.28:0/2704753219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff180071ce0 0x7ff180072150 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7ff178009e50 tx=0x7ff17802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c818091da3a018c3 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:40.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.879+0000 7ff17d600640 1 -- 172.21.15.28:0/2704753219 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff17802ca80 con 0x7ff180071ce0 2024-02-21T23:10:40.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.879+0000 7ff17d600640 1 -- 172.21.15.28:0/2704753219 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff17802cc20 con 0x7ff180071ce0 2024-02-21T23:10:40.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.879+0000 7ff17d600640 1 -- 172.21.15.28:0/2704753219 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1780343f0 con 0x7ff180071ce0 2024-02-21T23:10:40.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.880+0000 7ff17fe00640 1 -- 172.21.15.28:0/2704753219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff180071ce0 msgr2=0x7ff180072150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:40.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.880+0000 7ff17fe00640 1 --2- 172.21.15.28:0/2704753219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff180071ce0 0x7ff180072150 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7ff178009e50 tx=0x7ff17802bee0 comp rx=0 tx=0).stop 2024-02-21T23:10:40.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.880+0000 7ff17fe00640 1 -- 172.21.15.28:0/2704753219 shutdown_connections 2024-02-21T23:10:40.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.880+0000 7ff17fe00640 1 --2- 172.21.15.28:0/2704753219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff180071ce0 0x7ff180072150 unknown :-1 s=CLOSED pgs=1860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:40.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.880+0000 7ff17fe00640 1 --2- 172.21.15.28:0/2704753219 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff1800c06b0 0x7ff18006b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:40.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.880+0000 7ff17fe00640 1 -- 172.21.15.28:0/2704753219 >> 172.21.15.28:0/2704753219 conn(0x7ff1800c6610 msgr2=0x7ff1800c69f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:40.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.880+0000 7ff17fe00640 1 -- 172.21.15.28:0/2704753219 shutdown_connections 2024-02-21T23:10:40.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.880+0000 7ff17fe00640 1 -- 172.21.15.28:0/2704753219 wait complete. 2024-02-21T23:10:40.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.882+0000 7ff17fe00640 1 Processor -- start 2024-02-21T23:10:40.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.882+0000 7ff17fe00640 1 -- start start 2024-02-21T23:10:40.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.883+0000 7ff17fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff18006ae70 0x7ff180153a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:40.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.883+0000 7ff17fe00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff180071ce0 0x7ff1800ac190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:40.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.883+0000 7ff17fe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff1800c06b0 0x7ff1800c5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:40.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.883+0000 7ff17fe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff180072690 con 0x7ff18006ae70 2024-02-21T23:10:40.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.883+0000 7ff17fe00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff18006ea20 con 0x7ff1800c06b0 2024-02-21T23:10:40.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.883+0000 7ff17fe00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff180152880 con 0x7ff180071ce0 2024-02-21T23:10:40.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.883+0000 7ff17ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff18006ae70 0x7ff180153a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:40.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.883+0000 7ff17ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff18006ae70 0x7ff180153a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53550/0 (socket says 172.21.15.28:53550) 2024-02-21T23:10:40.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.883+0000 7ff17ea00640 1 -- 172.21.15.28:0/2601270596 learned_addr learned my addr 172.21.15.28:0/2601270596 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:40.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.884+0000 7ff17f400640 1 --2- 172.21.15.28:0/2601270596 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff1800c06b0 0x7ff1800c5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:40.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.884+0000 7ff17e000640 1 --2- 172.21.15.28:0/2601270596 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff180071ce0 0x7ff1800ac190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:40.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.884+0000 7ff17ea00640 1 -- 172.21.15.28:0/2601270596 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff180071ce0 msgr2=0x7ff1800ac190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:40.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.884+0000 7ff17ea00640 1 --2- 172.21.15.28:0/2601270596 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff180071ce0 0x7ff1800ac190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:40.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.884+0000 7ff17ea00640 1 -- 172.21.15.28:0/2601270596 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff1800c06b0 msgr2=0x7ff1800c5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:40.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.884+0000 7ff17ea00640 1 --2- 172.21.15.28:0/2601270596 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff1800c06b0 0x7ff1800c5a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:40.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.884+0000 7ff17ea00640 1 -- 172.21.15.28:0/2601270596 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff180154ae0 con 0x7ff18006ae70 2024-02-21T23:10:40.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.884+0000 7ff17f400640 1 --2- 172.21.15.28:0/2601270596 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff1800c06b0 0x7ff1800c5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:10:40.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.884+0000 7ff17e000640 1 --2- 172.21.15.28:0/2601270596 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff180071ce0 0x7ff1800ac190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:10:40.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.884+0000 7ff17ea00640 1 --2- 172.21.15.28:0/2601270596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff18006ae70 0x7ff180153a70 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7ff17000c430 tx=0x7ff17000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:40.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.884+0000 7ff176a00640 1 -- 172.21.15.28:0/2601270596 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff170011790 con 0x7ff18006ae70 2024-02-21T23:10:40.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.885+0000 7ff176a00640 1 -- 172.21.15.28:0/2601270596 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff170017070 con 0x7ff18006ae70 2024-02-21T23:10:40.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.885+0000 7ff176a00640 1 -- 172.21.15.28:0/2601270596 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff170011bb0 con 0x7ff18006ae70 2024-02-21T23:10:40.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.885+0000 7ff17fe00640 1 -- 172.21.15.28:0/2601270596 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff180154dc0 con 0x7ff18006ae70 2024-02-21T23:10:40.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.885+0000 7ff17fe00640 1 -- 172.21.15.28:0/2601270596 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff180155ad0 con 0x7ff18006ae70 2024-02-21T23:10:40.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.886+0000 7ff17fe00640 1 -- 172.21.15.28:0/2601270596 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff18004f290 con 0x7ff18006ae70 2024-02-21T23:10:40.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.887+0000 7ff176a00640 1 -- 172.21.15.28:0/2601270596 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff17001c460 con 0x7ff18006ae70 2024-02-21T23:10:40.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.888+0000 7ff176a00640 1 --2- 172.21.15.28:0/2601270596 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff16006b4d0 0x7ff16006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:40.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.888+0000 7ff176a00640 1 -- 172.21.15.28:0/2601270596 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 6622+0+0 (secure 0 0 0) 0x7ff17008a490 con 0x7ff18006ae70 2024-02-21T23:10:40.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.889+0000 7ff17e000640 1 --2- 172.21.15.28:0/2601270596 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff16006b4d0 0x7ff16006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:40.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.890+0000 7ff17e000640 1 --2- 172.21.15.28:0/2601270596 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff16006b4d0 0x7ff16006d990 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7ff16c006fd0 tx=0x7ff16c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:40.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:40.892+0000 7ff176a00640 1 -- 172.21.15.28:0/2601270596 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ff170058f70 con 0x7ff18006ae70 2024-02-21T23:10:41.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:41.098+0000 7ff17fe00640 1 -- 172.21.15.28:0/2601270596 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs2_metadata", "pool2": "cephfs2_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ff18006bc70 con 0x7ff18006ae70 2024-02-21T23:10:41.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:41.748+0000 7ff176a00640 1 -- 172.21.15.28:0/2601270596 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs2_metadata", "pool2": "cephfs2_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs2_metadata' removed v165) v1 ==== 184+0+0 (secure 0 0 0) 0x7ff170058870 con 0x7ff18006ae70 2024-02-21T23:10:41.738 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs2_metadata' removed 2024-02-21T23:10:41.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:41.752+0000 7ff17fe00640 1 -- 172.21.15.28:0/2601270596 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff16006b4d0 msgr2=0x7ff16006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:41.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:41.752+0000 7ff17fe00640 1 --2- 172.21.15.28:0/2601270596 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff16006b4d0 0x7ff16006d990 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7ff16c006fd0 tx=0x7ff16c008040 comp rx=0 tx=0).stop 2024-02-21T23:10:41.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:41.752+0000 7ff17fe00640 1 -- 172.21.15.28:0/2601270596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff18006ae70 msgr2=0x7ff180153a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:41.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:41.752+0000 7ff17fe00640 1 --2- 172.21.15.28:0/2601270596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff18006ae70 0x7ff180153a70 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7ff17000c430 tx=0x7ff17000c8f0 comp rx=0 tx=0).stop 2024-02-21T23:10:41.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:41.753+0000 7ff17fe00640 1 -- 172.21.15.28:0/2601270596 shutdown_connections 2024-02-21T23:10:41.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:41.753+0000 7ff17fe00640 1 --2- 172.21.15.28:0/2601270596 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff16006b4d0 0x7ff16006d990 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:41.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:41.753+0000 7ff17fe00640 1 --2- 172.21.15.28:0/2601270596 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff180071ce0 0x7ff1800ac190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:41.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:41.753+0000 7ff17fe00640 1 --2- 172.21.15.28:0/2601270596 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff1800c06b0 0x7ff1800c5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:41.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:41.753+0000 7ff17fe00640 1 --2- 172.21.15.28:0/2601270596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff18006ae70 0x7ff180153a70 unknown :-1 s=CLOSED pgs=1861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:41.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:41.753+0000 7ff17fe00640 1 -- 172.21.15.28:0/2601270596 >> 172.21.15.28:0/2601270596 conn(0x7ff1800c6610 msgr2=0x7ff18014d120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:41.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:41.754+0000 7ff17fe00640 1 -- 172.21.15.28:0/2601270596 shutdown_connections 2024-02-21T23:10:41.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:41.754+0000 7ff17fe00640 1 -- 172.21.15.28:0/2601270596 wait complete. 2024-02-21T23:10:41.766 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs2_data cephfs2_data --yes-i-really-really-mean-it 2024-02-21T23:10:41.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:41.998+0000 7f46eb200640 1 Processor -- start 2024-02-21T23:10:41.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:41.999+0000 7f46eb200640 1 -- start start 2024-02-21T23:10:41.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.000+0000 7f46eb200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f46ec06cfa0 0x7f46ec0ca540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:41.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.000+0000 7f46eb200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f46ec158460 0x7f46ec0c6290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:41.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.000+0000 7f46eb200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f46ec0c8f70 0x7f46ec1522c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:41.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.000+0000 7f46eb200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46ec0c8ae0 con 0x7f46ec0c8f70 2024-02-21T23:10:41.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.000+0000 7f46eb200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46ec0caf40 con 0x7f46ec06cfa0 2024-02-21T23:10:41.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.000+0000 7f46eb200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46ec1588f0 con 0x7f46ec158460 2024-02-21T23:10:41.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.001+0000 7f46ea800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f46ec0c8f70 0x7f46ec1522c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:41.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.001+0000 7f46ea800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f46ec0c8f70 0x7f46ec1522c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53554/0 (socket says 172.21.15.28:53554) 2024-02-21T23:10:41.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.001+0000 7f46ea800640 1 -- 172.21.15.28:0/3878168931 learned_addr learned my addr 172.21.15.28:0/3878168931 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:41.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.001+0000 7f46e9400640 1 --2- 172.21.15.28:0/3878168931 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f46ec158460 0x7f46ec0c6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:41.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.001+0000 7f46e9e00640 1 --2- 172.21.15.28:0/3878168931 >> v2:172.21.15.81:3300/0 conn(0x7f46ec06cfa0 0x7f46ec0ca540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:41.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.001+0000 7f46ea800640 1 -- 172.21.15.28:0/3878168931 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f46ec158460 msgr2=0x7f46ec0c6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:41.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.001+0000 7f46ea800640 1 --2- 172.21.15.28:0/3878168931 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f46ec158460 0x7f46ec0c6290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:41.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.001+0000 7f46ea800640 1 -- 172.21.15.28:0/3878168931 >> v2:172.21.15.81:3300/0 conn(0x7f46ec06cfa0 msgr2=0x7f46ec0ca540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:41.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.001+0000 7f46ea800640 1 --2- 172.21.15.28:0/3878168931 >> v2:172.21.15.81:3300/0 conn(0x7f46ec06cfa0 0x7f46ec0ca540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:41.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.001+0000 7f46ea800640 1 -- 172.21.15.28:0/3878168931 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46ec0c87b0 con 0x7f46ec0c8f70 2024-02-21T23:10:41.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.001+0000 7f46e9e00640 1 --2- 172.21.15.28:0/3878168931 >> v2:172.21.15.81:3300/0 conn(0x7f46ec06cfa0 0x7f46ec0ca540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:10:41.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.002+0000 7f46ea800640 1 --2- 172.21.15.28:0/3878168931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46ec0c8f70 0x7f46ec1522c0 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7f46ec0aa1b0 tx=0x7f46e402bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c8f04f4d0e3298b server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:41.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.003+0000 7f46e8a00640 1 -- 172.21.15.28:0/3878168931 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46e402c960 con 0x7f46ec0c8f70 2024-02-21T23:10:41.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.003+0000 7f46e8a00640 1 -- 172.21.15.28:0/3878168931 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46e402cb00 con 0x7f46ec0c8f70 2024-02-21T23:10:41.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.003+0000 7f46e8a00640 1 -- 172.21.15.28:0/3878168931 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46e40343f0 con 0x7f46ec0c8f70 2024-02-21T23:10:41.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.003+0000 7f46eb200640 1 -- 172.21.15.28:0/3878168931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46ec0c8f70 msgr2=0x7f46ec1522c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:41.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.003+0000 7f46eb200640 1 --2- 172.21.15.28:0/3878168931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46ec0c8f70 0x7f46ec1522c0 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7f46ec0aa1b0 tx=0x7f46e402bdc0 comp rx=0 tx=0).stop 2024-02-21T23:10:41.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.004+0000 7f46eb200640 1 -- 172.21.15.28:0/3878168931 shutdown_connections 2024-02-21T23:10:41.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.004+0000 7f46eb200640 1 --2- 172.21.15.28:0/3878168931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46ec0c8f70 0x7f46ec1522c0 unknown :-1 s=CLOSED pgs=1862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:41.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.004+0000 7f46eb200640 1 --2- 172.21.15.28:0/3878168931 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f46ec158460 0x7f46ec0c6290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:41.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.004+0000 7f46eb200640 1 --2- 172.21.15.28:0/3878168931 >> v2:172.21.15.81:3300/0 conn(0x7f46ec06cfa0 0x7f46ec0ca540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:41.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.004+0000 7f46eb200640 1 -- 172.21.15.28:0/3878168931 >> 172.21.15.28:0/3878168931 conn(0x7f46ec0725a0 msgr2=0x7f46ec0729b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:41.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.004+0000 7f46eb200640 1 -- 172.21.15.28:0/3878168931 shutdown_connections 2024-02-21T23:10:41.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.004+0000 7f46eb200640 1 -- 172.21.15.28:0/3878168931 wait complete. 2024-02-21T23:10:41.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.005+0000 7f46eb200640 1 Processor -- start 2024-02-21T23:10:41.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.005+0000 7f46eb200640 1 -- start start 2024-02-21T23:10:41.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.006+0000 7f46eb200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46ec06cfa0 0x7f46ec06d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:41.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.006+0000 7f46eb200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46ec0c8f70 0x7f46ec0ab7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:41.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.006+0000 7f46eb200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f46ec158460 0x7f46ec0abce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:41.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.006+0000 7f46eb200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46ec0c93c0 con 0x7f46ec06cfa0 2024-02-21T23:10:41.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.006+0000 7f46eb200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46ec0c6c90 con 0x7f46ec0c8f70 2024-02-21T23:10:41.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.006+0000 7f46eb200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46ec0c9540 con 0x7f46ec158460 2024-02-21T23:10:41.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.007+0000 7f46e9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46ec06cfa0 0x7f46ec06d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:41.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.007+0000 7f46e9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46ec06cfa0 0x7f46ec06d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53568/0 (socket says 172.21.15.28:53568) 2024-02-21T23:10:41.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.007+0000 7f46e9e00640 1 -- 172.21.15.28:0/1171103213 learned_addr learned my addr 172.21.15.28:0/1171103213 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:41.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.007+0000 7f46e9400640 1 --2- 172.21.15.28:0/1171103213 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46ec0c8f70 0x7f46ec0ab7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:41.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.007+0000 7f46ea800640 1 --2- 172.21.15.28:0/1171103213 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f46ec158460 0x7f46ec0abce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:41.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.007+0000 7f46e9e00640 1 -- 172.21.15.28:0/1171103213 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f46ec158460 msgr2=0x7f46ec0abce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:41.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.007+0000 7f46e9e00640 1 --2- 172.21.15.28:0/1171103213 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f46ec158460 0x7f46ec0abce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:41.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.007+0000 7f46e9e00640 1 -- 172.21.15.28:0/1171103213 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46ec0c8f70 msgr2=0x7f46ec0ab7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:41.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.007+0000 7f46e9e00640 1 --2- 172.21.15.28:0/1171103213 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46ec0c8f70 0x7f46ec0ab7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:41.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.007+0000 7f46e9e00640 1 -- 172.21.15.28:0/1171103213 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46ec0c7930 con 0x7f46ec06cfa0 2024-02-21T23:10:41.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.007+0000 7f46ea800640 1 --2- 172.21.15.28:0/1171103213 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f46ec158460 0x7f46ec0abce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:10:41.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.008+0000 7f46e9e00640 1 --2- 172.21.15.28:0/1171103213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46ec06cfa0 0x7f46ec06d5f0 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7f46dc00c820 tx=0x7f46dc00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:41.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.008+0000 7f46e2000640 1 -- 172.21.15.28:0/1171103213 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46dc015070 con 0x7f46ec06cfa0 2024-02-21T23:10:41.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.008+0000 7f46e2000640 1 -- 172.21.15.28:0/1171103213 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46dc007a30 con 0x7f46ec06cfa0 2024-02-21T23:10:41.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.008+0000 7f46e2000640 1 -- 172.21.15.28:0/1171103213 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46dc007e20 con 0x7f46ec06cfa0 2024-02-21T23:10:41.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.008+0000 7f46eb200640 1 -- 172.21.15.28:0/1171103213 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46ec1de730 con 0x7f46ec06cfa0 2024-02-21T23:10:41.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.008+0000 7f46eb200640 1 -- 172.21.15.28:0/1171103213 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46ec0aaa60 con 0x7f46ec06cfa0 2024-02-21T23:10:41.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.009+0000 7f46eb200640 1 -- 172.21.15.28:0/1171103213 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46ec04f290 con 0x7f46ec06cfa0 2024-02-21T23:10:41.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.010+0000 7f46e2000640 1 -- 172.21.15.28:0/1171103213 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f46dc018070 con 0x7f46ec06cfa0 2024-02-21T23:10:42.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.011+0000 7f46e2000640 1 --2- 172.21.15.28:0/1171103213 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f46c806b4d0 0x7f46c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:42.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.011+0000 7f46e2000640 1 -- 172.21.15.28:0/1171103213 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 6182+0+0 (secure 0 0 0) 0x7f46dc08a2c0 con 0x7f46ec06cfa0 2024-02-21T23:10:42.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.012+0000 7f46e9400640 1 --2- 172.21.15.28:0/1171103213 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f46c806b4d0 0x7f46c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:42.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.013+0000 7f46e9400640 1 --2- 172.21.15.28:0/1171103213 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f46c806b4d0 0x7f46c806d990 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f46d4006fd0 tx=0x7f46d4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:42.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.016+0000 7f46e2000640 1 -- 172.21.15.28:0/1171103213 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f46ec1de730 con 0x7f46ec06cfa0 2024-02-21T23:10:42.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.225+0000 7f46eb200640 1 -- 172.21.15.28:0/1171103213 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs2_data", "pool2": "cephfs2_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f46ec158220 con 0x7f46ec06cfa0 2024-02-21T23:10:42.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.752+0000 7f46e2000640 1 -- 172.21.15.28:0/1171103213 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs2_data", "pool2": "cephfs2_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs2_data' removed v166) v1 ==== 172+0+0 (secure 0 0 0) 0x7f46ec0c7930 con 0x7f46ec06cfa0 2024-02-21T23:10:42.741 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs2_data' removed 2024-02-21T23:10:42.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.756+0000 7f46eb200640 1 -- 172.21.15.28:0/1171103213 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f46c806b4d0 msgr2=0x7f46c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:42.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.756+0000 7f46eb200640 1 --2- 172.21.15.28:0/1171103213 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f46c806b4d0 0x7f46c806d990 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f46d4006fd0 tx=0x7f46d4008040 comp rx=0 tx=0).stop 2024-02-21T23:10:42.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.756+0000 7f46eb200640 1 -- 172.21.15.28:0/1171103213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46ec06cfa0 msgr2=0x7f46ec06d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:42.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.756+0000 7f46eb200640 1 --2- 172.21.15.28:0/1171103213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46ec06cfa0 0x7f46ec06d5f0 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7f46dc00c820 tx=0x7f46dc00cce0 comp rx=0 tx=0).stop 2024-02-21T23:10:42.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.761+0000 7f46eb200640 1 -- 172.21.15.28:0/1171103213 shutdown_connections 2024-02-21T23:10:42.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.761+0000 7f46eb200640 1 --2- 172.21.15.28:0/1171103213 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f46c806b4d0 0x7f46c806d990 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:42.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.761+0000 7f46eb200640 1 --2- 172.21.15.28:0/1171103213 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f46ec158460 0x7f46ec0abce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:42.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.761+0000 7f46eb200640 1 --2- 172.21.15.28:0/1171103213 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f46ec0c8f70 0x7f46ec0ab7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:42.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.761+0000 7f46eb200640 1 --2- 172.21.15.28:0/1171103213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46ec06cfa0 0x7f46ec06d5f0 unknown :-1 s=CLOSED pgs=1863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:42.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.761+0000 7f46eb200640 1 -- 172.21.15.28:0/1171103213 >> 172.21.15.28:0/1171103213 conn(0x7f46ec0725a0 msgr2=0x7f46ec154650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:42.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.761+0000 7f46eb200640 1 -- 172.21.15.28:0/1171103213 shutdown_connections 2024-02-21T23:10:42.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.761+0000 7f46eb200640 1 -- 172.21.15.28:0/1171103213 wait complete. 2024-02-21T23:10:42.773 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.b mds_join_fs 2024-02-21T23:10:42.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.984+0000 7f99afa00640 1 Processor -- start 2024-02-21T23:10:42.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.984+0000 7f99afa00640 1 -- start start 2024-02-21T23:10:42.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.986+0000 7f99afa00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f99a806b0e0 0x7f99a80c5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:42.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.986+0000 7f99afa00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f99a8150700 0x7f99a8156500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:42.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.986+0000 7f99afa00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99a80c61e0 con 0x7f99a80c5d80 2024-02-21T23:10:42.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.986+0000 7f99afa00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99a8153bc0 con 0x7f99a8150700 2024-02-21T23:10:42.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.986+0000 7f99afa00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99a8152fb0 con 0x7f99a806b0e0 2024-02-21T23:10:42.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.986+0000 7f99a7e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f99a80c5d80 0x7f99a814fff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:47882/0 (socket says 172.21.15.28:47882) 2024-02-21T23:10:42.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.986+0000 7f99a7e00640 1 -- 172.21.15.28:0/303431223 learned_addr learned my addr 172.21.15.28:0/303431223 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:10:42.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.987+0000 7f99ada00640 1 --2- 172.21.15.28:0/303431223 >> v2:172.21.15.81:3300/0 conn(0x7f99a8150700 0x7f99a8156500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:42.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.987+0000 7f99a7400640 1 -- 172.21.15.28:0/303431223 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 872165671 0 0) 0x7f99a80c61e0 con 0x7f99a80c5d80 2024-02-21T23:10:42.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.987+0000 7f99ad000640 1 --2- 172.21.15.28:0/303431223 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f99a806b0e0 0x7f99a80c5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:42.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.987+0000 7f99a7400640 1 -- 172.21.15.28:0/303431223 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f998c003520 con 0x7f99a80c5d80 2024-02-21T23:10:42.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.987+0000 7f99a7400640 1 -- 172.21.15.28:0/303431223 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 905764142 0 0) 0x7f998c003520 con 0x7f99a80c5d80 2024-02-21T23:10:42.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.987+0000 7f99a7400640 1 -- 172.21.15.28:0/303431223 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f99a806b0e0 msgr2=0x7f99a80c5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:42.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.987+0000 7f99a7400640 1 --2- 172.21.15.28:0/303431223 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f99a806b0e0 0x7f99a80c5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:42.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.987+0000 7f99a7400640 1 -- 172.21.15.28:0/303431223 >> v2:172.21.15.81:3300/0 conn(0x7f99a8150700 msgr2=0x7f99a8156500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:42.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.987+0000 7f99a7400640 1 --2- 172.21.15.28:0/303431223 >> v2:172.21.15.81:3300/0 conn(0x7f99a8150700 0x7f99a8156500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:42.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.987+0000 7f99a7400640 1 -- 172.21.15.28:0/303431223 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99a8153290 con 0x7f99a80c5d80 2024-02-21T23:10:42.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.987+0000 7f99a7400640 1 -- 172.21.15.28:0/303431223 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f9990001670 con 0x7f99a80c5d80 2024-02-21T23:10:42.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.987+0000 7f99a7400640 1 -- 172.21.15.28:0/303431223 >> v1:172.21.15.28:6789/0 conn(0x7f99a80c5d80 legacy=0x7f99a814fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:42.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.987+0000 7f99a7400640 1 --2- 172.21.15.28:0/303431223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f998c0039b0 0x7f998c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:42.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.987+0000 7f99a7400640 1 --2- 172.21.15.28:0/303431223 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f998c004800 0x7f998c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:42.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.987+0000 7f99a7400640 1 --2- 172.21.15.28:0/303431223 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f99a806b0e0 0x7f998c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:42.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.987+0000 7f99a7400640 1 -- 172.21.15.28:0/303431223 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99a8153bc0 con 0x7f998c0039b0 2024-02-21T23:10:42.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.987+0000 7f99a7400640 1 -- 172.21.15.28:0/303431223 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99a8152fb0 con 0x7f998c004800 2024-02-21T23:10:42.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.987+0000 7f99a7400640 1 -- 172.21.15.28:0/303431223 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f998c00bb20 con 0x7f99a806b0e0 2024-02-21T23:10:42.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.987+0000 7f99ad000640 1 --2- 172.21.15.28:0/303431223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f998c0039b0 0x7f998c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:42.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.988+0000 7f99a7e00640 1 --2- 172.21.15.28:0/303431223 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f998c004800 0x7f998c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:42.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.988+0000 7f99ada00640 1 --2- 172.21.15.28:0/303431223 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f99a806b0e0 0x7f998c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:42.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.988+0000 7f99ad000640 1 -- 172.21.15.28:0/303431223 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f99a806b0e0 msgr2=0x7f998c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:42.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.988+0000 7f99ad000640 1 --2- 172.21.15.28:0/303431223 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f99a806b0e0 0x7f998c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:42.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.988+0000 7f99ad000640 1 -- 172.21.15.28:0/303431223 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f998c004800 msgr2=0x7f998c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:42.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.988+0000 7f99ad000640 1 --2- 172.21.15.28:0/303431223 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f998c004800 0x7f998c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:42.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.988+0000 7f99ad000640 1 -- 172.21.15.28:0/303431223 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f998c003520 con 0x7f998c0039b0 2024-02-21T23:10:42.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.988+0000 7f99ada00640 1 --2- 172.21.15.28:0/303431223 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f99a806b0e0 0x7f998c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:10:42.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.988+0000 7f99a7e00640 1 --2- 172.21.15.28:0/303431223 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f998c004800 0x7f998c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:10:42.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.990+0000 7f99ad000640 1 --2- 172.21.15.28:0/303431223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f998c0039b0 0x7f998c003dd0 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7f9998007330 tx=0x7f998c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4046cb46ceae195 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:42.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.990+0000 7f99a7400640 1 -- 172.21.15.28:0/303431223 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9998039070 con 0x7f998c0039b0 2024-02-21T23:10:42.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.990+0000 7f99a7400640 1 -- 172.21.15.28:0/303431223 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f999802cc70 con 0x7f998c0039b0 2024-02-21T23:10:42.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.990+0000 7f99a7400640 1 -- 172.21.15.28:0/303431223 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9998009be0 con 0x7f998c0039b0 2024-02-21T23:10:42.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.991+0000 7f99afa00640 1 -- 172.21.15.28:0/303431223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f998c0039b0 msgr2=0x7f998c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:42.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.991+0000 7f99afa00640 1 --2- 172.21.15.28:0/303431223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f998c0039b0 0x7f998c003dd0 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7f9998007330 tx=0x7f998c005830 comp rx=0 tx=0).stop 2024-02-21T23:10:42.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.991+0000 7f99ada00640 1 -- 172.21.15.28:0/303431223 reap_dead start 2024-02-21T23:10:42.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.991+0000 7f99afa00640 1 -- 172.21.15.28:0/303431223 shutdown_connections 2024-02-21T23:10:42.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.991+0000 7f99afa00640 1 -- 172.21.15.28:0/303431223 >> 172.21.15.28:0/303431223 conn(0x7f99a80a85a0 msgr2=0x7f99a80a8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:42.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.991+0000 7f99afa00640 1 -- 172.21.15.28:0/303431223 shutdown_connections 2024-02-21T23:10:42.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.991+0000 7f99afa00640 1 -- 172.21.15.28:0/303431223 wait complete. 2024-02-21T23:10:42.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.993+0000 7f99afa00640 1 Processor -- start 2024-02-21T23:10:42.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.993+0000 7f99afa00640 1 -- start start 2024-02-21T23:10:42.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.994+0000 7f99afa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99a806e960 0x7f99a8147ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:42.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.994+0000 7f99afa00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f99a814e410 0x7f99a81547a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:42.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.994+0000 7f99afa00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f99a8154150 0x7f99a81438e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:42.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.994+0000 7f99afa00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99a806b4c0 con 0x7f99a806e960 2024-02-21T23:10:42.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.994+0000 7f99afa00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99a8153850 con 0x7f99a814e410 2024-02-21T23:10:42.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.994+0000 7f99afa00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99a806b640 con 0x7f99a8154150 2024-02-21T23:10:42.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.994+0000 7f99ad000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99a806e960 0x7f99a8147ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:42.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.994+0000 7f99ad000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99a806e960 0x7f99a8147ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53592/0 (socket says 172.21.15.28:53592) 2024-02-21T23:10:42.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.994+0000 7f99ad000640 1 -- 172.21.15.28:0/994058926 learned_addr learned my addr 172.21.15.28:0/994058926 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:42.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.994+0000 7f99ada00640 1 --2- 172.21.15.28:0/994058926 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f99a8154150 0x7f99a81438e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:42.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.994+0000 7f99a7e00640 1 --2- 172.21.15.28:0/994058926 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f99a814e410 0x7f99a81547a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:42.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.994+0000 7f99ad000640 1 -- 172.21.15.28:0/994058926 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f99a8154150 msgr2=0x7f99a81438e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:42.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.994+0000 7f99ad000640 1 --2- 172.21.15.28:0/994058926 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f99a8154150 0x7f99a81438e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:42.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.994+0000 7f99ad000640 1 -- 172.21.15.28:0/994058926 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f99a814e410 msgr2=0x7f99a81547a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:42.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.994+0000 7f99ad000640 1 --2- 172.21.15.28:0/994058926 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f99a814e410 0x7f99a81547a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:42.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.994+0000 7f99ad000640 1 -- 172.21.15.28:0/994058926 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99a81e7140 con 0x7f99a806e960 2024-02-21T23:10:42.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.995+0000 7f99ada00640 1 --2- 172.21.15.28:0/994058926 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f99a8154150 0x7f99a81438e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:10:42.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.995+0000 7f99a7e00640 1 --2- 172.21.15.28:0/994058926 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f99a814e410 0x7f99a81547a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:10:42.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.995+0000 7f99ad000640 1 --2- 172.21.15.28:0/994058926 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99a806e960 0x7f99a8147ab0 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7f9998037040 tx=0x7f9998004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:42.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.995+0000 7f99a4c00640 1 -- 172.21.15.28:0/994058926 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9998042030 con 0x7f99a806e960 2024-02-21T23:10:42.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.995+0000 7f99a4c00640 1 -- 172.21.15.28:0/994058926 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f999803a040 con 0x7f99a806e960 2024-02-21T23:10:42.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.995+0000 7f99a4c00640 1 -- 172.21.15.28:0/994058926 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9998039070 con 0x7f99a806e960 2024-02-21T23:10:42.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.995+0000 7f99afa00640 1 -- 172.21.15.28:0/994058926 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99a81e7420 con 0x7f99a806e960 2024-02-21T23:10:42.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.995+0000 7f99afa00640 1 -- 172.21.15.28:0/994058926 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99a814dfb0 con 0x7f99a806e960 2024-02-21T23:10:42.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.997+0000 7f99afa00640 1 -- 172.21.15.28:0/994058926 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99a806f8c0 con 0x7f99a806e960 2024-02-21T23:10:42.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.998+0000 7f99a4c00640 1 -- 172.21.15.28:0/994058926 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9998045070 con 0x7f99a806e960 2024-02-21T23:10:42.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.998+0000 7f99a4c00640 1 --2- 172.21.15.28:0/994058926 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f998406b4d0 0x7f998406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:42.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.999+0000 7f99a4c00640 1 -- 172.21.15.28:0/994058926 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 5679+0+0 (secure 0 0 0) 0x7f99980abd30 con 0x7f99a806e960 2024-02-21T23:10:42.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:42.999+0000 7f99a7e00640 1 --2- 172.21.15.28:0/994058926 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f998406b4d0 0x7f998406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:42.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.001+0000 7f99a7e00640 1 --2- 172.21.15.28:0/994058926 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f998406b4d0 0x7f998406d990 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f99a8148cd0 tx=0x7f99900047e0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:42.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.004+0000 7f99a4c00640 1 -- 172.21.15.28:0/994058926 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f99a806f8c0 con 0x7f99a806e960 2024-02-21T23:10:43.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.207+0000 7f99afa00640 1 -- 172.21.15.28:0/994058926 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.b", "name": "mds_join_fs"} v 0) v1 -- 0x7f99a8067f30 con 0x7f99a806e960 2024-02-21T23:10:43.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.215+0000 7f99a4c00640 1 -- 172.21.15.28:0/994058926 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.b", "name": "mds_join_fs"}]=0 v54) v1 ==== 96+0+0 (secure 0 0 0) 0x7f999804c070 con 0x7f99a806e960 2024-02-21T23:10:43.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.217+0000 7f99afa00640 1 -- 172.21.15.28:0/994058926 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f998406b4d0 msgr2=0x7f998406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:43.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.217+0000 7f99afa00640 1 --2- 172.21.15.28:0/994058926 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f998406b4d0 0x7f998406d990 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f99a8148cd0 tx=0x7f99900047e0 comp rx=0 tx=0).stop 2024-02-21T23:10:43.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.217+0000 7f99afa00640 1 -- 172.21.15.28:0/994058926 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99a806e960 msgr2=0x7f99a8147ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:43.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.217+0000 7f99afa00640 1 --2- 172.21.15.28:0/994058926 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99a806e960 0x7f99a8147ab0 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7f9998037040 tx=0x7f9998004ab0 comp rx=0 tx=0).stop 2024-02-21T23:10:43.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.218+0000 7f99afa00640 1 -- 172.21.15.28:0/994058926 shutdown_connections 2024-02-21T23:10:43.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.218+0000 7f99afa00640 1 --2- 172.21.15.28:0/994058926 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f998406b4d0 0x7f998406d990 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:43.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.218+0000 7f99afa00640 1 --2- 172.21.15.28:0/994058926 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f99a8154150 0x7f99a81438e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:43.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.218+0000 7f99afa00640 1 --2- 172.21.15.28:0/994058926 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f99a814e410 0x7f99a81547a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:43.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.218+0000 7f99afa00640 1 --2- 172.21.15.28:0/994058926 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99a806e960 0x7f99a8147ab0 unknown :-1 s=CLOSED pgs=1866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:43.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.218+0000 7f99afa00640 1 -- 172.21.15.28:0/994058926 >> 172.21.15.28:0/994058926 conn(0x7f99a80a85a0 msgr2=0x7f99a80729d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:43.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.218+0000 7f99afa00640 1 -- 172.21.15.28:0/994058926 shutdown_connections 2024-02-21T23:10:43.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.218+0000 7f99afa00640 1 -- 172.21.15.28:0/994058926 wait complete. 2024-02-21T23:10:43.227 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds.c mds_join_fs 2024-02-21T23:10:43.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.456+0000 7fa9b9800640 1 Processor -- start 2024-02-21T23:10:43.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.456+0000 7fa9b9800640 1 -- start start 2024-02-21T23:10:43.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.458+0000 7fa9b9800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa9b41500f0 0x7fa9b406ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:43.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.458+0000 7fa9b9800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa9b406b3b0 0x7fa9b406e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:43.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.458+0000 7fa9b9800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9b406db10 con 0x7fa9b40c2c60 2024-02-21T23:10:43.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.458+0000 7fa9b9800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9b40c1080 con 0x7fa9b41500f0 2024-02-21T23:10:43.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.458+0000 7fa9b9800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b41539a0 con 0x7fa9b406b3b0 2024-02-21T23:10:43.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.458+0000 7fa9b3e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa9b41500f0 0x7fa9b406ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:43.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.458+0000 7fa9b3400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa9b40c2c60 0x7fa9b406dff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:47884/0 (socket says 172.21.15.28:47884) 2024-02-21T23:10:43.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.458+0000 7fa9b3400640 1 -- 172.21.15.28:0/1465228228 learned_addr learned my addr 172.21.15.28:0/1465228228 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:10:43.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.459+0000 7fa9b8e00640 1 --2- 172.21.15.28:0/1465228228 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa9b406b3b0 0x7fa9b406e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:43.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.459+0000 7fa9b3e00640 1 -- 172.21.15.28:0/1465228228 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa9b406b3b0 msgr2=0x7fa9b406e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:43.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.459+0000 7fa9b3e00640 1 --2- 172.21.15.28:0/1465228228 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa9b406b3b0 0x7fa9b406e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:43.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.459+0000 7fa9b3e00640 1 -- 172.21.15.28:0/1465228228 >> v1:172.21.15.28:6789/0 conn(0x7fa9b40c2c60 legacy=0x7fa9b406dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:43.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.459+0000 7fa9b3e00640 1 -- 172.21.15.28:0/1465228228 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9b40c12d0 con 0x7fa9b41500f0 2024-02-21T23:10:43.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.459+0000 7fa9b8e00640 1 --2- 172.21.15.28:0/1465228228 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa9b406b3b0 0x7fa9b406e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:10:43.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.459+0000 7fa9b2a00640 1 -- 172.21.15.28:0/1465228228 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1479270667 0 0) 0x7fa9b406db10 con 0x7fa9b40c2c60 2024-02-21T23:10:43.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.460+0000 7fa9b3e00640 1 --2- 172.21.15.28:0/1465228228 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9b41500f0 0x7fa9b406ae70 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7fa9a4009d90 tx=0x7fa9a402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=440d35f66e6fc79 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:43.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.460+0000 7fa9b2a00640 1 -- 172.21.15.28:0/1465228228 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a4039070 con 0x7fa9b41500f0 2024-02-21T23:10:43.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.461+0000 7fa9b2a00640 1 -- 172.21.15.28:0/1465228228 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9a402cc50 con 0x7fa9b41500f0 2024-02-21T23:10:43.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.461+0000 7fa9b2a00640 1 -- 172.21.15.28:0/1465228228 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a4034440 con 0x7fa9b41500f0 2024-02-21T23:10:43.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.461+0000 7fa9b9800640 1 -- 172.21.15.28:0/1465228228 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9b41500f0 msgr2=0x7fa9b406ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:43.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.461+0000 7fa9b9800640 1 --2- 172.21.15.28:0/1465228228 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9b41500f0 0x7fa9b406ae70 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7fa9a4009d90 tx=0x7fa9a402bea0 comp rx=0 tx=0).stop 2024-02-21T23:10:43.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.461+0000 7fa9b9800640 1 -- 172.21.15.28:0/1465228228 shutdown_connections 2024-02-21T23:10:43.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.461+0000 7fa9b9800640 1 --2- 172.21.15.28:0/1465228228 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa9b406b3b0 0x7fa9b406e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:43.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.461+0000 7fa9b9800640 1 --2- 172.21.15.28:0/1465228228 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9b41500f0 0x7fa9b406ae70 unknown :-1 s=CLOSED pgs=1868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:43.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.461+0000 7fa9b9800640 1 -- 172.21.15.28:0/1465228228 >> 172.21.15.28:0/1465228228 conn(0x7fa9b40a8530 msgr2=0x7fa9b40a8910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:43.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.461+0000 7fa9b9800640 1 -- 172.21.15.28:0/1465228228 shutdown_connections 2024-02-21T23:10:43.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.461+0000 7fa9b9800640 1 -- 172.21.15.28:0/1465228228 wait complete. 2024-02-21T23:10:43.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.463+0000 7fa9b9800640 1 Processor -- start 2024-02-21T23:10:43.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.463+0000 7fa9b9800640 1 -- start start 2024-02-21T23:10:43.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.464+0000 7fa9b9800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa9b406b3b0 0x7fa9b41edf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:43.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.464+0000 7fa9b9800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9b40c2c60 0x7fa9b4150c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:43.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.464+0000 7fa9b9800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa9b41500f0 0x7fa9b4145920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:43.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.464+0000 7fa9b9800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b41535e0 con 0x7fa9b40c2c60 2024-02-21T23:10:43.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.464+0000 7fa9b9800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b41504d0 con 0x7fa9b41500f0 2024-02-21T23:10:43.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.464+0000 7fa9b9800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b4153760 con 0x7fa9b406b3b0 2024-02-21T23:10:43.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.464+0000 7fa9b3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9b40c2c60 0x7fa9b4150c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:43.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.464+0000 7fa9b3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9b40c2c60 0x7fa9b4150c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53612/0 (socket says 172.21.15.28:53612) 2024-02-21T23:10:43.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.464+0000 7fa9b3400640 1 -- 172.21.15.28:0/3283869712 learned_addr learned my addr 172.21.15.28:0/3283869712 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:43.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.465+0000 7fa9b3e00640 1 --2- 172.21.15.28:0/3283869712 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa9b406b3b0 0x7fa9b41edf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:43.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.465+0000 7fa9b3400640 1 -- 172.21.15.28:0/3283869712 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa9b406b3b0 msgr2=0x7fa9b41edf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:43.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.465+0000 7fa9b3400640 1 --2- 172.21.15.28:0/3283869712 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa9b406b3b0 0x7fa9b41edf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:43.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.465+0000 7fa9b3400640 1 -- 172.21.15.28:0/3283869712 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa9b41500f0 msgr2=0x7fa9b4145920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:43.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.465+0000 7fa9b3400640 1 --2- 172.21.15.28:0/3283869712 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa9b41500f0 0x7fa9b4145920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:43.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.465+0000 7fa9b3400640 1 -- 172.21.15.28:0/3283869712 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9b4152b50 con 0x7fa9b40c2c60 2024-02-21T23:10:43.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.465+0000 7fa9b3400640 1 --2- 172.21.15.28:0/3283869712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9b40c2c60 0x7fa9b4150c10 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7fa9a000c4b0 tx=0x7fa9a000c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:43.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.465+0000 7fa9abe00640 1 -- 172.21.15.28:0/3283869712 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a0018070 con 0x7fa9b40c2c60 2024-02-21T23:10:43.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.466+0000 7fa9abe00640 1 -- 172.21.15.28:0/3283869712 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9a0007500 con 0x7fa9b40c2c60 2024-02-21T23:10:43.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.466+0000 7fa9abe00640 1 -- 172.21.15.28:0/3283869712 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a0004a20 con 0x7fa9b40c2c60 2024-02-21T23:10:43.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.466+0000 7fa9b9800640 1 -- 172.21.15.28:0/3283869712 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9b4072ed0 con 0x7fa9b40c2c60 2024-02-21T23:10:43.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.466+0000 7fa9b9800640 1 -- 172.21.15.28:0/3283869712 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9b40731b0 con 0x7fa9b40c2c60 2024-02-21T23:10:43.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.467+0000 7fa9b9800640 1 -- 172.21.15.28:0/3283869712 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9b406c440 con 0x7fa9b40c2c60 2024-02-21T23:10:43.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.468+0000 7fa9abe00640 1 -- 172.21.15.28:0/3283869712 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa9a001b070 con 0x7fa9b40c2c60 2024-02-21T23:10:43.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.469+0000 7fa9abe00640 1 --2- 172.21.15.28:0/3283869712 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa99406b4d0 0x7fa99406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:43.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.469+0000 7fa9abe00640 1 -- 172.21.15.28:0/3283869712 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 5679+0+0 (secure 0 0 0) 0x7fa9a0089560 con 0x7fa9b40c2c60 2024-02-21T23:10:43.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.469+0000 7fa9b3e00640 1 --2- 172.21.15.28:0/3283869712 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa99406b4d0 0x7fa99406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:43.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.471+0000 7fa9b3e00640 1 --2- 172.21.15.28:0/3283869712 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa99406b4d0 0x7fa99406d990 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fa9a4036000 tx=0x7fa9a4030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:43.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.473+0000 7fa9abe00640 1 -- 172.21.15.28:0/3283869712 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fa9b406c440 con 0x7fa9b40c2c60 2024-02-21T23:10:43.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.674+0000 7fa9b9800640 1 -- 172.21.15.28:0/3283869712 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config rm", "who": "mds.c", "name": "mds_join_fs"} v 0) v1 -- 0x7fa9b404f290 con 0x7fa9b40c2c60 2024-02-21T23:10:43.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.682+0000 7fa9abe00640 1 -- 172.21.15.28:0/3283869712 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds.c", "name": "mds_join_fs"}]=0 v55) v1 ==== 96+0+0 (secure 0 0 0) 0x7fa9a0058420 con 0x7fa9b40c2c60 2024-02-21T23:10:43.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.685+0000 7fa9b9800640 1 -- 172.21.15.28:0/3283869712 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa99406b4d0 msgr2=0x7fa99406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:43.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.685+0000 7fa9b9800640 1 --2- 172.21.15.28:0/3283869712 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa99406b4d0 0x7fa99406d990 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fa9a4036000 tx=0x7fa9a4030000 comp rx=0 tx=0).stop 2024-02-21T23:10:43.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.685+0000 7fa9b9800640 1 -- 172.21.15.28:0/3283869712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9b40c2c60 msgr2=0x7fa9b4150c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:43.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.685+0000 7fa9b9800640 1 --2- 172.21.15.28:0/3283869712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9b40c2c60 0x7fa9b4150c10 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7fa9a000c4b0 tx=0x7fa9a000c970 comp rx=0 tx=0).stop 2024-02-21T23:10:43.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.685+0000 7fa9b9800640 1 -- 172.21.15.28:0/3283869712 shutdown_connections 2024-02-21T23:10:43.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.685+0000 7fa9b9800640 1 --2- 172.21.15.28:0/3283869712 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa99406b4d0 0x7fa99406d990 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:43.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.685+0000 7fa9b9800640 1 --2- 172.21.15.28:0/3283869712 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa9b41500f0 0x7fa9b4145920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:43.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.685+0000 7fa9b9800640 1 --2- 172.21.15.28:0/3283869712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9b40c2c60 0x7fa9b4150c10 unknown :-1 s=CLOSED pgs=1869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:43.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.685+0000 7fa9b9800640 1 --2- 172.21.15.28:0/3283869712 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa9b406b3b0 0x7fa9b41edf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:43.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.685+0000 7fa9b9800640 1 -- 172.21.15.28:0/3283869712 >> 172.21.15.28:0/3283869712 conn(0x7fa9b40a8530 msgr2=0x7fa9b40c39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:43.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.685+0000 7fa9b9800640 1 -- 172.21.15.28:0/3283869712 shutdown_connections 2024-02-21T23:10:43.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.685+0000 7fa9b9800640 1 -- 172.21.15.28:0/3283869712 wait complete. 2024-02-21T23:10:43.694 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-02-21T23:10:43.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.920+0000 7fc319e00640 1 Processor -- start 2024-02-21T23:10:43.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.921+0000 7fc319e00640 1 -- start start 2024-02-21T23:10:43.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.922+0000 7fc319e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc3140c2c60 0x7fc31406dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:43.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.922+0000 7fc319e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc31406db10 con 0x7fc31406f400 2024-02-21T23:10:43.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.922+0000 7fc319e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3141539a0 con 0x7fc3141500f0 2024-02-21T23:10:43.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.922+0000 7fc319e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3140c1080 con 0x7fc3140c2c60 2024-02-21T23:10:43.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.923+0000 7fc313e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc3140c2c60 0x7fc31406dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:43.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.923+0000 7fc319400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fc31406f400 0x7fc31406e530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:47898/0 (socket says 172.21.15.28:47898) 2024-02-21T23:10:43.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.923+0000 7fc319400640 1 -- 172.21.15.28:0/4056680564 learned_addr learned my addr 172.21.15.28:0/4056680564 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:10:43.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.923+0000 7fc313e00640 1 -- 172.21.15.28:0/4056680564 >> v1:172.21.15.81:6789/0 conn(0x7fc3141500f0 legacy=0x7fc31406ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:43.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.923+0000 7fc313e00640 1 -- 172.21.15.28:0/4056680564 >> v1:172.21.15.28:6789/0 conn(0x7fc31406f400 legacy=0x7fc31406e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:43.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.923+0000 7fc313e00640 1 -- 172.21.15.28:0/4056680564 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3140c12d0 con 0x7fc3140c2c60 2024-02-21T23:10:43.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.924+0000 7fc313e00640 1 --2- 172.21.15.28:0/4056680564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3140c2c60 0x7fc31406dff0 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7fc3140a94a0 tx=0x7fc30002b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f233c6a781a2ed7e server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:43.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.925+0000 7fc313400640 1 -- 172.21.15.28:0/4056680564 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3000317f0 con 0x7fc3140c2c60 2024-02-21T23:10:43.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.925+0000 7fc313400640 1 -- 172.21.15.28:0/4056680564 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc300037070 con 0x7fc3140c2c60 2024-02-21T23:10:43.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.925+0000 7fc313400640 1 -- 172.21.15.28:0/4056680564 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc300031bb0 con 0x7fc3140c2c60 2024-02-21T23:10:43.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.925+0000 7fc319e00640 1 -- 172.21.15.28:0/4056680564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3140c2c60 msgr2=0x7fc31406dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:43.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.925+0000 7fc319e00640 1 --2- 172.21.15.28:0/4056680564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3140c2c60 0x7fc31406dff0 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7fc3140a94a0 tx=0x7fc30002b890 comp rx=0 tx=0).stop 2024-02-21T23:10:43.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.925+0000 7fc319e00640 1 -- 172.21.15.28:0/4056680564 shutdown_connections 2024-02-21T23:10:43.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.925+0000 7fc319e00640 1 --2- 172.21.15.28:0/4056680564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3140c2c60 0x7fc31406dff0 unknown :-1 s=CLOSED pgs=1871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:43.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.925+0000 7fc319e00640 1 -- 172.21.15.28:0/4056680564 >> 172.21.15.28:0/4056680564 conn(0x7fc3140a8530 msgr2=0x7fc3140a8910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:43.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.925+0000 7fc319e00640 1 -- 172.21.15.28:0/4056680564 shutdown_connections 2024-02-21T23:10:43.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.925+0000 7fc319e00640 1 -- 172.21.15.28:0/4056680564 wait complete. 2024-02-21T23:10:43.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.927+0000 7fc319e00640 1 Processor -- start 2024-02-21T23:10:43.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.927+0000 7fc319e00640 1 -- start start 2024-02-21T23:10:43.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.928+0000 7fc319e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc31406f400 0x7fc31414f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:43.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.928+0000 7fc319e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3140c2c60 0x7fc3141e0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:43.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.928+0000 7fc319e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc3141500f0 0x7fc314151090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:43.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.928+0000 7fc319e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc31406b580 con 0x7fc3140c2c60 2024-02-21T23:10:43.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.928+0000 7fc319e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3141504d0 con 0x7fc31406f400 2024-02-21T23:10:43.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.928+0000 7fc319e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc31406b700 con 0x7fc3141500f0 2024-02-21T23:10:43.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.928+0000 7fc313e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3140c2c60 0x7fc3141e0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:43.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.929+0000 7fc313e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3140c2c60 0x7fc3141e0b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53624/0 (socket says 172.21.15.28:53624) 2024-02-21T23:10:43.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.929+0000 7fc313e00640 1 -- 172.21.15.28:0/1247962495 learned_addr learned my addr 172.21.15.28:0/1247962495 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:43.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.929+0000 7fc318a00640 1 --2- 172.21.15.28:0/1247962495 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc31406f400 0x7fc31414f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:43.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.929+0000 7fc319400640 1 --2- 172.21.15.28:0/1247962495 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc3141500f0 0x7fc314151090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:43.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.929+0000 7fc313e00640 1 -- 172.21.15.28:0/1247962495 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc3141500f0 msgr2=0x7fc314151090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:43.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.929+0000 7fc313e00640 1 --2- 172.21.15.28:0/1247962495 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc3141500f0 0x7fc314151090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:43.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.929+0000 7fc313e00640 1 -- 172.21.15.28:0/1247962495 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc31406f400 msgr2=0x7fc31414f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:43.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.929+0000 7fc313e00640 1 --2- 172.21.15.28:0/1247962495 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc31406f400 0x7fc31414f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:43.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.929+0000 7fc313e00640 1 -- 172.21.15.28:0/1247962495 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc314146500 con 0x7fc3140c2c60 2024-02-21T23:10:43.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.929+0000 7fc318a00640 1 --2- 172.21.15.28:0/1247962495 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc31406f400 0x7fc31414f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:10:43.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.929+0000 7fc313e00640 1 --2- 172.21.15.28:0/1247962495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3140c2c60 0x7fc3141e0b50 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7fc300000c00 tx=0x7fc3000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:43.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.930+0000 7fc310c00640 1 -- 172.21.15.28:0/1247962495 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc300032890 con 0x7fc3140c2c60 2024-02-21T23:10:43.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.930+0000 7fc310c00640 1 -- 172.21.15.28:0/1247962495 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc30002e070 con 0x7fc3140c2c60 2024-02-21T23:10:43.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.930+0000 7fc310c00640 1 -- 172.21.15.28:0/1247962495 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc300032c90 con 0x7fc3140c2c60 2024-02-21T23:10:43.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.930+0000 7fc319e00640 1 -- 172.21.15.28:0/1247962495 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc31414df70 con 0x7fc3140c2c60 2024-02-21T23:10:43.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.930+0000 7fc319e00640 1 -- 172.21.15.28:0/1247962495 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc31414e530 con 0x7fc3140c2c60 2024-02-21T23:10:43.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.931+0000 7fc319e00640 1 -- 172.21.15.28:0/1247962495 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc31404f290 con 0x7fc3140c2c60 2024-02-21T23:10:43.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.932+0000 7fc310c00640 1 -- 172.21.15.28:0/1247962495 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc3000453f0 con 0x7fc3140c2c60 2024-02-21T23:10:43.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.933+0000 7fc310c00640 1 --2- 172.21.15.28:0/1247962495 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc2f406b500 0x7fc2f406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:43.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.933+0000 7fc310c00640 1 -- 172.21.15.28:0/1247962495 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 5679+0+0 (secure 0 0 0) 0x7fc3000ac510 con 0x7fc3140c2c60 2024-02-21T23:10:43.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.933+0000 7fc318a00640 1 --2- 172.21.15.28:0/1247962495 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc2f406b500 0x7fc2f406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:43.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.935+0000 7fc318a00640 1 --2- 172.21.15.28:0/1247962495 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc2f406b500 0x7fc2f406d9c0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fc308006fd0 tx=0x7fc308008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:43.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:43.937+0000 7fc310c00640 1 -- 172.21.15.28:0/1247962495 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc314146500 con 0x7fc3140c2c60 2024-02-21T23:10:44.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.138+0000 7fc319e00640 1 -- 172.21.15.28:0/1247962495 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fc31406f1e0 con 0x7fc3140c2c60 2024-02-21T23:10:44.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.144+0000 7fc310c00640 1 -- 172.21.15.28:0/1247962495 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v56) v1 ==== 104+0+0 (secure 0 0 0) 0x7fc300040070 con 0x7fc3140c2c60 2024-02-21T23:10:44.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.147+0000 7fc319e00640 1 -- 172.21.15.28:0/1247962495 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc2f406b500 msgr2=0x7fc2f406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:44.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.147+0000 7fc319e00640 1 --2- 172.21.15.28:0/1247962495 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc2f406b500 0x7fc2f406d9c0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fc308006fd0 tx=0x7fc308008040 comp rx=0 tx=0).stop 2024-02-21T23:10:44.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.147+0000 7fc319e00640 1 -- 172.21.15.28:0/1247962495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3140c2c60 msgr2=0x7fc3141e0b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:44.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.147+0000 7fc319e00640 1 --2- 172.21.15.28:0/1247962495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3140c2c60 0x7fc3141e0b50 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7fc300000c00 tx=0x7fc3000046c0 comp rx=0 tx=0).stop 2024-02-21T23:10:44.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.147+0000 7fc319e00640 1 -- 172.21.15.28:0/1247962495 shutdown_connections 2024-02-21T23:10:44.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.147+0000 7fc319e00640 1 --2- 172.21.15.28:0/1247962495 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc2f406b500 0x7fc2f406d9c0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:44.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.147+0000 7fc319e00640 1 --2- 172.21.15.28:0/1247962495 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc3141500f0 0x7fc314151090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:44.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.147+0000 7fc319e00640 1 --2- 172.21.15.28:0/1247962495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3140c2c60 0x7fc3141e0b50 unknown :-1 s=CLOSED pgs=1872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:44.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.147+0000 7fc319e00640 1 --2- 172.21.15.28:0/1247962495 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc31406f400 0x7fc31414f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:44.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.147+0000 7fc319e00640 1 -- 172.21.15.28:0/1247962495 >> 172.21.15.28:0/1247962495 conn(0x7fc3140a8530 msgr2=0x7fc31406b960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:44.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.147+0000 7fc319e00640 1 -- 172.21.15.28:0/1247962495 shutdown_connections 2024-02-21T23:10:44.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.147+0000 7fc319e00640 1 -- 172.21.15.28:0/1247962495 wait complete. 2024-02-21T23:10:44.157 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_vanilla' 2024-02-21T23:10:44.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.391+0000 7fa9df200640 1 Processor -- start 2024-02-21T23:10:44.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.392+0000 7fa9df200640 1 -- start start 2024-02-21T23:10:44.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.393+0000 7fa9df200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa9e0155960 0x7fa9e01608a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:44.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.393+0000 7fa9df200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9e00c74c0 con 0x7fa9e006dbf0 2024-02-21T23:10:44.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.393+0000 7fa9df200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9e00c77a0 con 0x7fa9e00aff90 2024-02-21T23:10:44.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.393+0000 7fa9df200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e00c7270 con 0x7fa9e0155960 2024-02-21T23:10:44.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.393+0000 7fa9dd400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa9e006dbf0 0x7fa9e0158ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:47904/0 (socket says 172.21.15.28:47904) 2024-02-21T23:10:44.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.393+0000 7fa9dd400640 1 -- 172.21.15.28:0/1471370479 learned_addr learned my addr 172.21.15.28:0/1471370479 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:10:44.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.393+0000 7fa9de800640 1 --2- 172.21.15.28:0/1471370479 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa9e0155960 0x7fa9e01608a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:44.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.394+0000 7fa9dca00640 1 -- 172.21.15.28:0/1471370479 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1932924990 0 0) 0x7fa9e00c74c0 con 0x7fa9e006dbf0 2024-02-21T23:10:44.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.394+0000 7fa9dca00640 1 -- 172.21.15.28:0/1471370479 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9c4003520 con 0x7fa9e006dbf0 2024-02-21T23:10:44.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.394+0000 7fa9dca00640 1 -- 172.21.15.28:0/1471370479 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2854659925 0 0) 0x7fa9c4003520 con 0x7fa9e006dbf0 2024-02-21T23:10:44.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.394+0000 7fa9dca00640 1 -- 172.21.15.28:0/1471370479 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa9e0155960 msgr2=0x7fa9e01608a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:44.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.394+0000 7fa9dca00640 1 --2- 172.21.15.28:0/1471370479 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa9e0155960 0x7fa9e01608a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:44.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.394+0000 7fa9dca00640 1 -- 172.21.15.28:0/1471370479 >> v1:172.21.15.81:6789/0 conn(0x7fa9e00aff90 legacy=0x7fa9e00c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:44.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.394+0000 7fa9dca00640 1 -- 172.21.15.28:0/1471370479 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9e00c7a80 con 0x7fa9e006dbf0 2024-02-21T23:10:44.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.394+0000 7fa9dca00640 1 -- 172.21.15.28:0/1471370479 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fa9c8001670 con 0x7fa9e006dbf0 2024-02-21T23:10:44.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.394+0000 7fa9dca00640 1 -- 172.21.15.28:0/1471370479 >> v1:172.21.15.28:6789/0 conn(0x7fa9e006dbf0 legacy=0x7fa9e0158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:44.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.394+0000 7fa9dca00640 1 --2- 172.21.15.28:0/1471370479 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa9e0155960 0x7fa9c40039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:44.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.394+0000 7fa9dca00640 1 --2- 172.21.15.28:0/1471370479 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa9c4003fb0 0x7fa9c4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:44.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.394+0000 7fa9dca00640 1 --2- 172.21.15.28:0/1471370479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9c4004d40 0x7fa9c40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:44.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.394+0000 7fa9dca00640 1 -- 172.21.15.28:0/1471370479 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e00c7270 con 0x7fa9c4004d40 2024-02-21T23:10:44.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.394+0000 7fa9dca00640 1 -- 172.21.15.28:0/1471370479 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c400bb00 con 0x7fa9c4003fb0 2024-02-21T23:10:44.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.394+0000 7fa9dca00640 1 -- 172.21.15.28:0/1471370479 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c400bd00 con 0x7fa9e0155960 2024-02-21T23:10:44.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.394+0000 7fa9de800640 1 --2- 172.21.15.28:0/1471370479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9c4004d40 0x7fa9c40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:44.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.395+0000 7fa9dde00640 1 --2- 172.21.15.28:0/1471370479 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa9e0155960 0x7fa9c40039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:44.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.395+0000 7fa9dd400640 1 --2- 172.21.15.28:0/1471370479 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa9c4003fb0 0x7fa9c4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:44.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.395+0000 7fa9de800640 1 -- 172.21.15.28:0/1471370479 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa9e0155960 msgr2=0x7fa9c40039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:44.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.395+0000 7fa9de800640 1 --2- 172.21.15.28:0/1471370479 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa9e0155960 0x7fa9c40039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:44.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.395+0000 7fa9de800640 1 -- 172.21.15.28:0/1471370479 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa9c4003fb0 msgr2=0x7fa9c4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:44.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.395+0000 7fa9de800640 1 --2- 172.21.15.28:0/1471370479 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa9c4003fb0 0x7fa9c4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:44.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.395+0000 7fa9de800640 1 -- 172.21.15.28:0/1471370479 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9c4003520 con 0x7fa9c4004d40 2024-02-21T23:10:44.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.395+0000 7fa9dde00640 1 --2- 172.21.15.28:0/1471370479 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa9e0155960 0x7fa9c40039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:10:44.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.396+0000 7fa9de800640 1 --2- 172.21.15.28:0/1471370479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9c4004d40 0x7fa9c40051c0 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7fa9d8009ac0 tx=0x7fa9c4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1579a6b4f5792700 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:44.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.396+0000 7fa9dca00640 1 -- 172.21.15.28:0/1471370479 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d8039070 con 0x7fa9c4004d40 2024-02-21T23:10:44.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.397+0000 7fa9dca00640 1 -- 172.21.15.28:0/1471370479 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9d802c6b0 con 0x7fa9c4004d40 2024-02-21T23:10:44.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.397+0000 7fa9dca00640 1 -- 172.21.15.28:0/1471370479 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d8009690 con 0x7fa9c4004d40 2024-02-21T23:10:44.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.397+0000 7fa9df200640 1 -- 172.21.15.28:0/1471370479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9c4004d40 msgr2=0x7fa9c40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:44.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.397+0000 7fa9df200640 1 --2- 172.21.15.28:0/1471370479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9c4004d40 0x7fa9c40051c0 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7fa9d8009ac0 tx=0x7fa9c4005830 comp rx=0 tx=0).stop 2024-02-21T23:10:44.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.397+0000 7fa9de800640 1 -- 172.21.15.28:0/1471370479 reap_dead start 2024-02-21T23:10:44.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.397+0000 7fa9df200640 1 -- 172.21.15.28:0/1471370479 shutdown_connections 2024-02-21T23:10:44.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.397+0000 7fa9df200640 1 -- 172.21.15.28:0/1471370479 >> 172.21.15.28:0/1471370479 conn(0x7fa9e00ac370 msgr2=0x7fa9e00ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:44.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.397+0000 7fa9df200640 1 -- 172.21.15.28:0/1471370479 shutdown_connections 2024-02-21T23:10:44.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.398+0000 7fa9df200640 1 -- 172.21.15.28:0/1471370479 wait complete. 2024-02-21T23:10:44.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.399+0000 7fa9df200640 1 Processor -- start 2024-02-21T23:10:44.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.399+0000 7fa9df200640 1 -- start start 2024-02-21T23:10:44.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.400+0000 7fa9df200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9c4004d40 0x7fa9e00c2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:44.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.400+0000 7fa9df200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa9e00be7b0 0x7fa9e01ee2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:44.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.400+0000 7fa9df200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa9e01ee820 0x7fa9e01474d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:44.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.400+0000 7fa9df200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e00c58f0 con 0x7fa9c4004d40 2024-02-21T23:10:44.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.400+0000 7fa9df200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e0158630 con 0x7fa9e00be7b0 2024-02-21T23:10:44.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.400+0000 7fa9df200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e00c5a70 con 0x7fa9e01ee820 2024-02-21T23:10:44.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.400+0000 7fa9dde00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9c4004d40 0x7fa9e00c2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:44.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.401+0000 7fa9dde00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9c4004d40 0x7fa9e00c2e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53632/0 (socket says 172.21.15.28:53632) 2024-02-21T23:10:44.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.401+0000 7fa9dde00640 1 -- 172.21.15.28:0/3358273846 learned_addr learned my addr 172.21.15.28:0/3358273846 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:44.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.401+0000 7fa9de800640 1 --2- 172.21.15.28:0/3358273846 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa9e01ee820 0x7fa9e01474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:44.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.401+0000 7fa9dd400640 1 --2- 172.21.15.28:0/3358273846 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa9e00be7b0 0x7fa9e01ee2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:44.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.401+0000 7fa9dde00640 1 -- 172.21.15.28:0/3358273846 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa9e01ee820 msgr2=0x7fa9e01474d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:44.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.401+0000 7fa9dde00640 1 --2- 172.21.15.28:0/3358273846 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa9e01ee820 0x7fa9e01474d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:44.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.401+0000 7fa9dde00640 1 -- 172.21.15.28:0/3358273846 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa9e00be7b0 msgr2=0x7fa9e01ee2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:44.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.401+0000 7fa9dde00640 1 --2- 172.21.15.28:0/3358273846 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa9e00be7b0 0x7fa9e01ee2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:44.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.401+0000 7fa9dde00640 1 -- 172.21.15.28:0/3358273846 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9e00c3d50 con 0x7fa9c4004d40 2024-02-21T23:10:44.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.401+0000 7fa9de800640 1 --2- 172.21.15.28:0/3358273846 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa9e01ee820 0x7fa9e01474d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:10:44.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.401+0000 7fa9dde00640 1 --2- 172.21.15.28:0/3358273846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9c4004d40 0x7fa9e00c2e20 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7fa9d000eb20 tx=0x7fa9d00020b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:44.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.402+0000 7fa9d6000640 1 -- 172.21.15.28:0/3358273846 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d0003b80 con 0x7fa9c4004d40 2024-02-21T23:10:44.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.402+0000 7fa9d6000640 1 -- 172.21.15.28:0/3358273846 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9d0018070 con 0x7fa9c4004d40 2024-02-21T23:10:44.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.402+0000 7fa9d6000640 1 -- 172.21.15.28:0/3358273846 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d00050d0 con 0x7fa9c4004d40 2024-02-21T23:10:44.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.402+0000 7fa9df200640 1 -- 172.21.15.28:0/3358273846 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9e00c4030 con 0x7fa9c4004d40 2024-02-21T23:10:44.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.402+0000 7fa9df200640 1 -- 172.21.15.28:0/3358273846 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9e00c2620 con 0x7fa9c4004d40 2024-02-21T23:10:44.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.403+0000 7fa9df200640 1 -- 172.21.15.28:0/3358273846 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9e004f290 con 0x7fa9c4004d40 2024-02-21T23:10:44.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.404+0000 7fa9d6000640 1 -- 172.21.15.28:0/3358273846 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa9d0005270 con 0x7fa9c4004d40 2024-02-21T23:10:44.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.405+0000 7fa9d6000640 1 --2- 172.21.15.28:0/3358273846 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa9bc06b500 0x7fa9bc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:44.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.405+0000 7fa9d6000640 1 -- 172.21.15.28:0/3358273846 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 5679+0+0 (secure 0 0 0) 0x7fa9d008a2b0 con 0x7fa9c4004d40 2024-02-21T23:10:44.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.406+0000 7fa9dd400640 1 --2- 172.21.15.28:0/3358273846 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa9bc06b500 0x7fa9bc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:44.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.407+0000 7fa9dd400640 1 --2- 172.21.15.28:0/3358273846 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa9bc06b500 0x7fa9bc06d9c0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fa9c8001eb0 tx=0x7fa9c8000fa0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:44.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.410+0000 7fa9d6000640 1 -- 172.21.15.28:0/3358273846 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fa9d001a310 con 0x7fa9c4004d40 2024-02-21T23:10:44.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.682+0000 7fa9df200640 1 -- 172.21.15.28:0/3358273846 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_vanilla"]} v 0) v1 -- 0x7fa9e006d9d0 con 0x7fa9c4004d40 2024-02-21T23:10:44.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.774+0000 7fa9d6000640 1 -- 172.21.15.28:0/3358273846 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterAffinity.test_join_fs_vanilla"]}]=0 v470) v1 ==== 146+0+0 (secure 0 0 0) 0x7fa9d0059140 con 0x7fa9c4004d40 2024-02-21T23:10:44.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.777+0000 7fa9df200640 1 -- 172.21.15.28:0/3358273846 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa9bc06b500 msgr2=0x7fa9bc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:44.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.777+0000 7fa9df200640 1 --2- 172.21.15.28:0/3358273846 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa9bc06b500 0x7fa9bc06d9c0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fa9c8001eb0 tx=0x7fa9c8000fa0 comp rx=0 tx=0).stop 2024-02-21T23:10:44.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.777+0000 7fa9df200640 1 -- 172.21.15.28:0/3358273846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9c4004d40 msgr2=0x7fa9e00c2e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:44.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.777+0000 7fa9df200640 1 --2- 172.21.15.28:0/3358273846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9c4004d40 0x7fa9e00c2e20 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7fa9d000eb20 tx=0x7fa9d00020b0 comp rx=0 tx=0).stop 2024-02-21T23:10:44.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.777+0000 7fa9df200640 1 -- 172.21.15.28:0/3358273846 shutdown_connections 2024-02-21T23:10:44.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.777+0000 7fa9df200640 1 --2- 172.21.15.28:0/3358273846 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa9bc06b500 0x7fa9bc06d9c0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:44.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.777+0000 7fa9df200640 1 --2- 172.21.15.28:0/3358273846 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa9e01ee820 0x7fa9e01474d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:44.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.777+0000 7fa9df200640 1 --2- 172.21.15.28:0/3358273846 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa9e00be7b0 0x7fa9e01ee2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:44.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.777+0000 7fa9df200640 1 --2- 172.21.15.28:0/3358273846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9c4004d40 0x7fa9e00c2e20 unknown :-1 s=CLOSED pgs=1875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:44.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.777+0000 7fa9df200640 1 -- 172.21.15.28:0/3358273846 >> 172.21.15.28:0/3358273846 conn(0x7fa9e00ac370 msgr2=0x7fa9e00abb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:44.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.778+0000 7fa9df200640 1 -- 172.21.15.28:0/3358273846 shutdown_connections 2024-02-21T23:10:44.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:44.778+0000 7fa9df200640 1 -- 172.21.15.28:0/3358273846 wait complete. 2024-02-21T23:10:44.786 INFO:tasks.cephfs_test_runner:That a vanilla standby is preferred over others with mds_join_fs set to another fs. ... ok 2024-02-21T23:10:44.787 INFO:tasks.cephfs_test_runner:Starting test: test_down (tasks.cephfs.test_failover.TestClusterResize) That down setting toggles and sets max_mds appropriately. 2024-02-21T23:10:44.787 INFO:tasks.cephfs_test_runner:test_down (tasks.cephfs.test_failover.TestClusterResize) 2024-02-21T23:10:44.787 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:10:45.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.018+0000 7fe1a9a00640 1 Processor -- start 2024-02-21T23:10:45.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.018+0000 7fe1a9a00640 1 -- start start 2024-02-21T23:10:45.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.020+0000 7fe1a9a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fe1a406dbf0 0x7fe1a4158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:45.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.020+0000 7fe1a9a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe1a4155960 0x7fe1a4159000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:45.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.020+0000 7fe1a9a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1a40c74c0 con 0x7fe1a40aff90 2024-02-21T23:10:45.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.020+0000 7fe1a9a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1a40c7270 con 0x7fe1a4155960 2024-02-21T23:10:45.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.020+0000 7fe1a9a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1a4158630 con 0x7fe1a406dbf0 2024-02-21T23:10:45.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.020+0000 7fe1a3e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe1a40aff90 0x7fe1a40c51e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41246/0 (socket says 172.21.15.28:41246) 2024-02-21T23:10:45.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.020+0000 7fe1a3e00640 1 -- 172.21.15.28:0/1559200002 learned_addr learned my addr 172.21.15.28:0/1559200002 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:10:45.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.020+0000 7fe1a9000640 1 --2- 172.21.15.28:0/1559200002 >> v2:172.21.15.28:3300/0 conn(0x7fe1a4155960 0x7fe1a4159000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:45.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.021+0000 7fe19be00640 1 --2- 172.21.15.28:0/1559200002 >> v2:172.21.15.81:3300/0 conn(0x7fe1a406dbf0 0x7fe1a4158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:45.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.021+0000 7fe1a9000640 1 -- 172.21.15.28:0/1559200002 >> v2:172.21.15.81:3300/0 conn(0x7fe1a406dbf0 msgr2=0x7fe1a4158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:45.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.021+0000 7fe1a9000640 1 --2- 172.21.15.28:0/1559200002 >> v2:172.21.15.81:3300/0 conn(0x7fe1a406dbf0 0x7fe1a4158ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:45.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.021+0000 7fe1a9000640 1 -- 172.21.15.28:0/1559200002 >> v1:172.21.15.28:6789/0 conn(0x7fe1a40aff90 legacy=0x7fe1a40c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:45.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.021+0000 7fe1a9000640 1 -- 172.21.15.28:0/1559200002 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1a40c77a0 con 0x7fe1a4155960 2024-02-21T23:10:45.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.021+0000 7fe1a3400640 1 -- 172.21.15.28:0/1559200002 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1788585137 0 0) 0x7fe1a40c74c0 con 0x7fe1a40aff90 2024-02-21T23:10:45.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.022+0000 7fe1a9000640 1 --2- 172.21.15.28:0/1559200002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1a4155960 0x7fe1a4159000 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7fe1a40cb8c0 tx=0x7fe19c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aef2505e112e15d0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:45.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.022+0000 7fe1a3400640 1 -- 172.21.15.28:0/1559200002 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe19c039070 con 0x7fe1a4155960 2024-02-21T23:10:45.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.023+0000 7fe1a3400640 1 -- 172.21.15.28:0/1559200002 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe19c02cc80 con 0x7fe1a4155960 2024-02-21T23:10:45.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.023+0000 7fe1a3400640 1 -- 172.21.15.28:0/1559200002 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe19c034660 con 0x7fe1a4155960 2024-02-21T23:10:45.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.023+0000 7fe1a9a00640 1 -- 172.21.15.28:0/1559200002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1a4155960 msgr2=0x7fe1a4159000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:45.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.023+0000 7fe1a9a00640 1 --2- 172.21.15.28:0/1559200002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1a4155960 0x7fe1a4159000 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7fe1a40cb8c0 tx=0x7fe19c02bed0 comp rx=0 tx=0).stop 2024-02-21T23:10:45.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.023+0000 7fe1a9a00640 1 -- 172.21.15.28:0/1559200002 shutdown_connections 2024-02-21T23:10:45.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.023+0000 7fe1a9a00640 1 --2- 172.21.15.28:0/1559200002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1a4155960 0x7fe1a4159000 unknown :-1 s=CLOSED pgs=1877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:45.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.023+0000 7fe1a9a00640 1 --2- 172.21.15.28:0/1559200002 >> v2:172.21.15.81:3300/0 conn(0x7fe1a406dbf0 0x7fe1a4158ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:45.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.023+0000 7fe1a9a00640 1 -- 172.21.15.28:0/1559200002 >> 172.21.15.28:0/1559200002 conn(0x7fe1a40ac370 msgr2=0x7fe1a40ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:45.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.023+0000 7fe1a9a00640 1 -- 172.21.15.28:0/1559200002 shutdown_connections 2024-02-21T23:10:45.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.023+0000 7fe1a9a00640 1 -- 172.21.15.28:0/1559200002 wait complete. 2024-02-21T23:10:45.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.025+0000 7fe1a9a00640 1 Processor -- start 2024-02-21T23:10:45.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.025+0000 7fe1a9a00640 1 -- start start 2024-02-21T23:10:45.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.026+0000 7fe1a9a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe1a406dbf0 0x7fe1a41edea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:45.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.026+0000 7fe1a9a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1a40aff90 0x7fe1a40bd040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:45.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.026+0000 7fe1a9a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe1a4155960 0x7fe1a40c2720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:45.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.026+0000 7fe1a9a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a40c58f0 con 0x7fe1a40aff90 2024-02-21T23:10:45.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.026+0000 7fe1a9a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a40be0c0 con 0x7fe1a406dbf0 2024-02-21T23:10:45.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.026+0000 7fe1a9a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a40c5a70 con 0x7fe1a4155960 2024-02-21T23:10:45.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.026+0000 7fe19be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1a40aff90 0x7fe1a40bd040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:45.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.026+0000 7fe19be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1a40aff90 0x7fe1a40bd040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49098/0 (socket says 172.21.15.28:49098) 2024-02-21T23:10:45.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.026+0000 7fe19be00640 1 -- 172.21.15.28:0/2790079350 learned_addr learned my addr 172.21.15.28:0/2790079350 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:45.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.027+0000 7fe1a3e00640 1 --2- 172.21.15.28:0/2790079350 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe1a406dbf0 0x7fe1a41edea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:45.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.027+0000 7fe1a9000640 1 --2- 172.21.15.28:0/2790079350 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe1a4155960 0x7fe1a40c2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:45.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.027+0000 7fe19be00640 1 -- 172.21.15.28:0/2790079350 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe1a4155960 msgr2=0x7fe1a40c2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:45.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.027+0000 7fe19be00640 1 --2- 172.21.15.28:0/2790079350 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe1a4155960 0x7fe1a40c2720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:45.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.027+0000 7fe19be00640 1 -- 172.21.15.28:0/2790079350 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe1a406dbf0 msgr2=0x7fe1a41edea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:45.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.027+0000 7fe19be00640 1 --2- 172.21.15.28:0/2790079350 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe1a406dbf0 0x7fe1a41edea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:45.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.027+0000 7fe19be00640 1 -- 172.21.15.28:0/2790079350 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1a41ee7e0 con 0x7fe1a40aff90 2024-02-21T23:10:45.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.027+0000 7fe19be00640 1 --2- 172.21.15.28:0/2790079350 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1a40aff90 0x7fe1a40bd040 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7fe18c00e9b0 tx=0x7fe18c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:45.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.027+0000 7fe1a0c00640 1 -- 172.21.15.28:0/2790079350 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe18c017070 con 0x7fe1a40aff90 2024-02-21T23:10:45.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.027+0000 7fe1a0c00640 1 -- 172.21.15.28:0/2790079350 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe18c00c9a0 con 0x7fe1a40aff90 2024-02-21T23:10:45.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.027+0000 7fe1a9a00640 1 -- 172.21.15.28:0/2790079350 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1a40c3e00 con 0x7fe1a40aff90 2024-02-21T23:10:45.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.027+0000 7fe1a0c00640 1 -- 172.21.15.28:0/2790079350 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe18c00cd70 con 0x7fe1a40aff90 2024-02-21T23:10:45.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.028+0000 7fe1a9a00640 1 -- 172.21.15.28:0/2790079350 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1a40c1190 con 0x7fe1a40aff90 2024-02-21T23:10:45.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.029+0000 7fe1a9a00640 1 -- 172.21.15.28:0/2790079350 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1a40c4320 con 0x7fe1a40aff90 2024-02-21T23:10:45.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.029+0000 7fe1a0c00640 1 -- 172.21.15.28:0/2790079350 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe18c01a070 con 0x7fe1a40aff90 2024-02-21T23:10:45.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.030+0000 7fe1a0c00640 1 --2- 172.21.15.28:0/2790079350 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe18006b4d0 0x7fe18006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:45.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.030+0000 7fe1a0c00640 1 -- 172.21.15.28:0/2790079350 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 5679+0+0 (secure 0 0 0) 0x7fe18c08a5a0 con 0x7fe1a40aff90 2024-02-21T23:10:45.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.031+0000 7fe1a3e00640 1 --2- 172.21.15.28:0/2790079350 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe18006b4d0 0x7fe18006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:45.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.032+0000 7fe1a3e00640 1 --2- 172.21.15.28:0/2790079350 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe18006b4d0 0x7fe18006d990 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fe194000f60 tx=0x7fe194009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:45.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.035+0000 7fe1a0c00640 1 -- 172.21.15.28:0/2790079350 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe1a40c4320 con 0x7fe1a40aff90 2024-02-21T23:10:45.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.237+0000 7fe1a9a00640 1 -- 172.21.15.28:0/2790079350 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe1a404f290 con 0x7fe1a40aff90 2024-02-21T23:10:45.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.239+0000 7fe1a0c00640 1 -- 172.21.15.28:0/2790079350 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v166) v1 ==== 74+0+13761 (secure 0 0 0) 0x7fe18c059430 con 0x7fe1a40aff90 2024-02-21T23:10:45.228 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:10:45.228 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":166,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:10:42.749785+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6832/2178509171":"2024-02-22T23:10:39.576935+0000","172.21.15.81:6835/1320495906":"2024-02-22T23:10:32.538190+0000","172.21.15.81:6834/1320495906":"2024-02-22T23:10:32.538190+0000","172.21.15.81:6837/521625945":"2024-02-22T23:10:31.712025+0000","172.21.15.28:6833/2975378719":"2024-02-22T23:10:21.581948+0000","172.21.15.28:6833/2178509171":"2024-02-22T23:10:39.576935+0000","172.21.15.28:6832/2975378719":"2024-02-22T23:10:21.581948+0000","172.21.15.81:6836/521625945":"2024-02-22T23:10:31.712025+0000","172.21.15.28:6835/2928639298":"2024-02-22T23:10:19.550812+0000","172.21.15.28:6834/2928639298":"2024-02-22T23:10:19.550812+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:10:45.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.241+0000 7fe1a9a00640 1 -- 172.21.15.28:0/2790079350 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe18006b4d0 msgr2=0x7fe18006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:45.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.241+0000 7fe1a9a00640 1 --2- 172.21.15.28:0/2790079350 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe18006b4d0 0x7fe18006d990 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fe194000f60 tx=0x7fe194009040 comp rx=0 tx=0).stop 2024-02-21T23:10:45.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.242+0000 7fe1a9a00640 1 -- 172.21.15.28:0/2790079350 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1a40aff90 msgr2=0x7fe1a40bd040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:45.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.242+0000 7fe1a9a00640 1 --2- 172.21.15.28:0/2790079350 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1a40aff90 0x7fe1a40bd040 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7fe18c00e9b0 tx=0x7fe18c00ee70 comp rx=0 tx=0).stop 2024-02-21T23:10:45.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.242+0000 7fe1a9a00640 1 -- 172.21.15.28:0/2790079350 shutdown_connections 2024-02-21T23:10:45.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.242+0000 7fe1a9a00640 1 --2- 172.21.15.28:0/2790079350 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe18006b4d0 0x7fe18006d990 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:45.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.242+0000 7fe1a9a00640 1 --2- 172.21.15.28:0/2790079350 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe1a4155960 0x7fe1a40c2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:45.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.242+0000 7fe1a9a00640 1 --2- 172.21.15.28:0/2790079350 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1a40aff90 0x7fe1a40bd040 unknown :-1 s=CLOSED pgs=1878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:45.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.242+0000 7fe1a9a00640 1 --2- 172.21.15.28:0/2790079350 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe1a406dbf0 0x7fe1a41edea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:45.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.242+0000 7fe1a9a00640 1 -- 172.21.15.28:0/2790079350 >> 172.21.15.28:0/2790079350 conn(0x7fe1a40ac370 msgr2=0x7fe1a40c50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:45.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.242+0000 7fe1a9a00640 1 -- 172.21.15.28:0/2790079350 shutdown_connections 2024-02-21T23:10:45.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.242+0000 7fe1a9a00640 1 -- 172.21.15.28:0/2790079350 wait complete. 2024-02-21T23:10:45.252 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-21T23:02:30.728329+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-21T23:10:45.253 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-21T23:10:45.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.491+0000 7ff881e00640 1 Processor -- start 2024-02-21T23:10:45.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.491+0000 7ff881e00640 1 -- start start 2024-02-21T23:10:45.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.493+0000 7ff881e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7ff87c0c4580 0x7ff87c0c12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:45.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.493+0000 7ff881e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff87c06e220 con 0x7ff87c150240 2024-02-21T23:10:45.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.493+0000 7ff881e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff87c06f780 con 0x7ff87c0c2c10 2024-02-21T23:10:45.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.493+0000 7ff881e00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff87c150620 con 0x7ff87c0c4580 2024-02-21T23:10:45.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.493+0000 7ff880a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff87c150240 0x7ff87c06db10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41256/0 (socket says 172.21.15.28:41256) 2024-02-21T23:10:45.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.493+0000 7ff880a00640 1 -- 172.21.15.28:0/1492837001 learned_addr learned my addr 172.21.15.28:0/1492837001 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:10:45.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.494+0000 7ff87be00640 1 --2- 172.21.15.28:0/1492837001 >> v2:172.21.15.81:3300/0 conn(0x7ff87c0c4580 0x7ff87c0c12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:45.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.494+0000 7ff87b400640 1 -- 172.21.15.28:0/1492837001 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1208290909 0 0) 0x7ff87c06e220 con 0x7ff87c150240 2024-02-21T23:10:45.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.494+0000 7ff87b400640 1 -- 172.21.15.28:0/1492837001 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff864003520 con 0x7ff87c150240 2024-02-21T23:10:45.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.494+0000 7ff87b400640 1 -- 172.21.15.28:0/1492837001 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3790446992 0 0) 0x7ff864003520 con 0x7ff87c150240 2024-02-21T23:10:45.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.494+0000 7ff87b400640 1 -- 172.21.15.28:0/1492837001 >> v2:172.21.15.81:3300/0 conn(0x7ff87c0c4580 msgr2=0x7ff87c0c12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:45.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.494+0000 7ff87b400640 1 --2- 172.21.15.28:0/1492837001 >> v2:172.21.15.81:3300/0 conn(0x7ff87c0c4580 0x7ff87c0c12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:45.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.494+0000 7ff87b400640 1 -- 172.21.15.28:0/1492837001 >> v1:172.21.15.81:6789/0 conn(0x7ff87c0c2c10 legacy=0x7ff87c06ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:45.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.494+0000 7ff87b400640 1 -- 172.21.15.28:0/1492837001 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff87c06f4a0 con 0x7ff87c150240 2024-02-21T23:10:45.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.494+0000 7ff87b400640 1 -- 172.21.15.28:0/1492837001 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7ff870001670 con 0x7ff87c150240 2024-02-21T23:10:45.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.494+0000 7ff87b400640 1 -- 172.21.15.28:0/1492837001 >> v1:172.21.15.28:6789/0 conn(0x7ff87c150240 legacy=0x7ff87c06db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:45.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.494+0000 7ff87b400640 1 --2- 172.21.15.28:0/1492837001 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff8640039b0 0x7ff864003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:45.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.494+0000 7ff87b400640 1 --2- 172.21.15.28:0/1492837001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff864004800 0x7ff864004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:45.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.494+0000 7ff87b400640 1 --2- 172.21.15.28:0/1492837001 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff8640051c0 0x7ff86400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:45.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.494+0000 7ff87b400640 1 -- 172.21.15.28:0/1492837001 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff86400c130 con 0x7ff864004800 2024-02-21T23:10:45.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.494+0000 7ff87b400640 1 -- 172.21.15.28:0/1492837001 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff86400c2e0 con 0x7ff8640051c0 2024-02-21T23:10:45.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.494+0000 7ff87b400640 1 -- 172.21.15.28:0/1492837001 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff86400c4e0 con 0x7ff8640039b0 2024-02-21T23:10:45.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.495+0000 7ff87be00640 1 --2- 172.21.15.28:0/1492837001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff864004800 0x7ff864004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:45.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.495+0000 7ff881400640 1 --2- 172.21.15.28:0/1492837001 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff8640051c0 0x7ff86400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:45.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.495+0000 7ff880a00640 1 --2- 172.21.15.28:0/1492837001 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff8640039b0 0x7ff864003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:45.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.495+0000 7ff87be00640 1 -- 172.21.15.28:0/1492837001 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff8640039b0 msgr2=0x7ff864003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:45.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.495+0000 7ff87be00640 1 --2- 172.21.15.28:0/1492837001 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff8640039b0 0x7ff864003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:45.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.495+0000 7ff87be00640 1 -- 172.21.15.28:0/1492837001 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff8640051c0 msgr2=0x7ff86400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:45.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.495+0000 7ff87be00640 1 --2- 172.21.15.28:0/1492837001 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff8640051c0 0x7ff86400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:45.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.495+0000 7ff87be00640 1 -- 172.21.15.28:0/1492837001 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff864003520 con 0x7ff864004800 2024-02-21T23:10:45.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.495+0000 7ff881400640 1 -- 172.21.15.28:0/1492837001 reap_dead start 2024-02-21T23:10:45.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.495+0000 7ff880a00640 1 --2- 172.21.15.28:0/1492837001 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff8640039b0 0x7ff864003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:10:45.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.496+0000 7ff87be00640 1 --2- 172.21.15.28:0/1492837001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff864004800 0x7ff864004c80 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7ff868009a90 tx=0x7ff864005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4536705cdd7a67d server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:45.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.497+0000 7ff87b400640 1 -- 172.21.15.28:0/1492837001 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff868039070 con 0x7ff864004800 2024-02-21T23:10:45.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.497+0000 7ff87b400640 1 -- 172.21.15.28:0/1492837001 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff86802c680 con 0x7ff864004800 2024-02-21T23:10:45.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.497+0000 7ff87b400640 1 -- 172.21.15.28:0/1492837001 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff868009690 con 0x7ff864004800 2024-02-21T23:10:45.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.497+0000 7ff881e00640 1 -- 172.21.15.28:0/1492837001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff864004800 msgr2=0x7ff864004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:45.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.497+0000 7ff881e00640 1 --2- 172.21.15.28:0/1492837001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff864004800 0x7ff864004c80 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7ff868009a90 tx=0x7ff864005830 comp rx=0 tx=0).stop 2024-02-21T23:10:45.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.497+0000 7ff881e00640 1 -- 172.21.15.28:0/1492837001 shutdown_connections 2024-02-21T23:10:45.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.497+0000 7ff881e00640 1 --2- 172.21.15.28:0/1492837001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff864004800 0x7ff864004c80 unknown :-1 s=CLOSED pgs=1880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:45.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.497+0000 7ff881e00640 1 -- 172.21.15.28:0/1492837001 >> 172.21.15.28:0/1492837001 conn(0x7ff87c0a8530 msgr2=0x7ff87c0a8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:45.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.497+0000 7ff881e00640 1 -- 172.21.15.28:0/1492837001 shutdown_connections 2024-02-21T23:10:45.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.498+0000 7ff881e00640 1 -- 172.21.15.28:0/1492837001 wait complete. 2024-02-21T23:10:45.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.499+0000 7ff881e00640 1 Processor -- start 2024-02-21T23:10:45.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.500+0000 7ff881e00640 1 -- start start 2024-02-21T23:10:45.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.501+0000 7ff881e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff864004800 0x7ff87c0a5db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:45.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.501+0000 7ff881e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff87c149e70 0x7ff87c1e9e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:45.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.501+0000 7ff881e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff87c1ea380 0x7ff87c146480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:45.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.501+0000 7ff881e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff87c0c4a10 con 0x7ff87c149e70 2024-02-21T23:10:45.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.501+0000 7ff881e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff87c153a90 con 0x7ff864004800 2024-02-21T23:10:45.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.501+0000 7ff881e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff87c0c4b90 con 0x7ff87c1ea380 2024-02-21T23:10:45.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.501+0000 7ff87be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff87c149e70 0x7ff87c1e9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:45.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.501+0000 7ff87be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff87c149e70 0x7ff87c1e9e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49122/0 (socket says 172.21.15.28:49122) 2024-02-21T23:10:45.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.501+0000 7ff87be00640 1 -- 172.21.15.28:0/2941337100 learned_addr learned my addr 172.21.15.28:0/2941337100 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:45.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.501+0000 7ff880a00640 1 --2- 172.21.15.28:0/2941337100 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff864004800 0x7ff87c0a5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:45.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.501+0000 7ff881400640 1 --2- 172.21.15.28:0/2941337100 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff87c1ea380 0x7ff87c146480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:45.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.501+0000 7ff87be00640 1 -- 172.21.15.28:0/2941337100 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff87c1ea380 msgr2=0x7ff87c146480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:45.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.501+0000 7ff87be00640 1 --2- 172.21.15.28:0/2941337100 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff87c1ea380 0x7ff87c146480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:45.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.501+0000 7ff87be00640 1 -- 172.21.15.28:0/2941337100 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff864004800 msgr2=0x7ff87c0a5db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:45.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.501+0000 7ff87be00640 1 --2- 172.21.15.28:0/2941337100 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff864004800 0x7ff87c0a5db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:45.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.501+0000 7ff87be00640 1 -- 172.21.15.28:0/2941337100 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff87c0abfd0 con 0x7ff87c149e70 2024-02-21T23:10:45.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.501+0000 7ff880a00640 1 --2- 172.21.15.28:0/2941337100 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff864004800 0x7ff87c0a5db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:10:45.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.502+0000 7ff87be00640 1 --2- 172.21.15.28:0/2941337100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff87c149e70 0x7ff87c1e9e40 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7ff868009bb0 tx=0x7ff868004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:45.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.502+0000 7ff878c00640 1 -- 172.21.15.28:0/2941337100 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff868042020 con 0x7ff87c149e70 2024-02-21T23:10:45.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.502+0000 7ff878c00640 1 -- 172.21.15.28:0/2941337100 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff86803a040 con 0x7ff87c149e70 2024-02-21T23:10:45.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.502+0000 7ff878c00640 1 -- 172.21.15.28:0/2941337100 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff868039070 con 0x7ff87c149e70 2024-02-21T23:10:45.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.502+0000 7ff881e00640 1 -- 172.21.15.28:0/2941337100 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff87c0a9780 con 0x7ff87c149e70 2024-02-21T23:10:45.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.503+0000 7ff881e00640 1 -- 172.21.15.28:0/2941337100 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff87c0a82b0 con 0x7ff87c149e70 2024-02-21T23:10:45.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.504+0000 7ff881e00640 1 -- 172.21.15.28:0/2941337100 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff87c04f290 con 0x7ff87c149e70 2024-02-21T23:10:45.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.505+0000 7ff878c00640 1 -- 172.21.15.28:0/2941337100 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff868045020 con 0x7ff87c149e70 2024-02-21T23:10:45.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.506+0000 7ff878c00640 1 --2- 172.21.15.28:0/2941337100 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff85c06b500 0x7ff85c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:45.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.506+0000 7ff878c00640 1 -- 172.21.15.28:0/2941337100 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 5679+0+0 (secure 0 0 0) 0x7ff8680ab8f0 con 0x7ff87c149e70 2024-02-21T23:10:45.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.506+0000 7ff880a00640 1 --2- 172.21.15.28:0/2941337100 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff85c06b500 0x7ff85c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:45.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.507+0000 7ff880a00640 1 --2- 172.21.15.28:0/2941337100 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff85c06b500 0x7ff85c06d9c0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7ff870001200 tx=0x7ff870009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:45.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.510+0000 7ff878c00640 1 -- 172.21.15.28:0/2941337100 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ff86804c030 con 0x7ff87c149e70 2024-02-21T23:10:45.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.710+0000 7ff881e00640 1 -- 172.21.15.28:0/2941337100 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff87c06f0e0 con 0x7ff87c149e70 2024-02-21T23:10:45.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.711+0000 7ff878c00640 1 -- 172.21.15.28:0/2941337100 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v166) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff86807a8b0 con 0x7ff87c149e70 2024-02-21T23:10:45.700 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-02-21T23:10:45.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.713+0000 7ff881e00640 1 -- 172.21.15.28:0/2941337100 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff85c06b500 msgr2=0x7ff85c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:45.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.713+0000 7ff881e00640 1 --2- 172.21.15.28:0/2941337100 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff85c06b500 0x7ff85c06d9c0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7ff870001200 tx=0x7ff870009040 comp rx=0 tx=0).stop 2024-02-21T23:10:45.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.714+0000 7ff881e00640 1 -- 172.21.15.28:0/2941337100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff87c149e70 msgr2=0x7ff87c1e9e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:45.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.714+0000 7ff881e00640 1 --2- 172.21.15.28:0/2941337100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff87c149e70 0x7ff87c1e9e40 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7ff868009bb0 tx=0x7ff868004730 comp rx=0 tx=0).stop 2024-02-21T23:10:45.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.714+0000 7ff881e00640 1 -- 172.21.15.28:0/2941337100 shutdown_connections 2024-02-21T23:10:45.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.714+0000 7ff881e00640 1 --2- 172.21.15.28:0/2941337100 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff85c06b500 0x7ff85c06d9c0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:45.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.714+0000 7ff881e00640 1 --2- 172.21.15.28:0/2941337100 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff87c1ea380 0x7ff87c146480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:45.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.714+0000 7ff881e00640 1 --2- 172.21.15.28:0/2941337100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff87c149e70 0x7ff87c1e9e40 unknown :-1 s=CLOSED pgs=1881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:45.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.714+0000 7ff881e00640 1 --2- 172.21.15.28:0/2941337100 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff864004800 0x7ff87c0a5db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:45.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.714+0000 7ff881e00640 1 -- 172.21.15.28:0/2941337100 >> 172.21.15.28:0/2941337100 conn(0x7ff87c0a8530 msgr2=0x7ff87c06ae70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:45.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.714+0000 7ff881e00640 1 -- 172.21.15.28:0/2941337100 shutdown_connections 2024-02-21T23:10:45.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.714+0000 7ff881e00640 1 -- 172.21.15.28:0/2941337100 wait complete. 2024-02-21T23:10:45.723 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestClusterResize.test_down' 2024-02-21T23:10:45.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.954+0000 7f7a68a00640 1 Processor -- start 2024-02-21T23:10:45.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.954+0000 7f7a68a00640 1 -- start start 2024-02-21T23:10:45.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.955+0000 7f7a68a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7a64150230 0x7f7a641536d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:45.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.955+0000 7f7a68a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f7a6406f0c0 0x7f7a6406f520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:45.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.955+0000 7f7a68a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a640c3760 con 0x7f7a6406fa60 2024-02-21T23:10:45.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.955+0000 7f7a68a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a64070350 con 0x7f7a6406f0c0 2024-02-21T23:10:45.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.955+0000 7f7a68a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a640c31a0 con 0x7f7a64150230 2024-02-21T23:10:45.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.956+0000 7f7a63400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7a64150230 0x7f7a641536d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:45.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.956+0000 7f7a62a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f7a6406f0c0 0x7f7a6406f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:45.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.956+0000 7f7a63e00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f7a6406fa60 0x7f7a640c3f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.28:44134/0 (socket says 172.21.15.28:44134) 2024-02-21T23:10:45.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.956+0000 7f7a62a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f7a6406f0c0 0x7f7a6406f520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.28:36066/0 (socket says 172.21.15.28:36066) 2024-02-21T23:10:45.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.956+0000 7f7a63e00640 1 -- 172.21.15.28:0/2301526555 learned_addr learned my addr 172.21.15.28:0/2301526555 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:10:45.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.956+0000 7f7a63400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7a64150230 0x7f7a641536d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3301/0 says I am v2:172.21.15.28:34290/0 (socket says 172.21.15.28:34290) 2024-02-21T23:10:45.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.957+0000 7f7a63400640 1 -- 172.21.15.28:0/2301526555 >> v2:172.21.15.81:3300/0 conn(0x7f7a6406f0c0 msgr2=0x7f7a6406f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:45.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.957+0000 7f7a63400640 1 --2- 172.21.15.28:0/2301526555 >> v2:172.21.15.81:3300/0 conn(0x7f7a6406f0c0 0x7f7a6406f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:45.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.957+0000 7f7a63400640 1 -- 172.21.15.28:0/2301526555 >> v1:172.21.15.81:6789/0 conn(0x7f7a6406fa60 legacy=0x7f7a640c3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:45.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.957+0000 7f7a63400640 1 -- 172.21.15.28:0/2301526555 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a640c3480 con 0x7f7a64150230 2024-02-21T23:10:45.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.958+0000 7f7a63400640 1 --2- 172.21.15.28:0/2301526555 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7a64150230 0x7f7a641536d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7a540097a0 tx=0x7f7a5402b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d8c528b38cf59ea server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:45.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.959+0000 7f7a62000640 1 -- 172.21.15.28:0/2301526555 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a540315a0 con 0x7f7a64150230 2024-02-21T23:10:45.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.959+0000 7f7a62000640 1 -- 172.21.15.28:0/2301526555 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a54037070 con 0x7f7a64150230 2024-02-21T23:10:45.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.959+0000 7f7a62000640 1 -- 172.21.15.28:0/2301526555 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a54031960 con 0x7f7a64150230 2024-02-21T23:10:45.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.959+0000 7f7a68a00640 1 -- 172.21.15.28:0/2301526555 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7a64150230 msgr2=0x7f7a641536d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:45.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.959+0000 7f7a68a00640 1 --2- 172.21.15.28:0/2301526555 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7a64150230 0x7f7a641536d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7a540097a0 tx=0x7f7a5402b840 comp rx=0 tx=0).stop 2024-02-21T23:10:45.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.960+0000 7f7a68a00640 1 -- 172.21.15.28:0/2301526555 shutdown_connections 2024-02-21T23:10:45.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.960+0000 7f7a68a00640 1 --2- 172.21.15.28:0/2301526555 >> v2:172.21.15.81:3300/0 conn(0x7f7a6406f0c0 0x7f7a6406f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:45.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.960+0000 7f7a68a00640 1 --2- 172.21.15.28:0/2301526555 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7a64150230 0x7f7a641536d0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:45.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.960+0000 7f7a68a00640 1 -- 172.21.15.28:0/2301526555 >> 172.21.15.28:0/2301526555 conn(0x7f7a640aa3d0 msgr2=0x7f7a640aa7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:45.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.960+0000 7f7a68a00640 1 -- 172.21.15.28:0/2301526555 shutdown_connections 2024-02-21T23:10:45.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.960+0000 7f7a68a00640 1 -- 172.21.15.28:0/2301526555 wait complete. 2024-02-21T23:10:45.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.961+0000 7f7a68a00640 1 Processor -- start 2024-02-21T23:10:45.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.962+0000 7f7a68a00640 1 -- start start 2024-02-21T23:10:45.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.963+0000 7f7a68a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a6406f0c0 0x7f7a6406bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:45.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.963+0000 7f7a68a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a6406fa60 0x7f7a640c1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:45.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.963+0000 7f7a68a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7a64150230 0x7f7a641eb140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:45.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.963+0000 7f7a68a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a64071a40 con 0x7f7a6406fa60 2024-02-21T23:10:45.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.963+0000 7f7a68a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a64150610 con 0x7f7a6406f0c0 2024-02-21T23:10:45.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.963+0000 7f7a68a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a64071bc0 con 0x7f7a64150230 2024-02-21T23:10:45.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.963+0000 7f7a62a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a6406fa60 0x7f7a640c1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:45.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.963+0000 7f7a62a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a6406fa60 0x7f7a640c1cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49130/0 (socket says 172.21.15.28:49130) 2024-02-21T23:10:45.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.963+0000 7f7a62a00640 1 -- 172.21.15.28:0/2880835400 learned_addr learned my addr 172.21.15.28:0/2880835400 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:45.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.963+0000 7f7a63e00640 1 --2- 172.21.15.28:0/2880835400 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7a64150230 0x7f7a641eb140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:45.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.963+0000 7f7a63400640 1 --2- 172.21.15.28:0/2880835400 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a6406f0c0 0x7f7a6406bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:45.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.963+0000 7f7a62a00640 1 -- 172.21.15.28:0/2880835400 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7a64150230 msgr2=0x7f7a641eb140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:45.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.963+0000 7f7a62a00640 1 --2- 172.21.15.28:0/2880835400 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7a64150230 0x7f7a641eb140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:45.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.963+0000 7f7a62a00640 1 -- 172.21.15.28:0/2880835400 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a6406f0c0 msgr2=0x7f7a6406bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:45.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.963+0000 7f7a62a00640 1 --2- 172.21.15.28:0/2880835400 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a6406f0c0 0x7f7a6406bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:45.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.963+0000 7f7a62a00640 1 -- 172.21.15.28:0/2880835400 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a641ea960 con 0x7f7a6406fa60 2024-02-21T23:10:45.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.963+0000 7f7a63400640 1 --2- 172.21.15.28:0/2880835400 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a6406f0c0 0x7f7a6406bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:10:45.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.964+0000 7f7a62a00640 1 --2- 172.21.15.28:0/2880835400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a6406fa60 0x7f7a640c1cc0 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7f7a5000e9b0 tx=0x7f7a5000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:45.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.964+0000 7f7a5b400640 1 -- 172.21.15.28:0/2880835400 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a50017070 con 0x7f7a6406fa60 2024-02-21T23:10:45.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.964+0000 7f7a5b400640 1 -- 172.21.15.28:0/2880835400 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a5000c9a0 con 0x7f7a6406fa60 2024-02-21T23:10:45.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.964+0000 7f7a5b400640 1 -- 172.21.15.28:0/2880835400 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a5000cd90 con 0x7f7a6406fa60 2024-02-21T23:10:45.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.964+0000 7f7a68a00640 1 -- 172.21.15.28:0/2880835400 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a6406db70 con 0x7f7a6406fa60 2024-02-21T23:10:45.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.964+0000 7f7a68a00640 1 -- 172.21.15.28:0/2880835400 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a641eac20 con 0x7f7a6406fa60 2024-02-21T23:10:45.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.965+0000 7f7a68a00640 1 -- 172.21.15.28:0/2880835400 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a6404f290 con 0x7f7a6406fa60 2024-02-21T23:10:45.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.966+0000 7f7a5b400640 1 -- 172.21.15.28:0/2880835400 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7a5001a070 con 0x7f7a6406fa60 2024-02-21T23:10:45.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.968+0000 7f7a5b400640 1 --2- 172.21.15.28:0/2880835400 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7a4406b4d0 0x7f7a4406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:45.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.968+0000 7f7a5b400640 1 -- 172.21.15.28:0/2880835400 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 5679+0+0 (secure 0 0 0) 0x7f7a5008a2c0 con 0x7f7a6406fa60 2024-02-21T23:10:45.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.969+0000 7f7a63400640 1 --2- 172.21.15.28:0/2880835400 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7a4406b4d0 0x7f7a4406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:45.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.970+0000 7f7a63400640 1 --2- 172.21.15.28:0/2880835400 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7a4406b4d0 0x7f7a4406d990 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f7a54036000 tx=0x7f7a54008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:45.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:45.973+0000 7f7a5b400640 1 -- 172.21.15.28:0/2880835400 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7a500590c0 con 0x7f7a6406fa60 2024-02-21T23:10:46.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:46.247+0000 7f7a68a00640 1 -- 172.21.15.28:0/2880835400 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterResize.test_down"]} v 0) v1 -- 0x7f7a640c7410 con 0x7f7a6406fa60 2024-02-21T23:10:46.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:46.785+0000 7f7a5b400640 1 -- 172.21.15.28:0/2880835400 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterResize.test_down"]}]=0 v472) v1 ==== 136+0+0 (secure 0 0 0) 0x7f7a500589c0 con 0x7f7a6406fa60 2024-02-21T23:10:46.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:46.789+0000 7f7a68a00640 1 -- 172.21.15.28:0/2880835400 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7a4406b4d0 msgr2=0x7f7a4406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:46.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:46.789+0000 7f7a68a00640 1 --2- 172.21.15.28:0/2880835400 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7a4406b4d0 0x7f7a4406d990 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f7a54036000 tx=0x7f7a54008030 comp rx=0 tx=0).stop 2024-02-21T23:10:46.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:46.789+0000 7f7a68a00640 1 -- 172.21.15.28:0/2880835400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a6406fa60 msgr2=0x7f7a640c1cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:46.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:46.789+0000 7f7a68a00640 1 --2- 172.21.15.28:0/2880835400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a6406fa60 0x7f7a640c1cc0 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7f7a5000e9b0 tx=0x7f7a5000ee70 comp rx=0 tx=0).stop 2024-02-21T23:10:46.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:46.789+0000 7f7a68a00640 1 -- 172.21.15.28:0/2880835400 shutdown_connections 2024-02-21T23:10:46.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:46.789+0000 7f7a68a00640 1 --2- 172.21.15.28:0/2880835400 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7a4406b4d0 0x7f7a4406d990 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:46.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:46.789+0000 7f7a68a00640 1 --2- 172.21.15.28:0/2880835400 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7a64150230 0x7f7a641eb140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:46.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:46.789+0000 7f7a68a00640 1 --2- 172.21.15.28:0/2880835400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a6406fa60 0x7f7a640c1cc0 unknown :-1 s=CLOSED pgs=1882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:46.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:46.789+0000 7f7a68a00640 1 --2- 172.21.15.28:0/2880835400 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a6406f0c0 0x7f7a6406bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:46.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:46.789+0000 7f7a68a00640 1 -- 172.21.15.28:0/2880835400 >> 172.21.15.28:0/2880835400 conn(0x7f7a640aa3d0 msgr2=0x7f7a64150820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:46.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:46.789+0000 7f7a68a00640 1 -- 172.21.15.28:0/2880835400 shutdown_connections 2024-02-21T23:10:46.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:46.789+0000 7f7a68a00640 1 -- 172.21.15.28:0/2880835400 wait complete. 2024-02-21T23:10:46.800 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-02-21T23:10:47.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.033+0000 7fe317200640 1 Processor -- start 2024-02-21T23:10:47.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.033+0000 7fe317200640 1 -- start start 2024-02-21T23:10:47.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.034+0000 7fe317200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe318150240 0x7fe3180c12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:47.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.034+0000 7fe317200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fe3180c2c10 0x7fe31806ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:47.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.034+0000 7fe317200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe31806e220 con 0x7fe3180c4580 2024-02-21T23:10:47.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.034+0000 7fe317200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe318150620 con 0x7fe318150240 2024-02-21T23:10:47.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.034+0000 7fe317200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe318153a90 con 0x7fe3180c2c10 2024-02-21T23:10:47.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.035+0000 7fe315e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe318150240 0x7fe3180c12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:47.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.035+0000 7fe315e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe318150240 0x7fe3180c12c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49140/0 (socket says 172.21.15.28:49140) 2024-02-21T23:10:47.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.035+0000 7fe315e00640 1 -- 172.21.15.28:0/129018843 learned_addr learned my addr 172.21.15.28:0/129018843 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:47.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.035+0000 7fe315e00640 1 -- 172.21.15.28:0/129018843 >> v2:172.21.15.81:3300/0 conn(0x7fe3180c2c10 msgr2=0x7fe31806ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:47.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.035+0000 7fe316800640 1 --2- 172.21.15.28:0/129018843 >> v2:172.21.15.81:3300/0 conn(0x7fe3180c2c10 0x7fe31806ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:47.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.035+0000 7fe315e00640 1 --2- 172.21.15.28:0/129018843 >> v2:172.21.15.81:3300/0 conn(0x7fe3180c2c10 0x7fe31806ae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:47.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.035+0000 7fe315e00640 1 -- 172.21.15.28:0/129018843 >> v1:172.21.15.81:6789/0 conn(0x7fe3180c4580 legacy=0x7fe31806db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:47.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.035+0000 7fe315e00640 1 -- 172.21.15.28:0/129018843 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe31806f780 con 0x7fe318150240 2024-02-21T23:10:47.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.036+0000 7fe315e00640 1 --2- 172.21.15.28:0/129018843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe318150240 0x7fe3180c12c0 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7fe31806d140 tx=0x7fe30802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdc272277aa29977 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:47.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.037+0000 7fe314a00640 1 -- 172.21.15.28:0/129018843 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe30802e070 con 0x7fe318150240 2024-02-21T23:10:47.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.037+0000 7fe314a00640 1 -- 172.21.15.28:0/129018843 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe308037070 con 0x7fe318150240 2024-02-21T23:10:47.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.037+0000 7fe314a00640 1 -- 172.21.15.28:0/129018843 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe308031a00 con 0x7fe318150240 2024-02-21T23:10:47.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.038+0000 7fe317200640 1 -- 172.21.15.28:0/129018843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe318150240 msgr2=0x7fe3180c12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:47.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.038+0000 7fe317200640 1 --2- 172.21.15.28:0/129018843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe318150240 0x7fe3180c12c0 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7fe31806d140 tx=0x7fe30802b880 comp rx=0 tx=0).stop 2024-02-21T23:10:47.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.038+0000 7fe317200640 1 -- 172.21.15.28:0/129018843 shutdown_connections 2024-02-21T23:10:47.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.038+0000 7fe317200640 1 --2- 172.21.15.28:0/129018843 >> v2:172.21.15.81:3300/0 conn(0x7fe3180c2c10 0x7fe31806ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:47.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.038+0000 7fe317200640 1 --2- 172.21.15.28:0/129018843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe318150240 0x7fe3180c12c0 unknown :-1 s=CLOSED pgs=1883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:47.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.038+0000 7fe317200640 1 -- 172.21.15.28:0/129018843 >> 172.21.15.28:0/129018843 conn(0x7fe3180a8530 msgr2=0x7fe3180a8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:47.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.038+0000 7fe317200640 1 -- 172.21.15.28:0/129018843 shutdown_connections 2024-02-21T23:10:47.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.038+0000 7fe317200640 1 -- 172.21.15.28:0/129018843 wait complete. 2024-02-21T23:10:47.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.040+0000 7fe317200640 1 Processor -- start 2024-02-21T23:10:47.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.040+0000 7fe317200640 1 -- start start 2024-02-21T23:10:47.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.041+0000 7fe317200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3180c2c10 0x7fe3181473f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:47.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.041+0000 7fe317200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3180c4580 0x7fe3180a66f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:47.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.041+0000 7fe317200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe318150240 0x7fe3180a6c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:47.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.041+0000 7fe317200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3180c4a10 con 0x7fe3180c4580 2024-02-21T23:10:47.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.041+0000 7fe317200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe318153720 con 0x7fe3180c2c10 2024-02-21T23:10:47.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.041+0000 7fe317200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3180c4b90 con 0x7fe318150240 2024-02-21T23:10:47.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.041+0000 7fe315400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3180c4580 0x7fe3180a66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:47.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.041+0000 7fe315400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3180c4580 0x7fe3180a66f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49156/0 (socket says 172.21.15.28:49156) 2024-02-21T23:10:47.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.041+0000 7fe315400640 1 -- 172.21.15.28:0/4081536371 learned_addr learned my addr 172.21.15.28:0/4081536371 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:47.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.041+0000 7fe315e00640 1 --2- 172.21.15.28:0/4081536371 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3180c2c10 0x7fe3181473f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:47.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.041+0000 7fe316800640 1 --2- 172.21.15.28:0/4081536371 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe318150240 0x7fe3180a6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:47.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.041+0000 7fe315400640 1 -- 172.21.15.28:0/4081536371 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe318150240 msgr2=0x7fe3180a6c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:47.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.041+0000 7fe315400640 1 --2- 172.21.15.28:0/4081536371 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe318150240 0x7fe3180a6c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:47.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.041+0000 7fe315400640 1 -- 172.21.15.28:0/4081536371 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3180c2c10 msgr2=0x7fe3181473f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:47.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.041+0000 7fe315400640 1 --2- 172.21.15.28:0/4081536371 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3180c2c10 0x7fe3181473f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:47.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.042+0000 7fe315400640 1 -- 172.21.15.28:0/4081536371 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe318153e70 con 0x7fe3180c4580 2024-02-21T23:10:47.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.042+0000 7fe316800640 1 --2- 172.21.15.28:0/4081536371 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe318150240 0x7fe3180a6c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:10:47.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.042+0000 7fe315400640 1 --2- 172.21.15.28:0/4081536371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3180c4580 0x7fe3180a66f0 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7fe30000c430 tx=0x7fe30000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:47.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.042+0000 7fe30e000640 1 -- 172.21.15.28:0/4081536371 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe300017070 con 0x7fe3180c4580 2024-02-21T23:10:47.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.042+0000 7fe30e000640 1 -- 172.21.15.28:0/4081536371 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3000035f0 con 0x7fe3180c4580 2024-02-21T23:10:47.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.042+0000 7fe30e000640 1 -- 172.21.15.28:0/4081536371 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3000039e0 con 0x7fe3180c4580 2024-02-21T23:10:47.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.042+0000 7fe317200640 1 -- 172.21.15.28:0/4081536371 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe318154150 con 0x7fe3180c4580 2024-02-21T23:10:47.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.042+0000 7fe317200640 1 -- 172.21.15.28:0/4081536371 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe318156e50 con 0x7fe3180c4580 2024-02-21T23:10:47.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.043+0000 7fe317200640 1 -- 172.21.15.28:0/4081536371 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe318154350 con 0x7fe3180c4580 2024-02-21T23:10:47.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.044+0000 7fe30e000640 1 -- 172.21.15.28:0/4081536371 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe300003b80 con 0x7fe3180c4580 2024-02-21T23:10:47.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.045+0000 7fe30e000640 1 --2- 172.21.15.28:0/4081536371 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe2f406b500 0x7fe2f406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:47.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.045+0000 7fe30e000640 1 -- 172.21.15.28:0/4081536371 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 5679+0+0 (secure 0 0 0) 0x7fe300089630 con 0x7fe3180c4580 2024-02-21T23:10:47.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.046+0000 7fe315e00640 1 --2- 172.21.15.28:0/4081536371 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe2f406b500 0x7fe2f406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:47.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.047+0000 7fe315e00640 1 --2- 172.21.15.28:0/4081536371 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe2f406b500 0x7fe2f406d9c0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fe308036000 tx=0x7fe30803d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:47.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.050+0000 7fe30e000640 1 -- 172.21.15.28:0/4081536371 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe300058430 con 0x7fe3180c4580 2024-02-21T23:10:47.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.251+0000 7fe317200640 1 -- 172.21.15.28:0/4081536371 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fe31804f290 con 0x7fe3180c4580 2024-02-21T23:10:47.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.256+0000 7fe30e000640 1 -- 172.21.15.28:0/4081536371 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v57)=0 v57) v1 ==== 122+0+0 (secure 0 0 0) 0x7fe300057d30 con 0x7fe3180c4580 2024-02-21T23:10:47.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.259+0000 7fe317200640 1 -- 172.21.15.28:0/4081536371 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe2f406b500 msgr2=0x7fe2f406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:47.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.259+0000 7fe317200640 1 --2- 172.21.15.28:0/4081536371 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe2f406b500 0x7fe2f406d9c0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fe308036000 tx=0x7fe30803d040 comp rx=0 tx=0).stop 2024-02-21T23:10:47.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.259+0000 7fe317200640 1 -- 172.21.15.28:0/4081536371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3180c4580 msgr2=0x7fe3180a66f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:47.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.259+0000 7fe317200640 1 --2- 172.21.15.28:0/4081536371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3180c4580 0x7fe3180a66f0 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7fe30000c430 tx=0x7fe30000c8f0 comp rx=0 tx=0).stop 2024-02-21T23:10:47.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.259+0000 7fe317200640 1 -- 172.21.15.28:0/4081536371 shutdown_connections 2024-02-21T23:10:47.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.259+0000 7fe317200640 1 --2- 172.21.15.28:0/4081536371 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe2f406b500 0x7fe2f406d9c0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:47.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.259+0000 7fe317200640 1 --2- 172.21.15.28:0/4081536371 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe318150240 0x7fe3180a6c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:47.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.259+0000 7fe317200640 1 --2- 172.21.15.28:0/4081536371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3180c4580 0x7fe3180a66f0 unknown :-1 s=CLOSED pgs=1884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:47.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.259+0000 7fe317200640 1 --2- 172.21.15.28:0/4081536371 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3180c2c10 0x7fe3181473f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:47.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.259+0000 7fe317200640 1 -- 172.21.15.28:0/4081536371 >> 172.21.15.28:0/4081536371 conn(0x7fe3180a8530 msgr2=0x7fe31814da60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:47.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.259+0000 7fe317200640 1 -- 172.21.15.28:0/4081536371 shutdown_connections 2024-02-21T23:10:47.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.259+0000 7fe317200640 1 -- 172.21.15.28:0/4081536371 wait complete. 2024-02-21T23:10:47.269 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-02-21T23:10:47.269 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-21T23:10:47.385 DEBUG:teuthology.orchestra.run.smithi081:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-21T23:10:47.503 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-02-21T23:10:47.504 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-21T23:10:47.504 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-21T23:10:47.563 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054608,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-02-21T23:10:47.563 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-02-21T23:10:47.563 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-21T23:10:47.619 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053833,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498812k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249404k,nr_inodes=812351,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498812k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-02-21T23:10:47.619 INFO:tasks.cephfs.cephfs_test_case:[, ] 2024-02-21T23:10:47.619 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:10:47.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.863+0000 7f324fe00640 1 Processor -- start 2024-02-21T23:10:47.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.863+0000 7f324fe00640 1 -- start start 2024-02-21T23:10:47.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.865+0000 7f324fe00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f325006ee40 0x7f32500a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:47.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.865+0000 7f324fe00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3250156af0 0x7f325006dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:47.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.865+0000 7f324fe00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32500c2760 con 0x7f3250150450 2024-02-21T23:10:47.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.865+0000 7f324fe00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32500c22c0 con 0x7f325006ee40 2024-02-21T23:10:47.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.865+0000 7f324fe00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3250156660 con 0x7f3250156af0 2024-02-21T23:10:47.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.866+0000 7f324e000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f325006ee40 0x7f32500a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:47.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.866+0000 7f324f400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3250156af0 0x7f325006dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:47.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.866+0000 7f324ea00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f3250150450 0x7f32501536f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.28:44154/0 (socket says 172.21.15.28:44154) 2024-02-21T23:10:47.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.866+0000 7f324ea00640 1 -- 172.21.15.28:0/3431035905 learned_addr learned my addr 172.21.15.28:0/3431035905 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:10:47.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.867+0000 7f324f400640 1 -- 172.21.15.28:0/3431035905 >> v2:172.21.15.81:3300/0 conn(0x7f325006ee40 msgr2=0x7f32500a5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:47.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.867+0000 7f324f400640 1 --2- 172.21.15.28:0/3431035905 >> v2:172.21.15.81:3300/0 conn(0x7f325006ee40 0x7f32500a5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:47.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.867+0000 7f324f400640 1 -- 172.21.15.28:0/3431035905 >> v1:172.21.15.81:6789/0 conn(0x7f3250150450 legacy=0x7f32501536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:47.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.867+0000 7f324f400640 1 -- 172.21.15.28:0/3431035905 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3250156310 con 0x7f3250156af0 2024-02-21T23:10:47.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.867+0000 7f324e000640 1 --2- 172.21.15.28:0/3431035905 >> v2:172.21.15.81:3300/0 conn(0x7f325006ee40 0x7f32500a5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-21T23:10:47.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.868+0000 7f324f400640 1 --2- 172.21.15.28:0/3431035905 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3250156af0 0x7f325006dff0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3248009d90 tx=0x7f324802beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af2be579192000d9 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:47.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.869+0000 7f324d600640 1 -- 172.21.15.28:0/3431035905 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3248039070 con 0x7f3250156af0 2024-02-21T23:10:47.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.869+0000 7f324d600640 1 -- 172.21.15.28:0/3431035905 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f324802cc60 con 0x7f3250156af0 2024-02-21T23:10:47.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.869+0000 7f324d600640 1 -- 172.21.15.28:0/3431035905 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3248034660 con 0x7f3250156af0 2024-02-21T23:10:47.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.869+0000 7f324fe00640 1 -- 172.21.15.28:0/3431035905 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3250156af0 msgr2=0x7f325006dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:47.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.869+0000 7f324fe00640 1 --2- 172.21.15.28:0/3431035905 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3250156af0 0x7f325006dff0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3248009d90 tx=0x7f324802beb0 comp rx=0 tx=0).stop 2024-02-21T23:10:47.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.869+0000 7f324fe00640 1 -- 172.21.15.28:0/3431035905 shutdown_connections 2024-02-21T23:10:47.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.869+0000 7f324fe00640 1 --2- 172.21.15.28:0/3431035905 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3250156af0 0x7f325006dff0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:47.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.869+0000 7f324fe00640 1 --2- 172.21.15.28:0/3431035905 >> v2:172.21.15.81:3300/0 conn(0x7f325006ee40 0x7f32500a5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:47.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.870+0000 7f324fe00640 1 -- 172.21.15.28:0/3431035905 >> 172.21.15.28:0/3431035905 conn(0x7f325006b630 msgr2=0x7f32500c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:47.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.870+0000 7f324fe00640 1 -- 172.21.15.28:0/3431035905 shutdown_connections 2024-02-21T23:10:47.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.870+0000 7f324fe00640 1 -- 172.21.15.28:0/3431035905 wait complete. 2024-02-21T23:10:47.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.871+0000 7f324fe00640 1 Processor -- start 2024-02-21T23:10:47.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.871+0000 7f324fe00640 1 -- start start 2024-02-21T23:10:47.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.872+0000 7f324fe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f325006ee40 0x7f32501ebd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:47.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.872+0000 7f324fe00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3250150450 0x7f32500c01f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:47.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.872+0000 7f324fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3250156af0 0x7f32500c0730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:47.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.872+0000 7f324fe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3250156f80 con 0x7f3250156af0 2024-02-21T23:10:47.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.872+0000 7f324fe00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32500c2cb0 con 0x7f325006ee40 2024-02-21T23:10:47.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.872+0000 7f324fe00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3250157100 con 0x7f3250150450 2024-02-21T23:10:47.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.873+0000 7f324f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3250156af0 0x7f32500c0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:47.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.873+0000 7f324f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3250156af0 0x7f32500c0730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49164/0 (socket says 172.21.15.28:49164) 2024-02-21T23:10:47.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.873+0000 7f324f400640 1 -- 172.21.15.28:0/2514735913 learned_addr learned my addr 172.21.15.28:0/2514735913 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:47.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.873+0000 7f324ea00640 1 --2- 172.21.15.28:0/2514735913 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f325006ee40 0x7f32501ebd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:47.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.873+0000 7f324e000640 1 --2- 172.21.15.28:0/2514735913 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3250150450 0x7f32500c01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:47.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.873+0000 7f324f400640 1 -- 172.21.15.28:0/2514735913 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3250150450 msgr2=0x7f32500c01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:47.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.873+0000 7f324f400640 1 --2- 172.21.15.28:0/2514735913 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3250150450 0x7f32500c01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:47.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.873+0000 7f324f400640 1 -- 172.21.15.28:0/2514735913 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f325006ee40 msgr2=0x7f32501ebd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:47.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.873+0000 7f324f400640 1 --2- 172.21.15.28:0/2514735913 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f325006ee40 0x7f32501ebd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:47.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.873+0000 7f324f400640 1 -- 172.21.15.28:0/2514735913 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32501ec630 con 0x7f3250156af0 2024-02-21T23:10:47.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.873+0000 7f324e000640 1 --2- 172.21.15.28:0/2514735913 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3250150450 0x7f32500c01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:10:47.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.874+0000 7f324f400640 1 --2- 172.21.15.28:0/2514735913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3250156af0 0x7f32500c0730 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7f3248009eb0 tx=0x7f3248004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:47.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.874+0000 7f3246a00640 1 -- 172.21.15.28:0/2514735913 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3248039050 con 0x7f3250156af0 2024-02-21T23:10:47.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.874+0000 7f3246a00640 1 -- 172.21.15.28:0/2514735913 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3248045070 con 0x7f3250156af0 2024-02-21T23:10:47.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.874+0000 7f3246a00640 1 -- 172.21.15.28:0/2514735913 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3248009450 con 0x7f3250156af0 2024-02-21T23:10:47.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.874+0000 7f324fe00640 1 -- 172.21.15.28:0/2514735913 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32500a9b50 con 0x7f3250156af0 2024-02-21T23:10:47.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.874+0000 7f324fe00640 1 -- 172.21.15.28:0/2514735913 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f325014d410 con 0x7f3250156af0 2024-02-21T23:10:47.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.875+0000 7f324fe00640 1 -- 172.21.15.28:0/2514735913 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f325004f290 con 0x7f3250156af0 2024-02-21T23:10:47.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.876+0000 7f3246a00640 1 -- 172.21.15.28:0/2514735913 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f324803e850 con 0x7f3250156af0 2024-02-21T23:10:47.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.877+0000 7f3246a00640 1 --2- 172.21.15.28:0/2514735913 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f323006b4d0 0x7f323006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:47.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.877+0000 7f3246a00640 1 -- 172.21.15.28:0/2514735913 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 5679+0+0 (secure 0 0 0) 0x7f32480ad1f0 con 0x7f3250156af0 2024-02-21T23:10:47.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.878+0000 7f324ea00640 1 --2- 172.21.15.28:0/2514735913 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f323006b4d0 0x7f323006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:47.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.879+0000 7f324ea00640 1 --2- 172.21.15.28:0/2514735913 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f323006b4d0 0x7f323006d990 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f3240005880 tx=0x7f3240009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:47.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:47.882+0000 7f3246a00640 1 -- 172.21.15.28:0/2514735913 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f324807c1a0 con 0x7f3250156af0 2024-02-21T23:10:48.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.121+0000 7f324fe00640 1 -- 172.21.15.28:0/2514735913 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f32500c1f10 con 0x7f3250156af0 2024-02-21T23:10:48.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.122+0000 7f3246a00640 1 -- 172.21.15.28:0/2514735913 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 178 v178) v1 ==== 95+0+3701 (secure 0 0 0) 0x7f324807baa0 con 0x7f3250156af0 2024-02-21T23:10:48.111 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:10:48.111 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":178,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6599,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/704098282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":704098282},{"type":"v1","addr":"172.21.15.81:6837","nonce":704098282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":173},{"gid":7372,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1804589714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1804589714},{"type":"v1","addr":"172.21.15.81:6835","nonce":1804589714}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":178},{"gid":9141,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2895215997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2895215997},{"type":"v1","addr":"172.21.15.28:6835","nonce":2895215997}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":166},{"gid":9390,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/4031206977","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":4031206977},{"type":"v1","addr":"172.21.15.28:6833","nonce":4031206977}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":178}],"filesystems":[]} 2024-02-21T23:10:48.112 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 178 2024-02-21T23:10:48.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.125+0000 7f324fe00640 1 -- 172.21.15.28:0/2514735913 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f323006b4d0 msgr2=0x7f323006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:48.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.125+0000 7f324fe00640 1 --2- 172.21.15.28:0/2514735913 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f323006b4d0 0x7f323006d990 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f3240005880 tx=0x7f3240009040 comp rx=0 tx=0).stop 2024-02-21T23:10:48.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.125+0000 7f324fe00640 1 -- 172.21.15.28:0/2514735913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3250156af0 msgr2=0x7f32500c0730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:48.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.125+0000 7f324fe00640 1 --2- 172.21.15.28:0/2514735913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3250156af0 0x7f32500c0730 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7f3248009eb0 tx=0x7f3248004a40 comp rx=0 tx=0).stop 2024-02-21T23:10:48.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.125+0000 7f324fe00640 1 -- 172.21.15.28:0/2514735913 shutdown_connections 2024-02-21T23:10:48.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.125+0000 7f324fe00640 1 --2- 172.21.15.28:0/2514735913 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f323006b4d0 0x7f323006d990 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:48.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.125+0000 7f324fe00640 1 --2- 172.21.15.28:0/2514735913 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3250150450 0x7f32500c01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:48.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.125+0000 7f324fe00640 1 --2- 172.21.15.28:0/2514735913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3250156af0 0x7f32500c0730 unknown :-1 s=CLOSED pgs=1885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:48.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.125+0000 7f324fe00640 1 --2- 172.21.15.28:0/2514735913 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f325006ee40 0x7f32501ebd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:48.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.125+0000 7f324fe00640 1 -- 172.21.15.28:0/2514735913 >> 172.21.15.28:0/2514735913 conn(0x7f325006b630 msgr2=0x7f32500a7570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:48.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.125+0000 7f324fe00640 1 -- 172.21.15.28:0/2514735913 shutdown_connections 2024-02-21T23:10:48.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.126+0000 7f324fe00640 1 -- 172.21.15.28:0/2514735913 wait complete. 2024-02-21T23:10:48.132 INFO:tasks.ceph.mds.a:Restarting daemon 2024-02-21T23:10:48.133 INFO:tasks.ceph.mds.a:Stopping old one... 2024-02-21T23:10:48.133 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-02-21T23:10:48.133 INFO:teuthology.orchestra.run:waiting for 300 2024-02-21T23:10:48.133 INFO:tasks.ceph.mds.c:Restarting daemon 2024-02-21T23:10:48.133 INFO:tasks.ceph.mds.c:Stopping old one... 2024-02-21T23:10:48.133 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-02-21T23:10:48.133 INFO:teuthology.orchestra.run:waiting for 300 2024-02-21T23:10:48.134 INFO:tasks.ceph.mds.b:Restarting daemon 2024-02-21T23:10:48.134 INFO:tasks.ceph.mds.b:Stopping old one... 2024-02-21T23:10:48.134 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-02-21T23:10:48.134 INFO:teuthology.orchestra.run:waiting for 300 2024-02-21T23:10:48.134 INFO:tasks.ceph.mds.d:Restarting daemon 2024-02-21T23:10:48.134 INFO:tasks.ceph.mds.d:Stopping old one... 2024-02-21T23:10:48.134 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-02-21T23:10:48.134 INFO:teuthology.orchestra.run:waiting for 300 2024-02-21T23:10:48.151 INFO:tasks.ceph.mds.c:Stopped 2024-02-21T23:10:48.151 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-02-21T23:10:48.156 INFO:tasks.ceph.mds.d:Stopped 2024-02-21T23:10:48.156 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-02-21T23:10:48.161 INFO:tasks.ceph.mds.b:Stopped 2024-02-21T23:10:48.161 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-02-21T23:10:48.161 INFO:tasks.ceph.mds.a:Stopped 2024-02-21T23:10:48.162 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-02-21T23:10:48.208 INFO:tasks.ceph.mds.d:Started 2024-02-21T23:10:48.208 DEBUG:teuthology.parallel:result is None 2024-02-21T23:10:48.209 INFO:tasks.ceph.mds.c:Started 2024-02-21T23:10:48.209 DEBUG:teuthology.parallel:result is None 2024-02-21T23:10:48.214 INFO:tasks.ceph.mds.b:Started 2024-02-21T23:10:48.215 DEBUG:teuthology.parallel:result is None 2024-02-21T23:10:48.215 INFO:tasks.ceph.mds.a:Started 2024-02-21T23:10:48.215 DEBUG:teuthology.parallel:result is None 2024-02-21T23:10:48.216 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-02-21T23:10:48.369 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-02-21T23:10:48.373 INFO:tasks.ceph.mds.b.smithi081.stdout:starting mds.b at 2024-02-21T23:10:48.374 INFO:tasks.ceph.mds.d.smithi081.stdout:starting mds.d at 2024-02-21T23:10:48.378 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-02-21T23:10:48.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.422+0000 7f44f6000640 1 Processor -- start 2024-02-21T23:10:48.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.422+0000 7f44f6000640 1 -- start start 2024-02-21T23:10:48.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.423+0000 7f44f6000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f44f01524c0 0x7f44f0155a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:48.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.423+0000 7f44f6000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f44f00c85c0 0x7f44f00c70d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:48.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.423+0000 7f44f6000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44f006df50 con 0x7f44f00c5040 2024-02-21T23:10:48.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.423+0000 7f44f6000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44f00c5520 con 0x7f44f01524c0 2024-02-21T23:10:48.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.423+0000 7f44f6000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44f00c5800 con 0x7f44f00c85c0 2024-02-21T23:10:48.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.423+0000 7f44f4c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f44f01524c0 0x7f44f0155a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:48.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.423+0000 7f44f4c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f44f01524c0 0x7f44f0155a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49218/0 (socket says 172.21.15.28:49218) 2024-02-21T23:10:48.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.423+0000 7f44f4c00640 1 -- 172.21.15.28:0/1125276919 learned_addr learned my addr 172.21.15.28:0/1125276919 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:48.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.424+0000 7f44f5600640 1 --2- 172.21.15.28:0/1125276919 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f44f00c85c0 0x7f44f00c70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:48.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.424+0000 7f44f4c00640 1 -- 172.21.15.28:0/1125276919 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f44f00c85c0 msgr2=0x7f44f00c70d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:48.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.424+0000 7f44f4c00640 1 --2- 172.21.15.28:0/1125276919 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f44f00c85c0 0x7f44f00c70d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:48.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.424+0000 7f44f4c00640 1 -- 172.21.15.28:0/1125276919 >> v1:172.21.15.81:6789/0 conn(0x7f44f00c5040 legacy=0x7f44f006ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:48.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.424+0000 7f44f4c00640 1 -- 172.21.15.28:0/1125276919 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44f006c670 con 0x7f44f01524c0 2024-02-21T23:10:48.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.425+0000 7f44f4c00640 1 --2- 172.21.15.28:0/1125276919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44f01524c0 0x7f44f0155a10 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7f44e4009870 tx=0x7f44e402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57afe6d45bf8d778 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:48.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.425+0000 7f44ef400640 1 -- 172.21.15.28:0/1125276919 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e40317c0 con 0x7f44f01524c0 2024-02-21T23:10:48.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.425+0000 7f44ef400640 1 -- 172.21.15.28:0/1125276919 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44e4037070 con 0x7f44f01524c0 2024-02-21T23:10:48.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.425+0000 7f44ef400640 1 -- 172.21.15.28:0/1125276919 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e4031b80 con 0x7f44f01524c0 2024-02-21T23:10:48.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.426+0000 7f44f6000640 1 -- 172.21.15.28:0/1125276919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44f01524c0 msgr2=0x7f44f0155a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:48.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.426+0000 7f44f6000640 1 --2- 172.21.15.28:0/1125276919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44f01524c0 0x7f44f0155a10 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7f44e4009870 tx=0x7f44e402b860 comp rx=0 tx=0).stop 2024-02-21T23:10:48.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.426+0000 7f44f6000640 1 -- 172.21.15.28:0/1125276919 shutdown_connections 2024-02-21T23:10:48.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.426+0000 7f44f6000640 1 --2- 172.21.15.28:0/1125276919 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f44f00c85c0 0x7f44f00c70d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:48.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.426+0000 7f44f6000640 1 --2- 172.21.15.28:0/1125276919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44f01524c0 0x7f44f0155a10 unknown :-1 s=CLOSED pgs=1891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:48.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.426+0000 7f44f6000640 1 -- 172.21.15.28:0/1125276919 >> 172.21.15.28:0/1125276919 conn(0x7f44f00ac2b0 msgr2=0x7f44f00ac6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:48.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.426+0000 7f44f6000640 1 -- 172.21.15.28:0/1125276919 shutdown_connections 2024-02-21T23:10:48.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.426+0000 7f44f6000640 1 -- 172.21.15.28:0/1125276919 wait complete. 2024-02-21T23:10:48.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.428+0000 7f44f6000640 1 Processor -- start 2024-02-21T23:10:48.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.428+0000 7f44f6000640 1 -- start start 2024-02-21T23:10:48.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.429+0000 7f44f6000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44f00c5040 0x7f44f014ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:48.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.429+0000 7f44f6000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f44f00c85c0 0x7f44f014a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:48.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.429+0000 7f44f6000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44f01524c0 0x7f44f01ee060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:48.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.429+0000 7f44f6000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44f00c8a70 con 0x7f44f00c5040 2024-02-21T23:10:48.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.429+0000 7f44f6000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44f01528a0 con 0x7f44f01524c0 2024-02-21T23:10:48.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.429+0000 7f44f6000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44f00c7610 con 0x7f44f00c85c0 2024-02-21T23:10:48.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.429+0000 7f44f4c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44f00c5040 0x7f44f014ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:48.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.429+0000 7f44f4c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44f00c5040 0x7f44f014ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49230/0 (socket says 172.21.15.28:49230) 2024-02-21T23:10:48.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.429+0000 7f44f4c00640 1 -- 172.21.15.28:0/3447891247 learned_addr learned my addr 172.21.15.28:0/3447891247 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:48.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.429+0000 7f44f5600640 1 --2- 172.21.15.28:0/3447891247 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44f01524c0 0x7f44f01ee060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:48.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.429+0000 7f44efe00640 1 --2- 172.21.15.28:0/3447891247 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f44f00c85c0 0x7f44f014a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:48.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.430+0000 7f44f4c00640 1 -- 172.21.15.28:0/3447891247 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f44f00c85c0 msgr2=0x7f44f014a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:48.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.430+0000 7f44f4c00640 1 --2- 172.21.15.28:0/3447891247 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f44f00c85c0 0x7f44f014a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:48.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.430+0000 7f44f4c00640 1 -- 172.21.15.28:0/3447891247 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44f01524c0 msgr2=0x7f44f01ee060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:48.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.430+0000 7f44f4c00640 1 --2- 172.21.15.28:0/3447891247 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44f01524c0 0x7f44f01ee060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:48.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.430+0000 7f44f4c00640 1 -- 172.21.15.28:0/3447891247 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44f00c2800 con 0x7f44f00c5040 2024-02-21T23:10:48.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.430+0000 7f44f4c00640 1 --2- 172.21.15.28:0/3447891247 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44f00c5040 0x7f44f014ef90 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7f44e402b8a0 tx=0x7f44e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:48.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.430+0000 7f44ecc00640 1 -- 172.21.15.28:0/3447891247 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e4032860 con 0x7f44f00c5040 2024-02-21T23:10:48.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.431+0000 7f44ecc00640 1 -- 172.21.15.28:0/3447891247 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44e4032a20 con 0x7f44f00c5040 2024-02-21T23:10:48.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.431+0000 7f44ecc00640 1 -- 172.21.15.28:0/3447891247 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e4032e10 con 0x7f44f00c5040 2024-02-21T23:10:48.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.431+0000 7f44f6000640 1 -- 172.21.15.28:0/3447891247 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44f00c2c20 con 0x7f44f00c5040 2024-02-21T23:10:48.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.431+0000 7f44f6000640 1 -- 172.21.15.28:0/3447891247 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44f014cd10 con 0x7f44f00c5040 2024-02-21T23:10:48.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.432+0000 7f44f6000640 1 -- 172.21.15.28:0/3447891247 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44f004f290 con 0x7f44f00c5040 2024-02-21T23:10:48.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.433+0000 7f44ecc00640 1 -- 172.21.15.28:0/3447891247 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f44e404c060 con 0x7f44f00c5040 2024-02-21T23:10:48.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.434+0000 7f44ecc00640 1 --2- 172.21.15.28:0/3447891247 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f44d006b4d0 0x7f44d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:48.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.434+0000 7f44ecc00640 1 -- 172.21.15.28:0/3447891247 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 5679+0+0 (secure 0 0 0) 0x7f44e40abc40 con 0x7f44f00c5040 2024-02-21T23:10:48.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.435+0000 7f44efe00640 1 --2- 172.21.15.28:0/3447891247 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f44d006b4d0 0x7f44d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:48.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.435+0000 7f44efe00640 1 --2- 172.21.15.28:0/3447891247 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f44d006b4d0 0x7f44d006d990 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f44f01e84d0 tx=0x7f44dc009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:48.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.438+0000 7f44ecc00640 1 -- 172.21.15.28:0/3447891247 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f44e4040070 con 0x7f44f00c5040 2024-02-21T23:10:48.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.653+0000 7f44f6000640 1 -- 172.21.15.28:0/3447891247 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f44f006dcc0 con 0x7f44f00c5040 2024-02-21T23:10:48.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.817+0000 7f44ecc00640 1 -- 172.21.15.28:0/3447891247 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v168) v1 ==== 97+0+0 (secure 0 0 0) 0x7f44e4031e90 con 0x7f44f00c5040 2024-02-21T23:10:48.807 INFO:teuthology.orchestra.run.smithi028.stderr: removed all blocklist entries 2024-02-21T23:10:48.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.821+0000 7f44f6000640 1 -- 172.21.15.28:0/3447891247 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f44d006b4d0 msgr2=0x7f44d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:48.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.821+0000 7f44f6000640 1 --2- 172.21.15.28:0/3447891247 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f44d006b4d0 0x7f44d006d990 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f44f01e84d0 tx=0x7f44dc009040 comp rx=0 tx=0).stop 2024-02-21T23:10:48.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.821+0000 7f44f6000640 1 -- 172.21.15.28:0/3447891247 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44f00c5040 msgr2=0x7f44f014ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:48.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.821+0000 7f44f6000640 1 --2- 172.21.15.28:0/3447891247 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44f00c5040 0x7f44f014ef90 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7f44e402b8a0 tx=0x7f44e40046c0 comp rx=0 tx=0).stop 2024-02-21T23:10:48.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.822+0000 7f44f6000640 1 -- 172.21.15.28:0/3447891247 shutdown_connections 2024-02-21T23:10:48.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.822+0000 7f44f6000640 1 --2- 172.21.15.28:0/3447891247 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f44d006b4d0 0x7f44d006d990 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:48.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.822+0000 7f44f6000640 1 --2- 172.21.15.28:0/3447891247 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f44f00c85c0 0x7f44f014a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:48.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.822+0000 7f44f6000640 1 --2- 172.21.15.28:0/3447891247 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44f01524c0 0x7f44f01ee060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:48.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.822+0000 7f44f6000640 1 --2- 172.21.15.28:0/3447891247 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44f00c5040 0x7f44f014ef90 unknown :-1 s=CLOSED pgs=1892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:48.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.822+0000 7f44f6000640 1 -- 172.21.15.28:0/3447891247 >> 172.21.15.28:0/3447891247 conn(0x7f44f00ac2b0 msgr2=0x7f44f00aa860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:48.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.822+0000 7f44f6000640 1 -- 172.21.15.28:0/3447891247 shutdown_connections 2024-02-21T23:10:48.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:48.822+0000 7f44f6000640 1 -- 172.21.15.28:0/3447891247 wait complete. 2024-02-21T23:10:48.833 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-02-21T23:10:49.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.052+0000 7f28e0a00640 1 Processor -- start 2024-02-21T23:10:49.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.052+0000 7f28e0a00640 1 -- start start 2024-02-21T23:10:49.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.053+0000 7f28e0a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f28dc0c2bb0 0x7f28dc152c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:49.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.053+0000 7f28e0a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f28dc0a5f80 0x7f28dc153190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:49.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.054+0000 7f28e0a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28dc0a6540 con 0x7f28dc150450 2024-02-21T23:10:49.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.054+0000 7f28e0a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28dc0c6df0 con 0x7f28dc0a5f80 2024-02-21T23:10:49.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.054+0000 7f28e0a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28dc0c26d0 con 0x7f28dc0c2bb0 2024-02-21T23:10:49.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.054+0000 7f28db400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f28dc150450 0x7f28dc06e010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41272/0 (socket says 172.21.15.28:41272) 2024-02-21T23:10:49.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.054+0000 7f28db400640 1 -- 172.21.15.28:0/2163823166 learned_addr learned my addr 172.21.15.28:0/2163823166 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:10:49.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.054+0000 7f28dbe00640 1 --2- 172.21.15.28:0/2163823166 >> v2:172.21.15.28:3300/0 conn(0x7f28dc0a5f80 0x7f28dc153190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:49.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.054+0000 7f28daa00640 1 --2- 172.21.15.28:0/2163823166 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f28dc0c2bb0 0x7f28dc152c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:49.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.054+0000 7f28da000640 1 -- 172.21.15.28:0/2163823166 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4161308385 0 0) 0x7f28dc0a6540 con 0x7f28dc150450 2024-02-21T23:10:49.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.054+0000 7f28da000640 1 -- 172.21.15.28:0/2163823166 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28c0003520 con 0x7f28dc150450 2024-02-21T23:10:49.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.055+0000 7f28dbe00640 1 -- 172.21.15.28:0/2163823166 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f28dc0c2bb0 msgr2=0x7f28dc152c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:49.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.055+0000 7f28dbe00640 1 --2- 172.21.15.28:0/2163823166 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f28dc0c2bb0 0x7f28dc152c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:49.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.055+0000 7f28dbe00640 1 -- 172.21.15.28:0/2163823166 >> v1:172.21.15.28:6789/0 conn(0x7f28dc150450 legacy=0x7f28dc06e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:49.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.055+0000 7f28dbe00640 1 -- 172.21.15.28:0/2163823166 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28dc0c7090 con 0x7f28dc0a5f80 2024-02-21T23:10:49.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.056+0000 7f28dbe00640 1 --2- 172.21.15.28:0/2163823166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28dc0a5f80 0x7f28dc153190 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7f28d4009d90 tx=0x7f28d402bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c8984667e820e3c server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:49.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.056+0000 7f28da000640 1 -- 172.21.15.28:0/2163823166 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28d4039070 con 0x7f28dc0a5f80 2024-02-21T23:10:49.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.056+0000 7f28da000640 1 -- 172.21.15.28:0/2163823166 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28d402cb30 con 0x7f28dc0a5f80 2024-02-21T23:10:49.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.056+0000 7f28da000640 1 -- 172.21.15.28:0/2163823166 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28d4034440 con 0x7f28dc0a5f80 2024-02-21T23:10:49.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.057+0000 7f28e0a00640 1 -- 172.21.15.28:0/2163823166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28dc0a5f80 msgr2=0x7f28dc153190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:49.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.057+0000 7f28e0a00640 1 --2- 172.21.15.28:0/2163823166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28dc0a5f80 0x7f28dc153190 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7f28d4009d90 tx=0x7f28d402bd80 comp rx=0 tx=0).stop 2024-02-21T23:10:49.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.057+0000 7f28e0a00640 1 -- 172.21.15.28:0/2163823166 shutdown_connections 2024-02-21T23:10:49.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.057+0000 7f28e0a00640 1 --2- 172.21.15.28:0/2163823166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28dc0a5f80 0x7f28dc153190 unknown :-1 s=CLOSED pgs=1894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:49.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.057+0000 7f28e0a00640 1 --2- 172.21.15.28:0/2163823166 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f28dc0c2bb0 0x7f28dc152c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:49.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.057+0000 7f28e0a00640 1 -- 172.21.15.28:0/2163823166 >> 172.21.15.28:0/2163823166 conn(0x7f28dc06b550 msgr2=0x7f28dc0c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:49.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.057+0000 7f28e0a00640 1 -- 172.21.15.28:0/2163823166 shutdown_connections 2024-02-21T23:10:49.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.057+0000 7f28e0a00640 1 -- 172.21.15.28:0/2163823166 wait complete. 2024-02-21T23:10:49.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.058+0000 7f28e0a00640 1 Processor -- start 2024-02-21T23:10:49.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.059+0000 7f28e0a00640 1 -- start start 2024-02-21T23:10:49.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.059+0000 7f28e0a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f28dc0a5f80 0x7f28dc149690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:49.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.059+0000 7f28e0a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28dc0c2bb0 0x7f28dc1edef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:49.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.060+0000 7f28e0a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f28dc150450 0x7f28dc148a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:49.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.060+0000 7f28e0a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28dc06e8a0 con 0x7f28dc0c2bb0 2024-02-21T23:10:49.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.060+0000 7f28e0a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28dc06e720 con 0x7f28dc0a5f80 2024-02-21T23:10:49.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.060+0000 7f28e0a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28dc0a7a00 con 0x7f28dc150450 2024-02-21T23:10:49.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.060+0000 7f28daa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28dc0c2bb0 0x7f28dc1edef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:49.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.060+0000 7f28daa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28dc0c2bb0 0x7f28dc1edef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49240/0 (socket says 172.21.15.28:49240) 2024-02-21T23:10:49.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.060+0000 7f28daa00640 1 -- 172.21.15.28:0/2018704116 learned_addr learned my addr 172.21.15.28:0/2018704116 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:49.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.060+0000 7f28daa00640 1 -- 172.21.15.28:0/2018704116 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f28dc150450 msgr2=0x7f28dc148a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:49.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.060+0000 7f28dbe00640 1 --2- 172.21.15.28:0/2018704116 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f28dc150450 0x7f28dc148a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:49.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.060+0000 7f28db400640 1 --2- 172.21.15.28:0/2018704116 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f28dc0a5f80 0x7f28dc149690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:49.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.060+0000 7f28daa00640 1 --2- 172.21.15.28:0/2018704116 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f28dc150450 0x7f28dc148a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:49.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.060+0000 7f28daa00640 1 -- 172.21.15.28:0/2018704116 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f28dc0a5f80 msgr2=0x7f28dc149690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:49.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.060+0000 7f28daa00640 1 --2- 172.21.15.28:0/2018704116 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f28dc0a5f80 0x7f28dc149690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:49.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.060+0000 7f28daa00640 1 -- 172.21.15.28:0/2018704116 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28dc1ee970 con 0x7f28dc0c2bb0 2024-02-21T23:10:49.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.060+0000 7f28db400640 1 --2- 172.21.15.28:0/2018704116 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f28dc0a5f80 0x7f28dc149690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:10:49.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.061+0000 7f28daa00640 1 --2- 172.21.15.28:0/2018704116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28dc0c2bb0 0x7f28dc1edef0 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7f28c800e9e0 tx=0x7f28c800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:49.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.061+0000 7f28d3400640 1 -- 172.21.15.28:0/2018704116 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28c8017070 con 0x7f28dc0c2bb0 2024-02-21T23:10:49.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.061+0000 7f28d3400640 1 -- 172.21.15.28:0/2018704116 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28c800c9d0 con 0x7f28dc0c2bb0 2024-02-21T23:10:49.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.061+0000 7f28d3400640 1 -- 172.21.15.28:0/2018704116 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28c800cdc0 con 0x7f28dc0c2bb0 2024-02-21T23:10:49.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.061+0000 7f28e0a00640 1 -- 172.21.15.28:0/2018704116 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28dc0ab0a0 con 0x7f28dc0c2bb0 2024-02-21T23:10:49.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.061+0000 7f28e0a00640 1 -- 172.21.15.28:0/2018704116 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28dc1ee430 con 0x7f28dc0c2bb0 2024-02-21T23:10:49.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.063+0000 7f28e0a00640 1 -- 172.21.15.28:0/2018704116 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28dc04f290 con 0x7f28dc0c2bb0 2024-02-21T23:10:49.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.063+0000 7f28d3400640 1 -- 172.21.15.28:0/2018704116 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f28c801a070 con 0x7f28dc0c2bb0 2024-02-21T23:10:49.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.064+0000 7f28d3400640 1 --2- 172.21.15.28:0/2018704116 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f28bc06b4d0 0x7f28bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:49.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.064+0000 7f28d3400640 1 -- 172.21.15.28:0/2018704116 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f28c808a490 con 0x7f28dc0c2bb0 2024-02-21T23:10:49.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.065+0000 7f28db400640 1 --2- 172.21.15.28:0/2018704116 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f28bc06b4d0 0x7f28bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:49.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.066+0000 7f28db400640 1 --2- 172.21.15.28:0/2018704116 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f28bc06b4d0 0x7f28bc06d990 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f28cc005880 tx=0x7f28cc009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:49.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.069+0000 7f28d3400640 1 -- 172.21.15.28:0/2018704116 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f28c80594c0 con 0x7f28dc0c2bb0 2024-02-21T23:10:49.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.343+0000 7f28e0a00640 1 -- 172.21.15.28:0/2018704116 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f28dc0c1f10 con 0x7f28dc0c2bb0 2024-02-21T23:10:49.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.344+0000 7f28d3400640 1 -- 172.21.15.28:0/2018704116 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v19) v1 ==== 80+0+4724 (secure 0 0 0) 0x7f28c8058dc0 con 0x7f28dc0c2bb0 2024-02-21T23:10:49.333 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:10:49.333 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-21T23:10:49.333 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_dump": [ 2024-02-21T23:10:49.333 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:10:49.333 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.a", 2024-02-21T23:10:49.333 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDSgNZl4FY1KRAAJSK5Ibb0ibKiKXYpTLgtmQ==", 2024-02-21T23:10:49.333 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:10:49.333 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:10:49.333 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-02-21T23:10:49.333 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-02-21T23:10:49.333 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:10:49.334 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:10:49.334 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:10:49.334 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:10:49.334 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.b", 2024-02-21T23:10:49.334 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDTgNZlwRcAERAAgsYmmkuobKelBEJVCRpXUg==", 2024-02-21T23:10:49.334 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:10:49.334 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:10:49.334 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-02-21T23:10:49.334 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-02-21T23:10:49.334 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:10:49.334 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:10:49.334 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:10:49.334 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:10:49.334 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.c", 2024-02-21T23:10:49.334 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDSgNZloFpQOxAAK2WBArXRVNTgqH9WWDEPDA==", 2024-02-21T23:10:49.335 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:10:49.335 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:10:49.335 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-02-21T23:10:49.335 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-02-21T23:10:49.335 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:10:49.335 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:10:49.335 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:10:49.335 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:10:49.335 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.d", 2024-02-21T23:10:49.335 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDTgNZlblOmIRAAxe2HfSKGSkkzJDN4aVdzgQ==", 2024-02-21T23:10:49.335 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:10:49.335 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:10:49.335 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-02-21T23:10:49.335 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-02-21T23:10:49.336 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:10:49.336 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:10:49.336 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:10:49.336 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:10:49.336 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.0", 2024-02-21T23:10:49.336 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDVgNZlxNJbLxAAXot7Q3chrLEBsFNO8kT6QQ==", 2024-02-21T23:10:49.336 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:10:49.336 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:10:49.336 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:10:49.336 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:10:49.336 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:10:49.336 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:10:49.336 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:10:49.336 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.1", 2024-02-21T23:10:49.337 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDYgNZlZZfWORAAr4XkOxLYLmumVbu362N2HQ==", 2024-02-21T23:10:49.337 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:10:49.337 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:10:49.337 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:10:49.337 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:10:49.337 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:10:49.337 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:10:49.337 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:10:49.337 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.2", 2024-02-21T23:10:49.339 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDcgNZl9mcDGBAAIZGOF8IUMenwo386x95mwg==", 2024-02-21T23:10:49.339 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:10:49.339 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:10:49.339 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:10:49.339 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:10:49.339 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:10:49.339 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:10:49.340 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:10:49.340 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.3", 2024-02-21T23:10:49.340 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDfgNZl8ALSMBAAag9cmGH0/4xALWSUmrZowg==", 2024-02-21T23:10:49.340 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:10:49.340 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:10:49.340 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:10:49.340 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:10:49.340 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:10:49.340 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:10:49.340 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:10:49.340 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.4", 2024-02-21T23:10:49.340 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDngNZlBgeKABAAeW99RVfCBvB1lp7w7mQBlQ==", 2024-02-21T23:10:49.340 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:10:49.340 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:10:49.341 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:10:49.341 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:10:49.341 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:10:49.341 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:10:49.341 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:10:49.341 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.5", 2024-02-21T23:10:49.341 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDqgNZljQG3ChAAY/IZEIcvcGNvlFGIWib3gg==", 2024-02-21T23:10:49.341 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:10:49.341 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:10:49.341 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:10:49.341 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:10:49.341 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:10:49.341 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:10:49.341 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:10:49.341 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.6", 2024-02-21T23:10:49.342 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDtgNZlKnakIxAAkW5xlkshzvEHJdmpgPxR+g==", 2024-02-21T23:10:49.342 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:10:49.342 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:10:49.342 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:10:49.342 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:10:49.342 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:10:49.342 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:10:49.342 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:10:49.342 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.7", 2024-02-21T23:10:49.342 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDxgNZlTWxYARAASX1/gmqO+8mQPP8uA4xjiQ==", 2024-02-21T23:10:49.342 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:10:49.342 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:10:49.342 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:10:49.342 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:10:49.343 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:10:49.343 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:10:49.343 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:10:49.343 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.0", 2024-02-21T23:10:49.343 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDTgNZlILvYLxAAoREWxKKT7YcXnhkgk8YvoQ==", 2024-02-21T23:10:49.343 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:10:49.343 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:10:49.343 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow r", 2024-02-21T23:10:49.343 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-02-21T23:10:49.343 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:10:49.343 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:10:49.343 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:10:49.343 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.1", 2024-02-21T23:10:49.343 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDTgNZlm7ImOxAA8iRcXlzFPZPaXQfsKhuO/g==", 2024-02-21T23:10:49.344 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:10:49.344 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:10:49.344 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-02-21T23:10:49.344 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-02-21T23:10:49.344 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-02-21T23:10:49.344 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:10:49.344 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:10:49.344 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:10:49.344 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.admin", 2024-02-21T23:10:49.344 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDRgNZlBYd1GhAAbK5qBCIkf0Ej5nEyPnTxIQ==", 2024-02-21T23:10:49.344 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:10:49.344 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-02-21T23:10:49.344 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-02-21T23:10:49.344 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-02-21T23:10:49.345 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:10:49.345 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:10:49.345 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:10:49.345 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:10:49.345 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.x", 2024-02-21T23:10:49.345 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDSgNZlJYKeHBAAYPskmLRr7Jw44TdKzM/JMg==", 2024-02-21T23:10:49.345 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:10:49.345 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-02-21T23:10:49.345 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-02-21T23:10:49.345 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:10:49.345 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:10:49.345 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:10:49.345 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:10:49.345 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.y", 2024-02-21T23:10:49.345 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDSgNZlSRkOEBAAOhRj9PsZDpjoV6WT4drrZA==", 2024-02-21T23:10:49.346 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:10:49.346 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-02-21T23:10:49.346 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-02-21T23:10:49.346 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:10:49.346 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:10:49.346 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:10:49.346 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-21T23:10:49.346 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-21T23:10:49.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.346+0000 7f28e0a00640 1 -- 172.21.15.28:0/2018704116 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f28bc06b4d0 msgr2=0x7f28bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:49.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.346+0000 7f28e0a00640 1 --2- 172.21.15.28:0/2018704116 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f28bc06b4d0 0x7f28bc06d990 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f28cc005880 tx=0x7f28cc009040 comp rx=0 tx=0).stop 2024-02-21T23:10:49.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.346+0000 7f28e0a00640 1 -- 172.21.15.28:0/2018704116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28dc0c2bb0 msgr2=0x7f28dc1edef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:49.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.346+0000 7f28e0a00640 1 --2- 172.21.15.28:0/2018704116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28dc0c2bb0 0x7f28dc1edef0 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7f28c800e9e0 tx=0x7f28c800eea0 comp rx=0 tx=0).stop 2024-02-21T23:10:49.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.346+0000 7f28e0a00640 1 -- 172.21.15.28:0/2018704116 shutdown_connections 2024-02-21T23:10:49.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.346+0000 7f28e0a00640 1 --2- 172.21.15.28:0/2018704116 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f28bc06b4d0 0x7f28bc06d990 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:49.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.346+0000 7f28e0a00640 1 --2- 172.21.15.28:0/2018704116 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f28dc150450 0x7f28dc148a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:49.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.346+0000 7f28e0a00640 1 --2- 172.21.15.28:0/2018704116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28dc0c2bb0 0x7f28dc1edef0 unknown :-1 s=CLOSED pgs=1895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:49.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.346+0000 7f28e0a00640 1 --2- 172.21.15.28:0/2018704116 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f28dc0a5f80 0x7f28dc149690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:49.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.346+0000 7f28e0a00640 1 -- 172.21.15.28:0/2018704116 >> 172.21.15.28:0/2018704116 conn(0x7f28dc06b550 msgr2=0x7f28dc14e410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:49.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.347+0000 7f28e0a00640 1 -- 172.21.15.28:0/2018704116 shutdown_connections 2024-02-21T23:10:49.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.347+0000 7f28e0a00640 1 -- 172.21.15.28:0/2018704116 wait complete. 2024-02-21T23:10:49.354 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:10:49.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.572+0000 7fd908a00640 1 Processor -- start 2024-02-21T23:10:49.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.573+0000 7fd908a00640 1 -- start start 2024-02-21T23:10:49.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.574+0000 7fd908a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd9040c3280 0x7fd9040c3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:49.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.574+0000 7fd908a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fd904155860 0x7fd904155d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:49.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.574+0000 7fd908a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9040c6560 con 0x7fd904072430 2024-02-21T23:10:49.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.574+0000 7fd908a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9040c5cc0 con 0x7fd904155860 2024-02-21T23:10:49.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.574+0000 7fd908a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9040c5fa0 con 0x7fd9040c3280 2024-02-21T23:10:49.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.575+0000 7fd901e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd9040c3280 0x7fd9040c3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:49.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.575+0000 7fd901e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd9040c3280 0x7fd9040c3660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3301/0 says I am v2:172.21.15.28:34386/0 (socket says 172.21.15.28:34386) 2024-02-21T23:10:49.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.575+0000 7fd901e00640 1 -- 172.21.15.28:0/228750754 learned_addr learned my addr 172.21.15.28:0/228750754 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:49.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.575+0000 7fd902800640 1 --2- 172.21.15.28:0/228750754 >> v2:172.21.15.81:3300/0 conn(0x7fd904155860 0x7fd904155d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:49.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.575+0000 7fd901e00640 1 -- 172.21.15.28:0/228750754 >> v2:172.21.15.81:3300/0 conn(0x7fd904155860 msgr2=0x7fd904155d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:49.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.575+0000 7fd901e00640 1 --2- 172.21.15.28:0/228750754 >> v2:172.21.15.81:3300/0 conn(0x7fd904155860 0x7fd904155d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:49.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.575+0000 7fd901e00640 1 -- 172.21.15.28:0/228750754 >> v1:172.21.15.81:6789/0 conn(0x7fd904072430 legacy=0x7fd9041526e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:49.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.576+0000 7fd901e00640 1 -- 172.21.15.28:0/228750754 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9040c6280 con 0x7fd9040c3280 2024-02-21T23:10:49.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.577+0000 7fd901e00640 1 --2- 172.21.15.28:0/228750754 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd9040c3280 0x7fd9040c3660 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd8f4009e50 tx=0x7fd8f402bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc09d8d2e4cf7410 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:49.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.577+0000 7fd900a00640 1 -- 172.21.15.28:0/228750754 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f402ca70 con 0x7fd9040c3280 2024-02-21T23:10:49.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.577+0000 7fd900a00640 1 -- 172.21.15.28:0/228750754 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8f402cc10 con 0x7fd9040c3280 2024-02-21T23:10:49.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.577+0000 7fd900a00640 1 -- 172.21.15.28:0/228750754 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f40343f0 con 0x7fd9040c3280 2024-02-21T23:10:49.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.578+0000 7fd908a00640 1 -- 172.21.15.28:0/228750754 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd9040c3280 msgr2=0x7fd9040c3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:49.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.578+0000 7fd908a00640 1 --2- 172.21.15.28:0/228750754 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd9040c3280 0x7fd9040c3660 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd8f4009e50 tx=0x7fd8f402bed0 comp rx=0 tx=0).stop 2024-02-21T23:10:49.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.578+0000 7fd908a00640 1 -- 172.21.15.28:0/228750754 shutdown_connections 2024-02-21T23:10:49.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.578+0000 7fd908a00640 1 --2- 172.21.15.28:0/228750754 >> v2:172.21.15.81:3300/0 conn(0x7fd904155860 0x7fd904155d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:49.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.578+0000 7fd908a00640 1 --2- 172.21.15.28:0/228750754 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd9040c3280 0x7fd9040c3660 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:49.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.578+0000 7fd908a00640 1 -- 172.21.15.28:0/228750754 >> 172.21.15.28:0/228750754 conn(0x7fd9040c67f0 msgr2=0x7fd9040c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:49.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.578+0000 7fd908a00640 1 -- 172.21.15.28:0/228750754 shutdown_connections 2024-02-21T23:10:49.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.578+0000 7fd908a00640 1 -- 172.21.15.28:0/228750754 wait complete. 2024-02-21T23:10:49.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.580+0000 7fd908a00640 1 Processor -- start 2024-02-21T23:10:49.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.580+0000 7fd908a00640 1 -- start start 2024-02-21T23:10:49.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.581+0000 7fd908a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd904072430 0x7fd9041ee1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:49.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.581+0000 7fd908a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9040c3280 0x7fd9041dfb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:49.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.581+0000 7fd908a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd904155860 0x7fd9041502a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:49.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.581+0000 7fd908a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9040c5aa0 con 0x7fd9040c3280 2024-02-21T23:10:49.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.581+0000 7fd908a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9040c5920 con 0x7fd904155860 2024-02-21T23:10:49.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.581+0000 7fd908a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9040733e0 con 0x7fd904072430 2024-02-21T23:10:49.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.581+0000 7fd901400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9040c3280 0x7fd9041dfb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:49.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.581+0000 7fd901400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9040c3280 0x7fd9041dfb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49252/0 (socket says 172.21.15.28:49252) 2024-02-21T23:10:49.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.581+0000 7fd901400640 1 -- 172.21.15.28:0/3580534163 learned_addr learned my addr 172.21.15.28:0/3580534163 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:49.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.582+0000 7fd902800640 1 --2- 172.21.15.28:0/3580534163 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd904155860 0x7fd9041502a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:49.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.582+0000 7fd901e00640 1 --2- 172.21.15.28:0/3580534163 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd904072430 0x7fd9041ee1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:49.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.582+0000 7fd901400640 1 -- 172.21.15.28:0/3580534163 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd904072430 msgr2=0x7fd9041ee1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:49.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.582+0000 7fd901400640 1 --2- 172.21.15.28:0/3580534163 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd904072430 0x7fd9041ee1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:49.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.582+0000 7fd901400640 1 -- 172.21.15.28:0/3580534163 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd904155860 msgr2=0x7fd9041502a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:49.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.582+0000 7fd901400640 1 --2- 172.21.15.28:0/3580534163 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd904155860 0x7fd9041502a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:49.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.582+0000 7fd901400640 1 -- 172.21.15.28:0/3580534163 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd904152480 con 0x7fd9040c3280 2024-02-21T23:10:49.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.582+0000 7fd901e00640 1 --2- 172.21.15.28:0/3580534163 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd904072430 0x7fd9041ee1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:10:49.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.582+0000 7fd901400640 1 --2- 172.21.15.28:0/3580534163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9040c3280 0x7fd9041dfb80 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec00c430 tx=0x7fd8ec00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:49.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.583+0000 7fd8fa000640 1 -- 172.21.15.28:0/3580534163 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8ec017070 con 0x7fd9040c3280 2024-02-21T23:10:49.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.583+0000 7fd8fa000640 1 -- 172.21.15.28:0/3580534163 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8ec0052d0 con 0x7fd9040c3280 2024-02-21T23:10:49.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.583+0000 7fd8fa000640 1 -- 172.21.15.28:0/3580534163 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8ec003830 con 0x7fd9040c3280 2024-02-21T23:10:49.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.583+0000 7fd908a00640 1 -- 172.21.15.28:0/3580534163 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd90414be20 con 0x7fd9040c3280 2024-02-21T23:10:49.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.583+0000 7fd908a00640 1 -- 172.21.15.28:0/3580534163 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9040a73f0 con 0x7fd9040c3280 2024-02-21T23:10:49.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.584+0000 7fd908a00640 1 -- 172.21.15.28:0/3580534163 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd904067f30 con 0x7fd9040c3280 2024-02-21T23:10:49.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.585+0000 7fd8fa000640 1 -- 172.21.15.28:0/3580534163 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd8ec01a070 con 0x7fd9040c3280 2024-02-21T23:10:49.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.586+0000 7fd8fa000640 1 --2- 172.21.15.28:0/3580534163 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd8e006b4d0 0x7fd8e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:49.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.586+0000 7fd8fa000640 1 -- 172.21.15.28:0/3580534163 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fd8ec089430 con 0x7fd9040c3280 2024-02-21T23:10:49.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.587+0000 7fd901e00640 1 --2- 172.21.15.28:0/3580534163 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd8e006b4d0 0x7fd8e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:49.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.588+0000 7fd901e00640 1 --2- 172.21.15.28:0/3580534163 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd8e006b4d0 0x7fd8e006d990 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fd8f4035000 tx=0x7fd8f4030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:49.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.591+0000 7fd8fa000640 1 -- 172.21.15.28:0/3580534163 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd8ec058460 con 0x7fd9040c3280 2024-02-21T23:10:49.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.791+0000 7fd908a00640 1 -- 172.21.15.28:0/3580534163 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd90406f2d0 con 0x7fd9040c3280 2024-02-21T23:10:49.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.793+0000 7fd8fa000640 1 -- 172.21.15.28:0/3580534163 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v170) v1 ==== 74+0+13114 (secure 0 0 0) 0x7fd8ec057d60 con 0x7fd9040c3280 2024-02-21T23:10:49.782 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:10:49.782 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":170,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:10:48.834763+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":153,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:10:49.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.795+0000 7fd908a00640 1 -- 172.21.15.28:0/3580534163 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd8e006b4d0 msgr2=0x7fd8e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:49.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.795+0000 7fd908a00640 1 --2- 172.21.15.28:0/3580534163 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd8e006b4d0 0x7fd8e006d990 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fd8f4035000 tx=0x7fd8f4030000 comp rx=0 tx=0).stop 2024-02-21T23:10:49.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.795+0000 7fd908a00640 1 -- 172.21.15.28:0/3580534163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9040c3280 msgr2=0x7fd9041dfb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:49.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.795+0000 7fd908a00640 1 --2- 172.21.15.28:0/3580534163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9040c3280 0x7fd9041dfb80 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec00c430 tx=0x7fd8ec00c8f0 comp rx=0 tx=0).stop 2024-02-21T23:10:49.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.796+0000 7fd908a00640 1 -- 172.21.15.28:0/3580534163 shutdown_connections 2024-02-21T23:10:49.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.796+0000 7fd908a00640 1 --2- 172.21.15.28:0/3580534163 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd8e006b4d0 0x7fd8e006d990 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:49.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.796+0000 7fd908a00640 1 --2- 172.21.15.28:0/3580534163 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd904155860 0x7fd9041502a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:49.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.796+0000 7fd908a00640 1 --2- 172.21.15.28:0/3580534163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9040c3280 0x7fd9041dfb80 unknown :-1 s=CLOSED pgs=1896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:49.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.796+0000 7fd908a00640 1 --2- 172.21.15.28:0/3580534163 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd904072430 0x7fd9041ee1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:49.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.796+0000 7fd908a00640 1 -- 172.21.15.28:0/3580534163 >> 172.21.15.28:0/3580534163 conn(0x7fd9040c67f0 msgr2=0x7fd9040c27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:49.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.796+0000 7fd908a00640 1 -- 172.21.15.28:0/3580534163 shutdown_connections 2024-02-21T23:10:49.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:49.796+0000 7fd908a00640 1 -- 172.21.15.28:0/3580534163 wait complete. 2024-02-21T23:10:49.803 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-21T23:02:30.728329+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-21T23:10:49.803 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-21T23:10:50.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.028+0000 7f9ff1000640 1 Processor -- start 2024-02-21T23:10:50.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.028+0000 7f9ff1000640 1 -- start start 2024-02-21T23:10:50.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.029+0000 7f9ff1000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f9fec06ae70 0x7f9fec06b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:50.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.030+0000 7f9ff1000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9fec159c50 0x7f9fec163b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:50.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.030+0000 7f9ff1000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fec0c0d40 con 0x7f9fec0c4c10 2024-02-21T23:10:50.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.030+0000 7f9ff1000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fec15caf0 con 0x7f9fec159c50 2024-02-21T23:10:50.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.030+0000 7f9ff1000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fec15cdd0 con 0x7f9fec06ae70 2024-02-21T23:10:50.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.030+0000 7f9febe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9fec159c50 0x7f9fec163b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:50.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.030+0000 7f9feaa00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f9fec0c4c10 0x7f9fec159540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41278/0 (socket says 172.21.15.28:41278) 2024-02-21T23:10:50.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.030+0000 7f9febe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9fec159c50 0x7f9fec163b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49262/0 (socket says 172.21.15.28:49262) 2024-02-21T23:10:50.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.030+0000 7f9feaa00640 1 -- 172.21.15.28:0/3895343216 learned_addr learned my addr 172.21.15.28:0/3895343216 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:10:50.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.030+0000 7f9feb400640 1 --2- 172.21.15.28:0/3895343216 >> v2:172.21.15.81:3300/0 conn(0x7f9fec06ae70 0x7f9fec06b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:50.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.031+0000 7f9febe00640 1 -- 172.21.15.28:0/3895343216 >> v2:172.21.15.81:3300/0 conn(0x7f9fec06ae70 msgr2=0x7f9fec06b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:50.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.031+0000 7f9febe00640 1 --2- 172.21.15.28:0/3895343216 >> v2:172.21.15.81:3300/0 conn(0x7f9fec06ae70 0x7f9fec06b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:50.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.031+0000 7f9febe00640 1 -- 172.21.15.28:0/3895343216 >> v1:172.21.15.28:6789/0 conn(0x7f9fec0c4c10 legacy=0x7f9fec159540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:50.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.031+0000 7f9febe00640 1 -- 172.21.15.28:0/3895343216 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fec15d0b0 con 0x7f9fec159c50 2024-02-21T23:10:50.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.032+0000 7f9febe00640 1 --2- 172.21.15.28:0/3895343216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9fec159c50 0x7f9fec163b70 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7f9fec06ed60 tx=0x7f9fe402be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64af37e6c442cba6 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:50.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.032+0000 7f9fea000640 1 -- 172.21.15.28:0/3895343216 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fe4039070 con 0x7f9fec159c50 2024-02-21T23:10:50.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.032+0000 7f9fea000640 1 -- 172.21.15.28:0/3895343216 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fe402cbe0 con 0x7f9fec159c50 2024-02-21T23:10:50.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.032+0000 7f9fea000640 1 -- 172.21.15.28:0/3895343216 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fe4034440 con 0x7f9fec159c50 2024-02-21T23:10:50.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.033+0000 7f9ff1000640 1 -- 172.21.15.28:0/3895343216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9fec159c50 msgr2=0x7f9fec163b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:50.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.033+0000 7f9ff1000640 1 --2- 172.21.15.28:0/3895343216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9fec159c50 0x7f9fec163b70 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7f9fec06ed60 tx=0x7f9fe402be30 comp rx=0 tx=0).stop 2024-02-21T23:10:50.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.033+0000 7f9ff1000640 1 -- 172.21.15.28:0/3895343216 shutdown_connections 2024-02-21T23:10:50.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.033+0000 7f9ff1000640 1 --2- 172.21.15.28:0/3895343216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9fec159c50 0x7f9fec163b70 unknown :-1 s=CLOSED pgs=1898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:50.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.033+0000 7f9ff1000640 1 --2- 172.21.15.28:0/3895343216 >> v2:172.21.15.81:3300/0 conn(0x7f9fec06ae70 0x7f9fec06b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:50.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.033+0000 7f9ff1000640 1 -- 172.21.15.28:0/3895343216 >> 172.21.15.28:0/3895343216 conn(0x7f9fec0a8530 msgr2=0x7f9fec0a8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:50.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.033+0000 7f9ff1000640 1 -- 172.21.15.28:0/3895343216 shutdown_connections 2024-02-21T23:10:50.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.033+0000 7f9ff1000640 1 -- 172.21.15.28:0/3895343216 wait complete. 2024-02-21T23:10:50.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.035+0000 7f9ff1000640 1 Processor -- start 2024-02-21T23:10:50.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.035+0000 7f9ff1000640 1 -- start start 2024-02-21T23:10:50.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.036+0000 7f9ff1000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9fec06ae70 0x7f9fec158450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:50.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.036+0000 7f9ff1000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9fec0c4c10 0x7f9fec152ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:50.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.036+0000 7f9ff1000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9fec159c50 0x7f9fec14e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:50.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.036+0000 7f9ff1000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fec15c7e0 con 0x7f9fec06ae70 2024-02-21T23:10:50.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.036+0000 7f9ff1000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fec15d670 con 0x7f9fec0c4c10 2024-02-21T23:10:50.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.036+0000 7f9ff1000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fec06f6b0 con 0x7f9fec159c50 2024-02-21T23:10:50.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.036+0000 7f9feb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9fec06ae70 0x7f9fec158450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:50.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.036+0000 7f9feb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9fec06ae70 0x7f9fec158450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49266/0 (socket says 172.21.15.28:49266) 2024-02-21T23:10:50.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.036+0000 7f9feb400640 1 -- 172.21.15.28:0/1639594377 learned_addr learned my addr 172.21.15.28:0/1639594377 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:50.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.036+0000 7f9feb400640 1 -- 172.21.15.28:0/1639594377 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9fec159c50 msgr2=0x7f9fec14e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:50.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.036+0000 7f9feaa00640 1 --2- 172.21.15.28:0/1639594377 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9fec0c4c10 0x7f9fec152ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:50.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.036+0000 7f9febe00640 1 --2- 172.21.15.28:0/1639594377 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9fec159c50 0x7f9fec14e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:50.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.036+0000 7f9feb400640 1 --2- 172.21.15.28:0/1639594377 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9fec159c50 0x7f9fec14e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:50.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.036+0000 7f9feb400640 1 -- 172.21.15.28:0/1639594377 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9fec0c4c10 msgr2=0x7f9fec152ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:50.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.036+0000 7f9feb400640 1 --2- 172.21.15.28:0/1639594377 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9fec0c4c10 0x7f9fec152ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:50.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.036+0000 7f9feb400640 1 -- 172.21.15.28:0/1639594377 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fec159150 con 0x7f9fec06ae70 2024-02-21T23:10:50.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.037+0000 7f9feb400640 1 --2- 172.21.15.28:0/1639594377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9fec06ae70 0x7f9fec158450 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc00d800 tx=0x7f9fdc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:50.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.037+0000 7f9fe3400640 1 -- 172.21.15.28:0/1639594377 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fdc00b6c0 con 0x7f9fec06ae70 2024-02-21T23:10:50.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.037+0000 7f9fe3400640 1 -- 172.21.15.28:0/1639594377 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fdc019070 con 0x7f9fec06ae70 2024-02-21T23:10:50.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.037+0000 7f9ff1000640 1 -- 172.21.15.28:0/1639594377 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fec158eb0 con 0x7f9fec06ae70 2024-02-21T23:10:50.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.037+0000 7f9fe3400640 1 -- 172.21.15.28:0/1639594377 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fdc00ba80 con 0x7f9fec06ae70 2024-02-21T23:10:50.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.037+0000 7f9ff1000640 1 -- 172.21.15.28:0/1639594377 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fec1ef490 con 0x7f9fec06ae70 2024-02-21T23:10:50.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.039+0000 7f9ff1000640 1 -- 172.21.15.28:0/1639594377 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fec04f290 con 0x7f9fec06ae70 2024-02-21T23:10:50.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.039+0000 7f9fe3400640 1 -- 172.21.15.28:0/1639594377 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9fdc00bc20 con 0x7f9fec06ae70 2024-02-21T23:10:50.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.040+0000 7f9fe3400640 1 --2- 172.21.15.28:0/1639594377 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9fcc06b500 0x7f9fcc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:50.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.040+0000 7f9fe3400640 1 -- 172.21.15.28:0/1639594377 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f9fdc08a410 con 0x7f9fec06ae70 2024-02-21T23:10:50.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.041+0000 7f9feaa00640 1 --2- 172.21.15.28:0/1639594377 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9fcc06b500 0x7f9fcc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:50.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.042+0000 7f9feaa00640 1 --2- 172.21.15.28:0/1639594377 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9fcc06b500 0x7f9fcc06d9c0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f9fec1551d0 tx=0x7f9fd8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:50.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.044+0000 7f9fe3400640 1 -- 172.21.15.28:0/1639594377 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f9fdc01c360 con 0x7f9fec06ae70 2024-02-21T23:10:50.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.243+0000 7f9ff1000640 1 -- 172.21.15.28:0/1639594377 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9fec0c1ac0 con 0x7f9fec06ae70 2024-02-21T23:10:50.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.244+0000 7f9fe3400640 1 -- 172.21.15.28:0/1639594377 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v170) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9fdc0594e0 con 0x7f9fec06ae70 2024-02-21T23:10:50.233 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-02-21T23:10:50.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.246+0000 7f9ff1000640 1 -- 172.21.15.28:0/1639594377 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9fcc06b500 msgr2=0x7f9fcc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:50.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.246+0000 7f9ff1000640 1 --2- 172.21.15.28:0/1639594377 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9fcc06b500 0x7f9fcc06d9c0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f9fec1551d0 tx=0x7f9fd8009040 comp rx=0 tx=0).stop 2024-02-21T23:10:50.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.246+0000 7f9ff1000640 1 -- 172.21.15.28:0/1639594377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9fec06ae70 msgr2=0x7f9fec158450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:50.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.246+0000 7f9ff1000640 1 --2- 172.21.15.28:0/1639594377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9fec06ae70 0x7f9fec158450 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc00d800 tx=0x7f9fdc00dcc0 comp rx=0 tx=0).stop 2024-02-21T23:10:50.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.246+0000 7f9ff1000640 1 -- 172.21.15.28:0/1639594377 shutdown_connections 2024-02-21T23:10:50.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.246+0000 7f9ff1000640 1 --2- 172.21.15.28:0/1639594377 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9fcc06b500 0x7f9fcc06d9c0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:50.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.246+0000 7f9ff1000640 1 --2- 172.21.15.28:0/1639594377 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9fec159c50 0x7f9fec14e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:50.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.246+0000 7f9ff1000640 1 --2- 172.21.15.28:0/1639594377 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9fec0c4c10 0x7f9fec152ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:50.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.246+0000 7f9ff1000640 1 --2- 172.21.15.28:0/1639594377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9fec06ae70 0x7f9fec158450 unknown :-1 s=CLOSED pgs=1899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:50.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.246+0000 7f9ff1000640 1 -- 172.21.15.28:0/1639594377 >> 172.21.15.28:0/1639594377 conn(0x7f9fec0a8530 msgr2=0x7f9fec06b250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:50.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.247+0000 7f9ff1000640 1 -- 172.21.15.28:0/1639594377 shutdown_connections 2024-02-21T23:10:50.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.247+0000 7f9ff1000640 1 -- 172.21.15.28:0/1639594377 wait complete. 2024-02-21T23:10:50.253 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-02-21T23:10:50.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.482+0000 7fd2db200640 1 Processor -- start 2024-02-21T23:10:50.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.483+0000 7fd2db200640 1 -- start start 2024-02-21T23:10:50.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.484+0000 7fd2db200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd2d40c49d0 0x7fd2d406f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:50.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.484+0000 7fd2db200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2d4153bf0 con 0x7fd2d4150230 2024-02-21T23:10:50.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.484+0000 7fd2db200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2d4153910 con 0x7fd2d406f670 2024-02-21T23:10:50.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.484+0000 7fd2db200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2d41536c0 con 0x7fd2d40c49d0 2024-02-21T23:10:50.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.484+0000 7fd2d9400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd2d40c49d0 0x7fd2d406f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:50.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.484+0000 7fd2d9e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd2d4150230 0x7fd2d40c42c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41286/0 (socket says 172.21.15.28:41286) 2024-02-21T23:10:50.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.484+0000 7fd2d9400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd2d40c49d0 0x7fd2d406f130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49270/0 (socket says 172.21.15.28:49270) 2024-02-21T23:10:50.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.485+0000 7fd2d9e00640 1 -- 172.21.15.28:0/2072281645 learned_addr learned my addr 172.21.15.28:0/2072281645 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:10:50.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.485+0000 7fd2d9400640 1 -- 172.21.15.28:0/2072281645 >> v1:172.21.15.81:6789/0 conn(0x7fd2d406f670 legacy=0x7fd2d415d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:50.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.485+0000 7fd2d9400640 1 -- 172.21.15.28:0/2072281645 >> v1:172.21.15.28:6789/0 conn(0x7fd2d4150230 legacy=0x7fd2d40c42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:50.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.485+0000 7fd2d9400640 1 -- 172.21.15.28:0/2072281645 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2d40c3ac0 con 0x7fd2d40c49d0 2024-02-21T23:10:50.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.486+0000 7fd2d9400640 1 --2- 172.21.15.28:0/2072281645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2d40c49d0 0x7fd2d406f130 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7fd2c40097a0 tx=0x7fd2c402b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=939955e9214b734b server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:50.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.486+0000 7fd2d8a00640 1 -- 172.21.15.28:0/2072281645 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2c4038070 con 0x7fd2d40c49d0 2024-02-21T23:10:50.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.487+0000 7fd2d8a00640 1 -- 172.21.15.28:0/2072281645 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2c4031700 con 0x7fd2d40c49d0 2024-02-21T23:10:50.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.487+0000 7fd2d8a00640 1 -- 172.21.15.28:0/2072281645 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2c4031ad0 con 0x7fd2d40c49d0 2024-02-21T23:10:50.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.487+0000 7fd2db200640 1 -- 172.21.15.28:0/2072281645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2d40c49d0 msgr2=0x7fd2d406f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:50.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.487+0000 7fd2db200640 1 --2- 172.21.15.28:0/2072281645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2d40c49d0 0x7fd2d406f130 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7fd2c40097a0 tx=0x7fd2c402b7a0 comp rx=0 tx=0).stop 2024-02-21T23:10:50.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.487+0000 7fd2db200640 1 -- 172.21.15.28:0/2072281645 shutdown_connections 2024-02-21T23:10:50.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.487+0000 7fd2db200640 1 --2- 172.21.15.28:0/2072281645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2d40c49d0 0x7fd2d406f130 unknown :-1 s=CLOSED pgs=1901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:50.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.487+0000 7fd2db200640 1 -- 172.21.15.28:0/2072281645 >> 172.21.15.28:0/2072281645 conn(0x7fd2d40aa3d0 msgr2=0x7fd2d40aa7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:50.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.487+0000 7fd2db200640 1 -- 172.21.15.28:0/2072281645 shutdown_connections 2024-02-21T23:10:50.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.487+0000 7fd2db200640 1 -- 172.21.15.28:0/2072281645 wait complete. 2024-02-21T23:10:50.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.489+0000 7fd2db200640 1 Processor -- start 2024-02-21T23:10:50.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.489+0000 7fd2db200640 1 -- start start 2024-02-21T23:10:50.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.490+0000 7fd2db200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd2d406f670 0x7fd2d41ee890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:50.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.490+0000 7fd2db200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2d4150230 0x7fd2d41eedd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:50.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.490+0000 7fd2db200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2d40ad830 0x7fd2d40adcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:50.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.490+0000 7fd2db200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d40712f0 con 0x7fd2d40ad830 2024-02-21T23:10:50.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.490+0000 7fd2db200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d4150610 con 0x7fd2d4150230 2024-02-21T23:10:50.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.490+0000 7fd2db200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d40c4e30 con 0x7fd2d406f670 2024-02-21T23:10:50.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.490+0000 7fd2da800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2d40ad830 0x7fd2d40adcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:50.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.490+0000 7fd2da800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2d40ad830 0x7fd2d40adcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49276/0 (socket says 172.21.15.28:49276) 2024-02-21T23:10:50.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.490+0000 7fd2da800640 1 -- 172.21.15.28:0/3682931536 learned_addr learned my addr 172.21.15.28:0/3682931536 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:50.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.491+0000 7fd2d9400640 1 --2- 172.21.15.28:0/3682931536 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2d4150230 0x7fd2d41eedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:50.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.491+0000 7fd2d9e00640 1 --2- 172.21.15.28:0/3682931536 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd2d406f670 0x7fd2d41ee890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:50.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.491+0000 7fd2da800640 1 -- 172.21.15.28:0/3682931536 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd2d406f670 msgr2=0x7fd2d41ee890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:50.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.491+0000 7fd2da800640 1 --2- 172.21.15.28:0/3682931536 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd2d406f670 0x7fd2d41ee890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:50.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.491+0000 7fd2da800640 1 -- 172.21.15.28:0/3682931536 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2d4150230 msgr2=0x7fd2d41eedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:50.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.491+0000 7fd2da800640 1 --2- 172.21.15.28:0/3682931536 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2d4150230 0x7fd2d41eedd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:50.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.491+0000 7fd2da800640 1 -- 172.21.15.28:0/3682931536 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2d41dee20 con 0x7fd2d40ad830 2024-02-21T23:10:50.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.491+0000 7fd2da800640 1 --2- 172.21.15.28:0/3682931536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2d40ad830 0x7fd2d40adcc0 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7fd2d000c780 tx=0x7fd2d000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:50.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.491+0000 7fd2ce000640 1 -- 172.21.15.28:0/3682931536 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d0004b60 con 0x7fd2d40ad830 2024-02-21T23:10:50.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.491+0000 7fd2ce000640 1 -- 172.21.15.28:0/3682931536 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2d0004d00 con 0x7fd2d40ad830 2024-02-21T23:10:50.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.491+0000 7fd2ce000640 1 -- 172.21.15.28:0/3682931536 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d000aa90 con 0x7fd2d40ad830 2024-02-21T23:10:50.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.491+0000 7fd2db200640 1 -- 172.21.15.28:0/3682931536 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2d4151b00 con 0x7fd2d40ad830 2024-02-21T23:10:50.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.491+0000 7fd2db200640 1 -- 172.21.15.28:0/3682931536 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2d40ae200 con 0x7fd2d40ad830 2024-02-21T23:10:50.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.493+0000 7fd2db200640 1 -- 172.21.15.28:0/3682931536 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2d404f290 con 0x7fd2d40ad830 2024-02-21T23:10:50.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.494+0000 7fd2ce000640 1 -- 172.21.15.28:0/3682931536 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd2d000ac30 con 0x7fd2d40ad830 2024-02-21T23:10:50.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.494+0000 7fd2ce000640 1 --2- 172.21.15.28:0/3682931536 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd2b806b4d0 0x7fd2b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:50.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.495+0000 7fd2ce000640 1 -- 172.21.15.28:0/3682931536 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fd2d008a3b0 con 0x7fd2d40ad830 2024-02-21T23:10:50.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.495+0000 7fd2d9e00640 1 --2- 172.21.15.28:0/3682931536 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd2b806b4d0 0x7fd2b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:50.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.496+0000 7fd2d9e00640 1 --2- 172.21.15.28:0/3682931536 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd2b806b4d0 0x7fd2b806d990 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fd2c8000f60 tx=0x7fd2c8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:50.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.499+0000 7fd2ce000640 1 -- 172.21.15.28:0/3682931536 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd2d0020070 con 0x7fd2d40ad830 2024-02-21T23:10:50.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.698+0000 7fd2db200640 1 -- 172.21.15.28:0/3682931536 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fd2d40c8160 con 0x7fd2d40ad830 2024-02-21T23:10:50.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.699+0000 7fd2ce000640 1 -- 172.21.15.28:0/3682931536 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v170) v1 ==== 84+0+65 (secure 0 0 0) 0x7fd2d0088e20 con 0x7fd2d40ad830 2024-02-21T23:10:50.688 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:10:50.688 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-02-21T23:10:50.688 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:10:50.688 INFO:teuthology.orchestra.run.smithi028.stdout: "poolnum": 1, 2024-02-21T23:10:50.688 INFO:teuthology.orchestra.run.smithi028.stdout: "poolname": ".mgr" 2024-02-21T23:10:50.688 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:10:50.688 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-02-21T23:10:50.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.701+0000 7fd2db200640 1 -- 172.21.15.28:0/3682931536 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd2b806b4d0 msgr2=0x7fd2b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:50.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.701+0000 7fd2db200640 1 --2- 172.21.15.28:0/3682931536 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd2b806b4d0 0x7fd2b806d990 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fd2c8000f60 tx=0x7fd2c8009040 comp rx=0 tx=0).stop 2024-02-21T23:10:50.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.701+0000 7fd2db200640 1 -- 172.21.15.28:0/3682931536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2d40ad830 msgr2=0x7fd2d40adcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:50.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.701+0000 7fd2db200640 1 --2- 172.21.15.28:0/3682931536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2d40ad830 0x7fd2d40adcc0 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7fd2d000c780 tx=0x7fd2d000cc40 comp rx=0 tx=0).stop 2024-02-21T23:10:50.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.701+0000 7fd2db200640 1 -- 172.21.15.28:0/3682931536 shutdown_connections 2024-02-21T23:10:50.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.701+0000 7fd2db200640 1 --2- 172.21.15.28:0/3682931536 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd2b806b4d0 0x7fd2b806d990 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:50.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.701+0000 7fd2db200640 1 --2- 172.21.15.28:0/3682931536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2d40ad830 0x7fd2d40adcc0 unknown :-1 s=CLOSED pgs=1902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:50.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.701+0000 7fd2db200640 1 --2- 172.21.15.28:0/3682931536 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2d4150230 0x7fd2d41eedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:50.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.701+0000 7fd2db200640 1 --2- 172.21.15.28:0/3682931536 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd2d406f670 0x7fd2d41ee890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:50.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.701+0000 7fd2db200640 1 -- 172.21.15.28:0/3682931536 >> 172.21.15.28:0/3682931536 conn(0x7fd2d40aa3d0 msgr2=0x7fd2d40c42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:50.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.702+0000 7fd2db200640 1 -- 172.21.15.28:0/3682931536 shutdown_connections 2024-02-21T23:10:50.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.702+0000 7fd2db200640 1 -- 172.21.15.28:0/3682931536 wait complete. 2024-02-21T23:10:50.710 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-02-21T23:10:50.710 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-02-21T23:10:50.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.948+0000 7f1b86c00640 1 Processor -- start 2024-02-21T23:10:50.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.948+0000 7f1b86c00640 1 -- start start 2024-02-21T23:10:50.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.950+0000 7f1b86c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f1b800c2a90 0x7f1b8006b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:50.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.950+0000 7f1b86c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1b80155870 0x7f1b8015c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:50.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.950+0000 7f1b86c00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b801527d0 con 0x7f1b8006ae70 2024-02-21T23:10:50.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.950+0000 7f1b86c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b800724f0 con 0x7f1b80155870 2024-02-21T23:10:50.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.950+0000 7f1b86c00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b800727d0 con 0x7f1b800c2a90 2024-02-21T23:10:50.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.950+0000 7f1b86200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1b80155870 0x7f1b8015c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:50.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.950+0000 7f1b86200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1b80155870 0x7f1b8015c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49284/0 (socket says 172.21.15.28:49284) 2024-02-21T23:10:50.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.950+0000 7f1b86200640 1 -- 172.21.15.28:0/245451763 learned_addr learned my addr 172.21.15.28:0/245451763 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:50.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.951+0000 7f1b84e00640 1 --2- 172.21.15.28:0/245451763 >> v2:172.21.15.81:3300/0 conn(0x7f1b800c2a90 0x7f1b8006b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:50.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.951+0000 7f1b86200640 1 -- 172.21.15.28:0/245451763 >> v2:172.21.15.81:3300/0 conn(0x7f1b800c2a90 msgr2=0x7f1b8006b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:50.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.951+0000 7f1b86200640 1 --2- 172.21.15.28:0/245451763 >> v2:172.21.15.81:3300/0 conn(0x7f1b800c2a90 0x7f1b8006b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:50.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.951+0000 7f1b86200640 1 -- 172.21.15.28:0/245451763 >> v1:172.21.15.81:6789/0 conn(0x7f1b8006ae70 legacy=0x7f1b80155160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:50.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.951+0000 7f1b86200640 1 -- 172.21.15.28:0/245451763 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b8006f720 con 0x7f1b80155870 2024-02-21T23:10:50.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.951+0000 7f1b84e00640 1 --2- 172.21.15.28:0/245451763 >> v2:172.21.15.81:3300/0 conn(0x7f1b800c2a90 0x7f1b8006b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:10:50.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.952+0000 7f1b86200640 1 --2- 172.21.15.28:0/245451763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b80155870 0x7f1b8015c3f0 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7f1b800c75f0 tx=0x7f1b7802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd8646917ef0b0bf server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:50.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.952+0000 7f1b7fe00640 1 -- 172.21.15.28:0/245451763 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b7802ca80 con 0x7f1b80155870 2024-02-21T23:10:50.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.952+0000 7f1b7fe00640 1 -- 172.21.15.28:0/245451763 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b7802cc20 con 0x7f1b80155870 2024-02-21T23:10:50.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.952+0000 7f1b7fe00640 1 -- 172.21.15.28:0/245451763 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b780343f0 con 0x7f1b80155870 2024-02-21T23:10:50.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.953+0000 7f1b86c00640 1 -- 172.21.15.28:0/245451763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b80155870 msgr2=0x7f1b8015c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:50.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.953+0000 7f1b86c00640 1 --2- 172.21.15.28:0/245451763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b80155870 0x7f1b8015c3f0 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7f1b800c75f0 tx=0x7f1b7802bee0 comp rx=0 tx=0).stop 2024-02-21T23:10:50.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.953+0000 7f1b86c00640 1 -- 172.21.15.28:0/245451763 shutdown_connections 2024-02-21T23:10:50.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.953+0000 7f1b86c00640 1 --2- 172.21.15.28:0/245451763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b80155870 0x7f1b8015c3f0 unknown :-1 s=CLOSED pgs=1903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:50.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.953+0000 7f1b86c00640 1 --2- 172.21.15.28:0/245451763 >> v2:172.21.15.81:3300/0 conn(0x7f1b800c2a90 0x7f1b8006b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:50.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.953+0000 7f1b86c00640 1 -- 172.21.15.28:0/245451763 >> 172.21.15.28:0/245451763 conn(0x7f1b800c6610 msgr2=0x7f1b800c69f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:50.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.953+0000 7f1b86c00640 1 -- 172.21.15.28:0/245451763 shutdown_connections 2024-02-21T23:10:50.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.953+0000 7f1b86c00640 1 -- 172.21.15.28:0/245451763 wait complete. 2024-02-21T23:10:50.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.955+0000 7f1b86c00640 1 Processor -- start 2024-02-21T23:10:50.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.955+0000 7f1b86c00640 1 -- start start 2024-02-21T23:10:50.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.956+0000 7f1b86c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1b8006ae70 0x7f1b801edfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:50.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.956+0000 7f1b86c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b800c2a90 0x7f1b80147900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:50.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.956+0000 7f1b86c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1b80155870 0x7f1b800ab6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:50.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.956+0000 7f1b86c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b80155d00 con 0x7f1b800c2a90 2024-02-21T23:10:50.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.956+0000 7f1b86c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b800c2240 con 0x7f1b80155870 2024-02-21T23:10:50.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.956+0000 7f1b86c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b800ab230 con 0x7f1b8006ae70 2024-02-21T23:10:50.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.956+0000 7f1b84e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b800c2a90 0x7f1b80147900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:50.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.956+0000 7f1b84e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b800c2a90 0x7f1b80147900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49296/0 (socket says 172.21.15.28:49296) 2024-02-21T23:10:50.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.956+0000 7f1b84e00640 1 -- 172.21.15.28:0/4099333081 learned_addr learned my addr 172.21.15.28:0/4099333081 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:50.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.957+0000 7f1b85800640 1 --2- 172.21.15.28:0/4099333081 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1b8006ae70 0x7f1b801edfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:50.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.957+0000 7f1b86200640 1 --2- 172.21.15.28:0/4099333081 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1b80155870 0x7f1b800ab6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:50.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.957+0000 7f1b84e00640 1 -- 172.21.15.28:0/4099333081 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1b8006ae70 msgr2=0x7f1b801edfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:50.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.957+0000 7f1b84e00640 1 --2- 172.21.15.28:0/4099333081 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1b8006ae70 0x7f1b801edfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:50.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.957+0000 7f1b84e00640 1 -- 172.21.15.28:0/4099333081 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1b80155870 msgr2=0x7f1b800ab6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:50.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.957+0000 7f1b84e00640 1 --2- 172.21.15.28:0/4099333081 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1b80155870 0x7f1b800ab6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:50.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.957+0000 7f1b84e00640 1 -- 172.21.15.28:0/4099333081 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b80151300 con 0x7f1b800c2a90 2024-02-21T23:10:50.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.957+0000 7f1b84e00640 1 --2- 172.21.15.28:0/4099333081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b800c2a90 0x7f1b80147900 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7f1b6c00c7f0 tx=0x7f1b6c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:50.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.958+0000 7f1b7d600640 1 -- 172.21.15.28:0/4099333081 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b6c015070 con 0x7f1b800c2a90 2024-02-21T23:10:50.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.958+0000 7f1b7d600640 1 -- 172.21.15.28:0/4099333081 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b6c007a30 con 0x7f1b800c2a90 2024-02-21T23:10:50.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.958+0000 7f1b7d600640 1 -- 172.21.15.28:0/4099333081 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b6c007e00 con 0x7f1b800c2a90 2024-02-21T23:10:50.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.958+0000 7f1b86c00640 1 -- 172.21.15.28:0/4099333081 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b801515e0 con 0x7f1b800c2a90 2024-02-21T23:10:50.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.958+0000 7f1b86c00640 1 -- 172.21.15.28:0/4099333081 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b80151ba0 con 0x7f1b800c2a90 2024-02-21T23:10:50.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.959+0000 7f1b86c00640 1 -- 172.21.15.28:0/4099333081 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b8004f290 con 0x7f1b800c2a90 2024-02-21T23:10:50.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.960+0000 7f1b7d600640 1 -- 172.21.15.28:0/4099333081 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1b6c018070 con 0x7f1b800c2a90 2024-02-21T23:10:50.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.961+0000 7f1b7d600640 1 --2- 172.21.15.28:0/4099333081 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1b6006b4d0 0x7f1b6006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:50.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.961+0000 7f1b7d600640 1 -- 172.21.15.28:0/4099333081 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f1b6c08a090 con 0x7f1b800c2a90 2024-02-21T23:10:50.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.961+0000 7f1b85800640 1 --2- 172.21.15.28:0/4099333081 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1b6006b4d0 0x7f1b6006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:50.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.962+0000 7f1b85800640 1 --2- 172.21.15.28:0/4099333081 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1b6006b4d0 0x7f1b6006d990 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f1b74004ea0 tx=0x7f1b74009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:50.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:50.965+0000 7f1b7d600640 1 -- 172.21.15.28:0/4099333081 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1b6c059030 con 0x7f1b800c2a90 2024-02-21T23:10:51.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:51.169+0000 7f1b86c00640 1 -- 172.21.15.28:0/4099333081 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f1b80072cb0 con 0x7f1b800c2a90 2024-02-21T23:10:51.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:51.840+0000 7f1b7d600640 1 -- 172.21.15.28:0/4099333081 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v171) v1 ==== 138+0+0 (secure 0 0 0) 0x7f1b6c058930 con 0x7f1b800c2a90 2024-02-21T23:10:51.830 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' created 2024-02-21T23:10:51.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:51.844+0000 7f1b86c00640 1 -- 172.21.15.28:0/4099333081 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1b6006b4d0 msgr2=0x7f1b6006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:51.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:51.844+0000 7f1b86c00640 1 --2- 172.21.15.28:0/4099333081 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1b6006b4d0 0x7f1b6006d990 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f1b74004ea0 tx=0x7f1b74009040 comp rx=0 tx=0).stop 2024-02-21T23:10:51.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:51.844+0000 7f1b86c00640 1 -- 172.21.15.28:0/4099333081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b800c2a90 msgr2=0x7f1b80147900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:51.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:51.844+0000 7f1b86c00640 1 --2- 172.21.15.28:0/4099333081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b800c2a90 0x7f1b80147900 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7f1b6c00c7f0 tx=0x7f1b6c00cbf0 comp rx=0 tx=0).stop 2024-02-21T23:10:51.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:51.844+0000 7f1b86c00640 1 -- 172.21.15.28:0/4099333081 shutdown_connections 2024-02-21T23:10:51.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:51.844+0000 7f1b86c00640 1 --2- 172.21.15.28:0/4099333081 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1b6006b4d0 0x7f1b6006d990 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:51.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:51.845+0000 7f1b86c00640 1 --2- 172.21.15.28:0/4099333081 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1b80155870 0x7f1b800ab6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:51.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:51.845+0000 7f1b86c00640 1 --2- 172.21.15.28:0/4099333081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b800c2a90 0x7f1b80147900 secure :-1 s=CLOSED pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7f1b6c00c7f0 tx=0x7f1b6c00cbf0 comp rx=0 tx=0).stop 2024-02-21T23:10:51.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:51.845+0000 7f1b86c00640 1 --2- 172.21.15.28:0/4099333081 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1b8006ae70 0x7f1b801edfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:51.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:51.845+0000 7f1b86c00640 1 -- 172.21.15.28:0/4099333081 >> 172.21.15.28:0/4099333081 conn(0x7f1b800c6610 msgr2=0x7f1b800c3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:51.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:51.845+0000 7f1b86c00640 1 -- 172.21.15.28:0/4099333081 shutdown_connections 2024-02-21T23:10:51.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:51.845+0000 7f1b86c00640 1 -- 172.21.15.28:0/4099333081 wait complete. 2024-02-21T23:10:51.865 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-02-21T23:10:52.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.091+0000 7f1aa3000640 1 Processor -- start 2024-02-21T23:10:52.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.091+0000 7f1aa3000640 1 -- start start 2024-02-21T23:10:52.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.092+0000 7f1aa3000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f1a9c0c35c0 0x7f1a9c156bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:52.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.093+0000 7f1aa3000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1a9c150640 0x7f1a9c152c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:52.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.093+0000 7f1aa3000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a9c06f1b0 con 0x7f1a9c0a96c0 2024-02-21T23:10:52.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.093+0000 7f1aa3000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a9c06ea40 con 0x7f1a9c150640 2024-02-21T23:10:52.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.093+0000 7f1aa3000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a9c0c2ae0 con 0x7f1a9c0c35c0 2024-02-21T23:10:52.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.093+0000 7f1aa2600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1a9c150640 0x7f1a9c152c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:52.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.093+0000 7f1aa2600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1a9c150640 0x7f1a9c152c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49308/0 (socket says 172.21.15.28:49308) 2024-02-21T23:10:52.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.093+0000 7f1aa2600640 1 -- 172.21.15.28:0/1617804020 learned_addr learned my addr 172.21.15.28:0/1617804020 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:52.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.093+0000 7f1aa1c00640 1 --2- 172.21.15.28:0/1617804020 >> v2:172.21.15.81:3300/0 conn(0x7f1a9c0c35c0 0x7f1a9c156bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:52.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.094+0000 7f1aa2600640 1 -- 172.21.15.28:0/1617804020 >> v2:172.21.15.81:3300/0 conn(0x7f1a9c0c35c0 msgr2=0x7f1a9c156bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:52.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.094+0000 7f1aa2600640 1 --2- 172.21.15.28:0/1617804020 >> v2:172.21.15.81:3300/0 conn(0x7f1a9c0c35c0 0x7f1a9c156bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:52.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.094+0000 7f1aa2600640 1 -- 172.21.15.28:0/1617804020 >> v1:172.21.15.81:6789/0 conn(0x7f1a9c0a96c0 legacy=0x7f1a9c14ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:52.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.094+0000 7f1aa2600640 1 -- 172.21.15.28:0/1617804020 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a9c0c2dc0 con 0x7f1a9c150640 2024-02-21T23:10:52.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.095+0000 7f1aa2600640 1 --2- 172.21.15.28:0/1617804020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a9c150640 0x7f1a9c152c40 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7f1a9c0734e0 tx=0x7f1a9402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a56d523ac2b06b49 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:52.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.096+0000 7f1a9be00640 1 -- 172.21.15.28:0/1617804020 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a9402ca80 con 0x7f1a9c150640 2024-02-21T23:10:52.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.096+0000 7f1a9be00640 1 -- 172.21.15.28:0/1617804020 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a9402cc20 con 0x7f1a9c150640 2024-02-21T23:10:52.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.096+0000 7f1a9be00640 1 -- 172.21.15.28:0/1617804020 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a940343f0 con 0x7f1a9c150640 2024-02-21T23:10:52.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.096+0000 7f1aa3000640 1 -- 172.21.15.28:0/1617804020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a9c150640 msgr2=0x7f1a9c152c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:52.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.096+0000 7f1aa3000640 1 --2- 172.21.15.28:0/1617804020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a9c150640 0x7f1a9c152c40 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7f1a9c0734e0 tx=0x7f1a9402bee0 comp rx=0 tx=0).stop 2024-02-21T23:10:52.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.096+0000 7f1aa3000640 1 -- 172.21.15.28:0/1617804020 shutdown_connections 2024-02-21T23:10:52.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.096+0000 7f1aa3000640 1 --2- 172.21.15.28:0/1617804020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a9c150640 0x7f1a9c152c40 unknown :-1 s=CLOSED pgs=1905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:52.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.096+0000 7f1aa3000640 1 --2- 172.21.15.28:0/1617804020 >> v2:172.21.15.81:3300/0 conn(0x7f1a9c0c35c0 0x7f1a9c156bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:52.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.096+0000 7f1aa3000640 1 -- 172.21.15.28:0/1617804020 >> 172.21.15.28:0/1617804020 conn(0x7f1a9c0a8690 msgr2=0x7f1a9c0a8a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:52.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.097+0000 7f1aa3000640 1 -- 172.21.15.28:0/1617804020 shutdown_connections 2024-02-21T23:10:52.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.097+0000 7f1aa3000640 1 -- 172.21.15.28:0/1617804020 wait complete. 2024-02-21T23:10:52.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.098+0000 7f1aa3000640 1 Processor -- start 2024-02-21T23:10:52.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.099+0000 7f1aa3000640 1 -- start start 2024-02-21T23:10:52.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.099+0000 7f1aa3000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a9c0a96c0 0x7f1a9c141ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:52.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.099+0000 7f1aa3000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1a9c0c35c0 0x7f1a9c142220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:52.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.099+0000 7f1aa3000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a9c150640 0x7f1a9c1eb590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:52.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.099+0000 7f1aa3000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a9c0c39a0 con 0x7f1a9c150640 2024-02-21T23:10:52.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.099+0000 7f1aa3000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a9c06e7b0 con 0x7f1a9c0a96c0 2024-02-21T23:10:52.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.099+0000 7f1aa3000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a9c0c3b20 con 0x7f1a9c0c35c0 2024-02-21T23:10:52.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.100+0000 7f1aa2600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a9c150640 0x7f1a9c1eb590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:52.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.100+0000 7f1aa2600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a9c150640 0x7f1a9c1eb590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49314/0 (socket says 172.21.15.28:49314) 2024-02-21T23:10:52.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.100+0000 7f1aa2600640 1 -- 172.21.15.28:0/433889341 learned_addr learned my addr 172.21.15.28:0/433889341 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:52.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.100+0000 7f1aa1c00640 1 --2- 172.21.15.28:0/433889341 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a9c0a96c0 0x7f1a9c141ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:52.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.100+0000 7f1aa1200640 1 --2- 172.21.15.28:0/433889341 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1a9c0c35c0 0x7f1a9c142220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:52.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.100+0000 7f1aa2600640 1 -- 172.21.15.28:0/433889341 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1a9c0c35c0 msgr2=0x7f1a9c142220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:52.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.100+0000 7f1aa2600640 1 --2- 172.21.15.28:0/433889341 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1a9c0c35c0 0x7f1a9c142220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:52.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.100+0000 7f1aa2600640 1 -- 172.21.15.28:0/433889341 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a9c0a96c0 msgr2=0x7f1a9c141ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:52.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.100+0000 7f1aa2600640 1 --2- 172.21.15.28:0/433889341 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a9c0a96c0 0x7f1a9c141ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:52.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.100+0000 7f1aa2600640 1 -- 172.21.15.28:0/433889341 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a9c14a990 con 0x7f1a9c150640 2024-02-21T23:10:52.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.100+0000 7f1aa1200640 1 --2- 172.21.15.28:0/433889341 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1a9c0c35c0 0x7f1a9c142220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:10:52.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.101+0000 7f1aa2600640 1 --2- 172.21.15.28:0/433889341 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a9c150640 0x7f1a9c1eb590 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7f1a9402c4f0 tx=0x7f1a94004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:52.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.101+0000 7f1a99600640 1 -- 172.21.15.28:0/433889341 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a94043020 con 0x7f1a9c150640 2024-02-21T23:10:52.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.101+0000 7f1a99600640 1 -- 172.21.15.28:0/433889341 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a9403e450 con 0x7f1a9c150640 2024-02-21T23:10:52.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.101+0000 7f1aa3000640 1 -- 172.21.15.28:0/433889341 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a9c151e90 con 0x7f1a9c150640 2024-02-21T23:10:52.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.101+0000 7f1a99600640 1 -- 172.21.15.28:0/433889341 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a94009510 con 0x7f1a9c150640 2024-02-21T23:10:52.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.101+0000 7f1aa3000640 1 -- 172.21.15.28:0/433889341 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a9c152450 con 0x7f1a9c150640 2024-02-21T23:10:52.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.102+0000 7f1aa3000640 1 -- 172.21.15.28:0/433889341 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a9c04f290 con 0x7f1a9c150640 2024-02-21T23:10:52.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.104+0000 7f1a99600640 1 -- 172.21.15.28:0/433889341 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1a94046020 con 0x7f1a9c150640 2024-02-21T23:10:52.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.104+0000 7f1a99600640 1 --2- 172.21.15.28:0/433889341 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1a7c06b4d0 0x7f1a7c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:52.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.105+0000 7f1a99600640 1 -- 172.21.15.28:0/433889341 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 5619+0+0 (secure 0 0 0) 0x7f1a940aca40 con 0x7f1a9c150640 2024-02-21T23:10:52.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.105+0000 7f1aa1c00640 1 --2- 172.21.15.28:0/433889341 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1a7c06b4d0 0x7f1a7c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:52.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.106+0000 7f1aa1c00640 1 --2- 172.21.15.28:0/433889341 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1a7c06b4d0 0x7f1a7c06d990 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f1a90009ad0 tx=0x7f1a90008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:52.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.110+0000 7f1a99600640 1 -- 172.21.15.28:0/433889341 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1a9404d030 con 0x7f1a9c150640 2024-02-21T23:10:52.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.311+0000 7f1aa3000640 1 -- 172.21.15.28:0/433889341 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f1a9c14ef40 con 0x7f1a9c150640 2024-02-21T23:10:52.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.859+0000 7f1a99600640 1 -- 172.21.15.28:0/433889341 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v172) v1 ==== 170+0+0 (secure 0 0 0) 0x7f1a9407b7e0 con 0x7f1a9c150640 2024-02-21T23:10:52.847 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' created 2024-02-21T23:10:52.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.864+0000 7f1aa3000640 1 -- 172.21.15.28:0/433889341 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1a7c06b4d0 msgr2=0x7f1a7c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:52.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.864+0000 7f1aa3000640 1 --2- 172.21.15.28:0/433889341 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1a7c06b4d0 0x7f1a7c06d990 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f1a90009ad0 tx=0x7f1a90008040 comp rx=0 tx=0).stop 2024-02-21T23:10:52.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.864+0000 7f1aa3000640 1 -- 172.21.15.28:0/433889341 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a9c150640 msgr2=0x7f1a9c1eb590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:52.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.864+0000 7f1aa3000640 1 --2- 172.21.15.28:0/433889341 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a9c150640 0x7f1a9c1eb590 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7f1a9402c4f0 tx=0x7f1a94004a40 comp rx=0 tx=0).stop 2024-02-21T23:10:52.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.864+0000 7f1aa3000640 1 -- 172.21.15.28:0/433889341 shutdown_connections 2024-02-21T23:10:52.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.864+0000 7f1aa3000640 1 --2- 172.21.15.28:0/433889341 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1a7c06b4d0 0x7f1a7c06d990 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:52.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.864+0000 7f1aa3000640 1 --2- 172.21.15.28:0/433889341 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1a9c0c35c0 0x7f1a9c142220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:52.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.864+0000 7f1aa3000640 1 --2- 172.21.15.28:0/433889341 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a9c150640 0x7f1a9c1eb590 unknown :-1 s=CLOSED pgs=1906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:52.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.864+0000 7f1aa3000640 1 --2- 172.21.15.28:0/433889341 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a9c0a96c0 0x7f1a9c141ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:52.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.864+0000 7f1aa3000640 1 -- 172.21.15.28:0/433889341 >> 172.21.15.28:0/433889341 conn(0x7f1a9c0a8690 msgr2=0x7f1a9c0ab4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:52.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.865+0000 7f1aa3000640 1 -- 172.21.15.28:0/433889341 shutdown_connections 2024-02-21T23:10:52.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:52.865+0000 7f1aa3000640 1 -- 172.21.15.28:0/433889341 wait complete. 2024-02-21T23:10:52.885 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-02-21T23:10:53.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.086+0000 7f6b02e00640 1 Processor -- start 2024-02-21T23:10:53.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.086+0000 7f6b02e00640 1 -- start start 2024-02-21T23:10:53.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.087+0000 7f6b02e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6afc0c5a70 0x7f6afc155a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:53.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.087+0000 7f6b02e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6afc071970 con 0x7f6afc0c22e0 2024-02-21T23:10:53.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.088+0000 7f6b02e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6afc071c50 con 0x7f6afc0c2b30 2024-02-21T23:10:53.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.088+0000 7f6b02e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6afc072a10 con 0x7f6afc0c5a70 2024-02-21T23:10:53.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.088+0000 7f6afb400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f6afc0c22e0 0x7f6afc06d120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41292/0 (socket says 172.21.15.28:41292) 2024-02-21T23:10:53.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.088+0000 7f6afb400640 1 -- 172.21.15.28:0/2708794947 learned_addr learned my addr 172.21.15.28:0/2708794947 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:10:53.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.088+0000 7f6afbe00640 1 --2- 172.21.15.28:0/2708794947 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6afc0c5a70 0x7f6afc155a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:53.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.089+0000 7f6afaa00640 1 -- 172.21.15.28:0/2708794947 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3828072251 0 0) 0x7f6afc071970 con 0x7f6afc0c22e0 2024-02-21T23:10:53.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.089+0000 7f6afaa00640 1 -- 172.21.15.28:0/2708794947 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ae0003520 con 0x7f6afc0c22e0 2024-02-21T23:10:53.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.089+0000 7f6afaa00640 1 -- 172.21.15.28:0/2708794947 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1935690615 0 0) 0x7f6ae0003520 con 0x7f6afc0c22e0 2024-02-21T23:10:53.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.089+0000 7f6afaa00640 1 -- 172.21.15.28:0/2708794947 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6afc0c5a70 msgr2=0x7f6afc155a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:53.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.089+0000 7f6afaa00640 1 --2- 172.21.15.28:0/2708794947 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6afc0c5a70 0x7f6afc155a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:53.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.089+0000 7f6afaa00640 1 -- 172.21.15.28:0/2708794947 >> v1:172.21.15.81:6789/0 conn(0x7f6afc0c2b30 legacy=0x7f6afc15c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:53.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.089+0000 7f6afaa00640 1 -- 172.21.15.28:0/2708794947 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6afc071f30 con 0x7f6afc0c22e0 2024-02-21T23:10:53.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.089+0000 7f6afaa00640 1 -- 172.21.15.28:0/2708794947 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f6ae8001670 con 0x7f6afc0c22e0 2024-02-21T23:10:53.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.089+0000 7f6afaa00640 1 -- 172.21.15.28:0/2708794947 >> v1:172.21.15.28:6789/0 conn(0x7f6afc0c22e0 legacy=0x7f6afc06d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:53.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.089+0000 7f6afaa00640 1 --2- 172.21.15.28:0/2708794947 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ae00039e0 0x7f6ae0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:53.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.089+0000 7f6afaa00640 1 --2- 172.21.15.28:0/2708794947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ae0004800 0x7f6ae0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:53.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.089+0000 7f6afaa00640 1 --2- 172.21.15.28:0/2708794947 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6afc0c5a70 0x7f6ae0005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:53.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.089+0000 7f6afaa00640 1 -- 172.21.15.28:0/2708794947 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6afc072a10 con 0x7f6ae0004800 2024-02-21T23:10:53.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.089+0000 7f6afaa00640 1 -- 172.21.15.28:0/2708794947 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae000baf0 con 0x7f6ae00039e0 2024-02-21T23:10:53.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.089+0000 7f6afaa00640 1 -- 172.21.15.28:0/2708794947 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae000bcf0 con 0x7f6afc0c5a70 2024-02-21T23:10:53.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.090+0000 7f6afb400640 1 --2- 172.21.15.28:0/2708794947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ae0004800 0x7f6ae0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:53.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.090+0000 7f6afbe00640 1 --2- 172.21.15.28:0/2708794947 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ae00039e0 0x7f6ae0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:53.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.090+0000 7f6b00e00640 1 --2- 172.21.15.28:0/2708794947 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6afc0c5a70 0x7f6ae0005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:53.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.090+0000 7f6afb400640 1 -- 172.21.15.28:0/2708794947 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6afc0c5a70 msgr2=0x7f6ae0005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:53.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.090+0000 7f6afb400640 1 --2- 172.21.15.28:0/2708794947 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6afc0c5a70 0x7f6ae0005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:53.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.090+0000 7f6afb400640 1 -- 172.21.15.28:0/2708794947 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ae00039e0 msgr2=0x7f6ae0003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:53.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.090+0000 7f6afb400640 1 --2- 172.21.15.28:0/2708794947 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ae00039e0 0x7f6ae0003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:53.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.090+0000 7f6afb400640 1 -- 172.21.15.28:0/2708794947 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ae0003520 con 0x7f6ae0004800 2024-02-21T23:10:53.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.091+0000 7f6afb400640 1 --2- 172.21.15.28:0/2708794947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ae0004800 0x7f6ae0004c80 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7f6ae800c210 tx=0x7f6ae0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bb33c14b98fc37 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:53.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.091+0000 7f6afaa00640 1 -- 172.21.15.28:0/2708794947 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ae8007d10 con 0x7f6ae0004800 2024-02-21T23:10:53.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.092+0000 7f6afaa00640 1 -- 172.21.15.28:0/2708794947 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ae8030070 con 0x7f6ae0004800 2024-02-21T23:10:53.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.092+0000 7f6afaa00640 1 -- 172.21.15.28:0/2708794947 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ae8005d50 con 0x7f6ae0004800 2024-02-21T23:10:53.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.092+0000 7f6b02e00640 1 -- 172.21.15.28:0/2708794947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ae0004800 msgr2=0x7f6ae0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:53.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.092+0000 7f6b02e00640 1 --2- 172.21.15.28:0/2708794947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ae0004800 0x7f6ae0004c80 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7f6ae800c210 tx=0x7f6ae0005830 comp rx=0 tx=0).stop 2024-02-21T23:10:53.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.092+0000 7f6b00e00640 1 -- 172.21.15.28:0/2708794947 reap_dead start 2024-02-21T23:10:53.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.092+0000 7f6b02e00640 1 -- 172.21.15.28:0/2708794947 shutdown_connections 2024-02-21T23:10:53.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.092+0000 7f6b02e00640 1 -- 172.21.15.28:0/2708794947 >> 172.21.15.28:0/2708794947 conn(0x7f6afc0a85a0 msgr2=0x7f6afc0a8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:53.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.092+0000 7f6b02e00640 1 -- 172.21.15.28:0/2708794947 shutdown_connections 2024-02-21T23:10:53.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.092+0000 7f6b02e00640 1 -- 172.21.15.28:0/2708794947 wait complete. 2024-02-21T23:10:53.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.094+0000 7f6b02e00640 1 Processor -- start 2024-02-21T23:10:53.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.094+0000 7f6b02e00640 1 -- start start 2024-02-21T23:10:53.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.095+0000 7f6b02e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6ae0004800 0x7f6afc153c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:53.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.095+0000 7f6b02e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6afc154140 0x7f6afc152960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:53.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.095+0000 7f6b02e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6afc152ea0 0x7f6afc1edf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:53.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.095+0000 7f6b02e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6afc0a6160 con 0x7f6afc154140 2024-02-21T23:10:53.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.095+0000 7f6b02e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6afc072580 con 0x7f6afc152ea0 2024-02-21T23:10:53.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.095+0000 7f6b02e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6afc0a62e0 con 0x7f6ae0004800 2024-02-21T23:10:53.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.095+0000 7f6afb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6afc154140 0x7f6afc152960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:53.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.096+0000 7f6afb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6afc154140 0x7f6afc152960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49324/0 (socket says 172.21.15.28:49324) 2024-02-21T23:10:53.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.096+0000 7f6afb400640 1 -- 172.21.15.28:0/2479443877 learned_addr learned my addr 172.21.15.28:0/2479443877 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:53.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.096+0000 7f6afbe00640 1 --2- 172.21.15.28:0/2479443877 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6ae0004800 0x7f6afc153c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:53.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.096+0000 7f6b00e00640 1 --2- 172.21.15.28:0/2479443877 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6afc152ea0 0x7f6afc1edf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:53.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.096+0000 7f6afb400640 1 -- 172.21.15.28:0/2479443877 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6ae0004800 msgr2=0x7f6afc153c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:53.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.096+0000 7f6afb400640 1 --2- 172.21.15.28:0/2479443877 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6ae0004800 0x7f6afc153c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:53.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.096+0000 7f6afb400640 1 -- 172.21.15.28:0/2479443877 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6afc152ea0 msgr2=0x7f6afc1edf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:53.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.096+0000 7f6afb400640 1 --2- 172.21.15.28:0/2479443877 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6afc152ea0 0x7f6afc1edf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:53.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.096+0000 7f6afb400640 1 -- 172.21.15.28:0/2479443877 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6afc156860 con 0x7f6afc154140 2024-02-21T23:10:53.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.096+0000 7f6afb400640 1 --2- 172.21.15.28:0/2479443877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6afc154140 0x7f6afc152960 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7f6ae8001970 tx=0x7f6afc0724f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:53.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.097+0000 7f6af3e00640 1 -- 172.21.15.28:0/2479443877 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ae8045070 con 0x7f6afc154140 2024-02-21T23:10:53.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.097+0000 7f6af3e00640 1 -- 172.21.15.28:0/2479443877 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ae8004bf0 con 0x7f6afc154140 2024-02-21T23:10:53.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.097+0000 7f6b02e00640 1 -- 172.21.15.28:0/2479443877 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6afc154660 con 0x7f6afc154140 2024-02-21T23:10:53.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.097+0000 7f6af3e00640 1 -- 172.21.15.28:0/2479443877 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ae8007d10 con 0x7f6afc154140 2024-02-21T23:10:53.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.097+0000 7f6b02e00640 1 -- 172.21.15.28:0/2479443877 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6afc0c0800 con 0x7f6afc154140 2024-02-21T23:10:53.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.098+0000 7f6b02e00640 1 -- 172.21.15.28:0/2479443877 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6afc067f30 con 0x7f6afc154140 2024-02-21T23:10:53.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.099+0000 7f6af3e00640 1 -- 172.21.15.28:0/2479443877 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6ae8007070 con 0x7f6afc154140 2024-02-21T23:10:53.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.100+0000 7f6af3e00640 1 --2- 172.21.15.28:0/2479443877 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6adc06b500 0x7f6adc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:53.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.100+0000 7f6af3e00640 1 -- 172.21.15.28:0/2479443877 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(172..172 src has 1..172) v4 ==== 6001+0+0 (secure 0 0 0) 0x7f6ae80ab420 con 0x7f6afc154140 2024-02-21T23:10:53.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.101+0000 7f6afbe00640 1 --2- 172.21.15.28:0/2479443877 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6adc06b500 0x7f6adc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:53.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.102+0000 7f6afbe00640 1 --2- 172.21.15.28:0/2479443877 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6adc06b500 0x7f6adc06d9c0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f6aec009a10 tx=0x7f6aec008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:53.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.106+0000 7f6af3e00640 1 -- 172.21.15.28:0/2479443877 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f6ae807a1e0 con 0x7f6afc154140 2024-02-21T23:10:53.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.348+0000 7f6b02e00640 1 -- 172.21.15.28:0/2479443877 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f6afc0c16b0 con 0x7f6afc154140 2024-02-21T23:10:53.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.903+0000 7f6af3e00640 1 -- 172.21.15.28:0/2479443877 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 27 and data pool 28 v181) v1 ==== 174+0+0 (secure 0 0 0) 0x7f6ae8079ae0 con 0x7f6afc154140 2024-02-21T23:10:53.898 INFO:teuthology.orchestra.run.smithi028.stderr:new fs with metadata pool 27 and data pool 28 2024-02-21T23:10:53.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.914+0000 7f6b02e00640 1 -- 172.21.15.28:0/2479443877 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6adc06b500 msgr2=0x7f6adc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:53.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.914+0000 7f6b02e00640 1 --2- 172.21.15.28:0/2479443877 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6adc06b500 0x7f6adc06d9c0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f6aec009a10 tx=0x7f6aec008040 comp rx=0 tx=0).stop 2024-02-21T23:10:53.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.915+0000 7f6b02e00640 1 -- 172.21.15.28:0/2479443877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6afc154140 msgr2=0x7f6afc152960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:53.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.915+0000 7f6b02e00640 1 --2- 172.21.15.28:0/2479443877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6afc154140 0x7f6afc152960 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7f6ae8001970 tx=0x7f6afc0724f0 comp rx=0 tx=0).stop 2024-02-21T23:10:53.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.916+0000 7f6b02e00640 1 -- 172.21.15.28:0/2479443877 shutdown_connections 2024-02-21T23:10:53.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.916+0000 7f6b02e00640 1 --2- 172.21.15.28:0/2479443877 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6adc06b500 0x7f6adc06d9c0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:53.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.916+0000 7f6b02e00640 1 --2- 172.21.15.28:0/2479443877 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6afc152ea0 0x7f6afc1edf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:53.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.916+0000 7f6b02e00640 1 --2- 172.21.15.28:0/2479443877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6afc154140 0x7f6afc152960 unknown :-1 s=CLOSED pgs=1909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:53.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.916+0000 7f6b02e00640 1 --2- 172.21.15.28:0/2479443877 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6ae0004800 0x7f6afc153c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:53.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.916+0000 7f6b02e00640 1 -- 172.21.15.28:0/2479443877 >> 172.21.15.28:0/2479443877 conn(0x7f6afc0a85a0 msgr2=0x7f6afc0a8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:53.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.916+0000 7f6b02e00640 1 -- 172.21.15.28:0/2479443877 shutdown_connections 2024-02-21T23:10:53.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:53.917+0000 7f6b02e00640 1 -- 172.21.15.28:0/2479443877 wait complete. 2024-02-21T23:10:53.931 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:10:54.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.130+0000 7fd8c6000640 1 Processor -- start 2024-02-21T23:10:54.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.130+0000 7fd8c6000640 1 -- start start 2024-02-21T23:10:54.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.131+0000 7fd8c6000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd8c0150450 0x7fd8c00a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:54.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.131+0000 7fd8c6000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8c00c2760 con 0x7fd8c006ee40 2024-02-21T23:10:54.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.131+0000 7fd8c6000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8c0156310 con 0x7fd8c0156af0 2024-02-21T23:10:54.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.131+0000 7fd8c6000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8c00c22c0 con 0x7fd8c0150450 2024-02-21T23:10:54.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.131+0000 7fd8c4c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd8c0150450 0x7fd8c00a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:54.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.132+0000 7fd8bfe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd8c006ee40 0x7fd8c01536f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41296/0 (socket says 172.21.15.28:41296) 2024-02-21T23:10:54.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.132+0000 7fd8bfe00640 1 -- 172.21.15.28:0/2368061171 learned_addr learned my addr 172.21.15.28:0/2368061171 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:10:54.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.132+0000 7fd8c4c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd8c0150450 0x7fd8c00a5fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49334/0 (socket says 172.21.15.28:49334) 2024-02-21T23:10:54.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.132+0000 7fd8c4c00640 1 -- 172.21.15.28:0/2368061171 >> v1:172.21.15.81:6789/0 conn(0x7fd8c0156af0 legacy=0x7fd8c006dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:54.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.132+0000 7fd8c4c00640 1 -- 172.21.15.28:0/2368061171 >> v1:172.21.15.28:6789/0 conn(0x7fd8c006ee40 legacy=0x7fd8c01536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:54.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.132+0000 7fd8c4c00640 1 -- 172.21.15.28:0/2368061171 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8c00c29d0 con 0x7fd8c0150450 2024-02-21T23:10:54.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.133+0000 7fd8c4c00640 1 --2- 172.21.15.28:0/2368061171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8c0150450 0x7fd8c00a5fd0 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0009d90 tx=0x7fd8b002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e85205d520d9143 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:54.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.133+0000 7fd8bf400640 1 -- 172.21.15.28:0/2368061171 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b0039070 con 0x7fd8c0150450 2024-02-21T23:10:54.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.133+0000 7fd8bf400640 1 -- 172.21.15.28:0/2368061171 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8b002cc60 con 0x7fd8c0150450 2024-02-21T23:10:54.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.133+0000 7fd8bf400640 1 -- 172.21.15.28:0/2368061171 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b0034660 con 0x7fd8c0150450 2024-02-21T23:10:54.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.134+0000 7fd8c6000640 1 -- 172.21.15.28:0/2368061171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8c0150450 msgr2=0x7fd8c00a5fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:54.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.134+0000 7fd8c6000640 1 --2- 172.21.15.28:0/2368061171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8c0150450 0x7fd8c00a5fd0 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0009d90 tx=0x7fd8b002beb0 comp rx=0 tx=0).stop 2024-02-21T23:10:54.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.134+0000 7fd8c6000640 1 -- 172.21.15.28:0/2368061171 shutdown_connections 2024-02-21T23:10:54.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.134+0000 7fd8c6000640 1 --2- 172.21.15.28:0/2368061171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8c0150450 0x7fd8c00a5fd0 unknown :-1 s=CLOSED pgs=1911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:54.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.134+0000 7fd8c6000640 1 -- 172.21.15.28:0/2368061171 >> 172.21.15.28:0/2368061171 conn(0x7fd8c006b630 msgr2=0x7fd8c00c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:54.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.134+0000 7fd8c6000640 1 -- 172.21.15.28:0/2368061171 shutdown_connections 2024-02-21T23:10:54.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.134+0000 7fd8c6000640 1 -- 172.21.15.28:0/2368061171 wait complete. 2024-02-21T23:10:54.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.136+0000 7fd8c6000640 1 Processor -- start 2024-02-21T23:10:54.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.136+0000 7fd8c6000640 1 -- start start 2024-02-21T23:10:54.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.137+0000 7fd8c6000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd8c006ee40 0x7fd8c00bfba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:54.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.137+0000 7fd8c6000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8c0150450 0x7fd8c0145550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:54.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.137+0000 7fd8c6000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd8c0156af0 0x7fd8c0155250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:54.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.137+0000 7fd8c6000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c006e700 con 0x7fd8c0150450 2024-02-21T23:10:54.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.137+0000 7fd8c6000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c0156660 con 0x7fd8c0156af0 2024-02-21T23:10:54.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.137+0000 7fd8c6000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c006e880 con 0x7fd8c006ee40 2024-02-21T23:10:54.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.137+0000 7fd8bfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8c0150450 0x7fd8c0145550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:54.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.137+0000 7fd8bfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8c0150450 0x7fd8c0145550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49346/0 (socket says 172.21.15.28:49346) 2024-02-21T23:10:54.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.137+0000 7fd8bfe00640 1 -- 172.21.15.28:0/1148795216 learned_addr learned my addr 172.21.15.28:0/1148795216 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:54.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.138+0000 7fd8c4c00640 1 --2- 172.21.15.28:0/1148795216 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd8c006ee40 0x7fd8c00bfba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:54.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.138+0000 7fd8c5600640 1 --2- 172.21.15.28:0/1148795216 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd8c0156af0 0x7fd8c0155250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:54.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.138+0000 7fd8bfe00640 1 -- 172.21.15.28:0/1148795216 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd8c006ee40 msgr2=0x7fd8c00bfba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:54.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.138+0000 7fd8bfe00640 1 --2- 172.21.15.28:0/1148795216 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd8c006ee40 0x7fd8c00bfba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:54.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.138+0000 7fd8bfe00640 1 -- 172.21.15.28:0/1148795216 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd8c0156af0 msgr2=0x7fd8c0155250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:54.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.138+0000 7fd8bfe00640 1 --2- 172.21.15.28:0/1148795216 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd8c0156af0 0x7fd8c0155250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:54.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.138+0000 7fd8bfe00640 1 -- 172.21.15.28:0/1148795216 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8c01edfa0 con 0x7fd8c0150450 2024-02-21T23:10:54.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.138+0000 7fd8bfe00640 1 --2- 172.21.15.28:0/1148795216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8c0150450 0x7fd8c0145550 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7fd8ac00c430 tx=0x7fd8ac00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:54.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.139+0000 7fd8bcc00640 1 -- 172.21.15.28:0/1148795216 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8ac017070 con 0x7fd8c0150450 2024-02-21T23:10:54.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.139+0000 7fd8bcc00640 1 -- 172.21.15.28:0/1148795216 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8ac005ce0 con 0x7fd8c0150450 2024-02-21T23:10:54.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.139+0000 7fd8bcc00640 1 -- 172.21.15.28:0/1148795216 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8ac0072f0 con 0x7fd8c0150450 2024-02-21T23:10:54.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.139+0000 7fd8c6000640 1 -- 172.21.15.28:0/1148795216 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8c01ee280 con 0x7fd8c0150450 2024-02-21T23:10:54.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.139+0000 7fd8c6000640 1 -- 172.21.15.28:0/1148795216 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8c01ee840 con 0x7fd8c0150450 2024-02-21T23:10:54.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.140+0000 7fd8c6000640 1 -- 172.21.15.28:0/1148795216 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8c004f290 con 0x7fd8c0150450 2024-02-21T23:10:54.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.141+0000 7fd8bcc00640 1 -- 172.21.15.28:0/1148795216 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd8ac01a070 con 0x7fd8c0150450 2024-02-21T23:10:54.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.142+0000 7fd8bcc00640 1 --2- 172.21.15.28:0/1148795216 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd8a006b500 0x7fd8a006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:54.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.142+0000 7fd8bcc00640 1 -- 172.21.15.28:0/1148795216 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd8ac089840 con 0x7fd8c0150450 2024-02-21T23:10:54.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.142+0000 7fd8c4c00640 1 --2- 172.21.15.28:0/1148795216 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd8a006b500 0x7fd8a006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:54.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.144+0000 7fd8c4c00640 1 --2- 172.21.15.28:0/1148795216 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd8a006b500 0x7fd8a006d9c0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0036000 tx=0x7fd8b0030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:54.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.146+0000 7fd8bcc00640 1 -- 172.21.15.28:0/1148795216 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd8ac0584a0 con 0x7fd8c0150450 2024-02-21T23:10:54.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.347+0000 7fd8c6000640 1 -- 172.21.15.28:0/1148795216 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd8c00c1f10 con 0x7fd8c0150450 2024-02-21T23:10:54.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.349+0000 7fd8bcc00640 1 -- 172.21.15.28:0/1148795216 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v173) v1 ==== 74+0+16905 (secure 0 0 0) 0x7fd8ac057da0 con 0x7fd8c0150450 2024-02-21T23:10:54.337 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:10:54.337 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":173,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:10:53.350387+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:10:51.223834+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"173","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2024-02-21T23:10:52.368371+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"173","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:10:54.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.351+0000 7fd8c6000640 1 -- 172.21.15.28:0/1148795216 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd8a006b500 msgr2=0x7fd8a006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:54.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.351+0000 7fd8c6000640 1 --2- 172.21.15.28:0/1148795216 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd8a006b500 0x7fd8a006d9c0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0036000 tx=0x7fd8b0030000 comp rx=0 tx=0).stop 2024-02-21T23:10:54.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.351+0000 7fd8c6000640 1 -- 172.21.15.28:0/1148795216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8c0150450 msgr2=0x7fd8c0145550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:54.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.351+0000 7fd8c6000640 1 --2- 172.21.15.28:0/1148795216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8c0150450 0x7fd8c0145550 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7fd8ac00c430 tx=0x7fd8ac00c8f0 comp rx=0 tx=0).stop 2024-02-21T23:10:54.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.351+0000 7fd8c6000640 1 -- 172.21.15.28:0/1148795216 shutdown_connections 2024-02-21T23:10:54.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.351+0000 7fd8c6000640 1 --2- 172.21.15.28:0/1148795216 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd8a006b500 0x7fd8a006d9c0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:54.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.351+0000 7fd8c6000640 1 --2- 172.21.15.28:0/1148795216 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd8c0156af0 0x7fd8c0155250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:54.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.351+0000 7fd8c6000640 1 --2- 172.21.15.28:0/1148795216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8c0150450 0x7fd8c0145550 unknown :-1 s=CLOSED pgs=1912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:54.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.351+0000 7fd8c6000640 1 --2- 172.21.15.28:0/1148795216 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd8c006ee40 0x7fd8c00bfba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:54.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.351+0000 7fd8c6000640 1 -- 172.21.15.28:0/1148795216 >> 172.21.15.28:0/1148795216 conn(0x7fd8c006b630 msgr2=0x7fd8c00c0530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:54.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.352+0000 7fd8c6000640 1 -- 172.21.15.28:0/1148795216 shutdown_connections 2024-02-21T23:10:54.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.352+0000 7fd8c6000640 1 -- 172.21.15.28:0/1148795216 wait complete. 2024-02-21T23:10:54.358 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:10:54.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.592+0000 7f3cf8a00640 1 Processor -- start 2024-02-21T23:10:54.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.593+0000 7f3cf8a00640 1 -- start start 2024-02-21T23:10:54.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.594+0000 7f3cf8a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3cf40aff90 0x7f3cf40c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:54.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.594+0000 7f3cf8a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3cf40c5720 0x7f3cf415f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:54.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.594+0000 7f3cf8a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cf40c74c0 con 0x7f3cf406dbf0 2024-02-21T23:10:54.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.594+0000 7f3cf8a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cf40c7270 con 0x7f3cf40aff90 2024-02-21T23:10:54.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.594+0000 7f3cf8a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cf4158630 con 0x7f3cf40c5720 2024-02-21T23:10:54.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.594+0000 7f3cf2800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3cf40aff90 0x7f3cf40c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:54.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.594+0000 7f3cf1e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f3cf406dbf0 0x7f3cf4158ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41308/0 (socket says 172.21.15.28:41308) 2024-02-21T23:10:54.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.594+0000 7f3cf1e00640 1 -- 172.21.15.28:0/3929367806 learned_addr learned my addr 172.21.15.28:0/3929367806 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:10:54.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.595+0000 7f3cf3200640 1 --2- 172.21.15.28:0/3929367806 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3cf40c5720 0x7f3cf415f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:54.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.595+0000 7f3cf2800640 1 -- 172.21.15.28:0/3929367806 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3cf40c5720 msgr2=0x7f3cf415f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:54.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.595+0000 7f3cf2800640 1 --2- 172.21.15.28:0/3929367806 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3cf40c5720 0x7f3cf415f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:54.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.595+0000 7f3cf2800640 1 -- 172.21.15.28:0/3929367806 >> v1:172.21.15.28:6789/0 conn(0x7f3cf406dbf0 legacy=0x7f3cf4158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:54.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.595+0000 7f3cf2800640 1 -- 172.21.15.28:0/3929367806 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cf40c77a0 con 0x7f3cf40aff90 2024-02-21T23:10:54.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.596+0000 7f3cf2800640 1 --2- 172.21.15.28:0/3929367806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cf40aff90 0x7f3cf40c51e0 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7f3cdc0097a0 tx=0x7f3cdc02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4632fd6d626d571c server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:54.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.596+0000 7f3cf1400640 1 -- 172.21.15.28:0/3929367806 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cdc035020 con 0x7f3cf40aff90 2024-02-21T23:10:54.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.597+0000 7f3cf1400640 1 -- 172.21.15.28:0/3929367806 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cdc031810 con 0x7f3cf40aff90 2024-02-21T23:10:54.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.597+0000 7f3cf1400640 1 -- 172.21.15.28:0/3929367806 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cdc031be0 con 0x7f3cf40aff90 2024-02-21T23:10:54.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.597+0000 7f3cf8a00640 1 -- 172.21.15.28:0/3929367806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cf40aff90 msgr2=0x7f3cf40c51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:54.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.597+0000 7f3cf8a00640 1 --2- 172.21.15.28:0/3929367806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cf40aff90 0x7f3cf40c51e0 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7f3cdc0097a0 tx=0x7f3cdc02b8b0 comp rx=0 tx=0).stop 2024-02-21T23:10:54.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.597+0000 7f3cf8a00640 1 -- 172.21.15.28:0/3929367806 shutdown_connections 2024-02-21T23:10:54.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.597+0000 7f3cf8a00640 1 --2- 172.21.15.28:0/3929367806 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3cf40c5720 0x7f3cf415f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:54.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.597+0000 7f3cf8a00640 1 --2- 172.21.15.28:0/3929367806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cf40aff90 0x7f3cf40c51e0 unknown :-1 s=CLOSED pgs=1914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:54.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.597+0000 7f3cf8a00640 1 -- 172.21.15.28:0/3929367806 >> 172.21.15.28:0/3929367806 conn(0x7f3cf40ac370 msgr2=0x7f3cf40ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:54.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.597+0000 7f3cf8a00640 1 -- 172.21.15.28:0/3929367806 shutdown_connections 2024-02-21T23:10:54.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.597+0000 7f3cf8a00640 1 -- 172.21.15.28:0/3929367806 wait complete. 2024-02-21T23:10:54.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.599+0000 7f3cf8a00640 1 Processor -- start 2024-02-21T23:10:54.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.599+0000 7f3cf8a00640 1 -- start start 2024-02-21T23:10:54.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.600+0000 7f3cf8a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cf406dbf0 0x7f3cf41edf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:54.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.600+0000 7f3cf8a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3cf40aff90 0x7f3cf40ae700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:54.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.600+0000 7f3cf8a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3cf40c5720 0x7f3cf414ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:54.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.600+0000 7f3cf8a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cf40c8760 con 0x7f3cf406dbf0 2024-02-21T23:10:54.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.600+0000 7f3cf8a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cf40be0c0 con 0x7f3cf40aff90 2024-02-21T23:10:54.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.600+0000 7f3cf8a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cf40c88e0 con 0x7f3cf40c5720 2024-02-21T23:10:54.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.600+0000 7f3cf2800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cf406dbf0 0x7f3cf41edf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:54.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.600+0000 7f3cf2800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cf406dbf0 0x7f3cf41edf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49362/0 (socket says 172.21.15.28:49362) 2024-02-21T23:10:54.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.600+0000 7f3cf2800640 1 -- 172.21.15.28:0/1736465706 learned_addr learned my addr 172.21.15.28:0/1736465706 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:54.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.601+0000 7f3cf3200640 1 --2- 172.21.15.28:0/1736465706 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3cf40c5720 0x7f3cf414ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:54.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.601+0000 7f3cf2800640 1 -- 172.21.15.28:0/1736465706 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3cf40c5720 msgr2=0x7f3cf414ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:54.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.601+0000 7f3cf2800640 1 --2- 172.21.15.28:0/1736465706 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3cf40c5720 0x7f3cf414ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:54.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.601+0000 7f3cf2800640 1 -- 172.21.15.28:0/1736465706 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3cf40aff90 msgr2=0x7f3cf40ae700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:54.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.601+0000 7f3cf2800640 1 --2- 172.21.15.28:0/1736465706 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3cf40aff90 0x7f3cf40ae700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:54.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.601+0000 7f3cf2800640 1 -- 172.21.15.28:0/1736465706 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cf41ee9a0 con 0x7f3cf406dbf0 2024-02-21T23:10:54.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.601+0000 7f3cf2800640 1 --2- 172.21.15.28:0/1736465706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cf406dbf0 0x7f3cf41edf20 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7f3cdc02bec0 tx=0x7f3cdc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:54.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.601+0000 7f3ceaa00640 1 -- 172.21.15.28:0/1736465706 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cdc041070 con 0x7f3cf406dbf0 2024-02-21T23:10:54.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.602+0000 7f3ceaa00640 1 -- 172.21.15.28:0/1736465706 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cdc03ad00 con 0x7f3cf406dbf0 2024-02-21T23:10:54.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.602+0000 7f3cf8a00640 1 -- 172.21.15.28:0/1736465706 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cf40ae2b0 con 0x7f3cf406dbf0 2024-02-21T23:10:54.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.602+0000 7f3ceaa00640 1 -- 172.21.15.28:0/1736465706 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cdc035040 con 0x7f3cf406dbf0 2024-02-21T23:10:54.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.602+0000 7f3cf8a00640 1 -- 172.21.15.28:0/1736465706 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cf41e11e0 con 0x7f3cf406dbf0 2024-02-21T23:10:54.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.603+0000 7f3cf8a00640 1 -- 172.21.15.28:0/1736465706 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cf404f290 con 0x7f3cf406dbf0 2024-02-21T23:10:54.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.604+0000 7f3ceaa00640 1 -- 172.21.15.28:0/1736465706 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3cdc04c060 con 0x7f3cf406dbf0 2024-02-21T23:10:54.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.605+0000 7f3ceaa00640 1 --2- 172.21.15.28:0/1736465706 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3cd006b4d0 0x7f3cd006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:54.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.605+0000 7f3ceaa00640 1 -- 172.21.15.28:0/1736465706 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(173..173 src has 1..173) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f3cdc0ac080 con 0x7f3cf406dbf0 2024-02-21T23:10:54.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.605+0000 7f3cf1e00640 1 --2- 172.21.15.28:0/1736465706 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3cd006b4d0 0x7f3cd006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:54.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.607+0000 7f3cf1e00640 1 --2- 172.21.15.28:0/1736465706 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3cd006b4d0 0x7f3cd006d990 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f3ce40012e0 tx=0x7f3ce4001630 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:54.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.609+0000 7f3ceaa00640 1 -- 172.21.15.28:0/1736465706 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3cdc07ad60 con 0x7f3cf406dbf0 2024-02-21T23:10:54.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.808+0000 7f3cf8a00640 1 -- 172.21.15.28:0/1736465706 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3cf406b4a0 con 0x7f3cf406dbf0 2024-02-21T23:10:54.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.810+0000 7f3ceaa00640 1 -- 172.21.15.28:0/1736465706 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v173) v1 ==== 74+0+16905 (secure 0 0 0) 0x7f3cdc07a660 con 0x7f3cf406dbf0 2024-02-21T23:10:54.798 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:10:54.798 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":173,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:10:53.350387+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:10:51.223834+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"173","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2024-02-21T23:10:52.368371+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"173","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:10:54.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.812+0000 7f3cf8a00640 1 -- 172.21.15.28:0/1736465706 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3cd006b4d0 msgr2=0x7f3cd006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:54.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.812+0000 7f3cf8a00640 1 --2- 172.21.15.28:0/1736465706 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3cd006b4d0 0x7f3cd006d990 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f3ce40012e0 tx=0x7f3ce4001630 comp rx=0 tx=0).stop 2024-02-21T23:10:54.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.812+0000 7f3cf8a00640 1 -- 172.21.15.28:0/1736465706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cf406dbf0 msgr2=0x7f3cf41edf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:54.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.812+0000 7f3cf8a00640 1 --2- 172.21.15.28:0/1736465706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cf406dbf0 0x7f3cf41edf20 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7f3cdc02bec0 tx=0x7f3cdc0046c0 comp rx=0 tx=0).stop 2024-02-21T23:10:54.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.813+0000 7f3cf8a00640 1 -- 172.21.15.28:0/1736465706 shutdown_connections 2024-02-21T23:10:54.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.813+0000 7f3cf8a00640 1 --2- 172.21.15.28:0/1736465706 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3cd006b4d0 0x7f3cd006d990 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:54.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.813+0000 7f3cf8a00640 1 --2- 172.21.15.28:0/1736465706 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3cf40c5720 0x7f3cf414ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:54.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.813+0000 7f3cf8a00640 1 --2- 172.21.15.28:0/1736465706 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3cf40aff90 0x7f3cf40ae700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:54.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.813+0000 7f3cf8a00640 1 --2- 172.21.15.28:0/1736465706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cf406dbf0 0x7f3cf41edf20 unknown :-1 s=CLOSED pgs=1915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:54.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.813+0000 7f3cf8a00640 1 -- 172.21.15.28:0/1736465706 >> 172.21.15.28:0/1736465706 conn(0x7f3cf40ac370 msgr2=0x7f3cf40ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:54.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.813+0000 7f3cf8a00640 1 -- 172.21.15.28:0/1736465706 shutdown_connections 2024-02-21T23:10:54.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:54.813+0000 7f3cf8a00640 1 -- 172.21.15.28:0/1736465706 wait complete. 2024-02-21T23:10:54.819 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-02-21T23:10:55.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:55.044+0000 7f71ce800640 1 Processor -- start 2024-02-21T23:10:55.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:55.044+0000 7f71ce800640 1 -- start start 2024-02-21T23:10:55.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:55.046+0000 7f71ce800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f71c80c4030 0x7f71c80c5d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:55.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:55.046+0000 7f71ce800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71c81594d0 con 0x7f71c806d3b0 2024-02-21T23:10:55.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:55.046+0000 7f71ce800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71c8155500 con 0x7f71c80c4d30 2024-02-21T23:10:55.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:55.046+0000 7f71ce800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71c81557e0 con 0x7f71c80c4030 2024-02-21T23:10:55.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:55.046+0000 7f71c7400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f71c80c4030 0x7f71c80c5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:55.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:55.046+0000 7f71c7e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f71c806d3b0 0x7f71c815fa50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55698/0 (socket says 172.21.15.28:55698) 2024-02-21T23:10:55.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:55.046+0000 7f71c7e00640 1 -- 172.21.15.28:0/144843410 learned_addr learned my addr 172.21.15.28:0/144843410 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:10:55.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:55.047+0000 7f71c7400640 1 -- 172.21.15.28:0/144843410 >> v1:172.21.15.81:6789/0 conn(0x7f71c80c4d30 legacy=0x7f71c806cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:55.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:55.047+0000 7f71c7400640 1 -- 172.21.15.28:0/144843410 >> v1:172.21.15.28:6789/0 conn(0x7f71c806d3b0 legacy=0x7f71c815fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:55.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:55.047+0000 7f71c7400640 1 -- 172.21.15.28:0/144843410 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71c8155ac0 con 0x7f71c80c4030 2024-02-21T23:10:55.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:55.047+0000 7f71c6000640 1 -- 172.21.15.28:0/144843410 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4009226947 0 0) 0x7f71c81594d0 con 0x7f71c806d3b0 2024-02-21T23:10:55.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:55.048+0000 7f71c7400640 1 --2- 172.21.15.28:0/144843410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71c80c4030 0x7f71c80c5d60 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7f71b80097a0 tx=0x7f71b802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=489cbec232edbb28 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:55.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:55.048+0000 7f71c6000640 1 -- 172.21.15.28:0/144843410 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71b80315a0 con 0x7f71c80c4030 2024-02-21T23:10:55.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:55.048+0000 7f71c6000640 1 -- 172.21.15.28:0/144843410 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71b8037070 con 0x7f71c80c4030 2024-02-21T23:10:55.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:55.048+0000 7f71c6000640 1 -- 172.21.15.28:0/144843410 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71b8031960 con 0x7f71c80c4030 2024-02-21T23:10:55.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:55.049+0000 7f71ce800640 1 -- 172.21.15.28:0/144843410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71c80c4030 msgr2=0x7f71c80c5d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:55.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:55.049+0000 7f71ce800640 1 --2- 172.21.15.28:0/144843410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71c80c4030 0x7f71c80c5d60 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7f71b80097a0 tx=0x7f71b802b840 comp rx=0 tx=0).stop 2024-02-21T23:10:55.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:55.049+0000 7f71ce800640 1 -- 172.21.15.28:0/144843410 shutdown_connections 2024-02-21T23:10:55.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:55.049+0000 7f71ce800640 1 --2- 172.21.15.28:0/144843410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71c80c4030 0x7f71c80c5d60 unknown :-1 s=CLOSED pgs=1917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:55.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:55.049+0000 7f71ce800640 1 -- 172.21.15.28:0/144843410 >> 172.21.15.28:0/144843410 conn(0x7f71c80725a0 msgr2=0x7f71c80729b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:55.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:55.049+0000 7f71ce800640 1 -- 172.21.15.28:0/144843410 shutdown_connections 2024-02-21T23:10:55.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:55.049+0000 7f71ce800640 1 -- 172.21.15.28:0/144843410 wait complete. 2024-02-21T23:10:55.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:55.051+0000 7f71ce800640 1 Processor -- start 2024-02-21T23:10:55.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:55.051+0000 7f71ce800640 1 -- start start 2024-02-21T23:10:55.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:55.052+0000 7f71ce800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71c806d3b0 0x7f71c81503a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:55.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:55.052+0000 7f71ce800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71c80c4030 0x7f71c80aa370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:55.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:55.052+0000 7f71ce800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f71c80c4d30 0x7f71c814e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:55.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:55.052+0000 7f71ce800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71c81592b0 con 0x7f71c80c4030 2024-02-21T23:10:55.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:55.052+0000 7f71ce800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71c8159130 con 0x7f71c806d3b0 2024-02-21T23:10:55.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:55.052+0000 7f71ce800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71c8155fe0 con 0x7f71c80c4d30 2024-02-21T23:10:55.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:55.052+0000 7f71c6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71c80c4030 0x7f71c80aa370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:55.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:55.052+0000 7f71c6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71c80c4030 0x7f71c80aa370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32984/0 (socket says 172.21.15.28:32984) 2024-02-21T23:10:55.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:55.052+0000 7f71c6a00640 1 -- 172.21.15.28:0/1461211015 learned_addr learned my addr 172.21.15.28:0/1461211015 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:55.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:55.052+0000 7f71c7400640 1 --2- 172.21.15.28:0/1461211015 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71c806d3b0 0x7f71c81503a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:55.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:55.052+0000 7f71c7e00640 1 --2- 172.21.15.28:0/1461211015 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f71c80c4d30 0x7f71c814e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:55.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:55.053+0000 7f71c6a00640 1 -- 172.21.15.28:0/1461211015 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f71c80c4d30 msgr2=0x7f71c814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:55.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:55.053+0000 7f71c6a00640 1 --2- 172.21.15.28:0/1461211015 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f71c80c4d30 0x7f71c814e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:55.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:55.053+0000 7f71c6a00640 1 -- 172.21.15.28:0/1461211015 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71c806d3b0 msgr2=0x7f71c81503a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:55.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:55.053+0000 7f71c6a00640 1 --2- 172.21.15.28:0/1461211015 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71c806d3b0 0x7f71c81503a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:55.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:55.053+0000 7f71c6a00640 1 -- 172.21.15.28:0/1461211015 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71c80ad090 con 0x7f71c80c4030 2024-02-21T23:10:55.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:55.053+0000 7f71c6a00640 1 --2- 172.21.15.28:0/1461211015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71c80c4030 0x7f71c80aa370 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7f71b400c3c0 tx=0x7f71b400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:55.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:55.053+0000 7f71bf400640 1 -- 172.21.15.28:0/1461211015 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71b4017070 con 0x7f71c80c4030 2024-02-21T23:10:55.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:55.053+0000 7f71bf400640 1 -- 172.21.15.28:0/1461211015 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71b40052f0 con 0x7f71c80c4030 2024-02-21T23:10:55.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:55.053+0000 7f71bf400640 1 -- 172.21.15.28:0/1461211015 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71b4003830 con 0x7f71c80c4030 2024-02-21T23:10:55.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:55.053+0000 7f71ce800640 1 -- 172.21.15.28:0/1461211015 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71c81512e0 con 0x7f71c80c4030 2024-02-21T23:10:55.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:55.053+0000 7f71ce800640 1 -- 172.21.15.28:0/1461211015 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71c814f8c0 con 0x7f71c80c4030 2024-02-21T23:10:55.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:55.055+0000 7f71ce800640 1 -- 172.21.15.28:0/1461211015 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71c8067f30 con 0x7f71c80c4030 2024-02-21T23:10:55.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:55.055+0000 7f71bf400640 1 -- 172.21.15.28:0/1461211015 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f71b401a070 con 0x7f71c80c4030 2024-02-21T23:10:55.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:55.057+0000 7f71bf400640 1 --2- 172.21.15.28:0/1461211015 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f71a806b4d0 0x7f71a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:55.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:55.057+0000 7f71bf400640 1 -- 172.21.15.28:0/1461211015 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f71b4089660 con 0x7f71c80c4030 2024-02-21T23:10:55.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:55.057+0000 7f71c7400640 1 --2- 172.21.15.28:0/1461211015 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f71a806b4d0 0x7f71a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:55.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:55.059+0000 7f71c7400640 1 --2- 172.21.15.28:0/1461211015 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f71a806b4d0 0x7f71a806d990 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f71b8036000 tx=0x7f71b8008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:55.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:55.061+0000 7f71bf400640 1 -- 172.21.15.28:0/1461211015 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f71b4058340 con 0x7f71c80c4030 2024-02-21T23:10:55.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:55.430+0000 7f71ce800640 1 -- 172.21.15.28:0/1461211015 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f71c80c46b0 con 0x7f71c80c4030 2024-02-21T23:10:55.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:55.915+0000 7f71bf400640 1 -- 172.21.15.28:0/1461211015 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v184) v1 ==== 118+0+0 (secure 0 0 0) 0x7f71b4057c40 con 0x7f71c80c4030 2024-02-21T23:10:55.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:55.919+0000 7f71ce800640 1 -- 172.21.15.28:0/1461211015 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f71a806b4d0 msgr2=0x7f71a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:55.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:55.919+0000 7f71ce800640 1 --2- 172.21.15.28:0/1461211015 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f71a806b4d0 0x7f71a806d990 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f71b8036000 tx=0x7f71b8008030 comp rx=0 tx=0).stop 2024-02-21T23:10:55.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:55.919+0000 7f71ce800640 1 -- 172.21.15.28:0/1461211015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71c80c4030 msgr2=0x7f71c80aa370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:55.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:55.919+0000 7f71ce800640 1 --2- 172.21.15.28:0/1461211015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71c80c4030 0x7f71c80aa370 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7f71b400c3c0 tx=0x7f71b400c880 comp rx=0 tx=0).stop 2024-02-21T23:10:55.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:55.919+0000 7f71ce800640 1 -- 172.21.15.28:0/1461211015 shutdown_connections 2024-02-21T23:10:55.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:55.919+0000 7f71ce800640 1 --2- 172.21.15.28:0/1461211015 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f71a806b4d0 0x7f71a806d990 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:55.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:55.919+0000 7f71ce800640 1 --2- 172.21.15.28:0/1461211015 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f71c80c4d30 0x7f71c814e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:55.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:55.919+0000 7f71ce800640 1 --2- 172.21.15.28:0/1461211015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71c80c4030 0x7f71c80aa370 unknown :-1 s=CLOSED pgs=1918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:55.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:55.919+0000 7f71ce800640 1 --2- 172.21.15.28:0/1461211015 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f71c806d3b0 0x7f71c81503a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:55.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:55.919+0000 7f71ce800640 1 -- 172.21.15.28:0/1461211015 >> 172.21.15.28:0/1461211015 conn(0x7f71c80725a0 msgr2=0x7f71c80ca920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:55.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:55.919+0000 7f71ce800640 1 -- 172.21.15.28:0/1461211015 shutdown_connections 2024-02-21T23:10:55.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:55.919+0000 7f71ce800640 1 -- 172.21.15.28:0/1461211015 wait complete. 2024-02-21T23:10:55.927 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-02-21T23:10:55.927 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-02-21T23:10:56.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.155+0000 7ff347e00640 1 Processor -- start 2024-02-21T23:10:56.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.155+0000 7ff347e00640 1 -- start start 2024-02-21T23:10:56.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.157+0000 7ff347e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff3400a70f0 0x7ff340072390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:56.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.157+0000 7ff347e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff3400728d0 0x7ff3401585d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:56.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.157+0000 7ff347e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3401550b0 con 0x7ff3401527d0 2024-02-21T23:10:56.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.157+0000 7ff347e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff340155cc0 con 0x7ff3400728d0 2024-02-21T23:10:56.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.157+0000 7ff347e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff340155390 con 0x7ff3400a70f0 2024-02-21T23:10:56.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.157+0000 7ff345e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff3400728d0 0x7ff3401585d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:56.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.157+0000 7ff345400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff3401527d0 0x7ff3400c5700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55710/0 (socket says 172.21.15.28:55710) 2024-02-21T23:10:56.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.157+0000 7ff345400640 1 -- 172.21.15.28:0/2693275958 learned_addr learned my addr 172.21.15.28:0/2693275958 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:10:56.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.158+0000 7ff344a00640 1 --2- 172.21.15.28:0/2693275958 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff3400a70f0 0x7ff340072390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:56.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.158+0000 7ff345e00640 1 -- 172.21.15.28:0/2693275958 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff3400a70f0 msgr2=0x7ff340072390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:56.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.158+0000 7ff345e00640 1 --2- 172.21.15.28:0/2693275958 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff3400a70f0 0x7ff340072390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:56.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.158+0000 7ff345e00640 1 -- 172.21.15.28:0/2693275958 >> v1:172.21.15.28:6789/0 conn(0x7ff3401527d0 legacy=0x7ff3400c5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:56.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.158+0000 7ff345e00640 1 -- 172.21.15.28:0/2693275958 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff340155670 con 0x7ff3400728d0 2024-02-21T23:10:56.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.158+0000 7ff344a00640 1 --2- 172.21.15.28:0/2693275958 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff3400a70f0 0x7ff340072390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:10:56.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.159+0000 7ff345e00640 1 --2- 172.21.15.28:0/2693275958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3400728d0 0x7ff3401585d0 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7ff338009dc0 tx=0x7ff33802be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7739c86b0871d64 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:56.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.159+0000 7ff33fe00640 1 -- 172.21.15.28:0/2693275958 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff338039070 con 0x7ff3400728d0 2024-02-21T23:10:56.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.159+0000 7ff33fe00640 1 -- 172.21.15.28:0/2693275958 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff33802cbb0 con 0x7ff3400728d0 2024-02-21T23:10:56.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.159+0000 7ff33fe00640 1 -- 172.21.15.28:0/2693275958 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff338034440 con 0x7ff3400728d0 2024-02-21T23:10:56.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.160+0000 7ff347e00640 1 -- 172.21.15.28:0/2693275958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3400728d0 msgr2=0x7ff3401585d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:56.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.160+0000 7ff347e00640 1 --2- 172.21.15.28:0/2693275958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3400728d0 0x7ff3401585d0 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7ff338009dc0 tx=0x7ff33802be00 comp rx=0 tx=0).stop 2024-02-21T23:10:56.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.160+0000 7ff347e00640 1 -- 172.21.15.28:0/2693275958 shutdown_connections 2024-02-21T23:10:56.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.160+0000 7ff347e00640 1 --2- 172.21.15.28:0/2693275958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3400728d0 0x7ff3401585d0 unknown :-1 s=CLOSED pgs=1920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:56.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.160+0000 7ff347e00640 1 --2- 172.21.15.28:0/2693275958 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff3400a70f0 0x7ff340072390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:56.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.160+0000 7ff347e00640 1 -- 172.21.15.28:0/2693275958 >> 172.21.15.28:0/2693275958 conn(0x7ff3400a95b0 msgr2=0x7ff3400a9990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:56.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.160+0000 7ff347e00640 1 -- 172.21.15.28:0/2693275958 shutdown_connections 2024-02-21T23:10:56.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.160+0000 7ff347e00640 1 -- 172.21.15.28:0/2693275958 wait complete. 2024-02-21T23:10:56.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.163+0000 7ff347e00640 1 Processor -- start 2024-02-21T23:10:56.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.163+0000 7ff347e00640 1 -- start start 2024-02-21T23:10:56.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.164+0000 7ff347e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff3400728d0 0x7ff3400c40b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:56.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.164+0000 7ff347e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3400a70f0 0x7ff3400c45f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:56.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.164+0000 7ff347e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3401527d0 0x7ff34006c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:56.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.165+0000 7ff347e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff340158c90 con 0x7ff3401527d0 2024-02-21T23:10:56.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.165+0000 7ff347e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff340158b10 con 0x7ff3400a70f0 2024-02-21T23:10:56.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.165+0000 7ff347e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff340158e10 con 0x7ff3400728d0 2024-02-21T23:10:56.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.165+0000 7ff345e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3401527d0 0x7ff34006c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:56.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.165+0000 7ff345e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3401527d0 0x7ff34006c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33000/0 (socket says 172.21.15.28:33000) 2024-02-21T23:10:56.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.165+0000 7ff345e00640 1 -- 172.21.15.28:0/3207301488 learned_addr learned my addr 172.21.15.28:0/3207301488 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:56.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.165+0000 7ff345400640 1 --2- 172.21.15.28:0/3207301488 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff3400728d0 0x7ff3400c40b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:56.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.165+0000 7ff345e00640 1 -- 172.21.15.28:0/3207301488 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff3400728d0 msgr2=0x7ff3400c40b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:56.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.165+0000 7ff345e00640 1 --2- 172.21.15.28:0/3207301488 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff3400728d0 0x7ff3400c40b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:56.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.165+0000 7ff344a00640 1 --2- 172.21.15.28:0/3207301488 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3400a70f0 0x7ff3400c45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:56.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.165+0000 7ff345e00640 1 -- 172.21.15.28:0/3207301488 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3400a70f0 msgr2=0x7ff3400c45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:56.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.165+0000 7ff345e00640 1 --2- 172.21.15.28:0/3207301488 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3400a70f0 0x7ff3400c45f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:56.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.165+0000 7ff345e00640 1 -- 172.21.15.28:0/3207301488 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3401ebb40 con 0x7ff3401527d0 2024-02-21T23:10:56.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.165+0000 7ff344a00640 1 --2- 172.21.15.28:0/3207301488 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3400a70f0 0x7ff3400c45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:10:56.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.166+0000 7ff345e00640 1 --2- 172.21.15.28:0/3207301488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3401527d0 0x7ff34006c320 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7ff338037040 tx=0x7ff338004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:56.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.166+0000 7ff33d600640 1 -- 172.21.15.28:0/3207301488 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff338039050 con 0x7ff3401527d0 2024-02-21T23:10:56.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.166+0000 7ff33d600640 1 -- 172.21.15.28:0/3207301488 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff338045070 con 0x7ff3401527d0 2024-02-21T23:10:56.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.166+0000 7ff347e00640 1 -- 172.21.15.28:0/3207301488 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff34006bb20 con 0x7ff3401527d0 2024-02-21T23:10:56.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.166+0000 7ff33d600640 1 -- 172.21.15.28:0/3207301488 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff33803e850 con 0x7ff3401527d0 2024-02-21T23:10:56.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.166+0000 7ff347e00640 1 -- 172.21.15.28:0/3207301488 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3400c3810 con 0x7ff3401527d0 2024-02-21T23:10:56.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.168+0000 7ff347e00640 1 -- 172.21.15.28:0/3207301488 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff340068640 con 0x7ff3401527d0 2024-02-21T23:10:56.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.168+0000 7ff33d600640 1 -- 172.21.15.28:0/3207301488 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff33803e9f0 con 0x7ff3401527d0 2024-02-21T23:10:56.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.170+0000 7ff33d600640 1 --2- 172.21.15.28:0/3207301488 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff32006b4d0 0x7ff32006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:56.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.170+0000 7ff33d600640 1 -- 172.21.15.28:0/3207301488 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff3380ad470 con 0x7ff3401527d0 2024-02-21T23:10:56.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.171+0000 7ff345400640 1 --2- 172.21.15.28:0/3207301488 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff32006b4d0 0x7ff32006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:56.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.172+0000 7ff345400640 1 --2- 172.21.15.28:0/3207301488 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff32006b4d0 0x7ff32006d990 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7ff334000f60 tx=0x7ff334009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:56.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.174+0000 7ff33d600640 1 -- 172.21.15.28:0/3207301488 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ff33807c270 con 0x7ff3401527d0 2024-02-21T23:10:56.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.417+0000 7ff347e00640 1 -- 172.21.15.28:0/3207301488 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7ff3400adb90 con 0x7ff3401527d0 2024-02-21T23:10:56.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.927+0000 7ff33d600640 1 -- 172.21.15.28:0/3207301488 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v185) v1 ==== 122+0+0 (secure 0 0 0) 0x7ff33807bb70 con 0x7ff3401527d0 2024-02-21T23:10:56.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.932+0000 7ff347e00640 1 -- 172.21.15.28:0/3207301488 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff32006b4d0 msgr2=0x7ff32006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:56.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.932+0000 7ff347e00640 1 --2- 172.21.15.28:0/3207301488 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff32006b4d0 0x7ff32006d990 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7ff334000f60 tx=0x7ff334009040 comp rx=0 tx=0).stop 2024-02-21T23:10:56.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.932+0000 7ff347e00640 1 -- 172.21.15.28:0/3207301488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3401527d0 msgr2=0x7ff34006c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:56.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.932+0000 7ff347e00640 1 --2- 172.21.15.28:0/3207301488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3401527d0 0x7ff34006c320 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7ff338037040 tx=0x7ff338004a40 comp rx=0 tx=0).stop 2024-02-21T23:10:56.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.932+0000 7ff347e00640 1 -- 172.21.15.28:0/3207301488 shutdown_connections 2024-02-21T23:10:56.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.932+0000 7ff347e00640 1 --2- 172.21.15.28:0/3207301488 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff32006b4d0 0x7ff32006d990 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:56.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.932+0000 7ff347e00640 1 --2- 172.21.15.28:0/3207301488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3401527d0 0x7ff34006c320 unknown :-1 s=CLOSED pgs=1921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:56.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.932+0000 7ff347e00640 1 --2- 172.21.15.28:0/3207301488 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3400a70f0 0x7ff3400c45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:56.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.932+0000 7ff347e00640 1 --2- 172.21.15.28:0/3207301488 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff3400728d0 0x7ff3400c40b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:56.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.932+0000 7ff347e00640 1 -- 172.21.15.28:0/3207301488 >> 172.21.15.28:0/3207301488 conn(0x7ff3400a95b0 msgr2=0x7ff34006efa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:56.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.933+0000 7ff347e00640 1 -- 172.21.15.28:0/3207301488 shutdown_connections 2024-02-21T23:10:56.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:56.933+0000 7ff347e00640 1 -- 172.21.15.28:0/3207301488 wait complete. 2024-02-21T23:10:56.946 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-02-21T23:10:56.946 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:10:57.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.180+0000 7f7a45200640 1 Processor -- start 2024-02-21T23:10:57.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.180+0000 7f7a45200640 1 -- start start 2024-02-21T23:10:57.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.182+0000 7f7a45200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7a4006dbf0 0x7f7a40158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:57.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.182+0000 7f7a45200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7a40155960 0x7f7a40159000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:57.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.182+0000 7f7a45200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a400c74c0 con 0x7f7a400aff90 2024-02-21T23:10:57.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.182+0000 7f7a45200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a400c7270 con 0x7f7a40155960 2024-02-21T23:10:57.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.182+0000 7f7a45200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a40158630 con 0x7f7a4006dbf0 2024-02-21T23:10:57.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.182+0000 7f7a3fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7a40155960 0x7f7a40159000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:57.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.182+0000 7f7a3fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7a40155960 0x7f7a40159000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33002/0 (socket says 172.21.15.28:33002) 2024-02-21T23:10:57.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.182+0000 7f7a3fe00640 1 -- 172.21.15.28:0/1317027412 learned_addr learned my addr 172.21.15.28:0/1317027412 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:57.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.183+0000 7f7a36a00640 1 --2- 172.21.15.28:0/1317027412 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7a4006dbf0 0x7f7a40158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:57.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.183+0000 7f7a3fe00640 1 -- 172.21.15.28:0/1317027412 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7a4006dbf0 msgr2=0x7f7a40158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:57.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.183+0000 7f7a3fe00640 1 --2- 172.21.15.28:0/1317027412 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7a4006dbf0 0x7f7a40158ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:57.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.183+0000 7f7a3fe00640 1 -- 172.21.15.28:0/1317027412 >> v1:172.21.15.81:6789/0 conn(0x7f7a400aff90 legacy=0x7f7a400c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:57.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.183+0000 7f7a3fe00640 1 -- 172.21.15.28:0/1317027412 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a400c77a0 con 0x7f7a40155960 2024-02-21T23:10:57.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.184+0000 7f7a3fe00640 1 --2- 172.21.15.28:0/1317027412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a40155960 0x7f7a40159000 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7f7a38009870 tx=0x7f7a3802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd7720683a08add6 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:57.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.185+0000 7f7a3ea00640 1 -- 172.21.15.28:0/1317027412 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a38034070 con 0x7f7a40155960 2024-02-21T23:10:57.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.185+0000 7f7a3ea00640 1 -- 172.21.15.28:0/1317027412 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a38030840 con 0x7f7a40155960 2024-02-21T23:10:57.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.185+0000 7f7a3ea00640 1 -- 172.21.15.28:0/1317027412 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a38030c10 con 0x7f7a40155960 2024-02-21T23:10:57.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.185+0000 7f7a45200640 1 -- 172.21.15.28:0/1317027412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a40155960 msgr2=0x7f7a40159000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:57.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.185+0000 7f7a45200640 1 --2- 172.21.15.28:0/1317027412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a40155960 0x7f7a40159000 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7f7a38009870 tx=0x7f7a3802b980 comp rx=0 tx=0).stop 2024-02-21T23:10:57.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.185+0000 7f7a45200640 1 -- 172.21.15.28:0/1317027412 shutdown_connections 2024-02-21T23:10:57.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.185+0000 7f7a45200640 1 --2- 172.21.15.28:0/1317027412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a40155960 0x7f7a40159000 unknown :-1 s=CLOSED pgs=1922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:57.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.185+0000 7f7a45200640 1 --2- 172.21.15.28:0/1317027412 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7a4006dbf0 0x7f7a40158ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:57.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.185+0000 7f7a45200640 1 -- 172.21.15.28:0/1317027412 >> 172.21.15.28:0/1317027412 conn(0x7f7a400ac370 msgr2=0x7f7a400ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:57.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.185+0000 7f7a45200640 1 -- 172.21.15.28:0/1317027412 shutdown_connections 2024-02-21T23:10:57.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.186+0000 7f7a45200640 1 -- 172.21.15.28:0/1317027412 wait complete. 2024-02-21T23:10:57.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.188+0000 7f7a45200640 1 Processor -- start 2024-02-21T23:10:57.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.188+0000 7f7a45200640 1 -- start start 2024-02-21T23:10:57.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.189+0000 7f7a45200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a4006dbf0 0x7f7a400c3560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:57.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.189+0000 7f7a45200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a400aff90 0x7f7a401518c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:57.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.189+0000 7f7a45200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7a40155960 0x7f7a40151e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:57.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.189+0000 7f7a45200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a400c58f0 con 0x7f7a4006dbf0 2024-02-21T23:10:57.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.189+0000 7f7a45200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a400be0c0 con 0x7f7a400aff90 2024-02-21T23:10:57.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.189+0000 7f7a45200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a400c5a70 con 0x7f7a40155960 2024-02-21T23:10:57.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.190+0000 7f7a3f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a4006dbf0 0x7f7a400c3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:57.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.190+0000 7f7a3f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a4006dbf0 0x7f7a400c3560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33008/0 (socket says 172.21.15.28:33008) 2024-02-21T23:10:57.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.190+0000 7f7a3f400640 1 -- 172.21.15.28:0/3687518917 learned_addr learned my addr 172.21.15.28:0/3687518917 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:57.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.190+0000 7f7a36a00640 1 --2- 172.21.15.28:0/3687518917 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a400aff90 0x7f7a401518c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:57.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.190+0000 7f7a3fe00640 1 --2- 172.21.15.28:0/3687518917 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7a40155960 0x7f7a40151e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:57.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.190+0000 7f7a3f400640 1 -- 172.21.15.28:0/3687518917 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7a40155960 msgr2=0x7f7a40151e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:57.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.190+0000 7f7a3f400640 1 --2- 172.21.15.28:0/3687518917 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7a40155960 0x7f7a40151e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:57.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.190+0000 7f7a3f400640 1 -- 172.21.15.28:0/3687518917 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a400aff90 msgr2=0x7f7a401518c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:57.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.190+0000 7f7a3f400640 1 --2- 172.21.15.28:0/3687518917 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a400aff90 0x7f7a401518c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:57.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.190+0000 7f7a3f400640 1 -- 172.21.15.28:0/3687518917 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a400c1510 con 0x7f7a4006dbf0 2024-02-21T23:10:57.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.191+0000 7f7a3f400640 1 --2- 172.21.15.28:0/3687518917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a4006dbf0 0x7f7a400c3560 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7f7a3000c790 tx=0x7f7a3000cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:57.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.191+0000 7f7a37e00640 1 -- 172.21.15.28:0/3687518917 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a30016020 con 0x7f7a4006dbf0 2024-02-21T23:10:57.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.191+0000 7f7a37e00640 1 -- 172.21.15.28:0/3687518917 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a30006d50 con 0x7f7a4006dbf0 2024-02-21T23:10:57.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.191+0000 7f7a37e00640 1 -- 172.21.15.28:0/3687518917 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a30003bb0 con 0x7f7a4006dbf0 2024-02-21T23:10:57.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.191+0000 7f7a45200640 1 -- 172.21.15.28:0/3687518917 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a400c1a40 con 0x7f7a4006dbf0 2024-02-21T23:10:57.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.191+0000 7f7a45200640 1 -- 172.21.15.28:0/3687518917 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a40157580 con 0x7f7a4006dbf0 2024-02-21T23:10:57.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.193+0000 7f7a45200640 1 -- 172.21.15.28:0/3687518917 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a400c31b0 con 0x7f7a4006dbf0 2024-02-21T23:10:57.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.193+0000 7f7a37e00640 1 -- 172.21.15.28:0/3687518917 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7a3000a850 con 0x7f7a4006dbf0 2024-02-21T23:10:57.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.194+0000 7f7a37e00640 1 --2- 172.21.15.28:0/3687518917 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7a1c06b4d0 0x7f7a1c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:57.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.194+0000 7f7a37e00640 1 -- 172.21.15.28:0/3687518917 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f7a3008a0a0 con 0x7f7a4006dbf0 2024-02-21T23:10:57.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.195+0000 7f7a36a00640 1 --2- 172.21.15.28:0/3687518917 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7a1c06b4d0 0x7f7a1c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:57.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.196+0000 7f7a36a00640 1 --2- 172.21.15.28:0/3687518917 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7a1c06b4d0 0x7f7a1c06d990 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f7a401e21e0 tx=0x7f7a28008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:57.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.198+0000 7f7a37e00640 1 -- 172.21.15.28:0/3687518917 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7a400c31b0 con 0x7f7a4006dbf0 2024-02-21T23:10:57.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.510+0000 7f7a45200640 1 -- 172.21.15.28:0/3687518917 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7a4004f290 con 0x7f7a4006dbf0 2024-02-21T23:10:57.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.511+0000 7f7a37e00640 1 -- 172.21.15.28:0/3687518917 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 185 v185) v1 ==== 95+0+4945 (secure 0 0 0) 0x7f7a30020020 con 0x7f7a4006dbf0 2024-02-21T23:10:57.499 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:10:57.499 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":185,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6686,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/155806034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":155806034},{"type":"v1","addr":"172.21.15.81:6836","nonce":155806034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":180},{"gid":7480,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3897243745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":3897243745},{"type":"v1","addr":"172.21.15.81:6837","nonce":3897243745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":180},{"gid":7486,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2827135632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2827135632},{"type":"v1","addr":"172.21.15.28:6835","nonce":2827135632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":180}],"filesystems":[{"mdsmap":{"epoch":185,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:10:53.351225+0000","modified":"2024-02-21T23:10:56.419054+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9480},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9480":{"gid":9480,"name":"c","rank":0,"incarnation":181,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6833/1850463609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1850463609},{"type":"v1","addr":"172.21.15.28:6833","nonce":1850463609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2024-02-21T23:10:57.499 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 185 2024-02-21T23:10:57.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.514+0000 7f7a45200640 1 -- 172.21.15.28:0/3687518917 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7a1c06b4d0 msgr2=0x7f7a1c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:57.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.514+0000 7f7a45200640 1 --2- 172.21.15.28:0/3687518917 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7a1c06b4d0 0x7f7a1c06d990 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f7a401e21e0 tx=0x7f7a28008040 comp rx=0 tx=0).stop 2024-02-21T23:10:57.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.514+0000 7f7a45200640 1 -- 172.21.15.28:0/3687518917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a4006dbf0 msgr2=0x7f7a400c3560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:57.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.514+0000 7f7a45200640 1 --2- 172.21.15.28:0/3687518917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a4006dbf0 0x7f7a400c3560 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7f7a3000c790 tx=0x7f7a3000cc50 comp rx=0 tx=0).stop 2024-02-21T23:10:57.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.514+0000 7f7a45200640 1 -- 172.21.15.28:0/3687518917 shutdown_connections 2024-02-21T23:10:57.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.514+0000 7f7a45200640 1 --2- 172.21.15.28:0/3687518917 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7a1c06b4d0 0x7f7a1c06d990 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:57.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.514+0000 7f7a45200640 1 --2- 172.21.15.28:0/3687518917 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7a40155960 0x7f7a40151e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:57.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.514+0000 7f7a45200640 1 --2- 172.21.15.28:0/3687518917 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a400aff90 0x7f7a401518c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:57.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.514+0000 7f7a45200640 1 --2- 172.21.15.28:0/3687518917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a4006dbf0 0x7f7a400c3560 unknown :-1 s=CLOSED pgs=1923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:57.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.514+0000 7f7a45200640 1 -- 172.21.15.28:0/3687518917 >> 172.21.15.28:0/3687518917 conn(0x7f7a400ac370 msgr2=0x7f7a400c50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:57.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.515+0000 7f7a45200640 1 -- 172.21.15.28:0/3687518917 shutdown_connections 2024-02-21T23:10:57.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.515+0000 7f7a45200640 1 -- 172.21.15.28:0/3687518917 wait complete. 2024-02-21T23:10:57.523 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:10:57.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.753+0000 7fb2c6600640 1 Processor -- start 2024-02-21T23:10:57.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.753+0000 7fb2c6600640 1 -- start start 2024-02-21T23:10:57.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.756+0000 7fb2c6600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fb2c01524a0 0x7fb2c00c7620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:57.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.756+0000 7fb2c6600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb2c00c45d0 0x7fb2c0155510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:57.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.756+0000 7fb2c6600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2c006ba00 con 0x7fb2c00c7b60 2024-02-21T23:10:57.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.756+0000 7fb2c6600640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2c006f2d0 con 0x7fb2c01524a0 2024-02-21T23:10:57.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.756+0000 7fb2c6600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c0154ff0 con 0x7fb2c00c45d0 2024-02-21T23:10:57.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.757+0000 7fb2bfe00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb2c00c45d0 0x7fb2c0155510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:57.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.757+0000 7fb2bfe00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb2c00c45d0 0x7fb2c0155510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3301/0 says I am v2:172.21.15.28:56598/0 (socket says 172.21.15.28:56598) 2024-02-21T23:10:57.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.757+0000 7fb2bfe00640 1 -- 172.21.15.28:0/1517602837 learned_addr learned my addr 172.21.15.28:0/1517602837 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:57.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.757+0000 7fb2c5c00640 1 --1- 172.21.15.28:0/1517602837 >> v1:172.21.15.81:6789/0 conn(0x7fb2c00c7b60 0x7fb2c00a93d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.28:35586/0 (socket says 172.21.15.28:35586) 2024-02-21T23:10:57.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.757+0000 7fb2c5200640 1 --2- 172.21.15.28:0/1517602837 >> v2:172.21.15.81:3300/0 conn(0x7fb2c01524a0 0x7fb2c00c7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:57.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.758+0000 7fb2bfe00640 1 -- 172.21.15.28:0/1517602837 >> v2:172.21.15.81:3300/0 conn(0x7fb2c01524a0 msgr2=0x7fb2c00c7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:57.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.758+0000 7fb2bfe00640 1 --2- 172.21.15.28:0/1517602837 >> v2:172.21.15.81:3300/0 conn(0x7fb2c01524a0 0x7fb2c00c7620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:57.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.758+0000 7fb2bfe00640 1 -- 172.21.15.28:0/1517602837 >> v1:172.21.15.81:6789/0 conn(0x7fb2c00c7b60 legacy=0x7fb2c00a93d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:57.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.758+0000 7fb2bfe00640 1 -- 172.21.15.28:0/1517602837 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2c006aee0 con 0x7fb2c00c45d0 2024-02-21T23:10:57.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.759+0000 7fb2bfe00640 1 --2- 172.21.15.28:0/1517602837 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb2c00c45d0 0x7fb2c0155510 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb2ac009870 tx=0x7fb2ac02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39c01d63809327cc server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:57.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.760+0000 7fb2bf400640 1 -- 172.21.15.28:0/1517602837 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2ac034070 con 0x7fb2c00c45d0 2024-02-21T23:10:57.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.760+0000 7fb2bf400640 1 -- 172.21.15.28:0/1517602837 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2ac030840 con 0x7fb2c00c45d0 2024-02-21T23:10:57.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.760+0000 7fb2bf400640 1 -- 172.21.15.28:0/1517602837 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2ac030c10 con 0x7fb2c00c45d0 2024-02-21T23:10:57.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.760+0000 7fb2c6600640 1 -- 172.21.15.28:0/1517602837 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb2c00c45d0 msgr2=0x7fb2c0155510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:57.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.760+0000 7fb2c6600640 1 --2- 172.21.15.28:0/1517602837 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb2c00c45d0 0x7fb2c0155510 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb2ac009870 tx=0x7fb2ac02b980 comp rx=0 tx=0).stop 2024-02-21T23:10:57.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.761+0000 7fb2c6600640 1 -- 172.21.15.28:0/1517602837 shutdown_connections 2024-02-21T23:10:57.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.761+0000 7fb2c6600640 1 --2- 172.21.15.28:0/1517602837 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb2c00c45d0 0x7fb2c0155510 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:57.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.761+0000 7fb2c6600640 1 --2- 172.21.15.28:0/1517602837 >> v2:172.21.15.81:3300/0 conn(0x7fb2c01524a0 0x7fb2c00c7620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:57.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.761+0000 7fb2c6600640 1 -- 172.21.15.28:0/1517602837 >> 172.21.15.28:0/1517602837 conn(0x7fb2c0067050 msgr2=0x7fb2c00a84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:57.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.761+0000 7fb2c6600640 1 -- 172.21.15.28:0/1517602837 shutdown_connections 2024-02-21T23:10:57.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.761+0000 7fb2c6600640 1 -- 172.21.15.28:0/1517602837 wait complete. 2024-02-21T23:10:57.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.762+0000 7fb2c6600640 1 Processor -- start 2024-02-21T23:10:57.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.763+0000 7fb2c6600640 1 -- start start 2024-02-21T23:10:57.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.763+0000 7fb2c6600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2c00c45d0 0x7fb2c00c54c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:57.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.764+0000 7fb2c6600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb2c00c7b60 0x7fb2c0152d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:57.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.764+0000 7fb2c6600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2c01524a0 0x7fb2c00c5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:57.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.764+0000 7fb2c6600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c00c4fa0 con 0x7fb2c01524a0 2024-02-21T23:10:57.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.764+0000 7fb2c6600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c006f5b0 con 0x7fb2c00c45d0 2024-02-21T23:10:57.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.764+0000 7fb2c6600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c006b460 con 0x7fb2c00c7b60 2024-02-21T23:10:57.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.764+0000 7fb2c5c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2c01524a0 0x7fb2c00c5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:57.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.764+0000 7fb2c5c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2c01524a0 0x7fb2c00c5fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33012/0 (socket says 172.21.15.28:33012) 2024-02-21T23:10:57.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.764+0000 7fb2c5c00640 1 -- 172.21.15.28:0/79471899 learned_addr learned my addr 172.21.15.28:0/79471899 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:57.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.764+0000 7fb2bfe00640 1 --2- 172.21.15.28:0/79471899 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb2c00c7b60 0x7fb2c0152d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:57.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.764+0000 7fb2c5200640 1 --2- 172.21.15.28:0/79471899 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2c00c45d0 0x7fb2c00c54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:57.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.765+0000 7fb2c5c00640 1 -- 172.21.15.28:0/79471899 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb2c00c7b60 msgr2=0x7fb2c0152d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:57.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.765+0000 7fb2c5c00640 1 --2- 172.21.15.28:0/79471899 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb2c00c7b60 0x7fb2c0152d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:57.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.765+0000 7fb2c5c00640 1 -- 172.21.15.28:0/79471899 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2c00c45d0 msgr2=0x7fb2c00c54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:57.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.765+0000 7fb2c5c00640 1 --2- 172.21.15.28:0/79471899 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2c00c45d0 0x7fb2c00c54c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:57.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.765+0000 7fb2c5c00640 1 -- 172.21.15.28:0/79471899 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2c01e2370 con 0x7fb2c01524a0 2024-02-21T23:10:57.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.765+0000 7fb2bfe00640 1 --2- 172.21.15.28:0/79471899 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb2c00c7b60 0x7fb2c0152d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:10:57.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.765+0000 7fb2c5200640 1 --2- 172.21.15.28:0/79471899 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2c00c45d0 0x7fb2c00c54c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:10:57.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.765+0000 7fb2c5c00640 1 --2- 172.21.15.28:0/79471899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2c01524a0 0x7fb2c00c5fc0 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7fb2b800c7b0 tx=0x7fb2b800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:57.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.765+0000 7fb2bcc00640 1 -- 172.21.15.28:0/79471899 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2b8017070 con 0x7fb2c01524a0 2024-02-21T23:10:57.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.765+0000 7fb2bcc00640 1 -- 172.21.15.28:0/79471899 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2b800a640 con 0x7fb2c01524a0 2024-02-21T23:10:57.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.765+0000 7fb2bcc00640 1 -- 172.21.15.28:0/79471899 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2b800a9f0 con 0x7fb2c01524a0 2024-02-21T23:10:57.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.765+0000 7fb2c6600640 1 -- 172.21.15.28:0/79471899 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2c01e2090 con 0x7fb2c01524a0 2024-02-21T23:10:57.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.765+0000 7fb2c6600640 1 -- 172.21.15.28:0/79471899 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2c01eeaa0 con 0x7fb2c01524a0 2024-02-21T23:10:57.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.767+0000 7fb2c6600640 1 -- 172.21.15.28:0/79471899 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2c004f290 con 0x7fb2c01524a0 2024-02-21T23:10:57.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.768+0000 7fb2bcc00640 1 -- 172.21.15.28:0/79471899 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb2b801a070 con 0x7fb2c01524a0 2024-02-21T23:10:57.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.769+0000 7fb2bcc00640 1 --2- 172.21.15.28:0/79471899 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb2a006b4d0 0x7fb2a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:57.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.769+0000 7fb2bcc00640 1 -- 172.21.15.28:0/79471899 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb2b8089860 con 0x7fb2c01524a0 2024-02-21T23:10:57.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.769+0000 7fb2c5200640 1 --2- 172.21.15.28:0/79471899 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb2a006b4d0 0x7fb2a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:57.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.770+0000 7fb2c5200640 1 --2- 172.21.15.28:0/79471899 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb2a006b4d0 0x7fb2a006d990 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fb2c01df6a0 tx=0x7fb2b4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:57.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.778+0000 7fb2bcc00640 1 -- 172.21.15.28:0/79471899 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb2b8057f90 con 0x7fb2c01524a0 2024-02-21T23:10:57.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.987+0000 7fb2c6600640 1 -- 172.21.15.28:0/79471899 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb2c00c4a70 con 0x7fb2c01524a0 2024-02-21T23:10:57.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.989+0000 7fb2bcc00640 1 -- 172.21.15.28:0/79471899 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v174) v1 ==== 74+0+16892 (secure 0 0 0) 0x7fb2b805c030 con 0x7fb2c01524a0 2024-02-21T23:10:57.977 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:10:57.977 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":174,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:10:54.863442+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:10:51.223834+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"173","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2024-02-21T23:10:52.368371+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"174","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:10:57.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.992+0000 7fb2c6600640 1 -- 172.21.15.28:0/79471899 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb2a006b4d0 msgr2=0x7fb2a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:57.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.992+0000 7fb2c6600640 1 --2- 172.21.15.28:0/79471899 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb2a006b4d0 0x7fb2a006d990 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fb2c01df6a0 tx=0x7fb2b4008040 comp rx=0 tx=0).stop 2024-02-21T23:10:57.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.992+0000 7fb2c6600640 1 -- 172.21.15.28:0/79471899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2c01524a0 msgr2=0x7fb2c00c5fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:57.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.992+0000 7fb2c6600640 1 --2- 172.21.15.28:0/79471899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2c01524a0 0x7fb2c00c5fc0 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7fb2b800c7b0 tx=0x7fb2b800cc70 comp rx=0 tx=0).stop 2024-02-21T23:10:57.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.992+0000 7fb2c6600640 1 -- 172.21.15.28:0/79471899 shutdown_connections 2024-02-21T23:10:57.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.992+0000 7fb2c6600640 1 --2- 172.21.15.28:0/79471899 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb2a006b4d0 0x7fb2a006d990 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:57.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.992+0000 7fb2c6600640 1 --2- 172.21.15.28:0/79471899 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb2c00c7b60 0x7fb2c0152d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:57.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.992+0000 7fb2c6600640 1 --2- 172.21.15.28:0/79471899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2c01524a0 0x7fb2c00c5fc0 unknown :-1 s=CLOSED pgs=1924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:57.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.992+0000 7fb2c6600640 1 --2- 172.21.15.28:0/79471899 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2c00c45d0 0x7fb2c00c54c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:57.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.992+0000 7fb2c6600640 1 -- 172.21.15.28:0/79471899 >> 172.21.15.28:0/79471899 conn(0x7fb2c0067050 msgr2=0x7fb2c0150520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:57.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.992+0000 7fb2c6600640 1 -- 172.21.15.28:0/79471899 shutdown_connections 2024-02-21T23:10:57.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:57.993+0000 7fb2c6600640 1 -- 172.21.15.28:0/79471899 wait complete. 2024-02-21T23:10:58.006 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-02-21T23:10:58.006 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-21T23:10:58.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.217+0000 7fe592a00640 1 Processor -- start 2024-02-21T23:10:58.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.217+0000 7fe592a00640 1 -- start start 2024-02-21T23:10:58.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.218+0000 7fe592a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe58c150450 0x7fe58c0a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:58.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.218+0000 7fe592a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe58c06ee40 0x7fe58c1536f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:58.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.218+0000 7fe592a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe58c0c2760 con 0x7fe58c156af0 2024-02-21T23:10:58.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.218+0000 7fe592a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe58c0c22c0 con 0x7fe58c150450 2024-02-21T23:10:58.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.218+0000 7fe592a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe58c156660 con 0x7fe58c06ee40 2024-02-21T23:10:58.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.219+0000 7fe591600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe58c150450 0x7fe58c0a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:58.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.219+0000 7fe591600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe58c150450 0x7fe58c0a5fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33022/0 (socket says 172.21.15.28:33022) 2024-02-21T23:10:58.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.219+0000 7fe591600640 1 -- 172.21.15.28:0/3735861506 learned_addr learned my addr 172.21.15.28:0/3735861506 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:58.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.219+0000 7fe590c00640 1 --2- 172.21.15.28:0/3735861506 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe58c06ee40 0x7fe58c1536f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:58.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.219+0000 7fe591600640 1 -- 172.21.15.28:0/3735861506 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe58c06ee40 msgr2=0x7fe58c1536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:58.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.219+0000 7fe591600640 1 --2- 172.21.15.28:0/3735861506 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe58c06ee40 0x7fe58c1536f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:58.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.219+0000 7fe591600640 1 -- 172.21.15.28:0/3735861506 >> v1:172.21.15.81:6789/0 conn(0x7fe58c156af0 legacy=0x7fe58c06dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:58.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.219+0000 7fe591600640 1 -- 172.21.15.28:0/3735861506 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe58c153c30 con 0x7fe58c150450 2024-02-21T23:10:58.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.221+0000 7fe591600640 1 --2- 172.21.15.28:0/3735861506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe58c150450 0x7fe58c0a5fd0 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7fe580009870 tx=0x7fe58002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2208d8fa5dc1de2 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:58.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.221+0000 7fe58be00640 1 -- 172.21.15.28:0/3735861506 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe580034070 con 0x7fe58c150450 2024-02-21T23:10:58.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.221+0000 7fe58be00640 1 -- 172.21.15.28:0/3735861506 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe580030840 con 0x7fe58c150450 2024-02-21T23:10:58.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.221+0000 7fe58be00640 1 -- 172.21.15.28:0/3735861506 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe580030c10 con 0x7fe58c150450 2024-02-21T23:10:58.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.222+0000 7fe592a00640 1 -- 172.21.15.28:0/3735861506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe58c150450 msgr2=0x7fe58c0a5fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:58.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.222+0000 7fe592a00640 1 --2- 172.21.15.28:0/3735861506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe58c150450 0x7fe58c0a5fd0 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7fe580009870 tx=0x7fe58002b980 comp rx=0 tx=0).stop 2024-02-21T23:10:58.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.222+0000 7fe592a00640 1 -- 172.21.15.28:0/3735861506 shutdown_connections 2024-02-21T23:10:58.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.222+0000 7fe592a00640 1 --2- 172.21.15.28:0/3735861506 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe58c06ee40 0x7fe58c1536f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:58.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.222+0000 7fe592a00640 1 --2- 172.21.15.28:0/3735861506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe58c150450 0x7fe58c0a5fd0 unknown :-1 s=CLOSED pgs=1925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:58.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.222+0000 7fe592a00640 1 -- 172.21.15.28:0/3735861506 >> 172.21.15.28:0/3735861506 conn(0x7fe58c06b630 msgr2=0x7fe58c0c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:58.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.222+0000 7fe592a00640 1 -- 172.21.15.28:0/3735861506 shutdown_connections 2024-02-21T23:10:58.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.222+0000 7fe592a00640 1 -- 172.21.15.28:0/3735861506 wait complete. 2024-02-21T23:10:58.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.224+0000 7fe592a00640 1 Processor -- start 2024-02-21T23:10:58.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.224+0000 7fe592a00640 1 -- start start 2024-02-21T23:10:58.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.225+0000 7fe592a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe58c06ee40 0x7fe58c151f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:58.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.225+0000 7fe592a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe58c150450 0x7fe58c0aa190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:58.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.225+0000 7fe592a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe58c156af0 0x7fe58c147730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:58.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.225+0000 7fe592a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe58c0c2cb0 con 0x7fe58c06ee40 2024-02-21T23:10:58.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.225+0000 7fe592a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe58c0c29d0 con 0x7fe58c150450 2024-02-21T23:10:58.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.225+0000 7fe592a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe58c06e700 con 0x7fe58c156af0 2024-02-21T23:10:58.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.225+0000 7fe591600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe58c06ee40 0x7fe58c151f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:58.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.225+0000 7fe591600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe58c06ee40 0x7fe58c151f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33032/0 (socket says 172.21.15.28:33032) 2024-02-21T23:10:58.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.225+0000 7fe591600640 1 -- 172.21.15.28:0/1825121003 learned_addr learned my addr 172.21.15.28:0/1825121003 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:58.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.225+0000 7fe590c00640 1 --2- 172.21.15.28:0/1825121003 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe58c150450 0x7fe58c0aa190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:58.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.225+0000 7fe592000640 1 --2- 172.21.15.28:0/1825121003 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe58c156af0 0x7fe58c147730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:58.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.225+0000 7fe591600640 1 -- 172.21.15.28:0/1825121003 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe58c156af0 msgr2=0x7fe58c147730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:58.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.226+0000 7fe591600640 1 --2- 172.21.15.28:0/1825121003 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe58c156af0 0x7fe58c147730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:58.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.226+0000 7fe591600640 1 -- 172.21.15.28:0/1825121003 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe58c150450 msgr2=0x7fe58c0aa190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:58.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.226+0000 7fe591600640 1 --2- 172.21.15.28:0/1825121003 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe58c150450 0x7fe58c0aa190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:58.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.226+0000 7fe591600640 1 -- 172.21.15.28:0/1825121003 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe58c0abaf0 con 0x7fe58c06ee40 2024-02-21T23:10:58.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.226+0000 7fe592000640 1 --2- 172.21.15.28:0/1825121003 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe58c156af0 0x7fe58c147730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:10:58.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.226+0000 7fe591600640 1 --2- 172.21.15.28:0/1825121003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe58c06ee40 0x7fe58c151f20 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7fe580000c00 tx=0x7fe580004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:58.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.226+0000 7fe589600640 1 -- 172.21.15.28:0/1825121003 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe580048070 con 0x7fe58c06ee40 2024-02-21T23:10:58.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.226+0000 7fe589600640 1 -- 172.21.15.28:0/1825121003 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe580030640 con 0x7fe58c06ee40 2024-02-21T23:10:58.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.226+0000 7fe592a00640 1 -- 172.21.15.28:0/1825121003 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe58c0a9440 con 0x7fe58c06ee40 2024-02-21T23:10:58.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.226+0000 7fe589600640 1 -- 172.21.15.28:0/1825121003 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe580034040 con 0x7fe58c06ee40 2024-02-21T23:10:58.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.226+0000 7fe592a00640 1 -- 172.21.15.28:0/1825121003 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe58c0ab2f0 con 0x7fe58c06ee40 2024-02-21T23:10:58.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.228+0000 7fe592a00640 1 -- 172.21.15.28:0/1825121003 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe58c04f290 con 0x7fe58c06ee40 2024-02-21T23:10:58.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.229+0000 7fe589600640 1 -- 172.21.15.28:0/1825121003 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe58004b070 con 0x7fe58c06ee40 2024-02-21T23:10:58.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.230+0000 7fe589600640 1 --2- 172.21.15.28:0/1825121003 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe56c06b4d0 0x7fe56c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:58.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.230+0000 7fe589600640 1 -- 172.21.15.28:0/1825121003 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe5800ac2f0 con 0x7fe58c06ee40 2024-02-21T23:10:58.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.230+0000 7fe590c00640 1 --2- 172.21.15.28:0/1825121003 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe56c06b4d0 0x7fe56c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:58.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.232+0000 7fe590c00640 1 --2- 172.21.15.28:0/1825121003 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe56c06b4d0 0x7fe56c06d990 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fe58c14dd00 tx=0x7fe578008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:58.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.234+0000 7fe589600640 1 -- 172.21.15.28:0/1825121003 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe58007af50 con 0x7fe58c06ee40 2024-02-21T23:10:58.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.429+0000 7fe592a00640 1 -- 172.21.15.28:0/1825121003 --> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe58c06be20 con 0x7fe56c06b4d0 2024-02-21T23:10:58.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.448+0000 7fe589600640 1 -- 172.21.15.28:0/1825121003 <== mgr.4101 v2:172.21.15.81:6832/1848538577 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252549 (secure 0 0 0) 0x7fe58c06be20 con 0x7fe56c06b4d0 2024-02-21T23:10:58.439 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:10:58.441 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-21T23:10:58.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.453+0000 7fe592a00640 1 -- 172.21.15.28:0/1825121003 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe56c06b4d0 msgr2=0x7fe56c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:58.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.453+0000 7fe592a00640 1 --2- 172.21.15.28:0/1825121003 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe56c06b4d0 0x7fe56c06d990 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fe58c14dd00 tx=0x7fe578008040 comp rx=0 tx=0).stop 2024-02-21T23:10:58.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.453+0000 7fe592a00640 1 -- 172.21.15.28:0/1825121003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe58c06ee40 msgr2=0x7fe58c151f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:58.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.453+0000 7fe592a00640 1 --2- 172.21.15.28:0/1825121003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe58c06ee40 0x7fe58c151f20 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7fe580000c00 tx=0x7fe580004730 comp rx=0 tx=0).stop 2024-02-21T23:10:58.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.453+0000 7fe592a00640 1 -- 172.21.15.28:0/1825121003 shutdown_connections 2024-02-21T23:10:58.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.453+0000 7fe592a00640 1 --2- 172.21.15.28:0/1825121003 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe56c06b4d0 0x7fe56c06d990 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:58.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.453+0000 7fe592a00640 1 --2- 172.21.15.28:0/1825121003 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe58c156af0 0x7fe58c147730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:58.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.453+0000 7fe592a00640 1 --2- 172.21.15.28:0/1825121003 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe58c150450 0x7fe58c0aa190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:58.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.453+0000 7fe592a00640 1 --2- 172.21.15.28:0/1825121003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe58c06ee40 0x7fe58c151f20 unknown :-1 s=CLOSED pgs=1926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:58.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.453+0000 7fe592a00640 1 -- 172.21.15.28:0/1825121003 >> 172.21.15.28:0/1825121003 conn(0x7fe58c06b630 msgr2=0x7fe58c0c3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:58.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.453+0000 7fe592a00640 1 -- 172.21.15.28:0/1825121003 shutdown_connections 2024-02-21T23:10:58.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.453+0000 7fe592a00640 1 -- 172.21.15.28:0/1825121003 wait complete. 2024-02-21T23:10:58.464 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":420,"stamp":"2024-02-21T23:10:56.843030+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":121,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":270960,"kb_used_data":17340,"kb_used_omap":12,"kb_used_meta":253491,"kb_avail":754703760,"statfs":{"total":773094113280,"available":772816650240,"internally_reserved":0,"allocated":17756160,"data_stored":9844632,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":259575345},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.999380"},"pg_stats":[{"pgid":"28.3f","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:55.599550+0000","last_change":"2024-02-21T23:10:53.898099+0000","last_active":"2024-02-21T23:10:55.599550+0000","last_peered":"2024-02-21T23:10:55.599550+0000","last_clean":"2024-02-21T23:10:55.599550+0000","last_became_active":"2024-02-21T23:10:53.897696+0000","last_became_peered":"2024-02-21T23:10:53.897696+0000","last_unstale":"2024-02-21T23:10:55.599550+0000","last_undegraded":"2024-02-21T23:10:55.599550+0000","last_fullsized":"2024-02-21T23:10:55.599550+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:14:32.880334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3e","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.871049+0000","last_change":"2024-02-21T23:10:53.905951+0000","last_active":"2024-02-21T23:10:54.871049+0000","last_peered":"2024-02-21T23:10:54.871049+0000","last_clean":"2024-02-21T23:10:54.871049+0000","last_became_active":"2024-02-21T23:10:53.905655+0000","last_became_peered":"2024-02-21T23:10:53.905655+0000","last_unstale":"2024-02-21T23:10:54.871049+0000","last_undegraded":"2024-02-21T23:10:54.871049+0000","last_fullsized":"2024-02-21T23:10:54.871049+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:17:27.590244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.3d","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.929341+0000","last_change":"2024-02-21T23:10:53.896548+0000","last_active":"2024-02-21T23:10:54.929341+0000","last_peered":"2024-02-21T23:10:54.929341+0000","last_clean":"2024-02-21T23:10:54.929341+0000","last_became_active":"2024-02-21T23:10:53.894772+0000","last_became_peered":"2024-02-21T23:10:53.894772+0000","last_unstale":"2024-02-21T23:10:54.929341+0000","last_undegraded":"2024-02-21T23:10:54.929341+0000","last_fullsized":"2024-02-21T23:10:54.929341+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:57:09.153263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.3c","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-21T23:10:53.917820+0000","last_change":"2024-02-21T23:10:53.915191+0000","last_active":"2024-02-21T23:10:53.917820+0000","last_peered":"2024-02-21T23:10:53.917820+0000","last_clean":"2024-02-21T23:10:53.917820+0000","last_became_active":"2024-02-21T23:10:53.914070+0000","last_became_peered":"2024-02-21T23:10:53.914070+0000","last_unstale":"2024-02-21T23:10:53.917820+0000","last_undegraded":"2024-02-21T23:10:53.917820+0000","last_fullsized":"2024-02-21T23:10:53.917820+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:22:51.734335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.3b","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.869144+0000","last_change":"2024-02-21T23:10:53.892135+0000","last_active":"2024-02-21T23:10:54.869144+0000","last_peered":"2024-02-21T23:10:54.869144+0000","last_clean":"2024-02-21T23:10:54.869144+0000","last_became_active":"2024-02-21T23:10:53.889621+0000","last_became_peered":"2024-02-21T23:10:53.889621+0000","last_unstale":"2024-02-21T23:10:54.869144+0000","last_undegraded":"2024-02-21T23:10:54.869144+0000","last_fullsized":"2024-02-21T23:10:54.869144+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:37:41.533392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.3a","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-21T23:10:53.917888+0000","last_change":"2024-02-21T23:10:53.914762+0000","last_active":"2024-02-21T23:10:53.917888+0000","last_peered":"2024-02-21T23:10:53.917888+0000","last_clean":"2024-02-21T23:10:53.917888+0000","last_became_active":"2024-02-21T23:10:53.911360+0000","last_became_peered":"2024-02-21T23:10:53.911360+0000","last_unstale":"2024-02-21T23:10:53.917888+0000","last_undegraded":"2024-02-21T23:10:53.917888+0000","last_fullsized":"2024-02-21T23:10:53.917888+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:45:29.399147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.39","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.868523+0000","last_change":"2024-02-21T23:10:53.889749+0000","last_active":"2024-02-21T23:10:54.868523+0000","last_peered":"2024-02-21T23:10:54.868523+0000","last_clean":"2024-02-21T23:10:54.868523+0000","last_became_active":"2024-02-21T23:10:53.889165+0000","last_became_peered":"2024-02-21T23:10:53.889165+0000","last_unstale":"2024-02-21T23:10:54.868523+0000","last_undegraded":"2024-02-21T23:10:54.868523+0000","last_fullsized":"2024-02-21T23:10:54.868523+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:00:59.328401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.38","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.924902+0000","last_change":"2024-02-21T23:10:53.901315+0000","last_active":"2024-02-21T23:10:54.924902+0000","last_peered":"2024-02-21T23:10:54.924902+0000","last_clean":"2024-02-21T23:10:54.924902+0000","last_became_active":"2024-02-21T23:10:53.901036+0000","last_became_peered":"2024-02-21T23:10:53.901036+0000","last_unstale":"2024-02-21T23:10:54.924902+0000","last_undegraded":"2024-02-21T23:10:54.924902+0000","last_fullsized":"2024-02-21T23:10:54.924902+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:30:57.274763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.37","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-21T23:10:53.912549+0000","last_change":"2024-02-21T23:10:53.907101+0000","last_active":"2024-02-21T23:10:53.912549+0000","last_peered":"2024-02-21T23:10:53.912549+0000","last_clean":"2024-02-21T23:10:53.912549+0000","last_became_active":"2024-02-21T23:10:53.905915+0000","last_became_peered":"2024-02-21T23:10:53.905915+0000","last_unstale":"2024-02-21T23:10:53.912549+0000","last_undegraded":"2024-02-21T23:10:53.912549+0000","last_fullsized":"2024-02-21T23:10:53.912549+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:03:35.208914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.36","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.926078+0000","last_change":"2024-02-21T23:10:53.904481+0000","last_active":"2024-02-21T23:10:54.926078+0000","last_peered":"2024-02-21T23:10:54.926078+0000","last_clean":"2024-02-21T23:10:54.926078+0000","last_became_active":"2024-02-21T23:10:53.903914+0000","last_became_peered":"2024-02-21T23:10:53.903914+0000","last_unstale":"2024-02-21T23:10:54.926078+0000","last_undegraded":"2024-02-21T23:10:54.926078+0000","last_fullsized":"2024-02-21T23:10:54.926078+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:47:33.190260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.35","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.929309+0000","last_change":"2024-02-21T23:10:53.913289+0000","last_active":"2024-02-21T23:10:54.929309+0000","last_peered":"2024-02-21T23:10:54.929309+0000","last_clean":"2024-02-21T23:10:54.929309+0000","last_became_active":"2024-02-21T23:10:53.911721+0000","last_became_peered":"2024-02-21T23:10:53.911721+0000","last_unstale":"2024-02-21T23:10:54.929309+0000","last_undegraded":"2024-02-21T23:10:54.929309+0000","last_fullsized":"2024-02-21T23:10:54.929309+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:07:06.999165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.34","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.868745+0000","last_change":"2024-02-21T23:10:53.901430+0000","last_active":"2024-02-21T23:10:54.868745+0000","last_peered":"2024-02-21T23:10:54.868745+0000","last_clean":"2024-02-21T23:10:54.868745+0000","last_became_active":"2024-02-21T23:10:53.901127+0000","last_became_peered":"2024-02-21T23:10:53.901127+0000","last_unstale":"2024-02-21T23:10:54.868745+0000","last_undegraded":"2024-02-21T23:10:54.868745+0000","last_fullsized":"2024-02-21T23:10:54.868745+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:31:45.048178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.33","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-21T23:10:53.917559+0000","last_change":"2024-02-21T23:10:53.912389+0000","last_active":"2024-02-21T23:10:53.917559+0000","last_peered":"2024-02-21T23:10:53.917559+0000","last_clean":"2024-02-21T23:10:53.917559+0000","last_became_active":"2024-02-21T23:10:53.912025+0000","last_became_peered":"2024-02-21T23:10:53.912025+0000","last_unstale":"2024-02-21T23:10:53.917559+0000","last_undegraded":"2024-02-21T23:10:53.917559+0000","last_fullsized":"2024-02-21T23:10:53.917559+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:03:41.983896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.32","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.925107+0000","last_change":"2024-02-21T23:10:53.908511+0000","last_active":"2024-02-21T23:10:54.925107+0000","last_peered":"2024-02-21T23:10:54.925107+0000","last_clean":"2024-02-21T23:10:54.925107+0000","last_became_active":"2024-02-21T23:10:53.908186+0000","last_became_peered":"2024-02-21T23:10:53.908186+0000","last_unstale":"2024-02-21T23:10:54.925107+0000","last_undegraded":"2024-02-21T23:10:54.925107+0000","last_fullsized":"2024-02-21T23:10:54.925107+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:06:24.650495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.31","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.872305+0000","last_change":"2024-02-21T23:10:53.902086+0000","last_active":"2024-02-21T23:10:54.872305+0000","last_peered":"2024-02-21T23:10:54.872305+0000","last_clean":"2024-02-21T23:10:54.872305+0000","last_became_active":"2024-02-21T23:10:53.901797+0000","last_became_peered":"2024-02-21T23:10:53.901797+0000","last_unstale":"2024-02-21T23:10:54.872305+0000","last_undegraded":"2024-02-21T23:10:54.872305+0000","last_fullsized":"2024-02-21T23:10:54.872305+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:04:38.458011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.30","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.929047+0000","last_change":"2024-02-21T23:10:53.899134+0000","last_active":"2024-02-21T23:10:54.929047+0000","last_peered":"2024-02-21T23:10:54.929047+0000","last_clean":"2024-02-21T23:10:54.929047+0000","last_became_active":"2024-02-21T23:10:53.898401+0000","last_became_peered":"2024-02-21T23:10:53.898401+0000","last_unstale":"2024-02-21T23:10:54.929047+0000","last_undegraded":"2024-02-21T23:10:54.929047+0000","last_fullsized":"2024-02-21T23:10:54.929047+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:02:32.099002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.2f","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-21T23:10:53.913458+0000","last_change":"2024-02-21T23:10:53.909987+0000","last_active":"2024-02-21T23:10:53.913458+0000","last_peered":"2024-02-21T23:10:53.913458+0000","last_clean":"2024-02-21T23:10:53.913458+0000","last_became_active":"2024-02-21T23:10:53.909694+0000","last_became_peered":"2024-02-21T23:10:53.909694+0000","last_unstale":"2024-02-21T23:10:53.913458+0000","last_undegraded":"2024-02-21T23:10:53.913458+0000","last_fullsized":"2024-02-21T23:10:53.913458+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:27:30.670285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.2e","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-21T23:10:53.912301+0000","last_change":"2024-02-21T23:10:53.896968+0000","last_active":"2024-02-21T23:10:53.912301+0000","last_peered":"2024-02-21T23:10:53.912301+0000","last_clean":"2024-02-21T23:10:53.912301+0000","last_became_active":"2024-02-21T23:10:53.896636+0000","last_became_peered":"2024-02-21T23:10:53.896636+0000","last_unstale":"2024-02-21T23:10:53.912301+0000","last_undegraded":"2024-02-21T23:10:53.912301+0000","last_fullsized":"2024-02-21T23:10:53.912301+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:15:19.170148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.2d","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.926216+0000","last_change":"2024-02-21T23:10:53.915122+0000","last_active":"2024-02-21T23:10:54.926216+0000","last_peered":"2024-02-21T23:10:54.926216+0000","last_clean":"2024-02-21T23:10:54.926216+0000","last_became_active":"2024-02-21T23:10:53.914742+0000","last_became_peered":"2024-02-21T23:10:53.914742+0000","last_unstale":"2024-02-21T23:10:54.926216+0000","last_undegraded":"2024-02-21T23:10:54.926216+0000","last_fullsized":"2024-02-21T23:10:54.926216+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:12:07.683138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2c","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.926570+0000","last_change":"2024-02-21T23:10:53.915192+0000","last_active":"2024-02-21T23:10:54.926570+0000","last_peered":"2024-02-21T23:10:54.926570+0000","last_clean":"2024-02-21T23:10:54.926570+0000","last_became_active":"2024-02-21T23:10:53.909666+0000","last_became_peered":"2024-02-21T23:10:53.909666+0000","last_unstale":"2024-02-21T23:10:54.926570+0000","last_undegraded":"2024-02-21T23:10:54.926570+0000","last_fullsized":"2024-02-21T23:10:54.926570+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:02:14.168850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2b","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.925130+0000","last_change":"2024-02-21T23:10:53.906254+0000","last_active":"2024-02-21T23:10:54.925130+0000","last_peered":"2024-02-21T23:10:54.925130+0000","last_clean":"2024-02-21T23:10:54.925130+0000","last_became_active":"2024-02-21T23:10:53.905868+0000","last_became_peered":"2024-02-21T23:10:53.905868+0000","last_unstale":"2024-02-21T23:10:54.925130+0000","last_undegraded":"2024-02-21T23:10:54.925130+0000","last_fullsized":"2024-02-21T23:10:54.925130+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:18:25.131127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2a","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.926000+0000","last_change":"2024-02-21T23:10:53.909219+0000","last_active":"2024-02-21T23:10:54.926000+0000","last_peered":"2024-02-21T23:10:54.926000+0000","last_clean":"2024-02-21T23:10:54.926000+0000","last_became_active":"2024-02-21T23:10:53.908915+0000","last_became_peered":"2024-02-21T23:10:53.908915+0000","last_unstale":"2024-02-21T23:10:54.926000+0000","last_undegraded":"2024-02-21T23:10:54.926000+0000","last_fullsized":"2024-02-21T23:10:54.926000+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:29:03.024967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.29","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.870691+0000","last_change":"2024-02-21T23:10:53.910615+0000","last_active":"2024-02-21T23:10:54.870691+0000","last_peered":"2024-02-21T23:10:54.870691+0000","last_clean":"2024-02-21T23:10:54.870691+0000","last_became_active":"2024-02-21T23:10:53.908789+0000","last_became_peered":"2024-02-21T23:10:53.908789+0000","last_unstale":"2024-02-21T23:10:54.870691+0000","last_undegraded":"2024-02-21T23:10:54.870691+0000","last_fullsized":"2024-02-21T23:10:54.870691+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:23:47.446321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.28","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.868396+0000","last_change":"2024-02-21T23:10:53.886666+0000","last_active":"2024-02-21T23:10:54.868396+0000","last_peered":"2024-02-21T23:10:54.868396+0000","last_clean":"2024-02-21T23:10:54.868396+0000","last_became_active":"2024-02-21T23:10:53.886095+0000","last_became_peered":"2024-02-21T23:10:53.886095+0000","last_unstale":"2024-02-21T23:10:54.868396+0000","last_undegraded":"2024-02-21T23:10:54.868396+0000","last_fullsized":"2024-02-21T23:10:54.868396+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:34:59.396395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.27","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-21T23:10:53.912473+0000","last_change":"2024-02-21T23:10:53.905919+0000","last_active":"2024-02-21T23:10:53.912473+0000","last_peered":"2024-02-21T23:10:53.912473+0000","last_clean":"2024-02-21T23:10:53.912473+0000","last_became_active":"2024-02-21T23:10:53.905404+0000","last_became_peered":"2024-02-21T23:10:53.905404+0000","last_unstale":"2024-02-21T23:10:53.912473+0000","last_undegraded":"2024-02-21T23:10:53.912473+0000","last_fullsized":"2024-02-21T23:10:53.912473+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:51:37.937318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.26","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:55.599545+0000","last_change":"2024-02-21T23:10:53.906183+0000","last_active":"2024-02-21T23:10:55.599545+0000","last_peered":"2024-02-21T23:10:55.599545+0000","last_clean":"2024-02-21T23:10:55.599545+0000","last_became_active":"2024-02-21T23:10:53.905887+0000","last_became_peered":"2024-02-21T23:10:53.905887+0000","last_unstale":"2024-02-21T23:10:55.599545+0000","last_undegraded":"2024-02-21T23:10:55.599545+0000","last_fullsized":"2024-02-21T23:10:55.599545+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:57:09.153263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.25","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.929483+0000","last_change":"2024-02-21T23:10:53.905545+0000","last_active":"2024-02-21T23:10:54.929483+0000","last_peered":"2024-02-21T23:10:54.929483+0000","last_clean":"2024-02-21T23:10:54.929483+0000","last_became_active":"2024-02-21T23:10:53.905234+0000","last_became_peered":"2024-02-21T23:10:53.905234+0000","last_unstale":"2024-02-21T23:10:54.929483+0000","last_undegraded":"2024-02-21T23:10:54.929483+0000","last_fullsized":"2024-02-21T23:10:54.929483+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:49:22.732546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.24","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.868899+0000","last_change":"2024-02-21T23:10:53.895871+0000","last_active":"2024-02-21T23:10:54.868899+0000","last_peered":"2024-02-21T23:10:54.868899+0000","last_clean":"2024-02-21T23:10:54.868899+0000","last_became_active":"2024-02-21T23:10:53.895230+0000","last_became_peered":"2024-02-21T23:10:53.895230+0000","last_unstale":"2024-02-21T23:10:54.868899+0000","last_undegraded":"2024-02-21T23:10:54.868899+0000","last_fullsized":"2024-02-21T23:10:54.868899+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:48:33.675885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.23","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-21T23:10:53.911987+0000","last_change":"2024-02-21T23:10:53.891940+0000","last_active":"2024-02-21T23:10:53.911987+0000","last_peered":"2024-02-21T23:10:53.911987+0000","last_clean":"2024-02-21T23:10:53.911987+0000","last_became_active":"2024-02-21T23:10:53.890248+0000","last_became_peered":"2024-02-21T23:10:53.890248+0000","last_unstale":"2024-02-21T23:10:53.911987+0000","last_undegraded":"2024-02-21T23:10:53.911987+0000","last_fullsized":"2024-02-21T23:10:53.911987+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:01:37.314686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.22","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.872395+0000","last_change":"2024-02-21T23:10:53.904573+0000","last_active":"2024-02-21T23:10:54.872395+0000","last_peered":"2024-02-21T23:10:54.872395+0000","last_clean":"2024-02-21T23:10:54.872395+0000","last_became_active":"2024-02-21T23:10:53.904225+0000","last_became_peered":"2024-02-21T23:10:53.904225+0000","last_unstale":"2024-02-21T23:10:54.872395+0000","last_undegraded":"2024-02-21T23:10:54.872395+0000","last_fullsized":"2024-02-21T23:10:54.872395+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:21:49.171138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.21","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-21T23:10:53.917784+0000","last_change":"2024-02-21T23:10:53.914745+0000","last_active":"2024-02-21T23:10:53.917784+0000","last_peered":"2024-02-21T23:10:53.917784+0000","last_clean":"2024-02-21T23:10:53.917784+0000","last_became_active":"2024-02-21T23:10:53.912720+0000","last_became_peered":"2024-02-21T23:10:53.912720+0000","last_unstale":"2024-02-21T23:10:53.917784+0000","last_undegraded":"2024-02-21T23:10:53.917784+0000","last_fullsized":"2024-02-21T23:10:53.917784+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:38:50.515853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.20","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-21T23:10:53.917580+0000","last_change":"2024-02-21T23:10:53.912834+0000","last_active":"2024-02-21T23:10:53.917580+0000","last_peered":"2024-02-21T23:10:53.917580+0000","last_clean":"2024-02-21T23:10:53.917580+0000","last_became_active":"2024-02-21T23:10:53.912462+0000","last_became_peered":"2024-02-21T23:10:53.912462+0000","last_unstale":"2024-02-21T23:10:53.917580+0000","last_undegraded":"2024-02-21T23:10:53.917580+0000","last_fullsized":"2024-02-21T23:10:53.917580+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:25:24.967052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.870437+0000","last_change":"2024-02-21T23:10:53.904551+0000","last_active":"2024-02-21T23:10:54.870437+0000","last_peered":"2024-02-21T23:10:54.870437+0000","last_clean":"2024-02-21T23:10:54.870437+0000","last_became_active":"2024-02-21T23:10:53.904267+0000","last_became_peered":"2024-02-21T23:10:53.904267+0000","last_unstale":"2024-02-21T23:10:54.870437+0000","last_undegraded":"2024-02-21T23:10:54.870437+0000","last_fullsized":"2024-02-21T23:10:54.870437+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:48:12.418011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.1f","version":"173'4","reported_seq":18,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.873133+0000","last_change":"2024-02-21T23:10:52.900397+0000","last_active":"2024-02-21T23:10:54.873133+0000","last_peered":"2024-02-21T23:10:54.873133+0000","last_clean":"2024-02-21T23:10:54.873133+0000","last_became_active":"2024-02-21T23:10:52.899782+0000","last_became_peered":"2024-02-21T23:10:52.899782+0000","last_unstale":"2024-02-21T23:10:54.873133+0000","last_undegraded":"2024-02-21T23:10:54.873133+0000","last_fullsized":"2024-02-21T23:10:54.873133+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_clean_scrub_stamp":"2024-02-21T23:10:51.827439+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:12:06.667909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:55.598768+0000","last_change":"2024-02-21T23:10:53.886408+0000","last_active":"2024-02-21T23:10:55.598768+0000","last_peered":"2024-02-21T23:10:55.598768+0000","last_clean":"2024-02-21T23:10:55.598768+0000","last_became_active":"2024-02-21T23:10:53.885669+0000","last_became_peered":"2024-02-21T23:10:53.885669+0000","last_unstale":"2024-02-21T23:10:55.598768+0000","last_undegraded":"2024-02-21T23:10:55.598768+0000","last_fullsized":"2024-02-21T23:10:55.598768+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:36:37.264098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1e","version":"173'1","reported_seq":15,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:55.599642+0000","last_change":"2024-02-21T23:10:52.895166+0000","last_active":"2024-02-21T23:10:55.599642+0000","last_peered":"2024-02-21T23:10:55.599642+0000","last_clean":"2024-02-21T23:10:55.599642+0000","last_became_active":"2024-02-21T23:10:52.894688+0000","last_became_peered":"2024-02-21T23:10:52.894688+0000","last_unstale":"2024-02-21T23:10:55.599642+0000","last_undegraded":"2024-02-21T23:10:55.599642+0000","last_fullsized":"2024-02-21T23:10:55.599642+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_clean_scrub_stamp":"2024-02-21T23:10:51.827439+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:03:40.968667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.926326+0000","last_change":"2024-02-21T23:10:53.899207+0000","last_active":"2024-02-21T23:10:54.926326+0000","last_peered":"2024-02-21T23:10:54.926326+0000","last_clean":"2024-02-21T23:10:54.926326+0000","last_became_active":"2024-02-21T23:10:53.898837+0000","last_became_peered":"2024-02-21T23:10:53.898837+0000","last_unstale":"2024-02-21T23:10:54.926326+0000","last_undegraded":"2024-02-21T23:10:54.926326+0000","last_fullsized":"2024-02-21T23:10:54.926326+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:49:24.548041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1d","version":"173'2","reported_seq":16,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.870902+0000","last_change":"2024-02-21T23:10:52.881910+0000","last_active":"2024-02-21T23:10:54.870902+0000","last_peered":"2024-02-21T23:10:54.870902+0000","last_clean":"2024-02-21T23:10:54.870902+0000","last_became_active":"2024-02-21T23:10:52.881576+0000","last_became_peered":"2024-02-21T23:10:52.881576+0000","last_unstale":"2024-02-21T23:10:54.870902+0000","last_undegraded":"2024-02-21T23:10:54.870902+0000","last_fullsized":"2024-02-21T23:10:54.870902+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_clean_scrub_stamp":"2024-02-21T23:10:51.827439+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:03:34.193685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.869463+0000","last_change":"2024-02-21T23:10:53.900136+0000","last_active":"2024-02-21T23:10:54.869463+0000","last_peered":"2024-02-21T23:10:54.869463+0000","last_clean":"2024-02-21T23:10:54.869463+0000","last_became_active":"2024-02-21T23:10:53.899784+0000","last_became_peered":"2024-02-21T23:10:53.899784+0000","last_unstale":"2024-02-21T23:10:54.869463+0000","last_undegraded":"2024-02-21T23:10:54.869463+0000","last_fullsized":"2024-02-21T23:10:54.869463+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:40:51.691428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1c","version":"173'2","reported_seq":14,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-21T23:10:53.925070+0000","last_change":"2024-02-21T23:10:52.888364+0000","last_active":"2024-02-21T23:10:53.925070+0000","last_peered":"2024-02-21T23:10:53.925070+0000","last_clean":"2024-02-21T23:10:53.925070+0000","last_became_active":"2024-02-21T23:10:52.887973+0000","last_became_peered":"2024-02-21T23:10:52.887973+0000","last_unstale":"2024-02-21T23:10:53.925070+0000","last_undegraded":"2024-02-21T23:10:53.925070+0000","last_fullsized":"2024-02-21T23:10:53.925070+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_clean_scrub_stamp":"2024-02-21T23:10:51.827439+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:14:31.865105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.873335+0000","last_change":"2024-02-21T23:10:53.900084+0000","last_active":"2024-02-21T23:10:54.873335+0000","last_peered":"2024-02-21T23:10:54.873335+0000","last_clean":"2024-02-21T23:10:54.873335+0000","last_became_active":"2024-02-21T23:10:53.898640+0000","last_became_peered":"2024-02-21T23:10:53.898640+0000","last_unstale":"2024-02-21T23:10:54.873335+0000","last_undegraded":"2024-02-21T23:10:54.873335+0000","last_fullsized":"2024-02-21T23:10:54.873335+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:59:46.467802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.1b","version":"173'1","reported_seq":13,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-21T23:10:53.924672+0000","last_change":"2024-02-21T23:10:52.897974+0000","last_active":"2024-02-21T23:10:53.924672+0000","last_peered":"2024-02-21T23:10:53.924672+0000","last_clean":"2024-02-21T23:10:53.924672+0000","last_became_active":"2024-02-21T23:10:52.896320+0000","last_became_peered":"2024-02-21T23:10:52.896320+0000","last_unstale":"2024-02-21T23:10:53.924672+0000","last_undegraded":"2024-02-21T23:10:53.924672+0000","last_fullsized":"2024-02-21T23:10:53.924672+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_clean_scrub_stamp":"2024-02-21T23:10:51.827439+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:16:28.625769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.872856+0000","last_change":"2024-02-21T23:10:53.895418+0000","last_active":"2024-02-21T23:10:54.872856+0000","last_peered":"2024-02-21T23:10:54.872856+0000","last_clean":"2024-02-21T23:10:54.872856+0000","last_became_active":"2024-02-21T23:10:53.894976+0000","last_became_peered":"2024-02-21T23:10:53.894976+0000","last_unstale":"2024-02-21T23:10:54.872856+0000","last_undegraded":"2024-02-21T23:10:54.872856+0000","last_fullsized":"2024-02-21T23:10:54.872856+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:15:26.533905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":12,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-21T23:10:53.898663+0000","last_change":"2024-02-21T23:10:52.889108+0000","last_active":"2024-02-21T23:10:53.898663+0000","last_peered":"2024-02-21T23:10:53.898663+0000","last_clean":"2024-02-21T23:10:53.898663+0000","last_became_active":"2024-02-21T23:10:52.888823+0000","last_became_peered":"2024-02-21T23:10:52.888823+0000","last_unstale":"2024-02-21T23:10:53.898663+0000","last_undegraded":"2024-02-21T23:10:53.898663+0000","last_fullsized":"2024-02-21T23:10:53.898663+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_clean_scrub_stamp":"2024-02-21T23:10:51.827439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:56:18.758854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-21T23:10:53.912450+0000","last_change":"2024-02-21T23:10:53.904904+0000","last_active":"2024-02-21T23:10:53.912450+0000","last_peered":"2024-02-21T23:10:53.912450+0000","last_clean":"2024-02-21T23:10:53.912450+0000","last_became_active":"2024-02-21T23:10:53.903873+0000","last_became_peered":"2024-02-21T23:10:53.903873+0000","last_unstale":"2024-02-21T23:10:53.912450+0000","last_undegraded":"2024-02-21T23:10:53.912450+0000","last_fullsized":"2024-02-21T23:10:53.912450+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:12:18.987945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":14,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.929025+0000","last_change":"2024-02-21T23:10:52.889283+0000","last_active":"2024-02-21T23:10:54.929025+0000","last_peered":"2024-02-21T23:10:54.929025+0000","last_clean":"2024-02-21T23:10:54.929025+0000","last_became_active":"2024-02-21T23:10:52.888932+0000","last_became_peered":"2024-02-21T23:10:52.888932+0000","last_unstale":"2024-02-21T23:10:54.929025+0000","last_undegraded":"2024-02-21T23:10:54.929025+0000","last_fullsized":"2024-02-21T23:10:54.929025+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_clean_scrub_stamp":"2024-02-21T23:10:51.827439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:25:23.951823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.872302+0000","last_change":"2024-02-21T23:10:53.915103+0000","last_active":"2024-02-21T23:10:54.872302+0000","last_peered":"2024-02-21T23:10:54.872302+0000","last_clean":"2024-02-21T23:10:54.872302+0000","last_became_active":"2024-02-21T23:10:53.913553+0000","last_became_peered":"2024-02-21T23:10:53.913553+0000","last_unstale":"2024-02-21T23:10:54.872302+0000","last_undegraded":"2024-02-21T23:10:54.872302+0000","last_fullsized":"2024-02-21T23:10:54.872302+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:23:07.424809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":14,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.868931+0000","last_change":"2024-02-21T23:10:52.899068+0000","last_active":"2024-02-21T23:10:54.868931+0000","last_peered":"2024-02-21T23:10:54.868931+0000","last_clean":"2024-02-21T23:10:54.868931+0000","last_became_active":"2024-02-21T23:10:52.898730+0000","last_became_peered":"2024-02-21T23:10:52.898730+0000","last_unstale":"2024-02-21T23:10:54.868931+0000","last_undegraded":"2024-02-21T23:10:54.868931+0000","last_fullsized":"2024-02-21T23:10:54.868931+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_clean_scrub_stamp":"2024-02-21T23:10:51.827439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:00:33.242390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.929103+0000","last_change":"2024-02-21T23:10:53.895655+0000","last_active":"2024-02-21T23:10:54.929103+0000","last_peered":"2024-02-21T23:10:54.929103+0000","last_clean":"2024-02-21T23:10:54.929103+0000","last_became_active":"2024-02-21T23:10:53.895254+0000","last_became_peered":"2024-02-21T23:10:53.895254+0000","last_unstale":"2024-02-21T23:10:54.929103+0000","last_undegraded":"2024-02-21T23:10:54.929103+0000","last_fullsized":"2024-02-21T23:10:54.929103+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:15:08.969451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":12,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-21T23:10:53.903517+0000","last_change":"2024-02-21T23:10:52.901737+0000","last_active":"2024-02-21T23:10:53.903517+0000","last_peered":"2024-02-21T23:10:53.903517+0000","last_clean":"2024-02-21T23:10:53.903517+0000","last_became_active":"2024-02-21T23:10:52.901431+0000","last_became_peered":"2024-02-21T23:10:52.901431+0000","last_unstale":"2024-02-21T23:10:53.903517+0000","last_undegraded":"2024-02-21T23:10:53.903517+0000","last_fullsized":"2024-02-21T23:10:53.903517+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_clean_scrub_stamp":"2024-02-21T23:10:51.827439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:26:03.509779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-21T23:10:53.912024+0000","last_change":"2024-02-21T23:10:53.896153+0000","last_active":"2024-02-21T23:10:53.912024+0000","last_peered":"2024-02-21T23:10:53.912024+0000","last_clean":"2024-02-21T23:10:53.912024+0000","last_became_active":"2024-02-21T23:10:53.892498+0000","last_became_peered":"2024-02-21T23:10:53.892498+0000","last_unstale":"2024-02-21T23:10:53.912024+0000","last_undegraded":"2024-02-21T23:10:53.912024+0000","last_fullsized":"2024-02-21T23:10:53.912024+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:49:22.732546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.15","version":"173'3","reported_seq":15,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-21T23:10:53.928552+0000","last_change":"2024-02-21T23:10:52.880063+0000","last_active":"2024-02-21T23:10:53.928552+0000","last_peered":"2024-02-21T23:10:53.928552+0000","last_clean":"2024-02-21T23:10:53.928552+0000","last_became_active":"2024-02-21T23:10:52.879756+0000","last_became_peered":"2024-02-21T23:10:52.879756+0000","last_unstale":"2024-02-21T23:10:53.928552+0000","last_undegraded":"2024-02-21T23:10:53.928552+0000","last_fullsized":"2024-02-21T23:10:53.928552+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_clean_scrub_stamp":"2024-02-21T23:10:51.827439+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:06:09.990674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.869282+0000","last_change":"2024-02-21T23:10:53.898474+0000","last_active":"2024-02-21T23:10:54.869282+0000","last_peered":"2024-02-21T23:10:54.869282+0000","last_clean":"2024-02-21T23:10:54.869282+0000","last_became_active":"2024-02-21T23:10:53.898135+0000","last_became_peered":"2024-02-21T23:10:53.898135+0000","last_unstale":"2024-02-21T23:10:54.869282+0000","last_undegraded":"2024-02-21T23:10:54.869282+0000","last_fullsized":"2024-02-21T23:10:54.869282+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:17:10.407396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.14","version":"173'6","reported_seq":18,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-21T23:10:53.929259+0000","last_change":"2024-02-21T23:10:52.903798+0000","last_active":"2024-02-21T23:10:53.929259+0000","last_peered":"2024-02-21T23:10:53.929259+0000","last_clean":"2024-02-21T23:10:53.929259+0000","last_became_active":"2024-02-21T23:10:52.903462+0000","last_became_peered":"2024-02-21T23:10:52.903462+0000","last_unstale":"2024-02-21T23:10:53.929259+0000","last_undegraded":"2024-02-21T23:10:53.929259+0000","last_fullsized":"2024-02-21T23:10:53.929259+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_clean_scrub_stamp":"2024-02-21T23:10:51.827439+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:52:00.130991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.870748+0000","last_change":"2024-02-21T23:10:53.897321+0000","last_active":"2024-02-21T23:10:54.870748+0000","last_peered":"2024-02-21T23:10:54.870748+0000","last_clean":"2024-02-21T23:10:54.870748+0000","last_became_active":"2024-02-21T23:10:53.896989+0000","last_became_peered":"2024-02-21T23:10:53.896989+0000","last_unstale":"2024-02-21T23:10:54.870748+0000","last_undegraded":"2024-02-21T23:10:54.870748+0000","last_fullsized":"2024-02-21T23:10:54.870748+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:47:32.514797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.13","version":"173'2","reported_seq":16,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.870621+0000","last_change":"2024-02-21T23:10:52.898947+0000","last_active":"2024-02-21T23:10:54.870621+0000","last_peered":"2024-02-21T23:10:54.870621+0000","last_clean":"2024-02-21T23:10:54.870621+0000","last_became_active":"2024-02-21T23:10:52.898348+0000","last_became_peered":"2024-02-21T23:10:52.898348+0000","last_unstale":"2024-02-21T23:10:54.870621+0000","last_undegraded":"2024-02-21T23:10:54.870621+0000","last_fullsized":"2024-02-21T23:10:54.870621+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_clean_scrub_stamp":"2024-02-21T23:10:51.827439+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:51:36.922089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.929499+0000","last_change":"2024-02-21T23:10:53.910740+0000","last_active":"2024-02-21T23:10:54.929499+0000","last_peered":"2024-02-21T23:10:54.929499+0000","last_clean":"2024-02-21T23:10:54.929499+0000","last_became_active":"2024-02-21T23:10:53.910423+0000","last_became_peered":"2024-02-21T23:10:53.910423+0000","last_unstale":"2024-02-21T23:10:54.929499+0000","last_undegraded":"2024-02-21T23:10:54.929499+0000","last_fullsized":"2024-02-21T23:10:54.929499+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:15:19.170148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":14,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.929135+0000","last_change":"2024-02-21T23:10:52.883726+0000","last_active":"2024-02-21T23:10:54.929135+0000","last_peered":"2024-02-21T23:10:54.929135+0000","last_clean":"2024-02-21T23:10:54.929135+0000","last_became_active":"2024-02-21T23:10:52.883288+0000","last_became_peered":"2024-02-21T23:10:52.883288+0000","last_unstale":"2024-02-21T23:10:54.929135+0000","last_undegraded":"2024-02-21T23:10:54.929135+0000","last_fullsized":"2024-02-21T23:10:54.929135+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_clean_scrub_stamp":"2024-02-21T23:10:51.827439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:27:18.911280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-21T23:10:53.912357+0000","last_change":"2024-02-21T23:10:53.900285+0000","last_active":"2024-02-21T23:10:53.912357+0000","last_peered":"2024-02-21T23:10:53.912357+0000","last_clean":"2024-02-21T23:10:53.912357+0000","last_became_active":"2024-02-21T23:10:53.899976+0000","last_became_peered":"2024-02-21T23:10:53.899976+0000","last_unstale":"2024-02-21T23:10:53.912357+0000","last_undegraded":"2024-02-21T23:10:53.912357+0000","last_fullsized":"2024-02-21T23:10:53.912357+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:09:04.785491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":12,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-21T23:10:53.889742+0000","last_change":"2024-02-21T23:10:52.894474+0000","last_active":"2024-02-21T23:10:53.889742+0000","last_peered":"2024-02-21T23:10:53.889742+0000","last_clean":"2024-02-21T23:10:53.889742+0000","last_became_active":"2024-02-21T23:10:52.894209+0000","last_became_peered":"2024-02-21T23:10:52.894209+0000","last_unstale":"2024-02-21T23:10:53.889742+0000","last_undegraded":"2024-02-21T23:10:53.889742+0000","last_fullsized":"2024-02-21T23:10:53.889742+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_clean_scrub_stamp":"2024-02-21T23:10:51.827439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:15:07.954222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-21T23:10:53.917416+0000","last_change":"2024-02-21T23:10:53.912060+0000","last_active":"2024-02-21T23:10:53.917416+0000","last_peered":"2024-02-21T23:10:53.917416+0000","last_clean":"2024-02-21T23:10:53.917416+0000","last_became_active":"2024-02-21T23:10:53.911032+0000","last_became_peered":"2024-02-21T23:10:53.911032+0000","last_unstale":"2024-02-21T23:10:53.917416+0000","last_undegraded":"2024-02-21T23:10:53.917416+0000","last_fullsized":"2024-02-21T23:10:53.917416+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:27:04.076278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.10","version":"173'8","reported_seq":22,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.868355+0000","last_change":"2024-02-21T23:10:52.891863+0000","last_active":"2024-02-21T23:10:54.868355+0000","last_peered":"2024-02-21T23:10:54.868355+0000","last_clean":"2024-02-21T23:10:54.868355+0000","last_became_active":"2024-02-21T23:10:52.891250+0000","last_became_peered":"2024-02-21T23:10:52.891250+0000","last_unstale":"2024-02-21T23:10:54.868355+0000","last_undegraded":"2024-02-21T23:10:54.868355+0000","last_fullsized":"2024-02-21T23:10:54.868355+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_clean_scrub_stamp":"2024-02-21T23:10:51.827439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:08:22.900530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.924880+0000","last_change":"2024-02-21T23:10:53.905223+0000","last_active":"2024-02-21T23:10:54.924880+0000","last_peered":"2024-02-21T23:10:54.924880+0000","last_clean":"2024-02-21T23:10:54.924880+0000","last_became_active":"2024-02-21T23:10:53.904815+0000","last_became_peered":"2024-02-21T23:10:53.904815+0000","last_unstale":"2024-02-21T23:10:54.924880+0000","last_undegraded":"2024-02-21T23:10:54.924880+0000","last_fullsized":"2024-02-21T23:10:54.924880+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:46:01.374626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.f","version":"173'1","reported_seq":15,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:55.599570+0000","last_change":"2024-02-21T23:10:52.895206+0000","last_active":"2024-02-21T23:10:55.599570+0000","last_peered":"2024-02-21T23:10:55.599570+0000","last_clean":"2024-02-21T23:10:55.599570+0000","last_became_active":"2024-02-21T23:10:52.894487+0000","last_became_peered":"2024-02-21T23:10:52.894487+0000","last_unstale":"2024-02-21T23:10:55.599570+0000","last_undegraded":"2024-02-21T23:10:55.599570+0000","last_fullsized":"2024-02-21T23:10:55.599570+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_clean_scrub_stamp":"2024-02-21T23:10:51.827439+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:27:03.061049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:55.598894+0000","last_change":"2024-02-21T23:10:53.905613+0000","last_active":"2024-02-21T23:10:55.598894+0000","last_peered":"2024-02-21T23:10:55.598894+0000","last_clean":"2024-02-21T23:10:55.598894+0000","last_became_active":"2024-02-21T23:10:53.902571+0000","last_became_peered":"2024-02-21T23:10:53.902571+0000","last_unstale":"2024-02-21T23:10:55.598894+0000","last_undegraded":"2024-02-21T23:10:55.598894+0000","last_fullsized":"2024-02-21T23:10:55.598894+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:02:32.099002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.e","version":"173'2","reported_seq":16,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:55.599773+0000","last_change":"2024-02-21T23:10:52.897066+0000","last_active":"2024-02-21T23:10:55.599773+0000","last_peered":"2024-02-21T23:10:55.599773+0000","last_clean":"2024-02-21T23:10:55.599773+0000","last_became_active":"2024-02-21T23:10:52.896587+0000","last_became_peered":"2024-02-21T23:10:52.896587+0000","last_unstale":"2024-02-21T23:10:55.599773+0000","last_undegraded":"2024-02-21T23:10:55.599773+0000","last_fullsized":"2024-02-21T23:10:55.599773+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_clean_scrub_stamp":"2024-02-21T23:10:51.827439+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:25:23.951823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:55.599346+0000","last_change":"2024-02-21T23:10:53.906960+0000","last_active":"2024-02-21T23:10:55.599346+0000","last_peered":"2024-02-21T23:10:55.599346+0000","last_clean":"2024-02-21T23:10:55.599346+0000","last_became_active":"2024-02-21T23:10:53.904761+0000","last_became_peered":"2024-02-21T23:10:53.904761+0000","last_unstale":"2024-02-21T23:10:55.599346+0000","last_undegraded":"2024-02-21T23:10:55.599346+0000","last_fullsized":"2024-02-21T23:10:55.599346+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:05:47.827081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.d","version":"173'1","reported_seq":15,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.870939+0000","last_change":"2024-02-21T23:10:52.892519+0000","last_active":"2024-02-21T23:10:54.870939+0000","last_peered":"2024-02-21T23:10:54.870939+0000","last_clean":"2024-02-21T23:10:54.870939+0000","last_became_active":"2024-02-21T23:10:52.892211+0000","last_became_peered":"2024-02-21T23:10:52.892211+0000","last_unstale":"2024-02-21T23:10:54.870939+0000","last_undegraded":"2024-02-21T23:10:54.870939+0000","last_fullsized":"2024-02-21T23:10:54.870939+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_clean_scrub_stamp":"2024-02-21T23:10:51.827439+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:27:29.655056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.925325+0000","last_change":"2024-02-21T23:10:53.905439+0000","last_active":"2024-02-21T23:10:54.925325+0000","last_peered":"2024-02-21T23:10:54.925325+0000","last_clean":"2024-02-21T23:10:54.925325+0000","last_became_active":"2024-02-21T23:10:53.905045+0000","last_became_peered":"2024-02-21T23:10:53.905045+0000","last_unstale":"2024-02-21T23:10:54.925325+0000","last_undegraded":"2024-02-21T23:10:54.925325+0000","last_fullsized":"2024-02-21T23:10:54.925325+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:25:20.360028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":14,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.868991+0000","last_change":"2024-02-21T23:10:52.898907+0000","last_active":"2024-02-21T23:10:54.868991+0000","last_peered":"2024-02-21T23:10:54.868991+0000","last_clean":"2024-02-21T23:10:54.868991+0000","last_became_active":"2024-02-21T23:10:52.898548+0000","last_became_peered":"2024-02-21T23:10:52.898548+0000","last_unstale":"2024-02-21T23:10:54.868991+0000","last_undegraded":"2024-02-21T23:10:54.868991+0000","last_fullsized":"2024-02-21T23:10:54.868991+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_clean_scrub_stamp":"2024-02-21T23:10:51.827439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:34:05.957909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.870829+0000","last_change":"2024-02-21T23:10:53.910083+0000","last_active":"2024-02-21T23:10:54.870829+0000","last_peered":"2024-02-21T23:10:54.870829+0000","last_clean":"2024-02-21T23:10:54.870829+0000","last_became_active":"2024-02-21T23:10:53.908966+0000","last_became_peered":"2024-02-21T23:10:53.908966+0000","last_unstale":"2024-02-21T23:10:54.870829+0000","last_undegraded":"2024-02-21T23:10:54.870829+0000","last_fullsized":"2024-02-21T23:10:54.870829+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:59:23.327887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.b","version":"173'2","reported_seq":16,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.869384+0000","last_change":"2024-02-21T23:10:52.883528+0000","last_active":"2024-02-21T23:10:54.869384+0000","last_peered":"2024-02-21T23:10:54.869384+0000","last_clean":"2024-02-21T23:10:54.869384+0000","last_became_active":"2024-02-21T23:10:52.883211+0000","last_became_peered":"2024-02-21T23:10:52.883211+0000","last_unstale":"2024-02-21T23:10:54.869384+0000","last_undegraded":"2024-02-21T23:10:54.869384+0000","last_fullsized":"2024-02-21T23:10:54.869384+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_clean_scrub_stamp":"2024-02-21T23:10:51.827439+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:43:15.583941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-21T23:10:53.917296+0000","last_change":"2024-02-21T23:10:53.911159+0000","last_active":"2024-02-21T23:10:53.917296+0000","last_peered":"2024-02-21T23:10:53.917296+0000","last_clean":"2024-02-21T23:10:53.917296+0000","last_became_active":"2024-02-21T23:10:53.909911+0000","last_became_peered":"2024-02-21T23:10:53.909911+0000","last_unstale":"2024-02-21T23:10:53.917296+0000","last_undegraded":"2024-02-21T23:10:53.917296+0000","last_fullsized":"2024-02-21T23:10:53.917296+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:28:34.665447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":14,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.870442+0000","last_change":"2024-02-21T23:10:52.882518+0000","last_active":"2024-02-21T23:10:54.870442+0000","last_peered":"2024-02-21T23:10:54.870442+0000","last_clean":"2024-02-21T23:10:54.870442+0000","last_became_active":"2024-02-21T23:10:52.882195+0000","last_became_peered":"2024-02-21T23:10:52.882195+0000","last_unstale":"2024-02-21T23:10:54.870442+0000","last_undegraded":"2024-02-21T23:10:54.870442+0000","last_fullsized":"2024-02-21T23:10:54.870442+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_clean_scrub_stamp":"2024-02-21T23:10:51.827439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:09:18.744867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-21T23:10:53.912377+0000","last_change":"2024-02-21T23:10:53.904097+0000","last_active":"2024-02-21T23:10:53.912377+0000","last_peered":"2024-02-21T23:10:53.912377+0000","last_clean":"2024-02-21T23:10:53.912377+0000","last_became_active":"2024-02-21T23:10:53.903781+0000","last_became_peered":"2024-02-21T23:10:53.903781+0000","last_unstale":"2024-02-21T23:10:53.912377+0000","last_undegraded":"2024-02-21T23:10:53.912377+0000","last_fullsized":"2024-02-21T23:10:53.912377+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:57:58.410605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":14,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.872475+0000","last_change":"2024-02-21T23:10:52.897082+0000","last_active":"2024-02-21T23:10:54.872475+0000","last_peered":"2024-02-21T23:10:54.872475+0000","last_clean":"2024-02-21T23:10:54.872475+0000","last_became_active":"2024-02-21T23:10:52.896761+0000","last_became_peered":"2024-02-21T23:10:52.896761+0000","last_unstale":"2024-02-21T23:10:54.872475+0000","last_undegraded":"2024-02-21T23:10:54.872475+0000","last_fullsized":"2024-02-21T23:10:54.872475+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_clean_scrub_stamp":"2024-02-21T23:10:51.827439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:49:23.532812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.926076+0000","last_change":"2024-02-21T23:10:53.904953+0000","last_active":"2024-02-21T23:10:54.926076+0000","last_peered":"2024-02-21T23:10:54.926076+0000","last_clean":"2024-02-21T23:10:54.926076+0000","last_became_active":"2024-02-21T23:10:53.904060+0000","last_became_peered":"2024-02-21T23:10:53.904060+0000","last_unstale":"2024-02-21T23:10:54.926076+0000","last_undegraded":"2024-02-21T23:10:54.926076+0000","last_fullsized":"2024-02-21T23:10:54.926076+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:51:21.189125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":14,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:55.598555+0000","last_change":"2024-02-21T23:10:52.900386+0000","last_active":"2024-02-21T23:10:55.598555+0000","last_peered":"2024-02-21T23:10:55.598555+0000","last_clean":"2024-02-21T23:10:55.598555+0000","last_became_active":"2024-02-21T23:10:52.899985+0000","last_became_peered":"2024-02-21T23:10:52.899985+0000","last_unstale":"2024-02-21T23:10:55.598555+0000","last_undegraded":"2024-02-21T23:10:55.598555+0000","last_fullsized":"2024-02-21T23:10:55.598555+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_clean_scrub_stamp":"2024-02-21T23:10:51.827439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:26:03.509779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.925331+0000","last_change":"2024-02-21T23:10:53.904892+0000","last_active":"2024-02-21T23:10:54.925331+0000","last_peered":"2024-02-21T23:10:54.925331+0000","last_clean":"2024-02-21T23:10:54.925331+0000","last_became_active":"2024-02-21T23:10:53.904403+0000","last_became_peered":"2024-02-21T23:10:53.904403+0000","last_unstale":"2024-02-21T23:10:54.925331+0000","last_undegraded":"2024-02-21T23:10:54.925331+0000","last_fullsized":"2024-02-21T23:10:54.925331+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:10:04.860878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.7","version":"173'2","reported_seq":14,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-21T23:10:53.927293+0000","last_change":"2024-02-21T23:10:52.900882+0000","last_active":"2024-02-21T23:10:53.927293+0000","last_peered":"2024-02-21T23:10:53.927293+0000","last_clean":"2024-02-21T23:10:53.927293+0000","last_became_active":"2024-02-21T23:10:52.900538+0000","last_became_peered":"2024-02-21T23:10:52.900538+0000","last_unstale":"2024-02-21T23:10:53.927293+0000","last_undegraded":"2024-02-21T23:10:53.927293+0000","last_fullsized":"2024-02-21T23:10:53.927293+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_clean_scrub_stamp":"2024-02-21T23:10:51.827439+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:31:09.669165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.929056+0000","last_change":"2024-02-21T23:10:53.905351+0000","last_active":"2024-02-21T23:10:54.929056+0000","last_peered":"2024-02-21T23:10:54.929056+0000","last_clean":"2024-02-21T23:10:54.929056+0000","last_became_active":"2024-02-21T23:10:53.905049+0000","last_became_peered":"2024-02-21T23:10:53.905049+0000","last_unstale":"2024-02-21T23:10:54.929056+0000","last_undegraded":"2024-02-21T23:10:54.929056+0000","last_fullsized":"2024-02-21T23:10:54.929056+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:05:47.827081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.6","version":"174'1","reported_seq":15,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.929149+0000","last_change":"2024-02-21T23:10:52.889360+0000","last_active":"2024-02-21T23:10:54.929149+0000","last_peered":"2024-02-21T23:10:54.929149+0000","last_clean":"2024-02-21T23:10:54.929149+0000","last_became_active":"2024-02-21T23:10:52.889022+0000","last_became_peered":"2024-02-21T23:10:52.889022+0000","last_unstale":"2024-02-21T23:10:54.929149+0000","last_undegraded":"2024-02-21T23:10:54.929149+0000","last_fullsized":"2024-02-21T23:10:54.929149+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_clean_scrub_stamp":"2024-02-21T23:10:51.827439+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:03:10.936536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-21T23:10:53.912129+0000","last_change":"2024-02-21T23:10:53.897689+0000","last_active":"2024-02-21T23:10:53.912129+0000","last_peered":"2024-02-21T23:10:53.912129+0000","last_clean":"2024-02-21T23:10:53.912129+0000","last_became_active":"2024-02-21T23:10:53.897304+0000","last_became_peered":"2024-02-21T23:10:53.897304+0000","last_unstale":"2024-02-21T23:10:53.912129+0000","last_undegraded":"2024-02-21T23:10:53.912129+0000","last_fullsized":"2024-02-21T23:10:53.912129+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:20:14.355944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":14,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.872627+0000","last_change":"2024-02-21T23:10:52.889401+0000","last_active":"2024-02-21T23:10:54.872627+0000","last_peered":"2024-02-21T23:10:54.872627+0000","last_clean":"2024-02-21T23:10:54.872627+0000","last_became_active":"2024-02-21T23:10:52.889142+0000","last_became_peered":"2024-02-21T23:10:52.889142+0000","last_unstale":"2024-02-21T23:10:54.872627+0000","last_undegraded":"2024-02-21T23:10:54.872627+0000","last_fullsized":"2024-02-21T23:10:54.872627+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_clean_scrub_stamp":"2024-02-21T23:10:51.827439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:51:20.173897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:55.599384+0000","last_change":"2024-02-21T23:10:53.903959+0000","last_active":"2024-02-21T23:10:55.599384+0000","last_peered":"2024-02-21T23:10:55.599384+0000","last_clean":"2024-02-21T23:10:55.599384+0000","last_became_active":"2024-02-21T23:10:53.903001+0000","last_became_peered":"2024-02-21T23:10:53.903001+0000","last_unstale":"2024-02-21T23:10:55.599384+0000","last_undegraded":"2024-02-21T23:10:55.599384+0000","last_fullsized":"2024-02-21T23:10:55.599384+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:15:08.969451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":14,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:55.599104+0000","last_change":"2024-02-21T23:10:52.894606+0000","last_active":"2024-02-21T23:10:55.599104+0000","last_peered":"2024-02-21T23:10:55.599104+0000","last_clean":"2024-02-21T23:10:55.599104+0000","last_became_active":"2024-02-21T23:10:52.894230+0000","last_became_peered":"2024-02-21T23:10:52.894230+0000","last_unstale":"2024-02-21T23:10:55.599104+0000","last_undegraded":"2024-02-21T23:10:55.599104+0000","last_fullsized":"2024-02-21T23:10:55.599104+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_clean_scrub_stamp":"2024-02-21T23:10:51.827439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:16:28.625769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.926566+0000","last_change":"2024-02-21T23:10:53.900914+0000","last_active":"2024-02-21T23:10:54.926566+0000","last_peered":"2024-02-21T23:10:54.926566+0000","last_clean":"2024-02-21T23:10:54.926566+0000","last_became_active":"2024-02-21T23:10:53.899241+0000","last_became_peered":"2024-02-21T23:10:53.899241+0000","last_unstale":"2024-02-21T23:10:54.926566+0000","last_undegraded":"2024-02-21T23:10:54.926566+0000","last_fullsized":"2024-02-21T23:10:54.926566+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:15:27.742832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":14,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.925493+0000","last_change":"2024-02-21T23:10:52.892844+0000","last_active":"2024-02-21T23:10:54.925493+0000","last_peered":"2024-02-21T23:10:54.925493+0000","last_clean":"2024-02-21T23:10:54.925493+0000","last_became_active":"2024-02-21T23:10:52.890532+0000","last_became_peered":"2024-02-21T23:10:52.890532+0000","last_unstale":"2024-02-21T23:10:54.925493+0000","last_undegraded":"2024-02-21T23:10:54.925493+0000","last_fullsized":"2024-02-21T23:10:54.925493+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_clean_scrub_stamp":"2024-02-21T23:10:51.827439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:16:06.680298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-21T23:10:53.917656+0000","last_change":"2024-02-21T23:10:53.913555+0000","last_active":"2024-02-21T23:10:53.917656+0000","last_peered":"2024-02-21T23:10:53.917656+0000","last_clean":"2024-02-21T23:10:53.917656+0000","last_became_active":"2024-02-21T23:10:53.912695+0000","last_became_peered":"2024-02-21T23:10:53.912695+0000","last_unstale":"2024-02-21T23:10:53.917656+0000","last_undegraded":"2024-02-21T23:10:53.917656+0000","last_fullsized":"2024-02-21T23:10:53.917656+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:27:19.926509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":12,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-21T23:10:53.890206+0000","last_change":"2024-02-21T23:10:52.886664+0000","last_active":"2024-02-21T23:10:53.890206+0000","last_peered":"2024-02-21T23:10:53.890206+0000","last_clean":"2024-02-21T23:10:53.890206+0000","last_became_active":"2024-02-21T23:10:52.886240+0000","last_became_peered":"2024-02-21T23:10:52.886240+0000","last_unstale":"2024-02-21T23:10:53.890206+0000","last_undegraded":"2024-02-21T23:10:53.890206+0000","last_fullsized":"2024-02-21T23:10:53.890206+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_clean_scrub_stamp":"2024-02-21T23:10:51.827439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:57:08.138034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-21T23:10:53.918765+0000","last_change":"2024-02-21T23:10:53.917131+0000","last_active":"2024-02-21T23:10:53.918765+0000","last_peered":"2024-02-21T23:10:53.918765+0000","last_clean":"2024-02-21T23:10:53.918765+0000","last_became_active":"2024-02-21T23:10:53.916596+0000","last_became_peered":"2024-02-21T23:10:53.916596+0000","last_unstale":"2024-02-21T23:10:53.918765+0000","last_undegraded":"2024-02-21T23:10:53.918765+0000","last_fullsized":"2024-02-21T23:10:53.918765+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:46:22.804312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":14,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:55.599201+0000","last_change":"2024-02-21T23:10:52.903783+0000","last_active":"2024-02-21T23:10:55.599201+0000","last_peered":"2024-02-21T23:10:55.599201+0000","last_clean":"2024-02-21T23:10:55.599201+0000","last_became_active":"2024-02-21T23:10:52.903375+0000","last_became_peered":"2024-02-21T23:10:52.903375+0000","last_unstale":"2024-02-21T23:10:55.599201+0000","last_undegraded":"2024-02-21T23:10:55.599201+0000","last_fullsized":"2024-02-21T23:10:55.599201+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_clean_scrub_stamp":"2024-02-21T23:10:51.827439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:28:33.650218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.929422+0000","last_change":"2024-02-21T23:10:53.906613+0000","last_active":"2024-02-21T23:10:54.929422+0000","last_peered":"2024-02-21T23:10:54.929422+0000","last_clean":"2024-02-21T23:10:54.929422+0000","last_became_active":"2024-02-21T23:10:53.906303+0000","last_became_peered":"2024-02-21T23:10:53.906303+0000","last_unstale":"2024-02-21T23:10:54.929422+0000","last_undegraded":"2024-02-21T23:10:54.929422+0000","last_fullsized":"2024-02-21T23:10:54.929422+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:14:32.880334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":14,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.868462+0000","last_change":"2024-02-21T23:10:52.891131+0000","last_active":"2024-02-21T23:10:54.868462+0000","last_peered":"2024-02-21T23:10:54.868462+0000","last_clean":"2024-02-21T23:10:54.868462+0000","last_became_active":"2024-02-21T23:10:52.890739+0000","last_became_peered":"2024-02-21T23:10:52.890739+0000","last_unstale":"2024-02-21T23:10:54.868462+0000","last_undegraded":"2024-02-21T23:10:54.868462+0000","last_fullsized":"2024-02-21T23:10:54.868462+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_clean_scrub_stamp":"2024-02-21T23:10:51.827439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:06:07.785035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:55.599384+0000","last_change":"2024-02-21T23:10:53.911117+0000","last_active":"2024-02-21T23:10:55.599384+0000","last_peered":"2024-02-21T23:10:55.599384+0000","last_clean":"2024-02-21T23:10:55.599384+0000","last_became_active":"2024-02-21T23:10:53.910822+0000","last_became_peered":"2024-02-21T23:10:53.910822+0000","last_unstale":"2024-02-21T23:10:55.599384+0000","last_undegraded":"2024-02-21T23:10:55.599384+0000","last_fullsized":"2024-02-21T23:10:55.599384+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:07:06.999165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1a","version":"173'1","reported_seq":13,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-21T23:10:53.927992+0000","last_change":"2024-02-21T23:10:52.887161+0000","last_active":"2024-02-21T23:10:53.927992+0000","last_peered":"2024-02-21T23:10:53.927992+0000","last_clean":"2024-02-21T23:10:53.927992+0000","last_became_active":"2024-02-21T23:10:52.885853+0000","last_became_peered":"2024-02-21T23:10:52.885853+0000","last_unstale":"2024-02-21T23:10:53.927992+0000","last_undegraded":"2024-02-21T23:10:53.927992+0000","last_fullsized":"2024-02-21T23:10:53.927992+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_clean_scrub_stamp":"2024-02-21T23:10:51.827439+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:07:05.983937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":458,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-21T23:10:53.894890+0000","last_change":"2024-02-21T23:02:32.289335+0000","last_active":"2024-02-21T23:10:53.894890+0000","last_peered":"2024-02-21T23:10:53.894890+0000","last_clean":"2024-02-21T23:10:53.894890+0000","last_became_active":"2024-02-21T23:02:32.286913+0000","last_became_peered":"2024-02-21T23:02:32.286913+0000","last_unstale":"2024-02-21T23:10:53.894890+0000","last_undegraded":"2024-02-21T23:10:53.894890+0000","last_fullsized":"2024-02-21T23:10:53.894890+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_clean_scrub_stamp":"2024-02-21T23:02:31.172569+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:36:23.987374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":28,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":27,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574951,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33400,"kb_used_data":2216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338440,"statfs":{"total":96636764160,"available":96602562560,"internally_reserved":0,"allocated":2269184,"data_stored":1670252,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574952,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33344,"kb_used_data":2164,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338496,"statfs":{"total":96636764160,"available":96602619904,"internally_reserved":0,"allocated":2215936,"data_stored":1087088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574951,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33320,"kb_used_data":2144,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338520,"statfs":{"total":96636764160,"available":96602644480,"internally_reserved":0,"allocated":2195456,"data_stored":1078258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574952,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33336,"kb_used_data":2152,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338504,"statfs":{"total":96636764160,"available":96602628096,"internally_reserved":0,"allocated":2203648,"data_stored":1085476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574952,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33344,"kb_used_data":2160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338496,"statfs":{"total":96636764160,"available":96602619904,"internally_reserved":0,"allocated":2211840,"data_stored":1085020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574953,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33336,"kb_used_data":2152,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338504,"statfs":{"total":96636764160,"available":96602628096,"internally_reserved":0,"allocated":2203648,"data_stored":1084984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574951,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33332,"kb_used_data":2148,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338508,"statfs":{"total":96636764160,"available":96602632192,"internally_reserved":0,"allocated":2199552,"data_stored":1084894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574951,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37548,"kb_used_data":2204,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334292,"statfs":{"total":96636764160,"available":96598315008,"internally_reserved":0,"allocated":2256896,"data_stored":1668660,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-21T23:10:58.469 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-21T23:10:58.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.741+0000 7f18c7e00640 1 Processor -- start 2024-02-21T23:10:58.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.741+0000 7f18c7e00640 1 -- start start 2024-02-21T23:10:58.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.743+0000 7f18c7e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f18c81524a0 0x7f18c80c7620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:58.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.743+0000 7f18c7e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f18c8155c20 0x7f18c815f860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:58.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.743+0000 7f18c7e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18c806ba00 con 0x7f18c80c45d0 2024-02-21T23:10:58.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.743+0000 7f18c7e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18c806f2d0 con 0x7f18c81524a0 2024-02-21T23:10:58.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.743+0000 7f18c7e00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18c8154ff0 con 0x7f18c8155c20 2024-02-21T23:10:58.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.743+0000 7f18c6a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f18c81524a0 0x7f18c80c7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:58.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.743+0000 7f18c6000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f18c80c45d0 0x7f18c8155510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55718/0 (socket says 172.21.15.28:55718) 2024-02-21T23:10:58.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.743+0000 7f18c6000640 1 -- 172.21.15.28:0/4000177779 learned_addr learned my addr 172.21.15.28:0/4000177779 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:10:58.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.743+0000 7f18c7400640 1 --2- 172.21.15.28:0/4000177779 >> v2:172.21.15.81:3300/0 conn(0x7f18c8155c20 0x7f18c815f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:58.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.744+0000 7f18c6a00640 1 -- 172.21.15.28:0/4000177779 >> v2:172.21.15.81:3300/0 conn(0x7f18c8155c20 msgr2=0x7f18c815f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:58.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.744+0000 7f18c6a00640 1 --2- 172.21.15.28:0/4000177779 >> v2:172.21.15.81:3300/0 conn(0x7f18c8155c20 0x7f18c815f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:58.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.744+0000 7f18c6a00640 1 -- 172.21.15.28:0/4000177779 >> v1:172.21.15.28:6789/0 conn(0x7f18c80c45d0 legacy=0x7f18c8155510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:58.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.744+0000 7f18c6a00640 1 -- 172.21.15.28:0/4000177779 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18c806aee0 con 0x7f18c81524a0 2024-02-21T23:10:58.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.744+0000 7f18c5600640 1 -- 172.21.15.28:0/4000177779 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 922618789 0 0) 0x7f18c806ba00 con 0x7f18c80c45d0 2024-02-21T23:10:58.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.745+0000 7f18c6a00640 1 --2- 172.21.15.28:0/4000177779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18c81524a0 0x7f18c80c7620 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7f18c806de70 tx=0x7f18b802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bbe5b1e2dac50a server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:58.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.745+0000 7f18c5600640 1 -- 172.21.15.28:0/4000177779 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18b8035020 con 0x7f18c81524a0 2024-02-21T23:10:58.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.745+0000 7f18c5600640 1 -- 172.21.15.28:0/4000177779 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18b8031830 con 0x7f18c81524a0 2024-02-21T23:10:58.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.745+0000 7f18c5600640 1 -- 172.21.15.28:0/4000177779 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18b8031c00 con 0x7f18c81524a0 2024-02-21T23:10:58.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.746+0000 7f18c7e00640 1 -- 172.21.15.28:0/4000177779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18c81524a0 msgr2=0x7f18c80c7620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:58.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.746+0000 7f18c7e00640 1 --2- 172.21.15.28:0/4000177779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18c81524a0 0x7f18c80c7620 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7f18c806de70 tx=0x7f18b802b8d0 comp rx=0 tx=0).stop 2024-02-21T23:10:58.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.746+0000 7f18c7e00640 1 -- 172.21.15.28:0/4000177779 shutdown_connections 2024-02-21T23:10:58.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.746+0000 7f18c7e00640 1 --2- 172.21.15.28:0/4000177779 >> v2:172.21.15.81:3300/0 conn(0x7f18c8155c20 0x7f18c815f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:58.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.746+0000 7f18c7e00640 1 --2- 172.21.15.28:0/4000177779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18c81524a0 0x7f18c80c7620 unknown :-1 s=CLOSED pgs=1928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:58.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.746+0000 7f18c7e00640 1 -- 172.21.15.28:0/4000177779 >> 172.21.15.28:0/4000177779 conn(0x7f18c8067050 msgr2=0x7f18c80a84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:58.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.746+0000 7f18c7e00640 1 -- 172.21.15.28:0/4000177779 shutdown_connections 2024-02-21T23:10:58.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.746+0000 7f18c7e00640 1 -- 172.21.15.28:0/4000177779 wait complete. 2024-02-21T23:10:58.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.748+0000 7f18c7e00640 1 Processor -- start 2024-02-21T23:10:58.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.748+0000 7f18c7e00640 1 -- start start 2024-02-21T23:10:58.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.749+0000 7f18c7e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f18c80c45d0 0x7f18c8120810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:58.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.749+0000 7f18c7e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f18c81524a0 0x7f18c8153560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:58.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.749+0000 7f18c7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18c81580d0 0x7f18c81edb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:58.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.749+0000 7f18c7e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18c80c4fa0 con 0x7f18c81580d0 2024-02-21T23:10:58.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.749+0000 7f18c7e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18c806f5b0 con 0x7f18c81524a0 2024-02-21T23:10:58.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.749+0000 7f18c7e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18c806b460 con 0x7f18c80c45d0 2024-02-21T23:10:58.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.749+0000 7f18c7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18c81580d0 0x7f18c81edb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:58.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.749+0000 7f18c7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18c81580d0 0x7f18c81edb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33050/0 (socket says 172.21.15.28:33050) 2024-02-21T23:10:58.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.749+0000 7f18c7400640 1 -- 172.21.15.28:0/1108716026 learned_addr learned my addr 172.21.15.28:0/1108716026 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:58.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.749+0000 7f18c6a00640 1 --2- 172.21.15.28:0/1108716026 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f18c80c45d0 0x7f18c8120810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:58.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.749+0000 7f18c6000640 1 --2- 172.21.15.28:0/1108716026 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f18c81524a0 0x7f18c8153560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:58.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.749+0000 7f18c7400640 1 -- 172.21.15.28:0/1108716026 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f18c80c45d0 msgr2=0x7f18c8120810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:58.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.750+0000 7f18c7400640 1 --2- 172.21.15.28:0/1108716026 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f18c80c45d0 0x7f18c8120810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:58.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.750+0000 7f18c7400640 1 -- 172.21.15.28:0/1108716026 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f18c81524a0 msgr2=0x7f18c8153560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:58.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.750+0000 7f18c7400640 1 --2- 172.21.15.28:0/1108716026 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f18c81524a0 0x7f18c8153560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:58.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.750+0000 7f18c7400640 1 -- 172.21.15.28:0/1108716026 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18c80aa850 con 0x7f18c81580d0 2024-02-21T23:10:58.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.750+0000 7f18c7400640 1 --2- 172.21.15.28:0/1108716026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18c81580d0 0x7f18c81edb10 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7f18c0007e60 tx=0x7f18c000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:58.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.750+0000 7f18bf400640 1 -- 172.21.15.28:0/1108716026 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18c0014420 con 0x7f18c81580d0 2024-02-21T23:10:58.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.750+0000 7f18bf400640 1 -- 172.21.15.28:0/1108716026 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18c0019070 con 0x7f18c81580d0 2024-02-21T23:10:58.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.750+0000 7f18bf400640 1 -- 172.21.15.28:0/1108716026 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18c0014820 con 0x7f18c81580d0 2024-02-21T23:10:58.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.750+0000 7f18c7e00640 1 -- 172.21.15.28:0/1108716026 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18c81ea700 con 0x7f18c81580d0 2024-02-21T23:10:58.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.751+0000 7f18c7e00640 1 -- 172.21.15.28:0/1108716026 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18c81eacc0 con 0x7f18c81580d0 2024-02-21T23:10:58.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.752+0000 7f18c7e00640 1 -- 172.21.15.28:0/1108716026 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18c81eaec0 con 0x7f18c81580d0 2024-02-21T23:10:58.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.753+0000 7f18bf400640 1 -- 172.21.15.28:0/1108716026 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f18c00149c0 con 0x7f18c81580d0 2024-02-21T23:10:58.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.754+0000 7f18bf400640 1 --2- 172.21.15.28:0/1108716026 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f18a806b500 0x7f18a806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:58.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.754+0000 7f18bf400640 1 -- 172.21.15.28:0/1108716026 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f18c008ab30 con 0x7f18c81580d0 2024-02-21T23:10:58.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.755+0000 7f18c6a00640 1 --2- 172.21.15.28:0/1108716026 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f18a806b500 0x7f18a806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:58.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.756+0000 7f18c6a00640 1 --2- 172.21.15.28:0/1108716026 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f18a806b500 0x7f18a806d9c0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f18b8036000 tx=0x7f18b8008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:58.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.758+0000 7f18bf400640 1 -- 172.21.15.28:0/1108716026 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f18c81eaec0 con 0x7f18c81580d0 2024-02-21T23:10:58.805 INFO:tasks.ceph.mds.c.smithi028.stderr:2024-02-21T23:10:58.817+0000 7fcf9f400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-21T23:10:58.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.958+0000 7f18c7e00640 1 -- 172.21.15.28:0/1108716026 --> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f18c80aab30 con 0x7f18a806b500 2024-02-21T23:10:58.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.976+0000 7f18bf400640 1 -- 172.21.15.28:0/1108716026 <== mgr.4101 v2:172.21.15.81:6832/1848538577 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252549 (secure 0 0 0) 0x7f18b8086aa0 con 0x7f18a806b500 2024-02-21T23:10:58.967 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:10:58.969 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-21T23:10:58.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.981+0000 7f18c7e00640 1 -- 172.21.15.28:0/1108716026 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f18a806b500 msgr2=0x7f18a806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:58.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.981+0000 7f18c7e00640 1 --2- 172.21.15.28:0/1108716026 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f18a806b500 0x7f18a806d9c0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f18b8036000 tx=0x7f18b8008030 comp rx=0 tx=0).stop 2024-02-21T23:10:58.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.981+0000 7f18c7e00640 1 -- 172.21.15.28:0/1108716026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18c81580d0 msgr2=0x7f18c81edb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:58.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.981+0000 7f18c7e00640 1 --2- 172.21.15.28:0/1108716026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18c81580d0 0x7f18c81edb10 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7f18c0007e60 tx=0x7f18c000e4b0 comp rx=0 tx=0).stop 2024-02-21T23:10:58.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.981+0000 7f18c7e00640 1 -- 172.21.15.28:0/1108716026 shutdown_connections 2024-02-21T23:10:58.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.981+0000 7f18c7e00640 1 --2- 172.21.15.28:0/1108716026 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f18a806b500 0x7f18a806d9c0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:58.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.981+0000 7f18c7e00640 1 --2- 172.21.15.28:0/1108716026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18c81580d0 0x7f18c81edb10 unknown :-1 s=CLOSED pgs=1929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:58.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.981+0000 7f18c7e00640 1 --2- 172.21.15.28:0/1108716026 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f18c81524a0 0x7f18c8153560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:58.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.981+0000 7f18c7e00640 1 --2- 172.21.15.28:0/1108716026 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f18c80c45d0 0x7f18c8120810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:58.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.981+0000 7f18c7e00640 1 -- 172.21.15.28:0/1108716026 >> 172.21.15.28:0/1108716026 conn(0x7f18c8067050 msgr2=0x7f18c8072b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:58.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.981+0000 7f18c7e00640 1 -- 172.21.15.28:0/1108716026 shutdown_connections 2024-02-21T23:10:58.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:58.981+0000 7f18c7e00640 1 -- 172.21.15.28:0/1108716026 wait complete. 2024-02-21T23:10:58.989 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":421,"stamp":"2024-02-21T23:10:58.843909+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":121,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":270960,"kb_used_data":17340,"kb_used_omap":12,"kb_used_meta":253491,"kb_avail":754703760,"statfs":{"total":773094113280,"available":772816650240,"internally_reserved":0,"allocated":17756160,"data_stored":9844632,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":259575345},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002150"},"pg_stats":[{"pgid":"28.3f","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:55.599550+0000","last_change":"2024-02-21T23:10:53.898099+0000","last_active":"2024-02-21T23:10:55.599550+0000","last_peered":"2024-02-21T23:10:55.599550+0000","last_clean":"2024-02-21T23:10:55.599550+0000","last_became_active":"2024-02-21T23:10:53.897696+0000","last_became_peered":"2024-02-21T23:10:53.897696+0000","last_unstale":"2024-02-21T23:10:55.599550+0000","last_undegraded":"2024-02-21T23:10:55.599550+0000","last_fullsized":"2024-02-21T23:10:55.599550+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:14:32.880334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3e","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.871049+0000","last_change":"2024-02-21T23:10:53.905951+0000","last_active":"2024-02-21T23:10:54.871049+0000","last_peered":"2024-02-21T23:10:54.871049+0000","last_clean":"2024-02-21T23:10:54.871049+0000","last_became_active":"2024-02-21T23:10:53.905655+0000","last_became_peered":"2024-02-21T23:10:53.905655+0000","last_unstale":"2024-02-21T23:10:54.871049+0000","last_undegraded":"2024-02-21T23:10:54.871049+0000","last_fullsized":"2024-02-21T23:10:54.871049+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:17:27.590244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.3d","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.929341+0000","last_change":"2024-02-21T23:10:53.896548+0000","last_active":"2024-02-21T23:10:54.929341+0000","last_peered":"2024-02-21T23:10:54.929341+0000","last_clean":"2024-02-21T23:10:54.929341+0000","last_became_active":"2024-02-21T23:10:53.894772+0000","last_became_peered":"2024-02-21T23:10:53.894772+0000","last_unstale":"2024-02-21T23:10:54.929341+0000","last_undegraded":"2024-02-21T23:10:54.929341+0000","last_fullsized":"2024-02-21T23:10:54.929341+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:57:09.153263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.3c","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-21T23:10:53.917820+0000","last_change":"2024-02-21T23:10:53.915191+0000","last_active":"2024-02-21T23:10:53.917820+0000","last_peered":"2024-02-21T23:10:53.917820+0000","last_clean":"2024-02-21T23:10:53.917820+0000","last_became_active":"2024-02-21T23:10:53.914070+0000","last_became_peered":"2024-02-21T23:10:53.914070+0000","last_unstale":"2024-02-21T23:10:53.917820+0000","last_undegraded":"2024-02-21T23:10:53.917820+0000","last_fullsized":"2024-02-21T23:10:53.917820+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:22:51.734335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.3b","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.869144+0000","last_change":"2024-02-21T23:10:53.892135+0000","last_active":"2024-02-21T23:10:54.869144+0000","last_peered":"2024-02-21T23:10:54.869144+0000","last_clean":"2024-02-21T23:10:54.869144+0000","last_became_active":"2024-02-21T23:10:53.889621+0000","last_became_peered":"2024-02-21T23:10:53.889621+0000","last_unstale":"2024-02-21T23:10:54.869144+0000","last_undegraded":"2024-02-21T23:10:54.869144+0000","last_fullsized":"2024-02-21T23:10:54.869144+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:37:41.533392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.3a","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-21T23:10:53.917888+0000","last_change":"2024-02-21T23:10:53.914762+0000","last_active":"2024-02-21T23:10:53.917888+0000","last_peered":"2024-02-21T23:10:53.917888+0000","last_clean":"2024-02-21T23:10:53.917888+0000","last_became_active":"2024-02-21T23:10:53.911360+0000","last_became_peered":"2024-02-21T23:10:53.911360+0000","last_unstale":"2024-02-21T23:10:53.917888+0000","last_undegraded":"2024-02-21T23:10:53.917888+0000","last_fullsized":"2024-02-21T23:10:53.917888+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:45:29.399147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.39","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.868523+0000","last_change":"2024-02-21T23:10:53.889749+0000","last_active":"2024-02-21T23:10:54.868523+0000","last_peered":"2024-02-21T23:10:54.868523+0000","last_clean":"2024-02-21T23:10:54.868523+0000","last_became_active":"2024-02-21T23:10:53.889165+0000","last_became_peered":"2024-02-21T23:10:53.889165+0000","last_unstale":"2024-02-21T23:10:54.868523+0000","last_undegraded":"2024-02-21T23:10:54.868523+0000","last_fullsized":"2024-02-21T23:10:54.868523+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:00:59.328401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.38","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.924902+0000","last_change":"2024-02-21T23:10:53.901315+0000","last_active":"2024-02-21T23:10:54.924902+0000","last_peered":"2024-02-21T23:10:54.924902+0000","last_clean":"2024-02-21T23:10:54.924902+0000","last_became_active":"2024-02-21T23:10:53.901036+0000","last_became_peered":"2024-02-21T23:10:53.901036+0000","last_unstale":"2024-02-21T23:10:54.924902+0000","last_undegraded":"2024-02-21T23:10:54.924902+0000","last_fullsized":"2024-02-21T23:10:54.924902+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:30:57.274763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.37","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-21T23:10:53.912549+0000","last_change":"2024-02-21T23:10:53.907101+0000","last_active":"2024-02-21T23:10:53.912549+0000","last_peered":"2024-02-21T23:10:53.912549+0000","last_clean":"2024-02-21T23:10:53.912549+0000","last_became_active":"2024-02-21T23:10:53.905915+0000","last_became_peered":"2024-02-21T23:10:53.905915+0000","last_unstale":"2024-02-21T23:10:53.912549+0000","last_undegraded":"2024-02-21T23:10:53.912549+0000","last_fullsized":"2024-02-21T23:10:53.912549+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:03:35.208914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.36","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.926078+0000","last_change":"2024-02-21T23:10:53.904481+0000","last_active":"2024-02-21T23:10:54.926078+0000","last_peered":"2024-02-21T23:10:54.926078+0000","last_clean":"2024-02-21T23:10:54.926078+0000","last_became_active":"2024-02-21T23:10:53.903914+0000","last_became_peered":"2024-02-21T23:10:53.903914+0000","last_unstale":"2024-02-21T23:10:54.926078+0000","last_undegraded":"2024-02-21T23:10:54.926078+0000","last_fullsized":"2024-02-21T23:10:54.926078+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:47:33.190260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.35","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.929309+0000","last_change":"2024-02-21T23:10:53.913289+0000","last_active":"2024-02-21T23:10:54.929309+0000","last_peered":"2024-02-21T23:10:54.929309+0000","last_clean":"2024-02-21T23:10:54.929309+0000","last_became_active":"2024-02-21T23:10:53.911721+0000","last_became_peered":"2024-02-21T23:10:53.911721+0000","last_unstale":"2024-02-21T23:10:54.929309+0000","last_undegraded":"2024-02-21T23:10:54.929309+0000","last_fullsized":"2024-02-21T23:10:54.929309+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:07:06.999165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.34","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.868745+0000","last_change":"2024-02-21T23:10:53.901430+0000","last_active":"2024-02-21T23:10:54.868745+0000","last_peered":"2024-02-21T23:10:54.868745+0000","last_clean":"2024-02-21T23:10:54.868745+0000","last_became_active":"2024-02-21T23:10:53.901127+0000","last_became_peered":"2024-02-21T23:10:53.901127+0000","last_unstale":"2024-02-21T23:10:54.868745+0000","last_undegraded":"2024-02-21T23:10:54.868745+0000","last_fullsized":"2024-02-21T23:10:54.868745+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:31:45.048178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.33","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-21T23:10:53.917559+0000","last_change":"2024-02-21T23:10:53.912389+0000","last_active":"2024-02-21T23:10:53.917559+0000","last_peered":"2024-02-21T23:10:53.917559+0000","last_clean":"2024-02-21T23:10:53.917559+0000","last_became_active":"2024-02-21T23:10:53.912025+0000","last_became_peered":"2024-02-21T23:10:53.912025+0000","last_unstale":"2024-02-21T23:10:53.917559+0000","last_undegraded":"2024-02-21T23:10:53.917559+0000","last_fullsized":"2024-02-21T23:10:53.917559+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:03:41.983896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.32","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.925107+0000","last_change":"2024-02-21T23:10:53.908511+0000","last_active":"2024-02-21T23:10:54.925107+0000","last_peered":"2024-02-21T23:10:54.925107+0000","last_clean":"2024-02-21T23:10:54.925107+0000","last_became_active":"2024-02-21T23:10:53.908186+0000","last_became_peered":"2024-02-21T23:10:53.908186+0000","last_unstale":"2024-02-21T23:10:54.925107+0000","last_undegraded":"2024-02-21T23:10:54.925107+0000","last_fullsized":"2024-02-21T23:10:54.925107+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:06:24.650495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.31","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.872305+0000","last_change":"2024-02-21T23:10:53.902086+0000","last_active":"2024-02-21T23:10:54.872305+0000","last_peered":"2024-02-21T23:10:54.872305+0000","last_clean":"2024-02-21T23:10:54.872305+0000","last_became_active":"2024-02-21T23:10:53.901797+0000","last_became_peered":"2024-02-21T23:10:53.901797+0000","last_unstale":"2024-02-21T23:10:54.872305+0000","last_undegraded":"2024-02-21T23:10:54.872305+0000","last_fullsized":"2024-02-21T23:10:54.872305+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:04:38.458011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.30","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.929047+0000","last_change":"2024-02-21T23:10:53.899134+0000","last_active":"2024-02-21T23:10:54.929047+0000","last_peered":"2024-02-21T23:10:54.929047+0000","last_clean":"2024-02-21T23:10:54.929047+0000","last_became_active":"2024-02-21T23:10:53.898401+0000","last_became_peered":"2024-02-21T23:10:53.898401+0000","last_unstale":"2024-02-21T23:10:54.929047+0000","last_undegraded":"2024-02-21T23:10:54.929047+0000","last_fullsized":"2024-02-21T23:10:54.929047+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:02:32.099002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.2f","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-21T23:10:53.913458+0000","last_change":"2024-02-21T23:10:53.909987+0000","last_active":"2024-02-21T23:10:53.913458+0000","last_peered":"2024-02-21T23:10:53.913458+0000","last_clean":"2024-02-21T23:10:53.913458+0000","last_became_active":"2024-02-21T23:10:53.909694+0000","last_became_peered":"2024-02-21T23:10:53.909694+0000","last_unstale":"2024-02-21T23:10:53.913458+0000","last_undegraded":"2024-02-21T23:10:53.913458+0000","last_fullsized":"2024-02-21T23:10:53.913458+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:27:30.670285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.2e","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-21T23:10:53.912301+0000","last_change":"2024-02-21T23:10:53.896968+0000","last_active":"2024-02-21T23:10:53.912301+0000","last_peered":"2024-02-21T23:10:53.912301+0000","last_clean":"2024-02-21T23:10:53.912301+0000","last_became_active":"2024-02-21T23:10:53.896636+0000","last_became_peered":"2024-02-21T23:10:53.896636+0000","last_unstale":"2024-02-21T23:10:53.912301+0000","last_undegraded":"2024-02-21T23:10:53.912301+0000","last_fullsized":"2024-02-21T23:10:53.912301+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:15:19.170148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.2d","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.926216+0000","last_change":"2024-02-21T23:10:53.915122+0000","last_active":"2024-02-21T23:10:54.926216+0000","last_peered":"2024-02-21T23:10:54.926216+0000","last_clean":"2024-02-21T23:10:54.926216+0000","last_became_active":"2024-02-21T23:10:53.914742+0000","last_became_peered":"2024-02-21T23:10:53.914742+0000","last_unstale":"2024-02-21T23:10:54.926216+0000","last_undegraded":"2024-02-21T23:10:54.926216+0000","last_fullsized":"2024-02-21T23:10:54.926216+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:12:07.683138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2c","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.926570+0000","last_change":"2024-02-21T23:10:53.915192+0000","last_active":"2024-02-21T23:10:54.926570+0000","last_peered":"2024-02-21T23:10:54.926570+0000","last_clean":"2024-02-21T23:10:54.926570+0000","last_became_active":"2024-02-21T23:10:53.909666+0000","last_became_peered":"2024-02-21T23:10:53.909666+0000","last_unstale":"2024-02-21T23:10:54.926570+0000","last_undegraded":"2024-02-21T23:10:54.926570+0000","last_fullsized":"2024-02-21T23:10:54.926570+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:02:14.168850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2b","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.925130+0000","last_change":"2024-02-21T23:10:53.906254+0000","last_active":"2024-02-21T23:10:54.925130+0000","last_peered":"2024-02-21T23:10:54.925130+0000","last_clean":"2024-02-21T23:10:54.925130+0000","last_became_active":"2024-02-21T23:10:53.905868+0000","last_became_peered":"2024-02-21T23:10:53.905868+0000","last_unstale":"2024-02-21T23:10:54.925130+0000","last_undegraded":"2024-02-21T23:10:54.925130+0000","last_fullsized":"2024-02-21T23:10:54.925130+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:18:25.131127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2a","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.926000+0000","last_change":"2024-02-21T23:10:53.909219+0000","last_active":"2024-02-21T23:10:54.926000+0000","last_peered":"2024-02-21T23:10:54.926000+0000","last_clean":"2024-02-21T23:10:54.926000+0000","last_became_active":"2024-02-21T23:10:53.908915+0000","last_became_peered":"2024-02-21T23:10:53.908915+0000","last_unstale":"2024-02-21T23:10:54.926000+0000","last_undegraded":"2024-02-21T23:10:54.926000+0000","last_fullsized":"2024-02-21T23:10:54.926000+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:29:03.024967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.29","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.870691+0000","last_change":"2024-02-21T23:10:53.910615+0000","last_active":"2024-02-21T23:10:54.870691+0000","last_peered":"2024-02-21T23:10:54.870691+0000","last_clean":"2024-02-21T23:10:54.870691+0000","last_became_active":"2024-02-21T23:10:53.908789+0000","last_became_peered":"2024-02-21T23:10:53.908789+0000","last_unstale":"2024-02-21T23:10:54.870691+0000","last_undegraded":"2024-02-21T23:10:54.870691+0000","last_fullsized":"2024-02-21T23:10:54.870691+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:23:47.446321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.28","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.868396+0000","last_change":"2024-02-21T23:10:53.886666+0000","last_active":"2024-02-21T23:10:54.868396+0000","last_peered":"2024-02-21T23:10:54.868396+0000","last_clean":"2024-02-21T23:10:54.868396+0000","last_became_active":"2024-02-21T23:10:53.886095+0000","last_became_peered":"2024-02-21T23:10:53.886095+0000","last_unstale":"2024-02-21T23:10:54.868396+0000","last_undegraded":"2024-02-21T23:10:54.868396+0000","last_fullsized":"2024-02-21T23:10:54.868396+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:34:59.396395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.27","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-21T23:10:53.912473+0000","last_change":"2024-02-21T23:10:53.905919+0000","last_active":"2024-02-21T23:10:53.912473+0000","last_peered":"2024-02-21T23:10:53.912473+0000","last_clean":"2024-02-21T23:10:53.912473+0000","last_became_active":"2024-02-21T23:10:53.905404+0000","last_became_peered":"2024-02-21T23:10:53.905404+0000","last_unstale":"2024-02-21T23:10:53.912473+0000","last_undegraded":"2024-02-21T23:10:53.912473+0000","last_fullsized":"2024-02-21T23:10:53.912473+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:51:37.937318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.26","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:55.599545+0000","last_change":"2024-02-21T23:10:53.906183+0000","last_active":"2024-02-21T23:10:55.599545+0000","last_peered":"2024-02-21T23:10:55.599545+0000","last_clean":"2024-02-21T23:10:55.599545+0000","last_became_active":"2024-02-21T23:10:53.905887+0000","last_became_peered":"2024-02-21T23:10:53.905887+0000","last_unstale":"2024-02-21T23:10:55.599545+0000","last_undegraded":"2024-02-21T23:10:55.599545+0000","last_fullsized":"2024-02-21T23:10:55.599545+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:57:09.153263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.25","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.929483+0000","last_change":"2024-02-21T23:10:53.905545+0000","last_active":"2024-02-21T23:10:54.929483+0000","last_peered":"2024-02-21T23:10:54.929483+0000","last_clean":"2024-02-21T23:10:54.929483+0000","last_became_active":"2024-02-21T23:10:53.905234+0000","last_became_peered":"2024-02-21T23:10:53.905234+0000","last_unstale":"2024-02-21T23:10:54.929483+0000","last_undegraded":"2024-02-21T23:10:54.929483+0000","last_fullsized":"2024-02-21T23:10:54.929483+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:49:22.732546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.24","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.868899+0000","last_change":"2024-02-21T23:10:53.895871+0000","last_active":"2024-02-21T23:10:54.868899+0000","last_peered":"2024-02-21T23:10:54.868899+0000","last_clean":"2024-02-21T23:10:54.868899+0000","last_became_active":"2024-02-21T23:10:53.895230+0000","last_became_peered":"2024-02-21T23:10:53.895230+0000","last_unstale":"2024-02-21T23:10:54.868899+0000","last_undegraded":"2024-02-21T23:10:54.868899+0000","last_fullsized":"2024-02-21T23:10:54.868899+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:48:33.675885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.23","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-21T23:10:53.911987+0000","last_change":"2024-02-21T23:10:53.891940+0000","last_active":"2024-02-21T23:10:53.911987+0000","last_peered":"2024-02-21T23:10:53.911987+0000","last_clean":"2024-02-21T23:10:53.911987+0000","last_became_active":"2024-02-21T23:10:53.890248+0000","last_became_peered":"2024-02-21T23:10:53.890248+0000","last_unstale":"2024-02-21T23:10:53.911987+0000","last_undegraded":"2024-02-21T23:10:53.911987+0000","last_fullsized":"2024-02-21T23:10:53.911987+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:01:37.314686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.22","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.872395+0000","last_change":"2024-02-21T23:10:53.904573+0000","last_active":"2024-02-21T23:10:54.872395+0000","last_peered":"2024-02-21T23:10:54.872395+0000","last_clean":"2024-02-21T23:10:54.872395+0000","last_became_active":"2024-02-21T23:10:53.904225+0000","last_became_peered":"2024-02-21T23:10:53.904225+0000","last_unstale":"2024-02-21T23:10:54.872395+0000","last_undegraded":"2024-02-21T23:10:54.872395+0000","last_fullsized":"2024-02-21T23:10:54.872395+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:21:49.171138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.21","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-21T23:10:53.917784+0000","last_change":"2024-02-21T23:10:53.914745+0000","last_active":"2024-02-21T23:10:53.917784+0000","last_peered":"2024-02-21T23:10:53.917784+0000","last_clean":"2024-02-21T23:10:53.917784+0000","last_became_active":"2024-02-21T23:10:53.912720+0000","last_became_peered":"2024-02-21T23:10:53.912720+0000","last_unstale":"2024-02-21T23:10:53.917784+0000","last_undegraded":"2024-02-21T23:10:53.917784+0000","last_fullsized":"2024-02-21T23:10:53.917784+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:38:50.515853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.20","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-21T23:10:53.917580+0000","last_change":"2024-02-21T23:10:53.912834+0000","last_active":"2024-02-21T23:10:53.917580+0000","last_peered":"2024-02-21T23:10:53.917580+0000","last_clean":"2024-02-21T23:10:53.917580+0000","last_became_active":"2024-02-21T23:10:53.912462+0000","last_became_peered":"2024-02-21T23:10:53.912462+0000","last_unstale":"2024-02-21T23:10:53.917580+0000","last_undegraded":"2024-02-21T23:10:53.917580+0000","last_fullsized":"2024-02-21T23:10:53.917580+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:25:24.967052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.870437+0000","last_change":"2024-02-21T23:10:53.904551+0000","last_active":"2024-02-21T23:10:54.870437+0000","last_peered":"2024-02-21T23:10:54.870437+0000","last_clean":"2024-02-21T23:10:54.870437+0000","last_became_active":"2024-02-21T23:10:53.904267+0000","last_became_peered":"2024-02-21T23:10:53.904267+0000","last_unstale":"2024-02-21T23:10:54.870437+0000","last_undegraded":"2024-02-21T23:10:54.870437+0000","last_fullsized":"2024-02-21T23:10:54.870437+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:48:12.418011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.1f","version":"173'4","reported_seq":18,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.873133+0000","last_change":"2024-02-21T23:10:52.900397+0000","last_active":"2024-02-21T23:10:54.873133+0000","last_peered":"2024-02-21T23:10:54.873133+0000","last_clean":"2024-02-21T23:10:54.873133+0000","last_became_active":"2024-02-21T23:10:52.899782+0000","last_became_peered":"2024-02-21T23:10:52.899782+0000","last_unstale":"2024-02-21T23:10:54.873133+0000","last_undegraded":"2024-02-21T23:10:54.873133+0000","last_fullsized":"2024-02-21T23:10:54.873133+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_clean_scrub_stamp":"2024-02-21T23:10:51.827439+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:12:06.667909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:55.598768+0000","last_change":"2024-02-21T23:10:53.886408+0000","last_active":"2024-02-21T23:10:55.598768+0000","last_peered":"2024-02-21T23:10:55.598768+0000","last_clean":"2024-02-21T23:10:55.598768+0000","last_became_active":"2024-02-21T23:10:53.885669+0000","last_became_peered":"2024-02-21T23:10:53.885669+0000","last_unstale":"2024-02-21T23:10:55.598768+0000","last_undegraded":"2024-02-21T23:10:55.598768+0000","last_fullsized":"2024-02-21T23:10:55.598768+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:36:37.264098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1e","version":"173'1","reported_seq":15,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:55.599642+0000","last_change":"2024-02-21T23:10:52.895166+0000","last_active":"2024-02-21T23:10:55.599642+0000","last_peered":"2024-02-21T23:10:55.599642+0000","last_clean":"2024-02-21T23:10:55.599642+0000","last_became_active":"2024-02-21T23:10:52.894688+0000","last_became_peered":"2024-02-21T23:10:52.894688+0000","last_unstale":"2024-02-21T23:10:55.599642+0000","last_undegraded":"2024-02-21T23:10:55.599642+0000","last_fullsized":"2024-02-21T23:10:55.599642+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_clean_scrub_stamp":"2024-02-21T23:10:51.827439+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:03:40.968667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.1a","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.926326+0000","last_change":"2024-02-21T23:10:53.899207+0000","last_active":"2024-02-21T23:10:54.926326+0000","last_peered":"2024-02-21T23:10:54.926326+0000","last_clean":"2024-02-21T23:10:54.926326+0000","last_became_active":"2024-02-21T23:10:53.898837+0000","last_became_peered":"2024-02-21T23:10:53.898837+0000","last_unstale":"2024-02-21T23:10:54.926326+0000","last_undegraded":"2024-02-21T23:10:54.926326+0000","last_fullsized":"2024-02-21T23:10:54.926326+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:49:24.548041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1d","version":"173'2","reported_seq":16,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.870902+0000","last_change":"2024-02-21T23:10:52.881910+0000","last_active":"2024-02-21T23:10:54.870902+0000","last_peered":"2024-02-21T23:10:54.870902+0000","last_clean":"2024-02-21T23:10:54.870902+0000","last_became_active":"2024-02-21T23:10:52.881576+0000","last_became_peered":"2024-02-21T23:10:52.881576+0000","last_unstale":"2024-02-21T23:10:54.870902+0000","last_undegraded":"2024-02-21T23:10:54.870902+0000","last_fullsized":"2024-02-21T23:10:54.870902+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_clean_scrub_stamp":"2024-02-21T23:10:51.827439+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:03:34.193685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1b","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.869463+0000","last_change":"2024-02-21T23:10:53.900136+0000","last_active":"2024-02-21T23:10:54.869463+0000","last_peered":"2024-02-21T23:10:54.869463+0000","last_clean":"2024-02-21T23:10:54.869463+0000","last_became_active":"2024-02-21T23:10:53.899784+0000","last_became_peered":"2024-02-21T23:10:53.899784+0000","last_unstale":"2024-02-21T23:10:54.869463+0000","last_undegraded":"2024-02-21T23:10:54.869463+0000","last_fullsized":"2024-02-21T23:10:54.869463+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:40:51.691428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1c","version":"173'2","reported_seq":14,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-21T23:10:53.925070+0000","last_change":"2024-02-21T23:10:52.888364+0000","last_active":"2024-02-21T23:10:53.925070+0000","last_peered":"2024-02-21T23:10:53.925070+0000","last_clean":"2024-02-21T23:10:53.925070+0000","last_became_active":"2024-02-21T23:10:52.887973+0000","last_became_peered":"2024-02-21T23:10:52.887973+0000","last_unstale":"2024-02-21T23:10:53.925070+0000","last_undegraded":"2024-02-21T23:10:53.925070+0000","last_fullsized":"2024-02-21T23:10:53.925070+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_clean_scrub_stamp":"2024-02-21T23:10:51.827439+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:14:31.865105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1c","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.873335+0000","last_change":"2024-02-21T23:10:53.900084+0000","last_active":"2024-02-21T23:10:54.873335+0000","last_peered":"2024-02-21T23:10:54.873335+0000","last_clean":"2024-02-21T23:10:54.873335+0000","last_became_active":"2024-02-21T23:10:53.898640+0000","last_became_peered":"2024-02-21T23:10:53.898640+0000","last_unstale":"2024-02-21T23:10:54.873335+0000","last_undegraded":"2024-02-21T23:10:54.873335+0000","last_fullsized":"2024-02-21T23:10:54.873335+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:59:46.467802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.1b","version":"173'1","reported_seq":13,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-21T23:10:53.924672+0000","last_change":"2024-02-21T23:10:52.897974+0000","last_active":"2024-02-21T23:10:53.924672+0000","last_peered":"2024-02-21T23:10:53.924672+0000","last_clean":"2024-02-21T23:10:53.924672+0000","last_became_active":"2024-02-21T23:10:52.896320+0000","last_became_peered":"2024-02-21T23:10:52.896320+0000","last_unstale":"2024-02-21T23:10:53.924672+0000","last_undegraded":"2024-02-21T23:10:53.924672+0000","last_fullsized":"2024-02-21T23:10:53.924672+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_clean_scrub_stamp":"2024-02-21T23:10:51.827439+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:16:28.625769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1e","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.872856+0000","last_change":"2024-02-21T23:10:53.895418+0000","last_active":"2024-02-21T23:10:54.872856+0000","last_peered":"2024-02-21T23:10:54.872856+0000","last_clean":"2024-02-21T23:10:54.872856+0000","last_became_active":"2024-02-21T23:10:53.894976+0000","last_became_peered":"2024-02-21T23:10:53.894976+0000","last_unstale":"2024-02-21T23:10:54.872856+0000","last_undegraded":"2024-02-21T23:10:54.872856+0000","last_fullsized":"2024-02-21T23:10:54.872856+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:15:26.533905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":12,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-21T23:10:53.898663+0000","last_change":"2024-02-21T23:10:52.889108+0000","last_active":"2024-02-21T23:10:53.898663+0000","last_peered":"2024-02-21T23:10:53.898663+0000","last_clean":"2024-02-21T23:10:53.898663+0000","last_became_active":"2024-02-21T23:10:52.888823+0000","last_became_peered":"2024-02-21T23:10:52.888823+0000","last_unstale":"2024-02-21T23:10:53.898663+0000","last_undegraded":"2024-02-21T23:10:53.898663+0000","last_fullsized":"2024-02-21T23:10:53.898663+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_clean_scrub_stamp":"2024-02-21T23:10:51.827439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:56:18.758854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1f","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-21T23:10:53.912450+0000","last_change":"2024-02-21T23:10:53.904904+0000","last_active":"2024-02-21T23:10:53.912450+0000","last_peered":"2024-02-21T23:10:53.912450+0000","last_clean":"2024-02-21T23:10:53.912450+0000","last_became_active":"2024-02-21T23:10:53.903873+0000","last_became_peered":"2024-02-21T23:10:53.903873+0000","last_unstale":"2024-02-21T23:10:53.912450+0000","last_undegraded":"2024-02-21T23:10:53.912450+0000","last_fullsized":"2024-02-21T23:10:53.912450+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:12:18.987945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":14,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.929025+0000","last_change":"2024-02-21T23:10:52.889283+0000","last_active":"2024-02-21T23:10:54.929025+0000","last_peered":"2024-02-21T23:10:54.929025+0000","last_clean":"2024-02-21T23:10:54.929025+0000","last_became_active":"2024-02-21T23:10:52.888932+0000","last_became_peered":"2024-02-21T23:10:52.888932+0000","last_unstale":"2024-02-21T23:10:54.929025+0000","last_undegraded":"2024-02-21T23:10:54.929025+0000","last_fullsized":"2024-02-21T23:10:54.929025+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_clean_scrub_stamp":"2024-02-21T23:10:51.827439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:25:23.951823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.10","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.872302+0000","last_change":"2024-02-21T23:10:53.915103+0000","last_active":"2024-02-21T23:10:54.872302+0000","last_peered":"2024-02-21T23:10:54.872302+0000","last_clean":"2024-02-21T23:10:54.872302+0000","last_became_active":"2024-02-21T23:10:53.913553+0000","last_became_peered":"2024-02-21T23:10:53.913553+0000","last_unstale":"2024-02-21T23:10:54.872302+0000","last_undegraded":"2024-02-21T23:10:54.872302+0000","last_fullsized":"2024-02-21T23:10:54.872302+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:23:07.424809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":14,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.868931+0000","last_change":"2024-02-21T23:10:52.899068+0000","last_active":"2024-02-21T23:10:54.868931+0000","last_peered":"2024-02-21T23:10:54.868931+0000","last_clean":"2024-02-21T23:10:54.868931+0000","last_became_active":"2024-02-21T23:10:52.898730+0000","last_became_peered":"2024-02-21T23:10:52.898730+0000","last_unstale":"2024-02-21T23:10:54.868931+0000","last_undegraded":"2024-02-21T23:10:54.868931+0000","last_fullsized":"2024-02-21T23:10:54.868931+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_clean_scrub_stamp":"2024-02-21T23:10:51.827439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:00:33.242390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.929103+0000","last_change":"2024-02-21T23:10:53.895655+0000","last_active":"2024-02-21T23:10:54.929103+0000","last_peered":"2024-02-21T23:10:54.929103+0000","last_clean":"2024-02-21T23:10:54.929103+0000","last_became_active":"2024-02-21T23:10:53.895254+0000","last_became_peered":"2024-02-21T23:10:53.895254+0000","last_unstale":"2024-02-21T23:10:54.929103+0000","last_undegraded":"2024-02-21T23:10:54.929103+0000","last_fullsized":"2024-02-21T23:10:54.929103+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:15:08.969451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":12,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-21T23:10:53.903517+0000","last_change":"2024-02-21T23:10:52.901737+0000","last_active":"2024-02-21T23:10:53.903517+0000","last_peered":"2024-02-21T23:10:53.903517+0000","last_clean":"2024-02-21T23:10:53.903517+0000","last_became_active":"2024-02-21T23:10:52.901431+0000","last_became_peered":"2024-02-21T23:10:52.901431+0000","last_unstale":"2024-02-21T23:10:53.903517+0000","last_undegraded":"2024-02-21T23:10:53.903517+0000","last_fullsized":"2024-02-21T23:10:53.903517+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_clean_scrub_stamp":"2024-02-21T23:10:51.827439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:26:03.509779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-21T23:10:53.912024+0000","last_change":"2024-02-21T23:10:53.896153+0000","last_active":"2024-02-21T23:10:53.912024+0000","last_peered":"2024-02-21T23:10:53.912024+0000","last_clean":"2024-02-21T23:10:53.912024+0000","last_became_active":"2024-02-21T23:10:53.892498+0000","last_became_peered":"2024-02-21T23:10:53.892498+0000","last_unstale":"2024-02-21T23:10:53.912024+0000","last_undegraded":"2024-02-21T23:10:53.912024+0000","last_fullsized":"2024-02-21T23:10:53.912024+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:49:22.732546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.15","version":"173'3","reported_seq":15,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-21T23:10:53.928552+0000","last_change":"2024-02-21T23:10:52.880063+0000","last_active":"2024-02-21T23:10:53.928552+0000","last_peered":"2024-02-21T23:10:53.928552+0000","last_clean":"2024-02-21T23:10:53.928552+0000","last_became_active":"2024-02-21T23:10:52.879756+0000","last_became_peered":"2024-02-21T23:10:52.879756+0000","last_unstale":"2024-02-21T23:10:53.928552+0000","last_undegraded":"2024-02-21T23:10:53.928552+0000","last_fullsized":"2024-02-21T23:10:53.928552+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_clean_scrub_stamp":"2024-02-21T23:10:51.827439+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:06:09.990674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.13","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.869282+0000","last_change":"2024-02-21T23:10:53.898474+0000","last_active":"2024-02-21T23:10:54.869282+0000","last_peered":"2024-02-21T23:10:54.869282+0000","last_clean":"2024-02-21T23:10:54.869282+0000","last_became_active":"2024-02-21T23:10:53.898135+0000","last_became_peered":"2024-02-21T23:10:53.898135+0000","last_unstale":"2024-02-21T23:10:54.869282+0000","last_undegraded":"2024-02-21T23:10:54.869282+0000","last_fullsized":"2024-02-21T23:10:54.869282+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:17:10.407396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.14","version":"173'6","reported_seq":18,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-21T23:10:53.929259+0000","last_change":"2024-02-21T23:10:52.903798+0000","last_active":"2024-02-21T23:10:53.929259+0000","last_peered":"2024-02-21T23:10:53.929259+0000","last_clean":"2024-02-21T23:10:53.929259+0000","last_became_active":"2024-02-21T23:10:52.903462+0000","last_became_peered":"2024-02-21T23:10:52.903462+0000","last_unstale":"2024-02-21T23:10:53.929259+0000","last_undegraded":"2024-02-21T23:10:53.929259+0000","last_fullsized":"2024-02-21T23:10:53.929259+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_clean_scrub_stamp":"2024-02-21T23:10:51.827439+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:52:00.130991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.14","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.870748+0000","last_change":"2024-02-21T23:10:53.897321+0000","last_active":"2024-02-21T23:10:54.870748+0000","last_peered":"2024-02-21T23:10:54.870748+0000","last_clean":"2024-02-21T23:10:54.870748+0000","last_became_active":"2024-02-21T23:10:53.896989+0000","last_became_peered":"2024-02-21T23:10:53.896989+0000","last_unstale":"2024-02-21T23:10:54.870748+0000","last_undegraded":"2024-02-21T23:10:54.870748+0000","last_fullsized":"2024-02-21T23:10:54.870748+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:47:32.514797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.13","version":"173'2","reported_seq":16,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.870621+0000","last_change":"2024-02-21T23:10:52.898947+0000","last_active":"2024-02-21T23:10:54.870621+0000","last_peered":"2024-02-21T23:10:54.870621+0000","last_clean":"2024-02-21T23:10:54.870621+0000","last_became_active":"2024-02-21T23:10:52.898348+0000","last_became_peered":"2024-02-21T23:10:52.898348+0000","last_unstale":"2024-02-21T23:10:54.870621+0000","last_undegraded":"2024-02-21T23:10:54.870621+0000","last_fullsized":"2024-02-21T23:10:54.870621+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_clean_scrub_stamp":"2024-02-21T23:10:51.827439+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:51:36.922089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.15","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.929499+0000","last_change":"2024-02-21T23:10:53.910740+0000","last_active":"2024-02-21T23:10:54.929499+0000","last_peered":"2024-02-21T23:10:54.929499+0000","last_clean":"2024-02-21T23:10:54.929499+0000","last_became_active":"2024-02-21T23:10:53.910423+0000","last_became_peered":"2024-02-21T23:10:53.910423+0000","last_unstale":"2024-02-21T23:10:54.929499+0000","last_undegraded":"2024-02-21T23:10:54.929499+0000","last_fullsized":"2024-02-21T23:10:54.929499+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:15:19.170148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":14,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.929135+0000","last_change":"2024-02-21T23:10:52.883726+0000","last_active":"2024-02-21T23:10:54.929135+0000","last_peered":"2024-02-21T23:10:54.929135+0000","last_clean":"2024-02-21T23:10:54.929135+0000","last_became_active":"2024-02-21T23:10:52.883288+0000","last_became_peered":"2024-02-21T23:10:52.883288+0000","last_unstale":"2024-02-21T23:10:54.929135+0000","last_undegraded":"2024-02-21T23:10:54.929135+0000","last_fullsized":"2024-02-21T23:10:54.929135+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_clean_scrub_stamp":"2024-02-21T23:10:51.827439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:27:18.911280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-21T23:10:53.912357+0000","last_change":"2024-02-21T23:10:53.900285+0000","last_active":"2024-02-21T23:10:53.912357+0000","last_peered":"2024-02-21T23:10:53.912357+0000","last_clean":"2024-02-21T23:10:53.912357+0000","last_became_active":"2024-02-21T23:10:53.899976+0000","last_became_peered":"2024-02-21T23:10:53.899976+0000","last_unstale":"2024-02-21T23:10:53.912357+0000","last_undegraded":"2024-02-21T23:10:53.912357+0000","last_fullsized":"2024-02-21T23:10:53.912357+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:09:04.785491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":12,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-21T23:10:53.889742+0000","last_change":"2024-02-21T23:10:52.894474+0000","last_active":"2024-02-21T23:10:53.889742+0000","last_peered":"2024-02-21T23:10:53.889742+0000","last_clean":"2024-02-21T23:10:53.889742+0000","last_became_active":"2024-02-21T23:10:52.894209+0000","last_became_peered":"2024-02-21T23:10:52.894209+0000","last_unstale":"2024-02-21T23:10:53.889742+0000","last_undegraded":"2024-02-21T23:10:53.889742+0000","last_fullsized":"2024-02-21T23:10:53.889742+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_clean_scrub_stamp":"2024-02-21T23:10:51.827439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:15:07.954222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-21T23:10:53.917416+0000","last_change":"2024-02-21T23:10:53.912060+0000","last_active":"2024-02-21T23:10:53.917416+0000","last_peered":"2024-02-21T23:10:53.917416+0000","last_clean":"2024-02-21T23:10:53.917416+0000","last_became_active":"2024-02-21T23:10:53.911032+0000","last_became_peered":"2024-02-21T23:10:53.911032+0000","last_unstale":"2024-02-21T23:10:53.917416+0000","last_undegraded":"2024-02-21T23:10:53.917416+0000","last_fullsized":"2024-02-21T23:10:53.917416+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:27:04.076278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.10","version":"173'8","reported_seq":22,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.868355+0000","last_change":"2024-02-21T23:10:52.891863+0000","last_active":"2024-02-21T23:10:54.868355+0000","last_peered":"2024-02-21T23:10:54.868355+0000","last_clean":"2024-02-21T23:10:54.868355+0000","last_became_active":"2024-02-21T23:10:52.891250+0000","last_became_peered":"2024-02-21T23:10:52.891250+0000","last_unstale":"2024-02-21T23:10:54.868355+0000","last_undegraded":"2024-02-21T23:10:54.868355+0000","last_fullsized":"2024-02-21T23:10:54.868355+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_clean_scrub_stamp":"2024-02-21T23:10:51.827439+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:08:22.900530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.924880+0000","last_change":"2024-02-21T23:10:53.905223+0000","last_active":"2024-02-21T23:10:54.924880+0000","last_peered":"2024-02-21T23:10:54.924880+0000","last_clean":"2024-02-21T23:10:54.924880+0000","last_became_active":"2024-02-21T23:10:53.904815+0000","last_became_peered":"2024-02-21T23:10:53.904815+0000","last_unstale":"2024-02-21T23:10:54.924880+0000","last_undegraded":"2024-02-21T23:10:54.924880+0000","last_fullsized":"2024-02-21T23:10:54.924880+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:46:01.374626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.f","version":"173'1","reported_seq":15,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:55.599570+0000","last_change":"2024-02-21T23:10:52.895206+0000","last_active":"2024-02-21T23:10:55.599570+0000","last_peered":"2024-02-21T23:10:55.599570+0000","last_clean":"2024-02-21T23:10:55.599570+0000","last_became_active":"2024-02-21T23:10:52.894487+0000","last_became_peered":"2024-02-21T23:10:52.894487+0000","last_unstale":"2024-02-21T23:10:55.599570+0000","last_undegraded":"2024-02-21T23:10:55.599570+0000","last_fullsized":"2024-02-21T23:10:55.599570+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_clean_scrub_stamp":"2024-02-21T23:10:51.827439+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:27:03.061049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:55.598894+0000","last_change":"2024-02-21T23:10:53.905613+0000","last_active":"2024-02-21T23:10:55.598894+0000","last_peered":"2024-02-21T23:10:55.598894+0000","last_clean":"2024-02-21T23:10:55.598894+0000","last_became_active":"2024-02-21T23:10:53.902571+0000","last_became_peered":"2024-02-21T23:10:53.902571+0000","last_unstale":"2024-02-21T23:10:55.598894+0000","last_undegraded":"2024-02-21T23:10:55.598894+0000","last_fullsized":"2024-02-21T23:10:55.598894+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:02:32.099002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.e","version":"173'2","reported_seq":16,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:55.599773+0000","last_change":"2024-02-21T23:10:52.897066+0000","last_active":"2024-02-21T23:10:55.599773+0000","last_peered":"2024-02-21T23:10:55.599773+0000","last_clean":"2024-02-21T23:10:55.599773+0000","last_became_active":"2024-02-21T23:10:52.896587+0000","last_became_peered":"2024-02-21T23:10:52.896587+0000","last_unstale":"2024-02-21T23:10:55.599773+0000","last_undegraded":"2024-02-21T23:10:55.599773+0000","last_fullsized":"2024-02-21T23:10:55.599773+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_clean_scrub_stamp":"2024-02-21T23:10:51.827439+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:25:23.951823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:55.599346+0000","last_change":"2024-02-21T23:10:53.906960+0000","last_active":"2024-02-21T23:10:55.599346+0000","last_peered":"2024-02-21T23:10:55.599346+0000","last_clean":"2024-02-21T23:10:55.599346+0000","last_became_active":"2024-02-21T23:10:53.904761+0000","last_became_peered":"2024-02-21T23:10:53.904761+0000","last_unstale":"2024-02-21T23:10:55.599346+0000","last_undegraded":"2024-02-21T23:10:55.599346+0000","last_fullsized":"2024-02-21T23:10:55.599346+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:05:47.827081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.d","version":"173'1","reported_seq":15,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.870939+0000","last_change":"2024-02-21T23:10:52.892519+0000","last_active":"2024-02-21T23:10:54.870939+0000","last_peered":"2024-02-21T23:10:54.870939+0000","last_clean":"2024-02-21T23:10:54.870939+0000","last_became_active":"2024-02-21T23:10:52.892211+0000","last_became_peered":"2024-02-21T23:10:52.892211+0000","last_unstale":"2024-02-21T23:10:54.870939+0000","last_undegraded":"2024-02-21T23:10:54.870939+0000","last_fullsized":"2024-02-21T23:10:54.870939+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_clean_scrub_stamp":"2024-02-21T23:10:51.827439+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:27:29.655056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.b","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.925325+0000","last_change":"2024-02-21T23:10:53.905439+0000","last_active":"2024-02-21T23:10:54.925325+0000","last_peered":"2024-02-21T23:10:54.925325+0000","last_clean":"2024-02-21T23:10:54.925325+0000","last_became_active":"2024-02-21T23:10:53.905045+0000","last_became_peered":"2024-02-21T23:10:53.905045+0000","last_unstale":"2024-02-21T23:10:54.925325+0000","last_undegraded":"2024-02-21T23:10:54.925325+0000","last_fullsized":"2024-02-21T23:10:54.925325+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:25:20.360028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":14,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.868991+0000","last_change":"2024-02-21T23:10:52.898907+0000","last_active":"2024-02-21T23:10:54.868991+0000","last_peered":"2024-02-21T23:10:54.868991+0000","last_clean":"2024-02-21T23:10:54.868991+0000","last_became_active":"2024-02-21T23:10:52.898548+0000","last_became_peered":"2024-02-21T23:10:52.898548+0000","last_unstale":"2024-02-21T23:10:54.868991+0000","last_undegraded":"2024-02-21T23:10:54.868991+0000","last_fullsized":"2024-02-21T23:10:54.868991+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_clean_scrub_stamp":"2024-02-21T23:10:51.827439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:34:05.957909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.870829+0000","last_change":"2024-02-21T23:10:53.910083+0000","last_active":"2024-02-21T23:10:54.870829+0000","last_peered":"2024-02-21T23:10:54.870829+0000","last_clean":"2024-02-21T23:10:54.870829+0000","last_became_active":"2024-02-21T23:10:53.908966+0000","last_became_peered":"2024-02-21T23:10:53.908966+0000","last_unstale":"2024-02-21T23:10:54.870829+0000","last_undegraded":"2024-02-21T23:10:54.870829+0000","last_fullsized":"2024-02-21T23:10:54.870829+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:59:23.327887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.b","version":"173'2","reported_seq":16,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.869384+0000","last_change":"2024-02-21T23:10:52.883528+0000","last_active":"2024-02-21T23:10:54.869384+0000","last_peered":"2024-02-21T23:10:54.869384+0000","last_clean":"2024-02-21T23:10:54.869384+0000","last_became_active":"2024-02-21T23:10:52.883211+0000","last_became_peered":"2024-02-21T23:10:52.883211+0000","last_unstale":"2024-02-21T23:10:54.869384+0000","last_undegraded":"2024-02-21T23:10:54.869384+0000","last_fullsized":"2024-02-21T23:10:54.869384+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_clean_scrub_stamp":"2024-02-21T23:10:51.827439+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:43:15.583941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.d","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-21T23:10:53.917296+0000","last_change":"2024-02-21T23:10:53.911159+0000","last_active":"2024-02-21T23:10:53.917296+0000","last_peered":"2024-02-21T23:10:53.917296+0000","last_clean":"2024-02-21T23:10:53.917296+0000","last_became_active":"2024-02-21T23:10:53.909911+0000","last_became_peered":"2024-02-21T23:10:53.909911+0000","last_unstale":"2024-02-21T23:10:53.917296+0000","last_undegraded":"2024-02-21T23:10:53.917296+0000","last_fullsized":"2024-02-21T23:10:53.917296+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:28:34.665447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":14,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.870442+0000","last_change":"2024-02-21T23:10:52.882518+0000","last_active":"2024-02-21T23:10:54.870442+0000","last_peered":"2024-02-21T23:10:54.870442+0000","last_clean":"2024-02-21T23:10:54.870442+0000","last_became_active":"2024-02-21T23:10:52.882195+0000","last_became_peered":"2024-02-21T23:10:52.882195+0000","last_unstale":"2024-02-21T23:10:54.870442+0000","last_undegraded":"2024-02-21T23:10:54.870442+0000","last_fullsized":"2024-02-21T23:10:54.870442+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_clean_scrub_stamp":"2024-02-21T23:10:51.827439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:09:18.744867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.e","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-21T23:10:53.912377+0000","last_change":"2024-02-21T23:10:53.904097+0000","last_active":"2024-02-21T23:10:53.912377+0000","last_peered":"2024-02-21T23:10:53.912377+0000","last_clean":"2024-02-21T23:10:53.912377+0000","last_became_active":"2024-02-21T23:10:53.903781+0000","last_became_peered":"2024-02-21T23:10:53.903781+0000","last_unstale":"2024-02-21T23:10:53.912377+0000","last_undegraded":"2024-02-21T23:10:53.912377+0000","last_fullsized":"2024-02-21T23:10:53.912377+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:57:58.410605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":14,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.872475+0000","last_change":"2024-02-21T23:10:52.897082+0000","last_active":"2024-02-21T23:10:54.872475+0000","last_peered":"2024-02-21T23:10:54.872475+0000","last_clean":"2024-02-21T23:10:54.872475+0000","last_became_active":"2024-02-21T23:10:52.896761+0000","last_became_peered":"2024-02-21T23:10:52.896761+0000","last_unstale":"2024-02-21T23:10:54.872475+0000","last_undegraded":"2024-02-21T23:10:54.872475+0000","last_fullsized":"2024-02-21T23:10:54.872475+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_clean_scrub_stamp":"2024-02-21T23:10:51.827439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:49:23.532812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.f","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.926076+0000","last_change":"2024-02-21T23:10:53.904953+0000","last_active":"2024-02-21T23:10:54.926076+0000","last_peered":"2024-02-21T23:10:54.926076+0000","last_clean":"2024-02-21T23:10:54.926076+0000","last_became_active":"2024-02-21T23:10:53.904060+0000","last_became_peered":"2024-02-21T23:10:53.904060+0000","last_unstale":"2024-02-21T23:10:54.926076+0000","last_undegraded":"2024-02-21T23:10:54.926076+0000","last_fullsized":"2024-02-21T23:10:54.926076+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:51:21.189125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":14,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:55.598555+0000","last_change":"2024-02-21T23:10:52.900386+0000","last_active":"2024-02-21T23:10:55.598555+0000","last_peered":"2024-02-21T23:10:55.598555+0000","last_clean":"2024-02-21T23:10:55.598555+0000","last_became_active":"2024-02-21T23:10:52.899985+0000","last_became_peered":"2024-02-21T23:10:52.899985+0000","last_unstale":"2024-02-21T23:10:55.598555+0000","last_undegraded":"2024-02-21T23:10:55.598555+0000","last_fullsized":"2024-02-21T23:10:55.598555+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_clean_scrub_stamp":"2024-02-21T23:10:51.827439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:26:03.509779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.925331+0000","last_change":"2024-02-21T23:10:53.904892+0000","last_active":"2024-02-21T23:10:54.925331+0000","last_peered":"2024-02-21T23:10:54.925331+0000","last_clean":"2024-02-21T23:10:54.925331+0000","last_became_active":"2024-02-21T23:10:53.904403+0000","last_became_peered":"2024-02-21T23:10:53.904403+0000","last_unstale":"2024-02-21T23:10:54.925331+0000","last_undegraded":"2024-02-21T23:10:54.925331+0000","last_fullsized":"2024-02-21T23:10:54.925331+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:10:04.860878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.7","version":"173'2","reported_seq":14,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-21T23:10:53.927293+0000","last_change":"2024-02-21T23:10:52.900882+0000","last_active":"2024-02-21T23:10:53.927293+0000","last_peered":"2024-02-21T23:10:53.927293+0000","last_clean":"2024-02-21T23:10:53.927293+0000","last_became_active":"2024-02-21T23:10:52.900538+0000","last_became_peered":"2024-02-21T23:10:52.900538+0000","last_unstale":"2024-02-21T23:10:53.927293+0000","last_undegraded":"2024-02-21T23:10:53.927293+0000","last_fullsized":"2024-02-21T23:10:53.927293+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_clean_scrub_stamp":"2024-02-21T23:10:51.827439+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:31:09.669165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.929056+0000","last_change":"2024-02-21T23:10:53.905351+0000","last_active":"2024-02-21T23:10:54.929056+0000","last_peered":"2024-02-21T23:10:54.929056+0000","last_clean":"2024-02-21T23:10:54.929056+0000","last_became_active":"2024-02-21T23:10:53.905049+0000","last_became_peered":"2024-02-21T23:10:53.905049+0000","last_unstale":"2024-02-21T23:10:54.929056+0000","last_undegraded":"2024-02-21T23:10:54.929056+0000","last_fullsized":"2024-02-21T23:10:54.929056+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:05:47.827081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.6","version":"174'1","reported_seq":15,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.929149+0000","last_change":"2024-02-21T23:10:52.889360+0000","last_active":"2024-02-21T23:10:54.929149+0000","last_peered":"2024-02-21T23:10:54.929149+0000","last_clean":"2024-02-21T23:10:54.929149+0000","last_became_active":"2024-02-21T23:10:52.889022+0000","last_became_peered":"2024-02-21T23:10:52.889022+0000","last_unstale":"2024-02-21T23:10:54.929149+0000","last_undegraded":"2024-02-21T23:10:54.929149+0000","last_fullsized":"2024-02-21T23:10:54.929149+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_clean_scrub_stamp":"2024-02-21T23:10:51.827439+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:03:10.936536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-21T23:10:53.912129+0000","last_change":"2024-02-21T23:10:53.897689+0000","last_active":"2024-02-21T23:10:53.912129+0000","last_peered":"2024-02-21T23:10:53.912129+0000","last_clean":"2024-02-21T23:10:53.912129+0000","last_became_active":"2024-02-21T23:10:53.897304+0000","last_became_peered":"2024-02-21T23:10:53.897304+0000","last_unstale":"2024-02-21T23:10:53.912129+0000","last_undegraded":"2024-02-21T23:10:53.912129+0000","last_fullsized":"2024-02-21T23:10:53.912129+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:20:14.355944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":14,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.872627+0000","last_change":"2024-02-21T23:10:52.889401+0000","last_active":"2024-02-21T23:10:54.872627+0000","last_peered":"2024-02-21T23:10:54.872627+0000","last_clean":"2024-02-21T23:10:54.872627+0000","last_became_active":"2024-02-21T23:10:52.889142+0000","last_became_peered":"2024-02-21T23:10:52.889142+0000","last_unstale":"2024-02-21T23:10:54.872627+0000","last_undegraded":"2024-02-21T23:10:54.872627+0000","last_fullsized":"2024-02-21T23:10:54.872627+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_clean_scrub_stamp":"2024-02-21T23:10:51.827439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:51:20.173897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:55.599384+0000","last_change":"2024-02-21T23:10:53.903959+0000","last_active":"2024-02-21T23:10:55.599384+0000","last_peered":"2024-02-21T23:10:55.599384+0000","last_clean":"2024-02-21T23:10:55.599384+0000","last_became_active":"2024-02-21T23:10:53.903001+0000","last_became_peered":"2024-02-21T23:10:53.903001+0000","last_unstale":"2024-02-21T23:10:55.599384+0000","last_undegraded":"2024-02-21T23:10:55.599384+0000","last_fullsized":"2024-02-21T23:10:55.599384+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:15:08.969451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":14,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:55.599104+0000","last_change":"2024-02-21T23:10:52.894606+0000","last_active":"2024-02-21T23:10:55.599104+0000","last_peered":"2024-02-21T23:10:55.599104+0000","last_clean":"2024-02-21T23:10:55.599104+0000","last_became_active":"2024-02-21T23:10:52.894230+0000","last_became_peered":"2024-02-21T23:10:52.894230+0000","last_unstale":"2024-02-21T23:10:55.599104+0000","last_undegraded":"2024-02-21T23:10:55.599104+0000","last_fullsized":"2024-02-21T23:10:55.599104+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_clean_scrub_stamp":"2024-02-21T23:10:51.827439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:16:28.625769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.926566+0000","last_change":"2024-02-21T23:10:53.900914+0000","last_active":"2024-02-21T23:10:54.926566+0000","last_peered":"2024-02-21T23:10:54.926566+0000","last_clean":"2024-02-21T23:10:54.926566+0000","last_became_active":"2024-02-21T23:10:53.899241+0000","last_became_peered":"2024-02-21T23:10:53.899241+0000","last_unstale":"2024-02-21T23:10:54.926566+0000","last_undegraded":"2024-02-21T23:10:54.926566+0000","last_fullsized":"2024-02-21T23:10:54.926566+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:15:27.742832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":14,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.925493+0000","last_change":"2024-02-21T23:10:52.892844+0000","last_active":"2024-02-21T23:10:54.925493+0000","last_peered":"2024-02-21T23:10:54.925493+0000","last_clean":"2024-02-21T23:10:54.925493+0000","last_became_active":"2024-02-21T23:10:52.890532+0000","last_became_peered":"2024-02-21T23:10:52.890532+0000","last_unstale":"2024-02-21T23:10:54.925493+0000","last_undegraded":"2024-02-21T23:10:54.925493+0000","last_fullsized":"2024-02-21T23:10:54.925493+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_clean_scrub_stamp":"2024-02-21T23:10:51.827439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:16:06.680298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-21T23:10:53.917656+0000","last_change":"2024-02-21T23:10:53.913555+0000","last_active":"2024-02-21T23:10:53.917656+0000","last_peered":"2024-02-21T23:10:53.917656+0000","last_clean":"2024-02-21T23:10:53.917656+0000","last_became_active":"2024-02-21T23:10:53.912695+0000","last_became_peered":"2024-02-21T23:10:53.912695+0000","last_unstale":"2024-02-21T23:10:53.917656+0000","last_undegraded":"2024-02-21T23:10:53.917656+0000","last_fullsized":"2024-02-21T23:10:53.917656+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:27:19.926509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":12,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-21T23:10:53.890206+0000","last_change":"2024-02-21T23:10:52.886664+0000","last_active":"2024-02-21T23:10:53.890206+0000","last_peered":"2024-02-21T23:10:53.890206+0000","last_clean":"2024-02-21T23:10:53.890206+0000","last_became_active":"2024-02-21T23:10:52.886240+0000","last_became_peered":"2024-02-21T23:10:52.886240+0000","last_unstale":"2024-02-21T23:10:53.890206+0000","last_undegraded":"2024-02-21T23:10:53.890206+0000","last_fullsized":"2024-02-21T23:10:53.890206+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_clean_scrub_stamp":"2024-02-21T23:10:51.827439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:57:08.138034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":9,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-21T23:10:53.918765+0000","last_change":"2024-02-21T23:10:53.917131+0000","last_active":"2024-02-21T23:10:53.918765+0000","last_peered":"2024-02-21T23:10:53.918765+0000","last_clean":"2024-02-21T23:10:53.918765+0000","last_became_active":"2024-02-21T23:10:53.916596+0000","last_became_peered":"2024-02-21T23:10:53.916596+0000","last_unstale":"2024-02-21T23:10:53.918765+0000","last_undegraded":"2024-02-21T23:10:53.918765+0000","last_fullsized":"2024-02-21T23:10:53.918765+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:46:22.804312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":14,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:55.599201+0000","last_change":"2024-02-21T23:10:52.903783+0000","last_active":"2024-02-21T23:10:55.599201+0000","last_peered":"2024-02-21T23:10:55.599201+0000","last_clean":"2024-02-21T23:10:55.599201+0000","last_became_active":"2024-02-21T23:10:52.903375+0000","last_became_peered":"2024-02-21T23:10:52.903375+0000","last_unstale":"2024-02-21T23:10:55.599201+0000","last_undegraded":"2024-02-21T23:10:55.599201+0000","last_fullsized":"2024-02-21T23:10:55.599201+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_clean_scrub_stamp":"2024-02-21T23:10:51.827439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:28:33.650218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.7","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.929422+0000","last_change":"2024-02-21T23:10:53.906613+0000","last_active":"2024-02-21T23:10:54.929422+0000","last_peered":"2024-02-21T23:10:54.929422+0000","last_clean":"2024-02-21T23:10:54.929422+0000","last_became_active":"2024-02-21T23:10:53.906303+0000","last_became_peered":"2024-02-21T23:10:53.906303+0000","last_unstale":"2024-02-21T23:10:54.929422+0000","last_undegraded":"2024-02-21T23:10:54.929422+0000","last_fullsized":"2024-02-21T23:10:54.929422+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:14:32.880334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":14,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:54.868462+0000","last_change":"2024-02-21T23:10:52.891131+0000","last_active":"2024-02-21T23:10:54.868462+0000","last_peered":"2024-02-21T23:10:54.868462+0000","last_clean":"2024-02-21T23:10:54.868462+0000","last_became_active":"2024-02-21T23:10:52.890739+0000","last_became_peered":"2024-02-21T23:10:52.890739+0000","last_unstale":"2024-02-21T23:10:54.868462+0000","last_undegraded":"2024-02-21T23:10:54.868462+0000","last_fullsized":"2024-02-21T23:10:54.868462+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_clean_scrub_stamp":"2024-02-21T23:10:51.827439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:06:07.785035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.1d","version":"0'0","reported_seq":12,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-02-21T23:10:55.599384+0000","last_change":"2024-02-21T23:10:53.911117+0000","last_active":"2024-02-21T23:10:55.599384+0000","last_peered":"2024-02-21T23:10:55.599384+0000","last_clean":"2024-02-21T23:10:55.599384+0000","last_became_active":"2024-02-21T23:10:53.910822+0000","last_became_peered":"2024-02-21T23:10:53.910822+0000","last_unstale":"2024-02-21T23:10:55.599384+0000","last_undegraded":"2024-02-21T23:10:55.599384+0000","last_fullsized":"2024-02-21T23:10:55.599384+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:52.842668+0000","last_clean_scrub_stamp":"2024-02-21T23:10:52.842668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:07:06.999165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1a","version":"173'1","reported_seq":13,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-21T23:10:53.927992+0000","last_change":"2024-02-21T23:10:52.887161+0000","last_active":"2024-02-21T23:10:53.927992+0000","last_peered":"2024-02-21T23:10:53.927992+0000","last_clean":"2024-02-21T23:10:53.927992+0000","last_became_active":"2024-02-21T23:10:52.885853+0000","last_became_peered":"2024-02-21T23:10:52.885853+0000","last_unstale":"2024-02-21T23:10:53.927992+0000","last_undegraded":"2024-02-21T23:10:53.927992+0000","last_fullsized":"2024-02-21T23:10:53.927992+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:10:51.827439+0000","last_clean_scrub_stamp":"2024-02-21T23:10:51.827439+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:07:05.983937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":458,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-02-21T23:10:53.894890+0000","last_change":"2024-02-21T23:02:32.289335+0000","last_active":"2024-02-21T23:10:53.894890+0000","last_peered":"2024-02-21T23:10:53.894890+0000","last_clean":"2024-02-21T23:10:53.894890+0000","last_became_active":"2024-02-21T23:02:32.286913+0000","last_became_peered":"2024-02-21T23:02:32.286913+0000","last_unstale":"2024-02-21T23:10:53.894890+0000","last_undegraded":"2024-02-21T23:10:53.894890+0000","last_fullsized":"2024-02-21T23:10:53.894890+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_clean_scrub_stamp":"2024-02-21T23:02:31.172569+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:36:23.987374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":28,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":27,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574951,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33400,"kb_used_data":2216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338440,"statfs":{"total":96636764160,"available":96602562560,"internally_reserved":0,"allocated":2269184,"data_stored":1670252,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574952,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33344,"kb_used_data":2164,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338496,"statfs":{"total":96636764160,"available":96602619904,"internally_reserved":0,"allocated":2215936,"data_stored":1087088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574951,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33320,"kb_used_data":2144,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338520,"statfs":{"total":96636764160,"available":96602644480,"internally_reserved":0,"allocated":2195456,"data_stored":1078258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574952,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33336,"kb_used_data":2152,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338504,"statfs":{"total":96636764160,"available":96602628096,"internally_reserved":0,"allocated":2203648,"data_stored":1085476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574952,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33344,"kb_used_data":2160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338496,"statfs":{"total":96636764160,"available":96602619904,"internally_reserved":0,"allocated":2211840,"data_stored":1085020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574953,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33336,"kb_used_data":2152,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338504,"statfs":{"total":96636764160,"available":96602628096,"internally_reserved":0,"allocated":2203648,"data_stored":1084984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574951,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33332,"kb_used_data":2148,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338508,"statfs":{"total":96636764160,"available":96602632192,"internally_reserved":0,"allocated":2199552,"data_stored":1084894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574951,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37548,"kb_used_data":2204,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334292,"statfs":{"total":96636764160,"available":96598315008,"internally_reserved":0,"allocated":2256896,"data_stored":1668660,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-21T23:10:58.993 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-02-21T23:10:58.993 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-02-21T23:10:59.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.273+0000 7f7b9b800640 1 Processor -- start 2024-02-21T23:10:59.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.274+0000 7f7b9b800640 1 -- start start 2024-02-21T23:10:59.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.276+0000 7f7b9b800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f7b940c7590 0x7f7b940c6c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:59.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.276+0000 7f7b9b800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7b940c5cc0 0x7f7b941524e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:59.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.276+0000 7f7b9b800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b94159580 con 0x7f7b94152a20 2024-02-21T23:10:59.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.276+0000 7f7b9b800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b940c6230 con 0x7f7b940c5cc0 2024-02-21T23:10:59.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.276+0000 7f7b9b800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b941587c0 con 0x7f7b940c7590 2024-02-21T23:10:59.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.276+0000 7f7b93e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7b940c5cc0 0x7f7b941524e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:59.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.276+0000 7f7b93e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7b940c5cc0 0x7f7b941524e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33056/0 (socket says 172.21.15.28:33056) 2024-02-21T23:10:59.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.276+0000 7f7b93e00640 1 -- 172.21.15.28:0/4189233081 learned_addr learned my addr 172.21.15.28:0/4189233081 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:59.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.277+0000 7f7b93e00640 1 -- 172.21.15.28:0/4189233081 >> v2:172.21.15.81:3300/0 conn(0x7f7b940c7590 msgr2=0x7f7b940c6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:59.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.277+0000 7f7b93e00640 1 --2- 172.21.15.28:0/4189233081 >> v2:172.21.15.81:3300/0 conn(0x7f7b940c7590 0x7f7b940c6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:59.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.277+0000 7f7b93e00640 1 -- 172.21.15.28:0/4189233081 >> v1:172.21.15.81:6789/0 conn(0x7f7b94152a20 legacy=0x7f7b940c3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:59.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.277+0000 7f7b93e00640 1 -- 172.21.15.28:0/4189233081 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b94158aa0 con 0x7f7b940c5cc0 2024-02-21T23:10:59.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.278+0000 7f7b93e00640 1 --2- 172.21.15.28:0/4189233081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b940c5cc0 0x7f7b941524e0 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7f7b940cb970 tx=0x7f7b8002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccecc103e33839d9 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:59.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.278+0000 7f7b93400640 1 -- 172.21.15.28:0/4189233081 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b8002e070 con 0x7f7b940c5cc0 2024-02-21T23:10:59.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.279+0000 7f7b93400640 1 -- 172.21.15.28:0/4189233081 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b80037070 con 0x7f7b940c5cc0 2024-02-21T23:10:59.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.279+0000 7f7b93400640 1 -- 172.21.15.28:0/4189233081 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b80031a00 con 0x7f7b940c5cc0 2024-02-21T23:10:59.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.279+0000 7f7b9b800640 1 -- 172.21.15.28:0/4189233081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b940c5cc0 msgr2=0x7f7b941524e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:59.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.279+0000 7f7b9b800640 1 --2- 172.21.15.28:0/4189233081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b940c5cc0 0x7f7b941524e0 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7f7b940cb970 tx=0x7f7b8002b880 comp rx=0 tx=0).stop 2024-02-21T23:10:59.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.279+0000 7f7b9b800640 1 -- 172.21.15.28:0/4189233081 shutdown_connections 2024-02-21T23:10:59.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.279+0000 7f7b9b800640 1 --2- 172.21.15.28:0/4189233081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b940c5cc0 0x7f7b941524e0 unknown :-1 s=CLOSED pgs=1930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:59.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.279+0000 7f7b9b800640 1 --2- 172.21.15.28:0/4189233081 >> v2:172.21.15.81:3300/0 conn(0x7f7b940c7590 0x7f7b940c6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:59.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.279+0000 7f7b9b800640 1 -- 172.21.15.28:0/4189233081 >> 172.21.15.28:0/4189233081 conn(0x7f7b9408afe0 msgr2=0x7f7b940725a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:59.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.279+0000 7f7b9b800640 1 -- 172.21.15.28:0/4189233081 shutdown_connections 2024-02-21T23:10:59.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.279+0000 7f7b9b800640 1 -- 172.21.15.28:0/4189233081 wait complete. 2024-02-21T23:10:59.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.281+0000 7f7b9b800640 1 Processor -- start 2024-02-21T23:10:59.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.281+0000 7f7b9b800640 1 -- start start 2024-02-21T23:10:59.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.282+0000 7f7b9b800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b940c5cc0 0x7f7b941557c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:59.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.282+0000 7f7b9b800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7b940c7590 0x7f7b9414d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:59.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.282+0000 7f7b9b800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7b94152a20 0x7f7b94144390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:59.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.282+0000 7f7b9b800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b940c6720 con 0x7f7b940c5cc0 2024-02-21T23:10:59.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.282+0000 7f7b9b800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b94159060 con 0x7f7b94152a20 2024-02-21T23:10:59.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.282+0000 7f7b9b800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b940c68a0 con 0x7f7b940c7590 2024-02-21T23:10:59.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.282+0000 7f7b98e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b940c5cc0 0x7f7b941557c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:59.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.282+0000 7f7b98e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b940c5cc0 0x7f7b941557c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33072/0 (socket says 172.21.15.28:33072) 2024-02-21T23:10:59.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.282+0000 7f7b98e00640 1 -- 172.21.15.28:0/4024930157 learned_addr learned my addr 172.21.15.28:0/4024930157 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:59.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.283+0000 7f7b93e00640 1 --2- 172.21.15.28:0/4024930157 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7b940c7590 0x7f7b9414d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:59.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.283+0000 7f7b98e00640 1 -- 172.21.15.28:0/4024930157 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7b940c7590 msgr2=0x7f7b9414d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:59.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.283+0000 7f7b98e00640 1 --2- 172.21.15.28:0/4024930157 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7b940c7590 0x7f7b9414d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:59.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.283+0000 7f7b98e00640 1 -- 172.21.15.28:0/4024930157 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7b94152a20 msgr2=0x7f7b94144390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:59.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.283+0000 7f7b99800640 1 --2- 172.21.15.28:0/4024930157 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7b94152a20 0x7f7b94144390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:59.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.283+0000 7f7b98e00640 1 --2- 172.21.15.28:0/4024930157 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7b94152a20 0x7f7b94144390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:59.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.283+0000 7f7b98e00640 1 -- 172.21.15.28:0/4024930157 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b940ca410 con 0x7f7b940c5cc0 2024-02-21T23:10:59.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.283+0000 7f7b98e00640 1 --2- 172.21.15.28:0/4024930157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b940c5cc0 0x7f7b941557c0 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7f7b8800b4a0 tx=0x7f7b8800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:59.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.283+0000 7f7b90c00640 1 -- 172.21.15.28:0/4024930157 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b88014070 con 0x7f7b940c5cc0 2024-02-21T23:10:59.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.284+0000 7f7b90c00640 1 -- 172.21.15.28:0/4024930157 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b88010610 con 0x7f7b940c5cc0 2024-02-21T23:10:59.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.284+0000 7f7b9b800640 1 -- 172.21.15.28:0/4024930157 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b941edd70 con 0x7f7b940c5cc0 2024-02-21T23:10:59.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.284+0000 7f7b90c00640 1 -- 172.21.15.28:0/4024930157 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b88010a00 con 0x7f7b940c5cc0 2024-02-21T23:10:59.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.284+0000 7f7b9b800640 1 -- 172.21.15.28:0/4024930157 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b941ee330 con 0x7f7b940c5cc0 2024-02-21T23:10:59.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.285+0000 7f7b9b800640 1 -- 172.21.15.28:0/4024930157 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b94067f30 con 0x7f7b940c5cc0 2024-02-21T23:10:59.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.286+0000 7f7b90c00640 1 -- 172.21.15.28:0/4024930157 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7b88017070 con 0x7f7b940c5cc0 2024-02-21T23:10:59.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.288+0000 7f7b90c00640 1 --2- 172.21.15.28:0/4024930157 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7b7406b500 0x7f7b7406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:59.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.288+0000 7f7b90c00640 1 -- 172.21.15.28:0/4024930157 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f7b88089dc0 con 0x7f7b940c5cc0 2024-02-21T23:10:59.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.288+0000 7f7b93e00640 1 --2- 172.21.15.28:0/4024930157 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7b7406b500 0x7f7b7406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:59.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.290+0000 7f7b93e00640 1 --2- 172.21.15.28:0/4024930157 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7b7406b500 0x7f7b7406d9c0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f7b80036000 tx=0x7f7b8003d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:59.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.291+0000 7f7b90c00640 1 -- 172.21.15.28:0/4024930157 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7b88058aa0 con 0x7f7b940c5cc0 2024-02-21T23:10:59.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.562+0000 7f7b9b800640 1 -- 172.21.15.28:0/4024930157 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f7b940ca5e0 con 0x7f7b940c5cc0 2024-02-21T23:10:59.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.567+0000 7f7b90c00640 1 -- 172.21.15.28:0/4024930157 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v20) v1 ==== 190+0+147 (secure 0 0 0) 0x7f7b880583a0 con 0x7f7b940c5cc0 2024-02-21T23:10:59.554 INFO:teuthology.orchestra.run.smithi028.stdout:[client.0] 2024-02-21T23:10:59.555 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQDTgNZlILvYLxAAoREWxKKT7YcXnhkgk8YvoQ== 2024-02-21T23:10:59.555 INFO:teuthology.orchestra.run.smithi028.stdout: caps mds = "allow" 2024-02-21T23:10:59.555 INFO:teuthology.orchestra.run.smithi028.stdout: caps mon = "allow r" 2024-02-21T23:10:59.555 INFO:teuthology.orchestra.run.smithi028.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-02-21T23:10:59.555 INFO:teuthology.orchestra.run.smithi028.stderr:updated caps for client.0 2024-02-21T23:10:59.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.569+0000 7f7b9b800640 1 -- 172.21.15.28:0/4024930157 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7b7406b500 msgr2=0x7f7b7406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:59.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.569+0000 7f7b9b800640 1 --2- 172.21.15.28:0/4024930157 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7b7406b500 0x7f7b7406d9c0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f7b80036000 tx=0x7f7b8003d040 comp rx=0 tx=0).stop 2024-02-21T23:10:59.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.569+0000 7f7b9b800640 1 -- 172.21.15.28:0/4024930157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b940c5cc0 msgr2=0x7f7b941557c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:59.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.569+0000 7f7b9b800640 1 --2- 172.21.15.28:0/4024930157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b940c5cc0 0x7f7b941557c0 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7f7b8800b4a0 tx=0x7f7b8800b960 comp rx=0 tx=0).stop 2024-02-21T23:10:59.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.569+0000 7f7b9b800640 1 -- 172.21.15.28:0/4024930157 shutdown_connections 2024-02-21T23:10:59.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.569+0000 7f7b9b800640 1 --2- 172.21.15.28:0/4024930157 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7b7406b500 0x7f7b7406d9c0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:59.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.569+0000 7f7b9b800640 1 --2- 172.21.15.28:0/4024930157 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7b940c7590 0x7f7b9414d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:59.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.569+0000 7f7b9b800640 1 --2- 172.21.15.28:0/4024930157 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7b94152a20 0x7f7b94144390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:59.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.569+0000 7f7b9b800640 1 --2- 172.21.15.28:0/4024930157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b940c5cc0 0x7f7b941557c0 unknown :-1 s=CLOSED pgs=1931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:59.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.569+0000 7f7b9b800640 1 -- 172.21.15.28:0/4024930157 >> 172.21.15.28:0/4024930157 conn(0x7f7b9408afe0 msgr2=0x7f7b94151590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:59.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.569+0000 7f7b9b800640 1 -- 172.21.15.28:0/4024930157 shutdown_connections 2024-02-21T23:10:59.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.569+0000 7f7b9b800640 1 -- 172.21.15.28:0/4024930157 wait complete. 2024-02-21T23:10:59.576 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:10:59.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.801+0000 7fed2ec00640 1 Processor -- start 2024-02-21T23:10:59.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.801+0000 7fed2ec00640 1 -- start start 2024-02-21T23:10:59.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.803+0000 7fed2ec00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fed280aff90 0x7fed280c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:59.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.803+0000 7fed2ec00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fed2806dbf0 0x7fed28158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:59.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.803+0000 7fed2ec00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fed280c5720 0x7fed28159000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:59.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.803+0000 7fed2ec00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed280c7270 con 0x7fed280aff90 2024-02-21T23:10:59.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.803+0000 7fed2ec00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed28158630 con 0x7fed2806dbf0 2024-02-21T23:10:59.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.803+0000 7fed2ec00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed280c74c0 con 0x7fed280c5720 2024-02-21T23:10:59.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.803+0000 7fed2d800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fed280aff90 0x7fed280c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:59.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.804+0000 7fed2d800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fed280aff90 0x7fed280c51e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33086/0 (socket says 172.21.15.28:33086) 2024-02-21T23:10:59.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.804+0000 7fed2d800640 1 -- 172.21.15.28:0/2316441170 learned_addr learned my addr 172.21.15.28:0/2316441170 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:59.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.804+0000 7fed2ce00640 1 --2- 172.21.15.28:0/2316441170 >> v2:172.21.15.81:3300/0 conn(0x7fed2806dbf0 0x7fed28158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:59.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.804+0000 7fed2e200640 1 --2- 172.21.15.28:0/2316441170 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fed280c5720 0x7fed28159000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:59.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.804+0000 7fed2d800640 1 -- 172.21.15.28:0/2316441170 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fed280c5720 msgr2=0x7fed28159000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:59.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.804+0000 7fed2d800640 1 --2- 172.21.15.28:0/2316441170 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fed280c5720 0x7fed28159000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:59.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.804+0000 7fed2d800640 1 -- 172.21.15.28:0/2316441170 >> v2:172.21.15.81:3300/0 conn(0x7fed2806dbf0 msgr2=0x7fed28158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:10:59.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.804+0000 7fed2d800640 1 --2- 172.21.15.28:0/2316441170 >> v2:172.21.15.81:3300/0 conn(0x7fed2806dbf0 0x7fed28158ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:59.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.804+0000 7fed2d800640 1 -- 172.21.15.28:0/2316441170 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed280c77a0 con 0x7fed280aff90 2024-02-21T23:10:59.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.806+0000 7fed2d800640 1 --2- 172.21.15.28:0/2316441170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed280aff90 0x7fed280c51e0 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7fed280cb8c0 tx=0x7fed1c02b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d74b4557c05aa25 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:59.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.806+0000 7fed27e00640 1 -- 172.21.15.28:0/2316441170 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed1c035020 con 0x7fed280aff90 2024-02-21T23:10:59.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.806+0000 7fed27e00640 1 -- 172.21.15.28:0/2316441170 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed1c02d040 con 0x7fed280aff90 2024-02-21T23:10:59.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.806+0000 7fed27e00640 1 -- 172.21.15.28:0/2316441170 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed1c031a80 con 0x7fed280aff90 2024-02-21T23:10:59.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.807+0000 7fed2ec00640 1 -- 172.21.15.28:0/2316441170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed280aff90 msgr2=0x7fed280c51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:59.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.807+0000 7fed2ec00640 1 --2- 172.21.15.28:0/2316441170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed280aff90 0x7fed280c51e0 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7fed280cb8c0 tx=0x7fed1c02b900 comp rx=0 tx=0).stop 2024-02-21T23:10:59.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.807+0000 7fed2ec00640 1 -- 172.21.15.28:0/2316441170 shutdown_connections 2024-02-21T23:10:59.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.807+0000 7fed2ec00640 1 --2- 172.21.15.28:0/2316441170 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fed280c5720 0x7fed28159000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:59.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.807+0000 7fed2ec00640 1 --2- 172.21.15.28:0/2316441170 >> v2:172.21.15.81:3300/0 conn(0x7fed2806dbf0 0x7fed28158ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:59.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.807+0000 7fed2ec00640 1 --2- 172.21.15.28:0/2316441170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed280aff90 0x7fed280c51e0 unknown :-1 s=CLOSED pgs=1932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:59.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.807+0000 7fed2ec00640 1 -- 172.21.15.28:0/2316441170 >> 172.21.15.28:0/2316441170 conn(0x7fed280ac370 msgr2=0x7fed280ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:10:59.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.807+0000 7fed2ec00640 1 -- 172.21.15.28:0/2316441170 shutdown_connections 2024-02-21T23:10:59.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.807+0000 7fed2ec00640 1 -- 172.21.15.28:0/2316441170 wait complete. 2024-02-21T23:10:59.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.810+0000 7fed2ec00640 1 Processor -- start 2024-02-21T23:10:59.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.810+0000 7fed2ec00640 1 -- start start 2024-02-21T23:10:59.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.811+0000 7fed2ec00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed2806dbf0 0x7fed280be930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:59.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.811+0000 7fed2ec00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed280aff90 0x7fed280c4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:59.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.811+0000 7fed2ec00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fed280c5720 0x7fed280c2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:59.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.811+0000 7fed2ec00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed280c8760 con 0x7fed280aff90 2024-02-21T23:10:59.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.811+0000 7fed2ec00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed280be0c0 con 0x7fed2806dbf0 2024-02-21T23:10:59.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.811+0000 7fed2ec00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed280c88e0 con 0x7fed280c5720 2024-02-21T23:10:59.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.811+0000 7fed2ce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed280aff90 0x7fed280c4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:59.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.812+0000 7fed2ce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed280aff90 0x7fed280c4090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33098/0 (socket says 172.21.15.28:33098) 2024-02-21T23:10:59.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.812+0000 7fed2ce00640 1 -- 172.21.15.28:0/675281575 learned_addr learned my addr 172.21.15.28:0/675281575 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:10:59.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.812+0000 7fed2d800640 1 --2- 172.21.15.28:0/675281575 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed2806dbf0 0x7fed280be930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:59.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.812+0000 7fed2ce00640 1 -- 172.21.15.28:0/675281575 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fed280c5720 msgr2=0x7fed280c2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:59.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.812+0000 7fed2ce00640 1 --2- 172.21.15.28:0/675281575 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fed280c5720 0x7fed280c2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:59.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.812+0000 7fed2ce00640 1 -- 172.21.15.28:0/675281575 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed2806dbf0 msgr2=0x7fed280be930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:10:59.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.812+0000 7fed2ce00640 1 --2- 172.21.15.28:0/675281575 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed2806dbf0 0x7fed280be930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:10:59.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.812+0000 7fed2ce00640 1 -- 172.21.15.28:0/675281575 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed280c1ad0 con 0x7fed280aff90 2024-02-21T23:10:59.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.812+0000 7fed2d800640 1 --2- 172.21.15.28:0/675281575 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed2806dbf0 0x7fed280be930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:10:59.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.812+0000 7fed2ce00640 1 --2- 172.21.15.28:0/675281575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed280aff90 0x7fed280c4090 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7fed1400e860 tx=0x7fed1400ed20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:59.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.813+0000 7fed25600640 1 -- 172.21.15.28:0/675281575 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed14017070 con 0x7fed280aff90 2024-02-21T23:10:59.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.813+0000 7fed25600640 1 -- 172.21.15.28:0/675281575 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed1400c940 con 0x7fed280aff90 2024-02-21T23:10:59.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.813+0000 7fed2ec00640 1 -- 172.21.15.28:0/675281575 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed280c1db0 con 0x7fed280aff90 2024-02-21T23:10:59.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.813+0000 7fed25600640 1 -- 172.21.15.28:0/675281575 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed1400cd10 con 0x7fed280aff90 2024-02-21T23:10:59.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.813+0000 7fed2ec00640 1 -- 172.21.15.28:0/675281575 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed280c2370 con 0x7fed280aff90 2024-02-21T23:10:59.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.814+0000 7fed2ec00640 1 -- 172.21.15.28:0/675281575 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed2804f290 con 0x7fed280aff90 2024-02-21T23:10:59.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.815+0000 7fed25600640 1 -- 172.21.15.28:0/675281575 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fed1401a070 con 0x7fed280aff90 2024-02-21T23:10:59.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.816+0000 7fed25600640 1 --2- 172.21.15.28:0/675281575 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fed0806b4d0 0x7fed0806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:10:59.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.816+0000 7fed25600640 1 -- 172.21.15.28:0/675281575 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fed1408a780 con 0x7fed280aff90 2024-02-21T23:10:59.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.817+0000 7fed2d800640 1 --2- 172.21.15.28:0/675281575 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fed0806b4d0 0x7fed0806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:10:59.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.818+0000 7fed2d800640 1 --2- 172.21.15.28:0/675281575 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fed0806b4d0 0x7fed0806d990 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fed1c036000 tx=0x7fed1c008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:10:59.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:10:59.825+0000 7fed25600640 1 -- 172.21.15.28:0/675281575 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fed14059460 con 0x7fed280aff90 2024-02-21T23:11:00.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.066+0000 7fed2ec00640 1 -- 172.21.15.28:0/675281575 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fed2806d9d0 con 0x7fed280aff90 2024-02-21T23:11:00.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.067+0000 7fed25600640 1 -- 172.21.15.28:0/675281575 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 185 v185) v1 ==== 95+0+4945 (secure 0 0 0) 0x7fed14058d60 con 0x7fed280aff90 2024-02-21T23:11:00.055 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:11:00.055 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":185,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6686,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/155806034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":155806034},{"type":"v1","addr":"172.21.15.81:6836","nonce":155806034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":180},{"gid":7480,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3897243745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":3897243745},{"type":"v1","addr":"172.21.15.81:6837","nonce":3897243745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":180},{"gid":7486,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2827135632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2827135632},{"type":"v1","addr":"172.21.15.28:6835","nonce":2827135632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":180}],"filesystems":[{"mdsmap":{"epoch":185,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:10:53.351225+0000","modified":"2024-02-21T23:10:56.419054+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9480},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9480":{"gid":9480,"name":"c","rank":0,"incarnation":181,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6833/1850463609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1850463609},{"type":"v1","addr":"172.21.15.28:6833","nonce":1850463609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2024-02-21T23:11:00.056 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 185 2024-02-21T23:11:00.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.070+0000 7fed2ec00640 1 -- 172.21.15.28:0/675281575 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fed0806b4d0 msgr2=0x7fed0806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:00.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.070+0000 7fed2ec00640 1 --2- 172.21.15.28:0/675281575 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fed0806b4d0 0x7fed0806d990 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fed1c036000 tx=0x7fed1c008030 comp rx=0 tx=0).stop 2024-02-21T23:11:00.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.070+0000 7fed2ec00640 1 -- 172.21.15.28:0/675281575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed280aff90 msgr2=0x7fed280c4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:00.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.070+0000 7fed2ec00640 1 --2- 172.21.15.28:0/675281575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed280aff90 0x7fed280c4090 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7fed1400e860 tx=0x7fed1400ed20 comp rx=0 tx=0).stop 2024-02-21T23:11:00.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.070+0000 7fed2ec00640 1 -- 172.21.15.28:0/675281575 shutdown_connections 2024-02-21T23:11:00.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.070+0000 7fed2ec00640 1 --2- 172.21.15.28:0/675281575 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fed0806b4d0 0x7fed0806d990 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:00.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.070+0000 7fed2ec00640 1 --2- 172.21.15.28:0/675281575 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fed280c5720 0x7fed280c2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:00.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.070+0000 7fed2ec00640 1 --2- 172.21.15.28:0/675281575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed280aff90 0x7fed280c4090 unknown :-1 s=CLOSED pgs=1933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:00.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.070+0000 7fed2ec00640 1 --2- 172.21.15.28:0/675281575 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed2806dbf0 0x7fed280be930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:00.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.070+0000 7fed2ec00640 1 -- 172.21.15.28:0/675281575 >> 172.21.15.28:0/675281575 conn(0x7fed280ac370 msgr2=0x7fed280ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:00.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.070+0000 7fed2ec00640 1 -- 172.21.15.28:0/675281575 shutdown_connections 2024-02-21T23:11:00.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.070+0000 7fed2ec00640 1 -- 172.21.15.28:0/675281575 wait complete. 2024-02-21T23:11:00.481 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 185, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:10:53.351225+0000', 'modified': '2024-02-21T23:10:56.419054+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9480}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9480': {'gid': 9480, 'name': 'c', 'rank': 0, 'incarnation': 181, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6833/1850463609', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 1850463609}, {'type': 'v1', 'addr': '172.21.15.28:6833', 'nonce': 1850463609}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:11:00.481 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-21T23:11:00.482 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-02-21T23:11:00.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.720+0000 7fc0b6a00640 1 Processor -- start 2024-02-21T23:11:00.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.721+0000 7fc0b6a00640 1 -- start start 2024-02-21T23:11:00.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.723+0000 7fc0b6a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc0b01524c0 0x7fc0b0155a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:00.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.723+0000 7fc0b6a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fc0b00c5040 0x7fc0b006ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:00.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.723+0000 7fc0b6a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc0b00c85c0 0x7fc0b00c70d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:00.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.723+0000 7fc0b6a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0b006df50 con 0x7fc0b01524c0 2024-02-21T23:11:00.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.723+0000 7fc0b6a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0b006d3b0 con 0x7fc0b00c5040 2024-02-21T23:11:00.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.723+0000 7fc0b6a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b00c5520 con 0x7fc0b00c85c0 2024-02-21T23:11:00.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.723+0000 7fc0b5600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc0b01524c0 0x7fc0b0155a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:00.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.723+0000 7fc0b5600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc0b01524c0 0x7fc0b0155a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33102/0 (socket says 172.21.15.28:33102) 2024-02-21T23:11:00.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.723+0000 7fc0b5600640 1 -- 172.21.15.28:0/2108666253 learned_addr learned my addr 172.21.15.28:0/2108666253 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:00.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.723+0000 7fc0b6000640 1 --2- 172.21.15.28:0/2108666253 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc0b00c85c0 0x7fc0b00c70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:00.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.723+0000 7fc0b4c00640 1 --2- 172.21.15.28:0/2108666253 >> v2:172.21.15.81:3300/0 conn(0x7fc0b00c5040 0x7fc0b006ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:00.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.724+0000 7fc0b5600640 1 -- 172.21.15.28:0/2108666253 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc0b00c85c0 msgr2=0x7fc0b00c70d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:00.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.724+0000 7fc0b5600640 1 --2- 172.21.15.28:0/2108666253 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc0b00c85c0 0x7fc0b00c70d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:00.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.724+0000 7fc0b5600640 1 -- 172.21.15.28:0/2108666253 >> v2:172.21.15.81:3300/0 conn(0x7fc0b00c5040 msgr2=0x7fc0b006ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:00.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.724+0000 7fc0b5600640 1 --2- 172.21.15.28:0/2108666253 >> v2:172.21.15.81:3300/0 conn(0x7fc0b00c5040 0x7fc0b006ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:00.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.724+0000 7fc0b5600640 1 -- 172.21.15.28:0/2108666253 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0b00c5800 con 0x7fc0b01524c0 2024-02-21T23:11:00.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.724+0000 7fc0b4c00640 1 --2- 172.21.15.28:0/2108666253 >> v2:172.21.15.81:3300/0 conn(0x7fc0b00c5040 0x7fc0b006ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:11:00.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.726+0000 7fc0b5600640 1 --2- 172.21.15.28:0/2108666253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0b01524c0 0x7fc0b0155a10 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7fc0b00cb760 tx=0x7fc0a402b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da3b1de5d566c6d3 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:00.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.726+0000 7fc0afe00640 1 -- 172.21.15.28:0/2108666253 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a4038070 con 0x7fc0b01524c0 2024-02-21T23:11:00.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.726+0000 7fc0afe00640 1 -- 172.21.15.28:0/2108666253 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0a4031740 con 0x7fc0b01524c0 2024-02-21T23:11:00.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.726+0000 7fc0afe00640 1 -- 172.21.15.28:0/2108666253 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a402f050 con 0x7fc0b01524c0 2024-02-21T23:11:00.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.727+0000 7fc0b6a00640 1 -- 172.21.15.28:0/2108666253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0b01524c0 msgr2=0x7fc0b0155a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:00.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.727+0000 7fc0b6a00640 1 --2- 172.21.15.28:0/2108666253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0b01524c0 0x7fc0b0155a10 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7fc0b00cb760 tx=0x7fc0a402b7e0 comp rx=0 tx=0).stop 2024-02-21T23:11:00.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.727+0000 7fc0b6a00640 1 -- 172.21.15.28:0/2108666253 shutdown_connections 2024-02-21T23:11:00.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.727+0000 7fc0b6a00640 1 --2- 172.21.15.28:0/2108666253 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc0b00c85c0 0x7fc0b00c70d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:00.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.727+0000 7fc0b6a00640 1 --2- 172.21.15.28:0/2108666253 >> v2:172.21.15.81:3300/0 conn(0x7fc0b00c5040 0x7fc0b006ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:00.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.727+0000 7fc0b6a00640 1 --2- 172.21.15.28:0/2108666253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0b01524c0 0x7fc0b0155a10 unknown :-1 s=CLOSED pgs=1934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:00.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.727+0000 7fc0b6a00640 1 -- 172.21.15.28:0/2108666253 >> 172.21.15.28:0/2108666253 conn(0x7fc0b00ac2b0 msgr2=0x7fc0b00ac6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:00.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.727+0000 7fc0b6a00640 1 -- 172.21.15.28:0/2108666253 shutdown_connections 2024-02-21T23:11:00.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.727+0000 7fc0b6a00640 1 -- 172.21.15.28:0/2108666253 wait complete. 2024-02-21T23:11:00.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.730+0000 7fc0b6a00640 1 Processor -- start 2024-02-21T23:11:00.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.730+0000 7fc0b6a00640 1 -- start start 2024-02-21T23:11:00.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.732+0000 7fc0b6a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc0b00c5040 0x7fc0b00aa090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:00.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.732+0000 7fc0b6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0b00c85c0 0x7fc0b0158fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:00.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.732+0000 7fc0b6a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0b01524c0 0x7fc0b0153dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:00.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.732+0000 7fc0b6a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b01528a0 con 0x7fc0b00c85c0 2024-02-21T23:11:00.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.732+0000 7fc0b6a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b006c950 con 0x7fc0b01524c0 2024-02-21T23:11:00.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.732+0000 7fc0b6a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b00c8a70 con 0x7fc0b00c5040 2024-02-21T23:11:00.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.732+0000 7fc0b4c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0b00c85c0 0x7fc0b0158fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:00.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.732+0000 7fc0b4c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0b00c85c0 0x7fc0b0158fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33106/0 (socket says 172.21.15.28:33106) 2024-02-21T23:11:00.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.732+0000 7fc0b4c00640 1 -- 172.21.15.28:0/2599789225 learned_addr learned my addr 172.21.15.28:0/2599789225 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:00.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.732+0000 7fc0b5600640 1 --2- 172.21.15.28:0/2599789225 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc0b00c5040 0x7fc0b00aa090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:00.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.732+0000 7fc0b6000640 1 --2- 172.21.15.28:0/2599789225 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0b01524c0 0x7fc0b0153dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:00.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.733+0000 7fc0b4c00640 1 -- 172.21.15.28:0/2599789225 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc0b00c5040 msgr2=0x7fc0b00aa090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:00.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.733+0000 7fc0b4c00640 1 --2- 172.21.15.28:0/2599789225 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc0b00c5040 0x7fc0b00aa090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:00.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.733+0000 7fc0b4c00640 1 -- 172.21.15.28:0/2599789225 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0b01524c0 msgr2=0x7fc0b0153dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:00.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.733+0000 7fc0b4c00640 1 --2- 172.21.15.28:0/2599789225 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0b01524c0 0x7fc0b0153dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:00.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.733+0000 7fc0b4c00640 1 -- 172.21.15.28:0/2599789225 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0b0155360 con 0x7fc0b00c85c0 2024-02-21T23:11:00.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.733+0000 7fc0b4c00640 1 --2- 172.21.15.28:0/2599789225 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0b00c85c0 0x7fc0b0158fb0 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7fc09c00e840 tx=0x7fc09c00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:00.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.733+0000 7fc0ad600640 1 -- 172.21.15.28:0/2599789225 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc09c00f040 con 0x7fc0b00c85c0 2024-02-21T23:11:00.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.733+0000 7fc0ad600640 1 -- 172.21.15.28:0/2599789225 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc09c004030 con 0x7fc0b00c85c0 2024-02-21T23:11:00.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.733+0000 7fc0ad600640 1 -- 172.21.15.28:0/2599789225 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc09c004400 con 0x7fc0b00c85c0 2024-02-21T23:11:00.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.733+0000 7fc0b6a00640 1 -- 172.21.15.28:0/2599789225 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0b0155080 con 0x7fc0b00c85c0 2024-02-21T23:11:00.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.733+0000 7fc0b6a00640 1 -- 172.21.15.28:0/2599789225 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0b01535c0 con 0x7fc0b00c85c0 2024-02-21T23:11:00.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.736+0000 7fc0ad600640 1 -- 172.21.15.28:0/2599789225 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc09c00c6d0 con 0x7fc0b00c85c0 2024-02-21T23:11:00.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.737+0000 7fc0ad600640 1 --2- 172.21.15.28:0/2599789225 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc09006b4d0 0x7fc09006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:00.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.737+0000 7fc0bafee740 1 Processor -- start 2024-02-21T23:11:00.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.737+0000 7fc0bafee740 1 -- start start 2024-02-21T23:11:00.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.737+0000 7fc0ad600640 1 -- 172.21.15.28:0/2599789225 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc09c089fe0 con 0x7fc0b00c85c0 2024-02-21T23:11:00.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.738+0000 7fc0b5600640 1 --2- 172.21.15.28:0/2599789225 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc09006b4d0 0x7fc09006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:00.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.738+0000 7fc0bafee740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560c5745f020 0x560c5745f400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:00.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.738+0000 7fc0bafee740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560c57460330 0x560c5749b170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:00.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.738+0000 7fc0bafee740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x560c5749b6b0 0x560c57499720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:00.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.738+0000 7fc0bafee740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c573c3470 con 0x560c5745f020 2024-02-21T23:11:00.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.738+0000 7fc0bafee740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c5736caf0 con 0x560c57460330 2024-02-21T23:11:00.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.738+0000 7fc0bafee740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x560c573bf510 con 0x560c5749b6b0 2024-02-21T23:11:00.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.739+0000 7fc0b5600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560c5745f020 0x560c5745f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:00.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.739+0000 7fc0b5600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560c5745f020 0x560c5745f400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33110/0 (socket says 172.21.15.28:33110) 2024-02-21T23:11:00.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.739+0000 7fc0b5600640 1 -- 172.21.15.28:0/900459155 learned_addr learned my addr 172.21.15.28:0/900459155 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:00.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.739+0000 7fc0b4c00640 1 --2- 172.21.15.28:0/900459155 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560c57460330 0x560c5749b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:00.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.739+0000 7fc0b6000640 1 --2- 172.21.15.28:0/900459155 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x560c5749b6b0 0x560c57499720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:00.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.739+0000 7fc0b5600640 1 -- 172.21.15.28:0/900459155 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x560c5749b6b0 msgr2=0x560c57499720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:00.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.739+0000 7fc0b5600640 1 --2- 172.21.15.28:0/900459155 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x560c5749b6b0 0x560c57499720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:00.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.739+0000 7fc0b5600640 1 -- 172.21.15.28:0/900459155 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560c57460330 msgr2=0x560c5749b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:00.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.739+0000 7fc0b5600640 1 --2- 172.21.15.28:0/900459155 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560c57460330 0x560c5749b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:00.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.739+0000 7fc0b5600640 1 -- 172.21.15.28:0/900459155 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560c57373d90 con 0x560c5745f020 2024-02-21T23:11:00.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.739+0000 7fc0b6000640 1 --2- 172.21.15.28:0/900459155 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x560c5749b6b0 0x560c57499720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:11:00.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.740+0000 7fc0b5600640 1 --2- 172.21.15.28:0/900459155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560c5745f020 0x560c5745f400 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7fc0a4009960 tx=0x7fc0a4045000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecf41be368d93f74 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:00.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.740+0000 7fc0b5600640 1 --2- 172.21.15.28:0/2599789225 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc09006b4d0 0x7fc09006d990 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fc0a4036040 tx=0x7fc0a402e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:00.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.740+0000 7fc0a1c00640 1 -- 172.21.15.28:0/900459155 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a4038040 con 0x560c5745f020 2024-02-21T23:11:00.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.740+0000 7fc0a1c00640 1 -- 172.21.15.28:0/900459155 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0a4051070 con 0x560c5745f020 2024-02-21T23:11:00.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.740+0000 7fc0a1c00640 1 -- 172.21.15.28:0/900459155 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a404c6a0 con 0x560c5745f020 2024-02-21T23:11:00.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.741+0000 7fc0bafee740 1 -- 172.21.15.28:0/900459155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560c5745f020 msgr2=0x560c5745f400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:00.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.741+0000 7fc0bafee740 1 --2- 172.21.15.28:0/900459155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560c5745f020 0x560c5745f400 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7fc0a4009960 tx=0x7fc0a4045000 comp rx=0 tx=0).stop 2024-02-21T23:11:00.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.741+0000 7fc0bafee740 1 -- 172.21.15.28:0/900459155 shutdown_connections 2024-02-21T23:11:00.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.741+0000 7fc0bafee740 1 --2- 172.21.15.28:0/900459155 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x560c5749b6b0 0x560c57499720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:00.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.741+0000 7fc0bafee740 1 --2- 172.21.15.28:0/900459155 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560c57460330 0x560c5749b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:00.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.741+0000 7fc0bafee740 1 --2- 172.21.15.28:0/900459155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560c5745f020 0x560c5745f400 unknown :-1 s=CLOSED pgs=1936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:00.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.741+0000 7fc0bafee740 1 -- 172.21.15.28:0/900459155 >> 172.21.15.28:0/900459155 conn(0x560c5745b970 msgr2=0x560c57480380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:00.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.741+0000 7fc0bafee740 1 -- 172.21.15.28:0/900459155 shutdown_connections 2024-02-21T23:11:00.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.741+0000 7fc0bafee740 1 -- 172.21.15.28:0/900459155 wait complete. 2024-02-21T23:11:00.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.742+0000 7fc0bafee740 1 -- start start 2024-02-21T23:11:00.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.742+0000 7fc0bafee740 1 Processor -- start 2024-02-21T23:11:00.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.743+0000 7fc0bafee740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560c57460330 0x560c5747adf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:00.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.743+0000 7fc0bafee740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x560c5749b6b0 0x560c57499c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:00.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.743+0000 7fc0bafee740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560c5749a150 0x560c574ab380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:00.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.743+0000 7fc0bafee740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c57362040 con 0x560c57460330 2024-02-21T23:11:00.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.743+0000 7fc0bafee740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c57480a90 con 0x560c5749a150 2024-02-21T23:11:00.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.743+0000 7fc0bafee740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x560c574abf70 con 0x560c5749b6b0 2024-02-21T23:11:00.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.743+0000 7fc0bafee740 10 client.0 _ll_register_callbacks cb 0x560c5737f920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:11:00.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.743+0000 7fc0b5600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560c57460330 0x560c5747adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:00.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.743+0000 7fc0b5600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560c57460330 0x560c5747adf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33122/0 (socket says 172.21.15.28:33122) 2024-02-21T23:11:00.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.744+0000 7fc0b5600640 1 -- 172.21.15.28:0/3050152041 learned_addr learned my addr 172.21.15.28:0/3050152041 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:00.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.744+0000 7fc0b4c00640 1 --2- 172.21.15.28:0/3050152041 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x560c5749b6b0 0x560c57499c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:00.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.744+0000 7fc0b5600640 1 -- 172.21.15.28:0/3050152041 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x560c5749b6b0 msgr2=0x560c57499c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:00.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.744+0000 7fc0b5600640 1 --2- 172.21.15.28:0/3050152041 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x560c5749b6b0 0x560c57499c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:00.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.744+0000 7fc0b5600640 1 -- 172.21.15.28:0/3050152041 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560c5749a150 msgr2=0x560c574ab380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:00.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.744+0000 7fc0b6000640 1 --2- 172.21.15.28:0/3050152041 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560c5749a150 0x560c574ab380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:00.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.744+0000 7fc0b5600640 1 --2- 172.21.15.28:0/3050152041 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560c5749a150 0x560c574ab380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:00.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.744+0000 7fc0b5600640 1 -- 172.21.15.28:0/3050152041 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560c57381aa0 con 0x560c57460330 2024-02-21T23:11:00.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.744+0000 7fc0b5600640 1 --2- 172.21.15.28:0/3050152041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560c57460330 0x560c5747adf0 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7fc0a4009930 tx=0x7fc0a4031000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55d24e93ed01d169 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:00.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.744+0000 7fc096000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:11:00.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.745+0000 7fc096000640 1 -- 172.21.15.28:0/3050152041 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a404c660 con 0x560c57460330 2024-02-21T23:11:00.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.745+0000 7fc0bafee740 1 -- 172.21.15.28:0/3050152041 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc0a4051070 con 0x560c57460330 2024-02-21T23:11:00.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.745+0000 7fc096000640 1 -- 172.21.15.28:0/3050152041 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(174..174 src has 1..174) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc0a404c800 con 0x560c57460330 2024-02-21T23:11:00.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.745+0000 7fc096000640 1 -- 172.21.15.28:0/3050152041 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=185) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc0a404ca00 con 0x560c57460330 2024-02-21T23:11:00.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.745+0000 7fc0bafee740 10 client.9627 fetch_fsmap learned FSMap version 185 2024-02-21T23:11:00.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.745+0000 7fc0bafee740 1 -- 172.21.15.28:0/3050152041 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=185,monmap=2+}) v3 -- 0x560c570d82d0 con 0x560c57460330 2024-02-21T23:11:00.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.746+0000 7fc096000640 1 -- 172.21.15.28:0/3050152041 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0a4051070 con 0x560c57460330 2024-02-21T23:11:00.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.746+0000 7fc096000640 1 -- 172.21.15.28:0/3050152041 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 185) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fc0a404cba0 con 0x560c57460330 2024-02-21T23:11:00.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.746+0000 7fc0bafee740 10 client.9627 fetch_fsmap finished waiting for FSMap version 185 2024-02-21T23:11:00.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.746+0000 7fc0bafee740 10 client.9627 resolve_mds: resolved name 'c' to daemon mds.c 2024-02-21T23:11:00.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.746+0000 7fc0bafee740 20 client.9627 populate_metadata read hostname 'smithi028' 2024-02-21T23:11:00.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.746+0000 7fc0bafee740 1 --2- 172.21.15.28:0/3050152041 >> [v2:172.21.15.28:6832/1850463609,v1:172.21.15.28:6833/1850463609] conn(0x560c574aeb30 0x560c574aef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:00.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.746+0000 7fc0bafee740 4 client.9627 mds_command: new command op to 9480 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-21T23:11:00.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.746+0000 7fc0bafee740 1 -- 172.21.15.28:0/3050152041 --> [v2:172.21.15.28:6832/1850463609,v1:172.21.15.28:6833/1850463609] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560c573d09b0 con 0x560c574aeb30 2024-02-21T23:11:00.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.746+0000 7fc0b4c00640 1 --2- 172.21.15.28:0/3050152041 >> [v2:172.21.15.28:6832/1850463609,v1:172.21.15.28:6833/1850463609] conn(0x560c574aeb30 0x560c574aef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:00.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.747+0000 7fc0b4c00640 1 --2- 172.21.15.28:0/3050152041 >> [v2:172.21.15.28:6832/1850463609,v1:172.21.15.28:6833/1850463609] conn(0x560c574aeb30 0x560c574aef10 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=80ec42daaa837272 server_cookie=3b5e18424f9dcaeb in_seq=0 out_seq=0 2024-02-21T23:11:00.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.747+0000 7fc096000640 10 client.9627 ms_handle_connect on v2:172.21.15.28:6832/1850463609 2024-02-21T23:11:00.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.747+0000 7fc096000640 1 -- 172.21.15.28:0/3050152041 <== mds.0 v2:172.21.15.28:6832/1850463609 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x560c573d09b0 con 0x560c574aeb30 2024-02-21T23:11:00.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.747+0000 7fc096000640 10 client.9627 handle_command_reply: tid=0 2024-02-21T23:11:00.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.747+0000 7fc0bafee740 1 client.9627 shutdown 2024-02-21T23:11:00.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.748+0000 7fc0bafee740 1 -- 172.21.15.28:0/3050152041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560c57460330 msgr2=0x560c5747adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:00.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.748+0000 7fc0bafee740 1 --2- 172.21.15.28:0/3050152041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560c57460330 0x560c5747adf0 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7fc0a4009930 tx=0x7fc0a4031000 comp rx=0 tx=0).stop 2024-02-21T23:11:00.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.748+0000 7fc0bafee740 1 -- 172.21.15.28:0/3050152041 shutdown_connections 2024-02-21T23:11:00.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.748+0000 7fc0bafee740 1 --2- 172.21.15.28:0/3050152041 >> [v2:172.21.15.28:6832/1850463609,v1:172.21.15.28:6833/1850463609] conn(0x560c574aeb30 0x560c574aef10 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:00.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.748+0000 7fc0bafee740 1 --2- 172.21.15.28:0/3050152041 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x560c5749b6b0 0x560c57499c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:00.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.748+0000 7fc0bafee740 1 --2- 172.21.15.28:0/3050152041 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560c5749a150 0x560c574ab380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:00.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.748+0000 7fc0bafee740 1 --2- 172.21.15.28:0/3050152041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560c57460330 0x560c5747adf0 unknown :-1 s=CLOSED pgs=1937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:00.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.748+0000 7fc0bafee740 1 -- 172.21.15.28:0/3050152041 >> 172.21.15.28:0/3050152041 conn(0x560c5745f020 msgr2=0x560c5749e790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:00.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.748+0000 7fc096000640 0 client.9627 ms_handle_reset on v2:172.21.15.28:6832/1850463609 2024-02-21T23:11:00.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.748+0000 7fc0bafee740 1 -- 172.21.15.28:0/3050152041 shutdown_connections 2024-02-21T23:11:00.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.748+0000 7fc0bafee740 1 -- 172.21.15.28:0/3050152041 wait complete. 2024-02-21T23:11:00.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.748+0000 7fc0bafee740 20 client.9627 trim_cache size 0 max 16384 2024-02-21T23:11:00.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.766+0000 7fc0bafee740 1 Processor -- start 2024-02-21T23:11:00.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.767+0000 7fc0bafee740 1 -- start start 2024-02-21T23:11:00.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.767+0000 7fc0bafee740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x560c57460330 0x560c574aef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:00.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.767+0000 7fc0bafee740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560c5749a150 0x560c574a7920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:00.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.767+0000 7fc0bafee740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560c5749b6b0 0x560c574a7e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:00.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.767+0000 7fc0bafee740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b01f6c70 con 0x560c5749a150 2024-02-21T23:11:00.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.767+0000 7fc0bafee740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b0148230 con 0x560c5749b6b0 2024-02-21T23:11:00.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.767+0000 7fc0bafee740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b01f6670 con 0x560c57460330 2024-02-21T23:11:00.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.768+0000 7fc0b4c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560c5749a150 0x560c574a7920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:00.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.768+0000 7fc0b4c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560c5749a150 0x560c574a7920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33124/0 (socket says 172.21.15.28:33124) 2024-02-21T23:11:00.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.768+0000 7fc0b4c00640 1 -- 172.21.15.28:0/2421481049 learned_addr learned my addr 172.21.15.28:0/2421481049 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:00.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.768+0000 7fc0b5600640 1 --2- 172.21.15.28:0/2421481049 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x560c57460330 0x560c574aef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:00.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.768+0000 7fc0b6000640 1 --2- 172.21.15.28:0/2421481049 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560c5749b6b0 0x560c574a7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:00.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.768+0000 7fc0b4c00640 1 -- 172.21.15.28:0/2421481049 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x560c57460330 msgr2=0x560c574aef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:00.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.768+0000 7fc0b4c00640 1 --2- 172.21.15.28:0/2421481049 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x560c57460330 0x560c574aef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:00.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.768+0000 7fc0b4c00640 1 -- 172.21.15.28:0/2421481049 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560c5749b6b0 msgr2=0x560c574a7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:00.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.768+0000 7fc0b4c00640 1 --2- 172.21.15.28:0/2421481049 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560c5749b6b0 0x560c574a7e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:00.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.768+0000 7fc0b4c00640 1 -- 172.21.15.28:0/2421481049 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560c5745f770 con 0x560c5749a150 2024-02-21T23:11:00.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.768+0000 7fc0b5600640 1 --2- 172.21.15.28:0/2421481049 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x560c57460330 0x560c574aef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:11:00.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.768+0000 7fc0b6000640 1 --2- 172.21.15.28:0/2421481049 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560c5749b6b0 0x560c574a7e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:11:00.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.769+0000 7fc0b4c00640 1 --2- 172.21.15.28:0/2421481049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560c5749a150 0x560c574a7920 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7fc09c0585f0 tx=0x7fc09c01d040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4df93ec31558e96 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:00.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.769+0000 7fc0a1200640 1 -- 172.21.15.28:0/2421481049 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc09c020040 con 0x560c5749a150 2024-02-21T23:11:00.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.769+0000 7fc0a1200640 1 -- 172.21.15.28:0/2421481049 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc09c085020 con 0x560c5749a150 2024-02-21T23:11:00.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.769+0000 7fc0a1200640 1 -- 172.21.15.28:0/2421481049 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc09c058430 con 0x560c5749a150 2024-02-21T23:11:00.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.769+0000 7fc0bafee740 1 -- 172.21.15.28:0/2421481049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560c5749a150 msgr2=0x560c574a7920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:00.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.769+0000 7fc0bafee740 1 --2- 172.21.15.28:0/2421481049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560c5749a150 0x560c574a7920 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7fc09c0585f0 tx=0x7fc09c01d040 comp rx=0 tx=0).stop 2024-02-21T23:11:00.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.770+0000 7fc0bafee740 1 -- 172.21.15.28:0/2421481049 shutdown_connections 2024-02-21T23:11:00.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.770+0000 7fc0bafee740 1 --2- 172.21.15.28:0/2421481049 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560c5749b6b0 0x560c574a7e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:00.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.770+0000 7fc0bafee740 1 --2- 172.21.15.28:0/2421481049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560c5749a150 0x560c574a7920 unknown :-1 s=CLOSED pgs=1938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:00.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.770+0000 7fc0bafee740 1 --2- 172.21.15.28:0/2421481049 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x560c57460330 0x560c574aef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:00.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.770+0000 7fc0bafee740 1 -- 172.21.15.28:0/2421481049 >> 172.21.15.28:0/2421481049 conn(0x560c5745f020 msgr2=0x560c5747fd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:00.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.770+0000 7fc0bafee740 1 -- 172.21.15.28:0/2421481049 shutdown_connections 2024-02-21T23:11:00.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.770+0000 7fc0bafee740 1 -- 172.21.15.28:0/2421481049 wait complete. 2024-02-21T23:11:00.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.770+0000 7fc0bafee740 1 -- start start 2024-02-21T23:11:00.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.770+0000 7fc0bafee740 1 Processor -- start 2024-02-21T23:11:00.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.771+0000 7fc0bafee740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x560c57460330 0x560c57455110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:00.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.771+0000 7fc0bafee740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560c5749a150 0x560c5749f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:00.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.771+0000 7fc0bafee740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560c5749b6b0 0x560c575077b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:00.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.771+0000 7fc0bafee740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c57362040 con 0x560c5749a150 2024-02-21T23:11:00.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.771+0000 7fc0bafee740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c57480a90 con 0x560c5749b6b0 2024-02-21T23:11:00.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.771+0000 7fc0bafee740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x560c574959d0 con 0x560c57460330 2024-02-21T23:11:00.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.771+0000 7fc0bafee740 10 client.0 _ll_register_callbacks cb 0x7fc0a40096f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:11:00.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.772+0000 7fc0b4c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560c5749a150 0x560c5749f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:00.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.772+0000 7fc0b4c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560c5749a150 0x560c5749f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33140/0 (socket says 172.21.15.28:33140) 2024-02-21T23:11:00.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.772+0000 7fc0b4c00640 1 -- 172.21.15.28:0/1899947953 learned_addr learned my addr 172.21.15.28:0/1899947953 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:00.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.772+0000 7fc0b5600640 1 --2- 172.21.15.28:0/1899947953 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x560c57460330 0x560c57455110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:00.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.772+0000 7fc0b6000640 1 --2- 172.21.15.28:0/1899947953 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560c5749b6b0 0x560c575077b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:00.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.772+0000 7fc0b4c00640 1 -- 172.21.15.28:0/1899947953 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x560c57460330 msgr2=0x560c57455110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:00.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.772+0000 7fc0b4c00640 1 --2- 172.21.15.28:0/1899947953 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x560c57460330 0x560c57455110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:00.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.772+0000 7fc0b4c00640 1 -- 172.21.15.28:0/1899947953 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560c5749b6b0 msgr2=0x560c575077b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:00.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.772+0000 7fc0b4c00640 1 --2- 172.21.15.28:0/1899947953 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560c5749b6b0 0x560c575077b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:00.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.772+0000 7fc0b4c00640 1 -- 172.21.15.28:0/1899947953 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560c571a26b0 con 0x560c5749a150 2024-02-21T23:11:00.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.772+0000 7fc0b4c00640 1 --2- 172.21.15.28:0/1899947953 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560c5749a150 0x560c5749f030 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7fc09c018000 tx=0x7fc09c06e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edf49a9b73d609ab server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:00.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.772+0000 7fc096000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:11:00.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.773+0000 7fc096000640 1 -- 172.21.15.28:0/1899947953 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc09c058430 con 0x560c5749a150 2024-02-21T23:11:00.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.773+0000 7fc096000640 1 -- 172.21.15.28:0/1899947953 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(174..174 src has 1..174) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc09c054cd0 con 0x560c5749a150 2024-02-21T23:11:00.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.773+0000 7fc0bafee740 1 -- 172.21.15.28:0/1899947953 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc09c085020 con 0x560c5749a150 2024-02-21T23:11:00.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.773+0000 7fc096000640 1 -- 172.21.15.28:0/1899947953 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=185) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc09c07f050 con 0x560c5749a150 2024-02-21T23:11:00.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.773+0000 7fc0bafee740 10 client.9633 fetch_fsmap learned FSMap version 185 2024-02-21T23:11:00.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.773+0000 7fc0bafee740 1 -- 172.21.15.28:0/1899947953 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=185,monmap=2+}) v3 -- 0x560c574a7920 con 0x560c5749a150 2024-02-21T23:11:00.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.773+0000 7fc096000640 1 -- 172.21.15.28:0/1899947953 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc09c085020 con 0x560c5749a150 2024-02-21T23:11:00.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.773+0000 7fc096000640 1 -- 172.21.15.28:0/1899947953 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 185) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fc09c012050 con 0x560c5749a150 2024-02-21T23:11:00.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.773+0000 7fc0bafee740 10 client.9633 fetch_fsmap finished waiting for FSMap version 185 2024-02-21T23:11:00.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.773+0000 7fc0bafee740 10 client.9633 resolve_mds: resolved name 'c' to daemon mds.c 2024-02-21T23:11:00.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.773+0000 7fc0bafee740 20 client.9633 populate_metadata read hostname 'smithi028' 2024-02-21T23:11:00.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.773+0000 7fc0bafee740 1 --2- 172.21.15.28:0/1899947953 >> [v2:172.21.15.28:6832/1850463609,v1:172.21.15.28:6833/1850463609] conn(0x560c57507cf0 0x560c5750a360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:00.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.773+0000 7fc0bafee740 4 client.9633 mds_command: new command op to 9480 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-21T23:11:00.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.773+0000 7fc0bafee740 1 -- 172.21.15.28:0/1899947953 --> [v2:172.21.15.28:6832/1850463609,v1:172.21.15.28:6833/1850463609] -- command(tid 0: {"prefix": "status"}) v1 -- 0x560c573cad00 con 0x560c57507cf0 2024-02-21T23:11:00.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.774+0000 7fc0b5600640 1 --2- 172.21.15.28:0/1899947953 >> [v2:172.21.15.28:6832/1850463609,v1:172.21.15.28:6833/1850463609] conn(0x560c57507cf0 0x560c5750a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:00.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.774+0000 7fc0b5600640 1 --2- 172.21.15.28:0/1899947953 >> [v2:172.21.15.28:6832/1850463609,v1:172.21.15.28:6833/1850463609] conn(0x560c57507cf0 0x560c5750a360 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a2446a3dea41cf0 server_cookie=19f6670417f12f70 in_seq=0 out_seq=0 2024-02-21T23:11:00.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.774+0000 7fc096000640 10 client.9633 ms_handle_connect on v2:172.21.15.28:6832/1850463609 2024-02-21T23:11:00.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.774+0000 7fc096000640 1 -- 172.21.15.28:0/1899947953 <== mds.0 v2:172.21.15.28:6832/1850463609 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+334 (crc 0 0 0) 0x560c573cad00 con 0x560c57507cf0 2024-02-21T23:11:00.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.774+0000 7fc096000640 10 client.9633 handle_command_reply: tid=0 2024-02-21T23:11:00.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.774+0000 7fc0bafee740 1 client.9633 shutdown 2024-02-21T23:11:00.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.775+0000 7fc0bafee740 1 -- 172.21.15.28:0/1899947953 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560c5749a150 msgr2=0x560c5749f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:00.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.775+0000 7fc0bafee740 1 --2- 172.21.15.28:0/1899947953 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560c5749a150 0x560c5749f030 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7fc09c018000 tx=0x7fc09c06e040 comp rx=0 tx=0).stop 2024-02-21T23:11:00.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.775+0000 7fc0bafee740 1 -- 172.21.15.28:0/1899947953 shutdown_connections 2024-02-21T23:11:00.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.775+0000 7fc0bafee740 1 --2- 172.21.15.28:0/1899947953 >> [v2:172.21.15.28:6832/1850463609,v1:172.21.15.28:6833/1850463609] conn(0x560c57507cf0 0x560c5750a360 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:00.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.775+0000 7fc0bafee740 1 --2- 172.21.15.28:0/1899947953 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x560c5749b6b0 0x560c575077b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:00.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.775+0000 7fc0bafee740 1 --2- 172.21.15.28:0/1899947953 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560c5749a150 0x560c5749f030 unknown :-1 s=CLOSED pgs=1939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:00.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.775+0000 7fc0bafee740 1 --2- 172.21.15.28:0/1899947953 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x560c57460330 0x560c57455110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:00.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.775+0000 7fc096000640 0 client.9633 ms_handle_reset on v2:172.21.15.28:6832/1850463609 2024-02-21T23:11:00.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.775+0000 7fc0bafee740 1 -- 172.21.15.28:0/1899947953 >> 172.21.15.28:0/1899947953 conn(0x560c5745f020 msgr2=0x560c5749da40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:00.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.775+0000 7fc0bafee740 1 -- 172.21.15.28:0/1899947953 shutdown_connections 2024-02-21T23:11:00.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.775+0000 7fc0bafee740 1 -- 172.21.15.28:0/1899947953 wait complete. 2024-02-21T23:11:00.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.775+0000 7fc0bafee740 20 client.9633 trim_cache size 0 max 16384 2024-02-21T23:11:00.769 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-21T23:11:00.769 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac", 2024-02-21T23:11:00.769 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-02-21T23:11:00.769 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 9480, 2024-02-21T23:11:00.769 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-02-21T23:11:00.769 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-02-21T23:11:00.769 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-02-21T23:11:00.770 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 6.8686399270000003, 2024-02-21T23:11:00.770 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 185, 2024-02-21T23:11:00.770 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 174, 2024-02-21T23:11:00.770 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 174, 2024-02-21T23:11:00.770 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 12.388543848999999 2024-02-21T23:11:00.770 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-21T23:11:00.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.776+0000 7fc0b6a00640 1 -- 172.21.15.28:0/2599789225 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc09006b4d0 msgr2=0x7fc09006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:00.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.776+0000 7fc0b6a00640 1 --2- 172.21.15.28:0/2599789225 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc09006b4d0 0x7fc09006d990 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fc0a4036040 tx=0x7fc0a402e000 comp rx=0 tx=0).stop 2024-02-21T23:11:00.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.776+0000 7fc0b6a00640 1 -- 172.21.15.28:0/2599789225 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0b00c85c0 msgr2=0x7fc0b0158fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:00.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.776+0000 7fc0b6a00640 1 --2- 172.21.15.28:0/2599789225 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0b00c85c0 0x7fc0b0158fb0 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7fc09c00e840 tx=0x7fc09c00ed00 comp rx=0 tx=0).stop 2024-02-21T23:11:00.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.776+0000 7fc0b6a00640 1 -- 172.21.15.28:0/2599789225 shutdown_connections 2024-02-21T23:11:00.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.776+0000 7fc0b6a00640 1 --2- 172.21.15.28:0/2599789225 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc09006b4d0 0x7fc09006d990 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:00.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.776+0000 7fc0b6a00640 1 --2- 172.21.15.28:0/2599789225 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0b01524c0 0x7fc0b0153dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:00.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.776+0000 7fc0b6a00640 1 --2- 172.21.15.28:0/2599789225 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0b00c85c0 0x7fc0b0158fb0 unknown :-1 s=CLOSED pgs=1935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:00.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.776+0000 7fc0b6a00640 1 --2- 172.21.15.28:0/2599789225 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc0b00c5040 0x7fc0b00aa090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:00.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.776+0000 7fc0b6a00640 1 -- 172.21.15.28:0/2599789225 >> 172.21.15.28:0/2599789225 conn(0x7fc0b00ac2b0 msgr2=0x7fc0b0150590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:00.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.776+0000 7fc0b6a00640 1 -- 172.21.15.28:0/2599789225 shutdown_connections 2024-02-21T23:11:00.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:00.776+0000 7fc0b6a00640 1 -- 172.21.15.28:0/2599789225 wait complete. 2024-02-21T23:11:00.782 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs down true 2024-02-21T23:11:01.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:01.016+0000 7f7ed3e00640 1 Processor -- start 2024-02-21T23:11:01.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:01.017+0000 7f7ed3e00640 1 -- start start 2024-02-21T23:11:01.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:01.020+0000 7f7ed3e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7ed40c4610 0x7f7ed41550e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:01.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:01.020+0000 7f7ed3e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ed40c4a90 con 0x7f7ed414f0b0 2024-02-21T23:11:01.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:01.020+0000 7f7ed2a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f7ed414f0b0 0x7f7ed40c3f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55732/0 (socket says 172.21.15.28:55732) 2024-02-21T23:11:01.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:01.020+0000 7f7ed2a00640 1 -- 172.21.15.28:0/411462319 learned_addr learned my addr 172.21.15.28:0/411462319 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:11:01.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:01.020+0000 7f7ed2000640 1 --2- 172.21.15.28:0/411462319 >> v2:172.21.15.28:3300/0 conn(0x7f7ed40c4610 0x7f7ed41550e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:01.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:01.020+0000 7f7ed3e00640 1 -- 172.21.15.28:0/411462319 --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ed406fdf0 con 0x7f7ed4155620 2024-02-21T23:11:01.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:01.020+0000 7f7ed3e00640 1 -- 172.21.15.28:0/411462319 --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ed40c31f0 con 0x7f7ed40c4610 2024-02-21T23:11:01.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:01.021+0000 7f7ed1600640 1 -- 172.21.15.28:0/411462319 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2918955020 0 0) 0x7f7ed40c4a90 con 0x7f7ed414f0b0 2024-02-21T23:11:01.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:01.021+0000 7f7ed1600640 1 -- 172.21.15.28:0/411462319 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7eb8003520 con 0x7f7ed414f0b0 2024-02-21T23:11:01.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:01.021+0000 7f7ed2000640 1 -- 172.21.15.28:0/411462319 >> v1:172.21.15.81:6789/0 conn(0x7f7ed4155620 legacy=0x7f7ed4155a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:01.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:01.021+0000 7f7ed2000640 1 -- 172.21.15.28:0/411462319 >> v1:172.21.15.28:6789/0 conn(0x7f7ed414f0b0 legacy=0x7f7ed40c3f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:01.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:01.021+0000 7f7ed2000640 1 -- 172.21.15.28:0/411462319 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ed40c3420 con 0x7f7ed40c4610 2024-02-21T23:11:01.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:01.022+0000 7f7ed2000640 1 --2- 172.21.15.28:0/411462319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ed40c4610 0x7f7ed41550e0 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc0097a0 tx=0x7f7ebc02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bde270df4339cf12 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:01.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:01.022+0000 7f7ed1600640 1 -- 172.21.15.28:0/411462319 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ebc034070 con 0x7f7ed40c4610 2024-02-21T23:11:01.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:01.023+0000 7f7ed1600640 1 -- 172.21.15.28:0/411462319 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ebc030610 con 0x7f7ed40c4610 2024-02-21T23:11:01.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:01.023+0000 7f7ed1600640 1 -- 172.21.15.28:0/411462319 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ebc0309e0 con 0x7f7ed40c4610 2024-02-21T23:11:01.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:01.023+0000 7f7ed3e00640 1 -- 172.21.15.28:0/411462319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ed40c4610 msgr2=0x7f7ed41550e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:01.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:01.023+0000 7f7ed3e00640 1 --2- 172.21.15.28:0/411462319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ed40c4610 0x7f7ed41550e0 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc0097a0 tx=0x7f7ebc02b960 comp rx=0 tx=0).stop 2024-02-21T23:11:01.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:01.023+0000 7f7ed3e00640 1 -- 172.21.15.28:0/411462319 shutdown_connections 2024-02-21T23:11:01.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:01.023+0000 7f7ed3e00640 1 --2- 172.21.15.28:0/411462319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ed40c4610 0x7f7ed41550e0 unknown :-1 s=CLOSED pgs=1941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:01.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:01.023+0000 7f7ed3e00640 1 -- 172.21.15.28:0/411462319 >> 172.21.15.28:0/411462319 conn(0x7f7ed4067050 msgr2=0x7f7ed406bf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:01.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:01.023+0000 7f7ed3e00640 1 -- 172.21.15.28:0/411462319 shutdown_connections 2024-02-21T23:11:01.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:01.023+0000 7f7ed3e00640 1 -- 172.21.15.28:0/411462319 wait complete. 2024-02-21T23:11:01.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:01.025+0000 7f7ed3e00640 1 Processor -- start 2024-02-21T23:11:01.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:01.025+0000 7f7ed3e00640 1 -- start start 2024-02-21T23:11:01.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:01.026+0000 7f7ed3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ed40c4610 0x7f7ed41dcf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:01.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:01.026+0000 7f7ed3e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7ed414f0b0 0x7f7ed4151a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:01.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:01.026+0000 7f7ed3e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ed4155620 0x7f7ed41eae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:01.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:01.026+0000 7f7ed3e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ed414f490 con 0x7f7ed40c4610 2024-02-21T23:11:01.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:01.026+0000 7f7ed3e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ed40c3a70 con 0x7f7ed4155620 2024-02-21T23:11:01.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:01.026+0000 7f7ed3e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ed406fa80 con 0x7f7ed414f0b0 2024-02-21T23:11:01.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:01.026+0000 7f7ed2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ed40c4610 0x7f7ed41dcf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:01.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:01.026+0000 7f7ed2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ed40c4610 0x7f7ed41dcf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33154/0 (socket says 172.21.15.28:33154) 2024-02-21T23:11:01.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:01.026+0000 7f7ed2a00640 1 -- 172.21.15.28:0/2879955825 learned_addr learned my addr 172.21.15.28:0/2879955825 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:01.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:01.027+0000 7f7ed3400640 1 --2- 172.21.15.28:0/2879955825 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ed4155620 0x7f7ed41eae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:01.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:01.027+0000 7f7ed2000640 1 --2- 172.21.15.28:0/2879955825 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7ed414f0b0 0x7f7ed4151a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:01.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:01.027+0000 7f7ed2a00640 1 -- 172.21.15.28:0/2879955825 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7ed414f0b0 msgr2=0x7f7ed4151a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:01.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:01.027+0000 7f7ed2a00640 1 --2- 172.21.15.28:0/2879955825 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7ed414f0b0 0x7f7ed4151a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:01.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:01.027+0000 7f7ed2a00640 1 -- 172.21.15.28:0/2879955825 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ed4155620 msgr2=0x7f7ed41eae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:01.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:01.027+0000 7f7ed2a00640 1 --2- 172.21.15.28:0/2879955825 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ed4155620 0x7f7ed41eae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:01.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:01.027+0000 7f7ed2a00640 1 -- 172.21.15.28:0/2879955825 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ed41dbe00 con 0x7f7ed40c4610 2024-02-21T23:11:01.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:01.027+0000 7f7ed2a00640 1 --2- 172.21.15.28:0/2879955825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ed40c4610 0x7f7ed41dcf40 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc00cbe0 tx=0x7f7ecc00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:01.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:01.027+0000 7f7ecaa00640 1 -- 172.21.15.28:0/2879955825 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ecc017070 con 0x7f7ed40c4610 2024-02-21T23:11:01.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:01.028+0000 7f7ecaa00640 1 -- 172.21.15.28:0/2879955825 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ecc00ad40 con 0x7f7ed40c4610 2024-02-21T23:11:01.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:01.028+0000 7f7ecaa00640 1 -- 172.21.15.28:0/2879955825 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ecc012600 con 0x7f7ed40c4610 2024-02-21T23:11:01.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:01.028+0000 7f7ed3e00640 1 -- 172.21.15.28:0/2879955825 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ed41dc0e0 con 0x7f7ed40c4610 2024-02-21T23:11:01.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:01.028+0000 7f7ed3e00640 1 -- 172.21.15.28:0/2879955825 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ed4151f80 con 0x7f7ed40c4610 2024-02-21T23:11:01.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:01.029+0000 7f7ed3e00640 1 -- 172.21.15.28:0/2879955825 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ed404f290 con 0x7f7ed40c4610 2024-02-21T23:11:01.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:01.030+0000 7f7ecaa00640 1 -- 172.21.15.28:0/2879955825 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7ecc01a070 con 0x7f7ed40c4610 2024-02-21T23:11:01.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:01.031+0000 7f7ecaa00640 1 --2- 172.21.15.28:0/2879955825 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7eb006b500 0x7f7eb006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:01.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:01.031+0000 7f7ecaa00640 1 -- 172.21.15.28:0/2879955825 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f7ecc08a0c0 con 0x7f7ed40c4610 2024-02-21T23:11:01.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:01.032+0000 7f7ed2000640 1 --2- 172.21.15.28:0/2879955825 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7eb006b500 0x7f7eb006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:01.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:01.033+0000 7f7ed2000640 1 --2- 172.21.15.28:0/2879955825 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7eb006b500 0x7f7eb006d9c0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc036000 tx=0x7f7ebc03b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:01.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:01.040+0000 7f7ecaa00640 1 -- 172.21.15.28:0/2879955825 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7ed404f290 con 0x7f7ed40c4610 2024-02-21T23:11:01.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:01.286+0000 7f7ed3e00640 1 -- 172.21.15.28:0/2879955825 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "down", "val": "true"} v 0) v1 -- 0x7f7ed4151380 con 0x7f7ed40c4610 2024-02-21T23:11:01.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:01.963+0000 7f7ecaa00640 1 -- 172.21.15.28:0/2879955825 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "down", "val": "true"}]=0 cephfs marked down. v186) v1 ==== 125+0+0 (secure 0 0 0) 0x7f7ecc058750 con 0x7f7ed40c4610 2024-02-21T23:11:01.951 INFO:teuthology.orchestra.run.smithi028.stderr:cephfs marked down. 2024-02-21T23:11:01.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:01.967+0000 7f7ed3e00640 1 -- 172.21.15.28:0/2879955825 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7eb006b500 msgr2=0x7f7eb006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:01.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:01.967+0000 7f7ed3e00640 1 --2- 172.21.15.28:0/2879955825 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7eb006b500 0x7f7eb006d9c0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc036000 tx=0x7f7ebc03b040 comp rx=0 tx=0).stop 2024-02-21T23:11:01.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:01.967+0000 7f7ed3e00640 1 -- 172.21.15.28:0/2879955825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ed40c4610 msgr2=0x7f7ed41dcf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:01.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:01.967+0000 7f7ed3e00640 1 --2- 172.21.15.28:0/2879955825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ed40c4610 0x7f7ed41dcf40 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc00cbe0 tx=0x7f7ecc00a430 comp rx=0 tx=0).stop 2024-02-21T23:11:01.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:01.967+0000 7f7ed3e00640 1 -- 172.21.15.28:0/2879955825 shutdown_connections 2024-02-21T23:11:01.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:01.967+0000 7f7ed3e00640 1 --2- 172.21.15.28:0/2879955825 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7eb006b500 0x7f7eb006d9c0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:01.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:01.967+0000 7f7ed3e00640 1 --2- 172.21.15.28:0/2879955825 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7ed414f0b0 0x7f7ed4151a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:01.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:01.967+0000 7f7ed3e00640 1 --2- 172.21.15.28:0/2879955825 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ed4155620 0x7f7ed41eae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:01.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:01.967+0000 7f7ed3e00640 1 --2- 172.21.15.28:0/2879955825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ed40c4610 0x7f7ed41dcf40 unknown :-1 s=CLOSED pgs=1942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:01.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:01.967+0000 7f7ed3e00640 1 -- 172.21.15.28:0/2879955825 >> 172.21.15.28:0/2879955825 conn(0x7f7ed4067050 msgr2=0x7f7ed40c3f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:01.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:01.967+0000 7f7ed3e00640 1 -- 172.21.15.28:0/2879955825 shutdown_connections 2024-02-21T23:11:01.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:01.967+0000 7f7ed3e00640 1 -- 172.21.15.28:0/2879955825 wait complete. 2024-02-21T23:11:01.977 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:11:02.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.202+0000 7faa36400640 1 Processor -- start 2024-02-21T23:11:02.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.202+0000 7faa36400640 1 -- start start 2024-02-21T23:11:02.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.203+0000 7faa36400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7faa300aff90 0x7faa300c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:02.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.203+0000 7faa36400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa300c74c0 con 0x7faa3006dbf0 2024-02-21T23:11:02.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.203+0000 7faa36400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa300c77a0 con 0x7faa300c5720 2024-02-21T23:11:02.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.203+0000 7faa36400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa300c7270 con 0x7faa300aff90 2024-02-21T23:11:02.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.203+0000 7faa35000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7faa300aff90 0x7faa300c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:02.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.203+0000 7faa27e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7faa3006dbf0 0x7faa30158ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55738/0 (socket says 172.21.15.28:55738) 2024-02-21T23:11:02.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.203+0000 7faa35000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7faa300aff90 0x7faa300c51e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33164/0 (socket says 172.21.15.28:33164) 2024-02-21T23:11:02.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.203+0000 7faa27e00640 1 -- 172.21.15.28:0/3723471335 learned_addr learned my addr 172.21.15.28:0/3723471335 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:11:02.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.204+0000 7faa2fe00640 1 -- 172.21.15.28:0/3723471335 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3965603150 0 0) 0x7faa300c74c0 con 0x7faa3006dbf0 2024-02-21T23:11:02.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.204+0000 7faa2fe00640 1 -- 172.21.15.28:0/3723471335 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa10003520 con 0x7faa3006dbf0 2024-02-21T23:11:02.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.204+0000 7faa35000640 1 -- 172.21.15.28:0/3723471335 >> v1:172.21.15.81:6789/0 conn(0x7faa300c5720 legacy=0x7faa3015f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:02.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.204+0000 7faa35000640 1 -- 172.21.15.28:0/3723471335 >> v1:172.21.15.28:6789/0 conn(0x7faa3006dbf0 legacy=0x7faa30158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:02.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.204+0000 7faa35000640 1 -- 172.21.15.28:0/3723471335 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa300c7a80 con 0x7faa300aff90 2024-02-21T23:11:02.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.205+0000 7faa35000640 1 --2- 172.21.15.28:0/3723471335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa300aff90 0x7faa300c51e0 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7faa180097a0 tx=0x7faa1802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f9d8a1da1385d4a server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:02.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.205+0000 7faa2fe00640 1 -- 172.21.15.28:0/3723471335 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa18034070 con 0x7faa300aff90 2024-02-21T23:11:02.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.206+0000 7faa2fe00640 1 -- 172.21.15.28:0/3723471335 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa18030610 con 0x7faa300aff90 2024-02-21T23:11:02.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.206+0000 7faa2fe00640 1 -- 172.21.15.28:0/3723471335 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa180309c0 con 0x7faa300aff90 2024-02-21T23:11:02.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.206+0000 7faa36400640 1 -- 172.21.15.28:0/3723471335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa300aff90 msgr2=0x7faa300c51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:02.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.206+0000 7faa36400640 1 --2- 172.21.15.28:0/3723471335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa300aff90 0x7faa300c51e0 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7faa180097a0 tx=0x7faa1802b960 comp rx=0 tx=0).stop 2024-02-21T23:11:02.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.206+0000 7faa36400640 1 -- 172.21.15.28:0/3723471335 shutdown_connections 2024-02-21T23:11:02.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.206+0000 7faa36400640 1 --2- 172.21.15.28:0/3723471335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa300aff90 0x7faa300c51e0 unknown :-1 s=CLOSED pgs=1944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:02.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.206+0000 7faa36400640 1 -- 172.21.15.28:0/3723471335 >> 172.21.15.28:0/3723471335 conn(0x7faa300ac370 msgr2=0x7faa300ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:02.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.206+0000 7faa36400640 1 -- 172.21.15.28:0/3723471335 shutdown_connections 2024-02-21T23:11:02.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.206+0000 7faa36400640 1 -- 172.21.15.28:0/3723471335 wait complete. 2024-02-21T23:11:02.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.208+0000 7faa36400640 1 Processor -- start 2024-02-21T23:11:02.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.208+0000 7faa36400640 1 -- start start 2024-02-21T23:11:02.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.209+0000 7faa36400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faa3006dbf0 0x7faa300be840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:02.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.209+0000 7faa36400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faa300aff90 0x7faa301eb330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:02.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.209+0000 7faa36400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa300c5720 0x7faa30154710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:02.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.209+0000 7faa36400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa300c8760 con 0x7faa300c5720 2024-02-21T23:11:02.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.209+0000 7faa36400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa30158630 con 0x7faa3006dbf0 2024-02-21T23:11:02.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.209+0000 7faa36400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa300c88e0 con 0x7faa300aff90 2024-02-21T23:11:02.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.209+0000 7faa35a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa300c5720 0x7faa30154710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:02.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.209+0000 7faa35a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa300c5720 0x7faa30154710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33170/0 (socket says 172.21.15.28:33170) 2024-02-21T23:11:02.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.209+0000 7faa35a00640 1 -- 172.21.15.28:0/726160490 learned_addr learned my addr 172.21.15.28:0/726160490 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:02.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.210+0000 7faa35000640 1 --2- 172.21.15.28:0/726160490 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faa3006dbf0 0x7faa300be840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:02.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.210+0000 7faa27e00640 1 --2- 172.21.15.28:0/726160490 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faa300aff90 0x7faa301eb330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:02.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.210+0000 7faa35a00640 1 -- 172.21.15.28:0/726160490 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faa300aff90 msgr2=0x7faa301eb330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:02.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.210+0000 7faa35a00640 1 --2- 172.21.15.28:0/726160490 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faa300aff90 0x7faa301eb330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:02.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.210+0000 7faa35a00640 1 -- 172.21.15.28:0/726160490 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faa3006dbf0 msgr2=0x7faa300be840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:02.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.210+0000 7faa35a00640 1 --2- 172.21.15.28:0/726160490 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faa3006dbf0 0x7faa300be840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:02.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.210+0000 7faa35a00640 1 -- 172.21.15.28:0/726160490 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa301eb870 con 0x7faa300c5720 2024-02-21T23:11:02.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.210+0000 7faa27e00640 1 --2- 172.21.15.28:0/726160490 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faa300aff90 0x7faa301eb330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:11:02.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.210+0000 7faa35a00640 1 --2- 172.21.15.28:0/726160490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa300c5720 0x7faa30154710 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7faa2800c6b0 tx=0x7faa2800cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:02.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.210+0000 7faa2d600640 1 -- 172.21.15.28:0/726160490 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa28003b70 con 0x7faa300c5720 2024-02-21T23:11:02.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.211+0000 7faa2d600640 1 -- 172.21.15.28:0/726160490 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa28003d30 con 0x7faa300c5720 2024-02-21T23:11:02.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.211+0000 7faa2d600640 1 -- 172.21.15.28:0/726160490 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa28005310 con 0x7faa300c5720 2024-02-21T23:11:02.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.211+0000 7faa36400640 1 -- 172.21.15.28:0/726160490 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa300c3e50 con 0x7faa300c5720 2024-02-21T23:11:02.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.211+0000 7faa36400640 1 -- 172.21.15.28:0/726160490 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa300c4410 con 0x7faa300c5720 2024-02-21T23:11:02.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.212+0000 7faa36400640 1 -- 172.21.15.28:0/726160490 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa3004f290 con 0x7faa300c5720 2024-02-21T23:11:02.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.213+0000 7faa2d600640 1 -- 172.21.15.28:0/726160490 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faa280054b0 con 0x7faa300c5720 2024-02-21T23:11:02.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.214+0000 7faa2d600640 1 --2- 172.21.15.28:0/726160490 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7faa0c06b500 0x7faa0c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:02.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.214+0000 7faa2d600640 1 -- 172.21.15.28:0/726160490 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6101+0+0 (secure 0 0 0) 0x7faa2808a1c0 con 0x7faa300c5720 2024-02-21T23:11:02.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.215+0000 7faa35000640 1 --2- 172.21.15.28:0/726160490 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7faa0c06b500 0x7faa0c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:02.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.216+0000 7faa35000640 1 --2- 172.21.15.28:0/726160490 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7faa0c06b500 0x7faa0c06d9c0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7faa18036000 tx=0x7faa18008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:02.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.218+0000 7faa2d600640 1 -- 172.21.15.28:0/726160490 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7faa301eb870 con 0x7faa300c5720 2024-02-21T23:11:02.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.457+0000 7faa36400640 1 -- 172.21.15.28:0/726160490 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faa30158380 con 0x7faa300c5720 2024-02-21T23:11:02.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.459+0000 7faa2d600640 1 -- 172.21.15.28:0/726160490 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 187 v187) v1 ==== 95+0+4947 (secure 0 0 0) 0x7faa28058ea0 con 0x7faa300c5720 2024-02-21T23:11:02.448 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:11:02.448 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":187,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6686,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/155806034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":155806034},{"type":"v1","addr":"172.21.15.81:6836","nonce":155806034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":180},{"gid":7480,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3897243745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":3897243745},{"type":"v1","addr":"172.21.15.81:6837","nonce":3897243745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":180},{"gid":7486,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2827135632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2827135632},{"type":"v1","addr":"172.21.15.28:6835","nonce":2827135632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":180}],"filesystems":[{"mdsmap":{"epoch":187,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:10:53.351225+0000","modified":"2024-02-21T23:11:01.967539+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0],"up":{"mds_0":9480},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9480":{"gid":9480,"name":"c","rank":0,"incarnation":181,"state":"up:stopping","state_seq":3,"addr":"172.21.15.28:6833/1850463609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1850463609},{"type":"v1","addr":"172.21.15.28:6833","nonce":1850463609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2024-02-21T23:11:02.450 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 187 2024-02-21T23:11:02.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.461+0000 7faa36400640 1 -- 172.21.15.28:0/726160490 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7faa0c06b500 msgr2=0x7faa0c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:02.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.461+0000 7faa36400640 1 --2- 172.21.15.28:0/726160490 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7faa0c06b500 0x7faa0c06d9c0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7faa18036000 tx=0x7faa18008030 comp rx=0 tx=0).stop 2024-02-21T23:11:02.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.461+0000 7faa36400640 1 -- 172.21.15.28:0/726160490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa300c5720 msgr2=0x7faa30154710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:02.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.461+0000 7faa36400640 1 --2- 172.21.15.28:0/726160490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa300c5720 0x7faa30154710 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7faa2800c6b0 tx=0x7faa2800cb70 comp rx=0 tx=0).stop 2024-02-21T23:11:02.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.461+0000 7faa36400640 1 -- 172.21.15.28:0/726160490 shutdown_connections 2024-02-21T23:11:02.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.461+0000 7faa36400640 1 --2- 172.21.15.28:0/726160490 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7faa0c06b500 0x7faa0c06d9c0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:02.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.461+0000 7faa36400640 1 --2- 172.21.15.28:0/726160490 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faa300aff90 0x7faa301eb330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:02.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.461+0000 7faa36400640 1 --2- 172.21.15.28:0/726160490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa300c5720 0x7faa30154710 unknown :-1 s=CLOSED pgs=1945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:02.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.461+0000 7faa36400640 1 --2- 172.21.15.28:0/726160490 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faa3006dbf0 0x7faa300be840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:02.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.461+0000 7faa36400640 1 -- 172.21.15.28:0/726160490 >> 172.21.15.28:0/726160490 conn(0x7faa300ac370 msgr2=0x7faa300c8ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:02.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.462+0000 7faa36400640 1 -- 172.21.15.28:0/726160490 shutdown_connections 2024-02-21T23:11:02.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:02.462+0000 7faa36400640 1 -- 172.21.15.28:0/726160490 wait complete. 2024-02-21T23:11:02.872 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 187, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:10:53.351225+0000', 'modified': '2024-02-21T23:11:01.967539+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 9480}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9480': {'gid': 9480, 'name': 'c', 'rank': 0, 'incarnation': 181, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.28:6833/1850463609', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 1850463609}, {'type': 'v1', 'addr': '172.21.15.28:6833', 'nonce': 1850463609}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:11:02.872 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9480:up:stopping 2024-02-21T23:11:03.874 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:11:04.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:04.115+0000 7f0b67e00640 1 Processor -- start 2024-02-21T23:11:04.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:04.115+0000 7f0b67e00640 1 -- start start 2024-02-21T23:11:04.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:04.117+0000 7f0b67e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0b68150450 0x7f0b680a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:04.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:04.117+0000 7f0b67e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b680c2760 con 0x7f0b68156af0 2024-02-21T23:11:04.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:04.117+0000 7f0b67e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b68156310 con 0x7f0b6806ee40 2024-02-21T23:11:04.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:04.117+0000 7f0b67e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b680c22c0 con 0x7f0b68150450 2024-02-21T23:11:04.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:04.117+0000 7f0b66a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0b68150450 0x7f0b680a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:04.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:04.117+0000 7f0b66a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0b68150450 0x7f0b680a5fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33174/0 (socket says 172.21.15.28:33174) 2024-02-21T23:11:04.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:04.117+0000 7f0b66a00640 1 -- 172.21.15.28:0/2077063473 learned_addr learned my addr 172.21.15.28:0/2077063473 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:04.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:04.118+0000 7f0b66a00640 1 -- 172.21.15.28:0/2077063473 >> v1:172.21.15.81:6789/0 conn(0x7f0b6806ee40 legacy=0x7f0b681536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:04.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:04.118+0000 7f0b66a00640 1 -- 172.21.15.28:0/2077063473 >> v1:172.21.15.28:6789/0 conn(0x7f0b68156af0 legacy=0x7f0b6806dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:04.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:04.118+0000 7f0b66a00640 1 -- 172.21.15.28:0/2077063473 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b680c29d0 con 0x7f0b68150450 2024-02-21T23:11:04.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:04.119+0000 7f0b66a00640 1 --2- 172.21.15.28:0/2077063473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b68150450 0x7f0b680a5fd0 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7f0b58009870 tx=0x7f0b5802b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ef93995bc832b41 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:04.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:04.119+0000 7f0b66000640 1 -- 172.21.15.28:0/2077063473 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b58034070 con 0x7f0b68150450 2024-02-21T23:11:04.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:04.119+0000 7f0b66000640 1 -- 172.21.15.28:0/2077063473 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b58030850 con 0x7f0b68150450 2024-02-21T23:11:04.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:04.119+0000 7f0b66000640 1 -- 172.21.15.28:0/2077063473 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b58030c20 con 0x7f0b68150450 2024-02-21T23:11:04.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:04.120+0000 7f0b67e00640 1 -- 172.21.15.28:0/2077063473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b68150450 msgr2=0x7f0b680a5fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:04.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:04.120+0000 7f0b67e00640 1 --2- 172.21.15.28:0/2077063473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b68150450 0x7f0b680a5fd0 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7f0b58009870 tx=0x7f0b5802b990 comp rx=0 tx=0).stop 2024-02-21T23:11:04.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:04.120+0000 7f0b67e00640 1 -- 172.21.15.28:0/2077063473 shutdown_connections 2024-02-21T23:11:04.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:04.120+0000 7f0b67e00640 1 --2- 172.21.15.28:0/2077063473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b68150450 0x7f0b680a5fd0 unknown :-1 s=CLOSED pgs=1947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:04.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:04.120+0000 7f0b67e00640 1 -- 172.21.15.28:0/2077063473 >> 172.21.15.28:0/2077063473 conn(0x7f0b6806b630 msgr2=0x7f0b680c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:04.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:04.120+0000 7f0b67e00640 1 -- 172.21.15.28:0/2077063473 shutdown_connections 2024-02-21T23:11:04.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:04.120+0000 7f0b67e00640 1 -- 172.21.15.28:0/2077063473 wait complete. 2024-02-21T23:11:04.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:04.122+0000 7f0b67e00640 1 Processor -- start 2024-02-21T23:11:04.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:04.122+0000 7f0b67e00640 1 -- start start 2024-02-21T23:11:04.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:04.123+0000 7f0b67e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0b6806ee40 0x7f0b68146510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:04.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:04.123+0000 7f0b67e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b68150450 0x7f0b68146a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:04.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:04.123+0000 7f0b67e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0b68156af0 0x7f0b681ee640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:04.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:04.123+0000 7f0b67e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b6806e700 con 0x7f0b68150450 2024-02-21T23:11:04.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:04.123+0000 7f0b67e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b68156660 con 0x7f0b68156af0 2024-02-21T23:11:04.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:04.123+0000 7f0b67e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b6806e880 con 0x7f0b6806ee40 2024-02-21T23:11:04.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:04.123+0000 7f0b5fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b68150450 0x7f0b68146a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:04.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:04.123+0000 7f0b5fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b68150450 0x7f0b68146a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33186/0 (socket says 172.21.15.28:33186) 2024-02-21T23:11:04.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:04.123+0000 7f0b5fe00640 1 -- 172.21.15.28:0/482086585 learned_addr learned my addr 172.21.15.28:0/482086585 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:04.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:04.123+0000 7f0b66a00640 1 --2- 172.21.15.28:0/482086585 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0b6806ee40 0x7f0b68146510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:04.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:04.123+0000 7f0b67400640 1 --2- 172.21.15.28:0/482086585 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0b68156af0 0x7f0b681ee640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:04.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:04.124+0000 7f0b5fe00640 1 -- 172.21.15.28:0/482086585 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0b6806ee40 msgr2=0x7f0b68146510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:04.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:04.124+0000 7f0b5fe00640 1 --2- 172.21.15.28:0/482086585 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0b6806ee40 0x7f0b68146510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:04.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:04.124+0000 7f0b5fe00640 1 -- 172.21.15.28:0/482086585 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0b68156af0 msgr2=0x7f0b681ee640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:04.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:04.124+0000 7f0b5fe00640 1 --2- 172.21.15.28:0/482086585 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0b68156af0 0x7f0b681ee640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:04.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:04.124+0000 7f0b5fe00640 1 -- 172.21.15.28:0/482086585 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b681549f0 con 0x7f0b68150450 2024-02-21T23:11:04.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:04.124+0000 7f0b5fe00640 1 --2- 172.21.15.28:0/482086585 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b68150450 0x7f0b68146a50 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7f0b5000c430 tx=0x7f0b5000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:04.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:04.124+0000 7f0b5f400640 1 -- 172.21.15.28:0/482086585 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b50011790 con 0x7f0b68150450 2024-02-21T23:11:04.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:04.124+0000 7f0b5f400640 1 -- 172.21.15.28:0/482086585 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b50017070 con 0x7f0b68150450 2024-02-21T23:11:04.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:04.124+0000 7f0b67e00640 1 -- 172.21.15.28:0/482086585 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b68150fd0 con 0x7f0b68150450 2024-02-21T23:11:04.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:04.124+0000 7f0b5f400640 1 -- 172.21.15.28:0/482086585 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b50011bb0 con 0x7f0b68150450 2024-02-21T23:11:04.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:04.125+0000 7f0b67e00640 1 -- 172.21.15.28:0/482086585 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b68150cc0 con 0x7f0b68150450 2024-02-21T23:11:04.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:04.126+0000 7f0b67e00640 1 -- 172.21.15.28:0/482086585 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b6804f290 con 0x7f0b68150450 2024-02-21T23:11:04.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:04.127+0000 7f0b5f400640 1 -- 172.21.15.28:0/482086585 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0b5001c460 con 0x7f0b68150450 2024-02-21T23:11:04.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:04.127+0000 7f0b5f400640 1 --2- 172.21.15.28:0/482086585 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0b4406b4d0 0x7f0b4406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:04.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:04.128+0000 7f0b5f400640 1 -- 172.21.15.28:0/482086585 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f0b5008a290 con 0x7f0b68150450 2024-02-21T23:11:04.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:04.128+0000 7f0b66a00640 1 --2- 172.21.15.28:0/482086585 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0b4406b4d0 0x7f0b4406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:04.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:04.129+0000 7f0b66a00640 1 --2- 172.21.15.28:0/482086585 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0b4406b4d0 0x7f0b4406d990 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f0b58036000 tx=0x7f0b5803b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:04.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:04.132+0000 7f0b5f400640 1 -- 172.21.15.28:0/482086585 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f0b50058f70 con 0x7f0b68150450 2024-02-21T23:11:04.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:04.373+0000 7f0b67e00640 1 -- 172.21.15.28:0/482086585 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b680a6720 con 0x7f0b68150450 2024-02-21T23:11:04.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:04.374+0000 7f0b5f400640 1 -- 172.21.15.28:0/482086585 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 187 v187) v1 ==== 95+0+4947 (secure 0 0 0) 0x7f0b50058870 con 0x7f0b68150450 2024-02-21T23:11:04.362 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:11:04.362 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":187,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6686,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/155806034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":155806034},{"type":"v1","addr":"172.21.15.81:6836","nonce":155806034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":180},{"gid":7480,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3897243745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":3897243745},{"type":"v1","addr":"172.21.15.81:6837","nonce":3897243745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":180},{"gid":7486,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2827135632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2827135632},{"type":"v1","addr":"172.21.15.28:6835","nonce":2827135632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":180}],"filesystems":[{"mdsmap":{"epoch":187,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:10:53.351225+0000","modified":"2024-02-21T23:11:01.967539+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0],"up":{"mds_0":9480},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9480":{"gid":9480,"name":"c","rank":0,"incarnation":181,"state":"up:stopping","state_seq":3,"addr":"172.21.15.28:6833/1850463609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1850463609},{"type":"v1","addr":"172.21.15.28:6833","nonce":1850463609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2024-02-21T23:11:04.363 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 187 2024-02-21T23:11:04.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:04.377+0000 7f0b67e00640 1 -- 172.21.15.28:0/482086585 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0b4406b4d0 msgr2=0x7f0b4406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:04.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:04.377+0000 7f0b67e00640 1 --2- 172.21.15.28:0/482086585 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0b4406b4d0 0x7f0b4406d990 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f0b58036000 tx=0x7f0b5803b040 comp rx=0 tx=0).stop 2024-02-21T23:11:04.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:04.377+0000 7f0b67e00640 1 -- 172.21.15.28:0/482086585 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b68150450 msgr2=0x7f0b68146a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:04.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:04.377+0000 7f0b67e00640 1 --2- 172.21.15.28:0/482086585 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b68150450 0x7f0b68146a50 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7f0b5000c430 tx=0x7f0b5000c8f0 comp rx=0 tx=0).stop 2024-02-21T23:11:04.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:04.377+0000 7f0b67e00640 1 -- 172.21.15.28:0/482086585 shutdown_connections 2024-02-21T23:11:04.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:04.377+0000 7f0b67e00640 1 --2- 172.21.15.28:0/482086585 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0b4406b4d0 0x7f0b4406d990 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:04.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:04.377+0000 7f0b67e00640 1 --2- 172.21.15.28:0/482086585 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0b68156af0 0x7f0b681ee640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:04.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:04.377+0000 7f0b67e00640 1 --2- 172.21.15.28:0/482086585 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b68150450 0x7f0b68146a50 unknown :-1 s=CLOSED pgs=1948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:04.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:04.377+0000 7f0b67e00640 1 --2- 172.21.15.28:0/482086585 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0b6806ee40 0x7f0b68146510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:04.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:04.377+0000 7f0b67e00640 1 -- 172.21.15.28:0/482086585 >> 172.21.15.28:0/482086585 conn(0x7f0b6806b630 msgr2=0x7f0b68141420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:04.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:04.377+0000 7f0b67e00640 1 -- 172.21.15.28:0/482086585 shutdown_connections 2024-02-21T23:11:04.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:04.378+0000 7f0b67e00640 1 -- 172.21.15.28:0/482086585 wait complete. 2024-02-21T23:11:04.787 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 187, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:10:53.351225+0000', 'modified': '2024-02-21T23:11:01.967539+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 9480}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9480': {'gid': 9480, 'name': 'c', 'rank': 0, 'incarnation': 181, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.28:6833/1850463609', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 1850463609}, {'type': 'v1', 'addr': '172.21.15.28:6833', 'nonce': 1850463609}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:11:04.787 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9480:up:stopping 2024-02-21T23:11:05.787 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:11:06.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:06.026+0000 7faa2bc00640 1 Processor -- start 2024-02-21T23:11:06.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:06.026+0000 7faa2bc00640 1 -- start start 2024-02-21T23:11:06.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:06.028+0000 7faa2bc00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7faa24153950 0x7faa240c15d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:06.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:06.028+0000 7faa2bc00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa24152e70 con 0x7faa240c11f0 2024-02-21T23:11:06.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:06.028+0000 7faa2bc00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa24153150 con 0x7faa2415b6d0 2024-02-21T23:11:06.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:06.028+0000 7faa2bc00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa240a5ec0 con 0x7faa24153950 2024-02-21T23:11:06.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:06.028+0000 7faa1be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7faa24153950 0x7faa240c15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:06.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:06.028+0000 7faa29200640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7faa240c11f0 0x7faa2406dce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:44024/0 (socket says 172.21.15.28:44024) 2024-02-21T23:11:06.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:06.028+0000 7faa29200640 1 -- 172.21.15.28:0/1187157467 learned_addr learned my addr 172.21.15.28:0/1187157467 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:11:06.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:06.029+0000 7faa1be00640 1 -- 172.21.15.28:0/1187157467 >> v1:172.21.15.81:6789/0 conn(0x7faa2415b6d0 legacy=0x7faa2415dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:06.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:06.029+0000 7faa1be00640 1 -- 172.21.15.28:0/1187157467 >> v1:172.21.15.28:6789/0 conn(0x7faa240c11f0 legacy=0x7faa2406dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:06.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:06.029+0000 7faa1be00640 1 -- 172.21.15.28:0/1187157467 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa24153430 con 0x7faa24153950 2024-02-21T23:11:06.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:06.030+0000 7faa1be00640 1 --2- 172.21.15.28:0/1187157467 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa24153950 0x7faa240c15d0 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7faa0c0097a0 tx=0x7faa0c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69acce78f99d4e9e server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:06.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:06.030+0000 7faa23e00640 1 -- 172.21.15.28:0/1187157467 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa0c0315a0 con 0x7faa24153950 2024-02-21T23:11:06.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:06.030+0000 7faa23e00640 1 -- 172.21.15.28:0/1187157467 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa0c037070 con 0x7faa24153950 2024-02-21T23:11:06.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:06.030+0000 7faa23e00640 1 -- 172.21.15.28:0/1187157467 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa0c031960 con 0x7faa24153950 2024-02-21T23:11:06.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:06.031+0000 7faa2bc00640 1 -- 172.21.15.28:0/1187157467 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa24153950 msgr2=0x7faa240c15d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:06.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:06.031+0000 7faa2bc00640 1 --2- 172.21.15.28:0/1187157467 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa24153950 0x7faa240c15d0 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7faa0c0097a0 tx=0x7faa0c02b840 comp rx=0 tx=0).stop 2024-02-21T23:11:06.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:06.031+0000 7faa2bc00640 1 -- 172.21.15.28:0/1187157467 shutdown_connections 2024-02-21T23:11:06.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:06.031+0000 7faa2bc00640 1 --2- 172.21.15.28:0/1187157467 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa24153950 0x7faa240c15d0 unknown :-1 s=CLOSED pgs=1950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:06.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:06.031+0000 7faa2bc00640 1 -- 172.21.15.28:0/1187157467 >> 172.21.15.28:0/1187157467 conn(0x7faa240c67f0 msgr2=0x7faa240c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:06.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:06.031+0000 7faa2bc00640 1 -- 172.21.15.28:0/1187157467 shutdown_connections 2024-02-21T23:11:06.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:06.031+0000 7faa2bc00640 1 -- 172.21.15.28:0/1187157467 wait complete. 2024-02-21T23:11:06.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:06.033+0000 7faa2bc00640 1 Processor -- start 2024-02-21T23:11:06.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:06.033+0000 7faa2bc00640 1 -- start start 2024-02-21T23:11:06.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:06.034+0000 7faa2bc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa240c11f0 0x7faa241e9840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:06.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:06.034+0000 7faa2bc00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faa24153950 0x7faa240c55e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:06.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:06.034+0000 7faa2bc00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faa2415b6d0 0x7faa241448f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:06.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:06.034+0000 7faa2bc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa240a6110 con 0x7faa240c11f0 2024-02-21T23:11:06.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:06.034+0000 7faa2bc00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa240c4c00 con 0x7faa24153950 2024-02-21T23:11:06.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:06.034+0000 7faa2bc00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa240c0d90 con 0x7faa2415b6d0 2024-02-21T23:11:06.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:06.034+0000 7faa29200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa240c11f0 0x7faa241e9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:06.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:06.034+0000 7faa29200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa240c11f0 0x7faa241e9840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59562/0 (socket says 172.21.15.28:59562) 2024-02-21T23:11:06.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:06.034+0000 7faa29200640 1 -- 172.21.15.28:0/2227189613 learned_addr learned my addr 172.21.15.28:0/2227189613 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:06.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:06.034+0000 7faa1be00640 1 --2- 172.21.15.28:0/2227189613 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faa24153950 0x7faa240c55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:06.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:06.034+0000 7faa29c00640 1 --2- 172.21.15.28:0/2227189613 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faa2415b6d0 0x7faa241448f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:06.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:06.034+0000 7faa29200640 1 -- 172.21.15.28:0/2227189613 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faa2415b6d0 msgr2=0x7faa241448f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:06.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:06.034+0000 7faa29200640 1 --2- 172.21.15.28:0/2227189613 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faa2415b6d0 0x7faa241448f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:06.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:06.035+0000 7faa29200640 1 -- 172.21.15.28:0/2227189613 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faa24153950 msgr2=0x7faa240c55e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:06.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:06.035+0000 7faa29200640 1 --2- 172.21.15.28:0/2227189613 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faa24153950 0x7faa240c55e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:06.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:06.035+0000 7faa29200640 1 -- 172.21.15.28:0/2227189613 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa240a9500 con 0x7faa240c11f0 2024-02-21T23:11:06.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:06.035+0000 7faa1be00640 1 --2- 172.21.15.28:0/2227189613 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faa24153950 0x7faa240c55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:11:06.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:06.035+0000 7faa29200640 1 --2- 172.21.15.28:0/2227189613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa240c11f0 0x7faa241e9840 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7faa1400c3c0 tx=0x7faa1400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:06.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:06.035+0000 7faa21600640 1 -- 172.21.15.28:0/2227189613 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa14017070 con 0x7faa240c11f0 2024-02-21T23:11:06.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:06.035+0000 7faa21600640 1 -- 172.21.15.28:0/2227189613 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa14005ce0 con 0x7faa240c11f0 2024-02-21T23:11:06.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:06.035+0000 7faa21600640 1 -- 172.21.15.28:0/2227189613 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa140072f0 con 0x7faa240c11f0 2024-02-21T23:11:06.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:06.035+0000 7faa2bc00640 1 -- 172.21.15.28:0/2227189613 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa240a8720 con 0x7faa240c11f0 2024-02-21T23:11:06.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:06.035+0000 7faa2bc00640 1 -- 172.21.15.28:0/2227189613 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa240c5b20 con 0x7faa240c11f0 2024-02-21T23:11:06.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:06.036+0000 7faa2bc00640 1 -- 172.21.15.28:0/2227189613 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa24067f30 con 0x7faa240c11f0 2024-02-21T23:11:06.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:06.037+0000 7faa21600640 1 -- 172.21.15.28:0/2227189613 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faa1401a070 con 0x7faa240c11f0 2024-02-21T23:11:06.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:06.039+0000 7faa21600640 1 --2- 172.21.15.28:0/2227189613 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7faa0006b4d0 0x7faa0006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:06.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:06.039+0000 7faa21600640 1 -- 172.21.15.28:0/2227189613 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6101+0+0 (secure 0 0 0) 0x7faa14089740 con 0x7faa240c11f0 2024-02-21T23:11:06.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:06.040+0000 7faa1be00640 1 --2- 172.21.15.28:0/2227189613 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7faa0006b4d0 0x7faa0006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:06.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:06.041+0000 7faa1be00640 1 --2- 172.21.15.28:0/2227189613 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7faa0006b4d0 0x7faa0006d990 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7faa0c036000 tx=0x7faa0c008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:06.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:06.044+0000 7faa21600640 1 -- 172.21.15.28:0/2227189613 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7faa14058420 con 0x7faa240c11f0 2024-02-21T23:11:06.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:06.281+0000 7faa2bc00640 1 -- 172.21.15.28:0/2227189613 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faa240c0960 con 0x7faa240c11f0 2024-02-21T23:11:06.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:06.283+0000 7faa21600640 1 -- 172.21.15.28:0/2227189613 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 187 v187) v1 ==== 95+0+4947 (secure 0 0 0) 0x7faa14057d20 con 0x7faa240c11f0 2024-02-21T23:11:06.271 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:11:06.271 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":187,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6686,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/155806034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":155806034},{"type":"v1","addr":"172.21.15.81:6836","nonce":155806034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":180},{"gid":7480,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3897243745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":3897243745},{"type":"v1","addr":"172.21.15.81:6837","nonce":3897243745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":180},{"gid":7486,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2827135632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2827135632},{"type":"v1","addr":"172.21.15.28:6835","nonce":2827135632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":180}],"filesystems":[{"mdsmap":{"epoch":187,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:10:53.351225+0000","modified":"2024-02-21T23:11:01.967539+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0],"up":{"mds_0":9480},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9480":{"gid":9480,"name":"c","rank":0,"incarnation":181,"state":"up:stopping","state_seq":3,"addr":"172.21.15.28:6833/1850463609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1850463609},{"type":"v1","addr":"172.21.15.28:6833","nonce":1850463609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2024-02-21T23:11:06.271 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 187 2024-02-21T23:11:06.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:06.285+0000 7faa2bc00640 1 -- 172.21.15.28:0/2227189613 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7faa0006b4d0 msgr2=0x7faa0006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:06.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:06.285+0000 7faa2bc00640 1 --2- 172.21.15.28:0/2227189613 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7faa0006b4d0 0x7faa0006d990 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7faa0c036000 tx=0x7faa0c008030 comp rx=0 tx=0).stop 2024-02-21T23:11:06.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:06.285+0000 7faa2bc00640 1 -- 172.21.15.28:0/2227189613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa240c11f0 msgr2=0x7faa241e9840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:06.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:06.285+0000 7faa2bc00640 1 --2- 172.21.15.28:0/2227189613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa240c11f0 0x7faa241e9840 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7faa1400c3c0 tx=0x7faa1400c880 comp rx=0 tx=0).stop 2024-02-21T23:11:06.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:06.285+0000 7faa2bc00640 1 -- 172.21.15.28:0/2227189613 shutdown_connections 2024-02-21T23:11:06.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:06.285+0000 7faa2bc00640 1 --2- 172.21.15.28:0/2227189613 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7faa0006b4d0 0x7faa0006d990 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:06.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:06.285+0000 7faa2bc00640 1 --2- 172.21.15.28:0/2227189613 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faa2415b6d0 0x7faa241448f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:06.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:06.285+0000 7faa2bc00640 1 --2- 172.21.15.28:0/2227189613 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faa24153950 0x7faa240c55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:06.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:06.285+0000 7faa2bc00640 1 --2- 172.21.15.28:0/2227189613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa240c11f0 0x7faa241e9840 unknown :-1 s=CLOSED pgs=1951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:06.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:06.285+0000 7faa2bc00640 1 -- 172.21.15.28:0/2227189613 >> 172.21.15.28:0/2227189613 conn(0x7faa240c67f0 msgr2=0x7faa240c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:06.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:06.286+0000 7faa2bc00640 1 -- 172.21.15.28:0/2227189613 shutdown_connections 2024-02-21T23:11:06.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:06.286+0000 7faa2bc00640 1 -- 172.21.15.28:0/2227189613 wait complete. 2024-02-21T23:11:06.695 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 187, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:10:53.351225+0000', 'modified': '2024-02-21T23:11:01.967539+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 9480}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9480': {'gid': 9480, 'name': 'c', 'rank': 0, 'incarnation': 181, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.28:6833/1850463609', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 1850463609}, {'type': 'v1', 'addr': '172.21.15.28:6833', 'nonce': 1850463609}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:11:06.695 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9480:up:stopping 2024-02-21T23:11:07.484 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-21T23:11:07.492 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-21T23:11:07.697 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:11:07.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:07.939+0000 7f3f57e00640 1 Processor -- start 2024-02-21T23:11:07.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:07.940+0000 7f3f57e00640 1 -- start start 2024-02-21T23:11:07.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:07.941+0000 7f3f57e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3f501569d0 0x7f3f500c3280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:07.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:07.941+0000 7f3f57e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3f500c4fe0 0x7f3f5006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:07.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:07.941+0000 7f3f57e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f5006ed10 con 0x7f3f50159400 2024-02-21T23:11:07.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:07.941+0000 7f3f57e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f50159900 con 0x7f3f500c4fe0 2024-02-21T23:11:07.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:07.941+0000 7f3f57e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f50159be0 con 0x7f3f501569d0 2024-02-21T23:11:07.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:07.942+0000 7f3f55e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f3f50159400 0x7f3f5006e600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:44032/0 (socket says 172.21.15.28:44032) 2024-02-21T23:11:07.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:07.942+0000 7f3f55e00640 1 -- 172.21.15.28:0/4112557937 learned_addr learned my addr 172.21.15.28:0/4112557937 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:11:07.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:07.941+0000 7f3f54a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3f500c4fe0 0x7f3f5006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:07.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:07.942+0000 7f3f55400640 1 --2- 172.21.15.28:0/4112557937 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3f501569d0 0x7f3f500c3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:07.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:07.942+0000 7f3f4fe00640 1 -- 172.21.15.28:0/4112557937 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1642679563 0 0) 0x7f3f5006ed10 con 0x7f3f50159400 2024-02-21T23:11:07.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:07.942+0000 7f3f4fe00640 1 -- 172.21.15.28:0/4112557937 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f38003520 con 0x7f3f50159400 2024-02-21T23:11:07.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:07.942+0000 7f3f54a00640 1 -- 172.21.15.28:0/4112557937 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3f501569d0 msgr2=0x7f3f500c3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:07.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:07.942+0000 7f3f54a00640 1 --2- 172.21.15.28:0/4112557937 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3f501569d0 0x7f3f500c3280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:07.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:07.942+0000 7f3f54a00640 1 -- 172.21.15.28:0/4112557937 >> v1:172.21.15.28:6789/0 conn(0x7f3f50159400 legacy=0x7f3f5006e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:07.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:07.942+0000 7f3f54a00640 1 -- 172.21.15.28:0/4112557937 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f50159ec0 con 0x7f3f500c4fe0 2024-02-21T23:11:07.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:07.944+0000 7f3f54a00640 1 --2- 172.21.15.28:0/4112557937 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f500c4fe0 0x7f3f5006e0c0 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c0097a0 tx=0x7f3f3c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=932a1dbb1765ab8d server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:07.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:07.944+0000 7f3f4fe00640 1 -- 172.21.15.28:0/4112557937 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f3c0315a0 con 0x7f3f500c4fe0 2024-02-21T23:11:07.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:07.944+0000 7f3f4fe00640 1 -- 172.21.15.28:0/4112557937 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f3c037070 con 0x7f3f500c4fe0 2024-02-21T23:11:07.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:07.944+0000 7f3f4fe00640 1 -- 172.21.15.28:0/4112557937 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f3c031960 con 0x7f3f500c4fe0 2024-02-21T23:11:07.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:07.944+0000 7f3f57e00640 1 -- 172.21.15.28:0/4112557937 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f500c4fe0 msgr2=0x7f3f5006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:07.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:07.944+0000 7f3f57e00640 1 --2- 172.21.15.28:0/4112557937 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f500c4fe0 0x7f3f5006e0c0 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c0097a0 tx=0x7f3f3c02b840 comp rx=0 tx=0).stop 2024-02-21T23:11:07.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:07.945+0000 7f3f57e00640 1 -- 172.21.15.28:0/4112557937 shutdown_connections 2024-02-21T23:11:07.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:07.945+0000 7f3f57e00640 1 --2- 172.21.15.28:0/4112557937 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f500c4fe0 0x7f3f5006e0c0 unknown :-1 s=CLOSED pgs=1953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:07.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:07.945+0000 7f3f57e00640 1 --2- 172.21.15.28:0/4112557937 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3f501569d0 0x7f3f500c3280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:07.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:07.945+0000 7f3f57e00640 1 -- 172.21.15.28:0/4112557937 >> 172.21.15.28:0/4112557937 conn(0x7f3f500c67f0 msgr2=0x7f3f500c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:07.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:07.945+0000 7f3f57e00640 1 -- 172.21.15.28:0/4112557937 shutdown_connections 2024-02-21T23:11:07.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:07.945+0000 7f3f57e00640 1 -- 172.21.15.28:0/4112557937 wait complete. 2024-02-21T23:11:07.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:07.946+0000 7f3f57e00640 1 Processor -- start 2024-02-21T23:11:07.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:07.946+0000 7f3f57e00640 1 -- start start 2024-02-21T23:11:07.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:07.947+0000 7f3f57e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3f500c4fe0 0x7f3f5014cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:07.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:07.947+0000 7f3f57e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f501569d0 0x7f3f5014d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:07.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:07.947+0000 7f3f57e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f50159400 0x7f3f501f2900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:07.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:07.947+0000 7f3f57e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f500c37c0 con 0x7f3f501569d0 2024-02-21T23:11:07.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:07.947+0000 7f3f57e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f50156db0 con 0x7f3f50159400 2024-02-21T23:11:07.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:07.947+0000 7f3f57e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f500c3940 con 0x7f3f500c4fe0 2024-02-21T23:11:07.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:07.948+0000 7f3f54a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f501569d0 0x7f3f5014d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:07.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:07.948+0000 7f3f54a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f501569d0 0x7f3f5014d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59584/0 (socket says 172.21.15.28:59584) 2024-02-21T23:11:07.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:07.948+0000 7f3f54a00640 1 -- 172.21.15.28:0/2024950392 learned_addr learned my addr 172.21.15.28:0/2024950392 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:07.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:07.948+0000 7f3f55e00640 1 --2- 172.21.15.28:0/2024950392 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f50159400 0x7f3f501f2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:07.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:07.948+0000 7f3f55400640 1 --2- 172.21.15.28:0/2024950392 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3f500c4fe0 0x7f3f5014cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:07.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:07.948+0000 7f3f54a00640 1 -- 172.21.15.28:0/2024950392 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3f500c4fe0 msgr2=0x7f3f5014cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:07.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:07.948+0000 7f3f54a00640 1 --2- 172.21.15.28:0/2024950392 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3f500c4fe0 0x7f3f5014cf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:07.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:07.948+0000 7f3f54a00640 1 -- 172.21.15.28:0/2024950392 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f50159400 msgr2=0x7f3f501f2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:07.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:07.948+0000 7f3f54a00640 1 --2- 172.21.15.28:0/2024950392 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f50159400 0x7f3f501f2900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:07.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:07.948+0000 7f3f54a00640 1 -- 172.21.15.28:0/2024950392 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f501f3610 con 0x7f3f501569d0 2024-02-21T23:11:07.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:07.949+0000 7f3f54a00640 1 --2- 172.21.15.28:0/2024950392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f501569d0 0x7f3f5014d4c0 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c034040 tx=0x7f3f3c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:07.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:07.949+0000 7f3f4d600640 1 -- 172.21.15.28:0/2024950392 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f3c032630 con 0x7f3f501569d0 2024-02-21T23:11:07.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:07.949+0000 7f3f4d600640 1 -- 172.21.15.28:0/2024950392 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f3c0327f0 con 0x7f3f501569d0 2024-02-21T23:11:07.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:07.949+0000 7f3f4d600640 1 -- 172.21.15.28:0/2024950392 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f3c032be0 con 0x7f3f501569d0 2024-02-21T23:11:07.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:07.949+0000 7f3f57e00640 1 -- 172.21.15.28:0/2024950392 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f5015b4e0 con 0x7f3f501569d0 2024-02-21T23:11:07.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:07.949+0000 7f3f57e00640 1 -- 172.21.15.28:0/2024950392 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f5015b200 con 0x7f3f501569d0 2024-02-21T23:11:07.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:07.950+0000 7f3f57e00640 1 -- 172.21.15.28:0/2024950392 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f50067f30 con 0x7f3f501569d0 2024-02-21T23:11:07.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:07.951+0000 7f3f4d600640 1 -- 172.21.15.28:0/2024950392 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3f3c04e430 con 0x7f3f501569d0 2024-02-21T23:11:07.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:07.953+0000 7f3f4d600640 1 --2- 172.21.15.28:0/2024950392 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3f3006b4d0 0x7f3f3006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:07.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:07.953+0000 7f3f4d600640 1 -- 172.21.15.28:0/2024950392 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f3f3c02f030 con 0x7f3f501569d0 2024-02-21T23:11:07.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:07.953+0000 7f3f55400640 1 --2- 172.21.15.28:0/2024950392 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3f3006b4d0 0x7f3f3006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:07.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:07.954+0000 7f3f55400640 1 --2- 172.21.15.28:0/2024950392 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3f3006b4d0 0x7f3f3006d990 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f3f44009f30 tx=0x7f3f4400b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:07.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:07.958+0000 7f3f4d600640 1 -- 172.21.15.28:0/2024950392 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3f3c040070 con 0x7f3f501569d0 2024-02-21T23:11:08.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:08.197+0000 7f3f57e00640 1 -- 172.21.15.28:0/2024950392 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3f5006f1d0 con 0x7f3f501569d0 2024-02-21T23:11:08.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:08.199+0000 7f3f4d600640 1 -- 172.21.15.28:0/2024950392 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 187 v187) v1 ==== 95+0+4947 (secure 0 0 0) 0x7f3f3c07aee0 con 0x7f3f501569d0 2024-02-21T23:11:08.187 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:11:08.187 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":187,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6686,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/155806034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":155806034},{"type":"v1","addr":"172.21.15.81:6836","nonce":155806034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":180},{"gid":7480,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3897243745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":3897243745},{"type":"v1","addr":"172.21.15.81:6837","nonce":3897243745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":180},{"gid":7486,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2827135632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2827135632},{"type":"v1","addr":"172.21.15.28:6835","nonce":2827135632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":180}],"filesystems":[{"mdsmap":{"epoch":187,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:10:53.351225+0000","modified":"2024-02-21T23:11:01.967539+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0],"up":{"mds_0":9480},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9480":{"gid":9480,"name":"c","rank":0,"incarnation":181,"state":"up:stopping","state_seq":3,"addr":"172.21.15.28:6833/1850463609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1850463609},{"type":"v1","addr":"172.21.15.28:6833","nonce":1850463609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2024-02-21T23:11:08.187 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 187 2024-02-21T23:11:08.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:08.202+0000 7f3f57e00640 1 -- 172.21.15.28:0/2024950392 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3f3006b4d0 msgr2=0x7f3f3006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:08.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:08.202+0000 7f3f57e00640 1 --2- 172.21.15.28:0/2024950392 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3f3006b4d0 0x7f3f3006d990 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f3f44009f30 tx=0x7f3f4400b040 comp rx=0 tx=0).stop 2024-02-21T23:11:08.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:08.202+0000 7f3f57e00640 1 -- 172.21.15.28:0/2024950392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f501569d0 msgr2=0x7f3f5014d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:08.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:08.202+0000 7f3f57e00640 1 --2- 172.21.15.28:0/2024950392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f501569d0 0x7f3f5014d4c0 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c034040 tx=0x7f3f3c0046c0 comp rx=0 tx=0).stop 2024-02-21T23:11:08.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:08.202+0000 7f3f57e00640 1 -- 172.21.15.28:0/2024950392 shutdown_connections 2024-02-21T23:11:08.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:08.202+0000 7f3f57e00640 1 --2- 172.21.15.28:0/2024950392 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3f3006b4d0 0x7f3f3006d990 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:08.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:08.202+0000 7f3f57e00640 1 --2- 172.21.15.28:0/2024950392 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f50159400 0x7f3f501f2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:08.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:08.202+0000 7f3f57e00640 1 --2- 172.21.15.28:0/2024950392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f501569d0 0x7f3f5014d4c0 unknown :-1 s=CLOSED pgs=1954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:08.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:08.202+0000 7f3f57e00640 1 --2- 172.21.15.28:0/2024950392 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3f500c4fe0 0x7f3f5014cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:08.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:08.202+0000 7f3f57e00640 1 -- 172.21.15.28:0/2024950392 >> 172.21.15.28:0/2024950392 conn(0x7f3f500c67f0 msgr2=0x7f3f50157270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:08.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:08.202+0000 7f3f57e00640 1 -- 172.21.15.28:0/2024950392 shutdown_connections 2024-02-21T23:11:08.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:08.202+0000 7f3f57e00640 1 -- 172.21.15.28:0/2024950392 wait complete. 2024-02-21T23:11:08.613 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 187, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:10:53.351225+0000', 'modified': '2024-02-21T23:11:01.967539+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 9480}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9480': {'gid': 9480, 'name': 'c', 'rank': 0, 'incarnation': 181, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.28:6833/1850463609', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 1850463609}, {'type': 'v1', 'addr': '172.21.15.28:6833', 'nonce': 1850463609}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:11:08.614 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9480:up:stopping 2024-02-21T23:11:09.615 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:11:09.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:09.857+0000 7f5501800640 1 Processor -- start 2024-02-21T23:11:09.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:09.857+0000 7f5501800640 1 -- start start 2024-02-21T23:11:09.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:09.859+0000 7f5501800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f54fc06e3d0 0x7f54fc14b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:09.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:09.859+0000 7f5501800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54fc0a8b40 con 0x7f54fc06dff0 2024-02-21T23:11:09.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:09.859+0000 7f5501800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54fc06e7d0 con 0x7f54fc0c0fe0 2024-02-21T23:11:09.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:09.859+0000 7f5501800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54fc0a8e00 con 0x7f54fc06e3d0 2024-02-21T23:11:09.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:09.859+0000 7f5500e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f54fc06e3d0 0x7f54fc14b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:09.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:09.859+0000 7f54fbe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f54fc06dff0 0x7f54fc071350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:44044/0 (socket says 172.21.15.28:44044) 2024-02-21T23:11:09.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:09.859+0000 7f54fbe00640 1 -- 172.21.15.28:0/2527490982 learned_addr learned my addr 172.21.15.28:0/2527490982 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:11:09.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:09.860+0000 7f5500e00640 1 -- 172.21.15.28:0/2527490982 >> v1:172.21.15.81:6789/0 conn(0x7f54fc0c0fe0 legacy=0x7f54fc158580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:09.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:09.860+0000 7f5500e00640 1 -- 172.21.15.28:0/2527490982 >> v1:172.21.15.28:6789/0 conn(0x7f54fc06dff0 legacy=0x7f54fc071350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:09.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:09.860+0000 7f5500e00640 1 -- 172.21.15.28:0/2527490982 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54fc0a90e0 con 0x7f54fc06e3d0 2024-02-21T23:11:09.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:09.860+0000 7f54fb400640 1 -- 172.21.15.28:0/2527490982 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2723861560 0 0) 0x7f54fc0a8b40 con 0x7f54fc06dff0 2024-02-21T23:11:09.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:09.862+0000 7f5500e00640 1 --2- 172.21.15.28:0/2527490982 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54fc06e3d0 0x7f54fc14b2b0 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7f54f4009d90 tx=0x7f54f402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bcf52ece15e0028 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:09.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:09.862+0000 7f54fb400640 1 -- 172.21.15.28:0/2527490982 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54f4039070 con 0x7f54fc06e3d0 2024-02-21T23:11:09.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:09.862+0000 7f54fb400640 1 -- 172.21.15.28:0/2527490982 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54f402cd00 con 0x7f54fc06e3d0 2024-02-21T23:11:09.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:09.862+0000 7f54fb400640 1 -- 172.21.15.28:0/2527490982 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54f4034660 con 0x7f54fc06e3d0 2024-02-21T23:11:09.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:09.863+0000 7f5501800640 1 -- 172.21.15.28:0/2527490982 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54fc06e3d0 msgr2=0x7f54fc14b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:09.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:09.863+0000 7f5501800640 1 --2- 172.21.15.28:0/2527490982 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54fc06e3d0 0x7f54fc14b2b0 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7f54f4009d90 tx=0x7f54f402bf50 comp rx=0 tx=0).stop 2024-02-21T23:11:09.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:09.863+0000 7f5501800640 1 -- 172.21.15.28:0/2527490982 shutdown_connections 2024-02-21T23:11:09.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:09.863+0000 7f5501800640 1 --2- 172.21.15.28:0/2527490982 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54fc06e3d0 0x7f54fc14b2b0 unknown :-1 s=CLOSED pgs=1956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:09.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:09.863+0000 7f5501800640 1 -- 172.21.15.28:0/2527490982 >> 172.21.15.28:0/2527490982 conn(0x7f54fc06b630 msgr2=0x7f54fc0c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:09.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:09.863+0000 7f5501800640 1 -- 172.21.15.28:0/2527490982 shutdown_connections 2024-02-21T23:11:09.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:09.863+0000 7f5501800640 1 -- 172.21.15.28:0/2527490982 wait complete. 2024-02-21T23:11:09.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:09.865+0000 7f5501800640 1 Processor -- start 2024-02-21T23:11:09.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:09.865+0000 7f5501800640 1 -- start start 2024-02-21T23:11:09.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:09.866+0000 7f5501800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54fc06dff0 0x7f54fc1eb650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:09.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:09.866+0000 7f5501800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54fc06e3d0 0x7f54fc0ab6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:09.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:09.866+0000 7f5501800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f54fc0c0fe0 0x7f54fc150c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:09.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:09.866+0000 7f5501800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54fc073520 con 0x7f54fc06e3d0 2024-02-21T23:11:09.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:09.866+0000 7f5501800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54fc0731b0 con 0x7f54fc06dff0 2024-02-21T23:11:09.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:09.866+0000 7f5501800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54fc14b7f0 con 0x7f54fc0c0fe0 2024-02-21T23:11:09.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:09.866+0000 7f54f3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54fc06e3d0 0x7f54fc0ab6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:09.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:09.866+0000 7f54f3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54fc06e3d0 0x7f54fc0ab6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59596/0 (socket says 172.21.15.28:59596) 2024-02-21T23:11:09.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:09.866+0000 7f54f3e00640 1 -- 172.21.15.28:0/3447934042 learned_addr learned my addr 172.21.15.28:0/3447934042 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:09.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:09.866+0000 7f54fbe00640 1 --2- 172.21.15.28:0/3447934042 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54fc06dff0 0x7f54fc1eb650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:09.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:09.866+0000 7f5500e00640 1 --2- 172.21.15.28:0/3447934042 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f54fc0c0fe0 0x7f54fc150c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:09.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:09.867+0000 7f54f3e00640 1 -- 172.21.15.28:0/3447934042 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f54fc0c0fe0 msgr2=0x7f54fc150c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:09.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:09.867+0000 7f54f3e00640 1 --2- 172.21.15.28:0/3447934042 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f54fc0c0fe0 0x7f54fc150c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:09.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:09.867+0000 7f54f3e00640 1 -- 172.21.15.28:0/3447934042 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54fc06dff0 msgr2=0x7f54fc1eb650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:09.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:09.867+0000 7f54f3e00640 1 --2- 172.21.15.28:0/3447934042 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54fc06dff0 0x7f54fc1eb650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:09.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:09.867+0000 7f54f3e00640 1 -- 172.21.15.28:0/3447934042 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54fc0abc20 con 0x7f54fc06e3d0 2024-02-21T23:11:09.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:09.867+0000 7f5500e00640 1 --2- 172.21.15.28:0/3447934042 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f54fc0c0fe0 0x7f54fc150c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:11:09.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:09.867+0000 7f54f3e00640 1 --2- 172.21.15.28:0/3447934042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54fc06e3d0 0x7f54fc0ab6e0 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7f54e400c480 tx=0x7f54e400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:09.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:09.867+0000 7f54f8c00640 1 -- 172.21.15.28:0/3447934042 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54e40052d0 con 0x7f54fc06e3d0 2024-02-21T23:11:09.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:09.867+0000 7f54f8c00640 1 -- 172.21.15.28:0/3447934042 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54e4005470 con 0x7f54fc06e3d0 2024-02-21T23:11:09.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:09.867+0000 7f54f8c00640 1 -- 172.21.15.28:0/3447934042 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54e4012870 con 0x7f54fc06e3d0 2024-02-21T23:11:09.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:09.867+0000 7f5501800640 1 -- 172.21.15.28:0/3447934042 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54fc0aa640 con 0x7f54fc06e3d0 2024-02-21T23:11:09.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:09.867+0000 7f5501800640 1 -- 172.21.15.28:0/3447934042 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54fc0aac00 con 0x7f54fc06e3d0 2024-02-21T23:11:09.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:09.869+0000 7f5501800640 1 -- 172.21.15.28:0/3447934042 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54fc04f290 con 0x7f54fc06e3d0 2024-02-21T23:11:09.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:09.869+0000 7f54f8c00640 1 -- 172.21.15.28:0/3447934042 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f54e4012a10 con 0x7f54fc06e3d0 2024-02-21T23:11:09.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:09.870+0000 7f54f8c00640 1 --2- 172.21.15.28:0/3447934042 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f54d806b500 0x7f54d806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:09.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:09.870+0000 7f54f8c00640 1 -- 172.21.15.28:0/3447934042 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f54e4089ff0 con 0x7f54fc06e3d0 2024-02-21T23:11:09.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:09.871+0000 7f54fbe00640 1 --2- 172.21.15.28:0/3447934042 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f54d806b500 0x7f54d806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:09.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:09.872+0000 7f54fbe00640 1 --2- 172.21.15.28:0/3447934042 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f54d806b500 0x7f54d806d9c0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f54ec005ea0 tx=0x7f54ec007d40 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:09.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:09.875+0000 7f54f8c00640 1 -- 172.21.15.28:0/3447934042 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f54e4058d10 con 0x7f54fc06e3d0 2024-02-21T23:11:10.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:10.124+0000 7f5501800640 1 -- 172.21.15.28:0/3447934042 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f54fc156160 con 0x7f54fc06e3d0 2024-02-21T23:11:10.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:10.126+0000 7f54f8c00640 1 -- 172.21.15.28:0/3447934042 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 187 v187) v1 ==== 95+0+4947 (secure 0 0 0) 0x7f54e4058610 con 0x7f54fc06e3d0 2024-02-21T23:11:10.114 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:11:10.114 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":187,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6686,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/155806034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":155806034},{"type":"v1","addr":"172.21.15.81:6836","nonce":155806034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":180},{"gid":7480,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3897243745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":3897243745},{"type":"v1","addr":"172.21.15.81:6837","nonce":3897243745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":180},{"gid":7486,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2827135632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2827135632},{"type":"v1","addr":"172.21.15.28:6835","nonce":2827135632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":180}],"filesystems":[{"mdsmap":{"epoch":187,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:10:53.351225+0000","modified":"2024-02-21T23:11:01.967539+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0],"up":{"mds_0":9480},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9480":{"gid":9480,"name":"c","rank":0,"incarnation":181,"state":"up:stopping","state_seq":3,"addr":"172.21.15.28:6833/1850463609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1850463609},{"type":"v1","addr":"172.21.15.28:6833","nonce":1850463609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2024-02-21T23:11:10.114 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 187 2024-02-21T23:11:10.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:10.129+0000 7f5501800640 1 -- 172.21.15.28:0/3447934042 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f54d806b500 msgr2=0x7f54d806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:10.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:10.129+0000 7f5501800640 1 --2- 172.21.15.28:0/3447934042 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f54d806b500 0x7f54d806d9c0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f54ec005ea0 tx=0x7f54ec007d40 comp rx=0 tx=0).stop 2024-02-21T23:11:10.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:10.129+0000 7f5501800640 1 -- 172.21.15.28:0/3447934042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54fc06e3d0 msgr2=0x7f54fc0ab6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:10.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:10.129+0000 7f5501800640 1 --2- 172.21.15.28:0/3447934042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54fc06e3d0 0x7f54fc0ab6e0 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7f54e400c480 tx=0x7f54e400c940 comp rx=0 tx=0).stop 2024-02-21T23:11:10.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:10.129+0000 7f5501800640 1 -- 172.21.15.28:0/3447934042 shutdown_connections 2024-02-21T23:11:10.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:10.129+0000 7f5501800640 1 --2- 172.21.15.28:0/3447934042 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f54d806b500 0x7f54d806d9c0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:10.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:10.129+0000 7f5501800640 1 --2- 172.21.15.28:0/3447934042 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f54fc0c0fe0 0x7f54fc150c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:10.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:10.129+0000 7f5501800640 1 --2- 172.21.15.28:0/3447934042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54fc06e3d0 0x7f54fc0ab6e0 unknown :-1 s=CLOSED pgs=1957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:10.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:10.129+0000 7f5501800640 1 --2- 172.21.15.28:0/3447934042 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54fc06dff0 0x7f54fc1eb650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:10.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:10.129+0000 7f5501800640 1 -- 172.21.15.28:0/3447934042 >> 172.21.15.28:0/3447934042 conn(0x7f54fc06b630 msgr2=0x7f54fc0c4e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:10.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:10.129+0000 7f5501800640 1 -- 172.21.15.28:0/3447934042 shutdown_connections 2024-02-21T23:11:10.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:10.129+0000 7f5501800640 1 -- 172.21.15.28:0/3447934042 wait complete. 2024-02-21T23:11:10.540 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 187, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:10:53.351225+0000', 'modified': '2024-02-21T23:11:01.967539+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 9480}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9480': {'gid': 9480, 'name': 'c', 'rank': 0, 'incarnation': 181, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.28:6833/1850463609', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 1850463609}, {'type': 'v1', 'addr': '172.21.15.28:6833', 'nonce': 1850463609}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:11:10.540 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9480:up:stopping 2024-02-21T23:11:11.541 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:11:11.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:11.777+0000 7f392de00640 1 Processor -- start 2024-02-21T23:11:11.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:11.778+0000 7f392de00640 1 -- start start 2024-02-21T23:11:11.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:11.779+0000 7f392de00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f39280aff90 0x7f39280c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:11.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:11.779+0000 7f392de00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f39280c5720 0x7f392815f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:11.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:11.779+0000 7f392de00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39280c74c0 con 0x7f392806dbf0 2024-02-21T23:11:11.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:11.779+0000 7f392de00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39280c7270 con 0x7f39280c5720 2024-02-21T23:11:11.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:11.779+0000 7f392de00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3928158630 con 0x7f39280aff90 2024-02-21T23:11:11.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:11.779+0000 7f392d400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f39280c5720 0x7f392815f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:11.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:11.780+0000 7f392d400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f39280c5720 0x7f392815f890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59606/0 (socket says 172.21.15.28:59606) 2024-02-21T23:11:11.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:11.780+0000 7f392d400640 1 -- 172.21.15.28:0/2362295407 learned_addr learned my addr 172.21.15.28:0/2362295407 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:11.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:11.780+0000 7f392ca00640 1 --2- 172.21.15.28:0/2362295407 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f39280aff90 0x7f39280c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:11.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:11.780+0000 7f392d400640 1 -- 172.21.15.28:0/2362295407 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f39280aff90 msgr2=0x7f39280c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:11.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:11.780+0000 7f392d400640 1 --2- 172.21.15.28:0/2362295407 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f39280aff90 0x7f39280c51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:11.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:11.780+0000 7f392d400640 1 -- 172.21.15.28:0/2362295407 >> v1:172.21.15.81:6789/0 conn(0x7f392806dbf0 legacy=0x7f3928158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:11.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:11.780+0000 7f392d400640 1 -- 172.21.15.28:0/2362295407 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39280c77a0 con 0x7f39280c5720 2024-02-21T23:11:11.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:11.781+0000 7f392d400640 1 --2- 172.21.15.28:0/2362295407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39280c5720 0x7f392815f890 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7f3920009e50 tx=0x7f392002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7526dd98c0cc631 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:11.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:11.781+0000 7f3927400640 1 -- 172.21.15.28:0/2362295407 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f392002ca70 con 0x7f39280c5720 2024-02-21T23:11:11.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:11.782+0000 7f3927400640 1 -- 172.21.15.28:0/2362295407 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f392002cc10 con 0x7f39280c5720 2024-02-21T23:11:11.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:11.782+0000 7f3927400640 1 -- 172.21.15.28:0/2362295407 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39200343f0 con 0x7f39280c5720 2024-02-21T23:11:11.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:11.782+0000 7f392de00640 1 -- 172.21.15.28:0/2362295407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39280c5720 msgr2=0x7f392815f890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:11.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:11.782+0000 7f392de00640 1 --2- 172.21.15.28:0/2362295407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39280c5720 0x7f392815f890 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7f3920009e50 tx=0x7f392002bed0 comp rx=0 tx=0).stop 2024-02-21T23:11:11.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:11.782+0000 7f392de00640 1 -- 172.21.15.28:0/2362295407 shutdown_connections 2024-02-21T23:11:11.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:11.782+0000 7f392de00640 1 --2- 172.21.15.28:0/2362295407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39280c5720 0x7f392815f890 unknown :-1 s=CLOSED pgs=1958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:11.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:11.782+0000 7f392de00640 1 --2- 172.21.15.28:0/2362295407 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f39280aff90 0x7f39280c51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:11.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:11.782+0000 7f392de00640 1 -- 172.21.15.28:0/2362295407 >> 172.21.15.28:0/2362295407 conn(0x7f39280ac370 msgr2=0x7f39280ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:11.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:11.782+0000 7f392de00640 1 -- 172.21.15.28:0/2362295407 shutdown_connections 2024-02-21T23:11:11.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:11.782+0000 7f392de00640 1 -- 172.21.15.28:0/2362295407 wait complete. 2024-02-21T23:11:11.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:11.784+0000 7f392de00640 1 Processor -- start 2024-02-21T23:11:11.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:11.784+0000 7f392de00640 1 -- start start 2024-02-21T23:11:11.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:11.785+0000 7f392de00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f392806dbf0 0x7f39280c21b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:11.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:11.785+0000 7f392de00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39280aff90 0x7f39281eb250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:11.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:11.785+0000 7f392de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39280c5720 0x7f39280aabe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:11.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:11.785+0000 7f392de00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39280c8760 con 0x7f39280c5720 2024-02-21T23:11:11.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:11.785+0000 7f392de00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39280be0c0 con 0x7f39280aff90 2024-02-21T23:11:11.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:11.785+0000 7f392de00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39280c88e0 con 0x7f392806dbf0 2024-02-21T23:11:11.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:11.785+0000 7f392d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39280c5720 0x7f39280aabe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:11.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:11.785+0000 7f392d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39280c5720 0x7f39280aabe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59610/0 (socket says 172.21.15.28:59610) 2024-02-21T23:11:11.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:11.785+0000 7f392d400640 1 -- 172.21.15.28:0/3371611461 learned_addr learned my addr 172.21.15.28:0/3371611461 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:11.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:11.786+0000 7f392ca00640 1 --2- 172.21.15.28:0/3371611461 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f392806dbf0 0x7f39280c21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:11.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:11.786+0000 7f3927e00640 1 --2- 172.21.15.28:0/3371611461 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39280aff90 0x7f39281eb250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:11.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:11.786+0000 7f392d400640 1 -- 172.21.15.28:0/3371611461 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f392806dbf0 msgr2=0x7f39280c21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:11.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:11.786+0000 7f392d400640 1 --2- 172.21.15.28:0/3371611461 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f392806dbf0 0x7f39280c21b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:11.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:11.786+0000 7f392d400640 1 -- 172.21.15.28:0/3371611461 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39280aff90 msgr2=0x7f39281eb250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:11.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:11.786+0000 7f392d400640 1 --2- 172.21.15.28:0/3371611461 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39280aff90 0x7f39281eb250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:11.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:11.786+0000 7f392d400640 1 -- 172.21.15.28:0/3371611461 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39281ede80 con 0x7f39280c5720 2024-02-21T23:11:11.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:11.786+0000 7f3927e00640 1 --2- 172.21.15.28:0/3371611461 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39280aff90 0x7f39281eb250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:11:11.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:11.786+0000 7f392ca00640 1 --2- 172.21.15.28:0/3371611461 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f392806dbf0 0x7f39280c21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:11:11.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:11.786+0000 7f392d400640 1 --2- 172.21.15.28:0/3371611461 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39280c5720 0x7f39280aabe0 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7f3920036040 tx=0x7f3920004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:11.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:11.787+0000 7f3924c00640 1 -- 172.21.15.28:0/3371611461 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3920044020 con 0x7f39280c5720 2024-02-21T23:11:11.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:11.787+0000 7f3924c00640 1 -- 172.21.15.28:0/3371611461 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3920040690 con 0x7f39280c5720 2024-02-21T23:11:11.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:11.787+0000 7f3924c00640 1 -- 172.21.15.28:0/3371611461 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3920009510 con 0x7f39280c5720 2024-02-21T23:11:11.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:11.787+0000 7f392de00640 1 -- 172.21.15.28:0/3371611461 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39281eb790 con 0x7f39280c5720 2024-02-21T23:11:11.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:11.787+0000 7f392de00640 1 -- 172.21.15.28:0/3371611461 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3928154a40 con 0x7f39280c5720 2024-02-21T23:11:11.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:11.788+0000 7f392de00640 1 -- 172.21.15.28:0/3371611461 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f392804f290 con 0x7f39280c5720 2024-02-21T23:11:11.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:11.789+0000 7f3924c00640 1 -- 172.21.15.28:0/3371611461 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3920047020 con 0x7f39280c5720 2024-02-21T23:11:11.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:11.790+0000 7f3924c00640 1 --2- 172.21.15.28:0/3371611461 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f390806b4d0 0x7f390806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:11.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:11.790+0000 7f3924c00640 1 -- 172.21.15.28:0/3371611461 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f39200aceb0 con 0x7f39280c5720 2024-02-21T23:11:11.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:11.791+0000 7f392ca00640 1 --2- 172.21.15.28:0/3371611461 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f390806b4d0 0x7f390806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:11.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:11.792+0000 7f392ca00640 1 --2- 172.21.15.28:0/3371611461 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f390806b4d0 0x7f390806d990 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f391c006fd0 tx=0x7f391c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:11.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:11.795+0000 7f3924c00640 1 -- 172.21.15.28:0/3371611461 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f392004e030 con 0x7f39280c5720 2024-02-21T23:11:12.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:12.032+0000 7f392de00640 1 -- 172.21.15.28:0/3371611461 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3928158140 con 0x7f39280c5720 2024-02-21T23:11:12.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:12.034+0000 7f3924c00640 1 -- 172.21.15.28:0/3371611461 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 187 v187) v1 ==== 95+0+4947 (secure 0 0 0) 0x7f392007bb90 con 0x7f39280c5720 2024-02-21T23:11:12.022 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:11:12.022 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":187,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6686,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/155806034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":155806034},{"type":"v1","addr":"172.21.15.81:6836","nonce":155806034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":180},{"gid":7480,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3897243745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":3897243745},{"type":"v1","addr":"172.21.15.81:6837","nonce":3897243745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":180},{"gid":7486,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2827135632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2827135632},{"type":"v1","addr":"172.21.15.28:6835","nonce":2827135632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":180}],"filesystems":[{"mdsmap":{"epoch":187,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:10:53.351225+0000","modified":"2024-02-21T23:11:01.967539+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0],"up":{"mds_0":9480},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9480":{"gid":9480,"name":"c","rank":0,"incarnation":181,"state":"up:stopping","state_seq":3,"addr":"172.21.15.28:6833/1850463609","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1850463609},{"type":"v1","addr":"172.21.15.28:6833","nonce":1850463609}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2024-02-21T23:11:12.022 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 187 2024-02-21T23:11:12.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:12.036+0000 7f392de00640 1 -- 172.21.15.28:0/3371611461 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f390806b4d0 msgr2=0x7f390806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:12.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:12.036+0000 7f392de00640 1 --2- 172.21.15.28:0/3371611461 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f390806b4d0 0x7f390806d990 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f391c006fd0 tx=0x7f391c008040 comp rx=0 tx=0).stop 2024-02-21T23:11:12.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:12.036+0000 7f392de00640 1 -- 172.21.15.28:0/3371611461 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39280c5720 msgr2=0x7f39280aabe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:12.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:12.036+0000 7f392de00640 1 --2- 172.21.15.28:0/3371611461 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39280c5720 0x7f39280aabe0 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7f3920036040 tx=0x7f3920004a40 comp rx=0 tx=0).stop 2024-02-21T23:11:12.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:12.036+0000 7f392de00640 1 -- 172.21.15.28:0/3371611461 shutdown_connections 2024-02-21T23:11:12.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:12.036+0000 7f392de00640 1 --2- 172.21.15.28:0/3371611461 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f390806b4d0 0x7f390806d990 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:12.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:12.036+0000 7f392de00640 1 --2- 172.21.15.28:0/3371611461 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39280c5720 0x7f39280aabe0 unknown :-1 s=CLOSED pgs=1959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:12.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:12.036+0000 7f392de00640 1 --2- 172.21.15.28:0/3371611461 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f39280aff90 0x7f39281eb250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:12.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:12.036+0000 7f392de00640 1 --2- 172.21.15.28:0/3371611461 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f392806dbf0 0x7f39280c21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:12.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:12.036+0000 7f392de00640 1 -- 172.21.15.28:0/3371611461 >> 172.21.15.28:0/3371611461 conn(0x7f39280ac370 msgr2=0x7f39280ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:12.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:12.036+0000 7f392de00640 1 -- 172.21.15.28:0/3371611461 shutdown_connections 2024-02-21T23:11:12.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:12.036+0000 7f392de00640 1 -- 172.21.15.28:0/3371611461 wait complete. 2024-02-21T23:11:12.445 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 187, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:10:53.351225+0000', 'modified': '2024-02-21T23:11:01.967539+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 9480}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9480': {'gid': 9480, 'name': 'c', 'rank': 0, 'incarnation': 181, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.28:6833/1850463609', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 1850463609}, {'type': 'v1', 'addr': '172.21.15.28:6833', 'nonce': 1850463609}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:11:12.445 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9480:up:stopping 2024-02-21T23:11:13.447 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:11:13.605 INFO:tasks.ceph.mds.c.smithi028.stderr: -651> 2024-02-21T23:10:58.817+0000 7fcf9f400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-21T23:11:13.659 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-02-21T23:11:13.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.691+0000 7f7ea1c00640 1 Processor -- start 2024-02-21T23:11:13.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.691+0000 7f7ea1c00640 1 -- start start 2024-02-21T23:11:13.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.692+0000 7f7ea1c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7e9c1523f0 0x7f7e9c158dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:13.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.692+0000 7f7ea1c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7e9c154ed0 0x7f7e9c155360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:13.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.692+0000 7f7ea1c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e9c070570 con 0x7f7e9c072770 2024-02-21T23:11:13.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.692+0000 7f7ea1c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e9c1527d0 con 0x7f7e9c154ed0 2024-02-21T23:11:13.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.692+0000 7f7ea1c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e9c070810 con 0x7f7e9c1523f0 2024-02-21T23:11:13.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.692+0000 7f7ea1200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7e9c154ed0 0x7f7e9c155360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:13.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.692+0000 7f7ea1200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7e9c154ed0 0x7f7e9c155360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59642/0 (socket says 172.21.15.28:59642) 2024-02-21T23:11:13.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.692+0000 7f7ea1200640 1 -- 172.21.15.28:0/2863513044 learned_addr learned my addr 172.21.15.28:0/2863513044 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:13.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.692+0000 7f7e9b400640 1 --1- 172.21.15.28:0/2863513044 >> v1:172.21.15.28:6789/0 conn(0x7f7e9c072770 0x7f7e9c06ae70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:44046/0 (socket says 172.21.15.28:44046) 2024-02-21T23:11:13.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.693+0000 7f7e9be00640 1 --2- 172.21.15.28:0/2863513044 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7e9c1523f0 0x7f7e9c158dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:13.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.693+0000 7f7ea1200640 1 -- 172.21.15.28:0/2863513044 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7e9c1523f0 msgr2=0x7f7e9c158dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:13.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.693+0000 7f7ea1200640 1 --2- 172.21.15.28:0/2863513044 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7e9c1523f0 0x7f7e9c158dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:13.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.693+0000 7f7ea1200640 1 -- 172.21.15.28:0/2863513044 >> v1:172.21.15.28:6789/0 conn(0x7f7e9c072770 legacy=0x7f7e9c06ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:13.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.693+0000 7f7ea1200640 1 -- 172.21.15.28:0/2863513044 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e9c070af0 con 0x7f7e9c154ed0 2024-02-21T23:11:13.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.694+0000 7f7ea1200640 1 --2- 172.21.15.28:0/2863513044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e9c154ed0 0x7f7e9c155360 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7f7e94009e50 tx=0x7f7e9402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53c71386968ad4e9 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:13.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.694+0000 7f7e9aa00640 1 -- 172.21.15.28:0/2863513044 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e9402ca80 con 0x7f7e9c154ed0 2024-02-21T23:11:13.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.694+0000 7f7e9aa00640 1 -- 172.21.15.28:0/2863513044 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e9402cc20 con 0x7f7e9c154ed0 2024-02-21T23:11:13.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.694+0000 7f7e9aa00640 1 -- 172.21.15.28:0/2863513044 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e940343f0 con 0x7f7e9c154ed0 2024-02-21T23:11:13.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.695+0000 7f7ea1c00640 1 -- 172.21.15.28:0/2863513044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e9c154ed0 msgr2=0x7f7e9c155360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:13.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.695+0000 7f7ea1c00640 1 --2- 172.21.15.28:0/2863513044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e9c154ed0 0x7f7e9c155360 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7f7e94009e50 tx=0x7f7e9402bee0 comp rx=0 tx=0).stop 2024-02-21T23:11:13.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.695+0000 7f7ea1c00640 1 -- 172.21.15.28:0/2863513044 shutdown_connections 2024-02-21T23:11:13.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.695+0000 7f7ea1c00640 1 --2- 172.21.15.28:0/2863513044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e9c154ed0 0x7f7e9c155360 unknown :-1 s=CLOSED pgs=1963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:13.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.695+0000 7f7ea1c00640 1 --2- 172.21.15.28:0/2863513044 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7e9c1523f0 0x7f7e9c158dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:13.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.695+0000 7f7ea1c00640 1 -- 172.21.15.28:0/2863513044 >> 172.21.15.28:0/2863513044 conn(0x7f7e9c0c7720 msgr2=0x7f7e9c0c7b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:13.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.695+0000 7f7ea1c00640 1 -- 172.21.15.28:0/2863513044 shutdown_connections 2024-02-21T23:11:13.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.695+0000 7f7ea1c00640 1 -- 172.21.15.28:0/2863513044 wait complete. 2024-02-21T23:11:13.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.697+0000 7f7ea1c00640 1 Processor -- start 2024-02-21T23:11:13.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.697+0000 7f7ea1c00640 1 -- start start 2024-02-21T23:11:13.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.698+0000 7f7ea1c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7e9c072770 0x7f7e9c156bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:13.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.698+0000 7f7ea1c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e9c1523f0 0x7f7e9c1eae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:13.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.698+0000 7f7ea1c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e9c154ed0 0x7f7e9c1eb340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:13.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.698+0000 7f7ea1c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e9c06b580 con 0x7f7e9c154ed0 2024-02-21T23:11:13.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.698+0000 7f7ea1c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e9c0c3c60 con 0x7f7e9c1523f0 2024-02-21T23:11:13.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.698+0000 7f7ea1c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e9c06b700 con 0x7f7e9c072770 2024-02-21T23:11:13.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.698+0000 7f7ea1200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e9c154ed0 0x7f7e9c1eb340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:13.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.698+0000 7f7ea1200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e9c154ed0 0x7f7e9c1eb340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59648/0 (socket says 172.21.15.28:59648) 2024-02-21T23:11:13.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.698+0000 7f7ea1200640 1 -- 172.21.15.28:0/1953440923 learned_addr learned my addr 172.21.15.28:0/1953440923 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:13.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.698+0000 7f7ea1200640 1 -- 172.21.15.28:0/1953440923 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7e9c072770 msgr2=0x7f7e9c156bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:13.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.698+0000 7f7ea1200640 1 --2- 172.21.15.28:0/1953440923 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7e9c072770 0x7f7e9c156bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:13.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.698+0000 7f7ea1200640 1 -- 172.21.15.28:0/1953440923 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e9c1523f0 msgr2=0x7f7e9c1eae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:13.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.698+0000 7f7ea1200640 1 --2- 172.21.15.28:0/1953440923 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e9c1523f0 0x7f7e9c1eae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:13.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.698+0000 7f7ea1200640 1 -- 172.21.15.28:0/1953440923 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e9c1ee350 con 0x7f7e9c154ed0 2024-02-21T23:11:13.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.699+0000 7f7ea1200640 1 --2- 172.21.15.28:0/1953440923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e9c154ed0 0x7f7e9c1eb340 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7f7e94009e50 tx=0x7f7e94004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:13.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.699+0000 7f7e93e00640 1 -- 172.21.15.28:0/1953440923 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e94043020 con 0x7f7e9c154ed0 2024-02-21T23:11:13.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.699+0000 7f7e93e00640 1 -- 172.21.15.28:0/1953440923 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e9403e450 con 0x7f7e9c154ed0 2024-02-21T23:11:13.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.699+0000 7f7e93e00640 1 -- 172.21.15.28:0/1953440923 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e94009510 con 0x7f7e9c154ed0 2024-02-21T23:11:13.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.699+0000 7f7ea1c00640 1 -- 172.21.15.28:0/1953440923 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e9c1ee630 con 0x7f7e9c154ed0 2024-02-21T23:11:13.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.699+0000 7f7ea1c00640 1 -- 172.21.15.28:0/1953440923 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e9c1eebf0 con 0x7f7e9c154ed0 2024-02-21T23:11:13.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.700+0000 7f7ea1c00640 1 -- 172.21.15.28:0/1953440923 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e9c04f290 con 0x7f7e9c154ed0 2024-02-21T23:11:13.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.701+0000 7f7e93e00640 1 -- 172.21.15.28:0/1953440923 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7e94046020 con 0x7f7e9c154ed0 2024-02-21T23:11:13.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.702+0000 7f7e93e00640 1 --2- 172.21.15.28:0/1953440923 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7e7c06b500 0x7f7e7c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:13.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.702+0000 7f7e93e00640 1 -- 172.21.15.28:0/1953440923 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f7e940acba0 con 0x7f7e9c154ed0 2024-02-21T23:11:13.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.703+0000 7f7e9be00640 1 --2- 172.21.15.28:0/1953440923 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7e7c06b500 0x7f7e7c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:13.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.704+0000 7f7e9be00640 1 --2- 172.21.15.28:0/1953440923 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7e7c06b500 0x7f7e7c06d9c0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f7e9c0736b0 tx=0x7f7e8c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:13.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.707+0000 7f7e93e00640 1 -- 172.21.15.28:0/1953440923 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7e9404d030 con 0x7f7e9c154ed0 2024-02-21T23:11:13.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.949+0000 7f7ea1c00640 1 -- 172.21.15.28:0/1953440923 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e9c0c2220 con 0x7f7e9c154ed0 2024-02-21T23:11:13.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.950+0000 7f7e93e00640 1 -- 172.21.15.28:0/1953440923 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 188 v188) v1 ==== 95+0+4143 (secure 0 0 0) 0x7f7e9407b880 con 0x7f7e9c154ed0 2024-02-21T23:11:13.938 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:11:13.938 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":188,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6686,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/155806034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":155806034},{"type":"v1","addr":"172.21.15.81:6836","nonce":155806034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":180},{"gid":7480,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3897243745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":3897243745},{"type":"v1","addr":"172.21.15.81:6837","nonce":3897243745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":180},{"gid":7486,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2827135632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2827135632},{"type":"v1","addr":"172.21.15.28:6835","nonce":2827135632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":180}],"filesystems":[{"mdsmap":{"epoch":188,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:10:53.351225+0000","modified":"2024-02-21T23:11:13.391452+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[],"up":{},"failed":[],"damaged":[],"stopped":[0],"info":{},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2024-02-21T23:11:13.938 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 188 2024-02-21T23:11:13.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.953+0000 7f7ea1c00640 1 -- 172.21.15.28:0/1953440923 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7e7c06b500 msgr2=0x7f7e7c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:13.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.953+0000 7f7ea1c00640 1 --2- 172.21.15.28:0/1953440923 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7e7c06b500 0x7f7e7c06d9c0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f7e9c0736b0 tx=0x7f7e8c008040 comp rx=0 tx=0).stop 2024-02-21T23:11:13.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.953+0000 7f7ea1c00640 1 -- 172.21.15.28:0/1953440923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e9c154ed0 msgr2=0x7f7e9c1eb340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:13.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.953+0000 7f7ea1c00640 1 --2- 172.21.15.28:0/1953440923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e9c154ed0 0x7f7e9c1eb340 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7f7e94009e50 tx=0x7f7e94004a40 comp rx=0 tx=0).stop 2024-02-21T23:11:13.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.953+0000 7f7ea1c00640 1 -- 172.21.15.28:0/1953440923 shutdown_connections 2024-02-21T23:11:13.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.953+0000 7f7ea1c00640 1 --2- 172.21.15.28:0/1953440923 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7e7c06b500 0x7f7e7c06d9c0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:13.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.953+0000 7f7ea1c00640 1 --2- 172.21.15.28:0/1953440923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e9c154ed0 0x7f7e9c1eb340 unknown :-1 s=CLOSED pgs=1964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:13.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.953+0000 7f7ea1c00640 1 --2- 172.21.15.28:0/1953440923 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e9c1523f0 0x7f7e9c1eae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:13.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.953+0000 7f7ea1c00640 1 --2- 172.21.15.28:0/1953440923 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7e9c072770 0x7f7e9c156bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:13.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.953+0000 7f7ea1c00640 1 -- 172.21.15.28:0/1953440923 >> 172.21.15.28:0/1953440923 conn(0x7f7e9c0c7720 msgr2=0x7f7e9c06ae70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:13.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.953+0000 7f7ea1c00640 1 -- 172.21.15.28:0/1953440923 shutdown_connections 2024-02-21T23:11:13.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:13.953+0000 7f7ea1c00640 1 -- 172.21.15.28:0/1953440923 wait complete. 2024-02-21T23:11:14.362 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 188, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:10:53.351225+0000', 'modified': '2024-02-21T23:11:13.391452+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [], 'up': {}, 'failed': [], 'damaged': [], 'stopped': [0], 'info': {}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:11:14.362 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/0 2024-02-21T23:11:14.363 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:11:14.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:14.604+0000 7f1bada00640 1 Processor -- start 2024-02-21T23:11:14.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:14.604+0000 7f1bada00640 1 -- start start 2024-02-21T23:11:14.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:14.605+0000 7f1bada00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1ba80c8560 0x7f1ba80c7ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:14.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:14.605+0000 7f1bada00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ba80c2c20 con 0x7f1ba80c8940 2024-02-21T23:11:14.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:14.606+0000 7f1bada00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ba8155100 con 0x7f1ba80c64d0 2024-02-21T23:11:14.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:14.606+0000 7f1bada00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba80c8dd0 con 0x7f1ba80c8560 2024-02-21T23:11:14.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:14.606+0000 7f1bad000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f1ba80c8940 0x7f1ba80c54a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:44048/0 (socket says 172.21.15.28:44048) 2024-02-21T23:11:14.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:14.606+0000 7f1bad000640 1 -- 172.21.15.28:0/1856135439 learned_addr learned my addr 172.21.15.28:0/1856135439 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:11:14.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:14.606+0000 7f1ba7e00640 1 --2- 172.21.15.28:0/1856135439 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1ba80c8560 0x7f1ba80c7ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:14.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:14.607+0000 7f1ba7e00640 1 -- 172.21.15.28:0/1856135439 >> v1:172.21.15.81:6789/0 conn(0x7f1ba80c64d0 legacy=0x7f1ba80c4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:14.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:14.607+0000 7f1ba7e00640 1 -- 172.21.15.28:0/1856135439 >> v1:172.21.15.28:6789/0 conn(0x7f1ba80c8940 legacy=0x7f1ba80c54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:14.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:14.607+0000 7f1ba7e00640 1 -- 172.21.15.28:0/1856135439 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ba81553e0 con 0x7f1ba80c8560 2024-02-21T23:11:14.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:14.609+0000 7f1ba7e00640 1 --2- 172.21.15.28:0/1856135439 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1ba80c8560 0x7f1ba80c7ef0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f1b980097d0 tx=0x7f1b9802b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bcca67b2a5956196 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:14.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:14.615+0000 7f1ba6a00640 1 -- 172.21.15.28:0/1856135439 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b98035020 con 0x7f1ba80c8560 2024-02-21T23:11:14.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:14.615+0000 7f1ba6a00640 1 -- 172.21.15.28:0/1856135439 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b98031800 con 0x7f1ba80c8560 2024-02-21T23:11:14.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:14.615+0000 7f1ba6a00640 1 -- 172.21.15.28:0/1856135439 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b98031bd0 con 0x7f1ba80c8560 2024-02-21T23:11:14.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:14.616+0000 7f1bada00640 1 -- 172.21.15.28:0/1856135439 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1ba80c8560 msgr2=0x7f1ba80c7ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:14.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:14.616+0000 7f1bada00640 1 --2- 172.21.15.28:0/1856135439 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1ba80c8560 0x7f1ba80c7ef0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f1b980097d0 tx=0x7f1b9802b8a0 comp rx=0 tx=0).stop 2024-02-21T23:11:14.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:14.616+0000 7f1bada00640 1 -- 172.21.15.28:0/1856135439 shutdown_connections 2024-02-21T23:11:14.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:14.616+0000 7f1bada00640 1 --2- 172.21.15.28:0/1856135439 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1ba80c8560 0x7f1ba80c7ef0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:14.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:14.616+0000 7f1bada00640 1 -- 172.21.15.28:0/1856135439 >> 172.21.15.28:0/1856135439 conn(0x7f1ba806ae70 msgr2=0x7f1ba806b280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:14.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:14.616+0000 7f1bada00640 1 -- 172.21.15.28:0/1856135439 shutdown_connections 2024-02-21T23:11:14.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:14.616+0000 7f1bada00640 1 -- 172.21.15.28:0/1856135439 wait complete. 2024-02-21T23:11:14.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:14.618+0000 7f1bada00640 1 Processor -- start 2024-02-21T23:11:14.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:14.618+0000 7f1bada00640 1 -- start start 2024-02-21T23:11:14.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:14.619+0000 7f1bada00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ba80c64d0 0x7f1ba80a8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:14.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:14.619+0000 7f1bada00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1ba80c8560 0x7f1ba80ab530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:14.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:14.619+0000 7f1bada00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ba80c8940 0x7f1ba8156770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:14.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:14.619+0000 7f1bada00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba81559a0 con 0x7f1ba80c64d0 2024-02-21T23:11:14.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:14.619+0000 7f1bada00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba80c6910 con 0x7f1ba80c8940 2024-02-21T23:11:14.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:14.619+0000 7f1bada00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba8154db0 con 0x7f1ba80c8560 2024-02-21T23:11:14.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:14.620+0000 7f1ba7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ba80c64d0 0x7f1ba80a8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:14.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:14.620+0000 7f1ba7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ba80c64d0 0x7f1ba80a8160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59650/0 (socket says 172.21.15.28:59650) 2024-02-21T23:11:14.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:14.620+0000 7f1ba7e00640 1 -- 172.21.15.28:0/3522115026 learned_addr learned my addr 172.21.15.28:0/3522115026 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:14.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:14.620+0000 7f1ba7400640 1 --2- 172.21.15.28:0/3522115026 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1ba80c8560 0x7f1ba80ab530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:14.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:14.620+0000 7f1bad000640 1 --2- 172.21.15.28:0/3522115026 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ba80c8940 0x7f1ba8156770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:14.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:14.620+0000 7f1ba7e00640 1 -- 172.21.15.28:0/3522115026 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1ba80c8560 msgr2=0x7f1ba80ab530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:14.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:14.620+0000 7f1ba7e00640 1 --2- 172.21.15.28:0/3522115026 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1ba80c8560 0x7f1ba80ab530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:14.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:14.620+0000 7f1ba7e00640 1 -- 172.21.15.28:0/3522115026 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ba80c8940 msgr2=0x7f1ba8156770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:14.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:14.620+0000 7f1ba7e00640 1 --2- 172.21.15.28:0/3522115026 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ba80c8940 0x7f1ba8156770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:14.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:14.620+0000 7f1ba7e00640 1 -- 172.21.15.28:0/3522115026 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ba80aa750 con 0x7f1ba80c64d0 2024-02-21T23:11:14.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:14.620+0000 7f1ba7400640 1 --2- 172.21.15.28:0/3522115026 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1ba80c8560 0x7f1ba80ab530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:11:14.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:14.621+0000 7f1ba7e00640 1 --2- 172.21.15.28:0/3522115026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ba80c64d0 0x7f1ba80a8160 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 crypto rx=0x7f1b980098f0 tx=0x7f1b980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:14.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:14.621+0000 7f1b9fe00640 1 -- 172.21.15.28:0/3522115026 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b98041070 con 0x7f1ba80c64d0 2024-02-21T23:11:14.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:14.621+0000 7f1b9fe00640 1 -- 172.21.15.28:0/3522115026 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b9803ad00 con 0x7f1ba80c64d0 2024-02-21T23:11:14.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:14.621+0000 7f1b9fe00640 1 -- 172.21.15.28:0/3522115026 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b98035040 con 0x7f1ba80c64d0 2024-02-21T23:11:14.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:14.621+0000 7f1bada00640 1 -- 172.21.15.28:0/3522115026 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ba80ad420 con 0x7f1ba80c64d0 2024-02-21T23:11:14.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:14.621+0000 7f1bada00640 1 -- 172.21.15.28:0/3522115026 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ba80aaa10 con 0x7f1ba80c64d0 2024-02-21T23:11:14.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:14.623+0000 7f1bada00640 1 -- 172.21.15.28:0/3522115026 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ba804f290 con 0x7f1ba80c64d0 2024-02-21T23:11:14.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:14.624+0000 7f1b9fe00640 1 -- 172.21.15.28:0/3522115026 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1b9804c060 con 0x7f1ba80c64d0 2024-02-21T23:11:14.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:14.625+0000 7f1b9fe00640 1 --2- 172.21.15.28:0/3522115026 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1b8806b4d0 0x7f1b8806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:14.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:14.625+0000 7f1b9fe00640 1 -- 172.21.15.28:0/3522115026 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1b980ac060 con 0x7f1ba80c64d0 2024-02-21T23:11:14.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:14.626+0000 7f1ba7400640 1 --2- 172.21.15.28:0/3522115026 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1b8806b4d0 0x7f1b8806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:14.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:14.627+0000 7f1ba7400640 1 --2- 172.21.15.28:0/3522115026 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1b8806b4d0 0x7f1b8806d990 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f1ba80adad0 tx=0x7f1b94009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:14.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:14.630+0000 7f1b9fe00640 1 -- 172.21.15.28:0/3522115026 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1b9807ad40 con 0x7f1ba80c64d0 2024-02-21T23:11:14.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:14.871+0000 7f1bada00640 1 -- 172.21.15.28:0/3522115026 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1ba80c1f40 con 0x7f1ba80c64d0 2024-02-21T23:11:14.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:14.872+0000 7f1b9fe00640 1 -- 172.21.15.28:0/3522115026 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 189 v189) v1 ==== 95+0+4935 (secure 0 0 0) 0x7f1b9807a640 con 0x7f1ba80c64d0 2024-02-21T23:11:14.861 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:11:14.861 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":189,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6686,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/155806034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":155806034},{"type":"v1","addr":"172.21.15.81:6836","nonce":155806034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":180},{"gid":7480,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3897243745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":3897243745},{"type":"v1","addr":"172.21.15.81:6837","nonce":3897243745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":180},{"gid":7486,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2827135632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2827135632},{"type":"v1","addr":"172.21.15.28:6835","nonce":2827135632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":180},{"gid":9699,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/2364631414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2364631414},{"type":"v1","addr":"172.21.15.28:6833","nonce":2364631414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":189}],"filesystems":[{"mdsmap":{"epoch":188,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:10:53.351225+0000","modified":"2024-02-21T23:11:13.391452+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[],"up":{},"failed":[],"damaged":[],"stopped":[0],"info":{},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2024-02-21T23:11:14.861 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 189 2024-02-21T23:11:14.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:14.875+0000 7f1bada00640 1 -- 172.21.15.28:0/3522115026 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1b8806b4d0 msgr2=0x7f1b8806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:14.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:14.875+0000 7f1bada00640 1 --2- 172.21.15.28:0/3522115026 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1b8806b4d0 0x7f1b8806d990 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f1ba80adad0 tx=0x7f1b94009040 comp rx=0 tx=0).stop 2024-02-21T23:11:14.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:14.875+0000 7f1bada00640 1 -- 172.21.15.28:0/3522115026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ba80c64d0 msgr2=0x7f1ba80a8160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:14.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:14.875+0000 7f1bada00640 1 --2- 172.21.15.28:0/3522115026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ba80c64d0 0x7f1ba80a8160 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 crypto rx=0x7f1b980098f0 tx=0x7f1b980046c0 comp rx=0 tx=0).stop 2024-02-21T23:11:14.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:14.875+0000 7f1bada00640 1 -- 172.21.15.28:0/3522115026 shutdown_connections 2024-02-21T23:11:14.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:14.875+0000 7f1bada00640 1 --2- 172.21.15.28:0/3522115026 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1b8806b4d0 0x7f1b8806d990 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:14.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:14.875+0000 7f1bada00640 1 --2- 172.21.15.28:0/3522115026 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1ba80c8560 0x7f1ba80ab530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:14.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:14.875+0000 7f1bada00640 1 --2- 172.21.15.28:0/3522115026 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ba80c8940 0x7f1ba8156770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:14.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:14.875+0000 7f1bada00640 1 --2- 172.21.15.28:0/3522115026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ba80c64d0 0x7f1ba80a8160 unknown :-1 s=CLOSED pgs=1966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:14.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:14.875+0000 7f1bada00640 1 -- 172.21.15.28:0/3522115026 >> 172.21.15.28:0/3522115026 conn(0x7f1ba806ae70 msgr2=0x7f1ba8153c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:14.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:14.876+0000 7f1bada00640 1 -- 172.21.15.28:0/3522115026 shutdown_connections 2024-02-21T23:11:14.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:14.876+0000 7f1bada00640 1 -- 172.21.15.28:0/3522115026 wait complete. 2024-02-21T23:11:14.885 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs down false 2024-02-21T23:11:15.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.114+0000 7fe2dd600640 1 Processor -- start 2024-02-21T23:11:15.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.115+0000 7fe2dd600640 1 -- start start 2024-02-21T23:11:15.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.116+0000 7fe2dd600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fe2d813e0c0 0x7fe2d80c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:15.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.116+0000 7fe2dd600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe2d806dbc0 0x7fe2d806c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:15.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.116+0000 7fe2dd600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2d8158550 con 0x7fe2d80c76f0 2024-02-21T23:11:15.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.116+0000 7fe2dd600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2d80c89a0 con 0x7fe2d806dbc0 2024-02-21T23:11:15.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.116+0000 7fe2dd600640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2d81587d0 con 0x7fe2d813e0c0 2024-02-21T23:11:15.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.116+0000 7fe2d7400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe2d806dbc0 0x7fe2d806c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:15.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.116+0000 7fe2d7400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe2d806dbc0 0x7fe2d806c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58778/0 (socket says 172.21.15.28:58778) 2024-02-21T23:11:15.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.116+0000 7fe2d7400640 1 -- 172.21.15.28:0/2597526905 learned_addr learned my addr 172.21.15.28:0/2597526905 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:15.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.117+0000 7fe2d7e00640 1 --2- 172.21.15.28:0/2597526905 >> v2:172.21.15.81:3300/0 conn(0x7fe2d813e0c0 0x7fe2d80c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:15.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.117+0000 7fe2d7400640 1 -- 172.21.15.28:0/2597526905 >> v2:172.21.15.81:3300/0 conn(0x7fe2d813e0c0 msgr2=0x7fe2d80c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:15.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.117+0000 7fe2d7400640 1 --2- 172.21.15.28:0/2597526905 >> v2:172.21.15.81:3300/0 conn(0x7fe2d813e0c0 0x7fe2d80c71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:15.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.117+0000 7fe2d7400640 1 -- 172.21.15.28:0/2597526905 >> v1:172.21.15.81:6789/0 conn(0x7fe2d80c76f0 legacy=0x7fe2d806c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:15.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.117+0000 7fe2d7400640 1 -- 172.21.15.28:0/2597526905 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2d8158ab0 con 0x7fe2d806dbc0 2024-02-21T23:11:15.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.118+0000 7fe2d7400640 1 --2- 172.21.15.28:0/2597526905 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2d806dbc0 0x7fe2d806c280 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7fe2d80ad180 tx=0x7fe2c402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d91c433338923694 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:15.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.118+0000 7fe2d6a00640 1 -- 172.21.15.28:0/2597526905 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c402e070 con 0x7fe2d806dbc0 2024-02-21T23:11:15.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.118+0000 7fe2d6a00640 1 -- 172.21.15.28:0/2597526905 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2c4037070 con 0x7fe2d806dbc0 2024-02-21T23:11:15.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.118+0000 7fe2d6a00640 1 -- 172.21.15.28:0/2597526905 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c4031a00 con 0x7fe2d806dbc0 2024-02-21T23:11:15.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.119+0000 7fe2dd600640 1 -- 172.21.15.28:0/2597526905 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2d806dbc0 msgr2=0x7fe2d806c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:15.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.119+0000 7fe2dd600640 1 --2- 172.21.15.28:0/2597526905 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2d806dbc0 0x7fe2d806c280 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7fe2d80ad180 tx=0x7fe2c402b880 comp rx=0 tx=0).stop 2024-02-21T23:11:15.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.119+0000 7fe2dd600640 1 -- 172.21.15.28:0/2597526905 shutdown_connections 2024-02-21T23:11:15.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.119+0000 7fe2dd600640 1 --2- 172.21.15.28:0/2597526905 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2d806dbc0 0x7fe2d806c280 unknown :-1 s=CLOSED pgs=1967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:15.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.119+0000 7fe2dd600640 1 --2- 172.21.15.28:0/2597526905 >> v2:172.21.15.81:3300/0 conn(0x7fe2d813e0c0 0x7fe2d80c71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:15.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.119+0000 7fe2dd600640 1 -- 172.21.15.28:0/2597526905 >> 172.21.15.28:0/2597526905 conn(0x7fe2d8073110 msgr2=0x7fe2d80ac490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:15.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.119+0000 7fe2dd600640 1 -- 172.21.15.28:0/2597526905 shutdown_connections 2024-02-21T23:11:15.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.119+0000 7fe2dd600640 1 -- 172.21.15.28:0/2597526905 wait complete. 2024-02-21T23:11:15.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.121+0000 7fe2dd600640 1 Processor -- start 2024-02-21T23:11:15.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.121+0000 7fe2dd600640 1 -- start start 2024-02-21T23:11:15.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.122+0000 7fe2dd600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2d806dbc0 0x7fe2d8137210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:15.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.122+0000 7fe2dd600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2d80c76f0 0x7fe2d81ea9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:15.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.122+0000 7fe2dd600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe2d813e0c0 0x7fe2d81543e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:15.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.122+0000 7fe2dd600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2d806ced0 con 0x7fe2d80c76f0 2024-02-21T23:11:15.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.122+0000 7fe2dd600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2d8159070 con 0x7fe2d806dbc0 2024-02-21T23:11:15.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.122+0000 7fe2dd600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2d813e520 con 0x7fe2d813e0c0 2024-02-21T23:11:15.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.122+0000 7fe2d7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2d80c76f0 0x7fe2d81ea9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:15.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.122+0000 7fe2d7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2d80c76f0 0x7fe2d81ea9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58794/0 (socket says 172.21.15.28:58794) 2024-02-21T23:11:15.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.122+0000 7fe2d7400640 1 -- 172.21.15.28:0/3545783105 learned_addr learned my addr 172.21.15.28:0/3545783105 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:15.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.123+0000 7fe2d7e00640 1 --2- 172.21.15.28:0/3545783105 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2d806dbc0 0x7fe2d8137210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:15.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.123+0000 7fe2dcc00640 1 --2- 172.21.15.28:0/3545783105 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe2d813e0c0 0x7fe2d81543e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:15.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.123+0000 7fe2d7400640 1 -- 172.21.15.28:0/3545783105 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe2d813e0c0 msgr2=0x7fe2d81543e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:15.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.123+0000 7fe2d7400640 1 --2- 172.21.15.28:0/3545783105 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe2d813e0c0 0x7fe2d81543e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:15.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.123+0000 7fe2d7400640 1 -- 172.21.15.28:0/3545783105 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2d806dbc0 msgr2=0x7fe2d8137210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:15.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.123+0000 7fe2d7400640 1 --2- 172.21.15.28:0/3545783105 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2d806dbc0 0x7fe2d8137210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:15.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.123+0000 7fe2d7400640 1 -- 172.21.15.28:0/3545783105 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2d81edf10 con 0x7fe2d80c76f0 2024-02-21T23:11:15.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.123+0000 7fe2dcc00640 1 --2- 172.21.15.28:0/3545783105 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe2d813e0c0 0x7fe2d81543e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:11:15.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.123+0000 7fe2d7400640 1 --2- 172.21.15.28:0/3545783105 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2d80c76f0 0x7fe2d81ea9a0 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7fe2c402b8c0 tx=0x7fe2c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:15.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.123+0000 7fe2cfe00640 1 -- 172.21.15.28:0/3545783105 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c4032670 con 0x7fe2d80c76f0 2024-02-21T23:11:15.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.124+0000 7fe2cfe00640 1 -- 172.21.15.28:0/3545783105 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2c4032830 con 0x7fe2d80c76f0 2024-02-21T23:11:15.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.124+0000 7fe2cfe00640 1 -- 172.21.15.28:0/3545783105 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c4032c20 con 0x7fe2d80c76f0 2024-02-21T23:11:15.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.124+0000 7fe2dd600640 1 -- 172.21.15.28:0/3545783105 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2d80af430 con 0x7fe2d80c76f0 2024-02-21T23:11:15.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.124+0000 7fe2dd600640 1 -- 172.21.15.28:0/3545783105 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2d80af9f0 con 0x7fe2d80c76f0 2024-02-21T23:11:15.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.125+0000 7fe2dd600640 1 -- 172.21.15.28:0/3545783105 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2d80ac870 con 0x7fe2d80c76f0 2024-02-21T23:11:15.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.126+0000 7fe2cfe00640 1 -- 172.21.15.28:0/3545783105 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe2c404c060 con 0x7fe2d80c76f0 2024-02-21T23:11:15.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.126+0000 7fe2cfe00640 1 --2- 172.21.15.28:0/3545783105 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe2b806b4d0 0x7fe2b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:15.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.127+0000 7fe2cfe00640 1 -- 172.21.15.28:0/3545783105 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe2c40abd50 con 0x7fe2d80c76f0 2024-02-21T23:11:15.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.127+0000 7fe2d7e00640 1 --2- 172.21.15.28:0/3545783105 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe2b806b4d0 0x7fe2b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:15.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.128+0000 7fe2d7e00640 1 --2- 172.21.15.28:0/3545783105 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe2b806b4d0 0x7fe2b806d990 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fe2d80ac480 tx=0x7fe2c80039b0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:15.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.131+0000 7fe2cfe00640 1 -- 172.21.15.28:0/3545783105 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe2d80ac870 con 0x7fe2d80c76f0 2024-02-21T23:11:15.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.373+0000 7fe2dd600640 1 -- 172.21.15.28:0/3545783105 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "down", "val": "false"} v 0) v1 -- 0x7fe2d804f290 con 0x7fe2d80c76f0 2024-02-21T23:11:15.605 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-02-21T23:11:15.616+0000 7f5ddf600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-21T23:11:15.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.627+0000 7fe2cfe00640 1 -- 172.21.15.28:0/3545783105 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "down", "val": "false"}]=0 cephfs marked up, max_mds = 1 v190) v1 ==== 135+0+0 (secure 0 0 0) 0x7fe2c40af030 con 0x7fe2d80c76f0 2024-02-21T23:11:15.616 INFO:teuthology.orchestra.run.smithi028.stderr:cephfs marked up, max_mds = 1 2024-02-21T23:11:15.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.631+0000 7fe2dd600640 1 -- 172.21.15.28:0/3545783105 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe2b806b4d0 msgr2=0x7fe2b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:15.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.631+0000 7fe2dd600640 1 --2- 172.21.15.28:0/3545783105 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe2b806b4d0 0x7fe2b806d990 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fe2d80ac480 tx=0x7fe2c80039b0 comp rx=0 tx=0).stop 2024-02-21T23:11:15.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.631+0000 7fe2dd600640 1 -- 172.21.15.28:0/3545783105 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2d80c76f0 msgr2=0x7fe2d81ea9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:15.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.631+0000 7fe2dd600640 1 --2- 172.21.15.28:0/3545783105 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2d80c76f0 0x7fe2d81ea9a0 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7fe2c402b8c0 tx=0x7fe2c40046c0 comp rx=0 tx=0).stop 2024-02-21T23:11:15.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.631+0000 7fe2dd600640 1 -- 172.21.15.28:0/3545783105 shutdown_connections 2024-02-21T23:11:15.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.631+0000 7fe2dd600640 1 --2- 172.21.15.28:0/3545783105 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe2b806b4d0 0x7fe2b806d990 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:15.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.631+0000 7fe2dd600640 1 --2- 172.21.15.28:0/3545783105 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe2d813e0c0 0x7fe2d81543e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:15.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.631+0000 7fe2dd600640 1 --2- 172.21.15.28:0/3545783105 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2d80c76f0 0x7fe2d81ea9a0 unknown :-1 s=CLOSED pgs=1968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:15.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.631+0000 7fe2dd600640 1 --2- 172.21.15.28:0/3545783105 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2d806dbc0 0x7fe2d8137210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:15.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.631+0000 7fe2dd600640 1 -- 172.21.15.28:0/3545783105 >> 172.21.15.28:0/3545783105 conn(0x7fe2d8073110 msgr2=0x7fe2d80c5f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:15.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.631+0000 7fe2dd600640 1 -- 172.21.15.28:0/3545783105 shutdown_connections 2024-02-21T23:11:15.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.631+0000 7fe2dd600640 1 -- 172.21.15.28:0/3545783105 wait complete. 2024-02-21T23:11:15.642 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:11:15.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.850+0000 7f970f000640 1 Processor -- start 2024-02-21T23:11:15.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.851+0000 7f970f000640 1 -- start start 2024-02-21T23:11:15.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.852+0000 7f970f000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f97080aff90 0x7f97080c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:15.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.852+0000 7f970f000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f970806dbf0 0x7f9708158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:15.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.852+0000 7f970f000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97080c74c0 con 0x7f97080c5720 2024-02-21T23:11:15.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.852+0000 7f970f000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97080c7270 con 0x7f97080aff90 2024-02-21T23:11:15.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.852+0000 7f970f000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9708158630 con 0x7f970806dbf0 2024-02-21T23:11:15.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.852+0000 7f970e600640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f97080c5720 0x7f970806c270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37836/0 (socket says 172.21.15.28:37836) 2024-02-21T23:11:15.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.852+0000 7f970e600640 1 -- 172.21.15.28:0/2866630777 learned_addr learned my addr 172.21.15.28:0/2866630777 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:11:15.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.852+0000 7f970d200640 1 --2- 172.21.15.28:0/2866630777 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f970806dbf0 0x7f9708158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:15.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.853+0000 7f970dc00640 1 --2- 172.21.15.28:0/2866630777 >> v2:172.21.15.81:3300/0 conn(0x7f97080aff90 0x7f97080c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:15.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.853+0000 7f9707e00640 1 -- 172.21.15.28:0/2866630777 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 831271222 0 0) 0x7f97080c74c0 con 0x7f97080c5720 2024-02-21T23:11:15.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.853+0000 7f9707e00640 1 -- 172.21.15.28:0/2866630777 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96f0003520 con 0x7f97080c5720 2024-02-21T23:11:15.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.853+0000 7f9707e00640 1 -- 172.21.15.28:0/2866630777 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2645036568 0 0) 0x7f96f0003520 con 0x7f97080c5720 2024-02-21T23:11:15.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.853+0000 7f9707e00640 1 -- 172.21.15.28:0/2866630777 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f970806dbf0 msgr2=0x7f9708158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:15.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.853+0000 7f9707e00640 1 --2- 172.21.15.28:0/2866630777 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f970806dbf0 0x7f9708158ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:15.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.853+0000 7f9707e00640 1 -- 172.21.15.28:0/2866630777 >> v2:172.21.15.81:3300/0 conn(0x7f97080aff90 msgr2=0x7f97080c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:15.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.853+0000 7f9707e00640 1 --2- 172.21.15.28:0/2866630777 >> v2:172.21.15.81:3300/0 conn(0x7f97080aff90 0x7f97080c51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:15.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.853+0000 7f9707e00640 1 -- 172.21.15.28:0/2866630777 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97080c77a0 con 0x7f97080c5720 2024-02-21T23:11:15.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.853+0000 7f9707e00640 1 -- 172.21.15.28:0/2866630777 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f97000019f0 con 0x7f97080c5720 2024-02-21T23:11:15.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.853+0000 7f970dc00640 1 --2- 172.21.15.28:0/2866630777 >> v2:172.21.15.81:3300/0 conn(0x7f97080aff90 0x7f97080c51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:11:15.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.853+0000 7f9707e00640 1 -- 172.21.15.28:0/2866630777 >> v1:172.21.15.28:6789/0 conn(0x7f97080c5720 legacy=0x7f970806c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:15.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.853+0000 7f9707e00640 1 --2- 172.21.15.28:0/2866630777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96f00039b0 0x7f96f0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:15.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.853+0000 7f9707e00640 1 --2- 172.21.15.28:0/2866630777 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f970806dbf0 0x7f96f0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:15.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.853+0000 7f9707e00640 1 --2- 172.21.15.28:0/2866630777 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f96f0004d40 0x7f96f00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:15.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.853+0000 7f9707e00640 1 -- 172.21.15.28:0/2866630777 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97080c7270 con 0x7f96f00039b0 2024-02-21T23:11:15.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.853+0000 7f9707e00640 1 -- 172.21.15.28:0/2866630777 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9708158630 con 0x7f96f0004d40 2024-02-21T23:11:15.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.853+0000 7f9707e00640 1 -- 172.21.15.28:0/2866630777 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96f000ba40 con 0x7f970806dbf0 2024-02-21T23:11:15.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.854+0000 7f970dc00640 1 --2- 172.21.15.28:0/2866630777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96f00039b0 0x7f96f0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:15.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.854+0000 7f970dc00640 1 -- 172.21.15.28:0/2866630777 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f970806dbf0 msgr2=0x7f96f0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:15.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.854+0000 7f970d200640 1 --2- 172.21.15.28:0/2866630777 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f970806dbf0 0x7f96f0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:15.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.854+0000 7f970e600640 1 --2- 172.21.15.28:0/2866630777 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f96f0004d40 0x7f96f00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:15.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.854+0000 7f970dc00640 1 --2- 172.21.15.28:0/2866630777 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f970806dbf0 0x7f96f0004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:15.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.854+0000 7f970dc00640 1 -- 172.21.15.28:0/2866630777 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f96f0004d40 msgr2=0x7f96f00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:15.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.854+0000 7f970dc00640 1 --2- 172.21.15.28:0/2866630777 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f96f0004d40 0x7f96f00051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:15.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.854+0000 7f970dc00640 1 -- 172.21.15.28:0/2866630777 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96f0003520 con 0x7f96f00039b0 2024-02-21T23:11:15.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.855+0000 7f970dc00640 1 --2- 172.21.15.28:0/2866630777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96f00039b0 0x7f96f0003dd0 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7f96fc0098a0 tx=0x7f96f0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a3ef4600d4082cb server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:15.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.855+0000 7f9707e00640 1 -- 172.21.15.28:0/2866630777 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96fc00f040 con 0x7f96f00039b0 2024-02-21T23:11:15.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.855+0000 7f9707e00640 1 -- 172.21.15.28:0/2866630777 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96fc004230 con 0x7f96f00039b0 2024-02-21T23:11:15.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.855+0000 7f9707e00640 1 -- 172.21.15.28:0/2866630777 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96fc003610 con 0x7f96f00039b0 2024-02-21T23:11:15.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.856+0000 7f970f000640 1 -- 172.21.15.28:0/2866630777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96f00039b0 msgr2=0x7f96f0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:15.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.856+0000 7f970f000640 1 --2- 172.21.15.28:0/2866630777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96f00039b0 0x7f96f0003dd0 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7f96fc0098a0 tx=0x7f96f0005830 comp rx=0 tx=0).stop 2024-02-21T23:11:15.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.856+0000 7f970e600640 1 -- 172.21.15.28:0/2866630777 reap_dead start 2024-02-21T23:11:15.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.856+0000 7f970f000640 1 -- 172.21.15.28:0/2866630777 shutdown_connections 2024-02-21T23:11:15.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.856+0000 7f970f000640 1 -- 172.21.15.28:0/2866630777 >> 172.21.15.28:0/2866630777 conn(0x7f97080ac370 msgr2=0x7f97080ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:15.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.856+0000 7f970f000640 1 -- 172.21.15.28:0/2866630777 shutdown_connections 2024-02-21T23:11:15.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.856+0000 7f970f000640 1 -- 172.21.15.28:0/2866630777 wait complete. 2024-02-21T23:11:15.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.858+0000 7f970f000640 1 Processor -- start 2024-02-21T23:11:15.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.858+0000 7f970f000640 1 -- start start 2024-02-21T23:11:15.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.859+0000 7f970f000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f96f00039b0 0x7f970814f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:15.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.859+0000 7f970f000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f97080c1930 0x7f97080aea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:15.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.859+0000 7f970f000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97080c1200 0x7f97081eaf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:15.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.859+0000 7f970f000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97080c8760 con 0x7f97080c1200 2024-02-21T23:11:15.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.859+0000 7f970f000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97080be0c0 con 0x7f97080c1930 2024-02-21T23:11:15.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.859+0000 7f970f000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97080c88e0 con 0x7f96f00039b0 2024-02-21T23:11:15.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.859+0000 7f970e600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97080c1200 0x7f97081eaf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:15.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.859+0000 7f970e600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97080c1200 0x7f97081eaf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58812/0 (socket says 172.21.15.28:58812) 2024-02-21T23:11:15.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.859+0000 7f970e600640 1 -- 172.21.15.28:0/1197490366 learned_addr learned my addr 172.21.15.28:0/1197490366 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:15.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.859+0000 7f970dc00640 1 --2- 172.21.15.28:0/1197490366 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f96f00039b0 0x7f970814f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:15.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.859+0000 7f970d200640 1 --2- 172.21.15.28:0/1197490366 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f97080c1930 0x7f97080aea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:15.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.859+0000 7f970e600640 1 -- 172.21.15.28:0/1197490366 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f96f00039b0 msgr2=0x7f970814f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:15.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.860+0000 7f970e600640 1 --2- 172.21.15.28:0/1197490366 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f96f00039b0 0x7f970814f270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:15.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.860+0000 7f970e600640 1 -- 172.21.15.28:0/1197490366 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f97080c1930 msgr2=0x7f97080aea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:15.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.860+0000 7f970e600640 1 --2- 172.21.15.28:0/1197490366 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f97080c1930 0x7f97080aea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:15.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.860+0000 7f970e600640 1 -- 172.21.15.28:0/1197490366 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97081ee860 con 0x7f97080c1200 2024-02-21T23:11:15.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.860+0000 7f970e600640 1 --2- 172.21.15.28:0/1197490366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97080c1200 0x7f97081eaf70 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7f97080c83e0 tx=0x7f970000b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:15.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.860+0000 7f9705600640 1 -- 172.21.15.28:0/1197490366 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9700004d60 con 0x7f97080c1200 2024-02-21T23:11:15.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.860+0000 7f9705600640 1 -- 172.21.15.28:0/1197490366 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f970000bb20 con 0x7f97080c1200 2024-02-21T23:11:15.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.860+0000 7f9705600640 1 -- 172.21.15.28:0/1197490366 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9700007110 con 0x7f97080c1200 2024-02-21T23:11:15.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.860+0000 7f970f000640 1 -- 172.21.15.28:0/1197490366 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97081eede0 con 0x7f97080c1200 2024-02-21T23:11:15.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.860+0000 7f970f000640 1 -- 172.21.15.28:0/1197490366 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97081eeb00 con 0x7f97080c1200 2024-02-21T23:11:15.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.861+0000 7f970f000640 1 -- 172.21.15.28:0/1197490366 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f970804f290 con 0x7f97080c1200 2024-02-21T23:11:15.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.862+0000 7f9705600640 1 -- 172.21.15.28:0/1197490366 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f970001b020 con 0x7f97080c1200 2024-02-21T23:11:15.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.863+0000 7f9705600640 1 --2- 172.21.15.28:0/1197490366 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f96e806b500 0x7f96e806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:15.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.863+0000 7f9705600640 1 -- 172.21.15.28:0/1197490366 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f970008a4e0 con 0x7f97080c1200 2024-02-21T23:11:15.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.864+0000 7f970dc00640 1 --2- 172.21.15.28:0/1197490366 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f96e806b500 0x7f96e806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:15.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.865+0000 7f970dc00640 1 --2- 172.21.15.28:0/1197490366 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f96e806b500 0x7f96e806d9c0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f96fc035000 tx=0x7f96fc039040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:15.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:15.868+0000 7f9705600640 1 -- 172.21.15.28:0/1197490366 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f97000591c0 con 0x7f97080c1200 2024-02-21T23:11:16.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.105+0000 7f970f000640 1 -- 172.21.15.28:0/1197490366 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f970806d9d0 con 0x7f97080c1200 2024-02-21T23:11:16.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.106+0000 7f9705600640 1 -- 172.21.15.28:0/1197490366 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 191 v191) v1 ==== 95+0+4947 (secure 0 0 0) 0x7f9700058ac0 con 0x7f97080c1200 2024-02-21T23:11:16.096 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:11:16.096 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":191,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6686,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/155806034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":155806034},{"type":"v1","addr":"172.21.15.81:6836","nonce":155806034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":180},{"gid":7480,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3897243745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":3897243745},{"type":"v1","addr":"172.21.15.81:6837","nonce":3897243745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":180},{"gid":7486,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2827135632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2827135632},{"type":"v1","addr":"172.21.15.28:6835","nonce":2827135632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":180}],"filesystems":[{"mdsmap":{"epoch":191,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:10:53.351225+0000","modified":"2024-02-21T23:11:15.634013+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9699},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9699":{"gid":9699,"name":"c","rank":0,"incarnation":191,"state":"up:starting","state_seq":1,"addr":"172.21.15.28:6833/2364631414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2364631414},{"type":"v1","addr":"172.21.15.28:6833","nonce":2364631414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2024-02-21T23:11:16.096 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 191 2024-02-21T23:11:16.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.109+0000 7f970f000640 1 -- 172.21.15.28:0/1197490366 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f96e806b500 msgr2=0x7f96e806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:16.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.109+0000 7f970f000640 1 --2- 172.21.15.28:0/1197490366 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f96e806b500 0x7f96e806d9c0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f96fc035000 tx=0x7f96fc039040 comp rx=0 tx=0).stop 2024-02-21T23:11:16.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.109+0000 7f970f000640 1 -- 172.21.15.28:0/1197490366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97080c1200 msgr2=0x7f97081eaf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:16.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.109+0000 7f970f000640 1 --2- 172.21.15.28:0/1197490366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97080c1200 0x7f97081eaf70 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7f97080c83e0 tx=0x7f970000b4a0 comp rx=0 tx=0).stop 2024-02-21T23:11:16.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.109+0000 7f970f000640 1 -- 172.21.15.28:0/1197490366 shutdown_connections 2024-02-21T23:11:16.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.109+0000 7f970f000640 1 --2- 172.21.15.28:0/1197490366 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f96e806b500 0x7f96e806d9c0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:16.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.109+0000 7f970f000640 1 --2- 172.21.15.28:0/1197490366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97080c1200 0x7f97081eaf70 unknown :-1 s=CLOSED pgs=1971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:16.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.109+0000 7f970f000640 1 --2- 172.21.15.28:0/1197490366 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f97080c1930 0x7f97080aea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:16.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.109+0000 7f970f000640 1 --2- 172.21.15.28:0/1197490366 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f96f00039b0 0x7f970814f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:16.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.109+0000 7f970f000640 1 -- 172.21.15.28:0/1197490366 >> 172.21.15.28:0/1197490366 conn(0x7f97080ac370 msgr2=0x7f97080ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:16.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.110+0000 7f970f000640 1 -- 172.21.15.28:0/1197490366 shutdown_connections 2024-02-21T23:11:16.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.110+0000 7f970f000640 1 -- 172.21.15.28:0/1197490366 wait complete. 2024-02-21T23:11:16.118 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:11:16.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.350+0000 7f1514a00640 1 Processor -- start 2024-02-21T23:11:16.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.350+0000 7f1514a00640 1 -- start start 2024-02-21T23:11:16.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.351+0000 7f1514a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f151014f0b0 0x7f1510155180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:16.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.351+0000 7f1514a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f15101521c0 0x7f1510152610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:16.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.352+0000 7f1514a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1510155760 con 0x7f15100a6160 2024-02-21T23:11:16.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.352+0000 7f1514a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1510152b50 con 0x7f151014f0b0 2024-02-21T23:11:16.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.352+0000 7f1514a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1510151d30 con 0x7f15101521c0 2024-02-21T23:11:16.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.352+0000 7f150e800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f151014f0b0 0x7f1510155180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:16.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.352+0000 7f150e800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f151014f0b0 0x7f1510155180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58826/0 (socket says 172.21.15.28:58826) 2024-02-21T23:11:16.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.352+0000 7f150e800640 1 -- 172.21.15.28:0/3421839076 learned_addr learned my addr 172.21.15.28:0/3421839076 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:16.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.352+0000 7f150e800640 1 -- 172.21.15.28:0/3421839076 >> v2:172.21.15.81:3300/0 conn(0x7f15101521c0 msgr2=0x7f1510152610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:16.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.352+0000 7f150f200640 1 --2- 172.21.15.28:0/3421839076 >> v2:172.21.15.81:3300/0 conn(0x7f15101521c0 0x7f1510152610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:16.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.352+0000 7f150e800640 1 --2- 172.21.15.28:0/3421839076 >> v2:172.21.15.81:3300/0 conn(0x7f15101521c0 0x7f1510152610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:16.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.352+0000 7f150e800640 1 -- 172.21.15.28:0/3421839076 >> v1:172.21.15.81:6789/0 conn(0x7f15100a6160 legacy=0x7f15100c2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:16.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.352+0000 7f150e800640 1 -- 172.21.15.28:0/3421839076 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1510155a40 con 0x7f151014f0b0 2024-02-21T23:11:16.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.353+0000 7f150e800640 1 --2- 172.21.15.28:0/3421839076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f151014f0b0 0x7f1510155180 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7f1510151eb0 tx=0x7f150002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b96a496cfc4ecbe3 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:16.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.354+0000 7f150d400640 1 -- 172.21.15.28:0/3421839076 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1500035020 con 0x7f151014f0b0 2024-02-21T23:11:16.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.354+0000 7f150d400640 1 -- 172.21.15.28:0/3421839076 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1500031870 con 0x7f151014f0b0 2024-02-21T23:11:16.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.354+0000 7f150d400640 1 -- 172.21.15.28:0/3421839076 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1500031c40 con 0x7f151014f0b0 2024-02-21T23:11:16.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.354+0000 7f1514a00640 1 -- 172.21.15.28:0/3421839076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f151014f0b0 msgr2=0x7f1510155180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:16.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.354+0000 7f1514a00640 1 --2- 172.21.15.28:0/3421839076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f151014f0b0 0x7f1510155180 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7f1510151eb0 tx=0x7f150002b910 comp rx=0 tx=0).stop 2024-02-21T23:11:16.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.354+0000 7f1514a00640 1 -- 172.21.15.28:0/3421839076 shutdown_connections 2024-02-21T23:11:16.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.354+0000 7f1514a00640 1 --2- 172.21.15.28:0/3421839076 >> v2:172.21.15.81:3300/0 conn(0x7f15101521c0 0x7f1510152610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:16.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.354+0000 7f1514a00640 1 --2- 172.21.15.28:0/3421839076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f151014f0b0 0x7f1510155180 unknown :-1 s=CLOSED pgs=1972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:16.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.355+0000 7f1514a00640 1 -- 172.21.15.28:0/3421839076 >> 172.21.15.28:0/3421839076 conn(0x7f151006b630 msgr2=0x7f15100c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:16.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.355+0000 7f1514a00640 1 -- 172.21.15.28:0/3421839076 shutdown_connections 2024-02-21T23:11:16.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.355+0000 7f1514a00640 1 -- 172.21.15.28:0/3421839076 wait complete. 2024-02-21T23:11:16.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.356+0000 7f1514a00640 1 Processor -- start 2024-02-21T23:11:16.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.356+0000 7f1514a00640 1 -- start start 2024-02-21T23:11:16.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.357+0000 7f1514a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f15100a6160 0x7f15101edee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:16.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.357+0000 7f1514a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f151014f0b0 0x7f15100c40f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:16.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.357+0000 7f1514a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f15101521c0 0x7f15100c4630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:16.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.357+0000 7f1514a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15100a68d0 con 0x7f151014f0b0 2024-02-21T23:11:16.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.357+0000 7f1514a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1510156000 con 0x7f15100a6160 2024-02-21T23:11:16.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.357+0000 7f1514a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15100c2720 con 0x7f15101521c0 2024-02-21T23:11:16.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.357+0000 7f150de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f151014f0b0 0x7f15100c40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:16.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.358+0000 7f150de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f151014f0b0 0x7f15100c40f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58828/0 (socket says 172.21.15.28:58828) 2024-02-21T23:11:16.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.358+0000 7f150de00640 1 -- 172.21.15.28:0/3169024902 learned_addr learned my addr 172.21.15.28:0/3169024902 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:16.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.358+0000 7f150de00640 1 -- 172.21.15.28:0/3169024902 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f15101521c0 msgr2=0x7f15100c4630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:16.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.358+0000 7f150de00640 1 --2- 172.21.15.28:0/3169024902 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f15101521c0 0x7f15100c4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:16.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.358+0000 7f150de00640 1 -- 172.21.15.28:0/3169024902 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f15100a6160 msgr2=0x7f15101edee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:16.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.358+0000 7f150de00640 1 --2- 172.21.15.28:0/3169024902 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f15100a6160 0x7f15101edee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:16.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.358+0000 7f150de00640 1 -- 172.21.15.28:0/3169024902 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15101504b0 con 0x7f151014f0b0 2024-02-21T23:11:16.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.358+0000 7f150de00640 1 --2- 172.21.15.28:0/3169024902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f151014f0b0 0x7f15100c40f0 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7f14fc00c460 tx=0x7f14fc00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:16.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.359+0000 7f1506a00640 1 -- 172.21.15.28:0/3169024902 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14fc016020 con 0x7f151014f0b0 2024-02-21T23:11:16.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.359+0000 7f1506a00640 1 -- 172.21.15.28:0/3169024902 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14fc012870 con 0x7f151014f0b0 2024-02-21T23:11:16.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.359+0000 7f1506a00640 1 -- 172.21.15.28:0/3169024902 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14fc012c60 con 0x7f151014f0b0 2024-02-21T23:11:16.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.359+0000 7f1514a00640 1 -- 172.21.15.28:0/3169024902 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15101ee820 con 0x7f151014f0b0 2024-02-21T23:11:16.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.359+0000 7f1514a00640 1 -- 172.21.15.28:0/3169024902 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15101eeb00 con 0x7f151014f0b0 2024-02-21T23:11:16.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.360+0000 7f1514a00640 1 -- 172.21.15.28:0/3169024902 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f151004f290 con 0x7f151014f0b0 2024-02-21T23:11:16.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.361+0000 7f1506a00640 1 -- 172.21.15.28:0/3169024902 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f14fc019020 con 0x7f151014f0b0 2024-02-21T23:11:16.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.362+0000 7f1506a00640 1 --2- 172.21.15.28:0/3169024902 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f14f006b4d0 0x7f14f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:16.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.362+0000 7f1506a00640 1 -- 172.21.15.28:0/3169024902 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f14fc089e20 con 0x7f151014f0b0 2024-02-21T23:11:16.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.363+0000 7f150e800640 1 --2- 172.21.15.28:0/3169024902 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f14f006b4d0 0x7f14f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:16.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.363+0000 7f150e800640 1 --2- 172.21.15.28:0/3169024902 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f14f006b4d0 0x7f14f006d990 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f1500036000 tx=0x7f1500030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:16.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.366+0000 7f1506a00640 1 -- 172.21.15.28:0/3169024902 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f14fc020070 con 0x7f151014f0b0 2024-02-21T23:11:16.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.606+0000 7f1514a00640 1 -- 172.21.15.28:0/3169024902 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f15100c0be0 con 0x7f151014f0b0 2024-02-21T23:11:16.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.607+0000 7f1506a00640 1 -- 172.21.15.28:0/3169024902 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 191 v191) v1 ==== 95+0+4947 (secure 0 0 0) 0x7f14fc058b00 con 0x7f151014f0b0 2024-02-21T23:11:16.596 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:11:16.597 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":191,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6686,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/155806034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":155806034},{"type":"v1","addr":"172.21.15.81:6836","nonce":155806034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":180},{"gid":7480,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3897243745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":3897243745},{"type":"v1","addr":"172.21.15.81:6837","nonce":3897243745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":180},{"gid":7486,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2827135632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2827135632},{"type":"v1","addr":"172.21.15.28:6835","nonce":2827135632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":180}],"filesystems":[{"mdsmap":{"epoch":191,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:10:53.351225+0000","modified":"2024-02-21T23:11:15.634013+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9699},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9699":{"gid":9699,"name":"c","rank":0,"incarnation":191,"state":"up:starting","state_seq":1,"addr":"172.21.15.28:6833/2364631414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2364631414},{"type":"v1","addr":"172.21.15.28:6833","nonce":2364631414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2024-02-21T23:11:16.597 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 191 2024-02-21T23:11:16.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.610+0000 7f1514a00640 1 -- 172.21.15.28:0/3169024902 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f14f006b4d0 msgr2=0x7f14f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:16.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.610+0000 7f1514a00640 1 --2- 172.21.15.28:0/3169024902 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f14f006b4d0 0x7f14f006d990 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f1500036000 tx=0x7f1500030000 comp rx=0 tx=0).stop 2024-02-21T23:11:16.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.610+0000 7f1514a00640 1 -- 172.21.15.28:0/3169024902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f151014f0b0 msgr2=0x7f15100c40f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:16.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.610+0000 7f1514a00640 1 --2- 172.21.15.28:0/3169024902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f151014f0b0 0x7f15100c40f0 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7f14fc00c460 tx=0x7f14fc00c920 comp rx=0 tx=0).stop 2024-02-21T23:11:16.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.611+0000 7f1514a00640 1 -- 172.21.15.28:0/3169024902 shutdown_connections 2024-02-21T23:11:16.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.611+0000 7f1514a00640 1 --2- 172.21.15.28:0/3169024902 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f14f006b4d0 0x7f14f006d990 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:16.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.611+0000 7f1514a00640 1 --2- 172.21.15.28:0/3169024902 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f15101521c0 0x7f15100c4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:16.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.611+0000 7f1514a00640 1 --2- 172.21.15.28:0/3169024902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f151014f0b0 0x7f15100c40f0 unknown :-1 s=CLOSED pgs=1973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:16.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.611+0000 7f1514a00640 1 --2- 172.21.15.28:0/3169024902 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f15100a6160 0x7f15101edee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:16.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.611+0000 7f1514a00640 1 -- 172.21.15.28:0/3169024902 >> 172.21.15.28:0/3169024902 conn(0x7f151006b630 msgr2=0x7f15100bfa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:16.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.611+0000 7f1514a00640 1 -- 172.21.15.28:0/3169024902 shutdown_connections 2024-02-21T23:11:16.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:16.611+0000 7f1514a00640 1 -- 172.21.15.28:0/3169024902 wait complete. 2024-02-21T23:11:17.021 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 191, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:10:53.351225+0000', 'modified': '2024-02-21T23:11:15.634013+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9699}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9699': {'gid': 9699, 'name': 'c', 'rank': 0, 'incarnation': 191, 'state': 'up:starting', 'state_seq': 1, 'addr': '172.21.15.28:6833/2364631414', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 2364631414}, {'type': 'v1', 'addr': '172.21.15.28:6833', 'nonce': 2364631414}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:11:17.021 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9699:up:starting 2024-02-21T23:11:18.022 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:11:18.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.262+0000 7f732fe00640 1 Processor -- start 2024-02-21T23:11:18.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.262+0000 7f732fe00640 1 -- start start 2024-02-21T23:11:18.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.264+0000 7f732fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7330155960 0x7f73301608a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:18.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.264+0000 7f732fe00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73300c74c0 con 0x7f73300aff90 2024-02-21T23:11:18.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.264+0000 7f732fe00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73300c77a0 con 0x7f733006dbf0 2024-02-21T23:11:18.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.264+0000 7f732fe00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73300c7270 con 0x7f7330155960 2024-02-21T23:11:18.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.264+0000 7f732ea00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f73300aff90 0x7f73300c51e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37846/0 (socket says 172.21.15.28:37846) 2024-02-21T23:11:18.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.264+0000 7f732ea00640 1 -- 172.21.15.28:0/2234417051 learned_addr learned my addr 172.21.15.28:0/2234417051 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:11:18.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.264+0000 7f732f400640 1 --2- 172.21.15.28:0/2234417051 >> v2:172.21.15.28:3300/0 conn(0x7f7330155960 0x7f73301608a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:18.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.265+0000 7f732e000640 1 -- 172.21.15.28:0/2234417051 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1182437510 0 0) 0x7f73300c74c0 con 0x7f73300aff90 2024-02-21T23:11:18.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.265+0000 7f732e000640 1 -- 172.21.15.28:0/2234417051 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7314003520 con 0x7f73300aff90 2024-02-21T23:11:18.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.265+0000 7f732e000640 1 -- 172.21.15.28:0/2234417051 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3745787154 0 0) 0x7f7314003520 con 0x7f73300aff90 2024-02-21T23:11:18.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.265+0000 7f732f400640 1 -- 172.21.15.28:0/2234417051 >> v1:172.21.15.81:6789/0 conn(0x7f733006dbf0 legacy=0x7f7330158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:18.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.265+0000 7f732f400640 1 -- 172.21.15.28:0/2234417051 >> v1:172.21.15.28:6789/0 conn(0x7f73300aff90 legacy=0x7f73300c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:18.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.265+0000 7f732f400640 1 -- 172.21.15.28:0/2234417051 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73300c7a80 con 0x7f7330155960 2024-02-21T23:11:18.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.267+0000 7f732f400640 1 --2- 172.21.15.28:0/2234417051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7330155960 0x7f73301608a0 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7f7328009d90 tx=0x7f732802be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6589410095f1bc0a server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:18.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.267+0000 7f732e000640 1 -- 172.21.15.28:0/2234417051 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7328039070 con 0x7f7330155960 2024-02-21T23:11:18.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.267+0000 7f732e000640 1 -- 172.21.15.28:0/2234417051 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f732802c940 con 0x7f7330155960 2024-02-21T23:11:18.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.267+0000 7f732e000640 1 -- 172.21.15.28:0/2234417051 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7320001670 con 0x7f7330155960 2024-02-21T23:11:18.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.268+0000 7f732fe00640 1 -- 172.21.15.28:0/2234417051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7330155960 msgr2=0x7f73301608a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:18.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.268+0000 7f732fe00640 1 --2- 172.21.15.28:0/2234417051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7330155960 0x7f73301608a0 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7f7328009d90 tx=0x7f732802be10 comp rx=0 tx=0).stop 2024-02-21T23:11:18.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.268+0000 7f732fe00640 1 -- 172.21.15.28:0/2234417051 shutdown_connections 2024-02-21T23:11:18.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.268+0000 7f732fe00640 1 --2- 172.21.15.28:0/2234417051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7330155960 0x7f73301608a0 unknown :-1 s=CLOSED pgs=1975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:18.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.268+0000 7f732fe00640 1 -- 172.21.15.28:0/2234417051 >> 172.21.15.28:0/2234417051 conn(0x7f73300ac370 msgr2=0x7f73300ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:18.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.268+0000 7f732fe00640 1 -- 172.21.15.28:0/2234417051 shutdown_connections 2024-02-21T23:11:18.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.268+0000 7f732fe00640 1 -- 172.21.15.28:0/2234417051 wait complete. 2024-02-21T23:11:18.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.270+0000 7f732fe00640 1 Processor -- start 2024-02-21T23:11:18.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.270+0000 7f732fe00640 1 -- start start 2024-02-21T23:11:18.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.271+0000 7f732fe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f733006dbf0 0x7f73300c0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:18.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.271+0000 7f732fe00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f73300aff90 0x7f73301512b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:18.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.271+0000 7f732fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7330155960 0x7f73300c40d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:18.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.271+0000 7f732fe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73300c58f0 con 0x7f7330155960 2024-02-21T23:11:18.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.271+0000 7f732fe00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7330158630 con 0x7f733006dbf0 2024-02-21T23:11:18.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.271+0000 7f732fe00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73300c5a70 con 0x7f73300aff90 2024-02-21T23:11:18.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.271+0000 7f732f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7330155960 0x7f73300c40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:18.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.271+0000 7f732f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7330155960 0x7f73300c40d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58854/0 (socket says 172.21.15.28:58854) 2024-02-21T23:11:18.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.271+0000 7f732f400640 1 -- 172.21.15.28:0/3065044248 learned_addr learned my addr 172.21.15.28:0/3065044248 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:18.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.272+0000 7f732ea00640 1 --2- 172.21.15.28:0/3065044248 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f733006dbf0 0x7f73300c0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:18.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.272+0000 7f7326000640 1 --2- 172.21.15.28:0/3065044248 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f73300aff90 0x7f73301512b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:18.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.272+0000 7f732f400640 1 -- 172.21.15.28:0/3065044248 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f73300aff90 msgr2=0x7f73301512b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:18.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.272+0000 7f732f400640 1 --2- 172.21.15.28:0/3065044248 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f73300aff90 0x7f73301512b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:18.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.272+0000 7f732f400640 1 -- 172.21.15.28:0/3065044248 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f733006dbf0 msgr2=0x7f73300c0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:18.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.272+0000 7f732f400640 1 --2- 172.21.15.28:0/3065044248 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f733006dbf0 0x7f73300c0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:18.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.272+0000 7f732f400640 1 -- 172.21.15.28:0/3065044248 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73300be780 con 0x7f7330155960 2024-02-21T23:11:18.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.272+0000 7f732f400640 1 --2- 172.21.15.28:0/3065044248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7330155960 0x7f73300c40d0 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7f7328037040 tx=0x7f7328004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:18.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.272+0000 7f7327400640 1 -- 172.21.15.28:0/3065044248 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7328039040 con 0x7f7330155960 2024-02-21T23:11:18.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.272+0000 7f7327400640 1 -- 172.21.15.28:0/3065044248 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7328034d10 con 0x7f7330155960 2024-02-21T23:11:18.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.272+0000 7f7327400640 1 -- 172.21.15.28:0/3065044248 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7328009450 con 0x7f7330155960 2024-02-21T23:11:18.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.272+0000 7f732fe00640 1 -- 172.21.15.28:0/3065044248 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73301eb3b0 con 0x7f7330155960 2024-02-21T23:11:18.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.273+0000 7f732fe00640 1 -- 172.21.15.28:0/3065044248 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7330154980 con 0x7f7330155960 2024-02-21T23:11:18.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.274+0000 7f732fe00640 1 -- 172.21.15.28:0/3065044248 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f733004f290 con 0x7f7330155960 2024-02-21T23:11:18.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.275+0000 7f7327400640 1 -- 172.21.15.28:0/3065044248 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7328003bd0 con 0x7f7330155960 2024-02-21T23:11:18.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.276+0000 7f7327400640 1 --2- 172.21.15.28:0/3065044248 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f730c06b4d0 0x7f730c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:18.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.276+0000 7f7327400640 1 -- 172.21.15.28:0/3065044248 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f73280ace80 con 0x7f7330155960 2024-02-21T23:11:18.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.277+0000 7f732ea00640 1 --2- 172.21.15.28:0/3065044248 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f730c06b4d0 0x7f730c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:18.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.278+0000 7f732ea00640 1 --2- 172.21.15.28:0/3065044248 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f730c06b4d0 0x7f730c06d990 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f7320005880 tx=0x7f7320009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:18.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.280+0000 7f7327400640 1 -- 172.21.15.28:0/3065044248 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f732804e070 con 0x7f7330155960 2024-02-21T23:11:18.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.609+0000 7f732fe00640 1 -- 172.21.15.28:0/3065044248 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f733006d9d0 con 0x7f7330155960 2024-02-21T23:11:18.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.611+0000 7f7327400640 1 -- 172.21.15.28:0/3065044248 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 192 v192) v1 ==== 95+0+4945 (secure 0 0 0) 0x7f73300be780 con 0x7f7330155960 2024-02-21T23:11:18.600 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:11:18.600 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":192,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6686,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/155806034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":155806034},{"type":"v1","addr":"172.21.15.81:6836","nonce":155806034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":180},{"gid":7480,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3897243745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":3897243745},{"type":"v1","addr":"172.21.15.81:6837","nonce":3897243745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":180},{"gid":7486,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2827135632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2827135632},{"type":"v1","addr":"172.21.15.28:6835","nonce":2827135632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":180}],"filesystems":[{"mdsmap":{"epoch":192,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:10:53.351225+0000","modified":"2024-02-21T23:11:15.654038+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9699},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9699":{"gid":9699,"name":"c","rank":0,"incarnation":191,"state":"up:active","state_seq":2,"addr":"172.21.15.28:6833/2364631414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2364631414},{"type":"v1","addr":"172.21.15.28:6833","nonce":2364631414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2024-02-21T23:11:18.600 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 192 2024-02-21T23:11:18.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.613+0000 7f732fe00640 1 -- 172.21.15.28:0/3065044248 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f730c06b4d0 msgr2=0x7f730c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:18.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.613+0000 7f732fe00640 1 --2- 172.21.15.28:0/3065044248 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f730c06b4d0 0x7f730c06d990 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f7320005880 tx=0x7f7320009040 comp rx=0 tx=0).stop 2024-02-21T23:11:18.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.613+0000 7f732fe00640 1 -- 172.21.15.28:0/3065044248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7330155960 msgr2=0x7f73300c40d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:18.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.613+0000 7f732fe00640 1 --2- 172.21.15.28:0/3065044248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7330155960 0x7f73300c40d0 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7f7328037040 tx=0x7f7328004a40 comp rx=0 tx=0).stop 2024-02-21T23:11:18.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.613+0000 7f732fe00640 1 -- 172.21.15.28:0/3065044248 shutdown_connections 2024-02-21T23:11:18.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.613+0000 7f732fe00640 1 --2- 172.21.15.28:0/3065044248 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f730c06b4d0 0x7f730c06d990 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:18.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.613+0000 7f732fe00640 1 --2- 172.21.15.28:0/3065044248 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f73300aff90 0x7f73301512b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:18.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.613+0000 7f732fe00640 1 --2- 172.21.15.28:0/3065044248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7330155960 0x7f73300c40d0 unknown :-1 s=CLOSED pgs=1976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:18.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.614+0000 7f732fe00640 1 --2- 172.21.15.28:0/3065044248 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f733006dbf0 0x7f73300c0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:18.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.614+0000 7f732fe00640 1 -- 172.21.15.28:0/3065044248 >> 172.21.15.28:0/3065044248 conn(0x7f73300ac370 msgr2=0x7f73300ac750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:18.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.614+0000 7f732fe00640 1 -- 172.21.15.28:0/3065044248 shutdown_connections 2024-02-21T23:11:18.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:18.614+0000 7f732fe00640 1 -- 172.21.15.28:0/3065044248 wait complete. 2024-02-21T23:11:19.025 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 192, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:10:53.351225+0000', 'modified': '2024-02-21T23:11:15.654038+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9699}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9699': {'gid': 9699, 'name': 'c', 'rank': 0, 'incarnation': 191, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.28:6833/2364631414', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 2364631414}, {'type': 'v1', 'addr': '172.21.15.28:6833', 'nonce': 2364631414}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:11:19.025 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-21T23:11:19.026 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-02-21T23:11:19.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.265+0000 7fe5c6c00640 1 Processor -- start 2024-02-21T23:11:19.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.266+0000 7fe5c6c00640 1 -- start start 2024-02-21T23:11:19.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.267+0000 7fe5c6c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe5c00bf880 0x7fe5c00c6a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:19.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.267+0000 7fe5c6c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe5c00c5260 0x7fe5c00c56f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:19.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.268+0000 7fe5c6c00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5c00c6640 con 0x7fe5c0152740 2024-02-21T23:11:19.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.268+0000 7fe5c6c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5c0155fc0 con 0x7fe5c00c5260 2024-02-21T23:11:19.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.268+0000 7fe5c6c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c0155200 con 0x7fe5c00bf880 2024-02-21T23:11:19.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.268+0000 7fe5c5800640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fe5c0152740 0x7fe5c00c72f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.28:35568/0 (socket says 172.21.15.28:35568) 2024-02-21T23:11:19.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.268+0000 7fe5c5800640 1 -- 172.21.15.28:0/3415132770 learned_addr learned my addr 172.21.15.28:0/3415132770 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:11:19.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.268+0000 7fe5c6200640 1 --2- 172.21.15.28:0/3415132770 >> v2:172.21.15.28:3300/0 conn(0x7fe5c00c5260 0x7fe5c00c56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:19.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.268+0000 7fe5c4e00640 1 --2- 172.21.15.28:0/3415132770 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe5c00bf880 0x7fe5c00c6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:19.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.269+0000 7fe5c6200640 1 -- 172.21.15.28:0/3415132770 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe5c00bf880 msgr2=0x7fe5c00c6a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:19.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.269+0000 7fe5c6200640 1 --2- 172.21.15.28:0/3415132770 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe5c00bf880 0x7fe5c00c6a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:19.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.269+0000 7fe5c6200640 1 -- 172.21.15.28:0/3415132770 >> v1:172.21.15.81:6789/0 conn(0x7fe5c0152740 legacy=0x7fe5c00c72f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:19.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.269+0000 7fe5c6200640 1 -- 172.21.15.28:0/3415132770 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5c01554e0 con 0x7fe5c00c5260 2024-02-21T23:11:19.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.270+0000 7fe5c6200640 1 --2- 172.21.15.28:0/3415132770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5c00c5260 0x7fe5c00c56f0 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7fe5b80097a0 tx=0x7fe5b802b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f55b6992b381923 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:19.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.270+0000 7fe5bfe00640 1 -- 172.21.15.28:0/3415132770 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5b8031780 con 0x7fe5c00c5260 2024-02-21T23:11:19.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.270+0000 7fe5bfe00640 1 -- 172.21.15.28:0/3415132770 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5b8037070 con 0x7fe5c00c5260 2024-02-21T23:11:19.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.270+0000 7fe5bfe00640 1 -- 172.21.15.28:0/3415132770 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5b8031b40 con 0x7fe5c00c5260 2024-02-21T23:11:19.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.271+0000 7fe5c6c00640 1 -- 172.21.15.28:0/3415132770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5c00c5260 msgr2=0x7fe5c00c56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:19.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.271+0000 7fe5c6c00640 1 --2- 172.21.15.28:0/3415132770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5c00c5260 0x7fe5c00c56f0 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7fe5b80097a0 tx=0x7fe5b802b820 comp rx=0 tx=0).stop 2024-02-21T23:11:19.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.271+0000 7fe5c6c00640 1 -- 172.21.15.28:0/3415132770 shutdown_connections 2024-02-21T23:11:19.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.271+0000 7fe5c6c00640 1 --2- 172.21.15.28:0/3415132770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5c00c5260 0x7fe5c00c56f0 unknown :-1 s=CLOSED pgs=1977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:19.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.271+0000 7fe5c6c00640 1 --2- 172.21.15.28:0/3415132770 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe5c00bf880 0x7fe5c00c6a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:19.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.271+0000 7fe5c6c00640 1 -- 172.21.15.28:0/3415132770 >> 172.21.15.28:0/3415132770 conn(0x7fe5c0067080 msgr2=0x7fe5c00725a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:19.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.271+0000 7fe5c6c00640 1 -- 172.21.15.28:0/3415132770 shutdown_connections 2024-02-21T23:11:19.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.272+0000 7fe5c6c00640 1 -- 172.21.15.28:0/3415132770 wait complete. 2024-02-21T23:11:19.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.274+0000 7fe5c6c00640 1 Processor -- start 2024-02-21T23:11:19.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.274+0000 7fe5c6c00640 1 -- start start 2024-02-21T23:11:19.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.276+0000 7fe5c6c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe5c00bf880 0x7fe5c01ee1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:19.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.276+0000 7fe5c6c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe5c00c5260 0x7fe5c0149990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:19.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.276+0000 7fe5c6c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5c0152740 0x7fe5c0149ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:19.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.276+0000 7fe5c6c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c00c7a00 con 0x7fe5c0152740 2024-02-21T23:11:19.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.276+0000 7fe5c6c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c0155aa0 con 0x7fe5c00c5260 2024-02-21T23:11:19.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.276+0000 7fe5c6c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c00c7b80 con 0x7fe5c00bf880 2024-02-21T23:11:19.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.276+0000 7fe5c6200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5c0152740 0x7fe5c0149ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:19.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.276+0000 7fe5c6200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5c0152740 0x7fe5c0149ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58862/0 (socket says 172.21.15.28:58862) 2024-02-21T23:11:19.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.276+0000 7fe5c6200640 1 -- 172.21.15.28:0/2130829962 learned_addr learned my addr 172.21.15.28:0/2130829962 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:19.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.277+0000 7fe5c5800640 1 --2- 172.21.15.28:0/2130829962 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe5c00bf880 0x7fe5c01ee1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:19.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.277+0000 7fe5c4e00640 1 --2- 172.21.15.28:0/2130829962 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe5c00c5260 0x7fe5c0149990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:19.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.277+0000 7fe5c6200640 1 -- 172.21.15.28:0/2130829962 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe5c00bf880 msgr2=0x7fe5c01ee1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:19.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.277+0000 7fe5c6200640 1 --2- 172.21.15.28:0/2130829962 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe5c00bf880 0x7fe5c01ee1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:19.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.277+0000 7fe5c6200640 1 -- 172.21.15.28:0/2130829962 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe5c00c5260 msgr2=0x7fe5c0149990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:19.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.277+0000 7fe5c6200640 1 --2- 172.21.15.28:0/2130829962 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe5c00c5260 0x7fe5c0149990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:19.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.277+0000 7fe5c6200640 1 -- 172.21.15.28:0/2130829962 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5c01eecd0 con 0x7fe5c0152740 2024-02-21T23:11:19.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.277+0000 7fe5c6200640 1 --2- 172.21.15.28:0/2130829962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5c0152740 0x7fe5c0149ed0 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7fe5b8034040 tx=0x7fe5b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:19.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.277+0000 7fe5bd600640 1 -- 172.21.15.28:0/2130829962 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5b8032610 con 0x7fe5c0152740 2024-02-21T23:11:19.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.278+0000 7fe5bd600640 1 -- 172.21.15.28:0/2130829962 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5b80327d0 con 0x7fe5c0152740 2024-02-21T23:11:19.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.278+0000 7fe5c6c00640 1 -- 172.21.15.28:0/2130829962 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5c00c8e50 con 0x7fe5c0152740 2024-02-21T23:11:19.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.278+0000 7fe5bd600640 1 -- 172.21.15.28:0/2130829962 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5b8032bc0 con 0x7fe5c0152740 2024-02-21T23:11:19.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.278+0000 7fe5c6c00640 1 -- 172.21.15.28:0/2130829962 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5c00c9410 con 0x7fe5c0152740 2024-02-21T23:11:19.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.280+0000 7fe5bd600640 1 -- 172.21.15.28:0/2130829962 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe5b804c060 con 0x7fe5c0152740 2024-02-21T23:11:19.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.281+0000 7fe5bd600640 1 --2- 172.21.15.28:0/2130829962 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe5a006b4d0 0x7fe5a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:19.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.281+0000 7fe5bd600640 1 -- 172.21.15.28:0/2130829962 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe5b80abcb0 con 0x7fe5c0152740 2024-02-21T23:11:19.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.282+0000 7fe5c5800640 1 --2- 172.21.15.28:0/2130829962 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe5a006b4d0 0x7fe5a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:19.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.283+0000 7fe5c5800640 1 --2- 172.21.15.28:0/2130829962 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe5a006b4d0 0x7fe5a006d990 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fe5b0005c00 tx=0x7fe5b0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:19.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.283+0000 7fe5cb080740 1 Processor -- start 2024-02-21T23:11:19.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.284+0000 7fe5cb080740 1 -- start start 2024-02-21T23:11:19.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.285+0000 7fe5cb080740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626baa8b020 0x5626baa8b400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:19.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.285+0000 7fe5cb080740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5626baa8c330 0x5626baac7170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:19.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.285+0000 7fe5cb080740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5626baac76b0 0x5626baac5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:19.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.285+0000 7fe5cb080740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626ba9ef470 con 0x5626baa8b020 2024-02-21T23:11:19.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.285+0000 7fe5cb080740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626ba998af0 con 0x5626baac76b0 2024-02-21T23:11:19.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.285+0000 7fe5cb080740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x5626ba9eb510 con 0x5626baa8c330 2024-02-21T23:11:19.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.285+0000 7fe5c5800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626baa8b020 0x5626baa8b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:19.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.286+0000 7fe5c5800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626baa8b020 0x5626baa8b400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58878/0 (socket says 172.21.15.28:58878) 2024-02-21T23:11:19.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.286+0000 7fe5c5800640 1 -- 172.21.15.28:0/264388179 learned_addr learned my addr 172.21.15.28:0/264388179 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:19.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.286+0000 7fe5c4e00640 1 --2- 172.21.15.28:0/264388179 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5626baa8c330 0x5626baac7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:19.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.286+0000 7fe5c6200640 1 --2- 172.21.15.28:0/264388179 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5626baac76b0 0x5626baac5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:19.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.286+0000 7fe5c5800640 1 -- 172.21.15.28:0/264388179 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5626baa8c330 msgr2=0x5626baac7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:19.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.286+0000 7fe5c5800640 1 --2- 172.21.15.28:0/264388179 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5626baa8c330 0x5626baac7170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:19.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.286+0000 7fe5c5800640 1 -- 172.21.15.28:0/264388179 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5626baac76b0 msgr2=0x5626baac5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:19.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.286+0000 7fe5c5800640 1 --2- 172.21.15.28:0/264388179 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5626baac76b0 0x5626baac5720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:19.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.286+0000 7fe5c5800640 1 -- 172.21.15.28:0/264388179 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5626ba99fd90 con 0x5626baa8b020 2024-02-21T23:11:19.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.286+0000 7fe5c5800640 1 --2- 172.21.15.28:0/264388179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626baa8b020 0x5626baa8b400 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7fe5b000ee20 tx=0x7fe5b0011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c12b3039702c721b server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:19.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.287+0000 7fe5b6000640 1 -- 172.21.15.28:0/264388179 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5b0007860 con 0x5626baa8b020 2024-02-21T23:11:19.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.287+0000 7fe5b6000640 1 -- 172.21.15.28:0/264388179 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5b0007a00 con 0x5626baa8b020 2024-02-21T23:11:19.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.287+0000 7fe5b6000640 1 -- 172.21.15.28:0/264388179 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5b0021650 con 0x5626baa8b020 2024-02-21T23:11:19.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.288+0000 7fe5cb080740 1 -- 172.21.15.28:0/264388179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626baa8b020 msgr2=0x5626baa8b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:19.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.288+0000 7fe5cb080740 1 --2- 172.21.15.28:0/264388179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626baa8b020 0x5626baa8b400 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7fe5b000ee20 tx=0x7fe5b0011000 comp rx=0 tx=0).stop 2024-02-21T23:11:19.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.288+0000 7fe5cb080740 1 -- 172.21.15.28:0/264388179 shutdown_connections 2024-02-21T23:11:19.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.288+0000 7fe5cb080740 1 --2- 172.21.15.28:0/264388179 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5626baa8c330 0x5626baac7170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:19.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.288+0000 7fe5cb080740 1 --2- 172.21.15.28:0/264388179 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5626baac76b0 0x5626baac5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:19.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.288+0000 7fe5cb080740 1 --2- 172.21.15.28:0/264388179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626baa8b020 0x5626baa8b400 unknown :-1 s=CLOSED pgs=1979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:19.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.288+0000 7fe5cb080740 1 -- 172.21.15.28:0/264388179 >> 172.21.15.28:0/264388179 conn(0x5626baa87970 msgr2=0x5626baaac380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:19.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.288+0000 7fe5cb080740 1 -- 172.21.15.28:0/264388179 shutdown_connections 2024-02-21T23:11:19.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.288+0000 7fe5cb080740 1 -- 172.21.15.28:0/264388179 wait complete. 2024-02-21T23:11:19.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.289+0000 7fe5cb080740 1 -- start start 2024-02-21T23:11:19.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.289+0000 7fe5cb080740 1 Processor -- start 2024-02-21T23:11:19.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.290+0000 7fe5cb080740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626baa8c330 0x5626baacceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:19.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.290+0000 7fe5cb080740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5626baac76b0 0x5626baadaaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:19.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.290+0000 7fe5cb080740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5626baaab5a0 0x5626baaa7130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:19.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.290+0000 7fe5cb080740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626ba98e040 con 0x5626baa8c330 2024-02-21T23:11:19.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.290+0000 7fe5cb080740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626baaaca90 con 0x5626baaab5a0 2024-02-21T23:11:19.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.290+0000 7fe5cb080740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x5626baaae6a0 con 0x5626baac76b0 2024-02-21T23:11:19.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.290+0000 7fe5cb080740 10 client.0 _ll_register_callbacks cb 0x5626ba9ab920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:11:19.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.290+0000 7fe5c5800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626baa8c330 0x5626baacceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:19.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.290+0000 7fe5c5800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626baa8c330 0x5626baacceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58886/0 (socket says 172.21.15.28:58886) 2024-02-21T23:11:19.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.290+0000 7fe5c5800640 1 -- 172.21.15.28:0/1335305117 learned_addr learned my addr 172.21.15.28:0/1335305117 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:19.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.290+0000 7fe5c6200640 1 --2- 172.21.15.28:0/1335305117 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5626baaab5a0 0x5626baaa7130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:19.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.290+0000 7fe5c4e00640 1 --2- 172.21.15.28:0/1335305117 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5626baac76b0 0x5626baadaaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:19.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.291+0000 7fe5c5800640 1 -- 172.21.15.28:0/1335305117 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5626baac76b0 msgr2=0x5626baadaaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:19.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.291+0000 7fe5c5800640 1 --2- 172.21.15.28:0/1335305117 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5626baac76b0 0x5626baadaaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:19.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.291+0000 7fe5c5800640 1 -- 172.21.15.28:0/1335305117 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5626baaab5a0 msgr2=0x5626baaa7130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:19.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.291+0000 7fe5c5800640 1 --2- 172.21.15.28:0/1335305117 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5626baaab5a0 0x5626baaa7130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:19.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.291+0000 7fe5c5800640 1 -- 172.21.15.28:0/1335305117 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5626ba9adaa0 con 0x5626baa8c330 2024-02-21T23:11:19.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.291+0000 7fe5c5800640 1 --2- 172.21.15.28:0/1335305117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626baa8c330 0x5626baacceb0 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 crypto rx=0x7fe5b00074a0 tx=0x7fe5b0001250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4492472d2cb5e6f1 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:19.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.291+0000 7fe5aaa00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:11:19.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.291+0000 7fe5aaa00640 1 -- 172.21.15.28:0/1335305117 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5b0022bd0 con 0x5626baa8c330 2024-02-21T23:11:19.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.291+0000 7fe5aaa00640 1 -- 172.21.15.28:0/1335305117 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(174..174 src has 1..174) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe5b0022d70 con 0x5626baa8c330 2024-02-21T23:11:19.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.291+0000 7fe5cb080740 1 -- 172.21.15.28:0/1335305117 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe5b0007a00 con 0x5626baa8c330 2024-02-21T23:11:19.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.292+0000 7fe5aaa00640 1 -- 172.21.15.28:0/1335305117 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=192) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe5b0031050 con 0x5626baa8c330 2024-02-21T23:11:19.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.292+0000 7fe5cb080740 10 client.9750 fetch_fsmap learned FSMap version 192 2024-02-21T23:11:19.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.292+0000 7fe5cb080740 1 -- 172.21.15.28:0/1335305117 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=192,monmap=2+}) v3 -- 0x5626ba7042d0 con 0x5626baa8c330 2024-02-21T23:11:19.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.292+0000 7fe5aaa00640 1 -- 172.21.15.28:0/1335305117 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5b0007a00 con 0x5626baa8c330 2024-02-21T23:11:19.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.292+0000 7fe5aaa00640 1 -- 172.21.15.28:0/1335305117 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 192) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fe5b0034020 con 0x5626baa8c330 2024-02-21T23:11:19.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.292+0000 7fe5cb080740 10 client.9750 fetch_fsmap finished waiting for FSMap version 192 2024-02-21T23:11:19.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.292+0000 7fe5cb080740 10 client.9750 resolve_mds: resolved name 'c' to daemon mds.c 2024-02-21T23:11:19.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.292+0000 7fe5cb080740 20 client.9750 populate_metadata read hostname 'smithi028' 2024-02-21T23:11:19.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.292+0000 7fe5cb080740 1 --2- 172.21.15.28:0/1335305117 >> [v2:172.21.15.28:6832/2364631414,v1:172.21.15.28:6833/2364631414] conn(0x5626baad7760 0x5626baad7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:19.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.292+0000 7fe5cb080740 4 client.9750 mds_command: new command op to 9699 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-21T23:11:19.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.292+0000 7fe5cb080740 1 -- 172.21.15.28:0/1335305117 --> [v2:172.21.15.28:6832/2364631414,v1:172.21.15.28:6833/2364631414] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5626ba9fc9b0 con 0x5626baad7760 2024-02-21T23:11:19.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.293+0000 7fe5c4e00640 1 --2- 172.21.15.28:0/1335305117 >> [v2:172.21.15.28:6832/2364631414,v1:172.21.15.28:6833/2364631414] conn(0x5626baad7760 0x5626baad7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:19.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.293+0000 7fe5c4e00640 1 --2- 172.21.15.28:0/1335305117 >> [v2:172.21.15.28:6832/2364631414,v1:172.21.15.28:6833/2364631414] conn(0x5626baad7760 0x5626baad7b80 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5eaa952d27a6cf5b server_cookie=7364154d11d64ddd in_seq=0 out_seq=0 2024-02-21T23:11:19.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.293+0000 7fe5aaa00640 10 client.9750 ms_handle_connect on v2:172.21.15.28:6832/2364631414 2024-02-21T23:11:19.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.294+0000 7fe5aaa00640 1 -- 172.21.15.28:0/1335305117 <== mds.0 v2:172.21.15.28:6832/2364631414 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5626ba9fc9b0 con 0x5626baad7760 2024-02-21T23:11:19.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.294+0000 7fe5aaa00640 10 client.9750 handle_command_reply: tid=0 2024-02-21T23:11:19.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.294+0000 7fe5cb080740 1 client.9750 shutdown 2024-02-21T23:11:19.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.294+0000 7fe5cb080740 1 -- 172.21.15.28:0/1335305117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626baa8c330 msgr2=0x5626baacceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:19.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.294+0000 7fe5cb080740 1 --2- 172.21.15.28:0/1335305117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626baa8c330 0x5626baacceb0 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 crypto rx=0x7fe5b00074a0 tx=0x7fe5b0001250 comp rx=0 tx=0).stop 2024-02-21T23:11:19.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.294+0000 7fe5cb080740 1 -- 172.21.15.28:0/1335305117 shutdown_connections 2024-02-21T23:11:19.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.294+0000 7fe5cb080740 1 --2- 172.21.15.28:0/1335305117 >> [v2:172.21.15.28:6832/2364631414,v1:172.21.15.28:6833/2364631414] conn(0x5626baad7760 0x5626baad7b80 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:19.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.294+0000 7fe5cb080740 1 --2- 172.21.15.28:0/1335305117 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5626baac76b0 0x5626baadaaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:19.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.294+0000 7fe5cb080740 1 --2- 172.21.15.28:0/1335305117 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5626baaab5a0 0x5626baaa7130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:19.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.294+0000 7fe5cb080740 1 --2- 172.21.15.28:0/1335305117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626baa8c330 0x5626baacceb0 unknown :-1 s=CLOSED pgs=1980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:19.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.294+0000 7fe5aaa00640 0 client.9750 ms_handle_reset on v2:172.21.15.28:6832/2364631414 2024-02-21T23:11:19.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.294+0000 7fe5cb080740 1 -- 172.21.15.28:0/1335305117 >> 172.21.15.28:0/1335305117 conn(0x5626baa8b020 msgr2=0x5626baac5c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:19.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.295+0000 7fe5cb080740 1 -- 172.21.15.28:0/1335305117 shutdown_connections 2024-02-21T23:11:19.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.295+0000 7fe5cb080740 1 -- 172.21.15.28:0/1335305117 wait complete. 2024-02-21T23:11:19.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.295+0000 7fe5cb080740 20 client.9750 trim_cache size 0 max 16384 2024-02-21T23:11:19.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.313+0000 7fe5cb080740 1 Processor -- start 2024-02-21T23:11:19.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.313+0000 7fe5cb080740 1 -- start start 2024-02-21T23:11:19.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.314+0000 7fe5cb080740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626baa8c330 0x5626baac90e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:19.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.314+0000 7fe5cb080740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5626baaab5a0 0x5626bab2d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:19.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.314+0000 7fe5cb080740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5626baac76b0 0x5626bab2df10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:19.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.314+0000 7fe5cb080740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c01f6c00 con 0x5626baa8c330 2024-02-21T23:11:19.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.314+0000 7fe5cb080740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c0148e20 con 0x5626baac76b0 2024-02-21T23:11:19.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.314+0000 7fe5cb080740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c01f6600 con 0x5626baaab5a0 2024-02-21T23:11:19.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.314+0000 7fe5c5800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626baa8c330 0x5626baac90e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:19.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.314+0000 7fe5c5800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626baa8c330 0x5626baac90e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58888/0 (socket says 172.21.15.28:58888) 2024-02-21T23:11:19.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.314+0000 7fe5c5800640 1 -- 172.21.15.28:0/3203011091 learned_addr learned my addr 172.21.15.28:0/3203011091 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:19.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.315+0000 7fe5c5800640 1 -- 172.21.15.28:0/3203011091 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5626baaab5a0 msgr2=0x5626bab2d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:19.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.315+0000 7fe5c5800640 1 --2- 172.21.15.28:0/3203011091 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5626baaab5a0 0x5626bab2d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:19.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.315+0000 7fe5c5800640 1 -- 172.21.15.28:0/3203011091 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5626baac76b0 msgr2=0x5626bab2df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:19.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.315+0000 7fe5c5800640 1 --2- 172.21.15.28:0/3203011091 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5626baac76b0 0x5626bab2df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:19.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.315+0000 7fe5c5800640 1 -- 172.21.15.28:0/3203011091 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5626ba7ce6b0 con 0x5626baa8c330 2024-02-21T23:11:19.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.315+0000 7fe5c5800640 1 --2- 172.21.15.28:0/3203011091 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626baa8c330 0x5626baac90e0 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7fe5b0013440 tx=0x7fe5b0026000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81034e557c0cb518 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:19.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.315+0000 7fe5b5600640 1 -- 172.21.15.28:0/3203011091 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5b00134c0 con 0x5626baa8c330 2024-02-21T23:11:19.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.316+0000 7fe5b5600640 1 -- 172.21.15.28:0/3203011091 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5b00229d0 con 0x5626baa8c330 2024-02-21T23:11:19.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.316+0000 7fe5b5600640 1 -- 172.21.15.28:0/3203011091 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5b0022d80 con 0x5626baa8c330 2024-02-21T23:11:19.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.316+0000 7fe5cb080740 1 -- 172.21.15.28:0/3203011091 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626baa8c330 msgr2=0x5626baac90e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:19.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.316+0000 7fe5cb080740 1 --2- 172.21.15.28:0/3203011091 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626baa8c330 0x5626baac90e0 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7fe5b0013440 tx=0x7fe5b0026000 comp rx=0 tx=0).stop 2024-02-21T23:11:19.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.316+0000 7fe5cb080740 1 -- 172.21.15.28:0/3203011091 shutdown_connections 2024-02-21T23:11:19.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.316+0000 7fe5cb080740 1 --2- 172.21.15.28:0/3203011091 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5626baaab5a0 0x5626bab2d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:19.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.316+0000 7fe5cb080740 1 --2- 172.21.15.28:0/3203011091 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5626baac76b0 0x5626bab2df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:19.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.316+0000 7fe5cb080740 1 --2- 172.21.15.28:0/3203011091 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626baa8c330 0x5626baac90e0 unknown :-1 s=CLOSED pgs=1981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:19.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.316+0000 7fe5cb080740 1 -- 172.21.15.28:0/3203011091 >> 172.21.15.28:0/3203011091 conn(0x5626baa8b020 msgr2=0x5626baa80f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:19.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.316+0000 7fe5cb080740 1 -- 172.21.15.28:0/3203011091 shutdown_connections 2024-02-21T23:11:19.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.316+0000 7fe5cb080740 1 -- 172.21.15.28:0/3203011091 wait complete. 2024-02-21T23:11:19.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.317+0000 7fe5cb080740 1 -- start start 2024-02-21T23:11:19.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.317+0000 7fe5cb080740 1 Processor -- start 2024-02-21T23:11:19.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.318+0000 7fe5cb080740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5626baa8c330 0x5626baaa3b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:19.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.318+0000 7fe5cb080740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626baaab5a0 0x5626bab2d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:19.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.318+0000 7fe5cb080740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5626baac76b0 0x5626baadd880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:19.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.318+0000 7fe5cb080740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626baaabee0 con 0x5626baaab5a0 2024-02-21T23:11:19.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.318+0000 7fe5cb080740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626baaad470 con 0x5626baac76b0 2024-02-21T23:11:19.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.318+0000 7fe5cb080740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x5626baaabb70 con 0x5626baa8c330 2024-02-21T23:11:19.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.318+0000 7fe5cb080740 10 client.0 _ll_register_callbacks cb 0x7fe5b0007860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:11:19.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.318+0000 7fe5c4e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626baaab5a0 0x5626bab2d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:19.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.318+0000 7fe5c4e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626baaab5a0 0x5626bab2d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58892/0 (socket says 172.21.15.28:58892) 2024-02-21T23:11:19.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.319+0000 7fe5c4e00640 1 -- 172.21.15.28:0/2309060458 learned_addr learned my addr 172.21.15.28:0/2309060458 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:19.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.319+0000 7fe5c6200640 1 --2- 172.21.15.28:0/2309060458 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5626baac76b0 0x5626baadd880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:19.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.319+0000 7fe5c4e00640 1 -- 172.21.15.28:0/2309060458 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5626baa8c330 msgr2=0x5626baaa3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:19.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.319+0000 7fe5c4e00640 1 --2- 172.21.15.28:0/2309060458 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5626baa8c330 0x5626baaa3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:19.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.319+0000 7fe5c4e00640 1 -- 172.21.15.28:0/2309060458 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5626baac76b0 msgr2=0x5626baadd880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:19.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.319+0000 7fe5c4e00640 1 --2- 172.21.15.28:0/2309060458 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5626baac76b0 0x5626baadd880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:19.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.319+0000 7fe5c4e00640 1 -- 172.21.15.28:0/2309060458 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5626baac9af0 con 0x5626baaab5a0 2024-02-21T23:11:19.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.319+0000 7fe5c4e00640 1 --2- 172.21.15.28:0/2309060458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626baaab5a0 0x5626bab2d570 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7fe5c01e1730 tx=0x7fe5ac017810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1540316f7ddab02e server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:19.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.319+0000 7fe5aaa00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:11:19.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.319+0000 7fe5aaa00640 1 -- 172.21.15.28:0/2309060458 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5ac022070 con 0x5626baaab5a0 2024-02-21T23:11:19.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.320+0000 7fe5aaa00640 1 -- 172.21.15.28:0/2309060458 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(174..174 src has 1..174) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe5ac01e520 con 0x5626baaab5a0 2024-02-21T23:11:19.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.320+0000 7fe5cb080740 1 -- 172.21.15.28:0/2309060458 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe5b00229d0 con 0x5626baaab5a0 2024-02-21T23:11:19.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.320+0000 7fe5aaa00640 1 -- 172.21.15.28:0/2309060458 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=192) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe5ac02d050 con 0x5626baaab5a0 2024-02-21T23:11:19.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.320+0000 7fe5cb080740 10 client.9756 fetch_fsmap learned FSMap version 192 2024-02-21T23:11:19.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.320+0000 7fe5cb080740 1 -- 172.21.15.28:0/2309060458 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=192,monmap=2+}) v3 -- 0x5626bab368b0 con 0x5626baaab5a0 2024-02-21T23:11:19.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.320+0000 7fe5aaa00640 1 -- 172.21.15.28:0/2309060458 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5b00229d0 con 0x5626baaab5a0 2024-02-21T23:11:19.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.320+0000 7fe5aaa00640 1 -- 172.21.15.28:0/2309060458 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 192) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fe5ac030020 con 0x5626baaab5a0 2024-02-21T23:11:19.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.320+0000 7fe5cb080740 10 client.9756 fetch_fsmap finished waiting for FSMap version 192 2024-02-21T23:11:19.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.320+0000 7fe5cb080740 10 client.9756 resolve_mds: resolved name 'c' to daemon mds.c 2024-02-21T23:11:19.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.320+0000 7fe5cb080740 20 client.9756 populate_metadata read hostname 'smithi028' 2024-02-21T23:11:19.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.320+0000 7fe5cb080740 1 --2- 172.21.15.28:0/2309060458 >> [v2:172.21.15.28:6832/2364631414,v1:172.21.15.28:6833/2364631414] conn(0x5626baaca5d0 0x5626baaca9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:19.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.320+0000 7fe5cb080740 4 client.9756 mds_command: new command op to 9699 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-21T23:11:19.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.320+0000 7fe5cb080740 1 -- 172.21.15.28:0/2309060458 --> [v2:172.21.15.28:6832/2364631414,v1:172.21.15.28:6833/2364631414] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5626ba9f6d00 con 0x5626baaca5d0 2024-02-21T23:11:19.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.321+0000 7fe5c5800640 1 --2- 172.21.15.28:0/2309060458 >> [v2:172.21.15.28:6832/2364631414,v1:172.21.15.28:6833/2364631414] conn(0x5626baaca5d0 0x5626baaca9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:19.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.321+0000 7fe5c5800640 1 --2- 172.21.15.28:0/2309060458 >> [v2:172.21.15.28:6832/2364631414,v1:172.21.15.28:6833/2364631414] conn(0x5626baaca5d0 0x5626baaca9b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=578bd122da70bd84 server_cookie=b12c9f430c68154b in_seq=0 out_seq=0 2024-02-21T23:11:19.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.321+0000 7fe5aaa00640 10 client.9756 ms_handle_connect on v2:172.21.15.28:6832/2364631414 2024-02-21T23:11:19.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.321+0000 7fe5aaa00640 1 -- 172.21.15.28:0/2309060458 <== mds.0 v2:172.21.15.28:6832/2364631414 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+334 (crc 0 0 0) 0x5626ba9f6d00 con 0x5626baaca5d0 2024-02-21T23:11:19.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.321+0000 7fe5aaa00640 10 client.9756 handle_command_reply: tid=0 2024-02-21T23:11:19.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.321+0000 7fe5cb080740 1 client.9756 shutdown 2024-02-21T23:11:19.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.322+0000 7fe5cb080740 1 -- 172.21.15.28:0/2309060458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626baaab5a0 msgr2=0x5626bab2d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:19.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.322+0000 7fe5cb080740 1 --2- 172.21.15.28:0/2309060458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626baaab5a0 0x5626bab2d570 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7fe5c01e1730 tx=0x7fe5ac017810 comp rx=0 tx=0).stop 2024-02-21T23:11:19.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.322+0000 7fe5cb080740 1 -- 172.21.15.28:0/2309060458 shutdown_connections 2024-02-21T23:11:19.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.322+0000 7fe5cb080740 1 --2- 172.21.15.28:0/2309060458 >> [v2:172.21.15.28:6832/2364631414,v1:172.21.15.28:6833/2364631414] conn(0x5626baaca5d0 0x5626baaca9b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:19.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.322+0000 7fe5cb080740 1 --2- 172.21.15.28:0/2309060458 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5626baac76b0 0x5626baadd880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:19.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.322+0000 7fe5cb080740 1 --2- 172.21.15.28:0/2309060458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5626baaab5a0 0x5626bab2d570 unknown :-1 s=CLOSED pgs=1982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:19.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.322+0000 7fe5aaa00640 0 client.9756 ms_handle_reset on v2:172.21.15.28:6832/2364631414 2024-02-21T23:11:19.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.322+0000 7fe5cb080740 1 --2- 172.21.15.28:0/2309060458 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5626baa8c330 0x5626baaa3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:19.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.322+0000 7fe5cb080740 1 -- 172.21.15.28:0/2309060458 >> 172.21.15.28:0/2309060458 conn(0x5626baa8b020 msgr2=0x5626baaa6cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:19.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.322+0000 7fe5cb080740 1 -- 172.21.15.28:0/2309060458 shutdown_connections 2024-02-21T23:11:19.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.322+0000 7fe5cb080740 1 -- 172.21.15.28:0/2309060458 wait complete. 2024-02-21T23:11:19.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.322+0000 7fe5cb080740 20 client.9756 trim_cache size 0 max 16384 2024-02-21T23:11:19.318 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-21T23:11:19.318 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac", 2024-02-21T23:11:19.318 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-02-21T23:11:19.318 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 9699, 2024-02-21T23:11:19.318 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-02-21T23:11:19.318 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-02-21T23:11:19.318 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-02-21T23:11:19.319 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 3.6854693100000002, 2024-02-21T23:11:19.319 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 192, 2024-02-21T23:11:19.319 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 174, 2024-02-21T23:11:19.319 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 174, 2024-02-21T23:11:19.319 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 5.6475680869999998 2024-02-21T23:11:19.319 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-21T23:11:19.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.323+0000 7fe5c6c00640 1 -- 172.21.15.28:0/2130829962 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe5a006b4d0 msgr2=0x7fe5a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:19.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.323+0000 7fe5c6c00640 1 --2- 172.21.15.28:0/2130829962 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe5a006b4d0 0x7fe5a006d990 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fe5b0005c00 tx=0x7fe5b0009040 comp rx=0 tx=0).stop 2024-02-21T23:11:19.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.323+0000 7fe5c6c00640 1 -- 172.21.15.28:0/2130829962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5c0152740 msgr2=0x7fe5c0149ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:19.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.323+0000 7fe5c6c00640 1 --2- 172.21.15.28:0/2130829962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5c0152740 0x7fe5c0149ed0 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7fe5b8034040 tx=0x7fe5b80046c0 comp rx=0 tx=0).stop 2024-02-21T23:11:19.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.323+0000 7fe5c6c00640 1 -- 172.21.15.28:0/2130829962 shutdown_connections 2024-02-21T23:11:19.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.323+0000 7fe5c6c00640 1 --2- 172.21.15.28:0/2130829962 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe5a006b4d0 0x7fe5a006d990 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:19.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.323+0000 7fe5c6c00640 1 --2- 172.21.15.28:0/2130829962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5c0152740 0x7fe5c0149ed0 unknown :-1 s=CLOSED pgs=1978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:19.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.323+0000 7fe5c6c00640 1 --2- 172.21.15.28:0/2130829962 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe5c00c5260 0x7fe5c0149990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:19.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.323+0000 7fe5c6c00640 1 --2- 172.21.15.28:0/2130829962 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe5c00bf880 0x7fe5c01ee1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:19.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.323+0000 7fe5c6c00640 1 -- 172.21.15.28:0/2130829962 >> 172.21.15.28:0/2130829962 conn(0x7fe5c0067080 msgr2=0x7fe5c00c0120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:19.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.323+0000 7fe5c6c00640 1 -- 172.21.15.28:0/2130829962 shutdown_connections 2024-02-21T23:11:19.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.323+0000 7fe5c6c00640 1 -- 172.21.15.28:0/2130829962 wait complete. 2024-02-21T23:11:19.335 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:11:19.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.558+0000 7f8286600640 1 Processor -- start 2024-02-21T23:11:19.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.558+0000 7f8286600640 1 -- start start 2024-02-21T23:11:19.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.560+0000 7f8286600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f82800aff90 0x7f82800c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:19.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.560+0000 7f8286600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f828006dbf0 0x7f8280158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:19.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.560+0000 7f8286600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82800c74c0 con 0x7f82800c5720 2024-02-21T23:11:19.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.560+0000 7f8286600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82800c7270 con 0x7f82800aff90 2024-02-21T23:11:19.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.560+0000 7f8286600640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8280158630 con 0x7f828006dbf0 2024-02-21T23:11:19.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.560+0000 7f8285200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f82800aff90 0x7f82800c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:19.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.560+0000 7f8285c00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f82800c5720 0x7f828006c270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37854/0 (socket says 172.21.15.28:37854) 2024-02-21T23:11:19.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.560+0000 7f8285c00640 1 -- 172.21.15.28:0/881023358 learned_addr learned my addr 172.21.15.28:0/881023358 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:11:19.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.561+0000 7f8277e00640 1 --2- 172.21.15.28:0/881023358 >> v2:172.21.15.81:3300/0 conn(0x7f828006dbf0 0x7f8280158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:19.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.561+0000 7f827fe00640 1 -- 172.21.15.28:0/881023358 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1594655509 0 0) 0x7f82800c74c0 con 0x7f82800c5720 2024-02-21T23:11:19.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.561+0000 7f827fe00640 1 -- 172.21.15.28:0/881023358 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8264003520 con 0x7f82800c5720 2024-02-21T23:11:19.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.561+0000 7f8285200640 1 -- 172.21.15.28:0/881023358 >> v2:172.21.15.81:3300/0 conn(0x7f828006dbf0 msgr2=0x7f8280158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:19.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.561+0000 7f8285200640 1 --2- 172.21.15.28:0/881023358 >> v2:172.21.15.81:3300/0 conn(0x7f828006dbf0 0x7f8280158ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:19.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.561+0000 7f8285200640 1 -- 172.21.15.28:0/881023358 >> v1:172.21.15.28:6789/0 conn(0x7f82800c5720 legacy=0x7f828006c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:19.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.561+0000 7f8285200640 1 -- 172.21.15.28:0/881023358 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82800c77a0 con 0x7f82800aff90 2024-02-21T23:11:19.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.562+0000 7f8285200640 1 --2- 172.21.15.28:0/881023358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82800aff90 0x7f82800c51e0 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7f82800c73f0 tx=0x7f827002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=445cc31d3aa98a1a server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:19.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.563+0000 7f827fe00640 1 -- 172.21.15.28:0/881023358 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8270035020 con 0x7f82800aff90 2024-02-21T23:11:19.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.563+0000 7f827fe00640 1 -- 172.21.15.28:0/881023358 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8270031830 con 0x7f82800aff90 2024-02-21T23:11:19.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.563+0000 7f827fe00640 1 -- 172.21.15.28:0/881023358 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8270031c00 con 0x7f82800aff90 2024-02-21T23:11:19.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.564+0000 7f8286600640 1 -- 172.21.15.28:0/881023358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82800aff90 msgr2=0x7f82800c51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:19.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.564+0000 7f8286600640 1 --2- 172.21.15.28:0/881023358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82800aff90 0x7f82800c51e0 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7f82800c73f0 tx=0x7f827002b8d0 comp rx=0 tx=0).stop 2024-02-21T23:11:19.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.564+0000 7f8286600640 1 -- 172.21.15.28:0/881023358 shutdown_connections 2024-02-21T23:11:19.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.564+0000 7f8286600640 1 --2- 172.21.15.28:0/881023358 >> v2:172.21.15.81:3300/0 conn(0x7f828006dbf0 0x7f8280158ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:19.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.564+0000 7f8286600640 1 --2- 172.21.15.28:0/881023358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82800aff90 0x7f82800c51e0 unknown :-1 s=CLOSED pgs=1984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:19.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.564+0000 7f8286600640 1 -- 172.21.15.28:0/881023358 >> 172.21.15.28:0/881023358 conn(0x7f82800ac370 msgr2=0x7f82800ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:19.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.564+0000 7f8286600640 1 -- 172.21.15.28:0/881023358 shutdown_connections 2024-02-21T23:11:19.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.564+0000 7f8286600640 1 -- 172.21.15.28:0/881023358 wait complete. 2024-02-21T23:11:19.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.566+0000 7f8286600640 1 Processor -- start 2024-02-21T23:11:19.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.566+0000 7f8286600640 1 -- start start 2024-02-21T23:11:19.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.568+0000 7f8286600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f828006dbf0 0x7f82800c1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:19.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.568+0000 7f8286600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f82800aff90 0x7f82800aebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:19.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.568+0000 7f8286600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82800c5720 0x7f82801eb180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:19.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.568+0000 7f8286600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82800c8760 con 0x7f82800c5720 2024-02-21T23:11:19.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.568+0000 7f8286600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82800be0c0 con 0x7f828006dbf0 2024-02-21T23:11:19.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.568+0000 7f8286600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82800c88e0 con 0x7f82800aff90 2024-02-21T23:11:19.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.568+0000 7f8285c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82800c5720 0x7f82801eb180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:19.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.568+0000 7f8285c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82800c5720 0x7f82801eb180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58900/0 (socket says 172.21.15.28:58900) 2024-02-21T23:11:19.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.568+0000 7f8285c00640 1 -- 172.21.15.28:0/3231777133 learned_addr learned my addr 172.21.15.28:0/3231777133 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:19.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.568+0000 7f8285200640 1 --2- 172.21.15.28:0/3231777133 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f828006dbf0 0x7f82800c1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:19.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.568+0000 7f8277e00640 1 --2- 172.21.15.28:0/3231777133 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f82800aff90 0x7f82800aebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:19.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.569+0000 7f8285c00640 1 -- 172.21.15.28:0/3231777133 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f82800aff90 msgr2=0x7f82800aebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:19.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.569+0000 7f8285c00640 1 --2- 172.21.15.28:0/3231777133 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f82800aff90 0x7f82800aebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:19.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.569+0000 7f8285c00640 1 -- 172.21.15.28:0/3231777133 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f828006dbf0 msgr2=0x7f82800c1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:19.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.569+0000 7f8285c00640 1 --2- 172.21.15.28:0/3231777133 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f828006dbf0 0x7f82800c1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:19.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.569+0000 7f8285c00640 1 -- 172.21.15.28:0/3231777133 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82801eead0 con 0x7f82800c5720 2024-02-21T23:11:19.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.569+0000 7f8277e00640 1 --2- 172.21.15.28:0/3231777133 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f82800aff90 0x7f82800aebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:11:19.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.569+0000 7f8285c00640 1 --2- 172.21.15.28:0/3231777133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82800c5720 0x7f82801eb180 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7f827800caa0 tx=0x7f827800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:19.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.569+0000 7f827d600640 1 -- 172.21.15.28:0/3231777133 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8278017070 con 0x7f82800c5720 2024-02-21T23:11:19.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.569+0000 7f827d600640 1 -- 172.21.15.28:0/3231777133 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f827800aba0 con 0x7f82800c5720 2024-02-21T23:11:19.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.570+0000 7f8286600640 1 -- 172.21.15.28:0/3231777133 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82801eedb0 con 0x7f82800c5720 2024-02-21T23:11:19.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.570+0000 7f827d600640 1 -- 172.21.15.28:0/3231777133 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8278012400 con 0x7f82800c5720 2024-02-21T23:11:19.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.570+0000 7f8286600640 1 -- 172.21.15.28:0/3231777133 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82801eac60 con 0x7f82800c5720 2024-02-21T23:11:19.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.571+0000 7f8286600640 1 -- 172.21.15.28:0/3231777133 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8280156e90 con 0x7f82800c5720 2024-02-21T23:11:19.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.572+0000 7f827d600640 1 -- 172.21.15.28:0/3231777133 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f827801a070 con 0x7f82800c5720 2024-02-21T23:11:19.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.573+0000 7f827d600640 1 --2- 172.21.15.28:0/3231777133 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f825c06b4d0 0x7f825c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:19.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.573+0000 7f827d600640 1 -- 172.21.15.28:0/3231777133 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f827808a0b0 con 0x7f82800c5720 2024-02-21T23:11:19.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.573+0000 7f8285200640 1 --2- 172.21.15.28:0/3231777133 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f825c06b4d0 0x7f825c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:19.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.575+0000 7f8285200640 1 --2- 172.21.15.28:0/3231777133 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f825c06b4d0 0x7f825c06d990 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f8270036000 tx=0x7f8270008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:19.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.580+0000 7f827d600640 1 -- 172.21.15.28:0/3231777133 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f8280156e90 con 0x7f82800c5720 2024-02-21T23:11:19.614 INFO:tasks.ceph.mds.c.smithi028.stderr:2024-02-21T23:11:19.625+0000 7f6d00c00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-21T23:11:19.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.824+0000 7f8286600640 1 -- 172.21.15.28:0/3231777133 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f828004f290 con 0x7f82800c5720 2024-02-21T23:11:19.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.825+0000 7f827d600640 1 -- 172.21.15.28:0/3231777133 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 192 v192) v1 ==== 95+0+4945 (secure 0 0 0) 0x7f8278058740 con 0x7f82800c5720 2024-02-21T23:11:19.814 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:11:19.814 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":192,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6686,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/155806034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":155806034},{"type":"v1","addr":"172.21.15.81:6836","nonce":155806034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":180},{"gid":7480,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3897243745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":3897243745},{"type":"v1","addr":"172.21.15.81:6837","nonce":3897243745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":180},{"gid":7486,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2827135632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2827135632},{"type":"v1","addr":"172.21.15.28:6835","nonce":2827135632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":180}],"filesystems":[{"mdsmap":{"epoch":192,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:10:53.351225+0000","modified":"2024-02-21T23:11:15.654038+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9699},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9699":{"gid":9699,"name":"c","rank":0,"incarnation":191,"state":"up:active","state_seq":2,"addr":"172.21.15.28:6833/2364631414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2364631414},{"type":"v1","addr":"172.21.15.28:6833","nonce":2364631414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2024-02-21T23:11:19.815 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 192 2024-02-21T23:11:19.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.828+0000 7f8286600640 1 -- 172.21.15.28:0/3231777133 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f825c06b4d0 msgr2=0x7f825c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:19.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.828+0000 7f8286600640 1 --2- 172.21.15.28:0/3231777133 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f825c06b4d0 0x7f825c06d990 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f8270036000 tx=0x7f8270008030 comp rx=0 tx=0).stop 2024-02-21T23:11:19.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.828+0000 7f8286600640 1 -- 172.21.15.28:0/3231777133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82800c5720 msgr2=0x7f82801eb180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:19.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.828+0000 7f8286600640 1 --2- 172.21.15.28:0/3231777133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82800c5720 0x7f82801eb180 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7f827800caa0 tx=0x7f827800cf60 comp rx=0 tx=0).stop 2024-02-21T23:11:19.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.828+0000 7f8286600640 1 -- 172.21.15.28:0/3231777133 shutdown_connections 2024-02-21T23:11:19.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.828+0000 7f8286600640 1 --2- 172.21.15.28:0/3231777133 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f825c06b4d0 0x7f825c06d990 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:19.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.828+0000 7f8286600640 1 --2- 172.21.15.28:0/3231777133 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f82800aff90 0x7f82800aebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:19.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.828+0000 7f8286600640 1 --2- 172.21.15.28:0/3231777133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82800c5720 0x7f82801eb180 unknown :-1 s=CLOSED pgs=1985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:19.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.828+0000 7f8286600640 1 --2- 172.21.15.28:0/3231777133 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f828006dbf0 0x7f82800c1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:19.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.828+0000 7f8286600640 1 -- 172.21.15.28:0/3231777133 >> 172.21.15.28:0/3231777133 conn(0x7f82800ac370 msgr2=0x7f82800ac750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:19.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.828+0000 7f8286600640 1 -- 172.21.15.28:0/3231777133 shutdown_connections 2024-02-21T23:11:19.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:19.828+0000 7f8286600640 1 -- 172.21.15.28:0/3231777133 wait complete. 2024-02-21T23:11:19.837 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-02-21T23:11:19.837 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-21T23:11:19.946 DEBUG:teuthology.orchestra.run.smithi081:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-21T23:11:20.068 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-02-21T23:11:20.069 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-21T23:11:20.069 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-21T23:11:20.126 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054608,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-02-21T23:11:20.127 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-02-21T23:11:20.127 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-21T23:11:20.185 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053833,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498812k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249404k,nr_inodes=812351,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498812k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-02-21T23:11:20.185 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:11:20.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.443+0000 7f72a3200640 1 Processor -- start 2024-02-21T23:11:20.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.443+0000 7f72a3200640 1 -- start start 2024-02-21T23:11:20.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.444+0000 7f72a3200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f72a4150450 0x7f72a40a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:20.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.444+0000 7f72a3200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f72a406ee40 0x7f72a41536f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:20.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.444+0000 7f72a3200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72a40c2760 con 0x7f72a4156af0 2024-02-21T23:11:20.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.444+0000 7f72a3200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72a40c22c0 con 0x7f72a406ee40 2024-02-21T23:11:20.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.444+0000 7f72a3200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72a4156660 con 0x7f72a4150450 2024-02-21T23:11:20.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.445+0000 7f72a1400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f72a406ee40 0x7f72a41536f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:20.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.445+0000 7f72a1400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f72a406ee40 0x7f72a41536f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58912/0 (socket says 172.21.15.28:58912) 2024-02-21T23:11:20.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.445+0000 7f72a1400640 1 -- 172.21.15.28:0/3169524914 learned_addr learned my addr 172.21.15.28:0/3169524914 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:20.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.445+0000 7f72a2800640 1 --1- 172.21.15.28:0/3169524914 >> v1:172.21.15.28:6789/0 conn(0x7f72a4156af0 0x7f72a406dff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37856/0 (socket says 172.21.15.28:37856) 2024-02-21T23:11:20.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.445+0000 7f72a1e00640 1 --2- 172.21.15.28:0/3169524914 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f72a4150450 0x7f72a40a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:20.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.445+0000 7f72a1400640 1 -- 172.21.15.28:0/3169524914 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f72a4150450 msgr2=0x7f72a40a5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:20.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.445+0000 7f72a1400640 1 --2- 172.21.15.28:0/3169524914 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f72a4150450 0x7f72a40a5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:20.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.445+0000 7f72a1400640 1 -- 172.21.15.28:0/3169524914 >> v1:172.21.15.28:6789/0 conn(0x7f72a4156af0 legacy=0x7f72a406dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:20.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.445+0000 7f72a0a00640 1 -- 172.21.15.28:0/3169524914 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 226385218 0 0) 0x7f72a40c2760 con 0x7f72a4156af0 2024-02-21T23:11:20.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.445+0000 7f72a1400640 1 -- 172.21.15.28:0/3169524914 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72a4153c30 con 0x7f72a406ee40 2024-02-21T23:11:20.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.447+0000 7f72a1400640 1 --2- 172.21.15.28:0/3169524914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72a406ee40 0x7f72a41536f0 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7f7294009e50 tx=0x7f729402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aac2a3bd761009a0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:20.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.447+0000 7f72a0a00640 1 -- 172.21.15.28:0/3169524914 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f729402cb00 con 0x7f72a406ee40 2024-02-21T23:11:20.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.447+0000 7f72a0a00640 1 -- 172.21.15.28:0/3169524914 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f729402cca0 con 0x7f72a406ee40 2024-02-21T23:11:20.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.447+0000 7f72a0a00640 1 -- 172.21.15.28:0/3169524914 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7294034610 con 0x7f72a406ee40 2024-02-21T23:11:20.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.448+0000 7f72a3200640 1 -- 172.21.15.28:0/3169524914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72a406ee40 msgr2=0x7f72a41536f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:20.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.448+0000 7f72a3200640 1 --2- 172.21.15.28:0/3169524914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72a406ee40 0x7f72a41536f0 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7f7294009e50 tx=0x7f729402bf60 comp rx=0 tx=0).stop 2024-02-21T23:11:20.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.448+0000 7f72a3200640 1 -- 172.21.15.28:0/3169524914 shutdown_connections 2024-02-21T23:11:20.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.448+0000 7f72a3200640 1 --2- 172.21.15.28:0/3169524914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72a406ee40 0x7f72a41536f0 unknown :-1 s=CLOSED pgs=1987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:20.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.448+0000 7f72a3200640 1 --2- 172.21.15.28:0/3169524914 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f72a4150450 0x7f72a40a5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:20.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.448+0000 7f72a3200640 1 -- 172.21.15.28:0/3169524914 >> 172.21.15.28:0/3169524914 conn(0x7f72a406b630 msgr2=0x7f72a40c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:20.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.448+0000 7f72a3200640 1 -- 172.21.15.28:0/3169524914 shutdown_connections 2024-02-21T23:11:20.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.448+0000 7f72a3200640 1 -- 172.21.15.28:0/3169524914 wait complete. 2024-02-21T23:11:20.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.450+0000 7f72a3200640 1 Processor -- start 2024-02-21T23:11:20.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.450+0000 7f72a3200640 1 -- start start 2024-02-21T23:11:20.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.451+0000 7f72a3200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f72a406ee40 0x7f72a41ee760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:20.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.451+0000 7f72a3200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72a4150450 0x7f72a40bfba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:20.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.451+0000 7f72a3200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72a4156af0 0x7f72a41ea7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:20.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.451+0000 7f72a3200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72a40c2cb0 con 0x7f72a4150450 2024-02-21T23:11:20.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.451+0000 7f72a3200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72a40c29d0 con 0x7f72a4156af0 2024-02-21T23:11:20.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.451+0000 7f72a3200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72a406e700 con 0x7f72a406ee40 2024-02-21T23:11:20.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.451+0000 7f72a1400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72a4150450 0x7f72a40bfba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:20.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.451+0000 7f72a1400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72a4150450 0x7f72a40bfba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58918/0 (socket says 172.21.15.28:58918) 2024-02-21T23:11:20.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.451+0000 7f72a1400640 1 -- 172.21.15.28:0/2726618060 learned_addr learned my addr 172.21.15.28:0/2726618060 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:20.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.451+0000 7f72a2800640 1 --2- 172.21.15.28:0/2726618060 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72a4156af0 0x7f72a41ea7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:20.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.451+0000 7f72a1400640 1 -- 172.21.15.28:0/2726618060 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f72a406ee40 msgr2=0x7f72a41ee760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:20.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.451+0000 7f72a1e00640 1 --2- 172.21.15.28:0/2726618060 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f72a406ee40 0x7f72a41ee760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:20.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.451+0000 7f72a1400640 1 --2- 172.21.15.28:0/2726618060 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f72a406ee40 0x7f72a41ee760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:20.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.451+0000 7f72a1400640 1 -- 172.21.15.28:0/2726618060 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72a4156af0 msgr2=0x7f72a41ea7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:20.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.451+0000 7f72a1400640 1 --2- 172.21.15.28:0/2726618060 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72a4156af0 0x7f72a41ea7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:20.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.451+0000 7f72a1400640 1 -- 172.21.15.28:0/2726618060 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72a4073070 con 0x7f72a4150450 2024-02-21T23:11:20.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.452+0000 7f72a1400640 1 --2- 172.21.15.28:0/2726618060 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72a4150450 0x7f72a40bfba0 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7f7294036040 tx=0x7f7294004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:20.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.452+0000 7f729a000640 1 -- 172.21.15.28:0/2726618060 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7294043020 con 0x7f72a4150450 2024-02-21T23:11:20.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.452+0000 7f729a000640 1 -- 172.21.15.28:0/2726618060 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f729402ce50 con 0x7f72a4150450 2024-02-21T23:11:20.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.452+0000 7f729a000640 1 -- 172.21.15.28:0/2726618060 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f729403e690 con 0x7f72a4150450 2024-02-21T23:11:20.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.452+0000 7f72a3200640 1 -- 172.21.15.28:0/2726618060 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72a4151550 con 0x7f72a4150450 2024-02-21T23:11:20.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.453+0000 7f72a3200640 1 -- 172.21.15.28:0/2726618060 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72a40c00e0 con 0x7f72a4150450 2024-02-21T23:11:20.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.454+0000 7f72a3200640 1 -- 172.21.15.28:0/2726618060 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72a404f290 con 0x7f72a4150450 2024-02-21T23:11:20.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.455+0000 7f729a000640 1 -- 172.21.15.28:0/2726618060 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7294046020 con 0x7f72a4150450 2024-02-21T23:11:20.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.456+0000 7f729a000640 1 --2- 172.21.15.28:0/2726618060 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f728006b4d0 0x7f728006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:20.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.456+0000 7f729a000640 1 -- 172.21.15.28:0/2726618060 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f72940ac900 con 0x7f72a4150450 2024-02-21T23:11:20.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.456+0000 7f72a1e00640 1 --2- 172.21.15.28:0/2726618060 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f728006b4d0 0x7f728006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:20.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.458+0000 7f72a1e00640 1 --2- 172.21.15.28:0/2726618060 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f728006b4d0 0x7f728006d990 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f728c006fd0 tx=0x7f728c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:20.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.460+0000 7f729a000640 1 -- 172.21.15.28:0/2726618060 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f729404d030 con 0x7f72a4150450 2024-02-21T23:11:20.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.699+0000 7f72a3200640 1 -- 172.21.15.28:0/2726618060 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f72a4153700 con 0x7f72a4150450 2024-02-21T23:11:20.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.700+0000 7f729a000640 1 -- 172.21.15.28:0/2726618060 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 192 v192) v1 ==== 95+0+4945 (secure 0 0 0) 0x7f729407b680 con 0x7f72a4150450 2024-02-21T23:11:20.689 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:11:20.689 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":192,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6686,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/155806034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":155806034},{"type":"v1","addr":"172.21.15.81:6836","nonce":155806034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":180},{"gid":7480,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3897243745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":3897243745},{"type":"v1","addr":"172.21.15.81:6837","nonce":3897243745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":180},{"gid":7486,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2827135632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2827135632},{"type":"v1","addr":"172.21.15.28:6835","nonce":2827135632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":180}],"filesystems":[{"mdsmap":{"epoch":192,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:10:53.351225+0000","modified":"2024-02-21T23:11:15.654038+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9699},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9699":{"gid":9699,"name":"c","rank":0,"incarnation":191,"state":"up:active","state_seq":2,"addr":"172.21.15.28:6833/2364631414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2364631414},{"type":"v1","addr":"172.21.15.28:6833","nonce":2364631414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2024-02-21T23:11:20.689 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 192 2024-02-21T23:11:20.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.703+0000 7f72a3200640 1 -- 172.21.15.28:0/2726618060 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f728006b4d0 msgr2=0x7f728006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:20.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.703+0000 7f72a3200640 1 --2- 172.21.15.28:0/2726618060 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f728006b4d0 0x7f728006d990 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f728c006fd0 tx=0x7f728c008040 comp rx=0 tx=0).stop 2024-02-21T23:11:20.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.703+0000 7f72a3200640 1 -- 172.21.15.28:0/2726618060 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72a4150450 msgr2=0x7f72a40bfba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:20.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.703+0000 7f72a3200640 1 --2- 172.21.15.28:0/2726618060 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72a4150450 0x7f72a40bfba0 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7f7294036040 tx=0x7f7294004a40 comp rx=0 tx=0).stop 2024-02-21T23:11:20.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.703+0000 7f72a3200640 1 -- 172.21.15.28:0/2726618060 shutdown_connections 2024-02-21T23:11:20.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.703+0000 7f72a3200640 1 --2- 172.21.15.28:0/2726618060 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f728006b4d0 0x7f728006d990 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:20.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.703+0000 7f72a3200640 1 --2- 172.21.15.28:0/2726618060 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72a4156af0 0x7f72a41ea7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:20.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.703+0000 7f72a3200640 1 --2- 172.21.15.28:0/2726618060 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72a4150450 0x7f72a40bfba0 unknown :-1 s=CLOSED pgs=1988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:20.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.703+0000 7f72a3200640 1 --2- 172.21.15.28:0/2726618060 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f72a406ee40 0x7f72a41ee760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:20.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.703+0000 7f72a3200640 1 -- 172.21.15.28:0/2726618060 >> 172.21.15.28:0/2726618060 conn(0x7f72a406b630 msgr2=0x7f72a414c8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:20.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.703+0000 7f72a3200640 1 -- 172.21.15.28:0/2726618060 shutdown_connections 2024-02-21T23:11:20.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.703+0000 7f72a3200640 1 -- 172.21.15.28:0/2726618060 wait complete. 2024-02-21T23:11:20.711 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:11:20.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.943+0000 7f1b13a00640 1 Processor -- start 2024-02-21T23:11:20.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.943+0000 7f1b13a00640 1 -- start start 2024-02-21T23:11:20.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.944+0000 7f1b13a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1b0c0c3280 0x7f1b0c0c3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:20.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.944+0000 7f1b13a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1b0c072430 0x7f1b0c1526e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:20.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.944+0000 7f1b13a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f1b0c155860 0x7f1b0c155d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:20.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.944+0000 7f1b13a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b0c152c20 con 0x7f1b0c072430 2024-02-21T23:11:20.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.944+0000 7f1b13a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b0c0c6560 con 0x7f1b0c155860 2024-02-21T23:11:20.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.944+0000 7f1b13a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b0c0c5cc0 con 0x7f1b0c0c3280 2024-02-21T23:11:20.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.945+0000 7f1b0be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1b0c072430 0x7f1b0c1526e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:20.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.945+0000 7f1b0be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1b0c072430 0x7f1b0c1526e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58920/0 (socket says 172.21.15.28:58920) 2024-02-21T23:11:20.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.945+0000 7f1b0be00640 1 -- 172.21.15.28:0/2322427333 learned_addr learned my addr 172.21.15.28:0/2322427333 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:20.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.945+0000 7f1b0be00640 1 -- 172.21.15.28:0/2322427333 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1b0c0c3280 msgr2=0x7f1b0c0c3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:20.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.945+0000 7f1b0be00640 1 --2- 172.21.15.28:0/2322427333 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1b0c0c3280 0x7f1b0c0c3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:20.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.945+0000 7f1b0be00640 1 -- 172.21.15.28:0/2322427333 >> v2:172.21.15.81:3300/0 conn(0x7f1b0c155860 msgr2=0x7f1b0c155d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:20.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.945+0000 7f1b0be00640 1 --2- 172.21.15.28:0/2322427333 >> v2:172.21.15.81:3300/0 conn(0x7f1b0c155860 0x7f1b0c155d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:20.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.945+0000 7f1b0be00640 1 -- 172.21.15.28:0/2322427333 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b0c0c5fa0 con 0x7f1b0c072430 2024-02-21T23:11:20.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.946+0000 7f1b0be00640 1 --2- 172.21.15.28:0/2322427333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b0c072430 0x7f1b0c1526e0 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c0c0c80 tx=0x7f1af402b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbb269638d832efb server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:20.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.947+0000 7f1b0b400640 1 -- 172.21.15.28:0/2322427333 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1af40317d0 con 0x7f1b0c072430 2024-02-21T23:11:20.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.947+0000 7f1b0b400640 1 -- 172.21.15.28:0/2322427333 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1af4037070 con 0x7f1b0c072430 2024-02-21T23:11:20.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.947+0000 7f1b0b400640 1 -- 172.21.15.28:0/2322427333 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1af4031b90 con 0x7f1b0c072430 2024-02-21T23:11:20.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.947+0000 7f1b13a00640 1 -- 172.21.15.28:0/2322427333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b0c072430 msgr2=0x7f1b0c1526e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:20.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.947+0000 7f1b13a00640 1 --2- 172.21.15.28:0/2322427333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b0c072430 0x7f1b0c1526e0 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c0c0c80 tx=0x7f1af402b870 comp rx=0 tx=0).stop 2024-02-21T23:11:20.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.947+0000 7f1b13a00640 1 -- 172.21.15.28:0/2322427333 shutdown_connections 2024-02-21T23:11:20.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.947+0000 7f1b13a00640 1 --2- 172.21.15.28:0/2322427333 >> v2:172.21.15.81:3300/0 conn(0x7f1b0c155860 0x7f1b0c155d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:20.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.947+0000 7f1b13a00640 1 --2- 172.21.15.28:0/2322427333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b0c072430 0x7f1b0c1526e0 unknown :-1 s=CLOSED pgs=1989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:20.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.947+0000 7f1b13a00640 1 --2- 172.21.15.28:0/2322427333 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1b0c0c3280 0x7f1b0c0c3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:20.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.947+0000 7f1b13a00640 1 -- 172.21.15.28:0/2322427333 >> 172.21.15.28:0/2322427333 conn(0x7f1b0c0c67f0 msgr2=0x7f1b0c0c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:20.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.948+0000 7f1b13a00640 1 -- 172.21.15.28:0/2322427333 shutdown_connections 2024-02-21T23:11:20.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.948+0000 7f1b13a00640 1 -- 172.21.15.28:0/2322427333 wait complete. 2024-02-21T23:11:20.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.949+0000 7f1b13a00640 1 Processor -- start 2024-02-21T23:11:20.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.949+0000 7f1b13a00640 1 -- start start 2024-02-21T23:11:20.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.950+0000 7f1b13a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1b0c072430 0x7f1b0c1eabb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:20.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.950+0000 7f1b13a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b0c0c3280 0x7f1b0c143f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:20.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.950+0000 7f1b13a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1b0c155860 0x7f1b0c13e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:20.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.950+0000 7f1b13a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b0c0c5920 con 0x7f1b0c0c3280 2024-02-21T23:11:20.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.950+0000 7f1b13a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b0c0a6100 con 0x7f1b0c072430 2024-02-21T23:11:20.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.950+0000 7f1b13a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b0c0c5aa0 con 0x7f1b0c155860 2024-02-21T23:11:20.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.951+0000 7f1b0be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b0c0c3280 0x7f1b0c143f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:20.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.951+0000 7f1b0be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b0c0c3280 0x7f1b0c143f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58924/0 (socket says 172.21.15.28:58924) 2024-02-21T23:11:20.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.951+0000 7f1b0be00640 1 -- 172.21.15.28:0/2011718595 learned_addr learned my addr 172.21.15.28:0/2011718595 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:20.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.951+0000 7f1b0be00640 1 -- 172.21.15.28:0/2011718595 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1b0c155860 msgr2=0x7f1b0c13e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:20.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.951+0000 7f1b0be00640 1 --2- 172.21.15.28:0/2011718595 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1b0c155860 0x7f1b0c13e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:20.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.951+0000 7f1b0be00640 1 -- 172.21.15.28:0/2011718595 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1b0c072430 msgr2=0x7f1b0c1eabb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:20.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.951+0000 7f1b11000640 1 --2- 172.21.15.28:0/2011718595 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1b0c072430 0x7f1b0c1eabb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:20.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.951+0000 7f1b0be00640 1 --2- 172.21.15.28:0/2011718595 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1b0c072430 0x7f1b0c1eabb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:20.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.951+0000 7f1b0be00640 1 -- 172.21.15.28:0/2011718595 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b0c142ff0 con 0x7f1b0c0c3280 2024-02-21T23:11:20.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.951+0000 7f1b0be00640 1 --2- 172.21.15.28:0/2011718595 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b0c0c3280 0x7f1b0c143f70 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7f1af4000c00 tx=0x7f1af40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:20.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.952+0000 7f1b08c00640 1 -- 172.21.15.28:0/2011718595 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1af4032870 con 0x7f1b0c0c3280 2024-02-21T23:11:20.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.952+0000 7f1b08c00640 1 -- 172.21.15.28:0/2011718595 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1af4032a10 con 0x7f1b0c0c3280 2024-02-21T23:11:20.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.952+0000 7f1b13a00640 1 -- 172.21.15.28:0/2011718595 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b0c1432d0 con 0x7f1b0c0c3280 2024-02-21T23:11:20.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.952+0000 7f1b08c00640 1 -- 172.21.15.28:0/2011718595 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1af4032e00 con 0x7f1b0c0c3280 2024-02-21T23:11:20.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.952+0000 7f1b13a00640 1 -- 172.21.15.28:0/2011718595 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b0c143890 con 0x7f1b0c0c3280 2024-02-21T23:11:20.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.953+0000 7f1b13a00640 1 -- 172.21.15.28:0/2011718595 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b0c067f30 con 0x7f1b0c0c3280 2024-02-21T23:11:20.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.954+0000 7f1b08c00640 1 -- 172.21.15.28:0/2011718595 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1af404c060 con 0x7f1b0c0c3280 2024-02-21T23:11:20.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.955+0000 7f1b08c00640 1 --2- 172.21.15.28:0/2011718595 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1ae806b4d0 0x7f1ae806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:20.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.955+0000 7f1b08c00640 1 -- 172.21.15.28:0/2011718595 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1af40ac1c0 con 0x7f1b0c0c3280 2024-02-21T23:11:20.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.956+0000 7f1b11000640 1 --2- 172.21.15.28:0/2011718595 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1ae806b4d0 0x7f1ae806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:20.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.957+0000 7f1b11000640 1 --2- 172.21.15.28:0/2011718595 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1ae806b4d0 0x7f1ae806d990 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f1afc007dc0 tx=0x7f1afc0072f0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:20.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:20.964+0000 7f1b08c00640 1 -- 172.21.15.28:0/2011718595 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1af4040070 con 0x7f1b0c0c3280 2024-02-21T23:11:21.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.169+0000 7f1b13a00640 1 -- 172.21.15.28:0/2011718595 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1b0c06f2d0 con 0x7f1b0c0c3280 2024-02-21T23:11:21.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.172+0000 7f1b08c00640 1 -- 172.21.15.28:0/2011718595 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v174) v1 ==== 74+0+16892 (secure 0 0 0) 0x7f1af407aea0 con 0x7f1b0c0c3280 2024-02-21T23:11:21.161 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:11:21.161 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":174,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:10:54.863442+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:10:51.223834+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"173","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2024-02-21T23:10:52.368371+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"174","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:11:21.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.174+0000 7f1b13a00640 1 -- 172.21.15.28:0/2011718595 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1ae806b4d0 msgr2=0x7f1ae806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:21.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.174+0000 7f1b13a00640 1 --2- 172.21.15.28:0/2011718595 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1ae806b4d0 0x7f1ae806d990 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f1afc007dc0 tx=0x7f1afc0072f0 comp rx=0 tx=0).stop 2024-02-21T23:11:21.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.174+0000 7f1b13a00640 1 -- 172.21.15.28:0/2011718595 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b0c0c3280 msgr2=0x7f1b0c143f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:21.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.174+0000 7f1b13a00640 1 --2- 172.21.15.28:0/2011718595 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b0c0c3280 0x7f1b0c143f70 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7f1af4000c00 tx=0x7f1af40046c0 comp rx=0 tx=0).stop 2024-02-21T23:11:21.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.175+0000 7f1b13a00640 1 -- 172.21.15.28:0/2011718595 shutdown_connections 2024-02-21T23:11:21.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.175+0000 7f1b13a00640 1 --2- 172.21.15.28:0/2011718595 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1ae806b4d0 0x7f1ae806d990 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:21.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.175+0000 7f1b13a00640 1 --2- 172.21.15.28:0/2011718595 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1b0c155860 0x7f1b0c13e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:21.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.175+0000 7f1b13a00640 1 --2- 172.21.15.28:0/2011718595 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b0c0c3280 0x7f1b0c143f70 unknown :-1 s=CLOSED pgs=1990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:21.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.175+0000 7f1b13a00640 1 --2- 172.21.15.28:0/2011718595 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1b0c072430 0x7f1b0c1eabb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:21.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.175+0000 7f1b13a00640 1 -- 172.21.15.28:0/2011718595 >> 172.21.15.28:0/2011718595 conn(0x7f1b0c0c67f0 msgr2=0x7f1b0c0c04b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:21.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.175+0000 7f1b13a00640 1 -- 172.21.15.28:0/2011718595 shutdown_connections 2024-02-21T23:11:21.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.175+0000 7f1b13a00640 1 -- 172.21.15.28:0/2011718595 wait complete. 2024-02-21T23:11:21.186 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-21T23:02:30.728329+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 27, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-21T23:10:51.223834+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '173', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 28, 'pool_name': 'cephfs_data', 'create_time': '2024-02-21T23:10:52.368371+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '174', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-21T23:11:21.186 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-21T23:11:21.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.400+0000 7f9f46e00640 1 Processor -- start 2024-02-21T23:11:21.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.401+0000 7f9f46e00640 1 -- start start 2024-02-21T23:11:21.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.402+0000 7f9f46e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f9f400c6fd0 0x7f9f400c5a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:21.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.402+0000 7f9f46e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9f40072da0 0x7f9f401584d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:21.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.402+0000 7f9f46e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f400c2dd0 con 0x7f9f40073230 2024-02-21T23:11:21.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.402+0000 7f9f46e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f400c6070 con 0x7f9f40072da0 2024-02-21T23:11:21.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.402+0000 7f9f46e00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f400c6350 con 0x7f9f400c6fd0 2024-02-21T23:11:21.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.402+0000 7f9f46400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f9f40073230 0x7f9f401520f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37868/0 (socket says 172.21.15.28:37868) 2024-02-21T23:11:21.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.402+0000 7f9f46400640 1 -- 172.21.15.28:0/791427174 learned_addr learned my addr 172.21.15.28:0/791427174 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:11:21.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.403+0000 7f9f45000640 1 --2- 172.21.15.28:0/791427174 >> v2:172.21.15.28:3300/0 conn(0x7f9f40072da0 0x7f9f401584d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:21.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.403+0000 7f9f45a00640 1 --2- 172.21.15.28:0/791427174 >> v2:172.21.15.81:3300/0 conn(0x7f9f400c6fd0 0x7f9f400c5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:21.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.403+0000 7f9f3fe00640 1 -- 172.21.15.28:0/791427174 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3084027146 0 0) 0x7f9f400c2dd0 con 0x7f9f40073230 2024-02-21T23:11:21.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.403+0000 7f9f3fe00640 1 -- 172.21.15.28:0/791427174 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f28003520 con 0x7f9f40073230 2024-02-21T23:11:21.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.403+0000 7f9f45000640 1 -- 172.21.15.28:0/791427174 >> v2:172.21.15.81:3300/0 conn(0x7f9f400c6fd0 msgr2=0x7f9f400c5a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:21.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.403+0000 7f9f45000640 1 --2- 172.21.15.28:0/791427174 >> v2:172.21.15.81:3300/0 conn(0x7f9f400c6fd0 0x7f9f400c5a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:21.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.403+0000 7f9f45000640 1 -- 172.21.15.28:0/791427174 >> v1:172.21.15.28:6789/0 conn(0x7f9f40073230 legacy=0x7f9f401520f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:21.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.403+0000 7f9f45000640 1 -- 172.21.15.28:0/791427174 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f400725a0 con 0x7f9f40072da0 2024-02-21T23:11:21.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.405+0000 7f9f45000640 1 --2- 172.21.15.28:0/791427174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f40072da0 0x7f9f401584d0 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7f9f400c8800 tx=0x7f9f2c02b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=716988a0496d5c79 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:21.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.405+0000 7f9f3fe00640 1 -- 172.21.15.28:0/791427174 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f2c038070 con 0x7f9f40072da0 2024-02-21T23:11:21.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.405+0000 7f9f3fe00640 1 -- 172.21.15.28:0/791427174 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f2c031520 con 0x7f9f40072da0 2024-02-21T23:11:21.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.405+0000 7f9f3fe00640 1 -- 172.21.15.28:0/791427174 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f2c0318f0 con 0x7f9f40072da0 2024-02-21T23:11:21.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.406+0000 7f9f46e00640 1 -- 172.21.15.28:0/791427174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f40072da0 msgr2=0x7f9f401584d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:21.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.406+0000 7f9f46e00640 1 --2- 172.21.15.28:0/791427174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f40072da0 0x7f9f401584d0 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7f9f400c8800 tx=0x7f9f2c02b7c0 comp rx=0 tx=0).stop 2024-02-21T23:11:21.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.406+0000 7f9f46e00640 1 -- 172.21.15.28:0/791427174 shutdown_connections 2024-02-21T23:11:21.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.406+0000 7f9f46e00640 1 --2- 172.21.15.28:0/791427174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f40072da0 0x7f9f401584d0 unknown :-1 s=CLOSED pgs=1992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:21.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.406+0000 7f9f46e00640 1 --2- 172.21.15.28:0/791427174 >> v2:172.21.15.81:3300/0 conn(0x7f9f400c6fd0 0x7f9f400c5a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:21.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.406+0000 7f9f46e00640 1 -- 172.21.15.28:0/791427174 >> 172.21.15.28:0/791427174 conn(0x7f9f400a9690 msgr2=0x7f9f400a9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:21.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.406+0000 7f9f46e00640 1 -- 172.21.15.28:0/791427174 shutdown_connections 2024-02-21T23:11:21.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.406+0000 7f9f46e00640 1 -- 172.21.15.28:0/791427174 wait complete. 2024-02-21T23:11:21.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.409+0000 7f9f46e00640 1 Processor -- start 2024-02-21T23:11:21.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.409+0000 7f9f46e00640 1 -- start start 2024-02-21T23:11:21.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.411+0000 7f9f46e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f40072da0 0x7f9f40148ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:21.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.411+0000 7f9f46e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9f40073230 0x7f9f40149430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:21.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.411+0000 7f9f46e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f400c6fd0 0x7f9f401edd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:21.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.411+0000 7f9f46e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f400c7530 con 0x7f9f400c6fd0 2024-02-21T23:11:21.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.411+0000 7f9f46e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f400c73b0 con 0x7f9f40072da0 2024-02-21T23:11:21.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.411+0000 7f9f46e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f40152800 con 0x7f9f40073230 2024-02-21T23:11:21.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.411+0000 7f9f46400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f400c6fd0 0x7f9f401edd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:21.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.412+0000 7f9f46400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f400c6fd0 0x7f9f401edd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58932/0 (socket says 172.21.15.28:58932) 2024-02-21T23:11:21.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.412+0000 7f9f46400640 1 -- 172.21.15.28:0/2680320311 learned_addr learned my addr 172.21.15.28:0/2680320311 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:21.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.412+0000 7f9f45a00640 1 --2- 172.21.15.28:0/2680320311 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f40072da0 0x7f9f40148ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:21.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.412+0000 7f9f45000640 1 --2- 172.21.15.28:0/2680320311 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9f40073230 0x7f9f40149430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:21.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.412+0000 7f9f46400640 1 -- 172.21.15.28:0/2680320311 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9f40073230 msgr2=0x7f9f40149430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:21.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.412+0000 7f9f46400640 1 --2- 172.21.15.28:0/2680320311 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9f40073230 0x7f9f40149430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:21.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.412+0000 7f9f46400640 1 -- 172.21.15.28:0/2680320311 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f40072da0 msgr2=0x7f9f40148ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:21.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.412+0000 7f9f46400640 1 --2- 172.21.15.28:0/2680320311 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f40072da0 0x7f9f40148ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:21.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.412+0000 7f9f46400640 1 -- 172.21.15.28:0/2680320311 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f400abb20 con 0x7f9f400c6fd0 2024-02-21T23:11:21.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.412+0000 7f9f46400640 1 --2- 172.21.15.28:0/2680320311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f400c6fd0 0x7f9f401edd80 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7f9f3800cb60 tx=0x7f9f3800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:21.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.413+0000 7f9f3d600640 1 -- 172.21.15.28:0/2680320311 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f38017070 con 0x7f9f400c6fd0 2024-02-21T23:11:21.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.413+0000 7f9f3d600640 1 -- 172.21.15.28:0/2680320311 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f3800ad40 con 0x7f9f400c6fd0 2024-02-21T23:11:21.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.413+0000 7f9f3d600640 1 -- 172.21.15.28:0/2680320311 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f38012600 con 0x7f9f400c6fd0 2024-02-21T23:11:21.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.413+0000 7f9f46e00640 1 -- 172.21.15.28:0/2680320311 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f401589c0 con 0x7f9f400c6fd0 2024-02-21T23:11:21.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.413+0000 7f9f46e00640 1 -- 172.21.15.28:0/2680320311 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f400ab4d0 con 0x7f9f400c6fd0 2024-02-21T23:11:21.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.414+0000 7f9f46e00640 1 -- 172.21.15.28:0/2680320311 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f4004f290 con 0x7f9f400c6fd0 2024-02-21T23:11:21.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.415+0000 7f9f3d600640 1 -- 172.21.15.28:0/2680320311 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9f3801a070 con 0x7f9f400c6fd0 2024-02-21T23:11:21.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.416+0000 7f9f3d600640 1 --2- 172.21.15.28:0/2680320311 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9f2006b4d0 0x7f9f2006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:21.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.416+0000 7f9f3d600640 1 -- 172.21.15.28:0/2680320311 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f9f38089fe0 con 0x7f9f400c6fd0 2024-02-21T23:11:21.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.416+0000 7f9f45a00640 1 --2- 172.21.15.28:0/2680320311 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9f2006b4d0 0x7f9f2006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:21.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.418+0000 7f9f45a00640 1 --2- 172.21.15.28:0/2680320311 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9f2006b4d0 0x7f9f2006d990 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f9f34009a30 tx=0x7f9f34008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:21.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.420+0000 7f9f3d600640 1 -- 172.21.15.28:0/2680320311 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f9f38058670 con 0x7f9f400c6fd0 2024-02-21T23:11:21.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.744+0000 7f9f46e00640 1 -- 172.21.15.28:0/2680320311 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9f400c57e0 con 0x7f9f400c6fd0 2024-02-21T23:11:21.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.745+0000 7f9f3d600640 1 -- 172.21.15.28:0/2680320311 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v174) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9f3805c710 con 0x7f9f400c6fd0 2024-02-21T23:11:21.734 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-02-21T23:11:21.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.747+0000 7f9f46e00640 1 -- 172.21.15.28:0/2680320311 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9f2006b4d0 msgr2=0x7f9f2006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:21.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.747+0000 7f9f46e00640 1 --2- 172.21.15.28:0/2680320311 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9f2006b4d0 0x7f9f2006d990 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f9f34009a30 tx=0x7f9f34008040 comp rx=0 tx=0).stop 2024-02-21T23:11:21.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.747+0000 7f9f46e00640 1 -- 172.21.15.28:0/2680320311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f400c6fd0 msgr2=0x7f9f401edd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:21.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.747+0000 7f9f46e00640 1 --2- 172.21.15.28:0/2680320311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f400c6fd0 0x7f9f401edd80 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7f9f3800cb60 tx=0x7f9f3800a430 comp rx=0 tx=0).stop 2024-02-21T23:11:21.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.747+0000 7f9f46e00640 1 -- 172.21.15.28:0/2680320311 shutdown_connections 2024-02-21T23:11:21.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.747+0000 7f9f46e00640 1 --2- 172.21.15.28:0/2680320311 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9f2006b4d0 0x7f9f2006d990 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:21.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.747+0000 7f9f46e00640 1 --2- 172.21.15.28:0/2680320311 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9f40073230 0x7f9f40149430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:21.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.747+0000 7f9f46e00640 1 --2- 172.21.15.28:0/2680320311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f400c6fd0 0x7f9f401edd80 unknown :-1 s=CLOSED pgs=1993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:21.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.747+0000 7f9f46e00640 1 --2- 172.21.15.28:0/2680320311 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f40072da0 0x7f9f40148ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:21.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.747+0000 7f9f46e00640 1 -- 172.21.15.28:0/2680320311 >> 172.21.15.28:0/2680320311 conn(0x7f9f400a9690 msgr2=0x7f9f40074360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:21.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.747+0000 7f9f46e00640 1 -- 172.21.15.28:0/2680320311 shutdown_connections 2024-02-21T23:11:21.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:21.747+0000 7f9f46e00640 1 -- 172.21.15.28:0/2680320311 wait complete. 2024-02-21T23:11:21.754 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-21T23:11:21.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.003+0000 7f659be00640 1 Processor -- start 2024-02-21T23:11:21.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.003+0000 7f659be00640 1 -- start start 2024-02-21T23:11:21.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.005+0000 7f659be00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f659c0c2c10 0x7f659c0c12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:21.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.005+0000 7f659be00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f659c06e220 con 0x7f659c0c4580 2024-02-21T23:11:21.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.005+0000 7f659be00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f659c06f780 con 0x7f659c150240 2024-02-21T23:11:21.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.005+0000 7f659be00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f659c150620 con 0x7f659c0c2c10 2024-02-21T23:11:21.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.005+0000 7f659a000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f659c0c4580 0x7f659c06ae70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37882/0 (socket says 172.21.15.28:37882) 2024-02-21T23:11:21.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.005+0000 7f659a000640 1 -- 172.21.15.28:0/1211602159 learned_addr learned my addr 172.21.15.28:0/1211602159 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:11:21.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.006+0000 7f659b400640 1 --2- 172.21.15.28:0/1211602159 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f659c0c2c10 0x7f659c0c12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:21.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.006+0000 7f6599600640 1 -- 172.21.15.28:0/1211602159 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4178353042 0 0) 0x7f659c06e220 con 0x7f659c0c4580 2024-02-21T23:11:21.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.006+0000 7f6599600640 1 -- 172.21.15.28:0/1211602159 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6580003520 con 0x7f659c0c4580 2024-02-21T23:11:21.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.006+0000 7f6599600640 1 -- 172.21.15.28:0/1211602159 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 487069180 0 0) 0x7f6580003520 con 0x7f659c0c4580 2024-02-21T23:11:21.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.006+0000 7f6599600640 1 -- 172.21.15.28:0/1211602159 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f659c0c2c10 msgr2=0x7f659c0c12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:21.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.006+0000 7f6599600640 1 --2- 172.21.15.28:0/1211602159 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f659c0c2c10 0x7f659c0c12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:21.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.006+0000 7f6599600640 1 -- 172.21.15.28:0/1211602159 >> v1:172.21.15.81:6789/0 conn(0x7f659c150240 legacy=0x7f659c06db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:21.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.006+0000 7f6599600640 1 -- 172.21.15.28:0/1211602159 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f659c06f4a0 con 0x7f659c0c4580 2024-02-21T23:11:21.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.006+0000 7f6599600640 1 -- 172.21.15.28:0/1211602159 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f6588001670 con 0x7f659c0c4580 2024-02-21T23:11:21.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.006+0000 7f6599600640 1 -- 172.21.15.28:0/1211602159 >> v1:172.21.15.28:6789/0 conn(0x7f659c0c4580 legacy=0x7f659c06ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:21.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.006+0000 7f6599600640 1 --2- 172.21.15.28:0/1211602159 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f659c0c2c10 0x7f65800039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:21.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.006+0000 7f6599600640 1 --2- 172.21.15.28:0/1211602159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6580003fb0 0x7f6580004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:21.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.007+0000 7f6599600640 1 --2- 172.21.15.28:0/1211602159 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6580004d40 0x7f65800051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:21.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.007+0000 7f6599600640 1 -- 172.21.15.28:0/1211602159 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f658000bb80 con 0x7f6580003fb0 2024-02-21T23:11:21.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.007+0000 7f6599600640 1 -- 172.21.15.28:0/1211602159 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f658000bd30 con 0x7f6580004d40 2024-02-21T23:11:21.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.007+0000 7f6599600640 1 -- 172.21.15.28:0/1211602159 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f658000bf30 con 0x7f659c0c2c10 2024-02-21T23:11:21.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.007+0000 7f659a000640 1 --2- 172.21.15.28:0/1211602159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6580003fb0 0x7f6580004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:21.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.007+0000 7f659aa00640 1 --2- 172.21.15.28:0/1211602159 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f659c0c2c10 0x7f65800039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:21.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.007+0000 7f659b400640 1 --2- 172.21.15.28:0/1211602159 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6580004d40 0x7f65800051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:21.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.007+0000 7f659a000640 1 -- 172.21.15.28:0/1211602159 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f659c0c2c10 msgr2=0x7f65800039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:21.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.007+0000 7f659a000640 1 --2- 172.21.15.28:0/1211602159 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f659c0c2c10 0x7f65800039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:21.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.007+0000 7f659a000640 1 -- 172.21.15.28:0/1211602159 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6580004d40 msgr2=0x7f65800051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:21.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.007+0000 7f659a000640 1 --2- 172.21.15.28:0/1211602159 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6580004d40 0x7f65800051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:21.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.007+0000 7f659a000640 1 -- 172.21.15.28:0/1211602159 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6580003520 con 0x7f6580003fb0 2024-02-21T23:11:21.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.007+0000 7f659b400640 1 --2- 172.21.15.28:0/1211602159 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6580004d40 0x7f65800051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:11:22.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.009+0000 7f659a000640 1 --2- 172.21.15.28:0/1211602159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6580003fb0 0x7f6580004800 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7f658800c0e0 tx=0x7f6580005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36184f2d138ba4cb server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:22.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.009+0000 7f6599600640 1 -- 172.21.15.28:0/1211602159 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6588007c40 con 0x7f6580003fb0 2024-02-21T23:11:22.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.010+0000 7f6599600640 1 -- 172.21.15.28:0/1211602159 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6588030070 con 0x7f6580003fb0 2024-02-21T23:11:22.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.010+0000 7f6599600640 1 -- 172.21.15.28:0/1211602159 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6588005d70 con 0x7f6580003fb0 2024-02-21T23:11:22.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.010+0000 7f659be00640 1 -- 172.21.15.28:0/1211602159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6580003fb0 msgr2=0x7f6580004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:22.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.010+0000 7f659be00640 1 --2- 172.21.15.28:0/1211602159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6580003fb0 0x7f6580004800 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7f658800c0e0 tx=0x7f6580005830 comp rx=0 tx=0).stop 2024-02-21T23:11:22.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.010+0000 7f659b400640 1 -- 172.21.15.28:0/1211602159 reap_dead start 2024-02-21T23:11:22.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.010+0000 7f659be00640 1 -- 172.21.15.28:0/1211602159 shutdown_connections 2024-02-21T23:11:22.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.010+0000 7f659be00640 1 -- 172.21.15.28:0/1211602159 >> 172.21.15.28:0/1211602159 conn(0x7f659c0a8530 msgr2=0x7f659c0a8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:22.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.010+0000 7f659be00640 1 -- 172.21.15.28:0/1211602159 shutdown_connections 2024-02-21T23:11:22.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.010+0000 7f659be00640 1 -- 172.21.15.28:0/1211602159 wait complete. 2024-02-21T23:11:22.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.013+0000 7f659be00640 1 Processor -- start 2024-02-21T23:11:22.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.013+0000 7f659be00640 1 -- start start 2024-02-21T23:11:22.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.015+0000 7f659be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6580003fb0 0x7f659c0bfe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:22.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.015+0000 7f659be00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f659c0ab5c0 0x7f659c072bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:22.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.015+0000 7f659be00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f659c1ee100 0x7f659c1ee590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:22.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.015+0000 7f659be00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f659c0c4a10 con 0x7f6580003fb0 2024-02-21T23:11:22.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.015+0000 7f659be00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f659c153a90 con 0x7f659c0ab5c0 2024-02-21T23:11:22.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.015+0000 7f659be00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f659c0c4b90 con 0x7f659c1ee100 2024-02-21T23:11:22.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.015+0000 7f659aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6580003fb0 0x7f659c0bfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:22.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.015+0000 7f659aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6580003fb0 0x7f659c0bfe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58960/0 (socket says 172.21.15.28:58960) 2024-02-21T23:11:22.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.015+0000 7f659aa00640 1 -- 172.21.15.28:0/884192752 learned_addr learned my addr 172.21.15.28:0/884192752 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:22.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.015+0000 7f659a000640 1 --2- 172.21.15.28:0/884192752 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f659c0ab5c0 0x7f659c072bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:22.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.016+0000 7f659b400640 1 --2- 172.21.15.28:0/884192752 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f659c1ee100 0x7f659c1ee590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:22.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.016+0000 7f659aa00640 1 -- 172.21.15.28:0/884192752 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f659c1ee100 msgr2=0x7f659c1ee590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:22.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.016+0000 7f659aa00640 1 --2- 172.21.15.28:0/884192752 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f659c1ee100 0x7f659c1ee590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:22.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.016+0000 7f659aa00640 1 -- 172.21.15.28:0/884192752 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f659c0ab5c0 msgr2=0x7f659c072bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:22.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.016+0000 7f659aa00640 1 --2- 172.21.15.28:0/884192752 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f659c0ab5c0 0x7f659c072bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:22.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.016+0000 7f659aa00640 1 -- 172.21.15.28:0/884192752 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f659c0abc10 con 0x7f6580003fb0 2024-02-21T23:11:22.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.016+0000 7f659b400640 1 --2- 172.21.15.28:0/884192752 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f659c1ee100 0x7f659c1ee590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:11:22.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.016+0000 7f659a000640 1 --2- 172.21.15.28:0/884192752 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f659c0ab5c0 0x7f659c072bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:11:22.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.016+0000 7f659aa00640 1 --2- 172.21.15.28:0/884192752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6580003fb0 0x7f659c0bfe20 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7f658c00e550 tx=0x7f658c00ea10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:22.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.016+0000 7f6592a00640 1 -- 172.21.15.28:0/884192752 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f658c0052e0 con 0x7f6580003fb0 2024-02-21T23:11:22.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.016+0000 7f6592a00640 1 -- 172.21.15.28:0/884192752 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f658c018070 con 0x7f6580003fb0 2024-02-21T23:11:22.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.016+0000 7f6592a00640 1 -- 172.21.15.28:0/884192752 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f658c003810 con 0x7f6580003fb0 2024-02-21T23:11:22.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.016+0000 7f659be00640 1 -- 172.21.15.28:0/884192752 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f659c1eead0 con 0x7f6580003fb0 2024-02-21T23:11:22.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.016+0000 7f659be00640 1 -- 172.21.15.28:0/884192752 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f659c0ac190 con 0x7f6580003fb0 2024-02-21T23:11:22.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.018+0000 7f659be00640 1 -- 172.21.15.28:0/884192752 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f659c04f290 con 0x7f6580003fb0 2024-02-21T23:11:22.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.019+0000 7f6592a00640 1 -- 172.21.15.28:0/884192752 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f658c0039b0 con 0x7f6580003fb0 2024-02-21T23:11:22.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.020+0000 7f6592a00640 1 --2- 172.21.15.28:0/884192752 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f657c06b4d0 0x7f657c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:22.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.020+0000 7f6592a00640 1 -- 172.21.15.28:0/884192752 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f658c08a0f0 con 0x7f6580003fb0 2024-02-21T23:11:22.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.021+0000 7f659a000640 1 --2- 172.21.15.28:0/884192752 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f657c06b4d0 0x7f657c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:22.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.022+0000 7f659a000640 1 --2- 172.21.15.28:0/884192752 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f657c06b4d0 0x7f657c06d990 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f6588034000 tx=0x7f6588009030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:22.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.024+0000 7f6592a00640 1 -- 172.21.15.28:0/884192752 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f658c058d40 con 0x7f6580003fb0 2024-02-21T23:11:22.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.224+0000 7f659be00640 1 -- 172.21.15.28:0/884192752 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f659c1edd20 con 0x7f6580003fb0 2024-02-21T23:11:22.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.225+0000 7f6592a00640 1 -- 172.21.15.28:0/884192752 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v174) v1 ==== 104+0+11 (secure 0 0 0) 0x7f658c058640 con 0x7f6580003fb0 2024-02-21T23:11:22.214 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-02-21T23:11:22.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.228+0000 7f659be00640 1 -- 172.21.15.28:0/884192752 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f657c06b4d0 msgr2=0x7f657c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:22.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.228+0000 7f659be00640 1 --2- 172.21.15.28:0/884192752 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f657c06b4d0 0x7f657c06d990 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f6588034000 tx=0x7f6588009030 comp rx=0 tx=0).stop 2024-02-21T23:11:22.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.228+0000 7f659be00640 1 -- 172.21.15.28:0/884192752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6580003fb0 msgr2=0x7f659c0bfe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:22.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.228+0000 7f659be00640 1 --2- 172.21.15.28:0/884192752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6580003fb0 0x7f659c0bfe20 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7f658c00e550 tx=0x7f658c00ea10 comp rx=0 tx=0).stop 2024-02-21T23:11:22.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.228+0000 7f659be00640 1 -- 172.21.15.28:0/884192752 shutdown_connections 2024-02-21T23:11:22.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.228+0000 7f659be00640 1 --2- 172.21.15.28:0/884192752 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f657c06b4d0 0x7f657c06d990 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:22.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.228+0000 7f659be00640 1 --2- 172.21.15.28:0/884192752 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f659c1ee100 0x7f659c1ee590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:22.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.228+0000 7f659be00640 1 --2- 172.21.15.28:0/884192752 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f659c0ab5c0 0x7f659c072bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:22.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.228+0000 7f659be00640 1 --2- 172.21.15.28:0/884192752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6580003fb0 0x7f659c0bfe20 unknown :-1 s=CLOSED pgs=1996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:22.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.228+0000 7f659be00640 1 -- 172.21.15.28:0/884192752 >> 172.21.15.28:0/884192752 conn(0x7f659c0a8530 msgr2=0x7f659c14c5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:22.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.228+0000 7f659be00640 1 -- 172.21.15.28:0/884192752 shutdown_connections 2024-02-21T23:11:22.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.229+0000 7f659be00640 1 -- 172.21.15.28:0/884192752 wait complete. 2024-02-21T23:11:22.236 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-21T23:11:22.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.506+0000 7fc87ae00640 1 Processor -- start 2024-02-21T23:11:22.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.506+0000 7fc87ae00640 1 -- start start 2024-02-21T23:11:22.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.508+0000 7fc87ae00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fc8740c5cc0 0x7fc8740c6c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:22.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.508+0000 7fc87ae00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc8740c3030 0x7fc8740c3460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:22.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.508+0000 7fc87ae00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc874159580 con 0x7fc8740c7590 2024-02-21T23:11:22.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.508+0000 7fc87ae00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8740c6230 con 0x7fc8740c5cc0 2024-02-21T23:11:22.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.508+0000 7fc87ae00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8741587c0 con 0x7fc8740c3030 2024-02-21T23:11:22.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.508+0000 7fc878e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc8740c3030 0x7fc8740c3460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:22.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.508+0000 7fc878e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc8740c3030 0x7fc8740c3460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3301/0 says I am v2:172.21.15.28:36154/0 (socket says 172.21.15.28:36154) 2024-02-21T23:11:22.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.508+0000 7fc878e00640 1 -- 172.21.15.28:0/2343310969 learned_addr learned my addr 172.21.15.28:0/2343310969 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:22.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.509+0000 7fc86be00640 1 --2- 172.21.15.28:0/2343310969 >> v2:172.21.15.81:3300/0 conn(0x7fc8740c5cc0 0x7fc8740c6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:22.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.509+0000 7fc878e00640 1 -- 172.21.15.28:0/2343310969 >> v2:172.21.15.81:3300/0 conn(0x7fc8740c5cc0 msgr2=0x7fc8740c6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:22.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.509+0000 7fc878e00640 1 --2- 172.21.15.28:0/2343310969 >> v2:172.21.15.81:3300/0 conn(0x7fc8740c5cc0 0x7fc8740c6c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:22.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.509+0000 7fc878e00640 1 -- 172.21.15.28:0/2343310969 >> v1:172.21.15.81:6789/0 conn(0x7fc8740c7590 legacy=0x7fc8741524e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:22.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.509+0000 7fc878e00640 1 -- 172.21.15.28:0/2343310969 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc874158aa0 con 0x7fc8740c3030 2024-02-21T23:11:22.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.510+0000 7fc878e00640 1 --2- 172.21.15.28:0/2343310969 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc8740c3030 0x7fc8740c3460 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fc86c009e50 tx=0x7fc86c02bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f536558eaf80b600 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:22.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.511+0000 7fc873400640 1 -- 172.21.15.28:0/2343310969 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc86c02ca70 con 0x7fc8740c3030 2024-02-21T23:11:22.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.511+0000 7fc873400640 1 -- 172.21.15.28:0/2343310969 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc86c02cc10 con 0x7fc8740c3030 2024-02-21T23:11:22.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.511+0000 7fc873400640 1 -- 172.21.15.28:0/2343310969 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc86c0343f0 con 0x7fc8740c3030 2024-02-21T23:11:22.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.512+0000 7fc87ae00640 1 -- 172.21.15.28:0/2343310969 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc8740c3030 msgr2=0x7fc8740c3460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:22.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.512+0000 7fc87ae00640 1 --2- 172.21.15.28:0/2343310969 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc8740c3030 0x7fc8740c3460 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fc86c009e50 tx=0x7fc86c02bed0 comp rx=0 tx=0).stop 2024-02-21T23:11:22.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.512+0000 7fc87ae00640 1 -- 172.21.15.28:0/2343310969 shutdown_connections 2024-02-21T23:11:22.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.512+0000 7fc87ae00640 1 --2- 172.21.15.28:0/2343310969 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc8740c3030 0x7fc8740c3460 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:22.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.512+0000 7fc87ae00640 1 --2- 172.21.15.28:0/2343310969 >> v2:172.21.15.81:3300/0 conn(0x7fc8740c5cc0 0x7fc8740c6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:22.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.512+0000 7fc87ae00640 1 -- 172.21.15.28:0/2343310969 >> 172.21.15.28:0/2343310969 conn(0x7fc87408afe0 msgr2=0x7fc8740725a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:22.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.512+0000 7fc87ae00640 1 -- 172.21.15.28:0/2343310969 shutdown_connections 2024-02-21T23:11:22.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.512+0000 7fc87ae00640 1 -- 172.21.15.28:0/2343310969 wait complete. 2024-02-21T23:11:22.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.515+0000 7fc87ae00640 1 Processor -- start 2024-02-21T23:11:22.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.515+0000 7fc87ae00640 1 -- start start 2024-02-21T23:11:22.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.517+0000 7fc87ae00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc8740c3030 0x7fc874156d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:22.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.517+0000 7fc87ae00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8740c5cc0 0x7fc874144600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:22.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.517+0000 7fc87ae00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc8740c7590 0x7fc87414cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:22.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.517+0000 7fc87ae00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8740c6720 con 0x7fc8740c5cc0 2024-02-21T23:11:22.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.517+0000 7fc87ae00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc874159060 con 0x7fc8740c7590 2024-02-21T23:11:22.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.517+0000 7fc87ae00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8740c68a0 con 0x7fc8740c3030 2024-02-21T23:11:22.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.518+0000 7fc86be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8740c5cc0 0x7fc874144600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:22.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.518+0000 7fc86be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8740c5cc0 0x7fc874144600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58966/0 (socket says 172.21.15.28:58966) 2024-02-21T23:11:22.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.518+0000 7fc86be00640 1 -- 172.21.15.28:0/2199218546 learned_addr learned my addr 172.21.15.28:0/2199218546 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:22.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.518+0000 7fc873e00640 1 --2- 172.21.15.28:0/2199218546 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc8740c3030 0x7fc874156d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:22.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.518+0000 7fc878e00640 1 --2- 172.21.15.28:0/2199218546 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc8740c7590 0x7fc87414cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:22.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.518+0000 7fc86be00640 1 -- 172.21.15.28:0/2199218546 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc8740c3030 msgr2=0x7fc874156d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:22.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.518+0000 7fc86be00640 1 --2- 172.21.15.28:0/2199218546 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc8740c3030 0x7fc874156d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:22.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.518+0000 7fc86be00640 1 -- 172.21.15.28:0/2199218546 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc8740c7590 msgr2=0x7fc87414cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:22.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.518+0000 7fc86be00640 1 --2- 172.21.15.28:0/2199218546 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc8740c7590 0x7fc87414cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:22.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.518+0000 7fc86be00640 1 -- 172.21.15.28:0/2199218546 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8741ef0b0 con 0x7fc8740c5cc0 2024-02-21T23:11:22.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.519+0000 7fc86be00640 1 --2- 172.21.15.28:0/2199218546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8740c5cc0 0x7fc874144600 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7fc85c00e9e0 tx=0x7fc85c00eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:22.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.519+0000 7fc870c00640 1 -- 172.21.15.28:0/2199218546 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc85c017070 con 0x7fc8740c5cc0 2024-02-21T23:11:22.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.519+0000 7fc870c00640 1 -- 172.21.15.28:0/2199218546 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc85c00c9d0 con 0x7fc8740c5cc0 2024-02-21T23:11:22.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.519+0000 7fc870c00640 1 -- 172.21.15.28:0/2199218546 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc85c00cdc0 con 0x7fc8740c5cc0 2024-02-21T23:11:22.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.519+0000 7fc87ae00640 1 -- 172.21.15.28:0/2199218546 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc874144b40 con 0x7fc8740c5cc0 2024-02-21T23:11:22.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.519+0000 7fc87ae00640 1 -- 172.21.15.28:0/2199218546 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8741eeb30 con 0x7fc8740c5cc0 2024-02-21T23:11:22.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.520+0000 7fc87ae00640 1 -- 172.21.15.28:0/2199218546 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc874146590 con 0x7fc8740c5cc0 2024-02-21T23:11:22.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.521+0000 7fc870c00640 1 -- 172.21.15.28:0/2199218546 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc85c01a070 con 0x7fc8740c5cc0 2024-02-21T23:11:22.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.522+0000 7fc870c00640 1 --2- 172.21.15.28:0/2199218546 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc85006b4d0 0x7fc85006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:22.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.522+0000 7fc870c00640 1 -- 172.21.15.28:0/2199218546 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc85c08a3e0 con 0x7fc8740c5cc0 2024-02-21T23:11:22.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.523+0000 7fc873e00640 1 --2- 172.21.15.28:0/2199218546 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc85006b4d0 0x7fc85006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:22.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.524+0000 7fc873e00640 1 --2- 172.21.15.28:0/2199218546 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc85006b4d0 0x7fc85006d990 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fc864007fd0 tx=0x7fc864009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:22.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.527+0000 7fc870c00640 1 -- 172.21.15.28:0/2199218546 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc874146590 con 0x7fc8740c5cc0 2024-02-21T23:11:22.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.758+0000 7fc87ae00640 1 -- 172.21.15.28:0/2199218546 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc874067f30 con 0x7fc8740c5cc0 2024-02-21T23:11:22.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.759+0000 7fc870c00640 1 -- 172.21.15.28:0/2199218546 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v174) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc874144b40 con 0x7fc8740c5cc0 2024-02-21T23:11:22.747 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-02-21T23:11:22.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.762+0000 7fc87ae00640 1 -- 172.21.15.28:0/2199218546 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc85006b4d0 msgr2=0x7fc85006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:22.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.762+0000 7fc87ae00640 1 --2- 172.21.15.28:0/2199218546 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc85006b4d0 0x7fc85006d990 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fc864007fd0 tx=0x7fc864009040 comp rx=0 tx=0).stop 2024-02-21T23:11:22.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.762+0000 7fc87ae00640 1 -- 172.21.15.28:0/2199218546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8740c5cc0 msgr2=0x7fc874144600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:22.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.762+0000 7fc87ae00640 1 --2- 172.21.15.28:0/2199218546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8740c5cc0 0x7fc874144600 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7fc85c00e9e0 tx=0x7fc85c00eea0 comp rx=0 tx=0).stop 2024-02-21T23:11:22.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.762+0000 7fc87ae00640 1 -- 172.21.15.28:0/2199218546 shutdown_connections 2024-02-21T23:11:22.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.762+0000 7fc87ae00640 1 --2- 172.21.15.28:0/2199218546 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc85006b4d0 0x7fc85006d990 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:22.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.762+0000 7fc87ae00640 1 --2- 172.21.15.28:0/2199218546 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc8740c7590 0x7fc87414cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:22.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.762+0000 7fc87ae00640 1 --2- 172.21.15.28:0/2199218546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8740c5cc0 0x7fc874144600 unknown :-1 s=CLOSED pgs=1997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:22.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.762+0000 7fc87ae00640 1 --2- 172.21.15.28:0/2199218546 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc8740c3030 0x7fc874156d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:22.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.762+0000 7fc87ae00640 1 -- 172.21.15.28:0/2199218546 >> 172.21.15.28:0/2199218546 conn(0x7fc87408afe0 msgr2=0x7fc874150c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:22.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.762+0000 7fc87ae00640 1 -- 172.21.15.28:0/2199218546 shutdown_connections 2024-02-21T23:11:22.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.762+0000 7fc87ae00640 1 -- 172.21.15.28:0/2199218546 wait complete. 2024-02-21T23:11:22.768 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:11:22.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.984+0000 7fc93d000640 1 Processor -- start 2024-02-21T23:11:22.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.984+0000 7fc93d000640 1 -- start start 2024-02-21T23:11:22.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.986+0000 7fc93d000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fc9380aff90 0x7fc9380c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:22.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.986+0000 7fc93d000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc93806dbf0 0x7fc938158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:22.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.986+0000 7fc93d000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9380c74c0 con 0x7fc9380c5720 2024-02-21T23:11:22.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.986+0000 7fc93d000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9380c7270 con 0x7fc93806dbf0 2024-02-21T23:11:22.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.986+0000 7fc93d000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc938158630 con 0x7fc9380aff90 2024-02-21T23:11:22.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.986+0000 7fc936a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc93806dbf0 0x7fc938158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:22.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.986+0000 7fc936a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc93806dbf0 0x7fc938158ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58982/0 (socket says 172.21.15.28:58982) 2024-02-21T23:11:22.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.986+0000 7fc936a00640 1 -- 172.21.15.28:0/1016904977 learned_addr learned my addr 172.21.15.28:0/1016904977 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:22.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.987+0000 7fc937400640 1 --2- 172.21.15.28:0/1016904977 >> v2:172.21.15.81:3300/0 conn(0x7fc9380aff90 0x7fc9380c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:22.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.987+0000 7fc936a00640 1 -- 172.21.15.28:0/1016904977 >> v2:172.21.15.81:3300/0 conn(0x7fc9380aff90 msgr2=0x7fc9380c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:22.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.987+0000 7fc936a00640 1 --2- 172.21.15.28:0/1016904977 >> v2:172.21.15.81:3300/0 conn(0x7fc9380aff90 0x7fc9380c51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:22.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.987+0000 7fc936a00640 1 -- 172.21.15.28:0/1016904977 >> v1:172.21.15.81:6789/0 conn(0x7fc9380c5720 legacy=0x7fc93806c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:22.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.987+0000 7fc936a00640 1 -- 172.21.15.28:0/1016904977 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9380c77a0 con 0x7fc93806dbf0 2024-02-21T23:11:22.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.988+0000 7fc936a00640 1 --2- 172.21.15.28:0/1016904977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc93806dbf0 0x7fc938158ac0 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7fc9380c73f0 tx=0x7fc92402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa994a33c14a8991 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:22.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.988+0000 7fc936000640 1 -- 172.21.15.28:0/1016904977 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc924034070 con 0x7fc93806dbf0 2024-02-21T23:11:22.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.989+0000 7fc936000640 1 -- 172.21.15.28:0/1016904977 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc924030860 con 0x7fc93806dbf0 2024-02-21T23:11:22.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.989+0000 7fc936000640 1 -- 172.21.15.28:0/1016904977 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc924030c30 con 0x7fc93806dbf0 2024-02-21T23:11:22.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.989+0000 7fc93d000640 1 -- 172.21.15.28:0/1016904977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc93806dbf0 msgr2=0x7fc938158ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:22.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.989+0000 7fc93d000640 1 --2- 172.21.15.28:0/1016904977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc93806dbf0 0x7fc938158ac0 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7fc9380c73f0 tx=0x7fc92402b9a0 comp rx=0 tx=0).stop 2024-02-21T23:11:22.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.989+0000 7fc93d000640 1 -- 172.21.15.28:0/1016904977 shutdown_connections 2024-02-21T23:11:22.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.989+0000 7fc93d000640 1 --2- 172.21.15.28:0/1016904977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc93806dbf0 0x7fc938158ac0 unknown :-1 s=CLOSED pgs=1998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:22.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.989+0000 7fc93d000640 1 --2- 172.21.15.28:0/1016904977 >> v2:172.21.15.81:3300/0 conn(0x7fc9380aff90 0x7fc9380c51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:22.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.989+0000 7fc93d000640 1 -- 172.21.15.28:0/1016904977 >> 172.21.15.28:0/1016904977 conn(0x7fc9380ac370 msgr2=0x7fc9380ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:22.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.989+0000 7fc93d000640 1 -- 172.21.15.28:0/1016904977 shutdown_connections 2024-02-21T23:11:22.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.989+0000 7fc93d000640 1 -- 172.21.15.28:0/1016904977 wait complete. 2024-02-21T23:11:22.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.991+0000 7fc93d000640 1 Processor -- start 2024-02-21T23:11:22.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.991+0000 7fc93d000640 1 -- start start 2024-02-21T23:11:22.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.992+0000 7fc93d000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc93806dbf0 0x7fc93814c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:22.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.992+0000 7fc93d000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9380aff90 0x7fc9381512d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:22.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.992+0000 7fc93d000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc9380c5720 0x7fc9381ebd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:22.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.992+0000 7fc93d000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9380c8760 con 0x7fc9380aff90 2024-02-21T23:11:22.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.992+0000 7fc93d000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9380be0c0 con 0x7fc9380c5720 2024-02-21T23:11:22.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.992+0000 7fc93d000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9380c88e0 con 0x7fc93806dbf0 2024-02-21T23:11:22.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.992+0000 7fc936a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9380aff90 0x7fc9381512d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:22.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.992+0000 7fc936a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9380aff90 0x7fc9381512d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58992/0 (socket says 172.21.15.28:58992) 2024-02-21T23:11:22.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.992+0000 7fc936a00640 1 -- 172.21.15.28:0/4219695140 learned_addr learned my addr 172.21.15.28:0/4219695140 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:22.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.993+0000 7fc936a00640 1 -- 172.21.15.28:0/4219695140 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc93806dbf0 msgr2=0x7fc93814c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:22.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.993+0000 7fc936a00640 1 --2- 172.21.15.28:0/4219695140 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc93806dbf0 0x7fc93814c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:22.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.993+0000 7fc936a00640 1 -- 172.21.15.28:0/4219695140 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc9380c5720 msgr2=0x7fc9381ebd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:22.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.993+0000 7fc936a00640 1 --2- 172.21.15.28:0/4219695140 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc9380c5720 0x7fc9381ebd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:22.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.993+0000 7fc936a00640 1 -- 172.21.15.28:0/4219695140 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc938151e40 con 0x7fc9380aff90 2024-02-21T23:11:22.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.993+0000 7fc936a00640 1 --2- 172.21.15.28:0/4219695140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9380aff90 0x7fc9381512d0 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7fc92402b9e0 tx=0x7fc924004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:22.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.994+0000 7fc92f400640 1 -- 172.21.15.28:0/4219695140 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc924048070 con 0x7fc9380aff90 2024-02-21T23:11:22.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.994+0000 7fc92f400640 1 -- 172.21.15.28:0/4219695140 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc924030660 con 0x7fc9380aff90 2024-02-21T23:11:22.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.994+0000 7fc92f400640 1 -- 172.21.15.28:0/4219695140 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc924034040 con 0x7fc9380aff90 2024-02-21T23:11:22.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.994+0000 7fc93d000640 1 -- 172.21.15.28:0/4219695140 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc938154740 con 0x7fc9380aff90 2024-02-21T23:11:22.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.994+0000 7fc93d000640 1 -- 172.21.15.28:0/4219695140 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc938154a00 con 0x7fc9380aff90 2024-02-21T23:11:22.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.995+0000 7fc93d000640 1 -- 172.21.15.28:0/4219695140 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc93804f290 con 0x7fc9380aff90 2024-02-21T23:11:22.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.996+0000 7fc92f400640 1 -- 172.21.15.28:0/4219695140 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc924037070 con 0x7fc9380aff90 2024-02-21T23:11:22.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.997+0000 7fc92f400640 1 --2- 172.21.15.28:0/4219695140 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc91806b500 0x7fc91806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:22.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.997+0000 7fc92f400640 1 -- 172.21.15.28:0/4219695140 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc9240ac110 con 0x7fc9380aff90 2024-02-21T23:11:22.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.998+0000 7fc937400640 1 --2- 172.21.15.28:0/4219695140 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc91806b500 0x7fc91806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:22.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:22.999+0000 7fc937400640 1 --2- 172.21.15.28:0/4219695140 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc91806b500 0x7fc91806d9c0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fc9280098f0 tx=0x7fc928008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:22.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.007+0000 7fc92f400640 1 -- 172.21.15.28:0/4219695140 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc92407adf0 con 0x7fc9380aff90 2024-02-21T23:11:23.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.250+0000 7fc93d000640 1 -- 172.21.15.28:0/4219695140 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc93806d9d0 con 0x7fc9380aff90 2024-02-21T23:11:23.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.251+0000 7fc92f400640 1 -- 172.21.15.28:0/4219695140 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 192 v192) v1 ==== 95+0+4945 (secure 0 0 0) 0x7fc92407a6f0 con 0x7fc9380aff90 2024-02-21T23:11:23.239 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:11:23.240 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":192,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6686,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/155806034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":155806034},{"type":"v1","addr":"172.21.15.81:6836","nonce":155806034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":180},{"gid":7480,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3897243745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":3897243745},{"type":"v1","addr":"172.21.15.81:6837","nonce":3897243745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":180},{"gid":7486,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2827135632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2827135632},{"type":"v1","addr":"172.21.15.28:6835","nonce":2827135632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":180}],"filesystems":[{"mdsmap":{"epoch":192,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:10:53.351225+0000","modified":"2024-02-21T23:11:15.654038+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9699},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9699":{"gid":9699,"name":"c","rank":0,"incarnation":191,"state":"up:active","state_seq":2,"addr":"172.21.15.28:6833/2364631414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2364631414},{"type":"v1","addr":"172.21.15.28:6833","nonce":2364631414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2024-02-21T23:11:23.240 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 192 2024-02-21T23:11:23.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.254+0000 7fc93d000640 1 -- 172.21.15.28:0/4219695140 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc91806b500 msgr2=0x7fc91806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:23.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.254+0000 7fc93d000640 1 --2- 172.21.15.28:0/4219695140 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc91806b500 0x7fc91806d9c0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fc9280098f0 tx=0x7fc928008040 comp rx=0 tx=0).stop 2024-02-21T23:11:23.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.254+0000 7fc93d000640 1 -- 172.21.15.28:0/4219695140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9380aff90 msgr2=0x7fc9381512d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:23.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.254+0000 7fc93d000640 1 --2- 172.21.15.28:0/4219695140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9380aff90 0x7fc9381512d0 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7fc92402b9e0 tx=0x7fc924004730 comp rx=0 tx=0).stop 2024-02-21T23:11:23.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.254+0000 7fc93d000640 1 -- 172.21.15.28:0/4219695140 shutdown_connections 2024-02-21T23:11:23.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.254+0000 7fc93d000640 1 --2- 172.21.15.28:0/4219695140 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc91806b500 0x7fc91806d9c0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:23.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.254+0000 7fc93d000640 1 --2- 172.21.15.28:0/4219695140 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc9380c5720 0x7fc9381ebd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:23.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.254+0000 7fc93d000640 1 --2- 172.21.15.28:0/4219695140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9380aff90 0x7fc9381512d0 unknown :-1 s=CLOSED pgs=1999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:23.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.254+0000 7fc93d000640 1 --2- 172.21.15.28:0/4219695140 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc93806dbf0 0x7fc93814c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:23.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.254+0000 7fc93d000640 1 -- 172.21.15.28:0/4219695140 >> 172.21.15.28:0/4219695140 conn(0x7fc9380ac370 msgr2=0x7fc938157210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:23.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.254+0000 7fc93d000640 1 -- 172.21.15.28:0/4219695140 shutdown_connections 2024-02-21T23:11:23.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.255+0000 7fc93d000640 1 -- 172.21.15.28:0/4219695140 wait complete. 2024-02-21T23:11:23.261 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:11:23.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.507+0000 7f70a7200640 1 Processor -- start 2024-02-21T23:11:23.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.508+0000 7f70a7200640 1 -- start start 2024-02-21T23:11:23.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.510+0000 7f70a7200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f70a8150450 0x7f70a80a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:23.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.510+0000 7f70a7200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70a80c2760 con 0x7f70a8156af0 2024-02-21T23:11:23.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.510+0000 7f70a7200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70a8156310 con 0x7f70a806ee40 2024-02-21T23:11:23.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.510+0000 7f70a7200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70a80c22c0 con 0x7f70a8150450 2024-02-21T23:11:23.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.511+0000 7f70a6800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f70a8156af0 0x7f70a806dff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37892/0 (socket says 172.21.15.28:37892) 2024-02-21T23:11:23.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.511+0000 7f70a6800640 1 -- 172.21.15.28:0/2925953333 learned_addr learned my addr 172.21.15.28:0/2925953333 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:11:23.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.511+0000 7f70a5e00640 1 --2- 172.21.15.28:0/2925953333 >> v2:172.21.15.81:3300/0 conn(0x7f70a8150450 0x7f70a80a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:23.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.511+0000 7f70a4a00640 1 -- 172.21.15.28:0/2925953333 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2268845358 0 0) 0x7f70a80c2760 con 0x7f70a8156af0 2024-02-21T23:11:23.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.511+0000 7f70a4a00640 1 -- 172.21.15.28:0/2925953333 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f708c003520 con 0x7f70a8156af0 2024-02-21T23:11:23.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.512+0000 7f70a4a00640 1 -- 172.21.15.28:0/2925953333 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3868362086 0 0) 0x7f708c003520 con 0x7f70a8156af0 2024-02-21T23:11:23.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.512+0000 7f70a4a00640 1 -- 172.21.15.28:0/2925953333 >> v2:172.21.15.81:3300/0 conn(0x7f70a8150450 msgr2=0x7f70a80a5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:23.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.512+0000 7f70a4a00640 1 --2- 172.21.15.28:0/2925953333 >> v2:172.21.15.81:3300/0 conn(0x7f70a8150450 0x7f70a80a5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:23.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.512+0000 7f70a4a00640 1 -- 172.21.15.28:0/2925953333 >> v1:172.21.15.81:6789/0 conn(0x7f70a806ee40 legacy=0x7f70a81536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:23.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.512+0000 7f70a4a00640 1 -- 172.21.15.28:0/2925953333 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70a80c29d0 con 0x7f70a8156af0 2024-02-21T23:11:23.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.512+0000 7f70a4a00640 1 -- 172.21.15.28:0/2925953333 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f70a00019f0 con 0x7f70a8156af0 2024-02-21T23:11:23.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.512+0000 7f70a4a00640 1 -- 172.21.15.28:0/2925953333 >> v1:172.21.15.28:6789/0 conn(0x7f70a8156af0 legacy=0x7f70a806dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:23.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.512+0000 7f70a4a00640 1 --2- 172.21.15.28:0/2925953333 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f708c0039b0 0x7f708c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:23.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.512+0000 7f70a4a00640 1 --2- 172.21.15.28:0/2925953333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f708c004800 0x7f708c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:23.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.512+0000 7f70a4a00640 1 --2- 172.21.15.28:0/2925953333 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f708c0051c0 0x7f708c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:23.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.512+0000 7f70a4a00640 1 -- 172.21.15.28:0/2925953333 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f708c00c030 con 0x7f708c004800 2024-02-21T23:11:23.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.512+0000 7f70a4a00640 1 -- 172.21.15.28:0/2925953333 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f708c00c1e0 con 0x7f708c0039b0 2024-02-21T23:11:23.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.512+0000 7f70a4a00640 1 -- 172.21.15.28:0/2925953333 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f708c00c3e0 con 0x7f708c0051c0 2024-02-21T23:11:23.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.512+0000 7f70a5400640 1 --2- 172.21.15.28:0/2925953333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f708c004800 0x7f708c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:23.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.513+0000 7f70a5400640 1 -- 172.21.15.28:0/2925953333 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f708c0051c0 msgr2=0x7f708c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:23.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.513+0000 7f70a5400640 1 --2- 172.21.15.28:0/2925953333 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f708c0051c0 0x7f708c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:23.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.513+0000 7f70a5400640 1 -- 172.21.15.28:0/2925953333 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f708c0039b0 msgr2=0x7f708c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:23.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.513+0000 7f70a5400640 1 --2- 172.21.15.28:0/2925953333 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f708c0039b0 0x7f708c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:23.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.513+0000 7f70a5400640 1 -- 172.21.15.28:0/2925953333 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f708c003520 con 0x7f708c004800 2024-02-21T23:11:23.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.513+0000 7f70a6800640 1 -- 172.21.15.28:0/2925953333 reap_dead start 2024-02-21T23:11:23.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.514+0000 7f70a5400640 1 --2- 172.21.15.28:0/2925953333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f708c004800 0x7f708c004c80 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7f7094001750 tx=0x7f708c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab9b6abfa28f9712 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:23.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.514+0000 7f70a4a00640 1 -- 172.21.15.28:0/2925953333 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7094001370 con 0x7f708c004800 2024-02-21T23:11:23.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.514+0000 7f70a4a00640 1 -- 172.21.15.28:0/2925953333 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70940052d0 con 0x7f708c004800 2024-02-21T23:11:23.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.514+0000 7f70a4a00640 1 -- 172.21.15.28:0/2925953333 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7094003830 con 0x7f708c004800 2024-02-21T23:11:23.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.515+0000 7f70a7200640 1 -- 172.21.15.28:0/2925953333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f708c004800 msgr2=0x7f708c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:23.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.515+0000 7f70a7200640 1 --2- 172.21.15.28:0/2925953333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f708c004800 0x7f708c004c80 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7f7094001750 tx=0x7f708c005830 comp rx=0 tx=0).stop 2024-02-21T23:11:23.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.515+0000 7f70a7200640 1 -- 172.21.15.28:0/2925953333 shutdown_connections 2024-02-21T23:11:23.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.515+0000 7f70a7200640 1 --2- 172.21.15.28:0/2925953333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f708c004800 0x7f708c004c80 unknown :-1 s=CLOSED pgs=2001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:23.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.515+0000 7f70a7200640 1 -- 172.21.15.28:0/2925953333 >> 172.21.15.28:0/2925953333 conn(0x7f70a806b630 msgr2=0x7f70a80c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:23.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.515+0000 7f70a7200640 1 -- 172.21.15.28:0/2925953333 shutdown_connections 2024-02-21T23:11:23.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.515+0000 7f70a7200640 1 -- 172.21.15.28:0/2925953333 wait complete. 2024-02-21T23:11:23.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.517+0000 7f70a7200640 1 Processor -- start 2024-02-21T23:11:23.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.517+0000 7f70a7200640 1 -- start start 2024-02-21T23:11:23.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.518+0000 7f70a7200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f708c004800 0x7f70a8147a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:23.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.518+0000 7f70a7200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f70a8154030 0x7f70a8141760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:23.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.518+0000 7f70a7200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70a8141ca0 0x7f70a8154c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:23.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.518+0000 7f70a7200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70a8156f80 con 0x7f708c004800 2024-02-21T23:11:23.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.518+0000 7f70a7200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70a8156660 con 0x7f70a8141ca0 2024-02-21T23:11:23.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.518+0000 7f70a7200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70a8157100 con 0x7f70a8154030 2024-02-21T23:11:23.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.518+0000 7f70a5e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f708c004800 0x7f70a8147a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:23.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.518+0000 7f70a5e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f708c004800 0x7f70a8147a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59004/0 (socket says 172.21.15.28:59004) 2024-02-21T23:11:23.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.518+0000 7f70a5e00640 1 -- 172.21.15.28:0/1234838574 learned_addr learned my addr 172.21.15.28:0/1234838574 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:23.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.518+0000 7f70a6800640 1 --2- 172.21.15.28:0/1234838574 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70a8141ca0 0x7f70a8154c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:23.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.518+0000 7f70a5400640 1 --2- 172.21.15.28:0/1234838574 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f70a8154030 0x7f70a8141760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:23.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.518+0000 7f70a5e00640 1 -- 172.21.15.28:0/1234838574 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f70a8154030 msgr2=0x7f70a8141760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:23.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.518+0000 7f70a5e00640 1 --2- 172.21.15.28:0/1234838574 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f70a8154030 0x7f70a8141760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:23.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.518+0000 7f70a5e00640 1 -- 172.21.15.28:0/1234838574 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70a8141ca0 msgr2=0x7f70a8154c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:23.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.518+0000 7f70a5e00640 1 --2- 172.21.15.28:0/1234838574 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70a8141ca0 0x7f70a8154c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:23.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.518+0000 7f70a5e00640 1 -- 172.21.15.28:0/1234838574 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70a8142150 con 0x7f708c004800 2024-02-21T23:11:23.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.519+0000 7f70a5e00640 1 --2- 172.21.15.28:0/1234838574 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f708c004800 0x7f70a8147a50 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7f709800e7d0 tx=0x7f709800ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:23.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.519+0000 7f709e000640 1 -- 172.21.15.28:0/1234838574 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7098017070 con 0x7f708c004800 2024-02-21T23:11:23.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.519+0000 7f709e000640 1 -- 172.21.15.28:0/1234838574 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f709800ee80 con 0x7f708c004800 2024-02-21T23:11:23.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.519+0000 7f709e000640 1 -- 172.21.15.28:0/1234838574 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f709800cb20 con 0x7f708c004800 2024-02-21T23:11:23.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.519+0000 7f70a7200640 1 -- 172.21.15.28:0/1234838574 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70a81eec30 con 0x7f708c004800 2024-02-21T23:11:23.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.519+0000 7f70a7200640 1 -- 172.21.15.28:0/1234838574 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70a814a870 con 0x7f708c004800 2024-02-21T23:11:23.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.520+0000 7f70a7200640 1 -- 172.21.15.28:0/1234838574 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70a804f290 con 0x7f708c004800 2024-02-21T23:11:23.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.522+0000 7f709e000640 1 -- 172.21.15.28:0/1234838574 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f709801a070 con 0x7f708c004800 2024-02-21T23:11:23.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.523+0000 7f709e000640 1 --2- 172.21.15.28:0/1234838574 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f708806b500 0x7f708806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:23.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.523+0000 7f709e000640 1 -- 172.21.15.28:0/1234838574 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f709808a7f0 con 0x7f708c004800 2024-02-21T23:11:23.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.524+0000 7f70a5400640 1 --2- 172.21.15.28:0/1234838574 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f708806b500 0x7f708806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:23.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.525+0000 7f70a5400640 1 --2- 172.21.15.28:0/1234838574 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f708806b500 0x7f708806d9c0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f7094033000 tx=0x7f7094009030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:23.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.531+0000 7f709e000640 1 -- 172.21.15.28:0/1234838574 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7098059650 con 0x7f708c004800 2024-02-21T23:11:23.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.760+0000 7f70a7200640 1 -- 172.21.15.28:0/1234838574 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f70a80a78b0 con 0x7f708c004800 2024-02-21T23:11:23.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.762+0000 7f709e000640 1 -- 172.21.15.28:0/1234838574 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v174) v1 ==== 74+0+16892 (secure 0 0 0) 0x7f7098058f50 con 0x7f708c004800 2024-02-21T23:11:23.751 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:11:23.752 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":174,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:10:54.863442+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:10:51.223834+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"173","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2024-02-21T23:10:52.368371+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"174","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:11:23.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.765+0000 7f70a7200640 1 -- 172.21.15.28:0/1234838574 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f708806b500 msgr2=0x7f708806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:23.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.765+0000 7f70a7200640 1 --2- 172.21.15.28:0/1234838574 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f708806b500 0x7f708806d9c0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f7094033000 tx=0x7f7094009030 comp rx=0 tx=0).stop 2024-02-21T23:11:23.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.765+0000 7f70a7200640 1 -- 172.21.15.28:0/1234838574 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f708c004800 msgr2=0x7f70a8147a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:23.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.765+0000 7f70a7200640 1 --2- 172.21.15.28:0/1234838574 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f708c004800 0x7f70a8147a50 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7f709800e7d0 tx=0x7f709800ec90 comp rx=0 tx=0).stop 2024-02-21T23:11:23.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.765+0000 7f70a7200640 1 -- 172.21.15.28:0/1234838574 shutdown_connections 2024-02-21T23:11:23.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.765+0000 7f70a7200640 1 --2- 172.21.15.28:0/1234838574 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f708806b500 0x7f708806d9c0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:23.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.765+0000 7f70a7200640 1 --2- 172.21.15.28:0/1234838574 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f70a8154030 0x7f70a8141760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:23.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.765+0000 7f70a7200640 1 --2- 172.21.15.28:0/1234838574 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f70a8141ca0 0x7f70a8154c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:23.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.765+0000 7f70a7200640 1 --2- 172.21.15.28:0/1234838574 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f708c004800 0x7f70a8147a50 unknown :-1 s=CLOSED pgs=2002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:23.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.765+0000 7f70a7200640 1 -- 172.21.15.28:0/1234838574 >> 172.21.15.28:0/1234838574 conn(0x7f70a806b630 msgr2=0x7f70a814f960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:23.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.765+0000 7f70a7200640 1 -- 172.21.15.28:0/1234838574 shutdown_connections 2024-02-21T23:11:23.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.765+0000 7f70a7200640 1 -- 172.21.15.28:0/1234838574 wait complete. 2024-02-21T23:11:23.779 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-02-21T23:11:23.780 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:11:23.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.992+0000 7f88b4a00640 1 Processor -- start 2024-02-21T23:11:23.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.992+0000 7f88b4a00640 1 -- start start 2024-02-21T23:11:23.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.994+0000 7f88b4a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f88b01559c0 0x7f88b00c76f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:23.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.994+0000 7f88b4a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f88b00c7c30 0x7f88b015f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:23.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.994+0000 7f88b4a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88b00c6f30 con 0x7f88b01526b0 2024-02-21T23:11:23.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.994+0000 7f88b4a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88b00c61b0 con 0x7f88b01559c0 2024-02-21T23:11:23.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.994+0000 7f88b4a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88b00c6450 con 0x7f88b00c7c30 2024-02-21T23:11:23.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.994+0000 7f88ad400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f88b01559c0 0x7f88b00c76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:23.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.994+0000 7f88ad400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f88b01559c0 0x7f88b00c76f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59016/0 (socket says 172.21.15.28:59016) 2024-02-21T23:11:23.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.994+0000 7f88ad400640 1 -- 172.21.15.28:0/3582231369 learned_addr learned my addr 172.21.15.28:0/3582231369 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:23.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.994+0000 7f88ae800640 1 --2- 172.21.15.28:0/3582231369 >> v2:172.21.15.81:3300/0 conn(0x7f88b00c7c30 0x7f88b015f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:23.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.995+0000 7f88ad400640 1 -- 172.21.15.28:0/3582231369 >> v2:172.21.15.81:3300/0 conn(0x7f88b00c7c30 msgr2=0x7f88b015f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:23.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.995+0000 7f88ad400640 1 --2- 172.21.15.28:0/3582231369 >> v2:172.21.15.81:3300/0 conn(0x7f88b00c7c30 0x7f88b015f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:23.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.995+0000 7f88ad400640 1 -- 172.21.15.28:0/3582231369 >> v1:172.21.15.81:6789/0 conn(0x7f88b01526b0 legacy=0x7f88b00ca940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:23.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.995+0000 7f88ad400640 1 -- 172.21.15.28:0/3582231369 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88b00c6710 con 0x7f88b01559c0 2024-02-21T23:11:23.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.995+0000 7f88ae800640 1 --2- 172.21.15.28:0/3582231369 >> v2:172.21.15.81:3300/0 conn(0x7f88b00c7c30 0x7f88b015f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:11:23.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.996+0000 7f88ad400640 1 --2- 172.21.15.28:0/3582231369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88b01559c0 0x7f88b00c76f0 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7f88b006bdc0 tx=0x7f889c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1eb88383cca2f152 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:23.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.996+0000 7f88aca00640 1 -- 172.21.15.28:0/3582231369 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f889c035020 con 0x7f88b01559c0 2024-02-21T23:11:23.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.996+0000 7f88aca00640 1 -- 172.21.15.28:0/3582231369 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f889c031870 con 0x7f88b01559c0 2024-02-21T23:11:23.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.996+0000 7f88aca00640 1 -- 172.21.15.28:0/3582231369 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f889c031c40 con 0x7f88b01559c0 2024-02-21T23:11:23.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.997+0000 7f88b4a00640 1 -- 172.21.15.28:0/3582231369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88b01559c0 msgr2=0x7f88b00c76f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:23.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.997+0000 7f88b4a00640 1 --2- 172.21.15.28:0/3582231369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88b01559c0 0x7f88b00c76f0 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7f88b006bdc0 tx=0x7f889c02b910 comp rx=0 tx=0).stop 2024-02-21T23:11:23.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.997+0000 7f88b4a00640 1 -- 172.21.15.28:0/3582231369 shutdown_connections 2024-02-21T23:11:23.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.997+0000 7f88b4a00640 1 --2- 172.21.15.28:0/3582231369 >> v2:172.21.15.81:3300/0 conn(0x7f88b00c7c30 0x7f88b015f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:23.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.997+0000 7f88b4a00640 1 --2- 172.21.15.28:0/3582231369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88b01559c0 0x7f88b00c76f0 unknown :-1 s=CLOSED pgs=2003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:23.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.997+0000 7f88b4a00640 1 -- 172.21.15.28:0/3582231369 >> 172.21.15.28:0/3582231369 conn(0x7f88b006d3d0 msgr2=0x7f88b008afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:23.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.997+0000 7f88b4a00640 1 -- 172.21.15.28:0/3582231369 shutdown_connections 2024-02-21T23:11:23.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.997+0000 7f88b4a00640 1 -- 172.21.15.28:0/3582231369 wait complete. 2024-02-21T23:11:23.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.999+0000 7f88b4a00640 1 Processor -- start 2024-02-21T23:11:23.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:23.999+0000 7f88b4a00640 1 -- start start 2024-02-21T23:11:23.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.000+0000 7f88b4a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88b00c7c30 0x7f88b0144380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:23.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.000+0000 7f88b4a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f88b01526b0 0x7f88b014afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:23.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.000+0000 7f88b4a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f88b01559c0 0x7f88b0150560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:23.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.000+0000 7f88b4a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88b00c5c90 con 0x7f88b00c7c30 2024-02-21T23:11:23.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.000+0000 7f88b4a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88b00c71d0 con 0x7f88b01559c0 2024-02-21T23:11:23.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.000+0000 7f88b4a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88b00c4e70 con 0x7f88b01526b0 2024-02-21T23:11:23.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.000+0000 7f88ade00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88b00c7c30 0x7f88b0144380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:23.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.000+0000 7f88ade00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88b00c7c30 0x7f88b0144380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59022/0 (socket says 172.21.15.28:59022) 2024-02-21T23:11:23.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.000+0000 7f88ade00640 1 -- 172.21.15.28:0/2085646997 learned_addr learned my addr 172.21.15.28:0/2085646997 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:23.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.000+0000 7f88ade00640 1 -- 172.21.15.28:0/2085646997 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f88b01526b0 msgr2=0x7f88b014afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:23.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.000+0000 7f88ad400640 1 --2- 172.21.15.28:0/2085646997 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f88b01526b0 0x7f88b014afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:23.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.000+0000 7f88ade00640 1 --2- 172.21.15.28:0/2085646997 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f88b01526b0 0x7f88b014afc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:23.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.000+0000 7f88ade00640 1 -- 172.21.15.28:0/2085646997 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f88b01559c0 msgr2=0x7f88b0150560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:23.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.000+0000 7f88ae800640 1 --2- 172.21.15.28:0/2085646997 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f88b01559c0 0x7f88b0150560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:23.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.001+0000 7f88ade00640 1 --2- 172.21.15.28:0/2085646997 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f88b01559c0 0x7f88b0150560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:23.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.001+0000 7f88ade00640 1 -- 172.21.15.28:0/2085646997 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88b014f720 con 0x7f88b00c7c30 2024-02-21T23:11:23.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.001+0000 7f88ae800640 1 --2- 172.21.15.28:0/2085646997 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f88b01559c0 0x7f88b0150560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:11:23.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.001+0000 7f88ade00640 1 --2- 172.21.15.28:0/2085646997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88b00c7c30 0x7f88b0144380 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7f88a000c460 tx=0x7f88a000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:23.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.001+0000 7f88a6000640 1 -- 172.21.15.28:0/2085646997 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a0016020 con 0x7f88b00c7c30 2024-02-21T23:11:23.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.001+0000 7f88a6000640 1 -- 172.21.15.28:0/2085646997 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88a0012870 con 0x7f88b00c7c30 2024-02-21T23:11:23.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.001+0000 7f88b4a00640 1 -- 172.21.15.28:0/2085646997 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88b014fa00 con 0x7f88b00c7c30 2024-02-21T23:11:23.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.001+0000 7f88a6000640 1 -- 172.21.15.28:0/2085646997 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a0012c60 con 0x7f88b00c7c30 2024-02-21T23:11:23.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.001+0000 7f88b4a00640 1 -- 172.21.15.28:0/2085646997 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88b014ffc0 con 0x7f88b00c7c30 2024-02-21T23:11:23.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.003+0000 7f88b4a00640 1 -- 172.21.15.28:0/2085646997 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88b01448c0 con 0x7f88b00c7c30 2024-02-21T23:11:23.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.004+0000 7f88a6000640 1 -- 172.21.15.28:0/2085646997 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f88a0019020 con 0x7f88b00c7c30 2024-02-21T23:11:23.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.004+0000 7f88a6000640 1 --2- 172.21.15.28:0/2085646997 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f889006b4d0 0x7f889006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:23.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.004+0000 7f88a6000640 1 -- 172.21.15.28:0/2085646997 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f88a008a080 con 0x7f88b00c7c30 2024-02-21T23:11:23.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.005+0000 7f88ad400640 1 --2- 172.21.15.28:0/2085646997 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f889006b4d0 0x7f889006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:23.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.006+0000 7f88ad400640 1 --2- 172.21.15.28:0/2085646997 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f889006b4d0 0x7f889006d990 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f889c036000 tx=0x7f889c008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:23.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.009+0000 7f88a6000640 1 -- 172.21.15.28:0/2085646997 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f88a0020030 con 0x7f88b00c7c30 2024-02-21T23:11:24.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.246+0000 7f88b4a00640 1 -- 172.21.15.28:0/2085646997 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f88b0067f30 con 0x7f88b00c7c30 2024-02-21T23:11:24.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.247+0000 7f88a6000640 1 -- 172.21.15.28:0/2085646997 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 192 v192) v1 ==== 95+0+4945 (secure 0 0 0) 0x7f88a0058d60 con 0x7f88b00c7c30 2024-02-21T23:11:24.236 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:11:24.236 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":192,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6686,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/155806034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":155806034},{"type":"v1","addr":"172.21.15.81:6836","nonce":155806034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":180},{"gid":7480,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3897243745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":3897243745},{"type":"v1","addr":"172.21.15.81:6837","nonce":3897243745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":180},{"gid":7486,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2827135632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2827135632},{"type":"v1","addr":"172.21.15.28:6835","nonce":2827135632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":180}],"filesystems":[{"mdsmap":{"epoch":192,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:10:53.351225+0000","modified":"2024-02-21T23:11:15.654038+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9699},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9699":{"gid":9699,"name":"c","rank":0,"incarnation":191,"state":"up:active","state_seq":2,"addr":"172.21.15.28:6833/2364631414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2364631414},{"type":"v1","addr":"172.21.15.28:6833","nonce":2364631414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2024-02-21T23:11:24.236 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 192 2024-02-21T23:11:24.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.249+0000 7f88b4a00640 1 -- 172.21.15.28:0/2085646997 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f889006b4d0 msgr2=0x7f889006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:24.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.249+0000 7f88b4a00640 1 --2- 172.21.15.28:0/2085646997 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f889006b4d0 0x7f889006d990 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f889c036000 tx=0x7f889c008030 comp rx=0 tx=0).stop 2024-02-21T23:11:24.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.249+0000 7f88b4a00640 1 -- 172.21.15.28:0/2085646997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88b00c7c30 msgr2=0x7f88b0144380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:24.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.249+0000 7f88b4a00640 1 --2- 172.21.15.28:0/2085646997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88b00c7c30 0x7f88b0144380 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7f88a000c460 tx=0x7f88a000c920 comp rx=0 tx=0).stop 2024-02-21T23:11:24.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.249+0000 7f88b4a00640 1 -- 172.21.15.28:0/2085646997 shutdown_connections 2024-02-21T23:11:24.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.249+0000 7f88b4a00640 1 --2- 172.21.15.28:0/2085646997 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f889006b4d0 0x7f889006d990 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:24.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.249+0000 7f88b4a00640 1 --2- 172.21.15.28:0/2085646997 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f88b01526b0 0x7f88b014afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:24.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.249+0000 7f88b4a00640 1 --2- 172.21.15.28:0/2085646997 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f88b01559c0 0x7f88b0150560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:24.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.249+0000 7f88b4a00640 1 --2- 172.21.15.28:0/2085646997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88b00c7c30 0x7f88b0144380 unknown :-1 s=CLOSED pgs=2004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:24.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.249+0000 7f88b4a00640 1 -- 172.21.15.28:0/2085646997 >> 172.21.15.28:0/2085646997 conn(0x7f88b006d3d0 msgr2=0x7f88b008afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:24.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.249+0000 7f88b4a00640 1 -- 172.21.15.28:0/2085646997 shutdown_connections 2024-02-21T23:11:24.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.249+0000 7f88b4a00640 1 -- 172.21.15.28:0/2085646997 wait complete. 2024-02-21T23:11:24.260 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:11:24.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.490+0000 7fe9c7200640 1 Processor -- start 2024-02-21T23:11:24.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.490+0000 7fe9c7200640 1 -- start start 2024-02-21T23:11:24.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.491+0000 7fe9c7200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe9c8070800 0x7fe9c8158e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:24.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.491+0000 7fe9c7200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9c8070be0 con 0x7fe9c8072750 2024-02-21T23:11:24.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.491+0000 7fe9c7200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9c80a9e70 con 0x7fe9c8154e00 2024-02-21T23:11:24.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.491+0000 7fe9c7200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c80c37c0 con 0x7fe9c8070800 2024-02-21T23:11:24.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.491+0000 7fe9c5400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe9c8072750 0x7fe9c806ae70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37906/0 (socket says 172.21.15.28:37906) 2024-02-21T23:11:24.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.491+0000 7fe9c5400640 1 -- 172.21.15.28:0/3853153604 learned_addr learned my addr 172.21.15.28:0/3853153604 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:11:24.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.492+0000 7fe9c5e00640 1 --2- 172.21.15.28:0/3853153604 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe9c8070800 0x7fe9c8158e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:24.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.492+0000 7fe9c4a00640 1 -- 172.21.15.28:0/3853153604 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1331337304 0 0) 0x7fe9c8070be0 con 0x7fe9c8072750 2024-02-21T23:11:24.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.492+0000 7fe9c4a00640 1 -- 172.21.15.28:0/3853153604 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9ac003520 con 0x7fe9c8072750 2024-02-21T23:11:24.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.492+0000 7fe9c4a00640 1 -- 172.21.15.28:0/3853153604 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 600158640 0 0) 0x7fe9c80a9e70 con 0x7fe9c8154e00 2024-02-21T23:11:24.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.492+0000 7fe9c4a00640 1 -- 172.21.15.28:0/3853153604 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9c8070be0 con 0x7fe9c8154e00 2024-02-21T23:11:24.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.492+0000 7fe9c4a00640 1 -- 172.21.15.28:0/3853153604 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3460918963 0 0) 0x7fe9ac003520 con 0x7fe9c8072750 2024-02-21T23:11:24.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.492+0000 7fe9c4a00640 1 -- 172.21.15.28:0/3853153604 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe9c8070800 msgr2=0x7fe9c8158e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:24.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.492+0000 7fe9c4a00640 1 --2- 172.21.15.28:0/3853153604 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe9c8070800 0x7fe9c8158e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:24.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.492+0000 7fe9c4a00640 1 -- 172.21.15.28:0/3853153604 >> v1:172.21.15.81:6789/0 conn(0x7fe9c8154e00 legacy=0x7fe9c8155290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:24.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.492+0000 7fe9c4a00640 1 -- 172.21.15.28:0/3853153604 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c80aa150 con 0x7fe9c8072750 2024-02-21T23:11:24.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.492+0000 7fe9c4a00640 1 -- 172.21.15.28:0/3853153604 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fe9b40015f0 con 0x7fe9c8072750 2024-02-21T23:11:24.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.493+0000 7fe9c4a00640 1 -- 172.21.15.28:0/3853153604 >> v1:172.21.15.28:6789/0 conn(0x7fe9c8072750 legacy=0x7fe9c806ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:24.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.493+0000 7fe9c4a00640 1 --2- 172.21.15.28:0/3853153604 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9ac003b70 0x7fe9ac003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:24.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.493+0000 7fe9c4a00640 1 --2- 172.21.15.28:0/3853153604 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe9c8070800 0x7fe9ac004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:24.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.493+0000 7fe9c4a00640 1 --2- 172.21.15.28:0/3853153604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9ac004d40 0x7fe9ac0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:24.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.493+0000 7fe9c4a00640 1 -- 172.21.15.28:0/3853153604 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9ac00bb50 con 0x7fe9ac004d40 2024-02-21T23:11:24.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.493+0000 7fe9c4a00640 1 -- 172.21.15.28:0/3853153604 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9ac00bcd0 con 0x7fe9ac003b70 2024-02-21T23:11:24.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.493+0000 7fe9c4a00640 1 -- 172.21.15.28:0/3853153604 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9ac00bea0 con 0x7fe9c8070800 2024-02-21T23:11:24.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.493+0000 7fe9c6800640 1 --2- 172.21.15.28:0/3853153604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9ac004d40 0x7fe9ac0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:24.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.493+0000 7fe9c5e00640 1 --2- 172.21.15.28:0/3853153604 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9ac003b70 0x7fe9ac003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:24.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.493+0000 7fe9c5400640 1 --2- 172.21.15.28:0/3853153604 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe9c8070800 0x7fe9ac004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:24.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.493+0000 7fe9c6800640 1 -- 172.21.15.28:0/3853153604 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe9c8070800 msgr2=0x7fe9ac004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:24.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.493+0000 7fe9c6800640 1 --2- 172.21.15.28:0/3853153604 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe9c8070800 0x7fe9ac004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:24.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.494+0000 7fe9c6800640 1 -- 172.21.15.28:0/3853153604 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9ac003b70 msgr2=0x7fe9ac003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:24.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.494+0000 7fe9c6800640 1 --2- 172.21.15.28:0/3853153604 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9ac003b70 0x7fe9ac003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:24.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.494+0000 7fe9c6800640 1 -- 172.21.15.28:0/3853153604 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9ac003520 con 0x7fe9ac004d40 2024-02-21T23:11:24.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.494+0000 7fe9c5400640 1 --2- 172.21.15.28:0/3853153604 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe9c8070800 0x7fe9ac004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:11:24.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.495+0000 7fe9c6800640 1 --2- 172.21.15.28:0/3853153604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9ac004d40 0x7fe9ac0051c0 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7fe9c0002000 tx=0x7fe9ac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c5156852ecb6ceb server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:24.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.495+0000 7fe9c4a00640 1 -- 172.21.15.28:0/3853153604 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9c0001bf0 con 0x7fe9ac004d40 2024-02-21T23:11:24.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.495+0000 7fe9c4a00640 1 -- 172.21.15.28:0/3853153604 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9c0003b70 con 0x7fe9ac004d40 2024-02-21T23:11:24.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.495+0000 7fe9c4a00640 1 -- 172.21.15.28:0/3853153604 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9c00050d0 con 0x7fe9ac004d40 2024-02-21T23:11:24.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.496+0000 7fe9c7200640 1 -- 172.21.15.28:0/3853153604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9ac004d40 msgr2=0x7fe9ac0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:24.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.496+0000 7fe9c7200640 1 --2- 172.21.15.28:0/3853153604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9ac004d40 0x7fe9ac0051c0 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7fe9c0002000 tx=0x7fe9ac005830 comp rx=0 tx=0).stop 2024-02-21T23:11:24.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.496+0000 7fe9c6800640 1 -- 172.21.15.28:0/3853153604 reap_dead start 2024-02-21T23:11:24.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.496+0000 7fe9c7200640 1 -- 172.21.15.28:0/3853153604 shutdown_connections 2024-02-21T23:11:24.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.496+0000 7fe9c7200640 1 -- 172.21.15.28:0/3853153604 >> 172.21.15.28:0/3853153604 conn(0x7fe9c80c7720 msgr2=0x7fe9c80c7b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:24.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.496+0000 7fe9c7200640 1 -- 172.21.15.28:0/3853153604 shutdown_connections 2024-02-21T23:11:24.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.496+0000 7fe9c7200640 1 -- 172.21.15.28:0/3853153604 wait complete. 2024-02-21T23:11:24.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.498+0000 7fe9c7200640 1 Processor -- start 2024-02-21T23:11:24.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.498+0000 7fe9c7200640 1 -- start start 2024-02-21T23:11:24.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.499+0000 7fe9c7200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe9ac004d40 0x7fe9c814c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:24.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.499+0000 7fe9c7200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9c81de820 0x7fe9c81e0b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:24.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.499+0000 7fe9c7200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9c81e10a0 0x7fe9c81df800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:24.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.499+0000 7fe9c7200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c806b580 con 0x7fe9c81de820 2024-02-21T23:11:24.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.499+0000 7fe9c7200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c80c3a80 con 0x7fe9c81e10a0 2024-02-21T23:11:24.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.499+0000 7fe9c7200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c806b700 con 0x7fe9ac004d40 2024-02-21T23:11:24.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.499+0000 7fe9c5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9c81de820 0x7fe9c81e0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:24.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.499+0000 7fe9c5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9c81de820 0x7fe9c81e0b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59030/0 (socket says 172.21.15.28:59030) 2024-02-21T23:11:24.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.499+0000 7fe9c5400640 1 -- 172.21.15.28:0/2374504724 learned_addr learned my addr 172.21.15.28:0/2374504724 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:24.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.499+0000 7fe9c5e00640 1 --2- 172.21.15.28:0/2374504724 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe9ac004d40 0x7fe9c814c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:24.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.499+0000 7fe9c6800640 1 --2- 172.21.15.28:0/2374504724 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9c81e10a0 0x7fe9c81df800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:24.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.499+0000 7fe9c5400640 1 -- 172.21.15.28:0/2374504724 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe9ac004d40 msgr2=0x7fe9c814c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:24.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.499+0000 7fe9c5400640 1 --2- 172.21.15.28:0/2374504724 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe9ac004d40 0x7fe9c814c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:24.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.500+0000 7fe9c5400640 1 -- 172.21.15.28:0/2374504724 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9c81e10a0 msgr2=0x7fe9c81df800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:24.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.500+0000 7fe9c5400640 1 --2- 172.21.15.28:0/2374504724 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9c81e10a0 0x7fe9c81df800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:24.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.500+0000 7fe9c5400640 1 -- 172.21.15.28:0/2374504724 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c814b770 con 0x7fe9c81de820 2024-02-21T23:11:24.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.500+0000 7fe9c6800640 1 --2- 172.21.15.28:0/2374504724 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9c81e10a0 0x7fe9c81df800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:11:24.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.500+0000 7fe9c5400640 1 --2- 172.21.15.28:0/2374504724 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9c81de820 0x7fe9c81e0b60 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7fe9b400d7a0 tx=0x7fe9b400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:24.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.500+0000 7fe9be000640 1 -- 172.21.15.28:0/2374504724 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9b4019070 con 0x7fe9c81de820 2024-02-21T23:11:24.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.500+0000 7fe9be000640 1 -- 172.21.15.28:0/2374504724 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9b40072d0 con 0x7fe9c81de820 2024-02-21T23:11:24.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.500+0000 7fe9c7200640 1 -- 172.21.15.28:0/2374504724 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9c8156d50 con 0x7fe9c81de820 2024-02-21T23:11:24.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.500+0000 7fe9be000640 1 -- 172.21.15.28:0/2374504724 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9b40047e0 con 0x7fe9c81de820 2024-02-21T23:11:24.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.500+0000 7fe9c7200640 1 -- 172.21.15.28:0/2374504724 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9c8157310 con 0x7fe9c81de820 2024-02-21T23:11:24.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.502+0000 7fe9c7200640 1 -- 172.21.15.28:0/2374504724 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9c804f290 con 0x7fe9c81de820 2024-02-21T23:11:24.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.502+0000 7fe9be000640 1 -- 172.21.15.28:0/2374504724 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe9b401c070 con 0x7fe9c81de820 2024-02-21T23:11:24.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.503+0000 7fe9be000640 1 --2- 172.21.15.28:0/2374504724 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe9a806b500 0x7fe9a806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:24.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.503+0000 7fe9be000640 1 -- 172.21.15.28:0/2374504724 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe9b4089c20 con 0x7fe9c81de820 2024-02-21T23:11:24.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.504+0000 7fe9c5e00640 1 --2- 172.21.15.28:0/2374504724 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe9a806b500 0x7fe9a806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:24.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.505+0000 7fe9c5e00640 1 --2- 172.21.15.28:0/2374504724 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe9a806b500 0x7fe9a806d9c0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fe9b8009950 tx=0x7fe9b8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:24.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.508+0000 7fe9be000640 1 -- 172.21.15.28:0/2374504724 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe9b40589a0 con 0x7fe9c81de820 2024-02-21T23:11:24.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.752+0000 7fe9c7200640 1 -- 172.21.15.28:0/2374504724 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe9c8072530 con 0x7fe9c81de820 2024-02-21T23:11:24.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.753+0000 7fe9be000640 1 -- 172.21.15.28:0/2374504724 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 192 v192) v1 ==== 95+0+4945 (secure 0 0 0) 0x7fe9b40582a0 con 0x7fe9c81de820 2024-02-21T23:11:24.823 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:11:24.823 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":192,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6686,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/155806034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":155806034},{"type":"v1","addr":"172.21.15.81:6836","nonce":155806034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":180},{"gid":7480,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3897243745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":3897243745},{"type":"v1","addr":"172.21.15.81:6837","nonce":3897243745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":180},{"gid":7486,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2827135632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2827135632},{"type":"v1","addr":"172.21.15.28:6835","nonce":2827135632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":180}],"filesystems":[{"mdsmap":{"epoch":192,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:10:53.351225+0000","modified":"2024-02-21T23:11:15.654038+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9699},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9699":{"gid":9699,"name":"c","rank":0,"incarnation":191,"state":"up:active","state_seq":2,"addr":"172.21.15.28:6833/2364631414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2364631414},{"type":"v1","addr":"172.21.15.28:6833","nonce":2364631414}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2024-02-21T23:11:24.824 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 192 2024-02-21T23:11:24.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.757+0000 7fe9c7200640 1 -- 172.21.15.28:0/2374504724 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe9a806b500 msgr2=0x7fe9a806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:24.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.757+0000 7fe9c7200640 1 --2- 172.21.15.28:0/2374504724 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe9a806b500 0x7fe9a806d9c0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fe9b8009950 tx=0x7fe9b8008040 comp rx=0 tx=0).stop 2024-02-21T23:11:24.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.757+0000 7fe9c7200640 1 -- 172.21.15.28:0/2374504724 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9c81de820 msgr2=0x7fe9c81e0b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:24.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.757+0000 7fe9c7200640 1 --2- 172.21.15.28:0/2374504724 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9c81de820 0x7fe9c81e0b60 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7fe9b400d7a0 tx=0x7fe9b400dc60 comp rx=0 tx=0).stop 2024-02-21T23:11:24.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.757+0000 7fe9c7200640 1 -- 172.21.15.28:0/2374504724 shutdown_connections 2024-02-21T23:11:24.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.757+0000 7fe9c7200640 1 --2- 172.21.15.28:0/2374504724 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe9a806b500 0x7fe9a806d9c0 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:24.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.757+0000 7fe9c7200640 1 --2- 172.21.15.28:0/2374504724 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9c81e10a0 0x7fe9c81df800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:24.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.757+0000 7fe9c7200640 1 --2- 172.21.15.28:0/2374504724 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9c81de820 0x7fe9c81e0b60 unknown :-1 s=CLOSED pgs=2007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:24.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.757+0000 7fe9c7200640 1 --2- 172.21.15.28:0/2374504724 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe9ac004d40 0x7fe9c814c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:24.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.757+0000 7fe9c7200640 1 -- 172.21.15.28:0/2374504724 >> 172.21.15.28:0/2374504724 conn(0x7fe9c80c7720 msgr2=0x7fe9c80c5310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:24.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.757+0000 7fe9c7200640 1 -- 172.21.15.28:0/2374504724 shutdown_connections 2024-02-21T23:11:24.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:24.757+0000 7fe9c7200640 1 -- 172.21.15.28:0/2374504724 wait complete. 2024-02-21T23:11:24.826 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:11:25.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.050+0000 7f26f2400640 1 Processor -- start 2024-02-21T23:11:25.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.050+0000 7f26f2400640 1 -- start start 2024-02-21T23:11:25.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.051+0000 7f26f2400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f26ec0c45d0 0x7f26ec155510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:25.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.051+0000 7f26f2400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f26ec155a50 0x7f26ec15f860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:25.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.052+0000 7f26f2400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26ec06ba00 con 0x7f26ec1524a0 2024-02-21T23:11:25.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.052+0000 7f26f2400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26ec06f2d0 con 0x7f26ec155a50 2024-02-21T23:11:25.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.052+0000 7f26f2400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26ec154ff0 con 0x7f26ec0c45d0 2024-02-21T23:11:25.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.052+0000 7f26f1000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f26ec1524a0 0x7f26ec0c7620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60940/0 (socket says 172.21.15.28:60940) 2024-02-21T23:11:25.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.052+0000 7f26f1000640 1 -- 172.21.15.28:0/1915027346 learned_addr learned my addr 172.21.15.28:0/1915027346 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:11:25.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.052+0000 7f26f1a00640 1 --2- 172.21.15.28:0/1915027346 >> v2:172.21.15.28:3300/0 conn(0x7f26ec155a50 0x7f26ec15f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:25.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.052+0000 7f26e3e00640 1 --2- 172.21.15.28:0/1915027346 >> v2:172.21.15.81:3300/0 conn(0x7f26ec0c45d0 0x7f26ec155510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:25.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.052+0000 7f26ebe00640 1 -- 172.21.15.28:0/1915027346 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2064784855 0 0) 0x7f26ec06ba00 con 0x7f26ec1524a0 2024-02-21T23:11:25.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.052+0000 7f26ebe00640 1 -- 172.21.15.28:0/1915027346 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26d0003520 con 0x7f26ec1524a0 2024-02-21T23:11:25.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.052+0000 7f26f1a00640 1 -- 172.21.15.28:0/1915027346 >> v2:172.21.15.81:3300/0 conn(0x7f26ec0c45d0 msgr2=0x7f26ec155510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:25.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.052+0000 7f26f1a00640 1 --2- 172.21.15.28:0/1915027346 >> v2:172.21.15.81:3300/0 conn(0x7f26ec0c45d0 0x7f26ec155510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:25.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.053+0000 7f26f1a00640 1 -- 172.21.15.28:0/1915027346 >> v1:172.21.15.28:6789/0 conn(0x7f26ec1524a0 legacy=0x7f26ec0c7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:25.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.053+0000 7f26f1a00640 1 -- 172.21.15.28:0/1915027346 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26ec06aee0 con 0x7f26ec155a50 2024-02-21T23:11:25.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.055+0000 7f26f1a00640 1 --2- 172.21.15.28:0/1915027346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26ec155a50 0x7f26ec15f860 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7f26ec0c83e0 tx=0x7f26e402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52b6caf972b5b410 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:25.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.055+0000 7f26ebe00640 1 -- 172.21.15.28:0/1915027346 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e4035020 con 0x7f26ec155a50 2024-02-21T23:11:25.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.055+0000 7f26ebe00640 1 -- 172.21.15.28:0/1915027346 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26e4031830 con 0x7f26ec155a50 2024-02-21T23:11:25.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.055+0000 7f26ebe00640 1 -- 172.21.15.28:0/1915027346 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e4031c00 con 0x7f26ec155a50 2024-02-21T23:11:25.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.056+0000 7f26f2400640 1 -- 172.21.15.28:0/1915027346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26ec155a50 msgr2=0x7f26ec15f860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:25.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.056+0000 7f26f2400640 1 --2- 172.21.15.28:0/1915027346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26ec155a50 0x7f26ec15f860 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7f26ec0c83e0 tx=0x7f26e402b8d0 comp rx=0 tx=0).stop 2024-02-21T23:11:25.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.056+0000 7f26f2400640 1 -- 172.21.15.28:0/1915027346 shutdown_connections 2024-02-21T23:11:25.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.056+0000 7f26f2400640 1 --2- 172.21.15.28:0/1915027346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26ec155a50 0x7f26ec15f860 unknown :-1 s=CLOSED pgs=2009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:25.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.056+0000 7f26f2400640 1 --2- 172.21.15.28:0/1915027346 >> v2:172.21.15.81:3300/0 conn(0x7f26ec0c45d0 0x7f26ec155510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:25.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.056+0000 7f26f2400640 1 -- 172.21.15.28:0/1915027346 >> 172.21.15.28:0/1915027346 conn(0x7f26ec067050 msgr2=0x7f26ec0a84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:25.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.056+0000 7f26f2400640 1 -- 172.21.15.28:0/1915027346 shutdown_connections 2024-02-21T23:11:25.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.056+0000 7f26f2400640 1 -- 172.21.15.28:0/1915027346 wait complete. 2024-02-21T23:11:25.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.058+0000 7f26f2400640 1 Processor -- start 2024-02-21T23:11:25.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.058+0000 7f26f2400640 1 -- start start 2024-02-21T23:11:25.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.059+0000 7f26f2400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f26ec0c45d0 0x7f26ec1eb400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:25.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.059+0000 7f26f2400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f26ec1524a0 0x7f26ec121a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:25.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.059+0000 7f26f2400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26ec155a50 0x7f26ec1560d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:25.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.059+0000 7f26f2400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26ec0c4fa0 con 0x7f26ec155a50 2024-02-21T23:11:25.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.059+0000 7f26f2400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26ec06f5b0 con 0x7f26ec0c45d0 2024-02-21T23:11:25.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.059+0000 7f26f2400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26ec06b460 con 0x7f26ec1524a0 2024-02-21T23:11:25.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.059+0000 7f26f1a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26ec155a50 0x7f26ec1560d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:25.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.059+0000 7f26f1a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26ec155a50 0x7f26ec1560d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41322/0 (socket says 172.21.15.28:41322) 2024-02-21T23:11:25.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.059+0000 7f26f1a00640 1 -- 172.21.15.28:0/543307879 learned_addr learned my addr 172.21.15.28:0/543307879 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:25.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.060+0000 7f26e3e00640 1 --2- 172.21.15.28:0/543307879 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f26ec1524a0 0x7f26ec121a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:25.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.060+0000 7f26f1000640 1 --2- 172.21.15.28:0/543307879 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f26ec0c45d0 0x7f26ec1eb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:25.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.060+0000 7f26f1a00640 1 -- 172.21.15.28:0/543307879 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f26ec1524a0 msgr2=0x7f26ec121a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:25.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.060+0000 7f26f1a00640 1 --2- 172.21.15.28:0/543307879 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f26ec1524a0 0x7f26ec121a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:25.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.060+0000 7f26f1a00640 1 -- 172.21.15.28:0/543307879 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f26ec0c45d0 msgr2=0x7f26ec1eb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:25.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.060+0000 7f26f1a00640 1 --2- 172.21.15.28:0/543307879 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f26ec0c45d0 0x7f26ec1eb400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:25.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.060+0000 7f26f1a00640 1 -- 172.21.15.28:0/543307879 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26ec1209f0 con 0x7f26ec155a50 2024-02-21T23:11:25.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.060+0000 7f26f1a00640 1 --2- 172.21.15.28:0/543307879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26ec155a50 0x7f26ec1560d0 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 crypto rx=0x7f26e4009860 tx=0x7f26e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:25.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.060+0000 7f26e9600640 1 -- 172.21.15.28:0/543307879 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e4035040 con 0x7f26ec155a50 2024-02-21T23:11:25.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.060+0000 7f26e9600640 1 -- 172.21.15.28:0/543307879 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26e403f070 con 0x7f26ec155a50 2024-02-21T23:11:25.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.060+0000 7f26e9600640 1 -- 172.21.15.28:0/543307879 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e403baa0 con 0x7f26ec155a50 2024-02-21T23:11:25.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.061+0000 7f26f2400640 1 -- 172.21.15.28:0/543307879 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26ec157120 con 0x7f26ec155a50 2024-02-21T23:11:25.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.061+0000 7f26f2400640 1 -- 172.21.15.28:0/543307879 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26ec120cd0 con 0x7f26ec155a50 2024-02-21T23:11:25.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.062+0000 7f26f2400640 1 -- 172.21.15.28:0/543307879 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26ec04f290 con 0x7f26ec155a50 2024-02-21T23:11:25.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.063+0000 7f26e9600640 1 -- 172.21.15.28:0/543307879 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f26e4038020 con 0x7f26ec155a50 2024-02-21T23:11:25.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.064+0000 7f26e9600640 1 --2- 172.21.15.28:0/543307879 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f26c806b500 0x7f26c806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:25.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.065+0000 7f26e9600640 1 -- 172.21.15.28:0/543307879 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f26e40abec0 con 0x7f26ec155a50 2024-02-21T23:11:25.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.065+0000 7f26f1000640 1 --2- 172.21.15.28:0/543307879 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f26c806b500 0x7f26c806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:25.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.066+0000 7f26f1000640 1 --2- 172.21.15.28:0/543307879 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f26c806b500 0x7f26c806d9c0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f26dc0012e0 tx=0x7f26dc009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:25.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.069+0000 7f26e9600640 1 -- 172.21.15.28:0/543307879 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f26e4031800 con 0x7f26ec155a50 2024-02-21T23:11:25.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.272+0000 7f26f2400640 1 -- 172.21.15.28:0/543307879 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f26ec0c4a70 con 0x7f26ec155a50 2024-02-21T23:11:25.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.274+0000 7f26e9600640 1 -- 172.21.15.28:0/543307879 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v174) v1 ==== 74+0+16892 (secure 0 0 0) 0x7f26e407aba0 con 0x7f26ec155a50 2024-02-21T23:11:25.263 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:11:25.263 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":174,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:10:54.863442+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:10:51.223834+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"173","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_data","create_time":"2024-02-21T23:10:52.368371+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"174","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:11:25.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.277+0000 7f26f2400640 1 -- 172.21.15.28:0/543307879 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f26c806b500 msgr2=0x7f26c806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:25.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.277+0000 7f26f2400640 1 --2- 172.21.15.28:0/543307879 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f26c806b500 0x7f26c806d9c0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f26dc0012e0 tx=0x7f26dc009040 comp rx=0 tx=0).stop 2024-02-21T23:11:25.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.277+0000 7f26f2400640 1 -- 172.21.15.28:0/543307879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26ec155a50 msgr2=0x7f26ec1560d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:25.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.277+0000 7f26f2400640 1 --2- 172.21.15.28:0/543307879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26ec155a50 0x7f26ec1560d0 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 crypto rx=0x7f26e4009860 tx=0x7f26e40046c0 comp rx=0 tx=0).stop 2024-02-21T23:11:25.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.277+0000 7f26f2400640 1 -- 172.21.15.28:0/543307879 shutdown_connections 2024-02-21T23:11:25.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.277+0000 7f26f2400640 1 --2- 172.21.15.28:0/543307879 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f26c806b500 0x7f26c806d9c0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:25.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.277+0000 7f26f2400640 1 --2- 172.21.15.28:0/543307879 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f26ec1524a0 0x7f26ec121a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:25.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.277+0000 7f26f2400640 1 --2- 172.21.15.28:0/543307879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26ec155a50 0x7f26ec1560d0 unknown :-1 s=CLOSED pgs=2010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:25.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.277+0000 7f26f2400640 1 --2- 172.21.15.28:0/543307879 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f26ec0c45d0 0x7f26ec1eb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:25.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.277+0000 7f26f2400640 1 -- 172.21.15.28:0/543307879 >> 172.21.15.28:0/543307879 conn(0x7f26ec067050 msgr2=0x7f26ec14fcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:25.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.277+0000 7f26f2400640 1 -- 172.21.15.28:0/543307879 shutdown_connections 2024-02-21T23:11:25.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.277+0000 7f26f2400640 1 -- 172.21.15.28:0/543307879 wait complete. 2024-02-21T23:11:25.287 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-02-21T23:11:25.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.525+0000 7f86ae000640 1 Processor -- start 2024-02-21T23:11:25.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.525+0000 7f86ae000640 1 -- start start 2024-02-21T23:11:25.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.526+0000 7f86ae000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f86a81524c0 0x7f86a8155a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:25.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.526+0000 7f86ae000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86a806df50 con 0x7f86a80c85c0 2024-02-21T23:11:25.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.526+0000 7f86ae000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86a80c5520 con 0x7f86a80c5040 2024-02-21T23:11:25.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.526+0000 7f86ae000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86a80c5800 con 0x7f86a81524c0 2024-02-21T23:11:25.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.527+0000 7f86acc00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f86a81524c0 0x7f86a8155a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:25.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.527+0000 7f86ad600640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f86a80c85c0 0x7f86a80c70d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60946/0 (socket says 172.21.15.28:60946) 2024-02-21T23:11:25.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.527+0000 7f86ad600640 1 -- 172.21.15.28:0/1675420811 learned_addr learned my addr 172.21.15.28:0/1675420811 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:11:25.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.527+0000 7f86acc00640 1 -- 172.21.15.28:0/1675420811 >> v1:172.21.15.81:6789/0 conn(0x7f86a80c5040 legacy=0x7f86a806ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:25.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.527+0000 7f86acc00640 1 -- 172.21.15.28:0/1675420811 >> v1:172.21.15.28:6789/0 conn(0x7f86a80c85c0 legacy=0x7f86a80c70d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:25.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.527+0000 7f86acc00640 1 -- 172.21.15.28:0/1675420811 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86a806c670 con 0x7f86a81524c0 2024-02-21T23:11:25.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.529+0000 7f86acc00640 1 --2- 172.21.15.28:0/1675420811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86a81524c0 0x7f86a8155a10 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7f869c0097a0 tx=0x7f869c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1de997da2e3b14ff server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:25.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.529+0000 7f86a7400640 1 -- 172.21.15.28:0/1675420811 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f869c0315a0 con 0x7f86a81524c0 2024-02-21T23:11:25.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.529+0000 7f86a7400640 1 -- 172.21.15.28:0/1675420811 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f869c037070 con 0x7f86a81524c0 2024-02-21T23:11:25.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.529+0000 7f86a7400640 1 -- 172.21.15.28:0/1675420811 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f869c031960 con 0x7f86a81524c0 2024-02-21T23:11:25.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.530+0000 7f86ae000640 1 -- 172.21.15.28:0/1675420811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86a81524c0 msgr2=0x7f86a8155a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:25.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.530+0000 7f86ae000640 1 --2- 172.21.15.28:0/1675420811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86a81524c0 0x7f86a8155a10 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7f869c0097a0 tx=0x7f869c02b840 comp rx=0 tx=0).stop 2024-02-21T23:11:25.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.530+0000 7f86ae000640 1 -- 172.21.15.28:0/1675420811 shutdown_connections 2024-02-21T23:11:25.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.530+0000 7f86ae000640 1 --2- 172.21.15.28:0/1675420811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86a81524c0 0x7f86a8155a10 unknown :-1 s=CLOSED pgs=2012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:25.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.530+0000 7f86ae000640 1 -- 172.21.15.28:0/1675420811 >> 172.21.15.28:0/1675420811 conn(0x7f86a80ac2b0 msgr2=0x7f86a80ac6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:25.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.530+0000 7f86ae000640 1 -- 172.21.15.28:0/1675420811 shutdown_connections 2024-02-21T23:11:25.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.530+0000 7f86ae000640 1 -- 172.21.15.28:0/1675420811 wait complete. 2024-02-21T23:11:25.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.532+0000 7f86ae000640 1 Processor -- start 2024-02-21T23:11:25.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.532+0000 7f86ae000640 1 -- start start 2024-02-21T23:11:25.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.533+0000 7f86ae000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86a80c5040 0x7f86a8149600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:25.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.533+0000 7f86ae000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86a80c85c0 0x7f86a8158420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:25.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.533+0000 7f86ae000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f86a81524c0 0x7f86a80c3050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:25.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.533+0000 7f86ae000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86a80c8a70 con 0x7f86a80c85c0 2024-02-21T23:11:25.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.534+0000 7f86ae000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86a81528a0 con 0x7f86a80c5040 2024-02-21T23:11:25.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.534+0000 7f86ae000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86a80c77e0 con 0x7f86a81524c0 2024-02-21T23:11:25.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.534+0000 7f86a7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86a80c85c0 0x7f86a8158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:25.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.534+0000 7f86a7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86a80c85c0 0x7f86a8158420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41342/0 (socket says 172.21.15.28:41342) 2024-02-21T23:11:25.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.534+0000 7f86a7e00640 1 -- 172.21.15.28:0/2998446231 learned_addr learned my addr 172.21.15.28:0/2998446231 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:25.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.534+0000 7f86acc00640 1 --2- 172.21.15.28:0/2998446231 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86a80c5040 0x7f86a8149600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:25.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.534+0000 7f86ad600640 1 --2- 172.21.15.28:0/2998446231 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f86a81524c0 0x7f86a80c3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:25.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.534+0000 7f86a7e00640 1 -- 172.21.15.28:0/2998446231 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f86a81524c0 msgr2=0x7f86a80c3050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:25.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.535+0000 7f86a7e00640 1 --2- 172.21.15.28:0/2998446231 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f86a81524c0 0x7f86a80c3050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:25.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.535+0000 7f86a7e00640 1 -- 172.21.15.28:0/2998446231 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86a80c5040 msgr2=0x7f86a8149600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:25.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.535+0000 7f86a7e00640 1 --2- 172.21.15.28:0/2998446231 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86a80c5040 0x7f86a8149600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:25.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.535+0000 7f86a7e00640 1 -- 172.21.15.28:0/2998446231 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86a8158e80 con 0x7f86a80c85c0 2024-02-21T23:11:25.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.535+0000 7f86acc00640 1 --2- 172.21.15.28:0/2998446231 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86a80c5040 0x7f86a8149600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:11:25.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.535+0000 7f86a7e00640 1 --2- 172.21.15.28:0/2998446231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86a80c85c0 0x7f86a8158420 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 crypto rx=0x7f869400c430 tx=0x7f869400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:25.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.535+0000 7f86a4c00640 1 -- 172.21.15.28:0/2998446231 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86940052d0 con 0x7f86a80c85c0 2024-02-21T23:11:25.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.535+0000 7f86a4c00640 1 -- 172.21.15.28:0/2998446231 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8694005490 con 0x7f86a80c85c0 2024-02-21T23:11:25.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.535+0000 7f86a4c00640 1 -- 172.21.15.28:0/2998446231 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8694007260 con 0x7f86a80c85c0 2024-02-21T23:11:25.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.535+0000 7f86ae000640 1 -- 172.21.15.28:0/2998446231 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86a80c4140 con 0x7f86a80c85c0 2024-02-21T23:11:25.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.535+0000 7f86ae000640 1 -- 172.21.15.28:0/2998446231 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86a80c3590 con 0x7f86a80c85c0 2024-02-21T23:11:25.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.537+0000 7f86ae000640 1 -- 172.21.15.28:0/2998446231 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86a804f290 con 0x7f86a80c85c0 2024-02-21T23:11:25.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.538+0000 7f86a4c00640 1 -- 172.21.15.28:0/2998446231 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8694007400 con 0x7f86a80c85c0 2024-02-21T23:11:25.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.539+0000 7f86a4c00640 1 --2- 172.21.15.28:0/2998446231 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f868806b4d0 0x7f868806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:25.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.539+0000 7f86a4c00640 1 -- 172.21.15.28:0/2998446231 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f8694089890 con 0x7f86a80c85c0 2024-02-21T23:11:25.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.540+0000 7f86acc00640 1 --2- 172.21.15.28:0/2998446231 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f868806b4d0 0x7f868806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:25.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.541+0000 7f86acc00640 1 --2- 172.21.15.28:0/2998446231 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f868806b4d0 0x7f868806d990 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f869c036000 tx=0x7f869c03d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:25.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.544+0000 7f86a4c00640 1 -- 172.21.15.28:0/2998446231 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f8694058570 con 0x7f86a80c85c0 2024-02-21T23:11:25.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.805+0000 7f86ae000640 1 -- 172.21.15.28:0/2998446231 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f86a80aadb0 con 0x7f86a80c85c0 2024-02-21T23:11:25.794 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-02-21T23:11:25.806+0000 7f5ddc400640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-21T23:11:25.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.809+0000 7f86a4c00640 1 -- 172.21.15.28:0/2998446231 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v193) v1 ==== 161+0+0 (secure 0 0 0) 0x7f8694057e70 con 0x7f86a80c85c0 2024-02-21T23:11:25.797 INFO:teuthology.orchestra.run.smithi028.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-02-21T23:11:25.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.811+0000 7f86ae000640 1 -- 172.21.15.28:0/2998446231 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f868806b4d0 msgr2=0x7f868806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:25.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.811+0000 7f86ae000640 1 --2- 172.21.15.28:0/2998446231 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f868806b4d0 0x7f868806d990 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f869c036000 tx=0x7f869c03d040 comp rx=0 tx=0).stop 2024-02-21T23:11:25.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.811+0000 7f86ae000640 1 -- 172.21.15.28:0/2998446231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86a80c85c0 msgr2=0x7f86a8158420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:25.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.811+0000 7f86ae000640 1 --2- 172.21.15.28:0/2998446231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86a80c85c0 0x7f86a8158420 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 crypto rx=0x7f869400c430 tx=0x7f869400c8f0 comp rx=0 tx=0).stop 2024-02-21T23:11:25.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.812+0000 7f86ae000640 1 -- 172.21.15.28:0/2998446231 shutdown_connections 2024-02-21T23:11:25.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.812+0000 7f86ae000640 1 --2- 172.21.15.28:0/2998446231 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f868806b4d0 0x7f868806d990 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:25.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.812+0000 7f86ae000640 1 --2- 172.21.15.28:0/2998446231 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f86a81524c0 0x7f86a80c3050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:25.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.812+0000 7f86ae000640 1 --2- 172.21.15.28:0/2998446231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86a80c85c0 0x7f86a8158420 unknown :-1 s=CLOSED pgs=2013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:25.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.812+0000 7f86ae000640 1 --2- 172.21.15.28:0/2998446231 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86a80c5040 0x7f86a8149600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:25.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.812+0000 7f86ae000640 1 -- 172.21.15.28:0/2998446231 >> 172.21.15.28:0/2998446231 conn(0x7f86a80ac2b0 msgr2=0x7f86a80c7cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:25.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.812+0000 7f86ae000640 1 -- 172.21.15.28:0/2998446231 shutdown_connections 2024-02-21T23:11:25.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:25.812+0000 7f86ae000640 1 -- 172.21.15.28:0/2998446231 wait complete. 2024-02-21T23:11:25.809 INFO:tasks.ceph.mds.c.smithi028.stderr: -210> 2024-02-21T23:11:19.625+0000 7f6d00c00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-21T23:11:25.820 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-02-21T23:11:25.851 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-02-21T23:11:26.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.019+0000 7fe509600640 1 Processor -- start 2024-02-21T23:11:26.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.019+0000 7fe509600640 1 -- start start 2024-02-21T23:11:26.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.020+0000 7fe509600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fe50406b0e0 0x7fe5040c5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:26.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.020+0000 7fe509600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe5040c5d80 0x7fe50414fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:26.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.020+0000 7fe509600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe504150530 0x7fe504156500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:26.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.020+0000 7fe509600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe504153bc0 con 0x7fe5040c5d80 2024-02-21T23:11:26.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.020+0000 7fe509600640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5040c61e0 con 0x7fe50406b0e0 2024-02-21T23:11:26.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.020+0000 7fe509600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe504152fb0 con 0x7fe504150530 2024-02-21T23:11:26.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.021+0000 7fe502000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe5040c5d80 0x7fe50414fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:26.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.021+0000 7fe502000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe5040c5d80 0x7fe50414fff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41374/0 (socket says 172.21.15.28:41374) 2024-02-21T23:11:26.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.021+0000 7fe502000640 1 -- 172.21.15.28:0/1426042423 learned_addr learned my addr 172.21.15.28:0/1426042423 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:26.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.021+0000 7fe502a00640 1 --2- 172.21.15.28:0/1426042423 >> v2:172.21.15.81:3300/0 conn(0x7fe50406b0e0 0x7fe5040c5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:26.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.021+0000 7fe503400640 1 --2- 172.21.15.28:0/1426042423 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe504150530 0x7fe504156500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:26.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.021+0000 7fe502000640 1 -- 172.21.15.28:0/1426042423 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe504150530 msgr2=0x7fe504156500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:26.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.021+0000 7fe502000640 1 --2- 172.21.15.28:0/1426042423 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe504150530 0x7fe504156500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:26.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.021+0000 7fe502000640 1 -- 172.21.15.28:0/1426042423 >> v2:172.21.15.81:3300/0 conn(0x7fe50406b0e0 msgr2=0x7fe5040c5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:26.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.021+0000 7fe502000640 1 --2- 172.21.15.28:0/1426042423 >> v2:172.21.15.81:3300/0 conn(0x7fe50406b0e0 0x7fe5040c5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:26.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.021+0000 7fe502000640 1 -- 172.21.15.28:0/1426042423 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe504153290 con 0x7fe5040c5d80 2024-02-21T23:11:26.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.022+0000 7fe502000640 1 --2- 172.21.15.28:0/1426042423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5040c5d80 0x7fe50414fff0 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7fe5040c77e0 tx=0x7fe4ec02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=953f4c8f087bb2f8 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:26.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.023+0000 7fe501600640 1 -- 172.21.15.28:0/1426042423 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4ec02c9e0 con 0x7fe5040c5d80 2024-02-21T23:11:26.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.023+0000 7fe501600640 1 -- 172.21.15.28:0/1426042423 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4ec02cb80 con 0x7fe5040c5d80 2024-02-21T23:11:26.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.023+0000 7fe501600640 1 -- 172.21.15.28:0/1426042423 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4ec0343f0 con 0x7fe5040c5d80 2024-02-21T23:11:26.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.023+0000 7fe509600640 1 -- 172.21.15.28:0/1426042423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5040c5d80 msgr2=0x7fe50414fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:26.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.023+0000 7fe509600640 1 --2- 172.21.15.28:0/1426042423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5040c5d80 0x7fe50414fff0 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7fe5040c77e0 tx=0x7fe4ec02be40 comp rx=0 tx=0).stop 2024-02-21T23:11:26.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.024+0000 7fe509600640 1 -- 172.21.15.28:0/1426042423 shutdown_connections 2024-02-21T23:11:26.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.024+0000 7fe509600640 1 --2- 172.21.15.28:0/1426042423 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe504150530 0x7fe504156500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:26.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.024+0000 7fe509600640 1 --2- 172.21.15.28:0/1426042423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5040c5d80 0x7fe50414fff0 unknown :-1 s=CLOSED pgs=2017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:26.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.024+0000 7fe509600640 1 --2- 172.21.15.28:0/1426042423 >> v2:172.21.15.81:3300/0 conn(0x7fe50406b0e0 0x7fe5040c5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:26.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.024+0000 7fe509600640 1 -- 172.21.15.28:0/1426042423 >> 172.21.15.28:0/1426042423 conn(0x7fe5040a85a0 msgr2=0x7fe5040a8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:26.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.024+0000 7fe509600640 1 -- 172.21.15.28:0/1426042423 shutdown_connections 2024-02-21T23:11:26.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.024+0000 7fe509600640 1 -- 172.21.15.28:0/1426042423 wait complete. 2024-02-21T23:11:26.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.025+0000 7fe509600640 1 Processor -- start 2024-02-21T23:11:26.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.025+0000 7fe509600640 1 -- start start 2024-02-21T23:11:26.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.026+0000 7fe509600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe50406b0e0 0x7fe5041eae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:26.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.026+0000 7fe509600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5040c5d80 0x7fe5041eb390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:26.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.026+0000 7fe509600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe504150530 0x7fe50406e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:26.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.026+0000 7fe509600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5041509c0 con 0x7fe5040c5d80 2024-02-21T23:11:26.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.026+0000 7fe509600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe504153850 con 0x7fe50406b0e0 2024-02-21T23:11:26.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.026+0000 7fe509600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe50406b4c0 con 0x7fe504150530 2024-02-21T23:11:26.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.027+0000 7fe502000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5040c5d80 0x7fe5041eb390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:26.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.027+0000 7fe502000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5040c5d80 0x7fe5041eb390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41380/0 (socket says 172.21.15.28:41380) 2024-02-21T23:11:26.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.027+0000 7fe502000640 1 -- 172.21.15.28:0/2540564045 learned_addr learned my addr 172.21.15.28:0/2540564045 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:26.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.027+0000 7fe502a00640 1 --2- 172.21.15.28:0/2540564045 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe50406b0e0 0x7fe5041eae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:26.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.027+0000 7fe503400640 1 --2- 172.21.15.28:0/2540564045 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe504150530 0x7fe50406e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:26.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.027+0000 7fe502000640 1 -- 172.21.15.28:0/2540564045 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe504150530 msgr2=0x7fe50406e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:26.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.027+0000 7fe502000640 1 --2- 172.21.15.28:0/2540564045 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe504150530 0x7fe50406e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:26.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.027+0000 7fe502000640 1 -- 172.21.15.28:0/2540564045 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe50406b0e0 msgr2=0x7fe5041eae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:26.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.027+0000 7fe502000640 1 --2- 172.21.15.28:0/2540564045 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe50406b0e0 0x7fe5041eae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:26.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.027+0000 7fe502000640 1 -- 172.21.15.28:0/2540564045 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5041eec20 con 0x7fe5040c5d80 2024-02-21T23:11:26.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.027+0000 7fe502000640 1 --2- 172.21.15.28:0/2540564045 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5040c5d80 0x7fe5041eb390 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7fe4ec000f80 tx=0x7fe4ec004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:26.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.028+0000 7fe4faa00640 1 -- 172.21.15.28:0/2540564045 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4ec043020 con 0x7fe5040c5d80 2024-02-21T23:11:26.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.028+0000 7fe4faa00640 1 -- 172.21.15.28:0/2540564045 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4ec03e450 con 0x7fe5040c5d80 2024-02-21T23:11:26.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.028+0000 7fe4faa00640 1 -- 172.21.15.28:0/2540564045 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4ec009510 con 0x7fe5040c5d80 2024-02-21T23:11:26.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.028+0000 7fe509600640 1 -- 172.21.15.28:0/2540564045 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe504152140 con 0x7fe5040c5d80 2024-02-21T23:11:26.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.028+0000 7fe509600640 1 -- 172.21.15.28:0/2540564045 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe504152700 con 0x7fe5040c5d80 2024-02-21T23:11:26.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.029+0000 7fe509600640 1 -- 172.21.15.28:0/2540564045 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe504067f30 con 0x7fe5040c5d80 2024-02-21T23:11:26.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.030+0000 7fe4faa00640 1 -- 172.21.15.28:0/2540564045 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe4ec046020 con 0x7fe5040c5d80 2024-02-21T23:11:26.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.031+0000 7fe4faa00640 1 --2- 172.21.15.28:0/2540564045 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe4e006b4d0 0x7fe4e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:26.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.031+0000 7fe4faa00640 1 -- 172.21.15.28:0/2540564045 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fe4ec0acc40 con 0x7fe5040c5d80 2024-02-21T23:11:26.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.031+0000 7fe502a00640 1 --2- 172.21.15.28:0/2540564045 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe4e006b4d0 0x7fe4e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:26.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.032+0000 7fe502a00640 1 --2- 172.21.15.28:0/2540564045 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe4e006b4d0 0x7fe4e006d990 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fe4f4006fd0 tx=0x7fe4f4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:26.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.035+0000 7fe4faa00640 1 -- 172.21.15.28:0/2540564045 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe4ec04d030 con 0x7fe5040c5d80 2024-02-21T23:11:26.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.273+0000 7fe509600640 1 -- 172.21.15.28:0/2540564045 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fe5040a8980 con 0x7fe5040c5d80 2024-02-21T23:11:26.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.812+0000 7fe4faa00640 1 -- 172.21.15.28:0/2540564045 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v194) v1 ==== 104+0+0 (secure 0 0 0) 0x7fe4ec07b9f0 con 0x7fe5040c5d80 2024-02-21T23:11:26.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.816+0000 7fe509600640 1 -- 172.21.15.28:0/2540564045 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe4e006b4d0 msgr2=0x7fe4e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:26.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.816+0000 7fe509600640 1 --2- 172.21.15.28:0/2540564045 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe4e006b4d0 0x7fe4e006d990 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fe4f4006fd0 tx=0x7fe4f4008040 comp rx=0 tx=0).stop 2024-02-21T23:11:26.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.816+0000 7fe509600640 1 -- 172.21.15.28:0/2540564045 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5040c5d80 msgr2=0x7fe5041eb390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:26.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.816+0000 7fe509600640 1 --2- 172.21.15.28:0/2540564045 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5040c5d80 0x7fe5041eb390 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7fe4ec000f80 tx=0x7fe4ec004a40 comp rx=0 tx=0).stop 2024-02-21T23:11:26.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.816+0000 7fe509600640 1 -- 172.21.15.28:0/2540564045 shutdown_connections 2024-02-21T23:11:26.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.816+0000 7fe509600640 1 --2- 172.21.15.28:0/2540564045 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe4e006b4d0 0x7fe4e006d990 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:26.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.816+0000 7fe509600640 1 --2- 172.21.15.28:0/2540564045 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe504150530 0x7fe50406e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:26.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.816+0000 7fe509600640 1 --2- 172.21.15.28:0/2540564045 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5040c5d80 0x7fe5041eb390 unknown :-1 s=CLOSED pgs=2018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:26.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.816+0000 7fe509600640 1 --2- 172.21.15.28:0/2540564045 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe50406b0e0 0x7fe5041eae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:26.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.816+0000 7fe509600640 1 -- 172.21.15.28:0/2540564045 >> 172.21.15.28:0/2540564045 conn(0x7fe5040a85a0 msgr2=0x7fe5040729d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:26.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.816+0000 7fe509600640 1 -- 172.21.15.28:0/2540564045 shutdown_connections 2024-02-21T23:11:26.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:26.816+0000 7fe509600640 1 -- 172.21.15.28:0/2540564045 wait complete. 2024-02-21T23:11:26.833 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-02-21T23:11:27.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.115+0000 7fdc29e00640 1 Processor -- start 2024-02-21T23:11:27.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.115+0000 7fdc29e00640 1 -- start start 2024-02-21T23:11:27.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.117+0000 7fdc29e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdc24155b00 0x7fdc24155f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:27.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.117+0000 7fdc29e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fdc240c29d0 0x7fdc240c2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:27.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.117+0000 7fdc29e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc240c2490 con 0x7fdc240c5a70 2024-02-21T23:11:27.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.117+0000 7fdc29e00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc2406f200 con 0x7fdc240c29d0 2024-02-21T23:11:27.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.117+0000 7fdc29e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc240c1910 con 0x7fdc24155b00 2024-02-21T23:11:27.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.117+0000 7fdc22a00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fdc240c5a70 0x7fdc24072310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.28:39420/0 (socket says 172.21.15.28:39420) 2024-02-21T23:11:27.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.117+0000 7fdc22a00640 1 -- 172.21.15.28:0/2938101698 learned_addr learned my addr 172.21.15.28:0/2938101698 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:11:27.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.117+0000 7fdc1be00640 1 --2- 172.21.15.28:0/2938101698 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdc24155b00 0x7fdc24155f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:27.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.117+0000 7fdc23400640 1 --2- 172.21.15.28:0/2938101698 >> v2:172.21.15.81:3300/0 conn(0x7fdc240c29d0 0x7fdc240c2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:27.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.117+0000 7fdc1be00640 1 -- 172.21.15.28:0/2938101698 >> v2:172.21.15.81:3300/0 conn(0x7fdc240c29d0 msgr2=0x7fdc240c2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:27.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.117+0000 7fdc1be00640 1 --2- 172.21.15.28:0/2938101698 >> v2:172.21.15.81:3300/0 conn(0x7fdc240c29d0 0x7fdc240c2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:27.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.117+0000 7fdc1be00640 1 -- 172.21.15.28:0/2938101698 >> v1:172.21.15.81:6789/0 conn(0x7fdc240c5a70 legacy=0x7fdc24072310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:27.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.117+0000 7fdc1be00640 1 -- 172.21.15.28:0/2938101698 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc240c1bf0 con 0x7fdc24155b00 2024-02-21T23:11:27.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.120+0000 7fdc1be00640 1 --2- 172.21.15.28:0/2938101698 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdc24155b00 0x7fdc24155f20 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c009790 tx=0x7fdc0c02b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae99720f7dbba744 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:27.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.120+0000 7fdc22000640 1 -- 172.21.15.28:0/2938101698 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc0c039070 con 0x7fdc24155b00 2024-02-21T23:11:27.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.120+0000 7fdc22000640 1 -- 172.21.15.28:0/2938101698 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc0c0316f0 con 0x7fdc24155b00 2024-02-21T23:11:27.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.120+0000 7fdc22000640 1 -- 172.21.15.28:0/2938101698 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc0c031ac0 con 0x7fdc24155b00 2024-02-21T23:11:27.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.121+0000 7fdc29e00640 1 -- 172.21.15.28:0/2938101698 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdc24155b00 msgr2=0x7fdc24155f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:27.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.121+0000 7fdc29e00640 1 --2- 172.21.15.28:0/2938101698 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdc24155b00 0x7fdc24155f20 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c009790 tx=0x7fdc0c02b790 comp rx=0 tx=0).stop 2024-02-21T23:11:27.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.121+0000 7fdc29e00640 1 -- 172.21.15.28:0/2938101698 shutdown_connections 2024-02-21T23:11:27.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.121+0000 7fdc29e00640 1 --2- 172.21.15.28:0/2938101698 >> v2:172.21.15.81:3300/0 conn(0x7fdc240c29d0 0x7fdc240c2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:27.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.121+0000 7fdc29e00640 1 --2- 172.21.15.28:0/2938101698 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdc24155b00 0x7fdc24155f20 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:27.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.121+0000 7fdc29e00640 1 -- 172.21.15.28:0/2938101698 >> 172.21.15.28:0/2938101698 conn(0x7fdc2406b830 msgr2=0x7fdc240c67f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:27.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.121+0000 7fdc29e00640 1 -- 172.21.15.28:0/2938101698 shutdown_connections 2024-02-21T23:11:27.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.121+0000 7fdc29e00640 1 -- 172.21.15.28:0/2938101698 wait complete. 2024-02-21T23:11:27.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.123+0000 7fdc29e00640 1 Processor -- start 2024-02-21T23:11:27.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.123+0000 7fdc29e00640 1 -- start start 2024-02-21T23:11:27.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.124+0000 7fdc29e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdc240c29d0 0x7fdc240bfe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:27.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.124+0000 7fdc29e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc240c5a70 0x7fdc2414c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:27.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.124+0000 7fdc29e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc24155b00 0x7fdc241eeb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:27.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.124+0000 7fdc29e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc24072a20 con 0x7fdc240c5a70 2024-02-21T23:11:27.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.124+0000 7fdc29e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc240c21b0 con 0x7fdc24155b00 2024-02-21T23:11:27.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.124+0000 7fdc29e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc24156460 con 0x7fdc240c29d0 2024-02-21T23:11:27.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.124+0000 7fdc1be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc240c5a70 0x7fdc2414c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:27.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.124+0000 7fdc1be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc240c5a70 0x7fdc2414c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41394/0 (socket says 172.21.15.28:41394) 2024-02-21T23:11:27.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.124+0000 7fdc1be00640 1 -- 172.21.15.28:0/1343105604 learned_addr learned my addr 172.21.15.28:0/1343105604 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:27.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.125+0000 7fdc22a00640 1 --2- 172.21.15.28:0/1343105604 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdc240c29d0 0x7fdc240bfe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:27.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.125+0000 7fdc23400640 1 --2- 172.21.15.28:0/1343105604 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc24155b00 0x7fdc241eeb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:27.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.125+0000 7fdc1be00640 1 -- 172.21.15.28:0/1343105604 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdc240c29d0 msgr2=0x7fdc240bfe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:27.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.125+0000 7fdc1be00640 1 --2- 172.21.15.28:0/1343105604 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdc240c29d0 0x7fdc240bfe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:27.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.125+0000 7fdc1be00640 1 -- 172.21.15.28:0/1343105604 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc24155b00 msgr2=0x7fdc241eeb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:27.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.125+0000 7fdc1be00640 1 --2- 172.21.15.28:0/1343105604 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc24155b00 0x7fdc241eeb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:27.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.125+0000 7fdc1be00640 1 -- 172.21.15.28:0/1343105604 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc2414bf20 con 0x7fdc240c5a70 2024-02-21T23:11:27.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.125+0000 7fdc1be00640 1 --2- 172.21.15.28:0/1343105604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc240c5a70 0x7fdc2414c5d0 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c02bda0 tx=0x7fdc0c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:27.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.126+0000 7fdc1b400640 1 -- 172.21.15.28:0/1343105604 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc0c042030 con 0x7fdc240c5a70 2024-02-21T23:11:27.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.126+0000 7fdc1b400640 1 -- 172.21.15.28:0/1343105604 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc0c031ea0 con 0x7fdc240c5a70 2024-02-21T23:11:27.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.126+0000 7fdc1b400640 1 -- 172.21.15.28:0/1343105604 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc0c039040 con 0x7fdc240c5a70 2024-02-21T23:11:27.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.126+0000 7fdc29e00640 1 -- 172.21.15.28:0/1343105604 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc2414c200 con 0x7fdc240c5a70 2024-02-21T23:11:27.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.126+0000 7fdc29e00640 1 -- 172.21.15.28:0/1343105604 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc240aafb0 con 0x7fdc240c5a70 2024-02-21T23:11:27.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.127+0000 7fdc29e00640 1 -- 172.21.15.28:0/1343105604 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc24067f30 con 0x7fdc240c5a70 2024-02-21T23:11:27.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.128+0000 7fdc1b400640 1 -- 172.21.15.28:0/1343105604 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdc0c045070 con 0x7fdc240c5a70 2024-02-21T23:11:27.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.129+0000 7fdc1b400640 1 --2- 172.21.15.28:0/1343105604 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdc0006b500 0x7fdc0006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:27.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.129+0000 7fdc1b400640 1 -- 172.21.15.28:0/1343105604 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 6274+0+0 (secure 0 0 0) 0x7fdc0c0ab720 con 0x7fdc240c5a70 2024-02-21T23:11:27.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.130+0000 7fdc22a00640 1 --2- 172.21.15.28:0/1343105604 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdc0006b500 0x7fdc0006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:27.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.131+0000 7fdc22a00640 1 --2- 172.21.15.28:0/1343105604 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdc0006b500 0x7fdc0006d9c0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fdc14005880 tx=0x7fdc14009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:27.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.133+0000 7fdc1b400640 1 -- 172.21.15.28:0/1343105604 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fdc0c07a470 con 0x7fdc240c5a70 2024-02-21T23:11:27.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.334+0000 7fdc29e00640 1 -- 172.21.15.28:0/1343105604 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fdc240ab870 con 0x7fdc240c5a70 2024-02-21T23:11:27.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.831+0000 7fdc1b400640 1 -- 172.21.15.28:0/1343105604 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v177) v1 ==== 181+0+0 (secure 0 0 0) 0x7fdc0c079d70 con 0x7fdc240c5a70 2024-02-21T23:11:27.819 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' removed 2024-02-21T23:11:27.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.835+0000 7fdc29e00640 1 -- 172.21.15.28:0/1343105604 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdc0006b500 msgr2=0x7fdc0006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:27.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.835+0000 7fdc29e00640 1 --2- 172.21.15.28:0/1343105604 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdc0006b500 0x7fdc0006d9c0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fdc14005880 tx=0x7fdc14009040 comp rx=0 tx=0).stop 2024-02-21T23:11:27.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.835+0000 7fdc29e00640 1 -- 172.21.15.28:0/1343105604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc240c5a70 msgr2=0x7fdc2414c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:27.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.835+0000 7fdc29e00640 1 --2- 172.21.15.28:0/1343105604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc240c5a70 0x7fdc2414c5d0 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c02bda0 tx=0x7fdc0c0046c0 comp rx=0 tx=0).stop 2024-02-21T23:11:27.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.836+0000 7fdc29e00640 1 -- 172.21.15.28:0/1343105604 shutdown_connections 2024-02-21T23:11:27.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.836+0000 7fdc29e00640 1 --2- 172.21.15.28:0/1343105604 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdc0006b500 0x7fdc0006d9c0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:27.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.836+0000 7fdc29e00640 1 --2- 172.21.15.28:0/1343105604 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc24155b00 0x7fdc241eeb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:27.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.836+0000 7fdc29e00640 1 --2- 172.21.15.28:0/1343105604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc240c5a70 0x7fdc2414c5d0 unknown :-1 s=CLOSED pgs=2019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:27.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.836+0000 7fdc29e00640 1 --2- 172.21.15.28:0/1343105604 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdc240c29d0 0x7fdc240bfe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:27.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.836+0000 7fdc29e00640 1 -- 172.21.15.28:0/1343105604 >> 172.21.15.28:0/1343105604 conn(0x7fdc2406b830 msgr2=0x7fdc240c4f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:27.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.837+0000 7fdc29e00640 1 -- 172.21.15.28:0/1343105604 shutdown_connections 2024-02-21T23:11:27.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:27.837+0000 7fdc29e00640 1 -- 172.21.15.28:0/1343105604 wait complete. 2024-02-21T23:11:27.852 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-02-21T23:11:28.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.063+0000 7ff3f0e00640 1 Processor -- start 2024-02-21T23:11:28.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.064+0000 7ff3f0e00640 1 -- start start 2024-02-21T23:11:28.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.065+0000 7ff3f0e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff3ec0ca540 0x7ff3ec158460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:28.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.065+0000 7ff3f0e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3ec0caf40 con 0x7ff3ec0c6290 2024-02-21T23:11:28.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.065+0000 7ff3f0e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3ec0c87b0 con 0x7ff3ec06cfa0 2024-02-21T23:11:28.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.065+0000 7ff3f0e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ec0c8ae0 con 0x7ff3ec0ca540 2024-02-21T23:11:28.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.066+0000 7ff3ebe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff3ec0c6290 0x7ff3ec1522c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60956/0 (socket says 172.21.15.28:60956) 2024-02-21T23:11:28.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.066+0000 7ff3ebe00640 1 -- 172.21.15.28:0/480125253 learned_addr learned my addr 172.21.15.28:0/480125253 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:11:28.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.066+0000 7ff3eaa00640 1 --2- 172.21.15.28:0/480125253 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff3ec0ca540 0x7ff3ec158460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:28.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.066+0000 7ff3ea000640 1 -- 172.21.15.28:0/480125253 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1155713011 0 0) 0x7ff3ec0caf40 con 0x7ff3ec0c6290 2024-02-21T23:11:28.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.066+0000 7ff3ea000640 1 -- 172.21.15.28:0/480125253 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3d0003520 con 0x7ff3ec0c6290 2024-02-21T23:11:28.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.067+0000 7ff3ea000640 1 -- 172.21.15.28:0/480125253 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1229533321 0 0) 0x7ff3d0003520 con 0x7ff3ec0c6290 2024-02-21T23:11:28.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.067+0000 7ff3ea000640 1 -- 172.21.15.28:0/480125253 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff3ec0ca540 msgr2=0x7ff3ec158460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:28.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.067+0000 7ff3ea000640 1 --2- 172.21.15.28:0/480125253 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff3ec0ca540 0x7ff3ec158460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:28.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.067+0000 7ff3ea000640 1 -- 172.21.15.28:0/480125253 >> v1:172.21.15.81:6789/0 conn(0x7ff3ec06cfa0 legacy=0x7ff3ec0c8f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:28.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.067+0000 7ff3ea000640 1 -- 172.21.15.28:0/480125253 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3ec0c7f80 con 0x7ff3ec0c6290 2024-02-21T23:11:28.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.067+0000 7ff3ea000640 1 -- 172.21.15.28:0/480125253 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7ff3e40019f0 con 0x7ff3ec0c6290 2024-02-21T23:11:28.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.067+0000 7ff3ea000640 1 -- 172.21.15.28:0/480125253 >> v1:172.21.15.28:6789/0 conn(0x7ff3ec0c6290 legacy=0x7ff3ec1522c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:28.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.067+0000 7ff3ea000640 1 --2- 172.21.15.28:0/480125253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3d00039e0 0x7ff3d0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:28.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.067+0000 7ff3ea000640 1 --2- 172.21.15.28:0/480125253 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff3ec0ca540 0x7ff3d0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:28.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.067+0000 7ff3ea000640 1 --2- 172.21.15.28:0/480125253 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3d0004d40 0x7ff3d00051a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:28.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.067+0000 7ff3ea000640 1 -- 172.21.15.28:0/480125253 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ec0c8ae0 con 0x7ff3d00039e0 2024-02-21T23:11:28.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.067+0000 7ff3ea000640 1 -- 172.21.15.28:0/480125253 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d000bc00 con 0x7ff3d0004d40 2024-02-21T23:11:28.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.067+0000 7ff3ea000640 1 -- 172.21.15.28:0/480125253 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d000be00 con 0x7ff3ec0ca540 2024-02-21T23:11:28.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.067+0000 7ff3eb400640 1 --2- 172.21.15.28:0/480125253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3d00039e0 0x7ff3d0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:28.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.067+0000 7ff3eaa00640 1 --2- 172.21.15.28:0/480125253 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff3ec0ca540 0x7ff3d0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:28.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.068+0000 7ff3ebe00640 1 --2- 172.21.15.28:0/480125253 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3d0004d40 0x7ff3d00051a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:28.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.068+0000 7ff3eb400640 1 -- 172.21.15.28:0/480125253 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff3ec0ca540 msgr2=0x7ff3d0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:28.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.068+0000 7ff3eb400640 1 --2- 172.21.15.28:0/480125253 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff3ec0ca540 0x7ff3d0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:28.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.068+0000 7ff3eb400640 1 -- 172.21.15.28:0/480125253 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3d0004d40 msgr2=0x7ff3d00051a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:28.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.068+0000 7ff3eb400640 1 --2- 172.21.15.28:0/480125253 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3d0004d40 0x7ff3d00051a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:28.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.068+0000 7ff3eb400640 1 -- 172.21.15.28:0/480125253 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3d0003520 con 0x7ff3d00039e0 2024-02-21T23:11:28.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.068+0000 7ff3ebe00640 1 --2- 172.21.15.28:0/480125253 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3d0004d40 0x7ff3d00051a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:11:28.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.069+0000 7ff3eb400640 1 --2- 172.21.15.28:0/480125253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3d00039e0 0x7ff3d0003e00 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7ff3dc001750 tx=0x7ff3d0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8c523362a049082 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:28.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.069+0000 7ff3ea000640 1 -- 172.21.15.28:0/480125253 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3dc001370 con 0x7ff3d00039e0 2024-02-21T23:11:28.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.069+0000 7ff3ea000640 1 -- 172.21.15.28:0/480125253 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3dc0052d0 con 0x7ff3d00039e0 2024-02-21T23:11:28.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.069+0000 7ff3ea000640 1 -- 172.21.15.28:0/480125253 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3dc003830 con 0x7ff3d00039e0 2024-02-21T23:11:28.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.070+0000 7ff3f0e00640 1 -- 172.21.15.28:0/480125253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3d00039e0 msgr2=0x7ff3d0003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:28.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.070+0000 7ff3f0e00640 1 --2- 172.21.15.28:0/480125253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3d00039e0 0x7ff3d0003e00 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7ff3dc001750 tx=0x7ff3d0005830 comp rx=0 tx=0).stop 2024-02-21T23:11:28.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.070+0000 7ff3ebe00640 1 -- 172.21.15.28:0/480125253 reap_dead start 2024-02-21T23:11:28.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.070+0000 7ff3f0e00640 1 -- 172.21.15.28:0/480125253 shutdown_connections 2024-02-21T23:11:28.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.070+0000 7ff3f0e00640 1 -- 172.21.15.28:0/480125253 >> 172.21.15.28:0/480125253 conn(0x7ff3ec0725a0 msgr2=0x7ff3ec0729b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:28.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.070+0000 7ff3f0e00640 1 -- 172.21.15.28:0/480125253 shutdown_connections 2024-02-21T23:11:28.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.070+0000 7ff3f0e00640 1 -- 172.21.15.28:0/480125253 wait complete. 2024-02-21T23:11:28.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.072+0000 7ff3f0e00640 1 Processor -- start 2024-02-21T23:11:28.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.072+0000 7ff3f0e00640 1 -- start start 2024-02-21T23:11:28.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.073+0000 7ff3f0e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3d00039e0 0x7ff3ec0aa4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:28.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.073+0000 7ff3f0e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3ec14afc0 0x7ff3ec1ede50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:28.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.073+0000 7ff3f0e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff3ec1530f0 0x7ff3ec1ee390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:28.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.073+0000 7ff3f0e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ec0c6c90 con 0x7ff3d00039e0 2024-02-21T23:11:28.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.073+0000 7ff3f0e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ec0c69b0 con 0x7ff3ec14afc0 2024-02-21T23:11:28.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.073+0000 7ff3f0e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ec1529d0 con 0x7ff3ec1530f0 2024-02-21T23:11:28.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.074+0000 7ff3eb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3d00039e0 0x7ff3ec0aa4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:28.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.074+0000 7ff3eb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3d00039e0 0x7ff3ec0aa4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41418/0 (socket says 172.21.15.28:41418) 2024-02-21T23:11:28.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.074+0000 7ff3eb400640 1 -- 172.21.15.28:0/1570737237 learned_addr learned my addr 172.21.15.28:0/1570737237 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:28.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.074+0000 7ff3eaa00640 1 --2- 172.21.15.28:0/1570737237 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3ec14afc0 0x7ff3ec1ede50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:28.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.074+0000 7ff3ebe00640 1 --2- 172.21.15.28:0/1570737237 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff3ec1530f0 0x7ff3ec1ee390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:28.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.074+0000 7ff3eb400640 1 -- 172.21.15.28:0/1570737237 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff3ec1530f0 msgr2=0x7ff3ec1ee390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:28.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.074+0000 7ff3eb400640 1 --2- 172.21.15.28:0/1570737237 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff3ec1530f0 0x7ff3ec1ee390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:28.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.074+0000 7ff3eb400640 1 -- 172.21.15.28:0/1570737237 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3ec14afc0 msgr2=0x7ff3ec1ede50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:28.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.074+0000 7ff3eb400640 1 --2- 172.21.15.28:0/1570737237 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3ec14afc0 0x7ff3ec1ede50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:28.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.074+0000 7ff3eb400640 1 -- 172.21.15.28:0/1570737237 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3ec14ace0 con 0x7ff3d00039e0 2024-02-21T23:11:28.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.074+0000 7ff3eaa00640 1 --2- 172.21.15.28:0/1570737237 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3ec14afc0 0x7ff3ec1ede50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:11:28.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.075+0000 7ff3eb400640 1 --2- 172.21.15.28:0/1570737237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3d00039e0 0x7ff3ec0aa4b0 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7ff3dc008fd0 tx=0x7ff3dc001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:28.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.075+0000 7ff3e3400640 1 -- 172.21.15.28:0/1570737237 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3dc041070 con 0x7ff3d00039e0 2024-02-21T23:11:28.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.075+0000 7ff3e3400640 1 -- 172.21.15.28:0/1570737237 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3dc005480 con 0x7ff3d00039e0 2024-02-21T23:11:28.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.075+0000 7ff3e3400640 1 -- 172.21.15.28:0/1570737237 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3dc007a30 con 0x7ff3d00039e0 2024-02-21T23:11:28.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.075+0000 7ff3f0e00640 1 -- 172.21.15.28:0/1570737237 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3ec14ebf0 con 0x7ff3d00039e0 2024-02-21T23:11:28.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.075+0000 7ff3f0e00640 1 -- 172.21.15.28:0/1570737237 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3ec0aa9f0 con 0x7ff3d00039e0 2024-02-21T23:11:28.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.077+0000 7ff3f0e00640 1 -- 172.21.15.28:0/1570737237 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3ec04f290 con 0x7ff3d00039e0 2024-02-21T23:11:28.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.078+0000 7ff3e3400640 1 -- 172.21.15.28:0/1570737237 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff3dc044070 con 0x7ff3d00039e0 2024-02-21T23:11:28.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.079+0000 7ff3e3400640 1 --2- 172.21.15.28:0/1570737237 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff3cc06b500 0x7ff3cc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:28.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.079+0000 7ff3e3400640 1 -- 172.21.15.28:0/1570737237 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 5836+0+0 (secure 0 0 0) 0x7ff3dc0abd00 con 0x7ff3d00039e0 2024-02-21T23:11:28.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.080+0000 7ff3eaa00640 1 --2- 172.21.15.28:0/1570737237 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff3cc06b500 0x7ff3cc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:28.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.081+0000 7ff3eaa00640 1 --2- 172.21.15.28:0/1570737237 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff3cc06b500 0x7ff3cc06d9c0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7ff3d8006ce0 tx=0x7ff3d8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:28.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.084+0000 7ff3e3400640 1 -- 172.21.15.28:0/1570737237 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ff3dc07ab00 con 0x7ff3d00039e0 2024-02-21T23:11:28.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.284+0000 7ff3f0e00640 1 -- 172.21.15.28:0/1570737237 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ff3ec0ab8c0 con 0x7ff3d00039e0 2024-02-21T23:11:28.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.834+0000 7ff3e3400640 1 -- 172.21.15.28:0/1570737237 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v178) v1 ==== 169+0+0 (secure 0 0 0) 0x7ff3dc07a400 con 0x7ff3d00039e0 2024-02-21T23:11:28.823 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' removed 2024-02-21T23:11:28.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.840+0000 7ff3f0e00640 1 -- 172.21.15.28:0/1570737237 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff3cc06b500 msgr2=0x7ff3cc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:28.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.840+0000 7ff3f0e00640 1 --2- 172.21.15.28:0/1570737237 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff3cc06b500 0x7ff3cc06d9c0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7ff3d8006ce0 tx=0x7ff3d8008040 comp rx=0 tx=0).stop 2024-02-21T23:11:28.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.840+0000 7ff3f0e00640 1 -- 172.21.15.28:0/1570737237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3d00039e0 msgr2=0x7ff3ec0aa4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:28.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.840+0000 7ff3f0e00640 1 --2- 172.21.15.28:0/1570737237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3d00039e0 0x7ff3ec0aa4b0 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7ff3dc008fd0 tx=0x7ff3dc001120 comp rx=0 tx=0).stop 2024-02-21T23:11:28.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.840+0000 7ff3f0e00640 1 -- 172.21.15.28:0/1570737237 shutdown_connections 2024-02-21T23:11:28.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.840+0000 7ff3f0e00640 1 --2- 172.21.15.28:0/1570737237 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff3cc06b500 0x7ff3cc06d9c0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:28.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.840+0000 7ff3f0e00640 1 --2- 172.21.15.28:0/1570737237 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff3ec1530f0 0x7ff3ec1ee390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:28.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.840+0000 7ff3f0e00640 1 --2- 172.21.15.28:0/1570737237 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3ec14afc0 0x7ff3ec1ede50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:28.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.840+0000 7ff3f0e00640 1 --2- 172.21.15.28:0/1570737237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3d00039e0 0x7ff3ec0aa4b0 unknown :-1 s=CLOSED pgs=2022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:28.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.840+0000 7ff3f0e00640 1 -- 172.21.15.28:0/1570737237 >> 172.21.15.28:0/1570737237 conn(0x7ff3ec0725a0 msgr2=0x7ff3ec1de2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:28.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.842+0000 7ff3f0e00640 1 -- 172.21.15.28:0/1570737237 shutdown_connections 2024-02-21T23:11:28.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:28.842+0000 7ff3f0e00640 1 -- 172.21.15.28:0/1570737237 wait complete. 2024-02-21T23:11:28.854 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-02-21T23:11:29.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.069+0000 7f4ce9e00640 1 Processor -- start 2024-02-21T23:11:29.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.069+0000 7f4ce9e00640 1 -- start start 2024-02-21T23:11:29.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.070+0000 7f4ce9e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4ce40c2c60 0x7f4ce406dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:29.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.070+0000 7f4ce9e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4ce406f400 0x7f4ce406e530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:29.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.070+0000 7f4ce9e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ce406db10 con 0x7f4ce41500f0 2024-02-21T23:11:29.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.070+0000 7f4ce9e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ce40c1080 con 0x7f4ce40c2c60 2024-02-21T23:11:29.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.070+0000 7f4ce9e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce41539a0 con 0x7f4ce406f400 2024-02-21T23:11:29.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.070+0000 7f4ce3e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4ce40c2c60 0x7f4ce406dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:29.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.071+0000 7f4ce3e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4ce40c2c60 0x7f4ce406dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41420/0 (socket says 172.21.15.28:41420) 2024-02-21T23:11:29.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.071+0000 7f4ce3e00640 1 -- 172.21.15.28:0/1044845411 learned_addr learned my addr 172.21.15.28:0/1044845411 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:29.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.071+0000 7f4ce9400640 1 --2- 172.21.15.28:0/1044845411 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4ce406f400 0x7f4ce406e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:29.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.071+0000 7f4ce3e00640 1 -- 172.21.15.28:0/1044845411 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4ce406f400 msgr2=0x7f4ce406e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:29.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.071+0000 7f4ce3e00640 1 --2- 172.21.15.28:0/1044845411 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4ce406f400 0x7f4ce406e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:29.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.071+0000 7f4ce3e00640 1 -- 172.21.15.28:0/1044845411 >> v1:172.21.15.81:6789/0 conn(0x7f4ce41500f0 legacy=0x7f4ce406ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:29.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.071+0000 7f4ce3e00640 1 -- 172.21.15.28:0/1044845411 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ce40c12d0 con 0x7f4ce40c2c60 2024-02-21T23:11:29.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.072+0000 7f4ce3e00640 1 --2- 172.21.15.28:0/1044845411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ce40c2c60 0x7f4ce406dff0 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 crypto rx=0x7f4cd0009870 tx=0x7f4cd002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be1202d628ab65a9 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:29.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.072+0000 7f4ce3400640 1 -- 172.21.15.28:0/1044845411 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cd0034070 con 0x7f4ce40c2c60 2024-02-21T23:11:29.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.073+0000 7f4ce3400640 1 -- 172.21.15.28:0/1044845411 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cd0030840 con 0x7f4ce40c2c60 2024-02-21T23:11:29.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.073+0000 7f4ce3400640 1 -- 172.21.15.28:0/1044845411 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cd0030c10 con 0x7f4ce40c2c60 2024-02-21T23:11:29.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.073+0000 7f4ce9e00640 1 -- 172.21.15.28:0/1044845411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ce40c2c60 msgr2=0x7f4ce406dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:29.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.073+0000 7f4ce9e00640 1 --2- 172.21.15.28:0/1044845411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ce40c2c60 0x7f4ce406dff0 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 crypto rx=0x7f4cd0009870 tx=0x7f4cd002b980 comp rx=0 tx=0).stop 2024-02-21T23:11:29.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.073+0000 7f4ce9e00640 1 -- 172.21.15.28:0/1044845411 shutdown_connections 2024-02-21T23:11:29.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.073+0000 7f4ce9e00640 1 --2- 172.21.15.28:0/1044845411 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4ce406f400 0x7f4ce406e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:29.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.073+0000 7f4ce9e00640 1 --2- 172.21.15.28:0/1044845411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ce40c2c60 0x7f4ce406dff0 unknown :-1 s=CLOSED pgs=2023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:29.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.073+0000 7f4ce9e00640 1 -- 172.21.15.28:0/1044845411 >> 172.21.15.28:0/1044845411 conn(0x7f4ce40a8530 msgr2=0x7f4ce40a8910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:29.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.073+0000 7f4ce9e00640 1 -- 172.21.15.28:0/1044845411 shutdown_connections 2024-02-21T23:11:29.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.073+0000 7f4ce9e00640 1 -- 172.21.15.28:0/1044845411 wait complete. 2024-02-21T23:11:29.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.075+0000 7f4ce9e00640 1 Processor -- start 2024-02-21T23:11:29.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.075+0000 7f4ce9e00640 1 -- start start 2024-02-21T23:11:29.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.076+0000 7f4ce9e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ce406f400 0x7f4ce41515e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:29.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.076+0000 7f4ce9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ce40c2c60 0x7f4ce41dddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:29.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.076+0000 7f4ce9e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4ce41500f0 0x7f4ce41de330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:29.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.076+0000 7f4ce9e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce406b580 con 0x7f4ce40c2c60 2024-02-21T23:11:29.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.076+0000 7f4ce9e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce41504d0 con 0x7f4ce406f400 2024-02-21T23:11:29.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.076+0000 7f4ce9e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce406b700 con 0x7f4ce41500f0 2024-02-21T23:11:29.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.076+0000 7f4ce3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ce40c2c60 0x7f4ce41dddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:29.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.076+0000 7f4ce3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ce40c2c60 0x7f4ce41dddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41422/0 (socket says 172.21.15.28:41422) 2024-02-21T23:11:29.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.076+0000 7f4ce3e00640 1 -- 172.21.15.28:0/4056650267 learned_addr learned my addr 172.21.15.28:0/4056650267 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:29.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.077+0000 7f4ce8a00640 1 --2- 172.21.15.28:0/4056650267 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ce406f400 0x7f4ce41515e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:29.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.077+0000 7f4ce9400640 1 --2- 172.21.15.28:0/4056650267 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4ce41500f0 0x7f4ce41de330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:29.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.077+0000 7f4ce3e00640 1 -- 172.21.15.28:0/4056650267 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4ce41500f0 msgr2=0x7f4ce41de330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:29.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.077+0000 7f4ce3e00640 1 --2- 172.21.15.28:0/4056650267 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4ce41500f0 0x7f4ce41de330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:29.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.077+0000 7f4ce3e00640 1 -- 172.21.15.28:0/4056650267 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ce406f400 msgr2=0x7f4ce41515e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:29.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.077+0000 7f4ce3e00640 1 --2- 172.21.15.28:0/4056650267 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ce406f400 0x7f4ce41515e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:29.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.077+0000 7f4ce3e00640 1 -- 172.21.15.28:0/4056650267 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ce41ebb90 con 0x7f4ce40c2c60 2024-02-21T23:11:29.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.077+0000 7f4ce3e00640 1 --2- 172.21.15.28:0/4056650267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ce40c2c60 0x7f4ce41dddf0 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7f4cd0000c00 tx=0x7f4cd0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:29.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.078+0000 7f4ce0c00640 1 -- 172.21.15.28:0/4056650267 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cd0048070 con 0x7f4ce40c2c60 2024-02-21T23:11:29.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.078+0000 7f4ce0c00640 1 -- 172.21.15.28:0/4056650267 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cd0030640 con 0x7f4ce40c2c60 2024-02-21T23:11:29.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.078+0000 7f4ce0c00640 1 -- 172.21.15.28:0/4056650267 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cd0034040 con 0x7f4ce40c2c60 2024-02-21T23:11:29.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.078+0000 7f4ce9e00640 1 -- 172.21.15.28:0/4056650267 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ce4150b00 con 0x7f4ce40c2c60 2024-02-21T23:11:29.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.078+0000 7f4ce9e00640 1 -- 172.21.15.28:0/4056650267 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ce41510c0 con 0x7f4ce40c2c60 2024-02-21T23:11:29.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.079+0000 7f4ce9e00640 1 -- 172.21.15.28:0/4056650267 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ce404f290 con 0x7f4ce40c2c60 2024-02-21T23:11:29.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.080+0000 7f4ce0c00640 1 -- 172.21.15.28:0/4056650267 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4cd004b070 con 0x7f4ce40c2c60 2024-02-21T23:11:29.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.081+0000 7f4ce0c00640 1 --2- 172.21.15.28:0/4056650267 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4cc406b520 0x7f4cc406d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:29.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.081+0000 7f4ce0c00640 1 -- 172.21.15.28:0/4056650267 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f4cd00ac0b0 con 0x7f4ce40c2c60 2024-02-21T23:11:29.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.082+0000 7f4ce8a00640 1 --2- 172.21.15.28:0/4056650267 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4cc406b520 0x7f4cc406d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:29.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.083+0000 7f4ce8a00640 1 --2- 172.21.15.28:0/4056650267 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4cc406b520 0x7f4cc406d9e0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f4cd8007fd0 tx=0x7f4cd8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:29.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.086+0000 7f4ce0c00640 1 -- 172.21.15.28:0/4056650267 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4cd007b110 con 0x7f4ce40c2c60 2024-02-21T23:11:29.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.285+0000 7f4ce9e00640 1 -- 172.21.15.28:0/4056650267 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f4ce406f1e0 con 0x7f4ce40c2c60 2024-02-21T23:11:29.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.290+0000 7f4ce0c00640 1 -- 172.21.15.28:0/4056650267 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v58) v1 ==== 104+0+0 (secure 0 0 0) 0x7f4cd007aa10 con 0x7f4ce40c2c60 2024-02-21T23:11:29.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.293+0000 7f4ce9e00640 1 -- 172.21.15.28:0/4056650267 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4cc406b520 msgr2=0x7f4cc406d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:29.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.293+0000 7f4ce9e00640 1 --2- 172.21.15.28:0/4056650267 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4cc406b520 0x7f4cc406d9e0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f4cd8007fd0 tx=0x7f4cd8009040 comp rx=0 tx=0).stop 2024-02-21T23:11:29.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.293+0000 7f4ce9e00640 1 -- 172.21.15.28:0/4056650267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ce40c2c60 msgr2=0x7f4ce41dddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:29.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.293+0000 7f4ce9e00640 1 --2- 172.21.15.28:0/4056650267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ce40c2c60 0x7f4ce41dddf0 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7f4cd0000c00 tx=0x7f4cd0004730 comp rx=0 tx=0).stop 2024-02-21T23:11:29.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.293+0000 7f4ce9e00640 1 -- 172.21.15.28:0/4056650267 shutdown_connections 2024-02-21T23:11:29.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.293+0000 7f4ce9e00640 1 --2- 172.21.15.28:0/4056650267 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4cc406b520 0x7f4cc406d9e0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:29.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.293+0000 7f4ce9e00640 1 --2- 172.21.15.28:0/4056650267 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4ce41500f0 0x7f4ce41de330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:29.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.293+0000 7f4ce9e00640 1 --2- 172.21.15.28:0/4056650267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ce40c2c60 0x7f4ce41dddf0 unknown :-1 s=CLOSED pgs=2024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:29.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.293+0000 7f4ce9e00640 1 --2- 172.21.15.28:0/4056650267 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ce406f400 0x7f4ce41515e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:29.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.293+0000 7f4ce9e00640 1 -- 172.21.15.28:0/4056650267 >> 172.21.15.28:0/4056650267 conn(0x7f4ce40a8530 msgr2=0x7f4ce406ae70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:29.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.293+0000 7f4ce9e00640 1 -- 172.21.15.28:0/4056650267 shutdown_connections 2024-02-21T23:11:29.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.293+0000 7f4ce9e00640 1 -- 172.21.15.28:0/4056650267 wait complete. 2024-02-21T23:11:29.300 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestClusterResize.test_down' 2024-02-21T23:11:29.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.515+0000 7f0125c00640 1 Processor -- start 2024-02-21T23:11:29.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.515+0000 7f0125c00640 1 -- start start 2024-02-21T23:11:29.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.517+0000 7f0125c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f01200c6b60 0x7f0120158930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:29.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.517+0000 7f0125c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01201529e0 con 0x7f0120152600 2024-02-21T23:11:29.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.517+0000 7f0125c00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01200c7200 con 0x7f01200c6190 2024-02-21T23:11:29.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.517+0000 7f0125c00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01200c74e0 con 0x7f01200c6b60 2024-02-21T23:11:29.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.517+0000 7f011fe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f0120152600 0x7f01200c0f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60964/0 (socket says 172.21.15.28:60964) 2024-02-21T23:11:29.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.517+0000 7f011fe00640 1 -- 172.21.15.28:0/3644137770 learned_addr learned my addr 172.21.15.28:0/3644137770 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:11:29.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.518+0000 7f0117400640 1 --2- 172.21.15.28:0/3644137770 >> v2:172.21.15.81:3300/0 conn(0x7f01200c6b60 0x7f0120158930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:29.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.518+0000 7f011f400640 1 -- 172.21.15.28:0/3644137770 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3855094378 0 0) 0x7f01201529e0 con 0x7f0120152600 2024-02-21T23:11:29.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.518+0000 7f011f400640 1 -- 172.21.15.28:0/3644137770 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0104003520 con 0x7f0120152600 2024-02-21T23:11:29.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.518+0000 7f011f400640 1 -- 172.21.15.28:0/3644137770 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2213843702 0 0) 0x7f0104003520 con 0x7f0120152600 2024-02-21T23:11:29.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.518+0000 7f011f400640 1 -- 172.21.15.28:0/3644137770 >> v2:172.21.15.81:3300/0 conn(0x7f01200c6b60 msgr2=0x7f0120158930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:29.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.518+0000 7f011f400640 1 --2- 172.21.15.28:0/3644137770 >> v2:172.21.15.81:3300/0 conn(0x7f01200c6b60 0x7f0120158930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:29.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.518+0000 7f011f400640 1 -- 172.21.15.28:0/3644137770 >> v1:172.21.15.81:6789/0 conn(0x7f01200c6190 legacy=0x7f0120158e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:29.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.518+0000 7f011f400640 1 -- 172.21.15.28:0/3644137770 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01200c77c0 con 0x7f0120152600 2024-02-21T23:11:29.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.518+0000 7f011f400640 1 -- 172.21.15.28:0/3644137770 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f0110001670 con 0x7f0120152600 2024-02-21T23:11:29.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.518+0000 7f011f400640 1 -- 172.21.15.28:0/3644137770 >> v1:172.21.15.28:6789/0 conn(0x7f0120152600 legacy=0x7f01200c0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:29.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.518+0000 7f011f400640 1 --2- 172.21.15.28:0/3644137770 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01040039b0 0x7f0104003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:29.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.519+0000 7f011f400640 1 --2- 172.21.15.28:0/3644137770 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0104004800 0x7f0104004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:29.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.519+0000 7f011f400640 1 --2- 172.21.15.28:0/3644137770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01040051c0 0x7f010400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:29.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.519+0000 7f011f400640 1 -- 172.21.15.28:0/3644137770 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01200c74e0 con 0x7f01040051c0 2024-02-21T23:11:29.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.519+0000 7f011f400640 1 -- 172.21.15.28:0/3644137770 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f010400c060 con 0x7f01040039b0 2024-02-21T23:11:29.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.519+0000 7f011f400640 1 -- 172.21.15.28:0/3644137770 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f010400c260 con 0x7f0104004800 2024-02-21T23:11:29.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.519+0000 7f0125200640 1 --2- 172.21.15.28:0/3644137770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01040051c0 0x7f010400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:29.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.519+0000 7f0117400640 1 --2- 172.21.15.28:0/3644137770 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0104004800 0x7f0104004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:29.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.519+0000 7f011fe00640 1 --2- 172.21.15.28:0/3644137770 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01040039b0 0x7f0104003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:29.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.519+0000 7f0125200640 1 -- 172.21.15.28:0/3644137770 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0104004800 msgr2=0x7f0104004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:29.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.519+0000 7f0125200640 1 --2- 172.21.15.28:0/3644137770 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0104004800 0x7f0104004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:29.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.519+0000 7f0125200640 1 -- 172.21.15.28:0/3644137770 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01040039b0 msgr2=0x7f0104003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:29.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.519+0000 7f0125200640 1 --2- 172.21.15.28:0/3644137770 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01040039b0 0x7f0104003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:29.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.519+0000 7f0125200640 1 -- 172.21.15.28:0/3644137770 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0104003520 con 0x7f01040051c0 2024-02-21T23:11:29.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.520+0000 7f0125200640 1 -- 172.21.15.28:0/3644137770 reap_dead start 2024-02-21T23:11:29.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.520+0000 7f0125200640 1 --2- 172.21.15.28:0/3644137770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01040051c0 0x7f010400b9c0 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7f0118001e50 tx=0x7f0104005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2aa7dd5abacbfab6 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:29.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.521+0000 7f011f400640 1 -- 172.21.15.28:0/3644137770 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0118001a70 con 0x7f01040051c0 2024-02-21T23:11:29.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.521+0000 7f011f400640 1 -- 172.21.15.28:0/3644137770 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0118003bb0 con 0x7f01040051c0 2024-02-21T23:11:29.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.521+0000 7f011f400640 1 -- 172.21.15.28:0/3644137770 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01180050d0 con 0x7f01040051c0 2024-02-21T23:11:29.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.521+0000 7f0125c00640 1 -- 172.21.15.28:0/3644137770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01040051c0 msgr2=0x7f010400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:29.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.521+0000 7f0125c00640 1 --2- 172.21.15.28:0/3644137770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01040051c0 0x7f010400b9c0 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7f0118001e50 tx=0x7f0104005830 comp rx=0 tx=0).stop 2024-02-21T23:11:29.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.521+0000 7f0125c00640 1 -- 172.21.15.28:0/3644137770 shutdown_connections 2024-02-21T23:11:29.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.521+0000 7f0125c00640 1 --2- 172.21.15.28:0/3644137770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01040051c0 0x7f010400b9c0 unknown :-1 s=CLOSED pgs=2026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:29.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.521+0000 7f0125c00640 1 -- 172.21.15.28:0/3644137770 >> 172.21.15.28:0/3644137770 conn(0x7f012006d1a0 msgr2=0x7f01200725a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:29.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.521+0000 7f0125c00640 1 -- 172.21.15.28:0/3644137770 shutdown_connections 2024-02-21T23:11:29.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.521+0000 7f0125c00640 1 -- 172.21.15.28:0/3644137770 wait complete. 2024-02-21T23:11:29.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.523+0000 7f0125c00640 1 Processor -- start 2024-02-21T23:11:29.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.523+0000 7f0125c00640 1 -- start start 2024-02-21T23:11:29.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.524+0000 7f0125c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f01040051c0 0x7f01201e0280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:29.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.524+0000 7f0125c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01201eeaa0 0x7f012014e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:29.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.524+0000 7f0125c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f012014d210 0x7f01201de6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:29.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.524+0000 7f0125c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0120155ae0 con 0x7f01201eeaa0 2024-02-21T23:11:29.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.524+0000 7f0125c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01200c7aa0 con 0x7f012014d210 2024-02-21T23:11:29.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.524+0000 7f0125c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0120155c60 con 0x7f01040051c0 2024-02-21T23:11:29.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.524+0000 7f0117400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01201eeaa0 0x7f012014e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:29.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.524+0000 7f0117400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01201eeaa0 0x7f012014e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41428/0 (socket says 172.21.15.28:41428) 2024-02-21T23:11:29.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.524+0000 7f0117400640 1 -- 172.21.15.28:0/1100115825 learned_addr learned my addr 172.21.15.28:0/1100115825 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:29.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.525+0000 7f011fe00640 1 --2- 172.21.15.28:0/1100115825 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f01040051c0 0x7f01201e0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:29.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.525+0000 7f0117400640 1 -- 172.21.15.28:0/1100115825 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f01040051c0 msgr2=0x7f01201e0280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:29.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.525+0000 7f0117400640 1 --2- 172.21.15.28:0/1100115825 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f01040051c0 0x7f01201e0280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:29.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.525+0000 7f0117400640 1 -- 172.21.15.28:0/1100115825 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f012014d210 msgr2=0x7f01201de6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:29.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.525+0000 7f0117400640 1 --2- 172.21.15.28:0/1100115825 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f012014d210 0x7f01201de6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:29.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.525+0000 7f0117400640 1 -- 172.21.15.28:0/1100115825 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01201e07c0 con 0x7f01201eeaa0 2024-02-21T23:11:29.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.525+0000 7f011fe00640 1 --2- 172.21.15.28:0/1100115825 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f01040051c0 0x7f01201e0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:11:29.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.525+0000 7f0117400640 1 --2- 172.21.15.28:0/1100115825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01201eeaa0 0x7f012014e710 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7f0108011590 tx=0x7f0108011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:29.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.525+0000 7f011cc00640 1 -- 172.21.15.28:0/1100115825 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0108019070 con 0x7f01201eeaa0 2024-02-21T23:11:29.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.526+0000 7f011cc00640 1 -- 172.21.15.28:0/1100115825 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0108003610 con 0x7f01201eeaa0 2024-02-21T23:11:29.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.526+0000 7f011cc00640 1 -- 172.21.15.28:0/1100115825 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01080039e0 con 0x7f01201eeaa0 2024-02-21T23:11:29.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.526+0000 7f0125c00640 1 -- 172.21.15.28:0/1100115825 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01201e19a0 con 0x7f01201eeaa0 2024-02-21T23:11:29.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.526+0000 7f0125c00640 1 -- 172.21.15.28:0/1100115825 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01201ee2a0 con 0x7f01201eeaa0 2024-02-21T23:11:29.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.527+0000 7f0125c00640 1 -- 172.21.15.28:0/1100115825 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f012004f290 con 0x7f01201eeaa0 2024-02-21T23:11:29.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.528+0000 7f011cc00640 1 -- 172.21.15.28:0/1100115825 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f010801c070 con 0x7f01201eeaa0 2024-02-21T23:11:29.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.529+0000 7f011cc00640 1 --2- 172.21.15.28:0/1100115825 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f00fc06b4d0 0x7f00fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:29.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.529+0000 7f011cc00640 1 -- 172.21.15.28:0/1100115825 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f010808a510 con 0x7f01201eeaa0 2024-02-21T23:11:29.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.529+0000 7f011fe00640 1 --2- 172.21.15.28:0/1100115825 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f00fc06b4d0 0x7f00fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:29.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.530+0000 7f011fe00640 1 --2- 172.21.15.28:0/1100115825 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f00fc06b4d0 0x7f00fc06d990 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f01100058b0 tx=0x7f0110001430 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:29.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.533+0000 7f011cc00640 1 -- 172.21.15.28:0/1100115825 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f0108059460 con 0x7f01201eeaa0 2024-02-21T23:11:29.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.806+0000 7f0125c00640 1 -- 172.21.15.28:0/1100115825 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterResize.test_down"]} v 0) v1 -- 0x7f01200c6630 con 0x7f01201eeaa0 2024-02-21T23:11:29.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.837+0000 7f011cc00640 1 -- 172.21.15.28:0/1100115825 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterResize.test_down"]}]=0 v509) v1 ==== 133+0+0 (secure 0 0 0) 0x7f0108058d60 con 0x7f01201eeaa0 2024-02-21T23:11:29.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.840+0000 7f0125c00640 1 -- 172.21.15.28:0/1100115825 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f00fc06b4d0 msgr2=0x7f00fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:29.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.840+0000 7f0125c00640 1 --2- 172.21.15.28:0/1100115825 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f00fc06b4d0 0x7f00fc06d990 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f01100058b0 tx=0x7f0110001430 comp rx=0 tx=0).stop 2024-02-21T23:11:29.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.840+0000 7f0125c00640 1 -- 172.21.15.28:0/1100115825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01201eeaa0 msgr2=0x7f012014e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:29.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.840+0000 7f0125c00640 1 --2- 172.21.15.28:0/1100115825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01201eeaa0 0x7f012014e710 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7f0108011590 tx=0x7f0108011a50 comp rx=0 tx=0).stop 2024-02-21T23:11:29.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.840+0000 7f0125c00640 1 -- 172.21.15.28:0/1100115825 shutdown_connections 2024-02-21T23:11:29.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.840+0000 7f0125c00640 1 --2- 172.21.15.28:0/1100115825 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f00fc06b4d0 0x7f00fc06d990 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:29.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.840+0000 7f0125c00640 1 --2- 172.21.15.28:0/1100115825 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f012014d210 0x7f01201de6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:29.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.840+0000 7f0125c00640 1 --2- 172.21.15.28:0/1100115825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01201eeaa0 0x7f012014e710 unknown :-1 s=CLOSED pgs=2027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:29.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.840+0000 7f0125c00640 1 --2- 172.21.15.28:0/1100115825 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f01040051c0 0x7f01201e0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:29.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.840+0000 7f0125c00640 1 -- 172.21.15.28:0/1100115825 >> 172.21.15.28:0/1100115825 conn(0x7f012006d1a0 msgr2=0x7f01200bfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:29.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.840+0000 7f0125c00640 1 -- 172.21.15.28:0/1100115825 shutdown_connections 2024-02-21T23:11:29.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:29.841+0000 7f0125c00640 1 -- 172.21.15.28:0/1100115825 wait complete. 2024-02-21T23:11:29.848 INFO:tasks.cephfs_test_runner:That down setting toggles and sets max_mds appropriately. ... ok 2024-02-21T23:11:29.849 INFO:tasks.cephfs_test_runner:Starting test: test_down_grow (tasks.cephfs.test_failover.TestClusterResize) That setting max_mds undoes down. 2024-02-21T23:11:29.849 INFO:tasks.cephfs_test_runner:test_down_grow (tasks.cephfs.test_failover.TestClusterResize) 2024-02-21T23:11:29.849 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:11:30.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.084+0000 7fdc31200640 1 Processor -- start 2024-02-21T23:11:30.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.084+0000 7fdc31200640 1 -- start start 2024-02-21T23:11:30.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.085+0000 7fdc31200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fdc2c155ca0 0x7fdc2c06cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:30.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.085+0000 7fdc31200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdc2c0c4ce0 0x7fdc2c06d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:30.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.085+0000 7fdc31200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc2c0c5e60 con 0x7fdc2c0af970 2024-02-21T23:11:30.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.085+0000 7fdc31200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc2c1561a0 con 0x7fdc2c155ca0 2024-02-21T23:11:30.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.085+0000 7fdc31200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2c0c6140 con 0x7fdc2c0c4ce0 2024-02-21T23:11:30.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.086+0000 7fdc2a800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdc2c0c4ce0 0x7fdc2c06d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:30.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.086+0000 7fdc29400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fdc2c155ca0 0x7fdc2c06cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:30.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.086+0000 7fdc29e00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fdc2c0af970 0x7fdc2c0c7350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.28:39464/0 (socket says 172.21.15.28:39464) 2024-02-21T23:11:30.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.086+0000 7fdc29e00640 1 -- 172.21.15.28:0/4114957364 learned_addr learned my addr 172.21.15.28:0/4114957364 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:11:30.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.087+0000 7fdc29400640 1 -- 172.21.15.28:0/4114957364 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdc2c0c4ce0 msgr2=0x7fdc2c06d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:30.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.087+0000 7fdc29400640 1 --2- 172.21.15.28:0/4114957364 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdc2c0c4ce0 0x7fdc2c06d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:30.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.087+0000 7fdc29400640 1 -- 172.21.15.28:0/4114957364 >> v1:172.21.15.81:6789/0 conn(0x7fdc2c0af970 legacy=0x7fdc2c0c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:30.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.087+0000 7fdc29400640 1 -- 172.21.15.28:0/4114957364 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc2c0c6420 con 0x7fdc2c155ca0 2024-02-21T23:11:30.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.089+0000 7fdc29400640 1 --2- 172.21.15.28:0/4114957364 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc2c155ca0 0x7fdc2c06cea0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fdc140097a0 tx=0x7fdc1402b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c9d1bcda2a478485 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:30.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.089+0000 7fdc28a00640 1 -- 172.21.15.28:0/4114957364 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc14039070 con 0x7fdc2c155ca0 2024-02-21T23:11:30.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.090+0000 7fdc28a00640 1 -- 172.21.15.28:0/4114957364 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc140316f0 con 0x7fdc2c155ca0 2024-02-21T23:11:30.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.090+0000 7fdc28a00640 1 -- 172.21.15.28:0/4114957364 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc14031ac0 con 0x7fdc2c155ca0 2024-02-21T23:11:30.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.090+0000 7fdc31200640 1 -- 172.21.15.28:0/4114957364 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc2c155ca0 msgr2=0x7fdc2c06cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:30.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.090+0000 7fdc31200640 1 --2- 172.21.15.28:0/4114957364 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc2c155ca0 0x7fdc2c06cea0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fdc140097a0 tx=0x7fdc1402b790 comp rx=0 tx=0).stop 2024-02-21T23:11:30.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.090+0000 7fdc31200640 1 -- 172.21.15.28:0/4114957364 shutdown_connections 2024-02-21T23:11:30.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.090+0000 7fdc31200640 1 --2- 172.21.15.28:0/4114957364 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdc2c0c4ce0 0x7fdc2c06d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:30.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.090+0000 7fdc31200640 1 --2- 172.21.15.28:0/4114957364 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc2c155ca0 0x7fdc2c06cea0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:30.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.090+0000 7fdc31200640 1 -- 172.21.15.28:0/4114957364 >> 172.21.15.28:0/4114957364 conn(0x7fdc2c0725a0 msgr2=0x7fdc2c0729b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:30.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.090+0000 7fdc31200640 1 -- 172.21.15.28:0/4114957364 shutdown_connections 2024-02-21T23:11:30.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.090+0000 7fdc31200640 1 -- 172.21.15.28:0/4114957364 wait complete. 2024-02-21T23:11:30.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.092+0000 7fdc31200640 1 Processor -- start 2024-02-21T23:11:30.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.092+0000 7fdc31200640 1 -- start start 2024-02-21T23:11:30.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.093+0000 7fdc31200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc2c0af970 0x7fdc2c0c9790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:30.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.093+0000 7fdc31200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc2c0c4ce0 0x7fdc2c14b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:30.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.093+0000 7fdc31200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdc2c155ca0 0x7fdc2c14bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:30.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.093+0000 7fdc31200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2c0c7be0 con 0x7fdc2c0c4ce0 2024-02-21T23:11:30.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.093+0000 7fdc31200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2c0c7a60 con 0x7fdc2c0af970 2024-02-21T23:11:30.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.093+0000 7fdc31200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2c0afd50 con 0x7fdc2c155ca0 2024-02-21T23:11:30.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.093+0000 7fdc29400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc2c0c4ce0 0x7fdc2c14b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:30.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.093+0000 7fdc29400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc2c0c4ce0 0x7fdc2c14b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41442/0 (socket says 172.21.15.28:41442) 2024-02-21T23:11:30.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.093+0000 7fdc29400640 1 -- 172.21.15.28:0/1095141060 learned_addr learned my addr 172.21.15.28:0/1095141060 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:30.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.094+0000 7fdc29e00640 1 --2- 172.21.15.28:0/1095141060 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc2c0af970 0x7fdc2c0c9790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:30.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.094+0000 7fdc2a800640 1 --2- 172.21.15.28:0/1095141060 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdc2c155ca0 0x7fdc2c14bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:30.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.094+0000 7fdc29400640 1 -- 172.21.15.28:0/1095141060 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdc2c155ca0 msgr2=0x7fdc2c14bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:30.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.094+0000 7fdc29400640 1 --2- 172.21.15.28:0/1095141060 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdc2c155ca0 0x7fdc2c14bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:30.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.094+0000 7fdc29400640 1 -- 172.21.15.28:0/1095141060 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc2c0af970 msgr2=0x7fdc2c0c9790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:30.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.094+0000 7fdc29400640 1 --2- 172.21.15.28:0/1095141060 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc2c0af970 0x7fdc2c0c9790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:30.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.094+0000 7fdc29400640 1 -- 172.21.15.28:0/1095141060 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc2c14c300 con 0x7fdc2c0c4ce0 2024-02-21T23:11:30.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.094+0000 7fdc2a800640 1 --2- 172.21.15.28:0/1095141060 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdc2c155ca0 0x7fdc2c14bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:11:30.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.094+0000 7fdc29e00640 1 --2- 172.21.15.28:0/1095141060 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc2c0af970 0x7fdc2c0c9790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:11:30.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.094+0000 7fdc29400640 1 --2- 172.21.15.28:0/1095141060 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc2c0c4ce0 0x7fdc2c14b7a0 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7fdc140098c0 tx=0x7fdc140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:30.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.095+0000 7fdc22000640 1 -- 172.21.15.28:0/1095141060 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc14043070 con 0x7fdc2c0c4ce0 2024-02-21T23:11:30.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.095+0000 7fdc22000640 1 -- 172.21.15.28:0/1095141060 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc14034480 con 0x7fdc2c0c4ce0 2024-02-21T23:11:30.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.095+0000 7fdc31200640 1 -- 172.21.15.28:0/1095141060 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc2c14d750 con 0x7fdc2c0c4ce0 2024-02-21T23:11:30.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.095+0000 7fdc22000640 1 -- 172.21.15.28:0/1095141060 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc14039040 con 0x7fdc2c0c4ce0 2024-02-21T23:11:30.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.095+0000 7fdc31200640 1 -- 172.21.15.28:0/1095141060 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc2c14dd10 con 0x7fdc2c0c4ce0 2024-02-21T23:11:30.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.096+0000 7fdc31200640 1 -- 172.21.15.28:0/1095141060 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc2c14e1d0 con 0x7fdc2c0c4ce0 2024-02-21T23:11:30.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.097+0000 7fdc22000640 1 -- 172.21.15.28:0/1095141060 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdc1403c070 con 0x7fdc2c0c4ce0 2024-02-21T23:11:30.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.098+0000 7fdc22000640 1 --2- 172.21.15.28:0/1095141060 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdc0806b4d0 0x7fdc0806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:30.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.098+0000 7fdc22000640 1 -- 172.21.15.28:0/1095141060 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 5335+0+0 (secure 0 0 0) 0x7fdc140ab7c0 con 0x7fdc2c0c4ce0 2024-02-21T23:11:30.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.098+0000 7fdc29e00640 1 --2- 172.21.15.28:0/1095141060 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdc0806b4d0 0x7fdc0806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:30.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.099+0000 7fdc29e00640 1 --2- 172.21.15.28:0/1095141060 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdc0806b4d0 0x7fdc0806d990 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c000f60 tx=0x7fdc1c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:30.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.102+0000 7fdc22000640 1 -- 172.21.15.28:0/1095141060 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fdc2c14e1d0 con 0x7fdc2c0c4ce0 2024-02-21T23:11:30.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.303+0000 7fdc31200640 1 -- 172.21.15.28:0/1095141060 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdc2c068640 con 0x7fdc2c0c4ce0 2024-02-21T23:11:30.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.305+0000 7fdc22000640 1 -- 172.21.15.28:0/1095141060 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v178) v1 ==== 74+0+13243 (secure 0 0 0) 0x7fdc14080020 con 0x7fdc2c0c4ce0 2024-02-21T23:11:30.293 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:11:30.293 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":178,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:11:28.831346+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6833/2364631414":"2024-02-22T23:11:25.807341+0000","172.21.15.28:6832/2364631414":"2024-02-22T23:11:25.807341+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:11:30.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.308+0000 7fdc31200640 1 -- 172.21.15.28:0/1095141060 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdc0806b4d0 msgr2=0x7fdc0806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:30.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.308+0000 7fdc31200640 1 --2- 172.21.15.28:0/1095141060 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdc0806b4d0 0x7fdc0806d990 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c000f60 tx=0x7fdc1c009040 comp rx=0 tx=0).stop 2024-02-21T23:11:30.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.308+0000 7fdc31200640 1 -- 172.21.15.28:0/1095141060 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc2c0c4ce0 msgr2=0x7fdc2c14b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:30.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.308+0000 7fdc31200640 1 --2- 172.21.15.28:0/1095141060 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc2c0c4ce0 0x7fdc2c14b7a0 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7fdc140098c0 tx=0x7fdc140046c0 comp rx=0 tx=0).stop 2024-02-21T23:11:30.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.308+0000 7fdc31200640 1 -- 172.21.15.28:0/1095141060 shutdown_connections 2024-02-21T23:11:30.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.308+0000 7fdc31200640 1 --2- 172.21.15.28:0/1095141060 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdc0806b4d0 0x7fdc0806d990 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:30.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.308+0000 7fdc31200640 1 --2- 172.21.15.28:0/1095141060 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdc2c155ca0 0x7fdc2c14bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:30.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.308+0000 7fdc31200640 1 --2- 172.21.15.28:0/1095141060 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc2c0c4ce0 0x7fdc2c14b7a0 unknown :-1 s=CLOSED pgs=2028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:30.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.308+0000 7fdc31200640 1 --2- 172.21.15.28:0/1095141060 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc2c0af970 0x7fdc2c0c9790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:30.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.308+0000 7fdc31200640 1 -- 172.21.15.28:0/1095141060 >> 172.21.15.28:0/1095141060 conn(0x7fdc2c0725a0 msgr2=0x7fdc2c1552e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:30.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.308+0000 7fdc31200640 1 -- 172.21.15.28:0/1095141060 shutdown_connections 2024-02-21T23:11:30.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.308+0000 7fdc31200640 1 -- 172.21.15.28:0/1095141060 wait complete. 2024-02-21T23:11:30.315 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-21T23:02:30.728329+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-21T23:11:30.315 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-21T23:11:30.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.534+0000 7f7acca00640 1 Processor -- start 2024-02-21T23:11:30.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.534+0000 7f7acca00640 1 -- start start 2024-02-21T23:11:30.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.536+0000 7f7acca00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7ac8158480 0x7f7ac8154f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:30.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.536+0000 7f7acca00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f7ac806e9d0 0x7f7ac806ee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:30.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.536+0000 7f7acca00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ac806f3a0 con 0x7f7ac8152560 2024-02-21T23:11:30.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.536+0000 7f7acca00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ac81554a0 con 0x7f7ac806e9d0 2024-02-21T23:11:30.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.536+0000 7f7acca00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac80a6270 con 0x7f7ac8158480 2024-02-21T23:11:30.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.536+0000 7f7ac7400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f7ac8152560 0x7f7ac806ae70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60972/0 (socket says 172.21.15.28:60972) 2024-02-21T23:11:30.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.536+0000 7f7ac7400640 1 -- 172.21.15.28:0/1345209125 learned_addr learned my addr 172.21.15.28:0/1345209125 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:11:30.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.537+0000 7f7ac7e00640 1 --2- 172.21.15.28:0/1345209125 >> v2:172.21.15.81:3300/0 conn(0x7f7ac806e9d0 0x7f7ac806ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:30.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.537+0000 7f7ac6a00640 1 --2- 172.21.15.28:0/1345209125 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7ac8158480 0x7f7ac8154f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:30.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.537+0000 7f7ac6000640 1 -- 172.21.15.28:0/1345209125 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1281844640 0 0) 0x7f7ac806f3a0 con 0x7f7ac8152560 2024-02-21T23:11:30.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.537+0000 7f7ac6000640 1 -- 172.21.15.28:0/1345209125 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7aac003520 con 0x7f7ac8152560 2024-02-21T23:11:30.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.537+0000 7f7ac6000640 1 -- 172.21.15.28:0/1345209125 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1879539783 0 0) 0x7f7aac003520 con 0x7f7ac8152560 2024-02-21T23:11:30.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.537+0000 7f7ac6000640 1 -- 172.21.15.28:0/1345209125 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7ac8158480 msgr2=0x7f7ac8154f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:30.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.537+0000 7f7ac6000640 1 --2- 172.21.15.28:0/1345209125 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7ac8158480 0x7f7ac8154f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:30.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.537+0000 7f7ac6000640 1 -- 172.21.15.28:0/1345209125 >> v2:172.21.15.81:3300/0 conn(0x7f7ac806e9d0 msgr2=0x7f7ac806ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:30.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.537+0000 7f7ac6000640 1 --2- 172.21.15.28:0/1345209125 >> v2:172.21.15.81:3300/0 conn(0x7f7ac806e9d0 0x7f7ac806ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:30.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.537+0000 7f7ac6000640 1 -- 172.21.15.28:0/1345209125 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ac81556d0 con 0x7f7ac8152560 2024-02-21T23:11:30.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.537+0000 7f7ac6000640 1 -- 172.21.15.28:0/1345209125 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f7ab8001670 con 0x7f7ac8152560 2024-02-21T23:11:30.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.537+0000 7f7ac6000640 1 -- 172.21.15.28:0/1345209125 >> v1:172.21.15.28:6789/0 conn(0x7f7ac8152560 legacy=0x7f7ac806ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:30.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.537+0000 7f7ac6000640 1 --2- 172.21.15.28:0/1345209125 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7ac8158480 0x7f7aac0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:30.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.537+0000 7f7ac6000640 1 --2- 172.21.15.28:0/1345209125 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7aac003f80 0x7f7aac004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:30.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.537+0000 7f7ac6000640 1 --2- 172.21.15.28:0/1345209125 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7aac004d40 0x7f7aac0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:30.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.537+0000 7f7ac6000640 1 -- 172.21.15.28:0/1345209125 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac80a6270 con 0x7f7aac004d40 2024-02-21T23:11:30.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.537+0000 7f7ac6000640 1 -- 172.21.15.28:0/1345209125 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aac00b9c0 con 0x7f7aac003f80 2024-02-21T23:11:30.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.538+0000 7f7ac6000640 1 -- 172.21.15.28:0/1345209125 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7aac00bb40 con 0x7f7ac8158480 2024-02-21T23:11:30.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.538+0000 7f7ac7e00640 1 --2- 172.21.15.28:0/1345209125 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7aac004d40 0x7f7aac0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:30.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.538+0000 7f7ac6a00640 1 --2- 172.21.15.28:0/1345209125 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7aac003f80 0x7f7aac004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:30.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.538+0000 7f7ac7e00640 1 -- 172.21.15.28:0/1345209125 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7ac8158480 msgr2=0x7f7aac0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:30.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.538+0000 7f7ac7e00640 1 --2- 172.21.15.28:0/1345209125 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7ac8158480 0x7f7aac0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:30.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.538+0000 7f7ac7e00640 1 -- 172.21.15.28:0/1345209125 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7aac003f80 msgr2=0x7f7aac004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:30.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.538+0000 7f7ac7e00640 1 --2- 172.21.15.28:0/1345209125 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7aac003f80 0x7f7aac004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:30.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.538+0000 7f7ac7e00640 1 -- 172.21.15.28:0/1345209125 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aac003520 con 0x7f7aac004d40 2024-02-21T23:11:30.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.539+0000 7f7ac7e00640 1 --2- 172.21.15.28:0/1345209125 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7aac004d40 0x7f7aac0051c0 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7f7ac000e170 tx=0x7f7aac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe3a20443b543836 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:30.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.539+0000 7f7ac6000640 1 -- 172.21.15.28:0/1345209125 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ac0039070 con 0x7f7aac004d40 2024-02-21T23:11:30.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.539+0000 7f7ac6000640 1 -- 172.21.15.28:0/1345209125 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ac002cc20 con 0x7f7aac004d40 2024-02-21T23:11:30.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.540+0000 7f7ac6000640 1 -- 172.21.15.28:0/1345209125 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ac0009c10 con 0x7f7aac004d40 2024-02-21T23:11:30.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.540+0000 7f7acca00640 1 -- 172.21.15.28:0/1345209125 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7aac004d40 msgr2=0x7f7aac0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:30.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.540+0000 7f7acca00640 1 --2- 172.21.15.28:0/1345209125 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7aac004d40 0x7f7aac0051c0 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7f7ac000e170 tx=0x7f7aac005830 comp rx=0 tx=0).stop 2024-02-21T23:11:30.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.540+0000 7f7ac7e00640 1 -- 172.21.15.28:0/1345209125 reap_dead start 2024-02-21T23:11:30.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.540+0000 7f7acca00640 1 -- 172.21.15.28:0/1345209125 shutdown_connections 2024-02-21T23:11:30.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.540+0000 7f7acca00640 1 -- 172.21.15.28:0/1345209125 >> 172.21.15.28:0/1345209125 conn(0x7f7ac80a8530 msgr2=0x7f7ac80a8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:30.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.540+0000 7f7acca00640 1 -- 172.21.15.28:0/1345209125 shutdown_connections 2024-02-21T23:11:30.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.540+0000 7f7acca00640 1 -- 172.21.15.28:0/1345209125 wait complete. 2024-02-21T23:11:30.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.542+0000 7f7acca00640 1 Processor -- start 2024-02-21T23:11:30.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.542+0000 7f7acca00640 1 -- start start 2024-02-21T23:11:30.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.543+0000 7f7acca00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7aac004d40 0x7f7ac80c3400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:30.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.543+0000 7f7acca00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ac8152dd0 0x7f7ac814a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:30.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.543+0000 7f7acca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ac8072ef0 0x7f7ac81ea490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:30.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.543+0000 7f7acca00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac806e4b0 con 0x7f7ac8072ef0 2024-02-21T23:11:30.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.543+0000 7f7acca00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac8155c90 con 0x7f7ac8152dd0 2024-02-21T23:11:30.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.543+0000 7f7acca00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac8152940 con 0x7f7aac004d40 2024-02-21T23:11:30.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.543+0000 7f7ac7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ac8072ef0 0x7f7ac81ea490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:30.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.543+0000 7f7ac7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ac8072ef0 0x7f7ac81ea490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41458/0 (socket says 172.21.15.28:41458) 2024-02-21T23:11:30.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.543+0000 7f7ac7e00640 1 -- 172.21.15.28:0/4166588687 learned_addr learned my addr 172.21.15.28:0/4166588687 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:30.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.543+0000 7f7ac6a00640 1 --2- 172.21.15.28:0/4166588687 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ac8152dd0 0x7f7ac814a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:30.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.543+0000 7f7ac7400640 1 --2- 172.21.15.28:0/4166588687 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7aac004d40 0x7f7ac80c3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:30.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.543+0000 7f7ac7e00640 1 -- 172.21.15.28:0/4166588687 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7aac004d40 msgr2=0x7f7ac80c3400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:30.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.543+0000 7f7ac7e00640 1 --2- 172.21.15.28:0/4166588687 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7aac004d40 0x7f7ac80c3400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:30.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.544+0000 7f7ac7e00640 1 -- 172.21.15.28:0/4166588687 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ac8152dd0 msgr2=0x7f7ac814a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:30.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.544+0000 7f7ac7e00640 1 --2- 172.21.15.28:0/4166588687 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ac8152dd0 0x7f7ac814a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:30.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.544+0000 7f7ac7e00640 1 -- 172.21.15.28:0/4166588687 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ac80aad20 con 0x7f7ac8072ef0 2024-02-21T23:11:30.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.544+0000 7f7ac7e00640 1 --2- 172.21.15.28:0/4166588687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ac8072ef0 0x7f7ac81ea490 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7f7ac000e170 tx=0x7f7ac0009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:30.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.544+0000 7f7abf400640 1 -- 172.21.15.28:0/4166588687 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ac0039070 con 0x7f7ac8072ef0 2024-02-21T23:11:30.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.544+0000 7f7abf400640 1 -- 172.21.15.28:0/4166588687 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ac003a040 con 0x7f7ac8072ef0 2024-02-21T23:11:30.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.544+0000 7f7abf400640 1 -- 172.21.15.28:0/4166588687 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ac0003990 con 0x7f7ac8072ef0 2024-02-21T23:11:30.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.544+0000 7f7acca00640 1 -- 172.21.15.28:0/4166588687 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ac80ab000 con 0x7f7ac8072ef0 2024-02-21T23:11:30.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.544+0000 7f7acca00640 1 -- 172.21.15.28:0/4166588687 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ac80ab5c0 con 0x7f7ac8072ef0 2024-02-21T23:11:30.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.546+0000 7f7acca00640 1 -- 172.21.15.28:0/4166588687 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ac804f290 con 0x7f7ac8072ef0 2024-02-21T23:11:30.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.546+0000 7f7abf400640 1 -- 172.21.15.28:0/4166588687 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7ac0046020 con 0x7f7ac8072ef0 2024-02-21T23:11:30.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.548+0000 7f7abf400640 1 --2- 172.21.15.28:0/4166588687 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7aa806b4d0 0x7f7aa806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:30.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.548+0000 7f7abf400640 1 -- 172.21.15.28:0/4166588687 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f7ac00abcd0 con 0x7f7ac8072ef0 2024-02-21T23:11:30.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.548+0000 7f7ac7400640 1 --2- 172.21.15.28:0/4166588687 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7aa806b4d0 0x7f7aa806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:30.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.549+0000 7f7ac7400640 1 --2- 172.21.15.28:0/4166588687 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7aa806b4d0 0x7f7aa806d990 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f7ab80017e0 tx=0x7f7ab8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:30.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.552+0000 7f7abf400640 1 -- 172.21.15.28:0/4166588687 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7ac007acb0 con 0x7f7ac8072ef0 2024-02-21T23:11:30.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.750+0000 7f7acca00640 1 -- 172.21.15.28:0/4166588687 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7ac806e0f0 con 0x7f7ac8072ef0 2024-02-21T23:11:30.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.751+0000 7f7abf400640 1 -- 172.21.15.28:0/4166588687 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v178) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7ac007a5b0 con 0x7f7ac8072ef0 2024-02-21T23:11:30.739 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-02-21T23:11:30.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.754+0000 7f7acca00640 1 -- 172.21.15.28:0/4166588687 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7aa806b4d0 msgr2=0x7f7aa806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:30.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.754+0000 7f7acca00640 1 --2- 172.21.15.28:0/4166588687 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7aa806b4d0 0x7f7aa806d990 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f7ab80017e0 tx=0x7f7ab8009040 comp rx=0 tx=0).stop 2024-02-21T23:11:30.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.754+0000 7f7acca00640 1 -- 172.21.15.28:0/4166588687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ac8072ef0 msgr2=0x7f7ac81ea490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:30.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.754+0000 7f7acca00640 1 --2- 172.21.15.28:0/4166588687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ac8072ef0 0x7f7ac81ea490 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7f7ac000e170 tx=0x7f7ac0009750 comp rx=0 tx=0).stop 2024-02-21T23:11:30.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.754+0000 7f7acca00640 1 -- 172.21.15.28:0/4166588687 shutdown_connections 2024-02-21T23:11:30.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.754+0000 7f7acca00640 1 --2- 172.21.15.28:0/4166588687 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7aa806b4d0 0x7f7aa806d990 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:30.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.754+0000 7f7acca00640 1 --2- 172.21.15.28:0/4166588687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ac8072ef0 0x7f7ac81ea490 unknown :-1 s=CLOSED pgs=2031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:30.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.754+0000 7f7acca00640 1 --2- 172.21.15.28:0/4166588687 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7ac8152dd0 0x7f7ac814a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:30.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.754+0000 7f7acca00640 1 --2- 172.21.15.28:0/4166588687 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7aac004d40 0x7f7ac80c3400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:30.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.754+0000 7f7acca00640 1 -- 172.21.15.28:0/4166588687 >> 172.21.15.28:0/4166588687 conn(0x7f7ac80a8530 msgr2=0x7f7ac80a6a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:30.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.754+0000 7f7acca00640 1 -- 172.21.15.28:0/4166588687 shutdown_connections 2024-02-21T23:11:30.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.754+0000 7f7acca00640 1 -- 172.21.15.28:0/4166588687 wait complete. 2024-02-21T23:11:30.762 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestClusterResize.test_down_grow' 2024-02-21T23:11:30.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.997+0000 7f6af5200640 1 Processor -- start 2024-02-21T23:11:30.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.998+0000 7f6af5200640 1 -- start start 2024-02-21T23:11:30.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.999+0000 7f6af5200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6af00a6160 0x7f6af0155180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:30.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.999+0000 7f6af5200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6af01521c0 0x7f6af0152650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:30.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.999+0000 7f6af5200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6af0155760 con 0x7f6af014f0b0 2024-02-21T23:11:30.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.999+0000 7f6af5200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6af0151d30 con 0x7f6af01521c0 2024-02-21T23:11:30.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:30.999+0000 7f6af5200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6af0155a40 con 0x7f6af00a6160 2024-02-21T23:11:30.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:31.000+0000 7f6aefe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6af01521c0 0x7f6af0152650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:30.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:31.000+0000 7f6aefe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6af01521c0 0x7f6af0152650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41472/0 (socket says 172.21.15.28:41472) 2024-02-21T23:11:30.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:31.000+0000 7f6aefe00640 1 -- 172.21.15.28:0/2748143398 learned_addr learned my addr 172.21.15.28:0/2748143398 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:30.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:31.000+0000 7f6ae7e00640 1 --2- 172.21.15.28:0/2748143398 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6af00a6160 0x7f6af0155180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:30.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:31.000+0000 7f6aefe00640 1 -- 172.21.15.28:0/2748143398 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6af00a6160 msgr2=0x7f6af0155180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:30.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:31.000+0000 7f6aefe00640 1 --2- 172.21.15.28:0/2748143398 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6af00a6160 0x7f6af0155180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:30.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:31.000+0000 7f6aefe00640 1 -- 172.21.15.28:0/2748143398 >> v1:172.21.15.81:6789/0 conn(0x7f6af014f0b0 legacy=0x7f6af00c2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:30.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:31.000+0000 7f6aefe00640 1 -- 172.21.15.28:0/2748143398 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6af0155d20 con 0x7f6af01521c0 2024-02-21T23:11:30.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:31.001+0000 7f6aefe00640 1 --2- 172.21.15.28:0/2748143398 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6af01521c0 0x7f6af0152650 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7f6ae8009e50 tx=0x7f6ae802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4509067a92eddc52 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:30.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:31.002+0000 7f6aeea00640 1 -- 172.21.15.28:0/2748143398 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ae802ca70 con 0x7f6af01521c0 2024-02-21T23:11:30.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:31.002+0000 7f6aeea00640 1 -- 172.21.15.28:0/2748143398 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ae802cc10 con 0x7f6af01521c0 2024-02-21T23:11:30.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:31.002+0000 7f6aeea00640 1 -- 172.21.15.28:0/2748143398 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ae80343f0 con 0x7f6af01521c0 2024-02-21T23:11:30.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:31.002+0000 7f6af5200640 1 -- 172.21.15.28:0/2748143398 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6af01521c0 msgr2=0x7f6af0152650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:30.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:31.002+0000 7f6af5200640 1 --2- 172.21.15.28:0/2748143398 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6af01521c0 0x7f6af0152650 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7f6ae8009e50 tx=0x7f6ae802bed0 comp rx=0 tx=0).stop 2024-02-21T23:11:30.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:31.002+0000 7f6af5200640 1 -- 172.21.15.28:0/2748143398 shutdown_connections 2024-02-21T23:11:30.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:31.002+0000 7f6af5200640 1 --2- 172.21.15.28:0/2748143398 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6af01521c0 0x7f6af0152650 unknown :-1 s=CLOSED pgs=2032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:30.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:31.002+0000 7f6af5200640 1 --2- 172.21.15.28:0/2748143398 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6af00a6160 0x7f6af0155180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:30.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:31.002+0000 7f6af5200640 1 -- 172.21.15.28:0/2748143398 >> 172.21.15.28:0/2748143398 conn(0x7f6af006b630 msgr2=0x7f6af00c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:30.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:31.003+0000 7f6af5200640 1 -- 172.21.15.28:0/2748143398 shutdown_connections 2024-02-21T23:11:30.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:31.003+0000 7f6af5200640 1 -- 172.21.15.28:0/2748143398 wait complete. 2024-02-21T23:11:30.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:31.004+0000 7f6af5200640 1 Processor -- start 2024-02-21T23:11:30.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:31.004+0000 7f6af5200640 1 -- start start 2024-02-21T23:11:30.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:31.005+0000 7f6af5200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6af00a6160 0x7f6af00c5960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:30.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:31.005+0000 7f6af5200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6af014f0b0 0x7f6af00c4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:30.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:31.005+0000 7f6af5200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6af01521c0 0x7f6af0148880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:30.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:31.005+0000 7f6af5200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6af00c2720 con 0x7f6af01521c0 2024-02-21T23:11:30.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:31.005+0000 7f6af5200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6af00a68d0 con 0x7f6af014f0b0 2024-02-21T23:11:30.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:31.005+0000 7f6af5200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6af00c28a0 con 0x7f6af00a6160 2024-02-21T23:11:30.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:31.006+0000 7f6aefe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6af01521c0 0x7f6af0148880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:30.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:31.006+0000 7f6aefe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6af01521c0 0x7f6af0148880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41484/0 (socket says 172.21.15.28:41484) 2024-02-21T23:11:30.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:31.006+0000 7f6aefe00640 1 -- 172.21.15.28:0/2898810011 learned_addr learned my addr 172.21.15.28:0/2898810011 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:30.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:31.006+0000 7f6aefe00640 1 -- 172.21.15.28:0/2898810011 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6af00a6160 msgr2=0x7f6af00c5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:30.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:31.006+0000 7f6aefe00640 1 --2- 172.21.15.28:0/2898810011 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6af00a6160 0x7f6af00c5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:30.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:31.006+0000 7f6aefe00640 1 -- 172.21.15.28:0/2898810011 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6af014f0b0 msgr2=0x7f6af00c4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:30.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:31.006+0000 7f6aefe00640 1 --2- 172.21.15.28:0/2898810011 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6af014f0b0 0x7f6af00c4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:30.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:31.006+0000 7f6aefe00640 1 -- 172.21.15.28:0/2898810011 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6af014e1e0 con 0x7f6af01521c0 2024-02-21T23:11:30.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:31.006+0000 7f6aefe00640 1 --2- 172.21.15.28:0/2898810011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6af01521c0 0x7f6af0148880 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 crypto rx=0x7f6ae8009e50 tx=0x7f6ae8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:30.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:31.007+0000 7f6ae7400640 1 -- 172.21.15.28:0/2898810011 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ae8043020 con 0x7f6af01521c0 2024-02-21T23:11:30.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:31.007+0000 7f6ae7400640 1 -- 172.21.15.28:0/2898810011 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ae803e450 con 0x7f6af01521c0 2024-02-21T23:11:30.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:31.007+0000 7f6ae7400640 1 -- 172.21.15.28:0/2898810011 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ae8009510 con 0x7f6af01521c0 2024-02-21T23:11:30.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:31.007+0000 7f6af5200640 1 -- 172.21.15.28:0/2898810011 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6af014d940 con 0x7f6af01521c0 2024-02-21T23:11:30.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:31.007+0000 7f6af5200640 1 -- 172.21.15.28:0/2898810011 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6af014df00 con 0x7f6af01521c0 2024-02-21T23:11:30.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:31.008+0000 7f6af5200640 1 -- 172.21.15.28:0/2898810011 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6af004f290 con 0x7f6af01521c0 2024-02-21T23:11:30.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:31.009+0000 7f6ae7400640 1 -- 172.21.15.28:0/2898810011 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6ae8046020 con 0x7f6af01521c0 2024-02-21T23:11:30.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:31.010+0000 7f6ae7400640 1 --2- 172.21.15.28:0/2898810011 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6acc06b4d0 0x7f6acc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:30.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:31.010+0000 7f6ae7400640 1 -- 172.21.15.28:0/2898810011 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f6ae80aca10 con 0x7f6af01521c0 2024-02-21T23:11:30.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:31.010+0000 7f6aef400640 1 --2- 172.21.15.28:0/2898810011 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6acc06b4d0 0x7f6acc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:30.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:31.011+0000 7f6aef400640 1 --2- 172.21.15.28:0/2898810011 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6acc06b4d0 0x7f6acc06d990 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f6ae0003cc0 tx=0x7f6ae00079b0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:31.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:31.014+0000 7f6ae7400640 1 -- 172.21.15.28:0/2898810011 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f6ae804d030 con 0x7f6af01521c0 2024-02-21T23:11:31.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:31.277+0000 7f6af5200640 1 -- 172.21.15.28:0/2898810011 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterResize.test_down_grow"]} v 0) v1 -- 0x7f6af00c1d60 con 0x7f6af01521c0 2024-02-21T23:11:31.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:31.849+0000 7f6ae7400640 1 -- 172.21.15.28:0/2898810011 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterResize.test_down_grow"]}]=0 v511) v1 ==== 141+0+0 (secure 0 0 0) 0x7f6ae807b9f0 con 0x7f6af01521c0 2024-02-21T23:11:31.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:31.853+0000 7f6af5200640 1 -- 172.21.15.28:0/2898810011 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6acc06b4d0 msgr2=0x7f6acc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:31.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:31.853+0000 7f6af5200640 1 --2- 172.21.15.28:0/2898810011 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6acc06b4d0 0x7f6acc06d990 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f6ae0003cc0 tx=0x7f6ae00079b0 comp rx=0 tx=0).stop 2024-02-21T23:11:31.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:31.853+0000 7f6af5200640 1 -- 172.21.15.28:0/2898810011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6af01521c0 msgr2=0x7f6af0148880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:31.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:31.853+0000 7f6af5200640 1 --2- 172.21.15.28:0/2898810011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6af01521c0 0x7f6af0148880 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 crypto rx=0x7f6ae8009e50 tx=0x7f6ae8004a40 comp rx=0 tx=0).stop 2024-02-21T23:11:31.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:31.853+0000 7f6af5200640 1 -- 172.21.15.28:0/2898810011 shutdown_connections 2024-02-21T23:11:31.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:31.853+0000 7f6af5200640 1 --2- 172.21.15.28:0/2898810011 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6acc06b4d0 0x7f6acc06d990 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:31.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:31.853+0000 7f6af5200640 1 --2- 172.21.15.28:0/2898810011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6af01521c0 0x7f6af0148880 unknown :-1 s=CLOSED pgs=2033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:31.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:31.853+0000 7f6af5200640 1 --2- 172.21.15.28:0/2898810011 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6af014f0b0 0x7f6af00c4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:31.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:31.853+0000 7f6af5200640 1 --2- 172.21.15.28:0/2898810011 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6af00a6160 0x7f6af00c5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:31.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:31.853+0000 7f6af5200640 1 -- 172.21.15.28:0/2898810011 >> 172.21.15.28:0/2898810011 conn(0x7f6af006b630 msgr2=0x7f6af00c64e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:31.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:31.854+0000 7f6af5200640 1 -- 172.21.15.28:0/2898810011 shutdown_connections 2024-02-21T23:11:31.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:31.854+0000 7f6af5200640 1 -- 172.21.15.28:0/2898810011 wait complete. 2024-02-21T23:11:31.865 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-02-21T23:11:32.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.102+0000 7f6877200640 1 Processor -- start 2024-02-21T23:11:32.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.103+0000 7f6877200640 1 -- start start 2024-02-21T23:11:32.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.104+0000 7f6877200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f687806ae70 0x7f687806b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:32.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.104+0000 7f6877200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f68780c4c10 0x7f6878159540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:32.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.104+0000 7f6877200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68780c0d40 con 0x7f6878159a80 2024-02-21T23:11:32.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.104+0000 7f6877200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f687815caf0 con 0x7f687806ae70 2024-02-21T23:11:32.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.104+0000 7f6877200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f687815cdd0 con 0x7f68780c4c10 2024-02-21T23:11:32.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.104+0000 7f6875e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f687806ae70 0x7f687806b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:32.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.104+0000 7f6876800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f6878159a80 0x7f687806db10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60984/0 (socket says 172.21.15.28:60984) 2024-02-21T23:11:32.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.104+0000 7f6876800640 1 -- 172.21.15.28:0/2418414238 learned_addr learned my addr 172.21.15.28:0/2418414238 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:11:32.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.105+0000 7f686fe00640 1 --2- 172.21.15.28:0/2418414238 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f68780c4c10 0x7f6878159540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:32.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.105+0000 7f6875e00640 1 -- 172.21.15.28:0/2418414238 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f68780c4c10 msgr2=0x7f6878159540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:32.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.105+0000 7f6875e00640 1 --2- 172.21.15.28:0/2418414238 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f68780c4c10 0x7f6878159540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:32.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.105+0000 7f6875e00640 1 -- 172.21.15.28:0/2418414238 >> v1:172.21.15.28:6789/0 conn(0x7f6878159a80 legacy=0x7f687806db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:32.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.105+0000 7f6875e00640 1 -- 172.21.15.28:0/2418414238 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f687815d0b0 con 0x7f687806ae70 2024-02-21T23:11:32.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.105+0000 7f6875400640 1 -- 172.21.15.28:0/2418414238 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 816783191 0 0) 0x7f68780c0d40 con 0x7f6878159a80 2024-02-21T23:11:32.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.106+0000 7f6875e00640 1 --2- 172.21.15.28:0/2418414238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f687806ae70 0x7f687806b250 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 crypto rx=0x7f68680097a0 tx=0x7f686802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb487b381a0509aa server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:32.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.106+0000 7f6875400640 1 -- 172.21.15.28:0/2418414238 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68680315a0 con 0x7f687806ae70 2024-02-21T23:11:32.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.107+0000 7f6875400640 1 -- 172.21.15.28:0/2418414238 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6868037070 con 0x7f687806ae70 2024-02-21T23:11:32.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.107+0000 7f6875400640 1 -- 172.21.15.28:0/2418414238 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6868031960 con 0x7f687806ae70 2024-02-21T23:11:32.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.107+0000 7f6877200640 1 -- 172.21.15.28:0/2418414238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f687806ae70 msgr2=0x7f687806b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:32.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.107+0000 7f6877200640 1 --2- 172.21.15.28:0/2418414238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f687806ae70 0x7f687806b250 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 crypto rx=0x7f68680097a0 tx=0x7f686802b840 comp rx=0 tx=0).stop 2024-02-21T23:11:32.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.107+0000 7f6877200640 1 -- 172.21.15.28:0/2418414238 shutdown_connections 2024-02-21T23:11:32.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.107+0000 7f6877200640 1 --2- 172.21.15.28:0/2418414238 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f68780c4c10 0x7f6878159540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:32.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.107+0000 7f6877200640 1 --2- 172.21.15.28:0/2418414238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f687806ae70 0x7f687806b250 unknown :-1 s=CLOSED pgs=2035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:32.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.107+0000 7f6877200640 1 -- 172.21.15.28:0/2418414238 >> 172.21.15.28:0/2418414238 conn(0x7f68780a8530 msgr2=0x7f68780a8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:32.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.107+0000 7f6877200640 1 -- 172.21.15.28:0/2418414238 shutdown_connections 2024-02-21T23:11:32.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.107+0000 7f6877200640 1 -- 172.21.15.28:0/2418414238 wait complete. 2024-02-21T23:11:32.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.109+0000 7f6877200640 1 Processor -- start 2024-02-21T23:11:32.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.109+0000 7f6877200640 1 -- start start 2024-02-21T23:11:32.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.110+0000 7f6877200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f687806ae70 0x7f6878152210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:32.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.110+0000 7f6877200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68780c4c10 0x7f6878157d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:32.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.110+0000 7f6877200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6878159a80 0x7f6878158250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:32.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.110+0000 7f6877200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f687815c7e0 con 0x7f68780c4c10 2024-02-21T23:11:32.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.110+0000 7f6877200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f687815d670 con 0x7f687806ae70 2024-02-21T23:11:32.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.110+0000 7f6877200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f687806f6b0 con 0x7f6878159a80 2024-02-21T23:11:32.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.110+0000 7f686fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68780c4c10 0x7f6878157d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:32.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.110+0000 7f686fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68780c4c10 0x7f6878157d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41492/0 (socket says 172.21.15.28:41492) 2024-02-21T23:11:32.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.110+0000 7f686fe00640 1 -- 172.21.15.28:0/3087613140 learned_addr learned my addr 172.21.15.28:0/3087613140 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:32.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.111+0000 7f6875e00640 1 --2- 172.21.15.28:0/3087613140 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f687806ae70 0x7f6878152210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:32.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.111+0000 7f6876800640 1 --2- 172.21.15.28:0/3087613140 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6878159a80 0x7f6878158250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:32.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.111+0000 7f686fe00640 1 -- 172.21.15.28:0/3087613140 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6878159a80 msgr2=0x7f6878158250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:32.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.111+0000 7f686fe00640 1 --2- 172.21.15.28:0/3087613140 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6878159a80 0x7f6878158250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:32.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.111+0000 7f686fe00640 1 -- 172.21.15.28:0/3087613140 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f687806ae70 msgr2=0x7f6878152210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:32.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.111+0000 7f686fe00640 1 --2- 172.21.15.28:0/3087613140 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f687806ae70 0x7f6878152210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:32.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.111+0000 7f686fe00640 1 -- 172.21.15.28:0/3087613140 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68781eef20 con 0x7f68780c4c10 2024-02-21T23:11:32.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.111+0000 7f686fe00640 1 --2- 172.21.15.28:0/3087613140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68780c4c10 0x7f6878157d10 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7f686000c7f0 tx=0x7f686000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:32.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.112+0000 7f686e000640 1 -- 172.21.15.28:0/3087613140 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6860015070 con 0x7f68780c4c10 2024-02-21T23:11:32.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.112+0000 7f686e000640 1 -- 172.21.15.28:0/3087613140 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6860004240 con 0x7f68780c4c10 2024-02-21T23:11:32.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.112+0000 7f686e000640 1 -- 172.21.15.28:0/3087613140 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6860007620 con 0x7f68780c4c10 2024-02-21T23:11:32.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.112+0000 7f6877200640 1 -- 172.21.15.28:0/3087613140 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68781ef200 con 0x7f68780c4c10 2024-02-21T23:11:32.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.112+0000 7f6877200640 1 -- 172.21.15.28:0/3087613140 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68781ef7c0 con 0x7f68780c4c10 2024-02-21T23:11:32.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.113+0000 7f6877200640 1 -- 172.21.15.28:0/3087613140 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68781ef9c0 con 0x7f68780c4c10 2024-02-21T23:11:32.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.114+0000 7f686e000640 1 -- 172.21.15.28:0/3087613140 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f68600079d0 con 0x7f68780c4c10 2024-02-21T23:11:32.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.115+0000 7f686e000640 1 --2- 172.21.15.28:0/3087613140 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f685406b4d0 0x7f685406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:32.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.115+0000 7f686e000640 1 -- 172.21.15.28:0/3087613140 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f686008a0b0 con 0x7f68780c4c10 2024-02-21T23:11:32.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.115+0000 7f6875e00640 1 --2- 172.21.15.28:0/3087613140 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f685406b4d0 0x7f685406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:32.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.117+0000 7f6875e00640 1 --2- 172.21.15.28:0/3087613140 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f685406b4d0 0x7f685406d990 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f6868036000 tx=0x7f6868008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:32.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.120+0000 7f686e000640 1 -- 172.21.15.28:0/3087613140 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f68781ef9c0 con 0x7f68780c4c10 2024-02-21T23:11:32.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.318+0000 7f6877200640 1 -- 172.21.15.28:0/3087613140 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f687804f290 con 0x7f68780c4c10 2024-02-21T23:11:32.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.324+0000 7f686e000640 1 -- 172.21.15.28:0/3087613140 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v59)=0 v59) v1 ==== 122+0+0 (secure 0 0 0) 0x7f6860059130 con 0x7f68780c4c10 2024-02-21T23:11:32.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.326+0000 7f6877200640 1 -- 172.21.15.28:0/3087613140 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f685406b4d0 msgr2=0x7f685406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:32.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.326+0000 7f6877200640 1 --2- 172.21.15.28:0/3087613140 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f685406b4d0 0x7f685406d990 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f6868036000 tx=0x7f6868008030 comp rx=0 tx=0).stop 2024-02-21T23:11:32.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.326+0000 7f6877200640 1 -- 172.21.15.28:0/3087613140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68780c4c10 msgr2=0x7f6878157d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:32.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.326+0000 7f6877200640 1 --2- 172.21.15.28:0/3087613140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68780c4c10 0x7f6878157d10 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7f686000c7f0 tx=0x7f686000ccb0 comp rx=0 tx=0).stop 2024-02-21T23:11:32.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.327+0000 7f6877200640 1 -- 172.21.15.28:0/3087613140 shutdown_connections 2024-02-21T23:11:32.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.327+0000 7f6877200640 1 --2- 172.21.15.28:0/3087613140 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f685406b4d0 0x7f685406d990 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:32.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.327+0000 7f6877200640 1 --2- 172.21.15.28:0/3087613140 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6878159a80 0x7f6878158250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:32.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.327+0000 7f6877200640 1 --2- 172.21.15.28:0/3087613140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68780c4c10 0x7f6878157d10 unknown :-1 s=CLOSED pgs=2036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:32.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.327+0000 7f6877200640 1 --2- 172.21.15.28:0/3087613140 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f687806ae70 0x7f6878152210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:32.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.327+0000 7f6877200640 1 -- 172.21.15.28:0/3087613140 >> 172.21.15.28:0/3087613140 conn(0x7f68780a8530 msgr2=0x7f687806b250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:32.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.327+0000 7f6877200640 1 -- 172.21.15.28:0/3087613140 shutdown_connections 2024-02-21T23:11:32.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.327+0000 7f6877200640 1 -- 172.21.15.28:0/3087613140 wait complete. 2024-02-21T23:11:32.335 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-02-21T23:11:32.336 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-21T23:11:32.450 DEBUG:teuthology.orchestra.run.smithi081:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-21T23:11:32.573 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-02-21T23:11:32.573 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-21T23:11:32.573 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-21T23:11:32.634 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054608,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-02-21T23:11:32.634 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-02-21T23:11:32.635 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-21T23:11:32.692 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053833,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498812k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249404k,nr_inodes=812351,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498812k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-02-21T23:11:32.692 INFO:tasks.cephfs.cephfs_test_case:[, ] 2024-02-21T23:11:32.692 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:11:32.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.938+0000 7fac33e00640 1 Processor -- start 2024-02-21T23:11:32.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.938+0000 7fac33e00640 1 -- start start 2024-02-21T23:11:32.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.940+0000 7fac33e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fac34150450 0x7fac340a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:32.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.940+0000 7fac33e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fac34156af0 0x7fac3406dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:32.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.940+0000 7fac33e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac340c2760 con 0x7fac3406ee40 2024-02-21T23:11:32.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.940+0000 7fac33e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac340c22c0 con 0x7fac34156af0 2024-02-21T23:11:32.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.940+0000 7fac33e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac34156660 con 0x7fac34150450 2024-02-21T23:11:32.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.940+0000 7fac2a000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fac3406ee40 0x7fac341536f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60998/0 (socket says 172.21.15.28:60998) 2024-02-21T23:11:32.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.940+0000 7fac2a000640 1 -- 172.21.15.28:0/2988343932 learned_addr learned my addr 172.21.15.28:0/2988343932 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:11:32.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.940+0000 7fac33400640 1 --2- 172.21.15.28:0/2988343932 >> v2:172.21.15.28:3300/0 conn(0x7fac34156af0 0x7fac3406dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:32.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.940+0000 7fac32a00640 1 --2- 172.21.15.28:0/2988343932 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fac34150450 0x7fac340a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:32.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.941+0000 7fac32000640 1 -- 172.21.15.28:0/2988343932 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 458556757 0 0) 0x7fac340c2760 con 0x7fac3406ee40 2024-02-21T23:11:32.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.941+0000 7fac32000640 1 -- 172.21.15.28:0/2988343932 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac18003520 con 0x7fac3406ee40 2024-02-21T23:11:32.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.941+0000 7fac33400640 1 -- 172.21.15.28:0/2988343932 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fac34150450 msgr2=0x7fac340a5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:32.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.941+0000 7fac33400640 1 --2- 172.21.15.28:0/2988343932 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fac34150450 0x7fac340a5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:32.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.941+0000 7fac33400640 1 -- 172.21.15.28:0/2988343932 >> v1:172.21.15.28:6789/0 conn(0x7fac3406ee40 legacy=0x7fac341536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:32.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.941+0000 7fac33400640 1 -- 172.21.15.28:0/2988343932 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac34156310 con 0x7fac34156af0 2024-02-21T23:11:32.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.942+0000 7fac33400640 1 --2- 172.21.15.28:0/2988343932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac34156af0 0x7fac3406dff0 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7fac2c009d90 tx=0x7fac2c02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc93a6e031163f15 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:32.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.942+0000 7fac32000640 1 -- 172.21.15.28:0/2988343932 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac2c039070 con 0x7fac34156af0 2024-02-21T23:11:32.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.942+0000 7fac32000640 1 -- 172.21.15.28:0/2988343932 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac2c02cc50 con 0x7fac34156af0 2024-02-21T23:11:32.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.942+0000 7fac32000640 1 -- 172.21.15.28:0/2988343932 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac2c034440 con 0x7fac34156af0 2024-02-21T23:11:32.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.943+0000 7fac33e00640 1 -- 172.21.15.28:0/2988343932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac34156af0 msgr2=0x7fac3406dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:32.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.943+0000 7fac33e00640 1 --2- 172.21.15.28:0/2988343932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac34156af0 0x7fac3406dff0 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7fac2c009d90 tx=0x7fac2c02bea0 comp rx=0 tx=0).stop 2024-02-21T23:11:32.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.943+0000 7fac33e00640 1 -- 172.21.15.28:0/2988343932 shutdown_connections 2024-02-21T23:11:32.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.943+0000 7fac33e00640 1 --2- 172.21.15.28:0/2988343932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac34156af0 0x7fac3406dff0 unknown :-1 s=CLOSED pgs=2038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:32.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.943+0000 7fac33e00640 1 --2- 172.21.15.28:0/2988343932 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fac34150450 0x7fac340a5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:32.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.943+0000 7fac33e00640 1 -- 172.21.15.28:0/2988343932 >> 172.21.15.28:0/2988343932 conn(0x7fac3406b630 msgr2=0x7fac340c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:32.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.943+0000 7fac33e00640 1 -- 172.21.15.28:0/2988343932 shutdown_connections 2024-02-21T23:11:32.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.943+0000 7fac33e00640 1 -- 172.21.15.28:0/2988343932 wait complete. 2024-02-21T23:11:32.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.945+0000 7fac33e00640 1 Processor -- start 2024-02-21T23:11:32.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.945+0000 7fac33e00640 1 -- start start 2024-02-21T23:11:32.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.946+0000 7fac33e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fac3406ee40 0x7fac340bfba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:32.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.946+0000 7fac33e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac34150450 0x7fac34145710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:32.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.946+0000 7fac33e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac34156af0 0x7fac34145c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:32.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.946+0000 7fac33e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac34156f80 con 0x7fac34156af0 2024-02-21T23:11:32.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.946+0000 7fac33e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac340c2cb0 con 0x7fac34150450 2024-02-21T23:11:32.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.946+0000 7fac33e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac34157100 con 0x7fac3406ee40 2024-02-21T23:11:32.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.946+0000 7fac33400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac34156af0 0x7fac34145c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:32.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.946+0000 7fac33400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac34156af0 0x7fac34145c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41504/0 (socket says 172.21.15.28:41504) 2024-02-21T23:11:32.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.946+0000 7fac33400640 1 -- 172.21.15.28:0/1028102030 learned_addr learned my addr 172.21.15.28:0/1028102030 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:32.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.946+0000 7fac32a00640 1 --2- 172.21.15.28:0/1028102030 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fac3406ee40 0x7fac340bfba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:32.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.947+0000 7fac33400640 1 -- 172.21.15.28:0/1028102030 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fac3406ee40 msgr2=0x7fac340bfba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:32.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.947+0000 7fac2a000640 1 --2- 172.21.15.28:0/1028102030 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac34150450 0x7fac34145710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:32.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.947+0000 7fac33400640 1 --2- 172.21.15.28:0/1028102030 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fac3406ee40 0x7fac340bfba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:32.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.947+0000 7fac33400640 1 -- 172.21.15.28:0/1028102030 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac34150450 msgr2=0x7fac34145710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:32.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.947+0000 7fac33400640 1 --2- 172.21.15.28:0/1028102030 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac34150450 0x7fac34145710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:32.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.947+0000 7fac33400640 1 -- 172.21.15.28:0/1028102030 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac341ea5a0 con 0x7fac34156af0 2024-02-21T23:11:32.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.947+0000 7fac32a00640 1 --2- 172.21.15.28:0/1028102030 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fac3406ee40 0x7fac340bfba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:11:32.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.947+0000 7fac33400640 1 --2- 172.21.15.28:0/1028102030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac34156af0 0x7fac34145c50 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7fac2c009eb0 tx=0x7fac2c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:32.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.947+0000 7fac2be00640 1 -- 172.21.15.28:0/1028102030 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac2c039050 con 0x7fac34156af0 2024-02-21T23:11:32.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.947+0000 7fac2be00640 1 -- 172.21.15.28:0/1028102030 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac2c045070 con 0x7fac34156af0 2024-02-21T23:11:32.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.947+0000 7fac2be00640 1 -- 172.21.15.28:0/1028102030 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac2c009450 con 0x7fac34156af0 2024-02-21T23:11:32.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.947+0000 7fac33e00640 1 -- 172.21.15.28:0/1028102030 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac34071070 con 0x7fac34156af0 2024-02-21T23:11:32.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.948+0000 7fac33e00640 1 -- 172.21.15.28:0/1028102030 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac34072a60 con 0x7fac34156af0 2024-02-21T23:11:32.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.949+0000 7fac33e00640 1 -- 172.21.15.28:0/1028102030 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac3404f290 con 0x7fac34156af0 2024-02-21T23:11:32.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.950+0000 7fac2be00640 1 -- 172.21.15.28:0/1028102030 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fac2c03e610 con 0x7fac34156af0 2024-02-21T23:11:32.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.951+0000 7fac2be00640 1 --2- 172.21.15.28:0/1028102030 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fac1006b4d0 0x7fac1006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:32.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.951+0000 7fac2be00640 1 -- 172.21.15.28:0/1028102030 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 5335+0+0 (secure 0 0 0) 0x7fac2c0aca90 con 0x7fac34156af0 2024-02-21T23:11:32.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.952+0000 7fac32a00640 1 --2- 172.21.15.28:0/1028102030 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fac1006b4d0 0x7fac1006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:32.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.953+0000 7fac32a00640 1 --2- 172.21.15.28:0/1028102030 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fac1006b4d0 0x7fac1006d990 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fac34153c60 tx=0x7fac24008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:32.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:32.956+0000 7fac2be00640 1 -- 172.21.15.28:0/1028102030 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fac341ea5a0 con 0x7fac34156af0 2024-02-21T23:11:33.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.197+0000 7fac33e00640 1 -- 172.21.15.28:0/1028102030 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac340c1f10 con 0x7fac34156af0 2024-02-21T23:11:33.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.198+0000 7fac2be00640 1 -- 172.21.15.28:0/1028102030 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 194 v194) v1 ==== 95+0+3698 (secure 0 0 0) 0x7fac2c07ba70 con 0x7fac34156af0 2024-02-21T23:11:33.186 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:11:33.187 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":194,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6686,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/155806034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":155806034},{"type":"v1","addr":"172.21.15.81:6836","nonce":155806034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":180},{"gid":7480,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3897243745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":3897243745},{"type":"v1","addr":"172.21.15.81:6837","nonce":3897243745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":180},{"gid":7486,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2827135632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2827135632},{"type":"v1","addr":"172.21.15.28:6835","nonce":2827135632}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":180},{"gid":9849,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/634506885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":634506885},{"type":"v1","addr":"172.21.15.28:6833","nonce":634506885}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":194}],"filesystems":[]} 2024-02-21T23:11:33.187 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 194 2024-02-21T23:11:33.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.201+0000 7fac33e00640 1 -- 172.21.15.28:0/1028102030 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fac1006b4d0 msgr2=0x7fac1006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:33.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.201+0000 7fac33e00640 1 --2- 172.21.15.28:0/1028102030 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fac1006b4d0 0x7fac1006d990 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fac34153c60 tx=0x7fac24008040 comp rx=0 tx=0).stop 2024-02-21T23:11:33.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.201+0000 7fac33e00640 1 -- 172.21.15.28:0/1028102030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac34156af0 msgr2=0x7fac34145c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:33.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.201+0000 7fac33e00640 1 --2- 172.21.15.28:0/1028102030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac34156af0 0x7fac34145c50 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7fac2c009eb0 tx=0x7fac2c004a40 comp rx=0 tx=0).stop 2024-02-21T23:11:33.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.201+0000 7fac33e00640 1 -- 172.21.15.28:0/1028102030 shutdown_connections 2024-02-21T23:11:33.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.201+0000 7fac33e00640 1 --2- 172.21.15.28:0/1028102030 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fac1006b4d0 0x7fac1006d990 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:33.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.201+0000 7fac33e00640 1 --2- 172.21.15.28:0/1028102030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac34156af0 0x7fac34145c50 unknown :-1 s=CLOSED pgs=2039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:33.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.201+0000 7fac33e00640 1 --2- 172.21.15.28:0/1028102030 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fac34150450 0x7fac34145710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:33.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.201+0000 7fac33e00640 1 --2- 172.21.15.28:0/1028102030 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fac3406ee40 0x7fac340bfba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:33.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.201+0000 7fac33e00640 1 -- 172.21.15.28:0/1028102030 >> 172.21.15.28:0/1028102030 conn(0x7fac3406b630 msgr2=0x7fac34148940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:33.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.202+0000 7fac33e00640 1 -- 172.21.15.28:0/1028102030 shutdown_connections 2024-02-21T23:11:33.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.202+0000 7fac33e00640 1 -- 172.21.15.28:0/1028102030 wait complete. 2024-02-21T23:11:33.208 INFO:tasks.ceph.mds.a:Restarting daemon 2024-02-21T23:11:33.208 INFO:tasks.ceph.mds.a:Stopping old one... 2024-02-21T23:11:33.208 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-02-21T23:11:33.208 INFO:teuthology.orchestra.run:waiting for 300 2024-02-21T23:11:33.209 INFO:tasks.ceph.mds.c:Restarting daemon 2024-02-21T23:11:33.209 INFO:tasks.ceph.mds.c:Stopping old one... 2024-02-21T23:11:33.209 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-02-21T23:11:33.209 INFO:teuthology.orchestra.run:waiting for 300 2024-02-21T23:11:33.209 INFO:tasks.ceph.mds.b:Restarting daemon 2024-02-21T23:11:33.210 INFO:tasks.ceph.mds.b:Stopping old one... 2024-02-21T23:11:33.210 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-02-21T23:11:33.210 INFO:teuthology.orchestra.run:waiting for 300 2024-02-21T23:11:33.210 INFO:tasks.ceph.mds.d:Restarting daemon 2024-02-21T23:11:33.210 INFO:tasks.ceph.mds.d:Stopping old one... 2024-02-21T23:11:33.210 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-02-21T23:11:33.210 INFO:teuthology.orchestra.run:waiting for 300 2024-02-21T23:11:33.227 INFO:tasks.ceph.mds.a:Stopped 2024-02-21T23:11:33.227 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-02-21T23:11:33.230 INFO:tasks.ceph.mds.d:Stopped 2024-02-21T23:11:33.231 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-02-21T23:11:33.232 INFO:tasks.ceph.mds.c:Stopped 2024-02-21T23:11:33.232 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-02-21T23:11:33.235 INFO:tasks.ceph.mds.b:Stopped 2024-02-21T23:11:33.235 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-02-21T23:11:33.280 INFO:tasks.ceph.mds.a:Started 2024-02-21T23:11:33.280 DEBUG:teuthology.parallel:result is None 2024-02-21T23:11:33.282 INFO:tasks.ceph.mds.d:Started 2024-02-21T23:11:33.283 DEBUG:teuthology.parallel:result is None 2024-02-21T23:11:33.286 INFO:tasks.ceph.mds.c:Started 2024-02-21T23:11:33.286 DEBUG:teuthology.parallel:result is None 2024-02-21T23:11:33.289 INFO:tasks.ceph.mds.b:Started 2024-02-21T23:11:33.290 DEBUG:teuthology.parallel:result is None 2024-02-21T23:11:33.290 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-02-21T23:11:33.440 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-02-21T23:11:33.448 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-02-21T23:11:33.450 INFO:tasks.ceph.mds.d.smithi081.stdout:starting mds.d at 2024-02-21T23:11:33.450 INFO:tasks.ceph.mds.b.smithi081.stdout:starting mds.b at 2024-02-21T23:11:33.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.498+0000 7f4bb2600640 1 Processor -- start 2024-02-21T23:11:33.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.498+0000 7f4bb2600640 1 -- start start 2024-02-21T23:11:33.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.499+0000 7f4bb2600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4bac0c2df0 0x7f4bac156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:33.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.499+0000 7f4bb2600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f4bac0c5f20 0x7f4bac0c31d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:33.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.499+0000 7f4bb2600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bac0c5c10 con 0x7f4bac0c3710 2024-02-21T23:11:33.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.499+0000 7f4bb2600640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bac06e6a0 con 0x7f4bac0c5f20 2024-02-21T23:11:33.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.499+0000 7f4bb2600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bac06e8d0 con 0x7f4bac0c2df0 2024-02-21T23:11:33.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.499+0000 7f4bab400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4bac0c2df0 0x7f4bac156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:33.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.499+0000 7f4bab400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4bac0c2df0 0x7f4bac156ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3301/0 says I am v2:172.21.15.28:43924/0 (socket says 172.21.15.28:43924) 2024-02-21T23:11:33.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.499+0000 7f4ba2a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f4bac0c5f20 0x7f4bac0c31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:33.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.499+0000 7f4bab400640 1 -- 172.21.15.28:0/1814279498 learned_addr learned my addr 172.21.15.28:0/1814279498 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:33.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.500+0000 7f4bab400640 1 -- 172.21.15.28:0/1814279498 >> v2:172.21.15.81:3300/0 conn(0x7f4bac0c5f20 msgr2=0x7f4bac0c31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:33.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.500+0000 7f4bab400640 1 --2- 172.21.15.28:0/1814279498 >> v2:172.21.15.81:3300/0 conn(0x7f4bac0c5f20 0x7f4bac0c31d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:33.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.500+0000 7f4bab400640 1 -- 172.21.15.28:0/1814279498 >> v1:172.21.15.81:6789/0 conn(0x7f4bac0c3710 legacy=0x7f4bac153220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:33.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.500+0000 7f4bab400640 1 -- 172.21.15.28:0/1814279498 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bac06ebb0 con 0x7f4bac0c2df0 2024-02-21T23:11:33.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.501+0000 7f4bab400640 1 --2- 172.21.15.28:0/1814279498 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4bac0c2df0 0x7f4bac156ed0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c009870 tx=0x7f4b9c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6456ce7d6cf4d48 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:33.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.501+0000 7f4baaa00640 1 -- 172.21.15.28:0/1814279498 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b9c034070 con 0x7f4bac0c2df0 2024-02-21T23:11:33.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.501+0000 7f4baaa00640 1 -- 172.21.15.28:0/1814279498 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b9c030840 con 0x7f4bac0c2df0 2024-02-21T23:11:33.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.501+0000 7f4baaa00640 1 -- 172.21.15.28:0/1814279498 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b9c030c10 con 0x7f4bac0c2df0 2024-02-21T23:11:33.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.502+0000 7f4bb2600640 1 -- 172.21.15.28:0/1814279498 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4bac0c2df0 msgr2=0x7f4bac156ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:33.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.502+0000 7f4bb2600640 1 --2- 172.21.15.28:0/1814279498 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4bac0c2df0 0x7f4bac156ed0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c009870 tx=0x7f4b9c02b980 comp rx=0 tx=0).stop 2024-02-21T23:11:33.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.502+0000 7f4bb2600640 1 -- 172.21.15.28:0/1814279498 shutdown_connections 2024-02-21T23:11:33.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.502+0000 7f4bb2600640 1 --2- 172.21.15.28:0/1814279498 >> v2:172.21.15.81:3300/0 conn(0x7f4bac0c5f20 0x7f4bac0c31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:33.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.502+0000 7f4bb2600640 1 --2- 172.21.15.28:0/1814279498 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4bac0c2df0 0x7f4bac156ed0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:33.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.502+0000 7f4bb2600640 1 -- 172.21.15.28:0/1814279498 >> 172.21.15.28:0/1814279498 conn(0x7f4bac06b820 msgr2=0x7f4bac0c67f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:33.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.502+0000 7f4bb2600640 1 -- 172.21.15.28:0/1814279498 shutdown_connections 2024-02-21T23:11:33.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.502+0000 7f4bb2600640 1 -- 172.21.15.28:0/1814279498 wait complete. 2024-02-21T23:11:33.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.504+0000 7f4bb2600640 1 Processor -- start 2024-02-21T23:11:33.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.504+0000 7f4bb2600640 1 -- start start 2024-02-21T23:11:33.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.505+0000 7f4bb2600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4bac0c2df0 0x7f4bac1ec0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:33.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.505+0000 7f4bb2600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4bac0c3710 0x7f4bac1526d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:33.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.505+0000 7f4bb2600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4bac0c5f20 0x7f4bac1eebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:33.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.505+0000 7f4bb2600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bac153930 con 0x7f4bac0c5f20 2024-02-21T23:11:33.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.505+0000 7f4bb2600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bac06f170 con 0x7f4bac0c3710 2024-02-21T23:11:33.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.505+0000 7f4bb2600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bac153ab0 con 0x7f4bac0c2df0 2024-02-21T23:11:33.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.505+0000 7f4babe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4bac0c5f20 0x7f4bac1eebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:33.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.505+0000 7f4ba2a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4bac0c3710 0x7f4bac1526d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:33.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.505+0000 7f4bab400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4bac0c2df0 0x7f4bac1ec0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:33.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.505+0000 7f4ba2a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4bac0c3710 0x7f4bac1526d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.28:37014/0 (socket says 172.21.15.28:37014) 2024-02-21T23:11:33.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.505+0000 7f4babe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4bac0c5f20 0x7f4bac1eebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41546/0 (socket says 172.21.15.28:41546) 2024-02-21T23:11:33.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.505+0000 7f4bab400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4bac0c2df0 0x7f4bac1ec0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3301/0 says I am v2:172.21.15.28:43928/0 (socket says 172.21.15.28:43928) 2024-02-21T23:11:33.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.505+0000 7f4ba2a00640 1 -- 172.21.15.28:0/392039426 learned_addr learned my addr 172.21.15.28:0/392039426 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:33.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.506+0000 7f4babe00640 1 -- 172.21.15.28:0/392039426 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4bac0c2df0 msgr2=0x7f4bac1ec0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:33.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.506+0000 7f4babe00640 1 --2- 172.21.15.28:0/392039426 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4bac0c2df0 0x7f4bac1ec0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:33.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.506+0000 7f4babe00640 1 -- 172.21.15.28:0/392039426 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4bac0c3710 msgr2=0x7f4bac1526d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:33.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.506+0000 7f4babe00640 1 --2- 172.21.15.28:0/392039426 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4bac0c3710 0x7f4bac1526d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:33.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.506+0000 7f4babe00640 1 -- 172.21.15.28:0/392039426 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bac1eb620 con 0x7f4bac0c5f20 2024-02-21T23:11:33.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.506+0000 7f4bab400640 1 --2- 172.21.15.28:0/392039426 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4bac0c2df0 0x7f4bac1ec0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-21T23:11:33.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.506+0000 7f4babe00640 1 --2- 172.21.15.28:0/392039426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4bac0c5f20 0x7f4bac1eebb0 secure :-1 s=READY pgs=2044 cs=0 l=1 rev1=1 crypto rx=0x7f4ba400c690 tx=0x7f4ba400cb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:33.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.506+0000 7f4ba3e00640 1 -- 172.21.15.28:0/392039426 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba4003b70 con 0x7f4bac0c5f20 2024-02-21T23:11:33.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.506+0000 7f4ba3e00640 1 -- 172.21.15.28:0/392039426 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ba4003d10 con 0x7f4bac0c5f20 2024-02-21T23:11:33.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.506+0000 7f4ba3e00640 1 -- 172.21.15.28:0/392039426 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba4005310 con 0x7f4bac0c5f20 2024-02-21T23:11:33.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.506+0000 7f4bb2600640 1 -- 172.21.15.28:0/392039426 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bac1dde30 con 0x7f4bac0c5f20 2024-02-21T23:11:33.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.507+0000 7f4bb2600640 1 -- 172.21.15.28:0/392039426 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bac0ab180 con 0x7f4bac0c5f20 2024-02-21T23:11:33.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.508+0000 7f4bb2600640 1 -- 172.21.15.28:0/392039426 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bac06f400 con 0x7f4bac0c5f20 2024-02-21T23:11:33.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.509+0000 7f4ba3e00640 1 -- 172.21.15.28:0/392039426 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4ba40054b0 con 0x7f4bac0c5f20 2024-02-21T23:11:33.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.510+0000 7f4ba3e00640 1 --2- 172.21.15.28:0/392039426 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4b8806b500 0x7f4b8806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:33.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.510+0000 7f4ba3e00640 1 -- 172.21.15.28:0/392039426 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f4ba4089d70 con 0x7f4bac0c5f20 2024-02-21T23:11:33.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.510+0000 7f4bab400640 1 --2- 172.21.15.28:0/392039426 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4b8806b500 0x7f4b8806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:33.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.511+0000 7f4bab400640 1 --2- 172.21.15.28:0/392039426 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4b8806b500 0x7f4b8806d9c0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c036000 tx=0x7f4b9c008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:33.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.515+0000 7f4ba3e00640 1 -- 172.21.15.28:0/392039426 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4bac1eb620 con 0x7f4bac0c5f20 2024-02-21T23:11:33.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.726+0000 7f4bb2600640 1 -- 172.21.15.28:0/392039426 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f4bac04f370 con 0x7f4bac0c5f20 2024-02-21T23:11:33.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.877+0000 7f4ba3e00640 1 -- 172.21.15.28:0/392039426 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v180) v1 ==== 97+0+0 (secure 0 0 0) 0x7f4ba4058700 con 0x7f4bac0c5f20 2024-02-21T23:11:33.865 INFO:teuthology.orchestra.run.smithi028.stderr: removed all blocklist entries 2024-02-21T23:11:33.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.880+0000 7f4bb2600640 1 -- 172.21.15.28:0/392039426 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4b8806b500 msgr2=0x7f4b8806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:33.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.880+0000 7f4bb2600640 1 --2- 172.21.15.28:0/392039426 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4b8806b500 0x7f4b8806d9c0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c036000 tx=0x7f4b9c008030 comp rx=0 tx=0).stop 2024-02-21T23:11:33.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.880+0000 7f4bb2600640 1 -- 172.21.15.28:0/392039426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4bac0c5f20 msgr2=0x7f4bac1eebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:33.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.880+0000 7f4bb2600640 1 --2- 172.21.15.28:0/392039426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4bac0c5f20 0x7f4bac1eebb0 secure :-1 s=READY pgs=2044 cs=0 l=1 rev1=1 crypto rx=0x7f4ba400c690 tx=0x7f4ba400cb50 comp rx=0 tx=0).stop 2024-02-21T23:11:33.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.881+0000 7f4bb2600640 1 -- 172.21.15.28:0/392039426 shutdown_connections 2024-02-21T23:11:33.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.881+0000 7f4bb2600640 1 --2- 172.21.15.28:0/392039426 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4b8806b500 0x7f4b8806d9c0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:33.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.881+0000 7f4bb2600640 1 --2- 172.21.15.28:0/392039426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4bac0c5f20 0x7f4bac1eebb0 unknown :-1 s=CLOSED pgs=2044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:33.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.881+0000 7f4bb2600640 1 --2- 172.21.15.28:0/392039426 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4bac0c3710 0x7f4bac1526d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:33.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.881+0000 7f4bb2600640 1 --2- 172.21.15.28:0/392039426 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4bac0c2df0 0x7f4bac1ec0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:33.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.881+0000 7f4bb2600640 1 -- 172.21.15.28:0/392039426 >> 172.21.15.28:0/392039426 conn(0x7f4bac06b820 msgr2=0x7f4bac156980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:33.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.881+0000 7f4bb2600640 1 -- 172.21.15.28:0/392039426 shutdown_connections 2024-02-21T23:11:33.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:33.881+0000 7f4bb2600640 1 -- 172.21.15.28:0/392039426 wait complete. 2024-02-21T23:11:33.894 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-02-21T23:11:34.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.116+0000 7faf66c00640 1 Processor -- start 2024-02-21T23:11:34.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.116+0000 7faf66c00640 1 -- start start 2024-02-21T23:11:34.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.117+0000 7faf66c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faf600712e0 0x7faf60072040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:34.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.117+0000 7faf66c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7faf600a60f0 0x7faf60152730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:34.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.117+0000 7faf66c00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf600a6590 con 0x7faf600a8a30 2024-02-21T23:11:34.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.117+0000 7faf66c00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf600a85a0 con 0x7faf600a60f0 2024-02-21T23:11:34.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.117+0000 7faf66c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf600a7f50 con 0x7faf600712e0 2024-02-21T23:11:34.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.118+0000 7faf65800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faf600712e0 0x7faf60072040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:34.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.118+0000 7faf64e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7faf600a60f0 0x7faf60152730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:34.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.118+0000 7faf65800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faf600712e0 0x7faf60072040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3301/0 says I am v2:172.21.15.28:43936/0 (socket says 172.21.15.28:43936) 2024-02-21T23:11:34.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.118+0000 7faf65800640 1 -- 172.21.15.28:0/1605708679 learned_addr learned my addr 172.21.15.28:0/1605708679 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:34.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.119+0000 7faf65800640 1 -- 172.21.15.28:0/1605708679 >> v2:172.21.15.81:3300/0 conn(0x7faf600a60f0 msgr2=0x7faf60152730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:34.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.119+0000 7faf65800640 1 --2- 172.21.15.28:0/1605708679 >> v2:172.21.15.81:3300/0 conn(0x7faf600a60f0 0x7faf60152730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:34.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.119+0000 7faf65800640 1 -- 172.21.15.28:0/1605708679 >> v1:172.21.15.81:6789/0 conn(0x7faf600a8a30 legacy=0x7faf601551a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:34.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.119+0000 7faf65800640 1 -- 172.21.15.28:0/1605708679 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf600a8230 con 0x7faf600712e0 2024-02-21T23:11:34.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.120+0000 7faf65800640 1 --2- 172.21.15.28:0/1605708679 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faf600712e0 0x7faf60072040 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7faf5400bb10 tx=0x7faf5402e0c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6b82b586f6f54e server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:34.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.121+0000 7faf5fe00640 1 -- 172.21.15.28:0/1605708679 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf5403b070 con 0x7faf600712e0 2024-02-21T23:11:34.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.121+0000 7faf5fe00640 1 -- 172.21.15.28:0/1605708679 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf54036440 con 0x7faf600712e0 2024-02-21T23:11:34.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.121+0000 7faf5fe00640 1 -- 172.21.15.28:0/1605708679 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf54036810 con 0x7faf600712e0 2024-02-21T23:11:34.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.122+0000 7faf66c00640 1 -- 172.21.15.28:0/1605708679 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faf600712e0 msgr2=0x7faf60072040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:34.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.122+0000 7faf66c00640 1 --2- 172.21.15.28:0/1605708679 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faf600712e0 0x7faf60072040 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7faf5400bb10 tx=0x7faf5402e0c0 comp rx=0 tx=0).stop 2024-02-21T23:11:34.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.122+0000 7faf66c00640 1 -- 172.21.15.28:0/1605708679 shutdown_connections 2024-02-21T23:11:34.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.122+0000 7faf66c00640 1 --2- 172.21.15.28:0/1605708679 >> v2:172.21.15.81:3300/0 conn(0x7faf600a60f0 0x7faf60152730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:34.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.122+0000 7faf66c00640 1 --2- 172.21.15.28:0/1605708679 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faf600712e0 0x7faf60072040 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:34.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.122+0000 7faf66c00640 1 -- 172.21.15.28:0/1605708679 >> 172.21.15.28:0/1605708679 conn(0x7faf6006b550 msgr2=0x7faf600c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:34.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.122+0000 7faf66c00640 1 -- 172.21.15.28:0/1605708679 shutdown_connections 2024-02-21T23:11:34.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.122+0000 7faf66c00640 1 -- 172.21.15.28:0/1605708679 wait complete. 2024-02-21T23:11:34.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.123+0000 7faf66c00640 1 Processor -- start 2024-02-21T23:11:34.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.124+0000 7faf66c00640 1 -- start start 2024-02-21T23:11:34.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.124+0000 7faf66c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf600712e0 0x7faf600c55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:34.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.124+0000 7faf66c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf600a60f0 0x7faf600bf930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:34.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.124+0000 7faf66c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faf600a8a30 0x7faf601ead20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:34.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.125+0000 7faf66c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf600716c0 con 0x7faf600712e0 2024-02-21T23:11:34.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.125+0000 7faf66c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf600a6870 con 0x7faf600a60f0 2024-02-21T23:11:34.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.125+0000 7faf66c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf60071840 con 0x7faf600a8a30 2024-02-21T23:11:34.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.125+0000 7faf65800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf600712e0 0x7faf600c55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:34.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.125+0000 7faf64e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf600a60f0 0x7faf600bf930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:34.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.125+0000 7faf65800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf600712e0 0x7faf600c55f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41548/0 (socket says 172.21.15.28:41548) 2024-02-21T23:11:34.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.125+0000 7faf65800640 1 -- 172.21.15.28:0/3179006424 learned_addr learned my addr 172.21.15.28:0/3179006424 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:34.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.125+0000 7faf66200640 1 --2- 172.21.15.28:0/3179006424 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faf600a8a30 0x7faf601ead20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:34.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.126+0000 7faf65800640 1 -- 172.21.15.28:0/3179006424 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faf600a8a30 msgr2=0x7faf601ead20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:34.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.126+0000 7faf65800640 1 --2- 172.21.15.28:0/3179006424 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faf600a8a30 0x7faf601ead20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:34.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.126+0000 7faf65800640 1 -- 172.21.15.28:0/3179006424 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf600a60f0 msgr2=0x7faf600bf930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:34.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.126+0000 7faf65800640 1 --2- 172.21.15.28:0/3179006424 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf600a60f0 0x7faf600bf930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:34.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.126+0000 7faf65800640 1 -- 172.21.15.28:0/3179006424 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf60154fa0 con 0x7faf600712e0 2024-02-21T23:11:34.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.126+0000 7faf65800640 1 --2- 172.21.15.28:0/3179006424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf600712e0 0x7faf600c55f0 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7faf5402e100 tx=0x7faf540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:34.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.126+0000 7faf5d600640 1 -- 172.21.15.28:0/3179006424 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf5403b030 con 0x7faf600712e0 2024-02-21T23:11:34.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.126+0000 7faf5d600640 1 -- 172.21.15.28:0/3179006424 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf5403d050 con 0x7faf600712e0 2024-02-21T23:11:34.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.126+0000 7faf5d600640 1 -- 172.21.15.28:0/3179006424 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf54034870 con 0x7faf600712e0 2024-02-21T23:11:34.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.126+0000 7faf66c00640 1 -- 172.21.15.28:0/3179006424 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf601ea860 con 0x7faf600712e0 2024-02-21T23:11:34.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.126+0000 7faf66c00640 1 -- 172.21.15.28:0/3179006424 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf601554f0 con 0x7faf600712e0 2024-02-21T23:11:34.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.128+0000 7faf66c00640 1 -- 172.21.15.28:0/3179006424 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf600a6c30 con 0x7faf600712e0 2024-02-21T23:11:34.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.129+0000 7faf5d600640 1 -- 172.21.15.28:0/3179006424 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faf54048060 con 0x7faf600712e0 2024-02-21T23:11:34.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.129+0000 7faf5d600640 1 --2- 172.21.15.28:0/3179006424 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7faf4006b4d0 0x7faf4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:34.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.130+0000 7faf5d600640 1 -- 172.21.15.28:0/3179006424 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 5249+0+0 (secure 0 0 0) 0x7faf540ab4e0 con 0x7faf600712e0 2024-02-21T23:11:34.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.130+0000 7faf64e00640 1 --2- 172.21.15.28:0/3179006424 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7faf4006b4d0 0x7faf4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:34.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.131+0000 7faf64e00640 1 --2- 172.21.15.28:0/3179006424 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7faf4006b4d0 0x7faf4006d990 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7faf60072100 tx=0x7faf4c009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:34.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.134+0000 7faf5d600640 1 -- 172.21.15.28:0/3179006424 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7faf600a6c30 con 0x7faf600712e0 2024-02-21T23:11:34.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.404+0000 7faf66c00640 1 -- 172.21.15.28:0/3179006424 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7faf6004f290 con 0x7faf600712e0 2024-02-21T23:11:34.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.405+0000 7faf5d600640 1 -- 172.21.15.28:0/3179006424 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v20) v1 ==== 80+0+4724 (secure 0 0 0) 0x7faf601ea860 con 0x7faf600712e0 2024-02-21T23:11:34.393 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:11:34.394 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-21T23:11:34.394 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_dump": [ 2024-02-21T23:11:34.394 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:11:34.394 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.a", 2024-02-21T23:11:34.394 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDSgNZl4FY1KRAAJSK5Ibb0ibKiKXYpTLgtmQ==", 2024-02-21T23:11:34.394 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:11:34.394 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:11:34.395 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-02-21T23:11:34.395 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-02-21T23:11:34.395 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:11:34.395 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:11:34.395 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:11:34.395 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:11:34.395 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.b", 2024-02-21T23:11:34.395 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDTgNZlwRcAERAAgsYmmkuobKelBEJVCRpXUg==", 2024-02-21T23:11:34.396 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:11:34.396 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:11:34.396 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-02-21T23:11:34.396 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-02-21T23:11:34.396 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:11:34.396 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:11:34.397 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:11:34.397 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:11:34.397 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.c", 2024-02-21T23:11:34.397 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDSgNZloFpQOxAAK2WBArXRVNTgqH9WWDEPDA==", 2024-02-21T23:11:34.397 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:11:34.397 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:11:34.397 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-02-21T23:11:34.397 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-02-21T23:11:34.397 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:11:34.397 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:11:34.397 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:11:34.398 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:11:34.398 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.d", 2024-02-21T23:11:34.398 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDTgNZlblOmIRAAxe2HfSKGSkkzJDN4aVdzgQ==", 2024-02-21T23:11:34.398 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:11:34.398 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:11:34.398 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-02-21T23:11:34.398 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-02-21T23:11:34.398 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:11:34.398 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:11:34.398 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:11:34.398 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:11:34.398 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.0", 2024-02-21T23:11:34.399 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDVgNZlxNJbLxAAXot7Q3chrLEBsFNO8kT6QQ==", 2024-02-21T23:11:34.399 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:11:34.399 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:11:34.399 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:11:34.399 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:11:34.399 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:11:34.399 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:11:34.399 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:11:34.399 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.1", 2024-02-21T23:11:34.400 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDYgNZlZZfWORAAr4XkOxLYLmumVbu362N2HQ==", 2024-02-21T23:11:34.400 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:11:34.400 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:11:34.400 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:11:34.400 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:11:34.400 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:11:34.400 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:11:34.400 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:11:34.400 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.2", 2024-02-21T23:11:34.400 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDcgNZl9mcDGBAAIZGOF8IUMenwo386x95mwg==", 2024-02-21T23:11:34.400 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:11:34.400 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:11:34.400 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:11:34.401 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:11:34.401 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:11:34.401 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:11:34.401 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:11:34.401 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.3", 2024-02-21T23:11:34.401 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDfgNZl8ALSMBAAag9cmGH0/4xALWSUmrZowg==", 2024-02-21T23:11:34.401 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:11:34.401 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:11:34.401 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:11:34.401 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:11:34.401 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:11:34.401 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:11:34.401 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:11:34.401 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.4", 2024-02-21T23:11:34.401 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDngNZlBgeKABAAeW99RVfCBvB1lp7w7mQBlQ==", 2024-02-21T23:11:34.402 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:11:34.402 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:11:34.402 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:11:34.402 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:11:34.402 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:11:34.402 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:11:34.402 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:11:34.402 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.5", 2024-02-21T23:11:34.402 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDqgNZljQG3ChAAY/IZEIcvcGNvlFGIWib3gg==", 2024-02-21T23:11:34.402 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:11:34.402 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:11:34.402 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:11:34.402 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:11:34.403 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:11:34.403 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:11:34.403 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:11:34.403 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.6", 2024-02-21T23:11:34.403 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDtgNZlKnakIxAAkW5xlkshzvEHJdmpgPxR+g==", 2024-02-21T23:11:34.403 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:11:34.403 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:11:34.403 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:11:34.403 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:11:34.403 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:11:34.403 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:11:34.403 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:11:34.403 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.7", 2024-02-21T23:11:34.403 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDxgNZlTWxYARAASX1/gmqO+8mQPP8uA4xjiQ==", 2024-02-21T23:11:34.403 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:11:34.404 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:11:34.404 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:11:34.404 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:11:34.404 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:11:34.404 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:11:34.404 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:11:34.404 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.0", 2024-02-21T23:11:34.404 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDTgNZlILvYLxAAoREWxKKT7YcXnhkgk8YvoQ==", 2024-02-21T23:11:34.404 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:11:34.404 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:11:34.404 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow r", 2024-02-21T23:11:34.404 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-02-21T23:11:34.404 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:11:34.404 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:11:34.405 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:11:34.405 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.1", 2024-02-21T23:11:34.405 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDTgNZlm7ImOxAA8iRcXlzFPZPaXQfsKhuO/g==", 2024-02-21T23:11:34.405 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:11:34.405 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:11:34.405 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-02-21T23:11:34.405 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-02-21T23:11:34.405 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-02-21T23:11:34.405 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:11:34.405 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:11:34.405 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:11:34.405 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.admin", 2024-02-21T23:11:34.405 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDRgNZlBYd1GhAAbK5qBCIkf0Ej5nEyPnTxIQ==", 2024-02-21T23:11:34.405 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:11:34.406 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-02-21T23:11:34.406 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-02-21T23:11:34.406 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-02-21T23:11:34.406 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:11:34.406 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:11:34.406 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:11:34.406 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:11:34.406 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.x", 2024-02-21T23:11:34.406 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDSgNZlJYKeHBAAYPskmLRr7Jw44TdKzM/JMg==", 2024-02-21T23:11:34.406 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:11:34.406 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-02-21T23:11:34.406 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-02-21T23:11:34.406 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:11:34.407 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:11:34.407 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:11:34.407 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:11:34.407 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.y", 2024-02-21T23:11:34.407 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDSgNZlSRkOEBAAOhRj9PsZDpjoV6WT4drrZA==", 2024-02-21T23:11:34.407 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:11:34.407 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-02-21T23:11:34.407 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-02-21T23:11:34.407 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:11:34.407 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:11:34.407 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:11:34.407 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-21T23:11:34.407 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-21T23:11:34.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.408+0000 7faf66c00640 1 -- 172.21.15.28:0/3179006424 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7faf4006b4d0 msgr2=0x7faf4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:34.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.408+0000 7faf66c00640 1 --2- 172.21.15.28:0/3179006424 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7faf4006b4d0 0x7faf4006d990 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7faf60072100 tx=0x7faf4c009190 comp rx=0 tx=0).stop 2024-02-21T23:11:34.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.408+0000 7faf66c00640 1 -- 172.21.15.28:0/3179006424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf600712e0 msgr2=0x7faf600c55f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:34.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.408+0000 7faf66c00640 1 --2- 172.21.15.28:0/3179006424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf600712e0 0x7faf600c55f0 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7faf5402e100 tx=0x7faf540046c0 comp rx=0 tx=0).stop 2024-02-21T23:11:34.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.408+0000 7faf66c00640 1 -- 172.21.15.28:0/3179006424 shutdown_connections 2024-02-21T23:11:34.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.408+0000 7faf66c00640 1 --2- 172.21.15.28:0/3179006424 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7faf4006b4d0 0x7faf4006d990 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:34.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.408+0000 7faf66c00640 1 --2- 172.21.15.28:0/3179006424 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faf600a8a30 0x7faf601ead20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:34.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.408+0000 7faf66c00640 1 --2- 172.21.15.28:0/3179006424 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faf600a60f0 0x7faf600bf930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:34.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.408+0000 7faf66c00640 1 --2- 172.21.15.28:0/3179006424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf600712e0 0x7faf600c55f0 unknown :-1 s=CLOSED pgs=2045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:34.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.408+0000 7faf66c00640 1 -- 172.21.15.28:0/3179006424 >> 172.21.15.28:0/3179006424 conn(0x7faf6006b550 msgr2=0x7faf6014e160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:34.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.408+0000 7faf66c00640 1 -- 172.21.15.28:0/3179006424 shutdown_connections 2024-02-21T23:11:34.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.408+0000 7faf66c00640 1 -- 172.21.15.28:0/3179006424 wait complete. 2024-02-21T23:11:34.416 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:11:34.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.656+0000 7f2fb1c00640 1 Processor -- start 2024-02-21T23:11:34.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.656+0000 7f2fb1c00640 1 -- start start 2024-02-21T23:11:34.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.657+0000 7f2fb1c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2fac0a5fd0 0x7f2fac06dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:34.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.657+0000 7f2fb1c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fac0c2760 con 0x7f2fac150450 2024-02-21T23:11:34.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.657+0000 7f2fb1c00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fac156310 con 0x7f2fac06ee40 2024-02-21T23:11:34.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.657+0000 7f2fb1c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fac0c22c0 con 0x7f2fac0a5fd0 2024-02-21T23:11:34.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.658+0000 7f2fabe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f2fac150450 0x7f2fac1536f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:32794/0 (socket says 172.21.15.28:32794) 2024-02-21T23:11:34.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.658+0000 7f2fabe00640 1 -- 172.21.15.28:0/207342493 learned_addr learned my addr 172.21.15.28:0/207342493 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:11:34.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.658+0000 7f2fb1200640 1 --2- 172.21.15.28:0/207342493 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2fac0a5fd0 0x7f2fac06dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:34.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.658+0000 7f2faaa00640 1 -- 172.21.15.28:0/207342493 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 438848146 0 0) 0x7f2fac0c2760 con 0x7f2fac150450 2024-02-21T23:11:34.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.658+0000 7f2faaa00640 1 -- 172.21.15.28:0/207342493 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f90003520 con 0x7f2fac150450 2024-02-21T23:11:34.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.659+0000 7f2faaa00640 1 -- 172.21.15.28:0/207342493 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2912151241 0 0) 0x7f2f90003520 con 0x7f2fac150450 2024-02-21T23:11:34.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.659+0000 7f2faaa00640 1 -- 172.21.15.28:0/207342493 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2fac0a5fd0 msgr2=0x7f2fac06dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:34.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.659+0000 7f2faaa00640 1 --2- 172.21.15.28:0/207342493 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2fac0a5fd0 0x7f2fac06dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:34.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.659+0000 7f2faaa00640 1 -- 172.21.15.28:0/207342493 >> v1:172.21.15.81:6789/0 conn(0x7f2fac06ee40 legacy=0x7f2fac156af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:34.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.659+0000 7f2faaa00640 1 -- 172.21.15.28:0/207342493 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fac0c29d0 con 0x7f2fac150450 2024-02-21T23:11:34.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.659+0000 7f2faaa00640 1 -- 172.21.15.28:0/207342493 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f2f9c001670 con 0x7f2fac150450 2024-02-21T23:11:34.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.659+0000 7f2faaa00640 1 -- 172.21.15.28:0/207342493 >> v1:172.21.15.28:6789/0 conn(0x7f2fac150450 legacy=0x7f2fac1536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:34.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.659+0000 7f2faaa00640 1 --2- 172.21.15.28:0/207342493 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2f900039e0 0x7f2f90003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:34.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.659+0000 7f2faaa00640 1 --2- 172.21.15.28:0/207342493 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f90004800 0x7f2f90004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:34.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.659+0000 7f2faaa00640 1 --2- 172.21.15.28:0/207342493 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2fac0a5fd0 0x7f2f900051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:34.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.659+0000 7f2faaa00640 1 -- 172.21.15.28:0/207342493 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f9000baa0 con 0x7f2f90004800 2024-02-21T23:11:34.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.659+0000 7f2faaa00640 1 -- 172.21.15.28:0/207342493 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f9000bc50 con 0x7f2f900039e0 2024-02-21T23:11:34.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.659+0000 7f2faaa00640 1 -- 172.21.15.28:0/207342493 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f9000be50 con 0x7f2fac0a5fd0 2024-02-21T23:11:34.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.659+0000 7f2fab400640 1 --2- 172.21.15.28:0/207342493 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f90004800 0x7f2f90004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:34.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.659+0000 7f2fab400640 1 -- 172.21.15.28:0/207342493 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2fac0a5fd0 msgr2=0x7f2f900051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:34.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.659+0000 7f2fab400640 1 --2- 172.21.15.28:0/207342493 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2fac0a5fd0 0x7f2f900051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:34.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.659+0000 7f2fabe00640 1 --2- 172.21.15.28:0/207342493 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2f900039e0 0x7f2f90003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:34.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.659+0000 7f2fab400640 1 -- 172.21.15.28:0/207342493 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2f900039e0 msgr2=0x7f2f90003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:34.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.660+0000 7f2fab400640 1 --2- 172.21.15.28:0/207342493 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2f900039e0 0x7f2f90003e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:34.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.660+0000 7f2fab400640 1 -- 172.21.15.28:0/207342493 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f90003520 con 0x7f2f90004800 2024-02-21T23:11:34.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.661+0000 7f2fab400640 1 --2- 172.21.15.28:0/207342493 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f90004800 0x7f2f90004c80 secure :-1 s=READY pgs=2047 cs=0 l=1 rev1=1 crypto rx=0x7f2f98001720 tx=0x7f2f90005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5a4ef68e832b6dd server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:34.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.661+0000 7f2faaa00640 1 -- 172.21.15.28:0/207342493 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f98001370 con 0x7f2f90004800 2024-02-21T23:11:34.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.661+0000 7f2faaa00640 1 -- 172.21.15.28:0/207342493 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f980052d0 con 0x7f2f90004800 2024-02-21T23:11:34.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.661+0000 7f2faaa00640 1 -- 172.21.15.28:0/207342493 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f98003830 con 0x7f2f90004800 2024-02-21T23:11:34.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.661+0000 7f2fb1c00640 1 -- 172.21.15.28:0/207342493 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f90004800 msgr2=0x7f2f90004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:34.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.661+0000 7f2fb1c00640 1 --2- 172.21.15.28:0/207342493 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f90004800 0x7f2f90004c80 secure :-1 s=READY pgs=2047 cs=0 l=1 rev1=1 crypto rx=0x7f2f98001720 tx=0x7f2f90005830 comp rx=0 tx=0).stop 2024-02-21T23:11:34.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.661+0000 7f2fb1200640 1 -- 172.21.15.28:0/207342493 reap_dead start 2024-02-21T23:11:34.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.662+0000 7f2fb1c00640 1 -- 172.21.15.28:0/207342493 shutdown_connections 2024-02-21T23:11:34.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.662+0000 7f2fb1c00640 1 -- 172.21.15.28:0/207342493 >> 172.21.15.28:0/207342493 conn(0x7f2fac06b630 msgr2=0x7f2fac0c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:34.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.662+0000 7f2fb1c00640 1 -- 172.21.15.28:0/207342493 shutdown_connections 2024-02-21T23:11:34.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.662+0000 7f2fb1c00640 1 -- 172.21.15.28:0/207342493 wait complete. 2024-02-21T23:11:34.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.663+0000 7f2fb1c00640 1 Processor -- start 2024-02-21T23:11:34.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.664+0000 7f2fb1c00640 1 -- start start 2024-02-21T23:11:34.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.665+0000 7f2fb1c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2f90004800 0x7f2fac1e9a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:34.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.665+0000 7f2fb1c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fac1e9fc0 0x7f2fac1436f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:34.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.665+0000 7f2fb1c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2fac143c30 0x7f2fac144fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:34.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.665+0000 7f2fb1c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fac157200 con 0x7f2fac1e9fc0 2024-02-21T23:11:34.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.665+0000 7f2fb1c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fac156660 con 0x7f2fac143c30 2024-02-21T23:11:34.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.665+0000 7f2fb1c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fac06dff0 con 0x7f2f90004800 2024-02-21T23:11:34.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.665+0000 7f2fab400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fac1e9fc0 0x7f2fac1436f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:34.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.665+0000 7f2fab400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fac1e9fc0 0x7f2fac1436f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41552/0 (socket says 172.21.15.28:41552) 2024-02-21T23:11:34.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.665+0000 7f2fab400640 1 -- 172.21.15.28:0/3161914347 learned_addr learned my addr 172.21.15.28:0/3161914347 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:34.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.665+0000 7f2fb1200640 1 --2- 172.21.15.28:0/3161914347 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2fac143c30 0x7f2fac144fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:34.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.665+0000 7f2fabe00640 1 --2- 172.21.15.28:0/3161914347 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2f90004800 0x7f2fac1e9a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:34.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.665+0000 7f2fab400640 1 -- 172.21.15.28:0/3161914347 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2f90004800 msgr2=0x7f2fac1e9a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:34.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.665+0000 7f2fab400640 1 --2- 172.21.15.28:0/3161914347 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2f90004800 0x7f2fac1e9a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:34.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.665+0000 7f2fab400640 1 -- 172.21.15.28:0/3161914347 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2fac143c30 msgr2=0x7f2fac144fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:34.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.665+0000 7f2fab400640 1 --2- 172.21.15.28:0/3161914347 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2fac143c30 0x7f2fac144fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:34.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.665+0000 7f2fab400640 1 -- 172.21.15.28:0/3161914347 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fac072720 con 0x7f2fac1e9fc0 2024-02-21T23:11:34.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.665+0000 7f2fabe00640 1 --2- 172.21.15.28:0/3161914347 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2f90004800 0x7f2fac1e9a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:11:34.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.666+0000 7f2fab400640 1 --2- 172.21.15.28:0/3161914347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fac1e9fc0 0x7f2fac1436f0 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7f2f98005040 tx=0x7f2f98001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:34.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.666+0000 7f2fa3e00640 1 -- 172.21.15.28:0/3161914347 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f98041070 con 0x7f2fac1e9fc0 2024-02-21T23:11:34.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.666+0000 7f2fa3e00640 1 -- 172.21.15.28:0/3161914347 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f98005480 con 0x7f2fac1e9fc0 2024-02-21T23:11:34.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.666+0000 7f2fa3e00640 1 -- 172.21.15.28:0/3161914347 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f98007a30 con 0x7f2fac1e9fc0 2024-02-21T23:11:34.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.666+0000 7f2fb1c00640 1 -- 172.21.15.28:0/3161914347 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fac154920 con 0x7f2fac1e9fc0 2024-02-21T23:11:34.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.666+0000 7f2fb1c00640 1 -- 172.21.15.28:0/3161914347 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fac154bc0 con 0x7f2fac1e9fc0 2024-02-21T23:11:34.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.667+0000 7f2fb1c00640 1 -- 172.21.15.28:0/3161914347 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fac04f290 con 0x7f2fac1e9fc0 2024-02-21T23:11:34.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.668+0000 7f2fa3e00640 1 -- 172.21.15.28:0/3161914347 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2f98044070 con 0x7f2fac1e9fc0 2024-02-21T23:11:34.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.669+0000 7f2fa3e00640 1 --2- 172.21.15.28:0/3161914347 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2f8c06b500 0x7f2f8c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:34.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.669+0000 7f2fa3e00640 1 -- 172.21.15.28:0/3161914347 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f2f980abb40 con 0x7f2fac1e9fc0 2024-02-21T23:11:34.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.670+0000 7f2fabe00640 1 --2- 172.21.15.28:0/3161914347 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2f8c06b500 0x7f2f8c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:34.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.671+0000 7f2fabe00640 1 --2- 172.21.15.28:0/3161914347 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2f8c06b500 0x7f2f8c06d9c0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c0078d0 tx=0x7f2f9c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:34.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.674+0000 7f2fa3e00640 1 -- 172.21.15.28:0/3161914347 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f2f9807ab70 con 0x7f2fac1e9fc0 2024-02-21T23:11:34.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.875+0000 7f2fb1c00640 1 -- 172.21.15.28:0/3161914347 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2fac14a340 con 0x7f2fac1e9fc0 2024-02-21T23:11:34.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.877+0000 7f2fa3e00640 1 -- 172.21.15.28:0/3161914347 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v182) v1 ==== 74+0+13114 (secure 0 0 0) 0x7f2f9807a470 con 0x7f2fac1e9fc0 2024-02-21T23:11:34.865 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:11:34.866 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":182,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:11:33.888300+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:11:34.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.880+0000 7f2fb1c00640 1 -- 172.21.15.28:0/3161914347 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2f8c06b500 msgr2=0x7f2f8c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:34.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.880+0000 7f2fb1c00640 1 --2- 172.21.15.28:0/3161914347 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2f8c06b500 0x7f2f8c06d9c0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c0078d0 tx=0x7f2f9c009040 comp rx=0 tx=0).stop 2024-02-21T23:11:34.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.880+0000 7f2fb1c00640 1 -- 172.21.15.28:0/3161914347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fac1e9fc0 msgr2=0x7f2fac1436f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:34.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.880+0000 7f2fb1c00640 1 --2- 172.21.15.28:0/3161914347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fac1e9fc0 0x7f2fac1436f0 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7f2f98005040 tx=0x7f2f98001120 comp rx=0 tx=0).stop 2024-02-21T23:11:34.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.880+0000 7f2fb1c00640 1 -- 172.21.15.28:0/3161914347 shutdown_connections 2024-02-21T23:11:34.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.880+0000 7f2fb1c00640 1 --2- 172.21.15.28:0/3161914347 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2f8c06b500 0x7f2f8c06d9c0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:34.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.880+0000 7f2fb1c00640 1 --2- 172.21.15.28:0/3161914347 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2fac143c30 0x7f2fac144fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:34.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.880+0000 7f2fb1c00640 1 --2- 172.21.15.28:0/3161914347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fac1e9fc0 0x7f2fac1436f0 unknown :-1 s=CLOSED pgs=2048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:34.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.880+0000 7f2fb1c00640 1 --2- 172.21.15.28:0/3161914347 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2f90004800 0x7f2fac1e9a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:34.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.880+0000 7f2fb1c00640 1 -- 172.21.15.28:0/3161914347 >> 172.21.15.28:0/3161914347 conn(0x7f2fac06b630 msgr2=0x7f2fac0c64e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:34.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.881+0000 7f2fb1c00640 1 -- 172.21.15.28:0/3161914347 shutdown_connections 2024-02-21T23:11:34.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:34.881+0000 7f2fb1c00640 1 -- 172.21.15.28:0/3161914347 wait complete. 2024-02-21T23:11:34.888 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-21T23:02:30.728329+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-21T23:11:34.888 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-21T23:11:35.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.100+0000 7fb0ff000640 1 Processor -- start 2024-02-21T23:11:35.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.101+0000 7fb0ff000640 1 -- start start 2024-02-21T23:11:35.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.102+0000 7fb0ff000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fb0f806b0e0 0x7fb0f80c5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:35.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.102+0000 7fb0ff000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb0f8150700 0x7fb0f8156500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:35.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.102+0000 7fb0ff000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0f80c61e0 con 0x7fb0f80c5d80 2024-02-21T23:11:35.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.102+0000 7fb0ff000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0f8153bc0 con 0x7fb0f8150700 2024-02-21T23:11:35.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.102+0000 7fb0ff000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0f8152fb0 con 0x7fb0f806b0e0 2024-02-21T23:11:35.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.102+0000 7fb0fd000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb0f8150700 0x7fb0f8156500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:35.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.103+0000 7fb0fd000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb0f8150700 0x7fb0f8156500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45686/0 (socket says 172.21.15.28:45686) 2024-02-21T23:11:35.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.103+0000 7fb0f7400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb0f80c5d80 0x7fb0f814fff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:40964/0 (socket says 172.21.15.28:40964) 2024-02-21T23:11:35.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.103+0000 7fb0fd000640 1 -- 172.21.15.28:0/2893007436 learned_addr learned my addr 172.21.15.28:0/2893007436 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:35.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.103+0000 7fb0f7e00640 1 --2- 172.21.15.28:0/2893007436 >> v2:172.21.15.81:3300/0 conn(0x7fb0f806b0e0 0x7fb0f80c5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:35.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.103+0000 7fb0fd000640 1 -- 172.21.15.28:0/2893007436 >> v2:172.21.15.81:3300/0 conn(0x7fb0f806b0e0 msgr2=0x7fb0f80c5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:35.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.103+0000 7fb0fd000640 1 --2- 172.21.15.28:0/2893007436 >> v2:172.21.15.81:3300/0 conn(0x7fb0f806b0e0 0x7fb0f80c5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:35.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.103+0000 7fb0fd000640 1 -- 172.21.15.28:0/2893007436 >> v1:172.21.15.28:6789/0 conn(0x7fb0f80c5d80 legacy=0x7fb0f814fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:35.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.103+0000 7fb0fd000640 1 -- 172.21.15.28:0/2893007436 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0f8153290 con 0x7fb0f8150700 2024-02-21T23:11:35.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.103+0000 7fb0f6a00640 1 -- 172.21.15.28:0/2893007436 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2663459371 0 0) 0x7fb0f80c61e0 con 0x7fb0f80c5d80 2024-02-21T23:11:35.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.104+0000 7fb0fd000640 1 --2- 172.21.15.28:0/2893007436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0f8150700 0x7fb0f8156500 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x7fb0f80c77e0 tx=0x7fb0f002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7a0775291757513 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:35.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.105+0000 7fb0f6a00640 1 -- 172.21.15.28:0/2893007436 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0f002ca80 con 0x7fb0f8150700 2024-02-21T23:11:35.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.105+0000 7fb0f6a00640 1 -- 172.21.15.28:0/2893007436 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0f002cc20 con 0x7fb0f8150700 2024-02-21T23:11:35.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.105+0000 7fb0f6a00640 1 -- 172.21.15.28:0/2893007436 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0f00343f0 con 0x7fb0f8150700 2024-02-21T23:11:35.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.105+0000 7fb0ff000640 1 -- 172.21.15.28:0/2893007436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0f8150700 msgr2=0x7fb0f8156500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:35.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.105+0000 7fb0ff000640 1 --2- 172.21.15.28:0/2893007436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0f8150700 0x7fb0f8156500 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x7fb0f80c77e0 tx=0x7fb0f002bee0 comp rx=0 tx=0).stop 2024-02-21T23:11:35.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.105+0000 7fb0ff000640 1 -- 172.21.15.28:0/2893007436 shutdown_connections 2024-02-21T23:11:35.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.105+0000 7fb0ff000640 1 --2- 172.21.15.28:0/2893007436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0f8150700 0x7fb0f8156500 unknown :-1 s=CLOSED pgs=2050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:35.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.105+0000 7fb0ff000640 1 --2- 172.21.15.28:0/2893007436 >> v2:172.21.15.81:3300/0 conn(0x7fb0f806b0e0 0x7fb0f80c5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:35.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.105+0000 7fb0ff000640 1 -- 172.21.15.28:0/2893007436 >> 172.21.15.28:0/2893007436 conn(0x7fb0f80a85a0 msgr2=0x7fb0f80a8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:35.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.105+0000 7fb0ff000640 1 -- 172.21.15.28:0/2893007436 shutdown_connections 2024-02-21T23:11:35.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.105+0000 7fb0ff000640 1 -- 172.21.15.28:0/2893007436 wait complete. 2024-02-21T23:11:35.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.107+0000 7fb0ff000640 1 Processor -- start 2024-02-21T23:11:35.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.107+0000 7fb0ff000640 1 -- start start 2024-02-21T23:11:35.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.108+0000 7fb0ff000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0f806b0e0 0x7fb0f8155740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:35.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.108+0000 7fb0ff000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb0f80c5d80 0x7fb0f80c2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:35.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.108+0000 7fb0ff000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0f8150700 0x7fb0f80c3000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:35.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.108+0000 7fb0ff000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0f806b4c0 con 0x7fb0f8150700 2024-02-21T23:11:35.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.108+0000 7fb0ff000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0f8153850 con 0x7fb0f806b0e0 2024-02-21T23:11:35.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.108+0000 7fb0ff000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0f806b640 con 0x7fb0f80c5d80 2024-02-21T23:11:35.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.108+0000 7fb0fd000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0f8150700 0x7fb0f80c3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:35.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.109+0000 7fb0fd000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0f8150700 0x7fb0f80c3000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45694/0 (socket says 172.21.15.28:45694) 2024-02-21T23:11:35.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.109+0000 7fb0fd000640 1 -- 172.21.15.28:0/3177735550 learned_addr learned my addr 172.21.15.28:0/3177735550 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:35.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.109+0000 7fb0f7e00640 1 --2- 172.21.15.28:0/3177735550 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0f806b0e0 0x7fb0f8155740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:35.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.109+0000 7fb0f7400640 1 --2- 172.21.15.28:0/3177735550 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb0f80c5d80 0x7fb0f80c2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:35.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.109+0000 7fb0fd000640 1 -- 172.21.15.28:0/3177735550 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb0f80c5d80 msgr2=0x7fb0f80c2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:35.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.109+0000 7fb0fd000640 1 --2- 172.21.15.28:0/3177735550 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb0f80c5d80 0x7fb0f80c2ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:35.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.109+0000 7fb0fd000640 1 -- 172.21.15.28:0/3177735550 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0f806b0e0 msgr2=0x7fb0f8155740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:35.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.109+0000 7fb0fd000640 1 --2- 172.21.15.28:0/3177735550 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0f806b0e0 0x7fb0f8155740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:35.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.109+0000 7fb0fd000640 1 -- 172.21.15.28:0/3177735550 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0f814d840 con 0x7fb0f8150700 2024-02-21T23:11:35.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.109+0000 7fb0f7e00640 1 --2- 172.21.15.28:0/3177735550 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0f806b0e0 0x7fb0f8155740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:11:35.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.110+0000 7fb0fd000640 1 --2- 172.21.15.28:0/3177735550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0f8150700 0x7fb0f80c3000 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7fb0f0036040 tx=0x7fb0f0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:35.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.110+0000 7fb0efe00640 1 -- 172.21.15.28:0/3177735550 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0f0043020 con 0x7fb0f8150700 2024-02-21T23:11:35.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.110+0000 7fb0efe00640 1 -- 172.21.15.28:0/3177735550 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0f003e450 con 0x7fb0f8150700 2024-02-21T23:11:35.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.110+0000 7fb0ff000640 1 -- 172.21.15.28:0/3177735550 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0f81e8470 con 0x7fb0f8150700 2024-02-21T23:11:35.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.110+0000 7fb0efe00640 1 -- 172.21.15.28:0/3177735550 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0f0009510 con 0x7fb0f8150700 2024-02-21T23:11:35.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.110+0000 7fb0ff000640 1 -- 172.21.15.28:0/3177735550 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0f81e8730 con 0x7fb0f8150700 2024-02-21T23:11:35.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.112+0000 7fb0ff000640 1 -- 172.21.15.28:0/3177735550 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0f8067f30 con 0x7fb0f8150700 2024-02-21T23:11:35.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.112+0000 7fb0efe00640 1 -- 172.21.15.28:0/3177735550 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb0f0005d10 con 0x7fb0f8150700 2024-02-21T23:11:35.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.113+0000 7fb0efe00640 1 --2- 172.21.15.28:0/3177735550 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb0d806b4d0 0x7fb0d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:35.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.113+0000 7fb0efe00640 1 -- 172.21.15.28:0/3177735550 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fb0f00acb40 con 0x7fb0f8150700 2024-02-21T23:11:35.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.114+0000 7fb0f7e00640 1 --2- 172.21.15.28:0/3177735550 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb0d806b4d0 0x7fb0d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:35.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.115+0000 7fb0f7e00640 1 --2- 172.21.15.28:0/3177735550 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb0d806b4d0 0x7fb0d806d990 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fb0e8006fd0 tx=0x7fb0e8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:35.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.118+0000 7fb0efe00640 1 -- 172.21.15.28:0/3177735550 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb0f814d840 con 0x7fb0f8150700 2024-02-21T23:11:35.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.325+0000 7fb0ff000640 1 -- 172.21.15.28:0/3177735550 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb0f80ab8c0 con 0x7fb0f8150700 2024-02-21T23:11:35.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.326+0000 7fb0efe00640 1 -- 172.21.15.28:0/3177735550 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v182) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb0f007bb60 con 0x7fb0f8150700 2024-02-21T23:11:35.313 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-02-21T23:11:35.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.328+0000 7fb0ff000640 1 -- 172.21.15.28:0/3177735550 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb0d806b4d0 msgr2=0x7fb0d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:35.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.328+0000 7fb0ff000640 1 --2- 172.21.15.28:0/3177735550 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb0d806b4d0 0x7fb0d806d990 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fb0e8006fd0 tx=0x7fb0e8008040 comp rx=0 tx=0).stop 2024-02-21T23:11:35.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.328+0000 7fb0ff000640 1 -- 172.21.15.28:0/3177735550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0f8150700 msgr2=0x7fb0f80c3000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:35.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.328+0000 7fb0ff000640 1 --2- 172.21.15.28:0/3177735550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0f8150700 0x7fb0f80c3000 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7fb0f0036040 tx=0x7fb0f0004a40 comp rx=0 tx=0).stop 2024-02-21T23:11:35.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.329+0000 7fb0ff000640 1 -- 172.21.15.28:0/3177735550 shutdown_connections 2024-02-21T23:11:35.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.329+0000 7fb0ff000640 1 --2- 172.21.15.28:0/3177735550 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb0d806b4d0 0x7fb0d806d990 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:35.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.329+0000 7fb0ff000640 1 --2- 172.21.15.28:0/3177735550 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb0f80c5d80 0x7fb0f80c2ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:35.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.329+0000 7fb0ff000640 1 --2- 172.21.15.28:0/3177735550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0f8150700 0x7fb0f80c3000 unknown :-1 s=CLOSED pgs=2051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:35.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.329+0000 7fb0ff000640 1 --2- 172.21.15.28:0/3177735550 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0f806b0e0 0x7fb0f8155740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:35.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.329+0000 7fb0ff000640 1 -- 172.21.15.28:0/3177735550 >> 172.21.15.28:0/3177735550 conn(0x7fb0f80a85a0 msgr2=0x7fb0f80a6ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:35.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.329+0000 7fb0ff000640 1 -- 172.21.15.28:0/3177735550 shutdown_connections 2024-02-21T23:11:35.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.329+0000 7fb0ff000640 1 -- 172.21.15.28:0/3177735550 wait complete. 2024-02-21T23:11:35.335 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-02-21T23:11:35.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.566+0000 7f64d3e00640 1 Processor -- start 2024-02-21T23:11:35.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.566+0000 7f64d3e00640 1 -- start start 2024-02-21T23:11:35.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.567+0000 7f64d3e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f64d4150450 0x7f64d406e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:35.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.567+0000 7f64d3e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f64d40c2bb0 0x7f64d4152c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:35.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.567+0000 7f64d3e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64d40a6540 con 0x7f64d406e550 2024-02-21T23:11:35.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.567+0000 7f64d3e00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64d40c6df0 con 0x7f64d40c2bb0 2024-02-21T23:11:35.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.567+0000 7f64d3e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64d40c26d0 con 0x7f64d4150450 2024-02-21T23:11:35.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.568+0000 7f64cbe00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f64d40c2bb0 0x7f64d4152c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:35.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.568+0000 7f64d2a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f64d4150450 0x7f64d406e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:35.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.568+0000 7f64cbe00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f64d40c2bb0 0x7f64d4152c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.28:37972/0 (socket says 172.21.15.28:37972) 2024-02-21T23:11:35.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.568+0000 7f64cbe00640 1 -- 172.21.15.28:0/4175843082 learned_addr learned my addr 172.21.15.28:0/4175843082 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:35.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.569+0000 7f64d2a00640 1 -- 172.21.15.28:0/4175843082 >> v2:172.21.15.81:3300/0 conn(0x7f64d40c2bb0 msgr2=0x7f64d4152c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:35.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.569+0000 7f64d2a00640 1 --2- 172.21.15.28:0/4175843082 >> v2:172.21.15.81:3300/0 conn(0x7f64d40c2bb0 0x7f64d4152c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:35.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.569+0000 7f64d2a00640 1 -- 172.21.15.28:0/4175843082 >> v1:172.21.15.81:6789/0 conn(0x7f64d406e550 legacy=0x7f64d40c0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:35.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.569+0000 7f64d2a00640 1 -- 172.21.15.28:0/4175843082 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64d40c7090 con 0x7f64d4150450 2024-02-21T23:11:35.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.570+0000 7f64d2a00640 1 --2- 172.21.15.28:0/4175843082 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f64d4150450 0x7f64d406e010 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f64c40097a0 tx=0x7f64c402b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0425ad6c7acb714 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:35.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.571+0000 7f64d2000640 1 -- 172.21.15.28:0/4175843082 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64c40315a0 con 0x7f64d4150450 2024-02-21T23:11:35.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.571+0000 7f64d2000640 1 -- 172.21.15.28:0/4175843082 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64c4037070 con 0x7f64d4150450 2024-02-21T23:11:35.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.571+0000 7f64d2000640 1 -- 172.21.15.28:0/4175843082 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64c4031960 con 0x7f64d4150450 2024-02-21T23:11:35.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.571+0000 7f64d3e00640 1 -- 172.21.15.28:0/4175843082 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f64d4150450 msgr2=0x7f64d406e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:35.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.571+0000 7f64d3e00640 1 --2- 172.21.15.28:0/4175843082 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f64d4150450 0x7f64d406e010 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f64c40097a0 tx=0x7f64c402b840 comp rx=0 tx=0).stop 2024-02-21T23:11:35.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.572+0000 7f64d3e00640 1 -- 172.21.15.28:0/4175843082 shutdown_connections 2024-02-21T23:11:35.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.572+0000 7f64d3e00640 1 --2- 172.21.15.28:0/4175843082 >> v2:172.21.15.81:3300/0 conn(0x7f64d40c2bb0 0x7f64d4152c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:35.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.572+0000 7f64d3e00640 1 --2- 172.21.15.28:0/4175843082 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f64d4150450 0x7f64d406e010 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:35.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.572+0000 7f64d3e00640 1 -- 172.21.15.28:0/4175843082 >> 172.21.15.28:0/4175843082 conn(0x7f64d406b550 msgr2=0x7f64d40c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:35.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.572+0000 7f64d3e00640 1 -- 172.21.15.28:0/4175843082 shutdown_connections 2024-02-21T23:11:35.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.572+0000 7f64d3e00640 1 -- 172.21.15.28:0/4175843082 wait complete. 2024-02-21T23:11:35.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.573+0000 7f64d3e00640 1 Processor -- start 2024-02-21T23:11:35.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.574+0000 7f64d3e00640 1 -- start start 2024-02-21T23:11:35.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.574+0000 7f64d3e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f64d406e550 0x7f64d414ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:35.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.574+0000 7f64d3e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64d40c2bb0 0x7f64d41496e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:35.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.574+0000 7f64d3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64d4150450 0x7f64d4149c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:35.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.574+0000 7f64d3e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64d40a7b80 con 0x7f64d4150450 2024-02-21T23:11:35.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.574+0000 7f64d3e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64d40a7a00 con 0x7f64d40c2bb0 2024-02-21T23:11:35.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.574+0000 7f64d3e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64d40a5f80 con 0x7f64d406e550 2024-02-21T23:11:35.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.575+0000 7f64d3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64d4150450 0x7f64d4149c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:35.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.575+0000 7f64d3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64d4150450 0x7f64d4149c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45708/0 (socket says 172.21.15.28:45708) 2024-02-21T23:11:35.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.575+0000 7f64d3400640 1 -- 172.21.15.28:0/164133743 learned_addr learned my addr 172.21.15.28:0/164133743 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:35.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.575+0000 7f64d2a00640 1 --2- 172.21.15.28:0/164133743 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f64d406e550 0x7f64d414ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:35.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.575+0000 7f64cbe00640 1 --2- 172.21.15.28:0/164133743 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64d40c2bb0 0x7f64d41496e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:35.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.575+0000 7f64d3400640 1 -- 172.21.15.28:0/164133743 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f64d406e550 msgr2=0x7f64d414ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:35.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.575+0000 7f64d3400640 1 --2- 172.21.15.28:0/164133743 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f64d406e550 0x7f64d414ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:35.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.575+0000 7f64d3400640 1 -- 172.21.15.28:0/164133743 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64d40c2bb0 msgr2=0x7f64d41496e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:35.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.575+0000 7f64d3400640 1 --2- 172.21.15.28:0/164133743 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64d40c2bb0 0x7f64d41496e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:35.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.575+0000 7f64d3400640 1 -- 172.21.15.28:0/164133743 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64d41ebc80 con 0x7f64d4150450 2024-02-21T23:11:35.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.576+0000 7f64d3400640 1 --2- 172.21.15.28:0/164133743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64d4150450 0x7f64d4149c20 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 crypto rx=0x7f64cc00c7d0 tx=0x7f64cc00cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:35.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.576+0000 7f64caa00640 1 -- 172.21.15.28:0/164133743 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64cc003b70 con 0x7f64d4150450 2024-02-21T23:11:35.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.576+0000 7f64caa00640 1 -- 172.21.15.28:0/164133743 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64cc003d30 con 0x7f64d4150450 2024-02-21T23:11:35.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.576+0000 7f64caa00640 1 -- 172.21.15.28:0/164133743 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64cc00aa90 con 0x7f64d4150450 2024-02-21T23:11:35.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.576+0000 7f64d3e00640 1 -- 172.21.15.28:0/164133743 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64d41eeb70 con 0x7f64d4150450 2024-02-21T23:11:35.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.576+0000 7f64d3e00640 1 -- 172.21.15.28:0/164133743 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64d41ec200 con 0x7f64d4150450 2024-02-21T23:11:35.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.577+0000 7f64d3e00640 1 -- 172.21.15.28:0/164133743 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64d40a5bf0 con 0x7f64d4150450 2024-02-21T23:11:35.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.578+0000 7f64caa00640 1 -- 172.21.15.28:0/164133743 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f64cc00ac30 con 0x7f64d4150450 2024-02-21T23:11:35.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.579+0000 7f64caa00640 1 --2- 172.21.15.28:0/164133743 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f64b006b4d0 0x7f64b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:35.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.579+0000 7f64caa00640 1 -- 172.21.15.28:0/164133743 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f64cc08a0a0 con 0x7f64d4150450 2024-02-21T23:11:35.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.580+0000 7f64d2a00640 1 --2- 172.21.15.28:0/164133743 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f64b006b4d0 0x7f64b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:35.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.581+0000 7f64d2a00640 1 --2- 172.21.15.28:0/164133743 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f64b006b4d0 0x7f64b006d990 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f64c4036000 tx=0x7f64c4008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:35.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.584+0000 7f64caa00640 1 -- 172.21.15.28:0/164133743 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f64d40a5bf0 con 0x7f64d4150450 2024-02-21T23:11:35.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.786+0000 7f64d3e00640 1 -- 172.21.15.28:0/164133743 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f64d404f290 con 0x7f64d4150450 2024-02-21T23:11:35.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.787+0000 7f64caa00640 1 -- 172.21.15.28:0/164133743 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v182) v1 ==== 84+0+65 (secure 0 0 0) 0x7f64cc058fb0 con 0x7f64d4150450 2024-02-21T23:11:35.775 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:11:35.775 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-02-21T23:11:35.775 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:11:35.775 INFO:teuthology.orchestra.run.smithi028.stdout: "poolnum": 1, 2024-02-21T23:11:35.775 INFO:teuthology.orchestra.run.smithi028.stdout: "poolname": ".mgr" 2024-02-21T23:11:35.775 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:11:35.775 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-02-21T23:11:35.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.789+0000 7f64d3e00640 1 -- 172.21.15.28:0/164133743 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f64b006b4d0 msgr2=0x7f64b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:35.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.789+0000 7f64d3e00640 1 --2- 172.21.15.28:0/164133743 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f64b006b4d0 0x7f64b006d990 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f64c4036000 tx=0x7f64c4008030 comp rx=0 tx=0).stop 2024-02-21T23:11:35.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.789+0000 7f64d3e00640 1 -- 172.21.15.28:0/164133743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64d4150450 msgr2=0x7f64d4149c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:35.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.789+0000 7f64d3e00640 1 --2- 172.21.15.28:0/164133743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64d4150450 0x7f64d4149c20 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 crypto rx=0x7f64cc00c7d0 tx=0x7f64cc00cc90 comp rx=0 tx=0).stop 2024-02-21T23:11:35.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.790+0000 7f64d3e00640 1 -- 172.21.15.28:0/164133743 shutdown_connections 2024-02-21T23:11:35.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.790+0000 7f64d3e00640 1 --2- 172.21.15.28:0/164133743 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f64b006b4d0 0x7f64b006d990 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:35.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.790+0000 7f64d3e00640 1 --2- 172.21.15.28:0/164133743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64d4150450 0x7f64d4149c20 unknown :-1 s=CLOSED pgs=2052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:35.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.790+0000 7f64d3e00640 1 --2- 172.21.15.28:0/164133743 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64d40c2bb0 0x7f64d41496e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:35.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.790+0000 7f64d3e00640 1 --2- 172.21.15.28:0/164133743 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f64d406e550 0x7f64d414ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:35.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.790+0000 7f64d3e00640 1 -- 172.21.15.28:0/164133743 >> 172.21.15.28:0/164133743 conn(0x7f64d406b550 msgr2=0x7f64d414f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:35.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.790+0000 7f64d3e00640 1 -- 172.21.15.28:0/164133743 shutdown_connections 2024-02-21T23:11:35.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:35.790+0000 7f64d3e00640 1 -- 172.21.15.28:0/164133743 wait complete. 2024-02-21T23:11:35.797 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-02-21T23:11:35.798 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-02-21T23:11:36.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.041+0000 7f845aa00640 1 Processor -- start 2024-02-21T23:11:36.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.041+0000 7f845aa00640 1 -- start start 2024-02-21T23:11:36.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.043+0000 7f845aa00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8454158460 0x7f84540c6290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:36.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.043+0000 7f845aa00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84540caf40 con 0x7f84540ca540 2024-02-21T23:11:36.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.043+0000 7f845aa00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84540c87b0 con 0x7f845406cfa0 2024-02-21T23:11:36.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.043+0000 7f845aa00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84540c8ae0 con 0x7f8454158460 2024-02-21T23:11:36.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.043+0000 7f8458c00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f84540ca540 0x7f84541522c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:40972/0 (socket says 172.21.15.28:40972) 2024-02-21T23:11:36.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.043+0000 7f8458c00640 1 -- 172.21.15.28:0/369782904 learned_addr learned my addr 172.21.15.28:0/369782904 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:11:36.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.044+0000 7f845a000640 1 --2- 172.21.15.28:0/369782904 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8454158460 0x7f84540c6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:36.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.044+0000 7f8453e00640 1 -- 172.21.15.28:0/369782904 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1016657322 0 0) 0x7f84540caf40 con 0x7f84540ca540 2024-02-21T23:11:36.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.044+0000 7f8453e00640 1 -- 172.21.15.28:0/369782904 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f843c003520 con 0x7f84540ca540 2024-02-21T23:11:36.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.044+0000 7f8453e00640 1 -- 172.21.15.28:0/369782904 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1934898514 0 0) 0x7f843c003520 con 0x7f84540ca540 2024-02-21T23:11:36.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.044+0000 7f8453e00640 1 -- 172.21.15.28:0/369782904 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8454158460 msgr2=0x7f84540c6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:36.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.044+0000 7f8453e00640 1 --2- 172.21.15.28:0/369782904 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8454158460 0x7f84540c6290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:36.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.044+0000 7f8453e00640 1 -- 172.21.15.28:0/369782904 >> v1:172.21.15.81:6789/0 conn(0x7f845406cfa0 legacy=0x7f84540c8f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:36.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.044+0000 7f8453e00640 1 -- 172.21.15.28:0/369782904 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84540c7f80 con 0x7f84540ca540 2024-02-21T23:11:36.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.044+0000 7f8453e00640 1 -- 172.21.15.28:0/369782904 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f8440001670 con 0x7f84540ca540 2024-02-21T23:11:36.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.044+0000 7f8453e00640 1 -- 172.21.15.28:0/369782904 >> v1:172.21.15.28:6789/0 conn(0x7f84540ca540 legacy=0x7f84541522c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:36.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.044+0000 7f8453e00640 1 --2- 172.21.15.28:0/369782904 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f843c0039e0 0x7f843c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:36.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.044+0000 7f8453e00640 1 --2- 172.21.15.28:0/369782904 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8454158460 0x7f843c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:36.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.044+0000 7f8453e00640 1 --2- 172.21.15.28:0/369782904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f843c004d40 0x7f843c0051a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:36.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.044+0000 7f8453e00640 1 -- 172.21.15.28:0/369782904 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84540c8ae0 con 0x7f843c004d40 2024-02-21T23:11:36.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.044+0000 7f8453e00640 1 -- 172.21.15.28:0/369782904 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f843c00bbd0 con 0x7f843c0039e0 2024-02-21T23:11:36.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.044+0000 7f8453e00640 1 -- 172.21.15.28:0/369782904 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f843c00bdd0 con 0x7f8454158460 2024-02-21T23:11:36.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.045+0000 7f845a000640 1 --2- 172.21.15.28:0/369782904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f843c004d40 0x7f843c0051a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:36.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.045+0000 7f8459600640 1 --2- 172.21.15.28:0/369782904 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f843c0039e0 0x7f843c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:36.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.045+0000 7f8458c00640 1 --2- 172.21.15.28:0/369782904 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8454158460 0x7f843c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:36.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.045+0000 7f845a000640 1 -- 172.21.15.28:0/369782904 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8454158460 msgr2=0x7f843c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:36.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.045+0000 7f845a000640 1 --2- 172.21.15.28:0/369782904 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8454158460 0x7f843c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:36.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.045+0000 7f845a000640 1 -- 172.21.15.28:0/369782904 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f843c0039e0 msgr2=0x7f843c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:36.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.045+0000 7f845a000640 1 --2- 172.21.15.28:0/369782904 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f843c0039e0 0x7f843c003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:36.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.045+0000 7f845a000640 1 -- 172.21.15.28:0/369782904 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f843c003520 con 0x7f843c004d40 2024-02-21T23:11:36.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.045+0000 7f8458c00640 1 --2- 172.21.15.28:0/369782904 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8454158460 0x7f843c004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:11:36.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.046+0000 7f845a000640 1 --2- 172.21.15.28:0/369782904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f843c004d40 0x7f843c0051a0 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7f844c009fc0 tx=0x7f843c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a58df3837ea76585 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:36.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.047+0000 7f8453e00640 1 -- 172.21.15.28:0/369782904 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f844c039070 con 0x7f843c004d40 2024-02-21T23:11:36.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.047+0000 7f8453e00640 1 -- 172.21.15.28:0/369782904 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f844c02cbf0 con 0x7f843c004d40 2024-02-21T23:11:36.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.047+0000 7f8453e00640 1 -- 172.21.15.28:0/369782904 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f844c009c10 con 0x7f843c004d40 2024-02-21T23:11:36.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.047+0000 7f845aa00640 1 -- 172.21.15.28:0/369782904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f843c004d40 msgr2=0x7f843c0051a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:36.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.047+0000 7f845aa00640 1 --2- 172.21.15.28:0/369782904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f843c004d40 0x7f843c0051a0 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7f844c009fc0 tx=0x7f843c005830 comp rx=0 tx=0).stop 2024-02-21T23:11:36.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.047+0000 7f845a000640 1 -- 172.21.15.28:0/369782904 reap_dead start 2024-02-21T23:11:36.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.047+0000 7f845aa00640 1 -- 172.21.15.28:0/369782904 shutdown_connections 2024-02-21T23:11:36.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.047+0000 7f845aa00640 1 -- 172.21.15.28:0/369782904 >> 172.21.15.28:0/369782904 conn(0x7f84540725a0 msgr2=0x7f84540729b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:36.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.047+0000 7f845aa00640 1 -- 172.21.15.28:0/369782904 shutdown_connections 2024-02-21T23:11:36.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.047+0000 7f845aa00640 1 -- 172.21.15.28:0/369782904 wait complete. 2024-02-21T23:11:36.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.049+0000 7f845aa00640 1 Processor -- start 2024-02-21T23:11:36.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.049+0000 7f845aa00640 1 -- start start 2024-02-21T23:11:36.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.050+0000 7f845aa00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f845414c710 0x7f84541e1b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:36.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.050+0000 7f845aa00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84541531a0 0x7f84540aa900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:36.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.050+0000 7f845aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84540aae40 0x7f845414a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:36.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.050+0000 7f845aa00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8454158870 con 0x7f84540aae40 2024-02-21T23:11:36.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.050+0000 7f845aa00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84540c6c90 con 0x7f84541531a0 2024-02-21T23:11:36.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.050+0000 7f845aa00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84541529d0 con 0x7f845414c710 2024-02-21T23:11:36.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.050+0000 7f845a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84540aae40 0x7f845414a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:36.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.051+0000 7f845a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84540aae40 0x7f845414a460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45726/0 (socket says 172.21.15.28:45726) 2024-02-21T23:11:36.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.051+0000 7f845a000640 1 -- 172.21.15.28:0/3738040027 learned_addr learned my addr 172.21.15.28:0/3738040027 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:36.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.051+0000 7f8459600640 1 --2- 172.21.15.28:0/3738040027 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f845414c710 0x7f84541e1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:36.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.051+0000 7f8458c00640 1 --2- 172.21.15.28:0/3738040027 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84541531a0 0x7f84540aa900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:36.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.051+0000 7f845a000640 1 -- 172.21.15.28:0/3738040027 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f845414c710 msgr2=0x7f84541e1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:36.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.051+0000 7f845a000640 1 --2- 172.21.15.28:0/3738040027 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f845414c710 0x7f84541e1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:36.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.051+0000 7f845a000640 1 -- 172.21.15.28:0/3738040027 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84541531a0 msgr2=0x7f84540aa900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:36.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.051+0000 7f845a000640 1 --2- 172.21.15.28:0/3738040027 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84541531a0 0x7f84540aa900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:36.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.051+0000 7f845a000640 1 -- 172.21.15.28:0/3738040027 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8454152f80 con 0x7f84540aae40 2024-02-21T23:11:36.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.051+0000 7f845a000640 1 --2- 172.21.15.28:0/3738040027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84540aae40 0x7f845414a460 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7f844c00e230 tx=0x7f844c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:36.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.052+0000 7f8451600640 1 -- 172.21.15.28:0/3738040027 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f844c042030 con 0x7f84540aae40 2024-02-21T23:11:36.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.052+0000 7f8451600640 1 -- 172.21.15.28:0/3738040027 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f844c03a040 con 0x7f84540aae40 2024-02-21T23:11:36.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.052+0000 7f8451600640 1 -- 172.21.15.28:0/3738040027 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f844c039070 con 0x7f84540aae40 2024-02-21T23:11:36.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.052+0000 7f845aa00640 1 -- 172.21.15.28:0/3738040027 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84541555d0 con 0x7f84540aae40 2024-02-21T23:11:36.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.052+0000 7f845aa00640 1 -- 172.21.15.28:0/3738040027 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8454155b90 con 0x7f84540aae40 2024-02-21T23:11:36.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.053+0000 7f845aa00640 1 -- 172.21.15.28:0/3738040027 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f845404f290 con 0x7f84540aae40 2024-02-21T23:11:36.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.054+0000 7f8451600640 1 -- 172.21.15.28:0/3738040027 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f844c045070 con 0x7f84540aae40 2024-02-21T23:11:36.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.055+0000 7f8451600640 1 --2- 172.21.15.28:0/3738040027 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f843406b500 0x7f843406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:36.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.055+0000 7f8451600640 1 -- 172.21.15.28:0/3738040027 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f844c0abb00 con 0x7f84540aae40 2024-02-21T23:11:36.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.056+0000 7f8459600640 1 --2- 172.21.15.28:0/3738040027 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f843406b500 0x7f843406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:36.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.057+0000 7f8459600640 1 --2- 172.21.15.28:0/3738040027 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f843406b500 0x7f843406d9c0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f8448007fd0 tx=0x7f8448009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:36.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.060+0000 7f8451600640 1 -- 172.21.15.28:0/3738040027 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f844c04c020 con 0x7f84540aae40 2024-02-21T23:11:36.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.259+0000 7f845aa00640 1 -- 172.21.15.28:0/3738040027 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f8454072a50 con 0x7f84540aae40 2024-02-21T23:11:36.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.889+0000 7f8451600640 1 -- 172.21.15.28:0/3738040027 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v183) v1 ==== 138+0+0 (secure 0 0 0) 0x7f844c07ab30 con 0x7f84540aae40 2024-02-21T23:11:36.877 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' created 2024-02-21T23:11:36.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.893+0000 7f845aa00640 1 -- 172.21.15.28:0/3738040027 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f843406b500 msgr2=0x7f843406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:36.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.893+0000 7f845aa00640 1 --2- 172.21.15.28:0/3738040027 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f843406b500 0x7f843406d9c0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f8448007fd0 tx=0x7f8448009040 comp rx=0 tx=0).stop 2024-02-21T23:11:36.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.893+0000 7f845aa00640 1 -- 172.21.15.28:0/3738040027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84540aae40 msgr2=0x7f845414a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:36.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.893+0000 7f845aa00640 1 --2- 172.21.15.28:0/3738040027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84540aae40 0x7f845414a460 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7f844c00e230 tx=0x7f844c009750 comp rx=0 tx=0).stop 2024-02-21T23:11:36.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.893+0000 7f845aa00640 1 -- 172.21.15.28:0/3738040027 shutdown_connections 2024-02-21T23:11:36.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.893+0000 7f845aa00640 1 --2- 172.21.15.28:0/3738040027 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f843406b500 0x7f843406d9c0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:36.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.893+0000 7f845aa00640 1 --2- 172.21.15.28:0/3738040027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84540aae40 0x7f845414a460 unknown :-1 s=CLOSED pgs=2055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:36.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.893+0000 7f845aa00640 1 --2- 172.21.15.28:0/3738040027 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84541531a0 0x7f84540aa900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:36.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.893+0000 7f845aa00640 1 --2- 172.21.15.28:0/3738040027 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f845414c710 0x7f84541e1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:36.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.893+0000 7f845aa00640 1 -- 172.21.15.28:0/3738040027 >> 172.21.15.28:0/3738040027 conn(0x7f84540725a0 msgr2=0x7f84540af290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:36.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.893+0000 7f845aa00640 1 -- 172.21.15.28:0/3738040027 shutdown_connections 2024-02-21T23:11:36.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:36.893+0000 7f845aa00640 1 -- 172.21.15.28:0/3738040027 wait complete. 2024-02-21T23:11:36.908 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-02-21T23:11:37.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.144+0000 7f2890a00640 1 Processor -- start 2024-02-21T23:11:37.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.144+0000 7f2890a00640 1 -- start start 2024-02-21T23:11:37.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.146+0000 7f2890a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f288c0c35c0 0x7f288c156bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:37.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.146+0000 7f2890a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f288c0a96c0 0x7f288c14ff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:37.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.146+0000 7f2890a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f288c06f1b0 con 0x7f288c150470 2024-02-21T23:11:37.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.146+0000 7f2890a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f288c06ea40 con 0x7f288c0c35c0 2024-02-21T23:11:37.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.146+0000 7f2890a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f288c0c2ae0 con 0x7f288c0a96c0 2024-02-21T23:11:37.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.146+0000 7f288a800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f288c0c35c0 0x7f288c156bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:37.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.146+0000 7f288a800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f288c0c35c0 0x7f288c156bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45742/0 (socket says 172.21.15.28:45742) 2024-02-21T23:11:37.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.146+0000 7f288a800640 1 -- 172.21.15.28:0/4133320777 learned_addr learned my addr 172.21.15.28:0/4133320777 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:37.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.147+0000 7f2881e00640 1 --2- 172.21.15.28:0/4133320777 >> v2:172.21.15.81:3300/0 conn(0x7f288c0a96c0 0x7f288c14ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:37.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.147+0000 7f288a800640 1 -- 172.21.15.28:0/4133320777 >> v2:172.21.15.81:3300/0 conn(0x7f288c0a96c0 msgr2=0x7f288c14ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:37.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.147+0000 7f288a800640 1 --2- 172.21.15.28:0/4133320777 >> v2:172.21.15.81:3300/0 conn(0x7f288c0a96c0 0x7f288c14ff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:37.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.147+0000 7f288a800640 1 -- 172.21.15.28:0/4133320777 >> v1:172.21.15.81:6789/0 conn(0x7f288c150470 legacy=0x7f288c152c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:37.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.147+0000 7f288a800640 1 -- 172.21.15.28:0/4133320777 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f288c0c2dc0 con 0x7f288c0c35c0 2024-02-21T23:11:37.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.147+0000 7f2881e00640 1 --2- 172.21.15.28:0/4133320777 >> v2:172.21.15.81:3300/0 conn(0x7f288c0a96c0 0x7f288c14ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:11:37.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.148+0000 7f288a800640 1 --2- 172.21.15.28:0/4133320777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f288c0c35c0 0x7f288c156bf0 secure :-1 s=READY pgs=2056 cs=0 l=1 rev1=1 crypto rx=0x7f288c06f830 tx=0x7f287c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a23bb5677321907 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:37.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.149+0000 7f2889e00640 1 -- 172.21.15.28:0/4133320777 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f287c02e070 con 0x7f288c0c35c0 2024-02-21T23:11:37.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.149+0000 7f2889e00640 1 -- 172.21.15.28:0/4133320777 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f287c037070 con 0x7f288c0c35c0 2024-02-21T23:11:37.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.149+0000 7f2889e00640 1 -- 172.21.15.28:0/4133320777 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f287c031a00 con 0x7f288c0c35c0 2024-02-21T23:11:37.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.149+0000 7f2890a00640 1 -- 172.21.15.28:0/4133320777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f288c0c35c0 msgr2=0x7f288c156bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:37.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.149+0000 7f2890a00640 1 --2- 172.21.15.28:0/4133320777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f288c0c35c0 0x7f288c156bf0 secure :-1 s=READY pgs=2056 cs=0 l=1 rev1=1 crypto rx=0x7f288c06f830 tx=0x7f287c02b880 comp rx=0 tx=0).stop 2024-02-21T23:11:37.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.149+0000 7f2890a00640 1 -- 172.21.15.28:0/4133320777 shutdown_connections 2024-02-21T23:11:37.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.149+0000 7f2890a00640 1 --2- 172.21.15.28:0/4133320777 >> v2:172.21.15.81:3300/0 conn(0x7f288c0a96c0 0x7f288c14ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:37.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.149+0000 7f2890a00640 1 --2- 172.21.15.28:0/4133320777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f288c0c35c0 0x7f288c156bf0 unknown :-1 s=CLOSED pgs=2056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:37.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.149+0000 7f2890a00640 1 -- 172.21.15.28:0/4133320777 >> 172.21.15.28:0/4133320777 conn(0x7f288c0a8690 msgr2=0x7f288c0a8a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:37.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.150+0000 7f2890a00640 1 -- 172.21.15.28:0/4133320777 shutdown_connections 2024-02-21T23:11:37.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.150+0000 7f2890a00640 1 -- 172.21.15.28:0/4133320777 wait complete. 2024-02-21T23:11:37.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.151+0000 7f2890a00640 1 Processor -- start 2024-02-21T23:11:37.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.151+0000 7f2890a00640 1 -- start start 2024-02-21T23:11:37.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.152+0000 7f2890a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f288c0a96c0 0x7f288c1549a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:37.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.152+0000 7f2890a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f288c0c35c0 0x7f288c1eddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:37.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.152+0000 7f2890a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f288c150470 0x7f288c0c5c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:37.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.152+0000 7f2890a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f288c06e7b0 con 0x7f288c0c35c0 2024-02-21T23:11:37.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.152+0000 7f2890a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f288c150870 con 0x7f288c0a96c0 2024-02-21T23:11:37.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.152+0000 7f2890a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f288c0c39a0 con 0x7f288c150470 2024-02-21T23:11:37.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.153+0000 7f2881e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f288c0c35c0 0x7f288c1eddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:37.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.153+0000 7f2881e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f288c0c35c0 0x7f288c1eddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45752/0 (socket says 172.21.15.28:45752) 2024-02-21T23:11:37.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.153+0000 7f2881e00640 1 -- 172.21.15.28:0/324855939 learned_addr learned my addr 172.21.15.28:0/324855939 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:37.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.153+0000 7f288a800640 1 --2- 172.21.15.28:0/324855939 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f288c0a96c0 0x7f288c1549a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:37.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.153+0000 7f288b200640 1 --2- 172.21.15.28:0/324855939 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f288c150470 0x7f288c0c5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:37.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.153+0000 7f2881e00640 1 -- 172.21.15.28:0/324855939 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f288c150470 msgr2=0x7f288c0c5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:37.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.153+0000 7f2881e00640 1 --2- 172.21.15.28:0/324855939 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f288c150470 0x7f288c0c5c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:37.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.153+0000 7f2881e00640 1 -- 172.21.15.28:0/324855939 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f288c0a96c0 msgr2=0x7f288c1549a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:37.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.153+0000 7f2881e00640 1 --2- 172.21.15.28:0/324855939 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f288c0a96c0 0x7f288c1549a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:37.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.153+0000 7f2881e00640 1 -- 172.21.15.28:0/324855939 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f288c1ee620 con 0x7f288c0c35c0 2024-02-21T23:11:37.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.153+0000 7f288a800640 1 --2- 172.21.15.28:0/324855939 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f288c0a96c0 0x7f288c1549a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:11:37.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.154+0000 7f2881e00640 1 --2- 172.21.15.28:0/324855939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f288c0c35c0 0x7f288c1eddd0 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7f287400c7f0 tx=0x7f287400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:37.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.154+0000 7f2883400640 1 -- 172.21.15.28:0/324855939 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2874015070 con 0x7f288c0c35c0 2024-02-21T23:11:37.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.154+0000 7f2883400640 1 -- 172.21.15.28:0/324855939 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2874007a30 con 0x7f288c0c35c0 2024-02-21T23:11:37.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.154+0000 7f2883400640 1 -- 172.21.15.28:0/324855939 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2874007e00 con 0x7f288c0c35c0 2024-02-21T23:11:37.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.154+0000 7f2890a00640 1 -- 172.21.15.28:0/324855939 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f288c1ee900 con 0x7f288c0c35c0 2024-02-21T23:11:37.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.154+0000 7f2890a00640 1 -- 172.21.15.28:0/324855939 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f288c1ee310 con 0x7f288c0c35c0 2024-02-21T23:11:37.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.155+0000 7f2890a00640 1 -- 172.21.15.28:0/324855939 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f288c04f290 con 0x7f288c0c35c0 2024-02-21T23:11:37.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.156+0000 7f2883400640 1 -- 172.21.15.28:0/324855939 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2874018070 con 0x7f288c0c35c0 2024-02-21T23:11:37.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.157+0000 7f2883400640 1 --2- 172.21.15.28:0/324855939 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f286806b500 0x7f286806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:37.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.158+0000 7f2883400640 1 -- 172.21.15.28:0/324855939 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 5619+0+0 (secure 0 0 0) 0x7f287408a130 con 0x7f288c0c35c0 2024-02-21T23:11:37.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.158+0000 7f288a800640 1 --2- 172.21.15.28:0/324855939 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f286806b500 0x7f286806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:37.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.159+0000 7f288a800640 1 --2- 172.21.15.28:0/324855939 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f286806b500 0x7f286806d9c0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f287c036000 tx=0x7f287c008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:37.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.163+0000 7f2883400640 1 -- 172.21.15.28:0/324855939 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f2874058ff0 con 0x7f288c0c35c0 2024-02-21T23:11:37.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.364+0000 7f2890a00640 1 -- 172.21.15.28:0/324855939 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f288c067fd0 con 0x7f288c0c35c0 2024-02-21T23:11:37.631 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-21T23:11:37.639 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-21T23:11:37.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.895+0000 7f2883400640 1 -- 172.21.15.28:0/324855939 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v184) v1 ==== 170+0+0 (secure 0 0 0) 0x7f28740588f0 con 0x7f288c0c35c0 2024-02-21T23:11:37.883 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' created 2024-02-21T23:11:37.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.899+0000 7f2890a00640 1 -- 172.21.15.28:0/324855939 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f286806b500 msgr2=0x7f286806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:37.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.899+0000 7f2890a00640 1 --2- 172.21.15.28:0/324855939 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f286806b500 0x7f286806d9c0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f287c036000 tx=0x7f287c008030 comp rx=0 tx=0).stop 2024-02-21T23:11:37.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.899+0000 7f2890a00640 1 -- 172.21.15.28:0/324855939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f288c0c35c0 msgr2=0x7f288c1eddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:37.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.899+0000 7f2890a00640 1 --2- 172.21.15.28:0/324855939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f288c0c35c0 0x7f288c1eddd0 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7f287400c7f0 tx=0x7f287400cbf0 comp rx=0 tx=0).stop 2024-02-21T23:11:37.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.900+0000 7f2890a00640 1 -- 172.21.15.28:0/324855939 shutdown_connections 2024-02-21T23:11:37.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.900+0000 7f2890a00640 1 --2- 172.21.15.28:0/324855939 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f286806b500 0x7f286806d9c0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:37.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.900+0000 7f2890a00640 1 --2- 172.21.15.28:0/324855939 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f288c150470 0x7f288c0c5c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:37.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.900+0000 7f2890a00640 1 --2- 172.21.15.28:0/324855939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f288c0c35c0 0x7f288c1eddd0 unknown :-1 s=CLOSED pgs=2057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:37.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.900+0000 7f2890a00640 1 --2- 172.21.15.28:0/324855939 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f288c0a96c0 0x7f288c1549a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:37.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.900+0000 7f2890a00640 1 -- 172.21.15.28:0/324855939 >> 172.21.15.28:0/324855939 conn(0x7f288c0a8690 msgr2=0x7f288c06b6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:37.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.900+0000 7f2890a00640 1 -- 172.21.15.28:0/324855939 shutdown_connections 2024-02-21T23:11:37.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:37.900+0000 7f2890a00640 1 -- 172.21.15.28:0/324855939 wait complete. 2024-02-21T23:11:37.920 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-02-21T23:11:38.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.120+0000 7fb761000640 1 Processor -- start 2024-02-21T23:11:38.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.120+0000 7fb761000640 1 -- start start 2024-02-21T23:11:38.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.122+0000 7fb761000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb75c0c22e0 0x7fb75c06d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:38.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.122+0000 7fb761000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fb75c1561a0 0x7fb75c06d660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:38.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.122+0000 7fb761000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb75c071970 con 0x7fb75c0c5a70 2024-02-21T23:11:38.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.122+0000 7fb761000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb75c072a10 con 0x7fb75c0c22e0 2024-02-21T23:11:38.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.122+0000 7fb761000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb75c072580 con 0x7fb75c1561a0 2024-02-21T23:11:38.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.122+0000 7fb759400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb75c0c22e0 0x7fb75c06d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:38.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.122+0000 7fb759400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb75c0c22e0 0x7fb75c06d120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45758/0 (socket says 172.21.15.28:45758) 2024-02-21T23:11:38.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.122+0000 7fb759400640 1 -- 172.21.15.28:0/423752659 learned_addr learned my addr 172.21.15.28:0/423752659 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:38.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.122+0000 7fb75a800640 1 --2- 172.21.15.28:0/423752659 >> v2:172.21.15.81:3300/0 conn(0x7fb75c1561a0 0x7fb75c06d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:38.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.122+0000 7fb759400640 1 -- 172.21.15.28:0/423752659 >> v2:172.21.15.81:3300/0 conn(0x7fb75c1561a0 msgr2=0x7fb75c06d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:38.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.123+0000 7fb759400640 1 --2- 172.21.15.28:0/423752659 >> v2:172.21.15.81:3300/0 conn(0x7fb75c1561a0 0x7fb75c06d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:38.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.123+0000 7fb759400640 1 -- 172.21.15.28:0/423752659 >> v1:172.21.15.81:6789/0 conn(0x7fb75c0c5a70 legacy=0x7fb75c155a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:38.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.123+0000 7fb759400640 1 -- 172.21.15.28:0/423752659 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb75c071c50 con 0x7fb75c0c22e0 2024-02-21T23:11:38.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.123+0000 7fb75a800640 1 --2- 172.21.15.28:0/423752659 >> v2:172.21.15.81:3300/0 conn(0x7fb75c1561a0 0x7fb75c06d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:11:38.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.124+0000 7fb759400640 1 --2- 172.21.15.28:0/423752659 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb75c0c22e0 0x7fb75c06d120 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7fb75c071310 tx=0x7fb74802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=829f72055971946 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:38.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.124+0000 7fb758a00640 1 -- 172.21.15.28:0/423752659 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb74802e070 con 0x7fb75c0c22e0 2024-02-21T23:11:38.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.124+0000 7fb758a00640 1 -- 172.21.15.28:0/423752659 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb748037070 con 0x7fb75c0c22e0 2024-02-21T23:11:38.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.124+0000 7fb758a00640 1 -- 172.21.15.28:0/423752659 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb748031a00 con 0x7fb75c0c22e0 2024-02-21T23:11:38.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.125+0000 7fb761000640 1 -- 172.21.15.28:0/423752659 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb75c0c22e0 msgr2=0x7fb75c06d120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:38.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.125+0000 7fb761000640 1 --2- 172.21.15.28:0/423752659 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb75c0c22e0 0x7fb75c06d120 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7fb75c071310 tx=0x7fb74802b880 comp rx=0 tx=0).stop 2024-02-21T23:11:38.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.125+0000 7fb761000640 1 -- 172.21.15.28:0/423752659 shutdown_connections 2024-02-21T23:11:38.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.125+0000 7fb761000640 1 --2- 172.21.15.28:0/423752659 >> v2:172.21.15.81:3300/0 conn(0x7fb75c1561a0 0x7fb75c06d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:38.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.125+0000 7fb761000640 1 --2- 172.21.15.28:0/423752659 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb75c0c22e0 0x7fb75c06d120 unknown :-1 s=CLOSED pgs=2058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:38.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.125+0000 7fb761000640 1 -- 172.21.15.28:0/423752659 >> 172.21.15.28:0/423752659 conn(0x7fb75c0a85a0 msgr2=0x7fb75c0a8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:38.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.125+0000 7fb761000640 1 -- 172.21.15.28:0/423752659 shutdown_connections 2024-02-21T23:11:38.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.125+0000 7fb761000640 1 -- 172.21.15.28:0/423752659 wait complete. 2024-02-21T23:11:38.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.127+0000 7fb761000640 1 Processor -- start 2024-02-21T23:11:38.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.127+0000 7fb761000640 1 -- start start 2024-02-21T23:11:38.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.128+0000 7fb761000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb75c0c22e0 0x7fb75c0bfc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:38.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.128+0000 7fb761000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb75c0c5a70 0x7fb75c1eb5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:38.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.128+0000 7fb761000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb75c1561a0 0x7fb75c1ebb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:38.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.128+0000 7fb761000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb75c06dba0 con 0x7fb75c0c22e0 2024-02-21T23:11:38.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.128+0000 7fb761000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb75c072210 con 0x7fb75c0c5a70 2024-02-21T23:11:38.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.128+0000 7fb761000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb75c0a6160 con 0x7fb75c1561a0 2024-02-21T23:11:38.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.128+0000 7fb759e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb75c0c22e0 0x7fb75c0bfc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:38.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.129+0000 7fb759e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb75c0c22e0 0x7fb75c0bfc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45774/0 (socket says 172.21.15.28:45774) 2024-02-21T23:11:38.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.129+0000 7fb759e00640 1 -- 172.21.15.28:0/1362039673 learned_addr learned my addr 172.21.15.28:0/1362039673 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:38.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.129+0000 7fb759e00640 1 -- 172.21.15.28:0/1362039673 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb75c1561a0 msgr2=0x7fb75c1ebb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:38.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.129+0000 7fb75a800640 1 --2- 172.21.15.28:0/1362039673 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb75c1561a0 0x7fb75c1ebb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:38.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.129+0000 7fb759400640 1 --2- 172.21.15.28:0/1362039673 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb75c0c5a70 0x7fb75c1eb5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:38.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.129+0000 7fb759e00640 1 --2- 172.21.15.28:0/1362039673 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb75c1561a0 0x7fb75c1ebb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:38.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.129+0000 7fb759e00640 1 -- 172.21.15.28:0/1362039673 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb75c0c5a70 msgr2=0x7fb75c1eb5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:38.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.129+0000 7fb759e00640 1 --2- 172.21.15.28:0/1362039673 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb75c0c5a70 0x7fb75c1eb5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:38.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.129+0000 7fb759e00640 1 -- 172.21.15.28:0/1362039673 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb75c0aad50 con 0x7fb75c0c22e0 2024-02-21T23:11:38.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.129+0000 7fb759400640 1 --2- 172.21.15.28:0/1362039673 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb75c0c5a70 0x7fb75c1eb5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:11:38.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.129+0000 7fb759e00640 1 --2- 172.21.15.28:0/1362039673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb75c0c22e0 0x7fb75c0bfc20 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7fb74c00c460 tx=0x7fb74c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:38.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.130+0000 7fb752000640 1 -- 172.21.15.28:0/1362039673 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb74c011790 con 0x7fb75c0c22e0 2024-02-21T23:11:38.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.130+0000 7fb752000640 1 -- 172.21.15.28:0/1362039673 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb74c017070 con 0x7fb75c0c22e0 2024-02-21T23:11:38.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.130+0000 7fb761000640 1 -- 172.21.15.28:0/1362039673 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb75c0ab030 con 0x7fb75c0c22e0 2024-02-21T23:11:38.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.130+0000 7fb752000640 1 -- 172.21.15.28:0/1362039673 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb74c011bb0 con 0x7fb75c0c22e0 2024-02-21T23:11:38.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.130+0000 7fb761000640 1 -- 172.21.15.28:0/1362039673 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb75c0ab5f0 con 0x7fb75c0c22e0 2024-02-21T23:11:38.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.131+0000 7fb761000640 1 -- 172.21.15.28:0/1362039673 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb75c067f30 con 0x7fb75c0c22e0 2024-02-21T23:11:38.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.132+0000 7fb752000640 1 -- 172.21.15.28:0/1362039673 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb74c01c460 con 0x7fb75c0c22e0 2024-02-21T23:11:38.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.133+0000 7fb752000640 1 --2- 172.21.15.28:0/1362039673 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb73c06b4d0 0x7fb73c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:38.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.133+0000 7fb752000640 1 -- 172.21.15.28:0/1362039673 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 6001+0+0 (secure 0 0 0) 0x7fb74c08a250 con 0x7fb75c0c22e0 2024-02-21T23:11:38.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.134+0000 7fb759400640 1 --2- 172.21.15.28:0/1362039673 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb73c06b4d0 0x7fb73c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:38.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.135+0000 7fb759400640 1 --2- 172.21.15.28:0/1362039673 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb73c06b4d0 0x7fb73c06d990 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fb748036000 tx=0x7fb748008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:38.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.138+0000 7fb752000640 1 -- 172.21.15.28:0/1362039673 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb74c058f00 con 0x7fb75c0c22e0 2024-02-21T23:11:38.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.382+0000 7fb761000640 1 -- 172.21.15.28:0/1362039673 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fb75c0c16b0 con 0x7fb75c0c22e0 2024-02-21T23:11:38.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.942+0000 7fb752000640 1 -- 172.21.15.28:0/1362039673 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 29 and data pool 30 v197) v1 ==== 174+0+0 (secure 0 0 0) 0x7fb74c058800 con 0x7fb75c0c22e0 2024-02-21T23:11:38.933 INFO:teuthology.orchestra.run.smithi028.stderr:new fs with metadata pool 29 and data pool 30 2024-02-21T23:11:38.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.951+0000 7fb761000640 1 -- 172.21.15.28:0/1362039673 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb73c06b4d0 msgr2=0x7fb73c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:38.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.951+0000 7fb761000640 1 --2- 172.21.15.28:0/1362039673 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb73c06b4d0 0x7fb73c06d990 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fb748036000 tx=0x7fb748008030 comp rx=0 tx=0).stop 2024-02-21T23:11:38.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.951+0000 7fb761000640 1 -- 172.21.15.28:0/1362039673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb75c0c22e0 msgr2=0x7fb75c0bfc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:38.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.951+0000 7fb761000640 1 --2- 172.21.15.28:0/1362039673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb75c0c22e0 0x7fb75c0bfc20 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7fb74c00c460 tx=0x7fb74c00c920 comp rx=0 tx=0).stop 2024-02-21T23:11:38.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.951+0000 7fb761000640 1 -- 172.21.15.28:0/1362039673 shutdown_connections 2024-02-21T23:11:38.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.951+0000 7fb761000640 1 --2- 172.21.15.28:0/1362039673 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb73c06b4d0 0x7fb73c06d990 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:38.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.951+0000 7fb761000640 1 --2- 172.21.15.28:0/1362039673 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb75c1561a0 0x7fb75c1ebb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:38.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.951+0000 7fb761000640 1 --2- 172.21.15.28:0/1362039673 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb75c0c5a70 0x7fb75c1eb5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:38.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.951+0000 7fb761000640 1 --2- 172.21.15.28:0/1362039673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb75c0c22e0 0x7fb75c0bfc20 secure :-1 s=CLOSED pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7fb74c00c460 tx=0x7fb74c00c920 comp rx=0 tx=0).stop 2024-02-21T23:11:38.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.951+0000 7fb761000640 1 -- 172.21.15.28:0/1362039673 >> 172.21.15.28:0/1362039673 conn(0x7fb75c0a85a0 msgr2=0x7fb75c0c3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:38.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.954+0000 7fb761000640 1 -- 172.21.15.28:0/1362039673 shutdown_connections 2024-02-21T23:11:38.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:38.955+0000 7fb761000640 1 -- 172.21.15.28:0/1362039673 wait complete. 2024-02-21T23:11:38.972 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:11:39.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.169+0000 7f2210c00640 1 Processor -- start 2024-02-21T23:11:39.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.169+0000 7f2210c00640 1 -- start start 2024-02-21T23:11:39.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.171+0000 7f2210c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f220c153950 0x7f220c0c15d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:39.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.171+0000 7f2210c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f220c15b6d0 0x7f220c15dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:39.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.171+0000 7f2210c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f220c152e70 con 0x7f220c0c11f0 2024-02-21T23:11:39.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.171+0000 7f2210c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f220c0a5ec0 con 0x7f220c153950 2024-02-21T23:11:39.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.171+0000 7f2210c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f220c153150 con 0x7f220c15b6d0 2024-02-21T23:11:39.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.171+0000 7f2209400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f220c153950 0x7f220c0c15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:39.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.171+0000 7f2209400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f220c153950 0x7f220c0c15d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45780/0 (socket says 172.21.15.28:45780) 2024-02-21T23:11:39.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.171+0000 7f2209400640 1 -- 172.21.15.28:0/1203151912 learned_addr learned my addr 172.21.15.28:0/1203151912 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:39.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.172+0000 7f220a800640 1 --2- 172.21.15.28:0/1203151912 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f220c15b6d0 0x7f220c15dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:39.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.172+0000 7f2209400640 1 -- 172.21.15.28:0/1203151912 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f220c15b6d0 msgr2=0x7f220c15dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:39.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.172+0000 7f2209400640 1 --2- 172.21.15.28:0/1203151912 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f220c15b6d0 0x7f220c15dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:39.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.172+0000 7f2209400640 1 -- 172.21.15.28:0/1203151912 >> v1:172.21.15.28:6789/0 conn(0x7f220c0c11f0 legacy=0x7f220c06dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:39.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.172+0000 7f2209400640 1 -- 172.21.15.28:0/1203151912 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f220c153430 con 0x7f220c153950 2024-02-21T23:11:39.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.173+0000 7f2209400640 1 --2- 172.21.15.28:0/1203151912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f220c153950 0x7f220c0c15d0 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7f21f4009870 tx=0x7f21f402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac994e44576029f server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:39.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.173+0000 7f2208a00640 1 -- 172.21.15.28:0/1203151912 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21f4035020 con 0x7f220c153950 2024-02-21T23:11:39.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.173+0000 7f2208a00640 1 -- 172.21.15.28:0/1203151912 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21f4031640 con 0x7f220c153950 2024-02-21T23:11:39.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.173+0000 7f2208a00640 1 -- 172.21.15.28:0/1203151912 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21f402d040 con 0x7f220c153950 2024-02-21T23:11:39.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.174+0000 7f2210c00640 1 -- 172.21.15.28:0/1203151912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f220c153950 msgr2=0x7f220c0c15d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:39.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.174+0000 7f2210c00640 1 --2- 172.21.15.28:0/1203151912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f220c153950 0x7f220c0c15d0 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7f21f4009870 tx=0x7f21f402b8f0 comp rx=0 tx=0).stop 2024-02-21T23:11:39.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.174+0000 7f2210c00640 1 -- 172.21.15.28:0/1203151912 shutdown_connections 2024-02-21T23:11:39.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.174+0000 7f2210c00640 1 --2- 172.21.15.28:0/1203151912 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f220c15b6d0 0x7f220c15dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:39.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.174+0000 7f2210c00640 1 --2- 172.21.15.28:0/1203151912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f220c153950 0x7f220c0c15d0 unknown :-1 s=CLOSED pgs=2061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:39.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.174+0000 7f2210c00640 1 -- 172.21.15.28:0/1203151912 >> 172.21.15.28:0/1203151912 conn(0x7f220c0c67f0 msgr2=0x7f220c0c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:39.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.174+0000 7f2210c00640 1 -- 172.21.15.28:0/1203151912 shutdown_connections 2024-02-21T23:11:39.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.174+0000 7f2210c00640 1 -- 172.21.15.28:0/1203151912 wait complete. 2024-02-21T23:11:39.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.176+0000 7f2210c00640 1 Processor -- start 2024-02-21T23:11:39.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.176+0000 7f2210c00640 1 -- start start 2024-02-21T23:11:39.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.177+0000 7f2210c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f220c0c11f0 0x7f220c146a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:39.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.177+0000 7f2210c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f220c153950 0x7f220c14eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:39.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.177+0000 7f2210c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f220c15b6d0 0x7f220c150c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:39.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.177+0000 7f2210c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f220c0a6110 con 0x7f220c153950 2024-02-21T23:11:39.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.177+0000 7f2210c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f220c0c4c00 con 0x7f220c0c11f0 2024-02-21T23:11:39.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.177+0000 7f2210c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f220c0c0d90 con 0x7f220c15b6d0 2024-02-21T23:11:39.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.177+0000 7f2209400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f220c153950 0x7f220c14eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:39.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.177+0000 7f2209400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f220c153950 0x7f220c14eda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45790/0 (socket says 172.21.15.28:45790) 2024-02-21T23:11:39.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.177+0000 7f2209400640 1 -- 172.21.15.28:0/596731709 learned_addr learned my addr 172.21.15.28:0/596731709 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:39.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.178+0000 7f2209e00640 1 --2- 172.21.15.28:0/596731709 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f220c0c11f0 0x7f220c146a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:39.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.178+0000 7f2209400640 1 -- 172.21.15.28:0/596731709 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f220c15b6d0 msgr2=0x7f220c150c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:39.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.178+0000 7f2209400640 1 --2- 172.21.15.28:0/596731709 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f220c15b6d0 0x7f220c150c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:39.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.178+0000 7f2209400640 1 -- 172.21.15.28:0/596731709 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f220c0c11f0 msgr2=0x7f220c146a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:39.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.178+0000 7f2209400640 1 --2- 172.21.15.28:0/596731709 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f220c0c11f0 0x7f220c146a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:39.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.178+0000 7f2209400640 1 -- 172.21.15.28:0/596731709 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f220c1eafe0 con 0x7f220c153950 2024-02-21T23:11:39.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.178+0000 7f2209e00640 1 --2- 172.21.15.28:0/596731709 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f220c0c11f0 0x7f220c146a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:11:39.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.178+0000 7f2209400640 1 --2- 172.21.15.28:0/596731709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f220c153950 0x7f220c14eda0 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7f21f4000c00 tx=0x7f21f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:39.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.178+0000 7f2202000640 1 -- 172.21.15.28:0/596731709 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21f4040070 con 0x7f220c153950 2024-02-21T23:11:39.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.179+0000 7f2202000640 1 -- 172.21.15.28:0/596731709 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21f40399f0 con 0x7f220c153950 2024-02-21T23:11:39.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.179+0000 7f2202000640 1 -- 172.21.15.28:0/596731709 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21f4035040 con 0x7f220c153950 2024-02-21T23:11:39.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.179+0000 7f2210c00640 1 -- 172.21.15.28:0/596731709 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f220c1eb2c0 con 0x7f220c153950 2024-02-21T23:11:39.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.179+0000 7f2210c00640 1 -- 172.21.15.28:0/596731709 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f220c1eb880 con 0x7f220c153950 2024-02-21T23:11:39.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.180+0000 7f2210c00640 1 -- 172.21.15.28:0/596731709 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f220c067f30 con 0x7f220c153950 2024-02-21T23:11:39.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.181+0000 7f2202000640 1 -- 172.21.15.28:0/596731709 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f21f404b060 con 0x7f220c153950 2024-02-21T23:11:39.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.182+0000 7f2202000640 1 --2- 172.21.15.28:0/596731709 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f21e806b500 0x7f21e806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:39.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.182+0000 7f2202000640 1 -- 172.21.15.28:0/596731709 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f21f40abcd0 con 0x7f220c153950 2024-02-21T23:11:39.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.183+0000 7f2209e00640 1 --2- 172.21.15.28:0/596731709 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f21e806b500 0x7f21e806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:39.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.184+0000 7f2209e00640 1 --2- 172.21.15.28:0/596731709 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f21e806b500 0x7f21e806d9c0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f21fc001680 tx=0x7f21fc009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:39.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.187+0000 7f2202000640 1 -- 172.21.15.28:0/596731709 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f21f407a9b0 con 0x7f220c153950 2024-02-21T23:11:39.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.392+0000 7f2210c00640 1 -- 172.21.15.28:0/596731709 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f220c06f890 con 0x7f220c153950 2024-02-21T23:11:39.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.394+0000 7f2202000640 1 -- 172.21.15.28:0/596731709 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v185) v1 ==== 74+0+16837 (secure 0 0 0) 0x7f21f407a2b0 con 0x7f220c153950 2024-02-21T23:11:39.383 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:11:39.383 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":185,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:11:38.383553+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:11:36.316421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"185","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_data","create_time":"2024-02-21T23:11:37.418709+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"185","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:11:39.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.398+0000 7f2210c00640 1 -- 172.21.15.28:0/596731709 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f21e806b500 msgr2=0x7f21e806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:39.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.398+0000 7f2210c00640 1 --2- 172.21.15.28:0/596731709 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f21e806b500 0x7f21e806d9c0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f21fc001680 tx=0x7f21fc009040 comp rx=0 tx=0).stop 2024-02-21T23:11:39.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.398+0000 7f2210c00640 1 -- 172.21.15.28:0/596731709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f220c153950 msgr2=0x7f220c14eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:39.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.398+0000 7f2210c00640 1 --2- 172.21.15.28:0/596731709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f220c153950 0x7f220c14eda0 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7f21f4000c00 tx=0x7f21f40046c0 comp rx=0 tx=0).stop 2024-02-21T23:11:39.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.398+0000 7f2210c00640 1 -- 172.21.15.28:0/596731709 shutdown_connections 2024-02-21T23:11:39.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.398+0000 7f2210c00640 1 --2- 172.21.15.28:0/596731709 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f21e806b500 0x7f21e806d9c0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:39.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.398+0000 7f2210c00640 1 --2- 172.21.15.28:0/596731709 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f220c15b6d0 0x7f220c150c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:39.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.398+0000 7f2210c00640 1 --2- 172.21.15.28:0/596731709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f220c153950 0x7f220c14eda0 unknown :-1 s=CLOSED pgs=2062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:39.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.398+0000 7f2210c00640 1 --2- 172.21.15.28:0/596731709 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f220c0c11f0 0x7f220c146a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:39.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.398+0000 7f2210c00640 1 -- 172.21.15.28:0/596731709 >> 172.21.15.28:0/596731709 conn(0x7f220c0c67f0 msgr2=0x7f220c0c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:39.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.398+0000 7f2210c00640 1 -- 172.21.15.28:0/596731709 shutdown_connections 2024-02-21T23:11:39.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.398+0000 7f2210c00640 1 -- 172.21.15.28:0/596731709 wait complete. 2024-02-21T23:11:39.405 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:11:39.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.642+0000 7f3d96200640 1 Processor -- start 2024-02-21T23:11:39.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.643+0000 7f3d96200640 1 -- start start 2024-02-21T23:11:39.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.644+0000 7f3d96200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f3d900c4fe0 0x7f3d9006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:39.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.644+0000 7f3d96200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3d90159400 0x7f3d9006e600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:39.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.644+0000 7f3d96200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d90159900 con 0x7f3d901569d0 2024-02-21T23:11:39.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.644+0000 7f3d96200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d90159be0 con 0x7f3d900c4fe0 2024-02-21T23:11:39.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.644+0000 7f3d96200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d90159ec0 con 0x7f3d90159400 2024-02-21T23:11:39.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.644+0000 7f3d8f400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f3d901569d0 0x7f3d900c3280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:40988/0 (socket says 172.21.15.28:40988) 2024-02-21T23:11:39.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.644+0000 7f3d8f400640 1 -- 172.21.15.28:0/2630084128 learned_addr learned my addr 172.21.15.28:0/2630084128 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:11:39.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.645+0000 7f3d8fe00640 1 --2- 172.21.15.28:0/2630084128 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3d90159400 0x7f3d9006e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:39.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.645+0000 7f3d8ea00640 1 --2- 172.21.15.28:0/2630084128 >> v2:172.21.15.81:3300/0 conn(0x7f3d900c4fe0 0x7f3d9006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:39.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.645+0000 7f3d8e000640 1 -- 172.21.15.28:0/2630084128 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1057613819 0 0) 0x7f3d90159900 con 0x7f3d901569d0 2024-02-21T23:11:39.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.645+0000 7f3d8e000640 1 -- 172.21.15.28:0/2630084128 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d74003520 con 0x7f3d901569d0 2024-02-21T23:11:39.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.645+0000 7f3d8e000640 1 -- 172.21.15.28:0/2630084128 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4154477241 0 0) 0x7f3d74003520 con 0x7f3d901569d0 2024-02-21T23:11:39.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.645+0000 7f3d8e000640 1 -- 172.21.15.28:0/2630084128 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3d90159400 msgr2=0x7f3d9006e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:39.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.645+0000 7f3d8e000640 1 --2- 172.21.15.28:0/2630084128 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3d90159400 0x7f3d9006e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:39.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.645+0000 7f3d8e000640 1 -- 172.21.15.28:0/2630084128 >> v2:172.21.15.81:3300/0 conn(0x7f3d900c4fe0 msgr2=0x7f3d9006e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:39.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.645+0000 7f3d8e000640 1 --2- 172.21.15.28:0/2630084128 >> v2:172.21.15.81:3300/0 conn(0x7f3d900c4fe0 0x7f3d9006e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:39.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.645+0000 7f3d8e000640 1 -- 172.21.15.28:0/2630084128 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d9015a1a0 con 0x7f3d901569d0 2024-02-21T23:11:39.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.645+0000 7f3d8e000640 1 -- 172.21.15.28:0/2630084128 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f3d80001690 con 0x7f3d901569d0 2024-02-21T23:11:39.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.645+0000 7f3d8e000640 1 -- 172.21.15.28:0/2630084128 >> v1:172.21.15.28:6789/0 conn(0x7f3d901569d0 legacy=0x7f3d900c3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:39.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.645+0000 7f3d8e000640 1 --2- 172.21.15.28:0/2630084128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d740039b0 0x7f3d74003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:39.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.646+0000 7f3d8e000640 1 --2- 172.21.15.28:0/2630084128 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d74004800 0x7f3d74004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:39.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.646+0000 7f3d8e000640 1 --2- 172.21.15.28:0/2630084128 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3d90159400 0x7f3d740051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:39.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.646+0000 7f3d8e000640 1 -- 172.21.15.28:0/2630084128 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d90159be0 con 0x7f3d740039b0 2024-02-21T23:11:39.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.646+0000 7f3d8e000640 1 -- 172.21.15.28:0/2630084128 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d90159ec0 con 0x7f3d74004800 2024-02-21T23:11:39.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.646+0000 7f3d8e000640 1 -- 172.21.15.28:0/2630084128 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d7400ba40 con 0x7f3d90159400 2024-02-21T23:11:39.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.646+0000 7f3d8f400640 1 --2- 172.21.15.28:0/2630084128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d740039b0 0x7f3d74003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:39.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.646+0000 7f3d8ea00640 1 --2- 172.21.15.28:0/2630084128 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d74004800 0x7f3d74004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:39.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.646+0000 7f3d8fe00640 1 --2- 172.21.15.28:0/2630084128 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3d90159400 0x7f3d740051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:39.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.646+0000 7f3d8f400640 1 -- 172.21.15.28:0/2630084128 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3d90159400 msgr2=0x7f3d740051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:39.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.646+0000 7f3d8f400640 1 --2- 172.21.15.28:0/2630084128 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3d90159400 0x7f3d740051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:39.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.646+0000 7f3d8f400640 1 -- 172.21.15.28:0/2630084128 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d74004800 msgr2=0x7f3d74004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:39.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.646+0000 7f3d8f400640 1 --2- 172.21.15.28:0/2630084128 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d74004800 0x7f3d74004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:39.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.646+0000 7f3d8f400640 1 -- 172.21.15.28:0/2630084128 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d74003520 con 0x7f3d740039b0 2024-02-21T23:11:39.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.646+0000 7f3d8ea00640 1 --2- 172.21.15.28:0/2630084128 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d74004800 0x7f3d74004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:11:39.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.648+0000 7f3d8f400640 1 --2- 172.21.15.28:0/2630084128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d740039b0 0x7f3d74003dd0 secure :-1 s=READY pgs=2064 cs=0 l=1 rev1=1 crypto rx=0x7f3d8000c160 tx=0x7f3d74005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67c6252be44df569 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:39.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.648+0000 7f3d8e000640 1 -- 172.21.15.28:0/2630084128 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d80007c80 con 0x7f3d740039b0 2024-02-21T23:11:39.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.648+0000 7f3d8e000640 1 -- 172.21.15.28:0/2630084128 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d80030070 con 0x7f3d740039b0 2024-02-21T23:11:39.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.648+0000 7f3d8e000640 1 -- 172.21.15.28:0/2630084128 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d80005cd0 con 0x7f3d740039b0 2024-02-21T23:11:39.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.648+0000 7f3d96200640 1 -- 172.21.15.28:0/2630084128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d740039b0 msgr2=0x7f3d74003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:39.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.648+0000 7f3d96200640 1 --2- 172.21.15.28:0/2630084128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d740039b0 0x7f3d74003dd0 secure :-1 s=READY pgs=2064 cs=0 l=1 rev1=1 crypto rx=0x7f3d8000c160 tx=0x7f3d74005830 comp rx=0 tx=0).stop 2024-02-21T23:11:39.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.648+0000 7f3d8fe00640 1 -- 172.21.15.28:0/2630084128 reap_dead start 2024-02-21T23:11:39.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.649+0000 7f3d96200640 1 -- 172.21.15.28:0/2630084128 shutdown_connections 2024-02-21T23:11:39.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.649+0000 7f3d96200640 1 -- 172.21.15.28:0/2630084128 >> 172.21.15.28:0/2630084128 conn(0x7f3d900c67f0 msgr2=0x7f3d900c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:39.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.649+0000 7f3d96200640 1 -- 172.21.15.28:0/2630084128 shutdown_connections 2024-02-21T23:11:39.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.649+0000 7f3d96200640 1 -- 172.21.15.28:0/2630084128 wait complete. 2024-02-21T23:11:39.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.650+0000 7f3d96200640 1 Processor -- start 2024-02-21T23:11:39.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.650+0000 7f3d96200640 1 -- start start 2024-02-21T23:11:39.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.651+0000 7f3d96200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d740039b0 0x7f3d9015d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:39.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.651+0000 7f3d96200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d9015a7f0 0x7f3d901ef620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:39.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.651+0000 7f3d96200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3d901efb60 0x7f3d901f2690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:39.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.651+0000 7f3d96200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d900c3b10 con 0x7f3d740039b0 2024-02-21T23:11:39.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.651+0000 7f3d96200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d900c3990 con 0x7f3d9015a7f0 2024-02-21T23:11:39.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.651+0000 7f3d96200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d900c46f0 con 0x7f3d901efb60 2024-02-21T23:11:39.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.652+0000 7f3d8f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d740039b0 0x7f3d9015d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:39.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.652+0000 7f3d8f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d740039b0 0x7f3d9015d390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45812/0 (socket says 172.21.15.28:45812) 2024-02-21T23:11:39.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.652+0000 7f3d8f400640 1 -- 172.21.15.28:0/1343947814 learned_addr learned my addr 172.21.15.28:0/1343947814 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:39.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.652+0000 7f3d8ea00640 1 --2- 172.21.15.28:0/1343947814 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d9015a7f0 0x7f3d901ef620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:39.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.652+0000 7f3d8fe00640 1 --2- 172.21.15.28:0/1343947814 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3d901efb60 0x7f3d901f2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:39.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.652+0000 7f3d8f400640 1 -- 172.21.15.28:0/1343947814 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3d901efb60 msgr2=0x7f3d901f2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:39.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.652+0000 7f3d8f400640 1 --2- 172.21.15.28:0/1343947814 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3d901efb60 0x7f3d901f2690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:39.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.652+0000 7f3d8f400640 1 -- 172.21.15.28:0/1343947814 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d9015a7f0 msgr2=0x7f3d901ef620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:39.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.652+0000 7f3d8f400640 1 --2- 172.21.15.28:0/1343947814 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d9015a7f0 0x7f3d901ef620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:39.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.652+0000 7f3d8f400640 1 -- 172.21.15.28:0/1343947814 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d90150d50 con 0x7f3d740039b0 2024-02-21T23:11:39.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.652+0000 7f3d8ea00640 1 --2- 172.21.15.28:0/1343947814 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d9015a7f0 0x7f3d901ef620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:11:39.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.652+0000 7f3d8f400640 1 --2- 172.21.15.28:0/1343947814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d740039b0 0x7f3d9015d390 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 crypto rx=0x7f3d8000c280 tx=0x7f3d9015a480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:39.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.653+0000 7f3d87400640 1 -- 172.21.15.28:0/1343947814 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d80046070 con 0x7f3d740039b0 2024-02-21T23:11:39.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.653+0000 7f3d87400640 1 -- 172.21.15.28:0/1343947814 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d80004bf0 con 0x7f3d740039b0 2024-02-21T23:11:39.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.653+0000 7f3d87400640 1 -- 172.21.15.28:0/1343947814 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d80001bc0 con 0x7f3d740039b0 2024-02-21T23:11:39.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.653+0000 7f3d96200640 1 -- 172.21.15.28:0/1343947814 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d901ef120 con 0x7f3d740039b0 2024-02-21T23:11:39.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.653+0000 7f3d96200640 1 -- 172.21.15.28:0/1343947814 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d9015c2b0 con 0x7f3d740039b0 2024-02-21T23:11:39.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.654+0000 7f3d96200640 1 -- 172.21.15.28:0/1343947814 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d90067f30 con 0x7f3d740039b0 2024-02-21T23:11:39.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.655+0000 7f3d87400640 1 -- 172.21.15.28:0/1343947814 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3d80007070 con 0x7f3d740039b0 2024-02-21T23:11:39.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.656+0000 7f3d87400640 1 --2- 172.21.15.28:0/1343947814 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3d7006b500 0x7f3d7006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:39.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.656+0000 7f3d87400640 1 -- 172.21.15.28:0/1343947814 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(185..185 src has 1..185) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f3d800ab510 con 0x7f3d740039b0 2024-02-21T23:11:39.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.657+0000 7f3d8ea00640 1 --2- 172.21.15.28:0/1343947814 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3d7006b500 0x7f3d7006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:39.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.658+0000 7f3d8ea00640 1 --2- 172.21.15.28:0/1343947814 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3d7006b500 0x7f3d7006d9c0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f3d9014dd80 tx=0x7f3d7c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:39.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.660+0000 7f3d87400640 1 -- 172.21.15.28:0/1343947814 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3d8007a270 con 0x7f3d740039b0 2024-02-21T23:11:39.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.855+0000 7f3d96200640 1 -- 172.21.15.28:0/1343947814 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3d900c64b0 con 0x7f3d740039b0 2024-02-21T23:11:39.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.857+0000 7f3d87400640 1 -- 172.21.15.28:0/1343947814 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v185) v1 ==== 74+0+16837 (secure 0 0 0) 0x7f3d80079b70 con 0x7f3d740039b0 2024-02-21T23:11:39.845 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:11:39.845 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":185,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:11:38.383553+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:11:36.316421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"185","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_data","create_time":"2024-02-21T23:11:37.418709+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"185","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:11:39.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.859+0000 7f3d96200640 1 -- 172.21.15.28:0/1343947814 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3d7006b500 msgr2=0x7f3d7006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:39.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.859+0000 7f3d96200640 1 --2- 172.21.15.28:0/1343947814 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3d7006b500 0x7f3d7006d9c0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f3d9014dd80 tx=0x7f3d7c008040 comp rx=0 tx=0).stop 2024-02-21T23:11:39.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.859+0000 7f3d96200640 1 -- 172.21.15.28:0/1343947814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d740039b0 msgr2=0x7f3d9015d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:39.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.859+0000 7f3d96200640 1 --2- 172.21.15.28:0/1343947814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d740039b0 0x7f3d9015d390 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 crypto rx=0x7f3d8000c280 tx=0x7f3d9015a480 comp rx=0 tx=0).stop 2024-02-21T23:11:39.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.859+0000 7f3d96200640 1 -- 172.21.15.28:0/1343947814 shutdown_connections 2024-02-21T23:11:39.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.859+0000 7f3d96200640 1 --2- 172.21.15.28:0/1343947814 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3d7006b500 0x7f3d7006d9c0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:39.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.859+0000 7f3d96200640 1 --2- 172.21.15.28:0/1343947814 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3d901efb60 0x7f3d901f2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:39.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.859+0000 7f3d96200640 1 --2- 172.21.15.28:0/1343947814 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d9015a7f0 0x7f3d901ef620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:39.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.860+0000 7f3d96200640 1 --2- 172.21.15.28:0/1343947814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d740039b0 0x7f3d9015d390 unknown :-1 s=CLOSED pgs=2065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:39.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.860+0000 7f3d96200640 1 -- 172.21.15.28:0/1343947814 >> 172.21.15.28:0/1343947814 conn(0x7f3d900c67f0 msgr2=0x7f3d90158ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:39.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.860+0000 7f3d96200640 1 -- 172.21.15.28:0/1343947814 shutdown_connections 2024-02-21T23:11:39.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:39.860+0000 7f3d96200640 1 -- 172.21.15.28:0/1343947814 wait complete. 2024-02-21T23:11:39.864 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-02-21T23:11:40.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.082+0000 7f5763e00640 1 Processor -- start 2024-02-21T23:11:40.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.082+0000 7f5763e00640 1 -- start start 2024-02-21T23:11:40.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.084+0000 7f5763e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f576413e0c0 0x7f57640c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:40.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.084+0000 7f5763e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f576406dbc0 0x7f576406c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:40.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.084+0000 7f5763e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f57640c76f0 0x7f576406c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:40.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.084+0000 7f5763e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57640c89a0 con 0x7f57640c76f0 2024-02-21T23:11:40.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.084+0000 7f5763e00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5764158550 con 0x7f576406dbc0 2024-02-21T23:11:40.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.084+0000 7f5763e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57641587d0 con 0x7f576413e0c0 2024-02-21T23:11:40.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.084+0000 7f5763400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f57640c76f0 0x7f576406c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:40.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.084+0000 7f5763400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f57640c76f0 0x7f576406c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45826/0 (socket says 172.21.15.28:45826) 2024-02-21T23:11:40.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.084+0000 7f5763400640 1 -- 172.21.15.28:0/1809190975 learned_addr learned my addr 172.21.15.28:0/1809190975 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:40.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.085+0000 7f5762000640 1 --2- 172.21.15.28:0/1809190975 >> v2:172.21.15.81:3300/0 conn(0x7f576406dbc0 0x7f576406c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:40.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.085+0000 7f5762a00640 1 --2- 172.21.15.28:0/1809190975 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f576413e0c0 0x7f57640c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:40.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.085+0000 7f5763400640 1 -- 172.21.15.28:0/1809190975 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f576413e0c0 msgr2=0x7f57640c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:40.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.085+0000 7f5763400640 1 --2- 172.21.15.28:0/1809190975 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f576413e0c0 0x7f57640c71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:40.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.085+0000 7f5763400640 1 -- 172.21.15.28:0/1809190975 >> v2:172.21.15.81:3300/0 conn(0x7f576406dbc0 msgr2=0x7f576406c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:40.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.085+0000 7f5763400640 1 --2- 172.21.15.28:0/1809190975 >> v2:172.21.15.81:3300/0 conn(0x7f576406dbc0 0x7f576406c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:40.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.085+0000 7f5763400640 1 -- 172.21.15.28:0/1809190975 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5764158ab0 con 0x7f57640c76f0 2024-02-21T23:11:40.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.085+0000 7f5762000640 1 --2- 172.21.15.28:0/1809190975 >> v2:172.21.15.81:3300/0 conn(0x7f576406dbc0 0x7f576406c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:11:40.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.086+0000 7f5763400640 1 --2- 172.21.15.28:0/1809190975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57640c76f0 0x7f576406c7c0 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7f57640c8320 tx=0x7f575c02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55ee2e89b8ab70a2 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:40.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.086+0000 7f5761600640 1 -- 172.21.15.28:0/1809190975 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f575c02c9f0 con 0x7f57640c76f0 2024-02-21T23:11:40.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.086+0000 7f5761600640 1 -- 172.21.15.28:0/1809190975 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f575c02cb90 con 0x7f57640c76f0 2024-02-21T23:11:40.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.086+0000 7f5761600640 1 -- 172.21.15.28:0/1809190975 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f575c0343f0 con 0x7f57640c76f0 2024-02-21T23:11:40.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.087+0000 7f5763e00640 1 -- 172.21.15.28:0/1809190975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57640c76f0 msgr2=0x7f576406c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:40.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.087+0000 7f5763e00640 1 --2- 172.21.15.28:0/1809190975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57640c76f0 0x7f576406c7c0 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7f57640c8320 tx=0x7f575c02be50 comp rx=0 tx=0).stop 2024-02-21T23:11:40.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.087+0000 7f5763e00640 1 -- 172.21.15.28:0/1809190975 shutdown_connections 2024-02-21T23:11:40.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.087+0000 7f5763e00640 1 --2- 172.21.15.28:0/1809190975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57640c76f0 0x7f576406c7c0 unknown :-1 s=CLOSED pgs=2066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:40.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.087+0000 7f5763e00640 1 --2- 172.21.15.28:0/1809190975 >> v2:172.21.15.81:3300/0 conn(0x7f576406dbc0 0x7f576406c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:40.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.087+0000 7f5763e00640 1 --2- 172.21.15.28:0/1809190975 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f576413e0c0 0x7f57640c71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:40.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.087+0000 7f5763e00640 1 -- 172.21.15.28:0/1809190975 >> 172.21.15.28:0/1809190975 conn(0x7f5764073110 msgr2=0x7f57640ac490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:40.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.087+0000 7f5763e00640 1 -- 172.21.15.28:0/1809190975 shutdown_connections 2024-02-21T23:11:40.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.087+0000 7f5763e00640 1 -- 172.21.15.28:0/1809190975 wait complete. 2024-02-21T23:11:40.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.089+0000 7f5763e00640 1 Processor -- start 2024-02-21T23:11:40.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.090+0000 7f5763e00640 1 -- start start 2024-02-21T23:11:40.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.091+0000 7f5763e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f576406dbc0 0x7f5764157640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:40.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.091+0000 7f5763e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f57640c76f0 0x7f57640ad940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:40.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.091+0000 7f5763e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f576413e0c0 0x7f57640ade80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:40.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.091+0000 7f5763e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f576406cd00 con 0x7f576413e0c0 2024-02-21T23:11:40.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.091+0000 7f5763e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5764159070 con 0x7f57640c76f0 2024-02-21T23:11:40.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.091+0000 7f5763e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f576406ce80 con 0x7f576406dbc0 2024-02-21T23:11:40.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.091+0000 7f5763400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f576413e0c0 0x7f57640ade80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:40.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.091+0000 7f5763400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f576413e0c0 0x7f57640ade80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45830/0 (socket says 172.21.15.28:45830) 2024-02-21T23:11:40.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.091+0000 7f5763400640 1 -- 172.21.15.28:0/2192941769 learned_addr learned my addr 172.21.15.28:0/2192941769 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:40.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.091+0000 7f5762a00640 1 --2- 172.21.15.28:0/2192941769 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f576406dbc0 0x7f5764157640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:40.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.091+0000 7f5762000640 1 --2- 172.21.15.28:0/2192941769 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f57640c76f0 0x7f57640ad940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:40.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.091+0000 7f5763400640 1 -- 172.21.15.28:0/2192941769 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f576406dbc0 msgr2=0x7f5764157640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:40.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.091+0000 7f5763400640 1 --2- 172.21.15.28:0/2192941769 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f576406dbc0 0x7f5764157640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:40.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.091+0000 7f5763400640 1 -- 172.21.15.28:0/2192941769 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f57640c76f0 msgr2=0x7f57640ad940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:40.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.091+0000 7f5763400640 1 --2- 172.21.15.28:0/2192941769 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f57640c76f0 0x7f57640ad940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:40.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.091+0000 7f5763400640 1 -- 172.21.15.28:0/2192941769 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57640aa300 con 0x7f576413e0c0 2024-02-21T23:11:40.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.092+0000 7f5763400640 1 --2- 172.21.15.28:0/2192941769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f576413e0c0 0x7f57640ade80 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 crypto rx=0x7f575c036040 tx=0x7f575c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:40.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.092+0000 7f575b400640 1 -- 172.21.15.28:0/2192941769 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f575c043020 con 0x7f576413e0c0 2024-02-21T23:11:40.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.092+0000 7f575b400640 1 -- 172.21.15.28:0/2192941769 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f575c03e450 con 0x7f576413e0c0 2024-02-21T23:11:40.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.092+0000 7f5763e00640 1 -- 172.21.15.28:0/2192941769 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57640aa5e0 con 0x7f576413e0c0 2024-02-21T23:11:40.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.092+0000 7f575b400640 1 -- 172.21.15.28:0/2192941769 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f575c009510 con 0x7f576413e0c0 2024-02-21T23:11:40.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.092+0000 7f5763e00640 1 -- 172.21.15.28:0/2192941769 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57640aaba0 con 0x7f576413e0c0 2024-02-21T23:11:40.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.093+0000 7f5763e00640 1 -- 172.21.15.28:0/2192941769 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f576413c300 con 0x7f576413e0c0 2024-02-21T23:11:40.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.094+0000 7f575b400640 1 -- 172.21.15.28:0/2192941769 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f575c005d10 con 0x7f576413e0c0 2024-02-21T23:11:40.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.095+0000 7f575b400640 1 --2- 172.21.15.28:0/2192941769 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f574006b4d0 0x7f574006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:40.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.095+0000 7f575b400640 1 -- 172.21.15.28:0/2192941769 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f575c0acf00 con 0x7f576413e0c0 2024-02-21T23:11:40.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.096+0000 7f5762a00640 1 --2- 172.21.15.28:0/2192941769 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f574006b4d0 0x7f574006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:40.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.097+0000 7f5762a00640 1 --2- 172.21.15.28:0/2192941769 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f574006b4d0 0x7f574006d990 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f5754006fd0 tx=0x7f5754008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:40.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.099+0000 7f575b400640 1 -- 172.21.15.28:0/2192941769 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f576413c300 con 0x7f576413e0c0 2024-02-21T23:11:40.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.337+0000 7f5763e00640 1 -- 172.21.15.28:0/2192941769 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f576404f290 con 0x7f576413e0c0 2024-02-21T23:11:40.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.955+0000 7f575b400640 1 -- 172.21.15.28:0/2192941769 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v200) v1 ==== 118+0+0 (secure 0 0 0) 0x7f575c07bbe0 con 0x7f576413e0c0 2024-02-21T23:11:40.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.959+0000 7f5763e00640 1 -- 172.21.15.28:0/2192941769 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f574006b4d0 msgr2=0x7f574006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:40.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.959+0000 7f5763e00640 1 --2- 172.21.15.28:0/2192941769 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f574006b4d0 0x7f574006d990 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f5754006fd0 tx=0x7f5754008040 comp rx=0 tx=0).stop 2024-02-21T23:11:40.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.959+0000 7f5763e00640 1 -- 172.21.15.28:0/2192941769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f576413e0c0 msgr2=0x7f57640ade80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:40.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.959+0000 7f5763e00640 1 --2- 172.21.15.28:0/2192941769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f576413e0c0 0x7f57640ade80 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 crypto rx=0x7f575c036040 tx=0x7f575c004a40 comp rx=0 tx=0).stop 2024-02-21T23:11:40.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.959+0000 7f5763e00640 1 -- 172.21.15.28:0/2192941769 shutdown_connections 2024-02-21T23:11:40.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.959+0000 7f5763e00640 1 --2- 172.21.15.28:0/2192941769 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f574006b4d0 0x7f574006d990 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:40.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.959+0000 7f5763e00640 1 --2- 172.21.15.28:0/2192941769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f576413e0c0 0x7f57640ade80 unknown :-1 s=CLOSED pgs=2067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:40.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.959+0000 7f5763e00640 1 --2- 172.21.15.28:0/2192941769 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f57640c76f0 0x7f57640ad940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:40.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.959+0000 7f5763e00640 1 --2- 172.21.15.28:0/2192941769 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f576406dbc0 0x7f5764157640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:40.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.959+0000 7f5763e00640 1 -- 172.21.15.28:0/2192941769 >> 172.21.15.28:0/2192941769 conn(0x7f5764073110 msgr2=0x7f57640c94c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:40.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.959+0000 7f5763e00640 1 -- 172.21.15.28:0/2192941769 shutdown_connections 2024-02-21T23:11:40.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:40.959+0000 7f5763e00640 1 -- 172.21.15.28:0/2192941769 wait complete. 2024-02-21T23:11:40.969 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-02-21T23:11:40.969 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-02-21T23:11:41.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.207+0000 7f1930a00640 1 Processor -- start 2024-02-21T23:11:41.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.207+0000 7f1930a00640 1 -- start start 2024-02-21T23:11:41.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.209+0000 7f1930a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f192c0c4c10 0x7f192c06b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:41.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.209+0000 7f1930a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f192c159c50 0x7f192c163b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:41.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.209+0000 7f1930a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f192c0c0d40 con 0x7f192c06ae70 2024-02-21T23:11:41.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.209+0000 7f1930a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f192c15caf0 con 0x7f192c159c50 2024-02-21T23:11:41.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.209+0000 7f1930a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f192c15cdd0 con 0x7f192c0c4c10 2024-02-21T23:11:41.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.209+0000 7f192be00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f192c159c50 0x7f192c163b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:41.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.209+0000 7f192aa00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f192c0c4c10 0x7f192c06b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:41.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.210+0000 7f192be00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f192c159c50 0x7f192c163b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.28:38106/0 (socket says 172.21.15.28:38106) 2024-02-21T23:11:41.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.210+0000 7f192aa00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f192c0c4c10 0x7f192c06b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3301/0 says I am v2:172.21.15.28:60422/0 (socket says 172.21.15.28:60422) 2024-02-21T23:11:41.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.210+0000 7f192be00640 1 -- 172.21.15.28:0/3828484577 learned_addr learned my addr 172.21.15.28:0/3828484577 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:41.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.210+0000 7f192aa00640 1 -- 172.21.15.28:0/3828484577 >> v2:172.21.15.81:3300/0 conn(0x7f192c159c50 msgr2=0x7f192c163b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:41.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.210+0000 7f192aa00640 1 --2- 172.21.15.28:0/3828484577 >> v2:172.21.15.81:3300/0 conn(0x7f192c159c50 0x7f192c163b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:41.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.210+0000 7f192aa00640 1 -- 172.21.15.28:0/3828484577 >> v1:172.21.15.81:6789/0 conn(0x7f192c06ae70 legacy=0x7f192c159540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:41.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.210+0000 7f192aa00640 1 -- 172.21.15.28:0/3828484577 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f192c15d0b0 con 0x7f192c0c4c10 2024-02-21T23:11:41.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.212+0000 7f192aa00640 1 --2- 172.21.15.28:0/3828484577 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f192c0c4c10 0x7f192c06b250 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f19180097a0 tx=0x7f191802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bbb6aa5dbbfe26e9 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:41.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.212+0000 7f192a000640 1 -- 172.21.15.28:0/3828484577 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19180315a0 con 0x7f192c0c4c10 2024-02-21T23:11:41.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.212+0000 7f192a000640 1 -- 172.21.15.28:0/3828484577 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1918037070 con 0x7f192c0c4c10 2024-02-21T23:11:41.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.212+0000 7f192a000640 1 -- 172.21.15.28:0/3828484577 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1918031960 con 0x7f192c0c4c10 2024-02-21T23:11:41.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.213+0000 7f1930a00640 1 -- 172.21.15.28:0/3828484577 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f192c0c4c10 msgr2=0x7f192c06b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:41.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.213+0000 7f1930a00640 1 --2- 172.21.15.28:0/3828484577 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f192c0c4c10 0x7f192c06b250 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f19180097a0 tx=0x7f191802b840 comp rx=0 tx=0).stop 2024-02-21T23:11:41.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.213+0000 7f1930a00640 1 -- 172.21.15.28:0/3828484577 shutdown_connections 2024-02-21T23:11:41.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.213+0000 7f1930a00640 1 --2- 172.21.15.28:0/3828484577 >> v2:172.21.15.81:3300/0 conn(0x7f192c159c50 0x7f192c163b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:41.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.213+0000 7f1930a00640 1 --2- 172.21.15.28:0/3828484577 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f192c0c4c10 0x7f192c06b250 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:41.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.213+0000 7f1930a00640 1 -- 172.21.15.28:0/3828484577 >> 172.21.15.28:0/3828484577 conn(0x7f192c0a8530 msgr2=0x7f192c0a8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:41.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.213+0000 7f1930a00640 1 -- 172.21.15.28:0/3828484577 shutdown_connections 2024-02-21T23:11:41.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.213+0000 7f1930a00640 1 -- 172.21.15.28:0/3828484577 wait complete. 2024-02-21T23:11:41.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.215+0000 7f1930a00640 1 Processor -- start 2024-02-21T23:11:41.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.215+0000 7f1930a00640 1 -- start start 2024-02-21T23:11:41.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.216+0000 7f1930a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f192c06ae70 0x7f192c151aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:41.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.216+0000 7f1930a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f192c0c4c10 0x7f192c151fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:41.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.216+0000 7f1930a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f192c159c50 0x7f192c1487c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:41.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.216+0000 7f1930a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f192c15c7e0 con 0x7f192c159c50 2024-02-21T23:11:41.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.216+0000 7f1930a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f192c15d670 con 0x7f192c0c4c10 2024-02-21T23:11:41.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.216+0000 7f1930a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f192c06f6b0 con 0x7f192c06ae70 2024-02-21T23:11:41.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.216+0000 7f192be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f192c159c50 0x7f192c1487c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:41.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.217+0000 7f192be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f192c159c50 0x7f192c1487c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45840/0 (socket says 172.21.15.28:45840) 2024-02-21T23:11:41.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.217+0000 7f192be00640 1 -- 172.21.15.28:0/2696244392 learned_addr learned my addr 172.21.15.28:0/2696244392 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:41.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.217+0000 7f192b400640 1 --2- 172.21.15.28:0/2696244392 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f192c06ae70 0x7f192c151aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:41.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.217+0000 7f192aa00640 1 --2- 172.21.15.28:0/2696244392 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f192c0c4c10 0x7f192c151fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:41.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.217+0000 7f192be00640 1 -- 172.21.15.28:0/2696244392 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f192c06ae70 msgr2=0x7f192c151aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:41.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.217+0000 7f192be00640 1 --2- 172.21.15.28:0/2696244392 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f192c06ae70 0x7f192c151aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:41.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.217+0000 7f192be00640 1 -- 172.21.15.28:0/2696244392 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f192c0c4c10 msgr2=0x7f192c151fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:41.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.217+0000 7f192be00640 1 --2- 172.21.15.28:0/2696244392 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f192c0c4c10 0x7f192c151fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:41.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.217+0000 7f192be00640 1 -- 172.21.15.28:0/2696244392 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f192c0731c0 con 0x7f192c159c50 2024-02-21T23:11:41.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.217+0000 7f192b400640 1 --2- 172.21.15.28:0/2696244392 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f192c06ae70 0x7f192c151aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:11:41.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.217+0000 7f192aa00640 1 --2- 172.21.15.28:0/2696244392 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f192c0c4c10 0x7f192c151fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:11:41.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.217+0000 7f192be00640 1 --2- 172.21.15.28:0/2696244392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f192c159c50 0x7f192c1487c0 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 crypto rx=0x7f1924009cf0 tx=0x7f192400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:41.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.218+0000 7f1923400640 1 -- 172.21.15.28:0/2696244392 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f192400cd20 con 0x7f192c159c50 2024-02-21T23:11:41.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.218+0000 7f1923400640 1 -- 172.21.15.28:0/2696244392 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1924003990 con 0x7f192c159c50 2024-02-21T23:11:41.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.218+0000 7f1930a00640 1 -- 172.21.15.28:0/2696244392 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f192c072ee0 con 0x7f192c159c50 2024-02-21T23:11:41.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.218+0000 7f1923400640 1 -- 172.21.15.28:0/2696244392 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1924003d80 con 0x7f192c159c50 2024-02-21T23:11:41.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.218+0000 7f1930a00640 1 -- 172.21.15.28:0/2696244392 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f192c071ed0 con 0x7f192c159c50 2024-02-21T23:11:41.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.219+0000 7f1930a00640 1 -- 172.21.15.28:0/2696244392 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f192c04f290 con 0x7f192c159c50 2024-02-21T23:11:41.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.220+0000 7f1923400640 1 -- 172.21.15.28:0/2696244392 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1924012050 con 0x7f192c159c50 2024-02-21T23:11:41.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.221+0000 7f1923400640 1 --2- 172.21.15.28:0/2696244392 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f190c06b500 0x7f190c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:41.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.221+0000 7f1923400640 1 -- 172.21.15.28:0/2696244392 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f192408a810 con 0x7f192c159c50 2024-02-21T23:11:41.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.221+0000 7f192b400640 1 --2- 172.21.15.28:0/2696244392 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f190c06b500 0x7f190c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:41.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.223+0000 7f192b400640 1 --2- 172.21.15.28:0/2696244392 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f190c06b500 0x7f190c06d9c0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f191c000f60 tx=0x7f191c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:41.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.226+0000 7f1923400640 1 -- 172.21.15.28:0/2696244392 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1924010040 con 0x7f192c159c50 2024-02-21T23:11:41.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.470+0000 7f1930a00640 1 -- 172.21.15.28:0/2696244392 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f192c0c0980 con 0x7f192c159c50 2024-02-21T23:11:41.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.966+0000 7f1923400640 1 -- 172.21.15.28:0/2696244392 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v201) v1 ==== 122+0+0 (secure 0 0 0) 0x7f19240594f0 con 0x7f192c159c50 2024-02-21T23:11:41.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.969+0000 7f1930a00640 1 -- 172.21.15.28:0/2696244392 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f190c06b500 msgr2=0x7f190c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:41.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.969+0000 7f1930a00640 1 --2- 172.21.15.28:0/2696244392 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f190c06b500 0x7f190c06d9c0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f191c000f60 tx=0x7f191c009040 comp rx=0 tx=0).stop 2024-02-21T23:11:41.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.969+0000 7f1930a00640 1 -- 172.21.15.28:0/2696244392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f192c159c50 msgr2=0x7f192c1487c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:41.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.969+0000 7f1930a00640 1 --2- 172.21.15.28:0/2696244392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f192c159c50 0x7f192c1487c0 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 crypto rx=0x7f1924009cf0 tx=0x7f192400c490 comp rx=0 tx=0).stop 2024-02-21T23:11:41.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.969+0000 7f1930a00640 1 -- 172.21.15.28:0/2696244392 shutdown_connections 2024-02-21T23:11:41.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.969+0000 7f1930a00640 1 --2- 172.21.15.28:0/2696244392 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f190c06b500 0x7f190c06d9c0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:41.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.969+0000 7f1930a00640 1 --2- 172.21.15.28:0/2696244392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f192c159c50 0x7f192c1487c0 unknown :-1 s=CLOSED pgs=2068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:41.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.969+0000 7f1930a00640 1 --2- 172.21.15.28:0/2696244392 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f192c0c4c10 0x7f192c151fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:41.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.969+0000 7f1930a00640 1 --2- 172.21.15.28:0/2696244392 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f192c06ae70 0x7f192c151aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:41.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.969+0000 7f1930a00640 1 -- 172.21.15.28:0/2696244392 >> 172.21.15.28:0/2696244392 conn(0x7f192c0a8530 msgr2=0x7f192c1556f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:41.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.970+0000 7f1930a00640 1 -- 172.21.15.28:0/2696244392 shutdown_connections 2024-02-21T23:11:41.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:41.970+0000 7f1930a00640 1 -- 172.21.15.28:0/2696244392 wait complete. 2024-02-21T23:11:41.980 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-02-21T23:11:41.980 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:11:42.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.223+0000 7f6793000640 1 Processor -- start 2024-02-21T23:11:42.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.223+0000 7f6793000640 1 -- start start 2024-02-21T23:11:42.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.224+0000 7f6793000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f678c0a5fd0 0x7f678c06dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:42.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.224+0000 7f6793000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f678c0c2760 con 0x7f678c150450 2024-02-21T23:11:42.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.225+0000 7f6793000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f678c156310 con 0x7f678c06ee40 2024-02-21T23:11:42.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.225+0000 7f6793000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f678c0c22c0 con 0x7f678c0a5fd0 2024-02-21T23:11:42.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.225+0000 7f6792600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f678c0a5fd0 0x7f678c06dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:42.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.225+0000 7f6792600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f678c0a5fd0 0x7f678c06dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45848/0 (socket says 172.21.15.28:45848) 2024-02-21T23:11:42.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.225+0000 7f6792600640 1 -- 172.21.15.28:0/2565602307 learned_addr learned my addr 172.21.15.28:0/2565602307 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:42.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.225+0000 7f678be00640 1 -- 172.21.15.28:0/2565602307 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2376846312 0 0) 0x7f678c0c2760 con 0x7f678c150450 2024-02-21T23:11:42.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.225+0000 7f6792600640 1 -- 172.21.15.28:0/2565602307 >> v1:172.21.15.81:6789/0 conn(0x7f678c06ee40 legacy=0x7f678c156af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:42.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.225+0000 7f6792600640 1 -- 172.21.15.28:0/2565602307 >> v1:172.21.15.28:6789/0 conn(0x7f678c150450 legacy=0x7f678c1536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:42.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.225+0000 7f6792600640 1 -- 172.21.15.28:0/2565602307 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f678c0c29d0 con 0x7f678c0a5fd0 2024-02-21T23:11:42.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.227+0000 7f6792600640 1 --2- 172.21.15.28:0/2565602307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f678c0a5fd0 0x7f678c06dff0 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 crypto rx=0x7f6784009e50 tx=0x7f678402c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=758eb38801203e24 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:42.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.227+0000 7f678be00640 1 -- 172.21.15.28:0/2565602307 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f678402cbb0 con 0x7f678c0a5fd0 2024-02-21T23:11:42.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.227+0000 7f678be00640 1 -- 172.21.15.28:0/2565602307 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f678402cd50 con 0x7f678c0a5fd0 2024-02-21T23:11:42.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.227+0000 7f678be00640 1 -- 172.21.15.28:0/2565602307 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6784034610 con 0x7f678c0a5fd0 2024-02-21T23:11:42.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.228+0000 7f6793000640 1 -- 172.21.15.28:0/2565602307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f678c0a5fd0 msgr2=0x7f678c06dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:42.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.228+0000 7f6793000640 1 --2- 172.21.15.28:0/2565602307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f678c0a5fd0 0x7f678c06dff0 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 crypto rx=0x7f6784009e50 tx=0x7f678402c010 comp rx=0 tx=0).stop 2024-02-21T23:11:42.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.228+0000 7f6793000640 1 -- 172.21.15.28:0/2565602307 shutdown_connections 2024-02-21T23:11:42.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.228+0000 7f6793000640 1 --2- 172.21.15.28:0/2565602307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f678c0a5fd0 0x7f678c06dff0 unknown :-1 s=CLOSED pgs=2070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:42.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.228+0000 7f6793000640 1 -- 172.21.15.28:0/2565602307 >> 172.21.15.28:0/2565602307 conn(0x7f678c06b630 msgr2=0x7f678c0c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:42.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.228+0000 7f6793000640 1 -- 172.21.15.28:0/2565602307 shutdown_connections 2024-02-21T23:11:42.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.228+0000 7f6793000640 1 -- 172.21.15.28:0/2565602307 wait complete. 2024-02-21T23:11:42.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.229+0000 7f6793000640 1 Processor -- start 2024-02-21T23:11:42.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.230+0000 7f6793000640 1 -- start start 2024-02-21T23:11:42.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.230+0000 7f6793000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f678c06ee40 0x7f678c0bfba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:42.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.231+0000 7f6793000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f678c0a5fd0 0x7f678c1436f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:42.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.231+0000 7f6793000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f678c150450 0x7f678c143c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:42.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.231+0000 7f6793000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f678c157200 con 0x7f678c0a5fd0 2024-02-21T23:11:42.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.231+0000 7f6793000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f678c156660 con 0x7f678c06ee40 2024-02-21T23:11:42.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.231+0000 7f6793000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f678c06e530 con 0x7f678c150450 2024-02-21T23:11:42.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.231+0000 7f6791200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f678c0a5fd0 0x7f678c1436f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:42.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.231+0000 7f6791200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f678c0a5fd0 0x7f678c1436f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45854/0 (socket says 172.21.15.28:45854) 2024-02-21T23:11:42.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.231+0000 7f6791200640 1 -- 172.21.15.28:0/3584810347 learned_addr learned my addr 172.21.15.28:0/3584810347 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:42.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.231+0000 7f6792600640 1 --2- 172.21.15.28:0/3584810347 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f678c150450 0x7f678c143c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:42.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.231+0000 7f6791c00640 1 --2- 172.21.15.28:0/3584810347 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f678c06ee40 0x7f678c0bfba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:42.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.231+0000 7f6791200640 1 -- 172.21.15.28:0/3584810347 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f678c150450 msgr2=0x7f678c143c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:42.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.231+0000 7f6791200640 1 --2- 172.21.15.28:0/3584810347 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f678c150450 0x7f678c143c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:42.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.231+0000 7f6791200640 1 -- 172.21.15.28:0/3584810347 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f678c06ee40 msgr2=0x7f678c0bfba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:42.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.231+0000 7f6791200640 1 --2- 172.21.15.28:0/3584810347 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f678c06ee40 0x7f678c0bfba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:42.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.231+0000 7f6791200640 1 -- 172.21.15.28:0/3584810347 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f678c0c07f0 con 0x7f678c0a5fd0 2024-02-21T23:11:42.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.231+0000 7f6791c00640 1 --2- 172.21.15.28:0/3584810347 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f678c06ee40 0x7f678c0bfba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:11:42.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.232+0000 7f6791200640 1 --2- 172.21.15.28:0/3584810347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f678c0a5fd0 0x7f678c1436f0 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 crypto rx=0x7f677800c480 tx=0x7f677800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:42.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.232+0000 7f6789600640 1 -- 172.21.15.28:0/3584810347 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67780052d0 con 0x7f678c0a5fd0 2024-02-21T23:11:42.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.232+0000 7f6789600640 1 -- 172.21.15.28:0/3584810347 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6778005470 con 0x7f678c0a5fd0 2024-02-21T23:11:42.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.232+0000 7f6789600640 1 -- 172.21.15.28:0/3584810347 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6778012870 con 0x7f678c0a5fd0 2024-02-21T23:11:42.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.232+0000 7f6793000640 1 -- 172.21.15.28:0/3584810347 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f678c0ac1d0 con 0x7f678c0a5fd0 2024-02-21T23:11:42.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.232+0000 7f6793000640 1 -- 172.21.15.28:0/3584810347 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f678c0abe60 con 0x7f678c0a5fd0 2024-02-21T23:11:42.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.233+0000 7f6793000640 1 -- 172.21.15.28:0/3584810347 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f678c14ece0 con 0x7f678c0a5fd0 2024-02-21T23:11:42.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.235+0000 7f6789600640 1 -- 172.21.15.28:0/3584810347 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6778012a10 con 0x7f678c0a5fd0 2024-02-21T23:11:42.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.235+0000 7f6789600640 1 --2- 172.21.15.28:0/3584810347 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f676c06b520 0x7f676c06d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:42.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.236+0000 7f6789600640 1 -- 172.21.15.28:0/3584810347 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6778089eb0 con 0x7f678c0a5fd0 2024-02-21T23:11:42.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.236+0000 7f6791c00640 1 --2- 172.21.15.28:0/3584810347 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f676c06b520 0x7f676c06d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:42.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.237+0000 7f6791c00640 1 --2- 172.21.15.28:0/3584810347 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f676c06b520 0x7f676c06d9e0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f6780000f60 tx=0x7f6780009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:42.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.240+0000 7f6789600640 1 -- 172.21.15.28:0/3584810347 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f678c14ece0 con 0x7f678c0a5fd0 2024-02-21T23:11:42.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.482+0000 7f6793000640 1 -- 172.21.15.28:0/3584810347 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f678c04f290 con 0x7f678c0a5fd0 2024-02-21T23:11:42.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.483+0000 7f6789600640 1 -- 172.21.15.28:0/3584810347 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 201 v201) v1 ==== 95+0+4948 (secure 0 0 0) 0x7f678c0ac1d0 con 0x7f678c0a5fd0 2024-02-21T23:11:42.471 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:11:42.471 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":201,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6848,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3439896042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3439896042},{"type":"v1","addr":"172.21.15.28:6835","nonce":3439896042}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":196},{"gid":7729,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/3213249808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3213249808},{"type":"v1","addr":"172.21.15.28:6833","nonce":3213249808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":195},{"gid":7735,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/2895690310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2895690310},{"type":"v1","addr":"172.21.15.81:6836","nonce":2895690310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":196}],"filesystems":[{"mdsmap":{"epoch":201,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:11:38.384619+0000","modified":"2024-02-21T23:11:41.472133+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7738},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7738":{"gid":7738,"name":"b","rank":0,"incarnation":197,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6837/1709165826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":1709165826},{"type":"v1","addr":"172.21.15.81:6837","nonce":1709165826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-02-21T23:11:42.471 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 201 2024-02-21T23:11:42.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.486+0000 7f6793000640 1 -- 172.21.15.28:0/3584810347 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f676c06b520 msgr2=0x7f676c06d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:42.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.486+0000 7f6793000640 1 --2- 172.21.15.28:0/3584810347 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f676c06b520 0x7f676c06d9e0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f6780000f60 tx=0x7f6780009040 comp rx=0 tx=0).stop 2024-02-21T23:11:42.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.486+0000 7f6793000640 1 -- 172.21.15.28:0/3584810347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f678c0a5fd0 msgr2=0x7f678c1436f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:42.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.486+0000 7f6793000640 1 --2- 172.21.15.28:0/3584810347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f678c0a5fd0 0x7f678c1436f0 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 crypto rx=0x7f677800c480 tx=0x7f677800c940 comp rx=0 tx=0).stop 2024-02-21T23:11:42.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.486+0000 7f6793000640 1 -- 172.21.15.28:0/3584810347 shutdown_connections 2024-02-21T23:11:42.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.486+0000 7f6793000640 1 --2- 172.21.15.28:0/3584810347 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f676c06b520 0x7f676c06d9e0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:42.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.486+0000 7f6793000640 1 --2- 172.21.15.28:0/3584810347 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f678c150450 0x7f678c143c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:42.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.486+0000 7f6793000640 1 --2- 172.21.15.28:0/3584810347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f678c0a5fd0 0x7f678c1436f0 unknown :-1 s=CLOSED pgs=2071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:42.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.486+0000 7f6793000640 1 --2- 172.21.15.28:0/3584810347 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f678c06ee40 0x7f678c0bfba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:42.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.486+0000 7f6793000640 1 -- 172.21.15.28:0/3584810347 >> 172.21.15.28:0/3584810347 conn(0x7f678c06b630 msgr2=0x7f678c0c64e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:42.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.486+0000 7f6793000640 1 -- 172.21.15.28:0/3584810347 shutdown_connections 2024-02-21T23:11:42.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.486+0000 7f6793000640 1 -- 172.21.15.28:0/3584810347 wait complete. 2024-02-21T23:11:42.493 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:11:42.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.723+0000 7f2177c00640 1 Processor -- start 2024-02-21T23:11:42.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.724+0000 7f2177c00640 1 -- start start 2024-02-21T23:11:42.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.725+0000 7f2177c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f21701525c0 0x7f21700cb390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:42.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.725+0000 7f2177c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f21700c3280 0x7f21700c3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:42.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.725+0000 7f2177c00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2170155120 con 0x7f2170159010 2024-02-21T23:11:42.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.725+0000 7f2177c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2170155390 con 0x7f21700c3280 2024-02-21T23:11:42.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.725+0000 7f2177c00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2170155670 con 0x7f21701525c0 2024-02-21T23:11:42.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.725+0000 7f2175c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f21700c3280 0x7f21700c3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:42.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.725+0000 7f2175c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f21700c3280 0x7f21700c3690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45856/0 (socket says 172.21.15.28:45856) 2024-02-21T23:11:42.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.725+0000 7f2175c00640 1 -- 172.21.15.28:0/2827312243 learned_addr learned my addr 172.21.15.28:0/2827312243 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:42.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.726+0000 7f2175200640 1 --2- 172.21.15.28:0/2827312243 >> v2:172.21.15.81:3300/0 conn(0x7f21701525c0 0x7f21700cb390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:42.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.726+0000 7f2175c00640 1 -- 172.21.15.28:0/2827312243 >> v2:172.21.15.81:3300/0 conn(0x7f21701525c0 msgr2=0x7f21700cb390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:42.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.726+0000 7f2175c00640 1 --2- 172.21.15.28:0/2827312243 >> v2:172.21.15.81:3300/0 conn(0x7f21701525c0 0x7f21700cb390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:42.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.726+0000 7f2175c00640 1 -- 172.21.15.28:0/2827312243 >> v1:172.21.15.81:6789/0 conn(0x7f2170159010 legacy=0x7f21700c6a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:42.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.726+0000 7f2175c00640 1 -- 172.21.15.28:0/2827312243 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2170155950 con 0x7f21700c3280 2024-02-21T23:11:42.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.727+0000 7f2175c00640 1 --2- 172.21.15.28:0/2827312243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21700c3280 0x7f21700c3690 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 crypto rx=0x7f217008cbb0 tx=0x7f216802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5af9346bd21639d server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:42.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.728+0000 7f216f400640 1 -- 172.21.15.28:0/2827312243 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f216802ca80 con 0x7f21700c3280 2024-02-21T23:11:42.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.728+0000 7f216f400640 1 -- 172.21.15.28:0/2827312243 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f216802cc20 con 0x7f21700c3280 2024-02-21T23:11:42.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.728+0000 7f216f400640 1 -- 172.21.15.28:0/2827312243 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21680343f0 con 0x7f21700c3280 2024-02-21T23:11:42.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.728+0000 7f2177c00640 1 -- 172.21.15.28:0/2827312243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21700c3280 msgr2=0x7f21700c3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:42.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.728+0000 7f2177c00640 1 --2- 172.21.15.28:0/2827312243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21700c3280 0x7f21700c3690 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 crypto rx=0x7f217008cbb0 tx=0x7f216802bee0 comp rx=0 tx=0).stop 2024-02-21T23:11:42.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.728+0000 7f2177c00640 1 -- 172.21.15.28:0/2827312243 shutdown_connections 2024-02-21T23:11:42.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.728+0000 7f2177c00640 1 --2- 172.21.15.28:0/2827312243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21700c3280 0x7f21700c3690 unknown :-1 s=CLOSED pgs=2072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:42.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.728+0000 7f2177c00640 1 --2- 172.21.15.28:0/2827312243 >> v2:172.21.15.81:3300/0 conn(0x7f21701525c0 0x7f21700cb390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:42.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.728+0000 7f2177c00640 1 -- 172.21.15.28:0/2827312243 >> 172.21.15.28:0/2827312243 conn(0x7f21700729c0 msgr2=0x7f2170072dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:42.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.729+0000 7f2177c00640 1 -- 172.21.15.28:0/2827312243 shutdown_connections 2024-02-21T23:11:42.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.729+0000 7f2177c00640 1 -- 172.21.15.28:0/2827312243 wait complete. 2024-02-21T23:11:42.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.730+0000 7f2177c00640 1 Processor -- start 2024-02-21T23:11:42.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.730+0000 7f2177c00640 1 -- start start 2024-02-21T23:11:42.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.731+0000 7f2177c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21700c3280 0x7f21700c0230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:42.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.731+0000 7f2177c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f21701525c0 0x7f21700c0770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:42.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.731+0000 7f2177c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f217014c860 0x7f21701539f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:42.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.731+0000 7f2177c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21701594a0 con 0x7f21700c3280 2024-02-21T23:11:42.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.731+0000 7f2177c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21700caff0 con 0x7f21701525c0 2024-02-21T23:11:42.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.731+0000 7f2177c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21700c3bd0 con 0x7f217014c860 2024-02-21T23:11:42.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.732+0000 7f2175200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21700c3280 0x7f21700c0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:42.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.732+0000 7f2175200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21700c3280 0x7f21700c0230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45872/0 (socket says 172.21.15.28:45872) 2024-02-21T23:11:42.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.732+0000 7f2175200640 1 -- 172.21.15.28:0/538402522 learned_addr learned my addr 172.21.15.28:0/538402522 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:42.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.732+0000 7f2175200640 1 -- 172.21.15.28:0/538402522 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f217014c860 msgr2=0x7f21701539f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:42.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.732+0000 7f2175200640 1 --2- 172.21.15.28:0/538402522 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f217014c860 0x7f21701539f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:42.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.732+0000 7f2175200640 1 -- 172.21.15.28:0/538402522 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f21701525c0 msgr2=0x7f21700c0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:42.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.732+0000 7f216fe00640 1 --2- 172.21.15.28:0/538402522 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f21701525c0 0x7f21700c0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:42.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.732+0000 7f2175200640 1 --2- 172.21.15.28:0/538402522 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f21701525c0 0x7f21700c0770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:42.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.732+0000 7f2175200640 1 -- 172.21.15.28:0/538402522 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2170153210 con 0x7f21700c3280 2024-02-21T23:11:42.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.732+0000 7f2175200640 1 --2- 172.21.15.28:0/538402522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21700c3280 0x7f21700c0230 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 crypto rx=0x7f216000d830 tx=0x7f216000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:42.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.733+0000 7f216cc00640 1 -- 172.21.15.28:0/538402522 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f216000b6c0 con 0x7f21700c3280 2024-02-21T23:11:42.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.733+0000 7f216cc00640 1 -- 172.21.15.28:0/538402522 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2160019070 con 0x7f21700c3280 2024-02-21T23:11:42.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.733+0000 7f216cc00640 1 -- 172.21.15.28:0/538402522 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f216000ba80 con 0x7f21700c3280 2024-02-21T23:11:42.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.733+0000 7f2177c00640 1 -- 172.21.15.28:0/538402522 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f217014d780 con 0x7f21700c3280 2024-02-21T23:11:42.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.733+0000 7f2177c00640 1 -- 172.21.15.28:0/538402522 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f217014dd40 con 0x7f21700c3280 2024-02-21T23:11:42.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.734+0000 7f2177c00640 1 -- 172.21.15.28:0/538402522 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2170067f30 con 0x7f21700c3280 2024-02-21T23:11:42.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.735+0000 7f216cc00640 1 -- 172.21.15.28:0/538402522 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f216001e460 con 0x7f21700c3280 2024-02-21T23:11:42.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.736+0000 7f216cc00640 1 --2- 172.21.15.28:0/538402522 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f215006b4d0 0x7f215006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:42.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.736+0000 7f216cc00640 1 -- 172.21.15.28:0/538402522 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f216008aa90 con 0x7f21700c3280 2024-02-21T23:11:42.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.737+0000 7f216fe00640 1 --2- 172.21.15.28:0/538402522 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f215006b4d0 0x7f215006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:42.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.738+0000 7f216fe00640 1 --2- 172.21.15.28:0/538402522 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f215006b4d0 0x7f215006d990 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f2170158bb0 tx=0x7f215c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:42.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.741+0000 7f216cc00640 1 -- 172.21.15.28:0/538402522 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f21600596e0 con 0x7f21700c3280 2024-02-21T23:11:42.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.945+0000 7f2177c00640 1 -- 172.21.15.28:0/538402522 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21700cb320 con 0x7f21700c3280 2024-02-21T23:11:42.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.946+0000 7f216cc00640 1 -- 172.21.15.28:0/538402522 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v186) v1 ==== 74+0+16824 (secure 0 0 0) 0x7f216008a070 con 0x7f21700c3280 2024-02-21T23:11:42.935 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:11:42.935 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":186,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:11:39.917106+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:11:36.316421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"185","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_data","create_time":"2024-02-21T23:11:37.418709+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"186","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:11:42.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.949+0000 7f2177c00640 1 -- 172.21.15.28:0/538402522 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f215006b4d0 msgr2=0x7f215006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:42.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.949+0000 7f2177c00640 1 --2- 172.21.15.28:0/538402522 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f215006b4d0 0x7f215006d990 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f2170158bb0 tx=0x7f215c008040 comp rx=0 tx=0).stop 2024-02-21T23:11:42.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.949+0000 7f2177c00640 1 -- 172.21.15.28:0/538402522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21700c3280 msgr2=0x7f21700c0230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:42.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.949+0000 7f2177c00640 1 --2- 172.21.15.28:0/538402522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21700c3280 0x7f21700c0230 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 crypto rx=0x7f216000d830 tx=0x7f216000dcf0 comp rx=0 tx=0).stop 2024-02-21T23:11:42.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.949+0000 7f2177c00640 1 -- 172.21.15.28:0/538402522 shutdown_connections 2024-02-21T23:11:42.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.949+0000 7f2177c00640 1 --2- 172.21.15.28:0/538402522 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f215006b4d0 0x7f215006d990 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:42.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.949+0000 7f2177c00640 1 --2- 172.21.15.28:0/538402522 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f217014c860 0x7f21701539f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:42.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.949+0000 7f2177c00640 1 --2- 172.21.15.28:0/538402522 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f21701525c0 0x7f21700c0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:42.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.949+0000 7f2177c00640 1 --2- 172.21.15.28:0/538402522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21700c3280 0x7f21700c0230 unknown :-1 s=CLOSED pgs=2073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:42.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.949+0000 7f2177c00640 1 -- 172.21.15.28:0/538402522 >> 172.21.15.28:0/538402522 conn(0x7f21700729c0 msgr2=0x7f21700c73a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:42.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.950+0000 7f2177c00640 1 -- 172.21.15.28:0/538402522 shutdown_connections 2024-02-21T23:11:42.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:42.950+0000 7f2177c00640 1 -- 172.21.15.28:0/538402522 wait complete. 2024-02-21T23:11:42.958 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-02-21T23:11:42.958 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-21T23:11:43.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.189+0000 7fec84e00640 1 Processor -- start 2024-02-21T23:11:43.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.189+0000 7fec84e00640 1 -- start start 2024-02-21T23:11:43.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.191+0000 7fec84e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fec80072ca0 0x7fec80151f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:43.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.191+0000 7fec84e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fec80072770 0x7fec800c32f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:43.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.191+0000 7fec84e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fec8006ec70 0x7fec800c11f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:43.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.191+0000 7fec84e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec80071a60 con 0x7fec8006ec70 2024-02-21T23:11:43.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.191+0000 7fec84e00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec8006f230 con 0x7fec80072ca0 2024-02-21T23:11:43.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.191+0000 7fec84e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec80071c90 con 0x7fec80072770 2024-02-21T23:11:43.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.191+0000 7fec7e800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fec8006ec70 0x7fec800c11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:43.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.191+0000 7fec7e800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fec8006ec70 0x7fec800c11f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45882/0 (socket says 172.21.15.28:45882) 2024-02-21T23:11:43.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.191+0000 7fec7e800640 1 -- 172.21.15.28:0/3901715621 learned_addr learned my addr 172.21.15.28:0/3901715621 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:43.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.192+0000 7fec7de00640 1 --2- 172.21.15.28:0/3901715621 >> v2:172.21.15.81:3300/0 conn(0x7fec80072ca0 0x7fec80151f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:43.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.192+0000 7fec7d400640 1 --2- 172.21.15.28:0/3901715621 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fec80072770 0x7fec800c32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:43.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.192+0000 7fec7e800640 1 -- 172.21.15.28:0/3901715621 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fec80072770 msgr2=0x7fec800c32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:43.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.192+0000 7fec7e800640 1 --2- 172.21.15.28:0/3901715621 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fec80072770 0x7fec800c32f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:43.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.192+0000 7fec7e800640 1 -- 172.21.15.28:0/3901715621 >> v2:172.21.15.81:3300/0 conn(0x7fec80072ca0 msgr2=0x7fec80151f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:43.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.192+0000 7fec7e800640 1 --2- 172.21.15.28:0/3901715621 >> v2:172.21.15.81:3300/0 conn(0x7fec80072ca0 0x7fec80151f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:43.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.192+0000 7fec7e800640 1 -- 172.21.15.28:0/3901715621 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec80071f70 con 0x7fec8006ec70 2024-02-21T23:11:43.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.192+0000 7fec7d400640 1 --2- 172.21.15.28:0/3901715621 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fec80072770 0x7fec800c32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:11:43.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.193+0000 7fec7e800640 1 --2- 172.21.15.28:0/3901715621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec8006ec70 0x7fec800c11f0 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 crypto rx=0x7fec800c3d80 tx=0x7fec7802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac3c2d9414baa6a4 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:43.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.194+0000 7fec7ca00640 1 -- 172.21.15.28:0/3901715621 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec7802ca70 con 0x7fec8006ec70 2024-02-21T23:11:43.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.194+0000 7fec7ca00640 1 -- 172.21.15.28:0/3901715621 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec7802cc10 con 0x7fec8006ec70 2024-02-21T23:11:43.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.194+0000 7fec7ca00640 1 -- 172.21.15.28:0/3901715621 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec780343f0 con 0x7fec8006ec70 2024-02-21T23:11:43.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.195+0000 7fec84e00640 1 -- 172.21.15.28:0/3901715621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec8006ec70 msgr2=0x7fec800c11f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:43.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.195+0000 7fec84e00640 1 --2- 172.21.15.28:0/3901715621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec8006ec70 0x7fec800c11f0 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 crypto rx=0x7fec800c3d80 tx=0x7fec7802bed0 comp rx=0 tx=0).stop 2024-02-21T23:11:43.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.195+0000 7fec84e00640 1 -- 172.21.15.28:0/3901715621 shutdown_connections 2024-02-21T23:11:43.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.195+0000 7fec84e00640 1 --2- 172.21.15.28:0/3901715621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec8006ec70 0x7fec800c11f0 unknown :-1 s=CLOSED pgs=2074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:43.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.195+0000 7fec84e00640 1 --2- 172.21.15.28:0/3901715621 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fec80072770 0x7fec800c32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:43.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.195+0000 7fec84e00640 1 --2- 172.21.15.28:0/3901715621 >> v2:172.21.15.81:3300/0 conn(0x7fec80072ca0 0x7fec80151f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:43.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.195+0000 7fec84e00640 1 -- 172.21.15.28:0/3901715621 >> 172.21.15.28:0/3901715621 conn(0x7fec800c67f0 msgr2=0x7fec800c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:43.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.195+0000 7fec84e00640 1 -- 172.21.15.28:0/3901715621 shutdown_connections 2024-02-21T23:11:43.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.195+0000 7fec84e00640 1 -- 172.21.15.28:0/3901715621 wait complete. 2024-02-21T23:11:43.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.196+0000 7fec84e00640 1 Processor -- start 2024-02-21T23:11:43.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.197+0000 7fec84e00640 1 -- start start 2024-02-21T23:11:43.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.197+0000 7fec84e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fec8006ec70 0x7fec801ee410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:43.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.198+0000 7fec84e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec80072770 0x7fec801ecd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:43.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.198+0000 7fec84e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fec80072ca0 0x7fec801ed260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:43.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.198+0000 7fec84e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec800a6210 con 0x7fec80072770 2024-02-21T23:11:43.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.198+0000 7fec84e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec800a6090 con 0x7fec8006ec70 2024-02-21T23:11:43.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.198+0000 7fec84e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec800c3830 con 0x7fec80072ca0 2024-02-21T23:11:43.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.198+0000 7fec7d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec80072770 0x7fec801ecd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:43.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.198+0000 7fec7d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec80072770 0x7fec801ecd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45888/0 (socket says 172.21.15.28:45888) 2024-02-21T23:11:43.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.198+0000 7fec7d400640 1 -- 172.21.15.28:0/3085949722 learned_addr learned my addr 172.21.15.28:0/3085949722 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:43.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.198+0000 7fec7de00640 1 --2- 172.21.15.28:0/3085949722 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fec8006ec70 0x7fec801ee410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:43.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.198+0000 7fec7e800640 1 --2- 172.21.15.28:0/3085949722 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fec80072ca0 0x7fec801ed260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:43.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.198+0000 7fec7d400640 1 -- 172.21.15.28:0/3085949722 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fec80072ca0 msgr2=0x7fec801ed260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:43.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.198+0000 7fec7d400640 1 --2- 172.21.15.28:0/3085949722 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fec80072ca0 0x7fec801ed260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:43.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.198+0000 7fec7d400640 1 -- 172.21.15.28:0/3085949722 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fec8006ec70 msgr2=0x7fec801ee410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:43.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.198+0000 7fec7d400640 1 --2- 172.21.15.28:0/3085949722 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fec8006ec70 0x7fec801ee410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:43.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.198+0000 7fec7d400640 1 -- 172.21.15.28:0/3085949722 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec801eee00 con 0x7fec80072770 2024-02-21T23:11:43.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.198+0000 7fec7de00640 1 --2- 172.21.15.28:0/3085949722 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fec8006ec70 0x7fec801ee410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:11:43.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.199+0000 7fec7d400640 1 --2- 172.21.15.28:0/3085949722 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec80072770 0x7fec801ecd20 secure :-1 s=READY pgs=2075 cs=0 l=1 rev1=1 crypto rx=0x7fec6c00e810 tx=0x7fec6c00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:43.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.199+0000 7fec76000640 1 -- 172.21.15.28:0/3085949722 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec6c00f040 con 0x7fec80072770 2024-02-21T23:11:43.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.199+0000 7fec76000640 1 -- 172.21.15.28:0/3085949722 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec6c004030 con 0x7fec80072770 2024-02-21T23:11:43.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.199+0000 7fec84e00640 1 -- 172.21.15.28:0/3085949722 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec801ec240 con 0x7fec80072770 2024-02-21T23:11:43.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.199+0000 7fec76000640 1 -- 172.21.15.28:0/3085949722 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec6c004400 con 0x7fec80072770 2024-02-21T23:11:43.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.199+0000 7fec84e00640 1 -- 172.21.15.28:0/3085949722 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec800a9270 con 0x7fec80072770 2024-02-21T23:11:43.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.200+0000 7fec84e00640 1 -- 172.21.15.28:0/3085949722 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec80067f30 con 0x7fec80072770 2024-02-21T23:11:43.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.201+0000 7fec76000640 1 -- 172.21.15.28:0/3085949722 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fec6c00c6d0 con 0x7fec80072770 2024-02-21T23:11:43.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.203+0000 7fec76000640 1 --2- 172.21.15.28:0/3085949722 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fec6006b500 0x7fec6006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:43.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.203+0000 7fec76000640 1 -- 172.21.15.28:0/3085949722 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fec6c089fe0 con 0x7fec80072770 2024-02-21T23:11:43.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.203+0000 7fec7de00640 1 --2- 172.21.15.28:0/3085949722 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fec6006b500 0x7fec6006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:43.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.204+0000 7fec7de00640 1 --2- 172.21.15.28:0/3085949722 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fec6006b500 0x7fec6006d9c0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fec70009af0 tx=0x7fec70008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:43.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.207+0000 7fec76000640 1 -- 172.21.15.28:0/3085949722 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fec6c017020 con 0x7fec80072770 2024-02-21T23:11:43.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.403+0000 7fec84e00640 1 -- 172.21.15.28:0/3085949722 --> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fec801ef0e0 con 0x7fec6006b500 2024-02-21T23:11:43.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.422+0000 7fec76000640 1 -- 172.21.15.28:0/3085949722 <== mgr.4101 v2:172.21.15.81:6832/1848538577 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252564 (secure 0 0 0) 0x7fec801ef0e0 con 0x7fec6006b500 2024-02-21T23:11:43.413 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:11:43.415 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-21T23:11:43.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.428+0000 7fec84e00640 1 -- 172.21.15.28:0/3085949722 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fec6006b500 msgr2=0x7fec6006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:43.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.428+0000 7fec84e00640 1 --2- 172.21.15.28:0/3085949722 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fec6006b500 0x7fec6006d9c0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fec70009af0 tx=0x7fec70008040 comp rx=0 tx=0).stop 2024-02-21T23:11:43.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.428+0000 7fec84e00640 1 -- 172.21.15.28:0/3085949722 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec80072770 msgr2=0x7fec801ecd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:43.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.428+0000 7fec84e00640 1 --2- 172.21.15.28:0/3085949722 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec80072770 0x7fec801ecd20 secure :-1 s=READY pgs=2075 cs=0 l=1 rev1=1 crypto rx=0x7fec6c00e810 tx=0x7fec6c00ecd0 comp rx=0 tx=0).stop 2024-02-21T23:11:43.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.428+0000 7fec84e00640 1 -- 172.21.15.28:0/3085949722 shutdown_connections 2024-02-21T23:11:43.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.428+0000 7fec84e00640 1 --2- 172.21.15.28:0/3085949722 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fec6006b500 0x7fec6006d9c0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:43.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.428+0000 7fec84e00640 1 --2- 172.21.15.28:0/3085949722 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fec80072ca0 0x7fec801ed260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:43.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.428+0000 7fec84e00640 1 --2- 172.21.15.28:0/3085949722 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec80072770 0x7fec801ecd20 unknown :-1 s=CLOSED pgs=2075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:43.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.428+0000 7fec84e00640 1 --2- 172.21.15.28:0/3085949722 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fec8006ec70 0x7fec801ee410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:43.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.428+0000 7fec84e00640 1 -- 172.21.15.28:0/3085949722 >> 172.21.15.28:0/3085949722 conn(0x7fec800c67f0 msgr2=0x7fec801552d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:43.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.428+0000 7fec84e00640 1 -- 172.21.15.28:0/3085949722 shutdown_connections 2024-02-21T23:11:43.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.428+0000 7fec84e00640 1 -- 172.21.15.28:0/3085949722 wait complete. 2024-02-21T23:11:43.437 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":455,"stamp":"2024-02-21T23:11:42.859619+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":138,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":272208,"kb_used_data":18580,"kb_used_omap":12,"kb_used_meta":253491,"kb_avail":754702512,"statfs":{"total":773094113280,"available":772815372288,"internally_reserved":0,"allocated":19025920,"data_stored":10451832,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":259575345},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.968240"},"pg_stats":[{"pgid":"30.3f","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.927639+0000","last_change":"2024-02-21T23:11:38.964544+0000","last_active":"2024-02-21T23:11:39.927639+0000","last_peered":"2024-02-21T23:11:39.927639+0000","last_clean":"2024-02-21T23:11:39.927639+0000","last_became_active":"2024-02-21T23:11:38.961190+0000","last_became_peered":"2024-02-21T23:11:38.961190+0000","last_unstale":"2024-02-21T23:11:39.927639+0000","last_undegraded":"2024-02-21T23:11:39.927639+0000","last_fullsized":"2024-02-21T23:11:39.927639+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:02:02.202127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3e","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.922442+0000","last_change":"2024-02-21T23:11:38.952680+0000","last_active":"2024-02-21T23:11:39.922442+0000","last_peered":"2024-02-21T23:11:39.922442+0000","last_clean":"2024-02-21T23:11:39.922442+0000","last_became_active":"2024-02-21T23:11:38.952372+0000","last_became_peered":"2024-02-21T23:11:38.952372+0000","last_unstale":"2024-02-21T23:11:39.922442+0000","last_undegraded":"2024-02-21T23:11:39.922442+0000","last_fullsized":"2024-02-21T23:11:39.922442+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:04:45.366291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.3d","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.924036+0000","last_change":"2024-02-21T23:11:38.957179+0000","last_active":"2024-02-21T23:11:39.924036+0000","last_peered":"2024-02-21T23:11:39.924036+0000","last_clean":"2024-02-21T23:11:39.924036+0000","last_became_active":"2024-02-21T23:11:38.956906+0000","last_became_peered":"2024-02-21T23:11:38.956906+0000","last_unstale":"2024-02-21T23:11:39.924036+0000","last_undegraded":"2024-02-21T23:11:39.924036+0000","last_fullsized":"2024-02-21T23:11:39.924036+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:13:21.907850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.3c","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.968963+0000","last_change":"2024-02-21T23:11:38.960140+0000","last_active":"2024-02-21T23:11:39.968963+0000","last_peered":"2024-02-21T23:11:39.968963+0000","last_clean":"2024-02-21T23:11:39.968963+0000","last_became_active":"2024-02-21T23:11:38.959847+0000","last_became_peered":"2024-02-21T23:11:38.959847+0000","last_unstale":"2024-02-21T23:11:39.968963+0000","last_undegraded":"2024-02-21T23:11:39.968963+0000","last_fullsized":"2024-02-21T23:11:39.968963+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:09:49.832064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.3b","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.926736+0000","last_change":"2024-02-21T23:11:38.956959+0000","last_active":"2024-02-21T23:11:39.926736+0000","last_peered":"2024-02-21T23:11:39.926736+0000","last_clean":"2024-02-21T23:11:39.926736+0000","last_became_active":"2024-02-21T23:11:38.956652+0000","last_became_peered":"2024-02-21T23:11:38.956652+0000","last_unstale":"2024-02-21T23:11:39.926736+0000","last_undegraded":"2024-02-21T23:11:39.926736+0000","last_fullsized":"2024-02-21T23:11:39.926736+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:26:31.677128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3a","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.434308+0000","last_change":"2024-02-21T23:11:38.966303+0000","last_active":"2024-02-21T23:11:40.434308+0000","last_peered":"2024-02-21T23:11:40.434308+0000","last_clean":"2024-02-21T23:11:40.434308+0000","last_became_active":"2024-02-21T23:11:38.965917+0000","last_became_peered":"2024-02-21T23:11:38.965917+0000","last_unstale":"2024-02-21T23:11:40.434308+0000","last_undegraded":"2024-02-21T23:11:40.434308+0000","last_fullsized":"2024-02-21T23:11:40.434308+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:03:50.259304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.39","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.924121+0000","last_change":"2024-02-21T23:11:38.953945+0000","last_active":"2024-02-21T23:11:39.924121+0000","last_peered":"2024-02-21T23:11:39.924121+0000","last_clean":"2024-02-21T23:11:39.924121+0000","last_became_active":"2024-02-21T23:11:38.952962+0000","last_became_peered":"2024-02-21T23:11:38.952962+0000","last_unstale":"2024-02-21T23:11:39.924121+0000","last_undegraded":"2024-02-21T23:11:39.924121+0000","last_fullsized":"2024-02-21T23:11:39.924121+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:08:06.979800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.38","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.434182+0000","last_change":"2024-02-21T23:11:38.961837+0000","last_active":"2024-02-21T23:11:40.434182+0000","last_peered":"2024-02-21T23:11:40.434182+0000","last_clean":"2024-02-21T23:11:40.434182+0000","last_became_active":"2024-02-21T23:11:38.961484+0000","last_became_peered":"2024-02-21T23:11:38.961484+0000","last_unstale":"2024-02-21T23:11:40.434182+0000","last_undegraded":"2024-02-21T23:11:40.434182+0000","last_fullsized":"2024-02-21T23:11:40.434182+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:26:31.677128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.37","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.434522+0000","last_change":"2024-02-21T23:11:38.957666+0000","last_active":"2024-02-21T23:11:40.434522+0000","last_peered":"2024-02-21T23:11:40.434522+0000","last_clean":"2024-02-21T23:11:40.434522+0000","last_became_active":"2024-02-21T23:11:38.957355+0000","last_became_peered":"2024-02-21T23:11:38.957355+0000","last_unstale":"2024-02-21T23:11:40.434522+0000","last_undegraded":"2024-02-21T23:11:40.434522+0000","last_fullsized":"2024-02-21T23:11:40.434522+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:57:04.820656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.36","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.924435+0000","last_change":"2024-02-21T23:11:38.964360+0000","last_active":"2024-02-21T23:11:39.924435+0000","last_peered":"2024-02-21T23:11:39.924435+0000","last_clean":"2024-02-21T23:11:39.924435+0000","last_became_active":"2024-02-21T23:11:38.961529+0000","last_became_peered":"2024-02-21T23:11:38.961529+0000","last_unstale":"2024-02-21T23:11:39.924435+0000","last_undegraded":"2024-02-21T23:11:39.924435+0000","last_fullsized":"2024-02-21T23:11:39.924435+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:29:32.351053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.35","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.969343+0000","last_change":"2024-02-21T23:11:38.945662+0000","last_active":"2024-02-21T23:11:39.969343+0000","last_peered":"2024-02-21T23:11:39.969343+0000","last_clean":"2024-02-21T23:11:39.969343+0000","last_became_active":"2024-02-21T23:11:38.945165+0000","last_became_peered":"2024-02-21T23:11:38.945165+0000","last_unstale":"2024-02-21T23:11:39.969343+0000","last_undegraded":"2024-02-21T23:11:39.969343+0000","last_fullsized":"2024-02-21T23:11:39.969343+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:06:56.052476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.34","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.436410+0000","last_change":"2024-02-21T23:11:38.961228+0000","last_active":"2024-02-21T23:11:40.436410+0000","last_peered":"2024-02-21T23:11:40.436410+0000","last_clean":"2024-02-21T23:11:40.436410+0000","last_became_active":"2024-02-21T23:11:38.960910+0000","last_became_peered":"2024-02-21T23:11:38.960910+0000","last_unstale":"2024-02-21T23:11:40.436410+0000","last_undegraded":"2024-02-21T23:11:40.436410+0000","last_fullsized":"2024-02-21T23:11:40.436410+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:59:13.487532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.33","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.434543+0000","last_change":"2024-02-21T23:11:38.963302+0000","last_active":"2024-02-21T23:11:40.434543+0000","last_peered":"2024-02-21T23:11:40.434543+0000","last_clean":"2024-02-21T23:11:40.434543+0000","last_became_active":"2024-02-21T23:11:38.963026+0000","last_became_peered":"2024-02-21T23:11:38.963026+0000","last_unstale":"2024-02-21T23:11:40.434543+0000","last_undegraded":"2024-02-21T23:11:40.434543+0000","last_fullsized":"2024-02-21T23:11:40.434543+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:31:19.660504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.32","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.435800+0000","last_change":"2024-02-21T23:11:38.957576+0000","last_active":"2024-02-21T23:11:40.435800+0000","last_peered":"2024-02-21T23:11:40.435800+0000","last_clean":"2024-02-21T23:11:40.435800+0000","last_became_active":"2024-02-21T23:11:38.957225+0000","last_became_peered":"2024-02-21T23:11:38.957225+0000","last_unstale":"2024-02-21T23:11:40.435800+0000","last_undegraded":"2024-02-21T23:11:40.435800+0000","last_fullsized":"2024-02-21T23:11:40.435800+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:21:04.858085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.31","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.964102+0000","last_change":"2024-02-21T23:11:38.951853+0000","last_active":"2024-02-21T23:11:39.964102+0000","last_peered":"2024-02-21T23:11:39.964102+0000","last_clean":"2024-02-21T23:11:39.964102+0000","last_became_active":"2024-02-21T23:11:38.951290+0000","last_became_peered":"2024-02-21T23:11:38.951290+0000","last_unstale":"2024-02-21T23:11:39.964102+0000","last_undegraded":"2024-02-21T23:11:39.964102+0000","last_fullsized":"2024-02-21T23:11:39.964102+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:31:19.660504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.30","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.434595+0000","last_change":"2024-02-21T23:11:38.956435+0000","last_active":"2024-02-21T23:11:40.434595+0000","last_peered":"2024-02-21T23:11:40.434595+0000","last_clean":"2024-02-21T23:11:40.434595+0000","last_became_active":"2024-02-21T23:11:38.956104+0000","last_became_peered":"2024-02-21T23:11:38.956104+0000","last_unstale":"2024-02-21T23:11:40.434595+0000","last_undegraded":"2024-02-21T23:11:40.434595+0000","last_fullsized":"2024-02-21T23:11:40.434595+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:17:16.149146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.2f","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.437462+0000","last_change":"2024-02-21T23:11:38.962805+0000","last_active":"2024-02-21T23:11:40.437462+0000","last_peered":"2024-02-21T23:11:40.437462+0000","last_clean":"2024-02-21T23:11:40.437462+0000","last_became_active":"2024-02-21T23:11:38.962491+0000","last_became_peered":"2024-02-21T23:11:38.962491+0000","last_unstale":"2024-02-21T23:11:40.437462+0000","last_undegraded":"2024-02-21T23:11:40.437462+0000","last_fullsized":"2024-02-21T23:11:40.437462+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:00:08.374460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2e","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.924542+0000","last_change":"2024-02-21T23:11:38.959763+0000","last_active":"2024-02-21T23:11:39.924542+0000","last_peered":"2024-02-21T23:11:39.924542+0000","last_clean":"2024-02-21T23:11:39.924542+0000","last_became_active":"2024-02-21T23:11:38.959444+0000","last_became_peered":"2024-02-21T23:11:38.959444+0000","last_unstale":"2024-02-21T23:11:39.924542+0000","last_undegraded":"2024-02-21T23:11:39.924542+0000","last_fullsized":"2024-02-21T23:11:39.924542+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:55:37.022532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.2d","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.926769+0000","last_change":"2024-02-21T23:11:38.954192+0000","last_active":"2024-02-21T23:11:39.926769+0000","last_peered":"2024-02-21T23:11:39.926769+0000","last_clean":"2024-02-21T23:11:39.926769+0000","last_became_active":"2024-02-21T23:11:38.951922+0000","last_became_peered":"2024-02-21T23:11:38.951922+0000","last_unstale":"2024-02-21T23:11:39.926769+0000","last_undegraded":"2024-02-21T23:11:39.926769+0000","last_fullsized":"2024-02-21T23:11:39.926769+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:03:50.259304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.2c","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.434812+0000","last_change":"2024-02-21T23:11:38.958868+0000","last_active":"2024-02-21T23:11:40.434812+0000","last_peered":"2024-02-21T23:11:40.434812+0000","last_clean":"2024-02-21T23:11:40.434812+0000","last_became_active":"2024-02-21T23:11:38.958581+0000","last_became_peered":"2024-02-21T23:11:38.958581+0000","last_unstale":"2024-02-21T23:11:40.434812+0000","last_undegraded":"2024-02-21T23:11:40.434812+0000","last_fullsized":"2024-02-21T23:11:40.434812+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:12:34.312401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.2b","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.924084+0000","last_change":"2024-02-21T23:11:38.959096+0000","last_active":"2024-02-21T23:11:39.924084+0000","last_peered":"2024-02-21T23:11:39.924084+0000","last_clean":"2024-02-21T23:11:39.924084+0000","last_became_active":"2024-02-21T23:11:38.958785+0000","last_became_peered":"2024-02-21T23:11:38.958785+0000","last_unstale":"2024-02-21T23:11:39.924084+0000","last_undegraded":"2024-02-21T23:11:39.924084+0000","last_fullsized":"2024-02-21T23:11:39.924084+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:57:30.581817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.2a","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.963952+0000","last_change":"2024-02-21T23:11:38.965338+0000","last_active":"2024-02-21T23:11:39.963952+0000","last_peered":"2024-02-21T23:11:39.963952+0000","last_clean":"2024-02-21T23:11:39.963952+0000","last_became_active":"2024-02-21T23:11:38.964929+0000","last_became_peered":"2024-02-21T23:11:38.964929+0000","last_unstale":"2024-02-21T23:11:39.963952+0000","last_undegraded":"2024-02-21T23:11:39.963952+0000","last_fullsized":"2024-02-21T23:11:39.963952+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:03:50.259304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.29","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.434256+0000","last_change":"2024-02-21T23:11:38.948385+0000","last_active":"2024-02-21T23:11:40.434256+0000","last_peered":"2024-02-21T23:11:40.434256+0000","last_clean":"2024-02-21T23:11:40.434256+0000","last_became_active":"2024-02-21T23:11:38.948029+0000","last_became_peered":"2024-02-21T23:11:38.948029+0000","last_unstale":"2024-02-21T23:11:40.434256+0000","last_undegraded":"2024-02-21T23:11:40.434256+0000","last_fullsized":"2024-02-21T23:11:40.434256+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:00:31.514375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.28","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.926039+0000","last_change":"2024-02-21T23:11:38.949777+0000","last_active":"2024-02-21T23:11:39.926039+0000","last_peered":"2024-02-21T23:11:39.926039+0000","last_clean":"2024-02-21T23:11:39.926039+0000","last_became_active":"2024-02-21T23:11:38.949390+0000","last_became_peered":"2024-02-21T23:11:38.949390+0000","last_unstale":"2024-02-21T23:11:39.926039+0000","last_undegraded":"2024-02-21T23:11:39.926039+0000","last_fullsized":"2024-02-21T23:11:39.926039+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:59:23.931824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.27","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.927718+0000","last_change":"2024-02-21T23:11:38.954781+0000","last_active":"2024-02-21T23:11:39.927718+0000","last_peered":"2024-02-21T23:11:39.927718+0000","last_clean":"2024-02-21T23:11:39.927718+0000","last_became_active":"2024-02-21T23:11:38.954507+0000","last_became_peered":"2024-02-21T23:11:38.954507+0000","last_unstale":"2024-02-21T23:11:39.927718+0000","last_undegraded":"2024-02-21T23:11:39.927718+0000","last_fullsized":"2024-02-21T23:11:39.927718+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:51:37.502229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.26","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.924611+0000","last_change":"2024-02-21T23:11:38.966080+0000","last_active":"2024-02-21T23:11:39.924611+0000","last_peered":"2024-02-21T23:11:39.924611+0000","last_clean":"2024-02-21T23:11:39.924611+0000","last_became_active":"2024-02-21T23:11:38.965723+0000","last_became_peered":"2024-02-21T23:11:38.965723+0000","last_unstale":"2024-02-21T23:11:39.924611+0000","last_undegraded":"2024-02-21T23:11:39.924611+0000","last_fullsized":"2024-02-21T23:11:39.924611+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:27:35.936913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.25","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.969309+0000","last_change":"2024-02-21T23:11:38.948536+0000","last_active":"2024-02-21T23:11:39.969309+0000","last_peered":"2024-02-21T23:11:39.969309+0000","last_clean":"2024-02-21T23:11:39.969309+0000","last_became_active":"2024-02-21T23:11:38.947950+0000","last_became_peered":"2024-02-21T23:11:38.947950+0000","last_unstale":"2024-02-21T23:11:39.969309+0000","last_undegraded":"2024-02-21T23:11:39.969309+0000","last_fullsized":"2024-02-21T23:11:39.969309+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:31:55.730966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.24","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.436784+0000","last_change":"2024-02-21T23:11:38.963502+0000","last_active":"2024-02-21T23:11:40.436784+0000","last_peered":"2024-02-21T23:11:40.436784+0000","last_clean":"2024-02-21T23:11:40.436784+0000","last_became_active":"2024-02-21T23:11:38.963163+0000","last_became_peered":"2024-02-21T23:11:38.963163+0000","last_unstale":"2024-02-21T23:11:40.436784+0000","last_undegraded":"2024-02-21T23:11:40.436784+0000","last_fullsized":"2024-02-21T23:11:40.436784+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:48:57.464584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.23","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.922220+0000","last_change":"2024-02-21T23:11:38.954372+0000","last_active":"2024-02-21T23:11:39.922220+0000","last_peered":"2024-02-21T23:11:39.922220+0000","last_clean":"2024-02-21T23:11:39.922220+0000","last_became_active":"2024-02-21T23:11:38.954003+0000","last_became_peered":"2024-02-21T23:11:38.954003+0000","last_unstale":"2024-02-21T23:11:39.922220+0000","last_undegraded":"2024-02-21T23:11:39.922220+0000","last_fullsized":"2024-02-21T23:11:39.922220+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:19:08.443946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.22","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.433889+0000","last_change":"2024-02-21T23:11:38.968611+0000","last_active":"2024-02-21T23:11:40.433889+0000","last_peered":"2024-02-21T23:11:40.433889+0000","last_clean":"2024-02-21T23:11:40.433889+0000","last_became_active":"2024-02-21T23:11:38.967853+0000","last_became_peered":"2024-02-21T23:11:38.967853+0000","last_unstale":"2024-02-21T23:11:40.433889+0000","last_undegraded":"2024-02-21T23:11:40.433889+0000","last_fullsized":"2024-02-21T23:11:40.433889+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:21:58.147248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.21","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.434048+0000","last_change":"2024-02-21T23:11:38.946771+0000","last_active":"2024-02-21T23:11:40.434048+0000","last_peered":"2024-02-21T23:11:40.434048+0000","last_clean":"2024-02-21T23:11:40.434048+0000","last_became_active":"2024-02-21T23:11:38.943409+0000","last_became_peered":"2024-02-21T23:11:38.943409+0000","last_unstale":"2024-02-21T23:11:40.434048+0000","last_undegraded":"2024-02-21T23:11:40.434048+0000","last_fullsized":"2024-02-21T23:11:40.434048+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:16:11.580478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.20","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.434678+0000","last_change":"2024-02-21T23:11:38.969678+0000","last_active":"2024-02-21T23:11:40.434678+0000","last_peered":"2024-02-21T23:11:40.434678+0000","last_clean":"2024-02-21T23:11:40.434678+0000","last_became_active":"2024-02-21T23:11:38.969387+0000","last_became_peered":"2024-02-21T23:11:38.969387+0000","last_unstale":"2024-02-21T23:11:40.434678+0000","last_undegraded":"2024-02-21T23:11:40.434678+0000","last_fullsized":"2024-02-21T23:11:40.434678+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:51:37.502229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1c","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.924328+0000","last_change":"2024-02-21T23:11:38.966212+0000","last_active":"2024-02-21T23:11:39.924328+0000","last_peered":"2024-02-21T23:11:39.924328+0000","last_clean":"2024-02-21T23:11:39.924328+0000","last_became_active":"2024-02-21T23:11:38.962483+0000","last_became_peered":"2024-02-21T23:11:38.962483+0000","last_unstale":"2024-02-21T23:11:39.924328+0000","last_undegraded":"2024-02-21T23:11:39.924328+0000","last_fullsized":"2024-02-21T23:11:39.924328+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:19:56.590895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1f","version":"185'4","reported_seq":18,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.434557+0000","last_change":"2024-02-21T23:11:37.930503+0000","last_active":"2024-02-21T23:11:40.434557+0000","last_peered":"2024-02-21T23:11:40.434557+0000","last_clean":"2024-02-21T23:11:40.434557+0000","last_became_active":"2024-02-21T23:11:37.929937+0000","last_became_peered":"2024-02-21T23:11:37.929937+0000","last_unstale":"2024-02-21T23:11:40.434557+0000","last_undegraded":"2024-02-21T23:11:40.434557+0000","last_fullsized":"2024-02-21T23:11:40.434557+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_clean_scrub_stamp":"2024-02-21T23:11:36.873263+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:50:08.578636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.1d","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.924391+0000","last_change":"2024-02-21T23:11:38.950075+0000","last_active":"2024-02-21T23:11:39.924391+0000","last_peered":"2024-02-21T23:11:39.924391+0000","last_clean":"2024-02-21T23:11:39.924391+0000","last_became_active":"2024-02-21T23:11:38.949082+0000","last_became_peered":"2024-02-21T23:11:38.949082+0000","last_unstale":"2024-02-21T23:11:39.924391+0000","last_undegraded":"2024-02-21T23:11:39.924391+0000","last_fullsized":"2024-02-21T23:11:39.924391+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:25:24.775484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1e","version":"185'1","reported_seq":15,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.437864+0000","last_change":"2024-02-21T23:11:37.941756+0000","last_active":"2024-02-21T23:11:40.437864+0000","last_peered":"2024-02-21T23:11:40.437864+0000","last_clean":"2024-02-21T23:11:40.437864+0000","last_became_active":"2024-02-21T23:11:37.941250+0000","last_became_peered":"2024-02-21T23:11:37.941250+0000","last_unstale":"2024-02-21T23:11:40.437864+0000","last_undegraded":"2024-02-21T23:11:40.437864+0000","last_fullsized":"2024-02-21T23:11:40.437864+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_clean_scrub_stamp":"2024-02-21T23:11:36.873263+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:20:58.386539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1e","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.437255+0000","last_change":"2024-02-21T23:11:38.961071+0000","last_active":"2024-02-21T23:11:40.437255+0000","last_peered":"2024-02-21T23:11:40.437255+0000","last_clean":"2024-02-21T23:11:40.437255+0000","last_became_active":"2024-02-21T23:11:38.960702+0000","last_became_peered":"2024-02-21T23:11:38.960702+0000","last_unstale":"2024-02-21T23:11:40.437255+0000","last_undegraded":"2024-02-21T23:11:40.437255+0000","last_fullsized":"2024-02-21T23:11:40.437255+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:48:17.561370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1d","version":"185'2","reported_seq":16,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.433955+0000","last_change":"2024-02-21T23:11:37.939613+0000","last_active":"2024-02-21T23:11:40.433955+0000","last_peered":"2024-02-21T23:11:40.433955+0000","last_clean":"2024-02-21T23:11:40.433955+0000","last_became_active":"2024-02-21T23:11:37.939055+0000","last_became_peered":"2024-02-21T23:11:37.939055+0000","last_unstale":"2024-02-21T23:11:40.433955+0000","last_undegraded":"2024-02-21T23:11:40.433955+0000","last_fullsized":"2024-02-21T23:11:40.433955+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_clean_scrub_stamp":"2024-02-21T23:11:36.873263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:48:56.448606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.923929+0000","last_change":"2024-02-21T23:11:38.960612+0000","last_active":"2024-02-21T23:11:39.923929+0000","last_peered":"2024-02-21T23:11:39.923929+0000","last_clean":"2024-02-21T23:11:39.923929+0000","last_became_active":"2024-02-21T23:11:38.957029+0000","last_became_peered":"2024-02-21T23:11:38.957029+0000","last_unstale":"2024-02-21T23:11:39.923929+0000","last_undegraded":"2024-02-21T23:11:39.923929+0000","last_fullsized":"2024-02-21T23:11:39.923929+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:05:18.211754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1b","version":"185'1","reported_seq":15,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.964114+0000","last_change":"2024-02-21T23:11:37.938169+0000","last_active":"2024-02-21T23:11:39.964114+0000","last_peered":"2024-02-21T23:11:39.964114+0000","last_clean":"2024-02-21T23:11:39.964114+0000","last_became_active":"2024-02-21T23:11:37.937884+0000","last_became_peered":"2024-02-21T23:11:37.937884+0000","last_unstale":"2024-02-21T23:11:39.964114+0000","last_undegraded":"2024-02-21T23:11:39.964114+0000","last_fullsized":"2024-02-21T23:11:39.964114+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_clean_scrub_stamp":"2024-02-21T23:11:36.873263+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:18:11.620839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.436256+0000","last_change":"2024-02-21T23:11:38.959036+0000","last_active":"2024-02-21T23:11:40.436256+0000","last_peered":"2024-02-21T23:11:40.436256+0000","last_clean":"2024-02-21T23:11:40.436256+0000","last_became_active":"2024-02-21T23:11:38.958669+0000","last_became_peered":"2024-02-21T23:11:38.958669+0000","last_unstale":"2024-02-21T23:11:40.436256+0000","last_undegraded":"2024-02-21T23:11:40.436256+0000","last_fullsized":"2024-02-21T23:11:40.436256+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:49:02.531250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1a","version":"185'1","reported_seq":15,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.926489+0000","last_change":"2024-02-21T23:11:37.943744+0000","last_active":"2024-02-21T23:11:39.926489+0000","last_peered":"2024-02-21T23:11:39.926489+0000","last_clean":"2024-02-21T23:11:39.926489+0000","last_became_active":"2024-02-21T23:11:37.943425+0000","last_became_peered":"2024-02-21T23:11:37.943425+0000","last_unstale":"2024-02-21T23:11:39.926489+0000","last_undegraded":"2024-02-21T23:11:39.926489+0000","last_fullsized":"2024-02-21T23:11:39.926489+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_clean_scrub_stamp":"2024-02-21T23:11:36.873263+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:24:31.476916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.1a","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.963984+0000","last_change":"2024-02-21T23:11:38.965961+0000","last_active":"2024-02-21T23:11:39.963984+0000","last_peered":"2024-02-21T23:11:39.963984+0000","last_clean":"2024-02-21T23:11:39.963984+0000","last_became_active":"2024-02-21T23:11:38.964488+0000","last_became_peered":"2024-02-21T23:11:38.964488+0000","last_unstale":"2024-02-21T23:11:39.963984+0000","last_undegraded":"2024-02-21T23:11:39.963984+0000","last_fullsized":"2024-02-21T23:11:39.963984+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:26:31.677128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":14,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.968674+0000","last_change":"2024-02-21T23:11:37.940706+0000","last_active":"2024-02-21T23:11:39.968674+0000","last_peered":"2024-02-21T23:11:39.968674+0000","last_clean":"2024-02-21T23:11:39.968674+0000","last_became_active":"2024-02-21T23:11:37.940426+0000","last_became_peered":"2024-02-21T23:11:37.940426+0000","last_unstale":"2024-02-21T23:11:39.968674+0000","last_undegraded":"2024-02-21T23:11:39.968674+0000","last_fullsized":"2024-02-21T23:11:39.968674+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_clean_scrub_stamp":"2024-02-21T23:11:36.873263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:07:51.029760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1b","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.434037+0000","last_change":"2024-02-21T23:11:38.952592+0000","last_active":"2024-02-21T23:11:40.434037+0000","last_peered":"2024-02-21T23:11:40.434037+0000","last_clean":"2024-02-21T23:11:40.434037+0000","last_became_active":"2024-02-21T23:11:38.952229+0000","last_became_peered":"2024-02-21T23:11:38.952229+0000","last_unstale":"2024-02-21T23:11:40.434037+0000","last_undegraded":"2024-02-21T23:11:40.434037+0000","last_fullsized":"2024-02-21T23:11:40.434037+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:22:34.217711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":14,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.924097+0000","last_change":"2024-02-21T23:11:37.924750+0000","last_active":"2024-02-21T23:11:39.924097+0000","last_peered":"2024-02-21T23:11:39.924097+0000","last_clean":"2024-02-21T23:11:39.924097+0000","last_became_active":"2024-02-21T23:11:37.924416+0000","last_became_peered":"2024-02-21T23:11:37.924416+0000","last_unstale":"2024-02-21T23:11:39.924097+0000","last_undegraded":"2024-02-21T23:11:39.924097+0000","last_fullsized":"2024-02-21T23:11:39.924097+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_clean_scrub_stamp":"2024-02-21T23:11:36.873263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:09:40.459144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.14","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.964270+0000","last_change":"2024-02-21T23:11:38.958101+0000","last_active":"2024-02-21T23:11:39.964270+0000","last_peered":"2024-02-21T23:11:39.964270+0000","last_clean":"2024-02-21T23:11:39.964270+0000","last_became_active":"2024-02-21T23:11:38.957576+0000","last_became_peered":"2024-02-21T23:11:38.957576+0000","last_unstale":"2024-02-21T23:11:39.964270+0000","last_undegraded":"2024-02-21T23:11:39.964270+0000","last_fullsized":"2024-02-21T23:11:39.964270+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:17:16.149146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":14,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.927900+0000","last_change":"2024-02-21T23:11:37.938249+0000","last_active":"2024-02-21T23:11:39.927900+0000","last_peered":"2024-02-21T23:11:39.927900+0000","last_clean":"2024-02-21T23:11:39.927900+0000","last_became_active":"2024-02-21T23:11:37.936634+0000","last_became_peered":"2024-02-21T23:11:37.936634+0000","last_unstale":"2024-02-21T23:11:39.927900+0000","last_undegraded":"2024-02-21T23:11:39.927900+0000","last_fullsized":"2024-02-21T23:11:39.927900+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_clean_scrub_stamp":"2024-02-21T23:11:36.873263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:59:12.471554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.15","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.964344+0000","last_change":"2024-02-21T23:11:38.952575+0000","last_active":"2024-02-21T23:11:39.964344+0000","last_peered":"2024-02-21T23:11:39.964344+0000","last_clean":"2024-02-21T23:11:39.964344+0000","last_became_active":"2024-02-21T23:11:38.952309+0000","last_became_peered":"2024-02-21T23:11:38.952309+0000","last_unstale":"2024-02-21T23:11:39.964344+0000","last_undegraded":"2024-02-21T23:11:39.964344+0000","last_fullsized":"2024-02-21T23:11:39.964344+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:02:02.202127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":14,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.969142+0000","last_change":"2024-02-21T23:11:37.943486+0000","last_active":"2024-02-21T23:11:39.969142+0000","last_peered":"2024-02-21T23:11:39.969142+0000","last_clean":"2024-02-21T23:11:39.969142+0000","last_became_active":"2024-02-21T23:11:37.943117+0000","last_became_peered":"2024-02-21T23:11:37.943117+0000","last_unstale":"2024-02-21T23:11:39.969142+0000","last_undegraded":"2024-02-21T23:11:39.969142+0000","last_fullsized":"2024-02-21T23:11:39.969142+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_clean_scrub_stamp":"2024-02-21T23:11:36.873263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:15:16.910929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.924833+0000","last_change":"2024-02-21T23:11:38.951990+0000","last_active":"2024-02-21T23:11:39.924833+0000","last_peered":"2024-02-21T23:11:39.924833+0000","last_clean":"2024-02-21T23:11:39.924833+0000","last_became_active":"2024-02-21T23:11:38.949433+0000","last_became_peered":"2024-02-21T23:11:38.949433+0000","last_unstale":"2024-02-21T23:11:39.924833+0000","last_undegraded":"2024-02-21T23:11:39.924833+0000","last_fullsized":"2024-02-21T23:11:39.924833+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:34:42.869435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.15","version":"185'3","reported_seq":17,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.927503+0000","last_change":"2024-02-21T23:11:37.936967+0000","last_active":"2024-02-21T23:11:39.927503+0000","last_peered":"2024-02-21T23:11:39.927503+0000","last_clean":"2024-02-21T23:11:39.927503+0000","last_became_active":"2024-02-21T23:11:37.936530+0000","last_became_peered":"2024-02-21T23:11:37.936530+0000","last_unstale":"2024-02-21T23:11:39.927503+0000","last_undegraded":"2024-02-21T23:11:39.927503+0000","last_fullsized":"2024-02-21T23:11:39.927503+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_clean_scrub_stamp":"2024-02-21T23:11:36.873263+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:48:56.448606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.969445+0000","last_change":"2024-02-21T23:11:38.949681+0000","last_active":"2024-02-21T23:11:39.969445+0000","last_peered":"2024-02-21T23:11:39.969445+0000","last_clean":"2024-02-21T23:11:39.969445+0000","last_became_active":"2024-02-21T23:11:38.944954+0000","last_became_peered":"2024-02-21T23:11:38.944954+0000","last_unstale":"2024-02-21T23:11:39.969445+0000","last_undegraded":"2024-02-21T23:11:39.969445+0000","last_fullsized":"2024-02-21T23:11:39.969445+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:58:43.457178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.14","version":"185'6","reported_seq":20,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.434839+0000","last_change":"2024-02-21T23:11:37.939784+0000","last_active":"2024-02-21T23:11:40.434839+0000","last_peered":"2024-02-21T23:11:40.434839+0000","last_clean":"2024-02-21T23:11:40.434839+0000","last_became_active":"2024-02-21T23:11:37.939524+0000","last_became_peered":"2024-02-21T23:11:37.939524+0000","last_unstale":"2024-02-21T23:11:40.434839+0000","last_undegraded":"2024-02-21T23:11:40.434839+0000","last_fullsized":"2024-02-21T23:11:40.434839+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_clean_scrub_stamp":"2024-02-21T23:11:36.873263+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:24:31.476916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.10","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.921928+0000","last_change":"2024-02-21T23:11:38.954494+0000","last_active":"2024-02-21T23:11:39.921928+0000","last_peered":"2024-02-21T23:11:39.921928+0000","last_clean":"2024-02-21T23:11:39.921928+0000","last_became_active":"2024-02-21T23:11:38.954120+0000","last_became_peered":"2024-02-21T23:11:38.954120+0000","last_unstale":"2024-02-21T23:11:39.921928+0000","last_undegraded":"2024-02-21T23:11:39.921928+0000","last_fullsized":"2024-02-21T23:11:39.921928+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:51:46.324459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.13","version":"185'2","reported_seq":16,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.436322+0000","last_change":"2024-02-21T23:11:37.937137+0000","last_active":"2024-02-21T23:11:40.436322+0000","last_peered":"2024-02-21T23:11:40.436322+0000","last_clean":"2024-02-21T23:11:40.436322+0000","last_became_active":"2024-02-21T23:11:37.936274+0000","last_became_peered":"2024-02-21T23:11:37.936274+0000","last_unstale":"2024-02-21T23:11:40.436322+0000","last_undegraded":"2024-02-21T23:11:40.436322+0000","last_fullsized":"2024-02-21T23:11:40.436322+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_clean_scrub_stamp":"2024-02-21T23:11:36.873263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:15:16.910929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.436031+0000","last_change":"2024-02-21T23:11:38.961331+0000","last_active":"2024-02-21T23:11:40.436031+0000","last_peered":"2024-02-21T23:11:40.436031+0000","last_clean":"2024-02-21T23:11:40.436031+0000","last_became_active":"2024-02-21T23:11:38.960863+0000","last_became_peered":"2024-02-21T23:11:38.960863+0000","last_unstale":"2024-02-21T23:11:40.436031+0000","last_undegraded":"2024-02-21T23:11:40.436031+0000","last_fullsized":"2024-02-21T23:11:40.436031+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:36:58.572937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":14,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.922279+0000","last_change":"2024-02-21T23:11:37.916816+0000","last_active":"2024-02-21T23:11:39.922279+0000","last_peered":"2024-02-21T23:11:39.922279+0000","last_clean":"2024-02-21T23:11:39.922279+0000","last_became_active":"2024-02-21T23:11:37.916468+0000","last_became_peered":"2024-02-21T23:11:37.916468+0000","last_unstale":"2024-02-21T23:11:39.922279+0000","last_undegraded":"2024-02-21T23:11:39.922279+0000","last_fullsized":"2024-02-21T23:11:39.922279+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_clean_scrub_stamp":"2024-02-21T23:11:36.873263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:51:36.486251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.924309+0000","last_change":"2024-02-21T23:11:38.954537+0000","last_active":"2024-02-21T23:11:39.924309+0000","last_peered":"2024-02-21T23:11:39.924309+0000","last_clean":"2024-02-21T23:11:39.924309+0000","last_became_active":"2024-02-21T23:11:38.952533+0000","last_became_peered":"2024-02-21T23:11:38.952533+0000","last_unstale":"2024-02-21T23:11:39.924309+0000","last_undegraded":"2024-02-21T23:11:39.924309+0000","last_fullsized":"2024-02-21T23:11:39.924309+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:47:54.894682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":14,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.433602+0000","last_change":"2024-02-21T23:11:37.926644+0000","last_active":"2024-02-21T23:11:40.433602+0000","last_peered":"2024-02-21T23:11:40.433602+0000","last_clean":"2024-02-21T23:11:40.433602+0000","last_became_active":"2024-02-21T23:11:37.926222+0000","last_became_peered":"2024-02-21T23:11:37.926222+0000","last_unstale":"2024-02-21T23:11:40.433602+0000","last_undegraded":"2024-02-21T23:11:40.433602+0000","last_fullsized":"2024-02-21T23:11:40.433602+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_clean_scrub_stamp":"2024-02-21T23:11:36.873263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:59:12.471554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.13","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.964661+0000","last_change":"2024-02-21T23:11:38.961360+0000","last_active":"2024-02-21T23:11:39.964661+0000","last_peered":"2024-02-21T23:11:39.964661+0000","last_clean":"2024-02-21T23:11:39.964661+0000","last_became_active":"2024-02-21T23:11:38.961075+0000","last_became_peered":"2024-02-21T23:11:38.961075+0000","last_unstale":"2024-02-21T23:11:39.964661+0000","last_undegraded":"2024-02-21T23:11:39.964661+0000","last_fullsized":"2024-02-21T23:11:39.964661+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:51:37.502229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.10","version":"185'8","reported_seq":22,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.435920+0000","last_change":"2024-02-21T23:11:37.934144+0000","last_active":"2024-02-21T23:11:40.435920+0000","last_peered":"2024-02-21T23:11:40.435920+0000","last_clean":"2024-02-21T23:11:40.435920+0000","last_became_active":"2024-02-21T23:11:37.933846+0000","last_became_peered":"2024-02-21T23:11:37.933846+0000","last_unstale":"2024-02-21T23:11:40.435920+0000","last_undegraded":"2024-02-21T23:11:40.435920+0000","last_fullsized":"2024-02-21T23:11:40.435920+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_clean_scrub_stamp":"2024-02-21T23:11:36.873263+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:06:31.857676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.922454+0000","last_change":"2024-02-21T23:11:38.957366+0000","last_active":"2024-02-21T23:11:39.922454+0000","last_peered":"2024-02-21T23:11:39.922454+0000","last_clean":"2024-02-21T23:11:39.922454+0000","last_became_active":"2024-02-21T23:11:38.957080+0000","last_became_peered":"2024-02-21T23:11:38.957080+0000","last_unstale":"2024-02-21T23:11:39.922454+0000","last_undegraded":"2024-02-21T23:11:39.922454+0000","last_fullsized":"2024-02-21T23:11:39.922454+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:16:04.419840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.f","version":"185'1","reported_seq":15,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.924421+0000","last_change":"2024-02-21T23:11:37.940069+0000","last_active":"2024-02-21T23:11:39.924421+0000","last_peered":"2024-02-21T23:11:39.924421+0000","last_clean":"2024-02-21T23:11:39.924421+0000","last_became_active":"2024-02-21T23:11:37.939722+0000","last_became_peered":"2024-02-21T23:11:37.939722+0000","last_unstale":"2024-02-21T23:11:39.924421+0000","last_undegraded":"2024-02-21T23:11:39.924421+0000","last_fullsized":"2024-02-21T23:11:39.924421+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_clean_scrub_stamp":"2024-02-21T23:11:36.873263+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:51:45.308481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.d","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.927204+0000","last_change":"2024-02-21T23:11:38.954257+0000","last_active":"2024-02-21T23:11:39.927204+0000","last_peered":"2024-02-21T23:11:39.927204+0000","last_clean":"2024-02-21T23:11:39.927204+0000","last_became_active":"2024-02-21T23:11:38.953429+0000","last_became_peered":"2024-02-21T23:11:38.953429+0000","last_unstale":"2024-02-21T23:11:39.927204+0000","last_undegraded":"2024-02-21T23:11:39.927204+0000","last_fullsized":"2024-02-21T23:11:39.927204+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:31:19.660504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.e","version":"185'2","reported_seq":16,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.434141+0000","last_change":"2024-02-21T23:11:37.925861+0000","last_active":"2024-02-21T23:11:40.434141+0000","last_peered":"2024-02-21T23:11:40.434141+0000","last_clean":"2024-02-21T23:11:40.434141+0000","last_became_active":"2024-02-21T23:11:37.925521+0000","last_became_peered":"2024-02-21T23:11:37.925521+0000","last_unstale":"2024-02-21T23:11:40.434141+0000","last_undegraded":"2024-02-21T23:11:40.434141+0000","last_fullsized":"2024-02-21T23:11:40.434141+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_clean_scrub_stamp":"2024-02-21T23:11:36.873263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:12:51.713733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.e","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.924741+0000","last_change":"2024-02-21T23:11:38.956255+0000","last_active":"2024-02-21T23:11:39.924741+0000","last_peered":"2024-02-21T23:11:39.924741+0000","last_clean":"2024-02-21T23:11:39.924741+0000","last_became_active":"2024-02-21T23:11:38.955978+0000","last_became_peered":"2024-02-21T23:11:38.955978+0000","last_unstale":"2024-02-21T23:11:39.924741+0000","last_undegraded":"2024-02-21T23:11:39.924741+0000","last_fullsized":"2024-02-21T23:11:39.924741+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:41:46.543922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.d","version":"185'1","reported_seq":15,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.436887+0000","last_change":"2024-02-21T23:11:37.938652+0000","last_active":"2024-02-21T23:11:40.436887+0000","last_peered":"2024-02-21T23:11:40.436887+0000","last_clean":"2024-02-21T23:11:40.436887+0000","last_became_active":"2024-02-21T23:11:37.938415+0000","last_became_peered":"2024-02-21T23:11:37.938415+0000","last_unstale":"2024-02-21T23:11:40.436887+0000","last_undegraded":"2024-02-21T23:11:40.436887+0000","last_fullsized":"2024-02-21T23:11:40.436887+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_clean_scrub_stamp":"2024-02-21T23:11:36.873263+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:03:16.129597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.f","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.437589+0000","last_change":"2024-02-21T23:11:38.953284+0000","last_active":"2024-02-21T23:11:40.437589+0000","last_peered":"2024-02-21T23:11:40.437589+0000","last_clean":"2024-02-21T23:11:40.437589+0000","last_became_active":"2024-02-21T23:11:38.952202+0000","last_became_peered":"2024-02-21T23:11:38.952202+0000","last_unstale":"2024-02-21T23:11:40.437589+0000","last_undegraded":"2024-02-21T23:11:40.437589+0000","last_fullsized":"2024-02-21T23:11:40.437589+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:18:12.636817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":14,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.436809+0000","last_change":"2024-02-21T23:11:37.933601+0000","last_active":"2024-02-21T23:11:40.436809+0000","last_peered":"2024-02-21T23:11:40.436809+0000","last_clean":"2024-02-21T23:11:40.436809+0000","last_became_active":"2024-02-21T23:11:37.933213+0000","last_became_peered":"2024-02-21T23:11:37.933213+0000","last_unstale":"2024-02-21T23:11:40.436809+0000","last_undegraded":"2024-02-21T23:11:40.436809+0000","last_fullsized":"2024-02-21T23:11:40.436809+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_clean_scrub_stamp":"2024-02-21T23:11:36.873263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:07:51.029760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.435860+0000","last_change":"2024-02-21T23:11:38.961819+0000","last_active":"2024-02-21T23:11:40.435860+0000","last_peered":"2024-02-21T23:11:40.435860+0000","last_clean":"2024-02-21T23:11:40.435860+0000","last_became_active":"2024-02-21T23:11:38.959941+0000","last_became_peered":"2024-02-21T23:11:38.959941+0000","last_unstale":"2024-02-21T23:11:40.435860+0000","last_undegraded":"2024-02-21T23:11:40.435860+0000","last_fullsized":"2024-02-21T23:11:40.435860+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:24:36.931053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.b","version":"185'2","reported_seq":16,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.436529+0000","last_change":"2024-02-21T23:11:37.942046+0000","last_active":"2024-02-21T23:11:40.436529+0000","last_peered":"2024-02-21T23:11:40.436529+0000","last_clean":"2024-02-21T23:11:40.436529+0000","last_became_active":"2024-02-21T23:11:37.941669+0000","last_became_peered":"2024-02-21T23:11:37.941669+0000","last_unstale":"2024-02-21T23:11:40.436529+0000","last_undegraded":"2024-02-21T23:11:40.436529+0000","last_fullsized":"2024-02-21T23:11:40.436529+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_clean_scrub_stamp":"2024-02-21T23:11:36.873263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:50:06.763141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.926641+0000","last_change":"2024-02-21T23:11:38.960907+0000","last_active":"2024-02-21T23:11:39.926641+0000","last_peered":"2024-02-21T23:11:39.926641+0000","last_clean":"2024-02-21T23:11:39.926641+0000","last_became_active":"2024-02-21T23:11:38.952193+0000","last_became_peered":"2024-02-21T23:11:38.952193+0000","last_unstale":"2024-02-21T23:11:39.926641+0000","last_undegraded":"2024-02-21T23:11:39.926641+0000","last_fullsized":"2024-02-21T23:11:39.926641+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:21:58.147248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":14,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.964078+0000","last_change":"2024-02-21T23:11:37.921582+0000","last_active":"2024-02-21T23:11:39.964078+0000","last_peered":"2024-02-21T23:11:39.964078+0000","last_clean":"2024-02-21T23:11:39.964078+0000","last_became_active":"2024-02-21T23:11:37.920276+0000","last_became_peered":"2024-02-21T23:11:37.920276+0000","last_unstale":"2024-02-21T23:11:39.964078+0000","last_undegraded":"2024-02-21T23:11:39.964078+0000","last_fullsized":"2024-02-21T23:11:39.964078+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_clean_scrub_stamp":"2024-02-21T23:11:36.873263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:48:16.545392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.922138+0000","last_change":"2024-02-21T23:11:38.955632+0000","last_active":"2024-02-21T23:11:39.922138+0000","last_peered":"2024-02-21T23:11:39.922138+0000","last_clean":"2024-02-21T23:11:39.922138+0000","last_became_active":"2024-02-21T23:11:38.955320+0000","last_became_peered":"2024-02-21T23:11:38.955320+0000","last_unstale":"2024-02-21T23:11:39.922138+0000","last_undegraded":"2024-02-21T23:11:39.922138+0000","last_fullsized":"2024-02-21T23:11:39.922138+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:50:43.736144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":14,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.963766+0000","last_change":"2024-02-21T23:11:37.921234+0000","last_active":"2024-02-21T23:11:39.963766+0000","last_peered":"2024-02-21T23:11:39.963766+0000","last_clean":"2024-02-21T23:11:39.963766+0000","last_became_active":"2024-02-21T23:11:37.919915+0000","last_became_peered":"2024-02-21T23:11:37.919915+0000","last_unstale":"2024-02-21T23:11:39.963766+0000","last_undegraded":"2024-02-21T23:11:39.963766+0000","last_fullsized":"2024-02-21T23:11:39.963766+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_clean_scrub_stamp":"2024-02-21T23:11:36.873263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:14:00.966802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.b","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.924523+0000","last_change":"2024-02-21T23:11:38.966253+0000","last_active":"2024-02-21T23:11:39.924523+0000","last_peered":"2024-02-21T23:11:39.924523+0000","last_clean":"2024-02-21T23:11:39.924523+0000","last_became_active":"2024-02-21T23:11:38.965709+0000","last_became_peered":"2024-02-21T23:11:38.965709+0000","last_unstale":"2024-02-21T23:11:39.924523+0000","last_undegraded":"2024-02-21T23:11:39.924523+0000","last_fullsized":"2024-02-21T23:11:39.924523+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:01:38.108705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":14,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.433930+0000","last_change":"2024-02-21T23:11:37.927184+0000","last_active":"2024-02-21T23:11:40.433930+0000","last_peered":"2024-02-21T23:11:40.433930+0000","last_clean":"2024-02-21T23:11:40.433930+0000","last_became_active":"2024-02-21T23:11:37.925905+0000","last_became_peered":"2024-02-21T23:11:37.925905+0000","last_unstale":"2024-02-21T23:11:40.433930+0000","last_undegraded":"2024-02-21T23:11:40.433930+0000","last_fullsized":"2024-02-21T23:11:40.433930+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_clean_scrub_stamp":"2024-02-21T23:11:36.873263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:52:05.219720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.922680+0000","last_change":"2024-02-21T23:11:38.954536+0000","last_active":"2024-02-21T23:11:39.922680+0000","last_peered":"2024-02-21T23:11:39.922680+0000","last_clean":"2024-02-21T23:11:39.922680+0000","last_became_active":"2024-02-21T23:11:38.954244+0000","last_became_peered":"2024-02-21T23:11:38.954244+0000","last_unstale":"2024-02-21T23:11:39.922680+0000","last_undegraded":"2024-02-21T23:11:39.922680+0000","last_fullsized":"2024-02-21T23:11:39.922680+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:56:33.085954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.7","version":"185'2","reported_seq":16,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.437674+0000","last_change":"2024-02-21T23:11:37.938802+0000","last_active":"2024-02-21T23:11:40.437674+0000","last_peered":"2024-02-21T23:11:40.437674+0000","last_clean":"2024-02-21T23:11:40.437674+0000","last_became_active":"2024-02-21T23:11:37.938174+0000","last_became_peered":"2024-02-21T23:11:37.938174+0000","last_unstale":"2024-02-21T23:11:40.437674+0000","last_undegraded":"2024-02-21T23:11:40.437674+0000","last_fullsized":"2024-02-21T23:11:40.437674+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_clean_scrub_stamp":"2024-02-21T23:11:36.873263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:57:53.183858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.927457+0000","last_change":"2024-02-21T23:11:38.954022+0000","last_active":"2024-02-21T23:11:39.927457+0000","last_peered":"2024-02-21T23:11:39.927457+0000","last_clean":"2024-02-21T23:11:39.927457+0000","last_became_active":"2024-02-21T23:11:38.952209+0000","last_became_peered":"2024-02-21T23:11:38.952209+0000","last_unstale":"2024-02-21T23:11:39.927457+0000","last_undegraded":"2024-02-21T23:11:39.927457+0000","last_fullsized":"2024-02-21T23:11:39.927457+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:17:16.149146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.6","version":"186'1","reported_seq":15,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.966877+0000","last_change":"2024-02-21T23:11:37.928985+0000","last_active":"2024-02-21T23:11:39.966877+0000","last_peered":"2024-02-21T23:11:39.966877+0000","last_clean":"2024-02-21T23:11:39.966877+0000","last_became_active":"2024-02-21T23:11:37.928557+0000","last_became_peered":"2024-02-21T23:11:37.928557+0000","last_unstale":"2024-02-21T23:11:39.966877+0000","last_undegraded":"2024-02-21T23:11:39.966877+0000","last_fullsized":"2024-02-21T23:11:39.966877+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_clean_scrub_stamp":"2024-02-21T23:11:36.873263+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:24:31.476916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.6","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.434639+0000","last_change":"2024-02-21T23:11:38.969553+0000","last_active":"2024-02-21T23:11:40.434639+0000","last_peered":"2024-02-21T23:11:40.434639+0000","last_clean":"2024-02-21T23:11:40.434639+0000","last_became_active":"2024-02-21T23:11:38.967824+0000","last_became_peered":"2024-02-21T23:11:38.967824+0000","last_unstale":"2024-02-21T23:11:40.434639+0000","last_undegraded":"2024-02-21T23:11:40.434639+0000","last_fullsized":"2024-02-21T23:11:40.434639+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:02:02.202127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":14,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.964476+0000","last_change":"2024-02-21T23:11:37.921548+0000","last_active":"2024-02-21T23:11:39.964476+0000","last_peered":"2024-02-21T23:11:39.964476+0000","last_clean":"2024-02-21T23:11:39.964476+0000","last_became_active":"2024-02-21T23:11:37.919431+0000","last_became_peered":"2024-02-21T23:11:37.919431+0000","last_unstale":"2024-02-21T23:11:39.964476+0000","last_undegraded":"2024-02-21T23:11:39.964476+0000","last_fullsized":"2024-02-21T23:11:39.964476+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_clean_scrub_stamp":"2024-02-21T23:11:36.873263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:00:07.358482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.7","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.434373+0000","last_change":"2024-02-21T23:11:38.948702+0000","last_active":"2024-02-21T23:11:40.434373+0000","last_peered":"2024-02-21T23:11:40.434373+0000","last_clean":"2024-02-21T23:11:40.434373+0000","last_became_active":"2024-02-21T23:11:38.945165+0000","last_became_peered":"2024-02-21T23:11:38.945165+0000","last_unstale":"2024-02-21T23:11:40.434373+0000","last_undegraded":"2024-02-21T23:11:40.434373+0000","last_fullsized":"2024-02-21T23:11:40.434373+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:52:46.192792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":14,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.437103+0000","last_change":"2024-02-21T23:11:37.938305+0000","last_active":"2024-02-21T23:11:40.437103+0000","last_peered":"2024-02-21T23:11:40.437103+0000","last_clean":"2024-02-21T23:11:40.437103+0000","last_became_active":"2024-02-21T23:11:37.938026+0000","last_became_peered":"2024-02-21T23:11:37.938026+0000","last_unstale":"2024-02-21T23:11:40.437103+0000","last_undegraded":"2024-02-21T23:11:40.437103+0000","last_fullsized":"2024-02-21T23:11:40.437103+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_clean_scrub_stamp":"2024-02-21T23:11:36.873263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:02:21.345281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.924020+0000","last_change":"2024-02-21T23:11:38.964961+0000","last_active":"2024-02-21T23:11:39.924020+0000","last_peered":"2024-02-21T23:11:39.924020+0000","last_clean":"2024-02-21T23:11:39.924020+0000","last_became_active":"2024-02-21T23:11:38.964642+0000","last_became_peered":"2024-02-21T23:11:38.964642+0000","last_unstale":"2024-02-21T23:11:39.924020+0000","last_undegraded":"2024-02-21T23:11:39.924020+0000","last_fullsized":"2024-02-21T23:11:39.924020+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:48:25.208558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":14,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.436724+0000","last_change":"2024-02-21T23:11:37.936246+0000","last_active":"2024-02-21T23:11:40.436724+0000","last_peered":"2024-02-21T23:11:40.436724+0000","last_clean":"2024-02-21T23:11:40.436724+0000","last_became_active":"2024-02-21T23:11:37.935943+0000","last_became_peered":"2024-02-21T23:11:37.935943+0000","last_unstale":"2024-02-21T23:11:40.436724+0000","last_undegraded":"2024-02-21T23:11:40.436724+0000","last_fullsized":"2024-02-21T23:11:40.436724+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_clean_scrub_stamp":"2024-02-21T23:11:36.873263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:16:03.200743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.924215+0000","last_change":"2024-02-21T23:11:38.954196+0000","last_active":"2024-02-21T23:11:39.924215+0000","last_peered":"2024-02-21T23:11:39.924215+0000","last_clean":"2024-02-21T23:11:39.924215+0000","last_became_active":"2024-02-21T23:11:38.953389+0000","last_became_peered":"2024-02-21T23:11:38.953389+0000","last_unstale":"2024-02-21T23:11:39.924215+0000","last_undegraded":"2024-02-21T23:11:39.924215+0000","last_fullsized":"2024-02-21T23:11:39.924215+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:19:00.167755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":14,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.968921+0000","last_change":"2024-02-21T23:11:37.925809+0000","last_active":"2024-02-21T23:11:39.968921+0000","last_peered":"2024-02-21T23:11:39.968921+0000","last_clean":"2024-02-21T23:11:39.968921+0000","last_became_active":"2024-02-21T23:11:37.921615+0000","last_became_peered":"2024-02-21T23:11:37.921615+0000","last_unstale":"2024-02-21T23:11:39.968921+0000","last_undegraded":"2024-02-21T23:11:39.968921+0000","last_fullsized":"2024-02-21T23:11:39.968921+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_clean_scrub_stamp":"2024-02-21T23:11:36.873263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:57:53.183858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.433763+0000","last_change":"2024-02-21T23:11:38.935912+0000","last_active":"2024-02-21T23:11:40.433763+0000","last_peered":"2024-02-21T23:11:40.433763+0000","last_clean":"2024-02-21T23:11:40.433763+0000","last_became_active":"2024-02-21T23:11:38.935153+0000","last_became_peered":"2024-02-21T23:11:38.935153+0000","last_unstale":"2024-02-21T23:11:40.433763+0000","last_undegraded":"2024-02-21T23:11:40.433763+0000","last_fullsized":"2024-02-21T23:11:40.433763+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:05:23.504584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":14,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.926219+0000","last_change":"2024-02-21T23:11:37.935839+0000","last_active":"2024-02-21T23:11:39.926219+0000","last_peered":"2024-02-21T23:11:39.926219+0000","last_clean":"2024-02-21T23:11:39.926219+0000","last_became_active":"2024-02-21T23:11:37.935520+0000","last_became_peered":"2024-02-21T23:11:37.935520+0000","last_unstale":"2024-02-21T23:11:39.926219+0000","last_undegraded":"2024-02-21T23:11:39.926219+0000","last_fullsized":"2024-02-21T23:11:39.926219+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_clean_scrub_stamp":"2024-02-21T23:11:36.873263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:36:57.556959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.436999+0000","last_change":"2024-02-21T23:11:38.950883+0000","last_active":"2024-02-21T23:11:40.436999+0000","last_peered":"2024-02-21T23:11:40.436999+0000","last_clean":"2024-02-21T23:11:40.436999+0000","last_became_active":"2024-02-21T23:11:38.950586+0000","last_became_peered":"2024-02-21T23:11:38.950586+0000","last_unstale":"2024-02-21T23:11:40.436999+0000","last_undegraded":"2024-02-21T23:11:40.436999+0000","last_fullsized":"2024-02-21T23:11:40.436999+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:24:32.492894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":14,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.436072+0000","last_change":"2024-02-21T23:11:37.939402+0000","last_active":"2024-02-21T23:11:40.436072+0000","last_peered":"2024-02-21T23:11:40.436072+0000","last_clean":"2024-02-21T23:11:40.436072+0000","last_became_active":"2024-02-21T23:11:37.938655+0000","last_became_peered":"2024-02-21T23:11:37.938655+0000","last_unstale":"2024-02-21T23:11:40.436072+0000","last_undegraded":"2024-02-21T23:11:40.436072+0000","last_fullsized":"2024-02-21T23:11:40.436072+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_clean_scrub_stamp":"2024-02-21T23:11:36.873263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:15:53.000046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1f","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.924322+0000","last_change":"2024-02-21T23:11:38.966924+0000","last_active":"2024-02-21T23:11:39.924322+0000","last_peered":"2024-02-21T23:11:39.924322+0000","last_clean":"2024-02-21T23:11:39.924322+0000","last_became_active":"2024-02-21T23:11:38.966487+0000","last_became_peered":"2024-02-21T23:11:38.966487+0000","last_unstale":"2024-02-21T23:11:39.924322+0000","last_undegraded":"2024-02-21T23:11:39.924322+0000","last_fullsized":"2024-02-21T23:11:39.924322+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:48:06.592788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1c","version":"185'2","reported_seq":16,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.926804+0000","last_change":"2024-02-21T23:11:37.931776+0000","last_active":"2024-02-21T23:11:39.926804+0000","last_peered":"2024-02-21T23:11:39.926804+0000","last_clean":"2024-02-21T23:11:39.926804+0000","last_became_active":"2024-02-21T23:11:37.931475+0000","last_became_peered":"2024-02-21T23:11:37.931475+0000","last_unstale":"2024-02-21T23:11:39.926804+0000","last_undegraded":"2024-02-21T23:11:39.926804+0000","last_fullsized":"2024-02-21T23:11:39.926804+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_clean_scrub_stamp":"2024-02-21T23:11:36.873263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:49:01.515272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":481,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.925899+0000","last_change":"2024-02-21T23:02:32.289335+0000","last_active":"2024-02-21T23:11:39.925899+0000","last_peered":"2024-02-21T23:11:39.925899+0000","last_clean":"2024-02-21T23:11:39.925899+0000","last_became_active":"2024-02-21T23:02:32.286913+0000","last_became_peered":"2024-02-21T23:02:32.286913+0000","last_unstale":"2024-02-21T23:11:39.925899+0000","last_undegraded":"2024-02-21T23:11:39.925899+0000","last_fullsized":"2024-02-21T23:11:39.925899+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_clean_scrub_stamp":"2024-02-21T23:02:31.172569+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:36:23.987374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":30,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":29,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574960,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33560,"kb_used_data":2372,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338280,"statfs":{"total":96636764160,"available":96602398720,"internally_reserved":0,"allocated":2428928,"data_stored":1750161,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574961,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33504,"kb_used_data":2324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338336,"statfs":{"total":96636764160,"available":96602456064,"internally_reserved":0,"allocated":2379776,"data_stored":1160859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574960,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33496,"kb_used_data":2316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338344,"statfs":{"total":96636764160,"available":96602464256,"internally_reserved":0,"allocated":2371584,"data_stored":1160803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574961,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33488,"kb_used_data":2304,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338352,"statfs":{"total":96636764160,"available":96602472448,"internally_reserved":0,"allocated":2359296,"data_stored":1159211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574961,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33488,"kb_used_data":2304,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338352,"statfs":{"total":96636764160,"available":96602472448,"internally_reserved":0,"allocated":2359296,"data_stored":1159143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574962,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33492,"kb_used_data":2308,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338348,"statfs":{"total":96636764160,"available":96602468352,"internally_reserved":0,"allocated":2363392,"data_stored":1159737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574960,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33468,"kb_used_data":2288,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338372,"statfs":{"total":96636764160,"available":96602492928,"internally_reserved":0,"allocated":2342912,"data_stored":1152429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574960,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37712,"kb_used_data":2364,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334128,"statfs":{"total":96636764160,"available":96598147072,"internally_reserved":0,"allocated":2420736,"data_stored":1749489,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-21T23:11:43.441 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-21T23:11:43.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.715+0000 7facbf200640 1 Processor -- start 2024-02-21T23:11:43.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.715+0000 7facbf200640 1 -- start start 2024-02-21T23:11:43.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.716+0000 7facbf200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7facc00a6160 0x7facc0155180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:43.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.717+0000 7facbf200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7facc01521c0 0x7facc0152610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:43.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.717+0000 7facbf200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facc0155760 con 0x7facc014f0b0 2024-02-21T23:11:43.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.717+0000 7facbf200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facc0152b50 con 0x7facc00a6160 2024-02-21T23:11:43.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.717+0000 7facbf200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facc0151d30 con 0x7facc01521c0 2024-02-21T23:11:43.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.717+0000 7facbd400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7facc00a6160 0x7facc0155180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:43.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.717+0000 7facbd400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7facc00a6160 0x7facc0155180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45894/0 (socket says 172.21.15.28:45894) 2024-02-21T23:11:43.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.717+0000 7facbd400640 1 -- 172.21.15.28:0/454359468 learned_addr learned my addr 172.21.15.28:0/454359468 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:43.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.717+0000 7facbe800640 1 --2- 172.21.15.28:0/454359468 >> v2:172.21.15.81:3300/0 conn(0x7facc01521c0 0x7facc0152610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:43.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.718+0000 7facbd400640 1 -- 172.21.15.28:0/454359468 >> v2:172.21.15.81:3300/0 conn(0x7facc01521c0 msgr2=0x7facc0152610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:43.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.718+0000 7facbd400640 1 --2- 172.21.15.28:0/454359468 >> v2:172.21.15.81:3300/0 conn(0x7facc01521c0 0x7facc0152610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:43.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.718+0000 7facbd400640 1 -- 172.21.15.28:0/454359468 >> v1:172.21.15.81:6789/0 conn(0x7facc014f0b0 legacy=0x7facc00c2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:43.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.718+0000 7facbd400640 1 -- 172.21.15.28:0/454359468 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facc0155a40 con 0x7facc00a6160 2024-02-21T23:11:43.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.719+0000 7facbd400640 1 --2- 172.21.15.28:0/454359468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facc00a6160 0x7facc0155180 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x7facc0151eb0 tx=0x7faca802e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cdc7b8c2af7088f server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:43.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.719+0000 7facbca00640 1 -- 172.21.15.28:0/454359468 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faca803b070 con 0x7facc00a6160 2024-02-21T23:11:43.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.719+0000 7facbca00640 1 -- 172.21.15.28:0/454359468 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faca8036440 con 0x7facc00a6160 2024-02-21T23:11:43.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.719+0000 7facbca00640 1 -- 172.21.15.28:0/454359468 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faca8036810 con 0x7facc00a6160 2024-02-21T23:11:43.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.720+0000 7facbf200640 1 -- 172.21.15.28:0/454359468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facc00a6160 msgr2=0x7facc0155180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:43.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.720+0000 7facbf200640 1 --2- 172.21.15.28:0/454359468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facc00a6160 0x7facc0155180 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x7facc0151eb0 tx=0x7faca802e0e0 comp rx=0 tx=0).stop 2024-02-21T23:11:43.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.720+0000 7facbf200640 1 -- 172.21.15.28:0/454359468 shutdown_connections 2024-02-21T23:11:43.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.720+0000 7facbf200640 1 --2- 172.21.15.28:0/454359468 >> v2:172.21.15.81:3300/0 conn(0x7facc01521c0 0x7facc0152610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:43.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.720+0000 7facbf200640 1 --2- 172.21.15.28:0/454359468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facc00a6160 0x7facc0155180 unknown :-1 s=CLOSED pgs=2076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:43.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.720+0000 7facbf200640 1 -- 172.21.15.28:0/454359468 >> 172.21.15.28:0/454359468 conn(0x7facc006b630 msgr2=0x7facc00c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:43.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.720+0000 7facbf200640 1 -- 172.21.15.28:0/454359468 shutdown_connections 2024-02-21T23:11:43.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.720+0000 7facbf200640 1 -- 172.21.15.28:0/454359468 wait complete. 2024-02-21T23:11:43.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.722+0000 7facbf200640 1 Processor -- start 2024-02-21T23:11:43.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.722+0000 7facbf200640 1 -- start start 2024-02-21T23:11:43.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.723+0000 7facbf200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7facc00a6160 0x7facc01e70e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:43.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.723+0000 7facbf200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7facc014f0b0 0x7facc00abbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:43.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.723+0000 7facbf200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facc01521c0 0x7facc01ea5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:43.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.723+0000 7facbf200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facc00a68d0 con 0x7facc01521c0 2024-02-21T23:11:43.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.723+0000 7facbf200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facc0156000 con 0x7facc00a6160 2024-02-21T23:11:43.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.723+0000 7facbf200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facc00c2720 con 0x7facc014f0b0 2024-02-21T23:11:43.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.723+0000 7facbe800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facc01521c0 0x7facc01ea5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:43.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.723+0000 7facbe800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facc01521c0 0x7facc01ea5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45908/0 (socket says 172.21.15.28:45908) 2024-02-21T23:11:43.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.723+0000 7facbe800640 1 -- 172.21.15.28:0/1453571462 learned_addr learned my addr 172.21.15.28:0/1453571462 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:43.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.724+0000 7facbde00640 1 --2- 172.21.15.28:0/1453571462 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7facc00a6160 0x7facc01e70e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:43.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.724+0000 7facbd400640 1 --2- 172.21.15.28:0/1453571462 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7facc014f0b0 0x7facc00abbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:43.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.724+0000 7facbe800640 1 -- 172.21.15.28:0/1453571462 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7facc014f0b0 msgr2=0x7facc00abbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:43.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.724+0000 7facbe800640 1 --2- 172.21.15.28:0/1453571462 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7facc014f0b0 0x7facc00abbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:43.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.724+0000 7facbe800640 1 -- 172.21.15.28:0/1453571462 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7facc00a6160 msgr2=0x7facc01e70e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:43.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.724+0000 7facbe800640 1 --2- 172.21.15.28:0/1453571462 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7facc00a6160 0x7facc01e70e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:43.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.724+0000 7facbe800640 1 -- 172.21.15.28:0/1453571462 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facc01e6620 con 0x7facc01521c0 2024-02-21T23:11:43.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.724+0000 7facbe800640 1 --2- 172.21.15.28:0/1453571462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facc01521c0 0x7facc01ea5a0 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 crypto rx=0x7facb8007e60 tx=0x7facb800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:43.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.724+0000 7facb6000640 1 -- 172.21.15.28:0/1453571462 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facb8014420 con 0x7facc01521c0 2024-02-21T23:11:43.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.724+0000 7facb6000640 1 -- 172.21.15.28:0/1453571462 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facb8019070 con 0x7facc01521c0 2024-02-21T23:11:43.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.724+0000 7facbf200640 1 -- 172.21.15.28:0/1453571462 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facc01547e0 con 0x7facc01521c0 2024-02-21T23:11:43.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.724+0000 7facb6000640 1 -- 172.21.15.28:0/1453571462 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facb8014820 con 0x7facc01521c0 2024-02-21T23:11:43.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.724+0000 7facbf200640 1 -- 172.21.15.28:0/1453571462 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facc01e68e0 con 0x7facc01521c0 2024-02-21T23:11:43.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.726+0000 7facbf200640 1 -- 172.21.15.28:0/1453571462 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facc004f290 con 0x7facc01521c0 2024-02-21T23:11:43.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.727+0000 7facb6000640 1 -- 172.21.15.28:0/1453571462 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7facb80149c0 con 0x7facc01521c0 2024-02-21T23:11:43.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.728+0000 7facb6000640 1 --2- 172.21.15.28:0/1453571462 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fac9c06b4d0 0x7fac9c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:43.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.728+0000 7facb6000640 1 -- 172.21.15.28:0/1453571462 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6101+0+0 (secure 0 0 0) 0x7facb808ac00 con 0x7facc01521c0 2024-02-21T23:11:43.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.728+0000 7facbde00640 1 --2- 172.21.15.28:0/1453571462 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fac9c06b4d0 0x7fac9c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:43.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.730+0000 7facbde00640 1 --2- 172.21.15.28:0/1453571462 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fac9c06b4d0 0x7fac9c06d990 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7facb0004ea0 tx=0x7facb0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:43.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.732+0000 7facb6000640 1 -- 172.21.15.28:0/1453571462 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7facb80598e0 con 0x7facc01521c0 2024-02-21T23:11:43.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.929+0000 7facbf200640 1 -- 172.21.15.28:0/1453571462 --> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7facc00ac840 con 0x7fac9c06b4d0 2024-02-21T23:11:43.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.947+0000 7facb6000640 1 -- 172.21.15.28:0/1453571462 <== mgr.4101 v2:172.21.15.81:6832/1848538577 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252564 (secure 0 0 0) 0x7facc00ac840 con 0x7fac9c06b4d0 2024-02-21T23:11:43.938 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:11:43.940 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-21T23:11:43.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.952+0000 7facbf200640 1 -- 172.21.15.28:0/1453571462 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fac9c06b4d0 msgr2=0x7fac9c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:43.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.952+0000 7facbf200640 1 --2- 172.21.15.28:0/1453571462 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fac9c06b4d0 0x7fac9c06d990 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7facb0004ea0 tx=0x7facb0009040 comp rx=0 tx=0).stop 2024-02-21T23:11:43.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.952+0000 7facbf200640 1 -- 172.21.15.28:0/1453571462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facc01521c0 msgr2=0x7facc01ea5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:43.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.952+0000 7facbf200640 1 --2- 172.21.15.28:0/1453571462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facc01521c0 0x7facc01ea5a0 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 crypto rx=0x7facb8007e60 tx=0x7facb800e4b0 comp rx=0 tx=0).stop 2024-02-21T23:11:43.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.953+0000 7facbf200640 1 -- 172.21.15.28:0/1453571462 shutdown_connections 2024-02-21T23:11:43.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.953+0000 7facbf200640 1 --2- 172.21.15.28:0/1453571462 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fac9c06b4d0 0x7fac9c06d990 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:43.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.953+0000 7facbf200640 1 --2- 172.21.15.28:0/1453571462 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7facc014f0b0 0x7facc00abbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:43.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.953+0000 7facbf200640 1 --2- 172.21.15.28:0/1453571462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facc01521c0 0x7facc01ea5a0 unknown :-1 s=CLOSED pgs=2077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:43.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.953+0000 7facbf200640 1 --2- 172.21.15.28:0/1453571462 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7facc00a6160 0x7facc01e70e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:43.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.953+0000 7facbf200640 1 -- 172.21.15.28:0/1453571462 >> 172.21.15.28:0/1453571462 conn(0x7facc006b630 msgr2=0x7facc0071e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:43.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.953+0000 7facbf200640 1 -- 172.21.15.28:0/1453571462 shutdown_connections 2024-02-21T23:11:43.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:43.953+0000 7facbf200640 1 -- 172.21.15.28:0/1453571462 wait complete. 2024-02-21T23:11:43.961 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":455,"stamp":"2024-02-21T23:11:42.859619+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":138,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":272208,"kb_used_data":18580,"kb_used_omap":12,"kb_used_meta":253491,"kb_avail":754702512,"statfs":{"total":773094113280,"available":772815372288,"internally_reserved":0,"allocated":19025920,"data_stored":10451832,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":259575345},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.968240"},"pg_stats":[{"pgid":"30.3f","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.927639+0000","last_change":"2024-02-21T23:11:38.964544+0000","last_active":"2024-02-21T23:11:39.927639+0000","last_peered":"2024-02-21T23:11:39.927639+0000","last_clean":"2024-02-21T23:11:39.927639+0000","last_became_active":"2024-02-21T23:11:38.961190+0000","last_became_peered":"2024-02-21T23:11:38.961190+0000","last_unstale":"2024-02-21T23:11:39.927639+0000","last_undegraded":"2024-02-21T23:11:39.927639+0000","last_fullsized":"2024-02-21T23:11:39.927639+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:02:02.202127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3e","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.922442+0000","last_change":"2024-02-21T23:11:38.952680+0000","last_active":"2024-02-21T23:11:39.922442+0000","last_peered":"2024-02-21T23:11:39.922442+0000","last_clean":"2024-02-21T23:11:39.922442+0000","last_became_active":"2024-02-21T23:11:38.952372+0000","last_became_peered":"2024-02-21T23:11:38.952372+0000","last_unstale":"2024-02-21T23:11:39.922442+0000","last_undegraded":"2024-02-21T23:11:39.922442+0000","last_fullsized":"2024-02-21T23:11:39.922442+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:04:45.366291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.3d","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.924036+0000","last_change":"2024-02-21T23:11:38.957179+0000","last_active":"2024-02-21T23:11:39.924036+0000","last_peered":"2024-02-21T23:11:39.924036+0000","last_clean":"2024-02-21T23:11:39.924036+0000","last_became_active":"2024-02-21T23:11:38.956906+0000","last_became_peered":"2024-02-21T23:11:38.956906+0000","last_unstale":"2024-02-21T23:11:39.924036+0000","last_undegraded":"2024-02-21T23:11:39.924036+0000","last_fullsized":"2024-02-21T23:11:39.924036+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:13:21.907850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.3c","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.968963+0000","last_change":"2024-02-21T23:11:38.960140+0000","last_active":"2024-02-21T23:11:39.968963+0000","last_peered":"2024-02-21T23:11:39.968963+0000","last_clean":"2024-02-21T23:11:39.968963+0000","last_became_active":"2024-02-21T23:11:38.959847+0000","last_became_peered":"2024-02-21T23:11:38.959847+0000","last_unstale":"2024-02-21T23:11:39.968963+0000","last_undegraded":"2024-02-21T23:11:39.968963+0000","last_fullsized":"2024-02-21T23:11:39.968963+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:09:49.832064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.3b","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.926736+0000","last_change":"2024-02-21T23:11:38.956959+0000","last_active":"2024-02-21T23:11:39.926736+0000","last_peered":"2024-02-21T23:11:39.926736+0000","last_clean":"2024-02-21T23:11:39.926736+0000","last_became_active":"2024-02-21T23:11:38.956652+0000","last_became_peered":"2024-02-21T23:11:38.956652+0000","last_unstale":"2024-02-21T23:11:39.926736+0000","last_undegraded":"2024-02-21T23:11:39.926736+0000","last_fullsized":"2024-02-21T23:11:39.926736+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:26:31.677128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3a","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.434308+0000","last_change":"2024-02-21T23:11:38.966303+0000","last_active":"2024-02-21T23:11:40.434308+0000","last_peered":"2024-02-21T23:11:40.434308+0000","last_clean":"2024-02-21T23:11:40.434308+0000","last_became_active":"2024-02-21T23:11:38.965917+0000","last_became_peered":"2024-02-21T23:11:38.965917+0000","last_unstale":"2024-02-21T23:11:40.434308+0000","last_undegraded":"2024-02-21T23:11:40.434308+0000","last_fullsized":"2024-02-21T23:11:40.434308+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:03:50.259304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.39","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.924121+0000","last_change":"2024-02-21T23:11:38.953945+0000","last_active":"2024-02-21T23:11:39.924121+0000","last_peered":"2024-02-21T23:11:39.924121+0000","last_clean":"2024-02-21T23:11:39.924121+0000","last_became_active":"2024-02-21T23:11:38.952962+0000","last_became_peered":"2024-02-21T23:11:38.952962+0000","last_unstale":"2024-02-21T23:11:39.924121+0000","last_undegraded":"2024-02-21T23:11:39.924121+0000","last_fullsized":"2024-02-21T23:11:39.924121+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:08:06.979800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.38","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.434182+0000","last_change":"2024-02-21T23:11:38.961837+0000","last_active":"2024-02-21T23:11:40.434182+0000","last_peered":"2024-02-21T23:11:40.434182+0000","last_clean":"2024-02-21T23:11:40.434182+0000","last_became_active":"2024-02-21T23:11:38.961484+0000","last_became_peered":"2024-02-21T23:11:38.961484+0000","last_unstale":"2024-02-21T23:11:40.434182+0000","last_undegraded":"2024-02-21T23:11:40.434182+0000","last_fullsized":"2024-02-21T23:11:40.434182+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:26:31.677128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.37","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.434522+0000","last_change":"2024-02-21T23:11:38.957666+0000","last_active":"2024-02-21T23:11:40.434522+0000","last_peered":"2024-02-21T23:11:40.434522+0000","last_clean":"2024-02-21T23:11:40.434522+0000","last_became_active":"2024-02-21T23:11:38.957355+0000","last_became_peered":"2024-02-21T23:11:38.957355+0000","last_unstale":"2024-02-21T23:11:40.434522+0000","last_undegraded":"2024-02-21T23:11:40.434522+0000","last_fullsized":"2024-02-21T23:11:40.434522+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:57:04.820656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.36","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.924435+0000","last_change":"2024-02-21T23:11:38.964360+0000","last_active":"2024-02-21T23:11:39.924435+0000","last_peered":"2024-02-21T23:11:39.924435+0000","last_clean":"2024-02-21T23:11:39.924435+0000","last_became_active":"2024-02-21T23:11:38.961529+0000","last_became_peered":"2024-02-21T23:11:38.961529+0000","last_unstale":"2024-02-21T23:11:39.924435+0000","last_undegraded":"2024-02-21T23:11:39.924435+0000","last_fullsized":"2024-02-21T23:11:39.924435+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:29:32.351053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.35","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.969343+0000","last_change":"2024-02-21T23:11:38.945662+0000","last_active":"2024-02-21T23:11:39.969343+0000","last_peered":"2024-02-21T23:11:39.969343+0000","last_clean":"2024-02-21T23:11:39.969343+0000","last_became_active":"2024-02-21T23:11:38.945165+0000","last_became_peered":"2024-02-21T23:11:38.945165+0000","last_unstale":"2024-02-21T23:11:39.969343+0000","last_undegraded":"2024-02-21T23:11:39.969343+0000","last_fullsized":"2024-02-21T23:11:39.969343+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:06:56.052476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.34","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.436410+0000","last_change":"2024-02-21T23:11:38.961228+0000","last_active":"2024-02-21T23:11:40.436410+0000","last_peered":"2024-02-21T23:11:40.436410+0000","last_clean":"2024-02-21T23:11:40.436410+0000","last_became_active":"2024-02-21T23:11:38.960910+0000","last_became_peered":"2024-02-21T23:11:38.960910+0000","last_unstale":"2024-02-21T23:11:40.436410+0000","last_undegraded":"2024-02-21T23:11:40.436410+0000","last_fullsized":"2024-02-21T23:11:40.436410+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:59:13.487532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.33","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.434543+0000","last_change":"2024-02-21T23:11:38.963302+0000","last_active":"2024-02-21T23:11:40.434543+0000","last_peered":"2024-02-21T23:11:40.434543+0000","last_clean":"2024-02-21T23:11:40.434543+0000","last_became_active":"2024-02-21T23:11:38.963026+0000","last_became_peered":"2024-02-21T23:11:38.963026+0000","last_unstale":"2024-02-21T23:11:40.434543+0000","last_undegraded":"2024-02-21T23:11:40.434543+0000","last_fullsized":"2024-02-21T23:11:40.434543+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:31:19.660504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.32","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.435800+0000","last_change":"2024-02-21T23:11:38.957576+0000","last_active":"2024-02-21T23:11:40.435800+0000","last_peered":"2024-02-21T23:11:40.435800+0000","last_clean":"2024-02-21T23:11:40.435800+0000","last_became_active":"2024-02-21T23:11:38.957225+0000","last_became_peered":"2024-02-21T23:11:38.957225+0000","last_unstale":"2024-02-21T23:11:40.435800+0000","last_undegraded":"2024-02-21T23:11:40.435800+0000","last_fullsized":"2024-02-21T23:11:40.435800+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:21:04.858085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.31","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.964102+0000","last_change":"2024-02-21T23:11:38.951853+0000","last_active":"2024-02-21T23:11:39.964102+0000","last_peered":"2024-02-21T23:11:39.964102+0000","last_clean":"2024-02-21T23:11:39.964102+0000","last_became_active":"2024-02-21T23:11:38.951290+0000","last_became_peered":"2024-02-21T23:11:38.951290+0000","last_unstale":"2024-02-21T23:11:39.964102+0000","last_undegraded":"2024-02-21T23:11:39.964102+0000","last_fullsized":"2024-02-21T23:11:39.964102+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:31:19.660504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.30","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.434595+0000","last_change":"2024-02-21T23:11:38.956435+0000","last_active":"2024-02-21T23:11:40.434595+0000","last_peered":"2024-02-21T23:11:40.434595+0000","last_clean":"2024-02-21T23:11:40.434595+0000","last_became_active":"2024-02-21T23:11:38.956104+0000","last_became_peered":"2024-02-21T23:11:38.956104+0000","last_unstale":"2024-02-21T23:11:40.434595+0000","last_undegraded":"2024-02-21T23:11:40.434595+0000","last_fullsized":"2024-02-21T23:11:40.434595+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:17:16.149146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.2f","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.437462+0000","last_change":"2024-02-21T23:11:38.962805+0000","last_active":"2024-02-21T23:11:40.437462+0000","last_peered":"2024-02-21T23:11:40.437462+0000","last_clean":"2024-02-21T23:11:40.437462+0000","last_became_active":"2024-02-21T23:11:38.962491+0000","last_became_peered":"2024-02-21T23:11:38.962491+0000","last_unstale":"2024-02-21T23:11:40.437462+0000","last_undegraded":"2024-02-21T23:11:40.437462+0000","last_fullsized":"2024-02-21T23:11:40.437462+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:00:08.374460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.2e","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.924542+0000","last_change":"2024-02-21T23:11:38.959763+0000","last_active":"2024-02-21T23:11:39.924542+0000","last_peered":"2024-02-21T23:11:39.924542+0000","last_clean":"2024-02-21T23:11:39.924542+0000","last_became_active":"2024-02-21T23:11:38.959444+0000","last_became_peered":"2024-02-21T23:11:38.959444+0000","last_unstale":"2024-02-21T23:11:39.924542+0000","last_undegraded":"2024-02-21T23:11:39.924542+0000","last_fullsized":"2024-02-21T23:11:39.924542+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:55:37.022532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.2d","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.926769+0000","last_change":"2024-02-21T23:11:38.954192+0000","last_active":"2024-02-21T23:11:39.926769+0000","last_peered":"2024-02-21T23:11:39.926769+0000","last_clean":"2024-02-21T23:11:39.926769+0000","last_became_active":"2024-02-21T23:11:38.951922+0000","last_became_peered":"2024-02-21T23:11:38.951922+0000","last_unstale":"2024-02-21T23:11:39.926769+0000","last_undegraded":"2024-02-21T23:11:39.926769+0000","last_fullsized":"2024-02-21T23:11:39.926769+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:03:50.259304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.2c","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.434812+0000","last_change":"2024-02-21T23:11:38.958868+0000","last_active":"2024-02-21T23:11:40.434812+0000","last_peered":"2024-02-21T23:11:40.434812+0000","last_clean":"2024-02-21T23:11:40.434812+0000","last_became_active":"2024-02-21T23:11:38.958581+0000","last_became_peered":"2024-02-21T23:11:38.958581+0000","last_unstale":"2024-02-21T23:11:40.434812+0000","last_undegraded":"2024-02-21T23:11:40.434812+0000","last_fullsized":"2024-02-21T23:11:40.434812+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:12:34.312401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.2b","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.924084+0000","last_change":"2024-02-21T23:11:38.959096+0000","last_active":"2024-02-21T23:11:39.924084+0000","last_peered":"2024-02-21T23:11:39.924084+0000","last_clean":"2024-02-21T23:11:39.924084+0000","last_became_active":"2024-02-21T23:11:38.958785+0000","last_became_peered":"2024-02-21T23:11:38.958785+0000","last_unstale":"2024-02-21T23:11:39.924084+0000","last_undegraded":"2024-02-21T23:11:39.924084+0000","last_fullsized":"2024-02-21T23:11:39.924084+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:57:30.581817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.2a","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.963952+0000","last_change":"2024-02-21T23:11:38.965338+0000","last_active":"2024-02-21T23:11:39.963952+0000","last_peered":"2024-02-21T23:11:39.963952+0000","last_clean":"2024-02-21T23:11:39.963952+0000","last_became_active":"2024-02-21T23:11:38.964929+0000","last_became_peered":"2024-02-21T23:11:38.964929+0000","last_unstale":"2024-02-21T23:11:39.963952+0000","last_undegraded":"2024-02-21T23:11:39.963952+0000","last_fullsized":"2024-02-21T23:11:39.963952+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:03:50.259304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.29","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.434256+0000","last_change":"2024-02-21T23:11:38.948385+0000","last_active":"2024-02-21T23:11:40.434256+0000","last_peered":"2024-02-21T23:11:40.434256+0000","last_clean":"2024-02-21T23:11:40.434256+0000","last_became_active":"2024-02-21T23:11:38.948029+0000","last_became_peered":"2024-02-21T23:11:38.948029+0000","last_unstale":"2024-02-21T23:11:40.434256+0000","last_undegraded":"2024-02-21T23:11:40.434256+0000","last_fullsized":"2024-02-21T23:11:40.434256+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:00:31.514375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.28","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.926039+0000","last_change":"2024-02-21T23:11:38.949777+0000","last_active":"2024-02-21T23:11:39.926039+0000","last_peered":"2024-02-21T23:11:39.926039+0000","last_clean":"2024-02-21T23:11:39.926039+0000","last_became_active":"2024-02-21T23:11:38.949390+0000","last_became_peered":"2024-02-21T23:11:38.949390+0000","last_unstale":"2024-02-21T23:11:39.926039+0000","last_undegraded":"2024-02-21T23:11:39.926039+0000","last_fullsized":"2024-02-21T23:11:39.926039+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:59:23.931824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.27","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.927718+0000","last_change":"2024-02-21T23:11:38.954781+0000","last_active":"2024-02-21T23:11:39.927718+0000","last_peered":"2024-02-21T23:11:39.927718+0000","last_clean":"2024-02-21T23:11:39.927718+0000","last_became_active":"2024-02-21T23:11:38.954507+0000","last_became_peered":"2024-02-21T23:11:38.954507+0000","last_unstale":"2024-02-21T23:11:39.927718+0000","last_undegraded":"2024-02-21T23:11:39.927718+0000","last_fullsized":"2024-02-21T23:11:39.927718+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:51:37.502229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.26","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.924611+0000","last_change":"2024-02-21T23:11:38.966080+0000","last_active":"2024-02-21T23:11:39.924611+0000","last_peered":"2024-02-21T23:11:39.924611+0000","last_clean":"2024-02-21T23:11:39.924611+0000","last_became_active":"2024-02-21T23:11:38.965723+0000","last_became_peered":"2024-02-21T23:11:38.965723+0000","last_unstale":"2024-02-21T23:11:39.924611+0000","last_undegraded":"2024-02-21T23:11:39.924611+0000","last_fullsized":"2024-02-21T23:11:39.924611+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:27:35.936913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.25","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.969309+0000","last_change":"2024-02-21T23:11:38.948536+0000","last_active":"2024-02-21T23:11:39.969309+0000","last_peered":"2024-02-21T23:11:39.969309+0000","last_clean":"2024-02-21T23:11:39.969309+0000","last_became_active":"2024-02-21T23:11:38.947950+0000","last_became_peered":"2024-02-21T23:11:38.947950+0000","last_unstale":"2024-02-21T23:11:39.969309+0000","last_undegraded":"2024-02-21T23:11:39.969309+0000","last_fullsized":"2024-02-21T23:11:39.969309+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:31:55.730966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.24","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.436784+0000","last_change":"2024-02-21T23:11:38.963502+0000","last_active":"2024-02-21T23:11:40.436784+0000","last_peered":"2024-02-21T23:11:40.436784+0000","last_clean":"2024-02-21T23:11:40.436784+0000","last_became_active":"2024-02-21T23:11:38.963163+0000","last_became_peered":"2024-02-21T23:11:38.963163+0000","last_unstale":"2024-02-21T23:11:40.436784+0000","last_undegraded":"2024-02-21T23:11:40.436784+0000","last_fullsized":"2024-02-21T23:11:40.436784+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:48:57.464584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.23","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.922220+0000","last_change":"2024-02-21T23:11:38.954372+0000","last_active":"2024-02-21T23:11:39.922220+0000","last_peered":"2024-02-21T23:11:39.922220+0000","last_clean":"2024-02-21T23:11:39.922220+0000","last_became_active":"2024-02-21T23:11:38.954003+0000","last_became_peered":"2024-02-21T23:11:38.954003+0000","last_unstale":"2024-02-21T23:11:39.922220+0000","last_undegraded":"2024-02-21T23:11:39.922220+0000","last_fullsized":"2024-02-21T23:11:39.922220+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:19:08.443946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.22","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.433889+0000","last_change":"2024-02-21T23:11:38.968611+0000","last_active":"2024-02-21T23:11:40.433889+0000","last_peered":"2024-02-21T23:11:40.433889+0000","last_clean":"2024-02-21T23:11:40.433889+0000","last_became_active":"2024-02-21T23:11:38.967853+0000","last_became_peered":"2024-02-21T23:11:38.967853+0000","last_unstale":"2024-02-21T23:11:40.433889+0000","last_undegraded":"2024-02-21T23:11:40.433889+0000","last_fullsized":"2024-02-21T23:11:40.433889+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:21:58.147248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.21","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.434048+0000","last_change":"2024-02-21T23:11:38.946771+0000","last_active":"2024-02-21T23:11:40.434048+0000","last_peered":"2024-02-21T23:11:40.434048+0000","last_clean":"2024-02-21T23:11:40.434048+0000","last_became_active":"2024-02-21T23:11:38.943409+0000","last_became_peered":"2024-02-21T23:11:38.943409+0000","last_unstale":"2024-02-21T23:11:40.434048+0000","last_undegraded":"2024-02-21T23:11:40.434048+0000","last_fullsized":"2024-02-21T23:11:40.434048+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:16:11.580478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.20","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.434678+0000","last_change":"2024-02-21T23:11:38.969678+0000","last_active":"2024-02-21T23:11:40.434678+0000","last_peered":"2024-02-21T23:11:40.434678+0000","last_clean":"2024-02-21T23:11:40.434678+0000","last_became_active":"2024-02-21T23:11:38.969387+0000","last_became_peered":"2024-02-21T23:11:38.969387+0000","last_unstale":"2024-02-21T23:11:40.434678+0000","last_undegraded":"2024-02-21T23:11:40.434678+0000","last_fullsized":"2024-02-21T23:11:40.434678+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:51:37.502229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1c","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.924328+0000","last_change":"2024-02-21T23:11:38.966212+0000","last_active":"2024-02-21T23:11:39.924328+0000","last_peered":"2024-02-21T23:11:39.924328+0000","last_clean":"2024-02-21T23:11:39.924328+0000","last_became_active":"2024-02-21T23:11:38.962483+0000","last_became_peered":"2024-02-21T23:11:38.962483+0000","last_unstale":"2024-02-21T23:11:39.924328+0000","last_undegraded":"2024-02-21T23:11:39.924328+0000","last_fullsized":"2024-02-21T23:11:39.924328+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:19:56.590895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1f","version":"185'4","reported_seq":18,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.434557+0000","last_change":"2024-02-21T23:11:37.930503+0000","last_active":"2024-02-21T23:11:40.434557+0000","last_peered":"2024-02-21T23:11:40.434557+0000","last_clean":"2024-02-21T23:11:40.434557+0000","last_became_active":"2024-02-21T23:11:37.929937+0000","last_became_peered":"2024-02-21T23:11:37.929937+0000","last_unstale":"2024-02-21T23:11:40.434557+0000","last_undegraded":"2024-02-21T23:11:40.434557+0000","last_fullsized":"2024-02-21T23:11:40.434557+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_clean_scrub_stamp":"2024-02-21T23:11:36.873263+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:50:08.578636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.1d","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.924391+0000","last_change":"2024-02-21T23:11:38.950075+0000","last_active":"2024-02-21T23:11:39.924391+0000","last_peered":"2024-02-21T23:11:39.924391+0000","last_clean":"2024-02-21T23:11:39.924391+0000","last_became_active":"2024-02-21T23:11:38.949082+0000","last_became_peered":"2024-02-21T23:11:38.949082+0000","last_unstale":"2024-02-21T23:11:39.924391+0000","last_undegraded":"2024-02-21T23:11:39.924391+0000","last_fullsized":"2024-02-21T23:11:39.924391+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:25:24.775484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1e","version":"185'1","reported_seq":15,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.437864+0000","last_change":"2024-02-21T23:11:37.941756+0000","last_active":"2024-02-21T23:11:40.437864+0000","last_peered":"2024-02-21T23:11:40.437864+0000","last_clean":"2024-02-21T23:11:40.437864+0000","last_became_active":"2024-02-21T23:11:37.941250+0000","last_became_peered":"2024-02-21T23:11:37.941250+0000","last_unstale":"2024-02-21T23:11:40.437864+0000","last_undegraded":"2024-02-21T23:11:40.437864+0000","last_fullsized":"2024-02-21T23:11:40.437864+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_clean_scrub_stamp":"2024-02-21T23:11:36.873263+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:20:58.386539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1e","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.437255+0000","last_change":"2024-02-21T23:11:38.961071+0000","last_active":"2024-02-21T23:11:40.437255+0000","last_peered":"2024-02-21T23:11:40.437255+0000","last_clean":"2024-02-21T23:11:40.437255+0000","last_became_active":"2024-02-21T23:11:38.960702+0000","last_became_peered":"2024-02-21T23:11:38.960702+0000","last_unstale":"2024-02-21T23:11:40.437255+0000","last_undegraded":"2024-02-21T23:11:40.437255+0000","last_fullsized":"2024-02-21T23:11:40.437255+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:48:17.561370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1d","version":"185'2","reported_seq":16,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.433955+0000","last_change":"2024-02-21T23:11:37.939613+0000","last_active":"2024-02-21T23:11:40.433955+0000","last_peered":"2024-02-21T23:11:40.433955+0000","last_clean":"2024-02-21T23:11:40.433955+0000","last_became_active":"2024-02-21T23:11:37.939055+0000","last_became_peered":"2024-02-21T23:11:37.939055+0000","last_unstale":"2024-02-21T23:11:40.433955+0000","last_undegraded":"2024-02-21T23:11:40.433955+0000","last_fullsized":"2024-02-21T23:11:40.433955+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_clean_scrub_stamp":"2024-02-21T23:11:36.873263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:48:56.448606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.923929+0000","last_change":"2024-02-21T23:11:38.960612+0000","last_active":"2024-02-21T23:11:39.923929+0000","last_peered":"2024-02-21T23:11:39.923929+0000","last_clean":"2024-02-21T23:11:39.923929+0000","last_became_active":"2024-02-21T23:11:38.957029+0000","last_became_peered":"2024-02-21T23:11:38.957029+0000","last_unstale":"2024-02-21T23:11:39.923929+0000","last_undegraded":"2024-02-21T23:11:39.923929+0000","last_fullsized":"2024-02-21T23:11:39.923929+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:05:18.211754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1b","version":"185'1","reported_seq":15,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.964114+0000","last_change":"2024-02-21T23:11:37.938169+0000","last_active":"2024-02-21T23:11:39.964114+0000","last_peered":"2024-02-21T23:11:39.964114+0000","last_clean":"2024-02-21T23:11:39.964114+0000","last_became_active":"2024-02-21T23:11:37.937884+0000","last_became_peered":"2024-02-21T23:11:37.937884+0000","last_unstale":"2024-02-21T23:11:39.964114+0000","last_undegraded":"2024-02-21T23:11:39.964114+0000","last_fullsized":"2024-02-21T23:11:39.964114+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_clean_scrub_stamp":"2024-02-21T23:11:36.873263+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:18:11.620839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.436256+0000","last_change":"2024-02-21T23:11:38.959036+0000","last_active":"2024-02-21T23:11:40.436256+0000","last_peered":"2024-02-21T23:11:40.436256+0000","last_clean":"2024-02-21T23:11:40.436256+0000","last_became_active":"2024-02-21T23:11:38.958669+0000","last_became_peered":"2024-02-21T23:11:38.958669+0000","last_unstale":"2024-02-21T23:11:40.436256+0000","last_undegraded":"2024-02-21T23:11:40.436256+0000","last_fullsized":"2024-02-21T23:11:40.436256+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:49:02.531250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1a","version":"185'1","reported_seq":15,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.926489+0000","last_change":"2024-02-21T23:11:37.943744+0000","last_active":"2024-02-21T23:11:39.926489+0000","last_peered":"2024-02-21T23:11:39.926489+0000","last_clean":"2024-02-21T23:11:39.926489+0000","last_became_active":"2024-02-21T23:11:37.943425+0000","last_became_peered":"2024-02-21T23:11:37.943425+0000","last_unstale":"2024-02-21T23:11:39.926489+0000","last_undegraded":"2024-02-21T23:11:39.926489+0000","last_fullsized":"2024-02-21T23:11:39.926489+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_clean_scrub_stamp":"2024-02-21T23:11:36.873263+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:24:31.476916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.1a","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.963984+0000","last_change":"2024-02-21T23:11:38.965961+0000","last_active":"2024-02-21T23:11:39.963984+0000","last_peered":"2024-02-21T23:11:39.963984+0000","last_clean":"2024-02-21T23:11:39.963984+0000","last_became_active":"2024-02-21T23:11:38.964488+0000","last_became_peered":"2024-02-21T23:11:38.964488+0000","last_unstale":"2024-02-21T23:11:39.963984+0000","last_undegraded":"2024-02-21T23:11:39.963984+0000","last_fullsized":"2024-02-21T23:11:39.963984+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:26:31.677128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":14,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.968674+0000","last_change":"2024-02-21T23:11:37.940706+0000","last_active":"2024-02-21T23:11:39.968674+0000","last_peered":"2024-02-21T23:11:39.968674+0000","last_clean":"2024-02-21T23:11:39.968674+0000","last_became_active":"2024-02-21T23:11:37.940426+0000","last_became_peered":"2024-02-21T23:11:37.940426+0000","last_unstale":"2024-02-21T23:11:39.968674+0000","last_undegraded":"2024-02-21T23:11:39.968674+0000","last_fullsized":"2024-02-21T23:11:39.968674+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_clean_scrub_stamp":"2024-02-21T23:11:36.873263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:07:51.029760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1b","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.434037+0000","last_change":"2024-02-21T23:11:38.952592+0000","last_active":"2024-02-21T23:11:40.434037+0000","last_peered":"2024-02-21T23:11:40.434037+0000","last_clean":"2024-02-21T23:11:40.434037+0000","last_became_active":"2024-02-21T23:11:38.952229+0000","last_became_peered":"2024-02-21T23:11:38.952229+0000","last_unstale":"2024-02-21T23:11:40.434037+0000","last_undegraded":"2024-02-21T23:11:40.434037+0000","last_fullsized":"2024-02-21T23:11:40.434037+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:22:34.217711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":14,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.924097+0000","last_change":"2024-02-21T23:11:37.924750+0000","last_active":"2024-02-21T23:11:39.924097+0000","last_peered":"2024-02-21T23:11:39.924097+0000","last_clean":"2024-02-21T23:11:39.924097+0000","last_became_active":"2024-02-21T23:11:37.924416+0000","last_became_peered":"2024-02-21T23:11:37.924416+0000","last_unstale":"2024-02-21T23:11:39.924097+0000","last_undegraded":"2024-02-21T23:11:39.924097+0000","last_fullsized":"2024-02-21T23:11:39.924097+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_clean_scrub_stamp":"2024-02-21T23:11:36.873263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:09:40.459144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.14","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.964270+0000","last_change":"2024-02-21T23:11:38.958101+0000","last_active":"2024-02-21T23:11:39.964270+0000","last_peered":"2024-02-21T23:11:39.964270+0000","last_clean":"2024-02-21T23:11:39.964270+0000","last_became_active":"2024-02-21T23:11:38.957576+0000","last_became_peered":"2024-02-21T23:11:38.957576+0000","last_unstale":"2024-02-21T23:11:39.964270+0000","last_undegraded":"2024-02-21T23:11:39.964270+0000","last_fullsized":"2024-02-21T23:11:39.964270+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:17:16.149146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":14,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.927900+0000","last_change":"2024-02-21T23:11:37.938249+0000","last_active":"2024-02-21T23:11:39.927900+0000","last_peered":"2024-02-21T23:11:39.927900+0000","last_clean":"2024-02-21T23:11:39.927900+0000","last_became_active":"2024-02-21T23:11:37.936634+0000","last_became_peered":"2024-02-21T23:11:37.936634+0000","last_unstale":"2024-02-21T23:11:39.927900+0000","last_undegraded":"2024-02-21T23:11:39.927900+0000","last_fullsized":"2024-02-21T23:11:39.927900+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_clean_scrub_stamp":"2024-02-21T23:11:36.873263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:59:12.471554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.15","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.964344+0000","last_change":"2024-02-21T23:11:38.952575+0000","last_active":"2024-02-21T23:11:39.964344+0000","last_peered":"2024-02-21T23:11:39.964344+0000","last_clean":"2024-02-21T23:11:39.964344+0000","last_became_active":"2024-02-21T23:11:38.952309+0000","last_became_peered":"2024-02-21T23:11:38.952309+0000","last_unstale":"2024-02-21T23:11:39.964344+0000","last_undegraded":"2024-02-21T23:11:39.964344+0000","last_fullsized":"2024-02-21T23:11:39.964344+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:02:02.202127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":14,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.969142+0000","last_change":"2024-02-21T23:11:37.943486+0000","last_active":"2024-02-21T23:11:39.969142+0000","last_peered":"2024-02-21T23:11:39.969142+0000","last_clean":"2024-02-21T23:11:39.969142+0000","last_became_active":"2024-02-21T23:11:37.943117+0000","last_became_peered":"2024-02-21T23:11:37.943117+0000","last_unstale":"2024-02-21T23:11:39.969142+0000","last_undegraded":"2024-02-21T23:11:39.969142+0000","last_fullsized":"2024-02-21T23:11:39.969142+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_clean_scrub_stamp":"2024-02-21T23:11:36.873263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:15:16.910929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.924833+0000","last_change":"2024-02-21T23:11:38.951990+0000","last_active":"2024-02-21T23:11:39.924833+0000","last_peered":"2024-02-21T23:11:39.924833+0000","last_clean":"2024-02-21T23:11:39.924833+0000","last_became_active":"2024-02-21T23:11:38.949433+0000","last_became_peered":"2024-02-21T23:11:38.949433+0000","last_unstale":"2024-02-21T23:11:39.924833+0000","last_undegraded":"2024-02-21T23:11:39.924833+0000","last_fullsized":"2024-02-21T23:11:39.924833+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:34:42.869435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.15","version":"185'3","reported_seq":17,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.927503+0000","last_change":"2024-02-21T23:11:37.936967+0000","last_active":"2024-02-21T23:11:39.927503+0000","last_peered":"2024-02-21T23:11:39.927503+0000","last_clean":"2024-02-21T23:11:39.927503+0000","last_became_active":"2024-02-21T23:11:37.936530+0000","last_became_peered":"2024-02-21T23:11:37.936530+0000","last_unstale":"2024-02-21T23:11:39.927503+0000","last_undegraded":"2024-02-21T23:11:39.927503+0000","last_fullsized":"2024-02-21T23:11:39.927503+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_clean_scrub_stamp":"2024-02-21T23:11:36.873263+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:48:56.448606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.969445+0000","last_change":"2024-02-21T23:11:38.949681+0000","last_active":"2024-02-21T23:11:39.969445+0000","last_peered":"2024-02-21T23:11:39.969445+0000","last_clean":"2024-02-21T23:11:39.969445+0000","last_became_active":"2024-02-21T23:11:38.944954+0000","last_became_peered":"2024-02-21T23:11:38.944954+0000","last_unstale":"2024-02-21T23:11:39.969445+0000","last_undegraded":"2024-02-21T23:11:39.969445+0000","last_fullsized":"2024-02-21T23:11:39.969445+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:58:43.457178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.14","version":"185'6","reported_seq":20,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.434839+0000","last_change":"2024-02-21T23:11:37.939784+0000","last_active":"2024-02-21T23:11:40.434839+0000","last_peered":"2024-02-21T23:11:40.434839+0000","last_clean":"2024-02-21T23:11:40.434839+0000","last_became_active":"2024-02-21T23:11:37.939524+0000","last_became_peered":"2024-02-21T23:11:37.939524+0000","last_unstale":"2024-02-21T23:11:40.434839+0000","last_undegraded":"2024-02-21T23:11:40.434839+0000","last_fullsized":"2024-02-21T23:11:40.434839+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_clean_scrub_stamp":"2024-02-21T23:11:36.873263+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:24:31.476916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.10","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.921928+0000","last_change":"2024-02-21T23:11:38.954494+0000","last_active":"2024-02-21T23:11:39.921928+0000","last_peered":"2024-02-21T23:11:39.921928+0000","last_clean":"2024-02-21T23:11:39.921928+0000","last_became_active":"2024-02-21T23:11:38.954120+0000","last_became_peered":"2024-02-21T23:11:38.954120+0000","last_unstale":"2024-02-21T23:11:39.921928+0000","last_undegraded":"2024-02-21T23:11:39.921928+0000","last_fullsized":"2024-02-21T23:11:39.921928+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:51:46.324459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.13","version":"185'2","reported_seq":16,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.436322+0000","last_change":"2024-02-21T23:11:37.937137+0000","last_active":"2024-02-21T23:11:40.436322+0000","last_peered":"2024-02-21T23:11:40.436322+0000","last_clean":"2024-02-21T23:11:40.436322+0000","last_became_active":"2024-02-21T23:11:37.936274+0000","last_became_peered":"2024-02-21T23:11:37.936274+0000","last_unstale":"2024-02-21T23:11:40.436322+0000","last_undegraded":"2024-02-21T23:11:40.436322+0000","last_fullsized":"2024-02-21T23:11:40.436322+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_clean_scrub_stamp":"2024-02-21T23:11:36.873263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:15:16.910929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.436031+0000","last_change":"2024-02-21T23:11:38.961331+0000","last_active":"2024-02-21T23:11:40.436031+0000","last_peered":"2024-02-21T23:11:40.436031+0000","last_clean":"2024-02-21T23:11:40.436031+0000","last_became_active":"2024-02-21T23:11:38.960863+0000","last_became_peered":"2024-02-21T23:11:38.960863+0000","last_unstale":"2024-02-21T23:11:40.436031+0000","last_undegraded":"2024-02-21T23:11:40.436031+0000","last_fullsized":"2024-02-21T23:11:40.436031+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:36:58.572937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":14,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.922279+0000","last_change":"2024-02-21T23:11:37.916816+0000","last_active":"2024-02-21T23:11:39.922279+0000","last_peered":"2024-02-21T23:11:39.922279+0000","last_clean":"2024-02-21T23:11:39.922279+0000","last_became_active":"2024-02-21T23:11:37.916468+0000","last_became_peered":"2024-02-21T23:11:37.916468+0000","last_unstale":"2024-02-21T23:11:39.922279+0000","last_undegraded":"2024-02-21T23:11:39.922279+0000","last_fullsized":"2024-02-21T23:11:39.922279+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_clean_scrub_stamp":"2024-02-21T23:11:36.873263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:51:36.486251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.924309+0000","last_change":"2024-02-21T23:11:38.954537+0000","last_active":"2024-02-21T23:11:39.924309+0000","last_peered":"2024-02-21T23:11:39.924309+0000","last_clean":"2024-02-21T23:11:39.924309+0000","last_became_active":"2024-02-21T23:11:38.952533+0000","last_became_peered":"2024-02-21T23:11:38.952533+0000","last_unstale":"2024-02-21T23:11:39.924309+0000","last_undegraded":"2024-02-21T23:11:39.924309+0000","last_fullsized":"2024-02-21T23:11:39.924309+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:47:54.894682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":14,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.433602+0000","last_change":"2024-02-21T23:11:37.926644+0000","last_active":"2024-02-21T23:11:40.433602+0000","last_peered":"2024-02-21T23:11:40.433602+0000","last_clean":"2024-02-21T23:11:40.433602+0000","last_became_active":"2024-02-21T23:11:37.926222+0000","last_became_peered":"2024-02-21T23:11:37.926222+0000","last_unstale":"2024-02-21T23:11:40.433602+0000","last_undegraded":"2024-02-21T23:11:40.433602+0000","last_fullsized":"2024-02-21T23:11:40.433602+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_clean_scrub_stamp":"2024-02-21T23:11:36.873263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:59:12.471554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.13","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.964661+0000","last_change":"2024-02-21T23:11:38.961360+0000","last_active":"2024-02-21T23:11:39.964661+0000","last_peered":"2024-02-21T23:11:39.964661+0000","last_clean":"2024-02-21T23:11:39.964661+0000","last_became_active":"2024-02-21T23:11:38.961075+0000","last_became_peered":"2024-02-21T23:11:38.961075+0000","last_unstale":"2024-02-21T23:11:39.964661+0000","last_undegraded":"2024-02-21T23:11:39.964661+0000","last_fullsized":"2024-02-21T23:11:39.964661+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:51:37.502229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.10","version":"185'8","reported_seq":22,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.435920+0000","last_change":"2024-02-21T23:11:37.934144+0000","last_active":"2024-02-21T23:11:40.435920+0000","last_peered":"2024-02-21T23:11:40.435920+0000","last_clean":"2024-02-21T23:11:40.435920+0000","last_became_active":"2024-02-21T23:11:37.933846+0000","last_became_peered":"2024-02-21T23:11:37.933846+0000","last_unstale":"2024-02-21T23:11:40.435920+0000","last_undegraded":"2024-02-21T23:11:40.435920+0000","last_fullsized":"2024-02-21T23:11:40.435920+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_clean_scrub_stamp":"2024-02-21T23:11:36.873263+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:06:31.857676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.922454+0000","last_change":"2024-02-21T23:11:38.957366+0000","last_active":"2024-02-21T23:11:39.922454+0000","last_peered":"2024-02-21T23:11:39.922454+0000","last_clean":"2024-02-21T23:11:39.922454+0000","last_became_active":"2024-02-21T23:11:38.957080+0000","last_became_peered":"2024-02-21T23:11:38.957080+0000","last_unstale":"2024-02-21T23:11:39.922454+0000","last_undegraded":"2024-02-21T23:11:39.922454+0000","last_fullsized":"2024-02-21T23:11:39.922454+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:16:04.419840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.f","version":"185'1","reported_seq":15,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.924421+0000","last_change":"2024-02-21T23:11:37.940069+0000","last_active":"2024-02-21T23:11:39.924421+0000","last_peered":"2024-02-21T23:11:39.924421+0000","last_clean":"2024-02-21T23:11:39.924421+0000","last_became_active":"2024-02-21T23:11:37.939722+0000","last_became_peered":"2024-02-21T23:11:37.939722+0000","last_unstale":"2024-02-21T23:11:39.924421+0000","last_undegraded":"2024-02-21T23:11:39.924421+0000","last_fullsized":"2024-02-21T23:11:39.924421+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_clean_scrub_stamp":"2024-02-21T23:11:36.873263+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:51:45.308481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.d","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.927204+0000","last_change":"2024-02-21T23:11:38.954257+0000","last_active":"2024-02-21T23:11:39.927204+0000","last_peered":"2024-02-21T23:11:39.927204+0000","last_clean":"2024-02-21T23:11:39.927204+0000","last_became_active":"2024-02-21T23:11:38.953429+0000","last_became_peered":"2024-02-21T23:11:38.953429+0000","last_unstale":"2024-02-21T23:11:39.927204+0000","last_undegraded":"2024-02-21T23:11:39.927204+0000","last_fullsized":"2024-02-21T23:11:39.927204+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:31:19.660504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.e","version":"185'2","reported_seq":16,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.434141+0000","last_change":"2024-02-21T23:11:37.925861+0000","last_active":"2024-02-21T23:11:40.434141+0000","last_peered":"2024-02-21T23:11:40.434141+0000","last_clean":"2024-02-21T23:11:40.434141+0000","last_became_active":"2024-02-21T23:11:37.925521+0000","last_became_peered":"2024-02-21T23:11:37.925521+0000","last_unstale":"2024-02-21T23:11:40.434141+0000","last_undegraded":"2024-02-21T23:11:40.434141+0000","last_fullsized":"2024-02-21T23:11:40.434141+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_clean_scrub_stamp":"2024-02-21T23:11:36.873263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:12:51.713733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.e","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.924741+0000","last_change":"2024-02-21T23:11:38.956255+0000","last_active":"2024-02-21T23:11:39.924741+0000","last_peered":"2024-02-21T23:11:39.924741+0000","last_clean":"2024-02-21T23:11:39.924741+0000","last_became_active":"2024-02-21T23:11:38.955978+0000","last_became_peered":"2024-02-21T23:11:38.955978+0000","last_unstale":"2024-02-21T23:11:39.924741+0000","last_undegraded":"2024-02-21T23:11:39.924741+0000","last_fullsized":"2024-02-21T23:11:39.924741+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:41:46.543922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.d","version":"185'1","reported_seq":15,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.436887+0000","last_change":"2024-02-21T23:11:37.938652+0000","last_active":"2024-02-21T23:11:40.436887+0000","last_peered":"2024-02-21T23:11:40.436887+0000","last_clean":"2024-02-21T23:11:40.436887+0000","last_became_active":"2024-02-21T23:11:37.938415+0000","last_became_peered":"2024-02-21T23:11:37.938415+0000","last_unstale":"2024-02-21T23:11:40.436887+0000","last_undegraded":"2024-02-21T23:11:40.436887+0000","last_fullsized":"2024-02-21T23:11:40.436887+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_clean_scrub_stamp":"2024-02-21T23:11:36.873263+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:03:16.129597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.f","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.437589+0000","last_change":"2024-02-21T23:11:38.953284+0000","last_active":"2024-02-21T23:11:40.437589+0000","last_peered":"2024-02-21T23:11:40.437589+0000","last_clean":"2024-02-21T23:11:40.437589+0000","last_became_active":"2024-02-21T23:11:38.952202+0000","last_became_peered":"2024-02-21T23:11:38.952202+0000","last_unstale":"2024-02-21T23:11:40.437589+0000","last_undegraded":"2024-02-21T23:11:40.437589+0000","last_fullsized":"2024-02-21T23:11:40.437589+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:18:12.636817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":14,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.436809+0000","last_change":"2024-02-21T23:11:37.933601+0000","last_active":"2024-02-21T23:11:40.436809+0000","last_peered":"2024-02-21T23:11:40.436809+0000","last_clean":"2024-02-21T23:11:40.436809+0000","last_became_active":"2024-02-21T23:11:37.933213+0000","last_became_peered":"2024-02-21T23:11:37.933213+0000","last_unstale":"2024-02-21T23:11:40.436809+0000","last_undegraded":"2024-02-21T23:11:40.436809+0000","last_fullsized":"2024-02-21T23:11:40.436809+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_clean_scrub_stamp":"2024-02-21T23:11:36.873263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:07:51.029760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.435860+0000","last_change":"2024-02-21T23:11:38.961819+0000","last_active":"2024-02-21T23:11:40.435860+0000","last_peered":"2024-02-21T23:11:40.435860+0000","last_clean":"2024-02-21T23:11:40.435860+0000","last_became_active":"2024-02-21T23:11:38.959941+0000","last_became_peered":"2024-02-21T23:11:38.959941+0000","last_unstale":"2024-02-21T23:11:40.435860+0000","last_undegraded":"2024-02-21T23:11:40.435860+0000","last_fullsized":"2024-02-21T23:11:40.435860+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:24:36.931053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.b","version":"185'2","reported_seq":16,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.436529+0000","last_change":"2024-02-21T23:11:37.942046+0000","last_active":"2024-02-21T23:11:40.436529+0000","last_peered":"2024-02-21T23:11:40.436529+0000","last_clean":"2024-02-21T23:11:40.436529+0000","last_became_active":"2024-02-21T23:11:37.941669+0000","last_became_peered":"2024-02-21T23:11:37.941669+0000","last_unstale":"2024-02-21T23:11:40.436529+0000","last_undegraded":"2024-02-21T23:11:40.436529+0000","last_fullsized":"2024-02-21T23:11:40.436529+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_clean_scrub_stamp":"2024-02-21T23:11:36.873263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:50:06.763141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.926641+0000","last_change":"2024-02-21T23:11:38.960907+0000","last_active":"2024-02-21T23:11:39.926641+0000","last_peered":"2024-02-21T23:11:39.926641+0000","last_clean":"2024-02-21T23:11:39.926641+0000","last_became_active":"2024-02-21T23:11:38.952193+0000","last_became_peered":"2024-02-21T23:11:38.952193+0000","last_unstale":"2024-02-21T23:11:39.926641+0000","last_undegraded":"2024-02-21T23:11:39.926641+0000","last_fullsized":"2024-02-21T23:11:39.926641+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:21:58.147248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":14,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.964078+0000","last_change":"2024-02-21T23:11:37.921582+0000","last_active":"2024-02-21T23:11:39.964078+0000","last_peered":"2024-02-21T23:11:39.964078+0000","last_clean":"2024-02-21T23:11:39.964078+0000","last_became_active":"2024-02-21T23:11:37.920276+0000","last_became_peered":"2024-02-21T23:11:37.920276+0000","last_unstale":"2024-02-21T23:11:39.964078+0000","last_undegraded":"2024-02-21T23:11:39.964078+0000","last_fullsized":"2024-02-21T23:11:39.964078+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_clean_scrub_stamp":"2024-02-21T23:11:36.873263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:48:16.545392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.922138+0000","last_change":"2024-02-21T23:11:38.955632+0000","last_active":"2024-02-21T23:11:39.922138+0000","last_peered":"2024-02-21T23:11:39.922138+0000","last_clean":"2024-02-21T23:11:39.922138+0000","last_became_active":"2024-02-21T23:11:38.955320+0000","last_became_peered":"2024-02-21T23:11:38.955320+0000","last_unstale":"2024-02-21T23:11:39.922138+0000","last_undegraded":"2024-02-21T23:11:39.922138+0000","last_fullsized":"2024-02-21T23:11:39.922138+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:50:43.736144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":14,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.963766+0000","last_change":"2024-02-21T23:11:37.921234+0000","last_active":"2024-02-21T23:11:39.963766+0000","last_peered":"2024-02-21T23:11:39.963766+0000","last_clean":"2024-02-21T23:11:39.963766+0000","last_became_active":"2024-02-21T23:11:37.919915+0000","last_became_peered":"2024-02-21T23:11:37.919915+0000","last_unstale":"2024-02-21T23:11:39.963766+0000","last_undegraded":"2024-02-21T23:11:39.963766+0000","last_fullsized":"2024-02-21T23:11:39.963766+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_clean_scrub_stamp":"2024-02-21T23:11:36.873263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:14:00.966802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.b","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.924523+0000","last_change":"2024-02-21T23:11:38.966253+0000","last_active":"2024-02-21T23:11:39.924523+0000","last_peered":"2024-02-21T23:11:39.924523+0000","last_clean":"2024-02-21T23:11:39.924523+0000","last_became_active":"2024-02-21T23:11:38.965709+0000","last_became_peered":"2024-02-21T23:11:38.965709+0000","last_unstale":"2024-02-21T23:11:39.924523+0000","last_undegraded":"2024-02-21T23:11:39.924523+0000","last_fullsized":"2024-02-21T23:11:39.924523+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:01:38.108705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":14,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.433930+0000","last_change":"2024-02-21T23:11:37.927184+0000","last_active":"2024-02-21T23:11:40.433930+0000","last_peered":"2024-02-21T23:11:40.433930+0000","last_clean":"2024-02-21T23:11:40.433930+0000","last_became_active":"2024-02-21T23:11:37.925905+0000","last_became_peered":"2024-02-21T23:11:37.925905+0000","last_unstale":"2024-02-21T23:11:40.433930+0000","last_undegraded":"2024-02-21T23:11:40.433930+0000","last_fullsized":"2024-02-21T23:11:40.433930+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_clean_scrub_stamp":"2024-02-21T23:11:36.873263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:52:05.219720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.922680+0000","last_change":"2024-02-21T23:11:38.954536+0000","last_active":"2024-02-21T23:11:39.922680+0000","last_peered":"2024-02-21T23:11:39.922680+0000","last_clean":"2024-02-21T23:11:39.922680+0000","last_became_active":"2024-02-21T23:11:38.954244+0000","last_became_peered":"2024-02-21T23:11:38.954244+0000","last_unstale":"2024-02-21T23:11:39.922680+0000","last_undegraded":"2024-02-21T23:11:39.922680+0000","last_fullsized":"2024-02-21T23:11:39.922680+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:56:33.085954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.7","version":"185'2","reported_seq":16,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.437674+0000","last_change":"2024-02-21T23:11:37.938802+0000","last_active":"2024-02-21T23:11:40.437674+0000","last_peered":"2024-02-21T23:11:40.437674+0000","last_clean":"2024-02-21T23:11:40.437674+0000","last_became_active":"2024-02-21T23:11:37.938174+0000","last_became_peered":"2024-02-21T23:11:37.938174+0000","last_unstale":"2024-02-21T23:11:40.437674+0000","last_undegraded":"2024-02-21T23:11:40.437674+0000","last_fullsized":"2024-02-21T23:11:40.437674+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_clean_scrub_stamp":"2024-02-21T23:11:36.873263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:57:53.183858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.927457+0000","last_change":"2024-02-21T23:11:38.954022+0000","last_active":"2024-02-21T23:11:39.927457+0000","last_peered":"2024-02-21T23:11:39.927457+0000","last_clean":"2024-02-21T23:11:39.927457+0000","last_became_active":"2024-02-21T23:11:38.952209+0000","last_became_peered":"2024-02-21T23:11:38.952209+0000","last_unstale":"2024-02-21T23:11:39.927457+0000","last_undegraded":"2024-02-21T23:11:39.927457+0000","last_fullsized":"2024-02-21T23:11:39.927457+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:17:16.149146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.6","version":"186'1","reported_seq":15,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.966877+0000","last_change":"2024-02-21T23:11:37.928985+0000","last_active":"2024-02-21T23:11:39.966877+0000","last_peered":"2024-02-21T23:11:39.966877+0000","last_clean":"2024-02-21T23:11:39.966877+0000","last_became_active":"2024-02-21T23:11:37.928557+0000","last_became_peered":"2024-02-21T23:11:37.928557+0000","last_unstale":"2024-02-21T23:11:39.966877+0000","last_undegraded":"2024-02-21T23:11:39.966877+0000","last_fullsized":"2024-02-21T23:11:39.966877+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_clean_scrub_stamp":"2024-02-21T23:11:36.873263+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:24:31.476916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.6","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.434639+0000","last_change":"2024-02-21T23:11:38.969553+0000","last_active":"2024-02-21T23:11:40.434639+0000","last_peered":"2024-02-21T23:11:40.434639+0000","last_clean":"2024-02-21T23:11:40.434639+0000","last_became_active":"2024-02-21T23:11:38.967824+0000","last_became_peered":"2024-02-21T23:11:38.967824+0000","last_unstale":"2024-02-21T23:11:40.434639+0000","last_undegraded":"2024-02-21T23:11:40.434639+0000","last_fullsized":"2024-02-21T23:11:40.434639+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:02:02.202127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":14,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.964476+0000","last_change":"2024-02-21T23:11:37.921548+0000","last_active":"2024-02-21T23:11:39.964476+0000","last_peered":"2024-02-21T23:11:39.964476+0000","last_clean":"2024-02-21T23:11:39.964476+0000","last_became_active":"2024-02-21T23:11:37.919431+0000","last_became_peered":"2024-02-21T23:11:37.919431+0000","last_unstale":"2024-02-21T23:11:39.964476+0000","last_undegraded":"2024-02-21T23:11:39.964476+0000","last_fullsized":"2024-02-21T23:11:39.964476+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_clean_scrub_stamp":"2024-02-21T23:11:36.873263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:00:07.358482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.7","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.434373+0000","last_change":"2024-02-21T23:11:38.948702+0000","last_active":"2024-02-21T23:11:40.434373+0000","last_peered":"2024-02-21T23:11:40.434373+0000","last_clean":"2024-02-21T23:11:40.434373+0000","last_became_active":"2024-02-21T23:11:38.945165+0000","last_became_peered":"2024-02-21T23:11:38.945165+0000","last_unstale":"2024-02-21T23:11:40.434373+0000","last_undegraded":"2024-02-21T23:11:40.434373+0000","last_fullsized":"2024-02-21T23:11:40.434373+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:52:46.192792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":14,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.437103+0000","last_change":"2024-02-21T23:11:37.938305+0000","last_active":"2024-02-21T23:11:40.437103+0000","last_peered":"2024-02-21T23:11:40.437103+0000","last_clean":"2024-02-21T23:11:40.437103+0000","last_became_active":"2024-02-21T23:11:37.938026+0000","last_became_peered":"2024-02-21T23:11:37.938026+0000","last_unstale":"2024-02-21T23:11:40.437103+0000","last_undegraded":"2024-02-21T23:11:40.437103+0000","last_fullsized":"2024-02-21T23:11:40.437103+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_clean_scrub_stamp":"2024-02-21T23:11:36.873263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:02:21.345281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.924020+0000","last_change":"2024-02-21T23:11:38.964961+0000","last_active":"2024-02-21T23:11:39.924020+0000","last_peered":"2024-02-21T23:11:39.924020+0000","last_clean":"2024-02-21T23:11:39.924020+0000","last_became_active":"2024-02-21T23:11:38.964642+0000","last_became_peered":"2024-02-21T23:11:38.964642+0000","last_unstale":"2024-02-21T23:11:39.924020+0000","last_undegraded":"2024-02-21T23:11:39.924020+0000","last_fullsized":"2024-02-21T23:11:39.924020+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:48:25.208558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":14,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.436724+0000","last_change":"2024-02-21T23:11:37.936246+0000","last_active":"2024-02-21T23:11:40.436724+0000","last_peered":"2024-02-21T23:11:40.436724+0000","last_clean":"2024-02-21T23:11:40.436724+0000","last_became_active":"2024-02-21T23:11:37.935943+0000","last_became_peered":"2024-02-21T23:11:37.935943+0000","last_unstale":"2024-02-21T23:11:40.436724+0000","last_undegraded":"2024-02-21T23:11:40.436724+0000","last_fullsized":"2024-02-21T23:11:40.436724+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_clean_scrub_stamp":"2024-02-21T23:11:36.873263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:16:03.200743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.924215+0000","last_change":"2024-02-21T23:11:38.954196+0000","last_active":"2024-02-21T23:11:39.924215+0000","last_peered":"2024-02-21T23:11:39.924215+0000","last_clean":"2024-02-21T23:11:39.924215+0000","last_became_active":"2024-02-21T23:11:38.953389+0000","last_became_peered":"2024-02-21T23:11:38.953389+0000","last_unstale":"2024-02-21T23:11:39.924215+0000","last_undegraded":"2024-02-21T23:11:39.924215+0000","last_fullsized":"2024-02-21T23:11:39.924215+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:19:00.167755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":14,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.968921+0000","last_change":"2024-02-21T23:11:37.925809+0000","last_active":"2024-02-21T23:11:39.968921+0000","last_peered":"2024-02-21T23:11:39.968921+0000","last_clean":"2024-02-21T23:11:39.968921+0000","last_became_active":"2024-02-21T23:11:37.921615+0000","last_became_peered":"2024-02-21T23:11:37.921615+0000","last_unstale":"2024-02-21T23:11:39.968921+0000","last_undegraded":"2024-02-21T23:11:39.968921+0000","last_fullsized":"2024-02-21T23:11:39.968921+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_clean_scrub_stamp":"2024-02-21T23:11:36.873263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:57:53.183858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.433763+0000","last_change":"2024-02-21T23:11:38.935912+0000","last_active":"2024-02-21T23:11:40.433763+0000","last_peered":"2024-02-21T23:11:40.433763+0000","last_clean":"2024-02-21T23:11:40.433763+0000","last_became_active":"2024-02-21T23:11:38.935153+0000","last_became_peered":"2024-02-21T23:11:38.935153+0000","last_unstale":"2024-02-21T23:11:40.433763+0000","last_undegraded":"2024-02-21T23:11:40.433763+0000","last_fullsized":"2024-02-21T23:11:40.433763+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:05:23.504584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":14,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.926219+0000","last_change":"2024-02-21T23:11:37.935839+0000","last_active":"2024-02-21T23:11:39.926219+0000","last_peered":"2024-02-21T23:11:39.926219+0000","last_clean":"2024-02-21T23:11:39.926219+0000","last_became_active":"2024-02-21T23:11:37.935520+0000","last_became_peered":"2024-02-21T23:11:37.935520+0000","last_unstale":"2024-02-21T23:11:39.926219+0000","last_undegraded":"2024-02-21T23:11:39.926219+0000","last_fullsized":"2024-02-21T23:11:39.926219+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_clean_scrub_stamp":"2024-02-21T23:11:36.873263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:36:57.556959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.436999+0000","last_change":"2024-02-21T23:11:38.950883+0000","last_active":"2024-02-21T23:11:40.436999+0000","last_peered":"2024-02-21T23:11:40.436999+0000","last_clean":"2024-02-21T23:11:40.436999+0000","last_became_active":"2024-02-21T23:11:38.950586+0000","last_became_peered":"2024-02-21T23:11:38.950586+0000","last_unstale":"2024-02-21T23:11:40.436999+0000","last_undegraded":"2024-02-21T23:11:40.436999+0000","last_fullsized":"2024-02-21T23:11:40.436999+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:24:32.492894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":14,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:40.436072+0000","last_change":"2024-02-21T23:11:37.939402+0000","last_active":"2024-02-21T23:11:40.436072+0000","last_peered":"2024-02-21T23:11:40.436072+0000","last_clean":"2024-02-21T23:11:40.436072+0000","last_became_active":"2024-02-21T23:11:37.938655+0000","last_became_peered":"2024-02-21T23:11:37.938655+0000","last_unstale":"2024-02-21T23:11:40.436072+0000","last_undegraded":"2024-02-21T23:11:40.436072+0000","last_fullsized":"2024-02-21T23:11:40.436072+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_clean_scrub_stamp":"2024-02-21T23:11:36.873263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:15:53.000046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.1f","version":"0'0","reported_seq":12,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.924322+0000","last_change":"2024-02-21T23:11:38.966924+0000","last_active":"2024-02-21T23:11:39.924322+0000","last_peered":"2024-02-21T23:11:39.924322+0000","last_clean":"2024-02-21T23:11:39.924322+0000","last_became_active":"2024-02-21T23:11:38.966487+0000","last_became_peered":"2024-02-21T23:11:38.966487+0000","last_unstale":"2024-02-21T23:11:39.924322+0000","last_undegraded":"2024-02-21T23:11:39.924322+0000","last_fullsized":"2024-02-21T23:11:39.924322+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:37.889241+0000","last_clean_scrub_stamp":"2024-02-21T23:11:37.889241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:48:06.592788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.1c","version":"185'2","reported_seq":16,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.926804+0000","last_change":"2024-02-21T23:11:37.931776+0000","last_active":"2024-02-21T23:11:39.926804+0000","last_peered":"2024-02-21T23:11:39.926804+0000","last_clean":"2024-02-21T23:11:39.926804+0000","last_became_active":"2024-02-21T23:11:37.931475+0000","last_became_peered":"2024-02-21T23:11:37.931475+0000","last_unstale":"2024-02-21T23:11:39.926804+0000","last_undegraded":"2024-02-21T23:11:39.926804+0000","last_fullsized":"2024-02-21T23:11:39.926804+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:11:36.873263+0000","last_clean_scrub_stamp":"2024-02-21T23:11:36.873263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:49:01.515272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":481,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-02-21T23:11:39.925899+0000","last_change":"2024-02-21T23:02:32.289335+0000","last_active":"2024-02-21T23:11:39.925899+0000","last_peered":"2024-02-21T23:11:39.925899+0000","last_clean":"2024-02-21T23:11:39.925899+0000","last_became_active":"2024-02-21T23:02:32.286913+0000","last_became_peered":"2024-02-21T23:02:32.286913+0000","last_unstale":"2024-02-21T23:11:39.925899+0000","last_undegraded":"2024-02-21T23:11:39.925899+0000","last_fullsized":"2024-02-21T23:11:39.925899+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_clean_scrub_stamp":"2024-02-21T23:02:31.172569+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:36:23.987374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":30,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":29,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574960,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33560,"kb_used_data":2372,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338280,"statfs":{"total":96636764160,"available":96602398720,"internally_reserved":0,"allocated":2428928,"data_stored":1750161,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574961,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33504,"kb_used_data":2324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338336,"statfs":{"total":96636764160,"available":96602456064,"internally_reserved":0,"allocated":2379776,"data_stored":1160859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574960,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33496,"kb_used_data":2316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338344,"statfs":{"total":96636764160,"available":96602464256,"internally_reserved":0,"allocated":2371584,"data_stored":1160803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574961,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33488,"kb_used_data":2304,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338352,"statfs":{"total":96636764160,"available":96602472448,"internally_reserved":0,"allocated":2359296,"data_stored":1159211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574961,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33488,"kb_used_data":2304,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338352,"statfs":{"total":96636764160,"available":96602472448,"internally_reserved":0,"allocated":2359296,"data_stored":1159143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574962,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33492,"kb_used_data":2308,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338348,"statfs":{"total":96636764160,"available":96602468352,"internally_reserved":0,"allocated":2363392,"data_stored":1159737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574960,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33468,"kb_used_data":2288,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338372,"statfs":{"total":96636764160,"available":96602492928,"internally_reserved":0,"allocated":2342912,"data_stored":1152429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574960,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37712,"kb_used_data":2364,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334128,"statfs":{"total":96636764160,"available":96598147072,"internally_reserved":0,"allocated":2420736,"data_stored":1749489,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-21T23:11:43.965 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-02-21T23:11:43.965 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-02-21T23:11:44.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.253+0000 7f2740400640 1 Processor -- start 2024-02-21T23:11:44.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.253+0000 7f2740400640 1 -- start start 2024-02-21T23:11:44.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.255+0000 7f2740400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f27380c26a0 0x7f27380728b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:44.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.255+0000 7f2740400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27380c2d70 con 0x7f27380a7180 2024-02-21T23:11:44.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.255+0000 7f2740400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27380c7200 con 0x7f27381524b0 2024-02-21T23:11:44.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.255+0000 7f2740400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27380c74e0 con 0x7f27380c26a0 2024-02-21T23:11:44.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.255+0000 7f273d000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f27380a7180 0x7f27380c3550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41016/0 (socket says 172.21.15.28:41016) 2024-02-21T23:11:44.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.255+0000 7f273d000640 1 -- 172.21.15.28:0/1160924180 learned_addr learned my addr 172.21.15.28:0/1160924180 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:11:44.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.256+0000 7f273e400640 1 --2- 172.21.15.28:0/1160924180 >> v2:172.21.15.81:3300/0 conn(0x7f27380c26a0 0x7f27380728b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:44.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.256+0000 7f2737e00640 1 -- 172.21.15.28:0/1160924180 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 764136133 0 0) 0x7f27380c2d70 con 0x7f27380a7180 2024-02-21T23:11:44.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.256+0000 7f2737e00640 1 -- 172.21.15.28:0/1160924180 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2720003520 con 0x7f27380a7180 2024-02-21T23:11:44.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.256+0000 7f2737e00640 1 -- 172.21.15.28:0/1160924180 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4264090455 0 0) 0x7f2720003520 con 0x7f27380a7180 2024-02-21T23:11:44.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.256+0000 7f2737e00640 1 -- 172.21.15.28:0/1160924180 >> v2:172.21.15.81:3300/0 conn(0x7f27380c26a0 msgr2=0x7f27380728b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:44.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.256+0000 7f2737e00640 1 --2- 172.21.15.28:0/1160924180 >> v2:172.21.15.81:3300/0 conn(0x7f27380c26a0 0x7f27380728b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:44.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.256+0000 7f2737e00640 1 -- 172.21.15.28:0/1160924180 >> v1:172.21.15.81:6789/0 conn(0x7f27381524b0 legacy=0x7f2738154fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:44.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.256+0000 7f2737e00640 1 -- 172.21.15.28:0/1160924180 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27380c3030 con 0x7f27380a7180 2024-02-21T23:11:44.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.256+0000 7f2737e00640 1 -- 172.21.15.28:0/1160924180 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f2724001670 con 0x7f27380a7180 2024-02-21T23:11:44.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.256+0000 7f2737e00640 1 -- 172.21.15.28:0/1160924180 >> v1:172.21.15.28:6789/0 conn(0x7f27380a7180 legacy=0x7f27380c3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:44.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.256+0000 7f2737e00640 1 --2- 172.21.15.28:0/1160924180 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f27200039b0 0x7f2720003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:44.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.256+0000 7f2737e00640 1 --2- 172.21.15.28:0/1160924180 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2720004800 0x7f2720004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:44.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.256+0000 7f2737e00640 1 --2- 172.21.15.28:0/1160924180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27200051c0 0x7f272000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:44.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.257+0000 7f2737e00640 1 -- 172.21.15.28:0/1160924180 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27380c74e0 con 0x7f27200051c0 2024-02-21T23:11:44.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.257+0000 7f2737e00640 1 -- 172.21.15.28:0/1160924180 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f272000c160 con 0x7f2720004800 2024-02-21T23:11:44.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.257+0000 7f2737e00640 1 -- 172.21.15.28:0/1160924180 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f272000c360 con 0x7f27200039b0 2024-02-21T23:11:44.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.257+0000 7f273e400640 1 --2- 172.21.15.28:0/1160924180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27200051c0 0x7f272000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:44.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.257+0000 7f273d000640 1 --2- 172.21.15.28:0/1160924180 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2720004800 0x7f2720004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:44.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.257+0000 7f273da00640 1 --2- 172.21.15.28:0/1160924180 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f27200039b0 0x7f2720003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:44.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.257+0000 7f273e400640 1 -- 172.21.15.28:0/1160924180 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f27200039b0 msgr2=0x7f2720003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:44.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.257+0000 7f273e400640 1 --2- 172.21.15.28:0/1160924180 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f27200039b0 0x7f2720003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:44.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.257+0000 7f273e400640 1 -- 172.21.15.28:0/1160924180 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2720004800 msgr2=0x7f2720004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:44.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.257+0000 7f273e400640 1 --2- 172.21.15.28:0/1160924180 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2720004800 0x7f2720004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:44.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.257+0000 7f273e400640 1 -- 172.21.15.28:0/1160924180 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2720003520 con 0x7f27200051c0 2024-02-21T23:11:44.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.258+0000 7f273e400640 1 -- 172.21.15.28:0/1160924180 reap_dead start 2024-02-21T23:11:44.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.259+0000 7f273e400640 1 --2- 172.21.15.28:0/1160924180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27200051c0 0x7f272000b9c0 secure :-1 s=READY pgs=2079 cs=0 l=1 rev1=1 crypto rx=0x7f2730009ac0 tx=0x7f2720005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a153a5206fc86c3 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:44.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.259+0000 7f2737e00640 1 -- 172.21.15.28:0/1160924180 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2730009690 con 0x7f27200051c0 2024-02-21T23:11:44.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.259+0000 7f2737e00640 1 -- 172.21.15.28:0/1160924180 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2730004020 con 0x7f27200051c0 2024-02-21T23:11:44.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.259+0000 7f2737e00640 1 -- 172.21.15.28:0/1160924180 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2730004410 con 0x7f27200051c0 2024-02-21T23:11:44.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.259+0000 7f2740400640 1 -- 172.21.15.28:0/1160924180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27200051c0 msgr2=0x7f272000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:44.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.259+0000 7f2740400640 1 --2- 172.21.15.28:0/1160924180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27200051c0 0x7f272000b9c0 secure :-1 s=READY pgs=2079 cs=0 l=1 rev1=1 crypto rx=0x7f2730009ac0 tx=0x7f2720005830 comp rx=0 tx=0).stop 2024-02-21T23:11:44.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.260+0000 7f2740400640 1 -- 172.21.15.28:0/1160924180 shutdown_connections 2024-02-21T23:11:44.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.260+0000 7f2740400640 1 --2- 172.21.15.28:0/1160924180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27200051c0 0x7f272000b9c0 unknown :-1 s=CLOSED pgs=2079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:44.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.260+0000 7f2740400640 1 -- 172.21.15.28:0/1160924180 >> 172.21.15.28:0/1160924180 conn(0x7f27380a9ab0 msgr2=0x7f27380a9ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:44.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.260+0000 7f2740400640 1 -- 172.21.15.28:0/1160924180 shutdown_connections 2024-02-21T23:11:44.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.260+0000 7f2740400640 1 -- 172.21.15.28:0/1160924180 wait complete. 2024-02-21T23:11:44.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.262+0000 7f2740400640 1 Processor -- start 2024-02-21T23:11:44.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.262+0000 7f2740400640 1 -- start start 2024-02-21T23:11:44.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.263+0000 7f2740400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f27200051c0 0x7f2738149c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:44.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.263+0000 7f2740400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27381eb980 0x7f27381510c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:44.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.263+0000 7f2740400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f27381df6e0 0x7f27381dfb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:44.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.263+0000 7f2740400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2738152a10 con 0x7f27381eb980 2024-02-21T23:11:44.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.263+0000 7f2740400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2738152890 con 0x7f27200051c0 2024-02-21T23:11:44.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.263+0000 7f2740400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2738154fd0 con 0x7f27381df6e0 2024-02-21T23:11:44.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.263+0000 7f273d000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27381eb980 0x7f27381510c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:44.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.263+0000 7f273d000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27381eb980 0x7f27381510c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45930/0 (socket says 172.21.15.28:45930) 2024-02-21T23:11:44.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.263+0000 7f273d000640 1 -- 172.21.15.28:0/2684430927 learned_addr learned my addr 172.21.15.28:0/2684430927 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:44.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.263+0000 7f273da00640 1 --2- 172.21.15.28:0/2684430927 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f27200051c0 0x7f2738149c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:44.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.263+0000 7f273e400640 1 --2- 172.21.15.28:0/2684430927 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f27381df6e0 0x7f27381dfb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:44.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.264+0000 7f273d000640 1 -- 172.21.15.28:0/2684430927 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f27381df6e0 msgr2=0x7f27381dfb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:44.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.264+0000 7f273d000640 1 --2- 172.21.15.28:0/2684430927 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f27381df6e0 0x7f27381dfb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:44.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.264+0000 7f273d000640 1 -- 172.21.15.28:0/2684430927 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f27200051c0 msgr2=0x7f2738149c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:44.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.264+0000 7f273d000640 1 --2- 172.21.15.28:0/2684430927 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f27200051c0 0x7f2738149c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:44.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.264+0000 7f273d000640 1 -- 172.21.15.28:0/2684430927 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f273814a1d0 con 0x7f27381eb980 2024-02-21T23:11:44.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.264+0000 7f273d000640 1 --2- 172.21.15.28:0/2684430927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27381eb980 0x7f27381510c0 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 crypto rx=0x7f2724000f60 tx=0x7f272400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:44.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.264+0000 7f2735600640 1 -- 172.21.15.28:0/2684430927 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f272400c870 con 0x7f27381eb980 2024-02-21T23:11:44.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.264+0000 7f2735600640 1 -- 172.21.15.28:0/2684430927 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2724018070 con 0x7f27381eb980 2024-02-21T23:11:44.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.264+0000 7f2740400640 1 -- 172.21.15.28:0/2684430927 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27381eaee0 con 0x7f27381eb980 2024-02-21T23:11:44.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.264+0000 7f2735600640 1 -- 172.21.15.28:0/2684430927 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f272400cc70 con 0x7f27381eb980 2024-02-21T23:11:44.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.265+0000 7f2740400640 1 -- 172.21.15.28:0/2684430927 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27381e1380 con 0x7f27381eb980 2024-02-21T23:11:44.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.266+0000 7f2740400640 1 -- 172.21.15.28:0/2684430927 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27381eacd0 con 0x7f27381eb980 2024-02-21T23:11:44.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.267+0000 7f2735600640 1 -- 172.21.15.28:0/2684430927 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f272401d460 con 0x7f27381eb980 2024-02-21T23:11:44.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.268+0000 7f2735600640 1 --2- 172.21.15.28:0/2684430927 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f271806b4d0 0x7f271806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:44.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.268+0000 7f2735600640 1 -- 172.21.15.28:0/2684430927 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f272408a230 con 0x7f27381eb980 2024-02-21T23:11:44.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.269+0000 7f273da00640 1 --2- 172.21.15.28:0/2684430927 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f271806b4d0 0x7f271806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:44.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.270+0000 7f273da00640 1 --2- 172.21.15.28:0/2684430927 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f271806b4d0 0x7f271806d990 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f272c007fd0 tx=0x7f272c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:44.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.273+0000 7f2735600640 1 -- 172.21.15.28:0/2684430927 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f273814a1d0 con 0x7f27381eb980 2024-02-21T23:11:44.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.547+0000 7f2740400640 1 -- 172.21.15.28:0/2684430927 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f2738068640 con 0x7f27381eb980 2024-02-21T23:11:44.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.552+0000 7f2735600640 1 -- 172.21.15.28:0/2684430927 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v21) v1 ==== 190+0+147 (secure 0 0 0) 0x7f2724058f10 con 0x7f27381eb980 2024-02-21T23:11:44.540 INFO:teuthology.orchestra.run.smithi028.stdout:[client.0] 2024-02-21T23:11:44.540 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQDTgNZlILvYLxAAoREWxKKT7YcXnhkgk8YvoQ== 2024-02-21T23:11:44.540 INFO:teuthology.orchestra.run.smithi028.stdout: caps mds = "allow" 2024-02-21T23:11:44.540 INFO:teuthology.orchestra.run.smithi028.stdout: caps mon = "allow r" 2024-02-21T23:11:44.541 INFO:teuthology.orchestra.run.smithi028.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-02-21T23:11:44.541 INFO:teuthology.orchestra.run.smithi028.stderr:updated caps for client.0 2024-02-21T23:11:44.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.555+0000 7f2740400640 1 -- 172.21.15.28:0/2684430927 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f271806b4d0 msgr2=0x7f271806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:44.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.555+0000 7f2740400640 1 --2- 172.21.15.28:0/2684430927 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f271806b4d0 0x7f271806d990 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f272c007fd0 tx=0x7f272c009040 comp rx=0 tx=0).stop 2024-02-21T23:11:44.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.555+0000 7f2740400640 1 -- 172.21.15.28:0/2684430927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27381eb980 msgr2=0x7f27381510c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:44.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.555+0000 7f2740400640 1 --2- 172.21.15.28:0/2684430927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27381eb980 0x7f27381510c0 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 crypto rx=0x7f2724000f60 tx=0x7f272400ec80 comp rx=0 tx=0).stop 2024-02-21T23:11:44.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.555+0000 7f2740400640 1 -- 172.21.15.28:0/2684430927 shutdown_connections 2024-02-21T23:11:44.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.555+0000 7f2740400640 1 --2- 172.21.15.28:0/2684430927 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f271806b4d0 0x7f271806d990 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:44.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.555+0000 7f2740400640 1 --2- 172.21.15.28:0/2684430927 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f27381df6e0 0x7f27381dfb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:44.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.555+0000 7f2740400640 1 --2- 172.21.15.28:0/2684430927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27381eb980 0x7f27381510c0 unknown :-1 s=CLOSED pgs=2080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:44.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.555+0000 7f2740400640 1 --2- 172.21.15.28:0/2684430927 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f27200051c0 0x7f2738149c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:44.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.555+0000 7f2740400640 1 -- 172.21.15.28:0/2684430927 >> 172.21.15.28:0/2684430927 conn(0x7f27380a9ab0 msgr2=0x7f2738155460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:44.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.556+0000 7f2740400640 1 -- 172.21.15.28:0/2684430927 shutdown_connections 2024-02-21T23:11:44.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.556+0000 7f2740400640 1 -- 172.21.15.28:0/2684430927 wait complete. 2024-02-21T23:11:44.566 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:11:44.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.798+0000 7efd36800640 1 Processor -- start 2024-02-21T23:11:44.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.798+0000 7efd36800640 1 -- start start 2024-02-21T23:11:44.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.800+0000 7efd36800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efd300c0f00 0x7efd30159db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:44.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.800+0000 7efd36800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7efd30071070 0x7efd30071500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:44.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.800+0000 7efd36800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd300c1390 con 0x7efd301569b0 2024-02-21T23:11:44.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.800+0000 7efd36800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd300c1fa0 con 0x7efd30071070 2024-02-21T23:11:44.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.800+0000 7efd36800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd300c1670 con 0x7efd300c0f00 2024-02-21T23:11:44.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.800+0000 7efd35400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7efd301569b0 0x7efd3006dff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41020/0 (socket says 172.21.15.28:41020) 2024-02-21T23:11:44.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.800+0000 7efd35400640 1 -- 172.21.15.28:0/2225459621 learned_addr learned my addr 172.21.15.28:0/2225459621 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:11:44.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.800+0000 7efd35e00640 1 --2- 172.21.15.28:0/2225459621 >> v2:172.21.15.81:3300/0 conn(0x7efd30071070 0x7efd30071500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:44.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.800+0000 7efd34a00640 1 --2- 172.21.15.28:0/2225459621 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efd300c0f00 0x7efd30159db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:44.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.800+0000 7efd2fe00640 1 -- 172.21.15.28:0/2225459621 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4084817597 0 0) 0x7efd300c1390 con 0x7efd301569b0 2024-02-21T23:11:44.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.801+0000 7efd2fe00640 1 -- 172.21.15.28:0/2225459621 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd18003520 con 0x7efd301569b0 2024-02-21T23:11:44.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.801+0000 7efd2fe00640 1 -- 172.21.15.28:0/2225459621 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 410474095 0 0) 0x7efd18003520 con 0x7efd301569b0 2024-02-21T23:11:44.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.801+0000 7efd2fe00640 1 -- 172.21.15.28:0/2225459621 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efd300c0f00 msgr2=0x7efd30159db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:44.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.801+0000 7efd2fe00640 1 --2- 172.21.15.28:0/2225459621 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efd300c0f00 0x7efd30159db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:44.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.801+0000 7efd2fe00640 1 -- 172.21.15.28:0/2225459621 >> v2:172.21.15.81:3300/0 conn(0x7efd30071070 msgr2=0x7efd30071500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:44.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.801+0000 7efd2fe00640 1 --2- 172.21.15.28:0/2225459621 >> v2:172.21.15.81:3300/0 conn(0x7efd30071070 0x7efd30071500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:44.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.801+0000 7efd2fe00640 1 -- 172.21.15.28:0/2225459621 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd300c1950 con 0x7efd301569b0 2024-02-21T23:11:44.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.801+0000 7efd2fe00640 1 -- 172.21.15.28:0/2225459621 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7efd24001670 con 0x7efd301569b0 2024-02-21T23:11:44.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.801+0000 7efd35e00640 1 --2- 172.21.15.28:0/2225459621 >> v2:172.21.15.81:3300/0 conn(0x7efd30071070 0x7efd30071500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-21T23:11:44.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.801+0000 7efd2fe00640 1 -- 172.21.15.28:0/2225459621 >> v1:172.21.15.28:6789/0 conn(0x7efd301569b0 legacy=0x7efd3006dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:44.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.801+0000 7efd2fe00640 1 --2- 172.21.15.28:0/2225459621 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd180039b0 0x7efd18003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:44.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.801+0000 7efd2fe00640 1 --2- 172.21.15.28:0/2225459621 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efd18004800 0x7efd18004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:44.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.801+0000 7efd2fe00640 1 --2- 172.21.15.28:0/2225459621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd18005180 0x7efd1800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:44.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.801+0000 7efd2fe00640 1 -- 172.21.15.28:0/2225459621 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd300c1fa0 con 0x7efd18005180 2024-02-21T23:11:44.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.801+0000 7efd2fe00640 1 -- 172.21.15.28:0/2225459621 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd300c1670 con 0x7efd180039b0 2024-02-21T23:11:44.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.801+0000 7efd2fe00640 1 -- 172.21.15.28:0/2225459621 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd18004630 con 0x7efd18004800 2024-02-21T23:11:44.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.802+0000 7efd35e00640 1 --2- 172.21.15.28:0/2225459621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd18005180 0x7efd1800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:44.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.802+0000 7efd35400640 1 --2- 172.21.15.28:0/2225459621 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd180039b0 0x7efd18003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:44.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.802+0000 7efd34a00640 1 --2- 172.21.15.28:0/2225459621 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efd18004800 0x7efd18004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:44.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.802+0000 7efd35e00640 1 -- 172.21.15.28:0/2225459621 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efd18004800 msgr2=0x7efd18004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:44.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.802+0000 7efd35e00640 1 --2- 172.21.15.28:0/2225459621 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efd18004800 0x7efd18004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:44.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.802+0000 7efd35e00640 1 -- 172.21.15.28:0/2225459621 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd180039b0 msgr2=0x7efd18003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:44.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.802+0000 7efd35e00640 1 --2- 172.21.15.28:0/2225459621 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd180039b0 0x7efd18003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:44.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.802+0000 7efd35e00640 1 -- 172.21.15.28:0/2225459621 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd18003520 con 0x7efd18005180 2024-02-21T23:11:44.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.803+0000 7efd35e00640 1 --2- 172.21.15.28:0/2225459621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd18005180 0x7efd1800b9c0 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 crypto rx=0x7efd2800e350 tx=0x7efd18005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb3e3cb5791240d0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:44.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.803+0000 7efd2fe00640 1 -- 172.21.15.28:0/2225459621 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd28039070 con 0x7efd18005180 2024-02-21T23:11:44.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.804+0000 7efd2fe00640 1 -- 172.21.15.28:0/2225459621 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd2802ce00 con 0x7efd18005180 2024-02-21T23:11:44.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.804+0000 7efd2fe00640 1 -- 172.21.15.28:0/2225459621 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd28009de0 con 0x7efd18005180 2024-02-21T23:11:44.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.804+0000 7efd36800640 1 -- 172.21.15.28:0/2225459621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd18005180 msgr2=0x7efd1800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:44.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.804+0000 7efd36800640 1 --2- 172.21.15.28:0/2225459621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd18005180 0x7efd1800b9c0 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 crypto rx=0x7efd2800e350 tx=0x7efd18005830 comp rx=0 tx=0).stop 2024-02-21T23:11:44.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.804+0000 7efd35e00640 1 -- 172.21.15.28:0/2225459621 reap_dead start 2024-02-21T23:11:44.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.804+0000 7efd36800640 1 -- 172.21.15.28:0/2225459621 shutdown_connections 2024-02-21T23:11:44.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.804+0000 7efd36800640 1 -- 172.21.15.28:0/2225459621 >> 172.21.15.28:0/2225459621 conn(0x7efd3006b630 msgr2=0x7efd300c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:44.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.804+0000 7efd36800640 1 -- 172.21.15.28:0/2225459621 shutdown_connections 2024-02-21T23:11:44.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.804+0000 7efd36800640 1 -- 172.21.15.28:0/2225459621 wait complete. 2024-02-21T23:11:44.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.806+0000 7efd36800640 1 Processor -- start 2024-02-21T23:11:44.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.806+0000 7efd36800640 1 -- start start 2024-02-21T23:11:44.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.807+0000 7efd36800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efd18005180 0x7efd301ef310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:44.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.807+0000 7efd36800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd3015bc80 0x7efd300a5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:44.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.807+0000 7efd36800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd300a7080 0x7efd301f1e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:44.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.807+0000 7efd36800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd3006e700 con 0x7efd3015bc80 2024-02-21T23:11:44.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.807+0000 7efd36800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd300c2ba0 con 0x7efd300a7080 2024-02-21T23:11:44.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.807+0000 7efd36800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd3006e880 con 0x7efd18005180 2024-02-21T23:11:44.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.807+0000 7efd34a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd3015bc80 0x7efd300a5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:44.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.808+0000 7efd34a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd3015bc80 0x7efd300a5110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45954/0 (socket says 172.21.15.28:45954) 2024-02-21T23:11:44.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.808+0000 7efd34a00640 1 -- 172.21.15.28:0/1339303313 learned_addr learned my addr 172.21.15.28:0/1339303313 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:44.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.808+0000 7efd35400640 1 --2- 172.21.15.28:0/1339303313 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efd18005180 0x7efd301ef310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:44.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.808+0000 7efd35e00640 1 --2- 172.21.15.28:0/1339303313 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd300a7080 0x7efd301f1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:44.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.808+0000 7efd34a00640 1 -- 172.21.15.28:0/1339303313 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efd18005180 msgr2=0x7efd301ef310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:44.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.808+0000 7efd34a00640 1 --2- 172.21.15.28:0/1339303313 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efd18005180 0x7efd301ef310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:44.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.808+0000 7efd34a00640 1 -- 172.21.15.28:0/1339303313 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd300a7080 msgr2=0x7efd301f1e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:44.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.808+0000 7efd34a00640 1 --2- 172.21.15.28:0/1339303313 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd300a7080 0x7efd301f1e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:44.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.808+0000 7efd34a00640 1 -- 172.21.15.28:0/1339303313 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd300a5650 con 0x7efd3015bc80 2024-02-21T23:11:44.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.808+0000 7efd35e00640 1 --2- 172.21.15.28:0/1339303313 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd300a7080 0x7efd301f1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:11:44.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.809+0000 7efd34a00640 1 --2- 172.21.15.28:0/1339303313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd3015bc80 0x7efd300a5110 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x7efd1c010860 tx=0x7efd1c010d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:44.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.809+0000 7efd2d600640 1 -- 172.21.15.28:0/1339303313 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd1c00f040 con 0x7efd3015bc80 2024-02-21T23:11:44.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.809+0000 7efd2d600640 1 -- 172.21.15.28:0/1339303313 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd1c00eb60 con 0x7efd3015bc80 2024-02-21T23:11:44.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.809+0000 7efd2d600640 1 -- 172.21.15.28:0/1339303313 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd1c01c400 con 0x7efd3015bc80 2024-02-21T23:11:44.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.809+0000 7efd36800640 1 -- 172.21.15.28:0/1339303313 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd300a25c0 con 0x7efd3015bc80 2024-02-21T23:11:44.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.809+0000 7efd36800640 1 -- 172.21.15.28:0/1339303313 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd300a4630 con 0x7efd3015bc80 2024-02-21T23:11:44.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.810+0000 7efd36800640 1 -- 172.21.15.28:0/1339303313 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd300c62b0 con 0x7efd3015bc80 2024-02-21T23:11:44.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.811+0000 7efd2d600640 1 -- 172.21.15.28:0/1339303313 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efd1c01c5a0 con 0x7efd3015bc80 2024-02-21T23:11:44.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.812+0000 7efd2d600640 1 --2- 172.21.15.28:0/1339303313 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7efd1006b500 0x7efd1006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:44.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.812+0000 7efd2d600640 1 -- 172.21.15.28:0/1339303313 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6101+0+0 (secure 0 0 0) 0x7efd1c08a9b0 con 0x7efd3015bc80 2024-02-21T23:11:44.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.813+0000 7efd35400640 1 --2- 172.21.15.28:0/1339303313 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7efd1006b500 0x7efd1006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:44.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.814+0000 7efd35400640 1 --2- 172.21.15.28:0/1339303313 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7efd1006b500 0x7efd1006d9c0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7efd24005b90 tx=0x7efd24007040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:44.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:44.816+0000 7efd2d600640 1 -- 172.21.15.28:0/1339303313 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7efd300c62b0 con 0x7efd3015bc80 2024-02-21T23:11:44.882 INFO:tasks.ceph.mds.b.smithi081.stderr:2024-02-21T23:11:44.894+0000 7f45e4800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-21T23:11:45.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.057+0000 7efd36800640 1 -- 172.21.15.28:0/1339303313 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efd3004f290 con 0x7efd3015bc80 2024-02-21T23:11:45.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.058+0000 7efd2d600640 1 -- 172.21.15.28:0/1339303313 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 201 v201) v1 ==== 95+0+4948 (secure 0 0 0) 0x7efd1c018030 con 0x7efd3015bc80 2024-02-21T23:11:45.046 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:11:45.046 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":201,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6848,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3439896042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3439896042},{"type":"v1","addr":"172.21.15.28:6835","nonce":3439896042}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":196},{"gid":7729,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/3213249808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3213249808},{"type":"v1","addr":"172.21.15.28:6833","nonce":3213249808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":195},{"gid":7735,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/2895690310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2895690310},{"type":"v1","addr":"172.21.15.81:6836","nonce":2895690310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":196}],"filesystems":[{"mdsmap":{"epoch":201,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:11:38.384619+0000","modified":"2024-02-21T23:11:41.472133+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7738},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7738":{"gid":7738,"name":"b","rank":0,"incarnation":197,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6837/1709165826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":1709165826},{"type":"v1","addr":"172.21.15.81:6837","nonce":1709165826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-02-21T23:11:45.046 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 201 2024-02-21T23:11:45.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.061+0000 7efd36800640 1 -- 172.21.15.28:0/1339303313 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7efd1006b500 msgr2=0x7efd1006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:45.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.061+0000 7efd36800640 1 --2- 172.21.15.28:0/1339303313 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7efd1006b500 0x7efd1006d9c0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7efd24005b90 tx=0x7efd24007040 comp rx=0 tx=0).stop 2024-02-21T23:11:45.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.061+0000 7efd36800640 1 -- 172.21.15.28:0/1339303313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd3015bc80 msgr2=0x7efd300a5110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:45.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.061+0000 7efd36800640 1 --2- 172.21.15.28:0/1339303313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd3015bc80 0x7efd300a5110 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x7efd1c010860 tx=0x7efd1c010d20 comp rx=0 tx=0).stop 2024-02-21T23:11:45.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.061+0000 7efd36800640 1 -- 172.21.15.28:0/1339303313 shutdown_connections 2024-02-21T23:11:45.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.061+0000 7efd36800640 1 --2- 172.21.15.28:0/1339303313 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7efd1006b500 0x7efd1006d9c0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:45.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.061+0000 7efd36800640 1 --2- 172.21.15.28:0/1339303313 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efd300a7080 0x7efd301f1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:45.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.061+0000 7efd36800640 1 --2- 172.21.15.28:0/1339303313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd3015bc80 0x7efd300a5110 unknown :-1 s=CLOSED pgs=2083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:45.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.061+0000 7efd36800640 1 --2- 172.21.15.28:0/1339303313 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7efd18005180 0x7efd301ef310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:45.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.061+0000 7efd36800640 1 -- 172.21.15.28:0/1339303313 >> 172.21.15.28:0/1339303313 conn(0x7efd3006b630 msgr2=0x7efd30157db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:45.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.062+0000 7efd36800640 1 -- 172.21.15.28:0/1339303313 shutdown_connections 2024-02-21T23:11:45.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.062+0000 7efd36800640 1 -- 172.21.15.28:0/1339303313 wait complete. 2024-02-21T23:11:45.472 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 201, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:11:38.384619+0000', 'modified': '2024-02-21T23:11:41.472133+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7738}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7738': {'gid': 7738, 'name': 'b', 'rank': 0, 'incarnation': 197, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.81:6837/1709165826', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6835', 'nonce': 1709165826}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 1709165826}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:11:45.472 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-21T23:11:45.472 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-02-21T23:11:45.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.715+0000 7fd2fe400640 1 Processor -- start 2024-02-21T23:11:45.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.716+0000 7fd2fe400640 1 -- start start 2024-02-21T23:11:45.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.717+0000 7fd2fe400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd2f8155a10 0x7fd2f80c70d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:45.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.717+0000 7fd2fe400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2f8155ec0 con 0x7fd2f80c5040 2024-02-21T23:11:45.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.717+0000 7fd2fe400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2f806df50 con 0x7fd2f81524c0 2024-02-21T23:11:45.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.717+0000 7fd2fe400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2f80c5520 con 0x7fd2f8155a10 2024-02-21T23:11:45.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.718+0000 7fd2fda00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd2f8155a10 0x7fd2f80c70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:45.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.718+0000 7fd2fda00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd2f8155a10 0x7fd2f80c70d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55032/0 (socket says 172.21.15.28:55032) 2024-02-21T23:11:45.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.718+0000 7fd2fda00640 1 -- 172.21.15.28:0/4062920109 learned_addr learned my addr 172.21.15.28:0/4062920109 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:45.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.718+0000 7fd2fda00640 1 -- 172.21.15.28:0/4062920109 >> v1:172.21.15.81:6789/0 conn(0x7fd2f81524c0 legacy=0x7fd2f806ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:45.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.718+0000 7fd2fda00640 1 -- 172.21.15.28:0/4062920109 >> v1:172.21.15.28:6789/0 conn(0x7fd2f80c5040 legacy=0x7fd2f80c85c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:45.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.718+0000 7fd2fda00640 1 -- 172.21.15.28:0/4062920109 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2f80c5800 con 0x7fd2f8155a10 2024-02-21T23:11:45.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.718+0000 7fd2f7400640 1 -- 172.21.15.28:0/4062920109 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1010792862 0 0) 0x7fd2f8155ec0 con 0x7fd2f80c5040 2024-02-21T23:11:45.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.719+0000 7fd2fda00640 1 --2- 172.21.15.28:0/4062920109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2f8155a10 0x7fd2f80c70d0 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 crypto rx=0x7fd2f0009e50 tx=0x7fd2f002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c7ab1f124867396 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:45.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.720+0000 7fd2f7400640 1 -- 172.21.15.28:0/4062920109 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f002ca80 con 0x7fd2f8155a10 2024-02-21T23:11:45.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.720+0000 7fd2f7400640 1 -- 172.21.15.28:0/4062920109 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2f002cc20 con 0x7fd2f8155a10 2024-02-21T23:11:45.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.720+0000 7fd2f7400640 1 -- 172.21.15.28:0/4062920109 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f00343f0 con 0x7fd2f8155a10 2024-02-21T23:11:45.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.720+0000 7fd2fe400640 1 -- 172.21.15.28:0/4062920109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2f8155a10 msgr2=0x7fd2f80c70d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:45.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.720+0000 7fd2fe400640 1 --2- 172.21.15.28:0/4062920109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2f8155a10 0x7fd2f80c70d0 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 crypto rx=0x7fd2f0009e50 tx=0x7fd2f002bee0 comp rx=0 tx=0).stop 2024-02-21T23:11:45.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.720+0000 7fd2fe400640 1 -- 172.21.15.28:0/4062920109 shutdown_connections 2024-02-21T23:11:45.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.720+0000 7fd2fe400640 1 --2- 172.21.15.28:0/4062920109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2f8155a10 0x7fd2f80c70d0 unknown :-1 s=CLOSED pgs=2085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:45.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.720+0000 7fd2fe400640 1 -- 172.21.15.28:0/4062920109 >> 172.21.15.28:0/4062920109 conn(0x7fd2f80ac2b0 msgr2=0x7fd2f80ac6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:45.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.720+0000 7fd2fe400640 1 -- 172.21.15.28:0/4062920109 shutdown_connections 2024-02-21T23:11:45.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.720+0000 7fd2fe400640 1 -- 172.21.15.28:0/4062920109 wait complete. 2024-02-21T23:11:45.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.722+0000 7fd2fe400640 1 Processor -- start 2024-02-21T23:11:45.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.722+0000 7fd2fe400640 1 -- start start 2024-02-21T23:11:45.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.723+0000 7fd2fe400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd2f80c5040 0x7fd2f80c0730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:45.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.723+0000 7fd2fe400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2f81524c0 0x7fd2f80c0c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:45.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.723+0000 7fd2fe400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2f8155a10 0x7fd2f80bf150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:45.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.723+0000 7fd2fe400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f81528a0 con 0x7fd2f8155a10 2024-02-21T23:11:45.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.723+0000 7fd2fe400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f806c950 con 0x7fd2f81524c0 2024-02-21T23:11:45.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.723+0000 7fd2fe400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f80c7610 con 0x7fd2f80c5040 2024-02-21T23:11:45.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.723+0000 7fd2fda00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2f8155a10 0x7fd2f80bf150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:45.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.724+0000 7fd2fda00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2f8155a10 0x7fd2f80bf150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55044/0 (socket says 172.21.15.28:55044) 2024-02-21T23:11:45.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.724+0000 7fd2fda00640 1 -- 172.21.15.28:0/3751242815 learned_addr learned my addr 172.21.15.28:0/3751242815 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:45.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.724+0000 7fd2fd000640 1 --2- 172.21.15.28:0/3751242815 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd2f80c5040 0x7fd2f80c0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:45.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.724+0000 7fd2f7e00640 1 --2- 172.21.15.28:0/3751242815 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2f81524c0 0x7fd2f80c0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:45.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.724+0000 7fd2fda00640 1 -- 172.21.15.28:0/3751242815 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd2f80c5040 msgr2=0x7fd2f80c0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:45.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.724+0000 7fd2fda00640 1 --2- 172.21.15.28:0/3751242815 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd2f80c5040 0x7fd2f80c0730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:45.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.724+0000 7fd2fda00640 1 -- 172.21.15.28:0/3751242815 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2f81524c0 msgr2=0x7fd2f80c0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:45.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.724+0000 7fd2fda00640 1 --2- 172.21.15.28:0/3751242815 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2f81524c0 0x7fd2f80c0c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:45.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.724+0000 7fd2fda00640 1 -- 172.21.15.28:0/3751242815 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2f80be390 con 0x7fd2f8155a10 2024-02-21T23:11:45.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.724+0000 7fd2f7e00640 1 --2- 172.21.15.28:0/3751242815 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2f81524c0 0x7fd2f80c0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:11:45.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.724+0000 7fd2fda00640 1 --2- 172.21.15.28:0/3751242815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2f8155a10 0x7fd2f80bf150 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 crypto rx=0x7fd2f0036040 tx=0x7fd2f0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:45.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.725+0000 7fd2f4c00640 1 -- 172.21.15.28:0/3751242815 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f0043070 con 0x7fd2f8155a10 2024-02-21T23:11:45.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.725+0000 7fd2f4c00640 1 -- 172.21.15.28:0/3751242815 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2f0003990 con 0x7fd2f8155a10 2024-02-21T23:11:45.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.725+0000 7fd2fe400640 1 -- 172.21.15.28:0/3751242815 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2f80be670 con 0x7fd2f8155a10 2024-02-21T23:11:45.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.725+0000 7fd2f4c00640 1 -- 172.21.15.28:0/3751242815 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f0009510 con 0x7fd2f8155a10 2024-02-21T23:11:45.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.725+0000 7fd2fe400640 1 -- 172.21.15.28:0/3751242815 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2f80bec30 con 0x7fd2f8155a10 2024-02-21T23:11:45.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.727+0000 7fd2f4c00640 1 -- 172.21.15.28:0/3751242815 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd2f002ca90 con 0x7fd2f8155a10 2024-02-21T23:11:45.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.728+0000 7fd2f4c00640 1 --2- 172.21.15.28:0/3751242815 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd2d806b4d0 0x7fd2d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:45.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.728+0000 7fd2f4c00640 1 -- 172.21.15.28:0/3751242815 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd2f00aca00 con 0x7fd2f8155a10 2024-02-21T23:11:45.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.729+0000 7fd30297b740 1 Processor -- start 2024-02-21T23:11:45.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.729+0000 7fd30297b740 1 -- start start 2024-02-21T23:11:45.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.729+0000 7fd2fd000640 1 --2- 172.21.15.28:0/3751242815 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd2d806b4d0 0x7fd2d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:45.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.730+0000 7fd30297b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c38ecdc020 0x55c38ecdc400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:45.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.730+0000 7fd30297b740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c38ecdd330 0x55c38ed18170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:45.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.730+0000 7fd30297b740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c38ed186b0 0x55c38ed16720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:45.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.730+0000 7fd30297b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c38ec40470 con 0x55c38ecdc020 2024-02-21T23:11:45.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.730+0000 7fd30297b740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c38ebe9af0 con 0x55c38ecdd330 2024-02-21T23:11:45.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.730+0000 7fd30297b740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c38ec3c510 con 0x55c38ed186b0 2024-02-21T23:11:45.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.730+0000 7fd2fd000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c38ecdc020 0x55c38ecdc400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:45.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.730+0000 7fd2fd000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c38ecdc020 0x55c38ecdc400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55058/0 (socket says 172.21.15.28:55058) 2024-02-21T23:11:45.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.730+0000 7fd2fd000640 1 -- 172.21.15.28:0/2487166700 learned_addr learned my addr 172.21.15.28:0/2487166700 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:45.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.730+0000 7fd2f7e00640 1 --2- 172.21.15.28:0/2487166700 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c38ecdd330 0x55c38ed18170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:45.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.730+0000 7fd2fda00640 1 --2- 172.21.15.28:0/2487166700 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c38ed186b0 0x55c38ed16720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:45.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.730+0000 7fd2fd000640 1 -- 172.21.15.28:0/2487166700 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c38ed186b0 msgr2=0x55c38ed16720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:45.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.730+0000 7fd2fd000640 1 --2- 172.21.15.28:0/2487166700 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c38ed186b0 0x55c38ed16720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:45.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.730+0000 7fd2fd000640 1 -- 172.21.15.28:0/2487166700 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c38ecdd330 msgr2=0x55c38ed18170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:45.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.730+0000 7fd2fd000640 1 --2- 172.21.15.28:0/2487166700 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c38ecdd330 0x55c38ed18170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:45.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.730+0000 7fd2fd000640 1 -- 172.21.15.28:0/2487166700 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c38ebf0d90 con 0x55c38ecdc020 2024-02-21T23:11:45.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.731+0000 7fd2fd000640 1 --2- 172.21.15.28:0/3751242815 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd2d806b4d0 0x7fd2d806d990 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fd2f81e82c0 tx=0x7fd2e8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:45.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.731+0000 7fd2fd000640 1 --2- 172.21.15.28:0/2487166700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c38ecdc020 0x55c38ecdc400 secure :-1 s=READY pgs=2087 cs=0 l=1 rev1=1 crypto rx=0x7fd2e800d870 tx=0x7fd2e801a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f4b7ca52c9b1a18 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:45.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.731+0000 7fd2ed600640 1 -- 172.21.15.28:0/2487166700 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e801c070 con 0x55c38ecdc020 2024-02-21T23:11:45.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.731+0000 7fd2ed600640 1 -- 172.21.15.28:0/2487166700 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2e8014ce0 con 0x55c38ecdc020 2024-02-21T23:11:45.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.731+0000 7fd2ed600640 1 -- 172.21.15.28:0/2487166700 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e8005310 con 0x55c38ecdc020 2024-02-21T23:11:45.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.732+0000 7fd30297b740 1 -- 172.21.15.28:0/2487166700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c38ecdc020 msgr2=0x55c38ecdc400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:45.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.732+0000 7fd30297b740 1 --2- 172.21.15.28:0/2487166700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c38ecdc020 0x55c38ecdc400 secure :-1 s=READY pgs=2087 cs=0 l=1 rev1=1 crypto rx=0x7fd2e800d870 tx=0x7fd2e801a000 comp rx=0 tx=0).stop 2024-02-21T23:11:45.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.732+0000 7fd30297b740 1 -- 172.21.15.28:0/2487166700 shutdown_connections 2024-02-21T23:11:45.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.732+0000 7fd30297b740 1 --2- 172.21.15.28:0/2487166700 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c38ed186b0 0x55c38ed16720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:45.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.732+0000 7fd30297b740 1 --2- 172.21.15.28:0/2487166700 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c38ecdd330 0x55c38ed18170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:45.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.732+0000 7fd30297b740 1 --2- 172.21.15.28:0/2487166700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c38ecdc020 0x55c38ecdc400 unknown :-1 s=CLOSED pgs=2087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:45.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.732+0000 7fd30297b740 1 -- 172.21.15.28:0/2487166700 >> 172.21.15.28:0/2487166700 conn(0x55c38ecd8970 msgr2=0x55c38ecfd380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:45.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.732+0000 7fd30297b740 1 -- 172.21.15.28:0/2487166700 shutdown_connections 2024-02-21T23:11:45.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.732+0000 7fd30297b740 1 -- 172.21.15.28:0/2487166700 wait complete. 2024-02-21T23:11:45.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.733+0000 7fd30297b740 1 -- start start 2024-02-21T23:11:45.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.733+0000 7fd30297b740 1 Processor -- start 2024-02-21T23:11:45.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.734+0000 7fd30297b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c38ecdd330 0x55c38ecd6b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:45.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.734+0000 7fd30297b740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c38ed186b0 0x55c38ed286c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:45.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.734+0000 7fd30297b740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c38ecf87c0 0x55c38ecf7200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:45.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.734+0000 7fd30297b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c38ebdf040 con 0x55c38ecdd330 2024-02-21T23:11:45.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.734+0000 7fd30297b740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c38ecfda90 con 0x55c38ecf87c0 2024-02-21T23:11:45.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.734+0000 7fd30297b740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c38ecfb570 con 0x55c38ed186b0 2024-02-21T23:11:45.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.734+0000 7fd30297b740 10 client.0 _ll_register_callbacks cb 0x55c38ebfc920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:11:45.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.734+0000 7fd2fd000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c38ecdd330 0x55c38ecd6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:45.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.734+0000 7fd2fd000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c38ecdd330 0x55c38ecd6b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55070/0 (socket says 172.21.15.28:55070) 2024-02-21T23:11:45.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.734+0000 7fd2fd000640 1 -- 172.21.15.28:0/2470044988 learned_addr learned my addr 172.21.15.28:0/2470044988 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:45.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.734+0000 7fd2f7e00640 1 --2- 172.21.15.28:0/2470044988 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c38ed186b0 0x55c38ed286c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:45.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.734+0000 7fd2fda00640 1 --2- 172.21.15.28:0/2470044988 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c38ecf87c0 0x55c38ecf7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:45.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.735+0000 7fd2fd000640 1 -- 172.21.15.28:0/2470044988 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c38ed186b0 msgr2=0x55c38ed286c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:45.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.735+0000 7fd2fd000640 1 --2- 172.21.15.28:0/2470044988 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c38ed186b0 0x55c38ed286c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:45.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.735+0000 7fd2fd000640 1 -- 172.21.15.28:0/2470044988 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c38ecf87c0 msgr2=0x55c38ecf7200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:45.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.735+0000 7fd2fd000640 1 --2- 172.21.15.28:0/2470044988 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c38ecf87c0 0x55c38ecf7200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:45.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.735+0000 7fd2fd000640 1 -- 172.21.15.28:0/2470044988 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c38ebfeaa0 con 0x55c38ecdd330 2024-02-21T23:11:45.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.735+0000 7fd2fd000640 1 --2- 172.21.15.28:0/2470044988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c38ecdd330 0x55c38ecd6b30 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7fd2e8005040 tx=0x7fd2e8000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95dcedf6dfa493aa server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:45.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.735+0000 7fd2e2000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:11:45.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.735+0000 7fd2e2000640 1 -- 172.21.15.28:0/2470044988 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e80013e0 con 0x55c38ecdd330 2024-02-21T23:11:45.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.735+0000 7fd2e2000640 1 -- 172.21.15.28:0/2470044988 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(186..186 src has 1..186) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd2e8014ae0 con 0x55c38ecdd330 2024-02-21T23:11:45.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.735+0000 7fd30297b740 1 -- 172.21.15.28:0/2470044988 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd2e8014ce0 con 0x55c38ecdd330 2024-02-21T23:11:45.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.736+0000 7fd2e2000640 1 -- 172.21.15.28:0/2470044988 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=201) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd2e8003800 con 0x55c38ecdd330 2024-02-21T23:11:45.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.736+0000 7fd30297b740 10 client.10056 fetch_fsmap learned FSMap version 201 2024-02-21T23:11:45.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.736+0000 7fd30297b740 1 -- 172.21.15.28:0/2470044988 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=201,monmap=2+}) v3 -- 0x55c38e9552d0 con 0x55c38ecdd330 2024-02-21T23:11:45.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.736+0000 7fd2e2000640 1 -- 172.21.15.28:0/2470044988 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2e8014ce0 con 0x55c38ecdd330 2024-02-21T23:11:45.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.736+0000 7fd2e2000640 1 -- 172.21.15.28:0/2470044988 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 201) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fd2e8013040 con 0x55c38ecdd330 2024-02-21T23:11:45.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.736+0000 7fd30297b740 10 client.10056 fetch_fsmap finished waiting for FSMap version 201 2024-02-21T23:11:45.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.736+0000 7fd30297b740 10 client.10056 resolve_mds: resolved name 'b' to daemon mds.b 2024-02-21T23:11:45.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.736+0000 7fd30297b740 20 client.10056 populate_metadata read hostname 'smithi028' 2024-02-21T23:11:45.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.736+0000 7fd30297b740 1 --2- 172.21.15.28:0/2470044988 >> [v2:172.21.15.81:6835/1709165826,v1:172.21.15.81:6837/1709165826] conn(0x55c38ed2bb70 0x55c38ed2bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:45.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.736+0000 7fd30297b740 4 client.10056 mds_command: new command op to 7738 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-21T23:11:45.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.736+0000 7fd30297b740 1 -- 172.21.15.28:0/2470044988 --> [v2:172.21.15.81:6835/1709165826,v1:172.21.15.81:6837/1709165826] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c38ec4d9b0 con 0x55c38ed2bb70 2024-02-21T23:11:45.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.737+0000 7fd2f7e00640 1 --2- 172.21.15.28:0/2470044988 >> [v2:172.21.15.81:6835/1709165826,v1:172.21.15.81:6837/1709165826] conn(0x55c38ed2bb70 0x55c38ed2bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:45.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.738+0000 7fd2f7e00640 1 --2- 172.21.15.28:0/2470044988 >> [v2:172.21.15.81:6835/1709165826,v1:172.21.15.81:6837/1709165826] conn(0x55c38ed2bb70 0x55c38ed2bf70 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a51e87f85a69ec3f server_cookie=e47bcccf659add56 in_seq=0 out_seq=0 2024-02-21T23:11:45.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.738+0000 7fd2e2000640 10 client.10056 ms_handle_connect on v2:172.21.15.81:6835/1709165826 2024-02-21T23:11:45.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.740+0000 7fd2e2000640 1 -- 172.21.15.28:0/2470044988 <== mds.0 v2:172.21.15.81:6835/1709165826 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c38ec4d9b0 con 0x55c38ed2bb70 2024-02-21T23:11:45.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.740+0000 7fd2e2000640 10 client.10056 handle_command_reply: tid=0 2024-02-21T23:11:45.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.740+0000 7fd30297b740 1 client.10056 shutdown 2024-02-21T23:11:45.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.740+0000 7fd30297b740 1 -- 172.21.15.28:0/2470044988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c38ecdd330 msgr2=0x55c38ecd6b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:45.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.740+0000 7fd30297b740 1 --2- 172.21.15.28:0/2470044988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c38ecdd330 0x55c38ecd6b30 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7fd2e8005040 tx=0x7fd2e8000ed0 comp rx=0 tx=0).stop 2024-02-21T23:11:45.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.740+0000 7fd30297b740 1 -- 172.21.15.28:0/2470044988 shutdown_connections 2024-02-21T23:11:45.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.740+0000 7fd30297b740 1 --2- 172.21.15.28:0/2470044988 >> [v2:172.21.15.81:6835/1709165826,v1:172.21.15.81:6837/1709165826] conn(0x55c38ed2bb70 0x55c38ed2bf70 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:45.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.740+0000 7fd30297b740 1 --2- 172.21.15.28:0/2470044988 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c38ed186b0 0x55c38ed286c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:45.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.740+0000 7fd30297b740 1 --2- 172.21.15.28:0/2470044988 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c38ecf87c0 0x55c38ecf7200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:45.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.740+0000 7fd30297b740 1 --2- 172.21.15.28:0/2470044988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c38ecdd330 0x55c38ecd6b30 unknown :-1 s=CLOSED pgs=2088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:45.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.740+0000 7fd2e2000640 0 client.10056 ms_handle_reset on v2:172.21.15.81:6835/1709165826 2024-02-21T23:11:45.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.740+0000 7fd30297b740 1 -- 172.21.15.28:0/2470044988 >> 172.21.15.28:0/2470044988 conn(0x55c38ecdc020 msgr2=0x55c38ed1b790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:45.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.740+0000 7fd30297b740 1 -- 172.21.15.28:0/2470044988 shutdown_connections 2024-02-21T23:11:45.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.740+0000 7fd30297b740 1 -- 172.21.15.28:0/2470044988 wait complete. 2024-02-21T23:11:45.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.741+0000 7fd30297b740 20 client.10056 trim_cache size 0 max 16384 2024-02-21T23:11:45.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.759+0000 7fd30297b740 1 Processor -- start 2024-02-21T23:11:45.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.759+0000 7fd30297b740 1 -- start start 2024-02-21T23:11:45.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.760+0000 7fd30297b740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c38ecdd330 0x55c38ed11bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:45.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.760+0000 7fd30297b740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c38ecf87c0 0x55c38ecf7c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:45.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.760+0000 7fd30297b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c38ed186b0 0x55c38ed1b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:45.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.760+0000 7fd30297b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f81f6b40 con 0x55c38ed186b0 2024-02-21T23:11:45.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.760+0000 7fd30297b740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f81f6010 con 0x55c38ecf87c0 2024-02-21T23:11:45.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.760+0000 7fd30297b740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f81f6540 con 0x55c38ecdd330 2024-02-21T23:11:45.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.761+0000 7fd2fda00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c38ed186b0 0x55c38ed1b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:45.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.761+0000 7fd2fd000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c38ecdd330 0x55c38ed11bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:45.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.761+0000 7fd2fda00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c38ed186b0 0x55c38ed1b6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55080/0 (socket says 172.21.15.28:55080) 2024-02-21T23:11:45.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.761+0000 7fd2fda00640 1 -- 172.21.15.28:0/3889445101 learned_addr learned my addr 172.21.15.28:0/3889445101 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:45.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.761+0000 7fd2fd000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c38ecdd330 0x55c38ed11bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3301/0 says I am v2:172.21.15.28:33528/0 (socket says 172.21.15.28:33528) 2024-02-21T23:11:45.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.761+0000 7fd2f7e00640 1 --2- 172.21.15.28:0/3889445101 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c38ecf87c0 0x55c38ecf7c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:45.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.761+0000 7fd2fda00640 1 -- 172.21.15.28:0/3889445101 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c38ecdd330 msgr2=0x55c38ed11bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:45.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.761+0000 7fd2fda00640 1 --2- 172.21.15.28:0/3889445101 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c38ecdd330 0x55c38ed11bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:45.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.761+0000 7fd2fda00640 1 -- 172.21.15.28:0/3889445101 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c38ecf87c0 msgr2=0x55c38ecf7c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:45.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.761+0000 7fd2fda00640 1 --2- 172.21.15.28:0/3889445101 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c38ecf87c0 0x55c38ecf7c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:45.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.761+0000 7fd2fda00640 1 -- 172.21.15.28:0/3889445101 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c38ea1f6b0 con 0x55c38ed186b0 2024-02-21T23:11:45.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.761+0000 7fd2fda00640 1 --2- 172.21.15.28:0/3889445101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c38ed186b0 0x55c38ed1b6b0 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 crypto rx=0x7fd2f007ace0 tx=0x7fd2f007b6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55c5948ae6216fc2 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:45.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.762+0000 7fd2ecc00640 1 -- 172.21.15.28:0/3889445101 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f00a8070 con 0x55c38ed186b0 2024-02-21T23:11:45.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.762+0000 7fd2ecc00640 1 -- 172.21.15.28:0/3889445101 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2f00836f0 con 0x55c38ed186b0 2024-02-21T23:11:45.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.762+0000 7fd2ecc00640 1 -- 172.21.15.28:0/3889445101 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f0077650 con 0x55c38ed186b0 2024-02-21T23:11:45.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.762+0000 7fd30297b740 1 -- 172.21.15.28:0/3889445101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c38ed186b0 msgr2=0x55c38ed1b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:45.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.762+0000 7fd30297b740 1 --2- 172.21.15.28:0/3889445101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c38ed186b0 0x55c38ed1b6b0 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 crypto rx=0x7fd2f007ace0 tx=0x7fd2f007b6e0 comp rx=0 tx=0).stop 2024-02-21T23:11:45.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.762+0000 7fd30297b740 1 -- 172.21.15.28:0/3889445101 shutdown_connections 2024-02-21T23:11:45.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.762+0000 7fd30297b740 1 --2- 172.21.15.28:0/3889445101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c38ed186b0 0x55c38ed1b6b0 unknown :-1 s=CLOSED pgs=2089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:45.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.762+0000 7fd30297b740 1 --2- 172.21.15.28:0/3889445101 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c38ecf87c0 0x55c38ecf7c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:45.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.762+0000 7fd30297b740 1 --2- 172.21.15.28:0/3889445101 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c38ecdd330 0x55c38ed11bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:45.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.762+0000 7fd30297b740 1 -- 172.21.15.28:0/3889445101 >> 172.21.15.28:0/3889445101 conn(0x55c38ecdc020 msgr2=0x55c38ed17d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:45.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.763+0000 7fd30297b740 1 -- 172.21.15.28:0/3889445101 shutdown_connections 2024-02-21T23:11:45.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.763+0000 7fd30297b740 1 -- 172.21.15.28:0/3889445101 wait complete. 2024-02-21T23:11:45.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.763+0000 7fd30297b740 1 -- start start 2024-02-21T23:11:45.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.763+0000 7fd30297b740 1 Processor -- start 2024-02-21T23:11:45.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.764+0000 7fd30297b740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c38ecdd330 0x55c38ed1a360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:45.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.764+0000 7fd30297b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c38ecf87c0 0x55c38ed1bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:45.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.764+0000 7fd30297b740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c38ed186b0 0x55c38ed87cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:45.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.765+0000 7fd30297b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c38ed28c00 con 0x55c38ecf87c0 2024-02-21T23:11:45.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.765+0000 7fd30297b740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c38ecfb2f0 con 0x55c38ed186b0 2024-02-21T23:11:45.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.765+0000 7fd30297b740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c38ecf7790 con 0x55c38ecdd330 2024-02-21T23:11:45.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.765+0000 7fd30297b740 10 client.0 _ll_register_callbacks cb 0x7fd2e80054d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:11:45.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.765+0000 7fd2f7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c38ecf87c0 0x55c38ed1bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:45.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.765+0000 7fd2f7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c38ecf87c0 0x55c38ed1bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55096/0 (socket says 172.21.15.28:55096) 2024-02-21T23:11:45.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.765+0000 7fd2f7e00640 1 -- 172.21.15.28:0/2995732278 learned_addr learned my addr 172.21.15.28:0/2995732278 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:45.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.765+0000 7fd2fd000640 1 --2- 172.21.15.28:0/2995732278 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c38ecdd330 0x55c38ed1a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:45.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.765+0000 7fd2f7e00640 1 -- 172.21.15.28:0/2995732278 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c38ecdd330 msgr2=0x55c38ed1a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:45.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.765+0000 7fd2f7e00640 1 --2- 172.21.15.28:0/2995732278 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c38ecdd330 0x55c38ed1a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:45.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.765+0000 7fd2f7e00640 1 -- 172.21.15.28:0/2995732278 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c38ed186b0 msgr2=0x55c38ed87cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:45.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.765+0000 7fd2f7e00640 1 --2- 172.21.15.28:0/2995732278 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c38ed186b0 0x55c38ed87cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:45.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.765+0000 7fd2f7e00640 1 -- 172.21.15.28:0/2995732278 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c38ecd72b0 con 0x55c38ecf87c0 2024-02-21T23:11:45.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.766+0000 7fd2f7e00640 1 --2- 172.21.15.28:0/2995732278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c38ecf87c0 0x55c38ed1bab0 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 crypto rx=0x7fd2e4000f10 tx=0x7fd2e4019be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6396c0e66e4b22e1 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:45.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.766+0000 7fd2e2000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:11:45.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.766+0000 7fd2e2000640 1 -- 172.21.15.28:0/2995732278 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e4019e30 con 0x55c38ecf87c0 2024-02-21T23:11:45.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.766+0000 7fd2e2000640 1 -- 172.21.15.28:0/2995732278 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(186..186 src has 1..186) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd2e4010040 con 0x55c38ecf87c0 2024-02-21T23:11:45.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.766+0000 7fd30297b740 1 -- 172.21.15.28:0/2995732278 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd2f00836f0 con 0x55c38ecf87c0 2024-02-21T23:11:45.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.766+0000 7fd2e2000640 1 -- 172.21.15.28:0/2995732278 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=201) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd2e4004c60 con 0x55c38ecf87c0 2024-02-21T23:11:45.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.766+0000 7fd30297b740 10 client.10062 fetch_fsmap learned FSMap version 201 2024-02-21T23:11:45.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.766+0000 7fd30297b740 1 -- 172.21.15.28:0/2995732278 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=201,monmap=2+}) v3 -- 0x55c38ed0f400 con 0x55c38ecf87c0 2024-02-21T23:11:45.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.767+0000 7fd2e2000640 1 -- 172.21.15.28:0/2995732278 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2f00836f0 con 0x55c38ecf87c0 2024-02-21T23:11:45.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.767+0000 7fd2e2000640 1 -- 172.21.15.28:0/2995732278 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 201) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fd2e402b050 con 0x55c38ecf87c0 2024-02-21T23:11:45.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.767+0000 7fd30297b740 10 client.10062 fetch_fsmap finished waiting for FSMap version 201 2024-02-21T23:11:45.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.767+0000 7fd30297b740 10 client.10062 resolve_mds: resolved name 'b' to daemon mds.b 2024-02-21T23:11:45.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.767+0000 7fd30297b740 20 client.10062 populate_metadata read hostname 'smithi028' 2024-02-21T23:11:45.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.767+0000 7fd30297b740 1 --2- 172.21.15.28:0/2995732278 >> [v2:172.21.15.81:6835/1709165826,v1:172.21.15.81:6837/1709165826] conn(0x55c38ed8ac10 0x55c38ed8aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:45.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.767+0000 7fd30297b740 4 client.10062 mds_command: new command op to 7738 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-21T23:11:45.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.767+0000 7fd30297b740 1 -- 172.21.15.28:0/2995732278 --> [v2:172.21.15.81:6835/1709165826,v1:172.21.15.81:6837/1709165826] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55c38ec47d00 con 0x55c38ed8ac10 2024-02-21T23:11:45.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.767+0000 7fd2fd000640 1 --2- 172.21.15.28:0/2995732278 >> [v2:172.21.15.81:6835/1709165826,v1:172.21.15.81:6837/1709165826] conn(0x55c38ed8ac10 0x55c38ed8aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:45.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.768+0000 7fd2fd000640 1 --2- 172.21.15.28:0/2995732278 >> [v2:172.21.15.81:6835/1709165826,v1:172.21.15.81:6837/1709165826] conn(0x55c38ed8ac10 0x55c38ed8aff0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d1f0973dccf3d9c2 server_cookie=231b9d48c77837c4 in_seq=0 out_seq=0 2024-02-21T23:11:45.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.768+0000 7fd2e2000640 10 client.10062 ms_handle_connect on v2:172.21.15.81:6835/1709165826 2024-02-21T23:11:45.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.769+0000 7fd2e2000640 1 -- 172.21.15.28:0/2995732278 <== mds.0 v2:172.21.15.81:6835/1709165826 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55c38ec47d00 con 0x55c38ed8ac10 2024-02-21T23:11:45.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.769+0000 7fd2e2000640 10 client.10062 handle_command_reply: tid=0 2024-02-21T23:11:45.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.769+0000 7fd30297b740 1 client.10062 shutdown 2024-02-21T23:11:45.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.769+0000 7fd30297b740 1 -- 172.21.15.28:0/2995732278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c38ecf87c0 msgr2=0x55c38ed1bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:45.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.769+0000 7fd30297b740 1 --2- 172.21.15.28:0/2995732278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c38ecf87c0 0x55c38ed1bab0 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 crypto rx=0x7fd2e4000f10 tx=0x7fd2e4019be0 comp rx=0 tx=0).stop 2024-02-21T23:11:45.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.769+0000 7fd30297b740 1 -- 172.21.15.28:0/2995732278 shutdown_connections 2024-02-21T23:11:45.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.769+0000 7fd30297b740 1 --2- 172.21.15.28:0/2995732278 >> [v2:172.21.15.81:6835/1709165826,v1:172.21.15.81:6837/1709165826] conn(0x55c38ed8ac10 0x55c38ed8aff0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:45.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.769+0000 7fd30297b740 1 --2- 172.21.15.28:0/2995732278 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c38ed186b0 0x55c38ed87cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:45.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.769+0000 7fd30297b740 1 --2- 172.21.15.28:0/2995732278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c38ecf87c0 0x55c38ed1bab0 unknown :-1 s=CLOSED pgs=2090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:45.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.769+0000 7fd30297b740 1 --2- 172.21.15.28:0/2995732278 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c38ecdd330 0x55c38ed1a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:45.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.769+0000 7fd2e2000640 0 client.10062 ms_handle_reset on v2:172.21.15.81:6835/1709165826 2024-02-21T23:11:45.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.769+0000 7fd30297b740 1 -- 172.21.15.28:0/2995732278 >> 172.21.15.28:0/2995732278 conn(0x55c38ecdc020 msgr2=0x55c38ed17d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:45.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.769+0000 7fd30297b740 1 -- 172.21.15.28:0/2995732278 shutdown_connections 2024-02-21T23:11:45.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.769+0000 7fd30297b740 1 -- 172.21.15.28:0/2995732278 wait complete. 2024-02-21T23:11:45.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.770+0000 7fd30297b740 20 client.10062 trim_cache size 0 max 16384 2024-02-21T23:11:45.760 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-21T23:11:45.760 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac", 2024-02-21T23:11:45.760 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-02-21T23:11:45.760 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 7738, 2024-02-21T23:11:45.760 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-02-21T23:11:45.760 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-02-21T23:11:45.760 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-02-21T23:11:45.761 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 6.821209895, 2024-02-21T23:11:45.761 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 201, 2024-02-21T23:11:45.761 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 186, 2024-02-21T23:11:45.761 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 186, 2024-02-21T23:11:45.761 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 12.302022232000001 2024-02-21T23:11:45.761 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-21T23:11:45.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.770+0000 7fd2fe400640 1 -- 172.21.15.28:0/3751242815 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd2d806b4d0 msgr2=0x7fd2d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:45.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.770+0000 7fd2fe400640 1 --2- 172.21.15.28:0/3751242815 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd2d806b4d0 0x7fd2d806d990 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fd2f81e82c0 tx=0x7fd2e8009040 comp rx=0 tx=0).stop 2024-02-21T23:11:45.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.770+0000 7fd2fe400640 1 -- 172.21.15.28:0/3751242815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2f8155a10 msgr2=0x7fd2f80bf150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:45.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.770+0000 7fd2fe400640 1 --2- 172.21.15.28:0/3751242815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2f8155a10 0x7fd2f80bf150 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 crypto rx=0x7fd2f0036040 tx=0x7fd2f0004a40 comp rx=0 tx=0).stop 2024-02-21T23:11:45.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.770+0000 7fd2fe400640 1 -- 172.21.15.28:0/3751242815 shutdown_connections 2024-02-21T23:11:45.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.770+0000 7fd2fe400640 1 --2- 172.21.15.28:0/3751242815 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd2d806b4d0 0x7fd2d806d990 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:45.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.770+0000 7fd2fe400640 1 --2- 172.21.15.28:0/3751242815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2f8155a10 0x7fd2f80bf150 unknown :-1 s=CLOSED pgs=2086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:45.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.770+0000 7fd2fe400640 1 --2- 172.21.15.28:0/3751242815 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2f81524c0 0x7fd2f80c0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:45.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.770+0000 7fd2fe400640 1 --2- 172.21.15.28:0/3751242815 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd2f80c5040 0x7fd2f80c0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:45.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.770+0000 7fd2fe400640 1 -- 172.21.15.28:0/3751242815 >> 172.21.15.28:0/3751242815 conn(0x7fd2f80ac2b0 msgr2=0x7fd2f80c7cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:45.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.771+0000 7fd2fe400640 1 -- 172.21.15.28:0/3751242815 shutdown_connections 2024-02-21T23:11:45.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:45.771+0000 7fd2fe400640 1 -- 172.21.15.28:0/3751242815 wait complete. 2024-02-21T23:11:45.778 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs down true 2024-02-21T23:11:46.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.012+0000 7f2a8de00640 1 Processor -- start 2024-02-21T23:11:46.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.013+0000 7f2a8de00640 1 -- start start 2024-02-21T23:11:46.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.014+0000 7f2a8de00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2a880c5df0 0x7f2a881556a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:46.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.014+0000 7f2a8de00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a880c5b80 con 0x7f2a8814f000 2024-02-21T23:11:46.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.014+0000 7f2a8de00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a88152950 con 0x7f2a88155be0 2024-02-21T23:11:46.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.014+0000 7f2a8de00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a880c4fe0 con 0x7f2a880c5df0 2024-02-21T23:11:46.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.014+0000 7f2a8ca00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f2a8814f000 0x7f2a8806ec20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55436/0 (socket says 172.21.15.28:55436) 2024-02-21T23:11:46.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.014+0000 7f2a8ca00640 1 -- 172.21.15.28:0/2906495427 learned_addr learned my addr 172.21.15.28:0/2906495427 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:11:46.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.015+0000 7f2a87e00640 1 --2- 172.21.15.28:0/2906495427 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2a880c5df0 0x7f2a881556a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:46.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.015+0000 7f2a87400640 1 -- 172.21.15.28:0/2906495427 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3868321821 0 0) 0x7f2a880c5b80 con 0x7f2a8814f000 2024-02-21T23:11:46.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.015+0000 7f2a87400640 1 -- 172.21.15.28:0/2906495427 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a6c005730 con 0x7f2a8814f000 2024-02-21T23:11:46.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.015+0000 7f2a87400640 1 -- 172.21.15.28:0/2906495427 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3257159302 0 0) 0x7f2a6c005730 con 0x7f2a8814f000 2024-02-21T23:11:46.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.015+0000 7f2a87400640 1 -- 172.21.15.28:0/2906495427 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2a880c5df0 msgr2=0x7f2a881556a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:46.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.015+0000 7f2a87400640 1 --2- 172.21.15.28:0/2906495427 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2a880c5df0 0x7f2a881556a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:46.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.015+0000 7f2a87400640 1 -- 172.21.15.28:0/2906495427 >> v1:172.21.15.81:6789/0 conn(0x7f2a88155be0 legacy=0x7f2a8815c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:46.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.015+0000 7f2a87400640 1 -- 172.21.15.28:0/2906495427 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a880c58a0 con 0x7f2a8814f000 2024-02-21T23:11:46.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.015+0000 7f2a87400640 1 -- 172.21.15.28:0/2906495427 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f2a780019f0 con 0x7f2a8814f000 2024-02-21T23:11:46.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.015+0000 7f2a87400640 1 -- 172.21.15.28:0/2906495427 >> v1:172.21.15.28:6789/0 conn(0x7f2a8814f000 legacy=0x7f2a8806ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:46.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.015+0000 7f2a87400640 1 --2- 172.21.15.28:0/2906495427 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2a880c5df0 0x7f2a6c005bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:46.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.016+0000 7f2a87400640 1 --2- 172.21.15.28:0/2906495427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a6c0061c0 0x7f2a6c006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:46.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.016+0000 7f2a87400640 1 --2- 172.21.15.28:0/2906495427 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2a6c006f50 0x7f2a6c0073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:46.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.016+0000 7f2a87400640 1 -- 172.21.15.28:0/2906495427 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a6c00dce0 con 0x7f2a6c0061c0 2024-02-21T23:11:46.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.016+0000 7f2a87400640 1 -- 172.21.15.28:0/2906495427 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a6c00de90 con 0x7f2a6c006f50 2024-02-21T23:11:46.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.016+0000 7f2a87400640 1 -- 172.21.15.28:0/2906495427 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a6c00e090 con 0x7f2a880c5df0 2024-02-21T23:11:46.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.016+0000 7f2a87e00640 1 --2- 172.21.15.28:0/2906495427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a6c0061c0 0x7f2a6c006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:46.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.016+0000 7f2a8d400640 1 --2- 172.21.15.28:0/2906495427 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2a6c006f50 0x7f2a6c0073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:46.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.016+0000 7f2a8ca00640 1 --2- 172.21.15.28:0/2906495427 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2a880c5df0 0x7f2a6c005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:46.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.016+0000 7f2a87e00640 1 -- 172.21.15.28:0/2906495427 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2a880c5df0 msgr2=0x7f2a6c005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:46.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.016+0000 7f2a87e00640 1 --2- 172.21.15.28:0/2906495427 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2a880c5df0 0x7f2a6c005bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:46.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.016+0000 7f2a87e00640 1 -- 172.21.15.28:0/2906495427 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2a6c006f50 msgr2=0x7f2a6c0073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:46.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.016+0000 7f2a87e00640 1 --2- 172.21.15.28:0/2906495427 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2a6c006f50 0x7f2a6c0073d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:46.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.016+0000 7f2a87e00640 1 -- 172.21.15.28:0/2906495427 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a6c005730 con 0x7f2a6c0061c0 2024-02-21T23:11:46.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.016+0000 7f2a8d400640 1 --2- 172.21.15.28:0/2906495427 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2a6c006f50 0x7f2a6c0073d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:11:46.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.017+0000 7f2a87e00640 1 --2- 172.21.15.28:0/2906495427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a6c0061c0 0x7f2a6c006a10 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7f2a74009ac0 tx=0x7f2a6c007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1eb5557dab57721 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:46.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.018+0000 7f2a87400640 1 -- 172.21.15.28:0/2906495427 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a74039070 con 0x7f2a6c0061c0 2024-02-21T23:11:46.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.018+0000 7f2a87400640 1 -- 172.21.15.28:0/2906495427 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a7402c600 con 0x7f2a6c0061c0 2024-02-21T23:11:46.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.018+0000 7f2a87400640 1 -- 172.21.15.28:0/2906495427 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a74009690 con 0x7f2a6c0061c0 2024-02-21T23:11:46.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.018+0000 7f2a8de00640 1 -- 172.21.15.28:0/2906495427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a6c0061c0 msgr2=0x7f2a6c006a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:46.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.018+0000 7f2a8de00640 1 --2- 172.21.15.28:0/2906495427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a6c0061c0 0x7f2a6c006a10 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7f2a74009ac0 tx=0x7f2a6c007a40 comp rx=0 tx=0).stop 2024-02-21T23:11:46.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.018+0000 7f2a8d400640 1 -- 172.21.15.28:0/2906495427 reap_dead start 2024-02-21T23:11:46.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.018+0000 7f2a8de00640 1 -- 172.21.15.28:0/2906495427 shutdown_connections 2024-02-21T23:11:46.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.018+0000 7f2a8de00640 1 -- 172.21.15.28:0/2906495427 >> 172.21.15.28:0/2906495427 conn(0x7f2a880a8530 msgr2=0x7f2a880a8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:46.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.018+0000 7f2a8de00640 1 -- 172.21.15.28:0/2906495427 shutdown_connections 2024-02-21T23:11:46.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.019+0000 7f2a8de00640 1 -- 172.21.15.28:0/2906495427 wait complete. 2024-02-21T23:11:46.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.020+0000 7f2a8de00640 1 Processor -- start 2024-02-21T23:11:46.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.020+0000 7f2a8de00640 1 -- start start 2024-02-21T23:11:46.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.021+0000 7f2a8de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a6c0061c0 0x7f2a8814f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:46.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.021+0000 7f2a8de00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2a88147f20 0x7f2a881473f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:46.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.021+0000 7f2a8de00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2a880a6170 0x7f2a88147930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:46.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.021+0000 7f2a8de00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a8806f330 con 0x7f2a6c0061c0 2024-02-21T23:11:46.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.021+0000 7f2a8de00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a880c5650 con 0x7f2a88147f20 2024-02-21T23:11:46.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.021+0000 7f2a8de00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a8806f4b0 con 0x7f2a880a6170 2024-02-21T23:11:46.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.021+0000 7f2a8ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a6c0061c0 0x7f2a8814f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:46.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.022+0000 7f2a8ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a6c0061c0 0x7f2a8814f9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55114/0 (socket says 172.21.15.28:55114) 2024-02-21T23:11:46.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.022+0000 7f2a8ca00640 1 -- 172.21.15.28:0/2288176119 learned_addr learned my addr 172.21.15.28:0/2288176119 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:46.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.022+0000 7f2a8ca00640 1 -- 172.21.15.28:0/2288176119 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2a880a6170 msgr2=0x7f2a88147930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:46.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.022+0000 7f2a8ca00640 1 --2- 172.21.15.28:0/2288176119 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2a880a6170 0x7f2a88147930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:46.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.022+0000 7f2a87e00640 1 --2- 172.21.15.28:0/2288176119 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2a88147f20 0x7f2a881473f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:46.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.022+0000 7f2a8ca00640 1 -- 172.21.15.28:0/2288176119 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2a88147f20 msgr2=0x7f2a881473f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:46.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.022+0000 7f2a8ca00640 1 --2- 172.21.15.28:0/2288176119 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2a88147f20 0x7f2a881473f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:46.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.022+0000 7f2a8ca00640 1 -- 172.21.15.28:0/2288176119 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a8806e770 con 0x7f2a6c0061c0 2024-02-21T23:11:46.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.022+0000 7f2a8ca00640 1 --2- 172.21.15.28:0/2288176119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a6c0061c0 0x7f2a8814f9f0 secure :-1 s=READY pgs=2093 cs=0 l=1 rev1=1 crypto rx=0x7f2a78002140 tx=0x7f2a7800b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:46.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.022+0000 7f2a84c00640 1 -- 172.21.15.28:0/2288176119 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a78019070 con 0x7f2a6c0061c0 2024-02-21T23:11:46.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.023+0000 7f2a84c00640 1 -- 172.21.15.28:0/2288176119 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a78004d60 con 0x7f2a6c0061c0 2024-02-21T23:11:46.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.023+0000 7f2a84c00640 1 -- 172.21.15.28:0/2288176119 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a78007310 con 0x7f2a6c0061c0 2024-02-21T23:11:46.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.023+0000 7f2a8de00640 1 -- 172.21.15.28:0/2288176119 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a881eb490 con 0x7f2a6c0061c0 2024-02-21T23:11:46.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.023+0000 7f2a8de00640 1 -- 172.21.15.28:0/2288176119 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a8814ff30 con 0x7f2a6c0061c0 2024-02-21T23:11:46.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.024+0000 7f2a8de00640 1 -- 172.21.15.28:0/2288176119 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a8804f290 con 0x7f2a6c0061c0 2024-02-21T23:11:46.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.025+0000 7f2a84c00640 1 -- 172.21.15.28:0/2288176119 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2a7801c070 con 0x7f2a6c0061c0 2024-02-21T23:11:46.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.026+0000 7f2a84c00640 1 --2- 172.21.15.28:0/2288176119 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2a6806b4d0 0x7f2a6806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:46.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.026+0000 7f2a84c00640 1 -- 172.21.15.28:0/2288176119 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f2a7808a560 con 0x7f2a6c0061c0 2024-02-21T23:11:46.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.027+0000 7f2a87e00640 1 --2- 172.21.15.28:0/2288176119 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2a6806b4d0 0x7f2a6806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:46.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.028+0000 7f2a87e00640 1 --2- 172.21.15.28:0/2288176119 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2a6806b4d0 0x7f2a6806d990 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f2a74036000 tx=0x7f2a74030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:46.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.030+0000 7f2a84c00640 1 -- 172.21.15.28:0/2288176119 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f2a780592e0 con 0x7f2a6c0061c0 2024-02-21T23:11:46.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.270+0000 7f2a8de00640 1 -- 172.21.15.28:0/2288176119 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "down", "val": "true"} v 0) v1 -- 0x7f2a88151360 con 0x7f2a6c0061c0 2024-02-21T23:11:46.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:46.996+0000 7f2a84c00640 1 -- 172.21.15.28:0/2288176119 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "down", "val": "true"}]=0 cephfs marked down. v202) v1 ==== 125+0+0 (secure 0 0 0) 0x7f2a78058be0 con 0x7f2a6c0061c0 2024-02-21T23:11:46.984 INFO:teuthology.orchestra.run.smithi028.stderr:cephfs marked down. 2024-02-21T23:11:46.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.000+0000 7f2a8de00640 1 -- 172.21.15.28:0/2288176119 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2a6806b4d0 msgr2=0x7f2a6806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:46.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.000+0000 7f2a8de00640 1 --2- 172.21.15.28:0/2288176119 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2a6806b4d0 0x7f2a6806d990 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f2a74036000 tx=0x7f2a74030000 comp rx=0 tx=0).stop 2024-02-21T23:11:46.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.000+0000 7f2a8de00640 1 -- 172.21.15.28:0/2288176119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a6c0061c0 msgr2=0x7f2a8814f9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:46.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.000+0000 7f2a8de00640 1 --2- 172.21.15.28:0/2288176119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a6c0061c0 0x7f2a8814f9f0 secure :-1 s=READY pgs=2093 cs=0 l=1 rev1=1 crypto rx=0x7f2a78002140 tx=0x7f2a7800b4a0 comp rx=0 tx=0).stop 2024-02-21T23:11:46.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.000+0000 7f2a8de00640 1 -- 172.21.15.28:0/2288176119 shutdown_connections 2024-02-21T23:11:46.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.000+0000 7f2a8de00640 1 --2- 172.21.15.28:0/2288176119 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2a6806b4d0 0x7f2a6806d990 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:46.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.000+0000 7f2a8de00640 1 --2- 172.21.15.28:0/2288176119 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2a880a6170 0x7f2a88147930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:46.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.000+0000 7f2a8de00640 1 --2- 172.21.15.28:0/2288176119 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2a88147f20 0x7f2a881473f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:46.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.000+0000 7f2a8de00640 1 --2- 172.21.15.28:0/2288176119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a6c0061c0 0x7f2a8814f9f0 unknown :-1 s=CLOSED pgs=2093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:46.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.000+0000 7f2a8de00640 1 -- 172.21.15.28:0/2288176119 >> 172.21.15.28:0/2288176119 conn(0x7f2a880a8530 msgr2=0x7f2a880c3040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:46.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.001+0000 7f2a8de00640 1 -- 172.21.15.28:0/2288176119 shutdown_connections 2024-02-21T23:11:46.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.001+0000 7f2a8de00640 1 -- 172.21.15.28:0/2288176119 wait complete. 2024-02-21T23:11:47.012 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:11:47.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.242+0000 7fee95000640 1 Processor -- start 2024-02-21T23:11:47.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.242+0000 7fee95000640 1 -- start start 2024-02-21T23:11:47.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.243+0000 7fee95000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fee900aff90 0x7fee900c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:47.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.243+0000 7fee95000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee900c74c0 con 0x7fee900c5720 2024-02-21T23:11:47.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.243+0000 7fee95000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee900c77a0 con 0x7fee9006dbf0 2024-02-21T23:11:47.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.243+0000 7fee95000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee900c7270 con 0x7fee900aff90 2024-02-21T23:11:47.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.244+0000 7fee8f400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fee900aff90 0x7fee900c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:47.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.244+0000 7fee8f400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fee900aff90 0x7fee900c51e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55124/0 (socket says 172.21.15.28:55124) 2024-02-21T23:11:47.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.244+0000 7fee8f400640 1 -- 172.21.15.28:0/4234473340 learned_addr learned my addr 172.21.15.28:0/4234473340 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:47.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.244+0000 7fee8f400640 1 -- 172.21.15.28:0/4234473340 >> v1:172.21.15.81:6789/0 conn(0x7fee9006dbf0 legacy=0x7fee90158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:47.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.244+0000 7fee8f400640 1 -- 172.21.15.28:0/4234473340 >> v1:172.21.15.28:6789/0 conn(0x7fee900c5720 legacy=0x7fee9015f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:47.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.244+0000 7fee8f400640 1 -- 172.21.15.28:0/4234473340 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee900c7a80 con 0x7fee900aff90 2024-02-21T23:11:47.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.245+0000 7fee8f400640 1 --2- 172.21.15.28:0/4234473340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee900aff90 0x7fee900c51e0 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7fee80009870 tx=0x7fee8002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bf6ac8ecf9b6cac server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:47.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.246+0000 7fee8e000640 1 -- 172.21.15.28:0/4234473340 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee80034070 con 0x7fee900aff90 2024-02-21T23:11:47.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.246+0000 7fee8e000640 1 -- 172.21.15.28:0/4234473340 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee80030860 con 0x7fee900aff90 2024-02-21T23:11:47.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.246+0000 7fee8e000640 1 -- 172.21.15.28:0/4234473340 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee80030c30 con 0x7fee900aff90 2024-02-21T23:11:47.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.246+0000 7fee95000640 1 -- 172.21.15.28:0/4234473340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee900aff90 msgr2=0x7fee900c51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:47.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.246+0000 7fee95000640 1 --2- 172.21.15.28:0/4234473340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee900aff90 0x7fee900c51e0 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7fee80009870 tx=0x7fee8002b9a0 comp rx=0 tx=0).stop 2024-02-21T23:11:47.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.247+0000 7fee95000640 1 -- 172.21.15.28:0/4234473340 shutdown_connections 2024-02-21T23:11:47.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.247+0000 7fee95000640 1 --2- 172.21.15.28:0/4234473340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee900aff90 0x7fee900c51e0 unknown :-1 s=CLOSED pgs=2095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:47.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.247+0000 7fee95000640 1 -- 172.21.15.28:0/4234473340 >> 172.21.15.28:0/4234473340 conn(0x7fee900ac370 msgr2=0x7fee900ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:47.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.247+0000 7fee95000640 1 -- 172.21.15.28:0/4234473340 shutdown_connections 2024-02-21T23:11:47.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.247+0000 7fee95000640 1 -- 172.21.15.28:0/4234473340 wait complete. 2024-02-21T23:11:47.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.249+0000 7fee95000640 1 Processor -- start 2024-02-21T23:11:47.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.249+0000 7fee95000640 1 -- start start 2024-02-21T23:11:47.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.250+0000 7fee95000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fee9006dbf0 0x7fee901ee170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:47.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.250+0000 7fee95000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee900c5720 0x7fee90150e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:47.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.250+0000 7fee95000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee9014c1b0 0x7fee90153380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:47.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.250+0000 7fee95000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee900c8760 con 0x7fee9014c1b0 2024-02-21T23:11:47.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.250+0000 7fee95000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee90158630 con 0x7fee900c5720 2024-02-21T23:11:47.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.250+0000 7fee95000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee900c88e0 con 0x7fee9006dbf0 2024-02-21T23:11:47.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.250+0000 7fee8fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee9014c1b0 0x7fee90153380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:47.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.250+0000 7fee8fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee9014c1b0 0x7fee90153380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55136/0 (socket says 172.21.15.28:55136) 2024-02-21T23:11:47.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.250+0000 7fee8fe00640 1 -- 172.21.15.28:0/2807391046 learned_addr learned my addr 172.21.15.28:0/2807391046 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:47.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.250+0000 7fee8f400640 1 --2- 172.21.15.28:0/2807391046 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fee9006dbf0 0x7fee901ee170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:47.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.250+0000 7fee8ea00640 1 --2- 172.21.15.28:0/2807391046 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee900c5720 0x7fee90150e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:47.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.251+0000 7fee8fe00640 1 -- 172.21.15.28:0/2807391046 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fee9006dbf0 msgr2=0x7fee901ee170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:47.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.251+0000 7fee8fe00640 1 --2- 172.21.15.28:0/2807391046 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fee9006dbf0 0x7fee901ee170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:47.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.251+0000 7fee8fe00640 1 -- 172.21.15.28:0/2807391046 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee900c5720 msgr2=0x7fee90150e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:47.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.251+0000 7fee8fe00640 1 --2- 172.21.15.28:0/2807391046 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee900c5720 0x7fee90150e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:47.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.251+0000 7fee8fe00640 1 -- 172.21.15.28:0/2807391046 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee901519f0 con 0x7fee9014c1b0 2024-02-21T23:11:47.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.251+0000 7fee8ea00640 1 --2- 172.21.15.28:0/2807391046 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee900c5720 0x7fee90150e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:11:47.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.251+0000 7fee8fe00640 1 --2- 172.21.15.28:0/2807391046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee9014c1b0 0x7fee90153380 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 crypto rx=0x7fee8800c6e0 tx=0x7fee8800cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:47.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.251+0000 7fee87400640 1 -- 172.21.15.28:0/2807391046 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee88003b70 con 0x7fee9014c1b0 2024-02-21T23:11:47.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.251+0000 7fee87400640 1 -- 172.21.15.28:0/2807391046 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee88003d30 con 0x7fee9014c1b0 2024-02-21T23:11:47.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.251+0000 7fee87400640 1 -- 172.21.15.28:0/2807391046 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee88005310 con 0x7fee9014c1b0 2024-02-21T23:11:47.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.251+0000 7fee95000640 1 -- 172.21.15.28:0/2807391046 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee90151cd0 con 0x7fee9014c1b0 2024-02-21T23:11:47.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.252+0000 7fee95000640 1 -- 172.21.15.28:0/2807391046 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee90152e60 con 0x7fee9014c1b0 2024-02-21T23:11:47.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.253+0000 7fee95000640 1 -- 172.21.15.28:0/2807391046 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee90153060 con 0x7fee9014c1b0 2024-02-21T23:11:47.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.254+0000 7fee87400640 1 -- 172.21.15.28:0/2807391046 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fee880054b0 con 0x7fee9014c1b0 2024-02-21T23:11:47.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.254+0000 7fee87400640 1 --2- 172.21.15.28:0/2807391046 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fee7006b4d0 0x7fee7006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:47.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.255+0000 7fee87400640 1 -- 172.21.15.28:0/2807391046 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fee8808a130 con 0x7fee9014c1b0 2024-02-21T23:11:47.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.255+0000 7fee8f400640 1 --2- 172.21.15.28:0/2807391046 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fee7006b4d0 0x7fee7006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:47.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.256+0000 7fee8f400640 1 --2- 172.21.15.28:0/2807391046 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fee7006b4d0 0x7fee7006d990 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fee80036000 tx=0x7fee8003b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:47.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.259+0000 7fee87400640 1 -- 172.21.15.28:0/2807391046 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fee90153060 con 0x7fee9014c1b0 2024-02-21T23:11:47.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.501+0000 7fee95000640 1 -- 172.21.15.28:0/2807391046 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fee9004f290 con 0x7fee9014c1b0 2024-02-21T23:11:47.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.502+0000 7fee87400640 1 -- 172.21.15.28:0/2807391046 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 203 v203) v1 ==== 95+0+4950 (secure 0 0 0) 0x7fee88058eb0 con 0x7fee9014c1b0 2024-02-21T23:11:47.490 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:11:47.490 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":203,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6848,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3439896042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3439896042},{"type":"v1","addr":"172.21.15.28:6835","nonce":3439896042}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":196},{"gid":7729,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/3213249808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3213249808},{"type":"v1","addr":"172.21.15.28:6833","nonce":3213249808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":195},{"gid":7735,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/2895690310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2895690310},{"type":"v1","addr":"172.21.15.81:6836","nonce":2895690310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":196}],"filesystems":[{"mdsmap":{"epoch":203,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:11:38.384619+0000","modified":"2024-02-21T23:11:47.000014+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0],"up":{"mds_0":7738},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7738":{"gid":7738,"name":"b","rank":0,"incarnation":197,"state":"up:stopping","state_seq":3,"addr":"172.21.15.81:6837/1709165826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":1709165826},{"type":"v1","addr":"172.21.15.81:6837","nonce":1709165826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-02-21T23:11:47.491 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 203 2024-02-21T23:11:47.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.505+0000 7fee95000640 1 -- 172.21.15.28:0/2807391046 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fee7006b4d0 msgr2=0x7fee7006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:47.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.505+0000 7fee95000640 1 --2- 172.21.15.28:0/2807391046 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fee7006b4d0 0x7fee7006d990 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fee80036000 tx=0x7fee8003b040 comp rx=0 tx=0).stop 2024-02-21T23:11:47.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.505+0000 7fee95000640 1 -- 172.21.15.28:0/2807391046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee9014c1b0 msgr2=0x7fee90153380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:47.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.505+0000 7fee95000640 1 --2- 172.21.15.28:0/2807391046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee9014c1b0 0x7fee90153380 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 crypto rx=0x7fee8800c6e0 tx=0x7fee8800cba0 comp rx=0 tx=0).stop 2024-02-21T23:11:47.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.505+0000 7fee95000640 1 -- 172.21.15.28:0/2807391046 shutdown_connections 2024-02-21T23:11:47.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.505+0000 7fee95000640 1 --2- 172.21.15.28:0/2807391046 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fee7006b4d0 0x7fee7006d990 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:47.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.505+0000 7fee95000640 1 --2- 172.21.15.28:0/2807391046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee9014c1b0 0x7fee90153380 unknown :-1 s=CLOSED pgs=2096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:47.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.505+0000 7fee95000640 1 --2- 172.21.15.28:0/2807391046 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee900c5720 0x7fee90150e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:47.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.505+0000 7fee95000640 1 --2- 172.21.15.28:0/2807391046 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fee9006dbf0 0x7fee901ee170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:47.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.505+0000 7fee95000640 1 -- 172.21.15.28:0/2807391046 >> 172.21.15.28:0/2807391046 conn(0x7fee900ac370 msgr2=0x7fee900aa1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:47.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.506+0000 7fee95000640 1 -- 172.21.15.28:0/2807391046 shutdown_connections 2024-02-21T23:11:47.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:47.506+0000 7fee95000640 1 -- 172.21.15.28:0/2807391046 wait complete. 2024-02-21T23:11:47.916 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 203, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:11:38.384619+0000', 'modified': '2024-02-21T23:11:47.000014+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 7738}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7738': {'gid': 7738, 'name': 'b', 'rank': 0, 'incarnation': 197, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.81:6837/1709165826', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6835', 'nonce': 1709165826}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 1709165826}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:11:47.916 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7738:up:stopping 2024-02-21T23:11:48.917 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:11:49.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.156+0000 7f935e800640 1 Processor -- start 2024-02-21T23:11:49.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.156+0000 7f935e800640 1 -- start start 2024-02-21T23:11:49.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.158+0000 7f935e800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9358150450 0x7f93580a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:49.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.158+0000 7f935e800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f935806ee40 0x7f93581536f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:49.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.158+0000 7f935e800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f9358156af0 0x7f935806dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:49.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.158+0000 7f935e800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93580c22c0 con 0x7f9358150450 2024-02-21T23:11:49.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.158+0000 7f935e800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9358156660 con 0x7f9358156af0 2024-02-21T23:11:49.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.158+0000 7f935e800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93580c2760 con 0x7f935806ee40 2024-02-21T23:11:49.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.158+0000 7f935d400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9358150450 0x7f93580a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:49.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.158+0000 7f935d400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9358150450 0x7f93580a5fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55148/0 (socket says 172.21.15.28:55148) 2024-02-21T23:11:49.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.158+0000 7f935d400640 1 -- 172.21.15.28:0/268513435 learned_addr learned my addr 172.21.15.28:0/268513435 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:49.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.159+0000 7f935ca00640 1 --2- 172.21.15.28:0/268513435 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f935806ee40 0x7f93581536f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:49.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.159+0000 7f935de00640 1 --2- 172.21.15.28:0/268513435 >> v2:172.21.15.81:3300/0 conn(0x7f9358156af0 0x7f935806dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:49.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.159+0000 7f935d400640 1 -- 172.21.15.28:0/268513435 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f935806ee40 msgr2=0x7f93581536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:49.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.159+0000 7f935d400640 1 --2- 172.21.15.28:0/268513435 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f935806ee40 0x7f93581536f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:49.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.159+0000 7f935d400640 1 -- 172.21.15.28:0/268513435 >> v2:172.21.15.81:3300/0 conn(0x7f9358156af0 msgr2=0x7f935806dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:49.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.159+0000 7f935d400640 1 --2- 172.21.15.28:0/268513435 >> v2:172.21.15.81:3300/0 conn(0x7f9358156af0 0x7f935806dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:49.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.159+0000 7f935d400640 1 -- 172.21.15.28:0/268513435 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9358153c30 con 0x7f9358150450 2024-02-21T23:11:49.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.161+0000 7f935d400640 1 --2- 172.21.15.28:0/268513435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9358150450 0x7f93580a5fd0 secure :-1 s=READY pgs=2097 cs=0 l=1 rev1=1 crypto rx=0x7f93580a87d0 tx=0x7f934c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3b8d6c22956b354 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:49.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.161+0000 7f9357e00640 1 -- 172.21.15.28:0/268513435 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f934c035020 con 0x7f9358150450 2024-02-21T23:11:49.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.161+0000 7f9357e00640 1 -- 172.21.15.28:0/268513435 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f934c031870 con 0x7f9358150450 2024-02-21T23:11:49.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.161+0000 7f9357e00640 1 -- 172.21.15.28:0/268513435 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f934c031c40 con 0x7f9358150450 2024-02-21T23:11:49.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.162+0000 7f935e800640 1 -- 172.21.15.28:0/268513435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9358150450 msgr2=0x7f93580a5fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:49.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.162+0000 7f935e800640 1 --2- 172.21.15.28:0/268513435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9358150450 0x7f93580a5fd0 secure :-1 s=READY pgs=2097 cs=0 l=1 rev1=1 crypto rx=0x7f93580a87d0 tx=0x7f934c02b910 comp rx=0 tx=0).stop 2024-02-21T23:11:49.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.162+0000 7f935e800640 1 -- 172.21.15.28:0/268513435 shutdown_connections 2024-02-21T23:11:49.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.162+0000 7f935e800640 1 --2- 172.21.15.28:0/268513435 >> v2:172.21.15.81:3300/0 conn(0x7f9358156af0 0x7f935806dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:49.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.162+0000 7f935e800640 1 --2- 172.21.15.28:0/268513435 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f935806ee40 0x7f93581536f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:49.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.162+0000 7f935e800640 1 --2- 172.21.15.28:0/268513435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9358150450 0x7f93580a5fd0 unknown :-1 s=CLOSED pgs=2097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:49.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.162+0000 7f935e800640 1 -- 172.21.15.28:0/268513435 >> 172.21.15.28:0/268513435 conn(0x7f935806b630 msgr2=0x7f93580c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:49.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.162+0000 7f935e800640 1 -- 172.21.15.28:0/268513435 shutdown_connections 2024-02-21T23:11:49.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.162+0000 7f935e800640 1 -- 172.21.15.28:0/268513435 wait complete. 2024-02-21T23:11:49.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.164+0000 7f935e800640 1 Processor -- start 2024-02-21T23:11:49.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.165+0000 7f935e800640 1 -- start start 2024-02-21T23:11:49.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.166+0000 7f935e800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f935806ee40 0x7f9358149290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:49.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.166+0000 7f935e800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9358150450 0x7f9358072360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:49.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.166+0000 7f935e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9358156af0 0x7f9358150f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:49.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.166+0000 7f935e800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93580c2cb0 con 0x7f9358156af0 2024-02-21T23:11:49.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.166+0000 7f935e800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93580c29d0 con 0x7f935806ee40 2024-02-21T23:11:49.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.166+0000 7f935e800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9358156f80 con 0x7f9358150450 2024-02-21T23:11:49.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.166+0000 7f935de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9358156af0 0x7f9358150f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:49.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.166+0000 7f935de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9358156af0 0x7f9358150f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55154/0 (socket says 172.21.15.28:55154) 2024-02-21T23:11:49.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.166+0000 7f935de00640 1 -- 172.21.15.28:0/2218862095 learned_addr learned my addr 172.21.15.28:0/2218862095 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:49.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.167+0000 7f935ca00640 1 --2- 172.21.15.28:0/2218862095 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9358150450 0x7f9358072360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:49.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.167+0000 7f935d400640 1 --2- 172.21.15.28:0/2218862095 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f935806ee40 0x7f9358149290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:49.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.167+0000 7f935de00640 1 -- 172.21.15.28:0/2218862095 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9358150450 msgr2=0x7f9358072360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:49.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.167+0000 7f935de00640 1 --2- 172.21.15.28:0/2218862095 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9358150450 0x7f9358072360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:49.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.167+0000 7f935de00640 1 -- 172.21.15.28:0/2218862095 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f935806ee40 msgr2=0x7f9358149290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:49.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.167+0000 7f935de00640 1 --2- 172.21.15.28:0/2218862095 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f935806ee40 0x7f9358149290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:49.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.167+0000 7f935de00640 1 -- 172.21.15.28:0/2218862095 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93581eb8e0 con 0x7f9358156af0 2024-02-21T23:11:49.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.167+0000 7f935de00640 1 --2- 172.21.15.28:0/2218862095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9358156af0 0x7f9358150f10 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 crypto rx=0x7f9350007e60 tx=0x7f935000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:49.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.167+0000 7f9355600640 1 -- 172.21.15.28:0/2218862095 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9350014420 con 0x7f9358156af0 2024-02-21T23:11:49.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.168+0000 7f9355600640 1 -- 172.21.15.28:0/2218862095 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9350019070 con 0x7f9358156af0 2024-02-21T23:11:49.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.168+0000 7f9355600640 1 -- 172.21.15.28:0/2218862095 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9350014820 con 0x7f9358156af0 2024-02-21T23:11:49.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.168+0000 7f935e800640 1 -- 172.21.15.28:0/2218862095 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9358152aa0 con 0x7f9358156af0 2024-02-21T23:11:49.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.168+0000 7f935e800640 1 -- 172.21.15.28:0/2218862095 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f935814a030 con 0x7f9358156af0 2024-02-21T23:11:49.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.169+0000 7f935e800640 1 -- 172.21.15.28:0/2218862095 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9358152fc0 con 0x7f9358156af0 2024-02-21T23:11:49.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.170+0000 7f9355600640 1 -- 172.21.15.28:0/2218862095 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f93500149c0 con 0x7f9358156af0 2024-02-21T23:11:49.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.171+0000 7f9355600640 1 --2- 172.21.15.28:0/2218862095 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f933806b4d0 0x7f933806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:49.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.171+0000 7f9355600640 1 -- 172.21.15.28:0/2218862095 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f935008ac00 con 0x7f9358156af0 2024-02-21T23:11:49.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.171+0000 7f935d400640 1 --2- 172.21.15.28:0/2218862095 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f933806b4d0 0x7f933806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:49.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.172+0000 7f935d400640 1 --2- 172.21.15.28:0/2218862095 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f933806b4d0 0x7f933806d990 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f934c036000 tx=0x7f934c03c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:49.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.176+0000 7f9355600640 1 -- 172.21.15.28:0/2218862095 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f9358152fc0 con 0x7f9358156af0 2024-02-21T23:11:49.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.410+0000 7f935e800640 1 -- 172.21.15.28:0/2218862095 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f935804f290 con 0x7f9358156af0 2024-02-21T23:11:49.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.412+0000 7f9355600640 1 -- 172.21.15.28:0/2218862095 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 203 v203) v1 ==== 95+0+4950 (secure 0 0 0) 0x7f93500598e0 con 0x7f9358156af0 2024-02-21T23:11:49.400 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:11:49.400 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":203,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6848,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3439896042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3439896042},{"type":"v1","addr":"172.21.15.28:6835","nonce":3439896042}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":196},{"gid":7729,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/3213249808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3213249808},{"type":"v1","addr":"172.21.15.28:6833","nonce":3213249808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":195},{"gid":7735,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/2895690310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2895690310},{"type":"v1","addr":"172.21.15.81:6836","nonce":2895690310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":196}],"filesystems":[{"mdsmap":{"epoch":203,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:11:38.384619+0000","modified":"2024-02-21T23:11:47.000014+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0],"up":{"mds_0":7738},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7738":{"gid":7738,"name":"b","rank":0,"incarnation":197,"state":"up:stopping","state_seq":3,"addr":"172.21.15.81:6837/1709165826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":1709165826},{"type":"v1","addr":"172.21.15.81:6837","nonce":1709165826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-02-21T23:11:49.401 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 203 2024-02-21T23:11:49.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.414+0000 7f935e800640 1 -- 172.21.15.28:0/2218862095 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f933806b4d0 msgr2=0x7f933806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:49.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.414+0000 7f935e800640 1 --2- 172.21.15.28:0/2218862095 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f933806b4d0 0x7f933806d990 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f934c036000 tx=0x7f934c03c040 comp rx=0 tx=0).stop 2024-02-21T23:11:49.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.414+0000 7f935e800640 1 -- 172.21.15.28:0/2218862095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9358156af0 msgr2=0x7f9358150f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:49.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.414+0000 7f935e800640 1 --2- 172.21.15.28:0/2218862095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9358156af0 0x7f9358150f10 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 crypto rx=0x7f9350007e60 tx=0x7f935000e4b0 comp rx=0 tx=0).stop 2024-02-21T23:11:49.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.414+0000 7f935e800640 1 -- 172.21.15.28:0/2218862095 shutdown_connections 2024-02-21T23:11:49.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.414+0000 7f935e800640 1 --2- 172.21.15.28:0/2218862095 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f933806b4d0 0x7f933806d990 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:49.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.414+0000 7f935e800640 1 --2- 172.21.15.28:0/2218862095 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9358150450 0x7f9358072360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:49.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.414+0000 7f935e800640 1 --2- 172.21.15.28:0/2218862095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9358156af0 0x7f9358150f10 unknown :-1 s=CLOSED pgs=2098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:49.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.414+0000 7f935e800640 1 --2- 172.21.15.28:0/2218862095 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f935806ee40 0x7f9358149290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:49.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.414+0000 7f935e800640 1 -- 172.21.15.28:0/2218862095 >> 172.21.15.28:0/2218862095 conn(0x7f935806b630 msgr2=0x7f935814e330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:49.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.414+0000 7f935e800640 1 -- 172.21.15.28:0/2218862095 shutdown_connections 2024-02-21T23:11:49.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:49.414+0000 7f935e800640 1 -- 172.21.15.28:0/2218862095 wait complete. 2024-02-21T23:11:49.823 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 203, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:11:38.384619+0000', 'modified': '2024-02-21T23:11:47.000014+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 7738}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7738': {'gid': 7738, 'name': 'b', 'rank': 0, 'incarnation': 197, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.81:6837/1709165826', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6835', 'nonce': 1709165826}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 1709165826}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:11:49.823 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7738:up:stopping 2024-02-21T23:11:50.824 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:11:51.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.065+0000 7fe351800640 1 Processor -- start 2024-02-21T23:11:51.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.066+0000 7fe351800640 1 -- start start 2024-02-21T23:11:51.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.067+0000 7fe351800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe34c0c0f00 0x7fe34c159db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:51.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.067+0000 7fe351800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe34c071070 0x7fe34c071500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:51.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.067+0000 7fe351800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe34c0c1390 con 0x7fe34c1569b0 2024-02-21T23:11:51.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.067+0000 7fe351800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe34c0c1fa0 con 0x7fe34c071070 2024-02-21T23:11:51.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.067+0000 7fe351800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe34c0c1670 con 0x7fe34c0c0f00 2024-02-21T23:11:51.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.068+0000 7fe350e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe34c071070 0x7fe34c071500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:51.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.068+0000 7fe350e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe34c071070 0x7fe34c071500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55170/0 (socket says 172.21.15.28:55170) 2024-02-21T23:11:51.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.068+0000 7fe350e00640 1 -- 172.21.15.28:0/4268604513 learned_addr learned my addr 172.21.15.28:0/4268604513 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:51.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.068+0000 7fe343e00640 1 --2- 172.21.15.28:0/4268604513 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe34c0c0f00 0x7fe34c159db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:51.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.068+0000 7fe350e00640 1 -- 172.21.15.28:0/4268604513 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe34c0c0f00 msgr2=0x7fe34c159db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:51.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.068+0000 7fe350e00640 1 --2- 172.21.15.28:0/4268604513 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe34c0c0f00 0x7fe34c159db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:51.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.068+0000 7fe350e00640 1 -- 172.21.15.28:0/4268604513 >> v1:172.21.15.81:6789/0 conn(0x7fe34c1569b0 legacy=0x7fe34c06dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:51.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.068+0000 7fe350e00640 1 -- 172.21.15.28:0/4268604513 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe34c0c1950 con 0x7fe34c071070 2024-02-21T23:11:51.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.069+0000 7fe350e00640 1 --2- 172.21.15.28:0/4268604513 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe34c071070 0x7fe34c071500 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 crypto rx=0x7fe344009e50 tx=0x7fe34402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5e08e085b99dda6 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:51.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.070+0000 7fe34b400640 1 -- 172.21.15.28:0/4268604513 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe34402c9e0 con 0x7fe34c071070 2024-02-21T23:11:51.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.070+0000 7fe34b400640 1 -- 172.21.15.28:0/4268604513 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe34402cb80 con 0x7fe34c071070 2024-02-21T23:11:51.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.070+0000 7fe34b400640 1 -- 172.21.15.28:0/4268604513 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3440343f0 con 0x7fe34c071070 2024-02-21T23:11:51.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.070+0000 7fe351800640 1 -- 172.21.15.28:0/4268604513 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe34c071070 msgr2=0x7fe34c071500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:51.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.070+0000 7fe351800640 1 --2- 172.21.15.28:0/4268604513 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe34c071070 0x7fe34c071500 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 crypto rx=0x7fe344009e50 tx=0x7fe34402be40 comp rx=0 tx=0).stop 2024-02-21T23:11:51.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.071+0000 7fe351800640 1 -- 172.21.15.28:0/4268604513 shutdown_connections 2024-02-21T23:11:51.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.071+0000 7fe351800640 1 --2- 172.21.15.28:0/4268604513 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe34c071070 0x7fe34c071500 unknown :-1 s=CLOSED pgs=2099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:51.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.071+0000 7fe351800640 1 --2- 172.21.15.28:0/4268604513 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe34c0c0f00 0x7fe34c159db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:51.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.071+0000 7fe351800640 1 -- 172.21.15.28:0/4268604513 >> 172.21.15.28:0/4268604513 conn(0x7fe34c06b630 msgr2=0x7fe34c0c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:51.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.071+0000 7fe351800640 1 -- 172.21.15.28:0/4268604513 shutdown_connections 2024-02-21T23:11:51.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.071+0000 7fe351800640 1 -- 172.21.15.28:0/4268604513 wait complete. 2024-02-21T23:11:51.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.073+0000 7fe351800640 1 Processor -- start 2024-02-21T23:11:51.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.073+0000 7fe351800640 1 -- start start 2024-02-21T23:11:51.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.074+0000 7fe351800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe34c071070 0x7fe34c15b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:51.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.074+0000 7fe351800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe34c1569b0 0x7fe34c0a3f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:51.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.074+0000 7fe351800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe34c15bbf0 0x7fe34c0a44a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:51.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.074+0000 7fe351800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe34c06e700 con 0x7fe34c15bbf0 2024-02-21T23:11:51.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.074+0000 7fe351800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe34c0c2ba0 con 0x7fe34c1569b0 2024-02-21T23:11:51.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.074+0000 7fe351800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe34c06e880 con 0x7fe34c071070 2024-02-21T23:11:51.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.074+0000 7fe350e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe34c15bbf0 0x7fe34c0a44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:51.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.075+0000 7fe350e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe34c15bbf0 0x7fe34c0a44a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55182/0 (socket says 172.21.15.28:55182) 2024-02-21T23:11:51.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.075+0000 7fe350e00640 1 -- 172.21.15.28:0/218446201 learned_addr learned my addr 172.21.15.28:0/218446201 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:51.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.075+0000 7fe343e00640 1 --2- 172.21.15.28:0/218446201 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe34c1569b0 0x7fe34c0a3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:51.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.075+0000 7fe34be00640 1 --2- 172.21.15.28:0/218446201 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe34c071070 0x7fe34c15b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:51.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.075+0000 7fe350e00640 1 -- 172.21.15.28:0/218446201 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe34c071070 msgr2=0x7fe34c15b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:51.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.075+0000 7fe350e00640 1 --2- 172.21.15.28:0/218446201 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe34c071070 0x7fe34c15b6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:51.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.075+0000 7fe350e00640 1 -- 172.21.15.28:0/218446201 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe34c1569b0 msgr2=0x7fe34c0a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:51.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.075+0000 7fe350e00640 1 --2- 172.21.15.28:0/218446201 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe34c1569b0 0x7fe34c0a3f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:51.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.075+0000 7fe350e00640 1 -- 172.21.15.28:0/218446201 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe34c155a50 con 0x7fe34c15bbf0 2024-02-21T23:11:51.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.075+0000 7fe34be00640 1 --2- 172.21.15.28:0/218446201 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe34c071070 0x7fe34c15b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:11:51.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.075+0000 7fe343e00640 1 --2- 172.21.15.28:0/218446201 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe34c1569b0 0x7fe34c0a3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:11:51.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.075+0000 7fe350e00640 1 --2- 172.21.15.28:0/218446201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe34c15bbf0 0x7fe34c0a44a0 secure :-1 s=READY pgs=2100 cs=0 l=1 rev1=1 crypto rx=0x7fe344036040 tx=0x7fe344004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:51.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.076+0000 7fe348c00640 1 -- 172.21.15.28:0/218446201 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe344043020 con 0x7fe34c15bbf0 2024-02-21T23:11:51.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.076+0000 7fe348c00640 1 -- 172.21.15.28:0/218446201 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe34403e450 con 0x7fe34c15bbf0 2024-02-21T23:11:51.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.076+0000 7fe348c00640 1 -- 172.21.15.28:0/218446201 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe344009510 con 0x7fe34c15bbf0 2024-02-21T23:11:51.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.076+0000 7fe351800640 1 -- 172.21.15.28:0/218446201 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe34c14f830 con 0x7fe34c15bbf0 2024-02-21T23:11:51.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.076+0000 7fe351800640 1 -- 172.21.15.28:0/218446201 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe34c14fc00 con 0x7fe34c15bbf0 2024-02-21T23:11:51.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.077+0000 7fe351800640 1 -- 172.21.15.28:0/218446201 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe34c14fa00 con 0x7fe34c15bbf0 2024-02-21T23:11:51.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.078+0000 7fe348c00640 1 -- 172.21.15.28:0/218446201 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe344046020 con 0x7fe34c15bbf0 2024-02-21T23:11:51.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.079+0000 7fe348c00640 1 --2- 172.21.15.28:0/218446201 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe32806b4d0 0x7fe32806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:51.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.079+0000 7fe348c00640 1 -- 172.21.15.28:0/218446201 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe3440acb80 con 0x7fe34c15bbf0 2024-02-21T23:11:51.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.080+0000 7fe34be00640 1 --2- 172.21.15.28:0/218446201 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe32806b4d0 0x7fe32806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:51.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.081+0000 7fe34be00640 1 --2- 172.21.15.28:0/218446201 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe32806b4d0 0x7fe32806d990 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fe33c00a830 tx=0x7fe33c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:51.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.083+0000 7fe348c00640 1 -- 172.21.15.28:0/218446201 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe34c14fa00 con 0x7fe34c15bbf0 2024-02-21T23:11:51.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.327+0000 7fe351800640 1 -- 172.21.15.28:0/218446201 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe34c0a2260 con 0x7fe34c15bbf0 2024-02-21T23:11:51.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.328+0000 7fe348c00640 1 -- 172.21.15.28:0/218446201 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 203 v203) v1 ==== 95+0+4950 (secure 0 0 0) 0x7fe34404d030 con 0x7fe34c15bbf0 2024-02-21T23:11:51.317 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:11:51.317 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":203,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6848,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3439896042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3439896042},{"type":"v1","addr":"172.21.15.28:6835","nonce":3439896042}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":196},{"gid":7729,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/3213249808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3213249808},{"type":"v1","addr":"172.21.15.28:6833","nonce":3213249808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":195},{"gid":7735,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/2895690310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2895690310},{"type":"v1","addr":"172.21.15.81:6836","nonce":2895690310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":196}],"filesystems":[{"mdsmap":{"epoch":203,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:11:38.384619+0000","modified":"2024-02-21T23:11:47.000014+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0],"up":{"mds_0":7738},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7738":{"gid":7738,"name":"b","rank":0,"incarnation":197,"state":"up:stopping","state_seq":3,"addr":"172.21.15.81:6837/1709165826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":1709165826},{"type":"v1","addr":"172.21.15.81:6837","nonce":1709165826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-02-21T23:11:51.317 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 203 2024-02-21T23:11:51.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.330+0000 7fe351800640 1 -- 172.21.15.28:0/218446201 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe32806b4d0 msgr2=0x7fe32806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:51.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.330+0000 7fe351800640 1 --2- 172.21.15.28:0/218446201 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe32806b4d0 0x7fe32806d990 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fe33c00a830 tx=0x7fe33c008040 comp rx=0 tx=0).stop 2024-02-21T23:11:51.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.330+0000 7fe351800640 1 -- 172.21.15.28:0/218446201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe34c15bbf0 msgr2=0x7fe34c0a44a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:51.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.330+0000 7fe351800640 1 --2- 172.21.15.28:0/218446201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe34c15bbf0 0x7fe34c0a44a0 secure :-1 s=READY pgs=2100 cs=0 l=1 rev1=1 crypto rx=0x7fe344036040 tx=0x7fe344004a40 comp rx=0 tx=0).stop 2024-02-21T23:11:51.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.330+0000 7fe351800640 1 -- 172.21.15.28:0/218446201 shutdown_connections 2024-02-21T23:11:51.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.330+0000 7fe351800640 1 --2- 172.21.15.28:0/218446201 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe32806b4d0 0x7fe32806d990 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:51.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.330+0000 7fe351800640 1 --2- 172.21.15.28:0/218446201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe34c15bbf0 0x7fe34c0a44a0 unknown :-1 s=CLOSED pgs=2100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:51.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.330+0000 7fe351800640 1 --2- 172.21.15.28:0/218446201 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe34c1569b0 0x7fe34c0a3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:51.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.330+0000 7fe351800640 1 --2- 172.21.15.28:0/218446201 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe34c071070 0x7fe34c15b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:51.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.330+0000 7fe351800640 1 -- 172.21.15.28:0/218446201 >> 172.21.15.28:0/218446201 conn(0x7fe34c06b630 msgr2=0x7fe34c06dff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:51.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.330+0000 7fe351800640 1 -- 172.21.15.28:0/218446201 shutdown_connections 2024-02-21T23:11:51.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:51.330+0000 7fe351800640 1 -- 172.21.15.28:0/218446201 wait complete. 2024-02-21T23:11:51.743 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 203, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:11:38.384619+0000', 'modified': '2024-02-21T23:11:47.000014+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 7738}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7738': {'gid': 7738, 'name': 'b', 'rank': 0, 'incarnation': 197, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.81:6837/1709165826', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6835', 'nonce': 1709165826}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 1709165826}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:11:51.743 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7738:up:stopping 2024-02-21T23:11:52.744 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:11:52.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.985+0000 7f6505c00640 1 Processor -- start 2024-02-21T23:11:52.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.985+0000 7f6505c00640 1 -- start start 2024-02-21T23:11:52.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.987+0000 7f6505c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f6500150450 0x7f65000a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:52.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.987+0000 7f6505c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65000c2760 con 0x7f650006ee40 2024-02-21T23:11:52.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.987+0000 7f6505c00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6500156310 con 0x7f6500156af0 2024-02-21T23:11:52.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.987+0000 7f6505c00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65000c22c0 con 0x7f6500150450 2024-02-21T23:11:52.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.987+0000 7f64ff400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f650006ee40 0x7f65001536f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55452/0 (socket says 172.21.15.28:55452) 2024-02-21T23:11:52.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.987+0000 7f64ff400640 1 -- 172.21.15.28:0/1912513595 learned_addr learned my addr 172.21.15.28:0/1912513595 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:11:52.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.988+0000 7f64ffe00640 1 --2- 172.21.15.28:0/1912513595 >> v2:172.21.15.81:3300/0 conn(0x7f6500150450 0x7f65000a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:52.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.988+0000 7f64fea00640 1 -- 172.21.15.28:0/1912513595 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4081724485 0 0) 0x7f65000c2760 con 0x7f650006ee40 2024-02-21T23:11:52.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.988+0000 7f64fea00640 1 -- 172.21.15.28:0/1912513595 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64e4003520 con 0x7f650006ee40 2024-02-21T23:11:52.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.988+0000 7f64fea00640 1 -- 172.21.15.28:0/1912513595 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3847043221 0 0) 0x7f64e4003520 con 0x7f650006ee40 2024-02-21T23:11:52.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.988+0000 7f64fea00640 1 -- 172.21.15.28:0/1912513595 >> v2:172.21.15.81:3300/0 conn(0x7f6500150450 msgr2=0x7f65000a5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:52.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.988+0000 7f64fea00640 1 --2- 172.21.15.28:0/1912513595 >> v2:172.21.15.81:3300/0 conn(0x7f6500150450 0x7f65000a5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:52.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.988+0000 7f64fea00640 1 -- 172.21.15.28:0/1912513595 >> v1:172.21.15.81:6789/0 conn(0x7f6500156af0 legacy=0x7f650006dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:52.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.988+0000 7f64fea00640 1 -- 172.21.15.28:0/1912513595 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65000c29d0 con 0x7f650006ee40 2024-02-21T23:11:52.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.989+0000 7f64fea00640 1 -- 172.21.15.28:0/1912513595 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f64ec001670 con 0x7f650006ee40 2024-02-21T23:11:52.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.989+0000 7f64fea00640 1 -- 172.21.15.28:0/1912513595 >> v1:172.21.15.28:6789/0 conn(0x7f650006ee40 legacy=0x7f65001536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:52.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.989+0000 7f64fea00640 1 --2- 172.21.15.28:0/1912513595 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64e40039b0 0x7f64e4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:52.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.989+0000 7f64fea00640 1 --2- 172.21.15.28:0/1912513595 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f64e4004800 0x7f64e4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:52.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.989+0000 7f64fea00640 1 --2- 172.21.15.28:0/1912513595 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64e40051c0 0x7f64e400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:52.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.989+0000 7f64fea00640 1 -- 172.21.15.28:0/1912513595 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64e400c030 con 0x7f64e40051c0 2024-02-21T23:11:52.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.989+0000 7f64fea00640 1 -- 172.21.15.28:0/1912513595 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64e400c1e0 con 0x7f64e40039b0 2024-02-21T23:11:52.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.989+0000 7f64fea00640 1 -- 172.21.15.28:0/1912513595 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64e400c3e0 con 0x7f64e4004800 2024-02-21T23:11:52.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.989+0000 7f6505200640 1 --2- 172.21.15.28:0/1912513595 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64e40051c0 0x7f64e400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:52.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.989+0000 7f64ffe00640 1 --2- 172.21.15.28:0/1912513595 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64e40039b0 0x7f64e4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:52.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.989+0000 7f64ff400640 1 --2- 172.21.15.28:0/1912513595 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f64e4004800 0x7f64e4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:52.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.989+0000 7f6505200640 1 -- 172.21.15.28:0/1912513595 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f64e4004800 msgr2=0x7f64e4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:52.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.989+0000 7f6505200640 1 --2- 172.21.15.28:0/1912513595 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f64e4004800 0x7f64e4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:52.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.989+0000 7f6505200640 1 -- 172.21.15.28:0/1912513595 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64e40039b0 msgr2=0x7f64e4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:52.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.989+0000 7f6505200640 1 --2- 172.21.15.28:0/1912513595 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64e40039b0 0x7f64e4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:52.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.989+0000 7f6505200640 1 -- 172.21.15.28:0/1912513595 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64e4003520 con 0x7f64e40051c0 2024-02-21T23:11:52.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.990+0000 7f6505200640 1 -- 172.21.15.28:0/1912513595 reap_dead start 2024-02-21T23:11:52.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.991+0000 7f6505200640 1 --2- 172.21.15.28:0/1912513595 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64e40051c0 0x7f64e400b9c0 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 crypto rx=0x7f64f8001e50 tx=0x7f64e4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9bf62e300f08085 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:52.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.991+0000 7f64fea00640 1 -- 172.21.15.28:0/1912513595 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64f8001a70 con 0x7f64e40051c0 2024-02-21T23:11:52.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.991+0000 7f64fea00640 1 -- 172.21.15.28:0/1912513595 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64f8003b90 con 0x7f64e40051c0 2024-02-21T23:11:52.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.991+0000 7f64fea00640 1 -- 172.21.15.28:0/1912513595 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64f80050d0 con 0x7f64e40051c0 2024-02-21T23:11:52.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.991+0000 7f6505c00640 1 -- 172.21.15.28:0/1912513595 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64e40051c0 msgr2=0x7f64e400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:52.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.991+0000 7f6505c00640 1 --2- 172.21.15.28:0/1912513595 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64e40051c0 0x7f64e400b9c0 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 crypto rx=0x7f64f8001e50 tx=0x7f64e4005830 comp rx=0 tx=0).stop 2024-02-21T23:11:52.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.992+0000 7f6505c00640 1 -- 172.21.15.28:0/1912513595 shutdown_connections 2024-02-21T23:11:52.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.992+0000 7f6505c00640 1 --2- 172.21.15.28:0/1912513595 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64e40051c0 0x7f64e400b9c0 unknown :-1 s=CLOSED pgs=2102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:52.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.992+0000 7f6505c00640 1 -- 172.21.15.28:0/1912513595 >> 172.21.15.28:0/1912513595 conn(0x7f650006b630 msgr2=0x7f65000c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:52.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.992+0000 7f6505c00640 1 -- 172.21.15.28:0/1912513595 shutdown_connections 2024-02-21T23:11:52.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.992+0000 7f6505c00640 1 -- 172.21.15.28:0/1912513595 wait complete. 2024-02-21T23:11:52.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.993+0000 7f6505c00640 1 Processor -- start 2024-02-21T23:11:52.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.994+0000 7f6505c00640 1 -- start start 2024-02-21T23:11:52.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.994+0000 7f6505c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f64e40051c0 0x7f65000a9330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:52.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.994+0000 7f6505c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65000ac5f0 0x7f65000ab4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:52.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.994+0000 7f6505c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6500144240 0x7f65001edf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:52.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.994+0000 7f6505c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6500156f00 con 0x7f65000ac5f0 2024-02-21T23:11:52.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.994+0000 7f6505c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6500156660 con 0x7f6500144240 2024-02-21T23:11:52.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.994+0000 7f6505c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6500157080 con 0x7f64e40051c0 2024-02-21T23:11:52.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.995+0000 7f64ff400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65000ac5f0 0x7f65000ab4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:52.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.995+0000 7f64ff400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65000ac5f0 0x7f65000ab4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55208/0 (socket says 172.21.15.28:55208) 2024-02-21T23:11:52.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.995+0000 7f64ff400640 1 -- 172.21.15.28:0/3428034743 learned_addr learned my addr 172.21.15.28:0/3428034743 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:52.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.995+0000 7f64ffe00640 1 --2- 172.21.15.28:0/3428034743 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f64e40051c0 0x7f65000a9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:52.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.995+0000 7f6505200640 1 --2- 172.21.15.28:0/3428034743 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6500144240 0x7f65001edf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:52.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.995+0000 7f64ff400640 1 -- 172.21.15.28:0/3428034743 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f64e40051c0 msgr2=0x7f65000a9330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:52.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.995+0000 7f64ff400640 1 --2- 172.21.15.28:0/3428034743 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f64e40051c0 0x7f65000a9330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:52.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.995+0000 7f64ff400640 1 -- 172.21.15.28:0/3428034743 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6500144240 msgr2=0x7f65001edf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:52.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.995+0000 7f64ff400640 1 --2- 172.21.15.28:0/3428034743 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6500144240 0x7f65001edf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:52.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.995+0000 7f64ff400640 1 -- 172.21.15.28:0/3428034743 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6500152350 con 0x7f65000ac5f0 2024-02-21T23:11:52.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.996+0000 7f64ff400640 1 --2- 172.21.15.28:0/3428034743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65000ac5f0 0x7f65000ab4a0 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 crypto rx=0x7f64ec00e830 tx=0x7f64ec00ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:52.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.996+0000 7f64f7e00640 1 -- 172.21.15.28:0/3428034743 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64ec00c8d0 con 0x7f65000ac5f0 2024-02-21T23:11:52.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.996+0000 7f64f7e00640 1 -- 172.21.15.28:0/3428034743 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64ec018070 con 0x7f65000ac5f0 2024-02-21T23:11:52.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.996+0000 7f64f7e00640 1 -- 172.21.15.28:0/3428034743 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64ec00ccb0 con 0x7f65000ac5f0 2024-02-21T23:11:52.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.996+0000 7f6505c00640 1 -- 172.21.15.28:0/3428034743 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65000aace0 con 0x7f65000ac5f0 2024-02-21T23:11:52.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.996+0000 7f6505c00640 1 -- 172.21.15.28:0/3428034743 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65001516e0 con 0x7f65000ac5f0 2024-02-21T23:11:52.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.997+0000 7f6505c00640 1 -- 172.21.15.28:0/3428034743 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f650004f290 con 0x7f65000ac5f0 2024-02-21T23:11:52.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.998+0000 7f64f7e00640 1 -- 172.21.15.28:0/3428034743 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f64ec01d460 con 0x7f65000ac5f0 2024-02-21T23:11:52.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.999+0000 7f64f7e00640 1 --2- 172.21.15.28:0/3428034743 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f64e006b4d0 0x7f64e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:52.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:52.999+0000 7f64f7e00640 1 -- 172.21.15.28:0/3428034743 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f64ec08a2a0 con 0x7f65000ac5f0 2024-02-21T23:11:52.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:53.000+0000 7f64ffe00640 1 --2- 172.21.15.28:0/3428034743 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f64e006b4d0 0x7f64e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:52.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:53.001+0000 7f64ffe00640 1 --2- 172.21.15.28:0/3428034743 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f64e006b4d0 0x7f64e006d990 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f64f0009990 tx=0x7f64f0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:52.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:53.004+0000 7f64f7e00640 1 -- 172.21.15.28:0/3428034743 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f64ec058f80 con 0x7f65000ac5f0 2024-02-21T23:11:53.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:53.249+0000 7f6505c00640 1 -- 172.21.15.28:0/3428034743 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f65000c1f10 con 0x7f65000ac5f0 2024-02-21T23:11:53.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:53.250+0000 7f64f7e00640 1 -- 172.21.15.28:0/3428034743 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 203 v203) v1 ==== 95+0+4950 (secure 0 0 0) 0x7f64ec058880 con 0x7f65000ac5f0 2024-02-21T23:11:53.239 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:11:53.239 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":203,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6848,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3439896042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3439896042},{"type":"v1","addr":"172.21.15.28:6835","nonce":3439896042}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":196},{"gid":7729,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/3213249808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3213249808},{"type":"v1","addr":"172.21.15.28:6833","nonce":3213249808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":195},{"gid":7735,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/2895690310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2895690310},{"type":"v1","addr":"172.21.15.81:6836","nonce":2895690310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":196}],"filesystems":[{"mdsmap":{"epoch":203,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:11:38.384619+0000","modified":"2024-02-21T23:11:47.000014+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0],"up":{"mds_0":7738},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7738":{"gid":7738,"name":"b","rank":0,"incarnation":197,"state":"up:stopping","state_seq":3,"addr":"172.21.15.81:6837/1709165826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":1709165826},{"type":"v1","addr":"172.21.15.81:6837","nonce":1709165826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-02-21T23:11:53.239 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 203 2024-02-21T23:11:53.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:53.252+0000 7f6505c00640 1 -- 172.21.15.28:0/3428034743 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f64e006b4d0 msgr2=0x7f64e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:53.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:53.252+0000 7f6505c00640 1 --2- 172.21.15.28:0/3428034743 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f64e006b4d0 0x7f64e006d990 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f64f0009990 tx=0x7f64f0008040 comp rx=0 tx=0).stop 2024-02-21T23:11:53.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:53.252+0000 7f6505c00640 1 -- 172.21.15.28:0/3428034743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65000ac5f0 msgr2=0x7f65000ab4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:53.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:53.252+0000 7f6505c00640 1 --2- 172.21.15.28:0/3428034743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65000ac5f0 0x7f65000ab4a0 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 crypto rx=0x7f64ec00e830 tx=0x7f64ec00ec30 comp rx=0 tx=0).stop 2024-02-21T23:11:53.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:53.253+0000 7f6505c00640 1 -- 172.21.15.28:0/3428034743 shutdown_connections 2024-02-21T23:11:53.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:53.253+0000 7f6505c00640 1 --2- 172.21.15.28:0/3428034743 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f64e006b4d0 0x7f64e006d990 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:53.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:53.253+0000 7f6505c00640 1 --2- 172.21.15.28:0/3428034743 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6500144240 0x7f65001edf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:53.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:53.253+0000 7f6505c00640 1 --2- 172.21.15.28:0/3428034743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65000ac5f0 0x7f65000ab4a0 unknown :-1 s=CLOSED pgs=2103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:53.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:53.253+0000 7f6505c00640 1 --2- 172.21.15.28:0/3428034743 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f64e40051c0 0x7f65000a9330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:53.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:53.253+0000 7f6505c00640 1 -- 172.21.15.28:0/3428034743 >> 172.21.15.28:0/3428034743 conn(0x7f650006b630 msgr2=0x7f650014b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:53.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:53.253+0000 7f6505c00640 1 -- 172.21.15.28:0/3428034743 shutdown_connections 2024-02-21T23:11:53.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:53.253+0000 7f6505c00640 1 -- 172.21.15.28:0/3428034743 wait complete. 2024-02-21T23:11:53.665 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 203, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:11:38.384619+0000', 'modified': '2024-02-21T23:11:47.000014+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 7738}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7738': {'gid': 7738, 'name': 'b', 'rank': 0, 'incarnation': 197, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.81:6837/1709165826', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6835', 'nonce': 1709165826}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 1709165826}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:11:53.666 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7738:up:stopping 2024-02-21T23:11:54.667 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:11:54.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:54.912+0000 7f120f200640 1 Processor -- start 2024-02-21T23:11:54.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:54.912+0000 7f120f200640 1 -- start start 2024-02-21T23:11:54.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:54.913+0000 7f120f200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f121014f0b0 0x7f1210155180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:54.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:54.914+0000 7f120f200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f12100a6160 0x7f12100c2010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:54.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:54.914+0000 7f120f200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1210155760 con 0x7f12100c2550 2024-02-21T23:11:54.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:54.914+0000 7f120f200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1210151d30 con 0x7f121014f0b0 2024-02-21T23:11:54.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:54.914+0000 7f120f200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1210155a40 con 0x7f12100a6160 2024-02-21T23:11:54.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:54.914+0000 7f120de00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f121014f0b0 0x7f1210155180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:54.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:54.914+0000 7f120e800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f12100c2550 0x7f12101521c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:47036/0 (socket says 172.21.15.28:47036) 2024-02-21T23:11:54.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:54.914+0000 7f120e800640 1 -- 172.21.15.28:0/1938466375 learned_addr learned my addr 172.21.15.28:0/1938466375 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:11:54.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:54.914+0000 7f120d400640 1 --2- 172.21.15.28:0/1938466375 >> v2:172.21.15.81:3300/0 conn(0x7f12100a6160 0x7f12100c2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:54.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:54.914+0000 7f120de00640 1 -- 172.21.15.28:0/1938466375 >> v2:172.21.15.81:3300/0 conn(0x7f12100a6160 msgr2=0x7f12100c2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:54.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:54.914+0000 7f120de00640 1 --2- 172.21.15.28:0/1938466375 >> v2:172.21.15.81:3300/0 conn(0x7f12100a6160 0x7f12100c2010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:54.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:54.914+0000 7f120ca00640 1 -- 172.21.15.28:0/1938466375 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3686666119 0 0) 0x7f1210155760 con 0x7f12100c2550 2024-02-21T23:11:54.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:54.914+0000 7f120de00640 1 -- 172.21.15.28:0/1938466375 >> v1:172.21.15.28:6789/0 conn(0x7f12100c2550 legacy=0x7f12101521c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:54.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:54.914+0000 7f120de00640 1 -- 172.21.15.28:0/1938466375 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1210155d20 con 0x7f121014f0b0 2024-02-21T23:11:54.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:54.915+0000 7f120d400640 1 --2- 172.21.15.28:0/1938466375 >> v2:172.21.15.81:3300/0 conn(0x7f12100a6160 0x7f12100c2010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:11:54.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:54.916+0000 7f120de00640 1 --2- 172.21.15.28:0/1938466375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f121014f0b0 0x7f1210155180 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 crypto rx=0x7f1210151eb0 tx=0x7f120002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da6d8b1c31dd3c48 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:54.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:54.916+0000 7f120ca00640 1 -- 172.21.15.28:0/1938466375 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1200035020 con 0x7f121014f0b0 2024-02-21T23:11:54.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:54.916+0000 7f120ca00640 1 -- 172.21.15.28:0/1938466375 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1200031830 con 0x7f121014f0b0 2024-02-21T23:11:54.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:54.916+0000 7f120ca00640 1 -- 172.21.15.28:0/1938466375 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1200031c00 con 0x7f121014f0b0 2024-02-21T23:11:54.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:54.917+0000 7f120f200640 1 -- 172.21.15.28:0/1938466375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f121014f0b0 msgr2=0x7f1210155180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:54.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:54.917+0000 7f120f200640 1 --2- 172.21.15.28:0/1938466375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f121014f0b0 0x7f1210155180 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 crypto rx=0x7f1210151eb0 tx=0x7f120002b8d0 comp rx=0 tx=0).stop 2024-02-21T23:11:54.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:54.917+0000 7f120f200640 1 -- 172.21.15.28:0/1938466375 shutdown_connections 2024-02-21T23:11:54.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:54.917+0000 7f120f200640 1 --2- 172.21.15.28:0/1938466375 >> v2:172.21.15.81:3300/0 conn(0x7f12100a6160 0x7f12100c2010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:54.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:54.917+0000 7f120f200640 1 --2- 172.21.15.28:0/1938466375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f121014f0b0 0x7f1210155180 unknown :-1 s=CLOSED pgs=2105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:54.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:54.917+0000 7f120f200640 1 -- 172.21.15.28:0/1938466375 >> 172.21.15.28:0/1938466375 conn(0x7f121006b630 msgr2=0x7f12100c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:54.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:54.917+0000 7f120f200640 1 -- 172.21.15.28:0/1938466375 shutdown_connections 2024-02-21T23:11:54.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:54.917+0000 7f120f200640 1 -- 172.21.15.28:0/1938466375 wait complete. 2024-02-21T23:11:54.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:54.918+0000 7f120f200640 1 Processor -- start 2024-02-21T23:11:54.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:54.919+0000 7f120f200640 1 -- start start 2024-02-21T23:11:54.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:54.920+0000 7f120f200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12100a6160 0x7f12101ee3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:54.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:54.920+0000 7f120f200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f12100c2550 0x7f1210146870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:54.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:54.920+0000 7f120f200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f121014f0b0 0x7f1210146db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:54.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:54.920+0000 7f120f200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12100a7f70 con 0x7f12100a6160 2024-02-21T23:11:54.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:54.920+0000 7f120f200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12100a68d0 con 0x7f121014f0b0 2024-02-21T23:11:54.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:54.920+0000 7f120f200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12100a80f0 con 0x7f12100c2550 2024-02-21T23:11:54.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:54.920+0000 7f120de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12100a6160 0x7f12101ee3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:54.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:54.920+0000 7f120de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12100a6160 0x7f12101ee3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60342/0 (socket says 172.21.15.28:60342) 2024-02-21T23:11:54.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:54.920+0000 7f120de00640 1 -- 172.21.15.28:0/719197562 learned_addr learned my addr 172.21.15.28:0/719197562 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:54.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:54.920+0000 7f120e800640 1 --2- 172.21.15.28:0/719197562 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f121014f0b0 0x7f1210146db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:54.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:54.920+0000 7f120de00640 1 -- 172.21.15.28:0/719197562 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f12100c2550 msgr2=0x7f1210146870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:54.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:54.920+0000 7f120d400640 1 --2- 172.21.15.28:0/719197562 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f12100c2550 0x7f1210146870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:54.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:54.920+0000 7f120de00640 1 --2- 172.21.15.28:0/719197562 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f12100c2550 0x7f1210146870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:54.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:54.920+0000 7f120de00640 1 -- 172.21.15.28:0/719197562 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f121014f0b0 msgr2=0x7f1210146db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:54.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:54.920+0000 7f120de00640 1 --2- 172.21.15.28:0/719197562 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f121014f0b0 0x7f1210146db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:54.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:54.920+0000 7f120de00640 1 -- 172.21.15.28:0/719197562 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12101ea660 con 0x7f12100a6160 2024-02-21T23:11:54.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:54.920+0000 7f120e800640 1 --2- 172.21.15.28:0/719197562 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f121014f0b0 0x7f1210146db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:11:54.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:54.920+0000 7f120d400640 1 --2- 172.21.15.28:0/719197562 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f12100c2550 0x7f1210146870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:11:54.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:54.921+0000 7f120de00640 1 --2- 172.21.15.28:0/719197562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12100a6160 0x7f12101ee3a0 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 crypto rx=0x7f120002bee0 tx=0x7f12000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:54.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:54.921+0000 7f1206000640 1 -- 172.21.15.28:0/719197562 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1200041070 con 0x7f12100a6160 2024-02-21T23:11:54.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:54.921+0000 7f1206000640 1 -- 172.21.15.28:0/719197562 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f120003aa30 con 0x7f12100a6160 2024-02-21T23:11:54.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:54.921+0000 7f1206000640 1 -- 172.21.15.28:0/719197562 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1200035040 con 0x7f12100a6160 2024-02-21T23:11:54.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:54.921+0000 7f120f200640 1 -- 172.21.15.28:0/719197562 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12101ea940 con 0x7f12100a6160 2024-02-21T23:11:54.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:54.921+0000 7f120f200640 1 -- 172.21.15.28:0/719197562 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12100c6140 con 0x7f12100a6160 2024-02-21T23:11:54.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:54.922+0000 7f120f200640 1 -- 172.21.15.28:0/719197562 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f121004f290 con 0x7f12100a6160 2024-02-21T23:11:54.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:54.923+0000 7f1206000640 1 -- 172.21.15.28:0/719197562 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1200038020 con 0x7f12100a6160 2024-02-21T23:11:54.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:54.924+0000 7f1206000640 1 --2- 172.21.15.28:0/719197562 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f11f006b500 0x7f11f006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:54.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:54.924+0000 7f1206000640 1 -- 172.21.15.28:0/719197562 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f12000abb70 con 0x7f12100a6160 2024-02-21T23:11:54.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:54.925+0000 7f120d400640 1 --2- 172.21.15.28:0/719197562 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f11f006b500 0x7f11f006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:54.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:54.926+0000 7f120d400640 1 --2- 172.21.15.28:0/719197562 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f11f006b500 0x7f11f006d9c0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f11fc006fd0 tx=0x7f11fc008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:54.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:54.928+0000 7f1206000640 1 -- 172.21.15.28:0/719197562 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f120007a850 con 0x7f12100a6160 2024-02-21T23:11:55.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:55.167+0000 7f120f200640 1 -- 172.21.15.28:0/719197562 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f12100c0be0 con 0x7f12100a6160 2024-02-21T23:11:55.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:55.168+0000 7f1206000640 1 -- 172.21.15.28:0/719197562 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 203 v203) v1 ==== 95+0+4950 (secure 0 0 0) 0x7f120007a150 con 0x7f12100a6160 2024-02-21T23:11:55.157 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:11:55.157 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":203,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6848,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3439896042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3439896042},{"type":"v1","addr":"172.21.15.28:6835","nonce":3439896042}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":196},{"gid":7729,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/3213249808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3213249808},{"type":"v1","addr":"172.21.15.28:6833","nonce":3213249808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":195},{"gid":7735,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/2895690310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2895690310},{"type":"v1","addr":"172.21.15.81:6836","nonce":2895690310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":196}],"filesystems":[{"mdsmap":{"epoch":203,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:11:38.384619+0000","modified":"2024-02-21T23:11:47.000014+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0],"up":{"mds_0":7738},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7738":{"gid":7738,"name":"b","rank":0,"incarnation":197,"state":"up:stopping","state_seq":3,"addr":"172.21.15.81:6837/1709165826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":1709165826},{"type":"v1","addr":"172.21.15.81:6837","nonce":1709165826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-02-21T23:11:55.157 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 203 2024-02-21T23:11:55.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:55.171+0000 7f120f200640 1 -- 172.21.15.28:0/719197562 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f11f006b500 msgr2=0x7f11f006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:55.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:55.171+0000 7f120f200640 1 --2- 172.21.15.28:0/719197562 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f11f006b500 0x7f11f006d9c0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f11fc006fd0 tx=0x7f11fc008040 comp rx=0 tx=0).stop 2024-02-21T23:11:55.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:55.171+0000 7f120f200640 1 -- 172.21.15.28:0/719197562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12100a6160 msgr2=0x7f12101ee3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:55.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:55.171+0000 7f120f200640 1 --2- 172.21.15.28:0/719197562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12100a6160 0x7f12101ee3a0 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 crypto rx=0x7f120002bee0 tx=0x7f12000046c0 comp rx=0 tx=0).stop 2024-02-21T23:11:55.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:55.171+0000 7f120f200640 1 -- 172.21.15.28:0/719197562 shutdown_connections 2024-02-21T23:11:55.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:55.171+0000 7f120f200640 1 --2- 172.21.15.28:0/719197562 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f11f006b500 0x7f11f006d9c0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:55.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:55.171+0000 7f120f200640 1 --2- 172.21.15.28:0/719197562 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f12100c2550 0x7f1210146870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:55.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:55.171+0000 7f120f200640 1 --2- 172.21.15.28:0/719197562 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f121014f0b0 0x7f1210146db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:55.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:55.171+0000 7f120f200640 1 --2- 172.21.15.28:0/719197562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12100a6160 0x7f12101ee3a0 unknown :-1 s=CLOSED pgs=2106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:55.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:55.171+0000 7f120f200640 1 -- 172.21.15.28:0/719197562 >> 172.21.15.28:0/719197562 conn(0x7f121006b630 msgr2=0x7f1210071d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:55.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:55.171+0000 7f120f200640 1 -- 172.21.15.28:0/719197562 shutdown_connections 2024-02-21T23:11:55.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:55.171+0000 7f120f200640 1 -- 172.21.15.28:0/719197562 wait complete. 2024-02-21T23:11:55.582 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 203, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:11:38.384619+0000', 'modified': '2024-02-21T23:11:47.000014+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 7738}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7738': {'gid': 7738, 'name': 'b', 'rank': 0, 'incarnation': 197, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.81:6837/1709165826', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6835', 'nonce': 1709165826}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 1709165826}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:11:55.583 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7738:up:stopping 2024-02-21T23:11:56.584 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:11:56.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:56.823+0000 7f95b1600640 1 Processor -- start 2024-02-21T23:11:56.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:56.823+0000 7f95b1600640 1 -- start start 2024-02-21T23:11:56.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:56.824+0000 7f95b1600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f95ac06dbf0 0x7f95ac158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:56.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:56.824+0000 7f95b1600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f95ac155960 0x7f95ac159000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:56.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:56.824+0000 7f95b1600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95ac0c74c0 con 0x7f95ac0aff90 2024-02-21T23:11:56.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:56.825+0000 7f95b1600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95ac0c7270 con 0x7f95ac155960 2024-02-21T23:11:56.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:56.825+0000 7f95b1600640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95ac158630 con 0x7f95ac06dbf0 2024-02-21T23:11:56.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:56.825+0000 7f95b0c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f95ac155960 0x7f95ac159000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:56.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:56.825+0000 7f95b0c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f95ac155960 0x7f95ac159000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60356/0 (socket says 172.21.15.28:60356) 2024-02-21T23:11:56.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:56.825+0000 7f95b0c00640 1 -- 172.21.15.28:0/921117104 learned_addr learned my addr 172.21.15.28:0/921117104 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:56.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:56.825+0000 7f95a3e00640 1 --2- 172.21.15.28:0/921117104 >> v2:172.21.15.81:3300/0 conn(0x7f95ac06dbf0 0x7f95ac158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:56.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:56.825+0000 7f95b0c00640 1 -- 172.21.15.28:0/921117104 >> v2:172.21.15.81:3300/0 conn(0x7f95ac06dbf0 msgr2=0x7f95ac158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:56.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:56.825+0000 7f95b0c00640 1 --2- 172.21.15.28:0/921117104 >> v2:172.21.15.81:3300/0 conn(0x7f95ac06dbf0 0x7f95ac158ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:56.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:56.825+0000 7f95b0c00640 1 -- 172.21.15.28:0/921117104 >> v1:172.21.15.81:6789/0 conn(0x7f95ac0aff90 legacy=0x7f95ac0c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:56.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:56.826+0000 7f95b0c00640 1 -- 172.21.15.28:0/921117104 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95ac0c77a0 con 0x7f95ac155960 2024-02-21T23:11:56.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:56.827+0000 7f95b0c00640 1 --2- 172.21.15.28:0/921117104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95ac155960 0x7f95ac159000 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 crypto rx=0x7f95ac0cb8c0 tx=0x7f95a402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90359fc02041ccf7 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:56.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:56.827+0000 7f95ab400640 1 -- 172.21.15.28:0/921117104 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95a402ca80 con 0x7f95ac155960 2024-02-21T23:11:56.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:56.827+0000 7f95ab400640 1 -- 172.21.15.28:0/921117104 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95a402cc20 con 0x7f95ac155960 2024-02-21T23:11:56.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:56.827+0000 7f95ab400640 1 -- 172.21.15.28:0/921117104 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95a40343f0 con 0x7f95ac155960 2024-02-21T23:11:56.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:56.828+0000 7f95b1600640 1 -- 172.21.15.28:0/921117104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95ac155960 msgr2=0x7f95ac159000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:56.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:56.828+0000 7f95b1600640 1 --2- 172.21.15.28:0/921117104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95ac155960 0x7f95ac159000 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 crypto rx=0x7f95ac0cb8c0 tx=0x7f95a402bee0 comp rx=0 tx=0).stop 2024-02-21T23:11:56.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:56.828+0000 7f95b1600640 1 -- 172.21.15.28:0/921117104 shutdown_connections 2024-02-21T23:11:56.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:56.828+0000 7f95b1600640 1 --2- 172.21.15.28:0/921117104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95ac155960 0x7f95ac159000 unknown :-1 s=CLOSED pgs=2107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:56.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:56.828+0000 7f95b1600640 1 --2- 172.21.15.28:0/921117104 >> v2:172.21.15.81:3300/0 conn(0x7f95ac06dbf0 0x7f95ac158ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:56.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:56.828+0000 7f95b1600640 1 -- 172.21.15.28:0/921117104 >> 172.21.15.28:0/921117104 conn(0x7f95ac0ac370 msgr2=0x7f95ac0ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:56.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:56.828+0000 7f95b1600640 1 -- 172.21.15.28:0/921117104 shutdown_connections 2024-02-21T23:11:56.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:56.828+0000 7f95b1600640 1 -- 172.21.15.28:0/921117104 wait complete. 2024-02-21T23:11:56.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:56.830+0000 7f95b1600640 1 Processor -- start 2024-02-21T23:11:56.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:56.830+0000 7f95b1600640 1 -- start start 2024-02-21T23:11:56.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:56.831+0000 7f95b1600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95ac06dbf0 0x7f95ac0ae740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:56.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:56.831+0000 7f95b1600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95ac0aff90 0x7f95ac152d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:56.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:56.831+0000 7f95b1600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f95ac155960 0x7f95ac0c38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:56.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:56.831+0000 7f95b1600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95ac0c58f0 con 0x7f95ac0aff90 2024-02-21T23:11:56.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:56.831+0000 7f95b1600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95ac0be0c0 con 0x7f95ac06dbf0 2024-02-21T23:11:56.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:56.831+0000 7f95b1600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95ac0c5a70 con 0x7f95ac155960 2024-02-21T23:11:56.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:56.831+0000 7f95a3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95ac0aff90 0x7f95ac152d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:56.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:56.831+0000 7f95a3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95ac0aff90 0x7f95ac152d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60368/0 (socket says 172.21.15.28:60368) 2024-02-21T23:11:56.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:56.831+0000 7f95a3e00640 1 -- 172.21.15.28:0/3941958325 learned_addr learned my addr 172.21.15.28:0/3941958325 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:56.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:56.831+0000 7f95b0c00640 1 --2- 172.21.15.28:0/3941958325 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f95ac155960 0x7f95ac0c38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:56.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:56.831+0000 7f95abe00640 1 --2- 172.21.15.28:0/3941958325 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95ac06dbf0 0x7f95ac0ae740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:56.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:56.832+0000 7f95a3e00640 1 -- 172.21.15.28:0/3941958325 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f95ac155960 msgr2=0x7f95ac0c38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:56.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:56.832+0000 7f95a3e00640 1 --2- 172.21.15.28:0/3941958325 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f95ac155960 0x7f95ac0c38b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:56.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:56.832+0000 7f95a3e00640 1 -- 172.21.15.28:0/3941958325 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95ac06dbf0 msgr2=0x7f95ac0ae740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:56.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:56.832+0000 7f95a3e00640 1 --2- 172.21.15.28:0/3941958325 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95ac06dbf0 0x7f95ac0ae740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:56.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:56.832+0000 7f95a3e00640 1 -- 172.21.15.28:0/3941958325 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95ac0af480 con 0x7f95ac0aff90 2024-02-21T23:11:56.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:56.832+0000 7f95abe00640 1 --2- 172.21.15.28:0/3941958325 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95ac06dbf0 0x7f95ac0ae740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:11:56.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:56.832+0000 7f95a3e00640 1 --2- 172.21.15.28:0/3941958325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95ac0aff90 0x7f95ac152d80 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7f959400d7d0 tx=0x7f959400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:56.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:56.832+0000 7f95a8c00640 1 -- 172.21.15.28:0/3941958325 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f959400b6c0 con 0x7f95ac0aff90 2024-02-21T23:11:56.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:56.832+0000 7f95a8c00640 1 -- 172.21.15.28:0/3941958325 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9594019070 con 0x7f95ac0aff90 2024-02-21T23:11:56.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:56.832+0000 7f95a8c00640 1 -- 172.21.15.28:0/3941958325 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f959400ba80 con 0x7f95ac0aff90 2024-02-21T23:11:56.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:56.832+0000 7f95b1600640 1 -- 172.21.15.28:0/3941958325 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95ac14f370 con 0x7f95ac0aff90 2024-02-21T23:11:56.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:56.832+0000 7f95b1600640 1 -- 172.21.15.28:0/3941958325 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95ac14ebe0 con 0x7f95ac0aff90 2024-02-21T23:11:56.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:56.834+0000 7f95b1600640 1 -- 172.21.15.28:0/3941958325 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95ac04f290 con 0x7f95ac0aff90 2024-02-21T23:11:56.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:56.835+0000 7f95a8c00640 1 -- 172.21.15.28:0/3941958325 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f959400bc20 con 0x7f95ac0aff90 2024-02-21T23:11:56.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:56.835+0000 7f95a8c00640 1 --2- 172.21.15.28:0/3941958325 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f958806b4d0 0x7f958806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:56.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:56.836+0000 7f95a8c00640 1 -- 172.21.15.28:0/3941958325 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f959408a7e0 con 0x7f95ac0aff90 2024-02-21T23:11:56.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:56.836+0000 7f95abe00640 1 --2- 172.21.15.28:0/3941958325 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f958806b4d0 0x7f958806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:56.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:56.837+0000 7f95abe00640 1 --2- 172.21.15.28:0/3941958325 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f958806b4d0 0x7f958806d990 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f959c007fd0 tx=0x7f959c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:56.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:56.840+0000 7f95a8c00640 1 -- 172.21.15.28:0/3941958325 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f95940594c0 con 0x7f95ac0aff90 2024-02-21T23:11:57.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:57.086+0000 7f95b1600640 1 -- 172.21.15.28:0/3941958325 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f95ac1eb9b0 con 0x7f95ac0aff90 2024-02-21T23:11:57.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:57.088+0000 7f95a8c00640 1 -- 172.21.15.28:0/3941958325 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 203 v203) v1 ==== 95+0+4950 (secure 0 0 0) 0x7f9594058dc0 con 0x7f95ac0aff90 2024-02-21T23:11:57.077 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:11:57.077 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":203,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6848,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3439896042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3439896042},{"type":"v1","addr":"172.21.15.28:6835","nonce":3439896042}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":196},{"gid":7729,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/3213249808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3213249808},{"type":"v1","addr":"172.21.15.28:6833","nonce":3213249808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":195},{"gid":7735,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/2895690310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2895690310},{"type":"v1","addr":"172.21.15.81:6836","nonce":2895690310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":196}],"filesystems":[{"mdsmap":{"epoch":203,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:11:38.384619+0000","modified":"2024-02-21T23:11:47.000014+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0],"up":{"mds_0":7738},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7738":{"gid":7738,"name":"b","rank":0,"incarnation":197,"state":"up:stopping","state_seq":3,"addr":"172.21.15.81:6837/1709165826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":1709165826},{"type":"v1","addr":"172.21.15.81:6837","nonce":1709165826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-02-21T23:11:57.077 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 203 2024-02-21T23:11:57.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:57.091+0000 7f95b1600640 1 -- 172.21.15.28:0/3941958325 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f958806b4d0 msgr2=0x7f958806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:57.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:57.091+0000 7f95b1600640 1 --2- 172.21.15.28:0/3941958325 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f958806b4d0 0x7f958806d990 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f959c007fd0 tx=0x7f959c009040 comp rx=0 tx=0).stop 2024-02-21T23:11:57.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:57.091+0000 7f95b1600640 1 -- 172.21.15.28:0/3941958325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95ac0aff90 msgr2=0x7f95ac152d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:57.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:57.091+0000 7f95b1600640 1 --2- 172.21.15.28:0/3941958325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95ac0aff90 0x7f95ac152d80 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7f959400d7d0 tx=0x7f959400dc90 comp rx=0 tx=0).stop 2024-02-21T23:11:57.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:57.091+0000 7f95b1600640 1 -- 172.21.15.28:0/3941958325 shutdown_connections 2024-02-21T23:11:57.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:57.091+0000 7f95b1600640 1 --2- 172.21.15.28:0/3941958325 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f958806b4d0 0x7f958806d990 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:57.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:57.091+0000 7f95b1600640 1 --2- 172.21.15.28:0/3941958325 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f95ac155960 0x7f95ac0c38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:57.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:57.091+0000 7f95b1600640 1 --2- 172.21.15.28:0/3941958325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95ac0aff90 0x7f95ac152d80 unknown :-1 s=CLOSED pgs=2108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:57.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:57.091+0000 7f95b1600640 1 --2- 172.21.15.28:0/3941958325 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f95ac06dbf0 0x7f95ac0ae740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:57.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:57.091+0000 7f95b1600640 1 -- 172.21.15.28:0/3941958325 >> 172.21.15.28:0/3941958325 conn(0x7f95ac0ac370 msgr2=0x7f95ac0c50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:57.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:57.091+0000 7f95b1600640 1 -- 172.21.15.28:0/3941958325 shutdown_connections 2024-02-21T23:11:57.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:57.091+0000 7f95b1600640 1 -- 172.21.15.28:0/3941958325 wait complete. 2024-02-21T23:11:57.503 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 203, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:11:38.384619+0000', 'modified': '2024-02-21T23:11:47.000014+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 7738}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7738': {'gid': 7738, 'name': 'b', 'rank': 0, 'incarnation': 197, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.81:6837/1709165826', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6835', 'nonce': 1709165826}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 1709165826}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:11:57.504 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7738:up:stopping 2024-02-21T23:11:58.504 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:11:58.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.744+0000 7fe3a5800640 1 Processor -- start 2024-02-21T23:11:58.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.744+0000 7fe3a5800640 1 -- start start 2024-02-21T23:11:58.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.746+0000 7fe3a5800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe3a0153950 0x7fe3a00c15d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:58.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.746+0000 7fe3a5800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fe3a015b6d0 0x7fe3a015dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:58.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.746+0000 7fe3a5800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3a0152e70 con 0x7fe3a00c11f0 2024-02-21T23:11:58.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.746+0000 7fe3a5800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3a00a5ec0 con 0x7fe3a015b6d0 2024-02-21T23:11:58.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.746+0000 7fe3a5800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a0153150 con 0x7fe3a0153950 2024-02-21T23:11:58.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.746+0000 7fe39ea00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe3a00c11f0 0x7fe3a006dce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:47040/0 (socket says 172.21.15.28:47040) 2024-02-21T23:11:58.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.746+0000 7fe39ea00640 1 -- 172.21.15.28:0/20874880 learned_addr learned my addr 172.21.15.28:0/20874880 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:11:58.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.747+0000 7fe397e00640 1 --2- 172.21.15.28:0/20874880 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe3a0153950 0x7fe3a00c15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:58.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.747+0000 7fe39f400640 1 --2- 172.21.15.28:0/20874880 >> v2:172.21.15.81:3300/0 conn(0x7fe3a015b6d0 0x7fe3a015dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:58.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.747+0000 7fe39e000640 1 -- 172.21.15.28:0/20874880 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3841031078 0 0) 0x7fe3a0152e70 con 0x7fe3a00c11f0 2024-02-21T23:11:58.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.747+0000 7fe39e000640 1 -- 172.21.15.28:0/20874880 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe384003520 con 0x7fe3a00c11f0 2024-02-21T23:11:58.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.747+0000 7fe39e000640 1 -- 172.21.15.28:0/20874880 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3205923011 0 0) 0x7fe384003520 con 0x7fe3a00c11f0 2024-02-21T23:11:58.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.747+0000 7fe39e000640 1 -- 172.21.15.28:0/20874880 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe3a0153950 msgr2=0x7fe3a00c15d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:58.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.747+0000 7fe39e000640 1 --2- 172.21.15.28:0/20874880 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe3a0153950 0x7fe3a00c15d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:58.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.747+0000 7fe39e000640 1 -- 172.21.15.28:0/20874880 >> v2:172.21.15.81:3300/0 conn(0x7fe3a015b6d0 msgr2=0x7fe3a015dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:58.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.747+0000 7fe39e000640 1 --2- 172.21.15.28:0/20874880 >> v2:172.21.15.81:3300/0 conn(0x7fe3a015b6d0 0x7fe3a015dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:58.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.747+0000 7fe39e000640 1 -- 172.21.15.28:0/20874880 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3a0153430 con 0x7fe3a00c11f0 2024-02-21T23:11:58.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.747+0000 7fe39e000640 1 -- 172.21.15.28:0/20874880 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fe390001670 con 0x7fe3a00c11f0 2024-02-21T23:11:58.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.747+0000 7fe39e000640 1 -- 172.21.15.28:0/20874880 >> v1:172.21.15.28:6789/0 conn(0x7fe3a00c11f0 legacy=0x7fe3a006dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:58.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.747+0000 7fe39e000640 1 --2- 172.21.15.28:0/20874880 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe3a0153950 0x7fe3840039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:58.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.747+0000 7fe39e000640 1 --2- 172.21.15.28:0/20874880 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe384003f80 0x7fe384004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:58.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.747+0000 7fe39e000640 1 --2- 172.21.15.28:0/20874880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe384004d40 0x7fe3840051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:58.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.747+0000 7fe39e000640 1 -- 172.21.15.28:0/20874880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a00a5ec0 con 0x7fe384004d40 2024-02-21T23:11:58.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.747+0000 7fe39e000640 1 -- 172.21.15.28:0/20874880 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a0153150 con 0x7fe384003f80 2024-02-21T23:11:58.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.747+0000 7fe39e000640 1 -- 172.21.15.28:0/20874880 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe38400ba70 con 0x7fe3a0153950 2024-02-21T23:11:58.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.748+0000 7fe39f400640 1 --2- 172.21.15.28:0/20874880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe384004d40 0x7fe3840051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:58.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.748+0000 7fe397e00640 1 --2- 172.21.15.28:0/20874880 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe384003f80 0x7fe384004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:58.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.748+0000 7fe39ea00640 1 --2- 172.21.15.28:0/20874880 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe3a0153950 0x7fe3840039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:58.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.748+0000 7fe39f400640 1 -- 172.21.15.28:0/20874880 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe3a0153950 msgr2=0x7fe3840039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:58.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.748+0000 7fe39f400640 1 --2- 172.21.15.28:0/20874880 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe3a0153950 0x7fe3840039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:58.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.748+0000 7fe39f400640 1 -- 172.21.15.28:0/20874880 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe384003f80 msgr2=0x7fe384004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:11:58.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.748+0000 7fe39f400640 1 --2- 172.21.15.28:0/20874880 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe384003f80 0x7fe384004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:58.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.748+0000 7fe39f400640 1 -- 172.21.15.28:0/20874880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe384003520 con 0x7fe384004d40 2024-02-21T23:11:58.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.748+0000 7fe397e00640 1 --2- 172.21.15.28:0/20874880 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe384003f80 0x7fe384004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:11:58.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.748+0000 7fe39ea00640 1 --2- 172.21.15.28:0/20874880 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe3a0153950 0x7fe3840039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:11:58.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.749+0000 7fe39f400640 1 --2- 172.21.15.28:0/20874880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe384004d40 0x7fe3840051c0 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7fe398009f90 tx=0x7fe384005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d871cc80cccd22 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:58.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.750+0000 7fe39e000640 1 -- 172.21.15.28:0/20874880 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe398039070 con 0x7fe384004d40 2024-02-21T23:11:58.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.750+0000 7fe39e000640 1 -- 172.21.15.28:0/20874880 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe39802cc20 con 0x7fe384004d40 2024-02-21T23:11:58.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.750+0000 7fe39e000640 1 -- 172.21.15.28:0/20874880 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe398009c10 con 0x7fe384004d40 2024-02-21T23:11:58.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.750+0000 7fe3a5800640 1 -- 172.21.15.28:0/20874880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe384004d40 msgr2=0x7fe3840051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:58.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.750+0000 7fe3a5800640 1 --2- 172.21.15.28:0/20874880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe384004d40 0x7fe3840051c0 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7fe398009f90 tx=0x7fe384005830 comp rx=0 tx=0).stop 2024-02-21T23:11:58.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.750+0000 7fe39f400640 1 -- 172.21.15.28:0/20874880 reap_dead start 2024-02-21T23:11:58.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.750+0000 7fe3a5800640 1 -- 172.21.15.28:0/20874880 shutdown_connections 2024-02-21T23:11:58.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.750+0000 7fe3a5800640 1 -- 172.21.15.28:0/20874880 >> 172.21.15.28:0/20874880 conn(0x7fe3a00c67f0 msgr2=0x7fe3a00c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:58.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.751+0000 7fe3a5800640 1 -- 172.21.15.28:0/20874880 shutdown_connections 2024-02-21T23:11:58.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.751+0000 7fe3a5800640 1 -- 172.21.15.28:0/20874880 wait complete. 2024-02-21T23:11:58.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.752+0000 7fe3a5800640 1 Processor -- start 2024-02-21T23:11:58.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.753+0000 7fe3a5800640 1 -- start start 2024-02-21T23:11:58.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.754+0000 7fe3a5800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe384004d40 0x7fe3a0146ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:58.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.754+0000 7fe3a5800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3a0147a90 0x7fe3a0143f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:58.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.754+0000 7fe3a5800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe3a0144bb0 0x7fe3a014e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:58.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.754+0000 7fe3a5800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a00a6110 con 0x7fe384004d40 2024-02-21T23:11:58.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.754+0000 7fe3a5800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a00c4c00 con 0x7fe3a0147a90 2024-02-21T23:11:58.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.754+0000 7fe3a5800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a00c0d90 con 0x7fe3a0144bb0 2024-02-21T23:11:58.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.754+0000 7fe39ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe384004d40 0x7fe3a0146ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:58.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.754+0000 7fe39ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe384004d40 0x7fe3a0146ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60384/0 (socket says 172.21.15.28:60384) 2024-02-21T23:11:58.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.754+0000 7fe39ea00640 1 -- 172.21.15.28:0/3189172909 learned_addr learned my addr 172.21.15.28:0/3189172909 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:11:58.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.755+0000 7fe39f400640 1 --2- 172.21.15.28:0/3189172909 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe3a0144bb0 0x7fe3a014e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:58.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.755+0000 7fe397e00640 1 --2- 172.21.15.28:0/3189172909 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3a0147a90 0x7fe3a0143f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:58.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.755+0000 7fe39ea00640 1 -- 172.21.15.28:0/3189172909 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe3a0144bb0 msgr2=0x7fe3a014e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:58.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.755+0000 7fe39ea00640 1 --2- 172.21.15.28:0/3189172909 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe3a0144bb0 0x7fe3a014e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:58.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.755+0000 7fe39ea00640 1 -- 172.21.15.28:0/3189172909 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3a0147a90 msgr2=0x7fe3a0143f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:58.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.755+0000 7fe39ea00640 1 --2- 172.21.15.28:0/3189172909 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3a0147a90 0x7fe3a0143f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:58.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.755+0000 7fe39ea00640 1 -- 172.21.15.28:0/3189172909 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3a01e98b0 con 0x7fe384004d40 2024-02-21T23:11:58.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.755+0000 7fe39ea00640 1 --2- 172.21.15.28:0/3189172909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe384004d40 0x7fe3a0146ec0 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7fe39000d8b0 tx=0x7fe39000dd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:58.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.755+0000 7fe396a00640 1 -- 172.21.15.28:0/3189172909 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe390019070 con 0x7fe384004d40 2024-02-21T23:11:58.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.756+0000 7fe396a00640 1 -- 172.21.15.28:0/3189172909 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3900074e0 con 0x7fe384004d40 2024-02-21T23:11:58.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.756+0000 7fe396a00640 1 -- 172.21.15.28:0/3189172909 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe390004a00 con 0x7fe384004d40 2024-02-21T23:11:58.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.756+0000 7fe3a5800640 1 -- 172.21.15.28:0/3189172909 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3a0143320 con 0x7fe384004d40 2024-02-21T23:11:58.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.756+0000 7fe3a5800640 1 -- 172.21.15.28:0/3189172909 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3a01e95d0 con 0x7fe384004d40 2024-02-21T23:11:58.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.757+0000 7fe3a5800640 1 -- 172.21.15.28:0/3189172909 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3a0067f30 con 0x7fe384004d40 2024-02-21T23:11:58.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.758+0000 7fe396a00640 1 -- 172.21.15.28:0/3189172909 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe39001c070 con 0x7fe384004d40 2024-02-21T23:11:58.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.759+0000 7fe396a00640 1 --2- 172.21.15.28:0/3189172909 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe37c06b4d0 0x7fe37c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:11:58.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.759+0000 7fe396a00640 1 -- 172.21.15.28:0/3189172909 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe390089f40 con 0x7fe384004d40 2024-02-21T23:11:58.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.759+0000 7fe397e00640 1 --2- 172.21.15.28:0/3189172909 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe37c06b4d0 0x7fe37c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:11:58.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.761+0000 7fe397e00640 1 --2- 172.21.15.28:0/3189172909 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe37c06b4d0 0x7fe37c06d990 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fe3a0144830 tx=0x7fe388008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:11:58.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:58.764+0000 7fe396a00640 1 -- 172.21.15.28:0/3189172909 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe390058c20 con 0x7fe384004d40 2024-02-21T23:11:58.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:59.003+0000 7fe3a5800640 1 -- 172.21.15.28:0/3189172909 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe3a0146c20 con 0x7fe384004d40 2024-02-21T23:11:58.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:59.004+0000 7fe396a00640 1 -- 172.21.15.28:0/3189172909 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 203 v203) v1 ==== 95+0+4950 (secure 0 0 0) 0x7fe390058520 con 0x7fe384004d40 2024-02-21T23:11:58.993 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:11:58.993 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":203,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6848,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3439896042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3439896042},{"type":"v1","addr":"172.21.15.28:6835","nonce":3439896042}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":196},{"gid":7729,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/3213249808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3213249808},{"type":"v1","addr":"172.21.15.28:6833","nonce":3213249808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":195},{"gid":7735,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/2895690310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2895690310},{"type":"v1","addr":"172.21.15.81:6836","nonce":2895690310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":196}],"filesystems":[{"mdsmap":{"epoch":203,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:11:38.384619+0000","modified":"2024-02-21T23:11:47.000014+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0],"up":{"mds_0":7738},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7738":{"gid":7738,"name":"b","rank":0,"incarnation":197,"state":"up:stopping","state_seq":3,"addr":"172.21.15.81:6837/1709165826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":1709165826},{"type":"v1","addr":"172.21.15.81:6837","nonce":1709165826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-02-21T23:11:58.993 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 203 2024-02-21T23:11:58.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:59.006+0000 7fe3a5800640 1 -- 172.21.15.28:0/3189172909 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe37c06b4d0 msgr2=0x7fe37c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:58.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:59.007+0000 7fe3a5800640 1 --2- 172.21.15.28:0/3189172909 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe37c06b4d0 0x7fe37c06d990 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fe3a0144830 tx=0x7fe388008040 comp rx=0 tx=0).stop 2024-02-21T23:11:58.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:59.007+0000 7fe3a5800640 1 -- 172.21.15.28:0/3189172909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe384004d40 msgr2=0x7fe3a0146ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:11:58.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:59.007+0000 7fe3a5800640 1 --2- 172.21.15.28:0/3189172909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe384004d40 0x7fe3a0146ec0 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7fe39000d8b0 tx=0x7fe39000dd70 comp rx=0 tx=0).stop 2024-02-21T23:11:58.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:59.007+0000 7fe3a5800640 1 -- 172.21.15.28:0/3189172909 shutdown_connections 2024-02-21T23:11:58.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:59.007+0000 7fe3a5800640 1 --2- 172.21.15.28:0/3189172909 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe37c06b4d0 0x7fe37c06d990 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:58.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:59.007+0000 7fe3a5800640 1 --2- 172.21.15.28:0/3189172909 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe3a0144bb0 0x7fe3a014e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:58.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:59.007+0000 7fe3a5800640 1 --2- 172.21.15.28:0/3189172909 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe3a0147a90 0x7fe3a0143f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:58.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:59.007+0000 7fe3a5800640 1 --2- 172.21.15.28:0/3189172909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe384004d40 0x7fe3a0146ec0 unknown :-1 s=CLOSED pgs=2111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:11:58.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:59.007+0000 7fe3a5800640 1 -- 172.21.15.28:0/3189172909 >> 172.21.15.28:0/3189172909 conn(0x7fe3a00c67f0 msgr2=0x7fe3a00c5bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:11:58.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:59.007+0000 7fe3a5800640 1 -- 172.21.15.28:0/3189172909 shutdown_connections 2024-02-21T23:11:58.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:11:59.007+0000 7fe3a5800640 1 -- 172.21.15.28:0/3189172909 wait complete. 2024-02-21T23:11:59.049 INFO:tasks.ceph.mds.b.smithi081.stderr: -643> 2024-02-21T23:11:44.894+0000 7f45e4800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-21T23:11:59.124 INFO:tasks.ceph.mds.b.smithi081.stdout:starting mds.b at 2024-02-21T23:11:59.417 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 203, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:11:38.384619+0000', 'modified': '2024-02-21T23:11:47.000014+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0], 'up': {'mds_0': 7738}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7738': {'gid': 7738, 'name': 'b', 'rank': 0, 'incarnation': 197, 'state': 'up:stopping', 'state_seq': 3, 'addr': '172.21.15.81:6837/1709165826', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6835', 'nonce': 1709165826}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 1709165826}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:11:59.417 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7738:up:stopping 2024-02-21T23:12:00.419 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:12:00.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.658+0000 7f343f200640 1 Processor -- start 2024-02-21T23:12:00.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.658+0000 7f343f200640 1 -- start start 2024-02-21T23:12:00.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.660+0000 7f343f200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f34400c0fe0 0x7f344006e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:00.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.660+0000 7f343f200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f3440158580 0x7f344014b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:00.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.660+0000 7f343f200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34400a8b40 con 0x7f344006dff0 2024-02-21T23:12:00.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.660+0000 7f343f200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34400a8e00 con 0x7f34400c0fe0 2024-02-21T23:12:00.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.660+0000 7f343f200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34400a90e0 con 0x7f3440158580 2024-02-21T23:12:00.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.660+0000 7f343de00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f344006dff0 0x7f3440071350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:47052/0 (socket says 172.21.15.28:47052) 2024-02-21T23:12:00.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.660+0000 7f343de00640 1 -- 172.21.15.28:0/1923404179 learned_addr learned my addr 172.21.15.28:0/1923404179 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:12:00.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.660+0000 7f343d400640 1 --2- 172.21.15.28:0/1923404179 >> v2:172.21.15.28:3300/0 conn(0x7f34400c0fe0 0x7f344006e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:00.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.661+0000 7f343e800640 1 --2- 172.21.15.28:0/1923404179 >> v2:172.21.15.81:3300/0 conn(0x7f3440158580 0x7f344014b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:00.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.661+0000 7f343d400640 1 -- 172.21.15.28:0/1923404179 >> v2:172.21.15.81:3300/0 conn(0x7f3440158580 msgr2=0x7f344014b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:00.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.661+0000 7f343d400640 1 --2- 172.21.15.28:0/1923404179 >> v2:172.21.15.81:3300/0 conn(0x7f3440158580 0x7f344014b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:00.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.661+0000 7f343d400640 1 -- 172.21.15.28:0/1923404179 >> v1:172.21.15.28:6789/0 conn(0x7f344006dff0 legacy=0x7f3440071350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:00.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.661+0000 7f343d400640 1 -- 172.21.15.28:0/1923404179 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3440072ed0 con 0x7f34400c0fe0 2024-02-21T23:12:00.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.662+0000 7f343d400640 1 --2- 172.21.15.28:0/1923404179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34400c0fe0 0x7f344006e3d0 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 crypto rx=0x7f344006f270 tx=0x7f342c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b17a7592f7af2e1 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:00.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.662+0000 7f343ca00640 1 -- 172.21.15.28:0/1923404179 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f342c035020 con 0x7f34400c0fe0 2024-02-21T23:12:00.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.662+0000 7f343ca00640 1 -- 172.21.15.28:0/1923404179 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f342c031830 con 0x7f34400c0fe0 2024-02-21T23:12:00.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.662+0000 7f343ca00640 1 -- 172.21.15.28:0/1923404179 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f342c031c00 con 0x7f34400c0fe0 2024-02-21T23:12:00.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.663+0000 7f343f200640 1 -- 172.21.15.28:0/1923404179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34400c0fe0 msgr2=0x7f344006e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:00.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.663+0000 7f343f200640 1 --2- 172.21.15.28:0/1923404179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34400c0fe0 0x7f344006e3d0 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 crypto rx=0x7f344006f270 tx=0x7f342c02b8d0 comp rx=0 tx=0).stop 2024-02-21T23:12:00.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.663+0000 7f343f200640 1 -- 172.21.15.28:0/1923404179 shutdown_connections 2024-02-21T23:12:00.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.663+0000 7f343f200640 1 --2- 172.21.15.28:0/1923404179 >> v2:172.21.15.81:3300/0 conn(0x7f3440158580 0x7f344014b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:00.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.663+0000 7f343f200640 1 --2- 172.21.15.28:0/1923404179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34400c0fe0 0x7f344006e3d0 unknown :-1 s=CLOSED pgs=2114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:00.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.663+0000 7f343f200640 1 -- 172.21.15.28:0/1923404179 >> 172.21.15.28:0/1923404179 conn(0x7f344006b630 msgr2=0x7f34400c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:00.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.663+0000 7f343f200640 1 -- 172.21.15.28:0/1923404179 shutdown_connections 2024-02-21T23:12:00.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.663+0000 7f343f200640 1 -- 172.21.15.28:0/1923404179 wait complete. 2024-02-21T23:12:00.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.665+0000 7f343f200640 1 Processor -- start 2024-02-21T23:12:00.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.665+0000 7f343f200640 1 -- start start 2024-02-21T23:12:00.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.666+0000 7f343f200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f344006dff0 0x7f34401eeac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:00.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.666+0000 7f343f200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f34400c0fe0 0x7f3440154a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:00.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.666+0000 7f343f200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3440158580 0x7f3440156db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:00.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.666+0000 7f343f200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3440158980 con 0x7f3440158580 2024-02-21T23:12:00.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.666+0000 7f343f200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3440073520 con 0x7f34400c0fe0 2024-02-21T23:12:00.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.666+0000 7f343f200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3440158b00 con 0x7f344006dff0 2024-02-21T23:12:00.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.666+0000 7f343e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3440158580 0x7f3440156db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:00.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.666+0000 7f343e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3440158580 0x7f3440156db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60398/0 (socket says 172.21.15.28:60398) 2024-02-21T23:12:00.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.666+0000 7f343e800640 1 -- 172.21.15.28:0/3599715183 learned_addr learned my addr 172.21.15.28:0/3599715183 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:00.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.667+0000 7f343d400640 1 --2- 172.21.15.28:0/3599715183 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f34400c0fe0 0x7f3440154a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:00.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.667+0000 7f343de00640 1 --2- 172.21.15.28:0/3599715183 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f344006dff0 0x7f34401eeac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:00.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.667+0000 7f343e800640 1 -- 172.21.15.28:0/3599715183 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f344006dff0 msgr2=0x7f34401eeac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:00.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.667+0000 7f343e800640 1 --2- 172.21.15.28:0/3599715183 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f344006dff0 0x7f34401eeac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:00.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.667+0000 7f343e800640 1 -- 172.21.15.28:0/3599715183 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f34400c0fe0 msgr2=0x7f3440154a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:00.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.667+0000 7f343e800640 1 --2- 172.21.15.28:0/3599715183 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f34400c0fe0 0x7f3440154a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:00.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.667+0000 7f343e800640 1 -- 172.21.15.28:0/3599715183 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34401eac50 con 0x7f3440158580 2024-02-21T23:12:00.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.667+0000 7f343de00640 1 --2- 172.21.15.28:0/3599715183 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f344006dff0 0x7f34401eeac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:12:00.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.667+0000 7f343e800640 1 --2- 172.21.15.28:0/3599715183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3440158580 0x7f3440156db0 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 crypto rx=0x7f3438007e70 tx=0x7f343800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:00.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.667+0000 7f3436000640 1 -- 172.21.15.28:0/3599715183 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3438014620 con 0x7f3440158580 2024-02-21T23:12:00.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.667+0000 7f3436000640 1 -- 172.21.15.28:0/3599715183 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3438019070 con 0x7f3440158580 2024-02-21T23:12:00.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.667+0000 7f343f200640 1 -- 172.21.15.28:0/3599715183 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3440152e50 con 0x7f3440158580 2024-02-21T23:12:00.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.667+0000 7f3436000640 1 -- 172.21.15.28:0/3599715183 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3438014a20 con 0x7f3440158580 2024-02-21T23:12:00.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.667+0000 7f343f200640 1 -- 172.21.15.28:0/3599715183 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34401eaef0 con 0x7f3440158580 2024-02-21T23:12:00.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.669+0000 7f343f200640 1 -- 172.21.15.28:0/3599715183 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34400c2340 con 0x7f3440158580 2024-02-21T23:12:00.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.670+0000 7f3436000640 1 -- 172.21.15.28:0/3599715183 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3438014bc0 con 0x7f3440158580 2024-02-21T23:12:00.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.671+0000 7f3436000640 1 --2- 172.21.15.28:0/3599715183 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f342006b4d0 0x7f342006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:00.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.671+0000 7f3436000640 1 -- 172.21.15.28:0/3599715183 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f343808ac00 con 0x7f3440158580 2024-02-21T23:12:00.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.671+0000 7f343de00640 1 --2- 172.21.15.28:0/3599715183 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f342006b4d0 0x7f342006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:00.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.672+0000 7f343de00640 1 --2- 172.21.15.28:0/3599715183 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f342006b4d0 0x7f342006d990 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f344014b7b0 tx=0x7f3430009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:00.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.675+0000 7f3436000640 1 -- 172.21.15.28:0/3599715183 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f34400c2340 con 0x7f3440158580 2024-02-21T23:12:00.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.917+0000 7f343f200640 1 -- 172.21.15.28:0/3599715183 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f344004f290 con 0x7f3440158580 2024-02-21T23:12:00.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.918+0000 7f3436000640 1 -- 172.21.15.28:0/3599715183 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 205 v205) v1 ==== 95+0+4935 (secure 0 0 0) 0x7f3438091080 con 0x7f3440158580 2024-02-21T23:12:00.906 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:12:00.907 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":205,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6848,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3439896042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3439896042},{"type":"v1","addr":"172.21.15.28:6835","nonce":3439896042}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":196},{"gid":7729,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/3213249808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3213249808},{"type":"v1","addr":"172.21.15.28:6833","nonce":3213249808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":195},{"gid":7735,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/2895690310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2895690310},{"type":"v1","addr":"172.21.15.81:6836","nonce":2895690310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":196},{"gid":7867,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/416021079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":416021079},{"type":"v1","addr":"172.21.15.81:6837","nonce":416021079}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":205}],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:11:38.384619+0000","modified":"2024-02-21T23:11:58.473381+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[],"up":{},"failed":[],"damaged":[],"stopped":[0],"info":{},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-02-21T23:12:00.907 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 205 2024-02-21T23:12:00.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.920+0000 7f343f200640 1 -- 172.21.15.28:0/3599715183 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f342006b4d0 msgr2=0x7f342006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:00.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.920+0000 7f343f200640 1 --2- 172.21.15.28:0/3599715183 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f342006b4d0 0x7f342006d990 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f344014b7b0 tx=0x7f3430009040 comp rx=0 tx=0).stop 2024-02-21T23:12:00.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.920+0000 7f343f200640 1 -- 172.21.15.28:0/3599715183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3440158580 msgr2=0x7f3440156db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:00.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.920+0000 7f343f200640 1 --2- 172.21.15.28:0/3599715183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3440158580 0x7f3440156db0 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 crypto rx=0x7f3438007e70 tx=0x7f343800e4b0 comp rx=0 tx=0).stop 2024-02-21T23:12:00.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.921+0000 7f343f200640 1 -- 172.21.15.28:0/3599715183 shutdown_connections 2024-02-21T23:12:00.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.921+0000 7f343f200640 1 --2- 172.21.15.28:0/3599715183 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f342006b4d0 0x7f342006d990 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:00.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.921+0000 7f343f200640 1 --2- 172.21.15.28:0/3599715183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3440158580 0x7f3440156db0 unknown :-1 s=CLOSED pgs=2115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:00.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.921+0000 7f343f200640 1 --2- 172.21.15.28:0/3599715183 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f34400c0fe0 0x7f3440154a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:00.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.921+0000 7f343f200640 1 --2- 172.21.15.28:0/3599715183 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f344006dff0 0x7f34401eeac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:00.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.921+0000 7f343f200640 1 -- 172.21.15.28:0/3599715183 >> 172.21.15.28:0/3599715183 conn(0x7f344006b630 msgr2=0x7f34400745b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:00.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.921+0000 7f343f200640 1 -- 172.21.15.28:0/3599715183 shutdown_connections 2024-02-21T23:12:00.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:00.921+0000 7f343f200640 1 -- 172.21.15.28:0/3599715183 wait complete. 2024-02-21T23:12:01.331 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 204, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:11:38.384619+0000', 'modified': '2024-02-21T23:11:58.473381+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [], 'up': {}, 'failed': [], 'damaged': [], 'stopped': [0], 'info': {}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:12:01.332 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/0 2024-02-21T23:12:01.332 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:12:01.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.575+0000 7fc0b2600640 1 Processor -- start 2024-02-21T23:12:01.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.575+0000 7fc0b2600640 1 -- start start 2024-02-21T23:12:01.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.576+0000 7fc0b2600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc0ac06ee40 0x7fc0ac0a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:01.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.576+0000 7fc0b2600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc0ac156af0 0x7fc0ac06dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:01.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.576+0000 7fc0b2600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0ac0c2760 con 0x7fc0ac150450 2024-02-21T23:12:01.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.576+0000 7fc0b2600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0ac0c22c0 con 0x7fc0ac156af0 2024-02-21T23:12:01.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.576+0000 7fc0b2600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0ac156660 con 0x7fc0ac06ee40 2024-02-21T23:12:01.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.577+0000 7fc0b1c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc0ac156af0 0x7fc0ac06dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:01.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.577+0000 7fc0b1c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc0ac156af0 0x7fc0ac06dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60404/0 (socket says 172.21.15.28:60404) 2024-02-21T23:12:01.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.577+0000 7fc0b1c00640 1 -- 172.21.15.28:0/928695094 learned_addr learned my addr 172.21.15.28:0/928695094 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:01.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.577+0000 7fc0abe00640 1 --2- 172.21.15.28:0/928695094 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc0ac06ee40 0x7fc0ac0a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:01.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.577+0000 7fc0b1c00640 1 -- 172.21.15.28:0/928695094 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc0ac06ee40 msgr2=0x7fc0ac0a5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:01.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.577+0000 7fc0b1c00640 1 --2- 172.21.15.28:0/928695094 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc0ac06ee40 0x7fc0ac0a5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:01.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.577+0000 7fc0b1c00640 1 -- 172.21.15.28:0/928695094 >> v1:172.21.15.81:6789/0 conn(0x7fc0ac150450 legacy=0x7fc0ac1536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:01.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.577+0000 7fc0b1c00640 1 -- 172.21.15.28:0/928695094 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0ac156310 con 0x7fc0ac156af0 2024-02-21T23:12:01.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.578+0000 7fc0b1c00640 1 --2- 172.21.15.28:0/928695094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0ac156af0 0x7fc0ac06dff0 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 crypto rx=0x7fc0a4009e50 tx=0x7fc0a402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80887ece005a60ef server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:01.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.579+0000 7fc0ab400640 1 -- 172.21.15.28:0/928695094 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a402ca70 con 0x7fc0ac156af0 2024-02-21T23:12:01.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.579+0000 7fc0ab400640 1 -- 172.21.15.28:0/928695094 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0a402cc10 con 0x7fc0ac156af0 2024-02-21T23:12:01.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.579+0000 7fc0ab400640 1 -- 172.21.15.28:0/928695094 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a40343f0 con 0x7fc0ac156af0 2024-02-21T23:12:01.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.579+0000 7fc0b2600640 1 -- 172.21.15.28:0/928695094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0ac156af0 msgr2=0x7fc0ac06dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:01.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.579+0000 7fc0b2600640 1 --2- 172.21.15.28:0/928695094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0ac156af0 0x7fc0ac06dff0 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 crypto rx=0x7fc0a4009e50 tx=0x7fc0a402bed0 comp rx=0 tx=0).stop 2024-02-21T23:12:01.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.579+0000 7fc0b2600640 1 -- 172.21.15.28:0/928695094 shutdown_connections 2024-02-21T23:12:01.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.579+0000 7fc0b2600640 1 --2- 172.21.15.28:0/928695094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0ac156af0 0x7fc0ac06dff0 unknown :-1 s=CLOSED pgs=2116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:01.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.579+0000 7fc0b2600640 1 --2- 172.21.15.28:0/928695094 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc0ac06ee40 0x7fc0ac0a5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:01.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.579+0000 7fc0b2600640 1 -- 172.21.15.28:0/928695094 >> 172.21.15.28:0/928695094 conn(0x7fc0ac06b630 msgr2=0x7fc0ac0c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:01.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.580+0000 7fc0b2600640 1 -- 172.21.15.28:0/928695094 shutdown_connections 2024-02-21T23:12:01.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.580+0000 7fc0b2600640 1 -- 172.21.15.28:0/928695094 wait complete. 2024-02-21T23:12:01.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.581+0000 7fc0b2600640 1 Processor -- start 2024-02-21T23:12:01.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.581+0000 7fc0b2600640 1 -- start start 2024-02-21T23:12:01.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.582+0000 7fc0b2600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0ac06ee40 0x7fc0ac145690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:01.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.582+0000 7fc0b2600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc0ac150450 0x7fc0ac144690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:01.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.582+0000 7fc0b2600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0ac156af0 0x7fc0ac144bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:01.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.582+0000 7fc0b2600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0ac156f80 con 0x7fc0ac156af0 2024-02-21T23:12:01.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.582+0000 7fc0b2600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0ac0c2cb0 con 0x7fc0ac06ee40 2024-02-21T23:12:01.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.582+0000 7fc0b2600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0ac157100 con 0x7fc0ac150450 2024-02-21T23:12:01.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.582+0000 7fc0b1c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0ac156af0 0x7fc0ac144bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:01.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.582+0000 7fc0b1c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0ac156af0 0x7fc0ac144bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60414/0 (socket says 172.21.15.28:60414) 2024-02-21T23:12:01.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.582+0000 7fc0b1c00640 1 -- 172.21.15.28:0/454420480 learned_addr learned my addr 172.21.15.28:0/454420480 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:01.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.583+0000 7fc0b1200640 1 --2- 172.21.15.28:0/454420480 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0ac06ee40 0x7fc0ac145690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:01.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.583+0000 7fc0abe00640 1 --2- 172.21.15.28:0/454420480 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc0ac150450 0x7fc0ac144690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:01.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.583+0000 7fc0b1c00640 1 -- 172.21.15.28:0/454420480 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc0ac150450 msgr2=0x7fc0ac144690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:01.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.583+0000 7fc0b1c00640 1 --2- 172.21.15.28:0/454420480 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc0ac150450 0x7fc0ac144690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:01.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.583+0000 7fc0b1c00640 1 -- 172.21.15.28:0/454420480 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0ac06ee40 msgr2=0x7fc0ac145690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:01.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.583+0000 7fc0b1c00640 1 --2- 172.21.15.28:0/454420480 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0ac06ee40 0x7fc0ac145690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:01.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.583+0000 7fc0b1c00640 1 -- 172.21.15.28:0/454420480 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0ac0ab220 con 0x7fc0ac156af0 2024-02-21T23:12:01.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.583+0000 7fc0b1c00640 1 --2- 172.21.15.28:0/454420480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0ac156af0 0x7fc0ac144bd0 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 crypto rx=0x7fc0a4009f70 tx=0x7fc0a4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:01.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.584+0000 7fc0a8c00640 1 -- 172.21.15.28:0/454420480 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a4044020 con 0x7fc0ac156af0 2024-02-21T23:12:01.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.584+0000 7fc0a8c00640 1 -- 172.21.15.28:0/454420480 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0a4040670 con 0x7fc0ac156af0 2024-02-21T23:12:01.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.584+0000 7fc0a8c00640 1 -- 172.21.15.28:0/454420480 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a4009510 con 0x7fc0ac156af0 2024-02-21T23:12:01.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.584+0000 7fc0b2600640 1 -- 172.21.15.28:0/454420480 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0ac0ab500 con 0x7fc0ac156af0 2024-02-21T23:12:01.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.584+0000 7fc0b2600640 1 -- 172.21.15.28:0/454420480 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0ac0c40f0 con 0x7fc0ac156af0 2024-02-21T23:12:01.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.585+0000 7fc0b2600640 1 -- 172.21.15.28:0/454420480 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0ac04f290 con 0x7fc0ac156af0 2024-02-21T23:12:01.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.586+0000 7fc0a8c00640 1 -- 172.21.15.28:0/454420480 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc0a4047020 con 0x7fc0ac156af0 2024-02-21T23:12:01.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.587+0000 7fc0a8c00640 1 --2- 172.21.15.28:0/454420480 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc08c06b500 0x7fc08c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:01.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.587+0000 7fc0a8c00640 1 -- 172.21.15.28:0/454420480 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc0a40acc20 con 0x7fc0ac156af0 2024-02-21T23:12:01.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.587+0000 7fc0b1200640 1 --2- 172.21.15.28:0/454420480 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc08c06b500 0x7fc08c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:01.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.589+0000 7fc0b1200640 1 --2- 172.21.15.28:0/454420480 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc08c06b500 0x7fc08c06d9c0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fc0a0004ea0 tx=0x7fc0a0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:01.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.591+0000 7fc0a8c00640 1 -- 172.21.15.28:0/454420480 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc0a404e070 con 0x7fc0ac156af0 2024-02-21T23:12:01.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.831+0000 7fc0b2600640 1 -- 172.21.15.28:0/454420480 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc0ac0c1f10 con 0x7fc0ac156af0 2024-02-21T23:12:01.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.832+0000 7fc0a8c00640 1 -- 172.21.15.28:0/454420480 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 205 v205) v1 ==== 95+0+4935 (secure 0 0 0) 0x7fc0a407b990 con 0x7fc0ac156af0 2024-02-21T23:12:01.820 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:12:01.820 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":205,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6848,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3439896042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3439896042},{"type":"v1","addr":"172.21.15.28:6835","nonce":3439896042}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":196},{"gid":7729,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/3213249808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3213249808},{"type":"v1","addr":"172.21.15.28:6833","nonce":3213249808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":195},{"gid":7735,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/2895690310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2895690310},{"type":"v1","addr":"172.21.15.81:6836","nonce":2895690310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":196},{"gid":7867,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/416021079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":416021079},{"type":"v1","addr":"172.21.15.81:6837","nonce":416021079}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":205}],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:11:38.384619+0000","modified":"2024-02-21T23:11:58.473381+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[],"up":{},"failed":[],"damaged":[],"stopped":[0],"info":{},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-02-21T23:12:01.820 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 205 2024-02-21T23:12:01.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.835+0000 7fc0b2600640 1 -- 172.21.15.28:0/454420480 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc08c06b500 msgr2=0x7fc08c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:01.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.835+0000 7fc0b2600640 1 --2- 172.21.15.28:0/454420480 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc08c06b500 0x7fc08c06d9c0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fc0a0004ea0 tx=0x7fc0a0009040 comp rx=0 tx=0).stop 2024-02-21T23:12:01.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.835+0000 7fc0b2600640 1 -- 172.21.15.28:0/454420480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0ac156af0 msgr2=0x7fc0ac144bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:01.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.835+0000 7fc0b2600640 1 --2- 172.21.15.28:0/454420480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0ac156af0 0x7fc0ac144bd0 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 crypto rx=0x7fc0a4009f70 tx=0x7fc0a4004a40 comp rx=0 tx=0).stop 2024-02-21T23:12:01.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.835+0000 7fc0b2600640 1 -- 172.21.15.28:0/454420480 shutdown_connections 2024-02-21T23:12:01.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.835+0000 7fc0b2600640 1 --2- 172.21.15.28:0/454420480 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc08c06b500 0x7fc08c06d9c0 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:01.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.835+0000 7fc0b2600640 1 --2- 172.21.15.28:0/454420480 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc0ac150450 0x7fc0ac144690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:01.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.835+0000 7fc0b2600640 1 --2- 172.21.15.28:0/454420480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0ac156af0 0x7fc0ac144bd0 unknown :-1 s=CLOSED pgs=2117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:01.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.835+0000 7fc0b2600640 1 --2- 172.21.15.28:0/454420480 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0ac06ee40 0x7fc0ac145690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:01.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.835+0000 7fc0b2600640 1 -- 172.21.15.28:0/454420480 >> 172.21.15.28:0/454420480 conn(0x7fc0ac06b630 msgr2=0x7fc0ac0a74d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:01.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.835+0000 7fc0b2600640 1 -- 172.21.15.28:0/454420480 shutdown_connections 2024-02-21T23:12:01.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:01.835+0000 7fc0b2600640 1 -- 172.21.15.28:0/454420480 wait complete. 2024-02-21T23:12:01.841 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2024-02-21T23:12:02.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.062+0000 7feb4d200640 1 Processor -- start 2024-02-21T23:12:02.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.063+0000 7feb4d200640 1 -- start start 2024-02-21T23:12:02.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.064+0000 7feb4d200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feb48152650 0x7feb480c8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:02.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.064+0000 7feb4d200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7feb481550e0 0x7feb48155550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:02.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.064+0000 7feb4d200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb48158920 con 0x7feb4806daa0 2024-02-21T23:12:02.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.064+0000 7feb4d200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb481586d0 con 0x7feb481550e0 2024-02-21T23:12:02.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.064+0000 7feb4d200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb48158c00 con 0x7feb48152650 2024-02-21T23:12:02.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.065+0000 7feb47400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feb48152650 0x7feb480c8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:02.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.065+0000 7feb47400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feb48152650 0x7feb480c8ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3301/0 says I am v2:172.21.15.28:55596/0 (socket says 172.21.15.28:55596) 2024-02-21T23:12:02.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.065+0000 7feb47400640 1 -- 172.21.15.28:0/990276629 learned_addr learned my addr 172.21.15.28:0/990276629 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:02.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.065+0000 7feb47e00640 1 --2- 172.21.15.28:0/990276629 >> v2:172.21.15.81:3300/0 conn(0x7feb481550e0 0x7feb48155550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:02.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.066+0000 7feb47e00640 1 -- 172.21.15.28:0/990276629 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feb48152650 msgr2=0x7feb480c8ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:02.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.066+0000 7feb47e00640 1 --2- 172.21.15.28:0/990276629 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feb48152650 0x7feb480c8ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:02.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.066+0000 7feb47e00640 1 -- 172.21.15.28:0/990276629 >> v1:172.21.15.81:6789/0 conn(0x7feb4806daa0 legacy=0x7feb480c7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:02.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.066+0000 7feb47e00640 1 -- 172.21.15.28:0/990276629 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb48158ee0 con 0x7feb481550e0 2024-02-21T23:12:02.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.066+0000 7feb47400640 1 --2- 172.21.15.28:0/990276629 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feb48152650 0x7feb480c8ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-21T23:12:02.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.067+0000 7feb47e00640 1 --2- 172.21.15.28:0/990276629 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feb481550e0 0x7feb48155550 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7feb40009d90 tx=0x7feb4002be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb8a346e4ecb3589 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:02.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.074+0000 7feb46a00640 1 -- 172.21.15.28:0/990276629 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb40039070 con 0x7feb481550e0 2024-02-21T23:12:02.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.074+0000 7feb46a00640 1 -- 172.21.15.28:0/990276629 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb4002cbd0 con 0x7feb481550e0 2024-02-21T23:12:02.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.074+0000 7feb46a00640 1 -- 172.21.15.28:0/990276629 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb40034440 con 0x7feb481550e0 2024-02-21T23:12:02.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.074+0000 7feb4d200640 1 -- 172.21.15.28:0/990276629 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feb481550e0 msgr2=0x7feb48155550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:02.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.074+0000 7feb4d200640 1 --2- 172.21.15.28:0/990276629 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feb481550e0 0x7feb48155550 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7feb40009d90 tx=0x7feb4002be20 comp rx=0 tx=0).stop 2024-02-21T23:12:02.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.074+0000 7feb4d200640 1 -- 172.21.15.28:0/990276629 shutdown_connections 2024-02-21T23:12:02.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.074+0000 7feb4d200640 1 --2- 172.21.15.28:0/990276629 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feb481550e0 0x7feb48155550 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:02.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.074+0000 7feb4d200640 1 --2- 172.21.15.28:0/990276629 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feb48152650 0x7feb480c8ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:02.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.074+0000 7feb4d200640 1 -- 172.21.15.28:0/990276629 >> 172.21.15.28:0/990276629 conn(0x7feb48072ce0 msgr2=0x7feb480ac340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:02.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.074+0000 7feb4d200640 1 -- 172.21.15.28:0/990276629 shutdown_connections 2024-02-21T23:12:02.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.075+0000 7feb4d200640 1 -- 172.21.15.28:0/990276629 wait complete. 2024-02-21T23:12:02.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.076+0000 7feb4d200640 1 Processor -- start 2024-02-21T23:12:02.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.077+0000 7feb4d200640 1 -- start start 2024-02-21T23:12:02.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.077+0000 7feb4d200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb4806daa0 0x7feb48151260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:02.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.077+0000 7feb4d200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feb48152650 0x7feb481eece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:02.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.078+0000 7feb4d200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feb481550e0 0x7feb481ef220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:02.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.078+0000 7feb4d200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb48152a30 con 0x7feb4806daa0 2024-02-21T23:12:02.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.078+0000 7feb4d200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb480bdfb0 con 0x7feb48152650 2024-02-21T23:12:02.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.078+0000 7feb4d200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb480c7870 con 0x7feb481550e0 2024-02-21T23:12:02.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.078+0000 7feb47400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb4806daa0 0x7feb48151260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:02.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.078+0000 7feb47400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb4806daa0 0x7feb48151260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60428/0 (socket says 172.21.15.28:60428) 2024-02-21T23:12:02.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.078+0000 7feb47400640 1 -- 172.21.15.28:0/3831378100 learned_addr learned my addr 172.21.15.28:0/3831378100 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:02.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.078+0000 7feb47e00640 1 --2- 172.21.15.28:0/3831378100 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feb481550e0 0x7feb481ef220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:02.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.078+0000 7feb3fe00640 1 --2- 172.21.15.28:0/3831378100 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feb48152650 0x7feb481eece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:02.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.079+0000 7feb47400640 1 -- 172.21.15.28:0/3831378100 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feb481550e0 msgr2=0x7feb481ef220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:02.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.079+0000 7feb47400640 1 --2- 172.21.15.28:0/3831378100 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feb481550e0 0x7feb481ef220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:02.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.079+0000 7feb47400640 1 -- 172.21.15.28:0/3831378100 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feb48152650 msgr2=0x7feb481eece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:02.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.079+0000 7feb47400640 1 --2- 172.21.15.28:0/3831378100 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feb48152650 0x7feb481eece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:02.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.079+0000 7feb47400640 1 -- 172.21.15.28:0/3831378100 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb480c4100 con 0x7feb4806daa0 2024-02-21T23:12:02.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.079+0000 7feb47400640 1 --2- 172.21.15.28:0/3831378100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb4806daa0 0x7feb48151260 secure :-1 s=READY pgs=2118 cs=0 l=1 rev1=1 crypto rx=0x7feb3800e9d0 tx=0x7feb3800ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:02.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.079+0000 7feb3f400640 1 -- 172.21.15.28:0/3831378100 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb38017070 con 0x7feb4806daa0 2024-02-21T23:12:02.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.079+0000 7feb3f400640 1 -- 172.21.15.28:0/3831378100 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb3800ca00 con 0x7feb4806daa0 2024-02-21T23:12:02.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.079+0000 7feb3f400640 1 -- 172.21.15.28:0/3831378100 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb3800cdb0 con 0x7feb4806daa0 2024-02-21T23:12:02.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.079+0000 7feb4d200640 1 -- 172.21.15.28:0/3831378100 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb4814d760 con 0x7feb4806daa0 2024-02-21T23:12:02.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.080+0000 7feb4d200640 1 -- 172.21.15.28:0/3831378100 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb4814dd20 con 0x7feb4806daa0 2024-02-21T23:12:02.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.081+0000 7feb4d200640 1 -- 172.21.15.28:0/3831378100 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb4804f290 con 0x7feb4806daa0 2024-02-21T23:12:02.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.082+0000 7feb3f400640 1 -- 172.21.15.28:0/3831378100 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feb3801a070 con 0x7feb4806daa0 2024-02-21T23:12:02.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.083+0000 7feb3f400640 1 --2- 172.21.15.28:0/3831378100 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7feb2406b4d0 0x7feb2406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:02.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.083+0000 7feb3f400640 1 -- 172.21.15.28:0/3831378100 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6101+0+0 (secure 0 0 0) 0x7feb3808a2a0 con 0x7feb4806daa0 2024-02-21T23:12:02.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.083+0000 7feb3fe00640 1 --2- 172.21.15.28:0/3831378100 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7feb2406b4d0 0x7feb2406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:02.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.084+0000 7feb3fe00640 1 --2- 172.21.15.28:0/3831378100 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7feb2406b4d0 0x7feb2406d990 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7feb480ac3a0 tx=0x7feb30009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:02.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.087+0000 7feb3f400640 1 -- 172.21.15.28:0/3831378100 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7feb38058f80 con 0x7feb4806daa0 2024-02-21T23:12:02.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.329+0000 7feb4d200640 1 -- 172.21.15.28:0/3831378100 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) v1 -- 0x7feb48156080 con 0x7feb4806daa0 2024-02-21T23:12:02.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.581+0000 7feb3f400640 1 -- 172.21.15.28:0/3831378100 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v206) v1 ==== 105+0+0 (secure 0 0 0) 0x7feb38058880 con 0x7feb4806daa0 2024-02-21T23:12:02.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.584+0000 7feb4d200640 1 -- 172.21.15.28:0/3831378100 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7feb2406b4d0 msgr2=0x7feb2406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:02.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.584+0000 7feb4d200640 1 --2- 172.21.15.28:0/3831378100 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7feb2406b4d0 0x7feb2406d990 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7feb480ac3a0 tx=0x7feb30009040 comp rx=0 tx=0).stop 2024-02-21T23:12:02.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.585+0000 7feb4d200640 1 -- 172.21.15.28:0/3831378100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb4806daa0 msgr2=0x7feb48151260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:02.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.585+0000 7feb4d200640 1 --2- 172.21.15.28:0/3831378100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb4806daa0 0x7feb48151260 secure :-1 s=READY pgs=2118 cs=0 l=1 rev1=1 crypto rx=0x7feb3800e9d0 tx=0x7feb3800ee90 comp rx=0 tx=0).stop 2024-02-21T23:12:02.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.585+0000 7feb4d200640 1 -- 172.21.15.28:0/3831378100 shutdown_connections 2024-02-21T23:12:02.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.585+0000 7feb4d200640 1 --2- 172.21.15.28:0/3831378100 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7feb2406b4d0 0x7feb2406d990 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:02.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.585+0000 7feb4d200640 1 --2- 172.21.15.28:0/3831378100 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feb481550e0 0x7feb481ef220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:02.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.585+0000 7feb4d200640 1 --2- 172.21.15.28:0/3831378100 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feb48152650 0x7feb481eece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:02.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.585+0000 7feb4d200640 1 --2- 172.21.15.28:0/3831378100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb4806daa0 0x7feb48151260 unknown :-1 s=CLOSED pgs=2118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:02.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.585+0000 7feb4d200640 1 -- 172.21.15.28:0/3831378100 >> 172.21.15.28:0/3831378100 conn(0x7feb48072ce0 msgr2=0x7feb480c93b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:02.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.585+0000 7feb4d200640 1 -- 172.21.15.28:0/3831378100 shutdown_connections 2024-02-21T23:12:02.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.585+0000 7feb4d200640 1 -- 172.21.15.28:0/3831378100 wait complete. 2024-02-21T23:12:02.594 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:12:02.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.823+0000 7faa34a00640 1 Processor -- start 2024-02-21T23:12:02.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.823+0000 7faa34a00640 1 -- start start 2024-02-21T23:12:02.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.825+0000 7faa34a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7faa300c11f0 0x7faa30153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:02.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.825+0000 7faa34a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faa3015b6d0 0x7faa3015dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:02.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.825+0000 7faa34a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa30152e70 con 0x7faa300c15d0 2024-02-21T23:12:02.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.825+0000 7faa34a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa300c1a60 con 0x7faa300c11f0 2024-02-21T23:12:02.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.825+0000 7faa34a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa300a5ec0 con 0x7faa3015b6d0 2024-02-21T23:12:02.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.825+0000 7faa2de00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7faa300c11f0 0x7faa30153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:02.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.825+0000 7faa2de00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7faa300c11f0 0x7faa30153950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60438/0 (socket says 172.21.15.28:60438) 2024-02-21T23:12:02.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.825+0000 7faa2de00640 1 -- 172.21.15.28:0/1305468724 learned_addr learned my addr 172.21.15.28:0/1305468724 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:02.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.825+0000 7faa2e800640 1 --2- 172.21.15.28:0/1305468724 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faa3015b6d0 0x7faa3015dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:02.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.826+0000 7faa2de00640 1 -- 172.21.15.28:0/1305468724 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faa3015b6d0 msgr2=0x7faa3015dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:02.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.826+0000 7faa2de00640 1 --2- 172.21.15.28:0/1305468724 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faa3015b6d0 0x7faa3015dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:02.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.826+0000 7faa2de00640 1 -- 172.21.15.28:0/1305468724 >> v1:172.21.15.81:6789/0 conn(0x7faa300c15d0 legacy=0x7faa3006dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:02.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.826+0000 7faa2de00640 1 -- 172.21.15.28:0/1305468724 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa30153150 con 0x7faa300c11f0 2024-02-21T23:12:02.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.827+0000 7faa2de00640 1 --2- 172.21.15.28:0/1305468724 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa300c11f0 0x7faa30153950 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 crypto rx=0x7faa20009870 tx=0x7faa2002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a571ddcf1777806 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:02.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.828+0000 7faa2d400640 1 -- 172.21.15.28:0/1305468724 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa200317c0 con 0x7faa300c11f0 2024-02-21T23:12:02.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.828+0000 7faa2d400640 1 -- 172.21.15.28:0/1305468724 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa20037070 con 0x7faa300c11f0 2024-02-21T23:12:02.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.828+0000 7faa2d400640 1 -- 172.21.15.28:0/1305468724 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa20031b80 con 0x7faa300c11f0 2024-02-21T23:12:02.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.829+0000 7faa34a00640 1 -- 172.21.15.28:0/1305468724 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa300c11f0 msgr2=0x7faa30153950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:02.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.829+0000 7faa34a00640 1 --2- 172.21.15.28:0/1305468724 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa300c11f0 0x7faa30153950 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 crypto rx=0x7faa20009870 tx=0x7faa2002b860 comp rx=0 tx=0).stop 2024-02-21T23:12:02.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.829+0000 7faa34a00640 1 -- 172.21.15.28:0/1305468724 shutdown_connections 2024-02-21T23:12:02.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.829+0000 7faa34a00640 1 --2- 172.21.15.28:0/1305468724 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faa3015b6d0 0x7faa3015dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:02.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.829+0000 7faa34a00640 1 --2- 172.21.15.28:0/1305468724 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa300c11f0 0x7faa30153950 unknown :-1 s=CLOSED pgs=2119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:02.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.829+0000 7faa34a00640 1 -- 172.21.15.28:0/1305468724 >> 172.21.15.28:0/1305468724 conn(0x7faa300c67f0 msgr2=0x7faa300c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:02.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.829+0000 7faa34a00640 1 -- 172.21.15.28:0/1305468724 shutdown_connections 2024-02-21T23:12:02.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.829+0000 7faa34a00640 1 -- 172.21.15.28:0/1305468724 wait complete. 2024-02-21T23:12:02.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.831+0000 7faa34a00640 1 Processor -- start 2024-02-21T23:12:02.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.831+0000 7faa34a00640 1 -- start start 2024-02-21T23:12:02.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.832+0000 7faa34a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faa300c11f0 0x7faa30071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:02.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.832+0000 7faa34a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faa300c15d0 0x7faa300a63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:02.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.832+0000 7faa34a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa3015b6d0 0x7faa301eb790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:02.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.832+0000 7faa34a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa300c4c00 con 0x7faa3015b6d0 2024-02-21T23:12:02.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.832+0000 7faa34a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa3014d1c0 con 0x7faa300c11f0 2024-02-21T23:12:02.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.832+0000 7faa34a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa300a6110 con 0x7faa300c15d0 2024-02-21T23:12:02.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.832+0000 7faa2e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa3015b6d0 0x7faa301eb790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:02.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.832+0000 7faa2e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa3015b6d0 0x7faa301eb790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60440/0 (socket says 172.21.15.28:60440) 2024-02-21T23:12:02.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.832+0000 7faa2e800640 1 -- 172.21.15.28:0/601138610 learned_addr learned my addr 172.21.15.28:0/601138610 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:02.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.832+0000 7faa2de00640 1 --2- 172.21.15.28:0/601138610 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faa300c11f0 0x7faa30071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:02.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.832+0000 7faa27e00640 1 --2- 172.21.15.28:0/601138610 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faa300c15d0 0x7faa300a63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:02.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.833+0000 7faa2e800640 1 -- 172.21.15.28:0/601138610 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faa300c15d0 msgr2=0x7faa300a63d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:02.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.833+0000 7faa2e800640 1 --2- 172.21.15.28:0/601138610 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faa300c15d0 0x7faa300a63d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:02.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.833+0000 7faa2e800640 1 -- 172.21.15.28:0/601138610 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faa300c11f0 msgr2=0x7faa30071380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:02.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.833+0000 7faa2e800640 1 --2- 172.21.15.28:0/601138610 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faa300c11f0 0x7faa30071380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:02.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.833+0000 7faa2e800640 1 -- 172.21.15.28:0/601138610 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa30152230 con 0x7faa3015b6d0 2024-02-21T23:12:02.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.833+0000 7faa27e00640 1 --2- 172.21.15.28:0/601138610 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faa300c15d0 0x7faa300a63d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:12:02.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.833+0000 7faa2e800640 1 --2- 172.21.15.28:0/601138610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa3015b6d0 0x7faa301eb790 secure :-1 s=READY pgs=2120 cs=0 l=1 rev1=1 crypto rx=0x7faa28007e90 tx=0x7faa2800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:02.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.833+0000 7faa26000640 1 -- 172.21.15.28:0/601138610 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa28014420 con 0x7faa3015b6d0 2024-02-21T23:12:02.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.833+0000 7faa26000640 1 -- 172.21.15.28:0/601138610 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa28019070 con 0x7faa3015b6d0 2024-02-21T23:12:02.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.833+0000 7faa34a00640 1 -- 172.21.15.28:0/601138610 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa300a6910 con 0x7faa3015b6d0 2024-02-21T23:12:02.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.833+0000 7faa26000640 1 -- 172.21.15.28:0/601138610 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa28014800 con 0x7faa3015b6d0 2024-02-21T23:12:02.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.834+0000 7faa34a00640 1 -- 172.21.15.28:0/601138610 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa30072e60 con 0x7faa3015b6d0 2024-02-21T23:12:02.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.835+0000 7faa34a00640 1 -- 172.21.15.28:0/601138610 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa30067f30 con 0x7faa3015b6d0 2024-02-21T23:12:02.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.836+0000 7faa26000640 1 -- 172.21.15.28:0/601138610 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faa280149a0 con 0x7faa3015b6d0 2024-02-21T23:12:02.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.837+0000 7faa26000640 1 --2- 172.21.15.28:0/601138610 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7faa0c06b500 0x7faa0c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:02.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.837+0000 7faa26000640 1 -- 172.21.15.28:0/601138610 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6101+0+0 (secure 0 0 0) 0x7faa2808ab70 con 0x7faa3015b6d0 2024-02-21T23:12:02.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.838+0000 7faa2de00640 1 --2- 172.21.15.28:0/601138610 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7faa0c06b500 0x7faa0c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:02.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.839+0000 7faa2de00640 1 --2- 172.21.15.28:0/601138610 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7faa0c06b500 0x7faa0c06d9c0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7faa20036000 tx=0x7faa2003d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:02.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:02.842+0000 7faa26000640 1 -- 172.21.15.28:0/601138610 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7faa30152230 con 0x7faa3015b6d0 2024-02-21T23:12:03.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:03.094+0000 7faa34a00640 1 -- 172.21.15.28:0/601138610 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faa3006f0e0 con 0x7faa3015b6d0 2024-02-21T23:12:03.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:03.095+0000 7faa26000640 1 -- 172.21.15.28:0/601138610 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 206 v206) v1 ==== 95+0+4947 (secure 0 0 0) 0x7faa28059860 con 0x7faa3015b6d0 2024-02-21T23:12:03.083 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:12:03.083 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":206,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6848,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3439896042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3439896042},{"type":"v1","addr":"172.21.15.28:6835","nonce":3439896042}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":196},{"gid":7729,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/3213249808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3213249808},{"type":"v1","addr":"172.21.15.28:6833","nonce":3213249808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":195},{"gid":7735,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/2895690310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2895690310},{"type":"v1","addr":"172.21.15.81:6836","nonce":2895690310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":196}],"filesystems":[{"mdsmap":{"epoch":206,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:11:38.384619+0000","modified":"2024-02-21T23:12:02.578054+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0],"up":{"mds_0":7867},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7867":{"gid":7867,"name":"b","rank":0,"incarnation":206,"state":"up:starting","state_seq":1,"addr":"172.21.15.81:6837/416021079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":416021079},{"type":"v1","addr":"172.21.15.81:6837","nonce":416021079}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-02-21T23:12:03.083 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 206 2024-02-21T23:12:03.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:03.097+0000 7faa34a00640 1 -- 172.21.15.28:0/601138610 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7faa0c06b500 msgr2=0x7faa0c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:03.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:03.097+0000 7faa34a00640 1 --2- 172.21.15.28:0/601138610 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7faa0c06b500 0x7faa0c06d9c0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7faa20036000 tx=0x7faa2003d040 comp rx=0 tx=0).stop 2024-02-21T23:12:03.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:03.098+0000 7faa34a00640 1 -- 172.21.15.28:0/601138610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa3015b6d0 msgr2=0x7faa301eb790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:03.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:03.098+0000 7faa34a00640 1 --2- 172.21.15.28:0/601138610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa3015b6d0 0x7faa301eb790 secure :-1 s=READY pgs=2120 cs=0 l=1 rev1=1 crypto rx=0x7faa28007e90 tx=0x7faa2800e4b0 comp rx=0 tx=0).stop 2024-02-21T23:12:03.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:03.098+0000 7faa34a00640 1 -- 172.21.15.28:0/601138610 shutdown_connections 2024-02-21T23:12:03.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:03.098+0000 7faa34a00640 1 --2- 172.21.15.28:0/601138610 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7faa0c06b500 0x7faa0c06d9c0 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:03.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:03.098+0000 7faa34a00640 1 --2- 172.21.15.28:0/601138610 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faa300c15d0 0x7faa300a63d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:03.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:03.098+0000 7faa34a00640 1 --2- 172.21.15.28:0/601138610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa3015b6d0 0x7faa301eb790 unknown :-1 s=CLOSED pgs=2120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:03.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:03.098+0000 7faa34a00640 1 --2- 172.21.15.28:0/601138610 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faa300c11f0 0x7faa30071380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:03.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:03.098+0000 7faa34a00640 1 -- 172.21.15.28:0/601138610 >> 172.21.15.28:0/601138610 conn(0x7faa300c67f0 msgr2=0x7faa3014a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:03.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:03.098+0000 7faa34a00640 1 -- 172.21.15.28:0/601138610 shutdown_connections 2024-02-21T23:12:03.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:03.098+0000 7faa34a00640 1 -- 172.21.15.28:0/601138610 wait complete. 2024-02-21T23:12:03.507 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 206, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:11:38.384619+0000', 'modified': '2024-02-21T23:12:02.578054+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 2, 'in': [0], 'up': {'mds_0': 7867}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7867': {'gid': 7867, 'name': 'b', 'rank': 0, 'incarnation': 206, 'state': 'up:starting', 'state_seq': 1, 'addr': '172.21.15.81:6837/416021079', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6835', 'nonce': 416021079}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 416021079}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:12:03.508 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7867:up:starting 2024-02-21T23:12:04.511 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:12:04.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:04.753+0000 7fd836600640 1 Processor -- start 2024-02-21T23:12:04.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:04.753+0000 7fd836600640 1 -- start start 2024-02-21T23:12:04.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:04.755+0000 7fd836600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd8300c11f0 0x7fd830153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:04.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:04.755+0000 7fd836600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fd8300c15d0 0x7fd83006dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:04.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:04.755+0000 7fd836600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8300c1a30 con 0x7fd830156350 2024-02-21T23:12:04.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:04.755+0000 7fd836600640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8300a5ec0 con 0x7fd8300c15d0 2024-02-21T23:12:04.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:04.755+0000 7fd836600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd830152e70 con 0x7fd8300c11f0 2024-02-21T23:12:04.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:04.755+0000 7fd827e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fd8300c15d0 0x7fd83006dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:04.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:04.755+0000 7fd82f400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd8300c11f0 0x7fd830153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:04.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:04.755+0000 7fd827e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fd8300c15d0 0x7fd83006dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.28:33198/0 (socket says 172.21.15.28:33198) 2024-02-21T23:12:04.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:04.755+0000 7fd827e00640 1 -- 172.21.15.28:0/341118814 learned_addr learned my addr 172.21.15.28:0/341118814 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:04.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:04.756+0000 7fd827e00640 1 -- 172.21.15.28:0/341118814 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd8300c11f0 msgr2=0x7fd830153950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:04.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:04.756+0000 7fd827e00640 1 --2- 172.21.15.28:0/341118814 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd8300c11f0 0x7fd830153950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:04.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:04.756+0000 7fd827e00640 1 -- 172.21.15.28:0/341118814 >> v1:172.21.15.81:6789/0 conn(0x7fd830156350 legacy=0x7fd8301567a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:04.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:04.756+0000 7fd82ea00640 1 -- 172.21.15.28:0/341118814 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1547954193 0 0) 0x7fd8300c1a30 con 0x7fd830156350 2024-02-21T23:12:04.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:04.756+0000 7fd827e00640 1 -- 172.21.15.28:0/341118814 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd830153150 con 0x7fd8300c15d0 2024-02-21T23:12:04.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:04.757+0000 7fd827e00640 1 --2- 172.21.15.28:0/341118814 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd8300c15d0 0x7fd83006dce0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fd818009870 tx=0x7fd81802b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cc42c346b53ce7e3 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:04.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:04.758+0000 7fd82ea00640 1 -- 172.21.15.28:0/341118814 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8180317c0 con 0x7fd8300c15d0 2024-02-21T23:12:04.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:04.758+0000 7fd82ea00640 1 -- 172.21.15.28:0/341118814 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd818037070 con 0x7fd8300c15d0 2024-02-21T23:12:04.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:04.758+0000 7fd82ea00640 1 -- 172.21.15.28:0/341118814 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd818031b80 con 0x7fd8300c15d0 2024-02-21T23:12:04.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:04.758+0000 7fd836600640 1 -- 172.21.15.28:0/341118814 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd8300c15d0 msgr2=0x7fd83006dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:04.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:04.758+0000 7fd836600640 1 --2- 172.21.15.28:0/341118814 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd8300c15d0 0x7fd83006dce0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fd818009870 tx=0x7fd81802b860 comp rx=0 tx=0).stop 2024-02-21T23:12:04.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:04.759+0000 7fd836600640 1 -- 172.21.15.28:0/341118814 shutdown_connections 2024-02-21T23:12:04.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:04.759+0000 7fd836600640 1 --2- 172.21.15.28:0/341118814 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd8300c15d0 0x7fd83006dce0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:04.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:04.759+0000 7fd836600640 1 --2- 172.21.15.28:0/341118814 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd8300c11f0 0x7fd830153950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:04.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:04.759+0000 7fd836600640 1 -- 172.21.15.28:0/341118814 >> 172.21.15.28:0/341118814 conn(0x7fd8300c67f0 msgr2=0x7fd8300c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:04.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:04.759+0000 7fd836600640 1 -- 172.21.15.28:0/341118814 shutdown_connections 2024-02-21T23:12:04.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:04.759+0000 7fd836600640 1 -- 172.21.15.28:0/341118814 wait complete. 2024-02-21T23:12:04.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:04.760+0000 7fd836600640 1 Processor -- start 2024-02-21T23:12:04.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:04.761+0000 7fd836600640 1 -- start start 2024-02-21T23:12:04.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:04.761+0000 7fd836600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8300c11f0 0x7fd830148c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:04.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:04.761+0000 7fd836600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd8300c15d0 0x7fd8301ead00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:04.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:04.761+0000 7fd836600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd830156350 0x7fd8301eb240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:04.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:04.761+0000 7fd836600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8300c4c00 con 0x7fd8300c11f0 2024-02-21T23:12:04.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:04.761+0000 7fd836600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd83014d1c0 con 0x7fd830156350 2024-02-21T23:12:04.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:04.761+0000 7fd836600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8300a6110 con 0x7fd8300c15d0 2024-02-21T23:12:04.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:04.762+0000 7fd82f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8300c11f0 0x7fd830148c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:04.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:04.762+0000 7fd82f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8300c11f0 0x7fd830148c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60456/0 (socket says 172.21.15.28:60456) 2024-02-21T23:12:04.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:04.762+0000 7fd82f400640 1 -- 172.21.15.28:0/1296333433 learned_addr learned my addr 172.21.15.28:0/1296333433 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:04.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:04.762+0000 7fd827e00640 1 --2- 172.21.15.28:0/1296333433 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd8300c15d0 0x7fd8301ead00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:04.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:04.762+0000 7fd82fe00640 1 --2- 172.21.15.28:0/1296333433 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd830156350 0x7fd8301eb240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:04.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:04.762+0000 7fd82f400640 1 -- 172.21.15.28:0/1296333433 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd8300c15d0 msgr2=0x7fd8301ead00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:04.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:04.762+0000 7fd82f400640 1 --2- 172.21.15.28:0/1296333433 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd8300c15d0 0x7fd8301ead00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:04.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:04.762+0000 7fd82f400640 1 -- 172.21.15.28:0/1296333433 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd830156350 msgr2=0x7fd8301eb240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:04.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:04.762+0000 7fd82f400640 1 --2- 172.21.15.28:0/1296333433 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd830156350 0x7fd8301eb240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:04.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:04.762+0000 7fd82f400640 1 -- 172.21.15.28:0/1296333433 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd830144d20 con 0x7fd8300c11f0 2024-02-21T23:12:04.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:04.762+0000 7fd82fe00640 1 --2- 172.21.15.28:0/1296333433 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd830156350 0x7fd8301eb240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:12:04.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:04.763+0000 7fd82f400640 1 --2- 172.21.15.28:0/1296333433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8300c11f0 0x7fd830148c60 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 crypto rx=0x7fd82000e9b0 tx=0x7fd82000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:04.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:04.763+0000 7fd827400640 1 -- 172.21.15.28:0/1296333433 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd820017070 con 0x7fd8300c11f0 2024-02-21T23:12:04.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:04.763+0000 7fd827400640 1 -- 172.21.15.28:0/1296333433 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd82000c9a0 con 0x7fd8300c11f0 2024-02-21T23:12:04.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:04.763+0000 7fd827400640 1 -- 172.21.15.28:0/1296333433 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd82000cd90 con 0x7fd8300c11f0 2024-02-21T23:12:04.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:04.763+0000 7fd836600640 1 -- 172.21.15.28:0/1296333433 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8301eec10 con 0x7fd8300c11f0 2024-02-21T23:12:04.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:04.763+0000 7fd836600640 1 -- 172.21.15.28:0/1296333433 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8301452c0 con 0x7fd8300c11f0 2024-02-21T23:12:04.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:04.764+0000 7fd836600640 1 -- 172.21.15.28:0/1296333433 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd830067f30 con 0x7fd8300c11f0 2024-02-21T23:12:04.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:04.766+0000 7fd827400640 1 -- 172.21.15.28:0/1296333433 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd82001a070 con 0x7fd8300c11f0 2024-02-21T23:12:04.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:04.766+0000 7fd827400640 1 --2- 172.21.15.28:0/1296333433 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd80c06b4d0 0x7fd80c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:04.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:04.767+0000 7fd827400640 1 -- 172.21.15.28:0/1296333433 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd82008a770 con 0x7fd8300c11f0 2024-02-21T23:12:04.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:04.767+0000 7fd827e00640 1 --2- 172.21.15.28:0/1296333433 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd80c06b4d0 0x7fd80c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:04.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:04.768+0000 7fd827e00640 1 --2- 172.21.15.28:0/1296333433 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd80c06b4d0 0x7fd80c06d990 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fd818036000 tx=0x7fd818008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:04.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:04.771+0000 7fd827400640 1 -- 172.21.15.28:0/1296333433 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd820059460 con 0x7fd8300c11f0 2024-02-21T23:12:04.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.010+0000 7fd836600640 1 -- 172.21.15.28:0/1296333433 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd8300c0960 con 0x7fd8300c11f0 2024-02-21T23:12:04.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.011+0000 7fd827400640 1 -- 172.21.15.28:0/1296333433 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 209 v209) v1 ==== 95+0+4959 (secure 0 0 0) 0x7fd820058d60 con 0x7fd8300c11f0 2024-02-21T23:12:04.999 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:12:04.999 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":209,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6848,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3439896042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3439896042},{"type":"v1","addr":"172.21.15.28:6835","nonce":3439896042}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":196},{"gid":7729,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/3213249808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3213249808},{"type":"v1","addr":"172.21.15.28:6833","nonce":3213249808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":195}],"filesystems":[{"mdsmap":{"epoch":209,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:11:38.384619+0000","modified":"2024-02-21T23:12:03.626987+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7867,"mds_1":7735},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7735":{"gid":7735,"name":"d","rank":1,"incarnation":208,"state":"up:active","state_seq":9,"addr":"172.21.15.81:6836/2895690310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2895690310},{"type":"v1","addr":"172.21.15.81:6836","nonce":2895690310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7867":{"gid":7867,"name":"b","rank":0,"incarnation":206,"state":"up:active","state_seq":2,"addr":"172.21.15.81:6837/416021079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":416021079},{"type":"v1","addr":"172.21.15.81:6837","nonce":416021079}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-02-21T23:12:04.999 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 209 2024-02-21T23:12:05.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.014+0000 7fd836600640 1 -- 172.21.15.28:0/1296333433 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd80c06b4d0 msgr2=0x7fd80c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:05.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.014+0000 7fd836600640 1 --2- 172.21.15.28:0/1296333433 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd80c06b4d0 0x7fd80c06d990 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fd818036000 tx=0x7fd818008030 comp rx=0 tx=0).stop 2024-02-21T23:12:05.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.014+0000 7fd836600640 1 -- 172.21.15.28:0/1296333433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8300c11f0 msgr2=0x7fd830148c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:05.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.014+0000 7fd836600640 1 --2- 172.21.15.28:0/1296333433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8300c11f0 0x7fd830148c60 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 crypto rx=0x7fd82000e9b0 tx=0x7fd82000ee70 comp rx=0 tx=0).stop 2024-02-21T23:12:05.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.014+0000 7fd836600640 1 -- 172.21.15.28:0/1296333433 shutdown_connections 2024-02-21T23:12:05.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.014+0000 7fd836600640 1 --2- 172.21.15.28:0/1296333433 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd80c06b4d0 0x7fd80c06d990 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:05.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.014+0000 7fd836600640 1 --2- 172.21.15.28:0/1296333433 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd8300c15d0 0x7fd8301ead00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:05.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.014+0000 7fd836600640 1 --2- 172.21.15.28:0/1296333433 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd830156350 0x7fd8301eb240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:05.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.014+0000 7fd836600640 1 --2- 172.21.15.28:0/1296333433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8300c11f0 0x7fd830148c60 unknown :-1 s=CLOSED pgs=2121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:05.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.014+0000 7fd836600640 1 -- 172.21.15.28:0/1296333433 >> 172.21.15.28:0/1296333433 conn(0x7fd8300c67f0 msgr2=0x7fd8300c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:05.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.014+0000 7fd836600640 1 -- 172.21.15.28:0/1296333433 shutdown_connections 2024-02-21T23:12:05.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.014+0000 7fd836600640 1 -- 172.21.15.28:0/1296333433 wait complete. 2024-02-21T23:12:05.048 INFO:tasks.ceph.mds.b.smithi081.stderr:2024-02-21T23:12:05.061+0000 7f1d0d000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-21T23:12:05.049 INFO:tasks.ceph.mds.b.smithi081.stderr:2024-02-21T23:12:05.061+0000 7f1d0d000640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-02-21T23:12:05.425 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 209, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:11:38.384619+0000', 'modified': '2024-02-21T23:12:03.626987+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 7867, 'mds_1': 7735}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7735': {'gid': 7735, 'name': 'd', 'rank': 1, 'incarnation': 208, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.81:6836/2895690310', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6834', 'nonce': 2895690310}, {'type': 'v1', 'addr': '172.21.15.81:6836', 'nonce': 2895690310}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_7867': {'gid': 7867, 'name': 'b', 'rank': 0, 'incarnation': 206, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.81:6837/416021079', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6835', 'nonce': 416021079}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 416021079}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:12:05.425 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 2/2 2024-02-21T23:12:05.426 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-02-21T23:12:05.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.670+0000 7f7632c00640 1 Processor -- start 2024-02-21T23:12:05.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.670+0000 7f7632c00640 1 -- start start 2024-02-21T23:12:05.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.672+0000 7f7632c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f762c0c5f20 0x7f762c156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:05.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.672+0000 7f7632c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f762c0c5c10 con 0x7f762c153220 2024-02-21T23:12:05.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.672+0000 7f7632c00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f762c06e8d0 con 0x7f762c0c2df0 2024-02-21T23:12:05.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.672+0000 7f7632c00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f762c06e6a0 con 0x7f762c0c5f20 2024-02-21T23:12:05.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.672+0000 7f7630c00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f762c153220 0x7f762c153630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53728/0 (socket says 172.21.15.28:53728) 2024-02-21T23:12:05.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.672+0000 7f7630c00640 1 -- 172.21.15.28:0/1897975471 learned_addr learned my addr 172.21.15.28:0/1897975471 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:12:05.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.672+0000 7f762b400640 1 --2- 172.21.15.28:0/1897975471 >> v2:172.21.15.81:3300/0 conn(0x7f762c0c5f20 0x7f762c156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:05.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.673+0000 7f762aa00640 1 -- 172.21.15.28:0/1897975471 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2562382370 0 0) 0x7f762c0c5c10 con 0x7f762c153220 2024-02-21T23:12:05.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.673+0000 7f762aa00640 1 -- 172.21.15.28:0/1897975471 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7610003520 con 0x7f762c153220 2024-02-21T23:12:05.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.673+0000 7f762aa00640 1 -- 172.21.15.28:0/1897975471 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2805462067 0 0) 0x7f7610003520 con 0x7f762c153220 2024-02-21T23:12:05.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.673+0000 7f762aa00640 1 -- 172.21.15.28:0/1897975471 >> v2:172.21.15.81:3300/0 conn(0x7f762c0c5f20 msgr2=0x7f762c156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:05.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.673+0000 7f762aa00640 1 --2- 172.21.15.28:0/1897975471 >> v2:172.21.15.81:3300/0 conn(0x7f762c0c5f20 0x7f762c156ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:05.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.673+0000 7f762aa00640 1 -- 172.21.15.28:0/1897975471 >> v1:172.21.15.81:6789/0 conn(0x7f762c0c2df0 legacy=0x7f762c0c31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:05.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.673+0000 7f762aa00640 1 -- 172.21.15.28:0/1897975471 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f762c06ebb0 con 0x7f762c153220 2024-02-21T23:12:05.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.673+0000 7f762aa00640 1 -- 172.21.15.28:0/1897975471 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f76240019f0 con 0x7f762c153220 2024-02-21T23:12:05.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.673+0000 7f762b400640 1 --2- 172.21.15.28:0/1897975471 >> v2:172.21.15.81:3300/0 conn(0x7f762c0c5f20 0x7f762c156ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-21T23:12:05.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.673+0000 7f762aa00640 1 -- 172.21.15.28:0/1897975471 >> v1:172.21.15.28:6789/0 conn(0x7f762c153220 legacy=0x7f762c153630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:05.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.673+0000 7f762aa00640 1 --2- 172.21.15.28:0/1897975471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76100039b0 0x7f7610003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:05.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.673+0000 7f762aa00640 1 --2- 172.21.15.28:0/1897975471 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7610004800 0x7f7610004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:05.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.673+0000 7f762aa00640 1 --2- 172.21.15.28:0/1897975471 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f76100051c0 0x7f761000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:05.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.673+0000 7f762aa00640 1 -- 172.21.15.28:0/1897975471 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f761000c030 con 0x7f76100039b0 2024-02-21T23:12:05.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.673+0000 7f762aa00640 1 -- 172.21.15.28:0/1897975471 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f761000c1e0 con 0x7f7610004800 2024-02-21T23:12:05.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.673+0000 7f762aa00640 1 -- 172.21.15.28:0/1897975471 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f761000c3e0 con 0x7f76100051c0 2024-02-21T23:12:05.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.674+0000 7f762be00640 1 --2- 172.21.15.28:0/1897975471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76100039b0 0x7f7610003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:05.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.674+0000 7f762b400640 1 --2- 172.21.15.28:0/1897975471 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7610004800 0x7f7610004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:05.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.674+0000 7f7630c00640 1 --2- 172.21.15.28:0/1897975471 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f76100051c0 0x7f761000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:05.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.674+0000 7f762be00640 1 -- 172.21.15.28:0/1897975471 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f76100051c0 msgr2=0x7f761000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:05.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.674+0000 7f762be00640 1 --2- 172.21.15.28:0/1897975471 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f76100051c0 0x7f761000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:05.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.674+0000 7f762be00640 1 -- 172.21.15.28:0/1897975471 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7610004800 msgr2=0x7f7610004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:05.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.674+0000 7f762be00640 1 --2- 172.21.15.28:0/1897975471 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7610004800 0x7f7610004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:05.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.674+0000 7f762be00640 1 -- 172.21.15.28:0/1897975471 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7610003520 con 0x7f76100039b0 2024-02-21T23:12:05.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.674+0000 7f7630c00640 1 -- 172.21.15.28:0/1897975471 reap_dead start 2024-02-21T23:12:05.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.675+0000 7f762be00640 1 --2- 172.21.15.28:0/1897975471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76100039b0 0x7f7610003dd0 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 crypto rx=0x7f761c001750 tx=0x7f7610005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c28084fdfd37c36c server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:05.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.676+0000 7f762aa00640 1 -- 172.21.15.28:0/1897975471 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f761c001370 con 0x7f76100039b0 2024-02-21T23:12:05.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.676+0000 7f762aa00640 1 -- 172.21.15.28:0/1897975471 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f761c0052d0 con 0x7f76100039b0 2024-02-21T23:12:05.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.676+0000 7f762aa00640 1 -- 172.21.15.28:0/1897975471 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f761c003830 con 0x7f76100039b0 2024-02-21T23:12:05.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.676+0000 7f7632c00640 1 -- 172.21.15.28:0/1897975471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76100039b0 msgr2=0x7f7610003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:05.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.676+0000 7f7632c00640 1 --2- 172.21.15.28:0/1897975471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76100039b0 0x7f7610003dd0 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 crypto rx=0x7f761c001750 tx=0x7f7610005830 comp rx=0 tx=0).stop 2024-02-21T23:12:05.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.676+0000 7f7632c00640 1 -- 172.21.15.28:0/1897975471 shutdown_connections 2024-02-21T23:12:05.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.676+0000 7f7632c00640 1 --2- 172.21.15.28:0/1897975471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76100039b0 0x7f7610003dd0 unknown :-1 s=CLOSED pgs=2123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:05.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.676+0000 7f7632c00640 1 -- 172.21.15.28:0/1897975471 >> 172.21.15.28:0/1897975471 conn(0x7f762c06b820 msgr2=0x7f762c0c67f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:05.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.676+0000 7f7632c00640 1 -- 172.21.15.28:0/1897975471 shutdown_connections 2024-02-21T23:12:05.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.676+0000 7f7632c00640 1 -- 172.21.15.28:0/1897975471 wait complete. 2024-02-21T23:12:05.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.678+0000 7f7632c00640 1 Processor -- start 2024-02-21T23:12:05.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.678+0000 7f7632c00640 1 -- start start 2024-02-21T23:12:05.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.679+0000 7f7632c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f76100039b0 0x7f762c14fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:05.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.679+0000 7f7632c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f762c0bb810 0x7f762c14e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:05.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.679+0000 7f7632c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f762c0bbc70 0x7f762c14ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:05.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.679+0000 7f7632c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f762c156ed0 con 0x7f762c0bbc70 2024-02-21T23:12:05.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.679+0000 7f7632c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f762c06f170 con 0x7f762c0bb810 2024-02-21T23:12:05.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.679+0000 7f7632c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f762c157050 con 0x7f76100039b0 2024-02-21T23:12:05.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.679+0000 7f7630c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f762c0bbc70 0x7f762c14ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:05.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.680+0000 7f7630c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f762c0bbc70 0x7f762c14ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60054/0 (socket says 172.21.15.28:60054) 2024-02-21T23:12:05.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.680+0000 7f7630c00640 1 -- 172.21.15.28:0/1084905162 learned_addr learned my addr 172.21.15.28:0/1084905162 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:05.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.680+0000 7f762be00640 1 --2- 172.21.15.28:0/1084905162 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f76100039b0 0x7f762c14fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:05.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.680+0000 7f762b400640 1 --2- 172.21.15.28:0/1084905162 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f762c0bb810 0x7f762c14e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:05.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.680+0000 7f7630c00640 1 -- 172.21.15.28:0/1084905162 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f76100039b0 msgr2=0x7f762c14fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:05.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.680+0000 7f7630c00640 1 --2- 172.21.15.28:0/1084905162 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f76100039b0 0x7f762c14fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:05.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.680+0000 7f7630c00640 1 -- 172.21.15.28:0/1084905162 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f762c0bb810 msgr2=0x7f762c14e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:05.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.680+0000 7f7630c00640 1 --2- 172.21.15.28:0/1084905162 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f762c0bb810 0x7f762c14e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:05.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.680+0000 7f7630c00640 1 -- 172.21.15.28:0/1084905162 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f762c1ea290 con 0x7f762c0bbc70 2024-02-21T23:12:05.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.680+0000 7f762be00640 1 --2- 172.21.15.28:0/1084905162 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f76100039b0 0x7f762c14fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:12:05.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.680+0000 7f762b400640 1 --2- 172.21.15.28:0/1084905162 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f762c0bb810 0x7f762c14e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:12:05.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.680+0000 7f7630c00640 1 --2- 172.21.15.28:0/1084905162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f762c0bbc70 0x7f762c14ee60 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 crypto rx=0x7f762c153600 tx=0x7f762400e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:05.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.681+0000 7f7623e00640 1 -- 172.21.15.28:0/1084905162 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f762400ebb0 con 0x7f762c0bbc70 2024-02-21T23:12:05.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.681+0000 7f7623e00640 1 -- 172.21.15.28:0/1084905162 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7624018070 con 0x7f762c0bbc70 2024-02-21T23:12:05.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.681+0000 7f7632c00640 1 -- 172.21.15.28:0/1084905162 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f762c14f3a0 con 0x7f762c0bbc70 2024-02-21T23:12:05.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.681+0000 7f7623e00640 1 -- 172.21.15.28:0/1084905162 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7624013420 con 0x7f762c0bbc70 2024-02-21T23:12:05.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.681+0000 7f7632c00640 1 -- 172.21.15.28:0/1084905162 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f762c1ea830 con 0x7f762c0bbc70 2024-02-21T23:12:05.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.683+0000 7f7623e00640 1 -- 172.21.15.28:0/1084905162 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f76240135c0 con 0x7f762c0bbc70 2024-02-21T23:12:05.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.684+0000 7f7623e00640 1 --2- 172.21.15.28:0/1084905162 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f760c06b500 0x7f760c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:05.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.684+0000 7f7623e00640 1 -- 172.21.15.28:0/1084905162 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f762408a720 con 0x7f762c0bbc70 2024-02-21T23:12:05.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.685+0000 7f762be00640 1 --2- 172.21.15.28:0/1084905162 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f760c06b500 0x7f760c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:05.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.685+0000 7f7635bcf740 1 Processor -- start 2024-02-21T23:12:05.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.685+0000 7f7635bcf740 1 -- start start 2024-02-21T23:12:05.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.686+0000 7f762be00640 1 --2- 172.21.15.28:0/1084905162 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f760c06b500 0x7f760c06d9c0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f761c033000 tx=0x7f761c009030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:05.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.686+0000 7f7635bcf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55892f25ae40 0x55892f25b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:05.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.686+0000 7f7635bcf740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55892f25c150 0x55892f296f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:05.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.686+0000 7f7635bcf740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55892f2974d0 0x55892f295540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:05.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.686+0000 7f7635bcf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55892f1bf470 con 0x55892f25ae40 2024-02-21T23:12:05.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.686+0000 7f7635bcf740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55892f168af0 con 0x55892f2974d0 2024-02-21T23:12:05.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.686+0000 7f7635bcf740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x55892f1bb510 con 0x55892f25c150 2024-02-21T23:12:05.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.686+0000 7f762be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55892f25ae40 0x55892f25b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:05.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.686+0000 7f762be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55892f25ae40 0x55892f25b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60060/0 (socket says 172.21.15.28:60060) 2024-02-21T23:12:05.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.686+0000 7f762be00640 1 -- 172.21.15.28:0/19417689 learned_addr learned my addr 172.21.15.28:0/19417689 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:05.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.686+0000 7f762b400640 1 --2- 172.21.15.28:0/19417689 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55892f25c150 0x55892f296f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:05.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.686+0000 7f7630c00640 1 --2- 172.21.15.28:0/19417689 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55892f2974d0 0x55892f295540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:05.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.687+0000 7f762be00640 1 -- 172.21.15.28:0/19417689 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55892f25c150 msgr2=0x55892f296f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:05.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.687+0000 7f762be00640 1 --2- 172.21.15.28:0/19417689 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55892f25c150 0x55892f296f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:05.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.687+0000 7f762be00640 1 -- 172.21.15.28:0/19417689 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55892f2974d0 msgr2=0x55892f295540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:05.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.687+0000 7f762be00640 1 --2- 172.21.15.28:0/19417689 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55892f2974d0 0x55892f295540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:05.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.687+0000 7f762be00640 1 -- 172.21.15.28:0/19417689 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55892f16fd90 con 0x55892f25ae40 2024-02-21T23:12:05.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.687+0000 7f762b400640 1 --2- 172.21.15.28:0/19417689 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55892f25c150 0x55892f296f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:12:05.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.687+0000 7f762be00640 1 --2- 172.21.15.28:0/19417689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55892f25ae40 0x55892f25b220 secure :-1 s=READY pgs=2125 cs=0 l=1 rev1=1 crypto rx=0x7f761c005480 tx=0x7f761c02f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81ea76718864a21d server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:05.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.687+0000 7f7617e00640 1 -- 172.21.15.28:0/19417689 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f761c007680 con 0x55892f25ae40 2024-02-21T23:12:05.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.687+0000 7f7617e00640 1 -- 172.21.15.28:0/19417689 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f761c04d070 con 0x55892f25ae40 2024-02-21T23:12:05.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.687+0000 7f7617e00640 1 -- 172.21.15.28:0/19417689 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f761c0069d0 con 0x55892f25ae40 2024-02-21T23:12:05.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.688+0000 7f7635bcf740 1 -- 172.21.15.28:0/19417689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55892f25ae40 msgr2=0x55892f25b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:05.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.688+0000 7f7635bcf740 1 --2- 172.21.15.28:0/19417689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55892f25ae40 0x55892f25b220 secure :-1 s=READY pgs=2125 cs=0 l=1 rev1=1 crypto rx=0x7f761c005480 tx=0x7f761c02f000 comp rx=0 tx=0).stop 2024-02-21T23:12:05.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.688+0000 7f7635bcf740 1 -- 172.21.15.28:0/19417689 shutdown_connections 2024-02-21T23:12:05.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.688+0000 7f7635bcf740 1 --2- 172.21.15.28:0/19417689 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55892f25c150 0x55892f296f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:05.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.688+0000 7f7635bcf740 1 --2- 172.21.15.28:0/19417689 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55892f2974d0 0x55892f295540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:05.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.688+0000 7f7635bcf740 1 --2- 172.21.15.28:0/19417689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55892f25ae40 0x55892f25b220 unknown :-1 s=CLOSED pgs=2125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:05.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.688+0000 7f7635bcf740 1 -- 172.21.15.28:0/19417689 >> 172.21.15.28:0/19417689 conn(0x55892f257790 msgr2=0x55892f27c1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:05.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.688+0000 7f7635bcf740 1 -- 172.21.15.28:0/19417689 shutdown_connections 2024-02-21T23:12:05.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.688+0000 7f7635bcf740 1 -- 172.21.15.28:0/19417689 wait complete. 2024-02-21T23:12:05.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.689+0000 7f7635bcf740 1 -- start start 2024-02-21T23:12:05.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.689+0000 7f7635bcf740 1 Processor -- start 2024-02-21T23:12:05.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.690+0000 7f7635bcf740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55892f25c150 0x55892f27b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:05.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.690+0000 7f7635bcf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55892f2974d0 0x55892f2aa930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:05.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.690+0000 7f7635bcf740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55892f27c1e0 0x55892f276cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:05.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.690+0000 7f7635bcf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55892f15e040 con 0x55892f2974d0 2024-02-21T23:12:05.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.690+0000 7f7635bcf740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55892f27c8b0 con 0x55892f27c1e0 2024-02-21T23:12:05.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.690+0000 7f7635bcf740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x55892f29cd00 con 0x55892f25c150 2024-02-21T23:12:05.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.690+0000 7f7635bcf740 10 client.0 _ll_register_callbacks cb 0x55892f17b920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:12:05.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.690+0000 7f762b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55892f2974d0 0x55892f2aa930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:05.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.691+0000 7f762b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55892f2974d0 0x55892f2aa930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60074/0 (socket says 172.21.15.28:60074) 2024-02-21T23:12:05.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.691+0000 7f762b400640 1 -- 172.21.15.28:0/1282208230 learned_addr learned my addr 172.21.15.28:0/1282208230 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:05.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.691+0000 7f762be00640 1 --2- 172.21.15.28:0/1282208230 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55892f25c150 0x55892f27b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:05.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.691+0000 7f7630c00640 1 --2- 172.21.15.28:0/1282208230 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55892f27c1e0 0x55892f276cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:05.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.691+0000 7f762b400640 1 -- 172.21.15.28:0/1282208230 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55892f25c150 msgr2=0x55892f27b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:05.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.691+0000 7f762b400640 1 --2- 172.21.15.28:0/1282208230 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55892f25c150 0x55892f27b6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:05.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.691+0000 7f762b400640 1 -- 172.21.15.28:0/1282208230 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55892f27c1e0 msgr2=0x55892f276cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:05.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.691+0000 7f762b400640 1 --2- 172.21.15.28:0/1282208230 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55892f27c1e0 0x55892f276cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:05.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.691+0000 7f762b400640 1 -- 172.21.15.28:0/1282208230 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55892f17daa0 con 0x55892f2974d0 2024-02-21T23:12:05.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.691+0000 7f762b400640 1 --2- 172.21.15.28:0/1282208230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55892f2974d0 0x55892f2aa930 secure :-1 s=READY pgs=2126 cs=0 l=1 rev1=1 crypto rx=0x7f762c0699f0 tx=0x7f7618013a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0c9a5f1ea37e28c server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:05.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.691+0000 7f7614c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:12:05.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.692+0000 7f7614c00640 1 -- 172.21.15.28:0/1282208230 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76180158f0 con 0x55892f2974d0 2024-02-21T23:12:05.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.692+0000 7f7614c00640 1 -- 172.21.15.28:0/1282208230 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(186..186 src has 1..186) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f7618015a90 con 0x55892f2974d0 2024-02-21T23:12:05.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.692+0000 7f7635bcf740 1 -- 172.21.15.28:0/1282208230 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f761c04d070 con 0x55892f2974d0 2024-02-21T23:12:05.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.692+0000 7f7614c00640 1 -- 172.21.15.28:0/1282208230 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=209) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7618015c90 con 0x55892f2974d0 2024-02-21T23:12:05.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.692+0000 7f7635bcf740 10 client.10155 fetch_fsmap learned FSMap version 209 2024-02-21T23:12:05.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.692+0000 7f7635bcf740 1 -- 172.21.15.28:0/1282208230 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=209,monmap=2+}) v3 -- 0x55892eed42d0 con 0x55892f2974d0 2024-02-21T23:12:05.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.692+0000 7f7614c00640 1 -- 172.21.15.28:0/1282208230 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f761c04d070 con 0x55892f2974d0 2024-02-21T23:12:05.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.692+0000 7f7614c00640 1 -- 172.21.15.28:0/1282208230 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 209) v1 ==== 3306+0+0 (secure 0 0 0) 0x7f761802a080 con 0x55892f2974d0 2024-02-21T23:12:05.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.692+0000 7f7635bcf740 10 client.10155 fetch_fsmap finished waiting for FSMap version 209 2024-02-21T23:12:05.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.693+0000 7f7635bcf740 10 client.10155 resolve_mds: resolved name 'd' to daemon mds.d 2024-02-21T23:12:05.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.693+0000 7f7635bcf740 20 client.10155 populate_metadata read hostname 'smithi028' 2024-02-21T23:12:05.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.693+0000 7f7635bcf740 1 --2- 172.21.15.28:0/1282208230 >> [v2:172.21.15.81:6834/2895690310,v1:172.21.15.81:6836/2895690310] conn(0x55892f2a77a0 0x55892f2b5b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:05.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.693+0000 7f7635bcf740 4 client.10155 mds_command: new command op to 7735 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-21T23:12:05.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.693+0000 7f7635bcf740 1 -- 172.21.15.28:0/1282208230 --> [v2:172.21.15.81:6834/2895690310,v1:172.21.15.81:6836/2895690310] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55892f1cc9b0 con 0x55892f2a77a0 2024-02-21T23:12:05.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.693+0000 7f762be00640 1 --2- 172.21.15.28:0/1282208230 >> [v2:172.21.15.81:6834/2895690310,v1:172.21.15.81:6836/2895690310] conn(0x55892f2a77a0 0x55892f2b5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:05.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.694+0000 7f762be00640 1 --2- 172.21.15.28:0/1282208230 >> [v2:172.21.15.81:6834/2895690310,v1:172.21.15.81:6836/2895690310] conn(0x55892f2a77a0 0x55892f2b5b70 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=57224aa73c1eaae5 server_cookie=411f15c7bb22ab90 in_seq=0 out_seq=0 2024-02-21T23:12:05.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.694+0000 7f7614c00640 10 client.10155 ms_handle_connect on v2:172.21.15.81:6834/2895690310 2024-02-21T23:12:05.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.696+0000 7f7614c00640 1 -- 172.21.15.28:0/1282208230 <== mds.1 v2:172.21.15.81:6834/2895690310 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55892f1cc9b0 con 0x55892f2a77a0 2024-02-21T23:12:05.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.696+0000 7f7614c00640 10 client.10155 handle_command_reply: tid=0 2024-02-21T23:12:05.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.696+0000 7f7635bcf740 1 client.10155 shutdown 2024-02-21T23:12:05.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.697+0000 7f7635bcf740 1 -- 172.21.15.28:0/1282208230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55892f2974d0 msgr2=0x55892f2aa930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:05.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.697+0000 7f7635bcf740 1 --2- 172.21.15.28:0/1282208230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55892f2974d0 0x55892f2aa930 secure :-1 s=READY pgs=2126 cs=0 l=1 rev1=1 crypto rx=0x7f762c0699f0 tx=0x7f7618013a00 comp rx=0 tx=0).stop 2024-02-21T23:12:05.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.697+0000 7f7635bcf740 1 -- 172.21.15.28:0/1282208230 shutdown_connections 2024-02-21T23:12:05.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.697+0000 7f7635bcf740 1 --2- 172.21.15.28:0/1282208230 >> [v2:172.21.15.81:6834/2895690310,v1:172.21.15.81:6836/2895690310] conn(0x55892f2a77a0 0x55892f2b5b70 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:05.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.697+0000 7f7635bcf740 1 --2- 172.21.15.28:0/1282208230 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55892f27c1e0 0x55892f276cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:05.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.697+0000 7f7635bcf740 1 --2- 172.21.15.28:0/1282208230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55892f2974d0 0x55892f2aa930 unknown :-1 s=CLOSED pgs=2126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:05.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.697+0000 7f7635bcf740 1 --2- 172.21.15.28:0/1282208230 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55892f25c150 0x55892f27b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:05.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.697+0000 7f7614c00640 0 client.10155 ms_handle_reset on v2:172.21.15.81:6834/2895690310 2024-02-21T23:12:05.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.697+0000 7f7635bcf740 1 -- 172.21.15.28:0/1282208230 >> 172.21.15.28:0/1282208230 conn(0x55892f25ae40 msgr2=0x55892f2545f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:05.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.697+0000 7f7635bcf740 1 -- 172.21.15.28:0/1282208230 shutdown_connections 2024-02-21T23:12:05.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.697+0000 7f7635bcf740 1 -- 172.21.15.28:0/1282208230 wait complete. 2024-02-21T23:12:05.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.697+0000 7f7635bcf740 20 client.10155 trim_cache size 0 max 16384 2024-02-21T23:12:05.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.716+0000 7f7635bcf740 1 Processor -- start 2024-02-21T23:12:05.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.716+0000 7f7635bcf740 1 -- start start 2024-02-21T23:12:05.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.716+0000 7f7635bcf740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55892f25c150 0x55892f27a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:05.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.716+0000 7f7635bcf740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55892f2974d0 0x55892f27c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:05.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.716+0000 7f7635bcf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55892f270920 0x55892f279dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:05.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.716+0000 7f7635bcf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f762c1f3e70 con 0x55892f270920 2024-02-21T23:12:05.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.716+0000 7f7635bcf740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f762c0aa2c0 con 0x55892f2974d0 2024-02-21T23:12:05.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.716+0000 7f7635bcf740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f762c1f2a00 con 0x55892f25c150 2024-02-21T23:12:05.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.717+0000 7f7630c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55892f270920 0x55892f279dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:05.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.717+0000 7f7630c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55892f270920 0x55892f279dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60076/0 (socket says 172.21.15.28:60076) 2024-02-21T23:12:05.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.717+0000 7f7630c00640 1 -- 172.21.15.28:0/3472393860 learned_addr learned my addr 172.21.15.28:0/3472393860 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:05.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.717+0000 7f762b400640 1 --2- 172.21.15.28:0/3472393860 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55892f2974d0 0x55892f27c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:05.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.717+0000 7f762be00640 1 --2- 172.21.15.28:0/3472393860 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55892f25c150 0x55892f27a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:05.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.717+0000 7f7630c00640 1 -- 172.21.15.28:0/3472393860 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55892f25c150 msgr2=0x55892f27a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:05.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.717+0000 7f7630c00640 1 --2- 172.21.15.28:0/3472393860 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55892f25c150 0x55892f27a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:05.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.717+0000 7f7630c00640 1 -- 172.21.15.28:0/3472393860 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55892f2974d0 msgr2=0x55892f27c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:05.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.717+0000 7f7630c00640 1 --2- 172.21.15.28:0/3472393860 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55892f2974d0 0x55892f27c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:05.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.717+0000 7f7630c00640 1 -- 172.21.15.28:0/3472393860 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55892ef9e6b0 con 0x55892f270920 2024-02-21T23:12:05.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.717+0000 7f762b400640 1 --2- 172.21.15.28:0/3472393860 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55892f2974d0 0x55892f27c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:12:05.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.717+0000 7f762be00640 1 --2- 172.21.15.28:0/3472393860 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55892f25c150 0x55892f27a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:12:05.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.717+0000 7f7630c00640 1 --2- 172.21.15.28:0/3472393860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55892f270920 0x55892f279dd0 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 crypto rx=0x7f7624000f80 tx=0x7f762408a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1de0f909c2787b31 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:05.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.718+0000 7f7617400640 1 -- 172.21.15.28:0/3472393860 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7624070020 con 0x55892f270920 2024-02-21T23:12:05.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.718+0000 7f7617400640 1 -- 172.21.15.28:0/3472393860 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7624058c70 con 0x55892f270920 2024-02-21T23:12:05.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.718+0000 7f7617400640 1 -- 172.21.15.28:0/3472393860 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f762405cdc0 con 0x55892f270920 2024-02-21T23:12:05.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.718+0000 7f7635bcf740 1 -- 172.21.15.28:0/3472393860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55892f270920 msgr2=0x55892f279dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:05.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.718+0000 7f7635bcf740 1 --2- 172.21.15.28:0/3472393860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55892f270920 0x55892f279dd0 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 crypto rx=0x7f7624000f80 tx=0x7f762408a040 comp rx=0 tx=0).stop 2024-02-21T23:12:05.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.718+0000 7f7635bcf740 1 -- 172.21.15.28:0/3472393860 shutdown_connections 2024-02-21T23:12:05.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.718+0000 7f7635bcf740 1 --2- 172.21.15.28:0/3472393860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55892f270920 0x55892f279dd0 unknown :-1 s=CLOSED pgs=2127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:05.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.718+0000 7f7635bcf740 1 --2- 172.21.15.28:0/3472393860 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55892f2974d0 0x55892f27c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:05.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.718+0000 7f7635bcf740 1 --2- 172.21.15.28:0/3472393860 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55892f25c150 0x55892f27a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:05.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.718+0000 7f7635bcf740 1 -- 172.21.15.28:0/3472393860 >> 172.21.15.28:0/3472393860 conn(0x55892f25ae40 msgr2=0x55892f27b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:05.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.719+0000 7f7635bcf740 1 -- 172.21.15.28:0/3472393860 shutdown_connections 2024-02-21T23:12:05.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.719+0000 7f7635bcf740 1 -- 172.21.15.28:0/3472393860 wait complete. 2024-02-21T23:12:05.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.719+0000 7f7635bcf740 1 -- start start 2024-02-21T23:12:05.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.719+0000 7f7635bcf740 1 Processor -- start 2024-02-21T23:12:05.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.720+0000 7f7635bcf740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55892f25c150 0x55892f29a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:05.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.720+0000 7f7635bcf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55892f270920 0x55892f2732e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:05.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.720+0000 7f7635bcf740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55892f2974d0 0x55892f3002e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:05.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.720+0000 7f7635bcf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55892f29c710 con 0x55892f270920 2024-02-21T23:12:05.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.720+0000 7f7635bcf740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55892f29caf0 con 0x55892f25c150 2024-02-21T23:12:05.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.720+0000 7f7635bcf740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x55892f27d9e0 con 0x55892f2974d0 2024-02-21T23:12:05.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.720+0000 7f7635bcf740 10 client.0 _ll_register_callbacks cb 0x7f7618009630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:12:05.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.721+0000 7f762b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55892f270920 0x55892f2732e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:05.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.721+0000 7f762b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55892f270920 0x55892f2732e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60080/0 (socket says 172.21.15.28:60080) 2024-02-21T23:12:05.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.721+0000 7f762b400640 1 -- 172.21.15.28:0/2901108272 learned_addr learned my addr 172.21.15.28:0/2901108272 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:05.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.721+0000 7f762be00640 1 --2- 172.21.15.28:0/2901108272 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55892f25c150 0x55892f29a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:05.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.721+0000 7f7630c00640 1 --2- 172.21.15.28:0/2901108272 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55892f2974d0 0x55892f3002e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:05.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.721+0000 7f762b400640 1 -- 172.21.15.28:0/2901108272 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55892f2974d0 msgr2=0x55892f3002e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:05.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.721+0000 7f762b400640 1 --2- 172.21.15.28:0/2901108272 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55892f2974d0 0x55892f3002e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:05.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.721+0000 7f762b400640 1 -- 172.21.15.28:0/2901108272 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55892f25c150 msgr2=0x55892f29a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:05.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.721+0000 7f762b400640 1 --2- 172.21.15.28:0/2901108272 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55892f25c150 0x55892f29a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:05.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.721+0000 7f762b400640 1 -- 172.21.15.28:0/2901108272 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55892f294670 con 0x55892f270920 2024-02-21T23:12:05.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.722+0000 7f762b400640 1 --2- 172.21.15.28:0/2901108272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55892f270920 0x55892f2732e0 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 crypto rx=0x55892f2518f0 tx=0x7f7618004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8db00d3c0be2836 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:05.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.722+0000 7f7614c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:12:05.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.722+0000 7f7614c00640 1 -- 172.21.15.28:0/2901108272 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f761801d620 con 0x55892f270920 2024-02-21T23:12:05.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.722+0000 7f7614c00640 1 -- 172.21.15.28:0/2901108272 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(186..186 src has 1..186) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f7618024ae0 con 0x55892f270920 2024-02-21T23:12:05.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.722+0000 7f7635bcf740 1 -- 172.21.15.28:0/2901108272 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7624058c70 con 0x55892f270920 2024-02-21T23:12:05.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.722+0000 7f7614c00640 1 -- 172.21.15.28:0/2901108272 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=209) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7618024ce0 con 0x55892f270920 2024-02-21T23:12:05.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.722+0000 7f7635bcf740 10 client.10161 fetch_fsmap learned FSMap version 209 2024-02-21T23:12:05.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.722+0000 7f7635bcf740 1 -- 172.21.15.28:0/2901108272 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=209,monmap=2+}) v3 -- 0x55892f274600 con 0x55892f270920 2024-02-21T23:12:05.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.722+0000 7f7614c00640 1 -- 172.21.15.28:0/2901108272 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7624058c70 con 0x55892f270920 2024-02-21T23:12:05.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.723+0000 7f7614c00640 1 -- 172.21.15.28:0/2901108272 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 209) v1 ==== 3306+0+0 (secure 0 0 0) 0x7f76180242f0 con 0x55892f270920 2024-02-21T23:12:05.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.723+0000 7f7635bcf740 10 client.10161 fetch_fsmap finished waiting for FSMap version 209 2024-02-21T23:12:05.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.723+0000 7f7635bcf740 10 client.10161 resolve_mds: resolved name 'd' to daemon mds.d 2024-02-21T23:12:05.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.723+0000 7f7635bcf740 20 client.10161 populate_metadata read hostname 'smithi028' 2024-02-21T23:12:05.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.723+0000 7f7635bcf740 1 --2- 172.21.15.28:0/2901108272 >> [v2:172.21.15.81:6834/2895690310,v1:172.21.15.81:6836/2895690310] conn(0x55892f279cd0 0x55892f27a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:05.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.723+0000 7f7635bcf740 4 client.10161 mds_command: new command op to 7735 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-21T23:12:05.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.723+0000 7f7635bcf740 1 -- 172.21.15.28:0/2901108272 --> [v2:172.21.15.81:6834/2895690310,v1:172.21.15.81:6836/2895690310] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55892f1c6d00 con 0x55892f279cd0 2024-02-21T23:12:05.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.723+0000 7f762be00640 1 --2- 172.21.15.28:0/2901108272 >> [v2:172.21.15.81:6834/2895690310,v1:172.21.15.81:6836/2895690310] conn(0x55892f279cd0 0x55892f27a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:05.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.724+0000 7f762be00640 1 --2- 172.21.15.28:0/2901108272 >> [v2:172.21.15.81:6834/2895690310,v1:172.21.15.81:6836/2895690310] conn(0x55892f279cd0 0x55892f27a0b0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=573c6c5b22eb4362 server_cookie=ea3c1c7ef7f8fc1d in_seq=0 out_seq=0 2024-02-21T23:12:05.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.724+0000 7f7614c00640 10 client.10161 ms_handle_connect on v2:172.21.15.81:6834/2895690310 2024-02-21T23:12:05.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.725+0000 7f7614c00640 1 -- 172.21.15.28:0/2901108272 <== mds.1 v2:172.21.15.81:6834/2895690310 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+334 (crc 0 0 0) 0x55892f1c6d00 con 0x55892f279cd0 2024-02-21T23:12:05.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.725+0000 7f7614c00640 10 client.10161 handle_command_reply: tid=0 2024-02-21T23:12:05.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.725+0000 7f7635bcf740 1 client.10161 shutdown 2024-02-21T23:12:05.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.725+0000 7f7635bcf740 1 -- 172.21.15.28:0/2901108272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55892f270920 msgr2=0x55892f2732e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:05.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.725+0000 7f7635bcf740 1 --2- 172.21.15.28:0/2901108272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55892f270920 0x55892f2732e0 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 crypto rx=0x55892f2518f0 tx=0x7f7618004730 comp rx=0 tx=0).stop 2024-02-21T23:12:05.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.725+0000 7f7635bcf740 1 -- 172.21.15.28:0/2901108272 shutdown_connections 2024-02-21T23:12:05.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.725+0000 7f7635bcf740 1 --2- 172.21.15.28:0/2901108272 >> [v2:172.21.15.81:6834/2895690310,v1:172.21.15.81:6836/2895690310] conn(0x55892f279cd0 0x55892f27a0b0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:05.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.725+0000 7f7635bcf740 1 --2- 172.21.15.28:0/2901108272 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55892f2974d0 0x55892f3002e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:05.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.725+0000 7f7635bcf740 1 --2- 172.21.15.28:0/2901108272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55892f270920 0x55892f2732e0 unknown :-1 s=CLOSED pgs=2128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:05.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.725+0000 7f7635bcf740 1 --2- 172.21.15.28:0/2901108272 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55892f25c150 0x55892f29a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:05.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.725+0000 7f7614c00640 0 client.10161 ms_handle_reset on v2:172.21.15.81:6834/2895690310 2024-02-21T23:12:05.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.725+0000 7f7635bcf740 1 -- 172.21.15.28:0/2901108272 >> 172.21.15.28:0/2901108272 conn(0x55892f25ae40 msgr2=0x55892f27c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:05.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.725+0000 7f7635bcf740 1 -- 172.21.15.28:0/2901108272 shutdown_connections 2024-02-21T23:12:05.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.725+0000 7f7635bcf740 1 -- 172.21.15.28:0/2901108272 wait complete. 2024-02-21T23:12:05.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.725+0000 7f7635bcf740 20 client.10161 trim_cache size 0 max 16384 2024-02-21T23:12:05.809 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-21T23:12:05.810 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac", 2024-02-21T23:12:05.810 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 1, 2024-02-21T23:12:05.810 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 7735, 2024-02-21T23:12:05.810 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-02-21T23:12:05.810 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-02-21T23:12:05.810 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-02-21T23:12:05.811 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 2.1232590689999999, 2024-02-21T23:12:05.811 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 209, 2024-02-21T23:12:05.811 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 186, 2024-02-21T23:12:05.812 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 186, 2024-02-21T23:12:05.812 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 32.258585179000001 2024-02-21T23:12:05.812 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-21T23:12:05.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.726+0000 7f7632c00640 1 -- 172.21.15.28:0/1084905162 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f760c06b500 msgr2=0x7f760c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:05.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.726+0000 7f7632c00640 1 --2- 172.21.15.28:0/1084905162 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f760c06b500 0x7f760c06d9c0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f761c033000 tx=0x7f761c009030 comp rx=0 tx=0).stop 2024-02-21T23:12:05.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.726+0000 7f7632c00640 1 -- 172.21.15.28:0/1084905162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f762c0bbc70 msgr2=0x7f762c14ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:05.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.726+0000 7f7632c00640 1 --2- 172.21.15.28:0/1084905162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f762c0bbc70 0x7f762c14ee60 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 crypto rx=0x7f762c153600 tx=0x7f762400e530 comp rx=0 tx=0).stop 2024-02-21T23:12:05.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.726+0000 7f7632c00640 1 -- 172.21.15.28:0/1084905162 shutdown_connections 2024-02-21T23:12:05.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.726+0000 7f7632c00640 1 --2- 172.21.15.28:0/1084905162 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f760c06b500 0x7f760c06d9c0 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:05.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.726+0000 7f7632c00640 1 --2- 172.21.15.28:0/1084905162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f762c0bbc70 0x7f762c14ee60 unknown :-1 s=CLOSED pgs=2124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:05.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.726+0000 7f7632c00640 1 --2- 172.21.15.28:0/1084905162 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f762c0bb810 0x7f762c14e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:05.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.726+0000 7f7632c00640 1 --2- 172.21.15.28:0/1084905162 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f76100039b0 0x7f762c14fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:05.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.726+0000 7f7632c00640 1 -- 172.21.15.28:0/1084905162 >> 172.21.15.28:0/1084905162 conn(0x7f762c06b820 msgr2=0x7f762c0c31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:05.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.726+0000 7f7632c00640 1 -- 172.21.15.28:0/1084905162 shutdown_connections 2024-02-21T23:12:05.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:05.726+0000 7f7632c00640 1 -- 172.21.15.28:0/1084905162 wait complete. 2024-02-21T23:12:05.816 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-02-21T23:12:06.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.059+0000 7feb52400640 1 Processor -- start 2024-02-21T23:12:06.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.059+0000 7feb52400640 1 -- start start 2024-02-21T23:12:06.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.061+0000 7feb52400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7feb4c06e110 0x7feb4c155c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:06.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.061+0000 7feb52400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7feb4c156190 0x7feb4c0c2170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:06.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.061+0000 7feb52400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb4c0c2e60 con 0x7feb4c0c5a70 2024-02-21T23:12:06.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.061+0000 7feb52400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb4c1565b0 con 0x7feb4c156190 2024-02-21T23:12:06.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.061+0000 7feb52400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb4c06dc80 con 0x7feb4c06e110 2024-02-21T23:12:06.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.061+0000 7feb4b400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7feb4c0c5a70 0x7feb4c071900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53734/0 (socket says 172.21.15.28:53734) 2024-02-21T23:12:06.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.061+0000 7feb4b400640 1 -- 172.21.15.28:0/629834373 learned_addr learned my addr 172.21.15.28:0/629834373 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:12:06.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.061+0000 7feb4be00640 1 --2- 172.21.15.28:0/629834373 >> v2:172.21.15.28:3300/0 conn(0x7feb4c156190 0x7feb4c0c2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:06.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.061+0000 7feb42a00640 1 --2- 172.21.15.28:0/629834373 >> v2:172.21.15.81:3300/0 conn(0x7feb4c06e110 0x7feb4c155c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:06.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.061+0000 7feb4be00640 1 -- 172.21.15.28:0/629834373 >> v2:172.21.15.81:3300/0 conn(0x7feb4c06e110 msgr2=0x7feb4c155c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:06.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.061+0000 7feb4be00640 1 --2- 172.21.15.28:0/629834373 >> v2:172.21.15.81:3300/0 conn(0x7feb4c06e110 0x7feb4c155c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:06.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.061+0000 7feb4be00640 1 -- 172.21.15.28:0/629834373 >> v1:172.21.15.28:6789/0 conn(0x7feb4c0c5a70 legacy=0x7feb4c071900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:06.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.062+0000 7feb4be00640 1 -- 172.21.15.28:0/629834373 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb4c06d630 con 0x7feb4c156190 2024-02-21T23:12:06.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.061+0000 7feb4aa00640 1 -- 172.21.15.28:0/629834373 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2137701351 0 0) 0x7feb4c0c2e60 con 0x7feb4c0c5a70 2024-02-21T23:12:06.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.062+0000 7feb42a00640 1 --2- 172.21.15.28:0/629834373 >> v2:172.21.15.81:3300/0 conn(0x7feb4c06e110 0x7feb4c155c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:12:06.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.063+0000 7feb4be00640 1 --2- 172.21.15.28:0/629834373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb4c156190 0x7feb4c0c2170 secure :-1 s=READY pgs=2130 cs=0 l=1 rev1=1 crypto rx=0x7feb4c0c4390 tx=0x7feb4402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bdf6e6bc419cbf server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:06.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.063+0000 7feb4aa00640 1 -- 172.21.15.28:0/629834373 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb44039070 con 0x7feb4c156190 2024-02-21T23:12:06.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.063+0000 7feb4aa00640 1 -- 172.21.15.28:0/629834373 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb4402cc80 con 0x7feb4c156190 2024-02-21T23:12:06.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.063+0000 7feb4aa00640 1 -- 172.21.15.28:0/629834373 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb44034660 con 0x7feb4c156190 2024-02-21T23:12:06.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.063+0000 7feb52400640 1 -- 172.21.15.28:0/629834373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb4c156190 msgr2=0x7feb4c0c2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:06.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.063+0000 7feb52400640 1 --2- 172.21.15.28:0/629834373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb4c156190 0x7feb4c0c2170 secure :-1 s=READY pgs=2130 cs=0 l=1 rev1=1 crypto rx=0x7feb4c0c4390 tx=0x7feb4402bed0 comp rx=0 tx=0).stop 2024-02-21T23:12:06.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.064+0000 7feb52400640 1 -- 172.21.15.28:0/629834373 shutdown_connections 2024-02-21T23:12:06.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.064+0000 7feb52400640 1 --2- 172.21.15.28:0/629834373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb4c156190 0x7feb4c0c2170 unknown :-1 s=CLOSED pgs=2130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:06.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.064+0000 7feb52400640 1 --2- 172.21.15.28:0/629834373 >> v2:172.21.15.81:3300/0 conn(0x7feb4c06e110 0x7feb4c155c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:06.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.064+0000 7feb52400640 1 -- 172.21.15.28:0/629834373 >> 172.21.15.28:0/629834373 conn(0x7feb4c06b820 msgr2=0x7feb4c0c67f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:06.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.064+0000 7feb52400640 1 -- 172.21.15.28:0/629834373 shutdown_connections 2024-02-21T23:12:06.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.064+0000 7feb52400640 1 -- 172.21.15.28:0/629834373 wait complete. 2024-02-21T23:12:06.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.065+0000 7feb52400640 1 Processor -- start 2024-02-21T23:12:06.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.066+0000 7feb52400640 1 -- start start 2024-02-21T23:12:06.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.066+0000 7feb52400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feb4c06e110 0x7feb4c150610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:06.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.066+0000 7feb52400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb4c0c5a70 0x7feb4c0bcb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:06.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.067+0000 7feb52400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feb4c156190 0x7feb4c14c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:06.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.067+0000 7feb52400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb4c072190 con 0x7feb4c0c5a70 2024-02-21T23:12:06.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.067+0000 7feb52400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb4c072010 con 0x7feb4c156190 2024-02-21T23:12:06.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.067+0000 7feb52400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb4c0c5e50 con 0x7feb4c06e110 2024-02-21T23:12:06.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.067+0000 7feb42a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb4c0c5a70 0x7feb4c0bcb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:06.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.067+0000 7feb42a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb4c0c5a70 0x7feb4c0bcb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60098/0 (socket says 172.21.15.28:60098) 2024-02-21T23:12:06.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.067+0000 7feb42a00640 1 -- 172.21.15.28:0/1635732342 learned_addr learned my addr 172.21.15.28:0/1635732342 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:06.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.067+0000 7feb4b400640 1 --2- 172.21.15.28:0/1635732342 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feb4c06e110 0x7feb4c150610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:06.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.067+0000 7feb42a00640 1 -- 172.21.15.28:0/1635732342 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feb4c06e110 msgr2=0x7feb4c150610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:06.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.067+0000 7feb42a00640 1 --2- 172.21.15.28:0/1635732342 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feb4c06e110 0x7feb4c150610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:06.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.067+0000 7feb42a00640 1 -- 172.21.15.28:0/1635732342 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feb4c156190 msgr2=0x7feb4c14c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:06.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.067+0000 7feb4be00640 1 --2- 172.21.15.28:0/1635732342 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feb4c156190 0x7feb4c14c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:06.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.067+0000 7feb42a00640 1 --2- 172.21.15.28:0/1635732342 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feb4c156190 0x7feb4c14c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:06.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.067+0000 7feb42a00640 1 -- 172.21.15.28:0/1635732342 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb4c0c1010 con 0x7feb4c0c5a70 2024-02-21T23:12:06.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.068+0000 7feb42a00640 1 --2- 172.21.15.28:0/1635732342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb4c0c5a70 0x7feb4c0bcb30 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 crypto rx=0x7feb3400c820 tx=0x7feb3400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:06.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.068+0000 7feb43e00640 1 -- 172.21.15.28:0/1635732342 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb34015070 con 0x7feb4c0c5a70 2024-02-21T23:12:06.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.068+0000 7feb43e00640 1 -- 172.21.15.28:0/1635732342 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb34007a30 con 0x7feb4c0c5a70 2024-02-21T23:12:06.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.068+0000 7feb52400640 1 -- 172.21.15.28:0/1635732342 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb4c0bdd50 con 0x7feb4c0c5a70 2024-02-21T23:12:06.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.068+0000 7feb43e00640 1 -- 172.21.15.28:0/1635732342 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb34007e00 con 0x7feb4c0c5a70 2024-02-21T23:12:06.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.068+0000 7feb52400640 1 -- 172.21.15.28:0/1635732342 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb4c0be310 con 0x7feb4c0c5a70 2024-02-21T23:12:06.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.070+0000 7feb43e00640 1 -- 172.21.15.28:0/1635732342 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feb34018070 con 0x7feb4c0c5a70 2024-02-21T23:12:06.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.071+0000 7feb43e00640 1 --2- 172.21.15.28:0/1635732342 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7feb2806b500 0x7feb2806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:06.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.071+0000 7feb43e00640 1 -- 172.21.15.28:0/1635732342 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6101+0+0 (secure 0 0 0) 0x7feb3408a2e0 con 0x7feb4c0c5a70 2024-02-21T23:12:06.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.072+0000 7feb55216740 1 Processor -- start 2024-02-21T23:12:06.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.072+0000 7feb4b400640 1 --2- 172.21.15.28:0/1635732342 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7feb2806b500 0x7feb2806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:06.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.072+0000 7feb55216740 1 -- start start 2024-02-21T23:12:06.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.073+0000 7feb4b400640 1 --2- 172.21.15.28:0/1635732342 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7feb2806b500 0x7feb2806d9c0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7feb3c000f60 tx=0x7feb3c007b50 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:06.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.073+0000 7feb55216740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d54ede40 0x5602d54ee220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:06.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.073+0000 7feb55216740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5602d54ef150 0x5602d5529f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:06.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.073+0000 7feb55216740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5602d552a4d0 0x5602d5528540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:06.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.073+0000 7feb55216740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602d5452470 con 0x5602d54ede40 2024-02-21T23:12:06.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.073+0000 7feb55216740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602d53fbaf0 con 0x5602d54ef150 2024-02-21T23:12:06.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.073+0000 7feb55216740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x5602d544e510 con 0x5602d552a4d0 2024-02-21T23:12:06.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.073+0000 7feb4b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d54ede40 0x5602d54ee220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:06.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.074+0000 7feb4b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d54ede40 0x5602d54ee220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60106/0 (socket says 172.21.15.28:60106) 2024-02-21T23:12:06.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.074+0000 7feb4b400640 1 -- 172.21.15.28:0/2406609763 learned_addr learned my addr 172.21.15.28:0/2406609763 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:06.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.074+0000 7feb42a00640 1 --2- 172.21.15.28:0/2406609763 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5602d54ef150 0x5602d5529f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:06.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.074+0000 7feb4be00640 1 --2- 172.21.15.28:0/2406609763 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5602d552a4d0 0x5602d5528540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:06.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.074+0000 7feb4b400640 1 -- 172.21.15.28:0/2406609763 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5602d552a4d0 msgr2=0x5602d5528540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:06.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.074+0000 7feb4b400640 1 --2- 172.21.15.28:0/2406609763 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5602d552a4d0 0x5602d5528540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:06.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.074+0000 7feb4b400640 1 -- 172.21.15.28:0/2406609763 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5602d54ef150 msgr2=0x5602d5529f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:06.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.074+0000 7feb4b400640 1 --2- 172.21.15.28:0/2406609763 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5602d54ef150 0x5602d5529f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:06.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.074+0000 7feb4b400640 1 -- 172.21.15.28:0/2406609763 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5602d5402d90 con 0x5602d54ede40 2024-02-21T23:12:06.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.074+0000 7feb4b400640 1 --2- 172.21.15.28:0/2406609763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d54ede40 0x5602d54ee220 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7feb3c005de0 tx=0x7feb3c012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f74fea1d61fd8ee server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:06.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.075+0000 7feb3be00640 1 -- 172.21.15.28:0/2406609763 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb3c005e10 con 0x5602d54ede40 2024-02-21T23:12:06.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.075+0000 7feb3be00640 1 -- 172.21.15.28:0/2406609763 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb3c0071a0 con 0x5602d54ede40 2024-02-21T23:12:06.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.075+0000 7feb3be00640 1 -- 172.21.15.28:0/2406609763 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb3c015bc0 con 0x5602d54ede40 2024-02-21T23:12:06.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.075+0000 7feb55216740 1 -- 172.21.15.28:0/2406609763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d54ede40 msgr2=0x5602d54ee220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:06.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.075+0000 7feb55216740 1 --2- 172.21.15.28:0/2406609763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d54ede40 0x5602d54ee220 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7feb3c005de0 tx=0x7feb3c012000 comp rx=0 tx=0).stop 2024-02-21T23:12:06.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.075+0000 7feb55216740 1 -- 172.21.15.28:0/2406609763 shutdown_connections 2024-02-21T23:12:06.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.075+0000 7feb55216740 1 --2- 172.21.15.28:0/2406609763 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5602d552a4d0 0x5602d5528540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:06.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.075+0000 7feb55216740 1 --2- 172.21.15.28:0/2406609763 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5602d54ef150 0x5602d5529f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:06.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.075+0000 7feb55216740 1 --2- 172.21.15.28:0/2406609763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d54ede40 0x5602d54ee220 unknown :-1 s=CLOSED pgs=2132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:06.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.075+0000 7feb55216740 1 -- 172.21.15.28:0/2406609763 >> 172.21.15.28:0/2406609763 conn(0x5602d54ea790 msgr2=0x5602d550f1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:06.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.075+0000 7feb55216740 1 -- 172.21.15.28:0/2406609763 shutdown_connections 2024-02-21T23:12:06.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.075+0000 7feb55216740 1 -- 172.21.15.28:0/2406609763 wait complete. 2024-02-21T23:12:06.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.076+0000 7feb55216740 1 -- start start 2024-02-21T23:12:06.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.076+0000 7feb55216740 1 Processor -- start 2024-02-21T23:12:06.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.077+0000 7feb55216740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5602d54ef150 0x5602d553d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:06.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.077+0000 7feb55216740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d552a4d0 0x5602d550dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:06.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.077+0000 7feb55216740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5602d550a560 0x5602d54fbc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:06.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.077+0000 7feb55216740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602d53f1040 con 0x5602d552a4d0 2024-02-21T23:12:06.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.077+0000 7feb55216740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602d550f8b0 con 0x5602d54ef150 2024-02-21T23:12:06.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.077+0000 7feb55216740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x5602d550e9b0 con 0x5602d550a560 2024-02-21T23:12:06.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.077+0000 7feb55216740 10 client.0 _ll_register_callbacks cb 0x5602d540e920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:12:06.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.078+0000 7feb42a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d552a4d0 0x5602d550dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:06.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.078+0000 7feb42a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d552a4d0 0x5602d550dc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60120/0 (socket says 172.21.15.28:60120) 2024-02-21T23:12:06.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.078+0000 7feb42a00640 1 -- 172.21.15.28:0/2223177563 learned_addr learned my addr 172.21.15.28:0/2223177563 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:06.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.078+0000 7feb4be00640 1 --2- 172.21.15.28:0/2223177563 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5602d550a560 0x5602d54fbc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:06.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.078+0000 7feb4b400640 1 --2- 172.21.15.28:0/2223177563 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5602d54ef150 0x5602d553d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:06.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.078+0000 7feb42a00640 1 -- 172.21.15.28:0/2223177563 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5602d550a560 msgr2=0x5602d54fbc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:06.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.078+0000 7feb42a00640 1 --2- 172.21.15.28:0/2223177563 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5602d550a560 0x5602d54fbc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:06.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.078+0000 7feb42a00640 1 -- 172.21.15.28:0/2223177563 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5602d54ef150 msgr2=0x5602d553d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:06.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.078+0000 7feb42a00640 1 --2- 172.21.15.28:0/2223177563 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5602d54ef150 0x5602d553d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:06.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.078+0000 7feb42a00640 1 -- 172.21.15.28:0/2223177563 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5602d5410aa0 con 0x5602d552a4d0 2024-02-21T23:12:06.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.078+0000 7feb42a00640 1 --2- 172.21.15.28:0/2223177563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d552a4d0 0x5602d550dc90 secure :-1 s=READY pgs=2133 cs=0 l=1 rev1=1 crypto rx=0x7feb34088670 tx=0x7feb34058fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c0f707203023bfb server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:06.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.078+0000 7feb38c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:12:06.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.079+0000 7feb38c00640 1 -- 172.21.15.28:0/2223177563 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb3401f040 con 0x5602d552a4d0 2024-02-21T23:12:06.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.079+0000 7feb38c00640 1 -- 172.21.15.28:0/2223177563 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(186..186 src has 1..186) v4 ==== 6101+0+0 (secure 0 0 0) 0x7feb34070380 con 0x5602d552a4d0 2024-02-21T23:12:06.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.079+0000 7feb55216740 1 -- 172.21.15.28:0/2223177563 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7feb3c0071a0 con 0x5602d552a4d0 2024-02-21T23:12:06.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.079+0000 7feb38c00640 1 -- 172.21.15.28:0/2223177563 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=209) v2 ==== 24+0+0 (secure 0 0 0) 0x7feb3405ca10 con 0x5602d552a4d0 2024-02-21T23:12:06.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.079+0000 7feb55216740 10 client.10176 fetch_fsmap learned FSMap version 209 2024-02-21T23:12:06.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.079+0000 7feb55216740 1 -- 172.21.15.28:0/2223177563 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=209,monmap=2+}) v3 -- 0x5602d51672d0 con 0x5602d552a4d0 2024-02-21T23:12:06.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.079+0000 7feb38c00640 1 -- 172.21.15.28:0/2223177563 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb3c0071a0 con 0x5602d552a4d0 2024-02-21T23:12:06.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.079+0000 7feb38c00640 1 -- 172.21.15.28:0/2223177563 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 209) v1 ==== 3306+0+0 (secure 0 0 0) 0x7feb34088120 con 0x5602d552a4d0 2024-02-21T23:12:06.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.079+0000 7feb55216740 10 client.10176 fetch_fsmap finished waiting for FSMap version 209 2024-02-21T23:12:06.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.079+0000 7feb55216740 10 client.10176 resolve_mds: resolved name 'b' to daemon mds.b 2024-02-21T23:12:06.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.079+0000 7feb55216740 20 client.10176 populate_metadata read hostname 'smithi028' 2024-02-21T23:12:06.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.079+0000 7feb55216740 1 --2- 172.21.15.28:0/2223177563 >> [v2:172.21.15.81:6835/416021079,v1:172.21.15.81:6837/416021079] conn(0x5602d553a740 0x5602d5548b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:06.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.079+0000 7feb55216740 4 client.10176 mds_command: new command op to 7867 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-21T23:12:06.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.079+0000 7feb55216740 1 -- 172.21.15.28:0/2223177563 --> [v2:172.21.15.81:6835/416021079,v1:172.21.15.81:6837/416021079] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5602d545f9b0 con 0x5602d553a740 2024-02-21T23:12:06.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.080+0000 7feb4b400640 1 --2- 172.21.15.28:0/2223177563 >> [v2:172.21.15.81:6835/416021079,v1:172.21.15.81:6837/416021079] conn(0x5602d553a740 0x5602d5548b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:06.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.081+0000 7feb4b400640 1 --2- 172.21.15.28:0/2223177563 >> [v2:172.21.15.81:6835/416021079,v1:172.21.15.81:6837/416021079] conn(0x5602d553a740 0x5602d5548b00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b3298f3d8e52a53e server_cookie=eda89dfe7b9aacfb in_seq=0 out_seq=0 2024-02-21T23:12:06.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.081+0000 7feb38c00640 10 client.10176 ms_handle_connect on v2:172.21.15.81:6835/416021079 2024-02-21T23:12:06.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.082+0000 7feb38c00640 1 -- 172.21.15.28:0/2223177563 <== mds.0 v2:172.21.15.81:6835/416021079 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5602d545f9b0 con 0x5602d553a740 2024-02-21T23:12:06.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.082+0000 7feb38c00640 10 client.10176 handle_command_reply: tid=0 2024-02-21T23:12:06.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.083+0000 7feb55216740 1 client.10176 shutdown 2024-02-21T23:12:06.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.083+0000 7feb55216740 1 -- 172.21.15.28:0/2223177563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d552a4d0 msgr2=0x5602d550dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:06.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.083+0000 7feb55216740 1 --2- 172.21.15.28:0/2223177563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d552a4d0 0x5602d550dc90 secure :-1 s=READY pgs=2133 cs=0 l=1 rev1=1 crypto rx=0x7feb34088670 tx=0x7feb34058fc0 comp rx=0 tx=0).stop 2024-02-21T23:12:06.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.083+0000 7feb55216740 1 -- 172.21.15.28:0/2223177563 shutdown_connections 2024-02-21T23:12:06.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.083+0000 7feb55216740 1 --2- 172.21.15.28:0/2223177563 >> [v2:172.21.15.81:6835/416021079,v1:172.21.15.81:6837/416021079] conn(0x5602d553a740 0x5602d5548b00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:06.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.083+0000 7feb55216740 1 --2- 172.21.15.28:0/2223177563 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5602d550a560 0x5602d54fbc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:06.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.083+0000 7feb55216740 1 --2- 172.21.15.28:0/2223177563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d552a4d0 0x5602d550dc90 unknown :-1 s=CLOSED pgs=2133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:06.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.083+0000 7feb55216740 1 --2- 172.21.15.28:0/2223177563 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5602d54ef150 0x5602d553d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:06.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.083+0000 7feb38c00640 0 client.10176 ms_handle_reset on v2:172.21.15.81:6835/416021079 2024-02-21T23:12:06.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.083+0000 7feb55216740 1 -- 172.21.15.28:0/2223177563 >> 172.21.15.28:0/2223177563 conn(0x5602d54ede40 msgr2=0x5602d5528da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:06.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.083+0000 7feb55216740 1 -- 172.21.15.28:0/2223177563 shutdown_connections 2024-02-21T23:12:06.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.083+0000 7feb55216740 1 -- 172.21.15.28:0/2223177563 wait complete. 2024-02-21T23:12:06.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.083+0000 7feb55216740 20 client.10176 trim_cache size 0 max 16384 2024-02-21T23:12:06.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.102+0000 7feb55216740 1 Processor -- start 2024-02-21T23:12:06.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.103+0000 7feb55216740 1 -- start start 2024-02-21T23:12:06.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.103+0000 7feb55216740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d54ef150 0x5602d5529a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:06.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.103+0000 7feb55216740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5602d550a560 0x5602d5509700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:06.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.103+0000 7feb55216740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5602d552a4d0 0x5602d5509c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:06.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.103+0000 7feb55216740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb4c1f7860 con 0x5602d54ef150 2024-02-21T23:12:06.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.103+0000 7feb55216740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb4c1ee910 con 0x5602d550a560 2024-02-21T23:12:06.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.103+0000 7feb55216740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb4c1f6470 con 0x5602d552a4d0 2024-02-21T23:12:06.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.104+0000 7feb4b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d54ef150 0x5602d5529a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:06.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.104+0000 7feb4b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d54ef150 0x5602d5529a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60128/0 (socket says 172.21.15.28:60128) 2024-02-21T23:12:06.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.104+0000 7feb4b400640 1 -- 172.21.15.28:0/2644262246 learned_addr learned my addr 172.21.15.28:0/2644262246 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:06.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.104+0000 7feb42a00640 1 --2- 172.21.15.28:0/2644262246 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5602d550a560 0x5602d5509700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:06.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.104+0000 7feb4be00640 1 --2- 172.21.15.28:0/2644262246 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5602d552a4d0 0x5602d5509c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:06.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.104+0000 7feb4b400640 1 -- 172.21.15.28:0/2644262246 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5602d552a4d0 msgr2=0x5602d5509c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:06.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.104+0000 7feb4b400640 1 --2- 172.21.15.28:0/2644262246 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5602d552a4d0 0x5602d5509c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:06.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.104+0000 7feb4b400640 1 -- 172.21.15.28:0/2644262246 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5602d550a560 msgr2=0x5602d5509700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:06.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.104+0000 7feb4b400640 1 --2- 172.21.15.28:0/2644262246 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5602d550a560 0x5602d5509700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:06.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.104+0000 7feb4b400640 1 -- 172.21.15.28:0/2644262246 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5602d52316b0 con 0x5602d54ef150 2024-02-21T23:12:06.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.105+0000 7feb4b400640 1 --2- 172.21.15.28:0/2644262246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d54ef150 0x5602d5529a90 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 crypto rx=0x7feb3c005db0 tx=0x7feb3c013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=893961ab87935660 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:06.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.105+0000 7feb3b400640 1 -- 172.21.15.28:0/2644262246 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb3c010040 con 0x5602d54ef150 2024-02-21T23:12:06.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.105+0000 7feb3b400640 1 -- 172.21.15.28:0/2644262246 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb3c015a40 con 0x5602d54ef150 2024-02-21T23:12:06.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.105+0000 7feb3b400640 1 -- 172.21.15.28:0/2644262246 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb3c015df0 con 0x5602d54ef150 2024-02-21T23:12:06.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.105+0000 7feb55216740 1 -- 172.21.15.28:0/2644262246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d54ef150 msgr2=0x5602d5529a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:06.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.105+0000 7feb55216740 1 --2- 172.21.15.28:0/2644262246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d54ef150 0x5602d5529a90 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 crypto rx=0x7feb3c005db0 tx=0x7feb3c013000 comp rx=0 tx=0).stop 2024-02-21T23:12:06.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.106+0000 7feb55216740 1 -- 172.21.15.28:0/2644262246 shutdown_connections 2024-02-21T23:12:06.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.106+0000 7feb55216740 1 --2- 172.21.15.28:0/2644262246 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5602d552a4d0 0x5602d5509c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:06.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.106+0000 7feb55216740 1 --2- 172.21.15.28:0/2644262246 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5602d550a560 0x5602d5509700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:06.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.106+0000 7feb55216740 1 --2- 172.21.15.28:0/2644262246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d54ef150 0x5602d5529a90 unknown :-1 s=CLOSED pgs=2134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:06.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.106+0000 7feb55216740 1 -- 172.21.15.28:0/2644262246 >> 172.21.15.28:0/2644262246 conn(0x5602d54ede40 msgr2=0x5602d554ddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:06.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.106+0000 7feb55216740 1 -- 172.21.15.28:0/2644262246 shutdown_connections 2024-02-21T23:12:06.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.106+0000 7feb55216740 1 -- 172.21.15.28:0/2644262246 wait complete. 2024-02-21T23:12:06.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.106+0000 7feb55216740 1 -- start start 2024-02-21T23:12:06.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.107+0000 7feb55216740 1 Processor -- start 2024-02-21T23:12:06.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.108+0000 7feb55216740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d54ef150 0x5602d55437b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:06.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.108+0000 7feb55216740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5602d550a560 0x5602d5595f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:06.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.108+0000 7feb55216740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5602d552a4d0 0x5602d55964d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:06.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.108+0000 7feb55216740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602d550e6f0 con 0x5602d54ef150 2024-02-21T23:12:06.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.108+0000 7feb55216740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602d5510290 con 0x5602d550a560 2024-02-21T23:12:06.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.108+0000 7feb55216740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x5602d54fc3c0 con 0x5602d552a4d0 2024-02-21T23:12:06.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.108+0000 7feb55216740 10 client.0 _ll_register_callbacks cb 0x7feb340584b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:12:06.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.108+0000 7feb4b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d54ef150 0x5602d55437b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:06.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.108+0000 7feb4b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d54ef150 0x5602d55437b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60142/0 (socket says 172.21.15.28:60142) 2024-02-21T23:12:06.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.108+0000 7feb4b400640 1 -- 172.21.15.28:0/145502671 learned_addr learned my addr 172.21.15.28:0/145502671 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:06.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.108+0000 7feb42a00640 1 --2- 172.21.15.28:0/145502671 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5602d550a560 0x5602d5595f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:06.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.108+0000 7feb4be00640 1 --2- 172.21.15.28:0/145502671 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5602d552a4d0 0x5602d55964d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:06.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.108+0000 7feb4b400640 1 -- 172.21.15.28:0/145502671 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5602d552a4d0 msgr2=0x5602d55964d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:06.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.108+0000 7feb4b400640 1 --2- 172.21.15.28:0/145502671 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5602d552a4d0 0x5602d55964d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:06.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.108+0000 7feb4b400640 1 -- 172.21.15.28:0/145502671 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5602d550a560 msgr2=0x5602d5595f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:06.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.109+0000 7feb4b400640 1 --2- 172.21.15.28:0/145502671 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5602d550a560 0x5602d5595f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:06.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.109+0000 7feb4b400640 1 -- 172.21.15.28:0/145502671 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5602d54e9730 con 0x5602d54ef150 2024-02-21T23:12:06.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.109+0000 7feb4b400640 1 --2- 172.21.15.28:0/145502671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d54ef150 0x5602d55437b0 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 crypto rx=0x7feb3c015810 tx=0x7feb3c000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de293d4c524d51fc server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:06.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.109+0000 7feb38c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:12:06.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.109+0000 7feb38c00640 1 -- 172.21.15.28:0/145502671 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb3c015df0 con 0x5602d54ef150 2024-02-21T23:12:06.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.109+0000 7feb38c00640 1 -- 172.21.15.28:0/145502671 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(186..186 src has 1..186) v4 ==== 6101+0+0 (secure 0 0 0) 0x7feb3c032990 con 0x5602d54ef150 2024-02-21T23:12:06.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.109+0000 7feb55216740 1 -- 172.21.15.28:0/145502671 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7feb3c015a40 con 0x5602d54ef150 2024-02-21T23:12:06.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.110+0000 7feb38c00640 1 -- 172.21.15.28:0/145502671 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=209) v2 ==== 24+0+0 (secure 0 0 0) 0x7feb3c039050 con 0x5602d54ef150 2024-02-21T23:12:06.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.110+0000 7feb55216740 10 client.10182 fetch_fsmap learned FSMap version 209 2024-02-21T23:12:06.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.110+0000 7feb55216740 1 -- 172.21.15.28:0/145502671 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=209,monmap=2+}) v3 -- 0x5602d5543cf0 con 0x5602d54ef150 2024-02-21T23:12:06.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.110+0000 7feb38c00640 1 -- 172.21.15.28:0/145502671 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb3c015a40 con 0x5602d54ef150 2024-02-21T23:12:06.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.110+0000 7feb38c00640 1 -- 172.21.15.28:0/145502671 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 209) v1 ==== 3306+0+0 (secure 0 0 0) 0x7feb3c03c020 con 0x5602d54ef150 2024-02-21T23:12:06.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.110+0000 7feb55216740 10 client.10182 fetch_fsmap finished waiting for FSMap version 209 2024-02-21T23:12:06.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.110+0000 7feb55216740 10 client.10182 resolve_mds: resolved name 'b' to daemon mds.b 2024-02-21T23:12:06.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.110+0000 7feb55216740 20 client.10182 populate_metadata read hostname 'smithi028' 2024-02-21T23:12:06.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.110+0000 7feb55216740 1 --2- 172.21.15.28:0/145502671 >> [v2:172.21.15.81:6835/416021079,v1:172.21.15.81:6837/416021079] conn(0x5602d559ce80 0x5602d5598e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:06.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.110+0000 7feb55216740 4 client.10182 mds_command: new command op to 7867 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-21T23:12:06.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.110+0000 7feb55216740 1 -- 172.21.15.28:0/145502671 --> [v2:172.21.15.81:6835/416021079,v1:172.21.15.81:6837/416021079] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5602d5459d00 con 0x5602d559ce80 2024-02-21T23:12:06.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.111+0000 7feb42a00640 1 --2- 172.21.15.28:0/145502671 >> [v2:172.21.15.81:6835/416021079,v1:172.21.15.81:6837/416021079] conn(0x5602d559ce80 0x5602d5598e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:06.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.112+0000 7feb42a00640 1 --2- 172.21.15.28:0/145502671 >> [v2:172.21.15.81:6835/416021079,v1:172.21.15.81:6837/416021079] conn(0x5602d559ce80 0x5602d5598e20 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1079fc8a2f7df071 server_cookie=a6881fe2cbacd6fb in_seq=0 out_seq=0 2024-02-21T23:12:06.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.112+0000 7feb38c00640 10 client.10182 ms_handle_connect on v2:172.21.15.81:6835/416021079 2024-02-21T23:12:06.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.112+0000 7feb38c00640 1 -- 172.21.15.28:0/145502671 <== mds.0 v2:172.21.15.81:6835/416021079 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+334 (crc 0 0 0) 0x5602d5459d00 con 0x5602d559ce80 2024-02-21T23:12:06.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.112+0000 7feb38c00640 10 client.10182 handle_command_reply: tid=0 2024-02-21T23:12:06.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.112+0000 7feb55216740 1 client.10182 shutdown 2024-02-21T23:12:06.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.113+0000 7feb55216740 1 -- 172.21.15.28:0/145502671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d54ef150 msgr2=0x5602d55437b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:06.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.113+0000 7feb55216740 1 --2- 172.21.15.28:0/145502671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d54ef150 0x5602d55437b0 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 crypto rx=0x7feb3c015810 tx=0x7feb3c000ed0 comp rx=0 tx=0).stop 2024-02-21T23:12:06.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.113+0000 7feb55216740 1 -- 172.21.15.28:0/145502671 shutdown_connections 2024-02-21T23:12:06.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.113+0000 7feb55216740 1 --2- 172.21.15.28:0/145502671 >> [v2:172.21.15.81:6835/416021079,v1:172.21.15.81:6837/416021079] conn(0x5602d559ce80 0x5602d5598e20 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:06.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.113+0000 7feb55216740 1 --2- 172.21.15.28:0/145502671 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x5602d552a4d0 0x5602d55964d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:06.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.113+0000 7feb55216740 1 --2- 172.21.15.28:0/145502671 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5602d550a560 0x5602d5595f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:06.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.113+0000 7feb55216740 1 --2- 172.21.15.28:0/145502671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602d54ef150 0x5602d55437b0 unknown :-1 s=CLOSED pgs=2135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:06.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.113+0000 7feb55216740 1 -- 172.21.15.28:0/145502671 >> 172.21.15.28:0/145502671 conn(0x5602d54ede40 msgr2=0x5602d550d9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:06.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.113+0000 7feb38c00640 0 client.10182 ms_handle_reset on v2:172.21.15.81:6835/416021079 2024-02-21T23:12:06.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.113+0000 7feb55216740 1 -- 172.21.15.28:0/145502671 shutdown_connections 2024-02-21T23:12:06.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.113+0000 7feb55216740 1 -- 172.21.15.28:0/145502671 wait complete. 2024-02-21T23:12:06.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.113+0000 7feb55216740 20 client.10182 trim_cache size 0 max 16384 2024-02-21T23:12:06.108 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-21T23:12:06.108 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac", 2024-02-21T23:12:06.108 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-02-21T23:12:06.108 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 7867, 2024-02-21T23:12:06.109 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-02-21T23:12:06.109 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-02-21T23:12:06.109 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-02-21T23:12:06.109 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 3.5285070049999998, 2024-02-21T23:12:06.109 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 209, 2024-02-21T23:12:06.109 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 186, 2024-02-21T23:12:06.109 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 186, 2024-02-21T23:12:06.109 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 6.9718546870000004 2024-02-21T23:12:06.109 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-21T23:12:06.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.114+0000 7feb52400640 1 -- 172.21.15.28:0/1635732342 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7feb2806b500 msgr2=0x7feb2806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:06.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.114+0000 7feb52400640 1 --2- 172.21.15.28:0/1635732342 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7feb2806b500 0x7feb2806d9c0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7feb3c000f60 tx=0x7feb3c007b50 comp rx=0 tx=0).stop 2024-02-21T23:12:06.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.114+0000 7feb52400640 1 -- 172.21.15.28:0/1635732342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb4c0c5a70 msgr2=0x7feb4c0bcb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:06.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.114+0000 7feb52400640 1 --2- 172.21.15.28:0/1635732342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb4c0c5a70 0x7feb4c0bcb30 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 crypto rx=0x7feb3400c820 tx=0x7feb3400cc20 comp rx=0 tx=0).stop 2024-02-21T23:12:06.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.114+0000 7feb52400640 1 -- 172.21.15.28:0/1635732342 shutdown_connections 2024-02-21T23:12:06.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.114+0000 7feb52400640 1 --2- 172.21.15.28:0/1635732342 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7feb2806b500 0x7feb2806d9c0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:06.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.114+0000 7feb52400640 1 --2- 172.21.15.28:0/1635732342 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feb4c156190 0x7feb4c14c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:06.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.114+0000 7feb52400640 1 --2- 172.21.15.28:0/1635732342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb4c0c5a70 0x7feb4c0bcb30 unknown :-1 s=CLOSED pgs=2131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:06.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.114+0000 7feb52400640 1 --2- 172.21.15.28:0/1635732342 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7feb4c06e110 0x7feb4c150610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:06.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.114+0000 7feb52400640 1 -- 172.21.15.28:0/1635732342 >> 172.21.15.28:0/1635732342 conn(0x7feb4c06b820 msgr2=0x7feb4c1511b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:06.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.114+0000 7feb52400640 1 -- 172.21.15.28:0/1635732342 shutdown_connections 2024-02-21T23:12:06.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.114+0000 7feb52400640 1 -- 172.21.15.28:0/1635732342 wait complete. 2024-02-21T23:12:06.120 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:12:06.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.348+0000 7fe237200640 1 Processor -- start 2024-02-21T23:12:06.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.348+0000 7fe237200640 1 -- start start 2024-02-21T23:12:06.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.349+0000 7fe237200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe2380aff90 0x7fe2380c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:06.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.349+0000 7fe237200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2380c74c0 con 0x7fe23806dbf0 2024-02-21T23:12:06.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.349+0000 7fe237200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2380c77a0 con 0x7fe2380c5720 2024-02-21T23:12:06.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.349+0000 7fe237200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2380c7270 con 0x7fe2380aff90 2024-02-21T23:12:06.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.349+0000 7fe235e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe2380aff90 0x7fe2380c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:06.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.349+0000 7fe235e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe2380aff90 0x7fe2380c51e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60146/0 (socket says 172.21.15.28:60146) 2024-02-21T23:12:06.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.349+0000 7fe235e00640 1 -- 172.21.15.28:0/1191024218 learned_addr learned my addr 172.21.15.28:0/1191024218 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:06.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.349+0000 7fe235400640 1 --1- 172.21.15.28:0/1191024218 >> v1:172.21.15.28:6789/0 conn(0x7fe23806dbf0 0x7fe238158ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53748/0 (socket says 172.21.15.28:53748) 2024-02-21T23:12:06.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.350+0000 7fe235e00640 1 -- 172.21.15.28:0/1191024218 >> v1:172.21.15.81:6789/0 conn(0x7fe2380c5720 legacy=0x7fe23815f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:06.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.350+0000 7fe235e00640 1 -- 172.21.15.28:0/1191024218 >> v1:172.21.15.28:6789/0 conn(0x7fe23806dbf0 legacy=0x7fe238158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:06.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.350+0000 7fe235e00640 1 -- 172.21.15.28:0/1191024218 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2380c7a80 con 0x7fe2380aff90 2024-02-21T23:12:06.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.350+0000 7fe234a00640 1 -- 172.21.15.28:0/1191024218 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3590149622 0 0) 0x7fe2380c74c0 con 0x7fe23806dbf0 2024-02-21T23:12:06.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.351+0000 7fe235e00640 1 --2- 172.21.15.28:0/1191024218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2380aff90 0x7fe2380c51e0 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 crypto rx=0x7fe228009e50 tx=0x7fe22802bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a4c2dee15877404 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:06.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.351+0000 7fe234a00640 1 -- 172.21.15.28:0/1191024218 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe22802cb10 con 0x7fe2380aff90 2024-02-21T23:12:06.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.351+0000 7fe234a00640 1 -- 172.21.15.28:0/1191024218 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe22802ccb0 con 0x7fe2380aff90 2024-02-21T23:12:06.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.351+0000 7fe234a00640 1 -- 172.21.15.28:0/1191024218 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe22802ce60 con 0x7fe2380aff90 2024-02-21T23:12:06.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.352+0000 7fe237200640 1 -- 172.21.15.28:0/1191024218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2380aff90 msgr2=0x7fe2380c51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:06.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.352+0000 7fe237200640 1 --2- 172.21.15.28:0/1191024218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2380aff90 0x7fe2380c51e0 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 crypto rx=0x7fe228009e50 tx=0x7fe22802bf70 comp rx=0 tx=0).stop 2024-02-21T23:12:06.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.352+0000 7fe237200640 1 -- 172.21.15.28:0/1191024218 shutdown_connections 2024-02-21T23:12:06.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.352+0000 7fe237200640 1 --2- 172.21.15.28:0/1191024218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2380aff90 0x7fe2380c51e0 unknown :-1 s=CLOSED pgs=2137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:06.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.352+0000 7fe237200640 1 -- 172.21.15.28:0/1191024218 >> 172.21.15.28:0/1191024218 conn(0x7fe2380ac370 msgr2=0x7fe2380ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:06.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.352+0000 7fe237200640 1 -- 172.21.15.28:0/1191024218 shutdown_connections 2024-02-21T23:12:06.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.352+0000 7fe237200640 1 -- 172.21.15.28:0/1191024218 wait complete. 2024-02-21T23:12:06.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.354+0000 7fe237200640 1 Processor -- start 2024-02-21T23:12:06.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.354+0000 7fe237200640 1 -- start start 2024-02-21T23:12:06.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.355+0000 7fe237200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe23806dbf0 0x7fe2380c09a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:06.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.355+0000 7fe237200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2380aff90 0x7fe238156e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:06.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.355+0000 7fe237200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2380c5720 0x7fe2381edd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:06.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.355+0000 7fe237200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2380c8760 con 0x7fe2380aff90 2024-02-21T23:12:06.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.355+0000 7fe237200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe238158630 con 0x7fe2380c5720 2024-02-21T23:12:06.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.355+0000 7fe237200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2380c88e0 con 0x7fe23806dbf0 2024-02-21T23:12:06.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.355+0000 7fe235400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2380aff90 0x7fe238156e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:06.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.355+0000 7fe235400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2380aff90 0x7fe238156e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60158/0 (socket says 172.21.15.28:60158) 2024-02-21T23:12:06.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.355+0000 7fe235400640 1 -- 172.21.15.28:0/2589759804 learned_addr learned my addr 172.21.15.28:0/2589759804 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:06.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.355+0000 7fe236800640 1 --2- 172.21.15.28:0/2589759804 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2380c5720 0x7fe2381edd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:06.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.355+0000 7fe235e00640 1 --2- 172.21.15.28:0/2589759804 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe23806dbf0 0x7fe2380c09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:06.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.356+0000 7fe235400640 1 -- 172.21.15.28:0/2589759804 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe23806dbf0 msgr2=0x7fe2380c09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:06.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.356+0000 7fe235400640 1 --2- 172.21.15.28:0/2589759804 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe23806dbf0 0x7fe2380c09a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:06.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.356+0000 7fe235400640 1 -- 172.21.15.28:0/2589759804 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2380c5720 msgr2=0x7fe2381edd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:06.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.356+0000 7fe235400640 1 --2- 172.21.15.28:0/2589759804 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2380c5720 0x7fe2381edd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:06.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.356+0000 7fe235400640 1 -- 172.21.15.28:0/2589759804 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2380c4a20 con 0x7fe2380aff90 2024-02-21T23:12:06.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.356+0000 7fe235e00640 1 --2- 172.21.15.28:0/2589759804 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe23806dbf0 0x7fe2380c09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:12:06.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.356+0000 7fe235400640 1 --2- 172.21.15.28:0/2589759804 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2380aff90 0x7fe238156e40 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 crypto rx=0x7fe22400c4b0 tx=0x7fe22400c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:06.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.356+0000 7fe22e000640 1 -- 172.21.15.28:0/2589759804 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe224018070 con 0x7fe2380aff90 2024-02-21T23:12:06.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.356+0000 7fe22e000640 1 -- 172.21.15.28:0/2589759804 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe224007500 con 0x7fe2380aff90 2024-02-21T23:12:06.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.356+0000 7fe22e000640 1 -- 172.21.15.28:0/2589759804 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe224004a20 con 0x7fe2380aff90 2024-02-21T23:12:06.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.356+0000 7fe237200640 1 -- 172.21.15.28:0/2589759804 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2380c3800 con 0x7fe2380aff90 2024-02-21T23:12:06.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.356+0000 7fe237200640 1 -- 172.21.15.28:0/2589759804 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe238157380 con 0x7fe2380aff90 2024-02-21T23:12:06.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.358+0000 7fe237200640 1 -- 172.21.15.28:0/2589759804 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2380c4550 con 0x7fe2380aff90 2024-02-21T23:12:06.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.359+0000 7fe22e000640 1 -- 172.21.15.28:0/2589759804 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe22401b070 con 0x7fe2380aff90 2024-02-21T23:12:06.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.359+0000 7fe22e000640 1 --2- 172.21.15.28:0/2589759804 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe21806b500 0x7fe21806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:06.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.360+0000 7fe22e000640 1 -- 172.21.15.28:0/2589759804 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe2240895f0 con 0x7fe2380aff90 2024-02-21T23:12:06.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.360+0000 7fe235e00640 1 --2- 172.21.15.28:0/2589759804 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe21806b500 0x7fe21806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:06.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.361+0000 7fe235e00640 1 --2- 172.21.15.28:0/2589759804 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe21806b500 0x7fe21806d9c0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fe228035000 tx=0x7fe228030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:06.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.364+0000 7fe22e000640 1 -- 172.21.15.28:0/2589759804 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe224058390 con 0x7fe2380aff90 2024-02-21T23:12:06.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.602+0000 7fe237200640 1 -- 172.21.15.28:0/2589759804 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe23804f290 con 0x7fe2380aff90 2024-02-21T23:12:06.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.603+0000 7fe22e000640 1 -- 172.21.15.28:0/2589759804 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 209 v209) v1 ==== 95+0+4959 (secure 0 0 0) 0x7fe224057c90 con 0x7fe2380aff90 2024-02-21T23:12:06.591 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:12:06.591 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":209,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6848,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3439896042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3439896042},{"type":"v1","addr":"172.21.15.28:6835","nonce":3439896042}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":196},{"gid":7729,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/3213249808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3213249808},{"type":"v1","addr":"172.21.15.28:6833","nonce":3213249808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":195}],"filesystems":[{"mdsmap":{"epoch":209,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:11:38.384619+0000","modified":"2024-02-21T23:12:03.626987+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7867,"mds_1":7735},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7735":{"gid":7735,"name":"d","rank":1,"incarnation":208,"state":"up:active","state_seq":9,"addr":"172.21.15.81:6836/2895690310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2895690310},{"type":"v1","addr":"172.21.15.81:6836","nonce":2895690310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7867":{"gid":7867,"name":"b","rank":0,"incarnation":206,"state":"up:active","state_seq":2,"addr":"172.21.15.81:6837/416021079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":416021079},{"type":"v1","addr":"172.21.15.81:6837","nonce":416021079}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-02-21T23:12:06.591 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 209 2024-02-21T23:12:06.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.605+0000 7fe237200640 1 -- 172.21.15.28:0/2589759804 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe21806b500 msgr2=0x7fe21806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:06.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.605+0000 7fe237200640 1 --2- 172.21.15.28:0/2589759804 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe21806b500 0x7fe21806d9c0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fe228035000 tx=0x7fe228030000 comp rx=0 tx=0).stop 2024-02-21T23:12:06.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.606+0000 7fe237200640 1 -- 172.21.15.28:0/2589759804 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2380aff90 msgr2=0x7fe238156e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:06.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.606+0000 7fe237200640 1 --2- 172.21.15.28:0/2589759804 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2380aff90 0x7fe238156e40 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 crypto rx=0x7fe22400c4b0 tx=0x7fe22400c970 comp rx=0 tx=0).stop 2024-02-21T23:12:06.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.606+0000 7fe237200640 1 -- 172.21.15.28:0/2589759804 shutdown_connections 2024-02-21T23:12:06.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.606+0000 7fe237200640 1 --2- 172.21.15.28:0/2589759804 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe21806b500 0x7fe21806d9c0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:06.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.606+0000 7fe237200640 1 --2- 172.21.15.28:0/2589759804 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2380c5720 0x7fe2381edd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:06.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.606+0000 7fe237200640 1 --2- 172.21.15.28:0/2589759804 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2380aff90 0x7fe238156e40 unknown :-1 s=CLOSED pgs=2138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:06.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.606+0000 7fe237200640 1 --2- 172.21.15.28:0/2589759804 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe23806dbf0 0x7fe2380c09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:06.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.606+0000 7fe237200640 1 -- 172.21.15.28:0/2589759804 >> 172.21.15.28:0/2589759804 conn(0x7fe2380ac370 msgr2=0x7fe2381562e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:06.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.606+0000 7fe237200640 1 -- 172.21.15.28:0/2589759804 shutdown_connections 2024-02-21T23:12:06.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:06.606+0000 7fe237200640 1 -- 172.21.15.28:0/2589759804 wait complete. 2024-02-21T23:12:07.017 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 209, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:11:38.384619+0000', 'modified': '2024-02-21T23:12:03.626987+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 7867, 'mds_1': 7735}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7735': {'gid': 7735, 'name': 'd', 'rank': 1, 'incarnation': 208, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.81:6836/2895690310', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6834', 'nonce': 2895690310}, {'type': 'v1', 'addr': '172.21.15.81:6836', 'nonce': 2895690310}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_7867': {'gid': 7867, 'name': 'b', 'rank': 0, 'incarnation': 206, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.81:6837/416021079', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6835', 'nonce': 416021079}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 416021079}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:12:07.017 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 2/2 2024-02-21T23:12:07.017 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-02-21T23:12:07.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.263+0000 7f2bc3200640 1 Processor -- start 2024-02-21T23:12:07.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.263+0000 7f2bc3200640 1 -- start start 2024-02-21T23:12:07.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.264+0000 7f2bc3200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f2bc406d410 0x7f2bc40a8e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:07.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.264+0000 7f2bc3200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2bc40c43e0 0x7f2bc40c5a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:07.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.264+0000 7f2bc3200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bc4149c10 con 0x7f2bc4149830 2024-02-21T23:12:07.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.264+0000 7f2bc3200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bc41537d0 con 0x7f2bc40c43e0 2024-02-21T23:12:07.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.264+0000 7f2bc3200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bc4153ab0 con 0x7f2bc406d410 2024-02-21T23:12:07.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.264+0000 7f2bc1e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f2bc4149830 0x7f2bc40c0ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53762/0 (socket says 172.21.15.28:53762) 2024-02-21T23:12:07.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.265+0000 7f2bc1e00640 1 -- 172.21.15.28:0/1243172820 learned_addr learned my addr 172.21.15.28:0/1243172820 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:12:07.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.265+0000 7f2bc2800640 1 --2- 172.21.15.28:0/1243172820 >> v2:172.21.15.28:3300/0 conn(0x7f2bc40c43e0 0x7f2bc40c5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:07.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.265+0000 7f2bc1400640 1 --2- 172.21.15.28:0/1243172820 >> v2:172.21.15.81:3300/0 conn(0x7f2bc406d410 0x7f2bc40a8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:07.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.265+0000 7f2bc0a00640 1 -- 172.21.15.28:0/1243172820 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1830586576 0 0) 0x7f2bc4149c10 con 0x7f2bc4149830 2024-02-21T23:12:07.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.265+0000 7f2bc0a00640 1 -- 172.21.15.28:0/1243172820 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ba8003520 con 0x7f2bc4149830 2024-02-21T23:12:07.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.265+0000 7f2bc2800640 1 -- 172.21.15.28:0/1243172820 >> v2:172.21.15.81:3300/0 conn(0x7f2bc406d410 msgr2=0x7f2bc40a8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:07.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.265+0000 7f2bc2800640 1 --2- 172.21.15.28:0/1243172820 >> v2:172.21.15.81:3300/0 conn(0x7f2bc406d410 0x7f2bc40a8e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:07.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.265+0000 7f2bc2800640 1 -- 172.21.15.28:0/1243172820 >> v1:172.21.15.28:6789/0 conn(0x7f2bc4149830 legacy=0x7f2bc40c0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:07.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.265+0000 7f2bc2800640 1 -- 172.21.15.28:0/1243172820 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bc40c4fa0 con 0x7f2bc40c43e0 2024-02-21T23:12:07.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.266+0000 7f2bc2800640 1 --2- 172.21.15.28:0/1243172820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bc40c43e0 0x7f2bc40c5a80 secure :-1 s=READY pgs=2140 cs=0 l=1 rev1=1 crypto rx=0x7f2bc40733f0 tx=0x7f2bbc02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0235c2582629534 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:07.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.267+0000 7f2bc0a00640 1 -- 172.21.15.28:0/1243172820 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bbc034070 con 0x7f2bc40c43e0 2024-02-21T23:12:07.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.267+0000 7f2bc0a00640 1 -- 172.21.15.28:0/1243172820 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bbc030610 con 0x7f2bc40c43e0 2024-02-21T23:12:07.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.267+0000 7f2bc0a00640 1 -- 172.21.15.28:0/1243172820 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bbc0309e0 con 0x7f2bc40c43e0 2024-02-21T23:12:07.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.267+0000 7f2bc3200640 1 -- 172.21.15.28:0/1243172820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bc40c43e0 msgr2=0x7f2bc40c5a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:07.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.267+0000 7f2bc3200640 1 --2- 172.21.15.28:0/1243172820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bc40c43e0 0x7f2bc40c5a80 secure :-1 s=READY pgs=2140 cs=0 l=1 rev1=1 crypto rx=0x7f2bc40733f0 tx=0x7f2bbc02b960 comp rx=0 tx=0).stop 2024-02-21T23:12:07.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.267+0000 7f2bc3200640 1 -- 172.21.15.28:0/1243172820 shutdown_connections 2024-02-21T23:12:07.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.267+0000 7f2bc3200640 1 --2- 172.21.15.28:0/1243172820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bc40c43e0 0x7f2bc40c5a80 unknown :-1 s=CLOSED pgs=2140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:07.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.267+0000 7f2bc3200640 1 --2- 172.21.15.28:0/1243172820 >> v2:172.21.15.81:3300/0 conn(0x7f2bc406d410 0x7f2bc40a8e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:07.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.267+0000 7f2bc3200640 1 -- 172.21.15.28:0/1243172820 >> 172.21.15.28:0/1243172820 conn(0x7f2bc40a8320 msgr2=0x7f2bc40a8730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:07.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.267+0000 7f2bc3200640 1 -- 172.21.15.28:0/1243172820 shutdown_connections 2024-02-21T23:12:07.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.268+0000 7f2bc3200640 1 -- 172.21.15.28:0/1243172820 wait complete. 2024-02-21T23:12:07.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.269+0000 7f2bc3200640 1 Processor -- start 2024-02-21T23:12:07.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.269+0000 7f2bc3200640 1 -- start start 2024-02-21T23:12:07.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.270+0000 7f2bc3200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2bc406d410 0x7f2bc4156b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:07.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.270+0000 7f2bc3200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2bc40c43e0 0x7f2bc4154140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:07.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.270+0000 7f2bc3200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bc4149830 0x7f2bc4154680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:07.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.270+0000 7f2bc3200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bc40c15f0 con 0x7f2bc4149830 2024-02-21T23:12:07.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.270+0000 7f2bc3200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bc40c5560 con 0x7f2bc406d410 2024-02-21T23:12:07.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.270+0000 7f2bc3200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bc40c1770 con 0x7f2bc40c43e0 2024-02-21T23:12:07.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.271+0000 7f2bc2800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bc4149830 0x7f2bc4154680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:07.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.271+0000 7f2bc2800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bc4149830 0x7f2bc4154680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60172/0 (socket says 172.21.15.28:60172) 2024-02-21T23:12:07.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.271+0000 7f2bc2800640 1 -- 172.21.15.28:0/810329043 learned_addr learned my addr 172.21.15.28:0/810329043 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:07.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.271+0000 7f2bc1400640 1 --2- 172.21.15.28:0/810329043 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2bc40c43e0 0x7f2bc4154140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:07.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.271+0000 7f2bc1e00640 1 --2- 172.21.15.28:0/810329043 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2bc406d410 0x7f2bc4156b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:07.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.271+0000 7f2bc2800640 1 -- 172.21.15.28:0/810329043 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2bc40c43e0 msgr2=0x7f2bc4154140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:07.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.271+0000 7f2bc2800640 1 --2- 172.21.15.28:0/810329043 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2bc40c43e0 0x7f2bc4154140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:07.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.271+0000 7f2bc2800640 1 -- 172.21.15.28:0/810329043 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2bc406d410 msgr2=0x7f2bc4156b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:07.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.271+0000 7f2bc2800640 1 --2- 172.21.15.28:0/810329043 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2bc406d410 0x7f2bc4156b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:07.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.271+0000 7f2bc2800640 1 -- 172.21.15.28:0/810329043 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bc41e7930 con 0x7f2bc4149830 2024-02-21T23:12:07.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.271+0000 7f2bc1e00640 1 --2- 172.21.15.28:0/810329043 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2bc406d410 0x7f2bc4156b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:12:07.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.271+0000 7f2bc2800640 1 --2- 172.21.15.28:0/810329043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bc4149830 0x7f2bc4154680 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc009860 tx=0x7f2bbc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:07.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.272+0000 7f2bba000640 1 -- 172.21.15.28:0/810329043 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bbc046020 con 0x7f2bc4149830 2024-02-21T23:12:07.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.272+0000 7f2bba000640 1 -- 172.21.15.28:0/810329043 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bbc030410 con 0x7f2bc4149830 2024-02-21T23:12:07.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.272+0000 7f2bba000640 1 -- 172.21.15.28:0/810329043 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bbc034040 con 0x7f2bc4149830 2024-02-21T23:12:07.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.272+0000 7f2bc3200640 1 -- 172.21.15.28:0/810329043 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bc41e7c10 con 0x7f2bc4149830 2024-02-21T23:12:07.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.272+0000 7f2bc3200640 1 -- 172.21.15.28:0/810329043 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bc41e81d0 con 0x7f2bc4149830 2024-02-21T23:12:07.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.274+0000 7f2bba000640 1 -- 172.21.15.28:0/810329043 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2bbc037070 con 0x7f2bc4149830 2024-02-21T23:12:07.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.275+0000 7f2bba000640 1 --2- 172.21.15.28:0/810329043 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2ba406b4d0 0x7f2ba406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:07.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.275+0000 7f2bba000640 1 -- 172.21.15.28:0/810329043 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f2bbc0abff0 con 0x7f2bc4149830 2024-02-21T23:12:07.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.275+0000 7f2bc1e00640 1 --2- 172.21.15.28:0/810329043 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2ba406b4d0 0x7f2ba406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:07.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.276+0000 7f2bcb9c4740 1 Processor -- start 2024-02-21T23:12:07.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.276+0000 7f2bcb9c4740 1 -- start start 2024-02-21T23:12:07.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.277+0000 7f2bc1e00640 1 --2- 172.21.15.28:0/810329043 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2ba406b4d0 0x7f2ba406d990 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f2bb4005c00 tx=0x7f2bb4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:07.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.277+0000 7f2bcb9c4740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555a59733020 0x555a59733400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:07.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.277+0000 7f2bcb9c4740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x555a59734330 0x555a5976f170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:07.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.277+0000 7f2bcb9c4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a5976f6b0 0x555a5976d720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:07.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.277+0000 7f2bcb9c4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a59697470 con 0x555a5976f6b0 2024-02-21T23:12:07.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.277+0000 7f2bcb9c4740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a59640af0 con 0x555a59733020 2024-02-21T23:12:07.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.277+0000 7f2bcb9c4740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x555a59693510 con 0x555a59734330 2024-02-21T23:12:07.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.277+0000 7f2bc2800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a5976f6b0 0x555a5976d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:07.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.277+0000 7f2bc2800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a5976f6b0 0x555a5976d720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60174/0 (socket says 172.21.15.28:60174) 2024-02-21T23:12:07.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.277+0000 7f2bc2800640 1 -- 172.21.15.28:0/4209427782 learned_addr learned my addr 172.21.15.28:0/4209427782 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:07.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.277+0000 7f2bc1400640 1 --2- 172.21.15.28:0/4209427782 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x555a59734330 0x555a5976f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:07.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.277+0000 7f2bc1e00640 1 --2- 172.21.15.28:0/4209427782 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555a59733020 0x555a59733400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:07.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.278+0000 7f2bc2800640 1 -- 172.21.15.28:0/4209427782 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x555a59734330 msgr2=0x555a5976f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:07.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.278+0000 7f2bc2800640 1 --2- 172.21.15.28:0/4209427782 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x555a59734330 0x555a5976f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:07.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.278+0000 7f2bc2800640 1 -- 172.21.15.28:0/4209427782 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555a59733020 msgr2=0x555a59733400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:07.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.278+0000 7f2bc2800640 1 --2- 172.21.15.28:0/4209427782 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555a59733020 0x555a59733400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:07.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.278+0000 7f2bc2800640 1 -- 172.21.15.28:0/4209427782 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555a59647d90 con 0x555a5976f6b0 2024-02-21T23:12:07.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.278+0000 7f2bc1400640 1 --2- 172.21.15.28:0/4209427782 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x555a59734330 0x555a5976f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:12:07.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.278+0000 7f2bc2800640 1 --2- 172.21.15.28:0/4209427782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a5976f6b0 0x555a5976d720 secure :-1 s=READY pgs=2142 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc02bef0 tx=0x7f2bbc0305e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb37a36d96b95507 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:07.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.278+0000 7f2bae600640 1 -- 172.21.15.28:0/4209427782 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bbc092020 con 0x555a5976f6b0 2024-02-21T23:12:07.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.278+0000 7f2bae600640 1 -- 172.21.15.28:0/4209427782 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bbc07a5f0 con 0x555a5976f6b0 2024-02-21T23:12:07.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.278+0000 7f2bae600640 1 -- 172.21.15.28:0/4209427782 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bbc07e740 con 0x555a5976f6b0 2024-02-21T23:12:07.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.279+0000 7f2bcb9c4740 1 -- 172.21.15.28:0/4209427782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a5976f6b0 msgr2=0x555a5976d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:07.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.279+0000 7f2bcb9c4740 1 --2- 172.21.15.28:0/4209427782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a5976f6b0 0x555a5976d720 secure :-1 s=READY pgs=2142 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc02bef0 tx=0x7f2bbc0305e0 comp rx=0 tx=0).stop 2024-02-21T23:12:07.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.279+0000 7f2bcb9c4740 1 -- 172.21.15.28:0/4209427782 shutdown_connections 2024-02-21T23:12:07.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.279+0000 7f2bcb9c4740 1 --2- 172.21.15.28:0/4209427782 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x555a59734330 0x555a5976f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:07.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.279+0000 7f2bcb9c4740 1 --2- 172.21.15.28:0/4209427782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a5976f6b0 0x555a5976d720 unknown :-1 s=CLOSED pgs=2142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:07.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.279+0000 7f2bcb9c4740 1 --2- 172.21.15.28:0/4209427782 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555a59733020 0x555a59733400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:07.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.279+0000 7f2bcb9c4740 1 -- 172.21.15.28:0/4209427782 >> 172.21.15.28:0/4209427782 conn(0x555a5972f970 msgr2=0x555a59754380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:07.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.279+0000 7f2bcb9c4740 1 -- 172.21.15.28:0/4209427782 shutdown_connections 2024-02-21T23:12:07.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.279+0000 7f2bcb9c4740 1 -- 172.21.15.28:0/4209427782 wait complete. 2024-02-21T23:12:07.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.280+0000 7f2bcb9c4740 1 -- start start 2024-02-21T23:12:07.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.280+0000 7f2bcb9c4740 1 Processor -- start 2024-02-21T23:12:07.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.281+0000 7f2bcb9c4740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555a59734330 0x555a59782bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:07.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.281+0000 7f2bcb9c4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a5976f6b0 0x555a59753280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:07.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.281+0000 7f2bcb9c4740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x555a597537c0 0x555a5976bc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:07.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.281+0000 7f2bcb9c4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a59697470 con 0x555a5976f6b0 2024-02-21T23:12:07.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.281+0000 7f2bcb9c4740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a59636040 con 0x555a59734330 2024-02-21T23:12:07.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.281+0000 7f2bcb9c4740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x555a59756250 con 0x555a597537c0 2024-02-21T23:12:07.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.281+0000 7f2bcb9c4740 10 client.0 _ll_register_callbacks cb 0x555a59653920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:12:07.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.281+0000 7f2bc1400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a5976f6b0 0x555a59753280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:07.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.281+0000 7f2bc1400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a5976f6b0 0x555a59753280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60176/0 (socket says 172.21.15.28:60176) 2024-02-21T23:12:07.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.281+0000 7f2bc1400640 1 -- 172.21.15.28:0/1378116057 learned_addr learned my addr 172.21.15.28:0/1378116057 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:07.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.281+0000 7f2bc2800640 1 --2- 172.21.15.28:0/1378116057 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x555a597537c0 0x555a5976bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:07.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.282+0000 7f2bc1400640 1 -- 172.21.15.28:0/1378116057 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x555a597537c0 msgr2=0x555a5976bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:07.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.282+0000 7f2bc1400640 1 --2- 172.21.15.28:0/1378116057 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x555a597537c0 0x555a5976bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:07.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.282+0000 7f2bc1e00640 1 --2- 172.21.15.28:0/1378116057 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555a59734330 0x555a59782bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:07.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.282+0000 7f2bc1400640 1 -- 172.21.15.28:0/1378116057 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555a59734330 msgr2=0x555a59782bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:07.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.282+0000 7f2bc1400640 1 --2- 172.21.15.28:0/1378116057 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555a59734330 0x555a59782bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:07.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.282+0000 7f2bc1400640 1 -- 172.21.15.28:0/1378116057 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555a59655aa0 con 0x555a5976f6b0 2024-02-21T23:12:07.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.282+0000 7f2bc1400640 1 --2- 172.21.15.28:0/1378116057 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a5976f6b0 0x555a59753280 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 crypto rx=0x7f2bb00125e0 tx=0x7f2bb0012aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53c23d809b9fe528 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:07.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.282+0000 7f2ba2a00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:12:07.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.282+0000 7f2ba2a00640 1 -- 172.21.15.28:0/1378116057 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bb00108e0 con 0x555a5976f6b0 2024-02-21T23:12:07.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.282+0000 7f2ba2a00640 1 -- 172.21.15.28:0/1378116057 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(186..186 src has 1..186) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f2bb0010a80 con 0x555a5976f6b0 2024-02-21T23:12:07.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.282+0000 7f2bcb9c4740 1 -- 172.21.15.28:0/1378116057 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2bbc07a5f0 con 0x555a5976f6b0 2024-02-21T23:12:07.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.282+0000 7f2ba2a00640 1 -- 172.21.15.28:0/1378116057 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=209) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2bb0010c80 con 0x555a5976f6b0 2024-02-21T23:12:07.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.283+0000 7f2bcb9c4740 10 client.10206 fetch_fsmap learned FSMap version 209 2024-02-21T23:12:07.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.283+0000 7f2bcb9c4740 1 -- 172.21.15.28:0/1378116057 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=209,monmap=2+}) v3 -- 0x555a593ac2d0 con 0x555a5976f6b0 2024-02-21T23:12:07.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.283+0000 7f2ba2a00640 1 -- 172.21.15.28:0/1378116057 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bbc07a5f0 con 0x555a5976f6b0 2024-02-21T23:12:07.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.283+0000 7f2ba2a00640 1 -- 172.21.15.28:0/1378116057 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 209) v1 ==== 3306+0+0 (secure 0 0 0) 0x7f2bb002a080 con 0x555a5976f6b0 2024-02-21T23:12:07.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.283+0000 7f2bcb9c4740 10 client.10206 fetch_fsmap finished waiting for FSMap version 209 2024-02-21T23:12:07.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.283+0000 7f2bcb9c4740 10 client.10206 resolve_mds: resolved name 'd' to daemon mds.d 2024-02-21T23:12:07.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.283+0000 7f2bcb9c4740 20 client.10206 populate_metadata read hostname 'smithi028' 2024-02-21T23:12:07.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.283+0000 7f2bcb9c4740 1 --2- 172.21.15.28:0/1378116057 >> [v2:172.21.15.81:6834/2895690310,v1:172.21.15.81:6836/2895690310] conn(0x555a5977f950 0x555a5977fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:07.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.283+0000 7f2bcb9c4740 4 client.10206 mds_command: new command op to 7735 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-21T23:12:07.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.283+0000 7f2bcb9c4740 1 -- 172.21.15.28:0/1378116057 --> [v2:172.21.15.81:6834/2895690310,v1:172.21.15.81:6836/2895690310] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555a596a49b0 con 0x555a5977f950 2024-02-21T23:12:07.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.283+0000 7f2bc1e00640 1 --2- 172.21.15.28:0/1378116057 >> [v2:172.21.15.81:6834/2895690310,v1:172.21.15.81:6836/2895690310] conn(0x555a5977f950 0x555a5977fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:07.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.284+0000 7f2bc1e00640 1 --2- 172.21.15.28:0/1378116057 >> [v2:172.21.15.81:6834/2895690310,v1:172.21.15.81:6836/2895690310] conn(0x555a5977f950 0x555a5977fd30 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7b32ae118bbb0c2c server_cookie=b3d1e5ee63584815 in_seq=0 out_seq=0 2024-02-21T23:12:07.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.284+0000 7f2ba2a00640 10 client.10206 ms_handle_connect on v2:172.21.15.81:6834/2895690310 2024-02-21T23:12:07.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.287+0000 7f2ba2a00640 1 -- 172.21.15.28:0/1378116057 <== mds.1 v2:172.21.15.81:6834/2895690310 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x555a596a49b0 con 0x555a5977f950 2024-02-21T23:12:07.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.287+0000 7f2ba2a00640 10 client.10206 handle_command_reply: tid=0 2024-02-21T23:12:07.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.287+0000 7f2bcb9c4740 1 client.10206 shutdown 2024-02-21T23:12:07.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.287+0000 7f2bcb9c4740 1 -- 172.21.15.28:0/1378116057 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a5976f6b0 msgr2=0x555a59753280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:07.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.287+0000 7f2bcb9c4740 1 --2- 172.21.15.28:0/1378116057 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a5976f6b0 0x555a59753280 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 crypto rx=0x7f2bb00125e0 tx=0x7f2bb0012aa0 comp rx=0 tx=0).stop 2024-02-21T23:12:07.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.287+0000 7f2bcb9c4740 1 -- 172.21.15.28:0/1378116057 shutdown_connections 2024-02-21T23:12:07.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.287+0000 7f2bcb9c4740 1 --2- 172.21.15.28:0/1378116057 >> [v2:172.21.15.81:6834/2895690310,v1:172.21.15.81:6836/2895690310] conn(0x555a5977f950 0x555a5977fd30 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:07.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.287+0000 7f2bcb9c4740 1 --2- 172.21.15.28:0/1378116057 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x555a597537c0 0x555a5976bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:07.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.287+0000 7f2bcb9c4740 1 --2- 172.21.15.28:0/1378116057 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a5976f6b0 0x555a59753280 unknown :-1 s=CLOSED pgs=2143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:07.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.287+0000 7f2bcb9c4740 1 --2- 172.21.15.28:0/1378116057 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555a59734330 0x555a59782bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:07.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.287+0000 7f2ba2a00640 0 client.10206 ms_handle_reset on v2:172.21.15.81:6834/2895690310 2024-02-21T23:12:07.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.287+0000 7f2bcb9c4740 1 -- 172.21.15.28:0/1378116057 >> 172.21.15.28:0/1378116057 conn(0x555a59733020 msgr2=0x555a5976e080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:07.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.287+0000 7f2bcb9c4740 1 -- 172.21.15.28:0/1378116057 shutdown_connections 2024-02-21T23:12:07.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.287+0000 7f2bcb9c4740 1 -- 172.21.15.28:0/1378116057 wait complete. 2024-02-21T23:12:07.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.287+0000 7f2bcb9c4740 20 client.10206 trim_cache size 0 max 16384 2024-02-21T23:12:07.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.305+0000 7f2bcb9c4740 1 Processor -- start 2024-02-21T23:12:07.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.305+0000 7f2bcb9c4740 1 -- start start 2024-02-21T23:12:07.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.306+0000 7f2bcb9c4740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x555a59734330 0x555a5977c500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:07.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.306+0000 7f2bcb9c4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a597537c0 0x555a5977ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:07.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.306+0000 7f2bcb9c4740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555a5976f6b0 0x555a5972c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:07.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.306+0000 7f2bcb9c4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bc41f78e0 con 0x555a597537c0 2024-02-21T23:12:07.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.306+0000 7f2bcb9c4740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bc41eea80 con 0x555a5976f6b0 2024-02-21T23:12:07.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.306+0000 7f2bcb9c4740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bc41eac40 con 0x555a59734330 2024-02-21T23:12:07.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.306+0000 7f2bc1400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a597537c0 0x555a5977ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:07.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.306+0000 7f2bc1400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a597537c0 0x555a5977ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60182/0 (socket says 172.21.15.28:60182) 2024-02-21T23:12:07.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.306+0000 7f2bc1400640 1 -- 172.21.15.28:0/3285076546 learned_addr learned my addr 172.21.15.28:0/3285076546 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:07.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.307+0000 7f2bc1e00640 1 --2- 172.21.15.28:0/3285076546 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x555a59734330 0x555a5977c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:07.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.307+0000 7f2bc2800640 1 --2- 172.21.15.28:0/3285076546 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555a5976f6b0 0x555a5972c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:07.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.307+0000 7f2bc1400640 1 -- 172.21.15.28:0/3285076546 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x555a59734330 msgr2=0x555a5977c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:07.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.307+0000 7f2bc1400640 1 --2- 172.21.15.28:0/3285076546 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x555a59734330 0x555a5977c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:07.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.307+0000 7f2bc1400640 1 -- 172.21.15.28:0/3285076546 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555a5976f6b0 msgr2=0x555a5972c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:07.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.307+0000 7f2bc1400640 1 --2- 172.21.15.28:0/3285076546 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555a5976f6b0 0x555a5972c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:07.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.307+0000 7f2bc1400640 1 -- 172.21.15.28:0/3285076546 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555a594766b0 con 0x555a597537c0 2024-02-21T23:12:07.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.307+0000 7f2bc1400640 1 --2- 172.21.15.28:0/3285076546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a597537c0 0x555a5977ca40 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 crypto rx=0x7f2bb0012ae0 tx=0x7f2bb0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c47b7f547dde65f5 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:07.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.307+0000 7f2badc00640 1 -- 172.21.15.28:0/3285076546 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bb00108e0 con 0x555a597537c0 2024-02-21T23:12:07.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.308+0000 7f2badc00640 1 -- 172.21.15.28:0/3285076546 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bb002d070 con 0x555a597537c0 2024-02-21T23:12:07.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.308+0000 7f2badc00640 1 -- 172.21.15.28:0/3285076546 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bb0005990 con 0x555a597537c0 2024-02-21T23:12:07.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.308+0000 7f2bcb9c4740 1 -- 172.21.15.28:0/3285076546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a597537c0 msgr2=0x555a5977ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:07.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.308+0000 7f2bcb9c4740 1 --2- 172.21.15.28:0/3285076546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a597537c0 0x555a5977ca40 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 crypto rx=0x7f2bb0012ae0 tx=0x7f2bb0017000 comp rx=0 tx=0).stop 2024-02-21T23:12:07.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.308+0000 7f2bcb9c4740 1 -- 172.21.15.28:0/3285076546 shutdown_connections 2024-02-21T23:12:07.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.308+0000 7f2bcb9c4740 1 --2- 172.21.15.28:0/3285076546 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555a5976f6b0 0x555a5972c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:07.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.308+0000 7f2bcb9c4740 1 --2- 172.21.15.28:0/3285076546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a597537c0 0x555a5977ca40 unknown :-1 s=CLOSED pgs=2144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:07.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.308+0000 7f2bcb9c4740 1 --2- 172.21.15.28:0/3285076546 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x555a59734330 0x555a5977c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:07.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.308+0000 7f2bcb9c4740 1 -- 172.21.15.28:0/3285076546 >> 172.21.15.28:0/3285076546 conn(0x555a59733020 msgr2=0x555a59728ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:07.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.308+0000 7f2bcb9c4740 1 -- 172.21.15.28:0/3285076546 shutdown_connections 2024-02-21T23:12:07.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.308+0000 7f2bcb9c4740 1 -- 172.21.15.28:0/3285076546 wait complete. 2024-02-21T23:12:07.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.309+0000 7f2bcb9c4740 1 -- start start 2024-02-21T23:12:07.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.309+0000 7f2bcb9c4740 1 Processor -- start 2024-02-21T23:12:07.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.310+0000 7f2bcb9c4740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x555a59734330 0x555a597827f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:07.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.310+0000 7f2bcb9c4740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555a597537c0 0x555a5972cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:07.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.310+0000 7f2bcb9c4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a5976f6b0 0x555a59780920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:07.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.310+0000 7f2bcb9c4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a59773af0 con 0x555a5976f6b0 2024-02-21T23:12:07.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.310+0000 7f2bcb9c4740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a59753da0 con 0x555a597537c0 2024-02-21T23:12:07.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.310+0000 7f2bcb9c4740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x555a5974f920 con 0x555a59734330 2024-02-21T23:12:07.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.310+0000 7f2bcb9c4740 10 client.0 _ll_register_callbacks cb 0x7f2bb0009e40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:12:07.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.310+0000 7f2bc2800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a5976f6b0 0x555a59780920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:07.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.310+0000 7f2bc2800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a5976f6b0 0x555a59780920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60192/0 (socket says 172.21.15.28:60192) 2024-02-21T23:12:07.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.310+0000 7f2bc2800640 1 -- 172.21.15.28:0/1425968546 learned_addr learned my addr 172.21.15.28:0/1425968546 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:07.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.311+0000 7f2bc1e00640 1 --2- 172.21.15.28:0/1425968546 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x555a59734330 0x555a597827f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:07.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.311+0000 7f2bc2800640 1 -- 172.21.15.28:0/1425968546 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x555a59734330 msgr2=0x555a597827f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:07.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.311+0000 7f2bc2800640 1 --2- 172.21.15.28:0/1425968546 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x555a59734330 0x555a597827f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:07.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.311+0000 7f2bc2800640 1 -- 172.21.15.28:0/1425968546 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555a597537c0 msgr2=0x555a5972cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:07.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.311+0000 7f2bc2800640 1 --2- 172.21.15.28:0/1425968546 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555a597537c0 0x555a5972cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:07.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.311+0000 7f2bc2800640 1 -- 172.21.15.28:0/1425968546 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555a5972c720 con 0x555a5976f6b0 2024-02-21T23:12:07.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.311+0000 7f2bc2800640 1 --2- 172.21.15.28:0/1425968546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a5976f6b0 0x555a59780920 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc02bef0 tx=0x7f2bbc02be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d04f90cc6e1be6c7 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:07.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.311+0000 7f2ba2a00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:12:07.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.311+0000 7f2ba2a00640 1 -- 172.21.15.28:0/1425968546 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bbc092040 con 0x555a5976f6b0 2024-02-21T23:12:07.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.311+0000 7f2ba2a00640 1 -- 172.21.15.28:0/1425968546 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(186..186 src has 1..186) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f2bbc082870 con 0x555a5976f6b0 2024-02-21T23:12:07.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.311+0000 7f2bcb9c4740 1 -- 172.21.15.28:0/1425968546 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2bb002d070 con 0x555a5976f6b0 2024-02-21T23:12:07.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.311+0000 7f2ba2a00640 1 -- 172.21.15.28:0/1425968546 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=209) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2bbc009110 con 0x555a5976f6b0 2024-02-21T23:12:07.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.311+0000 7f2bcb9c4740 10 client.10212 fetch_fsmap learned FSMap version 209 2024-02-21T23:12:07.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.311+0000 7f2bcb9c4740 1 -- 172.21.15.28:0/1425968546 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=209,monmap=2+}) v3 -- 0x555a597e2c90 con 0x555a5976f6b0 2024-02-21T23:12:07.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.312+0000 7f2ba2a00640 1 -- 172.21.15.28:0/1425968546 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bb002d070 con 0x555a5976f6b0 2024-02-21T23:12:07.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.312+0000 7f2ba2a00640 1 -- 172.21.15.28:0/1425968546 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 209) v1 ==== 3306+0+0 (secure 0 0 0) 0x7f2bbc041070 con 0x555a5976f6b0 2024-02-21T23:12:07.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.312+0000 7f2bcb9c4740 10 client.10212 fetch_fsmap finished waiting for FSMap version 209 2024-02-21T23:12:07.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.312+0000 7f2bcb9c4740 10 client.10212 resolve_mds: resolved name 'd' to daemon mds.d 2024-02-21T23:12:07.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.312+0000 7f2bcb9c4740 20 client.10212 populate_metadata read hostname 'smithi028' 2024-02-21T23:12:07.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.312+0000 7f2bcb9c4740 1 --2- 172.21.15.28:0/1425968546 >> [v2:172.21.15.81:6834/2895690310,v1:172.21.15.81:6836/2895690310] conn(0x555a597dc050 0x555a597725d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:07.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.312+0000 7f2bcb9c4740 4 client.10212 mds_command: new command op to 7735 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-21T23:12:07.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.312+0000 7f2bcb9c4740 1 -- 172.21.15.28:0/1425968546 --> [v2:172.21.15.81:6834/2895690310,v1:172.21.15.81:6836/2895690310] -- command(tid 0: {"prefix": "status"}) v1 -- 0x555a5969ed00 con 0x555a597dc050 2024-02-21T23:12:07.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.312+0000 7f2bc1e00640 1 --2- 172.21.15.28:0/1425968546 >> [v2:172.21.15.81:6834/2895690310,v1:172.21.15.81:6836/2895690310] conn(0x555a597dc050 0x555a597725d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:07.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.313+0000 7f2bc1e00640 1 --2- 172.21.15.28:0/1425968546 >> [v2:172.21.15.81:6834/2895690310,v1:172.21.15.81:6836/2895690310] conn(0x555a597dc050 0x555a597725d0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=84dca55350645bd server_cookie=4ba3cdecafbae00c in_seq=0 out_seq=0 2024-02-21T23:12:07.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.313+0000 7f2ba2a00640 10 client.10212 ms_handle_connect on v2:172.21.15.81:6834/2895690310 2024-02-21T23:12:07.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.314+0000 7f2ba2a00640 1 -- 172.21.15.28:0/1425968546 <== mds.1 v2:172.21.15.81:6834/2895690310 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+334 (crc 0 0 0) 0x555a5969ed00 con 0x555a597dc050 2024-02-21T23:12:07.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.314+0000 7f2ba2a00640 10 client.10212 handle_command_reply: tid=0 2024-02-21T23:12:07.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.314+0000 7f2bcb9c4740 1 client.10212 shutdown 2024-02-21T23:12:07.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.314+0000 7f2bcb9c4740 1 -- 172.21.15.28:0/1425968546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a5976f6b0 msgr2=0x555a59780920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:07.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.314+0000 7f2bcb9c4740 1 --2- 172.21.15.28:0/1425968546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a5976f6b0 0x555a59780920 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc02bef0 tx=0x7f2bbc02be80 comp rx=0 tx=0).stop 2024-02-21T23:12:07.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.314+0000 7f2bcb9c4740 1 -- 172.21.15.28:0/1425968546 shutdown_connections 2024-02-21T23:12:07.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.314+0000 7f2bcb9c4740 1 --2- 172.21.15.28:0/1425968546 >> [v2:172.21.15.81:6834/2895690310,v1:172.21.15.81:6836/2895690310] conn(0x555a597dc050 0x555a597725d0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:07.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.314+0000 7f2bcb9c4740 1 --2- 172.21.15.28:0/1425968546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a5976f6b0 0x555a59780920 unknown :-1 s=CLOSED pgs=2145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:07.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.314+0000 7f2bcb9c4740 1 --2- 172.21.15.28:0/1425968546 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555a597537c0 0x555a5972cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:07.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.314+0000 7f2bcb9c4740 1 --2- 172.21.15.28:0/1425968546 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x555a59734330 0x555a597827f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:07.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.314+0000 7f2ba2a00640 0 client.10212 ms_handle_reset on v2:172.21.15.81:6834/2895690310 2024-02-21T23:12:07.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.314+0000 7f2bcb9c4740 1 -- 172.21.15.28:0/1425968546 >> 172.21.15.28:0/1425968546 conn(0x555a59733020 msgr2=0x555a5977c500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:07.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.314+0000 7f2bcb9c4740 1 -- 172.21.15.28:0/1425968546 shutdown_connections 2024-02-21T23:12:07.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.314+0000 7f2bcb9c4740 1 -- 172.21.15.28:0/1425968546 wait complete. 2024-02-21T23:12:07.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.314+0000 7f2bcb9c4740 20 client.10212 trim_cache size 0 max 16384 2024-02-21T23:12:07.313 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-21T23:12:07.313 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac", 2024-02-21T23:12:07.313 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 1, 2024-02-21T23:12:07.314 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 7735, 2024-02-21T23:12:07.314 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-02-21T23:12:07.314 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-02-21T23:12:07.314 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-02-21T23:12:07.314 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 3.7121724149999999, 2024-02-21T23:12:07.314 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 209, 2024-02-21T23:12:07.314 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 186, 2024-02-21T23:12:07.314 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 186, 2024-02-21T23:12:07.314 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 33.847505755999997 2024-02-21T23:12:07.314 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-21T23:12:07.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.315+0000 7f2bc3200640 1 -- 172.21.15.28:0/810329043 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2ba406b4d0 msgr2=0x7f2ba406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:07.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.315+0000 7f2bc3200640 1 --2- 172.21.15.28:0/810329043 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2ba406b4d0 0x7f2ba406d990 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f2bb4005c00 tx=0x7f2bb4009040 comp rx=0 tx=0).stop 2024-02-21T23:12:07.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.315+0000 7f2bc3200640 1 -- 172.21.15.28:0/810329043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bc4149830 msgr2=0x7f2bc4154680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:07.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.315+0000 7f2bc3200640 1 --2- 172.21.15.28:0/810329043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bc4149830 0x7f2bc4154680 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc009860 tx=0x7f2bbc0046c0 comp rx=0 tx=0).stop 2024-02-21T23:12:07.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.315+0000 7f2bc3200640 1 -- 172.21.15.28:0/810329043 shutdown_connections 2024-02-21T23:12:07.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.315+0000 7f2bc3200640 1 --2- 172.21.15.28:0/810329043 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2ba406b4d0 0x7f2ba406d990 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:07.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.315+0000 7f2bc3200640 1 --2- 172.21.15.28:0/810329043 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2bc40c43e0 0x7f2bc4154140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:07.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.315+0000 7f2bc3200640 1 --2- 172.21.15.28:0/810329043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bc4149830 0x7f2bc4154680 unknown :-1 s=CLOSED pgs=2141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:07.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.315+0000 7f2bc3200640 1 --2- 172.21.15.28:0/810329043 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2bc406d410 0x7f2bc4156b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:07.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.315+0000 7f2bc3200640 1 -- 172.21.15.28:0/810329043 >> 172.21.15.28:0/810329043 conn(0x7f2bc40a8320 msgr2=0x7f2bc4071010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:07.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.315+0000 7f2bc3200640 1 -- 172.21.15.28:0/810329043 shutdown_connections 2024-02-21T23:12:07.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.315+0000 7f2bc3200640 1 -- 172.21.15.28:0/810329043 wait complete. 2024-02-21T23:12:07.319 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-02-21T23:12:07.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.540+0000 7fe233200640 1 Processor -- start 2024-02-21T23:12:07.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.540+0000 7fe233200640 1 -- start start 2024-02-21T23:12:07.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.541+0000 7fe233200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe2341524c0 0x7fe234155a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:07.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.541+0000 7fe233200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fe2340c85c0 0x7fe2340c70d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:07.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.541+0000 7fe233200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe23406df50 con 0x7fe2340c5040 2024-02-21T23:12:07.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.541+0000 7fe233200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2340c5520 con 0x7fe2340c85c0 2024-02-21T23:12:07.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.541+0000 7fe233200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2340c5800 con 0x7fe2341524c0 2024-02-21T23:12:07.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.542+0000 7fe231e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe2341524c0 0x7fe234155a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:07.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.542+0000 7fe231e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe2341524c0 0x7fe234155a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3301/0 says I am v2:172.21.15.28:60978/0 (socket says 172.21.15.28:60978) 2024-02-21T23:12:07.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.542+0000 7fe231e00640 1 -- 172.21.15.28:0/3535892259 learned_addr learned my addr 172.21.15.28:0/3535892259 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:07.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.542+0000 7fe232800640 1 --2- 172.21.15.28:0/3535892259 >> v2:172.21.15.81:3300/0 conn(0x7fe2340c85c0 0x7fe2340c70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:07.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.542+0000 7fe231e00640 1 -- 172.21.15.28:0/3535892259 >> v2:172.21.15.81:3300/0 conn(0x7fe2340c85c0 msgr2=0x7fe2340c70d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:07.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.542+0000 7fe231e00640 1 --2- 172.21.15.28:0/3535892259 >> v2:172.21.15.81:3300/0 conn(0x7fe2340c85c0 0x7fe2340c70d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:07.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.542+0000 7fe231e00640 1 -- 172.21.15.28:0/3535892259 >> v1:172.21.15.81:6789/0 conn(0x7fe2340c5040 legacy=0x7fe23406ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:07.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.542+0000 7fe231e00640 1 -- 172.21.15.28:0/3535892259 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe23406c670 con 0x7fe2341524c0 2024-02-21T23:12:07.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.544+0000 7fe231e00640 1 --2- 172.21.15.28:0/3535892259 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe2341524c0 0x7fe234155a10 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe224009e50 tx=0x7fe22402be50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b155410de30b600 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:07.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.544+0000 7fe230a00640 1 -- 172.21.15.28:0/3535892259 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe22402c9f0 con 0x7fe2341524c0 2024-02-21T23:12:07.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.544+0000 7fe230a00640 1 -- 172.21.15.28:0/3535892259 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe22402cb90 con 0x7fe2341524c0 2024-02-21T23:12:07.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.544+0000 7fe230a00640 1 -- 172.21.15.28:0/3535892259 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2240343f0 con 0x7fe2341524c0 2024-02-21T23:12:07.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.545+0000 7fe233200640 1 -- 172.21.15.28:0/3535892259 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe2341524c0 msgr2=0x7fe234155a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:07.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.545+0000 7fe233200640 1 --2- 172.21.15.28:0/3535892259 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe2341524c0 0x7fe234155a10 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe224009e50 tx=0x7fe22402be50 comp rx=0 tx=0).stop 2024-02-21T23:12:07.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.545+0000 7fe233200640 1 -- 172.21.15.28:0/3535892259 shutdown_connections 2024-02-21T23:12:07.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.545+0000 7fe233200640 1 --2- 172.21.15.28:0/3535892259 >> v2:172.21.15.81:3300/0 conn(0x7fe2340c85c0 0x7fe2340c70d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:07.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.545+0000 7fe233200640 1 --2- 172.21.15.28:0/3535892259 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe2341524c0 0x7fe234155a10 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:07.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.545+0000 7fe233200640 1 -- 172.21.15.28:0/3535892259 >> 172.21.15.28:0/3535892259 conn(0x7fe2340ac2b0 msgr2=0x7fe2340ac6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:07.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.545+0000 7fe233200640 1 -- 172.21.15.28:0/3535892259 shutdown_connections 2024-02-21T23:12:07.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.545+0000 7fe233200640 1 -- 172.21.15.28:0/3535892259 wait complete. 2024-02-21T23:12:07.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.547+0000 7fe233200640 1 Processor -- start 2024-02-21T23:12:07.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.547+0000 7fe233200640 1 -- start start 2024-02-21T23:12:07.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.548+0000 7fe233200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe2340c5040 0x7fe2341edcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:07.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.548+0000 7fe233200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2340c85c0 0x7fe234148c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:07.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.548+0000 7fe233200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2341524c0 0x7fe2341491b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:07.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.548+0000 7fe233200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2340c8a70 con 0x7fe2341524c0 2024-02-21T23:12:07.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.548+0000 7fe233200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2341528a0 con 0x7fe2340c85c0 2024-02-21T23:12:07.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.548+0000 7fe233200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2340c7610 con 0x7fe2340c5040 2024-02-21T23:12:07.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.548+0000 7fe232800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2341524c0 0x7fe2341491b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:07.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.548+0000 7fe232800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2341524c0 0x7fe2341491b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60208/0 (socket says 172.21.15.28:60208) 2024-02-21T23:12:07.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.548+0000 7fe232800640 1 -- 172.21.15.28:0/3406333386 learned_addr learned my addr 172.21.15.28:0/3406333386 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:07.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.548+0000 7fe231e00640 1 --2- 172.21.15.28:0/3406333386 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe2340c5040 0x7fe2341edcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:07.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.548+0000 7fe231400640 1 --2- 172.21.15.28:0/3406333386 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2340c85c0 0x7fe234148c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:07.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.549+0000 7fe232800640 1 -- 172.21.15.28:0/3406333386 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe2340c5040 msgr2=0x7fe2341edcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:07.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.549+0000 7fe232800640 1 --2- 172.21.15.28:0/3406333386 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe2340c5040 0x7fe2341edcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:07.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.549+0000 7fe232800640 1 -- 172.21.15.28:0/3406333386 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2340c85c0 msgr2=0x7fe234148c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:07.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.549+0000 7fe232800640 1 --2- 172.21.15.28:0/3406333386 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2340c85c0 0x7fe234148c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:07.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.549+0000 7fe232800640 1 -- 172.21.15.28:0/3406333386 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2340ae180 con 0x7fe2341524c0 2024-02-21T23:12:07.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.549+0000 7fe232800640 1 --2- 172.21.15.28:0/3406333386 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2341524c0 0x7fe2341491b0 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 crypto rx=0x7fe22c00e960 tx=0x7fe22c00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:07.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.549+0000 7fe22a000640 1 -- 172.21.15.28:0/3406333386 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe22c017070 con 0x7fe2341524c0 2024-02-21T23:12:07.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.549+0000 7fe22a000640 1 -- 172.21.15.28:0/3406333386 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe22c00c980 con 0x7fe2341524c0 2024-02-21T23:12:07.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.549+0000 7fe22a000640 1 -- 172.21.15.28:0/3406333386 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe22c00cd70 con 0x7fe2341524c0 2024-02-21T23:12:07.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.549+0000 7fe233200640 1 -- 172.21.15.28:0/3406333386 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2341499c0 con 0x7fe2341524c0 2024-02-21T23:12:07.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.550+0000 7fe233200640 1 -- 172.21.15.28:0/3406333386 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2341ee510 con 0x7fe2341524c0 2024-02-21T23:12:07.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.552+0000 7fe22a000640 1 -- 172.21.15.28:0/3406333386 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe22c01a070 con 0x7fe2341524c0 2024-02-21T23:12:07.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.552+0000 7fe22a000640 1 --2- 172.21.15.28:0/3406333386 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe21006b4d0 0x7fe21006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:07.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.553+0000 7fe22a000640 1 -- 172.21.15.28:0/3406333386 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe22c08a420 con 0x7fe2341524c0 2024-02-21T23:12:07.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.553+0000 7fe231e00640 1 --2- 172.21.15.28:0/3406333386 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe21006b4d0 0x7fe21006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:07.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.553+0000 7fe23b9ce740 1 Processor -- start 2024-02-21T23:12:07.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.554+0000 7fe23b9ce740 1 -- start start 2024-02-21T23:12:07.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.554+0000 7fe231e00640 1 --2- 172.21.15.28:0/3406333386 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe21006b4d0 0x7fe21006d990 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fe224035000 tx=0x7fe224030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:07.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.554+0000 7fe23b9ce740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a40aaa020 0x561a40aaa400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:07.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.554+0000 7fe23b9ce740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x561a40aab330 0x561a40ae6170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:07.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.555+0000 7fe23b9ce740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561a40ae66b0 0x561a40ae4720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:07.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.555+0000 7fe23b9ce740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a40a0e470 con 0x561a40aaa020 2024-02-21T23:12:07.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.555+0000 7fe23b9ce740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a409b7af0 con 0x561a40ae66b0 2024-02-21T23:12:07.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.555+0000 7fe23b9ce740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x561a40a0a510 con 0x561a40aab330 2024-02-21T23:12:07.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.555+0000 7fe231e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a40aaa020 0x561a40aaa400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:07.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.555+0000 7fe231e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a40aaa020 0x561a40aaa400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60218/0 (socket says 172.21.15.28:60218) 2024-02-21T23:12:07.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.555+0000 7fe231e00640 1 -- 172.21.15.28:0/1578361046 learned_addr learned my addr 172.21.15.28:0/1578361046 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:07.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.555+0000 7fe231400640 1 --2- 172.21.15.28:0/1578361046 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x561a40aab330 0x561a40ae6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:07.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.555+0000 7fe232800640 1 --2- 172.21.15.28:0/1578361046 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561a40ae66b0 0x561a40ae4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:07.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.555+0000 7fe231e00640 1 -- 172.21.15.28:0/1578361046 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x561a40aab330 msgr2=0x561a40ae6170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:07.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.555+0000 7fe231e00640 1 --2- 172.21.15.28:0/1578361046 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x561a40aab330 0x561a40ae6170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:07.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.555+0000 7fe231e00640 1 -- 172.21.15.28:0/1578361046 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561a40ae66b0 msgr2=0x561a40ae4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:07.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.555+0000 7fe231e00640 1 --2- 172.21.15.28:0/1578361046 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561a40ae66b0 0x561a40ae4720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:07.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.555+0000 7fe231e00640 1 -- 172.21.15.28:0/1578361046 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561a409bed90 con 0x561a40aaa020 2024-02-21T23:12:07.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.556+0000 7fe231e00640 1 --2- 172.21.15.28:0/1578361046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a40aaa020 0x561a40aaa400 secure :-1 s=READY pgs=2147 cs=0 l=1 rev1=1 crypto rx=0x7fe22402c460 tx=0x7fe22402e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8429b0a389730cdf server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:07.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.556+0000 7fe222600640 1 -- 172.21.15.28:0/1578361046 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe22404e070 con 0x561a40aaa020 2024-02-21T23:12:07.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.556+0000 7fe222600640 1 -- 172.21.15.28:0/1578361046 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe22403e7c0 con 0x561a40aaa020 2024-02-21T23:12:07.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.556+0000 7fe222600640 1 -- 172.21.15.28:0/1578361046 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe22403eb70 con 0x561a40aaa020 2024-02-21T23:12:07.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.557+0000 7fe23b9ce740 1 -- 172.21.15.28:0/1578361046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a40aaa020 msgr2=0x561a40aaa400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:07.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.557+0000 7fe23b9ce740 1 --2- 172.21.15.28:0/1578361046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a40aaa020 0x561a40aaa400 secure :-1 s=READY pgs=2147 cs=0 l=1 rev1=1 crypto rx=0x7fe22402c460 tx=0x7fe22402e000 comp rx=0 tx=0).stop 2024-02-21T23:12:07.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.557+0000 7fe23b9ce740 1 -- 172.21.15.28:0/1578361046 shutdown_connections 2024-02-21T23:12:07.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.557+0000 7fe23b9ce740 1 --2- 172.21.15.28:0/1578361046 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x561a40aab330 0x561a40ae6170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:07.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.557+0000 7fe23b9ce740 1 --2- 172.21.15.28:0/1578361046 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561a40ae66b0 0x561a40ae4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:07.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.557+0000 7fe23b9ce740 1 --2- 172.21.15.28:0/1578361046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a40aaa020 0x561a40aaa400 unknown :-1 s=CLOSED pgs=2147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:07.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.557+0000 7fe23b9ce740 1 -- 172.21.15.28:0/1578361046 >> 172.21.15.28:0/1578361046 conn(0x561a40aa6970 msgr2=0x561a40acb380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:07.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.557+0000 7fe23b9ce740 1 -- 172.21.15.28:0/1578361046 shutdown_connections 2024-02-21T23:12:07.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.557+0000 7fe23b9ce740 1 -- 172.21.15.28:0/1578361046 wait complete. 2024-02-21T23:12:07.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.557+0000 7fe23b9ce740 1 -- start start 2024-02-21T23:12:07.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.558+0000 7fe23b9ce740 1 Processor -- start 2024-02-21T23:12:07.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.559+0000 7fe23b9ce740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561a40aab330 0x561a40acf130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:07.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.559+0000 7fe23b9ce740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x561a40ae66b0 0x561a40acdad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:07.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.559+0000 7fe23b9ce740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a40ac6890 0x561a40ae31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:07.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.559+0000 7fe23b9ce740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a409ad040 con 0x561a40ac6890 2024-02-21T23:12:07.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.559+0000 7fe23b9ce740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a40acba90 con 0x561a40aab330 2024-02-21T23:12:07.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.559+0000 7fe23b9ce740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x561a40ace5c0 con 0x561a40ae66b0 2024-02-21T23:12:07.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.559+0000 7fe23b9ce740 10 client.0 _ll_register_callbacks cb 0x561a409ca920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:12:07.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.559+0000 7fe232800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a40ac6890 0x561a40ae31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:07.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.559+0000 7fe232800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a40ac6890 0x561a40ae31f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60220/0 (socket says 172.21.15.28:60220) 2024-02-21T23:12:07.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.559+0000 7fe232800640 1 -- 172.21.15.28:0/3119360796 learned_addr learned my addr 172.21.15.28:0/3119360796 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:07.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.559+0000 7fe231400640 1 --2- 172.21.15.28:0/3119360796 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x561a40ae66b0 0x561a40acdad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:07.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.559+0000 7fe231e00640 1 --2- 172.21.15.28:0/3119360796 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561a40aab330 0x561a40acf130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:07.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.559+0000 7fe232800640 1 -- 172.21.15.28:0/3119360796 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x561a40ae66b0 msgr2=0x561a40acdad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:07.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.559+0000 7fe232800640 1 --2- 172.21.15.28:0/3119360796 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x561a40ae66b0 0x561a40acdad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:07.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.559+0000 7fe232800640 1 -- 172.21.15.28:0/3119360796 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561a40aab330 msgr2=0x561a40acf130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:07.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.559+0000 7fe232800640 1 --2- 172.21.15.28:0/3119360796 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561a40aab330 0x561a40acf130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:07.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.559+0000 7fe232800640 1 -- 172.21.15.28:0/3119360796 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561a409ccaa0 con 0x561a40ac6890 2024-02-21T23:12:07.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.560+0000 7fe232800640 1 --2- 172.21.15.28:0/3119360796 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a40ac6890 0x561a40ae31f0 secure :-1 s=READY pgs=2148 cs=0 l=1 rev1=1 crypto rx=0x7fe22c00e930 tx=0x7fe22c059160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=882a343674ed5278 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:07.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.560+0000 7fe216a00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:12:07.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.560+0000 7fe216a00640 1 -- 172.21.15.28:0/3119360796 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe22c070020 con 0x561a40ac6890 2024-02-21T23:12:07.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.560+0000 7fe216a00640 1 -- 172.21.15.28:0/3119360796 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(186..186 src has 1..186) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe22c08a0c0 con 0x561a40ac6890 2024-02-21T23:12:07.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.560+0000 7fe23b9ce740 1 -- 172.21.15.28:0/3119360796 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe22403e7c0 con 0x561a40ac6890 2024-02-21T23:12:07.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.560+0000 7fe216a00640 1 -- 172.21.15.28:0/3119360796 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=209) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe22c075050 con 0x561a40ac6890 2024-02-21T23:12:07.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.560+0000 7fe23b9ce740 10 client.10221 fetch_fsmap learned FSMap version 209 2024-02-21T23:12:07.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.560+0000 7fe23b9ce740 1 -- 172.21.15.28:0/3119360796 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=209,monmap=2+}) v3 -- 0x561a407232d0 con 0x561a40ac6890 2024-02-21T23:12:07.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.560+0000 7fe216a00640 1 -- 172.21.15.28:0/3119360796 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe22403e7c0 con 0x561a40ac6890 2024-02-21T23:12:07.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.560+0000 7fe216a00640 1 -- 172.21.15.28:0/3119360796 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 209) v1 ==== 3306+0+0 (secure 0 0 0) 0x7fe22c024060 con 0x561a40ac6890 2024-02-21T23:12:07.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.561+0000 7fe23b9ce740 10 client.10221 fetch_fsmap finished waiting for FSMap version 209 2024-02-21T23:12:07.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.561+0000 7fe23b9ce740 10 client.10221 resolve_mds: resolved name 'b' to daemon mds.b 2024-02-21T23:12:07.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.561+0000 7fe23b9ce740 20 client.10221 populate_metadata read hostname 'smithi028' 2024-02-21T23:12:07.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.561+0000 7fe23b9ce740 1 --2- 172.21.15.28:0/3119360796 >> [v2:172.21.15.81:6835/416021079,v1:172.21.15.81:6837/416021079] conn(0x561a40af68b0 0x561a40b04c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:07.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.561+0000 7fe23b9ce740 4 client.10221 mds_command: new command op to 7867 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-21T23:12:07.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.561+0000 7fe23b9ce740 1 -- 172.21.15.28:0/3119360796 --> [v2:172.21.15.81:6835/416021079,v1:172.21.15.81:6837/416021079] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561a40a1b9b0 con 0x561a40af68b0 2024-02-21T23:12:07.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.561+0000 7fe231e00640 1 --2- 172.21.15.28:0/3119360796 >> [v2:172.21.15.81:6835/416021079,v1:172.21.15.81:6837/416021079] conn(0x561a40af68b0 0x561a40b04c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:07.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.562+0000 7fe231e00640 1 --2- 172.21.15.28:0/3119360796 >> [v2:172.21.15.81:6835/416021079,v1:172.21.15.81:6837/416021079] conn(0x561a40af68b0 0x561a40b04c80 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1544dc1ee28bd56e server_cookie=1a6c77ce24a9a9f3 in_seq=0 out_seq=0 2024-02-21T23:12:07.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.562+0000 7fe216a00640 10 client.10221 ms_handle_connect on v2:172.21.15.81:6835/416021079 2024-02-21T23:12:07.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.564+0000 7fe216a00640 1 -- 172.21.15.28:0/3119360796 <== mds.0 v2:172.21.15.81:6835/416021079 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x561a40a1b9b0 con 0x561a40af68b0 2024-02-21T23:12:07.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.564+0000 7fe216a00640 10 client.10221 handle_command_reply: tid=0 2024-02-21T23:12:07.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.564+0000 7fe23b9ce740 1 client.10221 shutdown 2024-02-21T23:12:07.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.564+0000 7fe23b9ce740 1 -- 172.21.15.28:0/3119360796 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a40ac6890 msgr2=0x561a40ae31f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:07.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.564+0000 7fe23b9ce740 1 --2- 172.21.15.28:0/3119360796 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a40ac6890 0x561a40ae31f0 secure :-1 s=READY pgs=2148 cs=0 l=1 rev1=1 crypto rx=0x7fe22c00e930 tx=0x7fe22c059160 comp rx=0 tx=0).stop 2024-02-21T23:12:07.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.564+0000 7fe23b9ce740 1 -- 172.21.15.28:0/3119360796 shutdown_connections 2024-02-21T23:12:07.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.564+0000 7fe23b9ce740 1 --2- 172.21.15.28:0/3119360796 >> [v2:172.21.15.81:6835/416021079,v1:172.21.15.81:6837/416021079] conn(0x561a40af68b0 0x561a40b04c80 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:07.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.564+0000 7fe23b9ce740 1 --2- 172.21.15.28:0/3119360796 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x561a40ae66b0 0x561a40acdad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:07.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.564+0000 7fe23b9ce740 1 --2- 172.21.15.28:0/3119360796 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a40ac6890 0x561a40ae31f0 unknown :-1 s=CLOSED pgs=2148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:07.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.564+0000 7fe23b9ce740 1 --2- 172.21.15.28:0/3119360796 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561a40aab330 0x561a40acf130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:07.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.564+0000 7fe216a00640 0 client.10221 ms_handle_reset on v2:172.21.15.81:6835/416021079 2024-02-21T23:12:07.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.564+0000 7fe23b9ce740 1 -- 172.21.15.28:0/3119360796 >> 172.21.15.28:0/3119360796 conn(0x561a40aaa020 msgr2=0x561a40ae9fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:07.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.564+0000 7fe23b9ce740 1 -- 172.21.15.28:0/3119360796 shutdown_connections 2024-02-21T23:12:07.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.564+0000 7fe23b9ce740 1 -- 172.21.15.28:0/3119360796 wait complete. 2024-02-21T23:12:07.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.565+0000 7fe23b9ce740 20 client.10221 trim_cache size 0 max 16384 2024-02-21T23:12:07.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.583+0000 7fe23b9ce740 1 Processor -- start 2024-02-21T23:12:07.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.583+0000 7fe23b9ce740 1 -- start start 2024-02-21T23:12:07.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.583+0000 7fe23b9ce740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561a40aab330 0x561a40ae2c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:07.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.583+0000 7fe23b9ce740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a40ac6890 0x561a40b08ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:07.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.583+0000 7fe23b9ce740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x561a40ae66b0 0x561a40b091e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:07.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.583+0000 7fe23b9ce740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2341f79d0 con 0x561a40ac6890 2024-02-21T23:12:07.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.583+0000 7fe23b9ce740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2341f5fa0 con 0x561a40aab330 2024-02-21T23:12:07.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.583+0000 7fe23b9ce740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2341f6520 con 0x561a40ae66b0 2024-02-21T23:12:07.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.584+0000 7fe231400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a40ac6890 0x561a40b08ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:07.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.584+0000 7fe231400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a40ac6890 0x561a40b08ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60236/0 (socket says 172.21.15.28:60236) 2024-02-21T23:12:07.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.584+0000 7fe231400640 1 -- 172.21.15.28:0/460528225 learned_addr learned my addr 172.21.15.28:0/460528225 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:07.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.584+0000 7fe231e00640 1 --2- 172.21.15.28:0/460528225 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561a40aab330 0x561a40ae2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:07.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.584+0000 7fe232800640 1 --2- 172.21.15.28:0/460528225 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x561a40ae66b0 0x561a40b091e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:07.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.584+0000 7fe231400640 1 -- 172.21.15.28:0/460528225 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x561a40ae66b0 msgr2=0x561a40b091e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:07.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.584+0000 7fe231400640 1 --2- 172.21.15.28:0/460528225 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x561a40ae66b0 0x561a40b091e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:07.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.584+0000 7fe231400640 1 -- 172.21.15.28:0/460528225 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561a40aab330 msgr2=0x561a40ae2c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:07.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.584+0000 7fe231400640 1 --2- 172.21.15.28:0/460528225 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561a40aab330 0x561a40ae2c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:07.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.584+0000 7fe231400640 1 -- 172.21.15.28:0/460528225 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561a407ed6b0 con 0x561a40ac6890 2024-02-21T23:12:07.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.585+0000 7fe231400640 1 --2- 172.21.15.28:0/460528225 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a40ac6890 0x561a40b08ca0 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 crypto rx=0x7fe21c0015d0 tx=0x7fe21c001a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c2b34f02095721a server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:07.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.585+0000 7fe221c00640 1 -- 172.21.15.28:0/460528225 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe21c01a070 con 0x561a40ac6890 2024-02-21T23:12:07.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.585+0000 7fe221c00640 1 -- 172.21.15.28:0/460528225 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe21c0052e0 con 0x561a40ac6890 2024-02-21T23:12:07.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.585+0000 7fe221c00640 1 -- 172.21.15.28:0/460528225 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe21c014ac0 con 0x561a40ac6890 2024-02-21T23:12:07.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.586+0000 7fe23b9ce740 1 -- 172.21.15.28:0/460528225 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a40ac6890 msgr2=0x561a40b08ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:07.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.586+0000 7fe23b9ce740 1 --2- 172.21.15.28:0/460528225 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a40ac6890 0x561a40b08ca0 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 crypto rx=0x7fe21c0015d0 tx=0x7fe21c001a90 comp rx=0 tx=0).stop 2024-02-21T23:12:07.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.586+0000 7fe23b9ce740 1 -- 172.21.15.28:0/460528225 shutdown_connections 2024-02-21T23:12:07.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.586+0000 7fe23b9ce740 1 --2- 172.21.15.28:0/460528225 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x561a40ae66b0 0x561a40b091e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:07.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.586+0000 7fe23b9ce740 1 --2- 172.21.15.28:0/460528225 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a40ac6890 0x561a40b08ca0 unknown :-1 s=CLOSED pgs=2149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:07.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.586+0000 7fe23b9ce740 1 --2- 172.21.15.28:0/460528225 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561a40aab330 0x561a40ae2c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:07.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.586+0000 7fe23b9ce740 1 -- 172.21.15.28:0/460528225 >> 172.21.15.28:0/460528225 conn(0x561a40aaa020 msgr2=0x561a40accf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:07.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.586+0000 7fe23b9ce740 1 -- 172.21.15.28:0/460528225 shutdown_connections 2024-02-21T23:12:07.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.586+0000 7fe23b9ce740 1 -- 172.21.15.28:0/460528225 wait complete. 2024-02-21T23:12:07.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.586+0000 7fe23b9ce740 1 -- start start 2024-02-21T23:12:07.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.587+0000 7fe23b9ce740 1 Processor -- start 2024-02-21T23:12:07.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.588+0000 7fe23b9ce740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a40aab330 0x561a40af2420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:07.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.588+0000 7fe23b9ce740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561a40ac6890 0x561a40b5e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:07.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.588+0000 7fe23b9ce740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x561a40ae66b0 0x561a40b5b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:07.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.588+0000 7fe23b9ce740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a40ace300 con 0x561a40aab330 2024-02-21T23:12:07.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.588+0000 7fe23b9ce740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a40acca80 con 0x561a40ac6890 2024-02-21T23:12:07.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.588+0000 7fe23b9ce740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x561a40acd730 con 0x561a40ae66b0 2024-02-21T23:12:07.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.588+0000 7fe23b9ce740 10 client.0 _ll_register_callbacks cb 0x7fe22c058340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:12:07.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.588+0000 7fe231e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a40aab330 0x561a40af2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:07.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.588+0000 7fe231e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a40aab330 0x561a40af2420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60246/0 (socket says 172.21.15.28:60246) 2024-02-21T23:12:07.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.588+0000 7fe231e00640 1 -- 172.21.15.28:0/3704062614 learned_addr learned my addr 172.21.15.28:0/3704062614 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:07.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.588+0000 7fe231400640 1 --2- 172.21.15.28:0/3704062614 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561a40ac6890 0x561a40b5e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:07.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.588+0000 7fe232800640 1 --2- 172.21.15.28:0/3704062614 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x561a40ae66b0 0x561a40b5b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:07.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.588+0000 7fe231e00640 1 -- 172.21.15.28:0/3704062614 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x561a40ae66b0 msgr2=0x561a40b5b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:07.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.588+0000 7fe231e00640 1 --2- 172.21.15.28:0/3704062614 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x561a40ae66b0 0x561a40b5b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:07.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.588+0000 7fe231e00640 1 -- 172.21.15.28:0/3704062614 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561a40ac6890 msgr2=0x561a40b5e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:07.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.588+0000 7fe231e00640 1 --2- 172.21.15.28:0/3704062614 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561a40ac6890 0x561a40b5e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:07.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.588+0000 7fe231e00640 1 -- 172.21.15.28:0/3704062614 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561a40ae8c90 con 0x561a40aab330 2024-02-21T23:12:07.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.589+0000 7fe231e00640 1 --2- 172.21.15.28:0/3704062614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a40aab330 0x561a40af2420 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 crypto rx=0x7fe224009f70 tx=0x7fe224004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60a3ff54db11b74c server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:07.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.589+0000 7fe216a00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:12:07.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.589+0000 7fe216a00640 1 -- 172.21.15.28:0/3704062614 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe224042050 con 0x561a40aab330 2024-02-21T23:12:07.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.589+0000 7fe216a00640 1 -- 172.21.15.28:0/3704062614 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(186..186 src has 1..186) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe22403ee00 con 0x561a40aab330 2024-02-21T23:12:07.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.589+0000 7fe23b9ce740 1 -- 172.21.15.28:0/3704062614 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe21c0052e0 con 0x561a40aab330 2024-02-21T23:12:07.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.589+0000 7fe216a00640 1 -- 172.21.15.28:0/3704062614 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=209) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe22404e040 con 0x561a40aab330 2024-02-21T23:12:07.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.589+0000 7fe23b9ce740 10 client.10227 fetch_fsmap learned FSMap version 209 2024-02-21T23:12:07.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.589+0000 7fe23b9ce740 1 -- 172.21.15.28:0/3704062614 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=209,monmap=2+}) v3 -- 0x561a40b5ecc0 con 0x561a40aab330 2024-02-21T23:12:07.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.589+0000 7fe216a00640 1 -- 172.21.15.28:0/3704062614 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe21c0052e0 con 0x561a40aab330 2024-02-21T23:12:07.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.590+0000 7fe216a00640 1 -- 172.21.15.28:0/3704062614 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 209) v1 ==== 3306+0+0 (secure 0 0 0) 0x7fe224008030 con 0x561a40aab330 2024-02-21T23:12:07.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.590+0000 7fe23b9ce740 10 client.10227 fetch_fsmap finished waiting for FSMap version 209 2024-02-21T23:12:07.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.590+0000 7fe23b9ce740 10 client.10227 resolve_mds: resolved name 'b' to daemon mds.b 2024-02-21T23:12:07.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.590+0000 7fe23b9ce740 20 client.10227 populate_metadata read hostname 'smithi028' 2024-02-21T23:12:07.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.590+0000 7fe23b9ce740 1 --2- 172.21.15.28:0/3704062614 >> [v2:172.21.15.81:6835/416021079,v1:172.21.15.81:6837/416021079] conn(0x561a40ac9140 0x561a40ac9520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:07.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.590+0000 7fe23b9ce740 4 client.10227 mds_command: new command op to 7867 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-21T23:12:07.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.590+0000 7fe23b9ce740 1 -- 172.21.15.28:0/3704062614 --> [v2:172.21.15.81:6835/416021079,v1:172.21.15.81:6837/416021079] -- command(tid 0: {"prefix": "status"}) v1 -- 0x561a40acc7e0 con 0x561a40ac9140 2024-02-21T23:12:07.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.590+0000 7fe231400640 1 --2- 172.21.15.28:0/3704062614 >> [v2:172.21.15.81:6835/416021079,v1:172.21.15.81:6837/416021079] conn(0x561a40ac9140 0x561a40ac9520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:07.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.591+0000 7fe231400640 1 --2- 172.21.15.28:0/3704062614 >> [v2:172.21.15.81:6835/416021079,v1:172.21.15.81:6837/416021079] conn(0x561a40ac9140 0x561a40ac9520 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eabe2d71ac3dfa3b server_cookie=5f60e711284227fe in_seq=0 out_seq=0 2024-02-21T23:12:07.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.591+0000 7fe216a00640 10 client.10227 ms_handle_connect on v2:172.21.15.81:6835/416021079 2024-02-21T23:12:07.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.591+0000 7fe216a00640 1 -- 172.21.15.28:0/3704062614 <== mds.0 v2:172.21.15.81:6835/416021079 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+320 (crc 0 0 0) 0x561a40acc7e0 con 0x561a40ac9140 2024-02-21T23:12:07.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.591+0000 7fe216a00640 10 client.10227 handle_command_reply: tid=0 2024-02-21T23:12:07.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.591+0000 7fe23b9ce740 1 client.10227 shutdown 2024-02-21T23:12:07.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.592+0000 7fe23b9ce740 1 -- 172.21.15.28:0/3704062614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a40aab330 msgr2=0x561a40af2420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:07.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.592+0000 7fe23b9ce740 1 --2- 172.21.15.28:0/3704062614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a40aab330 0x561a40af2420 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 crypto rx=0x7fe224009f70 tx=0x7fe224004ab0 comp rx=0 tx=0).stop 2024-02-21T23:12:07.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.592+0000 7fe23b9ce740 1 -- 172.21.15.28:0/3704062614 shutdown_connections 2024-02-21T23:12:07.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.592+0000 7fe23b9ce740 1 --2- 172.21.15.28:0/3704062614 >> [v2:172.21.15.81:6835/416021079,v1:172.21.15.81:6837/416021079] conn(0x561a40ac9140 0x561a40ac9520 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:07.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.592+0000 7fe23b9ce740 1 --2- 172.21.15.28:0/3704062614 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x561a40ae66b0 0x561a40b5b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:07.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.592+0000 7fe23b9ce740 1 --2- 172.21.15.28:0/3704062614 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561a40ac6890 0x561a40b5e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:07.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.592+0000 7fe23b9ce740 1 --2- 172.21.15.28:0/3704062614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561a40aab330 0x561a40af2420 unknown :-1 s=CLOSED pgs=2150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:07.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.592+0000 7fe216a00640 0 client.10227 ms_handle_reset on v2:172.21.15.81:6835/416021079 2024-02-21T23:12:07.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.592+0000 7fe23b9ce740 1 -- 172.21.15.28:0/3704062614 >> 172.21.15.28:0/3704062614 conn(0x561a40aaa020 msgr2=0x561a40ae2c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:07.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.592+0000 7fe23b9ce740 1 -- 172.21.15.28:0/3704062614 shutdown_connections 2024-02-21T23:12:07.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.592+0000 7fe23b9ce740 1 -- 172.21.15.28:0/3704062614 wait complete. 2024-02-21T23:12:07.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.592+0000 7fe23b9ce740 20 client.10227 trim_cache size 0 max 16384 2024-02-21T23:12:07.583 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-21T23:12:07.583 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac", 2024-02-21T23:12:07.583 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-02-21T23:12:07.583 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 7867, 2024-02-21T23:12:07.583 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-02-21T23:12:07.583 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-02-21T23:12:07.583 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-02-21T23:12:07.584 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 5.007736704, 2024-02-21T23:12:07.584 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 209, 2024-02-21T23:12:07.584 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 186, 2024-02-21T23:12:07.584 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 186, 2024-02-21T23:12:07.584 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 8.451080997 2024-02-21T23:12:07.584 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-21T23:12:07.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.593+0000 7fe233200640 1 -- 172.21.15.28:0/3406333386 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe21006b4d0 msgr2=0x7fe21006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:07.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.593+0000 7fe233200640 1 --2- 172.21.15.28:0/3406333386 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe21006b4d0 0x7fe21006d990 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fe224035000 tx=0x7fe224030000 comp rx=0 tx=0).stop 2024-02-21T23:12:07.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.593+0000 7fe233200640 1 -- 172.21.15.28:0/3406333386 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2341524c0 msgr2=0x7fe2341491b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:07.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.593+0000 7fe233200640 1 --2- 172.21.15.28:0/3406333386 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2341524c0 0x7fe2341491b0 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 crypto rx=0x7fe22c00e960 tx=0x7fe22c00ee20 comp rx=0 tx=0).stop 2024-02-21T23:12:07.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.593+0000 7fe233200640 1 -- 172.21.15.28:0/3406333386 shutdown_connections 2024-02-21T23:12:07.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.593+0000 7fe233200640 1 --2- 172.21.15.28:0/3406333386 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe21006b4d0 0x7fe21006d990 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:07.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.593+0000 7fe233200640 1 --2- 172.21.15.28:0/3406333386 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2341524c0 0x7fe2341491b0 unknown :-1 s=CLOSED pgs=2146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:07.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.593+0000 7fe233200640 1 --2- 172.21.15.28:0/3406333386 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2340c85c0 0x7fe234148c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:07.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.593+0000 7fe233200640 1 --2- 172.21.15.28:0/3406333386 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe2340c5040 0x7fe2341edcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:07.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.593+0000 7fe233200640 1 -- 172.21.15.28:0/3406333386 >> 172.21.15.28:0/3406333386 conn(0x7fe2340ac2b0 msgr2=0x7fe23414fba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:07.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.593+0000 7fe233200640 1 -- 172.21.15.28:0/3406333386 shutdown_connections 2024-02-21T23:12:07.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:07.593+0000 7fe233200640 1 -- 172.21.15.28:0/3406333386 wait complete. 2024-02-21T23:12:07.596 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-02-21T23:12:07.597 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-21T23:12:07.708 DEBUG:teuthology.orchestra.run.smithi081:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-21T23:12:07.780 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-21T23:12:07.788 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-21T23:12:07.821 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-02-21T23:12:07.822 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-21T23:12:07.822 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-21T23:12:07.882 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054608,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-02-21T23:12:07.882 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-02-21T23:12:07.882 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-21T23:12:07.975 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053833,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498812k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249404k,nr_inodes=812351,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498812k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-02-21T23:12:07.976 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:12:08.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.213+0000 7f351c400640 1 Processor -- start 2024-02-21T23:12:08.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.213+0000 7f351c400640 1 -- start start 2024-02-21T23:12:08.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.215+0000 7f351c400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3514072ca0 0x7f351406dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:08.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.215+0000 7f351c400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f351406f230 con 0x7f3514151f60 2024-02-21T23:12:08.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.215+0000 7f351c400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3514071c90 con 0x7f351406e220 2024-02-21T23:12:08.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.215+0000 7f351c400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3514071a60 con 0x7f3514072ca0 2024-02-21T23:12:08.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.215+0000 7f3519000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3514072ca0 0x7f351406dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:08.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.216+0000 7f3519000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3514072ca0 0x7f351406dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60254/0 (socket says 172.21.15.28:60254) 2024-02-21T23:12:08.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.216+0000 7f3519000640 1 -- 172.21.15.28:0/2092229733 learned_addr learned my addr 172.21.15.28:0/2092229733 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:08.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.216+0000 7f3519a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f3514151f60 0x7f35140c11f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53766/0 (socket says 172.21.15.28:53766) 2024-02-21T23:12:08.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.216+0000 7f3519000640 1 -- 172.21.15.28:0/2092229733 >> v1:172.21.15.81:6789/0 conn(0x7f351406e220 legacy=0x7f351415bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:08.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.216+0000 7f3519000640 1 -- 172.21.15.28:0/2092229733 >> v1:172.21.15.28:6789/0 conn(0x7f3514151f60 legacy=0x7f35140c11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:08.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.216+0000 7f3519000640 1 -- 172.21.15.28:0/2092229733 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3514071f70 con 0x7f3514072ca0 2024-02-21T23:12:08.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.216+0000 7f3513e00640 1 -- 172.21.15.28:0/2092229733 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3412636853 0 0) 0x7f351406f230 con 0x7f3514151f60 2024-02-21T23:12:08.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.217+0000 7f3519000640 1 --2- 172.21.15.28:0/2092229733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3514072ca0 0x7f351406dce0 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 crypto rx=0x7f3500009870 tx=0x7f350002b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5baa6202af9c08f server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:08.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.217+0000 7f3513e00640 1 -- 172.21.15.28:0/2092229733 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3500034070 con 0x7f3514072ca0 2024-02-21T23:12:08.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.217+0000 7f3513e00640 1 -- 172.21.15.28:0/2092229733 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3500030850 con 0x7f3514072ca0 2024-02-21T23:12:08.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.217+0000 7f3513e00640 1 -- 172.21.15.28:0/2092229733 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3500030c20 con 0x7f3514072ca0 2024-02-21T23:12:08.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.218+0000 7f351c400640 1 -- 172.21.15.28:0/2092229733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3514072ca0 msgr2=0x7f351406dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:08.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.218+0000 7f351c400640 1 --2- 172.21.15.28:0/2092229733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3514072ca0 0x7f351406dce0 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 crypto rx=0x7f3500009870 tx=0x7f350002b990 comp rx=0 tx=0).stop 2024-02-21T23:12:08.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.218+0000 7f351c400640 1 -- 172.21.15.28:0/2092229733 shutdown_connections 2024-02-21T23:12:08.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.218+0000 7f351c400640 1 --2- 172.21.15.28:0/2092229733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3514072ca0 0x7f351406dce0 unknown :-1 s=CLOSED pgs=2152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:08.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.218+0000 7f351c400640 1 -- 172.21.15.28:0/2092229733 >> 172.21.15.28:0/2092229733 conn(0x7f35140c67f0 msgr2=0x7f35140c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:08.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.218+0000 7f351c400640 1 -- 172.21.15.28:0/2092229733 shutdown_connections 2024-02-21T23:12:08.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.218+0000 7f351c400640 1 -- 172.21.15.28:0/2092229733 wait complete. 2024-02-21T23:12:08.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.220+0000 7f351c400640 1 Processor -- start 2024-02-21T23:12:08.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.220+0000 7f351c400640 1 -- start start 2024-02-21T23:12:08.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.221+0000 7f351c400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f351406e220 0x7f35140aa430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:08.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.221+0000 7f351c400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3514072ca0 0x7f3514140ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:08.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.221+0000 7f351c400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3514151f60 0x7f3514141010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:08.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.221+0000 7f351c400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3514152340 con 0x7f3514151f60 2024-02-21T23:12:08.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.221+0000 7f351c400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35140c1900 con 0x7f3514072ca0 2024-02-21T23:12:08.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.221+0000 7f351c400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35140a6090 con 0x7f351406e220 2024-02-21T23:12:08.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.221+0000 7f351a400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3514151f60 0x7f3514141010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:08.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.221+0000 7f351a400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3514151f60 0x7f3514141010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60270/0 (socket says 172.21.15.28:60270) 2024-02-21T23:12:08.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.221+0000 7f351a400640 1 -- 172.21.15.28:0/3619383971 learned_addr learned my addr 172.21.15.28:0/3619383971 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:08.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.221+0000 7f3519a00640 1 --2- 172.21.15.28:0/3619383971 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f351406e220 0x7f35140aa430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:08.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.222+0000 7f3519000640 1 --2- 172.21.15.28:0/3619383971 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3514072ca0 0x7f3514140ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:08.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.222+0000 7f351a400640 1 -- 172.21.15.28:0/3619383971 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f351406e220 msgr2=0x7f35140aa430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:08.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.222+0000 7f351a400640 1 --2- 172.21.15.28:0/3619383971 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f351406e220 0x7f35140aa430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:08.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.222+0000 7f351a400640 1 -- 172.21.15.28:0/3619383971 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3514072ca0 msgr2=0x7f3514140ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:08.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.222+0000 7f351a400640 1 --2- 172.21.15.28:0/3619383971 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3514072ca0 0x7f3514140ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:08.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.222+0000 7f351a400640 1 -- 172.21.15.28:0/3619383971 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3514147b10 con 0x7f3514151f60 2024-02-21T23:12:08.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.222+0000 7f351a400640 1 --2- 172.21.15.28:0/3619383971 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3514151f60 0x7f3514141010 secure :-1 s=READY pgs=2153 cs=0 l=1 rev1=1 crypto rx=0x7f350c00c390 tx=0x7f350c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:08.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.222+0000 7f3511600640 1 -- 172.21.15.28:0/3619383971 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f350c017070 con 0x7f3514151f60 2024-02-21T23:12:08.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.222+0000 7f3511600640 1 -- 172.21.15.28:0/3619383971 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f350c0035f0 con 0x7f3514151f60 2024-02-21T23:12:08.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.222+0000 7f3511600640 1 -- 172.21.15.28:0/3619383971 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f350c0039e0 con 0x7f3514151f60 2024-02-21T23:12:08.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.222+0000 7f351c400640 1 -- 172.21.15.28:0/3619383971 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35140ac100 con 0x7f3514151f60 2024-02-21T23:12:08.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.223+0000 7f351c400640 1 -- 172.21.15.28:0/3619383971 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3514148070 con 0x7f3514151f60 2024-02-21T23:12:08.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.224+0000 7f351c400640 1 -- 172.21.15.28:0/3619383971 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35141483a0 con 0x7f3514151f60 2024-02-21T23:12:08.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.225+0000 7f3511600640 1 -- 172.21.15.28:0/3619383971 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f350c01a070 con 0x7f3514151f60 2024-02-21T23:12:08.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.226+0000 7f3511600640 1 --2- 172.21.15.28:0/3619383971 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f34f406b4d0 0x7f34f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:08.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.226+0000 7f3511600640 1 -- 172.21.15.28:0/3619383971 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f350c089760 con 0x7f3514151f60 2024-02-21T23:12:08.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.226+0000 7f3519a00640 1 --2- 172.21.15.28:0/3619383971 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f34f406b4d0 0x7f34f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:08.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.227+0000 7f3519a00640 1 --2- 172.21.15.28:0/3619383971 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f34f406b4d0 0x7f34f406d990 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f35080012e0 tx=0x7f3508005eb0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:08.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.231+0000 7f3511600640 1 -- 172.21.15.28:0/3619383971 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3514147b10 con 0x7f3514151f60 2024-02-21T23:12:08.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.471+0000 7f351c400640 1 -- 172.21.15.28:0/3619383971 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3514067f30 con 0x7f3514151f60 2024-02-21T23:12:08.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.472+0000 7f3511600640 1 -- 172.21.15.28:0/3619383971 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 209 v209) v1 ==== 95+0+4959 (secure 0 0 0) 0x7f350c058440 con 0x7f3514151f60 2024-02-21T23:12:08.459 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:12:08.460 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":209,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6848,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3439896042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3439896042},{"type":"v1","addr":"172.21.15.28:6835","nonce":3439896042}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":196},{"gid":7729,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/3213249808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3213249808},{"type":"v1","addr":"172.21.15.28:6833","nonce":3213249808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":195}],"filesystems":[{"mdsmap":{"epoch":209,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:11:38.384619+0000","modified":"2024-02-21T23:12:03.626987+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7867,"mds_1":7735},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7735":{"gid":7735,"name":"d","rank":1,"incarnation":208,"state":"up:active","state_seq":9,"addr":"172.21.15.81:6836/2895690310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2895690310},{"type":"v1","addr":"172.21.15.81:6836","nonce":2895690310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7867":{"gid":7867,"name":"b","rank":0,"incarnation":206,"state":"up:active","state_seq":2,"addr":"172.21.15.81:6837/416021079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":416021079},{"type":"v1","addr":"172.21.15.81:6837","nonce":416021079}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-02-21T23:12:08.460 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 209 2024-02-21T23:12:08.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.474+0000 7f351c400640 1 -- 172.21.15.28:0/3619383971 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f34f406b4d0 msgr2=0x7f34f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:08.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.475+0000 7f351c400640 1 --2- 172.21.15.28:0/3619383971 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f34f406b4d0 0x7f34f406d990 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f35080012e0 tx=0x7f3508005eb0 comp rx=0 tx=0).stop 2024-02-21T23:12:08.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.475+0000 7f351c400640 1 -- 172.21.15.28:0/3619383971 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3514151f60 msgr2=0x7f3514141010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:08.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.475+0000 7f351c400640 1 --2- 172.21.15.28:0/3619383971 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3514151f60 0x7f3514141010 secure :-1 s=READY pgs=2153 cs=0 l=1 rev1=1 crypto rx=0x7f350c00c390 tx=0x7f350c00c850 comp rx=0 tx=0).stop 2024-02-21T23:12:08.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.475+0000 7f351c400640 1 -- 172.21.15.28:0/3619383971 shutdown_connections 2024-02-21T23:12:08.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.475+0000 7f351c400640 1 --2- 172.21.15.28:0/3619383971 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f34f406b4d0 0x7f34f406d990 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:08.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.475+0000 7f351c400640 1 --2- 172.21.15.28:0/3619383971 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3514151f60 0x7f3514141010 unknown :-1 s=CLOSED pgs=2153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:08.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.475+0000 7f351c400640 1 --2- 172.21.15.28:0/3619383971 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3514072ca0 0x7f3514140ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:08.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.475+0000 7f351c400640 1 --2- 172.21.15.28:0/3619383971 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f351406e220 0x7f35140aa430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:08.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.475+0000 7f351c400640 1 -- 172.21.15.28:0/3619383971 >> 172.21.15.28:0/3619383971 conn(0x7f35140c67f0 msgr2=0x7f35141553c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:08.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.475+0000 7f351c400640 1 -- 172.21.15.28:0/3619383971 shutdown_connections 2024-02-21T23:12:08.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.475+0000 7f351c400640 1 -- 172.21.15.28:0/3619383971 wait complete. 2024-02-21T23:12:08.483 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:12:08.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.716+0000 7fd585600640 1 Processor -- start 2024-02-21T23:12:08.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.716+0000 7fd585600640 1 -- start start 2024-02-21T23:12:08.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.717+0000 7fd585600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fd5801528f0 0x7fd5800c7130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:08.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.717+0000 7fd585600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd5800c3a80 0x7fd5800c3280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:08.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.717+0000 7fd585600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd5800c5130 0x7fd5800c55c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:08.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.717+0000 7fd585600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5800c7700 con 0x7fd5800c5130 2024-02-21T23:12:08.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.717+0000 7fd585600640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5800c83b0 con 0x7fd5801528f0 2024-02-21T23:12:08.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.717+0000 7fd585600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5800c8650 con 0x7fd5800c3a80 2024-02-21T23:12:08.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.718+0000 7fd57f400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd5800c5130 0x7fd5800c55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:08.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.718+0000 7fd57f400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd5800c5130 0x7fd5800c55c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60272/0 (socket says 172.21.15.28:60272) 2024-02-21T23:12:08.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.718+0000 7fd57f400640 1 -- 172.21.15.28:0/418219124 learned_addr learned my addr 172.21.15.28:0/418219124 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:08.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.718+0000 7fd57ea00640 1 --2- 172.21.15.28:0/418219124 >> v2:172.21.15.81:3300/0 conn(0x7fd5801528f0 0x7fd5800c7130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:08.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.718+0000 7fd57e000640 1 --2- 172.21.15.28:0/418219124 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd5800c3a80 0x7fd5800c3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:08.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.718+0000 7fd57f400640 1 -- 172.21.15.28:0/418219124 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd5800c3a80 msgr2=0x7fd5800c3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:08.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.718+0000 7fd57f400640 1 --2- 172.21.15.28:0/418219124 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd5800c3a80 0x7fd5800c3280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:08.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.718+0000 7fd57f400640 1 -- 172.21.15.28:0/418219124 >> v2:172.21.15.81:3300/0 conn(0x7fd5801528f0 msgr2=0x7fd5800c7130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:08.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.718+0000 7fd57f400640 1 --2- 172.21.15.28:0/418219124 >> v2:172.21.15.81:3300/0 conn(0x7fd5801528f0 0x7fd5800c7130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:08.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.718+0000 7fd57f400640 1 -- 172.21.15.28:0/418219124 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5801591d0 con 0x7fd5800c5130 2024-02-21T23:12:08.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.718+0000 7fd57e000640 1 --2- 172.21.15.28:0/418219124 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd5800c3a80 0x7fd5800c3280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:12:08.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.720+0000 7fd57f400640 1 --2- 172.21.15.28:0/418219124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5800c5130 0x7fd5800c55c0 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7fd5800c6140 tx=0x7fd57802bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36a841d066517322 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:08.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.720+0000 7fd57d600640 1 -- 172.21.15.28:0/418219124 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd57802c950 con 0x7fd5800c5130 2024-02-21T23:12:08.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.720+0000 7fd57d600640 1 -- 172.21.15.28:0/418219124 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd57802caf0 con 0x7fd5800c5130 2024-02-21T23:12:08.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.720+0000 7fd57d600640 1 -- 172.21.15.28:0/418219124 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5780343f0 con 0x7fd5800c5130 2024-02-21T23:12:08.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.721+0000 7fd585600640 1 -- 172.21.15.28:0/418219124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5800c5130 msgr2=0x7fd5800c55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:08.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.721+0000 7fd585600640 1 --2- 172.21.15.28:0/418219124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5800c5130 0x7fd5800c55c0 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7fd5800c6140 tx=0x7fd57802bdb0 comp rx=0 tx=0).stop 2024-02-21T23:12:08.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.721+0000 7fd585600640 1 -- 172.21.15.28:0/418219124 shutdown_connections 2024-02-21T23:12:08.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.721+0000 7fd585600640 1 --2- 172.21.15.28:0/418219124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5800c5130 0x7fd5800c55c0 unknown :-1 s=CLOSED pgs=2154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:08.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.721+0000 7fd585600640 1 --2- 172.21.15.28:0/418219124 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd5800c3a80 0x7fd5800c3280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:08.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.721+0000 7fd585600640 1 --2- 172.21.15.28:0/418219124 >> v2:172.21.15.81:3300/0 conn(0x7fd5801528f0 0x7fd5800c7130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:08.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.721+0000 7fd585600640 1 -- 172.21.15.28:0/418219124 >> 172.21.15.28:0/418219124 conn(0x7fd5800c8880 msgr2=0x7fd5800c8c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:08.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.721+0000 7fd585600640 1 -- 172.21.15.28:0/418219124 shutdown_connections 2024-02-21T23:12:08.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.721+0000 7fd585600640 1 -- 172.21.15.28:0/418219124 wait complete. 2024-02-21T23:12:08.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.723+0000 7fd585600640 1 Processor -- start 2024-02-21T23:12:08.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.723+0000 7fd585600640 1 -- start start 2024-02-21T23:12:08.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.724+0000 7fd585600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5800c3a80 0x7fd580070b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:08.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.724+0000 7fd585600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd5800c5130 0x7fd58014aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:08.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.724+0000 7fd585600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5801528f0 0x7fd58014b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:08.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.724+0000 7fd585600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5800a80f0 con 0x7fd5801528f0 2024-02-21T23:12:08.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.724+0000 7fd585600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5800c7b90 con 0x7fd5800c3a80 2024-02-21T23:12:08.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.724+0000 7fd585600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5800c37c0 con 0x7fd5800c5130 2024-02-21T23:12:08.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.724+0000 7fd57f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5801528f0 0x7fd58014b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:08.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.724+0000 7fd57f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5801528f0 0x7fd58014b420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60276/0 (socket says 172.21.15.28:60276) 2024-02-21T23:12:08.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.724+0000 7fd57f400640 1 -- 172.21.15.28:0/295701314 learned_addr learned my addr 172.21.15.28:0/295701314 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:08.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.724+0000 7fd57ea00640 1 --2- 172.21.15.28:0/295701314 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5800c3a80 0x7fd580070b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:08.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.725+0000 7fd57e000640 1 --2- 172.21.15.28:0/295701314 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd5800c5130 0x7fd58014aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:08.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.725+0000 7fd57f400640 1 -- 172.21.15.28:0/295701314 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd5800c5130 msgr2=0x7fd58014aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:08.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.725+0000 7fd57f400640 1 --2- 172.21.15.28:0/295701314 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd5800c5130 0x7fd58014aee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:08.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.725+0000 7fd57f400640 1 -- 172.21.15.28:0/295701314 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5800c3a80 msgr2=0x7fd580070b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:08.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.725+0000 7fd57f400640 1 --2- 172.21.15.28:0/295701314 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5800c3a80 0x7fd580070b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:08.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.725+0000 7fd57f400640 1 -- 172.21.15.28:0/295701314 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5800adb00 con 0x7fd5801528f0 2024-02-21T23:12:08.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.725+0000 7fd57ea00640 1 --2- 172.21.15.28:0/295701314 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5800c3a80 0x7fd580070b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:12:08.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.725+0000 7fd57f400640 1 --2- 172.21.15.28:0/295701314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5801528f0 0x7fd58014b420 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 crypto rx=0x7fd578009f40 tx=0x7fd578004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:08.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.725+0000 7fd576a00640 1 -- 172.21.15.28:0/295701314 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd578043020 con 0x7fd5801528f0 2024-02-21T23:12:08.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.725+0000 7fd576a00640 1 -- 172.21.15.28:0/295701314 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd57803e450 con 0x7fd5801528f0 2024-02-21T23:12:08.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.725+0000 7fd576a00640 1 -- 172.21.15.28:0/295701314 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd57803e840 con 0x7fd5801528f0 2024-02-21T23:12:08.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.726+0000 7fd585600640 1 -- 172.21.15.28:0/295701314 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5801ede00 con 0x7fd5801528f0 2024-02-21T23:12:08.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.726+0000 7fd585600640 1 -- 172.21.15.28:0/295701314 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5801ee3c0 con 0x7fd5801528f0 2024-02-21T23:12:08.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.727+0000 7fd585600640 1 -- 172.21.15.28:0/295701314 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd580068640 con 0x7fd5801528f0 2024-02-21T23:12:08.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.728+0000 7fd576a00640 1 -- 172.21.15.28:0/295701314 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd578046020 con 0x7fd5801528f0 2024-02-21T23:12:08.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.729+0000 7fd576a00640 1 --2- 172.21.15.28:0/295701314 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd55c06b4d0 0x7fd55c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:08.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.729+0000 7fd576a00640 1 -- 172.21.15.28:0/295701314 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd5780acbb0 con 0x7fd5801528f0 2024-02-21T23:12:08.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.729+0000 7fd57ea00640 1 --2- 172.21.15.28:0/295701314 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd55c06b4d0 0x7fd55c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:08.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.730+0000 7fd57ea00640 1 --2- 172.21.15.28:0/295701314 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd55c06b4d0 0x7fd55c06d990 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fd58014ab40 tx=0x7fd570008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:08.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.733+0000 7fd576a00640 1 -- 172.21.15.28:0/295701314 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd57804d030 con 0x7fd5801528f0 2024-02-21T23:12:08.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.934+0000 7fd585600640 1 -- 172.21.15.28:0/295701314 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd580158e50 con 0x7fd5801528f0 2024-02-21T23:12:08.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.938+0000 7fd576a00640 1 -- 172.21.15.28:0/295701314 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v186) v1 ==== 74+0+16824 (secure 0 0 0) 0x7fd57807b890 con 0x7fd5801528f0 2024-02-21T23:12:08.926 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:12:08.926 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":186,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:11:39.917106+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:11:36.316421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"185","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_data","create_time":"2024-02-21T23:11:37.418709+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"186","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:12:08.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.940+0000 7fd585600640 1 -- 172.21.15.28:0/295701314 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd55c06b4d0 msgr2=0x7fd55c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:08.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.940+0000 7fd585600640 1 --2- 172.21.15.28:0/295701314 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd55c06b4d0 0x7fd55c06d990 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fd58014ab40 tx=0x7fd570008040 comp rx=0 tx=0).stop 2024-02-21T23:12:08.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.940+0000 7fd585600640 1 -- 172.21.15.28:0/295701314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5801528f0 msgr2=0x7fd58014b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:08.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.940+0000 7fd585600640 1 --2- 172.21.15.28:0/295701314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5801528f0 0x7fd58014b420 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 crypto rx=0x7fd578009f40 tx=0x7fd578004a40 comp rx=0 tx=0).stop 2024-02-21T23:12:08.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.941+0000 7fd585600640 1 -- 172.21.15.28:0/295701314 shutdown_connections 2024-02-21T23:12:08.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.941+0000 7fd585600640 1 --2- 172.21.15.28:0/295701314 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd55c06b4d0 0x7fd55c06d990 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:08.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.941+0000 7fd585600640 1 --2- 172.21.15.28:0/295701314 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd5800c5130 0x7fd58014aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:08.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.941+0000 7fd585600640 1 --2- 172.21.15.28:0/295701314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5801528f0 0x7fd58014b420 unknown :-1 s=CLOSED pgs=2155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:08.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.941+0000 7fd585600640 1 --2- 172.21.15.28:0/295701314 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5800c3a80 0x7fd580070b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:08.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.941+0000 7fd585600640 1 -- 172.21.15.28:0/295701314 >> 172.21.15.28:0/295701314 conn(0x7fd5800c8880 msgr2=0x7fd580153020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:08.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.941+0000 7fd585600640 1 -- 172.21.15.28:0/295701314 shutdown_connections 2024-02-21T23:12:08.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:08.941+0000 7fd585600640 1 -- 172.21.15.28:0/295701314 wait complete. 2024-02-21T23:12:08.949 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-21T23:02:30.728329+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 29, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-21T23:11:36.316421+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '185', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 30, 'pool_name': 'cephfs_data', 'create_time': '2024-02-21T23:11:37.418709+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '186', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-21T23:12:08.950 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-21T23:12:09.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.186+0000 7f3b79a00640 1 Processor -- start 2024-02-21T23:12:09.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.187+0000 7f3b79a00640 1 -- start start 2024-02-21T23:12:09.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.188+0000 7f3b79a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3b7406dbc0 0x7f3b7406c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:09.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.188+0000 7f3b79a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b74158550 con 0x7f3b7413e0c0 2024-02-21T23:12:09.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.188+0000 7f3b79a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b741587d0 con 0x7f3b7413e520 2024-02-21T23:12:09.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.188+0000 7f3b79a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b740c89a0 con 0x7f3b7406dbc0 2024-02-21T23:12:09.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.189+0000 7f3b73e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f3b7413e0c0 0x7f3b740c71b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53768/0 (socket says 172.21.15.28:53768) 2024-02-21T23:12:09.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.189+0000 7f3b73e00640 1 -- 172.21.15.28:0/1036191376 learned_addr learned my addr 172.21.15.28:0/1036191376 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:12:09.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.189+0000 7f3b73400640 1 --2- 172.21.15.28:0/1036191376 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3b7406dbc0 0x7f3b7406c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:09.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.189+0000 7f3b72a00640 1 -- 172.21.15.28:0/1036191376 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 319891282 0 0) 0x7f3b64001110 con 0x7f3b7413e0c0 2024-02-21T23:12:09.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.189+0000 7f3b72a00640 1 -- 172.21.15.28:0/1036191376 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b54003520 con 0x7f3b7413e0c0 2024-02-21T23:12:09.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.190+0000 7f3b72a00640 1 -- 172.21.15.28:0/1036191376 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3037731578 0 0) 0x7f3b54003520 con 0x7f3b7413e0c0 2024-02-21T23:12:09.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.190+0000 7f3b72a00640 1 -- 172.21.15.28:0/1036191376 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3b7406dbc0 msgr2=0x7f3b7406c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:09.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.190+0000 7f3b72a00640 1 --2- 172.21.15.28:0/1036191376 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3b7406dbc0 0x7f3b7406c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:09.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.190+0000 7f3b72a00640 1 -- 172.21.15.28:0/1036191376 >> v1:172.21.15.81:6789/0 conn(0x7f3b7413e520 legacy=0x7f3b7406c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:09.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.190+0000 7f3b72a00640 1 -- 172.21.15.28:0/1036191376 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b74158ab0 con 0x7f3b7413e0c0 2024-02-21T23:12:09.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.190+0000 7f3b72a00640 1 -- 172.21.15.28:0/1036191376 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f3b64001840 con 0x7f3b7413e0c0 2024-02-21T23:12:09.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.190+0000 7f3b72a00640 1 -- 172.21.15.28:0/1036191376 >> v1:172.21.15.28:6789/0 conn(0x7f3b7413e0c0 legacy=0x7f3b740c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:09.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.190+0000 7f3b72a00640 1 --2- 172.21.15.28:0/1036191376 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b540039e0 0x7f3b54003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:09.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.190+0000 7f3b72a00640 1 --2- 172.21.15.28:0/1036191376 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3b7406dbc0 0x7f3b54004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:09.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.190+0000 7f3b72a00640 1 --2- 172.21.15.28:0/1036191376 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b54004d40 0x7f3b540051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:09.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.190+0000 7f3b72a00640 1 -- 172.21.15.28:0/1036191376 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b5400bac0 con 0x7f3b54004d40 2024-02-21T23:12:09.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.190+0000 7f3b72a00640 1 -- 172.21.15.28:0/1036191376 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b5400bc70 con 0x7f3b540039e0 2024-02-21T23:12:09.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.190+0000 7f3b72a00640 1 -- 172.21.15.28:0/1036191376 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b5400be70 con 0x7f3b7406dbc0 2024-02-21T23:12:09.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.190+0000 7f3b79000640 1 --2- 172.21.15.28:0/1036191376 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b54004d40 0x7f3b540051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:09.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.190+0000 7f3b73e00640 1 --2- 172.21.15.28:0/1036191376 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b540039e0 0x7f3b54003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:09.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.190+0000 7f3b73400640 1 --2- 172.21.15.28:0/1036191376 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3b7406dbc0 0x7f3b54004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:09.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.191+0000 7f3b79000640 1 -- 172.21.15.28:0/1036191376 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3b7406dbc0 msgr2=0x7f3b54004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:09.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.191+0000 7f3b79000640 1 --2- 172.21.15.28:0/1036191376 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3b7406dbc0 0x7f3b54004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:09.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.191+0000 7f3b79000640 1 -- 172.21.15.28:0/1036191376 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b540039e0 msgr2=0x7f3b54003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:09.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.191+0000 7f3b79000640 1 --2- 172.21.15.28:0/1036191376 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b540039e0 0x7f3b54003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:09.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.191+0000 7f3b79000640 1 -- 172.21.15.28:0/1036191376 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b54003520 con 0x7f3b54004d40 2024-02-21T23:12:09.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.192+0000 7f3b79000640 1 --2- 172.21.15.28:0/1036191376 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b54004d40 0x7f3b540051c0 secure :-1 s=READY pgs=2157 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c001e50 tx=0x7f3b54005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70cd011582d20caa server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:09.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.192+0000 7f3b72a00640 1 -- 172.21.15.28:0/1036191376 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b6c001a70 con 0x7f3b54004d40 2024-02-21T23:12:09.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.192+0000 7f3b72a00640 1 -- 172.21.15.28:0/1036191376 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b6c003b70 con 0x7f3b54004d40 2024-02-21T23:12:09.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.192+0000 7f3b72a00640 1 -- 172.21.15.28:0/1036191376 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b6c0050d0 con 0x7f3b54004d40 2024-02-21T23:12:09.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.193+0000 7f3b79a00640 1 -- 172.21.15.28:0/1036191376 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b54004d40 msgr2=0x7f3b540051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:09.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.193+0000 7f3b79a00640 1 --2- 172.21.15.28:0/1036191376 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b54004d40 0x7f3b540051c0 secure :-1 s=READY pgs=2157 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c001e50 tx=0x7f3b54005830 comp rx=0 tx=0).stop 2024-02-21T23:12:09.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.193+0000 7f3b79000640 1 -- 172.21.15.28:0/1036191376 reap_dead start 2024-02-21T23:12:09.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.193+0000 7f3b79a00640 1 -- 172.21.15.28:0/1036191376 shutdown_connections 2024-02-21T23:12:09.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.193+0000 7f3b79a00640 1 -- 172.21.15.28:0/1036191376 >> 172.21.15.28:0/1036191376 conn(0x7f3b74073110 msgr2=0x7f3b740ac490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:09.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.193+0000 7f3b79a00640 1 -- 172.21.15.28:0/1036191376 shutdown_connections 2024-02-21T23:12:09.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.193+0000 7f3b79a00640 1 -- 172.21.15.28:0/1036191376 wait complete. 2024-02-21T23:12:09.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.195+0000 7f3b79a00640 1 Processor -- start 2024-02-21T23:12:09.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.195+0000 7f3b79a00640 1 -- start start 2024-02-21T23:12:09.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.196+0000 7f3b79a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3b54004d40 0x7f3b74152e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:09.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.196+0000 7f3b79a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b741520b0 0x7f3b741ee030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:09.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.196+0000 7f3b79a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b740af5d0 0x7f3b74150ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:09.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.196+0000 7f3b79a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b7406ced0 con 0x7f3b740af5d0 2024-02-21T23:12:09.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.196+0000 7f3b79a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b74159070 con 0x7f3b741520b0 2024-02-21T23:12:09.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.196+0000 7f3b79a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b740c78c0 con 0x7f3b54004d40 2024-02-21T23:12:09.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.196+0000 7f3b79000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b740af5d0 0x7f3b74150ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:09.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.196+0000 7f3b79000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b740af5d0 0x7f3b74150ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60300/0 (socket says 172.21.15.28:60300) 2024-02-21T23:12:09.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.196+0000 7f3b79000640 1 -- 172.21.15.28:0/2639166275 learned_addr learned my addr 172.21.15.28:0/2639166275 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:09.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.196+0000 7f3b73e00640 1 --2- 172.21.15.28:0/2639166275 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3b54004d40 0x7f3b74152e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:09.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.196+0000 7f3b73400640 1 --2- 172.21.15.28:0/2639166275 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b741520b0 0x7f3b741ee030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:09.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.196+0000 7f3b79000640 1 -- 172.21.15.28:0/2639166275 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3b54004d40 msgr2=0x7f3b74152e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:09.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.196+0000 7f3b79000640 1 --2- 172.21.15.28:0/2639166275 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3b54004d40 0x7f3b74152e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:09.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.196+0000 7f3b79000640 1 -- 172.21.15.28:0/2639166275 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b741520b0 msgr2=0x7f3b741ee030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:09.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.196+0000 7f3b79000640 1 --2- 172.21.15.28:0/2639166275 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b741520b0 0x7f3b741ee030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:09.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.196+0000 7f3b79000640 1 -- 172.21.15.28:0/2639166275 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b741ee870 con 0x7f3b740af5d0 2024-02-21T23:12:09.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.197+0000 7f3b79000640 1 --2- 172.21.15.28:0/2639166275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b740af5d0 0x7f3b74150ec0 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c001f70 tx=0x7f3b6c0014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:09.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.197+0000 7f3b6be00640 1 -- 172.21.15.28:0/2639166275 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b6c047070 con 0x7f3b740af5d0 2024-02-21T23:12:09.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.197+0000 7f3b6be00640 1 -- 172.21.15.28:0/2639166275 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b6c006b50 con 0x7f3b740af5d0 2024-02-21T23:12:09.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.197+0000 7f3b6be00640 1 -- 172.21.15.28:0/2639166275 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b6c007db0 con 0x7f3b740af5d0 2024-02-21T23:12:09.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.197+0000 7f3b79a00640 1 -- 172.21.15.28:0/2639166275 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b74151e60 con 0x7f3b740af5d0 2024-02-21T23:12:09.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.197+0000 7f3b79a00640 1 -- 172.21.15.28:0/2639166275 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b740b0060 con 0x7f3b740af5d0 2024-02-21T23:12:09.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.199+0000 7f3b79a00640 1 -- 172.21.15.28:0/2639166275 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b7404f290 con 0x7f3b740af5d0 2024-02-21T23:12:09.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.199+0000 7f3b6be00640 1 -- 172.21.15.28:0/2639166275 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3b6c04a070 con 0x7f3b740af5d0 2024-02-21T23:12:09.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.200+0000 7f3b6be00640 1 --2- 172.21.15.28:0/2639166275 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3b5006b4d0 0x7f3b5006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:09.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.200+0000 7f3b6be00640 1 -- 172.21.15.28:0/2639166275 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f3b6c0abae0 con 0x7f3b740af5d0 2024-02-21T23:12:09.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.201+0000 7f3b73e00640 1 --2- 172.21.15.28:0/2639166275 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3b5006b4d0 0x7f3b5006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:09.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.202+0000 7f3b73e00640 1 --2- 172.21.15.28:0/2639166275 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3b5006b4d0 0x7f3b5006d990 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f3b64001050 tx=0x7f3b64009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:09.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.205+0000 7f3b6be00640 1 -- 172.21.15.28:0/2639166275 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3b6c033020 con 0x7f3b740af5d0 2024-02-21T23:12:09.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.406+0000 7f3b79a00640 1 -- 172.21.15.28:0/2639166275 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3b7406d9a0 con 0x7f3b740af5d0 2024-02-21T23:12:09.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.407+0000 7f3b6be00640 1 -- 172.21.15.28:0/2639166275 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v186) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3b6c07a7c0 con 0x7f3b740af5d0 2024-02-21T23:12:09.395 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-02-21T23:12:09.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.410+0000 7f3b79a00640 1 -- 172.21.15.28:0/2639166275 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3b5006b4d0 msgr2=0x7f3b5006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:09.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.410+0000 7f3b79a00640 1 --2- 172.21.15.28:0/2639166275 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3b5006b4d0 0x7f3b5006d990 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f3b64001050 tx=0x7f3b64009040 comp rx=0 tx=0).stop 2024-02-21T23:12:09.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.410+0000 7f3b79a00640 1 -- 172.21.15.28:0/2639166275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b740af5d0 msgr2=0x7f3b74150ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:09.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.410+0000 7f3b79a00640 1 --2- 172.21.15.28:0/2639166275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b740af5d0 0x7f3b74150ec0 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c001f70 tx=0x7f3b6c0014a0 comp rx=0 tx=0).stop 2024-02-21T23:12:09.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.410+0000 7f3b79a00640 1 -- 172.21.15.28:0/2639166275 shutdown_connections 2024-02-21T23:12:09.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.410+0000 7f3b79a00640 1 --2- 172.21.15.28:0/2639166275 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3b5006b4d0 0x7f3b5006d990 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:09.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.410+0000 7f3b79a00640 1 --2- 172.21.15.28:0/2639166275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b740af5d0 0x7f3b74150ec0 unknown :-1 s=CLOSED pgs=2158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:09.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.410+0000 7f3b79a00640 1 --2- 172.21.15.28:0/2639166275 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b741520b0 0x7f3b741ee030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:09.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.410+0000 7f3b79a00640 1 --2- 172.21.15.28:0/2639166275 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3b54004d40 0x7f3b74152e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:09.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.410+0000 7f3b79a00640 1 -- 172.21.15.28:0/2639166275 >> 172.21.15.28:0/2639166275 conn(0x7f3b74073110 msgr2=0x7f3b7406c280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:09.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.410+0000 7f3b79a00640 1 -- 172.21.15.28:0/2639166275 shutdown_connections 2024-02-21T23:12:09.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.410+0000 7f3b79a00640 1 -- 172.21.15.28:0/2639166275 wait complete. 2024-02-21T23:12:09.419 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-21T23:12:09.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.659+0000 7fa7e1200640 1 Processor -- start 2024-02-21T23:12:09.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.659+0000 7fa7e1200640 1 -- start start 2024-02-21T23:12:09.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.660+0000 7fa7e1200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fa7dc150240 0x7fa7dc0c12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:09.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.660+0000 7fa7e1200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa7dc0c2c10 0x7fa7dc06ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:09.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.660+0000 7fa7e1200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7dc06e220 con 0x7fa7dc0c4580 2024-02-21T23:12:09.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.660+0000 7fa7e1200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7dc150620 con 0x7fa7dc0c2c10 2024-02-21T23:12:09.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.660+0000 7fa7e1200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7dc153a90 con 0x7fa7dc150240 2024-02-21T23:12:09.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.661+0000 7fa7dbe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa7dc0c2c10 0x7fa7dc06ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:09.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.661+0000 7fa7dbe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa7dc0c2c10 0x7fa7dc06ae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60314/0 (socket says 172.21.15.28:60314) 2024-02-21T23:12:09.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.661+0000 7fa7daa00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa7dc0c4580 0x7fa7dc06db10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53774/0 (socket says 172.21.15.28:53774) 2024-02-21T23:12:09.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.661+0000 7fa7dbe00640 1 -- 172.21.15.28:0/1185970984 learned_addr learned my addr 172.21.15.28:0/1185970984 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:09.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.661+0000 7fa7db400640 1 --2- 172.21.15.28:0/1185970984 >> v2:172.21.15.81:3300/0 conn(0x7fa7dc150240 0x7fa7dc0c12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:09.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.661+0000 7fa7dbe00640 1 -- 172.21.15.28:0/1185970984 >> v2:172.21.15.81:3300/0 conn(0x7fa7dc150240 msgr2=0x7fa7dc0c12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:09.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.661+0000 7fa7dbe00640 1 --2- 172.21.15.28:0/1185970984 >> v2:172.21.15.81:3300/0 conn(0x7fa7dc150240 0x7fa7dc0c12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:09.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.661+0000 7fa7dbe00640 1 -- 172.21.15.28:0/1185970984 >> v1:172.21.15.28:6789/0 conn(0x7fa7dc0c4580 legacy=0x7fa7dc06db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:09.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.661+0000 7fa7dbe00640 1 -- 172.21.15.28:0/1185970984 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7dc06f780 con 0x7fa7dc0c2c10 2024-02-21T23:12:09.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.661+0000 7fa7da000640 1 -- 172.21.15.28:0/1185970984 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 732219300 0 0) 0x7fa7dc06e220 con 0x7fa7dc0c4580 2024-02-21T23:12:09.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.663+0000 7fa7dbe00640 1 --2- 172.21.15.28:0/1185970984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7dc0c2c10 0x7fa7dc06ae70 secure :-1 s=READY pgs=2160 cs=0 l=1 rev1=1 crypto rx=0x7fa7dc06bef0 tx=0x7fa7d402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ee774779f81d683 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:09.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.663+0000 7fa7da000640 1 -- 172.21.15.28:0/1185970984 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d402ca80 con 0x7fa7dc0c2c10 2024-02-21T23:12:09.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.663+0000 7fa7da000640 1 -- 172.21.15.28:0/1185970984 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7d402cc20 con 0x7fa7dc0c2c10 2024-02-21T23:12:09.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.663+0000 7fa7da000640 1 -- 172.21.15.28:0/1185970984 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d40343f0 con 0x7fa7dc0c2c10 2024-02-21T23:12:09.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.663+0000 7fa7e1200640 1 -- 172.21.15.28:0/1185970984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7dc0c2c10 msgr2=0x7fa7dc06ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:09.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.663+0000 7fa7e1200640 1 --2- 172.21.15.28:0/1185970984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7dc0c2c10 0x7fa7dc06ae70 secure :-1 s=READY pgs=2160 cs=0 l=1 rev1=1 crypto rx=0x7fa7dc06bef0 tx=0x7fa7d402bee0 comp rx=0 tx=0).stop 2024-02-21T23:12:09.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.663+0000 7fa7e1200640 1 -- 172.21.15.28:0/1185970984 shutdown_connections 2024-02-21T23:12:09.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.663+0000 7fa7e1200640 1 --2- 172.21.15.28:0/1185970984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7dc0c2c10 0x7fa7dc06ae70 unknown :-1 s=CLOSED pgs=2160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:09.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.663+0000 7fa7e1200640 1 --2- 172.21.15.28:0/1185970984 >> v2:172.21.15.81:3300/0 conn(0x7fa7dc150240 0x7fa7dc0c12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:09.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.664+0000 7fa7e1200640 1 -- 172.21.15.28:0/1185970984 >> 172.21.15.28:0/1185970984 conn(0x7fa7dc0a8530 msgr2=0x7fa7dc0a8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:09.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.664+0000 7fa7e1200640 1 -- 172.21.15.28:0/1185970984 shutdown_connections 2024-02-21T23:12:09.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.664+0000 7fa7e1200640 1 -- 172.21.15.28:0/1185970984 wait complete. 2024-02-21T23:12:09.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.665+0000 7fa7e1200640 1 Processor -- start 2024-02-21T23:12:09.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.665+0000 7fa7e1200640 1 -- start start 2024-02-21T23:12:09.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.666+0000 7fa7e1200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7dc0c2c10 0x7fa7dc0ab7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:09.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.666+0000 7fa7e1200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa7dc0c4580 0x7fa7dc1ebcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:09.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.666+0000 7fa7e1200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa7dc150240 0x7fa7dc1ec210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:09.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.666+0000 7fa7e1200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7dc0c4a10 con 0x7fa7dc0c2c10 2024-02-21T23:12:09.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.666+0000 7fa7e1200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7dc153720 con 0x7fa7dc150240 2024-02-21T23:12:09.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.666+0000 7fa7e1200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7dc0c4b90 con 0x7fa7dc0c4580 2024-02-21T23:12:09.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.667+0000 7fa7db400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7dc0c2c10 0x7fa7dc0ab7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:09.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.667+0000 7fa7db400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7dc0c2c10 0x7fa7dc0ab7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60326/0 (socket says 172.21.15.28:60326) 2024-02-21T23:12:09.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.667+0000 7fa7db400640 1 -- 172.21.15.28:0/4178248690 learned_addr learned my addr 172.21.15.28:0/4178248690 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:09.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.667+0000 7fa7daa00640 1 --2- 172.21.15.28:0/4178248690 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa7dc0c4580 0x7fa7dc1ebcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:09.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.667+0000 7fa7dbe00640 1 --2- 172.21.15.28:0/4178248690 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa7dc150240 0x7fa7dc1ec210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:09.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.667+0000 7fa7db400640 1 -- 172.21.15.28:0/4178248690 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa7dc0c4580 msgr2=0x7fa7dc1ebcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:09.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.667+0000 7fa7db400640 1 --2- 172.21.15.28:0/4178248690 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa7dc0c4580 0x7fa7dc1ebcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:09.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.667+0000 7fa7db400640 1 -- 172.21.15.28:0/4178248690 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa7dc150240 msgr2=0x7fa7dc1ec210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:09.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.667+0000 7fa7db400640 1 --2- 172.21.15.28:0/4178248690 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa7dc150240 0x7fa7dc1ec210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:09.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.667+0000 7fa7db400640 1 -- 172.21.15.28:0/4178248690 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7dc0bbc00 con 0x7fa7dc0c2c10 2024-02-21T23:12:09.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.667+0000 7fa7db400640 1 --2- 172.21.15.28:0/4178248690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7dc0c2c10 0x7fa7dc0ab7d0 secure :-1 s=READY pgs=2161 cs=0 l=1 rev1=1 crypto rx=0x7fa7cc00d830 tx=0x7fa7cc00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:09.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.668+0000 7fa7d3400640 1 -- 172.21.15.28:0/4178248690 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7cc00b6c0 con 0x7fa7dc0c2c10 2024-02-21T23:12:09.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.668+0000 7fa7d3400640 1 -- 172.21.15.28:0/4178248690 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7cc019070 con 0x7fa7dc0c2c10 2024-02-21T23:12:09.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.668+0000 7fa7d3400640 1 -- 172.21.15.28:0/4178248690 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7cc00ba80 con 0x7fa7dc0c2c10 2024-02-21T23:12:09.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.668+0000 7fa7e1200640 1 -- 172.21.15.28:0/4178248690 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7dc0bc400 con 0x7fa7dc0c2c10 2024-02-21T23:12:09.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.668+0000 7fa7e1200640 1 -- 172.21.15.28:0/4178248690 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7dc153c40 con 0x7fa7dc0c2c10 2024-02-21T23:12:09.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.669+0000 7fa7e1200640 1 -- 172.21.15.28:0/4178248690 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7dc04f290 con 0x7fa7dc0c2c10 2024-02-21T23:12:09.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.670+0000 7fa7d3400640 1 -- 172.21.15.28:0/4178248690 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa7cc01e460 con 0x7fa7dc0c2c10 2024-02-21T23:12:09.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.671+0000 7fa7d3400640 1 --2- 172.21.15.28:0/4178248690 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa7bc06b4d0 0x7fa7bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:09.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.671+0000 7fa7d3400640 1 -- 172.21.15.28:0/4178248690 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa7cc08aa50 con 0x7fa7dc0c2c10 2024-02-21T23:12:09.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.672+0000 7fa7daa00640 1 --2- 172.21.15.28:0/4178248690 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa7bc06b4d0 0x7fa7bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:09.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.672+0000 7fa7daa00640 1 --2- 172.21.15.28:0/4178248690 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa7bc06b4d0 0x7fa7bc06d990 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fa7dc1e60a0 tx=0x7fa7c8007b70 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:09.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.676+0000 7fa7d3400640 1 -- 172.21.15.28:0/4178248690 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fa7cc059730 con 0x7fa7dc0c2c10 2024-02-21T23:12:09.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.874+0000 7fa7e1200640 1 -- 172.21.15.28:0/4178248690 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa7dc06f0e0 con 0x7fa7dc0c2c10 2024-02-21T23:12:09.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.875+0000 7fa7d3400640 1 -- 172.21.15.28:0/4178248690 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v186) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa7cc059030 con 0x7fa7dc0c2c10 2024-02-21T23:12:09.863 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-02-21T23:12:09.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.878+0000 7fa7e1200640 1 -- 172.21.15.28:0/4178248690 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa7bc06b4d0 msgr2=0x7fa7bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:09.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.878+0000 7fa7e1200640 1 --2- 172.21.15.28:0/4178248690 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa7bc06b4d0 0x7fa7bc06d990 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fa7dc1e60a0 tx=0x7fa7c8007b70 comp rx=0 tx=0).stop 2024-02-21T23:12:09.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.878+0000 7fa7e1200640 1 -- 172.21.15.28:0/4178248690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7dc0c2c10 msgr2=0x7fa7dc0ab7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:09.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.878+0000 7fa7e1200640 1 --2- 172.21.15.28:0/4178248690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7dc0c2c10 0x7fa7dc0ab7d0 secure :-1 s=READY pgs=2161 cs=0 l=1 rev1=1 crypto rx=0x7fa7cc00d830 tx=0x7fa7cc00dcf0 comp rx=0 tx=0).stop 2024-02-21T23:12:09.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.878+0000 7fa7e1200640 1 -- 172.21.15.28:0/4178248690 shutdown_connections 2024-02-21T23:12:09.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.878+0000 7fa7e1200640 1 --2- 172.21.15.28:0/4178248690 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa7bc06b4d0 0x7fa7bc06d990 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:09.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.878+0000 7fa7e1200640 1 --2- 172.21.15.28:0/4178248690 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa7dc0c4580 0x7fa7dc1ebcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:09.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.878+0000 7fa7e1200640 1 --2- 172.21.15.28:0/4178248690 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa7dc150240 0x7fa7dc1ec210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:09.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.878+0000 7fa7e1200640 1 --2- 172.21.15.28:0/4178248690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7dc0c2c10 0x7fa7dc0ab7d0 unknown :-1 s=CLOSED pgs=2161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:09.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.878+0000 7fa7e1200640 1 -- 172.21.15.28:0/4178248690 >> 172.21.15.28:0/4178248690 conn(0x7fa7dc0a8530 msgr2=0x7fa7dc06ae70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:09.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.878+0000 7fa7e1200640 1 -- 172.21.15.28:0/4178248690 shutdown_connections 2024-02-21T23:12:09.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:09.878+0000 7fa7e1200640 1 -- 172.21.15.28:0/4178248690 wait complete. 2024-02-21T23:12:09.886 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-21T23:12:10.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.119+0000 7fa99be00640 1 Processor -- start 2024-02-21T23:12:10.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.120+0000 7fa99be00640 1 -- start start 2024-02-21T23:12:10.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.121+0000 7fa99be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa99c13e0c0 0x7fa99c0c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:10.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.121+0000 7fa99be00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fa99c06dbc0 0x7fa99c06c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:10.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.121+0000 7fa99be00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa99c158550 con 0x7fa99c0c76f0 2024-02-21T23:12:10.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.121+0000 7fa99be00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa99c0c89a0 con 0x7fa99c13e0c0 2024-02-21T23:12:10.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.121+0000 7fa99be00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa99c1587d0 con 0x7fa99c06dbc0 2024-02-21T23:12:10.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.121+0000 7fa99aa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa99c13e0c0 0x7fa99c0c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:10.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.121+0000 7fa99a000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fa99c06dbc0 0x7fa99c06c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:10.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.121+0000 7fa99b400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa99c0c76f0 0x7fa99c06c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53790/0 (socket says 172.21.15.28:53790) 2024-02-21T23:12:10.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.121+0000 7fa99b400640 1 -- 172.21.15.28:0/3317602736 learned_addr learned my addr 172.21.15.28:0/3317602736 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:12:10.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.121+0000 7fa99aa00640 1 --2- 172.21.15.28:0/3317602736 >> v2:172.21.15.28:3300/0 conn(0x7fa99c13e0c0 0x7fa99c0c71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60338/0 (socket says 172.21.15.28:60338) 2024-02-21T23:12:10.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.122+0000 7fa99aa00640 1 -- 172.21.15.28:0/3317602736 >> v2:172.21.15.81:3300/0 conn(0x7fa99c06dbc0 msgr2=0x7fa99c06c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:10.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.122+0000 7fa99aa00640 1 --2- 172.21.15.28:0/3317602736 >> v2:172.21.15.81:3300/0 conn(0x7fa99c06dbc0 0x7fa99c06c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:10.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.122+0000 7fa99aa00640 1 -- 172.21.15.28:0/3317602736 >> v1:172.21.15.28:6789/0 conn(0x7fa99c0c76f0 legacy=0x7fa99c06c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:10.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.122+0000 7fa99aa00640 1 -- 172.21.15.28:0/3317602736 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa99c158ab0 con 0x7fa99c13e0c0 2024-02-21T23:12:10.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.123+0000 7fa99aa00640 1 --2- 172.21.15.28:0/3317602736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa99c13e0c0 0x7fa99c0c71b0 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 crypto rx=0x7fa99c0c7100 tx=0x7fa98c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=528a512423823bab server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:10.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.123+0000 7fa999600640 1 -- 172.21.15.28:0/3317602736 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa98c039070 con 0x7fa99c13e0c0 2024-02-21T23:12:10.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.123+0000 7fa999600640 1 -- 172.21.15.28:0/3317602736 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa98c02cbf0 con 0x7fa99c13e0c0 2024-02-21T23:12:10.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.123+0000 7fa999600640 1 -- 172.21.15.28:0/3317602736 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa98c034440 con 0x7fa99c13e0c0 2024-02-21T23:12:10.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.124+0000 7fa99be00640 1 -- 172.21.15.28:0/3317602736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa99c13e0c0 msgr2=0x7fa99c0c71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:10.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.124+0000 7fa99be00640 1 --2- 172.21.15.28:0/3317602736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa99c13e0c0 0x7fa99c0c71b0 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 crypto rx=0x7fa99c0c7100 tx=0x7fa98c02be40 comp rx=0 tx=0).stop 2024-02-21T23:12:10.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.124+0000 7fa99be00640 1 -- 172.21.15.28:0/3317602736 shutdown_connections 2024-02-21T23:12:10.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.124+0000 7fa99be00640 1 --2- 172.21.15.28:0/3317602736 >> v2:172.21.15.81:3300/0 conn(0x7fa99c06dbc0 0x7fa99c06c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:10.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.124+0000 7fa99be00640 1 --2- 172.21.15.28:0/3317602736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa99c13e0c0 0x7fa99c0c71b0 unknown :-1 s=CLOSED pgs=2163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:10.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.124+0000 7fa99be00640 1 -- 172.21.15.28:0/3317602736 >> 172.21.15.28:0/3317602736 conn(0x7fa99c073110 msgr2=0x7fa99c0ac490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:10.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.124+0000 7fa99be00640 1 -- 172.21.15.28:0/3317602736 shutdown_connections 2024-02-21T23:12:10.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.124+0000 7fa99be00640 1 -- 172.21.15.28:0/3317602736 wait complete. 2024-02-21T23:12:10.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.126+0000 7fa99be00640 1 Processor -- start 2024-02-21T23:12:10.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.126+0000 7fa99be00640 1 -- start start 2024-02-21T23:12:10.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.127+0000 7fa99be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa99c06dbc0 0x7fa99c0b0120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:10.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.127+0000 7fa99be00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa99c0c76f0 0x7fa99c154ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:10.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.127+0000 7fa99be00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa99c13e0c0 0x7fa99c151d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:10.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.127+0000 7fa99be00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa99c06ced0 con 0x7fa99c06dbc0 2024-02-21T23:12:10.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.127+0000 7fa99be00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa99c159070 con 0x7fa99c13e0c0 2024-02-21T23:12:10.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.127+0000 7fa99be00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa99c13e520 con 0x7fa99c0c76f0 2024-02-21T23:12:10.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.127+0000 7fa99aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa99c06dbc0 0x7fa99c0b0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:10.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.127+0000 7fa99aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa99c06dbc0 0x7fa99c0b0120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60354/0 (socket says 172.21.15.28:60354) 2024-02-21T23:12:10.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.127+0000 7fa99aa00640 1 -- 172.21.15.28:0/2636328818 learned_addr learned my addr 172.21.15.28:0/2636328818 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:10.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.127+0000 7fa99a000640 1 --2- 172.21.15.28:0/2636328818 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa99c0c76f0 0x7fa99c154ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:10.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.127+0000 7fa99b400640 1 --2- 172.21.15.28:0/2636328818 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa99c13e0c0 0x7fa99c151d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:10.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.128+0000 7fa99aa00640 1 -- 172.21.15.28:0/2636328818 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa99c0c76f0 msgr2=0x7fa99c154ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:10.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.128+0000 7fa99aa00640 1 --2- 172.21.15.28:0/2636328818 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa99c0c76f0 0x7fa99c154ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:10.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.128+0000 7fa99aa00640 1 -- 172.21.15.28:0/2636328818 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa99c13e0c0 msgr2=0x7fa99c151d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:10.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.128+0000 7fa99aa00640 1 --2- 172.21.15.28:0/2636328818 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa99c13e0c0 0x7fa99c151d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:10.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.128+0000 7fa99aa00640 1 -- 172.21.15.28:0/2636328818 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa99c153f50 con 0x7fa99c06dbc0 2024-02-21T23:12:10.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.128+0000 7fa99b400640 1 --2- 172.21.15.28:0/2636328818 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa99c13e0c0 0x7fa99c151d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:12:10.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.128+0000 7fa99aa00640 1 --2- 172.21.15.28:0/2636328818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa99c06dbc0 0x7fa99c0b0120 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 crypto rx=0x7fa98c037040 tx=0x7fa98c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:10.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.128+0000 7fa992a00640 1 -- 172.21.15.28:0/2636328818 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa98c039050 con 0x7fa99c06dbc0 2024-02-21T23:12:10.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.128+0000 7fa992a00640 1 -- 172.21.15.28:0/2636328818 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa98c045070 con 0x7fa99c06dbc0 2024-02-21T23:12:10.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.128+0000 7fa992a00640 1 -- 172.21.15.28:0/2636328818 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa98c03e850 con 0x7fa99c06dbc0 2024-02-21T23:12:10.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.128+0000 7fa99be00640 1 -- 172.21.15.28:0/2636328818 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa99c154230 con 0x7fa99c06dbc0 2024-02-21T23:12:10.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.128+0000 7fa99be00640 1 -- 172.21.15.28:0/2636328818 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa99c0ad790 con 0x7fa99c06dbc0 2024-02-21T23:12:10.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.130+0000 7fa99be00640 1 -- 172.21.15.28:0/2636328818 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa99c154750 con 0x7fa99c06dbc0 2024-02-21T23:12:10.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.131+0000 7fa992a00640 1 -- 172.21.15.28:0/2636328818 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa98c03e9f0 con 0x7fa99c06dbc0 2024-02-21T23:12:10.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.131+0000 7fa992a00640 1 --2- 172.21.15.28:0/2636328818 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa97806b4d0 0x7fa97806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:10.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.132+0000 7fa992a00640 1 -- 172.21.15.28:0/2636328818 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa98c0ad120 con 0x7fa99c06dbc0 2024-02-21T23:12:10.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.132+0000 7fa99a000640 1 --2- 172.21.15.28:0/2636328818 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa97806b4d0 0x7fa97806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:10.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.133+0000 7fa99a000640 1 --2- 172.21.15.28:0/2636328818 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa97806b4d0 0x7fa97806d990 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fa99c156030 tx=0x7fa984008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:10.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.136+0000 7fa992a00640 1 -- 172.21.15.28:0/2636328818 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fa99c154750 con 0x7fa99c06dbc0 2024-02-21T23:12:10.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.336+0000 7fa99be00640 1 -- 172.21.15.28:0/2636328818 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa99c0ad580 con 0x7fa99c06dbc0 2024-02-21T23:12:10.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.338+0000 7fa992a00640 1 -- 172.21.15.28:0/2636328818 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v186) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa98c032e20 con 0x7fa99c06dbc0 2024-02-21T23:12:10.325 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-02-21T23:12:10.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.340+0000 7fa99be00640 1 -- 172.21.15.28:0/2636328818 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa97806b4d0 msgr2=0x7fa97806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:10.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.340+0000 7fa99be00640 1 --2- 172.21.15.28:0/2636328818 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa97806b4d0 0x7fa97806d990 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fa99c156030 tx=0x7fa984008040 comp rx=0 tx=0).stop 2024-02-21T23:12:10.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.340+0000 7fa99be00640 1 -- 172.21.15.28:0/2636328818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa99c06dbc0 msgr2=0x7fa99c0b0120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:10.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.340+0000 7fa99be00640 1 --2- 172.21.15.28:0/2636328818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa99c06dbc0 0x7fa99c0b0120 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 crypto rx=0x7fa98c037040 tx=0x7fa98c004a40 comp rx=0 tx=0).stop 2024-02-21T23:12:10.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.341+0000 7fa99be00640 1 -- 172.21.15.28:0/2636328818 shutdown_connections 2024-02-21T23:12:10.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.341+0000 7fa99be00640 1 --2- 172.21.15.28:0/2636328818 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa97806b4d0 0x7fa97806d990 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:10.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.341+0000 7fa99be00640 1 --2- 172.21.15.28:0/2636328818 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa99c0c76f0 0x7fa99c154ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:10.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.341+0000 7fa99be00640 1 --2- 172.21.15.28:0/2636328818 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa99c13e0c0 0x7fa99c151d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:10.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.341+0000 7fa99be00640 1 --2- 172.21.15.28:0/2636328818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa99c06dbc0 0x7fa99c0b0120 unknown :-1 s=CLOSED pgs=2164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:10.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.341+0000 7fa99be00640 1 -- 172.21.15.28:0/2636328818 >> 172.21.15.28:0/2636328818 conn(0x7fa99c073110 msgr2=0x7fa99c0c9f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:10.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.341+0000 7fa99be00640 1 -- 172.21.15.28:0/2636328818 shutdown_connections 2024-02-21T23:12:10.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.341+0000 7fa99be00640 1 -- 172.21.15.28:0/2636328818 wait complete. 2024-02-21T23:12:10.347 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:12:10.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.572+0000 7f05b8c00640 1 Processor -- start 2024-02-21T23:12:10.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.572+0000 7f05b8c00640 1 -- start start 2024-02-21T23:12:10.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.573+0000 7f05b8c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f05b4150450 0x7f05b40a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:10.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.573+0000 7f05b8c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f05b406ee40 0x7f05b41536f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:10.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.574+0000 7f05b8c00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05b40c2760 con 0x7f05b4156af0 2024-02-21T23:12:10.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.574+0000 7f05b8c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05b40c22c0 con 0x7f05b4150450 2024-02-21T23:12:10.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.574+0000 7f05b8c00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05b4156660 con 0x7f05b406ee40 2024-02-21T23:12:10.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.574+0000 7f05b3400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f05b4150450 0x7f05b40a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:10.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.574+0000 7f05b3400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f05b4150450 0x7f05b40a5fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60372/0 (socket says 172.21.15.28:60372) 2024-02-21T23:12:10.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.574+0000 7f05b3400640 1 -- 172.21.15.28:0/4048818280 learned_addr learned my addr 172.21.15.28:0/4048818280 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:10.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.574+0000 7f05aaa00640 1 --2- 172.21.15.28:0/4048818280 >> v2:172.21.15.81:3300/0 conn(0x7f05b406ee40 0x7f05b41536f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:10.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.574+0000 7f05b3400640 1 -- 172.21.15.28:0/4048818280 >> v2:172.21.15.81:3300/0 conn(0x7f05b406ee40 msgr2=0x7f05b41536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:10.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.574+0000 7f05b3400640 1 --2- 172.21.15.28:0/4048818280 >> v2:172.21.15.81:3300/0 conn(0x7f05b406ee40 0x7f05b41536f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:10.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.574+0000 7f05b3400640 1 -- 172.21.15.28:0/4048818280 >> v1:172.21.15.81:6789/0 conn(0x7f05b4156af0 legacy=0x7f05b406dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:10.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.574+0000 7f05b3400640 1 -- 172.21.15.28:0/4048818280 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05b4153c30 con 0x7f05b4150450 2024-02-21T23:12:10.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.575+0000 7f05b3400640 1 --2- 172.21.15.28:0/4048818280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05b4150450 0x7f05b40a5fd0 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 crypto rx=0x7f05b406bcc0 tx=0x7f05a402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0d8d78f74ff2e83 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:10.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.576+0000 7f05b2a00640 1 -- 172.21.15.28:0/4048818280 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a4035020 con 0x7f05b4150450 2024-02-21T23:12:10.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.576+0000 7f05b2a00640 1 -- 172.21.15.28:0/4048818280 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05a4031870 con 0x7f05b4150450 2024-02-21T23:12:10.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.576+0000 7f05b2a00640 1 -- 172.21.15.28:0/4048818280 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a4031c40 con 0x7f05b4150450 2024-02-21T23:12:10.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.576+0000 7f05b8c00640 1 -- 172.21.15.28:0/4048818280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05b4150450 msgr2=0x7f05b40a5fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:10.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.576+0000 7f05b8c00640 1 --2- 172.21.15.28:0/4048818280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05b4150450 0x7f05b40a5fd0 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 crypto rx=0x7f05b406bcc0 tx=0x7f05a402b910 comp rx=0 tx=0).stop 2024-02-21T23:12:10.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.577+0000 7f05b8c00640 1 -- 172.21.15.28:0/4048818280 shutdown_connections 2024-02-21T23:12:10.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.577+0000 7f05b8c00640 1 --2- 172.21.15.28:0/4048818280 >> v2:172.21.15.81:3300/0 conn(0x7f05b406ee40 0x7f05b41536f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:10.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.577+0000 7f05b8c00640 1 --2- 172.21.15.28:0/4048818280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05b4150450 0x7f05b40a5fd0 unknown :-1 s=CLOSED pgs=2165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:10.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.577+0000 7f05b8c00640 1 -- 172.21.15.28:0/4048818280 >> 172.21.15.28:0/4048818280 conn(0x7f05b406b630 msgr2=0x7f05b40c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:10.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.577+0000 7f05b8c00640 1 -- 172.21.15.28:0/4048818280 shutdown_connections 2024-02-21T23:12:10.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.577+0000 7f05b8c00640 1 -- 172.21.15.28:0/4048818280 wait complete. 2024-02-21T23:12:10.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.578+0000 7f05b8c00640 1 Processor -- start 2024-02-21T23:12:10.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.579+0000 7f05b8c00640 1 -- start start 2024-02-21T23:12:10.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.579+0000 7f05b8c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f05b406ee40 0x7f05b41e0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:10.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.579+0000 7f05b8c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05b4150450 0x7f05b40c40f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:10.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.579+0000 7f05b8c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05b4156af0 0x7f05b40c4630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:10.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.579+0000 7f05b8c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05b40c2cb0 con 0x7f05b4156af0 2024-02-21T23:12:10.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.579+0000 7f05b8c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05b40c29d0 con 0x7f05b4150450 2024-02-21T23:12:10.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.579+0000 7f05b8c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05b406e700 con 0x7f05b406ee40 2024-02-21T23:12:10.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.580+0000 7f05b3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05b4156af0 0x7f05b40c4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:10.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.580+0000 7f05b3400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f05b406ee40 0x7f05b41e0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:10.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.580+0000 7f05b3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05b4156af0 0x7f05b40c4630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60374/0 (socket says 172.21.15.28:60374) 2024-02-21T23:12:10.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.580+0000 7f05b3e00640 1 -- 172.21.15.28:0/3654425097 learned_addr learned my addr 172.21.15.28:0/3654425097 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:10.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.580+0000 7f05aaa00640 1 --2- 172.21.15.28:0/3654425097 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05b4150450 0x7f05b40c40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:10.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.580+0000 7f05b3e00640 1 -- 172.21.15.28:0/3654425097 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f05b406ee40 msgr2=0x7f05b41e0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:10.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.580+0000 7f05b3e00640 1 --2- 172.21.15.28:0/3654425097 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f05b406ee40 0x7f05b41e0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:10.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.580+0000 7f05b3e00640 1 -- 172.21.15.28:0/3654425097 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05b4150450 msgr2=0x7f05b40c40f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:10.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.580+0000 7f05b3e00640 1 --2- 172.21.15.28:0/3654425097 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05b4150450 0x7f05b40c40f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:10.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.580+0000 7f05b3e00640 1 -- 172.21.15.28:0/3654425097 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05b40c6250 con 0x7f05b4156af0 2024-02-21T23:12:10.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.581+0000 7f05b3e00640 1 --2- 172.21.15.28:0/3654425097 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05b4156af0 0x7f05b40c4630 secure :-1 s=READY pgs=2166 cs=0 l=1 rev1=1 crypto rx=0x7f05ac00c410 tx=0x7f05ac00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:10.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.581+0000 7f05abe00640 1 -- 172.21.15.28:0/3654425097 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05ac011790 con 0x7f05b4156af0 2024-02-21T23:12:10.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.581+0000 7f05abe00640 1 -- 172.21.15.28:0/3654425097 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05ac017070 con 0x7f05b4156af0 2024-02-21T23:12:10.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.581+0000 7f05b8c00640 1 -- 172.21.15.28:0/3654425097 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05b40c4b70 con 0x7f05b4156af0 2024-02-21T23:12:10.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.581+0000 7f05abe00640 1 -- 172.21.15.28:0/3654425097 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05ac011bb0 con 0x7f05b4156af0 2024-02-21T23:12:10.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.581+0000 7f05b8c00640 1 -- 172.21.15.28:0/3654425097 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05b40c5a50 con 0x7f05b4156af0 2024-02-21T23:12:10.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.582+0000 7f05b8c00640 1 -- 172.21.15.28:0/3654425097 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05b404f290 con 0x7f05b4156af0 2024-02-21T23:12:10.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.583+0000 7f05abe00640 1 -- 172.21.15.28:0/3654425097 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f05ac01c460 con 0x7f05b4156af0 2024-02-21T23:12:10.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.584+0000 7f05abe00640 1 --2- 172.21.15.28:0/3654425097 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f059006b4d0 0x7f059006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:10.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.584+0000 7f05abe00640 1 -- 172.21.15.28:0/3654425097 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f05ac089e40 con 0x7f05b4156af0 2024-02-21T23:12:10.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.585+0000 7f05b3400640 1 --2- 172.21.15.28:0/3654425097 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f059006b4d0 0x7f059006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:10.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.586+0000 7f05b3400640 1 --2- 172.21.15.28:0/3654425097 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f059006b4d0 0x7f059006d990 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f05a4036000 tx=0x7f05a403c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:10.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.588+0000 7f05abe00640 1 -- 172.21.15.28:0/3654425097 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f05ac021020 con 0x7f05b4156af0 2024-02-21T23:12:10.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.837+0000 7f05b8c00640 1 -- 172.21.15.28:0/3654425097 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f05b414a110 con 0x7f05b4156af0 2024-02-21T23:12:10.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.838+0000 7f05abe00640 1 -- 172.21.15.28:0/3654425097 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 209 v209) v1 ==== 95+0+4959 (secure 0 0 0) 0x7f05ac0584d0 con 0x7f05b4156af0 2024-02-21T23:12:10.826 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:12:10.826 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":209,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6848,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3439896042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3439896042},{"type":"v1","addr":"172.21.15.28:6835","nonce":3439896042}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":196},{"gid":7729,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/3213249808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3213249808},{"type":"v1","addr":"172.21.15.28:6833","nonce":3213249808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":195}],"filesystems":[{"mdsmap":{"epoch":209,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:11:38.384619+0000","modified":"2024-02-21T23:12:03.626987+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7867,"mds_1":7735},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7735":{"gid":7735,"name":"d","rank":1,"incarnation":208,"state":"up:active","state_seq":9,"addr":"172.21.15.81:6836/2895690310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2895690310},{"type":"v1","addr":"172.21.15.81:6836","nonce":2895690310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7867":{"gid":7867,"name":"b","rank":0,"incarnation":206,"state":"up:active","state_seq":2,"addr":"172.21.15.81:6837/416021079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":416021079},{"type":"v1","addr":"172.21.15.81:6837","nonce":416021079}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-02-21T23:12:10.826 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 209 2024-02-21T23:12:10.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.840+0000 7f05b8c00640 1 -- 172.21.15.28:0/3654425097 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f059006b4d0 msgr2=0x7f059006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:10.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.840+0000 7f05b8c00640 1 --2- 172.21.15.28:0/3654425097 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f059006b4d0 0x7f059006d990 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f05a4036000 tx=0x7f05a403c040 comp rx=0 tx=0).stop 2024-02-21T23:12:10.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.840+0000 7f05b8c00640 1 -- 172.21.15.28:0/3654425097 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05b4156af0 msgr2=0x7f05b40c4630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:10.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.840+0000 7f05b8c00640 1 --2- 172.21.15.28:0/3654425097 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05b4156af0 0x7f05b40c4630 secure :-1 s=READY pgs=2166 cs=0 l=1 rev1=1 crypto rx=0x7f05ac00c410 tx=0x7f05ac00c8d0 comp rx=0 tx=0).stop 2024-02-21T23:12:10.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.841+0000 7f05b8c00640 1 -- 172.21.15.28:0/3654425097 shutdown_connections 2024-02-21T23:12:10.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.841+0000 7f05b8c00640 1 --2- 172.21.15.28:0/3654425097 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f059006b4d0 0x7f059006d990 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:10.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.841+0000 7f05b8c00640 1 --2- 172.21.15.28:0/3654425097 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05b4156af0 0x7f05b40c4630 unknown :-1 s=CLOSED pgs=2166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:10.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.841+0000 7f05b8c00640 1 --2- 172.21.15.28:0/3654425097 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f05b4150450 0x7f05b40c40f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:10.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.841+0000 7f05b8c00640 1 --2- 172.21.15.28:0/3654425097 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f05b406ee40 0x7f05b41e0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:10.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.841+0000 7f05b8c00640 1 -- 172.21.15.28:0/3654425097 >> 172.21.15.28:0/3654425097 conn(0x7f05b406b630 msgr2=0x7f05b40c64e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:10.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.841+0000 7f05b8c00640 1 -- 172.21.15.28:0/3654425097 shutdown_connections 2024-02-21T23:12:10.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:10.841+0000 7f05b8c00640 1 -- 172.21.15.28:0/3654425097 wait complete. 2024-02-21T23:12:10.846 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:12:11.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.072+0000 7f325b200640 1 Processor -- start 2024-02-21T23:12:11.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.072+0000 7f325b200640 1 -- start start 2024-02-21T23:12:11.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.073+0000 7f325b200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f325c06dff0 0x7f325c06e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:11.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.073+0000 7f325b200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f325c0c0fe0 0x7f325c071350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:11.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.073+0000 7f325b200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f325c0a8b40 con 0x7f325c158580 2024-02-21T23:12:11.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.073+0000 7f325b200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f325c0a8e00 con 0x7f325c0c0fe0 2024-02-21T23:12:11.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.074+0000 7f325b200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f325c0a90e0 con 0x7f325c06dff0 2024-02-21T23:12:11.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.074+0000 7f3253e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f325c0c0fe0 0x7f325c071350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:11.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.074+0000 7f3253e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f325c0c0fe0 0x7f325c071350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60382/0 (socket says 172.21.15.28:60382) 2024-02-21T23:12:11.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.074+0000 7f3253e00640 1 -- 172.21.15.28:0/1930570628 learned_addr learned my addr 172.21.15.28:0/1930570628 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:11.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.074+0000 7f3259e00640 1 --2- 172.21.15.28:0/1930570628 >> v2:172.21.15.81:3300/0 conn(0x7f325c06dff0 0x7f325c06e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:11.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.074+0000 7f3253e00640 1 -- 172.21.15.28:0/1930570628 >> v2:172.21.15.81:3300/0 conn(0x7f325c06dff0 msgr2=0x7f325c06e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:11.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.075+0000 7f3253e00640 1 --2- 172.21.15.28:0/1930570628 >> v2:172.21.15.81:3300/0 conn(0x7f325c06dff0 0x7f325c06e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:11.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.075+0000 7f3253e00640 1 -- 172.21.15.28:0/1930570628 >> v1:172.21.15.81:6789/0 conn(0x7f325c158580 legacy=0x7f325c14b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:11.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.075+0000 7f3253e00640 1 -- 172.21.15.28:0/1930570628 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f325c072ed0 con 0x7f325c0c0fe0 2024-02-21T23:12:11.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.076+0000 7f3253e00640 1 --2- 172.21.15.28:0/1930570628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f325c0c0fe0 0x7f325c071350 secure :-1 s=READY pgs=2167 cs=0 l=1 rev1=1 crypto rx=0x7f325c06be50 tx=0x7f324402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0e6cb39dcb8817a server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:11.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.076+0000 7f3259400640 1 -- 172.21.15.28:0/1930570628 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3244034070 con 0x7f325c0c0fe0 2024-02-21T23:12:11.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.076+0000 7f3259400640 1 -- 172.21.15.28:0/1930570628 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3244030860 con 0x7f325c0c0fe0 2024-02-21T23:12:11.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.076+0000 7f3259400640 1 -- 172.21.15.28:0/1930570628 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3244030c30 con 0x7f325c0c0fe0 2024-02-21T23:12:11.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.077+0000 7f325b200640 1 -- 172.21.15.28:0/1930570628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f325c0c0fe0 msgr2=0x7f325c071350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:11.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.077+0000 7f325b200640 1 --2- 172.21.15.28:0/1930570628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f325c0c0fe0 0x7f325c071350 secure :-1 s=READY pgs=2167 cs=0 l=1 rev1=1 crypto rx=0x7f325c06be50 tx=0x7f324402b9a0 comp rx=0 tx=0).stop 2024-02-21T23:12:11.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.077+0000 7f325b200640 1 -- 172.21.15.28:0/1930570628 shutdown_connections 2024-02-21T23:12:11.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.077+0000 7f325b200640 1 --2- 172.21.15.28:0/1930570628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f325c0c0fe0 0x7f325c071350 unknown :-1 s=CLOSED pgs=2167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:11.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.077+0000 7f325b200640 1 --2- 172.21.15.28:0/1930570628 >> v2:172.21.15.81:3300/0 conn(0x7f325c06dff0 0x7f325c06e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:11.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.077+0000 7f325b200640 1 -- 172.21.15.28:0/1930570628 >> 172.21.15.28:0/1930570628 conn(0x7f325c06b630 msgr2=0x7f325c0c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:11.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.077+0000 7f325b200640 1 -- 172.21.15.28:0/1930570628 shutdown_connections 2024-02-21T23:12:11.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.077+0000 7f325b200640 1 -- 172.21.15.28:0/1930570628 wait complete. 2024-02-21T23:12:11.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.079+0000 7f325b200640 1 Processor -- start 2024-02-21T23:12:11.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.079+0000 7f325b200640 1 -- start start 2024-02-21T23:12:11.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.080+0000 7f325b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f325c06dff0 0x7f325c0a9610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:11.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.080+0000 7f325b200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f325c0c0fe0 0x7f325c1ee180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:11.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.080+0000 7f325b200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f325c158580 0x7f325c147b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:11.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.080+0000 7f325b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f325c071890 con 0x7f325c06dff0 2024-02-21T23:12:11.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.080+0000 7f325b200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f325c073520 con 0x7f325c0c0fe0 2024-02-21T23:12:11.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.080+0000 7f325b200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f325c158980 con 0x7f325c158580 2024-02-21T23:12:11.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.080+0000 7f3259e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f325c06dff0 0x7f325c0a9610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:11.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.081+0000 7f3259e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f325c06dff0 0x7f325c0a9610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60394/0 (socket says 172.21.15.28:60394) 2024-02-21T23:12:11.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.081+0000 7f3259e00640 1 -- 172.21.15.28:0/2412465246 learned_addr learned my addr 172.21.15.28:0/2412465246 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:11.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.081+0000 7f325a800640 1 --2- 172.21.15.28:0/2412465246 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f325c158580 0x7f325c147b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:11.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.081+0000 7f3253e00640 1 --2- 172.21.15.28:0/2412465246 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f325c0c0fe0 0x7f325c1ee180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:11.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.081+0000 7f3259e00640 1 -- 172.21.15.28:0/2412465246 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f325c158580 msgr2=0x7f325c147b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:11.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.081+0000 7f3259e00640 1 --2- 172.21.15.28:0/2412465246 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f325c158580 0x7f325c147b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:11.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.081+0000 7f3259e00640 1 -- 172.21.15.28:0/2412465246 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f325c0c0fe0 msgr2=0x7f325c1ee180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:11.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.081+0000 7f3259e00640 1 --2- 172.21.15.28:0/2412465246 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f325c0c0fe0 0x7f325c1ee180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:11.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.081+0000 7f3259e00640 1 -- 172.21.15.28:0/2412465246 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f325c14abf0 con 0x7f325c06dff0 2024-02-21T23:12:11.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.081+0000 7f3259e00640 1 --2- 172.21.15.28:0/2412465246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f325c06dff0 0x7f325c0a9610 secure :-1 s=READY pgs=2168 cs=0 l=1 rev1=1 crypto rx=0x7f324c00e8a0 tx=0x7f324c00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:11.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.081+0000 7f3252000640 1 -- 172.21.15.28:0/2412465246 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f324c017070 con 0x7f325c06dff0 2024-02-21T23:12:11.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.081+0000 7f3252000640 1 -- 172.21.15.28:0/2412465246 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f324c00c940 con 0x7f325c06dff0 2024-02-21T23:12:11.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.081+0000 7f3252000640 1 -- 172.21.15.28:0/2412465246 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f324c00cd30 con 0x7f325c06dff0 2024-02-21T23:12:11.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.082+0000 7f325b200640 1 -- 172.21.15.28:0/2412465246 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f325c0ab3d0 con 0x7f325c06dff0 2024-02-21T23:12:11.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.082+0000 7f325b200640 1 -- 172.21.15.28:0/2412465246 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f325c0a9b50 con 0x7f325c06dff0 2024-02-21T23:12:11.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.083+0000 7f325b200640 1 -- 172.21.15.28:0/2412465246 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f325c0c70c0 con 0x7f325c06dff0 2024-02-21T23:12:11.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.084+0000 7f3252000640 1 -- 172.21.15.28:0/2412465246 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f324c01a070 con 0x7f325c06dff0 2024-02-21T23:12:11.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.084+0000 7f3252000640 1 --2- 172.21.15.28:0/2412465246 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f323806b4d0 0x7f323806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:11.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.085+0000 7f3252000640 1 -- 172.21.15.28:0/2412465246 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f324c08a230 con 0x7f325c06dff0 2024-02-21T23:12:11.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.085+0000 7f3253e00640 1 --2- 172.21.15.28:0/2412465246 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f323806b4d0 0x7f323806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:11.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.086+0000 7f3253e00640 1 --2- 172.21.15.28:0/2412465246 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f323806b4d0 0x7f323806d990 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f3244036000 tx=0x7f324403b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:11.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.089+0000 7f3252000640 1 -- 172.21.15.28:0/2412465246 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f325c0c70c0 con 0x7f325c06dff0 2024-02-21T23:12:11.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.291+0000 7f325b200640 1 -- 172.21.15.28:0/2412465246 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f325c04f290 con 0x7f325c06dff0 2024-02-21T23:12:11.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.295+0000 7f3252000640 1 -- 172.21.15.28:0/2412465246 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v186) v1 ==== 74+0+16824 (secure 0 0 0) 0x7f324c058f10 con 0x7f325c06dff0 2024-02-21T23:12:11.283 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:12:11.283 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":186,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:11:39.917106+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:11:36.316421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"185","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_data","create_time":"2024-02-21T23:11:37.418709+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"186","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:12:11.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.297+0000 7f325b200640 1 -- 172.21.15.28:0/2412465246 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f323806b4d0 msgr2=0x7f323806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:11.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.297+0000 7f325b200640 1 --2- 172.21.15.28:0/2412465246 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f323806b4d0 0x7f323806d990 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f3244036000 tx=0x7f324403b040 comp rx=0 tx=0).stop 2024-02-21T23:12:11.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.297+0000 7f325b200640 1 -- 172.21.15.28:0/2412465246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f325c06dff0 msgr2=0x7f325c0a9610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:11.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.297+0000 7f325b200640 1 --2- 172.21.15.28:0/2412465246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f325c06dff0 0x7f325c0a9610 secure :-1 s=READY pgs=2168 cs=0 l=1 rev1=1 crypto rx=0x7f324c00e8a0 tx=0x7f324c00ed60 comp rx=0 tx=0).stop 2024-02-21T23:12:11.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.297+0000 7f325b200640 1 -- 172.21.15.28:0/2412465246 shutdown_connections 2024-02-21T23:12:11.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.298+0000 7f325b200640 1 --2- 172.21.15.28:0/2412465246 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f323806b4d0 0x7f323806d990 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:11.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.298+0000 7f325b200640 1 --2- 172.21.15.28:0/2412465246 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f325c158580 0x7f325c147b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:11.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.298+0000 7f325b200640 1 --2- 172.21.15.28:0/2412465246 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f325c0c0fe0 0x7f325c1ee180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:11.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.298+0000 7f325b200640 1 --2- 172.21.15.28:0/2412465246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f325c06dff0 0x7f325c0a9610 unknown :-1 s=CLOSED pgs=2168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:11.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.298+0000 7f325b200640 1 -- 172.21.15.28:0/2412465246 >> 172.21.15.28:0/2412465246 conn(0x7f325c06b630 msgr2=0x7f325c1516c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:11.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.298+0000 7f325b200640 1 -- 172.21.15.28:0/2412465246 shutdown_connections 2024-02-21T23:12:11.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.298+0000 7f325b200640 1 -- 172.21.15.28:0/2412465246 wait complete. 2024-02-21T23:12:11.305 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-02-21T23:12:11.306 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:12:11.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.544+0000 7f0752000640 1 Processor -- start 2024-02-21T23:12:11.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.544+0000 7f0752000640 1 -- start start 2024-02-21T23:12:11.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.546+0000 7f0752000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f074c150450 0x7f074c0a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:11.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.546+0000 7f0752000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f074c06ee40 0x7f074c1536f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:11.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.546+0000 7f0752000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f074c0c2760 con 0x7f074c156af0 2024-02-21T23:12:11.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.546+0000 7f0752000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f074c0c22c0 con 0x7f074c06ee40 2024-02-21T23:12:11.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.546+0000 7f0752000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f074c156660 con 0x7f074c150450 2024-02-21T23:12:11.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.546+0000 7f0750c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f074c150450 0x7f074c0a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:11.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.546+0000 7f074be00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f074c06ee40 0x7f074c1536f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:11.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.546+0000 7f0750c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f074c150450 0x7f074c0a5fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3301/0 says I am v2:172.21.15.28:32926/0 (socket says 172.21.15.28:32926) 2024-02-21T23:12:11.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.546+0000 7f0750c00640 1 -- 172.21.15.28:0/2874900221 learned_addr learned my addr 172.21.15.28:0/2874900221 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:11.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.547+0000 7f0750c00640 1 -- 172.21.15.28:0/2874900221 >> v2:172.21.15.81:3300/0 conn(0x7f074c06ee40 msgr2=0x7f074c1536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:11.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.547+0000 7f0750c00640 1 --2- 172.21.15.28:0/2874900221 >> v2:172.21.15.81:3300/0 conn(0x7f074c06ee40 0x7f074c1536f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:11.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.547+0000 7f0750c00640 1 -- 172.21.15.28:0/2874900221 >> v1:172.21.15.81:6789/0 conn(0x7f074c156af0 legacy=0x7f074c06dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:11.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.547+0000 7f0750c00640 1 -- 172.21.15.28:0/2874900221 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f074c153c30 con 0x7f074c150450 2024-02-21T23:12:11.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.549+0000 7f0750c00640 1 --2- 172.21.15.28:0/2874900221 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f074c150450 0x7f074c0a5fd0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0740009870 tx=0x7f074002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6fc424532d4ab3e1 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:11.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.549+0000 7f074b400640 1 -- 172.21.15.28:0/2874900221 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0740034070 con 0x7f074c150450 2024-02-21T23:12:11.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.549+0000 7f074b400640 1 -- 172.21.15.28:0/2874900221 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0740030840 con 0x7f074c150450 2024-02-21T23:12:11.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.549+0000 7f074b400640 1 -- 172.21.15.28:0/2874900221 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0740030c10 con 0x7f074c150450 2024-02-21T23:12:11.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.550+0000 7f0752000640 1 -- 172.21.15.28:0/2874900221 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f074c150450 msgr2=0x7f074c0a5fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:11.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.550+0000 7f0752000640 1 --2- 172.21.15.28:0/2874900221 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f074c150450 0x7f074c0a5fd0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0740009870 tx=0x7f074002b980 comp rx=0 tx=0).stop 2024-02-21T23:12:11.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.550+0000 7f0752000640 1 -- 172.21.15.28:0/2874900221 shutdown_connections 2024-02-21T23:12:11.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.550+0000 7f0752000640 1 --2- 172.21.15.28:0/2874900221 >> v2:172.21.15.81:3300/0 conn(0x7f074c06ee40 0x7f074c1536f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:11.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.550+0000 7f0752000640 1 --2- 172.21.15.28:0/2874900221 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f074c150450 0x7f074c0a5fd0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:11.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.550+0000 7f0752000640 1 -- 172.21.15.28:0/2874900221 >> 172.21.15.28:0/2874900221 conn(0x7f074c06b630 msgr2=0x7f074c0c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:11.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.550+0000 7f0752000640 1 -- 172.21.15.28:0/2874900221 shutdown_connections 2024-02-21T23:12:11.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.550+0000 7f0752000640 1 -- 172.21.15.28:0/2874900221 wait complete. 2024-02-21T23:12:11.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.552+0000 7f0752000640 1 Processor -- start 2024-02-21T23:12:11.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.552+0000 7f0752000640 1 -- start start 2024-02-21T23:12:11.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.553+0000 7f0752000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f074c06ee40 0x7f074c149730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:11.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.554+0000 7f0752000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f074c150450 0x7f074c0bfd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:11.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.554+0000 7f0752000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f074c156af0 0x7f074c0aaf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:11.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.554+0000 7f0752000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f074c0c2cb0 con 0x7f074c150450 2024-02-21T23:12:11.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.554+0000 7f0752000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f074c0c29d0 con 0x7f074c06ee40 2024-02-21T23:12:11.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.554+0000 7f0752000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f074c06e700 con 0x7f074c156af0 2024-02-21T23:12:11.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.554+0000 7f074be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f074c150450 0x7f074c0bfd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:11.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.554+0000 7f074be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f074c150450 0x7f074c0bfd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60396/0 (socket says 172.21.15.28:60396) 2024-02-21T23:12:11.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.554+0000 7f074be00640 1 -- 172.21.15.28:0/2403170440 learned_addr learned my addr 172.21.15.28:0/2403170440 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:11.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.554+0000 7f0751600640 1 --2- 172.21.15.28:0/2403170440 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f074c156af0 0x7f074c0aaf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:11.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.554+0000 7f0750c00640 1 --2- 172.21.15.28:0/2403170440 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f074c06ee40 0x7f074c149730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:11.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.554+0000 7f074be00640 1 -- 172.21.15.28:0/2403170440 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f074c156af0 msgr2=0x7f074c0aaf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:11.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.554+0000 7f074be00640 1 --2- 172.21.15.28:0/2403170440 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f074c156af0 0x7f074c0aaf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:11.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.554+0000 7f074be00640 1 -- 172.21.15.28:0/2403170440 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f074c06ee40 msgr2=0x7f074c149730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:11.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.554+0000 7f074be00640 1 --2- 172.21.15.28:0/2403170440 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f074c06ee40 0x7f074c149730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:11.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.554+0000 7f074be00640 1 -- 172.21.15.28:0/2403170440 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f074c149c70 con 0x7f074c150450 2024-02-21T23:12:11.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.555+0000 7f074be00640 1 --2- 172.21.15.28:0/2403170440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f074c150450 0x7f074c0bfd80 secure :-1 s=READY pgs=2169 cs=0 l=1 rev1=1 crypto rx=0x7f073800e9b0 tx=0x7f073800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:11.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.555+0000 7f0748c00640 1 -- 172.21.15.28:0/2403170440 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0738017070 con 0x7f074c150450 2024-02-21T23:12:11.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.555+0000 7f0748c00640 1 -- 172.21.15.28:0/2403170440 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f073800c9a0 con 0x7f074c150450 2024-02-21T23:12:11.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.555+0000 7f0748c00640 1 -- 172.21.15.28:0/2403170440 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f073800cd90 con 0x7f074c150450 2024-02-21T23:12:11.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.555+0000 7f0752000640 1 -- 172.21.15.28:0/2403170440 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f074c0c6160 con 0x7f074c150450 2024-02-21T23:12:11.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.555+0000 7f0752000640 1 -- 172.21.15.28:0/2403170440 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f074c0c6720 con 0x7f074c150450 2024-02-21T23:12:11.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.557+0000 7f0752000640 1 -- 172.21.15.28:0/2403170440 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f074c04f290 con 0x7f074c150450 2024-02-21T23:12:11.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.557+0000 7f0748c00640 1 -- 172.21.15.28:0/2403170440 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f073801a070 con 0x7f074c150450 2024-02-21T23:12:11.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.558+0000 7f0748c00640 1 --2- 172.21.15.28:0/2403170440 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f072c06b4d0 0x7f072c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:11.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.559+0000 7f0748c00640 1 -- 172.21.15.28:0/2403170440 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f073808a380 con 0x7f074c150450 2024-02-21T23:12:11.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.559+0000 7f0750c00640 1 --2- 172.21.15.28:0/2403170440 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f072c06b4d0 0x7f072c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:11.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.560+0000 7f0750c00640 1 --2- 172.21.15.28:0/2403170440 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f072c06b4d0 0x7f072c06d990 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f0740036000 tx=0x7f074003b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:11.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.563+0000 7f0748c00640 1 -- 172.21.15.28:0/2403170440 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f0738059070 con 0x7f074c150450 2024-02-21T23:12:11.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.812+0000 7f0752000640 1 -- 172.21.15.28:0/2403170440 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f074c0ac890 con 0x7f074c150450 2024-02-21T23:12:11.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.813+0000 7f0748c00640 1 -- 172.21.15.28:0/2403170440 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 209 v209) v1 ==== 95+0+4959 (secure 0 0 0) 0x7f0738058970 con 0x7f074c150450 2024-02-21T23:12:11.801 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:12:11.801 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":209,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6848,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3439896042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3439896042},{"type":"v1","addr":"172.21.15.28:6835","nonce":3439896042}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":196},{"gid":7729,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/3213249808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3213249808},{"type":"v1","addr":"172.21.15.28:6833","nonce":3213249808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":195}],"filesystems":[{"mdsmap":{"epoch":209,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:11:38.384619+0000","modified":"2024-02-21T23:12:03.626987+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7867,"mds_1":7735},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7735":{"gid":7735,"name":"d","rank":1,"incarnation":208,"state":"up:active","state_seq":9,"addr":"172.21.15.81:6836/2895690310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2895690310},{"type":"v1","addr":"172.21.15.81:6836","nonce":2895690310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7867":{"gid":7867,"name":"b","rank":0,"incarnation":206,"state":"up:active","state_seq":2,"addr":"172.21.15.81:6837/416021079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":416021079},{"type":"v1","addr":"172.21.15.81:6837","nonce":416021079}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-02-21T23:12:11.801 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 209 2024-02-21T23:12:11.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.816+0000 7f0752000640 1 -- 172.21.15.28:0/2403170440 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f072c06b4d0 msgr2=0x7f072c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:11.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.816+0000 7f0752000640 1 --2- 172.21.15.28:0/2403170440 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f072c06b4d0 0x7f072c06d990 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f0740036000 tx=0x7f074003b040 comp rx=0 tx=0).stop 2024-02-21T23:12:11.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.816+0000 7f0752000640 1 -- 172.21.15.28:0/2403170440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f074c150450 msgr2=0x7f074c0bfd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:11.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.816+0000 7f0752000640 1 --2- 172.21.15.28:0/2403170440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f074c150450 0x7f074c0bfd80 secure :-1 s=READY pgs=2169 cs=0 l=1 rev1=1 crypto rx=0x7f073800e9b0 tx=0x7f073800ee70 comp rx=0 tx=0).stop 2024-02-21T23:12:11.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.816+0000 7f0752000640 1 -- 172.21.15.28:0/2403170440 shutdown_connections 2024-02-21T23:12:11.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.816+0000 7f0752000640 1 --2- 172.21.15.28:0/2403170440 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f072c06b4d0 0x7f072c06d990 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:11.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.816+0000 7f0752000640 1 --2- 172.21.15.28:0/2403170440 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f074c156af0 0x7f074c0aaf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:11.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.816+0000 7f0752000640 1 --2- 172.21.15.28:0/2403170440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f074c150450 0x7f074c0bfd80 unknown :-1 s=CLOSED pgs=2169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:11.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.816+0000 7f0752000640 1 --2- 172.21.15.28:0/2403170440 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f074c06ee40 0x7f074c149730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:11.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.816+0000 7f0752000640 1 -- 172.21.15.28:0/2403170440 >> 172.21.15.28:0/2403170440 conn(0x7f074c06b630 msgr2=0x7f074c14ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:11.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.816+0000 7f0752000640 1 -- 172.21.15.28:0/2403170440 shutdown_connections 2024-02-21T23:12:11.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:11.816+0000 7f0752000640 1 -- 172.21.15.28:0/2403170440 wait complete. 2024-02-21T23:12:11.824 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:12:12.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.048+0000 7ffbc3e00640 1 Processor -- start 2024-02-21T23:12:12.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.049+0000 7ffbc3e00640 1 -- start start 2024-02-21T23:12:12.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.050+0000 7ffbc3e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7ffbc40aff90 0x7ffbc40c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:12.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.050+0000 7ffbc3e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbc40c74c0 con 0x7ffbc406dbf0 2024-02-21T23:12:12.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.050+0000 7ffbc3e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbc40c77a0 con 0x7ffbc40c5720 2024-02-21T23:12:12.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.050+0000 7ffbc3e00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbc40c7270 con 0x7ffbc40aff90 2024-02-21T23:12:12.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.050+0000 7ffbc2000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ffbc406dbf0 0x7ffbc4158ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53802/0 (socket says 172.21.15.28:53802) 2024-02-21T23:12:12.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.050+0000 7ffbc2000640 1 -- 172.21.15.28:0/1738845014 learned_addr learned my addr 172.21.15.28:0/1738845014 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:12:12.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.051+0000 7ffbc2a00640 1 --2- 172.21.15.28:0/1738845014 >> v2:172.21.15.81:3300/0 conn(0x7ffbc40aff90 0x7ffbc40c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:12.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.051+0000 7ffbc1600640 1 -- 172.21.15.28:0/1738845014 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3968694249 0 0) 0x7ffbc40c74c0 con 0x7ffbc406dbf0 2024-02-21T23:12:12.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.051+0000 7ffbc1600640 1 -- 172.21.15.28:0/1738845014 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffba8003520 con 0x7ffbc406dbf0 2024-02-21T23:12:12.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.051+0000 7ffbc1600640 1 -- 172.21.15.28:0/1738845014 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3477299390 0 0) 0x7ffba8003520 con 0x7ffbc406dbf0 2024-02-21T23:12:12.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.051+0000 7ffbc1600640 1 -- 172.21.15.28:0/1738845014 >> v2:172.21.15.81:3300/0 conn(0x7ffbc40aff90 msgr2=0x7ffbc40c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:12.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.051+0000 7ffbc1600640 1 --2- 172.21.15.28:0/1738845014 >> v2:172.21.15.81:3300/0 conn(0x7ffbc40aff90 0x7ffbc40c51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:12.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.051+0000 7ffbc1600640 1 -- 172.21.15.28:0/1738845014 >> v1:172.21.15.81:6789/0 conn(0x7ffbc40c5720 legacy=0x7ffbc415f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:12.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.051+0000 7ffbc1600640 1 -- 172.21.15.28:0/1738845014 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbc40c7a80 con 0x7ffbc406dbf0 2024-02-21T23:12:12.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.051+0000 7ffbc1600640 1 -- 172.21.15.28:0/1738845014 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7ffbb0001670 con 0x7ffbc406dbf0 2024-02-21T23:12:12.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.052+0000 7ffbc1600640 1 -- 172.21.15.28:0/1738845014 >> v1:172.21.15.28:6789/0 conn(0x7ffbc406dbf0 legacy=0x7ffbc4158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:12.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.052+0000 7ffbc1600640 1 --2- 172.21.15.28:0/1738845014 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffba8003a00 0x7ffba8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:12.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.052+0000 7ffbc1600640 1 --2- 172.21.15.28:0/1738845014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffba8004800 0x7ffba8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:12.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.052+0000 7ffbc1600640 1 --2- 172.21.15.28:0/1738845014 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffba80051c0 0x7ffba800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:12.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.052+0000 7ffbc1600640 1 -- 172.21.15.28:0/1738845014 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc40c7270 con 0x7ffba8004800 2024-02-21T23:12:12.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.052+0000 7ffbc1600640 1 -- 172.21.15.28:0/1738845014 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffba800c0c0 con 0x7ffba8003a00 2024-02-21T23:12:12.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.052+0000 7ffbc1600640 1 -- 172.21.15.28:0/1738845014 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffba800c2c0 con 0x7ffba80051c0 2024-02-21T23:12:12.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.052+0000 7ffbc2000640 1 --2- 172.21.15.28:0/1738845014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffba8004800 0x7ffba8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:12.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.052+0000 7ffbc2a00640 1 --2- 172.21.15.28:0/1738845014 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffba8003a00 0x7ffba8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:12.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.052+0000 7ffbc2000640 1 -- 172.21.15.28:0/1738845014 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffba80051c0 msgr2=0x7ffba800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:12.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.052+0000 7ffbc2000640 1 --2- 172.21.15.28:0/1738845014 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffba80051c0 0x7ffba800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:12.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.052+0000 7ffbc2000640 1 -- 172.21.15.28:0/1738845014 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffba8003a00 msgr2=0x7ffba8003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:12.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.052+0000 7ffbc2000640 1 --2- 172.21.15.28:0/1738845014 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffba8003a00 0x7ffba8003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:12.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.052+0000 7ffbc2000640 1 -- 172.21.15.28:0/1738845014 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffba8003520 con 0x7ffba8004800 2024-02-21T23:12:12.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.052+0000 7ffbc2a00640 1 --2- 172.21.15.28:0/1738845014 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffba8003a00 0x7ffba8003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:12:12.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.052+0000 7ffbc3400640 1 -- 172.21.15.28:0/1738845014 reap_dead start 2024-02-21T23:12:12.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.053+0000 7ffbc2000640 1 --2- 172.21.15.28:0/1738845014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffba8004800 0x7ffba8004c80 secure :-1 s=READY pgs=2171 cs=0 l=1 rev1=1 crypto rx=0x7ffbb000c160 tx=0x7ffba8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fb26f9bf8b27fda server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:12.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.054+0000 7ffbc1600640 1 -- 172.21.15.28:0/1738845014 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb0007c60 con 0x7ffba8004800 2024-02-21T23:12:12.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.054+0000 7ffbc1600640 1 -- 172.21.15.28:0/1738845014 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbb0030070 con 0x7ffba8004800 2024-02-21T23:12:12.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.054+0000 7ffbc1600640 1 -- 172.21.15.28:0/1738845014 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb0005d90 con 0x7ffba8004800 2024-02-21T23:12:12.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.054+0000 7ffbc3e00640 1 -- 172.21.15.28:0/1738845014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffba8004800 msgr2=0x7ffba8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:12.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.054+0000 7ffbc3e00640 1 --2- 172.21.15.28:0/1738845014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffba8004800 0x7ffba8004c80 secure :-1 s=READY pgs=2171 cs=0 l=1 rev1=1 crypto rx=0x7ffbb000c160 tx=0x7ffba8005830 comp rx=0 tx=0).stop 2024-02-21T23:12:12.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.054+0000 7ffbc3e00640 1 -- 172.21.15.28:0/1738845014 shutdown_connections 2024-02-21T23:12:12.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.054+0000 7ffbc3e00640 1 --2- 172.21.15.28:0/1738845014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffba8004800 0x7ffba8004c80 unknown :-1 s=CLOSED pgs=2171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:12.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.054+0000 7ffbc3e00640 1 -- 172.21.15.28:0/1738845014 >> 172.21.15.28:0/1738845014 conn(0x7ffbc40ac370 msgr2=0x7ffbc40ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:12.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.055+0000 7ffbc3e00640 1 -- 172.21.15.28:0/1738845014 shutdown_connections 2024-02-21T23:12:12.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.055+0000 7ffbc3e00640 1 -- 172.21.15.28:0/1738845014 wait complete. 2024-02-21T23:12:12.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.056+0000 7ffbc3e00640 1 Processor -- start 2024-02-21T23:12:12.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.056+0000 7ffbc3e00640 1 -- start start 2024-02-21T23:12:12.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.057+0000 7ffbc3e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffba8004800 0x7ffbc4154710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:12.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.057+0000 7ffbc3e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffbc41522b0 0x7ffbc41437e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:12.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.057+0000 7ffbc3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbc4151d60 0x7ffbc4143d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:12.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.057+0000 7ffbc3e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc40c8760 con 0x7ffbc4151d60 2024-02-21T23:12:12.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.057+0000 7ffbc3e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc4158630 con 0x7ffbc41522b0 2024-02-21T23:12:12.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.057+0000 7ffbc3e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc40c88e0 con 0x7ffba8004800 2024-02-21T23:12:12.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.058+0000 7ffbc3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbc4151d60 0x7ffbc4143d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:12.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.058+0000 7ffbc3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbc4151d60 0x7ffbc4143d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60416/0 (socket says 172.21.15.28:60416) 2024-02-21T23:12:12.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.058+0000 7ffbc3400640 1 -- 172.21.15.28:0/1538923788 learned_addr learned my addr 172.21.15.28:0/1538923788 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:12.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.058+0000 7ffbc2000640 1 --2- 172.21.15.28:0/1538923788 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffbc41522b0 0x7ffbc41437e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:12.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.058+0000 7ffbc2a00640 1 --2- 172.21.15.28:0/1538923788 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffba8004800 0x7ffbc4154710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:12.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.058+0000 7ffbc3400640 1 -- 172.21.15.28:0/1538923788 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffba8004800 msgr2=0x7ffbc4154710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:12.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.058+0000 7ffbc3400640 1 --2- 172.21.15.28:0/1538923788 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffba8004800 0x7ffbc4154710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:12.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.058+0000 7ffbc3400640 1 -- 172.21.15.28:0/1538923788 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffbc41522b0 msgr2=0x7ffbc41437e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:12.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.058+0000 7ffbc3400640 1 --2- 172.21.15.28:0/1538923788 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffbc41522b0 0x7ffbc41437e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:12.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.058+0000 7ffbc3400640 1 -- 172.21.15.28:0/1538923788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbc40c3ae0 con 0x7ffbc4151d60 2024-02-21T23:12:12.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.059+0000 7ffbc3400640 1 --2- 172.21.15.28:0/1538923788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbc4151d60 0x7ffbc4143d20 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 crypto rx=0x7ffbc40c83e0 tx=0x7ffbbc00cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:12.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.059+0000 7ffbbaa00640 1 -- 172.21.15.28:0/1538923788 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbbc017070 con 0x7ffbc4151d60 2024-02-21T23:12:12.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.059+0000 7ffbbaa00640 1 -- 172.21.15.28:0/1538923788 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbbc005d60 con 0x7ffbc4151d60 2024-02-21T23:12:12.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.059+0000 7ffbbaa00640 1 -- 172.21.15.28:0/1538923788 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbbc00a820 con 0x7ffbc4151d60 2024-02-21T23:12:12.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.059+0000 7ffbc3e00640 1 -- 172.21.15.28:0/1538923788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbc41eee30 con 0x7ffbc4151d60 2024-02-21T23:12:12.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.059+0000 7ffbc3e00640 1 -- 172.21.15.28:0/1538923788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbc40c4060 con 0x7ffbc4151d60 2024-02-21T23:12:12.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.060+0000 7ffbc3e00640 1 -- 172.21.15.28:0/1538923788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbc404f290 con 0x7ffbc4151d60 2024-02-21T23:12:12.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.061+0000 7ffbbaa00640 1 -- 172.21.15.28:0/1538923788 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffbbc01a070 con 0x7ffbc4151d60 2024-02-21T23:12:12.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.062+0000 7ffbbaa00640 1 --2- 172.21.15.28:0/1538923788 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ffba406b500 0x7ffba406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:12.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.062+0000 7ffbbaa00640 1 -- 172.21.15.28:0/1538923788 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ffbbc089410 con 0x7ffbc4151d60 2024-02-21T23:12:12.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.063+0000 7ffbc2a00640 1 --2- 172.21.15.28:0/1538923788 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ffba406b500 0x7ffba406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:12.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.063+0000 7ffbc2a00640 1 --2- 172.21.15.28:0/1538923788 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ffba406b500 0x7ffba406d9c0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7ffbc4050b90 tx=0x7ffbb4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:12.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.066+0000 7ffbbaa00640 1 -- 172.21.15.28:0/1538923788 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ffbbc057aa0 con 0x7ffbc4151d60 2024-02-21T23:12:12.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.306+0000 7ffbc3e00640 1 -- 172.21.15.28:0/1538923788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffbc40c5cf0 con 0x7ffbc4151d60 2024-02-21T23:12:12.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.307+0000 7ffbbaa00640 1 -- 172.21.15.28:0/1538923788 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 209 v209) v1 ==== 95+0+4959 (secure 0 0 0) 0x7ffbbc05bb40 con 0x7ffbc4151d60 2024-02-21T23:12:12.295 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:12:12.295 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":209,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6848,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3439896042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3439896042},{"type":"v1","addr":"172.21.15.28:6835","nonce":3439896042}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":196},{"gid":7729,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/3213249808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3213249808},{"type":"v1","addr":"172.21.15.28:6833","nonce":3213249808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":195}],"filesystems":[{"mdsmap":{"epoch":209,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:11:38.384619+0000","modified":"2024-02-21T23:12:03.626987+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7867,"mds_1":7735},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7735":{"gid":7735,"name":"d","rank":1,"incarnation":208,"state":"up:active","state_seq":9,"addr":"172.21.15.81:6836/2895690310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2895690310},{"type":"v1","addr":"172.21.15.81:6836","nonce":2895690310}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7867":{"gid":7867,"name":"b","rank":0,"incarnation":206,"state":"up:active","state_seq":2,"addr":"172.21.15.81:6837/416021079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":416021079},{"type":"v1","addr":"172.21.15.81:6837","nonce":416021079}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-02-21T23:12:12.295 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 209 2024-02-21T23:12:12.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.310+0000 7ffbc3e00640 1 -- 172.21.15.28:0/1538923788 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ffba406b500 msgr2=0x7ffba406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:12.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.310+0000 7ffbc3e00640 1 --2- 172.21.15.28:0/1538923788 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ffba406b500 0x7ffba406d9c0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7ffbc4050b90 tx=0x7ffbb4008040 comp rx=0 tx=0).stop 2024-02-21T23:12:12.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.310+0000 7ffbc3e00640 1 -- 172.21.15.28:0/1538923788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbc4151d60 msgr2=0x7ffbc4143d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:12.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.310+0000 7ffbc3e00640 1 --2- 172.21.15.28:0/1538923788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbc4151d60 0x7ffbc4143d20 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 crypto rx=0x7ffbc40c83e0 tx=0x7ffbbc00cb90 comp rx=0 tx=0).stop 2024-02-21T23:12:12.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.310+0000 7ffbc3e00640 1 -- 172.21.15.28:0/1538923788 shutdown_connections 2024-02-21T23:12:12.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.310+0000 7ffbc3e00640 1 --2- 172.21.15.28:0/1538923788 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ffba406b500 0x7ffba406d9c0 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:12.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.310+0000 7ffbc3e00640 1 --2- 172.21.15.28:0/1538923788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbc4151d60 0x7ffbc4143d20 unknown :-1 s=CLOSED pgs=2172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:12.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.310+0000 7ffbc3e00640 1 --2- 172.21.15.28:0/1538923788 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffbc41522b0 0x7ffbc41437e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:12.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.310+0000 7ffbc3e00640 1 --2- 172.21.15.28:0/1538923788 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffba8004800 0x7ffbc4154710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:12.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.310+0000 7ffbc3e00640 1 -- 172.21.15.28:0/1538923788 >> 172.21.15.28:0/1538923788 conn(0x7ffbc40ac370 msgr2=0x7ffbc40ac750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:12.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.310+0000 7ffbc3e00640 1 -- 172.21.15.28:0/1538923788 shutdown_connections 2024-02-21T23:12:12.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.311+0000 7ffbc3e00640 1 -- 172.21.15.28:0/1538923788 wait complete. 2024-02-21T23:12:12.318 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:12:12.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.553+0000 7f4ff6800640 1 Processor -- start 2024-02-21T23:12:12.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.553+0000 7f4ff6800640 1 -- start start 2024-02-21T23:12:12.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.555+0000 7f4ff6800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4ff00c3280 0x7f4ff00c3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:12.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.555+0000 7f4ff6800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4ff0072430 0x7f4ff01526e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:12.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.555+0000 7f4ff6800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ff0152c20 con 0x7f4ff0155860 2024-02-21T23:12:12.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.555+0000 7f4ff6800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ff00c6560 con 0x7f4ff0072430 2024-02-21T23:12:12.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.555+0000 7f4ff6800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff00c5cc0 con 0x7f4ff00c3280 2024-02-21T23:12:12.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.555+0000 7f4feea00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4ff0072430 0x7f4ff01526e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:12.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.555+0000 7f4feea00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4ff0072430 0x7f4ff01526e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60418/0 (socket says 172.21.15.28:60418) 2024-02-21T23:12:12.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.555+0000 7f4feea00640 1 -- 172.21.15.28:0/1056371032 learned_addr learned my addr 172.21.15.28:0/1056371032 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:12.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.555+0000 7f4fef400640 1 --2- 172.21.15.28:0/1056371032 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4ff00c3280 0x7f4ff00c3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:12.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.556+0000 7f4feea00640 1 -- 172.21.15.28:0/1056371032 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4ff00c3280 msgr2=0x7f4ff00c3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:12.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.556+0000 7f4feea00640 1 --2- 172.21.15.28:0/1056371032 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4ff00c3280 0x7f4ff00c3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:12.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.556+0000 7f4feea00640 1 -- 172.21.15.28:0/1056371032 >> v1:172.21.15.81:6789/0 conn(0x7f4ff0155860 legacy=0x7f4ff00c46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:12.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.556+0000 7f4feea00640 1 -- 172.21.15.28:0/1056371032 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ff00c5fa0 con 0x7f4ff0072430 2024-02-21T23:12:12.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.557+0000 7f4feea00640 1 --2- 172.21.15.28:0/1056371032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ff0072430 0x7f4ff01526e0 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc009870 tx=0x7f4fdc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=737d507a02696ff9 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:12.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.557+0000 7f4fee000640 1 -- 172.21.15.28:0/1056371032 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fdc035020 con 0x7f4ff0072430 2024-02-21T23:12:12.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.557+0000 7f4fee000640 1 -- 172.21.15.28:0/1056371032 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fdc031640 con 0x7f4ff0072430 2024-02-21T23:12:12.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.557+0000 7f4fee000640 1 -- 172.21.15.28:0/1056371032 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fdc02d040 con 0x7f4ff0072430 2024-02-21T23:12:12.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.558+0000 7f4ff6800640 1 -- 172.21.15.28:0/1056371032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ff0072430 msgr2=0x7f4ff01526e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:12.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.558+0000 7f4ff6800640 1 --2- 172.21.15.28:0/1056371032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ff0072430 0x7f4ff01526e0 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc009870 tx=0x7f4fdc02b8f0 comp rx=0 tx=0).stop 2024-02-21T23:12:12.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.558+0000 7f4ff6800640 1 -- 172.21.15.28:0/1056371032 shutdown_connections 2024-02-21T23:12:12.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.558+0000 7f4ff6800640 1 --2- 172.21.15.28:0/1056371032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ff0072430 0x7f4ff01526e0 unknown :-1 s=CLOSED pgs=2173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:12.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.558+0000 7f4ff6800640 1 --2- 172.21.15.28:0/1056371032 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4ff00c3280 0x7f4ff00c3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:12.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.558+0000 7f4ff6800640 1 -- 172.21.15.28:0/1056371032 >> 172.21.15.28:0/1056371032 conn(0x7f4ff00c67f0 msgr2=0x7f4ff00c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:12.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.558+0000 7f4ff6800640 1 -- 172.21.15.28:0/1056371032 shutdown_connections 2024-02-21T23:12:12.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.558+0000 7f4ff6800640 1 -- 172.21.15.28:0/1056371032 wait complete. 2024-02-21T23:12:12.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.560+0000 7f4ff6800640 1 Processor -- start 2024-02-21T23:12:12.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.560+0000 7f4ff6800640 1 -- start start 2024-02-21T23:12:12.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.562+0000 7f4ff6800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ff0072430 0x7f4ff01490d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:12.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.562+0000 7f4ff6800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4ff00c3280 0x7f4ff00bfe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:12.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.562+0000 7f4ff6800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ff0155860 0x7f4ff01eed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:12.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.562+0000 7f4ff6800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff00c5920 con 0x7f4ff0155860 2024-02-21T23:12:12.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.562+0000 7f4ff6800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff00a6100 con 0x7f4ff0072430 2024-02-21T23:12:12.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.562+0000 7f4ff6800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff00c5aa0 con 0x7f4ff00c3280 2024-02-21T23:12:12.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.562+0000 7f4fefe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ff0155860 0x7f4ff01eed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:12.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.562+0000 7f4fef400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ff0072430 0x7f4ff01490d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:12.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.562+0000 7f4fefe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ff0155860 0x7f4ff01eed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60422/0 (socket says 172.21.15.28:60422) 2024-02-21T23:12:12.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.562+0000 7f4fef400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ff0072430 0x7f4ff01490d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.28:54394/0 (socket says 172.21.15.28:54394) 2024-02-21T23:12:12.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.562+0000 7f4fefe00640 1 -- 172.21.15.28:0/2510388141 learned_addr learned my addr 172.21.15.28:0/2510388141 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:12.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.562+0000 7f4feea00640 1 --2- 172.21.15.28:0/2510388141 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4ff00c3280 0x7f4ff00bfe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:12.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.562+0000 7f4fefe00640 1 -- 172.21.15.28:0/2510388141 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4ff00c3280 msgr2=0x7f4ff00bfe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:12.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.562+0000 7f4fefe00640 1 --2- 172.21.15.28:0/2510388141 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4ff00c3280 0x7f4ff00bfe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:12.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.562+0000 7f4fefe00640 1 -- 172.21.15.28:0/2510388141 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ff0072430 msgr2=0x7f4ff01490d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:12.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.562+0000 7f4fefe00640 1 --2- 172.21.15.28:0/2510388141 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ff0072430 0x7f4ff01490d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:12.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.562+0000 7f4fefe00640 1 -- 172.21.15.28:0/2510388141 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ff014cc60 con 0x7f4ff0155860 2024-02-21T23:12:12.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.563+0000 7f4fefe00640 1 --2- 172.21.15.28:0/2510388141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ff0155860 0x7f4ff01eed20 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 crypto rx=0x7f4fe800c6b0 tx=0x7f4fe800cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:12.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.563+0000 7f4fe7400640 1 -- 172.21.15.28:0/2510388141 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe8003b70 con 0x7f4ff0155860 2024-02-21T23:12:12.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.563+0000 7f4fe7400640 1 -- 172.21.15.28:0/2510388141 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fe8003d30 con 0x7f4ff0155860 2024-02-21T23:12:12.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.563+0000 7f4fe7400640 1 -- 172.21.15.28:0/2510388141 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe8005310 con 0x7f4ff0155860 2024-02-21T23:12:12.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.563+0000 7f4ff6800640 1 -- 172.21.15.28:0/2510388141 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ff014cf40 con 0x7f4ff0155860 2024-02-21T23:12:12.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.563+0000 7f4ff6800640 1 -- 172.21.15.28:0/2510388141 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ff0154ee0 con 0x7f4ff0155860 2024-02-21T23:12:12.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.565+0000 7f4ff6800640 1 -- 172.21.15.28:0/2510388141 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ff00c43e0 con 0x7f4ff0155860 2024-02-21T23:12:12.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.565+0000 7f4fe7400640 1 -- 172.21.15.28:0/2510388141 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4fe80054b0 con 0x7f4ff0155860 2024-02-21T23:12:12.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.567+0000 7f4fe7400640 1 --2- 172.21.15.28:0/2510388141 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4fd006b4d0 0x7f4fd006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:12.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.567+0000 7f4fe7400640 1 -- 172.21.15.28:0/2510388141 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4fe808a0c0 con 0x7f4ff0155860 2024-02-21T23:12:12.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.567+0000 7f4fef400640 1 --2- 172.21.15.28:0/2510388141 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4fd006b4d0 0x7f4fd006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:12.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.568+0000 7f4fef400640 1 --2- 172.21.15.28:0/2510388141 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4fd006b4d0 0x7f4fd006d990 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f4fe0006fd0 tx=0x7f4fe0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:12.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.571+0000 7f4fe7400640 1 -- 172.21.15.28:0/2510388141 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4ff00c43e0 con 0x7f4ff0155860 2024-02-21T23:12:12.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.772+0000 7f4ff6800640 1 -- 172.21.15.28:0/2510388141 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4ff00c4100 con 0x7f4ff0155860 2024-02-21T23:12:12.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.774+0000 7f4fe7400640 1 -- 172.21.15.28:0/2510388141 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v186) v1 ==== 74+0+16824 (secure 0 0 0) 0x7f4ff00c4100 con 0x7f4ff0155860 2024-02-21T23:12:12.762 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:12:12.763 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":186,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:11:39.917106+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:11:36.316421+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"185","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_data","create_time":"2024-02-21T23:11:37.418709+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"186","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:12:12.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.776+0000 7f4ff6800640 1 -- 172.21.15.28:0/2510388141 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4fd006b4d0 msgr2=0x7f4fd006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:12.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.776+0000 7f4ff6800640 1 --2- 172.21.15.28:0/2510388141 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4fd006b4d0 0x7f4fd006d990 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f4fe0006fd0 tx=0x7f4fe0008040 comp rx=0 tx=0).stop 2024-02-21T23:12:12.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.776+0000 7f4ff6800640 1 -- 172.21.15.28:0/2510388141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ff0155860 msgr2=0x7f4ff01eed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:12.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.776+0000 7f4ff6800640 1 --2- 172.21.15.28:0/2510388141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ff0155860 0x7f4ff01eed20 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 crypto rx=0x7f4fe800c6b0 tx=0x7f4fe800cb70 comp rx=0 tx=0).stop 2024-02-21T23:12:12.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.776+0000 7f4ff6800640 1 -- 172.21.15.28:0/2510388141 shutdown_connections 2024-02-21T23:12:12.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.776+0000 7f4ff6800640 1 --2- 172.21.15.28:0/2510388141 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4fd006b4d0 0x7f4fd006d990 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:12.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.776+0000 7f4ff6800640 1 --2- 172.21.15.28:0/2510388141 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4ff00c3280 0x7f4ff00bfe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:12.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.776+0000 7f4ff6800640 1 --2- 172.21.15.28:0/2510388141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ff0155860 0x7f4ff01eed20 unknown :-1 s=CLOSED pgs=2174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:12.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.776+0000 7f4ff6800640 1 --2- 172.21.15.28:0/2510388141 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ff0072430 0x7f4ff01490d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:12.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.776+0000 7f4ff6800640 1 -- 172.21.15.28:0/2510388141 >> 172.21.15.28:0/2510388141 conn(0x7f4ff00c67f0 msgr2=0x7f4ff00c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:12.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.777+0000 7f4ff6800640 1 -- 172.21.15.28:0/2510388141 shutdown_connections 2024-02-21T23:12:12.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:12.777+0000 7f4ff6800640 1 -- 172.21.15.28:0/2510388141 wait complete. 2024-02-21T23:12:12.781 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-02-21T23:12:12.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.007+0000 7f4a3fe00640 1 Processor -- start 2024-02-21T23:12:12.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.007+0000 7f4a3fe00640 1 -- start start 2024-02-21T23:12:12.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.009+0000 7f4a3fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4a380c2df0 0x7f4a38156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:12.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.009+0000 7f4a3fe00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4a380c5f20 0x7f4a380c31d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:12.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.009+0000 7f4a3fe00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a380c5c10 con 0x7f4a380c3710 2024-02-21T23:12:12.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.009+0000 7f4a3fe00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a3806e6a0 con 0x7f4a380c2df0 2024-02-21T23:12:12.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.009+0000 7f4a3fe00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a3806e8d0 con 0x7f4a380c5f20 2024-02-21T23:12:12.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.009+0000 7f4a3d400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4a380c2df0 0x7f4a38156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:12.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.009+0000 7f4a3d400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4a380c2df0 0x7f4a38156ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60430/0 (socket says 172.21.15.28:60430) 2024-02-21T23:12:12.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.009+0000 7f4a3d400640 1 -- 172.21.15.28:0/111950611 learned_addr learned my addr 172.21.15.28:0/111950611 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:12.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.009+0000 7f4a3ca00640 1 --2- 172.21.15.28:0/111950611 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4a380c5f20 0x7f4a380c31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:12.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.010+0000 7f4a3d400640 1 -- 172.21.15.28:0/111950611 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4a380c5f20 msgr2=0x7f4a380c31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:12.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.010+0000 7f4a3d400640 1 --2- 172.21.15.28:0/111950611 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4a380c5f20 0x7f4a380c31d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:12.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.010+0000 7f4a3d400640 1 -- 172.21.15.28:0/111950611 >> v1:172.21.15.28:6789/0 conn(0x7f4a380c3710 legacy=0x7f4a38153220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:12.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.010+0000 7f4a3d400640 1 -- 172.21.15.28:0/111950611 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a3806ebb0 con 0x7f4a380c2df0 2024-02-21T23:12:12.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.010+0000 7f4a37e00640 1 -- 172.21.15.28:0/111950611 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3006751615 0 0) 0x7f4a380c5c10 con 0x7f4a380c3710 2024-02-21T23:12:12.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.011+0000 7f4a3d400640 1 --2- 172.21.15.28:0/111950611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a380c2df0 0x7f4a38156ed0 secure :-1 s=READY pgs=2176 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c009870 tx=0x7f4a2c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f885359c0128b7 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:12.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.011+0000 7f4a37e00640 1 -- 172.21.15.28:0/111950611 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a2c034070 con 0x7f4a380c2df0 2024-02-21T23:12:12.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.011+0000 7f4a37e00640 1 -- 172.21.15.28:0/111950611 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a2c030840 con 0x7f4a380c2df0 2024-02-21T23:12:13.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.011+0000 7f4a37e00640 1 -- 172.21.15.28:0/111950611 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a2c030c10 con 0x7f4a380c2df0 2024-02-21T23:12:13.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.012+0000 7f4a3fe00640 1 -- 172.21.15.28:0/111950611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a380c2df0 msgr2=0x7f4a38156ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:13.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.012+0000 7f4a3fe00640 1 --2- 172.21.15.28:0/111950611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a380c2df0 0x7f4a38156ed0 secure :-1 s=READY pgs=2176 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c009870 tx=0x7f4a2c02b980 comp rx=0 tx=0).stop 2024-02-21T23:12:13.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.012+0000 7f4a3fe00640 1 -- 172.21.15.28:0/111950611 shutdown_connections 2024-02-21T23:12:13.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.012+0000 7f4a3fe00640 1 --2- 172.21.15.28:0/111950611 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4a380c5f20 0x7f4a380c31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:13.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.012+0000 7f4a3fe00640 1 --2- 172.21.15.28:0/111950611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a380c2df0 0x7f4a38156ed0 unknown :-1 s=CLOSED pgs=2176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:13.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.012+0000 7f4a3fe00640 1 -- 172.21.15.28:0/111950611 >> 172.21.15.28:0/111950611 conn(0x7f4a3806b820 msgr2=0x7f4a380c67f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:13.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.012+0000 7f4a3fe00640 1 -- 172.21.15.28:0/111950611 shutdown_connections 2024-02-21T23:12:13.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.012+0000 7f4a3fe00640 1 -- 172.21.15.28:0/111950611 wait complete. 2024-02-21T23:12:13.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.013+0000 7f4a3fe00640 1 Processor -- start 2024-02-21T23:12:13.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.014+0000 7f4a3fe00640 1 -- start start 2024-02-21T23:12:13.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.014+0000 7f4a3fe00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4a380c2df0 0x7f4a38152370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:13.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.014+0000 7f4a3fe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a380c3710 0x7f4a381eec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:13.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.014+0000 7f4a3fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a380c5f20 0x7f4a380c4980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:13.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.015+0000 7f4a3fe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a38153930 con 0x7f4a380c5f20 2024-02-21T23:12:13.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.015+0000 7f4a3fe00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a3806f170 con 0x7f4a380c3710 2024-02-21T23:12:13.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.015+0000 7f4a3fe00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a38153ab0 con 0x7f4a380c2df0 2024-02-21T23:12:13.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.015+0000 7f4a3de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a380c5f20 0x7f4a380c4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:13.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.015+0000 7f4a3de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a380c5f20 0x7f4a380c4980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60446/0 (socket says 172.21.15.28:60446) 2024-02-21T23:12:13.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.015+0000 7f4a3de00640 1 -- 172.21.15.28:0/3589984167 learned_addr learned my addr 172.21.15.28:0/3589984167 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:13.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.015+0000 7f4a3ca00640 1 --2- 172.21.15.28:0/3589984167 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a380c3710 0x7f4a381eec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:13.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.015+0000 7f4a3de00640 1 -- 172.21.15.28:0/3589984167 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4a380c2df0 msgr2=0x7f4a38152370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:13.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.015+0000 7f4a3de00640 1 --2- 172.21.15.28:0/3589984167 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4a380c2df0 0x7f4a38152370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:13.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.015+0000 7f4a3de00640 1 -- 172.21.15.28:0/3589984167 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a380c3710 msgr2=0x7f4a381eec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:13.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.015+0000 7f4a3de00640 1 --2- 172.21.15.28:0/3589984167 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a380c3710 0x7f4a381eec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:13.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.015+0000 7f4a3de00640 1 -- 172.21.15.28:0/3589984167 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a381ee170 con 0x7f4a380c5f20 2024-02-21T23:12:13.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.016+0000 7f4a3de00640 1 --2- 172.21.15.28:0/3589984167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a380c5f20 0x7f4a380c4980 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 crypto rx=0x7f4a3000c790 tx=0x7f4a3000cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:13.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.016+0000 7f4a35600640 1 -- 172.21.15.28:0/3589984167 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a30003da0 con 0x7f4a380c5f20 2024-02-21T23:12:13.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.016+0000 7f4a35600640 1 -- 172.21.15.28:0/3589984167 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a30005070 con 0x7f4a380c5f20 2024-02-21T23:12:13.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.016+0000 7f4a35600640 1 -- 172.21.15.28:0/3589984167 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a30005460 con 0x7f4a380c5f20 2024-02-21T23:12:13.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.016+0000 7f4a3fe00640 1 -- 172.21.15.28:0/3589984167 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a38152950 con 0x7f4a380c5f20 2024-02-21T23:12:13.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.016+0000 7f4a3fe00640 1 -- 172.21.15.28:0/3589984167 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a381ee410 con 0x7f4a380c5f20 2024-02-21T23:12:13.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.017+0000 7f4a3fe00640 1 -- 172.21.15.28:0/3589984167 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a3806f400 con 0x7f4a380c5f20 2024-02-21T23:12:13.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.018+0000 7f4a35600640 1 -- 172.21.15.28:0/3589984167 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4a30005600 con 0x7f4a380c5f20 2024-02-21T23:12:13.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.019+0000 7f4a35600640 1 --2- 172.21.15.28:0/3589984167 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4a1806b500 0x7f4a1806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:13.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.019+0000 7f4a35600640 1 -- 172.21.15.28:0/3589984167 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(186..186 src has 1..186) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4a3008a340 con 0x7f4a380c5f20 2024-02-21T23:12:13.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.019+0000 7f4a3d400640 1 --2- 172.21.15.28:0/3589984167 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4a1806b500 0x7f4a1806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:13.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.020+0000 7f4a3d400640 1 --2- 172.21.15.28:0/3589984167 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4a1806b500 0x7f4a1806d9c0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c036000 tx=0x7f4a2c03b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:13.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.023+0000 7f4a35600640 1 -- 172.21.15.28:0/3589984167 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4a30058f90 con 0x7f4a380c5f20 2024-02-21T23:12:13.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.263+0000 7f4a3fe00640 1 -- 172.21.15.28:0/3589984167 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f4a380c59f0 con 0x7f4a380c5f20 2024-02-21T23:12:13.252 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-02-21T23:12:13.264+0000 7f5ddc400640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-21T23:12:13.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.268+0000 7f4a35600640 1 -- 172.21.15.28:0/3589984167 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v210) v1 ==== 161+0+0 (secure 0 0 0) 0x7f4a30058890 con 0x7f4a380c5f20 2024-02-21T23:12:13.256 INFO:teuthology.orchestra.run.smithi028.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-02-21T23:12:13.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.272+0000 7f4a3fe00640 1 -- 172.21.15.28:0/3589984167 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4a1806b500 msgr2=0x7f4a1806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:13.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.272+0000 7f4a3fe00640 1 --2- 172.21.15.28:0/3589984167 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4a1806b500 0x7f4a1806d9c0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c036000 tx=0x7f4a2c03b040 comp rx=0 tx=0).stop 2024-02-21T23:12:13.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.272+0000 7f4a3fe00640 1 -- 172.21.15.28:0/3589984167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a380c5f20 msgr2=0x7f4a380c4980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:13.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.272+0000 7f4a3fe00640 1 --2- 172.21.15.28:0/3589984167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a380c5f20 0x7f4a380c4980 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 crypto rx=0x7f4a3000c790 tx=0x7f4a3000cc50 comp rx=0 tx=0).stop 2024-02-21T23:12:13.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.273+0000 7f4a3fe00640 1 -- 172.21.15.28:0/3589984167 shutdown_connections 2024-02-21T23:12:13.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.273+0000 7f4a3fe00640 1 --2- 172.21.15.28:0/3589984167 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4a1806b500 0x7f4a1806d9c0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:13.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.273+0000 7f4a3fe00640 1 --2- 172.21.15.28:0/3589984167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a380c5f20 0x7f4a380c4980 unknown :-1 s=CLOSED pgs=2177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:13.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.273+0000 7f4a3fe00640 1 --2- 172.21.15.28:0/3589984167 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4a380c3710 0x7f4a381eec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:13.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.273+0000 7f4a3fe00640 1 --2- 172.21.15.28:0/3589984167 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4a380c2df0 0x7f4a38152370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:13.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.273+0000 7f4a3fe00640 1 -- 172.21.15.28:0/3589984167 >> 172.21.15.28:0/3589984167 conn(0x7f4a3806b820 msgr2=0x7f4a380c10f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:13.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.273+0000 7f4a3fe00640 1 -- 172.21.15.28:0/3589984167 shutdown_connections 2024-02-21T23:12:13.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.273+0000 7f4a3fe00640 1 -- 172.21.15.28:0/3589984167 wait complete. 2024-02-21T23:12:13.283 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-02-21T23:12:13.288 INFO:tasks.ceph.mds.b.smithi081.stderr: -341> 2024-02-21T23:12:05.061+0000 7f1d0d000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-21T23:12:13.288 INFO:tasks.ceph.mds.b.smithi081.stderr: -339> 2024-02-21T23:12:05.061+0000 7f1d0d000640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-02-21T23:12:13.338 INFO:tasks.ceph.mds.d.smithi081.stdout:starting mds.d at 2024-02-21T23:12:13.339 INFO:tasks.ceph.mds.b.smithi081.stdout:starting mds.b at 2024-02-21T23:12:13.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.506+0000 7f68b7e00640 1 Processor -- start 2024-02-21T23:12:13.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.506+0000 7f68b7e00640 1 -- start start 2024-02-21T23:12:13.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.507+0000 7f68b7e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f68b81500f0 0x7f68b806ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:13.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.507+0000 7f68b7e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f68b806b3b0 0x7f68b806e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:13.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.507+0000 7f68b7e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68b806db10 con 0x7f68b80c2c60 2024-02-21T23:12:13.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.507+0000 7f68b7e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68b80c1080 con 0x7f68b81500f0 2024-02-21T23:12:13.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.507+0000 7f68b7e00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68b81539a0 con 0x7f68b806b3b0 2024-02-21T23:12:13.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.508+0000 7f68b6a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f68b81500f0 0x7f68b806ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:13.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.508+0000 7f68b6a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f68b81500f0 0x7f68b806ae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60452/0 (socket says 172.21.15.28:60452) 2024-02-21T23:12:13.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.508+0000 7f68b6a00640 1 -- 172.21.15.28:0/214117680 learned_addr learned my addr 172.21.15.28:0/214117680 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:13.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.508+0000 7f68b6a00640 1 -- 172.21.15.28:0/214117680 >> v2:172.21.15.81:3300/0 conn(0x7f68b806b3b0 msgr2=0x7f68b806e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:13.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.508+0000 7f68b6a00640 1 --2- 172.21.15.28:0/214117680 >> v2:172.21.15.81:3300/0 conn(0x7f68b806b3b0 0x7f68b806e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:13.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.508+0000 7f68b6a00640 1 -- 172.21.15.28:0/214117680 >> v1:172.21.15.81:6789/0 conn(0x7f68b80c2c60 legacy=0x7f68b806dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:13.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.508+0000 7f68b6a00640 1 -- 172.21.15.28:0/214117680 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68b80c12d0 con 0x7f68b81500f0 2024-02-21T23:12:13.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.509+0000 7f68b6a00640 1 --2- 172.21.15.28:0/214117680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68b81500f0 0x7f68b806ae70 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 crypto rx=0x7f68b806dce0 tx=0x7f68a802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=327e3a78306905c7 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:13.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.509+0000 7f68b5600640 1 -- 172.21.15.28:0/214117680 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68a8035020 con 0x7f68b81500f0 2024-02-21T23:12:13.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.510+0000 7f68b5600640 1 -- 172.21.15.28:0/214117680 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68a8031870 con 0x7f68b81500f0 2024-02-21T23:12:13.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.510+0000 7f68b5600640 1 -- 172.21.15.28:0/214117680 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68a8031c40 con 0x7f68b81500f0 2024-02-21T23:12:13.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.510+0000 7f68b7e00640 1 -- 172.21.15.28:0/214117680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68b81500f0 msgr2=0x7f68b806ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:13.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.510+0000 7f68b7e00640 1 --2- 172.21.15.28:0/214117680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68b81500f0 0x7f68b806ae70 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 crypto rx=0x7f68b806dce0 tx=0x7f68a802b910 comp rx=0 tx=0).stop 2024-02-21T23:12:13.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.510+0000 7f68b7e00640 1 -- 172.21.15.28:0/214117680 shutdown_connections 2024-02-21T23:12:13.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.510+0000 7f68b7e00640 1 --2- 172.21.15.28:0/214117680 >> v2:172.21.15.81:3300/0 conn(0x7f68b806b3b0 0x7f68b806e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:13.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.510+0000 7f68b7e00640 1 --2- 172.21.15.28:0/214117680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68b81500f0 0x7f68b806ae70 unknown :-1 s=CLOSED pgs=2180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:13.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.510+0000 7f68b7e00640 1 -- 172.21.15.28:0/214117680 >> 172.21.15.28:0/214117680 conn(0x7f68b80a8530 msgr2=0x7f68b80a8910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:13.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.510+0000 7f68b7e00640 1 -- 172.21.15.28:0/214117680 shutdown_connections 2024-02-21T23:12:13.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.510+0000 7f68b7e00640 1 -- 172.21.15.28:0/214117680 wait complete. 2024-02-21T23:12:13.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.512+0000 7f68b7e00640 1 Processor -- start 2024-02-21T23:12:13.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.512+0000 7f68b7e00640 1 -- start start 2024-02-21T23:12:13.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.513+0000 7f68b7e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f68b806b3b0 0x7f68b8071b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:13.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.513+0000 7f68b7e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f68b80c2c60 0x7f68b8145790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:13.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.513+0000 7f68b7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68b81500f0 0x7f68b8145cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:13.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.513+0000 7f68b7e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68b81535e0 con 0x7f68b81500f0 2024-02-21T23:12:13.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.513+0000 7f68b7e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68b81504d0 con 0x7f68b80c2c60 2024-02-21T23:12:13.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.513+0000 7f68b7e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68b8153760 con 0x7f68b806b3b0 2024-02-21T23:12:13.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.513+0000 7f68b7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68b81500f0 0x7f68b8145cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:13.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.513+0000 7f68b7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68b81500f0 0x7f68b8145cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60454/0 (socket says 172.21.15.28:60454) 2024-02-21T23:12:13.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.513+0000 7f68b7400640 1 -- 172.21.15.28:0/1779077902 learned_addr learned my addr 172.21.15.28:0/1779077902 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:13.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.514+0000 7f68b6000640 1 --2- 172.21.15.28:0/1779077902 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f68b80c2c60 0x7f68b8145790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:13.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.514+0000 7f68b6a00640 1 --2- 172.21.15.28:0/1779077902 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f68b806b3b0 0x7f68b8071b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:13.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.514+0000 7f68b7400640 1 -- 172.21.15.28:0/1779077902 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f68b806b3b0 msgr2=0x7f68b8071b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:13.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.514+0000 7f68b7400640 1 --2- 172.21.15.28:0/1779077902 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f68b806b3b0 0x7f68b8071b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:13.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.514+0000 7f68b7400640 1 -- 172.21.15.28:0/1779077902 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f68b80c2c60 msgr2=0x7f68b8145790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:13.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.514+0000 7f68b7400640 1 --2- 172.21.15.28:0/1779077902 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f68b80c2c60 0x7f68b8145790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:13.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.514+0000 7f68b7400640 1 -- 172.21.15.28:0/1779077902 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68b80a7360 con 0x7f68b81500f0 2024-02-21T23:12:13.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.514+0000 7f68b7400640 1 --2- 172.21.15.28:0/1779077902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68b81500f0 0x7f68b8145cd0 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7f68b000b9a0 tx=0x7f68b000be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:13.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.514+0000 7f68aea00640 1 -- 172.21.15.28:0/1779077902 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68b0019070 con 0x7f68b81500f0 2024-02-21T23:12:13.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.514+0000 7f68aea00640 1 -- 172.21.15.28:0/1779077902 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68b000cbc0 con 0x7f68b81500f0 2024-02-21T23:12:13.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.514+0000 7f68aea00640 1 -- 172.21.15.28:0/1779077902 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68b0014440 con 0x7f68b81500f0 2024-02-21T23:12:13.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.514+0000 7f68b7e00640 1 -- 172.21.15.28:0/1779077902 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68b80a7640 con 0x7f68b81500f0 2024-02-21T23:12:13.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.515+0000 7f68b7e00640 1 -- 172.21.15.28:0/1779077902 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68b8146210 con 0x7f68b81500f0 2024-02-21T23:12:13.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.516+0000 7f68b7e00640 1 -- 172.21.15.28:0/1779077902 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68b80a7b60 con 0x7f68b81500f0 2024-02-21T23:12:13.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.517+0000 7f68aea00640 1 -- 172.21.15.28:0/1779077902 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f68b001c070 con 0x7f68b81500f0 2024-02-21T23:12:13.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.518+0000 7f68aea00640 1 --2- 172.21.15.28:0/1779077902 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f689806b4d0 0x7f689806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:13.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.518+0000 7f68aea00640 1 -- 172.21.15.28:0/1779077902 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f68b008a140 con 0x7f68b81500f0 2024-02-21T23:12:13.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.519+0000 7f68b6a00640 1 --2- 172.21.15.28:0/1779077902 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f689806b4d0 0x7f689806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:13.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.520+0000 7f68b6a00640 1 --2- 172.21.15.28:0/1779077902 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f689806b4d0 0x7f689806d990 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f68a8036000 tx=0x7f68a803c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:13.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.523+0000 7f68aea00640 1 -- 172.21.15.28:0/1779077902 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f68b80a7b60 con 0x7f68b81500f0 2024-02-21T23:12:13.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:13.768+0000 7f68b7e00640 1 -- 172.21.15.28:0/1779077902 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f68b804f290 con 0x7f68b81500f0 2024-02-21T23:12:14.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.271+0000 7f68aea00640 1 -- 172.21.15.28:0/1779077902 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v211) v1 ==== 104+0+0 (secure 0 0 0) 0x7f68b0058e10 con 0x7f68b81500f0 2024-02-21T23:12:14.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.276+0000 7f68b7e00640 1 -- 172.21.15.28:0/1779077902 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f689806b4d0 msgr2=0x7f689806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:14.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.276+0000 7f68b7e00640 1 --2- 172.21.15.28:0/1779077902 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f689806b4d0 0x7f689806d990 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f68a8036000 tx=0x7f68a803c040 comp rx=0 tx=0).stop 2024-02-21T23:12:14.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.276+0000 7f68b7e00640 1 -- 172.21.15.28:0/1779077902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68b81500f0 msgr2=0x7f68b8145cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:14.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.276+0000 7f68b7e00640 1 --2- 172.21.15.28:0/1779077902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68b81500f0 0x7f68b8145cd0 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7f68b000b9a0 tx=0x7f68b000be60 comp rx=0 tx=0).stop 2024-02-21T23:12:14.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.276+0000 7f68b7e00640 1 -- 172.21.15.28:0/1779077902 shutdown_connections 2024-02-21T23:12:14.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.276+0000 7f68b7e00640 1 --2- 172.21.15.28:0/1779077902 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f689806b4d0 0x7f689806d990 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:14.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.276+0000 7f68b7e00640 1 --2- 172.21.15.28:0/1779077902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68b81500f0 0x7f68b8145cd0 unknown :-1 s=CLOSED pgs=2181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:14.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.276+0000 7f68b7e00640 1 --2- 172.21.15.28:0/1779077902 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f68b80c2c60 0x7f68b8145790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:14.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.276+0000 7f68b7e00640 1 --2- 172.21.15.28:0/1779077902 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f68b806b3b0 0x7f68b8071b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:14.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.277+0000 7f68b7e00640 1 -- 172.21.15.28:0/1779077902 >> 172.21.15.28:0/1779077902 conn(0x7f68b80a8530 msgr2=0x7f68b806dff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:14.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.277+0000 7f68b7e00640 1 -- 172.21.15.28:0/1779077902 shutdown_connections 2024-02-21T23:12:14.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.277+0000 7f68b7e00640 1 -- 172.21.15.28:0/1779077902 wait complete. 2024-02-21T23:12:14.287 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-02-21T23:12:14.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.529+0000 7fa4b2800640 1 Processor -- start 2024-02-21T23:12:14.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.529+0000 7fa4b2800640 1 -- start start 2024-02-21T23:12:14.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.531+0000 7fa4b2800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fa4ac06e780 0x7fa4ac06eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:14.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.531+0000 7fa4b2800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa4ac0c5290 0x7fa4ac15d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:14.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.531+0000 7fa4b2800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4ac153cb0 con 0x7fa4ac0a5fc0 2024-02-21T23:12:14.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.531+0000 7fa4b2800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4ac0c5700 con 0x7fa4ac0c5290 2024-02-21T23:12:14.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.531+0000 7fa4b2800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4ac1543d0 con 0x7fa4ac06e780 2024-02-21T23:12:14.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.531+0000 7fa4abe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa4ac0c5290 0x7fa4ac15d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:14.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.531+0000 7fa4abe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa4ac0c5290 0x7fa4ac15d860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60460/0 (socket says 172.21.15.28:60460) 2024-02-21T23:12:14.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.531+0000 7fa4abe00640 1 -- 172.21.15.28:0/2140239855 learned_addr learned my addr 172.21.15.28:0/2140239855 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:14.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.531+0000 7fa4ab400640 1 --2- 172.21.15.28:0/2140239855 >> v2:172.21.15.81:3300/0 conn(0x7fa4ac06e780 0x7fa4ac06eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:14.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.532+0000 7fa4abe00640 1 -- 172.21.15.28:0/2140239855 >> v2:172.21.15.81:3300/0 conn(0x7fa4ac06e780 msgr2=0x7fa4ac06eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:14.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.532+0000 7fa4abe00640 1 --2- 172.21.15.28:0/2140239855 >> v2:172.21.15.81:3300/0 conn(0x7fa4ac06e780 0x7fa4ac06eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:14.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.532+0000 7fa4abe00640 1 -- 172.21.15.28:0/2140239855 >> v1:172.21.15.81:6789/0 conn(0x7fa4ac0a5fc0 legacy=0x7fa4ac0c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:14.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.532+0000 7fa4abe00640 1 -- 172.21.15.28:0/2140239855 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4ac0c5d20 con 0x7fa4ac0c5290 2024-02-21T23:12:14.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.533+0000 7fa4abe00640 1 --2- 172.21.15.28:0/2140239855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4ac0c5290 0x7fa4ac15d860 secure :-1 s=READY pgs=2182 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac06f810 tx=0x7fa4a402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6419c0739cee6acf server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:14.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.533+0000 7fa4aa000640 1 -- 172.21.15.28:0/2140239855 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a402c9f0 con 0x7fa4ac0c5290 2024-02-21T23:12:14.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.533+0000 7fa4aa000640 1 -- 172.21.15.28:0/2140239855 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4a402cb90 con 0x7fa4ac0c5290 2024-02-21T23:12:14.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.533+0000 7fa4aa000640 1 -- 172.21.15.28:0/2140239855 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a40343f0 con 0x7fa4ac0c5290 2024-02-21T23:12:14.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.534+0000 7fa4b2800640 1 -- 172.21.15.28:0/2140239855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4ac0c5290 msgr2=0x7fa4ac15d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:14.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.534+0000 7fa4b2800640 1 --2- 172.21.15.28:0/2140239855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4ac0c5290 0x7fa4ac15d860 secure :-1 s=READY pgs=2182 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac06f810 tx=0x7fa4a402be50 comp rx=0 tx=0).stop 2024-02-21T23:12:14.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.534+0000 7fa4b2800640 1 -- 172.21.15.28:0/2140239855 shutdown_connections 2024-02-21T23:12:14.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.534+0000 7fa4b2800640 1 --2- 172.21.15.28:0/2140239855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4ac0c5290 0x7fa4ac15d860 unknown :-1 s=CLOSED pgs=2182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:14.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.534+0000 7fa4b2800640 1 --2- 172.21.15.28:0/2140239855 >> v2:172.21.15.81:3300/0 conn(0x7fa4ac06e780 0x7fa4ac06eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:14.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.534+0000 7fa4b2800640 1 -- 172.21.15.28:0/2140239855 >> 172.21.15.28:0/2140239855 conn(0x7fa4ac06b830 msgr2=0x7fa4ac0c67f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:14.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.534+0000 7fa4b2800640 1 -- 172.21.15.28:0/2140239855 shutdown_connections 2024-02-21T23:12:14.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.534+0000 7fa4b2800640 1 -- 172.21.15.28:0/2140239855 wait complete. 2024-02-21T23:12:14.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.536+0000 7fa4b2800640 1 Processor -- start 2024-02-21T23:12:14.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.536+0000 7fa4b2800640 1 -- start start 2024-02-21T23:12:14.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.537+0000 7fa4b2800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4ac06e780 0x7fa4ac0a8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:14.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.537+0000 7fa4b2800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa4ac0a5fc0 0x7fa4ac0a6450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:14.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.537+0000 7fa4b2800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa4ac0c5290 0x7fa4ac0a6990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:14.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.537+0000 7fa4b2800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ac0c46f0 con 0x7fa4ac06e780 2024-02-21T23:12:14.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.537+0000 7fa4b2800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ac0c5ad0 con 0x7fa4ac0a5fc0 2024-02-21T23:12:14.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.537+0000 7fa4b2800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ac153960 con 0x7fa4ac0c5290 2024-02-21T23:12:14.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.537+0000 7fa4ab400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4ac06e780 0x7fa4ac0a8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:14.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.537+0000 7fa4ab400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4ac06e780 0x7fa4ac0a8a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60470/0 (socket says 172.21.15.28:60470) 2024-02-21T23:12:14.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.537+0000 7fa4ab400640 1 -- 172.21.15.28:0/1456625851 learned_addr learned my addr 172.21.15.28:0/1456625851 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:14.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.537+0000 7fa4ab400640 1 -- 172.21.15.28:0/1456625851 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa4ac0c5290 msgr2=0x7fa4ac0a6990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:14.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.537+0000 7fa4ab400640 1 --2- 172.21.15.28:0/1456625851 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa4ac0c5290 0x7fa4ac0a6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:14.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.537+0000 7fa4aaa00640 1 --2- 172.21.15.28:0/1456625851 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa4ac0a5fc0 0x7fa4ac0a6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:14.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.537+0000 7fa4ab400640 1 -- 172.21.15.28:0/1456625851 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa4ac0a5fc0 msgr2=0x7fa4ac0a6450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:14.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.538+0000 7fa4ab400640 1 --2- 172.21.15.28:0/1456625851 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa4ac0a5fc0 0x7fa4ac0a6450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:14.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.538+0000 7fa4ab400640 1 -- 172.21.15.28:0/1456625851 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4ac1e8910 con 0x7fa4ac06e780 2024-02-21T23:12:14.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.538+0000 7fa4aaa00640 1 --2- 172.21.15.28:0/1456625851 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa4ac0a5fc0 0x7fa4ac0a6450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:12:14.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.538+0000 7fa4ab400640 1 --2- 172.21.15.28:0/1456625851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4ac06e780 0x7fa4ac0a8a60 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 crypto rx=0x7fa49c00d620 tx=0x7fa49c00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:14.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.538+0000 7fa4a3400640 1 -- 172.21.15.28:0/1456625851 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa49c00b620 con 0x7fa4ac06e780 2024-02-21T23:12:14.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.538+0000 7fa4a3400640 1 -- 172.21.15.28:0/1456625851 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa49c019070 con 0x7fa4ac06e780 2024-02-21T23:12:14.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.538+0000 7fa4a3400640 1 -- 172.21.15.28:0/1456625851 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa49c00ba00 con 0x7fa4ac06e780 2024-02-21T23:12:14.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.538+0000 7fa4b2800640 1 -- 172.21.15.28:0/1456625851 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4ac1e01b0 con 0x7fa4ac06e780 2024-02-21T23:12:14.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.538+0000 7fa4b2800640 1 -- 172.21.15.28:0/1456625851 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4ac1e8eb0 con 0x7fa4ac06e780 2024-02-21T23:12:14.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.540+0000 7fa4b2800640 1 -- 172.21.15.28:0/1456625851 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4ac067f30 con 0x7fa4ac06e780 2024-02-21T23:12:14.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.540+0000 7fa4a3400640 1 -- 172.21.15.28:0/1456625851 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa49c00bba0 con 0x7fa4ac06e780 2024-02-21T23:12:14.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.541+0000 7fa4a3400640 1 --2- 172.21.15.28:0/1456625851 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa48c06b4d0 0x7fa48c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:14.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.541+0000 7fa4a3400640 1 -- 172.21.15.28:0/1456625851 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(187..187 src has 1..187) v4 ==== 6273+0+0 (secure 0 0 0) 0x7fa49c08a890 con 0x7fa4ac06e780 2024-02-21T23:12:14.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.542+0000 7fa4aaa00640 1 --2- 172.21.15.28:0/1456625851 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa48c06b4d0 0x7fa48c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:14.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.543+0000 7fa4aaa00640 1 --2- 172.21.15.28:0/1456625851 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa48c06b4d0 0x7fa48c06d990 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac0a8080 tx=0x7fa498008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:14.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.545+0000 7fa4a3400640 1 -- 172.21.15.28:0/1456625851 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fa49c0594c0 con 0x7fa4ac06e780 2024-02-21T23:12:14.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:14.739+0000 7fa4b2800640 1 -- 172.21.15.28:0/1456625851 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fa4ac0c58b0 con 0x7fa4ac06e780 2024-02-21T23:12:15.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.671+0000 7fa4a3400640 1 -- 172.21.15.28:0/1456625851 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v188) v1 ==== 181+0+0 (secure 0 0 0) 0x7fa49c058dc0 con 0x7fa4ac06e780 2024-02-21T23:12:15.660 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' removed 2024-02-21T23:12:15.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.676+0000 7fa4b2800640 1 -- 172.21.15.28:0/1456625851 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa48c06b4d0 msgr2=0x7fa48c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:15.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.676+0000 7fa4b2800640 1 --2- 172.21.15.28:0/1456625851 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa48c06b4d0 0x7fa48c06d990 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac0a8080 tx=0x7fa498008040 comp rx=0 tx=0).stop 2024-02-21T23:12:15.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.676+0000 7fa4b2800640 1 -- 172.21.15.28:0/1456625851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4ac06e780 msgr2=0x7fa4ac0a8a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:15.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.676+0000 7fa4b2800640 1 --2- 172.21.15.28:0/1456625851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4ac06e780 0x7fa4ac0a8a60 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 crypto rx=0x7fa49c00d620 tx=0x7fa49c00dae0 comp rx=0 tx=0).stop 2024-02-21T23:12:15.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.676+0000 7fa4b2800640 1 -- 172.21.15.28:0/1456625851 shutdown_connections 2024-02-21T23:12:15.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.676+0000 7fa4b2800640 1 --2- 172.21.15.28:0/1456625851 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa48c06b4d0 0x7fa48c06d990 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:15.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.676+0000 7fa4b2800640 1 --2- 172.21.15.28:0/1456625851 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa4ac0c5290 0x7fa4ac0a6990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:15.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.676+0000 7fa4b2800640 1 --2- 172.21.15.28:0/1456625851 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa4ac0a5fc0 0x7fa4ac0a6450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:15.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.676+0000 7fa4b2800640 1 --2- 172.21.15.28:0/1456625851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4ac06e780 0x7fa4ac0a8a60 unknown :-1 s=CLOSED pgs=2183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:15.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.676+0000 7fa4b2800640 1 -- 172.21.15.28:0/1456625851 >> 172.21.15.28:0/1456625851 conn(0x7fa4ac06b830 msgr2=0x7fa4ac0c67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:15.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.677+0000 7fa4b2800640 1 -- 172.21.15.28:0/1456625851 shutdown_connections 2024-02-21T23:12:15.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.678+0000 7fa4b2800640 1 -- 172.21.15.28:0/1456625851 wait complete. 2024-02-21T23:12:15.689 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-02-21T23:12:15.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.921+0000 7fb5bac00640 1 Processor -- start 2024-02-21T23:12:15.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.921+0000 7fb5bac00640 1 -- start start 2024-02-21T23:12:15.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.923+0000 7fb5bac00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb5b40c2a90 0x7fb5b406b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:15.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.923+0000 7fb5bac00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5b41527d0 con 0x7fb5b406ae70 2024-02-21T23:12:15.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.923+0000 7fb5bac00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5b40724f0 con 0x7fb5b4155870 2024-02-21T23:12:15.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.923+0000 7fb5bac00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b40727d0 con 0x7fb5b40c2a90 2024-02-21T23:12:15.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.923+0000 7fb5b9800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb5b406ae70 0x7fb5b4155160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39658/0 (socket says 172.21.15.28:39658) 2024-02-21T23:12:15.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.923+0000 7fb5b9800640 1 -- 172.21.15.28:0/3223676296 learned_addr learned my addr 172.21.15.28:0/3223676296 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:12:15.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.924+0000 7fb5b8e00640 1 --2- 172.21.15.28:0/3223676296 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb5b40c2a90 0x7fb5b406b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:15.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.924+0000 7fb5b3e00640 1 -- 172.21.15.28:0/3223676296 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1542094162 0 0) 0x7fb5b41527d0 con 0x7fb5b406ae70 2024-02-21T23:12:15.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.924+0000 7fb5b3e00640 1 -- 172.21.15.28:0/3223676296 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb598003520 con 0x7fb5b406ae70 2024-02-21T23:12:15.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.924+0000 7fb5b3e00640 1 -- 172.21.15.28:0/3223676296 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2779866833 0 0) 0x7fb598003520 con 0x7fb5b406ae70 2024-02-21T23:12:15.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.924+0000 7fb5b3e00640 1 -- 172.21.15.28:0/3223676296 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb5b40c2a90 msgr2=0x7fb5b406b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:15.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.924+0000 7fb5b3e00640 1 --2- 172.21.15.28:0/3223676296 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb5b40c2a90 0x7fb5b406b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:15.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.924+0000 7fb5b3e00640 1 -- 172.21.15.28:0/3223676296 >> v1:172.21.15.81:6789/0 conn(0x7fb5b4155870 legacy=0x7fb5b415c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:15.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.924+0000 7fb5b3e00640 1 -- 172.21.15.28:0/3223676296 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5b406f720 con 0x7fb5b406ae70 2024-02-21T23:12:15.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.924+0000 7fb5b3e00640 1 -- 172.21.15.28:0/3223676296 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fb5a4001670 con 0x7fb5b406ae70 2024-02-21T23:12:15.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.924+0000 7fb5b3e00640 1 -- 172.21.15.28:0/3223676296 >> v1:172.21.15.28:6789/0 conn(0x7fb5b406ae70 legacy=0x7fb5b4155160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:15.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.924+0000 7fb5b3e00640 1 --2- 172.21.15.28:0/3223676296 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb5b40c2a90 0x7fb5980039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:15.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.924+0000 7fb5b3e00640 1 --2- 172.21.15.28:0/3223676296 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb598003fb0 0x7fb598004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:15.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.924+0000 7fb5b3e00640 1 --2- 172.21.15.28:0/3223676296 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb598004d40 0x7fb5980051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:15.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.924+0000 7fb5b3e00640 1 -- 172.21.15.28:0/3223676296 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b40727d0 con 0x7fb598003fb0 2024-02-21T23:12:15.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.924+0000 7fb5b3e00640 1 -- 172.21.15.28:0/3223676296 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb59800bb00 con 0x7fb598004d40 2024-02-21T23:12:15.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.924+0000 7fb5b3e00640 1 -- 172.21.15.28:0/3223676296 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb59800df10 con 0x7fb5b40c2a90 2024-02-21T23:12:15.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.925+0000 7fb5b8e00640 1 --2- 172.21.15.28:0/3223676296 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb598003fb0 0x7fb598004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:15.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.925+0000 7fb5b8e00640 1 -- 172.21.15.28:0/3223676296 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb5b40c2a90 msgr2=0x7fb5980039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:15.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.925+0000 7fb5b8e00640 1 --2- 172.21.15.28:0/3223676296 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb5b40c2a90 0x7fb5980039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:15.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.925+0000 7fb5b8e00640 1 -- 172.21.15.28:0/3223676296 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb598004d40 msgr2=0x7fb5980051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:15.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.925+0000 7fb5b8e00640 1 --2- 172.21.15.28:0/3223676296 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb598004d40 0x7fb5980051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:15.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.925+0000 7fb5b8e00640 1 -- 172.21.15.28:0/3223676296 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb598003520 con 0x7fb598003fb0 2024-02-21T23:12:15.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.926+0000 7fb5b8e00640 1 --2- 172.21.15.28:0/3223676296 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb598003fb0 0x7fb598004800 secure :-1 s=READY pgs=2185 cs=0 l=1 rev1=1 crypto rx=0x7fb5a0009ac0 tx=0x7fb598005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c39c9179ad10f server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:15.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.926+0000 7fb5b3e00640 1 -- 172.21.15.28:0/3223676296 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5a0039070 con 0x7fb598003fb0 2024-02-21T23:12:15.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.926+0000 7fb5b3e00640 1 -- 172.21.15.28:0/3223676296 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5a002c630 con 0x7fb598003fb0 2024-02-21T23:12:15.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.926+0000 7fb5b3e00640 1 -- 172.21.15.28:0/3223676296 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5a0009690 con 0x7fb598003fb0 2024-02-21T23:12:15.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.927+0000 7fb5bac00640 1 -- 172.21.15.28:0/3223676296 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb598003fb0 msgr2=0x7fb598004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:15.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.927+0000 7fb5bac00640 1 --2- 172.21.15.28:0/3223676296 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb598003fb0 0x7fb598004800 secure :-1 s=READY pgs=2185 cs=0 l=1 rev1=1 crypto rx=0x7fb5a0009ac0 tx=0x7fb598005830 comp rx=0 tx=0).stop 2024-02-21T23:12:15.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.927+0000 7fb5ba200640 1 -- 172.21.15.28:0/3223676296 reap_dead start 2024-02-21T23:12:15.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.927+0000 7fb5bac00640 1 -- 172.21.15.28:0/3223676296 shutdown_connections 2024-02-21T23:12:15.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.927+0000 7fb5bac00640 1 -- 172.21.15.28:0/3223676296 >> 172.21.15.28:0/3223676296 conn(0x7fb5b40c6610 msgr2=0x7fb5b40c69f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:15.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.927+0000 7fb5bac00640 1 -- 172.21.15.28:0/3223676296 shutdown_connections 2024-02-21T23:12:15.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.927+0000 7fb5bac00640 1 -- 172.21.15.28:0/3223676296 wait complete. 2024-02-21T23:12:15.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.929+0000 7fb5bac00640 1 Processor -- start 2024-02-21T23:12:15.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.929+0000 7fb5bac00640 1 -- start start 2024-02-21T23:12:15.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.930+0000 7fb5bac00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb598003fb0 0x7fb5b40c5de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:15.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.930+0000 7fb5bac00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5b41420b0 0x7fb5b41475e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:15.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.930+0000 7fb5bac00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb5b4147b20 0x7fb5b41ea9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:15.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.930+0000 7fb5bac00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b4155cf0 con 0x7fb598003fb0 2024-02-21T23:12:15.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.930+0000 7fb5bac00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b40c2240 con 0x7fb5b41420b0 2024-02-21T23:12:15.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.930+0000 7fb5bac00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b4155e70 con 0x7fb5b4147b20 2024-02-21T23:12:15.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.930+0000 7fb5b9800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb598003fb0 0x7fb5b40c5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:15.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.930+0000 7fb5b9800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb598003fb0 0x7fb5b40c5de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41510/0 (socket says 172.21.15.28:41510) 2024-02-21T23:12:15.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.930+0000 7fb5b9800640 1 -- 172.21.15.28:0/3678847363 learned_addr learned my addr 172.21.15.28:0/3678847363 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:15.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.931+0000 7fb5b8e00640 1 --2- 172.21.15.28:0/3678847363 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5b41420b0 0x7fb5b41475e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:15.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.931+0000 7fb5ba200640 1 --2- 172.21.15.28:0/3678847363 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb5b4147b20 0x7fb5b41ea9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:15.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.931+0000 7fb5b9800640 1 -- 172.21.15.28:0/3678847363 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb5b4147b20 msgr2=0x7fb5b41ea9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:15.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.931+0000 7fb5b9800640 1 --2- 172.21.15.28:0/3678847363 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb5b4147b20 0x7fb5b41ea9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:15.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.931+0000 7fb5b9800640 1 -- 172.21.15.28:0/3678847363 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5b41420b0 msgr2=0x7fb5b41475e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:15.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.931+0000 7fb5b9800640 1 --2- 172.21.15.28:0/3678847363 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5b41420b0 0x7fb5b41475e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:15.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.931+0000 7fb5b9800640 1 -- 172.21.15.28:0/3678847363 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5b40c45d0 con 0x7fb598003fb0 2024-02-21T23:12:15.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.931+0000 7fb5b9800640 1 --2- 172.21.15.28:0/3678847363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb598003fb0 0x7fb5b40c5de0 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 crypto rx=0x7fb5a400c7a0 tx=0x7fb5a400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:15.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.931+0000 7fb5b1600640 1 -- 172.21.15.28:0/3678847363 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5a4018070 con 0x7fb598003fb0 2024-02-21T23:12:15.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.931+0000 7fb5b1600640 1 -- 172.21.15.28:0/3678847363 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5a40074e0 con 0x7fb598003fb0 2024-02-21T23:12:15.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.931+0000 7fb5b1600640 1 -- 172.21.15.28:0/3678847363 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5a4004a20 con 0x7fb598003fb0 2024-02-21T23:12:15.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.932+0000 7fb5bac00640 1 -- 172.21.15.28:0/3678847363 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5b4151500 con 0x7fb598003fb0 2024-02-21T23:12:15.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.932+0000 7fb5bac00640 1 -- 172.21.15.28:0/3678847363 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5b40abcc0 con 0x7fb598003fb0 2024-02-21T23:12:15.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.933+0000 7fb5bac00640 1 -- 172.21.15.28:0/3678847363 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5b404f290 con 0x7fb598003fb0 2024-02-21T23:12:15.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.934+0000 7fb5b1600640 1 -- 172.21.15.28:0/3678847363 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb5a4004bc0 con 0x7fb598003fb0 2024-02-21T23:12:15.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.935+0000 7fb5b1600640 1 --2- 172.21.15.28:0/3678847363 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb59406b4d0 0x7fb59406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:15.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.935+0000 7fb5b1600640 1 -- 172.21.15.28:0/3678847363 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 5835+0+0 (secure 0 0 0) 0x7fb5a40897c0 con 0x7fb598003fb0 2024-02-21T23:12:15.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.935+0000 7fb5b8e00640 1 --2- 172.21.15.28:0/3678847363 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb59406b4d0 0x7fb59406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:15.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.936+0000 7fb5b8e00640 1 --2- 172.21.15.28:0/3678847363 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb59406b4d0 0x7fb59406d990 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fb5a0036000 tx=0x7fb5a0030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:15.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:15.939+0000 7fb5b1600640 1 -- 172.21.15.28:0/3678847363 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb5a40585b0 con 0x7fb598003fb0 2024-02-21T23:12:16.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.138+0000 7fb5bac00640 1 -- 172.21.15.28:0/3678847363 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb5b4072cb0 con 0x7fb598003fb0 2024-02-21T23:12:16.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.675+0000 7fb5b1600640 1 -- 172.21.15.28:0/3678847363 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v189) v1 ==== 169+0+0 (secure 0 0 0) 0x7fb5a4057eb0 con 0x7fb598003fb0 2024-02-21T23:12:16.667 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' removed 2024-02-21T23:12:16.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.685+0000 7fb5bac00640 1 -- 172.21.15.28:0/3678847363 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb59406b4d0 msgr2=0x7fb59406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:16.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.685+0000 7fb5bac00640 1 --2- 172.21.15.28:0/3678847363 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb59406b4d0 0x7fb59406d990 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fb5a0036000 tx=0x7fb5a0030000 comp rx=0 tx=0).stop 2024-02-21T23:12:16.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.685+0000 7fb5bac00640 1 -- 172.21.15.28:0/3678847363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb598003fb0 msgr2=0x7fb5b40c5de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:16.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.685+0000 7fb5bac00640 1 --2- 172.21.15.28:0/3678847363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb598003fb0 0x7fb5b40c5de0 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 crypto rx=0x7fb5a400c7a0 tx=0x7fb5a400cc60 comp rx=0 tx=0).stop 2024-02-21T23:12:16.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.686+0000 7fb5bac00640 1 -- 172.21.15.28:0/3678847363 shutdown_connections 2024-02-21T23:12:16.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.686+0000 7fb5bac00640 1 --2- 172.21.15.28:0/3678847363 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb59406b4d0 0x7fb59406d990 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:16.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.686+0000 7fb5bac00640 1 --2- 172.21.15.28:0/3678847363 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb5b4147b20 0x7fb5b41ea9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:16.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.686+0000 7fb5bac00640 1 --2- 172.21.15.28:0/3678847363 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5b41420b0 0x7fb5b41475e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:16.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.686+0000 7fb5bac00640 1 --2- 172.21.15.28:0/3678847363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb598003fb0 0x7fb5b40c5de0 unknown :-1 s=CLOSED pgs=2186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:16.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.686+0000 7fb5bac00640 1 -- 172.21.15.28:0/3678847363 >> 172.21.15.28:0/3678847363 conn(0x7fb5b40c6610 msgr2=0x7fb5b406eb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:16.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.688+0000 7fb5bac00640 1 -- 172.21.15.28:0/3678847363 shutdown_connections 2024-02-21T23:12:16.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.688+0000 7fb5bac00640 1 -- 172.21.15.28:0/3678847363 wait complete. 2024-02-21T23:12:16.703 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-02-21T23:12:16.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.908+0000 7f2d5ee00640 1 Processor -- start 2024-02-21T23:12:16.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.908+0000 7f2d5ee00640 1 -- start start 2024-02-21T23:12:16.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.909+0000 7f2d5ee00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2d581500f0 0x7f2d5806ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:16.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.909+0000 7f2d5ee00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f2d5806b3b0 0x7f2d5806e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:16.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.909+0000 7f2d5ee00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d5806db10 con 0x7f2d580c2c60 2024-02-21T23:12:16.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.909+0000 7f2d5ee00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d580c1080 con 0x7f2d581500f0 2024-02-21T23:12:16.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.909+0000 7f2d5ee00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d581539a0 con 0x7f2d5806b3b0 2024-02-21T23:12:16.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.910+0000 7f2d5da00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2d581500f0 0x7f2d5806ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:16.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.910+0000 7f2d5da00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2d581500f0 0x7f2d5806ae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41520/0 (socket says 172.21.15.28:41520) 2024-02-21T23:12:16.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.910+0000 7f2d5da00640 1 -- 172.21.15.28:0/861098435 learned_addr learned my addr 172.21.15.28:0/861098435 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:16.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.910+0000 7f2d5da00640 1 -- 172.21.15.28:0/861098435 >> v2:172.21.15.81:3300/0 conn(0x7f2d5806b3b0 msgr2=0x7f2d5806e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:16.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.910+0000 7f2d5da00640 1 --2- 172.21.15.28:0/861098435 >> v2:172.21.15.81:3300/0 conn(0x7f2d5806b3b0 0x7f2d5806e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:16.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.910+0000 7f2d5da00640 1 -- 172.21.15.28:0/861098435 >> v1:172.21.15.81:6789/0 conn(0x7f2d580c2c60 legacy=0x7f2d5806dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:16.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.910+0000 7f2d5da00640 1 -- 172.21.15.28:0/861098435 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d580c12d0 con 0x7f2d581500f0 2024-02-21T23:12:16.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.911+0000 7f2d5da00640 1 --2- 172.21.15.28:0/861098435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d581500f0 0x7f2d5806ae70 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 crypto rx=0x7f2d5806dce0 tx=0x7f2d4c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab40c31da4933be1 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:16.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.912+0000 7f2d57e00640 1 -- 172.21.15.28:0/861098435 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d4c02ca80 con 0x7f2d581500f0 2024-02-21T23:12:16.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.912+0000 7f2d57e00640 1 -- 172.21.15.28:0/861098435 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d4c02cc20 con 0x7f2d581500f0 2024-02-21T23:12:16.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.912+0000 7f2d57e00640 1 -- 172.21.15.28:0/861098435 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d4c0343f0 con 0x7f2d581500f0 2024-02-21T23:12:16.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.912+0000 7f2d5ee00640 1 -- 172.21.15.28:0/861098435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d581500f0 msgr2=0x7f2d5806ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:16.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.912+0000 7f2d5ee00640 1 --2- 172.21.15.28:0/861098435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d581500f0 0x7f2d5806ae70 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 crypto rx=0x7f2d5806dce0 tx=0x7f2d4c02bee0 comp rx=0 tx=0).stop 2024-02-21T23:12:16.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.912+0000 7f2d5ee00640 1 -- 172.21.15.28:0/861098435 shutdown_connections 2024-02-21T23:12:16.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.912+0000 7f2d5ee00640 1 --2- 172.21.15.28:0/861098435 >> v2:172.21.15.81:3300/0 conn(0x7f2d5806b3b0 0x7f2d5806e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:16.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.912+0000 7f2d5ee00640 1 --2- 172.21.15.28:0/861098435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d581500f0 0x7f2d5806ae70 unknown :-1 s=CLOSED pgs=2187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:16.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.912+0000 7f2d5ee00640 1 -- 172.21.15.28:0/861098435 >> 172.21.15.28:0/861098435 conn(0x7f2d580a8530 msgr2=0x7f2d580a8910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:16.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.912+0000 7f2d5ee00640 1 -- 172.21.15.28:0/861098435 shutdown_connections 2024-02-21T23:12:16.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.913+0000 7f2d5ee00640 1 -- 172.21.15.28:0/861098435 wait complete. 2024-02-21T23:12:16.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.914+0000 7f2d5ee00640 1 Processor -- start 2024-02-21T23:12:16.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.914+0000 7f2d5ee00640 1 -- start start 2024-02-21T23:12:16.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.915+0000 7f2d5ee00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2d5806b3b0 0x7f2d580711f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:16.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.915+0000 7f2d5ee00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2d580c2c60 0x7f2d580bfbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:16.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.915+0000 7f2d5ee00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d581500f0 0x7f2d581453d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:16.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.915+0000 7f2d5ee00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d581535e0 con 0x7f2d581500f0 2024-02-21T23:12:16.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.915+0000 7f2d5ee00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d581504d0 con 0x7f2d580c2c60 2024-02-21T23:12:16.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.915+0000 7f2d5ee00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d58153760 con 0x7f2d5806b3b0 2024-02-21T23:12:16.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.916+0000 7f2d5e400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d581500f0 0x7f2d581453d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:16.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.916+0000 7f2d5e400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d581500f0 0x7f2d581453d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41524/0 (socket says 172.21.15.28:41524) 2024-02-21T23:12:16.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.916+0000 7f2d5e400640 1 -- 172.21.15.28:0/4098694241 learned_addr learned my addr 172.21.15.28:0/4098694241 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:16.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.916+0000 7f2d5da00640 1 --2- 172.21.15.28:0/4098694241 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2d5806b3b0 0x7f2d580711f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:16.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.916+0000 7f2d5d000640 1 --2- 172.21.15.28:0/4098694241 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2d580c2c60 0x7f2d580bfbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:16.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.916+0000 7f2d5e400640 1 -- 172.21.15.28:0/4098694241 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2d5806b3b0 msgr2=0x7f2d580711f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:16.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.916+0000 7f2d5e400640 1 --2- 172.21.15.28:0/4098694241 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2d5806b3b0 0x7f2d580711f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:16.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.916+0000 7f2d5e400640 1 -- 172.21.15.28:0/4098694241 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2d580c2c60 msgr2=0x7f2d580bfbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:16.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.916+0000 7f2d5e400640 1 --2- 172.21.15.28:0/4098694241 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2d580c2c60 0x7f2d580bfbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:16.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.916+0000 7f2d5e400640 1 -- 172.21.15.28:0/4098694241 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d58152a20 con 0x7f2d581500f0 2024-02-21T23:12:16.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.917+0000 7f2d5e400640 1 --2- 172.21.15.28:0/4098694241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d581500f0 0x7f2d581453d0 secure :-1 s=READY pgs=2188 cs=0 l=1 rev1=1 crypto rx=0x7f2d50009fd0 tx=0x7f2d5000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:16.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.917+0000 7f2d55600640 1 -- 172.21.15.28:0/4098694241 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d50011830 con 0x7f2d581500f0 2024-02-21T23:12:16.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.917+0000 7f2d55600640 1 -- 172.21.15.28:0/4098694241 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d50016070 con 0x7f2d581500f0 2024-02-21T23:12:16.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.917+0000 7f2d55600640 1 -- 172.21.15.28:0/4098694241 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d50011bf0 con 0x7f2d581500f0 2024-02-21T23:12:16.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.917+0000 7f2d5ee00640 1 -- 172.21.15.28:0/4098694241 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d580a61a0 con 0x7f2d581500f0 2024-02-21T23:12:16.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.917+0000 7f2d5ee00640 1 -- 172.21.15.28:0/4098694241 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d58072b70 con 0x7f2d581500f0 2024-02-21T23:12:16.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.918+0000 7f2d5ee00640 1 -- 172.21.15.28:0/4098694241 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d5804f290 con 0x7f2d581500f0 2024-02-21T23:12:16.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.919+0000 7f2d55600640 1 -- 172.21.15.28:0/4098694241 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2d5001b460 con 0x7f2d581500f0 2024-02-21T23:12:16.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.920+0000 7f2d55600640 1 --2- 172.21.15.28:0/4098694241 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2d3806b4d0 0x7f2d3806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:16.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.920+0000 7f2d55600640 1 -- 172.21.15.28:0/4098694241 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f2d50089e20 con 0x7f2d581500f0 2024-02-21T23:12:16.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.921+0000 7f2d5da00640 1 --2- 172.21.15.28:0/4098694241 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2d3806b4d0 0x7f2d3806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:16.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.922+0000 7f2d5da00640 1 --2- 172.21.15.28:0/4098694241 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2d3806b4d0 0x7f2d3806d990 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f2d4c035000 tx=0x7f2d4c030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:16.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:16.925+0000 7f2d55600640 1 -- 172.21.15.28:0/4098694241 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f2d50058db0 con 0x7f2d581500f0 2024-02-21T23:12:17.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.128+0000 7f2d5ee00640 1 -- 172.21.15.28:0/4098694241 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f2d5806f1e0 con 0x7f2d581500f0 2024-02-21T23:12:17.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.135+0000 7f2d55600640 1 -- 172.21.15.28:0/4098694241 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v60) v1 ==== 104+0+0 (secure 0 0 0) 0x7f2d500586b0 con 0x7f2d581500f0 2024-02-21T23:12:17.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.138+0000 7f2d5ee00640 1 -- 172.21.15.28:0/4098694241 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2d3806b4d0 msgr2=0x7f2d3806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:17.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.138+0000 7f2d5ee00640 1 --2- 172.21.15.28:0/4098694241 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2d3806b4d0 0x7f2d3806d990 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f2d4c035000 tx=0x7f2d4c030000 comp rx=0 tx=0).stop 2024-02-21T23:12:17.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.138+0000 7f2d5ee00640 1 -- 172.21.15.28:0/4098694241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d581500f0 msgr2=0x7f2d581453d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:17.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.138+0000 7f2d5ee00640 1 --2- 172.21.15.28:0/4098694241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d581500f0 0x7f2d581453d0 secure :-1 s=READY pgs=2188 cs=0 l=1 rev1=1 crypto rx=0x7f2d50009fd0 tx=0x7f2d5000b8e0 comp rx=0 tx=0).stop 2024-02-21T23:12:17.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.138+0000 7f2d5ee00640 1 -- 172.21.15.28:0/4098694241 shutdown_connections 2024-02-21T23:12:17.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.138+0000 7f2d5ee00640 1 --2- 172.21.15.28:0/4098694241 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2d3806b4d0 0x7f2d3806d990 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:17.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.138+0000 7f2d5ee00640 1 --2- 172.21.15.28:0/4098694241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d581500f0 0x7f2d581453d0 unknown :-1 s=CLOSED pgs=2188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:17.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.138+0000 7f2d5ee00640 1 --2- 172.21.15.28:0/4098694241 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2d580c2c60 0x7f2d580bfbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:17.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.138+0000 7f2d5ee00640 1 --2- 172.21.15.28:0/4098694241 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2d5806b3b0 0x7f2d580711f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:17.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.138+0000 7f2d5ee00640 1 -- 172.21.15.28:0/4098694241 >> 172.21.15.28:0/4098694241 conn(0x7f2d580a8530 msgr2=0x7f2d5814b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:17.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.138+0000 7f2d5ee00640 1 -- 172.21.15.28:0/4098694241 shutdown_connections 2024-02-21T23:12:17.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.139+0000 7f2d5ee00640 1 -- 172.21.15.28:0/4098694241 wait complete. 2024-02-21T23:12:17.146 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestClusterResize.test_down_grow' 2024-02-21T23:12:17.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.371+0000 7facfe600640 1 Processor -- start 2024-02-21T23:12:17.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.371+0000 7facfe600640 1 -- start start 2024-02-21T23:12:17.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.372+0000 7facfe600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7facf80a6160 0x7facf8155180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:17.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.373+0000 7facfe600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facf8155760 con 0x7facf81521c0 2024-02-21T23:12:17.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.373+0000 7facfe600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facf8155a40 con 0x7facf814f0b0 2024-02-21T23:12:17.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.373+0000 7facfe600640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facf8151d30 con 0x7facf80a6160 2024-02-21T23:12:17.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.373+0000 7facfdc00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7facf81521c0 0x7facf815c510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39666/0 (socket says 172.21.15.28:39666) 2024-02-21T23:12:17.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.373+0000 7facfdc00640 1 -- 172.21.15.28:0/4293574462 learned_addr learned my addr 172.21.15.28:0/4293574462 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:12:17.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.373+0000 7facf7e00640 1 --2- 172.21.15.28:0/4293574462 >> v2:172.21.15.81:3300/0 conn(0x7facf80a6160 0x7facf8155180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:17.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.373+0000 7facf7400640 1 -- 172.21.15.28:0/4293574462 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1583347459 0 0) 0x7facf8155760 con 0x7facf81521c0 2024-02-21T23:12:17.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.374+0000 7facf7400640 1 -- 172.21.15.28:0/4293574462 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facdc003520 con 0x7facf81521c0 2024-02-21T23:12:17.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.374+0000 7facf7400640 1 -- 172.21.15.28:0/4293574462 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1144753552 0 0) 0x7facf8155a40 con 0x7facf814f0b0 2024-02-21T23:12:17.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.374+0000 7facf7400640 1 -- 172.21.15.28:0/4293574462 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facf8155760 con 0x7facf814f0b0 2024-02-21T23:12:17.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.374+0000 7facf7400640 1 -- 172.21.15.28:0/4293574462 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2001684569 0 0) 0x7facdc003520 con 0x7facf81521c0 2024-02-21T23:12:17.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.374+0000 7facf7400640 1 -- 172.21.15.28:0/4293574462 >> v2:172.21.15.81:3300/0 conn(0x7facf80a6160 msgr2=0x7facf8155180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:17.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.374+0000 7facf7400640 1 --2- 172.21.15.28:0/4293574462 >> v2:172.21.15.81:3300/0 conn(0x7facf80a6160 0x7facf8155180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:17.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.374+0000 7facf7400640 1 -- 172.21.15.28:0/4293574462 >> v1:172.21.15.81:6789/0 conn(0x7facf814f0b0 legacy=0x7facf80c2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:17.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.374+0000 7facf7400640 1 -- 172.21.15.28:0/4293574462 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facf8155d20 con 0x7facf81521c0 2024-02-21T23:12:17.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.374+0000 7facf7400640 1 -- 172.21.15.28:0/4293574462 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7facf00019f0 con 0x7facf81521c0 2024-02-21T23:12:17.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.374+0000 7facf7400640 1 -- 172.21.15.28:0/4293574462 >> v1:172.21.15.28:6789/0 conn(0x7facf81521c0 legacy=0x7facf815c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:17.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.374+0000 7facf7400640 1 --2- 172.21.15.28:0/4293574462 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7facdc003b90 0x7facdc003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:17.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.374+0000 7facf7400640 1 --2- 172.21.15.28:0/4293574462 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7facdc004800 0x7facdc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:17.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.374+0000 7facf7400640 1 --2- 172.21.15.28:0/4293574462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facdc0051c0 0x7facdc00dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:17.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.374+0000 7facf7400640 1 -- 172.21.15.28:0/4293574462 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facf8151d30 con 0x7facdc0051c0 2024-02-21T23:12:17.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.374+0000 7facf7400640 1 -- 172.21.15.28:0/4293574462 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facdc00e2a0 con 0x7facdc004800 2024-02-21T23:12:17.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.374+0000 7facf7400640 1 -- 172.21.15.28:0/4293574462 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facdc00e4a0 con 0x7facdc003b90 2024-02-21T23:12:17.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.375+0000 7facfdc00640 1 --2- 172.21.15.28:0/4293574462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facdc0051c0 0x7facdc00dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:17.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.375+0000 7facfd200640 1 --2- 172.21.15.28:0/4293574462 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7facdc003b90 0x7facdc003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:17.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.375+0000 7facf7e00640 1 --2- 172.21.15.28:0/4293574462 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7facdc004800 0x7facdc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:17.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.375+0000 7facfdc00640 1 -- 172.21.15.28:0/4293574462 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7facdc003b90 msgr2=0x7facdc003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:17.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.375+0000 7facfdc00640 1 --2- 172.21.15.28:0/4293574462 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7facdc003b90 0x7facdc003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:17.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.375+0000 7facfdc00640 1 -- 172.21.15.28:0/4293574462 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7facdc004800 msgr2=0x7facdc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:17.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.375+0000 7facfdc00640 1 --2- 172.21.15.28:0/4293574462 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7facdc004800 0x7facdc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:17.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.375+0000 7facfdc00640 1 -- 172.21.15.28:0/4293574462 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facdc003520 con 0x7facdc0051c0 2024-02-21T23:12:17.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.376+0000 7facfdc00640 1 -- 172.21.15.28:0/4293574462 reap_dead start 2024-02-21T23:12:17.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.376+0000 7facfdc00640 1 --2- 172.21.15.28:0/4293574462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facdc0051c0 0x7facdc00dbd0 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 crypto rx=0x7facf000a730 tx=0x7facdc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b8740b58b36e909 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:17.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.376+0000 7facf7400640 1 -- 172.21.15.28:0/4293574462 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facf000a350 con 0x7facdc0051c0 2024-02-21T23:12:17.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.376+0000 7facf7400640 1 -- 172.21.15.28:0/4293574462 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facf0034650 con 0x7facdc0051c0 2024-02-21T23:12:17.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.376+0000 7facf7400640 1 -- 172.21.15.28:0/4293574462 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facf0034a40 con 0x7facdc0051c0 2024-02-21T23:12:17.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.377+0000 7facfe600640 1 -- 172.21.15.28:0/4293574462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facdc0051c0 msgr2=0x7facdc00dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:17.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.377+0000 7facfe600640 1 --2- 172.21.15.28:0/4293574462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facdc0051c0 0x7facdc00dbd0 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 crypto rx=0x7facf000a730 tx=0x7facdc005830 comp rx=0 tx=0).stop 2024-02-21T23:12:17.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.377+0000 7facfe600640 1 -- 172.21.15.28:0/4293574462 shutdown_connections 2024-02-21T23:12:17.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.377+0000 7facfe600640 1 --2- 172.21.15.28:0/4293574462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facdc0051c0 0x7facdc00dbd0 unknown :-1 s=CLOSED pgs=2190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:17.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.377+0000 7facfe600640 1 -- 172.21.15.28:0/4293574462 >> 172.21.15.28:0/4293574462 conn(0x7facf806b630 msgr2=0x7facf80c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:17.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.377+0000 7facfe600640 1 -- 172.21.15.28:0/4293574462 shutdown_connections 2024-02-21T23:12:17.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.377+0000 7facfe600640 1 -- 172.21.15.28:0/4293574462 wait complete. 2024-02-21T23:12:17.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.379+0000 7facfe600640 1 Processor -- start 2024-02-21T23:12:17.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.379+0000 7facfe600640 1 -- start start 2024-02-21T23:12:17.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.380+0000 7facfe600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7facdc0051c0 0x7facf80c1480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:17.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.380+0000 7facfe600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7facf80a95f0 0x7facf80a9a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:17.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.380+0000 7facfe600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facf8153bb0 0x7facf8154030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:17.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.381+0000 7facfe600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facf80a7f70 con 0x7facf8153bb0 2024-02-21T23:12:17.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.381+0000 7facfe600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facf80a68d0 con 0x7facdc0051c0 2024-02-21T23:12:17.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.381+0000 7facfe600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facf80a80f0 con 0x7facf80a95f0 2024-02-21T23:12:17.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.381+0000 7facfdc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facf8153bb0 0x7facf8154030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:17.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.381+0000 7facfdc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facf8153bb0 0x7facf8154030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41554/0 (socket says 172.21.15.28:41554) 2024-02-21T23:12:17.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.381+0000 7facfdc00640 1 -- 172.21.15.28:0/1418654730 learned_addr learned my addr 172.21.15.28:0/1418654730 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:17.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.381+0000 7facf7e00640 1 --2- 172.21.15.28:0/1418654730 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7facf80a95f0 0x7facf80a9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:17.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.381+0000 7facfd200640 1 --2- 172.21.15.28:0/1418654730 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7facdc0051c0 0x7facf80c1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:17.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.381+0000 7facfdc00640 1 -- 172.21.15.28:0/1418654730 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7facf80a95f0 msgr2=0x7facf80a9a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:17.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.381+0000 7facfdc00640 1 --2- 172.21.15.28:0/1418654730 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7facf80a95f0 0x7facf80a9a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:17.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.381+0000 7facfdc00640 1 -- 172.21.15.28:0/1418654730 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7facdc0051c0 msgr2=0x7facf80c1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:17.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.381+0000 7facfdc00640 1 --2- 172.21.15.28:0/1418654730 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7facdc0051c0 0x7facf80c1480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:17.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.381+0000 7facfdc00640 1 -- 172.21.15.28:0/1418654730 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facf8154570 con 0x7facf8153bb0 2024-02-21T23:12:17.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.381+0000 7facfd200640 1 --2- 172.21.15.28:0/1418654730 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7facdc0051c0 0x7facf80c1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:12:17.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.382+0000 7facfdc00640 1 --2- 172.21.15.28:0/1418654730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facf8153bb0 0x7facf8154030 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 crypto rx=0x7facf002cb10 tx=0x7facf8155930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:17.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.382+0000 7facf4c00640 1 -- 172.21.15.28:0/1418654730 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facf0039040 con 0x7facf8153bb0 2024-02-21T23:12:17.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.382+0000 7facf4c00640 1 -- 172.21.15.28:0/1418654730 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facf0043070 con 0x7facf8153bb0 2024-02-21T23:12:17.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.382+0000 7facf4c00640 1 -- 172.21.15.28:0/1418654730 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facf00019f0 con 0x7facf8153bb0 2024-02-21T23:12:17.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.382+0000 7facfe600640 1 -- 172.21.15.28:0/1418654730 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facf80c19c0 con 0x7facf8153bb0 2024-02-21T23:12:17.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.382+0000 7facfe600640 1 -- 172.21.15.28:0/1418654730 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facf8152df0 con 0x7facf8153bb0 2024-02-21T23:12:17.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.383+0000 7facfe600640 1 -- 172.21.15.28:0/1418654730 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facf8153890 con 0x7facf8153bb0 2024-02-21T23:12:17.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.385+0000 7facf4c00640 1 -- 172.21.15.28:0/1418654730 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7facf0034650 con 0x7facf8153bb0 2024-02-21T23:12:17.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.386+0000 7facf4c00640 1 --2- 172.21.15.28:0/1418654730 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7facd806b4d0 0x7facd806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:17.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.386+0000 7facf4c00640 1 -- 172.21.15.28:0/1418654730 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5421+0+0 (secure 0 0 0) 0x7facf00ac090 con 0x7facf8153bb0 2024-02-21T23:12:17.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.387+0000 7facfd200640 1 --2- 172.21.15.28:0/1418654730 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7facd806b4d0 0x7facd806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:17.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.388+0000 7facfd200640 1 --2- 172.21.15.28:0/1418654730 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7facd806b4d0 0x7facd806d990 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7face8007850 tx=0x7face8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:17.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.391+0000 7facf4c00640 1 -- 172.21.15.28:0/1418654730 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7facf8153890 con 0x7facf8153bb0 2024-02-21T23:12:17.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.667+0000 7facfe600640 1 -- 172.21.15.28:0/1418654730 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterResize.test_down_grow"]} v 0) v1 -- 0x7facf804f290 con 0x7facf8153bb0 2024-02-21T23:12:17.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.697+0000 7facf4c00640 1 -- 172.21.15.28:0/1418654730 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterResize.test_down_grow"]}]=0 v550) v1 ==== 138+0+0 (secure 0 0 0) 0x7facf007af90 con 0x7facf8153bb0 2024-02-21T23:12:17.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.701+0000 7facfe600640 1 -- 172.21.15.28:0/1418654730 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7facd806b4d0 msgr2=0x7facd806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:17.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.701+0000 7facfe600640 1 --2- 172.21.15.28:0/1418654730 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7facd806b4d0 0x7facd806d990 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7face8007850 tx=0x7face8009040 comp rx=0 tx=0).stop 2024-02-21T23:12:17.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.701+0000 7facfe600640 1 -- 172.21.15.28:0/1418654730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facf8153bb0 msgr2=0x7facf8154030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:17.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.701+0000 7facfe600640 1 --2- 172.21.15.28:0/1418654730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facf8153bb0 0x7facf8154030 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 crypto rx=0x7facf002cb10 tx=0x7facf8155930 comp rx=0 tx=0).stop 2024-02-21T23:12:17.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.701+0000 7facfe600640 1 -- 172.21.15.28:0/1418654730 shutdown_connections 2024-02-21T23:12:17.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.701+0000 7facfe600640 1 --2- 172.21.15.28:0/1418654730 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7facd806b4d0 0x7facd806d990 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:17.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.701+0000 7facfe600640 1 --2- 172.21.15.28:0/1418654730 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7facf80a95f0 0x7facf80a9a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:17.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.701+0000 7facfe600640 1 --2- 172.21.15.28:0/1418654730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facf8153bb0 0x7facf8154030 unknown :-1 s=CLOSED pgs=2191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:17.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.701+0000 7facfe600640 1 --2- 172.21.15.28:0/1418654730 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7facdc0051c0 0x7facf80c1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:17.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.701+0000 7facfe600640 1 -- 172.21.15.28:0/1418654730 >> 172.21.15.28:0/1418654730 conn(0x7facf806b630 msgr2=0x7facf8071cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:17.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.701+0000 7facfe600640 1 -- 172.21.15.28:0/1418654730 shutdown_connections 2024-02-21T23:12:17.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.701+0000 7facfe600640 1 -- 172.21.15.28:0/1418654730 wait complete. 2024-02-21T23:12:17.706 INFO:tasks.cephfs_test_runner:That setting max_mds undoes down. ... ok 2024-02-21T23:12:17.707 INFO:tasks.cephfs_test_runner:Starting test: test_down_health (tasks.cephfs.test_failover.TestClusterResize) That marking a FS down does not generate a health warning 2024-02-21T23:12:17.707 INFO:tasks.cephfs_test_runner:test_down_health (tasks.cephfs.test_failover.TestClusterResize) 2024-02-21T23:12:17.707 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:12:17.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.935+0000 7fcf44a00640 1 Processor -- start 2024-02-21T23:12:17.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.936+0000 7fcf44a00640 1 -- start start 2024-02-21T23:12:17.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.937+0000 7fcf44a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcf400aff90 0x7fcf400c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:17.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.937+0000 7fcf44a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcf400c5720 0x7fcf4015f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:17.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.937+0000 7fcf44a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf400c74c0 con 0x7fcf4006dbf0 2024-02-21T23:12:17.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.937+0000 7fcf44a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf400c7270 con 0x7fcf400aff90 2024-02-21T23:12:17.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.937+0000 7fcf44a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf40158630 con 0x7fcf400c5720 2024-02-21T23:12:17.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.937+0000 7fcf3e800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcf400aff90 0x7fcf400c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:17.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.937+0000 7fcf3e800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcf400aff90 0x7fcf400c51e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41564/0 (socket says 172.21.15.28:41564) 2024-02-21T23:12:17.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.937+0000 7fcf3e800640 1 -- 172.21.15.28:0/642240331 learned_addr learned my addr 172.21.15.28:0/642240331 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:17.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.938+0000 7fcf3e800640 1 -- 172.21.15.28:0/642240331 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcf400c5720 msgr2=0x7fcf4015f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:17.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.938+0000 7fcf3f200640 1 --2- 172.21.15.28:0/642240331 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcf400c5720 0x7fcf4015f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:17.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.938+0000 7fcf3e800640 1 --2- 172.21.15.28:0/642240331 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcf400c5720 0x7fcf4015f890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:17.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.938+0000 7fcf3e800640 1 -- 172.21.15.28:0/642240331 >> v1:172.21.15.81:6789/0 conn(0x7fcf4006dbf0 legacy=0x7fcf40158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:17.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.938+0000 7fcf3e800640 1 -- 172.21.15.28:0/642240331 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf400c77a0 con 0x7fcf400aff90 2024-02-21T23:12:17.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.939+0000 7fcf3e800640 1 --2- 172.21.15.28:0/642240331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf400aff90 0x7fcf400c51e0 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7fcf30009e50 tx=0x7fcf3002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1eb66b1ab98a27f6 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:17.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.939+0000 7fcf3d400640 1 -- 172.21.15.28:0/642240331 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf3002ca70 con 0x7fcf400aff90 2024-02-21T23:12:17.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.939+0000 7fcf3d400640 1 -- 172.21.15.28:0/642240331 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf3002cc10 con 0x7fcf400aff90 2024-02-21T23:12:17.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.939+0000 7fcf3d400640 1 -- 172.21.15.28:0/642240331 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf300343f0 con 0x7fcf400aff90 2024-02-21T23:12:17.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.940+0000 7fcf44a00640 1 -- 172.21.15.28:0/642240331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf400aff90 msgr2=0x7fcf400c51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:17.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.940+0000 7fcf44a00640 1 --2- 172.21.15.28:0/642240331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf400aff90 0x7fcf400c51e0 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7fcf30009e50 tx=0x7fcf3002bed0 comp rx=0 tx=0).stop 2024-02-21T23:12:17.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.940+0000 7fcf44a00640 1 -- 172.21.15.28:0/642240331 shutdown_connections 2024-02-21T23:12:17.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.940+0000 7fcf44a00640 1 --2- 172.21.15.28:0/642240331 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcf400c5720 0x7fcf4015f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:17.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.940+0000 7fcf44a00640 1 --2- 172.21.15.28:0/642240331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf400aff90 0x7fcf400c51e0 unknown :-1 s=CLOSED pgs=2192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:17.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.940+0000 7fcf44a00640 1 -- 172.21.15.28:0/642240331 >> 172.21.15.28:0/642240331 conn(0x7fcf400ac370 msgr2=0x7fcf400ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:17.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.940+0000 7fcf44a00640 1 -- 172.21.15.28:0/642240331 shutdown_connections 2024-02-21T23:12:17.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.940+0000 7fcf44a00640 1 -- 172.21.15.28:0/642240331 wait complete. 2024-02-21T23:12:17.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.942+0000 7fcf44a00640 1 Processor -- start 2024-02-21T23:12:17.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.942+0000 7fcf44a00640 1 -- start start 2024-02-21T23:12:17.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.943+0000 7fcf44a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf4006dbf0 0x7fcf4014bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:17.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.943+0000 7fcf44a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf400aff90 0x7fcf401ed570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:17.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.943+0000 7fcf44a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcf400c5720 0x7fcf40154100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:17.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.943+0000 7fcf44a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf400c8760 con 0x7fcf4006dbf0 2024-02-21T23:12:17.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.943+0000 7fcf44a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf400be0c0 con 0x7fcf400aff90 2024-02-21T23:12:17.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.943+0000 7fcf44a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf400c88e0 con 0x7fcf400c5720 2024-02-21T23:12:17.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.943+0000 7fcf3e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf4006dbf0 0x7fcf4014bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:17.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.943+0000 7fcf3e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf4006dbf0 0x7fcf4014bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41566/0 (socket says 172.21.15.28:41566) 2024-02-21T23:12:17.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.943+0000 7fcf3e800640 1 -- 172.21.15.28:0/3598853107 learned_addr learned my addr 172.21.15.28:0/3598853107 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:17.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.943+0000 7fcf3e800640 1 -- 172.21.15.28:0/3598853107 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcf400c5720 msgr2=0x7fcf40154100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:17.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.943+0000 7fcf3e800640 1 --2- 172.21.15.28:0/3598853107 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcf400c5720 0x7fcf40154100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:17.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.943+0000 7fcf3e800640 1 -- 172.21.15.28:0/3598853107 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf400aff90 msgr2=0x7fcf401ed570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:17.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.943+0000 7fcf3e800640 1 --2- 172.21.15.28:0/3598853107 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf400aff90 0x7fcf401ed570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:17.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.943+0000 7fcf3e800640 1 -- 172.21.15.28:0/3598853107 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf400af6d0 con 0x7fcf4006dbf0 2024-02-21T23:12:17.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.944+0000 7fcf3e800640 1 --2- 172.21.15.28:0/3598853107 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf4006dbf0 0x7fcf4014bda0 secure :-1 s=READY pgs=2193 cs=0 l=1 rev1=1 crypto rx=0x7fcf30009e50 tx=0x7fcf30004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:17.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.944+0000 7fcf36a00640 1 -- 172.21.15.28:0/3598853107 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf30043020 con 0x7fcf4006dbf0 2024-02-21T23:12:17.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.944+0000 7fcf36a00640 1 -- 172.21.15.28:0/3598853107 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf3003e450 con 0x7fcf4006dbf0 2024-02-21T23:12:17.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.944+0000 7fcf36a00640 1 -- 172.21.15.28:0/3598853107 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf3003e840 con 0x7fcf4006dbf0 2024-02-21T23:12:17.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.944+0000 7fcf44a00640 1 -- 172.21.15.28:0/3598853107 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf400c16a0 con 0x7fcf4006dbf0 2024-02-21T23:12:17.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.944+0000 7fcf44a00640 1 -- 172.21.15.28:0/3598853107 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf400af130 con 0x7fcf4006dbf0 2024-02-21T23:12:17.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.945+0000 7fcf44a00640 1 -- 172.21.15.28:0/3598853107 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf4004f290 con 0x7fcf4006dbf0 2024-02-21T23:12:17.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.946+0000 7fcf36a00640 1 -- 172.21.15.28:0/3598853107 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcf30046020 con 0x7fcf4006dbf0 2024-02-21T23:12:17.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.947+0000 7fcf36a00640 1 --2- 172.21.15.28:0/3598853107 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcf2006b4d0 0x7fcf2006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:17.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.947+0000 7fcf36a00640 1 -- 172.21.15.28:0/3598853107 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5421+0+0 (secure 0 0 0) 0x7fcf300aca60 con 0x7fcf4006dbf0 2024-02-21T23:12:17.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.948+0000 7fcf3de00640 1 --2- 172.21.15.28:0/3598853107 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcf2006b4d0 0x7fcf2006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:17.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.949+0000 7fcf3de00640 1 --2- 172.21.15.28:0/3598853107 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcf2006b4d0 0x7fcf2006d990 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fcf401575a0 tx=0x7fcf2c0079b0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:17.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:17.952+0000 7fcf36a00640 1 -- 172.21.15.28:0/3598853107 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fcf3004d030 con 0x7fcf4006dbf0 2024-02-21T23:12:18.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.153+0000 7fcf44a00640 1 -- 172.21.15.28:0/3598853107 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcf401ebd40 con 0x7fcf4006dbf0 2024-02-21T23:12:18.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.154+0000 7fcf36a00640 1 -- 172.21.15.28:0/3598853107 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v189) v1 ==== 74+0+13371 (secure 0 0 0) 0x7fcf3007b9f0 con 0x7fcf4006dbf0 2024-02-21T23:12:18.143 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:12:18.143 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":189,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:12:16.672196+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.81:6837/416021079":"2024-02-22T23:12:13.264945+0000","172.21.15.81:6835/416021079":"2024-02-22T23:12:13.264945+0000","172.21.15.81:6836/2895690310":"2024-02-22T23:12:13.264916+0000","172.21.15.81:6834/2895690310":"2024-02-22T23:12:13.264916+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:12:18.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.157+0000 7fcf44a00640 1 -- 172.21.15.28:0/3598853107 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcf2006b4d0 msgr2=0x7fcf2006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:18.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.157+0000 7fcf44a00640 1 --2- 172.21.15.28:0/3598853107 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcf2006b4d0 0x7fcf2006d990 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fcf401575a0 tx=0x7fcf2c0079b0 comp rx=0 tx=0).stop 2024-02-21T23:12:18.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.157+0000 7fcf44a00640 1 -- 172.21.15.28:0/3598853107 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf4006dbf0 msgr2=0x7fcf4014bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:18.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.157+0000 7fcf44a00640 1 --2- 172.21.15.28:0/3598853107 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf4006dbf0 0x7fcf4014bda0 secure :-1 s=READY pgs=2193 cs=0 l=1 rev1=1 crypto rx=0x7fcf30009e50 tx=0x7fcf30004a40 comp rx=0 tx=0).stop 2024-02-21T23:12:18.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.157+0000 7fcf44a00640 1 -- 172.21.15.28:0/3598853107 shutdown_connections 2024-02-21T23:12:18.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.157+0000 7fcf44a00640 1 --2- 172.21.15.28:0/3598853107 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcf2006b4d0 0x7fcf2006d990 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:18.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.157+0000 7fcf44a00640 1 --2- 172.21.15.28:0/3598853107 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcf400c5720 0x7fcf40154100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:18.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.157+0000 7fcf44a00640 1 --2- 172.21.15.28:0/3598853107 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf400aff90 0x7fcf401ed570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:18.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.157+0000 7fcf44a00640 1 --2- 172.21.15.28:0/3598853107 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf4006dbf0 0x7fcf4014bda0 unknown :-1 s=CLOSED pgs=2193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:18.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.157+0000 7fcf44a00640 1 -- 172.21.15.28:0/3598853107 >> 172.21.15.28:0/3598853107 conn(0x7fcf400ac370 msgr2=0x7fcf400ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:18.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.157+0000 7fcf44a00640 1 -- 172.21.15.28:0/3598853107 shutdown_connections 2024-02-21T23:12:18.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.157+0000 7fcf44a00640 1 -- 172.21.15.28:0/3598853107 wait complete. 2024-02-21T23:12:18.164 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-21T23:02:30.728329+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-21T23:12:18.164 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-21T23:12:18.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.403+0000 7f77a8a00640 1 Processor -- start 2024-02-21T23:12:18.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.403+0000 7f77a8a00640 1 -- start start 2024-02-21T23:12:18.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.404+0000 7f77a8a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f77a40c2c10 0x7f77a40c12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:18.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.404+0000 7f77a8a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77a406e220 con 0x7f77a40c4580 2024-02-21T23:12:18.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.404+0000 7f77a8a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77a406f780 con 0x7f77a4150240 2024-02-21T23:12:18.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.404+0000 7f77a8a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77a4150620 con 0x7f77a40c2c10 2024-02-21T23:12:18.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.405+0000 7f77a1e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f77a40c4580 0x7f77a406ae70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39678/0 (socket says 172.21.15.28:39678) 2024-02-21T23:12:18.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.405+0000 7f77a1e00640 1 -- 172.21.15.28:0/2426650573 learned_addr learned my addr 172.21.15.28:0/2426650573 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:12:18.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.405+0000 7f77a1400640 1 -- 172.21.15.28:0/2426650573 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 425437485 0 0) 0x7f77a406e220 con 0x7f77a40c4580 2024-02-21T23:12:18.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.405+0000 7f77a1400640 1 -- 172.21.15.28:0/2426650573 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7788003520 con 0x7f77a40c4580 2024-02-21T23:12:18.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.405+0000 7f77a3200640 1 --2- 172.21.15.28:0/2426650573 >> v2:172.21.15.81:3300/0 conn(0x7f77a40c2c10 0x7f77a40c12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:18.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.406+0000 7f77a1400640 1 -- 172.21.15.28:0/2426650573 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 983975563 0 0) 0x7f7788003520 con 0x7f77a40c4580 2024-02-21T23:12:18.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.406+0000 7f77a1400640 1 -- 172.21.15.28:0/2426650573 >> v2:172.21.15.81:3300/0 conn(0x7f77a40c2c10 msgr2=0x7f77a40c12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:18.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.406+0000 7f77a1400640 1 --2- 172.21.15.28:0/2426650573 >> v2:172.21.15.81:3300/0 conn(0x7f77a40c2c10 0x7f77a40c12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:18.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.406+0000 7f77a1400640 1 -- 172.21.15.28:0/2426650573 >> v1:172.21.15.81:6789/0 conn(0x7f77a4150240 legacy=0x7f77a406db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:18.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.406+0000 7f77a1400640 1 -- 172.21.15.28:0/2426650573 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77a406f4a0 con 0x7f77a40c4580 2024-02-21T23:12:18.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.406+0000 7f77a1400640 1 -- 172.21.15.28:0/2426650573 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f778c001670 con 0x7f77a40c4580 2024-02-21T23:12:18.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.406+0000 7f77a1400640 1 -- 172.21.15.28:0/2426650573 >> v1:172.21.15.28:6789/0 conn(0x7f77a40c4580 legacy=0x7f77a406ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:18.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.406+0000 7f77a1400640 1 --2- 172.21.15.28:0/2426650573 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f77880039b0 0x7f7788003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:18.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.406+0000 7f77a1400640 1 --2- 172.21.15.28:0/2426650573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7788004800 0x7f7788004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:18.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.406+0000 7f77a1400640 1 --2- 172.21.15.28:0/2426650573 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f77880052e0 0x7f778800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:18.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.406+0000 7f77a1400640 1 -- 172.21.15.28:0/2426650573 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f778800c180 con 0x7f7788004800 2024-02-21T23:12:18.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.406+0000 7f77a1400640 1 -- 172.21.15.28:0/2426650573 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f778800c330 con 0x7f77880039b0 2024-02-21T23:12:18.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.406+0000 7f77a1400640 1 -- 172.21.15.28:0/2426650573 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f778800c530 con 0x7f77880052e0 2024-02-21T23:12:18.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.406+0000 7f77a3200640 1 --2- 172.21.15.28:0/2426650573 >> v2:172.21.15.81:3300/0 conn(0x7f77a40c2c10 0x7f77a40c12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:12:18.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.406+0000 7f77a1e00640 1 --2- 172.21.15.28:0/2426650573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7788004800 0x7f7788004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:18.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.407+0000 7f77a1e00640 1 -- 172.21.15.28:0/2426650573 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f77880052e0 msgr2=0x7f778800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:18.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.407+0000 7f77a1e00640 1 --2- 172.21.15.28:0/2426650573 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f77880052e0 0x7f778800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:18.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.407+0000 7f77a1e00640 1 -- 172.21.15.28:0/2426650573 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f77880039b0 msgr2=0x7f7788003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:18.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.407+0000 7f77a1e00640 1 --2- 172.21.15.28:0/2426650573 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f77880039b0 0x7f7788003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:18.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.407+0000 7f77a1e00640 1 -- 172.21.15.28:0/2426650573 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7788003520 con 0x7f7788004800 2024-02-21T23:12:18.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.407+0000 7f77a3200640 1 -- 172.21.15.28:0/2426650573 reap_dead start 2024-02-21T23:12:18.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.408+0000 7f77a1e00640 1 --2- 172.21.15.28:0/2426650573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7788004800 0x7f7788004c80 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7f77a40c49c0 tx=0x7f7788005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf250351b168bcf6 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:18.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.408+0000 7f77a1400640 1 -- 172.21.15.28:0/2426650573 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f778c007d10 con 0x7f7788004800 2024-02-21T23:12:18.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.408+0000 7f77a1400640 1 -- 172.21.15.28:0/2426650573 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f778c030070 con 0x7f7788004800 2024-02-21T23:12:18.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.408+0000 7f77a1400640 1 -- 172.21.15.28:0/2426650573 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f778c005d70 con 0x7f7788004800 2024-02-21T23:12:18.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.409+0000 7f77a8a00640 1 -- 172.21.15.28:0/2426650573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7788004800 msgr2=0x7f7788004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:18.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.409+0000 7f77a8a00640 1 --2- 172.21.15.28:0/2426650573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7788004800 0x7f7788004c80 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7f77a40c49c0 tx=0x7f7788005830 comp rx=0 tx=0).stop 2024-02-21T23:12:18.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.409+0000 7f77a8a00640 1 -- 172.21.15.28:0/2426650573 shutdown_connections 2024-02-21T23:12:18.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.409+0000 7f77a8a00640 1 --2- 172.21.15.28:0/2426650573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7788004800 0x7f7788004c80 unknown :-1 s=CLOSED pgs=2195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:18.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.409+0000 7f77a8a00640 1 -- 172.21.15.28:0/2426650573 >> 172.21.15.28:0/2426650573 conn(0x7f77a40a8530 msgr2=0x7f77a40a8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:18.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.409+0000 7f77a8a00640 1 -- 172.21.15.28:0/2426650573 shutdown_connections 2024-02-21T23:12:18.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.409+0000 7f77a8a00640 1 -- 172.21.15.28:0/2426650573 wait complete. 2024-02-21T23:12:18.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.410+0000 7f77a8a00640 1 Processor -- start 2024-02-21T23:12:18.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.411+0000 7f77a8a00640 1 -- start start 2024-02-21T23:12:18.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.411+0000 7f77a8a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7788004800 0x7f77a4147ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:18.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.411+0000 7f77a8a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77a40bfbb0 0x7f77a40c0030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:18.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.411+0000 7f77a8a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f77a40714c0 0x7f77a4071940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:18.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.411+0000 7f77a8a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77a40c4a10 con 0x7f77a40bfbb0 2024-02-21T23:12:18.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.411+0000 7f77a8a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77a4153a90 con 0x7f77a40714c0 2024-02-21T23:12:18.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.411+0000 7f77a8a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77a40c4b90 con 0x7f7788004800 2024-02-21T23:12:18.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.412+0000 7f77a1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77a40bfbb0 0x7f77a40c0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:18.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.412+0000 7f77a1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77a40bfbb0 0x7f77a40c0030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41570/0 (socket says 172.21.15.28:41570) 2024-02-21T23:12:18.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.412+0000 7f77a1e00640 1 -- 172.21.15.28:0/3339801694 learned_addr learned my addr 172.21.15.28:0/3339801694 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:18.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.412+0000 7f77a2800640 1 --2- 172.21.15.28:0/3339801694 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7788004800 0x7f77a4147ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:18.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.412+0000 7f77a1e00640 1 -- 172.21.15.28:0/3339801694 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7788004800 msgr2=0x7f77a4147ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:18.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.412+0000 7f77a3200640 1 --2- 172.21.15.28:0/3339801694 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f77a40714c0 0x7f77a4071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:18.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.412+0000 7f77a1e00640 1 --2- 172.21.15.28:0/3339801694 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7788004800 0x7f77a4147ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:18.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.412+0000 7f77a1e00640 1 -- 172.21.15.28:0/3339801694 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f77a40714c0 msgr2=0x7f77a4071940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:18.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.412+0000 7f77a1e00640 1 --2- 172.21.15.28:0/3339801694 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f77a40714c0 0x7f77a4071940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:18.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.412+0000 7f77a1e00640 1 -- 172.21.15.28:0/3339801694 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77a41543a0 con 0x7f77a40bfbb0 2024-02-21T23:12:18.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.412+0000 7f77a2800640 1 --2- 172.21.15.28:0/3339801694 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7788004800 0x7f77a4147ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:12:18.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.412+0000 7f77a3200640 1 --2- 172.21.15.28:0/3339801694 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f77a40714c0 0x7f77a4071940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:12:18.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.413+0000 7f77a1e00640 1 --2- 172.21.15.28:0/3339801694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77a40bfbb0 0x7f77a40c0030 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 crypto rx=0x7f778c02c4e0 tx=0x7f778c02e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:18.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.413+0000 7f779aa00640 1 -- 172.21.15.28:0/3339801694 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f778c045070 con 0x7f77a40bfbb0 2024-02-21T23:12:18.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.413+0000 7f779aa00640 1 -- 172.21.15.28:0/3339801694 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f778c004bf0 con 0x7f77a40bfbb0 2024-02-21T23:12:18.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.413+0000 7f779aa00640 1 -- 172.21.15.28:0/3339801694 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f778c007d10 con 0x7f77a40bfbb0 2024-02-21T23:12:18.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.413+0000 7f77a8a00640 1 -- 172.21.15.28:0/3339801694 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77a4154890 con 0x7f77a40bfbb0 2024-02-21T23:12:18.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.413+0000 7f77a8a00640 1 -- 172.21.15.28:0/3339801694 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77a4153e80 con 0x7f77a40bfbb0 2024-02-21T23:12:18.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.414+0000 7f77a8a00640 1 -- 172.21.15.28:0/3339801694 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77a41545e0 con 0x7f77a40bfbb0 2024-02-21T23:12:18.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.415+0000 7f779aa00640 1 -- 172.21.15.28:0/3339801694 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f778c007070 con 0x7f77a40bfbb0 2024-02-21T23:12:18.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.416+0000 7f779aa00640 1 --2- 172.21.15.28:0/3339801694 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f778006b4d0 0x7f778006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:18.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.416+0000 7f779aa00640 1 -- 172.21.15.28:0/3339801694 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f778c0ab0a0 con 0x7f77a40bfbb0 2024-02-21T23:12:18.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.417+0000 7f77a2800640 1 --2- 172.21.15.28:0/3339801694 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f778006b4d0 0x7f778006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:18.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.418+0000 7f77a2800640 1 --2- 172.21.15.28:0/3339801694 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f778006b4d0 0x7f778006d990 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f7794007fd0 tx=0x7f7794009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:18.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.421+0000 7f779aa00640 1 -- 172.21.15.28:0/3339801694 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f778c079960 con 0x7f77a40bfbb0 2024-02-21T23:12:18.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.611+0000 7f77a8a00640 1 -- 172.21.15.28:0/3339801694 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f77a404f290 con 0x7f77a40bfbb0 2024-02-21T23:12:18.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.612+0000 7f779aa00640 1 -- 172.21.15.28:0/3339801694 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v189) v1 ==== 93+0+10 (secure 0 0 0) 0x7f778c07da00 con 0x7f77a40bfbb0 2024-02-21T23:12:18.599 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-02-21T23:12:18.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.614+0000 7f77a8a00640 1 -- 172.21.15.28:0/3339801694 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f778006b4d0 msgr2=0x7f778006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:18.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.614+0000 7f77a8a00640 1 --2- 172.21.15.28:0/3339801694 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f778006b4d0 0x7f778006d990 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f7794007fd0 tx=0x7f7794009040 comp rx=0 tx=0).stop 2024-02-21T23:12:18.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.614+0000 7f77a8a00640 1 -- 172.21.15.28:0/3339801694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77a40bfbb0 msgr2=0x7f77a40c0030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:18.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.614+0000 7f77a8a00640 1 --2- 172.21.15.28:0/3339801694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77a40bfbb0 0x7f77a40c0030 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 crypto rx=0x7f778c02c4e0 tx=0x7f778c02e070 comp rx=0 tx=0).stop 2024-02-21T23:12:18.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.614+0000 7f77a8a00640 1 -- 172.21.15.28:0/3339801694 shutdown_connections 2024-02-21T23:12:18.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.614+0000 7f77a8a00640 1 --2- 172.21.15.28:0/3339801694 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f778006b4d0 0x7f778006d990 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:18.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.614+0000 7f77a8a00640 1 --2- 172.21.15.28:0/3339801694 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f77a40714c0 0x7f77a4071940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:18.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.614+0000 7f77a8a00640 1 --2- 172.21.15.28:0/3339801694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77a40bfbb0 0x7f77a40c0030 unknown :-1 s=CLOSED pgs=2196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:18.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.614+0000 7f77a8a00640 1 --2- 172.21.15.28:0/3339801694 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7788004800 0x7f77a4147ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:18.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.614+0000 7f77a8a00640 1 -- 172.21.15.28:0/3339801694 >> 172.21.15.28:0/3339801694 conn(0x7f77a40a8530 msgr2=0x7f77a4150620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:18.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.615+0000 7f77a8a00640 1 -- 172.21.15.28:0/3339801694 shutdown_connections 2024-02-21T23:12:18.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.615+0000 7f77a8a00640 1 -- 172.21.15.28:0/3339801694 wait complete. 2024-02-21T23:12:18.621 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestClusterResize.test_down_health' 2024-02-21T23:12:18.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.854+0000 7f892ec00640 1 Processor -- start 2024-02-21T23:12:18.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.854+0000 7f892ec00640 1 -- start start 2024-02-21T23:12:18.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.855+0000 7f892ec00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f8928152ab0 0x7f89280c7fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:18.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.856+0000 7f892ec00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8928158850 0x7f8928158cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:18.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.856+0000 7f892ec00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89280a8180 con 0x7f89280c6710 2024-02-21T23:12:18.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.856+0000 7f892ec00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89280c3ce0 con 0x7f8928158850 2024-02-21T23:12:18.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.856+0000 7f892ec00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89280c33b0 con 0x7f8928152ab0 2024-02-21T23:12:18.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.856+0000 7f8927400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f89280c6710 0x7f89280c5210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39686/0 (socket says 172.21.15.28:39686) 2024-02-21T23:12:18.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.856+0000 7f8927400640 1 -- 172.21.15.28:0/3626278673 learned_addr learned my addr 172.21.15.28:0/3626278673 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:12:18.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.856+0000 7f892cc00640 1 --2- 172.21.15.28:0/3626278673 >> v2:172.21.15.28:3300/0 conn(0x7f8928158850 0x7f8928158cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:18.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.856+0000 7f8927e00640 1 --2- 172.21.15.28:0/3626278673 >> v2:172.21.15.81:3300/0 conn(0x7f8928152ab0 0x7f89280c7fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:18.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.856+0000 7f8926a00640 1 -- 172.21.15.28:0/3626278673 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 731070867 0 0) 0x7f8918001490 con 0x7f89280c6710 2024-02-21T23:12:18.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.857+0000 7f8926a00640 1 -- 172.21.15.28:0/3626278673 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f890c003520 con 0x7f89280c6710 2024-02-21T23:12:18.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.857+0000 7f892cc00640 1 -- 172.21.15.28:0/3626278673 >> v2:172.21.15.81:3300/0 conn(0x7f8928152ab0 msgr2=0x7f89280c7fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:18.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.857+0000 7f892cc00640 1 --2- 172.21.15.28:0/3626278673 >> v2:172.21.15.81:3300/0 conn(0x7f8928152ab0 0x7f89280c7fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:18.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.857+0000 7f892cc00640 1 -- 172.21.15.28:0/3626278673 >> v1:172.21.15.28:6789/0 conn(0x7f89280c6710 legacy=0x7f89280c5210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:18.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.857+0000 7f892cc00640 1 -- 172.21.15.28:0/3626278673 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89280c3690 con 0x7f8928158850 2024-02-21T23:12:18.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.858+0000 7f892cc00640 1 --2- 172.21.15.28:0/3626278673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8928158850 0x7f8928158cc0 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 crypto rx=0x7f89280c9850 tx=0x7f892002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ab39e6ea1fffcb9 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:18.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.858+0000 7f8926a00640 1 -- 172.21.15.28:0/3626278673 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8920035020 con 0x7f8928158850 2024-02-21T23:12:18.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.858+0000 7f8926a00640 1 -- 172.21.15.28:0/3626278673 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8920031830 con 0x7f8928158850 2024-02-21T23:12:18.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.858+0000 7f8926a00640 1 -- 172.21.15.28:0/3626278673 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8920031c00 con 0x7f8928158850 2024-02-21T23:12:18.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.859+0000 7f892ec00640 1 -- 172.21.15.28:0/3626278673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8928158850 msgr2=0x7f8928158cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:18.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.859+0000 7f892ec00640 1 --2- 172.21.15.28:0/3626278673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8928158850 0x7f8928158cc0 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 crypto rx=0x7f89280c9850 tx=0x7f892002b8d0 comp rx=0 tx=0).stop 2024-02-21T23:12:18.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.859+0000 7f892ec00640 1 -- 172.21.15.28:0/3626278673 shutdown_connections 2024-02-21T23:12:18.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.859+0000 7f892ec00640 1 --2- 172.21.15.28:0/3626278673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8928158850 0x7f8928158cc0 unknown :-1 s=CLOSED pgs=2198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:18.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.859+0000 7f892ec00640 1 --2- 172.21.15.28:0/3626278673 >> v2:172.21.15.81:3300/0 conn(0x7f8928152ab0 0x7f89280c7fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:18.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.859+0000 7f892ec00640 1 -- 172.21.15.28:0/3626278673 >> 172.21.15.28:0/3626278673 conn(0x7f89280c8810 msgr2=0x7f89280c8bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:18.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.859+0000 7f892ec00640 1 -- 172.21.15.28:0/3626278673 shutdown_connections 2024-02-21T23:12:18.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.859+0000 7f892ec00640 1 -- 172.21.15.28:0/3626278673 wait complete. 2024-02-21T23:12:18.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.861+0000 7f892ec00640 1 Processor -- start 2024-02-21T23:12:18.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.861+0000 7f892ec00640 1 -- start start 2024-02-21T23:12:18.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.862+0000 7f892ec00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f89280c6710 0x7f89281e0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:18.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.862+0000 7f892ec00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8928152ab0 0x7f892814db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:18.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.862+0000 7f892ec00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8928158850 0x7f892814e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:18.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.862+0000 7f892ec00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89280c5aa0 con 0x7f8928152ab0 2024-02-21T23:12:18.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.862+0000 7f892ec00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89280c5920 con 0x7f89280c6710 2024-02-21T23:12:18.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.862+0000 7f892ec00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8928159200 con 0x7f8928158850 2024-02-21T23:12:18.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.862+0000 7f8927400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8928152ab0 0x7f892814db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:18.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.862+0000 7f8927400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8928152ab0 0x7f892814db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41590/0 (socket says 172.21.15.28:41590) 2024-02-21T23:12:18.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.862+0000 7f8927400640 1 -- 172.21.15.28:0/1700603646 learned_addr learned my addr 172.21.15.28:0/1700603646 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:18.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.863+0000 7f8927e00640 1 --2- 172.21.15.28:0/1700603646 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f89280c6710 0x7f89281e0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:18.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.863+0000 7f892cc00640 1 --2- 172.21.15.28:0/1700603646 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8928158850 0x7f892814e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:18.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.863+0000 7f8927400640 1 -- 172.21.15.28:0/1700603646 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8928158850 msgr2=0x7f892814e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:18.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.863+0000 7f8927400640 1 --2- 172.21.15.28:0/1700603646 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8928158850 0x7f892814e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:18.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.863+0000 7f8927400640 1 -- 172.21.15.28:0/1700603646 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f89280c6710 msgr2=0x7f89281e0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:18.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.863+0000 7f8927400640 1 --2- 172.21.15.28:0/1700603646 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f89280c6710 0x7f89281e0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:18.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.863+0000 7f8927400640 1 -- 172.21.15.28:0/1700603646 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8928150600 con 0x7f8928152ab0 2024-02-21T23:12:18.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.863+0000 7f8927e00640 1 --2- 172.21.15.28:0/1700603646 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f89280c6710 0x7f89281e0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:12:18.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.863+0000 7f8927400640 1 --2- 172.21.15.28:0/1700603646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8928152ab0 0x7f892814db10 secure :-1 s=READY pgs=2199 cs=0 l=1 rev1=1 crypto rx=0x7f891800cb80 tx=0x7f891800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:18.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.864+0000 7f891fe00640 1 -- 172.21.15.28:0/1700603646 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8918017070 con 0x7f8928152ab0 2024-02-21T23:12:18.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.864+0000 7f891fe00640 1 -- 172.21.15.28:0/1700603646 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f891800ad40 con 0x7f8928152ab0 2024-02-21T23:12:18.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.864+0000 7f891fe00640 1 -- 172.21.15.28:0/1700603646 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8918012640 con 0x7f8928152ab0 2024-02-21T23:12:18.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.864+0000 7f892ec00640 1 -- 172.21.15.28:0/1700603646 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89280adc70 con 0x7f8928152ab0 2024-02-21T23:12:18.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.864+0000 7f892ec00640 1 -- 172.21.15.28:0/1700603646 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f892814f840 con 0x7f8928152ab0 2024-02-21T23:12:18.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.865+0000 7f892ec00640 1 -- 172.21.15.28:0/1700603646 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8928067f30 con 0x7f8928152ab0 2024-02-21T23:12:18.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.866+0000 7f891fe00640 1 -- 172.21.15.28:0/1700603646 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f891801a070 con 0x7f8928152ab0 2024-02-21T23:12:18.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.867+0000 7f891fe00640 1 --2- 172.21.15.28:0/1700603646 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f890406b500 0x7f890406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:18.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.867+0000 7f891fe00640 1 -- 172.21.15.28:0/1700603646 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f8918089f00 con 0x7f8928152ab0 2024-02-21T23:12:18.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.867+0000 7f8927e00640 1 --2- 172.21.15.28:0/1700603646 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f890406b500 0x7f890406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:18.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.869+0000 7f8927e00640 1 --2- 172.21.15.28:0/1700603646 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f890406b500 0x7f890406d9c0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f89281519f0 tx=0x7f8910008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:18.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:18.871+0000 7f891fe00640 1 -- 172.21.15.28:0/1700603646 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f89280a8180 con 0x7f8928152ab0 2024-02-21T23:12:19.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.150+0000 7f892ec00640 1 -- 172.21.15.28:0/1700603646 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterResize.test_down_health"]} v 0) v1 -- 0x7f89280c7500 con 0x7f8928152ab0 2024-02-21T23:12:19.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.712+0000 7f891fe00640 1 -- 172.21.15.28:0/1700603646 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterResize.test_down_health"]}]=0 v552) v1 ==== 143+0+0 (secure 0 0 0) 0x7f8918058e90 con 0x7f8928152ab0 2024-02-21T23:12:19.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.716+0000 7f892ec00640 1 -- 172.21.15.28:0/1700603646 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f890406b500 msgr2=0x7f890406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:19.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.716+0000 7f892ec00640 1 --2- 172.21.15.28:0/1700603646 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f890406b500 0x7f890406d9c0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f89281519f0 tx=0x7f8910008040 comp rx=0 tx=0).stop 2024-02-21T23:12:19.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.716+0000 7f892ec00640 1 -- 172.21.15.28:0/1700603646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8928152ab0 msgr2=0x7f892814db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:19.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.716+0000 7f892ec00640 1 --2- 172.21.15.28:0/1700603646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8928152ab0 0x7f892814db10 secure :-1 s=READY pgs=2199 cs=0 l=1 rev1=1 crypto rx=0x7f891800cb80 tx=0x7f891800a430 comp rx=0 tx=0).stop 2024-02-21T23:12:19.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.716+0000 7f892ec00640 1 -- 172.21.15.28:0/1700603646 shutdown_connections 2024-02-21T23:12:19.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.716+0000 7f892ec00640 1 --2- 172.21.15.28:0/1700603646 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f890406b500 0x7f890406d9c0 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:19.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.716+0000 7f892ec00640 1 --2- 172.21.15.28:0/1700603646 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8928158850 0x7f892814e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:19.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.716+0000 7f892ec00640 1 --2- 172.21.15.28:0/1700603646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8928152ab0 0x7f892814db10 unknown :-1 s=CLOSED pgs=2199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:19.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.716+0000 7f892ec00640 1 --2- 172.21.15.28:0/1700603646 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f89280c6710 0x7f89281e0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:19.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.716+0000 7f892ec00640 1 -- 172.21.15.28:0/1700603646 >> 172.21.15.28:0/1700603646 conn(0x7f89280c8810 msgr2=0x7f892806c950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:19.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.716+0000 7f892ec00640 1 -- 172.21.15.28:0/1700603646 shutdown_connections 2024-02-21T23:12:19.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.716+0000 7f892ec00640 1 -- 172.21.15.28:0/1700603646 wait complete. 2024-02-21T23:12:19.726 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-02-21T23:12:19.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.970+0000 7fc3f6a00640 1 Processor -- start 2024-02-21T23:12:19.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.970+0000 7fc3f6a00640 1 -- start start 2024-02-21T23:12:19.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.972+0000 7fc3f6a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fc3f006ae70 0x7fc3f006b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:19.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.972+0000 7fc3f6a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc3f0159c50 0x7fc3f0163b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:19.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.972+0000 7fc3f6a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3f00c0d40 con 0x7fc3f00c4c10 2024-02-21T23:12:19.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.972+0000 7fc3f6a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3f015caf0 con 0x7fc3f006ae70 2024-02-21T23:12:19.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.972+0000 7fc3f6a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f015cdd0 con 0x7fc3f0159c50 2024-02-21T23:12:19.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.972+0000 7fc3f4c00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fc3f00c4c10 0x7fc3f0159540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39692/0 (socket says 172.21.15.28:39692) 2024-02-21T23:12:19.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.972+0000 7fc3f4c00640 1 -- 172.21.15.28:0/2206311665 learned_addr learned my addr 172.21.15.28:0/2206311665 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:12:19.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.972+0000 7fc3f5600640 1 --2- 172.21.15.28:0/2206311665 >> v2:172.21.15.81:3300/0 conn(0x7fc3f006ae70 0x7fc3f006b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:19.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.973+0000 7fc3f6000640 1 --2- 172.21.15.28:0/2206311665 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc3f0159c50 0x7fc3f0163b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:19.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.973+0000 7fc3efe00640 1 -- 172.21.15.28:0/2206311665 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1615618822 0 0) 0x7fc3f00c0d40 con 0x7fc3f00c4c10 2024-02-21T23:12:19.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.973+0000 7fc3efe00640 1 -- 172.21.15.28:0/2206311665 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3d8003520 con 0x7fc3f00c4c10 2024-02-21T23:12:19.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.973+0000 7fc3efe00640 1 -- 172.21.15.28:0/2206311665 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 498609336 0 0) 0x7fc3d8003520 con 0x7fc3f00c4c10 2024-02-21T23:12:19.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.973+0000 7fc3efe00640 1 -- 172.21.15.28:0/2206311665 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc3f0159c50 msgr2=0x7fc3f0163b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:19.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.973+0000 7fc3efe00640 1 --2- 172.21.15.28:0/2206311665 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc3f0159c50 0x7fc3f0163b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:19.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.973+0000 7fc3efe00640 1 -- 172.21.15.28:0/2206311665 >> v2:172.21.15.81:3300/0 conn(0x7fc3f006ae70 msgr2=0x7fc3f006b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:19.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.973+0000 7fc3efe00640 1 --2- 172.21.15.28:0/2206311665 >> v2:172.21.15.81:3300/0 conn(0x7fc3f006ae70 0x7fc3f006b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:19.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.973+0000 7fc3efe00640 1 -- 172.21.15.28:0/2206311665 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3f015d0b0 con 0x7fc3f00c4c10 2024-02-21T23:12:19.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.973+0000 7fc3efe00640 1 -- 172.21.15.28:0/2206311665 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fc3dc001670 con 0x7fc3f00c4c10 2024-02-21T23:12:19.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.973+0000 7fc3efe00640 1 -- 172.21.15.28:0/2206311665 >> v1:172.21.15.28:6789/0 conn(0x7fc3f00c4c10 legacy=0x7fc3f0159540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:19.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.973+0000 7fc3efe00640 1 --2- 172.21.15.28:0/2206311665 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc3f0159c50 0x7fc3d80039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:19.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.973+0000 7fc3efe00640 1 --2- 172.21.15.28:0/2206311665 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc3d8003f30 0x7fc3d8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:19.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.973+0000 7fc3efe00640 1 --2- 172.21.15.28:0/2206311665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3d8004d40 0x7fc3d80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:19.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.973+0000 7fc3efe00640 1 -- 172.21.15.28:0/2206311665 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f015caf0 con 0x7fc3d8004d40 2024-02-21T23:12:19.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.973+0000 7fc3efe00640 1 -- 172.21.15.28:0/2206311665 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f015cdd0 con 0x7fc3d8003f30 2024-02-21T23:12:19.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.973+0000 7fc3efe00640 1 -- 172.21.15.28:0/2206311665 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d800ba10 con 0x7fc3f0159c50 2024-02-21T23:12:19.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.974+0000 7fc3f6000640 1 --2- 172.21.15.28:0/2206311665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3d8004d40 0x7fc3d80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:19.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.974+0000 7fc3f5600640 1 --2- 172.21.15.28:0/2206311665 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc3f0159c50 0x7fc3d80039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:19.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.974+0000 7fc3f4c00640 1 --2- 172.21.15.28:0/2206311665 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc3d8003f30 0x7fc3d8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:19.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.974+0000 7fc3f6000640 1 -- 172.21.15.28:0/2206311665 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc3f0159c50 msgr2=0x7fc3d80039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:19.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.974+0000 7fc3f6000640 1 --2- 172.21.15.28:0/2206311665 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc3f0159c50 0x7fc3d80039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:19.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.974+0000 7fc3f6000640 1 -- 172.21.15.28:0/2206311665 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc3d8003f30 msgr2=0x7fc3d8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:19.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.974+0000 7fc3f6000640 1 --2- 172.21.15.28:0/2206311665 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc3d8003f30 0x7fc3d8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:19.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.974+0000 7fc3f6000640 1 -- 172.21.15.28:0/2206311665 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3d8003520 con 0x7fc3d8004d40 2024-02-21T23:12:19.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.975+0000 7fc3f6000640 1 --2- 172.21.15.28:0/2206311665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3d8004d40 0x7fc3d80051c0 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 crypto rx=0x7fc3e800e170 tx=0x7fc3d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b953f26b8753e0dd server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:19.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.975+0000 7fc3efe00640 1 -- 172.21.15.28:0/2206311665 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e8039070 con 0x7fc3d8004d40 2024-02-21T23:12:19.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.975+0000 7fc3efe00640 1 -- 172.21.15.28:0/2206311665 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3e802cbf0 con 0x7fc3d8004d40 2024-02-21T23:12:19.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.975+0000 7fc3efe00640 1 -- 172.21.15.28:0/2206311665 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e8009c10 con 0x7fc3d8004d40 2024-02-21T23:12:19.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.976+0000 7fc3f6a00640 1 -- 172.21.15.28:0/2206311665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3d8004d40 msgr2=0x7fc3d80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:19.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.976+0000 7fc3f6a00640 1 --2- 172.21.15.28:0/2206311665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3d8004d40 0x7fc3d80051c0 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 crypto rx=0x7fc3e800e170 tx=0x7fc3d8005830 comp rx=0 tx=0).stop 2024-02-21T23:12:19.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.976+0000 7fc3f6000640 1 -- 172.21.15.28:0/2206311665 reap_dead start 2024-02-21T23:12:19.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.976+0000 7fc3f6a00640 1 -- 172.21.15.28:0/2206311665 shutdown_connections 2024-02-21T23:12:19.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.976+0000 7fc3f6a00640 1 -- 172.21.15.28:0/2206311665 >> 172.21.15.28:0/2206311665 conn(0x7fc3f00a8530 msgr2=0x7fc3f00a8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:19.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.976+0000 7fc3f6a00640 1 -- 172.21.15.28:0/2206311665 shutdown_connections 2024-02-21T23:12:19.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.976+0000 7fc3f6a00640 1 -- 172.21.15.28:0/2206311665 wait complete. 2024-02-21T23:12:19.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.978+0000 7fc3f6a00640 1 Processor -- start 2024-02-21T23:12:19.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.978+0000 7fc3f6a00640 1 -- start start 2024-02-21T23:12:19.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.979+0000 7fc3f6a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc3d8004d40 0x7fc3f0155e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:19.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.979+0000 7fc3f6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3f01536c0 0x7fc3f014e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:19.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.979+0000 7fc3f6a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc3f00711d0 0x7fc3f0154270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:19.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.979+0000 7fc3f6a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f015c7e0 con 0x7fc3f01536c0 2024-02-21T23:12:19.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.979+0000 7fc3f6a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f015d670 con 0x7fc3d8004d40 2024-02-21T23:12:19.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.979+0000 7fc3f6a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f006f6b0 con 0x7fc3f00711d0 2024-02-21T23:12:19.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.980+0000 7fc3f4c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3f01536c0 0x7fc3f014e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:19.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.980+0000 7fc3f4c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3f01536c0 0x7fc3f014e160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41610/0 (socket says 172.21.15.28:41610) 2024-02-21T23:12:19.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.980+0000 7fc3f4c00640 1 -- 172.21.15.28:0/372700182 learned_addr learned my addr 172.21.15.28:0/372700182 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:19.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.980+0000 7fc3f5600640 1 --2- 172.21.15.28:0/372700182 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc3d8004d40 0x7fc3f0155e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:19.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.980+0000 7fc3f6000640 1 --2- 172.21.15.28:0/372700182 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc3f00711d0 0x7fc3f0154270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:19.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.980+0000 7fc3f4c00640 1 -- 172.21.15.28:0/372700182 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc3f00711d0 msgr2=0x7fc3f0154270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:19.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.980+0000 7fc3f4c00640 1 --2- 172.21.15.28:0/372700182 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc3f00711d0 0x7fc3f0154270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:19.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.980+0000 7fc3f4c00640 1 -- 172.21.15.28:0/372700182 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc3d8004d40 msgr2=0x7fc3f0155e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:19.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.980+0000 7fc3f4c00640 1 --2- 172.21.15.28:0/372700182 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc3d8004d40 0x7fc3f0155e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:19.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.980+0000 7fc3f4c00640 1 -- 172.21.15.28:0/372700182 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3f014ba70 con 0x7fc3f01536c0 2024-02-21T23:12:19.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.980+0000 7fc3f5600640 1 --2- 172.21.15.28:0/372700182 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc3d8004d40 0x7fc3f0155e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:12:19.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.980+0000 7fc3f4c00640 1 --2- 172.21.15.28:0/372700182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3f01536c0 0x7fc3f014e160 secure :-1 s=READY pgs=2202 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc00e8f0 tx=0x7fc3dc00edb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:19.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.981+0000 7fc3ed600640 1 -- 172.21.15.28:0/372700182 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3dc00c870 con 0x7fc3f01536c0 2024-02-21T23:12:19.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.981+0000 7fc3ed600640 1 -- 172.21.15.28:0/372700182 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3dc018070 con 0x7fc3f01536c0 2024-02-21T23:12:19.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.981+0000 7fc3f6a00640 1 -- 172.21.15.28:0/372700182 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3f014bd50 con 0x7fc3f01536c0 2024-02-21T23:12:19.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.981+0000 7fc3ed600640 1 -- 172.21.15.28:0/372700182 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3dc00cc30 con 0x7fc3f01536c0 2024-02-21T23:12:19.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.981+0000 7fc3f6a00640 1 -- 172.21.15.28:0/372700182 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3f01531a0 con 0x7fc3f01536c0 2024-02-21T23:12:19.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.982+0000 7fc3f6a00640 1 -- 172.21.15.28:0/372700182 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3f004f290 con 0x7fc3f01536c0 2024-02-21T23:12:19.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.983+0000 7fc3ed600640 1 -- 172.21.15.28:0/372700182 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc3dc013420 con 0x7fc3f01536c0 2024-02-21T23:12:19.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.984+0000 7fc3ed600640 1 --2- 172.21.15.28:0/372700182 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc3d006b500 0x7fc3d006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:19.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.984+0000 7fc3ed600640 1 -- 172.21.15.28:0/372700182 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5421+0+0 (secure 0 0 0) 0x7fc3dc089cb0 con 0x7fc3f01536c0 2024-02-21T23:12:19.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.985+0000 7fc3f5600640 1 --2- 172.21.15.28:0/372700182 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc3d006b500 0x7fc3d006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:19.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.986+0000 7fc3f5600640 1 --2- 172.21.15.28:0/372700182 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc3d006b500 0x7fc3d006d9c0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fc3e4009a80 tx=0x7fc3e4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:19.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:19.989+0000 7fc3ed600640 1 -- 172.21.15.28:0/372700182 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc3dc058ce0 con 0x7fc3f01536c0 2024-02-21T23:12:20.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.190+0000 7fc3f6a00640 1 -- 172.21.15.28:0/372700182 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fc3f0153370 con 0x7fc3f01536c0 2024-02-21T23:12:20.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.197+0000 7fc3ed600640 1 -- 172.21.15.28:0/372700182 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v61)=0 v61) v1 ==== 122+0+0 (secure 0 0 0) 0x7fc3dc0585e0 con 0x7fc3f01536c0 2024-02-21T23:12:20.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.200+0000 7fc3f6a00640 1 -- 172.21.15.28:0/372700182 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc3d006b500 msgr2=0x7fc3d006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:20.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.200+0000 7fc3f6a00640 1 --2- 172.21.15.28:0/372700182 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc3d006b500 0x7fc3d006d9c0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fc3e4009a80 tx=0x7fc3e4008040 comp rx=0 tx=0).stop 2024-02-21T23:12:20.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.200+0000 7fc3f6a00640 1 -- 172.21.15.28:0/372700182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3f01536c0 msgr2=0x7fc3f014e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:20.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.200+0000 7fc3f6a00640 1 --2- 172.21.15.28:0/372700182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3f01536c0 0x7fc3f014e160 secure :-1 s=READY pgs=2202 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc00e8f0 tx=0x7fc3dc00edb0 comp rx=0 tx=0).stop 2024-02-21T23:12:20.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.200+0000 7fc3f6a00640 1 -- 172.21.15.28:0/372700182 shutdown_connections 2024-02-21T23:12:20.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.200+0000 7fc3f6a00640 1 --2- 172.21.15.28:0/372700182 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc3d006b500 0x7fc3d006d9c0 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:20.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.200+0000 7fc3f6a00640 1 --2- 172.21.15.28:0/372700182 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc3f00711d0 0x7fc3f0154270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:20.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.200+0000 7fc3f6a00640 1 --2- 172.21.15.28:0/372700182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3f01536c0 0x7fc3f014e160 unknown :-1 s=CLOSED pgs=2202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:20.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.200+0000 7fc3f6a00640 1 --2- 172.21.15.28:0/372700182 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc3d8004d40 0x7fc3f0155e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:20.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.200+0000 7fc3f6a00640 1 -- 172.21.15.28:0/372700182 >> 172.21.15.28:0/372700182 conn(0x7fc3f00a8530 msgr2=0x7fc3f00c0980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:20.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.200+0000 7fc3f6a00640 1 -- 172.21.15.28:0/372700182 shutdown_connections 2024-02-21T23:12:20.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.200+0000 7fc3f6a00640 1 -- 172.21.15.28:0/372700182 wait complete. 2024-02-21T23:12:20.207 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-02-21T23:12:20.207 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-21T23:12:20.321 DEBUG:teuthology.orchestra.run.smithi081:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-21T23:12:20.437 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-02-21T23:12:20.437 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-21T23:12:20.437 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-21T23:12:20.499 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054608,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-02-21T23:12:20.500 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-02-21T23:12:20.500 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-21T23:12:20.556 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053833,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498812k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249404k,nr_inodes=812351,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498812k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-02-21T23:12:20.556 INFO:tasks.cephfs.cephfs_test_case:[, ] 2024-02-21T23:12:20.556 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:12:20.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.805+0000 7f8002a00640 1 Processor -- start 2024-02-21T23:12:20.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.806+0000 7f8002a00640 1 -- start start 2024-02-21T23:12:20.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.807+0000 7f8002a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f7ffc0a5fd0 0x7f7ffc06dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:20.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.807+0000 7f8002a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ffc0c2760 con 0x7f7ffc150450 2024-02-21T23:12:20.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.807+0000 7f8002a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ffc156310 con 0x7f7ffc06ee40 2024-02-21T23:12:20.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.807+0000 7f8002a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ffc0c22c0 con 0x7f7ffc0a5fd0 2024-02-21T23:12:20.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.808+0000 7f8001600640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f7ffc150450 0x7f7ffc1536f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39696/0 (socket says 172.21.15.28:39696) 2024-02-21T23:12:20.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.808+0000 7f8001600640 1 -- 172.21.15.28:0/2279759515 learned_addr learned my addr 172.21.15.28:0/2279759515 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:12:20.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.808+0000 7f8002000640 1 --2- 172.21.15.28:0/2279759515 >> v2:172.21.15.81:3300/0 conn(0x7f7ffc0a5fd0 0x7f7ffc06dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:20.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.808+0000 7f7ffbe00640 1 -- 172.21.15.28:0/2279759515 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 803702497 0 0) 0x7f7ffc0c2760 con 0x7f7ffc150450 2024-02-21T23:12:20.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.808+0000 7f7ffbe00640 1 -- 172.21.15.28:0/2279759515 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fe4003520 con 0x7f7ffc150450 2024-02-21T23:12:20.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.809+0000 7f7ffbe00640 1 -- 172.21.15.28:0/2279759515 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3835794333 0 0) 0x7f7fe4003520 con 0x7f7ffc150450 2024-02-21T23:12:20.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.809+0000 7f7ffbe00640 1 -- 172.21.15.28:0/2279759515 >> v2:172.21.15.81:3300/0 conn(0x7f7ffc0a5fd0 msgr2=0x7f7ffc06dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:20.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.809+0000 7f7ffbe00640 1 --2- 172.21.15.28:0/2279759515 >> v2:172.21.15.81:3300/0 conn(0x7f7ffc0a5fd0 0x7f7ffc06dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:20.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.809+0000 7f7ffbe00640 1 -- 172.21.15.28:0/2279759515 >> v1:172.21.15.81:6789/0 conn(0x7f7ffc06ee40 legacy=0x7f7ffc156af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:20.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.809+0000 7f7ffbe00640 1 -- 172.21.15.28:0/2279759515 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ffc0c29d0 con 0x7f7ffc150450 2024-02-21T23:12:20.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.809+0000 7f7ffbe00640 1 -- 172.21.15.28:0/2279759515 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f7ff0001670 con 0x7f7ffc150450 2024-02-21T23:12:20.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.809+0000 7f7ffbe00640 1 -- 172.21.15.28:0/2279759515 >> v1:172.21.15.28:6789/0 conn(0x7f7ffc150450 legacy=0x7f7ffc1536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:20.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.809+0000 7f7ffbe00640 1 --2- 172.21.15.28:0/2279759515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fe40039b0 0x7f7fe4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:20.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.809+0000 7f7ffbe00640 1 --2- 172.21.15.28:0/2279759515 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7fe40043b0 0x7f7fe4004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:20.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.809+0000 7f7ffbe00640 1 --2- 172.21.15.28:0/2279759515 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7fe4005080 0x7f7fe400fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:20.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.809+0000 7f7ffbe00640 1 -- 172.21.15.28:0/2279759515 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe40103c0 con 0x7f7fe40039b0 2024-02-21T23:12:20.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.809+0000 7f7ffbe00640 1 -- 172.21.15.28:0/2279759515 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe4010570 con 0x7f7fe40043b0 2024-02-21T23:12:20.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.809+0000 7f7ffbe00640 1 -- 172.21.15.28:0/2279759515 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe4010770 con 0x7f7fe4005080 2024-02-21T23:12:20.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.809+0000 7f8001600640 1 --2- 172.21.15.28:0/2279759515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fe40039b0 0x7f7fe4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:20.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.809+0000 7f8000c00640 1 --2- 172.21.15.28:0/2279759515 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7fe40043b0 0x7f7fe4004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:20.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.810+0000 7f8002000640 1 --2- 172.21.15.28:0/2279759515 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7fe4005080 0x7f7fe400fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:20.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.810+0000 7f8001600640 1 -- 172.21.15.28:0/2279759515 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7fe4005080 msgr2=0x7f7fe400fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:20.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.810+0000 7f8001600640 1 --2- 172.21.15.28:0/2279759515 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7fe4005080 0x7f7fe400fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:20.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.810+0000 7f8001600640 1 -- 172.21.15.28:0/2279759515 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7fe40043b0 msgr2=0x7f7fe4004880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:20.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.810+0000 7f8001600640 1 --2- 172.21.15.28:0/2279759515 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7fe40043b0 0x7f7fe4004880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:20.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.810+0000 7f8001600640 1 -- 172.21.15.28:0/2279759515 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fe4003520 con 0x7f7fe40039b0 2024-02-21T23:12:20.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.810+0000 7f8002000640 1 --2- 172.21.15.28:0/2279759515 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7fe4005080 0x7f7fe400fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:12:20.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.810+0000 7f8002000640 1 -- 172.21.15.28:0/2279759515 reap_dead start 2024-02-21T23:12:20.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.811+0000 7f8001600640 1 --2- 172.21.15.28:0/2279759515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fe40039b0 0x7f7fe4003d90 secure :-1 s=READY pgs=2204 cs=0 l=1 rev1=1 crypto rx=0x7f7ff000c210 tx=0x7f7fe4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66be37a40976b30a server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:20.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.811+0000 7f7ffbe00640 1 -- 172.21.15.28:0/2279759515 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff0007ce0 con 0x7f7fe40039b0 2024-02-21T23:12:20.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.811+0000 7f7ffbe00640 1 -- 172.21.15.28:0/2279759515 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ff0030070 con 0x7f7fe40039b0 2024-02-21T23:12:20.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.811+0000 7f7ffbe00640 1 -- 172.21.15.28:0/2279759515 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff0005d70 con 0x7f7fe40039b0 2024-02-21T23:12:20.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.812+0000 7f8002a00640 1 -- 172.21.15.28:0/2279759515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fe40039b0 msgr2=0x7f7fe4003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:20.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.812+0000 7f8002a00640 1 --2- 172.21.15.28:0/2279759515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fe40039b0 0x7f7fe4003d90 secure :-1 s=READY pgs=2204 cs=0 l=1 rev1=1 crypto rx=0x7f7ff000c210 tx=0x7f7fe4005830 comp rx=0 tx=0).stop 2024-02-21T23:12:20.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.812+0000 7f8002a00640 1 -- 172.21.15.28:0/2279759515 shutdown_connections 2024-02-21T23:12:20.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.812+0000 7f8002a00640 1 --2- 172.21.15.28:0/2279759515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fe40039b0 0x7f7fe4003d90 unknown :-1 s=CLOSED pgs=2204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:20.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.812+0000 7f8002a00640 1 -- 172.21.15.28:0/2279759515 >> 172.21.15.28:0/2279759515 conn(0x7f7ffc06b630 msgr2=0x7f7ffc0c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:20.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.812+0000 7f8002a00640 1 -- 172.21.15.28:0/2279759515 shutdown_connections 2024-02-21T23:12:20.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.812+0000 7f8002a00640 1 -- 172.21.15.28:0/2279759515 wait complete. 2024-02-21T23:12:20.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.814+0000 7f8002a00640 1 Processor -- start 2024-02-21T23:12:20.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.814+0000 7f8002a00640 1 -- start start 2024-02-21T23:12:20.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.815+0000 7f8002a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7fe40039b0 0x7f7ffc1e6b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:20.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.815+0000 7f8002a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ffc0c5fc0 0x7f7ffc0c5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:20.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.815+0000 7f8002a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7ffc1e9c50 0x7f7ffc1ea0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:20.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.815+0000 7f8002a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ffc1536f0 con 0x7f7ffc0c5fc0 2024-02-21T23:12:20.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.815+0000 7f8002a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ffc156660 con 0x7f7fe40039b0 2024-02-21T23:12:20.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.815+0000 7f8002a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ffc153870 con 0x7f7ffc1e9c50 2024-02-21T23:12:20.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.815+0000 7f8000c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ffc0c5fc0 0x7f7ffc0c5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:20.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.815+0000 7f8000c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ffc0c5fc0 0x7f7ffc0c5580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41632/0 (socket says 172.21.15.28:41632) 2024-02-21T23:12:20.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.815+0000 7f8000c00640 1 -- 172.21.15.28:0/2749408662 learned_addr learned my addr 172.21.15.28:0/2749408662 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:20.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.815+0000 7f8001600640 1 --2- 172.21.15.28:0/2749408662 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7fe40039b0 0x7f7ffc1e6b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:20.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.816+0000 7f8002000640 1 --2- 172.21.15.28:0/2749408662 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7ffc1e9c50 0x7f7ffc1ea0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:20.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.816+0000 7f8000c00640 1 -- 172.21.15.28:0/2749408662 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7ffc1e9c50 msgr2=0x7f7ffc1ea0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:20.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.816+0000 7f8000c00640 1 --2- 172.21.15.28:0/2749408662 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7ffc1e9c50 0x7f7ffc1ea0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:20.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.816+0000 7f8000c00640 1 -- 172.21.15.28:0/2749408662 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7fe40039b0 msgr2=0x7f7ffc1e6b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:20.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.816+0000 7f8000c00640 1 --2- 172.21.15.28:0/2749408662 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7fe40039b0 0x7f7ffc1e6b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:20.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.816+0000 7f8000c00640 1 -- 172.21.15.28:0/2749408662 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ffc1ea610 con 0x7f7ffc0c5fc0 2024-02-21T23:12:20.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.816+0000 7f8001600640 1 --2- 172.21.15.28:0/2749408662 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7fe40039b0 0x7f7ffc1e6b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:12:20.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.816+0000 7f8000c00640 1 --2- 172.21.15.28:0/2749408662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ffc0c5fc0 0x7f7ffc0c5580 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 crypto rx=0x7f7fec00e580 tx=0x7f7fec00ea40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:20.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.816+0000 7f7ff9600640 1 -- 172.21.15.28:0/2749408662 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fec0052e0 con 0x7f7ffc0c5fc0 2024-02-21T23:12:20.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.816+0000 7f7ff9600640 1 -- 172.21.15.28:0/2749408662 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fec018070 con 0x7f7ffc0c5fc0 2024-02-21T23:12:20.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.816+0000 7f7ff9600640 1 -- 172.21.15.28:0/2749408662 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fec003830 con 0x7f7ffc0c5fc0 2024-02-21T23:12:20.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.816+0000 7f8002a00640 1 -- 172.21.15.28:0/2749408662 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ffc14b970 con 0x7f7ffc0c5fc0 2024-02-21T23:12:20.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.817+0000 7f8002a00640 1 -- 172.21.15.28:0/2749408662 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ffc14bf30 con 0x7f7ffc0c5fc0 2024-02-21T23:12:20.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.818+0000 7f8002a00640 1 -- 172.21.15.28:0/2749408662 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ffc04f290 con 0x7f7ffc0c5fc0 2024-02-21T23:12:20.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.819+0000 7f7ff9600640 1 -- 172.21.15.28:0/2749408662 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7fec0039d0 con 0x7f7ffc0c5fc0 2024-02-21T23:12:20.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.819+0000 7f7ff9600640 1 --2- 172.21.15.28:0/2749408662 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7fe006b4d0 0x7f7fe006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:20.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.820+0000 7f7ff9600640 1 -- 172.21.15.28:0/2749408662 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f7fec089ef0 con 0x7f7ffc0c5fc0 2024-02-21T23:12:20.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.820+0000 7f8001600640 1 --2- 172.21.15.28:0/2749408662 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7fe006b4d0 0x7f7fe006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:20.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.821+0000 7f8001600640 1 --2- 172.21.15.28:0/2749408662 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7fe006b4d0 0x7f7fe006d990 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f7ff0034000 tx=0x7f7ff0009030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:20.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:20.824+0000 7f7ff9600640 1 -- 172.21.15.28:0/2749408662 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7fec058e80 con 0x7f7ffc0c5fc0 2024-02-21T23:12:21.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.063+0000 7f8002a00640 1 -- 172.21.15.28:0/2749408662 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7ffc0a5c40 con 0x7f7ffc0c5fc0 2024-02-21T23:12:21.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.064+0000 7f7ff9600640 1 -- 172.21.15.28:0/2749408662 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 211 v211) v1 ==== 95+0+3698 (secure 0 0 0) 0x7f7fec01be40 con 0x7f7ffc0c5fc0 2024-02-21T23:12:21.051 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:12:21.051 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":211,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6848,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3439896042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3439896042},{"type":"v1","addr":"172.21.15.28:6835","nonce":3439896042}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":196},{"gid":7073,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/86793923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":86793923},{"type":"v1","addr":"172.21.15.81:6837","nonce":86793923}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":211},{"gid":7729,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/3213249808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3213249808},{"type":"v1","addr":"172.21.15.28:6833","nonce":3213249808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":195},{"gid":8023,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/3178479663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":3178479663},{"type":"v1","addr":"172.21.15.81:6836","nonce":3178479663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":211}],"filesystems":[]} 2024-02-21T23:12:21.052 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 211 2024-02-21T23:12:21.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.066+0000 7f8002a00640 1 -- 172.21.15.28:0/2749408662 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7fe006b4d0 msgr2=0x7f7fe006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:21.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.066+0000 7f8002a00640 1 --2- 172.21.15.28:0/2749408662 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7fe006b4d0 0x7f7fe006d990 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f7ff0034000 tx=0x7f7ff0009030 comp rx=0 tx=0).stop 2024-02-21T23:12:21.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.066+0000 7f8002a00640 1 -- 172.21.15.28:0/2749408662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ffc0c5fc0 msgr2=0x7f7ffc0c5580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:21.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.066+0000 7f8002a00640 1 --2- 172.21.15.28:0/2749408662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ffc0c5fc0 0x7f7ffc0c5580 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 crypto rx=0x7f7fec00e580 tx=0x7f7fec00ea40 comp rx=0 tx=0).stop 2024-02-21T23:12:21.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.066+0000 7f8002a00640 1 -- 172.21.15.28:0/2749408662 shutdown_connections 2024-02-21T23:12:21.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.066+0000 7f8002a00640 1 --2- 172.21.15.28:0/2749408662 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7fe006b4d0 0x7f7fe006d990 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:21.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.066+0000 7f8002a00640 1 --2- 172.21.15.28:0/2749408662 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7ffc1e9c50 0x7f7ffc1ea0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:21.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.066+0000 7f8002a00640 1 --2- 172.21.15.28:0/2749408662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ffc0c5fc0 0x7f7ffc0c5580 unknown :-1 s=CLOSED pgs=2205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:21.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.066+0000 7f8002a00640 1 --2- 172.21.15.28:0/2749408662 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7fe40039b0 0x7f7ffc1e6b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:21.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.066+0000 7f8002a00640 1 -- 172.21.15.28:0/2749408662 >> 172.21.15.28:0/2749408662 conn(0x7f7ffc06b630 msgr2=0x7f7ffc0a74b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:21.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.067+0000 7f8002a00640 1 -- 172.21.15.28:0/2749408662 shutdown_connections 2024-02-21T23:12:21.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.067+0000 7f8002a00640 1 -- 172.21.15.28:0/2749408662 wait complete. 2024-02-21T23:12:21.073 INFO:tasks.ceph.mds.a:Restarting daemon 2024-02-21T23:12:21.084 INFO:tasks.ceph.mds.a:Stopping old one... 2024-02-21T23:12:21.085 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-02-21T23:12:21.085 INFO:teuthology.orchestra.run:waiting for 300 2024-02-21T23:12:21.085 INFO:tasks.ceph.mds.c:Restarting daemon 2024-02-21T23:12:21.085 INFO:tasks.ceph.mds.c:Stopping old one... 2024-02-21T23:12:21.085 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-02-21T23:12:21.085 INFO:teuthology.orchestra.run:waiting for 300 2024-02-21T23:12:21.086 INFO:tasks.ceph.mds.b:Restarting daemon 2024-02-21T23:12:21.086 INFO:tasks.ceph.mds.b:Stopping old one... 2024-02-21T23:12:21.086 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-02-21T23:12:21.086 INFO:teuthology.orchestra.run:waiting for 300 2024-02-21T23:12:21.086 INFO:tasks.ceph.mds.d:Restarting daemon 2024-02-21T23:12:21.086 INFO:tasks.ceph.mds.d:Stopping old one... 2024-02-21T23:12:21.086 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-02-21T23:12:21.087 INFO:teuthology.orchestra.run:waiting for 300 2024-02-21T23:12:21.105 INFO:tasks.ceph.mds.a:Stopped 2024-02-21T23:12:21.106 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-02-21T23:12:21.107 INFO:tasks.ceph.mds.b:Stopped 2024-02-21T23:12:21.107 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-02-21T23:12:21.109 INFO:tasks.ceph.mds.c:Stopped 2024-02-21T23:12:21.109 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-02-21T23:12:21.113 INFO:tasks.ceph.mds.d:Stopped 2024-02-21T23:12:21.113 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-02-21T23:12:21.156 INFO:tasks.ceph.mds.a:Started 2024-02-21T23:12:21.157 DEBUG:teuthology.parallel:result is None 2024-02-21T23:12:21.159 INFO:tasks.ceph.mds.b:Started 2024-02-21T23:12:21.160 DEBUG:teuthology.parallel:result is None 2024-02-21T23:12:21.163 INFO:tasks.ceph.mds.c:Started 2024-02-21T23:12:21.163 DEBUG:teuthology.parallel:result is None 2024-02-21T23:12:21.166 INFO:tasks.ceph.mds.d:Started 2024-02-21T23:12:21.166 DEBUG:teuthology.parallel:result is None 2024-02-21T23:12:21.166 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-02-21T23:12:21.310 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-02-21T23:12:21.310 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-02-21T23:12:21.321 INFO:tasks.ceph.mds.d.smithi081.stdout:starting mds.d at 2024-02-21T23:12:21.321 INFO:tasks.ceph.mds.b.smithi081.stdout:starting mds.b at 2024-02-21T23:12:21.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.358+0000 7fba4f200640 1 Processor -- start 2024-02-21T23:12:21.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.358+0000 7fba4f200640 1 -- start start 2024-02-21T23:12:21.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.359+0000 7fba4f200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fba501524c0 0x7fba50155a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:21.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.359+0000 7fba4f200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fba500c5040 0x7fba5006ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:21.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.359+0000 7fba4f200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fba500c85c0 0x7fba500c70d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:21.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.359+0000 7fba4f200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba5006df50 con 0x7fba500c85c0 2024-02-21T23:12:21.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.359+0000 7fba4f200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba5006d3b0 con 0x7fba500c5040 2024-02-21T23:12:21.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.359+0000 7fba4f200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba500c5520 con 0x7fba501524c0 2024-02-21T23:12:21.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.359+0000 7fba4e800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fba500c85c0 0x7fba500c70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:21.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.359+0000 7fba4de00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fba501524c0 0x7fba50155a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:21.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.359+0000 7fba4e800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fba500c85c0 0x7fba500c70d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41650/0 (socket says 172.21.15.28:41650) 2024-02-21T23:12:21.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.359+0000 7fba4e800640 1 -- 172.21.15.28:0/1700679598 learned_addr learned my addr 172.21.15.28:0/1700679598 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:21.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.359+0000 7fba4d400640 1 --2- 172.21.15.28:0/1700679598 >> v2:172.21.15.81:3300/0 conn(0x7fba500c5040 0x7fba5006ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:21.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.360+0000 7fba4de00640 1 -- 172.21.15.28:0/1700679598 >> v2:172.21.15.81:3300/0 conn(0x7fba500c5040 msgr2=0x7fba5006ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:21.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.360+0000 7fba4de00640 1 --2- 172.21.15.28:0/1700679598 >> v2:172.21.15.81:3300/0 conn(0x7fba500c5040 0x7fba5006ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:21.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.360+0000 7fba4de00640 1 -- 172.21.15.28:0/1700679598 >> v2:172.21.15.28:3300/0 conn(0x7fba500c85c0 msgr2=0x7fba500c70d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:21.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.360+0000 7fba4de00640 1 --2- 172.21.15.28:0/1700679598 >> v2:172.21.15.28:3300/0 conn(0x7fba500c85c0 0x7fba500c70d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:21.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.360+0000 7fba4de00640 1 -- 172.21.15.28:0/1700679598 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba500c5800 con 0x7fba501524c0 2024-02-21T23:12:21.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.361+0000 7fba4de00640 1 --2- 172.21.15.28:0/1700679598 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fba501524c0 0x7fba50155a10 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fba500cb7c0 tx=0x7fba4002b710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=84b598f9c9c8384a server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:21.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.361+0000 7fba4ca00640 1 -- 172.21.15.28:0/1700679598 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba40037070 con 0x7fba501524c0 2024-02-21T23:12:21.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.361+0000 7fba4ca00640 1 -- 172.21.15.28:0/1700679598 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba40030710 con 0x7fba501524c0 2024-02-21T23:12:21.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.361+0000 7fba4ca00640 1 -- 172.21.15.28:0/1700679598 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba40030ae0 con 0x7fba501524c0 2024-02-21T23:12:21.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.362+0000 7fba4f200640 1 -- 172.21.15.28:0/1700679598 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fba501524c0 msgr2=0x7fba50155a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:21.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.362+0000 7fba4f200640 1 --2- 172.21.15.28:0/1700679598 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fba501524c0 0x7fba50155a10 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fba500cb7c0 tx=0x7fba4002b710 comp rx=0 tx=0).stop 2024-02-21T23:12:21.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.362+0000 7fba4f200640 1 -- 172.21.15.28:0/1700679598 shutdown_connections 2024-02-21T23:12:21.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.362+0000 7fba4f200640 1 --2- 172.21.15.28:0/1700679598 >> v2:172.21.15.28:3300/0 conn(0x7fba500c85c0 0x7fba500c70d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:21.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.362+0000 7fba4f200640 1 --2- 172.21.15.28:0/1700679598 >> v2:172.21.15.81:3300/0 conn(0x7fba500c5040 0x7fba5006ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:21.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.362+0000 7fba4f200640 1 --2- 172.21.15.28:0/1700679598 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fba501524c0 0x7fba50155a10 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:21.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.362+0000 7fba4f200640 1 -- 172.21.15.28:0/1700679598 >> 172.21.15.28:0/1700679598 conn(0x7fba500ac2b0 msgr2=0x7fba500ac6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:21.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.362+0000 7fba4f200640 1 -- 172.21.15.28:0/1700679598 shutdown_connections 2024-02-21T23:12:21.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.362+0000 7fba4f200640 1 -- 172.21.15.28:0/1700679598 wait complete. 2024-02-21T23:12:21.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.363+0000 7fba4f200640 1 Processor -- start 2024-02-21T23:12:21.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.364+0000 7fba4f200640 1 -- start start 2024-02-21T23:12:21.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.364+0000 7fba4f200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba500c5040 0x7fba500c32a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:21.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.364+0000 7fba4f200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fba500c85c0 0x7fba50156870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:21.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.364+0000 7fba4f200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fba501524c0 0x7fba501498c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:21.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.364+0000 7fba4f200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba501528a0 con 0x7fba500c5040 2024-02-21T23:12:21.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.364+0000 7fba4f200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba5006c950 con 0x7fba501524c0 2024-02-21T23:12:21.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.364+0000 7fba4f200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba500c8a70 con 0x7fba500c85c0 2024-02-21T23:12:21.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.365+0000 7fba4de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba500c5040 0x7fba500c32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:21.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.365+0000 7fba4de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba500c5040 0x7fba500c32a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41662/0 (socket says 172.21.15.28:41662) 2024-02-21T23:12:21.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.365+0000 7fba4de00640 1 -- 172.21.15.28:0/2760914591 learned_addr learned my addr 172.21.15.28:0/2760914591 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:21.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.365+0000 7fba4d400640 1 --2- 172.21.15.28:0/2760914591 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fba500c85c0 0x7fba50156870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:21.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.365+0000 7fba4e800640 1 --2- 172.21.15.28:0/2760914591 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fba501524c0 0x7fba501498c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:21.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.365+0000 7fba4de00640 1 -- 172.21.15.28:0/2760914591 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fba500c85c0 msgr2=0x7fba50156870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:21.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.365+0000 7fba4de00640 1 --2- 172.21.15.28:0/2760914591 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fba500c85c0 0x7fba50156870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:21.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.365+0000 7fba4de00640 1 -- 172.21.15.28:0/2760914591 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fba501524c0 msgr2=0x7fba501498c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:21.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.365+0000 7fba4de00640 1 --2- 172.21.15.28:0/2760914591 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fba501524c0 0x7fba501498c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:21.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.365+0000 7fba4de00640 1 -- 172.21.15.28:0/2760914591 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba500c3ae0 con 0x7fba500c5040 2024-02-21T23:12:21.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.365+0000 7fba4e800640 1 --2- 172.21.15.28:0/2760914591 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fba501524c0 0x7fba501498c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:12:21.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.365+0000 7fba4de00640 1 --2- 172.21.15.28:0/2760914591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba500c5040 0x7fba500c32a0 secure :-1 s=READY pgs=2212 cs=0 l=1 rev1=1 crypto rx=0x7fba4002bd20 tx=0x7fba400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:21.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.366+0000 7fba46000640 1 -- 172.21.15.28:0/2760914591 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba40042070 con 0x7fba500c5040 2024-02-21T23:12:21.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.366+0000 7fba46000640 1 -- 172.21.15.28:0/2760914591 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba4003e890 con 0x7fba500c5040 2024-02-21T23:12:21.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.366+0000 7fba46000640 1 -- 172.21.15.28:0/2760914591 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba40037040 con 0x7fba500c5040 2024-02-21T23:12:21.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.366+0000 7fba4f200640 1 -- 172.21.15.28:0/2760914591 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba500c3dc0 con 0x7fba500c5040 2024-02-21T23:12:21.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.366+0000 7fba4f200640 1 -- 172.21.15.28:0/2760914591 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba500c4350 con 0x7fba500c5040 2024-02-21T23:12:21.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.367+0000 7fba4f200640 1 -- 172.21.15.28:0/2760914591 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba5004f290 con 0x7fba500c5040 2024-02-21T23:12:21.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.368+0000 7fba46000640 1 -- 172.21.15.28:0/2760914591 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fba40045070 con 0x7fba500c5040 2024-02-21T23:12:21.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.369+0000 7fba46000640 1 --2- 172.21.15.28:0/2760914591 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fba3006b4d0 0x7fba3006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:21.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.369+0000 7fba46000640 1 -- 172.21.15.28:0/2760914591 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5421+0+0 (secure 0 0 0) 0x7fba400abc90 con 0x7fba500c5040 2024-02-21T23:12:21.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.369+0000 7fba4d400640 1 --2- 172.21.15.28:0/2760914591 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fba3006b4d0 0x7fba3006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:21.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.370+0000 7fba4d400640 1 --2- 172.21.15.28:0/2760914591 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fba3006b4d0 0x7fba3006d990 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fba500aca70 tx=0x7fba3c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:21.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.373+0000 7fba46000640 1 -- 172.21.15.28:0/2760914591 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fba4007ad40 con 0x7fba500c5040 2024-02-21T23:12:21.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.586+0000 7fba4f200640 1 -- 172.21.15.28:0/2760914591 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fba500ac7c0 con 0x7fba500c5040 2024-02-21T23:12:21.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.739+0000 7fba46000640 1 -- 172.21.15.28:0/2760914591 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v191) v1 ==== 97+0+0 (secure 0 0 0) 0x7fba4007a640 con 0x7fba500c5040 2024-02-21T23:12:21.727 INFO:teuthology.orchestra.run.smithi028.stderr: removed all blocklist entries 2024-02-21T23:12:21.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.742+0000 7fba4f200640 1 -- 172.21.15.28:0/2760914591 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fba3006b4d0 msgr2=0x7fba3006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:21.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.742+0000 7fba4f200640 1 --2- 172.21.15.28:0/2760914591 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fba3006b4d0 0x7fba3006d990 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fba500aca70 tx=0x7fba3c008040 comp rx=0 tx=0).stop 2024-02-21T23:12:21.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.742+0000 7fba4f200640 1 -- 172.21.15.28:0/2760914591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba500c5040 msgr2=0x7fba500c32a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:21.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.742+0000 7fba4f200640 1 --2- 172.21.15.28:0/2760914591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba500c5040 0x7fba500c32a0 secure :-1 s=READY pgs=2212 cs=0 l=1 rev1=1 crypto rx=0x7fba4002bd20 tx=0x7fba400046c0 comp rx=0 tx=0).stop 2024-02-21T23:12:21.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.742+0000 7fba4f200640 1 -- 172.21.15.28:0/2760914591 shutdown_connections 2024-02-21T23:12:21.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.742+0000 7fba4f200640 1 --2- 172.21.15.28:0/2760914591 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fba3006b4d0 0x7fba3006d990 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:21.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.742+0000 7fba4f200640 1 --2- 172.21.15.28:0/2760914591 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fba500c85c0 0x7fba50156870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:21.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.742+0000 7fba4f200640 1 --2- 172.21.15.28:0/2760914591 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fba501524c0 0x7fba501498c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:21.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.742+0000 7fba4f200640 1 --2- 172.21.15.28:0/2760914591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba500c5040 0x7fba500c32a0 unknown :-1 s=CLOSED pgs=2212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:21.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.742+0000 7fba4f200640 1 -- 172.21.15.28:0/2760914591 >> 172.21.15.28:0/2760914591 conn(0x7fba500ac2b0 msgr2=0x7fba50154e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:21.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.743+0000 7fba4f200640 1 -- 172.21.15.28:0/2760914591 shutdown_connections 2024-02-21T23:12:21.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.743+0000 7fba4f200640 1 -- 172.21.15.28:0/2760914591 wait complete. 2024-02-21T23:12:21.752 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-02-21T23:12:21.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.978+0000 7f2dbf200640 1 Processor -- start 2024-02-21T23:12:21.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.978+0000 7f2dbf200640 1 -- start start 2024-02-21T23:12:21.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.980+0000 7f2dbf200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2dc0156a90 0x7f2dc015c860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:21.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.980+0000 7f2dbf200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2dc00c29a0 0x7f2dc006e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:21.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.980+0000 7f2dbf200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dc015cf60 con 0x7f2dc006e550 2024-02-21T23:12:21.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.980+0000 7f2dbf200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dc00c7440 con 0x7f2dc0156a90 2024-02-21T23:12:21.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.980+0000 7f2dbf200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dc015d240 con 0x7f2dc00c29a0 2024-02-21T23:12:21.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.980+0000 7f2dbde00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2dc0156a90 0x7f2dc015c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:21.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.981+0000 7f2dbe800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f2dc006e550 0x7f2dc0071730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39702/0 (socket says 172.21.15.28:39702) 2024-02-21T23:12:21.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.981+0000 7f2dbde00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2dc0156a90 0x7f2dc015c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41666/0 (socket says 172.21.15.28:41666) 2024-02-21T23:12:21.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.981+0000 7f2dbe800640 1 -- 172.21.15.28:0/1523183844 learned_addr learned my addr 172.21.15.28:0/1523183844 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:12:21.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.981+0000 7f2dbd400640 1 --2- 172.21.15.28:0/1523183844 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2dc00c29a0 0x7f2dc006e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:21.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.981+0000 7f2dbde00640 1 -- 172.21.15.28:0/1523183844 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2dc00c29a0 msgr2=0x7f2dc006e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:21.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.981+0000 7f2dbde00640 1 --2- 172.21.15.28:0/1523183844 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2dc00c29a0 0x7f2dc006e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:21.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.981+0000 7f2dbde00640 1 -- 172.21.15.28:0/1523183844 >> v1:172.21.15.28:6789/0 conn(0x7f2dc006e550 legacy=0x7f2dc0071730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:21.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.981+0000 7f2dbde00640 1 -- 172.21.15.28:0/1523183844 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dc015d520 con 0x7f2dc0156a90 2024-02-21T23:12:21.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.981+0000 7f2dbca00640 1 -- 172.21.15.28:0/1523183844 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2489726200 0 0) 0x7f2dc015cf60 con 0x7f2dc006e550 2024-02-21T23:12:21.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.982+0000 7f2dbde00640 1 --2- 172.21.15.28:0/1523183844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2dc0156a90 0x7f2dc015c860 secure :-1 s=READY pgs=2214 cs=0 l=1 rev1=1 crypto rx=0x7f2da80097a0 tx=0x7f2da802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82ff0a8ee0224a9a server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:21.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.983+0000 7f2dbca00640 1 -- 172.21.15.28:0/1523183844 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da8035020 con 0x7f2dc0156a90 2024-02-21T23:12:21.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.983+0000 7f2dbca00640 1 -- 172.21.15.28:0/1523183844 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2da8031810 con 0x7f2dc0156a90 2024-02-21T23:12:21.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.983+0000 7f2dbca00640 1 -- 172.21.15.28:0/1523183844 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da8031be0 con 0x7f2dc0156a90 2024-02-21T23:12:21.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.983+0000 7f2dbf200640 1 -- 172.21.15.28:0/1523183844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2dc0156a90 msgr2=0x7f2dc015c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:21.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.983+0000 7f2dbf200640 1 --2- 172.21.15.28:0/1523183844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2dc0156a90 0x7f2dc015c860 secure :-1 s=READY pgs=2214 cs=0 l=1 rev1=1 crypto rx=0x7f2da80097a0 tx=0x7f2da802b8b0 comp rx=0 tx=0).stop 2024-02-21T23:12:21.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.983+0000 7f2dbf200640 1 -- 172.21.15.28:0/1523183844 shutdown_connections 2024-02-21T23:12:21.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.983+0000 7f2dbf200640 1 --2- 172.21.15.28:0/1523183844 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2dc00c29a0 0x7f2dc006e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:21.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.983+0000 7f2dbf200640 1 --2- 172.21.15.28:0/1523183844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2dc0156a90 0x7f2dc015c860 unknown :-1 s=CLOSED pgs=2214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:21.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.983+0000 7f2dbf200640 1 -- 172.21.15.28:0/1523183844 >> 172.21.15.28:0/1523183844 conn(0x7f2dc006b550 msgr2=0x7f2dc00c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:21.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.983+0000 7f2dbf200640 1 -- 172.21.15.28:0/1523183844 shutdown_connections 2024-02-21T23:12:21.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.983+0000 7f2dbf200640 1 -- 172.21.15.28:0/1523183844 wait complete. 2024-02-21T23:12:21.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.985+0000 7f2dbf200640 1 Processor -- start 2024-02-21T23:12:21.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.985+0000 7f2dbf200640 1 -- start start 2024-02-21T23:12:21.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.986+0000 7f2dbf200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2dc006e550 0x7f2dc0158840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:21.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.986+0000 7f2dbf200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2dc00c29a0 0x7f2dc00a3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:21.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.986+0000 7f2dbf200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2dc0156a90 0x7f2dc0152770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:21.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.986+0000 7f2dbf200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dc0071e40 con 0x7f2dc00c29a0 2024-02-21T23:12:21.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.986+0000 7f2dbf200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dc0156e70 con 0x7f2dc0156a90 2024-02-21T23:12:21.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.986+0000 7f2dbf200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dc0071fc0 con 0x7f2dc006e550 2024-02-21T23:12:21.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.986+0000 7f2dbd400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2dc00c29a0 0x7f2dc00a3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:21.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.986+0000 7f2dbd400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2dc00c29a0 0x7f2dc00a3ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41682/0 (socket says 172.21.15.28:41682) 2024-02-21T23:12:21.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.986+0000 7f2dbd400640 1 -- 172.21.15.28:0/3547290816 learned_addr learned my addr 172.21.15.28:0/3547290816 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:21.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.987+0000 7f2dbde00640 1 --2- 172.21.15.28:0/3547290816 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2dc006e550 0x7f2dc0158840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:21.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.987+0000 7f2dbe800640 1 --2- 172.21.15.28:0/3547290816 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2dc0156a90 0x7f2dc0152770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:21.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.987+0000 7f2dbd400640 1 -- 172.21.15.28:0/3547290816 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2dc006e550 msgr2=0x7f2dc0158840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:21.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.987+0000 7f2dbd400640 1 --2- 172.21.15.28:0/3547290816 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2dc006e550 0x7f2dc0158840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:21.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.987+0000 7f2dbd400640 1 -- 172.21.15.28:0/3547290816 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2dc0156a90 msgr2=0x7f2dc0152770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:21.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.987+0000 7f2dbd400640 1 --2- 172.21.15.28:0/3547290816 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2dc0156a90 0x7f2dc0152770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:21.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.987+0000 7f2dbd400640 1 -- 172.21.15.28:0/3547290816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dc01ee8c0 con 0x7f2dc00c29a0 2024-02-21T23:12:21.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.987+0000 7f2dbd400640 1 --2- 172.21.15.28:0/3547290816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2dc00c29a0 0x7f2dc00a3ff0 secure :-1 s=READY pgs=2215 cs=0 l=1 rev1=1 crypto rx=0x7f2db000d800 tx=0x7f2db000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:21.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.987+0000 7f2db6000640 1 -- 172.21.15.28:0/3547290816 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2db000b6c0 con 0x7f2dc00c29a0 2024-02-21T23:12:21.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.987+0000 7f2db6000640 1 -- 172.21.15.28:0/3547290816 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2db0019070 con 0x7f2dc00c29a0 2024-02-21T23:12:21.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.988+0000 7f2db6000640 1 -- 172.21.15.28:0/3547290816 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2db000ba80 con 0x7f2dc00c29a0 2024-02-21T23:12:21.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.988+0000 7f2dbf200640 1 -- 172.21.15.28:0/3547290816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dc01eeba0 con 0x7f2dc00c29a0 2024-02-21T23:12:21.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.988+0000 7f2dbf200640 1 -- 172.21.15.28:0/3547290816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dc01ef160 con 0x7f2dc00c29a0 2024-02-21T23:12:21.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.989+0000 7f2dbf200640 1 -- 172.21.15.28:0/3547290816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dc004f290 con 0x7f2dc00c29a0 2024-02-21T23:12:21.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.990+0000 7f2db6000640 1 -- 172.21.15.28:0/3547290816 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2db000bc20 con 0x7f2dc00c29a0 2024-02-21T23:12:21.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.991+0000 7f2db6000640 1 --2- 172.21.15.28:0/3547290816 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2d9c06b4d0 0x7f2d9c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:21.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.991+0000 7f2db6000640 1 -- 172.21.15.28:0/3547290816 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f2db008a530 con 0x7f2dc00c29a0 2024-02-21T23:12:21.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.992+0000 7f2dbde00640 1 --2- 172.21.15.28:0/3547290816 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2d9c06b4d0 0x7f2d9c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:21.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.993+0000 7f2dbde00640 1 --2- 172.21.15.28:0/3547290816 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2d9c06b4d0 0x7f2d9c06d990 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f2da8036000 tx=0x7f2da803c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:21.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:21.996+0000 7f2db6000640 1 -- 172.21.15.28:0/3547290816 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f2db0059560 con 0x7f2dc00c29a0 2024-02-21T23:12:22.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.267+0000 7f2dbf200640 1 -- 172.21.15.28:0/3547290816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f2dc01562b0 con 0x7f2dc00c29a0 2024-02-21T23:12:22.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.268+0000 7f2db6000640 1 -- 172.21.15.28:0/3547290816 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v21) v1 ==== 80+0+4724 (secure 0 0 0) 0x7f2db0058e60 con 0x7f2dc00c29a0 2024-02-21T23:12:22.256 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:12:22.256 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-21T23:12:22.256 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_dump": [ 2024-02-21T23:12:22.256 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:12:22.256 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.a", 2024-02-21T23:12:22.256 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDSgNZl4FY1KRAAJSK5Ibb0ibKiKXYpTLgtmQ==", 2024-02-21T23:12:22.256 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:12:22.256 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:12:22.256 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-02-21T23:12:22.256 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-02-21T23:12:22.256 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:12:22.257 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:12:22.257 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:12:22.257 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:12:22.257 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.b", 2024-02-21T23:12:22.257 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDTgNZlwRcAERAAgsYmmkuobKelBEJVCRpXUg==", 2024-02-21T23:12:22.257 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:12:22.257 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:12:22.257 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-02-21T23:12:22.257 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-02-21T23:12:22.257 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:12:22.257 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:12:22.258 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:12:22.258 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:12:22.258 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.c", 2024-02-21T23:12:22.258 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDSgNZloFpQOxAAK2WBArXRVNTgqH9WWDEPDA==", 2024-02-21T23:12:22.258 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:12:22.258 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:12:22.258 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-02-21T23:12:22.258 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-02-21T23:12:22.258 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:12:22.258 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:12:22.258 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:12:22.258 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:12:22.258 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.d", 2024-02-21T23:12:22.259 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDTgNZlblOmIRAAxe2HfSKGSkkzJDN4aVdzgQ==", 2024-02-21T23:12:22.259 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:12:22.259 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:12:22.259 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-02-21T23:12:22.259 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-02-21T23:12:22.259 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:12:22.259 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:12:22.259 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:12:22.259 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:12:22.259 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.0", 2024-02-21T23:12:22.259 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDVgNZlxNJbLxAAXot7Q3chrLEBsFNO8kT6QQ==", 2024-02-21T23:12:22.260 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:12:22.260 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:12:22.260 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:12:22.260 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:12:22.260 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:12:22.260 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:12:22.260 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:12:22.260 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.1", 2024-02-21T23:12:22.260 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDYgNZlZZfWORAAr4XkOxLYLmumVbu362N2HQ==", 2024-02-21T23:12:22.260 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:12:22.260 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:12:22.260 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:12:22.261 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:12:22.261 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:12:22.261 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:12:22.261 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:12:22.261 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.2", 2024-02-21T23:12:22.261 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDcgNZl9mcDGBAAIZGOF8IUMenwo386x95mwg==", 2024-02-21T23:12:22.261 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:12:22.261 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:12:22.261 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:12:22.262 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:12:22.262 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:12:22.262 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:12:22.262 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:12:22.262 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.3", 2024-02-21T23:12:22.262 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDfgNZl8ALSMBAAag9cmGH0/4xALWSUmrZowg==", 2024-02-21T23:12:22.262 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:12:22.262 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:12:22.262 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:12:22.262 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:12:22.262 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:12:22.263 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:12:22.263 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:12:22.263 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.4", 2024-02-21T23:12:22.263 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDngNZlBgeKABAAeW99RVfCBvB1lp7w7mQBlQ==", 2024-02-21T23:12:22.263 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:12:22.263 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:12:22.263 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:12:22.263 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:12:22.263 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:12:22.263 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:12:22.263 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:12:22.263 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.5", 2024-02-21T23:12:22.263 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDqgNZljQG3ChAAY/IZEIcvcGNvlFGIWib3gg==", 2024-02-21T23:12:22.263 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:12:22.264 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:12:22.264 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:12:22.264 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:12:22.264 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:12:22.264 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:12:22.264 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:12:22.264 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.6", 2024-02-21T23:12:22.264 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDtgNZlKnakIxAAkW5xlkshzvEHJdmpgPxR+g==", 2024-02-21T23:12:22.264 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:12:22.264 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:12:22.264 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:12:22.264 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:12:22.265 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:12:22.265 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:12:22.265 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:12:22.265 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.7", 2024-02-21T23:12:22.265 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDxgNZlTWxYARAASX1/gmqO+8mQPP8uA4xjiQ==", 2024-02-21T23:12:22.265 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:12:22.265 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:12:22.265 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:12:22.265 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:12:22.265 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:12:22.265 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:12:22.265 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:12:22.265 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.0", 2024-02-21T23:12:22.265 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDTgNZlILvYLxAAoREWxKKT7YcXnhkgk8YvoQ==", 2024-02-21T23:12:22.266 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:12:22.266 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:12:22.266 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow r", 2024-02-21T23:12:22.266 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-02-21T23:12:22.266 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:12:22.266 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:12:22.266 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:12:22.266 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.1", 2024-02-21T23:12:22.266 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDTgNZlm7ImOxAA8iRcXlzFPZPaXQfsKhuO/g==", 2024-02-21T23:12:22.266 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:12:22.266 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:12:22.266 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-02-21T23:12:22.266 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-02-21T23:12:22.266 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-02-21T23:12:22.267 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:12:22.267 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:12:22.267 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:12:22.267 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.admin", 2024-02-21T23:12:22.267 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDRgNZlBYd1GhAAbK5qBCIkf0Ej5nEyPnTxIQ==", 2024-02-21T23:12:22.267 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:12:22.267 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-02-21T23:12:22.267 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-02-21T23:12:22.267 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-02-21T23:12:22.267 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:12:22.267 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:12:22.267 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:12:22.267 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:12:22.267 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.x", 2024-02-21T23:12:22.267 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDSgNZlJYKeHBAAYPskmLRr7Jw44TdKzM/JMg==", 2024-02-21T23:12:22.268 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:12:22.268 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-02-21T23:12:22.268 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-02-21T23:12:22.268 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:12:22.268 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:12:22.268 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:12:22.268 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:12:22.268 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.y", 2024-02-21T23:12:22.268 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDSgNZlSRkOEBAAOhRj9PsZDpjoV6WT4drrZA==", 2024-02-21T23:12:22.268 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:12:22.268 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-02-21T23:12:22.268 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-02-21T23:12:22.268 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:12:22.268 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:12:22.269 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:12:22.269 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-21T23:12:22.269 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-21T23:12:22.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.270+0000 7f2dbf200640 1 -- 172.21.15.28:0/3547290816 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2d9c06b4d0 msgr2=0x7f2d9c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:22.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.270+0000 7f2dbf200640 1 --2- 172.21.15.28:0/3547290816 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2d9c06b4d0 0x7f2d9c06d990 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f2da8036000 tx=0x7f2da803c040 comp rx=0 tx=0).stop 2024-02-21T23:12:22.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.270+0000 7f2dbf200640 1 -- 172.21.15.28:0/3547290816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2dc00c29a0 msgr2=0x7f2dc00a3ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:22.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.270+0000 7f2dbf200640 1 --2- 172.21.15.28:0/3547290816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2dc00c29a0 0x7f2dc00a3ff0 secure :-1 s=READY pgs=2215 cs=0 l=1 rev1=1 crypto rx=0x7f2db000d800 tx=0x7f2db000dcc0 comp rx=0 tx=0).stop 2024-02-21T23:12:22.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.271+0000 7f2dbf200640 1 -- 172.21.15.28:0/3547290816 shutdown_connections 2024-02-21T23:12:22.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.271+0000 7f2dbf200640 1 --2- 172.21.15.28:0/3547290816 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2d9c06b4d0 0x7f2d9c06d990 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:22.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.271+0000 7f2dbf200640 1 --2- 172.21.15.28:0/3547290816 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2dc0156a90 0x7f2dc0152770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:22.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.271+0000 7f2dbf200640 1 --2- 172.21.15.28:0/3547290816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2dc00c29a0 0x7f2dc00a3ff0 unknown :-1 s=CLOSED pgs=2215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:22.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.271+0000 7f2dbf200640 1 --2- 172.21.15.28:0/3547290816 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2dc006e550 0x7f2dc0158840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:22.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.271+0000 7f2dbf200640 1 -- 172.21.15.28:0/3547290816 >> 172.21.15.28:0/3547290816 conn(0x7f2dc006b550 msgr2=0x7f2dc0153e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:22.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.271+0000 7f2dbf200640 1 -- 172.21.15.28:0/3547290816 shutdown_connections 2024-02-21T23:12:22.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.271+0000 7f2dbf200640 1 -- 172.21.15.28:0/3547290816 wait complete. 2024-02-21T23:12:22.278 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:12:22.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.511+0000 7f0d22a00640 1 Processor -- start 2024-02-21T23:12:22.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.511+0000 7f0d22a00640 1 -- start start 2024-02-21T23:12:22.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.512+0000 7f0d22a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0d1c072430 0x7f0d1c0c3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:22.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.512+0000 7f0d22a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0d1c155860 0x7f0d1c155d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:22.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.512+0000 7f0d22a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d1c0c6560 con 0x7f0d1c0c3280 2024-02-21T23:12:22.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.512+0000 7f0d22a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d1c0c5cc0 con 0x7f0d1c155860 2024-02-21T23:12:22.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.512+0000 7f0d22a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d1c0c5fa0 con 0x7f0d1c072430 2024-02-21T23:12:22.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.513+0000 7f0d20a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0d1c155860 0x7f0d1c155d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:22.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.513+0000 7f0d13e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0d1c072430 0x7f0d1c0c3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:22.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.513+0000 7f0d13e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0d1c072430 0x7f0d1c0c3660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3301/0 says I am v2:172.21.15.28:33584/0 (socket says 172.21.15.28:33584) 2024-02-21T23:12:22.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.513+0000 7f0d13e00640 1 -- 172.21.15.28:0/536935854 learned_addr learned my addr 172.21.15.28:0/536935854 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:22.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.514+0000 7f0d13e00640 1 -- 172.21.15.28:0/536935854 >> v2:172.21.15.81:3300/0 conn(0x7f0d1c155860 msgr2=0x7f0d1c155d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:22.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.514+0000 7f0d13e00640 1 --2- 172.21.15.28:0/536935854 >> v2:172.21.15.81:3300/0 conn(0x7f0d1c155860 0x7f0d1c155d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:22.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.514+0000 7f0d13e00640 1 -- 172.21.15.28:0/536935854 >> v1:172.21.15.81:6789/0 conn(0x7f0d1c0c3280 legacy=0x7f0d1c1526e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:22.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.514+0000 7f0d13e00640 1 -- 172.21.15.28:0/536935854 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d1c0c6280 con 0x7f0d1c072430 2024-02-21T23:12:22.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.515+0000 7f0d13e00640 1 --2- 172.21.15.28:0/536935854 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0d1c072430 0x7f0d1c0c3660 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f0d04009870 tx=0x7f0d0402db00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a88aa8f9aca6117 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:22.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.515+0000 7f0d1b400640 1 -- 172.21.15.28:0/536935854 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d0402f8c0 con 0x7f0d1c072430 2024-02-21T23:12:22.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.515+0000 7f0d1b400640 1 -- 172.21.15.28:0/536935854 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d0402fa60 con 0x7f0d1c072430 2024-02-21T23:12:22.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.515+0000 7f0d1b400640 1 -- 172.21.15.28:0/536935854 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d0402fe30 con 0x7f0d1c072430 2024-02-21T23:12:22.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.516+0000 7f0d22a00640 1 -- 172.21.15.28:0/536935854 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0d1c072430 msgr2=0x7f0d1c0c3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:22.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.516+0000 7f0d22a00640 1 --2- 172.21.15.28:0/536935854 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0d1c072430 0x7f0d1c0c3660 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f0d04009870 tx=0x7f0d0402db00 comp rx=0 tx=0).stop 2024-02-21T23:12:22.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.516+0000 7f0d22a00640 1 -- 172.21.15.28:0/536935854 shutdown_connections 2024-02-21T23:12:22.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.516+0000 7f0d22a00640 1 --2- 172.21.15.28:0/536935854 >> v2:172.21.15.81:3300/0 conn(0x7f0d1c155860 0x7f0d1c155d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:22.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.516+0000 7f0d22a00640 1 --2- 172.21.15.28:0/536935854 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0d1c072430 0x7f0d1c0c3660 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:22.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.516+0000 7f0d22a00640 1 -- 172.21.15.28:0/536935854 >> 172.21.15.28:0/536935854 conn(0x7f0d1c0c67f0 msgr2=0x7f0d1c0c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:22.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.516+0000 7f0d22a00640 1 -- 172.21.15.28:0/536935854 shutdown_connections 2024-02-21T23:12:22.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.516+0000 7f0d22a00640 1 -- 172.21.15.28:0/536935854 wait complete. 2024-02-21T23:12:22.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.518+0000 7f0d22a00640 1 Processor -- start 2024-02-21T23:12:22.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.518+0000 7f0d22a00640 1 -- start start 2024-02-21T23:12:22.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.519+0000 7f0d22a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0d1c0c3280 0x7f0d1c0bfe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:22.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.519+0000 7f0d22a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d1c155860 0x7f0d1c0ab940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:22.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.519+0000 7f0d22a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d1c0c2310 0x7f0d1c0abe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:22.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.519+0000 7f0d22a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d1c0c5aa0 con 0x7f0d1c0c2310 2024-02-21T23:12:22.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.519+0000 7f0d22a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d1c0c5920 con 0x7f0d1c155860 2024-02-21T23:12:22.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.519+0000 7f0d22a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d1c0733e0 con 0x7f0d1c0c3280 2024-02-21T23:12:22.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.519+0000 7f0d20a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d1c0c2310 0x7f0d1c0abe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:22.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.519+0000 7f0d20a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d1c0c2310 0x7f0d1c0abe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41692/0 (socket says 172.21.15.28:41692) 2024-02-21T23:12:22.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.519+0000 7f0d20a00640 1 -- 172.21.15.28:0/438365843 learned_addr learned my addr 172.21.15.28:0/438365843 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:22.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.520+0000 7f0d13e00640 1 --2- 172.21.15.28:0/438365843 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d1c155860 0x7f0d1c0ab940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:22.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.520+0000 7f0d1be00640 1 --2- 172.21.15.28:0/438365843 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0d1c0c3280 0x7f0d1c0bfe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:22.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.520+0000 7f0d20a00640 1 -- 172.21.15.28:0/438365843 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0d1c0c3280 msgr2=0x7f0d1c0bfe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:22.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.520+0000 7f0d20a00640 1 --2- 172.21.15.28:0/438365843 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0d1c0c3280 0x7f0d1c0bfe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:22.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.520+0000 7f0d20a00640 1 -- 172.21.15.28:0/438365843 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d1c155860 msgr2=0x7f0d1c0ab940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:22.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.520+0000 7f0d20a00640 1 --2- 172.21.15.28:0/438365843 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d1c155860 0x7f0d1c0ab940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:22.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.520+0000 7f0d20a00640 1 -- 172.21.15.28:0/438365843 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d1c1eb620 con 0x7f0d1c0c2310 2024-02-21T23:12:22.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.520+0000 7f0d20a00640 1 --2- 172.21.15.28:0/438365843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d1c0c2310 0x7f0d1c0abe80 secure :-1 s=READY pgs=2216 cs=0 l=1 rev1=1 crypto rx=0x7f0d14009dd0 tx=0x7f0d1400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:22.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.520+0000 7f0d18c00640 1 -- 172.21.15.28:0/438365843 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d14018020 con 0x7f0d1c0c2310 2024-02-21T23:12:22.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.520+0000 7f0d18c00640 1 -- 172.21.15.28:0/438365843 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d14014480 con 0x7f0d1c0c2310 2024-02-21T23:12:22.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.520+0000 7f0d18c00640 1 -- 172.21.15.28:0/438365843 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d14014870 con 0x7f0d1c0c2310 2024-02-21T23:12:22.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.521+0000 7f0d22a00640 1 -- 172.21.15.28:0/438365843 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d1c1eb900 con 0x7f0d1c0c2310 2024-02-21T23:12:22.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.521+0000 7f0d22a00640 1 -- 172.21.15.28:0/438365843 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d1c1ddde0 con 0x7f0d1c0c2310 2024-02-21T23:12:22.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.522+0000 7f0d22a00640 1 -- 172.21.15.28:0/438365843 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d1c1ebad0 con 0x7f0d1c0c2310 2024-02-21T23:12:22.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.523+0000 7f0d18c00640 1 -- 172.21.15.28:0/438365843 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0d14005d10 con 0x7f0d1c0c2310 2024-02-21T23:12:22.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.524+0000 7f0d18c00640 1 --2- 172.21.15.28:0/438365843 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0cf806b4d0 0x7f0cf806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:22.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.524+0000 7f0d18c00640 1 -- 172.21.15.28:0/438365843 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f0d1408ac00 con 0x7f0d1c0c2310 2024-02-21T23:12:22.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.524+0000 7f0d1be00640 1 --2- 172.21.15.28:0/438365843 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0cf806b4d0 0x7f0cf806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:22.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.525+0000 7f0d1be00640 1 --2- 172.21.15.28:0/438365843 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0cf806b4d0 0x7f0cf806d990 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c001640 tx=0x7f0d0c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:22.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.528+0000 7f0d18c00640 1 -- 172.21.15.28:0/438365843 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f0d14022070 con 0x7f0d1c0c2310 2024-02-21T23:12:22.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.728+0000 7f0d22a00640 1 -- 172.21.15.28:0/438365843 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0d1c067f30 con 0x7f0d1c0c2310 2024-02-21T23:12:22.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.730+0000 7f0d18c00640 1 -- 172.21.15.28:0/438365843 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v193) v1 ==== 74+0+13114 (secure 0 0 0) 0x7f0d14010040 con 0x7f0d1c0c2310 2024-02-21T23:12:22.717 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:12:22.718 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":193,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:12:21.751582+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:12:22.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.732+0000 7f0d22a00640 1 -- 172.21.15.28:0/438365843 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0cf806b4d0 msgr2=0x7f0cf806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:22.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.732+0000 7f0d22a00640 1 --2- 172.21.15.28:0/438365843 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0cf806b4d0 0x7f0cf806d990 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c001640 tx=0x7f0d0c009040 comp rx=0 tx=0).stop 2024-02-21T23:12:22.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.732+0000 7f0d22a00640 1 -- 172.21.15.28:0/438365843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d1c0c2310 msgr2=0x7f0d1c0abe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:22.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.732+0000 7f0d22a00640 1 --2- 172.21.15.28:0/438365843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d1c0c2310 0x7f0d1c0abe80 secure :-1 s=READY pgs=2216 cs=0 l=1 rev1=1 crypto rx=0x7f0d14009dd0 tx=0x7f0d1400c490 comp rx=0 tx=0).stop 2024-02-21T23:12:22.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.733+0000 7f0d22a00640 1 -- 172.21.15.28:0/438365843 shutdown_connections 2024-02-21T23:12:22.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.733+0000 7f0d22a00640 1 --2- 172.21.15.28:0/438365843 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0cf806b4d0 0x7f0cf806d990 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:22.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.733+0000 7f0d22a00640 1 --2- 172.21.15.28:0/438365843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d1c0c2310 0x7f0d1c0abe80 unknown :-1 s=CLOSED pgs=2216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:22.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.733+0000 7f0d22a00640 1 --2- 172.21.15.28:0/438365843 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0d1c155860 0x7f0d1c0ab940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:22.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.733+0000 7f0d22a00640 1 --2- 172.21.15.28:0/438365843 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0d1c0c3280 0x7f0d1c0bfe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:22.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.733+0000 7f0d22a00640 1 -- 172.21.15.28:0/438365843 >> 172.21.15.28:0/438365843 conn(0x7f0d1c0c67f0 msgr2=0x7f0d1c1536a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:22.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.733+0000 7f0d22a00640 1 -- 172.21.15.28:0/438365843 shutdown_connections 2024-02-21T23:12:22.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.733+0000 7f0d22a00640 1 -- 172.21.15.28:0/438365843 wait complete. 2024-02-21T23:12:22.742 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-21T23:02:30.728329+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-21T23:12:22.743 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-21T23:12:22.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.970+0000 7f31b1e00640 1 Processor -- start 2024-02-21T23:12:22.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.971+0000 7f31b1e00640 1 -- start start 2024-02-21T23:12:22.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.972+0000 7f31b1e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f31ac1525a0 0x7f31ac0c86a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:22.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.972+0000 7f31b1e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f31ac158cf0 0x7f31ac0c7120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:22.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.972+0000 7f31b1e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31ac0c5120 con 0x7f31ac06dd20 2024-02-21T23:12:22.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.972+0000 7f31b1e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31ac158530 con 0x7f31ac158cf0 2024-02-21T23:12:22.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.972+0000 7f31b1e00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31ac0c53a0 con 0x7f31ac1525a0 2024-02-21T23:12:22.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.973+0000 7f31b1400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f31ac158cf0 0x7f31ac0c7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:22.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.973+0000 7f31b1400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f31ac158cf0 0x7f31ac0c7120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41706/0 (socket says 172.21.15.28:41706) 2024-02-21T23:12:22.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.973+0000 7f31b1400640 1 -- 172.21.15.28:0/2120929148 learned_addr learned my addr 172.21.15.28:0/2120929148 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:22.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.973+0000 7f31b0a00640 1 --2- 172.21.15.28:0/2120929148 >> v2:172.21.15.81:3300/0 conn(0x7f31ac1525a0 0x7f31ac0c86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:22.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.973+0000 7f31b1400640 1 -- 172.21.15.28:0/2120929148 >> v2:172.21.15.81:3300/0 conn(0x7f31ac1525a0 msgr2=0x7f31ac0c86a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:22.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.973+0000 7f31b1400640 1 --2- 172.21.15.28:0/2120929148 >> v2:172.21.15.81:3300/0 conn(0x7f31ac1525a0 0x7f31ac0c86a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:22.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.973+0000 7f31b1400640 1 -- 172.21.15.28:0/2120929148 >> v1:172.21.15.81:6789/0 conn(0x7f31ac06dd20 legacy=0x7f31ac155920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:22.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.973+0000 7f31b1400640 1 -- 172.21.15.28:0/2120929148 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31ac0c5680 con 0x7f31ac158cf0 2024-02-21T23:12:22.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.974+0000 7f31b1400640 1 --2- 172.21.15.28:0/2120929148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31ac158cf0 0x7f31ac0c7120 secure :-1 s=READY pgs=2217 cs=0 l=1 rev1=1 crypto rx=0x7f31ac0c16d0 tx=0x7f31a402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ae8d3d92bb74fb1 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:22.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.975+0000 7f31ab400640 1 -- 172.21.15.28:0/2120929148 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a40317c0 con 0x7f31ac158cf0 2024-02-21T23:12:22.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.975+0000 7f31ab400640 1 -- 172.21.15.28:0/2120929148 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31a4037070 con 0x7f31ac158cf0 2024-02-21T23:12:22.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.975+0000 7f31ab400640 1 -- 172.21.15.28:0/2120929148 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a4031b80 con 0x7f31ac158cf0 2024-02-21T23:12:22.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.975+0000 7f31b1e00640 1 -- 172.21.15.28:0/2120929148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31ac158cf0 msgr2=0x7f31ac0c7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:22.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.975+0000 7f31b1e00640 1 --2- 172.21.15.28:0/2120929148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31ac158cf0 0x7f31ac0c7120 secure :-1 s=READY pgs=2217 cs=0 l=1 rev1=1 crypto rx=0x7f31ac0c16d0 tx=0x7f31a402b860 comp rx=0 tx=0).stop 2024-02-21T23:12:22.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.976+0000 7f31b1e00640 1 -- 172.21.15.28:0/2120929148 shutdown_connections 2024-02-21T23:12:22.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.976+0000 7f31b1e00640 1 --2- 172.21.15.28:0/2120929148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31ac158cf0 0x7f31ac0c7120 unknown :-1 s=CLOSED pgs=2217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:22.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.976+0000 7f31b1e00640 1 --2- 172.21.15.28:0/2120929148 >> v2:172.21.15.81:3300/0 conn(0x7f31ac1525a0 0x7f31ac0c86a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:22.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.976+0000 7f31b1e00640 1 -- 172.21.15.28:0/2120929148 >> 172.21.15.28:0/2120929148 conn(0x7f31ac0ac4d0 msgr2=0x7f31ac0ac8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:22.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.976+0000 7f31b1e00640 1 -- 172.21.15.28:0/2120929148 shutdown_connections 2024-02-21T23:12:22.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.976+0000 7f31b1e00640 1 -- 172.21.15.28:0/2120929148 wait complete. 2024-02-21T23:12:22.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.977+0000 7f31b1e00640 1 Processor -- start 2024-02-21T23:12:22.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.978+0000 7f31b1e00640 1 -- start start 2024-02-21T23:12:22.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.978+0000 7f31b1e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f31ac06dd20 0x7f31ac1548b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:22.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.978+0000 7f31b1e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f31ac1525a0 0x7f31ac153660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:22.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.978+0000 7f31b1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31ac158cf0 0x7f31ac1ea690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:22.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.978+0000 7f31b1e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31ac0c5ae0 con 0x7f31ac158cf0 2024-02-21T23:12:22.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.979+0000 7f31b1e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31ac0c5960 con 0x7f31ac06dd20 2024-02-21T23:12:22.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.979+0000 7f31b1e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31ac159100 con 0x7f31ac1525a0 2024-02-21T23:12:22.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.979+0000 7f31b1400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31ac158cf0 0x7f31ac1ea690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:22.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.979+0000 7f31b1400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31ac158cf0 0x7f31ac1ea690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41718/0 (socket says 172.21.15.28:41718) 2024-02-21T23:12:22.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.979+0000 7f31b1400640 1 -- 172.21.15.28:0/916823933 learned_addr learned my addr 172.21.15.28:0/916823933 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:22.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.979+0000 7f31abe00640 1 --2- 172.21.15.28:0/916823933 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f31ac1525a0 0x7f31ac153660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:22.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.979+0000 7f31b0a00640 1 --2- 172.21.15.28:0/916823933 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f31ac06dd20 0x7f31ac1548b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:22.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.979+0000 7f31b1400640 1 -- 172.21.15.28:0/916823933 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f31ac1525a0 msgr2=0x7f31ac153660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:22.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.979+0000 7f31b1400640 1 --2- 172.21.15.28:0/916823933 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f31ac1525a0 0x7f31ac153660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:22.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.980+0000 7f31b1400640 1 -- 172.21.15.28:0/916823933 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f31ac06dd20 msgr2=0x7f31ac1548b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:22.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.980+0000 7f31b1400640 1 --2- 172.21.15.28:0/916823933 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f31ac06dd20 0x7f31ac1548b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:22.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.980+0000 7f31b1400640 1 -- 172.21.15.28:0/916823933 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31ac1eb360 con 0x7f31ac158cf0 2024-02-21T23:12:22.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.980+0000 7f31b1400640 1 --2- 172.21.15.28:0/916823933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31ac158cf0 0x7f31ac1ea690 secure :-1 s=READY pgs=2218 cs=0 l=1 rev1=1 crypto rx=0x7f31a40090b0 tx=0x7f31a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:22.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.980+0000 7f31a8c00640 1 -- 172.21.15.28:0/916823933 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a4032650 con 0x7f31ac158cf0 2024-02-21T23:12:22.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.980+0000 7f31a8c00640 1 -- 172.21.15.28:0/916823933 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31a40327f0 con 0x7f31ac158cf0 2024-02-21T23:12:22.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.980+0000 7f31a8c00640 1 -- 172.21.15.28:0/916823933 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a4032be0 con 0x7f31ac158cf0 2024-02-21T23:12:22.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.980+0000 7f31b1e00640 1 -- 172.21.15.28:0/916823933 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31ac1eb080 con 0x7f31ac158cf0 2024-02-21T23:12:22.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.980+0000 7f31b1e00640 1 -- 172.21.15.28:0/916823933 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31ac153ba0 con 0x7f31ac158cf0 2024-02-21T23:12:22.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.982+0000 7f31b1e00640 1 -- 172.21.15.28:0/916823933 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31ac04f290 con 0x7f31ac158cf0 2024-02-21T23:12:22.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.983+0000 7f31a8c00640 1 -- 172.21.15.28:0/916823933 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f31a40453f0 con 0x7f31ac158cf0 2024-02-21T23:12:22.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.983+0000 7f31a8c00640 1 --2- 172.21.15.28:0/916823933 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f318806b4d0 0x7f318806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:22.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.984+0000 7f31a8c00640 1 -- 172.21.15.28:0/916823933 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f31a40ac100 con 0x7f31ac158cf0 2024-02-21T23:12:22.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.984+0000 7f31b0a00640 1 --2- 172.21.15.28:0/916823933 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f318806b4d0 0x7f318806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:22.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.985+0000 7f31b0a00640 1 --2- 172.21.15.28:0/916823933 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f318806b4d0 0x7f318806d990 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f3194006fd0 tx=0x7f3194008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:22.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:22.988+0000 7f31a8c00640 1 -- 172.21.15.28:0/916823933 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f31a4040070 con 0x7f31ac158cf0 2024-02-21T23:12:23.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.185+0000 7f31b1e00640 1 -- 172.21.15.28:0/916823933 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f31ac06da70 con 0x7f31ac158cf0 2024-02-21T23:12:23.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.186+0000 7f31a8c00640 1 -- 172.21.15.28:0/916823933 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v193) v1 ==== 93+0+10 (secure 0 0 0) 0x7f31ac0c5120 con 0x7f31ac158cf0 2024-02-21T23:12:23.174 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-02-21T23:12:23.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.188+0000 7f31b1e00640 1 -- 172.21.15.28:0/916823933 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f318806b4d0 msgr2=0x7f318806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:23.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.188+0000 7f31b1e00640 1 --2- 172.21.15.28:0/916823933 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f318806b4d0 0x7f318806d990 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f3194006fd0 tx=0x7f3194008040 comp rx=0 tx=0).stop 2024-02-21T23:12:23.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.188+0000 7f31b1e00640 1 -- 172.21.15.28:0/916823933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31ac158cf0 msgr2=0x7f31ac1ea690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:23.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.188+0000 7f31b1e00640 1 --2- 172.21.15.28:0/916823933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31ac158cf0 0x7f31ac1ea690 secure :-1 s=READY pgs=2218 cs=0 l=1 rev1=1 crypto rx=0x7f31a40090b0 tx=0x7f31a40046c0 comp rx=0 tx=0).stop 2024-02-21T23:12:23.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.188+0000 7f31b1e00640 1 -- 172.21.15.28:0/916823933 shutdown_connections 2024-02-21T23:12:23.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.188+0000 7f31b1e00640 1 --2- 172.21.15.28:0/916823933 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f318806b4d0 0x7f318806d990 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:23.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.188+0000 7f31b1e00640 1 --2- 172.21.15.28:0/916823933 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f31ac1525a0 0x7f31ac153660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:23.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.188+0000 7f31b1e00640 1 --2- 172.21.15.28:0/916823933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31ac158cf0 0x7f31ac1ea690 unknown :-1 s=CLOSED pgs=2218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:23.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.188+0000 7f31b1e00640 1 --2- 172.21.15.28:0/916823933 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f31ac06dd20 0x7f31ac1548b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:23.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.188+0000 7f31b1e00640 1 -- 172.21.15.28:0/916823933 >> 172.21.15.28:0/916823933 conn(0x7f31ac0ac4d0 msgr2=0x7f31ac0c7000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:23.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.188+0000 7f31b1e00640 1 -- 172.21.15.28:0/916823933 shutdown_connections 2024-02-21T23:12:23.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.188+0000 7f31b1e00640 1 -- 172.21.15.28:0/916823933 wait complete. 2024-02-21T23:12:23.197 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-02-21T23:12:23.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.431+0000 7f39bda00640 1 Processor -- start 2024-02-21T23:12:23.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.431+0000 7f39bda00640 1 -- start start 2024-02-21T23:12:23.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.433+0000 7f39bda00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f39b80c29a0 0x7f39b815c860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:23.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.433+0000 7f39bda00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f39b8071730 0x7f39b8071b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:23.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.433+0000 7f39bda00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39b815cf60 con 0x7f39b8156a90 2024-02-21T23:12:23.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.433+0000 7f39bda00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39b80c7440 con 0x7f39b80c29a0 2024-02-21T23:12:23.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.433+0000 7f39bda00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39b815d240 con 0x7f39b8071730 2024-02-21T23:12:23.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.433+0000 7f39b7e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f39b8156a90 0x7f39b806e010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39710/0 (socket says 172.21.15.28:39710) 2024-02-21T23:12:23.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.433+0000 7f39b7e00640 1 -- 172.21.15.28:0/3916728393 learned_addr learned my addr 172.21.15.28:0/3916728393 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:12:23.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.433+0000 7f39b7400640 1 --2- 172.21.15.28:0/3916728393 >> v2:172.21.15.81:3300/0 conn(0x7f39b80c29a0 0x7f39b815c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:23.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.433+0000 7f39bd000640 1 --2- 172.21.15.28:0/3916728393 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f39b8071730 0x7f39b8071b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:23.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.434+0000 7f39b6a00640 1 -- 172.21.15.28:0/3916728393 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2382812318 0 0) 0x7f39b815cf60 con 0x7f39b8156a90 2024-02-21T23:12:23.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.434+0000 7f39b6a00640 1 -- 172.21.15.28:0/3916728393 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f399c003520 con 0x7f39b8156a90 2024-02-21T23:12:23.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.434+0000 7f39b6a00640 1 -- 172.21.15.28:0/3916728393 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2271615414 0 0) 0x7f399c003520 con 0x7f39b8156a90 2024-02-21T23:12:23.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.434+0000 7f39b6a00640 1 -- 172.21.15.28:0/3916728393 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f39b8071730 msgr2=0x7f39b8071b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:23.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.434+0000 7f39b6a00640 1 --2- 172.21.15.28:0/3916728393 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f39b8071730 0x7f39b8071b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:23.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.434+0000 7f39b6a00640 1 -- 172.21.15.28:0/3916728393 >> v2:172.21.15.81:3300/0 conn(0x7f39b80c29a0 msgr2=0x7f39b815c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:23.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.434+0000 7f39b6a00640 1 --2- 172.21.15.28:0/3916728393 >> v2:172.21.15.81:3300/0 conn(0x7f39b80c29a0 0x7f39b815c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:23.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.434+0000 7f39b6a00640 1 -- 172.21.15.28:0/3916728393 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39b815d520 con 0x7f39b8156a90 2024-02-21T23:12:23.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.434+0000 7f39b6a00640 1 -- 172.21.15.28:0/3916728393 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f39a8001670 con 0x7f39b8156a90 2024-02-21T23:12:23.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.434+0000 7f39b6a00640 1 -- 172.21.15.28:0/3916728393 >> v1:172.21.15.28:6789/0 conn(0x7f39b8156a90 legacy=0x7f39b806e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:23.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.434+0000 7f39b6a00640 1 --2- 172.21.15.28:0/3916728393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f399c0039b0 0x7f399c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:23.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.434+0000 7f39b6a00640 1 --2- 172.21.15.28:0/3916728393 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f39b8071730 0x7f399c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:23.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.434+0000 7f39b6a00640 1 --2- 172.21.15.28:0/3916728393 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f399c004d40 0x7f399c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:23.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.434+0000 7f39b6a00640 1 -- 172.21.15.28:0/3916728393 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b80c7440 con 0x7f399c0039b0 2024-02-21T23:12:23.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.434+0000 7f39b6a00640 1 -- 172.21.15.28:0/3916728393 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b815d240 con 0x7f399c004d40 2024-02-21T23:12:23.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.434+0000 7f39b6a00640 1 -- 172.21.15.28:0/3916728393 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f399c00b9c0 con 0x7f39b8071730 2024-02-21T23:12:23.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.434+0000 7f39b7e00640 1 --2- 172.21.15.28:0/3916728393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f399c0039b0 0x7f399c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:23.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.435+0000 7f39b7400640 1 --2- 172.21.15.28:0/3916728393 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f39b8071730 0x7f399c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:23.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.435+0000 7f39b7e00640 1 -- 172.21.15.28:0/3916728393 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f39b8071730 msgr2=0x7f399c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:23.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.435+0000 7f39bd000640 1 --2- 172.21.15.28:0/3916728393 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f399c004d40 0x7f399c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:23.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.435+0000 7f39b7e00640 1 --2- 172.21.15.28:0/3916728393 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f39b8071730 0x7f399c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:23.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.435+0000 7f39b7e00640 1 -- 172.21.15.28:0/3916728393 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f399c004d40 msgr2=0x7f399c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:23.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.435+0000 7f39b7e00640 1 --2- 172.21.15.28:0/3916728393 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f399c004d40 0x7f399c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:23.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.435+0000 7f39b7e00640 1 -- 172.21.15.28:0/3916728393 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f399c003520 con 0x7f399c0039b0 2024-02-21T23:12:23.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.435+0000 7f39b7400640 1 --2- 172.21.15.28:0/3916728393 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f39b8071730 0x7f399c004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:12:23.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.435+0000 7f39bd000640 1 --2- 172.21.15.28:0/3916728393 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f399c004d40 0x7f399c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:12:23.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.436+0000 7f39b7e00640 1 --2- 172.21.15.28:0/3916728393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f399c0039b0 0x7f399c003dd0 secure :-1 s=READY pgs=2220 cs=0 l=1 rev1=1 crypto rx=0x7f39a800c160 tx=0x7f399c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc3b5856075efd40 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:23.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.436+0000 7f39b6a00640 1 -- 172.21.15.28:0/3916728393 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39a8007c80 con 0x7f399c0039b0 2024-02-21T23:12:23.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.437+0000 7f39b6a00640 1 -- 172.21.15.28:0/3916728393 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39a8030070 con 0x7f399c0039b0 2024-02-21T23:12:23.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.437+0000 7f39b6a00640 1 -- 172.21.15.28:0/3916728393 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39a8005d30 con 0x7f399c0039b0 2024-02-21T23:12:23.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.437+0000 7f39bda00640 1 -- 172.21.15.28:0/3916728393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f399c0039b0 msgr2=0x7f399c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:23.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.437+0000 7f39bda00640 1 --2- 172.21.15.28:0/3916728393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f399c0039b0 0x7f399c003dd0 secure :-1 s=READY pgs=2220 cs=0 l=1 rev1=1 crypto rx=0x7f39a800c160 tx=0x7f399c005830 comp rx=0 tx=0).stop 2024-02-21T23:12:23.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.437+0000 7f39bd000640 1 -- 172.21.15.28:0/3916728393 reap_dead start 2024-02-21T23:12:23.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.437+0000 7f39bda00640 1 -- 172.21.15.28:0/3916728393 shutdown_connections 2024-02-21T23:12:23.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.437+0000 7f39bda00640 1 -- 172.21.15.28:0/3916728393 >> 172.21.15.28:0/3916728393 conn(0x7f39b806b550 msgr2=0x7f39b80c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:23.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.437+0000 7f39bda00640 1 -- 172.21.15.28:0/3916728393 shutdown_connections 2024-02-21T23:12:23.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.437+0000 7f39bda00640 1 -- 172.21.15.28:0/3916728393 wait complete. 2024-02-21T23:12:23.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.439+0000 7f39bda00640 1 Processor -- start 2024-02-21T23:12:23.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.439+0000 7f39bda00640 1 -- start start 2024-02-21T23:12:23.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.440+0000 7f39bda00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f399c0039b0 0x7f39b80a24d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:23.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.440+0000 7f39bda00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f39b80a5320 0x7f39b814d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:23.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.440+0000 7f39bda00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39b814da40 0x7f39b80a7590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:23.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.440+0000 7f39bda00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b806e720 con 0x7f39b814da40 2024-02-21T23:12:23.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.440+0000 7f39bda00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b8156e70 con 0x7f399c0039b0 2024-02-21T23:12:23.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.440+0000 7f39bda00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39b806e8a0 con 0x7f39b80a5320 2024-02-21T23:12:23.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.440+0000 7f39bd000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39b814da40 0x7f39b80a7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:23.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.440+0000 7f39bd000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39b814da40 0x7f39b80a7590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41744/0 (socket says 172.21.15.28:41744) 2024-02-21T23:12:23.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.440+0000 7f39bd000640 1 -- 172.21.15.28:0/2167068468 learned_addr learned my addr 172.21.15.28:0/2167068468 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:23.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.441+0000 7f39b7e00640 1 --2- 172.21.15.28:0/2167068468 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f399c0039b0 0x7f39b80a24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:23.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.441+0000 7f39b7400640 1 --2- 172.21.15.28:0/2167068468 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f39b80a5320 0x7f39b814d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:23.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.441+0000 7f39bd000640 1 -- 172.21.15.28:0/2167068468 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f39b80a5320 msgr2=0x7f39b814d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:23.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.441+0000 7f39bd000640 1 --2- 172.21.15.28:0/2167068468 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f39b80a5320 0x7f39b814d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:23.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.441+0000 7f39bd000640 1 -- 172.21.15.28:0/2167068468 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f399c0039b0 msgr2=0x7f39b80a24d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:23.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.441+0000 7f39bd000640 1 --2- 172.21.15.28:0/2167068468 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f399c0039b0 0x7f39b80a24d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:23.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.441+0000 7f39bd000640 1 -- 172.21.15.28:0/2167068468 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39b80a97c0 con 0x7f39b814da40 2024-02-21T23:12:23.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.441+0000 7f39bd000640 1 --2- 172.21.15.28:0/2167068468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39b814da40 0x7f39b80a7590 secure :-1 s=READY pgs=2221 cs=0 l=1 rev1=1 crypto rx=0x7f39b80c0b00 tx=0x7f39b000f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:23.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.441+0000 7f39afe00640 1 -- 172.21.15.28:0/2167068468 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b001c070 con 0x7f39b814da40 2024-02-21T23:12:23.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.442+0000 7f39afe00640 1 -- 172.21.15.28:0/2167068468 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39b0017400 con 0x7f39b814da40 2024-02-21T23:12:23.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.442+0000 7f39afe00640 1 -- 172.21.15.28:0/2167068468 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b00177d0 con 0x7f39b814da40 2024-02-21T23:12:23.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.442+0000 7f39bda00640 1 -- 172.21.15.28:0/2167068468 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39b80a44b0 con 0x7f39b814da40 2024-02-21T23:12:23.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.442+0000 7f39bda00640 1 -- 172.21.15.28:0/2167068468 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39b81ef110 con 0x7f39b814da40 2024-02-21T23:12:23.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.443+0000 7f39bda00640 1 -- 172.21.15.28:0/2167068468 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39b80a8f20 con 0x7f39b814da40 2024-02-21T23:12:23.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.444+0000 7f39afe00640 1 -- 172.21.15.28:0/2167068468 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f39b0005d10 con 0x7f39b814da40 2024-02-21T23:12:23.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.445+0000 7f39afe00640 1 --2- 172.21.15.28:0/2167068468 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f399806b4d0 0x7f399806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:23.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.445+0000 7f39afe00640 1 -- 172.21.15.28:0/2167068468 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f39b008a2d0 con 0x7f39b814da40 2024-02-21T23:12:23.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.445+0000 7f39b7e00640 1 --2- 172.21.15.28:0/2167068468 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f399806b4d0 0x7f399806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:23.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.446+0000 7f39b7e00640 1 --2- 172.21.15.28:0/2167068468 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f399806b4d0 0x7f399806d990 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f39a8034000 tx=0x7f39a8007000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:23.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.449+0000 7f39afe00640 1 -- 172.21.15.28:0/2167068468 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f39b80a8f20 con 0x7f39b814da40 2024-02-21T23:12:23.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.651+0000 7f39bda00640 1 -- 172.21.15.28:0/2167068468 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f39b804f290 con 0x7f39b814da40 2024-02-21T23:12:23.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.652+0000 7f39afe00640 1 -- 172.21.15.28:0/2167068468 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v193) v1 ==== 84+0+65 (secure 0 0 0) 0x7f39b0012070 con 0x7f39b814da40 2024-02-21T23:12:23.641 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:12:23.641 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-02-21T23:12:23.641 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:12:23.641 INFO:teuthology.orchestra.run.smithi028.stdout: "poolnum": 1, 2024-02-21T23:12:23.641 INFO:teuthology.orchestra.run.smithi028.stdout: "poolname": ".mgr" 2024-02-21T23:12:23.641 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:12:23.641 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-02-21T23:12:23.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.655+0000 7f39bda00640 1 -- 172.21.15.28:0/2167068468 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f399806b4d0 msgr2=0x7f399806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:23.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.655+0000 7f39bda00640 1 --2- 172.21.15.28:0/2167068468 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f399806b4d0 0x7f399806d990 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f39a8034000 tx=0x7f39a8007000 comp rx=0 tx=0).stop 2024-02-21T23:12:23.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.655+0000 7f39bda00640 1 -- 172.21.15.28:0/2167068468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39b814da40 msgr2=0x7f39b80a7590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:23.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.655+0000 7f39bda00640 1 --2- 172.21.15.28:0/2167068468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39b814da40 0x7f39b80a7590 secure :-1 s=READY pgs=2221 cs=0 l=1 rev1=1 crypto rx=0x7f39b80c0b00 tx=0x7f39b000f4a0 comp rx=0 tx=0).stop 2024-02-21T23:12:23.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.655+0000 7f39bda00640 1 -- 172.21.15.28:0/2167068468 shutdown_connections 2024-02-21T23:12:23.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.655+0000 7f39bda00640 1 --2- 172.21.15.28:0/2167068468 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f399806b4d0 0x7f399806d990 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:23.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.655+0000 7f39bda00640 1 --2- 172.21.15.28:0/2167068468 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f39b80a5320 0x7f39b814d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:23.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.655+0000 7f39bda00640 1 --2- 172.21.15.28:0/2167068468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39b814da40 0x7f39b80a7590 unknown :-1 s=CLOSED pgs=2221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:23.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.655+0000 7f39bda00640 1 --2- 172.21.15.28:0/2167068468 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f399c0039b0 0x7f39b80a24d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:23.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.655+0000 7f39bda00640 1 -- 172.21.15.28:0/2167068468 >> 172.21.15.28:0/2167068468 conn(0x7f39b806b550 msgr2=0x7f39b815b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:23.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.655+0000 7f39bda00640 1 -- 172.21.15.28:0/2167068468 shutdown_connections 2024-02-21T23:12:23.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.655+0000 7f39bda00640 1 -- 172.21.15.28:0/2167068468 wait complete. 2024-02-21T23:12:23.664 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-02-21T23:12:23.664 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-02-21T23:12:23.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.915+0000 7f4830a00640 1 Processor -- start 2024-02-21T23:12:23.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.915+0000 7f4830a00640 1 -- start start 2024-02-21T23:12:23.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.917+0000 7f4830a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f482c1501c0 0x7f482c0a96c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:23.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.917+0000 7f4830a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f482c0c1010 0x7f482c06ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:23.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.917+0000 7f4830a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f482c152fd0 con 0x7f482c0a9c00 2024-02-21T23:12:23.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.917+0000 7f4830a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f482c1532b0 con 0x7f482c0c1010 2024-02-21T23:12:23.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.917+0000 7f4830a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f482c153590 con 0x7f482c1501c0 2024-02-21T23:12:23.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.918+0000 7f482be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f482c0c1010 0x7f482c06ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:23.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.918+0000 7f482be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f482c0c1010 0x7f482c06ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41760/0 (socket says 172.21.15.28:41760) 2024-02-21T23:12:23.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.918+0000 7f482be00640 1 -- 172.21.15.28:0/3388467711 learned_addr learned my addr 172.21.15.28:0/3388467711 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:23.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.918+0000 7f482b400640 1 --2- 172.21.15.28:0/3388467711 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f482c1501c0 0x7f482c0a96c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:23.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.918+0000 7f482be00640 1 -- 172.21.15.28:0/3388467711 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f482c1501c0 msgr2=0x7f482c0a96c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:23.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.918+0000 7f482be00640 1 --2- 172.21.15.28:0/3388467711 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f482c1501c0 0x7f482c0a96c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:23.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.918+0000 7f482be00640 1 -- 172.21.15.28:0/3388467711 >> v1:172.21.15.81:6789/0 conn(0x7f482c0a9c00 legacy=0x7f482c06e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:23.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.918+0000 7f482be00640 1 -- 172.21.15.28:0/3388467711 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f482c153870 con 0x7f482c0c1010 2024-02-21T23:12:23.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.919+0000 7f482be00640 1 --2- 172.21.15.28:0/3388467711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f482c0c1010 0x7f482c06ea40 secure :-1 s=READY pgs=2222 cs=0 l=1 rev1=1 crypto rx=0x7f4824009e50 tx=0x7f482402bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f14f8cd5762ce17 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:23.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.920+0000 7f482a000640 1 -- 172.21.15.28:0/3388467711 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f482402c950 con 0x7f482c0c1010 2024-02-21T23:12:23.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.920+0000 7f482a000640 1 -- 172.21.15.28:0/3388467711 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f482402caf0 con 0x7f482c0c1010 2024-02-21T23:12:23.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.920+0000 7f482a000640 1 -- 172.21.15.28:0/3388467711 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48240343f0 con 0x7f482c0c1010 2024-02-21T23:12:23.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.920+0000 7f4830a00640 1 -- 172.21.15.28:0/3388467711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f482c0c1010 msgr2=0x7f482c06ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:23.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.920+0000 7f4830a00640 1 --2- 172.21.15.28:0/3388467711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f482c0c1010 0x7f482c06ea40 secure :-1 s=READY pgs=2222 cs=0 l=1 rev1=1 crypto rx=0x7f4824009e50 tx=0x7f482402bdb0 comp rx=0 tx=0).stop 2024-02-21T23:12:23.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.921+0000 7f4830a00640 1 -- 172.21.15.28:0/3388467711 shutdown_connections 2024-02-21T23:12:23.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.921+0000 7f4830a00640 1 --2- 172.21.15.28:0/3388467711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f482c0c1010 0x7f482c06ea40 unknown :-1 s=CLOSED pgs=2222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:23.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.921+0000 7f4830a00640 1 --2- 172.21.15.28:0/3388467711 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f482c1501c0 0x7f482c0a96c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:23.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.921+0000 7f4830a00640 1 -- 172.21.15.28:0/3388467711 >> 172.21.15.28:0/3388467711 conn(0x7f482c0c6610 msgr2=0x7f482c0c69f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:23.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.921+0000 7f4830a00640 1 -- 172.21.15.28:0/3388467711 shutdown_connections 2024-02-21T23:12:23.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.921+0000 7f4830a00640 1 -- 172.21.15.28:0/3388467711 wait complete. 2024-02-21T23:12:23.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.922+0000 7f4830a00640 1 Processor -- start 2024-02-21T23:12:23.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.923+0000 7f4830a00640 1 -- start start 2024-02-21T23:12:23.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.923+0000 7f4830a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f482c0a9c00 0x7f482c145e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:23.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.923+0000 7f4830a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f482c0c1010 0x7f482c071dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:23.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.924+0000 7f4830a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f482c1501c0 0x7f482c0c4290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:23.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.924+0000 7f4830a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f482c0c0c20 con 0x7f482c0a9c00 2024-02-21T23:12:23.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.924+0000 7f4830a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f482c06ef80 con 0x7f482c0c1010 2024-02-21T23:12:23.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.924+0000 7f4830a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f482c0c7340 con 0x7f482c1501c0 2024-02-21T23:12:23.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.924+0000 7f482b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f482c0a9c00 0x7f482c145e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:23.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.924+0000 7f482b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f482c0a9c00 0x7f482c145e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41776/0 (socket says 172.21.15.28:41776) 2024-02-21T23:12:23.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.924+0000 7f482b400640 1 -- 172.21.15.28:0/3210918518 learned_addr learned my addr 172.21.15.28:0/3210918518 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:23.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.924+0000 7f482b400640 1 -- 172.21.15.28:0/3210918518 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f482c1501c0 msgr2=0x7f482c0c4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:23.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.924+0000 7f482b400640 1 --2- 172.21.15.28:0/3210918518 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f482c1501c0 0x7f482c0c4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:23.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.924+0000 7f482b400640 1 -- 172.21.15.28:0/3210918518 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f482c0c1010 msgr2=0x7f482c071dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:23.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.924+0000 7f482aa00640 1 --2- 172.21.15.28:0/3210918518 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f482c0c1010 0x7f482c071dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:23.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.924+0000 7f482b400640 1 --2- 172.21.15.28:0/3210918518 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f482c0c1010 0x7f482c071dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:23.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.924+0000 7f482b400640 1 -- 172.21.15.28:0/3210918518 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f482c1ee7e0 con 0x7f482c0a9c00 2024-02-21T23:12:23.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.924+0000 7f482aa00640 1 --2- 172.21.15.28:0/3210918518 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f482c0c1010 0x7f482c071dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:12:23.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.925+0000 7f482b400640 1 --2- 172.21.15.28:0/3210918518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f482c0a9c00 0x7f482c145e90 secure :-1 s=READY pgs=2223 cs=0 l=1 rev1=1 crypto rx=0x7f481c00d5f0 tx=0x7f481c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:23.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.925+0000 7f4823400640 1 -- 172.21.15.28:0/3210918518 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f481c0125f0 con 0x7f482c0a9c00 2024-02-21T23:12:23.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.925+0000 7f4823400640 1 -- 172.21.15.28:0/3210918518 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f481c019070 con 0x7f482c0a9c00 2024-02-21T23:12:23.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.925+0000 7f4823400640 1 -- 172.21.15.28:0/3210918518 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f481c0129d0 con 0x7f482c0a9c00 2024-02-21T23:12:23.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.925+0000 7f4830a00640 1 -- 172.21.15.28:0/3210918518 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f482c150cb0 con 0x7f482c0a9c00 2024-02-21T23:12:23.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.925+0000 7f4830a00640 1 -- 172.21.15.28:0/3210918518 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f482c14e350 con 0x7f482c0a9c00 2024-02-21T23:12:23.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.926+0000 7f4830a00640 1 -- 172.21.15.28:0/3210918518 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f482c150aa0 con 0x7f482c0a9c00 2024-02-21T23:12:23.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.927+0000 7f4823400640 1 -- 172.21.15.28:0/3210918518 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f481c012b70 con 0x7f482c0a9c00 2024-02-21T23:12:23.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.928+0000 7f4823400640 1 --2- 172.21.15.28:0/3210918518 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f480c06b520 0x7f480c06d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:23.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.929+0000 7f4823400640 1 -- 172.21.15.28:0/3210918518 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f481c08a3e0 con 0x7f482c0a9c00 2024-02-21T23:12:23.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.929+0000 7f482aa00640 1 --2- 172.21.15.28:0/3210918518 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f480c06b520 0x7f480c06d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:23.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.930+0000 7f482aa00640 1 --2- 172.21.15.28:0/3210918518 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f480c06b520 0x7f480c06d9e0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f482c1463d0 tx=0x7f4818009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:23.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:23.933+0000 7f4823400640 1 -- 172.21.15.28:0/3210918518 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f481c059490 con 0x7f482c0a9c00 2024-02-21T23:12:24.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:24.143+0000 7f4830a00640 1 -- 172.21.15.28:0/3210918518 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f482c04f290 con 0x7f482c0a9c00 2024-02-21T23:12:24.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:24.769+0000 7f4823400640 1 -- 172.21.15.28:0/3210918518 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v194) v1 ==== 138+0+0 (secure 0 0 0) 0x7f481c058d90 con 0x7f482c0a9c00 2024-02-21T23:12:24.758 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' created 2024-02-21T23:12:24.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:24.773+0000 7f4830a00640 1 -- 172.21.15.28:0/3210918518 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f480c06b520 msgr2=0x7f480c06d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:24.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:24.773+0000 7f4830a00640 1 --2- 172.21.15.28:0/3210918518 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f480c06b520 0x7f480c06d9e0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f482c1463d0 tx=0x7f4818009040 comp rx=0 tx=0).stop 2024-02-21T23:12:24.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:24.773+0000 7f4830a00640 1 -- 172.21.15.28:0/3210918518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f482c0a9c00 msgr2=0x7f482c145e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:24.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:24.773+0000 7f4830a00640 1 --2- 172.21.15.28:0/3210918518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f482c0a9c00 0x7f482c145e90 secure :-1 s=READY pgs=2223 cs=0 l=1 rev1=1 crypto rx=0x7f481c00d5f0 tx=0x7f481c00dab0 comp rx=0 tx=0).stop 2024-02-21T23:12:24.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:24.773+0000 7f4830a00640 1 -- 172.21.15.28:0/3210918518 shutdown_connections 2024-02-21T23:12:24.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:24.773+0000 7f4830a00640 1 --2- 172.21.15.28:0/3210918518 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f480c06b520 0x7f480c06d9e0 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:24.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:24.773+0000 7f4830a00640 1 --2- 172.21.15.28:0/3210918518 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f482c1501c0 0x7f482c0c4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:24.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:24.773+0000 7f4830a00640 1 --2- 172.21.15.28:0/3210918518 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f482c0c1010 0x7f482c071dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:24.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:24.773+0000 7f4830a00640 1 --2- 172.21.15.28:0/3210918518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f482c0a9c00 0x7f482c145e90 unknown :-1 s=CLOSED pgs=2223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:24.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:24.773+0000 7f4830a00640 1 -- 172.21.15.28:0/3210918518 >> 172.21.15.28:0/3210918518 conn(0x7f482c0c6610 msgr2=0x7f482c0743f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:24.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:24.773+0000 7f4830a00640 1 -- 172.21.15.28:0/3210918518 shutdown_connections 2024-02-21T23:12:24.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:24.774+0000 7f4830a00640 1 -- 172.21.15.28:0/3210918518 wait complete. 2024-02-21T23:12:24.789 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-02-21T23:12:25.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.016+0000 7ffacd000640 1 Processor -- start 2024-02-21T23:12:25.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.016+0000 7ffacd000640 1 -- start start 2024-02-21T23:12:25.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.018+0000 7ffacd000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ffac81505c0 0x7ffac8153800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:25.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.018+0000 7ffacd000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffac8072520 0x7ffac806be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:25.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.018+0000 7ffacd000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffac8072ac0 con 0x7ffac806c3c0 2024-02-21T23:12:25.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.018+0000 7ffacd000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffac80aaa00 con 0x7ffac81505c0 2024-02-21T23:12:25.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.018+0000 7ffacd000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffac8153d40 con 0x7ffac8072520 2024-02-21T23:12:25.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.018+0000 7ffac7400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ffac81505c0 0x7ffac8153800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:25.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.019+0000 7ffac7400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ffac81505c0 0x7ffac8153800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44198/0 (socket says 172.21.15.28:44198) 2024-02-21T23:12:25.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.019+0000 7ffac7400640 1 -- 172.21.15.28:0/3114612520 learned_addr learned my addr 172.21.15.28:0/3114612520 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:25.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.019+0000 7ffac6a00640 1 --2- 172.21.15.28:0/3114612520 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffac8072520 0x7ffac806be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:25.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.019+0000 7ffac7400640 1 -- 172.21.15.28:0/3114612520 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffac8072520 msgr2=0x7ffac806be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:25.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.019+0000 7ffac7400640 1 --2- 172.21.15.28:0/3114612520 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffac8072520 0x7ffac806be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:25.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.019+0000 7ffac7400640 1 -- 172.21.15.28:0/3114612520 >> v1:172.21.15.81:6789/0 conn(0x7ffac806c3c0 legacy=0x7ffac80aae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:25.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.019+0000 7ffac7400640 1 -- 172.21.15.28:0/3114612520 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffac80a8ec0 con 0x7ffac81505c0 2024-02-21T23:12:25.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.020+0000 7ffac7400640 1 --2- 172.21.15.28:0/3114612520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffac81505c0 0x7ffac8153800 secure :-1 s=READY pgs=2224 cs=0 l=1 rev1=1 crypto rx=0x7ffab8009870 tx=0x7ffab802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96d267e643ff6261 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:25.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.021+0000 7ffac6000640 1 -- 172.21.15.28:0/3114612520 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab8035020 con 0x7ffac81505c0 2024-02-21T23:12:25.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.021+0000 7ffac6000640 1 -- 172.21.15.28:0/3114612520 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffab8031640 con 0x7ffac81505c0 2024-02-21T23:12:25.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.021+0000 7ffac6000640 1 -- 172.21.15.28:0/3114612520 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab802d040 con 0x7ffac81505c0 2024-02-21T23:12:25.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.021+0000 7ffacd000640 1 -- 172.21.15.28:0/3114612520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffac81505c0 msgr2=0x7ffac8153800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:25.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.021+0000 7ffacd000640 1 --2- 172.21.15.28:0/3114612520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffac81505c0 0x7ffac8153800 secure :-1 s=READY pgs=2224 cs=0 l=1 rev1=1 crypto rx=0x7ffab8009870 tx=0x7ffab802b8f0 comp rx=0 tx=0).stop 2024-02-21T23:12:25.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.022+0000 7ffacd000640 1 -- 172.21.15.28:0/3114612520 shutdown_connections 2024-02-21T23:12:25.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.022+0000 7ffacd000640 1 --2- 172.21.15.28:0/3114612520 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffac8072520 0x7ffac806be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:25.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.022+0000 7ffacd000640 1 --2- 172.21.15.28:0/3114612520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffac81505c0 0x7ffac8153800 unknown :-1 s=CLOSED pgs=2224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:25.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.022+0000 7ffacd000640 1 -- 172.21.15.28:0/3114612520 >> 172.21.15.28:0/3114612520 conn(0x7ffac80c4000 msgr2=0x7ffac80719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:25.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.022+0000 7ffacd000640 1 -- 172.21.15.28:0/3114612520 shutdown_connections 2024-02-21T23:12:25.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.022+0000 7ffacd000640 1 -- 172.21.15.28:0/3114612520 wait complete. 2024-02-21T23:12:25.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.023+0000 7ffacd000640 1 Processor -- start 2024-02-21T23:12:25.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.024+0000 7ffacd000640 1 -- start start 2024-02-21T23:12:25.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.024+0000 7ffacd000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffac806c3c0 0x7ffac81ebf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:25.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.024+0000 7ffacd000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffac81505c0 0x7ffac81ea440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:25.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.024+0000 7ffacd000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffac81dc300 0x7ffac81ea980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:25.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.024+0000 7ffacd000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac80a9480 con 0x7ffac81505c0 2024-02-21T23:12:25.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.024+0000 7ffacd000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac8072d60 con 0x7ffac81dc300 2024-02-21T23:12:25.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.024+0000 7ffacd000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffac80ab580 con 0x7ffac806c3c0 2024-02-21T23:12:25.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.025+0000 7ffac6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffac81505c0 0x7ffac81ea440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:25.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.025+0000 7ffac6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffac81505c0 0x7ffac81ea440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44202/0 (socket says 172.21.15.28:44202) 2024-02-21T23:12:25.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.025+0000 7ffac6a00640 1 -- 172.21.15.28:0/1948883579 learned_addr learned my addr 172.21.15.28:0/1948883579 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:25.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.025+0000 7ffac7e00640 1 --2- 172.21.15.28:0/1948883579 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffac81dc300 0x7ffac81ea980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:25.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.025+0000 7ffac7400640 1 --2- 172.21.15.28:0/1948883579 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffac806c3c0 0x7ffac81ebf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:25.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.025+0000 7ffac6a00640 1 -- 172.21.15.28:0/1948883579 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffac806c3c0 msgr2=0x7ffac81ebf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:25.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.025+0000 7ffac6a00640 1 --2- 172.21.15.28:0/1948883579 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffac806c3c0 0x7ffac81ebf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:25.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.025+0000 7ffac6a00640 1 -- 172.21.15.28:0/1948883579 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffac81dc300 msgr2=0x7ffac81ea980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:25.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.025+0000 7ffac6a00640 1 --2- 172.21.15.28:0/1948883579 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffac81dc300 0x7ffac81ea980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:25.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.025+0000 7ffac6a00640 1 -- 172.21.15.28:0/1948883579 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffac81e9c80 con 0x7ffac81505c0 2024-02-21T23:12:25.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.025+0000 7ffac7400640 1 --2- 172.21.15.28:0/1948883579 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffac806c3c0 0x7ffac81ebf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:12:25.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.025+0000 7ffac7e00640 1 --2- 172.21.15.28:0/1948883579 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffac81dc300 0x7ffac81ea980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:12:25.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.026+0000 7ffac6a00640 1 --2- 172.21.15.28:0/1948883579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffac81505c0 0x7ffac81ea440 secure :-1 s=READY pgs=2225 cs=0 l=1 rev1=1 crypto rx=0x7ffab400d800 tx=0x7ffab400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:25.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.026+0000 7ffabf400640 1 -- 172.21.15.28:0/1948883579 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab400b6c0 con 0x7ffac81505c0 2024-02-21T23:12:25.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.026+0000 7ffabf400640 1 -- 172.21.15.28:0/1948883579 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffab4019070 con 0x7ffac81505c0 2024-02-21T23:12:25.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.026+0000 7ffabf400640 1 -- 172.21.15.28:0/1948883579 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab400ba80 con 0x7ffac81505c0 2024-02-21T23:12:25.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.026+0000 7ffacd000640 1 -- 172.21.15.28:0/1948883579 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffac81ec890 con 0x7ffac81505c0 2024-02-21T23:12:25.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.026+0000 7ffacd000640 1 -- 172.21.15.28:0/1948883579 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffac81ecb70 con 0x7ffac81505c0 2024-02-21T23:12:25.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.027+0000 7ffacd000640 1 -- 172.21.15.28:0/1948883579 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffac804f290 con 0x7ffac81505c0 2024-02-21T23:12:25.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.028+0000 7ffabf400640 1 -- 172.21.15.28:0/1948883579 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffab400bc20 con 0x7ffac81505c0 2024-02-21T23:12:25.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.029+0000 7ffabf400640 1 --2- 172.21.15.28:0/1948883579 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ffaa806b4d0 0x7ffaa806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:25.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.029+0000 7ffabf400640 1 -- 172.21.15.28:0/1948883579 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 5619+0+0 (secure 0 0 0) 0x7ffab408a6a0 con 0x7ffac81505c0 2024-02-21T23:12:25.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.030+0000 7ffac7400640 1 --2- 172.21.15.28:0/1948883579 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ffaa806b4d0 0x7ffaa806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:25.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.031+0000 7ffac7400640 1 --2- 172.21.15.28:0/1948883579 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ffaa806b4d0 0x7ffaa806d990 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7ffab8036000 tx=0x7ffab8008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:25.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.035+0000 7ffabf400640 1 -- 172.21.15.28:0/1948883579 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ffab4059560 con 0x7ffac81505c0 2024-02-21T23:12:25.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.236+0000 7ffacd000640 1 -- 172.21.15.28:0/1948883579 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7ffac8072220 con 0x7ffac81505c0 2024-02-21T23:12:25.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.776+0000 7ffabf400640 1 -- 172.21.15.28:0/1948883579 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v195) v1 ==== 170+0+0 (secure 0 0 0) 0x7ffab4058e60 con 0x7ffac81505c0 2024-02-21T23:12:25.765 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' created 2024-02-21T23:12:25.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.779+0000 7ffacd000640 1 -- 172.21.15.28:0/1948883579 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ffaa806b4d0 msgr2=0x7ffaa806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:25.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.779+0000 7ffacd000640 1 --2- 172.21.15.28:0/1948883579 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ffaa806b4d0 0x7ffaa806d990 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7ffab8036000 tx=0x7ffab8008030 comp rx=0 tx=0).stop 2024-02-21T23:12:25.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.779+0000 7ffacd000640 1 -- 172.21.15.28:0/1948883579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffac81505c0 msgr2=0x7ffac81ea440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:25.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.779+0000 7ffacd000640 1 --2- 172.21.15.28:0/1948883579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffac81505c0 0x7ffac81ea440 secure :-1 s=READY pgs=2225 cs=0 l=1 rev1=1 crypto rx=0x7ffab400d800 tx=0x7ffab400dcc0 comp rx=0 tx=0).stop 2024-02-21T23:12:25.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.780+0000 7ffacd000640 1 -- 172.21.15.28:0/1948883579 shutdown_connections 2024-02-21T23:12:25.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.780+0000 7ffacd000640 1 --2- 172.21.15.28:0/1948883579 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ffaa806b4d0 0x7ffaa806d990 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:25.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.780+0000 7ffacd000640 1 --2- 172.21.15.28:0/1948883579 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffac81dc300 0x7ffac81ea980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:25.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.780+0000 7ffacd000640 1 --2- 172.21.15.28:0/1948883579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffac81505c0 0x7ffac81ea440 unknown :-1 s=CLOSED pgs=2225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:25.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.780+0000 7ffacd000640 1 --2- 172.21.15.28:0/1948883579 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffac806c3c0 0x7ffac81ebf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:25.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.780+0000 7ffacd000640 1 -- 172.21.15.28:0/1948883579 >> 172.21.15.28:0/1948883579 conn(0x7ffac80c4000 msgr2=0x7ffac80a84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:25.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.780+0000 7ffacd000640 1 -- 172.21.15.28:0/1948883579 shutdown_connections 2024-02-21T23:12:25.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:25.781+0000 7ffacd000640 1 -- 172.21.15.28:0/1948883579 wait complete. 2024-02-21T23:12:25.801 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-02-21T23:12:25.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:26.000+0000 7fa0c5e00640 1 Processor -- start 2024-02-21T23:12:25.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:26.000+0000 7fa0c5e00640 1 -- start start 2024-02-21T23:12:25.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:26.002+0000 7fa0c5e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa0c0150700 0x7fa0c0156500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:25.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:26.002+0000 7fa0c5e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0c0152fb0 con 0x7fa0c00c5f50 2024-02-21T23:12:25.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:26.002+0000 7fa0c5e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0c0153290 con 0x7fa0c006b0e0 2024-02-21T23:12:25.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:26.002+0000 7fa0c5e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0c0153bc0 con 0x7fa0c0150700 2024-02-21T23:12:25.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:26.002+0000 7fa0be000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa0c00c5f50 0x7fa0c014fff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:32866/0 (socket says 172.21.15.28:32866) 2024-02-21T23:12:25.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:26.002+0000 7fa0be000640 1 -- 172.21.15.28:0/1369253948 learned_addr learned my addr 172.21.15.28:0/1369253948 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:12:25.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:26.002+0000 7fa0bf400640 1 --2- 172.21.15.28:0/1369253948 >> v2:172.21.15.28:3300/0 conn(0x7fa0c0150700 0x7fa0c0156500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:25.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:26.002+0000 7fa0bd600640 1 -- 172.21.15.28:0/1369253948 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2274371986 0 0) 0x7fa0c0152fb0 con 0x7fa0c00c5f50 2024-02-21T23:12:25.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:26.003+0000 7fa0bd600640 1 -- 172.21.15.28:0/1369253948 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0a4003520 con 0x7fa0c00c5f50 2024-02-21T23:12:25.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:26.003+0000 7fa0bf400640 1 -- 172.21.15.28:0/1369253948 >> v1:172.21.15.81:6789/0 conn(0x7fa0c006b0e0 legacy=0x7fa0c00c5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:25.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:26.003+0000 7fa0bf400640 1 -- 172.21.15.28:0/1369253948 >> v1:172.21.15.28:6789/0 conn(0x7fa0c00c5f50 legacy=0x7fa0c014fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:25.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:26.003+0000 7fa0bf400640 1 -- 172.21.15.28:0/1369253948 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0c0153570 con 0x7fa0c0150700 2024-02-21T23:12:25.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:26.004+0000 7fa0bf400640 1 --2- 172.21.15.28:0/1369253948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0c0150700 0x7fa0c0156500 secure :-1 s=READY pgs=2227 cs=0 l=1 rev1=1 crypto rx=0x7fa0b80097a0 tx=0x7fa0b802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ba6b167aaf99935 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:25.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:26.004+0000 7fa0bd600640 1 -- 172.21.15.28:0/1369253948 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b8035020 con 0x7fa0c0150700 2024-02-21T23:12:25.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:26.004+0000 7fa0bd600640 1 -- 172.21.15.28:0/1369253948 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0b8031830 con 0x7fa0c0150700 2024-02-21T23:12:25.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:26.004+0000 7fa0bd600640 1 -- 172.21.15.28:0/1369253948 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b8031c00 con 0x7fa0c0150700 2024-02-21T23:12:25.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:26.005+0000 7fa0c5e00640 1 -- 172.21.15.28:0/1369253948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0c0150700 msgr2=0x7fa0c0156500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:25.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:26.005+0000 7fa0c5e00640 1 --2- 172.21.15.28:0/1369253948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0c0150700 0x7fa0c0156500 secure :-1 s=READY pgs=2227 cs=0 l=1 rev1=1 crypto rx=0x7fa0b80097a0 tx=0x7fa0b802b8d0 comp rx=0 tx=0).stop 2024-02-21T23:12:25.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:26.005+0000 7fa0c5e00640 1 -- 172.21.15.28:0/1369253948 shutdown_connections 2024-02-21T23:12:25.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:26.005+0000 7fa0c5e00640 1 --2- 172.21.15.28:0/1369253948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0c0150700 0x7fa0c0156500 unknown :-1 s=CLOSED pgs=2227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:25.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:26.005+0000 7fa0c5e00640 1 -- 172.21.15.28:0/1369253948 >> 172.21.15.28:0/1369253948 conn(0x7fa0c00a85a0 msgr2=0x7fa0c00a8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:25.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:26.005+0000 7fa0c5e00640 1 -- 172.21.15.28:0/1369253948 shutdown_connections 2024-02-21T23:12:25.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:26.005+0000 7fa0c5e00640 1 -- 172.21.15.28:0/1369253948 wait complete. 2024-02-21T23:12:25.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:26.006+0000 7fa0c5e00640 1 Processor -- start 2024-02-21T23:12:25.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:26.007+0000 7fa0c5e00640 1 -- start start 2024-02-21T23:12:25.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:26.007+0000 7fa0c5e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa0c006b0e0 0x7fa0c00ab7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:25.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:26.007+0000 7fa0c5e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0c00c5f50 0x7fa0c0146ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:25.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:26.008+0000 7fa0c5e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa0c0150700 0x7fa0c01ec0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:25.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:26.008+0000 7fa0c5e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c006b640 con 0x7fa0c00c5f50 2024-02-21T23:12:25.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:26.008+0000 7fa0c5e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c006b4c0 con 0x7fa0c0150700 2024-02-21T23:12:25.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:26.008+0000 7fa0c5e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c0156a40 con 0x7fa0c006b0e0 2024-02-21T23:12:25.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:26.008+0000 7fa0be000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0c00c5f50 0x7fa0c0146ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:25.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:26.008+0000 7fa0be000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0c00c5f50 0x7fa0c0146ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44222/0 (socket says 172.21.15.28:44222) 2024-02-21T23:12:25.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:26.008+0000 7fa0be000640 1 -- 172.21.15.28:0/4223857164 learned_addr learned my addr 172.21.15.28:0/4223857164 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:26.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:26.008+0000 7fa0bf400640 1 --2- 172.21.15.28:0/4223857164 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa0c0150700 0x7fa0c01ec0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:26.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:26.008+0000 7fa0bea00640 1 --2- 172.21.15.28:0/4223857164 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa0c006b0e0 0x7fa0c00ab7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:26.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:26.008+0000 7fa0be000640 1 -- 172.21.15.28:0/4223857164 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa0c006b0e0 msgr2=0x7fa0c00ab7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:26.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:26.008+0000 7fa0be000640 1 --2- 172.21.15.28:0/4223857164 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa0c006b0e0 0x7fa0c00ab7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:26.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:26.008+0000 7fa0be000640 1 -- 172.21.15.28:0/4223857164 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa0c0150700 msgr2=0x7fa0c01ec0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:26.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:26.008+0000 7fa0be000640 1 --2- 172.21.15.28:0/4223857164 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa0c0150700 0x7fa0c01ec0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:26.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:26.008+0000 7fa0be000640 1 -- 172.21.15.28:0/4223857164 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0c0147530 con 0x7fa0c00c5f50 2024-02-21T23:12:26.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:26.009+0000 7fa0be000640 1 --2- 172.21.15.28:0/4223857164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0c00c5f50 0x7fa0c0146ff0 secure :-1 s=READY pgs=2228 cs=0 l=1 rev1=1 crypto rx=0x7fa0ac00c530 tx=0x7fa0ac00c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:26.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:26.009+0000 7fa0b7400640 1 -- 172.21.15.28:0/4223857164 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0ac017070 con 0x7fa0c00c5f50 2024-02-21T23:12:26.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:26.009+0000 7fa0b7400640 1 -- 172.21.15.28:0/4223857164 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0ac005ce0 con 0x7fa0c00c5f50 2024-02-21T23:12:26.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:26.009+0000 7fa0c5e00640 1 -- 172.21.15.28:0/4223857164 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0c01ec610 con 0x7fa0c00c5f50 2024-02-21T23:12:26.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:26.009+0000 7fa0b7400640 1 -- 172.21.15.28:0/4223857164 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0ac0072b0 con 0x7fa0c00c5f50 2024-02-21T23:12:26.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:26.009+0000 7fa0c5e00640 1 -- 172.21.15.28:0/4223857164 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0c01524f0 con 0x7fa0c00c5f50 2024-02-21T23:12:26.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:26.010+0000 7fa0c5e00640 1 -- 172.21.15.28:0/4223857164 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0c0067f30 con 0x7fa0c00c5f50 2024-02-21T23:12:26.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:26.012+0000 7fa0b7400640 1 -- 172.21.15.28:0/4223857164 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa0ac01a070 con 0x7fa0c00c5f50 2024-02-21T23:12:26.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:26.012+0000 7fa0b7400640 1 --2- 172.21.15.28:0/4223857164 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa0a006b4d0 0x7fa0a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:26.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:26.013+0000 7fa0b7400640 1 -- 172.21.15.28:0/4223857164 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 6001+0+0 (secure 0 0 0) 0x7fa0ac0897d0 con 0x7fa0c00c5f50 2024-02-21T23:12:26.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:26.013+0000 7fa0bea00640 1 --2- 172.21.15.28:0/4223857164 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa0a006b4d0 0x7fa0a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:26.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:26.014+0000 7fa0bea00640 1 --2- 172.21.15.28:0/4223857164 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa0a006b4d0 0x7fa0a006d990 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fa0b0001d40 tx=0x7fa0b0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:26.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:26.018+0000 7fa0b7400640 1 -- 172.21.15.28:0/4223857164 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fa0c0147530 con 0x7fa0c00c5f50 2024-02-21T23:12:26.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:26.256+0000 7fa0c5e00640 1 -- 172.21.15.28:0/4223857164 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fa0c00c4160 con 0x7fa0c00c5f50 2024-02-21T23:12:26.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:26.806+0000 7fa0b7400640 1 -- 172.21.15.28:0/4223857164 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 31 and data pool 32 v214) v1 ==== 174+0+0 (secure 0 0 0) 0x7fa0ac058540 con 0x7fa0c00c5f50 2024-02-21T23:12:26.795 INFO:teuthology.orchestra.run.smithi028.stderr:new fs with metadata pool 31 and data pool 32 2024-02-21T23:12:26.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:26.812+0000 7fa0c5e00640 1 -- 172.21.15.28:0/4223857164 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa0a006b4d0 msgr2=0x7fa0a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:26.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:26.812+0000 7fa0c5e00640 1 --2- 172.21.15.28:0/4223857164 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa0a006b4d0 0x7fa0a006d990 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fa0b0001d40 tx=0x7fa0b0009040 comp rx=0 tx=0).stop 2024-02-21T23:12:26.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:26.812+0000 7fa0c5e00640 1 -- 172.21.15.28:0/4223857164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0c00c5f50 msgr2=0x7fa0c0146ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:26.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:26.812+0000 7fa0c5e00640 1 --2- 172.21.15.28:0/4223857164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0c00c5f50 0x7fa0c0146ff0 secure :-1 s=READY pgs=2228 cs=0 l=1 rev1=1 crypto rx=0x7fa0ac00c530 tx=0x7fa0ac00c9f0 comp rx=0 tx=0).stop 2024-02-21T23:12:26.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:26.813+0000 7fa0c5e00640 1 -- 172.21.15.28:0/4223857164 shutdown_connections 2024-02-21T23:12:26.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:26.813+0000 7fa0c5e00640 1 --2- 172.21.15.28:0/4223857164 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa0a006b4d0 0x7fa0a006d990 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:26.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:26.813+0000 7fa0c5e00640 1 --2- 172.21.15.28:0/4223857164 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa0c0150700 0x7fa0c01ec0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:26.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:26.813+0000 7fa0c5e00640 1 --2- 172.21.15.28:0/4223857164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0c00c5f50 0x7fa0c0146ff0 unknown :-1 s=CLOSED pgs=2228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:26.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:26.813+0000 7fa0c5e00640 1 --2- 172.21.15.28:0/4223857164 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa0c006b0e0 0x7fa0c00ab7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:26.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:26.813+0000 7fa0c5e00640 1 -- 172.21.15.28:0/4223857164 >> 172.21.15.28:0/4223857164 conn(0x7fa0c00a85a0 msgr2=0x7fa0c0074360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:26.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:26.813+0000 7fa0c5e00640 1 -- 172.21.15.28:0/4223857164 shutdown_connections 2024-02-21T23:12:26.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:26.814+0000 7fa0c5e00640 1 -- 172.21.15.28:0/4223857164 wait complete. 2024-02-21T23:12:26.843 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:12:27.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.037+0000 7f547b200640 1 Processor -- start 2024-02-21T23:12:27.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.037+0000 7f547b200640 1 -- start start 2024-02-21T23:12:27.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.038+0000 7f547b200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f547c06dbf0 0x7f547c158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:27.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.038+0000 7f547b200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f547c155960 0x7f547c159000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:27.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.038+0000 7f547b200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f547c0c74c0 con 0x7f547c0aff90 2024-02-21T23:12:27.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.038+0000 7f547b200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f547c0c7270 con 0x7f547c155960 2024-02-21T23:12:27.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.038+0000 7f547b200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f547c158630 con 0x7f547c06dbf0 2024-02-21T23:12:27.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.039+0000 7f5479e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f547c0aff90 0x7f547c0c51e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:32880/0 (socket says 172.21.15.28:32880) 2024-02-21T23:12:27.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.039+0000 7f5479e00640 1 -- 172.21.15.28:0/3739790603 learned_addr learned my addr 172.21.15.28:0/3739790603 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:12:27.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.039+0000 7f547a800640 1 --2- 172.21.15.28:0/3739790603 >> v2:172.21.15.28:3300/0 conn(0x7f547c155960 0x7f547c159000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:27.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.039+0000 7f5478a00640 1 -- 172.21.15.28:0/3739790603 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 823437799 0 0) 0x7f547c0c74c0 con 0x7f547c0aff90 2024-02-21T23:12:27.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.039+0000 7f5479400640 1 --2- 172.21.15.28:0/3739790603 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f547c06dbf0 0x7f547c158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:27.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.039+0000 7f5478a00640 1 -- 172.21.15.28:0/3739790603 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5460003520 con 0x7f547c0aff90 2024-02-21T23:12:27.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.039+0000 7f547a800640 1 -- 172.21.15.28:0/3739790603 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f547c06dbf0 msgr2=0x7f547c158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:27.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.039+0000 7f547a800640 1 --2- 172.21.15.28:0/3739790603 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f547c06dbf0 0x7f547c158ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:27.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.039+0000 7f547a800640 1 -- 172.21.15.28:0/3739790603 >> v1:172.21.15.28:6789/0 conn(0x7f547c0aff90 legacy=0x7f547c0c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:27.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.039+0000 7f547a800640 1 -- 172.21.15.28:0/3739790603 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f547c0c77a0 con 0x7f547c155960 2024-02-21T23:12:27.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.039+0000 7f5479400640 1 --2- 172.21.15.28:0/3739790603 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f547c06dbf0 0x7f547c158ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:12:27.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.041+0000 7f547a800640 1 --2- 172.21.15.28:0/3739790603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f547c155960 0x7f547c159000 secure :-1 s=READY pgs=2230 cs=0 l=1 rev1=1 crypto rx=0x7f5474009d90 tx=0x7f547402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48c71df9ede6e984 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:27.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.041+0000 7f5478a00640 1 -- 172.21.15.28:0/3739790603 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5474039070 con 0x7f547c155960 2024-02-21T23:12:27.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.041+0000 7f5478a00640 1 -- 172.21.15.28:0/3739790603 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f547402cc50 con 0x7f547c155960 2024-02-21T23:12:27.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.041+0000 7f5478a00640 1 -- 172.21.15.28:0/3739790603 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5474034440 con 0x7f547c155960 2024-02-21T23:12:27.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.041+0000 7f547b200640 1 -- 172.21.15.28:0/3739790603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f547c155960 msgr2=0x7f547c159000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:27.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.041+0000 7f547b200640 1 --2- 172.21.15.28:0/3739790603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f547c155960 0x7f547c159000 secure :-1 s=READY pgs=2230 cs=0 l=1 rev1=1 crypto rx=0x7f5474009d90 tx=0x7f547402bea0 comp rx=0 tx=0).stop 2024-02-21T23:12:27.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.041+0000 7f547b200640 1 -- 172.21.15.28:0/3739790603 shutdown_connections 2024-02-21T23:12:27.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.041+0000 7f547b200640 1 --2- 172.21.15.28:0/3739790603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f547c155960 0x7f547c159000 unknown :-1 s=CLOSED pgs=2230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:27.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.041+0000 7f547b200640 1 --2- 172.21.15.28:0/3739790603 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f547c06dbf0 0x7f547c158ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:27.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.042+0000 7f547b200640 1 -- 172.21.15.28:0/3739790603 >> 172.21.15.28:0/3739790603 conn(0x7f547c0ac370 msgr2=0x7f547c0ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:27.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.042+0000 7f547b200640 1 -- 172.21.15.28:0/3739790603 shutdown_connections 2024-02-21T23:12:27.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.042+0000 7f547b200640 1 -- 172.21.15.28:0/3739790603 wait complete. 2024-02-21T23:12:27.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.043+0000 7f547b200640 1 Processor -- start 2024-02-21T23:12:27.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.043+0000 7f547b200640 1 -- start start 2024-02-21T23:12:27.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.044+0000 7f547b200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f547c06dbf0 0x7f547c0af0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:27.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.044+0000 7f547b200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f547c0aff90 0x7f547c0c22d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:27.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.044+0000 7f547b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f547c155960 0x7f547c0c2810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:27.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.044+0000 7f547b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f547c0c58f0 con 0x7f547c155960 2024-02-21T23:12:27.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.044+0000 7f547b200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f547c0be0c0 con 0x7f547c06dbf0 2024-02-21T23:12:27.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.044+0000 7f547b200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f547c0c5a70 con 0x7f547c0aff90 2024-02-21T23:12:27.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.044+0000 7f547a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f547c155960 0x7f547c0c2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:27.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.045+0000 7f547a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f547c155960 0x7f547c0c2810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44246/0 (socket says 172.21.15.28:44246) 2024-02-21T23:12:27.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.045+0000 7f547a800640 1 -- 172.21.15.28:0/4269499573 learned_addr learned my addr 172.21.15.28:0/4269499573 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:27.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.045+0000 7f547a800640 1 -- 172.21.15.28:0/4269499573 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f547c0aff90 msgr2=0x7f547c0c22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:27.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.045+0000 7f547a800640 1 --2- 172.21.15.28:0/4269499573 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f547c0aff90 0x7f547c0c22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:27.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.045+0000 7f5479e00640 1 --2- 172.21.15.28:0/4269499573 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f547c06dbf0 0x7f547c0af0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:27.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.045+0000 7f547a800640 1 -- 172.21.15.28:0/4269499573 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f547c06dbf0 msgr2=0x7f547c0af0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:27.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.045+0000 7f547a800640 1 --2- 172.21.15.28:0/4269499573 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f547c06dbf0 0x7f547c0af0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:27.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.045+0000 7f547a800640 1 -- 172.21.15.28:0/4269499573 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f547c0ad8d0 con 0x7f547c155960 2024-02-21T23:12:27.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.045+0000 7f547a800640 1 --2- 172.21.15.28:0/4269499573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f547c155960 0x7f547c0c2810 secure :-1 s=READY pgs=2231 cs=0 l=1 rev1=1 crypto rx=0x7f5474009d90 tx=0x7f5474004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:27.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.046+0000 7f5472000640 1 -- 172.21.15.28:0/4269499573 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5474044020 con 0x7f547c155960 2024-02-21T23:12:27.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.046+0000 7f5472000640 1 -- 172.21.15.28:0/4269499573 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f547402ca50 con 0x7f547c155960 2024-02-21T23:12:27.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.046+0000 7f5472000640 1 -- 172.21.15.28:0/4269499573 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5474039040 con 0x7f547c155960 2024-02-21T23:12:27.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.046+0000 7f547b200640 1 -- 172.21.15.28:0/4269499573 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f547c0adbb0 con 0x7f547c155960 2024-02-21T23:12:27.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.046+0000 7f547b200640 1 -- 172.21.15.28:0/4269499573 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f547c0ae170 con 0x7f547c155960 2024-02-21T23:12:27.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.047+0000 7f547b200640 1 -- 172.21.15.28:0/4269499573 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f547c04f290 con 0x7f547c155960 2024-02-21T23:12:27.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.048+0000 7f5472000640 1 -- 172.21.15.28:0/4269499573 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5474047020 con 0x7f547c155960 2024-02-21T23:12:27.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.049+0000 7f5472000640 1 --2- 172.21.15.28:0/4269499573 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f545c06b4d0 0x7f545c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:27.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.049+0000 7f5472000640 1 -- 172.21.15.28:0/4269499573 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f54740acc60 con 0x7f547c155960 2024-02-21T23:12:27.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.050+0000 7f5479e00640 1 --2- 172.21.15.28:0/4269499573 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f545c06b4d0 0x7f545c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:27.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.051+0000 7f5479e00640 1 --2- 172.21.15.28:0/4269499573 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f545c06b4d0 0x7f545c06d990 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f547c0c40e0 tx=0x7f546c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:27.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.054+0000 7f5472000640 1 -- 172.21.15.28:0/4269499573 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f547404e030 con 0x7f547c155960 2024-02-21T23:12:27.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.260+0000 7f547b200640 1 -- 172.21.15.28:0/4269499573 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f547c0ac150 con 0x7f547c155960 2024-02-21T23:12:27.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.263+0000 7f5472000640 1 -- 172.21.15.28:0/4269499573 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v196) v1 ==== 74+0+16905 (secure 0 0 0) 0x7f547402cc20 con 0x7f547c155960 2024-02-21T23:12:27.252 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:12:27.252 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":196,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:12:26.258927+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:12:24.201589+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"196","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2024-02-21T23:12:25.295387+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"196","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:12:27.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.265+0000 7f547b200640 1 -- 172.21.15.28:0/4269499573 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f545c06b4d0 msgr2=0x7f545c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:27.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.265+0000 7f547b200640 1 --2- 172.21.15.28:0/4269499573 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f545c06b4d0 0x7f545c06d990 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f547c0c40e0 tx=0x7f546c009040 comp rx=0 tx=0).stop 2024-02-21T23:12:27.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.265+0000 7f547b200640 1 -- 172.21.15.28:0/4269499573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f547c155960 msgr2=0x7f547c0c2810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:27.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.265+0000 7f547b200640 1 --2- 172.21.15.28:0/4269499573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f547c155960 0x7f547c0c2810 secure :-1 s=READY pgs=2231 cs=0 l=1 rev1=1 crypto rx=0x7f5474009d90 tx=0x7f5474004a40 comp rx=0 tx=0).stop 2024-02-21T23:12:27.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.265+0000 7f547b200640 1 -- 172.21.15.28:0/4269499573 shutdown_connections 2024-02-21T23:12:27.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.265+0000 7f547b200640 1 --2- 172.21.15.28:0/4269499573 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f545c06b4d0 0x7f545c06d990 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:27.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.265+0000 7f547b200640 1 --2- 172.21.15.28:0/4269499573 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f547c0aff90 0x7f547c0c22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:27.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.265+0000 7f547b200640 1 --2- 172.21.15.28:0/4269499573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f547c155960 0x7f547c0c2810 unknown :-1 s=CLOSED pgs=2231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:27.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.265+0000 7f547b200640 1 --2- 172.21.15.28:0/4269499573 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f547c06dbf0 0x7f547c0af0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:27.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.265+0000 7f547b200640 1 -- 172.21.15.28:0/4269499573 >> 172.21.15.28:0/4269499573 conn(0x7f547c0ac370 msgr2=0x7f547c0c50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:27.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.265+0000 7f547b200640 1 -- 172.21.15.28:0/4269499573 shutdown_connections 2024-02-21T23:12:27.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.266+0000 7f547b200640 1 -- 172.21.15.28:0/4269499573 wait complete. 2024-02-21T23:12:27.275 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:12:27.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.506+0000 7f9704a00640 1 Processor -- start 2024-02-21T23:12:27.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.506+0000 7f9704a00640 1 -- start start 2024-02-21T23:12:27.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.507+0000 7f9704a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f97000a5f40 0x7f970006dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:27.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.507+0000 7f9704a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f970006e750 con 0x7f97000c0270 2024-02-21T23:12:27.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.507+0000 7f9704a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f970006ea30 con 0x7f97000c3490 2024-02-21T23:12:27.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.507+0000 7f9704a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f970006eff0 con 0x7f97000a5f40 2024-02-21T23:12:27.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.508+0000 7f96fd400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f97000a5f40 0x7f970006dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:27.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.508+0000 7f96fd400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f97000a5f40 0x7f970006dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44260/0 (socket says 172.21.15.28:44260) 2024-02-21T23:12:27.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.508+0000 7f96fd400640 1 -- 172.21.15.28:0/1293509514 learned_addr learned my addr 172.21.15.28:0/1293509514 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:27.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.508+0000 7f96fd400640 1 -- 172.21.15.28:0/1293509514 >> v1:172.21.15.81:6789/0 conn(0x7f97000c3490 legacy=0x7f9700152140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:27.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.508+0000 7f96fd400640 1 -- 172.21.15.28:0/1293509514 >> v1:172.21.15.28:6789/0 conn(0x7f97000c0270 legacy=0x7f97000c06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:27.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.508+0000 7f96fd400640 1 -- 172.21.15.28:0/1293509514 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f970006ed10 con 0x7f97000a5f40 2024-02-21T23:12:27.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.509+0000 7f96fd400640 1 --2- 172.21.15.28:0/1293509514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97000a5f40 0x7f970006dfb0 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 crypto rx=0x7f96e8009e50 tx=0x7f96e802bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85dc56bdf216a1c6 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:27.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.510+0000 7f96fca00640 1 -- 172.21.15.28:0/1293509514 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96e802cb10 con 0x7f97000a5f40 2024-02-21T23:12:27.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.510+0000 7f96fca00640 1 -- 172.21.15.28:0/1293509514 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96e802ccb0 con 0x7f97000a5f40 2024-02-21T23:12:27.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.510+0000 7f96fca00640 1 -- 172.21.15.28:0/1293509514 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96e802ce60 con 0x7f97000a5f40 2024-02-21T23:12:27.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.510+0000 7f9704a00640 1 -- 172.21.15.28:0/1293509514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97000a5f40 msgr2=0x7f970006dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:27.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.510+0000 7f9704a00640 1 --2- 172.21.15.28:0/1293509514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97000a5f40 0x7f970006dfb0 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 crypto rx=0x7f96e8009e50 tx=0x7f96e802bf70 comp rx=0 tx=0).stop 2024-02-21T23:12:27.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.510+0000 7f9704a00640 1 -- 172.21.15.28:0/1293509514 shutdown_connections 2024-02-21T23:12:27.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.510+0000 7f9704a00640 1 --2- 172.21.15.28:0/1293509514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97000a5f40 0x7f970006dfb0 unknown :-1 s=CLOSED pgs=2233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:27.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.510+0000 7f9704a00640 1 -- 172.21.15.28:0/1293509514 >> 172.21.15.28:0/1293509514 conn(0x7f97000c67f0 msgr2=0x7f97000c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:27.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.510+0000 7f9704a00640 1 -- 172.21.15.28:0/1293509514 shutdown_connections 2024-02-21T23:12:27.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.510+0000 7f9704a00640 1 -- 172.21.15.28:0/1293509514 wait complete. 2024-02-21T23:12:27.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.512+0000 7f9704a00640 1 Processor -- start 2024-02-21T23:12:27.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.512+0000 7f9704a00640 1 -- start start 2024-02-21T23:12:27.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.513+0000 7f9704a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f97000a5f40 0x7f9700072ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:27.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.513+0000 7f9704a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f97000c0270 0x7f97000a6ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:27.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.513+0000 7f9704a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97000c3490 0x7f97000a7000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:27.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.513+0000 7f9704a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f970006e4f0 con 0x7f97000c3490 2024-02-21T23:12:27.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.513+0000 7f9704a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97000bfde0 con 0x7f97000c0270 2024-02-21T23:12:27.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.513+0000 7f9704a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9700152850 con 0x7f97000a5f40 2024-02-21T23:12:27.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.513+0000 7f96fe800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97000c3490 0x7f97000a7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:27.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.513+0000 7f96fe800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97000c3490 0x7f97000a7000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44274/0 (socket says 172.21.15.28:44274) 2024-02-21T23:12:27.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.513+0000 7f96fe800640 1 -- 172.21.15.28:0/1593080500 learned_addr learned my addr 172.21.15.28:0/1593080500 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:27.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.514+0000 7f96fd400640 1 --2- 172.21.15.28:0/1593080500 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f97000c0270 0x7f97000a6ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:27.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.514+0000 7f96fde00640 1 --2- 172.21.15.28:0/1593080500 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f97000a5f40 0x7f9700072ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:27.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.514+0000 7f96fe800640 1 -- 172.21.15.28:0/1593080500 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f97000a5f40 msgr2=0x7f9700072ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:27.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.514+0000 7f96fe800640 1 --2- 172.21.15.28:0/1593080500 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f97000a5f40 0x7f9700072ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:27.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.514+0000 7f96fe800640 1 -- 172.21.15.28:0/1593080500 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f97000c0270 msgr2=0x7f97000a6ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:27.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.514+0000 7f96fe800640 1 --2- 172.21.15.28:0/1593080500 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f97000c0270 0x7f97000a6ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:27.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.514+0000 7f96fe800640 1 -- 172.21.15.28:0/1593080500 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97001531f0 con 0x7f97000c3490 2024-02-21T23:12:27.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.514+0000 7f96fd400640 1 --2- 172.21.15.28:0/1593080500 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f97000c0270 0x7f97000a6ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:12:27.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.514+0000 7f96fe800640 1 --2- 172.21.15.28:0/1593080500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97000c3490 0x7f97000a7000 secure :-1 s=READY pgs=2234 cs=0 l=1 rev1=1 crypto rx=0x7f96f8001eb0 tx=0x7f96f800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:27.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.514+0000 7f96f6000640 1 -- 172.21.15.28:0/1593080500 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96f8017070 con 0x7f97000c3490 2024-02-21T23:12:27.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.514+0000 7f96f6000640 1 -- 172.21.15.28:0/1593080500 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96f80052d0 con 0x7f97000c3490 2024-02-21T23:12:27.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.514+0000 7f96f6000640 1 -- 172.21.15.28:0/1593080500 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96f8003810 con 0x7f97000c3490 2024-02-21T23:12:27.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.514+0000 7f9704a00640 1 -- 172.21.15.28:0/1593080500 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97001ebc20 con 0x7f97000c3490 2024-02-21T23:12:27.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.515+0000 7f9704a00640 1 -- 172.21.15.28:0/1593080500 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97001eb930 con 0x7f97000c3490 2024-02-21T23:12:27.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.516+0000 7f9704a00640 1 -- 172.21.15.28:0/1593080500 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9700067f30 con 0x7f97000c3490 2024-02-21T23:12:27.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.517+0000 7f96f6000640 1 -- 172.21.15.28:0/1593080500 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f96f801a070 con 0x7f97000c3490 2024-02-21T23:12:27.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.518+0000 7f96f6000640 1 --2- 172.21.15.28:0/1593080500 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f96dc06b4d0 0x7f96dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:27.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.518+0000 7f96f6000640 1 -- 172.21.15.28:0/1593080500 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f96f8089700 con 0x7f97000c3490 2024-02-21T23:12:27.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.518+0000 7f96fde00640 1 --2- 172.21.15.28:0/1593080500 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f96dc06b4d0 0x7f96dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:27.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.519+0000 7f96fde00640 1 --2- 172.21.15.28:0/1593080500 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f96dc06b4d0 0x7f96dc06d990 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f9700156c10 tx=0x7f96f0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:27.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.522+0000 7f96f6000640 1 -- 172.21.15.28:0/1593080500 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f96f80583e0 con 0x7f97000c3490 2024-02-21T23:12:27.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.723+0000 7f9704a00640 1 -- 172.21.15.28:0/1593080500 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f97000732a0 con 0x7f97000c3490 2024-02-21T23:12:27.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.725+0000 7f96f6000640 1 -- 172.21.15.28:0/1593080500 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v196) v1 ==== 74+0+16905 (secure 0 0 0) 0x7f96f8087e20 con 0x7f97000c3490 2024-02-21T23:12:27.714 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:12:27.714 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":196,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:12:26.258927+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:12:24.201589+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"196","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2024-02-21T23:12:25.295387+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"196","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:12:27.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.728+0000 7f9704a00640 1 -- 172.21.15.28:0/1593080500 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f96dc06b4d0 msgr2=0x7f96dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:27.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.728+0000 7f9704a00640 1 --2- 172.21.15.28:0/1593080500 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f96dc06b4d0 0x7f96dc06d990 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f9700156c10 tx=0x7f96f0009040 comp rx=0 tx=0).stop 2024-02-21T23:12:27.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.728+0000 7f9704a00640 1 -- 172.21.15.28:0/1593080500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97000c3490 msgr2=0x7f97000a7000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:27.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.728+0000 7f9704a00640 1 --2- 172.21.15.28:0/1593080500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97000c3490 0x7f97000a7000 secure :-1 s=READY pgs=2234 cs=0 l=1 rev1=1 crypto rx=0x7f96f8001eb0 tx=0x7f96f800c890 comp rx=0 tx=0).stop 2024-02-21T23:12:27.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.728+0000 7f9704a00640 1 -- 172.21.15.28:0/1593080500 shutdown_connections 2024-02-21T23:12:27.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.728+0000 7f9704a00640 1 --2- 172.21.15.28:0/1593080500 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f96dc06b4d0 0x7f96dc06d990 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:27.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.728+0000 7f9704a00640 1 --2- 172.21.15.28:0/1593080500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97000c3490 0x7f97000a7000 unknown :-1 s=CLOSED pgs=2234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:27.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.728+0000 7f9704a00640 1 --2- 172.21.15.28:0/1593080500 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f97000c0270 0x7f97000a6ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:27.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.728+0000 7f9704a00640 1 --2- 172.21.15.28:0/1593080500 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f97000a5f40 0x7f9700072ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:27.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.728+0000 7f9704a00640 1 -- 172.21.15.28:0/1593080500 >> 172.21.15.28:0/1593080500 conn(0x7f97000c67f0 msgr2=0x7f970014b1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:27.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.728+0000 7f9704a00640 1 -- 172.21.15.28:0/1593080500 shutdown_connections 2024-02-21T23:12:27.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.728+0000 7f9704a00640 1 -- 172.21.15.28:0/1593080500 wait complete. 2024-02-21T23:12:27.738 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-02-21T23:12:27.744 INFO:tasks.ceph.mds.d.smithi081.stderr:2024-02-21T23:12:27.756+0000 7f9270400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-21T23:12:27.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.980+0000 7f6954c00640 1 Processor -- start 2024-02-21T23:12:27.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.980+0000 7f6954c00640 1 -- start start 2024-02-21T23:12:27.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.982+0000 7f6954c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f695013e0c0 0x7f69500c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:27.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.982+0000 7f6954c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f695006dbc0 0x7f695006c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:27.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.982+0000 7f6954c00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6950158550 con 0x7f69500c76f0 2024-02-21T23:12:27.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.982+0000 7f6954c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69500c89a0 con 0x7f695006dbc0 2024-02-21T23:12:27.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.982+0000 7f6954c00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69501587d0 con 0x7f695013e0c0 2024-02-21T23:12:27.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.982+0000 7f6946a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f695006dbc0 0x7f695006c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:27.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.982+0000 7f6946a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f695006dbc0 0x7f695006c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44290/0 (socket says 172.21.15.28:44290) 2024-02-21T23:12:27.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.982+0000 7f6946a00640 1 -- 172.21.15.28:0/1458610542 learned_addr learned my addr 172.21.15.28:0/1458610542 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:27.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.983+0000 7f694f400640 1 --2- 172.21.15.28:0/1458610542 >> v2:172.21.15.81:3300/0 conn(0x7f695013e0c0 0x7f69500c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:27.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.983+0000 7f6946a00640 1 -- 172.21.15.28:0/1458610542 >> v2:172.21.15.81:3300/0 conn(0x7f695013e0c0 msgr2=0x7f69500c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:27.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.983+0000 7f6946a00640 1 --2- 172.21.15.28:0/1458610542 >> v2:172.21.15.81:3300/0 conn(0x7f695013e0c0 0x7f69500c71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:27.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.983+0000 7f6946a00640 1 -- 172.21.15.28:0/1458610542 >> v1:172.21.15.81:6789/0 conn(0x7f69500c76f0 legacy=0x7f695006c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:27.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.983+0000 7f6946a00640 1 -- 172.21.15.28:0/1458610542 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6950158ab0 con 0x7f695006dbc0 2024-02-21T23:12:27.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.984+0000 7f6946a00640 1 --2- 172.21.15.28:0/1458610542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f695006dbc0 0x7f695006c280 secure :-1 s=READY pgs=2235 cs=0 l=1 rev1=1 crypto rx=0x7f69500ad180 tx=0x7f693802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bbf4dfeaa36c94c server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:27.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.984+0000 7f694ea00640 1 -- 172.21.15.28:0/1458610542 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f693802e070 con 0x7f695006dbc0 2024-02-21T23:12:27.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.984+0000 7f694ea00640 1 -- 172.21.15.28:0/1458610542 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6938037070 con 0x7f695006dbc0 2024-02-21T23:12:27.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.985+0000 7f694ea00640 1 -- 172.21.15.28:0/1458610542 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6938031a00 con 0x7f695006dbc0 2024-02-21T23:12:27.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.985+0000 7f6954c00640 1 -- 172.21.15.28:0/1458610542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f695006dbc0 msgr2=0x7f695006c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:27.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.985+0000 7f6954c00640 1 --2- 172.21.15.28:0/1458610542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f695006dbc0 0x7f695006c280 secure :-1 s=READY pgs=2235 cs=0 l=1 rev1=1 crypto rx=0x7f69500ad180 tx=0x7f693802b880 comp rx=0 tx=0).stop 2024-02-21T23:12:27.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.985+0000 7f6954c00640 1 -- 172.21.15.28:0/1458610542 shutdown_connections 2024-02-21T23:12:27.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.985+0000 7f6954c00640 1 --2- 172.21.15.28:0/1458610542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f695006dbc0 0x7f695006c280 unknown :-1 s=CLOSED pgs=2235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:27.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.985+0000 7f6954c00640 1 --2- 172.21.15.28:0/1458610542 >> v2:172.21.15.81:3300/0 conn(0x7f695013e0c0 0x7f69500c71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:27.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.985+0000 7f6954c00640 1 -- 172.21.15.28:0/1458610542 >> 172.21.15.28:0/1458610542 conn(0x7f6950073110 msgr2=0x7f69500ac490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:27.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.985+0000 7f6954c00640 1 -- 172.21.15.28:0/1458610542 shutdown_connections 2024-02-21T23:12:27.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.985+0000 7f6954c00640 1 -- 172.21.15.28:0/1458610542 wait complete. 2024-02-21T23:12:27.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.987+0000 7f6954c00640 1 Processor -- start 2024-02-21T23:12:27.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.987+0000 7f6954c00640 1 -- start start 2024-02-21T23:12:27.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.988+0000 7f6954c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f695006dbc0 0x7f6950136d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:27.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.988+0000 7f6954c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f69500c76f0 0x7f69501ee040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:27.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.988+0000 7f6954c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f695013e0c0 0x7f69501ec600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:27.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.988+0000 7f6954c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f695006ced0 con 0x7f695013e0c0 2024-02-21T23:12:27.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.988+0000 7f6954c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6950159070 con 0x7f69500c76f0 2024-02-21T23:12:27.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.988+0000 7f6954c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f695013e520 con 0x7f695006dbc0 2024-02-21T23:12:27.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.988+0000 7f694fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f695013e0c0 0x7f69501ec600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:27.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.989+0000 7f694fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f695013e0c0 0x7f69501ec600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44298/0 (socket says 172.21.15.28:44298) 2024-02-21T23:12:27.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.989+0000 7f694fe00640 1 -- 172.21.15.28:0/1550725553 learned_addr learned my addr 172.21.15.28:0/1550725553 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:27.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.989+0000 7f6946a00640 1 --2- 172.21.15.28:0/1550725553 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f69500c76f0 0x7f69501ee040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:27.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.989+0000 7f694f400640 1 --2- 172.21.15.28:0/1550725553 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f695006dbc0 0x7f6950136d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:27.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.989+0000 7f694fe00640 1 -- 172.21.15.28:0/1550725553 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f695006dbc0 msgr2=0x7f6950136d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:27.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.989+0000 7f694fe00640 1 --2- 172.21.15.28:0/1550725553 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f695006dbc0 0x7f6950136d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:27.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.989+0000 7f694fe00640 1 -- 172.21.15.28:0/1550725553 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f69500c76f0 msgr2=0x7f69501ee040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:27.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.989+0000 7f694fe00640 1 --2- 172.21.15.28:0/1550725553 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f69500c76f0 0x7f69501ee040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:27.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.989+0000 7f694fe00640 1 -- 172.21.15.28:0/1550725553 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69501eeaf0 con 0x7f695013e0c0 2024-02-21T23:12:27.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.989+0000 7f6946a00640 1 --2- 172.21.15.28:0/1550725553 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f69500c76f0 0x7f69501ee040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:12:27.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.989+0000 7f694fe00640 1 --2- 172.21.15.28:0/1550725553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f695013e0c0 0x7f69501ec600 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 crypto rx=0x7f694800c7b0 tx=0x7f694800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:27.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.989+0000 7f6947e00640 1 -- 172.21.15.28:0/1550725553 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6948017070 con 0x7f695013e0c0 2024-02-21T23:12:27.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.990+0000 7f6947e00640 1 -- 172.21.15.28:0/1550725553 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6948003da0 con 0x7f695013e0c0 2024-02-21T23:12:27.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.990+0000 7f6947e00640 1 -- 172.21.15.28:0/1550725553 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69480052b0 con 0x7f695013e0c0 2024-02-21T23:12:27.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.990+0000 7f6954c00640 1 -- 172.21.15.28:0/1550725553 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69501eedd0 con 0x7f695013e0c0 2024-02-21T23:12:27.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.990+0000 7f6954c00640 1 -- 172.21.15.28:0/1550725553 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69500b0570 con 0x7f695013e0c0 2024-02-21T23:12:27.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.991+0000 7f6954c00640 1 -- 172.21.15.28:0/1550725553 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f695004f290 con 0x7f695013e0c0 2024-02-21T23:12:27.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.992+0000 7f6947e00640 1 -- 172.21.15.28:0/1550725553 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f694801a070 con 0x7f695013e0c0 2024-02-21T23:12:27.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.993+0000 7f6947e00640 1 --2- 172.21.15.28:0/1550725553 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f692c06b4d0 0x7f692c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:27.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.993+0000 7f6947e00640 1 -- 172.21.15.28:0/1550725553 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6948089ad0 con 0x7f695013e0c0 2024-02-21T23:12:27.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.994+0000 7f694f400640 1 --2- 172.21.15.28:0/1550725553 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f692c06b4d0 0x7f692c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:27.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.995+0000 7f694f400640 1 --2- 172.21.15.28:0/1550725553 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f692c06b4d0 0x7f692c06d990 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f6940006fd0 tx=0x7f6940008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:27.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:27.998+0000 7f6947e00640 1 -- 172.21.15.28:0/1550725553 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f6948058720 con 0x7f695013e0c0 2024-02-21T23:12:28.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:28.241+0000 7f6954c00640 1 -- 172.21.15.28:0/1550725553 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f695013cec0 con 0x7f695013e0c0 2024-02-21T23:12:28.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:28.834+0000 7f6947e00640 1 -- 172.21.15.28:0/1550725553 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v216) v1 ==== 118+0+0 (secure 0 0 0) 0x7f6948058020 con 0x7f695013e0c0 2024-02-21T23:12:28.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:28.839+0000 7f6954c00640 1 -- 172.21.15.28:0/1550725553 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f692c06b4d0 msgr2=0x7f692c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:28.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:28.839+0000 7f6954c00640 1 --2- 172.21.15.28:0/1550725553 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f692c06b4d0 0x7f692c06d990 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f6940006fd0 tx=0x7f6940008040 comp rx=0 tx=0).stop 2024-02-21T23:12:28.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:28.839+0000 7f6954c00640 1 -- 172.21.15.28:0/1550725553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f695013e0c0 msgr2=0x7f69501ec600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:28.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:28.839+0000 7f6954c00640 1 --2- 172.21.15.28:0/1550725553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f695013e0c0 0x7f69501ec600 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 crypto rx=0x7f694800c7b0 tx=0x7f694800cc70 comp rx=0 tx=0).stop 2024-02-21T23:12:28.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:28.839+0000 7f6954c00640 1 -- 172.21.15.28:0/1550725553 shutdown_connections 2024-02-21T23:12:28.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:28.839+0000 7f6954c00640 1 --2- 172.21.15.28:0/1550725553 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f692c06b4d0 0x7f692c06d990 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:28.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:28.839+0000 7f6954c00640 1 --2- 172.21.15.28:0/1550725553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f695013e0c0 0x7f69501ec600 unknown :-1 s=CLOSED pgs=2236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:28.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:28.839+0000 7f6954c00640 1 --2- 172.21.15.28:0/1550725553 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f69500c76f0 0x7f69501ee040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:28.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:28.839+0000 7f6954c00640 1 --2- 172.21.15.28:0/1550725553 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f695006dbc0 0x7f6950136d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:28.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:28.839+0000 7f6954c00640 1 -- 172.21.15.28:0/1550725553 >> 172.21.15.28:0/1550725553 conn(0x7f6950073110 msgr2=0x7f6950157ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:28.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:28.839+0000 7f6954c00640 1 -- 172.21.15.28:0/1550725553 shutdown_connections 2024-02-21T23:12:28.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:28.839+0000 7f6954c00640 1 -- 172.21.15.28:0/1550725553 wait complete. 2024-02-21T23:12:28.849 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-02-21T23:12:28.850 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-02-21T23:12:29.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.082+0000 7f65eb200640 1 Processor -- start 2024-02-21T23:12:29.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.083+0000 7f65eb200640 1 -- start start 2024-02-21T23:12:29.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.084+0000 7f65eb200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f65ec0c4580 0x7f65ec0c12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:29.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.084+0000 7f65eb200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65ec06e220 con 0x7f65ec0c2c10 2024-02-21T23:12:29.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.084+0000 7f65eb200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65ec06f780 con 0x7f65ec150240 2024-02-21T23:12:29.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.084+0000 7f65eb200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65ec150620 con 0x7f65ec0c4580 2024-02-21T23:12:29.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.084+0000 7f65ea800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f65ec0c2c10 0x7f65ec06ae70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:32906/0 (socket says 172.21.15.28:32906) 2024-02-21T23:12:29.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.084+0000 7f65ea800640 1 -- 172.21.15.28:0/1961875807 learned_addr learned my addr 172.21.15.28:0/1961875807 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:12:29.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.085+0000 7f65e9400640 1 --2- 172.21.15.28:0/1961875807 >> v2:172.21.15.81:3300/0 conn(0x7f65ec0c4580 0x7f65ec0c12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:29.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.085+0000 7f65e8a00640 1 -- 172.21.15.28:0/1961875807 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1095155762 0 0) 0x7f65ec06e220 con 0x7f65ec0c2c10 2024-02-21T23:12:29.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.085+0000 7f65e8a00640 1 -- 172.21.15.28:0/1961875807 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65d0003520 con 0x7f65ec0c2c10 2024-02-21T23:12:29.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.085+0000 7f65e8a00640 1 -- 172.21.15.28:0/1961875807 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1131258391 0 0) 0x7f65d0003520 con 0x7f65ec0c2c10 2024-02-21T23:12:29.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.085+0000 7f65e8a00640 1 -- 172.21.15.28:0/1961875807 >> v2:172.21.15.81:3300/0 conn(0x7f65ec0c4580 msgr2=0x7f65ec0c12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:29.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.085+0000 7f65e8a00640 1 --2- 172.21.15.28:0/1961875807 >> v2:172.21.15.81:3300/0 conn(0x7f65ec0c4580 0x7f65ec0c12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:29.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.085+0000 7f65e8a00640 1 -- 172.21.15.28:0/1961875807 >> v1:172.21.15.81:6789/0 conn(0x7f65ec150240 legacy=0x7f65ec06db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:29.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.085+0000 7f65e8a00640 1 -- 172.21.15.28:0/1961875807 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65ec06f4a0 con 0x7f65ec0c2c10 2024-02-21T23:12:29.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.085+0000 7f65e8a00640 1 -- 172.21.15.28:0/1961875807 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f65e40019f0 con 0x7f65ec0c2c10 2024-02-21T23:12:29.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.086+0000 7f65e8a00640 1 -- 172.21.15.28:0/1961875807 >> v1:172.21.15.28:6789/0 conn(0x7f65ec0c2c10 legacy=0x7f65ec06ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:29.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.086+0000 7f65e8a00640 1 --2- 172.21.15.28:0/1961875807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65d00039b0 0x7f65d0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:29.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.086+0000 7f65e8a00640 1 --2- 172.21.15.28:0/1961875807 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f65d0004800 0x7f65d0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:29.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.086+0000 7f65e8a00640 1 --2- 172.21.15.28:0/1961875807 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65d00051c0 0x7f65d000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:29.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.086+0000 7f65e8a00640 1 -- 172.21.15.28:0/1961875807 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65d000c130 con 0x7f65d00039b0 2024-02-21T23:12:29.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.086+0000 7f65e8a00640 1 -- 172.21.15.28:0/1961875807 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65d000c2e0 con 0x7f65d00051c0 2024-02-21T23:12:29.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.086+0000 7f65e8a00640 1 -- 172.21.15.28:0/1961875807 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65d000c4e0 con 0x7f65d0004800 2024-02-21T23:12:29.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.086+0000 7f65e9e00640 1 --2- 172.21.15.28:0/1961875807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65d00039b0 0x7f65d0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:29.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.086+0000 7f65ea800640 1 --2- 172.21.15.28:0/1961875807 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65d00051c0 0x7f65d000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:29.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.086+0000 7f65e9e00640 1 -- 172.21.15.28:0/1961875807 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f65d0004800 msgr2=0x7f65d0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:29.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.086+0000 7f65e9e00640 1 --2- 172.21.15.28:0/1961875807 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f65d0004800 0x7f65d0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:29.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.086+0000 7f65e9e00640 1 -- 172.21.15.28:0/1961875807 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65d00051c0 msgr2=0x7f65d000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:29.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.086+0000 7f65e9e00640 1 --2- 172.21.15.28:0/1961875807 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65d00051c0 0x7f65d000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:29.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.086+0000 7f65e9e00640 1 -- 172.21.15.28:0/1961875807 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65d0003520 con 0x7f65d00039b0 2024-02-21T23:12:29.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.086+0000 7f65ea800640 1 --2- 172.21.15.28:0/1961875807 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65d00051c0 0x7f65d000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:12:29.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.086+0000 7f65ea800640 1 -- 172.21.15.28:0/1961875807 reap_dead start 2024-02-21T23:12:29.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.088+0000 7f65e9e00640 1 --2- 172.21.15.28:0/1961875807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65d00039b0 0x7f65d0003dd0 secure :-1 s=READY pgs=2238 cs=0 l=1 rev1=1 crypto rx=0x7f65dc001750 tx=0x7f65d0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=335e888f39359ff9 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:29.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.088+0000 7f65e8a00640 1 -- 172.21.15.28:0/1961875807 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65dc001370 con 0x7f65d00039b0 2024-02-21T23:12:29.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.088+0000 7f65e8a00640 1 -- 172.21.15.28:0/1961875807 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65dc0052d0 con 0x7f65d00039b0 2024-02-21T23:12:29.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.088+0000 7f65e8a00640 1 -- 172.21.15.28:0/1961875807 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65dc003830 con 0x7f65d00039b0 2024-02-21T23:12:29.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.088+0000 7f65eb200640 1 -- 172.21.15.28:0/1961875807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65d00039b0 msgr2=0x7f65d0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:29.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.088+0000 7f65eb200640 1 --2- 172.21.15.28:0/1961875807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65d00039b0 0x7f65d0003dd0 secure :-1 s=READY pgs=2238 cs=0 l=1 rev1=1 crypto rx=0x7f65dc001750 tx=0x7f65d0005830 comp rx=0 tx=0).stop 2024-02-21T23:12:29.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.089+0000 7f65eb200640 1 -- 172.21.15.28:0/1961875807 shutdown_connections 2024-02-21T23:12:29.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.089+0000 7f65eb200640 1 --2- 172.21.15.28:0/1961875807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65d00039b0 0x7f65d0003dd0 unknown :-1 s=CLOSED pgs=2238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:29.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.089+0000 7f65eb200640 1 -- 172.21.15.28:0/1961875807 >> 172.21.15.28:0/1961875807 conn(0x7f65ec0a8530 msgr2=0x7f65ec0a8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:29.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.089+0000 7f65eb200640 1 -- 172.21.15.28:0/1961875807 shutdown_connections 2024-02-21T23:12:29.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.089+0000 7f65eb200640 1 -- 172.21.15.28:0/1961875807 wait complete. 2024-02-21T23:12:29.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.090+0000 7f65eb200640 1 Processor -- start 2024-02-21T23:12:29.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.090+0000 7f65eb200640 1 -- start start 2024-02-21T23:12:29.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.091+0000 7f65eb200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65d00039b0 0x7f65ec156a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:29.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.091+0000 7f65eb200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65ec1497b0 0x7f65ec0bfe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:29.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.091+0000 7f65eb200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f65ec072030 0x7f65ec1eb080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:29.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.091+0000 7f65eb200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65ec0c4a10 con 0x7f65ec1497b0 2024-02-21T23:12:29.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.091+0000 7f65eb200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65ec153a90 con 0x7f65d00039b0 2024-02-21T23:12:29.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.091+0000 7f65eb200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65ec0c4b90 con 0x7f65ec072030 2024-02-21T23:12:29.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.092+0000 7f65e9400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65ec1497b0 0x7f65ec0bfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:29.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.092+0000 7f65e9400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65ec1497b0 0x7f65ec0bfe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44320/0 (socket says 172.21.15.28:44320) 2024-02-21T23:12:29.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.092+0000 7f65e9400640 1 -- 172.21.15.28:0/3696343112 learned_addr learned my addr 172.21.15.28:0/3696343112 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:29.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.092+0000 7f65e9e00640 1 --2- 172.21.15.28:0/3696343112 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65d00039b0 0x7f65ec156a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:29.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.092+0000 7f65ea800640 1 --2- 172.21.15.28:0/3696343112 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f65ec072030 0x7f65ec1eb080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:29.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.092+0000 7f65e9400640 1 -- 172.21.15.28:0/3696343112 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f65ec072030 msgr2=0x7f65ec1eb080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:29.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.092+0000 7f65e9400640 1 --2- 172.21.15.28:0/3696343112 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f65ec072030 0x7f65ec1eb080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:29.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.092+0000 7f65e9400640 1 -- 172.21.15.28:0/3696343112 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65d00039b0 msgr2=0x7f65ec156a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:29.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.092+0000 7f65e9400640 1 --2- 172.21.15.28:0/3696343112 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65d00039b0 0x7f65ec156a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:29.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.092+0000 7f65e9400640 1 -- 172.21.15.28:0/3696343112 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65ec1eab80 con 0x7f65ec1497b0 2024-02-21T23:12:29.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.092+0000 7f65ea800640 1 --2- 172.21.15.28:0/3696343112 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f65ec072030 0x7f65ec1eb080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:12:29.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.093+0000 7f65e9400640 1 --2- 172.21.15.28:0/3696343112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65ec1497b0 0x7f65ec0bfe20 secure :-1 s=READY pgs=2239 cs=0 l=1 rev1=1 crypto rx=0x7f65d800e7c0 tx=0x7f65d800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:29.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.093+0000 7f65e2000640 1 -- 172.21.15.28:0/3696343112 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d8017070 con 0x7f65ec1497b0 2024-02-21T23:12:29.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.093+0000 7f65e2000640 1 -- 172.21.15.28:0/3696343112 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65d800c940 con 0x7f65ec1497b0 2024-02-21T23:12:29.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.093+0000 7f65e2000640 1 -- 172.21.15.28:0/3696343112 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d800cd30 con 0x7f65ec1497b0 2024-02-21T23:12:29.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.093+0000 7f65eb200640 1 -- 172.21.15.28:0/3696343112 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65ec071d30 con 0x7f65ec1497b0 2024-02-21T23:12:29.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.093+0000 7f65eb200640 1 -- 172.21.15.28:0/3696343112 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65ec1eebf0 con 0x7f65ec1497b0 2024-02-21T23:12:29.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.094+0000 7f65eb200640 1 -- 172.21.15.28:0/3696343112 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65ec1ea990 con 0x7f65ec1497b0 2024-02-21T23:12:29.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.095+0000 7f65e2000640 1 -- 172.21.15.28:0/3696343112 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f65d801a070 con 0x7f65ec1497b0 2024-02-21T23:12:29.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.096+0000 7f65e2000640 1 --2- 172.21.15.28:0/3696343112 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f65cc06b500 0x7f65cc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:29.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.096+0000 7f65e2000640 1 -- 172.21.15.28:0/3696343112 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f65d808a240 con 0x7f65ec1497b0 2024-02-21T23:12:29.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.097+0000 7f65e9e00640 1 --2- 172.21.15.28:0/3696343112 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f65cc06b500 0x7f65cc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:29.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.098+0000 7f65e9e00640 1 --2- 172.21.15.28:0/3696343112 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f65cc06b500 0x7f65cc06d9c0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f65dc033000 tx=0x7f65dc009030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:29.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.101+0000 7f65e2000640 1 -- 172.21.15.28:0/3696343112 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f65d8058e90 con 0x7f65ec1497b0 2024-02-21T23:12:29.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.343+0000 7f65eb200640 1 -- 172.21.15.28:0/3696343112 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f65ec1eedf0 con 0x7f65ec1497b0 2024-02-21T23:12:29.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.853+0000 7f65e2000640 1 -- 172.21.15.28:0/3696343112 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v217) v1 ==== 122+0+0 (secure 0 0 0) 0x7f65d8058790 con 0x7f65ec1497b0 2024-02-21T23:12:29.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.857+0000 7f65eb200640 1 -- 172.21.15.28:0/3696343112 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f65cc06b500 msgr2=0x7f65cc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:29.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.857+0000 7f65eb200640 1 --2- 172.21.15.28:0/3696343112 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f65cc06b500 0x7f65cc06d9c0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f65dc033000 tx=0x7f65dc009030 comp rx=0 tx=0).stop 2024-02-21T23:12:29.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.857+0000 7f65eb200640 1 -- 172.21.15.28:0/3696343112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65ec1497b0 msgr2=0x7f65ec0bfe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:29.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.857+0000 7f65eb200640 1 --2- 172.21.15.28:0/3696343112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65ec1497b0 0x7f65ec0bfe20 secure :-1 s=READY pgs=2239 cs=0 l=1 rev1=1 crypto rx=0x7f65d800e7c0 tx=0x7f65d800ec80 comp rx=0 tx=0).stop 2024-02-21T23:12:29.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.857+0000 7f65eb200640 1 -- 172.21.15.28:0/3696343112 shutdown_connections 2024-02-21T23:12:29.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.857+0000 7f65eb200640 1 --2- 172.21.15.28:0/3696343112 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f65cc06b500 0x7f65cc06d9c0 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:29.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.857+0000 7f65eb200640 1 --2- 172.21.15.28:0/3696343112 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f65ec072030 0x7f65ec1eb080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:29.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.857+0000 7f65eb200640 1 --2- 172.21.15.28:0/3696343112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65ec1497b0 0x7f65ec0bfe20 unknown :-1 s=CLOSED pgs=2239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:29.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.857+0000 7f65eb200640 1 --2- 172.21.15.28:0/3696343112 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65d00039b0 0x7f65ec156a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:29.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.857+0000 7f65eb200640 1 -- 172.21.15.28:0/3696343112 >> 172.21.15.28:0/3696343112 conn(0x7f65ec0a8530 msgr2=0x7f65ec1522b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:29.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.858+0000 7f65eb200640 1 -- 172.21.15.28:0/3696343112 shutdown_connections 2024-02-21T23:12:29.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:29.858+0000 7f65eb200640 1 -- 172.21.15.28:0/3696343112 wait complete. 2024-02-21T23:12:29.866 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-02-21T23:12:29.867 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:12:30.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.089+0000 7f7767200640 1 Processor -- start 2024-02-21T23:12:30.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.089+0000 7f7767200640 1 -- start start 2024-02-21T23:12:30.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.091+0000 7f7767200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f77600a5f40 0x7f776006dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:30.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.091+0000 7f7767200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f776006e750 con 0x7f77600c3490 2024-02-21T23:12:30.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.091+0000 7f7767200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f776006ea30 con 0x7f77600c0270 2024-02-21T23:12:30.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.091+0000 7f7767200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f776006eff0 con 0x7f77600a5f40 2024-02-21T23:12:30.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.091+0000 7f775fe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f77600c3490 0x7f7760152140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:32920/0 (socket says 172.21.15.28:32920) 2024-02-21T23:12:30.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.091+0000 7f775fe00640 1 -- 172.21.15.28:0/2911152183 learned_addr learned my addr 172.21.15.28:0/2911152183 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:12:30.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.091+0000 7f7757e00640 1 --2- 172.21.15.28:0/2911152183 >> v2:172.21.15.28:3300/0 conn(0x7f77600a5f40 0x7f776006dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:30.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.092+0000 7f7757e00640 1 -- 172.21.15.28:0/2911152183 >> v1:172.21.15.81:6789/0 conn(0x7f77600c0270 legacy=0x7f77600c06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:30.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.092+0000 7f7757e00640 1 -- 172.21.15.28:0/2911152183 >> v1:172.21.15.28:6789/0 conn(0x7f77600c3490 legacy=0x7f7760152140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:30.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.092+0000 7f7757e00640 1 -- 172.21.15.28:0/2911152183 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f776006ed10 con 0x7f77600a5f40 2024-02-21T23:12:30.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.093+0000 7f7757e00640 1 --2- 172.21.15.28:0/2911152183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77600a5f40 0x7f776006dfb0 secure :-1 s=READY pgs=2241 cs=0 l=1 rev1=1 crypto rx=0x7f77480097a0 tx=0x7f774802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efb9d72c64d7c066 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:30.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.093+0000 7f775f400640 1 -- 172.21.15.28:0/2911152183 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7748035020 con 0x7f77600a5f40 2024-02-21T23:12:30.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.093+0000 7f775f400640 1 -- 172.21.15.28:0/2911152183 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7748031830 con 0x7f77600a5f40 2024-02-21T23:12:30.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.093+0000 7f775f400640 1 -- 172.21.15.28:0/2911152183 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7748031c00 con 0x7f77600a5f40 2024-02-21T23:12:30.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.094+0000 7f7767200640 1 -- 172.21.15.28:0/2911152183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77600a5f40 msgr2=0x7f776006dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:30.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.094+0000 7f7767200640 1 --2- 172.21.15.28:0/2911152183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77600a5f40 0x7f776006dfb0 secure :-1 s=READY pgs=2241 cs=0 l=1 rev1=1 crypto rx=0x7f77480097a0 tx=0x7f774802b8d0 comp rx=0 tx=0).stop 2024-02-21T23:12:30.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.094+0000 7f7767200640 1 -- 172.21.15.28:0/2911152183 shutdown_connections 2024-02-21T23:12:30.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.094+0000 7f7767200640 1 --2- 172.21.15.28:0/2911152183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77600a5f40 0x7f776006dfb0 unknown :-1 s=CLOSED pgs=2241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:30.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.094+0000 7f7767200640 1 -- 172.21.15.28:0/2911152183 >> 172.21.15.28:0/2911152183 conn(0x7f77600c67f0 msgr2=0x7f77600c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:30.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.094+0000 7f7767200640 1 -- 172.21.15.28:0/2911152183 shutdown_connections 2024-02-21T23:12:30.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.094+0000 7f7767200640 1 -- 172.21.15.28:0/2911152183 wait complete. 2024-02-21T23:12:30.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.096+0000 7f7767200640 1 Processor -- start 2024-02-21T23:12:30.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.096+0000 7f7767200640 1 -- start start 2024-02-21T23:12:30.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.097+0000 7f7767200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77600a5f40 0x7f7760072540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:30.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.097+0000 7f7767200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f77600c0270 0x7f77601eb130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:30.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.097+0000 7f7767200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f77600c3490 0x7f77601eb670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:30.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.097+0000 7f7767200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f776006e4f0 con 0x7f77600a5f40 2024-02-21T23:12:30.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.097+0000 7f7767200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77600bfde0 con 0x7f77600c0270 2024-02-21T23:12:30.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.097+0000 7f7767200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7760152850 con 0x7f77600c3490 2024-02-21T23:12:30.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.097+0000 7f775fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77600a5f40 0x7f7760072540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:30.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.097+0000 7f775fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77600a5f40 0x7f7760072540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44336/0 (socket says 172.21.15.28:44336) 2024-02-21T23:12:30.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.097+0000 7f775fe00640 1 -- 172.21.15.28:0/4091497691 learned_addr learned my addr 172.21.15.28:0/4091497691 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:30.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.097+0000 7f7757e00640 1 --2- 172.21.15.28:0/4091497691 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f77600c0270 0x7f77601eb130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:30.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.097+0000 7f7765200640 1 --2- 172.21.15.28:0/4091497691 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f77600c3490 0x7f77601eb670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:30.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.097+0000 7f775fe00640 1 -- 172.21.15.28:0/4091497691 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f77600c3490 msgr2=0x7f77601eb670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:30.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.097+0000 7f775fe00640 1 --2- 172.21.15.28:0/4091497691 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f77600c3490 0x7f77601eb670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:30.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.098+0000 7f775fe00640 1 -- 172.21.15.28:0/4091497691 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f77600c0270 msgr2=0x7f77601eb130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:30.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.098+0000 7f775fe00640 1 --2- 172.21.15.28:0/4091497691 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f77600c0270 0x7f77601eb130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:30.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.098+0000 7f775fe00640 1 -- 172.21.15.28:0/4091497691 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77601eac50 con 0x7f77600a5f40 2024-02-21T23:12:30.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.098+0000 7f7757e00640 1 --2- 172.21.15.28:0/4091497691 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f77600c0270 0x7f77601eb130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:12:30.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.098+0000 7f775fe00640 1 --2- 172.21.15.28:0/4091497691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77600a5f40 0x7f7760072540 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 crypto rx=0x7f775000c480 tx=0x7f775000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:30.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.098+0000 7f775cc00640 1 -- 172.21.15.28:0/4091497691 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7750018070 con 0x7f77600a5f40 2024-02-21T23:12:30.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.098+0000 7f775cc00640 1 -- 172.21.15.28:0/4091497691 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7750005ce0 con 0x7f77600a5f40 2024-02-21T23:12:30.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.098+0000 7f775cc00640 1 -- 172.21.15.28:0/4091497691 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77500072d0 con 0x7f77600a5f40 2024-02-21T23:12:30.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.098+0000 7f7767200640 1 -- 172.21.15.28:0/4091497691 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f776014d9f0 con 0x7f77600a5f40 2024-02-21T23:12:30.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.098+0000 7f7767200640 1 -- 172.21.15.28:0/4091497691 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7760072a80 con 0x7f77600a5f40 2024-02-21T23:12:30.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.099+0000 7f7767200640 1 -- 172.21.15.28:0/4091497691 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77601eae20 con 0x7f77600a5f40 2024-02-21T23:12:30.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.100+0000 7f775cc00640 1 -- 172.21.15.28:0/4091497691 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f775001b070 con 0x7f77600a5f40 2024-02-21T23:12:30.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.102+0000 7f775cc00640 1 --2- 172.21.15.28:0/4091497691 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f773c06b500 0x7f773c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:30.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.102+0000 7f775cc00640 1 -- 172.21.15.28:0/4091497691 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f7750089740 con 0x7f77600a5f40 2024-02-21T23:12:30.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.102+0000 7f7757e00640 1 --2- 172.21.15.28:0/4091497691 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f773c06b500 0x7f773c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:30.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.103+0000 7f7757e00640 1 --2- 172.21.15.28:0/4091497691 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f773c06b500 0x7f773c06d9c0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f7748036000 tx=0x7f7748008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:30.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.106+0000 7f775cc00640 1 -- 172.21.15.28:0/4091497691 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f77500583a0 con 0x7f77600a5f40 2024-02-21T23:12:30.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.349+0000 7f7767200640 1 -- 172.21.15.28:0/4091497691 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f77600732a0 con 0x7f77600a5f40 2024-02-21T23:12:30.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.351+0000 7f775cc00640 1 -- 172.21.15.28:0/4091497691 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 217 v217) v1 ==== 95+0+4952 (secure 0 0 0) 0x7f7750057ca0 con 0x7f77600a5f40 2024-02-21T23:12:30.339 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:12:30.340 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":217,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7100,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3213818811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3213818811},{"type":"v1","addr":"172.21.15.28:6835","nonce":3213818811}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":213},{"gid":7106,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/1322911156","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":1322911156},{"type":"v1","addr":"172.21.15.81:6837","nonce":1322911156}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":213},{"gid":10386,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3814985204","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3814985204},{"type":"v1","addr":"172.21.15.28:6834","nonce":3814985204}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":212}],"filesystems":[{"mdsmap":{"epoch":217,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:12:26.259457+0000","modified":"2024-02-21T23:12:29.346312+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10398},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10398":{"gid":10398,"name":"d","rank":0,"incarnation":214,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6836/2895278683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2895278683},{"type":"v1","addr":"172.21.15.81:6836","nonce":2895278683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15}]} 2024-02-21T23:12:30.340 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 217 2024-02-21T23:12:30.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.354+0000 7f7767200640 1 -- 172.21.15.28:0/4091497691 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f773c06b500 msgr2=0x7f773c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:30.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.354+0000 7f7767200640 1 --2- 172.21.15.28:0/4091497691 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f773c06b500 0x7f773c06d9c0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f7748036000 tx=0x7f7748008030 comp rx=0 tx=0).stop 2024-02-21T23:12:30.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.354+0000 7f7767200640 1 -- 172.21.15.28:0/4091497691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77600a5f40 msgr2=0x7f7760072540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:30.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.354+0000 7f7767200640 1 --2- 172.21.15.28:0/4091497691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77600a5f40 0x7f7760072540 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 crypto rx=0x7f775000c480 tx=0x7f775000c940 comp rx=0 tx=0).stop 2024-02-21T23:12:30.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.354+0000 7f7767200640 1 -- 172.21.15.28:0/4091497691 shutdown_connections 2024-02-21T23:12:30.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.354+0000 7f7767200640 1 --2- 172.21.15.28:0/4091497691 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f773c06b500 0x7f773c06d9c0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:30.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.354+0000 7f7767200640 1 --2- 172.21.15.28:0/4091497691 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f77600c3490 0x7f77601eb670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:30.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.354+0000 7f7767200640 1 --2- 172.21.15.28:0/4091497691 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f77600c0270 0x7f77601eb130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:30.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.354+0000 7f7767200640 1 --2- 172.21.15.28:0/4091497691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77600a5f40 0x7f7760072540 unknown :-1 s=CLOSED pgs=2242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:30.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.354+0000 7f7767200640 1 -- 172.21.15.28:0/4091497691 >> 172.21.15.28:0/4091497691 conn(0x7f77600c67f0 msgr2=0x7f776014a930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:30.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.354+0000 7f7767200640 1 -- 172.21.15.28:0/4091497691 shutdown_connections 2024-02-21T23:12:30.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.354+0000 7f7767200640 1 -- 172.21.15.28:0/4091497691 wait complete. 2024-02-21T23:12:30.363 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:12:30.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.589+0000 7fe021400640 1 Processor -- start 2024-02-21T23:12:30.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.589+0000 7fe021400640 1 -- start start 2024-02-21T23:12:30.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.591+0000 7fe021400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe01c1559c0 0x7fe01c0c76f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:30.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.591+0000 7fe021400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fe01c0c7c30 0x7fe01c15f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:30.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.591+0000 7fe021400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe01c0c6f30 con 0x7fe01c1526b0 2024-02-21T23:12:30.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.591+0000 7fe021400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe01c0c61b0 con 0x7fe01c1559c0 2024-02-21T23:12:30.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.591+0000 7fe021400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe01c0c6450 con 0x7fe01c0c7c30 2024-02-21T23:12:30.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.591+0000 7fe019e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe01c1559c0 0x7fe01c0c76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:30.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.592+0000 7fe01a800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe01c1526b0 0x7fe01c0ca940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:32926/0 (socket says 172.21.15.28:32926) 2024-02-21T23:12:30.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.592+0000 7fe01a800640 1 -- 172.21.15.28:0/253026516 learned_addr learned my addr 172.21.15.28:0/253026516 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:12:30.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.592+0000 7fe01b200640 1 --2- 172.21.15.28:0/253026516 >> v2:172.21.15.81:3300/0 conn(0x7fe01c0c7c30 0x7fe01c15f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:30.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.592+0000 7fe019e00640 1 -- 172.21.15.28:0/253026516 >> v2:172.21.15.81:3300/0 conn(0x7fe01c0c7c30 msgr2=0x7fe01c15f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:30.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.592+0000 7fe019e00640 1 --2- 172.21.15.28:0/253026516 >> v2:172.21.15.81:3300/0 conn(0x7fe01c0c7c30 0x7fe01c15f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:30.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.592+0000 7fe019e00640 1 -- 172.21.15.28:0/253026516 >> v1:172.21.15.28:6789/0 conn(0x7fe01c1526b0 legacy=0x7fe01c0ca940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:30.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.592+0000 7fe019400640 1 -- 172.21.15.28:0/253026516 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3285818330 0 0) 0x7fe01c0c6f30 con 0x7fe01c1526b0 2024-02-21T23:12:30.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.592+0000 7fe019e00640 1 -- 172.21.15.28:0/253026516 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe01c0c6710 con 0x7fe01c1559c0 2024-02-21T23:12:30.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.594+0000 7fe019e00640 1 --2- 172.21.15.28:0/253026516 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe01c1559c0 0x7fe01c0c76f0 secure :-1 s=READY pgs=2244 cs=0 l=1 rev1=1 crypto rx=0x7fe01c06bdc0 tx=0x7fe00802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=194fd3312a25bc6d server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:30.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.594+0000 7fe019400640 1 -- 172.21.15.28:0/253026516 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe008035020 con 0x7fe01c1559c0 2024-02-21T23:12:30.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.594+0000 7fe019400640 1 -- 172.21.15.28:0/253026516 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe008031830 con 0x7fe01c1559c0 2024-02-21T23:12:30.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.594+0000 7fe019400640 1 -- 172.21.15.28:0/253026516 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe008031c00 con 0x7fe01c1559c0 2024-02-21T23:12:30.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.595+0000 7fe021400640 1 -- 172.21.15.28:0/253026516 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe01c1559c0 msgr2=0x7fe01c0c76f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:30.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.595+0000 7fe021400640 1 --2- 172.21.15.28:0/253026516 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe01c1559c0 0x7fe01c0c76f0 secure :-1 s=READY pgs=2244 cs=0 l=1 rev1=1 crypto rx=0x7fe01c06bdc0 tx=0x7fe00802b8d0 comp rx=0 tx=0).stop 2024-02-21T23:12:30.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.595+0000 7fe021400640 1 -- 172.21.15.28:0/253026516 shutdown_connections 2024-02-21T23:12:30.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.595+0000 7fe021400640 1 --2- 172.21.15.28:0/253026516 >> v2:172.21.15.81:3300/0 conn(0x7fe01c0c7c30 0x7fe01c15f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:30.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.595+0000 7fe021400640 1 --2- 172.21.15.28:0/253026516 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe01c1559c0 0x7fe01c0c76f0 unknown :-1 s=CLOSED pgs=2244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:30.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.595+0000 7fe021400640 1 -- 172.21.15.28:0/253026516 >> 172.21.15.28:0/253026516 conn(0x7fe01c06d3d0 msgr2=0x7fe01c08afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:30.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.595+0000 7fe021400640 1 -- 172.21.15.28:0/253026516 shutdown_connections 2024-02-21T23:12:30.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.595+0000 7fe021400640 1 -- 172.21.15.28:0/253026516 wait complete. 2024-02-21T23:12:30.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.597+0000 7fe021400640 1 Processor -- start 2024-02-21T23:12:30.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.597+0000 7fe021400640 1 -- start start 2024-02-21T23:12:30.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.598+0000 7fe021400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe01c0c7c30 0x7fe01c14c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:30.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.598+0000 7fe021400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe01c1526b0 0x7fe01c146d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:30.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.598+0000 7fe021400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe01c1559c0 0x7fe01c1472c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:30.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.598+0000 7fe021400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe01c0c5c90 con 0x7fe01c1526b0 2024-02-21T23:12:30.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.598+0000 7fe021400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe01c0c71d0 con 0x7fe01c1559c0 2024-02-21T23:12:30.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.598+0000 7fe021400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe01c0c4e70 con 0x7fe01c0c7c30 2024-02-21T23:12:30.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.598+0000 7fe019e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe01c1526b0 0x7fe01c146d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:30.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.598+0000 7fe019e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe01c1526b0 0x7fe01c146d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44362/0 (socket says 172.21.15.28:44362) 2024-02-21T23:12:30.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.598+0000 7fe019e00640 1 -- 172.21.15.28:0/3703856557 learned_addr learned my addr 172.21.15.28:0/3703856557 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:30.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.599+0000 7fe01a800640 1 --2- 172.21.15.28:0/3703856557 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe01c0c7c30 0x7fe01c14c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:30.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.599+0000 7fe01b200640 1 --2- 172.21.15.28:0/3703856557 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe01c1559c0 0x7fe01c1472c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:30.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.599+0000 7fe019e00640 1 -- 172.21.15.28:0/3703856557 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe01c0c7c30 msgr2=0x7fe01c14c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:30.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.599+0000 7fe019e00640 1 --2- 172.21.15.28:0/3703856557 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe01c0c7c30 0x7fe01c14c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:30.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.599+0000 7fe019e00640 1 -- 172.21.15.28:0/3703856557 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe01c1559c0 msgr2=0x7fe01c1472c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:30.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.599+0000 7fe019e00640 1 --2- 172.21.15.28:0/3703856557 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe01c1559c0 0x7fe01c1472c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:30.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.599+0000 7fe019e00640 1 -- 172.21.15.28:0/3703856557 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe01c1eaaf0 con 0x7fe01c1526b0 2024-02-21T23:12:30.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.599+0000 7fe019e00640 1 --2- 172.21.15.28:0/3703856557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe01c1526b0 0x7fe01c146d80 secure :-1 s=READY pgs=2245 cs=0 l=1 rev1=1 crypto rx=0x7fe008000c00 tx=0x7fe0080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:30.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.600+0000 7fe012a00640 1 -- 172.21.15.28:0/3703856557 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe008040070 con 0x7fe01c1526b0 2024-02-21T23:12:30.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.600+0000 7fe012a00640 1 -- 172.21.15.28:0/3703856557 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe008039940 con 0x7fe01c1526b0 2024-02-21T23:12:30.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.600+0000 7fe012a00640 1 -- 172.21.15.28:0/3703856557 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe008035040 con 0x7fe01c1526b0 2024-02-21T23:12:30.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.600+0000 7fe021400640 1 -- 172.21.15.28:0/3703856557 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe01c1eadd0 con 0x7fe01c1526b0 2024-02-21T23:12:30.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.600+0000 7fe021400640 1 -- 172.21.15.28:0/3703856557 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe01c147800 con 0x7fe01c1526b0 2024-02-21T23:12:30.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.601+0000 7fe021400640 1 -- 172.21.15.28:0/3703856557 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe01c1557e0 con 0x7fe01c1526b0 2024-02-21T23:12:30.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.602+0000 7fe012a00640 1 -- 172.21.15.28:0/3703856557 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe00804b060 con 0x7fe01c1526b0 2024-02-21T23:12:30.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.603+0000 7fe012a00640 1 --2- 172.21.15.28:0/3703856557 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdffc06b4d0 0x7fdffc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:30.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.603+0000 7fe012a00640 1 -- 172.21.15.28:0/3703856557 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe0080abcf0 con 0x7fe01c1526b0 2024-02-21T23:12:30.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.603+0000 7fe01a800640 1 --2- 172.21.15.28:0/3703856557 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdffc06b4d0 0x7fdffc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:30.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.605+0000 7fe01a800640 1 --2- 172.21.15.28:0/3703856557 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdffc06b4d0 0x7fdffc06d990 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fe00c000f60 tx=0x7fe00c007b50 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:30.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.607+0000 7fe012a00640 1 -- 172.21.15.28:0/3703856557 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe01c1557e0 con 0x7fe01c1526b0 2024-02-21T23:12:30.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.808+0000 7fe021400640 1 -- 172.21.15.28:0/3703856557 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe01c067f30 con 0x7fe01c1526b0 2024-02-21T23:12:30.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.811+0000 7fe012a00640 1 -- 172.21.15.28:0/3703856557 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v197) v1 ==== 74+0+16892 (secure 0 0 0) 0x7fe00807a9d0 con 0x7fe01c1526b0 2024-02-21T23:12:30.800 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:12:30.801 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":197,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:12:27.821567+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:12:24.201589+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"196","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2024-02-21T23:12:25.295387+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"197","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:12:30.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.814+0000 7fe021400640 1 -- 172.21.15.28:0/3703856557 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdffc06b4d0 msgr2=0x7fdffc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:30.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.814+0000 7fe021400640 1 --2- 172.21.15.28:0/3703856557 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdffc06b4d0 0x7fdffc06d990 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fe00c000f60 tx=0x7fe00c007b50 comp rx=0 tx=0).stop 2024-02-21T23:12:30.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.814+0000 7fe021400640 1 -- 172.21.15.28:0/3703856557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe01c1526b0 msgr2=0x7fe01c146d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:30.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.814+0000 7fe021400640 1 --2- 172.21.15.28:0/3703856557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe01c1526b0 0x7fe01c146d80 secure :-1 s=READY pgs=2245 cs=0 l=1 rev1=1 crypto rx=0x7fe008000c00 tx=0x7fe0080046c0 comp rx=0 tx=0).stop 2024-02-21T23:12:30.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.814+0000 7fe021400640 1 -- 172.21.15.28:0/3703856557 shutdown_connections 2024-02-21T23:12:30.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.814+0000 7fe021400640 1 --2- 172.21.15.28:0/3703856557 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdffc06b4d0 0x7fdffc06d990 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:30.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.814+0000 7fe021400640 1 --2- 172.21.15.28:0/3703856557 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe01c1559c0 0x7fe01c1472c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:30.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.814+0000 7fe021400640 1 --2- 172.21.15.28:0/3703856557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe01c1526b0 0x7fe01c146d80 unknown :-1 s=CLOSED pgs=2245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:30.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.814+0000 7fe021400640 1 --2- 172.21.15.28:0/3703856557 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe01c0c7c30 0x7fe01c14c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:30.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.814+0000 7fe021400640 1 -- 172.21.15.28:0/3703856557 >> 172.21.15.28:0/3703856557 conn(0x7fe01c06d3d0 msgr2=0x7fe01c0c31e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:30.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.814+0000 7fe021400640 1 -- 172.21.15.28:0/3703856557 shutdown_connections 2024-02-21T23:12:30.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:30.814+0000 7fe021400640 1 -- 172.21.15.28:0/3703856557 wait complete. 2024-02-21T23:12:30.821 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-02-21T23:12:30.821 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-21T23:12:31.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.043+0000 7f5e55000640 1 Processor -- start 2024-02-21T23:12:31.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.043+0000 7f5e55000640 1 -- start start 2024-02-21T23:12:31.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.045+0000 7f5e55000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f5e50150450 0x7f5e500a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:31.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.045+0000 7f5e55000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5e5006ee40 0x7f5e501536f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:31.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.045+0000 7f5e55000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e500c2760 con 0x7f5e50156af0 2024-02-21T23:12:31.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.045+0000 7f5e55000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e500c22c0 con 0x7f5e50150450 2024-02-21T23:12:31.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.045+0000 7f5e55000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e50156660 con 0x7f5e5006ee40 2024-02-21T23:12:31.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.045+0000 7f5e4f400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f5e50150450 0x7f5e500a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:31.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.046+0000 7f5e4f400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f5e50150450 0x7f5e500a5fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.28:35558/0 (socket says 172.21.15.28:35558) 2024-02-21T23:12:31.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.046+0000 7f5e4f400640 1 -- 172.21.15.28:0/2817008095 learned_addr learned my addr 172.21.15.28:0/2817008095 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:31.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.046+0000 7f5e4ea00640 1 --2- 172.21.15.28:0/2817008095 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5e5006ee40 0x7f5e501536f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:31.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.046+0000 7f5e4f400640 1 -- 172.21.15.28:0/2817008095 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5e5006ee40 msgr2=0x7f5e501536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:31.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.046+0000 7f5e4f400640 1 --2- 172.21.15.28:0/2817008095 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5e5006ee40 0x7f5e501536f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:31.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.046+0000 7f5e4f400640 1 -- 172.21.15.28:0/2817008095 >> v1:172.21.15.81:6789/0 conn(0x7f5e50156af0 legacy=0x7f5e5006dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:31.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.046+0000 7f5e4f400640 1 -- 172.21.15.28:0/2817008095 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e50153c30 con 0x7f5e50150450 2024-02-21T23:12:31.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.048+0000 7f5e4f400640 1 --2- 172.21.15.28:0/2817008095 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e50150450 0x7f5e500a5fd0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f5e40009870 tx=0x7f5e4002b7e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e96d7e0081246f38 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:31.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.048+0000 7f5e4e000640 1 -- 172.21.15.28:0/2817008095 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e40037070 con 0x7f5e50150450 2024-02-21T23:12:31.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.048+0000 7f5e4e000640 1 -- 172.21.15.28:0/2817008095 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e40004230 con 0x7f5e50150450 2024-02-21T23:12:31.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.048+0000 7f5e4e000640 1 -- 172.21.15.28:0/2817008095 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e40003610 con 0x7f5e50150450 2024-02-21T23:12:31.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.049+0000 7f5e55000640 1 -- 172.21.15.28:0/2817008095 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e50150450 msgr2=0x7f5e500a5fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:31.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.049+0000 7f5e55000640 1 --2- 172.21.15.28:0/2817008095 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e50150450 0x7f5e500a5fd0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f5e40009870 tx=0x7f5e4002b7e0 comp rx=0 tx=0).stop 2024-02-21T23:12:31.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.049+0000 7f5e55000640 1 -- 172.21.15.28:0/2817008095 shutdown_connections 2024-02-21T23:12:31.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.049+0000 7f5e55000640 1 --2- 172.21.15.28:0/2817008095 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5e5006ee40 0x7f5e501536f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:31.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.049+0000 7f5e55000640 1 --2- 172.21.15.28:0/2817008095 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e50150450 0x7f5e500a5fd0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:31.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.049+0000 7f5e55000640 1 -- 172.21.15.28:0/2817008095 >> 172.21.15.28:0/2817008095 conn(0x7f5e5006b630 msgr2=0x7f5e500c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:31.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.049+0000 7f5e55000640 1 -- 172.21.15.28:0/2817008095 shutdown_connections 2024-02-21T23:12:31.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.049+0000 7f5e55000640 1 -- 172.21.15.28:0/2817008095 wait complete. 2024-02-21T23:12:31.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.051+0000 7f5e55000640 1 Processor -- start 2024-02-21T23:12:31.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.051+0000 7f5e55000640 1 -- start start 2024-02-21T23:12:31.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.052+0000 7f5e55000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e5006ee40 0x7f5e5014b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:31.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.052+0000 7f5e55000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e50150450 0x7f5e50154510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:31.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.052+0000 7f5e55000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5e50156af0 0x7f5e501552d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:31.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.052+0000 7f5e55000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e500c2cb0 con 0x7f5e5006ee40 2024-02-21T23:12:31.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.052+0000 7f5e55000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e500c29d0 con 0x7f5e50150450 2024-02-21T23:12:31.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.052+0000 7f5e55000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5006e700 con 0x7f5e50156af0 2024-02-21T23:12:31.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.052+0000 7f5e4f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e5006ee40 0x7f5e5014b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:31.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.052+0000 7f5e4f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e5006ee40 0x7f5e5014b720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44366/0 (socket says 172.21.15.28:44366) 2024-02-21T23:12:31.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.052+0000 7f5e4f400640 1 -- 172.21.15.28:0/3682633471 learned_addr learned my addr 172.21.15.28:0/3682633471 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:31.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.053+0000 7f5e4ea00640 1 --2- 172.21.15.28:0/3682633471 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e50150450 0x7f5e50154510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:31.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.053+0000 7f5e4fe00640 1 --2- 172.21.15.28:0/3682633471 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5e50156af0 0x7f5e501552d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:31.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.053+0000 7f5e4f400640 1 -- 172.21.15.28:0/3682633471 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5e50156af0 msgr2=0x7f5e501552d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:31.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.053+0000 7f5e4f400640 1 --2- 172.21.15.28:0/3682633471 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5e50156af0 0x7f5e501552d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:31.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.053+0000 7f5e4f400640 1 -- 172.21.15.28:0/3682633471 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e50150450 msgr2=0x7f5e50154510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:31.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.053+0000 7f5e4f400640 1 --2- 172.21.15.28:0/3682633471 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e50150450 0x7f5e50154510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:31.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.053+0000 7f5e4f400640 1 -- 172.21.15.28:0/3682633471 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e5014b010 con 0x7f5e5006ee40 2024-02-21T23:12:31.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.053+0000 7f5e4f400640 1 --2- 172.21.15.28:0/3682633471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e5006ee40 0x7f5e5014b720 secure :-1 s=READY pgs=2246 cs=0 l=1 rev1=1 crypto rx=0x7f5e400097c0 tx=0x7f5e400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:31.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.054+0000 7f5e47400640 1 -- 172.21.15.28:0/3682633471 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e40042020 con 0x7f5e5006ee40 2024-02-21T23:12:31.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.054+0000 7f5e47400640 1 -- 172.21.15.28:0/3682633471 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e40030510 con 0x7f5e5006ee40 2024-02-21T23:12:31.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.054+0000 7f5e47400640 1 -- 172.21.15.28:0/3682633471 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e40037040 con 0x7f5e5006ee40 2024-02-21T23:12:31.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.054+0000 7f5e55000640 1 -- 172.21.15.28:0/3682633471 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e5014ad30 con 0x7f5e5006ee40 2024-02-21T23:12:31.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.054+0000 7f5e55000640 1 -- 172.21.15.28:0/3682633471 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e501ebab0 con 0x7f5e5006ee40 2024-02-21T23:12:31.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.055+0000 7f5e55000640 1 -- 172.21.15.28:0/3682633471 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e5004f290 con 0x7f5e5006ee40 2024-02-21T23:12:31.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.056+0000 7f5e47400640 1 -- 172.21.15.28:0/3682633471 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5e40045020 con 0x7f5e5006ee40 2024-02-21T23:12:31.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.057+0000 7f5e47400640 1 --2- 172.21.15.28:0/3682633471 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5e3006b4d0 0x7f5e3006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:31.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.057+0000 7f5e47400640 1 -- 172.21.15.28:0/3682633471 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5e400abf00 con 0x7f5e5006ee40 2024-02-21T23:12:31.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.058+0000 7f5e4ea00640 1 --2- 172.21.15.28:0/3682633471 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5e3006b4d0 0x7f5e3006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:31.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.059+0000 7f5e4ea00640 1 --2- 172.21.15.28:0/3682633471 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5e3006b4d0 0x7f5e3006d990 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f5e501544a0 tx=0x7f5e3c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:31.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.062+0000 7f5e47400640 1 -- 172.21.15.28:0/3682633471 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f5e4004c030 con 0x7f5e5006ee40 2024-02-21T23:12:31.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.262+0000 7f5e55000640 1 -- 172.21.15.28:0/3682633471 --> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5e5006be20 con 0x7f5e3006b4d0 2024-02-21T23:12:31.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.281+0000 7f5e47400640 1 -- 172.21.15.28:0/3682633471 <== mgr.4101 v2:172.21.15.81:6832/1848538577 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252566 (secure 0 0 0) 0x7f5e5006be20 con 0x7f5e3006b4d0 2024-02-21T23:12:31.274 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:12:31.276 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-21T23:12:31.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.287+0000 7f5e55000640 1 -- 172.21.15.28:0/3682633471 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5e3006b4d0 msgr2=0x7f5e3006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:31.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.287+0000 7f5e55000640 1 --2- 172.21.15.28:0/3682633471 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5e3006b4d0 0x7f5e3006d990 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f5e501544a0 tx=0x7f5e3c008040 comp rx=0 tx=0).stop 2024-02-21T23:12:31.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.287+0000 7f5e55000640 1 -- 172.21.15.28:0/3682633471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e5006ee40 msgr2=0x7f5e5014b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:31.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.287+0000 7f5e55000640 1 --2- 172.21.15.28:0/3682633471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e5006ee40 0x7f5e5014b720 secure :-1 s=READY pgs=2246 cs=0 l=1 rev1=1 crypto rx=0x7f5e400097c0 tx=0x7f5e400046c0 comp rx=0 tx=0).stop 2024-02-21T23:12:31.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.287+0000 7f5e55000640 1 -- 172.21.15.28:0/3682633471 shutdown_connections 2024-02-21T23:12:31.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.287+0000 7f5e55000640 1 --2- 172.21.15.28:0/3682633471 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5e3006b4d0 0x7f5e3006d990 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:31.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.287+0000 7f5e55000640 1 --2- 172.21.15.28:0/3682633471 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5e50156af0 0x7f5e501552d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:31.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.287+0000 7f5e55000640 1 --2- 172.21.15.28:0/3682633471 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e50150450 0x7f5e50154510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:31.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.287+0000 7f5e55000640 1 --2- 172.21.15.28:0/3682633471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e5006ee40 0x7f5e5014b720 unknown :-1 s=CLOSED pgs=2246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:31.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.287+0000 7f5e55000640 1 -- 172.21.15.28:0/3682633471 >> 172.21.15.28:0/3682633471 conn(0x7f5e5006b630 msgr2=0x7f5e5006dff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:31.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.288+0000 7f5e55000640 1 -- 172.21.15.28:0/3682633471 shutdown_connections 2024-02-21T23:12:31.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.288+0000 7f5e55000640 1 -- 172.21.15.28:0/3682633471 wait complete. 2024-02-21T23:12:31.296 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":490,"stamp":"2024-02-21T23:12:30.876748+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":173,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":277492,"kb_used_data":19712,"kb_used_omap":12,"kb_used_meta":257651,"kb_avail":754697228,"statfs":{"total":773094113280,"available":772809961472,"internally_reserved":0,"allocated":20185088,"data_stored":11002620,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":263835185},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001856"},"pg_stats":[{"pgid":"32.1f","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:29.380264+0000","last_change":"2024-02-21T23:12:26.848756+0000","last_active":"2024-02-21T23:12:29.380264+0000","last_peered":"2024-02-21T23:12:29.380264+0000","last_clean":"2024-02-21T23:12:29.380264+0000","last_became_active":"2024-02-21T23:12:26.848437+0000","last_became_peered":"2024-02-21T23:12:26.848437+0000","last_unstale":"2024-02-21T23:12:29.380264+0000","last_undegraded":"2024-02-21T23:12:29.380264+0000","last_fullsized":"2024-02-21T23:12:29.380264+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:52:23.398556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.1e","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.836933+0000","last_change":"2024-02-21T23:12:26.847858+0000","last_active":"2024-02-21T23:12:27.836933+0000","last_peered":"2024-02-21T23:12:27.836933+0000","last_clean":"2024-02-21T23:12:27.836933+0000","last_became_active":"2024-02-21T23:12:26.845014+0000","last_became_peered":"2024-02-21T23:12:26.845014+0000","last_unstale":"2024-02-21T23:12:27.836933+0000","last_undegraded":"2024-02-21T23:12:27.836933+0000","last_fullsized":"2024-02-21T23:12:27.836933+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:21:52.739416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1d","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:28.278913+0000","last_change":"2024-02-21T23:12:26.830438+0000","last_active":"2024-02-21T23:12:28.278913+0000","last_peered":"2024-02-21T23:12:28.278913+0000","last_clean":"2024-02-21T23:12:28.278913+0000","last_became_active":"2024-02-21T23:12:26.829573+0000","last_became_peered":"2024-02-21T23:12:26.829573+0000","last_unstale":"2024-02-21T23:12:28.278913+0000","last_undegraded":"2024-02-21T23:12:28.278913+0000","last_fullsized":"2024-02-21T23:12:28.278913+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:20:49.569662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1c","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:28.278745+0000","last_change":"2024-02-21T23:12:26.835807+0000","last_active":"2024-02-21T23:12:28.278745+0000","last_peered":"2024-02-21T23:12:28.278745+0000","last_clean":"2024-02-21T23:12:28.278745+0000","last_became_active":"2024-02-21T23:12:26.835526+0000","last_became_peered":"2024-02-21T23:12:26.835526+0000","last_unstale":"2024-02-21T23:12:28.278745+0000","last_undegraded":"2024-02-21T23:12:28.278745+0000","last_fullsized":"2024-02-21T23:12:28.278745+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:39:57.281364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1b","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:28.068868+0000","last_change":"2024-02-21T23:12:26.842846+0000","last_active":"2024-02-21T23:12:28.068868+0000","last_peered":"2024-02-21T23:12:28.068868+0000","last_clean":"2024-02-21T23:12:28.068868+0000","last_became_active":"2024-02-21T23:12:26.842493+0000","last_became_peered":"2024-02-21T23:12:26.842493+0000","last_unstale":"2024-02-21T23:12:28.068868+0000","last_undegraded":"2024-02-21T23:12:28.068868+0000","last_fullsized":"2024-02-21T23:12:28.068868+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:58:18.463148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1a","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:29.380155+0000","last_change":"2024-02-21T23:12:26.844005+0000","last_active":"2024-02-21T23:12:29.380155+0000","last_peered":"2024-02-21T23:12:29.380155+0000","last_clean":"2024-02-21T23:12:29.380155+0000","last_became_active":"2024-02-21T23:12:26.843735+0000","last_became_peered":"2024-02-21T23:12:26.843735+0000","last_unstale":"2024-02-21T23:12:29.380155+0000","last_undegraded":"2024-02-21T23:12:29.380155+0000","last_fullsized":"2024-02-21T23:12:29.380155+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:15:27.708023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.838468+0000","last_change":"2024-02-21T23:12:26.827683+0000","last_active":"2024-02-21T23:12:27.838468+0000","last_peered":"2024-02-21T23:12:27.838468+0000","last_clean":"2024-02-21T23:12:27.838468+0000","last_became_active":"2024-02-21T23:12:26.827414+0000","last_became_peered":"2024-02-21T23:12:26.827414+0000","last_unstale":"2024-02-21T23:12:27.838468+0000","last_undegraded":"2024-02-21T23:12:27.838468+0000","last_fullsized":"2024-02-21T23:12:27.838468+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:19:48.049086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:28.068438+0000","last_change":"2024-02-21T23:12:26.842685+0000","last_active":"2024-02-21T23:12:28.068438+0000","last_peered":"2024-02-21T23:12:28.068438+0000","last_clean":"2024-02-21T23:12:28.068438+0000","last_became_active":"2024-02-21T23:12:26.841736+0000","last_became_peered":"2024-02-21T23:12:26.841736+0000","last_unstale":"2024-02-21T23:12:28.068438+0000","last_undegraded":"2024-02-21T23:12:28.068438+0000","last_fullsized":"2024-02-21T23:12:28.068438+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:49:13.089888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:29.380368+0000","last_change":"2024-02-21T23:12:26.849708+0000","last_active":"2024-02-21T23:12:29.380368+0000","last_peered":"2024-02-21T23:12:29.380368+0000","last_clean":"2024-02-21T23:12:29.380368+0000","last_became_active":"2024-02-21T23:12:26.848070+0000","last_became_peered":"2024-02-21T23:12:26.848070+0000","last_unstale":"2024-02-21T23:12:29.380368+0000","last_undegraded":"2024-02-21T23:12:29.380368+0000","last_fullsized":"2024-02-21T23:12:29.380368+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:56:36.503491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:28.279112+0000","last_change":"2024-02-21T23:12:26.835797+0000","last_active":"2024-02-21T23:12:28.279112+0000","last_peered":"2024-02-21T23:12:28.279112+0000","last_clean":"2024-02-21T23:12:28.279112+0000","last_became_active":"2024-02-21T23:12:26.835006+0000","last_became_peered":"2024-02-21T23:12:26.835006+0000","last_unstale":"2024-02-21T23:12:28.279112+0000","last_undegraded":"2024-02-21T23:12:28.279112+0000","last_fullsized":"2024-02-21T23:12:28.279112+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:33:47.725072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.15","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:28.279003+0000","last_change":"2024-02-21T23:12:26.835197+0000","last_active":"2024-02-21T23:12:28.279003+0000","last_peered":"2024-02-21T23:12:28.279003+0000","last_clean":"2024-02-21T23:12:28.279003+0000","last_became_active":"2024-02-21T23:12:26.834887+0000","last_became_peered":"2024-02-21T23:12:26.834887+0000","last_unstale":"2024-02-21T23:12:28.279003+0000","last_undegraded":"2024-02-21T23:12:28.279003+0000","last_fullsized":"2024-02-21T23:12:28.279003+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:21:43.387305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.14","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:29.380081+0000","last_change":"2024-02-21T23:12:26.853250+0000","last_active":"2024-02-21T23:12:29.380081+0000","last_peered":"2024-02-21T23:12:29.380081+0000","last_clean":"2024-02-21T23:12:29.380081+0000","last_became_active":"2024-02-21T23:12:26.847779+0000","last_became_peered":"2024-02-21T23:12:26.847779+0000","last_unstale":"2024-02-21T23:12:29.380081+0000","last_undegraded":"2024-02-21T23:12:29.380081+0000","last_fullsized":"2024-02-21T23:12:29.380081+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:41:45.251159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.13","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:29.380736+0000","last_change":"2024-02-21T23:12:26.838145+0000","last_active":"2024-02-21T23:12:29.380736+0000","last_peered":"2024-02-21T23:12:29.380736+0000","last_clean":"2024-02-21T23:12:29.380736+0000","last_became_active":"2024-02-21T23:12:26.834449+0000","last_became_peered":"2024-02-21T23:12:26.834449+0000","last_unstale":"2024-02-21T23:12:29.380736+0000","last_undegraded":"2024-02-21T23:12:29.380736+0000","last_fullsized":"2024-02-21T23:12:29.380736+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:16:52.301170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.948792+0000","last_change":"2024-02-21T23:12:26.843019+0000","last_active":"2024-02-21T23:12:27.948792+0000","last_peered":"2024-02-21T23:12:27.948792+0000","last_clean":"2024-02-21T23:12:27.948792+0000","last_became_active":"2024-02-21T23:12:26.842707+0000","last_became_peered":"2024-02-21T23:12:26.842707+0000","last_unstale":"2024-02-21T23:12:27.948792+0000","last_undegraded":"2024-02-21T23:12:27.948792+0000","last_fullsized":"2024-02-21T23:12:27.948792+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:53:34.566060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.840844+0000","last_change":"2024-02-21T23:12:26.844735+0000","last_active":"2024-02-21T23:12:27.840844+0000","last_peered":"2024-02-21T23:12:27.840844+0000","last_clean":"2024-02-21T23:12:27.840844+0000","last_became_active":"2024-02-21T23:12:26.844148+0000","last_became_peered":"2024-02-21T23:12:26.844148+0000","last_unstale":"2024-02-21T23:12:27.840844+0000","last_undegraded":"2024-02-21T23:12:27.840844+0000","last_fullsized":"2024-02-21T23:12:27.840844+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:20:44.472226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.10","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:28.278425+0000","last_change":"2024-02-21T23:12:26.828679+0000","last_active":"2024-02-21T23:12:28.278425+0000","last_peered":"2024-02-21T23:12:28.278425+0000","last_clean":"2024-02-21T23:12:28.278425+0000","last_became_active":"2024-02-21T23:12:26.828306+0000","last_became_peered":"2024-02-21T23:12:26.828306+0000","last_unstale":"2024-02-21T23:12:28.278425+0000","last_undegraded":"2024-02-21T23:12:28.278425+0000","last_fullsized":"2024-02-21T23:12:28.278425+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:42:34.425253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.f","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:29.381318+0000","last_change":"2024-02-21T23:12:26.832308+0000","last_active":"2024-02-21T23:12:29.381318+0000","last_peered":"2024-02-21T23:12:29.381318+0000","last_clean":"2024-02-21T23:12:29.381318+0000","last_became_active":"2024-02-21T23:12:26.831963+0000","last_became_peered":"2024-02-21T23:12:26.831963+0000","last_unstale":"2024-02-21T23:12:29.381318+0000","last_undegraded":"2024-02-21T23:12:29.381318+0000","last_fullsized":"2024-02-21T23:12:29.381318+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:54:11.876493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.e","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:28.279357+0000","last_change":"2024-02-21T23:12:26.837244+0000","last_active":"2024-02-21T23:12:28.279357+0000","last_peered":"2024-02-21T23:12:28.279357+0000","last_clean":"2024-02-21T23:12:28.279357+0000","last_became_active":"2024-02-21T23:12:26.836959+0000","last_became_peered":"2024-02-21T23:12:26.836959+0000","last_unstale":"2024-02-21T23:12:28.279357+0000","last_undegraded":"2024-02-21T23:12:28.279357+0000","last_fullsized":"2024-02-21T23:12:28.279357+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:51:59.978445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.d","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.841005+0000","last_change":"2024-02-21T23:12:26.836887+0000","last_active":"2024-02-21T23:12:27.841005+0000","last_peered":"2024-02-21T23:12:27.841005+0000","last_clean":"2024-02-21T23:12:27.841005+0000","last_became_active":"2024-02-21T23:12:26.836600+0000","last_became_peered":"2024-02-21T23:12:26.836600+0000","last_unstale":"2024-02-21T23:12:27.841005+0000","last_undegraded":"2024-02-21T23:12:27.841005+0000","last_fullsized":"2024-02-21T23:12:27.841005+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:26:12.656815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.836974+0000","last_change":"2024-02-21T23:12:26.825571+0000","last_active":"2024-02-21T23:12:27.836974+0000","last_peered":"2024-02-21T23:12:27.836974+0000","last_clean":"2024-02-21T23:12:27.836974+0000","last_became_active":"2024-02-21T23:12:26.824298+0000","last_became_peered":"2024-02-21T23:12:26.824298+0000","last_unstale":"2024-02-21T23:12:27.836974+0000","last_undegraded":"2024-02-21T23:12:27.836974+0000","last_fullsized":"2024-02-21T23:12:27.836974+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:25:24.812384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.b","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.838696+0000","last_change":"2024-02-21T23:12:26.836194+0000","last_active":"2024-02-21T23:12:27.838696+0000","last_peered":"2024-02-21T23:12:27.838696+0000","last_clean":"2024-02-21T23:12:27.838696+0000","last_became_active":"2024-02-21T23:12:26.835892+0000","last_became_peered":"2024-02-21T23:12:26.835892+0000","last_unstale":"2024-02-21T23:12:27.838696+0000","last_undegraded":"2024-02-21T23:12:27.838696+0000","last_fullsized":"2024-02-21T23:12:27.838696+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:48:54.474119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:29.380396+0000","last_change":"2024-02-21T23:12:26.852351+0000","last_active":"2024-02-21T23:12:29.380396+0000","last_peered":"2024-02-21T23:12:29.380396+0000","last_clean":"2024-02-21T23:12:29.380396+0000","last_became_active":"2024-02-21T23:12:26.852005+0000","last_became_peered":"2024-02-21T23:12:26.852005+0000","last_unstale":"2024-02-21T23:12:29.380396+0000","last_undegraded":"2024-02-21T23:12:29.380396+0000","last_fullsized":"2024-02-21T23:12:29.380396+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:05:33.247621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.841142+0000","last_change":"2024-02-21T23:12:26.839425+0000","last_active":"2024-02-21T23:12:27.841142+0000","last_peered":"2024-02-21T23:12:27.841142+0000","last_clean":"2024-02-21T23:12:27.841142+0000","last_became_active":"2024-02-21T23:12:26.839063+0000","last_became_peered":"2024-02-21T23:12:26.839063+0000","last_unstale":"2024-02-21T23:12:27.841142+0000","last_undegraded":"2024-02-21T23:12:27.841142+0000","last_fullsized":"2024-02-21T23:12:27.841142+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:30:20.232383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:29.380133+0000","last_change":"2024-02-21T23:12:26.835513+0000","last_active":"2024-02-21T23:12:29.380133+0000","last_peered":"2024-02-21T23:12:29.380133+0000","last_clean":"2024-02-21T23:12:29.380133+0000","last_became_active":"2024-02-21T23:12:26.835244+0000","last_became_peered":"2024-02-21T23:12:26.835244+0000","last_unstale":"2024-02-21T23:12:29.380133+0000","last_undegraded":"2024-02-21T23:12:29.380133+0000","last_fullsized":"2024-02-21T23:12:29.380133+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:51:31.617475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.7","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.837499+0000","last_change":"2024-02-21T23:12:26.830592+0000","last_active":"2024-02-21T23:12:27.837499+0000","last_peered":"2024-02-21T23:12:27.837499+0000","last_clean":"2024-02-21T23:12:27.837499+0000","last_became_active":"2024-02-21T23:12:26.830315+0000","last_became_peered":"2024-02-21T23:12:26.830315+0000","last_unstale":"2024-02-21T23:12:27.837499+0000","last_undegraded":"2024-02-21T23:12:27.837499+0000","last_fullsized":"2024-02-21T23:12:27.837499+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:49:50.412580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.839493+0000","last_change":"2024-02-21T23:12:26.839564+0000","last_active":"2024-02-21T23:12:27.839493+0000","last_peered":"2024-02-21T23:12:27.839493+0000","last_clean":"2024-02-21T23:12:27.839493+0000","last_became_active":"2024-02-21T23:12:26.839297+0000","last_became_peered":"2024-02-21T23:12:26.839297+0000","last_unstale":"2024-02-21T23:12:27.839493+0000","last_undegraded":"2024-02-21T23:12:27.839493+0000","last_fullsized":"2024-02-21T23:12:27.839493+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:30:20.232383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:28.069100+0000","last_change":"2024-02-21T23:12:26.825911+0000","last_active":"2024-02-21T23:12:28.069100+0000","last_peered":"2024-02-21T23:12:28.069100+0000","last_clean":"2024-02-21T23:12:28.069100+0000","last_became_active":"2024-02-21T23:12:26.825586+0000","last_became_peered":"2024-02-21T23:12:26.825586+0000","last_unstale":"2024-02-21T23:12:28.069100+0000","last_undegraded":"2024-02-21T23:12:28.069100+0000","last_fullsized":"2024-02-21T23:12:28.069100+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:08:54.861131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:29.380151+0000","last_change":"2024-02-21T23:12:26.855217+0000","last_active":"2024-02-21T23:12:29.380151+0000","last_peered":"2024-02-21T23:12:29.380151+0000","last_clean":"2024-02-21T23:12:29.380151+0000","last_became_active":"2024-02-21T23:12:26.853599+0000","last_became_peered":"2024-02-21T23:12:26.853599+0000","last_unstale":"2024-02-21T23:12:29.380151+0000","last_undegraded":"2024-02-21T23:12:29.380151+0000","last_fullsized":"2024-02-21T23:12:29.380151+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:22:02.553536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.948401+0000","last_change":"2024-02-21T23:12:26.844231+0000","last_active":"2024-02-21T23:12:27.948401+0000","last_peered":"2024-02-21T23:12:27.948401+0000","last_clean":"2024-02-21T23:12:27.948401+0000","last_became_active":"2024-02-21T23:12:26.842514+0000","last_became_peered":"2024-02-21T23:12:26.842514+0000","last_unstale":"2024-02-21T23:12:27.948401+0000","last_undegraded":"2024-02-21T23:12:27.948401+0000","last_fullsized":"2024-02-21T23:12:27.948401+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:33:47.725072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.948281+0000","last_change":"2024-02-21T23:12:26.834209+0000","last_active":"2024-02-21T23:12:27.948281+0000","last_peered":"2024-02-21T23:12:27.948281+0000","last_clean":"2024-02-21T23:12:27.948281+0000","last_became_active":"2024-02-21T23:12:26.833908+0000","last_became_peered":"2024-02-21T23:12:26.833908+0000","last_unstale":"2024-02-21T23:12:27.948281+0000","last_undegraded":"2024-02-21T23:12:27.948281+0000","last_fullsized":"2024-02-21T23:12:27.948281+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:18:39.370028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.836234+0000","last_change":"2024-02-21T23:12:26.839500+0000","last_active":"2024-02-21T23:12:27.836234+0000","last_peered":"2024-02-21T23:12:27.836234+0000","last_clean":"2024-02-21T23:12:27.836234+0000","last_became_active":"2024-02-21T23:12:26.838973+0000","last_became_peered":"2024-02-21T23:12:26.838973+0000","last_unstale":"2024-02-21T23:12:27.836234+0000","last_undegraded":"2024-02-21T23:12:27.836234+0000","last_fullsized":"2024-02-21T23:12:27.836234+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:53:10.865222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:29.379898+0000","last_change":"2024-02-21T23:12:26.838842+0000","last_active":"2024-02-21T23:12:29.379898+0000","last_peered":"2024-02-21T23:12:29.379898+0000","last_clean":"2024-02-21T23:12:29.379898+0000","last_became_active":"2024-02-21T23:12:26.838533+0000","last_became_peered":"2024-02-21T23:12:26.838533+0000","last_unstale":"2024-02-21T23:12:29.379898+0000","last_undegraded":"2024-02-21T23:12:29.379898+0000","last_fullsized":"2024-02-21T23:12:29.379898+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:42:28.345683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.20","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.836522+0000","last_change":"2024-02-21T23:12:26.847313+0000","last_active":"2024-02-21T23:12:27.836522+0000","last_peered":"2024-02-21T23:12:27.836522+0000","last_clean":"2024-02-21T23:12:27.836522+0000","last_became_active":"2024-02-21T23:12:26.846999+0000","last_became_peered":"2024-02-21T23:12:26.846999+0000","last_unstale":"2024-02-21T23:12:27.836522+0000","last_undegraded":"2024-02-21T23:12:27.836522+0000","last_fullsized":"2024-02-21T23:12:27.836522+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:29:03.598188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1f","version":"196'4","reported_seq":18,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:29.380144+0000","last_change":"2024-02-21T23:12:25.820534+0000","last_active":"2024-02-21T23:12:29.380144+0000","last_peered":"2024-02-21T23:12:29.380144+0000","last_clean":"2024-02-21T23:12:29.380144+0000","last_became_active":"2024-02-21T23:12:25.820259+0000","last_became_peered":"2024-02-21T23:12:25.820259+0000","last_unstale":"2024-02-21T23:12:29.380144+0000","last_undegraded":"2024-02-21T23:12:29.380144+0000","last_fullsized":"2024-02-21T23:12:29.380144+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_clean_scrub_stamp":"2024-02-21T23:12:24.754357+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:29:45.724598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.22","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.840221+0000","last_change":"2024-02-21T23:12:26.838027+0000","last_active":"2024-02-21T23:12:27.840221+0000","last_peered":"2024-02-21T23:12:27.840221+0000","last_clean":"2024-02-21T23:12:27.840221+0000","last_became_active":"2024-02-21T23:12:26.837691+0000","last_became_peered":"2024-02-21T23:12:26.837691+0000","last_unstale":"2024-02-21T23:12:27.840221+0000","last_undegraded":"2024-02-21T23:12:27.840221+0000","last_fullsized":"2024-02-21T23:12:27.840221+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:48:42.776013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.1d","version":"196'2","reported_seq":16,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:29.380638+0000","last_change":"2024-02-21T23:12:25.811795+0000","last_active":"2024-02-21T23:12:29.380638+0000","last_peered":"2024-02-21T23:12:29.380638+0000","last_clean":"2024-02-21T23:12:29.380638+0000","last_became_active":"2024-02-21T23:12:25.811505+0000","last_became_peered":"2024-02-21T23:12:25.811505+0000","last_unstale":"2024-02-21T23:12:29.380638+0000","last_undegraded":"2024-02-21T23:12:29.380638+0000","last_fullsized":"2024-02-21T23:12:29.380638+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_clean_scrub_stamp":"2024-02-21T23:12:24.754357+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:22:45.012364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.23","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:29.380532+0000","last_change":"2024-02-21T23:12:26.847364+0000","last_active":"2024-02-21T23:12:29.380532+0000","last_peered":"2024-02-21T23:12:29.380532+0000","last_clean":"2024-02-21T23:12:29.380532+0000","last_became_active":"2024-02-21T23:12:26.846720+0000","last_became_peered":"2024-02-21T23:12:26.846720+0000","last_unstale":"2024-02-21T23:12:29.380532+0000","last_undegraded":"2024-02-21T23:12:29.380532+0000","last_fullsized":"2024-02-21T23:12:29.380532+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:08:55.825903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1c","version":"196'2","reported_seq":16,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.949031+0000","last_change":"2024-02-21T23:12:25.811712+0000","last_active":"2024-02-21T23:12:27.949031+0000","last_peered":"2024-02-21T23:12:27.949031+0000","last_clean":"2024-02-21T23:12:27.949031+0000","last_became_active":"2024-02-21T23:12:25.811286+0000","last_became_peered":"2024-02-21T23:12:25.811286+0000","last_unstale":"2024-02-21T23:12:27.949031+0000","last_undegraded":"2024-02-21T23:12:27.949031+0000","last_fullsized":"2024-02-21T23:12:27.949031+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_clean_scrub_stamp":"2024-02-21T23:12:24.754357+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:42:33.409039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.24","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.948615+0000","last_change":"2024-02-21T23:12:26.842291+0000","last_active":"2024-02-21T23:12:27.948615+0000","last_peered":"2024-02-21T23:12:27.948615+0000","last_clean":"2024-02-21T23:12:27.948615+0000","last_became_active":"2024-02-21T23:12:26.841973+0000","last_became_peered":"2024-02-21T23:12:26.841973+0000","last_unstale":"2024-02-21T23:12:27.948615+0000","last_undegraded":"2024-02-21T23:12:27.948615+0000","last_fullsized":"2024-02-21T23:12:27.948615+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:23:03.917989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.1b","version":"196'1","reported_seq":15,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.840841+0000","last_change":"2024-02-21T23:12:25.814146+0000","last_active":"2024-02-21T23:12:27.840841+0000","last_peered":"2024-02-21T23:12:27.840841+0000","last_clean":"2024-02-21T23:12:27.840841+0000","last_became_active":"2024-02-21T23:12:25.813819+0000","last_became_peered":"2024-02-21T23:12:25.813819+0000","last_unstale":"2024-02-21T23:12:27.840841+0000","last_undegraded":"2024-02-21T23:12:27.840841+0000","last_fullsized":"2024-02-21T23:12:27.840841+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_clean_scrub_stamp":"2024-02-21T23:12:24.754357+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:22:51.087029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.25","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.839344+0000","last_change":"2024-02-21T23:12:26.840099+0000","last_active":"2024-02-21T23:12:27.839344+0000","last_peered":"2024-02-21T23:12:27.839344+0000","last_clean":"2024-02-21T23:12:27.839344+0000","last_became_active":"2024-02-21T23:12:26.839717+0000","last_became_peered":"2024-02-21T23:12:26.839717+0000","last_unstale":"2024-02-21T23:12:27.839344+0000","last_undegraded":"2024-02-21T23:12:27.839344+0000","last_fullsized":"2024-02-21T23:12:27.839344+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:26:12.656815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.1a","version":"196'1","reported_seq":15,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.948569+0000","last_change":"2024-02-21T23:12:25.809249+0000","last_active":"2024-02-21T23:12:27.948569+0000","last_peered":"2024-02-21T23:12:27.948569+0000","last_clean":"2024-02-21T23:12:27.948569+0000","last_became_active":"2024-02-21T23:12:25.808954+0000","last_became_peered":"2024-02-21T23:12:25.808954+0000","last_unstale":"2024-02-21T23:12:27.948569+0000","last_undegraded":"2024-02-21T23:12:27.948569+0000","last_fullsized":"2024-02-21T23:12:27.948569+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_clean_scrub_stamp":"2024-02-21T23:12:24.754357+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:02:24.973821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.26","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:28.278990+0000","last_change":"2024-02-21T23:12:26.836061+0000","last_active":"2024-02-21T23:12:28.278990+0000","last_peered":"2024-02-21T23:12:28.278990+0000","last_clean":"2024-02-21T23:12:28.278990+0000","last_became_active":"2024-02-21T23:12:26.835427+0000","last_became_peered":"2024-02-21T23:12:26.835427+0000","last_unstale":"2024-02-21T23:12:28.278990+0000","last_undegraded":"2024-02-21T23:12:28.278990+0000","last_fullsized":"2024-02-21T23:12:28.278990+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:16:24.977623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":14,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:28.278252+0000","last_change":"2024-02-21T23:12:25.819100+0000","last_active":"2024-02-21T23:12:28.278252+0000","last_peered":"2024-02-21T23:12:28.278252+0000","last_clean":"2024-02-21T23:12:28.278252+0000","last_became_active":"2024-02-21T23:12:25.818747+0000","last_became_peered":"2024-02-21T23:12:25.818747+0000","last_unstale":"2024-02-21T23:12:28.278252+0000","last_undegraded":"2024-02-21T23:12:28.278252+0000","last_fullsized":"2024-02-21T23:12:28.278252+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_clean_scrub_stamp":"2024-02-21T23:12:24.754357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:18:19.034287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.27","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:28.069628+0000","last_change":"2024-02-21T23:12:26.823356+0000","last_active":"2024-02-21T23:12:28.069628+0000","last_peered":"2024-02-21T23:12:28.069628+0000","last_clean":"2024-02-21T23:12:28.069628+0000","last_became_active":"2024-02-21T23:12:26.823071+0000","last_became_peered":"2024-02-21T23:12:26.823071+0000","last_unstale":"2024-02-21T23:12:28.069628+0000","last_undegraded":"2024-02-21T23:12:28.069628+0000","last_fullsized":"2024-02-21T23:12:28.069628+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:48:42.776013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":14,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:29.379617+0000","last_change":"2024-02-21T23:12:25.818025+0000","last_active":"2024-02-21T23:12:29.379617+0000","last_peered":"2024-02-21T23:12:29.379617+0000","last_clean":"2024-02-21T23:12:29.379617+0000","last_became_active":"2024-02-21T23:12:25.817398+0000","last_became_peered":"2024-02-21T23:12:25.817398+0000","last_unstale":"2024-02-21T23:12:29.379617+0000","last_undegraded":"2024-02-21T23:12:29.379617+0000","last_fullsized":"2024-02-21T23:12:29.379617+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_clean_scrub_stamp":"2024-02-21T23:12:24.754357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:44:47.202214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.28","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.838547+0000","last_change":"2024-02-21T23:12:26.837592+0000","last_active":"2024-02-21T23:12:27.838547+0000","last_peered":"2024-02-21T23:12:27.838547+0000","last_clean":"2024-02-21T23:12:27.838547+0000","last_became_active":"2024-02-21T23:12:26.837292+0000","last_became_peered":"2024-02-21T23:12:26.837292+0000","last_unstale":"2024-02-21T23:12:27.838547+0000","last_undegraded":"2024-02-21T23:12:27.838547+0000","last_fullsized":"2024-02-21T23:12:27.838547+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:48:42.776013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":14,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.949347+0000","last_change":"2024-02-21T23:12:25.819967+0000","last_active":"2024-02-21T23:12:27.949347+0000","last_peered":"2024-02-21T23:12:27.949347+0000","last_clean":"2024-02-21T23:12:27.949347+0000","last_became_active":"2024-02-21T23:12:25.819688+0000","last_became_peered":"2024-02-21T23:12:25.819688+0000","last_unstale":"2024-02-21T23:12:27.949347+0000","last_undegraded":"2024-02-21T23:12:27.949347+0000","last_fullsized":"2024-02-21T23:12:27.949347+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_clean_scrub_stamp":"2024-02-21T23:12:24.754357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:28:22.802030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.29","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:29.380204+0000","last_change":"2024-02-21T23:12:26.847975+0000","last_active":"2024-02-21T23:12:29.380204+0000","last_peered":"2024-02-21T23:12:29.380204+0000","last_clean":"2024-02-21T23:12:29.380204+0000","last_became_active":"2024-02-21T23:12:26.847564+0000","last_became_peered":"2024-02-21T23:12:26.847564+0000","last_unstale":"2024-02-21T23:12:29.380204+0000","last_undegraded":"2024-02-21T23:12:29.380204+0000","last_fullsized":"2024-02-21T23:12:29.380204+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:19:56.325276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":14,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:28.279321+0000","last_change":"2024-02-21T23:12:25.807532+0000","last_active":"2024-02-21T23:12:28.279321+0000","last_peered":"2024-02-21T23:12:28.279321+0000","last_clean":"2024-02-21T23:12:28.279321+0000","last_became_active":"2024-02-21T23:12:25.807238+0000","last_became_peered":"2024-02-21T23:12:25.807238+0000","last_unstale":"2024-02-21T23:12:28.279321+0000","last_undegraded":"2024-02-21T23:12:28.279321+0000","last_fullsized":"2024-02-21T23:12:28.279321+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_clean_scrub_stamp":"2024-02-21T23:12:24.754357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:59:14.985059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.2a","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:28.278971+0000","last_change":"2024-02-21T23:12:26.836694+0000","last_active":"2024-02-21T23:12:28.278971+0000","last_peered":"2024-02-21T23:12:28.278971+0000","last_clean":"2024-02-21T23:12:28.278971+0000","last_became_active":"2024-02-21T23:12:26.836415+0000","last_became_peered":"2024-02-21T23:12:26.836415+0000","last_unstale":"2024-02-21T23:12:28.278971+0000","last_undegraded":"2024-02-21T23:12:28.278971+0000","last_fullsized":"2024-02-21T23:12:28.278971+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:18:39.370028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"196'3","reported_seq":17,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:28.069102+0000","last_change":"2024-02-21T23:12:25.813698+0000","last_active":"2024-02-21T23:12:28.069102+0000","last_peered":"2024-02-21T23:12:28.069102+0000","last_clean":"2024-02-21T23:12:28.069102+0000","last_became_active":"2024-02-21T23:12:25.813397+0000","last_became_peered":"2024-02-21T23:12:25.813397+0000","last_unstale":"2024-02-21T23:12:28.069102+0000","last_undegraded":"2024-02-21T23:12:28.069102+0000","last_fullsized":"2024-02-21T23:12:28.069102+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_clean_scrub_stamp":"2024-02-21T23:12:24.754357+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:16:51.284956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2b","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.838403+0000","last_change":"2024-02-21T23:12:26.851712+0000","last_active":"2024-02-21T23:12:27.838403+0000","last_peered":"2024-02-21T23:12:27.838403+0000","last_clean":"2024-02-21T23:12:27.838403+0000","last_became_active":"2024-02-21T23:12:26.851408+0000","last_became_peered":"2024-02-21T23:12:26.851408+0000","last_unstale":"2024-02-21T23:12:27.838403+0000","last_undegraded":"2024-02-21T23:12:27.838403+0000","last_fullsized":"2024-02-21T23:12:27.838403+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:58:18.463148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.14","version":"196'6","reported_seq":20,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:29.380507+0000","last_change":"2024-02-21T23:12:25.815159+0000","last_active":"2024-02-21T23:12:29.380507+0000","last_peered":"2024-02-21T23:12:29.380507+0000","last_clean":"2024-02-21T23:12:29.380507+0000","last_became_active":"2024-02-21T23:12:25.814055+0000","last_became_peered":"2024-02-21T23:12:25.814055+0000","last_unstale":"2024-02-21T23:12:29.380507+0000","last_undegraded":"2024-02-21T23:12:29.380507+0000","last_fullsized":"2024-02-21T23:12:29.380507+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_clean_scrub_stamp":"2024-02-21T23:12:24.754357+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:47:40.186390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.2c","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.836799+0000","last_change":"2024-02-21T23:12:26.829222+0000","last_active":"2024-02-21T23:12:27.836799+0000","last_peered":"2024-02-21T23:12:27.836799+0000","last_clean":"2024-02-21T23:12:27.836799+0000","last_became_active":"2024-02-21T23:12:26.828860+0000","last_became_peered":"2024-02-21T23:12:26.828860+0000","last_unstale":"2024-02-21T23:12:27.836799+0000","last_undegraded":"2024-02-21T23:12:27.836799+0000","last_fullsized":"2024-02-21T23:12:27.836799+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:21:19.370427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.13","version":"196'2","reported_seq":16,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.841018+0000","last_change":"2024-02-21T23:12:25.817824+0000","last_active":"2024-02-21T23:12:27.841018+0000","last_peered":"2024-02-21T23:12:27.841018+0000","last_clean":"2024-02-21T23:12:27.841018+0000","last_became_active":"2024-02-21T23:12:25.817403+0000","last_became_peered":"2024-02-21T23:12:25.817403+0000","last_unstale":"2024-02-21T23:12:27.841018+0000","last_undegraded":"2024-02-21T23:12:27.841018+0000","last_fullsized":"2024-02-21T23:12:27.841018+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_clean_scrub_stamp":"2024-02-21T23:12:24.754357+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:18:19.034287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.2d","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.839175+0000","last_change":"2024-02-21T23:12:26.833962+0000","last_active":"2024-02-21T23:12:27.839175+0000","last_peered":"2024-02-21T23:12:27.839175+0000","last_clean":"2024-02-21T23:12:27.839175+0000","last_became_active":"2024-02-21T23:12:26.833516+0000","last_became_peered":"2024-02-21T23:12:26.833516+0000","last_unstale":"2024-02-21T23:12:27.839175+0000","last_undegraded":"2024-02-21T23:12:27.839175+0000","last_fullsized":"2024-02-21T23:12:27.839175+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:20:44.472226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":14,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:28.068617+0000","last_change":"2024-02-21T23:12:25.814231+0000","last_active":"2024-02-21T23:12:28.068617+0000","last_peered":"2024-02-21T23:12:28.068617+0000","last_clean":"2024-02-21T23:12:28.068617+0000","last_became_active":"2024-02-21T23:12:25.811379+0000","last_became_peered":"2024-02-21T23:12:25.811379+0000","last_unstale":"2024-02-21T23:12:28.068617+0000","last_undegraded":"2024-02-21T23:12:28.068617+0000","last_fullsized":"2024-02-21T23:12:28.068617+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_clean_scrub_stamp":"2024-02-21T23:12:24.754357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:57:19.951070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2e","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.840739+0000","last_change":"2024-02-21T23:12:26.839311+0000","last_active":"2024-02-21T23:12:27.840739+0000","last_peered":"2024-02-21T23:12:27.840739+0000","last_clean":"2024-02-21T23:12:27.840739+0000","last_became_active":"2024-02-21T23:12:26.839022+0000","last_became_peered":"2024-02-21T23:12:26.839022+0000","last_unstale":"2024-02-21T23:12:27.840739+0000","last_undegraded":"2024-02-21T23:12:27.840739+0000","last_fullsized":"2024-02-21T23:12:27.840739+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:48:54.474119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":14,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:28.068111+0000","last_change":"2024-02-21T23:12:25.807307+0000","last_active":"2024-02-21T23:12:28.068111+0000","last_peered":"2024-02-21T23:12:28.068111+0000","last_clean":"2024-02-21T23:12:28.068111+0000","last_became_active":"2024-02-21T23:12:25.806326+0000","last_became_peered":"2024-02-21T23:12:25.806326+0000","last_unstale":"2024-02-21T23:12:28.068111+0000","last_undegraded":"2024-02-21T23:12:28.068111+0000","last_fullsized":"2024-02-21T23:12:28.068111+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_clean_scrub_stamp":"2024-02-21T23:12:24.754357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:19:55.309062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2f","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:28.069511+0000","last_change":"2024-02-21T23:12:26.830317+0000","last_active":"2024-02-21T23:12:28.069511+0000","last_peered":"2024-02-21T23:12:28.069511+0000","last_clean":"2024-02-21T23:12:28.069511+0000","last_became_active":"2024-02-21T23:12:26.829662+0000","last_became_peered":"2024-02-21T23:12:26.829662+0000","last_unstale":"2024-02-21T23:12:28.069511+0000","last_undegraded":"2024-02-21T23:12:28.069511+0000","last_fullsized":"2024-02-21T23:12:28.069511+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:19:48.049086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.10","version":"196'8","reported_seq":22,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.836065+0000","last_change":"2024-02-21T23:12:25.816885+0000","last_active":"2024-02-21T23:12:27.836065+0000","last_peered":"2024-02-21T23:12:27.836065+0000","last_clean":"2024-02-21T23:12:27.836065+0000","last_became_active":"2024-02-21T23:12:25.816529+0000","last_became_peered":"2024-02-21T23:12:25.816529+0000","last_unstale":"2024-02-21T23:12:27.836065+0000","last_undegraded":"2024-02-21T23:12:27.836065+0000","last_fullsized":"2024-02-21T23:12:27.836065+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_clean_scrub_stamp":"2024-02-21T23:12:24.754357+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:58:41.064952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.30","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:28.068407+0000","last_change":"2024-02-21T23:12:26.824451+0000","last_active":"2024-02-21T23:12:28.068407+0000","last_peered":"2024-02-21T23:12:28.068407+0000","last_clean":"2024-02-21T23:12:28.068407+0000","last_became_active":"2024-02-21T23:12:26.823912+0000","last_became_peered":"2024-02-21T23:12:26.823912+0000","last_unstale":"2024-02-21T23:12:28.068407+0000","last_undegraded":"2024-02-21T23:12:28.068407+0000","last_fullsized":"2024-02-21T23:12:28.068407+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:06:06.093085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.f","version":"196'1","reported_seq":15,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:29.380569+0000","last_change":"2024-02-21T23:12:25.812632+0000","last_active":"2024-02-21T23:12:29.380569+0000","last_peered":"2024-02-21T23:12:29.380569+0000","last_clean":"2024-02-21T23:12:29.380569+0000","last_became_active":"2024-02-21T23:12:25.812290+0000","last_became_peered":"2024-02-21T23:12:25.812290+0000","last_unstale":"2024-02-21T23:12:29.380569+0000","last_undegraded":"2024-02-21T23:12:29.380569+0000","last_fullsized":"2024-02-21T23:12:29.380569+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_clean_scrub_stamp":"2024-02-21T23:12:24.754357+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:06:40.524381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.31","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:28.068224+0000","last_change":"2024-02-21T23:12:26.837195+0000","last_active":"2024-02-21T23:12:28.068224+0000","last_peered":"2024-02-21T23:12:28.068224+0000","last_clean":"2024-02-21T23:12:28.068224+0000","last_became_active":"2024-02-21T23:12:26.836872+0000","last_became_peered":"2024-02-21T23:12:26.836872+0000","last_unstale":"2024-02-21T23:12:28.068224+0000","last_undegraded":"2024-02-21T23:12:28.068224+0000","last_fullsized":"2024-02-21T23:12:28.068224+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:59:16.001273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.e","version":"196'2","reported_seq":16,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.837274+0000","last_change":"2024-02-21T23:12:25.814010+0000","last_active":"2024-02-21T23:12:27.837274+0000","last_peered":"2024-02-21T23:12:27.837274+0000","last_clean":"2024-02-21T23:12:27.837274+0000","last_became_active":"2024-02-21T23:12:25.813703+0000","last_became_peered":"2024-02-21T23:12:25.813703+0000","last_unstale":"2024-02-21T23:12:27.837274+0000","last_undegraded":"2024-02-21T23:12:27.837274+0000","last_fullsized":"2024-02-21T23:12:27.837274+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_clean_scrub_stamp":"2024-02-21T23:12:24.754357+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:16:04.792023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.32","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.836279+0000","last_change":"2024-02-21T23:12:26.822821+0000","last_active":"2024-02-21T23:12:27.836279+0000","last_peered":"2024-02-21T23:12:27.836279+0000","last_clean":"2024-02-21T23:12:27.836279+0000","last_became_active":"2024-02-21T23:12:26.822495+0000","last_became_peered":"2024-02-21T23:12:26.822495+0000","last_unstale":"2024-02-21T23:12:27.836279+0000","last_undegraded":"2024-02-21T23:12:27.836279+0000","last_fullsized":"2024-02-21T23:12:27.836279+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:05:08.136818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.d","version":"196'1","reported_seq":15,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:29.380718+0000","last_change":"2024-02-21T23:12:25.813104+0000","last_active":"2024-02-21T23:12:29.380718+0000","last_peered":"2024-02-21T23:12:29.380718+0000","last_clean":"2024-02-21T23:12:29.380718+0000","last_became_active":"2024-02-21T23:12:25.811515+0000","last_became_peered":"2024-02-21T23:12:25.811515+0000","last_unstale":"2024-02-21T23:12:29.380718+0000","last_undegraded":"2024-02-21T23:12:29.380718+0000","last_fullsized":"2024-02-21T23:12:29.380718+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_clean_scrub_stamp":"2024-02-21T23:12:24.754357+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:04:37.124420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.33","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:28.278502+0000","last_change":"2024-02-21T23:12:26.836283+0000","last_active":"2024-02-21T23:12:28.278502+0000","last_peered":"2024-02-21T23:12:28.278502+0000","last_clean":"2024-02-21T23:12:28.278502+0000","last_became_active":"2024-02-21T23:12:26.835922+0000","last_became_peered":"2024-02-21T23:12:26.835922+0000","last_unstale":"2024-02-21T23:12:28.278502+0000","last_undegraded":"2024-02-21T23:12:28.278502+0000","last_fullsized":"2024-02-21T23:12:28.278502+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:35:41.902303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":14,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.841272+0000","last_change":"2024-02-21T23:12:25.814942+0000","last_active":"2024-02-21T23:12:27.841272+0000","last_peered":"2024-02-21T23:12:27.841272+0000","last_clean":"2024-02-21T23:12:27.841272+0000","last_became_active":"2024-02-21T23:12:25.814624+0000","last_became_peered":"2024-02-21T23:12:25.814624+0000","last_unstale":"2024-02-21T23:12:27.841272+0000","last_undegraded":"2024-02-21T23:12:27.841272+0000","last_fullsized":"2024-02-21T23:12:27.841272+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_clean_scrub_stamp":"2024-02-21T23:12:24.754357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:32:11.595345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.34","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:28.278501+0000","last_change":"2024-02-21T23:12:26.833912+0000","last_active":"2024-02-21T23:12:28.278501+0000","last_peered":"2024-02-21T23:12:28.278501+0000","last_clean":"2024-02-21T23:12:28.278501+0000","last_became_active":"2024-02-21T23:12:26.833359+0000","last_became_peered":"2024-02-21T23:12:26.833359+0000","last_unstale":"2024-02-21T23:12:28.278501+0000","last_undegraded":"2024-02-21T23:12:28.278501+0000","last_fullsized":"2024-02-21T23:12:28.278501+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:35:30.750766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.b","version":"196'2","reported_seq":16,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:28.278816+0000","last_change":"2024-02-21T23:12:25.806765+0000","last_active":"2024-02-21T23:12:28.278816+0000","last_peered":"2024-02-21T23:12:28.278816+0000","last_clean":"2024-02-21T23:12:28.278816+0000","last_became_active":"2024-02-21T23:12:25.806306+0000","last_became_peered":"2024-02-21T23:12:25.806306+0000","last_unstale":"2024-02-21T23:12:28.278816+0000","last_undegraded":"2024-02-21T23:12:28.278816+0000","last_fullsized":"2024-02-21T23:12:28.278816+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_clean_scrub_stamp":"2024-02-21T23:12:24.754357+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:06:05.076870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.35","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.836576+0000","last_change":"2024-02-21T23:12:26.845618+0000","last_active":"2024-02-21T23:12:27.836576+0000","last_peered":"2024-02-21T23:12:27.836576+0000","last_clean":"2024-02-21T23:12:27.836576+0000","last_became_active":"2024-02-21T23:12:26.844335+0000","last_became_peered":"2024-02-21T23:12:26.844335+0000","last_unstale":"2024-02-21T23:12:27.836576+0000","last_undegraded":"2024-02-21T23:12:27.836576+0000","last_fullsized":"2024-02-21T23:12:27.836576+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:10:52.688000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":14,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:29.380326+0000","last_change":"2024-02-21T23:12:25.809819+0000","last_active":"2024-02-21T23:12:29.380326+0000","last_peered":"2024-02-21T23:12:29.380326+0000","last_clean":"2024-02-21T23:12:29.380326+0000","last_became_active":"2024-02-21T23:12:25.809519+0000","last_became_peered":"2024-02-21T23:12:25.809519+0000","last_unstale":"2024-02-21T23:12:29.380326+0000","last_undegraded":"2024-02-21T23:12:29.380326+0000","last_fullsized":"2024-02-21T23:12:29.380326+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_clean_scrub_stamp":"2024-02-21T23:12:24.754357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:27:18.542244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.36","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:29.381271+0000","last_change":"2024-02-21T23:12:26.838228+0000","last_active":"2024-02-21T23:12:29.381271+0000","last_peered":"2024-02-21T23:12:29.381271+0000","last_clean":"2024-02-21T23:12:29.381271+0000","last_became_active":"2024-02-21T23:12:26.837439+0000","last_became_peered":"2024-02-21T23:12:26.837439+0000","last_unstale":"2024-02-21T23:12:29.381271+0000","last_undegraded":"2024-02-21T23:12:29.381271+0000","last_fullsized":"2024-02-21T23:12:29.381271+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:10:15.570938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":14,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:29.380347+0000","last_change":"2024-02-21T23:12:25.811564+0000","last_active":"2024-02-21T23:12:29.380347+0000","last_peered":"2024-02-21T23:12:29.380347+0000","last_clean":"2024-02-21T23:12:29.380347+0000","last_became_active":"2024-02-21T23:12:25.811244+0000","last_became_peered":"2024-02-21T23:12:25.811244+0000","last_unstale":"2024-02-21T23:12:29.380347+0000","last_undegraded":"2024-02-21T23:12:29.380347+0000","last_fullsized":"2024-02-21T23:12:29.380347+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_clean_scrub_stamp":"2024-02-21T23:12:24.754357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:32:06.525620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.37","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:29.381134+0000","last_change":"2024-02-21T23:12:26.838881+0000","last_active":"2024-02-21T23:12:29.381134+0000","last_peered":"2024-02-21T23:12:29.381134+0000","last_clean":"2024-02-21T23:12:29.381134+0000","last_became_active":"2024-02-21T23:12:26.838574+0000","last_became_peered":"2024-02-21T23:12:26.838574+0000","last_unstale":"2024-02-21T23:12:29.381134+0000","last_undegraded":"2024-02-21T23:12:29.381134+0000","last_fullsized":"2024-02-21T23:12:29.381134+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:57:20.967285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":14,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.948002+0000","last_change":"2024-02-21T23:12:25.811248+0000","last_active":"2024-02-21T23:12:27.948002+0000","last_peered":"2024-02-21T23:12:27.948002+0000","last_clean":"2024-02-21T23:12:27.948002+0000","last_became_active":"2024-02-21T23:12:25.807780+0000","last_became_peered":"2024-02-21T23:12:25.807780+0000","last_unstale":"2024-02-21T23:12:27.948002+0000","last_undegraded":"2024-02-21T23:12:27.948002+0000","last_fullsized":"2024-02-21T23:12:27.948002+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_clean_scrub_stamp":"2024-02-21T23:12:24.754357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:30:19.216169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.38","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.838452+0000","last_change":"2024-02-21T23:12:26.828531+0000","last_active":"2024-02-21T23:12:27.838452+0000","last_peered":"2024-02-21T23:12:27.838452+0000","last_clean":"2024-02-21T23:12:27.838452+0000","last_became_active":"2024-02-21T23:12:26.828208+0000","last_became_peered":"2024-02-21T23:12:26.828208+0000","last_unstale":"2024-02-21T23:12:27.838452+0000","last_undegraded":"2024-02-21T23:12:27.838452+0000","last_fullsized":"2024-02-21T23:12:27.838452+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:08:54.861131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.7","version":"196'2","reported_seq":16,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:28.279189+0000","last_change":"2024-02-21T23:12:25.812189+0000","last_active":"2024-02-21T23:12:28.279189+0000","last_peered":"2024-02-21T23:12:28.279189+0000","last_clean":"2024-02-21T23:12:28.279189+0000","last_became_active":"2024-02-21T23:12:25.810343+0000","last_became_peered":"2024-02-21T23:12:25.810343+0000","last_unstale":"2024-02-21T23:12:28.279189+0000","last_undegraded":"2024-02-21T23:12:28.279189+0000","last_fullsized":"2024-02-21T23:12:28.279189+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_clean_scrub_stamp":"2024-02-21T23:12:24.754357+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:49:12.073674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.39","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.840010+0000","last_change":"2024-02-21T23:12:26.848953+0000","last_active":"2024-02-21T23:12:27.840010+0000","last_peered":"2024-02-21T23:12:27.840010+0000","last_clean":"2024-02-21T23:12:27.840010+0000","last_became_active":"2024-02-21T23:12:26.848631+0000","last_became_peered":"2024-02-21T23:12:26.848631+0000","last_unstale":"2024-02-21T23:12:27.840010+0000","last_undegraded":"2024-02-21T23:12:27.840010+0000","last_fullsized":"2024-02-21T23:12:27.840010+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:19:48.049086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.6","version":"196'1","reported_seq":15,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:28.069388+0000","last_change":"2024-02-21T23:12:25.805846+0000","last_active":"2024-02-21T23:12:28.069388+0000","last_peered":"2024-02-21T23:12:28.069388+0000","last_clean":"2024-02-21T23:12:28.069388+0000","last_became_active":"2024-02-21T23:12:25.805072+0000","last_became_peered":"2024-02-21T23:12:25.805072+0000","last_unstale":"2024-02-21T23:12:28.069388+0000","last_undegraded":"2024-02-21T23:12:28.069388+0000","last_fullsized":"2024-02-21T23:12:28.069388+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_clean_scrub_stamp":"2024-02-21T23:12:24.754357+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:05:32.231407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3a","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:28.278467+0000","last_change":"2024-02-21T23:12:26.840139+0000","last_active":"2024-02-21T23:12:28.278467+0000","last_peered":"2024-02-21T23:12:28.278467+0000","last_clean":"2024-02-21T23:12:28.278467+0000","last_became_active":"2024-02-21T23:12:26.839777+0000","last_became_peered":"2024-02-21T23:12:26.839777+0000","last_unstale":"2024-02-21T23:12:28.278467+0000","last_undegraded":"2024-02-21T23:12:28.278467+0000","last_fullsized":"2024-02-21T23:12:28.278467+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:35:54.372949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":14,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.948870+0000","last_change":"2024-02-21T23:12:25.814567+0000","last_active":"2024-02-21T23:12:27.948870+0000","last_peered":"2024-02-21T23:12:27.948870+0000","last_clean":"2024-02-21T23:12:27.948870+0000","last_became_active":"2024-02-21T23:12:25.814284+0000","last_became_peered":"2024-02-21T23:12:25.814284+0000","last_unstale":"2024-02-21T23:12:27.948870+0000","last_undegraded":"2024-02-21T23:12:27.948870+0000","last_fullsized":"2024-02-21T23:12:27.948870+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_clean_scrub_stamp":"2024-02-21T23:12:24.754357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:56:23.887648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3b","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:28.068737+0000","last_change":"2024-02-21T23:12:26.827404+0000","last_active":"2024-02-21T23:12:28.068737+0000","last_peered":"2024-02-21T23:12:28.068737+0000","last_clean":"2024-02-21T23:12:28.068737+0000","last_became_active":"2024-02-21T23:12:26.827135+0000","last_became_peered":"2024-02-21T23:12:26.827135+0000","last_unstale":"2024-02-21T23:12:28.068737+0000","last_undegraded":"2024-02-21T23:12:28.068737+0000","last_fullsized":"2024-02-21T23:12:28.068737+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:14:09.789181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":14,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.948917+0000","last_change":"2024-02-21T23:12:25.812483+0000","last_active":"2024-02-21T23:12:27.948917+0000","last_peered":"2024-02-21T23:12:27.948917+0000","last_clean":"2024-02-21T23:12:27.948917+0000","last_became_active":"2024-02-21T23:12:25.812211+0000","last_became_peered":"2024-02-21T23:12:25.812211+0000","last_unstale":"2024-02-21T23:12:27.948917+0000","last_undegraded":"2024-02-21T23:12:27.948917+0000","last_fullsized":"2024-02-21T23:12:27.948917+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_clean_scrub_stamp":"2024-02-21T23:12:24.754357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:35:29.734551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3c","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.948538+0000","last_change":"2024-02-21T23:12:26.829974+0000","last_active":"2024-02-21T23:12:27.948538+0000","last_peered":"2024-02-21T23:12:27.948538+0000","last_clean":"2024-02-21T23:12:27.948538+0000","last_became_active":"2024-02-21T23:12:26.829656+0000","last_became_peered":"2024-02-21T23:12:26.829656+0000","last_unstale":"2024-02-21T23:12:27.948538+0000","last_undegraded":"2024-02-21T23:12:27.948538+0000","last_fullsized":"2024-02-21T23:12:27.948538+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:51:59.978445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":14,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:29.380565+0000","last_change":"2024-02-21T23:12:25.815800+0000","last_active":"2024-02-21T23:12:29.380565+0000","last_peered":"2024-02-21T23:12:29.380565+0000","last_clean":"2024-02-21T23:12:29.380565+0000","last_became_active":"2024-02-21T23:12:25.815477+0000","last_became_peered":"2024-02-21T23:12:25.815477+0000","last_unstale":"2024-02-21T23:12:29.380565+0000","last_undegraded":"2024-02-21T23:12:29.380565+0000","last_fullsized":"2024-02-21T23:12:29.380565+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_clean_scrub_stamp":"2024-02-21T23:12:24.754357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:02:49.067243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.3d","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:29.380445+0000","last_change":"2024-02-21T23:12:26.855505+0000","last_active":"2024-02-21T23:12:29.380445+0000","last_peered":"2024-02-21T23:12:29.380445+0000","last_clean":"2024-02-21T23:12:29.380445+0000","last_became_active":"2024-02-21T23:12:26.854883+0000","last_became_peered":"2024-02-21T23:12:26.854883+0000","last_unstale":"2024-02-21T23:12:29.380445+0000","last_undegraded":"2024-02-21T23:12:29.380445+0000","last_fullsized":"2024-02-21T23:12:29.380445+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:12:03.094082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":14,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:28.068677+0000","last_change":"2024-02-21T23:12:25.805995+0000","last_active":"2024-02-21T23:12:28.068677+0000","last_peered":"2024-02-21T23:12:28.068677+0000","last_clean":"2024-02-21T23:12:28.068677+0000","last_became_active":"2024-02-21T23:12:25.805645+0000","last_became_peered":"2024-02-21T23:12:25.805645+0000","last_unstale":"2024-02-21T23:12:28.068677+0000","last_undegraded":"2024-02-21T23:12:28.068677+0000","last_fullsized":"2024-02-21T23:12:28.068677+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_clean_scrub_stamp":"2024-02-21T23:12:24.754357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:53:07.829677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3e","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.837164+0000","last_change":"2024-02-21T23:12:26.845823+0000","last_active":"2024-02-21T23:12:27.837164+0000","last_peered":"2024-02-21T23:12:27.837164+0000","last_clean":"2024-02-21T23:12:27.837164+0000","last_became_active":"2024-02-21T23:12:26.845537+0000","last_became_peered":"2024-02-21T23:12:26.845537+0000","last_unstale":"2024-02-21T23:12:27.837164+0000","last_undegraded":"2024-02-21T23:12:27.837164+0000","last_fullsized":"2024-02-21T23:12:27.837164+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:37:46.454268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":14,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:28.278556+0000","last_change":"2024-02-21T23:12:25.807027+0000","last_active":"2024-02-21T23:12:28.278556+0000","last_peered":"2024-02-21T23:12:28.278556+0000","last_clean":"2024-02-21T23:12:28.278556+0000","last_became_active":"2024-02-21T23:12:25.806737+0000","last_became_peered":"2024-02-21T23:12:25.806737+0000","last_unstale":"2024-02-21T23:12:28.278556+0000","last_undegraded":"2024-02-21T23:12:28.278556+0000","last_fullsized":"2024-02-21T23:12:28.278556+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_clean_scrub_stamp":"2024-02-21T23:12:24.754357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:38:51.815687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.3f","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:29.381089+0000","last_change":"2024-02-21T23:12:26.851964+0000","last_active":"2024-02-21T23:12:29.381089+0000","last_peered":"2024-02-21T23:12:29.381089+0000","last_clean":"2024-02-21T23:12:29.381089+0000","last_became_active":"2024-02-21T23:12:26.851609+0000","last_became_peered":"2024-02-21T23:12:26.851609+0000","last_unstale":"2024-02-21T23:12:29.381089+0000","last_undegraded":"2024-02-21T23:12:29.381089+0000","last_fullsized":"2024-02-21T23:12:29.381089+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:53:08.845892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":14,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:29.379911+0000","last_change":"2024-02-21T23:12:25.801596+0000","last_active":"2024-02-21T23:12:29.379911+0000","last_peered":"2024-02-21T23:12:29.379911+0000","last_clean":"2024-02-21T23:12:29.379911+0000","last_became_active":"2024-02-21T23:12:25.801215+0000","last_became_peered":"2024-02-21T23:12:25.801215+0000","last_unstale":"2024-02-21T23:12:29.379911+0000","last_undegraded":"2024-02-21T23:12:29.379911+0000","last_fullsized":"2024-02-21T23:12:29.379911+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_clean_scrub_stamp":"2024-02-21T23:12:24.754357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:18:03.014262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.21","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:29.379759+0000","last_change":"2024-02-21T23:12:26.850189+0000","last_active":"2024-02-21T23:12:29.379759+0000","last_peered":"2024-02-21T23:12:29.379759+0000","last_clean":"2024-02-21T23:12:29.379759+0000","last_became_active":"2024-02-21T23:12:26.849907+0000","last_became_peered":"2024-02-21T23:12:26.849907+0000","last_unstale":"2024-02-21T23:12:29.379759+0000","last_undegraded":"2024-02-21T23:12:29.379759+0000","last_fullsized":"2024-02-21T23:12:29.379759+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:54:05.753523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1e","version":"196'1","reported_seq":15,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.840251+0000","last_change":"2024-02-21T23:12:25.809712+0000","last_active":"2024-02-21T23:12:27.840251+0000","last_peered":"2024-02-21T23:12:27.840251+0000","last_clean":"2024-02-21T23:12:27.840251+0000","last_became_active":"2024-02-21T23:12:25.809422+0000","last_became_peered":"2024-02-21T23:12:25.809422+0000","last_unstale":"2024-02-21T23:12:27.840251+0000","last_undegraded":"2024-02-21T23:12:27.840251+0000","last_fullsized":"2024-02-21T23:12:27.840251+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_clean_scrub_stamp":"2024-02-21T23:12:24.754357+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:16:49.302319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":499,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.840007+0000","last_change":"2024-02-21T23:02:32.289335+0000","last_active":"2024-02-21T23:12:27.840007+0000","last_peered":"2024-02-21T23:12:27.840007+0000","last_clean":"2024-02-21T23:12:27.840007+0000","last_became_active":"2024-02-21T23:02:32.286913+0000","last_became_peered":"2024-02-21T23:02:32.286913+0000","last_unstale":"2024-02-21T23:12:27.840007+0000","last_undegraded":"2024-02-21T23:12:27.840007+0000","last_fullsized":"2024-02-21T23:12:27.840007+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_clean_scrub_stamp":"2024-02-21T23:02:31.172569+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:36:23.987374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":32,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":31,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574970,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33696,"kb_used_data":2512,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338144,"statfs":{"total":96636764160,"available":96602259456,"internally_reserved":0,"allocated":2572288,"data_stored":1817625,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574971,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33628,"kb_used_data":2448,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338212,"statfs":{"total":96636764160,"available":96602329088,"internally_reserved":0,"allocated":2506752,"data_stored":1227245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574970,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33620,"kb_used_data":2440,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338220,"statfs":{"total":96636764160,"available":96602337280,"internally_reserved":0,"allocated":2498560,"data_stored":1227133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574971,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33624,"kb_used_data":2444,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338216,"statfs":{"total":96636764160,"available":96602333184,"internally_reserved":0,"allocated":2502656,"data_stored":1227203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574971,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37808,"kb_used_data":2464,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334032,"statfs":{"total":96636764160,"available":96598048768,"internally_reserved":0,"allocated":2523136,"data_stored":1229383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574972,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33632,"kb_used_data":2448,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338208,"statfs":{"total":96636764160,"available":96602324992,"internally_reserved":0,"allocated":2506752,"data_stored":1227785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574970,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33624,"kb_used_data":2440,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338216,"statfs":{"total":96636764160,"available":96602333184,"internally_reserved":0,"allocated":2498560,"data_stored":1227133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574970,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37860,"kb_used_data":2516,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333980,"statfs":{"total":96636764160,"available":96597995520,"internally_reserved":0,"allocated":2576384,"data_stored":1819113,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-21T23:12:31.301 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-21T23:12:31.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.577+0000 7ff483000640 1 Processor -- start 2024-02-21T23:12:31.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.577+0000 7ff483000640 1 -- start start 2024-02-21T23:12:31.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.578+0000 7ff483000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff47c155960 0x7ff47c1608a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:31.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.578+0000 7ff483000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff47c0c74c0 con 0x7ff47c06dbf0 2024-02-21T23:12:31.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.578+0000 7ff483000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff47c0c77a0 con 0x7ff47c0aff90 2024-02-21T23:12:31.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.578+0000 7ff483000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff47c0c7270 con 0x7ff47c155960 2024-02-21T23:12:31.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.579+0000 7ff481200640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff47c06dbf0 0x7ff47c158ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:32928/0 (socket says 172.21.15.28:32928) 2024-02-21T23:12:31.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.579+0000 7ff481200640 1 -- 172.21.15.28:0/1229013072 learned_addr learned my addr 172.21.15.28:0/1229013072 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:12:31.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.579+0000 7ff482600640 1 --2- 172.21.15.28:0/1229013072 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff47c155960 0x7ff47c1608a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:31.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.579+0000 7ff47be00640 1 -- 172.21.15.28:0/1229013072 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3440537526 0 0) 0x7ff47c0c74c0 con 0x7ff47c06dbf0 2024-02-21T23:12:31.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.579+0000 7ff47be00640 1 -- 172.21.15.28:0/1229013072 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff464003520 con 0x7ff47c06dbf0 2024-02-21T23:12:31.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.579+0000 7ff47be00640 1 -- 172.21.15.28:0/1229013072 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1746188916 0 0) 0x7ff464003520 con 0x7ff47c06dbf0 2024-02-21T23:12:31.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.580+0000 7ff47be00640 1 -- 172.21.15.28:0/1229013072 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff47c155960 msgr2=0x7ff47c1608a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:31.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.580+0000 7ff47be00640 1 --2- 172.21.15.28:0/1229013072 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff47c155960 0x7ff47c1608a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:31.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.580+0000 7ff47be00640 1 -- 172.21.15.28:0/1229013072 >> v1:172.21.15.81:6789/0 conn(0x7ff47c0aff90 legacy=0x7ff47c0c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:31.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.580+0000 7ff47be00640 1 -- 172.21.15.28:0/1229013072 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff47c0c7a80 con 0x7ff47c06dbf0 2024-02-21T23:12:31.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.580+0000 7ff47be00640 1 -- 172.21.15.28:0/1229013072 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7ff468001670 con 0x7ff47c06dbf0 2024-02-21T23:12:31.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.580+0000 7ff47be00640 1 -- 172.21.15.28:0/1229013072 >> v1:172.21.15.28:6789/0 conn(0x7ff47c06dbf0 legacy=0x7ff47c158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:31.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.580+0000 7ff47be00640 1 --2- 172.21.15.28:0/1229013072 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff47c155960 0x7ff4640039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:31.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.580+0000 7ff47be00640 1 --2- 172.21.15.28:0/1229013072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff464004010 0x7ff464004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:31.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.580+0000 7ff47be00640 1 --2- 172.21.15.28:0/1229013072 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff464004d40 0x7ff4640051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:31.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.580+0000 7ff47be00640 1 -- 172.21.15.28:0/1229013072 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff47c0c7270 con 0x7ff464004010 2024-02-21T23:12:31.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.580+0000 7ff47be00640 1 -- 172.21.15.28:0/1229013072 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff46400bb80 con 0x7ff464004d40 2024-02-21T23:12:31.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.580+0000 7ff47be00640 1 -- 172.21.15.28:0/1229013072 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff46400bd80 con 0x7ff47c155960 2024-02-21T23:12:31.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.580+0000 7ff481200640 1 --2- 172.21.15.28:0/1229013072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff464004010 0x7ff464004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:31.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.580+0000 7ff482600640 1 --2- 172.21.15.28:0/1229013072 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff464004d40 0x7ff4640051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:31.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.580+0000 7ff481c00640 1 --2- 172.21.15.28:0/1229013072 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff47c155960 0x7ff4640039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:31.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.580+0000 7ff481200640 1 -- 172.21.15.28:0/1229013072 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff47c155960 msgr2=0x7ff4640039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:31.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.580+0000 7ff481200640 1 --2- 172.21.15.28:0/1229013072 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff47c155960 0x7ff4640039e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:31.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.581+0000 7ff481200640 1 -- 172.21.15.28:0/1229013072 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff464004d40 msgr2=0x7ff4640051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:31.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.581+0000 7ff481200640 1 --2- 172.21.15.28:0/1229013072 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff464004d40 0x7ff4640051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:31.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.581+0000 7ff481200640 1 -- 172.21.15.28:0/1229013072 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff464003520 con 0x7ff464004010 2024-02-21T23:12:31.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.581+0000 7ff482600640 1 --2- 172.21.15.28:0/1229013072 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff464004d40 0x7ff4640051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:12:31.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.582+0000 7ff481200640 1 --2- 172.21.15.28:0/1229013072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff464004010 0x7ff464004800 secure :-1 s=READY pgs=2248 cs=0 l=1 rev1=1 crypto rx=0x7ff47c06e080 tx=0x7ff464005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14d700da0b96240b server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:31.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.582+0000 7ff47be00640 1 -- 172.21.15.28:0/1229013072 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff468007d30 con 0x7ff464004010 2024-02-21T23:12:31.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.582+0000 7ff47be00640 1 -- 172.21.15.28:0/1229013072 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff468030070 con 0x7ff464004010 2024-02-21T23:12:31.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.582+0000 7ff47be00640 1 -- 172.21.15.28:0/1229013072 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff468005d70 con 0x7ff464004010 2024-02-21T23:12:31.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.583+0000 7ff483000640 1 -- 172.21.15.28:0/1229013072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff464004010 msgr2=0x7ff464004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:31.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.583+0000 7ff483000640 1 --2- 172.21.15.28:0/1229013072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff464004010 0x7ff464004800 secure :-1 s=READY pgs=2248 cs=0 l=1 rev1=1 crypto rx=0x7ff47c06e080 tx=0x7ff464005830 comp rx=0 tx=0).stop 2024-02-21T23:12:31.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.583+0000 7ff482600640 1 -- 172.21.15.28:0/1229013072 reap_dead start 2024-02-21T23:12:31.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.583+0000 7ff483000640 1 -- 172.21.15.28:0/1229013072 shutdown_connections 2024-02-21T23:12:31.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.583+0000 7ff483000640 1 -- 172.21.15.28:0/1229013072 >> 172.21.15.28:0/1229013072 conn(0x7ff47c0ac370 msgr2=0x7ff47c0ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:31.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.583+0000 7ff483000640 1 -- 172.21.15.28:0/1229013072 shutdown_connections 2024-02-21T23:12:31.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.583+0000 7ff483000640 1 -- 172.21.15.28:0/1229013072 wait complete. 2024-02-21T23:12:31.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.584+0000 7ff483000640 1 Processor -- start 2024-02-21T23:12:31.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.585+0000 7ff483000640 1 -- start start 2024-02-21T23:12:31.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.585+0000 7ff483000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff464004010 0x7ff47c0be310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:31.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.585+0000 7ff483000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff47c0c45d0 0x7ff47c0c2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:31.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.586+0000 7ff483000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff47c0bcdf0 0x7ff47c0bd270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:31.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.586+0000 7ff483000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff47c0c58f0 con 0x7ff47c0c45d0 2024-02-21T23:12:31.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.586+0000 7ff483000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff47c158630 con 0x7ff47c0bcdf0 2024-02-21T23:12:31.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.586+0000 7ff483000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff47c0c5a70 con 0x7ff464004010 2024-02-21T23:12:31.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.586+0000 7ff481200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff47c0c45d0 0x7ff47c0c2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:31.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.586+0000 7ff481200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff47c0c45d0 0x7ff47c0c2b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44388/0 (socket says 172.21.15.28:44388) 2024-02-21T23:12:31.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.586+0000 7ff481200640 1 -- 172.21.15.28:0/1105055817 learned_addr learned my addr 172.21.15.28:0/1105055817 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:31.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.586+0000 7ff482600640 1 --2- 172.21.15.28:0/1105055817 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff47c0bcdf0 0x7ff47c0bd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:31.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.586+0000 7ff481c00640 1 --2- 172.21.15.28:0/1105055817 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff464004010 0x7ff47c0be310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:31.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.586+0000 7ff481200640 1 -- 172.21.15.28:0/1105055817 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff464004010 msgr2=0x7ff47c0be310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:31.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.586+0000 7ff481200640 1 --2- 172.21.15.28:0/1105055817 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff464004010 0x7ff47c0be310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:31.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.586+0000 7ff481200640 1 -- 172.21.15.28:0/1105055817 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff47c0bcdf0 msgr2=0x7ff47c0bd270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:31.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.586+0000 7ff481200640 1 --2- 172.21.15.28:0/1105055817 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff47c0bcdf0 0x7ff47c0bd270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:31.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.586+0000 7ff481200640 1 -- 172.21.15.28:0/1105055817 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff47c0c1ad0 con 0x7ff47c0c45d0 2024-02-21T23:12:31.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.586+0000 7ff482600640 1 --2- 172.21.15.28:0/1105055817 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff47c0bcdf0 0x7ff47c0bd270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:12:31.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.587+0000 7ff481200640 1 --2- 172.21.15.28:0/1105055817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff47c0c45d0 0x7ff47c0c2b70 secure :-1 s=READY pgs=2249 cs=0 l=1 rev1=1 crypto rx=0x7ff46800c300 tx=0x7ff468001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:31.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.587+0000 7ff479600640 1 -- 172.21.15.28:0/1105055817 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff468004bf0 con 0x7ff47c0c45d0 2024-02-21T23:12:31.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.587+0000 7ff479600640 1 -- 172.21.15.28:0/1105055817 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff468004d90 con 0x7ff47c0c45d0 2024-02-21T23:12:31.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.587+0000 7ff483000640 1 -- 172.21.15.28:0/1105055817 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff47c0c1db0 con 0x7ff47c0c45d0 2024-02-21T23:12:31.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.587+0000 7ff479600640 1 -- 172.21.15.28:0/1105055817 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff468007270 con 0x7ff47c0c45d0 2024-02-21T23:12:31.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.587+0000 7ff483000640 1 -- 172.21.15.28:0/1105055817 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff47c0c2370 con 0x7ff47c0c45d0 2024-02-21T23:12:31.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.588+0000 7ff483000640 1 -- 172.21.15.28:0/1105055817 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff47c04f290 con 0x7ff47c0c45d0 2024-02-21T23:12:31.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.589+0000 7ff479600640 1 -- 172.21.15.28:0/1105055817 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff468047020 con 0x7ff47c0c45d0 2024-02-21T23:12:31.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.590+0000 7ff479600640 1 --2- 172.21.15.28:0/1105055817 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff45c06b4d0 0x7ff45c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:31.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.590+0000 7ff479600640 1 -- 172.21.15.28:0/1105055817 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff4680ab5c0 con 0x7ff47c0c45d0 2024-02-21T23:12:31.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.591+0000 7ff481c00640 1 --2- 172.21.15.28:0/1105055817 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff45c06b4d0 0x7ff45c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:31.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.592+0000 7ff481c00640 1 --2- 172.21.15.28:0/1105055817 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff45c06b4d0 0x7ff45c06d990 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7ff470001710 tx=0x7ff470009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:31.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.595+0000 7ff479600640 1 -- 172.21.15.28:0/1105055817 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ff46807a340 con 0x7ff47c0c45d0 2024-02-21T23:12:31.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.793+0000 7ff483000640 1 -- 172.21.15.28:0/1105055817 --> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff47c06d9d0 con 0x7ff45c06b4d0 2024-02-21T23:12:31.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.809+0000 7ff479600640 1 -- 172.21.15.28:0/1105055817 <== mgr.4101 v2:172.21.15.81:6832/1848538577 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252566 (secure 0 0 0) 0x7ff47c06d9d0 con 0x7ff45c06b4d0 2024-02-21T23:12:31.802 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:12:31.803 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-21T23:12:31.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.815+0000 7ff483000640 1 -- 172.21.15.28:0/1105055817 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff45c06b4d0 msgr2=0x7ff45c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:31.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.815+0000 7ff483000640 1 --2- 172.21.15.28:0/1105055817 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff45c06b4d0 0x7ff45c06d990 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7ff470001710 tx=0x7ff470009040 comp rx=0 tx=0).stop 2024-02-21T23:12:31.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.815+0000 7ff483000640 1 -- 172.21.15.28:0/1105055817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff47c0c45d0 msgr2=0x7ff47c0c2b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:31.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.815+0000 7ff483000640 1 --2- 172.21.15.28:0/1105055817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff47c0c45d0 0x7ff47c0c2b70 secure :-1 s=READY pgs=2249 cs=0 l=1 rev1=1 crypto rx=0x7ff46800c300 tx=0x7ff468001a40 comp rx=0 tx=0).stop 2024-02-21T23:12:31.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.815+0000 7ff483000640 1 -- 172.21.15.28:0/1105055817 shutdown_connections 2024-02-21T23:12:31.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.815+0000 7ff483000640 1 --2- 172.21.15.28:0/1105055817 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff45c06b4d0 0x7ff45c06d990 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:31.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.815+0000 7ff483000640 1 --2- 172.21.15.28:0/1105055817 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff47c0bcdf0 0x7ff47c0bd270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:31.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.815+0000 7ff483000640 1 --2- 172.21.15.28:0/1105055817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff47c0c45d0 0x7ff47c0c2b70 unknown :-1 s=CLOSED pgs=2249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:31.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.815+0000 7ff483000640 1 --2- 172.21.15.28:0/1105055817 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff464004010 0x7ff47c0be310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:31.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.815+0000 7ff483000640 1 -- 172.21.15.28:0/1105055817 >> 172.21.15.28:0/1105055817 conn(0x7ff47c0ac370 msgr2=0x7ff47c0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:31.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.815+0000 7ff483000640 1 -- 172.21.15.28:0/1105055817 shutdown_connections 2024-02-21T23:12:31.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:31.815+0000 7ff483000640 1 -- 172.21.15.28:0/1105055817 wait complete. 2024-02-21T23:12:31.823 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":490,"stamp":"2024-02-21T23:12:30.876748+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":173,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":277492,"kb_used_data":19712,"kb_used_omap":12,"kb_used_meta":257651,"kb_avail":754697228,"statfs":{"total":773094113280,"available":772809961472,"internally_reserved":0,"allocated":20185088,"data_stored":11002620,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":263835185},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001856"},"pg_stats":[{"pgid":"32.1f","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:29.380264+0000","last_change":"2024-02-21T23:12:26.848756+0000","last_active":"2024-02-21T23:12:29.380264+0000","last_peered":"2024-02-21T23:12:29.380264+0000","last_clean":"2024-02-21T23:12:29.380264+0000","last_became_active":"2024-02-21T23:12:26.848437+0000","last_became_peered":"2024-02-21T23:12:26.848437+0000","last_unstale":"2024-02-21T23:12:29.380264+0000","last_undegraded":"2024-02-21T23:12:29.380264+0000","last_fullsized":"2024-02-21T23:12:29.380264+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:52:23.398556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.1e","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.836933+0000","last_change":"2024-02-21T23:12:26.847858+0000","last_active":"2024-02-21T23:12:27.836933+0000","last_peered":"2024-02-21T23:12:27.836933+0000","last_clean":"2024-02-21T23:12:27.836933+0000","last_became_active":"2024-02-21T23:12:26.845014+0000","last_became_peered":"2024-02-21T23:12:26.845014+0000","last_unstale":"2024-02-21T23:12:27.836933+0000","last_undegraded":"2024-02-21T23:12:27.836933+0000","last_fullsized":"2024-02-21T23:12:27.836933+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:21:52.739416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1d","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:28.278913+0000","last_change":"2024-02-21T23:12:26.830438+0000","last_active":"2024-02-21T23:12:28.278913+0000","last_peered":"2024-02-21T23:12:28.278913+0000","last_clean":"2024-02-21T23:12:28.278913+0000","last_became_active":"2024-02-21T23:12:26.829573+0000","last_became_peered":"2024-02-21T23:12:26.829573+0000","last_unstale":"2024-02-21T23:12:28.278913+0000","last_undegraded":"2024-02-21T23:12:28.278913+0000","last_fullsized":"2024-02-21T23:12:28.278913+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:20:49.569662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1c","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:28.278745+0000","last_change":"2024-02-21T23:12:26.835807+0000","last_active":"2024-02-21T23:12:28.278745+0000","last_peered":"2024-02-21T23:12:28.278745+0000","last_clean":"2024-02-21T23:12:28.278745+0000","last_became_active":"2024-02-21T23:12:26.835526+0000","last_became_peered":"2024-02-21T23:12:26.835526+0000","last_unstale":"2024-02-21T23:12:28.278745+0000","last_undegraded":"2024-02-21T23:12:28.278745+0000","last_fullsized":"2024-02-21T23:12:28.278745+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:39:57.281364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1b","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:28.068868+0000","last_change":"2024-02-21T23:12:26.842846+0000","last_active":"2024-02-21T23:12:28.068868+0000","last_peered":"2024-02-21T23:12:28.068868+0000","last_clean":"2024-02-21T23:12:28.068868+0000","last_became_active":"2024-02-21T23:12:26.842493+0000","last_became_peered":"2024-02-21T23:12:26.842493+0000","last_unstale":"2024-02-21T23:12:28.068868+0000","last_undegraded":"2024-02-21T23:12:28.068868+0000","last_fullsized":"2024-02-21T23:12:28.068868+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:58:18.463148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1a","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:29.380155+0000","last_change":"2024-02-21T23:12:26.844005+0000","last_active":"2024-02-21T23:12:29.380155+0000","last_peered":"2024-02-21T23:12:29.380155+0000","last_clean":"2024-02-21T23:12:29.380155+0000","last_became_active":"2024-02-21T23:12:26.843735+0000","last_became_peered":"2024-02-21T23:12:26.843735+0000","last_unstale":"2024-02-21T23:12:29.380155+0000","last_undegraded":"2024-02-21T23:12:29.380155+0000","last_fullsized":"2024-02-21T23:12:29.380155+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:15:27.708023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.838468+0000","last_change":"2024-02-21T23:12:26.827683+0000","last_active":"2024-02-21T23:12:27.838468+0000","last_peered":"2024-02-21T23:12:27.838468+0000","last_clean":"2024-02-21T23:12:27.838468+0000","last_became_active":"2024-02-21T23:12:26.827414+0000","last_became_peered":"2024-02-21T23:12:26.827414+0000","last_unstale":"2024-02-21T23:12:27.838468+0000","last_undegraded":"2024-02-21T23:12:27.838468+0000","last_fullsized":"2024-02-21T23:12:27.838468+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:19:48.049086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:28.068438+0000","last_change":"2024-02-21T23:12:26.842685+0000","last_active":"2024-02-21T23:12:28.068438+0000","last_peered":"2024-02-21T23:12:28.068438+0000","last_clean":"2024-02-21T23:12:28.068438+0000","last_became_active":"2024-02-21T23:12:26.841736+0000","last_became_peered":"2024-02-21T23:12:26.841736+0000","last_unstale":"2024-02-21T23:12:28.068438+0000","last_undegraded":"2024-02-21T23:12:28.068438+0000","last_fullsized":"2024-02-21T23:12:28.068438+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:49:13.089888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:29.380368+0000","last_change":"2024-02-21T23:12:26.849708+0000","last_active":"2024-02-21T23:12:29.380368+0000","last_peered":"2024-02-21T23:12:29.380368+0000","last_clean":"2024-02-21T23:12:29.380368+0000","last_became_active":"2024-02-21T23:12:26.848070+0000","last_became_peered":"2024-02-21T23:12:26.848070+0000","last_unstale":"2024-02-21T23:12:29.380368+0000","last_undegraded":"2024-02-21T23:12:29.380368+0000","last_fullsized":"2024-02-21T23:12:29.380368+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:56:36.503491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:28.279112+0000","last_change":"2024-02-21T23:12:26.835797+0000","last_active":"2024-02-21T23:12:28.279112+0000","last_peered":"2024-02-21T23:12:28.279112+0000","last_clean":"2024-02-21T23:12:28.279112+0000","last_became_active":"2024-02-21T23:12:26.835006+0000","last_became_peered":"2024-02-21T23:12:26.835006+0000","last_unstale":"2024-02-21T23:12:28.279112+0000","last_undegraded":"2024-02-21T23:12:28.279112+0000","last_fullsized":"2024-02-21T23:12:28.279112+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:33:47.725072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.15","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:28.279003+0000","last_change":"2024-02-21T23:12:26.835197+0000","last_active":"2024-02-21T23:12:28.279003+0000","last_peered":"2024-02-21T23:12:28.279003+0000","last_clean":"2024-02-21T23:12:28.279003+0000","last_became_active":"2024-02-21T23:12:26.834887+0000","last_became_peered":"2024-02-21T23:12:26.834887+0000","last_unstale":"2024-02-21T23:12:28.279003+0000","last_undegraded":"2024-02-21T23:12:28.279003+0000","last_fullsized":"2024-02-21T23:12:28.279003+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:21:43.387305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.14","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:29.380081+0000","last_change":"2024-02-21T23:12:26.853250+0000","last_active":"2024-02-21T23:12:29.380081+0000","last_peered":"2024-02-21T23:12:29.380081+0000","last_clean":"2024-02-21T23:12:29.380081+0000","last_became_active":"2024-02-21T23:12:26.847779+0000","last_became_peered":"2024-02-21T23:12:26.847779+0000","last_unstale":"2024-02-21T23:12:29.380081+0000","last_undegraded":"2024-02-21T23:12:29.380081+0000","last_fullsized":"2024-02-21T23:12:29.380081+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:41:45.251159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.13","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:29.380736+0000","last_change":"2024-02-21T23:12:26.838145+0000","last_active":"2024-02-21T23:12:29.380736+0000","last_peered":"2024-02-21T23:12:29.380736+0000","last_clean":"2024-02-21T23:12:29.380736+0000","last_became_active":"2024-02-21T23:12:26.834449+0000","last_became_peered":"2024-02-21T23:12:26.834449+0000","last_unstale":"2024-02-21T23:12:29.380736+0000","last_undegraded":"2024-02-21T23:12:29.380736+0000","last_fullsized":"2024-02-21T23:12:29.380736+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:16:52.301170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.948792+0000","last_change":"2024-02-21T23:12:26.843019+0000","last_active":"2024-02-21T23:12:27.948792+0000","last_peered":"2024-02-21T23:12:27.948792+0000","last_clean":"2024-02-21T23:12:27.948792+0000","last_became_active":"2024-02-21T23:12:26.842707+0000","last_became_peered":"2024-02-21T23:12:26.842707+0000","last_unstale":"2024-02-21T23:12:27.948792+0000","last_undegraded":"2024-02-21T23:12:27.948792+0000","last_fullsized":"2024-02-21T23:12:27.948792+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:53:34.566060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.840844+0000","last_change":"2024-02-21T23:12:26.844735+0000","last_active":"2024-02-21T23:12:27.840844+0000","last_peered":"2024-02-21T23:12:27.840844+0000","last_clean":"2024-02-21T23:12:27.840844+0000","last_became_active":"2024-02-21T23:12:26.844148+0000","last_became_peered":"2024-02-21T23:12:26.844148+0000","last_unstale":"2024-02-21T23:12:27.840844+0000","last_undegraded":"2024-02-21T23:12:27.840844+0000","last_fullsized":"2024-02-21T23:12:27.840844+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:20:44.472226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.10","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:28.278425+0000","last_change":"2024-02-21T23:12:26.828679+0000","last_active":"2024-02-21T23:12:28.278425+0000","last_peered":"2024-02-21T23:12:28.278425+0000","last_clean":"2024-02-21T23:12:28.278425+0000","last_became_active":"2024-02-21T23:12:26.828306+0000","last_became_peered":"2024-02-21T23:12:26.828306+0000","last_unstale":"2024-02-21T23:12:28.278425+0000","last_undegraded":"2024-02-21T23:12:28.278425+0000","last_fullsized":"2024-02-21T23:12:28.278425+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:42:34.425253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.f","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:29.381318+0000","last_change":"2024-02-21T23:12:26.832308+0000","last_active":"2024-02-21T23:12:29.381318+0000","last_peered":"2024-02-21T23:12:29.381318+0000","last_clean":"2024-02-21T23:12:29.381318+0000","last_became_active":"2024-02-21T23:12:26.831963+0000","last_became_peered":"2024-02-21T23:12:26.831963+0000","last_unstale":"2024-02-21T23:12:29.381318+0000","last_undegraded":"2024-02-21T23:12:29.381318+0000","last_fullsized":"2024-02-21T23:12:29.381318+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:54:11.876493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.e","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:28.279357+0000","last_change":"2024-02-21T23:12:26.837244+0000","last_active":"2024-02-21T23:12:28.279357+0000","last_peered":"2024-02-21T23:12:28.279357+0000","last_clean":"2024-02-21T23:12:28.279357+0000","last_became_active":"2024-02-21T23:12:26.836959+0000","last_became_peered":"2024-02-21T23:12:26.836959+0000","last_unstale":"2024-02-21T23:12:28.279357+0000","last_undegraded":"2024-02-21T23:12:28.279357+0000","last_fullsized":"2024-02-21T23:12:28.279357+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:51:59.978445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.d","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.841005+0000","last_change":"2024-02-21T23:12:26.836887+0000","last_active":"2024-02-21T23:12:27.841005+0000","last_peered":"2024-02-21T23:12:27.841005+0000","last_clean":"2024-02-21T23:12:27.841005+0000","last_became_active":"2024-02-21T23:12:26.836600+0000","last_became_peered":"2024-02-21T23:12:26.836600+0000","last_unstale":"2024-02-21T23:12:27.841005+0000","last_undegraded":"2024-02-21T23:12:27.841005+0000","last_fullsized":"2024-02-21T23:12:27.841005+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:26:12.656815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.836974+0000","last_change":"2024-02-21T23:12:26.825571+0000","last_active":"2024-02-21T23:12:27.836974+0000","last_peered":"2024-02-21T23:12:27.836974+0000","last_clean":"2024-02-21T23:12:27.836974+0000","last_became_active":"2024-02-21T23:12:26.824298+0000","last_became_peered":"2024-02-21T23:12:26.824298+0000","last_unstale":"2024-02-21T23:12:27.836974+0000","last_undegraded":"2024-02-21T23:12:27.836974+0000","last_fullsized":"2024-02-21T23:12:27.836974+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:25:24.812384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.b","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.838696+0000","last_change":"2024-02-21T23:12:26.836194+0000","last_active":"2024-02-21T23:12:27.838696+0000","last_peered":"2024-02-21T23:12:27.838696+0000","last_clean":"2024-02-21T23:12:27.838696+0000","last_became_active":"2024-02-21T23:12:26.835892+0000","last_became_peered":"2024-02-21T23:12:26.835892+0000","last_unstale":"2024-02-21T23:12:27.838696+0000","last_undegraded":"2024-02-21T23:12:27.838696+0000","last_fullsized":"2024-02-21T23:12:27.838696+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:48:54.474119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:29.380396+0000","last_change":"2024-02-21T23:12:26.852351+0000","last_active":"2024-02-21T23:12:29.380396+0000","last_peered":"2024-02-21T23:12:29.380396+0000","last_clean":"2024-02-21T23:12:29.380396+0000","last_became_active":"2024-02-21T23:12:26.852005+0000","last_became_peered":"2024-02-21T23:12:26.852005+0000","last_unstale":"2024-02-21T23:12:29.380396+0000","last_undegraded":"2024-02-21T23:12:29.380396+0000","last_fullsized":"2024-02-21T23:12:29.380396+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:05:33.247621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.841142+0000","last_change":"2024-02-21T23:12:26.839425+0000","last_active":"2024-02-21T23:12:27.841142+0000","last_peered":"2024-02-21T23:12:27.841142+0000","last_clean":"2024-02-21T23:12:27.841142+0000","last_became_active":"2024-02-21T23:12:26.839063+0000","last_became_peered":"2024-02-21T23:12:26.839063+0000","last_unstale":"2024-02-21T23:12:27.841142+0000","last_undegraded":"2024-02-21T23:12:27.841142+0000","last_fullsized":"2024-02-21T23:12:27.841142+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:30:20.232383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:29.380133+0000","last_change":"2024-02-21T23:12:26.835513+0000","last_active":"2024-02-21T23:12:29.380133+0000","last_peered":"2024-02-21T23:12:29.380133+0000","last_clean":"2024-02-21T23:12:29.380133+0000","last_became_active":"2024-02-21T23:12:26.835244+0000","last_became_peered":"2024-02-21T23:12:26.835244+0000","last_unstale":"2024-02-21T23:12:29.380133+0000","last_undegraded":"2024-02-21T23:12:29.380133+0000","last_fullsized":"2024-02-21T23:12:29.380133+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:51:31.617475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.7","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.837499+0000","last_change":"2024-02-21T23:12:26.830592+0000","last_active":"2024-02-21T23:12:27.837499+0000","last_peered":"2024-02-21T23:12:27.837499+0000","last_clean":"2024-02-21T23:12:27.837499+0000","last_became_active":"2024-02-21T23:12:26.830315+0000","last_became_peered":"2024-02-21T23:12:26.830315+0000","last_unstale":"2024-02-21T23:12:27.837499+0000","last_undegraded":"2024-02-21T23:12:27.837499+0000","last_fullsized":"2024-02-21T23:12:27.837499+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:49:50.412580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.839493+0000","last_change":"2024-02-21T23:12:26.839564+0000","last_active":"2024-02-21T23:12:27.839493+0000","last_peered":"2024-02-21T23:12:27.839493+0000","last_clean":"2024-02-21T23:12:27.839493+0000","last_became_active":"2024-02-21T23:12:26.839297+0000","last_became_peered":"2024-02-21T23:12:26.839297+0000","last_unstale":"2024-02-21T23:12:27.839493+0000","last_undegraded":"2024-02-21T23:12:27.839493+0000","last_fullsized":"2024-02-21T23:12:27.839493+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:30:20.232383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:28.069100+0000","last_change":"2024-02-21T23:12:26.825911+0000","last_active":"2024-02-21T23:12:28.069100+0000","last_peered":"2024-02-21T23:12:28.069100+0000","last_clean":"2024-02-21T23:12:28.069100+0000","last_became_active":"2024-02-21T23:12:26.825586+0000","last_became_peered":"2024-02-21T23:12:26.825586+0000","last_unstale":"2024-02-21T23:12:28.069100+0000","last_undegraded":"2024-02-21T23:12:28.069100+0000","last_fullsized":"2024-02-21T23:12:28.069100+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:08:54.861131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:29.380151+0000","last_change":"2024-02-21T23:12:26.855217+0000","last_active":"2024-02-21T23:12:29.380151+0000","last_peered":"2024-02-21T23:12:29.380151+0000","last_clean":"2024-02-21T23:12:29.380151+0000","last_became_active":"2024-02-21T23:12:26.853599+0000","last_became_peered":"2024-02-21T23:12:26.853599+0000","last_unstale":"2024-02-21T23:12:29.380151+0000","last_undegraded":"2024-02-21T23:12:29.380151+0000","last_fullsized":"2024-02-21T23:12:29.380151+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:22:02.553536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.948401+0000","last_change":"2024-02-21T23:12:26.844231+0000","last_active":"2024-02-21T23:12:27.948401+0000","last_peered":"2024-02-21T23:12:27.948401+0000","last_clean":"2024-02-21T23:12:27.948401+0000","last_became_active":"2024-02-21T23:12:26.842514+0000","last_became_peered":"2024-02-21T23:12:26.842514+0000","last_unstale":"2024-02-21T23:12:27.948401+0000","last_undegraded":"2024-02-21T23:12:27.948401+0000","last_fullsized":"2024-02-21T23:12:27.948401+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:33:47.725072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.948281+0000","last_change":"2024-02-21T23:12:26.834209+0000","last_active":"2024-02-21T23:12:27.948281+0000","last_peered":"2024-02-21T23:12:27.948281+0000","last_clean":"2024-02-21T23:12:27.948281+0000","last_became_active":"2024-02-21T23:12:26.833908+0000","last_became_peered":"2024-02-21T23:12:26.833908+0000","last_unstale":"2024-02-21T23:12:27.948281+0000","last_undegraded":"2024-02-21T23:12:27.948281+0000","last_fullsized":"2024-02-21T23:12:27.948281+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:18:39.370028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.836234+0000","last_change":"2024-02-21T23:12:26.839500+0000","last_active":"2024-02-21T23:12:27.836234+0000","last_peered":"2024-02-21T23:12:27.836234+0000","last_clean":"2024-02-21T23:12:27.836234+0000","last_became_active":"2024-02-21T23:12:26.838973+0000","last_became_peered":"2024-02-21T23:12:26.838973+0000","last_unstale":"2024-02-21T23:12:27.836234+0000","last_undegraded":"2024-02-21T23:12:27.836234+0000","last_fullsized":"2024-02-21T23:12:27.836234+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:53:10.865222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:29.379898+0000","last_change":"2024-02-21T23:12:26.838842+0000","last_active":"2024-02-21T23:12:29.379898+0000","last_peered":"2024-02-21T23:12:29.379898+0000","last_clean":"2024-02-21T23:12:29.379898+0000","last_became_active":"2024-02-21T23:12:26.838533+0000","last_became_peered":"2024-02-21T23:12:26.838533+0000","last_unstale":"2024-02-21T23:12:29.379898+0000","last_undegraded":"2024-02-21T23:12:29.379898+0000","last_fullsized":"2024-02-21T23:12:29.379898+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:42:28.345683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.20","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.836522+0000","last_change":"2024-02-21T23:12:26.847313+0000","last_active":"2024-02-21T23:12:27.836522+0000","last_peered":"2024-02-21T23:12:27.836522+0000","last_clean":"2024-02-21T23:12:27.836522+0000","last_became_active":"2024-02-21T23:12:26.846999+0000","last_became_peered":"2024-02-21T23:12:26.846999+0000","last_unstale":"2024-02-21T23:12:27.836522+0000","last_undegraded":"2024-02-21T23:12:27.836522+0000","last_fullsized":"2024-02-21T23:12:27.836522+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:29:03.598188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1f","version":"196'4","reported_seq":18,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:29.380144+0000","last_change":"2024-02-21T23:12:25.820534+0000","last_active":"2024-02-21T23:12:29.380144+0000","last_peered":"2024-02-21T23:12:29.380144+0000","last_clean":"2024-02-21T23:12:29.380144+0000","last_became_active":"2024-02-21T23:12:25.820259+0000","last_became_peered":"2024-02-21T23:12:25.820259+0000","last_unstale":"2024-02-21T23:12:29.380144+0000","last_undegraded":"2024-02-21T23:12:29.380144+0000","last_fullsized":"2024-02-21T23:12:29.380144+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_clean_scrub_stamp":"2024-02-21T23:12:24.754357+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:29:45.724598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.22","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.840221+0000","last_change":"2024-02-21T23:12:26.838027+0000","last_active":"2024-02-21T23:12:27.840221+0000","last_peered":"2024-02-21T23:12:27.840221+0000","last_clean":"2024-02-21T23:12:27.840221+0000","last_became_active":"2024-02-21T23:12:26.837691+0000","last_became_peered":"2024-02-21T23:12:26.837691+0000","last_unstale":"2024-02-21T23:12:27.840221+0000","last_undegraded":"2024-02-21T23:12:27.840221+0000","last_fullsized":"2024-02-21T23:12:27.840221+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:48:42.776013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.1d","version":"196'2","reported_seq":16,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:29.380638+0000","last_change":"2024-02-21T23:12:25.811795+0000","last_active":"2024-02-21T23:12:29.380638+0000","last_peered":"2024-02-21T23:12:29.380638+0000","last_clean":"2024-02-21T23:12:29.380638+0000","last_became_active":"2024-02-21T23:12:25.811505+0000","last_became_peered":"2024-02-21T23:12:25.811505+0000","last_unstale":"2024-02-21T23:12:29.380638+0000","last_undegraded":"2024-02-21T23:12:29.380638+0000","last_fullsized":"2024-02-21T23:12:29.380638+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_clean_scrub_stamp":"2024-02-21T23:12:24.754357+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:22:45.012364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.23","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:29.380532+0000","last_change":"2024-02-21T23:12:26.847364+0000","last_active":"2024-02-21T23:12:29.380532+0000","last_peered":"2024-02-21T23:12:29.380532+0000","last_clean":"2024-02-21T23:12:29.380532+0000","last_became_active":"2024-02-21T23:12:26.846720+0000","last_became_peered":"2024-02-21T23:12:26.846720+0000","last_unstale":"2024-02-21T23:12:29.380532+0000","last_undegraded":"2024-02-21T23:12:29.380532+0000","last_fullsized":"2024-02-21T23:12:29.380532+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:08:55.825903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1c","version":"196'2","reported_seq":16,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.949031+0000","last_change":"2024-02-21T23:12:25.811712+0000","last_active":"2024-02-21T23:12:27.949031+0000","last_peered":"2024-02-21T23:12:27.949031+0000","last_clean":"2024-02-21T23:12:27.949031+0000","last_became_active":"2024-02-21T23:12:25.811286+0000","last_became_peered":"2024-02-21T23:12:25.811286+0000","last_unstale":"2024-02-21T23:12:27.949031+0000","last_undegraded":"2024-02-21T23:12:27.949031+0000","last_fullsized":"2024-02-21T23:12:27.949031+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_clean_scrub_stamp":"2024-02-21T23:12:24.754357+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:42:33.409039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.24","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.948615+0000","last_change":"2024-02-21T23:12:26.842291+0000","last_active":"2024-02-21T23:12:27.948615+0000","last_peered":"2024-02-21T23:12:27.948615+0000","last_clean":"2024-02-21T23:12:27.948615+0000","last_became_active":"2024-02-21T23:12:26.841973+0000","last_became_peered":"2024-02-21T23:12:26.841973+0000","last_unstale":"2024-02-21T23:12:27.948615+0000","last_undegraded":"2024-02-21T23:12:27.948615+0000","last_fullsized":"2024-02-21T23:12:27.948615+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:23:03.917989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.1b","version":"196'1","reported_seq":15,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.840841+0000","last_change":"2024-02-21T23:12:25.814146+0000","last_active":"2024-02-21T23:12:27.840841+0000","last_peered":"2024-02-21T23:12:27.840841+0000","last_clean":"2024-02-21T23:12:27.840841+0000","last_became_active":"2024-02-21T23:12:25.813819+0000","last_became_peered":"2024-02-21T23:12:25.813819+0000","last_unstale":"2024-02-21T23:12:27.840841+0000","last_undegraded":"2024-02-21T23:12:27.840841+0000","last_fullsized":"2024-02-21T23:12:27.840841+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_clean_scrub_stamp":"2024-02-21T23:12:24.754357+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:22:51.087029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.25","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.839344+0000","last_change":"2024-02-21T23:12:26.840099+0000","last_active":"2024-02-21T23:12:27.839344+0000","last_peered":"2024-02-21T23:12:27.839344+0000","last_clean":"2024-02-21T23:12:27.839344+0000","last_became_active":"2024-02-21T23:12:26.839717+0000","last_became_peered":"2024-02-21T23:12:26.839717+0000","last_unstale":"2024-02-21T23:12:27.839344+0000","last_undegraded":"2024-02-21T23:12:27.839344+0000","last_fullsized":"2024-02-21T23:12:27.839344+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:26:12.656815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.1a","version":"196'1","reported_seq":15,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.948569+0000","last_change":"2024-02-21T23:12:25.809249+0000","last_active":"2024-02-21T23:12:27.948569+0000","last_peered":"2024-02-21T23:12:27.948569+0000","last_clean":"2024-02-21T23:12:27.948569+0000","last_became_active":"2024-02-21T23:12:25.808954+0000","last_became_peered":"2024-02-21T23:12:25.808954+0000","last_unstale":"2024-02-21T23:12:27.948569+0000","last_undegraded":"2024-02-21T23:12:27.948569+0000","last_fullsized":"2024-02-21T23:12:27.948569+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_clean_scrub_stamp":"2024-02-21T23:12:24.754357+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:02:24.973821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.26","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:28.278990+0000","last_change":"2024-02-21T23:12:26.836061+0000","last_active":"2024-02-21T23:12:28.278990+0000","last_peered":"2024-02-21T23:12:28.278990+0000","last_clean":"2024-02-21T23:12:28.278990+0000","last_became_active":"2024-02-21T23:12:26.835427+0000","last_became_peered":"2024-02-21T23:12:26.835427+0000","last_unstale":"2024-02-21T23:12:28.278990+0000","last_undegraded":"2024-02-21T23:12:28.278990+0000","last_fullsized":"2024-02-21T23:12:28.278990+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:16:24.977623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":14,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:28.278252+0000","last_change":"2024-02-21T23:12:25.819100+0000","last_active":"2024-02-21T23:12:28.278252+0000","last_peered":"2024-02-21T23:12:28.278252+0000","last_clean":"2024-02-21T23:12:28.278252+0000","last_became_active":"2024-02-21T23:12:25.818747+0000","last_became_peered":"2024-02-21T23:12:25.818747+0000","last_unstale":"2024-02-21T23:12:28.278252+0000","last_undegraded":"2024-02-21T23:12:28.278252+0000","last_fullsized":"2024-02-21T23:12:28.278252+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_clean_scrub_stamp":"2024-02-21T23:12:24.754357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:18:19.034287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.27","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:28.069628+0000","last_change":"2024-02-21T23:12:26.823356+0000","last_active":"2024-02-21T23:12:28.069628+0000","last_peered":"2024-02-21T23:12:28.069628+0000","last_clean":"2024-02-21T23:12:28.069628+0000","last_became_active":"2024-02-21T23:12:26.823071+0000","last_became_peered":"2024-02-21T23:12:26.823071+0000","last_unstale":"2024-02-21T23:12:28.069628+0000","last_undegraded":"2024-02-21T23:12:28.069628+0000","last_fullsized":"2024-02-21T23:12:28.069628+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:48:42.776013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":14,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:29.379617+0000","last_change":"2024-02-21T23:12:25.818025+0000","last_active":"2024-02-21T23:12:29.379617+0000","last_peered":"2024-02-21T23:12:29.379617+0000","last_clean":"2024-02-21T23:12:29.379617+0000","last_became_active":"2024-02-21T23:12:25.817398+0000","last_became_peered":"2024-02-21T23:12:25.817398+0000","last_unstale":"2024-02-21T23:12:29.379617+0000","last_undegraded":"2024-02-21T23:12:29.379617+0000","last_fullsized":"2024-02-21T23:12:29.379617+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_clean_scrub_stamp":"2024-02-21T23:12:24.754357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:44:47.202214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.28","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.838547+0000","last_change":"2024-02-21T23:12:26.837592+0000","last_active":"2024-02-21T23:12:27.838547+0000","last_peered":"2024-02-21T23:12:27.838547+0000","last_clean":"2024-02-21T23:12:27.838547+0000","last_became_active":"2024-02-21T23:12:26.837292+0000","last_became_peered":"2024-02-21T23:12:26.837292+0000","last_unstale":"2024-02-21T23:12:27.838547+0000","last_undegraded":"2024-02-21T23:12:27.838547+0000","last_fullsized":"2024-02-21T23:12:27.838547+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:48:42.776013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":14,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.949347+0000","last_change":"2024-02-21T23:12:25.819967+0000","last_active":"2024-02-21T23:12:27.949347+0000","last_peered":"2024-02-21T23:12:27.949347+0000","last_clean":"2024-02-21T23:12:27.949347+0000","last_became_active":"2024-02-21T23:12:25.819688+0000","last_became_peered":"2024-02-21T23:12:25.819688+0000","last_unstale":"2024-02-21T23:12:27.949347+0000","last_undegraded":"2024-02-21T23:12:27.949347+0000","last_fullsized":"2024-02-21T23:12:27.949347+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_clean_scrub_stamp":"2024-02-21T23:12:24.754357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:28:22.802030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.29","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:29.380204+0000","last_change":"2024-02-21T23:12:26.847975+0000","last_active":"2024-02-21T23:12:29.380204+0000","last_peered":"2024-02-21T23:12:29.380204+0000","last_clean":"2024-02-21T23:12:29.380204+0000","last_became_active":"2024-02-21T23:12:26.847564+0000","last_became_peered":"2024-02-21T23:12:26.847564+0000","last_unstale":"2024-02-21T23:12:29.380204+0000","last_undegraded":"2024-02-21T23:12:29.380204+0000","last_fullsized":"2024-02-21T23:12:29.380204+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:19:56.325276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":14,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:28.279321+0000","last_change":"2024-02-21T23:12:25.807532+0000","last_active":"2024-02-21T23:12:28.279321+0000","last_peered":"2024-02-21T23:12:28.279321+0000","last_clean":"2024-02-21T23:12:28.279321+0000","last_became_active":"2024-02-21T23:12:25.807238+0000","last_became_peered":"2024-02-21T23:12:25.807238+0000","last_unstale":"2024-02-21T23:12:28.279321+0000","last_undegraded":"2024-02-21T23:12:28.279321+0000","last_fullsized":"2024-02-21T23:12:28.279321+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_clean_scrub_stamp":"2024-02-21T23:12:24.754357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:59:14.985059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.2a","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:28.278971+0000","last_change":"2024-02-21T23:12:26.836694+0000","last_active":"2024-02-21T23:12:28.278971+0000","last_peered":"2024-02-21T23:12:28.278971+0000","last_clean":"2024-02-21T23:12:28.278971+0000","last_became_active":"2024-02-21T23:12:26.836415+0000","last_became_peered":"2024-02-21T23:12:26.836415+0000","last_unstale":"2024-02-21T23:12:28.278971+0000","last_undegraded":"2024-02-21T23:12:28.278971+0000","last_fullsized":"2024-02-21T23:12:28.278971+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:18:39.370028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"196'3","reported_seq":17,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:28.069102+0000","last_change":"2024-02-21T23:12:25.813698+0000","last_active":"2024-02-21T23:12:28.069102+0000","last_peered":"2024-02-21T23:12:28.069102+0000","last_clean":"2024-02-21T23:12:28.069102+0000","last_became_active":"2024-02-21T23:12:25.813397+0000","last_became_peered":"2024-02-21T23:12:25.813397+0000","last_unstale":"2024-02-21T23:12:28.069102+0000","last_undegraded":"2024-02-21T23:12:28.069102+0000","last_fullsized":"2024-02-21T23:12:28.069102+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_clean_scrub_stamp":"2024-02-21T23:12:24.754357+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:16:51.284956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2b","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.838403+0000","last_change":"2024-02-21T23:12:26.851712+0000","last_active":"2024-02-21T23:12:27.838403+0000","last_peered":"2024-02-21T23:12:27.838403+0000","last_clean":"2024-02-21T23:12:27.838403+0000","last_became_active":"2024-02-21T23:12:26.851408+0000","last_became_peered":"2024-02-21T23:12:26.851408+0000","last_unstale":"2024-02-21T23:12:27.838403+0000","last_undegraded":"2024-02-21T23:12:27.838403+0000","last_fullsized":"2024-02-21T23:12:27.838403+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:58:18.463148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.14","version":"196'6","reported_seq":20,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:29.380507+0000","last_change":"2024-02-21T23:12:25.815159+0000","last_active":"2024-02-21T23:12:29.380507+0000","last_peered":"2024-02-21T23:12:29.380507+0000","last_clean":"2024-02-21T23:12:29.380507+0000","last_became_active":"2024-02-21T23:12:25.814055+0000","last_became_peered":"2024-02-21T23:12:25.814055+0000","last_unstale":"2024-02-21T23:12:29.380507+0000","last_undegraded":"2024-02-21T23:12:29.380507+0000","last_fullsized":"2024-02-21T23:12:29.380507+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_clean_scrub_stamp":"2024-02-21T23:12:24.754357+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:47:40.186390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.2c","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.836799+0000","last_change":"2024-02-21T23:12:26.829222+0000","last_active":"2024-02-21T23:12:27.836799+0000","last_peered":"2024-02-21T23:12:27.836799+0000","last_clean":"2024-02-21T23:12:27.836799+0000","last_became_active":"2024-02-21T23:12:26.828860+0000","last_became_peered":"2024-02-21T23:12:26.828860+0000","last_unstale":"2024-02-21T23:12:27.836799+0000","last_undegraded":"2024-02-21T23:12:27.836799+0000","last_fullsized":"2024-02-21T23:12:27.836799+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:21:19.370427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.13","version":"196'2","reported_seq":16,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.841018+0000","last_change":"2024-02-21T23:12:25.817824+0000","last_active":"2024-02-21T23:12:27.841018+0000","last_peered":"2024-02-21T23:12:27.841018+0000","last_clean":"2024-02-21T23:12:27.841018+0000","last_became_active":"2024-02-21T23:12:25.817403+0000","last_became_peered":"2024-02-21T23:12:25.817403+0000","last_unstale":"2024-02-21T23:12:27.841018+0000","last_undegraded":"2024-02-21T23:12:27.841018+0000","last_fullsized":"2024-02-21T23:12:27.841018+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_clean_scrub_stamp":"2024-02-21T23:12:24.754357+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:18:19.034287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.2d","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.839175+0000","last_change":"2024-02-21T23:12:26.833962+0000","last_active":"2024-02-21T23:12:27.839175+0000","last_peered":"2024-02-21T23:12:27.839175+0000","last_clean":"2024-02-21T23:12:27.839175+0000","last_became_active":"2024-02-21T23:12:26.833516+0000","last_became_peered":"2024-02-21T23:12:26.833516+0000","last_unstale":"2024-02-21T23:12:27.839175+0000","last_undegraded":"2024-02-21T23:12:27.839175+0000","last_fullsized":"2024-02-21T23:12:27.839175+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:20:44.472226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":14,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:28.068617+0000","last_change":"2024-02-21T23:12:25.814231+0000","last_active":"2024-02-21T23:12:28.068617+0000","last_peered":"2024-02-21T23:12:28.068617+0000","last_clean":"2024-02-21T23:12:28.068617+0000","last_became_active":"2024-02-21T23:12:25.811379+0000","last_became_peered":"2024-02-21T23:12:25.811379+0000","last_unstale":"2024-02-21T23:12:28.068617+0000","last_undegraded":"2024-02-21T23:12:28.068617+0000","last_fullsized":"2024-02-21T23:12:28.068617+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_clean_scrub_stamp":"2024-02-21T23:12:24.754357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:57:19.951070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2e","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.840739+0000","last_change":"2024-02-21T23:12:26.839311+0000","last_active":"2024-02-21T23:12:27.840739+0000","last_peered":"2024-02-21T23:12:27.840739+0000","last_clean":"2024-02-21T23:12:27.840739+0000","last_became_active":"2024-02-21T23:12:26.839022+0000","last_became_peered":"2024-02-21T23:12:26.839022+0000","last_unstale":"2024-02-21T23:12:27.840739+0000","last_undegraded":"2024-02-21T23:12:27.840739+0000","last_fullsized":"2024-02-21T23:12:27.840739+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:48:54.474119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":14,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:28.068111+0000","last_change":"2024-02-21T23:12:25.807307+0000","last_active":"2024-02-21T23:12:28.068111+0000","last_peered":"2024-02-21T23:12:28.068111+0000","last_clean":"2024-02-21T23:12:28.068111+0000","last_became_active":"2024-02-21T23:12:25.806326+0000","last_became_peered":"2024-02-21T23:12:25.806326+0000","last_unstale":"2024-02-21T23:12:28.068111+0000","last_undegraded":"2024-02-21T23:12:28.068111+0000","last_fullsized":"2024-02-21T23:12:28.068111+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_clean_scrub_stamp":"2024-02-21T23:12:24.754357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:19:55.309062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.2f","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:28.069511+0000","last_change":"2024-02-21T23:12:26.830317+0000","last_active":"2024-02-21T23:12:28.069511+0000","last_peered":"2024-02-21T23:12:28.069511+0000","last_clean":"2024-02-21T23:12:28.069511+0000","last_became_active":"2024-02-21T23:12:26.829662+0000","last_became_peered":"2024-02-21T23:12:26.829662+0000","last_unstale":"2024-02-21T23:12:28.069511+0000","last_undegraded":"2024-02-21T23:12:28.069511+0000","last_fullsized":"2024-02-21T23:12:28.069511+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:19:48.049086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.10","version":"196'8","reported_seq":22,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.836065+0000","last_change":"2024-02-21T23:12:25.816885+0000","last_active":"2024-02-21T23:12:27.836065+0000","last_peered":"2024-02-21T23:12:27.836065+0000","last_clean":"2024-02-21T23:12:27.836065+0000","last_became_active":"2024-02-21T23:12:25.816529+0000","last_became_peered":"2024-02-21T23:12:25.816529+0000","last_unstale":"2024-02-21T23:12:27.836065+0000","last_undegraded":"2024-02-21T23:12:27.836065+0000","last_fullsized":"2024-02-21T23:12:27.836065+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_clean_scrub_stamp":"2024-02-21T23:12:24.754357+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:58:41.064952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.30","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:28.068407+0000","last_change":"2024-02-21T23:12:26.824451+0000","last_active":"2024-02-21T23:12:28.068407+0000","last_peered":"2024-02-21T23:12:28.068407+0000","last_clean":"2024-02-21T23:12:28.068407+0000","last_became_active":"2024-02-21T23:12:26.823912+0000","last_became_peered":"2024-02-21T23:12:26.823912+0000","last_unstale":"2024-02-21T23:12:28.068407+0000","last_undegraded":"2024-02-21T23:12:28.068407+0000","last_fullsized":"2024-02-21T23:12:28.068407+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:06:06.093085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.f","version":"196'1","reported_seq":15,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:29.380569+0000","last_change":"2024-02-21T23:12:25.812632+0000","last_active":"2024-02-21T23:12:29.380569+0000","last_peered":"2024-02-21T23:12:29.380569+0000","last_clean":"2024-02-21T23:12:29.380569+0000","last_became_active":"2024-02-21T23:12:25.812290+0000","last_became_peered":"2024-02-21T23:12:25.812290+0000","last_unstale":"2024-02-21T23:12:29.380569+0000","last_undegraded":"2024-02-21T23:12:29.380569+0000","last_fullsized":"2024-02-21T23:12:29.380569+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_clean_scrub_stamp":"2024-02-21T23:12:24.754357+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:06:40.524381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.31","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:28.068224+0000","last_change":"2024-02-21T23:12:26.837195+0000","last_active":"2024-02-21T23:12:28.068224+0000","last_peered":"2024-02-21T23:12:28.068224+0000","last_clean":"2024-02-21T23:12:28.068224+0000","last_became_active":"2024-02-21T23:12:26.836872+0000","last_became_peered":"2024-02-21T23:12:26.836872+0000","last_unstale":"2024-02-21T23:12:28.068224+0000","last_undegraded":"2024-02-21T23:12:28.068224+0000","last_fullsized":"2024-02-21T23:12:28.068224+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:59:16.001273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.e","version":"196'2","reported_seq":16,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.837274+0000","last_change":"2024-02-21T23:12:25.814010+0000","last_active":"2024-02-21T23:12:27.837274+0000","last_peered":"2024-02-21T23:12:27.837274+0000","last_clean":"2024-02-21T23:12:27.837274+0000","last_became_active":"2024-02-21T23:12:25.813703+0000","last_became_peered":"2024-02-21T23:12:25.813703+0000","last_unstale":"2024-02-21T23:12:27.837274+0000","last_undegraded":"2024-02-21T23:12:27.837274+0000","last_fullsized":"2024-02-21T23:12:27.837274+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_clean_scrub_stamp":"2024-02-21T23:12:24.754357+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:16:04.792023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.32","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.836279+0000","last_change":"2024-02-21T23:12:26.822821+0000","last_active":"2024-02-21T23:12:27.836279+0000","last_peered":"2024-02-21T23:12:27.836279+0000","last_clean":"2024-02-21T23:12:27.836279+0000","last_became_active":"2024-02-21T23:12:26.822495+0000","last_became_peered":"2024-02-21T23:12:26.822495+0000","last_unstale":"2024-02-21T23:12:27.836279+0000","last_undegraded":"2024-02-21T23:12:27.836279+0000","last_fullsized":"2024-02-21T23:12:27.836279+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:05:08.136818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.d","version":"196'1","reported_seq":15,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:29.380718+0000","last_change":"2024-02-21T23:12:25.813104+0000","last_active":"2024-02-21T23:12:29.380718+0000","last_peered":"2024-02-21T23:12:29.380718+0000","last_clean":"2024-02-21T23:12:29.380718+0000","last_became_active":"2024-02-21T23:12:25.811515+0000","last_became_peered":"2024-02-21T23:12:25.811515+0000","last_unstale":"2024-02-21T23:12:29.380718+0000","last_undegraded":"2024-02-21T23:12:29.380718+0000","last_fullsized":"2024-02-21T23:12:29.380718+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_clean_scrub_stamp":"2024-02-21T23:12:24.754357+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:04:37.124420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.33","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:28.278502+0000","last_change":"2024-02-21T23:12:26.836283+0000","last_active":"2024-02-21T23:12:28.278502+0000","last_peered":"2024-02-21T23:12:28.278502+0000","last_clean":"2024-02-21T23:12:28.278502+0000","last_became_active":"2024-02-21T23:12:26.835922+0000","last_became_peered":"2024-02-21T23:12:26.835922+0000","last_unstale":"2024-02-21T23:12:28.278502+0000","last_undegraded":"2024-02-21T23:12:28.278502+0000","last_fullsized":"2024-02-21T23:12:28.278502+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:35:41.902303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":14,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.841272+0000","last_change":"2024-02-21T23:12:25.814942+0000","last_active":"2024-02-21T23:12:27.841272+0000","last_peered":"2024-02-21T23:12:27.841272+0000","last_clean":"2024-02-21T23:12:27.841272+0000","last_became_active":"2024-02-21T23:12:25.814624+0000","last_became_peered":"2024-02-21T23:12:25.814624+0000","last_unstale":"2024-02-21T23:12:27.841272+0000","last_undegraded":"2024-02-21T23:12:27.841272+0000","last_fullsized":"2024-02-21T23:12:27.841272+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_clean_scrub_stamp":"2024-02-21T23:12:24.754357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:32:11.595345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.34","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:28.278501+0000","last_change":"2024-02-21T23:12:26.833912+0000","last_active":"2024-02-21T23:12:28.278501+0000","last_peered":"2024-02-21T23:12:28.278501+0000","last_clean":"2024-02-21T23:12:28.278501+0000","last_became_active":"2024-02-21T23:12:26.833359+0000","last_became_peered":"2024-02-21T23:12:26.833359+0000","last_unstale":"2024-02-21T23:12:28.278501+0000","last_undegraded":"2024-02-21T23:12:28.278501+0000","last_fullsized":"2024-02-21T23:12:28.278501+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:35:30.750766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.b","version":"196'2","reported_seq":16,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:28.278816+0000","last_change":"2024-02-21T23:12:25.806765+0000","last_active":"2024-02-21T23:12:28.278816+0000","last_peered":"2024-02-21T23:12:28.278816+0000","last_clean":"2024-02-21T23:12:28.278816+0000","last_became_active":"2024-02-21T23:12:25.806306+0000","last_became_peered":"2024-02-21T23:12:25.806306+0000","last_unstale":"2024-02-21T23:12:28.278816+0000","last_undegraded":"2024-02-21T23:12:28.278816+0000","last_fullsized":"2024-02-21T23:12:28.278816+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_clean_scrub_stamp":"2024-02-21T23:12:24.754357+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:06:05.076870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.35","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.836576+0000","last_change":"2024-02-21T23:12:26.845618+0000","last_active":"2024-02-21T23:12:27.836576+0000","last_peered":"2024-02-21T23:12:27.836576+0000","last_clean":"2024-02-21T23:12:27.836576+0000","last_became_active":"2024-02-21T23:12:26.844335+0000","last_became_peered":"2024-02-21T23:12:26.844335+0000","last_unstale":"2024-02-21T23:12:27.836576+0000","last_undegraded":"2024-02-21T23:12:27.836576+0000","last_fullsized":"2024-02-21T23:12:27.836576+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:10:52.688000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":14,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:29.380326+0000","last_change":"2024-02-21T23:12:25.809819+0000","last_active":"2024-02-21T23:12:29.380326+0000","last_peered":"2024-02-21T23:12:29.380326+0000","last_clean":"2024-02-21T23:12:29.380326+0000","last_became_active":"2024-02-21T23:12:25.809519+0000","last_became_peered":"2024-02-21T23:12:25.809519+0000","last_unstale":"2024-02-21T23:12:29.380326+0000","last_undegraded":"2024-02-21T23:12:29.380326+0000","last_fullsized":"2024-02-21T23:12:29.380326+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_clean_scrub_stamp":"2024-02-21T23:12:24.754357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:27:18.542244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.36","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:29.381271+0000","last_change":"2024-02-21T23:12:26.838228+0000","last_active":"2024-02-21T23:12:29.381271+0000","last_peered":"2024-02-21T23:12:29.381271+0000","last_clean":"2024-02-21T23:12:29.381271+0000","last_became_active":"2024-02-21T23:12:26.837439+0000","last_became_peered":"2024-02-21T23:12:26.837439+0000","last_unstale":"2024-02-21T23:12:29.381271+0000","last_undegraded":"2024-02-21T23:12:29.381271+0000","last_fullsized":"2024-02-21T23:12:29.381271+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:10:15.570938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":14,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:29.380347+0000","last_change":"2024-02-21T23:12:25.811564+0000","last_active":"2024-02-21T23:12:29.380347+0000","last_peered":"2024-02-21T23:12:29.380347+0000","last_clean":"2024-02-21T23:12:29.380347+0000","last_became_active":"2024-02-21T23:12:25.811244+0000","last_became_peered":"2024-02-21T23:12:25.811244+0000","last_unstale":"2024-02-21T23:12:29.380347+0000","last_undegraded":"2024-02-21T23:12:29.380347+0000","last_fullsized":"2024-02-21T23:12:29.380347+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_clean_scrub_stamp":"2024-02-21T23:12:24.754357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:32:06.525620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.37","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:29.381134+0000","last_change":"2024-02-21T23:12:26.838881+0000","last_active":"2024-02-21T23:12:29.381134+0000","last_peered":"2024-02-21T23:12:29.381134+0000","last_clean":"2024-02-21T23:12:29.381134+0000","last_became_active":"2024-02-21T23:12:26.838574+0000","last_became_peered":"2024-02-21T23:12:26.838574+0000","last_unstale":"2024-02-21T23:12:29.381134+0000","last_undegraded":"2024-02-21T23:12:29.381134+0000","last_fullsized":"2024-02-21T23:12:29.381134+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:57:20.967285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":14,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.948002+0000","last_change":"2024-02-21T23:12:25.811248+0000","last_active":"2024-02-21T23:12:27.948002+0000","last_peered":"2024-02-21T23:12:27.948002+0000","last_clean":"2024-02-21T23:12:27.948002+0000","last_became_active":"2024-02-21T23:12:25.807780+0000","last_became_peered":"2024-02-21T23:12:25.807780+0000","last_unstale":"2024-02-21T23:12:27.948002+0000","last_undegraded":"2024-02-21T23:12:27.948002+0000","last_fullsized":"2024-02-21T23:12:27.948002+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_clean_scrub_stamp":"2024-02-21T23:12:24.754357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:30:19.216169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.38","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.838452+0000","last_change":"2024-02-21T23:12:26.828531+0000","last_active":"2024-02-21T23:12:27.838452+0000","last_peered":"2024-02-21T23:12:27.838452+0000","last_clean":"2024-02-21T23:12:27.838452+0000","last_became_active":"2024-02-21T23:12:26.828208+0000","last_became_peered":"2024-02-21T23:12:26.828208+0000","last_unstale":"2024-02-21T23:12:27.838452+0000","last_undegraded":"2024-02-21T23:12:27.838452+0000","last_fullsized":"2024-02-21T23:12:27.838452+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:08:54.861131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.7","version":"196'2","reported_seq":16,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:28.279189+0000","last_change":"2024-02-21T23:12:25.812189+0000","last_active":"2024-02-21T23:12:28.279189+0000","last_peered":"2024-02-21T23:12:28.279189+0000","last_clean":"2024-02-21T23:12:28.279189+0000","last_became_active":"2024-02-21T23:12:25.810343+0000","last_became_peered":"2024-02-21T23:12:25.810343+0000","last_unstale":"2024-02-21T23:12:28.279189+0000","last_undegraded":"2024-02-21T23:12:28.279189+0000","last_fullsized":"2024-02-21T23:12:28.279189+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_clean_scrub_stamp":"2024-02-21T23:12:24.754357+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:49:12.073674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.39","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.840010+0000","last_change":"2024-02-21T23:12:26.848953+0000","last_active":"2024-02-21T23:12:27.840010+0000","last_peered":"2024-02-21T23:12:27.840010+0000","last_clean":"2024-02-21T23:12:27.840010+0000","last_became_active":"2024-02-21T23:12:26.848631+0000","last_became_peered":"2024-02-21T23:12:26.848631+0000","last_unstale":"2024-02-21T23:12:27.840010+0000","last_undegraded":"2024-02-21T23:12:27.840010+0000","last_fullsized":"2024-02-21T23:12:27.840010+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:19:48.049086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.6","version":"196'1","reported_seq":15,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:28.069388+0000","last_change":"2024-02-21T23:12:25.805846+0000","last_active":"2024-02-21T23:12:28.069388+0000","last_peered":"2024-02-21T23:12:28.069388+0000","last_clean":"2024-02-21T23:12:28.069388+0000","last_became_active":"2024-02-21T23:12:25.805072+0000","last_became_peered":"2024-02-21T23:12:25.805072+0000","last_unstale":"2024-02-21T23:12:28.069388+0000","last_undegraded":"2024-02-21T23:12:28.069388+0000","last_fullsized":"2024-02-21T23:12:28.069388+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_clean_scrub_stamp":"2024-02-21T23:12:24.754357+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:05:32.231407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3a","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:28.278467+0000","last_change":"2024-02-21T23:12:26.840139+0000","last_active":"2024-02-21T23:12:28.278467+0000","last_peered":"2024-02-21T23:12:28.278467+0000","last_clean":"2024-02-21T23:12:28.278467+0000","last_became_active":"2024-02-21T23:12:26.839777+0000","last_became_peered":"2024-02-21T23:12:26.839777+0000","last_unstale":"2024-02-21T23:12:28.278467+0000","last_undegraded":"2024-02-21T23:12:28.278467+0000","last_fullsized":"2024-02-21T23:12:28.278467+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:35:54.372949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":14,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.948870+0000","last_change":"2024-02-21T23:12:25.814567+0000","last_active":"2024-02-21T23:12:27.948870+0000","last_peered":"2024-02-21T23:12:27.948870+0000","last_clean":"2024-02-21T23:12:27.948870+0000","last_became_active":"2024-02-21T23:12:25.814284+0000","last_became_peered":"2024-02-21T23:12:25.814284+0000","last_unstale":"2024-02-21T23:12:27.948870+0000","last_undegraded":"2024-02-21T23:12:27.948870+0000","last_fullsized":"2024-02-21T23:12:27.948870+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_clean_scrub_stamp":"2024-02-21T23:12:24.754357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:56:23.887648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3b","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:28.068737+0000","last_change":"2024-02-21T23:12:26.827404+0000","last_active":"2024-02-21T23:12:28.068737+0000","last_peered":"2024-02-21T23:12:28.068737+0000","last_clean":"2024-02-21T23:12:28.068737+0000","last_became_active":"2024-02-21T23:12:26.827135+0000","last_became_peered":"2024-02-21T23:12:26.827135+0000","last_unstale":"2024-02-21T23:12:28.068737+0000","last_undegraded":"2024-02-21T23:12:28.068737+0000","last_fullsized":"2024-02-21T23:12:28.068737+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:14:09.789181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":14,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.948917+0000","last_change":"2024-02-21T23:12:25.812483+0000","last_active":"2024-02-21T23:12:27.948917+0000","last_peered":"2024-02-21T23:12:27.948917+0000","last_clean":"2024-02-21T23:12:27.948917+0000","last_became_active":"2024-02-21T23:12:25.812211+0000","last_became_peered":"2024-02-21T23:12:25.812211+0000","last_unstale":"2024-02-21T23:12:27.948917+0000","last_undegraded":"2024-02-21T23:12:27.948917+0000","last_fullsized":"2024-02-21T23:12:27.948917+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_clean_scrub_stamp":"2024-02-21T23:12:24.754357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:35:29.734551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.3c","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.948538+0000","last_change":"2024-02-21T23:12:26.829974+0000","last_active":"2024-02-21T23:12:27.948538+0000","last_peered":"2024-02-21T23:12:27.948538+0000","last_clean":"2024-02-21T23:12:27.948538+0000","last_became_active":"2024-02-21T23:12:26.829656+0000","last_became_peered":"2024-02-21T23:12:26.829656+0000","last_unstale":"2024-02-21T23:12:27.948538+0000","last_undegraded":"2024-02-21T23:12:27.948538+0000","last_fullsized":"2024-02-21T23:12:27.948538+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:51:59.978445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":14,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:29.380565+0000","last_change":"2024-02-21T23:12:25.815800+0000","last_active":"2024-02-21T23:12:29.380565+0000","last_peered":"2024-02-21T23:12:29.380565+0000","last_clean":"2024-02-21T23:12:29.380565+0000","last_became_active":"2024-02-21T23:12:25.815477+0000","last_became_peered":"2024-02-21T23:12:25.815477+0000","last_unstale":"2024-02-21T23:12:29.380565+0000","last_undegraded":"2024-02-21T23:12:29.380565+0000","last_fullsized":"2024-02-21T23:12:29.380565+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_clean_scrub_stamp":"2024-02-21T23:12:24.754357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:02:49.067243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.3d","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:29.380445+0000","last_change":"2024-02-21T23:12:26.855505+0000","last_active":"2024-02-21T23:12:29.380445+0000","last_peered":"2024-02-21T23:12:29.380445+0000","last_clean":"2024-02-21T23:12:29.380445+0000","last_became_active":"2024-02-21T23:12:26.854883+0000","last_became_peered":"2024-02-21T23:12:26.854883+0000","last_unstale":"2024-02-21T23:12:29.380445+0000","last_undegraded":"2024-02-21T23:12:29.380445+0000","last_fullsized":"2024-02-21T23:12:29.380445+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:12:03.094082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":14,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:28.068677+0000","last_change":"2024-02-21T23:12:25.805995+0000","last_active":"2024-02-21T23:12:28.068677+0000","last_peered":"2024-02-21T23:12:28.068677+0000","last_clean":"2024-02-21T23:12:28.068677+0000","last_became_active":"2024-02-21T23:12:25.805645+0000","last_became_peered":"2024-02-21T23:12:25.805645+0000","last_unstale":"2024-02-21T23:12:28.068677+0000","last_undegraded":"2024-02-21T23:12:28.068677+0000","last_fullsized":"2024-02-21T23:12:28.068677+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_clean_scrub_stamp":"2024-02-21T23:12:24.754357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:53:07.829677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.3e","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.837164+0000","last_change":"2024-02-21T23:12:26.845823+0000","last_active":"2024-02-21T23:12:27.837164+0000","last_peered":"2024-02-21T23:12:27.837164+0000","last_clean":"2024-02-21T23:12:27.837164+0000","last_became_active":"2024-02-21T23:12:26.845537+0000","last_became_peered":"2024-02-21T23:12:26.845537+0000","last_unstale":"2024-02-21T23:12:27.837164+0000","last_undegraded":"2024-02-21T23:12:27.837164+0000","last_fullsized":"2024-02-21T23:12:27.837164+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:37:46.454268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":14,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:28.278556+0000","last_change":"2024-02-21T23:12:25.807027+0000","last_active":"2024-02-21T23:12:28.278556+0000","last_peered":"2024-02-21T23:12:28.278556+0000","last_clean":"2024-02-21T23:12:28.278556+0000","last_became_active":"2024-02-21T23:12:25.806737+0000","last_became_peered":"2024-02-21T23:12:25.806737+0000","last_unstale":"2024-02-21T23:12:28.278556+0000","last_undegraded":"2024-02-21T23:12:28.278556+0000","last_fullsized":"2024-02-21T23:12:28.278556+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_clean_scrub_stamp":"2024-02-21T23:12:24.754357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:38:51.815687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.3f","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:29.381089+0000","last_change":"2024-02-21T23:12:26.851964+0000","last_active":"2024-02-21T23:12:29.381089+0000","last_peered":"2024-02-21T23:12:29.381089+0000","last_clean":"2024-02-21T23:12:29.381089+0000","last_became_active":"2024-02-21T23:12:26.851609+0000","last_became_peered":"2024-02-21T23:12:26.851609+0000","last_unstale":"2024-02-21T23:12:29.381089+0000","last_undegraded":"2024-02-21T23:12:29.381089+0000","last_fullsized":"2024-02-21T23:12:29.381089+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:53:08.845892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":14,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:29.379911+0000","last_change":"2024-02-21T23:12:25.801596+0000","last_active":"2024-02-21T23:12:29.379911+0000","last_peered":"2024-02-21T23:12:29.379911+0000","last_clean":"2024-02-21T23:12:29.379911+0000","last_became_active":"2024-02-21T23:12:25.801215+0000","last_became_peered":"2024-02-21T23:12:25.801215+0000","last_unstale":"2024-02-21T23:12:29.379911+0000","last_undegraded":"2024-02-21T23:12:29.379911+0000","last_fullsized":"2024-02-21T23:12:29.379911+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_clean_scrub_stamp":"2024-02-21T23:12:24.754357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:18:03.014262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.21","version":"0'0","reported_seq":12,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:29.379759+0000","last_change":"2024-02-21T23:12:26.850189+0000","last_active":"2024-02-21T23:12:29.379759+0000","last_peered":"2024-02-21T23:12:29.379759+0000","last_clean":"2024-02-21T23:12:29.379759+0000","last_became_active":"2024-02-21T23:12:26.849907+0000","last_became_peered":"2024-02-21T23:12:26.849907+0000","last_unstale":"2024-02-21T23:12:29.379759+0000","last_undegraded":"2024-02-21T23:12:29.379759+0000","last_fullsized":"2024-02-21T23:12:29.379759+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":195,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:25.770572+0000","last_clean_scrub_stamp":"2024-02-21T23:12:25.770572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:54:05.753523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1e","version":"196'1","reported_seq":15,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.840251+0000","last_change":"2024-02-21T23:12:25.809712+0000","last_active":"2024-02-21T23:12:27.840251+0000","last_peered":"2024-02-21T23:12:27.840251+0000","last_clean":"2024-02-21T23:12:27.840251+0000","last_became_active":"2024-02-21T23:12:25.809422+0000","last_became_peered":"2024-02-21T23:12:25.809422+0000","last_unstale":"2024-02-21T23:12:27.840251+0000","last_undegraded":"2024-02-21T23:12:27.840251+0000","last_fullsized":"2024-02-21T23:12:27.840251+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":194,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:12:24.754357+0000","last_clean_scrub_stamp":"2024-02-21T23:12:24.754357+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:16:49.302319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":499,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-21T23:12:27.840007+0000","last_change":"2024-02-21T23:02:32.289335+0000","last_active":"2024-02-21T23:12:27.840007+0000","last_peered":"2024-02-21T23:12:27.840007+0000","last_clean":"2024-02-21T23:12:27.840007+0000","last_became_active":"2024-02-21T23:02:32.286913+0000","last_became_peered":"2024-02-21T23:02:32.286913+0000","last_unstale":"2024-02-21T23:12:27.840007+0000","last_undegraded":"2024-02-21T23:12:27.840007+0000","last_fullsized":"2024-02-21T23:12:27.840007+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_clean_scrub_stamp":"2024-02-21T23:02:31.172569+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:36:23.987374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":32,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":31,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574970,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33696,"kb_used_data":2512,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338144,"statfs":{"total":96636764160,"available":96602259456,"internally_reserved":0,"allocated":2572288,"data_stored":1817625,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574971,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33628,"kb_used_data":2448,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338212,"statfs":{"total":96636764160,"available":96602329088,"internally_reserved":0,"allocated":2506752,"data_stored":1227245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574970,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33620,"kb_used_data":2440,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338220,"statfs":{"total":96636764160,"available":96602337280,"internally_reserved":0,"allocated":2498560,"data_stored":1227133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574971,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33624,"kb_used_data":2444,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338216,"statfs":{"total":96636764160,"available":96602333184,"internally_reserved":0,"allocated":2502656,"data_stored":1227203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574971,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37808,"kb_used_data":2464,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334032,"statfs":{"total":96636764160,"available":96598048768,"internally_reserved":0,"allocated":2523136,"data_stored":1229383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574972,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33632,"kb_used_data":2448,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338208,"statfs":{"total":96636764160,"available":96602324992,"internally_reserved":0,"allocated":2506752,"data_stored":1227785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574970,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33624,"kb_used_data":2440,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338216,"statfs":{"total":96636764160,"available":96602333184,"internally_reserved":0,"allocated":2498560,"data_stored":1227133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574970,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37860,"kb_used_data":2516,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333980,"statfs":{"total":96636764160,"available":96597995520,"internally_reserved":0,"allocated":2576384,"data_stored":1819113,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-21T23:12:31.827 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-02-21T23:12:31.827 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-02-21T23:12:32.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.113+0000 7f7a19000640 1 Processor -- start 2024-02-21T23:12:32.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.113+0000 7f7a19000640 1 -- start start 2024-02-21T23:12:32.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.114+0000 7f7a19000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7a14152370 0x7f7a14155940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:32.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.114+0000 7f7a19000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f7a140c5110 0x7f7a140cab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:32.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.114+0000 7f7a19000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a14154ea0 con 0x7f7a140bde20 2024-02-21T23:12:32.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.114+0000 7f7a19000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a141554b0 con 0x7f7a14152370 2024-02-21T23:12:32.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.114+0000 7f7a19000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a140c5690 con 0x7f7a140c5110 2024-02-21T23:12:32.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.115+0000 7f7a12a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f7a140bde20 0x7f7a140c6f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:32942/0 (socket says 172.21.15.28:32942) 2024-02-21T23:12:32.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.115+0000 7f7a12a00640 1 -- 172.21.15.28:0/2370660162 learned_addr learned my addr 172.21.15.28:0/2370660162 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:12:32.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.115+0000 7f7a13400640 1 --2- 172.21.15.28:0/2370660162 >> v2:172.21.15.28:3300/0 conn(0x7f7a14152370 0x7f7a14155940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:32.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.115+0000 7f7a13e00640 1 --2- 172.21.15.28:0/2370660162 >> v2:172.21.15.81:3300/0 conn(0x7f7a140c5110 0x7f7a140cab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:32.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.115+0000 7f7a13400640 1 -- 172.21.15.28:0/2370660162 >> v2:172.21.15.81:3300/0 conn(0x7f7a140c5110 msgr2=0x7f7a140cab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:32.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.115+0000 7f7a13400640 1 --2- 172.21.15.28:0/2370660162 >> v2:172.21.15.81:3300/0 conn(0x7f7a140c5110 0x7f7a140cab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:32.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.115+0000 7f7a13400640 1 -- 172.21.15.28:0/2370660162 >> v1:172.21.15.28:6789/0 conn(0x7f7a140bde20 legacy=0x7f7a140c6f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:32.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.115+0000 7f7a13400640 1 -- 172.21.15.28:0/2370660162 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a14155140 con 0x7f7a14152370 2024-02-21T23:12:32.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.115+0000 7f7a13e00640 1 --2- 172.21.15.28:0/2370660162 >> v2:172.21.15.81:3300/0 conn(0x7f7a140c5110 0x7f7a140cab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:12:32.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.115+0000 7f7a12000640 1 -- 172.21.15.28:0/2370660162 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1670401534 0 0) 0x7f7a14154ea0 con 0x7f7a140bde20 2024-02-21T23:12:32.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.117+0000 7f7a13400640 1 --2- 172.21.15.28:0/2370660162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a14152370 0x7f7a14155940 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 crypto rx=0x7f7a14155630 tx=0x7f7a0402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea87a643b54e5cb7 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:32.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.117+0000 7f7a12000640 1 -- 172.21.15.28:0/2370660162 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a040315a0 con 0x7f7a14152370 2024-02-21T23:12:32.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.117+0000 7f7a12000640 1 -- 172.21.15.28:0/2370660162 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a04037070 con 0x7f7a14152370 2024-02-21T23:12:32.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.117+0000 7f7a12000640 1 -- 172.21.15.28:0/2370660162 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a04031960 con 0x7f7a14152370 2024-02-21T23:12:32.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.118+0000 7f7a19000640 1 -- 172.21.15.28:0/2370660162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a14152370 msgr2=0x7f7a14155940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:32.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.118+0000 7f7a19000640 1 --2- 172.21.15.28:0/2370660162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a14152370 0x7f7a14155940 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 crypto rx=0x7f7a14155630 tx=0x7f7a0402b840 comp rx=0 tx=0).stop 2024-02-21T23:12:32.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.118+0000 7f7a19000640 1 -- 172.21.15.28:0/2370660162 shutdown_connections 2024-02-21T23:12:32.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.118+0000 7f7a19000640 1 --2- 172.21.15.28:0/2370660162 >> v2:172.21.15.81:3300/0 conn(0x7f7a140c5110 0x7f7a140cab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:32.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.118+0000 7f7a19000640 1 --2- 172.21.15.28:0/2370660162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a14152370 0x7f7a14155940 unknown :-1 s=CLOSED pgs=2251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:32.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.118+0000 7f7a19000640 1 -- 172.21.15.28:0/2370660162 >> 172.21.15.28:0/2370660162 conn(0x7f7a14072780 msgr2=0x7f7a14072b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:32.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.118+0000 7f7a19000640 1 -- 172.21.15.28:0/2370660162 shutdown_connections 2024-02-21T23:12:32.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.118+0000 7f7a19000640 1 -- 172.21.15.28:0/2370660162 wait complete. 2024-02-21T23:12:32.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.119+0000 7f7a19000640 1 Processor -- start 2024-02-21T23:12:32.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.120+0000 7f7a19000640 1 -- start start 2024-02-21T23:12:32.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.121+0000 7f7a19000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a140bde20 0x7f7a141490f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:32.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.121+0000 7f7a19000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7a140c5110 0x7f7a141edbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:32.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.121+0000 7f7a19000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a14152370 0x7f7a140aa140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:32.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.121+0000 7f7a19000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1406dd00 con 0x7f7a140bde20 2024-02-21T23:12:32.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.121+0000 7f7a19000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a140c7670 con 0x7f7a14152370 2024-02-21T23:12:32.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.121+0000 7f7a19000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a140c5950 con 0x7f7a140c5110 2024-02-21T23:12:32.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.121+0000 7f7a13400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a140bde20 0x7f7a141490f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:32.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.121+0000 7f7a13400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a140bde20 0x7f7a141490f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44408/0 (socket says 172.21.15.28:44408) 2024-02-21T23:12:32.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.121+0000 7f7a13400640 1 -- 172.21.15.28:0/2028223491 learned_addr learned my addr 172.21.15.28:0/2028223491 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:32.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.121+0000 7f7a12a00640 1 --2- 172.21.15.28:0/2028223491 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7a140c5110 0x7f7a141edbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:32.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.121+0000 7f7a13e00640 1 --2- 172.21.15.28:0/2028223491 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a14152370 0x7f7a140aa140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:32.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.121+0000 7f7a13400640 1 -- 172.21.15.28:0/2028223491 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7a140c5110 msgr2=0x7f7a141edbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:32.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.121+0000 7f7a13400640 1 --2- 172.21.15.28:0/2028223491 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7a140c5110 0x7f7a141edbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:32.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.121+0000 7f7a13400640 1 -- 172.21.15.28:0/2028223491 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a14152370 msgr2=0x7f7a140aa140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:32.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.121+0000 7f7a13400640 1 --2- 172.21.15.28:0/2028223491 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a14152370 0x7f7a140aa140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:32.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.121+0000 7f7a13400640 1 -- 172.21.15.28:0/2028223491 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a140bf870 con 0x7f7a140bde20 2024-02-21T23:12:32.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.121+0000 7f7a12a00640 1 --2- 172.21.15.28:0/2028223491 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7a140c5110 0x7f7a141edbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:12:32.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.121+0000 7f7a13e00640 1 --2- 172.21.15.28:0/2028223491 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a14152370 0x7f7a140aa140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:12:32.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.122+0000 7f7a13400640 1 --2- 172.21.15.28:0/2028223491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a140bde20 0x7f7a141490f0 secure :-1 s=READY pgs=2252 cs=0 l=1 rev1=1 crypto rx=0x7f7a04000c00 tx=0x7f7a040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:32.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.122+0000 7f7a0b400640 1 -- 172.21.15.28:0/2028223491 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a04032840 con 0x7f7a140bde20 2024-02-21T23:12:32.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.122+0000 7f7a0b400640 1 -- 172.21.15.28:0/2028223491 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a04032a00 con 0x7f7a140bde20 2024-02-21T23:12:32.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.122+0000 7f7a0b400640 1 -- 172.21.15.28:0/2028223491 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a04032df0 con 0x7f7a140bde20 2024-02-21T23:12:32.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.122+0000 7f7a19000640 1 -- 172.21.15.28:0/2028223491 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a140c0730 con 0x7f7a140bde20 2024-02-21T23:12:32.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.122+0000 7f7a19000640 1 -- 172.21.15.28:0/2028223491 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a140be2c0 con 0x7f7a140bde20 2024-02-21T23:12:32.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.123+0000 7f7a19000640 1 -- 172.21.15.28:0/2028223491 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a1404f290 con 0x7f7a140bde20 2024-02-21T23:12:32.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.124+0000 7f7a0b400640 1 -- 172.21.15.28:0/2028223491 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7a0404d3f0 con 0x7f7a140bde20 2024-02-21T23:12:32.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.125+0000 7f7a0b400640 1 --2- 172.21.15.28:0/2028223491 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f79f006b4d0 0x7f79f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:32.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.125+0000 7f7a0b400640 1 -- 172.21.15.28:0/2028223491 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f7a0402f030 con 0x7f7a140bde20 2024-02-21T23:12:32.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.126+0000 7f7a12a00640 1 --2- 172.21.15.28:0/2028223491 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f79f006b4d0 0x7f79f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:32.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.127+0000 7f7a12a00640 1 --2- 172.21.15.28:0/2028223491 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f79f006b4d0 0x7f79f006d990 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f7a141eb780 tx=0x7f79fc009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:32.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.130+0000 7f7a0b400640 1 -- 172.21.15.28:0/2028223491 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7a04040070 con 0x7f7a140bde20 2024-02-21T23:12:32.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.404+0000 7f7a19000640 1 -- 172.21.15.28:0/2028223491 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f7a140c9c70 con 0x7f7a140bde20 2024-02-21T23:12:32.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.409+0000 7f7a0b400640 1 -- 172.21.15.28:0/2028223491 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v22) v1 ==== 190+0+147 (secure 0 0 0) 0x7f7a0407aef0 con 0x7f7a140bde20 2024-02-21T23:12:32.398 INFO:teuthology.orchestra.run.smithi028.stdout:[client.0] 2024-02-21T23:12:32.398 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQDTgNZlILvYLxAAoREWxKKT7YcXnhkgk8YvoQ== 2024-02-21T23:12:32.398 INFO:teuthology.orchestra.run.smithi028.stdout: caps mds = "allow" 2024-02-21T23:12:32.399 INFO:teuthology.orchestra.run.smithi028.stdout: caps mon = "allow r" 2024-02-21T23:12:32.399 INFO:teuthology.orchestra.run.smithi028.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-02-21T23:12:32.399 INFO:teuthology.orchestra.run.smithi028.stderr:updated caps for client.0 2024-02-21T23:12:32.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.412+0000 7f7a19000640 1 -- 172.21.15.28:0/2028223491 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f79f006b4d0 msgr2=0x7f79f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:32.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.412+0000 7f7a19000640 1 --2- 172.21.15.28:0/2028223491 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f79f006b4d0 0x7f79f006d990 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f7a141eb780 tx=0x7f79fc009040 comp rx=0 tx=0).stop 2024-02-21T23:12:32.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.412+0000 7f7a19000640 1 -- 172.21.15.28:0/2028223491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a140bde20 msgr2=0x7f7a141490f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:32.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.412+0000 7f7a19000640 1 --2- 172.21.15.28:0/2028223491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a140bde20 0x7f7a141490f0 secure :-1 s=READY pgs=2252 cs=0 l=1 rev1=1 crypto rx=0x7f7a04000c00 tx=0x7f7a040046c0 comp rx=0 tx=0).stop 2024-02-21T23:12:32.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.412+0000 7f7a19000640 1 -- 172.21.15.28:0/2028223491 shutdown_connections 2024-02-21T23:12:32.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.412+0000 7f7a19000640 1 --2- 172.21.15.28:0/2028223491 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f79f006b4d0 0x7f79f006d990 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:32.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.412+0000 7f7a19000640 1 --2- 172.21.15.28:0/2028223491 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7a140c5110 0x7f7a141edbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:32.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.412+0000 7f7a19000640 1 --2- 172.21.15.28:0/2028223491 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a14152370 0x7f7a140aa140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:32.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.412+0000 7f7a19000640 1 --2- 172.21.15.28:0/2028223491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a140bde20 0x7f7a141490f0 unknown :-1 s=CLOSED pgs=2252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:32.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.412+0000 7f7a19000640 1 -- 172.21.15.28:0/2028223491 >> 172.21.15.28:0/2028223491 conn(0x7f7a14072780 msgr2=0x7f7a140cab00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:32.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.412+0000 7f7a19000640 1 -- 172.21.15.28:0/2028223491 shutdown_connections 2024-02-21T23:12:32.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.412+0000 7f7a19000640 1 -- 172.21.15.28:0/2028223491 wait complete. 2024-02-21T23:12:32.421 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:12:32.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.651+0000 7f31c1600640 1 Processor -- start 2024-02-21T23:12:32.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.651+0000 7f31c1600640 1 -- start start 2024-02-21T23:12:32.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.652+0000 7f31c1600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f31bc0c5510 0x7f31bc0a7e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:32.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.652+0000 7f31c1600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f31bc0c7320 0x7f31bc1521d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:32.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.652+0000 7f31c1600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31bc0c8040 con 0x7f31bc152710 2024-02-21T23:12:32.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.652+0000 7f31c1600640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31bc0a8380 con 0x7f31bc0c5510 2024-02-21T23:12:32.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.652+0000 7f31c1600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31bc0c85c0 con 0x7f31bc0c7320 2024-02-21T23:12:32.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.653+0000 7f31bb400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f31bc152710 0x7f31bc155040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:32950/0 (socket says 172.21.15.28:32950) 2024-02-21T23:12:32.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.653+0000 7f31bb400640 1 -- 172.21.15.28:0/1026268366 learned_addr learned my addr 172.21.15.28:0/1026268366 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:12:32.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.653+0000 7f31ba000640 1 --2- 172.21.15.28:0/1026268366 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f31bc0c7320 0x7f31bc1521d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:32.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.653+0000 7f31baa00640 1 --2- 172.21.15.28:0/1026268366 >> v2:172.21.15.81:3300/0 conn(0x7f31bc0c5510 0x7f31bc0a7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:32.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.653+0000 7f31b9600640 1 -- 172.21.15.28:0/1026268366 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1285436641 0 0) 0x7f31bc0c8040 con 0x7f31bc152710 2024-02-21T23:12:32.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.653+0000 7f31b9600640 1 -- 172.21.15.28:0/1026268366 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31a0003520 con 0x7f31bc152710 2024-02-21T23:12:32.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.653+0000 7f31b9600640 1 -- 172.21.15.28:0/1026268366 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3797418240 0 0) 0x7f31a0003520 con 0x7f31bc152710 2024-02-21T23:12:32.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.653+0000 7f31b9600640 1 -- 172.21.15.28:0/1026268366 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f31bc0c7320 msgr2=0x7f31bc1521d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:32.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.654+0000 7f31b9600640 1 --2- 172.21.15.28:0/1026268366 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f31bc0c7320 0x7f31bc1521d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:32.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.654+0000 7f31b9600640 1 -- 172.21.15.28:0/1026268366 >> v2:172.21.15.81:3300/0 conn(0x7f31bc0c5510 msgr2=0x7f31bc0a7e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:32.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.654+0000 7f31b9600640 1 --2- 172.21.15.28:0/1026268366 >> v2:172.21.15.81:3300/0 conn(0x7f31bc0c5510 0x7f31bc0a7e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:32.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.654+0000 7f31b9600640 1 -- 172.21.15.28:0/1026268366 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31bc0c82e0 con 0x7f31bc152710 2024-02-21T23:12:32.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.654+0000 7f31b9600640 1 -- 172.21.15.28:0/1026268366 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f31b4001670 con 0x7f31bc152710 2024-02-21T23:12:32.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.654+0000 7f31b9600640 1 -- 172.21.15.28:0/1026268366 >> v1:172.21.15.28:6789/0 conn(0x7f31bc152710 legacy=0x7f31bc155040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:32.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.654+0000 7f31b9600640 1 --2- 172.21.15.28:0/1026268366 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f31a00039b0 0x7f31a0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:32.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.654+0000 7f31b9600640 1 --2- 172.21.15.28:0/1026268366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31a0004800 0x7f31a0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:32.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.654+0000 7f31b9600640 1 --2- 172.21.15.28:0/1026268366 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f31bc0c7320 0x7f31a00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:32.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.654+0000 7f31b9600640 1 -- 172.21.15.28:0/1026268366 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31bc0a8380 con 0x7f31a0004800 2024-02-21T23:12:32.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.654+0000 7f31b9600640 1 -- 172.21.15.28:0/1026268366 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31bc0c85c0 con 0x7f31a00039b0 2024-02-21T23:12:32.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.654+0000 7f31b9600640 1 -- 172.21.15.28:0/1026268366 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31a000b9c0 con 0x7f31bc0c7320 2024-02-21T23:12:32.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.654+0000 7f31ba000640 1 --2- 172.21.15.28:0/1026268366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31a0004800 0x7f31a0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:32.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.654+0000 7f31bb400640 1 --2- 172.21.15.28:0/1026268366 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f31bc0c7320 0x7f31a00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:32.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.654+0000 7f31baa00640 1 --2- 172.21.15.28:0/1026268366 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f31a00039b0 0x7f31a0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:32.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.654+0000 7f31ba000640 1 -- 172.21.15.28:0/1026268366 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f31bc0c7320 msgr2=0x7f31a00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:32.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.654+0000 7f31ba000640 1 --2- 172.21.15.28:0/1026268366 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f31bc0c7320 0x7f31a00051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:32.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.654+0000 7f31ba000640 1 -- 172.21.15.28:0/1026268366 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f31a00039b0 msgr2=0x7f31a0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:32.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.654+0000 7f31ba000640 1 --2- 172.21.15.28:0/1026268366 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f31a00039b0 0x7f31a0003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:32.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.654+0000 7f31ba000640 1 -- 172.21.15.28:0/1026268366 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31a0003520 con 0x7f31a0004800 2024-02-21T23:12:32.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.655+0000 7f31ba000640 1 --2- 172.21.15.28:0/1026268366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31a0004800 0x7f31a0004c80 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 crypto rx=0x7f31a8009a90 tx=0x7f31a0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78c26c5165a2c82c server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:32.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.656+0000 7f31b9600640 1 -- 172.21.15.28:0/1026268366 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a8039070 con 0x7f31a0004800 2024-02-21T23:12:32.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.656+0000 7f31b9600640 1 -- 172.21.15.28:0/1026268366 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31a802c680 con 0x7f31a0004800 2024-02-21T23:12:32.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.656+0000 7f31b9600640 1 -- 172.21.15.28:0/1026268366 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a8009690 con 0x7f31a0004800 2024-02-21T23:12:32.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.656+0000 7f31c1600640 1 -- 172.21.15.28:0/1026268366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31a0004800 msgr2=0x7f31a0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:32.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.656+0000 7f31c1600640 1 --2- 172.21.15.28:0/1026268366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31a0004800 0x7f31a0004c80 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 crypto rx=0x7f31a8009a90 tx=0x7f31a0005830 comp rx=0 tx=0).stop 2024-02-21T23:12:32.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.656+0000 7f31bb400640 1 -- 172.21.15.28:0/1026268366 reap_dead start 2024-02-21T23:12:32.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.656+0000 7f31c1600640 1 -- 172.21.15.28:0/1026268366 shutdown_connections 2024-02-21T23:12:32.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.656+0000 7f31c1600640 1 -- 172.21.15.28:0/1026268366 >> 172.21.15.28:0/1026268366 conn(0x7f31bc0c8810 msgr2=0x7f31bc0c8bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:32.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.657+0000 7f31c1600640 1 -- 172.21.15.28:0/1026268366 shutdown_connections 2024-02-21T23:12:32.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.657+0000 7f31c1600640 1 -- 172.21.15.28:0/1026268366 wait complete. 2024-02-21T23:12:32.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.658+0000 7f31c1600640 1 Processor -- start 2024-02-21T23:12:32.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.659+0000 7f31c1600640 1 -- start start 2024-02-21T23:12:32.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.659+0000 7f31c1600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f31a0004800 0x7f31bc149d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:32.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.659+0000 7f31c1600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31bc0ad9b0 0x7f31bc14b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:32.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.659+0000 7f31c1600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f31bc14be50 0x7f31bc1e1b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:32.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.659+0000 7f31c1600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31bc0c58f0 con 0x7f31bc0ad9b0 2024-02-21T23:12:32.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.659+0000 7f31c1600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31bc0c6cb0 con 0x7f31a0004800 2024-02-21T23:12:32.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.659+0000 7f31c1600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31bc0c5a70 con 0x7f31bc14be50 2024-02-21T23:12:32.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.660+0000 7f31ba000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31bc0ad9b0 0x7f31bc14b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:32.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.660+0000 7f31ba000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31bc0ad9b0 0x7f31bc14b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44422/0 (socket says 172.21.15.28:44422) 2024-02-21T23:12:32.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.660+0000 7f31ba000640 1 -- 172.21.15.28:0/1132005415 learned_addr learned my addr 172.21.15.28:0/1132005415 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:32.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.660+0000 7f31bb400640 1 --2- 172.21.15.28:0/1132005415 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f31bc14be50 0x7f31bc1e1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:32.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.660+0000 7f31baa00640 1 --2- 172.21.15.28:0/1132005415 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f31a0004800 0x7f31bc149d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:32.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.660+0000 7f31ba000640 1 -- 172.21.15.28:0/1132005415 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f31bc14be50 msgr2=0x7f31bc1e1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:32.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.660+0000 7f31ba000640 1 --2- 172.21.15.28:0/1132005415 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f31bc14be50 0x7f31bc1e1b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:32.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.660+0000 7f31ba000640 1 -- 172.21.15.28:0/1132005415 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f31a0004800 msgr2=0x7f31bc149d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:32.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.660+0000 7f31ba000640 1 --2- 172.21.15.28:0/1132005415 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f31a0004800 0x7f31bc149d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:32.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.660+0000 7f31ba000640 1 -- 172.21.15.28:0/1132005415 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31bc1eb880 con 0x7f31bc0ad9b0 2024-02-21T23:12:32.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.660+0000 7f31baa00640 1 --2- 172.21.15.28:0/1132005415 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f31a0004800 0x7f31bc149d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:12:32.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.661+0000 7f31ba000640 1 --2- 172.21.15.28:0/1132005415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31bc0ad9b0 0x7f31bc14b910 secure :-1 s=READY pgs=2255 cs=0 l=1 rev1=1 crypto rx=0x7f31a802c2f0 tx=0x7f31a8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:32.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.661+0000 7f31b2a00640 1 -- 172.21.15.28:0/1132005415 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a8042020 con 0x7f31bc0ad9b0 2024-02-21T23:12:32.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.661+0000 7f31b2a00640 1 -- 172.21.15.28:0/1132005415 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31a803a040 con 0x7f31bc0ad9b0 2024-02-21T23:12:32.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.661+0000 7f31b2a00640 1 -- 172.21.15.28:0/1132005415 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a8039070 con 0x7f31bc0ad9b0 2024-02-21T23:12:32.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.661+0000 7f31c1600640 1 -- 172.21.15.28:0/1132005415 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31bc14a340 con 0x7f31bc0ad9b0 2024-02-21T23:12:32.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.661+0000 7f31c1600640 1 -- 172.21.15.28:0/1132005415 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31bc1eada0 con 0x7f31bc0ad9b0 2024-02-21T23:12:32.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.663+0000 7f31c1600640 1 -- 172.21.15.28:0/1132005415 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31bc068640 con 0x7f31bc0ad9b0 2024-02-21T23:12:32.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.663+0000 7f31b2a00640 1 -- 172.21.15.28:0/1132005415 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f31a8045020 con 0x7f31bc0ad9b0 2024-02-21T23:12:32.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.664+0000 7f31b2a00640 1 --2- 172.21.15.28:0/1132005415 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f319c06b550 0x7f319c06da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:32.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.664+0000 7f31b2a00640 1 -- 172.21.15.28:0/1132005415 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f31a80aba40 con 0x7f31bc0ad9b0 2024-02-21T23:12:32.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.665+0000 7f31baa00640 1 --2- 172.21.15.28:0/1132005415 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f319c06b550 0x7f319c06da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:32.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.666+0000 7f31baa00640 1 --2- 172.21.15.28:0/1132005415 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f319c06b550 0x7f319c06da10 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f31ac007460 tx=0x7f31ac00b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:32.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.669+0000 7f31b2a00640 1 -- 172.21.15.28:0/1132005415 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f31a804c030 con 0x7f31bc0ad9b0 2024-02-21T23:12:32.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.917+0000 7f31c1600640 1 -- 172.21.15.28:0/1132005415 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f31bc0c6710 con 0x7f31bc0ad9b0 2024-02-21T23:12:32.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.919+0000 7f31b2a00640 1 -- 172.21.15.28:0/1132005415 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 217 v217) v1 ==== 95+0+4952 (secure 0 0 0) 0x7f31a807a710 con 0x7f31bc0ad9b0 2024-02-21T23:12:32.908 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:12:32.908 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":217,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7100,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3213818811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3213818811},{"type":"v1","addr":"172.21.15.28:6835","nonce":3213818811}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":213},{"gid":7106,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/1322911156","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":1322911156},{"type":"v1","addr":"172.21.15.81:6837","nonce":1322911156}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":213},{"gid":10386,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3814985204","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3814985204},{"type":"v1","addr":"172.21.15.28:6834","nonce":3814985204}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":212}],"filesystems":[{"mdsmap":{"epoch":217,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:12:26.259457+0000","modified":"2024-02-21T23:12:29.346312+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10398},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10398":{"gid":10398,"name":"d","rank":0,"incarnation":214,"state":"up:active","state_seq":3,"addr":"172.21.15.81:6836/2895278683","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":2895278683},{"type":"v1","addr":"172.21.15.81:6836","nonce":2895278683}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15}]} 2024-02-21T23:12:32.908 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 217 2024-02-21T23:12:32.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.922+0000 7f31c1600640 1 -- 172.21.15.28:0/1132005415 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f319c06b550 msgr2=0x7f319c06da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:32.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.922+0000 7f31c1600640 1 --2- 172.21.15.28:0/1132005415 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f319c06b550 0x7f319c06da10 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f31ac007460 tx=0x7f31ac00b040 comp rx=0 tx=0).stop 2024-02-21T23:12:32.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.922+0000 7f31c1600640 1 -- 172.21.15.28:0/1132005415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31bc0ad9b0 msgr2=0x7f31bc14b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:32.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.922+0000 7f31c1600640 1 --2- 172.21.15.28:0/1132005415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31bc0ad9b0 0x7f31bc14b910 secure :-1 s=READY pgs=2255 cs=0 l=1 rev1=1 crypto rx=0x7f31a802c2f0 tx=0x7f31a8004730 comp rx=0 tx=0).stop 2024-02-21T23:12:32.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.922+0000 7f31c1600640 1 -- 172.21.15.28:0/1132005415 shutdown_connections 2024-02-21T23:12:32.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.922+0000 7f31c1600640 1 --2- 172.21.15.28:0/1132005415 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f319c06b550 0x7f319c06da10 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:32.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.922+0000 7f31c1600640 1 --2- 172.21.15.28:0/1132005415 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f31bc14be50 0x7f31bc1e1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:32.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.922+0000 7f31c1600640 1 --2- 172.21.15.28:0/1132005415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31bc0ad9b0 0x7f31bc14b910 unknown :-1 s=CLOSED pgs=2255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:32.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.922+0000 7f31c1600640 1 --2- 172.21.15.28:0/1132005415 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f31a0004800 0x7f31bc149d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:32.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.922+0000 7f31c1600640 1 -- 172.21.15.28:0/1132005415 >> 172.21.15.28:0/1132005415 conn(0x7f31bc0c8810 msgr2=0x7f31bc06c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:32.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.922+0000 7f31c1600640 1 -- 172.21.15.28:0/1132005415 shutdown_connections 2024-02-21T23:12:32.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:32.922+0000 7f31c1600640 1 -- 172.21.15.28:0/1132005415 wait complete. 2024-02-21T23:12:33.335 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 217, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:12:26.259457+0000', 'modified': '2024-02-21T23:12:29.346312+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10398}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10398': {'gid': 10398, 'name': 'd', 'rank': 0, 'incarnation': 214, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.81:6836/2895278683', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6834', 'nonce': 2895278683}, {'type': 'v1', 'addr': '172.21.15.81:6836', 'nonce': 2895278683}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [32], 'metadata_pool': 31, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:12:33.336 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-21T23:12:33.336 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-02-21T23:12:33.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.581+0000 7fdac7000640 1 Processor -- start 2024-02-21T23:12:33.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.581+0000 7fdac7000640 1 -- start start 2024-02-21T23:12:33.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.582+0000 7fdac7000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fdac006d410 0x7fdac00a8e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:33.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.582+0000 7fdac7000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdac0149c10 con 0x7fdac00c43e0 2024-02-21T23:12:33.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.582+0000 7fdac7000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdac01537d0 con 0x7fdac0149830 2024-02-21T23:12:33.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.582+0000 7fdac7000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdac0153ab0 con 0x7fdac006d410 2024-02-21T23:12:33.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.583+0000 7fdac6600640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fdac00c43e0 0x7fdac00c5a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:32964/0 (socket says 172.21.15.28:32964) 2024-02-21T23:12:33.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.583+0000 7fdac6600640 1 -- 172.21.15.28:0/1066729551 learned_addr learned my addr 172.21.15.28:0/1066729551 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:12:33.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.583+0000 7fdac5200640 1 --2- 172.21.15.28:0/1066729551 >> v2:172.21.15.81:3300/0 conn(0x7fdac006d410 0x7fdac00a8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:33.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.584+0000 7fdabfe00640 1 -- 172.21.15.28:0/1066729551 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2973287545 0 0) 0x7fdac0149c10 con 0x7fdac00c43e0 2024-02-21T23:12:33.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.584+0000 7fdabfe00640 1 -- 172.21.15.28:0/1066729551 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdaa8003520 con 0x7fdac00c43e0 2024-02-21T23:12:33.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.584+0000 7fdabfe00640 1 -- 172.21.15.28:0/1066729551 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4052093870 0 0) 0x7fdaa8003520 con 0x7fdac00c43e0 2024-02-21T23:12:33.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.584+0000 7fdabfe00640 1 -- 172.21.15.28:0/1066729551 >> v2:172.21.15.81:3300/0 conn(0x7fdac006d410 msgr2=0x7fdac00a8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:33.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.584+0000 7fdabfe00640 1 --2- 172.21.15.28:0/1066729551 >> v2:172.21.15.81:3300/0 conn(0x7fdac006d410 0x7fdac00a8e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:33.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.584+0000 7fdabfe00640 1 -- 172.21.15.28:0/1066729551 >> v1:172.21.15.81:6789/0 conn(0x7fdac0149830 legacy=0x7fdac00c0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:33.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.584+0000 7fdabfe00640 1 -- 172.21.15.28:0/1066729551 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdac00c4fa0 con 0x7fdac00c43e0 2024-02-21T23:12:33.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.584+0000 7fdabfe00640 1 -- 172.21.15.28:0/1066729551 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fdab80019f0 con 0x7fdac00c43e0 2024-02-21T23:12:33.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.584+0000 7fdabfe00640 1 -- 172.21.15.28:0/1066729551 >> v1:172.21.15.28:6789/0 conn(0x7fdac00c43e0 legacy=0x7fdac00c5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:33.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.584+0000 7fdabfe00640 1 --2- 172.21.15.28:0/1066729551 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdaa80039b0 0x7fdaa8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:33.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.584+0000 7fdabfe00640 1 --2- 172.21.15.28:0/1066729551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdaa8004800 0x7fdaa8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:33.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.584+0000 7fdabfe00640 1 --2- 172.21.15.28:0/1066729551 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdaa80051c0 0x7fdaa800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:33.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.584+0000 7fdabfe00640 1 -- 172.21.15.28:0/1066729551 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdac0153ab0 con 0x7fdaa8004800 2024-02-21T23:12:33.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.584+0000 7fdabfe00640 1 -- 172.21.15.28:0/1066729551 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaa800c070 con 0x7fdaa80039b0 2024-02-21T23:12:33.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.584+0000 7fdabfe00640 1 -- 172.21.15.28:0/1066729551 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdaa800c270 con 0x7fdaa80051c0 2024-02-21T23:12:33.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.585+0000 7fdac5200640 1 --2- 172.21.15.28:0/1066729551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdaa8004800 0x7fdaa8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:33.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.585+0000 7fdac5c00640 1 --2- 172.21.15.28:0/1066729551 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdaa80039b0 0x7fdaa8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:33.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.585+0000 7fdac5200640 1 -- 172.21.15.28:0/1066729551 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdaa80051c0 msgr2=0x7fdaa800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:33.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.585+0000 7fdac5200640 1 --2- 172.21.15.28:0/1066729551 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdaa80051c0 0x7fdaa800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:33.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.585+0000 7fdac5200640 1 -- 172.21.15.28:0/1066729551 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdaa80039b0 msgr2=0x7fdaa8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:33.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.585+0000 7fdac5200640 1 --2- 172.21.15.28:0/1066729551 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdaa80039b0 0x7fdaa8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:33.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.585+0000 7fdac5200640 1 -- 172.21.15.28:0/1066729551 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaa8003520 con 0x7fdaa8004800 2024-02-21T23:12:33.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.585+0000 7fdac6600640 1 -- 172.21.15.28:0/1066729551 reap_dead start 2024-02-21T23:12:33.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.585+0000 7fdac5c00640 1 --2- 172.21.15.28:0/1066729551 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdaa80039b0 0x7fdaa8003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:12:33.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.587+0000 7fdac5200640 1 --2- 172.21.15.28:0/1066729551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdaa8004800 0x7fdaa8004c80 secure :-1 s=READY pgs=2257 cs=0 l=1 rev1=1 crypto rx=0x7fdaac009ac0 tx=0x7fdaa8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1826ff63eff65955 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:33.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.587+0000 7fdabfe00640 1 -- 172.21.15.28:0/1066729551 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaac039070 con 0x7fdaa8004800 2024-02-21T23:12:33.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.587+0000 7fdabfe00640 1 -- 172.21.15.28:0/1066729551 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdaac02c6b0 con 0x7fdaa8004800 2024-02-21T23:12:33.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.587+0000 7fdabfe00640 1 -- 172.21.15.28:0/1066729551 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaac009690 con 0x7fdaa8004800 2024-02-21T23:12:33.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.587+0000 7fdac7000640 1 -- 172.21.15.28:0/1066729551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdaa8004800 msgr2=0x7fdaa8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:33.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.587+0000 7fdac7000640 1 --2- 172.21.15.28:0/1066729551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdaa8004800 0x7fdaa8004c80 secure :-1 s=READY pgs=2257 cs=0 l=1 rev1=1 crypto rx=0x7fdaac009ac0 tx=0x7fdaa8005830 comp rx=0 tx=0).stop 2024-02-21T23:12:33.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.588+0000 7fdac7000640 1 -- 172.21.15.28:0/1066729551 shutdown_connections 2024-02-21T23:12:33.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.588+0000 7fdac7000640 1 --2- 172.21.15.28:0/1066729551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdaa8004800 0x7fdaa8004c80 unknown :-1 s=CLOSED pgs=2257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:33.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.588+0000 7fdac7000640 1 -- 172.21.15.28:0/1066729551 >> 172.21.15.28:0/1066729551 conn(0x7fdac00a8320 msgr2=0x7fdac00a8730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:33.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.588+0000 7fdac7000640 1 -- 172.21.15.28:0/1066729551 shutdown_connections 2024-02-21T23:12:33.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.588+0000 7fdac7000640 1 -- 172.21.15.28:0/1066729551 wait complete. 2024-02-21T23:12:33.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.589+0000 7fdac7000640 1 Processor -- start 2024-02-21T23:12:33.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.590+0000 7fdac7000640 1 -- start start 2024-02-21T23:12:33.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.591+0000 7fdac7000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdaa8004800 0x7fdac01448e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:33.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.591+0000 7fdac7000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdac01e9f90 0x7fdac01e9540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:33.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.591+0000 7fdac7000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdac01ec800 0x7fdac0146c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:33.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.591+0000 7fdac7000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdac00a8e40 con 0x7fdac01e9f90 2024-02-21T23:12:33.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.591+0000 7fdac7000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdac00c5560 con 0x7fdac01ec800 2024-02-21T23:12:33.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.591+0000 7fdac7000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdac00a8fc0 con 0x7fdaa8004800 2024-02-21T23:12:33.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.591+0000 7fdac5200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdac01e9f90 0x7fdac01e9540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:33.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.591+0000 7fdac5200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdac01e9f90 0x7fdac01e9540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44428/0 (socket says 172.21.15.28:44428) 2024-02-21T23:12:33.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.591+0000 7fdac5200640 1 -- 172.21.15.28:0/1073406007 learned_addr learned my addr 172.21.15.28:0/1073406007 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:33.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.591+0000 7fdac5200640 1 -- 172.21.15.28:0/1073406007 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdaa8004800 msgr2=0x7fdac01448e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:33.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.591+0000 7fdac5200640 1 --2- 172.21.15.28:0/1073406007 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdaa8004800 0x7fdac01448e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:33.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.591+0000 7fdac5200640 1 -- 172.21.15.28:0/1073406007 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdac01ec800 msgr2=0x7fdac0146c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:33.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.591+0000 7fdac5200640 1 --2- 172.21.15.28:0/1073406007 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdac01ec800 0x7fdac0146c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:33.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.591+0000 7fdac5200640 1 -- 172.21.15.28:0/1073406007 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdac00a6b00 con 0x7fdac01e9f90 2024-02-21T23:12:33.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.592+0000 7fdac5200640 1 --2- 172.21.15.28:0/1073406007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdac01e9f90 0x7fdac01e9540 secure :-1 s=READY pgs=2258 cs=0 l=1 rev1=1 crypto rx=0x7fdaac000c00 tx=0x7fdaac004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:33.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.592+0000 7fdabd600640 1 -- 172.21.15.28:0/1073406007 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaac042020 con 0x7fdac01e9f90 2024-02-21T23:12:33.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.592+0000 7fdabd600640 1 -- 172.21.15.28:0/1073406007 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdaac03a040 con 0x7fdac01e9f90 2024-02-21T23:12:33.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.592+0000 7fdabd600640 1 -- 172.21.15.28:0/1073406007 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaac039070 con 0x7fdac01e9f90 2024-02-21T23:12:33.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.592+0000 7fdac7000640 1 -- 172.21.15.28:0/1073406007 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdac00a6de0 con 0x7fdac01e9f90 2024-02-21T23:12:33.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.592+0000 7fdac7000640 1 -- 172.21.15.28:0/1073406007 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdac00a73a0 con 0x7fdac01e9f90 2024-02-21T23:12:33.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.594+0000 7fdabd600640 1 -- 172.21.15.28:0/1073406007 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdaac045020 con 0x7fdac01e9f90 2024-02-21T23:12:33.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.595+0000 7fdabd600640 1 --2- 172.21.15.28:0/1073406007 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdaa006b500 0x7fdaa006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:33.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.595+0000 7fdabd600640 1 -- 172.21.15.28:0/1073406007 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fdaac0ab7d0 con 0x7fdac01e9f90 2024-02-21T23:12:33.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.596+0000 7fdacb479740 1 Processor -- start 2024-02-21T23:12:33.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.596+0000 7fdac5c00640 1 --2- 172.21.15.28:0/1073406007 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdaa006b500 0x7fdaa006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:33.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.596+0000 7fdacb479740 1 -- start start 2024-02-21T23:12:33.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.597+0000 7fdacb479740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8160fb190 0x55c8160b7820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:33.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.597+0000 7fdacb479740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c8160b7d60 0x55c816106330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:33.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.597+0000 7fdacb479740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c8160fb570 0x55c816106870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:33.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.597+0000 7fdacb479740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c816021470 con 0x55c8160fb190 2024-02-21T23:12:33.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.597+0000 7fdacb479740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c815fcaaf0 con 0x55c8160b7d60 2024-02-21T23:12:33.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.597+0000 7fdacb479740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c81601d510 con 0x55c8160fb570 2024-02-21T23:12:33.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.597+0000 7fdac5c00640 1 --2- 172.21.15.28:0/1073406007 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdaa006b500 0x7fdaa006d9c0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fdac01e6900 tx=0x7fdab4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:33.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.597+0000 7fdac5c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8160fb190 0x55c8160b7820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:33.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.597+0000 7fdac5c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8160fb190 0x55c8160b7820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44430/0 (socket says 172.21.15.28:44430) 2024-02-21T23:12:33.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.597+0000 7fdac5c00640 1 -- 172.21.15.28:0/3546878131 learned_addr learned my addr 172.21.15.28:0/3546878131 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:33.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.598+0000 7fdac5200640 1 --2- 172.21.15.28:0/3546878131 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c8160b7d60 0x55c816106330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:33.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.598+0000 7fdac6600640 1 --2- 172.21.15.28:0/3546878131 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c8160fb570 0x55c816106870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:33.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.598+0000 7fdac5c00640 1 -- 172.21.15.28:0/3546878131 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c8160fb570 msgr2=0x55c816106870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:33.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.598+0000 7fdac5c00640 1 --2- 172.21.15.28:0/3546878131 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c8160fb570 0x55c816106870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:33.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.598+0000 7fdac5c00640 1 -- 172.21.15.28:0/3546878131 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c8160b7d60 msgr2=0x55c816106330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:33.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.598+0000 7fdac5c00640 1 --2- 172.21.15.28:0/3546878131 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c8160b7d60 0x55c816106330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:33.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.598+0000 7fdac5c00640 1 -- 172.21.15.28:0/3546878131 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c815fd1d90 con 0x55c8160fb190 2024-02-21T23:12:33.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.598+0000 7fdac5c00640 1 --2- 172.21.15.28:0/3546878131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8160fb190 0x55c8160b7820 secure :-1 s=READY pgs=2259 cs=0 l=1 rev1=1 crypto rx=0x7fdab4014790 tx=0x7fdab4010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=137446a79fc47886 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:33.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.599+0000 7fdab2000640 1 -- 172.21.15.28:0/3546878131 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdab4024070 con 0x55c8160fb190 2024-02-21T23:12:33.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.599+0000 7fdab2000640 1 -- 172.21.15.28:0/3546878131 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdab4012b50 con 0x55c8160fb190 2024-02-21T23:12:33.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.599+0000 7fdab2000640 1 -- 172.21.15.28:0/3546878131 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdab401f400 con 0x55c8160fb190 2024-02-21T23:12:33.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.599+0000 7fdacb479740 1 -- 172.21.15.28:0/3546878131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8160fb190 msgr2=0x55c8160b7820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:33.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.599+0000 7fdacb479740 1 --2- 172.21.15.28:0/3546878131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8160fb190 0x55c8160b7820 secure :-1 s=READY pgs=2259 cs=0 l=1 rev1=1 crypto rx=0x7fdab4014790 tx=0x7fdab4010000 comp rx=0 tx=0).stop 2024-02-21T23:12:33.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.600+0000 7fdacb479740 1 -- 172.21.15.28:0/3546878131 shutdown_connections 2024-02-21T23:12:33.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.600+0000 7fdacb479740 1 --2- 172.21.15.28:0/3546878131 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c8160fb570 0x55c816106870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:33.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.600+0000 7fdacb479740 1 --2- 172.21.15.28:0/3546878131 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c8160b7d60 0x55c816106330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:33.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.600+0000 7fdacb479740 1 --2- 172.21.15.28:0/3546878131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8160fb190 0x55c8160b7820 unknown :-1 s=CLOSED pgs=2259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:33.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.600+0000 7fdacb479740 1 -- 172.21.15.28:0/3546878131 >> 172.21.15.28:0/3546878131 conn(0x55c8160b9970 msgr2=0x55c8160de380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:33.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.600+0000 7fdacb479740 1 -- 172.21.15.28:0/3546878131 shutdown_connections 2024-02-21T23:12:33.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.600+0000 7fdacb479740 1 -- 172.21.15.28:0/3546878131 wait complete. 2024-02-21T23:12:33.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.600+0000 7fdacb479740 1 -- start start 2024-02-21T23:12:33.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.601+0000 7fdacb479740 1 Processor -- start 2024-02-21T23:12:33.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.602+0000 7fdacb479740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c8160fb190 0x55c8160dcff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:33.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.602+0000 7fdacb479740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8160fb570 0x55c81610d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:33.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.602+0000 7fdacb479740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c81610deb0 0x55c8160db270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:33.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.602+0000 7fdacb479740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8160fbc50 con 0x55c8160fb570 2024-02-21T23:12:33.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.602+0000 7fdacb479740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c815fc0040 con 0x55c8160fb190 2024-02-21T23:12:33.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.602+0000 7fdacb479740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c8160bd510 con 0x55c81610deb0 2024-02-21T23:12:33.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.602+0000 7fdacb479740 10 client.0 _ll_register_callbacks cb 0x55c815fdd920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:12:33.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.602+0000 7fdac5200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8160fb570 0x55c81610d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:33.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.602+0000 7fdac5200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8160fb570 0x55c81610d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44442/0 (socket says 172.21.15.28:44442) 2024-02-21T23:12:33.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.602+0000 7fdac5200640 1 -- 172.21.15.28:0/1918794689 learned_addr learned my addr 172.21.15.28:0/1918794689 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:33.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.602+0000 7fdac5c00640 1 --2- 172.21.15.28:0/1918794689 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c8160fb190 0x55c8160dcff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:33.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.602+0000 7fdac6600640 1 --2- 172.21.15.28:0/1918794689 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c81610deb0 0x55c8160db270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:33.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.602+0000 7fdac5200640 1 -- 172.21.15.28:0/1918794689 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c81610deb0 msgr2=0x55c8160db270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:33.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.602+0000 7fdac5200640 1 --2- 172.21.15.28:0/1918794689 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c81610deb0 0x55c8160db270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:33.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.602+0000 7fdac5200640 1 -- 172.21.15.28:0/1918794689 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c8160fb190 msgr2=0x55c8160dcff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:33.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.602+0000 7fdac5200640 1 --2- 172.21.15.28:0/1918794689 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c8160fb190 0x55c8160dcff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:33.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.603+0000 7fdac5200640 1 -- 172.21.15.28:0/1918794689 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c815fdfaa0 con 0x55c8160fb570 2024-02-21T23:12:33.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.603+0000 7fdac6600640 1 --2- 172.21.15.28:0/1918794689 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c81610deb0 0x55c8160db270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:12:33.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.603+0000 7fdac5200640 1 --2- 172.21.15.28:0/1918794689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8160fb570 0x55c81610d970 secure :-1 s=READY pgs=2260 cs=0 l=1 rev1=1 crypto rx=0x7fdaac0a9b00 tx=0x7fdaac0044e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bae5e15328c3a98a server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:33.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.603+0000 7fdaa6a00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:12:33.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.603+0000 7fdaa6a00640 1 -- 172.21.15.28:0/1918794689 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaac07df00 con 0x55c8160fb570 2024-02-21T23:12:33.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.603+0000 7fdaa6a00640 1 -- 172.21.15.28:0/1918794689 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(197..197 src has 1..197) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fdaac082030 con 0x55c8160fb570 2024-02-21T23:12:33.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.603+0000 7fdacb479740 1 -- 172.21.15.28:0/1918794689 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdab4012b50 con 0x55c8160fb570 2024-02-21T23:12:33.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.603+0000 7fdaa6a00640 1 -- 172.21.15.28:0/1918794689 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=217) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdaac0824c0 con 0x55c8160fb570 2024-02-21T23:12:33.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.603+0000 7fdacb479740 10 client.10536 fetch_fsmap learned FSMap version 217 2024-02-21T23:12:33.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.603+0000 7fdacb479740 1 -- 172.21.15.28:0/1918794689 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=217,monmap=2+}) v3 -- 0x55c815d362d0 con 0x55c8160fb570 2024-02-21T23:12:33.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.604+0000 7fdaa6a00640 1 -- 172.21.15.28:0/1918794689 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdab4012b50 con 0x55c8160fb570 2024-02-21T23:12:33.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.604+0000 7fdaa6a00640 1 -- 172.21.15.28:0/1918794689 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 217) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fdaac079200 con 0x55c8160fb570 2024-02-21T23:12:33.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.604+0000 7fdacb479740 10 client.10536 fetch_fsmap finished waiting for FSMap version 217 2024-02-21T23:12:33.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.604+0000 7fdacb479740 10 client.10536 resolve_mds: resolved name 'd' to daemon mds.d 2024-02-21T23:12:33.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.604+0000 7fdacb479740 20 client.10536 populate_metadata read hostname 'smithi028' 2024-02-21T23:12:33.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.604+0000 7fdacb479740 1 --2- 172.21.15.28:0/1918794689 >> [v2:172.21.15.81:6834/2895278683,v1:172.21.15.81:6836/2895278683] conn(0x55c8160e16a0 0x55c8160e1a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:33.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.604+0000 7fdacb479740 4 client.10536 mds_command: new command op to 10398 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-21T23:12:33.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.604+0000 7fdacb479740 1 -- 172.21.15.28:0/1918794689 --> [v2:172.21.15.81:6834/2895278683,v1:172.21.15.81:6836/2895278683] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c81602e9b0 con 0x55c8160e16a0 2024-02-21T23:12:33.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.605+0000 7fdac5c00640 1 --2- 172.21.15.28:0/1918794689 >> [v2:172.21.15.81:6834/2895278683,v1:172.21.15.81:6836/2895278683] conn(0x55c8160e16a0 0x55c8160e1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:33.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.606+0000 7fdaa6a00640 10 client.10536 ms_handle_connect on v2:172.21.15.81:6834/2895278683 2024-02-21T23:12:33.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.606+0000 7fdac5c00640 1 --2- 172.21.15.28:0/1918794689 >> [v2:172.21.15.81:6834/2895278683,v1:172.21.15.81:6836/2895278683] conn(0x55c8160e16a0 0x55c8160e1a80 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f0134d38c61c45df server_cookie=117d18f811f07347 in_seq=0 out_seq=0 2024-02-21T23:12:33.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.608+0000 7fdaa6a00640 1 -- 172.21.15.28:0/1918794689 <== mds.0 v2:172.21.15.81:6834/2895278683 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c81602e9b0 con 0x55c8160e16a0 2024-02-21T23:12:33.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.608+0000 7fdaa6a00640 10 client.10536 handle_command_reply: tid=0 2024-02-21T23:12:33.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.608+0000 7fdacb479740 1 client.10536 shutdown 2024-02-21T23:12:33.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.608+0000 7fdacb479740 1 -- 172.21.15.28:0/1918794689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8160fb570 msgr2=0x55c81610d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:33.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.608+0000 7fdacb479740 1 --2- 172.21.15.28:0/1918794689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8160fb570 0x55c81610d970 secure :-1 s=READY pgs=2260 cs=0 l=1 rev1=1 crypto rx=0x7fdaac0a9b00 tx=0x7fdaac0044e0 comp rx=0 tx=0).stop 2024-02-21T23:12:33.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.608+0000 7fdacb479740 1 -- 172.21.15.28:0/1918794689 shutdown_connections 2024-02-21T23:12:33.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.608+0000 7fdacb479740 1 --2- 172.21.15.28:0/1918794689 >> [v2:172.21.15.81:6834/2895278683,v1:172.21.15.81:6836/2895278683] conn(0x55c8160e16a0 0x55c8160e1a80 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:33.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.608+0000 7fdacb479740 1 --2- 172.21.15.28:0/1918794689 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c81610deb0 0x55c8160db270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:33.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.608+0000 7fdacb479740 1 --2- 172.21.15.28:0/1918794689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8160fb570 0x55c81610d970 unknown :-1 s=CLOSED pgs=2260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:33.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.608+0000 7fdacb479740 1 --2- 172.21.15.28:0/1918794689 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c8160fb190 0x55c8160dcff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:33.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.608+0000 7fdaa6a00640 0 client.10536 ms_handle_reset on v2:172.21.15.81:6834/2895278683 2024-02-21T23:12:33.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.608+0000 7fdacb479740 1 -- 172.21.15.28:0/1918794689 >> 172.21.15.28:0/1918794689 conn(0x55c8160b7d60 msgr2=0x55c8160b65f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:33.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.608+0000 7fdacb479740 1 -- 172.21.15.28:0/1918794689 shutdown_connections 2024-02-21T23:12:33.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.608+0000 7fdacb479740 1 -- 172.21.15.28:0/1918794689 wait complete. 2024-02-21T23:12:33.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.609+0000 7fdacb479740 20 client.10536 trim_cache size 0 max 16384 2024-02-21T23:12:33.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.627+0000 7fdacb479740 1 Processor -- start 2024-02-21T23:12:33.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.628+0000 7fdacb479740 1 -- start start 2024-02-21T23:12:33.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.628+0000 7fdacb479740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c8160fb190 0x55c8160e0580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:33.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.628+0000 7fdacb479740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c81610deb0 0x55c8160b2f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:33.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.628+0000 7fdacb479740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c8160e21c0 0x55c8160ddbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:33.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.628+0000 7fdacb479740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdac0072f60 con 0x55c81610deb0 2024-02-21T23:12:33.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.628+0000 7fdacb479740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdac0072360 con 0x55c8160fb190 2024-02-21T23:12:33.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.628+0000 7fdacb479740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdac0072960 con 0x55c8160e21c0 2024-02-21T23:12:33.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.629+0000 7fdac5200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c81610deb0 0x55c8160b2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:33.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.629+0000 7fdac5200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c81610deb0 0x55c8160b2f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44452/0 (socket says 172.21.15.28:44452) 2024-02-21T23:12:33.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.629+0000 7fdac5200640 1 -- 172.21.15.28:0/2021099120 learned_addr learned my addr 172.21.15.28:0/2021099120 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:33.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.629+0000 7fdac5200640 1 -- 172.21.15.28:0/2021099120 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c8160e21c0 msgr2=0x55c8160ddbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:33.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.629+0000 7fdac5c00640 1 --2- 172.21.15.28:0/2021099120 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c8160fb190 0x55c8160e0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:33.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.629+0000 7fdac5200640 1 --2- 172.21.15.28:0/2021099120 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c8160e21c0 0x55c8160ddbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:33.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.629+0000 7fdac5200640 1 -- 172.21.15.28:0/2021099120 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c8160fb190 msgr2=0x55c8160e0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:33.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.629+0000 7fdac5200640 1 --2- 172.21.15.28:0/2021099120 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c8160fb190 0x55c8160e0580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:33.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.629+0000 7fdac5200640 1 -- 172.21.15.28:0/2021099120 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c815e006b0 con 0x55c81610deb0 2024-02-21T23:12:33.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.630+0000 7fdac5200640 1 --2- 172.21.15.28:0/2021099120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c81610deb0 0x55c8160b2f00 secure :-1 s=READY pgs=2261 cs=0 l=1 rev1=1 crypto rx=0x7fdaac076210 tx=0x7fdaac080000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc28098d607a0030 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:33.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.630+0000 7fdab1600640 1 -- 172.21.15.28:0/2021099120 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaac07df00 con 0x55c81610deb0 2024-02-21T23:12:33.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.630+0000 7fdab1600640 1 -- 172.21.15.28:0/2021099120 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdaac095020 con 0x55c81610deb0 2024-02-21T23:12:33.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.630+0000 7fdab1600640 1 -- 172.21.15.28:0/2021099120 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaac07a4b0 con 0x55c81610deb0 2024-02-21T23:12:33.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.630+0000 7fdacb479740 1 -- 172.21.15.28:0/2021099120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c81610deb0 msgr2=0x55c8160b2f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:33.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.630+0000 7fdacb479740 1 --2- 172.21.15.28:0/2021099120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c81610deb0 0x55c8160b2f00 secure :-1 s=READY pgs=2261 cs=0 l=1 rev1=1 crypto rx=0x7fdaac076210 tx=0x7fdaac080000 comp rx=0 tx=0).stop 2024-02-21T23:12:33.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.630+0000 7fdacb479740 1 -- 172.21.15.28:0/2021099120 shutdown_connections 2024-02-21T23:12:33.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.630+0000 7fdacb479740 1 --2- 172.21.15.28:0/2021099120 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c8160e21c0 0x55c8160ddbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:33.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.630+0000 7fdacb479740 1 --2- 172.21.15.28:0/2021099120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c81610deb0 0x55c8160b2f00 unknown :-1 s=CLOSED pgs=2261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:33.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.630+0000 7fdacb479740 1 --2- 172.21.15.28:0/2021099120 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c8160fb190 0x55c8160e0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:33.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.630+0000 7fdacb479740 1 -- 172.21.15.28:0/2021099120 >> 172.21.15.28:0/2021099120 conn(0x55c8160b7d60 msgr2=0x55c81610ad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:33.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.631+0000 7fdacb479740 1 -- 172.21.15.28:0/2021099120 shutdown_connections 2024-02-21T23:12:33.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.631+0000 7fdacb479740 1 -- 172.21.15.28:0/2021099120 wait complete. 2024-02-21T23:12:33.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.631+0000 7fdacb479740 1 -- start start 2024-02-21T23:12:33.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.631+0000 7fdacb479740 1 Processor -- start 2024-02-21T23:12:33.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.632+0000 7fdacb479740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c8160e21c0 0x55c8160ddf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:33.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.632+0000 7fdacb479740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c8160fb190 0x55c8160de920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:33.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.632+0000 7fdacb479740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c81616e560 0x55c81616acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:33.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.632+0000 7fdacb479740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8160fbc50 con 0x55c81616e560 2024-02-21T23:12:33.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.632+0000 7fdacb479740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c815fc0040 con 0x55c8160fb190 2024-02-21T23:12:33.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.632+0000 7fdacb479740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c8160e0e90 con 0x55c8160e21c0 2024-02-21T23:12:33.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.632+0000 7fdacb479740 10 client.0 _ll_register_callbacks cb 0x7fdaac0798f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:12:33.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.633+0000 7fdac6600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c81616e560 0x55c81616acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:33.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.633+0000 7fdac6600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c81616e560 0x55c81616acd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44454/0 (socket says 172.21.15.28:44454) 2024-02-21T23:12:33.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.633+0000 7fdac6600640 1 -- 172.21.15.28:0/2528068473 learned_addr learned my addr 172.21.15.28:0/2528068473 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:33.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.633+0000 7fdac5200640 1 --2- 172.21.15.28:0/2528068473 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c8160fb190 0x55c8160de920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:33.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.633+0000 7fdac5c00640 1 --2- 172.21.15.28:0/2528068473 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c8160e21c0 0x55c8160ddf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:33.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.633+0000 7fdac6600640 1 -- 172.21.15.28:0/2528068473 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c8160e21c0 msgr2=0x55c8160ddf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:33.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.633+0000 7fdac6600640 1 --2- 172.21.15.28:0/2528068473 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c8160e21c0 0x55c8160ddf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:33.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.633+0000 7fdac6600640 1 -- 172.21.15.28:0/2528068473 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c8160fb190 msgr2=0x55c8160de920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:33.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.633+0000 7fdac6600640 1 --2- 172.21.15.28:0/2528068473 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c8160fb190 0x55c8160de920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:33.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.633+0000 7fdac6600640 1 -- 172.21.15.28:0/2528068473 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c8160def60 con 0x55c81616e560 2024-02-21T23:12:33.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.633+0000 7fdac6600640 1 --2- 172.21.15.28:0/2528068473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c81616e560 0x55c81616acd0 secure :-1 s=READY pgs=2262 cs=0 l=1 rev1=1 crypto rx=0x7fdac00c0ca0 tx=0x7fdab800eae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bab0af59e0d08ab4 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:33.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.633+0000 7fdaa6a00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:12:33.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.634+0000 7fdaa6a00640 1 -- 172.21.15.28:0/2528068473 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdab8019060 con 0x55c81616e560 2024-02-21T23:12:33.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.634+0000 7fdaa6a00640 1 -- 172.21.15.28:0/2528068473 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(197..197 src has 1..197) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fdab8004d70 con 0x55c81616e560 2024-02-21T23:12:33.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.634+0000 7fdacb479740 1 -- 172.21.15.28:0/2528068473 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdaac095020 con 0x55c81616e560 2024-02-21T23:12:33.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.634+0000 7fdaa6a00640 1 -- 172.21.15.28:0/2528068473 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=217) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdab801f050 con 0x55c81616e560 2024-02-21T23:12:33.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.634+0000 7fdacb479740 10 client.10542 fetch_fsmap learned FSMap version 217 2024-02-21T23:12:33.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.634+0000 7fdacb479740 1 -- 172.21.15.28:0/2528068473 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=217,monmap=2+}) v3 -- 0x55c816117530 con 0x55c81616e560 2024-02-21T23:12:33.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.634+0000 7fdaa6a00640 1 -- 172.21.15.28:0/2528068473 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdaac095020 con 0x55c81616e560 2024-02-21T23:12:33.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.634+0000 7fdaa6a00640 1 -- 172.21.15.28:0/2528068473 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 217) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fdab8022020 con 0x55c81616e560 2024-02-21T23:12:33.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.634+0000 7fdacb479740 10 client.10542 fetch_fsmap finished waiting for FSMap version 217 2024-02-21T23:12:33.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.635+0000 7fdacb479740 10 client.10542 resolve_mds: resolved name 'd' to daemon mds.d 2024-02-21T23:12:33.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.635+0000 7fdacb479740 20 client.10542 populate_metadata read hostname 'smithi028' 2024-02-21T23:12:33.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.635+0000 7fdacb479740 1 --2- 172.21.15.28:0/2528068473 >> [v2:172.21.15.81:6834/2895278683,v1:172.21.15.81:6836/2895278683] conn(0x55c8161716a0 0x55c81617a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:33.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.635+0000 7fdacb479740 4 client.10542 mds_command: new command op to 10398 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-21T23:12:33.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.635+0000 7fdacb479740 1 -- 172.21.15.28:0/2528068473 --> [v2:172.21.15.81:6834/2895278683,v1:172.21.15.81:6836/2895278683] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55c8160ba0d0 con 0x55c8161716a0 2024-02-21T23:12:33.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.635+0000 7fdac5c00640 1 --2- 172.21.15.28:0/2528068473 >> [v2:172.21.15.81:6834/2895278683,v1:172.21.15.81:6836/2895278683] conn(0x55c8161716a0 0x55c81617a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:33.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.636+0000 7fdac5c00640 1 --2- 172.21.15.28:0/2528068473 >> [v2:172.21.15.81:6834/2895278683,v1:172.21.15.81:6836/2895278683] conn(0x55c8161716a0 0x55c81617a230 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=49a5ac0410eb174c server_cookie=8615ef3e7bc678a6 in_seq=0 out_seq=0 2024-02-21T23:12:33.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.636+0000 7fdaa6a00640 10 client.10542 ms_handle_connect on v2:172.21.15.81:6834/2895278683 2024-02-21T23:12:33.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.637+0000 7fdaa6a00640 1 -- 172.21.15.28:0/2528068473 <== mds.0 v2:172.21.15.81:6834/2895278683 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+329 (crc 0 0 0) 0x55c8160ba0d0 con 0x55c8161716a0 2024-02-21T23:12:33.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.637+0000 7fdaa6a00640 10 client.10542 handle_command_reply: tid=0 2024-02-21T23:12:33.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.637+0000 7fdacb479740 1 client.10542 shutdown 2024-02-21T23:12:33.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.637+0000 7fdacb479740 1 -- 172.21.15.28:0/2528068473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c81616e560 msgr2=0x55c81616acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:33.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.637+0000 7fdacb479740 1 --2- 172.21.15.28:0/2528068473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c81616e560 0x55c81616acd0 secure :-1 s=READY pgs=2262 cs=0 l=1 rev1=1 crypto rx=0x7fdac00c0ca0 tx=0x7fdab800eae0 comp rx=0 tx=0).stop 2024-02-21T23:12:33.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.637+0000 7fdacb479740 1 -- 172.21.15.28:0/2528068473 shutdown_connections 2024-02-21T23:12:33.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.637+0000 7fdacb479740 1 --2- 172.21.15.28:0/2528068473 >> [v2:172.21.15.81:6834/2895278683,v1:172.21.15.81:6836/2895278683] conn(0x55c8161716a0 0x55c81617a230 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:33.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.637+0000 7fdacb479740 1 --2- 172.21.15.28:0/2528068473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c81616e560 0x55c81616acd0 unknown :-1 s=CLOSED pgs=2262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:33.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.637+0000 7fdacb479740 1 --2- 172.21.15.28:0/2528068473 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55c8160fb190 0x55c8160de920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:33.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.637+0000 7fdacb479740 1 --2- 172.21.15.28:0/2528068473 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55c8160e21c0 0x55c8160ddf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:33.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.637+0000 7fdaa6a00640 0 client.10542 ms_handle_reset on v2:172.21.15.81:6834/2895278683 2024-02-21T23:12:33.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.637+0000 7fdacb479740 1 -- 172.21.15.28:0/2528068473 >> 172.21.15.28:0/2528068473 conn(0x55c8160b7d60 msgr2=0x55c8160e1a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:33.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.637+0000 7fdacb479740 1 -- 172.21.15.28:0/2528068473 shutdown_connections 2024-02-21T23:12:33.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.637+0000 7fdacb479740 1 -- 172.21.15.28:0/2528068473 wait complete. 2024-02-21T23:12:33.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.638+0000 7fdacb479740 20 client.10542 trim_cache size 0 max 16384 2024-02-21T23:12:33.629 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-21T23:12:33.630 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac", 2024-02-21T23:12:33.630 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-02-21T23:12:33.630 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 10398, 2024-02-21T23:12:33.630 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-02-21T23:12:33.630 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-02-21T23:12:33.630 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-02-21T23:12:33.630 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 6.8279211679999996, 2024-02-21T23:12:33.630 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 217, 2024-02-21T23:12:33.630 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 197, 2024-02-21T23:12:33.630 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 197, 2024-02-21T23:12:33.630 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 12.299767315 2024-02-21T23:12:33.630 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-21T23:12:33.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.638+0000 7fdac7000640 1 -- 172.21.15.28:0/1073406007 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdaa006b500 msgr2=0x7fdaa006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:33.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.638+0000 7fdac7000640 1 --2- 172.21.15.28:0/1073406007 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdaa006b500 0x7fdaa006d9c0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fdac01e6900 tx=0x7fdab4009040 comp rx=0 tx=0).stop 2024-02-21T23:12:33.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.638+0000 7fdac7000640 1 -- 172.21.15.28:0/1073406007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdac01e9f90 msgr2=0x7fdac01e9540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:33.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.638+0000 7fdac7000640 1 --2- 172.21.15.28:0/1073406007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdac01e9f90 0x7fdac01e9540 secure :-1 s=READY pgs=2258 cs=0 l=1 rev1=1 crypto rx=0x7fdaac000c00 tx=0x7fdaac004730 comp rx=0 tx=0).stop 2024-02-21T23:12:33.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.638+0000 7fdac7000640 1 -- 172.21.15.28:0/1073406007 shutdown_connections 2024-02-21T23:12:33.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.638+0000 7fdac7000640 1 --2- 172.21.15.28:0/1073406007 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdaa006b500 0x7fdaa006d9c0 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:33.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.638+0000 7fdac7000640 1 --2- 172.21.15.28:0/1073406007 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdac01ec800 0x7fdac0146c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:33.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.638+0000 7fdac7000640 1 --2- 172.21.15.28:0/1073406007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdac01e9f90 0x7fdac01e9540 unknown :-1 s=CLOSED pgs=2258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:33.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.638+0000 7fdac7000640 1 --2- 172.21.15.28:0/1073406007 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdaa8004800 0x7fdac01448e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:33.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.638+0000 7fdac7000640 1 -- 172.21.15.28:0/1073406007 >> 172.21.15.28:0/1073406007 conn(0x7fdac00a8320 msgr2=0x7fdac00c5980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:33.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.638+0000 7fdac7000640 1 -- 172.21.15.28:0/1073406007 shutdown_connections 2024-02-21T23:12:33.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.638+0000 7fdac7000640 1 -- 172.21.15.28:0/1073406007 wait complete. 2024-02-21T23:12:33.644 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs down true 2024-02-21T23:12:33.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.871+0000 7ff07d000640 1 Processor -- start 2024-02-21T23:12:33.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.871+0000 7ff07d000640 1 -- start start 2024-02-21T23:12:33.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.873+0000 7ff07d000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7ff0780c4580 0x7ff0780c12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:33.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.873+0000 7ff07d000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff0780c2c10 0x7ff07806ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:33.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.873+0000 7ff07d000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff07806e220 con 0x7ff078150240 2024-02-21T23:12:33.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.873+0000 7ff07d000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff078150620 con 0x7ff0780c4580 2024-02-21T23:12:33.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.873+0000 7ff07d000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff078153a90 con 0x7ff0780c2c10 2024-02-21T23:12:33.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.873+0000 7ff077400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff078150240 0x7ff07806db10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:32972/0 (socket says 172.21.15.28:32972) 2024-02-21T23:12:33.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.873+0000 7ff077400640 1 -- 172.21.15.28:0/3108836138 learned_addr learned my addr 172.21.15.28:0/3108836138 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:12:33.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.874+0000 7ff077e00640 1 --2- 172.21.15.28:0/3108836138 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff0780c2c10 0x7ff07806ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:33.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.874+0000 7ff076a00640 1 -- 172.21.15.28:0/3108836138 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4125544262 0 0) 0x7ff07806e220 con 0x7ff078150240 2024-02-21T23:12:33.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.874+0000 7ff076a00640 1 -- 172.21.15.28:0/3108836138 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff05c003520 con 0x7ff078150240 2024-02-21T23:12:33.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.874+0000 7ff06fe00640 1 --2- 172.21.15.28:0/3108836138 >> v2:172.21.15.81:3300/0 conn(0x7ff0780c4580 0x7ff0780c12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:33.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.874+0000 7ff076a00640 1 -- 172.21.15.28:0/3108836138 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2857962377 0 0) 0x7ff05c003520 con 0x7ff078150240 2024-02-21T23:12:33.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.874+0000 7ff076a00640 1 -- 172.21.15.28:0/3108836138 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff0780c2c10 msgr2=0x7ff07806ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:33.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.874+0000 7ff076a00640 1 --2- 172.21.15.28:0/3108836138 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff0780c2c10 0x7ff07806ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:33.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.874+0000 7ff076a00640 1 -- 172.21.15.28:0/3108836138 >> v2:172.21.15.81:3300/0 conn(0x7ff0780c4580 msgr2=0x7ff0780c12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:33.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.874+0000 7ff076a00640 1 --2- 172.21.15.28:0/3108836138 >> v2:172.21.15.81:3300/0 conn(0x7ff0780c4580 0x7ff0780c12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:33.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.874+0000 7ff076a00640 1 -- 172.21.15.28:0/3108836138 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff07806f780 con 0x7ff078150240 2024-02-21T23:12:33.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.874+0000 7ff076a00640 1 -- 172.21.15.28:0/3108836138 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7ff068001670 con 0x7ff078150240 2024-02-21T23:12:33.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.874+0000 7ff077e00640 1 --2- 172.21.15.28:0/3108836138 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff0780c2c10 0x7ff07806ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:12:33.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.874+0000 7ff076a00640 1 -- 172.21.15.28:0/3108836138 >> v1:172.21.15.28:6789/0 conn(0x7ff078150240 legacy=0x7ff07806db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:33.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.874+0000 7ff076a00640 1 --2- 172.21.15.28:0/3108836138 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff05c0039b0 0x7ff05c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:33.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.874+0000 7ff076a00640 1 --2- 172.21.15.28:0/3108836138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff05c004800 0x7ff05c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:33.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.874+0000 7ff076a00640 1 --2- 172.21.15.28:0/3108836138 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff0780c2c10 0x7ff05c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:33.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.874+0000 7ff076a00640 1 -- 172.21.15.28:0/3108836138 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff078153a90 con 0x7ff05c004800 2024-02-21T23:12:33.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.874+0000 7ff076a00640 1 -- 172.21.15.28:0/3108836138 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff05c00baa0 con 0x7ff05c0039b0 2024-02-21T23:12:33.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.874+0000 7ff076a00640 1 -- 172.21.15.28:0/3108836138 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff05c00bca0 con 0x7ff0780c2c10 2024-02-21T23:12:33.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.875+0000 7ff06fe00640 1 --2- 172.21.15.28:0/3108836138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff05c004800 0x7ff05c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:33.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.875+0000 7ff077400640 1 --2- 172.21.15.28:0/3108836138 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff05c0039b0 0x7ff05c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:33.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.875+0000 7ff077e00640 1 --2- 172.21.15.28:0/3108836138 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff0780c2c10 0x7ff05c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:33.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.875+0000 7ff06fe00640 1 -- 172.21.15.28:0/3108836138 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff0780c2c10 msgr2=0x7ff05c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:33.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.875+0000 7ff06fe00640 1 --2- 172.21.15.28:0/3108836138 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff0780c2c10 0x7ff05c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:33.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.875+0000 7ff06fe00640 1 -- 172.21.15.28:0/3108836138 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff05c0039b0 msgr2=0x7ff05c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:33.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.875+0000 7ff06fe00640 1 --2- 172.21.15.28:0/3108836138 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff05c0039b0 0x7ff05c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:33.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.875+0000 7ff06fe00640 1 -- 172.21.15.28:0/3108836138 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff05c003520 con 0x7ff05c004800 2024-02-21T23:12:33.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.877+0000 7ff06fe00640 1 --2- 172.21.15.28:0/3108836138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff05c004800 0x7ff05c004c80 secure :-1 s=READY pgs=2264 cs=0 l=1 rev1=1 crypto rx=0x7ff0780c49c0 tx=0x7ff05c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56f5ada55ae14223 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:33.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.879+0000 7ff076a00640 1 -- 172.21.15.28:0/3108836138 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff060036020 con 0x7ff05c004800 2024-02-21T23:12:33.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.879+0000 7ff076a00640 1 -- 172.21.15.28:0/3108836138 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0600325f0 con 0x7ff05c004800 2024-02-21T23:12:33.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.879+0000 7ff076a00640 1 -- 172.21.15.28:0/3108836138 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff06000b600 con 0x7ff05c004800 2024-02-21T23:12:33.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.880+0000 7ff07d000640 1 -- 172.21.15.28:0/3108836138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff05c004800 msgr2=0x7ff05c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:33.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.880+0000 7ff07d000640 1 --2- 172.21.15.28:0/3108836138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff05c004800 0x7ff05c004c80 secure :-1 s=READY pgs=2264 cs=0 l=1 rev1=1 crypto rx=0x7ff0780c49c0 tx=0x7ff05c005830 comp rx=0 tx=0).stop 2024-02-21T23:12:33.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.880+0000 7ff077e00640 1 -- 172.21.15.28:0/3108836138 reap_dead start 2024-02-21T23:12:33.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.880+0000 7ff07d000640 1 -- 172.21.15.28:0/3108836138 shutdown_connections 2024-02-21T23:12:33.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.880+0000 7ff07d000640 1 -- 172.21.15.28:0/3108836138 >> 172.21.15.28:0/3108836138 conn(0x7ff0780a8530 msgr2=0x7ff0780a8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:33.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.880+0000 7ff07d000640 1 -- 172.21.15.28:0/3108836138 shutdown_connections 2024-02-21T23:12:33.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.880+0000 7ff07d000640 1 -- 172.21.15.28:0/3108836138 wait complete. 2024-02-21T23:12:33.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.882+0000 7ff07d000640 1 Processor -- start 2024-02-21T23:12:33.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.882+0000 7ff07d000640 1 -- start start 2024-02-21T23:12:33.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.883+0000 7ff07d000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff05c004800 0x7ff0781ee5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:33.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.883+0000 7ff07d000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff07814f340 0x7ff0780a7640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:33.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.883+0000 7ff07d000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0781eb390 0x7ff0781e8680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:33.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.883+0000 7ff07d000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0780c4a10 con 0x7ff05c004800 2024-02-21T23:12:33.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.883+0000 7ff07d000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff078153720 con 0x7ff0781eb390 2024-02-21T23:12:33.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.883+0000 7ff07d000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0780c4b90 con 0x7ff07814f340 2024-02-21T23:12:33.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.883+0000 7ff077400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff05c004800 0x7ff0781ee5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:33.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.883+0000 7ff077400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff05c004800 0x7ff0781ee5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44478/0 (socket says 172.21.15.28:44478) 2024-02-21T23:12:33.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.883+0000 7ff077400640 1 -- 172.21.15.28:0/892957051 learned_addr learned my addr 172.21.15.28:0/892957051 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:33.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.884+0000 7ff077e00640 1 --2- 172.21.15.28:0/892957051 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0781eb390 0x7ff0781e8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:33.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.884+0000 7ff06fe00640 1 --2- 172.21.15.28:0/892957051 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff07814f340 0x7ff0780a7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:33.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.884+0000 7ff077400640 1 -- 172.21.15.28:0/892957051 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff07814f340 msgr2=0x7ff0780a7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:33.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.884+0000 7ff077400640 1 --2- 172.21.15.28:0/892957051 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff07814f340 0x7ff0780a7640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:33.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.884+0000 7ff077400640 1 -- 172.21.15.28:0/892957051 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0781eb390 msgr2=0x7ff0781e8680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:33.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.884+0000 7ff077400640 1 --2- 172.21.15.28:0/892957051 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0781eb390 0x7ff0781e8680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:33.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.884+0000 7ff077400640 1 -- 172.21.15.28:0/892957051 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0781eeb10 con 0x7ff05c004800 2024-02-21T23:12:33.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.884+0000 7ff077400640 1 --2- 172.21.15.28:0/892957051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff05c004800 0x7ff0781ee5d0 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 crypto rx=0x7ff06800e7b0 tx=0x7ff06800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:33.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.884+0000 7ff06f400640 1 -- 172.21.15.28:0/892957051 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff06800c870 con 0x7ff05c004800 2024-02-21T23:12:33.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.885+0000 7ff06f400640 1 -- 172.21.15.28:0/892957051 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff068018070 con 0x7ff05c004800 2024-02-21T23:12:33.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.885+0000 7ff06f400640 1 -- 172.21.15.28:0/892957051 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff06800cc50 con 0x7ff05c004800 2024-02-21T23:12:33.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.885+0000 7ff07d000640 1 -- 172.21.15.28:0/892957051 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff07814f830 con 0x7ff05c004800 2024-02-21T23:12:33.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.885+0000 7ff07d000640 1 -- 172.21.15.28:0/892957051 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0781eab90 con 0x7ff05c004800 2024-02-21T23:12:33.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.886+0000 7ff07d000640 1 -- 172.21.15.28:0/892957051 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff07806c000 con 0x7ff05c004800 2024-02-21T23:12:33.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.887+0000 7ff06f400640 1 -- 172.21.15.28:0/892957051 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff06801d460 con 0x7ff05c004800 2024-02-21T23:12:33.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.888+0000 7ff06f400640 1 --2- 172.21.15.28:0/892957051 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff05406b500 0x7ff05406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:33.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.888+0000 7ff06f400640 1 -- 172.21.15.28:0/892957051 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff06808a150 con 0x7ff05c004800 2024-02-21T23:12:33.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.889+0000 7ff06fe00640 1 --2- 172.21.15.28:0/892957051 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff05406b500 0x7ff05406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:33.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.890+0000 7ff06fe00640 1 --2- 172.21.15.28:0/892957051 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff05406b500 0x7ff05406d9c0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7ff05c0057e0 tx=0x7ff06003d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:33.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:33.893+0000 7ff06f400640 1 -- 172.21.15.28:0/892957051 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ff068058ed0 con 0x7ff05c004800 2024-02-21T23:12:34.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:34.133+0000 7ff07d000640 1 -- 172.21.15.28:0/892957051 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "down", "val": "true"} v 0) v1 -- 0x7ff07806c440 con 0x7ff05c004800 2024-02-21T23:12:34.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:34.890+0000 7ff06f400640 1 -- 172.21.15.28:0/892957051 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "down", "val": "true"}]=0 cephfs marked down. v218) v1 ==== 125+0+0 (secure 0 0 0) 0x7ff07806c440 con 0x7ff05c004800 2024-02-21T23:12:34.879 INFO:teuthology.orchestra.run.smithi028.stderr:cephfs marked down. 2024-02-21T23:12:34.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:34.894+0000 7ff07d000640 1 -- 172.21.15.28:0/892957051 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff05406b500 msgr2=0x7ff05406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:34.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:34.894+0000 7ff07d000640 1 --2- 172.21.15.28:0/892957051 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff05406b500 0x7ff05406d9c0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7ff05c0057e0 tx=0x7ff06003d040 comp rx=0 tx=0).stop 2024-02-21T23:12:34.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:34.894+0000 7ff07d000640 1 -- 172.21.15.28:0/892957051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff05c004800 msgr2=0x7ff0781ee5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:34.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:34.894+0000 7ff07d000640 1 --2- 172.21.15.28:0/892957051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff05c004800 0x7ff0781ee5d0 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 crypto rx=0x7ff06800e7b0 tx=0x7ff06800ec70 comp rx=0 tx=0).stop 2024-02-21T23:12:34.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:34.894+0000 7ff07d000640 1 -- 172.21.15.28:0/892957051 shutdown_connections 2024-02-21T23:12:34.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:34.894+0000 7ff07d000640 1 --2- 172.21.15.28:0/892957051 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff05406b500 0x7ff05406d9c0 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:34.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:34.894+0000 7ff07d000640 1 --2- 172.21.15.28:0/892957051 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff07814f340 0x7ff0780a7640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:34.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:34.894+0000 7ff07d000640 1 --2- 172.21.15.28:0/892957051 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0781eb390 0x7ff0781e8680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:34.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:34.894+0000 7ff07d000640 1 --2- 172.21.15.28:0/892957051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff05c004800 0x7ff0781ee5d0 unknown :-1 s=CLOSED pgs=2265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:34.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:34.894+0000 7ff07d000640 1 -- 172.21.15.28:0/892957051 >> 172.21.15.28:0/892957051 conn(0x7ff0780a8530 msgr2=0x7ff07806ae70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:34.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:34.895+0000 7ff07d000640 1 -- 172.21.15.28:0/892957051 shutdown_connections 2024-02-21T23:12:34.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:34.895+0000 7ff07d000640 1 -- 172.21.15.28:0/892957051 wait complete. 2024-02-21T23:12:34.907 INFO:tasks.ceph_test_case:waiting 30s for health warning matching 2024-02-21T23:12:34.907 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-02-21T23:12:35.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.143+0000 7f0fdcc00640 1 Processor -- start 2024-02-21T23:12:35.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.143+0000 7f0fdcc00640 1 -- start start 2024-02-21T23:12:35.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.144+0000 7f0fdcc00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0fd80c3280 0x7f0fd80c3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:35.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.144+0000 7f0fdcc00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0fd8155860 0x7f0fd8155d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:35.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.144+0000 7f0fdcc00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fd80c6560 con 0x7f0fd8072430 2024-02-21T23:12:35.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.144+0000 7f0fdcc00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fd80c5cc0 con 0x7f0fd8155860 2024-02-21T23:12:35.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.144+0000 7f0fdcc00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd80c5fa0 con 0x7f0fd80c3280 2024-02-21T23:12:35.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.144+0000 7f0fd6800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0fd8155860 0x7f0fd8155d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:35.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.144+0000 7f0fd6800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0fd8155860 0x7f0fd8155d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56406/0 (socket says 172.21.15.28:56406) 2024-02-21T23:12:35.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.144+0000 7f0fd6800640 1 -- 172.21.15.28:0/3595502916 learned_addr learned my addr 172.21.15.28:0/3595502916 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:35.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.144+0000 7f0fcfe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f0fd8072430 0x7f0fd81526e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41112/0 (socket says 172.21.15.28:41112) 2024-02-21T23:12:35.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.145+0000 7f0fd5e00640 1 --2- 172.21.15.28:0/3595502916 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0fd80c3280 0x7f0fd80c3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:35.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.145+0000 7f0fd6800640 1 -- 172.21.15.28:0/3595502916 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0fd80c3280 msgr2=0x7f0fd80c3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:35.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.145+0000 7f0fd6800640 1 --2- 172.21.15.28:0/3595502916 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0fd80c3280 0x7f0fd80c3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:35.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.145+0000 7f0fd6800640 1 -- 172.21.15.28:0/3595502916 >> v1:172.21.15.28:6789/0 conn(0x7f0fd8072430 legacy=0x7f0fd81526e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:35.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.145+0000 7f0fd6800640 1 -- 172.21.15.28:0/3595502916 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fd80c6280 con 0x7f0fd8155860 2024-02-21T23:12:35.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.145+0000 7f0fd5400640 1 -- 172.21.15.28:0/3595502916 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3653340861 0 0) 0x7f0fd80c6560 con 0x7f0fd8072430 2024-02-21T23:12:35.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.146+0000 7f0fd6800640 1 --2- 172.21.15.28:0/3595502916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0fd8155860 0x7f0fd8155d10 secure :-1 s=READY pgs=2267 cs=0 l=1 rev1=1 crypto rx=0x7f0fd0009e50 tx=0x7f0fd002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85686ae0fee25c6d server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:35.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.146+0000 7f0fd5400640 1 -- 172.21.15.28:0/3595502916 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fd002ca70 con 0x7f0fd8155860 2024-02-21T23:12:35.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.146+0000 7f0fd5400640 1 -- 172.21.15.28:0/3595502916 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fd002cc10 con 0x7f0fd8155860 2024-02-21T23:12:35.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.146+0000 7f0fd5400640 1 -- 172.21.15.28:0/3595502916 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fd00343f0 con 0x7f0fd8155860 2024-02-21T23:12:35.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.147+0000 7f0fdcc00640 1 -- 172.21.15.28:0/3595502916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0fd8155860 msgr2=0x7f0fd8155d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:35.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.147+0000 7f0fdcc00640 1 --2- 172.21.15.28:0/3595502916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0fd8155860 0x7f0fd8155d10 secure :-1 s=READY pgs=2267 cs=0 l=1 rev1=1 crypto rx=0x7f0fd0009e50 tx=0x7f0fd002bed0 comp rx=0 tx=0).stop 2024-02-21T23:12:35.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.147+0000 7f0fdcc00640 1 -- 172.21.15.28:0/3595502916 shutdown_connections 2024-02-21T23:12:35.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.147+0000 7f0fdcc00640 1 --2- 172.21.15.28:0/3595502916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0fd8155860 0x7f0fd8155d10 unknown :-1 s=CLOSED pgs=2267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:35.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.147+0000 7f0fdcc00640 1 --2- 172.21.15.28:0/3595502916 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0fd80c3280 0x7f0fd80c3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:35.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.147+0000 7f0fdcc00640 1 -- 172.21.15.28:0/3595502916 >> 172.21.15.28:0/3595502916 conn(0x7f0fd80c67f0 msgr2=0x7f0fd80c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:35.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.147+0000 7f0fdcc00640 1 -- 172.21.15.28:0/3595502916 shutdown_connections 2024-02-21T23:12:35.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.147+0000 7f0fdcc00640 1 -- 172.21.15.28:0/3595502916 wait complete. 2024-02-21T23:12:35.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.149+0000 7f0fdcc00640 1 Processor -- start 2024-02-21T23:12:35.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.149+0000 7f0fdcc00640 1 -- start start 2024-02-21T23:12:35.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.150+0000 7f0fdcc00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0fd8072430 0x7f0fd81dcf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:35.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.150+0000 7f0fdcc00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0fd80c3280 0x7f0fd80a77d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:35.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.150+0000 7f0fdcc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0fd8155860 0x7f0fd814bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:35.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.150+0000 7f0fdcc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd80c5aa0 con 0x7f0fd8155860 2024-02-21T23:12:35.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.150+0000 7f0fdcc00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd80c5920 con 0x7f0fd80c3280 2024-02-21T23:12:35.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.150+0000 7f0fdcc00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd80733e0 con 0x7f0fd8072430 2024-02-21T23:12:35.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.150+0000 7f0fd6800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0fd8155860 0x7f0fd814bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:35.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.150+0000 7f0fd6800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0fd8155860 0x7f0fd814bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56418/0 (socket says 172.21.15.28:56418) 2024-02-21T23:12:35.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.150+0000 7f0fd6800640 1 -- 172.21.15.28:0/3787580390 learned_addr learned my addr 172.21.15.28:0/3787580390 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:35.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.151+0000 7f0fcfe00640 1 --2- 172.21.15.28:0/3787580390 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0fd80c3280 0x7f0fd80a77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:35.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.151+0000 7f0fd5e00640 1 --2- 172.21.15.28:0/3787580390 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0fd8072430 0x7f0fd81dcf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:35.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.151+0000 7f0fd6800640 1 -- 172.21.15.28:0/3787580390 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0fd8072430 msgr2=0x7f0fd81dcf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:35.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.151+0000 7f0fd6800640 1 --2- 172.21.15.28:0/3787580390 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0fd8072430 0x7f0fd81dcf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:35.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.151+0000 7f0fd6800640 1 -- 172.21.15.28:0/3787580390 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0fd80c3280 msgr2=0x7f0fd80a77d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:35.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.151+0000 7f0fd6800640 1 --2- 172.21.15.28:0/3787580390 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0fd80c3280 0x7f0fd80a77d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:35.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.151+0000 7f0fd6800640 1 -- 172.21.15.28:0/3787580390 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fd81ebb60 con 0x7f0fd8155860 2024-02-21T23:12:35.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.151+0000 7f0fd6800640 1 --2- 172.21.15.28:0/3787580390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0fd8155860 0x7f0fd814bc90 secure :-1 s=READY pgs=2268 cs=0 l=1 rev1=1 crypto rx=0x7f0fd0036040 tx=0x7f0fd0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:35.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.151+0000 7f0fce000640 1 -- 172.21.15.28:0/3787580390 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fd0043030 con 0x7f0fd8155860 2024-02-21T23:12:35.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.152+0000 7f0fce000640 1 -- 172.21.15.28:0/3787580390 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fd003e450 con 0x7f0fd8155860 2024-02-21T23:12:35.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.152+0000 7f0fce000640 1 -- 172.21.15.28:0/3787580390 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fd0009510 con 0x7f0fd8155860 2024-02-21T23:12:35.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.152+0000 7f0fdcc00640 1 -- 172.21.15.28:0/3787580390 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fd81eafe0 con 0x7f0fd8155860 2024-02-21T23:12:35.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.152+0000 7f0fdcc00640 1 -- 172.21.15.28:0/3787580390 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fd81eb5a0 con 0x7f0fd8155860 2024-02-21T23:12:35.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.153+0000 7f0fdcc00640 1 -- 172.21.15.28:0/3787580390 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fd8067f30 con 0x7f0fd8155860 2024-02-21T23:12:35.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.154+0000 7f0fce000640 1 -- 172.21.15.28:0/3787580390 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0fd0046070 con 0x7f0fd8155860 2024-02-21T23:12:35.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.155+0000 7f0fce000640 1 --2- 172.21.15.28:0/3787580390 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0fb406b4d0 0x7f0fb406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:35.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.155+0000 7f0fce000640 1 -- 172.21.15.28:0/3787580390 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f0fd00ac880 con 0x7f0fd8155860 2024-02-21T23:12:35.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.155+0000 7f0fd5e00640 1 --2- 172.21.15.28:0/3787580390 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0fb406b4d0 0x7f0fb406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:35.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.156+0000 7f0fd5e00640 1 --2- 172.21.15.28:0/3787580390 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0fb406b4d0 0x7f0fb406d990 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f0fc8006fd0 tx=0x7f0fc8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:35.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.159+0000 7f0fce000640 1 -- 172.21.15.28:0/3787580390 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f0fd004d070 con 0x7f0fd8155860 2024-02-21T23:12:35.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.403+0000 7f0fdcc00640 1 -- 172.21.15.28:0/3787580390 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f0fd806f2d0 con 0x7f0fd8155860 2024-02-21T23:12:35.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.403+0000 7f0fce000640 1 -- 172.21.15.28:0/3787580390 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f0fd007b560 con 0x7f0fd8155860 2024-02-21T23:12:35.392 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:12:35.392 INFO:teuthology.orchestra.run.smithi028.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-02-21T23:12:35.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.406+0000 7f0fdcc00640 1 -- 172.21.15.28:0/3787580390 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0fb406b4d0 msgr2=0x7f0fb406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:35.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.406+0000 7f0fdcc00640 1 --2- 172.21.15.28:0/3787580390 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0fb406b4d0 0x7f0fb406d990 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f0fc8006fd0 tx=0x7f0fc8008040 comp rx=0 tx=0).stop 2024-02-21T23:12:35.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.406+0000 7f0fdcc00640 1 -- 172.21.15.28:0/3787580390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0fd8155860 msgr2=0x7f0fd814bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:35.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.406+0000 7f0fdcc00640 1 --2- 172.21.15.28:0/3787580390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0fd8155860 0x7f0fd814bc90 secure :-1 s=READY pgs=2268 cs=0 l=1 rev1=1 crypto rx=0x7f0fd0036040 tx=0x7f0fd0004a40 comp rx=0 tx=0).stop 2024-02-21T23:12:35.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.406+0000 7f0fdcc00640 1 -- 172.21.15.28:0/3787580390 shutdown_connections 2024-02-21T23:12:35.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.406+0000 7f0fdcc00640 1 --2- 172.21.15.28:0/3787580390 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0fb406b4d0 0x7f0fb406d990 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:35.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.406+0000 7f0fdcc00640 1 --2- 172.21.15.28:0/3787580390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0fd8155860 0x7f0fd814bc90 unknown :-1 s=CLOSED pgs=2268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:35.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.406+0000 7f0fdcc00640 1 --2- 172.21.15.28:0/3787580390 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0fd80c3280 0x7f0fd80a77d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:35.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.406+0000 7f0fdcc00640 1 --2- 172.21.15.28:0/3787580390 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0fd8072430 0x7f0fd81dcf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:35.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.406+0000 7f0fdcc00640 1 -- 172.21.15.28:0/3787580390 >> 172.21.15.28:0/3787580390 conn(0x7f0fd80c67f0 msgr2=0x7f0fd8153050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:35.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.406+0000 7f0fdcc00640 1 -- 172.21.15.28:0/3787580390 shutdown_connections 2024-02-21T23:12:35.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:35.406+0000 7f0fdcc00640 1 -- 172.21.15.28:0/3787580390 wait complete. 2024-02-21T23:12:35.415 DEBUG:tasks.ceph_test_case:Not expected number of summary strings ([]) 2024-02-21T23:12:35.415 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-02-21T23:12:37.931 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-21T23:12:37.940 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-21T23:12:40.416 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-02-21T23:12:40.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.657+0000 7f4d76400640 1 Processor -- start 2024-02-21T23:12:40.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.657+0000 7f4d76400640 1 -- start start 2024-02-21T23:12:40.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.659+0000 7f4d76400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f4d700c11f0 0x7f4d70153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:40.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.659+0000 7f4d76400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d70152e70 con 0x7f4d700c15d0 2024-02-21T23:12:40.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.659+0000 7f4d76400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d70153150 con 0x7f4d7015b6d0 2024-02-21T23:12:40.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.659+0000 7f4d76400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d700c1a60 con 0x7f4d700c11f0 2024-02-21T23:12:40.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.659+0000 7f4d66a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f4d700c15d0 0x7f4d7006dce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41114/0 (socket says 172.21.15.28:41114) 2024-02-21T23:12:40.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.659+0000 7f4d66a00640 1 -- 172.21.15.28:0/1190909365 learned_addr learned my addr 172.21.15.28:0/1190909365 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:12:40.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.659+0000 7f4d6f400640 1 --2- 172.21.15.28:0/1190909365 >> v2:172.21.15.81:3300/0 conn(0x7f4d700c11f0 0x7f4d70153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:40.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.660+0000 7f4d6ea00640 1 -- 172.21.15.28:0/1190909365 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2768877256 0 0) 0x7f4d70152e70 con 0x7f4d700c15d0 2024-02-21T23:12:40.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.660+0000 7f4d6ea00640 1 -- 172.21.15.28:0/1190909365 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d54003520 con 0x7f4d700c15d0 2024-02-21T23:12:40.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.660+0000 7f4d6ea00640 1 -- 172.21.15.28:0/1190909365 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2777594272 0 0) 0x7f4d54003520 con 0x7f4d700c15d0 2024-02-21T23:12:40.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.660+0000 7f4d6ea00640 1 -- 172.21.15.28:0/1190909365 >> v2:172.21.15.81:3300/0 conn(0x7f4d700c11f0 msgr2=0x7f4d70153950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:40.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.660+0000 7f4d6ea00640 1 --2- 172.21.15.28:0/1190909365 >> v2:172.21.15.81:3300/0 conn(0x7f4d700c11f0 0x7f4d70153950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:40.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.660+0000 7f4d6ea00640 1 -- 172.21.15.28:0/1190909365 >> v1:172.21.15.81:6789/0 conn(0x7f4d7015b6d0 legacy=0x7f4d7015dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:40.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.660+0000 7f4d6ea00640 1 -- 172.21.15.28:0/1190909365 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d70153430 con 0x7f4d700c15d0 2024-02-21T23:12:40.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.660+0000 7f4d6ea00640 1 -- 172.21.15.28:0/1190909365 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f4d58001670 con 0x7f4d700c15d0 2024-02-21T23:12:40.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.660+0000 7f4d6ea00640 1 -- 172.21.15.28:0/1190909365 >> v1:172.21.15.28:6789/0 conn(0x7f4d700c15d0 legacy=0x7f4d7006dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:40.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.660+0000 7f4d6ea00640 1 --2- 172.21.15.28:0/1190909365 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4d540039b0 0x7f4d54003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:40.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.660+0000 7f4d6ea00640 1 --2- 172.21.15.28:0/1190909365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d54004800 0x7f4d54004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:40.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.660+0000 7f4d6ea00640 1 --2- 172.21.15.28:0/1190909365 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d540052e0 0x7f4d5400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:40.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.660+0000 7f4d6ea00640 1 -- 172.21.15.28:0/1190909365 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d700c1a60 con 0x7f4d54004800 2024-02-21T23:12:40.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.660+0000 7f4d6ea00640 1 -- 172.21.15.28:0/1190909365 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d5400c1b0 con 0x7f4d540052e0 2024-02-21T23:12:40.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.660+0000 7f4d6ea00640 1 -- 172.21.15.28:0/1190909365 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d5400c3b0 con 0x7f4d540039b0 2024-02-21T23:12:40.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.661+0000 7f4d66a00640 1 --2- 172.21.15.28:0/1190909365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d54004800 0x7f4d54004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:40.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.661+0000 7f4d6f400640 1 --2- 172.21.15.28:0/1190909365 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4d540039b0 0x7f4d54003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:40.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.661+0000 7f4d6fe00640 1 --2- 172.21.15.28:0/1190909365 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d540052e0 0x7f4d5400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:40.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.661+0000 7f4d66a00640 1 -- 172.21.15.28:0/1190909365 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4d540039b0 msgr2=0x7f4d54003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:40.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.661+0000 7f4d66a00640 1 --2- 172.21.15.28:0/1190909365 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4d540039b0 0x7f4d54003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:40.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.661+0000 7f4d66a00640 1 -- 172.21.15.28:0/1190909365 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d540052e0 msgr2=0x7f4d5400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:40.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.661+0000 7f4d66a00640 1 --2- 172.21.15.28:0/1190909365 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d540052e0 0x7f4d5400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:40.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.661+0000 7f4d66a00640 1 -- 172.21.15.28:0/1190909365 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d54003520 con 0x7f4d54004800 2024-02-21T23:12:40.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.661+0000 7f4d6fe00640 1 -- 172.21.15.28:0/1190909365 reap_dead start 2024-02-21T23:12:40.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.662+0000 7f4d66a00640 1 --2- 172.21.15.28:0/1190909365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d54004800 0x7f4d54004c80 secure :-1 s=READY pgs=2270 cs=0 l=1 rev1=1 crypto rx=0x7f4d5800c210 tx=0x7f4d54005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0df19cd8ab504a4 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:40.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.663+0000 7f4d6ea00640 1 -- 172.21.15.28:0/1190909365 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d58007cf0 con 0x7f4d54004800 2024-02-21T23:12:40.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.663+0000 7f4d6ea00640 1 -- 172.21.15.28:0/1190909365 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d58030070 con 0x7f4d54004800 2024-02-21T23:12:40.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.663+0000 7f4d6ea00640 1 -- 172.21.15.28:0/1190909365 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d58005d70 con 0x7f4d54004800 2024-02-21T23:12:40.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.663+0000 7f4d76400640 1 -- 172.21.15.28:0/1190909365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d54004800 msgr2=0x7f4d54004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:40.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.663+0000 7f4d76400640 1 --2- 172.21.15.28:0/1190909365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d54004800 0x7f4d54004c80 secure :-1 s=READY pgs=2270 cs=0 l=1 rev1=1 crypto rx=0x7f4d5800c210 tx=0x7f4d54005830 comp rx=0 tx=0).stop 2024-02-21T23:12:40.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.663+0000 7f4d76400640 1 -- 172.21.15.28:0/1190909365 shutdown_connections 2024-02-21T23:12:40.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.663+0000 7f4d76400640 1 --2- 172.21.15.28:0/1190909365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d54004800 0x7f4d54004c80 unknown :-1 s=CLOSED pgs=2270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:40.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.663+0000 7f4d76400640 1 -- 172.21.15.28:0/1190909365 >> 172.21.15.28:0/1190909365 conn(0x7f4d700c67f0 msgr2=0x7f4d700c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:40.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.663+0000 7f4d76400640 1 -- 172.21.15.28:0/1190909365 shutdown_connections 2024-02-21T23:12:40.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.664+0000 7f4d76400640 1 -- 172.21.15.28:0/1190909365 wait complete. 2024-02-21T23:12:40.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.665+0000 7f4d76400640 1 Processor -- start 2024-02-21T23:12:40.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.665+0000 7f4d76400640 1 -- start start 2024-02-21T23:12:40.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.666+0000 7f4d76400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d54004800 0x7f4d70143520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:40.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.666+0000 7f4d76400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d70143a60 0x7f4d700bfc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:40.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.666+0000 7f4d76400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4d700c0160 0x7f4d70143fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:40.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.666+0000 7f4d76400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d700c4c00 con 0x7f4d54004800 2024-02-21T23:12:40.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.666+0000 7f4d76400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d700a5ec0 con 0x7f4d70143a60 2024-02-21T23:12:40.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.666+0000 7f4d76400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d700a6110 con 0x7f4d700c0160 2024-02-21T23:12:40.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.667+0000 7f4d6f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d54004800 0x7f4d70143520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:40.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.667+0000 7f4d6f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d54004800 0x7f4d70143520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56450/0 (socket says 172.21.15.28:56450) 2024-02-21T23:12:40.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.667+0000 7f4d6f400640 1 -- 172.21.15.28:0/2007246066 learned_addr learned my addr 172.21.15.28:0/2007246066 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:40.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.667+0000 7f4d66a00640 1 --2- 172.21.15.28:0/2007246066 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d70143a60 0x7f4d700bfc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:40.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.667+0000 7f4d6fe00640 1 --2- 172.21.15.28:0/2007246066 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4d700c0160 0x7f4d70143fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:40.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.667+0000 7f4d6f400640 1 -- 172.21.15.28:0/2007246066 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4d700c0160 msgr2=0x7f4d70143fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:40.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.667+0000 7f4d6f400640 1 --2- 172.21.15.28:0/2007246066 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4d700c0160 0x7f4d70143fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:40.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.667+0000 7f4d6f400640 1 -- 172.21.15.28:0/2007246066 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d70143a60 msgr2=0x7f4d700bfc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:40.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.667+0000 7f4d6f400640 1 --2- 172.21.15.28:0/2007246066 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d70143a60 0x7f4d700bfc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:40.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.667+0000 7f4d6f400640 1 -- 172.21.15.28:0/2007246066 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d700ac4f0 con 0x7f4d54004800 2024-02-21T23:12:40.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.667+0000 7f4d6fe00640 1 --2- 172.21.15.28:0/2007246066 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4d700c0160 0x7f4d70143fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:12:40.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.668+0000 7f4d6f400640 1 --2- 172.21.15.28:0/2007246066 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d54004800 0x7f4d70143520 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 crypto rx=0x7f4d60009f60 tx=0x7f4d60009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:40.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.668+0000 7f4d67e00640 1 -- 172.21.15.28:0/2007246066 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d6001a020 con 0x7f4d54004800 2024-02-21T23:12:40.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.668+0000 7f4d67e00640 1 -- 172.21.15.28:0/2007246066 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d60011e30 con 0x7f4d54004800 2024-02-21T23:12:40.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.668+0000 7f4d67e00640 1 -- 172.21.15.28:0/2007246066 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d600166b0 con 0x7f4d54004800 2024-02-21T23:12:40.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.668+0000 7f4d76400640 1 -- 172.21.15.28:0/2007246066 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d701ee260 con 0x7f4d54004800 2024-02-21T23:12:40.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.668+0000 7f4d76400640 1 -- 172.21.15.28:0/2007246066 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d701ee820 con 0x7f4d54004800 2024-02-21T23:12:40.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.669+0000 7f4d76400640 1 -- 172.21.15.28:0/2007246066 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d70067f30 con 0x7f4d54004800 2024-02-21T23:12:40.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.670+0000 7f4d67e00640 1 -- 172.21.15.28:0/2007246066 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4d6001d020 con 0x7f4d54004800 2024-02-21T23:12:40.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.671+0000 7f4d67e00640 1 --2- 172.21.15.28:0/2007246066 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4d4c06b500 0x7f4d4c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:40.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.671+0000 7f4d67e00640 1 -- 172.21.15.28:0/2007246066 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4d6008b1b0 con 0x7f4d54004800 2024-02-21T23:12:40.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.672+0000 7f4d66a00640 1 --2- 172.21.15.28:0/2007246066 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4d4c06b500 0x7f4d4c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:40.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.673+0000 7f4d66a00640 1 --2- 172.21.15.28:0/2007246066 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4d4c06b500 0x7f4d4c06d9c0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f4d58034000 tx=0x7f4d58007000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:40.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.677+0000 7f4d67e00640 1 -- 172.21.15.28:0/2007246066 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4d60024020 con 0x7f4d54004800 2024-02-21T23:12:40.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.923+0000 7f4d76400640 1 -- 172.21.15.28:0/2007246066 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4d700c0960 con 0x7f4d54004800 2024-02-21T23:12:40.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.924+0000 7f4d67e00640 1 -- 172.21.15.28:0/2007246066 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f4d60059e90 con 0x7f4d54004800 2024-02-21T23:12:40.911 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:12:40.912 INFO:teuthology.orchestra.run.smithi028.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-02-21T23:12:40.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.927+0000 7f4d76400640 1 -- 172.21.15.28:0/2007246066 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4d4c06b500 msgr2=0x7f4d4c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:40.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.927+0000 7f4d76400640 1 --2- 172.21.15.28:0/2007246066 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4d4c06b500 0x7f4d4c06d9c0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f4d58034000 tx=0x7f4d58007000 comp rx=0 tx=0).stop 2024-02-21T23:12:40.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.927+0000 7f4d76400640 1 -- 172.21.15.28:0/2007246066 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d54004800 msgr2=0x7f4d70143520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:40.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.927+0000 7f4d76400640 1 --2- 172.21.15.28:0/2007246066 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d54004800 0x7f4d70143520 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 crypto rx=0x7f4d60009f60 tx=0x7f4d60009f90 comp rx=0 tx=0).stop 2024-02-21T23:12:40.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.927+0000 7f4d76400640 1 -- 172.21.15.28:0/2007246066 shutdown_connections 2024-02-21T23:12:40.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.927+0000 7f4d76400640 1 --2- 172.21.15.28:0/2007246066 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4d4c06b500 0x7f4d4c06d9c0 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:40.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.927+0000 7f4d76400640 1 --2- 172.21.15.28:0/2007246066 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4d700c0160 0x7f4d70143fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:40.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.927+0000 7f4d76400640 1 --2- 172.21.15.28:0/2007246066 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d70143a60 0x7f4d700bfc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:40.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.927+0000 7f4d76400640 1 --2- 172.21.15.28:0/2007246066 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d54004800 0x7f4d70143520 unknown :-1 s=CLOSED pgs=2271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:40.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.927+0000 7f4d76400640 1 -- 172.21.15.28:0/2007246066 >> 172.21.15.28:0/2007246066 conn(0x7f4d700c67f0 msgr2=0x7f4d700c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:40.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.927+0000 7f4d76400640 1 -- 172.21.15.28:0/2007246066 shutdown_connections 2024-02-21T23:12:40.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:40.927+0000 7f4d76400640 1 -- 172.21.15.28:0/2007246066 wait complete. 2024-02-21T23:12:40.932 DEBUG:tasks.ceph_test_case:Not expected number of summary strings ([]) 2024-02-21T23:12:40.933 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-02-21T23:12:45.933 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-02-21T23:12:46.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.176+0000 7fb633200640 1 Processor -- start 2024-02-21T23:12:46.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.177+0000 7fb633200640 1 -- start start 2024-02-21T23:12:46.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.178+0000 7fb633200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb63414f0b0 0x7fb634155180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:46.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.178+0000 7fb633200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fb6341521c0 0x7fb634152610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:46.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.178+0000 7fb633200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb634155760 con 0x7fb6340a6160 2024-02-21T23:12:46.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.178+0000 7fb633200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb634152b50 con 0x7fb63414f0b0 2024-02-21T23:12:46.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.178+0000 7fb633200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb634151d30 con 0x7fb6341521c0 2024-02-21T23:12:46.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.178+0000 7fb631e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb63414f0b0 0x7fb634155180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:46.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.179+0000 7fb631e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb63414f0b0 0x7fb634155180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36890/0 (socket says 172.21.15.28:36890) 2024-02-21T23:12:46.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.179+0000 7fb631e00640 1 -- 172.21.15.28:0/1829478505 learned_addr learned my addr 172.21.15.28:0/1829478505 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:46.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.179+0000 7fb632800640 1 --2- 172.21.15.28:0/1829478505 >> v2:172.21.15.81:3300/0 conn(0x7fb6341521c0 0x7fb634152610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:46.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.179+0000 7fb631e00640 1 -- 172.21.15.28:0/1829478505 >> v2:172.21.15.81:3300/0 conn(0x7fb6341521c0 msgr2=0x7fb634152610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:46.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.179+0000 7fb631e00640 1 --2- 172.21.15.28:0/1829478505 >> v2:172.21.15.81:3300/0 conn(0x7fb6341521c0 0x7fb634152610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:46.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.179+0000 7fb631e00640 1 -- 172.21.15.28:0/1829478505 >> v1:172.21.15.28:6789/0 conn(0x7fb6340a6160 legacy=0x7fb6340c2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:46.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.179+0000 7fb631e00640 1 -- 172.21.15.28:0/1829478505 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb634155a40 con 0x7fb63414f0b0 2024-02-21T23:12:46.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.179+0000 7fb632800640 1 --2- 172.21.15.28:0/1829478505 >> v2:172.21.15.81:3300/0 conn(0x7fb6341521c0 0x7fb634152610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:12:46.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.180+0000 7fb631e00640 1 --2- 172.21.15.28:0/1829478505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb63414f0b0 0x7fb634155180 secure :-1 s=READY pgs=2273 cs=0 l=1 rev1=1 crypto rx=0x7fb634151eb0 tx=0x7fb62402e170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3b5416972e14ea5 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:46.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.180+0000 7fb631400640 1 -- 172.21.15.28:0/1829478505 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb62403b070 con 0x7fb63414f0b0 2024-02-21T23:12:46.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.181+0000 7fb631400640 1 -- 172.21.15.28:0/1829478505 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb624036440 con 0x7fb63414f0b0 2024-02-21T23:12:46.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.181+0000 7fb631400640 1 -- 172.21.15.28:0/1829478505 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb624036810 con 0x7fb63414f0b0 2024-02-21T23:12:46.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.181+0000 7fb633200640 1 -- 172.21.15.28:0/1829478505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb63414f0b0 msgr2=0x7fb634155180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:46.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.181+0000 7fb633200640 1 --2- 172.21.15.28:0/1829478505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb63414f0b0 0x7fb634155180 secure :-1 s=READY pgs=2273 cs=0 l=1 rev1=1 crypto rx=0x7fb634151eb0 tx=0x7fb62402e170 comp rx=0 tx=0).stop 2024-02-21T23:12:46.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.181+0000 7fb633200640 1 -- 172.21.15.28:0/1829478505 shutdown_connections 2024-02-21T23:12:46.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.181+0000 7fb633200640 1 --2- 172.21.15.28:0/1829478505 >> v2:172.21.15.81:3300/0 conn(0x7fb6341521c0 0x7fb634152610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:46.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.181+0000 7fb633200640 1 --2- 172.21.15.28:0/1829478505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb63414f0b0 0x7fb634155180 unknown :-1 s=CLOSED pgs=2273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:46.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.181+0000 7fb633200640 1 -- 172.21.15.28:0/1829478505 >> 172.21.15.28:0/1829478505 conn(0x7fb63406b630 msgr2=0x7fb6340c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:46.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.181+0000 7fb633200640 1 -- 172.21.15.28:0/1829478505 shutdown_connections 2024-02-21T23:12:46.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.181+0000 7fb633200640 1 -- 172.21.15.28:0/1829478505 wait complete. 2024-02-21T23:12:46.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.183+0000 7fb633200640 1 Processor -- start 2024-02-21T23:12:46.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.183+0000 7fb633200640 1 -- start start 2024-02-21T23:12:46.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.184+0000 7fb633200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6340a6160 0x7fb634150700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:46.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.184+0000 7fb633200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb63414f0b0 0x7fb634153d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:46.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.184+0000 7fb633200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb6341521c0 0x7fb6341542d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:46.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.184+0000 7fb633200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6340a68d0 con 0x7fb6340a6160 2024-02-21T23:12:46.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.184+0000 7fb633200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb634156000 con 0x7fb6341521c0 2024-02-21T23:12:46.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.184+0000 7fb633200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6340c2720 con 0x7fb63414f0b0 2024-02-21T23:12:46.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.184+0000 7fb631e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6340a6160 0x7fb634150700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:46.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.184+0000 7fb631e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6340a6160 0x7fb634150700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36896/0 (socket says 172.21.15.28:36896) 2024-02-21T23:12:46.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.184+0000 7fb631e00640 1 -- 172.21.15.28:0/329329084 learned_addr learned my addr 172.21.15.28:0/329329084 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:46.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.185+0000 7fb632800640 1 --2- 172.21.15.28:0/329329084 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb6341521c0 0x7fb6341542d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:46.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.185+0000 7fb62be00640 1 --2- 172.21.15.28:0/329329084 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb63414f0b0 0x7fb634153d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:46.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.185+0000 7fb631e00640 1 -- 172.21.15.28:0/329329084 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb63414f0b0 msgr2=0x7fb634153d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:46.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.185+0000 7fb631e00640 1 --2- 172.21.15.28:0/329329084 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb63414f0b0 0x7fb634153d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:46.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.185+0000 7fb631e00640 1 -- 172.21.15.28:0/329329084 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb6341521c0 msgr2=0x7fb6341542d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:46.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.185+0000 7fb631e00640 1 --2- 172.21.15.28:0/329329084 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb6341521c0 0x7fb6341542d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:46.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.185+0000 7fb631e00640 1 -- 172.21.15.28:0/329329084 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb634154810 con 0x7fb6340a6160 2024-02-21T23:12:46.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.185+0000 7fb632800640 1 --2- 172.21.15.28:0/329329084 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb6341521c0 0x7fb6341542d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:12:46.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.185+0000 7fb631e00640 1 --2- 172.21.15.28:0/329329084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6340a6160 0x7fb634150700 secure :-1 s=READY pgs=2274 cs=0 l=1 rev1=1 crypto rx=0x7fb62402e1b0 tx=0x7fb6240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:46.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.186+0000 7fb62a000640 1 -- 172.21.15.28:0/329329084 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb62403b030 con 0x7fb6340a6160 2024-02-21T23:12:46.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.186+0000 7fb62a000640 1 -- 172.21.15.28:0/329329084 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb62403d070 con 0x7fb6340a6160 2024-02-21T23:12:46.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.186+0000 7fb62a000640 1 -- 172.21.15.28:0/329329084 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb624034bc0 con 0x7fb6340a6160 2024-02-21T23:12:46.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.186+0000 7fb633200640 1 -- 172.21.15.28:0/329329084 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6341eae60 con 0x7fb6340a6160 2024-02-21T23:12:46.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.186+0000 7fb633200640 1 -- 172.21.15.28:0/329329084 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6341532b0 con 0x7fb6340a6160 2024-02-21T23:12:46.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.187+0000 7fb633200640 1 -- 172.21.15.28:0/329329084 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb63404f290 con 0x7fb6340a6160 2024-02-21T23:12:46.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.188+0000 7fb62a000640 1 -- 172.21.15.28:0/329329084 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb624048060 con 0x7fb6340a6160 2024-02-21T23:12:46.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.189+0000 7fb62a000640 1 --2- 172.21.15.28:0/329329084 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb61006b4d0 0x7fb61006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:46.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.189+0000 7fb62a000640 1 -- 172.21.15.28:0/329329084 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb6240ac0c0 con 0x7fb6340a6160 2024-02-21T23:12:46.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.190+0000 7fb62be00640 1 --2- 172.21.15.28:0/329329084 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb61006b4d0 0x7fb61006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:46.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.191+0000 7fb62be00640 1 --2- 172.21.15.28:0/329329084 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb61006b4d0 0x7fb61006d990 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fb634155690 tx=0x7fb61c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:46.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.194+0000 7fb62a000640 1 -- 172.21.15.28:0/329329084 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb62407ada0 con 0x7fb6340a6160 2024-02-21T23:12:46.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.428+0000 7fb633200640 1 -- 172.21.15.28:0/329329084 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fb6340c67c0 con 0x7fb6340a6160 2024-02-21T23:12:46.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.428+0000 7fb62a000640 1 -- 172.21.15.28:0/329329084 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fb62407a6a0 con 0x7fb6340a6160 2024-02-21T23:12:46.416 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:12:46.416 INFO:teuthology.orchestra.run.smithi028.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-02-21T23:12:46.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.431+0000 7fb633200640 1 -- 172.21.15.28:0/329329084 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb61006b4d0 msgr2=0x7fb61006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:46.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.431+0000 7fb633200640 1 --2- 172.21.15.28:0/329329084 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb61006b4d0 0x7fb61006d990 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fb634155690 tx=0x7fb61c009040 comp rx=0 tx=0).stop 2024-02-21T23:12:46.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.431+0000 7fb633200640 1 -- 172.21.15.28:0/329329084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6340a6160 msgr2=0x7fb634150700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:46.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.431+0000 7fb633200640 1 --2- 172.21.15.28:0/329329084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6340a6160 0x7fb634150700 secure :-1 s=READY pgs=2274 cs=0 l=1 rev1=1 crypto rx=0x7fb62402e1b0 tx=0x7fb6240046c0 comp rx=0 tx=0).stop 2024-02-21T23:12:46.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.431+0000 7fb633200640 1 -- 172.21.15.28:0/329329084 shutdown_connections 2024-02-21T23:12:46.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.431+0000 7fb633200640 1 --2- 172.21.15.28:0/329329084 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb61006b4d0 0x7fb61006d990 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:46.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.431+0000 7fb633200640 1 --2- 172.21.15.28:0/329329084 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb63414f0b0 0x7fb634153d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:46.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.431+0000 7fb633200640 1 --2- 172.21.15.28:0/329329084 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb6341521c0 0x7fb6341542d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:46.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.431+0000 7fb633200640 1 --2- 172.21.15.28:0/329329084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6340a6160 0x7fb634150700 unknown :-1 s=CLOSED pgs=2274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:46.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.431+0000 7fb633200640 1 -- 172.21.15.28:0/329329084 >> 172.21.15.28:0/329329084 conn(0x7fb63406b630 msgr2=0x7fb6340bfa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:46.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.432+0000 7fb633200640 1 -- 172.21.15.28:0/329329084 shutdown_connections 2024-02-21T23:12:46.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:46.432+0000 7fb633200640 1 -- 172.21.15.28:0/329329084 wait complete. 2024-02-21T23:12:46.437 DEBUG:tasks.ceph_test_case:Not expected number of summary strings ([]) 2024-02-21T23:12:46.437 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-02-21T23:12:46.942 INFO:tasks.ceph.mds.d.smithi081.stderr: -842> 2024-02-21T23:12:27.756+0000 7f9270400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-21T23:12:46.997 INFO:tasks.ceph.mds.d.smithi081.stdout:starting mds.d at 2024-02-21T23:12:51.438 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-02-21T23:12:51.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.677+0000 7f1ffd000640 1 Processor -- start 2024-02-21T23:12:51.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.677+0000 7f1ffd000640 1 -- start start 2024-02-21T23:12:51.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.678+0000 7f1ffd000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f1ff80c2280 0x7f1ff80a6190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:51.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.678+0000 7f1ffd000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1ff80a66d0 0x7f1ff814ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:51.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.679+0000 7f1ffd000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ff8155200 con 0x7f1ff814f460 2024-02-21T23:12:51.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.679+0000 7f1ffd000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ff8155e10 con 0x7f1ff80a66d0 2024-02-21T23:12:51.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.679+0000 7f1ffd000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ff81554e0 con 0x7f1ff80c2280 2024-02-21T23:12:51.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.679+0000 7f1ff6a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1ff80a66d0 0x7f1ff814ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:51.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.679+0000 7f1ff6a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1ff80a66d0 0x7f1ff814ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36906/0 (socket says 172.21.15.28:36906) 2024-02-21T23:12:51.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.679+0000 7f1ff6a00640 1 -- 172.21.15.28:0/3000603656 learned_addr learned my addr 172.21.15.28:0/3000603656 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:51.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.679+0000 7f1ff7400640 1 --2- 172.21.15.28:0/3000603656 >> v2:172.21.15.81:3300/0 conn(0x7f1ff80c2280 0x7f1ff80a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:51.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.679+0000 7f1ff6a00640 1 -- 172.21.15.28:0/3000603656 >> v2:172.21.15.81:3300/0 conn(0x7f1ff80c2280 msgr2=0x7f1ff80a6190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:51.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.679+0000 7f1ff6a00640 1 --2- 172.21.15.28:0/3000603656 >> v2:172.21.15.81:3300/0 conn(0x7f1ff80c2280 0x7f1ff80a6190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:51.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.679+0000 7f1ff6a00640 1 -- 172.21.15.28:0/3000603656 >> v1:172.21.15.81:6789/0 conn(0x7f1ff814f460 legacy=0x7f1ff8151b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:51.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.679+0000 7f1ff6a00640 1 -- 172.21.15.28:0/3000603656 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ff81557c0 con 0x7f1ff80a66d0 2024-02-21T23:12:51.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.681+0000 7f1ff6a00640 1 --2- 172.21.15.28:0/3000603656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ff80a66d0 0x7f1ff814ef20 secure :-1 s=READY pgs=2275 cs=0 l=1 rev1=1 crypto rx=0x7f1ff80c7020 tx=0x7f1fe402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a77b9d242b279164 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:51.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.681+0000 7f1ff6000640 1 -- 172.21.15.28:0/3000603656 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fe402e070 con 0x7f1ff80a66d0 2024-02-21T23:12:51.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.681+0000 7f1ff6000640 1 -- 172.21.15.28:0/3000603656 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fe4037070 con 0x7f1ff80a66d0 2024-02-21T23:12:51.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.681+0000 7f1ff6000640 1 -- 172.21.15.28:0/3000603656 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fe4031a00 con 0x7f1ff80a66d0 2024-02-21T23:12:51.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.682+0000 7f1ffd000640 1 -- 172.21.15.28:0/3000603656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ff80a66d0 msgr2=0x7f1ff814ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:51.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.682+0000 7f1ffd000640 1 --2- 172.21.15.28:0/3000603656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ff80a66d0 0x7f1ff814ef20 secure :-1 s=READY pgs=2275 cs=0 l=1 rev1=1 crypto rx=0x7f1ff80c7020 tx=0x7f1fe402b880 comp rx=0 tx=0).stop 2024-02-21T23:12:51.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.682+0000 7f1ffd000640 1 -- 172.21.15.28:0/3000603656 shutdown_connections 2024-02-21T23:12:51.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.682+0000 7f1ffd000640 1 --2- 172.21.15.28:0/3000603656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ff80a66d0 0x7f1ff814ef20 unknown :-1 s=CLOSED pgs=2275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:51.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.682+0000 7f1ffd000640 1 --2- 172.21.15.28:0/3000603656 >> v2:172.21.15.81:3300/0 conn(0x7f1ff80c2280 0x7f1ff80a6190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:51.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.682+0000 7f1ffd000640 1 -- 172.21.15.28:0/3000603656 >> 172.21.15.28:0/3000603656 conn(0x7f1ff806b630 msgr2=0x7f1ff80c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:51.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.682+0000 7f1ffd000640 1 -- 172.21.15.28:0/3000603656 shutdown_connections 2024-02-21T23:12:51.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.682+0000 7f1ffd000640 1 -- 172.21.15.28:0/3000603656 wait complete. 2024-02-21T23:12:51.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.684+0000 7f1ffd000640 1 Processor -- start 2024-02-21T23:12:51.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.684+0000 7f1ffd000640 1 -- start start 2024-02-21T23:12:51.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.685+0000 7f1ffd000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1ff80a66d0 0x7f1ff8148790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:51.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.685+0000 7f1ffd000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ff80c2280 0x7f1ff814a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:51.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.685+0000 7f1ffd000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ff814f460 0x7f1ff81478f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:51.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.685+0000 7f1ffd000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff8071d40 con 0x7f1ff814f460 2024-02-21T23:12:51.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.685+0000 7f1ffd000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff8071bc0 con 0x7f1ff80c2280 2024-02-21T23:12:51.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.685+0000 7f1ffd000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff80c2660 con 0x7f1ff80a66d0 2024-02-21T23:12:51.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.685+0000 7f1ff7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ff814f460 0x7f1ff81478f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:51.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.685+0000 7f1ff7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ff814f460 0x7f1ff81478f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36912/0 (socket says 172.21.15.28:36912) 2024-02-21T23:12:51.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.685+0000 7f1ff7e00640 1 -- 172.21.15.28:0/4208673046 learned_addr learned my addr 172.21.15.28:0/4208673046 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:51.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.685+0000 7f1ff6a00640 1 --2- 172.21.15.28:0/4208673046 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ff80c2280 0x7f1ff814a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:51.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.685+0000 7f1ff7400640 1 --2- 172.21.15.28:0/4208673046 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1ff80a66d0 0x7f1ff8148790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:51.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.686+0000 7f1ff7e00640 1 -- 172.21.15.28:0/4208673046 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1ff80a66d0 msgr2=0x7f1ff8148790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:51.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.686+0000 7f1ff7e00640 1 --2- 172.21.15.28:0/4208673046 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1ff80a66d0 0x7f1ff8148790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:51.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.686+0000 7f1ff7e00640 1 -- 172.21.15.28:0/4208673046 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ff80c2280 msgr2=0x7f1ff814a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:51.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.686+0000 7f1ff7e00640 1 --2- 172.21.15.28:0/4208673046 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ff80c2280 0x7f1ff814a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:51.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.686+0000 7f1ff7e00640 1 -- 172.21.15.28:0/4208673046 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ff80a84e0 con 0x7f1ff814f460 2024-02-21T23:12:51.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.686+0000 7f1ff6a00640 1 --2- 172.21.15.28:0/4208673046 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ff80c2280 0x7f1ff814a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:12:51.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.686+0000 7f1ff7e00640 1 --2- 172.21.15.28:0/4208673046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ff814f460 0x7f1ff81478f0 secure :-1 s=READY pgs=2276 cs=0 l=1 rev1=1 crypto rx=0x7f1ff000b9a0 tx=0x7f1ff000be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:51.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.686+0000 7f1fef400640 1 -- 172.21.15.28:0/4208673046 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff0019070 con 0x7f1ff814f460 2024-02-21T23:12:51.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.686+0000 7f1fef400640 1 -- 172.21.15.28:0/4208673046 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ff000cba0 con 0x7f1ff814f460 2024-02-21T23:12:51.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.686+0000 7f1ffd000640 1 -- 172.21.15.28:0/4208673046 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ff81484b0 con 0x7f1ff814f460 2024-02-21T23:12:51.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.686+0000 7f1fef400640 1 -- 172.21.15.28:0/4208673046 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff0014440 con 0x7f1ff814f460 2024-02-21T23:12:51.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.686+0000 7f1ffd000640 1 -- 172.21.15.28:0/4208673046 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ff80aa2b0 con 0x7f1ff814f460 2024-02-21T23:12:51.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.688+0000 7f1ffd000640 1 -- 172.21.15.28:0/4208673046 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ff804f290 con 0x7f1ff814f460 2024-02-21T23:12:51.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.689+0000 7f1fef400640 1 -- 172.21.15.28:0/4208673046 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1ff001c070 con 0x7f1ff814f460 2024-02-21T23:12:51.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.690+0000 7f1fef400640 1 --2- 172.21.15.28:0/4208673046 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1fd806b4d0 0x7f1fd806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:51.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.690+0000 7f1fef400640 1 -- 172.21.15.28:0/4208673046 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1ff0089ff0 con 0x7f1ff814f460 2024-02-21T23:12:51.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.690+0000 7f1ff7400640 1 --2- 172.21.15.28:0/4208673046 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1fd806b4d0 0x7f1fd806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:51.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.691+0000 7f1ff7400640 1 --2- 172.21.15.28:0/4208673046 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1fd806b4d0 0x7f1fd806d990 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f1fe8006fd0 tx=0x7f1fe8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:51.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.695+0000 7f1fef400640 1 -- 172.21.15.28:0/4208673046 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1ff0058d30 con 0x7f1ff814f460 2024-02-21T23:12:51.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.936+0000 7f1ffd000640 1 -- 172.21.15.28:0/4208673046 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f1ff81527a0 con 0x7f1ff814f460 2024-02-21T23:12:51.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.937+0000 7f1fef400640 1 -- 172.21.15.28:0/4208673046 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f1ff0058630 con 0x7f1ff814f460 2024-02-21T23:12:51.925 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:12:51.925 INFO:teuthology.orchestra.run.smithi028.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-02-21T23:12:51.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.940+0000 7f1ffd000640 1 -- 172.21.15.28:0/4208673046 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1fd806b4d0 msgr2=0x7f1fd806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:51.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.940+0000 7f1ffd000640 1 --2- 172.21.15.28:0/4208673046 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1fd806b4d0 0x7f1fd806d990 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f1fe8006fd0 tx=0x7f1fe8008040 comp rx=0 tx=0).stop 2024-02-21T23:12:51.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.940+0000 7f1ffd000640 1 -- 172.21.15.28:0/4208673046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ff814f460 msgr2=0x7f1ff81478f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:51.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.940+0000 7f1ffd000640 1 --2- 172.21.15.28:0/4208673046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ff814f460 0x7f1ff81478f0 secure :-1 s=READY pgs=2276 cs=0 l=1 rev1=1 crypto rx=0x7f1ff000b9a0 tx=0x7f1ff000be60 comp rx=0 tx=0).stop 2024-02-21T23:12:51.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.940+0000 7f1ffd000640 1 -- 172.21.15.28:0/4208673046 shutdown_connections 2024-02-21T23:12:51.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.940+0000 7f1ffd000640 1 --2- 172.21.15.28:0/4208673046 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1fd806b4d0 0x7f1fd806d990 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:51.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.940+0000 7f1ffd000640 1 --2- 172.21.15.28:0/4208673046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ff814f460 0x7f1ff81478f0 unknown :-1 s=CLOSED pgs=2276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:51.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.940+0000 7f1ffd000640 1 --2- 172.21.15.28:0/4208673046 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1ff80c2280 0x7f1ff814a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:51.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.940+0000 7f1ffd000640 1 --2- 172.21.15.28:0/4208673046 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1ff80a66d0 0x7f1ff8148790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:51.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.940+0000 7f1ffd000640 1 -- 172.21.15.28:0/4208673046 >> 172.21.15.28:0/4208673046 conn(0x7f1ff806b630 msgr2=0x7f1ff8151cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:51.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.940+0000 7f1ffd000640 1 -- 172.21.15.28:0/4208673046 shutdown_connections 2024-02-21T23:12:51.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:51.941+0000 7f1ffd000640 1 -- 172.21.15.28:0/4208673046 wait complete. 2024-02-21T23:12:51.949 DEBUG:tasks.ceph_test_case:Not expected number of summary strings ([]) 2024-02-21T23:12:51.949 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-02-21T23:12:56.950 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-02-21T23:12:57.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.184+0000 7f1a39000640 1 Processor -- start 2024-02-21T23:12:57.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.185+0000 7f1a39000640 1 -- start start 2024-02-21T23:12:57.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.186+0000 7f1a39000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f1a3406dff0 0x7f1a3406e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:57.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.186+0000 7f1a39000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1a34158580 0x7f1a3414b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:57.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.186+0000 7f1a39000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a340a8b40 con 0x7f1a340c0fe0 2024-02-21T23:12:57.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.186+0000 7f1a39000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a340a8e00 con 0x7f1a34158580 2024-02-21T23:12:57.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.186+0000 7f1a39000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a340a90e0 con 0x7f1a3406dff0 2024-02-21T23:12:57.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.187+0000 7f1a33e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1a34158580 0x7f1a3414b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:57.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.187+0000 7f1a33e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1a34158580 0x7f1a3414b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33146/0 (socket says 172.21.15.28:33146) 2024-02-21T23:12:57.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.187+0000 7f1a33e00640 1 -- 172.21.15.28:0/357916038 learned_addr learned my addr 172.21.15.28:0/357916038 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:57.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.187+0000 7f1a33400640 1 --2- 172.21.15.28:0/357916038 >> v2:172.21.15.81:3300/0 conn(0x7f1a3406dff0 0x7f1a3406e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:57.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.187+0000 7f1a33e00640 1 -- 172.21.15.28:0/357916038 >> v2:172.21.15.81:3300/0 conn(0x7f1a3406dff0 msgr2=0x7f1a3406e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:57.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.187+0000 7f1a33e00640 1 --2- 172.21.15.28:0/357916038 >> v2:172.21.15.81:3300/0 conn(0x7f1a3406dff0 0x7f1a3406e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:57.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.187+0000 7f1a33e00640 1 -- 172.21.15.28:0/357916038 >> v1:172.21.15.81:6789/0 conn(0x7f1a340c0fe0 legacy=0x7f1a34071350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:12:57.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.187+0000 7f1a33e00640 1 -- 172.21.15.28:0/357916038 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a34072ed0 con 0x7f1a34158580 2024-02-21T23:12:57.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.188+0000 7f1a33e00640 1 --2- 172.21.15.28:0/357916038 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a34158580 0x7f1a3414b2b0 secure :-1 s=READY pgs=2277 cs=0 l=1 rev1=1 crypto rx=0x7f1a3406be50 tx=0x7f1a2c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f467f64fd91beff6 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:57.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.189+0000 7f1a32a00640 1 -- 172.21.15.28:0/357916038 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a2c02ca80 con 0x7f1a34158580 2024-02-21T23:12:57.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.189+0000 7f1a32a00640 1 -- 172.21.15.28:0/357916038 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a2c02cc20 con 0x7f1a34158580 2024-02-21T23:12:57.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.189+0000 7f1a32a00640 1 -- 172.21.15.28:0/357916038 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a2c0343f0 con 0x7f1a34158580 2024-02-21T23:12:57.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.189+0000 7f1a39000640 1 -- 172.21.15.28:0/357916038 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a34158580 msgr2=0x7f1a3414b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:57.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.189+0000 7f1a39000640 1 --2- 172.21.15.28:0/357916038 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a34158580 0x7f1a3414b2b0 secure :-1 s=READY pgs=2277 cs=0 l=1 rev1=1 crypto rx=0x7f1a3406be50 tx=0x7f1a2c02bee0 comp rx=0 tx=0).stop 2024-02-21T23:12:57.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.190+0000 7f1a39000640 1 -- 172.21.15.28:0/357916038 shutdown_connections 2024-02-21T23:12:57.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.190+0000 7f1a39000640 1 --2- 172.21.15.28:0/357916038 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a34158580 0x7f1a3414b2b0 unknown :-1 s=CLOSED pgs=2277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:57.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.190+0000 7f1a39000640 1 --2- 172.21.15.28:0/357916038 >> v2:172.21.15.81:3300/0 conn(0x7f1a3406dff0 0x7f1a3406e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:57.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.190+0000 7f1a39000640 1 -- 172.21.15.28:0/357916038 >> 172.21.15.28:0/357916038 conn(0x7f1a3406b630 msgr2=0x7f1a340c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:57.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.190+0000 7f1a39000640 1 -- 172.21.15.28:0/357916038 shutdown_connections 2024-02-21T23:12:57.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.190+0000 7f1a39000640 1 -- 172.21.15.28:0/357916038 wait complete. 2024-02-21T23:12:57.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.191+0000 7f1a39000640 1 Processor -- start 2024-02-21T23:12:57.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.192+0000 7f1a39000640 1 -- start start 2024-02-21T23:12:57.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.192+0000 7f1a39000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a3406dff0 0x7f1a340aa740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:57.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.192+0000 7f1a39000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a340c0fe0 0x7f1a34154b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:57.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.193+0000 7f1a39000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1a34158580 0x7f1a341518c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:57.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.193+0000 7f1a39000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a34158980 con 0x7f1a3406dff0 2024-02-21T23:12:57.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.193+0000 7f1a39000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a34073520 con 0x7f1a340c0fe0 2024-02-21T23:12:57.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.193+0000 7f1a39000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a34158b00 con 0x7f1a34158580 2024-02-21T23:12:57.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.193+0000 7f1a33400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a3406dff0 0x7f1a340aa740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:57.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.193+0000 7f1a33400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a3406dff0 0x7f1a340aa740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33154/0 (socket says 172.21.15.28:33154) 2024-02-21T23:12:57.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.193+0000 7f1a33400640 1 -- 172.21.15.28:0/4128585643 learned_addr learned my addr 172.21.15.28:0/4128585643 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:12:57.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.193+0000 7f1a2be00640 1 --2- 172.21.15.28:0/4128585643 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a340c0fe0 0x7f1a34154b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:57.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.193+0000 7f1a33400640 1 -- 172.21.15.28:0/4128585643 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1a34158580 msgr2=0x7f1a341518c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:57.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.193+0000 7f1a33400640 1 --2- 172.21.15.28:0/4128585643 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1a34158580 0x7f1a341518c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:57.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.193+0000 7f1a33400640 1 -- 172.21.15.28:0/4128585643 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a340c0fe0 msgr2=0x7f1a34154b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:57.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.193+0000 7f1a33400640 1 --2- 172.21.15.28:0/4128585643 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a340c0fe0 0x7f1a34154b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:57.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.193+0000 7f1a33400640 1 -- 172.21.15.28:0/4128585643 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a341514a0 con 0x7f1a3406dff0 2024-02-21T23:12:57.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.194+0000 7f1a33400640 1 --2- 172.21.15.28:0/4128585643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a3406dff0 0x7f1a340aa740 secure :-1 s=READY pgs=2278 cs=0 l=1 rev1=1 crypto rx=0x7f1a2400d620 tx=0x7f1a2400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:57.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.194+0000 7f1a2b400640 1 -- 172.21.15.28:0/4128585643 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a240125f0 con 0x7f1a3406dff0 2024-02-21T23:12:57.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.194+0000 7f1a2b400640 1 -- 172.21.15.28:0/4128585643 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a24019070 con 0x7f1a3406dff0 2024-02-21T23:12:57.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.194+0000 7f1a2b400640 1 -- 172.21.15.28:0/4128585643 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a240129d0 con 0x7f1a3406dff0 2024-02-21T23:12:57.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.194+0000 7f1a39000640 1 -- 172.21.15.28:0/4128585643 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a3414f760 con 0x7f1a3406dff0 2024-02-21T23:12:57.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.194+0000 7f1a39000640 1 -- 172.21.15.28:0/4128585643 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a341550b0 con 0x7f1a3406dff0 2024-02-21T23:12:57.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.195+0000 7f1a39000640 1 -- 172.21.15.28:0/4128585643 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a3404f290 con 0x7f1a3406dff0 2024-02-21T23:12:57.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.196+0000 7f1a2b400640 1 -- 172.21.15.28:0/4128585643 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1a24012b70 con 0x7f1a3406dff0 2024-02-21T23:12:57.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.197+0000 7f1a2b400640 1 --2- 172.21.15.28:0/4128585643 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1a1006b4d0 0x7f1a1006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:12:57.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.197+0000 7f1a2b400640 1 -- 172.21.15.28:0/4128585643 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1a2408a730 con 0x7f1a3406dff0 2024-02-21T23:12:57.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.198+0000 7f1a2be00640 1 --2- 172.21.15.28:0/4128585643 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1a1006b4d0 0x7f1a1006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:12:57.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.199+0000 7f1a2be00640 1 --2- 172.21.15.28:0/4128585643 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1a1006b4d0 0x7f1a1006d990 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f1a3414fc10 tx=0x7f1a1c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:12:57.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.201+0000 7f1a2b400640 1 -- 172.21.15.28:0/4128585643 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1a24059410 con 0x7f1a3406dff0 2024-02-21T23:12:57.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.436+0000 7f1a39000640 1 -- 172.21.15.28:0/4128585643 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f1a3406ecb0 con 0x7f1a3406dff0 2024-02-21T23:12:57.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.437+0000 7f1a2b400640 1 -- 172.21.15.28:0/4128585643 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f1a24058d10 con 0x7f1a3406dff0 2024-02-21T23:12:57.425 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:12:57.426 INFO:teuthology.orchestra.run.smithi028.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-02-21T23:12:57.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.439+0000 7f1a39000640 1 -- 172.21.15.28:0/4128585643 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1a1006b4d0 msgr2=0x7f1a1006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:57.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.439+0000 7f1a39000640 1 --2- 172.21.15.28:0/4128585643 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1a1006b4d0 0x7f1a1006d990 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f1a3414fc10 tx=0x7f1a1c009040 comp rx=0 tx=0).stop 2024-02-21T23:12:57.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.439+0000 7f1a39000640 1 -- 172.21.15.28:0/4128585643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a3406dff0 msgr2=0x7f1a340aa740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:12:57.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.439+0000 7f1a39000640 1 --2- 172.21.15.28:0/4128585643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a3406dff0 0x7f1a340aa740 secure :-1 s=READY pgs=2278 cs=0 l=1 rev1=1 crypto rx=0x7f1a2400d620 tx=0x7f1a2400dae0 comp rx=0 tx=0).stop 2024-02-21T23:12:57.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.440+0000 7f1a39000640 1 -- 172.21.15.28:0/4128585643 shutdown_connections 2024-02-21T23:12:57.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.440+0000 7f1a39000640 1 --2- 172.21.15.28:0/4128585643 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1a1006b4d0 0x7f1a1006d990 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:57.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.440+0000 7f1a39000640 1 --2- 172.21.15.28:0/4128585643 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1a34158580 0x7f1a341518c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:57.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.440+0000 7f1a39000640 1 --2- 172.21.15.28:0/4128585643 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a340c0fe0 0x7f1a34154b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:57.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.440+0000 7f1a39000640 1 --2- 172.21.15.28:0/4128585643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a3406dff0 0x7f1a340aa740 unknown :-1 s=CLOSED pgs=2278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:12:57.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.440+0000 7f1a39000640 1 -- 172.21.15.28:0/4128585643 >> 172.21.15.28:0/4128585643 conn(0x7f1a3406b630 msgr2=0x7f1a34157530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:12:57.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.440+0000 7f1a39000640 1 -- 172.21.15.28:0/4128585643 shutdown_connections 2024-02-21T23:12:57.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:12:57.440+0000 7f1a39000640 1 -- 172.21.15.28:0/4128585643 wait complete. 2024-02-21T23:12:57.447 DEBUG:tasks.ceph_test_case:Not expected number of summary strings ([]) 2024-02-21T23:12:57.447 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-02-21T23:13:02.448 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-02-21T23:13:02.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.689+0000 7fe279e00640 1 Processor -- start 2024-02-21T23:13:02.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.689+0000 7fe279e00640 1 -- start start 2024-02-21T23:13:02.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.691+0000 7fe279e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fe274150450 0x7fe2740a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:02.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.691+0000 7fe279e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe27406ee40 0x7fe2741536f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:02.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.691+0000 7fe279e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2740c2760 con 0x7fe274156af0 2024-02-21T23:13:02.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.691+0000 7fe279e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2740c22c0 con 0x7fe27406ee40 2024-02-21T23:13:02.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.691+0000 7fe279e00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe274156660 con 0x7fe274150450 2024-02-21T23:13:02.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.691+0000 7fe273e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe27406ee40 0x7fe2741536f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:02.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.691+0000 7fe273e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe27406ee40 0x7fe2741536f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33170/0 (socket says 172.21.15.28:33170) 2024-02-21T23:13:02.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.691+0000 7fe273e00640 1 -- 172.21.15.28:0/404856080 learned_addr learned my addr 172.21.15.28:0/404856080 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:02.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.691+0000 7fe273e00640 1 -- 172.21.15.28:0/404856080 >> v2:172.21.15.81:3300/0 conn(0x7fe274150450 msgr2=0x7fe2740a5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:02.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.692+0000 7fe278a00640 1 --2- 172.21.15.28:0/404856080 >> v2:172.21.15.81:3300/0 conn(0x7fe274150450 0x7fe2740a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:02.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.692+0000 7fe273e00640 1 --2- 172.21.15.28:0/404856080 >> v2:172.21.15.81:3300/0 conn(0x7fe274150450 0x7fe2740a5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:02.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.692+0000 7fe273e00640 1 -- 172.21.15.28:0/404856080 >> v1:172.21.15.81:6789/0 conn(0x7fe274156af0 legacy=0x7fe27406dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:02.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.692+0000 7fe273e00640 1 -- 172.21.15.28:0/404856080 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe274153c30 con 0x7fe27406ee40 2024-02-21T23:13:02.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.692+0000 7fe278a00640 1 --2- 172.21.15.28:0/404856080 >> v2:172.21.15.81:3300/0 conn(0x7fe274150450 0x7fe2740a5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:13:02.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.693+0000 7fe273e00640 1 --2- 172.21.15.28:0/404856080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe27406ee40 0x7fe2741536f0 secure :-1 s=READY pgs=2279 cs=0 l=1 rev1=1 crypto rx=0x7fe27406f600 tx=0x7fe26002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac859f8abf46bbb0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:02.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.693+0000 7fe273400640 1 -- 172.21.15.28:0/404856080 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe260035020 con 0x7fe27406ee40 2024-02-21T23:13:02.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.693+0000 7fe273400640 1 -- 172.21.15.28:0/404856080 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe260031870 con 0x7fe27406ee40 2024-02-21T23:13:02.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.693+0000 7fe273400640 1 -- 172.21.15.28:0/404856080 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe260031c40 con 0x7fe27406ee40 2024-02-21T23:13:02.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.694+0000 7fe279e00640 1 -- 172.21.15.28:0/404856080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe27406ee40 msgr2=0x7fe2741536f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:02.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.694+0000 7fe279e00640 1 --2- 172.21.15.28:0/404856080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe27406ee40 0x7fe2741536f0 secure :-1 s=READY pgs=2279 cs=0 l=1 rev1=1 crypto rx=0x7fe27406f600 tx=0x7fe26002b910 comp rx=0 tx=0).stop 2024-02-21T23:13:02.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.694+0000 7fe279e00640 1 -- 172.21.15.28:0/404856080 shutdown_connections 2024-02-21T23:13:02.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.694+0000 7fe279e00640 1 --2- 172.21.15.28:0/404856080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe27406ee40 0x7fe2741536f0 unknown :-1 s=CLOSED pgs=2279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:02.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.694+0000 7fe279e00640 1 --2- 172.21.15.28:0/404856080 >> v2:172.21.15.81:3300/0 conn(0x7fe274150450 0x7fe2740a5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:02.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.694+0000 7fe279e00640 1 -- 172.21.15.28:0/404856080 >> 172.21.15.28:0/404856080 conn(0x7fe27406b630 msgr2=0x7fe2740c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:02.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.694+0000 7fe279e00640 1 -- 172.21.15.28:0/404856080 shutdown_connections 2024-02-21T23:13:02.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.694+0000 7fe279e00640 1 -- 172.21.15.28:0/404856080 wait complete. 2024-02-21T23:13:02.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.696+0000 7fe279e00640 1 Processor -- start 2024-02-21T23:13:02.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.696+0000 7fe279e00640 1 -- start start 2024-02-21T23:13:02.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.697+0000 7fe279e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe27406ee40 0x7fe27414c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:02.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.697+0000 7fe279e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe274150450 0x7fe27414cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:02.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.697+0000 7fe279e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe274156af0 0x7fe27414a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:02.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.697+0000 7fe279e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2740c2cb0 con 0x7fe274156af0 2024-02-21T23:13:02.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.697+0000 7fe279e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2740c29d0 con 0x7fe27406ee40 2024-02-21T23:13:02.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.697+0000 7fe279e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe27406e700 con 0x7fe274150450 2024-02-21T23:13:02.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.698+0000 7fe279400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe274156af0 0x7fe27414a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:02.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.698+0000 7fe279400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe274156af0 0x7fe27414a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33186/0 (socket says 172.21.15.28:33186) 2024-02-21T23:13:02.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.698+0000 7fe279400640 1 -- 172.21.15.28:0/557932379 learned_addr learned my addr 172.21.15.28:0/557932379 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:02.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.698+0000 7fe278a00640 1 --2- 172.21.15.28:0/557932379 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe27406ee40 0x7fe27414c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:02.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.698+0000 7fe273e00640 1 --2- 172.21.15.28:0/557932379 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe274150450 0x7fe27414cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:02.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.698+0000 7fe279400640 1 -- 172.21.15.28:0/557932379 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe274150450 msgr2=0x7fe27414cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:02.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.698+0000 7fe279400640 1 --2- 172.21.15.28:0/557932379 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe274150450 0x7fe27414cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:02.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.698+0000 7fe279400640 1 -- 172.21.15.28:0/557932379 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe27406ee40 msgr2=0x7fe27414c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:02.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.698+0000 7fe279400640 1 --2- 172.21.15.28:0/557932379 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe27406ee40 0x7fe27414c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:02.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.698+0000 7fe279400640 1 -- 172.21.15.28:0/557932379 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2741eab80 con 0x7fe274156af0 2024-02-21T23:13:02.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.698+0000 7fe278a00640 1 --2- 172.21.15.28:0/557932379 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe27406ee40 0x7fe27414c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:13:02.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.698+0000 7fe279400640 1 --2- 172.21.15.28:0/557932379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe274156af0 0x7fe27414a6f0 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 crypto rx=0x7fe26c00c780 tx=0x7fe26c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:02.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.699+0000 7fe270c00640 1 -- 172.21.15.28:0/557932379 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe26c004b60 con 0x7fe274156af0 2024-02-21T23:13:02.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.699+0000 7fe270c00640 1 -- 172.21.15.28:0/557932379 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe26c004d00 con 0x7fe274156af0 2024-02-21T23:13:02.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.699+0000 7fe270c00640 1 -- 172.21.15.28:0/557932379 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe26c00aa30 con 0x7fe274156af0 2024-02-21T23:13:02.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.699+0000 7fe279e00640 1 -- 172.21.15.28:0/557932379 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2741512b0 con 0x7fe274156af0 2024-02-21T23:13:02.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.699+0000 7fe279e00640 1 -- 172.21.15.28:0/557932379 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe274151870 con 0x7fe274156af0 2024-02-21T23:13:02.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.700+0000 7fe279e00640 1 -- 172.21.15.28:0/557932379 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe274151d50 con 0x7fe274156af0 2024-02-21T23:13:02.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.701+0000 7fe270c00640 1 -- 172.21.15.28:0/557932379 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe26c00abd0 con 0x7fe274156af0 2024-02-21T23:13:02.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.702+0000 7fe270c00640 1 --2- 172.21.15.28:0/557932379 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe25406b500 0x7fe25406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:02.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.702+0000 7fe270c00640 1 -- 172.21.15.28:0/557932379 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe26c08a5a0 con 0x7fe274156af0 2024-02-21T23:13:02.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.703+0000 7fe278a00640 1 --2- 172.21.15.28:0/557932379 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe25406b500 0x7fe25406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:02.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.704+0000 7fe278a00640 1 --2- 172.21.15.28:0/557932379 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe25406b500 0x7fe25406d9c0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fe268006fd0 tx=0x7fe268008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:02.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.708+0000 7fe270c00640 1 -- 172.21.15.28:0/557932379 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe274151d50 con 0x7fe274156af0 2024-02-21T23:13:02.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.949+0000 7fe279e00640 1 -- 172.21.15.28:0/557932379 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe27404f290 con 0x7fe274156af0 2024-02-21T23:13:02.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.950+0000 7fe270c00640 1 -- 172.21.15.28:0/557932379 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fe26c020070 con 0x7fe274156af0 2024-02-21T23:13:02.938 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:13:02.938 INFO:teuthology.orchestra.run.smithi028.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-02-21T23:13:02.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.952+0000 7fe279e00640 1 -- 172.21.15.28:0/557932379 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe25406b500 msgr2=0x7fe25406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:02.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.952+0000 7fe279e00640 1 --2- 172.21.15.28:0/557932379 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe25406b500 0x7fe25406d9c0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fe268006fd0 tx=0x7fe268008040 comp rx=0 tx=0).stop 2024-02-21T23:13:02.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.952+0000 7fe279e00640 1 -- 172.21.15.28:0/557932379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe274156af0 msgr2=0x7fe27414a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:02.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.952+0000 7fe279e00640 1 --2- 172.21.15.28:0/557932379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe274156af0 0x7fe27414a6f0 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 crypto rx=0x7fe26c00c780 tx=0x7fe26c00cc40 comp rx=0 tx=0).stop 2024-02-21T23:13:02.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.953+0000 7fe279e00640 1 -- 172.21.15.28:0/557932379 shutdown_connections 2024-02-21T23:13:02.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.953+0000 7fe279e00640 1 --2- 172.21.15.28:0/557932379 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe25406b500 0x7fe25406d9c0 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:02.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.953+0000 7fe279e00640 1 --2- 172.21.15.28:0/557932379 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe274150450 0x7fe27414cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:02.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.953+0000 7fe279e00640 1 --2- 172.21.15.28:0/557932379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe274156af0 0x7fe27414a6f0 unknown :-1 s=CLOSED pgs=2280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:02.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.953+0000 7fe279e00640 1 --2- 172.21.15.28:0/557932379 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe27406ee40 0x7fe27414c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:02.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.953+0000 7fe279e00640 1 -- 172.21.15.28:0/557932379 >> 172.21.15.28:0/557932379 conn(0x7fe27406b630 msgr2=0x7fe2740c64e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:02.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.953+0000 7fe279e00640 1 -- 172.21.15.28:0/557932379 shutdown_connections 2024-02-21T23:13:02.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:02.953+0000 7fe279e00640 1 -- 172.21.15.28:0/557932379 wait complete. 2024-02-21T23:13:02.961 DEBUG:tasks.ceph_test_case:Not expected number of summary strings ([]) 2024-02-21T23:13:02.961 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-02-21T23:13:07.962 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-02-21T23:13:08.075 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-21T23:13:08.082 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-21T23:13:08.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.203+0000 7fe989c00640 1 Processor -- start 2024-02-21T23:13:08.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.203+0000 7fe989c00640 1 -- start start 2024-02-21T23:13:08.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.205+0000 7fe989c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe98413e620 0x7fe984159070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:08.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.205+0000 7fe989c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe9840bf990 0x7fe9840bfe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:08.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.205+0000 7fe989c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9841551b0 con 0x7fe9840c6460 2024-02-21T23:13:08.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.205+0000 7fe989c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9840c71d0 con 0x7fe9840bf990 2024-02-21T23:13:08.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.205+0000 7fe989c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9841595b0 con 0x7fe98413e620 2024-02-21T23:13:08.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.206+0000 7fe97be00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe9840c6460 0x7fe9840ca8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60062/0 (socket says 172.21.15.28:60062) 2024-02-21T23:13:08.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.206+0000 7fe97be00640 1 -- 172.21.15.28:0/1610852161 learned_addr learned my addr 172.21.15.28:0/1610852161 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:13:08.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.206+0000 7fe983400640 1 --2- 172.21.15.28:0/1610852161 >> v2:172.21.15.28:3300/0 conn(0x7fe9840bf990 0x7fe9840bfe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:08.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.206+0000 7fe982a00640 1 --2- 172.21.15.28:0/1610852161 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe98413e620 0x7fe984159070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:08.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.206+0000 7fe982000640 1 -- 172.21.15.28:0/1610852161 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3812249037 0 0) 0x7fe9841551b0 con 0x7fe9840c6460 2024-02-21T23:13:08.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.206+0000 7fe982000640 1 -- 172.21.15.28:0/1610852161 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe968003520 con 0x7fe9840c6460 2024-02-21T23:13:08.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.206+0000 7fe983400640 1 -- 172.21.15.28:0/1610852161 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe98413e620 msgr2=0x7fe984159070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:08.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.206+0000 7fe983400640 1 --2- 172.21.15.28:0/1610852161 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe98413e620 0x7fe984159070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:08.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.206+0000 7fe983400640 1 -- 172.21.15.28:0/1610852161 >> v1:172.21.15.28:6789/0 conn(0x7fe9840c6460 legacy=0x7fe9840ca8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:08.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.206+0000 7fe983400640 1 -- 172.21.15.28:0/1610852161 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe984155420 con 0x7fe9840bf990 2024-02-21T23:13:08.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.208+0000 7fe983400640 1 --2- 172.21.15.28:0/1610852161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9840bf990 0x7fe9840bfe00 secure :-1 s=READY pgs=2282 cs=0 l=1 rev1=1 crypto rx=0x7fe97c009d90 tx=0x7fe97c02be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb67f94ee0a44b42 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:08.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.208+0000 7fe982000640 1 -- 172.21.15.28:0/1610852161 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe97c039070 con 0x7fe9840bf990 2024-02-21T23:13:08.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.208+0000 7fe982000640 1 -- 172.21.15.28:0/1610852161 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe97c02cbc0 con 0x7fe9840bf990 2024-02-21T23:13:08.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.208+0000 7fe982000640 1 -- 172.21.15.28:0/1610852161 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe97c034440 con 0x7fe9840bf990 2024-02-21T23:13:08.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.208+0000 7fe989c00640 1 -- 172.21.15.28:0/1610852161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9840bf990 msgr2=0x7fe9840bfe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:08.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.208+0000 7fe989c00640 1 --2- 172.21.15.28:0/1610852161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9840bf990 0x7fe9840bfe00 secure :-1 s=READY pgs=2282 cs=0 l=1 rev1=1 crypto rx=0x7fe97c009d90 tx=0x7fe97c02be10 comp rx=0 tx=0).stop 2024-02-21T23:13:08.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.209+0000 7fe989c00640 1 -- 172.21.15.28:0/1610852161 shutdown_connections 2024-02-21T23:13:08.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.209+0000 7fe989c00640 1 --2- 172.21.15.28:0/1610852161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9840bf990 0x7fe9840bfe00 unknown :-1 s=CLOSED pgs=2282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:08.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.209+0000 7fe989c00640 1 --2- 172.21.15.28:0/1610852161 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe98413e620 0x7fe984159070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:08.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.209+0000 7fe989c00640 1 -- 172.21.15.28:0/1610852161 >> 172.21.15.28:0/1610852161 conn(0x7fe98408b1c0 msgr2=0x7fe9840725a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:08.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.209+0000 7fe989c00640 1 -- 172.21.15.28:0/1610852161 shutdown_connections 2024-02-21T23:13:08.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.209+0000 7fe989c00640 1 -- 172.21.15.28:0/1610852161 wait complete. 2024-02-21T23:13:08.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.210+0000 7fe989c00640 1 Processor -- start 2024-02-21T23:13:08.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.210+0000 7fe989c00640 1 -- start start 2024-02-21T23:13:08.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.211+0000 7fe989c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9840bf990 0x7fe9841de4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:08.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.211+0000 7fe989c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe9840c6460 0x7fe98413aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:08.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.211+0000 7fe989c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe98413e620 0x7fe9841ea8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:08.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.211+0000 7fe989c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe984155cc0 con 0x7fe98413e620 2024-02-21T23:13:08.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.211+0000 7fe989c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9841559e0 con 0x7fe9840bf990 2024-02-21T23:13:08.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.211+0000 7fe989c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe98413ea00 con 0x7fe9840c6460 2024-02-21T23:13:08.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.212+0000 7fe983400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe98413e620 0x7fe9841ea8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:08.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.212+0000 7fe983400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe98413e620 0x7fe9841ea8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55072/0 (socket says 172.21.15.28:55072) 2024-02-21T23:13:08.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.212+0000 7fe983400640 1 -- 172.21.15.28:0/2927467543 learned_addr learned my addr 172.21.15.28:0/2927467543 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:08.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.212+0000 7fe983400640 1 -- 172.21.15.28:0/2927467543 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe9840c6460 msgr2=0x7fe98413aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:08.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.212+0000 7fe983400640 1 --2- 172.21.15.28:0/2927467543 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe9840c6460 0x7fe98413aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:08.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.212+0000 7fe983400640 1 -- 172.21.15.28:0/2927467543 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9840bf990 msgr2=0x7fe9841de4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:08.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.212+0000 7fe983400640 1 --2- 172.21.15.28:0/2927467543 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9840bf990 0x7fe9841de4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:08.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.212+0000 7fe983400640 1 -- 172.21.15.28:0/2927467543 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9841ee8c0 con 0x7fe98413e620 2024-02-21T23:13:08.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.212+0000 7fe983400640 1 --2- 172.21.15.28:0/2927467543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe98413e620 0x7fe9841ea8a0 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 crypto rx=0x7fe97c009d90 tx=0x7fe97c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:08.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.213+0000 7fe97aa00640 1 -- 172.21.15.28:0/2927467543 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe97c039050 con 0x7fe98413e620 2024-02-21T23:13:08.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.213+0000 7fe97aa00640 1 -- 172.21.15.28:0/2927467543 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe97c045070 con 0x7fe98413e620 2024-02-21T23:13:08.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.213+0000 7fe97aa00640 1 -- 172.21.15.28:0/2927467543 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe97c009450 con 0x7fe98413e620 2024-02-21T23:13:08.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.213+0000 7fe989c00640 1 -- 172.21.15.28:0/2927467543 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe98413af90 con 0x7fe98413e620 2024-02-21T23:13:08.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.213+0000 7fe989c00640 1 -- 172.21.15.28:0/2927467543 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9841eeb30 con 0x7fe98413e620 2024-02-21T23:13:08.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.214+0000 7fe989c00640 1 -- 172.21.15.28:0/2927467543 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe984067f30 con 0x7fe98413e620 2024-02-21T23:13:08.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.215+0000 7fe97aa00640 1 -- 172.21.15.28:0/2927467543 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe97c03e830 con 0x7fe98413e620 2024-02-21T23:13:08.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.216+0000 7fe97aa00640 1 --2- 172.21.15.28:0/2927467543 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe96006b4d0 0x7fe96006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:08.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.216+0000 7fe97aa00640 1 -- 172.21.15.28:0/2927467543 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe97c0ad1f0 con 0x7fe98413e620 2024-02-21T23:13:08.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.217+0000 7fe982a00640 1 --2- 172.21.15.28:0/2927467543 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe96006b4d0 0x7fe96006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:08.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.218+0000 7fe982a00640 1 --2- 172.21.15.28:0/2927467543 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe96006b4d0 0x7fe96006d990 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fe974006a90 tx=0x7fe974006a20 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:08.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.221+0000 7fe97aa00640 1 -- 172.21.15.28:0/2927467543 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe97c032e20 con 0x7fe98413e620 2024-02-21T23:13:08.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.463+0000 7fe989c00640 1 -- 172.21.15.28:0/2927467543 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe9840c7e00 con 0x7fe98413e620 2024-02-21T23:13:08.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.464+0000 7fe97aa00640 1 -- 172.21.15.28:0/2927467543 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fe97c07c0b0 con 0x7fe98413e620 2024-02-21T23:13:08.452 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:13:08.452 INFO:teuthology.orchestra.run.smithi028.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-02-21T23:13:08.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.466+0000 7fe989c00640 1 -- 172.21.15.28:0/2927467543 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe96006b4d0 msgr2=0x7fe96006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:08.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.466+0000 7fe989c00640 1 --2- 172.21.15.28:0/2927467543 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe96006b4d0 0x7fe96006d990 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fe974006a90 tx=0x7fe974006a20 comp rx=0 tx=0).stop 2024-02-21T23:13:08.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.466+0000 7fe989c00640 1 -- 172.21.15.28:0/2927467543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe98413e620 msgr2=0x7fe9841ea8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:08.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.466+0000 7fe989c00640 1 --2- 172.21.15.28:0/2927467543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe98413e620 0x7fe9841ea8a0 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 crypto rx=0x7fe97c009d90 tx=0x7fe97c004a40 comp rx=0 tx=0).stop 2024-02-21T23:13:08.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.466+0000 7fe989c00640 1 -- 172.21.15.28:0/2927467543 shutdown_connections 2024-02-21T23:13:08.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.466+0000 7fe989c00640 1 --2- 172.21.15.28:0/2927467543 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe96006b4d0 0x7fe96006d990 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:08.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.466+0000 7fe989c00640 1 --2- 172.21.15.28:0/2927467543 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe9840c6460 0x7fe98413aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:08.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.466+0000 7fe989c00640 1 --2- 172.21.15.28:0/2927467543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe98413e620 0x7fe9841ea8a0 unknown :-1 s=CLOSED pgs=2283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:08.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.466+0000 7fe989c00640 1 --2- 172.21.15.28:0/2927467543 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe9840bf990 0x7fe9841de4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:08.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.467+0000 7fe989c00640 1 -- 172.21.15.28:0/2927467543 >> 172.21.15.28:0/2927467543 conn(0x7fe98408b1c0 msgr2=0x7fe9840c7350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:08.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.467+0000 7fe989c00640 1 -- 172.21.15.28:0/2927467543 shutdown_connections 2024-02-21T23:13:08.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:08.467+0000 7fe989c00640 1 -- 172.21.15.28:0/2927467543 wait complete. 2024-02-21T23:13:08.478 DEBUG:tasks.ceph_test_case:Not expected number of summary strings ([]) 2024-02-21T23:13:08.478 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-02-21T23:13:08.478 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-21T23:13:08.641 DEBUG:teuthology.orchestra.run.smithi081:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-21T23:13:08.760 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-02-21T23:13:08.760 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-21T23:13:08.760 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-21T23:13:08.822 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054608,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-02-21T23:13:08.822 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-02-21T23:13:08.822 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-21T23:13:08.881 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053833,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498812k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249404k,nr_inodes=812351,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498812k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-02-21T23:13:08.881 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:13:09.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.127+0000 7f819fe00640 1 Processor -- start 2024-02-21T23:13:09.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.127+0000 7f819fe00640 1 -- start start 2024-02-21T23:13:09.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.129+0000 7f819fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f81a006ee40 0x7f81a00a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:09.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.129+0000 7f819fe00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f81a0156af0 0x7f81a006dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:09.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.129+0000 7f819fe00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81a00c2760 con 0x7f81a0150450 2024-02-21T23:13:09.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.129+0000 7f819fe00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81a00c22c0 con 0x7f81a006ee40 2024-02-21T23:13:09.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.129+0000 7f819fe00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81a0156660 con 0x7f81a0156af0 2024-02-21T23:13:09.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.129+0000 7f8197e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f81a006ee40 0x7f81a00a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:09.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.129+0000 7f8197e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f81a006ee40 0x7f81a00a5fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55082/0 (socket says 172.21.15.28:55082) 2024-02-21T23:13:09.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.129+0000 7f8197e00640 1 -- 172.21.15.28:0/2796852405 learned_addr learned my addr 172.21.15.28:0/2796852405 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:09.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.130+0000 7f819f400640 1 --2- 172.21.15.28:0/2796852405 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f81a0156af0 0x7f81a006dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:09.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.130+0000 7f8197e00640 1 -- 172.21.15.28:0/2796852405 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f81a0156af0 msgr2=0x7f81a006dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:09.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.130+0000 7f8197e00640 1 --2- 172.21.15.28:0/2796852405 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f81a0156af0 0x7f81a006dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:09.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.130+0000 7f8197e00640 1 -- 172.21.15.28:0/2796852405 >> v1:172.21.15.81:6789/0 conn(0x7f81a0150450 legacy=0x7f81a01536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:09.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.130+0000 7f8197e00640 1 -- 172.21.15.28:0/2796852405 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81a0156310 con 0x7f81a006ee40 2024-02-21T23:13:09.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.131+0000 7f8197e00640 1 --2- 172.21.15.28:0/2796852405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81a006ee40 0x7f81a00a5fd0 secure :-1 s=READY pgs=2284 cs=0 l=1 rev1=1 crypto rx=0x7f8188009870 tx=0x7f818802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87bffb7e7b32c4db server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:09.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.131+0000 7f819e000640 1 -- 172.21.15.28:0/2796852405 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8188035020 con 0x7f81a006ee40 2024-02-21T23:13:09.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.131+0000 7f819e000640 1 -- 172.21.15.28:0/2796852405 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8188031640 con 0x7f81a006ee40 2024-02-21T23:13:09.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.131+0000 7f819e000640 1 -- 172.21.15.28:0/2796852405 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f818802d040 con 0x7f81a006ee40 2024-02-21T23:13:09.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.132+0000 7f819fe00640 1 -- 172.21.15.28:0/2796852405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81a006ee40 msgr2=0x7f81a00a5fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:09.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.132+0000 7f819fe00640 1 --2- 172.21.15.28:0/2796852405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81a006ee40 0x7f81a00a5fd0 secure :-1 s=READY pgs=2284 cs=0 l=1 rev1=1 crypto rx=0x7f8188009870 tx=0x7f818802b8f0 comp rx=0 tx=0).stop 2024-02-21T23:13:09.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.132+0000 7f819fe00640 1 -- 172.21.15.28:0/2796852405 shutdown_connections 2024-02-21T23:13:09.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.132+0000 7f819fe00640 1 --2- 172.21.15.28:0/2796852405 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f81a0156af0 0x7f81a006dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:09.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.132+0000 7f819fe00640 1 --2- 172.21.15.28:0/2796852405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81a006ee40 0x7f81a00a5fd0 unknown :-1 s=CLOSED pgs=2284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:09.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.132+0000 7f819fe00640 1 -- 172.21.15.28:0/2796852405 >> 172.21.15.28:0/2796852405 conn(0x7f81a006b630 msgr2=0x7f81a00c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:09.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.132+0000 7f819fe00640 1 -- 172.21.15.28:0/2796852405 shutdown_connections 2024-02-21T23:13:09.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.132+0000 7f819fe00640 1 -- 172.21.15.28:0/2796852405 wait complete. 2024-02-21T23:13:09.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.134+0000 7f819fe00640 1 Processor -- start 2024-02-21T23:13:09.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.134+0000 7f819fe00640 1 -- start start 2024-02-21T23:13:09.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.135+0000 7f819fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81a006ee40 0x7f81a0071070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:09.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.135+0000 7f819fe00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f81a0150450 0x7f81a00bfba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:09.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.135+0000 7f819fe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81a0156af0 0x7f81a00c5c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:09.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.135+0000 7f819fe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81a0156f80 con 0x7f81a006ee40 2024-02-21T23:13:09.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.135+0000 7f819fe00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81a00c2cb0 con 0x7f81a0156af0 2024-02-21T23:13:09.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.135+0000 7f819fe00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81a0157100 con 0x7f81a0150450 2024-02-21T23:13:09.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.135+0000 7f819ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81a006ee40 0x7f81a0071070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:09.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.135+0000 7f819ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81a006ee40 0x7f81a0071070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55088/0 (socket says 172.21.15.28:55088) 2024-02-21T23:13:09.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.135+0000 7f819ea00640 1 -- 172.21.15.28:0/3413548259 learned_addr learned my addr 172.21.15.28:0/3413548259 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:09.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.136+0000 7f8197e00640 1 --2- 172.21.15.28:0/3413548259 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f81a0150450 0x7f81a00bfba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:09.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.136+0000 7f819f400640 1 --2- 172.21.15.28:0/3413548259 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81a0156af0 0x7f81a00c5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:09.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.136+0000 7f819ea00640 1 -- 172.21.15.28:0/3413548259 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f81a0150450 msgr2=0x7f81a00bfba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:09.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.136+0000 7f819ea00640 1 --2- 172.21.15.28:0/3413548259 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f81a0150450 0x7f81a00bfba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:09.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.136+0000 7f819ea00640 1 -- 172.21.15.28:0/3413548259 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81a0156af0 msgr2=0x7f81a00c5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:09.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.136+0000 7f819ea00640 1 --2- 172.21.15.28:0/3413548259 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81a0156af0 0x7f81a00c5c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:09.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.136+0000 7f819ea00640 1 -- 172.21.15.28:0/3413548259 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81a0155ac0 con 0x7f81a006ee40 2024-02-21T23:13:09.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.136+0000 7f819ea00640 1 --2- 172.21.15.28:0/3413548259 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81a006ee40 0x7f81a0071070 secure :-1 s=READY pgs=2285 cs=0 l=1 rev1=1 crypto rx=0x7f819000c460 tx=0x7f819000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:09.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.136+0000 7f8196a00640 1 -- 172.21.15.28:0/3413548259 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8190016020 con 0x7f81a006ee40 2024-02-21T23:13:09.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.136+0000 7f8196a00640 1 -- 172.21.15.28:0/3413548259 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8190012870 con 0x7f81a006ee40 2024-02-21T23:13:09.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.136+0000 7f819fe00640 1 -- 172.21.15.28:0/3413548259 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81a0154110 con 0x7f81a006ee40 2024-02-21T23:13:09.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.137+0000 7f8196a00640 1 -- 172.21.15.28:0/3413548259 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8190012c60 con 0x7f81a006ee40 2024-02-21T23:13:09.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.137+0000 7f819fe00640 1 -- 172.21.15.28:0/3413548259 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81a00c00e0 con 0x7f81a006ee40 2024-02-21T23:13:09.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.138+0000 7f819fe00640 1 -- 172.21.15.28:0/3413548259 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81a004f290 con 0x7f81a006ee40 2024-02-21T23:13:09.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.139+0000 7f8196a00640 1 -- 172.21.15.28:0/3413548259 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8190019020 con 0x7f81a006ee40 2024-02-21T23:13:09.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.140+0000 7f8196a00640 1 --2- 172.21.15.28:0/3413548259 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f817c06b4d0 0x7f817c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:09.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.140+0000 7f8196a00640 1 -- 172.21.15.28:0/3413548259 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f819008a3b0 con 0x7f81a006ee40 2024-02-21T23:13:09.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.140+0000 7f8197e00640 1 --2- 172.21.15.28:0/3413548259 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f817c06b4d0 0x7f817c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:09.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.141+0000 7f8197e00640 1 --2- 172.21.15.28:0/3413548259 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f817c06b4d0 0x7f817c06d990 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f8188036000 tx=0x7f818803c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:09.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.144+0000 7f8196a00640 1 -- 172.21.15.28:0/3413548259 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f8190020020 con 0x7f81a006ee40 2024-02-21T23:13:09.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.389+0000 7f819fe00640 1 -- 172.21.15.28:0/3413548259 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f81a00c1f10 con 0x7f81a006ee40 2024-02-21T23:13:09.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.390+0000 7f8196a00640 1 -- 172.21.15.28:0/3413548259 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 221 v221) v1 ==== 95+0+4939 (secure 0 0 0) 0x7f81900182c0 con 0x7f81a006ee40 2024-02-21T23:13:09.379 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:13:09.379 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":221,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7100,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3213818811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3213818811},{"type":"v1","addr":"172.21.15.28:6835","nonce":3213818811}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":213},{"gid":7106,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/1322911156","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":1322911156},{"type":"v1","addr":"172.21.15.81:6837","nonce":1322911156}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":213},{"gid":7184,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/1214841323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1214841323},{"type":"v1","addr":"172.21.15.81:6836","nonce":1214841323}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":221},{"gid":10386,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3814985204","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3814985204},{"type":"v1","addr":"172.21.15.28:6834","nonce":3814985204}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":212}],"filesystems":[{"mdsmap":{"epoch":220,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:12:26.259457+0000","modified":"2024-02-21T23:12:46.343251+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[],"up":{},"failed":[],"damaged":[],"stopped":[0],"info":{},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15}]} 2024-02-21T23:13:09.379 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 221 2024-02-21T23:13:09.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.393+0000 7f819fe00640 1 -- 172.21.15.28:0/3413548259 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f817c06b4d0 msgr2=0x7f817c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:09.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.393+0000 7f819fe00640 1 --2- 172.21.15.28:0/3413548259 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f817c06b4d0 0x7f817c06d990 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f8188036000 tx=0x7f818803c040 comp rx=0 tx=0).stop 2024-02-21T23:13:09.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.393+0000 7f819fe00640 1 -- 172.21.15.28:0/3413548259 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81a006ee40 msgr2=0x7f81a0071070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:09.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.393+0000 7f819fe00640 1 --2- 172.21.15.28:0/3413548259 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81a006ee40 0x7f81a0071070 secure :-1 s=READY pgs=2285 cs=0 l=1 rev1=1 crypto rx=0x7f819000c460 tx=0x7f819000c920 comp rx=0 tx=0).stop 2024-02-21T23:13:09.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.393+0000 7f819fe00640 1 -- 172.21.15.28:0/3413548259 shutdown_connections 2024-02-21T23:13:09.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.393+0000 7f819fe00640 1 --2- 172.21.15.28:0/3413548259 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f817c06b4d0 0x7f817c06d990 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:09.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.393+0000 7f819fe00640 1 --2- 172.21.15.28:0/3413548259 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f81a0150450 0x7f81a00bfba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:09.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.393+0000 7f819fe00640 1 --2- 172.21.15.28:0/3413548259 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81a0156af0 0x7f81a00c5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:09.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.393+0000 7f819fe00640 1 --2- 172.21.15.28:0/3413548259 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81a006ee40 0x7f81a0071070 unknown :-1 s=CLOSED pgs=2285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:09.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.393+0000 7f819fe00640 1 -- 172.21.15.28:0/3413548259 >> 172.21.15.28:0/3413548259 conn(0x7f81a006b630 msgr2=0x7f81a00c64e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:09.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.393+0000 7f819fe00640 1 -- 172.21.15.28:0/3413548259 shutdown_connections 2024-02-21T23:13:09.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.393+0000 7f819fe00640 1 -- 172.21.15.28:0/3413548259 wait complete. 2024-02-21T23:13:09.402 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:13:09.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.639+0000 7f4766200640 1 Processor -- start 2024-02-21T23:13:09.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.639+0000 7f4766200640 1 -- start start 2024-02-21T23:13:09.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.639+0000 7f4766200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f476006ee40 0x7f47600a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:09.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.639+0000 7f4766200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f4760156af0 0x7f476006dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:09.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.639+0000 7f4766200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47600c2760 con 0x7f4760150450 2024-02-21T23:13:09.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.639+0000 7f4766200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47600c22c0 con 0x7f476006ee40 2024-02-21T23:13:09.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.639+0000 7f4766200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4760156660 con 0x7f4760156af0 2024-02-21T23:13:09.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.640+0000 7f475fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f476006ee40 0x7f47600a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:09.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.640+0000 7f475fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f476006ee40 0x7f47600a5fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55102/0 (socket says 172.21.15.28:55102) 2024-02-21T23:13:09.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.640+0000 7f475fe00640 1 -- 172.21.15.28:0/1930591760 learned_addr learned my addr 172.21.15.28:0/1930591760 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:09.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.640+0000 7f4764e00640 1 --1- 172.21.15.28:0/1930591760 >> v1:172.21.15.28:6789/0 conn(0x7f4760150450 0x7f47601536f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60064/0 (socket says 172.21.15.28:60064) 2024-02-21T23:13:09.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.640+0000 7f4765800640 1 --2- 172.21.15.28:0/1930591760 >> v2:172.21.15.81:3300/0 conn(0x7f4760156af0 0x7f476006dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:09.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.640+0000 7f475fe00640 1 -- 172.21.15.28:0/1930591760 >> v2:172.21.15.81:3300/0 conn(0x7f4760156af0 msgr2=0x7f476006dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:09.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.640+0000 7f475fe00640 1 --2- 172.21.15.28:0/1930591760 >> v2:172.21.15.81:3300/0 conn(0x7f4760156af0 0x7f476006dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:09.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.640+0000 7f475fe00640 1 -- 172.21.15.28:0/1930591760 >> v1:172.21.15.28:6789/0 conn(0x7f4760150450 legacy=0x7f47601536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:09.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.640+0000 7f475fe00640 1 -- 172.21.15.28:0/1930591760 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4760156310 con 0x7f476006ee40 2024-02-21T23:13:09.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.640+0000 7f4765800640 1 --2- 172.21.15.28:0/1930591760 >> v2:172.21.15.81:3300/0 conn(0x7f4760156af0 0x7f476006dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:13:09.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.641+0000 7f475fe00640 1 --2- 172.21.15.28:0/1930591760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f476006ee40 0x7f47600a5fd0 secure :-1 s=READY pgs=2287 cs=0 l=1 rev1=1 crypto rx=0x7f47600734b0 tx=0x7f474c02b9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21c4ce2e7865c93b server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:09.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.642+0000 7f475f400640 1 -- 172.21.15.28:0/1930591760 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f474c034070 con 0x7f476006ee40 2024-02-21T23:13:09.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.642+0000 7f475f400640 1 -- 172.21.15.28:0/1930591760 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f474c030880 con 0x7f476006ee40 2024-02-21T23:13:09.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.642+0000 7f475f400640 1 -- 172.21.15.28:0/1930591760 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f474c030c50 con 0x7f476006ee40 2024-02-21T23:13:09.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.642+0000 7f4766200640 1 -- 172.21.15.28:0/1930591760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f476006ee40 msgr2=0x7f47600a5fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:09.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.642+0000 7f4766200640 1 --2- 172.21.15.28:0/1930591760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f476006ee40 0x7f47600a5fd0 secure :-1 s=READY pgs=2287 cs=0 l=1 rev1=1 crypto rx=0x7f47600734b0 tx=0x7f474c02b9c0 comp rx=0 tx=0).stop 2024-02-21T23:13:09.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.642+0000 7f4766200640 1 -- 172.21.15.28:0/1930591760 shutdown_connections 2024-02-21T23:13:09.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.642+0000 7f4766200640 1 --2- 172.21.15.28:0/1930591760 >> v2:172.21.15.81:3300/0 conn(0x7f4760156af0 0x7f476006dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:09.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.642+0000 7f4766200640 1 --2- 172.21.15.28:0/1930591760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f476006ee40 0x7f47600a5fd0 unknown :-1 s=CLOSED pgs=2287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:09.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.643+0000 7f4766200640 1 -- 172.21.15.28:0/1930591760 >> 172.21.15.28:0/1930591760 conn(0x7f476006b630 msgr2=0x7f47600c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:09.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.643+0000 7f4766200640 1 -- 172.21.15.28:0/1930591760 shutdown_connections 2024-02-21T23:13:09.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.643+0000 7f4766200640 1 -- 172.21.15.28:0/1930591760 wait complete. 2024-02-21T23:13:09.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.644+0000 7f4766200640 1 Processor -- start 2024-02-21T23:13:09.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.644+0000 7f4766200640 1 -- start start 2024-02-21T23:13:09.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.645+0000 7f4766200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f476006ee40 0x7f4760155720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:09.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.645+0000 7f4766200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4760150450 0x7f47601eaa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:09.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.645+0000 7f4766200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4760156af0 0x7f47601eaf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:09.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.645+0000 7f4766200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4760156f80 con 0x7f4760156af0 2024-02-21T23:13:09.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.645+0000 7f4766200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47600c2cb0 con 0x7f4760150450 2024-02-21T23:13:09.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.645+0000 7f4766200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4760157100 con 0x7f476006ee40 2024-02-21T23:13:09.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.646+0000 7f4765800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4760156af0 0x7f47601eaf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:09.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.646+0000 7f4765800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4760156af0 0x7f47601eaf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55110/0 (socket says 172.21.15.28:55110) 2024-02-21T23:13:09.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.646+0000 7f4765800640 1 -- 172.21.15.28:0/3286117540 learned_addr learned my addr 172.21.15.28:0/3286117540 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:09.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.646+0000 7f4764e00640 1 --2- 172.21.15.28:0/3286117540 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f476006ee40 0x7f4760155720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:09.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.646+0000 7f475fe00640 1 --2- 172.21.15.28:0/3286117540 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4760150450 0x7f47601eaa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:09.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.646+0000 7f4765800640 1 -- 172.21.15.28:0/3286117540 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f476006ee40 msgr2=0x7f4760155720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:09.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.646+0000 7f4765800640 1 --2- 172.21.15.28:0/3286117540 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f476006ee40 0x7f4760155720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:09.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.646+0000 7f4765800640 1 -- 172.21.15.28:0/3286117540 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4760150450 msgr2=0x7f47601eaa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:09.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.646+0000 7f4765800640 1 --2- 172.21.15.28:0/3286117540 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4760150450 0x7f47601eaa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:09.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.646+0000 7f4765800640 1 -- 172.21.15.28:0/3286117540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47600aa2c0 con 0x7f4760156af0 2024-02-21T23:13:09.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.647+0000 7f4765800640 1 --2- 172.21.15.28:0/3286117540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4760156af0 0x7f47601eaf40 secure :-1 s=READY pgs=2288 cs=0 l=1 rev1=1 crypto rx=0x7f475800cd80 tx=0x7f475800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:09.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.647+0000 7f475cc00640 1 -- 172.21.15.28:0/3286117540 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f475800f680 con 0x7f4760156af0 2024-02-21T23:13:09.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.647+0000 7f475cc00640 1 -- 172.21.15.28:0/3286117540 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f475800f840 con 0x7f4760156af0 2024-02-21T23:13:09.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.647+0000 7f475cc00640 1 -- 172.21.15.28:0/3286117540 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f475800fc30 con 0x7f4760156af0 2024-02-21T23:13:09.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.647+0000 7f4766200640 1 -- 172.21.15.28:0/3286117540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f476014f810 con 0x7f4760156af0 2024-02-21T23:13:09.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.647+0000 7f4766200640 1 -- 172.21.15.28:0/3286117540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47601ee550 con 0x7f4760156af0 2024-02-21T23:13:09.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.648+0000 7f4766200640 1 -- 172.21.15.28:0/3286117540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f476004f290 con 0x7f4760156af0 2024-02-21T23:13:09.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.649+0000 7f475cc00640 1 -- 172.21.15.28:0/3286117540 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f475800e3e0 con 0x7f4760156af0 2024-02-21T23:13:09.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.650+0000 7f475cc00640 1 --2- 172.21.15.28:0/3286117540 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f474006b4d0 0x7f474006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:09.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.650+0000 7f475cc00640 1 -- 172.21.15.28:0/3286117540 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f475808b170 con 0x7f4760156af0 2024-02-21T23:13:09.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.651+0000 7f4764e00640 1 --2- 172.21.15.28:0/3286117540 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f474006b4d0 0x7f474006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:09.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.652+0000 7f4764e00640 1 --2- 172.21.15.28:0/3286117540 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f474006b4d0 0x7f474006d990 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f4760155f70 tx=0x7f4754007ce0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:09.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.655+0000 7f475cc00640 1 -- 172.21.15.28:0/3286117540 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4758014030 con 0x7f4760156af0 2024-02-21T23:13:09.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.857+0000 7f4766200640 1 -- 172.21.15.28:0/3286117540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f47600c38c0 con 0x7f4760156af0 2024-02-21T23:13:09.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.860+0000 7f475cc00640 1 -- 172.21.15.28:0/3286117540 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v197) v1 ==== 74+0+16892 (secure 0 0 0) 0x7f4758059e50 con 0x7f4760156af0 2024-02-21T23:13:09.849 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:13:09.850 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":197,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:12:27.821567+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:12:24.201589+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"196","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2024-02-21T23:12:25.295387+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"197","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:13:09.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.863+0000 7f4766200640 1 -- 172.21.15.28:0/3286117540 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f474006b4d0 msgr2=0x7f474006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:09.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.863+0000 7f4766200640 1 --2- 172.21.15.28:0/3286117540 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f474006b4d0 0x7f474006d990 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f4760155f70 tx=0x7f4754007ce0 comp rx=0 tx=0).stop 2024-02-21T23:13:09.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.863+0000 7f4766200640 1 -- 172.21.15.28:0/3286117540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4760156af0 msgr2=0x7f47601eaf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:09.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.863+0000 7f4766200640 1 --2- 172.21.15.28:0/3286117540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4760156af0 0x7f47601eaf40 secure :-1 s=READY pgs=2288 cs=0 l=1 rev1=1 crypto rx=0x7f475800cd80 tx=0x7f475800a650 comp rx=0 tx=0).stop 2024-02-21T23:13:09.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.864+0000 7f4766200640 1 -- 172.21.15.28:0/3286117540 shutdown_connections 2024-02-21T23:13:09.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.864+0000 7f4766200640 1 --2- 172.21.15.28:0/3286117540 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f474006b4d0 0x7f474006d990 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:09.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.864+0000 7f4766200640 1 --2- 172.21.15.28:0/3286117540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4760156af0 0x7f47601eaf40 unknown :-1 s=CLOSED pgs=2288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:09.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.864+0000 7f4766200640 1 --2- 172.21.15.28:0/3286117540 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4760150450 0x7f47601eaa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:09.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.864+0000 7f4766200640 1 --2- 172.21.15.28:0/3286117540 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f476006ee40 0x7f4760155720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:09.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.864+0000 7f4766200640 1 -- 172.21.15.28:0/3286117540 >> 172.21.15.28:0/3286117540 conn(0x7f476006b630 msgr2=0x7f4760149610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:09.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.864+0000 7f4766200640 1 -- 172.21.15.28:0/3286117540 shutdown_connections 2024-02-21T23:13:09.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:09.864+0000 7f4766200640 1 -- 172.21.15.28:0/3286117540 wait complete. 2024-02-21T23:13:09.873 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-21T23:02:30.728329+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 31, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-21T23:12:24.201589+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '196', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 32, 'pool_name': 'cephfs_data', 'create_time': '2024-02-21T23:12:25.295387+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '197', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-21T23:13:09.873 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-21T23:13:10.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.113+0000 7f8c51c00640 1 Processor -- start 2024-02-21T23:13:10.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.113+0000 7f8c51c00640 1 -- start start 2024-02-21T23:13:10.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.114+0000 7f8c51c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8c4c0c5df0 0x7f8c4c1556a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:10.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.114+0000 7f8c51c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8c4c155be0 0x7f8c4c15c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:10.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.115+0000 7f8c51c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c4c0c5b80 con 0x7f8c4c14f000 2024-02-21T23:13:10.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.115+0000 7f8c51c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c4c0c4fe0 con 0x7f8c4c155be0 2024-02-21T23:13:10.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.115+0000 7f8c51c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c4c0c5650 con 0x7f8c4c0c5df0 2024-02-21T23:13:10.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.115+0000 7f8c4be00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f8c4c14f000 0x7f8c4c06ec20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60072/0 (socket says 172.21.15.28:60072) 2024-02-21T23:13:10.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.115+0000 7f8c4be00640 1 -- 172.21.15.28:0/603052643 learned_addr learned my addr 172.21.15.28:0/603052643 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:13:10.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.115+0000 7f8c51200640 1 --2- 172.21.15.28:0/603052643 >> v2:172.21.15.28:3300/0 conn(0x7f8c4c155be0 0x7f8c4c15c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:10.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.115+0000 7f8c4b400640 1 --2- 172.21.15.28:0/603052643 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8c4c0c5df0 0x7f8c4c1556a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:10.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.115+0000 7f8c51200640 1 -- 172.21.15.28:0/603052643 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8c4c0c5df0 msgr2=0x7f8c4c1556a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:10.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.115+0000 7f8c51200640 1 --2- 172.21.15.28:0/603052643 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8c4c0c5df0 0x7f8c4c1556a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:10.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.116+0000 7f8c51200640 1 -- 172.21.15.28:0/603052643 >> v1:172.21.15.28:6789/0 conn(0x7f8c4c14f000 legacy=0x7f8c4c06ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:10.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.116+0000 7f8c51200640 1 -- 172.21.15.28:0/603052643 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c4c152950 con 0x7f8c4c155be0 2024-02-21T23:13:10.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.116+0000 7f8c4aa00640 1 -- 172.21.15.28:0/603052643 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 148932623 0 0) 0x7f8c4c0c5b80 con 0x7f8c4c14f000 2024-02-21T23:13:10.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.117+0000 7f8c51200640 1 --2- 172.21.15.28:0/603052643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c4c155be0 0x7f8c4c15c470 secure :-1 s=READY pgs=2290 cs=0 l=1 rev1=1 crypto rx=0x7f8c440097a0 tx=0x7f8c4402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b48eab0132d3ab2 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:10.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.117+0000 7f8c4aa00640 1 -- 172.21.15.28:0/603052643 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c44035020 con 0x7f8c4c155be0 2024-02-21T23:13:10.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.117+0000 7f8c4aa00640 1 -- 172.21.15.28:0/603052643 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c44031810 con 0x7f8c4c155be0 2024-02-21T23:13:10.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.117+0000 7f8c4aa00640 1 -- 172.21.15.28:0/603052643 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c44031be0 con 0x7f8c4c155be0 2024-02-21T23:13:10.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.118+0000 7f8c51c00640 1 -- 172.21.15.28:0/603052643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c4c155be0 msgr2=0x7f8c4c15c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:10.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.118+0000 7f8c51c00640 1 --2- 172.21.15.28:0/603052643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c4c155be0 0x7f8c4c15c470 secure :-1 s=READY pgs=2290 cs=0 l=1 rev1=1 crypto rx=0x7f8c440097a0 tx=0x7f8c4402b8b0 comp rx=0 tx=0).stop 2024-02-21T23:13:10.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.118+0000 7f8c51c00640 1 -- 172.21.15.28:0/603052643 shutdown_connections 2024-02-21T23:13:10.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.118+0000 7f8c51c00640 1 --2- 172.21.15.28:0/603052643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c4c155be0 0x7f8c4c15c470 unknown :-1 s=CLOSED pgs=2290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:10.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.118+0000 7f8c51c00640 1 --2- 172.21.15.28:0/603052643 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8c4c0c5df0 0x7f8c4c1556a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:10.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.118+0000 7f8c51c00640 1 -- 172.21.15.28:0/603052643 >> 172.21.15.28:0/603052643 conn(0x7f8c4c0a8530 msgr2=0x7f8c4c0a8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:10.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.118+0000 7f8c51c00640 1 -- 172.21.15.28:0/603052643 shutdown_connections 2024-02-21T23:13:10.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.118+0000 7f8c51c00640 1 -- 172.21.15.28:0/603052643 wait complete. 2024-02-21T23:13:10.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.120+0000 7f8c51c00640 1 Processor -- start 2024-02-21T23:13:10.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.121+0000 7f8c51c00640 1 -- start start 2024-02-21T23:13:10.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.121+0000 7f8c51c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8c4c0c5df0 0x7f8c4c154630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:10.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.121+0000 7f8c51c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c4c14f000 0x7f8c4c149640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:10.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.121+0000 7f8c51c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c4c155be0 0x7f8c4c149b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:10.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.121+0000 7f8c51c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c4c06f330 con 0x7f8c4c14f000 2024-02-21T23:13:10.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.121+0000 7f8c51c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c4c155180 con 0x7f8c4c155be0 2024-02-21T23:13:10.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.121+0000 7f8c51c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c4c06f4b0 con 0x7f8c4c0c5df0 2024-02-21T23:13:10.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.121+0000 7f8c4b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c4c14f000 0x7f8c4c149640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:10.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.121+0000 7f8c4b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c4c14f000 0x7f8c4c149640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55132/0 (socket says 172.21.15.28:55132) 2024-02-21T23:13:10.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.121+0000 7f8c4b400640 1 -- 172.21.15.28:0/125688201 learned_addr learned my addr 172.21.15.28:0/125688201 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:10.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.121+0000 7f8c4be00640 1 --2- 172.21.15.28:0/125688201 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8c4c0c5df0 0x7f8c4c154630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:10.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.121+0000 7f8c51200640 1 --2- 172.21.15.28:0/125688201 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c4c155be0 0x7f8c4c149b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:10.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.121+0000 7f8c4b400640 1 -- 172.21.15.28:0/125688201 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8c4c0c5df0 msgr2=0x7f8c4c154630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:10.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.121+0000 7f8c4b400640 1 --2- 172.21.15.28:0/125688201 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8c4c0c5df0 0x7f8c4c154630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:10.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.121+0000 7f8c4b400640 1 -- 172.21.15.28:0/125688201 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c4c155be0 msgr2=0x7f8c4c149b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:10.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.121+0000 7f8c4b400640 1 --2- 172.21.15.28:0/125688201 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c4c155be0 0x7f8c4c149b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:10.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.121+0000 7f8c4b400640 1 -- 172.21.15.28:0/125688201 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c4c0a9680 con 0x7f8c4c14f000 2024-02-21T23:13:10.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.121+0000 7f8c4be00640 1 --2- 172.21.15.28:0/125688201 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8c4c0c5df0 0x7f8c4c154630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:13:10.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.122+0000 7f8c4b400640 1 --2- 172.21.15.28:0/125688201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c4c14f000 0x7f8c4c149640 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 crypto rx=0x7f8c3400d830 tx=0x7f8c3400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:10.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.122+0000 7f8c43e00640 1 -- 172.21.15.28:0/125688201 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c3400b6c0 con 0x7f8c4c14f000 2024-02-21T23:13:10.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.122+0000 7f8c43e00640 1 -- 172.21.15.28:0/125688201 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c34019070 con 0x7f8c4c14f000 2024-02-21T23:13:10.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.122+0000 7f8c43e00640 1 -- 172.21.15.28:0/125688201 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c3400ba80 con 0x7f8c4c14f000 2024-02-21T23:13:10.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.122+0000 7f8c51c00640 1 -- 172.21.15.28:0/125688201 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c4c14a300 con 0x7f8c4c14f000 2024-02-21T23:13:10.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.122+0000 7f8c51c00640 1 -- 172.21.15.28:0/125688201 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c4c0a9b60 con 0x7f8c4c14f000 2024-02-21T23:13:10.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.123+0000 7f8c51c00640 1 -- 172.21.15.28:0/125688201 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c4c04f290 con 0x7f8c4c14f000 2024-02-21T23:13:10.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.124+0000 7f8c43e00640 1 -- 172.21.15.28:0/125688201 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8c3401e460 con 0x7f8c4c14f000 2024-02-21T23:13:10.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.125+0000 7f8c43e00640 1 --2- 172.21.15.28:0/125688201 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8c2806b4d0 0x7f8c2806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:10.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.125+0000 7f8c43e00640 1 -- 172.21.15.28:0/125688201 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f8c3408aa50 con 0x7f8c4c14f000 2024-02-21T23:13:10.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.126+0000 7f8c4be00640 1 --2- 172.21.15.28:0/125688201 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8c2806b4d0 0x7f8c2806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:10.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.127+0000 7f8c4be00640 1 --2- 172.21.15.28:0/125688201 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8c2806b4d0 0x7f8c2806d990 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c0012e0 tx=0x7f8c3c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:10.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.130+0000 7f8c43e00640 1 -- 172.21.15.28:0/125688201 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f8c34059730 con 0x7f8c4c14f000 2024-02-21T23:13:10.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.328+0000 7f8c51c00640 1 -- 172.21.15.28:0/125688201 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8c4c152580 con 0x7f8c4c14f000 2024-02-21T23:13:10.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.328+0000 7f8c43e00640 1 -- 172.21.15.28:0/125688201 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v197) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8c34059030 con 0x7f8c4c14f000 2024-02-21T23:13:10.316 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-02-21T23:13:10.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.330+0000 7f8c51c00640 1 -- 172.21.15.28:0/125688201 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8c2806b4d0 msgr2=0x7f8c2806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:10.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.330+0000 7f8c51c00640 1 --2- 172.21.15.28:0/125688201 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8c2806b4d0 0x7f8c2806d990 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c0012e0 tx=0x7f8c3c009040 comp rx=0 tx=0).stop 2024-02-21T23:13:10.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.330+0000 7f8c51c00640 1 -- 172.21.15.28:0/125688201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c4c14f000 msgr2=0x7f8c4c149640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:10.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.330+0000 7f8c51c00640 1 --2- 172.21.15.28:0/125688201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c4c14f000 0x7f8c4c149640 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 crypto rx=0x7f8c3400d830 tx=0x7f8c3400dcf0 comp rx=0 tx=0).stop 2024-02-21T23:13:10.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.330+0000 7f8c51c00640 1 -- 172.21.15.28:0/125688201 shutdown_connections 2024-02-21T23:13:10.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.330+0000 7f8c51c00640 1 --2- 172.21.15.28:0/125688201 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8c2806b4d0 0x7f8c2806d990 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:10.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.330+0000 7f8c51c00640 1 --2- 172.21.15.28:0/125688201 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c4c155be0 0x7f8c4c149b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:10.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.330+0000 7f8c51c00640 1 --2- 172.21.15.28:0/125688201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c4c14f000 0x7f8c4c149640 unknown :-1 s=CLOSED pgs=2291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:10.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.330+0000 7f8c51c00640 1 --2- 172.21.15.28:0/125688201 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8c4c0c5df0 0x7f8c4c154630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:10.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.330+0000 7f8c51c00640 1 -- 172.21.15.28:0/125688201 >> 172.21.15.28:0/125688201 conn(0x7f8c4c0a8530 msgr2=0x7f8c4c072f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:10.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.330+0000 7f8c51c00640 1 -- 172.21.15.28:0/125688201 shutdown_connections 2024-02-21T23:13:10.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.330+0000 7f8c51c00640 1 -- 172.21.15.28:0/125688201 wait complete. 2024-02-21T23:13:10.339 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-21T23:13:10.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.578+0000 7f4df4e00640 1 Processor -- start 2024-02-21T23:13:10.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.578+0000 7f4df4e00640 1 -- start start 2024-02-21T23:13:10.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.580+0000 7f4df4e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4df0152650 0x7f4df00c8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:10.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.580+0000 7f4df4e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f4df01550e0 0x7f4df01554f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:10.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.580+0000 7f4df4e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4df0158920 con 0x7f4df006daa0 2024-02-21T23:13:10.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.580+0000 7f4df4e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4df01586d0 con 0x7f4df0152650 2024-02-21T23:13:10.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.580+0000 7f4df4e00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4df0158c00 con 0x7f4df01550e0 2024-02-21T23:13:10.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.580+0000 7f4def400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4df0152650 0x7f4df00c8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:10.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.580+0000 7f4def400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4df0152650 0x7f4df00c8ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55134/0 (socket says 172.21.15.28:55134) 2024-02-21T23:13:10.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.580+0000 7f4def400640 1 -- 172.21.15.28:0/118201119 learned_addr learned my addr 172.21.15.28:0/118201119 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:10.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.580+0000 7f4defe00640 1 --2- 172.21.15.28:0/118201119 >> v2:172.21.15.81:3300/0 conn(0x7f4df01550e0 0x7f4df01554f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:10.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.581+0000 7f4def400640 1 -- 172.21.15.28:0/118201119 >> v2:172.21.15.81:3300/0 conn(0x7f4df01550e0 msgr2=0x7f4df01554f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:10.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.581+0000 7f4def400640 1 --2- 172.21.15.28:0/118201119 >> v2:172.21.15.81:3300/0 conn(0x7f4df01550e0 0x7f4df01554f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:10.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.581+0000 7f4def400640 1 -- 172.21.15.28:0/118201119 >> v1:172.21.15.81:6789/0 conn(0x7f4df006daa0 legacy=0x7f4df00c7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:10.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.581+0000 7f4def400640 1 -- 172.21.15.28:0/118201119 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4df0158ee0 con 0x7f4df0152650 2024-02-21T23:13:10.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.582+0000 7f4def400640 1 --2- 172.21.15.28:0/118201119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df0152650 0x7f4df00c8ce0 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 crypto rx=0x7f4df00aa260 tx=0x7f4de002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e92071e37dac1787 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:10.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.583+0000 7f4deea00640 1 -- 172.21.15.28:0/118201119 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4de0035020 con 0x7f4df0152650 2024-02-21T23:13:10.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.583+0000 7f4deea00640 1 -- 172.21.15.28:0/118201119 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4de0031870 con 0x7f4df0152650 2024-02-21T23:13:10.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.583+0000 7f4deea00640 1 -- 172.21.15.28:0/118201119 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4de0031c40 con 0x7f4df0152650 2024-02-21T23:13:10.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.583+0000 7f4df4e00640 1 -- 172.21.15.28:0/118201119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df0152650 msgr2=0x7f4df00c8ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:10.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.583+0000 7f4df4e00640 1 --2- 172.21.15.28:0/118201119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df0152650 0x7f4df00c8ce0 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 crypto rx=0x7f4df00aa260 tx=0x7f4de002b910 comp rx=0 tx=0).stop 2024-02-21T23:13:10.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.583+0000 7f4df4e00640 1 -- 172.21.15.28:0/118201119 shutdown_connections 2024-02-21T23:13:10.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.583+0000 7f4df4e00640 1 --2- 172.21.15.28:0/118201119 >> v2:172.21.15.81:3300/0 conn(0x7f4df01550e0 0x7f4df01554f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:10.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.583+0000 7f4df4e00640 1 --2- 172.21.15.28:0/118201119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df0152650 0x7f4df00c8ce0 unknown :-1 s=CLOSED pgs=2292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:10.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.583+0000 7f4df4e00640 1 -- 172.21.15.28:0/118201119 >> 172.21.15.28:0/118201119 conn(0x7f4df0072ce0 msgr2=0x7f4df00ac340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:10.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.583+0000 7f4df4e00640 1 -- 172.21.15.28:0/118201119 shutdown_connections 2024-02-21T23:13:10.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.584+0000 7f4df4e00640 1 -- 172.21.15.28:0/118201119 wait complete. 2024-02-21T23:13:10.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.585+0000 7f4df4e00640 1 Processor -- start 2024-02-21T23:13:10.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.585+0000 7f4df4e00640 1 -- start start 2024-02-21T23:13:10.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.586+0000 7f4df4e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4df006daa0 0x7f4df014b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:10.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.586+0000 7f4df4e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df0152650 0x7f4df01e0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:10.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.586+0000 7f4df4e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4df01550e0 0x7f4df014df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:10.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.586+0000 7f4df4e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4df0152a30 con 0x7f4df0152650 2024-02-21T23:13:10.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.586+0000 7f4df4e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4df00bdfb0 con 0x7f4df006daa0 2024-02-21T23:13:10.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.586+0000 7f4df4e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4df00c7870 con 0x7f4df01550e0 2024-02-21T23:13:10.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.587+0000 7f4de7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df0152650 0x7f4df01e0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:10.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.587+0000 7f4de7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df0152650 0x7f4df01e0810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55150/0 (socket says 172.21.15.28:55150) 2024-02-21T23:13:10.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.587+0000 7f4de7e00640 1 -- 172.21.15.28:0/2206182871 learned_addr learned my addr 172.21.15.28:0/2206182871 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:10.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.587+0000 7f4def400640 1 --2- 172.21.15.28:0/2206182871 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4df006daa0 0x7f4df014b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:10.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.587+0000 7f4defe00640 1 --2- 172.21.15.28:0/2206182871 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4df01550e0 0x7f4df014df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:10.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.587+0000 7f4de7e00640 1 -- 172.21.15.28:0/2206182871 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4df01550e0 msgr2=0x7f4df014df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:10.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.587+0000 7f4de7e00640 1 --2- 172.21.15.28:0/2206182871 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4df01550e0 0x7f4df014df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:10.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.587+0000 7f4de7e00640 1 -- 172.21.15.28:0/2206182871 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4df006daa0 msgr2=0x7f4df014b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:10.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.587+0000 7f4de7e00640 1 --2- 172.21.15.28:0/2206182871 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4df006daa0 0x7f4df014b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:10.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.587+0000 7f4de7e00640 1 -- 172.21.15.28:0/2206182871 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4df014eb90 con 0x7f4df0152650 2024-02-21T23:13:10.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.587+0000 7f4defe00640 1 --2- 172.21.15.28:0/2206182871 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4df01550e0 0x7f4df014df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:13:10.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.587+0000 7f4def400640 1 --2- 172.21.15.28:0/2206182871 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4df006daa0 0x7f4df014b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:13:10.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.588+0000 7f4de7e00640 1 --2- 172.21.15.28:0/2206182871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df0152650 0x7f4df01e0810 secure :-1 s=READY pgs=2293 cs=0 l=1 rev1=1 crypto rx=0x7f4dd800c400 tx=0x7f4dd800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:10.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.588+0000 7f4de7400640 1 -- 172.21.15.28:0/2206182871 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dd80052d0 con 0x7f4df0152650 2024-02-21T23:13:10.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.588+0000 7f4de7400640 1 -- 172.21.15.28:0/2206182871 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dd8005490 con 0x7f4df0152650 2024-02-21T23:13:10.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.588+0000 7f4de7400640 1 -- 172.21.15.28:0/2206182871 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dd8007260 con 0x7f4df0152650 2024-02-21T23:13:10.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.588+0000 7f4df4e00640 1 -- 172.21.15.28:0/2206182871 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4df014ee70 con 0x7f4df0152650 2024-02-21T23:13:10.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.588+0000 7f4df4e00640 1 -- 172.21.15.28:0/2206182871 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4df01df1e0 con 0x7f4df0152650 2024-02-21T23:13:10.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.589+0000 7f4df4e00640 1 -- 172.21.15.28:0/2206182871 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4df004f290 con 0x7f4df0152650 2024-02-21T23:13:10.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.590+0000 7f4de7400640 1 -- 172.21.15.28:0/2206182871 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4dd8007400 con 0x7f4df0152650 2024-02-21T23:13:10.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.591+0000 7f4de7400640 1 --2- 172.21.15.28:0/2206182871 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4dcc06b4d0 0x7f4dcc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:10.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.591+0000 7f4de7400640 1 -- 172.21.15.28:0/2206182871 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4dd8089980 con 0x7f4df0152650 2024-02-21T23:13:10.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.592+0000 7f4def400640 1 --2- 172.21.15.28:0/2206182871 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4dcc06b4d0 0x7f4dcc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:10.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.593+0000 7f4def400640 1 --2- 172.21.15.28:0/2206182871 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4dcc06b4d0 0x7f4dcc06d990 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f4de0036000 tx=0x7f4de0008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:10.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.596+0000 7f4de7400640 1 -- 172.21.15.28:0/2206182871 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4dd80585d0 con 0x7f4df0152650 2024-02-21T23:13:10.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.793+0000 7f4df4e00640 1 -- 172.21.15.28:0/2206182871 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4df0156080 con 0x7f4df0152650 2024-02-21T23:13:10.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.794+0000 7f4de7400640 1 -- 172.21.15.28:0/2206182871 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v197) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4dd8057ed0 con 0x7f4df0152650 2024-02-21T23:13:10.782 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-02-21T23:13:10.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.797+0000 7f4df4e00640 1 -- 172.21.15.28:0/2206182871 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4dcc06b4d0 msgr2=0x7f4dcc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:10.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.797+0000 7f4df4e00640 1 --2- 172.21.15.28:0/2206182871 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4dcc06b4d0 0x7f4dcc06d990 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f4de0036000 tx=0x7f4de0008030 comp rx=0 tx=0).stop 2024-02-21T23:13:10.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.797+0000 7f4df4e00640 1 -- 172.21.15.28:0/2206182871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df0152650 msgr2=0x7f4df01e0810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:10.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.797+0000 7f4df4e00640 1 --2- 172.21.15.28:0/2206182871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df0152650 0x7f4df01e0810 secure :-1 s=READY pgs=2293 cs=0 l=1 rev1=1 crypto rx=0x7f4dd800c400 tx=0x7f4dd800c8c0 comp rx=0 tx=0).stop 2024-02-21T23:13:10.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.797+0000 7f4df4e00640 1 -- 172.21.15.28:0/2206182871 shutdown_connections 2024-02-21T23:13:10.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.797+0000 7f4df4e00640 1 --2- 172.21.15.28:0/2206182871 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4dcc06b4d0 0x7f4dcc06d990 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:10.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.797+0000 7f4df4e00640 1 --2- 172.21.15.28:0/2206182871 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4df01550e0 0x7f4df014df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:10.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.797+0000 7f4df4e00640 1 --2- 172.21.15.28:0/2206182871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df0152650 0x7f4df01e0810 unknown :-1 s=CLOSED pgs=2293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:10.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.797+0000 7f4df4e00640 1 --2- 172.21.15.28:0/2206182871 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4df006daa0 0x7f4df014b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:10.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.797+0000 7f4df4e00640 1 -- 172.21.15.28:0/2206182871 >> 172.21.15.28:0/2206182871 conn(0x7f4df0072ce0 msgr2=0x7f4df0153eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:10.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.797+0000 7f4df4e00640 1 -- 172.21.15.28:0/2206182871 shutdown_connections 2024-02-21T23:13:10.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:10.797+0000 7f4df4e00640 1 -- 172.21.15.28:0/2206182871 wait complete. 2024-02-21T23:13:10.805 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-21T23:13:11.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.047+0000 7f9ade200640 1 Processor -- start 2024-02-21T23:13:11.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.047+0000 7f9ade200640 1 -- start start 2024-02-21T23:13:11.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.049+0000 7f9ade200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9ad806b0e0 0x7f9ad80c5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:11.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.049+0000 7f9ade200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9ad80c5d80 0x7f9ad814fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:11.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.049+0000 7f9ade200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ad80c61e0 con 0x7f9ad8150530 2024-02-21T23:13:11.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.049+0000 7f9ade200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ad8153bc0 con 0x7f9ad806b0e0 2024-02-21T23:13:11.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.049+0000 7f9ade200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad8152fb0 con 0x7f9ad80c5d80 2024-02-21T23:13:11.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.049+0000 7f9ad7400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9ad806b0e0 0x7f9ad80c5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:11.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.049+0000 7f9ad7400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9ad806b0e0 0x7f9ad80c5840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55162/0 (socket says 172.21.15.28:55162) 2024-02-21T23:13:11.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.049+0000 7f9ad7400640 1 -- 172.21.15.28:0/824185993 learned_addr learned my addr 172.21.15.28:0/824185993 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:11.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.050+0000 7f9ad6a00640 1 --2- 172.21.15.28:0/824185993 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9ad80c5d80 0x7f9ad814fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:11.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.050+0000 7f9ad7400640 1 -- 172.21.15.28:0/824185993 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9ad80c5d80 msgr2=0x7f9ad814fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:11.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.050+0000 7f9ad7400640 1 --2- 172.21.15.28:0/824185993 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9ad80c5d80 0x7f9ad814fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:11.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.050+0000 7f9ad7400640 1 -- 172.21.15.28:0/824185993 >> v1:172.21.15.81:6789/0 conn(0x7f9ad8150530 legacy=0x7f9ad8156500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:11.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.050+0000 7f9ad7400640 1 -- 172.21.15.28:0/824185993 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ad8153290 con 0x7f9ad806b0e0 2024-02-21T23:13:11.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.053+0000 7f9ad7400640 1 --2- 172.21.15.28:0/824185993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ad806b0e0 0x7f9ad80c5840 secure :-1 s=READY pgs=2294 cs=0 l=1 rev1=1 crypto rx=0x7f9ac8009870 tx=0x7f9ac802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e3d4625389be86c server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:11.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.056+0000 7f9ad6000640 1 -- 172.21.15.28:0/824185993 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ac80317c0 con 0x7f9ad806b0e0 2024-02-21T23:13:11.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.056+0000 7f9ad6000640 1 -- 172.21.15.28:0/824185993 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ac8037070 con 0x7f9ad806b0e0 2024-02-21T23:13:11.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.056+0000 7f9ad6000640 1 -- 172.21.15.28:0/824185993 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ac8031b80 con 0x7f9ad806b0e0 2024-02-21T23:13:11.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.057+0000 7f9ade200640 1 -- 172.21.15.28:0/824185993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ad806b0e0 msgr2=0x7f9ad80c5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:11.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.057+0000 7f9ade200640 1 --2- 172.21.15.28:0/824185993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ad806b0e0 0x7f9ad80c5840 secure :-1 s=READY pgs=2294 cs=0 l=1 rev1=1 crypto rx=0x7f9ac8009870 tx=0x7f9ac802b860 comp rx=0 tx=0).stop 2024-02-21T23:13:11.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.057+0000 7f9ade200640 1 -- 172.21.15.28:0/824185993 shutdown_connections 2024-02-21T23:13:11.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.057+0000 7f9ade200640 1 --2- 172.21.15.28:0/824185993 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9ad80c5d80 0x7f9ad814fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:11.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.057+0000 7f9ade200640 1 --2- 172.21.15.28:0/824185993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ad806b0e0 0x7f9ad80c5840 unknown :-1 s=CLOSED pgs=2294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:11.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.057+0000 7f9ade200640 1 -- 172.21.15.28:0/824185993 >> 172.21.15.28:0/824185993 conn(0x7f9ad80a85a0 msgr2=0x7f9ad80a8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:11.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.057+0000 7f9ade200640 1 -- 172.21.15.28:0/824185993 shutdown_connections 2024-02-21T23:13:11.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.057+0000 7f9ade200640 1 -- 172.21.15.28:0/824185993 wait complete. 2024-02-21T23:13:11.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.059+0000 7f9ade200640 1 Processor -- start 2024-02-21T23:13:11.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.059+0000 7f9ade200640 1 -- start start 2024-02-21T23:13:11.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.060+0000 7f9ade200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9ad806b0e0 0x7f9ad814f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:11.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.060+0000 7f9ade200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9ad80c5d80 0x7f9ad81516e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:11.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.060+0000 7f9ade200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ad8150530 0x7f9ad81e06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:11.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.060+0000 7f9ade200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad81509c0 con 0x7f9ad8150530 2024-02-21T23:13:11.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.060+0000 7f9ade200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad8153850 con 0x7f9ad80c5d80 2024-02-21T23:13:11.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.060+0000 7f9ade200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad806b4c0 con 0x7f9ad806b0e0 2024-02-21T23:13:11.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.060+0000 7f9ad7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ad8150530 0x7f9ad81e06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:11.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.061+0000 7f9ad7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ad8150530 0x7f9ad81e06b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55178/0 (socket says 172.21.15.28:55178) 2024-02-21T23:13:11.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.061+0000 7f9ad7e00640 1 -- 172.21.15.28:0/2600439256 learned_addr learned my addr 172.21.15.28:0/2600439256 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:11.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.061+0000 7f9ad6a00640 1 --2- 172.21.15.28:0/2600439256 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9ad80c5d80 0x7f9ad81516e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:11.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.061+0000 7f9ad7400640 1 --2- 172.21.15.28:0/2600439256 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9ad806b0e0 0x7f9ad814f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:11.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.061+0000 7f9ad7e00640 1 -- 172.21.15.28:0/2600439256 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9ad806b0e0 msgr2=0x7f9ad814f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:11.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.061+0000 7f9ad7e00640 1 --2- 172.21.15.28:0/2600439256 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9ad806b0e0 0x7f9ad814f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:11.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.061+0000 7f9ad7e00640 1 -- 172.21.15.28:0/2600439256 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9ad80c5d80 msgr2=0x7f9ad81516e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:11.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.061+0000 7f9ad7e00640 1 --2- 172.21.15.28:0/2600439256 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9ad80c5d80 0x7f9ad81516e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:11.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.061+0000 7f9ad7e00640 1 -- 172.21.15.28:0/2600439256 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ad814fa80 con 0x7f9ad8150530 2024-02-21T23:13:11.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.061+0000 7f9ad7e00640 1 --2- 172.21.15.28:0/2600439256 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ad8150530 0x7f9ad81e06b0 secure :-1 s=READY pgs=2295 cs=0 l=1 rev1=1 crypto rx=0x7f9ad000b9a0 tx=0x7f9ad000be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:11.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.062+0000 7f9acf400640 1 -- 172.21.15.28:0/2600439256 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad0019070 con 0x7f9ad8150530 2024-02-21T23:13:11.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.062+0000 7f9acf400640 1 -- 172.21.15.28:0/2600439256 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ad000cba0 con 0x7f9ad8150530 2024-02-21T23:13:11.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.062+0000 7f9acf400640 1 -- 172.21.15.28:0/2600439256 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad0014440 con 0x7f9ad8150530 2024-02-21T23:13:11.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.062+0000 7f9ade200640 1 -- 172.21.15.28:0/2600439256 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ad8155e90 con 0x7f9ad8150530 2024-02-21T23:13:11.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.062+0000 7f9ade200640 1 -- 172.21.15.28:0/2600439256 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ad81ee900 con 0x7f9ad8150530 2024-02-21T23:13:11.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.063+0000 7f9ade200640 1 -- 172.21.15.28:0/2600439256 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ad8067f30 con 0x7f9ad8150530 2024-02-21T23:13:11.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.064+0000 7f9acf400640 1 -- 172.21.15.28:0/2600439256 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9ad001c070 con 0x7f9ad8150530 2024-02-21T23:13:11.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.065+0000 7f9acf400640 1 --2- 172.21.15.28:0/2600439256 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9ab806b4d0 0x7f9ab806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:11.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.065+0000 7f9acf400640 1 -- 172.21.15.28:0/2600439256 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f9ad0089f60 con 0x7f9ad8150530 2024-02-21T23:13:11.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.066+0000 7f9ad7400640 1 --2- 172.21.15.28:0/2600439256 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9ab806b4d0 0x7f9ab806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:11.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.067+0000 7f9ad7400640 1 --2- 172.21.15.28:0/2600439256 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9ab806b4d0 0x7f9ab806d990 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f9ac8036000 tx=0x7f9ac8008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:11.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.070+0000 7f9acf400640 1 -- 172.21.15.28:0/2600439256 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f9ad0058c40 con 0x7f9ad8150530 2024-02-21T23:13:11.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.268+0000 7f9ade200640 1 -- 172.21.15.28:0/2600439256 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9ad80c5050 con 0x7f9ad8150530 2024-02-21T23:13:11.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.269+0000 7f9acf400640 1 -- 172.21.15.28:0/2600439256 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v197) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9ad0058540 con 0x7f9ad8150530 2024-02-21T23:13:11.257 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-02-21T23:13:11.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.272+0000 7f9ade200640 1 -- 172.21.15.28:0/2600439256 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9ab806b4d0 msgr2=0x7f9ab806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:11.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.272+0000 7f9ade200640 1 --2- 172.21.15.28:0/2600439256 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9ab806b4d0 0x7f9ab806d990 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f9ac8036000 tx=0x7f9ac8008030 comp rx=0 tx=0).stop 2024-02-21T23:13:11.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.272+0000 7f9ade200640 1 -- 172.21.15.28:0/2600439256 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ad8150530 msgr2=0x7f9ad81e06b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:11.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.272+0000 7f9ade200640 1 --2- 172.21.15.28:0/2600439256 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ad8150530 0x7f9ad81e06b0 secure :-1 s=READY pgs=2295 cs=0 l=1 rev1=1 crypto rx=0x7f9ad000b9a0 tx=0x7f9ad000be60 comp rx=0 tx=0).stop 2024-02-21T23:13:11.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.272+0000 7f9ade200640 1 -- 172.21.15.28:0/2600439256 shutdown_connections 2024-02-21T23:13:11.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.272+0000 7f9ade200640 1 --2- 172.21.15.28:0/2600439256 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9ab806b4d0 0x7f9ab806d990 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:11.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.272+0000 7f9ade200640 1 --2- 172.21.15.28:0/2600439256 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ad8150530 0x7f9ad81e06b0 unknown :-1 s=CLOSED pgs=2295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:11.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.272+0000 7f9ade200640 1 --2- 172.21.15.28:0/2600439256 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9ad80c5d80 0x7f9ad81516e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:11.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.272+0000 7f9ade200640 1 --2- 172.21.15.28:0/2600439256 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9ad806b0e0 0x7f9ad814f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:11.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.272+0000 7f9ade200640 1 -- 172.21.15.28:0/2600439256 >> 172.21.15.28:0/2600439256 conn(0x7f9ad80a85a0 msgr2=0x7f9ad80729d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:11.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.272+0000 7f9ade200640 1 -- 172.21.15.28:0/2600439256 shutdown_connections 2024-02-21T23:13:11.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.272+0000 7f9ade200640 1 -- 172.21.15.28:0/2600439256 wait complete. 2024-02-21T23:13:11.278 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:13:11.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.510+0000 7f4bc1c00640 1 Processor -- start 2024-02-21T23:13:11.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.510+0000 7f4bc1c00640 1 -- start start 2024-02-21T23:13:11.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.511+0000 7f4bc1c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4bbc151f60 0x7f4bbc0c11f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:11.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.511+0000 7f4bc1c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4bbc072ca0 0x7f4bbc15bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:11.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.511+0000 7f4bc1c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bbc06f230 con 0x7f4bbc0c1730 2024-02-21T23:13:11.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.512+0000 7f4bc1c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bbc071a60 con 0x7f4bbc151f60 2024-02-21T23:13:11.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.512+0000 7f4bc1c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bbc071c90 con 0x7f4bbc072ca0 2024-02-21T23:13:11.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.512+0000 7f4bbaa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4bbc151f60 0x7f4bbc0c11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:11.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.512+0000 7f4bba000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f4bbc0c1730 0x7f4bbc06dce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60080/0 (socket says 172.21.15.28:60080) 2024-02-21T23:13:11.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.512+0000 7f4bba000640 1 -- 172.21.15.28:0/3739872266 learned_addr learned my addr 172.21.15.28:0/3739872266 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:13:11.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.512+0000 7f4bbb400640 1 --2- 172.21.15.28:0/3739872266 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4bbc072ca0 0x7f4bbc15bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:11.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.512+0000 7f4bbaa00640 1 -- 172.21.15.28:0/3739872266 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4bbc072ca0 msgr2=0x7f4bbc15bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:11.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.512+0000 7f4bbaa00640 1 --2- 172.21.15.28:0/3739872266 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4bbc072ca0 0x7f4bbc15bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:11.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.512+0000 7f4bbaa00640 1 -- 172.21.15.28:0/3739872266 >> v1:172.21.15.28:6789/0 conn(0x7f4bbc0c1730 legacy=0x7f4bbc06dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:11.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.512+0000 7f4bbaa00640 1 -- 172.21.15.28:0/3739872266 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bbc071f70 con 0x7f4bbc151f60 2024-02-21T23:13:11.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.512+0000 7f4bb9600640 1 -- 172.21.15.28:0/3739872266 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1039861589 0 0) 0x7f4bbc06f230 con 0x7f4bbc0c1730 2024-02-21T23:13:11.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.514+0000 7f4bbaa00640 1 --2- 172.21.15.28:0/3739872266 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4bbc151f60 0x7f4bbc0c11f0 secure :-1 s=READY pgs=2297 cs=0 l=1 rev1=1 crypto rx=0x7f4bac009d90 tx=0x7f4bac02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1894f085522231f5 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:11.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.514+0000 7f4bb9600640 1 -- 172.21.15.28:0/3739872266 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bac039070 con 0x7f4bbc151f60 2024-02-21T23:13:11.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.514+0000 7f4bb9600640 1 -- 172.21.15.28:0/3739872266 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bac02cc50 con 0x7f4bbc151f60 2024-02-21T23:13:11.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.514+0000 7f4bb9600640 1 -- 172.21.15.28:0/3739872266 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bac034440 con 0x7f4bbc151f60 2024-02-21T23:13:11.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.514+0000 7f4bc1c00640 1 -- 172.21.15.28:0/3739872266 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4bbc151f60 msgr2=0x7f4bbc0c11f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:11.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.514+0000 7f4bc1c00640 1 --2- 172.21.15.28:0/3739872266 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4bbc151f60 0x7f4bbc0c11f0 secure :-1 s=READY pgs=2297 cs=0 l=1 rev1=1 crypto rx=0x7f4bac009d90 tx=0x7f4bac02bea0 comp rx=0 tx=0).stop 2024-02-21T23:13:11.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.515+0000 7f4bc1c00640 1 -- 172.21.15.28:0/3739872266 shutdown_connections 2024-02-21T23:13:11.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.515+0000 7f4bc1c00640 1 --2- 172.21.15.28:0/3739872266 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4bbc072ca0 0x7f4bbc15bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:11.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.515+0000 7f4bc1c00640 1 --2- 172.21.15.28:0/3739872266 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4bbc151f60 0x7f4bbc0c11f0 unknown :-1 s=CLOSED pgs=2297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:11.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.515+0000 7f4bc1c00640 1 -- 172.21.15.28:0/3739872266 >> 172.21.15.28:0/3739872266 conn(0x7f4bbc0c67f0 msgr2=0x7f4bbc0c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:11.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.515+0000 7f4bc1c00640 1 -- 172.21.15.28:0/3739872266 shutdown_connections 2024-02-21T23:13:11.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.515+0000 7f4bc1c00640 1 -- 172.21.15.28:0/3739872266 wait complete. 2024-02-21T23:13:11.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.516+0000 7f4bc1c00640 1 Processor -- start 2024-02-21T23:13:11.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.516+0000 7f4bc1c00640 1 -- start start 2024-02-21T23:13:11.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.517+0000 7f4bc1c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4bbc072ca0 0x7f4bbc0bfe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:11.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.517+0000 7f4bc1c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4bbc0c1730 0x7f4bbc15e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:11.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.517+0000 7f4bc1c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4bbc151f60 0x7f4bbc15ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:11.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.517+0000 7f4bc1c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bbc0a6090 con 0x7f4bbc151f60 2024-02-21T23:13:11.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.517+0000 7f4bc1c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bbc152340 con 0x7f4bbc0c1730 2024-02-21T23:13:11.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.517+0000 7f4bc1c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bbc0a6210 con 0x7f4bbc072ca0 2024-02-21T23:13:11.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.518+0000 7f4bbb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4bbc151f60 0x7f4bbc15ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:11.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.518+0000 7f4bbb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4bbc151f60 0x7f4bbc15ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55192/0 (socket says 172.21.15.28:55192) 2024-02-21T23:13:11.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.518+0000 7f4bbb400640 1 -- 172.21.15.28:0/3654670610 learned_addr learned my addr 172.21.15.28:0/3654670610 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:11.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.518+0000 7f4bba000640 1 --2- 172.21.15.28:0/3654670610 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4bbc0c1730 0x7f4bbc15e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:11.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.518+0000 7f4bbaa00640 1 --2- 172.21.15.28:0/3654670610 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4bbc072ca0 0x7f4bbc0bfe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:11.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.518+0000 7f4bbb400640 1 -- 172.21.15.28:0/3654670610 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4bbc072ca0 msgr2=0x7f4bbc0bfe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:11.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.518+0000 7f4bbb400640 1 --2- 172.21.15.28:0/3654670610 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4bbc072ca0 0x7f4bbc0bfe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:11.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.518+0000 7f4bbb400640 1 -- 172.21.15.28:0/3654670610 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4bbc0c1730 msgr2=0x7f4bbc15e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:11.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.518+0000 7f4bbb400640 1 --2- 172.21.15.28:0/3654670610 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4bbc0c1730 0x7f4bbc15e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:11.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.518+0000 7f4bbb400640 1 -- 172.21.15.28:0/3654670610 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bbc160020 con 0x7f4bbc151f60 2024-02-21T23:13:11.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.518+0000 7f4bba000640 1 --2- 172.21.15.28:0/3654670610 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4bbc0c1730 0x7f4bbc15e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:13:11.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.519+0000 7f4bbb400640 1 --2- 172.21.15.28:0/3654670610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4bbc151f60 0x7f4bbc15ecd0 secure :-1 s=READY pgs=2298 cs=0 l=1 rev1=1 crypto rx=0x7f4bb400d800 tx=0x7f4bb400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:11.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.519+0000 7f4bb2a00640 1 -- 172.21.15.28:0/3654670610 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bb400b6c0 con 0x7f4bbc151f60 2024-02-21T23:13:11.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.519+0000 7f4bb2a00640 1 -- 172.21.15.28:0/3654670610 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bb4019070 con 0x7f4bbc151f60 2024-02-21T23:13:11.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.519+0000 7f4bc1c00640 1 -- 172.21.15.28:0/3654670610 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bbc15f370 con 0x7f4bbc151f60 2024-02-21T23:13:11.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.519+0000 7f4bb2a00640 1 -- 172.21.15.28:0/3654670610 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bb400bac0 con 0x7f4bbc151f60 2024-02-21T23:13:11.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.519+0000 7f4bc1c00640 1 -- 172.21.15.28:0/3654670610 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bbc160300 con 0x7f4bbc151f60 2024-02-21T23:13:11.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.520+0000 7f4bc1c00640 1 -- 172.21.15.28:0/3654670610 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bbc067f30 con 0x7f4bbc151f60 2024-02-21T23:13:11.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.521+0000 7f4bb2a00640 1 -- 172.21.15.28:0/3654670610 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4bb400bc60 con 0x7f4bbc151f60 2024-02-21T23:13:11.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.522+0000 7f4bb2a00640 1 --2- 172.21.15.28:0/3654670610 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4b9c06b4d0 0x7f4b9c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:11.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.522+0000 7f4bb2a00640 1 -- 172.21.15.28:0/3654670610 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4bb408a880 con 0x7f4bbc151f60 2024-02-21T23:13:11.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.523+0000 7f4bbaa00640 1 --2- 172.21.15.28:0/3654670610 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4b9c06b4d0 0x7f4b9c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:11.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.524+0000 7f4bbaa00640 1 --2- 172.21.15.28:0/3654670610 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4b9c06b4d0 0x7f4b9c06d990 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f4bac036000 tx=0x7f4bac030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:11.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.527+0000 7f4bb2a00640 1 -- 172.21.15.28:0/3654670610 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4bb4059560 con 0x7f4bbc151f60 2024-02-21T23:13:11.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.767+0000 7f4bc1c00640 1 -- 172.21.15.28:0/3654670610 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4bbc06ea50 con 0x7f4bbc151f60 2024-02-21T23:13:11.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.768+0000 7f4bb2a00640 1 -- 172.21.15.28:0/3654670610 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 221 v221) v1 ==== 95+0+4939 (secure 0 0 0) 0x7f4bb4058e60 con 0x7f4bbc151f60 2024-02-21T23:13:11.755 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:13:11.756 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":221,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7100,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3213818811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3213818811},{"type":"v1","addr":"172.21.15.28:6835","nonce":3213818811}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":213},{"gid":7106,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/1322911156","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":1322911156},{"type":"v1","addr":"172.21.15.81:6837","nonce":1322911156}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":213},{"gid":7184,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/1214841323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1214841323},{"type":"v1","addr":"172.21.15.81:6836","nonce":1214841323}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":221},{"gid":10386,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3814985204","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3814985204},{"type":"v1","addr":"172.21.15.28:6834","nonce":3814985204}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":212}],"filesystems":[{"mdsmap":{"epoch":220,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:12:26.259457+0000","modified":"2024-02-21T23:12:46.343251+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[],"up":{},"failed":[],"damaged":[],"stopped":[0],"info":{},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15}]} 2024-02-21T23:13:11.756 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 221 2024-02-21T23:13:11.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.771+0000 7f4bc1c00640 1 -- 172.21.15.28:0/3654670610 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4b9c06b4d0 msgr2=0x7f4b9c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:11.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.771+0000 7f4bc1c00640 1 --2- 172.21.15.28:0/3654670610 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4b9c06b4d0 0x7f4b9c06d990 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f4bac036000 tx=0x7f4bac030000 comp rx=0 tx=0).stop 2024-02-21T23:13:11.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.771+0000 7f4bc1c00640 1 -- 172.21.15.28:0/3654670610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4bbc151f60 msgr2=0x7f4bbc15ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:11.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.771+0000 7f4bc1c00640 1 --2- 172.21.15.28:0/3654670610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4bbc151f60 0x7f4bbc15ecd0 secure :-1 s=READY pgs=2298 cs=0 l=1 rev1=1 crypto rx=0x7f4bb400d800 tx=0x7f4bb400dcc0 comp rx=0 tx=0).stop 2024-02-21T23:13:11.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.771+0000 7f4bc1c00640 1 -- 172.21.15.28:0/3654670610 shutdown_connections 2024-02-21T23:13:11.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.771+0000 7f4bc1c00640 1 --2- 172.21.15.28:0/3654670610 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4b9c06b4d0 0x7f4b9c06d990 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:11.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.771+0000 7f4bc1c00640 1 --2- 172.21.15.28:0/3654670610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4bbc151f60 0x7f4bbc15ecd0 unknown :-1 s=CLOSED pgs=2298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:11.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.771+0000 7f4bc1c00640 1 --2- 172.21.15.28:0/3654670610 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4bbc0c1730 0x7f4bbc15e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:11.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.771+0000 7f4bc1c00640 1 --2- 172.21.15.28:0/3654670610 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4bbc072ca0 0x7f4bbc0bfe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:11.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.771+0000 7f4bc1c00640 1 -- 172.21.15.28:0/3654670610 >> 172.21.15.28:0/3654670610 conn(0x7f4bbc0c67f0 msgr2=0x7f4bbc0a8420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:11.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.771+0000 7f4bc1c00640 1 -- 172.21.15.28:0/3654670610 shutdown_connections 2024-02-21T23:13:11.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:11.771+0000 7f4bc1c00640 1 -- 172.21.15.28:0/3654670610 wait complete. 2024-02-21T23:13:11.778 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:13:12.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.013+0000 7fe015e00640 1 Processor -- start 2024-02-21T23:13:12.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.014+0000 7fe015e00640 1 -- start start 2024-02-21T23:13:12.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.015+0000 7fe015e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe010153950 0x7fe0100c15d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:12.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.015+0000 7fe015e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fe01015b6d0 0x7fe01015dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:12.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.015+0000 7fe015e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe010152e70 con 0x7fe0100c11f0 2024-02-21T23:13:12.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.015+0000 7fe015e00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0100a5ec0 con 0x7fe01015b6d0 2024-02-21T23:13:12.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.015+0000 7fe015e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe010153150 con 0x7fe010153950 2024-02-21T23:13:12.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.016+0000 7fe00e000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe010153950 0x7fe0100c15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:12.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.016+0000 7fe00f400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fe01015b6d0 0x7fe01015dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:12.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.016+0000 7fe00f400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fe01015b6d0 0x7fe01015dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.28:35562/0 (socket says 172.21.15.28:35562) 2024-02-21T23:13:12.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.016+0000 7fe00e000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe010153950 0x7fe0100c15d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3301/0 says I am v2:172.21.15.28:41260/0 (socket says 172.21.15.28:41260) 2024-02-21T23:13:12.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.016+0000 7fe00f400640 1 -- 172.21.15.28:0/563861804 learned_addr learned my addr 172.21.15.28:0/563861804 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:12.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.016+0000 7fe00ea00640 1 --1- 172.21.15.28:0/563861804 >> v1:172.21.15.81:6789/0 conn(0x7fe0100c11f0 0x7fe01006dce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.28:38332/0 (socket says 172.21.15.28:38332) 2024-02-21T23:13:12.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.017+0000 7fe00f400640 1 -- 172.21.15.28:0/563861804 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe010153950 msgr2=0x7fe0100c15d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:12.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.017+0000 7fe00f400640 1 --2- 172.21.15.28:0/563861804 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe010153950 0x7fe0100c15d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:12.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.017+0000 7fe00f400640 1 -- 172.21.15.28:0/563861804 >> v1:172.21.15.81:6789/0 conn(0x7fe0100c11f0 legacy=0x7fe01006dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:12.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.017+0000 7fe00f400640 1 -- 172.21.15.28:0/563861804 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe010153430 con 0x7fe01015b6d0 2024-02-21T23:13:12.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.018+0000 7fe00f400640 1 --2- 172.21.15.28:0/563861804 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe01015b6d0 0x7fe01015dac0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fe008009870 tx=0x7fe00802b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec76a446105f12ec server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:12.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.019+0000 7fe00d600640 1 -- 172.21.15.28:0/563861804 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe008035020 con 0x7fe01015b6d0 2024-02-21T23:13:12.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.019+0000 7fe00d600640 1 -- 172.21.15.28:0/563861804 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe008031640 con 0x7fe01015b6d0 2024-02-21T23:13:12.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.019+0000 7fe00d600640 1 -- 172.21.15.28:0/563861804 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe00802d040 con 0x7fe01015b6d0 2024-02-21T23:13:12.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.019+0000 7fe015e00640 1 -- 172.21.15.28:0/563861804 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe01015b6d0 msgr2=0x7fe01015dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:12.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.019+0000 7fe015e00640 1 --2- 172.21.15.28:0/563861804 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe01015b6d0 0x7fe01015dac0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fe008009870 tx=0x7fe00802b8f0 comp rx=0 tx=0).stop 2024-02-21T23:13:12.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.019+0000 7fe015e00640 1 -- 172.21.15.28:0/563861804 shutdown_connections 2024-02-21T23:13:12.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.020+0000 7fe015e00640 1 --2- 172.21.15.28:0/563861804 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe01015b6d0 0x7fe01015dac0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:12.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.020+0000 7fe015e00640 1 --2- 172.21.15.28:0/563861804 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe010153950 0x7fe0100c15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:12.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.020+0000 7fe015e00640 1 -- 172.21.15.28:0/563861804 >> 172.21.15.28:0/563861804 conn(0x7fe0100c67f0 msgr2=0x7fe0100c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:12.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.020+0000 7fe015e00640 1 -- 172.21.15.28:0/563861804 shutdown_connections 2024-02-21T23:13:12.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.020+0000 7fe015e00640 1 -- 172.21.15.28:0/563861804 wait complete. 2024-02-21T23:13:12.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.021+0000 7fe015e00640 1 Processor -- start 2024-02-21T23:13:12.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.022+0000 7fe015e00640 1 -- start start 2024-02-21T23:13:12.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.023+0000 7fe015e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe0100c11f0 0x7fe0101eedb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:12.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.023+0000 7fe015e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe010153950 0x7fe010148db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:12.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.023+0000 7fe015e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe01015b6d0 0x7fe0101eaea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:12.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.023+0000 7fe015e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0100a6110 con 0x7fe010153950 2024-02-21T23:13:12.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.023+0000 7fe015e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0100c4c00 con 0x7fe0100c11f0 2024-02-21T23:13:12.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.023+0000 7fe015e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0100c0d90 con 0x7fe01015b6d0 2024-02-21T23:13:12.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.023+0000 7fe00e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe010153950 0x7fe010148db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:12.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.023+0000 7fe00e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe010153950 0x7fe010148db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55202/0 (socket says 172.21.15.28:55202) 2024-02-21T23:13:12.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.023+0000 7fe00e000640 1 -- 172.21.15.28:0/3759841954 learned_addr learned my addr 172.21.15.28:0/3759841954 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:12.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.023+0000 7fe00ea00640 1 --2- 172.21.15.28:0/3759841954 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe0100c11f0 0x7fe0101eedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:12.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.023+0000 7fe00f400640 1 --2- 172.21.15.28:0/3759841954 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe01015b6d0 0x7fe0101eaea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:12.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.023+0000 7fe00e000640 1 -- 172.21.15.28:0/3759841954 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe01015b6d0 msgr2=0x7fe0101eaea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:12.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.023+0000 7fe00e000640 1 --2- 172.21.15.28:0/3759841954 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe01015b6d0 0x7fe0101eaea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:12.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.023+0000 7fe00e000640 1 -- 172.21.15.28:0/3759841954 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe0100c11f0 msgr2=0x7fe0101eedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:12.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.023+0000 7fe00e000640 1 --2- 172.21.15.28:0/3759841954 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe0100c11f0 0x7fe0101eedb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:12.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.024+0000 7fe00e000640 1 -- 172.21.15.28:0/3759841954 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0101ef2f0 con 0x7fe010153950 2024-02-21T23:13:12.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.024+0000 7fe00e000640 1 --2- 172.21.15.28:0/3759841954 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe010153950 0x7fe010148db0 secure :-1 s=READY pgs=2299 cs=0 l=1 rev1=1 crypto rx=0x7fe00000efc0 tx=0x7fe00000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:12.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.024+0000 7fe007400640 1 -- 172.21.15.28:0/3759841954 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe000018020 con 0x7fe010153950 2024-02-21T23:13:12.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.024+0000 7fe007400640 1 -- 172.21.15.28:0/3759841954 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe000014480 con 0x7fe010153950 2024-02-21T23:13:12.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.024+0000 7fe007400640 1 -- 172.21.15.28:0/3759841954 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe000014850 con 0x7fe010153950 2024-02-21T23:13:12.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.024+0000 7fe015e00640 1 -- 172.21.15.28:0/3759841954 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe010072820 con 0x7fe010153950 2024-02-21T23:13:12.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.024+0000 7fe015e00640 1 -- 172.21.15.28:0/3759841954 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0101ee5b0 con 0x7fe010153950 2024-02-21T23:13:12.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.026+0000 7fe015e00640 1 -- 172.21.15.28:0/3759841954 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0100a7da0 con 0x7fe010153950 2024-02-21T23:13:12.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.027+0000 7fe007400640 1 -- 172.21.15.28:0/3759841954 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe000005d10 con 0x7fe010153950 2024-02-21T23:13:12.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.028+0000 7fe007400640 1 --2- 172.21.15.28:0/3759841954 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdfec06b4d0 0x7fdfec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:12.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.028+0000 7fe007400640 1 -- 172.21.15.28:0/3759841954 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe00008b240 con 0x7fe010153950 2024-02-21T23:13:12.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.028+0000 7fe00ea00640 1 --2- 172.21.15.28:0/3759841954 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdfec06b4d0 0x7fdfec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:12.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.029+0000 7fe00ea00640 1 --2- 172.21.15.28:0/3759841954 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdfec06b4d0 0x7fdfec06d990 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fdff8000f60 tx=0x7fdff8007ca0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:12.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.032+0000 7fe007400640 1 -- 172.21.15.28:0/3759841954 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe0100a7da0 con 0x7fe010153950 2024-02-21T23:13:12.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.237+0000 7fe015e00640 1 -- 172.21.15.28:0/3759841954 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe010067f30 con 0x7fe010153950 2024-02-21T23:13:12.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.240+0000 7fe007400640 1 -- 172.21.15.28:0/3759841954 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v197) v1 ==== 74+0+16892 (secure 0 0 0) 0x7fe000022030 con 0x7fe010153950 2024-02-21T23:13:12.228 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:13:12.228 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":197,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:12:27.821567+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:12:24.201589+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"196","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2024-02-21T23:12:25.295387+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"197","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:13:12.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.243+0000 7fe015e00640 1 -- 172.21.15.28:0/3759841954 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdfec06b4d0 msgr2=0x7fdfec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:12.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.243+0000 7fe015e00640 1 --2- 172.21.15.28:0/3759841954 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdfec06b4d0 0x7fdfec06d990 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fdff8000f60 tx=0x7fdff8007ca0 comp rx=0 tx=0).stop 2024-02-21T23:13:12.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.243+0000 7fe015e00640 1 -- 172.21.15.28:0/3759841954 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe010153950 msgr2=0x7fe010148db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:12.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.243+0000 7fe015e00640 1 --2- 172.21.15.28:0/3759841954 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe010153950 0x7fe010148db0 secure :-1 s=READY pgs=2299 cs=0 l=1 rev1=1 crypto rx=0x7fe00000efc0 tx=0x7fe00000c490 comp rx=0 tx=0).stop 2024-02-21T23:13:12.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.243+0000 7fe015e00640 1 -- 172.21.15.28:0/3759841954 shutdown_connections 2024-02-21T23:13:12.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.243+0000 7fe015e00640 1 --2- 172.21.15.28:0/3759841954 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdfec06b4d0 0x7fdfec06d990 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:12.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.243+0000 7fe015e00640 1 --2- 172.21.15.28:0/3759841954 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe01015b6d0 0x7fe0101eaea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:12.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.243+0000 7fe015e00640 1 --2- 172.21.15.28:0/3759841954 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe010153950 0x7fe010148db0 unknown :-1 s=CLOSED pgs=2299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:12.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.243+0000 7fe015e00640 1 --2- 172.21.15.28:0/3759841954 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe0100c11f0 0x7fe0101eedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:12.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.243+0000 7fe015e00640 1 -- 172.21.15.28:0/3759841954 >> 172.21.15.28:0/3759841954 conn(0x7fe0100c67f0 msgr2=0x7fe01006e4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:12.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.243+0000 7fe015e00640 1 -- 172.21.15.28:0/3759841954 shutdown_connections 2024-02-21T23:13:12.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.243+0000 7fe015e00640 1 -- 172.21.15.28:0/3759841954 wait complete. 2024-02-21T23:13:12.250 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-02-21T23:13:12.250 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:13:12.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.488+0000 7f0825000640 1 Processor -- start 2024-02-21T23:13:12.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.488+0000 7f0825000640 1 -- start start 2024-02-21T23:13:12.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.489+0000 7f0825000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f082006dbf0 0x7f0820158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:12.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.489+0000 7f0825000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0820155960 0x7f0820159000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:12.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.489+0000 7f0825000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08200c74c0 con 0x7f08200aff90 2024-02-21T23:13:12.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.489+0000 7f0825000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08200c7270 con 0x7f082006dbf0 2024-02-21T23:13:12.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.489+0000 7f0825000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0820158630 con 0x7f0820155960 2024-02-21T23:13:12.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.490+0000 7f081f400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f08200aff90 0x7f08200c51e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60090/0 (socket says 172.21.15.28:60090) 2024-02-21T23:13:12.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.490+0000 7f081f400640 1 -- 172.21.15.28:0/3291488341 learned_addr learned my addr 172.21.15.28:0/3291488341 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:13:12.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.490+0000 7f081ea00640 1 --2- 172.21.15.28:0/3291488341 >> v2:172.21.15.81:3300/0 conn(0x7f082006dbf0 0x7f0820158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:12.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.490+0000 7f081fe00640 1 --2- 172.21.15.28:0/3291488341 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0820155960 0x7f0820159000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:12.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.490+0000 7f081e000640 1 -- 172.21.15.28:0/3291488341 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4051458589 0 0) 0x7f08200c74c0 con 0x7f08200aff90 2024-02-21T23:13:12.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.490+0000 7f081e000640 1 -- 172.21.15.28:0/3291488341 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0804003520 con 0x7f08200aff90 2024-02-21T23:13:12.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.491+0000 7f081e000640 1 -- 172.21.15.28:0/3291488341 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 454563923 0 0) 0x7f0804003520 con 0x7f08200aff90 2024-02-21T23:13:12.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.491+0000 7f081ea00640 1 -- 172.21.15.28:0/3291488341 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0820155960 msgr2=0x7f0820159000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:12.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.491+0000 7f081ea00640 1 --2- 172.21.15.28:0/3291488341 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0820155960 0x7f0820159000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:12.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.491+0000 7f081ea00640 1 -- 172.21.15.28:0/3291488341 >> v1:172.21.15.28:6789/0 conn(0x7f08200aff90 legacy=0x7f08200c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:12.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.491+0000 7f081ea00640 1 -- 172.21.15.28:0/3291488341 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08200c77a0 con 0x7f082006dbf0 2024-02-21T23:13:12.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.492+0000 7f081ea00640 1 --2- 172.21.15.28:0/3291488341 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f082006dbf0 0x7f0820158ac0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f0810009d90 tx=0x7f081002bd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ddd0549f3989889f server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:12.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.492+0000 7f081e000640 1 -- 172.21.15.28:0/3291488341 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0810039070 con 0x7f082006dbf0 2024-02-21T23:13:12.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.493+0000 7f081e000640 1 -- 172.21.15.28:0/3291488341 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f081002c8a0 con 0x7f082006dbf0 2024-02-21T23:13:12.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.493+0000 7f081e000640 1 -- 172.21.15.28:0/3291488341 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0808001670 con 0x7f082006dbf0 2024-02-21T23:13:12.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.493+0000 7f0825000640 1 -- 172.21.15.28:0/3291488341 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f082006dbf0 msgr2=0x7f0820158ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:12.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.493+0000 7f0825000640 1 --2- 172.21.15.28:0/3291488341 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f082006dbf0 0x7f0820158ac0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f0810009d90 tx=0x7f081002bd70 comp rx=0 tx=0).stop 2024-02-21T23:13:12.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.493+0000 7f0825000640 1 -- 172.21.15.28:0/3291488341 shutdown_connections 2024-02-21T23:13:12.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.493+0000 7f0825000640 1 --2- 172.21.15.28:0/3291488341 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0820155960 0x7f0820159000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:12.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.493+0000 7f0825000640 1 --2- 172.21.15.28:0/3291488341 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f082006dbf0 0x7f0820158ac0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:12.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.493+0000 7f0825000640 1 -- 172.21.15.28:0/3291488341 >> 172.21.15.28:0/3291488341 conn(0x7f08200ac370 msgr2=0x7f08200ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:12.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.493+0000 7f0825000640 1 -- 172.21.15.28:0/3291488341 shutdown_connections 2024-02-21T23:13:12.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.494+0000 7f0825000640 1 -- 172.21.15.28:0/3291488341 wait complete. 2024-02-21T23:13:12.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.495+0000 7f0825000640 1 Processor -- start 2024-02-21T23:13:12.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.495+0000 7f0825000640 1 -- start start 2024-02-21T23:13:12.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.496+0000 7f0825000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f082006dbf0 0x7f08200adeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:12.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.496+0000 7f0825000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08200aff90 0x7f08201ea9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:12.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.496+0000 7f0825000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0820155960 0x7f08201eaf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:12.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.496+0000 7f0825000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08200c58f0 con 0x7f08200aff90 2024-02-21T23:13:12.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.496+0000 7f0825000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08200be0c0 con 0x7f082006dbf0 2024-02-21T23:13:12.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.496+0000 7f0825000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08200c5a70 con 0x7f0820155960 2024-02-21T23:13:12.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.496+0000 7f081ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08200aff90 0x7f08201ea9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:12.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.497+0000 7f081ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08200aff90 0x7f08201ea9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55214/0 (socket says 172.21.15.28:55214) 2024-02-21T23:13:12.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.497+0000 7f081ea00640 1 -- 172.21.15.28:0/1817731469 learned_addr learned my addr 172.21.15.28:0/1817731469 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:12.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.497+0000 7f081fe00640 1 --2- 172.21.15.28:0/1817731469 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0820155960 0x7f08201eaf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:12.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.497+0000 7f081f400640 1 --2- 172.21.15.28:0/1817731469 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f082006dbf0 0x7f08200adeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:12.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.497+0000 7f081ea00640 1 -- 172.21.15.28:0/1817731469 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0820155960 msgr2=0x7f08201eaf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:12.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.497+0000 7f081ea00640 1 --2- 172.21.15.28:0/1817731469 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0820155960 0x7f08201eaf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:12.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.497+0000 7f081ea00640 1 -- 172.21.15.28:0/1817731469 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f082006dbf0 msgr2=0x7f08200adeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:12.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.497+0000 7f081ea00640 1 --2- 172.21.15.28:0/1817731469 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f082006dbf0 0x7f08200adeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:12.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.497+0000 7f081ea00640 1 -- 172.21.15.28:0/1817731469 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08201eee30 con 0x7f08200aff90 2024-02-21T23:13:12.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.497+0000 7f081ea00640 1 --2- 172.21.15.28:0/1817731469 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08200aff90 0x7f08201ea9f0 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 crypto rx=0x7f0810009eb0 tx=0x7f0810004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:12.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.498+0000 7f0817400640 1 -- 172.21.15.28:0/1817731469 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0810039050 con 0x7f08200aff90 2024-02-21T23:13:12.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.498+0000 7f0817400640 1 -- 172.21.15.28:0/1817731469 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0810045070 con 0x7f08200aff90 2024-02-21T23:13:12.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.498+0000 7f0817400640 1 -- 172.21.15.28:0/1817731469 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0810009450 con 0x7f08200aff90 2024-02-21T23:13:12.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.498+0000 7f0825000640 1 -- 172.21.15.28:0/1817731469 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0820151bc0 con 0x7f08200aff90 2024-02-21T23:13:12.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.498+0000 7f0825000640 1 -- 172.21.15.28:0/1817731469 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08200ae3f0 con 0x7f08200aff90 2024-02-21T23:13:12.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.499+0000 7f0825000640 1 -- 172.21.15.28:0/1817731469 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08200c4eb0 con 0x7f08200aff90 2024-02-21T23:13:12.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.500+0000 7f0817400640 1 -- 172.21.15.28:0/1817731469 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0810004690 con 0x7f08200aff90 2024-02-21T23:13:12.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.501+0000 7f0817400640 1 --2- 172.21.15.28:0/1817731469 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f07fc06b4d0 0x7f07fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:12.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.501+0000 7f0817400640 1 -- 172.21.15.28:0/1817731469 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f08100acc00 con 0x7f08200aff90 2024-02-21T23:13:12.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.502+0000 7f081f400640 1 --2- 172.21.15.28:0/1817731469 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f07fc06b4d0 0x7f07fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:12.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.503+0000 7f081f400640 1 --2- 172.21.15.28:0/1817731469 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f07fc06b4d0 0x7f07fc06d990 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f08080011e0 tx=0x7f0808009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:12.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.505+0000 7f0817400640 1 -- 172.21.15.28:0/1817731469 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f08200c4eb0 con 0x7f08200aff90 2024-02-21T23:13:12.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.743+0000 7f0825000640 1 -- 172.21.15.28:0/1817731469 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0820074b20 con 0x7f08200aff90 2024-02-21T23:13:12.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.744+0000 7f0817400640 1 -- 172.21.15.28:0/1817731469 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 221 v221) v1 ==== 95+0+4939 (secure 0 0 0) 0x7f0820074b20 con 0x7f08200aff90 2024-02-21T23:13:12.732 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:13:12.732 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":221,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7100,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3213818811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3213818811},{"type":"v1","addr":"172.21.15.28:6835","nonce":3213818811}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":213},{"gid":7106,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/1322911156","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":1322911156},{"type":"v1","addr":"172.21.15.81:6837","nonce":1322911156}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":213},{"gid":7184,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/1214841323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1214841323},{"type":"v1","addr":"172.21.15.81:6836","nonce":1214841323}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":221},{"gid":10386,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3814985204","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3814985204},{"type":"v1","addr":"172.21.15.28:6834","nonce":3814985204}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":212}],"filesystems":[{"mdsmap":{"epoch":220,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:12:26.259457+0000","modified":"2024-02-21T23:12:46.343251+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[],"up":{},"failed":[],"damaged":[],"stopped":[0],"info":{},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15}]} 2024-02-21T23:13:12.733 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 221 2024-02-21T23:13:12.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.746+0000 7f0825000640 1 -- 172.21.15.28:0/1817731469 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f07fc06b4d0 msgr2=0x7f07fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:12.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.746+0000 7f0825000640 1 --2- 172.21.15.28:0/1817731469 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f07fc06b4d0 0x7f07fc06d990 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f08080011e0 tx=0x7f0808009040 comp rx=0 tx=0).stop 2024-02-21T23:13:12.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.747+0000 7f0825000640 1 -- 172.21.15.28:0/1817731469 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08200aff90 msgr2=0x7f08201ea9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:12.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.747+0000 7f0825000640 1 --2- 172.21.15.28:0/1817731469 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08200aff90 0x7f08201ea9f0 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 crypto rx=0x7f0810009eb0 tx=0x7f0810004a40 comp rx=0 tx=0).stop 2024-02-21T23:13:12.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.747+0000 7f0825000640 1 -- 172.21.15.28:0/1817731469 shutdown_connections 2024-02-21T23:13:12.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.747+0000 7f0825000640 1 --2- 172.21.15.28:0/1817731469 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f07fc06b4d0 0x7f07fc06d990 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:12.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.747+0000 7f0825000640 1 --2- 172.21.15.28:0/1817731469 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0820155960 0x7f08201eaf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:12.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.747+0000 7f0825000640 1 --2- 172.21.15.28:0/1817731469 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08200aff90 0x7f08201ea9f0 unknown :-1 s=CLOSED pgs=2301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:12.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.747+0000 7f0825000640 1 --2- 172.21.15.28:0/1817731469 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f082006dbf0 0x7f08200adeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:12.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.747+0000 7f0825000640 1 -- 172.21.15.28:0/1817731469 >> 172.21.15.28:0/1817731469 conn(0x7f08200ac370 msgr2=0x7f0820156300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:12.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.747+0000 7f0825000640 1 -- 172.21.15.28:0/1817731469 shutdown_connections 2024-02-21T23:13:12.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.747+0000 7f0825000640 1 -- 172.21.15.28:0/1817731469 wait complete. 2024-02-21T23:13:12.755 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:13:12.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.991+0000 7fdc7b200640 1 Processor -- start 2024-02-21T23:13:12.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.992+0000 7fdc7b200640 1 -- start start 2024-02-21T23:13:12.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.993+0000 7fdc7b200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdc7c152500 0x7fdc7c1558c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:12.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.993+0000 7fdc7b200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fdc7c0ca050 0x7fdc7c0bf6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:12.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.993+0000 7fdc7b200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdc7c0bfc00 0x7fdc7c06bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:12.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.993+0000 7fdc7b200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc7c1528e0 con 0x7fdc7c0bfc00 2024-02-21T23:13:12.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.993+0000 7fdc7b200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc7c0c86e0 con 0x7fdc7c0ca050 2024-02-21T23:13:12.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.993+0000 7fdc7b200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7c0c8980 con 0x7fdc7c152500 2024-02-21T23:13:12.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.994+0000 7fdc7a800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdc7c0bfc00 0x7fdc7c06bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:12.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.994+0000 7fdc7a800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdc7c0bfc00 0x7fdc7c06bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55218/0 (socket says 172.21.15.28:55218) 2024-02-21T23:13:12.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.994+0000 7fdc7a800640 1 -- 172.21.15.28:0/193209373 learned_addr learned my addr 172.21.15.28:0/193209373 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:12.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.994+0000 7fdc79e00640 1 --2- 172.21.15.28:0/193209373 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdc7c152500 0x7fdc7c1558c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:12.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.994+0000 7fdc79400640 1 --2- 172.21.15.28:0/193209373 >> v2:172.21.15.81:3300/0 conn(0x7fdc7c0ca050 0x7fdc7c0bf6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:12.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.994+0000 7fdc7a800640 1 -- 172.21.15.28:0/193209373 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdc7c152500 msgr2=0x7fdc7c1558c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:12.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.994+0000 7fdc7a800640 1 --2- 172.21.15.28:0/193209373 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdc7c152500 0x7fdc7c1558c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:12.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.994+0000 7fdc7a800640 1 -- 172.21.15.28:0/193209373 >> v2:172.21.15.81:3300/0 conn(0x7fdc7c0ca050 msgr2=0x7fdc7c0bf6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:12.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.994+0000 7fdc7a800640 1 --2- 172.21.15.28:0/193209373 >> v2:172.21.15.81:3300/0 conn(0x7fdc7c0ca050 0x7fdc7c0bf6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:12.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.994+0000 7fdc7a800640 1 -- 172.21.15.28:0/193209373 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc7c0c8c60 con 0x7fdc7c0bfc00 2024-02-21T23:13:12.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.994+0000 7fdc79400640 1 --2- 172.21.15.28:0/193209373 >> v2:172.21.15.81:3300/0 conn(0x7fdc7c0ca050 0x7fdc7c0bf6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:13:12.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.995+0000 7fdc7a800640 1 --2- 172.21.15.28:0/193209373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc7c0bfc00 0x7fdc7c06bbf0 secure :-1 s=READY pgs=2302 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c0c7d80 tx=0x7fdc7402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6020bd520a595a8d server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:12.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.995+0000 7fdc78a00640 1 -- 172.21.15.28:0/193209373 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc7402c9f0 con 0x7fdc7c0bfc00 2024-02-21T23:13:12.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.996+0000 7fdc78a00640 1 -- 172.21.15.28:0/193209373 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc7402cb90 con 0x7fdc7c0bfc00 2024-02-21T23:13:12.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.996+0000 7fdc78a00640 1 -- 172.21.15.28:0/193209373 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc740343f0 con 0x7fdc7c0bfc00 2024-02-21T23:13:12.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.996+0000 7fdc7b200640 1 -- 172.21.15.28:0/193209373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc7c0bfc00 msgr2=0x7fdc7c06bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:12.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.996+0000 7fdc7b200640 1 --2- 172.21.15.28:0/193209373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc7c0bfc00 0x7fdc7c06bbf0 secure :-1 s=READY pgs=2302 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c0c7d80 tx=0x7fdc7402be50 comp rx=0 tx=0).stop 2024-02-21T23:13:12.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.996+0000 7fdc7b200640 1 -- 172.21.15.28:0/193209373 shutdown_connections 2024-02-21T23:13:12.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.996+0000 7fdc7b200640 1 --2- 172.21.15.28:0/193209373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc7c0bfc00 0x7fdc7c06bbf0 unknown :-1 s=CLOSED pgs=2302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:12.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.996+0000 7fdc7b200640 1 --2- 172.21.15.28:0/193209373 >> v2:172.21.15.81:3300/0 conn(0x7fdc7c0ca050 0x7fdc7c0bf6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:12.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.996+0000 7fdc7b200640 1 --2- 172.21.15.28:0/193209373 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdc7c152500 0x7fdc7c1558c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:12.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.996+0000 7fdc7b200640 1 -- 172.21.15.28:0/193209373 >> 172.21.15.28:0/193209373 conn(0x7fdc7c0ac450 msgr2=0x7fdc7c0ac860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:12.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.996+0000 7fdc7b200640 1 -- 172.21.15.28:0/193209373 shutdown_connections 2024-02-21T23:13:12.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.996+0000 7fdc7b200640 1 -- 172.21.15.28:0/193209373 wait complete. 2024-02-21T23:13:12.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.998+0000 7fdc7b200640 1 Processor -- start 2024-02-21T23:13:12.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.998+0000 7fdc7b200640 1 -- start start 2024-02-21T23:13:12.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.999+0000 7fdc7b200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdc7c0bfc00 0x7fdc7c1edd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:12.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.999+0000 7fdc7b200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc7c0ca050 0x7fdc7c144790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:12.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.999+0000 7fdc7b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc7c152500 0x7fdc7c144cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:12.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.999+0000 7fdc7b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7c06c130 con 0x7fdc7c152500 2024-02-21T23:13:12.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.999+0000 7fdc7b200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7c0c9220 con 0x7fdc7c0ca050 2024-02-21T23:13:12.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.999+0000 7fdc7b200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7c06c2b0 con 0x7fdc7c0bfc00 2024-02-21T23:13:12.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.999+0000 7fdc7a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc7c152500 0x7fdc7c144cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:12.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.999+0000 7fdc7a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc7c152500 0x7fdc7c144cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55222/0 (socket says 172.21.15.28:55222) 2024-02-21T23:13:12.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:12.999+0000 7fdc7a800640 1 -- 172.21.15.28:0/3680563989 learned_addr learned my addr 172.21.15.28:0/3680563989 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:12.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.000+0000 7fdc79e00640 1 --2- 172.21.15.28:0/3680563989 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdc7c0bfc00 0x7fdc7c1edd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:12.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.000+0000 7fdc79400640 1 --2- 172.21.15.28:0/3680563989 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc7c0ca050 0x7fdc7c144790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:12.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.000+0000 7fdc7a800640 1 -- 172.21.15.28:0/3680563989 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdc7c0bfc00 msgr2=0x7fdc7c1edd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:12.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.000+0000 7fdc7a800640 1 --2- 172.21.15.28:0/3680563989 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdc7c0bfc00 0x7fdc7c1edd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:12.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.000+0000 7fdc7a800640 1 -- 172.21.15.28:0/3680563989 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc7c0ca050 msgr2=0x7fdc7c144790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:12.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.000+0000 7fdc7a800640 1 --2- 172.21.15.28:0/3680563989 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc7c0ca050 0x7fdc7c144790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:12.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.000+0000 7fdc7a800640 1 -- 172.21.15.28:0/3680563989 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc7c0b0730 con 0x7fdc7c152500 2024-02-21T23:13:12.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.000+0000 7fdc7a800640 1 --2- 172.21.15.28:0/3680563989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc7c152500 0x7fdc7c144cd0 secure :-1 s=READY pgs=2303 cs=0 l=1 rev1=1 crypto rx=0x7fdc74036040 tx=0x7fdc74004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:12.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.001+0000 7fdc72000640 1 -- 172.21.15.28:0/3680563989 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc74043020 con 0x7fdc7c152500 2024-02-21T23:13:12.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.001+0000 7fdc72000640 1 -- 172.21.15.28:0/3680563989 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc7403e450 con 0x7fdc7c152500 2024-02-21T23:13:12.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.001+0000 7fdc72000640 1 -- 172.21.15.28:0/3680563989 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc74009510 con 0x7fdc7c152500 2024-02-21T23:13:12.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.001+0000 7fdc7b200640 1 -- 172.21.15.28:0/3680563989 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc7c1ee600 con 0x7fdc7c152500 2024-02-21T23:13:12.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.001+0000 7fdc7b200640 1 -- 172.21.15.28:0/3680563989 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc7c1452b0 con 0x7fdc7c152500 2024-02-21T23:13:12.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.002+0000 7fdc7b200640 1 -- 172.21.15.28:0/3680563989 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc7c04f290 con 0x7fdc7c152500 2024-02-21T23:13:12.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.003+0000 7fdc72000640 1 -- 172.21.15.28:0/3680563989 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdc74046020 con 0x7fdc7c152500 2024-02-21T23:13:12.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.004+0000 7fdc72000640 1 --2- 172.21.15.28:0/3680563989 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdc5c06b4d0 0x7fdc5c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:12.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.004+0000 7fdc72000640 1 -- 172.21.15.28:0/3680563989 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fdc740acc80 con 0x7fdc7c152500 2024-02-21T23:13:12.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.004+0000 7fdc79e00640 1 --2- 172.21.15.28:0/3680563989 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdc5c06b4d0 0x7fdc5c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:12.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.005+0000 7fdc79e00640 1 --2- 172.21.15.28:0/3680563989 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdc5c06b4d0 0x7fdc5c06d990 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fdc6c006fd0 tx=0x7fdc6c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:12.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.009+0000 7fdc72000640 1 -- 172.21.15.28:0/3680563989 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fdc7404d030 con 0x7fdc7c152500 2024-02-21T23:13:13.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.253+0000 7fdc7b200640 1 -- 172.21.15.28:0/3680563989 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdc7c1ee7d0 con 0x7fdc7c152500 2024-02-21T23:13:13.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.254+0000 7fdc72000640 1 -- 172.21.15.28:0/3680563989 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 221 v221) v1 ==== 95+0+4939 (secure 0 0 0) 0x7fdc7407b8d0 con 0x7fdc7c152500 2024-02-21T23:13:13.242 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:13:13.242 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":221,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7100,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3213818811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3213818811},{"type":"v1","addr":"172.21.15.28:6835","nonce":3213818811}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":213},{"gid":7106,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/1322911156","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":1322911156},{"type":"v1","addr":"172.21.15.81:6837","nonce":1322911156}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":213},{"gid":7184,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/1214841323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1214841323},{"type":"v1","addr":"172.21.15.81:6836","nonce":1214841323}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":221},{"gid":10386,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3814985204","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3814985204},{"type":"v1","addr":"172.21.15.28:6834","nonce":3814985204}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":212}],"filesystems":[{"mdsmap":{"epoch":220,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:12:26.259457+0000","modified":"2024-02-21T23:12:46.343251+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[],"up":{},"failed":[],"damaged":[],"stopped":[0],"info":{},"data_pools":[32],"metadata_pool":31,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15}]} 2024-02-21T23:13:13.242 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 221 2024-02-21T23:13:13.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.257+0000 7fdc7b200640 1 -- 172.21.15.28:0/3680563989 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdc5c06b4d0 msgr2=0x7fdc5c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:13.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.257+0000 7fdc7b200640 1 --2- 172.21.15.28:0/3680563989 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdc5c06b4d0 0x7fdc5c06d990 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fdc6c006fd0 tx=0x7fdc6c008040 comp rx=0 tx=0).stop 2024-02-21T23:13:13.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.257+0000 7fdc7b200640 1 -- 172.21.15.28:0/3680563989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc7c152500 msgr2=0x7fdc7c144cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:13.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.257+0000 7fdc7b200640 1 --2- 172.21.15.28:0/3680563989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc7c152500 0x7fdc7c144cd0 secure :-1 s=READY pgs=2303 cs=0 l=1 rev1=1 crypto rx=0x7fdc74036040 tx=0x7fdc74004a40 comp rx=0 tx=0).stop 2024-02-21T23:13:13.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.257+0000 7fdc7b200640 1 -- 172.21.15.28:0/3680563989 shutdown_connections 2024-02-21T23:13:13.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.257+0000 7fdc7b200640 1 --2- 172.21.15.28:0/3680563989 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdc5c06b4d0 0x7fdc5c06d990 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:13.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.257+0000 7fdc7b200640 1 --2- 172.21.15.28:0/3680563989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc7c152500 0x7fdc7c144cd0 unknown :-1 s=CLOSED pgs=2303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:13.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.257+0000 7fdc7b200640 1 --2- 172.21.15.28:0/3680563989 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc7c0ca050 0x7fdc7c144790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:13.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.257+0000 7fdc7b200640 1 --2- 172.21.15.28:0/3680563989 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdc7c0bfc00 0x7fdc7c1edd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:13.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.257+0000 7fdc7b200640 1 -- 172.21.15.28:0/3680563989 >> 172.21.15.28:0/3680563989 conn(0x7fdc7c0ac450 msgr2=0x7fdc7c1540a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:13.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.257+0000 7fdc7b200640 1 -- 172.21.15.28:0/3680563989 shutdown_connections 2024-02-21T23:13:13.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.257+0000 7fdc7b200640 1 -- 172.21.15.28:0/3680563989 wait complete. 2024-02-21T23:13:13.266 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:13:13.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.501+0000 7f25ff000640 1 Processor -- start 2024-02-21T23:13:13.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.501+0000 7f25ff000640 1 -- start start 2024-02-21T23:13:13.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.503+0000 7f25ff000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f25f8150450 0x7f25f80a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:13.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.503+0000 7f25ff000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f25f806ee40 0x7f25f81536f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:13.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.503+0000 7f25ff000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25f80c2760 con 0x7f25f8156af0 2024-02-21T23:13:13.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.503+0000 7f25ff000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25f80c22c0 con 0x7f25f806ee40 2024-02-21T23:13:13.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.503+0000 7f25ff000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25f8156660 con 0x7f25f8150450 2024-02-21T23:13:13.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.503+0000 7f25fdc00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f25f8150450 0x7f25f80a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:13.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.504+0000 7f25fd200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f25f806ee40 0x7f25f81536f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:13.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.504+0000 7f25fdc00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f25f8150450 0x7f25f80a5fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3301/0 says I am v2:172.21.15.28:41312/0 (socket says 172.21.15.28:41312) 2024-02-21T23:13:13.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.504+0000 7f25fdc00640 1 -- 172.21.15.28:0/1003079868 learned_addr learned my addr 172.21.15.28:0/1003079868 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:13.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.504+0000 7f25fe600640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f25f8156af0 0x7f25f806dff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.28:38348/0 (socket says 172.21.15.28:38348) 2024-02-21T23:13:13.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.504+0000 7f25fdc00640 1 -- 172.21.15.28:0/1003079868 >> v2:172.21.15.81:3300/0 conn(0x7f25f806ee40 msgr2=0x7f25f81536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:13.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.504+0000 7f25fdc00640 1 --2- 172.21.15.28:0/1003079868 >> v2:172.21.15.81:3300/0 conn(0x7f25f806ee40 0x7f25f81536f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:13.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.504+0000 7f25fdc00640 1 -- 172.21.15.28:0/1003079868 >> v1:172.21.15.81:6789/0 conn(0x7f25f8156af0 legacy=0x7f25f806dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:13.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.504+0000 7f25fdc00640 1 -- 172.21.15.28:0/1003079868 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25f8153c30 con 0x7f25f8150450 2024-02-21T23:13:13.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.504+0000 7f25fd200640 1 --2- 172.21.15.28:0/1003079868 >> v2:172.21.15.81:3300/0 conn(0x7f25f806ee40 0x7f25f81536f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-21T23:13:13.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.506+0000 7f25fdc00640 1 --2- 172.21.15.28:0/1003079868 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f25f8150450 0x7f25f80a5fd0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f25ec009870 tx=0x7f25ec02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc846c7ec1cf2846 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:13.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.506+0000 7f25f7e00640 1 -- 172.21.15.28:0/1003079868 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25ec034070 con 0x7f25f8150450 2024-02-21T23:13:13.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.506+0000 7f25f7e00640 1 -- 172.21.15.28:0/1003079868 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25ec030840 con 0x7f25f8150450 2024-02-21T23:13:13.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.506+0000 7f25f7e00640 1 -- 172.21.15.28:0/1003079868 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25ec030c10 con 0x7f25f8150450 2024-02-21T23:13:13.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.507+0000 7f25ff000640 1 -- 172.21.15.28:0/1003079868 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f25f8150450 msgr2=0x7f25f80a5fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:13.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.507+0000 7f25ff000640 1 --2- 172.21.15.28:0/1003079868 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f25f8150450 0x7f25f80a5fd0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f25ec009870 tx=0x7f25ec02b980 comp rx=0 tx=0).stop 2024-02-21T23:13:13.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.507+0000 7f25ff000640 1 -- 172.21.15.28:0/1003079868 shutdown_connections 2024-02-21T23:13:13.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.507+0000 7f25ff000640 1 --2- 172.21.15.28:0/1003079868 >> v2:172.21.15.81:3300/0 conn(0x7f25f806ee40 0x7f25f81536f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:13.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.507+0000 7f25ff000640 1 --2- 172.21.15.28:0/1003079868 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f25f8150450 0x7f25f80a5fd0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:13.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.507+0000 7f25ff000640 1 -- 172.21.15.28:0/1003079868 >> 172.21.15.28:0/1003079868 conn(0x7f25f806b630 msgr2=0x7f25f80c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:13.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.507+0000 7f25ff000640 1 -- 172.21.15.28:0/1003079868 shutdown_connections 2024-02-21T23:13:13.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.507+0000 7f25ff000640 1 -- 172.21.15.28:0/1003079868 wait complete. 2024-02-21T23:13:13.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.509+0000 7f25ff000640 1 Processor -- start 2024-02-21T23:13:13.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.509+0000 7f25ff000640 1 -- start start 2024-02-21T23:13:13.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.510+0000 7f25ff000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25f806ee40 0x7f25f81ee080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:13.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.510+0000 7f25ff000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f25f8150450 0x7f25f80bfba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:13.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.510+0000 7f25ff000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f25f8156af0 0x7f25f8071340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:13.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.510+0000 7f25ff000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25f80c2cb0 con 0x7f25f806ee40 2024-02-21T23:13:13.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.510+0000 7f25ff000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25f80c29d0 con 0x7f25f8156af0 2024-02-21T23:13:13.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.510+0000 7f25ff000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25f806e700 con 0x7f25f8150450 2024-02-21T23:13:13.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.510+0000 7f25fdc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25f806ee40 0x7f25f81ee080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:13.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.510+0000 7f25fdc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25f806ee40 0x7f25f81ee080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55234/0 (socket says 172.21.15.28:55234) 2024-02-21T23:13:13.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.510+0000 7f25fdc00640 1 -- 172.21.15.28:0/674008081 learned_addr learned my addr 172.21.15.28:0/674008081 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:13.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.510+0000 7f25fe600640 1 --2- 172.21.15.28:0/674008081 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f25f8156af0 0x7f25f8071340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:13.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.510+0000 7f25fd200640 1 --2- 172.21.15.28:0/674008081 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f25f8150450 0x7f25f80bfba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:13.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.511+0000 7f25fdc00640 1 -- 172.21.15.28:0/674008081 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f25f8150450 msgr2=0x7f25f80bfba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:13.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.511+0000 7f25fdc00640 1 --2- 172.21.15.28:0/674008081 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f25f8150450 0x7f25f80bfba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:13.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.511+0000 7f25fdc00640 1 -- 172.21.15.28:0/674008081 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f25f8156af0 msgr2=0x7f25f8071340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:13.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.511+0000 7f25fdc00640 1 --2- 172.21.15.28:0/674008081 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f25f8156af0 0x7f25f8071340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:13.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.511+0000 7f25fdc00640 1 -- 172.21.15.28:0/674008081 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25f8072db0 con 0x7f25f806ee40 2024-02-21T23:13:13.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.511+0000 7f25fd200640 1 --2- 172.21.15.28:0/674008081 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f25f8150450 0x7f25f80bfba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:13:13.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.511+0000 7f25fdc00640 1 --2- 172.21.15.28:0/674008081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25f806ee40 0x7f25f81ee080 secure :-1 s=READY pgs=2304 cs=0 l=1 rev1=1 crypto rx=0x7f25ec0097c0 tx=0x7f25ec004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:13.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.511+0000 7f25f5600640 1 -- 172.21.15.28:0/674008081 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25ec048070 con 0x7f25f806ee40 2024-02-21T23:13:13.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.511+0000 7f25f5600640 1 -- 172.21.15.28:0/674008081 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25ec030640 con 0x7f25f806ee40 2024-02-21T23:13:13.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.511+0000 7f25f5600640 1 -- 172.21.15.28:0/674008081 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25ec034040 con 0x7f25f806ee40 2024-02-21T23:13:13.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.511+0000 7f25ff000640 1 -- 172.21.15.28:0/674008081 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25f8073050 con 0x7f25f806ee40 2024-02-21T23:13:13.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.512+0000 7f25ff000640 1 -- 172.21.15.28:0/674008081 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25f8154b20 con 0x7f25f806ee40 2024-02-21T23:13:13.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.513+0000 7f25ff000640 1 -- 172.21.15.28:0/674008081 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25f804f290 con 0x7f25f806ee40 2024-02-21T23:13:13.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.514+0000 7f25f5600640 1 -- 172.21.15.28:0/674008081 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f25ec037070 con 0x7f25f806ee40 2024-02-21T23:13:13.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.514+0000 7f25f5600640 1 --2- 172.21.15.28:0/674008081 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f25d806b4d0 0x7f25d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:13.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.515+0000 7f25f5600640 1 -- 172.21.15.28:0/674008081 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f25ec0abfe0 con 0x7f25f806ee40 2024-02-21T23:13:13.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.515+0000 7f25fd200640 1 --2- 172.21.15.28:0/674008081 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f25d806b4d0 0x7f25d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:13.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.516+0000 7f25fd200640 1 --2- 172.21.15.28:0/674008081 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f25d806b4d0 0x7f25d806d990 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f25f80a7260 tx=0x7f25e4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:13.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.519+0000 7f25f5600640 1 -- 172.21.15.28:0/674008081 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f25ec07ac10 con 0x7f25f806ee40 2024-02-21T23:13:13.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.718+0000 7f25ff000640 1 -- 172.21.15.28:0/674008081 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f25f80c1f10 con 0x7f25f806ee40 2024-02-21T23:13:13.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.721+0000 7f25f5600640 1 -- 172.21.15.28:0/674008081 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v197) v1 ==== 74+0+16892 (secure 0 0 0) 0x7f25ec07a510 con 0x7f25f806ee40 2024-02-21T23:13:13.709 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:13:13.709 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":197,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:12:27.821567+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:12:24.201589+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"196","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_data","create_time":"2024-02-21T23:12:25.295387+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"197","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:13:13.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.724+0000 7f25ff000640 1 -- 172.21.15.28:0/674008081 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f25d806b4d0 msgr2=0x7f25d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:13.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.724+0000 7f25ff000640 1 --2- 172.21.15.28:0/674008081 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f25d806b4d0 0x7f25d806d990 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f25f80a7260 tx=0x7f25e4008040 comp rx=0 tx=0).stop 2024-02-21T23:13:13.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.724+0000 7f25ff000640 1 -- 172.21.15.28:0/674008081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25f806ee40 msgr2=0x7f25f81ee080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:13.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.724+0000 7f25ff000640 1 --2- 172.21.15.28:0/674008081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25f806ee40 0x7f25f81ee080 secure :-1 s=READY pgs=2304 cs=0 l=1 rev1=1 crypto rx=0x7f25ec0097c0 tx=0x7f25ec004730 comp rx=0 tx=0).stop 2024-02-21T23:13:13.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.724+0000 7f25ff000640 1 -- 172.21.15.28:0/674008081 shutdown_connections 2024-02-21T23:13:13.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.724+0000 7f25ff000640 1 --2- 172.21.15.28:0/674008081 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f25d806b4d0 0x7f25d806d990 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:13.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.724+0000 7f25ff000640 1 --2- 172.21.15.28:0/674008081 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f25f8150450 0x7f25f80bfba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:13.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.724+0000 7f25ff000640 1 --2- 172.21.15.28:0/674008081 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f25f8156af0 0x7f25f8071340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:13.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.724+0000 7f25ff000640 1 --2- 172.21.15.28:0/674008081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25f806ee40 0x7f25f81ee080 unknown :-1 s=CLOSED pgs=2304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:13.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.724+0000 7f25ff000640 1 -- 172.21.15.28:0/674008081 >> 172.21.15.28:0/674008081 conn(0x7f25f806b630 msgr2=0x7f25f80a74d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:13.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.724+0000 7f25ff000640 1 -- 172.21.15.28:0/674008081 shutdown_connections 2024-02-21T23:13:13.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.724+0000 7f25ff000640 1 -- 172.21.15.28:0/674008081 wait complete. 2024-02-21T23:13:13.729 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-02-21T23:13:13.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.964+0000 7ff8db200640 1 Processor -- start 2024-02-21T23:13:13.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.965+0000 7ff8db200640 1 -- start start 2024-02-21T23:13:13.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.966+0000 7ff8db200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff8dc148700 0x7ff8dc06d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:13.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.966+0000 7ff8db200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff8dc155260 0x7ff8dc1556d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:13.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.966+0000 7ff8db200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8dc072e90 con 0x7ff8dc0c13e0 2024-02-21T23:13:13.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.966+0000 7ff8db200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8dc0c0ee0 con 0x7ff8dc148700 2024-02-21T23:13:13.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.966+0000 7ff8db200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8dc073100 con 0x7ff8dc155260 2024-02-21T23:13:13.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.967+0000 7ff8d9e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff8dc148700 0x7ff8dc06d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:13.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.967+0000 7ff8d9400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff8dc0c13e0 0x7ff8dc0c1830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60098/0 (socket says 172.21.15.28:60098) 2024-02-21T23:13:13.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.967+0000 7ff8d9e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff8dc148700 0x7ff8dc06d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55244/0 (socket says 172.21.15.28:55244) 2024-02-21T23:13:13.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.967+0000 7ff8d9400640 1 -- 172.21.15.28:0/634292662 learned_addr learned my addr 172.21.15.28:0/634292662 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:13:13.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.967+0000 7ff8da800640 1 --2- 172.21.15.28:0/634292662 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff8dc155260 0x7ff8dc1556d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:13.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.967+0000 7ff8d9e00640 1 -- 172.21.15.28:0/634292662 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff8dc155260 msgr2=0x7ff8dc1556d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:13.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.967+0000 7ff8d9e00640 1 --2- 172.21.15.28:0/634292662 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff8dc155260 0x7ff8dc1556d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:13.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.967+0000 7ff8d9e00640 1 -- 172.21.15.28:0/634292662 >> v1:172.21.15.28:6789/0 conn(0x7ff8dc0c13e0 legacy=0x7ff8dc0c1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:13.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.967+0000 7ff8d9e00640 1 -- 172.21.15.28:0/634292662 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8dc0733e0 con 0x7ff8dc148700 2024-02-21T23:13:13.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.967+0000 7ff8d8a00640 1 -- 172.21.15.28:0/634292662 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1443652147 0 0) 0x7ff8dc072e90 con 0x7ff8dc0c13e0 2024-02-21T23:13:13.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.967+0000 7ff8da800640 1 --2- 172.21.15.28:0/634292662 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff8dc155260 0x7ff8dc1556d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:13:13.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.968+0000 7ff8d9e00640 1 --2- 172.21.15.28:0/634292662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8dc148700 0x7ff8dc06d410 secure :-1 s=READY pgs=2306 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc009d90 tx=0x7ff8cc02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fac2fe7844a55ad server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:13.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.969+0000 7ff8d8a00640 1 -- 172.21.15.28:0/634292662 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8cc039070 con 0x7ff8dc148700 2024-02-21T23:13:13.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.969+0000 7ff8d8a00640 1 -- 172.21.15.28:0/634292662 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8cc02cc60 con 0x7ff8dc148700 2024-02-21T23:13:13.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.969+0000 7ff8d8a00640 1 -- 172.21.15.28:0/634292662 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8cc034660 con 0x7ff8dc148700 2024-02-21T23:13:13.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.969+0000 7ff8db200640 1 -- 172.21.15.28:0/634292662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8dc148700 msgr2=0x7ff8dc06d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:13.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.969+0000 7ff8db200640 1 --2- 172.21.15.28:0/634292662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8dc148700 0x7ff8dc06d410 secure :-1 s=READY pgs=2306 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc009d90 tx=0x7ff8cc02beb0 comp rx=0 tx=0).stop 2024-02-21T23:13:13.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.969+0000 7ff8db200640 1 -- 172.21.15.28:0/634292662 shutdown_connections 2024-02-21T23:13:13.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.969+0000 7ff8db200640 1 --2- 172.21.15.28:0/634292662 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff8dc155260 0x7ff8dc1556d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:13.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.969+0000 7ff8db200640 1 --2- 172.21.15.28:0/634292662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8dc148700 0x7ff8dc06d410 unknown :-1 s=CLOSED pgs=2306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:13.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.969+0000 7ff8db200640 1 -- 172.21.15.28:0/634292662 >> 172.21.15.28:0/634292662 conn(0x7ff8dc0a8320 msgr2=0x7ff8dc0a8730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:13.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.969+0000 7ff8db200640 1 -- 172.21.15.28:0/634292662 shutdown_connections 2024-02-21T23:13:13.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.969+0000 7ff8db200640 1 -- 172.21.15.28:0/634292662 wait complete. 2024-02-21T23:13:13.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.971+0000 7ff8db200640 1 Processor -- start 2024-02-21T23:13:13.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.971+0000 7ff8db200640 1 -- start start 2024-02-21T23:13:13.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.972+0000 7ff8db200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff8dc0c13e0 0x7ff8dc0aa670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:13.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.972+0000 7ff8db200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8dc148700 0x7ff8dc1475a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:13.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.972+0000 7ff8db200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff8dc155260 0x7ff8dc1ebac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:13.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.972+0000 7ff8db200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8dc06d9f0 con 0x7ff8dc148700 2024-02-21T23:13:13.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.972+0000 7ff8db200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8dc0a9080 con 0x7ff8dc0c13e0 2024-02-21T23:13:13.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.972+0000 7ff8db200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8dc0c1f40 con 0x7ff8dc155260 2024-02-21T23:13:13.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.973+0000 7ff8d9400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8dc148700 0x7ff8dc1475a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:13.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.973+0000 7ff8d9400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8dc148700 0x7ff8dc1475a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55254/0 (socket says 172.21.15.28:55254) 2024-02-21T23:13:13.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.973+0000 7ff8d9400640 1 -- 172.21.15.28:0/1509301720 learned_addr learned my addr 172.21.15.28:0/1509301720 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:13.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.973+0000 7ff8d9e00640 1 --2- 172.21.15.28:0/1509301720 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff8dc0c13e0 0x7ff8dc0aa670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:13.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.973+0000 7ff8da800640 1 --2- 172.21.15.28:0/1509301720 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff8dc155260 0x7ff8dc1ebac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:13.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.973+0000 7ff8d9400640 1 -- 172.21.15.28:0/1509301720 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff8dc155260 msgr2=0x7ff8dc1ebac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:13.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.973+0000 7ff8d9400640 1 --2- 172.21.15.28:0/1509301720 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff8dc155260 0x7ff8dc1ebac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:13.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.973+0000 7ff8d9400640 1 -- 172.21.15.28:0/1509301720 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff8dc0c13e0 msgr2=0x7ff8dc0aa670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:13.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.973+0000 7ff8d9400640 1 --2- 172.21.15.28:0/1509301720 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff8dc0c13e0 0x7ff8dc0aa670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:13.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.973+0000 7ff8d9400640 1 -- 172.21.15.28:0/1509301720 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8dc0a7470 con 0x7ff8dc148700 2024-02-21T23:13:13.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.973+0000 7ff8da800640 1 --2- 172.21.15.28:0/1509301720 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff8dc155260 0x7ff8dc1ebac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:13:13.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.973+0000 7ff8d9400640 1 --2- 172.21.15.28:0/1509301720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8dc148700 0x7ff8dc1475a0 secure :-1 s=READY pgs=2307 cs=0 l=1 rev1=1 crypto rx=0x7ff8c400c460 tx=0x7ff8c400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:13.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.974+0000 7ff8d2000640 1 -- 172.21.15.28:0/1509301720 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c4018070 con 0x7ff8dc148700 2024-02-21T23:13:13.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.974+0000 7ff8d2000640 1 -- 172.21.15.28:0/1509301720 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8c4007500 con 0x7ff8dc148700 2024-02-21T23:13:13.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.974+0000 7ff8d2000640 1 -- 172.21.15.28:0/1509301720 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c4004a20 con 0x7ff8dc148700 2024-02-21T23:13:13.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.974+0000 7ff8db200640 1 -- 172.21.15.28:0/1509301720 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8dc1ec390 con 0x7ff8dc148700 2024-02-21T23:13:13.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.974+0000 7ff8db200640 1 -- 172.21.15.28:0/1509301720 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8dc14f620 con 0x7ff8dc148700 2024-02-21T23:13:13.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.975+0000 7ff8db200640 1 -- 172.21.15.28:0/1509301720 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8dc0c73f0 con 0x7ff8dc148700 2024-02-21T23:13:13.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.976+0000 7ff8d2000640 1 -- 172.21.15.28:0/1509301720 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff8c4004bc0 con 0x7ff8dc148700 2024-02-21T23:13:13.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.977+0000 7ff8d2000640 1 --2- 172.21.15.28:0/1509301720 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff8b806b4d0 0x7ff8b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:13.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.977+0000 7ff8d2000640 1 -- 172.21.15.28:0/1509301720 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff8c4089740 con 0x7ff8dc148700 2024-02-21T23:13:13.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.978+0000 7ff8d9e00640 1 --2- 172.21.15.28:0/1509301720 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff8b806b4d0 0x7ff8b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:13.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.979+0000 7ff8d9e00640 1 --2- 172.21.15.28:0/1509301720 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff8b806b4d0 0x7ff8b806d990 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc036000 tx=0x7ff8cc030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:13.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:13.981+0000 7ff8d2000640 1 -- 172.21.15.28:0/1509301720 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ff8dc0c73f0 con 0x7ff8dc148700 2024-02-21T23:13:14.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:14.222+0000 7ff8db200640 1 -- 172.21.15.28:0/1509301720 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7ff8dc04f290 con 0x7ff8dc148700 2024-02-21T23:13:15.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:15.085+0000 7ff8d2000640 1 -- 172.21.15.28:0/1509301720 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v222) v1 ==== 161+0+0 (secure 0 0 0) 0x7ff8c4058390 con 0x7ff8dc148700 2024-02-21T23:13:15.073 INFO:teuthology.orchestra.run.smithi028.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-02-21T23:13:15.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:15.089+0000 7ff8db200640 1 -- 172.21.15.28:0/1509301720 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff8b806b4d0 msgr2=0x7ff8b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:15.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:15.089+0000 7ff8db200640 1 --2- 172.21.15.28:0/1509301720 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff8b806b4d0 0x7ff8b806d990 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc036000 tx=0x7ff8cc030000 comp rx=0 tx=0).stop 2024-02-21T23:13:15.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:15.089+0000 7ff8db200640 1 -- 172.21.15.28:0/1509301720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8dc148700 msgr2=0x7ff8dc1475a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:15.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:15.089+0000 7ff8db200640 1 --2- 172.21.15.28:0/1509301720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8dc148700 0x7ff8dc1475a0 secure :-1 s=READY pgs=2307 cs=0 l=1 rev1=1 crypto rx=0x7ff8c400c460 tx=0x7ff8c400c920 comp rx=0 tx=0).stop 2024-02-21T23:13:15.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:15.089+0000 7ff8db200640 1 -- 172.21.15.28:0/1509301720 shutdown_connections 2024-02-21T23:13:15.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:15.089+0000 7ff8db200640 1 --2- 172.21.15.28:0/1509301720 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff8b806b4d0 0x7ff8b806d990 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:15.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:15.089+0000 7ff8db200640 1 --2- 172.21.15.28:0/1509301720 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff8dc155260 0x7ff8dc1ebac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:15.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:15.089+0000 7ff8db200640 1 --2- 172.21.15.28:0/1509301720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8dc148700 0x7ff8dc1475a0 unknown :-1 s=CLOSED pgs=2307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:15.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:15.089+0000 7ff8db200640 1 --2- 172.21.15.28:0/1509301720 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff8dc0c13e0 0x7ff8dc0aa670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:15.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:15.089+0000 7ff8db200640 1 -- 172.21.15.28:0/1509301720 >> 172.21.15.28:0/1509301720 conn(0x7ff8dc0a8320 msgr2=0x7ff8dc0729c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:15.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:15.089+0000 7ff8db200640 1 -- 172.21.15.28:0/1509301720 shutdown_connections 2024-02-21T23:13:15.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:15.089+0000 7ff8db200640 1 -- 172.21.15.28:0/1509301720 wait complete. 2024-02-21T23:13:15.098 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-02-21T23:13:15.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:15.339+0000 7fa801200640 1 Processor -- start 2024-02-21T23:13:15.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:15.339+0000 7fa801200640 1 -- start start 2024-02-21T23:13:15.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:15.341+0000 7fa801200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa7fc1500f0 0x7fa7fc06ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:15.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:15.341+0000 7fa801200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7fc06db10 con 0x7fa7fc06b3b0 2024-02-21T23:13:15.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:15.341+0000 7fa801200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7fc1539a0 con 0x7fa7fc0c2c60 2024-02-21T23:13:15.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:15.341+0000 7fa801200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7fc0c1080 con 0x7fa7fc1500f0 2024-02-21T23:13:15.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:15.341+0000 7fa7fb400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa7fc1500f0 0x7fa7fc06ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:15.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:15.342+0000 7fa7fbe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa7fc06b3b0 0x7fa7fc15d4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:49990/0 (socket says 172.21.15.28:49990) 2024-02-21T23:13:15.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:15.342+0000 7fa7fbe00640 1 -- 172.21.15.28:0/1252538031 learned_addr learned my addr 172.21.15.28:0/1252538031 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:13:15.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:15.342+0000 7fa7fb400640 1 -- 172.21.15.28:0/1252538031 >> v1:172.21.15.81:6789/0 conn(0x7fa7fc0c2c60 legacy=0x7fa7fc06dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:15.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:15.342+0000 7fa7fb400640 1 -- 172.21.15.28:0/1252538031 >> v1:172.21.15.28:6789/0 conn(0x7fa7fc06b3b0 legacy=0x7fa7fc15d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:15.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:15.342+0000 7fa7fa000640 1 -- 172.21.15.28:0/1252538031 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3654971580 0 0) 0x7fa7fc06db10 con 0x7fa7fc06b3b0 2024-02-21T23:13:15.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:15.342+0000 7fa7fb400640 1 -- 172.21.15.28:0/1252538031 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7fc0c12d0 con 0x7fa7fc1500f0 2024-02-21T23:13:15.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:15.343+0000 7fa7fb400640 1 --2- 172.21.15.28:0/1252538031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7fc1500f0 0x7fa7fc06ae70 secure :-1 s=READY pgs=2309 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc06c010 tx=0x7fa7ec02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f08212ba1537c34 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:15.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:15.344+0000 7fa7fa000640 1 -- 172.21.15.28:0/1252538031 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7ec035020 con 0x7fa7fc1500f0 2024-02-21T23:13:15.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:15.344+0000 7fa7fa000640 1 -- 172.21.15.28:0/1252538031 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7ec031830 con 0x7fa7fc1500f0 2024-02-21T23:13:15.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:15.344+0000 7fa7fa000640 1 -- 172.21.15.28:0/1252538031 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7ec031c00 con 0x7fa7fc1500f0 2024-02-21T23:13:15.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:15.344+0000 7fa801200640 1 -- 172.21.15.28:0/1252538031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7fc1500f0 msgr2=0x7fa7fc06ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:15.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:15.344+0000 7fa801200640 1 --2- 172.21.15.28:0/1252538031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7fc1500f0 0x7fa7fc06ae70 secure :-1 s=READY pgs=2309 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc06c010 tx=0x7fa7ec02b8d0 comp rx=0 tx=0).stop 2024-02-21T23:13:15.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:15.344+0000 7fa801200640 1 -- 172.21.15.28:0/1252538031 shutdown_connections 2024-02-21T23:13:15.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:15.344+0000 7fa801200640 1 --2- 172.21.15.28:0/1252538031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7fc1500f0 0x7fa7fc06ae70 unknown :-1 s=CLOSED pgs=2309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:15.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:15.344+0000 7fa801200640 1 -- 172.21.15.28:0/1252538031 >> 172.21.15.28:0/1252538031 conn(0x7fa7fc0a8530 msgr2=0x7fa7fc0a8910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:15.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:15.344+0000 7fa801200640 1 -- 172.21.15.28:0/1252538031 shutdown_connections 2024-02-21T23:13:15.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:15.345+0000 7fa801200640 1 -- 172.21.15.28:0/1252538031 wait complete. 2024-02-21T23:13:15.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:15.346+0000 7fa801200640 1 Processor -- start 2024-02-21T23:13:15.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:15.346+0000 7fa801200640 1 -- start start 2024-02-21T23:13:15.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:15.347+0000 7fa801200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa7fc06b3b0 0x7fa7fc145d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:15.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:15.347+0000 7fa801200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7fc0c2c60 0x7fa7fc1479b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:15.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:15.347+0000 7fa801200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa7fc1500f0 0x7fa7fc0a9840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:15.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:15.347+0000 7fa801200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7fc1535e0 con 0x7fa7fc0c2c60 2024-02-21T23:13:15.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:15.347+0000 7fa801200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7fc1504d0 con 0x7fa7fc1500f0 2024-02-21T23:13:15.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:15.347+0000 7fa801200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7fc153760 con 0x7fa7fc06b3b0 2024-02-21T23:13:15.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:15.348+0000 7fa7faa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7fc0c2c60 0x7fa7fc1479b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:15.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:15.348+0000 7fa7faa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7fc0c2c60 0x7fa7fc1479b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51428/0 (socket says 172.21.15.28:51428) 2024-02-21T23:13:15.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:15.348+0000 7fa7faa00640 1 -- 172.21.15.28:0/4073374927 learned_addr learned my addr 172.21.15.28:0/4073374927 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:15.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:15.348+0000 7fa7faa00640 1 -- 172.21.15.28:0/4073374927 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa7fc06b3b0 msgr2=0x7fa7fc145d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:15.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:15.348+0000 7fa7faa00640 1 --2- 172.21.15.28:0/4073374927 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa7fc06b3b0 0x7fa7fc145d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:15.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:15.348+0000 7fa7fbe00640 1 --2- 172.21.15.28:0/4073374927 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa7fc1500f0 0x7fa7fc0a9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:15.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:15.348+0000 7fa7faa00640 1 -- 172.21.15.28:0/4073374927 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa7fc1500f0 msgr2=0x7fa7fc0a9840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:15.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:15.348+0000 7fa7faa00640 1 --2- 172.21.15.28:0/4073374927 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa7fc1500f0 0x7fa7fc0a9840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:15.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:15.348+0000 7fa7faa00640 1 -- 172.21.15.28:0/4073374927 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7fc0ac270 con 0x7fa7fc0c2c60 2024-02-21T23:13:15.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:15.348+0000 7fa7faa00640 1 --2- 172.21.15.28:0/4073374927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7fc0c2c60 0x7fa7fc1479b0 secure :-1 s=READY pgs=2310 cs=0 l=1 rev1=1 crypto rx=0x7fa7e400c400 tx=0x7fa7e400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:15.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:15.349+0000 7fa7f3400640 1 -- 172.21.15.28:0/4073374927 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7e4011790 con 0x7fa7fc0c2c60 2024-02-21T23:13:15.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:15.349+0000 7fa7f3400640 1 -- 172.21.15.28:0/4073374927 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7e4017070 con 0x7fa7fc0c2c60 2024-02-21T23:13:15.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:15.349+0000 7fa7f3400640 1 -- 172.21.15.28:0/4073374927 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7e4011bb0 con 0x7fa7fc0c2c60 2024-02-21T23:13:15.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:15.349+0000 7fa801200640 1 -- 172.21.15.28:0/4073374927 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7fc0ac550 con 0x7fa7fc0c2c60 2024-02-21T23:13:15.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:15.349+0000 7fa801200640 1 -- 172.21.15.28:0/4073374927 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7fc156c30 con 0x7fa7fc0c2c60 2024-02-21T23:13:15.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:15.350+0000 7fa801200640 1 -- 172.21.15.28:0/4073374927 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7fc04f290 con 0x7fa7fc0c2c60 2024-02-21T23:13:15.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:15.351+0000 7fa7f3400640 1 -- 172.21.15.28:0/4073374927 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa7e401c460 con 0x7fa7fc0c2c60 2024-02-21T23:13:15.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:15.352+0000 7fa7f3400640 1 --2- 172.21.15.28:0/4073374927 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa7d806b4d0 0x7fa7d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:15.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:15.352+0000 7fa7f3400640 1 -- 172.21.15.28:0/4073374927 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa7e408a1f0 con 0x7fa7fc0c2c60 2024-02-21T23:13:15.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:15.353+0000 7fa7fb400640 1 --2- 172.21.15.28:0/4073374927 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa7d806b4d0 0x7fa7d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:15.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:15.354+0000 7fa7fb400640 1 --2- 172.21.15.28:0/4073374927 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa7d806b4d0 0x7fa7d806d990 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fa7ec036000 tx=0x7fa7ec030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:15.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:15.356+0000 7fa7f3400640 1 -- 172.21.15.28:0/4073374927 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fa7e4058ed0 con 0x7fa7fc0c2c60 2024-02-21T23:13:15.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:15.597+0000 7fa801200640 1 -- 172.21.15.28:0/4073374927 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fa7fc06f1e0 con 0x7fa7fc0c2c60 2024-02-21T23:13:16.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.097+0000 7fa7f3400640 1 -- 172.21.15.28:0/4073374927 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v223) v1 ==== 104+0+0 (secure 0 0 0) 0x7fa7e40587d0 con 0x7fa7fc0c2c60 2024-02-21T23:13:16.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.101+0000 7fa801200640 1 -- 172.21.15.28:0/4073374927 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa7d806b4d0 msgr2=0x7fa7d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:16.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.101+0000 7fa801200640 1 --2- 172.21.15.28:0/4073374927 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa7d806b4d0 0x7fa7d806d990 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fa7ec036000 tx=0x7fa7ec030000 comp rx=0 tx=0).stop 2024-02-21T23:13:16.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.101+0000 7fa801200640 1 -- 172.21.15.28:0/4073374927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7fc0c2c60 msgr2=0x7fa7fc1479b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:16.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.101+0000 7fa801200640 1 --2- 172.21.15.28:0/4073374927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7fc0c2c60 0x7fa7fc1479b0 secure :-1 s=READY pgs=2310 cs=0 l=1 rev1=1 crypto rx=0x7fa7e400c400 tx=0x7fa7e400c8c0 comp rx=0 tx=0).stop 2024-02-21T23:13:16.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.101+0000 7fa801200640 1 -- 172.21.15.28:0/4073374927 shutdown_connections 2024-02-21T23:13:16.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.101+0000 7fa801200640 1 --2- 172.21.15.28:0/4073374927 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa7d806b4d0 0x7fa7d806d990 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:16.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.101+0000 7fa801200640 1 --2- 172.21.15.28:0/4073374927 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa7fc1500f0 0x7fa7fc0a9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:16.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.101+0000 7fa801200640 1 --2- 172.21.15.28:0/4073374927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7fc0c2c60 0x7fa7fc1479b0 unknown :-1 s=CLOSED pgs=2310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:16.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.101+0000 7fa801200640 1 --2- 172.21.15.28:0/4073374927 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa7fc06b3b0 0x7fa7fc145d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:16.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.101+0000 7fa801200640 1 -- 172.21.15.28:0/4073374927 >> 172.21.15.28:0/4073374927 conn(0x7fa7fc0a8530 msgr2=0x7fa7fc14a5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:16.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.102+0000 7fa801200640 1 -- 172.21.15.28:0/4073374927 shutdown_connections 2024-02-21T23:13:16.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.102+0000 7fa801200640 1 -- 172.21.15.28:0/4073374927 wait complete. 2024-02-21T23:13:16.110 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-02-21T23:13:16.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.349+0000 7fb9e1600640 1 Processor -- start 2024-02-21T23:13:16.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.350+0000 7fb9e1600640 1 -- start start 2024-02-21T23:13:16.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.352+0000 7fb9e1600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb9dc0c1010 0x7fb9dc06e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:16.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.352+0000 7fb9e1600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb9dc06e870 0x7fb9dc15d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:16.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.352+0000 7fb9e1600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9dc152fd0 con 0x7fb9dc1501c0 2024-02-21T23:13:16.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.352+0000 7fb9e1600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9dc1532b0 con 0x7fb9dc0c1010 2024-02-21T23:13:16.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.352+0000 7fb9e1600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9dc153590 con 0x7fb9dc06e870 2024-02-21T23:13:16.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.352+0000 7fb9db400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb9dc0c1010 0x7fb9dc06e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:16.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.352+0000 7fb9dbe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb9dc1501c0 0x7fb9dc0a96c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50006/0 (socket says 172.21.15.28:50006) 2024-02-21T23:13:16.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.352+0000 7fb9dbe00640 1 -- 172.21.15.28:0/1186918640 learned_addr learned my addr 172.21.15.28:0/1186918640 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:13:16.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.352+0000 7fb9e0c00640 1 --2- 172.21.15.28:0/1186918640 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb9dc06e870 0x7fb9dc15d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:16.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.353+0000 7fb9db400640 1 -- 172.21.15.28:0/1186918640 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb9dc06e870 msgr2=0x7fb9dc15d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:16.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.353+0000 7fb9db400640 1 --2- 172.21.15.28:0/1186918640 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb9dc06e870 0x7fb9dc15d5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:16.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.353+0000 7fb9db400640 1 -- 172.21.15.28:0/1186918640 >> v1:172.21.15.28:6789/0 conn(0x7fb9dc1501c0 legacy=0x7fb9dc0a96c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:16.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.353+0000 7fb9db400640 1 -- 172.21.15.28:0/1186918640 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9dc153870 con 0x7fb9dc0c1010 2024-02-21T23:13:16.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.353+0000 7fb9daa00640 1 -- 172.21.15.28:0/1186918640 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 910100864 0 0) 0x7fb9dc152fd0 con 0x7fb9dc1501c0 2024-02-21T23:13:16.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.354+0000 7fb9db400640 1 --2- 172.21.15.28:0/1186918640 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9dc0c1010 0x7fb9dc06e330 secure :-1 s=READY pgs=2312 cs=0 l=1 rev1=1 crypto rx=0x7fb9c40097a0 tx=0x7fb9c402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8408ed4abc040a03 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:16.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.354+0000 7fb9daa00640 1 -- 172.21.15.28:0/1186918640 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9c40315a0 con 0x7fb9dc0c1010 2024-02-21T23:13:16.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.354+0000 7fb9daa00640 1 -- 172.21.15.28:0/1186918640 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9c4037070 con 0x7fb9dc0c1010 2024-02-21T23:13:16.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.354+0000 7fb9daa00640 1 -- 172.21.15.28:0/1186918640 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9c4031960 con 0x7fb9dc0c1010 2024-02-21T23:13:16.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.355+0000 7fb9e1600640 1 -- 172.21.15.28:0/1186918640 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9dc0c1010 msgr2=0x7fb9dc06e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:16.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.355+0000 7fb9e1600640 1 --2- 172.21.15.28:0/1186918640 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9dc0c1010 0x7fb9dc06e330 secure :-1 s=READY pgs=2312 cs=0 l=1 rev1=1 crypto rx=0x7fb9c40097a0 tx=0x7fb9c402b840 comp rx=0 tx=0).stop 2024-02-21T23:13:16.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.355+0000 7fb9e1600640 1 -- 172.21.15.28:0/1186918640 shutdown_connections 2024-02-21T23:13:16.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.355+0000 7fb9e1600640 1 --2- 172.21.15.28:0/1186918640 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb9dc06e870 0x7fb9dc15d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:16.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.355+0000 7fb9e1600640 1 --2- 172.21.15.28:0/1186918640 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9dc0c1010 0x7fb9dc06e330 unknown :-1 s=CLOSED pgs=2312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:16.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.355+0000 7fb9e1600640 1 -- 172.21.15.28:0/1186918640 >> 172.21.15.28:0/1186918640 conn(0x7fb9dc0c6610 msgr2=0x7fb9dc0c69f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:16.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.355+0000 7fb9e1600640 1 -- 172.21.15.28:0/1186918640 shutdown_connections 2024-02-21T23:13:16.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.355+0000 7fb9e1600640 1 -- 172.21.15.28:0/1186918640 wait complete. 2024-02-21T23:13:16.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.357+0000 7fb9e1600640 1 Processor -- start 2024-02-21T23:13:16.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.357+0000 7fb9e1600640 1 -- start start 2024-02-21T23:13:16.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.358+0000 7fb9e1600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9dc06e870 0x7fb9dc0c54c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:16.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.358+0000 7fb9e1600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9dc0c1010 0x7fb9dc14a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:16.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.358+0000 7fb9e1600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb9dc1501c0 0x7fb9dc14a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:16.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.358+0000 7fb9e1600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9dc0c7340 con 0x7fb9dc0c1010 2024-02-21T23:13:16.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.358+0000 7fb9e1600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9dc0c0c20 con 0x7fb9dc06e870 2024-02-21T23:13:16.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.358+0000 7fb9e1600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9dc1505a0 con 0x7fb9dc1501c0 2024-02-21T23:13:16.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.358+0000 7fb9db400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9dc0c1010 0x7fb9dc14a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:16.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.358+0000 7fb9db400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9dc0c1010 0x7fb9dc14a460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51444/0 (socket says 172.21.15.28:51444) 2024-02-21T23:13:16.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.358+0000 7fb9db400640 1 -- 172.21.15.28:0/3904159938 learned_addr learned my addr 172.21.15.28:0/3904159938 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:16.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.359+0000 7fb9dbe00640 1 --2- 172.21.15.28:0/3904159938 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9dc06e870 0x7fb9dc0c54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:16.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.359+0000 7fb9e0c00640 1 --2- 172.21.15.28:0/3904159938 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb9dc1501c0 0x7fb9dc14a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:16.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.359+0000 7fb9db400640 1 -- 172.21.15.28:0/3904159938 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb9dc1501c0 msgr2=0x7fb9dc14a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:16.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.359+0000 7fb9db400640 1 --2- 172.21.15.28:0/3904159938 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb9dc1501c0 0x7fb9dc14a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:16.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.359+0000 7fb9db400640 1 -- 172.21.15.28:0/3904159938 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9dc06e870 msgr2=0x7fb9dc0c54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:16.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.359+0000 7fb9db400640 1 --2- 172.21.15.28:0/3904159938 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9dc06e870 0x7fb9dc0c54c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:16.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.359+0000 7fb9db400640 1 -- 172.21.15.28:0/3904159938 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9dc1ee840 con 0x7fb9dc0c1010 2024-02-21T23:13:16.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.359+0000 7fb9e0c00640 1 --2- 172.21.15.28:0/3904159938 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb9dc1501c0 0x7fb9dc14a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:13:16.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.359+0000 7fb9db400640 1 --2- 172.21.15.28:0/3904159938 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9dc0c1010 0x7fb9dc14a460 secure :-1 s=READY pgs=2313 cs=0 l=1 rev1=1 crypto rx=0x7fb9c40097a0 tx=0x7fb9c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:16.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.359+0000 7fb9d3e00640 1 -- 172.21.15.28:0/3904159938 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9c4004230 con 0x7fb9dc0c1010 2024-02-21T23:13:16.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.360+0000 7fb9d3e00640 1 -- 172.21.15.28:0/3904159938 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9c40043d0 con 0x7fb9dc0c1010 2024-02-21T23:13:16.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.360+0000 7fb9d3e00640 1 -- 172.21.15.28:0/3904159938 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9c4003850 con 0x7fb9dc0c1010 2024-02-21T23:13:16.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.360+0000 7fb9e1600640 1 -- 172.21.15.28:0/3904159938 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9dc1527b0 con 0x7fb9dc0c1010 2024-02-21T23:13:16.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.360+0000 7fb9e1600640 1 -- 172.21.15.28:0/3904159938 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9dc151bb0 con 0x7fb9dc0c1010 2024-02-21T23:13:16.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.361+0000 7fb9e1600640 1 -- 172.21.15.28:0/3904159938 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9dc1519a0 con 0x7fb9dc0c1010 2024-02-21T23:13:16.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.362+0000 7fb9d3e00640 1 -- 172.21.15.28:0/3904159938 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb9c40039f0 con 0x7fb9dc0c1010 2024-02-21T23:13:16.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.363+0000 7fb9d3e00640 1 --2- 172.21.15.28:0/3904159938 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb9b806b500 0x7fb9b806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:16.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.363+0000 7fb9d3e00640 1 -- 172.21.15.28:0/3904159938 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb9c402f030 con 0x7fb9dc0c1010 2024-02-21T23:13:16.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.363+0000 7fb9dbe00640 1 --2- 172.21.15.28:0/3904159938 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb9b806b500 0x7fb9b806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:16.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.365+0000 7fb9dbe00640 1 --2- 172.21.15.28:0/3904159938 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb9b806b500 0x7fb9b806d9c0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc000f60 tx=0x7fb9cc007b70 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:16.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.368+0000 7fb9d3e00640 1 -- 172.21.15.28:0/3904159938 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb9c4041070 con 0x7fb9dc0c1010 2024-02-21T23:13:16.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:16.568+0000 7fb9e1600640 1 -- 172.21.15.28:0/3904159938 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb9dc04f290 con 0x7fb9dc0c1010 2024-02-21T23:13:17.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.108+0000 7fb9d3e00640 1 -- 172.21.15.28:0/3904159938 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v198) v1 ==== 181+0+0 (secure 0 0 0) 0x7fb9c4039310 con 0x7fb9dc0c1010 2024-02-21T23:13:17.097 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' removed 2024-02-21T23:13:17.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.113+0000 7fb9e1600640 1 -- 172.21.15.28:0/3904159938 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb9b806b500 msgr2=0x7fb9b806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:17.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.113+0000 7fb9e1600640 1 --2- 172.21.15.28:0/3904159938 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb9b806b500 0x7fb9b806d9c0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc000f60 tx=0x7fb9cc007b70 comp rx=0 tx=0).stop 2024-02-21T23:13:17.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.113+0000 7fb9e1600640 1 -- 172.21.15.28:0/3904159938 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9dc0c1010 msgr2=0x7fb9dc14a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:17.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.113+0000 7fb9e1600640 1 --2- 172.21.15.28:0/3904159938 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9dc0c1010 0x7fb9dc14a460 secure :-1 s=READY pgs=2313 cs=0 l=1 rev1=1 crypto rx=0x7fb9c40097a0 tx=0x7fb9c40046c0 comp rx=0 tx=0).stop 2024-02-21T23:13:17.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.114+0000 7fb9e1600640 1 -- 172.21.15.28:0/3904159938 shutdown_connections 2024-02-21T23:13:17.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.114+0000 7fb9e1600640 1 --2- 172.21.15.28:0/3904159938 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb9b806b500 0x7fb9b806d9c0 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:17.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.114+0000 7fb9e1600640 1 --2- 172.21.15.28:0/3904159938 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb9dc1501c0 0x7fb9dc14a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:17.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.114+0000 7fb9e1600640 1 --2- 172.21.15.28:0/3904159938 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9dc0c1010 0x7fb9dc14a460 unknown :-1 s=CLOSED pgs=2313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:17.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.114+0000 7fb9e1600640 1 --2- 172.21.15.28:0/3904159938 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb9dc06e870 0x7fb9dc0c54c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:17.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.114+0000 7fb9e1600640 1 -- 172.21.15.28:0/3904159938 >> 172.21.15.28:0/3904159938 conn(0x7fb9dc0c6610 msgr2=0x7fb9dc0ab5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:17.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.115+0000 7fb9e1600640 1 -- 172.21.15.28:0/3904159938 shutdown_connections 2024-02-21T23:13:17.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.116+0000 7fb9e1600640 1 -- 172.21.15.28:0/3904159938 wait complete. 2024-02-21T23:13:17.130 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-02-21T23:13:17.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.356+0000 7ff436600640 1 Processor -- start 2024-02-21T23:13:17.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.356+0000 7ff436600640 1 -- start start 2024-02-21T23:13:17.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.358+0000 7ff436600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff43006e780 0x7ff43006eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:17.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.358+0000 7ff436600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff4300a5fc0 0x7ff4300c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:17.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.358+0000 7ff436600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7ff4300c50c0 0x7ff4301564d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:17.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.358+0000 7ff436600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4301543d0 con 0x7ff43006e780 2024-02-21T23:13:17.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.358+0000 7ff436600640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4300c5ad0 con 0x7ff4300c50c0 2024-02-21T23:13:17.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.358+0000 7ff436600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4300c46f0 con 0x7ff4300a5fc0 2024-02-21T23:13:17.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.358+0000 7ff42f400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff43006e780 0x7ff43006eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:17.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.358+0000 7ff42f400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff43006e780 0x7ff43006eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51456/0 (socket says 172.21.15.28:51456) 2024-02-21T23:13:17.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.358+0000 7ff42f400640 1 -- 172.21.15.28:0/708211526 learned_addr learned my addr 172.21.15.28:0/708211526 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:17.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.359+0000 7ff426a00640 1 --2- 172.21.15.28:0/708211526 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff4300a5fc0 0x7ff4300c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:17.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.359+0000 7ff42f400640 1 -- 172.21.15.28:0/708211526 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff4300a5fc0 msgr2=0x7ff4300c4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:17.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.359+0000 7ff42f400640 1 --2- 172.21.15.28:0/708211526 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff4300a5fc0 0x7ff4300c4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:17.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.359+0000 7ff42f400640 1 -- 172.21.15.28:0/708211526 >> v2:172.21.15.81:3300/0 conn(0x7ff4300c50c0 msgr2=0x7ff4301564d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:17.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.359+0000 7ff42f400640 1 --2- 172.21.15.28:0/708211526 >> v2:172.21.15.81:3300/0 conn(0x7ff4300c50c0 0x7ff4301564d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:17.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.359+0000 7ff42f400640 1 -- 172.21.15.28:0/708211526 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff430153cb0 con 0x7ff43006e780 2024-02-21T23:13:17.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.359+0000 7ff426a00640 1 --2- 172.21.15.28:0/708211526 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff4300a5fc0 0x7ff4300c4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:13:17.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.360+0000 7ff42f400640 1 --2- 172.21.15.28:0/708211526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff43006e780 0x7ff43006eb60 secure :-1 s=READY pgs=2314 cs=0 l=1 rev1=1 crypto rx=0x7ff4300c74d0 tx=0x7ff42002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e42fd74c71041dd0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:17.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.361+0000 7ff42ea00640 1 -- 172.21.15.28:0/708211526 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff42002e070 con 0x7ff43006e780 2024-02-21T23:13:17.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.361+0000 7ff42ea00640 1 -- 172.21.15.28:0/708211526 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff420037070 con 0x7ff43006e780 2024-02-21T23:13:17.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.361+0000 7ff42ea00640 1 -- 172.21.15.28:0/708211526 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff420031a00 con 0x7ff43006e780 2024-02-21T23:13:17.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.361+0000 7ff436600640 1 -- 172.21.15.28:0/708211526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff43006e780 msgr2=0x7ff43006eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:17.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.361+0000 7ff436600640 1 --2- 172.21.15.28:0/708211526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff43006e780 0x7ff43006eb60 secure :-1 s=READY pgs=2314 cs=0 l=1 rev1=1 crypto rx=0x7ff4300c74d0 tx=0x7ff42002b880 comp rx=0 tx=0).stop 2024-02-21T23:13:17.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.361+0000 7ff436600640 1 -- 172.21.15.28:0/708211526 shutdown_connections 2024-02-21T23:13:17.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.361+0000 7ff436600640 1 --2- 172.21.15.28:0/708211526 >> v2:172.21.15.81:3300/0 conn(0x7ff4300c50c0 0x7ff4301564d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:17.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.361+0000 7ff436600640 1 --2- 172.21.15.28:0/708211526 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff4300a5fc0 0x7ff4300c4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:17.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.361+0000 7ff436600640 1 --2- 172.21.15.28:0/708211526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff43006e780 0x7ff43006eb60 unknown :-1 s=CLOSED pgs=2314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:17.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.361+0000 7ff436600640 1 -- 172.21.15.28:0/708211526 >> 172.21.15.28:0/708211526 conn(0x7ff43006b830 msgr2=0x7ff4300c67f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:17.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.362+0000 7ff436600640 1 -- 172.21.15.28:0/708211526 shutdown_connections 2024-02-21T23:13:17.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.362+0000 7ff436600640 1 -- 172.21.15.28:0/708211526 wait complete. 2024-02-21T23:13:17.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.363+0000 7ff436600640 1 Processor -- start 2024-02-21T23:13:17.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.363+0000 7ff436600640 1 -- start start 2024-02-21T23:13:17.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.364+0000 7ff436600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff43006e780 0x7ff4301ec160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:17.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.364+0000 7ff436600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff4300a5fc0 0x7ff43014b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:17.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.364+0000 7ff436600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff4300c50c0 0x7ff43014b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:17.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.364+0000 7ff436600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff430153960 con 0x7ff43006e780 2024-02-21T23:13:17.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.364+0000 7ff436600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4300c6000 con 0x7ff4300c50c0 2024-02-21T23:13:17.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.364+0000 7ff436600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff43006d110 con 0x7ff4300a5fc0 2024-02-21T23:13:17.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.364+0000 7ff42f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff43006e780 0x7ff4301ec160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:17.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.365+0000 7ff42f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff43006e780 0x7ff4301ec160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51468/0 (socket says 172.21.15.28:51468) 2024-02-21T23:13:17.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.365+0000 7ff42f400640 1 -- 172.21.15.28:0/40050969 learned_addr learned my addr 172.21.15.28:0/40050969 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:17.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.365+0000 7ff42fe00640 1 --2- 172.21.15.28:0/40050969 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff4300c50c0 0x7ff43014b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:17.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.365+0000 7ff42f400640 1 -- 172.21.15.28:0/40050969 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff4300a5fc0 msgr2=0x7ff43014b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:17.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.365+0000 7ff42f400640 1 --2- 172.21.15.28:0/40050969 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff4300a5fc0 0x7ff43014b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:17.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.365+0000 7ff42f400640 1 -- 172.21.15.28:0/40050969 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff4300c50c0 msgr2=0x7ff43014b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:17.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.365+0000 7ff42f400640 1 --2- 172.21.15.28:0/40050969 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff4300c50c0 0x7ff43014b710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:17.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.365+0000 7ff42f400640 1 -- 172.21.15.28:0/40050969 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4301ecc70 con 0x7ff43006e780 2024-02-21T23:13:17.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.365+0000 7ff42f400640 1 --2- 172.21.15.28:0/40050969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff43006e780 0x7ff4301ec160 secure :-1 s=READY pgs=2315 cs=0 l=1 rev1=1 crypto rx=0x7ff42002b8c0 tx=0x7ff4200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:17.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.366+0000 7ff427e00640 1 -- 172.21.15.28:0/40050969 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff420032880 con 0x7ff43006e780 2024-02-21T23:13:17.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.366+0000 7ff427e00640 1 -- 172.21.15.28:0/40050969 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff420032a20 con 0x7ff43006e780 2024-02-21T23:13:17.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.366+0000 7ff436600640 1 -- 172.21.15.28:0/40050969 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4301dfe60 con 0x7ff43006e780 2024-02-21T23:13:17.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.366+0000 7ff427e00640 1 -- 172.21.15.28:0/40050969 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff420032e10 con 0x7ff43006e780 2024-02-21T23:13:17.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.366+0000 7ff436600640 1 -- 172.21.15.28:0/40050969 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4300c33a0 con 0x7ff43006e780 2024-02-21T23:13:17.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.367+0000 7ff436600640 1 -- 172.21.15.28:0/40050969 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4300c58b0 con 0x7ff43006e780 2024-02-21T23:13:17.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.368+0000 7ff427e00640 1 -- 172.21.15.28:0/40050969 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff42004c060 con 0x7ff43006e780 2024-02-21T23:13:17.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.369+0000 7ff427e00640 1 --2- 172.21.15.28:0/40050969 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff40c06b550 0x7ff40c06da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:17.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.369+0000 7ff427e00640 1 -- 172.21.15.28:0/40050969 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 5663+0+0 (secure 0 0 0) 0x7ff4200abb70 con 0x7ff43006e780 2024-02-21T23:13:17.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.369+0000 7ff426a00640 1 --2- 172.21.15.28:0/40050969 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff40c06b550 0x7ff40c06da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:17.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.371+0000 7ff426a00640 1 --2- 172.21.15.28:0/40050969 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff40c06b550 0x7ff40c06da10 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7ff4301dedf0 tx=0x7ff418008420 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:17.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.373+0000 7ff427e00640 1 -- 172.21.15.28:0/40050969 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ff420040070 con 0x7ff43006e780 2024-02-21T23:13:17.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:17.572+0000 7ff436600640 1 -- 172.21.15.28:0/40050969 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ff4301508b0 con 0x7ff43006e780 2024-02-21T23:13:18.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.120+0000 7ff427e00640 1 -- 172.21.15.28:0/40050969 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v199) v1 ==== 169+0+0 (secure 0 0 0) 0x7ff42007acb0 con 0x7ff43006e780 2024-02-21T23:13:18.108 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' removed 2024-02-21T23:13:18.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.125+0000 7ff436600640 1 -- 172.21.15.28:0/40050969 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff40c06b550 msgr2=0x7ff40c06da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:18.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.125+0000 7ff436600640 1 --2- 172.21.15.28:0/40050969 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff40c06b550 0x7ff40c06da10 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7ff4301dedf0 tx=0x7ff418008420 comp rx=0 tx=0).stop 2024-02-21T23:13:18.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.125+0000 7ff436600640 1 -- 172.21.15.28:0/40050969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff43006e780 msgr2=0x7ff4301ec160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:18.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.125+0000 7ff436600640 1 --2- 172.21.15.28:0/40050969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff43006e780 0x7ff4301ec160 secure :-1 s=READY pgs=2315 cs=0 l=1 rev1=1 crypto rx=0x7ff42002b8c0 tx=0x7ff4200046c0 comp rx=0 tx=0).stop 2024-02-21T23:13:18.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.126+0000 7ff436600640 1 -- 172.21.15.28:0/40050969 shutdown_connections 2024-02-21T23:13:18.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.126+0000 7ff436600640 1 --2- 172.21.15.28:0/40050969 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff40c06b550 0x7ff40c06da10 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:18.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.126+0000 7ff436600640 1 --2- 172.21.15.28:0/40050969 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff4300a5fc0 0x7ff43014b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:18.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.126+0000 7ff436600640 1 --2- 172.21.15.28:0/40050969 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff4300c50c0 0x7ff43014b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:18.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.126+0000 7ff436600640 1 --2- 172.21.15.28:0/40050969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff43006e780 0x7ff4301ec160 unknown :-1 s=CLOSED pgs=2315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:18.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.126+0000 7ff436600640 1 -- 172.21.15.28:0/40050969 >> 172.21.15.28:0/40050969 conn(0x7ff43006b830 msgr2=0x7ff4300c67f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:18.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.131+0000 7ff436600640 1 -- 172.21.15.28:0/40050969 shutdown_connections 2024-02-21T23:13:18.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.131+0000 7ff436600640 1 -- 172.21.15.28:0/40050969 wait complete. 2024-02-21T23:13:18.144 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-02-21T23:13:18.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.366+0000 7f1516200640 1 Processor -- start 2024-02-21T23:13:18.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.367+0000 7f1516200640 1 -- start start 2024-02-21T23:13:18.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.368+0000 7f1516200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f15101569d0 0x7f151006e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:18.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.368+0000 7f1516200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f151006d190 0x7f15100c5d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:18.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.368+0000 7f1516200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f15100c62d0 0x7f1510159470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:18.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.368+0000 7f1516200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f151006f370 con 0x7f151006d190 2024-02-21T23:13:18.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.368+0000 7f1516200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f151015ca90 con 0x7f15100c62d0 2024-02-21T23:13:18.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.368+0000 7f1516200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f151015cd70 con 0x7f15101569d0 2024-02-21T23:13:18.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.368+0000 7f1507e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f151006d190 0x7f15100c5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:18.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.368+0000 7f1507e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f151006d190 0x7f15100c5d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51474/0 (socket says 172.21.15.28:51474) 2024-02-21T23:13:18.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.368+0000 7f1507e00640 1 -- 172.21.15.28:0/3019796991 learned_addr learned my addr 172.21.15.28:0/3019796991 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:18.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.369+0000 7f150fe00640 1 --2- 172.21.15.28:0/3019796991 >> v2:172.21.15.81:3300/0 conn(0x7f15100c62d0 0x7f1510159470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:18.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.369+0000 7f150f400640 1 --2- 172.21.15.28:0/3019796991 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f15101569d0 0x7f151006e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:18.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.369+0000 7f1507e00640 1 -- 172.21.15.28:0/3019796991 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f15101569d0 msgr2=0x7f151006e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:18.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.369+0000 7f1507e00640 1 --2- 172.21.15.28:0/3019796991 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f15101569d0 0x7f151006e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:18.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.369+0000 7f1507e00640 1 -- 172.21.15.28:0/3019796991 >> v2:172.21.15.81:3300/0 conn(0x7f15100c62d0 msgr2=0x7f1510159470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:18.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.369+0000 7f1507e00640 1 --2- 172.21.15.28:0/3019796991 >> v2:172.21.15.81:3300/0 conn(0x7f15100c62d0 0x7f1510159470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:18.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.369+0000 7f1507e00640 1 -- 172.21.15.28:0/3019796991 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f151015d050 con 0x7f151006d190 2024-02-21T23:13:18.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.369+0000 7f150fe00640 1 --2- 172.21.15.28:0/3019796991 >> v2:172.21.15.81:3300/0 conn(0x7f15100c62d0 0x7f1510159470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:13:18.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.370+0000 7f1507e00640 1 --2- 172.21.15.28:0/3019796991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f151006d190 0x7f15100c5d90 secure :-1 s=READY pgs=2316 cs=0 l=1 rev1=1 crypto rx=0x7f15100a64c0 tx=0x7f14f802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b12a7ae9732fc6ea server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:18.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.370+0000 7f150ea00640 1 -- 172.21.15.28:0/3019796991 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f8034070 con 0x7f151006d190 2024-02-21T23:13:18.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.370+0000 7f150ea00640 1 -- 172.21.15.28:0/3019796991 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14f8030860 con 0x7f151006d190 2024-02-21T23:13:18.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.371+0000 7f150ea00640 1 -- 172.21.15.28:0/3019796991 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f8030c30 con 0x7f151006d190 2024-02-21T23:13:18.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.371+0000 7f1516200640 1 -- 172.21.15.28:0/3019796991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f151006d190 msgr2=0x7f15100c5d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:18.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.371+0000 7f1516200640 1 --2- 172.21.15.28:0/3019796991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f151006d190 0x7f15100c5d90 secure :-1 s=READY pgs=2316 cs=0 l=1 rev1=1 crypto rx=0x7f15100a64c0 tx=0x7f14f802b9a0 comp rx=0 tx=0).stop 2024-02-21T23:13:18.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.371+0000 7f1516200640 1 -- 172.21.15.28:0/3019796991 shutdown_connections 2024-02-21T23:13:18.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.371+0000 7f1516200640 1 --2- 172.21.15.28:0/3019796991 >> v2:172.21.15.81:3300/0 conn(0x7f15100c62d0 0x7f1510159470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:18.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.371+0000 7f1516200640 1 --2- 172.21.15.28:0/3019796991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f151006d190 0x7f15100c5d90 unknown :-1 s=CLOSED pgs=2316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:18.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.371+0000 7f1516200640 1 --2- 172.21.15.28:0/3019796991 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f15101569d0 0x7f151006e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:18.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.371+0000 7f1516200640 1 -- 172.21.15.28:0/3019796991 >> 172.21.15.28:0/3019796991 conn(0x7f15100a85a0 msgr2=0x7f15100a8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:18.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.371+0000 7f1516200640 1 -- 172.21.15.28:0/3019796991 shutdown_connections 2024-02-21T23:13:18.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.371+0000 7f1516200640 1 -- 172.21.15.28:0/3019796991 wait complete. 2024-02-21T23:13:18.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.373+0000 7f1516200640 1 Processor -- start 2024-02-21T23:13:18.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.373+0000 7f1516200640 1 -- start start 2024-02-21T23:13:18.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.374+0000 7f1516200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f151006d190 0x7f15101f2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:18.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.374+0000 7f1516200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f15100c62d0 0x7f151014e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:18.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.374+0000 7f1516200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15101569d0 0x7f151014b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:18.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.374+0000 7f1516200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f151015d850 con 0x7f15101569d0 2024-02-21T23:13:18.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.374+0000 7f1516200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1510156db0 con 0x7f15100c62d0 2024-02-21T23:13:18.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.374+0000 7f1516200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f151006ef20 con 0x7f151006d190 2024-02-21T23:13:18.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.374+0000 7f150fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15101569d0 0x7f151014b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:18.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.374+0000 7f150fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15101569d0 0x7f151014b960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51484/0 (socket says 172.21.15.28:51484) 2024-02-21T23:13:18.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.374+0000 7f150fe00640 1 -- 172.21.15.28:0/2750675282 learned_addr learned my addr 172.21.15.28:0/2750675282 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:18.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.375+0000 7f150f400640 1 --2- 172.21.15.28:0/2750675282 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f151006d190 0x7f15101f2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:18.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.375+0000 7f1507e00640 1 --2- 172.21.15.28:0/2750675282 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f15100c62d0 0x7f151014e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:18.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.375+0000 7f150fe00640 1 -- 172.21.15.28:0/2750675282 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f151006d190 msgr2=0x7f15101f2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:18.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.375+0000 7f150fe00640 1 --2- 172.21.15.28:0/2750675282 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f151006d190 0x7f15101f2fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:18.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.375+0000 7f150fe00640 1 -- 172.21.15.28:0/2750675282 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f15100c62d0 msgr2=0x7f151014e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:18.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.375+0000 7f150fe00640 1 --2- 172.21.15.28:0/2750675282 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f15100c62d0 0x7f151014e1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:18.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.375+0000 7f150fe00640 1 -- 172.21.15.28:0/2750675282 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15101f3520 con 0x7f15101569d0 2024-02-21T23:13:18.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.375+0000 7f150fe00640 1 --2- 172.21.15.28:0/2750675282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15101569d0 0x7f151014b960 secure :-1 s=READY pgs=2317 cs=0 l=1 rev1=1 crypto rx=0x7f150800cf20 tx=0x7f150800a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:18.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.375+0000 7f1507400640 1 -- 172.21.15.28:0/2750675282 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f150801a070 con 0x7f15101569d0 2024-02-21T23:13:18.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.376+0000 7f1507400640 1 -- 172.21.15.28:0/2750675282 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f150800d840 con 0x7f15101569d0 2024-02-21T23:13:18.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.376+0000 7f1507400640 1 -- 172.21.15.28:0/2750675282 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f150800dc30 con 0x7f15101569d0 2024-02-21T23:13:18.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.376+0000 7f1516200640 1 -- 172.21.15.28:0/2750675282 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15101ef1c0 con 0x7f15101569d0 2024-02-21T23:13:18.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.376+0000 7f1516200640 1 -- 172.21.15.28:0/2750675282 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15101ef780 con 0x7f15101569d0 2024-02-21T23:13:18.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.377+0000 7f1516200640 1 -- 172.21.15.28:0/2750675282 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1510067f30 con 0x7f15101569d0 2024-02-21T23:13:18.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.378+0000 7f1507400640 1 -- 172.21.15.28:0/2750675282 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f150801d070 con 0x7f15101569d0 2024-02-21T23:13:18.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.379+0000 7f1507400640 1 --2- 172.21.15.28:0/2750675282 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f14ec06b500 0x7f14ec06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:18.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.379+0000 7f1507400640 1 -- 172.21.15.28:0/2750675282 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f150808a870 con 0x7f15101569d0 2024-02-21T23:13:18.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.379+0000 7f150f400640 1 --2- 172.21.15.28:0/2750675282 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f14ec06b500 0x7f14ec06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:18.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.381+0000 7f150f400640 1 --2- 172.21.15.28:0/2750675282 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f14ec06b500 0x7f14ec06d9c0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f1500006fd0 tx=0x7f1500008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:18.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.383+0000 7f1507400640 1 -- 172.21.15.28:0/2750675282 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1508059940 con 0x7f15101569d0 2024-02-21T23:13:18.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.586+0000 7f1516200640 1 -- 172.21.15.28:0/2750675282 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f15100c5b70 con 0x7f15101569d0 2024-02-21T23:13:18.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.594+0000 7f1507400640 1 -- 172.21.15.28:0/2750675282 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v62) v1 ==== 104+0+0 (secure 0 0 0) 0x7f1508059240 con 0x7f15101569d0 2024-02-21T23:13:18.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.596+0000 7f1516200640 1 -- 172.21.15.28:0/2750675282 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f14ec06b500 msgr2=0x7f14ec06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:18.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.596+0000 7f1516200640 1 --2- 172.21.15.28:0/2750675282 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f14ec06b500 0x7f14ec06d9c0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f1500006fd0 tx=0x7f1500008040 comp rx=0 tx=0).stop 2024-02-21T23:13:18.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.596+0000 7f1516200640 1 -- 172.21.15.28:0/2750675282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15101569d0 msgr2=0x7f151014b960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:18.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.596+0000 7f1516200640 1 --2- 172.21.15.28:0/2750675282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15101569d0 0x7f151014b960 secure :-1 s=READY pgs=2317 cs=0 l=1 rev1=1 crypto rx=0x7f150800cf20 tx=0x7f150800a7d0 comp rx=0 tx=0).stop 2024-02-21T23:13:18.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.597+0000 7f1516200640 1 -- 172.21.15.28:0/2750675282 shutdown_connections 2024-02-21T23:13:18.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.597+0000 7f1516200640 1 --2- 172.21.15.28:0/2750675282 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f14ec06b500 0x7f14ec06d9c0 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:18.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.597+0000 7f1516200640 1 --2- 172.21.15.28:0/2750675282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15101569d0 0x7f151014b960 unknown :-1 s=CLOSED pgs=2317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:18.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.597+0000 7f1516200640 1 --2- 172.21.15.28:0/2750675282 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f15100c62d0 0x7f151014e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:18.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.597+0000 7f1516200640 1 --2- 172.21.15.28:0/2750675282 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f151006d190 0x7f15101f2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:18.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.597+0000 7f1516200640 1 -- 172.21.15.28:0/2750675282 >> 172.21.15.28:0/2750675282 conn(0x7f15100a85a0 msgr2=0x7f1510072040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:18.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.597+0000 7f1516200640 1 -- 172.21.15.28:0/2750675282 shutdown_connections 2024-02-21T23:13:18.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.597+0000 7f1516200640 1 -- 172.21.15.28:0/2750675282 wait complete. 2024-02-21T23:13:18.605 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestClusterResize.test_down_health' 2024-02-21T23:13:18.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.829+0000 7f8b2f200640 1 Processor -- start 2024-02-21T23:13:18.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.829+0000 7f8b2f200640 1 -- start start 2024-02-21T23:13:18.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.830+0000 7f8b2f200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8b3006ee40 0x7f8b300a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:18.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.830+0000 7f8b2f200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f8b30156af0 0x7f8b3006dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:18.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.831+0000 7f8b2f200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b300c2760 con 0x7f8b30150450 2024-02-21T23:13:18.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.831+0000 7f8b2f200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b300c22c0 con 0x7f8b3006ee40 2024-02-21T23:13:18.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.831+0000 7f8b2f200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b30156660 con 0x7f8b30156af0 2024-02-21T23:13:18.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.831+0000 7f8b2de00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f8b30150450 0x7f8b301536f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50020/0 (socket says 172.21.15.28:50020) 2024-02-21T23:13:18.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.831+0000 7f8b2de00640 1 -- 172.21.15.28:0/1547774131 learned_addr learned my addr 172.21.15.28:0/1547774131 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:13:18.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.831+0000 7f8b2d400640 1 --2- 172.21.15.28:0/1547774131 >> v2:172.21.15.28:3300/0 conn(0x7f8b3006ee40 0x7f8b300a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:18.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.831+0000 7f8b2d400640 1 -- 172.21.15.28:0/1547774131 >> v2:172.21.15.81:3300/0 conn(0x7f8b30156af0 msgr2=0x7f8b3006dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:18.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.831+0000 7f8b2e800640 1 --2- 172.21.15.28:0/1547774131 >> v2:172.21.15.81:3300/0 conn(0x7f8b30156af0 0x7f8b3006dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:18.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.831+0000 7f8b2d400640 1 --2- 172.21.15.28:0/1547774131 >> v2:172.21.15.81:3300/0 conn(0x7f8b30156af0 0x7f8b3006dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:18.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.831+0000 7f8b2d400640 1 -- 172.21.15.28:0/1547774131 >> v1:172.21.15.28:6789/0 conn(0x7f8b30150450 legacy=0x7f8b301536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:18.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.831+0000 7f8b2d400640 1 -- 172.21.15.28:0/1547774131 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b30156310 con 0x7f8b3006ee40 2024-02-21T23:13:18.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.831+0000 7f8b2ca00640 1 -- 172.21.15.28:0/1547774131 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1378510549 0 0) 0x7f8b300c2760 con 0x7f8b30150450 2024-02-21T23:13:18.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.833+0000 7f8b2d400640 1 --2- 172.21.15.28:0/1547774131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b3006ee40 0x7f8b300a5fd0 secure :-1 s=READY pgs=2319 cs=0 l=1 rev1=1 crypto rx=0x7f8b300734b0 tx=0x7f8b1802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fa9fdee2416e1c3 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:18.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.833+0000 7f8b2ca00640 1 -- 172.21.15.28:0/1547774131 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b18035020 con 0x7f8b3006ee40 2024-02-21T23:13:18.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.833+0000 7f8b2ca00640 1 -- 172.21.15.28:0/1547774131 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b18031830 con 0x7f8b3006ee40 2024-02-21T23:13:18.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.833+0000 7f8b2ca00640 1 -- 172.21.15.28:0/1547774131 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b18031c00 con 0x7f8b3006ee40 2024-02-21T23:13:18.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.834+0000 7f8b2f200640 1 -- 172.21.15.28:0/1547774131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b3006ee40 msgr2=0x7f8b300a5fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:18.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.834+0000 7f8b2f200640 1 --2- 172.21.15.28:0/1547774131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b3006ee40 0x7f8b300a5fd0 secure :-1 s=READY pgs=2319 cs=0 l=1 rev1=1 crypto rx=0x7f8b300734b0 tx=0x7f8b1802b8d0 comp rx=0 tx=0).stop 2024-02-21T23:13:18.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.834+0000 7f8b2f200640 1 -- 172.21.15.28:0/1547774131 shutdown_connections 2024-02-21T23:13:18.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.834+0000 7f8b2f200640 1 --2- 172.21.15.28:0/1547774131 >> v2:172.21.15.81:3300/0 conn(0x7f8b30156af0 0x7f8b3006dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:18.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.834+0000 7f8b2f200640 1 --2- 172.21.15.28:0/1547774131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b3006ee40 0x7f8b300a5fd0 unknown :-1 s=CLOSED pgs=2319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:18.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.834+0000 7f8b2f200640 1 -- 172.21.15.28:0/1547774131 >> 172.21.15.28:0/1547774131 conn(0x7f8b3006b630 msgr2=0x7f8b300c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:18.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.834+0000 7f8b2f200640 1 -- 172.21.15.28:0/1547774131 shutdown_connections 2024-02-21T23:13:18.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.834+0000 7f8b2f200640 1 -- 172.21.15.28:0/1547774131 wait complete. 2024-02-21T23:13:18.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.836+0000 7f8b2f200640 1 Processor -- start 2024-02-21T23:13:18.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.836+0000 7f8b2f200640 1 -- start start 2024-02-21T23:13:18.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.837+0000 7f8b2f200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b3006ee40 0x7f8b300c53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:18.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.837+0000 7f8b2f200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8b30150450 0x7f8b30144c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:18.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.837+0000 7f8b2f200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b30156af0 0x7f8b30145190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:18.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.837+0000 7f8b2f200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b30156f80 con 0x7f8b30156af0 2024-02-21T23:13:18.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.837+0000 7f8b2f200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b300c2cb0 con 0x7f8b3006ee40 2024-02-21T23:13:18.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.837+0000 7f8b2f200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b30157100 con 0x7f8b30150450 2024-02-21T23:13:18.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.837+0000 7f8b2e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b30156af0 0x7f8b30145190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:18.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.837+0000 7f8b2e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b30156af0 0x7f8b30145190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51500/0 (socket says 172.21.15.28:51500) 2024-02-21T23:13:18.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.837+0000 7f8b2e800640 1 -- 172.21.15.28:0/871364332 learned_addr learned my addr 172.21.15.28:0/871364332 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:18.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.837+0000 7f8b2d400640 1 --2- 172.21.15.28:0/871364332 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8b30150450 0x7f8b30144c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:18.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.837+0000 7f8b2e800640 1 -- 172.21.15.28:0/871364332 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8b30150450 msgr2=0x7f8b30144c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:18.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.837+0000 7f8b2de00640 1 --2- 172.21.15.28:0/871364332 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b3006ee40 0x7f8b300c53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:18.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.837+0000 7f8b2e800640 1 --2- 172.21.15.28:0/871364332 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8b30150450 0x7f8b30144c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:18.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.837+0000 7f8b2e800640 1 -- 172.21.15.28:0/871364332 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b3006ee40 msgr2=0x7f8b300c53b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:18.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.837+0000 7f8b2e800640 1 --2- 172.21.15.28:0/871364332 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b3006ee40 0x7f8b300c53b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:18.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.837+0000 7f8b2e800640 1 -- 172.21.15.28:0/871364332 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b3014d670 con 0x7f8b30156af0 2024-02-21T23:13:18.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.838+0000 7f8b2e800640 1 --2- 172.21.15.28:0/871364332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b30156af0 0x7f8b30145190 secure :-1 s=READY pgs=2320 cs=0 l=1 rev1=1 crypto rx=0x7f8b2800cc60 tx=0x7f8b2800a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:18.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.838+0000 7f8b26000640 1 -- 172.21.15.28:0/871364332 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b28003990 con 0x7f8b30156af0 2024-02-21T23:13:18.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.838+0000 7f8b26000640 1 -- 172.21.15.28:0/871364332 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b28003b30 con 0x7f8b30156af0 2024-02-21T23:13:18.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.838+0000 7f8b26000640 1 -- 172.21.15.28:0/871364332 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b28004020 con 0x7f8b30156af0 2024-02-21T23:13:18.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.838+0000 7f8b2f200640 1 -- 172.21.15.28:0/871364332 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b3014b330 con 0x7f8b30156af0 2024-02-21T23:13:18.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.838+0000 7f8b2f200640 1 -- 172.21.15.28:0/871364332 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b3014a890 con 0x7f8b30156af0 2024-02-21T23:13:18.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.840+0000 7f8b2f200640 1 -- 172.21.15.28:0/871364332 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b300c64e0 con 0x7f8b30156af0 2024-02-21T23:13:18.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.840+0000 7f8b26000640 1 -- 172.21.15.28:0/871364332 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8b280041c0 con 0x7f8b30156af0 2024-02-21T23:13:18.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.841+0000 7f8b26000640 1 --2- 172.21.15.28:0/871364332 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8b0c06b4d0 0x7f8b0c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:18.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.841+0000 7f8b26000640 1 -- 172.21.15.28:0/871364332 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f8b2808a4a0 con 0x7f8b30156af0 2024-02-21T23:13:18.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.842+0000 7f8b2de00640 1 --2- 172.21.15.28:0/871364332 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8b0c06b4d0 0x7f8b0c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:18.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.843+0000 7f8b2de00640 1 --2- 172.21.15.28:0/871364332 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8b0c06b4d0 0x7f8b0c06d990 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f8b20007fd0 tx=0x7f8b20009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:18.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:18.845+0000 7f8b26000640 1 -- 172.21.15.28:0/871364332 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f8b300c64e0 con 0x7f8b30156af0 2024-02-21T23:13:19.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:19.131+0000 7f8b2f200640 1 -- 172.21.15.28:0/871364332 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterResize.test_down_health"]} v 0) v1 -- 0x7f8b30150830 con 0x7f8b30156af0 2024-02-21T23:13:20.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.126+0000 7f8b26000640 1 -- 172.21.15.28:0/871364332 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterResize.test_down_health"]}]=0 v599) v1 ==== 140+0+0 (secure 0 0 0) 0x7f8b30150830 con 0x7f8b30156af0 2024-02-21T23:13:20.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.130+0000 7f8b2f200640 1 -- 172.21.15.28:0/871364332 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8b0c06b4d0 msgr2=0x7f8b0c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:20.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.130+0000 7f8b2f200640 1 --2- 172.21.15.28:0/871364332 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8b0c06b4d0 0x7f8b0c06d990 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f8b20007fd0 tx=0x7f8b20009040 comp rx=0 tx=0).stop 2024-02-21T23:13:20.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.130+0000 7f8b2f200640 1 -- 172.21.15.28:0/871364332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b30156af0 msgr2=0x7f8b30145190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:20.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.130+0000 7f8b2f200640 1 --2- 172.21.15.28:0/871364332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b30156af0 0x7f8b30145190 secure :-1 s=READY pgs=2320 cs=0 l=1 rev1=1 crypto rx=0x7f8b2800cc60 tx=0x7f8b2800a5c0 comp rx=0 tx=0).stop 2024-02-21T23:13:20.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.130+0000 7f8b2f200640 1 -- 172.21.15.28:0/871364332 shutdown_connections 2024-02-21T23:13:20.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.130+0000 7f8b2f200640 1 --2- 172.21.15.28:0/871364332 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8b0c06b4d0 0x7f8b0c06d990 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:20.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.130+0000 7f8b2f200640 1 --2- 172.21.15.28:0/871364332 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8b30150450 0x7f8b30144c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:20.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.130+0000 7f8b2f200640 1 --2- 172.21.15.28:0/871364332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b30156af0 0x7f8b30145190 unknown :-1 s=CLOSED pgs=2320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:20.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.130+0000 7f8b2f200640 1 --2- 172.21.15.28:0/871364332 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b3006ee40 0x7f8b300c53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:20.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.130+0000 7f8b2f200640 1 -- 172.21.15.28:0/871364332 >> 172.21.15.28:0/871364332 conn(0x7f8b3006b630 msgr2=0x7f8b3014da20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:20.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.130+0000 7f8b2f200640 1 -- 172.21.15.28:0/871364332 shutdown_connections 2024-02-21T23:13:20.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.131+0000 7f8b2f200640 1 -- 172.21.15.28:0/871364332 wait complete. 2024-02-21T23:13:20.138 INFO:tasks.cephfs_test_runner:That marking a FS down does not generate a health warning ... ok 2024-02-21T23:13:20.139 INFO:tasks.cephfs_test_runner:Starting test: test_down_twice (tasks.cephfs.test_failover.TestClusterResize) That marking a FS down twice does not wipe old_max_mds. 2024-02-21T23:13:20.139 INFO:tasks.cephfs_test_runner:test_down_twice (tasks.cephfs.test_failover.TestClusterResize) 2024-02-21T23:13:20.139 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:13:20.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.363+0000 7fb64d400640 1 Processor -- start 2024-02-21T23:13:20.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.364+0000 7fb64d400640 1 -- start start 2024-02-21T23:13:20.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.365+0000 7fb64d400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb6480c0fe0 0x7fb64806e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:20.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.365+0000 7fb64d400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6480a8b40 con 0x7fb648158580 2024-02-21T23:13:20.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.365+0000 7fb64d400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6480a8e00 con 0x7fb64806dff0 2024-02-21T23:13:20.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.365+0000 7fb64d400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6480a90e0 con 0x7fb6480c0fe0 2024-02-21T23:13:20.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.366+0000 7fb64ca00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb648158580 0x7fb64814b2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50036/0 (socket says 172.21.15.28:50036) 2024-02-21T23:13:20.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.366+0000 7fb64ca00640 1 -- 172.21.15.28:0/3764430203 learned_addr learned my addr 172.21.15.28:0/3764430203 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:13:20.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.366+0000 7fb647400640 1 --2- 172.21.15.28:0/3764430203 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb6480c0fe0 0x7fb64806e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:20.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.366+0000 7fb646a00640 1 -- 172.21.15.28:0/3764430203 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 477681979 0 0) 0x7fb6480a8b40 con 0x7fb648158580 2024-02-21T23:13:20.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.366+0000 7fb646a00640 1 -- 172.21.15.28:0/3764430203 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb62c003520 con 0x7fb648158580 2024-02-21T23:13:20.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.367+0000 7fb646a00640 1 -- 172.21.15.28:0/3764430203 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2161171680 0 0) 0x7fb62c003520 con 0x7fb648158580 2024-02-21T23:13:20.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.367+0000 7fb646a00640 1 -- 172.21.15.28:0/3764430203 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb6480c0fe0 msgr2=0x7fb64806e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:20.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.367+0000 7fb646a00640 1 --2- 172.21.15.28:0/3764430203 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb6480c0fe0 0x7fb64806e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:20.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.367+0000 7fb646a00640 1 -- 172.21.15.28:0/3764430203 >> v1:172.21.15.81:6789/0 conn(0x7fb64806dff0 legacy=0x7fb648071350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:20.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.367+0000 7fb646a00640 1 -- 172.21.15.28:0/3764430203 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb648072ed0 con 0x7fb648158580 2024-02-21T23:13:20.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.367+0000 7fb646a00640 1 -- 172.21.15.28:0/3764430203 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fb6400019f0 con 0x7fb648158580 2024-02-21T23:13:20.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.367+0000 7fb646a00640 1 -- 172.21.15.28:0/3764430203 >> v1:172.21.15.28:6789/0 conn(0x7fb648158580 legacy=0x7fb64814b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:20.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.367+0000 7fb646a00640 1 --2- 172.21.15.28:0/3764430203 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb62c0039e0 0x7fb62c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:20.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.367+0000 7fb646a00640 1 --2- 172.21.15.28:0/3764430203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb62c004800 0x7fb62c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:20.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.367+0000 7fb646a00640 1 --2- 172.21.15.28:0/3764430203 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb6480c0fe0 0x7fb62c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:20.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.367+0000 7fb646a00640 1 -- 172.21.15.28:0/3764430203 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6480a90e0 con 0x7fb62c004800 2024-02-21T23:13:20.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.367+0000 7fb646a00640 1 -- 172.21.15.28:0/3764430203 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb62c00baa0 con 0x7fb62c0039e0 2024-02-21T23:13:20.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.367+0000 7fb646a00640 1 -- 172.21.15.28:0/3764430203 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb62c00bca0 con 0x7fb6480c0fe0 2024-02-21T23:13:20.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.367+0000 7fb647400640 1 --2- 172.21.15.28:0/3764430203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb62c004800 0x7fb62c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:20.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.367+0000 7fb647e00640 1 --2- 172.21.15.28:0/3764430203 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb62c0039e0 0x7fb62c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:20.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.368+0000 7fb647400640 1 -- 172.21.15.28:0/3764430203 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb6480c0fe0 msgr2=0x7fb62c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:20.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.368+0000 7fb647400640 1 --2- 172.21.15.28:0/3764430203 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb6480c0fe0 0x7fb62c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:20.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.368+0000 7fb647400640 1 -- 172.21.15.28:0/3764430203 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb62c0039e0 msgr2=0x7fb62c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:20.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.368+0000 7fb647400640 1 --2- 172.21.15.28:0/3764430203 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb62c0039e0 0x7fb62c003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:20.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.368+0000 7fb647400640 1 -- 172.21.15.28:0/3764430203 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb62c003520 con 0x7fb62c004800 2024-02-21T23:13:20.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.369+0000 7fb647400640 1 --2- 172.21.15.28:0/3764430203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb62c004800 0x7fb62c004c80 secure :-1 s=READY pgs=2322 cs=0 l=1 rev1=1 crypto rx=0x7fb634009a90 tx=0x7fb62c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=812fc34b48c024bc server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:20.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.369+0000 7fb646a00640 1 -- 172.21.15.28:0/3764430203 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb634039070 con 0x7fb62c004800 2024-02-21T23:13:20.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.369+0000 7fb646a00640 1 -- 172.21.15.28:0/3764430203 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb63402c6b0 con 0x7fb62c004800 2024-02-21T23:13:20.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.369+0000 7fb646a00640 1 -- 172.21.15.28:0/3764430203 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb634009690 con 0x7fb62c004800 2024-02-21T23:13:20.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.370+0000 7fb64d400640 1 -- 172.21.15.28:0/3764430203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb62c004800 msgr2=0x7fb62c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:20.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.370+0000 7fb64d400640 1 --2- 172.21.15.28:0/3764430203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb62c004800 0x7fb62c004c80 secure :-1 s=READY pgs=2322 cs=0 l=1 rev1=1 crypto rx=0x7fb634009a90 tx=0x7fb62c005830 comp rx=0 tx=0).stop 2024-02-21T23:13:20.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.370+0000 7fb64ca00640 1 -- 172.21.15.28:0/3764430203 reap_dead start 2024-02-21T23:13:20.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.370+0000 7fb64d400640 1 -- 172.21.15.28:0/3764430203 shutdown_connections 2024-02-21T23:13:20.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.370+0000 7fb64d400640 1 -- 172.21.15.28:0/3764430203 >> 172.21.15.28:0/3764430203 conn(0x7fb64806b630 msgr2=0x7fb6480c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:20.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.370+0000 7fb64d400640 1 -- 172.21.15.28:0/3764430203 shutdown_connections 2024-02-21T23:13:20.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.370+0000 7fb64d400640 1 -- 172.21.15.28:0/3764430203 wait complete. 2024-02-21T23:13:20.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.372+0000 7fb64d400640 1 Processor -- start 2024-02-21T23:13:20.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.372+0000 7fb64d400640 1 -- start start 2024-02-21T23:13:20.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.373+0000 7fb64d400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb62c004800 0x7fb6481540a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:20.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.373+0000 7fb64d400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb6481eb190 0x7fb6480c5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:20.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.373+0000 7fb64d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6481ee5a0 0x7fb6480aa3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:20.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.373+0000 7fb64d400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb648158a10 con 0x7fb6481ee5a0 2024-02-21T23:13:20.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.373+0000 7fb64d400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb648073520 con 0x7fb62c004800 2024-02-21T23:13:20.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.373+0000 7fb64d400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb648158b90 con 0x7fb6481eb190 2024-02-21T23:13:20.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.373+0000 7fb64ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6481ee5a0 0x7fb6480aa3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:20.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.373+0000 7fb64ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6481ee5a0 0x7fb6480aa3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51520/0 (socket says 172.21.15.28:51520) 2024-02-21T23:13:20.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.373+0000 7fb64ca00640 1 -- 172.21.15.28:0/181353934 learned_addr learned my addr 172.21.15.28:0/181353934 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:20.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.374+0000 7fb647400640 1 --2- 172.21.15.28:0/181353934 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb6481eb190 0x7fb6480c5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:20.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.374+0000 7fb647e00640 1 --2- 172.21.15.28:0/181353934 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb62c004800 0x7fb6481540a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:20.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.374+0000 7fb64ca00640 1 -- 172.21.15.28:0/181353934 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb6481eb190 msgr2=0x7fb6480c5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:20.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.374+0000 7fb64ca00640 1 --2- 172.21.15.28:0/181353934 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb6481eb190 0x7fb6480c5370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:20.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.374+0000 7fb64ca00640 1 -- 172.21.15.28:0/181353934 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb62c004800 msgr2=0x7fb6481540a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:20.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.374+0000 7fb64ca00640 1 --2- 172.21.15.28:0/181353934 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb62c004800 0x7fb6481540a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:20.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.374+0000 7fb64ca00640 1 -- 172.21.15.28:0/181353934 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6481536f0 con 0x7fb6481ee5a0 2024-02-21T23:13:20.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.374+0000 7fb64ca00640 1 --2- 172.21.15.28:0/181353934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6481ee5a0 0x7fb6480aa3f0 secure :-1 s=READY pgs=2323 cs=0 l=1 rev1=1 crypto rx=0x7fb648152ad0 tx=0x7fb64000cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:20.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.374+0000 7fb63fe00640 1 -- 172.21.15.28:0/181353934 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb64000ce50 con 0x7fb6481ee5a0 2024-02-21T23:13:20.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.374+0000 7fb63fe00640 1 -- 172.21.15.28:0/181353934 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb64000a610 con 0x7fb6481ee5a0 2024-02-21T23:13:20.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.374+0000 7fb63fe00640 1 -- 172.21.15.28:0/181353934 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb64000aa00 con 0x7fb6481ee5a0 2024-02-21T23:13:20.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.374+0000 7fb64d400640 1 -- 172.21.15.28:0/181353934 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb648153130 con 0x7fb6481ee5a0 2024-02-21T23:13:20.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.375+0000 7fb64d400640 1 -- 172.21.15.28:0/181353934 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6481edda0 con 0x7fb6481ee5a0 2024-02-21T23:13:20.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.376+0000 7fb64d400640 1 -- 172.21.15.28:0/181353934 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb64804f290 con 0x7fb6481ee5a0 2024-02-21T23:13:20.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.377+0000 7fb63fe00640 1 -- 172.21.15.28:0/181353934 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb640011070 con 0x7fb6481ee5a0 2024-02-21T23:13:20.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.378+0000 7fb63fe00640 1 --2- 172.21.15.28:0/181353934 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb62806b4d0 0x7fb62806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:20.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.378+0000 7fb63fe00640 1 -- 172.21.15.28:0/181353934 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fb6400892a0 con 0x7fb6481ee5a0 2024-02-21T23:13:20.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.379+0000 7fb647e00640 1 --2- 172.21.15.28:0/181353934 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb62806b4d0 0x7fb62806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:20.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.380+0000 7fb647e00640 1 --2- 172.21.15.28:0/181353934 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb62806b4d0 0x7fb62806d990 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fb648068390 tx=0x7fb638009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:20.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.382+0000 7fb63fe00640 1 -- 172.21.15.28:0/181353934 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb6400582d0 con 0x7fb6481ee5a0 2024-02-21T23:13:20.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.589+0000 7fb64d400640 1 -- 172.21.15.28:0/181353934 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb64806ecb0 con 0x7fb6481ee5a0 2024-02-21T23:13:20.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.591+0000 7fb63fe00640 1 -- 172.21.15.28:0/181353934 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v199) v1 ==== 74+0+13114 (secure 0 0 0) 0x7fb640057bd0 con 0x7fb6481ee5a0 2024-02-21T23:13:20.579 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:13:20.579 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":199,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:13:18.109914+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:13:20.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.593+0000 7fb64d400640 1 -- 172.21.15.28:0/181353934 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb62806b4d0 msgr2=0x7fb62806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:20.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.594+0000 7fb64d400640 1 --2- 172.21.15.28:0/181353934 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb62806b4d0 0x7fb62806d990 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fb648068390 tx=0x7fb638009040 comp rx=0 tx=0).stop 2024-02-21T23:13:20.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.594+0000 7fb64d400640 1 -- 172.21.15.28:0/181353934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6481ee5a0 msgr2=0x7fb6480aa3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:20.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.594+0000 7fb64d400640 1 --2- 172.21.15.28:0/181353934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6481ee5a0 0x7fb6480aa3f0 secure :-1 s=READY pgs=2323 cs=0 l=1 rev1=1 crypto rx=0x7fb648152ad0 tx=0x7fb64000cc00 comp rx=0 tx=0).stop 2024-02-21T23:13:20.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.594+0000 7fb64d400640 1 -- 172.21.15.28:0/181353934 shutdown_connections 2024-02-21T23:13:20.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.594+0000 7fb64d400640 1 --2- 172.21.15.28:0/181353934 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb62806b4d0 0x7fb62806d990 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:20.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.594+0000 7fb64d400640 1 --2- 172.21.15.28:0/181353934 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb6481eb190 0x7fb6480c5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:20.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.594+0000 7fb64d400640 1 --2- 172.21.15.28:0/181353934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6481ee5a0 0x7fb6480aa3f0 unknown :-1 s=CLOSED pgs=2323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:20.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.594+0000 7fb64d400640 1 --2- 172.21.15.28:0/181353934 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb62c004800 0x7fb6481540a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:20.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.594+0000 7fb64d400640 1 -- 172.21.15.28:0/181353934 >> 172.21.15.28:0/181353934 conn(0x7fb64806b630 msgr2=0x7fb6480c42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:20.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.594+0000 7fb64d400640 1 -- 172.21.15.28:0/181353934 shutdown_connections 2024-02-21T23:13:20.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.594+0000 7fb64d400640 1 -- 172.21.15.28:0/181353934 wait complete. 2024-02-21T23:13:20.601 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-21T23:02:30.728329+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-21T23:13:20.601 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-21T23:13:20.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.834+0000 7f676aa00640 1 Processor -- start 2024-02-21T23:13:20.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.834+0000 7f676aa00640 1 -- start start 2024-02-21T23:13:20.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.836+0000 7f676aa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6764149830 0x7f6764071060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:20.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.836+0000 7f676aa00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f67641535b0 0x7f67640c8bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:20.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.836+0000 7f676aa00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67640c7b90 con 0x7f67640c7730 2024-02-21T23:13:20.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.836+0000 7f676aa00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6764153120 con 0x7f6764149830 2024-02-21T23:13:20.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.836+0000 7f676aa00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67640c7e30 con 0x7f67641535b0 2024-02-21T23:13:20.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.836+0000 7f6769600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6764149830 0x7f6764071060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:20.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.836+0000 7f6768c00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f67640c7730 0x7f6764156190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50046/0 (socket says 172.21.15.28:50046) 2024-02-21T23:13:20.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.836+0000 7f6769600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6764149830 0x7f6764071060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51530/0 (socket says 172.21.15.28:51530) 2024-02-21T23:13:20.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.836+0000 7f6768c00640 1 -- 172.21.15.28:0/1823213172 learned_addr learned my addr 172.21.15.28:0/1823213172 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:13:20.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.837+0000 7f6769600640 1 -- 172.21.15.28:0/1823213172 >> v2:172.21.15.81:3300/0 conn(0x7f67641535b0 msgr2=0x7f67640c8bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:20.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.837+0000 7f676a000640 1 --2- 172.21.15.28:0/1823213172 >> v2:172.21.15.81:3300/0 conn(0x7f67641535b0 0x7f67640c8bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:20.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.837+0000 7f6769600640 1 --2- 172.21.15.28:0/1823213172 >> v2:172.21.15.81:3300/0 conn(0x7f67641535b0 0x7f67640c8bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:20.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.837+0000 7f6769600640 1 -- 172.21.15.28:0/1823213172 >> v1:172.21.15.28:6789/0 conn(0x7f67640c7730 legacy=0x7f6764156190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:20.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.837+0000 7f6769600640 1 -- 172.21.15.28:0/1823213172 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67640c8110 con 0x7f6764149830 2024-02-21T23:13:20.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.837+0000 7f6763e00640 1 -- 172.21.15.28:0/1823213172 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1051931002 0 0) 0x7f67640c7b90 con 0x7f67640c7730 2024-02-21T23:13:20.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.837+0000 7f676a000640 1 --2- 172.21.15.28:0/1823213172 >> v2:172.21.15.81:3300/0 conn(0x7f67641535b0 0x7f67640c8bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:13:20.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.838+0000 7f6769600640 1 --2- 172.21.15.28:0/1823213172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6764149830 0x7f6764071060 secure :-1 s=READY pgs=2325 cs=0 l=1 rev1=1 crypto rx=0x7f676406efa0 tx=0x7f675802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85411f47d7d6f565 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:20.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.838+0000 7f6763e00640 1 -- 172.21.15.28:0/1823213172 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67580315a0 con 0x7f6764149830 2024-02-21T23:13:20.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.839+0000 7f6763e00640 1 -- 172.21.15.28:0/1823213172 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6758037070 con 0x7f6764149830 2024-02-21T23:13:20.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.839+0000 7f6763e00640 1 -- 172.21.15.28:0/1823213172 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6758031960 con 0x7f6764149830 2024-02-21T23:13:20.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.839+0000 7f676aa00640 1 -- 172.21.15.28:0/1823213172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6764149830 msgr2=0x7f6764071060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:20.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.839+0000 7f676aa00640 1 --2- 172.21.15.28:0/1823213172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6764149830 0x7f6764071060 secure :-1 s=READY pgs=2325 cs=0 l=1 rev1=1 crypto rx=0x7f676406efa0 tx=0x7f675802b840 comp rx=0 tx=0).stop 2024-02-21T23:13:20.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.839+0000 7f676aa00640 1 -- 172.21.15.28:0/1823213172 shutdown_connections 2024-02-21T23:13:20.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.839+0000 7f676aa00640 1 --2- 172.21.15.28:0/1823213172 >> v2:172.21.15.81:3300/0 conn(0x7f67641535b0 0x7f67640c8bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:20.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.839+0000 7f676aa00640 1 --2- 172.21.15.28:0/1823213172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6764149830 0x7f6764071060 unknown :-1 s=CLOSED pgs=2325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:20.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.839+0000 7f676aa00640 1 -- 172.21.15.28:0/1823213172 >> 172.21.15.28:0/1823213172 conn(0x7f676406b290 msgr2=0x7f676406b6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:20.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.839+0000 7f676aa00640 1 -- 172.21.15.28:0/1823213172 shutdown_connections 2024-02-21T23:13:20.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.839+0000 7f676aa00640 1 -- 172.21.15.28:0/1823213172 wait complete. 2024-02-21T23:13:20.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.841+0000 7f676aa00640 1 Processor -- start 2024-02-21T23:13:20.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.841+0000 7f676aa00640 1 -- start start 2024-02-21T23:13:20.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.842+0000 7f676aa00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f67640c7730 0x7f676414c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:20.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.842+0000 7f676aa00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6764149830 0x7f67640c1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:20.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.842+0000 7f676aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67641535b0 0x7f6764149d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:20.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.842+0000 7f676aa00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6764153b80 con 0x7f67641535b0 2024-02-21T23:13:20.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.842+0000 7f676aa00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6764153a00 con 0x7f67640c7730 2024-02-21T23:13:20.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.842+0000 7f676aa00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67640c90f0 con 0x7f6764149830 2024-02-21T23:13:20.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.842+0000 7f676a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67641535b0 0x7f6764149d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:20.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.842+0000 7f676a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67641535b0 0x7f6764149d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51532/0 (socket says 172.21.15.28:51532) 2024-02-21T23:13:20.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.842+0000 7f676a000640 1 -- 172.21.15.28:0/2131318621 learned_addr learned my addr 172.21.15.28:0/2131318621 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:20.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.843+0000 7f6768c00640 1 --2- 172.21.15.28:0/2131318621 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6764149830 0x7f67640c1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:20.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.843+0000 7f6769600640 1 --2- 172.21.15.28:0/2131318621 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f67640c7730 0x7f676414c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:20.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.843+0000 7f676a000640 1 -- 172.21.15.28:0/2131318621 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6764149830 msgr2=0x7f67640c1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:20.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.843+0000 7f676a000640 1 --2- 172.21.15.28:0/2131318621 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6764149830 0x7f67640c1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:20.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.843+0000 7f676a000640 1 -- 172.21.15.28:0/2131318621 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f67640c7730 msgr2=0x7f676414c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:20.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.843+0000 7f676a000640 1 --2- 172.21.15.28:0/2131318621 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f67640c7730 0x7f676414c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:20.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.843+0000 7f676a000640 1 -- 172.21.15.28:0/2131318621 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f676406c6e0 con 0x7f67641535b0 2024-02-21T23:13:20.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.843+0000 7f676a000640 1 --2- 172.21.15.28:0/2131318621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67641535b0 0x7f6764149d10 secure :-1 s=READY pgs=2326 cs=0 l=1 rev1=1 crypto rx=0x7f675c00cd80 tx=0x7f675c00a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:20.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.844+0000 7f6761600640 1 -- 172.21.15.28:0/2131318621 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f675c00f680 con 0x7f67641535b0 2024-02-21T23:13:20.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.844+0000 7f6761600640 1 -- 172.21.15.28:0/2131318621 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f675c00f840 con 0x7f67641535b0 2024-02-21T23:13:20.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.844+0000 7f6761600640 1 -- 172.21.15.28:0/2131318621 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f675c00fc30 con 0x7f67641535b0 2024-02-21T23:13:20.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.844+0000 7f676aa00640 1 -- 172.21.15.28:0/2131318621 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f676414aca0 con 0x7f67641535b0 2024-02-21T23:13:20.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.844+0000 7f676aa00640 1 -- 172.21.15.28:0/2131318621 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f676406c980 con 0x7f67641535b0 2024-02-21T23:13:20.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.845+0000 7f676aa00640 1 -- 172.21.15.28:0/2131318621 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f676404f290 con 0x7f67641535b0 2024-02-21T23:13:20.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.846+0000 7f6761600640 1 -- 172.21.15.28:0/2131318621 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f675c0213f0 con 0x7f67641535b0 2024-02-21T23:13:20.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.847+0000 7f6761600640 1 --2- 172.21.15.28:0/2131318621 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f674806b500 0x7f674806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:20.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.848+0000 7f6761600640 1 -- 172.21.15.28:0/2131318621 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f675c08ae40 con 0x7f67641535b0 2024-02-21T23:13:20.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.848+0000 7f6769600640 1 --2- 172.21.15.28:0/2131318621 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f674806b500 0x7f674806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:20.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.849+0000 7f6769600640 1 --2- 172.21.15.28:0/2131318621 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f674806b500 0x7f674806d9c0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f6758036000 tx=0x7f6758008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:20.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:20.851+0000 7f6761600640 1 -- 172.21.15.28:0/2131318621 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f675c014030 con 0x7f67641535b0 2024-02-21T23:13:21.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.052+0000 7f676aa00640 1 -- 172.21.15.28:0/2131318621 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f676406bb90 con 0x7f67641535b0 2024-02-21T23:13:21.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.053+0000 7f6761600640 1 -- 172.21.15.28:0/2131318621 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v199) v1 ==== 93+0+10 (secure 0 0 0) 0x7f675c059790 con 0x7f67641535b0 2024-02-21T23:13:21.040 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-02-21T23:13:21.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.055+0000 7f676aa00640 1 -- 172.21.15.28:0/2131318621 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f674806b500 msgr2=0x7f674806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:21.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.055+0000 7f676aa00640 1 --2- 172.21.15.28:0/2131318621 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f674806b500 0x7f674806d9c0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f6758036000 tx=0x7f6758008030 comp rx=0 tx=0).stop 2024-02-21T23:13:21.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.055+0000 7f676aa00640 1 -- 172.21.15.28:0/2131318621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67641535b0 msgr2=0x7f6764149d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:21.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.055+0000 7f676aa00640 1 --2- 172.21.15.28:0/2131318621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67641535b0 0x7f6764149d10 secure :-1 s=READY pgs=2326 cs=0 l=1 rev1=1 crypto rx=0x7f675c00cd80 tx=0x7f675c00a650 comp rx=0 tx=0).stop 2024-02-21T23:13:21.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.056+0000 7f676aa00640 1 -- 172.21.15.28:0/2131318621 shutdown_connections 2024-02-21T23:13:21.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.056+0000 7f676aa00640 1 --2- 172.21.15.28:0/2131318621 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f674806b500 0x7f674806d9c0 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:21.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.056+0000 7f676aa00640 1 --2- 172.21.15.28:0/2131318621 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6764149830 0x7f67640c1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:21.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.056+0000 7f676aa00640 1 --2- 172.21.15.28:0/2131318621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67641535b0 0x7f6764149d10 unknown :-1 s=CLOSED pgs=2326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:21.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.056+0000 7f676aa00640 1 --2- 172.21.15.28:0/2131318621 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f67640c7730 0x7f676414c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:21.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.056+0000 7f676aa00640 1 -- 172.21.15.28:0/2131318621 >> 172.21.15.28:0/2131318621 conn(0x7f676406b290 msgr2=0x7f6764150e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:21.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.056+0000 7f676aa00640 1 -- 172.21.15.28:0/2131318621 shutdown_connections 2024-02-21T23:13:21.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.056+0000 7f676aa00640 1 -- 172.21.15.28:0/2131318621 wait complete. 2024-02-21T23:13:21.064 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_failover.TestClusterResize.test_down_twice' 2024-02-21T23:13:21.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.303+0000 7f02a1400640 1 Processor -- start 2024-02-21T23:13:21.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.304+0000 7f02a1400640 1 -- start start 2024-02-21T23:13:21.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.305+0000 7f02a1400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f029c14b750 0x7f029c151e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:21.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.305+0000 7f02a1400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f029c155300 0x7f029c0a7690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:21.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.305+0000 7f02a1400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f029c0a7bd0 0x7f029c06e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:21.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.305+0000 7f02a1400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f029c1519a0 con 0x7f029c155300 2024-02-21T23:13:21.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.305+0000 7f02a1400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f029c0c2790 con 0x7f029c0a7bd0 2024-02-21T23:13:21.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.305+0000 7f02a1400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f029c0c2a70 con 0x7f029c14b750 2024-02-21T23:13:21.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.306+0000 7f0293400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f029c155300 0x7f029c0a7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:21.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.306+0000 7f0293400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f029c155300 0x7f029c0a7690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51538/0 (socket says 172.21.15.28:51538) 2024-02-21T23:13:21.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.306+0000 7f0293400640 1 -- 172.21.15.28:0/3553466197 learned_addr learned my addr 172.21.15.28:0/3553466197 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:21.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.306+0000 7f029be00640 1 --2- 172.21.15.28:0/3553466197 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f029c14b750 0x7f029c151e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:21.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.306+0000 7f02a0a00640 1 --2- 172.21.15.28:0/3553466197 >> v2:172.21.15.81:3300/0 conn(0x7f029c0a7bd0 0x7f029c06e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:21.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.306+0000 7f0293400640 1 -- 172.21.15.28:0/3553466197 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f029c14b750 msgr2=0x7f029c151e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:21.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.306+0000 7f0293400640 1 --2- 172.21.15.28:0/3553466197 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f029c14b750 0x7f029c151e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:21.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.306+0000 7f0293400640 1 -- 172.21.15.28:0/3553466197 >> v2:172.21.15.81:3300/0 conn(0x7f029c0a7bd0 msgr2=0x7f029c06e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:21.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.306+0000 7f0293400640 1 --2- 172.21.15.28:0/3553466197 >> v2:172.21.15.81:3300/0 conn(0x7f029c0a7bd0 0x7f029c06e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:21.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.306+0000 7f0293400640 1 -- 172.21.15.28:0/3553466197 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f029c151630 con 0x7f029c155300 2024-02-21T23:13:21.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.308+0000 7f0293400640 1 --2- 172.21.15.28:0/3553466197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f029c155300 0x7f029c0a7690 secure :-1 s=READY pgs=2327 cs=0 l=1 rev1=1 crypto rx=0x7f029c0a9370 tx=0x7f028402de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2da000c590a5fbb0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:21.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.308+0000 7f029b400640 1 -- 172.21.15.28:0/3553466197 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0284034070 con 0x7f029c155300 2024-02-21T23:13:21.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.308+0000 7f029b400640 1 -- 172.21.15.28:0/3553466197 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f028402ebc0 con 0x7f029c155300 2024-02-21T23:13:21.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.308+0000 7f029b400640 1 -- 172.21.15.28:0/3553466197 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0284038400 con 0x7f029c155300 2024-02-21T23:13:21.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.309+0000 7f02a1400640 1 -- 172.21.15.28:0/3553466197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f029c155300 msgr2=0x7f029c0a7690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:21.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.309+0000 7f02a1400640 1 --2- 172.21.15.28:0/3553466197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f029c155300 0x7f029c0a7690 secure :-1 s=READY pgs=2327 cs=0 l=1 rev1=1 crypto rx=0x7f029c0a9370 tx=0x7f028402de10 comp rx=0 tx=0).stop 2024-02-21T23:13:21.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.309+0000 7f02a1400640 1 -- 172.21.15.28:0/3553466197 shutdown_connections 2024-02-21T23:13:21.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.309+0000 7f02a1400640 1 --2- 172.21.15.28:0/3553466197 >> v2:172.21.15.81:3300/0 conn(0x7f029c0a7bd0 0x7f029c06e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:21.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.309+0000 7f02a1400640 1 --2- 172.21.15.28:0/3553466197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f029c155300 0x7f029c0a7690 unknown :-1 s=CLOSED pgs=2327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:21.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.309+0000 7f02a1400640 1 --2- 172.21.15.28:0/3553466197 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f029c14b750 0x7f029c151e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:21.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.309+0000 7f02a1400640 1 -- 172.21.15.28:0/3553466197 >> 172.21.15.28:0/3553466197 conn(0x7f029c06b630 msgr2=0x7f029c0c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:21.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.309+0000 7f02a1400640 1 -- 172.21.15.28:0/3553466197 shutdown_connections 2024-02-21T23:13:21.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.309+0000 7f02a1400640 1 -- 172.21.15.28:0/3553466197 wait complete. 2024-02-21T23:13:21.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.311+0000 7f02a1400640 1 Processor -- start 2024-02-21T23:13:21.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.311+0000 7f02a1400640 1 -- start start 2024-02-21T23:13:21.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.312+0000 7f02a1400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f029c0a7bd0 0x7f029c0c54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:21.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.312+0000 7f02a1400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f029c14b750 0x7f029c147270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:21.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.312+0000 7f02a1400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f029c155300 0x7f029c1477b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:21.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.312+0000 7f02a1400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f029c0c09c0 con 0x7f029c14b750 2024-02-21T23:13:21.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.312+0000 7f02a1400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f029c155760 con 0x7f029c155300 2024-02-21T23:13:21.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.312+0000 7f02a1400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f029c0c0b40 con 0x7f029c0a7bd0 2024-02-21T23:13:21.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.312+0000 7f0293400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f029c14b750 0x7f029c147270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:21.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.312+0000 7f0293400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f029c14b750 0x7f029c147270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51548/0 (socket says 172.21.15.28:51548) 2024-02-21T23:13:21.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.312+0000 7f0293400640 1 -- 172.21.15.28:0/3015297169 learned_addr learned my addr 172.21.15.28:0/3015297169 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:21.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.313+0000 7f029be00640 1 --2- 172.21.15.28:0/3015297169 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f029c0a7bd0 0x7f029c0c54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:21.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.313+0000 7f02a0a00640 1 --2- 172.21.15.28:0/3015297169 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f029c155300 0x7f029c1477b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:21.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.313+0000 7f0293400640 1 -- 172.21.15.28:0/3015297169 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f029c0a7bd0 msgr2=0x7f029c0c54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:21.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.313+0000 7f0293400640 1 --2- 172.21.15.28:0/3015297169 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f029c0a7bd0 0x7f029c0c54e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:21.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.313+0000 7f0293400640 1 -- 172.21.15.28:0/3015297169 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f029c155300 msgr2=0x7f029c1477b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:21.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.313+0000 7f0293400640 1 --2- 172.21.15.28:0/3015297169 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f029c155300 0x7f029c1477b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:21.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.313+0000 7f0293400640 1 -- 172.21.15.28:0/3015297169 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f029c1e9f50 con 0x7f029c14b750 2024-02-21T23:13:21.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.313+0000 7f0293400640 1 --2- 172.21.15.28:0/3015297169 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f029c14b750 0x7f029c147270 secure :-1 s=READY pgs=2328 cs=0 l=1 rev1=1 crypto rx=0x7f0284000c00 tx=0x7f02840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:21.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.314+0000 7f0298c00640 1 -- 172.21.15.28:0/3015297169 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0284047070 con 0x7f029c14b750 2024-02-21T23:13:21.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.314+0000 7f0298c00640 1 -- 172.21.15.28:0/3015297169 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0284043650 con 0x7f029c14b750 2024-02-21T23:13:21.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.314+0000 7f0298c00640 1 -- 172.21.15.28:0/3015297169 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0284043a20 con 0x7f029c14b750 2024-02-21T23:13:21.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.314+0000 7f02a1400640 1 -- 172.21.15.28:0/3015297169 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f029c1ea210 con 0x7f029c14b750 2024-02-21T23:13:21.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.314+0000 7f02a1400640 1 -- 172.21.15.28:0/3015297169 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f029c0c5a20 con 0x7f029c14b750 2024-02-21T23:13:21.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.315+0000 7f02a1400640 1 -- 172.21.15.28:0/3015297169 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f029c04f290 con 0x7f029c14b750 2024-02-21T23:13:21.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.316+0000 7f0298c00640 1 -- 172.21.15.28:0/3015297169 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0284043bc0 con 0x7f029c14b750 2024-02-21T23:13:21.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.317+0000 7f0298c00640 1 --2- 172.21.15.28:0/3015297169 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f027806b4d0 0x7f027806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:21.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.317+0000 7f0298c00640 1 -- 172.21.15.28:0/3015297169 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f02840ae530 con 0x7f029c14b750 2024-02-21T23:13:21.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.318+0000 7f029be00640 1 --2- 172.21.15.28:0/3015297169 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f027806b4d0 0x7f027806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:21.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.319+0000 7f029be00640 1 --2- 172.21.15.28:0/3015297169 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f027806b4d0 0x7f027806d990 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f028c009990 tx=0x7f028c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:21.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.322+0000 7f0298c00640 1 -- 172.21.15.28:0/3015297169 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f028407d4d0 con 0x7f029c14b750 2024-02-21T23:13:21.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:21.592+0000 7f02a1400640 1 -- 172.21.15.28:0/3015297169 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterResize.test_down_twice"]} v 0) v1 -- 0x7f029c1ea3e0 con 0x7f029c14b750 2024-02-21T23:13:22.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.140+0000 7f0298c00640 1 -- 172.21.15.28:0/3015297169 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_failover.TestClusterResize.test_down_twice"]}]=0 v601) v1 ==== 142+0+0 (secure 0 0 0) 0x7f028407cdd0 con 0x7f029c14b750 2024-02-21T23:13:22.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.143+0000 7f02a1400640 1 -- 172.21.15.28:0/3015297169 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f027806b4d0 msgr2=0x7f027806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:22.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.143+0000 7f02a1400640 1 --2- 172.21.15.28:0/3015297169 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f027806b4d0 0x7f027806d990 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f028c009990 tx=0x7f028c008040 comp rx=0 tx=0).stop 2024-02-21T23:13:22.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.143+0000 7f02a1400640 1 -- 172.21.15.28:0/3015297169 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f029c14b750 msgr2=0x7f029c147270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:22.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.143+0000 7f02a1400640 1 --2- 172.21.15.28:0/3015297169 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f029c14b750 0x7f029c147270 secure :-1 s=READY pgs=2328 cs=0 l=1 rev1=1 crypto rx=0x7f0284000c00 tx=0x7f02840046c0 comp rx=0 tx=0).stop 2024-02-21T23:13:22.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.144+0000 7f02a1400640 1 -- 172.21.15.28:0/3015297169 shutdown_connections 2024-02-21T23:13:22.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.144+0000 7f02a1400640 1 --2- 172.21.15.28:0/3015297169 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f027806b4d0 0x7f027806d990 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:22.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.144+0000 7f02a1400640 1 --2- 172.21.15.28:0/3015297169 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f029c155300 0x7f029c1477b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:22.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.144+0000 7f02a1400640 1 --2- 172.21.15.28:0/3015297169 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f029c14b750 0x7f029c147270 unknown :-1 s=CLOSED pgs=2328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:22.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.144+0000 7f02a1400640 1 --2- 172.21.15.28:0/3015297169 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f029c0a7bd0 0x7f029c0c54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:22.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.144+0000 7f02a1400640 1 -- 172.21.15.28:0/3015297169 >> 172.21.15.28:0/3015297169 conn(0x7f029c06b630 msgr2=0x7f029c071d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:22.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.144+0000 7f02a1400640 1 -- 172.21.15.28:0/3015297169 shutdown_connections 2024-02-21T23:13:22.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.144+0000 7f02a1400640 1 -- 172.21.15.28:0/3015297169 wait complete. 2024-02-21T23:13:22.154 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-02-21T23:13:22.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.384+0000 7f434de00640 1 Processor -- start 2024-02-21T23:13:22.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.384+0000 7f434de00640 1 -- start start 2024-02-21T23:13:22.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.385+0000 7f434de00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4348150240 0x7f43480c12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:22.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.385+0000 7f434de00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f43480c2c10 0x7f434806ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:22.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.385+0000 7f434de00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f434806e220 con 0x7f43480c4580 2024-02-21T23:13:22.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.385+0000 7f434de00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4348150620 con 0x7f4348150240 2024-02-21T23:13:22.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.385+0000 7f434de00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4348153a90 con 0x7f43480c2c10 2024-02-21T23:13:22.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.386+0000 7f434ca00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4348150240 0x7f43480c12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:22.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.386+0000 7f434ca00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4348150240 0x7f43480c12c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51552/0 (socket says 172.21.15.28:51552) 2024-02-21T23:13:22.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.386+0000 7f434ca00640 1 -- 172.21.15.28:0/2781205505 learned_addr learned my addr 172.21.15.28:0/2781205505 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:22.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.386+0000 7f434d400640 1 --2- 172.21.15.28:0/2781205505 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f43480c2c10 0x7f434806ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:22.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.387+0000 7f434ca00640 1 -- 172.21.15.28:0/2781205505 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f43480c2c10 msgr2=0x7f434806ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:22.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.387+0000 7f434ca00640 1 --2- 172.21.15.28:0/2781205505 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f43480c2c10 0x7f434806ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:22.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.387+0000 7f434ca00640 1 -- 172.21.15.28:0/2781205505 >> v1:172.21.15.81:6789/0 conn(0x7f43480c4580 legacy=0x7f434806db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:22.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.387+0000 7f434ca00640 1 -- 172.21.15.28:0/2781205505 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f434806f780 con 0x7f4348150240 2024-02-21T23:13:22.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.387+0000 7f434d400640 1 --2- 172.21.15.28:0/2781205505 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f43480c2c10 0x7f434806ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:13:22.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.388+0000 7f434ca00640 1 --2- 172.21.15.28:0/2781205505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4348150240 0x7f43480c12c0 secure :-1 s=READY pgs=2329 cs=0 l=1 rev1=1 crypto rx=0x7f433c009870 tx=0x7f433c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=136da63cb9c39a37 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:22.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.389+0000 7f4347400640 1 -- 172.21.15.28:0/2781205505 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f433c035020 con 0x7f4348150240 2024-02-21T23:13:22.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.389+0000 7f4347400640 1 -- 172.21.15.28:0/2781205505 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f433c031640 con 0x7f4348150240 2024-02-21T23:13:22.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.389+0000 7f4347400640 1 -- 172.21.15.28:0/2781205505 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f433c02d040 con 0x7f4348150240 2024-02-21T23:13:22.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.389+0000 7f434de00640 1 -- 172.21.15.28:0/2781205505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4348150240 msgr2=0x7f43480c12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:22.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.389+0000 7f434de00640 1 --2- 172.21.15.28:0/2781205505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4348150240 0x7f43480c12c0 secure :-1 s=READY pgs=2329 cs=0 l=1 rev1=1 crypto rx=0x7f433c009870 tx=0x7f433c02b8f0 comp rx=0 tx=0).stop 2024-02-21T23:13:22.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.389+0000 7f434de00640 1 -- 172.21.15.28:0/2781205505 shutdown_connections 2024-02-21T23:13:22.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.389+0000 7f434de00640 1 --2- 172.21.15.28:0/2781205505 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f43480c2c10 0x7f434806ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:22.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.389+0000 7f434de00640 1 --2- 172.21.15.28:0/2781205505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4348150240 0x7f43480c12c0 unknown :-1 s=CLOSED pgs=2329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:22.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.389+0000 7f434de00640 1 -- 172.21.15.28:0/2781205505 >> 172.21.15.28:0/2781205505 conn(0x7f43480a8530 msgr2=0x7f43480a8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:22.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.389+0000 7f434de00640 1 -- 172.21.15.28:0/2781205505 shutdown_connections 2024-02-21T23:13:22.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.390+0000 7f434de00640 1 -- 172.21.15.28:0/2781205505 wait complete. 2024-02-21T23:13:22.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.391+0000 7f434de00640 1 Processor -- start 2024-02-21T23:13:22.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.391+0000 7f434de00640 1 -- start start 2024-02-21T23:13:22.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.392+0000 7f434de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43480c2c10 0x7f43480ab760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:22.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.392+0000 7f434de00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f43480c4580 0x7f434814b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:22.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.393+0000 7f434de00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4348150240 0x7f434814bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:22.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.393+0000 7f434de00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43480c4a10 con 0x7f43480c2c10 2024-02-21T23:13:22.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.393+0000 7f434de00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4348153720 con 0x7f4348150240 2024-02-21T23:13:22.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.393+0000 7f434de00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43480c4b90 con 0x7f43480c4580 2024-02-21T23:13:22.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.393+0000 7f434ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43480c2c10 0x7f43480ab760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:22.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.393+0000 7f434ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43480c2c10 0x7f43480ab760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51558/0 (socket says 172.21.15.28:51558) 2024-02-21T23:13:22.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.393+0000 7f434ca00640 1 -- 172.21.15.28:0/348951413 learned_addr learned my addr 172.21.15.28:0/348951413 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:22.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.393+0000 7f4347e00640 1 --2- 172.21.15.28:0/348951413 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f43480c4580 0x7f434814b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:22.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.393+0000 7f434d400640 1 --2- 172.21.15.28:0/348951413 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4348150240 0x7f434814bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:22.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.393+0000 7f434ca00640 1 -- 172.21.15.28:0/348951413 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f43480c4580 msgr2=0x7f434814b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:22.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.393+0000 7f434ca00640 1 --2- 172.21.15.28:0/348951413 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f43480c4580 0x7f434814b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:22.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.393+0000 7f434ca00640 1 -- 172.21.15.28:0/348951413 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4348150240 msgr2=0x7f434814bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:22.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.393+0000 7f434ca00640 1 --2- 172.21.15.28:0/348951413 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4348150240 0x7f434814bec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:22.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.393+0000 7f434ca00640 1 -- 172.21.15.28:0/348951413 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43481e8dc0 con 0x7f43480c2c10 2024-02-21T23:13:22.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.393+0000 7f434d400640 1 --2- 172.21.15.28:0/348951413 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4348150240 0x7f434814bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:13:22.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.394+0000 7f434ca00640 1 --2- 172.21.15.28:0/348951413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43480c2c10 0x7f43480ab760 secure :-1 s=READY pgs=2330 cs=0 l=1 rev1=1 crypto rx=0x7f433c000c00 tx=0x7f433c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:22.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.394+0000 7f4344c00640 1 -- 172.21.15.28:0/348951413 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f433c040070 con 0x7f43480c2c10 2024-02-21T23:13:22.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.394+0000 7f4344c00640 1 -- 172.21.15.28:0/348951413 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f433c0399f0 con 0x7f43480c2c10 2024-02-21T23:13:22.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.394+0000 7f4344c00640 1 -- 172.21.15.28:0/348951413 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f433c035040 con 0x7f43480c2c10 2024-02-21T23:13:22.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.394+0000 7f434de00640 1 -- 172.21.15.28:0/348951413 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43481e90a0 con 0x7f43480c2c10 2024-02-21T23:13:22.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.394+0000 7f434de00640 1 -- 172.21.15.28:0/348951413 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43480abca0 con 0x7f43480c2c10 2024-02-21T23:13:22.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.396+0000 7f434de00640 1 -- 172.21.15.28:0/348951413 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f434804f290 con 0x7f43480c2c10 2024-02-21T23:13:22.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.397+0000 7f4344c00640 1 -- 172.21.15.28:0/348951413 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f433c04b060 con 0x7f43480c2c10 2024-02-21T23:13:22.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.397+0000 7f4344c00640 1 --2- 172.21.15.28:0/348951413 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f432806b4d0 0x7f432806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:22.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.398+0000 7f4344c00640 1 -- 172.21.15.28:0/348951413 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f433c0ab890 con 0x7f43480c2c10 2024-02-21T23:13:22.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.398+0000 7f4347e00640 1 --2- 172.21.15.28:0/348951413 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f432806b4d0 0x7f432806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:22.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.399+0000 7f4347e00640 1 --2- 172.21.15.28:0/348951413 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f432806b4d0 0x7f432806d990 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f43480ac1f0 tx=0x7f4334009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:22.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.402+0000 7f4344c00640 1 -- 172.21.15.28:0/348951413 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f433c07a960 con 0x7f43480c2c10 2024-02-21T23:13:22.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.606+0000 7f434de00640 1 -- 172.21.15.28:0/348951413 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f434806f0e0 con 0x7f43480c2c10 2024-02-21T23:13:22.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.611+0000 7f4344c00640 1 -- 172.21.15.28:0/348951413 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v63)=0 v63) v1 ==== 122+0+0 (secure 0 0 0) 0x7f433c07a260 con 0x7f43480c2c10 2024-02-21T23:13:22.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.614+0000 7f434de00640 1 -- 172.21.15.28:0/348951413 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f432806b4d0 msgr2=0x7f432806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:22.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.614+0000 7f434de00640 1 --2- 172.21.15.28:0/348951413 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f432806b4d0 0x7f432806d990 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f43480ac1f0 tx=0x7f4334009040 comp rx=0 tx=0).stop 2024-02-21T23:13:22.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.614+0000 7f434de00640 1 -- 172.21.15.28:0/348951413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43480c2c10 msgr2=0x7f43480ab760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:22.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.614+0000 7f434de00640 1 --2- 172.21.15.28:0/348951413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43480c2c10 0x7f43480ab760 secure :-1 s=READY pgs=2330 cs=0 l=1 rev1=1 crypto rx=0x7f433c000c00 tx=0x7f433c0046c0 comp rx=0 tx=0).stop 2024-02-21T23:13:22.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.614+0000 7f434de00640 1 -- 172.21.15.28:0/348951413 shutdown_connections 2024-02-21T23:13:22.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.614+0000 7f434de00640 1 --2- 172.21.15.28:0/348951413 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f432806b4d0 0x7f432806d990 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:22.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.614+0000 7f434de00640 1 --2- 172.21.15.28:0/348951413 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f43480c4580 0x7f434814b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:22.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.614+0000 7f434de00640 1 --2- 172.21.15.28:0/348951413 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4348150240 0x7f434814bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:22.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.614+0000 7f434de00640 1 --2- 172.21.15.28:0/348951413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43480c2c10 0x7f43480ab760 unknown :-1 s=CLOSED pgs=2330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:22.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.614+0000 7f434de00640 1 -- 172.21.15.28:0/348951413 >> 172.21.15.28:0/348951413 conn(0x7f43480a8530 msgr2=0x7f43480c5da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:22.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.614+0000 7f434de00640 1 -- 172.21.15.28:0/348951413 shutdown_connections 2024-02-21T23:13:22.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:22.614+0000 7f434de00640 1 -- 172.21.15.28:0/348951413 wait complete. 2024-02-21T23:13:22.621 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-02-21T23:13:22.621 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-21T23:13:22.743 DEBUG:teuthology.orchestra.run.smithi081:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-21T23:13:22.870 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-02-21T23:13:22.870 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-21T23:13:22.870 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-21T23:13:22.930 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054608,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-02-21T23:13:22.931 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-02-21T23:13:22.931 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-21T23:13:22.986 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053833,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498812k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249404k,nr_inodes=812351,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498812k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-02-21T23:13:22.987 INFO:tasks.cephfs.cephfs_test_case:[, ] 2024-02-21T23:13:22.987 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:13:23.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.230+0000 7eff2f200640 1 Processor -- start 2024-02-21T23:13:23.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.230+0000 7eff2f200640 1 -- start start 2024-02-21T23:13:23.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.232+0000 7eff2f200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7eff280c3490 0x7eff2806dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:23.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.232+0000 7eff2f200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7eff280a5f40 0x7eff28152140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:23.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.232+0000 7eff2f200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7eff28152680 0x7eff280c0270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:23.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.232+0000 7eff2f200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff2806eff0 con 0x7eff280a5f40 2024-02-21T23:13:23.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.232+0000 7eff2f200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff280bfde0 con 0x7eff28152680 2024-02-21T23:13:23.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.232+0000 7eff2f200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff2806e4f0 con 0x7eff280c3490 2024-02-21T23:13:23.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.232+0000 7eff1fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7eff280a5f40 0x7eff28152140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:23.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.232+0000 7eff1fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7eff280a5f40 0x7eff28152140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51560/0 (socket says 172.21.15.28:51560) 2024-02-21T23:13:23.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.232+0000 7eff1fe00640 1 -- 172.21.15.28:0/1347197474 learned_addr learned my addr 172.21.15.28:0/1347197474 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:23.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.233+0000 7eff27e00640 1 --2- 172.21.15.28:0/1347197474 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7eff280c3490 0x7eff2806dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:23.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.233+0000 7eff2d200640 1 --2- 172.21.15.28:0/1347197474 >> v2:172.21.15.81:3300/0 conn(0x7eff28152680 0x7eff280c0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:23.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.233+0000 7eff1fe00640 1 -- 172.21.15.28:0/1347197474 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7eff280c3490 msgr2=0x7eff2806dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:23.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.233+0000 7eff1fe00640 1 --2- 172.21.15.28:0/1347197474 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7eff280c3490 0x7eff2806dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:23.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.233+0000 7eff1fe00640 1 -- 172.21.15.28:0/1347197474 >> v2:172.21.15.81:3300/0 conn(0x7eff28152680 msgr2=0x7eff280c0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:23.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.233+0000 7eff1fe00640 1 --2- 172.21.15.28:0/1347197474 >> v2:172.21.15.81:3300/0 conn(0x7eff28152680 0x7eff280c0270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:23.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.233+0000 7eff1fe00640 1 -- 172.21.15.28:0/1347197474 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff2806e750 con 0x7eff280a5f40 2024-02-21T23:13:23.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.233+0000 7eff2d200640 1 --2- 172.21.15.28:0/1347197474 >> v2:172.21.15.81:3300/0 conn(0x7eff28152680 0x7eff280c0270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:13:23.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.234+0000 7eff1fe00640 1 --2- 172.21.15.28:0/1347197474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff280a5f40 0x7eff28152140 secure :-1 s=READY pgs=2331 cs=0 l=1 rev1=1 crypto rx=0x7eff280c10c0 tx=0x7eff1002b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be93252de29219f8 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:23.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.234+0000 7eff27400640 1 -- 172.21.15.28:0/1347197474 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff100317f0 con 0x7eff280a5f40 2024-02-21T23:13:23.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.235+0000 7eff27400640 1 -- 172.21.15.28:0/1347197474 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff10037070 con 0x7eff280a5f40 2024-02-21T23:13:23.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.235+0000 7eff27400640 1 -- 172.21.15.28:0/1347197474 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff10031bb0 con 0x7eff280a5f40 2024-02-21T23:13:23.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.235+0000 7eff2f200640 1 -- 172.21.15.28:0/1347197474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff280a5f40 msgr2=0x7eff28152140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:23.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.235+0000 7eff2f200640 1 --2- 172.21.15.28:0/1347197474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff280a5f40 0x7eff28152140 secure :-1 s=READY pgs=2331 cs=0 l=1 rev1=1 crypto rx=0x7eff280c10c0 tx=0x7eff1002b890 comp rx=0 tx=0).stop 2024-02-21T23:13:23.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.235+0000 7eff2f200640 1 -- 172.21.15.28:0/1347197474 shutdown_connections 2024-02-21T23:13:23.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.235+0000 7eff2f200640 1 --2- 172.21.15.28:0/1347197474 >> v2:172.21.15.81:3300/0 conn(0x7eff28152680 0x7eff280c0270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:23.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.235+0000 7eff2f200640 1 --2- 172.21.15.28:0/1347197474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff280a5f40 0x7eff28152140 unknown :-1 s=CLOSED pgs=2331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:23.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.235+0000 7eff2f200640 1 --2- 172.21.15.28:0/1347197474 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7eff280c3490 0x7eff2806dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:23.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.235+0000 7eff2f200640 1 -- 172.21.15.28:0/1347197474 >> 172.21.15.28:0/1347197474 conn(0x7eff280c67f0 msgr2=0x7eff280c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:23.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.235+0000 7eff2f200640 1 -- 172.21.15.28:0/1347197474 shutdown_connections 2024-02-21T23:13:23.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.236+0000 7eff2f200640 1 -- 172.21.15.28:0/1347197474 wait complete. 2024-02-21T23:13:23.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.237+0000 7eff2f200640 1 Processor -- start 2024-02-21T23:13:23.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.237+0000 7eff2f200640 1 -- start start 2024-02-21T23:13:23.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.238+0000 7eff2f200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7eff280a5f40 0x7eff281442d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:23.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.238+0000 7eff2f200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff280c3490 0x7eff28149fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:23.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.238+0000 7eff2f200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7eff28152680 0x7eff281eafc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:23.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.238+0000 7eff2f200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff280c1c70 con 0x7eff280c3490 2024-02-21T23:13:23.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.238+0000 7eff2f200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff2806ed10 con 0x7eff280a5f40 2024-02-21T23:13:23.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.238+0000 7eff2f200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff280c1df0 con 0x7eff28152680 2024-02-21T23:13:23.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.238+0000 7eff1fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff280c3490 0x7eff28149fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:23.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.239+0000 7eff1fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff280c3490 0x7eff28149fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51562/0 (socket says 172.21.15.28:51562) 2024-02-21T23:13:23.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.239+0000 7eff1fe00640 1 -- 172.21.15.28:0/3439532226 learned_addr learned my addr 172.21.15.28:0/3439532226 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:23.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.239+0000 7eff27e00640 1 --2- 172.21.15.28:0/3439532226 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7eff280a5f40 0x7eff281442d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:23.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.239+0000 7eff2d200640 1 --2- 172.21.15.28:0/3439532226 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7eff28152680 0x7eff281eafc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:23.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.239+0000 7eff1fe00640 1 -- 172.21.15.28:0/3439532226 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7eff28152680 msgr2=0x7eff281eafc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:23.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.239+0000 7eff1fe00640 1 --2- 172.21.15.28:0/3439532226 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7eff28152680 0x7eff281eafc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:23.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.239+0000 7eff1fe00640 1 -- 172.21.15.28:0/3439532226 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7eff280a5f40 msgr2=0x7eff281442d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:23.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.239+0000 7eff1fe00640 1 --2- 172.21.15.28:0/3439532226 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7eff280a5f40 0x7eff281442d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:23.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.239+0000 7eff1fe00640 1 -- 172.21.15.28:0/3439532226 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff28073100 con 0x7eff280c3490 2024-02-21T23:13:23.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.239+0000 7eff1fe00640 1 --2- 172.21.15.28:0/3439532226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff280c3490 0x7eff28149fd0 secure :-1 s=READY pgs=2332 cs=0 l=1 rev1=1 crypto rx=0x7eff10000c00 tx=0x7eff100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:23.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.240+0000 7eff24c00640 1 -- 172.21.15.28:0/3439532226 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff10032890 con 0x7eff280c3490 2024-02-21T23:13:23.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.240+0000 7eff24c00640 1 -- 172.21.15.28:0/3439532226 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff1002e070 con 0x7eff280c3490 2024-02-21T23:13:23.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.240+0000 7eff24c00640 1 -- 172.21.15.28:0/3439532226 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff10032c70 con 0x7eff280c3490 2024-02-21T23:13:23.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.240+0000 7eff2f200640 1 -- 172.21.15.28:0/3439532226 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff2814ae80 con 0x7eff280c3490 2024-02-21T23:13:23.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.240+0000 7eff2f200640 1 -- 172.21.15.28:0/3439532226 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff281eeac0 con 0x7eff280c3490 2024-02-21T23:13:23.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.241+0000 7eff2f200640 1 -- 172.21.15.28:0/3439532226 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff28067f30 con 0x7eff280c3490 2024-02-21T23:13:23.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.242+0000 7eff24c00640 1 -- 172.21.15.28:0/3439532226 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7eff1004c060 con 0x7eff280c3490 2024-02-21T23:13:23.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.243+0000 7eff24c00640 1 --2- 172.21.15.28:0/3439532226 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7eff0406b4d0 0x7eff0406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:23.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.243+0000 7eff24c00640 1 -- 172.21.15.28:0/3439532226 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 5249+0+0 (secure 0 0 0) 0x7eff100abbb0 con 0x7eff280c3490 2024-02-21T23:13:23.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.244+0000 7eff27e00640 1 --2- 172.21.15.28:0/3439532226 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7eff0406b4d0 0x7eff0406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:23.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.245+0000 7eff27e00640 1 --2- 172.21.15.28:0/3439532226 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7eff0406b4d0 0x7eff0406d990 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7eff18006fd0 tx=0x7eff18008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:23.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.247+0000 7eff24c00640 1 -- 172.21.15.28:0/3439532226 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7eff10040070 con 0x7eff280c3490 2024-02-21T23:13:23.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.494+0000 7eff2f200640 1 -- 172.21.15.28:0/3439532226 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7eff28145490 con 0x7eff280c3490 2024-02-21T23:13:23.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.496+0000 7eff24c00640 1 -- 172.21.15.28:0/3439532226 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 223 v223) v1 ==== 95+0+3705 (secure 0 0 0) 0x7eff1007abe0 con 0x7eff280c3490 2024-02-21T23:13:23.483 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:13:23.483 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":223,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7100,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3213818811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3213818811},{"type":"v1","addr":"172.21.15.28:6835","nonce":3213818811}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":213},{"gid":7106,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/1322911156","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6835","nonce":1322911156},{"type":"v1","addr":"172.21.15.81:6837","nonce":1322911156}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":213},{"gid":7184,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6836/1214841323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1214841323},{"type":"v1","addr":"172.21.15.81:6836","nonce":1214841323}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":221},{"gid":10386,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3814985204","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3814985204},{"type":"v1","addr":"172.21.15.28:6834","nonce":3814985204}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":212}],"filesystems":[]} 2024-02-21T23:13:23.484 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 223 2024-02-21T23:13:23.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.498+0000 7eff2f200640 1 -- 172.21.15.28:0/3439532226 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7eff0406b4d0 msgr2=0x7eff0406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:23.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.498+0000 7eff2f200640 1 --2- 172.21.15.28:0/3439532226 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7eff0406b4d0 0x7eff0406d990 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7eff18006fd0 tx=0x7eff18008040 comp rx=0 tx=0).stop 2024-02-21T23:13:23.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.498+0000 7eff2f200640 1 -- 172.21.15.28:0/3439532226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff280c3490 msgr2=0x7eff28149fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:23.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.498+0000 7eff2f200640 1 --2- 172.21.15.28:0/3439532226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff280c3490 0x7eff28149fd0 secure :-1 s=READY pgs=2332 cs=0 l=1 rev1=1 crypto rx=0x7eff10000c00 tx=0x7eff100046c0 comp rx=0 tx=0).stop 2024-02-21T23:13:23.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.498+0000 7eff2f200640 1 -- 172.21.15.28:0/3439532226 shutdown_connections 2024-02-21T23:13:23.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.498+0000 7eff2f200640 1 --2- 172.21.15.28:0/3439532226 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7eff0406b4d0 0x7eff0406d990 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:23.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.498+0000 7eff2f200640 1 --2- 172.21.15.28:0/3439532226 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7eff28152680 0x7eff281eafc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:23.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.498+0000 7eff2f200640 1 --2- 172.21.15.28:0/3439532226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff280c3490 0x7eff28149fd0 unknown :-1 s=CLOSED pgs=2332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:23.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.498+0000 7eff2f200640 1 --2- 172.21.15.28:0/3439532226 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7eff280a5f40 0x7eff281442d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:23.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.499+0000 7eff2f200640 1 -- 172.21.15.28:0/3439532226 >> 172.21.15.28:0/3439532226 conn(0x7eff280c67f0 msgr2=0x7eff280c5650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:23.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.499+0000 7eff2f200640 1 -- 172.21.15.28:0/3439532226 shutdown_connections 2024-02-21T23:13:23.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.499+0000 7eff2f200640 1 -- 172.21.15.28:0/3439532226 wait complete. 2024-02-21T23:13:23.508 INFO:tasks.ceph.mds.a:Restarting daemon 2024-02-21T23:13:23.509 INFO:tasks.ceph.mds.a:Stopping old one... 2024-02-21T23:13:23.509 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-02-21T23:13:23.509 INFO:teuthology.orchestra.run:waiting for 300 2024-02-21T23:13:23.510 INFO:tasks.ceph.mds.c:Restarting daemon 2024-02-21T23:13:23.510 INFO:tasks.ceph.mds.c:Stopping old one... 2024-02-21T23:13:23.510 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-02-21T23:13:23.510 INFO:teuthology.orchestra.run:waiting for 300 2024-02-21T23:13:23.510 INFO:tasks.ceph.mds.b:Restarting daemon 2024-02-21T23:13:23.510 INFO:tasks.ceph.mds.b:Stopping old one... 2024-02-21T23:13:23.511 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-02-21T23:13:23.511 INFO:teuthology.orchestra.run:waiting for 300 2024-02-21T23:13:23.511 INFO:tasks.ceph.mds.d:Restarting daemon 2024-02-21T23:13:23.511 INFO:tasks.ceph.mds.d:Stopping old one... 2024-02-21T23:13:23.511 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-02-21T23:13:23.511 INFO:teuthology.orchestra.run:waiting for 300 2024-02-21T23:13:23.529 INFO:tasks.ceph.mds.c:Stopped 2024-02-21T23:13:23.530 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-02-21T23:13:23.533 INFO:tasks.ceph.mds.a:Stopped 2024-02-21T23:13:23.534 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-02-21T23:13:23.535 INFO:tasks.ceph.mds.b:Stopped 2024-02-21T23:13:23.535 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-02-21T23:13:23.540 INFO:tasks.ceph.mds.d:Stopped 2024-02-21T23:13:23.540 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-02-21T23:13:23.580 INFO:tasks.ceph.mds.c:Started 2024-02-21T23:13:23.581 DEBUG:teuthology.parallel:result is None 2024-02-21T23:13:23.587 INFO:tasks.ceph.mds.a:Started 2024-02-21T23:13:23.587 DEBUG:teuthology.parallel:result is None 2024-02-21T23:13:23.587 INFO:tasks.ceph.mds.b:Started 2024-02-21T23:13:23.588 DEBUG:teuthology.parallel:result is None 2024-02-21T23:13:23.594 INFO:tasks.ceph.mds.d:Started 2024-02-21T23:13:23.594 DEBUG:teuthology.parallel:result is None 2024-02-21T23:13:23.594 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-02-21T23:13:23.744 INFO:tasks.ceph.mds.d.smithi081.stdout:starting mds.d at 2024-02-21T23:13:23.744 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-02-21T23:13:23.746 INFO:tasks.ceph.mds.b.smithi081.stdout:starting mds.b at 2024-02-21T23:13:23.750 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-02-21T23:13:23.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.804+0000 7fa015200640 1 Processor -- start 2024-02-21T23:13:23.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.804+0000 7fa015200640 1 -- start start 2024-02-21T23:13:23.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.805+0000 7fa015200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa010156ed0 0x7fa010153930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:23.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.805+0000 7fa015200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa010153e70 con 0x7fa0100c5f20 2024-02-21T23:13:23.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.805+0000 7fa015200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0100c5c10 con 0x7fa0100c2df0 2024-02-21T23:13:23.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.805+0000 7fa015200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa01006e6a0 con 0x7fa010156ed0 2024-02-21T23:13:23.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.805+0000 7fa00e800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa010156ed0 0x7fa010153930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:23.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.805+0000 7fa00e800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa010156ed0 0x7fa010153930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51608/0 (socket says 172.21.15.28:51608) 2024-02-21T23:13:23.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.805+0000 7fa007e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa0100c5f20 0x7fa010153220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50062/0 (socket says 172.21.15.28:50062) 2024-02-21T23:13:23.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.805+0000 7fa00e800640 1 -- 172.21.15.28:0/3040805659 learned_addr learned my addr 172.21.15.28:0/3040805659 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:23.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.806+0000 7fa00e800640 1 -- 172.21.15.28:0/3040805659 >> v1:172.21.15.81:6789/0 conn(0x7fa0100c2df0 legacy=0x7fa0100c31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:23.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.806+0000 7fa00e800640 1 -- 172.21.15.28:0/3040805659 >> v1:172.21.15.28:6789/0 conn(0x7fa0100c5f20 legacy=0x7fa010153220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:23.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.806+0000 7fa00e800640 1 -- 172.21.15.28:0/3040805659 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa01006e8d0 con 0x7fa010156ed0 2024-02-21T23:13:23.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.807+0000 7fa00e800640 1 --2- 172.21.15.28:0/3040805659 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa010156ed0 0x7fa010153930 secure :-1 s=READY pgs=2339 cs=0 l=1 rev1=1 crypto rx=0x7fa008009e50 tx=0x7fa00802bff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cdb1766744e9661 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:23.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.807+0000 7fa00d400640 1 -- 172.21.15.28:0/3040805659 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa00802cb90 con 0x7fa010156ed0 2024-02-21T23:13:23.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.807+0000 7fa00d400640 1 -- 172.21.15.28:0/3040805659 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa00802cd30 con 0x7fa010156ed0 2024-02-21T23:13:23.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.807+0000 7fa00d400640 1 -- 172.21.15.28:0/3040805659 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa008034610 con 0x7fa010156ed0 2024-02-21T23:13:23.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.808+0000 7fa015200640 1 -- 172.21.15.28:0/3040805659 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa010156ed0 msgr2=0x7fa010153930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:23.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.808+0000 7fa015200640 1 --2- 172.21.15.28:0/3040805659 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa010156ed0 0x7fa010153930 secure :-1 s=READY pgs=2339 cs=0 l=1 rev1=1 crypto rx=0x7fa008009e50 tx=0x7fa00802bff0 comp rx=0 tx=0).stop 2024-02-21T23:13:23.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.808+0000 7fa015200640 1 -- 172.21.15.28:0/3040805659 shutdown_connections 2024-02-21T23:13:23.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.808+0000 7fa015200640 1 --2- 172.21.15.28:0/3040805659 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa010156ed0 0x7fa010153930 unknown :-1 s=CLOSED pgs=2339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:23.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.808+0000 7fa015200640 1 -- 172.21.15.28:0/3040805659 >> 172.21.15.28:0/3040805659 conn(0x7fa01006b820 msgr2=0x7fa0100c67f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:23.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.808+0000 7fa015200640 1 -- 172.21.15.28:0/3040805659 shutdown_connections 2024-02-21T23:13:23.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.808+0000 7fa015200640 1 -- 172.21.15.28:0/3040805659 wait complete. 2024-02-21T23:13:23.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.810+0000 7fa015200640 1 Processor -- start 2024-02-21T23:13:23.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.810+0000 7fa015200640 1 -- start start 2024-02-21T23:13:23.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.811+0000 7fa015200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa0100c2df0 0x7fa01014f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:23.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.811+0000 7fa015200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0100c5f20 0x7fa01014fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:23.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.811+0000 7fa015200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa010156ed0 0x7fa010071340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:23.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.811+0000 7fa015200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa01006f170 con 0x7fa0100c5f20 2024-02-21T23:13:23.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.811+0000 7fa015200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa01006ee90 con 0x7fa010156ed0 2024-02-21T23:13:23.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.811+0000 7fa015200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0100c38e0 con 0x7fa0100c2df0 2024-02-21T23:13:23.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.811+0000 7fa007e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0100c5f20 0x7fa01014fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:23.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.811+0000 7fa007e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0100c5f20 0x7fa01014fd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51614/0 (socket says 172.21.15.28:51614) 2024-02-21T23:13:23.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.811+0000 7fa007e00640 1 -- 172.21.15.28:0/2410981671 learned_addr learned my addr 172.21.15.28:0/2410981671 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:23.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.811+0000 7fa00e800640 1 --2- 172.21.15.28:0/2410981671 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa010156ed0 0x7fa010071340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:23.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.811+0000 7fa00de00640 1 --2- 172.21.15.28:0/2410981671 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa0100c2df0 0x7fa01014f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:23.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.811+0000 7fa007e00640 1 -- 172.21.15.28:0/2410981671 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa0100c2df0 msgr2=0x7fa01014f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:23.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.811+0000 7fa007e00640 1 --2- 172.21.15.28:0/2410981671 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa0100c2df0 0x7fa01014f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:23.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.811+0000 7fa007e00640 1 -- 172.21.15.28:0/2410981671 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa010156ed0 msgr2=0x7fa010071340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:23.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.811+0000 7fa007e00640 1 --2- 172.21.15.28:0/2410981671 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa010156ed0 0x7fa010071340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:23.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.811+0000 7fa007e00640 1 -- 172.21.15.28:0/2410981671 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa01014fa30 con 0x7fa0100c5f20 2024-02-21T23:13:23.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.812+0000 7fa007e00640 1 --2- 172.21.15.28:0/2410981671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0100c5f20 0x7fa01014fd10 secure :-1 s=READY pgs=2340 cs=0 l=1 rev1=1 crypto rx=0x7f9ff800c500 tx=0x7f9ff800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:23.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.812+0000 7fa006000640 1 -- 172.21.15.28:0/2410981671 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ff8018070 con 0x7fa0100c5f20 2024-02-21T23:13:23.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.812+0000 7fa006000640 1 -- 172.21.15.28:0/2410981671 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ff8007500 con 0x7fa0100c5f20 2024-02-21T23:13:23.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.812+0000 7fa006000640 1 -- 172.21.15.28:0/2410981671 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ff80049e0 con 0x7fa0100c5f20 2024-02-21T23:13:23.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.812+0000 7fa015200640 1 -- 172.21.15.28:0/2410981671 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0101dc8c0 con 0x7fa0100c5f20 2024-02-21T23:13:23.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.812+0000 7fa015200640 1 -- 172.21.15.28:0/2410981671 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0101dcf40 con 0x7fa0100c5f20 2024-02-21T23:13:23.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.813+0000 7fa015200640 1 -- 172.21.15.28:0/2410981671 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0101de670 con 0x7fa0100c5f20 2024-02-21T23:13:23.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.814+0000 7fa006000640 1 -- 172.21.15.28:0/2410981671 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9ff8004b80 con 0x7fa0100c5f20 2024-02-21T23:13:23.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.815+0000 7fa006000640 1 --2- 172.21.15.28:0/2410981671 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9fec06b4d0 0x7f9fec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:23.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.815+0000 7fa006000640 1 -- 172.21.15.28:0/2410981671 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(199..199 src has 1..199) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f9ff8089280 con 0x7fa0100c5f20 2024-02-21T23:13:23.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.816+0000 7fa00de00640 1 --2- 172.21.15.28:0/2410981671 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9fec06b4d0 0x7f9fec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:23.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.816+0000 7fa00de00640 1 --2- 172.21.15.28:0/2410981671 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9fec06b4d0 0x7f9fec06d990 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fa000007fd0 tx=0x7fa000009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:23.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:23.819+0000 7fa006000640 1 -- 172.21.15.28:0/2410981671 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f9ff8057c60 con 0x7fa0100c5f20 2024-02-21T23:13:24.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:24.033+0000 7fa015200640 1 -- 172.21.15.28:0/2410981671 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fa01006f400 con 0x7fa0100c5f20 2024-02-21T23:13:25.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.181+0000 7fa006000640 1 -- 172.21.15.28:0/2410981671 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v204) v1 ==== 97+0+0 (secure 0 0 0) 0x7f9ff805bd00 con 0x7fa0100c5f20 2024-02-21T23:13:25.169 INFO:teuthology.orchestra.run.smithi028.stderr: removed all blocklist entries 2024-02-21T23:13:25.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.186+0000 7fa015200640 1 -- 172.21.15.28:0/2410981671 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9fec06b4d0 msgr2=0x7f9fec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:25.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.186+0000 7fa015200640 1 --2- 172.21.15.28:0/2410981671 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9fec06b4d0 0x7f9fec06d990 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fa000007fd0 tx=0x7fa000009040 comp rx=0 tx=0).stop 2024-02-21T23:13:25.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.186+0000 7fa015200640 1 -- 172.21.15.28:0/2410981671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0100c5f20 msgr2=0x7fa01014fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:25.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.186+0000 7fa015200640 1 --2- 172.21.15.28:0/2410981671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0100c5f20 0x7fa01014fd10 secure :-1 s=READY pgs=2340 cs=0 l=1 rev1=1 crypto rx=0x7f9ff800c500 tx=0x7f9ff800c9c0 comp rx=0 tx=0).stop 2024-02-21T23:13:25.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.186+0000 7fa015200640 1 -- 172.21.15.28:0/2410981671 shutdown_connections 2024-02-21T23:13:25.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.186+0000 7fa015200640 1 --2- 172.21.15.28:0/2410981671 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9fec06b4d0 0x7f9fec06d990 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:25.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.186+0000 7fa015200640 1 --2- 172.21.15.28:0/2410981671 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa010156ed0 0x7fa010071340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:25.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.186+0000 7fa015200640 1 --2- 172.21.15.28:0/2410981671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0100c5f20 0x7fa01014fd10 unknown :-1 s=CLOSED pgs=2340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:25.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.186+0000 7fa015200640 1 --2- 172.21.15.28:0/2410981671 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa0100c2df0 0x7fa01014f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:25.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.186+0000 7fa015200640 1 -- 172.21.15.28:0/2410981671 >> 172.21.15.28:0/2410981671 conn(0x7fa01006b820 msgr2=0x7fa0100c43e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:25.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.187+0000 7fa015200640 1 -- 172.21.15.28:0/2410981671 shutdown_connections 2024-02-21T23:13:25.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.187+0000 7fa015200640 1 -- 172.21.15.28:0/2410981671 wait complete. 2024-02-21T23:13:25.194 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-02-21T23:13:25.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.428+0000 7f9b9b200640 1 Processor -- start 2024-02-21T23:13:25.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.428+0000 7f9b9b200640 1 -- start start 2024-02-21T23:13:25.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.430+0000 7f9b9b200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f9b9c150450 0x7f9b9c06e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:25.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.430+0000 7f9b9b200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9b9c0c2bb0 0x7f9b9c152c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:25.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.430+0000 7f9b9b200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b9c0a6540 con 0x7f9b9c06e550 2024-02-21T23:13:25.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.430+0000 7f9b9b200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b9c0c6df0 con 0x7f9b9c0c2bb0 2024-02-21T23:13:25.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.430+0000 7f9b9b200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b9c0c26d0 con 0x7f9b9c150450 2024-02-21T23:13:25.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.430+0000 7f9b91400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9b9c0c2bb0 0x7f9b9c152c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:25.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.430+0000 7f9b9a800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f9b9c06e550 0x7f9b9c0c0f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59520/0 (socket says 172.21.15.28:59520) 2024-02-21T23:13:25.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.430+0000 7f9b9a800640 1 -- 172.21.15.28:0/1470589977 learned_addr learned my addr 172.21.15.28:0/1470589977 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:13:25.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.431+0000 7f9b99e00640 1 --2- 172.21.15.28:0/1470589977 >> v2:172.21.15.81:3300/0 conn(0x7f9b9c150450 0x7f9b9c06e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:25.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.431+0000 7f9b91400640 1 -- 172.21.15.28:0/1470589977 >> v2:172.21.15.81:3300/0 conn(0x7f9b9c150450 msgr2=0x7f9b9c06e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:25.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.431+0000 7f9b91400640 1 --2- 172.21.15.28:0/1470589977 >> v2:172.21.15.81:3300/0 conn(0x7f9b9c150450 0x7f9b9c06e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:25.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.431+0000 7f9b91400640 1 -- 172.21.15.28:0/1470589977 >> v1:172.21.15.28:6789/0 conn(0x7f9b9c06e550 legacy=0x7f9b9c0c0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:25.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.431+0000 7f9b91400640 1 -- 172.21.15.28:0/1470589977 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b9c0c7090 con 0x7f9b9c0c2bb0 2024-02-21T23:13:25.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.432+0000 7f9b91400640 1 --2- 172.21.15.28:0/1470589977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b9c0c2bb0 0x7f9b9c152c50 secure :-1 s=READY pgs=2342 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c06ef00 tx=0x7f9b8402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5243fa5e90f1efb server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:25.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.433+0000 7f9b99400640 1 -- 172.21.15.28:0/1470589977 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b840315a0 con 0x7f9b9c0c2bb0 2024-02-21T23:13:25.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.433+0000 7f9b99400640 1 -- 172.21.15.28:0/1470589977 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b84037070 con 0x7f9b9c0c2bb0 2024-02-21T23:13:25.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.433+0000 7f9b99400640 1 -- 172.21.15.28:0/1470589977 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b84031960 con 0x7f9b9c0c2bb0 2024-02-21T23:13:25.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.433+0000 7f9b9b200640 1 -- 172.21.15.28:0/1470589977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b9c0c2bb0 msgr2=0x7f9b9c152c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:25.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.433+0000 7f9b9b200640 1 --2- 172.21.15.28:0/1470589977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b9c0c2bb0 0x7f9b9c152c50 secure :-1 s=READY pgs=2342 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c06ef00 tx=0x7f9b8402b840 comp rx=0 tx=0).stop 2024-02-21T23:13:25.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.433+0000 7f9b9b200640 1 -- 172.21.15.28:0/1470589977 shutdown_connections 2024-02-21T23:13:25.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.433+0000 7f9b9b200640 1 --2- 172.21.15.28:0/1470589977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b9c0c2bb0 0x7f9b9c152c50 unknown :-1 s=CLOSED pgs=2342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:25.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.433+0000 7f9b9b200640 1 --2- 172.21.15.28:0/1470589977 >> v2:172.21.15.81:3300/0 conn(0x7f9b9c150450 0x7f9b9c06e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:25.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.433+0000 7f9b9b200640 1 -- 172.21.15.28:0/1470589977 >> 172.21.15.28:0/1470589977 conn(0x7f9b9c06b550 msgr2=0x7f9b9c0c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:25.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.434+0000 7f9b9b200640 1 -- 172.21.15.28:0/1470589977 shutdown_connections 2024-02-21T23:13:25.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.434+0000 7f9b9b200640 1 -- 172.21.15.28:0/1470589977 wait complete. 2024-02-21T23:13:25.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.435+0000 7f9b9b200640 1 Processor -- start 2024-02-21T23:13:25.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.435+0000 7f9b9b200640 1 -- start start 2024-02-21T23:13:25.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.436+0000 7f9b9b200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9b9c06e550 0x7f9b9c1ea750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:25.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.436+0000 7f9b9b200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9b9c0c2bb0 0x7f9b9c0bfba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:25.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.436+0000 7f9b9b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b9c150450 0x7f9b9c1ee430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:25.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.436+0000 7f9b9b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b9c0a7b80 con 0x7f9b9c150450 2024-02-21T23:13:25.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.436+0000 7f9b9b200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b9c0a7a00 con 0x7f9b9c0c2bb0 2024-02-21T23:13:25.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.436+0000 7f9b9b200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b9c0a5f80 con 0x7f9b9c06e550 2024-02-21T23:13:25.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.437+0000 7f9b9a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b9c150450 0x7f9b9c1ee430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:25.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.437+0000 7f9b9a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b9c150450 0x7f9b9c1ee430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33568/0 (socket says 172.21.15.28:33568) 2024-02-21T23:13:25.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.437+0000 7f9b9a800640 1 -- 172.21.15.28:0/3885650467 learned_addr learned my addr 172.21.15.28:0/3885650467 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:25.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.437+0000 7f9b91400640 1 --2- 172.21.15.28:0/3885650467 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9b9c0c2bb0 0x7f9b9c0bfba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:25.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.437+0000 7f9b99e00640 1 --2- 172.21.15.28:0/3885650467 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9b9c06e550 0x7f9b9c1ea750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:25.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.437+0000 7f9b9a800640 1 -- 172.21.15.28:0/3885650467 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9b9c06e550 msgr2=0x7f9b9c1ea750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:25.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.437+0000 7f9b9a800640 1 --2- 172.21.15.28:0/3885650467 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9b9c06e550 0x7f9b9c1ea750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:25.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.437+0000 7f9b9a800640 1 -- 172.21.15.28:0/3885650467 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9b9c0c2bb0 msgr2=0x7f9b9c0bfba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:25.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.437+0000 7f9b9a800640 1 --2- 172.21.15.28:0/3885650467 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9b9c0c2bb0 0x7f9b9c0bfba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:25.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.437+0000 7f9b9a800640 1 -- 172.21.15.28:0/3885650467 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b9c072380 con 0x7f9b9c150450 2024-02-21T23:13:25.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.437+0000 7f9b91400640 1 --2- 172.21.15.28:0/3885650467 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9b9c0c2bb0 0x7f9b9c0bfba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:13:25.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.437+0000 7f9b9a800640 1 --2- 172.21.15.28:0/3885650467 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b9c150450 0x7f9b9c1ee430 secure :-1 s=READY pgs=2343 cs=0 l=1 rev1=1 crypto rx=0x7f9b9400cbf0 tx=0x7f9b9400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:25.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.438+0000 7f9b92a00640 1 -- 172.21.15.28:0/3885650467 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b94017070 con 0x7f9b9c150450 2024-02-21T23:13:25.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.438+0000 7f9b92a00640 1 -- 172.21.15.28:0/3885650467 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b9400ad40 con 0x7f9b9c150450 2024-02-21T23:13:25.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.438+0000 7f9b92a00640 1 -- 172.21.15.28:0/3885650467 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b94012600 con 0x7f9b9c150450 2024-02-21T23:13:25.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.438+0000 7f9b9b200640 1 -- 172.21.15.28:0/3885650467 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b9c072660 con 0x7f9b9c150450 2024-02-21T23:13:25.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.438+0000 7f9b9b200640 1 -- 172.21.15.28:0/3885650467 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b9c1edf10 con 0x7f9b9c150450 2024-02-21T23:13:25.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.439+0000 7f9b9b200640 1 -- 172.21.15.28:0/3885650467 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b9c04f290 con 0x7f9b9c150450 2024-02-21T23:13:25.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.440+0000 7f9b92a00640 1 -- 172.21.15.28:0/3885650467 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9b9401a070 con 0x7f9b9c150450 2024-02-21T23:13:25.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.441+0000 7f9b92a00640 1 --2- 172.21.15.28:0/3885650467 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9b7806b4d0 0x7f9b7806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:25.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.441+0000 7f9b92a00640 1 -- 172.21.15.28:0/3885650467 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(204..204 src has 1..204) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f9b94089d00 con 0x7f9b9c150450 2024-02-21T23:13:25.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.441+0000 7f9b99e00640 1 --2- 172.21.15.28:0/3885650467 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9b7806b4d0 0x7f9b7806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:25.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.443+0000 7f9b99e00640 1 --2- 172.21.15.28:0/3885650467 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9b7806b4d0 0x7f9b7806d990 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c006fd0 tx=0x7f9b8c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:25.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.445+0000 7f9b92a00640 1 -- 172.21.15.28:0/3885650467 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f9b940586e0 con 0x7f9b9c150450 2024-02-21T23:13:25.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.706+0000 7f9b9b200640 1 -- 172.21.15.28:0/3885650467 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f9b9c06e010 con 0x7f9b9c150450 2024-02-21T23:13:25.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.707+0000 7f9b92a00640 1 -- 172.21.15.28:0/3885650467 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v22) v1 ==== 80+0+4724 (secure 0 0 0) 0x7f9b9405c780 con 0x7f9b9c150450 2024-02-21T23:13:25.694 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:13:25.695 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-21T23:13:25.695 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_dump": [ 2024-02-21T23:13:25.695 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:13:25.695 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.a", 2024-02-21T23:13:25.695 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDSgNZl4FY1KRAAJSK5Ibb0ibKiKXYpTLgtmQ==", 2024-02-21T23:13:25.695 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:13:25.695 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:13:25.695 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-02-21T23:13:25.695 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-02-21T23:13:25.695 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:13:25.695 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:13:25.695 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:13:25.695 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:13:25.696 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.b", 2024-02-21T23:13:25.696 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDTgNZlwRcAERAAgsYmmkuobKelBEJVCRpXUg==", 2024-02-21T23:13:25.696 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:13:25.696 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:13:25.696 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-02-21T23:13:25.696 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-02-21T23:13:25.696 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:13:25.696 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:13:25.696 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:13:25.697 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:13:25.697 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.c", 2024-02-21T23:13:25.697 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDSgNZloFpQOxAAK2WBArXRVNTgqH9WWDEPDA==", 2024-02-21T23:13:25.697 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:13:25.697 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:13:25.697 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-02-21T23:13:25.697 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-02-21T23:13:25.697 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:13:25.697 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:13:25.697 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:13:25.697 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:13:25.697 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.d", 2024-02-21T23:13:25.697 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDTgNZlblOmIRAAxe2HfSKGSkkzJDN4aVdzgQ==", 2024-02-21T23:13:25.697 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:13:25.698 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:13:25.698 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-02-21T23:13:25.698 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-02-21T23:13:25.698 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:13:25.698 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:13:25.698 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:13:25.698 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:13:25.698 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.0", 2024-02-21T23:13:25.698 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDVgNZlxNJbLxAAXot7Q3chrLEBsFNO8kT6QQ==", 2024-02-21T23:13:25.698 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:13:25.698 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:13:25.698 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:13:25.698 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:13:25.698 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:13:25.698 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:13:25.699 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:13:25.699 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.1", 2024-02-21T23:13:25.699 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDYgNZlZZfWORAAr4XkOxLYLmumVbu362N2HQ==", 2024-02-21T23:13:25.699 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:13:25.699 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:13:25.699 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:13:25.699 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:13:25.699 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:13:25.699 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:13:25.699 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:13:25.699 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.2", 2024-02-21T23:13:25.699 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDcgNZl9mcDGBAAIZGOF8IUMenwo386x95mwg==", 2024-02-21T23:13:25.699 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:13:25.699 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:13:25.699 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:13:25.700 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:13:25.700 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:13:25.700 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:13:25.700 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:13:25.700 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.3", 2024-02-21T23:13:25.700 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDfgNZl8ALSMBAAag9cmGH0/4xALWSUmrZowg==", 2024-02-21T23:13:25.700 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:13:25.700 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:13:25.700 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:13:25.700 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:13:25.700 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:13:25.700 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:13:25.700 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:13:25.701 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.4", 2024-02-21T23:13:25.701 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDngNZlBgeKABAAeW99RVfCBvB1lp7w7mQBlQ==", 2024-02-21T23:13:25.701 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:13:25.701 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:13:25.701 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:13:25.701 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:13:25.701 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:13:25.701 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:13:25.701 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:13:25.701 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.5", 2024-02-21T23:13:25.701 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDqgNZljQG3ChAAY/IZEIcvcGNvlFGIWib3gg==", 2024-02-21T23:13:25.701 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:13:25.701 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:13:25.701 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:13:25.701 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:13:25.702 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:13:25.702 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:13:25.702 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:13:25.702 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.6", 2024-02-21T23:13:25.702 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDtgNZlKnakIxAAkW5xlkshzvEHJdmpgPxR+g==", 2024-02-21T23:13:25.702 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:13:25.702 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:13:25.702 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:13:25.702 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:13:25.702 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:13:25.702 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:13:25.702 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:13:25.702 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.7", 2024-02-21T23:13:25.702 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDxgNZlTWxYARAASX1/gmqO+8mQPP8uA4xjiQ==", 2024-02-21T23:13:25.702 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:13:25.703 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-02-21T23:13:25.703 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-02-21T23:13:25.703 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:13:25.703 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:13:25.703 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:13:25.703 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:13:25.703 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.0", 2024-02-21T23:13:25.703 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDTgNZlILvYLxAAoREWxKKT7YcXnhkgk8YvoQ==", 2024-02-21T23:13:25.703 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:13:25.703 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:13:25.703 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow r", 2024-02-21T23:13:25.703 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-02-21T23:13:25.703 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:13:25.704 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:13:25.704 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:13:25.704 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.1", 2024-02-21T23:13:25.704 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDTgNZlm7ImOxAA8iRcXlzFPZPaXQfsKhuO/g==", 2024-02-21T23:13:25.704 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:13:25.704 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-02-21T23:13:25.704 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-02-21T23:13:25.705 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-02-21T23:13:25.705 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-02-21T23:13:25.705 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:13:25.705 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:13:25.705 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:13:25.705 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.admin", 2024-02-21T23:13:25.706 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDRgNZlBYd1GhAAbK5qBCIkf0Ej5nEyPnTxIQ==", 2024-02-21T23:13:25.706 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:13:25.706 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-02-21T23:13:25.706 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-02-21T23:13:25.706 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-02-21T23:13:25.706 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:13:25.706 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:13:25.706 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:13:25.707 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:13:25.707 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.x", 2024-02-21T23:13:25.707 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDSgNZlJYKeHBAAYPskmLRr7Jw44TdKzM/JMg==", 2024-02-21T23:13:25.707 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:13:25.707 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-02-21T23:13:25.707 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-02-21T23:13:25.707 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:13:25.707 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:13:25.707 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-21T23:13:25.707 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:13:25.707 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.y", 2024-02-21T23:13:25.707 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDSgNZlSRkOEBAAOhRj9PsZDpjoV6WT4drrZA==", 2024-02-21T23:13:25.707 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-02-21T23:13:25.707 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-02-21T23:13:25.708 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-02-21T23:13:25.708 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-02-21T23:13:25.708 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:13:25.708 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:13:25.708 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-21T23:13:25.708 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-21T23:13:25.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.709+0000 7f9b9b200640 1 -- 172.21.15.28:0/3885650467 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9b7806b4d0 msgr2=0x7f9b7806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:25.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.709+0000 7f9b9b200640 1 --2- 172.21.15.28:0/3885650467 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9b7806b4d0 0x7f9b7806d990 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c006fd0 tx=0x7f9b8c008040 comp rx=0 tx=0).stop 2024-02-21T23:13:25.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.709+0000 7f9b9b200640 1 -- 172.21.15.28:0/3885650467 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b9c150450 msgr2=0x7f9b9c1ee430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:25.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.709+0000 7f9b9b200640 1 --2- 172.21.15.28:0/3885650467 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b9c150450 0x7f9b9c1ee430 secure :-1 s=READY pgs=2343 cs=0 l=1 rev1=1 crypto rx=0x7f9b9400cbf0 tx=0x7f9b9400a430 comp rx=0 tx=0).stop 2024-02-21T23:13:25.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.709+0000 7f9b9b200640 1 -- 172.21.15.28:0/3885650467 shutdown_connections 2024-02-21T23:13:25.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.709+0000 7f9b9b200640 1 --2- 172.21.15.28:0/3885650467 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9b7806b4d0 0x7f9b7806d990 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:25.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.709+0000 7f9b9b200640 1 --2- 172.21.15.28:0/3885650467 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b9c150450 0x7f9b9c1ee430 unknown :-1 s=CLOSED pgs=2343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:25.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.709+0000 7f9b9b200640 1 --2- 172.21.15.28:0/3885650467 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9b9c0c2bb0 0x7f9b9c0bfba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:25.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.709+0000 7f9b9b200640 1 --2- 172.21.15.28:0/3885650467 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9b9c06e550 0x7f9b9c1ea750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:25.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.709+0000 7f9b9b200640 1 -- 172.21.15.28:0/3885650467 >> 172.21.15.28:0/3885650467 conn(0x7f9b9c06b550 msgr2=0x7f9b9c14dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:25.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.709+0000 7f9b9b200640 1 -- 172.21.15.28:0/3885650467 shutdown_connections 2024-02-21T23:13:25.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.709+0000 7f9b9b200640 1 -- 172.21.15.28:0/3885650467 wait complete. 2024-02-21T23:13:25.716 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:13:25.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.941+0000 7f3f47e00640 1 Processor -- start 2024-02-21T23:13:25.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.941+0000 7f3f47e00640 1 -- start start 2024-02-21T23:13:25.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.943+0000 7f3f47e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3f48155960 0x7f3f481608a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:25.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.943+0000 7f3f47e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f480c74c0 con 0x7f3f4806dbf0 2024-02-21T23:13:25.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.943+0000 7f3f47e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f480c77a0 con 0x7f3f480aff90 2024-02-21T23:13:25.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.943+0000 7f3f47e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f480c7270 con 0x7f3f48155960 2024-02-21T23:13:25.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.943+0000 7f3f46000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f3f4806dbf0 0x7f3f48158ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59534/0 (socket says 172.21.15.28:59534) 2024-02-21T23:13:25.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.943+0000 7f3f46000640 1 -- 172.21.15.28:0/2472884946 learned_addr learned my addr 172.21.15.28:0/2472884946 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:13:25.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.944+0000 7f3f47400640 1 --2- 172.21.15.28:0/2472884946 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3f48155960 0x7f3f481608a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:25.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.944+0000 7f3f45600640 1 -- 172.21.15.28:0/2472884946 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1251605751 0 0) 0x7f3f480c74c0 con 0x7f3f4806dbf0 2024-02-21T23:13:25.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.944+0000 7f3f45600640 1 -- 172.21.15.28:0/2472884946 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f2c003520 con 0x7f3f4806dbf0 2024-02-21T23:13:25.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.944+0000 7f3f45600640 1 -- 172.21.15.28:0/2472884946 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3098865712 0 0) 0x7f3f2c003520 con 0x7f3f4806dbf0 2024-02-21T23:13:25.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.944+0000 7f3f45600640 1 -- 172.21.15.28:0/2472884946 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3f48155960 msgr2=0x7f3f481608a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:25.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.944+0000 7f3f45600640 1 --2- 172.21.15.28:0/2472884946 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3f48155960 0x7f3f481608a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:25.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.944+0000 7f3f45600640 1 -- 172.21.15.28:0/2472884946 >> v1:172.21.15.81:6789/0 conn(0x7f3f480aff90 legacy=0x7f3f480c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:25.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.944+0000 7f3f45600640 1 -- 172.21.15.28:0/2472884946 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f480c7a80 con 0x7f3f4806dbf0 2024-02-21T23:13:25.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.944+0000 7f3f45600640 1 -- 172.21.15.28:0/2472884946 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f3f34001670 con 0x7f3f4806dbf0 2024-02-21T23:13:25.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.944+0000 7f3f45600640 1 -- 172.21.15.28:0/2472884946 >> v1:172.21.15.28:6789/0 conn(0x7f3f4806dbf0 legacy=0x7f3f48158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:25.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.945+0000 7f3f45600640 1 --2- 172.21.15.28:0/2472884946 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3f48155960 0x7f3f2c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:25.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.945+0000 7f3f45600640 1 --2- 172.21.15.28:0/2472884946 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f2c003f20 0x7f3f2c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:25.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.945+0000 7f3f45600640 1 --2- 172.21.15.28:0/2472884946 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f2c004d40 0x7f3f2c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:25.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.945+0000 7f3f45600640 1 -- 172.21.15.28:0/2472884946 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f480c7270 con 0x7f3f2c003f20 2024-02-21T23:13:25.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.945+0000 7f3f45600640 1 -- 172.21.15.28:0/2472884946 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f2c00ba20 con 0x7f3f2c004d40 2024-02-21T23:13:25.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.945+0000 7f3f45600640 1 -- 172.21.15.28:0/2472884946 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f2c00bc20 con 0x7f3f48155960 2024-02-21T23:13:25.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.945+0000 7f3f46000640 1 --2- 172.21.15.28:0/2472884946 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f2c003f20 0x7f3f2c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:25.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.945+0000 7f3f47400640 1 --2- 172.21.15.28:0/2472884946 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f2c004d40 0x7f3f2c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:25.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.945+0000 7f3f46a00640 1 --2- 172.21.15.28:0/2472884946 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3f48155960 0x7f3f2c0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:25.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.945+0000 7f3f46000640 1 -- 172.21.15.28:0/2472884946 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3f48155960 msgr2=0x7f3f2c0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:25.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.945+0000 7f3f46000640 1 --2- 172.21.15.28:0/2472884946 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3f48155960 0x7f3f2c0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:25.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.945+0000 7f3f46000640 1 -- 172.21.15.28:0/2472884946 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f2c004d40 msgr2=0x7f3f2c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:25.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.945+0000 7f3f46000640 1 --2- 172.21.15.28:0/2472884946 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f2c004d40 0x7f3f2c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:25.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.945+0000 7f3f46000640 1 -- 172.21.15.28:0/2472884946 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f2c003520 con 0x7f3f2c003f20 2024-02-21T23:13:25.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.945+0000 7f3f46a00640 1 --2- 172.21.15.28:0/2472884946 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3f48155960 0x7f3f2c0039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:13:25.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.947+0000 7f3f46000640 1 --2- 172.21.15.28:0/2472884946 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f2c003f20 0x7f3f2c004800 secure :-1 s=READY pgs=2345 cs=0 l=1 rev1=1 crypto rx=0x7f3f3400c160 tx=0x7f3f2c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f199c4d2862e664e server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:25.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.947+0000 7f3f45600640 1 -- 172.21.15.28:0/2472884946 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f34007cc0 con 0x7f3f2c003f20 2024-02-21T23:13:25.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.947+0000 7f3f45600640 1 -- 172.21.15.28:0/2472884946 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f34030070 con 0x7f3f2c003f20 2024-02-21T23:13:25.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.947+0000 7f3f45600640 1 -- 172.21.15.28:0/2472884946 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f34005d50 con 0x7f3f2c003f20 2024-02-21T23:13:25.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.947+0000 7f3f47e00640 1 -- 172.21.15.28:0/2472884946 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f2c003f20 msgr2=0x7f3f2c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:25.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.947+0000 7f3f47e00640 1 --2- 172.21.15.28:0/2472884946 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f2c003f20 0x7f3f2c004800 secure :-1 s=READY pgs=2345 cs=0 l=1 rev1=1 crypto rx=0x7f3f3400c160 tx=0x7f3f2c005830 comp rx=0 tx=0).stop 2024-02-21T23:13:25.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.947+0000 7f3f47400640 1 -- 172.21.15.28:0/2472884946 reap_dead start 2024-02-21T23:13:25.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.948+0000 7f3f47e00640 1 -- 172.21.15.28:0/2472884946 shutdown_connections 2024-02-21T23:13:25.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.948+0000 7f3f47e00640 1 -- 172.21.15.28:0/2472884946 >> 172.21.15.28:0/2472884946 conn(0x7f3f480ac370 msgr2=0x7f3f480ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:25.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.948+0000 7f3f47e00640 1 -- 172.21.15.28:0/2472884946 shutdown_connections 2024-02-21T23:13:25.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.948+0000 7f3f47e00640 1 -- 172.21.15.28:0/2472884946 wait complete. 2024-02-21T23:13:25.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.949+0000 7f3f47e00640 1 Processor -- start 2024-02-21T23:13:25.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.949+0000 7f3f47e00640 1 -- start start 2024-02-21T23:13:25.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.950+0000 7f3f47e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3f2c003f20 0x7f3f480c1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:25.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.950+0000 7f3f47e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f481eaf10 0x7f3f48151300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:25.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.950+0000 7f3f47e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f48153720 0x7f3f4814ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:25.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.950+0000 7f3f47e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f480c58f0 con 0x7f3f48153720 2024-02-21T23:13:25.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.950+0000 7f3f47e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f48158630 con 0x7f3f481eaf10 2024-02-21T23:13:25.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.950+0000 7f3f47e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f480c5a70 con 0x7f3f2c003f20 2024-02-21T23:13:25.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.951+0000 7f3f47400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f48153720 0x7f3f4814ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:25.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.951+0000 7f3f47400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f48153720 0x7f3f4814ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33576/0 (socket says 172.21.15.28:33576) 2024-02-21T23:13:25.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.951+0000 7f3f47400640 1 -- 172.21.15.28:0/2757137494 learned_addr learned my addr 172.21.15.28:0/2757137494 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:25.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.951+0000 7f3f46a00640 1 --2- 172.21.15.28:0/2757137494 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3f2c003f20 0x7f3f480c1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:25.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.951+0000 7f3f47400640 1 -- 172.21.15.28:0/2757137494 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3f2c003f20 msgr2=0x7f3f480c1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:25.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.951+0000 7f3f46000640 1 --2- 172.21.15.28:0/2757137494 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f481eaf10 0x7f3f48151300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:25.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.951+0000 7f3f47400640 1 --2- 172.21.15.28:0/2757137494 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3f2c003f20 0x7f3f480c1930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:25.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.951+0000 7f3f47400640 1 -- 172.21.15.28:0/2757137494 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f481eaf10 msgr2=0x7f3f48151300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:25.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.951+0000 7f3f47400640 1 --2- 172.21.15.28:0/2757137494 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f481eaf10 0x7f3f48151300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:25.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.951+0000 7f3f47400640 1 -- 172.21.15.28:0/2757137494 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f48154930 con 0x7f3f48153720 2024-02-21T23:13:25.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.951+0000 7f3f47400640 1 --2- 172.21.15.28:0/2757137494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f48153720 0x7f3f4814ede0 secure :-1 s=READY pgs=2346 cs=0 l=1 rev1=1 crypto rx=0x7f3f480c83e0 tx=0x7f3f400115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:25.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.952+0000 7f3f3ea00640 1 -- 172.21.15.28:0/2757137494 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f40011e40 con 0x7f3f48153720 2024-02-21T23:13:25.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.952+0000 7f3f3ea00640 1 -- 172.21.15.28:0/2757137494 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f4001b070 con 0x7f3f48153720 2024-02-21T23:13:25.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.952+0000 7f3f3ea00640 1 -- 172.21.15.28:0/2757137494 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f40003990 con 0x7f3f48153720 2024-02-21T23:13:25.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.952+0000 7f3f47e00640 1 -- 172.21.15.28:0/2757137494 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f48154bd0 con 0x7f3f48153720 2024-02-21T23:13:25.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.952+0000 7f3f47e00640 1 -- 172.21.15.28:0/2757137494 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f48151840 con 0x7f3f48153720 2024-02-21T23:13:25.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.953+0000 7f3f47e00640 1 -- 172.21.15.28:0/2757137494 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f4804f290 con 0x7f3f48153720 2024-02-21T23:13:25.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.954+0000 7f3f3ea00640 1 -- 172.21.15.28:0/2757137494 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3f40005d10 con 0x7f3f48153720 2024-02-21T23:13:25.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.955+0000 7f3f3ea00640 1 --2- 172.21.15.28:0/2757137494 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3f2806b500 0x7f3f2806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:25.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.955+0000 7f3f3ea00640 1 -- 172.21.15.28:0/2757137494 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(204..204 src has 1..204) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f3f4008a500 con 0x7f3f48153720 2024-02-21T23:13:25.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.956+0000 7f3f46a00640 1 --2- 172.21.15.28:0/2757137494 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3f2806b500 0x7f3f2806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:25.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.957+0000 7f3f46a00640 1 --2- 172.21.15.28:0/2757137494 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3f2806b500 0x7f3f2806d9c0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f3f38000f60 tx=0x7f3f38009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:25.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:25.960+0000 7f3f3ea00640 1 -- 172.21.15.28:0/2757137494 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3f40025020 con 0x7f3f48153720 2024-02-21T23:13:26.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.160+0000 7f3f47e00640 1 -- 172.21.15.28:0/2757137494 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3f4806d9d0 con 0x7f3f48153720 2024-02-21T23:13:26.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.162+0000 7f3f3ea00640 1 -- 172.21.15.28:0/2757137494 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v204) v1 ==== 74+0+13114 (secure 0 0 0) 0x7f3f400595d0 con 0x7f3f48153720 2024-02-21T23:13:26.150 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:13:26.150 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":204,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:13:25.178386+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":32,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":195,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:13:26.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.165+0000 7f3f47e00640 1 -- 172.21.15.28:0/2757137494 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3f2806b500 msgr2=0x7f3f2806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:26.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.165+0000 7f3f47e00640 1 --2- 172.21.15.28:0/2757137494 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3f2806b500 0x7f3f2806d9c0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f3f38000f60 tx=0x7f3f38009040 comp rx=0 tx=0).stop 2024-02-21T23:13:26.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.165+0000 7f3f47e00640 1 -- 172.21.15.28:0/2757137494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f48153720 msgr2=0x7f3f4814ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:26.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.165+0000 7f3f47e00640 1 --2- 172.21.15.28:0/2757137494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f48153720 0x7f3f4814ede0 secure :-1 s=READY pgs=2346 cs=0 l=1 rev1=1 crypto rx=0x7f3f480c83e0 tx=0x7f3f400115a0 comp rx=0 tx=0).stop 2024-02-21T23:13:26.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.165+0000 7f3f47e00640 1 -- 172.21.15.28:0/2757137494 shutdown_connections 2024-02-21T23:13:26.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.165+0000 7f3f47e00640 1 --2- 172.21.15.28:0/2757137494 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3f2806b500 0x7f3f2806d9c0 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:26.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.165+0000 7f3f47e00640 1 --2- 172.21.15.28:0/2757137494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f48153720 0x7f3f4814ede0 unknown :-1 s=CLOSED pgs=2346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:26.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.165+0000 7f3f47e00640 1 --2- 172.21.15.28:0/2757137494 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3f481eaf10 0x7f3f48151300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:26.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.165+0000 7f3f47e00640 1 --2- 172.21.15.28:0/2757137494 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3f2c003f20 0x7f3f480c1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:26.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.165+0000 7f3f47e00640 1 -- 172.21.15.28:0/2757137494 >> 172.21.15.28:0/2757137494 conn(0x7f3f480ac370 msgr2=0x7f3f48157d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:26.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.165+0000 7f3f47e00640 1 -- 172.21.15.28:0/2757137494 shutdown_connections 2024-02-21T23:13:26.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.165+0000 7f3f47e00640 1 -- 172.21.15.28:0/2757137494 wait complete. 2024-02-21T23:13:26.170 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-21T23:02:30.728329+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-21T23:13:26.171 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-21T23:13:26.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.413+0000 7ffbae600640 1 Processor -- start 2024-02-21T23:13:26.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.413+0000 7ffbae600640 1 -- start start 2024-02-21T23:13:26.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.415+0000 7ffbae600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffba806dbc0 0x7ffba806c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:26.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.415+0000 7ffbae600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffba8158550 con 0x7ffba813e520 2024-02-21T23:13:26.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.415+0000 7ffbae600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffba81587d0 con 0x7ffba813e0c0 2024-02-21T23:13:26.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.415+0000 7ffbae600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffba80c89a0 con 0x7ffba806dbc0 2024-02-21T23:13:26.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.415+0000 7ffbadc00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ffba813e520 0x7ffba806c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59538/0 (socket says 172.21.15.28:59538) 2024-02-21T23:13:26.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.415+0000 7ffbadc00640 1 -- 172.21.15.28:0/1901886057 learned_addr learned my addr 172.21.15.28:0/1901886057 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:13:26.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.416+0000 7ffba7e00640 1 --2- 172.21.15.28:0/1901886057 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffba806dbc0 0x7ffba806c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:26.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.416+0000 7ffba7400640 1 -- 172.21.15.28:0/1901886057 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2990176600 0 0) 0x7ffba0001490 con 0x7ffba813e520 2024-02-21T23:13:26.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.416+0000 7ffba7400640 1 -- 172.21.15.28:0/1901886057 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb90003520 con 0x7ffba813e520 2024-02-21T23:13:26.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.416+0000 7ffba7400640 1 -- 172.21.15.28:0/1901886057 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2280780526 0 0) 0x7ffb90003520 con 0x7ffba813e520 2024-02-21T23:13:26.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.416+0000 7ffba7400640 1 -- 172.21.15.28:0/1901886057 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffba806dbc0 msgr2=0x7ffba806c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:26.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.416+0000 7ffba7400640 1 --2- 172.21.15.28:0/1901886057 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffba806dbc0 0x7ffba806c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:26.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.416+0000 7ffba7400640 1 -- 172.21.15.28:0/1901886057 >> v1:172.21.15.81:6789/0 conn(0x7ffba813e0c0 legacy=0x7ffba80c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:26.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.416+0000 7ffba7400640 1 -- 172.21.15.28:0/1901886057 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffba8158ab0 con 0x7ffba813e520 2024-02-21T23:13:26.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.416+0000 7ffba7400640 1 -- 172.21.15.28:0/1901886057 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7ffba0001bc0 con 0x7ffba813e520 2024-02-21T23:13:26.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.416+0000 7ffba7400640 1 -- 172.21.15.28:0/1901886057 >> v1:172.21.15.28:6789/0 conn(0x7ffba813e520 legacy=0x7ffba806c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:26.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.416+0000 7ffba7400640 1 --2- 172.21.15.28:0/1901886057 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb900039e0 0x7ffb90003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:26.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.416+0000 7ffba7400640 1 --2- 172.21.15.28:0/1901886057 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffba806dbc0 0x7ffb90004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:26.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.416+0000 7ffba7400640 1 --2- 172.21.15.28:0/1901886057 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb90004d40 0x7ffb900051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:26.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.416+0000 7ffba7400640 1 -- 172.21.15.28:0/1901886057 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb9000ba70 con 0x7ffb90004d40 2024-02-21T23:13:26.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.416+0000 7ffba7400640 1 -- 172.21.15.28:0/1901886057 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb9000bbf0 con 0x7ffb900039e0 2024-02-21T23:13:26.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.416+0000 7ffba7400640 1 -- 172.21.15.28:0/1901886057 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb9000bdf0 con 0x7ffba806dbc0 2024-02-21T23:13:26.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.417+0000 7ffbadc00640 1 --2- 172.21.15.28:0/1901886057 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb90004d40 0x7ffb900051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:26.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.417+0000 7ffba7e00640 1 --2- 172.21.15.28:0/1901886057 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffba806dbc0 0x7ffb90004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:26.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.417+0000 7ffbad200640 1 --2- 172.21.15.28:0/1901886057 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb900039e0 0x7ffb90003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:26.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.417+0000 7ffbadc00640 1 -- 172.21.15.28:0/1901886057 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffba806dbc0 msgr2=0x7ffb90004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:26.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.417+0000 7ffbadc00640 1 --2- 172.21.15.28:0/1901886057 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffba806dbc0 0x7ffb90004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:26.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.417+0000 7ffbadc00640 1 -- 172.21.15.28:0/1901886057 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb900039e0 msgr2=0x7ffb90003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:26.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.417+0000 7ffbadc00640 1 --2- 172.21.15.28:0/1901886057 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb900039e0 0x7ffb90003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:26.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.417+0000 7ffbadc00640 1 -- 172.21.15.28:0/1901886057 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb90003520 con 0x7ffb90004d40 2024-02-21T23:13:26.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.418+0000 7ffbadc00640 1 --2- 172.21.15.28:0/1901886057 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb90004d40 0x7ffb900051c0 secure :-1 s=READY pgs=2348 cs=0 l=1 rev1=1 crypto rx=0x7ffba000aa70 tx=0x7ffb90005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82f4a2c18a02624e server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:26.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.419+0000 7ffba7400640 1 -- 172.21.15.28:0/1901886057 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffba0038070 con 0x7ffb90004d40 2024-02-21T23:13:26.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.419+0000 7ffba7400640 1 -- 172.21.15.28:0/1901886057 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffba00072d0 con 0x7ffb90004d40 2024-02-21T23:13:26.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.419+0000 7ffba7400640 1 -- 172.21.15.28:0/1901886057 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffba000a690 con 0x7ffb90004d40 2024-02-21T23:13:26.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.419+0000 7ffbae600640 1 -- 172.21.15.28:0/1901886057 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb90004d40 msgr2=0x7ffb900051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:26.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.419+0000 7ffbae600640 1 --2- 172.21.15.28:0/1901886057 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb90004d40 0x7ffb900051c0 secure :-1 s=READY pgs=2348 cs=0 l=1 rev1=1 crypto rx=0x7ffba000aa70 tx=0x7ffb90005830 comp rx=0 tx=0).stop 2024-02-21T23:13:26.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.419+0000 7ffbadc00640 1 -- 172.21.15.28:0/1901886057 reap_dead start 2024-02-21T23:13:26.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.419+0000 7ffbae600640 1 -- 172.21.15.28:0/1901886057 shutdown_connections 2024-02-21T23:13:26.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.419+0000 7ffbae600640 1 -- 172.21.15.28:0/1901886057 >> 172.21.15.28:0/1901886057 conn(0x7ffba8073110 msgr2=0x7ffba80ac490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:26.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.419+0000 7ffbae600640 1 -- 172.21.15.28:0/1901886057 shutdown_connections 2024-02-21T23:13:26.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.419+0000 7ffbae600640 1 -- 172.21.15.28:0/1901886057 wait complete. 2024-02-21T23:13:26.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.421+0000 7ffbae600640 1 Processor -- start 2024-02-21T23:13:26.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.421+0000 7ffbae600640 1 -- start start 2024-02-21T23:13:26.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.422+0000 7ffbae600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb90004d40 0x7ffba8150220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:26.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.422+0000 7ffbae600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffba81576c0 0x7ffba80aa5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:26.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.422+0000 7ffbae600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffba80aab30 0x7ffba80aafc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:26.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.422+0000 7ffbae600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffba806ced0 con 0x7ffb90004d40 2024-02-21T23:13:26.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.422+0000 7ffbae600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffba8159070 con 0x7ffba81576c0 2024-02-21T23:13:26.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.422+0000 7ffbae600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffba80c78c0 con 0x7ffba80aab30 2024-02-21T23:13:26.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.422+0000 7ffbad200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb90004d40 0x7ffba8150220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:26.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.422+0000 7ffbad200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb90004d40 0x7ffba8150220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33586/0 (socket says 172.21.15.28:33586) 2024-02-21T23:13:26.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.422+0000 7ffbad200640 1 -- 172.21.15.28:0/1215621318 learned_addr learned my addr 172.21.15.28:0/1215621318 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:26.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.423+0000 7ffba7e00640 1 --2- 172.21.15.28:0/1215621318 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffba81576c0 0x7ffba80aa5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:26.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.423+0000 7ffbadc00640 1 --2- 172.21.15.28:0/1215621318 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffba80aab30 0x7ffba80aafc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:26.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.423+0000 7ffbad200640 1 -- 172.21.15.28:0/1215621318 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffba80aab30 msgr2=0x7ffba80aafc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:26.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.423+0000 7ffbad200640 1 --2- 172.21.15.28:0/1215621318 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffba80aab30 0x7ffba80aafc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:26.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.423+0000 7ffbad200640 1 -- 172.21.15.28:0/1215621318 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffba81576c0 msgr2=0x7ffba80aa5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:26.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.423+0000 7ffbad200640 1 --2- 172.21.15.28:0/1215621318 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffba81576c0 0x7ffba80aa5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:26.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.423+0000 7ffbad200640 1 -- 172.21.15.28:0/1215621318 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffba81ee4e0 con 0x7ffb90004d40 2024-02-21T23:13:26.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.423+0000 7ffbad200640 1 --2- 172.21.15.28:0/1215621318 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb90004d40 0x7ffba8150220 secure :-1 s=READY pgs=2349 cs=0 l=1 rev1=1 crypto rx=0x7ffb9c00e7d0 tx=0x7ffb9c00ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:26.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.423+0000 7ffba4c00640 1 -- 172.21.15.28:0/1215621318 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb9c0050d0 con 0x7ffb90004d40 2024-02-21T23:13:26.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.423+0000 7ffba4c00640 1 -- 172.21.15.28:0/1215621318 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb9c018070 con 0x7ffb90004d40 2024-02-21T23:13:26.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.423+0000 7ffbae600640 1 -- 172.21.15.28:0/1215621318 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffba81ee7c0 con 0x7ffb90004d40 2024-02-21T23:13:26.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.424+0000 7ffba4c00640 1 -- 172.21.15.28:0/1215621318 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb9c00c6c0 con 0x7ffb90004d40 2024-02-21T23:13:26.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.424+0000 7ffbae600640 1 -- 172.21.15.28:0/1215621318 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffba8150760 con 0x7ffb90004d40 2024-02-21T23:13:26.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.425+0000 7ffbae600640 1 -- 172.21.15.28:0/1215621318 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffba81ee9e0 con 0x7ffb90004d40 2024-02-21T23:13:26.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.426+0000 7ffba4c00640 1 -- 172.21.15.28:0/1215621318 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffb9c00c860 con 0x7ffb90004d40 2024-02-21T23:13:26.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.426+0000 7ffba4c00640 1 --2- 172.21.15.28:0/1215621318 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ffb8806b4d0 0x7ffb8806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:26.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.427+0000 7ffba4c00640 1 -- 172.21.15.28:0/1215621318 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(204..204 src has 1..204) v4 ==== 5249+0+0 (secure 0 0 0) 0x7ffb9c089f10 con 0x7ffb90004d40 2024-02-21T23:13:26.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.427+0000 7ffba7e00640 1 --2- 172.21.15.28:0/1215621318 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ffb8806b4d0 0x7ffb8806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:26.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.428+0000 7ffba7e00640 1 --2- 172.21.15.28:0/1215621318 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ffb8806b4d0 0x7ffb8806d990 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7ffb94009a10 tx=0x7ffb94008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:26.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.431+0000 7ffba4c00640 1 -- 172.21.15.28:0/1215621318 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ffba81ee9e0 con 0x7ffb90004d40 2024-02-21T23:13:26.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.630+0000 7ffbae600640 1 -- 172.21.15.28:0/1215621318 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ffba813c400 con 0x7ffb90004d40 2024-02-21T23:13:26.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.630+0000 7ffba4c00640 1 -- 172.21.15.28:0/1215621318 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v204) v1 ==== 93+0+10 (secure 0 0 0) 0x7ffba813c400 con 0x7ffb90004d40 2024-02-21T23:13:26.618 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-02-21T23:13:26.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.633+0000 7ffbae600640 1 -- 172.21.15.28:0/1215621318 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ffb8806b4d0 msgr2=0x7ffb8806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:26.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.633+0000 7ffbae600640 1 --2- 172.21.15.28:0/1215621318 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ffb8806b4d0 0x7ffb8806d990 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7ffb94009a10 tx=0x7ffb94008040 comp rx=0 tx=0).stop 2024-02-21T23:13:26.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.633+0000 7ffbae600640 1 -- 172.21.15.28:0/1215621318 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb90004d40 msgr2=0x7ffba8150220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:26.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.633+0000 7ffbae600640 1 --2- 172.21.15.28:0/1215621318 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb90004d40 0x7ffba8150220 secure :-1 s=READY pgs=2349 cs=0 l=1 rev1=1 crypto rx=0x7ffb9c00e7d0 tx=0x7ffb9c00ec90 comp rx=0 tx=0).stop 2024-02-21T23:13:26.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.633+0000 7ffbae600640 1 -- 172.21.15.28:0/1215621318 shutdown_connections 2024-02-21T23:13:26.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.633+0000 7ffbae600640 1 --2- 172.21.15.28:0/1215621318 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ffb8806b4d0 0x7ffb8806d990 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:26.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.633+0000 7ffbae600640 1 --2- 172.21.15.28:0/1215621318 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffba80aab30 0x7ffba80aafc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:26.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.633+0000 7ffbae600640 1 --2- 172.21.15.28:0/1215621318 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffba81576c0 0x7ffba80aa5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:26.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.633+0000 7ffbae600640 1 --2- 172.21.15.28:0/1215621318 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb90004d40 0x7ffba8150220 unknown :-1 s=CLOSED pgs=2349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:26.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.633+0000 7ffbae600640 1 -- 172.21.15.28:0/1215621318 >> 172.21.15.28:0/1215621318 conn(0x7ffba8073110 msgr2=0x7ffba80ca780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:26.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.633+0000 7ffbae600640 1 -- 172.21.15.28:0/1215621318 shutdown_connections 2024-02-21T23:13:26.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.633+0000 7ffbae600640 1 -- 172.21.15.28:0/1215621318 wait complete. 2024-02-21T23:13:26.640 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-02-21T23:13:26.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.865+0000 7f6d9b600640 1 Processor -- start 2024-02-21T23:13:26.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.866+0000 7f6d9b600640 1 -- start start 2024-02-21T23:13:26.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.867+0000 7f6d9b600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6d94153950 0x7f6d940c15d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:26.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.867+0000 7f6d9b600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6d9415b6d0 0x7f6d9415dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:26.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.867+0000 7f6d9b600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d94152e70 con 0x7f6d940c11f0 2024-02-21T23:13:26.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.867+0000 7f6d9b600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d940a5ec0 con 0x7f6d94153950 2024-02-21T23:13:26.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.867+0000 7f6d9b600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d94153150 con 0x7f6d9415b6d0 2024-02-21T23:13:26.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.868+0000 7f6d93e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6d94153950 0x7f6d940c15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:26.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.868+0000 7f6d93e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6d94153950 0x7f6d940c15d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33592/0 (socket says 172.21.15.28:33592) 2024-02-21T23:13:26.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.868+0000 7f6d93e00640 1 -- 172.21.15.28:0/1112753398 learned_addr learned my addr 172.21.15.28:0/1112753398 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:26.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.868+0000 7f6d99600640 1 --2- 172.21.15.28:0/1112753398 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6d9415b6d0 0x7f6d9415dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:26.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.868+0000 7f6d93e00640 1 -- 172.21.15.28:0/1112753398 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6d9415b6d0 msgr2=0x7f6d9415dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:26.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.868+0000 7f6d93e00640 1 --2- 172.21.15.28:0/1112753398 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6d9415b6d0 0x7f6d9415dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:26.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.868+0000 7f6d93e00640 1 -- 172.21.15.28:0/1112753398 >> v1:172.21.15.81:6789/0 conn(0x7f6d940c11f0 legacy=0x7f6d9406dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:26.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.868+0000 7f6d93e00640 1 -- 172.21.15.28:0/1112753398 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d94153430 con 0x7f6d94153950 2024-02-21T23:13:26.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.870+0000 7f6d93e00640 1 --2- 172.21.15.28:0/1112753398 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d94153950 0x7f6d940c15d0 secure :-1 s=READY pgs=2350 cs=0 l=1 rev1=1 crypto rx=0x7f6d80009870 tx=0x7f6d8002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d1a35f6b3321270 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:26.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.870+0000 7f6d93400640 1 -- 172.21.15.28:0/1112753398 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d80035020 con 0x7f6d94153950 2024-02-21T23:13:26.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.870+0000 7f6d93400640 1 -- 172.21.15.28:0/1112753398 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d80031640 con 0x7f6d94153950 2024-02-21T23:13:26.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.870+0000 7f6d93400640 1 -- 172.21.15.28:0/1112753398 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d8002d040 con 0x7f6d94153950 2024-02-21T23:13:26.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.870+0000 7f6d9b600640 1 -- 172.21.15.28:0/1112753398 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d94153950 msgr2=0x7f6d940c15d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:26.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.870+0000 7f6d9b600640 1 --2- 172.21.15.28:0/1112753398 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d94153950 0x7f6d940c15d0 secure :-1 s=READY pgs=2350 cs=0 l=1 rev1=1 crypto rx=0x7f6d80009870 tx=0x7f6d8002b8f0 comp rx=0 tx=0).stop 2024-02-21T23:13:26.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.871+0000 7f6d9b600640 1 -- 172.21.15.28:0/1112753398 shutdown_connections 2024-02-21T23:13:26.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.871+0000 7f6d9b600640 1 --2- 172.21.15.28:0/1112753398 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6d9415b6d0 0x7f6d9415dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:26.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.871+0000 7f6d9b600640 1 --2- 172.21.15.28:0/1112753398 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d94153950 0x7f6d940c15d0 unknown :-1 s=CLOSED pgs=2350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:26.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.871+0000 7f6d9b600640 1 -- 172.21.15.28:0/1112753398 >> 172.21.15.28:0/1112753398 conn(0x7f6d940c67f0 msgr2=0x7f6d940c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:26.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.871+0000 7f6d9b600640 1 -- 172.21.15.28:0/1112753398 shutdown_connections 2024-02-21T23:13:26.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.871+0000 7f6d9b600640 1 -- 172.21.15.28:0/1112753398 wait complete. 2024-02-21T23:13:26.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.872+0000 7f6d9b600640 1 Processor -- start 2024-02-21T23:13:26.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.873+0000 7f6d9b600640 1 -- start start 2024-02-21T23:13:26.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.873+0000 7f6d9b600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6d940c11f0 0x7f6d940ab7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:26.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.874+0000 7f6d9b600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6d94153950 0x7f6d940abd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:26.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.874+0000 7f6d9b600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d9415b6d0 0x7f6d941ec980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:26.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.874+0000 7f6d9b600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d940a6110 con 0x7f6d9415b6d0 2024-02-21T23:13:26.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.874+0000 7f6d9b600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d940c4c00 con 0x7f6d940c11f0 2024-02-21T23:13:26.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.874+0000 7f6d9b600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d940c0d90 con 0x7f6d94153950 2024-02-21T23:13:26.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.874+0000 7f6d99600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d9415b6d0 0x7f6d941ec980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:26.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.874+0000 7f6d99600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d9415b6d0 0x7f6d941ec980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33598/0 (socket says 172.21.15.28:33598) 2024-02-21T23:13:26.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.874+0000 7f6d99600640 1 -- 172.21.15.28:0/2809332534 learned_addr learned my addr 172.21.15.28:0/2809332534 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:26.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.874+0000 7f6d93e00640 1 --2- 172.21.15.28:0/2809332534 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6d94153950 0x7f6d940abd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:26.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.874+0000 7f6d98c00640 1 --2- 172.21.15.28:0/2809332534 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6d940c11f0 0x7f6d940ab7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:26.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.874+0000 7f6d99600640 1 -- 172.21.15.28:0/2809332534 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6d94153950 msgr2=0x7f6d940abd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:26.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.874+0000 7f6d99600640 1 --2- 172.21.15.28:0/2809332534 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6d94153950 0x7f6d940abd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:26.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.874+0000 7f6d99600640 1 -- 172.21.15.28:0/2809332534 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6d940c11f0 msgr2=0x7f6d940ab7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:26.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.874+0000 7f6d99600640 1 --2- 172.21.15.28:0/2809332534 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6d940c11f0 0x7f6d940ab7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:26.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.874+0000 7f6d99600640 1 -- 172.21.15.28:0/2809332534 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d9414ef60 con 0x7f6d9415b6d0 2024-02-21T23:13:26.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.874+0000 7f6d98c00640 1 --2- 172.21.15.28:0/2809332534 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6d940c11f0 0x7f6d940ab7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:13:26.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.875+0000 7f6d99600640 1 --2- 172.21.15.28:0/2809332534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d9415b6d0 0x7f6d941ec980 secure :-1 s=READY pgs=2351 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c007e40 tx=0x7f6d8c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:26.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.875+0000 7f6d90c00640 1 -- 172.21.15.28:0/2809332534 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d8c014620 con 0x7f6d9415b6d0 2024-02-21T23:13:26.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.875+0000 7f6d90c00640 1 -- 172.21.15.28:0/2809332534 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d8c019070 con 0x7f6d9415b6d0 2024-02-21T23:13:26.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.875+0000 7f6d90c00640 1 -- 172.21.15.28:0/2809332534 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d8c014a20 con 0x7f6d9415b6d0 2024-02-21T23:13:26.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.875+0000 7f6d9b600640 1 -- 172.21.15.28:0/2809332534 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d941ed5b0 con 0x7f6d9415b6d0 2024-02-21T23:13:26.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.875+0000 7f6d9b600640 1 -- 172.21.15.28:0/2809332534 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d940a7f90 con 0x7f6d9415b6d0 2024-02-21T23:13:26.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.876+0000 7f6d9b600640 1 -- 172.21.15.28:0/2809332534 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d94067f30 con 0x7f6d9415b6d0 2024-02-21T23:13:26.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.878+0000 7f6d90c00640 1 -- 172.21.15.28:0/2809332534 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6d8c014bc0 con 0x7f6d9415b6d0 2024-02-21T23:13:26.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.878+0000 7f6d90c00640 1 --2- 172.21.15.28:0/2809332534 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6d7406b4d0 0x7f6d7406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:26.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.879+0000 7f6d90c00640 1 -- 172.21.15.28:0/2809332534 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(204..204 src has 1..204) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f6d8c08aa20 con 0x7f6d9415b6d0 2024-02-21T23:13:26.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.879+0000 7f6d98c00640 1 --2- 172.21.15.28:0/2809332534 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6d7406b4d0 0x7f6d7406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:26.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.881+0000 7f6d98c00640 1 --2- 172.21.15.28:0/2809332534 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6d7406b4d0 0x7f6d7406d990 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f6d88000f60 tx=0x7f6d88009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:26.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:26.883+0000 7f6d90c00640 1 -- 172.21.15.28:0/2809332534 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f6d8c0599c0 con 0x7f6d9415b6d0 2024-02-21T23:13:27.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.084+0000 7f6d9b600640 1 -- 172.21.15.28:0/2809332534 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f6d94150d70 con 0x7f6d9415b6d0 2024-02-21T23:13:27.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.085+0000 7f6d90c00640 1 -- 172.21.15.28:0/2809332534 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v204) v1 ==== 84+0+65 (secure 0 0 0) 0x7f6d8c0592c0 con 0x7f6d9415b6d0 2024-02-21T23:13:27.072 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:13:27.073 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-02-21T23:13:27.073 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-21T23:13:27.073 INFO:teuthology.orchestra.run.smithi028.stdout: "poolnum": 1, 2024-02-21T23:13:27.073 INFO:teuthology.orchestra.run.smithi028.stdout: "poolname": ".mgr" 2024-02-21T23:13:27.073 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-21T23:13:27.073 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-02-21T23:13:27.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.088+0000 7f6d9b600640 1 -- 172.21.15.28:0/2809332534 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6d7406b4d0 msgr2=0x7f6d7406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:27.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.088+0000 7f6d9b600640 1 --2- 172.21.15.28:0/2809332534 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6d7406b4d0 0x7f6d7406d990 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f6d88000f60 tx=0x7f6d88009040 comp rx=0 tx=0).stop 2024-02-21T23:13:27.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.088+0000 7f6d9b600640 1 -- 172.21.15.28:0/2809332534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d9415b6d0 msgr2=0x7f6d941ec980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:27.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.088+0000 7f6d9b600640 1 --2- 172.21.15.28:0/2809332534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d9415b6d0 0x7f6d941ec980 secure :-1 s=READY pgs=2351 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c007e40 tx=0x7f6d8c00e4b0 comp rx=0 tx=0).stop 2024-02-21T23:13:27.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.088+0000 7f6d9b600640 1 -- 172.21.15.28:0/2809332534 shutdown_connections 2024-02-21T23:13:27.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.088+0000 7f6d9b600640 1 --2- 172.21.15.28:0/2809332534 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6d7406b4d0 0x7f6d7406d990 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:27.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.088+0000 7f6d9b600640 1 --2- 172.21.15.28:0/2809332534 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6d94153950 0x7f6d940abd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:27.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.088+0000 7f6d9b600640 1 --2- 172.21.15.28:0/2809332534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d9415b6d0 0x7f6d941ec980 unknown :-1 s=CLOSED pgs=2351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:27.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.088+0000 7f6d9b600640 1 --2- 172.21.15.28:0/2809332534 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6d940c11f0 0x7f6d940ab7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:27.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.088+0000 7f6d9b600640 1 -- 172.21.15.28:0/2809332534 >> 172.21.15.28:0/2809332534 conn(0x7f6d940c67f0 msgr2=0x7f6d94148430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:27.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.088+0000 7f6d9b600640 1 -- 172.21.15.28:0/2809332534 shutdown_connections 2024-02-21T23:13:27.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.088+0000 7f6d9b600640 1 -- 172.21.15.28:0/2809332534 wait complete. 2024-02-21T23:13:27.096 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-02-21T23:13:27.096 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-02-21T23:13:27.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.338+0000 7f0006400640 1 Processor -- start 2024-02-21T23:13:27.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.339+0000 7f0006400640 1 -- start start 2024-02-21T23:13:27.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.341+0000 7f0006400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f00000c1010 0x7f000006e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:27.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.341+0000 7f0006400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f000006e870 0x7f000015d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:27.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.341+0000 7f0006400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0000152fd0 con 0x7f00001501c0 2024-02-21T23:13:27.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.341+0000 7f0006400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00001532b0 con 0x7f000006e870 2024-02-21T23:13:27.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.341+0000 7f0006400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0000153590 con 0x7f00000c1010 2024-02-21T23:13:27.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.341+0000 7f0005a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f000006e870 0x7f000015d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:27.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.341+0000 7f0005a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f000006e870 0x7f000015d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33608/0 (socket says 172.21.15.28:33608) 2024-02-21T23:13:27.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.341+0000 7f0005a00640 1 -- 172.21.15.28:0/2035814491 learned_addr learned my addr 172.21.15.28:0/2035814491 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:27.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.341+0000 7effffe00640 1 --2- 172.21.15.28:0/2035814491 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f00000c1010 0x7f000006e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:27.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.342+0000 7f0005a00640 1 -- 172.21.15.28:0/2035814491 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f00000c1010 msgr2=0x7f000006e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:27.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.342+0000 7f0005a00640 1 --2- 172.21.15.28:0/2035814491 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f00000c1010 0x7f000006e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:27.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.342+0000 7f0005a00640 1 -- 172.21.15.28:0/2035814491 >> v1:172.21.15.81:6789/0 conn(0x7f00001501c0 legacy=0x7f00000a96c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:27.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.342+0000 7f0005a00640 1 -- 172.21.15.28:0/2035814491 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0000153870 con 0x7f000006e870 2024-02-21T23:13:27.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.343+0000 7f0005a00640 1 --2- 172.21.15.28:0/2035814491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f000006e870 0x7f000015d5b0 secure :-1 s=READY pgs=2352 cs=0 l=1 rev1=1 crypto rx=0x7efff8009e50 tx=0x7efff802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee3d491b3a23b950 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:27.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.343+0000 7effff400640 1 -- 172.21.15.28:0/2035814491 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efff802ca70 con 0x7f000006e870 2024-02-21T23:13:27.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.343+0000 7effff400640 1 -- 172.21.15.28:0/2035814491 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efff802cc10 con 0x7f000006e870 2024-02-21T23:13:27.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.343+0000 7effff400640 1 -- 172.21.15.28:0/2035814491 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efff80343f0 con 0x7f000006e870 2024-02-21T23:13:27.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.344+0000 7f0006400640 1 -- 172.21.15.28:0/2035814491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f000006e870 msgr2=0x7f000015d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:27.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.344+0000 7f0006400640 1 --2- 172.21.15.28:0/2035814491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f000006e870 0x7f000015d5b0 secure :-1 s=READY pgs=2352 cs=0 l=1 rev1=1 crypto rx=0x7efff8009e50 tx=0x7efff802bed0 comp rx=0 tx=0).stop 2024-02-21T23:13:27.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.344+0000 7f0006400640 1 -- 172.21.15.28:0/2035814491 shutdown_connections 2024-02-21T23:13:27.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.344+0000 7f0006400640 1 --2- 172.21.15.28:0/2035814491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f000006e870 0x7f000015d5b0 unknown :-1 s=CLOSED pgs=2352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:27.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.344+0000 7f0006400640 1 --2- 172.21.15.28:0/2035814491 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f00000c1010 0x7f000006e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:27.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.344+0000 7f0006400640 1 -- 172.21.15.28:0/2035814491 >> 172.21.15.28:0/2035814491 conn(0x7f00000c6610 msgr2=0x7f00000c69f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:27.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.344+0000 7f0006400640 1 -- 172.21.15.28:0/2035814491 shutdown_connections 2024-02-21T23:13:27.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.344+0000 7f0006400640 1 -- 172.21.15.28:0/2035814491 wait complete. 2024-02-21T23:13:27.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.346+0000 7f0006400640 1 Processor -- start 2024-02-21T23:13:27.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.346+0000 7f0006400640 1 -- start start 2024-02-21T23:13:27.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.347+0000 7f0006400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f000006e870 0x7f00001521c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:27.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.347+0000 7f0006400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f00000c1010 0x7f0000141e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:27.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.347+0000 7f0006400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f00001501c0 0x7f0000142390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:27.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.347+0000 7f0006400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00000c7340 con 0x7f000006e870 2024-02-21T23:13:27.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.347+0000 7f0006400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00000c0c20 con 0x7f00001501c0 2024-02-21T23:13:27.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.347+0000 7f0006400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00001505a0 con 0x7f00000c1010 2024-02-21T23:13:27.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.347+0000 7f0005000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f000006e870 0x7f00001521c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:27.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.347+0000 7f0005000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f000006e870 0x7f00001521c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33610/0 (socket says 172.21.15.28:33610) 2024-02-21T23:13:27.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.347+0000 7f0005000640 1 -- 172.21.15.28:0/2340143058 learned_addr learned my addr 172.21.15.28:0/2340143058 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:27.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.347+0000 7effffe00640 1 --2- 172.21.15.28:0/2340143058 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f00000c1010 0x7f0000141e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:27.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.347+0000 7f0005a00640 1 --2- 172.21.15.28:0/2340143058 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f00001501c0 0x7f0000142390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:27.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.347+0000 7f0005000640 1 -- 172.21.15.28:0/2340143058 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f00000c1010 msgr2=0x7f0000141e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:27.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.347+0000 7f0005000640 1 --2- 172.21.15.28:0/2340143058 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f00000c1010 0x7f0000141e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:27.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.347+0000 7f0005000640 1 -- 172.21.15.28:0/2340143058 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f00001501c0 msgr2=0x7f0000142390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:27.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.347+0000 7f0005000640 1 --2- 172.21.15.28:0/2340143058 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f00001501c0 0x7f0000142390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:27.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.347+0000 7f0005000640 1 -- 172.21.15.28:0/2340143058 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00000c5aa0 con 0x7f000006e870 2024-02-21T23:13:27.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.348+0000 7effffe00640 1 --2- 172.21.15.28:0/2340143058 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f00000c1010 0x7f0000141e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:13:27.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.348+0000 7f0005000640 1 --2- 172.21.15.28:0/2340143058 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f000006e870 0x7f00001521c0 secure :-1 s=READY pgs=2353 cs=0 l=1 rev1=1 crypto rx=0x7efff000c450 tx=0x7efff000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:27.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.348+0000 7efffcc00640 1 -- 172.21.15.28:0/2340143058 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efff00052d0 con 0x7f000006e870 2024-02-21T23:13:27.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.348+0000 7efffcc00640 1 -- 172.21.15.28:0/2340143058 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efff0005470 con 0x7f000006e870 2024-02-21T23:13:27.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.348+0000 7efffcc00640 1 -- 172.21.15.28:0/2340143058 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efff0012870 con 0x7f000006e870 2024-02-21T23:13:27.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.348+0000 7f0006400640 1 -- 172.21.15.28:0/2340143058 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00000c5d80 con 0x7f000006e870 2024-02-21T23:13:27.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.348+0000 7f0006400640 1 -- 172.21.15.28:0/2340143058 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00001eaa50 con 0x7f000006e870 2024-02-21T23:13:27.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.350+0000 7f0006400640 1 -- 172.21.15.28:0/2340143058 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f000004f290 con 0x7f000006e870 2024-02-21T23:13:27.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.350+0000 7efffcc00640 1 -- 172.21.15.28:0/2340143058 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efff0012a10 con 0x7f000006e870 2024-02-21T23:13:27.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.352+0000 7efffcc00640 1 --2- 172.21.15.28:0/2340143058 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7effe006b4d0 0x7effe006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:27.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.352+0000 7efffcc00640 1 -- 172.21.15.28:0/2340143058 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(204..204 src has 1..204) v4 ==== 5249+0+0 (secure 0 0 0) 0x7efff0089bf0 con 0x7f000006e870 2024-02-21T23:13:27.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.353+0000 7effffe00640 1 --2- 172.21.15.28:0/2340143058 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7effe006b4d0 0x7effe006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:27.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.354+0000 7effffe00640 1 --2- 172.21.15.28:0/2340143058 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7effe006b4d0 0x7effe006d990 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f00000a9f40 tx=0x7effec008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:27.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.357+0000 7efffcc00640 1 -- 172.21.15.28:0/2340143058 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7efff0058cc0 con 0x7f000006e870 2024-02-21T23:13:27.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:27.556+0000 7f0006400640 1 -- 172.21.15.28:0/2340143058 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f0000067f30 con 0x7f000006e870 2024-02-21T23:13:28.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.212+0000 7efffcc00640 1 -- 172.21.15.28:0/2340143058 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v205) v1 ==== 138+0+0 (secure 0 0 0) 0x7efff00585c0 con 0x7f000006e870 2024-02-21T23:13:28.200 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' created 2024-02-21T23:13:28.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.217+0000 7f0006400640 1 -- 172.21.15.28:0/2340143058 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7effe006b4d0 msgr2=0x7effe006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:28.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.217+0000 7f0006400640 1 --2- 172.21.15.28:0/2340143058 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7effe006b4d0 0x7effe006d990 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f00000a9f40 tx=0x7effec008040 comp rx=0 tx=0).stop 2024-02-21T23:13:28.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.217+0000 7f0006400640 1 -- 172.21.15.28:0/2340143058 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f000006e870 msgr2=0x7f00001521c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:28.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.217+0000 7f0006400640 1 --2- 172.21.15.28:0/2340143058 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f000006e870 0x7f00001521c0 secure :-1 s=READY pgs=2353 cs=0 l=1 rev1=1 crypto rx=0x7efff000c450 tx=0x7efff000c910 comp rx=0 tx=0).stop 2024-02-21T23:13:28.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.217+0000 7f0006400640 1 -- 172.21.15.28:0/2340143058 shutdown_connections 2024-02-21T23:13:28.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.217+0000 7f0006400640 1 --2- 172.21.15.28:0/2340143058 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7effe006b4d0 0x7effe006d990 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:28.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.217+0000 7f0006400640 1 --2- 172.21.15.28:0/2340143058 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f00000c1010 0x7f0000141e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:28.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.217+0000 7f0006400640 1 --2- 172.21.15.28:0/2340143058 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f00001501c0 0x7f0000142390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:28.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.217+0000 7f0006400640 1 --2- 172.21.15.28:0/2340143058 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f000006e870 0x7f00001521c0 unknown :-1 s=CLOSED pgs=2353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:28.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.217+0000 7f0006400640 1 -- 172.21.15.28:0/2340143058 >> 172.21.15.28:0/2340143058 conn(0x7f00000c6610 msgr2=0x7f00000a9470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:28.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.218+0000 7f0006400640 1 -- 172.21.15.28:0/2340143058 shutdown_connections 2024-02-21T23:13:28.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.219+0000 7f0006400640 1 -- 172.21.15.28:0/2340143058 wait complete. 2024-02-21T23:13:28.233 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-02-21T23:13:28.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.461+0000 7ffafac00640 1 Processor -- start 2024-02-21T23:13:28.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.461+0000 7ffafac00640 1 -- start start 2024-02-21T23:13:28.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.463+0000 7ffafac00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ffaf40a96c0 0x7ffaf4156bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:28.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.463+0000 7ffafac00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaf406f1b0 con 0x7ffaf4150640 2024-02-21T23:13:28.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.463+0000 7ffafac00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaf40c2ae0 con 0x7ffaf40c35c0 2024-02-21T23:13:28.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.463+0000 7ffafac00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffaf406ea40 con 0x7ffaf40a96c0 2024-02-21T23:13:28.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.463+0000 7ffafa200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ffaf40a96c0 0x7ffaf4156bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:28.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.463+0000 7ffaf8e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ffaf4150640 0x7ffaf4152c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59552/0 (socket says 172.21.15.28:59552) 2024-02-21T23:13:28.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.463+0000 7ffaf8e00640 1 -- 172.21.15.28:0/2149045030 learned_addr learned my addr 172.21.15.28:0/2149045030 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:13:28.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.463+0000 7ffafa200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ffaf40a96c0 0x7ffaf4156bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33626/0 (socket says 172.21.15.28:33626) 2024-02-21T23:13:28.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.464+0000 7ffafa200640 1 -- 172.21.15.28:0/2149045030 >> v1:172.21.15.81:6789/0 conn(0x7ffaf40c35c0 legacy=0x7ffaf414ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:28.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.464+0000 7ffafa200640 1 -- 172.21.15.28:0/2149045030 >> v1:172.21.15.28:6789/0 conn(0x7ffaf4150640 legacy=0x7ffaf4152c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:28.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.464+0000 7ffafa200640 1 -- 172.21.15.28:0/2149045030 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaf40c2dc0 con 0x7ffaf40a96c0 2024-02-21T23:13:28.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.464+0000 7ffaf3e00640 1 -- 172.21.15.28:0/2149045030 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1290612258 0 0) 0x7ffaf406f1b0 con 0x7ffaf4150640 2024-02-21T23:13:28.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.466+0000 7ffafa200640 1 --2- 172.21.15.28:0/2149045030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffaf40a96c0 0x7ffaf4156bf0 secure :-1 s=READY pgs=2355 cs=0 l=1 rev1=1 crypto rx=0x7ffaec009d90 tx=0x7ffaec02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7f8fe11980077c server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:28.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.466+0000 7ffaf3e00640 1 -- 172.21.15.28:0/2149045030 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaec039070 con 0x7ffaf40a96c0 2024-02-21T23:13:28.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.466+0000 7ffaf3e00640 1 -- 172.21.15.28:0/2149045030 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaec02cc60 con 0x7ffaf40a96c0 2024-02-21T23:13:28.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.466+0000 7ffaf3e00640 1 -- 172.21.15.28:0/2149045030 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaec034660 con 0x7ffaf40a96c0 2024-02-21T23:13:28.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.467+0000 7ffafac00640 1 -- 172.21.15.28:0/2149045030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffaf40a96c0 msgr2=0x7ffaf4156bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:28.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.467+0000 7ffafac00640 1 --2- 172.21.15.28:0/2149045030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffaf40a96c0 0x7ffaf4156bf0 secure :-1 s=READY pgs=2355 cs=0 l=1 rev1=1 crypto rx=0x7ffaec009d90 tx=0x7ffaec02beb0 comp rx=0 tx=0).stop 2024-02-21T23:13:28.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.467+0000 7ffafac00640 1 -- 172.21.15.28:0/2149045030 shutdown_connections 2024-02-21T23:13:28.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.467+0000 7ffafac00640 1 --2- 172.21.15.28:0/2149045030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffaf40a96c0 0x7ffaf4156bf0 unknown :-1 s=CLOSED pgs=2355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:28.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.467+0000 7ffafac00640 1 -- 172.21.15.28:0/2149045030 >> 172.21.15.28:0/2149045030 conn(0x7ffaf40a8690 msgr2=0x7ffaf40a8a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:28.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.467+0000 7ffafac00640 1 -- 172.21.15.28:0/2149045030 shutdown_connections 2024-02-21T23:13:28.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.467+0000 7ffafac00640 1 -- 172.21.15.28:0/2149045030 wait complete. 2024-02-21T23:13:28.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.469+0000 7ffafac00640 1 Processor -- start 2024-02-21T23:13:28.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.470+0000 7ffafac00640 1 -- start start 2024-02-21T23:13:28.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.471+0000 7ffafac00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffaf40a96c0 0x7ffaf4155b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:28.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.471+0000 7ffafac00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffaf40c35c0 0x7ffaf41ee3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:28.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.471+0000 7ffafac00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffaf4150640 0x7ffaf40c5660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:28.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.471+0000 7ffafac00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf40c39a0 con 0x7ffaf40a96c0 2024-02-21T23:13:28.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.471+0000 7ffafac00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf406e7b0 con 0x7ffaf40c35c0 2024-02-21T23:13:28.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.471+0000 7ffafac00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf40c3b20 con 0x7ffaf4150640 2024-02-21T23:13:28.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.471+0000 7ffaf9800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffaf40a96c0 0x7ffaf4155b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:28.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.471+0000 7ffaf9800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffaf40a96c0 0x7ffaf4155b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33642/0 (socket says 172.21.15.28:33642) 2024-02-21T23:13:28.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.471+0000 7ffaf9800640 1 -- 172.21.15.28:0/1656038280 learned_addr learned my addr 172.21.15.28:0/1656038280 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:28.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.471+0000 7ffaf8e00640 1 --2- 172.21.15.28:0/1656038280 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffaf40c35c0 0x7ffaf41ee3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:28.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.472+0000 7ffafa200640 1 --2- 172.21.15.28:0/1656038280 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffaf4150640 0x7ffaf40c5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:28.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.472+0000 7ffaf9800640 1 -- 172.21.15.28:0/1656038280 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffaf4150640 msgr2=0x7ffaf40c5660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:28.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.472+0000 7ffaf9800640 1 --2- 172.21.15.28:0/1656038280 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffaf4150640 0x7ffaf40c5660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:28.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.472+0000 7ffaf9800640 1 -- 172.21.15.28:0/1656038280 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffaf40c35c0 msgr2=0x7ffaf41ee3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:28.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.472+0000 7ffaf9800640 1 --2- 172.21.15.28:0/1656038280 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffaf40c35c0 0x7ffaf41ee3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:28.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.472+0000 7ffaf9800640 1 -- 172.21.15.28:0/1656038280 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaf41eefe0 con 0x7ffaf40a96c0 2024-02-21T23:13:28.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.472+0000 7ffafa200640 1 --2- 172.21.15.28:0/1656038280 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffaf4150640 0x7ffaf40c5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:13:28.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.472+0000 7ffaf9800640 1 --2- 172.21.15.28:0/1656038280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffaf40a96c0 0x7ffaf4155b60 secure :-1 s=READY pgs=2356 cs=0 l=1 rev1=1 crypto rx=0x7ffae800c480 tx=0x7ffae800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:28.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.472+0000 7ffaf1600640 1 -- 172.21.15.28:0/1656038280 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae80052d0 con 0x7ffaf40a96c0 2024-02-21T23:13:28.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.472+0000 7ffaf1600640 1 -- 172.21.15.28:0/1656038280 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffae8005470 con 0x7ffaf40a96c0 2024-02-21T23:13:28.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.472+0000 7ffaf1600640 1 -- 172.21.15.28:0/1656038280 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae8012870 con 0x7ffaf40a96c0 2024-02-21T23:13:28.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.472+0000 7ffafac00640 1 -- 172.21.15.28:0/1656038280 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaf41eed00 con 0x7ffaf40a96c0 2024-02-21T23:13:28.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.473+0000 7ffafac00640 1 -- 172.21.15.28:0/1656038280 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaf4155320 con 0x7ffaf40a96c0 2024-02-21T23:13:28.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.474+0000 7ffafac00640 1 -- 172.21.15.28:0/1656038280 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffaf4146a60 con 0x7ffaf40a96c0 2024-02-21T23:13:28.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.475+0000 7ffaf1600640 1 -- 172.21.15.28:0/1656038280 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffae8012a10 con 0x7ffaf40a96c0 2024-02-21T23:13:28.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.476+0000 7ffaf1600640 1 --2- 172.21.15.28:0/1656038280 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ffad406b4d0 0x7ffad406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:28.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.476+0000 7ffaf1600640 1 -- 172.21.15.28:0/1656038280 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 5619+0+0 (secure 0 0 0) 0x7ffae8089db0 con 0x7ffaf40a96c0 2024-02-21T23:13:28.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.477+0000 7ffaf8e00640 1 --2- 172.21.15.28:0/1656038280 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ffad406b4d0 0x7ffad406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:28.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.478+0000 7ffaf8e00640 1 --2- 172.21.15.28:0/1656038280 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ffad406b4d0 0x7ffad406d990 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7ffaf41ebd10 tx=0x7ffae0007b90 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:28.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.482+0000 7ffaf1600640 1 -- 172.21.15.28:0/1656038280 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ffaf4146a60 con 0x7ffaf40a96c0 2024-02-21T23:13:28.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:28.689+0000 7ffafac00640 1 -- 172.21.15.28:0/1656038280 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7ffaf404f290 con 0x7ffaf40a96c0 2024-02-21T23:13:29.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.228+0000 7ffaf1600640 1 -- 172.21.15.28:0/1656038280 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v206) v1 ==== 170+0+0 (secure 0 0 0) 0x7ffae8058d10 con 0x7ffaf40a96c0 2024-02-21T23:13:29.216 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' created 2024-02-21T23:13:29.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.233+0000 7ffafac00640 1 -- 172.21.15.28:0/1656038280 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ffad406b4d0 msgr2=0x7ffad406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:29.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.233+0000 7ffafac00640 1 --2- 172.21.15.28:0/1656038280 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ffad406b4d0 0x7ffad406d990 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7ffaf41ebd10 tx=0x7ffae0007b90 comp rx=0 tx=0).stop 2024-02-21T23:13:29.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.233+0000 7ffafac00640 1 -- 172.21.15.28:0/1656038280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffaf40a96c0 msgr2=0x7ffaf4155b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:29.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.233+0000 7ffafac00640 1 --2- 172.21.15.28:0/1656038280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffaf40a96c0 0x7ffaf4155b60 secure :-1 s=READY pgs=2356 cs=0 l=1 rev1=1 crypto rx=0x7ffae800c480 tx=0x7ffae800c940 comp rx=0 tx=0).stop 2024-02-21T23:13:29.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.235+0000 7ffafac00640 1 -- 172.21.15.28:0/1656038280 shutdown_connections 2024-02-21T23:13:29.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.235+0000 7ffafac00640 1 --2- 172.21.15.28:0/1656038280 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ffad406b4d0 0x7ffad406d990 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:29.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.235+0000 7ffafac00640 1 --2- 172.21.15.28:0/1656038280 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffaf4150640 0x7ffaf40c5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:29.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.235+0000 7ffafac00640 1 --2- 172.21.15.28:0/1656038280 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffaf40c35c0 0x7ffaf41ee3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:29.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.235+0000 7ffafac00640 1 --2- 172.21.15.28:0/1656038280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffaf40a96c0 0x7ffaf4155b60 secure :-1 s=CLOSED pgs=2356 cs=0 l=1 rev1=1 crypto rx=0x7ffae800c480 tx=0x7ffae800c940 comp rx=0 tx=0).stop 2024-02-21T23:13:29.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.235+0000 7ffafac00640 1 -- 172.21.15.28:0/1656038280 >> 172.21.15.28:0/1656038280 conn(0x7ffaf40a8690 msgr2=0x7ffaf41450d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:29.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.237+0000 7ffafac00640 1 -- 172.21.15.28:0/1656038280 shutdown_connections 2024-02-21T23:13:29.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.237+0000 7ffafac00640 1 -- 172.21.15.28:0/1656038280 wait complete. 2024-02-21T23:13:29.261 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-02-21T23:13:29.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.462+0000 7f50a5c00640 1 Processor -- start 2024-02-21T23:13:29.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.463+0000 7f50a5c00640 1 -- start start 2024-02-21T23:13:29.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.464+0000 7f50a5c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f50a00c4580 0x7f50a00c12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:29.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.464+0000 7f50a5c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50a006e220 con 0x7f50a00c2c10 2024-02-21T23:13:29.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.464+0000 7f50a5c00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50a006f780 con 0x7f50a0150240 2024-02-21T23:13:29.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.464+0000 7f50a5c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50a0150620 con 0x7f50a00c4580 2024-02-21T23:13:29.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.464+0000 7f50a5200640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f50a00c2c10 0x7f50a006ae70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59554/0 (socket says 172.21.15.28:59554) 2024-02-21T23:13:29.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.465+0000 7f50a5200640 1 -- 172.21.15.28:0/1053185454 learned_addr learned my addr 172.21.15.28:0/1053185454 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:13:29.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.465+0000 7f509f400640 1 --2- 172.21.15.28:0/1053185454 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f50a00c4580 0x7f50a00c12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:29.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.465+0000 7f509ea00640 1 -- 172.21.15.28:0/1053185454 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3049271935 0 0) 0x7f50a006e220 con 0x7f50a00c2c10 2024-02-21T23:13:29.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.465+0000 7f509ea00640 1 -- 172.21.15.28:0/1053185454 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5084003520 con 0x7f50a00c2c10 2024-02-21T23:13:29.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.465+0000 7f509ea00640 1 -- 172.21.15.28:0/1053185454 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 428540753 0 0) 0x7f5084003520 con 0x7f50a00c2c10 2024-02-21T23:13:29.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.465+0000 7f509ea00640 1 -- 172.21.15.28:0/1053185454 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f50a00c4580 msgr2=0x7f50a00c12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:29.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.465+0000 7f509ea00640 1 --2- 172.21.15.28:0/1053185454 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f50a00c4580 0x7f50a00c12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:29.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.465+0000 7f509ea00640 1 -- 172.21.15.28:0/1053185454 >> v1:172.21.15.81:6789/0 conn(0x7f50a0150240 legacy=0x7f50a006db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:29.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.465+0000 7f509ea00640 1 -- 172.21.15.28:0/1053185454 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50a006f4a0 con 0x7f50a00c2c10 2024-02-21T23:13:29.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.465+0000 7f509ea00640 1 -- 172.21.15.28:0/1053185454 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f50980019f0 con 0x7f50a00c2c10 2024-02-21T23:13:29.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.466+0000 7f509ea00640 1 -- 172.21.15.28:0/1053185454 >> v1:172.21.15.28:6789/0 conn(0x7f50a00c2c10 legacy=0x7f50a006ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:29.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.466+0000 7f509ea00640 1 --2- 172.21.15.28:0/1053185454 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f50a00c4580 0x7f50840039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:29.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.466+0000 7f509ea00640 1 --2- 172.21.15.28:0/1053185454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5084003f60 0x7f5084004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:29.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.466+0000 7f509ea00640 1 --2- 172.21.15.28:0/1053185454 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5084004d40 0x7f50840051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:29.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.466+0000 7f509ea00640 1 -- 172.21.15.28:0/1053185454 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f508400bb50 con 0x7f5084003f60 2024-02-21T23:13:29.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.466+0000 7f509ea00640 1 -- 172.21.15.28:0/1053185454 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f508400bcd0 con 0x7f5084004d40 2024-02-21T23:13:29.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.466+0000 7f509ea00640 1 -- 172.21.15.28:0/1053185454 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f508400bed0 con 0x7f50a00c4580 2024-02-21T23:13:29.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.466+0000 7f509f400640 1 --2- 172.21.15.28:0/1053185454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5084003f60 0x7f5084004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:29.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.466+0000 7f50a5200640 1 --2- 172.21.15.28:0/1053185454 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5084004d40 0x7f50840051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:29.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.466+0000 7f509f400640 1 -- 172.21.15.28:0/1053185454 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f50a00c4580 msgr2=0x7f50840039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:29.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.466+0000 7f509f400640 1 --2- 172.21.15.28:0/1053185454 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f50a00c4580 0x7f50840039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:29.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.466+0000 7f509f400640 1 -- 172.21.15.28:0/1053185454 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5084004d40 msgr2=0x7f50840051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:29.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.466+0000 7f509f400640 1 --2- 172.21.15.28:0/1053185454 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5084004d40 0x7f50840051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:29.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.466+0000 7f509f400640 1 -- 172.21.15.28:0/1053185454 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5084003520 con 0x7f5084003f60 2024-02-21T23:13:29.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.467+0000 7f50a5200640 1 --2- 172.21.15.28:0/1053185454 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5084004d40 0x7f50840051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:13:29.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.468+0000 7f509f400640 1 --2- 172.21.15.28:0/1053185454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5084003f60 0x7f5084004800 secure :-1 s=READY pgs=2358 cs=0 l=1 rev1=1 crypto rx=0x7f5088009a90 tx=0x7f5084005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df9ddc956b85b994 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:29.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.468+0000 7f509ea00640 1 -- 172.21.15.28:0/1053185454 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5088039070 con 0x7f5084003f60 2024-02-21T23:13:29.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.468+0000 7f509ea00640 1 -- 172.21.15.28:0/1053185454 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f508802c6b0 con 0x7f5084003f60 2024-02-21T23:13:29.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.468+0000 7f509ea00640 1 -- 172.21.15.28:0/1053185454 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5088009690 con 0x7f5084003f60 2024-02-21T23:13:29.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.468+0000 7f50a5c00640 1 -- 172.21.15.28:0/1053185454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5084003f60 msgr2=0x7f5084004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:29.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.468+0000 7f50a5c00640 1 --2- 172.21.15.28:0/1053185454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5084003f60 0x7f5084004800 secure :-1 s=READY pgs=2358 cs=0 l=1 rev1=1 crypto rx=0x7f5088009a90 tx=0x7f5084005830 comp rx=0 tx=0).stop 2024-02-21T23:13:29.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.468+0000 7f50a5200640 1 -- 172.21.15.28:0/1053185454 reap_dead start 2024-02-21T23:13:29.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.469+0000 7f50a5c00640 1 -- 172.21.15.28:0/1053185454 shutdown_connections 2024-02-21T23:13:29.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.469+0000 7f50a5c00640 1 -- 172.21.15.28:0/1053185454 >> 172.21.15.28:0/1053185454 conn(0x7f50a00a8530 msgr2=0x7f50a00a8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:29.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.469+0000 7f50a5c00640 1 -- 172.21.15.28:0/1053185454 shutdown_connections 2024-02-21T23:13:29.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.469+0000 7f50a5c00640 1 -- 172.21.15.28:0/1053185454 wait complete. 2024-02-21T23:13:29.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.470+0000 7f50a5c00640 1 Processor -- start 2024-02-21T23:13:29.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.471+0000 7f50a5c00640 1 -- start start 2024-02-21T23:13:29.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.471+0000 7f50a5c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5084003f60 0x7f50a00710f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:29.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.471+0000 7f50a5c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50a0153f60 0x7f50a0149c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:29.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.471+0000 7f50a5c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f50a01e3240 0x7f50a01e36c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:29.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.471+0000 7f50a5c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50a00c4a10 con 0x7f50a0153f60 2024-02-21T23:13:29.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.471+0000 7f50a5c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50a0153a90 con 0x7f50a01e3240 2024-02-21T23:13:29.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.471+0000 7f50a5c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50a00c4b90 con 0x7f5084003f60 2024-02-21T23:13:29.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.472+0000 7f509f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50a0153f60 0x7f50a0149c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:29.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.472+0000 7f509f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50a0153f60 0x7f50a0149c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33666/0 (socket says 172.21.15.28:33666) 2024-02-21T23:13:29.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.472+0000 7f509f400640 1 -- 172.21.15.28:0/3794948092 learned_addr learned my addr 172.21.15.28:0/3794948092 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:29.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.472+0000 7f509fe00640 1 --2- 172.21.15.28:0/3794948092 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5084003f60 0x7f50a00710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:29.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.472+0000 7f509f400640 1 -- 172.21.15.28:0/3794948092 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5084003f60 msgr2=0x7f50a00710f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:29.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.472+0000 7f509f400640 1 --2- 172.21.15.28:0/3794948092 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5084003f60 0x7f50a00710f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:29.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.472+0000 7f509f400640 1 -- 172.21.15.28:0/3794948092 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f50a01e3240 msgr2=0x7f50a01e36c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:29.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.472+0000 7f50a5200640 1 --2- 172.21.15.28:0/3794948092 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f50a01e3240 0x7f50a01e36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:29.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.472+0000 7f509f400640 1 --2- 172.21.15.28:0/3794948092 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f50a01e3240 0x7f50a01e36c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:29.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.472+0000 7f509f400640 1 -- 172.21.15.28:0/3794948092 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50a0071630 con 0x7f50a0153f60 2024-02-21T23:13:29.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.473+0000 7f509f400640 1 --2- 172.21.15.28:0/3794948092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50a0153f60 0x7f50a0149c20 secure :-1 s=READY pgs=2359 cs=0 l=1 rev1=1 crypto rx=0x7f5088000c00 tx=0x7f5088004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:29.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.473+0000 7f5097e00640 1 -- 172.21.15.28:0/3794948092 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5088042020 con 0x7f50a0153f60 2024-02-21T23:13:29.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.473+0000 7f5097e00640 1 -- 172.21.15.28:0/3794948092 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f508803a040 con 0x7f50a0153f60 2024-02-21T23:13:29.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.473+0000 7f50a5c00640 1 -- 172.21.15.28:0/3794948092 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50a01ea050 con 0x7f50a0153f60 2024-02-21T23:13:29.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.473+0000 7f5097e00640 1 -- 172.21.15.28:0/3794948092 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5088039070 con 0x7f50a0153f60 2024-02-21T23:13:29.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.473+0000 7f50a5c00640 1 -- 172.21.15.28:0/3794948092 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50a00a7e70 con 0x7f50a0153f60 2024-02-21T23:13:29.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.474+0000 7f50a5c00640 1 -- 172.21.15.28:0/3794948092 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50a004f290 con 0x7f50a0153f60 2024-02-21T23:13:29.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.475+0000 7f5097e00640 1 -- 172.21.15.28:0/3794948092 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5088045020 con 0x7f50a0153f60 2024-02-21T23:13:29.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.476+0000 7f5097e00640 1 --2- 172.21.15.28:0/3794948092 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f507c06b4d0 0x7f507c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:29.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.476+0000 7f5097e00640 1 -- 172.21.15.28:0/3794948092 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(206..206 src has 1..206) v4 ==== 6001+0+0 (secure 0 0 0) 0x7f50880ab7f0 con 0x7f50a0153f60 2024-02-21T23:13:29.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.477+0000 7f509fe00640 1 --2- 172.21.15.28:0/3794948092 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f507c06b4d0 0x7f507c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:29.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.478+0000 7f509fe00640 1 --2- 172.21.15.28:0/3794948092 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f507c06b4d0 0x7f507c06d990 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f50900016f0 tx=0x7f5090009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:29.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.482+0000 7f5097e00640 1 -- 172.21.15.28:0/3794948092 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f508804c070 con 0x7f50a0153f60 2024-02-21T23:13:29.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:29.723+0000 7f50a5c00640 1 -- 172.21.15.28:0/3794948092 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f50a006dcb0 con 0x7f50a0153f60 2024-02-21T23:13:30.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.249+0000 7f5097e00640 1 -- 172.21.15.28:0/3794948092 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 33 and data pool 34 v225) v1 ==== 174+0+0 (secure 0 0 0) 0x7f508807a530 con 0x7f50a0153f60 2024-02-21T23:13:30.241 INFO:teuthology.orchestra.run.smithi028.stderr:new fs with metadata pool 33 and data pool 34 2024-02-21T23:13:30.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.262+0000 7f50a5c00640 1 -- 172.21.15.28:0/3794948092 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f507c06b4d0 msgr2=0x7f507c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:30.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.262+0000 7f50a5c00640 1 --2- 172.21.15.28:0/3794948092 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f507c06b4d0 0x7f507c06d990 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f50900016f0 tx=0x7f5090009040 comp rx=0 tx=0).stop 2024-02-21T23:13:30.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.262+0000 7f50a5c00640 1 -- 172.21.15.28:0/3794948092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50a0153f60 msgr2=0x7f50a0149c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:30.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.262+0000 7f50a5c00640 1 --2- 172.21.15.28:0/3794948092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50a0153f60 0x7f50a0149c20 secure :-1 s=READY pgs=2359 cs=0 l=1 rev1=1 crypto rx=0x7f5088000c00 tx=0x7f5088004730 comp rx=0 tx=0).stop 2024-02-21T23:13:30.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.262+0000 7f50a5c00640 1 -- 172.21.15.28:0/3794948092 shutdown_connections 2024-02-21T23:13:30.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.262+0000 7f50a5c00640 1 --2- 172.21.15.28:0/3794948092 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f507c06b4d0 0x7f507c06d990 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:30.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.262+0000 7f50a5c00640 1 --2- 172.21.15.28:0/3794948092 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f50a01e3240 0x7f50a01e36c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:30.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.262+0000 7f50a5c00640 1 --2- 172.21.15.28:0/3794948092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50a0153f60 0x7f50a0149c20 unknown :-1 s=CLOSED pgs=2359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:30.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.262+0000 7f50a5c00640 1 --2- 172.21.15.28:0/3794948092 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5084003f60 0x7f50a00710f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:30.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.263+0000 7f50a5c00640 1 -- 172.21.15.28:0/3794948092 >> 172.21.15.28:0/3794948092 conn(0x7f50a00a8530 msgr2=0x7f50a0151f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:30.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.265+0000 7f50a5c00640 1 -- 172.21.15.28:0/3794948092 shutdown_connections 2024-02-21T23:13:30.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.265+0000 7f50a5c00640 1 -- 172.21.15.28:0/3794948092 wait complete. 2024-02-21T23:13:30.282 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:13:30.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.476+0000 7fa493200640 1 Processor -- start 2024-02-21T23:13:30.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.476+0000 7fa493200640 1 -- start start 2024-02-21T23:13:30.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.478+0000 7fa493200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fa4940c6740 0x7fa494154fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:30.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.478+0000 7fa493200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa4941554f0 0x7fa4940c8a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:30.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.478+0000 7fa493200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa494155a00 con 0x7fa49414ef60 2024-02-21T23:13:30.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.478+0000 7fa493200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa494155ce0 con 0x7fa4941554f0 2024-02-21T23:13:30.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.478+0000 7fa493200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4941586d0 con 0x7fa4940c6740 2024-02-21T23:13:30.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.478+0000 7fa492800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa4941554f0 0x7fa4940c8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:30.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.478+0000 7fa492800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa4941554f0 0x7fa4940c8a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33682/0 (socket says 172.21.15.28:33682) 2024-02-21T23:13:30.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.478+0000 7fa492800640 1 -- 172.21.15.28:0/453999182 learned_addr learned my addr 172.21.15.28:0/453999182 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:30.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.478+0000 7fa491400640 1 --2- 172.21.15.28:0/453999182 >> v2:172.21.15.81:3300/0 conn(0x7fa4940c6740 0x7fa494154fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:30.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.479+0000 7fa492800640 1 -- 172.21.15.28:0/453999182 >> v2:172.21.15.81:3300/0 conn(0x7fa4940c6740 msgr2=0x7fa494154fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:30.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.479+0000 7fa492800640 1 --2- 172.21.15.28:0/453999182 >> v2:172.21.15.81:3300/0 conn(0x7fa4940c6740 0x7fa494154fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:30.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.479+0000 7fa492800640 1 -- 172.21.15.28:0/453999182 >> v1:172.21.15.81:6789/0 conn(0x7fa49414ef60 legacy=0x7fa494158ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:30.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.479+0000 7fa492800640 1 -- 172.21.15.28:0/453999182 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4941589b0 con 0x7fa4941554f0 2024-02-21T23:13:30.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.480+0000 7fa492800640 1 --2- 172.21.15.28:0/453999182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4941554f0 0x7fa4940c8a10 secure :-1 s=READY pgs=2360 cs=0 l=1 rev1=1 crypto rx=0x7fa4940c2ed0 tx=0x7fa48c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dafe9603145e98a3 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:30.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.480+0000 7fa490a00640 1 -- 172.21.15.28:0/453999182 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa48c035020 con 0x7fa4941554f0 2024-02-21T23:13:30.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.480+0000 7fa490a00640 1 -- 172.21.15.28:0/453999182 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa48c031640 con 0x7fa4941554f0 2024-02-21T23:13:30.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.480+0000 7fa490a00640 1 -- 172.21.15.28:0/453999182 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa48c02d040 con 0x7fa4941554f0 2024-02-21T23:13:30.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.481+0000 7fa493200640 1 -- 172.21.15.28:0/453999182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4941554f0 msgr2=0x7fa4940c8a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:30.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.481+0000 7fa493200640 1 --2- 172.21.15.28:0/453999182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4941554f0 0x7fa4940c8a10 secure :-1 s=READY pgs=2360 cs=0 l=1 rev1=1 crypto rx=0x7fa4940c2ed0 tx=0x7fa48c02b8f0 comp rx=0 tx=0).stop 2024-02-21T23:13:30.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.481+0000 7fa493200640 1 -- 172.21.15.28:0/453999182 shutdown_connections 2024-02-21T23:13:30.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.481+0000 7fa493200640 1 --2- 172.21.15.28:0/453999182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4941554f0 0x7fa4940c8a10 unknown :-1 s=CLOSED pgs=2360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:30.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.481+0000 7fa493200640 1 --2- 172.21.15.28:0/453999182 >> v2:172.21.15.81:3300/0 conn(0x7fa4940c6740 0x7fa494154fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:30.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.481+0000 7fa493200640 1 -- 172.21.15.28:0/453999182 >> 172.21.15.28:0/453999182 conn(0x7fa49406b260 msgr2=0x7fa49406b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:30.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.481+0000 7fa493200640 1 -- 172.21.15.28:0/453999182 shutdown_connections 2024-02-21T23:13:30.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.481+0000 7fa493200640 1 -- 172.21.15.28:0/453999182 wait complete. 2024-02-21T23:13:30.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.483+0000 7fa493200640 1 Processor -- start 2024-02-21T23:13:30.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.483+0000 7fa493200640 1 -- start start 2024-02-21T23:13:30.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.484+0000 7fa493200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa4940c6740 0x7fa494070bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:30.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.484+0000 7fa493200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa49414ef60 0x7fa4941ee530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:30.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.484+0000 7fa493200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4941554f0 0x7fa494156ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:30.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.484+0000 7fa493200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4940c2b80 con 0x7fa4941554f0 2024-02-21T23:13:30.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.484+0000 7fa493200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4940c2a00 con 0x7fa49414ef60 2024-02-21T23:13:30.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.484+0000 7fa493200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4940c6ba0 con 0x7fa4940c6740 2024-02-21T23:13:30.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.485+0000 7fa492800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4941554f0 0x7fa494156ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:30.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.485+0000 7fa492800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4941554f0 0x7fa494156ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33696/0 (socket says 172.21.15.28:33696) 2024-02-21T23:13:30.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.485+0000 7fa492800640 1 -- 172.21.15.28:0/3463712933 learned_addr learned my addr 172.21.15.28:0/3463712933 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:30.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.485+0000 7fa491e00640 1 --2- 172.21.15.28:0/3463712933 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa4940c6740 0x7fa494070bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:30.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.485+0000 7fa492800640 1 -- 172.21.15.28:0/3463712933 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa4940c6740 msgr2=0x7fa494070bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:30.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.485+0000 7fa492800640 1 --2- 172.21.15.28:0/3463712933 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa4940c6740 0x7fa494070bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:30.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.485+0000 7fa492800640 1 -- 172.21.15.28:0/3463712933 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa49414ef60 msgr2=0x7fa4941ee530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:30.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.485+0000 7fa492800640 1 --2- 172.21.15.28:0/3463712933 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa49414ef60 0x7fa4941ee530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:30.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.485+0000 7fa492800640 1 -- 172.21.15.28:0/3463712933 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4941eee00 con 0x7fa4941554f0 2024-02-21T23:13:30.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.486+0000 7fa492800640 1 --2- 172.21.15.28:0/3463712933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4941554f0 0x7fa494156ce0 secure :-1 s=READY pgs=2361 cs=0 l=1 rev1=1 crypto rx=0x7fa48c0090b0 tx=0x7fa48c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:30.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.486+0000 7fa48a000640 1 -- 172.21.15.28:0/3463712933 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa48c040070 con 0x7fa4941554f0 2024-02-21T23:13:30.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.486+0000 7fa48a000640 1 -- 172.21.15.28:0/3463712933 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa48c0399f0 con 0x7fa4941554f0 2024-02-21T23:13:30.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.486+0000 7fa48a000640 1 -- 172.21.15.28:0/3463712933 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa48c035040 con 0x7fa4941554f0 2024-02-21T23:13:30.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.486+0000 7fa493200640 1 -- 172.21.15.28:0/3463712933 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4941ef0e0 con 0x7fa4941554f0 2024-02-21T23:13:30.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.486+0000 7fa493200640 1 -- 172.21.15.28:0/3463712933 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa494157fb0 con 0x7fa4941554f0 2024-02-21T23:13:30.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.487+0000 7fa493200640 1 -- 172.21.15.28:0/3463712933 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa49404f290 con 0x7fa4941554f0 2024-02-21T23:13:30.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.488+0000 7fa48a000640 1 -- 172.21.15.28:0/3463712933 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa48c04b060 con 0x7fa4941554f0 2024-02-21T23:13:30.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.489+0000 7fa48a000640 1 --2- 172.21.15.28:0/3463712933 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa47006b4d0 0x7fa47006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:30.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.489+0000 7fa48a000640 1 -- 172.21.15.28:0/3463712933 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa48c0abcd0 con 0x7fa4941554f0 2024-02-21T23:13:30.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.490+0000 7fa491e00640 1 --2- 172.21.15.28:0/3463712933 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa47006b4d0 0x7fa47006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:30.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.492+0000 7fa491e00640 1 --2- 172.21.15.28:0/3463712933 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa47006b4d0 0x7fa47006d990 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fa494154fd0 tx=0x7fa484009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:30.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.494+0000 7fa48a000640 1 -- 172.21.15.28:0/3463712933 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fa48c07a9b0 con 0x7fa4941554f0 2024-02-21T23:13:30.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.694+0000 7fa493200640 1 -- 172.21.15.28:0/3463712933 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa4940a7d60 con 0x7fa4941554f0 2024-02-21T23:13:30.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.697+0000 7fa48a000640 1 -- 172.21.15.28:0/3463712933 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v207) v1 ==== 74+0+16841 (secure 0 0 0) 0x7fa48c07a2b0 con 0x7fa4941554f0 2024-02-21T23:13:30.685 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:13:30.685 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":207,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:13:29.724740+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:13:27.612392+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"207","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_data","create_time":"2024-02-21T23:13:28.744308+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"207","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:13:30.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.700+0000 7fa493200640 1 -- 172.21.15.28:0/3463712933 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa47006b4d0 msgr2=0x7fa47006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:30.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.700+0000 7fa493200640 1 --2- 172.21.15.28:0/3463712933 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa47006b4d0 0x7fa47006d990 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fa494154fd0 tx=0x7fa484009040 comp rx=0 tx=0).stop 2024-02-21T23:13:30.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.700+0000 7fa493200640 1 -- 172.21.15.28:0/3463712933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4941554f0 msgr2=0x7fa494156ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:30.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.700+0000 7fa493200640 1 --2- 172.21.15.28:0/3463712933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4941554f0 0x7fa494156ce0 secure :-1 s=READY pgs=2361 cs=0 l=1 rev1=1 crypto rx=0x7fa48c0090b0 tx=0x7fa48c0046c0 comp rx=0 tx=0).stop 2024-02-21T23:13:30.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.700+0000 7fa493200640 1 -- 172.21.15.28:0/3463712933 shutdown_connections 2024-02-21T23:13:30.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.700+0000 7fa493200640 1 --2- 172.21.15.28:0/3463712933 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa47006b4d0 0x7fa47006d990 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:30.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.700+0000 7fa493200640 1 --2- 172.21.15.28:0/3463712933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4941554f0 0x7fa494156ce0 unknown :-1 s=CLOSED pgs=2361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:30.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.700+0000 7fa493200640 1 --2- 172.21.15.28:0/3463712933 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa49414ef60 0x7fa4941ee530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:30.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.700+0000 7fa493200640 1 --2- 172.21.15.28:0/3463712933 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa4940c6740 0x7fa494070bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:30.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.700+0000 7fa493200640 1 -- 172.21.15.28:0/3463712933 >> 172.21.15.28:0/3463712933 conn(0x7fa49406b260 msgr2=0x7fa49406b640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:30.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.700+0000 7fa493200640 1 -- 172.21.15.28:0/3463712933 shutdown_connections 2024-02-21T23:13:30.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.700+0000 7fa493200640 1 -- 172.21.15.28:0/3463712933 wait complete. 2024-02-21T23:13:30.706 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:13:30.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.921+0000 7f5b9be00640 1 Processor -- start 2024-02-21T23:13:30.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.921+0000 7f5b9be00640 1 -- start start 2024-02-21T23:13:30.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.923+0000 7f5b9be00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f5b9c06ee40 0x7f5b9c0a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:30.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.923+0000 7f5b9be00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5b9c156af0 0x7f5b9c06dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:30.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.923+0000 7f5b9be00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b9c0c2760 con 0x7f5b9c150450 2024-02-21T23:13:30.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.923+0000 7f5b9be00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b9c0c22c0 con 0x7f5b9c06ee40 2024-02-21T23:13:30.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.923+0000 7f5b9be00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b9c156660 con 0x7f5b9c156af0 2024-02-21T23:13:30.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.923+0000 7f5b9aa00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f5b9c150450 0x7f5b9c1536f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59556/0 (socket says 172.21.15.28:59556) 2024-02-21T23:13:30.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.923+0000 7f5b9aa00640 1 -- 172.21.15.28:0/3542565055 learned_addr learned my addr 172.21.15.28:0/3542565055 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:13:30.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.923+0000 7f5b9a000640 1 --2- 172.21.15.28:0/3542565055 >> v2:172.21.15.81:3300/0 conn(0x7f5b9c06ee40 0x7f5b9c0a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:30.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.924+0000 7f5b99600640 1 -- 172.21.15.28:0/3542565055 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2726768052 0 0) 0x7f5b9c0c2760 con 0x7f5b9c150450 2024-02-21T23:13:30.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.924+0000 7f5b99600640 1 -- 172.21.15.28:0/3542565055 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b80003520 con 0x7f5b9c150450 2024-02-21T23:13:30.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.924+0000 7f5b9b400640 1 --2- 172.21.15.28:0/3542565055 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5b9c156af0 0x7f5b9c06dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:30.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.924+0000 7f5b99600640 1 -- 172.21.15.28:0/3542565055 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2335228264 0 0) 0x7f5b80003520 con 0x7f5b9c150450 2024-02-21T23:13:30.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.924+0000 7f5b99600640 1 -- 172.21.15.28:0/3542565055 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5b9c156af0 msgr2=0x7f5b9c06dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:30.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.924+0000 7f5b99600640 1 --2- 172.21.15.28:0/3542565055 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5b9c156af0 0x7f5b9c06dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:30.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.924+0000 7f5b99600640 1 -- 172.21.15.28:0/3542565055 >> v2:172.21.15.81:3300/0 conn(0x7f5b9c06ee40 msgr2=0x7f5b9c0a5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:30.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.924+0000 7f5b99600640 1 --2- 172.21.15.28:0/3542565055 >> v2:172.21.15.81:3300/0 conn(0x7f5b9c06ee40 0x7f5b9c0a5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:30.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.924+0000 7f5b99600640 1 -- 172.21.15.28:0/3542565055 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b9c156310 con 0x7f5b9c150450 2024-02-21T23:13:30.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.924+0000 7f5b99600640 1 -- 172.21.15.28:0/3542565055 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f5b8c001670 con 0x7f5b9c150450 2024-02-21T23:13:30.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.924+0000 7f5b99600640 1 -- 172.21.15.28:0/3542565055 >> v1:172.21.15.28:6789/0 conn(0x7f5b9c150450 legacy=0x7f5b9c1536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:30.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.924+0000 7f5b99600640 1 --2- 172.21.15.28:0/3542565055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b800039b0 0x7f5b80003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:30.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.924+0000 7f5b99600640 1 --2- 172.21.15.28:0/3542565055 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5b80004800 0x7f5b80004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:30.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.924+0000 7f5b99600640 1 --2- 172.21.15.28:0/3542565055 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b800051c0 0x7f5b8000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:30.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.924+0000 7f5b99600640 1 -- 172.21.15.28:0/3542565055 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b9c156660 con 0x7f5b800039b0 2024-02-21T23:13:30.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.924+0000 7f5b99600640 1 -- 172.21.15.28:0/3542565055 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b8000bf00 con 0x7f5b800051c0 2024-02-21T23:13:30.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.924+0000 7f5b99600640 1 -- 172.21.15.28:0/3542565055 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b8000c0b0 con 0x7f5b80004800 2024-02-21T23:13:30.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.924+0000 7f5b9b400640 1 --2- 172.21.15.28:0/3542565055 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5b9c156af0 0x7f5b9c06dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:13:30.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.924+0000 7f5b9aa00640 1 --2- 172.21.15.28:0/3542565055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b800039b0 0x7f5b80003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:30.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.925+0000 7f5b9a000640 1 --2- 172.21.15.28:0/3542565055 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5b80004800 0x7f5b80004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:30.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.925+0000 7f5b9b400640 1 --2- 172.21.15.28:0/3542565055 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b800051c0 0x7f5b8000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:30.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.925+0000 7f5b9aa00640 1 -- 172.21.15.28:0/3542565055 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5b80004800 msgr2=0x7f5b80004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:30.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.925+0000 7f5b9aa00640 1 --2- 172.21.15.28:0/3542565055 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5b80004800 0x7f5b80004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:30.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.925+0000 7f5b9aa00640 1 -- 172.21.15.28:0/3542565055 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b800051c0 msgr2=0x7f5b8000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:30.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.925+0000 7f5b9aa00640 1 --2- 172.21.15.28:0/3542565055 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b800051c0 0x7f5b8000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:30.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.925+0000 7f5b9aa00640 1 -- 172.21.15.28:0/3542565055 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b80003520 con 0x7f5b800039b0 2024-02-21T23:13:30.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.927+0000 7f5b9aa00640 1 --2- 172.21.15.28:0/3542565055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b800039b0 0x7f5b80003dd0 secure :-1 s=READY pgs=2363 cs=0 l=1 rev1=1 crypto rx=0x7f5b8c00c290 tx=0x7f5b80005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9dcfc1bcee70687 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:30.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.927+0000 7f5b99600640 1 -- 172.21.15.28:0/3542565055 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b8c007d50 con 0x7f5b800039b0 2024-02-21T23:13:30.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.927+0000 7f5b99600640 1 -- 172.21.15.28:0/3542565055 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b8c030070 con 0x7f5b800039b0 2024-02-21T23:13:30.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.927+0000 7f5b99600640 1 -- 172.21.15.28:0/3542565055 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b8c02cd80 con 0x7f5b800039b0 2024-02-21T23:13:30.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.927+0000 7f5b9be00640 1 -- 172.21.15.28:0/3542565055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b800039b0 msgr2=0x7f5b80003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:30.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.927+0000 7f5b9be00640 1 --2- 172.21.15.28:0/3542565055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b800039b0 0x7f5b80003dd0 secure :-1 s=READY pgs=2363 cs=0 l=1 rev1=1 crypto rx=0x7f5b8c00c290 tx=0x7f5b80005830 comp rx=0 tx=0).stop 2024-02-21T23:13:30.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.928+0000 7f5b9b400640 1 -- 172.21.15.28:0/3542565055 reap_dead start 2024-02-21T23:13:30.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.928+0000 7f5b9be00640 1 -- 172.21.15.28:0/3542565055 shutdown_connections 2024-02-21T23:13:30.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.928+0000 7f5b9be00640 1 -- 172.21.15.28:0/3542565055 >> 172.21.15.28:0/3542565055 conn(0x7f5b9c06b630 msgr2=0x7f5b9c0c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:30.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.928+0000 7f5b9be00640 1 -- 172.21.15.28:0/3542565055 shutdown_connections 2024-02-21T23:13:30.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.928+0000 7f5b9be00640 1 -- 172.21.15.28:0/3542565055 wait complete. 2024-02-21T23:13:30.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.929+0000 7f5b9be00640 1 Processor -- start 2024-02-21T23:13:30.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.930+0000 7f5b9be00640 1 -- start start 2024-02-21T23:13:30.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.930+0000 7f5b9be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b9c155840 0x7f5b9c147540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:30.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.930+0000 7f5b9be00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5b9c150b30 0x7f5b9c147a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:30.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.930+0000 7f5b9be00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b9c151ed0 0x7f5b9c152350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:30.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.930+0000 7f5b9be00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b9c156f80 con 0x7f5b9c155840 2024-02-21T23:13:30.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.930+0000 7f5b9be00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b9c0c2cb0 con 0x7f5b9c151ed0 2024-02-21T23:13:30.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.931+0000 7f5b9be00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b9c157100 con 0x7f5b9c150b30 2024-02-21T23:13:30.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.931+0000 7f5b9aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b9c155840 0x7f5b9c147540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:30.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.931+0000 7f5b9aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b9c155840 0x7f5b9c147540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33716/0 (socket says 172.21.15.28:33716) 2024-02-21T23:13:30.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.931+0000 7f5b9aa00640 1 -- 172.21.15.28:0/3249839917 learned_addr learned my addr 172.21.15.28:0/3249839917 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:30.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.931+0000 7f5b9aa00640 1 -- 172.21.15.28:0/3249839917 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5b9c150b30 msgr2=0x7f5b9c147a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:30.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.931+0000 7f5b9aa00640 1 --2- 172.21.15.28:0/3249839917 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5b9c150b30 0x7f5b9c147a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:30.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.931+0000 7f5b9aa00640 1 -- 172.21.15.28:0/3249839917 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b9c151ed0 msgr2=0x7f5b9c152350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:30.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.931+0000 7f5b9aa00640 1 --2- 172.21.15.28:0/3249839917 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b9c151ed0 0x7f5b9c152350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:30.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.931+0000 7f5b9aa00640 1 -- 172.21.15.28:0/3249839917 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b9c0c54f0 con 0x7f5b9c155840 2024-02-21T23:13:30.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.931+0000 7f5b9aa00640 1 --2- 172.21.15.28:0/3249839917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b9c155840 0x7f5b9c147540 secure :-1 s=READY pgs=2364 cs=0 l=1 rev1=1 crypto rx=0x7f5b8c00c3b0 tx=0x7f5b8c001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:30.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.932+0000 7f5b93400640 1 -- 172.21.15.28:0/3249839917 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b8c001c80 con 0x7f5b9c155840 2024-02-21T23:13:30.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.932+0000 7f5b93400640 1 -- 172.21.15.28:0/3249839917 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b8c004cb0 con 0x7f5b9c155840 2024-02-21T23:13:30.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.932+0000 7f5b93400640 1 -- 172.21.15.28:0/3249839917 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b8c007d50 con 0x7f5b9c155840 2024-02-21T23:13:30.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.932+0000 7f5b9be00640 1 -- 172.21.15.28:0/3249839917 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b9c147fc0 con 0x7f5b9c155840 2024-02-21T23:13:30.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.932+0000 7f5b9be00640 1 -- 172.21.15.28:0/3249839917 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b9c1ee1c0 con 0x7f5b9c155840 2024-02-21T23:13:30.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.933+0000 7f5b9be00640 1 -- 172.21.15.28:0/3249839917 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b9c04f290 con 0x7f5b9c155840 2024-02-21T23:13:30.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.934+0000 7f5b93400640 1 -- 172.21.15.28:0/3249839917 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5b8c007270 con 0x7f5b9c155840 2024-02-21T23:13:30.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.935+0000 7f5b93400640 1 --2- 172.21.15.28:0/3249839917 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5b7c06b500 0x7f5b7c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:30.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.935+0000 7f5b93400640 1 -- 172.21.15.28:0/3249839917 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5b8c0ab880 con 0x7f5b9c155840 2024-02-21T23:13:30.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.936+0000 7f5b9a000640 1 --2- 172.21.15.28:0/3249839917 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5b7c06b500 0x7f5b7c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:30.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.937+0000 7f5b9a000640 1 --2- 172.21.15.28:0/3249839917 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5b7c06b500 0x7f5b7c06d9c0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c156290 tx=0x7f5b88008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:30.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:30.940+0000 7f5b93400640 1 -- 172.21.15.28:0/3249839917 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f5b8c007560 con 0x7f5b9c155840 2024-02-21T23:13:31.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.139+0000 7f5b9be00640 1 -- 172.21.15.28:0/3249839917 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5b9c04f290 con 0x7f5b9c155840 2024-02-21T23:13:31.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.142+0000 7f5b93400640 1 -- 172.21.15.28:0/3249839917 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v207) v1 ==== 74+0+16841 (secure 0 0 0) 0x7f5b8c07a4e0 con 0x7f5b9c155840 2024-02-21T23:13:31.130 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:13:31.130 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":207,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:13:29.724740+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:13:27.612392+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"207","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_data","create_time":"2024-02-21T23:13:28.744308+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"207","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:13:31.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.144+0000 7f5b9be00640 1 -- 172.21.15.28:0/3249839917 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5b7c06b500 msgr2=0x7f5b7c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:31.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.144+0000 7f5b9be00640 1 --2- 172.21.15.28:0/3249839917 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5b7c06b500 0x7f5b7c06d9c0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c156290 tx=0x7f5b88008040 comp rx=0 tx=0).stop 2024-02-21T23:13:31.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.144+0000 7f5b9be00640 1 -- 172.21.15.28:0/3249839917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b9c155840 msgr2=0x7f5b9c147540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:31.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.144+0000 7f5b9be00640 1 --2- 172.21.15.28:0/3249839917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b9c155840 0x7f5b9c147540 secure :-1 s=READY pgs=2364 cs=0 l=1 rev1=1 crypto rx=0x7f5b8c00c3b0 tx=0x7f5b8c001a40 comp rx=0 tx=0).stop 2024-02-21T23:13:31.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.145+0000 7f5b9be00640 1 -- 172.21.15.28:0/3249839917 shutdown_connections 2024-02-21T23:13:31.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.145+0000 7f5b9be00640 1 --2- 172.21.15.28:0/3249839917 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5b7c06b500 0x7f5b7c06d9c0 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:31.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.145+0000 7f5b9be00640 1 --2- 172.21.15.28:0/3249839917 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5b9c150b30 0x7f5b9c147a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:31.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.145+0000 7f5b9be00640 1 --2- 172.21.15.28:0/3249839917 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b9c151ed0 0x7f5b9c152350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:31.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.145+0000 7f5b9be00640 1 --2- 172.21.15.28:0/3249839917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b9c155840 0x7f5b9c147540 unknown :-1 s=CLOSED pgs=2364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:31.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.145+0000 7f5b9be00640 1 -- 172.21.15.28:0/3249839917 >> 172.21.15.28:0/3249839917 conn(0x7f5b9c06b630 msgr2=0x7f5b9c0a6d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:31.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.145+0000 7f5b9be00640 1 -- 172.21.15.28:0/3249839917 shutdown_connections 2024-02-21T23:13:31.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.145+0000 7f5b9be00640 1 -- 172.21.15.28:0/3249839917 wait complete. 2024-02-21T23:13:31.152 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-02-21T23:13:31.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.382+0000 7f7022200640 1 Processor -- start 2024-02-21T23:13:31.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.383+0000 7f7022200640 1 -- start start 2024-02-21T23:13:31.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.384+0000 7f7022200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f701c06b0e0 0x7f701c0c5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:31.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.384+0000 7f7022200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f701c0c5d80 0x7f701c14fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:31.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.384+0000 7f7022200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f701c0c61e0 con 0x7f701c150530 2024-02-21T23:13:31.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.384+0000 7f7022200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f701c153bc0 con 0x7f701c0c5d80 2024-02-21T23:13:31.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.384+0000 7f7022200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f701c152fb0 con 0x7f701c06b0e0 2024-02-21T23:13:31.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.384+0000 7f701aa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f701c0c5d80 0x7f701c14fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:31.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.384+0000 7f701be00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f701c150530 0x7f701c156500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59570/0 (socket says 172.21.15.28:59570) 2024-02-21T23:13:31.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.384+0000 7f701be00640 1 -- 172.21.15.28:0/1094343932 learned_addr learned my addr 172.21.15.28:0/1094343932 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:13:31.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.385+0000 7f701b400640 1 --2- 172.21.15.28:0/1094343932 >> v2:172.21.15.81:3300/0 conn(0x7f701c06b0e0 0x7f701c0c5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:31.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.385+0000 7f701aa00640 1 -- 172.21.15.28:0/1094343932 >> v2:172.21.15.81:3300/0 conn(0x7f701c06b0e0 msgr2=0x7f701c0c5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:31.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.385+0000 7f701aa00640 1 --2- 172.21.15.28:0/1094343932 >> v2:172.21.15.81:3300/0 conn(0x7f701c06b0e0 0x7f701c0c5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:31.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.385+0000 7f701aa00640 1 -- 172.21.15.28:0/1094343932 >> v1:172.21.15.28:6789/0 conn(0x7f701c150530 legacy=0x7f701c156500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:31.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.385+0000 7f701aa00640 1 -- 172.21.15.28:0/1094343932 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f701c153290 con 0x7f701c0c5d80 2024-02-21T23:13:31.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.385+0000 7f701a000640 1 -- 172.21.15.28:0/1094343932 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1943494534 0 0) 0x7f701c0c61e0 con 0x7f701c150530 2024-02-21T23:13:31.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.386+0000 7f701aa00640 1 --2- 172.21.15.28:0/1094343932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f701c0c5d80 0x7f701c14fff0 secure :-1 s=READY pgs=2366 cs=0 l=1 rev1=1 crypto rx=0x7f701c0c77e0 tx=0x7f700802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be48f2275503064b server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:31.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.386+0000 7f701a000640 1 -- 172.21.15.28:0/1094343932 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70080315a0 con 0x7f701c0c5d80 2024-02-21T23:13:31.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.386+0000 7f701a000640 1 -- 172.21.15.28:0/1094343932 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7008037070 con 0x7f701c0c5d80 2024-02-21T23:13:31.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.386+0000 7f701a000640 1 -- 172.21.15.28:0/1094343932 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7008031960 con 0x7f701c0c5d80 2024-02-21T23:13:31.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.387+0000 7f7022200640 1 -- 172.21.15.28:0/1094343932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f701c0c5d80 msgr2=0x7f701c14fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:31.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.387+0000 7f7022200640 1 --2- 172.21.15.28:0/1094343932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f701c0c5d80 0x7f701c14fff0 secure :-1 s=READY pgs=2366 cs=0 l=1 rev1=1 crypto rx=0x7f701c0c77e0 tx=0x7f700802b840 comp rx=0 tx=0).stop 2024-02-21T23:13:31.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.387+0000 7f7022200640 1 -- 172.21.15.28:0/1094343932 shutdown_connections 2024-02-21T23:13:31.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.387+0000 7f7022200640 1 --2- 172.21.15.28:0/1094343932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f701c0c5d80 0x7f701c14fff0 unknown :-1 s=CLOSED pgs=2366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:31.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.387+0000 7f7022200640 1 --2- 172.21.15.28:0/1094343932 >> v2:172.21.15.81:3300/0 conn(0x7f701c06b0e0 0x7f701c0c5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:31.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.387+0000 7f7022200640 1 -- 172.21.15.28:0/1094343932 >> 172.21.15.28:0/1094343932 conn(0x7f701c0a85a0 msgr2=0x7f701c0a8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:31.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.387+0000 7f7022200640 1 -- 172.21.15.28:0/1094343932 shutdown_connections 2024-02-21T23:13:31.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.387+0000 7f7022200640 1 -- 172.21.15.28:0/1094343932 wait complete. 2024-02-21T23:13:31.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.389+0000 7f7022200640 1 Processor -- start 2024-02-21T23:13:31.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.389+0000 7f7022200640 1 -- start start 2024-02-21T23:13:31.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.390+0000 7f7022200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f701c06b0e0 0x7f701c149710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:31.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.390+0000 7f7022200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f701c0c5d80 0x7f701c1ed900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:31.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.390+0000 7f7022200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f701c150530 0x7f701c155410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:31.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.390+0000 7f7022200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f701c150970 con 0x7f701c0c5d80 2024-02-21T23:13:31.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.390+0000 7f7022200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f701c153850 con 0x7f701c06b0e0 2024-02-21T23:13:31.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.390+0000 7f7022200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f701c06b4c0 con 0x7f701c150530 2024-02-21T23:13:31.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.390+0000 7f701aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f701c0c5d80 0x7f701c1ed900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:31.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.391+0000 7f701aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f701c0c5d80 0x7f701c1ed900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33736/0 (socket says 172.21.15.28:33736) 2024-02-21T23:13:31.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.391+0000 7f701aa00640 1 -- 172.21.15.28:0/41620369 learned_addr learned my addr 172.21.15.28:0/41620369 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:31.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.391+0000 7f701b400640 1 --2- 172.21.15.28:0/41620369 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f701c06b0e0 0x7f701c149710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:31.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.391+0000 7f701be00640 1 --2- 172.21.15.28:0/41620369 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f701c150530 0x7f701c155410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:31.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.391+0000 7f701aa00640 1 -- 172.21.15.28:0/41620369 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f701c150530 msgr2=0x7f701c155410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:31.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.391+0000 7f701aa00640 1 --2- 172.21.15.28:0/41620369 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f701c150530 0x7f701c155410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:31.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.391+0000 7f701aa00640 1 -- 172.21.15.28:0/41620369 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f701c06b0e0 msgr2=0x7f701c149710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:31.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.391+0000 7f701aa00640 1 --2- 172.21.15.28:0/41620369 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f701c06b0e0 0x7f701c149710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:31.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.391+0000 7f701aa00640 1 -- 172.21.15.28:0/41620369 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f701c0a5dc0 con 0x7f701c0c5d80 2024-02-21T23:13:31.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.391+0000 7f701aa00640 1 --2- 172.21.15.28:0/41620369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f701c0c5d80 0x7f701c1ed900 secure :-1 s=READY pgs=2367 cs=0 l=1 rev1=1 crypto rx=0x7f7008000c00 tx=0x7f70080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:31.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.392+0000 7f7013400640 1 -- 172.21.15.28:0/41620369 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7008032840 con 0x7f701c0c5d80 2024-02-21T23:13:31.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.392+0000 7f7013400640 1 -- 172.21.15.28:0/41620369 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7008032a00 con 0x7f701c0c5d80 2024-02-21T23:13:31.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.392+0000 7f7013400640 1 -- 172.21.15.28:0/41620369 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7008032df0 con 0x7f701c0c5d80 2024-02-21T23:13:31.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.392+0000 7f7022200640 1 -- 172.21.15.28:0/41620369 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f701c06eea0 con 0x7f701c0c5d80 2024-02-21T23:13:31.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.392+0000 7f7022200640 1 -- 172.21.15.28:0/41620369 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f701c156030 con 0x7f701c0c5d80 2024-02-21T23:13:31.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.393+0000 7f7022200640 1 -- 172.21.15.28:0/41620369 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f701c0a6270 con 0x7f701c0c5d80 2024-02-21T23:13:31.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.394+0000 7f7013400640 1 -- 172.21.15.28:0/41620369 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f700804d3f0 con 0x7f701c0c5d80 2024-02-21T23:13:31.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.396+0000 7f7013400640 1 --2- 172.21.15.28:0/41620369 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6ffc06b500 0x7f6ffc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:31.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.396+0000 7f7013400640 1 -- 172.21.15.28:0/41620369 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f700802f030 con 0x7f701c0c5d80 2024-02-21T23:13:31.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.396+0000 7f701b400640 1 --2- 172.21.15.28:0/41620369 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6ffc06b500 0x7f6ffc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:31.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.397+0000 7f701b400640 1 --2- 172.21.15.28:0/41620369 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6ffc06b500 0x7f6ffc06d9c0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f700c006fd0 tx=0x7f700c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:31.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.399+0000 7f7013400640 1 -- 172.21.15.28:0/41620369 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f701c0a6270 con 0x7f701c0c5d80 2024-02-21T23:13:31.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:31.638+0000 7f7022200640 1 -- 172.21.15.28:0/41620369 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f701c067f30 con 0x7f701c0c5d80 2024-02-21T23:13:32.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.262+0000 7f7013400640 1 -- 172.21.15.28:0/41620369 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v228) v1 ==== 118+0+0 (secure 0 0 0) 0x7f7008040070 con 0x7f701c0c5d80 2024-02-21T23:13:32.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.264+0000 7f7022200640 1 -- 172.21.15.28:0/41620369 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6ffc06b500 msgr2=0x7f6ffc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:32.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.264+0000 7f7022200640 1 --2- 172.21.15.28:0/41620369 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6ffc06b500 0x7f6ffc06d9c0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f700c006fd0 tx=0x7f700c008040 comp rx=0 tx=0).stop 2024-02-21T23:13:32.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.264+0000 7f7022200640 1 -- 172.21.15.28:0/41620369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f701c0c5d80 msgr2=0x7f701c1ed900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:32.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.264+0000 7f7022200640 1 --2- 172.21.15.28:0/41620369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f701c0c5d80 0x7f701c1ed900 secure :-1 s=READY pgs=2367 cs=0 l=1 rev1=1 crypto rx=0x7f7008000c00 tx=0x7f70080046c0 comp rx=0 tx=0).stop 2024-02-21T23:13:32.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.265+0000 7f7022200640 1 -- 172.21.15.28:0/41620369 shutdown_connections 2024-02-21T23:13:32.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.265+0000 7f7022200640 1 --2- 172.21.15.28:0/41620369 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6ffc06b500 0x7f6ffc06d9c0 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:32.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.265+0000 7f7022200640 1 --2- 172.21.15.28:0/41620369 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f701c150530 0x7f701c155410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:32.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.265+0000 7f7022200640 1 --2- 172.21.15.28:0/41620369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f701c0c5d80 0x7f701c1ed900 unknown :-1 s=CLOSED pgs=2367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:32.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.265+0000 7f7022200640 1 --2- 172.21.15.28:0/41620369 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f701c06b0e0 0x7f701c149710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:32.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.265+0000 7f7022200640 1 -- 172.21.15.28:0/41620369 >> 172.21.15.28:0/41620369 conn(0x7f701c0a85a0 msgr2=0x7f701c14b030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:32.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.265+0000 7f7022200640 1 -- 172.21.15.28:0/41620369 shutdown_connections 2024-02-21T23:13:32.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.265+0000 7f7022200640 1 -- 172.21.15.28:0/41620369 wait complete. 2024-02-21T23:13:32.279 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-02-21T23:13:32.279 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-02-21T23:13:32.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.509+0000 7f1dd1800640 1 Processor -- start 2024-02-21T23:13:32.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.509+0000 7f1dd1800640 1 -- start start 2024-02-21T23:13:32.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.510+0000 7f1dd1800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1dcc06b0e0 0x7f1dcc0c5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:32.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.510+0000 7f1dd1800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1dcc0c5d80 0x7f1dcc14fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:32.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.510+0000 7f1dd1800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f1dcc150530 0x7f1dcc156500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:32.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.510+0000 7f1dd1800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dcc153bc0 con 0x7f1dcc0c5d80 2024-02-21T23:13:32.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.510+0000 7f1dd1800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dcc0c61e0 con 0x7f1dcc150530 2024-02-21T23:13:32.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.510+0000 7f1dd1800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dcc152fb0 con 0x7f1dcc06b0e0 2024-02-21T23:13:32.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.511+0000 7f1dca000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1dcc0c5d80 0x7f1dcc14fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:32.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.511+0000 7f1dca000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1dcc0c5d80 0x7f1dcc14fff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33752/0 (socket says 172.21.15.28:33752) 2024-02-21T23:13:32.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.511+0000 7f1dca000640 1 -- 172.21.15.28:0/3335893921 learned_addr learned my addr 172.21.15.28:0/3335893921 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:32.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.511+0000 7f1dcaa00640 1 --2- 172.21.15.28:0/3335893921 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1dcc06b0e0 0x7f1dcc0c5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:32.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.511+0000 7f1dcb400640 1 --2- 172.21.15.28:0/3335893921 >> v2:172.21.15.81:3300/0 conn(0x7f1dcc150530 0x7f1dcc156500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:32.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.511+0000 7f1dca000640 1 -- 172.21.15.28:0/3335893921 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1dcc06b0e0 msgr2=0x7f1dcc0c5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:32.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.511+0000 7f1dca000640 1 --2- 172.21.15.28:0/3335893921 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1dcc06b0e0 0x7f1dcc0c5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:32.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.511+0000 7f1dca000640 1 -- 172.21.15.28:0/3335893921 >> v2:172.21.15.81:3300/0 conn(0x7f1dcc150530 msgr2=0x7f1dcc156500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:32.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.511+0000 7f1dca000640 1 --2- 172.21.15.28:0/3335893921 >> v2:172.21.15.81:3300/0 conn(0x7f1dcc150530 0x7f1dcc156500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:32.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.511+0000 7f1dca000640 1 -- 172.21.15.28:0/3335893921 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dcc153290 con 0x7f1dcc0c5d80 2024-02-21T23:13:32.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.512+0000 7f1dca000640 1 --2- 172.21.15.28:0/3335893921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1dcc0c5d80 0x7f1dcc14fff0 secure :-1 s=READY pgs=2368 cs=0 l=1 rev1=1 crypto rx=0x7f1dcc0722f0 tx=0x7f1db402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7249ca14cbbf284 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:32.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.513+0000 7f1dc9600640 1 -- 172.21.15.28:0/3335893921 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1db402e070 con 0x7f1dcc0c5d80 2024-02-21T23:13:32.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.513+0000 7f1dc9600640 1 -- 172.21.15.28:0/3335893921 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1db4037070 con 0x7f1dcc0c5d80 2024-02-21T23:13:32.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.513+0000 7f1dc9600640 1 -- 172.21.15.28:0/3335893921 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1db4031a00 con 0x7f1dcc0c5d80 2024-02-21T23:13:32.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.513+0000 7f1dd1800640 1 -- 172.21.15.28:0/3335893921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1dcc0c5d80 msgr2=0x7f1dcc14fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:32.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.513+0000 7f1dd1800640 1 --2- 172.21.15.28:0/3335893921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1dcc0c5d80 0x7f1dcc14fff0 secure :-1 s=READY pgs=2368 cs=0 l=1 rev1=1 crypto rx=0x7f1dcc0722f0 tx=0x7f1db402b880 comp rx=0 tx=0).stop 2024-02-21T23:13:32.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.513+0000 7f1dd1800640 1 -- 172.21.15.28:0/3335893921 shutdown_connections 2024-02-21T23:13:32.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.513+0000 7f1dd1800640 1 --2- 172.21.15.28:0/3335893921 >> v2:172.21.15.81:3300/0 conn(0x7f1dcc150530 0x7f1dcc156500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:32.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.513+0000 7f1dd1800640 1 --2- 172.21.15.28:0/3335893921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1dcc0c5d80 0x7f1dcc14fff0 unknown :-1 s=CLOSED pgs=2368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:32.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.513+0000 7f1dd1800640 1 --2- 172.21.15.28:0/3335893921 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1dcc06b0e0 0x7f1dcc0c5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:32.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.513+0000 7f1dd1800640 1 -- 172.21.15.28:0/3335893921 >> 172.21.15.28:0/3335893921 conn(0x7f1dcc0a85a0 msgr2=0x7f1dcc0a8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:32.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.513+0000 7f1dd1800640 1 -- 172.21.15.28:0/3335893921 shutdown_connections 2024-02-21T23:13:32.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.514+0000 7f1dd1800640 1 -- 172.21.15.28:0/3335893921 wait complete. 2024-02-21T23:13:32.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.515+0000 7f1dd1800640 1 Processor -- start 2024-02-21T23:13:32.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.515+0000 7f1dd1800640 1 -- start start 2024-02-21T23:13:32.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.516+0000 7f1dd1800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1dcc06b0e0 0x7f1dcc0ab9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:32.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.516+0000 7f1dd1800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1dcc0c5d80 0x7f1dcc1e9a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:32.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.516+0000 7f1dd1800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1dcc150530 0x7f1dcc1541e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:32.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.516+0000 7f1dd1800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dcc1509c0 con 0x7f1dcc0c5d80 2024-02-21T23:13:32.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.516+0000 7f1dd1800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dcc153850 con 0x7f1dcc06b0e0 2024-02-21T23:13:32.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.516+0000 7f1dd1800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dcc06b4c0 con 0x7f1dcc150530 2024-02-21T23:13:32.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.516+0000 7f1dca000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1dcc0c5d80 0x7f1dcc1e9a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:32.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.516+0000 7f1dca000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1dcc0c5d80 0x7f1dcc1e9a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33768/0 (socket says 172.21.15.28:33768) 2024-02-21T23:13:32.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.516+0000 7f1dca000640 1 -- 172.21.15.28:0/1972235787 learned_addr learned my addr 172.21.15.28:0/1972235787 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:32.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.517+0000 7f1dcb400640 1 --2- 172.21.15.28:0/1972235787 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1dcc150530 0x7f1dcc1541e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:32.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.517+0000 7f1dcaa00640 1 --2- 172.21.15.28:0/1972235787 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1dcc06b0e0 0x7f1dcc0ab9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:32.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.517+0000 7f1dca000640 1 -- 172.21.15.28:0/1972235787 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1dcc150530 msgr2=0x7f1dcc1541e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:32.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.517+0000 7f1dca000640 1 --2- 172.21.15.28:0/1972235787 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1dcc150530 0x7f1dcc1541e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:32.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.517+0000 7f1dca000640 1 -- 172.21.15.28:0/1972235787 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1dcc06b0e0 msgr2=0x7f1dcc0ab9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:32.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.517+0000 7f1dca000640 1 --2- 172.21.15.28:0/1972235787 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1dcc06b0e0 0x7f1dcc0ab9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:32.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.517+0000 7f1dca000640 1 -- 172.21.15.28:0/1972235787 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dcc154bd0 con 0x7f1dcc0c5d80 2024-02-21T23:13:32.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.517+0000 7f1dcaa00640 1 --2- 172.21.15.28:0/1972235787 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1dcc06b0e0 0x7f1dcc0ab9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:13:32.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.517+0000 7f1dca000640 1 --2- 172.21.15.28:0/1972235787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1dcc0c5d80 0x7f1dcc1e9a80 secure :-1 s=READY pgs=2369 cs=0 l=1 rev1=1 crypto rx=0x7f1db402b8c0 tx=0x7f1db40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:32.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.518+0000 7f1dc2a00640 1 -- 172.21.15.28:0/1972235787 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1db4032880 con 0x7f1dcc0c5d80 2024-02-21T23:13:32.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.518+0000 7f1dc2a00640 1 -- 172.21.15.28:0/1972235787 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1db4032a40 con 0x7f1dcc0c5d80 2024-02-21T23:13:32.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.518+0000 7f1dc2a00640 1 -- 172.21.15.28:0/1972235787 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1db4032e30 con 0x7f1dcc0c5d80 2024-02-21T23:13:32.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.518+0000 7f1dd1800640 1 -- 172.21.15.28:0/1972235787 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dcc1e8fa0 con 0x7f1dcc0c5d80 2024-02-21T23:13:32.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.518+0000 7f1dd1800640 1 -- 172.21.15.28:0/1972235787 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dcc14fa00 con 0x7f1dcc0c5d80 2024-02-21T23:13:32.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.519+0000 7f1dd1800640 1 -- 172.21.15.28:0/1972235787 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dcc067f30 con 0x7f1dcc0c5d80 2024-02-21T23:13:32.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.520+0000 7f1dc2a00640 1 -- 172.21.15.28:0/1972235787 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1db404c060 con 0x7f1dcc0c5d80 2024-02-21T23:13:32.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.521+0000 7f1dc2a00640 1 --2- 172.21.15.28:0/1972235787 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1da806b4d0 0x7f1da806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:32.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.521+0000 7f1dc2a00640 1 -- 172.21.15.28:0/1972235787 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1db40abda0 con 0x7f1dcc0c5d80 2024-02-21T23:13:32.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.522+0000 7f1dcaa00640 1 --2- 172.21.15.28:0/1972235787 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1da806b4d0 0x7f1da806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:32.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.523+0000 7f1dcaa00640 1 --2- 172.21.15.28:0/1972235787 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1da806b4d0 0x7f1da806d990 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f1dbc006fd0 tx=0x7f1dbc008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:32.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.528+0000 7f1dc2a00640 1 -- 172.21.15.28:0/1972235787 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1db4040070 con 0x7f1dcc0c5d80 2024-02-21T23:13:32.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:32.766+0000 7f1dd1800640 1 -- 172.21.15.28:0/1972235787 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f1dcc14c1b0 con 0x7f1dcc0c5d80 2024-02-21T23:13:33.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.265+0000 7f1dc2a00640 1 -- 172.21.15.28:0/1972235787 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v229) v1 ==== 122+0+0 (secure 0 0 0) 0x7f1db407ab10 con 0x7f1dcc0c5d80 2024-02-21T23:13:33.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.269+0000 7f1dd1800640 1 -- 172.21.15.28:0/1972235787 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1da806b4d0 msgr2=0x7f1da806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:33.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.269+0000 7f1dd1800640 1 --2- 172.21.15.28:0/1972235787 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1da806b4d0 0x7f1da806d990 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f1dbc006fd0 tx=0x7f1dbc008040 comp rx=0 tx=0).stop 2024-02-21T23:13:33.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.269+0000 7f1dd1800640 1 -- 172.21.15.28:0/1972235787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1dcc0c5d80 msgr2=0x7f1dcc1e9a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:33.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.269+0000 7f1dd1800640 1 --2- 172.21.15.28:0/1972235787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1dcc0c5d80 0x7f1dcc1e9a80 secure :-1 s=READY pgs=2369 cs=0 l=1 rev1=1 crypto rx=0x7f1db402b8c0 tx=0x7f1db40046c0 comp rx=0 tx=0).stop 2024-02-21T23:13:33.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.269+0000 7f1dd1800640 1 -- 172.21.15.28:0/1972235787 shutdown_connections 2024-02-21T23:13:33.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.269+0000 7f1dd1800640 1 --2- 172.21.15.28:0/1972235787 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1da806b4d0 0x7f1da806d990 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:33.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.269+0000 7f1dd1800640 1 --2- 172.21.15.28:0/1972235787 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1dcc150530 0x7f1dcc1541e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:33.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.269+0000 7f1dd1800640 1 --2- 172.21.15.28:0/1972235787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1dcc0c5d80 0x7f1dcc1e9a80 unknown :-1 s=CLOSED pgs=2369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:33.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.269+0000 7f1dd1800640 1 --2- 172.21.15.28:0/1972235787 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1dcc06b0e0 0x7f1dcc0ab9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:33.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.269+0000 7f1dd1800640 1 -- 172.21.15.28:0/1972235787 >> 172.21.15.28:0/1972235787 conn(0x7f1dcc0a85a0 msgr2=0x7f1dcc14c3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:33.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.269+0000 7f1dd1800640 1 -- 172.21.15.28:0/1972235787 shutdown_connections 2024-02-21T23:13:33.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.269+0000 7f1dd1800640 1 -- 172.21.15.28:0/1972235787 wait complete. 2024-02-21T23:13:33.278 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-02-21T23:13:33.279 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:13:33.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.506+0000 7f304e000640 1 Processor -- start 2024-02-21T23:13:33.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.506+0000 7f304e000640 1 -- start start 2024-02-21T23:13:33.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.508+0000 7f304e000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f304806ee40 0x7f30480a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:33.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.508+0000 7f304e000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3048156af0 0x7f304806dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:33.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.508+0000 7f304e000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30480c2760 con 0x7f3048150450 2024-02-21T23:13:33.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.508+0000 7f304e000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30480c22c0 con 0x7f3048156af0 2024-02-21T23:13:33.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.508+0000 7f304e000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3048156660 con 0x7f304806ee40 2024-02-21T23:13:33.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.508+0000 7f304d600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3048156af0 0x7f304806dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:33.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.508+0000 7f304d600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3048156af0 0x7f304806dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33778/0 (socket says 172.21.15.28:33778) 2024-02-21T23:13:33.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.508+0000 7f304d600640 1 -- 172.21.15.28:0/102536827 learned_addr learned my addr 172.21.15.28:0/102536827 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:33.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.509+0000 7f303fe00640 1 --2- 172.21.15.28:0/102536827 >> v2:172.21.15.81:3300/0 conn(0x7f304806ee40 0x7f30480a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:33.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.509+0000 7f304d600640 1 -- 172.21.15.28:0/102536827 >> v2:172.21.15.81:3300/0 conn(0x7f304806ee40 msgr2=0x7f30480a5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:33.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.509+0000 7f304d600640 1 --2- 172.21.15.28:0/102536827 >> v2:172.21.15.81:3300/0 conn(0x7f304806ee40 0x7f30480a5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:33.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.509+0000 7f304d600640 1 -- 172.21.15.28:0/102536827 >> v1:172.21.15.81:6789/0 conn(0x7f3048150450 legacy=0x7f30481536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:33.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.509+0000 7f304d600640 1 -- 172.21.15.28:0/102536827 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3048156310 con 0x7f3048156af0 2024-02-21T23:13:33.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.510+0000 7f304d600640 1 --2- 172.21.15.28:0/102536827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3048156af0 0x7f304806dff0 secure :-1 s=READY pgs=2370 cs=0 l=1 rev1=1 crypto rx=0x7f304806bcc0 tx=0x7f304002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a104bc83f7d87add server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:33.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.510+0000 7f3047e00640 1 -- 172.21.15.28:0/102536827 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f304002ca80 con 0x7f3048156af0 2024-02-21T23:13:33.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.510+0000 7f3047e00640 1 -- 172.21.15.28:0/102536827 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f304002cc20 con 0x7f3048156af0 2024-02-21T23:13:33.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.510+0000 7f3047e00640 1 -- 172.21.15.28:0/102536827 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30400343f0 con 0x7f3048156af0 2024-02-21T23:13:33.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.511+0000 7f304e000640 1 -- 172.21.15.28:0/102536827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3048156af0 msgr2=0x7f304806dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:33.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.511+0000 7f304e000640 1 --2- 172.21.15.28:0/102536827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3048156af0 0x7f304806dff0 secure :-1 s=READY pgs=2370 cs=0 l=1 rev1=1 crypto rx=0x7f304806bcc0 tx=0x7f304002bee0 comp rx=0 tx=0).stop 2024-02-21T23:13:33.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.511+0000 7f304e000640 1 -- 172.21.15.28:0/102536827 shutdown_connections 2024-02-21T23:13:33.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.511+0000 7f304e000640 1 --2- 172.21.15.28:0/102536827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3048156af0 0x7f304806dff0 unknown :-1 s=CLOSED pgs=2370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:33.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.511+0000 7f304e000640 1 --2- 172.21.15.28:0/102536827 >> v2:172.21.15.81:3300/0 conn(0x7f304806ee40 0x7f30480a5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:33.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.511+0000 7f304e000640 1 -- 172.21.15.28:0/102536827 >> 172.21.15.28:0/102536827 conn(0x7f304806b630 msgr2=0x7f30480c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:33.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.511+0000 7f304e000640 1 -- 172.21.15.28:0/102536827 shutdown_connections 2024-02-21T23:13:33.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.511+0000 7f304e000640 1 -- 172.21.15.28:0/102536827 wait complete. 2024-02-21T23:13:33.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.514+0000 7f304e000640 1 Processor -- start 2024-02-21T23:13:33.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.514+0000 7f304e000640 1 -- start start 2024-02-21T23:13:33.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.516+0000 7f304e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f304806ee40 0x7f3048152180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:33.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.516+0000 7f304e000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3048150450 0x7f304814cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:33.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.516+0000 7f304e000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3048156af0 0x7f30481ee380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:33.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.516+0000 7f304e000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3048156f80 con 0x7f304806ee40 2024-02-21T23:13:33.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.516+0000 7f304e000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30480c2cb0 con 0x7f3048156af0 2024-02-21T23:13:33.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.516+0000 7f304e000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3048157100 con 0x7f3048150450 2024-02-21T23:13:33.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.516+0000 7f304cc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f304806ee40 0x7f3048152180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:33.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.516+0000 7f304cc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f304806ee40 0x7f3048152180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33794/0 (socket says 172.21.15.28:33794) 2024-02-21T23:13:33.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.516+0000 7f304cc00640 1 -- 172.21.15.28:0/814588218 learned_addr learned my addr 172.21.15.28:0/814588218 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:33.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.516+0000 7f304d600640 1 --2- 172.21.15.28:0/814588218 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3048156af0 0x7f30481ee380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:33.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.516+0000 7f304cc00640 1 -- 172.21.15.28:0/814588218 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3048150450 msgr2=0x7f304814cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:33.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.516+0000 7f304cc00640 1 --2- 172.21.15.28:0/814588218 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3048150450 0x7f304814cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:33.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.516+0000 7f304cc00640 1 -- 172.21.15.28:0/814588218 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3048156af0 msgr2=0x7f30481ee380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:33.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.516+0000 7f304cc00640 1 --2- 172.21.15.28:0/814588218 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3048156af0 0x7f30481ee380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:33.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.516+0000 7f304cc00640 1 -- 172.21.15.28:0/814588218 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f304814a9e0 con 0x7f304806ee40 2024-02-21T23:13:33.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.517+0000 7f304cc00640 1 --2- 172.21.15.28:0/814588218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f304806ee40 0x7f3048152180 secure :-1 s=READY pgs=2371 cs=0 l=1 rev1=1 crypto rx=0x7f303800c430 tx=0x7f303800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:33.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.517+0000 7f3045600640 1 -- 172.21.15.28:0/814588218 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3038011790 con 0x7f304806ee40 2024-02-21T23:13:33.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.517+0000 7f3045600640 1 -- 172.21.15.28:0/814588218 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3038017070 con 0x7f304806ee40 2024-02-21T23:13:33.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.517+0000 7f304e000640 1 -- 172.21.15.28:0/814588218 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f304814acc0 con 0x7f304806ee40 2024-02-21T23:13:33.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.517+0000 7f3045600640 1 -- 172.21.15.28:0/814588218 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3038011bb0 con 0x7f304806ee40 2024-02-21T23:13:33.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.517+0000 7f304e000640 1 -- 172.21.15.28:0/814588218 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f304814ca80 con 0x7f304806ee40 2024-02-21T23:13:33.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.519+0000 7f304e000640 1 -- 172.21.15.28:0/814588218 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30480a7390 con 0x7f304806ee40 2024-02-21T23:13:33.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.520+0000 7f3045600640 1 -- 172.21.15.28:0/814588218 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f303801c460 con 0x7f304806ee40 2024-02-21T23:13:33.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.521+0000 7f3045600640 1 --2- 172.21.15.28:0/814588218 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f302406b4d0 0x7f302406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:33.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.521+0000 7f3045600640 1 -- 172.21.15.28:0/814588218 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f303808a1b0 con 0x7f304806ee40 2024-02-21T23:13:33.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.522+0000 7f303fe00640 1 --2- 172.21.15.28:0/814588218 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f302406b4d0 0x7f302406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:33.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.523+0000 7f303fe00640 1 --2- 172.21.15.28:0/814588218 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f302406b4d0 0x7f302406d990 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f30481e89e0 tx=0x7f3030008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:33.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.526+0000 7f3045600640 1 -- 172.21.15.28:0/814588218 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f30480a7390 con 0x7f304806ee40 2024-02-21T23:13:33.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.765+0000 7f304e000640 1 -- 172.21.15.28:0/814588218 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f304804f290 con 0x7f304806ee40 2024-02-21T23:13:33.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.765+0000 7f3045600640 1 -- 172.21.15.28:0/814588218 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 229 v229) v1 ==== 95+0+4948 (secure 0 0 0) 0x7f3038058e90 con 0x7f304806ee40 2024-02-21T23:13:33.754 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:13:33.754 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":229,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7265,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":224},{"gid":8287,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1607587459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1607587459},{"type":"v1","addr":"172.21.15.81:6835","nonce":1607587459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":224},{"gid":8290,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/1864206570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1864206570},{"type":"v1","addr":"172.21.15.28:6833","nonce":1864206570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":224}],"filesystems":[{"mdsmap":{"epoch":229,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:13:32.769089+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:13:33.755 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 229 2024-02-21T23:13:33.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.768+0000 7f304e000640 1 -- 172.21.15.28:0/814588218 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f302406b4d0 msgr2=0x7f302406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:33.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.768+0000 7f304e000640 1 --2- 172.21.15.28:0/814588218 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f302406b4d0 0x7f302406d990 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f30481e89e0 tx=0x7f3030008040 comp rx=0 tx=0).stop 2024-02-21T23:13:33.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.768+0000 7f304e000640 1 -- 172.21.15.28:0/814588218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f304806ee40 msgr2=0x7f3048152180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:33.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.768+0000 7f304e000640 1 --2- 172.21.15.28:0/814588218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f304806ee40 0x7f3048152180 secure :-1 s=READY pgs=2371 cs=0 l=1 rev1=1 crypto rx=0x7f303800c430 tx=0x7f303800c8f0 comp rx=0 tx=0).stop 2024-02-21T23:13:33.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.768+0000 7f304e000640 1 -- 172.21.15.28:0/814588218 shutdown_connections 2024-02-21T23:13:33.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.768+0000 7f304e000640 1 --2- 172.21.15.28:0/814588218 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f302406b4d0 0x7f302406d990 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:33.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.768+0000 7f304e000640 1 --2- 172.21.15.28:0/814588218 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3048150450 0x7f304814cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:33.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.768+0000 7f304e000640 1 --2- 172.21.15.28:0/814588218 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3048156af0 0x7f30481ee380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:33.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.768+0000 7f304e000640 1 --2- 172.21.15.28:0/814588218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f304806ee40 0x7f3048152180 unknown :-1 s=CLOSED pgs=2371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:33.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.768+0000 7f304e000640 1 -- 172.21.15.28:0/814588218 >> 172.21.15.28:0/814588218 conn(0x7f304806b630 msgr2=0x7f304814eb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:33.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.768+0000 7f304e000640 1 -- 172.21.15.28:0/814588218 shutdown_connections 2024-02-21T23:13:33.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.768+0000 7f304e000640 1 -- 172.21.15.28:0/814588218 wait complete. 2024-02-21T23:13:33.774 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:13:33.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.977+0000 7f935f200640 1 Processor -- start 2024-02-21T23:13:33.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.978+0000 7f935f200640 1 -- start start 2024-02-21T23:13:33.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.979+0000 7f935f200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f936006f130 0x7f93600a9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:33.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.979+0000 7f935f200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f936006bad0 0x7f936006f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:33.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.980+0000 7f935f200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f93600c4d20 0x7f9360158580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:33.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.980+0000 7f935f200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f936006b6d0 con 0x7f93600c4d20 2024-02-21T23:13:33.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.980+0000 7f935f200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f936006fc80 con 0x7f936006bad0 2024-02-21T23:13:33.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.980+0000 7f935f200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f936006ff60 con 0x7f936006f130 2024-02-21T23:13:33.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.980+0000 7f935e800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f93600c4d20 0x7f9360158580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:33.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.980+0000 7f935e800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f93600c4d20 0x7f9360158580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33808/0 (socket says 172.21.15.28:33808) 2024-02-21T23:13:33.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.980+0000 7f935e800640 1 -- 172.21.15.28:0/2861608158 learned_addr learned my addr 172.21.15.28:0/2861608158 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:33.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.980+0000 7f935de00640 1 --2- 172.21.15.28:0/2861608158 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f936006f130 0x7f93600a9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:33.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.980+0000 7f9355400640 1 --2- 172.21.15.28:0/2861608158 >> v2:172.21.15.81:3300/0 conn(0x7f936006bad0 0x7f936006f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:33.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.980+0000 7f935e800640 1 -- 172.21.15.28:0/2861608158 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f936006f130 msgr2=0x7f93600a9eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:33.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.980+0000 7f935e800640 1 --2- 172.21.15.28:0/2861608158 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f936006f130 0x7f93600a9eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:33.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.980+0000 7f935e800640 1 -- 172.21.15.28:0/2861608158 >> v2:172.21.15.81:3300/0 conn(0x7f936006bad0 msgr2=0x7f936006f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:33.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.980+0000 7f935e800640 1 --2- 172.21.15.28:0/2861608158 >> v2:172.21.15.81:3300/0 conn(0x7f936006bad0 0x7f936006f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:33.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.981+0000 7f935e800640 1 -- 172.21.15.28:0/2861608158 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9360070240 con 0x7f93600c4d20 2024-02-21T23:13:33.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.982+0000 7f935e800640 1 --2- 172.21.15.28:0/2861608158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93600c4d20 0x7f9360158580 secure :-1 s=READY pgs=2372 cs=0 l=1 rev1=1 crypto rx=0x7f93600a8ed0 tx=0x7f935802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9f2904f9ffa172d server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:33.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.982+0000 7f935d400640 1 -- 172.21.15.28:0/2861608158 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f935802e070 con 0x7f93600c4d20 2024-02-21T23:13:33.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.982+0000 7f935d400640 1 -- 172.21.15.28:0/2861608158 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9358037070 con 0x7f93600c4d20 2024-02-21T23:13:33.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.982+0000 7f935d400640 1 -- 172.21.15.28:0/2861608158 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9358031a00 con 0x7f93600c4d20 2024-02-21T23:13:33.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.983+0000 7f935f200640 1 -- 172.21.15.28:0/2861608158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93600c4d20 msgr2=0x7f9360158580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:33.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.983+0000 7f935f200640 1 --2- 172.21.15.28:0/2861608158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93600c4d20 0x7f9360158580 secure :-1 s=READY pgs=2372 cs=0 l=1 rev1=1 crypto rx=0x7f93600a8ed0 tx=0x7f935802b880 comp rx=0 tx=0).stop 2024-02-21T23:13:33.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.983+0000 7f935f200640 1 -- 172.21.15.28:0/2861608158 shutdown_connections 2024-02-21T23:13:33.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.983+0000 7f935f200640 1 --2- 172.21.15.28:0/2861608158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93600c4d20 0x7f9360158580 unknown :-1 s=CLOSED pgs=2372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:33.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.983+0000 7f935f200640 1 --2- 172.21.15.28:0/2861608158 >> v2:172.21.15.81:3300/0 conn(0x7f936006bad0 0x7f936006f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:33.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.983+0000 7f935f200640 1 --2- 172.21.15.28:0/2861608158 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f936006f130 0x7f93600a9eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:33.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.983+0000 7f935f200640 1 -- 172.21.15.28:0/2861608158 >> 172.21.15.28:0/2861608158 conn(0x7f9360067050 msgr2=0x7f93600a84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:33.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.983+0000 7f935f200640 1 -- 172.21.15.28:0/2861608158 shutdown_connections 2024-02-21T23:13:33.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.983+0000 7f935f200640 1 -- 172.21.15.28:0/2861608158 wait complete. 2024-02-21T23:13:33.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.984+0000 7f935f200640 1 Processor -- start 2024-02-21T23:13:33.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.985+0000 7f935f200640 1 -- start start 2024-02-21T23:13:33.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.985+0000 7f935f200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f936006bad0 0x7f93601de860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:33.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.985+0000 7f935f200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f936006f130 0x7f93600ab310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:33.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.985+0000 7f935f200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f93600c4d20 0x7f93600ab850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:33.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.985+0000 7f935f200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93600a9990 con 0x7f936006f130 2024-02-21T23:13:33.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.986+0000 7f935f200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93600a96b0 con 0x7f93600c4d20 2024-02-21T23:13:33.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.986+0000 7f935f200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93601528d0 con 0x7f936006bad0 2024-02-21T23:13:33.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.986+0000 7f9355400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f936006f130 0x7f93600ab310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:33.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.986+0000 7f9355400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f936006f130 0x7f93600ab310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33810/0 (socket says 172.21.15.28:33810) 2024-02-21T23:13:33.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.986+0000 7f9355400640 1 -- 172.21.15.28:0/2002073873 learned_addr learned my addr 172.21.15.28:0/2002073873 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:33.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.986+0000 7f935de00640 1 --2- 172.21.15.28:0/2002073873 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f936006bad0 0x7f93601de860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:33.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.986+0000 7f935e800640 1 --2- 172.21.15.28:0/2002073873 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f93600c4d20 0x7f93600ab850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:33.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.986+0000 7f9355400640 1 -- 172.21.15.28:0/2002073873 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f936006bad0 msgr2=0x7f93601de860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:33.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.986+0000 7f9355400640 1 --2- 172.21.15.28:0/2002073873 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f936006bad0 0x7f93601de860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:33.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.986+0000 7f9355400640 1 -- 172.21.15.28:0/2002073873 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f93600c4d20 msgr2=0x7f93600ab850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:33.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.986+0000 7f9355400640 1 --2- 172.21.15.28:0/2002073873 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f93600c4d20 0x7f93600ab850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:33.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.986+0000 7f9355400640 1 -- 172.21.15.28:0/2002073873 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93601eb7b0 con 0x7f936006f130 2024-02-21T23:13:33.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.986+0000 7f935e800640 1 --2- 172.21.15.28:0/2002073873 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f93600c4d20 0x7f93600ab850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:13:33.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.986+0000 7f935de00640 1 --2- 172.21.15.28:0/2002073873 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f936006bad0 0x7f93601de860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:13:33.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.987+0000 7f9355400640 1 --2- 172.21.15.28:0/2002073873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f936006f130 0x7f93600ab310 secure :-1 s=READY pgs=2373 cs=0 l=1 rev1=1 crypto rx=0x7f934800d830 tx=0x7f934800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:33.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.987+0000 7f9356a00640 1 -- 172.21.15.28:0/2002073873 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9348018020 con 0x7f936006f130 2024-02-21T23:13:33.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.987+0000 7f9356a00640 1 -- 172.21.15.28:0/2002073873 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93480096a0 con 0x7f936006f130 2024-02-21T23:13:33.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.987+0000 7f9356a00640 1 -- 172.21.15.28:0/2002073873 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9348009a70 con 0x7f936006f130 2024-02-21T23:13:33.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.987+0000 7f935f200640 1 -- 172.21.15.28:0/2002073873 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93600aa550 con 0x7f936006f130 2024-02-21T23:13:33.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.987+0000 7f935f200640 1 -- 172.21.15.28:0/2002073873 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93600aab10 con 0x7f936006f130 2024-02-21T23:13:33.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.988+0000 7f935f200640 1 -- 172.21.15.28:0/2002073873 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f936004f290 con 0x7f936006f130 2024-02-21T23:13:33.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.989+0000 7f9356a00640 1 -- 172.21.15.28:0/2002073873 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f934801b020 con 0x7f936006f130 2024-02-21T23:13:33.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.990+0000 7f9356a00640 1 --2- 172.21.15.28:0/2002073873 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f933c06b4d0 0x7f933c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:33.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.990+0000 7f9356a00640 1 -- 172.21.15.28:0/2002073873 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f934808aae0 con 0x7f936006f130 2024-02-21T23:13:33.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.991+0000 7f935de00640 1 --2- 172.21.15.28:0/2002073873 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f933c06b4d0 0x7f933c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:33.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.992+0000 7f935de00640 1 --2- 172.21.15.28:0/2002073873 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f933c06b4d0 0x7f933c06d990 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f936014bcd0 tx=0x7f935000b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:33.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:33.995+0000 7f9356a00640 1 -- 172.21.15.28:0/2002073873 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f9348022020 con 0x7f936006f130 2024-02-21T23:13:34.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.195+0000 7f935f200640 1 -- 172.21.15.28:0/2002073873 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f93600c48c0 con 0x7f936006f130 2024-02-21T23:13:34.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.196+0000 7f9356a00640 1 -- 172.21.15.28:0/2002073873 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v208) v1 ==== 74+0+16828 (secure 0 0 0) 0x7f9348009e20 con 0x7f936006f130 2024-02-21T23:13:34.185 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:13:34.185 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":208,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:13:31.220694+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:13:27.612392+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"207","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_data","create_time":"2024-02-21T23:13:28.744308+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"208","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:13:34.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.199+0000 7f935f200640 1 -- 172.21.15.28:0/2002073873 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f933c06b4d0 msgr2=0x7f933c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:34.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.199+0000 7f935f200640 1 --2- 172.21.15.28:0/2002073873 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f933c06b4d0 0x7f933c06d990 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f936014bcd0 tx=0x7f935000b040 comp rx=0 tx=0).stop 2024-02-21T23:13:34.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.199+0000 7f935f200640 1 -- 172.21.15.28:0/2002073873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f936006f130 msgr2=0x7f93600ab310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:34.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.199+0000 7f935f200640 1 --2- 172.21.15.28:0/2002073873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f936006f130 0x7f93600ab310 secure :-1 s=READY pgs=2373 cs=0 l=1 rev1=1 crypto rx=0x7f934800d830 tx=0x7f934800dcf0 comp rx=0 tx=0).stop 2024-02-21T23:13:34.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.199+0000 7f935f200640 1 -- 172.21.15.28:0/2002073873 shutdown_connections 2024-02-21T23:13:34.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.199+0000 7f935f200640 1 --2- 172.21.15.28:0/2002073873 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f933c06b4d0 0x7f933c06d990 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:34.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.199+0000 7f935f200640 1 --2- 172.21.15.28:0/2002073873 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f93600c4d20 0x7f93600ab850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:34.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.199+0000 7f935f200640 1 --2- 172.21.15.28:0/2002073873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f936006f130 0x7f93600ab310 unknown :-1 s=CLOSED pgs=2373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:34.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.199+0000 7f935f200640 1 --2- 172.21.15.28:0/2002073873 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f936006bad0 0x7f93601de860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:34.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.199+0000 7f935f200640 1 -- 172.21.15.28:0/2002073873 >> 172.21.15.28:0/2002073873 conn(0x7f9360067050 msgr2=0x7f936014fd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:34.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.200+0000 7f935f200640 1 -- 172.21.15.28:0/2002073873 shutdown_connections 2024-02-21T23:13:34.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.200+0000 7f935f200640 1 -- 172.21.15.28:0/2002073873 wait complete. 2024-02-21T23:13:34.209 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-02-21T23:13:34.209 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-21T23:13:34.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.439+0000 7f92e1400640 1 Processor -- start 2024-02-21T23:13:34.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.440+0000 7f92e1400640 1 -- start start 2024-02-21T23:13:34.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.441+0000 7f92e1400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f92dc0c3280 0x7f92dc0c3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:34.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.441+0000 7f92e1400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f92dc072430 0x7f92dc1526e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:34.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.441+0000 7f92e1400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92dc152c20 con 0x7f92dc155860 2024-02-21T23:13:34.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.441+0000 7f92e1400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92dc0c6560 con 0x7f92dc0c3280 2024-02-21T23:13:34.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.441+0000 7f92e1400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92dc0c5cc0 con 0x7f92dc072430 2024-02-21T23:13:34.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.441+0000 7f92da800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f92dc0c3280 0x7f92dc0c3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:34.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.442+0000 7f92da800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f92dc0c3280 0x7f92dc0c3660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33822/0 (socket says 172.21.15.28:33822) 2024-02-21T23:13:34.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.442+0000 7f92da800640 1 -- 172.21.15.28:0/3738273861 learned_addr learned my addr 172.21.15.28:0/3738273861 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:34.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.442+0000 7f92db200640 1 --1- 172.21.15.28:0/3738273861 >> v1:172.21.15.28:6789/0 conn(0x7f92dc155860 0x7f92dc0c46f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59572/0 (socket says 172.21.15.28:59572) 2024-02-21T23:13:34.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.442+0000 7f92d9e00640 1 --2- 172.21.15.28:0/3738273861 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f92dc072430 0x7f92dc1526e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:34.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.442+0000 7f92da800640 1 -- 172.21.15.28:0/3738273861 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f92dc072430 msgr2=0x7f92dc1526e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:34.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.442+0000 7f92da800640 1 --2- 172.21.15.28:0/3738273861 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f92dc072430 0x7f92dc1526e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:34.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.442+0000 7f92da800640 1 -- 172.21.15.28:0/3738273861 >> v1:172.21.15.28:6789/0 conn(0x7f92dc155860 legacy=0x7f92dc0c46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:34.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.442+0000 7f92da800640 1 -- 172.21.15.28:0/3738273861 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92dc0c5fa0 con 0x7f92dc0c3280 2024-02-21T23:13:34.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.442+0000 7f92d9400640 1 -- 172.21.15.28:0/3738273861 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 664710618 0 0) 0x7f92d4001490 con 0x7f92dc155860 2024-02-21T23:13:34.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.443+0000 7f92da800640 1 --2- 172.21.15.28:0/3738273861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92dc0c3280 0x7f92dc0c3660 secure :-1 s=READY pgs=2375 cs=0 l=1 rev1=1 crypto rx=0x7f92cc009870 tx=0x7f92cc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd36f032cecf5d9d server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:34.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.444+0000 7f92d9400640 1 -- 172.21.15.28:0/3738273861 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92cc035020 con 0x7f92dc0c3280 2024-02-21T23:13:34.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.444+0000 7f92d9400640 1 -- 172.21.15.28:0/3738273861 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92cc031640 con 0x7f92dc0c3280 2024-02-21T23:13:34.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.444+0000 7f92d9400640 1 -- 172.21.15.28:0/3738273861 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92cc02d040 con 0x7f92dc0c3280 2024-02-21T23:13:34.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.444+0000 7f92e1400640 1 -- 172.21.15.28:0/3738273861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92dc0c3280 msgr2=0x7f92dc0c3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:34.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.444+0000 7f92e1400640 1 --2- 172.21.15.28:0/3738273861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92dc0c3280 0x7f92dc0c3660 secure :-1 s=READY pgs=2375 cs=0 l=1 rev1=1 crypto rx=0x7f92cc009870 tx=0x7f92cc02b8f0 comp rx=0 tx=0).stop 2024-02-21T23:13:34.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.444+0000 7f92e1400640 1 -- 172.21.15.28:0/3738273861 shutdown_connections 2024-02-21T23:13:34.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.444+0000 7f92e1400640 1 --2- 172.21.15.28:0/3738273861 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f92dc072430 0x7f92dc1526e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:34.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.444+0000 7f92e1400640 1 --2- 172.21.15.28:0/3738273861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92dc0c3280 0x7f92dc0c3660 unknown :-1 s=CLOSED pgs=2375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:34.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.444+0000 7f92e1400640 1 -- 172.21.15.28:0/3738273861 >> 172.21.15.28:0/3738273861 conn(0x7f92dc0c67f0 msgr2=0x7f92dc0c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:34.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.444+0000 7f92e1400640 1 -- 172.21.15.28:0/3738273861 shutdown_connections 2024-02-21T23:13:34.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.444+0000 7f92e1400640 1 -- 172.21.15.28:0/3738273861 wait complete. 2024-02-21T23:13:34.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.446+0000 7f92e1400640 1 Processor -- start 2024-02-21T23:13:34.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.446+0000 7f92e1400640 1 -- start start 2024-02-21T23:13:34.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.447+0000 7f92e1400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f92dc072430 0x7f92dc151050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:34.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.447+0000 7f92e1400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f92dc0c3280 0x7f92dc14da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:34.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.447+0000 7f92e1400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92dc155860 0x7f92dc14dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:34.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.447+0000 7f92e1400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92dc0c5920 con 0x7f92dc155860 2024-02-21T23:13:34.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.447+0000 7f92e1400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92dc0a6100 con 0x7f92dc0c3280 2024-02-21T23:13:34.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.447+0000 7f92e1400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92dc0c5aa0 con 0x7f92dc072430 2024-02-21T23:13:34.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.447+0000 7f92db200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92dc155860 0x7f92dc14dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:34.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.448+0000 7f92db200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92dc155860 0x7f92dc14dfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33824/0 (socket says 172.21.15.28:33824) 2024-02-21T23:13:34.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.448+0000 7f92db200640 1 -- 172.21.15.28:0/3994525810 learned_addr learned my addr 172.21.15.28:0/3994525810 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:34.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.448+0000 7f92da800640 1 --2- 172.21.15.28:0/3994525810 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f92dc072430 0x7f92dc151050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:34.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.448+0000 7f92d9e00640 1 --2- 172.21.15.28:0/3994525810 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f92dc0c3280 0x7f92dc14da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:34.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.448+0000 7f92db200640 1 -- 172.21.15.28:0/3994525810 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f92dc072430 msgr2=0x7f92dc151050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:34.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.448+0000 7f92db200640 1 --2- 172.21.15.28:0/3994525810 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f92dc072430 0x7f92dc151050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:34.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.448+0000 7f92db200640 1 -- 172.21.15.28:0/3994525810 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f92dc0c3280 msgr2=0x7f92dc14da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:34.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.448+0000 7f92db200640 1 --2- 172.21.15.28:0/3994525810 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f92dc0c3280 0x7f92dc14da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:34.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.448+0000 7f92db200640 1 -- 172.21.15.28:0/3994525810 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92dc15f050 con 0x7f92dc155860 2024-02-21T23:13:34.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.448+0000 7f92db200640 1 --2- 172.21.15.28:0/3994525810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92dc155860 0x7f92dc14dfd0 secure :-1 s=READY pgs=2376 cs=0 l=1 rev1=1 crypto rx=0x7f92d400c730 tx=0x7f92d400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:34.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.449+0000 7f92d2a00640 1 -- 172.21.15.28:0/3994525810 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92d4018070 con 0x7f92dc155860 2024-02-21T23:13:34.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.449+0000 7f92d2a00640 1 -- 172.21.15.28:0/3994525810 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92d400ce40 con 0x7f92dc155860 2024-02-21T23:13:34.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.449+0000 7f92d2a00640 1 -- 172.21.15.28:0/3994525810 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92d400a880 con 0x7f92dc155860 2024-02-21T23:13:34.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.449+0000 7f92e1400640 1 -- 172.21.15.28:0/3994525810 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92dc14cfb0 con 0x7f92dc155860 2024-02-21T23:13:34.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.449+0000 7f92e1400640 1 -- 172.21.15.28:0/3994525810 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92dc151590 con 0x7f92dc155860 2024-02-21T23:13:34.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.450+0000 7f92e1400640 1 -- 172.21.15.28:0/3994525810 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92dc14cda0 con 0x7f92dc155860 2024-02-21T23:13:34.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.451+0000 7f92d2a00640 1 -- 172.21.15.28:0/3994525810 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f92d400aa20 con 0x7f92dc155860 2024-02-21T23:13:34.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.453+0000 7f92d2a00640 1 --2- 172.21.15.28:0/3994525810 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f92bc06b4d0 0x7f92bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:34.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.453+0000 7f92d2a00640 1 -- 172.21.15.28:0/3994525810 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f92d40898a0 con 0x7f92dc155860 2024-02-21T23:13:34.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.453+0000 7f92da800640 1 --2- 172.21.15.28:0/3994525810 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f92bc06b4d0 0x7f92bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:34.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.455+0000 7f92da800640 1 --2- 172.21.15.28:0/3994525810 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f92bc06b4d0 0x7f92bc06d990 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f92cc036000 tx=0x7f92cc008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:34.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.456+0000 7f92d2a00640 1 -- 172.21.15.28:0/3994525810 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f92dc152c20 con 0x7f92dc155860 2024-02-21T23:13:34.622 INFO:tasks.ceph.mds.c.smithi028.stderr:2024-02-21T23:13:34.633+0000 7fc87a200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-21T23:13:34.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.653+0000 7f92e1400640 1 -- 172.21.15.28:0/3994525810 --> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f92dc067f30 con 0x7f92bc06b4d0 2024-02-21T23:13:34.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.671+0000 7f92d2a00640 1 -- 172.21.15.28:0/3994525810 <== mgr.4101 v2:172.21.15.81:6832/1848538577 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252507 (secure 0 0 0) 0x7f92dc067f30 con 0x7f92bc06b4d0 2024-02-21T23:13:34.663 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:13:34.665 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-21T23:13:34.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.677+0000 7f92e1400640 1 -- 172.21.15.28:0/3994525810 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f92bc06b4d0 msgr2=0x7f92bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:34.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.677+0000 7f92e1400640 1 --2- 172.21.15.28:0/3994525810 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f92bc06b4d0 0x7f92bc06d990 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f92cc036000 tx=0x7f92cc008030 comp rx=0 tx=0).stop 2024-02-21T23:13:34.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.677+0000 7f92e1400640 1 -- 172.21.15.28:0/3994525810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92dc155860 msgr2=0x7f92dc14dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:34.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.677+0000 7f92e1400640 1 --2- 172.21.15.28:0/3994525810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92dc155860 0x7f92dc14dfd0 secure :-1 s=READY pgs=2376 cs=0 l=1 rev1=1 crypto rx=0x7f92d400c730 tx=0x7f92d400cbf0 comp rx=0 tx=0).stop 2024-02-21T23:13:34.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.677+0000 7f92e1400640 1 -- 172.21.15.28:0/3994525810 shutdown_connections 2024-02-21T23:13:34.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.677+0000 7f92e1400640 1 --2- 172.21.15.28:0/3994525810 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f92bc06b4d0 0x7f92bc06d990 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:34.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.677+0000 7f92e1400640 1 --2- 172.21.15.28:0/3994525810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92dc155860 0x7f92dc14dfd0 unknown :-1 s=CLOSED pgs=2376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:34.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.677+0000 7f92e1400640 1 --2- 172.21.15.28:0/3994525810 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f92dc0c3280 0x7f92dc14da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:34.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.677+0000 7f92e1400640 1 --2- 172.21.15.28:0/3994525810 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f92dc072430 0x7f92dc151050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:34.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.677+0000 7f92e1400640 1 -- 172.21.15.28:0/3994525810 >> 172.21.15.28:0/3994525810 conn(0x7f92dc0c67f0 msgr2=0x7f92dc152e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:34.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.677+0000 7f92e1400640 1 -- 172.21.15.28:0/3994525810 shutdown_connections 2024-02-21T23:13:34.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.677+0000 7f92e1400640 1 -- 172.21.15.28:0/3994525810 wait complete. 2024-02-21T23:13:34.685 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":532,"stamp":"2024-02-21T23:13:32.901057+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":138,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":286788,"kb_used_data":20692,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754687932,"statfs":{"total":773094113280,"available":772800442368,"internally_reserved":0,"allocated":21188608,"data_stored":11463232,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":272354865},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.687203"},"pg_stats":[{"pgid":"34.3f","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.266624+0000","last_change":"2024-02-21T23:13:30.263436+0000","last_active":"2024-02-21T23:13:30.266624+0000","last_peered":"2024-02-21T23:13:30.266624+0000","last_clean":"2024-02-21T23:13:30.266624+0000","last_became_active":"2024-02-21T23:13:30.263101+0000","last_became_peered":"2024-02-21T23:13:30.263101+0000","last_unstale":"2024-02-21T23:13:30.266624+0000","last_undegraded":"2024-02-21T23:13:30.266624+0000","last_fullsized":"2024-02-21T23:13:30.266624+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:21:53.012342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.3e","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.262953+0000","last_change":"2024-02-21T23:13:30.259389+0000","last_active":"2024-02-21T23:13:30.262953+0000","last_peered":"2024-02-21T23:13:30.262953+0000","last_clean":"2024-02-21T23:13:30.262953+0000","last_became_active":"2024-02-21T23:13:30.259027+0000","last_became_peered":"2024-02-21T23:13:30.259027+0000","last_unstale":"2024-02-21T23:13:30.262953+0000","last_undegraded":"2024-02-21T23:13:30.262953+0000","last_fullsized":"2024-02-21T23:13:30.262953+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:23:05.996217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.3d","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.269473+0000","last_change":"2024-02-21T23:13:30.265498+0000","last_active":"2024-02-21T23:13:30.269473+0000","last_peered":"2024-02-21T23:13:30.269473+0000","last_clean":"2024-02-21T23:13:30.269473+0000","last_became_active":"2024-02-21T23:13:30.264675+0000","last_became_peered":"2024-02-21T23:13:30.264675+0000","last_unstale":"2024-02-21T23:13:30.269473+0000","last_undegraded":"2024-02-21T23:13:30.269473+0000","last_fullsized":"2024-02-21T23:13:30.269473+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:04:42.692957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.3c","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.267407+0000","last_change":"2024-02-21T23:13:30.260235+0000","last_active":"2024-02-21T23:13:30.267407+0000","last_peered":"2024-02-21T23:13:30.267407+0000","last_clean":"2024-02-21T23:13:30.267407+0000","last_became_active":"2024-02-21T23:13:30.259600+0000","last_became_peered":"2024-02-21T23:13:30.259600+0000","last_unstale":"2024-02-21T23:13:30.267407+0000","last_undegraded":"2024-02-21T23:13:30.267407+0000","last_fullsized":"2024-02-21T23:13:30.267407+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T11:04:39.059962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.3b","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.270250+0000","last_change":"2024-02-21T23:13:30.268330+0000","last_active":"2024-02-21T23:13:30.270250+0000","last_peered":"2024-02-21T23:13:30.270250+0000","last_clean":"2024-02-21T23:13:30.270250+0000","last_became_active":"2024-02-21T23:13:30.267457+0000","last_became_peered":"2024-02-21T23:13:30.267457+0000","last_unstale":"2024-02-21T23:13:30.270250+0000","last_undegraded":"2024-02-21T23:13:30.270250+0000","last_fullsized":"2024-02-21T23:13:30.270250+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:57:39.946172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.3a","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.269796+0000","last_change":"2024-02-21T23:13:30.264250+0000","last_active":"2024-02-21T23:13:30.269796+0000","last_peered":"2024-02-21T23:13:30.269796+0000","last_clean":"2024-02-21T23:13:30.269796+0000","last_became_active":"2024-02-21T23:13:30.263984+0000","last_became_peered":"2024-02-21T23:13:30.263984+0000","last_unstale":"2024-02-21T23:13:30.269796+0000","last_undegraded":"2024-02-21T23:13:30.269796+0000","last_fullsized":"2024-02-21T23:13:30.269796+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:54:21.479946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.39","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.264545+0000","last_change":"2024-02-21T23:13:30.257367+0000","last_active":"2024-02-21T23:13:30.264545+0000","last_peered":"2024-02-21T23:13:30.264545+0000","last_clean":"2024-02-21T23:13:30.264545+0000","last_became_active":"2024-02-21T23:13:30.257066+0000","last_became_peered":"2024-02-21T23:13:30.257066+0000","last_unstale":"2024-02-21T23:13:30.264545+0000","last_undegraded":"2024-02-21T23:13:30.264545+0000","last_fullsized":"2024-02-21T23:13:30.264545+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:43:37.680664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.38","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.267432+0000","last_change":"2024-02-21T23:13:30.257613+0000","last_active":"2024-02-21T23:13:30.267432+0000","last_peered":"2024-02-21T23:13:30.267432+0000","last_clean":"2024-02-21T23:13:30.267432+0000","last_became_active":"2024-02-21T23:13:30.256432+0000","last_became_peered":"2024-02-21T23:13:30.256432+0000","last_unstale":"2024-02-21T23:13:30.267432+0000","last_undegraded":"2024-02-21T23:13:30.267432+0000","last_fullsized":"2024-02-21T23:13:30.267432+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:57:56.254791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.37","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.271392+0000","last_change":"2024-02-21T23:13:30.267058+0000","last_active":"2024-02-21T23:13:30.271392+0000","last_peered":"2024-02-21T23:13:30.271392+0000","last_clean":"2024-02-21T23:13:30.271392+0000","last_became_active":"2024-02-21T23:13:30.266768+0000","last_became_peered":"2024-02-21T23:13:30.266768+0000","last_unstale":"2024-02-21T23:13:30.271392+0000","last_undegraded":"2024-02-21T23:13:30.271392+0000","last_fullsized":"2024-02-21T23:13:30.271392+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:45:51.661110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.36","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.261245+0000","last_change":"2024-02-21T23:13:30.255320+0000","last_active":"2024-02-21T23:13:30.261245+0000","last_peered":"2024-02-21T23:13:30.261245+0000","last_clean":"2024-02-21T23:13:30.261245+0000","last_became_active":"2024-02-21T23:13:30.254279+0000","last_became_peered":"2024-02-21T23:13:30.254279+0000","last_unstale":"2024-02-21T23:13:30.261245+0000","last_undegraded":"2024-02-21T23:13:30.261245+0000","last_fullsized":"2024-02-21T23:13:30.261245+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:07:44.983277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.35","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.265360+0000","last_change":"2024-02-21T23:13:30.257020+0000","last_active":"2024-02-21T23:13:30.265360+0000","last_peered":"2024-02-21T23:13:30.265360+0000","last_clean":"2024-02-21T23:13:30.265360+0000","last_became_active":"2024-02-21T23:13:30.255810+0000","last_became_peered":"2024-02-21T23:13:30.255810+0000","last_unstale":"2024-02-21T23:13:30.265360+0000","last_undegraded":"2024-02-21T23:13:30.265360+0000","last_fullsized":"2024-02-21T23:13:30.265360+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:41:00.724045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.34","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.274320+0000","last_change":"2024-02-21T23:13:30.270247+0000","last_active":"2024-02-21T23:13:30.274320+0000","last_peered":"2024-02-21T23:13:30.274320+0000","last_clean":"2024-02-21T23:13:30.274320+0000","last_became_active":"2024-02-21T23:13:30.268715+0000","last_became_peered":"2024-02-21T23:13:30.268715+0000","last_unstale":"2024-02-21T23:13:30.274320+0000","last_undegraded":"2024-02-21T23:13:30.274320+0000","last_fullsized":"2024-02-21T23:13:30.274320+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:59:38.547353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.33","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.268021+0000","last_change":"2024-02-21T23:13:30.262537+0000","last_active":"2024-02-21T23:13:30.268021+0000","last_peered":"2024-02-21T23:13:30.268021+0000","last_clean":"2024-02-21T23:13:30.268021+0000","last_became_active":"2024-02-21T23:13:30.257067+0000","last_became_peered":"2024-02-21T23:13:30.257067+0000","last_unstale":"2024-02-21T23:13:30.268021+0000","last_undegraded":"2024-02-21T23:13:30.268021+0000","last_fullsized":"2024-02-21T23:13:30.268021+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:49:43.777047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.32","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.280783+0000","last_change":"2024-02-21T23:13:30.277793+0000","last_active":"2024-02-21T23:13:30.280783+0000","last_peered":"2024-02-21T23:13:30.280783+0000","last_clean":"2024-02-21T23:13:30.280783+0000","last_became_active":"2024-02-21T23:13:30.270736+0000","last_became_peered":"2024-02-21T23:13:30.270736+0000","last_unstale":"2024-02-21T23:13:30.280783+0000","last_undegraded":"2024-02-21T23:13:30.280783+0000","last_fullsized":"2024-02-21T23:13:30.280783+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:28:32.097867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.31","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.278205+0000","last_change":"2024-02-21T23:13:30.263971+0000","last_active":"2024-02-21T23:13:30.278205+0000","last_peered":"2024-02-21T23:13:30.278205+0000","last_clean":"2024-02-21T23:13:30.278205+0000","last_became_active":"2024-02-21T23:13:30.262598+0000","last_became_peered":"2024-02-21T23:13:30.262598+0000","last_unstale":"2024-02-21T23:13:30.278205+0000","last_undegraded":"2024-02-21T23:13:30.278205+0000","last_fullsized":"2024-02-21T23:13:30.278205+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:09:59.268584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.30","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.268922+0000","last_change":"2024-02-21T23:13:30.263522+0000","last_active":"2024-02-21T23:13:30.268922+0000","last_peered":"2024-02-21T23:13:30.268922+0000","last_clean":"2024-02-21T23:13:30.268922+0000","last_became_active":"2024-02-21T23:13:30.262689+0000","last_became_peered":"2024-02-21T23:13:30.262689+0000","last_unstale":"2024-02-21T23:13:30.268922+0000","last_undegraded":"2024-02-21T23:13:30.268922+0000","last_fullsized":"2024-02-21T23:13:30.268922+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:59:38.547353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.2f","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.259832+0000","last_change":"2024-02-21T23:13:30.254805+0000","last_active":"2024-02-21T23:13:30.259832+0000","last_peered":"2024-02-21T23:13:30.259832+0000","last_clean":"2024-02-21T23:13:30.259832+0000","last_became_active":"2024-02-21T23:13:30.252766+0000","last_became_peered":"2024-02-21T23:13:30.252766+0000","last_unstale":"2024-02-21T23:13:30.259832+0000","last_undegraded":"2024-02-21T23:13:30.259832+0000","last_fullsized":"2024-02-21T23:13:30.259832+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:45:51.661110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.2e","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.254772+0000","last_change":"2024-02-21T23:13:30.250176+0000","last_active":"2024-02-21T23:13:30.254772+0000","last_peered":"2024-02-21T23:13:30.254772+0000","last_clean":"2024-02-21T23:13:30.254772+0000","last_became_active":"2024-02-21T23:13:30.249538+0000","last_became_peered":"2024-02-21T23:13:30.249538+0000","last_unstale":"2024-02-21T23:13:30.254772+0000","last_undegraded":"2024-02-21T23:13:30.254772+0000","last_fullsized":"2024-02-21T23:13:30.254772+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:17:53.761214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.2d","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.259732+0000","last_change":"2024-02-21T23:13:30.254034+0000","last_active":"2024-02-21T23:13:30.259732+0000","last_peered":"2024-02-21T23:13:30.259732+0000","last_clean":"2024-02-21T23:13:30.259732+0000","last_became_active":"2024-02-21T23:13:30.253625+0000","last_became_peered":"2024-02-21T23:13:30.253625+0000","last_unstale":"2024-02-21T23:13:30.259732+0000","last_undegraded":"2024-02-21T23:13:30.259732+0000","last_fullsized":"2024-02-21T23:13:30.259732+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:59:26.281603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.2c","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.274290+0000","last_change":"2024-02-21T23:13:30.269988+0000","last_active":"2024-02-21T23:13:30.274290+0000","last_peered":"2024-02-21T23:13:30.274290+0000","last_clean":"2024-02-21T23:13:30.274290+0000","last_became_active":"2024-02-21T23:13:30.268471+0000","last_became_peered":"2024-02-21T23:13:30.268471+0000","last_unstale":"2024-02-21T23:13:30.274290+0000","last_undegraded":"2024-02-21T23:13:30.274290+0000","last_fullsized":"2024-02-21T23:13:30.274290+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:07:44.983277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.2b","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.270272+0000","last_change":"2024-02-21T23:13:30.268351+0000","last_active":"2024-02-21T23:13:30.270272+0000","last_peered":"2024-02-21T23:13:30.270272+0000","last_clean":"2024-02-21T23:13:30.270272+0000","last_became_active":"2024-02-21T23:13:30.267688+0000","last_became_peered":"2024-02-21T23:13:30.267688+0000","last_unstale":"2024-02-21T23:13:30.270272+0000","last_undegraded":"2024-02-21T23:13:30.270272+0000","last_fullsized":"2024-02-21T23:13:30.270272+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:13:06.536763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.2a","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.270975+0000","last_change":"2024-02-21T23:13:30.266234+0000","last_active":"2024-02-21T23:13:30.270975+0000","last_peered":"2024-02-21T23:13:30.270975+0000","last_clean":"2024-02-21T23:13:30.270975+0000","last_became_active":"2024-02-21T23:13:30.264541+0000","last_became_peered":"2024-02-21T23:13:30.264541+0000","last_unstale":"2024-02-21T23:13:30.270975+0000","last_undegraded":"2024-02-21T23:13:30.270975+0000","last_fullsized":"2024-02-21T23:13:30.270975+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:34:51.167753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.29","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.280778+0000","last_change":"2024-02-21T23:13:30.277544+0000","last_active":"2024-02-21T23:13:30.280778+0000","last_peered":"2024-02-21T23:13:30.280778+0000","last_clean":"2024-02-21T23:13:30.280778+0000","last_became_active":"2024-02-21T23:13:30.268711+0000","last_became_peered":"2024-02-21T23:13:30.268711+0000","last_unstale":"2024-02-21T23:13:30.280778+0000","last_undegraded":"2024-02-21T23:13:30.280778+0000","last_fullsized":"2024-02-21T23:13:30.280778+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:49:38.979105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.28","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.261093+0000","last_change":"2024-02-21T23:13:30.251900+0000","last_active":"2024-02-21T23:13:30.261093+0000","last_peered":"2024-02-21T23:13:30.261093+0000","last_clean":"2024-02-21T23:13:30.261093+0000","last_became_active":"2024-02-21T23:13:30.251561+0000","last_became_peered":"2024-02-21T23:13:30.251561+0000","last_unstale":"2024-02-21T23:13:30.261093+0000","last_undegraded":"2024-02-21T23:13:30.261093+0000","last_fullsized":"2024-02-21T23:13:30.261093+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:36:34.193446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.27","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.261587+0000","last_change":"2024-02-21T23:13:30.255408+0000","last_active":"2024-02-21T23:13:30.261587+0000","last_peered":"2024-02-21T23:13:30.261587+0000","last_clean":"2024-02-21T23:13:30.261587+0000","last_became_active":"2024-02-21T23:13:30.255120+0000","last_became_peered":"2024-02-21T23:13:30.255120+0000","last_unstale":"2024-02-21T23:13:30.261587+0000","last_undegraded":"2024-02-21T23:13:30.261587+0000","last_fullsized":"2024-02-21T23:13:30.261587+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:48:44.645285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.26","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.271885+0000","last_change":"2024-02-21T23:13:30.263046+0000","last_active":"2024-02-21T23:13:30.271885+0000","last_peered":"2024-02-21T23:13:30.271885+0000","last_clean":"2024-02-21T23:13:30.271885+0000","last_became_active":"2024-02-21T23:13:30.262494+0000","last_became_peered":"2024-02-21T23:13:30.262494+0000","last_unstale":"2024-02-21T23:13:30.271885+0000","last_undegraded":"2024-02-21T23:13:30.271885+0000","last_fullsized":"2024-02-21T23:13:30.271885+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:53:26.841237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.25","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.271053+0000","last_change":"2024-02-21T23:13:30.265968+0000","last_active":"2024-02-21T23:13:30.271053+0000","last_peered":"2024-02-21T23:13:30.271053+0000","last_clean":"2024-02-21T23:13:30.271053+0000","last_became_active":"2024-02-21T23:13:30.264342+0000","last_became_peered":"2024-02-21T23:13:30.264342+0000","last_unstale":"2024-02-21T23:13:30.271053+0000","last_undegraded":"2024-02-21T23:13:30.271053+0000","last_fullsized":"2024-02-21T23:13:30.271053+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:19:42.812708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.24","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.267130+0000","last_change":"2024-02-21T23:13:30.263976+0000","last_active":"2024-02-21T23:13:30.267130+0000","last_peered":"2024-02-21T23:13:30.267130+0000","last_clean":"2024-02-21T23:13:30.267130+0000","last_became_active":"2024-02-21T23:13:30.263566+0000","last_became_peered":"2024-02-21T23:13:30.263566+0000","last_unstale":"2024-02-21T23:13:30.267130+0000","last_undegraded":"2024-02-21T23:13:30.267130+0000","last_fullsized":"2024-02-21T23:13:30.267130+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:17:28.420304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.22","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.269911+0000","last_change":"2024-02-21T23:13:30.266130+0000","last_active":"2024-02-21T23:13:30.269911+0000","last_peered":"2024-02-21T23:13:30.269911+0000","last_clean":"2024-02-21T23:13:30.269911+0000","last_became_active":"2024-02-21T23:13:30.262419+0000","last_became_peered":"2024-02-21T23:13:30.262419+0000","last_unstale":"2024-02-21T23:13:30.269911+0000","last_undegraded":"2024-02-21T23:13:30.269911+0000","last_fullsized":"2024-02-21T23:13:30.269911+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:59:26.281603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.21","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.278298+0000","last_change":"2024-02-21T23:13:30.270320+0000","last_active":"2024-02-21T23:13:30.278298+0000","last_peered":"2024-02-21T23:13:30.278298+0000","last_clean":"2024-02-21T23:13:30.278298+0000","last_became_active":"2024-02-21T23:13:30.270008+0000","last_became_peered":"2024-02-21T23:13:30.270008+0000","last_unstale":"2024-02-21T23:13:30.278298+0000","last_undegraded":"2024-02-21T23:13:30.278298+0000","last_fullsized":"2024-02-21T23:13:30.278298+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:49:57.308010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.20","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.282744+0000","last_change":"2024-02-21T23:13:30.278812+0000","last_active":"2024-02-21T23:13:30.282744+0000","last_peered":"2024-02-21T23:13:30.282744+0000","last_clean":"2024-02-21T23:13:30.282744+0000","last_became_active":"2024-02-21T23:13:30.278374+0000","last_became_peered":"2024-02-21T23:13:30.278374+0000","last_unstale":"2024-02-21T23:13:30.282744+0000","last_undegraded":"2024-02-21T23:13:30.282744+0000","last_fullsized":"2024-02-21T23:13:30.282744+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:30:31.245779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.1c","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.269400+0000","last_change":"2024-02-21T23:13:30.263843+0000","last_active":"2024-02-21T23:13:30.269400+0000","last_peered":"2024-02-21T23:13:30.269400+0000","last_clean":"2024-02-21T23:13:30.269400+0000","last_became_active":"2024-02-21T23:13:30.263555+0000","last_became_peered":"2024-02-21T23:13:30.263555+0000","last_unstale":"2024-02-21T23:13:30.269400+0000","last_undegraded":"2024-02-21T23:13:30.269400+0000","last_fullsized":"2024-02-21T23:13:30.269400+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:48:44.645285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.1f","version":"207'4","reported_seq":16,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.285881+0000","last_change":"2024-02-21T23:13:29.257460+0000","last_active":"2024-02-21T23:13:30.285881+0000","last_peered":"2024-02-21T23:13:30.285881+0000","last_clean":"2024-02-21T23:13:30.285881+0000","last_became_active":"2024-02-21T23:13:29.257196+0000","last_became_peered":"2024-02-21T23:13:29.257196+0000","last_unstale":"2024-02-21T23:13:30.285881+0000","last_undegraded":"2024-02-21T23:13:30.285881+0000","last_fullsized":"2024-02-21T23:13:30.285881+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_clean_scrub_stamp":"2024-02-21T23:13:28.197327+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:11:17.997693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.1d","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.256658+0000","last_change":"2024-02-21T23:13:30.253331+0000","last_active":"2024-02-21T23:13:30.256658+0000","last_peered":"2024-02-21T23:13:30.256658+0000","last_clean":"2024-02-21T23:13:30.256658+0000","last_became_active":"2024-02-21T23:13:30.252998+0000","last_became_peered":"2024-02-21T23:13:30.252998+0000","last_unstale":"2024-02-21T23:13:30.256658+0000","last_undegraded":"2024-02-21T23:13:30.256658+0000","last_fullsized":"2024-02-21T23:13:30.256658+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:23:55.545925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1e","version":"207'1","reported_seq":13,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.276595+0000","last_change":"2024-02-21T23:13:29.269012+0000","last_active":"2024-02-21T23:13:30.276595+0000","last_peered":"2024-02-21T23:13:30.276595+0000","last_clean":"2024-02-21T23:13:30.276595+0000","last_became_active":"2024-02-21T23:13:29.268669+0000","last_became_peered":"2024-02-21T23:13:29.268669+0000","last_unstale":"2024-02-21T23:13:30.276595+0000","last_undegraded":"2024-02-21T23:13:30.276595+0000","last_fullsized":"2024-02-21T23:13:30.276595+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_clean_scrub_stamp":"2024-02-21T23:13:28.197327+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:09:33.639743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.1e","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.242448+0000","last_change":"2024-02-21T23:13:30.240587+0000","last_active":"2024-02-21T23:13:30.242448+0000","last_peered":"2024-02-21T23:13:30.242448+0000","last_clean":"2024-02-21T23:13:30.242448+0000","last_became_active":"2024-02-21T23:13:30.239778+0000","last_became_peered":"2024-02-21T23:13:30.239778+0000","last_unstale":"2024-02-21T23:13:30.242448+0000","last_undegraded":"2024-02-21T23:13:30.242448+0000","last_fullsized":"2024-02-21T23:13:30.242448+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:43:45.804413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1d","version":"207'2","reported_seq":14,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.285949+0000","last_change":"2024-02-21T23:13:29.260853+0000","last_active":"2024-02-21T23:13:30.285949+0000","last_peered":"2024-02-21T23:13:30.285949+0000","last_clean":"2024-02-21T23:13:30.285949+0000","last_became_active":"2024-02-21T23:13:29.260225+0000","last_became_peered":"2024-02-21T23:13:29.260225+0000","last_unstale":"2024-02-21T23:13:30.285949+0000","last_undegraded":"2024-02-21T23:13:30.285949+0000","last_fullsized":"2024-02-21T23:13:30.285949+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_clean_scrub_stamp":"2024-02-21T23:13:28.197327+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:57:27.330618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.1f","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.281748+0000","last_change":"2024-02-21T23:13:30.280231+0000","last_active":"2024-02-21T23:13:30.281748+0000","last_peered":"2024-02-21T23:13:30.281748+0000","last_clean":"2024-02-21T23:13:30.281748+0000","last_became_active":"2024-02-21T23:13:30.271571+0000","last_became_peered":"2024-02-21T23:13:30.271571+0000","last_unstale":"2024-02-21T23:13:30.281748+0000","last_undegraded":"2024-02-21T23:13:30.281748+0000","last_fullsized":"2024-02-21T23:13:30.281748+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:14:29.166217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1c","version":"207'2","reported_seq":14,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.279895+0000","last_change":"2024-02-21T23:13:29.264981+0000","last_active":"2024-02-21T23:13:30.279895+0000","last_peered":"2024-02-21T23:13:30.279895+0000","last_clean":"2024-02-21T23:13:30.279895+0000","last_became_active":"2024-02-21T23:13:29.264527+0000","last_became_peered":"2024-02-21T23:13:29.264527+0000","last_unstale":"2024-02-21T23:13:30.279895+0000","last_undegraded":"2024-02-21T23:13:30.279895+0000","last_fullsized":"2024-02-21T23:13:30.279895+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_clean_scrub_stamp":"2024-02-21T23:13:28.197327+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:54:20.464021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.18","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.278568+0000","last_change":"2024-02-21T23:13:30.263784+0000","last_active":"2024-02-21T23:13:30.278568+0000","last_peered":"2024-02-21T23:13:30.278568+0000","last_clean":"2024-02-21T23:13:30.278568+0000","last_became_active":"2024-02-21T23:13:30.263191+0000","last_became_peered":"2024-02-21T23:13:30.263191+0000","last_unstale":"2024-02-21T23:13:30.278568+0000","last_undegraded":"2024-02-21T23:13:30.278568+0000","last_fullsized":"2024-02-21T23:13:30.278568+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:13:06.536763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1b","version":"207'1","reported_seq":13,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.277261+0000","last_change":"2024-02-21T23:13:29.262692+0000","last_active":"2024-02-21T23:13:30.277261+0000","last_peered":"2024-02-21T23:13:30.277261+0000","last_clean":"2024-02-21T23:13:30.277261+0000","last_became_active":"2024-02-21T23:13:29.262396+0000","last_became_peered":"2024-02-21T23:13:29.262396+0000","last_unstale":"2024-02-21T23:13:30.277261+0000","last_undegraded":"2024-02-21T23:13:30.277261+0000","last_fullsized":"2024-02-21T23:13:30.277261+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_clean_scrub_stamp":"2024-02-21T23:13:28.197327+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:55:08.180278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.19","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.278253+0000","last_change":"2024-02-21T23:13:30.264445+0000","last_active":"2024-02-21T23:13:30.278253+0000","last_peered":"2024-02-21T23:13:30.278253+0000","last_clean":"2024-02-21T23:13:30.278253+0000","last_became_active":"2024-02-21T23:13:30.260450+0000","last_became_peered":"2024-02-21T23:13:30.260450+0000","last_unstale":"2024-02-21T23:13:30.278253+0000","last_undegraded":"2024-02-21T23:13:30.278253+0000","last_fullsized":"2024-02-21T23:13:30.278253+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:05:15.802970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1a","version":"207'1","reported_seq":13,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.288937+0000","last_change":"2024-02-21T23:13:29.263680+0000","last_active":"2024-02-21T23:13:30.288937+0000","last_peered":"2024-02-21T23:13:30.288937+0000","last_clean":"2024-02-21T23:13:30.288937+0000","last_became_active":"2024-02-21T23:13:29.263017+0000","last_became_peered":"2024-02-21T23:13:29.263017+0000","last_unstale":"2024-02-21T23:13:30.288937+0000","last_undegraded":"2024-02-21T23:13:30.288937+0000","last_fullsized":"2024-02-21T23:13:30.288937+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_clean_scrub_stamp":"2024-02-21T23:13:28.197327+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:43:30.772438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.1a","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.269527+0000","last_change":"2024-02-21T23:13:30.265727+0000","last_active":"2024-02-21T23:13:30.269527+0000","last_peered":"2024-02-21T23:13:30.269527+0000","last_clean":"2024-02-21T23:13:30.269527+0000","last_became_active":"2024-02-21T23:13:30.264020+0000","last_became_peered":"2024-02-21T23:13:30.264020+0000","last_unstale":"2024-02-21T23:13:30.269527+0000","last_undegraded":"2024-02-21T23:13:30.269527+0000","last_fullsized":"2024-02-21T23:13:30.269527+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:27:53.857422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.19","version":"0'0","reported_seq":12,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.251087+0000","last_change":"2024-02-21T23:13:29.265683+0000","last_active":"2024-02-21T23:13:30.251087+0000","last_peered":"2024-02-21T23:13:30.251087+0000","last_clean":"2024-02-21T23:13:30.251087+0000","last_became_active":"2024-02-21T23:13:29.265362+0000","last_became_peered":"2024-02-21T23:13:29.265362+0000","last_unstale":"2024-02-21T23:13:30.251087+0000","last_undegraded":"2024-02-21T23:13:30.251087+0000","last_fullsized":"2024-02-21T23:13:30.251087+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_clean_scrub_stamp":"2024-02-21T23:13:28.197327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:52:02.987184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.1b","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.268001+0000","last_change":"2024-02-21T23:13:30.257729+0000","last_active":"2024-02-21T23:13:30.268001+0000","last_peered":"2024-02-21T23:13:30.268001+0000","last_clean":"2024-02-21T23:13:30.268001+0000","last_became_active":"2024-02-21T23:13:30.250419+0000","last_became_peered":"2024-02-21T23:13:30.250419+0000","last_unstale":"2024-02-21T23:13:30.268001+0000","last_undegraded":"2024-02-21T23:13:30.268001+0000","last_fullsized":"2024-02-21T23:13:30.268001+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:05:45.588776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.18","version":"0'0","reported_seq":12,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.241601+0000","last_change":"2024-02-21T23:13:29.261481+0000","last_active":"2024-02-21T23:13:30.241601+0000","last_peered":"2024-02-21T23:13:30.241601+0000","last_clean":"2024-02-21T23:13:30.241601+0000","last_became_active":"2024-02-21T23:13:29.260963+0000","last_became_peered":"2024-02-21T23:13:29.260963+0000","last_unstale":"2024-02-21T23:13:30.241601+0000","last_undegraded":"2024-02-21T23:13:30.241601+0000","last_fullsized":"2024-02-21T23:13:30.241601+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_clean_scrub_stamp":"2024-02-21T23:13:28.197327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:36:34.078459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.14","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.267433+0000","last_change":"2024-02-21T23:13:30.261499+0000","last_active":"2024-02-21T23:13:30.267433+0000","last_peered":"2024-02-21T23:13:30.267433+0000","last_clean":"2024-02-21T23:13:30.267433+0000","last_became_active":"2024-02-21T23:13:30.257552+0000","last_became_peered":"2024-02-21T23:13:30.257552+0000","last_unstale":"2024-02-21T23:13:30.267433+0000","last_undegraded":"2024-02-21T23:13:30.267433+0000","last_fullsized":"2024-02-21T23:13:30.267433+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:41:55.354628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.17","version":"0'0","reported_seq":12,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.232610+0000","last_change":"2024-02-21T23:13:29.250865+0000","last_active":"2024-02-21T23:13:30.232610+0000","last_peered":"2024-02-21T23:13:30.232610+0000","last_clean":"2024-02-21T23:13:30.232610+0000","last_became_active":"2024-02-21T23:13:29.250593+0000","last_became_peered":"2024-02-21T23:13:29.250593+0000","last_unstale":"2024-02-21T23:13:30.232610+0000","last_undegraded":"2024-02-21T23:13:30.232610+0000","last_fullsized":"2024-02-21T23:13:30.232610+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_clean_scrub_stamp":"2024-02-21T23:13:28.197327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:58:34.458280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.15","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.254549+0000","last_change":"2024-02-21T23:13:30.249851+0000","last_active":"2024-02-21T23:13:30.254549+0000","last_peered":"2024-02-21T23:13:30.254549+0000","last_clean":"2024-02-21T23:13:30.254549+0000","last_became_active":"2024-02-21T23:13:30.249210+0000","last_became_peered":"2024-02-21T23:13:30.249210+0000","last_unstale":"2024-02-21T23:13:30.254549+0000","last_undegraded":"2024-02-21T23:13:30.254549+0000","last_fullsized":"2024-02-21T23:13:30.254549+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:33:16.054241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.16","version":"0'0","reported_seq":12,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.250862+0000","last_change":"2024-02-21T23:13:29.268396+0000","last_active":"2024-02-21T23:13:30.250862+0000","last_peered":"2024-02-21T23:13:30.250862+0000","last_clean":"2024-02-21T23:13:30.250862+0000","last_became_active":"2024-02-21T23:13:29.268105+0000","last_became_peered":"2024-02-21T23:13:29.268105+0000","last_unstale":"2024-02-21T23:13:30.250862+0000","last_undegraded":"2024-02-21T23:13:30.250862+0000","last_fullsized":"2024-02-21T23:13:30.250862+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_clean_scrub_stamp":"2024-02-21T23:13:28.197327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:57:51.373442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.16","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.276407+0000","last_change":"2024-02-21T23:13:30.272466+0000","last_active":"2024-02-21T23:13:30.276407+0000","last_peered":"2024-02-21T23:13:30.276407+0000","last_clean":"2024-02-21T23:13:30.276407+0000","last_became_active":"2024-02-21T23:13:30.272195+0000","last_became_peered":"2024-02-21T23:13:30.272195+0000","last_unstale":"2024-02-21T23:13:30.276407+0000","last_undegraded":"2024-02-21T23:13:30.276407+0000","last_fullsized":"2024-02-21T23:13:30.276407+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:43:37.680664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.15","version":"207'3","reported_seq":15,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.288052+0000","last_change":"2024-02-21T23:13:29.250853+0000","last_active":"2024-02-21T23:13:30.288052+0000","last_peered":"2024-02-21T23:13:30.288052+0000","last_clean":"2024-02-21T23:13:30.288052+0000","last_became_active":"2024-02-21T23:13:29.250539+0000","last_became_peered":"2024-02-21T23:13:29.250539+0000","last_unstale":"2024-02-21T23:13:30.288052+0000","last_undegraded":"2024-02-21T23:13:30.288052+0000","last_fullsized":"2024-02-21T23:13:30.288052+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_clean_scrub_stamp":"2024-02-21T23:13:28.197327+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:48:57.675316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.17","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.260868+0000","last_change":"2024-02-21T23:13:30.254100+0000","last_active":"2024-02-21T23:13:30.260868+0000","last_peered":"2024-02-21T23:13:30.260868+0000","last_clean":"2024-02-21T23:13:30.260868+0000","last_became_active":"2024-02-21T23:13:30.253615+0000","last_became_peered":"2024-02-21T23:13:30.253615+0000","last_unstale":"2024-02-21T23:13:30.260868+0000","last_undegraded":"2024-02-21T23:13:30.260868+0000","last_fullsized":"2024-02-21T23:13:30.260868+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:36:45.344984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.14","version":"207'6","reported_seq":18,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.290724+0000","last_change":"2024-02-21T23:13:29.255335+0000","last_active":"2024-02-21T23:13:30.290724+0000","last_peered":"2024-02-21T23:13:30.290724+0000","last_clean":"2024-02-21T23:13:30.290724+0000","last_became_active":"2024-02-21T23:13:29.254755+0000","last_became_peered":"2024-02-21T23:13:29.254755+0000","last_unstale":"2024-02-21T23:13:30.290724+0000","last_undegraded":"2024-02-21T23:13:30.290724+0000","last_fullsized":"2024-02-21T23:13:30.290724+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_clean_scrub_stamp":"2024-02-21T23:13:28.197327+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:58:23.394041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.10","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.254448+0000","last_change":"2024-02-21T23:13:30.253618+0000","last_active":"2024-02-21T23:13:30.254448+0000","last_peered":"2024-02-21T23:13:30.254448+0000","last_clean":"2024-02-21T23:13:30.254448+0000","last_became_active":"2024-02-21T23:13:30.253331+0000","last_became_peered":"2024-02-21T23:13:30.253331+0000","last_unstale":"2024-02-21T23:13:30.254448+0000","last_undegraded":"2024-02-21T23:13:30.254448+0000","last_fullsized":"2024-02-21T23:13:30.254448+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:16:31.150704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.13","version":"207'2","reported_seq":14,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.279727+0000","last_change":"2024-02-21T23:13:29.268721+0000","last_active":"2024-02-21T23:13:30.279727+0000","last_peered":"2024-02-21T23:13:30.279727+0000","last_clean":"2024-02-21T23:13:30.279727+0000","last_became_active":"2024-02-21T23:13:29.268390+0000","last_became_peered":"2024-02-21T23:13:29.268390+0000","last_unstale":"2024-02-21T23:13:30.279727+0000","last_undegraded":"2024-02-21T23:13:30.279727+0000","last_fullsized":"2024-02-21T23:13:30.279727+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_clean_scrub_stamp":"2024-02-21T23:13:28.197327+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:43:30.772438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.11","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.270970+0000","last_change":"2024-02-21T23:13:30.266579+0000","last_active":"2024-02-21T23:13:30.270970+0000","last_peered":"2024-02-21T23:13:30.270970+0000","last_clean":"2024-02-21T23:13:30.270970+0000","last_became_active":"2024-02-21T23:13:30.264191+0000","last_became_peered":"2024-02-21T23:13:30.264191+0000","last_unstale":"2024-02-21T23:13:30.270970+0000","last_undegraded":"2024-02-21T23:13:30.270970+0000","last_fullsized":"2024-02-21T23:13:30.270970+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:53:03.421126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.12","version":"0'0","reported_seq":12,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.231969+0000","last_change":"2024-02-21T23:13:29.257416+0000","last_active":"2024-02-21T23:13:30.231969+0000","last_peered":"2024-02-21T23:13:30.231969+0000","last_clean":"2024-02-21T23:13:30.231969+0000","last_became_active":"2024-02-21T23:13:29.257124+0000","last_became_peered":"2024-02-21T23:13:29.257124+0000","last_unstale":"2024-02-21T23:13:30.231969+0000","last_undegraded":"2024-02-21T23:13:30.231969+0000","last_fullsized":"2024-02-21T23:13:30.231969+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_clean_scrub_stamp":"2024-02-21T23:13:28.197327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:29:47.250213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.12","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.264176+0000","last_change":"2024-02-21T23:13:30.261523+0000","last_active":"2024-02-21T23:13:30.264176+0000","last_peered":"2024-02-21T23:13:30.264176+0000","last_clean":"2024-02-21T23:13:30.264176+0000","last_became_active":"2024-02-21T23:13:30.261255+0000","last_became_peered":"2024-02-21T23:13:30.261255+0000","last_unstale":"2024-02-21T23:13:30.264176+0000","last_undegraded":"2024-02-21T23:13:30.264176+0000","last_fullsized":"2024-02-21T23:13:30.264176+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:23:05.996217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.11","version":"0'0","reported_seq":12,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.229445+0000","last_change":"2024-02-21T23:13:29.263790+0000","last_active":"2024-02-21T23:13:30.229445+0000","last_peered":"2024-02-21T23:13:30.229445+0000","last_clean":"2024-02-21T23:13:30.229445+0000","last_became_active":"2024-02-21T23:13:29.263448+0000","last_became_peered":"2024-02-21T23:13:29.263448+0000","last_unstale":"2024-02-21T23:13:30.229445+0000","last_undegraded":"2024-02-21T23:13:30.229445+0000","last_fullsized":"2024-02-21T23:13:30.229445+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_clean_scrub_stamp":"2024-02-21T23:13:28.197327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:26:45.312782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.13","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.268224+0000","last_change":"2024-02-21T23:13:30.264461+0000","last_active":"2024-02-21T23:13:30.268224+0000","last_peered":"2024-02-21T23:13:30.268224+0000","last_clean":"2024-02-21T23:13:30.268224+0000","last_became_active":"2024-02-21T23:13:30.264174+0000","last_became_peered":"2024-02-21T23:13:30.264174+0000","last_unstale":"2024-02-21T23:13:30.268224+0000","last_undegraded":"2024-02-21T23:13:30.268224+0000","last_fullsized":"2024-02-21T23:13:30.268224+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:53:26.841237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.10","version":"207'8","reported_seq":20,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.287890+0000","last_change":"2024-02-21T23:13:29.254779+0000","last_active":"2024-02-21T23:13:30.287890+0000","last_peered":"2024-02-21T23:13:30.287890+0000","last_clean":"2024-02-21T23:13:30.287890+0000","last_became_active":"2024-02-21T23:13:29.253958+0000","last_became_peered":"2024-02-21T23:13:29.253958+0000","last_unstale":"2024-02-21T23:13:30.287890+0000","last_undegraded":"2024-02-21T23:13:30.287890+0000","last_fullsized":"2024-02-21T23:13:30.287890+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_clean_scrub_stamp":"2024-02-21T23:13:28.197327+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:01:25.183534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.c","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.260321+0000","last_change":"2024-02-21T23:13:30.254017+0000","last_active":"2024-02-21T23:13:30.260321+0000","last_peered":"2024-02-21T23:13:30.260321+0000","last_clean":"2024-02-21T23:13:30.260321+0000","last_became_active":"2024-02-21T23:13:30.253691+0000","last_became_peered":"2024-02-21T23:13:30.253691+0000","last_unstale":"2024-02-21T23:13:30.260321+0000","last_undegraded":"2024-02-21T23:13:30.260321+0000","last_fullsized":"2024-02-21T23:13:30.260321+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:36:57.815629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.f","version":"207'1","reported_seq":13,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.277268+0000","last_change":"2024-02-21T23:13:29.266149+0000","last_active":"2024-02-21T23:13:30.277268+0000","last_peered":"2024-02-21T23:13:30.277268+0000","last_clean":"2024-02-21T23:13:30.277268+0000","last_became_active":"2024-02-21T23:13:29.265660+0000","last_became_peered":"2024-02-21T23:13:29.265660+0000","last_unstale":"2024-02-21T23:13:30.277268+0000","last_undegraded":"2024-02-21T23:13:30.277268+0000","last_fullsized":"2024-02-21T23:13:30.277268+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_clean_scrub_stamp":"2024-02-21T23:13:28.197327+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:09:33.639743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.d","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.260403+0000","last_change":"2024-02-21T23:13:30.250755+0000","last_active":"2024-02-21T23:13:30.260403+0000","last_peered":"2024-02-21T23:13:30.260403+0000","last_clean":"2024-02-21T23:13:30.260403+0000","last_became_active":"2024-02-21T23:13:30.250359+0000","last_became_peered":"2024-02-21T23:13:30.250359+0000","last_unstale":"2024-02-21T23:13:30.260403+0000","last_undegraded":"2024-02-21T23:13:30.260403+0000","last_fullsized":"2024-02-21T23:13:30.260403+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:43:37.867934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.e","version":"207'2","reported_seq":14,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.284083+0000","last_change":"2024-02-21T23:13:29.265760+0000","last_active":"2024-02-21T23:13:30.284083+0000","last_peered":"2024-02-21T23:13:30.284083+0000","last_clean":"2024-02-21T23:13:30.284083+0000","last_became_active":"2024-02-21T23:13:29.264991+0000","last_became_peered":"2024-02-21T23:13:29.264991+0000","last_unstale":"2024-02-21T23:13:30.284083+0000","last_undegraded":"2024-02-21T23:13:30.284083+0000","last_fullsized":"2024-02-21T23:13:30.284083+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_clean_scrub_stamp":"2024-02-21T23:13:28.197327+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:55:14.303249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.e","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.267879+0000","last_change":"2024-02-21T23:13:30.258943+0000","last_active":"2024-02-21T23:13:30.267879+0000","last_peered":"2024-02-21T23:13:30.267879+0000","last_clean":"2024-02-21T23:13:30.267879+0000","last_became_active":"2024-02-21T23:13:30.257306+0000","last_became_peered":"2024-02-21T23:13:30.257306+0000","last_unstale":"2024-02-21T23:13:30.267879+0000","last_undegraded":"2024-02-21T23:13:30.267879+0000","last_fullsized":"2024-02-21T23:13:30.267879+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:34:40.892453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.d","version":"207'1","reported_seq":13,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.286368+0000","last_change":"2024-02-21T23:13:29.264426+0000","last_active":"2024-02-21T23:13:30.286368+0000","last_peered":"2024-02-21T23:13:30.286368+0000","last_clean":"2024-02-21T23:13:30.286368+0000","last_became_active":"2024-02-21T23:13:29.264113+0000","last_became_peered":"2024-02-21T23:13:29.264113+0000","last_unstale":"2024-02-21T23:13:30.286368+0000","last_undegraded":"2024-02-21T23:13:30.286368+0000","last_fullsized":"2024-02-21T23:13:30.286368+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_clean_scrub_stamp":"2024-02-21T23:13:28.197327+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:30:49.167568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.f","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.266648+0000","last_change":"2024-02-21T23:13:30.264032+0000","last_active":"2024-02-21T23:13:30.266648+0000","last_peered":"2024-02-21T23:13:30.266648+0000","last_clean":"2024-02-21T23:13:30.266648+0000","last_became_active":"2024-02-21T23:13:30.261622+0000","last_became_peered":"2024-02-21T23:13:30.261622+0000","last_unstale":"2024-02-21T23:13:30.266648+0000","last_undegraded":"2024-02-21T23:13:30.266648+0000","last_fullsized":"2024-02-21T23:13:30.266648+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:22:46.829986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.c","version":"0'0","reported_seq":12,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.226083+0000","last_change":"2024-02-21T23:13:29.262574+0000","last_active":"2024-02-21T23:13:30.226083+0000","last_peered":"2024-02-21T23:13:30.226083+0000","last_clean":"2024-02-21T23:13:30.226083+0000","last_became_active":"2024-02-21T23:13:29.262221+0000","last_became_peered":"2024-02-21T23:13:29.262221+0000","last_unstale":"2024-02-21T23:13:30.226083+0000","last_undegraded":"2024-02-21T23:13:30.226083+0000","last_fullsized":"2024-02-21T23:13:30.226083+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_clean_scrub_stamp":"2024-02-21T23:13:28.197327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:54:11.272648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.8","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.262418+0000","last_change":"2024-02-21T23:13:30.258489+0000","last_active":"2024-02-21T23:13:30.262418+0000","last_peered":"2024-02-21T23:13:30.262418+0000","last_clean":"2024-02-21T23:13:30.262418+0000","last_became_active":"2024-02-21T23:13:30.258179+0000","last_became_peered":"2024-02-21T23:13:30.258179+0000","last_unstale":"2024-02-21T23:13:30.262418+0000","last_undegraded":"2024-02-21T23:13:30.262418+0000","last_fullsized":"2024-02-21T23:13:30.262418+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:16:31.150704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.b","version":"207'2","reported_seq":14,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.286247+0000","last_change":"2024-02-21T23:13:29.252890+0000","last_active":"2024-02-21T23:13:30.286247+0000","last_peered":"2024-02-21T23:13:30.286247+0000","last_clean":"2024-02-21T23:13:30.286247+0000","last_became_active":"2024-02-21T23:13:29.250884+0000","last_became_peered":"2024-02-21T23:13:29.250884+0000","last_unstale":"2024-02-21T23:13:30.286247+0000","last_undegraded":"2024-02-21T23:13:30.286247+0000","last_fullsized":"2024-02-21T23:13:30.286247+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_clean_scrub_stamp":"2024-02-21T23:13:28.197327+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:42:47.677914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.9","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.251544+0000","last_change":"2024-02-21T23:13:30.243777+0000","last_active":"2024-02-21T23:13:30.251544+0000","last_peered":"2024-02-21T23:13:30.251544+0000","last_clean":"2024-02-21T23:13:30.251544+0000","last_became_active":"2024-02-21T23:13:30.243061+0000","last_became_peered":"2024-02-21T23:13:30.243061+0000","last_unstale":"2024-02-21T23:13:30.251544+0000","last_undegraded":"2024-02-21T23:13:30.251544+0000","last_fullsized":"2024-02-21T23:13:30.251544+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:29:27.260925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.a","version":"0'0","reported_seq":12,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.239864+0000","last_change":"2024-02-21T23:13:29.258895+0000","last_active":"2024-02-21T23:13:30.239864+0000","last_peered":"2024-02-21T23:13:30.239864+0000","last_clean":"2024-02-21T23:13:30.239864+0000","last_became_active":"2024-02-21T23:13:29.258598+0000","last_became_peered":"2024-02-21T23:13:29.258598+0000","last_unstale":"2024-02-21T23:13:30.239864+0000","last_undegraded":"2024-02-21T23:13:30.239864+0000","last_fullsized":"2024-02-21T23:13:30.239864+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_clean_scrub_stamp":"2024-02-21T23:13:28.197327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:34:13.155725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.a","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.256530+0000","last_change":"2024-02-21T23:13:30.248928+0000","last_active":"2024-02-21T23:13:30.256530+0000","last_peered":"2024-02-21T23:13:30.256530+0000","last_clean":"2024-02-21T23:13:30.256530+0000","last_became_active":"2024-02-21T23:13:30.248452+0000","last_became_peered":"2024-02-21T23:13:30.248452+0000","last_unstale":"2024-02-21T23:13:30.256530+0000","last_undegraded":"2024-02-21T23:13:30.256530+0000","last_fullsized":"2024-02-21T23:13:30.256530+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:58:12.797695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.9","version":"0'0","reported_seq":12,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.254064+0000","last_change":"2024-02-21T23:13:29.254284+0000","last_active":"2024-02-21T23:13:30.254064+0000","last_peered":"2024-02-21T23:13:30.254064+0000","last_clean":"2024-02-21T23:13:30.254064+0000","last_became_active":"2024-02-21T23:13:29.253638+0000","last_became_peered":"2024-02-21T23:13:29.253638+0000","last_unstale":"2024-02-21T23:13:30.254064+0000","last_undegraded":"2024-02-21T23:13:30.254064+0000","last_fullsized":"2024-02-21T23:13:30.254064+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_clean_scrub_stamp":"2024-02-21T23:13:28.197327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:42:47.677914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.b","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.281158+0000","last_change":"2024-02-21T23:13:30.278409+0000","last_active":"2024-02-21T23:13:30.281158+0000","last_peered":"2024-02-21T23:13:30.281158+0000","last_clean":"2024-02-21T23:13:30.281158+0000","last_became_active":"2024-02-21T23:13:30.278081+0000","last_became_peered":"2024-02-21T23:13:30.278081+0000","last_unstale":"2024-02-21T23:13:30.281158+0000","last_undegraded":"2024-02-21T23:13:30.281158+0000","last_fullsized":"2024-02-21T23:13:30.281158+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:16:17.460840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.8","version":"0'0","reported_seq":12,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.233525+0000","last_change":"2024-02-21T23:13:29.248784+0000","last_active":"2024-02-21T23:13:30.233525+0000","last_peered":"2024-02-21T23:13:30.233525+0000","last_clean":"2024-02-21T23:13:30.233525+0000","last_became_active":"2024-02-21T23:13:29.248370+0000","last_became_peered":"2024-02-21T23:13:29.248370+0000","last_unstale":"2024-02-21T23:13:30.233525+0000","last_undegraded":"2024-02-21T23:13:30.233525+0000","last_fullsized":"2024-02-21T23:13:30.233525+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_clean_scrub_stamp":"2024-02-21T23:13:28.197327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:13:57.983133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.4","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.274340+0000","last_change":"2024-02-21T23:13:30.270565+0000","last_active":"2024-02-21T23:13:30.274340+0000","last_peered":"2024-02-21T23:13:30.274340+0000","last_clean":"2024-02-21T23:13:30.274340+0000","last_became_active":"2024-02-21T23:13:30.270030+0000","last_became_peered":"2024-02-21T23:13:30.270030+0000","last_unstale":"2024-02-21T23:13:30.274340+0000","last_undegraded":"2024-02-21T23:13:30.274340+0000","last_fullsized":"2024-02-21T23:13:30.274340+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:48:44.645285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.7","version":"207'2","reported_seq":14,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.285282+0000","last_change":"2024-02-21T23:13:29.256652+0000","last_active":"2024-02-21T23:13:30.285282+0000","last_peered":"2024-02-21T23:13:30.285282+0000","last_clean":"2024-02-21T23:13:30.285282+0000","last_became_active":"2024-02-21T23:13:29.255581+0000","last_became_peered":"2024-02-21T23:13:29.255581+0000","last_unstale":"2024-02-21T23:13:30.285282+0000","last_undegraded":"2024-02-21T23:13:30.285282+0000","last_fullsized":"2024-02-21T23:13:30.285282+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_clean_scrub_stamp":"2024-02-21T23:13:28.197327+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:21:27.555571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.5","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.274386+0000","last_change":"2024-02-21T23:13:30.270663+0000","last_active":"2024-02-21T23:13:30.274386+0000","last_peered":"2024-02-21T23:13:30.274386+0000","last_clean":"2024-02-21T23:13:30.274386+0000","last_became_active":"2024-02-21T23:13:30.270322+0000","last_became_peered":"2024-02-21T23:13:30.270322+0000","last_unstale":"2024-02-21T23:13:30.274386+0000","last_undegraded":"2024-02-21T23:13:30.274386+0000","last_fullsized":"2024-02-21T23:13:30.274386+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:57:45.202740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.6","version":"0'0","reported_seq":12,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.230826+0000","last_change":"2024-02-21T23:13:29.259152+0000","last_active":"2024-02-21T23:13:30.230826+0000","last_peered":"2024-02-21T23:13:30.230826+0000","last_clean":"2024-02-21T23:13:30.230826+0000","last_became_active":"2024-02-21T23:13:29.258871+0000","last_became_peered":"2024-02-21T23:13:29.258871+0000","last_unstale":"2024-02-21T23:13:30.230826+0000","last_undegraded":"2024-02-21T23:13:30.230826+0000","last_fullsized":"2024-02-21T23:13:30.230826+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_clean_scrub_stamp":"2024-02-21T23:13:28.197327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:30:50.944777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.6","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.264318+0000","last_change":"2024-02-21T23:13:30.257184+0000","last_active":"2024-02-21T23:13:30.264318+0000","last_peered":"2024-02-21T23:13:30.264318+0000","last_clean":"2024-02-21T23:13:30.264318+0000","last_became_active":"2024-02-21T23:13:30.256701+0000","last_became_peered":"2024-02-21T23:13:30.256701+0000","last_unstale":"2024-02-21T23:13:30.264318+0000","last_undegraded":"2024-02-21T23:13:30.264318+0000","last_fullsized":"2024-02-21T23:13:30.264318+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:57:45.202740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.5","version":"0'0","reported_seq":12,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.243561+0000","last_change":"2024-02-21T23:13:29.268039+0000","last_active":"2024-02-21T23:13:30.243561+0000","last_peered":"2024-02-21T23:13:30.243561+0000","last_clean":"2024-02-21T23:13:30.243561+0000","last_became_active":"2024-02-21T23:13:29.267311+0000","last_became_peered":"2024-02-21T23:13:29.267311+0000","last_unstale":"2024-02-21T23:13:30.243561+0000","last_undegraded":"2024-02-21T23:13:30.243561+0000","last_fullsized":"2024-02-21T23:13:30.243561+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_clean_scrub_stamp":"2024-02-21T23:13:28.197327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:07:43.967351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.7","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.274591+0000","last_change":"2024-02-21T23:13:30.269247+0000","last_active":"2024-02-21T23:13:30.274591+0000","last_peered":"2024-02-21T23:13:30.274591+0000","last_clean":"2024-02-21T23:13:30.274591+0000","last_became_active":"2024-02-21T23:13:30.268530+0000","last_became_peered":"2024-02-21T23:13:30.268530+0000","last_unstale":"2024-02-21T23:13:30.274591+0000","last_undegraded":"2024-02-21T23:13:30.274591+0000","last_fullsized":"2024-02-21T23:13:30.274591+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:30:50.183494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.4","version":"0'0","reported_seq":12,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.245474+0000","last_change":"2024-02-21T23:13:29.261707+0000","last_active":"2024-02-21T23:13:30.245474+0000","last_peered":"2024-02-21T23:13:30.245474+0000","last_clean":"2024-02-21T23:13:30.245474+0000","last_became_active":"2024-02-21T23:13:29.261351+0000","last_became_peered":"2024-02-21T23:13:29.261351+0000","last_unstale":"2024-02-21T23:13:30.245474+0000","last_undegraded":"2024-02-21T23:13:30.245474+0000","last_fullsized":"2024-02-21T23:13:30.245474+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_clean_scrub_stamp":"2024-02-21T23:13:28.197327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:59:37.531428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.0","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.267489+0000","last_change":"2024-02-21T23:13:30.262666+0000","last_active":"2024-02-21T23:13:30.267489+0000","last_peered":"2024-02-21T23:13:30.267489+0000","last_clean":"2024-02-21T23:13:30.267489+0000","last_became_active":"2024-02-21T23:13:30.262329+0000","last_became_peered":"2024-02-21T23:13:30.262329+0000","last_unstale":"2024-02-21T23:13:30.267489+0000","last_undegraded":"2024-02-21T23:13:30.267489+0000","last_fullsized":"2024-02-21T23:13:30.267489+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:07:27.307529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.3","version":"0'0","reported_seq":12,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.225427+0000","last_change":"2024-02-21T23:13:29.255576+0000","last_active":"2024-02-21T23:13:30.225427+0000","last_peered":"2024-02-21T23:13:30.225427+0000","last_clean":"2024-02-21T23:13:30.225427+0000","last_became_active":"2024-02-21T23:13:29.255315+0000","last_became_peered":"2024-02-21T23:13:29.255315+0000","last_unstale":"2024-02-21T23:13:30.225427+0000","last_undegraded":"2024-02-21T23:13:30.225427+0000","last_fullsized":"2024-02-21T23:13:30.225427+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_clean_scrub_stamp":"2024-02-21T23:13:28.197327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:06:35.674377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.1","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.278181+0000","last_change":"2024-02-21T23:13:30.263695+0000","last_active":"2024-02-21T23:13:30.278181+0000","last_peered":"2024-02-21T23:13:30.278181+0000","last_clean":"2024-02-21T23:13:30.278181+0000","last_became_active":"2024-02-21T23:13:30.262207+0000","last_became_peered":"2024-02-21T23:13:30.262207+0000","last_unstale":"2024-02-21T23:13:30.278181+0000","last_undegraded":"2024-02-21T23:13:30.278181+0000","last_fullsized":"2024-02-21T23:13:30.278181+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:57:39.946172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.2","version":"0'0","reported_seq":12,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.225440+0000","last_change":"2024-02-21T23:13:29.253396+0000","last_active":"2024-02-21T23:13:30.225440+0000","last_peered":"2024-02-21T23:13:30.225440+0000","last_clean":"2024-02-21T23:13:30.225440+0000","last_became_active":"2024-02-21T23:13:29.251990+0000","last_became_peered":"2024-02-21T23:13:29.251990+0000","last_unstale":"2024-02-21T23:13:30.225440+0000","last_undegraded":"2024-02-21T23:13:30.225440+0000","last_fullsized":"2024-02-21T23:13:30.225440+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_clean_scrub_stamp":"2024-02-21T23:13:28.197327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:17:54.727926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.2","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.278028+0000","last_change":"2024-02-21T23:13:30.271186+0000","last_active":"2024-02-21T23:13:30.278028+0000","last_peered":"2024-02-21T23:13:30.278028+0000","last_clean":"2024-02-21T23:13:30.278028+0000","last_became_active":"2024-02-21T23:13:30.270349+0000","last_became_peered":"2024-02-21T23:13:30.270349+0000","last_unstale":"2024-02-21T23:13:30.278028+0000","last_undegraded":"2024-02-21T23:13:30.278028+0000","last_fullsized":"2024-02-21T23:13:30.278028+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:20:12.018424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1","version":"0'0","reported_seq":12,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.241323+0000","last_change":"2024-02-21T23:13:29.266996+0000","last_active":"2024-02-21T23:13:30.241323+0000","last_peered":"2024-02-21T23:13:30.241323+0000","last_clean":"2024-02-21T23:13:30.241323+0000","last_became_active":"2024-02-21T23:13:29.266639+0000","last_became_peered":"2024-02-21T23:13:29.266639+0000","last_unstale":"2024-02-21T23:13:30.241323+0000","last_undegraded":"2024-02-21T23:13:30.241323+0000","last_fullsized":"2024-02-21T23:13:30.241323+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_clean_scrub_stamp":"2024-02-21T23:13:28.197327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:45:50.645184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.3","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.259885+0000","last_change":"2024-02-21T23:13:30.254837+0000","last_active":"2024-02-21T23:13:30.259885+0000","last_peered":"2024-02-21T23:13:30.259885+0000","last_clean":"2024-02-21T23:13:30.259885+0000","last_became_active":"2024-02-21T23:13:30.254523+0000","last_became_peered":"2024-02-21T23:13:30.254523+0000","last_unstale":"2024-02-21T23:13:30.259885+0000","last_undegraded":"2024-02-21T23:13:30.259885+0000","last_fullsized":"2024-02-21T23:13:30.259885+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:30:50.183494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.0","version":"0'0","reported_seq":12,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.225103+0000","last_change":"2024-02-21T23:13:29.267982+0000","last_active":"2024-02-21T23:13:30.225103+0000","last_peered":"2024-02-21T23:13:30.225103+0000","last_clean":"2024-02-21T23:13:30.225103+0000","last_became_active":"2024-02-21T23:13:29.267673+0000","last_became_peered":"2024-02-21T23:13:29.267673+0000","last_unstale":"2024-02-21T23:13:30.225103+0000","last_undegraded":"2024-02-21T23:13:30.225103+0000","last_fullsized":"2024-02-21T23:13:30.225103+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_clean_scrub_stamp":"2024-02-21T23:13:28.197327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:20:58.752032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.23","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.268040+0000","last_change":"2024-02-21T23:13:30.262998+0000","last_active":"2024-02-21T23:13:30.268040+0000","last_peered":"2024-02-21T23:13:30.268040+0000","last_clean":"2024-02-21T23:13:30.268040+0000","last_became_active":"2024-02-21T23:13:30.257390+0000","last_became_peered":"2024-02-21T23:13:30.257390+0000","last_unstale":"2024-02-21T23:13:30.268040+0000","last_undegraded":"2024-02-21T23:13:30.268040+0000","last_fullsized":"2024-02-21T23:13:30.268040+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:58:57.387154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":518,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.249810+0000","last_change":"2024-02-21T23:02:32.289335+0000","last_active":"2024-02-21T23:13:30.249810+0000","last_peered":"2024-02-21T23:13:30.249810+0000","last_clean":"2024-02-21T23:13:30.249810+0000","last_became_active":"2024-02-21T23:02:32.286913+0000","last_became_peered":"2024-02-21T23:02:32.286913+0000","last_unstale":"2024-02-21T23:13:30.249810+0000","last_undegraded":"2024-02-21T23:13:30.249810+0000","last_fullsized":"2024-02-21T23:13:30.249810+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_clean_scrub_stamp":"2024-02-21T23:02:31.172569+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:36:23.987374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":34,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":33,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574982,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33800,"kb_used_data":2620,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338040,"statfs":{"total":96636764160,"available":96602152960,"internally_reserved":0,"allocated":2682880,"data_stored":1869802,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574983,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33760,"kb_used_data":2572,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338080,"statfs":{"total":96636764160,"available":96602193920,"internally_reserved":0,"allocated":2633728,"data_stored":1286635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574982,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33760,"kb_used_data":2572,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338080,"statfs":{"total":96636764160,"available":96602193920,"internally_reserved":0,"allocated":2633728,"data_stored":1286671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574983,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33760,"kb_used_data":2580,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338080,"statfs":{"total":96636764160,"available":96602193920,"internally_reserved":0,"allocated":2641920,"data_stored":1288213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574983,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37912,"kb_used_data":2572,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333928,"statfs":{"total":96636764160,"available":96597942272,"internally_reserved":0,"allocated":2633728,"data_stored":1286691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574984,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37928,"kb_used_data":2588,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333912,"statfs":{"total":96636764160,"available":96597925888,"internally_reserved":0,"allocated":2650112,"data_stored":1288817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574982,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37908,"kb_used_data":2568,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333932,"statfs":{"total":96636764160,"available":96597946368,"internally_reserved":0,"allocated":2629632,"data_stored":1286601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574982,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37960,"kb_used_data":2620,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333880,"statfs":{"total":96636764160,"available":96597893120,"internally_reserved":0,"allocated":2682880,"data_stored":1869802,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-21T23:13:34.689 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-21T23:13:34.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.961+0000 7f2ec2000640 1 Processor -- start 2024-02-21T23:13:34.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.961+0000 7f2ec2000640 1 -- start start 2024-02-21T23:13:34.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.963+0000 7f2ec2000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2ebc13e620 0x7f2ebc159070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:34.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.963+0000 7f2ec2000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ebc1551b0 con 0x7f2ebc0c6460 2024-02-21T23:13:34.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.963+0000 7f2ec2000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ebc1595b0 con 0x7f2ebc0bf990 2024-02-21T23:13:34.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.963+0000 7f2ec2000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ebc0c71d0 con 0x7f2ebc13e620 2024-02-21T23:13:34.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.964+0000 7f2ebb400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2ebc13e620 0x7f2ebc159070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:34.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.964+0000 7f2ebb400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2ebc13e620 0x7f2ebc159070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39352/0 (socket says 172.21.15.28:39352) 2024-02-21T23:13:34.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.964+0000 7f2ebb400640 1 -- 172.21.15.28:0/1164055005 learned_addr learned my addr 172.21.15.28:0/1164055005 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:34.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.964+0000 7f2ebb400640 1 -- 172.21.15.28:0/1164055005 >> v1:172.21.15.81:6789/0 conn(0x7f2ebc0bf990 legacy=0x7f2ebc0bfdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:34.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.964+0000 7f2ebb400640 1 -- 172.21.15.28:0/1164055005 >> v1:172.21.15.28:6789/0 conn(0x7f2ebc0c6460 legacy=0x7f2ebc0ca8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:34.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.964+0000 7f2ebb400640 1 -- 172.21.15.28:0/1164055005 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ebc155420 con 0x7f2ebc13e620 2024-02-21T23:13:34.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.964+0000 7f2eba000640 1 -- 172.21.15.28:0/1164055005 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1231199653 0 0) 0x7f2ebc1551b0 con 0x7f2ebc0c6460 2024-02-21T23:13:34.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.966+0000 7f2ebb400640 1 --2- 172.21.15.28:0/1164055005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ebc13e620 0x7f2ebc159070 secure :-1 s=READY pgs=2378 cs=0 l=1 rev1=1 crypto rx=0x7f2eac009e50 tx=0x7f2eac02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62fbb5a2eab6563f server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:34.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.966+0000 7f2eba000640 1 -- 172.21.15.28:0/1164055005 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eac02cb20 con 0x7f2ebc13e620 2024-02-21T23:13:34.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.967+0000 7f2eba000640 1 -- 172.21.15.28:0/1164055005 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2eac02ccc0 con 0x7f2ebc13e620 2024-02-21T23:13:34.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.967+0000 7f2eba000640 1 -- 172.21.15.28:0/1164055005 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eac034610 con 0x7f2ebc13e620 2024-02-21T23:13:34.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.967+0000 7f2ec2000640 1 -- 172.21.15.28:0/1164055005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ebc13e620 msgr2=0x7f2ebc159070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:34.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.967+0000 7f2ec2000640 1 --2- 172.21.15.28:0/1164055005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ebc13e620 0x7f2ebc159070 secure :-1 s=READY pgs=2378 cs=0 l=1 rev1=1 crypto rx=0x7f2eac009e50 tx=0x7f2eac02bf80 comp rx=0 tx=0).stop 2024-02-21T23:13:34.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.967+0000 7f2ec2000640 1 -- 172.21.15.28:0/1164055005 shutdown_connections 2024-02-21T23:13:34.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.967+0000 7f2ec2000640 1 --2- 172.21.15.28:0/1164055005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ebc13e620 0x7f2ebc159070 unknown :-1 s=CLOSED pgs=2378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:34.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.967+0000 7f2ec2000640 1 -- 172.21.15.28:0/1164055005 >> 172.21.15.28:0/1164055005 conn(0x7f2ebc08b1c0 msgr2=0x7f2ebc0725a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:34.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.967+0000 7f2ec2000640 1 -- 172.21.15.28:0/1164055005 shutdown_connections 2024-02-21T23:13:34.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.968+0000 7f2ec2000640 1 -- 172.21.15.28:0/1164055005 wait complete. 2024-02-21T23:13:34.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.970+0000 7f2ec2000640 1 Processor -- start 2024-02-21T23:13:34.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.970+0000 7f2ec2000640 1 -- start start 2024-02-21T23:13:34.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.972+0000 7f2ec2000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ebc0bf990 0x7f2ebc0c9f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:34.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.972+0000 7f2ec2000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2ebc0c6460 0x7f2ebc1ee110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:34.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.972+0000 7f2ec2000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ebc13e620 0x7f2ebc13b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:34.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.972+0000 7f2ec2000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ebc155cc0 con 0x7f2ebc13e620 2024-02-21T23:13:34.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.972+0000 7f2ec2000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ebc1559e0 con 0x7f2ebc0bf990 2024-02-21T23:13:34.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.972+0000 7f2ec2000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ebc13ea00 con 0x7f2ebc0c6460 2024-02-21T23:13:34.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.972+0000 7f2ebbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ebc13e620 0x7f2ebc13b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:34.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.972+0000 7f2ebbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ebc13e620 0x7f2ebc13b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39354/0 (socket says 172.21.15.28:39354) 2024-02-21T23:13:34.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.972+0000 7f2ebbe00640 1 -- 172.21.15.28:0/609208349 learned_addr learned my addr 172.21.15.28:0/609208349 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:34.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.973+0000 7f2ebb400640 1 --2- 172.21.15.28:0/609208349 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ebc0bf990 0x7f2ebc0c9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:34.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.973+0000 7f2ebaa00640 1 --2- 172.21.15.28:0/609208349 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2ebc0c6460 0x7f2ebc1ee110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:34.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.973+0000 7f2ebbe00640 1 -- 172.21.15.28:0/609208349 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2ebc0c6460 msgr2=0x7f2ebc1ee110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:34.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.973+0000 7f2ebbe00640 1 --2- 172.21.15.28:0/609208349 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2ebc0c6460 0x7f2ebc1ee110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:34.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.973+0000 7f2ebbe00640 1 -- 172.21.15.28:0/609208349 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ebc0bf990 msgr2=0x7f2ebc0c9f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:34.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.973+0000 7f2ebbe00640 1 --2- 172.21.15.28:0/609208349 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ebc0bf990 0x7f2ebc0c9f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:34.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.973+0000 7f2ebbe00640 1 -- 172.21.15.28:0/609208349 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ebc0aae60 con 0x7f2ebc13e620 2024-02-21T23:13:34.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.973+0000 7f2ebbe00640 1 --2- 172.21.15.28:0/609208349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ebc13e620 0x7f2ebc13b0c0 secure :-1 s=READY pgs=2379 cs=0 l=1 rev1=1 crypto rx=0x7f2eb400c390 tx=0x7f2eb400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:34.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.973+0000 7f2eb3400640 1 -- 172.21.15.28:0/609208349 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eb4017070 con 0x7f2ebc13e620 2024-02-21T23:13:34.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.973+0000 7f2eb3400640 1 -- 172.21.15.28:0/609208349 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2eb40052d0 con 0x7f2ebc13e620 2024-02-21T23:13:34.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.973+0000 7f2ec2000640 1 -- 172.21.15.28:0/609208349 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ebc1eaf30 con 0x7f2ebc13e620 2024-02-21T23:13:34.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.974+0000 7f2eb3400640 1 -- 172.21.15.28:0/609208349 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eb4003810 con 0x7f2ebc13e620 2024-02-21T23:13:34.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.974+0000 7f2ec2000640 1 -- 172.21.15.28:0/609208349 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ebc1edbf0 con 0x7f2ebc13e620 2024-02-21T23:13:34.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.975+0000 7f2ec2000640 1 -- 172.21.15.28:0/609208349 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ebc1eddf0 con 0x7f2ebc13e620 2024-02-21T23:13:34.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.977+0000 7f2eb3400640 1 -- 172.21.15.28:0/609208349 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2eb401a070 con 0x7f2ebc13e620 2024-02-21T23:13:34.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.978+0000 7f2eb3400640 1 --2- 172.21.15.28:0/609208349 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2e9806b4d0 0x7f2e9806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:34.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.979+0000 7f2eb3400640 1 -- 172.21.15.28:0/609208349 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f2eb4089650 con 0x7f2ebc13e620 2024-02-21T23:13:34.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.979+0000 7f2ebb400640 1 --2- 172.21.15.28:0/609208349 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2e9806b4d0 0x7f2e9806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:34.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.980+0000 7f2ebb400640 1 --2- 172.21.15.28:0/609208349 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2e9806b4d0 0x7f2e9806d990 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f2eac035000 tx=0x7f2eac030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:34.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:34.983+0000 7f2eb3400640 1 -- 172.21.15.28:0/609208349 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f2ebc1eddf0 con 0x7f2ebc13e620 2024-02-21T23:13:35.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.184+0000 7f2ec2000640 1 -- 172.21.15.28:0/609208349 --> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2ebc154110 con 0x7f2e9806b4d0 2024-02-21T23:13:35.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.202+0000 7f2eb3400640 1 -- 172.21.15.28:0/609208349 <== mgr.4101 v2:172.21.15.81:6832/1848538577 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252507 (secure 0 0 0) 0x7f2ebc154110 con 0x7f2e9806b4d0 2024-02-21T23:13:35.194 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:13:35.195 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-21T23:13:35.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.208+0000 7f2ec2000640 1 -- 172.21.15.28:0/609208349 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2e9806b4d0 msgr2=0x7f2e9806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:35.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.208+0000 7f2ec2000640 1 --2- 172.21.15.28:0/609208349 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2e9806b4d0 0x7f2e9806d990 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f2eac035000 tx=0x7f2eac030000 comp rx=0 tx=0).stop 2024-02-21T23:13:35.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.208+0000 7f2ec2000640 1 -- 172.21.15.28:0/609208349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ebc13e620 msgr2=0x7f2ebc13b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:35.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.208+0000 7f2ec2000640 1 --2- 172.21.15.28:0/609208349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ebc13e620 0x7f2ebc13b0c0 secure :-1 s=READY pgs=2379 cs=0 l=1 rev1=1 crypto rx=0x7f2eb400c390 tx=0x7f2eb400c850 comp rx=0 tx=0).stop 2024-02-21T23:13:35.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.208+0000 7f2ec2000640 1 -- 172.21.15.28:0/609208349 shutdown_connections 2024-02-21T23:13:35.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.208+0000 7f2ec2000640 1 --2- 172.21.15.28:0/609208349 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2e9806b4d0 0x7f2e9806d990 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:35.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.208+0000 7f2ec2000640 1 --2- 172.21.15.28:0/609208349 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2ebc0c6460 0x7f2ebc1ee110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:35.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.208+0000 7f2ec2000640 1 --2- 172.21.15.28:0/609208349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ebc13e620 0x7f2ebc13b0c0 unknown :-1 s=CLOSED pgs=2379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:35.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.208+0000 7f2ec2000640 1 --2- 172.21.15.28:0/609208349 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ebc0bf990 0x7f2ebc0c9f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:35.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.208+0000 7f2ec2000640 1 -- 172.21.15.28:0/609208349 >> 172.21.15.28:0/609208349 conn(0x7f2ebc08b1c0 msgr2=0x7f2ebc13c720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:35.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.209+0000 7f2ec2000640 1 -- 172.21.15.28:0/609208349 shutdown_connections 2024-02-21T23:13:35.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.209+0000 7f2ec2000640 1 -- 172.21.15.28:0/609208349 wait complete. 2024-02-21T23:13:35.216 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":533,"stamp":"2024-02-21T23:13:34.901645+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":138,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":286788,"kb_used_data":20692,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754687932,"statfs":{"total":773094113280,"available":772800442368,"internally_reserved":0,"allocated":21188608,"data_stored":11463232,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":272354865},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002075"},"pg_stats":[{"pgid":"34.3f","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.266624+0000","last_change":"2024-02-21T23:13:30.263436+0000","last_active":"2024-02-21T23:13:30.266624+0000","last_peered":"2024-02-21T23:13:30.266624+0000","last_clean":"2024-02-21T23:13:30.266624+0000","last_became_active":"2024-02-21T23:13:30.263101+0000","last_became_peered":"2024-02-21T23:13:30.263101+0000","last_unstale":"2024-02-21T23:13:30.266624+0000","last_undegraded":"2024-02-21T23:13:30.266624+0000","last_fullsized":"2024-02-21T23:13:30.266624+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:21:53.012342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.3e","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.262953+0000","last_change":"2024-02-21T23:13:30.259389+0000","last_active":"2024-02-21T23:13:30.262953+0000","last_peered":"2024-02-21T23:13:30.262953+0000","last_clean":"2024-02-21T23:13:30.262953+0000","last_became_active":"2024-02-21T23:13:30.259027+0000","last_became_peered":"2024-02-21T23:13:30.259027+0000","last_unstale":"2024-02-21T23:13:30.262953+0000","last_undegraded":"2024-02-21T23:13:30.262953+0000","last_fullsized":"2024-02-21T23:13:30.262953+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:23:05.996217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.3d","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.269473+0000","last_change":"2024-02-21T23:13:30.265498+0000","last_active":"2024-02-21T23:13:30.269473+0000","last_peered":"2024-02-21T23:13:30.269473+0000","last_clean":"2024-02-21T23:13:30.269473+0000","last_became_active":"2024-02-21T23:13:30.264675+0000","last_became_peered":"2024-02-21T23:13:30.264675+0000","last_unstale":"2024-02-21T23:13:30.269473+0000","last_undegraded":"2024-02-21T23:13:30.269473+0000","last_fullsized":"2024-02-21T23:13:30.269473+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:04:42.692957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.3c","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.267407+0000","last_change":"2024-02-21T23:13:30.260235+0000","last_active":"2024-02-21T23:13:30.267407+0000","last_peered":"2024-02-21T23:13:30.267407+0000","last_clean":"2024-02-21T23:13:30.267407+0000","last_became_active":"2024-02-21T23:13:30.259600+0000","last_became_peered":"2024-02-21T23:13:30.259600+0000","last_unstale":"2024-02-21T23:13:30.267407+0000","last_undegraded":"2024-02-21T23:13:30.267407+0000","last_fullsized":"2024-02-21T23:13:30.267407+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T11:04:39.059962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.3b","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.270250+0000","last_change":"2024-02-21T23:13:30.268330+0000","last_active":"2024-02-21T23:13:30.270250+0000","last_peered":"2024-02-21T23:13:30.270250+0000","last_clean":"2024-02-21T23:13:30.270250+0000","last_became_active":"2024-02-21T23:13:30.267457+0000","last_became_peered":"2024-02-21T23:13:30.267457+0000","last_unstale":"2024-02-21T23:13:30.270250+0000","last_undegraded":"2024-02-21T23:13:30.270250+0000","last_fullsized":"2024-02-21T23:13:30.270250+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:57:39.946172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.3a","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.269796+0000","last_change":"2024-02-21T23:13:30.264250+0000","last_active":"2024-02-21T23:13:30.269796+0000","last_peered":"2024-02-21T23:13:30.269796+0000","last_clean":"2024-02-21T23:13:30.269796+0000","last_became_active":"2024-02-21T23:13:30.263984+0000","last_became_peered":"2024-02-21T23:13:30.263984+0000","last_unstale":"2024-02-21T23:13:30.269796+0000","last_undegraded":"2024-02-21T23:13:30.269796+0000","last_fullsized":"2024-02-21T23:13:30.269796+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:54:21.479946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.39","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.264545+0000","last_change":"2024-02-21T23:13:30.257367+0000","last_active":"2024-02-21T23:13:30.264545+0000","last_peered":"2024-02-21T23:13:30.264545+0000","last_clean":"2024-02-21T23:13:30.264545+0000","last_became_active":"2024-02-21T23:13:30.257066+0000","last_became_peered":"2024-02-21T23:13:30.257066+0000","last_unstale":"2024-02-21T23:13:30.264545+0000","last_undegraded":"2024-02-21T23:13:30.264545+0000","last_fullsized":"2024-02-21T23:13:30.264545+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:43:37.680664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.38","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.267432+0000","last_change":"2024-02-21T23:13:30.257613+0000","last_active":"2024-02-21T23:13:30.267432+0000","last_peered":"2024-02-21T23:13:30.267432+0000","last_clean":"2024-02-21T23:13:30.267432+0000","last_became_active":"2024-02-21T23:13:30.256432+0000","last_became_peered":"2024-02-21T23:13:30.256432+0000","last_unstale":"2024-02-21T23:13:30.267432+0000","last_undegraded":"2024-02-21T23:13:30.267432+0000","last_fullsized":"2024-02-21T23:13:30.267432+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:57:56.254791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.37","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.271392+0000","last_change":"2024-02-21T23:13:30.267058+0000","last_active":"2024-02-21T23:13:30.271392+0000","last_peered":"2024-02-21T23:13:30.271392+0000","last_clean":"2024-02-21T23:13:30.271392+0000","last_became_active":"2024-02-21T23:13:30.266768+0000","last_became_peered":"2024-02-21T23:13:30.266768+0000","last_unstale":"2024-02-21T23:13:30.271392+0000","last_undegraded":"2024-02-21T23:13:30.271392+0000","last_fullsized":"2024-02-21T23:13:30.271392+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:45:51.661110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.36","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.261245+0000","last_change":"2024-02-21T23:13:30.255320+0000","last_active":"2024-02-21T23:13:30.261245+0000","last_peered":"2024-02-21T23:13:30.261245+0000","last_clean":"2024-02-21T23:13:30.261245+0000","last_became_active":"2024-02-21T23:13:30.254279+0000","last_became_peered":"2024-02-21T23:13:30.254279+0000","last_unstale":"2024-02-21T23:13:30.261245+0000","last_undegraded":"2024-02-21T23:13:30.261245+0000","last_fullsized":"2024-02-21T23:13:30.261245+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:07:44.983277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.35","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.265360+0000","last_change":"2024-02-21T23:13:30.257020+0000","last_active":"2024-02-21T23:13:30.265360+0000","last_peered":"2024-02-21T23:13:30.265360+0000","last_clean":"2024-02-21T23:13:30.265360+0000","last_became_active":"2024-02-21T23:13:30.255810+0000","last_became_peered":"2024-02-21T23:13:30.255810+0000","last_unstale":"2024-02-21T23:13:30.265360+0000","last_undegraded":"2024-02-21T23:13:30.265360+0000","last_fullsized":"2024-02-21T23:13:30.265360+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:41:00.724045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.34","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.274320+0000","last_change":"2024-02-21T23:13:30.270247+0000","last_active":"2024-02-21T23:13:30.274320+0000","last_peered":"2024-02-21T23:13:30.274320+0000","last_clean":"2024-02-21T23:13:30.274320+0000","last_became_active":"2024-02-21T23:13:30.268715+0000","last_became_peered":"2024-02-21T23:13:30.268715+0000","last_unstale":"2024-02-21T23:13:30.274320+0000","last_undegraded":"2024-02-21T23:13:30.274320+0000","last_fullsized":"2024-02-21T23:13:30.274320+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:59:38.547353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.33","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.268021+0000","last_change":"2024-02-21T23:13:30.262537+0000","last_active":"2024-02-21T23:13:30.268021+0000","last_peered":"2024-02-21T23:13:30.268021+0000","last_clean":"2024-02-21T23:13:30.268021+0000","last_became_active":"2024-02-21T23:13:30.257067+0000","last_became_peered":"2024-02-21T23:13:30.257067+0000","last_unstale":"2024-02-21T23:13:30.268021+0000","last_undegraded":"2024-02-21T23:13:30.268021+0000","last_fullsized":"2024-02-21T23:13:30.268021+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:49:43.777047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.32","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.280783+0000","last_change":"2024-02-21T23:13:30.277793+0000","last_active":"2024-02-21T23:13:30.280783+0000","last_peered":"2024-02-21T23:13:30.280783+0000","last_clean":"2024-02-21T23:13:30.280783+0000","last_became_active":"2024-02-21T23:13:30.270736+0000","last_became_peered":"2024-02-21T23:13:30.270736+0000","last_unstale":"2024-02-21T23:13:30.280783+0000","last_undegraded":"2024-02-21T23:13:30.280783+0000","last_fullsized":"2024-02-21T23:13:30.280783+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:28:32.097867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.31","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.278205+0000","last_change":"2024-02-21T23:13:30.263971+0000","last_active":"2024-02-21T23:13:30.278205+0000","last_peered":"2024-02-21T23:13:30.278205+0000","last_clean":"2024-02-21T23:13:30.278205+0000","last_became_active":"2024-02-21T23:13:30.262598+0000","last_became_peered":"2024-02-21T23:13:30.262598+0000","last_unstale":"2024-02-21T23:13:30.278205+0000","last_undegraded":"2024-02-21T23:13:30.278205+0000","last_fullsized":"2024-02-21T23:13:30.278205+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:09:59.268584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.30","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.268922+0000","last_change":"2024-02-21T23:13:30.263522+0000","last_active":"2024-02-21T23:13:30.268922+0000","last_peered":"2024-02-21T23:13:30.268922+0000","last_clean":"2024-02-21T23:13:30.268922+0000","last_became_active":"2024-02-21T23:13:30.262689+0000","last_became_peered":"2024-02-21T23:13:30.262689+0000","last_unstale":"2024-02-21T23:13:30.268922+0000","last_undegraded":"2024-02-21T23:13:30.268922+0000","last_fullsized":"2024-02-21T23:13:30.268922+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:59:38.547353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.2f","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.259832+0000","last_change":"2024-02-21T23:13:30.254805+0000","last_active":"2024-02-21T23:13:30.259832+0000","last_peered":"2024-02-21T23:13:30.259832+0000","last_clean":"2024-02-21T23:13:30.259832+0000","last_became_active":"2024-02-21T23:13:30.252766+0000","last_became_peered":"2024-02-21T23:13:30.252766+0000","last_unstale":"2024-02-21T23:13:30.259832+0000","last_undegraded":"2024-02-21T23:13:30.259832+0000","last_fullsized":"2024-02-21T23:13:30.259832+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:45:51.661110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.2e","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.254772+0000","last_change":"2024-02-21T23:13:30.250176+0000","last_active":"2024-02-21T23:13:30.254772+0000","last_peered":"2024-02-21T23:13:30.254772+0000","last_clean":"2024-02-21T23:13:30.254772+0000","last_became_active":"2024-02-21T23:13:30.249538+0000","last_became_peered":"2024-02-21T23:13:30.249538+0000","last_unstale":"2024-02-21T23:13:30.254772+0000","last_undegraded":"2024-02-21T23:13:30.254772+0000","last_fullsized":"2024-02-21T23:13:30.254772+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:17:53.761214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.2d","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.259732+0000","last_change":"2024-02-21T23:13:30.254034+0000","last_active":"2024-02-21T23:13:30.259732+0000","last_peered":"2024-02-21T23:13:30.259732+0000","last_clean":"2024-02-21T23:13:30.259732+0000","last_became_active":"2024-02-21T23:13:30.253625+0000","last_became_peered":"2024-02-21T23:13:30.253625+0000","last_unstale":"2024-02-21T23:13:30.259732+0000","last_undegraded":"2024-02-21T23:13:30.259732+0000","last_fullsized":"2024-02-21T23:13:30.259732+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:59:26.281603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.2c","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.274290+0000","last_change":"2024-02-21T23:13:30.269988+0000","last_active":"2024-02-21T23:13:30.274290+0000","last_peered":"2024-02-21T23:13:30.274290+0000","last_clean":"2024-02-21T23:13:30.274290+0000","last_became_active":"2024-02-21T23:13:30.268471+0000","last_became_peered":"2024-02-21T23:13:30.268471+0000","last_unstale":"2024-02-21T23:13:30.274290+0000","last_undegraded":"2024-02-21T23:13:30.274290+0000","last_fullsized":"2024-02-21T23:13:30.274290+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:07:44.983277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.2b","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.270272+0000","last_change":"2024-02-21T23:13:30.268351+0000","last_active":"2024-02-21T23:13:30.270272+0000","last_peered":"2024-02-21T23:13:30.270272+0000","last_clean":"2024-02-21T23:13:30.270272+0000","last_became_active":"2024-02-21T23:13:30.267688+0000","last_became_peered":"2024-02-21T23:13:30.267688+0000","last_unstale":"2024-02-21T23:13:30.270272+0000","last_undegraded":"2024-02-21T23:13:30.270272+0000","last_fullsized":"2024-02-21T23:13:30.270272+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:13:06.536763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.2a","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.270975+0000","last_change":"2024-02-21T23:13:30.266234+0000","last_active":"2024-02-21T23:13:30.270975+0000","last_peered":"2024-02-21T23:13:30.270975+0000","last_clean":"2024-02-21T23:13:30.270975+0000","last_became_active":"2024-02-21T23:13:30.264541+0000","last_became_peered":"2024-02-21T23:13:30.264541+0000","last_unstale":"2024-02-21T23:13:30.270975+0000","last_undegraded":"2024-02-21T23:13:30.270975+0000","last_fullsized":"2024-02-21T23:13:30.270975+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:34:51.167753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.29","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.280778+0000","last_change":"2024-02-21T23:13:30.277544+0000","last_active":"2024-02-21T23:13:30.280778+0000","last_peered":"2024-02-21T23:13:30.280778+0000","last_clean":"2024-02-21T23:13:30.280778+0000","last_became_active":"2024-02-21T23:13:30.268711+0000","last_became_peered":"2024-02-21T23:13:30.268711+0000","last_unstale":"2024-02-21T23:13:30.280778+0000","last_undegraded":"2024-02-21T23:13:30.280778+0000","last_fullsized":"2024-02-21T23:13:30.280778+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:49:38.979105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.28","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.261093+0000","last_change":"2024-02-21T23:13:30.251900+0000","last_active":"2024-02-21T23:13:30.261093+0000","last_peered":"2024-02-21T23:13:30.261093+0000","last_clean":"2024-02-21T23:13:30.261093+0000","last_became_active":"2024-02-21T23:13:30.251561+0000","last_became_peered":"2024-02-21T23:13:30.251561+0000","last_unstale":"2024-02-21T23:13:30.261093+0000","last_undegraded":"2024-02-21T23:13:30.261093+0000","last_fullsized":"2024-02-21T23:13:30.261093+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:36:34.193446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.27","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.261587+0000","last_change":"2024-02-21T23:13:30.255408+0000","last_active":"2024-02-21T23:13:30.261587+0000","last_peered":"2024-02-21T23:13:30.261587+0000","last_clean":"2024-02-21T23:13:30.261587+0000","last_became_active":"2024-02-21T23:13:30.255120+0000","last_became_peered":"2024-02-21T23:13:30.255120+0000","last_unstale":"2024-02-21T23:13:30.261587+0000","last_undegraded":"2024-02-21T23:13:30.261587+0000","last_fullsized":"2024-02-21T23:13:30.261587+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:48:44.645285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.26","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.271885+0000","last_change":"2024-02-21T23:13:30.263046+0000","last_active":"2024-02-21T23:13:30.271885+0000","last_peered":"2024-02-21T23:13:30.271885+0000","last_clean":"2024-02-21T23:13:30.271885+0000","last_became_active":"2024-02-21T23:13:30.262494+0000","last_became_peered":"2024-02-21T23:13:30.262494+0000","last_unstale":"2024-02-21T23:13:30.271885+0000","last_undegraded":"2024-02-21T23:13:30.271885+0000","last_fullsized":"2024-02-21T23:13:30.271885+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:53:26.841237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.25","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.271053+0000","last_change":"2024-02-21T23:13:30.265968+0000","last_active":"2024-02-21T23:13:30.271053+0000","last_peered":"2024-02-21T23:13:30.271053+0000","last_clean":"2024-02-21T23:13:30.271053+0000","last_became_active":"2024-02-21T23:13:30.264342+0000","last_became_peered":"2024-02-21T23:13:30.264342+0000","last_unstale":"2024-02-21T23:13:30.271053+0000","last_undegraded":"2024-02-21T23:13:30.271053+0000","last_fullsized":"2024-02-21T23:13:30.271053+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:19:42.812708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.24","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.267130+0000","last_change":"2024-02-21T23:13:30.263976+0000","last_active":"2024-02-21T23:13:30.267130+0000","last_peered":"2024-02-21T23:13:30.267130+0000","last_clean":"2024-02-21T23:13:30.267130+0000","last_became_active":"2024-02-21T23:13:30.263566+0000","last_became_peered":"2024-02-21T23:13:30.263566+0000","last_unstale":"2024-02-21T23:13:30.267130+0000","last_undegraded":"2024-02-21T23:13:30.267130+0000","last_fullsized":"2024-02-21T23:13:30.267130+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:17:28.420304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.22","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.269911+0000","last_change":"2024-02-21T23:13:30.266130+0000","last_active":"2024-02-21T23:13:30.269911+0000","last_peered":"2024-02-21T23:13:30.269911+0000","last_clean":"2024-02-21T23:13:30.269911+0000","last_became_active":"2024-02-21T23:13:30.262419+0000","last_became_peered":"2024-02-21T23:13:30.262419+0000","last_unstale":"2024-02-21T23:13:30.269911+0000","last_undegraded":"2024-02-21T23:13:30.269911+0000","last_fullsized":"2024-02-21T23:13:30.269911+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:59:26.281603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.21","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.278298+0000","last_change":"2024-02-21T23:13:30.270320+0000","last_active":"2024-02-21T23:13:30.278298+0000","last_peered":"2024-02-21T23:13:30.278298+0000","last_clean":"2024-02-21T23:13:30.278298+0000","last_became_active":"2024-02-21T23:13:30.270008+0000","last_became_peered":"2024-02-21T23:13:30.270008+0000","last_unstale":"2024-02-21T23:13:30.278298+0000","last_undegraded":"2024-02-21T23:13:30.278298+0000","last_fullsized":"2024-02-21T23:13:30.278298+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:49:57.308010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.20","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.282744+0000","last_change":"2024-02-21T23:13:30.278812+0000","last_active":"2024-02-21T23:13:30.282744+0000","last_peered":"2024-02-21T23:13:30.282744+0000","last_clean":"2024-02-21T23:13:30.282744+0000","last_became_active":"2024-02-21T23:13:30.278374+0000","last_became_peered":"2024-02-21T23:13:30.278374+0000","last_unstale":"2024-02-21T23:13:30.282744+0000","last_undegraded":"2024-02-21T23:13:30.282744+0000","last_fullsized":"2024-02-21T23:13:30.282744+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:30:31.245779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.1c","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.269400+0000","last_change":"2024-02-21T23:13:30.263843+0000","last_active":"2024-02-21T23:13:30.269400+0000","last_peered":"2024-02-21T23:13:30.269400+0000","last_clean":"2024-02-21T23:13:30.269400+0000","last_became_active":"2024-02-21T23:13:30.263555+0000","last_became_peered":"2024-02-21T23:13:30.263555+0000","last_unstale":"2024-02-21T23:13:30.269400+0000","last_undegraded":"2024-02-21T23:13:30.269400+0000","last_fullsized":"2024-02-21T23:13:30.269400+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:48:44.645285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.1f","version":"207'4","reported_seq":16,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.285881+0000","last_change":"2024-02-21T23:13:29.257460+0000","last_active":"2024-02-21T23:13:30.285881+0000","last_peered":"2024-02-21T23:13:30.285881+0000","last_clean":"2024-02-21T23:13:30.285881+0000","last_became_active":"2024-02-21T23:13:29.257196+0000","last_became_peered":"2024-02-21T23:13:29.257196+0000","last_unstale":"2024-02-21T23:13:30.285881+0000","last_undegraded":"2024-02-21T23:13:30.285881+0000","last_fullsized":"2024-02-21T23:13:30.285881+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_clean_scrub_stamp":"2024-02-21T23:13:28.197327+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:11:17.997693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.1d","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.256658+0000","last_change":"2024-02-21T23:13:30.253331+0000","last_active":"2024-02-21T23:13:30.256658+0000","last_peered":"2024-02-21T23:13:30.256658+0000","last_clean":"2024-02-21T23:13:30.256658+0000","last_became_active":"2024-02-21T23:13:30.252998+0000","last_became_peered":"2024-02-21T23:13:30.252998+0000","last_unstale":"2024-02-21T23:13:30.256658+0000","last_undegraded":"2024-02-21T23:13:30.256658+0000","last_fullsized":"2024-02-21T23:13:30.256658+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:23:55.545925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1e","version":"207'1","reported_seq":13,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.276595+0000","last_change":"2024-02-21T23:13:29.269012+0000","last_active":"2024-02-21T23:13:30.276595+0000","last_peered":"2024-02-21T23:13:30.276595+0000","last_clean":"2024-02-21T23:13:30.276595+0000","last_became_active":"2024-02-21T23:13:29.268669+0000","last_became_peered":"2024-02-21T23:13:29.268669+0000","last_unstale":"2024-02-21T23:13:30.276595+0000","last_undegraded":"2024-02-21T23:13:30.276595+0000","last_fullsized":"2024-02-21T23:13:30.276595+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_clean_scrub_stamp":"2024-02-21T23:13:28.197327+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:09:33.639743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.1e","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.242448+0000","last_change":"2024-02-21T23:13:30.240587+0000","last_active":"2024-02-21T23:13:30.242448+0000","last_peered":"2024-02-21T23:13:30.242448+0000","last_clean":"2024-02-21T23:13:30.242448+0000","last_became_active":"2024-02-21T23:13:30.239778+0000","last_became_peered":"2024-02-21T23:13:30.239778+0000","last_unstale":"2024-02-21T23:13:30.242448+0000","last_undegraded":"2024-02-21T23:13:30.242448+0000","last_fullsized":"2024-02-21T23:13:30.242448+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:43:45.804413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1d","version":"207'2","reported_seq":14,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.285949+0000","last_change":"2024-02-21T23:13:29.260853+0000","last_active":"2024-02-21T23:13:30.285949+0000","last_peered":"2024-02-21T23:13:30.285949+0000","last_clean":"2024-02-21T23:13:30.285949+0000","last_became_active":"2024-02-21T23:13:29.260225+0000","last_became_peered":"2024-02-21T23:13:29.260225+0000","last_unstale":"2024-02-21T23:13:30.285949+0000","last_undegraded":"2024-02-21T23:13:30.285949+0000","last_fullsized":"2024-02-21T23:13:30.285949+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_clean_scrub_stamp":"2024-02-21T23:13:28.197327+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:57:27.330618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.1f","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.281748+0000","last_change":"2024-02-21T23:13:30.280231+0000","last_active":"2024-02-21T23:13:30.281748+0000","last_peered":"2024-02-21T23:13:30.281748+0000","last_clean":"2024-02-21T23:13:30.281748+0000","last_became_active":"2024-02-21T23:13:30.271571+0000","last_became_peered":"2024-02-21T23:13:30.271571+0000","last_unstale":"2024-02-21T23:13:30.281748+0000","last_undegraded":"2024-02-21T23:13:30.281748+0000","last_fullsized":"2024-02-21T23:13:30.281748+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:14:29.166217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1c","version":"207'2","reported_seq":14,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.279895+0000","last_change":"2024-02-21T23:13:29.264981+0000","last_active":"2024-02-21T23:13:30.279895+0000","last_peered":"2024-02-21T23:13:30.279895+0000","last_clean":"2024-02-21T23:13:30.279895+0000","last_became_active":"2024-02-21T23:13:29.264527+0000","last_became_peered":"2024-02-21T23:13:29.264527+0000","last_unstale":"2024-02-21T23:13:30.279895+0000","last_undegraded":"2024-02-21T23:13:30.279895+0000","last_fullsized":"2024-02-21T23:13:30.279895+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_clean_scrub_stamp":"2024-02-21T23:13:28.197327+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:54:20.464021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.18","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.278568+0000","last_change":"2024-02-21T23:13:30.263784+0000","last_active":"2024-02-21T23:13:30.278568+0000","last_peered":"2024-02-21T23:13:30.278568+0000","last_clean":"2024-02-21T23:13:30.278568+0000","last_became_active":"2024-02-21T23:13:30.263191+0000","last_became_peered":"2024-02-21T23:13:30.263191+0000","last_unstale":"2024-02-21T23:13:30.278568+0000","last_undegraded":"2024-02-21T23:13:30.278568+0000","last_fullsized":"2024-02-21T23:13:30.278568+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:13:06.536763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1b","version":"207'1","reported_seq":13,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.277261+0000","last_change":"2024-02-21T23:13:29.262692+0000","last_active":"2024-02-21T23:13:30.277261+0000","last_peered":"2024-02-21T23:13:30.277261+0000","last_clean":"2024-02-21T23:13:30.277261+0000","last_became_active":"2024-02-21T23:13:29.262396+0000","last_became_peered":"2024-02-21T23:13:29.262396+0000","last_unstale":"2024-02-21T23:13:30.277261+0000","last_undegraded":"2024-02-21T23:13:30.277261+0000","last_fullsized":"2024-02-21T23:13:30.277261+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_clean_scrub_stamp":"2024-02-21T23:13:28.197327+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:55:08.180278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.19","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.278253+0000","last_change":"2024-02-21T23:13:30.264445+0000","last_active":"2024-02-21T23:13:30.278253+0000","last_peered":"2024-02-21T23:13:30.278253+0000","last_clean":"2024-02-21T23:13:30.278253+0000","last_became_active":"2024-02-21T23:13:30.260450+0000","last_became_peered":"2024-02-21T23:13:30.260450+0000","last_unstale":"2024-02-21T23:13:30.278253+0000","last_undegraded":"2024-02-21T23:13:30.278253+0000","last_fullsized":"2024-02-21T23:13:30.278253+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:05:15.802970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1a","version":"207'1","reported_seq":13,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.288937+0000","last_change":"2024-02-21T23:13:29.263680+0000","last_active":"2024-02-21T23:13:30.288937+0000","last_peered":"2024-02-21T23:13:30.288937+0000","last_clean":"2024-02-21T23:13:30.288937+0000","last_became_active":"2024-02-21T23:13:29.263017+0000","last_became_peered":"2024-02-21T23:13:29.263017+0000","last_unstale":"2024-02-21T23:13:30.288937+0000","last_undegraded":"2024-02-21T23:13:30.288937+0000","last_fullsized":"2024-02-21T23:13:30.288937+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_clean_scrub_stamp":"2024-02-21T23:13:28.197327+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:43:30.772438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.1a","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.269527+0000","last_change":"2024-02-21T23:13:30.265727+0000","last_active":"2024-02-21T23:13:30.269527+0000","last_peered":"2024-02-21T23:13:30.269527+0000","last_clean":"2024-02-21T23:13:30.269527+0000","last_became_active":"2024-02-21T23:13:30.264020+0000","last_became_peered":"2024-02-21T23:13:30.264020+0000","last_unstale":"2024-02-21T23:13:30.269527+0000","last_undegraded":"2024-02-21T23:13:30.269527+0000","last_fullsized":"2024-02-21T23:13:30.269527+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:27:53.857422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.19","version":"0'0","reported_seq":12,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.251087+0000","last_change":"2024-02-21T23:13:29.265683+0000","last_active":"2024-02-21T23:13:30.251087+0000","last_peered":"2024-02-21T23:13:30.251087+0000","last_clean":"2024-02-21T23:13:30.251087+0000","last_became_active":"2024-02-21T23:13:29.265362+0000","last_became_peered":"2024-02-21T23:13:29.265362+0000","last_unstale":"2024-02-21T23:13:30.251087+0000","last_undegraded":"2024-02-21T23:13:30.251087+0000","last_fullsized":"2024-02-21T23:13:30.251087+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_clean_scrub_stamp":"2024-02-21T23:13:28.197327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:52:02.987184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.1b","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.268001+0000","last_change":"2024-02-21T23:13:30.257729+0000","last_active":"2024-02-21T23:13:30.268001+0000","last_peered":"2024-02-21T23:13:30.268001+0000","last_clean":"2024-02-21T23:13:30.268001+0000","last_became_active":"2024-02-21T23:13:30.250419+0000","last_became_peered":"2024-02-21T23:13:30.250419+0000","last_unstale":"2024-02-21T23:13:30.268001+0000","last_undegraded":"2024-02-21T23:13:30.268001+0000","last_fullsized":"2024-02-21T23:13:30.268001+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:05:45.588776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.18","version":"0'0","reported_seq":12,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.241601+0000","last_change":"2024-02-21T23:13:29.261481+0000","last_active":"2024-02-21T23:13:30.241601+0000","last_peered":"2024-02-21T23:13:30.241601+0000","last_clean":"2024-02-21T23:13:30.241601+0000","last_became_active":"2024-02-21T23:13:29.260963+0000","last_became_peered":"2024-02-21T23:13:29.260963+0000","last_unstale":"2024-02-21T23:13:30.241601+0000","last_undegraded":"2024-02-21T23:13:30.241601+0000","last_fullsized":"2024-02-21T23:13:30.241601+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_clean_scrub_stamp":"2024-02-21T23:13:28.197327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:36:34.078459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.14","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.267433+0000","last_change":"2024-02-21T23:13:30.261499+0000","last_active":"2024-02-21T23:13:30.267433+0000","last_peered":"2024-02-21T23:13:30.267433+0000","last_clean":"2024-02-21T23:13:30.267433+0000","last_became_active":"2024-02-21T23:13:30.257552+0000","last_became_peered":"2024-02-21T23:13:30.257552+0000","last_unstale":"2024-02-21T23:13:30.267433+0000","last_undegraded":"2024-02-21T23:13:30.267433+0000","last_fullsized":"2024-02-21T23:13:30.267433+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:41:55.354628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.17","version":"0'0","reported_seq":12,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.232610+0000","last_change":"2024-02-21T23:13:29.250865+0000","last_active":"2024-02-21T23:13:30.232610+0000","last_peered":"2024-02-21T23:13:30.232610+0000","last_clean":"2024-02-21T23:13:30.232610+0000","last_became_active":"2024-02-21T23:13:29.250593+0000","last_became_peered":"2024-02-21T23:13:29.250593+0000","last_unstale":"2024-02-21T23:13:30.232610+0000","last_undegraded":"2024-02-21T23:13:30.232610+0000","last_fullsized":"2024-02-21T23:13:30.232610+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_clean_scrub_stamp":"2024-02-21T23:13:28.197327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:58:34.458280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.15","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.254549+0000","last_change":"2024-02-21T23:13:30.249851+0000","last_active":"2024-02-21T23:13:30.254549+0000","last_peered":"2024-02-21T23:13:30.254549+0000","last_clean":"2024-02-21T23:13:30.254549+0000","last_became_active":"2024-02-21T23:13:30.249210+0000","last_became_peered":"2024-02-21T23:13:30.249210+0000","last_unstale":"2024-02-21T23:13:30.254549+0000","last_undegraded":"2024-02-21T23:13:30.254549+0000","last_fullsized":"2024-02-21T23:13:30.254549+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:33:16.054241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.16","version":"0'0","reported_seq":12,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.250862+0000","last_change":"2024-02-21T23:13:29.268396+0000","last_active":"2024-02-21T23:13:30.250862+0000","last_peered":"2024-02-21T23:13:30.250862+0000","last_clean":"2024-02-21T23:13:30.250862+0000","last_became_active":"2024-02-21T23:13:29.268105+0000","last_became_peered":"2024-02-21T23:13:29.268105+0000","last_unstale":"2024-02-21T23:13:30.250862+0000","last_undegraded":"2024-02-21T23:13:30.250862+0000","last_fullsized":"2024-02-21T23:13:30.250862+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_clean_scrub_stamp":"2024-02-21T23:13:28.197327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:57:51.373442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.16","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.276407+0000","last_change":"2024-02-21T23:13:30.272466+0000","last_active":"2024-02-21T23:13:30.276407+0000","last_peered":"2024-02-21T23:13:30.276407+0000","last_clean":"2024-02-21T23:13:30.276407+0000","last_became_active":"2024-02-21T23:13:30.272195+0000","last_became_peered":"2024-02-21T23:13:30.272195+0000","last_unstale":"2024-02-21T23:13:30.276407+0000","last_undegraded":"2024-02-21T23:13:30.276407+0000","last_fullsized":"2024-02-21T23:13:30.276407+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:43:37.680664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.15","version":"207'3","reported_seq":15,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.288052+0000","last_change":"2024-02-21T23:13:29.250853+0000","last_active":"2024-02-21T23:13:30.288052+0000","last_peered":"2024-02-21T23:13:30.288052+0000","last_clean":"2024-02-21T23:13:30.288052+0000","last_became_active":"2024-02-21T23:13:29.250539+0000","last_became_peered":"2024-02-21T23:13:29.250539+0000","last_unstale":"2024-02-21T23:13:30.288052+0000","last_undegraded":"2024-02-21T23:13:30.288052+0000","last_fullsized":"2024-02-21T23:13:30.288052+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_clean_scrub_stamp":"2024-02-21T23:13:28.197327+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:48:57.675316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.17","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.260868+0000","last_change":"2024-02-21T23:13:30.254100+0000","last_active":"2024-02-21T23:13:30.260868+0000","last_peered":"2024-02-21T23:13:30.260868+0000","last_clean":"2024-02-21T23:13:30.260868+0000","last_became_active":"2024-02-21T23:13:30.253615+0000","last_became_peered":"2024-02-21T23:13:30.253615+0000","last_unstale":"2024-02-21T23:13:30.260868+0000","last_undegraded":"2024-02-21T23:13:30.260868+0000","last_fullsized":"2024-02-21T23:13:30.260868+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:36:45.344984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.14","version":"207'6","reported_seq":18,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.290724+0000","last_change":"2024-02-21T23:13:29.255335+0000","last_active":"2024-02-21T23:13:30.290724+0000","last_peered":"2024-02-21T23:13:30.290724+0000","last_clean":"2024-02-21T23:13:30.290724+0000","last_became_active":"2024-02-21T23:13:29.254755+0000","last_became_peered":"2024-02-21T23:13:29.254755+0000","last_unstale":"2024-02-21T23:13:30.290724+0000","last_undegraded":"2024-02-21T23:13:30.290724+0000","last_fullsized":"2024-02-21T23:13:30.290724+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_clean_scrub_stamp":"2024-02-21T23:13:28.197327+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:58:23.394041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.10","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.254448+0000","last_change":"2024-02-21T23:13:30.253618+0000","last_active":"2024-02-21T23:13:30.254448+0000","last_peered":"2024-02-21T23:13:30.254448+0000","last_clean":"2024-02-21T23:13:30.254448+0000","last_became_active":"2024-02-21T23:13:30.253331+0000","last_became_peered":"2024-02-21T23:13:30.253331+0000","last_unstale":"2024-02-21T23:13:30.254448+0000","last_undegraded":"2024-02-21T23:13:30.254448+0000","last_fullsized":"2024-02-21T23:13:30.254448+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:16:31.150704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.13","version":"207'2","reported_seq":14,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.279727+0000","last_change":"2024-02-21T23:13:29.268721+0000","last_active":"2024-02-21T23:13:30.279727+0000","last_peered":"2024-02-21T23:13:30.279727+0000","last_clean":"2024-02-21T23:13:30.279727+0000","last_became_active":"2024-02-21T23:13:29.268390+0000","last_became_peered":"2024-02-21T23:13:29.268390+0000","last_unstale":"2024-02-21T23:13:30.279727+0000","last_undegraded":"2024-02-21T23:13:30.279727+0000","last_fullsized":"2024-02-21T23:13:30.279727+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_clean_scrub_stamp":"2024-02-21T23:13:28.197327+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:43:30.772438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.11","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.270970+0000","last_change":"2024-02-21T23:13:30.266579+0000","last_active":"2024-02-21T23:13:30.270970+0000","last_peered":"2024-02-21T23:13:30.270970+0000","last_clean":"2024-02-21T23:13:30.270970+0000","last_became_active":"2024-02-21T23:13:30.264191+0000","last_became_peered":"2024-02-21T23:13:30.264191+0000","last_unstale":"2024-02-21T23:13:30.270970+0000","last_undegraded":"2024-02-21T23:13:30.270970+0000","last_fullsized":"2024-02-21T23:13:30.270970+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:53:03.421126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.12","version":"0'0","reported_seq":12,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.231969+0000","last_change":"2024-02-21T23:13:29.257416+0000","last_active":"2024-02-21T23:13:30.231969+0000","last_peered":"2024-02-21T23:13:30.231969+0000","last_clean":"2024-02-21T23:13:30.231969+0000","last_became_active":"2024-02-21T23:13:29.257124+0000","last_became_peered":"2024-02-21T23:13:29.257124+0000","last_unstale":"2024-02-21T23:13:30.231969+0000","last_undegraded":"2024-02-21T23:13:30.231969+0000","last_fullsized":"2024-02-21T23:13:30.231969+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_clean_scrub_stamp":"2024-02-21T23:13:28.197327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:29:47.250213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.12","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.264176+0000","last_change":"2024-02-21T23:13:30.261523+0000","last_active":"2024-02-21T23:13:30.264176+0000","last_peered":"2024-02-21T23:13:30.264176+0000","last_clean":"2024-02-21T23:13:30.264176+0000","last_became_active":"2024-02-21T23:13:30.261255+0000","last_became_peered":"2024-02-21T23:13:30.261255+0000","last_unstale":"2024-02-21T23:13:30.264176+0000","last_undegraded":"2024-02-21T23:13:30.264176+0000","last_fullsized":"2024-02-21T23:13:30.264176+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:23:05.996217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.11","version":"0'0","reported_seq":12,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.229445+0000","last_change":"2024-02-21T23:13:29.263790+0000","last_active":"2024-02-21T23:13:30.229445+0000","last_peered":"2024-02-21T23:13:30.229445+0000","last_clean":"2024-02-21T23:13:30.229445+0000","last_became_active":"2024-02-21T23:13:29.263448+0000","last_became_peered":"2024-02-21T23:13:29.263448+0000","last_unstale":"2024-02-21T23:13:30.229445+0000","last_undegraded":"2024-02-21T23:13:30.229445+0000","last_fullsized":"2024-02-21T23:13:30.229445+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_clean_scrub_stamp":"2024-02-21T23:13:28.197327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:26:45.312782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.13","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.268224+0000","last_change":"2024-02-21T23:13:30.264461+0000","last_active":"2024-02-21T23:13:30.268224+0000","last_peered":"2024-02-21T23:13:30.268224+0000","last_clean":"2024-02-21T23:13:30.268224+0000","last_became_active":"2024-02-21T23:13:30.264174+0000","last_became_peered":"2024-02-21T23:13:30.264174+0000","last_unstale":"2024-02-21T23:13:30.268224+0000","last_undegraded":"2024-02-21T23:13:30.268224+0000","last_fullsized":"2024-02-21T23:13:30.268224+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:53:26.841237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.10","version":"207'8","reported_seq":20,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.287890+0000","last_change":"2024-02-21T23:13:29.254779+0000","last_active":"2024-02-21T23:13:30.287890+0000","last_peered":"2024-02-21T23:13:30.287890+0000","last_clean":"2024-02-21T23:13:30.287890+0000","last_became_active":"2024-02-21T23:13:29.253958+0000","last_became_peered":"2024-02-21T23:13:29.253958+0000","last_unstale":"2024-02-21T23:13:30.287890+0000","last_undegraded":"2024-02-21T23:13:30.287890+0000","last_fullsized":"2024-02-21T23:13:30.287890+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_clean_scrub_stamp":"2024-02-21T23:13:28.197327+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:01:25.183534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.c","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.260321+0000","last_change":"2024-02-21T23:13:30.254017+0000","last_active":"2024-02-21T23:13:30.260321+0000","last_peered":"2024-02-21T23:13:30.260321+0000","last_clean":"2024-02-21T23:13:30.260321+0000","last_became_active":"2024-02-21T23:13:30.253691+0000","last_became_peered":"2024-02-21T23:13:30.253691+0000","last_unstale":"2024-02-21T23:13:30.260321+0000","last_undegraded":"2024-02-21T23:13:30.260321+0000","last_fullsized":"2024-02-21T23:13:30.260321+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:36:57.815629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.f","version":"207'1","reported_seq":13,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.277268+0000","last_change":"2024-02-21T23:13:29.266149+0000","last_active":"2024-02-21T23:13:30.277268+0000","last_peered":"2024-02-21T23:13:30.277268+0000","last_clean":"2024-02-21T23:13:30.277268+0000","last_became_active":"2024-02-21T23:13:29.265660+0000","last_became_peered":"2024-02-21T23:13:29.265660+0000","last_unstale":"2024-02-21T23:13:30.277268+0000","last_undegraded":"2024-02-21T23:13:30.277268+0000","last_fullsized":"2024-02-21T23:13:30.277268+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_clean_scrub_stamp":"2024-02-21T23:13:28.197327+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T02:09:33.639743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.d","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.260403+0000","last_change":"2024-02-21T23:13:30.250755+0000","last_active":"2024-02-21T23:13:30.260403+0000","last_peered":"2024-02-21T23:13:30.260403+0000","last_clean":"2024-02-21T23:13:30.260403+0000","last_became_active":"2024-02-21T23:13:30.250359+0000","last_became_peered":"2024-02-21T23:13:30.250359+0000","last_unstale":"2024-02-21T23:13:30.260403+0000","last_undegraded":"2024-02-21T23:13:30.260403+0000","last_fullsized":"2024-02-21T23:13:30.260403+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:43:37.867934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.e","version":"207'2","reported_seq":14,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.284083+0000","last_change":"2024-02-21T23:13:29.265760+0000","last_active":"2024-02-21T23:13:30.284083+0000","last_peered":"2024-02-21T23:13:30.284083+0000","last_clean":"2024-02-21T23:13:30.284083+0000","last_became_active":"2024-02-21T23:13:29.264991+0000","last_became_peered":"2024-02-21T23:13:29.264991+0000","last_unstale":"2024-02-21T23:13:30.284083+0000","last_undegraded":"2024-02-21T23:13:30.284083+0000","last_fullsized":"2024-02-21T23:13:30.284083+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_clean_scrub_stamp":"2024-02-21T23:13:28.197327+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:55:14.303249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.e","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.267879+0000","last_change":"2024-02-21T23:13:30.258943+0000","last_active":"2024-02-21T23:13:30.267879+0000","last_peered":"2024-02-21T23:13:30.267879+0000","last_clean":"2024-02-21T23:13:30.267879+0000","last_became_active":"2024-02-21T23:13:30.257306+0000","last_became_peered":"2024-02-21T23:13:30.257306+0000","last_unstale":"2024-02-21T23:13:30.267879+0000","last_undegraded":"2024-02-21T23:13:30.267879+0000","last_fullsized":"2024-02-21T23:13:30.267879+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:34:40.892453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.d","version":"207'1","reported_seq":13,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.286368+0000","last_change":"2024-02-21T23:13:29.264426+0000","last_active":"2024-02-21T23:13:30.286368+0000","last_peered":"2024-02-21T23:13:30.286368+0000","last_clean":"2024-02-21T23:13:30.286368+0000","last_became_active":"2024-02-21T23:13:29.264113+0000","last_became_peered":"2024-02-21T23:13:29.264113+0000","last_unstale":"2024-02-21T23:13:30.286368+0000","last_undegraded":"2024-02-21T23:13:30.286368+0000","last_fullsized":"2024-02-21T23:13:30.286368+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_clean_scrub_stamp":"2024-02-21T23:13:28.197327+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:30:49.167568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.f","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.266648+0000","last_change":"2024-02-21T23:13:30.264032+0000","last_active":"2024-02-21T23:13:30.266648+0000","last_peered":"2024-02-21T23:13:30.266648+0000","last_clean":"2024-02-21T23:13:30.266648+0000","last_became_active":"2024-02-21T23:13:30.261622+0000","last_became_peered":"2024-02-21T23:13:30.261622+0000","last_unstale":"2024-02-21T23:13:30.266648+0000","last_undegraded":"2024-02-21T23:13:30.266648+0000","last_fullsized":"2024-02-21T23:13:30.266648+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:22:46.829986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.c","version":"0'0","reported_seq":12,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.226083+0000","last_change":"2024-02-21T23:13:29.262574+0000","last_active":"2024-02-21T23:13:30.226083+0000","last_peered":"2024-02-21T23:13:30.226083+0000","last_clean":"2024-02-21T23:13:30.226083+0000","last_became_active":"2024-02-21T23:13:29.262221+0000","last_became_peered":"2024-02-21T23:13:29.262221+0000","last_unstale":"2024-02-21T23:13:30.226083+0000","last_undegraded":"2024-02-21T23:13:30.226083+0000","last_fullsized":"2024-02-21T23:13:30.226083+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_clean_scrub_stamp":"2024-02-21T23:13:28.197327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:54:11.272648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.8","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.262418+0000","last_change":"2024-02-21T23:13:30.258489+0000","last_active":"2024-02-21T23:13:30.262418+0000","last_peered":"2024-02-21T23:13:30.262418+0000","last_clean":"2024-02-21T23:13:30.262418+0000","last_became_active":"2024-02-21T23:13:30.258179+0000","last_became_peered":"2024-02-21T23:13:30.258179+0000","last_unstale":"2024-02-21T23:13:30.262418+0000","last_undegraded":"2024-02-21T23:13:30.262418+0000","last_fullsized":"2024-02-21T23:13:30.262418+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:16:31.150704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.b","version":"207'2","reported_seq":14,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.286247+0000","last_change":"2024-02-21T23:13:29.252890+0000","last_active":"2024-02-21T23:13:30.286247+0000","last_peered":"2024-02-21T23:13:30.286247+0000","last_clean":"2024-02-21T23:13:30.286247+0000","last_became_active":"2024-02-21T23:13:29.250884+0000","last_became_peered":"2024-02-21T23:13:29.250884+0000","last_unstale":"2024-02-21T23:13:30.286247+0000","last_undegraded":"2024-02-21T23:13:30.286247+0000","last_fullsized":"2024-02-21T23:13:30.286247+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_clean_scrub_stamp":"2024-02-21T23:13:28.197327+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:42:47.677914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.9","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.251544+0000","last_change":"2024-02-21T23:13:30.243777+0000","last_active":"2024-02-21T23:13:30.251544+0000","last_peered":"2024-02-21T23:13:30.251544+0000","last_clean":"2024-02-21T23:13:30.251544+0000","last_became_active":"2024-02-21T23:13:30.243061+0000","last_became_peered":"2024-02-21T23:13:30.243061+0000","last_unstale":"2024-02-21T23:13:30.251544+0000","last_undegraded":"2024-02-21T23:13:30.251544+0000","last_fullsized":"2024-02-21T23:13:30.251544+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:29:27.260925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.a","version":"0'0","reported_seq":12,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.239864+0000","last_change":"2024-02-21T23:13:29.258895+0000","last_active":"2024-02-21T23:13:30.239864+0000","last_peered":"2024-02-21T23:13:30.239864+0000","last_clean":"2024-02-21T23:13:30.239864+0000","last_became_active":"2024-02-21T23:13:29.258598+0000","last_became_peered":"2024-02-21T23:13:29.258598+0000","last_unstale":"2024-02-21T23:13:30.239864+0000","last_undegraded":"2024-02-21T23:13:30.239864+0000","last_fullsized":"2024-02-21T23:13:30.239864+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_clean_scrub_stamp":"2024-02-21T23:13:28.197327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T10:34:13.155725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.a","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.256530+0000","last_change":"2024-02-21T23:13:30.248928+0000","last_active":"2024-02-21T23:13:30.256530+0000","last_peered":"2024-02-21T23:13:30.256530+0000","last_clean":"2024-02-21T23:13:30.256530+0000","last_became_active":"2024-02-21T23:13:30.248452+0000","last_became_peered":"2024-02-21T23:13:30.248452+0000","last_unstale":"2024-02-21T23:13:30.256530+0000","last_undegraded":"2024-02-21T23:13:30.256530+0000","last_fullsized":"2024-02-21T23:13:30.256530+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:58:12.797695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.9","version":"0'0","reported_seq":12,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.254064+0000","last_change":"2024-02-21T23:13:29.254284+0000","last_active":"2024-02-21T23:13:30.254064+0000","last_peered":"2024-02-21T23:13:30.254064+0000","last_clean":"2024-02-21T23:13:30.254064+0000","last_became_active":"2024-02-21T23:13:29.253638+0000","last_became_peered":"2024-02-21T23:13:29.253638+0000","last_unstale":"2024-02-21T23:13:30.254064+0000","last_undegraded":"2024-02-21T23:13:30.254064+0000","last_fullsized":"2024-02-21T23:13:30.254064+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_clean_scrub_stamp":"2024-02-21T23:13:28.197327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:42:47.677914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.b","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.281158+0000","last_change":"2024-02-21T23:13:30.278409+0000","last_active":"2024-02-21T23:13:30.281158+0000","last_peered":"2024-02-21T23:13:30.281158+0000","last_clean":"2024-02-21T23:13:30.281158+0000","last_became_active":"2024-02-21T23:13:30.278081+0000","last_became_peered":"2024-02-21T23:13:30.278081+0000","last_unstale":"2024-02-21T23:13:30.281158+0000","last_undegraded":"2024-02-21T23:13:30.281158+0000","last_fullsized":"2024-02-21T23:13:30.281158+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:16:17.460840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.8","version":"0'0","reported_seq":12,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.233525+0000","last_change":"2024-02-21T23:13:29.248784+0000","last_active":"2024-02-21T23:13:30.233525+0000","last_peered":"2024-02-21T23:13:30.233525+0000","last_clean":"2024-02-21T23:13:30.233525+0000","last_became_active":"2024-02-21T23:13:29.248370+0000","last_became_peered":"2024-02-21T23:13:29.248370+0000","last_unstale":"2024-02-21T23:13:30.233525+0000","last_undegraded":"2024-02-21T23:13:30.233525+0000","last_fullsized":"2024-02-21T23:13:30.233525+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_clean_scrub_stamp":"2024-02-21T23:13:28.197327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T00:13:57.983133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.4","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.274340+0000","last_change":"2024-02-21T23:13:30.270565+0000","last_active":"2024-02-21T23:13:30.274340+0000","last_peered":"2024-02-21T23:13:30.274340+0000","last_clean":"2024-02-21T23:13:30.274340+0000","last_became_active":"2024-02-21T23:13:30.270030+0000","last_became_peered":"2024-02-21T23:13:30.270030+0000","last_unstale":"2024-02-21T23:13:30.274340+0000","last_undegraded":"2024-02-21T23:13:30.274340+0000","last_fullsized":"2024-02-21T23:13:30.274340+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:48:44.645285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.7","version":"207'2","reported_seq":14,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.285282+0000","last_change":"2024-02-21T23:13:29.256652+0000","last_active":"2024-02-21T23:13:30.285282+0000","last_peered":"2024-02-21T23:13:30.285282+0000","last_clean":"2024-02-21T23:13:30.285282+0000","last_became_active":"2024-02-21T23:13:29.255581+0000","last_became_peered":"2024-02-21T23:13:29.255581+0000","last_unstale":"2024-02-21T23:13:30.285282+0000","last_undegraded":"2024-02-21T23:13:30.285282+0000","last_fullsized":"2024-02-21T23:13:30.285282+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_clean_scrub_stamp":"2024-02-21T23:13:28.197327+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:21:27.555571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.5","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.274386+0000","last_change":"2024-02-21T23:13:30.270663+0000","last_active":"2024-02-21T23:13:30.274386+0000","last_peered":"2024-02-21T23:13:30.274386+0000","last_clean":"2024-02-21T23:13:30.274386+0000","last_became_active":"2024-02-21T23:13:30.270322+0000","last_became_peered":"2024-02-21T23:13:30.270322+0000","last_unstale":"2024-02-21T23:13:30.274386+0000","last_undegraded":"2024-02-21T23:13:30.274386+0000","last_fullsized":"2024-02-21T23:13:30.274386+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:57:45.202740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.6","version":"0'0","reported_seq":12,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.230826+0000","last_change":"2024-02-21T23:13:29.259152+0000","last_active":"2024-02-21T23:13:30.230826+0000","last_peered":"2024-02-21T23:13:30.230826+0000","last_clean":"2024-02-21T23:13:30.230826+0000","last_became_active":"2024-02-21T23:13:29.258871+0000","last_became_peered":"2024-02-21T23:13:29.258871+0000","last_unstale":"2024-02-21T23:13:30.230826+0000","last_undegraded":"2024-02-21T23:13:30.230826+0000","last_fullsized":"2024-02-21T23:13:30.230826+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_clean_scrub_stamp":"2024-02-21T23:13:28.197327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:30:50.944777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.6","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.264318+0000","last_change":"2024-02-21T23:13:30.257184+0000","last_active":"2024-02-21T23:13:30.264318+0000","last_peered":"2024-02-21T23:13:30.264318+0000","last_clean":"2024-02-21T23:13:30.264318+0000","last_became_active":"2024-02-21T23:13:30.256701+0000","last_became_peered":"2024-02-21T23:13:30.256701+0000","last_unstale":"2024-02-21T23:13:30.264318+0000","last_undegraded":"2024-02-21T23:13:30.264318+0000","last_fullsized":"2024-02-21T23:13:30.264318+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T05:57:45.202740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.5","version":"0'0","reported_seq":12,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.243561+0000","last_change":"2024-02-21T23:13:29.268039+0000","last_active":"2024-02-21T23:13:30.243561+0000","last_peered":"2024-02-21T23:13:30.243561+0000","last_clean":"2024-02-21T23:13:30.243561+0000","last_became_active":"2024-02-21T23:13:29.267311+0000","last_became_peered":"2024-02-21T23:13:29.267311+0000","last_unstale":"2024-02-21T23:13:30.243561+0000","last_undegraded":"2024-02-21T23:13:30.243561+0000","last_fullsized":"2024-02-21T23:13:30.243561+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_clean_scrub_stamp":"2024-02-21T23:13:28.197327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:07:43.967351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.7","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.274591+0000","last_change":"2024-02-21T23:13:30.269247+0000","last_active":"2024-02-21T23:13:30.274591+0000","last_peered":"2024-02-21T23:13:30.274591+0000","last_clean":"2024-02-21T23:13:30.274591+0000","last_became_active":"2024-02-21T23:13:30.268530+0000","last_became_peered":"2024-02-21T23:13:30.268530+0000","last_unstale":"2024-02-21T23:13:30.274591+0000","last_undegraded":"2024-02-21T23:13:30.274591+0000","last_fullsized":"2024-02-21T23:13:30.274591+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:30:50.183494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.4","version":"0'0","reported_seq":12,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.245474+0000","last_change":"2024-02-21T23:13:29.261707+0000","last_active":"2024-02-21T23:13:30.245474+0000","last_peered":"2024-02-21T23:13:30.245474+0000","last_clean":"2024-02-21T23:13:30.245474+0000","last_became_active":"2024-02-21T23:13:29.261351+0000","last_became_peered":"2024-02-21T23:13:29.261351+0000","last_unstale":"2024-02-21T23:13:30.245474+0000","last_undegraded":"2024-02-21T23:13:30.245474+0000","last_fullsized":"2024-02-21T23:13:30.245474+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_clean_scrub_stamp":"2024-02-21T23:13:28.197327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:59:37.531428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.0","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.267489+0000","last_change":"2024-02-21T23:13:30.262666+0000","last_active":"2024-02-21T23:13:30.267489+0000","last_peered":"2024-02-21T23:13:30.267489+0000","last_clean":"2024-02-21T23:13:30.267489+0000","last_became_active":"2024-02-21T23:13:30.262329+0000","last_became_peered":"2024-02-21T23:13:30.262329+0000","last_unstale":"2024-02-21T23:13:30.267489+0000","last_undegraded":"2024-02-21T23:13:30.267489+0000","last_fullsized":"2024-02-21T23:13:30.267489+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:07:27.307529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.3","version":"0'0","reported_seq":12,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.225427+0000","last_change":"2024-02-21T23:13:29.255576+0000","last_active":"2024-02-21T23:13:30.225427+0000","last_peered":"2024-02-21T23:13:30.225427+0000","last_clean":"2024-02-21T23:13:30.225427+0000","last_became_active":"2024-02-21T23:13:29.255315+0000","last_became_peered":"2024-02-21T23:13:29.255315+0000","last_unstale":"2024-02-21T23:13:30.225427+0000","last_undegraded":"2024-02-21T23:13:30.225427+0000","last_fullsized":"2024-02-21T23:13:30.225427+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_clean_scrub_stamp":"2024-02-21T23:13:28.197327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:06:35.674377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.1","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.278181+0000","last_change":"2024-02-21T23:13:30.263695+0000","last_active":"2024-02-21T23:13:30.278181+0000","last_peered":"2024-02-21T23:13:30.278181+0000","last_clean":"2024-02-21T23:13:30.278181+0000","last_became_active":"2024-02-21T23:13:30.262207+0000","last_became_peered":"2024-02-21T23:13:30.262207+0000","last_unstale":"2024-02-21T23:13:30.278181+0000","last_undegraded":"2024-02-21T23:13:30.278181+0000","last_fullsized":"2024-02-21T23:13:30.278181+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T04:57:39.946172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.2","version":"0'0","reported_seq":12,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.225440+0000","last_change":"2024-02-21T23:13:29.253396+0000","last_active":"2024-02-21T23:13:30.225440+0000","last_peered":"2024-02-21T23:13:30.225440+0000","last_clean":"2024-02-21T23:13:30.225440+0000","last_became_active":"2024-02-21T23:13:29.251990+0000","last_became_peered":"2024-02-21T23:13:29.251990+0000","last_unstale":"2024-02-21T23:13:30.225440+0000","last_undegraded":"2024-02-21T23:13:30.225440+0000","last_fullsized":"2024-02-21T23:13:30.225440+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_clean_scrub_stamp":"2024-02-21T23:13:28.197327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T06:17:54.727926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.2","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.278028+0000","last_change":"2024-02-21T23:13:30.271186+0000","last_active":"2024-02-21T23:13:30.278028+0000","last_peered":"2024-02-21T23:13:30.278028+0000","last_clean":"2024-02-21T23:13:30.278028+0000","last_became_active":"2024-02-21T23:13:30.270349+0000","last_became_peered":"2024-02-21T23:13:30.270349+0000","last_unstale":"2024-02-21T23:13:30.278028+0000","last_undegraded":"2024-02-21T23:13:30.278028+0000","last_fullsized":"2024-02-21T23:13:30.278028+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:20:12.018424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1","version":"0'0","reported_seq":12,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.241323+0000","last_change":"2024-02-21T23:13:29.266996+0000","last_active":"2024-02-21T23:13:30.241323+0000","last_peered":"2024-02-21T23:13:30.241323+0000","last_clean":"2024-02-21T23:13:30.241323+0000","last_became_active":"2024-02-21T23:13:29.266639+0000","last_became_peered":"2024-02-21T23:13:29.266639+0000","last_unstale":"2024-02-21T23:13:30.241323+0000","last_undegraded":"2024-02-21T23:13:30.241323+0000","last_fullsized":"2024-02-21T23:13:30.241323+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_clean_scrub_stamp":"2024-02-21T23:13:28.197327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T03:45:50.645184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.3","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.259885+0000","last_change":"2024-02-21T23:13:30.254837+0000","last_active":"2024-02-21T23:13:30.259885+0000","last_peered":"2024-02-21T23:13:30.259885+0000","last_clean":"2024-02-21T23:13:30.259885+0000","last_became_active":"2024-02-21T23:13:30.254523+0000","last_became_peered":"2024-02-21T23:13:30.254523+0000","last_unstale":"2024-02-21T23:13:30.259885+0000","last_undegraded":"2024-02-21T23:13:30.259885+0000","last_fullsized":"2024-02-21T23:13:30.259885+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T07:30:50.183494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.0","version":"0'0","reported_seq":12,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.225103+0000","last_change":"2024-02-21T23:13:29.267982+0000","last_active":"2024-02-21T23:13:30.225103+0000","last_peered":"2024-02-21T23:13:30.225103+0000","last_clean":"2024-02-21T23:13:30.225103+0000","last_became_active":"2024-02-21T23:13:29.267673+0000","last_became_peered":"2024-02-21T23:13:29.267673+0000","last_unstale":"2024-02-21T23:13:30.225103+0000","last_undegraded":"2024-02-21T23:13:30.225103+0000","last_fullsized":"2024-02-21T23:13:30.225103+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":205,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:28.197327+0000","last_clean_scrub_stamp":"2024-02-21T23:13:28.197327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T01:20:58.752032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.23","version":"0'0","reported_seq":9,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.268040+0000","last_change":"2024-02-21T23:13:30.262998+0000","last_active":"2024-02-21T23:13:30.268040+0000","last_peered":"2024-02-21T23:13:30.268040+0000","last_clean":"2024-02-21T23:13:30.268040+0000","last_became_active":"2024-02-21T23:13:30.257390+0000","last_became_peered":"2024-02-21T23:13:30.257390+0000","last_unstale":"2024-02-21T23:13:30.268040+0000","last_undegraded":"2024-02-21T23:13:30.268040+0000","last_fullsized":"2024-02-21T23:13:30.268040+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":206,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:13:29.213253+0000","last_clean_scrub_stamp":"2024-02-21T23:13:29.213253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T09:58:57.387154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":518,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-02-21T23:13:30.249810+0000","last_change":"2024-02-21T23:02:32.289335+0000","last_active":"2024-02-21T23:13:30.249810+0000","last_peered":"2024-02-21T23:13:30.249810+0000","last_clean":"2024-02-21T23:13:30.249810+0000","last_became_active":"2024-02-21T23:02:32.286913+0000","last_became_peered":"2024-02-21T23:02:32.286913+0000","last_unstale":"2024-02-21T23:13:30.249810+0000","last_undegraded":"2024-02-21T23:13:30.249810+0000","last_fullsized":"2024-02-21T23:13:30.249810+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_clean_scrub_stamp":"2024-02-21T23:02:31.172569+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:36:23.987374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":34,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":33,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574982,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33800,"kb_used_data":2620,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338040,"statfs":{"total":96636764160,"available":96602152960,"internally_reserved":0,"allocated":2682880,"data_stored":1869802,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574983,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33760,"kb_used_data":2572,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338080,"statfs":{"total":96636764160,"available":96602193920,"internally_reserved":0,"allocated":2633728,"data_stored":1286635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574982,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33760,"kb_used_data":2572,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338080,"statfs":{"total":96636764160,"available":96602193920,"internally_reserved":0,"allocated":2633728,"data_stored":1286671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574983,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33760,"kb_used_data":2580,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338080,"statfs":{"total":96636764160,"available":96602193920,"internally_reserved":0,"allocated":2641920,"data_stored":1288213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574983,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37912,"kb_used_data":2572,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333928,"statfs":{"total":96636764160,"available":96597942272,"internally_reserved":0,"allocated":2633728,"data_stored":1286691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574984,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37928,"kb_used_data":2588,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333912,"statfs":{"total":96636764160,"available":96597925888,"internally_reserved":0,"allocated":2650112,"data_stored":1288817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574982,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37908,"kb_used_data":2568,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333932,"statfs":{"total":96636764160,"available":96597946368,"internally_reserved":0,"allocated":2629632,"data_stored":1286601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574982,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37960,"kb_used_data":2620,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333880,"statfs":{"total":96636764160,"available":96597893120,"internally_reserved":0,"allocated":2682880,"data_stored":1869802,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-21T23:13:35.220 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-02-21T23:13:35.220 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-02-21T23:13:35.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.502+0000 7f4d72200640 1 Processor -- start 2024-02-21T23:13:35.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.502+0000 7f4d72200640 1 -- start start 2024-02-21T23:13:35.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.504+0000 7f4d72200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4d6c14eb40 0x7f4d6c151fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:35.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.504+0000 7f4d72200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d6c0c38f0 con 0x7f4d6c072620 2024-02-21T23:13:35.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.504+0000 7f4d72200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d6c0c3330 con 0x7f4d6c072a80 2024-02-21T23:13:35.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.504+0000 7f4d72200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d6c0c3100 con 0x7f4d6c14eb40 2024-02-21T23:13:35.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.504+0000 7f4d6aa00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f4d6c072620 0x7f4d6c06b4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60962/0 (socket says 172.21.15.28:60962) 2024-02-21T23:13:35.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.504+0000 7f4d6aa00640 1 -- 172.21.15.28:0/643261074 learned_addr learned my addr 172.21.15.28:0/643261074 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:13:35.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.504+0000 7f4d6b400640 1 --2- 172.21.15.28:0/643261074 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4d6c14eb40 0x7f4d6c151fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:35.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.505+0000 7f4d6a000640 1 -- 172.21.15.28:0/643261074 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 478163728 0 0) 0x7f4d6c0c38f0 con 0x7f4d6c072620 2024-02-21T23:13:35.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.505+0000 7f4d6a000640 1 -- 172.21.15.28:0/643261074 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d50003520 con 0x7f4d6c072620 2024-02-21T23:13:35.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.505+0000 7f4d6a000640 1 -- 172.21.15.28:0/643261074 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 690618207 0 0) 0x7f4d6c0c3330 con 0x7f4d6c072a80 2024-02-21T23:13:35.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.505+0000 7f4d6a000640 1 -- 172.21.15.28:0/643261074 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d6c0c38f0 con 0x7f4d6c072a80 2024-02-21T23:13:35.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.505+0000 7f4d6a000640 1 -- 172.21.15.28:0/643261074 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2160928431 0 0) 0x7f4d50003520 con 0x7f4d6c072620 2024-02-21T23:13:35.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.505+0000 7f4d6a000640 1 -- 172.21.15.28:0/643261074 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4d6c14eb40 msgr2=0x7f4d6c151fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:35.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.505+0000 7f4d6a000640 1 --2- 172.21.15.28:0/643261074 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4d6c14eb40 0x7f4d6c151fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:35.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.505+0000 7f4d6a000640 1 -- 172.21.15.28:0/643261074 >> v1:172.21.15.81:6789/0 conn(0x7f4d6c072a80 legacy=0x7f4d6c06bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:35.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.505+0000 7f4d6a000640 1 -- 172.21.15.28:0/643261074 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d6c0c3610 con 0x7f4d6c072620 2024-02-21T23:13:35.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.505+0000 7f4d6a000640 1 -- 172.21.15.28:0/643261074 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f4d580015f0 con 0x7f4d6c072620 2024-02-21T23:13:35.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.506+0000 7f4d6a000640 1 -- 172.21.15.28:0/643261074 >> v1:172.21.15.28:6789/0 conn(0x7f4d6c072620 legacy=0x7f4d6c06b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:35.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.506+0000 7f4d6a000640 1 --2- 172.21.15.28:0/643261074 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d50003b70 0x7f4d50003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:35.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.506+0000 7f4d6a000640 1 --2- 172.21.15.28:0/643261074 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d50004800 0x7f4d50004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:35.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.506+0000 7f4d6a000640 1 --2- 172.21.15.28:0/643261074 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4d6c14eb40 0x7f4d50005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:35.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.506+0000 7f4d6a000640 1 -- 172.21.15.28:0/643261074 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d5000bb50 con 0x7f4d50003b70 2024-02-21T23:13:35.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.506+0000 7f4d6a000640 1 -- 172.21.15.28:0/643261074 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d5000bcd0 con 0x7f4d50004800 2024-02-21T23:13:35.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.506+0000 7f4d6a000640 1 -- 172.21.15.28:0/643261074 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d5000bea0 con 0x7f4d6c14eb40 2024-02-21T23:13:35.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.506+0000 7f4d6b400640 1 --2- 172.21.15.28:0/643261074 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d50003b70 0x7f4d50003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:35.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.506+0000 7f4d6aa00640 1 --2- 172.21.15.28:0/643261074 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d50004800 0x7f4d50004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:35.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.506+0000 7f4d6be00640 1 --2- 172.21.15.28:0/643261074 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4d6c14eb40 0x7f4d50005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:35.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.507+0000 7f4d6b400640 1 -- 172.21.15.28:0/643261074 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4d6c14eb40 msgr2=0x7f4d50005220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:35.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.507+0000 7f4d6b400640 1 --2- 172.21.15.28:0/643261074 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4d6c14eb40 0x7f4d50005220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:35.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.507+0000 7f4d6b400640 1 -- 172.21.15.28:0/643261074 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d50004800 msgr2=0x7f4d50004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:35.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.507+0000 7f4d6b400640 1 --2- 172.21.15.28:0/643261074 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d50004800 0x7f4d50004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:35.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.507+0000 7f4d6b400640 1 -- 172.21.15.28:0/643261074 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d50003520 con 0x7f4d50003b70 2024-02-21T23:13:35.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.508+0000 7f4d6b400640 1 --2- 172.21.15.28:0/643261074 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d50003b70 0x7f4d50003f70 secure :-1 s=READY pgs=2381 cs=0 l=1 rev1=1 crypto rx=0x7f4d5c009af0 tx=0x7f4d50005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c958dab6aba671b server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:35.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.508+0000 7f4d6a000640 1 -- 172.21.15.28:0/643261074 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d5c039070 con 0x7f4d50003b70 2024-02-21T23:13:35.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.508+0000 7f4d6a000640 1 -- 172.21.15.28:0/643261074 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d5c02c6b0 con 0x7f4d50003b70 2024-02-21T23:13:35.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.508+0000 7f4d6a000640 1 -- 172.21.15.28:0/643261074 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d5c009690 con 0x7f4d50003b70 2024-02-21T23:13:35.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.509+0000 7f4d72200640 1 -- 172.21.15.28:0/643261074 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d50003b70 msgr2=0x7f4d50003f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:35.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.509+0000 7f4d72200640 1 --2- 172.21.15.28:0/643261074 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d50003b70 0x7f4d50003f70 secure :-1 s=READY pgs=2381 cs=0 l=1 rev1=1 crypto rx=0x7f4d5c009af0 tx=0x7f4d50005830 comp rx=0 tx=0).stop 2024-02-21T23:13:35.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.509+0000 7f4d6be00640 1 -- 172.21.15.28:0/643261074 reap_dead start 2024-02-21T23:13:35.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.509+0000 7f4d72200640 1 -- 172.21.15.28:0/643261074 shutdown_connections 2024-02-21T23:13:35.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.509+0000 7f4d72200640 1 -- 172.21.15.28:0/643261074 >> 172.21.15.28:0/643261074 conn(0x7f4d6c071b00 msgr2=0x7f4d6c071f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:35.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.509+0000 7f4d72200640 1 -- 172.21.15.28:0/643261074 shutdown_connections 2024-02-21T23:13:35.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.509+0000 7f4d72200640 1 -- 172.21.15.28:0/643261074 wait complete. 2024-02-21T23:13:35.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.511+0000 7f4d72200640 1 Processor -- start 2024-02-21T23:13:35.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.511+0000 7f4d72200640 1 -- start start 2024-02-21T23:13:35.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.512+0000 7f4d72200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d50003b70 0x7f4d6c142810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:35.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.512+0000 7f4d72200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4d6c1406b0 0x7f4d6c14a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:35.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.512+0000 7f4d72200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d6c0c5cf0 0x7f4d6c13af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:35.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.512+0000 7f4d72200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d6c0c15c0 con 0x7f4d50003b70 2024-02-21T23:13:35.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.512+0000 7f4d72200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d6c14ef20 con 0x7f4d6c0c5cf0 2024-02-21T23:13:35.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.512+0000 7f4d72200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d6c0c1740 con 0x7f4d6c1406b0 2024-02-21T23:13:35.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.512+0000 7f4d6b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d50003b70 0x7f4d6c142810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:35.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.512+0000 7f4d6b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d50003b70 0x7f4d6c142810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39372/0 (socket says 172.21.15.28:39372) 2024-02-21T23:13:35.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.512+0000 7f4d6b400640 1 -- 172.21.15.28:0/542076252 learned_addr learned my addr 172.21.15.28:0/542076252 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:35.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.512+0000 7f4d6be00640 1 --2- 172.21.15.28:0/542076252 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d6c0c5cf0 0x7f4d6c13af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:35.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.512+0000 7f4d6b400640 1 -- 172.21.15.28:0/542076252 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4d6c1406b0 msgr2=0x7f4d6c14a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:35.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.512+0000 7f4d6b400640 1 --2- 172.21.15.28:0/542076252 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4d6c1406b0 0x7f4d6c14a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:35.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.512+0000 7f4d6b400640 1 -- 172.21.15.28:0/542076252 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d6c0c5cf0 msgr2=0x7f4d6c13af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:35.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.512+0000 7f4d6b400640 1 --2- 172.21.15.28:0/542076252 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d6c0c5cf0 0x7f4d6c13af00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:35.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.512+0000 7f4d6b400640 1 -- 172.21.15.28:0/542076252 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d6c14db80 con 0x7f4d50003b70 2024-02-21T23:13:35.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.513+0000 7f4d6b400640 1 --2- 172.21.15.28:0/542076252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d50003b70 0x7f4d6c142810 secure :-1 s=READY pgs=2382 cs=0 l=1 rev1=1 crypto rx=0x7f4d5c037040 tx=0x7f4d5c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:35.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.513+0000 7f4d63400640 1 -- 172.21.15.28:0/542076252 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d5c042020 con 0x7f4d50003b70 2024-02-21T23:13:35.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.513+0000 7f4d63400640 1 -- 172.21.15.28:0/542076252 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d5c03a040 con 0x7f4d50003b70 2024-02-21T23:13:35.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.513+0000 7f4d63400640 1 -- 172.21.15.28:0/542076252 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d5c039070 con 0x7f4d50003b70 2024-02-21T23:13:35.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.513+0000 7f4d72200640 1 -- 172.21.15.28:0/542076252 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d6c14de00 con 0x7f4d50003b70 2024-02-21T23:13:35.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.513+0000 7f4d72200640 1 -- 172.21.15.28:0/542076252 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d6c142010 con 0x7f4d50003b70 2024-02-21T23:13:35.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.515+0000 7f4d72200640 1 -- 172.21.15.28:0/542076252 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d6c14e320 con 0x7f4d50003b70 2024-02-21T23:13:35.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.516+0000 7f4d63400640 1 -- 172.21.15.28:0/542076252 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4d5c045020 con 0x7f4d50003b70 2024-02-21T23:13:35.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.517+0000 7f4d63400640 1 --2- 172.21.15.28:0/542076252 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4d4c06b500 0x7f4d4c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:35.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.517+0000 7f4d63400640 1 -- 172.21.15.28:0/542076252 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4d5c0ab770 con 0x7f4d50003b70 2024-02-21T23:13:35.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.518+0000 7f4d6aa00640 1 --2- 172.21.15.28:0/542076252 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4d4c06b500 0x7f4d4c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:35.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.519+0000 7f4d6aa00640 1 --2- 172.21.15.28:0/542076252 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4d4c06b500 0x7f4d4c06d9c0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f4d58001eb0 tx=0x7f4d58001e20 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:35.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.521+0000 7f4d63400640 1 -- 172.21.15.28:0/542076252 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4d6c14e320 con 0x7f4d50003b70 2024-02-21T23:13:35.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.796+0000 7f4d72200640 1 -- 172.21.15.28:0/542076252 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f4d6c151fb0 con 0x7f4d50003b70 2024-02-21T23:13:35.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.800+0000 7f4d63400640 1 -- 172.21.15.28:0/542076252 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v23) v1 ==== 190+0+147 (secure 0 0 0) 0x7f4d6c151fb0 con 0x7f4d50003b70 2024-02-21T23:13:35.789 INFO:teuthology.orchestra.run.smithi028.stdout:[client.0] 2024-02-21T23:13:35.789 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQDTgNZlILvYLxAAoREWxKKT7YcXnhkgk8YvoQ== 2024-02-21T23:13:35.789 INFO:teuthology.orchestra.run.smithi028.stdout: caps mds = "allow" 2024-02-21T23:13:35.789 INFO:teuthology.orchestra.run.smithi028.stdout: caps mon = "allow r" 2024-02-21T23:13:35.789 INFO:teuthology.orchestra.run.smithi028.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-02-21T23:13:35.790 INFO:teuthology.orchestra.run.smithi028.stderr:updated caps for client.0 2024-02-21T23:13:35.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.804+0000 7f4d72200640 1 -- 172.21.15.28:0/542076252 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4d4c06b500 msgr2=0x7f4d4c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:35.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.804+0000 7f4d72200640 1 --2- 172.21.15.28:0/542076252 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4d4c06b500 0x7f4d4c06d9c0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f4d58001eb0 tx=0x7f4d58001e20 comp rx=0 tx=0).stop 2024-02-21T23:13:35.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.804+0000 7f4d72200640 1 -- 172.21.15.28:0/542076252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d50003b70 msgr2=0x7f4d6c142810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:35.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.804+0000 7f4d72200640 1 --2- 172.21.15.28:0/542076252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d50003b70 0x7f4d6c142810 secure :-1 s=READY pgs=2382 cs=0 l=1 rev1=1 crypto rx=0x7f4d5c037040 tx=0x7f4d5c004730 comp rx=0 tx=0).stop 2024-02-21T23:13:35.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.804+0000 7f4d72200640 1 -- 172.21.15.28:0/542076252 shutdown_connections 2024-02-21T23:13:35.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.804+0000 7f4d72200640 1 --2- 172.21.15.28:0/542076252 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4d4c06b500 0x7f4d4c06d9c0 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:35.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.804+0000 7f4d72200640 1 --2- 172.21.15.28:0/542076252 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4d6c1406b0 0x7f4d6c14a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:35.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.804+0000 7f4d72200640 1 --2- 172.21.15.28:0/542076252 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4d6c0c5cf0 0x7f4d6c13af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:35.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.804+0000 7f4d72200640 1 --2- 172.21.15.28:0/542076252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d50003b70 0x7f4d6c142810 unknown :-1 s=CLOSED pgs=2382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:35.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.804+0000 7f4d72200640 1 -- 172.21.15.28:0/542076252 >> 172.21.15.28:0/542076252 conn(0x7f4d6c071b00 msgr2=0x7f4d6c14f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:35.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.804+0000 7f4d72200640 1 -- 172.21.15.28:0/542076252 shutdown_connections 2024-02-21T23:13:35.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:35.804+0000 7f4d72200640 1 -- 172.21.15.28:0/542076252 wait complete. 2024-02-21T23:13:35.813 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:13:36.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.039+0000 7f0992a00640 1 Processor -- start 2024-02-21T23:13:36.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.040+0000 7f0992a00640 1 -- start start 2024-02-21T23:13:36.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.041+0000 7f0992a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f098c0ca050 0x7f098c1558c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:36.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.041+0000 7f0992a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f098c06bbf0 0x7f098c06c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:36.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.041+0000 7f0992a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f098c1528e0 con 0x7f098c152500 2024-02-21T23:13:36.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.041+0000 7f0992a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f098c0c86e0 con 0x7f098c0ca050 2024-02-21T23:13:36.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.041+0000 7f0992a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f098c0c8980 con 0x7f098c06bbf0 2024-02-21T23:13:36.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.041+0000 7f0991600640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f098c152500 0x7f098c0bf6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60968/0 (socket says 172.21.15.28:60968) 2024-02-21T23:13:36.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.041+0000 7f0991600640 1 -- 172.21.15.28:0/2882880978 learned_addr learned my addr 172.21.15.28:0/2882880978 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:13:36.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.042+0000 7f0992000640 1 --2- 172.21.15.28:0/2882880978 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f098c06bbf0 0x7f098c06c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:36.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.042+0000 7f0990c00640 1 --2- 172.21.15.28:0/2882880978 >> v2:172.21.15.81:3300/0 conn(0x7f098c0ca050 0x7f098c1558c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:36.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.042+0000 7f098be00640 1 -- 172.21.15.28:0/2882880978 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3131622326 0 0) 0x7f098c1528e0 con 0x7f098c152500 2024-02-21T23:13:36.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.042+0000 7f098be00640 1 -- 172.21.15.28:0/2882880978 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0974003520 con 0x7f098c152500 2024-02-21T23:13:36.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.042+0000 7f098be00640 1 -- 172.21.15.28:0/2882880978 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2977749226 0 0) 0x7f0974003520 con 0x7f098c152500 2024-02-21T23:13:36.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.042+0000 7f098be00640 1 -- 172.21.15.28:0/2882880978 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f098c06bbf0 msgr2=0x7f098c06c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:36.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.042+0000 7f098be00640 1 --2- 172.21.15.28:0/2882880978 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f098c06bbf0 0x7f098c06c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:36.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.042+0000 7f098be00640 1 -- 172.21.15.28:0/2882880978 >> v2:172.21.15.81:3300/0 conn(0x7f098c0ca050 msgr2=0x7f098c1558c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:36.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.042+0000 7f098be00640 1 --2- 172.21.15.28:0/2882880978 >> v2:172.21.15.81:3300/0 conn(0x7f098c0ca050 0x7f098c1558c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:36.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.042+0000 7f098be00640 1 -- 172.21.15.28:0/2882880978 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f098c0c8c60 con 0x7f098c152500 2024-02-21T23:13:36.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.042+0000 7f098be00640 1 -- 172.21.15.28:0/2882880978 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f0980001690 con 0x7f098c152500 2024-02-21T23:13:36.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.042+0000 7f098be00640 1 -- 172.21.15.28:0/2882880978 >> v1:172.21.15.28:6789/0 conn(0x7f098c152500 legacy=0x7f098c0bf6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:36.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.043+0000 7f098be00640 1 --2- 172.21.15.28:0/2882880978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09740039b0 0x7f0974003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:36.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.043+0000 7f098be00640 1 --2- 172.21.15.28:0/2882880978 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0974004800 0x7f0974004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:36.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.043+0000 7f098be00640 1 --2- 172.21.15.28:0/2882880978 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f098c06bbf0 0x7f09740051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:36.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.043+0000 7f098be00640 1 -- 172.21.15.28:0/2882880978 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f098c0c86e0 con 0x7f09740039b0 2024-02-21T23:13:36.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.043+0000 7f098be00640 1 -- 172.21.15.28:0/2882880978 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f098c0c8980 con 0x7f0974004800 2024-02-21T23:13:36.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.043+0000 7f098be00640 1 -- 172.21.15.28:0/2882880978 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f097400b9c0 con 0x7f098c06bbf0 2024-02-21T23:13:36.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.043+0000 7f0991600640 1 --2- 172.21.15.28:0/2882880978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09740039b0 0x7f0974003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:36.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.043+0000 7f0992000640 1 --2- 172.21.15.28:0/2882880978 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f098c06bbf0 0x7f09740051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:36.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.043+0000 7f0990c00640 1 --2- 172.21.15.28:0/2882880978 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0974004800 0x7f0974004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:36.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.043+0000 7f0991600640 1 -- 172.21.15.28:0/2882880978 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f098c06bbf0 msgr2=0x7f09740051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:36.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.043+0000 7f0991600640 1 --2- 172.21.15.28:0/2882880978 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f098c06bbf0 0x7f09740051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:36.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.043+0000 7f0991600640 1 -- 172.21.15.28:0/2882880978 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0974004800 msgr2=0x7f0974004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:36.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.043+0000 7f0991600640 1 --2- 172.21.15.28:0/2882880978 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0974004800 0x7f0974004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:36.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.043+0000 7f0991600640 1 -- 172.21.15.28:0/2882880978 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0974003520 con 0x7f09740039b0 2024-02-21T23:13:36.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.045+0000 7f0991600640 1 --2- 172.21.15.28:0/2882880978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09740039b0 0x7f0974003dd0 secure :-1 s=READY pgs=2384 cs=0 l=1 rev1=1 crypto rx=0x7f098000c240 tx=0x7f0974005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a3249a9ee3bf7b9 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:36.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.045+0000 7f098be00640 1 -- 172.21.15.28:0/2882880978 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0980007d70 con 0x7f09740039b0 2024-02-21T23:13:36.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.045+0000 7f098be00640 1 -- 172.21.15.28:0/2882880978 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0980030070 con 0x7f09740039b0 2024-02-21T23:13:36.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.045+0000 7f098be00640 1 -- 172.21.15.28:0/2882880978 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f098002cd40 con 0x7f09740039b0 2024-02-21T23:13:36.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.046+0000 7f0992a00640 1 -- 172.21.15.28:0/2882880978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09740039b0 msgr2=0x7f0974003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:36.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.046+0000 7f0992a00640 1 --2- 172.21.15.28:0/2882880978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09740039b0 0x7f0974003dd0 secure :-1 s=READY pgs=2384 cs=0 l=1 rev1=1 crypto rx=0x7f098000c240 tx=0x7f0974005830 comp rx=0 tx=0).stop 2024-02-21T23:13:36.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.046+0000 7f0992000640 1 -- 172.21.15.28:0/2882880978 reap_dead start 2024-02-21T23:13:36.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.046+0000 7f0992a00640 1 -- 172.21.15.28:0/2882880978 shutdown_connections 2024-02-21T23:13:36.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.046+0000 7f0992a00640 1 -- 172.21.15.28:0/2882880978 >> 172.21.15.28:0/2882880978 conn(0x7f098c0ac450 msgr2=0x7f098c0ac860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:36.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.046+0000 7f0992a00640 1 -- 172.21.15.28:0/2882880978 shutdown_connections 2024-02-21T23:13:36.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.046+0000 7f0992a00640 1 -- 172.21.15.28:0/2882880978 wait complete. 2024-02-21T23:13:36.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.049+0000 7f0992a00640 1 Processor -- start 2024-02-21T23:13:36.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.049+0000 7f0992a00640 1 -- start start 2024-02-21T23:13:36.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.050+0000 7f0992a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09740039b0 0x7f098c1eb1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:36.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.050+0000 7f0992a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f098c148850 0x7f098c0c3950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:36.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.050+0000 7f0992a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f098c0b0270 0x7f098c0c3e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:36.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.050+0000 7f0992a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f098c0bfdd0 con 0x7f09740039b0 2024-02-21T23:13:36.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.050+0000 7f0992a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f098c0c9220 con 0x7f098c0b0270 2024-02-21T23:13:36.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.050+0000 7f0992a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f098c0bff50 con 0x7f098c148850 2024-02-21T23:13:36.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.051+0000 7f0991600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09740039b0 0x7f098c1eb1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:36.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.051+0000 7f0991600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09740039b0 0x7f098c1eb1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39388/0 (socket says 172.21.15.28:39388) 2024-02-21T23:13:36.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.051+0000 7f0991600640 1 -- 172.21.15.28:0/2023756905 learned_addr learned my addr 172.21.15.28:0/2023756905 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:36.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.051+0000 7f0990c00640 1 --2- 172.21.15.28:0/2023756905 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f098c148850 0x7f098c0c3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:36.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.051+0000 7f0991600640 1 -- 172.21.15.28:0/2023756905 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f098c148850 msgr2=0x7f098c0c3950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:36.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.051+0000 7f0991600640 1 --2- 172.21.15.28:0/2023756905 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f098c148850 0x7f098c0c3950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:36.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.051+0000 7f0991600640 1 -- 172.21.15.28:0/2023756905 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f098c0b0270 msgr2=0x7f098c0c3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:36.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.051+0000 7f0992000640 1 --2- 172.21.15.28:0/2023756905 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f098c0b0270 0x7f098c0c3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:36.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.051+0000 7f0991600640 1 --2- 172.21.15.28:0/2023756905 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f098c0b0270 0x7f098c0c3e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:36.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.051+0000 7f0991600640 1 -- 172.21.15.28:0/2023756905 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f098c148480 con 0x7f09740039b0 2024-02-21T23:13:36.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.051+0000 7f0992000640 1 --2- 172.21.15.28:0/2023756905 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f098c0b0270 0x7f098c0c3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:13:36.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.052+0000 7f0991600640 1 --2- 172.21.15.28:0/2023756905 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09740039b0 0x7f098c1eb1e0 secure :-1 s=READY pgs=2385 cs=0 l=1 rev1=1 crypto rx=0x7f098002c570 tx=0x7f0980001a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:36.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.052+0000 7f0989600640 1 -- 172.21.15.28:0/2023756905 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0980001cb0 con 0x7f09740039b0 2024-02-21T23:13:36.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.052+0000 7f0989600640 1 -- 172.21.15.28:0/2023756905 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0980004bf0 con 0x7f09740039b0 2024-02-21T23:13:36.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.052+0000 7f0989600640 1 -- 172.21.15.28:0/2023756905 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0980007070 con 0x7f09740039b0 2024-02-21T23:13:36.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.052+0000 7f0992a00640 1 -- 172.21.15.28:0/2023756905 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f098c156400 con 0x7f09740039b0 2024-02-21T23:13:36.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.052+0000 7f0992a00640 1 -- 172.21.15.28:0/2023756905 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f098c0c4b00 con 0x7f09740039b0 2024-02-21T23:13:36.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.054+0000 7f0992a00640 1 -- 172.21.15.28:0/2023756905 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f098c151d60 con 0x7f09740039b0 2024-02-21T23:13:36.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.054+0000 7f0989600640 1 -- 172.21.15.28:0/2023756905 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f098003f070 con 0x7f09740039b0 2024-02-21T23:13:36.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.055+0000 7f0989600640 1 --2- 172.21.15.28:0/2023756905 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f096c06b500 0x7f096c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:36.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.056+0000 7f0989600640 1 -- 172.21.15.28:0/2023756905 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f09800ab6a0 con 0x7f09740039b0 2024-02-21T23:13:36.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.056+0000 7f0990c00640 1 --2- 172.21.15.28:0/2023756905 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f096c06b500 0x7f096c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:36.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.057+0000 7f0990c00640 1 --2- 172.21.15.28:0/2023756905 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f096c06b500 0x7f096c06d9c0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f098c148c70 tx=0x7f0978008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:36.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.060+0000 7f0989600640 1 -- 172.21.15.28:0/2023756905 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f098c151d60 con 0x7f09740039b0 2024-02-21T23:13:36.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.344+0000 7f0992a00640 1 -- 172.21.15.28:0/2023756905 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f098c1561f0 con 0x7f09740039b0 2024-02-21T23:13:36.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.345+0000 7f0989600640 1 -- 172.21.15.28:0/2023756905 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 229 v229) v1 ==== 95+0+4948 (secure 0 0 0) 0x7f098007a500 con 0x7f09740039b0 2024-02-21T23:13:36.334 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:13:36.334 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":229,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7265,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":224},{"gid":8287,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1607587459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1607587459},{"type":"v1","addr":"172.21.15.81:6835","nonce":1607587459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":224},{"gid":8290,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/1864206570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1864206570},{"type":"v1","addr":"172.21.15.28:6833","nonce":1864206570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":224}],"filesystems":[{"mdsmap":{"epoch":229,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:13:32.769089+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:13:36.334 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 229 2024-02-21T23:13:36.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.348+0000 7f0992a00640 1 -- 172.21.15.28:0/2023756905 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f096c06b500 msgr2=0x7f096c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:36.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.348+0000 7f0992a00640 1 --2- 172.21.15.28:0/2023756905 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f096c06b500 0x7f096c06d9c0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f098c148c70 tx=0x7f0978008040 comp rx=0 tx=0).stop 2024-02-21T23:13:36.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.348+0000 7f0992a00640 1 -- 172.21.15.28:0/2023756905 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09740039b0 msgr2=0x7f098c1eb1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:36.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.348+0000 7f0992a00640 1 --2- 172.21.15.28:0/2023756905 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09740039b0 0x7f098c1eb1e0 secure :-1 s=READY pgs=2385 cs=0 l=1 rev1=1 crypto rx=0x7f098002c570 tx=0x7f0980001a60 comp rx=0 tx=0).stop 2024-02-21T23:13:36.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.348+0000 7f0992a00640 1 -- 172.21.15.28:0/2023756905 shutdown_connections 2024-02-21T23:13:36.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.348+0000 7f0992a00640 1 --2- 172.21.15.28:0/2023756905 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f096c06b500 0x7f096c06d9c0 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:36.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.348+0000 7f0992a00640 1 --2- 172.21.15.28:0/2023756905 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f098c148850 0x7f098c0c3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:36.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.348+0000 7f0992a00640 1 --2- 172.21.15.28:0/2023756905 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f098c0b0270 0x7f098c0c3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:36.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.348+0000 7f0992a00640 1 --2- 172.21.15.28:0/2023756905 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09740039b0 0x7f098c1eb1e0 unknown :-1 s=CLOSED pgs=2385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:36.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.348+0000 7f0992a00640 1 -- 172.21.15.28:0/2023756905 >> 172.21.15.28:0/2023756905 conn(0x7f098c0ac450 msgr2=0x7f098c0bf6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:36.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.348+0000 7f0992a00640 1 -- 172.21.15.28:0/2023756905 shutdown_connections 2024-02-21T23:13:36.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.348+0000 7f0992a00640 1 -- 172.21.15.28:0/2023756905 wait complete. 2024-02-21T23:13:36.759 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 229, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:13:32.769089+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8296}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:13:36.760 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-02-21T23:13:36.760 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-02-21T23:13:36.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.997+0000 7f4f20a00640 1 Processor -- start 2024-02-21T23:13:36.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.997+0000 7f4f20a00640 1 -- start start 2024-02-21T23:13:36.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.999+0000 7f4f20a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4f1c0bf880 0x7f4f1c0c6a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:36.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.999+0000 7f4f20a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f4f1c0c5260 0x7f4f1c0c56f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:36.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.999+0000 7f4f20a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f1c0c6640 con 0x7f4f1c152740 2024-02-21T23:13:36.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.999+0000 7f4f20a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f1c155fc0 con 0x7f4f1c0c5260 2024-02-21T23:13:36.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:36.999+0000 7f4f20a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1c155200 con 0x7f4f1c0bf880 2024-02-21T23:13:36.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.000+0000 7f4f1a800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f4f1c152740 0x7f4f1c0c72f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60976/0 (socket says 172.21.15.28:60976) 2024-02-21T23:13:36.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.000+0000 7f4f1a800640 1 -- 172.21.15.28:0/4120989416 learned_addr learned my addr 172.21.15.28:0/4120989416 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:13:36.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.000+0000 7f4f13e00640 1 --2- 172.21.15.28:0/4120989416 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4f1c0bf880 0x7f4f1c0c6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:36.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.000+0000 7f4f1b200640 1 --2- 172.21.15.28:0/4120989416 >> v2:172.21.15.81:3300/0 conn(0x7f4f1c0c5260 0x7f4f1c0c56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:36.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.000+0000 7f4f19e00640 1 -- 172.21.15.28:0/4120989416 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1501211862 0 0) 0x7f4f1c0c6640 con 0x7f4f1c152740 2024-02-21T23:13:36.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.000+0000 7f4f19e00640 1 -- 172.21.15.28:0/4120989416 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f00003520 con 0x7f4f1c152740 2024-02-21T23:13:36.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.001+0000 7f4f19e00640 1 -- 172.21.15.28:0/4120989416 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 85621768 0 0) 0x7f4f00003520 con 0x7f4f1c152740 2024-02-21T23:13:36.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.001+0000 7f4f19e00640 1 -- 172.21.15.28:0/4120989416 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4f1c0bf880 msgr2=0x7f4f1c0c6a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:36.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.001+0000 7f4f19e00640 1 --2- 172.21.15.28:0/4120989416 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4f1c0bf880 0x7f4f1c0c6a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:36.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.001+0000 7f4f19e00640 1 -- 172.21.15.28:0/4120989416 >> v2:172.21.15.81:3300/0 conn(0x7f4f1c0c5260 msgr2=0x7f4f1c0c56f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:36.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.001+0000 7f4f19e00640 1 --2- 172.21.15.28:0/4120989416 >> v2:172.21.15.81:3300/0 conn(0x7f4f1c0c5260 0x7f4f1c0c56f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:36.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.001+0000 7f4f19e00640 1 -- 172.21.15.28:0/4120989416 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f1c1554e0 con 0x7f4f1c152740 2024-02-21T23:13:36.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.001+0000 7f4f19e00640 1 -- 172.21.15.28:0/4120989416 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f4f0c001670 con 0x7f4f1c152740 2024-02-21T23:13:36.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.001+0000 7f4f19e00640 1 -- 172.21.15.28:0/4120989416 >> v1:172.21.15.28:6789/0 conn(0x7f4f1c152740 legacy=0x7f4f1c0c72f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:36.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.001+0000 7f4f19e00640 1 --2- 172.21.15.28:0/4120989416 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f000039b0 0x7f4f00003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:36.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.001+0000 7f4f19e00640 1 --2- 172.21.15.28:0/4120989416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f00004800 0x7f4f00004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:36.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.001+0000 7f4f19e00640 1 --2- 172.21.15.28:0/4120989416 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4f1c0bf880 0x7f4f000051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:36.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.001+0000 7f4f19e00640 1 -- 172.21.15.28:0/4120989416 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1c155fc0 con 0x7f4f00004800 2024-02-21T23:13:36.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.001+0000 7f4f19e00640 1 -- 172.21.15.28:0/4120989416 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1c155200 con 0x7f4f000039b0 2024-02-21T23:13:36.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.001+0000 7f4f19e00640 1 -- 172.21.15.28:0/4120989416 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f0000b9c0 con 0x7f4f1c0bf880 2024-02-21T23:13:36.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.001+0000 7f4f13e00640 1 --2- 172.21.15.28:0/4120989416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f00004800 0x7f4f00004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:36.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.002+0000 7f4f1a800640 1 --2- 172.21.15.28:0/4120989416 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f000039b0 0x7f4f00003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:36.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.002+0000 7f4f1b200640 1 --2- 172.21.15.28:0/4120989416 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4f1c0bf880 0x7f4f000051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:36.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.002+0000 7f4f13e00640 1 -- 172.21.15.28:0/4120989416 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4f1c0bf880 msgr2=0x7f4f000051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:36.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.002+0000 7f4f13e00640 1 --2- 172.21.15.28:0/4120989416 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4f1c0bf880 0x7f4f000051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:36.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.002+0000 7f4f13e00640 1 -- 172.21.15.28:0/4120989416 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f000039b0 msgr2=0x7f4f00003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:36.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.002+0000 7f4f13e00640 1 --2- 172.21.15.28:0/4120989416 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f000039b0 0x7f4f00003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:36.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.002+0000 7f4f13e00640 1 -- 172.21.15.28:0/4120989416 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f00003520 con 0x7f4f00004800 2024-02-21T23:13:36.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.002+0000 7f4f1b200640 1 --2- 172.21.15.28:0/4120989416 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4f1c0bf880 0x7f4f000051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:13:36.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.003+0000 7f4f13e00640 1 --2- 172.21.15.28:0/4120989416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f00004800 0x7f4f00004c80 secure :-1 s=READY pgs=2387 cs=0 l=1 rev1=1 crypto rx=0x7f4f04009af0 tx=0x7f4f00005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eceeae04200e1e53 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:36.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.003+0000 7f4f19e00640 1 -- 172.21.15.28:0/4120989416 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f04039070 con 0x7f4f00004800 2024-02-21T23:13:36.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.003+0000 7f4f19e00640 1 -- 172.21.15.28:0/4120989416 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f0402c6b0 con 0x7f4f00004800 2024-02-21T23:13:36.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.003+0000 7f4f19e00640 1 -- 172.21.15.28:0/4120989416 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f04009690 con 0x7f4f00004800 2024-02-21T23:13:36.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.004+0000 7f4f20a00640 1 -- 172.21.15.28:0/4120989416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f00004800 msgr2=0x7f4f00004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:36.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.004+0000 7f4f20a00640 1 --2- 172.21.15.28:0/4120989416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f00004800 0x7f4f00004c80 secure :-1 s=READY pgs=2387 cs=0 l=1 rev1=1 crypto rx=0x7f4f04009af0 tx=0x7f4f00005830 comp rx=0 tx=0).stop 2024-02-21T23:13:36.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.004+0000 7f4f1b200640 1 -- 172.21.15.28:0/4120989416 reap_dead start 2024-02-21T23:13:36.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.004+0000 7f4f20a00640 1 -- 172.21.15.28:0/4120989416 shutdown_connections 2024-02-21T23:13:36.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.004+0000 7f4f20a00640 1 -- 172.21.15.28:0/4120989416 >> 172.21.15.28:0/4120989416 conn(0x7f4f1c067080 msgr2=0x7f4f1c0725a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:36.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.004+0000 7f4f20a00640 1 -- 172.21.15.28:0/4120989416 shutdown_connections 2024-02-21T23:13:36.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.004+0000 7f4f20a00640 1 -- 172.21.15.28:0/4120989416 wait complete. 2024-02-21T23:13:36.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.006+0000 7f4f20a00640 1 Processor -- start 2024-02-21T23:13:36.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.006+0000 7f4f20a00640 1 -- start start 2024-02-21T23:13:36.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.007+0000 7f4f20a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4f00004800 0x7f4f1c0c03d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:36.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.007+0000 7f4f20a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f1c156190 0x7f4f1c1eb2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:36.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.007+0000 7f4f20a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f1c1ec010 0x7f4f1c14aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:36.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.007+0000 7f4f20a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1c0c7a00 con 0x7f4f1c156190 2024-02-21T23:13:36.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.007+0000 7f4f20a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1c155aa0 con 0x7f4f1c1ec010 2024-02-21T23:13:36.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.007+0000 7f4f20a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1c0c7b80 con 0x7f4f00004800 2024-02-21T23:13:36.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.007+0000 7f4f13e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f1c156190 0x7f4f1c1eb2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:36.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.007+0000 7f4f13e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f1c156190 0x7f4f1c1eb2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39404/0 (socket says 172.21.15.28:39404) 2024-02-21T23:13:36.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.007+0000 7f4f13e00640 1 -- 172.21.15.28:0/2897635480 learned_addr learned my addr 172.21.15.28:0/2897635480 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:36.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.008+0000 7f4f1a800640 1 --2- 172.21.15.28:0/2897635480 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4f00004800 0x7f4f1c0c03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:36.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.008+0000 7f4f1b200640 1 --2- 172.21.15.28:0/2897635480 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f1c1ec010 0x7f4f1c14aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:36.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.008+0000 7f4f13e00640 1 -- 172.21.15.28:0/2897635480 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4f00004800 msgr2=0x7f4f1c0c03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:36.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.008+0000 7f4f13e00640 1 --2- 172.21.15.28:0/2897635480 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4f00004800 0x7f4f1c0c03d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:36.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.008+0000 7f4f13e00640 1 -- 172.21.15.28:0/2897635480 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f1c1ec010 msgr2=0x7f4f1c14aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:36.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.008+0000 7f4f13e00640 1 --2- 172.21.15.28:0/2897635480 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f1c1ec010 0x7f4f1c14aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:36.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.008+0000 7f4f13e00640 1 -- 172.21.15.28:0/2897635480 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f1c1eebc0 con 0x7f4f1c156190 2024-02-21T23:13:36.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.008+0000 7f4f13e00640 1 --2- 172.21.15.28:0/2897635480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f1c156190 0x7f4f1c1eb2d0 secure :-1 s=READY pgs=2388 cs=0 l=1 rev1=1 crypto rx=0x7f4f04037040 tx=0x7f4f04004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:36.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.009+0000 7f4f12a00640 1 -- 172.21.15.28:0/2897635480 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f04042020 con 0x7f4f1c156190 2024-02-21T23:13:36.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.009+0000 7f4f12a00640 1 -- 172.21.15.28:0/2897635480 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f0403a040 con 0x7f4f1c156190 2024-02-21T23:13:36.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.009+0000 7f4f12a00640 1 -- 172.21.15.28:0/2897635480 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f04039070 con 0x7f4f1c156190 2024-02-21T23:13:36.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.009+0000 7f4f20a00640 1 -- 172.21.15.28:0/2897635480 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f1c1eeea0 con 0x7f4f1c156190 2024-02-21T23:13:36.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.009+0000 7f4f20a00640 1 -- 172.21.15.28:0/2897635480 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f1c1dea60 con 0x7f4f1c156190 2024-02-21T23:13:37.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.011+0000 7f4f12a00640 1 -- 172.21.15.28:0/2897635480 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4f04045020 con 0x7f4f1c156190 2024-02-21T23:13:37.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.012+0000 7f4f12a00640 1 --2- 172.21.15.28:0/2897635480 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4ef806b500 0x7f4ef806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:37.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.012+0000 7f4f12a00640 1 -- 172.21.15.28:0/2897635480 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4f040ab7e0 con 0x7f4f1c156190 2024-02-21T23:13:37.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.013+0000 7f4f243ca740 1 Processor -- start 2024-02-21T23:13:37.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.013+0000 7f4f1a800640 1 --2- 172.21.15.28:0/2897635480 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4ef806b500 0x7f4ef806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:37.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.013+0000 7f4f243ca740 1 -- start start 2024-02-21T23:13:37.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.014+0000 7f4f243ca740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55805d1e2020 0x55805d1e2400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:37.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.014+0000 7f4f243ca740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55805d1e3330 0x55805d21e170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:37.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.014+0000 7f4f243ca740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55805d21e6b0 0x55805d21c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:37.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.014+0000 7f4f243ca740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55805d146470 con 0x55805d21e6b0 2024-02-21T23:13:37.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.014+0000 7f4f243ca740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55805d0efaf0 con 0x55805d1e2020 2024-02-21T23:13:37.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.014+0000 7f4f243ca740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x55805d142510 con 0x55805d1e3330 2024-02-21T23:13:37.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.014+0000 7f4f1b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55805d21e6b0 0x55805d21c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:37.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.014+0000 7f4f1b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55805d21e6b0 0x55805d21c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39410/0 (socket says 172.21.15.28:39410) 2024-02-21T23:13:37.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.014+0000 7f4f1b200640 1 -- 172.21.15.28:0/2241956030 learned_addr learned my addr 172.21.15.28:0/2241956030 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:37.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.014+0000 7f4f1a800640 1 --2- 172.21.15.28:0/2897635480 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4ef806b500 0x7f4ef806d9c0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f4f0c0059b0 tx=0x7f4f0c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:37.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.014+0000 7f4f13e00640 1 --2- 172.21.15.28:0/2241956030 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55805d1e3330 0x55805d21e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:37.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.014+0000 7f4f1a800640 1 --2- 172.21.15.28:0/2241956030 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55805d1e2020 0x55805d1e2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:37.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.015+0000 7f4f1b200640 1 -- 172.21.15.28:0/2241956030 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55805d1e3330 msgr2=0x55805d21e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:37.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.015+0000 7f4f1b200640 1 --2- 172.21.15.28:0/2241956030 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55805d1e3330 0x55805d21e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:37.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.015+0000 7f4f1b200640 1 -- 172.21.15.28:0/2241956030 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55805d1e2020 msgr2=0x55805d1e2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:37.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.015+0000 7f4f1b200640 1 --2- 172.21.15.28:0/2241956030 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55805d1e2020 0x55805d1e2400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:37.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.015+0000 7f4f1b200640 1 -- 172.21.15.28:0/2241956030 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55805d0f6d90 con 0x55805d21e6b0 2024-02-21T23:13:37.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.015+0000 7f4f1b200640 1 --2- 172.21.15.28:0/2241956030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55805d21e6b0 0x55805d21c720 secure :-1 s=READY pgs=2389 cs=0 l=1 rev1=1 crypto rx=0x7f4f14003ad0 tx=0x7f4f14009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a4166d40e8691dd server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:37.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.015+0000 7f4f0b000640 1 -- 172.21.15.28:0/2241956030 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f14019020 con 0x55805d21e6b0 2024-02-21T23:13:37.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.015+0000 7f4f0b000640 1 -- 172.21.15.28:0/2241956030 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f140153f0 con 0x55805d21e6b0 2024-02-21T23:13:37.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.015+0000 7f4f0b000640 1 -- 172.21.15.28:0/2241956030 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f140157c0 con 0x55805d21e6b0 2024-02-21T23:13:37.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.016+0000 7f4f243ca740 1 -- 172.21.15.28:0/2241956030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55805d21e6b0 msgr2=0x55805d21c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:37.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.016+0000 7f4f243ca740 1 --2- 172.21.15.28:0/2241956030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55805d21e6b0 0x55805d21c720 secure :-1 s=READY pgs=2389 cs=0 l=1 rev1=1 crypto rx=0x7f4f14003ad0 tx=0x7f4f14009750 comp rx=0 tx=0).stop 2024-02-21T23:13:37.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.016+0000 7f4f243ca740 1 -- 172.21.15.28:0/2241956030 shutdown_connections 2024-02-21T23:13:37.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.016+0000 7f4f243ca740 1 --2- 172.21.15.28:0/2241956030 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55805d1e3330 0x55805d21e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:37.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.016+0000 7f4f243ca740 1 --2- 172.21.15.28:0/2241956030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55805d21e6b0 0x55805d21c720 unknown :-1 s=CLOSED pgs=2389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:37.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.016+0000 7f4f243ca740 1 --2- 172.21.15.28:0/2241956030 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55805d1e2020 0x55805d1e2400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:37.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.016+0000 7f4f243ca740 1 -- 172.21.15.28:0/2241956030 >> 172.21.15.28:0/2241956030 conn(0x55805d1de970 msgr2=0x55805d203380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:37.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.016+0000 7f4f243ca740 1 -- 172.21.15.28:0/2241956030 shutdown_connections 2024-02-21T23:13:37.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.016+0000 7f4f243ca740 1 -- 172.21.15.28:0/2241956030 wait complete. 2024-02-21T23:13:37.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.017+0000 7f4f243ca740 1 -- start start 2024-02-21T23:13:37.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.017+0000 7f4f243ca740 1 Processor -- start 2024-02-21T23:13:37.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.018+0000 7f4f243ca740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55805d1e3330 0x55805d231930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:37.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.018+0000 7f4f243ca740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55805d21e6b0 0x55805d201eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:37.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.018+0000 7f4f243ca740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55805d1fe6f0 0x55805d1fd160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:37.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.018+0000 7f4f243ca740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55805d0e5040 con 0x55805d21e6b0 2024-02-21T23:13:37.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.018+0000 7f4f243ca740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55805d203a90 con 0x55805d1e3330 2024-02-21T23:13:37.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.018+0000 7f4f243ca740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x55805d202bd0 con 0x55805d1fe6f0 2024-02-21T23:13:37.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.018+0000 7f4f243ca740 10 client.0 _ll_register_callbacks cb 0x55805d102920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:13:37.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.019+0000 7f4f13e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55805d21e6b0 0x55805d201eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:37.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.019+0000 7f4f13e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55805d21e6b0 0x55805d201eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39426/0 (socket says 172.21.15.28:39426) 2024-02-21T23:13:37.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.019+0000 7f4f13e00640 1 -- 172.21.15.28:0/1802156163 learned_addr learned my addr 172.21.15.28:0/1802156163 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:37.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.019+0000 7f4f1a800640 1 --2- 172.21.15.28:0/1802156163 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55805d1e3330 0x55805d231930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:37.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.019+0000 7f4f1b200640 1 --2- 172.21.15.28:0/1802156163 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55805d1fe6f0 0x55805d1fd160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:37.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.019+0000 7f4f13e00640 1 -- 172.21.15.28:0/1802156163 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55805d1fe6f0 msgr2=0x55805d1fd160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:37.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.019+0000 7f4f13e00640 1 --2- 172.21.15.28:0/1802156163 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55805d1fe6f0 0x55805d1fd160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:37.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.019+0000 7f4f13e00640 1 -- 172.21.15.28:0/1802156163 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55805d1e3330 msgr2=0x55805d231930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:37.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.019+0000 7f4f13e00640 1 --2- 172.21.15.28:0/1802156163 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55805d1e3330 0x55805d231930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:37.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.019+0000 7f4f13e00640 1 -- 172.21.15.28:0/1802156163 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55805d104aa0 con 0x55805d21e6b0 2024-02-21T23:13:37.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.020+0000 7f4f13e00640 1 --2- 172.21.15.28:0/1802156163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55805d21e6b0 0x55805d201eb0 secure :-1 s=READY pgs=2390 cs=0 l=1 rev1=1 crypto rx=0x7f4f04081f10 tx=0x7f4f040044e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63b2bbe947d689c server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:37.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.020+0000 7f4eff400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:13:37.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.020+0000 7f4eff400640 1 -- 172.21.15.28:0/1802156163 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f0407de80 con 0x55805d21e6b0 2024-02-21T23:13:37.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.020+0000 7f4eff400640 1 -- 172.21.15.28:0/1802156163 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(208..208 src has 1..208) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4f04081fb0 con 0x55805d21e6b0 2024-02-21T23:13:37.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.020+0000 7f4f243ca740 1 -- 172.21.15.28:0/1802156163 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4f140153f0 con 0x55805d21e6b0 2024-02-21T23:13:37.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.020+0000 7f4eff400640 1 -- 172.21.15.28:0/1802156163 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=229) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4f04082440 con 0x55805d21e6b0 2024-02-21T23:13:37.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.020+0000 7f4f243ca740 10 client.10914 fetch_fsmap learned FSMap version 229 2024-02-21T23:13:37.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.020+0000 7f4f243ca740 1 -- 172.21.15.28:0/1802156163 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=229,monmap=2+}) v3 -- 0x55805ce5b2d0 con 0x55805d21e6b0 2024-02-21T23:13:37.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.020+0000 7f4eff400640 1 -- 172.21.15.28:0/1802156163 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f140153f0 con 0x55805d21e6b0 2024-02-21T23:13:37.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.021+0000 7f4eff400640 1 -- 172.21.15.28:0/1802156163 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 229) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f4f04079180 con 0x55805d21e6b0 2024-02-21T23:13:37.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.021+0000 7f4f243ca740 10 client.10914 fetch_fsmap finished waiting for FSMap version 229 2024-02-21T23:13:37.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.021+0000 7f4f243ca740 10 client.10914 resolve_mds: resolved name 'c' to daemon mds.c 2024-02-21T23:13:37.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.021+0000 7f4f243ca740 20 client.10914 populate_metadata read hostname 'smithi028' 2024-02-21T23:13:37.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.021+0000 7f4f243ca740 1 --2- 172.21.15.28:0/1802156163 >> [v2:172.21.15.28:6834/3142544451,v1:172.21.15.28:6835/3142544451] conn(0x55805d22e0e0 0x55805d22e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:37.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.021+0000 7f4f243ca740 4 client.10914 mds_command: new command op to 8296 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-21T23:13:37.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.021+0000 7f4f243ca740 1 -- 172.21.15.28:0/1802156163 --> [v2:172.21.15.28:6834/3142544451,v1:172.21.15.28:6835/3142544451] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55805d1539b0 con 0x55805d22e0e0 2024-02-21T23:13:37.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.021+0000 7f4f1a800640 1 --2- 172.21.15.28:0/1802156163 >> [v2:172.21.15.28:6834/3142544451,v1:172.21.15.28:6835/3142544451] conn(0x55805d22e0e0 0x55805d22e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:37.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.022+0000 7f4f1a800640 1 --2- 172.21.15.28:0/1802156163 >> [v2:172.21.15.28:6834/3142544451,v1:172.21.15.28:6835/3142544451] conn(0x55805d22e0e0 0x55805d22e4c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9578c161ffdf7131 server_cookie=e4f442c3acc913ff in_seq=0 out_seq=0 2024-02-21T23:13:37.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.022+0000 7f4eff400640 10 client.10914 ms_handle_connect on v2:172.21.15.28:6834/3142544451 2024-02-21T23:13:37.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.022+0000 7f4eff400640 1 -- 172.21.15.28:0/1802156163 <== mds.0 v2:172.21.15.28:6834/3142544451 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55805d1539b0 con 0x55805d22e0e0 2024-02-21T23:13:37.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.022+0000 7f4eff400640 10 client.10914 handle_command_reply: tid=0 2024-02-21T23:13:37.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.022+0000 7f4f243ca740 1 client.10914 shutdown 2024-02-21T23:13:37.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.023+0000 7f4f243ca740 1 -- 172.21.15.28:0/1802156163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55805d21e6b0 msgr2=0x55805d201eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:37.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.023+0000 7f4f243ca740 1 --2- 172.21.15.28:0/1802156163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55805d21e6b0 0x55805d201eb0 secure :-1 s=READY pgs=2390 cs=0 l=1 rev1=1 crypto rx=0x7f4f04081f10 tx=0x7f4f040044e0 comp rx=0 tx=0).stop 2024-02-21T23:13:37.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.023+0000 7f4f243ca740 1 -- 172.21.15.28:0/1802156163 shutdown_connections 2024-02-21T23:13:37.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.023+0000 7f4f243ca740 1 --2- 172.21.15.28:0/1802156163 >> [v2:172.21.15.28:6834/3142544451,v1:172.21.15.28:6835/3142544451] conn(0x55805d22e0e0 0x55805d22e4c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:37.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.023+0000 7f4f243ca740 1 --2- 172.21.15.28:0/1802156163 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55805d1fe6f0 0x55805d1fd160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:37.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.023+0000 7f4f243ca740 1 --2- 172.21.15.28:0/1802156163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55805d21e6b0 0x55805d201eb0 unknown :-1 s=CLOSED pgs=2390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:37.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.023+0000 7f4f243ca740 1 --2- 172.21.15.28:0/1802156163 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55805d1e3330 0x55805d231930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:37.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.023+0000 7f4eff400640 0 client.10914 ms_handle_reset on v2:172.21.15.28:6834/3142544451 2024-02-21T23:13:37.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.023+0000 7f4f243ca740 1 -- 172.21.15.28:0/1802156163 >> 172.21.15.28:0/1802156163 conn(0x55805d1e2020 msgr2=0x55805d21cf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:37.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.023+0000 7f4f243ca740 1 -- 172.21.15.28:0/1802156163 shutdown_connections 2024-02-21T23:13:37.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.023+0000 7f4f243ca740 1 -- 172.21.15.28:0/1802156163 wait complete. 2024-02-21T23:13:37.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.023+0000 7f4f243ca740 20 client.10914 trim_cache size 0 max 16384 2024-02-21T23:13:37.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.042+0000 7f4f243ca740 1 Processor -- start 2024-02-21T23:13:37.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.042+0000 7f4f243ca740 1 -- start start 2024-02-21T23:13:37.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.042+0000 7f4f243ca740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55805d1e3330 0x55805d21dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:37.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.042+0000 7f4f243ca740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55805d1fe6f0 0x55805d21fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:37.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.043+0000 7f4f243ca740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55805d21e6b0 0x55805d2215d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:37.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.043+0000 7f4f243ca740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1c1f7370 con 0x55805d1fe6f0 2024-02-21T23:13:37.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.043+0000 7f4f243ca740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1c1f6770 con 0x55805d1e3330 2024-02-21T23:13:37.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.043+0000 7f4f243ca740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1c1f6d70 con 0x55805d21e6b0 2024-02-21T23:13:37.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.043+0000 7f4f13e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55805d1fe6f0 0x55805d21fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:37.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.043+0000 7f4f13e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55805d1fe6f0 0x55805d21fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39428/0 (socket says 172.21.15.28:39428) 2024-02-21T23:13:37.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.043+0000 7f4f13e00640 1 -- 172.21.15.28:0/1607441379 learned_addr learned my addr 172.21.15.28:0/1607441379 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:37.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.043+0000 7f4f13e00640 1 -- 172.21.15.28:0/1607441379 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55805d21e6b0 msgr2=0x55805d2215d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:37.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.043+0000 7f4f13e00640 1 --2- 172.21.15.28:0/1607441379 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55805d21e6b0 0x55805d2215d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:37.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.043+0000 7f4f13e00640 1 -- 172.21.15.28:0/1607441379 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55805d1e3330 msgr2=0x55805d21dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:37.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.043+0000 7f4f13e00640 1 --2- 172.21.15.28:0/1607441379 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55805d1e3330 0x55805d21dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:37.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.043+0000 7f4f13e00640 1 -- 172.21.15.28:0/1607441379 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55805d1e2770 con 0x55805d1fe6f0 2024-02-21T23:13:37.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.043+0000 7f4f13e00640 1 --2- 172.21.15.28:0/1607441379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55805d1fe6f0 0x55805d21fbe0 secure :-1 s=READY pgs=2391 cs=0 l=1 rev1=1 crypto rx=0x7f4f04079cc0 tx=0x7f4f04040000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d521a331d25641f server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:37.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.044+0000 7f4f0a600640 1 -- 172.21.15.28:0/1607441379 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f0407de80 con 0x55805d1fe6f0 2024-02-21T23:13:37.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.044+0000 7f4f0a600640 1 -- 172.21.15.28:0/1607441379 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f04079180 con 0x55805d1fe6f0 2024-02-21T23:13:37.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.044+0000 7f4f0a600640 1 -- 172.21.15.28:0/1607441379 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f04075a80 con 0x55805d1fe6f0 2024-02-21T23:13:37.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.044+0000 7f4f243ca740 1 -- 172.21.15.28:0/1607441379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55805d1fe6f0 msgr2=0x55805d21fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:37.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.044+0000 7f4f243ca740 1 --2- 172.21.15.28:0/1607441379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55805d1fe6f0 0x55805d21fbe0 secure :-1 s=READY pgs=2391 cs=0 l=1 rev1=1 crypto rx=0x7f4f04079cc0 tx=0x7f4f04040000 comp rx=0 tx=0).stop 2024-02-21T23:13:37.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.044+0000 7f4f243ca740 1 -- 172.21.15.28:0/1607441379 shutdown_connections 2024-02-21T23:13:37.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.044+0000 7f4f243ca740 1 --2- 172.21.15.28:0/1607441379 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55805d21e6b0 0x55805d2215d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:37.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.044+0000 7f4f243ca740 1 --2- 172.21.15.28:0/1607441379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55805d1fe6f0 0x55805d21fbe0 unknown :-1 s=CLOSED pgs=2391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:37.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.044+0000 7f4f243ca740 1 --2- 172.21.15.28:0/1607441379 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55805d1e3330 0x55805d21dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:37.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.044+0000 7f4f243ca740 1 -- 172.21.15.28:0/1607441379 >> 172.21.15.28:0/1607441379 conn(0x55805d1e2020 msgr2=0x55805d1fcfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:37.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.044+0000 7f4f243ca740 1 -- 172.21.15.28:0/1607441379 shutdown_connections 2024-02-21T23:13:37.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.045+0000 7f4f243ca740 1 -- 172.21.15.28:0/1607441379 wait complete. 2024-02-21T23:13:37.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.045+0000 7f4f243ca740 1 -- start start 2024-02-21T23:13:37.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.045+0000 7f4f243ca740 1 Processor -- start 2024-02-21T23:13:37.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.046+0000 7f4f243ca740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55805d1e3330 0x55805d241210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:37.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.046+0000 7f4f243ca740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55805d1fe6f0 0x55805d234360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:37.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.046+0000 7f4f243ca740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55805d21e6b0 0x55805d28acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:37.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.046+0000 7f4f243ca740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55805d202910 con 0x55805d21e6b0 2024-02-21T23:13:37.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.046+0000 7f4f243ca740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55805d0e5040 con 0x55805d1fe6f0 2024-02-21T23:13:37.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.046+0000 7f4f243ca740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x55805d1fd6f0 con 0x55805d1e3330 2024-02-21T23:13:37.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.046+0000 7f4f243ca740 10 client.0 _ll_register_callbacks cb 0x7f4f04079870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:13:37.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.047+0000 7f4f1b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55805d21e6b0 0x55805d28acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:37.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.047+0000 7f4f1b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55805d21e6b0 0x55805d28acd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39436/0 (socket says 172.21.15.28:39436) 2024-02-21T23:13:37.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.047+0000 7f4f1b200640 1 -- 172.21.15.28:0/2422364769 learned_addr learned my addr 172.21.15.28:0/2422364769 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:37.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.047+0000 7f4f13e00640 1 --2- 172.21.15.28:0/2422364769 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55805d1fe6f0 0x55805d234360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:37.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.047+0000 7f4f1a800640 1 --2- 172.21.15.28:0/2422364769 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55805d1e3330 0x55805d241210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:37.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.047+0000 7f4f1b200640 1 -- 172.21.15.28:0/2422364769 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55805d1e3330 msgr2=0x55805d241210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:37.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.047+0000 7f4f1b200640 1 --2- 172.21.15.28:0/2422364769 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55805d1e3330 0x55805d241210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:37.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.047+0000 7f4f1b200640 1 -- 172.21.15.28:0/2422364769 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55805d1fe6f0 msgr2=0x55805d234360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:37.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.047+0000 7f4f1b200640 1 --2- 172.21.15.28:0/2422364769 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55805d1fe6f0 0x55805d234360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:37.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.047+0000 7f4f1b200640 1 -- 172.21.15.28:0/2422364769 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55805cf256b0 con 0x55805d21e6b0 2024-02-21T23:13:37.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.048+0000 7f4f1b200640 1 --2- 172.21.15.28:0/2422364769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55805d21e6b0 0x55805d28acd0 secure :-1 s=READY pgs=2392 cs=0 l=1 rev1=1 crypto rx=0x7f4f1400b010 tx=0x7f4f14004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21640172d3a6a825 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:37.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.048+0000 7f4eff400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:13:37.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.048+0000 7f4eff400640 1 -- 172.21.15.28:0/2422364769 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f14019040 con 0x55805d21e6b0 2024-02-21T23:13:37.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.048+0000 7f4f243ca740 1 -- 172.21.15.28:0/2422364769 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4f04079180 con 0x55805d21e6b0 2024-02-21T23:13:37.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.048+0000 7f4eff400640 1 -- 172.21.15.28:0/2422364769 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(208..208 src has 1..208) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4f14015b30 con 0x55805d21e6b0 2024-02-21T23:13:37.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.048+0000 7f4eff400640 1 -- 172.21.15.28:0/2422364769 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=229) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4f14031050 con 0x55805d21e6b0 2024-02-21T23:13:37.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.048+0000 7f4f243ca740 10 client.10920 fetch_fsmap learned FSMap version 229 2024-02-21T23:13:37.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.048+0000 7f4f243ca740 1 -- 172.21.15.28:0/2422364769 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=229,monmap=2+}) v3 -- 0x55805d241b50 con 0x55805d21e6b0 2024-02-21T23:13:37.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.049+0000 7f4eff400640 1 -- 172.21.15.28:0/2422364769 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f04079180 con 0x55805d21e6b0 2024-02-21T23:13:37.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.049+0000 7f4eff400640 1 -- 172.21.15.28:0/2422364769 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 229) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f4f14034020 con 0x55805d21e6b0 2024-02-21T23:13:37.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.049+0000 7f4f243ca740 10 client.10920 fetch_fsmap finished waiting for FSMap version 229 2024-02-21T23:13:37.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.049+0000 7f4f243ca740 10 client.10920 resolve_mds: resolved name 'c' to daemon mds.c 2024-02-21T23:13:37.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.049+0000 7f4f243ca740 20 client.10920 populate_metadata read hostname 'smithi028' 2024-02-21T23:13:37.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.049+0000 7f4f243ca740 1 --2- 172.21.15.28:0/2422364769 >> [v2:172.21.15.28:6834/3142544451,v1:172.21.15.28:6835/3142544451] conn(0x55805d1f9b10 0x55805d1f9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:37.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.049+0000 7f4f243ca740 4 client.10920 mds_command: new command op to 8296 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-21T23:13:37.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.049+0000 7f4f243ca740 1 -- 172.21.15.28:0/2422364769 --> [v2:172.21.15.28:6834/3142544451,v1:172.21.15.28:6835/3142544451] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55805d14dd00 con 0x55805d1f9b10 2024-02-21T23:13:37.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.049+0000 7f4f1a800640 1 --2- 172.21.15.28:0/2422364769 >> [v2:172.21.15.28:6834/3142544451,v1:172.21.15.28:6835/3142544451] conn(0x55805d1f9b10 0x55805d1f9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:37.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.050+0000 7f4f1a800640 1 --2- 172.21.15.28:0/2422364769 >> [v2:172.21.15.28:6834/3142544451,v1:172.21.15.28:6835/3142544451] conn(0x55805d1f9b10 0x55805d1f9ef0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d5cbae8c07358442 server_cookie=ba9a64d7723e3964 in_seq=0 out_seq=0 2024-02-21T23:13:37.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.050+0000 7f4eff400640 10 client.10920 ms_handle_connect on v2:172.21.15.28:6834/3142544451 2024-02-21T23:13:37.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.050+0000 7f4eff400640 1 -- 172.21.15.28:0/2422364769 <== mds.0 v2:172.21.15.28:6834/3142544451 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+328 (crc 0 0 0) 0x55805d14dd00 con 0x55805d1f9b10 2024-02-21T23:13:37.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.050+0000 7f4eff400640 10 client.10920 handle_command_reply: tid=0 2024-02-21T23:13:37.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.050+0000 7f4f243ca740 1 client.10920 shutdown 2024-02-21T23:13:37.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.050+0000 7f4f243ca740 1 -- 172.21.15.28:0/2422364769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55805d21e6b0 msgr2=0x55805d28acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:37.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.050+0000 7f4f243ca740 1 --2- 172.21.15.28:0/2422364769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55805d21e6b0 0x55805d28acd0 secure :-1 s=READY pgs=2392 cs=0 l=1 rev1=1 crypto rx=0x7f4f1400b010 tx=0x7f4f14004ab0 comp rx=0 tx=0).stop 2024-02-21T23:13:37.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.050+0000 7f4f243ca740 1 -- 172.21.15.28:0/2422364769 shutdown_connections 2024-02-21T23:13:37.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.050+0000 7f4f243ca740 1 --2- 172.21.15.28:0/2422364769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55805d21e6b0 0x55805d28acd0 unknown :-1 s=CLOSED pgs=2392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:37.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.050+0000 7f4f243ca740 1 --2- 172.21.15.28:0/2422364769 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55805d1fe6f0 0x55805d234360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:37.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.050+0000 7f4f243ca740 1 --2- 172.21.15.28:0/2422364769 >> [v2:172.21.15.28:6834/3142544451,v1:172.21.15.28:6835/3142544451] conn(0x55805d1f9b10 0x55805d1f9ef0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:37.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.050+0000 7f4f243ca740 1 --2- 172.21.15.28:0/2422364769 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55805d1e3330 0x55805d241210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:37.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.050+0000 7f4f243ca740 1 -- 172.21.15.28:0/2422364769 >> 172.21.15.28:0/2422364769 conn(0x55805d1e2020 msgr2=0x55805d21dd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:37.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.050+0000 7f4eff400640 0 client.10920 ms_handle_reset on v2:172.21.15.28:6834/3142544451 2024-02-21T23:13:37.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.051+0000 7f4f243ca740 1 -- 172.21.15.28:0/2422364769 shutdown_connections 2024-02-21T23:13:37.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.051+0000 7f4f243ca740 1 -- 172.21.15.28:0/2422364769 wait complete. 2024-02-21T23:13:37.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.051+0000 7f4f243ca740 20 client.10920 trim_cache size 0 max 16384 2024-02-21T23:13:37.045 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-21T23:13:37.045 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac", 2024-02-21T23:13:37.045 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-02-21T23:13:37.045 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 8296, 2024-02-21T23:13:37.045 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-02-21T23:13:37.045 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-02-21T23:13:37.045 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-02-21T23:13:37.045 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 6.7946295460000004, 2024-02-21T23:13:37.045 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 229, 2024-02-21T23:13:37.045 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 208, 2024-02-21T23:13:37.046 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 208, 2024-02-21T23:13:37.046 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 13.283760185 2024-02-21T23:13:37.046 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-21T23:13:37.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.051+0000 7f4f20a00640 1 -- 172.21.15.28:0/2897635480 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4ef806b500 msgr2=0x7f4ef806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:37.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.051+0000 7f4f20a00640 1 --2- 172.21.15.28:0/2897635480 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4ef806b500 0x7f4ef806d9c0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f4f0c0059b0 tx=0x7f4f0c009040 comp rx=0 tx=0).stop 2024-02-21T23:13:37.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.051+0000 7f4f20a00640 1 -- 172.21.15.28:0/2897635480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f1c156190 msgr2=0x7f4f1c1eb2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:37.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.051+0000 7f4f20a00640 1 --2- 172.21.15.28:0/2897635480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f1c156190 0x7f4f1c1eb2d0 secure :-1 s=READY pgs=2388 cs=0 l=1 rev1=1 crypto rx=0x7f4f04037040 tx=0x7f4f04004730 comp rx=0 tx=0).stop 2024-02-21T23:13:37.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.052+0000 7f4f20a00640 1 -- 172.21.15.28:0/2897635480 shutdown_connections 2024-02-21T23:13:37.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.052+0000 7f4f20a00640 1 --2- 172.21.15.28:0/2897635480 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4ef806b500 0x7f4ef806d9c0 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:37.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.052+0000 7f4f20a00640 1 --2- 172.21.15.28:0/2897635480 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4f1c1ec010 0x7f4f1c14aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:37.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.052+0000 7f4f20a00640 1 --2- 172.21.15.28:0/2897635480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f1c156190 0x7f4f1c1eb2d0 unknown :-1 s=CLOSED pgs=2388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:37.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.052+0000 7f4f20a00640 1 --2- 172.21.15.28:0/2897635480 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4f00004800 0x7f4f1c0c03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:37.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.052+0000 7f4f20a00640 1 -- 172.21.15.28:0/2897635480 >> 172.21.15.28:0/2897635480 conn(0x7f4f1c067080 msgr2=0x7f4f1c074360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:37.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.052+0000 7f4f20a00640 1 -- 172.21.15.28:0/2897635480 shutdown_connections 2024-02-21T23:13:37.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.052+0000 7f4f20a00640 1 -- 172.21.15.28:0/2897635480 wait complete. 2024-02-21T23:13:37.058 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:13:37.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.269+0000 7f7325000640 1 Processor -- start 2024-02-21T23:13:37.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.269+0000 7f7325000640 1 -- start start 2024-02-21T23:13:37.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.271+0000 7f7325000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f73200aff90 0x7f73200c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:37.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.271+0000 7f7325000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f732006dbf0 0x7f7320158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:37.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.271+0000 7f7325000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f73200c5720 0x7f7320159000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:37.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.271+0000 7f7325000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73200c7270 con 0x7f732006dbf0 2024-02-21T23:13:37.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.271+0000 7f7325000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7320158630 con 0x7f73200aff90 2024-02-21T23:13:37.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.271+0000 7f7325000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73200c74c0 con 0x7f73200c5720 2024-02-21T23:13:37.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.272+0000 7f731ea00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f732006dbf0 0x7f7320158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:37.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.272+0000 7f731ea00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f732006dbf0 0x7f7320158ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39444/0 (socket says 172.21.15.28:39444) 2024-02-21T23:13:37.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.272+0000 7f731ea00640 1 -- 172.21.15.28:0/2995070990 learned_addr learned my addr 172.21.15.28:0/2995070990 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:37.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.272+0000 7f731f400640 1 --2- 172.21.15.28:0/2995070990 >> v2:172.21.15.81:3300/0 conn(0x7f73200aff90 0x7f73200c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:37.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.272+0000 7f731fe00640 1 --2- 172.21.15.28:0/2995070990 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f73200c5720 0x7f7320159000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:37.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.272+0000 7f731ea00640 1 -- 172.21.15.28:0/2995070990 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f73200c5720 msgr2=0x7f7320159000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:37.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.272+0000 7f731ea00640 1 --2- 172.21.15.28:0/2995070990 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f73200c5720 0x7f7320159000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:37.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.272+0000 7f731ea00640 1 -- 172.21.15.28:0/2995070990 >> v2:172.21.15.81:3300/0 conn(0x7f73200aff90 msgr2=0x7f73200c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:37.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.272+0000 7f731ea00640 1 --2- 172.21.15.28:0/2995070990 >> v2:172.21.15.81:3300/0 conn(0x7f73200aff90 0x7f73200c51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:37.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.272+0000 7f731ea00640 1 -- 172.21.15.28:0/2995070990 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73200c77a0 con 0x7f732006dbf0 2024-02-21T23:13:37.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.272+0000 7f731f400640 1 --2- 172.21.15.28:0/2995070990 >> v2:172.21.15.81:3300/0 conn(0x7f73200aff90 0x7f73200c51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:13:37.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.274+0000 7f731ea00640 1 --2- 172.21.15.28:0/2995070990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f732006dbf0 0x7f7320158ac0 secure :-1 s=READY pgs=2393 cs=0 l=1 rev1=1 crypto rx=0x7f73200aa1b0 tx=0x7f730c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c018e6c04551fa8 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:37.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.274+0000 7f731e000640 1 -- 172.21.15.28:0/2995070990 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f730c035020 con 0x7f732006dbf0 2024-02-21T23:13:37.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.274+0000 7f731e000640 1 -- 172.21.15.28:0/2995070990 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f730c031870 con 0x7f732006dbf0 2024-02-21T23:13:37.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.274+0000 7f731e000640 1 -- 172.21.15.28:0/2995070990 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f730c031c40 con 0x7f732006dbf0 2024-02-21T23:13:37.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.274+0000 7f7325000640 1 -- 172.21.15.28:0/2995070990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f732006dbf0 msgr2=0x7f7320158ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:37.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.274+0000 7f7325000640 1 --2- 172.21.15.28:0/2995070990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f732006dbf0 0x7f7320158ac0 secure :-1 s=READY pgs=2393 cs=0 l=1 rev1=1 crypto rx=0x7f73200aa1b0 tx=0x7f730c02b910 comp rx=0 tx=0).stop 2024-02-21T23:13:37.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.275+0000 7f7325000640 1 -- 172.21.15.28:0/2995070990 shutdown_connections 2024-02-21T23:13:37.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.275+0000 7f7325000640 1 --2- 172.21.15.28:0/2995070990 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f73200c5720 0x7f7320159000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:37.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.275+0000 7f7325000640 1 --2- 172.21.15.28:0/2995070990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f732006dbf0 0x7f7320158ac0 unknown :-1 s=CLOSED pgs=2393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:37.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.275+0000 7f7325000640 1 --2- 172.21.15.28:0/2995070990 >> v2:172.21.15.81:3300/0 conn(0x7f73200aff90 0x7f73200c51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:37.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.275+0000 7f7325000640 1 -- 172.21.15.28:0/2995070990 >> 172.21.15.28:0/2995070990 conn(0x7f73200ac370 msgr2=0x7f73200ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:37.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.275+0000 7f7325000640 1 -- 172.21.15.28:0/2995070990 shutdown_connections 2024-02-21T23:13:37.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.275+0000 7f7325000640 1 -- 172.21.15.28:0/2995070990 wait complete. 2024-02-21T23:13:37.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.276+0000 7f7325000640 1 Processor -- start 2024-02-21T23:13:37.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.277+0000 7f7325000640 1 -- start start 2024-02-21T23:13:37.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.277+0000 7f7325000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f732006dbf0 0x7f7320154760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:37.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.277+0000 7f7325000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f73200aff90 0x7f73201ee2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:37.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.277+0000 7f7325000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73200c5720 0x7f73201ea2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:37.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.277+0000 7f7325000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73200c8760 con 0x7f732006dbf0 2024-02-21T23:13:37.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.277+0000 7f7325000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73200be0c0 con 0x7f73200c5720 2024-02-21T23:13:37.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.277+0000 7f7325000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73200c88e0 con 0x7f73200aff90 2024-02-21T23:13:37.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.278+0000 7f731f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f732006dbf0 0x7f7320154760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:37.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.278+0000 7f731f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f732006dbf0 0x7f7320154760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39446/0 (socket says 172.21.15.28:39446) 2024-02-21T23:13:37.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.278+0000 7f731f400640 1 -- 172.21.15.28:0/279379541 learned_addr learned my addr 172.21.15.28:0/279379541 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:37.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.278+0000 7f731ea00640 1 --2- 172.21.15.28:0/279379541 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f73200aff90 0x7f73201ee2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:37.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.278+0000 7f731fe00640 1 --2- 172.21.15.28:0/279379541 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73200c5720 0x7f73201ea2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:37.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.278+0000 7f731f400640 1 -- 172.21.15.28:0/279379541 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f73200aff90 msgr2=0x7f73201ee2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:37.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.278+0000 7f731f400640 1 --2- 172.21.15.28:0/279379541 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f73200aff90 0x7f73201ee2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:37.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.278+0000 7f731f400640 1 -- 172.21.15.28:0/279379541 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73200c5720 msgr2=0x7f73201ea2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:37.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.278+0000 7f731f400640 1 --2- 172.21.15.28:0/279379541 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73200c5720 0x7f73201ea2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:37.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.278+0000 7f731f400640 1 -- 172.21.15.28:0/279379541 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73201ed840 con 0x7f732006dbf0 2024-02-21T23:13:37.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.278+0000 7f731fe00640 1 --2- 172.21.15.28:0/279379541 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73200c5720 0x7f73201ea2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:13:37.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.279+0000 7f731f400640 1 --2- 172.21.15.28:0/279379541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f732006dbf0 0x7f7320154760 secure :-1 s=READY pgs=2394 cs=0 l=1 rev1=1 crypto rx=0x7f731000e810 tx=0x7f731000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:37.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.279+0000 7f7317400640 1 -- 172.21.15.28:0/279379541 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f731000f040 con 0x7f732006dbf0 2024-02-21T23:13:37.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.279+0000 7f7317400640 1 -- 172.21.15.28:0/279379541 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7310004030 con 0x7f732006dbf0 2024-02-21T23:13:37.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.279+0000 7f7317400640 1 -- 172.21.15.28:0/279379541 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7310004400 con 0x7f732006dbf0 2024-02-21T23:13:37.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.279+0000 7f7325000640 1 -- 172.21.15.28:0/279379541 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7320154d40 con 0x7f732006dbf0 2024-02-21T23:13:37.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.279+0000 7f7325000640 1 -- 172.21.15.28:0/279379541 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73201edae0 con 0x7f732006dbf0 2024-02-21T23:13:37.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.280+0000 7f7325000640 1 -- 172.21.15.28:0/279379541 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f732004f290 con 0x7f732006dbf0 2024-02-21T23:13:37.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.281+0000 7f7317400640 1 -- 172.21.15.28:0/279379541 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f731000c6d0 con 0x7f732006dbf0 2024-02-21T23:13:37.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.282+0000 7f7317400640 1 --2- 172.21.15.28:0/279379541 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f730006b4d0 0x7f730006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:37.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.282+0000 7f7317400640 1 -- 172.21.15.28:0/279379541 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f731008a0b0 con 0x7f732006dbf0 2024-02-21T23:13:37.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.283+0000 7f731ea00640 1 --2- 172.21.15.28:0/279379541 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f730006b4d0 0x7f730006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:37.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.284+0000 7f731ea00640 1 --2- 172.21.15.28:0/279379541 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f730006b4d0 0x7f730006d990 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f730c036000 tx=0x7f730c008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:37.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.286+0000 7f7317400640 1 -- 172.21.15.28:0/279379541 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7310017020 con 0x7f732006dbf0 2024-02-21T23:13:37.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.527+0000 7f7325000640 1 -- 172.21.15.28:0/279379541 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f732006d9d0 con 0x7f732006dbf0 2024-02-21T23:13:37.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.528+0000 7f7317400640 1 -- 172.21.15.28:0/279379541 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 229 v229) v1 ==== 95+0+4948 (secure 0 0 0) 0x7f7310058d90 con 0x7f732006dbf0 2024-02-21T23:13:37.517 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:13:37.517 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":229,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7265,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":224},{"gid":8287,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1607587459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1607587459},{"type":"v1","addr":"172.21.15.81:6835","nonce":1607587459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":224},{"gid":8290,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/1864206570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1864206570},{"type":"v1","addr":"172.21.15.28:6833","nonce":1864206570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":224}],"filesystems":[{"mdsmap":{"epoch":229,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:13:32.769089+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8296},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:13:37.517 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 229 2024-02-21T23:13:37.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.532+0000 7f7325000640 1 -- 172.21.15.28:0/279379541 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f730006b4d0 msgr2=0x7f730006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:37.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.532+0000 7f7325000640 1 --2- 172.21.15.28:0/279379541 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f730006b4d0 0x7f730006d990 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f730c036000 tx=0x7f730c008030 comp rx=0 tx=0).stop 2024-02-21T23:13:37.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.532+0000 7f7325000640 1 -- 172.21.15.28:0/279379541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f732006dbf0 msgr2=0x7f7320154760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:37.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.532+0000 7f7325000640 1 --2- 172.21.15.28:0/279379541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f732006dbf0 0x7f7320154760 secure :-1 s=READY pgs=2394 cs=0 l=1 rev1=1 crypto rx=0x7f731000e810 tx=0x7f731000ecd0 comp rx=0 tx=0).stop 2024-02-21T23:13:37.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.532+0000 7f7325000640 1 -- 172.21.15.28:0/279379541 shutdown_connections 2024-02-21T23:13:37.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.532+0000 7f7325000640 1 --2- 172.21.15.28:0/279379541 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f730006b4d0 0x7f730006d990 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:37.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.532+0000 7f7325000640 1 --2- 172.21.15.28:0/279379541 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f73200aff90 0x7f73201ee2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:37.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.532+0000 7f7325000640 1 --2- 172.21.15.28:0/279379541 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73200c5720 0x7f73201ea2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:37.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.532+0000 7f7325000640 1 --2- 172.21.15.28:0/279379541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f732006dbf0 0x7f7320154760 unknown :-1 s=CLOSED pgs=2394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:37.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.532+0000 7f7325000640 1 -- 172.21.15.28:0/279379541 >> 172.21.15.28:0/279379541 conn(0x7f73200ac370 msgr2=0x7f73200ac750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:37.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.532+0000 7f7325000640 1 -- 172.21.15.28:0/279379541 shutdown_connections 2024-02-21T23:13:37.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.532+0000 7f7325000640 1 -- 172.21.15.28:0/279379541 wait complete. 2024-02-21T23:13:37.545 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2024-02-21T23:13:37.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.798+0000 7f55cfe00640 1 Processor -- start 2024-02-21T23:13:37.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.799+0000 7f55cfe00640 1 -- start start 2024-02-21T23:13:37.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.800+0000 7f55cfe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f55d013e0c0 0x7f55d00c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:37.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.800+0000 7f55cfe00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55d0158550 con 0x7f55d006dbc0 2024-02-21T23:13:37.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.800+0000 7f55cfe00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55d01587d0 con 0x7f55d00c76f0 2024-02-21T23:13:37.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.800+0000 7f55cfe00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55d00c89a0 con 0x7f55d013e0c0 2024-02-21T23:13:37.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.800+0000 7f55cea00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f55d013e0c0 0x7f55d00c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:37.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.801+0000 7f55c7e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f55d006dbc0 0x7f55d006c280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60986/0 (socket says 172.21.15.28:60986) 2024-02-21T23:13:37.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.801+0000 7f55c7e00640 1 -- 172.21.15.28:0/4165067099 learned_addr learned my addr 172.21.15.28:0/4165067099 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:13:37.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.801+0000 7f55cea00640 1 -- 172.21.15.28:0/4165067099 >> v1:172.21.15.81:6789/0 conn(0x7f55d00c76f0 legacy=0x7f55d006c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:37.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.801+0000 7f55cea00640 1 -- 172.21.15.28:0/4165067099 >> v1:172.21.15.28:6789/0 conn(0x7f55d006dbc0 legacy=0x7f55d006c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:37.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.801+0000 7f55cea00640 1 -- 172.21.15.28:0/4165067099 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55d0158ab0 con 0x7f55d013e0c0 2024-02-21T23:13:37.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.801+0000 7f55ce000640 1 -- 172.21.15.28:0/4165067099 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2619285304 0 0) 0x7f55b8001110 con 0x7f55d006dbc0 2024-02-21T23:13:37.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.802+0000 7f55cea00640 1 --2- 172.21.15.28:0/4165067099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55d013e0c0 0x7f55d00c71b0 secure :-1 s=READY pgs=2396 cs=0 l=1 rev1=1 crypto rx=0x7f55c00097a0 tx=0x7f55c002b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5855294f1cf8994 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:37.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.803+0000 7f55ce000640 1 -- 172.21.15.28:0/4165067099 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55c0038070 con 0x7f55d013e0c0 2024-02-21T23:13:37.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.803+0000 7f55ce000640 1 -- 172.21.15.28:0/4165067099 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55c0031700 con 0x7f55d013e0c0 2024-02-21T23:13:37.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.803+0000 7f55ce000640 1 -- 172.21.15.28:0/4165067099 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55c0031ad0 con 0x7f55d013e0c0 2024-02-21T23:13:37.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.803+0000 7f55cfe00640 1 -- 172.21.15.28:0/4165067099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55d013e0c0 msgr2=0x7f55d00c71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:37.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.803+0000 7f55cfe00640 1 --2- 172.21.15.28:0/4165067099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55d013e0c0 0x7f55d00c71b0 secure :-1 s=READY pgs=2396 cs=0 l=1 rev1=1 crypto rx=0x7f55c00097a0 tx=0x7f55c002b7a0 comp rx=0 tx=0).stop 2024-02-21T23:13:37.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.803+0000 7f55cfe00640 1 -- 172.21.15.28:0/4165067099 shutdown_connections 2024-02-21T23:13:37.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.803+0000 7f55cfe00640 1 --2- 172.21.15.28:0/4165067099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55d013e0c0 0x7f55d00c71b0 unknown :-1 s=CLOSED pgs=2396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:37.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.803+0000 7f55cfe00640 1 -- 172.21.15.28:0/4165067099 >> 172.21.15.28:0/4165067099 conn(0x7f55d0073110 msgr2=0x7f55d00ac490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:37.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.804+0000 7f55cfe00640 1 -- 172.21.15.28:0/4165067099 shutdown_connections 2024-02-21T23:13:37.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.804+0000 7f55cfe00640 1 -- 172.21.15.28:0/4165067099 wait complete. 2024-02-21T23:13:37.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.805+0000 7f55cfe00640 1 Processor -- start 2024-02-21T23:13:37.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.805+0000 7f55cfe00640 1 -- start start 2024-02-21T23:13:37.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.806+0000 7f55cfe00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f55d006dbc0 0x7f55d01ee0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:37.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.806+0000 7f55cfe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55d00c76f0 0x7f55d01ee5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:37.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.806+0000 7f55cfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55d013e0c0 0x7f55d01ecd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:37.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.806+0000 7f55cfe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55d013e520 con 0x7f55d013e0c0 2024-02-21T23:13:37.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.806+0000 7f55cfe00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55d0159070 con 0x7f55d00c76f0 2024-02-21T23:13:37.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.806+0000 7f55cfe00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55d013e6a0 con 0x7f55d006dbc0 2024-02-21T23:13:37.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.807+0000 7f55cf400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55d013e0c0 0x7f55d01ecd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:37.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.807+0000 7f55cf400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55d013e0c0 0x7f55d01ecd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39464/0 (socket says 172.21.15.28:39464) 2024-02-21T23:13:37.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.807+0000 7f55cf400640 1 -- 172.21.15.28:0/3950251021 learned_addr learned my addr 172.21.15.28:0/3950251021 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:37.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.807+0000 7f55c7e00640 1 --2- 172.21.15.28:0/3950251021 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55d00c76f0 0x7f55d01ee5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:37.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.807+0000 7f55cea00640 1 --2- 172.21.15.28:0/3950251021 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f55d006dbc0 0x7f55d01ee0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:37.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.807+0000 7f55cf400640 1 -- 172.21.15.28:0/3950251021 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f55d006dbc0 msgr2=0x7f55d01ee0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:37.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.807+0000 7f55cf400640 1 --2- 172.21.15.28:0/3950251021 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f55d006dbc0 0x7f55d01ee0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:37.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.807+0000 7f55cf400640 1 -- 172.21.15.28:0/3950251021 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55d00c76f0 msgr2=0x7f55d01ee5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:37.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.807+0000 7f55cf400640 1 --2- 172.21.15.28:0/3950251021 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55d00c76f0 0x7f55d01ee5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:37.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.807+0000 7f55cf400640 1 -- 172.21.15.28:0/3950251021 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55d01ebfe0 con 0x7f55d013e0c0 2024-02-21T23:13:37.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.807+0000 7f55cf400640 1 --2- 172.21.15.28:0/3950251021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55d013e0c0 0x7f55d01ecd50 secure :-1 s=READY pgs=2397 cs=0 l=1 rev1=1 crypto rx=0x7f55c800b970 tx=0x7f55c800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:37.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.808+0000 7f55c7400640 1 -- 172.21.15.28:0/3950251021 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55c8019070 con 0x7f55d013e0c0 2024-02-21T23:13:37.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.808+0000 7f55c7400640 1 -- 172.21.15.28:0/3950251021 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55c800cb40 con 0x7f55d013e0c0 2024-02-21T23:13:37.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.808+0000 7f55cfe00640 1 -- 172.21.15.28:0/3950251021 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55d01ec270 con 0x7f55d013e0c0 2024-02-21T23:13:37.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.808+0000 7f55c7400640 1 -- 172.21.15.28:0/3950251021 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55c8014440 con 0x7f55d013e0c0 2024-02-21T23:13:37.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.808+0000 7f55cfe00640 1 -- 172.21.15.28:0/3950251021 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55d01ec830 con 0x7f55d013e0c0 2024-02-21T23:13:37.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.810+0000 7f55cfe00640 1 -- 172.21.15.28:0/3950251021 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55d01eca30 con 0x7f55d013e0c0 2024-02-21T23:13:37.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.810+0000 7f55c7400640 1 -- 172.21.15.28:0/3950251021 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f55c801c070 con 0x7f55d013e0c0 2024-02-21T23:13:37.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.812+0000 7f55c7400640 1 --2- 172.21.15.28:0/3950251021 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f55ac06b4d0 0x7f55ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:37.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.812+0000 7f55c7400640 1 -- 172.21.15.28:0/3950251021 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f55c808a0a0 con 0x7f55d013e0c0 2024-02-21T23:13:37.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.812+0000 7f55cea00640 1 --2- 172.21.15.28:0/3950251021 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f55ac06b4d0 0x7f55ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:37.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.813+0000 7f55cea00640 1 --2- 172.21.15.28:0/3950251021 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f55ac06b4d0 0x7f55ac06d990 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f55c0035000 tx=0x7f55c0008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:37.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:37.816+0000 7f55c7400640 1 -- 172.21.15.28:0/3950251021 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f55d01eca30 con 0x7f55d013e0c0 2024-02-21T23:13:38.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.059+0000 7f55cfe00640 1 -- 172.21.15.28:0/3950251021 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) v1 -- 0x7f55d004f290 con 0x7f55d013e0c0 2024-02-21T23:13:38.231 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-21T23:13:38.240 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-21T23:13:38.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.306+0000 7f55c7400640 1 -- 172.21.15.28:0/3950251021 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v230) v1 ==== 105+0+0 (secure 0 0 0) 0x7f55c8058c60 con 0x7f55d013e0c0 2024-02-21T23:13:38.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.310+0000 7f55cfe00640 1 -- 172.21.15.28:0/3950251021 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f55ac06b4d0 msgr2=0x7f55ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:38.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.310+0000 7f55cfe00640 1 --2- 172.21.15.28:0/3950251021 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f55ac06b4d0 0x7f55ac06d990 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f55c0035000 tx=0x7f55c0008030 comp rx=0 tx=0).stop 2024-02-21T23:13:38.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.310+0000 7f55cfe00640 1 -- 172.21.15.28:0/3950251021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55d013e0c0 msgr2=0x7f55d01ecd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:38.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.310+0000 7f55cfe00640 1 --2- 172.21.15.28:0/3950251021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55d013e0c0 0x7f55d01ecd50 secure :-1 s=READY pgs=2397 cs=0 l=1 rev1=1 crypto rx=0x7f55c800b970 tx=0x7f55c800be30 comp rx=0 tx=0).stop 2024-02-21T23:13:38.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.310+0000 7f55cfe00640 1 -- 172.21.15.28:0/3950251021 shutdown_connections 2024-02-21T23:13:38.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.310+0000 7f55cfe00640 1 --2- 172.21.15.28:0/3950251021 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f55ac06b4d0 0x7f55ac06d990 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:38.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.310+0000 7f55cfe00640 1 --2- 172.21.15.28:0/3950251021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55d013e0c0 0x7f55d01ecd50 unknown :-1 s=CLOSED pgs=2397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:38.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.310+0000 7f55cfe00640 1 --2- 172.21.15.28:0/3950251021 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55d00c76f0 0x7f55d01ee5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:38.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.310+0000 7f55cfe00640 1 --2- 172.21.15.28:0/3950251021 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f55d006dbc0 0x7f55d01ee0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:38.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.310+0000 7f55cfe00640 1 -- 172.21.15.28:0/3950251021 >> 172.21.15.28:0/3950251021 conn(0x7f55d0073110 msgr2=0x7f55d00c9f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:38.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.310+0000 7f55cfe00640 1 -- 172.21.15.28:0/3950251021 shutdown_connections 2024-02-21T23:13:38.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.310+0000 7f55cfe00640 1 -- 172.21.15.28:0/3950251021 wait complete. 2024-02-21T23:13:38.326 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:13:38.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.520+0000 7f1c13e00640 1 Processor -- start 2024-02-21T23:13:38.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.520+0000 7f1c13e00640 1 -- start start 2024-02-21T23:13:38.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.522+0000 7f1c13e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1c140c8a80 0x7f1c14156c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:38.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.522+0000 7f1c13e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1c140c8e60 0x7f1c14153640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:38.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.522+0000 7f1c13e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c14153b80 con 0x7f1c141540b0 2024-02-21T23:13:38.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.522+0000 7f1c13e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c140c92e0 con 0x7f1c140c8e60 2024-02-21T23:13:38.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.522+0000 7f1c13e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c140c8000 con 0x7f1c140c8a80 2024-02-21T23:13:38.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.522+0000 7f1c0be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1c140c8e60 0x7f1c14153640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:38.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.522+0000 7f1c0be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1c140c8e60 0x7f1c14153640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39476/0 (socket says 172.21.15.28:39476) 2024-02-21T23:13:38.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.522+0000 7f1c0be00640 1 -- 172.21.15.28:0/2909770412 learned_addr learned my addr 172.21.15.28:0/2909770412 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:38.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.522+0000 7f1c13400640 1 --1- 172.21.15.28:0/2909770412 >> v1:172.21.15.28:6789/0 conn(0x7f1c141540b0 0x7f1c140705a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:32770/0 (socket says 172.21.15.28:32770) 2024-02-21T23:13:38.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.523+0000 7f1c12a00640 1 --2- 172.21.15.28:0/2909770412 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1c140c8a80 0x7f1c14156c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:38.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.523+0000 7f1c0be00640 1 -- 172.21.15.28:0/2909770412 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1c140c8a80 msgr2=0x7f1c14156c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:38.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.523+0000 7f1c0be00640 1 --2- 172.21.15.28:0/2909770412 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1c140c8a80 0x7f1c14156c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:38.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.523+0000 7f1c0be00640 1 -- 172.21.15.28:0/2909770412 >> v1:172.21.15.28:6789/0 conn(0x7f1c141540b0 legacy=0x7f1c140705a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:38.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.523+0000 7f1c0be00640 1 -- 172.21.15.28:0/2909770412 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c140c82e0 con 0x7f1c140c8e60 2024-02-21T23:13:38.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.524+0000 7f1c0be00640 1 --2- 172.21.15.28:0/2909770412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c140c8e60 0x7f1c14153640 secure :-1 s=READY pgs=2399 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc009870 tx=0x7f1bfc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ba6575d3cca87f4 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:38.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.525+0000 7f1c12000640 1 -- 172.21.15.28:0/2909770412 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bfc034070 con 0x7f1c140c8e60 2024-02-21T23:13:38.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.525+0000 7f1c12000640 1 -- 172.21.15.28:0/2909770412 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bfc030840 con 0x7f1c140c8e60 2024-02-21T23:13:38.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.525+0000 7f1c12000640 1 -- 172.21.15.28:0/2909770412 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bfc030c10 con 0x7f1c140c8e60 2024-02-21T23:13:38.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.525+0000 7f1c13e00640 1 -- 172.21.15.28:0/2909770412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c140c8e60 msgr2=0x7f1c14153640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:38.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.525+0000 7f1c13e00640 1 --2- 172.21.15.28:0/2909770412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c140c8e60 0x7f1c14153640 secure :-1 s=READY pgs=2399 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc009870 tx=0x7f1bfc02b980 comp rx=0 tx=0).stop 2024-02-21T23:13:38.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.525+0000 7f1c13e00640 1 -- 172.21.15.28:0/2909770412 shutdown_connections 2024-02-21T23:13:38.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.525+0000 7f1c13e00640 1 --2- 172.21.15.28:0/2909770412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c140c8e60 0x7f1c14153640 unknown :-1 s=CLOSED pgs=2399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:38.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.525+0000 7f1c13e00640 1 --2- 172.21.15.28:0/2909770412 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1c140c8a80 0x7f1c14156c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:38.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.525+0000 7f1c13e00640 1 -- 172.21.15.28:0/2909770412 >> 172.21.15.28:0/2909770412 conn(0x7f1c1406b260 msgr2=0x7f1c1406b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:38.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.525+0000 7f1c13e00640 1 -- 172.21.15.28:0/2909770412 shutdown_connections 2024-02-21T23:13:38.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.525+0000 7f1c13e00640 1 -- 172.21.15.28:0/2909770412 wait complete. 2024-02-21T23:13:38.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.527+0000 7f1c13e00640 1 Processor -- start 2024-02-21T23:13:38.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.527+0000 7f1c13e00640 1 -- start start 2024-02-21T23:13:38.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.528+0000 7f1c13e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c140c8a80 0x7f1c1414d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:38.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.528+0000 7f1c13e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1c140c8e60 0x7f1c1414e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:38.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.528+0000 7f1c13e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c141540b0 0x7f1c140ac7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:38.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.528+0000 7f1c13e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c14156770 con 0x7f1c140c8a80 2024-02-21T23:13:38.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.528+0000 7f1c13e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c14156490 con 0x7f1c141540b0 2024-02-21T23:13:38.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.528+0000 7f1c13e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c140c7c90 con 0x7f1c140c8e60 2024-02-21T23:13:38.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.528+0000 7f1c12a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c140c8a80 0x7f1c1414d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:38.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.528+0000 7f1c12a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c140c8a80 0x7f1c1414d980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39486/0 (socket says 172.21.15.28:39486) 2024-02-21T23:13:38.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.528+0000 7f1c12a00640 1 -- 172.21.15.28:0/3399787508 learned_addr learned my addr 172.21.15.28:0/3399787508 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:38.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.529+0000 7f1c0be00640 1 --2- 172.21.15.28:0/3399787508 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1c140c8e60 0x7f1c1414e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:38.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.529+0000 7f1c12a00640 1 -- 172.21.15.28:0/3399787508 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1c140c8e60 msgr2=0x7f1c1414e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:38.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.529+0000 7f1c12a00640 1 --2- 172.21.15.28:0/3399787508 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1c140c8e60 0x7f1c1414e360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:38.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.529+0000 7f1c12a00640 1 -- 172.21.15.28:0/3399787508 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c141540b0 msgr2=0x7f1c140ac7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:38.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.529+0000 7f1c12a00640 1 --2- 172.21.15.28:0/3399787508 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c141540b0 0x7f1c140ac7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:38.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.529+0000 7f1c12a00640 1 -- 172.21.15.28:0/3399787508 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c140a84e0 con 0x7f1c140c8a80 2024-02-21T23:13:38.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.529+0000 7f1c12a00640 1 --2- 172.21.15.28:0/3399787508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c140c8a80 0x7f1c1414d980 secure :-1 s=READY pgs=2400 cs=0 l=1 rev1=1 crypto rx=0x7f1c0400c7f0 tx=0x7f1c0400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:38.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.529+0000 7f1c0aa00640 1 -- 172.21.15.28:0/3399787508 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c04015070 con 0x7f1c140c8a80 2024-02-21T23:13:38.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.530+0000 7f1c0aa00640 1 -- 172.21.15.28:0/3399787508 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c04004240 con 0x7f1c140c8a80 2024-02-21T23:13:38.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.530+0000 7f1c0aa00640 1 -- 172.21.15.28:0/3399787508 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c04007620 con 0x7f1c140c8a80 2024-02-21T23:13:38.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.530+0000 7f1c13e00640 1 -- 172.21.15.28:0/3399787508 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c140a8780 con 0x7f1c140c8a80 2024-02-21T23:13:38.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.530+0000 7f1c13e00640 1 -- 172.21.15.28:0/3399787508 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c140ac2c0 con 0x7f1c140c8a80 2024-02-21T23:13:38.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.531+0000 7f1c13e00640 1 -- 172.21.15.28:0/3399787508 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c1404f290 con 0x7f1c140c8a80 2024-02-21T23:13:38.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.532+0000 7f1c0aa00640 1 -- 172.21.15.28:0/3399787508 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1c040079d0 con 0x7f1c140c8a80 2024-02-21T23:13:38.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.533+0000 7f1c0aa00640 1 --2- 172.21.15.28:0/3399787508 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1bf006b4d0 0x7f1bf006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:38.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.533+0000 7f1c0aa00640 1 -- 172.21.15.28:0/3399787508 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1c0408a7b0 con 0x7f1c140c8a80 2024-02-21T23:13:38.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.534+0000 7f1c0be00640 1 --2- 172.21.15.28:0/3399787508 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1bf006b4d0 0x7f1bf006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:38.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.535+0000 7f1c0be00640 1 --2- 172.21.15.28:0/3399787508 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1bf006b4d0 0x7f1bf006d990 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc036000 tx=0x7f1bfc03b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:38.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.537+0000 7f1c0aa00640 1 -- 172.21.15.28:0/3399787508 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1c04059490 con 0x7f1c140c8a80 2024-02-21T23:13:38.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.783+0000 7f1c13e00640 1 -- 172.21.15.28:0/3399787508 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1c14156b90 con 0x7f1c140c8a80 2024-02-21T23:13:38.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.784+0000 7f1c0aa00640 1 -- 172.21.15.28:0/3399787508 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 231 v231) v1 ==== 95+0+4964 (secure 0 0 0) 0x7f1c04058d90 con 0x7f1c140c8a80 2024-02-21T23:13:38.773 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:13:38.773 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":231,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7265,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":224},{"gid":8287,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1607587459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1607587459},{"type":"v1","addr":"172.21.15.81:6835","nonce":1607587459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":224}],"filesystems":[{"mdsmap":{"epoch":231,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:13:38.310845+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8296,"mds_1":8290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8290":{"gid":8290,"name":"a","rank":1,"incarnation":231,"state":"up:creating","state_seq":1,"addr":"172.21.15.28:6833/1864206570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1864206570},{"type":"v1","addr":"172.21.15.28:6833","nonce":1864206570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:13:38.773 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 231 2024-02-21T23:13:38.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.787+0000 7f1c13e00640 1 -- 172.21.15.28:0/3399787508 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1bf006b4d0 msgr2=0x7f1bf006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:38.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.787+0000 7f1c13e00640 1 --2- 172.21.15.28:0/3399787508 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1bf006b4d0 0x7f1bf006d990 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc036000 tx=0x7f1bfc03b040 comp rx=0 tx=0).stop 2024-02-21T23:13:38.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.787+0000 7f1c13e00640 1 -- 172.21.15.28:0/3399787508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c140c8a80 msgr2=0x7f1c1414d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:38.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.787+0000 7f1c13e00640 1 --2- 172.21.15.28:0/3399787508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c140c8a80 0x7f1c1414d980 secure :-1 s=READY pgs=2400 cs=0 l=1 rev1=1 crypto rx=0x7f1c0400c7f0 tx=0x7f1c0400ccb0 comp rx=0 tx=0).stop 2024-02-21T23:13:38.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.787+0000 7f1c13e00640 1 -- 172.21.15.28:0/3399787508 shutdown_connections 2024-02-21T23:13:38.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.787+0000 7f1c13e00640 1 --2- 172.21.15.28:0/3399787508 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1bf006b4d0 0x7f1bf006d990 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:38.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.787+0000 7f1c13e00640 1 --2- 172.21.15.28:0/3399787508 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1c140c8e60 0x7f1c1414e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:38.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.787+0000 7f1c13e00640 1 --2- 172.21.15.28:0/3399787508 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1c141540b0 0x7f1c140ac7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:38.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.787+0000 7f1c13e00640 1 --2- 172.21.15.28:0/3399787508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c140c8a80 0x7f1c1414d980 unknown :-1 s=CLOSED pgs=2400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:38.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.787+0000 7f1c13e00640 1 -- 172.21.15.28:0/3399787508 >> 172.21.15.28:0/3399787508 conn(0x7f1c1406b260 msgr2=0x7f1c14152450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:38.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.787+0000 7f1c13e00640 1 -- 172.21.15.28:0/3399787508 shutdown_connections 2024-02-21T23:13:38.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:38.787+0000 7f1c13e00640 1 -- 172.21.15.28:0/3399787508 wait complete. 2024-02-21T23:13:39.196 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 231, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:13:38.310845+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 8290}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8290': {'gid': 8290, 'name': 'a', 'rank': 1, 'incarnation': 231, 'state': 'up:creating', 'state_seq': 1, 'addr': '172.21.15.28:6833/1864206570', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 1864206570}, {'type': 'v1', 'addr': '172.21.15.28:6833', 'nonce': 1864206570}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:13:39.197 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8290:up:creating 2024-02-21T23:13:39.622 INFO:tasks.ceph.mds.c.smithi028.stderr:2024-02-21T23:13:39.634+0000 7fc87a200640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-02-21T23:13:40.198 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:13:40.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.474+0000 7f6ab7e00640 1 Processor -- start 2024-02-21T23:13:40.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.474+0000 7f6ab7e00640 1 -- start start 2024-02-21T23:13:40.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.476+0000 7f6ab7e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6ab806bad0 0x7f6ab80a9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:40.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.476+0000 7f6ab7e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f6ab80c4d20 0x7f6ab806f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:40.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.476+0000 7f6ab7e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ab806b6a0 con 0x7f6ab81527c0 2024-02-21T23:13:40.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.476+0000 7f6ab7e00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ab80a9a20 con 0x7f6ab80c4d20 2024-02-21T23:13:40.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.476+0000 7f6ab7e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab806fc80 con 0x7f6ab806bad0 2024-02-21T23:13:40.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.477+0000 7f6ab7400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f6ab80c4d20 0x7f6ab806f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:40.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.477+0000 7f6ab6000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6ab806bad0 0x7f6ab80a9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:40.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.477+0000 7f6ab6000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6ab806bad0 0x7f6ab80a9eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3301/0 says I am v2:172.21.15.28:44540/0 (socket says 172.21.15.28:44540) 2024-02-21T23:13:40.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.477+0000 7f6ab6000640 1 -- 172.21.15.28:0/1979058107 learned_addr learned my addr 172.21.15.28:0/1979058107 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:40.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.478+0000 7f6ab7400640 1 -- 172.21.15.28:0/1979058107 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6ab806bad0 msgr2=0x7f6ab80a9eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:40.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.478+0000 7f6ab7400640 1 --2- 172.21.15.28:0/1979058107 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6ab806bad0 0x7f6ab80a9eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:40.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.478+0000 7f6ab7400640 1 -- 172.21.15.28:0/1979058107 >> v1:172.21.15.81:6789/0 conn(0x7f6ab81527c0 legacy=0x7f6ab8158580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:40.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.478+0000 7f6ab7400640 1 -- 172.21.15.28:0/1979058107 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ab806ff60 con 0x7f6ab80c4d20 2024-02-21T23:13:40.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.478+0000 7f6ab6000640 1 --2- 172.21.15.28:0/1979058107 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6ab806bad0 0x7f6ab80a9eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-21T23:13:40.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.479+0000 7f6ab7400640 1 --2- 172.21.15.28:0/1979058107 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ab80c4d20 0x7f6ab806f130 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f6ab0009d90 tx=0x7f6ab002be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9557562ddc5ad807 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:40.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.480+0000 7f6ab5600640 1 -- 172.21.15.28:0/1979058107 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab0039070 con 0x7f6ab80c4d20 2024-02-21T23:13:40.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.480+0000 7f6ab5600640 1 -- 172.21.15.28:0/1979058107 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ab002cbc0 con 0x7f6ab80c4d20 2024-02-21T23:13:40.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.480+0000 7f6ab5600640 1 -- 172.21.15.28:0/1979058107 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab0034440 con 0x7f6ab80c4d20 2024-02-21T23:13:40.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.480+0000 7f6ab7e00640 1 -- 172.21.15.28:0/1979058107 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ab80c4d20 msgr2=0x7f6ab806f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:40.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.480+0000 7f6ab7e00640 1 --2- 172.21.15.28:0/1979058107 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ab80c4d20 0x7f6ab806f130 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f6ab0009d90 tx=0x7f6ab002be10 comp rx=0 tx=0).stop 2024-02-21T23:13:40.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.481+0000 7f6ab7e00640 1 -- 172.21.15.28:0/1979058107 shutdown_connections 2024-02-21T23:13:40.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.481+0000 7f6ab7e00640 1 --2- 172.21.15.28:0/1979058107 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ab80c4d20 0x7f6ab806f130 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:40.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.481+0000 7f6ab7e00640 1 --2- 172.21.15.28:0/1979058107 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6ab806bad0 0x7f6ab80a9eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:40.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.481+0000 7f6ab7e00640 1 -- 172.21.15.28:0/1979058107 >> 172.21.15.28:0/1979058107 conn(0x7f6ab8067050 msgr2=0x7f6ab80a84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:40.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.481+0000 7f6ab7e00640 1 -- 172.21.15.28:0/1979058107 shutdown_connections 2024-02-21T23:13:40.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.481+0000 7f6ab7e00640 1 -- 172.21.15.28:0/1979058107 wait complete. 2024-02-21T23:13:40.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.483+0000 7f6ab7e00640 1 Processor -- start 2024-02-21T23:13:40.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.483+0000 7f6ab7e00640 1 -- start start 2024-02-21T23:13:40.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.484+0000 7f6ab7e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ab806bad0 0x7f6ab80c3380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:40.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.485+0000 7f6ab7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ab80c4d20 0x7f6ab8158270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:40.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.485+0000 7f6ab7e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6ab81527c0 0x7f6ab814aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:40.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.485+0000 7f6ab7e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab80a96b0 con 0x7f6ab80c4d20 2024-02-21T23:13:40.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.485+0000 7f6ab7e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab80a93d0 con 0x7f6ab806bad0 2024-02-21T23:13:40.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.485+0000 7f6ab7e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab8158c90 con 0x7f6ab81527c0 2024-02-21T23:13:40.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.485+0000 7f6ab6000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ab80c4d20 0x7f6ab8158270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:40.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.485+0000 7f6ab6000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ab80c4d20 0x7f6ab8158270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39498/0 (socket says 172.21.15.28:39498) 2024-02-21T23:13:40.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.485+0000 7f6ab6000640 1 -- 172.21.15.28:0/3923070861 learned_addr learned my addr 172.21.15.28:0/3923070861 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:40.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.485+0000 7f6ab6a00640 1 --2- 172.21.15.28:0/3923070861 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ab806bad0 0x7f6ab80c3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:40.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.485+0000 7f6ab7400640 1 --2- 172.21.15.28:0/3923070861 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6ab81527c0 0x7f6ab814aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:40.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.485+0000 7f6ab6000640 1 -- 172.21.15.28:0/3923070861 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6ab81527c0 msgr2=0x7f6ab814aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:40.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.485+0000 7f6ab6000640 1 --2- 172.21.15.28:0/3923070861 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6ab81527c0 0x7f6ab814aee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:40.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.486+0000 7f6ab6000640 1 -- 172.21.15.28:0/3923070861 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ab806bad0 msgr2=0x7f6ab80c3380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:40.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.486+0000 7f6ab6000640 1 --2- 172.21.15.28:0/3923070861 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ab806bad0 0x7f6ab80c3380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:40.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.486+0000 7f6ab6000640 1 -- 172.21.15.28:0/3923070861 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ab80aa6b0 con 0x7f6ab80c4d20 2024-02-21T23:13:40.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.486+0000 7f6ab6a00640 1 --2- 172.21.15.28:0/3923070861 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ab806bad0 0x7f6ab80c3380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:13:40.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.486+0000 7f6ab6000640 1 --2- 172.21.15.28:0/3923070861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ab80c4d20 0x7f6ab8158270 secure :-1 s=READY pgs=2401 cs=0 l=1 rev1=1 crypto rx=0x7f6aa400e9d0 tx=0x7f6aa400ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:40.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.486+0000 7f6aaea00640 1 -- 172.21.15.28:0/3923070861 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aa4017070 con 0x7f6ab80c4d20 2024-02-21T23:13:40.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.486+0000 7f6aaea00640 1 -- 172.21.15.28:0/3923070861 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6aa400ca60 con 0x7f6ab80c4d20 2024-02-21T23:13:40.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.486+0000 7f6aaea00640 1 -- 172.21.15.28:0/3923070861 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aa400ce10 con 0x7f6ab80c4d20 2024-02-21T23:13:40.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.486+0000 7f6ab7e00640 1 -- 172.21.15.28:0/3923070861 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ab80c5280 con 0x7f6ab80c4d20 2024-02-21T23:13:40.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.487+0000 7f6ab7e00640 1 -- 172.21.15.28:0/3923070861 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ab80c5540 con 0x7f6ab80c4d20 2024-02-21T23:13:40.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.488+0000 7f6ab7e00640 1 -- 172.21.15.28:0/3923070861 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ab804f290 con 0x7f6ab80c4d20 2024-02-21T23:13:40.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.490+0000 7f6aaea00640 1 -- 172.21.15.28:0/3923070861 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6aa401a070 con 0x7f6ab80c4d20 2024-02-21T23:13:40.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.491+0000 7f6aaea00640 1 --2- 172.21.15.28:0/3923070861 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6a9806b4d0 0x7f6a9806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:40.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.491+0000 7f6aaea00640 1 -- 172.21.15.28:0/3923070861 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6aa408a350 con 0x7f6ab80c4d20 2024-02-21T23:13:40.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.492+0000 7f6ab6a00640 1 --2- 172.21.15.28:0/3923070861 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6a9806b4d0 0x7f6a9806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:40.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.493+0000 7f6ab6a00640 1 --2- 172.21.15.28:0/3923070861 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6a9806b4d0 0x7f6a9806d990 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f6aa8007fd0 tx=0x7f6aa8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:40.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.496+0000 7f6aaea00640 1 -- 172.21.15.28:0/3923070861 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f6aa4059040 con 0x7f6ab80c4d20 2024-02-21T23:13:40.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.737+0000 7f6ab7e00640 1 -- 172.21.15.28:0/3923070861 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6ab80c48c0 con 0x7f6ab80c4d20 2024-02-21T23:13:40.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.738+0000 7f6aaea00640 1 -- 172.21.15.28:0/3923070861 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 232 v232) v1 ==== 95+0+4962 (secure 0 0 0) 0x7f6aa4058940 con 0x7f6ab80c4d20 2024-02-21T23:13:40.727 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:13:40.727 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":232,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7265,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":224},{"gid":8287,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1607587459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1607587459},{"type":"v1","addr":"172.21.15.81:6835","nonce":1607587459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":224}],"filesystems":[{"mdsmap":{"epoch":232,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:13:38.336841+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8296,"mds_1":8290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8290":{"gid":8290,"name":"a","rank":1,"incarnation":231,"state":"up:active","state_seq":5,"addr":"172.21.15.28:6833/1864206570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1864206570},{"type":"v1","addr":"172.21.15.28:6833","nonce":1864206570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:13:40.727 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 232 2024-02-21T23:13:40.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.742+0000 7f6ab7e00640 1 -- 172.21.15.28:0/3923070861 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6a9806b4d0 msgr2=0x7f6a9806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:40.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.742+0000 7f6ab7e00640 1 --2- 172.21.15.28:0/3923070861 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6a9806b4d0 0x7f6a9806d990 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f6aa8007fd0 tx=0x7f6aa8009040 comp rx=0 tx=0).stop 2024-02-21T23:13:40.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.742+0000 7f6ab7e00640 1 -- 172.21.15.28:0/3923070861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ab80c4d20 msgr2=0x7f6ab8158270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:40.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.742+0000 7f6ab7e00640 1 --2- 172.21.15.28:0/3923070861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ab80c4d20 0x7f6ab8158270 secure :-1 s=READY pgs=2401 cs=0 l=1 rev1=1 crypto rx=0x7f6aa400e9d0 tx=0x7f6aa400ee90 comp rx=0 tx=0).stop 2024-02-21T23:13:40.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.742+0000 7f6ab7e00640 1 -- 172.21.15.28:0/3923070861 shutdown_connections 2024-02-21T23:13:40.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.742+0000 7f6ab7e00640 1 --2- 172.21.15.28:0/3923070861 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6a9806b4d0 0x7f6a9806d990 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:40.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.742+0000 7f6ab7e00640 1 --2- 172.21.15.28:0/3923070861 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6ab81527c0 0x7f6ab814aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:40.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.742+0000 7f6ab7e00640 1 --2- 172.21.15.28:0/3923070861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ab80c4d20 0x7f6ab8158270 unknown :-1 s=CLOSED pgs=2401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:40.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.742+0000 7f6ab7e00640 1 --2- 172.21.15.28:0/3923070861 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ab806bad0 0x7f6ab80c3380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:40.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.742+0000 7f6ab7e00640 1 -- 172.21.15.28:0/3923070861 >> 172.21.15.28:0/3923070861 conn(0x7f6ab8067050 msgr2=0x7f6ab80c7a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:40.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.742+0000 7f6ab7e00640 1 -- 172.21.15.28:0/3923070861 shutdown_connections 2024-02-21T23:13:40.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:40.742+0000 7f6ab7e00640 1 -- 172.21.15.28:0/3923070861 wait complete. 2024-02-21T23:13:41.154 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 232, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:13:38.336841+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 8290}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8290': {'gid': 8290, 'name': 'a', 'rank': 1, 'incarnation': 231, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.28:6833/1864206570', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 1864206570}, {'type': 'v1', 'addr': '172.21.15.28:6833', 'nonce': 1864206570}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:13:41.154 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 2/2 2024-02-21T23:13:41.154 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-02-21T23:13:41.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.392+0000 7fb271c00640 1 Processor -- start 2024-02-21T23:13:41.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.392+0000 7fb271c00640 1 -- start start 2024-02-21T23:13:41.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.393+0000 7fb271c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb26c0c29b0 0x7fb26c0c1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:41.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.393+0000 7fb271c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fb26c0aaec0 0x7fb26c15f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:41.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.393+0000 7fb271c00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb26c0c31c0 con 0x7fb26c0c1a50 2024-02-21T23:13:41.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.393+0000 7fb271c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb26c0c34a0 con 0x7fb26c0c29b0 2024-02-21T23:13:41.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.393+0000 7fb271c00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb26c0c3780 con 0x7fb26c0aaec0 2024-02-21T23:13:41.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.394+0000 7fb26b400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb26c0c29b0 0x7fb26c0c1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:41.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.394+0000 7fb271200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fb26c0aaec0 0x7fb26c15f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:41.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.394+0000 7fb26b400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb26c0c29b0 0x7fb26c0c1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39510/0 (socket says 172.21.15.28:39510) 2024-02-21T23:13:41.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.394+0000 7fb26b400640 1 -- 172.21.15.28:0/4184338920 learned_addr learned my addr 172.21.15.28:0/4184338920 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:41.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.394+0000 7fb26b400640 1 -- 172.21.15.28:0/4184338920 >> v2:172.21.15.81:3300/0 conn(0x7fb26c0aaec0 msgr2=0x7fb26c15f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:41.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.394+0000 7fb26b400640 1 --2- 172.21.15.28:0/4184338920 >> v2:172.21.15.81:3300/0 conn(0x7fb26c0aaec0 0x7fb26c15f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:41.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.394+0000 7fb26b400640 1 -- 172.21.15.28:0/4184338920 >> v1:172.21.15.81:6789/0 conn(0x7fb26c0c1a50 legacy=0x7fb26c06cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:41.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.394+0000 7fb26b400640 1 -- 172.21.15.28:0/4184338920 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb26c0c3a60 con 0x7fb26c0c29b0 2024-02-21T23:13:41.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.395+0000 7fb26b400640 1 --2- 172.21.15.28:0/4184338920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb26c0c29b0 0x7fb26c0c1e30 secure :-1 s=READY pgs=2402 cs=0 l=1 rev1=1 crypto rx=0x7fb26c06efa0 tx=0x7fb25802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8da2baf136b5b97d server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:41.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.396+0000 7fb26aa00640 1 -- 172.21.15.28:0/4184338920 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb258035020 con 0x7fb26c0c29b0 2024-02-21T23:13:41.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.396+0000 7fb26aa00640 1 -- 172.21.15.28:0/4184338920 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb258031870 con 0x7fb26c0c29b0 2024-02-21T23:13:41.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.396+0000 7fb26aa00640 1 -- 172.21.15.28:0/4184338920 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb258031c40 con 0x7fb26c0c29b0 2024-02-21T23:13:41.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.397+0000 7fb271c00640 1 -- 172.21.15.28:0/4184338920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb26c0c29b0 msgr2=0x7fb26c0c1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:41.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.397+0000 7fb271c00640 1 --2- 172.21.15.28:0/4184338920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb26c0c29b0 0x7fb26c0c1e30 secure :-1 s=READY pgs=2402 cs=0 l=1 rev1=1 crypto rx=0x7fb26c06efa0 tx=0x7fb25802b910 comp rx=0 tx=0).stop 2024-02-21T23:13:41.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.397+0000 7fb271c00640 1 -- 172.21.15.28:0/4184338920 shutdown_connections 2024-02-21T23:13:41.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.397+0000 7fb271c00640 1 --2- 172.21.15.28:0/4184338920 >> v2:172.21.15.81:3300/0 conn(0x7fb26c0aaec0 0x7fb26c15f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:41.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.397+0000 7fb271c00640 1 --2- 172.21.15.28:0/4184338920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb26c0c29b0 0x7fb26c0c1e30 unknown :-1 s=CLOSED pgs=2402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:41.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.397+0000 7fb271c00640 1 -- 172.21.15.28:0/4184338920 >> 172.21.15.28:0/4184338920 conn(0x7fb26c0aa3d0 msgr2=0x7fb26c0aa7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:41.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.397+0000 7fb271c00640 1 -- 172.21.15.28:0/4184338920 shutdown_connections 2024-02-21T23:13:41.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.397+0000 7fb271c00640 1 -- 172.21.15.28:0/4184338920 wait complete. 2024-02-21T23:13:41.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.399+0000 7fb271c00640 1 Processor -- start 2024-02-21T23:13:41.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.399+0000 7fb271c00640 1 -- start start 2024-02-21T23:13:41.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.400+0000 7fb271c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb26c0aaec0 0x7fb26c0a80a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:41.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.400+0000 7fb271c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb26c0c1a50 0x7fb26c14fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:41.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.400+0000 7fb271c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb26c0c29b0 0x7fb26c070460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:41.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.400+0000 7fb271c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb26c0c7d10 con 0x7fb26c0aaec0 2024-02-21T23:13:41.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.400+0000 7fb271c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb26c1528c0 con 0x7fb26c0c29b0 2024-02-21T23:13:41.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.400+0000 7fb271c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb26c0c7e90 con 0x7fb26c0c1a50 2024-02-21T23:13:41.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.400+0000 7fb26be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb26c0aaec0 0x7fb26c0a80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:41.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.400+0000 7fb26be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb26c0aaec0 0x7fb26c0a80a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39516/0 (socket says 172.21.15.28:39516) 2024-02-21T23:13:41.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.400+0000 7fb26be00640 1 -- 172.21.15.28:0/890258460 learned_addr learned my addr 172.21.15.28:0/890258460 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:41.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.401+0000 7fb26b400640 1 --2- 172.21.15.28:0/890258460 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb26c0c1a50 0x7fb26c14fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:41.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.401+0000 7fb271200640 1 --2- 172.21.15.28:0/890258460 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb26c0c29b0 0x7fb26c070460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:41.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.401+0000 7fb26be00640 1 -- 172.21.15.28:0/890258460 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb26c0c1a50 msgr2=0x7fb26c14fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:41.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.401+0000 7fb26be00640 1 --2- 172.21.15.28:0/890258460 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb26c0c1a50 0x7fb26c14fcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:41.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.401+0000 7fb26be00640 1 -- 172.21.15.28:0/890258460 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb26c0c29b0 msgr2=0x7fb26c070460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:41.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.401+0000 7fb26be00640 1 --2- 172.21.15.28:0/890258460 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb26c0c29b0 0x7fb26c070460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:41.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.401+0000 7fb26be00640 1 -- 172.21.15.28:0/890258460 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb26c0ae6f0 con 0x7fb26c0aaec0 2024-02-21T23:13:41.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.401+0000 7fb26be00640 1 --2- 172.21.15.28:0/890258460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb26c0aaec0 0x7fb26c0a80a0 secure :-1 s=READY pgs=2403 cs=0 l=1 rev1=1 crypto rx=0x7fb25c00c480 tx=0x7fb25c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:41.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.402+0000 7fb263e00640 1 -- 172.21.15.28:0/890258460 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb25c0052d0 con 0x7fb26c0aaec0 2024-02-21T23:13:41.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.402+0000 7fb263e00640 1 -- 172.21.15.28:0/890258460 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb25c005470 con 0x7fb26c0aaec0 2024-02-21T23:13:41.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.402+0000 7fb263e00640 1 -- 172.21.15.28:0/890258460 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb25c012870 con 0x7fb26c0aaec0 2024-02-21T23:13:41.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.402+0000 7fb271c00640 1 -- 172.21.15.28:0/890258460 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb26c14da30 con 0x7fb26c0aaec0 2024-02-21T23:13:41.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.402+0000 7fb271c00640 1 -- 172.21.15.28:0/890258460 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb26c150230 con 0x7fb26c0aaec0 2024-02-21T23:13:41.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.405+0000 7fb263e00640 1 -- 172.21.15.28:0/890258460 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb25c012a10 con 0x7fb26c0aaec0 2024-02-21T23:13:41.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.406+0000 7fb276042740 1 Processor -- start 2024-02-21T23:13:41.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.406+0000 7fb276042740 1 -- start start 2024-02-21T23:13:41.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.406+0000 7fb263e00640 1 --2- 172.21.15.28:0/890258460 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb24c06b4d0 0x7fb24c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:41.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.406+0000 7fb263e00640 1 -- 172.21.15.28:0/890258460 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb25c089fb0 con 0x7fb26c0aaec0 2024-02-21T23:13:41.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.407+0000 7fb276042740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55a33ff19020 0x55a33ff19400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:41.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.407+0000 7fb276042740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a33ff1a330 0x55a33ff55170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:41.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.407+0000 7fb271200640 1 --2- 172.21.15.28:0/890258460 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb24c06b4d0 0x7fb24c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:41.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.407+0000 7fb276042740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a33ff556b0 0x55a33ff53720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:41.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.407+0000 7fb276042740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a33fe7d470 con 0x55a33ff556b0 2024-02-21T23:13:41.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.407+0000 7fb276042740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a33fe26af0 con 0x55a33ff1a330 2024-02-21T23:13:41.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.407+0000 7fb276042740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a33fe79510 con 0x55a33ff19020 2024-02-21T23:13:41.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.408+0000 7fb271200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a33ff556b0 0x55a33ff53720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:41.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.408+0000 7fb271200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a33ff556b0 0x55a33ff53720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39528/0 (socket says 172.21.15.28:39528) 2024-02-21T23:13:41.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.408+0000 7fb271200640 1 -- 172.21.15.28:0/502641357 learned_addr learned my addr 172.21.15.28:0/502641357 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:41.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.408+0000 7fb26be00640 1 --2- 172.21.15.28:0/502641357 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55a33ff19020 0x55a33ff19400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:41.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.408+0000 7fb26b400640 1 --2- 172.21.15.28:0/502641357 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a33ff1a330 0x55a33ff55170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:41.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.408+0000 7fb271200640 1 -- 172.21.15.28:0/502641357 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55a33ff19020 msgr2=0x55a33ff19400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:41.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.408+0000 7fb271200640 1 --2- 172.21.15.28:0/502641357 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55a33ff19020 0x55a33ff19400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:41.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.408+0000 7fb271200640 1 -- 172.21.15.28:0/502641357 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a33ff1a330 msgr2=0x55a33ff55170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:41.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.408+0000 7fb271200640 1 --2- 172.21.15.28:0/502641357 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a33ff1a330 0x55a33ff55170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:41.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.408+0000 7fb271200640 1 -- 172.21.15.28:0/502641357 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a33fe2dd90 con 0x55a33ff556b0 2024-02-21T23:13:41.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.408+0000 7fb26b400640 1 --2- 172.21.15.28:0/502641357 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a33ff1a330 0x55a33ff55170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:13:41.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.408+0000 7fb271200640 1 --2- 172.21.15.28:0/502641357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a33ff556b0 0x55a33ff53720 secure :-1 s=READY pgs=2404 cs=0 l=1 rev1=1 crypto rx=0x7fb264007f80 tx=0x7fb26401a810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2a34eefb49eb4b server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:41.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.409+0000 7fb260c00640 1 -- 172.21.15.28:0/502641357 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb264025620 con 0x55a33ff556b0 2024-02-21T23:13:41.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.409+0000 7fb260c00640 1 -- 172.21.15.28:0/502641357 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb26401f070 con 0x55a33ff556b0 2024-02-21T23:13:41.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.409+0000 7fb260c00640 1 -- 172.21.15.28:0/502641357 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb264025a00 con 0x55a33ff556b0 2024-02-21T23:13:41.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.409+0000 7fb276042740 1 -- 172.21.15.28:0/502641357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a33ff556b0 msgr2=0x55a33ff53720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:41.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.409+0000 7fb271200640 1 --2- 172.21.15.28:0/890258460 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb24c06b4d0 0x7fb24c06d990 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fb264009fd0 tx=0x7fb264012000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:41.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.409+0000 7fb276042740 1 --2- 172.21.15.28:0/502641357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a33ff556b0 0x55a33ff53720 secure :-1 s=READY pgs=2404 cs=0 l=1 rev1=1 crypto rx=0x7fb264007f80 tx=0x7fb26401a810 comp rx=0 tx=0).stop 2024-02-21T23:13:41.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.410+0000 7fb276042740 1 -- 172.21.15.28:0/502641357 shutdown_connections 2024-02-21T23:13:41.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.410+0000 7fb276042740 1 --2- 172.21.15.28:0/502641357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a33ff556b0 0x55a33ff53720 unknown :-1 s=CLOSED pgs=2404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:41.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.410+0000 7fb276042740 1 --2- 172.21.15.28:0/502641357 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a33ff1a330 0x55a33ff55170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:41.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.410+0000 7fb276042740 1 --2- 172.21.15.28:0/502641357 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55a33ff19020 0x55a33ff19400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:41.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.410+0000 7fb276042740 1 -- 172.21.15.28:0/502641357 >> 172.21.15.28:0/502641357 conn(0x55a33ff15970 msgr2=0x55a33ff3a380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:41.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.410+0000 7fb276042740 1 -- 172.21.15.28:0/502641357 shutdown_connections 2024-02-21T23:13:41.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.410+0000 7fb276042740 1 -- 172.21.15.28:0/502641357 wait complete. 2024-02-21T23:13:41.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.411+0000 7fb276042740 1 -- start start 2024-02-21T23:13:41.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.411+0000 7fb276042740 1 Processor -- start 2024-02-21T23:13:41.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.413+0000 7fb276042740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a33ff1a330 0x55a33ff689a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:41.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.413+0000 7fb276042740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55a33ff556b0 0x55a33ff39470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:41.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.413+0000 7fb276042740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a33ff3a650 0x55a33ff650a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:41.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.413+0000 7fb276042740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a33fe1c040 con 0x55a33ff1a330 2024-02-21T23:13:41.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.413+0000 7fb276042740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a33ff3aa90 con 0x55a33ff3a650 2024-02-21T23:13:41.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.413+0000 7fb276042740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a33ff5aad0 con 0x55a33ff556b0 2024-02-21T23:13:41.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.413+0000 7fb276042740 10 client.0 _ll_register_callbacks cb 0x55a33fe39920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:13:41.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.413+0000 7fb26be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a33ff1a330 0x55a33ff689a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:41.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.413+0000 7fb26be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a33ff1a330 0x55a33ff689a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39542/0 (socket says 172.21.15.28:39542) 2024-02-21T23:13:41.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.413+0000 7fb26be00640 1 -- 172.21.15.28:0/4280628711 learned_addr learned my addr 172.21.15.28:0/4280628711 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:41.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.413+0000 7fb26b400640 1 --2- 172.21.15.28:0/4280628711 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55a33ff556b0 0x55a33ff39470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:41.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.413+0000 7fb271200640 1 --2- 172.21.15.28:0/4280628711 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a33ff3a650 0x55a33ff650a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:41.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.413+0000 7fb26be00640 1 -- 172.21.15.28:0/4280628711 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55a33ff556b0 msgr2=0x55a33ff39470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:41.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.413+0000 7fb26be00640 1 --2- 172.21.15.28:0/4280628711 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55a33ff556b0 0x55a33ff39470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:41.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.414+0000 7fb26be00640 1 -- 172.21.15.28:0/4280628711 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a33ff3a650 msgr2=0x55a33ff650a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:41.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.414+0000 7fb26be00640 1 --2- 172.21.15.28:0/4280628711 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a33ff3a650 0x55a33ff650a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:41.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.414+0000 7fb26be00640 1 -- 172.21.15.28:0/4280628711 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a33fe3baa0 con 0x55a33ff1a330 2024-02-21T23:13:41.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.414+0000 7fb271200640 1 --2- 172.21.15.28:0/4280628711 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a33ff3a650 0x55a33ff650a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:13:41.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.414+0000 7fb26be00640 1 --2- 172.21.15.28:0/4280628711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a33ff1a330 0x55a33ff689a0 secure :-1 s=READY pgs=2405 cs=0 l=1 rev1=1 crypto rx=0x7fb25c00c450 tx=0x7fb25c0586d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1965a9f15d5a1003 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:41.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.414+0000 7fb255600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:13:41.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.414+0000 7fb255600640 1 -- 172.21.15.28:0/4280628711 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb25c070180 con 0x55a33ff1a330 2024-02-21T23:13:41.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.414+0000 7fb255600640 1 -- 172.21.15.28:0/4280628711 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(208..208 src has 1..208) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb25c054c80 con 0x55a33ff1a330 2024-02-21T23:13:41.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.414+0000 7fb276042740 1 -- 172.21.15.28:0/4280628711 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb26401f070 con 0x55a33ff1a330 2024-02-21T23:13:41.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.414+0000 7fb255600640 1 -- 172.21.15.28:0/4280628711 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=232) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb25c060890 con 0x55a33ff1a330 2024-02-21T23:13:41.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.414+0000 7fb276042740 10 client.10959 fetch_fsmap learned FSMap version 232 2024-02-21T23:13:41.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.414+0000 7fb276042740 1 -- 172.21.15.28:0/4280628711 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=232,monmap=2+}) v3 -- 0x55a33fb922d0 con 0x55a33ff1a330 2024-02-21T23:13:41.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.415+0000 7fb255600640 1 -- 172.21.15.28:0/4280628711 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb26401f070 con 0x55a33ff1a330 2024-02-21T23:13:41.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.415+0000 7fb255600640 1 -- 172.21.15.28:0/4280628711 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 232) v1 ==== 3306+0+0 (secure 0 0 0) 0x7fb25c057a60 con 0x55a33ff1a330 2024-02-21T23:13:41.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.415+0000 7fb276042740 10 client.10959 fetch_fsmap finished waiting for FSMap version 232 2024-02-21T23:13:41.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.415+0000 7fb276042740 10 client.10959 resolve_mds: resolved name 'a' to daemon mds.a 2024-02-21T23:13:41.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.415+0000 7fb276042740 20 client.10959 populate_metadata read hostname 'smithi028' 2024-02-21T23:13:41.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.415+0000 7fb276042740 1 --2- 172.21.15.28:0/4280628711 >> [v2:172.21.15.28:6832/1864206570,v1:172.21.15.28:6833/1864206570] conn(0x55a33ff348d0 0x55a33ff34cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:41.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.415+0000 7fb276042740 4 client.10959 mds_command: new command op to 8290 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-21T23:13:41.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.415+0000 7fb276042740 1 -- 172.21.15.28:0/4280628711 --> [v2:172.21.15.28:6832/1864206570,v1:172.21.15.28:6833/1864206570] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a33fe8a9b0 con 0x55a33ff348d0 2024-02-21T23:13:41.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.415+0000 7fb26b400640 1 --2- 172.21.15.28:0/4280628711 >> [v2:172.21.15.28:6832/1864206570,v1:172.21.15.28:6833/1864206570] conn(0x55a33ff348d0 0x55a33ff34cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:41.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.416+0000 7fb26b400640 1 --2- 172.21.15.28:0/4280628711 >> [v2:172.21.15.28:6832/1864206570,v1:172.21.15.28:6833/1864206570] conn(0x55a33ff348d0 0x55a33ff34cb0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6459895fd32f4348 server_cookie=a69a48fa9bf3fffe in_seq=0 out_seq=0 2024-02-21T23:13:41.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.416+0000 7fb255600640 10 client.10959 ms_handle_connect on v2:172.21.15.28:6832/1864206570 2024-02-21T23:13:41.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.417+0000 7fb255600640 1 -- 172.21.15.28:0/4280628711 <== mds.1 v2:172.21.15.28:6832/1864206570 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55a33fe8a9b0 con 0x55a33ff348d0 2024-02-21T23:13:41.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.417+0000 7fb255600640 10 client.10959 handle_command_reply: tid=0 2024-02-21T23:13:41.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.417+0000 7fb276042740 1 client.10959 shutdown 2024-02-21T23:13:41.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.417+0000 7fb276042740 1 -- 172.21.15.28:0/4280628711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a33ff1a330 msgr2=0x55a33ff689a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:41.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.417+0000 7fb276042740 1 --2- 172.21.15.28:0/4280628711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a33ff1a330 0x55a33ff689a0 secure :-1 s=READY pgs=2405 cs=0 l=1 rev1=1 crypto rx=0x7fb25c00c450 tx=0x7fb25c0586d0 comp rx=0 tx=0).stop 2024-02-21T23:13:41.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.417+0000 7fb276042740 1 -- 172.21.15.28:0/4280628711 shutdown_connections 2024-02-21T23:13:41.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.417+0000 7fb276042740 1 --2- 172.21.15.28:0/4280628711 >> [v2:172.21.15.28:6832/1864206570,v1:172.21.15.28:6833/1864206570] conn(0x55a33ff348d0 0x55a33ff34cb0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:41.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.417+0000 7fb276042740 1 --2- 172.21.15.28:0/4280628711 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55a33ff556b0 0x55a33ff39470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:41.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.417+0000 7fb276042740 1 --2- 172.21.15.28:0/4280628711 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a33ff3a650 0x55a33ff650a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:41.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.417+0000 7fb276042740 1 --2- 172.21.15.28:0/4280628711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a33ff1a330 0x55a33ff689a0 unknown :-1 s=CLOSED pgs=2405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:41.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.417+0000 7fb255600640 0 client.10959 ms_handle_reset on v2:172.21.15.28:6832/1864206570 2024-02-21T23:13:41.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.417+0000 7fb276042740 1 -- 172.21.15.28:0/4280628711 >> 172.21.15.28:0/4280628711 conn(0x55a33ff19020 msgr2=0x55a33ff58790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:41.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.418+0000 7fb276042740 1 -- 172.21.15.28:0/4280628711 shutdown_connections 2024-02-21T23:13:41.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.418+0000 7fb276042740 1 -- 172.21.15.28:0/4280628711 wait complete. 2024-02-21T23:13:41.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.418+0000 7fb276042740 20 client.10959 trim_cache size 0 max 16384 2024-02-21T23:13:41.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.436+0000 7fb276042740 1 Processor -- start 2024-02-21T23:13:41.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.436+0000 7fb276042740 1 -- start start 2024-02-21T23:13:41.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.437+0000 7fb276042740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55a33ff1a330 0x55a33ff5c070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:41.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.437+0000 7fb276042740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a33ff3a650 0x55a33ff54890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:41.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.437+0000 7fb276042740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a33ff556b0 0x55a33ff54dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:41.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.437+0000 7fb276042740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb26c1f7a50 con 0x55a33ff556b0 2024-02-21T23:13:41.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.437+0000 7fb276042740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb26c1eeae0 con 0x55a33ff3a650 2024-02-21T23:13:41.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.437+0000 7fb276042740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb26c1f6620 con 0x55a33ff1a330 2024-02-21T23:13:41.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.437+0000 7fb271200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a33ff556b0 0x55a33ff54dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:41.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.437+0000 7fb271200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a33ff556b0 0x55a33ff54dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39544/0 (socket says 172.21.15.28:39544) 2024-02-21T23:13:41.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.437+0000 7fb271200640 1 -- 172.21.15.28:0/226879099 learned_addr learned my addr 172.21.15.28:0/226879099 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:41.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.437+0000 7fb26be00640 1 --2- 172.21.15.28:0/226879099 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55a33ff1a330 0x55a33ff5c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:41.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.438+0000 7fb26b400640 1 --2- 172.21.15.28:0/226879099 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a33ff3a650 0x55a33ff54890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:41.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.438+0000 7fb271200640 1 -- 172.21.15.28:0/226879099 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55a33ff1a330 msgr2=0x55a33ff5c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:41.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.438+0000 7fb271200640 1 --2- 172.21.15.28:0/226879099 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55a33ff1a330 0x55a33ff5c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:41.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.438+0000 7fb271200640 1 -- 172.21.15.28:0/226879099 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a33ff3a650 msgr2=0x55a33ff54890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:41.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.438+0000 7fb271200640 1 --2- 172.21.15.28:0/226879099 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a33ff3a650 0x55a33ff54890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:41.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.438+0000 7fb271200640 1 -- 172.21.15.28:0/226879099 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a33fc5c6b0 con 0x55a33ff556b0 2024-02-21T23:13:41.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.438+0000 7fb271200640 1 --2- 172.21.15.28:0/226879099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a33ff556b0 0x55a33ff54dd0 secure :-1 s=READY pgs=2406 cs=0 l=1 rev1=1 crypto rx=0x7fb26401ad10 tx=0x7fb26401d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcfc598ec2c9e5f9 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:41.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.439+0000 7fb257e00640 1 -- 172.21.15.28:0/226879099 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb264029070 con 0x55a33ff556b0 2024-02-21T23:13:41.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.439+0000 7fb257e00640 1 -- 172.21.15.28:0/226879099 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb26402a040 con 0x55a33ff556b0 2024-02-21T23:13:41.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.439+0000 7fb257e00640 1 -- 172.21.15.28:0/226879099 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb264007630 con 0x55a33ff556b0 2024-02-21T23:13:41.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.440+0000 7fb276042740 1 -- 172.21.15.28:0/226879099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a33ff556b0 msgr2=0x55a33ff54dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:41.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.440+0000 7fb276042740 1 --2- 172.21.15.28:0/226879099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a33ff556b0 0x55a33ff54dd0 secure :-1 s=READY pgs=2406 cs=0 l=1 rev1=1 crypto rx=0x7fb26401ad10 tx=0x7fb26401d000 comp rx=0 tx=0).stop 2024-02-21T23:13:41.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.440+0000 7fb276042740 1 -- 172.21.15.28:0/226879099 shutdown_connections 2024-02-21T23:13:41.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.440+0000 7fb276042740 1 --2- 172.21.15.28:0/226879099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a33ff556b0 0x55a33ff54dd0 unknown :-1 s=CLOSED pgs=2406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:41.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.440+0000 7fb276042740 1 --2- 172.21.15.28:0/226879099 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a33ff3a650 0x55a33ff54890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:41.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.440+0000 7fb276042740 1 --2- 172.21.15.28:0/226879099 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55a33ff1a330 0x55a33ff5c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:41.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.440+0000 7fb276042740 1 -- 172.21.15.28:0/226879099 >> 172.21.15.28:0/226879099 conn(0x55a33ff19020 msgr2=0x55a33ff64be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:41.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.440+0000 7fb276042740 1 -- 172.21.15.28:0/226879099 shutdown_connections 2024-02-21T23:13:41.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.440+0000 7fb276042740 1 -- 172.21.15.28:0/226879099 wait complete. 2024-02-21T23:13:41.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.441+0000 7fb276042740 1 -- start start 2024-02-21T23:13:41.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.441+0000 7fb276042740 1 Processor -- start 2024-02-21T23:13:41.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.442+0000 7fb276042740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a33ff1a330 0x55a33ffc8410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:41.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.442+0000 7fb276042740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a33ff3a650 0x55a33ff54c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:41.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.442+0000 7fb276042740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55a33ff556b0 0x55a33ffc16e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:41.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.442+0000 7fb276042740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a33ff39b40 con 0x55a33ff1a330 2024-02-21T23:13:41.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.442+0000 7fb276042740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a33ff59ed0 con 0x55a33ff3a650 2024-02-21T23:13:41.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.442+0000 7fb276042740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a33ff65840 con 0x55a33ff556b0 2024-02-21T23:13:41.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.442+0000 7fb276042740 10 client.0 _ll_register_callbacks cb 0x7fb25c0548b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:13:41.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.442+0000 7fb26be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a33ff1a330 0x55a33ffc8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:41.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.442+0000 7fb26be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a33ff1a330 0x55a33ffc8410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39546/0 (socket says 172.21.15.28:39546) 2024-02-21T23:13:41.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.442+0000 7fb26be00640 1 -- 172.21.15.28:0/4029419460 learned_addr learned my addr 172.21.15.28:0/4029419460 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:41.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.443+0000 7fb26b400640 1 --2- 172.21.15.28:0/4029419460 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a33ff3a650 0x55a33ff54c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:41.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.443+0000 7fb271200640 1 --2- 172.21.15.28:0/4029419460 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55a33ff556b0 0x55a33ffc16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:41.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.443+0000 7fb26be00640 1 -- 172.21.15.28:0/4029419460 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55a33ff556b0 msgr2=0x55a33ffc16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:41.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.443+0000 7fb26be00640 1 --2- 172.21.15.28:0/4029419460 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55a33ff556b0 0x55a33ffc16e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:41.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.443+0000 7fb26be00640 1 -- 172.21.15.28:0/4029419460 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a33ff3a650 msgr2=0x55a33ff54c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:41.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.443+0000 7fb26be00640 1 --2- 172.21.15.28:0/4029419460 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a33ff3a650 0x55a33ff54c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:41.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.443+0000 7fb26be00640 1 -- 172.21.15.28:0/4029419460 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a33ff36a50 con 0x55a33ff1a330 2024-02-21T23:13:41.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.443+0000 7fb26b400640 1 --2- 172.21.15.28:0/4029419460 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a33ff3a650 0x55a33ff54c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:13:41.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.443+0000 7fb271200640 1 --2- 172.21.15.28:0/4029419460 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55a33ff556b0 0x55a33ffc16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:13:41.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.443+0000 7fb26be00640 1 --2- 172.21.15.28:0/4029419460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a33ff1a330 0x55a33ffc8410 secure :-1 s=READY pgs=2407 cs=0 l=1 rev1=1 crypto rx=0x7fb25c056b40 tx=0x7fb25c088290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3607262946d3357 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:41.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.443+0000 7fb255600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:13:41.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.443+0000 7fb255600640 1 -- 172.21.15.28:0/4029419460 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb25c058150 con 0x55a33ff1a330 2024-02-21T23:13:41.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.443+0000 7fb255600640 1 -- 172.21.15.28:0/4029419460 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(208..208 src has 1..208) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb25c0770f0 con 0x55a33ff1a330 2024-02-21T23:13:41.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.443+0000 7fb276042740 1 -- 172.21.15.28:0/4029419460 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb26402a040 con 0x55a33ff1a330 2024-02-21T23:13:41.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.444+0000 7fb255600640 1 -- 172.21.15.28:0/4029419460 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=232) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb25c07d050 con 0x55a33ff1a330 2024-02-21T23:13:41.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.444+0000 7fb276042740 10 client.10965 fetch_fsmap learned FSMap version 232 2024-02-21T23:13:41.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.444+0000 7fb276042740 1 -- 172.21.15.28:0/4029419460 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=232,monmap=2+}) v3 -- 0x55a33ff5ba00 con 0x55a33ff1a330 2024-02-21T23:13:41.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.444+0000 7fb255600640 1 -- 172.21.15.28:0/4029419460 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb26402a040 con 0x55a33ff1a330 2024-02-21T23:13:41.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.444+0000 7fb255600640 1 -- 172.21.15.28:0/4029419460 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 232) v1 ==== 3306+0+0 (secure 0 0 0) 0x7fb25c080020 con 0x55a33ff1a330 2024-02-21T23:13:41.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.444+0000 7fb276042740 10 client.10965 fetch_fsmap finished waiting for FSMap version 232 2024-02-21T23:13:41.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.444+0000 7fb276042740 10 client.10965 resolve_mds: resolved name 'a' to daemon mds.a 2024-02-21T23:13:41.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.444+0000 7fb276042740 20 client.10965 populate_metadata read hostname 'smithi028' 2024-02-21T23:13:41.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.444+0000 7fb276042740 1 --2- 172.21.15.28:0/4029419460 >> [v2:172.21.15.28:6832/1864206570,v1:172.21.15.28:6833/1864206570] conn(0x55a33ffc1eb0 0x55a33ff56720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:41.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.444+0000 7fb276042740 4 client.10965 mds_command: new command op to 8290 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-21T23:13:41.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.444+0000 7fb276042740 1 -- 172.21.15.28:0/4029419460 --> [v2:172.21.15.28:6832/1864206570,v1:172.21.15.28:6833/1864206570] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55a33fe84d00 con 0x55a33ffc1eb0 2024-02-21T23:13:41.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.444+0000 7fb26b400640 1 --2- 172.21.15.28:0/4029419460 >> [v2:172.21.15.28:6832/1864206570,v1:172.21.15.28:6833/1864206570] conn(0x55a33ffc1eb0 0x55a33ff56720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:41.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.445+0000 7fb26b400640 1 --2- 172.21.15.28:0/4029419460 >> [v2:172.21.15.28:6832/1864206570,v1:172.21.15.28:6833/1864206570] conn(0x55a33ffc1eb0 0x55a33ff56720 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3914729f76917e74 server_cookie=7da32604cd9f5590 in_seq=0 out_seq=0 2024-02-21T23:13:41.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.445+0000 7fb255600640 10 client.10965 ms_handle_connect on v2:172.21.15.28:6832/1864206570 2024-02-21T23:13:41.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.446+0000 7fb255600640 1 -- 172.21.15.28:0/4029419460 <== mds.1 v2:172.21.15.28:6832/1864206570 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+328 (crc 0 0 0) 0x55a33fe84d00 con 0x55a33ffc1eb0 2024-02-21T23:13:41.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.446+0000 7fb255600640 10 client.10965 handle_command_reply: tid=0 2024-02-21T23:13:41.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.446+0000 7fb276042740 1 client.10965 shutdown 2024-02-21T23:13:41.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.446+0000 7fb276042740 1 -- 172.21.15.28:0/4029419460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a33ff1a330 msgr2=0x55a33ffc8410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:41.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.446+0000 7fb276042740 1 --2- 172.21.15.28:0/4029419460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a33ff1a330 0x55a33ffc8410 secure :-1 s=READY pgs=2407 cs=0 l=1 rev1=1 crypto rx=0x7fb25c056b40 tx=0x7fb25c088290 comp rx=0 tx=0).stop 2024-02-21T23:13:41.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.446+0000 7fb276042740 1 -- 172.21.15.28:0/4029419460 shutdown_connections 2024-02-21T23:13:41.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.446+0000 7fb276042740 1 --2- 172.21.15.28:0/4029419460 >> [v2:172.21.15.28:6832/1864206570,v1:172.21.15.28:6833/1864206570] conn(0x55a33ffc1eb0 0x55a33ff56720 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:41.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.446+0000 7fb276042740 1 --2- 172.21.15.28:0/4029419460 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x55a33ff556b0 0x55a33ffc16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:41.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.446+0000 7fb276042740 1 --2- 172.21.15.28:0/4029419460 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55a33ff3a650 0x55a33ff54c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:41.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.446+0000 7fb276042740 1 --2- 172.21.15.28:0/4029419460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a33ff1a330 0x55a33ffc8410 unknown :-1 s=CLOSED pgs=2407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:41.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.446+0000 7fb255600640 0 client.10965 ms_handle_reset on v2:172.21.15.28:6832/1864206570 2024-02-21T23:13:41.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.446+0000 7fb276042740 1 -- 172.21.15.28:0/4029419460 >> 172.21.15.28:0/4029419460 conn(0x55a33ff19020 msgr2=0x55a33ff0ed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:41.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.446+0000 7fb276042740 1 -- 172.21.15.28:0/4029419460 shutdown_connections 2024-02-21T23:13:41.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.446+0000 7fb276042740 1 -- 172.21.15.28:0/4029419460 wait complete. 2024-02-21T23:13:41.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.446+0000 7fb276042740 20 client.10965 trim_cache size 0 max 16384 2024-02-21T23:13:41.440 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-21T23:13:41.440 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac", 2024-02-21T23:13:41.440 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 1, 2024-02-21T23:13:41.440 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 8290, 2024-02-21T23:13:41.440 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-02-21T23:13:41.440 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-02-21T23:13:41.440 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-02-21T23:13:41.441 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 3.1325921170000002, 2024-02-21T23:13:41.441 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 232, 2024-02-21T23:13:41.441 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 208, 2024-02-21T23:13:41.441 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 208, 2024-02-21T23:13:41.441 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 17.685671142 2024-02-21T23:13:41.441 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-21T23:13:41.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.447+0000 7fb271c00640 1 -- 172.21.15.28:0/890258460 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb24c06b4d0 msgr2=0x7fb24c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:41.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.447+0000 7fb271c00640 1 --2- 172.21.15.28:0/890258460 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb24c06b4d0 0x7fb24c06d990 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fb264009fd0 tx=0x7fb264012000 comp rx=0 tx=0).stop 2024-02-21T23:13:41.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.447+0000 7fb271c00640 1 -- 172.21.15.28:0/890258460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb26c0aaec0 msgr2=0x7fb26c0a80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:41.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.447+0000 7fb271c00640 1 --2- 172.21.15.28:0/890258460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb26c0aaec0 0x7fb26c0a80a0 secure :-1 s=READY pgs=2403 cs=0 l=1 rev1=1 crypto rx=0x7fb25c00c480 tx=0x7fb25c00c940 comp rx=0 tx=0).stop 2024-02-21T23:13:41.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.447+0000 7fb271c00640 1 -- 172.21.15.28:0/890258460 shutdown_connections 2024-02-21T23:13:41.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.447+0000 7fb271c00640 1 --2- 172.21.15.28:0/890258460 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb24c06b4d0 0x7fb24c06d990 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:41.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.447+0000 7fb271c00640 1 --2- 172.21.15.28:0/890258460 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb26c0c1a50 0x7fb26c14fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:41.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.447+0000 7fb271c00640 1 --2- 172.21.15.28:0/890258460 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb26c0c29b0 0x7fb26c070460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:41.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.447+0000 7fb271c00640 1 --2- 172.21.15.28:0/890258460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb26c0aaec0 0x7fb26c0a80a0 unknown :-1 s=CLOSED pgs=2403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:41.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.447+0000 7fb271c00640 1 -- 172.21.15.28:0/890258460 >> 172.21.15.28:0/890258460 conn(0x7fb26c0aa3d0 msgr2=0x7fb26c150b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:41.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.447+0000 7fb271c00640 1 -- 172.21.15.28:0/890258460 shutdown_connections 2024-02-21T23:13:41.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.447+0000 7fb271c00640 1 -- 172.21.15.28:0/890258460 wait complete. 2024-02-21T23:13:41.459 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-02-21T23:13:41.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.669+0000 7f5e26c00640 1 Processor -- start 2024-02-21T23:13:41.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.670+0000 7f5e26c00640 1 -- start start 2024-02-21T23:13:41.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.671+0000 7f5e26c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5e200c6a00 0x7f5e200c6de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:41.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.671+0000 7f5e26c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f5e200bffd0 0x7f5e20154f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:41.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.671+0000 7f5e26c00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e20155700 con 0x7f5e200bf950 2024-02-21T23:13:41.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.671+0000 7f5e26c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e200c5030 con 0x7f5e200c6a00 2024-02-21T23:13:41.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.671+0000 7f5e26c00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e200b0b10 con 0x7f5e200bffd0 2024-02-21T23:13:41.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.672+0000 7f5e1fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5e200c6a00 0x7f5e200c6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:41.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.672+0000 7f5e1fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5e200c6a00 0x7f5e200c6de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39548/0 (socket says 172.21.15.28:39548) 2024-02-21T23:13:41.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.672+0000 7f5e1fe00640 1 -- 172.21.15.28:0/3661978889 learned_addr learned my addr 172.21.15.28:0/3661978889 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:41.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.672+0000 7f5e17400640 1 --2- 172.21.15.28:0/3661978889 >> v2:172.21.15.81:3300/0 conn(0x7f5e200bffd0 0x7f5e20154f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:41.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.672+0000 7f5e1fe00640 1 -- 172.21.15.28:0/3661978889 >> v2:172.21.15.81:3300/0 conn(0x7f5e200bffd0 msgr2=0x7f5e20154f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:41.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.672+0000 7f5e1fe00640 1 --2- 172.21.15.28:0/3661978889 >> v2:172.21.15.81:3300/0 conn(0x7f5e200bffd0 0x7f5e20154f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:41.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.672+0000 7f5e1fe00640 1 -- 172.21.15.28:0/3661978889 >> v1:172.21.15.81:6789/0 conn(0x7f5e200bf950 legacy=0x7f5e20158c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:41.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.672+0000 7f5e1fe00640 1 -- 172.21.15.28:0/3661978889 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e201559e0 con 0x7f5e200c6a00 2024-02-21T23:13:41.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.673+0000 7f5e1fe00640 1 --2- 172.21.15.28:0/3661978889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e200c6a00 0x7f5e200c6de0 secure :-1 s=READY pgs=2408 cs=0 l=1 rev1=1 crypto rx=0x7f5e2008c8e0 tx=0x7f5e1002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eaf8d7b94255c9a2 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:41.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.673+0000 7f5e1f400640 1 -- 172.21.15.28:0/3661978889 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e1002c9f0 con 0x7f5e200c6a00 2024-02-21T23:13:41.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.674+0000 7f5e1f400640 1 -- 172.21.15.28:0/3661978889 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e1002cb90 con 0x7f5e200c6a00 2024-02-21T23:13:41.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.674+0000 7f5e1f400640 1 -- 172.21.15.28:0/3661978889 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e100343f0 con 0x7f5e200c6a00 2024-02-21T23:13:41.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.674+0000 7f5e26c00640 1 -- 172.21.15.28:0/3661978889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e200c6a00 msgr2=0x7f5e200c6de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:41.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.674+0000 7f5e26c00640 1 --2- 172.21.15.28:0/3661978889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e200c6a00 0x7f5e200c6de0 secure :-1 s=READY pgs=2408 cs=0 l=1 rev1=1 crypto rx=0x7f5e2008c8e0 tx=0x7f5e1002be50 comp rx=0 tx=0).stop 2024-02-21T23:13:41.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.674+0000 7f5e26c00640 1 -- 172.21.15.28:0/3661978889 shutdown_connections 2024-02-21T23:13:41.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.674+0000 7f5e26c00640 1 --2- 172.21.15.28:0/3661978889 >> v2:172.21.15.81:3300/0 conn(0x7f5e200bffd0 0x7f5e20154f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:41.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.674+0000 7f5e26c00640 1 --2- 172.21.15.28:0/3661978889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e200c6a00 0x7f5e200c6de0 unknown :-1 s=CLOSED pgs=2408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:41.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.674+0000 7f5e26c00640 1 -- 172.21.15.28:0/3661978889 >> 172.21.15.28:0/3661978889 conn(0x7f5e2006d580 msgr2=0x7f5e2006d990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:41.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.674+0000 7f5e26c00640 1 -- 172.21.15.28:0/3661978889 shutdown_connections 2024-02-21T23:13:41.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.674+0000 7f5e26c00640 1 -- 172.21.15.28:0/3661978889 wait complete. 2024-02-21T23:13:41.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.677+0000 7f5e26c00640 1 Processor -- start 2024-02-21T23:13:41.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.677+0000 7f5e26c00640 1 -- start start 2024-02-21T23:13:41.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.679+0000 7f5e26c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e200bf950 0x7f5e201457c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:41.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.679+0000 7f5e26c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5e200bffd0 0x7f5e201deb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:41.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.679+0000 7f5e26c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e200c6a00 0x7f5e201eb1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:41.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.679+0000 7f5e26c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e20158720 con 0x7f5e200bf950 2024-02-21T23:13:41.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.679+0000 7f5e26c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e20159350 con 0x7f5e200c6a00 2024-02-21T23:13:41.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.679+0000 7f5e26c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e201594d0 con 0x7f5e200bffd0 2024-02-21T23:13:41.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.679+0000 7f5e1fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e200bf950 0x7f5e201457c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:41.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.679+0000 7f5e1fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e200bf950 0x7f5e201457c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39564/0 (socket says 172.21.15.28:39564) 2024-02-21T23:13:41.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.679+0000 7f5e1fe00640 1 -- 172.21.15.28:0/2197188734 learned_addr learned my addr 172.21.15.28:0/2197188734 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:41.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.679+0000 7f5e1fe00640 1 -- 172.21.15.28:0/2197188734 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5e200bffd0 msgr2=0x7f5e201deb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:41.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.679+0000 7f5e17400640 1 --2- 172.21.15.28:0/2197188734 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5e200bffd0 0x7f5e201deb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:41.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.679+0000 7f5e24c00640 1 --2- 172.21.15.28:0/2197188734 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e200c6a00 0x7f5e201eb1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:41.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.679+0000 7f5e1fe00640 1 --2- 172.21.15.28:0/2197188734 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5e200bffd0 0x7f5e201deb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:41.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.679+0000 7f5e1fe00640 1 -- 172.21.15.28:0/2197188734 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e200c6a00 msgr2=0x7f5e201eb1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:41.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.679+0000 7f5e1fe00640 1 --2- 172.21.15.28:0/2197188734 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e200c6a00 0x7f5e201eb1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:41.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.679+0000 7f5e1fe00640 1 -- 172.21.15.28:0/2197188734 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e20154de0 con 0x7f5e200bf950 2024-02-21T23:13:41.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.679+0000 7f5e24c00640 1 --2- 172.21.15.28:0/2197188734 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e200c6a00 0x7f5e201eb1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:13:41.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.680+0000 7f5e1fe00640 1 --2- 172.21.15.28:0/2197188734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e200bf950 0x7f5e201457c0 secure :-1 s=READY pgs=2409 cs=0 l=1 rev1=1 crypto rx=0x7f5e10036040 tx=0x7f5e10004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:41.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.680+0000 7f5e1cc00640 1 -- 172.21.15.28:0/2197188734 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e10043020 con 0x7f5e200bf950 2024-02-21T23:13:41.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.680+0000 7f5e1cc00640 1 -- 172.21.15.28:0/2197188734 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e1003e450 con 0x7f5e200bf950 2024-02-21T23:13:41.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.680+0000 7f5e1cc00640 1 -- 172.21.15.28:0/2197188734 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e10009510 con 0x7f5e200bf950 2024-02-21T23:13:41.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.680+0000 7f5e26c00640 1 -- 172.21.15.28:0/2197188734 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e201e0c90 con 0x7f5e200bf950 2024-02-21T23:13:41.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.680+0000 7f5e26c00640 1 -- 172.21.15.28:0/2197188734 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e201e1250 con 0x7f5e200bf950 2024-02-21T23:13:41.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.682+0000 7f5e1cc00640 1 -- 172.21.15.28:0/2197188734 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5e10046020 con 0x7f5e200bf950 2024-02-21T23:13:41.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.683+0000 7f5e1cc00640 1 --2- 172.21.15.28:0/2197188734 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5dfc06b4d0 0x7f5dfc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:41.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.684+0000 7f5e1cc00640 1 -- 172.21.15.28:0/2197188734 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5e100ace30 con 0x7f5e200bf950 2024-02-21T23:13:41.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.684+0000 7f5e29bca740 1 Processor -- start 2024-02-21T23:13:41.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.684+0000 7f5e17400640 1 --2- 172.21.15.28:0/2197188734 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5dfc06b4d0 0x7f5dfc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:41.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.684+0000 7f5e29bca740 1 -- start start 2024-02-21T23:13:41.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.685+0000 7f5e29bca740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557bdd366e40 0x557bdd367220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:41.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.685+0000 7f5e29bca740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557bdd368150 0x557bdd3a2f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:41.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.685+0000 7f5e29bca740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x557bdd3a34d0 0x557bdd3a1540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:41.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.685+0000 7f5e29bca740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x557bdd2cb470 con 0x557bdd368150 2024-02-21T23:13:41.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.685+0000 7f5e29bca740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x557bdd274af0 con 0x557bdd366e40 2024-02-21T23:13:41.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.685+0000 7f5e29bca740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x557bdd2c7510 con 0x557bdd3a34d0 2024-02-21T23:13:41.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.686+0000 7f5e17400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557bdd368150 0x557bdd3a2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:41.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.686+0000 7f5e17400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557bdd368150 0x557bdd3a2f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39566/0 (socket says 172.21.15.28:39566) 2024-02-21T23:13:41.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.686+0000 7f5e17400640 1 -- 172.21.15.28:0/2831089591 learned_addr learned my addr 172.21.15.28:0/2831089591 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:41.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.686+0000 7f5e1fe00640 1 --2- 172.21.15.28:0/2831089591 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557bdd366e40 0x557bdd367220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:41.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.686+0000 7f5e17400640 1 --2- 172.21.15.28:0/2197188734 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5dfc06b4d0 0x7f5dfc06d990 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f5e08006fd0 tx=0x7f5e08008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:41.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.686+0000 7f5e24c00640 1 --2- 172.21.15.28:0/2831089591 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x557bdd3a34d0 0x557bdd3a1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:41.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.686+0000 7f5e17400640 1 -- 172.21.15.28:0/2831089591 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x557bdd3a34d0 msgr2=0x557bdd3a1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:41.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.686+0000 7f5e17400640 1 --2- 172.21.15.28:0/2831089591 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x557bdd3a34d0 0x557bdd3a1540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:41.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.686+0000 7f5e17400640 1 -- 172.21.15.28:0/2831089591 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557bdd366e40 msgr2=0x557bdd367220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:41.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.686+0000 7f5e17400640 1 --2- 172.21.15.28:0/2831089591 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557bdd366e40 0x557bdd367220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:41.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.686+0000 7f5e17400640 1 -- 172.21.15.28:0/2831089591 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557bdd27bd90 con 0x557bdd368150 2024-02-21T23:13:41.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.686+0000 7f5e24c00640 1 --2- 172.21.15.28:0/2831089591 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x557bdd3a34d0 0x557bdd3a1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:13:41.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.687+0000 7f5e17400640 1 --2- 172.21.15.28:0/2831089591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557bdd368150 0x557bdd3a2f90 secure :-1 s=READY pgs=2410 cs=0 l=1 rev1=1 crypto rx=0x7f5e0800dbc0 tx=0x7f5e08010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15fdc17e355e0247 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:41.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.687+0000 7f5e0fe00640 1 -- 172.21.15.28:0/2831089591 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e08023070 con 0x557bdd368150 2024-02-21T23:13:41.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.687+0000 7f5e0fe00640 1 -- 172.21.15.28:0/2831089591 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e08017070 con 0x557bdd368150 2024-02-21T23:13:41.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.687+0000 7f5e0fe00640 1 -- 172.21.15.28:0/2831089591 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e08019050 con 0x557bdd368150 2024-02-21T23:13:41.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.688+0000 7f5e29bca740 1 -- 172.21.15.28:0/2831089591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557bdd368150 msgr2=0x557bdd3a2f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:41.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.688+0000 7f5e29bca740 1 --2- 172.21.15.28:0/2831089591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557bdd368150 0x557bdd3a2f90 secure :-1 s=READY pgs=2410 cs=0 l=1 rev1=1 crypto rx=0x7f5e0800dbc0 tx=0x7f5e08010000 comp rx=0 tx=0).stop 2024-02-21T23:13:41.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.688+0000 7f5e29bca740 1 -- 172.21.15.28:0/2831089591 shutdown_connections 2024-02-21T23:13:41.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.688+0000 7f5e29bca740 1 --2- 172.21.15.28:0/2831089591 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x557bdd3a34d0 0x557bdd3a1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:41.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.688+0000 7f5e29bca740 1 --2- 172.21.15.28:0/2831089591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557bdd368150 0x557bdd3a2f90 unknown :-1 s=CLOSED pgs=2410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:41.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.688+0000 7f5e29bca740 1 --2- 172.21.15.28:0/2831089591 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557bdd366e40 0x557bdd367220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:41.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.688+0000 7f5e29bca740 1 -- 172.21.15.28:0/2831089591 >> 172.21.15.28:0/2831089591 conn(0x557bdd363790 msgr2=0x557bdd3881a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:41.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.688+0000 7f5e29bca740 1 -- 172.21.15.28:0/2831089591 shutdown_connections 2024-02-21T23:13:41.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.688+0000 7f5e29bca740 1 -- 172.21.15.28:0/2831089591 wait complete. 2024-02-21T23:13:41.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.689+0000 7f5e29bca740 1 -- start start 2024-02-21T23:13:41.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.690+0000 7f5e29bca740 1 Processor -- start 2024-02-21T23:13:41.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.691+0000 7f5e29bca740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x557bdd368150 0x557bdd387e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:41.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.691+0000 7f5e29bca740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557bdd3a34d0 0x557bdd3b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:41.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.691+0000 7f5e29bca740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557bdd386f00 0x557bdd387360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:41.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.691+0000 7f5e29bca740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x557bdd26a040 con 0x557bdd386f00 2024-02-21T23:13:41.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.691+0000 7f5e29bca740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x557bdd3888b0 con 0x557bdd3a34d0 2024-02-21T23:13:41.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.691+0000 7f5e29bca740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x557bdd39e170 con 0x557bdd368150 2024-02-21T23:13:41.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.691+0000 7f5e29bca740 10 client.0 _ll_register_callbacks cb 0x557bdd287920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:13:41.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.692+0000 7f5e24c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557bdd386f00 0x557bdd387360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:41.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.692+0000 7f5e24c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557bdd386f00 0x557bdd387360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39574/0 (socket says 172.21.15.28:39574) 2024-02-21T23:13:41.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.692+0000 7f5e24c00640 1 -- 172.21.15.28:0/3012873285 learned_addr learned my addr 172.21.15.28:0/3012873285 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:41.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.692+0000 7f5e1fe00640 1 --2- 172.21.15.28:0/3012873285 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x557bdd368150 0x557bdd387e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:41.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.692+0000 7f5e17400640 1 --2- 172.21.15.28:0/3012873285 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557bdd3a34d0 0x557bdd3b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:41.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.692+0000 7f5e24c00640 1 -- 172.21.15.28:0/3012873285 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x557bdd368150 msgr2=0x557bdd387e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:41.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.692+0000 7f5e24c00640 1 --2- 172.21.15.28:0/3012873285 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x557bdd368150 0x557bdd387e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:41.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.692+0000 7f5e24c00640 1 -- 172.21.15.28:0/3012873285 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557bdd3a34d0 msgr2=0x557bdd3b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:41.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.692+0000 7f5e24c00640 1 --2- 172.21.15.28:0/3012873285 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557bdd3a34d0 0x557bdd3b37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:41.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.692+0000 7f5e24c00640 1 -- 172.21.15.28:0/3012873285 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557bdd289aa0 con 0x557bdd386f00 2024-02-21T23:13:41.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.692+0000 7f5e24c00640 1 --2- 172.21.15.28:0/3012873285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557bdd386f00 0x557bdd387360 secure :-1 s=READY pgs=2411 cs=0 l=1 rev1=1 crypto rx=0x7f5e1800f200 tx=0x7f5e1800f6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8a558b6dc34c906 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:41.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.693+0000 7f5e0cc00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:13:41.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.693+0000 7f5e0cc00640 1 -- 172.21.15.28:0/3012873285 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e180050d0 con 0x557bdd386f00 2024-02-21T23:13:41.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.693+0000 7f5e0cc00640 1 -- 172.21.15.28:0/3012873285 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(208..208 src has 1..208) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5e18005270 con 0x557bdd386f00 2024-02-21T23:13:41.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.693+0000 7f5e29bca740 1 -- 172.21.15.28:0/3012873285 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5e08017070 con 0x557bdd386f00 2024-02-21T23:13:41.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.693+0000 7f5e0cc00640 1 -- 172.21.15.28:0/3012873285 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=232) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5e180035f0 con 0x557bdd386f00 2024-02-21T23:13:41.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.693+0000 7f5e29bca740 10 client.10977 fetch_fsmap learned FSMap version 232 2024-02-21T23:13:41.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.693+0000 7f5e29bca740 1 -- 172.21.15.28:0/3012873285 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=232,monmap=2+}) v3 -- 0x557bdcfe02d0 con 0x557bdd386f00 2024-02-21T23:13:41.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.694+0000 7f5e0cc00640 1 -- 172.21.15.28:0/3012873285 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e08017070 con 0x557bdd386f00 2024-02-21T23:13:41.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.694+0000 7f5e0cc00640 1 -- 172.21.15.28:0/3012873285 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 232) v1 ==== 3306+0+0 (secure 0 0 0) 0x7f5e18023020 con 0x557bdd386f00 2024-02-21T23:13:41.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.694+0000 7f5e29bca740 10 client.10977 fetch_fsmap finished waiting for FSMap version 232 2024-02-21T23:13:41.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.694+0000 7f5e29bca740 10 client.10977 resolve_mds: resolved name 'c' to daemon mds.c 2024-02-21T23:13:41.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.694+0000 7f5e29bca740 20 client.10977 populate_metadata read hostname 'smithi028' 2024-02-21T23:13:41.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.694+0000 7f5e29bca740 1 --2- 172.21.15.28:0/3012873285 >> [v2:172.21.15.28:6834/3142544451,v1:172.21.15.28:6835/3142544451] conn(0x557bdd3b7180 0x557bdd3c1f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:41.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.694+0000 7f5e29bca740 4 client.10977 mds_command: new command op to 8296 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-21T23:13:41.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.694+0000 7f5e29bca740 1 -- 172.21.15.28:0/3012873285 --> [v2:172.21.15.28:6834/3142544451,v1:172.21.15.28:6835/3142544451] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557bdd2d89b0 con 0x557bdd3b7180 2024-02-21T23:13:41.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.694+0000 7f5e1fe00640 1 --2- 172.21.15.28:0/3012873285 >> [v2:172.21.15.28:6834/3142544451,v1:172.21.15.28:6835/3142544451] conn(0x557bdd3b7180 0x557bdd3c1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:41.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.695+0000 7f5e0cc00640 10 client.10977 ms_handle_connect on v2:172.21.15.28:6834/3142544451 2024-02-21T23:13:41.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.695+0000 7f5e1fe00640 1 --2- 172.21.15.28:0/3012873285 >> [v2:172.21.15.28:6834/3142544451,v1:172.21.15.28:6835/3142544451] conn(0x557bdd3b7180 0x557bdd3c1f90 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ded4f29061b18454 server_cookie=35ab8d64ccf2488c in_seq=0 out_seq=0 2024-02-21T23:13:41.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.695+0000 7f5e0cc00640 1 -- 172.21.15.28:0/3012873285 <== mds.0 v2:172.21.15.28:6834/3142544451 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x557bdd2d89b0 con 0x557bdd3b7180 2024-02-21T23:13:41.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.695+0000 7f5e0cc00640 10 client.10977 handle_command_reply: tid=0 2024-02-21T23:13:41.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.696+0000 7f5e29bca740 1 client.10977 shutdown 2024-02-21T23:13:41.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.696+0000 7f5e29bca740 1 -- 172.21.15.28:0/3012873285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557bdd386f00 msgr2=0x557bdd387360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:41.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.696+0000 7f5e29bca740 1 --2- 172.21.15.28:0/3012873285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557bdd386f00 0x557bdd387360 secure :-1 s=READY pgs=2411 cs=0 l=1 rev1=1 crypto rx=0x7f5e1800f200 tx=0x7f5e1800f6c0 comp rx=0 tx=0).stop 2024-02-21T23:13:41.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.696+0000 7f5e29bca740 1 -- 172.21.15.28:0/3012873285 shutdown_connections 2024-02-21T23:13:41.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.696+0000 7f5e29bca740 1 --2- 172.21.15.28:0/3012873285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557bdd386f00 0x557bdd387360 unknown :-1 s=CLOSED pgs=2411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:41.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.696+0000 7f5e29bca740 1 --2- 172.21.15.28:0/3012873285 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557bdd3a34d0 0x557bdd3b37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:41.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.696+0000 7f5e29bca740 1 --2- 172.21.15.28:0/3012873285 >> [v2:172.21.15.28:6834/3142544451,v1:172.21.15.28:6835/3142544451] conn(0x557bdd3b7180 0x557bdd3c1f90 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:41.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.696+0000 7f5e29bca740 1 --2- 172.21.15.28:0/3012873285 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x557bdd368150 0x557bdd387e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:41.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.696+0000 7f5e29bca740 1 -- 172.21.15.28:0/3012873285 >> 172.21.15.28:0/3012873285 conn(0x557bdd366e40 msgr2=0x557bdd3605f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:41.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.696+0000 7f5e0cc00640 0 client.10977 ms_handle_reset on v2:172.21.15.28:6834/3142544451 2024-02-21T23:13:41.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.696+0000 7f5e29bca740 1 -- 172.21.15.28:0/3012873285 shutdown_connections 2024-02-21T23:13:41.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.696+0000 7f5e29bca740 1 -- 172.21.15.28:0/3012873285 wait complete. 2024-02-21T23:13:41.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.696+0000 7f5e29bca740 20 client.10977 trim_cache size 0 max 16384 2024-02-21T23:13:41.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.715+0000 7f5e29bca740 1 Processor -- start 2024-02-21T23:13:41.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.715+0000 7f5e29bca740 1 -- start start 2024-02-21T23:13:41.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.716+0000 7f5e29bca740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x557bdd368150 0x557bdd387c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:41.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.716+0000 7f5e29bca740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557bdd386f00 0x557bdd3b0750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:41.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.716+0000 7f5e29bca740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557bdd3a34d0 0x557bdd3a2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:41.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.716+0000 7f5e29bca740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e201f7c40 con 0x557bdd3a34d0 2024-02-21T23:13:41.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.716+0000 7f5e29bca740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e201ebe40 con 0x557bdd386f00 2024-02-21T23:13:41.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.716+0000 7f5e29bca740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e201581c0 con 0x557bdd368150 2024-02-21T23:13:41.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.716+0000 7f5e24c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557bdd3a34d0 0x557bdd3a2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:41.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.716+0000 7f5e24c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557bdd3a34d0 0x557bdd3a2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39580/0 (socket says 172.21.15.28:39580) 2024-02-21T23:13:41.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.717+0000 7f5e24c00640 1 -- 172.21.15.28:0/334036823 learned_addr learned my addr 172.21.15.28:0/334036823 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:41.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.717+0000 7f5e1fe00640 1 --2- 172.21.15.28:0/334036823 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x557bdd368150 0x557bdd387c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:41.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.717+0000 7f5e17400640 1 --2- 172.21.15.28:0/334036823 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557bdd386f00 0x557bdd3b0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:41.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.717+0000 7f5e24c00640 1 -- 172.21.15.28:0/334036823 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x557bdd368150 msgr2=0x557bdd387c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:41.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.717+0000 7f5e24c00640 1 --2- 172.21.15.28:0/334036823 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x557bdd368150 0x557bdd387c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:41.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.717+0000 7f5e24c00640 1 -- 172.21.15.28:0/334036823 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557bdd386f00 msgr2=0x557bdd3b0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:41.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.717+0000 7f5e24c00640 1 --2- 172.21.15.28:0/334036823 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557bdd386f00 0x557bdd3b0750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:41.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.717+0000 7f5e24c00640 1 -- 172.21.15.28:0/334036823 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557bdd0aa6b0 con 0x557bdd3a34d0 2024-02-21T23:13:41.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.717+0000 7f5e17400640 1 --2- 172.21.15.28:0/334036823 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557bdd386f00 0x557bdd3b0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:13:41.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.717+0000 7f5e24c00640 1 --2- 172.21.15.28:0/334036823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557bdd3a34d0 0x557bdd3a2950 secure :-1 s=READY pgs=2412 cs=0 l=1 rev1=1 crypto rx=0x7f5e18007f20 tx=0x7f5e1800d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca48081b54fb88c8 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:41.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.718+0000 7f5e0f400640 1 -- 172.21.15.28:0/334036823 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e18005070 con 0x557bdd3a34d0 2024-02-21T23:13:41.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.718+0000 7f5e0f400640 1 -- 172.21.15.28:0/334036823 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e18028030 con 0x557bdd3a34d0 2024-02-21T23:13:41.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.718+0000 7f5e0f400640 1 -- 172.21.15.28:0/334036823 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e180039f0 con 0x557bdd3a34d0 2024-02-21T23:13:41.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.718+0000 7f5e29bca740 1 -- 172.21.15.28:0/334036823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557bdd3a34d0 msgr2=0x557bdd3a2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:41.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.718+0000 7f5e29bca740 1 --2- 172.21.15.28:0/334036823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557bdd3a34d0 0x557bdd3a2950 secure :-1 s=READY pgs=2412 cs=0 l=1 rev1=1 crypto rx=0x7f5e18007f20 tx=0x7f5e1800d000 comp rx=0 tx=0).stop 2024-02-21T23:13:41.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.718+0000 7f5e29bca740 1 -- 172.21.15.28:0/334036823 shutdown_connections 2024-02-21T23:13:41.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.718+0000 7f5e29bca740 1 --2- 172.21.15.28:0/334036823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557bdd3a34d0 0x557bdd3a2950 unknown :-1 s=CLOSED pgs=2412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:41.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.718+0000 7f5e29bca740 1 --2- 172.21.15.28:0/334036823 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557bdd386f00 0x557bdd3b0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:41.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.718+0000 7f5e29bca740 1 --2- 172.21.15.28:0/334036823 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x557bdd368150 0x557bdd387c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:41.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.718+0000 7f5e29bca740 1 -- 172.21.15.28:0/334036823 >> 172.21.15.28:0/334036823 conn(0x557bdd366e40 msgr2=0x557bdd3860e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:41.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.718+0000 7f5e29bca740 1 -- 172.21.15.28:0/334036823 shutdown_connections 2024-02-21T23:13:41.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.719+0000 7f5e29bca740 1 -- 172.21.15.28:0/334036823 wait complete. 2024-02-21T23:13:41.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.719+0000 7f5e29bca740 1 -- start start 2024-02-21T23:13:41.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.720+0000 7f5e29bca740 1 Processor -- start 2024-02-21T23:13:41.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.721+0000 7f5e29bca740 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x557bdd368150 0x557bdd3a2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:41.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.721+0000 7f5e29bca740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557bdd386f00 0x557bdd4162a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:41.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.721+0000 7f5e29bca740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557bdd3a34d0 0x557bdd40ff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:41.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.721+0000 7f5e29bca740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x557bdd387aa0 con 0x557bdd3a34d0 2024-02-21T23:13:41.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.721+0000 7f5e29bca740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x557bdd39a000 con 0x557bdd386f00 2024-02-21T23:13:41.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.721+0000 7f5e29bca740 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x557bdd383760 con 0x557bdd368150 2024-02-21T23:13:41.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.721+0000 7f5e29bca740 10 client.0 _ll_register_callbacks cb 0x7f5e18001a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-21T23:13:41.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.722+0000 7f5e24c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557bdd3a34d0 0x557bdd40ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:41.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.722+0000 7f5e24c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557bdd3a34d0 0x557bdd40ff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39582/0 (socket says 172.21.15.28:39582) 2024-02-21T23:13:41.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.722+0000 7f5e24c00640 1 -- 172.21.15.28:0/3199522310 learned_addr learned my addr 172.21.15.28:0/3199522310 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:41.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.722+0000 7f5e17400640 1 --2- 172.21.15.28:0/3199522310 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557bdd386f00 0x557bdd4162a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:41.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.722+0000 7f5e1fe00640 1 --2- 172.21.15.28:0/3199522310 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x557bdd368150 0x557bdd3a2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:41.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.722+0000 7f5e24c00640 1 -- 172.21.15.28:0/3199522310 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x557bdd368150 msgr2=0x557bdd3a2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:41.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.722+0000 7f5e24c00640 1 --2- 172.21.15.28:0/3199522310 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x557bdd368150 0x557bdd3a2cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:41.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.722+0000 7f5e24c00640 1 -- 172.21.15.28:0/3199522310 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557bdd386f00 msgr2=0x557bdd4162a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:41.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.722+0000 7f5e24c00640 1 --2- 172.21.15.28:0/3199522310 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557bdd386f00 0x557bdd4162a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:41.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.722+0000 7f5e24c00640 1 -- 172.21.15.28:0/3199522310 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557bdd39fe10 con 0x557bdd3a34d0 2024-02-21T23:13:41.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.723+0000 7f5e24c00640 1 --2- 172.21.15.28:0/3199522310 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557bdd3a34d0 0x557bdd40ff40 secure :-1 s=READY pgs=2413 cs=0 l=1 rev1=1 crypto rx=0x7f5e18008ff0 tx=0x7f5e18001020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc3c2fdeb5e8c854 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:41.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.723+0000 7f5e0cc00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-02-21T23:13:41.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.723+0000 7f5e0cc00640 1 -- 172.21.15.28:0/3199522310 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e1803b060 con 0x557bdd3a34d0 2024-02-21T23:13:41.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.723+0000 7f5e0cc00640 1 -- 172.21.15.28:0/3199522310 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(208..208 src has 1..208) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5e18003790 con 0x557bdd3a34d0 2024-02-21T23:13:41.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.723+0000 7f5e29bca740 1 -- 172.21.15.28:0/3199522310 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5e18028030 con 0x557bdd3a34d0 2024-02-21T23:13:41.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.723+0000 7f5e0cc00640 1 -- 172.21.15.28:0/3199522310 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=232) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5e18041050 con 0x557bdd3a34d0 2024-02-21T23:13:41.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.723+0000 7f5e29bca740 10 client.10983 fetch_fsmap learned FSMap version 232 2024-02-21T23:13:41.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.723+0000 7f5e29bca740 1 -- 172.21.15.28:0/3199522310 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=232,monmap=2+}) v3 -- 0x557bdd4167e0 con 0x557bdd3a34d0 2024-02-21T23:13:41.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.724+0000 7f5e0cc00640 1 -- 172.21.15.28:0/3199522310 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e18028030 con 0x557bdd3a34d0 2024-02-21T23:13:41.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.724+0000 7f5e0cc00640 1 -- 172.21.15.28:0/3199522310 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 232) v1 ==== 3306+0+0 (secure 0 0 0) 0x7f5e18044020 con 0x557bdd3a34d0 2024-02-21T23:13:41.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.724+0000 7f5e29bca740 10 client.10983 fetch_fsmap finished waiting for FSMap version 232 2024-02-21T23:13:41.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.724+0000 7f5e29bca740 10 client.10983 resolve_mds: resolved name 'c' to daemon mds.c 2024-02-21T23:13:41.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.724+0000 7f5e29bca740 20 client.10983 populate_metadata read hostname 'smithi028' 2024-02-21T23:13:41.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.724+0000 7f5e29bca740 1 --2- 172.21.15.28:0/3199522310 >> [v2:172.21.15.28:6834/3142544451,v1:172.21.15.28:6835/3142544451] conn(0x557bdd361350 0x557bdd361730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:41.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.724+0000 7f5e29bca740 4 client.10983 mds_command: new command op to 8296 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-21T23:13:41.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.724+0000 7f5e29bca740 1 -- 172.21.15.28:0/3199522310 --> [v2:172.21.15.28:6834/3142544451,v1:172.21.15.28:6835/3142544451] -- command(tid 0: {"prefix": "status"}) v1 -- 0x557bdd2d2d00 con 0x557bdd361350 2024-02-21T23:13:41.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.724+0000 7f5e1fe00640 1 --2- 172.21.15.28:0/3199522310 >> [v2:172.21.15.28:6834/3142544451,v1:172.21.15.28:6835/3142544451] conn(0x557bdd361350 0x557bdd361730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:41.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.725+0000 7f5e1fe00640 1 --2- 172.21.15.28:0/3199522310 >> [v2:172.21.15.28:6834/3142544451,v1:172.21.15.28:6835/3142544451] conn(0x557bdd361350 0x557bdd361730 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=16bc0a04cb7bcd6b server_cookie=284fb386efe8a6ac in_seq=0 out_seq=0 2024-02-21T23:13:41.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.725+0000 7f5e0cc00640 10 client.10983 ms_handle_connect on v2:172.21.15.28:6834/3142544451 2024-02-21T23:13:41.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.725+0000 7f5e0cc00640 1 -- 172.21.15.28:0/3199522310 <== mds.0 v2:172.21.15.28:6834/3142544451 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+322 (crc 0 0 0) 0x557bdd2d2d00 con 0x557bdd361350 2024-02-21T23:13:41.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.725+0000 7f5e0cc00640 10 client.10983 handle_command_reply: tid=0 2024-02-21T23:13:41.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.725+0000 7f5e29bca740 1 client.10983 shutdown 2024-02-21T23:13:41.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.726+0000 7f5e29bca740 1 -- 172.21.15.28:0/3199522310 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557bdd3a34d0 msgr2=0x557bdd40ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:41.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.726+0000 7f5e29bca740 1 --2- 172.21.15.28:0/3199522310 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557bdd3a34d0 0x557bdd40ff40 secure :-1 s=READY pgs=2413 cs=0 l=1 rev1=1 crypto rx=0x7f5e18008ff0 tx=0x7f5e18001020 comp rx=0 tx=0).stop 2024-02-21T23:13:41.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.726+0000 7f5e29bca740 1 -- 172.21.15.28:0/3199522310 shutdown_connections 2024-02-21T23:13:41.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.726+0000 7f5e29bca740 1 --2- 172.21.15.28:0/3199522310 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557bdd3a34d0 0x557bdd40ff40 unknown :-1 s=CLOSED pgs=2413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:41.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.726+0000 7f5e29bca740 1 --2- 172.21.15.28:0/3199522310 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557bdd386f00 0x557bdd4162a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:41.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.726+0000 7f5e29bca740 1 --2- 172.21.15.28:0/3199522310 >> [v2:172.21.15.28:6834/3142544451,v1:172.21.15.28:6835/3142544451] conn(0x557bdd361350 0x557bdd361730 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:41.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.726+0000 7f5e29bca740 1 --2- 172.21.15.28:0/3199522310 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x557bdd368150 0x557bdd3a2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:41.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.726+0000 7f5e29bca740 1 -- 172.21.15.28:0/3199522310 >> 172.21.15.28:0/3199522310 conn(0x557bdd366e40 msgr2=0x557bdd387c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:41.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.726+0000 7f5e0cc00640 0 client.10983 ms_handle_reset on v2:172.21.15.28:6834/3142544451 2024-02-21T23:13:41.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.726+0000 7f5e29bca740 1 -- 172.21.15.28:0/3199522310 shutdown_connections 2024-02-21T23:13:41.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.726+0000 7f5e29bca740 1 -- 172.21.15.28:0/3199522310 wait complete. 2024-02-21T23:13:41.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.726+0000 7f5e29bca740 20 client.10983 trim_cache size 0 max 16384 2024-02-21T23:13:41.722 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-21T23:13:41.722 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac", 2024-02-21T23:13:41.722 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-02-21T23:13:41.722 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 8296, 2024-02-21T23:13:41.722 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-02-21T23:13:41.722 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-02-21T23:13:41.722 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-02-21T23:13:41.722 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 11.469818814, 2024-02-21T23:13:41.723 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 232, 2024-02-21T23:13:41.723 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 208, 2024-02-21T23:13:41.723 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 208, 2024-02-21T23:13:41.723 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 17.958946199 2024-02-21T23:13:41.723 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-21T23:13:41.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.727+0000 7f5e26c00640 1 -- 172.21.15.28:0/2197188734 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5dfc06b4d0 msgr2=0x7f5dfc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:41.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.727+0000 7f5e26c00640 1 --2- 172.21.15.28:0/2197188734 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5dfc06b4d0 0x7f5dfc06d990 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f5e08006fd0 tx=0x7f5e08008040 comp rx=0 tx=0).stop 2024-02-21T23:13:41.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.727+0000 7f5e26c00640 1 -- 172.21.15.28:0/2197188734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e200bf950 msgr2=0x7f5e201457c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:41.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.727+0000 7f5e26c00640 1 --2- 172.21.15.28:0/2197188734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e200bf950 0x7f5e201457c0 secure :-1 s=READY pgs=2409 cs=0 l=1 rev1=1 crypto rx=0x7f5e10036040 tx=0x7f5e10004a40 comp rx=0 tx=0).stop 2024-02-21T23:13:41.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.727+0000 7f5e26c00640 1 -- 172.21.15.28:0/2197188734 shutdown_connections 2024-02-21T23:13:41.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.727+0000 7f5e26c00640 1 --2- 172.21.15.28:0/2197188734 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5dfc06b4d0 0x7f5dfc06d990 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:41.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.727+0000 7f5e26c00640 1 --2- 172.21.15.28:0/2197188734 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5e200bffd0 0x7f5e201deb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:41.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.727+0000 7f5e26c00640 1 --2- 172.21.15.28:0/2197188734 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e200c6a00 0x7f5e201eb1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:41.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.727+0000 7f5e26c00640 1 --2- 172.21.15.28:0/2197188734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e200bf950 0x7f5e201457c0 unknown :-1 s=CLOSED pgs=2409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:41.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.727+0000 7f5e26c00640 1 -- 172.21.15.28:0/2197188734 >> 172.21.15.28:0/2197188734 conn(0x7f5e2006d580 msgr2=0x7f5e2006d960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:41.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.727+0000 7f5e26c00640 1 -- 172.21.15.28:0/2197188734 shutdown_connections 2024-02-21T23:13:41.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.727+0000 7f5e26c00640 1 -- 172.21.15.28:0/2197188734 wait complete. 2024-02-21T23:13:41.736 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs down true 2024-02-21T23:13:41.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.944+0000 7f4c59800640 1 Processor -- start 2024-02-21T23:13:41.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.944+0000 7f4c59800640 1 -- start start 2024-02-21T23:13:41.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.945+0000 7f4c59800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4c540a6f30 0x7f4c540c2b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:41.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.945+0000 7f4c59800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c540c2380 con 0x7f4c541524b0 2024-02-21T23:13:41.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.945+0000 7f4c59800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c540c5e90 con 0x7f4c540c5a00 2024-02-21T23:13:41.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.945+0000 7f4c59800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c540c6170 con 0x7f4c540a6f30 2024-02-21T23:13:41.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.946+0000 7f4c53e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f4c541524b0 0x7f4c54158460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:32774/0 (socket says 172.21.15.28:32774) 2024-02-21T23:13:41.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.946+0000 7f4c53e00640 1 -- 172.21.15.28:0/1604489593 learned_addr learned my addr 172.21.15.28:0/1604489593 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:13:41.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.946+0000 7f4c58e00640 1 --2- 172.21.15.28:0/1604489593 >> v2:172.21.15.28:3300/0 conn(0x7f4c540a6f30 0x7f4c540c2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:41.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.946+0000 7f4c58e00640 1 -- 172.21.15.28:0/1604489593 >> v1:172.21.15.81:6789/0 conn(0x7f4c540c5a00 legacy=0x7f4c540c6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:41.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.946+0000 7f4c52a00640 1 -- 172.21.15.28:0/1604489593 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1530766887 0 0) 0x7f4c540c2380 con 0x7f4c541524b0 2024-02-21T23:13:41.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.946+0000 7f4c58e00640 1 -- 172.21.15.28:0/1604489593 >> v1:172.21.15.28:6789/0 conn(0x7f4c541524b0 legacy=0x7f4c54158460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:41.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.946+0000 7f4c58e00640 1 -- 172.21.15.28:0/1604489593 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c540c25f0 con 0x7f4c540a6f30 2024-02-21T23:13:41.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.948+0000 7f4c58e00640 1 --2- 172.21.15.28:0/1604489593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c540a6f30 0x7f4c540c2b10 secure :-1 s=READY pgs=2415 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c009d90 tx=0x7f4c4c02bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=336c393b3e92b5bf server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:41.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.949+0000 7f4c52a00640 1 -- 172.21.15.28:0/1604489593 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c039070 con 0x7f4c540a6f30 2024-02-21T23:13:41.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.949+0000 7f4c52a00640 1 -- 172.21.15.28:0/1604489593 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c4c02ccf0 con 0x7f4c540a6f30 2024-02-21T23:13:41.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.949+0000 7f4c52a00640 1 -- 172.21.15.28:0/1604489593 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c034660 con 0x7f4c540a6f30 2024-02-21T23:13:41.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.949+0000 7f4c59800640 1 -- 172.21.15.28:0/1604489593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c540a6f30 msgr2=0x7f4c540c2b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:41.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.949+0000 7f4c59800640 1 --2- 172.21.15.28:0/1604489593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c540a6f30 0x7f4c540c2b10 secure :-1 s=READY pgs=2415 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c009d90 tx=0x7f4c4c02bf40 comp rx=0 tx=0).stop 2024-02-21T23:13:41.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.949+0000 7f4c59800640 1 -- 172.21.15.28:0/1604489593 shutdown_connections 2024-02-21T23:13:41.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.949+0000 7f4c59800640 1 --2- 172.21.15.28:0/1604489593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c540a6f30 0x7f4c540c2b10 unknown :-1 s=CLOSED pgs=2415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:41.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.949+0000 7f4c59800640 1 -- 172.21.15.28:0/1604489593 >> 172.21.15.28:0/1604489593 conn(0x7f4c540a9690 msgr2=0x7f4c540a9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:41.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.950+0000 7f4c59800640 1 -- 172.21.15.28:0/1604489593 shutdown_connections 2024-02-21T23:13:41.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.950+0000 7f4c59800640 1 -- 172.21.15.28:0/1604489593 wait complete. 2024-02-21T23:13:41.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.951+0000 7f4c59800640 1 Processor -- start 2024-02-21T23:13:41.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.952+0000 7f4c59800640 1 -- start start 2024-02-21T23:13:41.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.953+0000 7f4c59800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4c540a6f30 0x7f4c541eb2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:41.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.953+0000 7f4c59800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c541524b0 0x7f4c54155e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:41.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.953+0000 7f4c59800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4c5406e280 0x7f4c541510c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:41.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.953+0000 7f4c59800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c54152890 con 0x7f4c541524b0 2024-02-21T23:13:41.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.953+0000 7f4c59800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c540c6450 con 0x7f4c540a6f30 2024-02-21T23:13:41.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.953+0000 7f4c59800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c540c74e0 con 0x7f4c5406e280 2024-02-21T23:13:41.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.953+0000 7f4c53400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c541524b0 0x7f4c54155e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:41.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.953+0000 7f4c53400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c541524b0 0x7f4c54155e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39610/0 (socket says 172.21.15.28:39610) 2024-02-21T23:13:41.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.953+0000 7f4c53400640 1 -- 172.21.15.28:0/988796924 learned_addr learned my addr 172.21.15.28:0/988796924 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:41.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.953+0000 7f4c53e00640 1 --2- 172.21.15.28:0/988796924 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4c540a6f30 0x7f4c541eb2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:41.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.953+0000 7f4c58e00640 1 --2- 172.21.15.28:0/988796924 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4c5406e280 0x7f4c541510c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:41.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.953+0000 7f4c53400640 1 -- 172.21.15.28:0/988796924 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4c5406e280 msgr2=0x7f4c541510c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:41.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.954+0000 7f4c53400640 1 --2- 172.21.15.28:0/988796924 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4c5406e280 0x7f4c541510c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:41.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.954+0000 7f4c53400640 1 -- 172.21.15.28:0/988796924 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4c540a6f30 msgr2=0x7f4c541eb2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:41.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.954+0000 7f4c53400640 1 --2- 172.21.15.28:0/988796924 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4c540a6f30 0x7f4c541eb2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:41.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.954+0000 7f4c53400640 1 -- 172.21.15.28:0/988796924 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c54156380 con 0x7f4c541524b0 2024-02-21T23:13:41.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.954+0000 7f4c53400640 1 --2- 172.21.15.28:0/988796924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c541524b0 0x7f4c54155e40 secure :-1 s=READY pgs=2416 cs=0 l=1 rev1=1 crypto rx=0x7f4c4000c480 tx=0x7f4c4000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:41.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.954+0000 7f4c4be00640 1 -- 172.21.15.28:0/988796924 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c400052d0 con 0x7f4c541524b0 2024-02-21T23:13:41.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.954+0000 7f4c4be00640 1 -- 172.21.15.28:0/988796924 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c40005470 con 0x7f4c541524b0 2024-02-21T23:13:41.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.954+0000 7f4c59800640 1 -- 172.21.15.28:0/988796924 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c5406cb80 con 0x7f4c541524b0 2024-02-21T23:13:41.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.954+0000 7f4c4be00640 1 -- 172.21.15.28:0/988796924 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c40012870 con 0x7f4c541524b0 2024-02-21T23:13:41.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.954+0000 7f4c59800640 1 -- 172.21.15.28:0/988796924 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c541eaaa0 con 0x7f4c541524b0 2024-02-21T23:13:41.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.956+0000 7f4c59800640 1 -- 172.21.15.28:0/988796924 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c5404f290 con 0x7f4c541524b0 2024-02-21T23:13:41.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.957+0000 7f4c4be00640 1 -- 172.21.15.28:0/988796924 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4c40012a10 con 0x7f4c541524b0 2024-02-21T23:13:41.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.958+0000 7f4c4be00640 1 --2- 172.21.15.28:0/988796924 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4c3406b4d0 0x7f4c3406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:41.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.958+0000 7f4c4be00640 1 -- 172.21.15.28:0/988796924 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4c40089fa0 con 0x7f4c541524b0 2024-02-21T23:13:41.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.958+0000 7f4c53e00640 1 --2- 172.21.15.28:0/988796924 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4c3406b4d0 0x7f4c3406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:41.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.960+0000 7f4c53e00640 1 --2- 172.21.15.28:0/988796924 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4c3406b4d0 0x7f4c3406d990 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f4c44000f60 tx=0x7f4c44007d00 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:41.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:41.962+0000 7f4c4be00640 1 -- 172.21.15.28:0/988796924 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4c54156380 con 0x7f4c541524b0 2024-02-21T23:13:42.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.204+0000 7f4c59800640 1 -- 172.21.15.28:0/988796924 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "down", "val": "true"} v 0) v1 -- 0x7f4c54072440 con 0x7f4c541524b0 2024-02-21T23:13:42.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.349+0000 7f4c4be00640 1 -- 172.21.15.28:0/988796924 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "down", "val": "true"}]=0 cephfs marked down. v233) v1 ==== 125+0+0 (secure 0 0 0) 0x7f4c40058d20 con 0x7f4c541524b0 2024-02-21T23:13:42.337 INFO:teuthology.orchestra.run.smithi028.stderr:cephfs marked down. 2024-02-21T23:13:42.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.352+0000 7f4c59800640 1 -- 172.21.15.28:0/988796924 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4c3406b4d0 msgr2=0x7f4c3406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:42.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.352+0000 7f4c59800640 1 --2- 172.21.15.28:0/988796924 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4c3406b4d0 0x7f4c3406d990 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f4c44000f60 tx=0x7f4c44007d00 comp rx=0 tx=0).stop 2024-02-21T23:13:42.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.352+0000 7f4c59800640 1 -- 172.21.15.28:0/988796924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c541524b0 msgr2=0x7f4c54155e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:42.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.352+0000 7f4c59800640 1 --2- 172.21.15.28:0/988796924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c541524b0 0x7f4c54155e40 secure :-1 s=READY pgs=2416 cs=0 l=1 rev1=1 crypto rx=0x7f4c4000c480 tx=0x7f4c4000c940 comp rx=0 tx=0).stop 2024-02-21T23:13:42.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.352+0000 7f4c59800640 1 -- 172.21.15.28:0/988796924 shutdown_connections 2024-02-21T23:13:42.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.352+0000 7f4c59800640 1 --2- 172.21.15.28:0/988796924 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4c3406b4d0 0x7f4c3406d990 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:42.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.352+0000 7f4c59800640 1 --2- 172.21.15.28:0/988796924 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4c5406e280 0x7f4c541510c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:42.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.352+0000 7f4c59800640 1 --2- 172.21.15.28:0/988796924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c541524b0 0x7f4c54155e40 unknown :-1 s=CLOSED pgs=2416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:42.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.352+0000 7f4c59800640 1 --2- 172.21.15.28:0/988796924 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4c540a6f30 0x7f4c541eb2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:42.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.352+0000 7f4c59800640 1 -- 172.21.15.28:0/988796924 >> 172.21.15.28:0/988796924 conn(0x7f4c540a9690 msgr2=0x7f4c540c0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:42.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.352+0000 7f4c59800640 1 -- 172.21.15.28:0/988796924 shutdown_connections 2024-02-21T23:13:42.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.352+0000 7f4c59800640 1 -- 172.21.15.28:0/988796924 wait complete. 2024-02-21T23:13:42.360 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:13:42.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.562+0000 7f4b26a00640 1 Processor -- start 2024-02-21T23:13:42.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.562+0000 7f4b26a00640 1 -- start start 2024-02-21T23:13:42.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.564+0000 7f4b26a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f4b201558c0 0x7f4b2006c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:42.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.564+0000 7f4b26a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b201528e0 con 0x7f4b200ca050 2024-02-21T23:13:42.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.564+0000 7f4b26a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b200c86e0 con 0x7f4b20152500 2024-02-21T23:13:42.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.564+0000 7f4b26a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b2006c840 con 0x7f4b201558c0 2024-02-21T23:13:42.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.564+0000 7f4b24c00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f4b200ca050 0x7f4b2006bbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:32788/0 (socket says 172.21.15.28:32788) 2024-02-21T23:13:42.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.564+0000 7f4b24c00640 1 -- 172.21.15.28:0/31036566 learned_addr learned my addr 172.21.15.28:0/31036566 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:13:42.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.564+0000 7f4b26000640 1 --2- 172.21.15.28:0/31036566 >> v2:172.21.15.81:3300/0 conn(0x7f4b201558c0 0x7f4b2006c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:42.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.565+0000 7f4b1fe00640 1 -- 172.21.15.28:0/31036566 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2996905431 0 0) 0x7f4b201528e0 con 0x7f4b200ca050 2024-02-21T23:13:42.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.565+0000 7f4b1fe00640 1 -- 172.21.15.28:0/31036566 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b08003520 con 0x7f4b200ca050 2024-02-21T23:13:42.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.565+0000 7f4b1fe00640 1 -- 172.21.15.28:0/31036566 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3187674558 0 0) 0x7f4b08003520 con 0x7f4b200ca050 2024-02-21T23:13:42.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.565+0000 7f4b1fe00640 1 -- 172.21.15.28:0/31036566 >> v2:172.21.15.81:3300/0 conn(0x7f4b201558c0 msgr2=0x7f4b2006c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:42.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.565+0000 7f4b1fe00640 1 --2- 172.21.15.28:0/31036566 >> v2:172.21.15.81:3300/0 conn(0x7f4b201558c0 0x7f4b2006c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:42.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.565+0000 7f4b1fe00640 1 -- 172.21.15.28:0/31036566 >> v1:172.21.15.81:6789/0 conn(0x7f4b20152500 legacy=0x7f4b200bf6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:42.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.565+0000 7f4b1fe00640 1 -- 172.21.15.28:0/31036566 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b200c8980 con 0x7f4b200ca050 2024-02-21T23:13:42.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.565+0000 7f4b1fe00640 1 -- 172.21.15.28:0/31036566 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f4b0c001670 con 0x7f4b200ca050 2024-02-21T23:13:42.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.565+0000 7f4b1fe00640 1 -- 172.21.15.28:0/31036566 >> v1:172.21.15.28:6789/0 conn(0x7f4b200ca050 legacy=0x7f4b2006bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:42.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.565+0000 7f4b1fe00640 1 --2- 172.21.15.28:0/31036566 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4b08003a00 0x7f4b08003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:42.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.565+0000 7f4b1fe00640 1 --2- 172.21.15.28:0/31036566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b08004800 0x7f4b08004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:42.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.565+0000 7f4b1fe00640 1 --2- 172.21.15.28:0/31036566 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4b08005210 0x7f4b0800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:42.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.565+0000 7f4b1fe00640 1 -- 172.21.15.28:0/31036566 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b2006c840 con 0x7f4b08004800 2024-02-21T23:13:42.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.565+0000 7f4b1fe00640 1 -- 172.21.15.28:0/31036566 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b0800c160 con 0x7f4b08005210 2024-02-21T23:13:42.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.565+0000 7f4b1fe00640 1 -- 172.21.15.28:0/31036566 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b0800c360 con 0x7f4b08003a00 2024-02-21T23:13:42.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.565+0000 7f4b24c00640 1 --2- 172.21.15.28:0/31036566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b08004800 0x7f4b08004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:42.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.566+0000 7f4b25600640 1 --2- 172.21.15.28:0/31036566 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4b08003a00 0x7f4b08003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:42.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.566+0000 7f4b24c00640 1 -- 172.21.15.28:0/31036566 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4b08003a00 msgr2=0x7f4b08003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:42.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.566+0000 7f4b24c00640 1 --2- 172.21.15.28:0/31036566 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4b08003a00 0x7f4b08003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:42.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.566+0000 7f4b24c00640 1 -- 172.21.15.28:0/31036566 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4b08005210 msgr2=0x7f4b0800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:42.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.566+0000 7f4b24c00640 1 --2- 172.21.15.28:0/31036566 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4b08005210 0x7f4b0800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:42.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.566+0000 7f4b24c00640 1 -- 172.21.15.28:0/31036566 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b08003520 con 0x7f4b08004800 2024-02-21T23:13:42.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.566+0000 7f4b26000640 1 -- 172.21.15.28:0/31036566 reap_dead start 2024-02-21T23:13:42.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.567+0000 7f4b24c00640 1 --2- 172.21.15.28:0/31036566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b08004800 0x7f4b08004c80 secure :-1 s=READY pgs=2418 cs=0 l=1 rev1=1 crypto rx=0x7f4b0c00c160 tx=0x7f4b08005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c907ca5118ed26c server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:42.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.567+0000 7f4b1fe00640 1 -- 172.21.15.28:0/31036566 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b0c007c60 con 0x7f4b08004800 2024-02-21T23:13:42.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.567+0000 7f4b1fe00640 1 -- 172.21.15.28:0/31036566 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b0c030070 con 0x7f4b08004800 2024-02-21T23:13:42.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.567+0000 7f4b1fe00640 1 -- 172.21.15.28:0/31036566 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b0c005d90 con 0x7f4b08004800 2024-02-21T23:13:42.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.568+0000 7f4b26a00640 1 -- 172.21.15.28:0/31036566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b08004800 msgr2=0x7f4b08004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:42.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.568+0000 7f4b26a00640 1 --2- 172.21.15.28:0/31036566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b08004800 0x7f4b08004c80 secure :-1 s=READY pgs=2418 cs=0 l=1 rev1=1 crypto rx=0x7f4b0c00c160 tx=0x7f4b08005830 comp rx=0 tx=0).stop 2024-02-21T23:13:42.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.568+0000 7f4b26a00640 1 -- 172.21.15.28:0/31036566 shutdown_connections 2024-02-21T23:13:42.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.568+0000 7f4b26a00640 1 --2- 172.21.15.28:0/31036566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b08004800 0x7f4b08004c80 unknown :-1 s=CLOSED pgs=2418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:42.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.568+0000 7f4b26a00640 1 -- 172.21.15.28:0/31036566 >> 172.21.15.28:0/31036566 conn(0x7f4b200ac450 msgr2=0x7f4b200ac860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:42.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.568+0000 7f4b26a00640 1 -- 172.21.15.28:0/31036566 shutdown_connections 2024-02-21T23:13:42.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.568+0000 7f4b26a00640 1 -- 172.21.15.28:0/31036566 wait complete. 2024-02-21T23:13:42.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.570+0000 7f4b26a00640 1 Processor -- start 2024-02-21T23:13:42.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.570+0000 7f4b26a00640 1 -- start start 2024-02-21T23:13:42.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.571+0000 7f4b26a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4b08004800 0x7f4b201ee160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:42.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.571+0000 7f4b26a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b201560b0 0x7f4b20158420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:42.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.571+0000 7f4b26a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4b20148450 0x7f4b20158960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:42.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.571+0000 7f4b26a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b200c9220 con 0x7f4b201560b0 2024-02-21T23:13:42.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.571+0000 7f4b26a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b200c8f40 con 0x7f4b08004800 2024-02-21T23:13:42.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.571+0000 7f4b26a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b20155cc0 con 0x7f4b20148450 2024-02-21T23:13:42.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.571+0000 7f4b24c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b201560b0 0x7f4b20158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:42.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.571+0000 7f4b24c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b201560b0 0x7f4b20158420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39634/0 (socket says 172.21.15.28:39634) 2024-02-21T23:13:42.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.571+0000 7f4b24c00640 1 -- 172.21.15.28:0/4120216848 learned_addr learned my addr 172.21.15.28:0/4120216848 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:42.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.571+0000 7f4b25600640 1 --2- 172.21.15.28:0/4120216848 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4b08004800 0x7f4b201ee160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:42.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.571+0000 7f4b26000640 1 --2- 172.21.15.28:0/4120216848 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4b20148450 0x7f4b20158960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:42.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.572+0000 7f4b24c00640 1 -- 172.21.15.28:0/4120216848 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4b20148450 msgr2=0x7f4b20158960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:42.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.572+0000 7f4b24c00640 1 --2- 172.21.15.28:0/4120216848 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4b20148450 0x7f4b20158960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:42.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.572+0000 7f4b24c00640 1 -- 172.21.15.28:0/4120216848 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4b08004800 msgr2=0x7f4b201ee160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:42.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.572+0000 7f4b24c00640 1 --2- 172.21.15.28:0/4120216848 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4b08004800 0x7f4b201ee160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:42.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.572+0000 7f4b24c00640 1 -- 172.21.15.28:0/4120216848 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b201eedf0 con 0x7f4b201560b0 2024-02-21T23:13:42.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.572+0000 7f4b24c00640 1 --2- 172.21.15.28:0/4120216848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b201560b0 0x7f4b20158420 secure :-1 s=READY pgs=2419 cs=0 l=1 rev1=1 crypto rx=0x7f4b0c02c480 tx=0x7f4b0c001970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:42.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.572+0000 7f4b1d600640 1 -- 172.21.15.28:0/4120216848 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b0c045070 con 0x7f4b201560b0 2024-02-21T23:13:42.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.573+0000 7f4b1d600640 1 -- 172.21.15.28:0/4120216848 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b0c0049f0 con 0x7f4b201560b0 2024-02-21T23:13:42.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.573+0000 7f4b1d600640 1 -- 172.21.15.28:0/4120216848 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b0c007c60 con 0x7f4b201560b0 2024-02-21T23:13:42.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.573+0000 7f4b26a00640 1 -- 172.21.15.28:0/4120216848 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b20149a90 con 0x7f4b201560b0 2024-02-21T23:13:42.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.573+0000 7f4b26a00640 1 -- 172.21.15.28:0/4120216848 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b20149220 con 0x7f4b201560b0 2024-02-21T23:13:42.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.574+0000 7f4b26a00640 1 -- 172.21.15.28:0/4120216848 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b20152d90 con 0x7f4b201560b0 2024-02-21T23:13:42.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.575+0000 7f4b1d600640 1 -- 172.21.15.28:0/4120216848 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4b0c007070 con 0x7f4b201560b0 2024-02-21T23:13:42.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.576+0000 7f4b1d600640 1 --2- 172.21.15.28:0/4120216848 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4b0006b4d0 0x7f4b0006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:42.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.576+0000 7f4b1d600640 1 -- 172.21.15.28:0/4120216848 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4b0c0aafc0 con 0x7f4b201560b0 2024-02-21T23:13:42.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.576+0000 7f4b25600640 1 --2- 172.21.15.28:0/4120216848 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4b0006b4d0 0x7f4b0006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:42.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.577+0000 7f4b25600640 1 --2- 172.21.15.28:0/4120216848 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4b0006b4d0 0x7f4b0006d990 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f4b14005490 tx=0x7f4b14009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:42.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.585+0000 7f4b1d600640 1 -- 172.21.15.28:0/4120216848 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4b20152d90 con 0x7f4b201560b0 2024-02-21T23:13:42.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.815+0000 7f4b26a00640 1 -- 172.21.15.28:0/4120216848 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b2004f290 con 0x7f4b201560b0 2024-02-21T23:13:42.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.816+0000 7f4b1d600640 1 -- 172.21.15.28:0/4120216848 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 234 v234) v1 ==== 95+0+4964 (secure 0 0 0) 0x7f4b0c079ca0 con 0x7f4b201560b0 2024-02-21T23:13:42.805 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:13:42.805 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":234,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7265,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":224},{"gid":8287,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1607587459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1607587459},{"type":"v1","addr":"172.21.15.81:6835","nonce":1607587459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":224}],"filesystems":[{"mdsmap":{"epoch":234,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:13:42.353072+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":8290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8290":{"gid":8290,"name":"a","rank":1,"incarnation":231,"state":"up:stopping","state_seq":5,"addr":"172.21.15.28:6833/1864206570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1864206570},{"type":"v1","addr":"172.21.15.28:6833","nonce":1864206570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:13:42.805 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 234 2024-02-21T23:13:42.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.819+0000 7f4b26a00640 1 -- 172.21.15.28:0/4120216848 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4b0006b4d0 msgr2=0x7f4b0006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:42.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.819+0000 7f4b26a00640 1 --2- 172.21.15.28:0/4120216848 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4b0006b4d0 0x7f4b0006d990 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f4b14005490 tx=0x7f4b14009040 comp rx=0 tx=0).stop 2024-02-21T23:13:42.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.819+0000 7f4b26a00640 1 -- 172.21.15.28:0/4120216848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b201560b0 msgr2=0x7f4b20158420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:42.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.819+0000 7f4b26a00640 1 --2- 172.21.15.28:0/4120216848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b201560b0 0x7f4b20158420 secure :-1 s=READY pgs=2419 cs=0 l=1 rev1=1 crypto rx=0x7f4b0c02c480 tx=0x7f4b0c001970 comp rx=0 tx=0).stop 2024-02-21T23:13:42.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.819+0000 7f4b26a00640 1 -- 172.21.15.28:0/4120216848 shutdown_connections 2024-02-21T23:13:42.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.819+0000 7f4b26a00640 1 --2- 172.21.15.28:0/4120216848 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4b0006b4d0 0x7f4b0006d990 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:42.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.819+0000 7f4b26a00640 1 --2- 172.21.15.28:0/4120216848 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4b20148450 0x7f4b20158960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:42.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.819+0000 7f4b26a00640 1 --2- 172.21.15.28:0/4120216848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b201560b0 0x7f4b20158420 unknown :-1 s=CLOSED pgs=2419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:42.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.819+0000 7f4b26a00640 1 --2- 172.21.15.28:0/4120216848 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4b08004800 0x7f4b201ee160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:42.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.819+0000 7f4b26a00640 1 -- 172.21.15.28:0/4120216848 >> 172.21.15.28:0/4120216848 conn(0x7f4b200ac450 msgr2=0x7f4b201550c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:42.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.819+0000 7f4b26a00640 1 -- 172.21.15.28:0/4120216848 shutdown_connections 2024-02-21T23:13:42.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:42.819+0000 7f4b26a00640 1 -- 172.21.15.28:0/4120216848 wait complete. 2024-02-21T23:13:43.232 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 234, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:13:42.353072+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 8290}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8290': {'gid': 8290, 'name': 'a', 'rank': 1, 'incarnation': 231, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.28:6833/1864206570', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 1864206570}, {'type': 'v1', 'addr': '172.21.15.28:6833', 'nonce': 1864206570}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:13:43.232 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8290:up:stopping 2024-02-21T23:13:44.232 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:13:44.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.473+0000 7f7cb1000640 1 Processor -- start 2024-02-21T23:13:44.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.473+0000 7f7cb1000640 1 -- start start 2024-02-21T23:13:44.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.475+0000 7f7cb1000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7cac156900 0x7f7cac0c8a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:44.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.475+0000 7f7cb1000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f7cac0c8f50 0x7f7cac15ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:44.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.475+0000 7f7cb1000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cac159230 con 0x7f7cac06ddb0 2024-02-21T23:13:44.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.475+0000 7f7cb1000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cac156ce0 con 0x7f7cac0c8f50 2024-02-21T23:13:44.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.475+0000 7f7cb1000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cac159e20 con 0x7f7cac156900 2024-02-21T23:13:44.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.475+0000 7f7ca3e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f7cac06ddb0 0x7f7cac15a720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:32800/0 (socket says 172.21.15.28:32800) 2024-02-21T23:13:44.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.475+0000 7f7ca3e00640 1 -- 172.21.15.28:0/2643179913 learned_addr learned my addr 172.21.15.28:0/2643179913 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:13:44.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.476+0000 7f7cabe00640 1 --2- 172.21.15.28:0/2643179913 >> v2:172.21.15.81:3300/0 conn(0x7f7cac0c8f50 0x7f7cac15ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:44.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.476+0000 7f7cab400640 1 --2- 172.21.15.28:0/2643179913 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7cac156900 0x7f7cac0c8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:44.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.476+0000 7f7caaa00640 1 -- 172.21.15.28:0/2643179913 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 85001985 0 0) 0x7f7cac159230 con 0x7f7cac06ddb0 2024-02-21T23:13:44.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.476+0000 7f7caaa00640 1 -- 172.21.15.28:0/2643179913 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c90003520 con 0x7f7cac06ddb0 2024-02-21T23:13:44.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.476+0000 7f7caaa00640 1 -- 172.21.15.28:0/2643179913 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2483739197 0 0) 0x7f7c90003520 con 0x7f7cac06ddb0 2024-02-21T23:13:44.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.476+0000 7f7caaa00640 1 -- 172.21.15.28:0/2643179913 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7cac156900 msgr2=0x7f7cac0c8a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:44.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.476+0000 7f7caaa00640 1 --2- 172.21.15.28:0/2643179913 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7cac156900 0x7f7cac0c8a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:44.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.476+0000 7f7caaa00640 1 -- 172.21.15.28:0/2643179913 >> v2:172.21.15.81:3300/0 conn(0x7f7cac0c8f50 msgr2=0x7f7cac15ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:44.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.476+0000 7f7caaa00640 1 --2- 172.21.15.28:0/2643179913 >> v2:172.21.15.81:3300/0 conn(0x7f7cac0c8f50 0x7f7cac15ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:44.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.476+0000 7f7caaa00640 1 -- 172.21.15.28:0/2643179913 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cac1594f0 con 0x7f7cac06ddb0 2024-02-21T23:13:44.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.476+0000 7f7caaa00640 1 -- 172.21.15.28:0/2643179913 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f7c94001670 con 0x7f7cac06ddb0 2024-02-21T23:13:44.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.477+0000 7f7caaa00640 1 -- 172.21.15.28:0/2643179913 >> v1:172.21.15.28:6789/0 conn(0x7f7cac06ddb0 legacy=0x7f7cac15a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:44.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.477+0000 7f7caaa00640 1 --2- 172.21.15.28:0/2643179913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c900039b0 0x7f7c90003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:44.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.477+0000 7f7caaa00640 1 --2- 172.21.15.28:0/2643179913 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7c90004800 0x7f7c90004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:44.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.477+0000 7f7caaa00640 1 --2- 172.21.15.28:0/2643179913 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7cac156900 0x7f7c900051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:44.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.477+0000 7f7caaa00640 1 -- 172.21.15.28:0/2643179913 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cac159e20 con 0x7f7c900039b0 2024-02-21T23:13:44.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.477+0000 7f7caaa00640 1 -- 172.21.15.28:0/2643179913 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c9000b9c0 con 0x7f7c90004800 2024-02-21T23:13:44.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.477+0000 7f7caaa00640 1 -- 172.21.15.28:0/2643179913 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c9000bb40 con 0x7f7cac156900 2024-02-21T23:13:44.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.477+0000 7f7cab400640 1 --2- 172.21.15.28:0/2643179913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c900039b0 0x7f7c90003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:44.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.477+0000 7f7ca3e00640 1 --2- 172.21.15.28:0/2643179913 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7c90004800 0x7f7c90004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:44.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.477+0000 7f7cabe00640 1 --2- 172.21.15.28:0/2643179913 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7cac156900 0x7f7c900051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:44.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.477+0000 7f7cab400640 1 -- 172.21.15.28:0/2643179913 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7cac156900 msgr2=0x7f7c900051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:44.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.477+0000 7f7cab400640 1 --2- 172.21.15.28:0/2643179913 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7cac156900 0x7f7c900051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:44.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.477+0000 7f7cab400640 1 -- 172.21.15.28:0/2643179913 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7c90004800 msgr2=0x7f7c90004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:44.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.477+0000 7f7cab400640 1 --2- 172.21.15.28:0/2643179913 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7c90004800 0x7f7c90004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:44.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.477+0000 7f7cab400640 1 -- 172.21.15.28:0/2643179913 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c90003520 con 0x7f7c900039b0 2024-02-21T23:13:44.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.477+0000 7f7cabe00640 1 --2- 172.21.15.28:0/2643179913 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7cac156900 0x7f7c900051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:13:44.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.479+0000 7f7cab400640 1 --2- 172.21.15.28:0/2643179913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c900039b0 0x7f7c90003dd0 secure :-1 s=READY pgs=2421 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c009af0 tx=0x7f7c90005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9a0578e352a064a server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:44.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.480+0000 7f7caaa00640 1 -- 172.21.15.28:0/2643179913 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c9c039070 con 0x7f7c900039b0 2024-02-21T23:13:44.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.480+0000 7f7caaa00640 1 -- 172.21.15.28:0/2643179913 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c9c02c6b0 con 0x7f7c900039b0 2024-02-21T23:13:44.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.480+0000 7f7caaa00640 1 -- 172.21.15.28:0/2643179913 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c9c009690 con 0x7f7c900039b0 2024-02-21T23:13:44.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.480+0000 7f7cb1000640 1 -- 172.21.15.28:0/2643179913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c900039b0 msgr2=0x7f7c90003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:44.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.480+0000 7f7cb1000640 1 --2- 172.21.15.28:0/2643179913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c900039b0 0x7f7c90003dd0 secure :-1 s=READY pgs=2421 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c009af0 tx=0x7f7c90005830 comp rx=0 tx=0).stop 2024-02-21T23:13:44.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.480+0000 7f7cabe00640 1 -- 172.21.15.28:0/2643179913 reap_dead start 2024-02-21T23:13:44.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.480+0000 7f7cb1000640 1 -- 172.21.15.28:0/2643179913 shutdown_connections 2024-02-21T23:13:44.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.480+0000 7f7cb1000640 1 -- 172.21.15.28:0/2643179913 >> 172.21.15.28:0/2643179913 conn(0x7f7cac06b260 msgr2=0x7f7cac06b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:44.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.480+0000 7f7cb1000640 1 -- 172.21.15.28:0/2643179913 shutdown_connections 2024-02-21T23:13:44.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.480+0000 7f7cb1000640 1 -- 172.21.15.28:0/2643179913 wait complete. 2024-02-21T23:13:44.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.482+0000 7f7cb1000640 1 Processor -- start 2024-02-21T23:13:44.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.482+0000 7f7cb1000640 1 -- start start 2024-02-21T23:13:44.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.483+0000 7f7cb1000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7c900039b0 0x7f7cac149770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:44.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.483+0000 7f7cb1000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7cac149cb0 0x7f7cac0c3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:44.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.483+0000 7f7cb1000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cac0c4130 0x7f7cac1f2670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:44.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.483+0000 7f7cb1000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cac0c7ef0 con 0x7f7cac0c4130 2024-02-21T23:13:44.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.483+0000 7f7cb1000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cac159ab0 con 0x7f7cac149cb0 2024-02-21T23:13:44.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.483+0000 7f7cb1000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cac0c8070 con 0x7f7c900039b0 2024-02-21T23:13:44.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.484+0000 7f7cabe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cac0c4130 0x7f7cac1f2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:44.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.484+0000 7f7cabe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cac0c4130 0x7f7cac1f2670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39648/0 (socket says 172.21.15.28:39648) 2024-02-21T23:13:44.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.484+0000 7f7cabe00640 1 -- 172.21.15.28:0/1009631608 learned_addr learned my addr 172.21.15.28:0/1009631608 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:44.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.484+0000 7f7cab400640 1 --2- 172.21.15.28:0/1009631608 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7c900039b0 0x7f7cac149770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:44.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.484+0000 7f7ca3e00640 1 --2- 172.21.15.28:0/1009631608 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7cac149cb0 0x7f7cac0c3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:44.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.484+0000 7f7cabe00640 1 -- 172.21.15.28:0/1009631608 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7c900039b0 msgr2=0x7f7cac149770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:44.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.484+0000 7f7cabe00640 1 --2- 172.21.15.28:0/1009631608 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7c900039b0 0x7f7cac149770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:44.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.484+0000 7f7cabe00640 1 -- 172.21.15.28:0/1009631608 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7cac149cb0 msgr2=0x7f7cac0c3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:44.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.484+0000 7f7cabe00640 1 --2- 172.21.15.28:0/1009631608 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7cac149cb0 0x7f7cac0c3bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:44.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.484+0000 7f7cabe00640 1 -- 172.21.15.28:0/1009631608 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cac0aba10 con 0x7f7cac0c4130 2024-02-21T23:13:44.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.485+0000 7f7cabe00640 1 --2- 172.21.15.28:0/1009631608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cac0c4130 0x7f7cac1f2670 secure :-1 s=READY pgs=2422 cs=0 l=1 rev1=1 crypto rx=0x7f7ca4009fd0 tx=0x7f7ca400f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:44.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.485+0000 7f7ca3400640 1 -- 172.21.15.28:0/1009631608 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ca401c070 con 0x7f7cac0c4130 2024-02-21T23:13:44.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.485+0000 7f7ca3400640 1 -- 172.21.15.28:0/1009631608 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ca4017400 con 0x7f7cac0c4130 2024-02-21T23:13:44.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.485+0000 7f7ca3400640 1 -- 172.21.15.28:0/1009631608 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ca40177d0 con 0x7f7cac0c4130 2024-02-21T23:13:44.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.485+0000 7f7cb1000640 1 -- 172.21.15.28:0/1009631608 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cac0abcf0 con 0x7f7cac0c4130 2024-02-21T23:13:44.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.485+0000 7f7cb1000640 1 -- 172.21.15.28:0/1009631608 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cac14a110 con 0x7f7cac0c4130 2024-02-21T23:13:44.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.486+0000 7f7cb1000640 1 -- 172.21.15.28:0/1009631608 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cac04f290 con 0x7f7cac0c4130 2024-02-21T23:13:44.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.487+0000 7f7ca3400640 1 -- 172.21.15.28:0/1009631608 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7ca401f070 con 0x7f7cac0c4130 2024-02-21T23:13:44.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.488+0000 7f7ca3400640 1 --2- 172.21.15.28:0/1009631608 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7c8806b4d0 0x7f7c8806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:44.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.488+0000 7f7ca3400640 1 -- 172.21.15.28:0/1009631608 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f7ca408a3b0 con 0x7f7cac0c4130 2024-02-21T23:13:44.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.489+0000 7f7cab400640 1 --2- 172.21.15.28:0/1009631608 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7c8806b4d0 0x7f7c8806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:44.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.490+0000 7f7cab400640 1 --2- 172.21.15.28:0/1009631608 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7c8806b4d0 0x7f7c8806d990 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c036000 tx=0x7f7c9c030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:44.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.496+0000 7f7ca3400640 1 -- 172.21.15.28:0/1009631608 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7ca4012070 con 0x7f7cac0c4130 2024-02-21T23:13:44.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.735+0000 7f7cb1000640 1 -- 172.21.15.28:0/1009631608 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7cac0aa110 con 0x7f7cac0c4130 2024-02-21T23:13:44.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.736+0000 7f7ca3400640 1 -- 172.21.15.28:0/1009631608 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 234 v234) v1 ==== 95+0+4964 (secure 0 0 0) 0x7f7ca4059090 con 0x7f7cac0c4130 2024-02-21T23:13:44.725 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:13:44.725 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":234,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7265,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":224},{"gid":8287,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1607587459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1607587459},{"type":"v1","addr":"172.21.15.81:6835","nonce":1607587459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":224}],"filesystems":[{"mdsmap":{"epoch":234,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:13:42.353072+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":8290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8290":{"gid":8290,"name":"a","rank":1,"incarnation":231,"state":"up:stopping","state_seq":5,"addr":"172.21.15.28:6833/1864206570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1864206570},{"type":"v1","addr":"172.21.15.28:6833","nonce":1864206570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:13:44.725 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 234 2024-02-21T23:13:44.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.740+0000 7f7cb1000640 1 -- 172.21.15.28:0/1009631608 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7c8806b4d0 msgr2=0x7f7c8806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:44.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.740+0000 7f7cb1000640 1 --2- 172.21.15.28:0/1009631608 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7c8806b4d0 0x7f7c8806d990 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c036000 tx=0x7f7c9c030000 comp rx=0 tx=0).stop 2024-02-21T23:13:44.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.740+0000 7f7cb1000640 1 -- 172.21.15.28:0/1009631608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cac0c4130 msgr2=0x7f7cac1f2670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:44.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.740+0000 7f7cb1000640 1 --2- 172.21.15.28:0/1009631608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cac0c4130 0x7f7cac1f2670 secure :-1 s=READY pgs=2422 cs=0 l=1 rev1=1 crypto rx=0x7f7ca4009fd0 tx=0x7f7ca400f4a0 comp rx=0 tx=0).stop 2024-02-21T23:13:44.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.740+0000 7f7cb1000640 1 -- 172.21.15.28:0/1009631608 shutdown_connections 2024-02-21T23:13:44.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.740+0000 7f7cb1000640 1 --2- 172.21.15.28:0/1009631608 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7c8806b4d0 0x7f7c8806d990 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:44.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.740+0000 7f7cb1000640 1 --2- 172.21.15.28:0/1009631608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cac0c4130 0x7f7cac1f2670 unknown :-1 s=CLOSED pgs=2422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:44.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.740+0000 7f7cb1000640 1 --2- 172.21.15.28:0/1009631608 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7cac149cb0 0x7f7cac0c3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:44.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.740+0000 7f7cb1000640 1 --2- 172.21.15.28:0/1009631608 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7c900039b0 0x7f7cac149770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:44.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.740+0000 7f7cb1000640 1 -- 172.21.15.28:0/1009631608 >> 172.21.15.28:0/1009631608 conn(0x7f7cac06b260 msgr2=0x7f7cac0a9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:44.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.740+0000 7f7cb1000640 1 -- 172.21.15.28:0/1009631608 shutdown_connections 2024-02-21T23:13:44.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:44.740+0000 7f7cb1000640 1 -- 172.21.15.28:0/1009631608 wait complete. 2024-02-21T23:13:45.152 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 234, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:13:42.353072+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 8290}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8290': {'gid': 8290, 'name': 'a', 'rank': 1, 'incarnation': 231, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.28:6833/1864206570', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 1864206570}, {'type': 'v1', 'addr': '172.21.15.28:6833', 'nonce': 1864206570}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:13:45.152 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8290:up:stopping 2024-02-21T23:13:46.153 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:13:46.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:46.397+0000 7fc1c8e00640 1 Processor -- start 2024-02-21T23:13:46.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:46.397+0000 7fc1c8e00640 1 -- start start 2024-02-21T23:13:46.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:46.399+0000 7fc1c8e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc1c4153950 0x7fc1c40c15d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:46.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:46.399+0000 7fc1c8e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1c4152e70 con 0x7fc1c415b6d0 2024-02-21T23:13:46.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:46.399+0000 7fc1c8e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1c4153150 con 0x7fc1c40c11f0 2024-02-21T23:13:46.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:46.399+0000 7fc1c8e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1c40a5ec0 con 0x7fc1c4153950 2024-02-21T23:13:46.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:46.399+0000 7fc1bbe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc1c4153950 0x7fc1c40c15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:46.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:46.399+0000 7fc1bbe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc1c4153950 0x7fc1c40c15d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54382/0 (socket says 172.21.15.28:54382) 2024-02-21T23:13:46.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:46.399+0000 7fc1c2800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fc1c415b6d0 0x7fc1c415dac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39006/0 (socket says 172.21.15.28:39006) 2024-02-21T23:13:46.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:46.399+0000 7fc1bbe00640 1 -- 172.21.15.28:0/2695114882 learned_addr learned my addr 172.21.15.28:0/2695114882 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:46.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:46.400+0000 7fc1bbe00640 1 -- 172.21.15.28:0/2695114882 >> v1:172.21.15.81:6789/0 conn(0x7fc1c40c11f0 legacy=0x7fc1c406dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:46.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:46.400+0000 7fc1bbe00640 1 -- 172.21.15.28:0/2695114882 >> v1:172.21.15.28:6789/0 conn(0x7fc1c415b6d0 legacy=0x7fc1c415dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:46.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:46.400+0000 7fc1bbe00640 1 -- 172.21.15.28:0/2695114882 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1c4153430 con 0x7fc1c4153950 2024-02-21T23:13:46.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:46.400+0000 7fc1c1400640 1 -- 172.21.15.28:0/2695114882 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 219648972 0 0) 0x7fc1c4152e70 con 0x7fc1c415b6d0 2024-02-21T23:13:46.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:46.401+0000 7fc1bbe00640 1 --2- 172.21.15.28:0/2695114882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1c4153950 0x7fc1c40c15d0 secure :-1 s=READY pgs=2424 cs=0 l=1 rev1=1 crypto rx=0x7fc1ac009870 tx=0x7fc1ac02b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0cf35ca446f40a7 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:46.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:46.401+0000 7fc1c1400640 1 -- 172.21.15.28:0/2695114882 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1ac034070 con 0x7fc1c4153950 2024-02-21T23:13:46.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:46.401+0000 7fc1c1400640 1 -- 172.21.15.28:0/2695114882 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1ac030850 con 0x7fc1c4153950 2024-02-21T23:13:46.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:46.401+0000 7fc1c1400640 1 -- 172.21.15.28:0/2695114882 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1ac030c20 con 0x7fc1c4153950 2024-02-21T23:13:46.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:46.402+0000 7fc1c8e00640 1 -- 172.21.15.28:0/2695114882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1c4153950 msgr2=0x7fc1c40c15d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:46.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:46.402+0000 7fc1c8e00640 1 --2- 172.21.15.28:0/2695114882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1c4153950 0x7fc1c40c15d0 secure :-1 s=READY pgs=2424 cs=0 l=1 rev1=1 crypto rx=0x7fc1ac009870 tx=0x7fc1ac02b990 comp rx=0 tx=0).stop 2024-02-21T23:13:46.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:46.402+0000 7fc1c8e00640 1 -- 172.21.15.28:0/2695114882 shutdown_connections 2024-02-21T23:13:46.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:46.402+0000 7fc1c8e00640 1 --2- 172.21.15.28:0/2695114882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1c4153950 0x7fc1c40c15d0 unknown :-1 s=CLOSED pgs=2424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:46.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:46.402+0000 7fc1c8e00640 1 -- 172.21.15.28:0/2695114882 >> 172.21.15.28:0/2695114882 conn(0x7fc1c40c67f0 msgr2=0x7fc1c40c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:46.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:46.402+0000 7fc1c8e00640 1 -- 172.21.15.28:0/2695114882 shutdown_connections 2024-02-21T23:13:46.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:46.402+0000 7fc1c8e00640 1 -- 172.21.15.28:0/2695114882 wait complete. 2024-02-21T23:13:46.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:46.403+0000 7fc1c8e00640 1 Processor -- start 2024-02-21T23:13:46.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:46.404+0000 7fc1c8e00640 1 -- start start 2024-02-21T23:13:46.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:46.404+0000 7fc1c8e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc1c40c11f0 0x7fc1c4140d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:46.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:46.404+0000 7fc1c8e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc1c4153950 0x7fc1c4141240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:46.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:46.404+0000 7fc1c8e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1c415b6d0 0x7fc1c4141bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:46.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:46.405+0000 7fc1c8e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c40a6110 con 0x7fc1c415b6d0 2024-02-21T23:13:46.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:46.405+0000 7fc1c8e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c40c4c00 con 0x7fc1c40c11f0 2024-02-21T23:13:46.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:46.405+0000 7fc1c8e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c40c0d90 con 0x7fc1c4153950 2024-02-21T23:13:46.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:46.405+0000 7fc1c2800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1c415b6d0 0x7fc1c4141bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:46.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:46.405+0000 7fc1c2800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1c415b6d0 0x7fc1c4141bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54384/0 (socket says 172.21.15.28:54384) 2024-02-21T23:13:46.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:46.405+0000 7fc1c2800640 1 -- 172.21.15.28:0/270647291 learned_addr learned my addr 172.21.15.28:0/270647291 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:46.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:46.405+0000 7fc1bbe00640 1 --2- 172.21.15.28:0/270647291 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc1c4153950 0x7fc1c4141240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:46.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:46.405+0000 7fc1c1e00640 1 --2- 172.21.15.28:0/270647291 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc1c40c11f0 0x7fc1c4140d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:46.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:46.405+0000 7fc1c2800640 1 -- 172.21.15.28:0/270647291 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc1c4153950 msgr2=0x7fc1c4141240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:46.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:46.405+0000 7fc1c2800640 1 --2- 172.21.15.28:0/270647291 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc1c4153950 0x7fc1c4141240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:46.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:46.405+0000 7fc1c2800640 1 -- 172.21.15.28:0/270647291 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc1c40c11f0 msgr2=0x7fc1c4140d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:46.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:46.405+0000 7fc1c2800640 1 --2- 172.21.15.28:0/270647291 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc1c40c11f0 0x7fc1c4140d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:46.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:46.405+0000 7fc1c2800640 1 -- 172.21.15.28:0/270647291 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1c40a7a40 con 0x7fc1c415b6d0 2024-02-21T23:13:46.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:46.406+0000 7fc1c2800640 1 --2- 172.21.15.28:0/270647291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1c415b6d0 0x7fc1c4141bc0 secure :-1 s=READY pgs=2425 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc00c7d0 tx=0x7fc1bc00cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:46.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:46.406+0000 7fc1ba000640 1 -- 172.21.15.28:0/270647291 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1bc018070 con 0x7fc1c415b6d0 2024-02-21T23:13:46.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:46.406+0000 7fc1ba000640 1 -- 172.21.15.28:0/270647291 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1bc00a640 con 0x7fc1c415b6d0 2024-02-21T23:13:46.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:46.406+0000 7fc1c8e00640 1 -- 172.21.15.28:0/270647291 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1c4144640 con 0x7fc1c415b6d0 2024-02-21T23:13:46.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:46.406+0000 7fc1ba000640 1 -- 172.21.15.28:0/270647291 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1bc00aa70 con 0x7fc1c415b6d0 2024-02-21T23:13:46.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:46.406+0000 7fc1c8e00640 1 -- 172.21.15.28:0/270647291 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1c40a81d0 con 0x7fc1c415b6d0 2024-02-21T23:13:46.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:46.407+0000 7fc1c8e00640 1 -- 172.21.15.28:0/270647291 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1c4067f30 con 0x7fc1c415b6d0 2024-02-21T23:13:46.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:46.409+0000 7fc1ba000640 1 -- 172.21.15.28:0/270647291 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc1bc01b070 con 0x7fc1c415b6d0 2024-02-21T23:13:46.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:46.410+0000 7fc1ba000640 1 --2- 172.21.15.28:0/270647291 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc1a006b500 0x7fc1a006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:46.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:46.410+0000 7fc1ba000640 1 -- 172.21.15.28:0/270647291 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc1bc089910 con 0x7fc1c415b6d0 2024-02-21T23:13:46.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:46.411+0000 7fc1c1e00640 1 --2- 172.21.15.28:0/270647291 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc1a006b500 0x7fc1a006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:46.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:46.412+0000 7fc1c1e00640 1 --2- 172.21.15.28:0/270647291 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc1a006b500 0x7fc1a006d9c0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fc1c414c8b0 tx=0x7fc1b4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:46.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:46.414+0000 7fc1ba000640 1 -- 172.21.15.28:0/270647291 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc1bc0585f0 con 0x7fc1c415b6d0 2024-02-21T23:13:46.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:46.655+0000 7fc1c8e00640 1 -- 172.21.15.28:0/270647291 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc1c40c0960 con 0x7fc1c415b6d0 2024-02-21T23:13:46.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:46.656+0000 7fc1ba000640 1 -- 172.21.15.28:0/270647291 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 234 v234) v1 ==== 95+0+4964 (secure 0 0 0) 0x7fc1bc057ef0 con 0x7fc1c415b6d0 2024-02-21T23:13:46.644 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:13:46.644 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":234,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7265,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":224},{"gid":8287,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1607587459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1607587459},{"type":"v1","addr":"172.21.15.81:6835","nonce":1607587459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":224}],"filesystems":[{"mdsmap":{"epoch":234,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:13:42.353072+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":8290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8290":{"gid":8290,"name":"a","rank":1,"incarnation":231,"state":"up:stopping","state_seq":5,"addr":"172.21.15.28:6833/1864206570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1864206570},{"type":"v1","addr":"172.21.15.28:6833","nonce":1864206570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:13:46.644 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 234 2024-02-21T23:13:46.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:46.659+0000 7fc1c8e00640 1 -- 172.21.15.28:0/270647291 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc1a006b500 msgr2=0x7fc1a006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:46.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:46.659+0000 7fc1c8e00640 1 --2- 172.21.15.28:0/270647291 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc1a006b500 0x7fc1a006d9c0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fc1c414c8b0 tx=0x7fc1b4009040 comp rx=0 tx=0).stop 2024-02-21T23:13:46.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:46.659+0000 7fc1c8e00640 1 -- 172.21.15.28:0/270647291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1c415b6d0 msgr2=0x7fc1c4141bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:46.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:46.659+0000 7fc1c8e00640 1 --2- 172.21.15.28:0/270647291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1c415b6d0 0x7fc1c4141bc0 secure :-1 s=READY pgs=2425 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc00c7d0 tx=0x7fc1bc00cc90 comp rx=0 tx=0).stop 2024-02-21T23:13:46.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:46.660+0000 7fc1c8e00640 1 -- 172.21.15.28:0/270647291 shutdown_connections 2024-02-21T23:13:46.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:46.660+0000 7fc1c8e00640 1 --2- 172.21.15.28:0/270647291 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc1a006b500 0x7fc1a006d9c0 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:46.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:46.660+0000 7fc1c8e00640 1 --2- 172.21.15.28:0/270647291 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc1c4153950 0x7fc1c4141240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:46.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:46.660+0000 7fc1c8e00640 1 --2- 172.21.15.28:0/270647291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1c415b6d0 0x7fc1c4141bc0 unknown :-1 s=CLOSED pgs=2425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:46.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:46.660+0000 7fc1c8e00640 1 --2- 172.21.15.28:0/270647291 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc1c40c11f0 0x7fc1c4140d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:46.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:46.660+0000 7fc1c8e00640 1 -- 172.21.15.28:0/270647291 >> 172.21.15.28:0/270647291 conn(0x7fc1c40c67f0 msgr2=0x7fc1c40c5370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:46.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:46.660+0000 7fc1c8e00640 1 -- 172.21.15.28:0/270647291 shutdown_connections 2024-02-21T23:13:46.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:46.660+0000 7fc1c8e00640 1 -- 172.21.15.28:0/270647291 wait complete. 2024-02-21T23:13:47.069 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 234, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:13:42.353072+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 8290}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8290': {'gid': 8290, 'name': 'a', 'rank': 1, 'incarnation': 231, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.28:6833/1864206570', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 1864206570}, {'type': 'v1', 'addr': '172.21.15.28:6833', 'nonce': 1864206570}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:13:47.069 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8290:up:stopping 2024-02-21T23:13:48.069 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:13:48.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:48.305+0000 7f659b000640 1 Processor -- start 2024-02-21T23:13:48.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:48.305+0000 7f659b000640 1 -- start start 2024-02-21T23:13:48.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:48.307+0000 7f659b000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f659406bad0 0x7f65940a9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:48.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:48.307+0000 7f659b000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f659406b6d0 con 0x7f659406f130 2024-02-21T23:13:48.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:48.307+0000 7f659b000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f659406fc80 con 0x7f65940c4d20 2024-02-21T23:13:48.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:48.307+0000 7f659b000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f659406ff60 con 0x7f659406bad0 2024-02-21T23:13:48.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:48.307+0000 7f6599c00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f659406f130 0x7f6594158580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39012/0 (socket says 172.21.15.28:39012) 2024-02-21T23:13:48.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:48.307+0000 7f6599c00640 1 -- 172.21.15.28:0/2606638377 learned_addr learned my addr 172.21.15.28:0/2606638377 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:13:48.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:48.307+0000 7f6599200640 1 --2- 172.21.15.28:0/2606638377 >> v2:172.21.15.28:3300/0 conn(0x7f659406bad0 0x7f65940a9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:48.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:48.308+0000 7f6599200640 1 -- 172.21.15.28:0/2606638377 >> v1:172.21.15.81:6789/0 conn(0x7f65940c4d20 legacy=0x7f659415f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:48.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:48.308+0000 7f6599200640 1 -- 172.21.15.28:0/2606638377 >> v1:172.21.15.28:6789/0 conn(0x7f659406f130 legacy=0x7f6594158580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:48.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:48.308+0000 7f6599200640 1 -- 172.21.15.28:0/2606638377 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6594070240 con 0x7f659406bad0 2024-02-21T23:13:48.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:48.308+0000 7f6593e00640 1 -- 172.21.15.28:0/2606638377 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2387266465 0 0) 0x7f659406b6d0 con 0x7f659406f130 2024-02-21T23:13:48.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:48.309+0000 7f6599200640 1 --2- 172.21.15.28:0/2606638377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f659406bad0 0x7f65940a9eb0 secure :-1 s=READY pgs=2427 cs=0 l=1 rev1=1 crypto rx=0x7f65800097a0 tx=0x7f658002b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=501878dfaceea7dc server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:48.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:48.310+0000 7f6593e00640 1 -- 172.21.15.28:0/2606638377 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6580038070 con 0x7f659406bad0 2024-02-21T23:13:48.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:48.310+0000 7f6593e00640 1 -- 172.21.15.28:0/2606638377 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6580031700 con 0x7f659406bad0 2024-02-21T23:13:48.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:48.310+0000 7f6593e00640 1 -- 172.21.15.28:0/2606638377 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6580031ad0 con 0x7f659406bad0 2024-02-21T23:13:48.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:48.310+0000 7f659b000640 1 -- 172.21.15.28:0/2606638377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f659406bad0 msgr2=0x7f65940a9eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:48.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:48.310+0000 7f659b000640 1 --2- 172.21.15.28:0/2606638377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f659406bad0 0x7f65940a9eb0 secure :-1 s=READY pgs=2427 cs=0 l=1 rev1=1 crypto rx=0x7f65800097a0 tx=0x7f658002b7a0 comp rx=0 tx=0).stop 2024-02-21T23:13:48.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:48.310+0000 7f659b000640 1 -- 172.21.15.28:0/2606638377 shutdown_connections 2024-02-21T23:13:48.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:48.310+0000 7f659b000640 1 --2- 172.21.15.28:0/2606638377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f659406bad0 0x7f65940a9eb0 unknown :-1 s=CLOSED pgs=2427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:48.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:48.310+0000 7f659b000640 1 -- 172.21.15.28:0/2606638377 >> 172.21.15.28:0/2606638377 conn(0x7f6594067050 msgr2=0x7f65940a84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:48.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:48.310+0000 7f659b000640 1 -- 172.21.15.28:0/2606638377 shutdown_connections 2024-02-21T23:13:48.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:48.310+0000 7f659b000640 1 -- 172.21.15.28:0/2606638377 wait complete. 2024-02-21T23:13:48.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:48.312+0000 7f659b000640 1 Processor -- start 2024-02-21T23:13:48.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:48.312+0000 7f659b000640 1 -- start start 2024-02-21T23:13:48.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:48.313+0000 7f659b000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f659406bad0 0x7f65940c2220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:48.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:48.313+0000 7f659b000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f659406f130 0x7f659414e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:48.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:48.313+0000 7f659b000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65940c4d20 0x7f65940c6670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:48.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:48.313+0000 7f659b000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65940a9990 con 0x7f659406bad0 2024-02-21T23:13:48.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:48.313+0000 7f659b000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65940a96b0 con 0x7f65940c4d20 2024-02-21T23:13:48.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:48.313+0000 7f659b000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6594158c90 con 0x7f659406f130 2024-02-21T23:13:48.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:48.314+0000 7f6599c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f659406bad0 0x7f65940c2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:48.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:48.314+0000 7f6599c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f659406bad0 0x7f65940c2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54412/0 (socket says 172.21.15.28:54412) 2024-02-21T23:13:48.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:48.314+0000 7f6599c00640 1 -- 172.21.15.28:0/1404842824 learned_addr learned my addr 172.21.15.28:0/1404842824 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:48.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:48.314+0000 7f6599200640 1 --2- 172.21.15.28:0/1404842824 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f659406f130 0x7f659414e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:48.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:48.314+0000 7f659a600640 1 --2- 172.21.15.28:0/1404842824 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65940c4d20 0x7f65940c6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:48.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:48.314+0000 7f6599c00640 1 -- 172.21.15.28:0/1404842824 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f659406f130 msgr2=0x7f659414e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:48.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:48.314+0000 7f6599c00640 1 --2- 172.21.15.28:0/1404842824 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f659406f130 0x7f659414e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:48.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:48.314+0000 7f6599c00640 1 -- 172.21.15.28:0/1404842824 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65940c4d20 msgr2=0x7f65940c6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:48.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:48.314+0000 7f6599c00640 1 --2- 172.21.15.28:0/1404842824 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65940c4d20 0x7f65940c6670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:48.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:48.314+0000 7f6599c00640 1 -- 172.21.15.28:0/1404842824 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f659414d110 con 0x7f659406bad0 2024-02-21T23:13:48.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:48.314+0000 7f6599200640 1 --2- 172.21.15.28:0/1404842824 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f659406f130 0x7f659414e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:13:48.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:48.314+0000 7f6599c00640 1 --2- 172.21.15.28:0/1404842824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f659406bad0 0x7f65940c2220 secure :-1 s=READY pgs=2428 cs=0 l=1 rev1=1 crypto rx=0x7f658800c460 tx=0x7f658800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:48.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:48.315+0000 7f6591600640 1 -- 172.21.15.28:0/1404842824 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6588018070 con 0x7f659406bad0 2024-02-21T23:13:48.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:48.315+0000 7f6591600640 1 -- 172.21.15.28:0/1404842824 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6588007500 con 0x7f659406bad0 2024-02-21T23:13:48.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:48.315+0000 7f6591600640 1 -- 172.21.15.28:0/1404842824 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6588004a20 con 0x7f659406bad0 2024-02-21T23:13:48.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:48.315+0000 7f659b000640 1 -- 172.21.15.28:0/1404842824 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f659414d3f0 con 0x7f659406bad0 2024-02-21T23:13:48.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:48.315+0000 7f659b000640 1 -- 172.21.15.28:0/1404842824 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65940c6bb0 con 0x7f659406bad0 2024-02-21T23:13:48.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:48.316+0000 7f659b000640 1 -- 172.21.15.28:0/1404842824 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f659404f290 con 0x7f659406bad0 2024-02-21T23:13:48.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:48.317+0000 7f6591600640 1 -- 172.21.15.28:0/1404842824 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f658801b070 con 0x7f659406bad0 2024-02-21T23:13:48.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:48.318+0000 7f6591600640 1 --2- 172.21.15.28:0/1404842824 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f657406b500 0x7f657406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:48.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:48.318+0000 7f6591600640 1 -- 172.21.15.28:0/1404842824 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f65880895e0 con 0x7f659406bad0 2024-02-21T23:13:48.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:48.319+0000 7f6599200640 1 --2- 172.21.15.28:0/1404842824 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f657406b500 0x7f657406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:48.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:48.320+0000 7f6599200640 1 --2- 172.21.15.28:0/1404842824 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f657406b500 0x7f657406d9c0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f6580035000 tx=0x7f6580008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:48.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:48.323+0000 7f6591600640 1 -- 172.21.15.28:0/1404842824 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f65880582d0 con 0x7f659406bad0 2024-02-21T23:13:48.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:48.565+0000 7f659b000640 1 -- 172.21.15.28:0/1404842824 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f659414d5c0 con 0x7f659406bad0 2024-02-21T23:13:48.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:48.566+0000 7f6591600640 1 -- 172.21.15.28:0/1404842824 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 234 v234) v1 ==== 95+0+4964 (secure 0 0 0) 0x7f6588057bd0 con 0x7f659406bad0 2024-02-21T23:13:48.554 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:13:48.554 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":234,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7265,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":224},{"gid":8287,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1607587459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1607587459},{"type":"v1","addr":"172.21.15.81:6835","nonce":1607587459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":224}],"filesystems":[{"mdsmap":{"epoch":234,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:13:42.353072+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":8290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8290":{"gid":8290,"name":"a","rank":1,"incarnation":231,"state":"up:stopping","state_seq":5,"addr":"172.21.15.28:6833/1864206570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1864206570},{"type":"v1","addr":"172.21.15.28:6833","nonce":1864206570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:13:48.554 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 234 2024-02-21T23:13:48.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:48.569+0000 7f659b000640 1 -- 172.21.15.28:0/1404842824 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f657406b500 msgr2=0x7f657406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:48.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:48.569+0000 7f659b000640 1 --2- 172.21.15.28:0/1404842824 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f657406b500 0x7f657406d9c0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f6580035000 tx=0x7f6580008030 comp rx=0 tx=0).stop 2024-02-21T23:13:48.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:48.569+0000 7f659b000640 1 -- 172.21.15.28:0/1404842824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f659406bad0 msgr2=0x7f65940c2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:48.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:48.569+0000 7f659b000640 1 --2- 172.21.15.28:0/1404842824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f659406bad0 0x7f65940c2220 secure :-1 s=READY pgs=2428 cs=0 l=1 rev1=1 crypto rx=0x7f658800c460 tx=0x7f658800c920 comp rx=0 tx=0).stop 2024-02-21T23:13:48.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:48.569+0000 7f659b000640 1 -- 172.21.15.28:0/1404842824 shutdown_connections 2024-02-21T23:13:48.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:48.569+0000 7f659b000640 1 --2- 172.21.15.28:0/1404842824 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f657406b500 0x7f657406d9c0 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:48.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:48.569+0000 7f659b000640 1 --2- 172.21.15.28:0/1404842824 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f659406f130 0x7f659414e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:48.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:48.569+0000 7f659b000640 1 --2- 172.21.15.28:0/1404842824 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65940c4d20 0x7f65940c6670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:48.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:48.569+0000 7f659b000640 1 --2- 172.21.15.28:0/1404842824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f659406bad0 0x7f65940c2220 unknown :-1 s=CLOSED pgs=2428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:48.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:48.569+0000 7f659b000640 1 -- 172.21.15.28:0/1404842824 >> 172.21.15.28:0/1404842824 conn(0x7f6594067050 msgr2=0x7f65940a84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:48.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:48.569+0000 7f659b000640 1 -- 172.21.15.28:0/1404842824 shutdown_connections 2024-02-21T23:13:48.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:48.569+0000 7f659b000640 1 -- 172.21.15.28:0/1404842824 wait complete. 2024-02-21T23:13:48.977 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 234, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:13:42.353072+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 8290}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8290': {'gid': 8290, 'name': 'a', 'rank': 1, 'incarnation': 231, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.28:6833/1864206570', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 1864206570}, {'type': 'v1', 'addr': '172.21.15.28:6833', 'nonce': 1864206570}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:13:48.978 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8290:up:stopping 2024-02-21T23:13:49.979 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:13:50.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.216+0000 7fc8b3e00640 1 Processor -- start 2024-02-21T23:13:50.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.216+0000 7fc8b3e00640 1 -- start start 2024-02-21T23:13:50.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.217+0000 7fc8b3e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc8b40aff90 0x7fc8b40c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:50.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.217+0000 7fc8b3e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fc8b406dbf0 0x7fc8b4158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:50.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.217+0000 7fc8b3e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8b40c74c0 con 0x7fc8b40c5720 2024-02-21T23:13:50.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.217+0000 7fc8b3e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8b40c7270 con 0x7fc8b40aff90 2024-02-21T23:13:50.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.217+0000 7fc8b3e00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8b4158630 con 0x7fc8b406dbf0 2024-02-21T23:13:50.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.217+0000 7fc8b2a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc8b40aff90 0x7fc8b40c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:50.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.218+0000 7fc8b2a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc8b40aff90 0x7fc8b40c51e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54428/0 (socket says 172.21.15.28:54428) 2024-02-21T23:13:50.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.218+0000 7fc8b2a00640 1 -- 172.21.15.28:0/1409656740 learned_addr learned my addr 172.21.15.28:0/1409656740 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:50.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.218+0000 7fc8abe00640 1 --2- 172.21.15.28:0/1409656740 >> v2:172.21.15.81:3300/0 conn(0x7fc8b406dbf0 0x7fc8b4158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:50.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.218+0000 7fc8b2a00640 1 -- 172.21.15.28:0/1409656740 >> v2:172.21.15.81:3300/0 conn(0x7fc8b406dbf0 msgr2=0x7fc8b4158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:50.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.218+0000 7fc8b2a00640 1 --2- 172.21.15.28:0/1409656740 >> v2:172.21.15.81:3300/0 conn(0x7fc8b406dbf0 0x7fc8b4158ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:50.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.218+0000 7fc8b2a00640 1 -- 172.21.15.28:0/1409656740 >> v1:172.21.15.81:6789/0 conn(0x7fc8b40c5720 legacy=0x7fc8b406c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:50.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.218+0000 7fc8b2a00640 1 -- 172.21.15.28:0/1409656740 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8b40c77a0 con 0x7fc8b40aff90 2024-02-21T23:13:50.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.219+0000 7fc8b2a00640 1 --2- 172.21.15.28:0/1409656740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8b40aff90 0x7fc8b40c51e0 secure :-1 s=READY pgs=2429 cs=0 l=1 rev1=1 crypto rx=0x7fc8b40c73f0 tx=0x7fc8a402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6858faa031fa0587 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:50.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.219+0000 7fc8b2000640 1 -- 172.21.15.28:0/1409656740 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8a4035020 con 0x7fc8b40aff90 2024-02-21T23:13:50.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.220+0000 7fc8b2000640 1 -- 172.21.15.28:0/1409656740 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8a4031870 con 0x7fc8b40aff90 2024-02-21T23:13:50.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.220+0000 7fc8b2000640 1 -- 172.21.15.28:0/1409656740 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8a4031c40 con 0x7fc8b40aff90 2024-02-21T23:13:50.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.220+0000 7fc8b3e00640 1 -- 172.21.15.28:0/1409656740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8b40aff90 msgr2=0x7fc8b40c51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:50.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.220+0000 7fc8b3e00640 1 --2- 172.21.15.28:0/1409656740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8b40aff90 0x7fc8b40c51e0 secure :-1 s=READY pgs=2429 cs=0 l=1 rev1=1 crypto rx=0x7fc8b40c73f0 tx=0x7fc8a402b910 comp rx=0 tx=0).stop 2024-02-21T23:13:50.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.220+0000 7fc8b3e00640 1 -- 172.21.15.28:0/1409656740 shutdown_connections 2024-02-21T23:13:50.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.220+0000 7fc8b3e00640 1 --2- 172.21.15.28:0/1409656740 >> v2:172.21.15.81:3300/0 conn(0x7fc8b406dbf0 0x7fc8b4158ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:50.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.220+0000 7fc8b3e00640 1 --2- 172.21.15.28:0/1409656740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8b40aff90 0x7fc8b40c51e0 unknown :-1 s=CLOSED pgs=2429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:50.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.220+0000 7fc8b3e00640 1 -- 172.21.15.28:0/1409656740 >> 172.21.15.28:0/1409656740 conn(0x7fc8b40ac370 msgr2=0x7fc8b40ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:50.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.220+0000 7fc8b3e00640 1 -- 172.21.15.28:0/1409656740 shutdown_connections 2024-02-21T23:13:50.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.220+0000 7fc8b3e00640 1 -- 172.21.15.28:0/1409656740 wait complete. 2024-02-21T23:13:50.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.223+0000 7fc8b3e00640 1 Processor -- start 2024-02-21T23:13:50.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.223+0000 7fc8b3e00640 1 -- start start 2024-02-21T23:13:50.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.225+0000 7fc8b3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8b406dbf0 0x7fc8b4152c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:50.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.225+0000 7fc8b3e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc8b40aff90 0x7fc8b41eb800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:50.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.225+0000 7fc8b3e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc8b40c5720 0x7fc8b40c0f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:50.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.225+0000 7fc8b3e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b40c8760 con 0x7fc8b406dbf0 2024-02-21T23:13:50.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.225+0000 7fc8b3e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b40be0c0 con 0x7fc8b40aff90 2024-02-21T23:13:50.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.225+0000 7fc8b3e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b40c88e0 con 0x7fc8b40c5720 2024-02-21T23:13:50.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.225+0000 7fc8b2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8b406dbf0 0x7fc8b4152c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:50.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.225+0000 7fc8b2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8b406dbf0 0x7fc8b4152c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54440/0 (socket says 172.21.15.28:54440) 2024-02-21T23:13:50.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.225+0000 7fc8b2a00640 1 -- 172.21.15.28:0/198085133 learned_addr learned my addr 172.21.15.28:0/198085133 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:50.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.225+0000 7fc8abe00640 1 --2- 172.21.15.28:0/198085133 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc8b40aff90 0x7fc8b41eb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:50.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.225+0000 7fc8b3400640 1 --2- 172.21.15.28:0/198085133 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc8b40c5720 0x7fc8b40c0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:50.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.225+0000 7fc8b2a00640 1 -- 172.21.15.28:0/198085133 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc8b40c5720 msgr2=0x7fc8b40c0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:50.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.225+0000 7fc8b2a00640 1 --2- 172.21.15.28:0/198085133 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc8b40c5720 0x7fc8b40c0f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:50.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.225+0000 7fc8b2a00640 1 -- 172.21.15.28:0/198085133 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc8b40aff90 msgr2=0x7fc8b41eb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:50.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.225+0000 7fc8b2a00640 1 --2- 172.21.15.28:0/198085133 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc8b40aff90 0x7fc8b41eb800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:50.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.225+0000 7fc8b2a00640 1 -- 172.21.15.28:0/198085133 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8b4153190 con 0x7fc8b406dbf0 2024-02-21T23:13:50.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.225+0000 7fc8abe00640 1 --2- 172.21.15.28:0/198085133 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc8b40aff90 0x7fc8b41eb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:13:50.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.225+0000 7fc8b3400640 1 --2- 172.21.15.28:0/198085133 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc8b40c5720 0x7fc8b40c0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:13:50.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.226+0000 7fc8b2a00640 1 --2- 172.21.15.28:0/198085133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8b406dbf0 0x7fc8b4152c50 secure :-1 s=READY pgs=2430 cs=0 l=1 rev1=1 crypto rx=0x7fc8a4000c00 tx=0x7fc8a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:50.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.226+0000 7fc8aaa00640 1 -- 172.21.15.28:0/198085133 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8a40443f0 con 0x7fc8b406dbf0 2024-02-21T23:13:50.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.226+0000 7fc8aaa00640 1 -- 172.21.15.28:0/198085133 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8a4044590 con 0x7fc8b406dbf0 2024-02-21T23:13:50.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.226+0000 7fc8aaa00640 1 -- 172.21.15.28:0/198085133 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8a4044980 con 0x7fc8b406dbf0 2024-02-21T23:13:50.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.226+0000 7fc8b3e00640 1 -- 172.21.15.28:0/198085133 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8b40ae890 con 0x7fc8b406dbf0 2024-02-21T23:13:50.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.226+0000 7fc8b3e00640 1 -- 172.21.15.28:0/198085133 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8b41eb020 con 0x7fc8b406dbf0 2024-02-21T23:13:50.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.227+0000 7fc8b3e00640 1 -- 172.21.15.28:0/198085133 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8b404f290 con 0x7fc8b406dbf0 2024-02-21T23:13:50.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.229+0000 7fc8aaa00640 1 -- 172.21.15.28:0/198085133 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc8a404b060 con 0x7fc8b406dbf0 2024-02-21T23:13:50.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.230+0000 7fc8aaa00640 1 --2- 172.21.15.28:0/198085133 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc89006b4d0 0x7fc89006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:50.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.230+0000 7fc8aaa00640 1 -- 172.21.15.28:0/198085133 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc8a40ac6d0 con 0x7fc8b406dbf0 2024-02-21T23:13:50.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.230+0000 7fc8abe00640 1 --2- 172.21.15.28:0/198085133 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc89006b4d0 0x7fc89006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:50.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.231+0000 7fc8abe00640 1 --2- 172.21.15.28:0/198085133 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc89006b4d0 0x7fc89006d990 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fc8b40c5580 tx=0x7fc89c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:50.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.238+0000 7fc8aaa00640 1 -- 172.21.15.28:0/198085133 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc8a403f070 con 0x7fc8b406dbf0 2024-02-21T23:13:50.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.480+0000 7fc8b3e00640 1 -- 172.21.15.28:0/198085133 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc8b40bdad0 con 0x7fc8b406dbf0 2024-02-21T23:13:50.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.481+0000 7fc8aaa00640 1 -- 172.21.15.28:0/198085133 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 234 v234) v1 ==== 95+0+4964 (secure 0 0 0) 0x7fc8a407b3b0 con 0x7fc8b406dbf0 2024-02-21T23:13:50.469 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:13:50.469 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":234,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7265,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":224},{"gid":8287,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1607587459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1607587459},{"type":"v1","addr":"172.21.15.81:6835","nonce":1607587459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":224}],"filesystems":[{"mdsmap":{"epoch":234,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:13:42.353072+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":8290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8290":{"gid":8290,"name":"a","rank":1,"incarnation":231,"state":"up:stopping","state_seq":5,"addr":"172.21.15.28:6833/1864206570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1864206570},{"type":"v1","addr":"172.21.15.28:6833","nonce":1864206570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:13:50.470 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 234 2024-02-21T23:13:50.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.484+0000 7fc8b3e00640 1 -- 172.21.15.28:0/198085133 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc89006b4d0 msgr2=0x7fc89006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:50.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.484+0000 7fc8b3e00640 1 --2- 172.21.15.28:0/198085133 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc89006b4d0 0x7fc89006d990 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fc8b40c5580 tx=0x7fc89c008040 comp rx=0 tx=0).stop 2024-02-21T23:13:50.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.484+0000 7fc8b3e00640 1 -- 172.21.15.28:0/198085133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8b406dbf0 msgr2=0x7fc8b4152c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:50.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.484+0000 7fc8b3e00640 1 --2- 172.21.15.28:0/198085133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8b406dbf0 0x7fc8b4152c50 secure :-1 s=READY pgs=2430 cs=0 l=1 rev1=1 crypto rx=0x7fc8a4000c00 tx=0x7fc8a40046c0 comp rx=0 tx=0).stop 2024-02-21T23:13:50.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.484+0000 7fc8b3e00640 1 -- 172.21.15.28:0/198085133 shutdown_connections 2024-02-21T23:13:50.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.484+0000 7fc8b3e00640 1 --2- 172.21.15.28:0/198085133 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc89006b4d0 0x7fc89006d990 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:50.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.484+0000 7fc8b3e00640 1 --2- 172.21.15.28:0/198085133 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc8b40c5720 0x7fc8b40c0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:50.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.484+0000 7fc8b3e00640 1 --2- 172.21.15.28:0/198085133 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc8b40aff90 0x7fc8b41eb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:50.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.484+0000 7fc8b3e00640 1 --2- 172.21.15.28:0/198085133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8b406dbf0 0x7fc8b4152c50 unknown :-1 s=CLOSED pgs=2430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:50.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.484+0000 7fc8b3e00640 1 -- 172.21.15.28:0/198085133 >> 172.21.15.28:0/198085133 conn(0x7fc8b40ac370 msgr2=0x7fc8b40ac750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:50.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.484+0000 7fc8b3e00640 1 -- 172.21.15.28:0/198085133 shutdown_connections 2024-02-21T23:13:50.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:50.484+0000 7fc8b3e00640 1 -- 172.21.15.28:0/198085133 wait complete. 2024-02-21T23:13:50.894 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 234, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:13:42.353072+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 8290}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8290': {'gid': 8290, 'name': 'a', 'rank': 1, 'incarnation': 231, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.28:6833/1864206570', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 1864206570}, {'type': 'v1', 'addr': '172.21.15.28:6833', 'nonce': 1864206570}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:13:50.895 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8290:up:stopping 2024-02-21T23:13:51.896 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:13:52.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.133+0000 7faa22200640 1 Processor -- start 2024-02-21T23:13:52.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.133+0000 7faa22200640 1 -- start start 2024-02-21T23:13:52.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.135+0000 7faa22200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faa1c0aff90 0x7faa1c0c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:52.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.135+0000 7faa22200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7faa1c06dbf0 0x7faa1c158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:52.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.135+0000 7faa22200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa1c0c74c0 con 0x7faa1c0c5720 2024-02-21T23:13:52.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.135+0000 7faa22200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa1c0c7270 con 0x7faa1c06dbf0 2024-02-21T23:13:52.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.135+0000 7faa22200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa1c158630 con 0x7faa1c0aff90 2024-02-21T23:13:52.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.135+0000 7faa21800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7faa1c0c5720 0x7faa1c06c270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39014/0 (socket says 172.21.15.28:39014) 2024-02-21T23:13:52.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.135+0000 7faa21800640 1 -- 172.21.15.28:0/982605828 learned_addr learned my addr 172.21.15.28:0/982605828 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:13:52.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.135+0000 7faa1be00640 1 --2- 172.21.15.28:0/982605828 >> v2:172.21.15.81:3300/0 conn(0x7faa1c06dbf0 0x7faa1c158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:52.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.135+0000 7faa20e00640 1 --2- 172.21.15.28:0/982605828 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faa1c0aff90 0x7faa1c0c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:52.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.136+0000 7faa1b400640 1 -- 172.21.15.28:0/982605828 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3416492735 0 0) 0x7faa1c0c74c0 con 0x7faa1c0c5720 2024-02-21T23:13:52.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.136+0000 7faa1b400640 1 -- 172.21.15.28:0/982605828 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa04003520 con 0x7faa1c0c5720 2024-02-21T23:13:52.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.136+0000 7faa1be00640 1 -- 172.21.15.28:0/982605828 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faa1c0aff90 msgr2=0x7faa1c0c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:52.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.136+0000 7faa1be00640 1 --2- 172.21.15.28:0/982605828 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faa1c0aff90 0x7faa1c0c51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:52.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.136+0000 7faa1be00640 1 -- 172.21.15.28:0/982605828 >> v1:172.21.15.28:6789/0 conn(0x7faa1c0c5720 legacy=0x7faa1c06c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:52.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.136+0000 7faa1b400640 1 -- 172.21.15.28:0/982605828 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4096746118 0 0) 0x7faa04003520 con 0x7faa1c0c5720 2024-02-21T23:13:52.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.136+0000 7faa1be00640 1 -- 172.21.15.28:0/982605828 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa1c0c77a0 con 0x7faa1c06dbf0 2024-02-21T23:13:52.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.138+0000 7faa1be00640 1 --2- 172.21.15.28:0/982605828 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faa1c06dbf0 0x7faa1c158ac0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7faa080097a0 tx=0x7faa0802b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ef4af22605973893 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:52.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.138+0000 7faa1b400640 1 -- 172.21.15.28:0/982605828 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa08037070 con 0x7faa1c06dbf0 2024-02-21T23:13:52.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.138+0000 7faa1b400640 1 -- 172.21.15.28:0/982605828 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa08030510 con 0x7faa1c06dbf0 2024-02-21T23:13:52.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.138+0000 7faa1b400640 1 -- 172.21.15.28:0/982605828 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa140019f0 con 0x7faa1c06dbf0 2024-02-21T23:13:52.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.139+0000 7faa22200640 1 -- 172.21.15.28:0/982605828 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faa1c06dbf0 msgr2=0x7faa1c158ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:52.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.139+0000 7faa22200640 1 --2- 172.21.15.28:0/982605828 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faa1c06dbf0 0x7faa1c158ac0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7faa080097a0 tx=0x7faa0802b790 comp rx=0 tx=0).stop 2024-02-21T23:13:52.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.139+0000 7faa22200640 1 -- 172.21.15.28:0/982605828 shutdown_connections 2024-02-21T23:13:52.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.139+0000 7faa22200640 1 --2- 172.21.15.28:0/982605828 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faa1c06dbf0 0x7faa1c158ac0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:52.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.139+0000 7faa22200640 1 --2- 172.21.15.28:0/982605828 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faa1c0aff90 0x7faa1c0c51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:52.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.139+0000 7faa22200640 1 -- 172.21.15.28:0/982605828 >> 172.21.15.28:0/982605828 conn(0x7faa1c0ac370 msgr2=0x7faa1c0ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:52.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.139+0000 7faa22200640 1 -- 172.21.15.28:0/982605828 shutdown_connections 2024-02-21T23:13:52.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.139+0000 7faa22200640 1 -- 172.21.15.28:0/982605828 wait complete. 2024-02-21T23:13:52.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.141+0000 7faa22200640 1 Processor -- start 2024-02-21T23:13:52.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.141+0000 7faa22200640 1 -- start start 2024-02-21T23:13:52.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.142+0000 7faa22200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faa1c06dbf0 0x7faa1c0c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:52.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.142+0000 7faa22200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faa1c0aff90 0x7faa1c1508b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:52.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.142+0000 7faa22200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa1c0c5720 0x7faa1c150df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:52.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.142+0000 7faa22200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa1c0c8760 con 0x7faa1c0c5720 2024-02-21T23:13:52.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.142+0000 7faa22200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa1c0be0c0 con 0x7faa1c06dbf0 2024-02-21T23:13:52.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.142+0000 7faa22200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa1c0c88e0 con 0x7faa1c0aff90 2024-02-21T23:13:52.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.142+0000 7faa21800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa1c0c5720 0x7faa1c150df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:52.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.142+0000 7faa21800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa1c0c5720 0x7faa1c150df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54444/0 (socket says 172.21.15.28:54444) 2024-02-21T23:13:52.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.142+0000 7faa21800640 1 -- 172.21.15.28:0/1727347066 learned_addr learned my addr 172.21.15.28:0/1727347066 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:52.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.143+0000 7faa20e00640 1 --2- 172.21.15.28:0/1727347066 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faa1c06dbf0 0x7faa1c0c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:52.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.143+0000 7faa1be00640 1 --2- 172.21.15.28:0/1727347066 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faa1c0aff90 0x7faa1c1508b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:52.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.143+0000 7faa21800640 1 -- 172.21.15.28:0/1727347066 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faa1c0aff90 msgr2=0x7faa1c1508b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:52.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.143+0000 7faa21800640 1 --2- 172.21.15.28:0/1727347066 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faa1c0aff90 0x7faa1c1508b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:52.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.143+0000 7faa21800640 1 -- 172.21.15.28:0/1727347066 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faa1c06dbf0 msgr2=0x7faa1c0c42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:52.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.143+0000 7faa21800640 1 --2- 172.21.15.28:0/1727347066 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faa1c06dbf0 0x7faa1c0c42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:52.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.143+0000 7faa21800640 1 -- 172.21.15.28:0/1727347066 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa1c0bfe80 con 0x7faa1c0c5720 2024-02-21T23:13:52.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.143+0000 7faa21800640 1 --2- 172.21.15.28:0/1727347066 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa1c0c5720 0x7faa1c150df0 secure :-1 s=READY pgs=2432 cs=0 l=1 rev1=1 crypto rx=0x7faa1400cc90 tx=0x7faa1400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:52.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.144+0000 7faa18c00640 1 -- 172.21.15.28:0/1727347066 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa1400ce40 con 0x7faa1c0c5720 2024-02-21T23:13:52.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.144+0000 7faa18c00640 1 -- 172.21.15.28:0/1727347066 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa14012400 con 0x7faa1c0c5720 2024-02-21T23:13:52.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.144+0000 7faa18c00640 1 -- 172.21.15.28:0/1727347066 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa140127f0 con 0x7faa1c0c5720 2024-02-21T23:13:52.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.144+0000 7faa22200640 1 -- 172.21.15.28:0/1727347066 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa1c0c1320 con 0x7faa1c0c5720 2024-02-21T23:13:52.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.144+0000 7faa22200640 1 -- 172.21.15.28:0/1727347066 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa1c0c3470 con 0x7faa1c0c5720 2024-02-21T23:13:52.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.145+0000 7faa22200640 1 -- 172.21.15.28:0/1727347066 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa1c04f290 con 0x7faa1c0c5720 2024-02-21T23:13:52.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.146+0000 7faa18c00640 1 -- 172.21.15.28:0/1727347066 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faa14012990 con 0x7faa1c0c5720 2024-02-21T23:13:52.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.147+0000 7faa18c00640 1 --2- 172.21.15.28:0/1727347066 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa9fc06b4d0 0x7fa9fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:52.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.147+0000 7faa18c00640 1 -- 172.21.15.28:0/1727347066 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 6101+0+0 (secure 0 0 0) 0x7faa1408ac50 con 0x7faa1c0c5720 2024-02-21T23:13:52.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.147+0000 7faa20e00640 1 --2- 172.21.15.28:0/1727347066 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa9fc06b4d0 0x7fa9fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:52.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.149+0000 7faa20e00640 1 --2- 172.21.15.28:0/1727347066 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa9fc06b4d0 0x7fa9fc06d990 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7faa10009ad0 tx=0x7faa10008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:52.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.151+0000 7faa18c00640 1 -- 172.21.15.28:0/1727347066 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7faa140598a0 con 0x7faa1c0c5720 2024-02-21T23:13:52.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.405+0000 7faa22200640 1 -- 172.21.15.28:0/1727347066 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faa1c06d9d0 con 0x7faa1c0c5720 2024-02-21T23:13:52.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.406+0000 7faa18c00640 1 -- 172.21.15.28:0/1727347066 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 234 v234) v1 ==== 95+0+4964 (secure 0 0 0) 0x7faa140591a0 con 0x7faa1c0c5720 2024-02-21T23:13:52.393 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:13:52.394 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":234,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7265,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":224},{"gid":8287,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1607587459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1607587459},{"type":"v1","addr":"172.21.15.81:6835","nonce":1607587459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":224}],"filesystems":[{"mdsmap":{"epoch":234,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:13:42.353072+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":8290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8290":{"gid":8290,"name":"a","rank":1,"incarnation":231,"state":"up:stopping","state_seq":5,"addr":"172.21.15.28:6833/1864206570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1864206570},{"type":"v1","addr":"172.21.15.28:6833","nonce":1864206570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:13:52.394 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 234 2024-02-21T23:13:52.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.408+0000 7faa22200640 1 -- 172.21.15.28:0/1727347066 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa9fc06b4d0 msgr2=0x7fa9fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:52.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.408+0000 7faa22200640 1 --2- 172.21.15.28:0/1727347066 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa9fc06b4d0 0x7fa9fc06d990 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7faa10009ad0 tx=0x7faa10008040 comp rx=0 tx=0).stop 2024-02-21T23:13:52.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.408+0000 7faa22200640 1 -- 172.21.15.28:0/1727347066 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa1c0c5720 msgr2=0x7faa1c150df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:52.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.408+0000 7faa22200640 1 --2- 172.21.15.28:0/1727347066 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa1c0c5720 0x7faa1c150df0 secure :-1 s=READY pgs=2432 cs=0 l=1 rev1=1 crypto rx=0x7faa1400cc90 tx=0x7faa1400a430 comp rx=0 tx=0).stop 2024-02-21T23:13:52.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.408+0000 7faa22200640 1 -- 172.21.15.28:0/1727347066 shutdown_connections 2024-02-21T23:13:52.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.409+0000 7faa22200640 1 --2- 172.21.15.28:0/1727347066 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa9fc06b4d0 0x7fa9fc06d990 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:52.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.409+0000 7faa22200640 1 --2- 172.21.15.28:0/1727347066 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faa1c0aff90 0x7faa1c1508b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:52.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.409+0000 7faa22200640 1 --2- 172.21.15.28:0/1727347066 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa1c0c5720 0x7faa1c150df0 unknown :-1 s=CLOSED pgs=2432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:52.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.409+0000 7faa22200640 1 --2- 172.21.15.28:0/1727347066 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faa1c06dbf0 0x7faa1c0c42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:52.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.409+0000 7faa22200640 1 -- 172.21.15.28:0/1727347066 >> 172.21.15.28:0/1727347066 conn(0x7faa1c0ac370 msgr2=0x7faa1c156f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:52.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.409+0000 7faa22200640 1 -- 172.21.15.28:0/1727347066 shutdown_connections 2024-02-21T23:13:52.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:52.409+0000 7faa22200640 1 -- 172.21.15.28:0/1727347066 wait complete. 2024-02-21T23:13:52.823 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 234, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:13:42.353072+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 8290}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8290': {'gid': 8290, 'name': 'a', 'rank': 1, 'incarnation': 231, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.28:6833/1864206570', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 1864206570}, {'type': 'v1', 'addr': '172.21.15.28:6833', 'nonce': 1864206570}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:13:52.823 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8290:up:stopping 2024-02-21T23:13:53.824 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:13:54.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.064+0000 7ff261800640 1 Processor -- start 2024-02-21T23:13:54.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.064+0000 7ff261800640 1 -- start start 2024-02-21T23:13:54.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.065+0000 7ff261800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff25c06dbf0 0x7ff25c158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:54.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.065+0000 7ff261800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff25c0c74c0 con 0x7ff25c155960 2024-02-21T23:13:54.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.065+0000 7ff261800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff25c0c77a0 con 0x7ff25c0aff90 2024-02-21T23:13:54.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.065+0000 7ff261800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff25c0c7270 con 0x7ff25c06dbf0 2024-02-21T23:13:54.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.066+0000 7ff253e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff25c06dbf0 0x7ff25c158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:54.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.066+0000 7ff260e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff25c155960 0x7ff25c15f890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39018/0 (socket says 172.21.15.28:39018) 2024-02-21T23:13:54.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.066+0000 7ff253e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff25c06dbf0 0x7ff25c158ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54458/0 (socket says 172.21.15.28:54458) 2024-02-21T23:13:54.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.066+0000 7ff260e00640 1 -- 172.21.15.28:0/3574608591 learned_addr learned my addr 172.21.15.28:0/3574608591 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:13:54.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.066+0000 7ff253e00640 1 -- 172.21.15.28:0/3574608591 >> v1:172.21.15.81:6789/0 conn(0x7ff25c0aff90 legacy=0x7ff25c0c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:54.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.066+0000 7ff253e00640 1 -- 172.21.15.28:0/3574608591 >> v1:172.21.15.28:6789/0 conn(0x7ff25c155960 legacy=0x7ff25c15f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:54.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.066+0000 7ff25b400640 1 -- 172.21.15.28:0/3574608591 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1172842445 0 0) 0x7ff25c0c74c0 con 0x7ff25c155960 2024-02-21T23:13:54.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.066+0000 7ff253e00640 1 -- 172.21.15.28:0/3574608591 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff25c0c7a80 con 0x7ff25c06dbf0 2024-02-21T23:13:54.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.067+0000 7ff253e00640 1 --2- 172.21.15.28:0/3574608591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff25c06dbf0 0x7ff25c158ac0 secure :-1 s=READY pgs=2434 cs=0 l=1 rev1=1 crypto rx=0x7ff2440097a0 tx=0x7ff24402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a5824c1384a011c server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:54.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.068+0000 7ff25b400640 1 -- 172.21.15.28:0/3574608591 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff244035020 con 0x7ff25c06dbf0 2024-02-21T23:13:54.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.068+0000 7ff25b400640 1 -- 172.21.15.28:0/3574608591 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff244031830 con 0x7ff25c06dbf0 2024-02-21T23:13:54.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.068+0000 7ff25b400640 1 -- 172.21.15.28:0/3574608591 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff244031c00 con 0x7ff25c06dbf0 2024-02-21T23:13:54.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.068+0000 7ff261800640 1 -- 172.21.15.28:0/3574608591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff25c06dbf0 msgr2=0x7ff25c158ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:54.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.068+0000 7ff261800640 1 --2- 172.21.15.28:0/3574608591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff25c06dbf0 0x7ff25c158ac0 secure :-1 s=READY pgs=2434 cs=0 l=1 rev1=1 crypto rx=0x7ff2440097a0 tx=0x7ff24402b8d0 comp rx=0 tx=0).stop 2024-02-21T23:13:54.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.068+0000 7ff261800640 1 -- 172.21.15.28:0/3574608591 shutdown_connections 2024-02-21T23:13:54.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.068+0000 7ff261800640 1 --2- 172.21.15.28:0/3574608591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff25c06dbf0 0x7ff25c158ac0 unknown :-1 s=CLOSED pgs=2434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:54.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.068+0000 7ff261800640 1 -- 172.21.15.28:0/3574608591 >> 172.21.15.28:0/3574608591 conn(0x7ff25c0ac370 msgr2=0x7ff25c0ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:54.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.069+0000 7ff261800640 1 -- 172.21.15.28:0/3574608591 shutdown_connections 2024-02-21T23:13:54.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.069+0000 7ff261800640 1 -- 172.21.15.28:0/3574608591 wait complete. 2024-02-21T23:13:54.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.070+0000 7ff261800640 1 Processor -- start 2024-02-21T23:13:54.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.070+0000 7ff261800640 1 -- start start 2024-02-21T23:13:54.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.071+0000 7ff261800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff25c06dbf0 0x7ff25c0aeca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:54.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.071+0000 7ff261800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff25c0aff90 0x7ff25c1ed550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:54.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.071+0000 7ff261800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff25c155960 0x7ff25c1eda90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:54.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.071+0000 7ff261800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff25c0c58f0 con 0x7ff25c155960 2024-02-21T23:13:54.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.071+0000 7ff261800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff25c158630 con 0x7ff25c0aff90 2024-02-21T23:13:54.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.071+0000 7ff261800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff25c0c5a70 con 0x7ff25c06dbf0 2024-02-21T23:13:54.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.072+0000 7ff260e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff25c155960 0x7ff25c1eda90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:54.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.072+0000 7ff260e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff25c155960 0x7ff25c1eda90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54474/0 (socket says 172.21.15.28:54474) 2024-02-21T23:13:54.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.072+0000 7ff260e00640 1 -- 172.21.15.28:0/3891305250 learned_addr learned my addr 172.21.15.28:0/3891305250 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:54.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.072+0000 7ff25be00640 1 --2- 172.21.15.28:0/3891305250 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff25c06dbf0 0x7ff25c0aeca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:54.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.072+0000 7ff253e00640 1 --2- 172.21.15.28:0/3891305250 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff25c0aff90 0x7ff25c1ed550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:54.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.072+0000 7ff260e00640 1 -- 172.21.15.28:0/3891305250 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff25c06dbf0 msgr2=0x7ff25c0aeca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:54.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.072+0000 7ff260e00640 1 --2- 172.21.15.28:0/3891305250 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff25c06dbf0 0x7ff25c0aeca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:54.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.072+0000 7ff260e00640 1 -- 172.21.15.28:0/3891305250 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff25c0aff90 msgr2=0x7ff25c1ed550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:54.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.072+0000 7ff260e00640 1 --2- 172.21.15.28:0/3891305250 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff25c0aff90 0x7ff25c1ed550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:54.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.072+0000 7ff260e00640 1 -- 172.21.15.28:0/3891305250 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff25c1eecd0 con 0x7ff25c155960 2024-02-21T23:13:54.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.072+0000 7ff253e00640 1 --2- 172.21.15.28:0/3891305250 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff25c0aff90 0x7ff25c1ed550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:13:54.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.073+0000 7ff260e00640 1 --2- 172.21.15.28:0/3891305250 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff25c155960 0x7ff25c1eda90 secure :-1 s=READY pgs=2435 cs=0 l=1 rev1=1 crypto rx=0x7ff25400c730 tx=0x7ff25400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:54.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.073+0000 7ff258c00640 1 -- 172.21.15.28:0/3891305250 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff254018070 con 0x7ff25c155960 2024-02-21T23:13:54.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.073+0000 7ff258c00640 1 -- 172.21.15.28:0/3891305250 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff25400ce40 con 0x7ff25c155960 2024-02-21T23:13:54.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.073+0000 7ff258c00640 1 -- 172.21.15.28:0/3891305250 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff25400a880 con 0x7ff25c155960 2024-02-21T23:13:54.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.073+0000 7ff261800640 1 -- 172.21.15.28:0/3891305250 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff25c154f50 con 0x7ff25c155960 2024-02-21T23:13:54.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.073+0000 7ff261800640 1 -- 172.21.15.28:0/3891305250 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff25c0af1e0 con 0x7ff25c155960 2024-02-21T23:13:54.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.074+0000 7ff261800640 1 -- 172.21.15.28:0/3891305250 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff25c04f290 con 0x7ff25c155960 2024-02-21T23:13:54.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.075+0000 7ff258c00640 1 -- 172.21.15.28:0/3891305250 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff25401b070 con 0x7ff25c155960 2024-02-21T23:13:54.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.076+0000 7ff258c00640 1 --2- 172.21.15.28:0/3891305250 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff23806b4d0 0x7ff23806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:54.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.077+0000 7ff258c00640 1 -- 172.21.15.28:0/3891305250 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff254089930 con 0x7ff25c155960 2024-02-21T23:13:54.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.077+0000 7ff25be00640 1 --2- 172.21.15.28:0/3891305250 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff23806b4d0 0x7ff23806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:54.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.078+0000 7ff25be00640 1 --2- 172.21.15.28:0/3891305250 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff23806b4d0 0x7ff23806d990 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7ff24c004ea0 tx=0x7ff24c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:54.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.081+0000 7ff258c00640 1 -- 172.21.15.28:0/3891305250 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ff254058610 con 0x7ff25c155960 2024-02-21T23:13:54.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.392+0000 7ff261800640 1 -- 172.21.15.28:0/3891305250 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff25c0b05d0 con 0x7ff25c155960 2024-02-21T23:13:54.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.393+0000 7ff258c00640 1 -- 172.21.15.28:0/3891305250 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 234 v234) v1 ==== 95+0+4964 (secure 0 0 0) 0x7ff254057f10 con 0x7ff25c155960 2024-02-21T23:13:54.381 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:13:54.381 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":234,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7265,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":224},{"gid":8287,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1607587459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1607587459},{"type":"v1","addr":"172.21.15.81:6835","nonce":1607587459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":224}],"filesystems":[{"mdsmap":{"epoch":234,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:13:42.353072+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":8290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8290":{"gid":8290,"name":"a","rank":1,"incarnation":231,"state":"up:stopping","state_seq":5,"addr":"172.21.15.28:6833/1864206570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1864206570},{"type":"v1","addr":"172.21.15.28:6833","nonce":1864206570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:13:54.381 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 234 2024-02-21T23:13:54.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.396+0000 7ff261800640 1 -- 172.21.15.28:0/3891305250 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff23806b4d0 msgr2=0x7ff23806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:54.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.396+0000 7ff261800640 1 --2- 172.21.15.28:0/3891305250 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff23806b4d0 0x7ff23806d990 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7ff24c004ea0 tx=0x7ff24c009040 comp rx=0 tx=0).stop 2024-02-21T23:13:54.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.396+0000 7ff261800640 1 -- 172.21.15.28:0/3891305250 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff25c155960 msgr2=0x7ff25c1eda90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:54.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.396+0000 7ff261800640 1 --2- 172.21.15.28:0/3891305250 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff25c155960 0x7ff25c1eda90 secure :-1 s=READY pgs=2435 cs=0 l=1 rev1=1 crypto rx=0x7ff25400c730 tx=0x7ff25400cbf0 comp rx=0 tx=0).stop 2024-02-21T23:13:54.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.397+0000 7ff261800640 1 -- 172.21.15.28:0/3891305250 shutdown_connections 2024-02-21T23:13:54.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.397+0000 7ff261800640 1 --2- 172.21.15.28:0/3891305250 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff23806b4d0 0x7ff23806d990 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:54.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.397+0000 7ff261800640 1 --2- 172.21.15.28:0/3891305250 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff25c155960 0x7ff25c1eda90 unknown :-1 s=CLOSED pgs=2435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:54.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.397+0000 7ff261800640 1 --2- 172.21.15.28:0/3891305250 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff25c0aff90 0x7ff25c1ed550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:54.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.397+0000 7ff261800640 1 --2- 172.21.15.28:0/3891305250 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff25c06dbf0 0x7ff25c0aeca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:54.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.397+0000 7ff261800640 1 -- 172.21.15.28:0/3891305250 >> 172.21.15.28:0/3891305250 conn(0x7ff25c0ac370 msgr2=0x7ff25c0c50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:54.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.397+0000 7ff261800640 1 -- 172.21.15.28:0/3891305250 shutdown_connections 2024-02-21T23:13:54.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:54.397+0000 7ff261800640 1 -- 172.21.15.28:0/3891305250 wait complete. 2024-02-21T23:13:54.809 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 234, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:13:42.353072+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 8290}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8290': {'gid': 8290, 'name': 'a', 'rank': 1, 'incarnation': 231, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.28:6833/1864206570', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 1864206570}, {'type': 'v1', 'addr': '172.21.15.28:6833', 'nonce': 1864206570}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:13:54.809 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8290:up:stopping 2024-02-21T23:13:55.810 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:13:56.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.046+0000 7f75fca00640 1 Processor -- start 2024-02-21T23:13:56.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.046+0000 7f75fca00640 1 -- start start 2024-02-21T23:13:56.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.047+0000 7f75fca00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f75f80aff90 0x7f75f80c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:56.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.047+0000 7f75fca00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f75f80c5720 0x7f75f815f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:56.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.047+0000 7f75fca00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75f80c74c0 con 0x7f75f806dbf0 2024-02-21T23:13:56.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.047+0000 7f75fca00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75f80c7270 con 0x7f75f80aff90 2024-02-21T23:13:56.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.047+0000 7f75fca00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75f8158630 con 0x7f75f80c5720 2024-02-21T23:13:56.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.048+0000 7f75f7400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f75f80aff90 0x7f75f80c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:56.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.048+0000 7f75f7e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f75f80c5720 0x7f75f815f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:56.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.048+0000 7f75f7400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f75f80aff90 0x7f75f80c51e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.28:56192/0 (socket says 172.21.15.28:56192) 2024-02-21T23:13:56.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.048+0000 7f75f7400640 1 -- 172.21.15.28:0/3837748348 learned_addr learned my addr 172.21.15.28:0/3837748348 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:56.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.049+0000 7f75f7e00640 1 -- 172.21.15.28:0/3837748348 >> v2:172.21.15.81:3300/0 conn(0x7f75f80aff90 msgr2=0x7f75f80c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:56.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.049+0000 7f75f7e00640 1 --2- 172.21.15.28:0/3837748348 >> v2:172.21.15.81:3300/0 conn(0x7f75f80aff90 0x7f75f80c51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:56.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.049+0000 7f75f7e00640 1 -- 172.21.15.28:0/3837748348 >> v1:172.21.15.81:6789/0 conn(0x7f75f806dbf0 legacy=0x7f75f8158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:56.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.049+0000 7f75f7e00640 1 -- 172.21.15.28:0/3837748348 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75f80c77a0 con 0x7f75f80c5720 2024-02-21T23:13:56.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.050+0000 7f75f7e00640 1 --2- 172.21.15.28:0/3837748348 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f75f80c5720 0x7f75f815f890 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f75f0009d90 tx=0x7f75f002bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=935a7d89666808f0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:56.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.051+0000 7f75f6a00640 1 -- 172.21.15.28:0/3837748348 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f0039070 con 0x7f75f80c5720 2024-02-21T23:13:56.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.051+0000 7f75f6a00640 1 -- 172.21.15.28:0/3837748348 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75f002cc50 con 0x7f75f80c5720 2024-02-21T23:13:56.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.051+0000 7f75f6a00640 1 -- 172.21.15.28:0/3837748348 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f0034440 con 0x7f75f80c5720 2024-02-21T23:13:56.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.052+0000 7f75fca00640 1 -- 172.21.15.28:0/3837748348 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f75f80c5720 msgr2=0x7f75f815f890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:56.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.052+0000 7f75fca00640 1 --2- 172.21.15.28:0/3837748348 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f75f80c5720 0x7f75f815f890 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f75f0009d90 tx=0x7f75f002bea0 comp rx=0 tx=0).stop 2024-02-21T23:13:56.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.052+0000 7f75fca00640 1 -- 172.21.15.28:0/3837748348 shutdown_connections 2024-02-21T23:13:56.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.052+0000 7f75fca00640 1 --2- 172.21.15.28:0/3837748348 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f75f80c5720 0x7f75f815f890 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:56.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.052+0000 7f75fca00640 1 --2- 172.21.15.28:0/3837748348 >> v2:172.21.15.81:3300/0 conn(0x7f75f80aff90 0x7f75f80c51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:56.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.052+0000 7f75fca00640 1 -- 172.21.15.28:0/3837748348 >> 172.21.15.28:0/3837748348 conn(0x7f75f80ac370 msgr2=0x7f75f80ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:56.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.052+0000 7f75fca00640 1 -- 172.21.15.28:0/3837748348 shutdown_connections 2024-02-21T23:13:56.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.052+0000 7f75fca00640 1 -- 172.21.15.28:0/3837748348 wait complete. 2024-02-21T23:13:56.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.054+0000 7f75fca00640 1 Processor -- start 2024-02-21T23:13:56.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.054+0000 7f75fca00640 1 -- start start 2024-02-21T23:13:56.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.055+0000 7f75fca00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f75f806dbf0 0x7f75f81ebbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:56.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.055+0000 7f75fca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75f80aff90 0x7f75f80c05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:56.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.055+0000 7f75fca00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f75f80c5720 0x7f75f80c42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:56.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.055+0000 7f75fca00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f80c8760 con 0x7f75f80aff90 2024-02-21T23:13:56.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.055+0000 7f75fca00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f80be0c0 con 0x7f75f806dbf0 2024-02-21T23:13:56.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.055+0000 7f75fca00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75f80c88e0 con 0x7f75f80c5720 2024-02-21T23:13:56.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.055+0000 7f75efe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75f80aff90 0x7f75f80c05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:56.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.055+0000 7f75efe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75f80aff90 0x7f75f80c05b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59176/0 (socket says 172.21.15.28:59176) 2024-02-21T23:13:56.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.055+0000 7f75efe00640 1 -- 172.21.15.28:0/1990143182 learned_addr learned my addr 172.21.15.28:0/1990143182 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:56.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.055+0000 7f75f7400640 1 --2- 172.21.15.28:0/1990143182 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f75f806dbf0 0x7f75f81ebbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:56.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.055+0000 7f75f7e00640 1 --2- 172.21.15.28:0/1990143182 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f75f80c5720 0x7f75f80c42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:56.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.056+0000 7f75efe00640 1 -- 172.21.15.28:0/1990143182 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f75f80c5720 msgr2=0x7f75f80c42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:56.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.056+0000 7f75efe00640 1 --2- 172.21.15.28:0/1990143182 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f75f80c5720 0x7f75f80c42c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:56.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.056+0000 7f75efe00640 1 -- 172.21.15.28:0/1990143182 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f75f806dbf0 msgr2=0x7f75f81ebbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:56.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.056+0000 7f75efe00640 1 --2- 172.21.15.28:0/1990143182 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f75f806dbf0 0x7f75f81ebbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:56.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.056+0000 7f75efe00640 1 -- 172.21.15.28:0/1990143182 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75f80c12f0 con 0x7f75f80aff90 2024-02-21T23:13:56.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.056+0000 7f75f7e00640 1 --2- 172.21.15.28:0/1990143182 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f75f80c5720 0x7f75f80c42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:13:56.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.056+0000 7f75efe00640 1 --2- 172.21.15.28:0/1990143182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75f80aff90 0x7f75f80c05b0 secure :-1 s=READY pgs=2436 cs=0 l=1 rev1=1 crypto rx=0x7f75e000c470 tx=0x7f75e0001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:56.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.056+0000 7f75ef400640 1 -- 172.21.15.28:0/1990143182 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75e00052d0 con 0x7f75f80aff90 2024-02-21T23:13:56.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.056+0000 7f75ef400640 1 -- 172.21.15.28:0/1990143182 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75e0005470 con 0x7f75f80aff90 2024-02-21T23:13:56.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.056+0000 7f75ef400640 1 -- 172.21.15.28:0/1990143182 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75e0007260 con 0x7f75f80aff90 2024-02-21T23:13:56.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.056+0000 7f75fca00640 1 -- 172.21.15.28:0/1990143182 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75f80c15d0 con 0x7f75f80aff90 2024-02-21T23:13:56.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.057+0000 7f75fca00640 1 -- 172.21.15.28:0/1990143182 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75f80c4800 con 0x7f75f80aff90 2024-02-21T23:13:56.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.058+0000 7f75fca00640 1 -- 172.21.15.28:0/1990143182 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75f804f290 con 0x7f75f80aff90 2024-02-21T23:13:56.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.059+0000 7f75ef400640 1 -- 172.21.15.28:0/1990143182 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f75e0007400 con 0x7f75f80aff90 2024-02-21T23:13:56.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.060+0000 7f75ef400640 1 --2- 172.21.15.28:0/1990143182 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f75d406b500 0x7f75d406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:56.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.060+0000 7f75ef400640 1 -- 172.21.15.28:0/1990143182 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f75e0089800 con 0x7f75f80aff90 2024-02-21T23:13:56.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.061+0000 7f75f7400640 1 --2- 172.21.15.28:0/1990143182 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f75d406b500 0x7f75d406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:56.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.062+0000 7f75f7400640 1 --2- 172.21.15.28:0/1990143182 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f75d406b500 0x7f75d406d9c0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f75e8009bd0 tx=0x7f75e8009240 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:56.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.065+0000 7f75ef400640 1 -- 172.21.15.28:0/1990143182 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f75e0057e90 con 0x7f75f80aff90 2024-02-21T23:13:56.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.429+0000 7f75fca00640 1 -- 172.21.15.28:0/1990143182 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f75f806d9d0 con 0x7f75f80aff90 2024-02-21T23:13:56.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.430+0000 7f75ef400640 1 -- 172.21.15.28:0/1990143182 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 234 v234) v1 ==== 95+0+4964 (secure 0 0 0) 0x7f75e005bf30 con 0x7f75f80aff90 2024-02-21T23:13:56.418 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:13:56.418 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":234,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7265,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":224},{"gid":8287,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1607587459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1607587459},{"type":"v1","addr":"172.21.15.81:6835","nonce":1607587459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":224}],"filesystems":[{"mdsmap":{"epoch":234,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:13:42.353072+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":8290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8290":{"gid":8290,"name":"a","rank":1,"incarnation":231,"state":"up:stopping","state_seq":5,"addr":"172.21.15.28:6833/1864206570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1864206570},{"type":"v1","addr":"172.21.15.28:6833","nonce":1864206570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:13:56.418 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 234 2024-02-21T23:13:56.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.433+0000 7f75fca00640 1 -- 172.21.15.28:0/1990143182 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f75d406b500 msgr2=0x7f75d406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:56.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.433+0000 7f75fca00640 1 --2- 172.21.15.28:0/1990143182 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f75d406b500 0x7f75d406d9c0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f75e8009bd0 tx=0x7f75e8009240 comp rx=0 tx=0).stop 2024-02-21T23:13:56.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.433+0000 7f75fca00640 1 -- 172.21.15.28:0/1990143182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75f80aff90 msgr2=0x7f75f80c05b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:56.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.433+0000 7f75fca00640 1 --2- 172.21.15.28:0/1990143182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75f80aff90 0x7f75f80c05b0 secure :-1 s=READY pgs=2436 cs=0 l=1 rev1=1 crypto rx=0x7f75e000c470 tx=0x7f75e0001e70 comp rx=0 tx=0).stop 2024-02-21T23:13:56.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.433+0000 7f75fca00640 1 -- 172.21.15.28:0/1990143182 shutdown_connections 2024-02-21T23:13:56.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.433+0000 7f75fca00640 1 --2- 172.21.15.28:0/1990143182 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f75d406b500 0x7f75d406d9c0 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:56.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.433+0000 7f75fca00640 1 --2- 172.21.15.28:0/1990143182 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f75f80c5720 0x7f75f80c42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:56.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.433+0000 7f75fca00640 1 --2- 172.21.15.28:0/1990143182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75f80aff90 0x7f75f80c05b0 unknown :-1 s=CLOSED pgs=2436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:56.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.433+0000 7f75fca00640 1 --2- 172.21.15.28:0/1990143182 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f75f806dbf0 0x7f75f81ebbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:56.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.433+0000 7f75fca00640 1 -- 172.21.15.28:0/1990143182 >> 172.21.15.28:0/1990143182 conn(0x7f75f80ac370 msgr2=0x7f75f80c8ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:56.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.433+0000 7f75fca00640 1 -- 172.21.15.28:0/1990143182 shutdown_connections 2024-02-21T23:13:56.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:56.433+0000 7f75fca00640 1 -- 172.21.15.28:0/1990143182 wait complete. 2024-02-21T23:13:56.845 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 234, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:13:42.353072+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 8290}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8290': {'gid': 8290, 'name': 'a', 'rank': 1, 'incarnation': 231, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.28:6833/1864206570', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 1864206570}, {'type': 'v1', 'addr': '172.21.15.28:6833', 'nonce': 1864206570}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:13:56.845 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8290:up:stopping 2024-02-21T23:13:57.846 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:13:58.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.086+0000 7f2cf0a00640 1 Processor -- start 2024-02-21T23:13:58.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.087+0000 7f2cf0a00640 1 -- start start 2024-02-21T23:13:58.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.089+0000 7f2cf0a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f2cec152ab0 0x7f2cec0c7fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:58.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.089+0000 7f2cf0a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2cec158850 0x7f2cec158cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:58.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.089+0000 7f2cf0a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cec0a8180 con 0x7f2cec0c6710 2024-02-21T23:13:58.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.089+0000 7f2cf0a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cec0c3ce0 con 0x7f2cec158850 2024-02-21T23:13:58.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.089+0000 7f2cf0a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cec0c33b0 con 0x7f2cec152ab0 2024-02-21T23:13:58.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.090+0000 7f2cea800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2cec158850 0x7f2cec158cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:58.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.090+0000 7f2cea800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2cec158850 0x7f2cec158cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59188/0 (socket says 172.21.15.28:59188) 2024-02-21T23:13:58.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.090+0000 7f2cea800640 1 -- 172.21.15.28:0/3070872547 learned_addr learned my addr 172.21.15.28:0/3070872547 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:58.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.090+0000 7f2ce9e00640 1 --2- 172.21.15.28:0/3070872547 >> v2:172.21.15.81:3300/0 conn(0x7f2cec152ab0 0x7f2cec0c7fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:58.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.090+0000 7f2cea800640 1 -- 172.21.15.28:0/3070872547 >> v2:172.21.15.81:3300/0 conn(0x7f2cec152ab0 msgr2=0x7f2cec0c7fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:58.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.090+0000 7f2cea800640 1 --2- 172.21.15.28:0/3070872547 >> v2:172.21.15.81:3300/0 conn(0x7f2cec152ab0 0x7f2cec0c7fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:58.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.090+0000 7f2cea800640 1 -- 172.21.15.28:0/3070872547 >> v1:172.21.15.81:6789/0 conn(0x7f2cec0c6710 legacy=0x7f2cec0c5210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:13:58.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.090+0000 7f2cea800640 1 -- 172.21.15.28:0/3070872547 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cec0c3690 con 0x7f2cec158850 2024-02-21T23:13:58.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.091+0000 7f2cea800640 1 --2- 172.21.15.28:0/3070872547 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cec158850 0x7f2cec158cc0 secure :-1 s=READY pgs=2437 cs=0 l=1 rev1=1 crypto rx=0x7f2cec0c9850 tx=0x7f2ce402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd725814e623984c server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:58.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.092+0000 7f2ce8a00640 1 -- 172.21.15.28:0/3070872547 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ce402ca80 con 0x7f2cec158850 2024-02-21T23:13:58.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.092+0000 7f2ce8a00640 1 -- 172.21.15.28:0/3070872547 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ce402cc20 con 0x7f2cec158850 2024-02-21T23:13:58.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.092+0000 7f2ce8a00640 1 -- 172.21.15.28:0/3070872547 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ce40343f0 con 0x7f2cec158850 2024-02-21T23:13:58.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.092+0000 7f2cf0a00640 1 -- 172.21.15.28:0/3070872547 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cec158850 msgr2=0x7f2cec158cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:58.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.092+0000 7f2cf0a00640 1 --2- 172.21.15.28:0/3070872547 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cec158850 0x7f2cec158cc0 secure :-1 s=READY pgs=2437 cs=0 l=1 rev1=1 crypto rx=0x7f2cec0c9850 tx=0x7f2ce402bee0 comp rx=0 tx=0).stop 2024-02-21T23:13:58.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.093+0000 7f2cf0a00640 1 -- 172.21.15.28:0/3070872547 shutdown_connections 2024-02-21T23:13:58.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.093+0000 7f2cf0a00640 1 --2- 172.21.15.28:0/3070872547 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cec158850 0x7f2cec158cc0 unknown :-1 s=CLOSED pgs=2437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:58.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.093+0000 7f2cf0a00640 1 --2- 172.21.15.28:0/3070872547 >> v2:172.21.15.81:3300/0 conn(0x7f2cec152ab0 0x7f2cec0c7fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:58.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.093+0000 7f2cf0a00640 1 -- 172.21.15.28:0/3070872547 >> 172.21.15.28:0/3070872547 conn(0x7f2cec0c8810 msgr2=0x7f2cec0c8bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:58.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.093+0000 7f2cf0a00640 1 -- 172.21.15.28:0/3070872547 shutdown_connections 2024-02-21T23:13:58.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.093+0000 7f2cf0a00640 1 -- 172.21.15.28:0/3070872547 wait complete. 2024-02-21T23:13:58.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.095+0000 7f2cf0a00640 1 Processor -- start 2024-02-21T23:13:58.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.095+0000 7f2cf0a00640 1 -- start start 2024-02-21T23:13:58.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.096+0000 7f2cf0a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cec0c6710 0x7f2cec0a9070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:58.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.096+0000 7f2cf0a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2cec152ab0 0x7f2cec155b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:58.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.096+0000 7f2cf0a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2cec158850 0x7f2cec1edff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:58.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.096+0000 7f2cf0a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cec0c5aa0 con 0x7f2cec0c6710 2024-02-21T23:13:58.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.096+0000 7f2cf0a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cec0c5920 con 0x7f2cec158850 2024-02-21T23:13:58.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.096+0000 7f2cf0a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cec159200 con 0x7f2cec152ab0 2024-02-21T23:13:58.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.096+0000 7f2ce9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cec0c6710 0x7f2cec0a9070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:58.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.096+0000 7f2ce9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cec0c6710 0x7f2cec0a9070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59194/0 (socket says 172.21.15.28:59194) 2024-02-21T23:13:58.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.096+0000 7f2ce9e00640 1 -- 172.21.15.28:0/4260948521 learned_addr learned my addr 172.21.15.28:0/4260948521 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:13:58.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.097+0000 7f2ce9400640 1 --2- 172.21.15.28:0/4260948521 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2cec152ab0 0x7f2cec155b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:58.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.097+0000 7f2cea800640 1 --2- 172.21.15.28:0/4260948521 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2cec158850 0x7f2cec1edff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:58.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.097+0000 7f2ce9e00640 1 -- 172.21.15.28:0/4260948521 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2cec152ab0 msgr2=0x7f2cec155b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:58.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.097+0000 7f2ce9e00640 1 --2- 172.21.15.28:0/4260948521 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2cec152ab0 0x7f2cec155b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:58.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.097+0000 7f2ce9e00640 1 -- 172.21.15.28:0/4260948521 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2cec158850 msgr2=0x7f2cec1edff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:58.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.097+0000 7f2ce9e00640 1 --2- 172.21.15.28:0/4260948521 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2cec158850 0x7f2cec1edff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:58.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.097+0000 7f2ce9e00640 1 -- 172.21.15.28:0/4260948521 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cec14da80 con 0x7f2cec0c6710 2024-02-21T23:13:58.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.097+0000 7f2ce9e00640 1 --2- 172.21.15.28:0/4260948521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cec0c6710 0x7f2cec0a9070 secure :-1 s=READY pgs=2438 cs=0 l=1 rev1=1 crypto rx=0x7f2cdc00d800 tx=0x7f2cdc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:58.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.097+0000 7f2ce2000640 1 -- 172.21.15.28:0/4260948521 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cdc00b6c0 con 0x7f2cec0c6710 2024-02-21T23:13:58.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.097+0000 7f2ce2000640 1 -- 172.21.15.28:0/4260948521 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cdc019070 con 0x7f2cec0c6710 2024-02-21T23:13:58.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.098+0000 7f2ce2000640 1 -- 172.21.15.28:0/4260948521 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cdc00ba80 con 0x7f2cec0c6710 2024-02-21T23:13:58.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.098+0000 7f2cf0a00640 1 -- 172.21.15.28:0/4260948521 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cec14dd40 con 0x7f2cec0c6710 2024-02-21T23:13:58.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.098+0000 7f2cf0a00640 1 -- 172.21.15.28:0/4260948521 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cec0708f0 con 0x7f2cec0c6710 2024-02-21T23:13:58.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.099+0000 7f2cf0a00640 1 -- 172.21.15.28:0/4260948521 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cec070d30 con 0x7f2cec0c6710 2024-02-21T23:13:58.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.100+0000 7f2ce2000640 1 -- 172.21.15.28:0/4260948521 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2cdc00bc20 con 0x7f2cec0c6710 2024-02-21T23:13:58.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.101+0000 7f2ce2000640 1 --2- 172.21.15.28:0/4260948521 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2ccc06b4d0 0x7f2ccc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:13:58.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.102+0000 7f2ce2000640 1 -- 172.21.15.28:0/4260948521 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f2cdc08a880 con 0x7f2cec0c6710 2024-02-21T23:13:58.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.102+0000 7f2ce9400640 1 --2- 172.21.15.28:0/4260948521 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2ccc06b4d0 0x7f2ccc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:13:58.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.103+0000 7f2ce9400640 1 --2- 172.21.15.28:0/4260948521 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2ccc06b4d0 0x7f2ccc06d990 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f2cd8007fd0 tx=0x7f2cd8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:13:58.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.106+0000 7f2ce2000640 1 -- 172.21.15.28:0/4260948521 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f2cdc059560 con 0x7f2cec0c6710 2024-02-21T23:13:58.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.354+0000 7f2cf0a00640 1 -- 172.21.15.28:0/4260948521 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2cec067f30 con 0x7f2cec0c6710 2024-02-21T23:13:58.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.355+0000 7f2ce2000640 1 -- 172.21.15.28:0/4260948521 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 234 v234) v1 ==== 95+0+4964 (secure 0 0 0) 0x7f2cdc058e60 con 0x7f2cec0c6710 2024-02-21T23:13:58.342 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:13:58.342 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":234,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7265,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":224},{"gid":8287,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1607587459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1607587459},{"type":"v1","addr":"172.21.15.81:6835","nonce":1607587459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":224}],"filesystems":[{"mdsmap":{"epoch":234,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:13:42.353072+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":8290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8290":{"gid":8290,"name":"a","rank":1,"incarnation":231,"state":"up:stopping","state_seq":5,"addr":"172.21.15.28:6833/1864206570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1864206570},{"type":"v1","addr":"172.21.15.28:6833","nonce":1864206570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:13:58.343 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 234 2024-02-21T23:13:58.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.358+0000 7f2cf0a00640 1 -- 172.21.15.28:0/4260948521 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2ccc06b4d0 msgr2=0x7f2ccc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:58.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.358+0000 7f2cf0a00640 1 --2- 172.21.15.28:0/4260948521 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2ccc06b4d0 0x7f2ccc06d990 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f2cd8007fd0 tx=0x7f2cd8009040 comp rx=0 tx=0).stop 2024-02-21T23:13:58.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.358+0000 7f2cf0a00640 1 -- 172.21.15.28:0/4260948521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cec0c6710 msgr2=0x7f2cec0a9070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:13:58.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.358+0000 7f2cf0a00640 1 --2- 172.21.15.28:0/4260948521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cec0c6710 0x7f2cec0a9070 secure :-1 s=READY pgs=2438 cs=0 l=1 rev1=1 crypto rx=0x7f2cdc00d800 tx=0x7f2cdc00dcc0 comp rx=0 tx=0).stop 2024-02-21T23:13:58.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.358+0000 7f2cf0a00640 1 -- 172.21.15.28:0/4260948521 shutdown_connections 2024-02-21T23:13:58.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.358+0000 7f2cf0a00640 1 --2- 172.21.15.28:0/4260948521 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2ccc06b4d0 0x7f2ccc06d990 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:58.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.358+0000 7f2cf0a00640 1 --2- 172.21.15.28:0/4260948521 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2cec152ab0 0x7f2cec155b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:58.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.358+0000 7f2cf0a00640 1 --2- 172.21.15.28:0/4260948521 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2cec158850 0x7f2cec1edff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:58.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.358+0000 7f2cf0a00640 1 --2- 172.21.15.28:0/4260948521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cec0c6710 0x7f2cec0a9070 unknown :-1 s=CLOSED pgs=2438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:13:58.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.358+0000 7f2cf0a00640 1 -- 172.21.15.28:0/4260948521 >> 172.21.15.28:0/4260948521 conn(0x7f2cec0c8810 msgr2=0x7f2cec06cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:13:58.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.359+0000 7f2cf0a00640 1 -- 172.21.15.28:0/4260948521 shutdown_connections 2024-02-21T23:13:58.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:13:58.359+0000 7f2cf0a00640 1 -- 172.21.15.28:0/4260948521 wait complete. 2024-02-21T23:13:58.775 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 234, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:13:42.353072+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 8290}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8290': {'gid': 8290, 'name': 'a', 'rank': 1, 'incarnation': 231, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.28:6833/1864206570', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 1864206570}, {'type': 'v1', 'addr': '172.21.15.28:6833', 'nonce': 1864206570}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:13:58.775 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8290:up:stopping 2024-02-21T23:13:59.776 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:14:00.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.019+0000 7fba7f200640 1 Processor -- start 2024-02-21T23:14:00.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.019+0000 7fba7f200640 1 -- start start 2024-02-21T23:14:00.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.021+0000 7fba7f200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fba800aff90 0x7fba800c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:00.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.021+0000 7fba7f200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fba8006dbf0 0x7fba80158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:00.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.021+0000 7fba7f200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba800c74c0 con 0x7fba800c5720 2024-02-21T23:14:00.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.021+0000 7fba7f200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba800c7270 con 0x7fba800aff90 2024-02-21T23:14:00.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.021+0000 7fba7f200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba80158630 con 0x7fba8006dbf0 2024-02-21T23:14:00.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.021+0000 7fba7de00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fba800aff90 0x7fba800c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:00.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.021+0000 7fba7de00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fba800aff90 0x7fba800c51e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59204/0 (socket says 172.21.15.28:59204) 2024-02-21T23:14:00.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.021+0000 7fba7de00640 1 -- 172.21.15.28:0/1350271014 learned_addr learned my addr 172.21.15.28:0/1350271014 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:14:00.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.022+0000 7fba7d400640 1 --2- 172.21.15.28:0/1350271014 >> v2:172.21.15.81:3300/0 conn(0x7fba8006dbf0 0x7fba80158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:00.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.022+0000 7fba7de00640 1 -- 172.21.15.28:0/1350271014 >> v2:172.21.15.81:3300/0 conn(0x7fba8006dbf0 msgr2=0x7fba80158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:14:00.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.022+0000 7fba7de00640 1 --2- 172.21.15.28:0/1350271014 >> v2:172.21.15.81:3300/0 conn(0x7fba8006dbf0 0x7fba80158ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:00.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.022+0000 7fba7de00640 1 -- 172.21.15.28:0/1350271014 >> v1:172.21.15.81:6789/0 conn(0x7fba800c5720 legacy=0x7fba8006c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:14:00.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.022+0000 7fba7de00640 1 -- 172.21.15.28:0/1350271014 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba800c77a0 con 0x7fba800aff90 2024-02-21T23:14:00.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.023+0000 7fba7de00640 1 --2- 172.21.15.28:0/1350271014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba800aff90 0x7fba800c51e0 secure :-1 s=READY pgs=2439 cs=0 l=1 rev1=1 crypto rx=0x7fba800c73f0 tx=0x7fba7002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac263ccc542cf663 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:00.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.023+0000 7fba7ca00640 1 -- 172.21.15.28:0/1350271014 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba7002ca80 con 0x7fba800aff90 2024-02-21T23:14:00.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.023+0000 7fba7ca00640 1 -- 172.21.15.28:0/1350271014 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba7002cc20 con 0x7fba800aff90 2024-02-21T23:14:00.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.023+0000 7fba7ca00640 1 -- 172.21.15.28:0/1350271014 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba700343f0 con 0x7fba800aff90 2024-02-21T23:14:00.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.024+0000 7fba7f200640 1 -- 172.21.15.28:0/1350271014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba800aff90 msgr2=0x7fba800c51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:00.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.024+0000 7fba7f200640 1 --2- 172.21.15.28:0/1350271014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba800aff90 0x7fba800c51e0 secure :-1 s=READY pgs=2439 cs=0 l=1 rev1=1 crypto rx=0x7fba800c73f0 tx=0x7fba7002bee0 comp rx=0 tx=0).stop 2024-02-21T23:14:00.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.024+0000 7fba7f200640 1 -- 172.21.15.28:0/1350271014 shutdown_connections 2024-02-21T23:14:00.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.024+0000 7fba7f200640 1 --2- 172.21.15.28:0/1350271014 >> v2:172.21.15.81:3300/0 conn(0x7fba8006dbf0 0x7fba80158ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:00.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.024+0000 7fba7f200640 1 --2- 172.21.15.28:0/1350271014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba800aff90 0x7fba800c51e0 unknown :-1 s=CLOSED pgs=2439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:00.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.024+0000 7fba7f200640 1 -- 172.21.15.28:0/1350271014 >> 172.21.15.28:0/1350271014 conn(0x7fba800ac370 msgr2=0x7fba800ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:14:00.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.024+0000 7fba7f200640 1 -- 172.21.15.28:0/1350271014 shutdown_connections 2024-02-21T23:14:00.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.024+0000 7fba7f200640 1 -- 172.21.15.28:0/1350271014 wait complete. 2024-02-21T23:14:00.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.026+0000 7fba7f200640 1 Processor -- start 2024-02-21T23:14:00.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.026+0000 7fba7f200640 1 -- start start 2024-02-21T23:14:00.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.027+0000 7fba7f200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba8006dbf0 0x7fba801eb4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:00.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.027+0000 7fba7f200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fba800aff90 0x7fba801515e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:00.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.027+0000 7fba7f200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fba800c5720 0x7fba800c1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:00.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.027+0000 7fba7f200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba800c8760 con 0x7fba8006dbf0 2024-02-21T23:14:00.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.027+0000 7fba7f200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba800be0c0 con 0x7fba800aff90 2024-02-21T23:14:00.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.027+0000 7fba7f200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba800c88e0 con 0x7fba800c5720 2024-02-21T23:14:00.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.027+0000 7fba7de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba8006dbf0 0x7fba801eb4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:00.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.027+0000 7fba7de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba8006dbf0 0x7fba801eb4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59216/0 (socket says 172.21.15.28:59216) 2024-02-21T23:14:00.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.027+0000 7fba7de00640 1 -- 172.21.15.28:0/2828812043 learned_addr learned my addr 172.21.15.28:0/2828812043 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:14:00.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.028+0000 7fba7d400640 1 --2- 172.21.15.28:0/2828812043 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fba800aff90 0x7fba801515e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:00.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.028+0000 7fba7e800640 1 --2- 172.21.15.28:0/2828812043 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fba800c5720 0x7fba800c1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:00.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.028+0000 7fba7de00640 1 -- 172.21.15.28:0/2828812043 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fba800c5720 msgr2=0x7fba800c1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:00.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.028+0000 7fba7de00640 1 --2- 172.21.15.28:0/2828812043 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fba800c5720 0x7fba800c1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:00.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.028+0000 7fba7de00640 1 -- 172.21.15.28:0/2828812043 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fba800aff90 msgr2=0x7fba801515e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:00.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.028+0000 7fba7de00640 1 --2- 172.21.15.28:0/2828812043 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fba800aff90 0x7fba801515e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:00.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.028+0000 7fba7de00640 1 -- 172.21.15.28:0/2828812043 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba80154a00 con 0x7fba8006dbf0 2024-02-21T23:14:00.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.028+0000 7fba7e800640 1 --2- 172.21.15.28:0/2828812043 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fba800c5720 0x7fba800c1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:14:00.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.028+0000 7fba7de00640 1 --2- 172.21.15.28:0/2828812043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba8006dbf0 0x7fba801eb4d0 secure :-1 s=READY pgs=2440 cs=0 l=1 rev1=1 crypto rx=0x7fba70000f80 tx=0x7fba70004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:00.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.029+0000 7fba76000640 1 -- 172.21.15.28:0/2828812043 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba70042020 con 0x7fba8006dbf0 2024-02-21T23:14:00.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.029+0000 7fba76000640 1 -- 172.21.15.28:0/2828812043 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba7003e450 con 0x7fba8006dbf0 2024-02-21T23:14:00.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.029+0000 7fba7f200640 1 -- 172.21.15.28:0/2828812043 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba80152af0 con 0x7fba8006dbf0 2024-02-21T23:14:00.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.029+0000 7fba76000640 1 -- 172.21.15.28:0/2828812043 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba70009510 con 0x7fba8006dbf0 2024-02-21T23:14:00.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.029+0000 7fba7f200640 1 -- 172.21.15.28:0/2828812043 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba801530b0 con 0x7fba8006dbf0 2024-02-21T23:14:00.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.030+0000 7fba7f200640 1 -- 172.21.15.28:0/2828812043 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba8004f290 con 0x7fba8006dbf0 2024-02-21T23:14:00.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.031+0000 7fba76000640 1 -- 172.21.15.28:0/2828812043 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fba70045020 con 0x7fba8006dbf0 2024-02-21T23:14:00.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.032+0000 7fba76000640 1 --2- 172.21.15.28:0/2828812043 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fba5c06b500 0x7fba5c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:00.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.032+0000 7fba76000640 1 -- 172.21.15.28:0/2828812043 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fba700acbc0 con 0x7fba8006dbf0 2024-02-21T23:14:00.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.033+0000 7fba7d400640 1 --2- 172.21.15.28:0/2828812043 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fba5c06b500 0x7fba5c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:00.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.034+0000 7fba7d400640 1 --2- 172.21.15.28:0/2828812043 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fba5c06b500 0x7fba5c06d9c0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fba800c5150 tx=0x7fba68008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:00.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.040+0000 7fba76000640 1 -- 172.21.15.28:0/2828812043 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fba7004c030 con 0x7fba8006dbf0 2024-02-21T23:14:00.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.282+0000 7fba7f200640 1 -- 172.21.15.28:0/2828812043 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fba801572a0 con 0x7fba8006dbf0 2024-02-21T23:14:00.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.283+0000 7fba76000640 1 -- 172.21.15.28:0/2828812043 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 234 v234) v1 ==== 95+0+4964 (secure 0 0 0) 0x7fba7007b820 con 0x7fba8006dbf0 2024-02-21T23:14:00.271 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:14:00.271 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":234,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7265,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":224},{"gid":8287,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1607587459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1607587459},{"type":"v1","addr":"172.21.15.81:6835","nonce":1607587459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":224}],"filesystems":[{"mdsmap":{"epoch":234,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:13:42.353072+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":8290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8290":{"gid":8290,"name":"a","rank":1,"incarnation":231,"state":"up:stopping","state_seq":5,"addr":"172.21.15.28:6833/1864206570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1864206570},{"type":"v1","addr":"172.21.15.28:6833","nonce":1864206570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:14:00.271 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 234 2024-02-21T23:14:00.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.286+0000 7fba7f200640 1 -- 172.21.15.28:0/2828812043 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fba5c06b500 msgr2=0x7fba5c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:00.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.286+0000 7fba7f200640 1 --2- 172.21.15.28:0/2828812043 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fba5c06b500 0x7fba5c06d9c0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fba800c5150 tx=0x7fba68008040 comp rx=0 tx=0).stop 2024-02-21T23:14:00.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.286+0000 7fba7f200640 1 -- 172.21.15.28:0/2828812043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba8006dbf0 msgr2=0x7fba801eb4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:00.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.286+0000 7fba7f200640 1 --2- 172.21.15.28:0/2828812043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba8006dbf0 0x7fba801eb4d0 secure :-1 s=READY pgs=2440 cs=0 l=1 rev1=1 crypto rx=0x7fba70000f80 tx=0x7fba70004a40 comp rx=0 tx=0).stop 2024-02-21T23:14:00.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.286+0000 7fba7f200640 1 -- 172.21.15.28:0/2828812043 shutdown_connections 2024-02-21T23:14:00.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.286+0000 7fba7f200640 1 --2- 172.21.15.28:0/2828812043 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fba5c06b500 0x7fba5c06d9c0 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:00.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.286+0000 7fba7f200640 1 --2- 172.21.15.28:0/2828812043 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fba800c5720 0x7fba800c1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:00.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.286+0000 7fba7f200640 1 --2- 172.21.15.28:0/2828812043 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fba800aff90 0x7fba801515e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:00.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.286+0000 7fba7f200640 1 --2- 172.21.15.28:0/2828812043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba8006dbf0 0x7fba801eb4d0 unknown :-1 s=CLOSED pgs=2440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:00.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.286+0000 7fba7f200640 1 -- 172.21.15.28:0/2828812043 >> 172.21.15.28:0/2828812043 conn(0x7fba800ac370 msgr2=0x7fba800ac750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:14:00.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.286+0000 7fba7f200640 1 -- 172.21.15.28:0/2828812043 shutdown_connections 2024-02-21T23:14:00.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:00.286+0000 7fba7f200640 1 -- 172.21.15.28:0/2828812043 wait complete. 2024-02-21T23:14:00.696 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 234, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:13:42.353072+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 8290}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8290': {'gid': 8290, 'name': 'a', 'rank': 1, 'incarnation': 231, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.28:6833/1864206570', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 1864206570}, {'type': 'v1', 'addr': '172.21.15.28:6833', 'nonce': 1864206570}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:14:00.696 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8290:up:stopping 2024-02-21T23:14:01.697 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:14:01.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.933+0000 7f810f200640 1 Processor -- start 2024-02-21T23:14:01.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.933+0000 7f810f200640 1 -- start start 2024-02-21T23:14:01.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.935+0000 7f810f200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f811006c0f0 0x7f8110155410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:01.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.935+0000 7f810f200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f8110151410 0x7f81101518a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:01.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.935+0000 7f810f200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81100c1610 con 0x7f81100a84b0 2024-02-21T23:14:01.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.935+0000 7f810f200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81100c3060 con 0x7f8110151410 2024-02-21T23:14:01.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.935+0000 7f810f200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81100c3340 con 0x7f811006c0f0 2024-02-21T23:14:01.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.935+0000 7f810d400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f81100a84b0 0x7f811014e7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:47562/0 (socket says 172.21.15.28:47562) 2024-02-21T23:14:01.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.935+0000 7f810d400640 1 -- 172.21.15.28:0/502100892 learned_addr learned my addr 172.21.15.28:0/502100892 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:14:01.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.936+0000 7f810e800640 1 --2- 172.21.15.28:0/502100892 >> v2:172.21.15.81:3300/0 conn(0x7f8110151410 0x7f81101518a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:01.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.936+0000 7f810de00640 1 --2- 172.21.15.28:0/502100892 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f811006c0f0 0x7f8110155410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:01.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.936+0000 7f810ca00640 1 -- 172.21.15.28:0/502100892 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2837211918 0 0) 0x7f81100c1610 con 0x7f81100a84b0 2024-02-21T23:14:01.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.936+0000 7f810ca00640 1 -- 172.21.15.28:0/502100892 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80f4003520 con 0x7f81100a84b0 2024-02-21T23:14:01.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.936+0000 7f810ca00640 1 -- 172.21.15.28:0/502100892 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3960454911 0 0) 0x7f80f4003520 con 0x7f81100a84b0 2024-02-21T23:14:01.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.936+0000 7f810ca00640 1 -- 172.21.15.28:0/502100892 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f811006c0f0 msgr2=0x7f8110155410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:14:01.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.936+0000 7f810ca00640 1 --2- 172.21.15.28:0/502100892 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f811006c0f0 0x7f8110155410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:01.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.936+0000 7f810ca00640 1 -- 172.21.15.28:0/502100892 >> v2:172.21.15.81:3300/0 conn(0x7f8110151410 msgr2=0x7f81101518a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:14:01.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.936+0000 7f810ca00640 1 --2- 172.21.15.28:0/502100892 >> v2:172.21.15.81:3300/0 conn(0x7f8110151410 0x7f81101518a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:01.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.936+0000 7f810ca00640 1 -- 172.21.15.28:0/502100892 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81100c3620 con 0x7f81100a84b0 2024-02-21T23:14:01.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.936+0000 7f810ca00640 1 -- 172.21.15.28:0/502100892 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f80fc0019f0 con 0x7f81100a84b0 2024-02-21T23:14:01.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.936+0000 7f810ca00640 1 -- 172.21.15.28:0/502100892 >> v1:172.21.15.28:6789/0 conn(0x7f81100a84b0 legacy=0x7f811014e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:01.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.936+0000 7f810ca00640 1 --2- 172.21.15.28:0/502100892 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f811006c0f0 0x7f80f40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:01.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.936+0000 7f810ca00640 1 --2- 172.21.15.28:0/502100892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80f4003f80 0x7f80f4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:01.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.937+0000 7f810ca00640 1 --2- 172.21.15.28:0/502100892 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f80f4004d40 0x7f80f40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:01.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.937+0000 7f810ca00640 1 -- 172.21.15.28:0/502100892 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81100c3060 con 0x7f80f4003f80 2024-02-21T23:14:01.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.937+0000 7f810ca00640 1 -- 172.21.15.28:0/502100892 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81100c3340 con 0x7f80f4004d40 2024-02-21T23:14:01.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.937+0000 7f810ca00640 1 -- 172.21.15.28:0/502100892 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80f400ba70 con 0x7f811006c0f0 2024-02-21T23:14:01.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.937+0000 7f810d400640 1 --2- 172.21.15.28:0/502100892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80f4003f80 0x7f80f4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:01.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.937+0000 7f810de00640 1 --2- 172.21.15.28:0/502100892 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f811006c0f0 0x7f80f40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:01.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.937+0000 7f810e800640 1 --2- 172.21.15.28:0/502100892 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f80f4004d40 0x7f80f40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:01.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.937+0000 7f810d400640 1 -- 172.21.15.28:0/502100892 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f811006c0f0 msgr2=0x7f80f40039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:14:01.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.937+0000 7f810d400640 1 --2- 172.21.15.28:0/502100892 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f811006c0f0 0x7f80f40039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:01.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.937+0000 7f810d400640 1 -- 172.21.15.28:0/502100892 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f80f4004d40 msgr2=0x7f80f40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:14:01.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.937+0000 7f810d400640 1 --2- 172.21.15.28:0/502100892 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f80f4004d40 0x7f80f40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:01.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.937+0000 7f810d400640 1 -- 172.21.15.28:0/502100892 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80f4003520 con 0x7f80f4003f80 2024-02-21T23:14:01.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.938+0000 7f810d400640 1 --2- 172.21.15.28:0/502100892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80f4003f80 0x7f80f4004800 secure :-1 s=READY pgs=2442 cs=0 l=1 rev1=1 crypto rx=0x7f80fc00a7d0 tx=0x7f80f4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9ee86b74de5a374 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:01.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.939+0000 7f810ca00640 1 -- 172.21.15.28:0/502100892 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80fc00a3f0 con 0x7f80f4003f80 2024-02-21T23:14:01.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.939+0000 7f810ca00640 1 -- 172.21.15.28:0/502100892 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80fc02ce50 con 0x7f80f4003f80 2024-02-21T23:14:01.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.939+0000 7f810ca00640 1 -- 172.21.15.28:0/502100892 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80fc0348d0 con 0x7f80f4003f80 2024-02-21T23:14:01.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.939+0000 7f810f200640 1 -- 172.21.15.28:0/502100892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80f4003f80 msgr2=0x7f80f4004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:01.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.939+0000 7f810f200640 1 --2- 172.21.15.28:0/502100892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80f4003f80 0x7f80f4004800 secure :-1 s=READY pgs=2442 cs=0 l=1 rev1=1 crypto rx=0x7f80fc00a7d0 tx=0x7f80f4005830 comp rx=0 tx=0).stop 2024-02-21T23:14:01.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.939+0000 7f810e800640 1 -- 172.21.15.28:0/502100892 reap_dead start 2024-02-21T23:14:01.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.940+0000 7f810f200640 1 -- 172.21.15.28:0/502100892 shutdown_connections 2024-02-21T23:14:01.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.940+0000 7f810f200640 1 -- 172.21.15.28:0/502100892 >> 172.21.15.28:0/502100892 conn(0x7f8110071910 msgr2=0x7f8110071cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:14:01.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.940+0000 7f810f200640 1 -- 172.21.15.28:0/502100892 shutdown_connections 2024-02-21T23:14:01.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.940+0000 7f810f200640 1 -- 172.21.15.28:0/502100892 wait complete. 2024-02-21T23:14:01.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.942+0000 7f810f200640 1 Processor -- start 2024-02-21T23:14:01.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.942+0000 7f810f200640 1 -- start start 2024-02-21T23:14:01.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.943+0000 7f810f200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f80f4003f80 0x7f81101ee350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:01.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.944+0000 7f810f200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f811015fac0 0x7f8110137c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:01.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.944+0000 7f810f200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81101ee890 0x7f8110138180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:01.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.944+0000 7f810f200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8110072ca0 con 0x7f811015fac0 2024-02-21T23:14:01.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.944+0000 7f810f200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8110151de0 con 0x7f81101ee890 2024-02-21T23:14:01.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.944+0000 7f810f200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81100c5540 con 0x7f80f4003f80 2024-02-21T23:14:01.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.944+0000 7f810d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f811015fac0 0x7f8110137c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:01.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.944+0000 7f810d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f811015fac0 0x7f8110137c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59242/0 (socket says 172.21.15.28:59242) 2024-02-21T23:14:01.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.944+0000 7f810d400640 1 -- 172.21.15.28:0/1541936986 learned_addr learned my addr 172.21.15.28:0/1541936986 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:14:01.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.944+0000 7f810de00640 1 --2- 172.21.15.28:0/1541936986 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f80f4003f80 0x7f81101ee350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:01.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.944+0000 7f810e800640 1 --2- 172.21.15.28:0/1541936986 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81101ee890 0x7f8110138180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:01.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.944+0000 7f810d400640 1 -- 172.21.15.28:0/1541936986 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f80f4003f80 msgr2=0x7f81101ee350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:01.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.944+0000 7f810d400640 1 --2- 172.21.15.28:0/1541936986 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f80f4003f80 0x7f81101ee350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:01.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.944+0000 7f810d400640 1 -- 172.21.15.28:0/1541936986 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81101ee890 msgr2=0x7f8110138180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:01.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.944+0000 7f810d400640 1 --2- 172.21.15.28:0/1541936986 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81101ee890 0x7f8110138180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:01.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.944+0000 7f810d400640 1 -- 172.21.15.28:0/1541936986 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81100c5d50 con 0x7f811015fac0 2024-02-21T23:14:01.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.944+0000 7f810de00640 1 --2- 172.21.15.28:0/1541936986 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f80f4003f80 0x7f81101ee350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:14:01.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.944+0000 7f810e800640 1 --2- 172.21.15.28:0/1541936986 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81101ee890 0x7f8110138180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:14:01.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.945+0000 7f810d400640 1 --2- 172.21.15.28:0/1541936986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f811015fac0 0x7f8110137c40 secure :-1 s=READY pgs=2443 cs=0 l=1 rev1=1 crypto rx=0x7f80fc00a7d0 tx=0x7f80fc001cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:01.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.945+0000 7f8106000640 1 -- 172.21.15.28:0/1541936986 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80fc034840 con 0x7f811015fac0 2024-02-21T23:14:01.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.945+0000 7f8106000640 1 -- 172.21.15.28:0/1541936986 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80fc03a040 con 0x7f811015fac0 2024-02-21T23:14:01.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.945+0000 7f8106000640 1 -- 172.21.15.28:0/1541936986 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80fc034c40 con 0x7f811015fac0 2024-02-21T23:14:01.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.945+0000 7f810f200640 1 -- 172.21.15.28:0/1541936986 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81101508b0 con 0x7f811015fac0 2024-02-21T23:14:01.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.945+0000 7f810f200640 1 -- 172.21.15.28:0/1541936986 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81100c5790 con 0x7f811015fac0 2024-02-21T23:14:01.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.946+0000 7f810f200640 1 -- 172.21.15.28:0/1541936986 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f811004f290 con 0x7f811015fac0 2024-02-21T23:14:01.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.947+0000 7f8106000640 1 -- 172.21.15.28:0/1541936986 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f80fc03f420 con 0x7f811015fac0 2024-02-21T23:14:01.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.948+0000 7f8106000640 1 --2- 172.21.15.28:0/1541936986 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f80f006b4d0 0x7f80f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:01.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.949+0000 7f8106000640 1 -- 172.21.15.28:0/1541936986 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f80fc0acd00 con 0x7f811015fac0 2024-02-21T23:14:01.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.949+0000 7f810de00640 1 --2- 172.21.15.28:0/1541936986 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f80f006b4d0 0x7f80f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:01.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.950+0000 7f810de00640 1 --2- 172.21.15.28:0/1541936986 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f80f006b4d0 0x7f80f006d990 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f811013f100 tx=0x7f8100008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:01.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:01.953+0000 7f8106000640 1 -- 172.21.15.28:0/1541936986 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f81100c5d50 con 0x7f811015fac0 2024-02-21T23:14:02.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:02.194+0000 7f810f200640 1 -- 172.21.15.28:0/1541936986 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8110155830 con 0x7f811015fac0 2024-02-21T23:14:02.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:02.196+0000 7f8106000640 1 -- 172.21.15.28:0/1541936986 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 234 v234) v1 ==== 95+0+4964 (secure 0 0 0) 0x7f80fc032380 con 0x7f811015fac0 2024-02-21T23:14:02.184 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:14:02.184 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":234,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7265,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":224},{"gid":8287,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1607587459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1607587459},{"type":"v1","addr":"172.21.15.81:6835","nonce":1607587459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":224}],"filesystems":[{"mdsmap":{"epoch":234,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:13:42.353072+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":8290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8290":{"gid":8290,"name":"a","rank":1,"incarnation":231,"state":"up:stopping","state_seq":5,"addr":"172.21.15.28:6833/1864206570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1864206570},{"type":"v1","addr":"172.21.15.28:6833","nonce":1864206570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:14:02.184 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 234 2024-02-21T23:14:02.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:02.199+0000 7f810f200640 1 -- 172.21.15.28:0/1541936986 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f80f006b4d0 msgr2=0x7f80f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:02.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:02.199+0000 7f810f200640 1 --2- 172.21.15.28:0/1541936986 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f80f006b4d0 0x7f80f006d990 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f811013f100 tx=0x7f8100008040 comp rx=0 tx=0).stop 2024-02-21T23:14:02.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:02.199+0000 7f810f200640 1 -- 172.21.15.28:0/1541936986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f811015fac0 msgr2=0x7f8110137c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:02.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:02.199+0000 7f810f200640 1 --2- 172.21.15.28:0/1541936986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f811015fac0 0x7f8110137c40 secure :-1 s=READY pgs=2443 cs=0 l=1 rev1=1 crypto rx=0x7f80fc00a7d0 tx=0x7f80fc001cf0 comp rx=0 tx=0).stop 2024-02-21T23:14:02.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:02.199+0000 7f810f200640 1 -- 172.21.15.28:0/1541936986 shutdown_connections 2024-02-21T23:14:02.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:02.199+0000 7f810f200640 1 --2- 172.21.15.28:0/1541936986 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f80f006b4d0 0x7f80f006d990 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:02.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:02.199+0000 7f810f200640 1 --2- 172.21.15.28:0/1541936986 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81101ee890 0x7f8110138180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:02.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:02.199+0000 7f810f200640 1 --2- 172.21.15.28:0/1541936986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f811015fac0 0x7f8110137c40 unknown :-1 s=CLOSED pgs=2443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:02.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:02.199+0000 7f810f200640 1 --2- 172.21.15.28:0/1541936986 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f80f4003f80 0x7f81101ee350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:02.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:02.199+0000 7f810f200640 1 -- 172.21.15.28:0/1541936986 >> 172.21.15.28:0/1541936986 conn(0x7f8110071910 msgr2=0x7f811014f230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:14:02.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:02.199+0000 7f810f200640 1 -- 172.21.15.28:0/1541936986 shutdown_connections 2024-02-21T23:14:02.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:02.199+0000 7f810f200640 1 -- 172.21.15.28:0/1541936986 wait complete. 2024-02-21T23:14:02.610 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 234, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:13:42.353072+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 8290}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8290': {'gid': 8290, 'name': 'a', 'rank': 1, 'incarnation': 231, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.28:6833/1864206570', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 1864206570}, {'type': 'v1', 'addr': '172.21.15.28:6833', 'nonce': 1864206570}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:14:02.610 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8290:up:stopping 2024-02-21T23:14:03.611 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:14:03.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:03.871+0000 7f442ac00640 1 Processor -- start 2024-02-21T23:14:03.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:03.872+0000 7f442ac00640 1 -- start start 2024-02-21T23:14:03.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:03.873+0000 7f442ac00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f44240c8a80 0x7f4424156c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:03.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:03.873+0000 7f442ac00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f44241540b0 0x7f442415d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:03.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:03.873+0000 7f442ac00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44240c8000 con 0x7f44240c8e60 2024-02-21T23:14:03.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:03.873+0000 7f442ac00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44240c9310 con 0x7f44240c8a80 2024-02-21T23:14:03.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:03.873+0000 7f442ac00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44240c82e0 con 0x7f44241540b0 2024-02-21T23:14:03.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:03.874+0000 7f4429800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f44240c8a80 0x7f4424156c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:03.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:03.874+0000 7f4429800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f44240c8a80 0x7f4424156c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59246/0 (socket says 172.21.15.28:59246) 2024-02-21T23:14:03.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:03.874+0000 7f4428e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f44240c8e60 0x7f4424153640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:47578/0 (socket says 172.21.15.28:47578) 2024-02-21T23:14:03.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:03.874+0000 7f4429800640 1 -- 172.21.15.28:0/3231318854 learned_addr learned my addr 172.21.15.28:0/3231318854 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:14:03.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:03.874+0000 7f442a200640 1 --2- 172.21.15.28:0/3231318854 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f44241540b0 0x7f442415d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:03.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:03.874+0000 7f4429800640 1 -- 172.21.15.28:0/3231318854 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f44241540b0 msgr2=0x7f442415d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:14:03.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:03.874+0000 7f4429800640 1 --2- 172.21.15.28:0/3231318854 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f44241540b0 0x7f442415d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:03.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:03.874+0000 7f4429800640 1 -- 172.21.15.28:0/3231318854 >> v1:172.21.15.28:6789/0 conn(0x7f44240c8e60 legacy=0x7f4424153640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:03.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:03.874+0000 7f4429800640 1 -- 172.21.15.28:0/3231318854 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44241561b0 con 0x7f44240c8a80 2024-02-21T23:14:03.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:03.874+0000 7f442a200640 1 --2- 172.21.15.28:0/3231318854 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f44241540b0 0x7f442415d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:14:03.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:03.875+0000 7f4429800640 1 --2- 172.21.15.28:0/3231318854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44240c8a80 0x7f4424156c90 secure :-1 s=READY pgs=2445 cs=0 l=1 rev1=1 crypto rx=0x7f4418009870 tx=0x7f441802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a414f96029aeeb3f server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:03.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:03.876+0000 7f4423e00640 1 -- 172.21.15.28:0/3231318854 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4418035020 con 0x7f44240c8a80 2024-02-21T23:14:03.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:03.876+0000 7f4423e00640 1 -- 172.21.15.28:0/3231318854 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4418031640 con 0x7f44240c8a80 2024-02-21T23:14:03.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:03.876+0000 7f4423e00640 1 -- 172.21.15.28:0/3231318854 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f441802d040 con 0x7f44240c8a80 2024-02-21T23:14:03.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:03.876+0000 7f442ac00640 1 -- 172.21.15.28:0/3231318854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44240c8a80 msgr2=0x7f4424156c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:03.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:03.876+0000 7f442ac00640 1 --2- 172.21.15.28:0/3231318854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44240c8a80 0x7f4424156c90 secure :-1 s=READY pgs=2445 cs=0 l=1 rev1=1 crypto rx=0x7f4418009870 tx=0x7f441802b8f0 comp rx=0 tx=0).stop 2024-02-21T23:14:03.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:03.877+0000 7f442ac00640 1 -- 172.21.15.28:0/3231318854 shutdown_connections 2024-02-21T23:14:03.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:03.877+0000 7f442ac00640 1 --2- 172.21.15.28:0/3231318854 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f44241540b0 0x7f442415d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:03.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:03.877+0000 7f442ac00640 1 --2- 172.21.15.28:0/3231318854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44240c8a80 0x7f4424156c90 unknown :-1 s=CLOSED pgs=2445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:03.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:03.877+0000 7f442ac00640 1 -- 172.21.15.28:0/3231318854 >> 172.21.15.28:0/3231318854 conn(0x7f442406b260 msgr2=0x7f442406b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:14:03.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:03.877+0000 7f442ac00640 1 -- 172.21.15.28:0/3231318854 shutdown_connections 2024-02-21T23:14:03.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:03.877+0000 7f442ac00640 1 -- 172.21.15.28:0/3231318854 wait complete. 2024-02-21T23:14:03.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:03.879+0000 7f442ac00640 1 Processor -- start 2024-02-21T23:14:03.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:03.880+0000 7f442ac00640 1 -- start start 2024-02-21T23:14:03.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:03.881+0000 7f442ac00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f44240c8a80 0x7f44240c1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:03.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:03.881+0000 7f442ac00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44240c8e60 0x7f4424149910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:03.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:03.881+0000 7f442ac00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44241540b0 0x7f4424149e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:03.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:03.881+0000 7f442ac00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44240c7c90 con 0x7f44241540b0 2024-02-21T23:14:03.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:03.881+0000 7f442ac00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4424156770 con 0x7f44240c8e60 2024-02-21T23:14:03.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:03.881+0000 7f442ac00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44240c2c30 con 0x7f44240c8a80 2024-02-21T23:14:03.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:03.882+0000 7f442a200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44241540b0 0x7f4424149e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:03.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:03.882+0000 7f442a200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44241540b0 0x7f4424149e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59248/0 (socket says 172.21.15.28:59248) 2024-02-21T23:14:03.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:03.882+0000 7f442a200640 1 -- 172.21.15.28:0/2495419180 learned_addr learned my addr 172.21.15.28:0/2495419180 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:14:03.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:03.882+0000 7f4428e00640 1 --2- 172.21.15.28:0/2495419180 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44240c8e60 0x7f4424149910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:03.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:03.882+0000 7f4429800640 1 --2- 172.21.15.28:0/2495419180 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f44240c8a80 0x7f44240c1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:03.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:03.882+0000 7f442a200640 1 -- 172.21.15.28:0/2495419180 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f44240c8a80 msgr2=0x7f44240c1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:03.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:03.882+0000 7f442a200640 1 --2- 172.21.15.28:0/2495419180 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f44240c8a80 0x7f44240c1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:03.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:03.882+0000 7f442a200640 1 -- 172.21.15.28:0/2495419180 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44240c8e60 msgr2=0x7f4424149910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:03.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:03.882+0000 7f442a200640 1 --2- 172.21.15.28:0/2495419180 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44240c8e60 0x7f4424149910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:03.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:03.882+0000 7f442a200640 1 -- 172.21.15.28:0/2495419180 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f442406cb20 con 0x7f44241540b0 2024-02-21T23:14:03.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:03.882+0000 7f4428e00640 1 --2- 172.21.15.28:0/2495419180 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44240c8e60 0x7f4424149910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:14:03.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:03.882+0000 7f4429800640 1 --2- 172.21.15.28:0/2495419180 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f44240c8a80 0x7f44240c1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:14:03.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:03.882+0000 7f442a200640 1 --2- 172.21.15.28:0/2495419180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44241540b0 0x7f4424149e50 secure :-1 s=READY pgs=2446 cs=0 l=1 rev1=1 crypto rx=0x7f441c00cef0 tx=0x7f441c00a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:03.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:03.883+0000 7f4421600640 1 -- 172.21.15.28:0/2495419180 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f441c01a070 con 0x7f44241540b0 2024-02-21T23:14:03.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:03.883+0000 7f4421600640 1 -- 172.21.15.28:0/2495419180 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f441c00d860 con 0x7f44241540b0 2024-02-21T23:14:03.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:03.883+0000 7f4421600640 1 -- 172.21.15.28:0/2495419180 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f441c00dc50 con 0x7f44241540b0 2024-02-21T23:14:03.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:03.883+0000 7f442ac00640 1 -- 172.21.15.28:0/2495419180 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f442406ce00 con 0x7f44241540b0 2024-02-21T23:14:03.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:03.883+0000 7f442ac00640 1 -- 172.21.15.28:0/2495419180 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f442406d3c0 con 0x7f44241540b0 2024-02-21T23:14:03.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:03.884+0000 7f442ac00640 1 -- 172.21.15.28:0/2495419180 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f442404f290 con 0x7f44241540b0 2024-02-21T23:14:03.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:03.885+0000 7f4421600640 1 -- 172.21.15.28:0/2495419180 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f441c01d070 con 0x7f44241540b0 2024-02-21T23:14:03.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:03.887+0000 7f4421600640 1 --2- 172.21.15.28:0/2495419180 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f440806b4d0 0x7f440806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:03.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:03.887+0000 7f4421600640 1 -- 172.21.15.28:0/2495419180 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f441c08ac30 con 0x7f44241540b0 2024-02-21T23:14:03.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:03.887+0000 7f4429800640 1 --2- 172.21.15.28:0/2495419180 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f440806b4d0 0x7f440806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:03.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:03.888+0000 7f4429800640 1 --2- 172.21.15.28:0/2495419180 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f440806b4d0 0x7f440806d990 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f4418036000 tx=0x7f4418008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:03.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:03.891+0000 7f4421600640 1 -- 172.21.15.28:0/2495419180 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f441c0599d0 con 0x7f44241540b0 2024-02-21T23:14:04.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:04.164+0000 7f442ac00640 1 -- 172.21.15.28:0/2495419180 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4424152800 con 0x7f44241540b0 2024-02-21T23:14:04.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:04.165+0000 7f4421600640 1 -- 172.21.15.28:0/2495419180 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 234 v234) v1 ==== 95+0+4964 (secure 0 0 0) 0x7f441c0592d0 con 0x7f44241540b0 2024-02-21T23:14:04.152 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:14:04.153 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":234,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7265,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":224},{"gid":8287,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1607587459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1607587459},{"type":"v1","addr":"172.21.15.81:6835","nonce":1607587459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":224}],"filesystems":[{"mdsmap":{"epoch":234,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:13:42.353072+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":8290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8290":{"gid":8290,"name":"a","rank":1,"incarnation":231,"state":"up:stopping","state_seq":5,"addr":"172.21.15.28:6833/1864206570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1864206570},{"type":"v1","addr":"172.21.15.28:6833","nonce":1864206570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:14:04.153 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 234 2024-02-21T23:14:04.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:04.168+0000 7f442ac00640 1 -- 172.21.15.28:0/2495419180 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f440806b4d0 msgr2=0x7f440806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:04.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:04.168+0000 7f442ac00640 1 --2- 172.21.15.28:0/2495419180 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f440806b4d0 0x7f440806d990 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f4418036000 tx=0x7f4418008030 comp rx=0 tx=0).stop 2024-02-21T23:14:04.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:04.168+0000 7f442ac00640 1 -- 172.21.15.28:0/2495419180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44241540b0 msgr2=0x7f4424149e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:04.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:04.168+0000 7f442ac00640 1 --2- 172.21.15.28:0/2495419180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44241540b0 0x7f4424149e50 secure :-1 s=READY pgs=2446 cs=0 l=1 rev1=1 crypto rx=0x7f441c00cef0 tx=0x7f441c00a710 comp rx=0 tx=0).stop 2024-02-21T23:14:04.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:04.168+0000 7f442ac00640 1 -- 172.21.15.28:0/2495419180 shutdown_connections 2024-02-21T23:14:04.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:04.168+0000 7f442ac00640 1 --2- 172.21.15.28:0/2495419180 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f440806b4d0 0x7f440806d990 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:04.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:04.168+0000 7f442ac00640 1 --2- 172.21.15.28:0/2495419180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44241540b0 0x7f4424149e50 unknown :-1 s=CLOSED pgs=2446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:04.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:04.168+0000 7f442ac00640 1 --2- 172.21.15.28:0/2495419180 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44240c8e60 0x7f4424149910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:04.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:04.168+0000 7f442ac00640 1 --2- 172.21.15.28:0/2495419180 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f44240c8a80 0x7f44240c1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:04.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:04.168+0000 7f442ac00640 1 -- 172.21.15.28:0/2495419180 >> 172.21.15.28:0/2495419180 conn(0x7f442406b260 msgr2=0x7f44240704f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:14:04.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:04.168+0000 7f442ac00640 1 -- 172.21.15.28:0/2495419180 shutdown_connections 2024-02-21T23:14:04.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:04.168+0000 7f442ac00640 1 -- 172.21.15.28:0/2495419180 wait complete. 2024-02-21T23:14:04.576 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 234, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:13:42.353072+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 8290}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8290': {'gid': 8290, 'name': 'a', 'rank': 1, 'incarnation': 231, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.28:6833/1864206570', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 1864206570}, {'type': 'v1', 'addr': '172.21.15.28:6833', 'nonce': 1864206570}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:14:04.576 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8290:up:stopping 2024-02-21T23:14:05.578 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:14:05.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:05.812+0000 7f4ff9400640 1 Processor -- start 2024-02-21T23:14:05.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:05.812+0000 7f4ff9400640 1 -- start start 2024-02-21T23:14:05.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:05.814+0000 7f4ff9400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4ff4155960 0x7f4ff41608a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:05.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:05.814+0000 7f4ff9400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ff40c74c0 con 0x7f4ff406dbf0 2024-02-21T23:14:05.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:05.814+0000 7f4ff9400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ff40c77a0 con 0x7f4ff40aff90 2024-02-21T23:14:05.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:05.814+0000 7f4ff9400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ff40c7270 con 0x7f4ff4155960 2024-02-21T23:14:05.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:05.814+0000 7f4ff8a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4ff4155960 0x7f4ff41608a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:05.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:05.814+0000 7f4ff3400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f4ff406dbf0 0x7f4ff4158ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:49952/0 (socket says 172.21.15.28:49952) 2024-02-21T23:14:05.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:05.814+0000 7f4ff3400640 1 -- 172.21.15.28:0/3428918315 learned_addr learned my addr 172.21.15.28:0/3428918315 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:14:05.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:05.814+0000 7f4ff8a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4ff4155960 0x7f4ff41608a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39292/0 (socket says 172.21.15.28:39292) 2024-02-21T23:14:05.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:05.815+0000 7f4ff8a00640 1 -- 172.21.15.28:0/3428918315 >> v1:172.21.15.81:6789/0 conn(0x7f4ff40aff90 legacy=0x7f4ff40c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:14:05.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:05.815+0000 7f4ff8a00640 1 -- 172.21.15.28:0/3428918315 >> v1:172.21.15.28:6789/0 conn(0x7f4ff406dbf0 legacy=0x7f4ff4158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:05.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:05.815+0000 7f4ff8a00640 1 -- 172.21.15.28:0/3428918315 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ff40c7a80 con 0x7f4ff4155960 2024-02-21T23:14:05.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:05.816+0000 7f4ff8a00640 1 --2- 172.21.15.28:0/3428918315 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ff4155960 0x7f4ff41608a0 secure :-1 s=READY pgs=2448 cs=0 l=1 rev1=1 crypto rx=0x7f4fec0097a0 tx=0x7f4fec02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6c4268bd46fe332 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:05.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:05.816+0000 7f4ff2a00640 1 -- 172.21.15.28:0/3428918315 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fec035020 con 0x7f4ff4155960 2024-02-21T23:14:05.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:05.816+0000 7f4ff2a00640 1 -- 172.21.15.28:0/3428918315 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fec031830 con 0x7f4ff4155960 2024-02-21T23:14:05.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:05.816+0000 7f4ff2a00640 1 -- 172.21.15.28:0/3428918315 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fec031c00 con 0x7f4ff4155960 2024-02-21T23:14:05.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:05.817+0000 7f4ff9400640 1 -- 172.21.15.28:0/3428918315 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ff4155960 msgr2=0x7f4ff41608a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:05.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:05.817+0000 7f4ff9400640 1 --2- 172.21.15.28:0/3428918315 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ff4155960 0x7f4ff41608a0 secure :-1 s=READY pgs=2448 cs=0 l=1 rev1=1 crypto rx=0x7f4fec0097a0 tx=0x7f4fec02b8d0 comp rx=0 tx=0).stop 2024-02-21T23:14:05.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:05.817+0000 7f4ff9400640 1 -- 172.21.15.28:0/3428918315 shutdown_connections 2024-02-21T23:14:05.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:05.817+0000 7f4ff9400640 1 --2- 172.21.15.28:0/3428918315 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ff4155960 0x7f4ff41608a0 unknown :-1 s=CLOSED pgs=2448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:05.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:05.817+0000 7f4ff9400640 1 -- 172.21.15.28:0/3428918315 >> 172.21.15.28:0/3428918315 conn(0x7f4ff40ac370 msgr2=0x7f4ff40ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:14:05.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:05.817+0000 7f4ff9400640 1 -- 172.21.15.28:0/3428918315 shutdown_connections 2024-02-21T23:14:05.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:05.817+0000 7f4ff9400640 1 -- 172.21.15.28:0/3428918315 wait complete. 2024-02-21T23:14:05.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:05.819+0000 7f4ff9400640 1 Processor -- start 2024-02-21T23:14:05.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:05.819+0000 7f4ff9400640 1 -- start start 2024-02-21T23:14:05.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:05.820+0000 7f4ff9400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ff406dbf0 0x7f4ff40c3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:05.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:05.821+0000 7f4ff9400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4ff40aff90 0x7f4ff41e1a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:05.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:05.821+0000 7f4ff9400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ff4155960 0x7f4ff41e1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:05.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:05.821+0000 7f4ff9400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff40c58f0 con 0x7f4ff406dbf0 2024-02-21T23:14:05.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:05.821+0000 7f4ff9400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff4158630 con 0x7f4ff4155960 2024-02-21T23:14:05.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:05.821+0000 7f4ff9400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff40c5a70 con 0x7f4ff40aff90 2024-02-21T23:14:05.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:05.821+0000 7f4ff3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ff406dbf0 0x7f4ff40c3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:05.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:05.821+0000 7f4ff3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ff406dbf0 0x7f4ff40c3ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39294/0 (socket says 172.21.15.28:39294) 2024-02-21T23:14:05.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:05.821+0000 7f4ff3e00640 1 -- 172.21.15.28:0/4073321612 learned_addr learned my addr 172.21.15.28:0/4073321612 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:14:05.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:05.821+0000 7f4ff3e00640 1 -- 172.21.15.28:0/4073321612 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4ff40aff90 msgr2=0x7f4ff41e1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:05.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:05.821+0000 7f4ff3400640 1 --2- 172.21.15.28:0/4073321612 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4ff40aff90 0x7f4ff41e1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:05.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:05.821+0000 7f4ff8a00640 1 --2- 172.21.15.28:0/4073321612 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ff4155960 0x7f4ff41e1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:05.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:05.821+0000 7f4ff3e00640 1 --2- 172.21.15.28:0/4073321612 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4ff40aff90 0x7f4ff41e1a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:05.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:05.821+0000 7f4ff3e00640 1 -- 172.21.15.28:0/4073321612 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ff4155960 msgr2=0x7f4ff41e1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:05.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:05.821+0000 7f4ff3e00640 1 --2- 172.21.15.28:0/4073321612 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ff4155960 0x7f4ff41e1fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:05.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:05.821+0000 7f4ff3e00640 1 -- 172.21.15.28:0/4073321612 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ff40c0e20 con 0x7f4ff406dbf0 2024-02-21T23:14:05.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:05.821+0000 7f4ff3400640 1 --2- 172.21.15.28:0/4073321612 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4ff40aff90 0x7f4ff41e1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:14:05.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:05.822+0000 7f4ff3e00640 1 --2- 172.21.15.28:0/4073321612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ff406dbf0 0x7f4ff40c3ce0 secure :-1 s=READY pgs=2449 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc00c430 tx=0x7f4fdc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:05.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:05.822+0000 7f4febe00640 1 -- 172.21.15.28:0/4073321612 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fdc011790 con 0x7f4ff406dbf0 2024-02-21T23:14:05.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:05.822+0000 7f4febe00640 1 -- 172.21.15.28:0/4073321612 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fdc017070 con 0x7f4ff406dbf0 2024-02-21T23:14:05.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:05.822+0000 7f4ff9400640 1 -- 172.21.15.28:0/4073321612 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ff40c1100 con 0x7f4ff406dbf0 2024-02-21T23:14:05.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:05.822+0000 7f4febe00640 1 -- 172.21.15.28:0/4073321612 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fdc011b70 con 0x7f4ff406dbf0 2024-02-21T23:14:05.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:05.822+0000 7f4ff9400640 1 -- 172.21.15.28:0/4073321612 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ff40c4220 con 0x7f4ff406dbf0 2024-02-21T23:14:05.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:05.824+0000 7f4ff9400640 1 -- 172.21.15.28:0/4073321612 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ff404f290 con 0x7f4ff406dbf0 2024-02-21T23:14:05.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:05.824+0000 7f4febe00640 1 -- 172.21.15.28:0/4073321612 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4fdc011d10 con 0x7f4ff406dbf0 2024-02-21T23:14:05.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:05.825+0000 7f4febe00640 1 --2- 172.21.15.28:0/4073321612 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4fd006b4d0 0x7f4fd006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:05.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:05.825+0000 7f4febe00640 1 -- 172.21.15.28:0/4073321612 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4fdc08a150 con 0x7f4ff406dbf0 2024-02-21T23:14:05.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:05.826+0000 7f4ff3400640 1 --2- 172.21.15.28:0/4073321612 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4fd006b4d0 0x7f4fd006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:05.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:05.827+0000 7f4ff3400640 1 --2- 172.21.15.28:0/4073321612 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4fd006b4d0 0x7f4fd006d990 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f4ff40af780 tx=0x7f4fe4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:05.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:05.835+0000 7f4febe00640 1 -- 172.21.15.28:0/4073321612 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4fdc058ed0 con 0x7f4ff406dbf0 2024-02-21T23:14:06.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:06.083+0000 7f4ff9400640 1 -- 172.21.15.28:0/4073321612 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4ff40bfeb0 con 0x7f4ff406dbf0 2024-02-21T23:14:06.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:06.084+0000 7f4febe00640 1 -- 172.21.15.28:0/4073321612 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 234 v234) v1 ==== 95+0+4964 (secure 0 0 0) 0x7f4fdc0587d0 con 0x7f4ff406dbf0 2024-02-21T23:14:06.072 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:14:06.072 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":234,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7265,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":224},{"gid":8287,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1607587459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1607587459},{"type":"v1","addr":"172.21.15.81:6835","nonce":1607587459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":224}],"filesystems":[{"mdsmap":{"epoch":234,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:13:42.353072+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":8290},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8290":{"gid":8290,"name":"a","rank":1,"incarnation":231,"state":"up:stopping","state_seq":5,"addr":"172.21.15.28:6833/1864206570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1864206570},{"type":"v1","addr":"172.21.15.28:6833","nonce":1864206570}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:14:06.073 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 234 2024-02-21T23:14:06.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:06.087+0000 7f4ff9400640 1 -- 172.21.15.28:0/4073321612 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4fd006b4d0 msgr2=0x7f4fd006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:06.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:06.087+0000 7f4ff9400640 1 --2- 172.21.15.28:0/4073321612 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4fd006b4d0 0x7f4fd006d990 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f4ff40af780 tx=0x7f4fe4009040 comp rx=0 tx=0).stop 2024-02-21T23:14:06.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:06.087+0000 7f4ff9400640 1 -- 172.21.15.28:0/4073321612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ff406dbf0 msgr2=0x7f4ff40c3ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:06.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:06.087+0000 7f4ff9400640 1 --2- 172.21.15.28:0/4073321612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ff406dbf0 0x7f4ff40c3ce0 secure :-1 s=READY pgs=2449 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc00c430 tx=0x7f4fdc00c8f0 comp rx=0 tx=0).stop 2024-02-21T23:14:06.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:06.087+0000 7f4ff9400640 1 -- 172.21.15.28:0/4073321612 shutdown_connections 2024-02-21T23:14:06.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:06.087+0000 7f4ff9400640 1 --2- 172.21.15.28:0/4073321612 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4fd006b4d0 0x7f4fd006d990 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:06.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:06.087+0000 7f4ff9400640 1 --2- 172.21.15.28:0/4073321612 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4ff40aff90 0x7f4ff41e1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:06.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:06.087+0000 7f4ff9400640 1 --2- 172.21.15.28:0/4073321612 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ff4155960 0x7f4ff41e1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:06.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:06.087+0000 7f4ff9400640 1 --2- 172.21.15.28:0/4073321612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ff406dbf0 0x7f4ff40c3ce0 unknown :-1 s=CLOSED pgs=2449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:06.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:06.087+0000 7f4ff9400640 1 -- 172.21.15.28:0/4073321612 >> 172.21.15.28:0/4073321612 conn(0x7f4ff40ac370 msgr2=0x7f4ff41569c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:14:06.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:06.088+0000 7f4ff9400640 1 -- 172.21.15.28:0/4073321612 shutdown_connections 2024-02-21T23:14:06.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:06.088+0000 7f4ff9400640 1 -- 172.21.15.28:0/4073321612 wait complete. 2024-02-21T23:14:06.498 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 234, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:13:42.353072+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 8290}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8290': {'gid': 8290, 'name': 'a', 'rank': 1, 'incarnation': 231, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.28:6833/1864206570', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 1864206570}, {'type': 'v1', 'addr': '172.21.15.28:6833', 'nonce': 1864206570}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:14:06.498 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8290:up:stopping 2024-02-21T23:14:07.499 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:14:07.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.737+0000 7f5c69c00640 1 Processor -- start 2024-02-21T23:14:07.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.737+0000 7f5c69c00640 1 -- start start 2024-02-21T23:14:07.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.739+0000 7f5c69c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f5c6406dbf0 0x7f5c64158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:07.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.739+0000 7f5c69c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5c64155960 0x7f5c64159000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:07.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.739+0000 7f5c69c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c640c74c0 con 0x7f5c640aff90 2024-02-21T23:14:07.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.739+0000 7f5c69c00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c640c7270 con 0x7f5c6406dbf0 2024-02-21T23:14:07.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.739+0000 7f5c69c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c64158630 con 0x7f5c64155960 2024-02-21T23:14:07.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.739+0000 7f5c63e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f5c640aff90 0x7f5c640c51e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:49958/0 (socket says 172.21.15.28:49958) 2024-02-21T23:14:07.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.739+0000 7f5c63e00640 1 -- 172.21.15.28:0/3261723918 learned_addr learned my addr 172.21.15.28:0/3261723918 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:14:07.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.739+0000 7f5c5be00640 1 --2- 172.21.15.28:0/3261723918 >> v2:172.21.15.81:3300/0 conn(0x7f5c6406dbf0 0x7f5c64158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:07.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.739+0000 7f5c69200640 1 --2- 172.21.15.28:0/3261723918 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5c64155960 0x7f5c64159000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:07.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.740+0000 7f5c63400640 1 -- 172.21.15.28:0/3261723918 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 822413892 0 0) 0x7f5c640c74c0 con 0x7f5c640aff90 2024-02-21T23:14:07.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.740+0000 7f5c63400640 1 -- 172.21.15.28:0/3261723918 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c48003520 con 0x7f5c640aff90 2024-02-21T23:14:07.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.740+0000 7f5c63400640 1 -- 172.21.15.28:0/3261723918 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1196356973 0 0) 0x7f5c48003520 con 0x7f5c640aff90 2024-02-21T23:14:07.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.740+0000 7f5c63400640 1 -- 172.21.15.28:0/3261723918 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5c64155960 msgr2=0x7f5c64159000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:14:07.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.740+0000 7f5c63400640 1 --2- 172.21.15.28:0/3261723918 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5c64155960 0x7f5c64159000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:07.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.740+0000 7f5c63400640 1 -- 172.21.15.28:0/3261723918 >> v2:172.21.15.81:3300/0 conn(0x7f5c6406dbf0 msgr2=0x7f5c64158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:14:07.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.740+0000 7f5c63400640 1 --2- 172.21.15.28:0/3261723918 >> v2:172.21.15.81:3300/0 conn(0x7f5c6406dbf0 0x7f5c64158ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:07.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.740+0000 7f5c63400640 1 -- 172.21.15.28:0/3261723918 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c640c77a0 con 0x7f5c640aff90 2024-02-21T23:14:07.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.740+0000 7f5c63400640 1 -- 172.21.15.28:0/3261723918 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f5c54001670 con 0x7f5c640aff90 2024-02-21T23:14:07.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.740+0000 7f5c63400640 1 -- 172.21.15.28:0/3261723918 >> v1:172.21.15.28:6789/0 conn(0x7f5c640aff90 legacy=0x7f5c640c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:07.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.740+0000 7f5c63400640 1 --2- 172.21.15.28:0/3261723918 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5c64155960 0x7f5c480039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:07.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.740+0000 7f5c63400640 1 --2- 172.21.15.28:0/3261723918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c48003f80 0x7f5c48004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:07.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.740+0000 7f5c63400640 1 --2- 172.21.15.28:0/3261723918 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5c48004d40 0x7f5c480051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:07.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.740+0000 7f5c63400640 1 -- 172.21.15.28:0/3261723918 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c640c7270 con 0x7f5c48003f80 2024-02-21T23:14:07.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.740+0000 7f5c63400640 1 -- 172.21.15.28:0/3261723918 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c64158630 con 0x7f5c48004d40 2024-02-21T23:14:07.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.740+0000 7f5c63400640 1 -- 172.21.15.28:0/3261723918 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c4800b9c0 con 0x7f5c64155960 2024-02-21T23:14:07.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.741+0000 7f5c5be00640 1 --2- 172.21.15.28:0/3261723918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c48003f80 0x7f5c48004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:07.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.741+0000 7f5c63e00640 1 --2- 172.21.15.28:0/3261723918 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5c64155960 0x7f5c480039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:07.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.741+0000 7f5c5be00640 1 -- 172.21.15.28:0/3261723918 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5c64155960 msgr2=0x7f5c480039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:14:07.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.741+0000 7f5c5be00640 1 --2- 172.21.15.28:0/3261723918 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5c64155960 0x7f5c480039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:07.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.741+0000 7f5c5be00640 1 -- 172.21.15.28:0/3261723918 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5c48004d40 msgr2=0x7f5c480051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:14:07.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.741+0000 7f5c5be00640 1 --2- 172.21.15.28:0/3261723918 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5c48004d40 0x7f5c480051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:07.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.741+0000 7f5c5be00640 1 -- 172.21.15.28:0/3261723918 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c48003520 con 0x7f5c48003f80 2024-02-21T23:14:07.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.742+0000 7f5c5be00640 1 --2- 172.21.15.28:0/3261723918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c48003f80 0x7f5c48004800 secure :-1 s=READY pgs=2451 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c009ac0 tx=0x7f5c48005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=984acc294c1aa841 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:07.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.742+0000 7f5c63400640 1 -- 172.21.15.28:0/3261723918 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c4c039070 con 0x7f5c48003f80 2024-02-21T23:14:07.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.743+0000 7f5c63400640 1 -- 172.21.15.28:0/3261723918 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c4c02c6b0 con 0x7f5c48003f80 2024-02-21T23:14:07.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.743+0000 7f5c63400640 1 -- 172.21.15.28:0/3261723918 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c4c009690 con 0x7f5c48003f80 2024-02-21T23:14:07.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.743+0000 7f5c69c00640 1 -- 172.21.15.28:0/3261723918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c48003f80 msgr2=0x7f5c48004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:07.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.743+0000 7f5c69c00640 1 --2- 172.21.15.28:0/3261723918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c48003f80 0x7f5c48004800 secure :-1 s=READY pgs=2451 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c009ac0 tx=0x7f5c48005830 comp rx=0 tx=0).stop 2024-02-21T23:14:07.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.743+0000 7f5c69200640 1 -- 172.21.15.28:0/3261723918 reap_dead start 2024-02-21T23:14:07.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.743+0000 7f5c69c00640 1 -- 172.21.15.28:0/3261723918 shutdown_connections 2024-02-21T23:14:07.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.743+0000 7f5c69c00640 1 -- 172.21.15.28:0/3261723918 >> 172.21.15.28:0/3261723918 conn(0x7f5c640ac370 msgr2=0x7f5c640ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:14:07.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.743+0000 7f5c69c00640 1 -- 172.21.15.28:0/3261723918 shutdown_connections 2024-02-21T23:14:07.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.744+0000 7f5c69c00640 1 -- 172.21.15.28:0/3261723918 wait complete. 2024-02-21T23:14:07.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.745+0000 7f5c69c00640 1 Processor -- start 2024-02-21T23:14:07.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.745+0000 7f5c69c00640 1 -- start start 2024-02-21T23:14:07.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.746+0000 7f5c69c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c48003f80 0x7f5c641eaac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:07.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.746+0000 7f5c69c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5c6414f380 0x7f5c640c3390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:07.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.746+0000 7f5c69c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5c6414ee30 0x7f5c641ee7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:07.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.746+0000 7f5c69c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c640c58f0 con 0x7f5c48003f80 2024-02-21T23:14:07.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.746+0000 7f5c69c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c640be0c0 con 0x7f5c6414ee30 2024-02-21T23:14:07.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.746+0000 7f5c69c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c640c5a70 con 0x7f5c6414f380 2024-02-21T23:14:07.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.746+0000 7f5c63e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c48003f80 0x7f5c641eaac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:07.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.747+0000 7f5c63e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c48003f80 0x7f5c641eaac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39302/0 (socket says 172.21.15.28:39302) 2024-02-21T23:14:07.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.747+0000 7f5c63e00640 1 -- 172.21.15.28:0/3516625390 learned_addr learned my addr 172.21.15.28:0/3516625390 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:14:07.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.747+0000 7f5c5be00640 1 --2- 172.21.15.28:0/3516625390 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5c6414f380 0x7f5c640c3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:07.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.747+0000 7f5c63e00640 1 -- 172.21.15.28:0/3516625390 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5c6414f380 msgr2=0x7f5c640c3390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:07.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.747+0000 7f5c63e00640 1 --2- 172.21.15.28:0/3516625390 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5c6414f380 0x7f5c640c3390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:07.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.747+0000 7f5c69200640 1 --2- 172.21.15.28:0/3516625390 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5c6414ee30 0x7f5c641ee7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:07.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.747+0000 7f5c63e00640 1 -- 172.21.15.28:0/3516625390 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5c6414ee30 msgr2=0x7f5c641ee7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:07.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.747+0000 7f5c63e00640 1 --2- 172.21.15.28:0/3516625390 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5c6414ee30 0x7f5c641ee7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:07.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.747+0000 7f5c63e00640 1 -- 172.21.15.28:0/3516625390 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c641ee030 con 0x7f5c48003f80 2024-02-21T23:14:07.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.747+0000 7f5c63e00640 1 --2- 172.21.15.28:0/3516625390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c48003f80 0x7f5c641eaac0 secure :-1 s=READY pgs=2452 cs=0 l=1 rev1=1 crypto rx=0x7f5c5400e780 tx=0x7f5c5400ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:07.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.748+0000 7f5c60c00640 1 -- 172.21.15.28:0/3516625390 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c5400c870 con 0x7f5c48003f80 2024-02-21T23:14:07.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.748+0000 7f5c60c00640 1 -- 172.21.15.28:0/3516625390 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c54018070 con 0x7f5c48003f80 2024-02-21T23:14:07.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.748+0000 7f5c69c00640 1 -- 172.21.15.28:0/3516625390 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c64154780 con 0x7f5c48003f80 2024-02-21T23:14:07.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.748+0000 7f5c60c00640 1 -- 172.21.15.28:0/3516625390 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c5400cc50 con 0x7f5c48003f80 2024-02-21T23:14:07.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.748+0000 7f5c69c00640 1 -- 172.21.15.28:0/3516625390 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c64154d40 con 0x7f5c48003f80 2024-02-21T23:14:07.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.749+0000 7f5c69c00640 1 -- 172.21.15.28:0/3516625390 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c6404f290 con 0x7f5c48003f80 2024-02-21T23:14:07.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.751+0000 7f5c60c00640 1 -- 172.21.15.28:0/3516625390 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5c54013420 con 0x7f5c48003f80 2024-02-21T23:14:07.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.752+0000 7f5c60c00640 1 --2- 172.21.15.28:0/3516625390 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5c4006b500 0x7f5c4006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:07.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.753+0000 7f5c60c00640 1 -- 172.21.15.28:0/3516625390 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f5c54089f90 con 0x7f5c48003f80 2024-02-21T23:14:07.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.753+0000 7f5c5be00640 1 --2- 172.21.15.28:0/3516625390 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5c4006b500 0x7f5c4006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:07.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.754+0000 7f5c5be00640 1 --2- 172.21.15.28:0/3516625390 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5c4006b500 0x7f5c4006d9c0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c036000 tx=0x7f5c4c030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:07.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:07.757+0000 7f5c60c00640 1 -- 172.21.15.28:0/3516625390 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f5c54022250 con 0x7f5c48003f80 2024-02-21T23:14:07.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:08.003+0000 7f5c69c00640 1 -- 172.21.15.28:0/3516625390 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5c640b0410 con 0x7f5c48003f80 2024-02-21T23:14:07.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:08.004+0000 7f5c60c00640 1 -- 172.21.15.28:0/3516625390 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 235 v235) v1 ==== 95+0+4172 (secure 0 0 0) 0x7f5c54058c20 con 0x7f5c48003f80 2024-02-21T23:14:07.992 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:14:07.992 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":235,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7265,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":224}],"filesystems":[{"mdsmap":{"epoch":235,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:14:07.624559+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":209,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":8287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8287":{"gid":8287,"name":"d","rank":1,"incarnation":235,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6835/1607587459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1607587459},{"type":"v1","addr":"172.21.15.81:6835","nonce":1607587459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:14:07.992 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 235 2024-02-21T23:14:07.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:08.006+0000 7f5c69c00640 1 -- 172.21.15.28:0/3516625390 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5c4006b500 msgr2=0x7f5c4006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:07.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:08.006+0000 7f5c69c00640 1 --2- 172.21.15.28:0/3516625390 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5c4006b500 0x7f5c4006d9c0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c036000 tx=0x7f5c4c030000 comp rx=0 tx=0).stop 2024-02-21T23:14:07.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:08.007+0000 7f5c69c00640 1 -- 172.21.15.28:0/3516625390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c48003f80 msgr2=0x7f5c641eaac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:07.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:08.007+0000 7f5c69c00640 1 --2- 172.21.15.28:0/3516625390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c48003f80 0x7f5c641eaac0 secure :-1 s=READY pgs=2452 cs=0 l=1 rev1=1 crypto rx=0x7f5c5400e780 tx=0x7f5c5400ec40 comp rx=0 tx=0).stop 2024-02-21T23:14:07.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:08.007+0000 7f5c69c00640 1 -- 172.21.15.28:0/3516625390 shutdown_connections 2024-02-21T23:14:07.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:08.007+0000 7f5c69c00640 1 --2- 172.21.15.28:0/3516625390 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5c4006b500 0x7f5c4006d9c0 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:07.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:08.007+0000 7f5c69c00640 1 --2- 172.21.15.28:0/3516625390 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5c6414f380 0x7f5c640c3390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:07.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:08.007+0000 7f5c69c00640 1 --2- 172.21.15.28:0/3516625390 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5c6414ee30 0x7f5c641ee7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:07.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:08.007+0000 7f5c69c00640 1 --2- 172.21.15.28:0/3516625390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c48003f80 0x7f5c641eaac0 unknown :-1 s=CLOSED pgs=2452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:07.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:08.007+0000 7f5c69c00640 1 -- 172.21.15.28:0/3516625390 >> 172.21.15.28:0/3516625390 conn(0x7f5c640ac370 msgr2=0x7f5c640c50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:14:07.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:08.007+0000 7f5c69c00640 1 -- 172.21.15.28:0/3516625390 shutdown_connections 2024-02-21T23:14:07.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:08.007+0000 7f5c69c00640 1 -- 172.21.15.28:0/3516625390 wait complete. 2024-02-21T23:14:08.369 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-21T23:14:08.378 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-21T23:14:08.418 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 235, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:14:07.624559+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 209, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 8287}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8287': {'gid': 8287, 'name': 'd', 'rank': 1, 'incarnation': 235, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.81:6835/1607587459', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6834', 'nonce': 1607587459}, {'type': 'v1', 'addr': '172.21.15.81:6835', 'nonce': 1607587459}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:14:08.418 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8287:up:replay 2024-02-21T23:14:09.419 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:14:09.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.656+0000 7f88d3400640 1 Processor -- start 2024-02-21T23:14:09.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.656+0000 7f88d3400640 1 -- start start 2024-02-21T23:14:09.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.658+0000 7f88d3400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f88cc0c11f0 0x7f88cc153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:09.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.658+0000 7f88d3400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88cc0c1a30 con 0x7f88cc0c15d0 2024-02-21T23:14:09.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.658+0000 7f88d3400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88cc152e70 con 0x7f88cc15b6d0 2024-02-21T23:14:09.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.658+0000 7f88d3400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88cc0a5ec0 con 0x7f88cc0c11f0 2024-02-21T23:14:09.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.658+0000 7f88cbe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f88cc0c15d0 0x7f88cc06dce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:49960/0 (socket says 172.21.15.28:49960) 2024-02-21T23:14:09.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.658+0000 7f88cbe00640 1 -- 172.21.15.28:0/2283164329 learned_addr learned my addr 172.21.15.28:0/2283164329 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:14:09.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.658+0000 7f88d0a00640 1 --2- 172.21.15.28:0/2283164329 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f88cc0c11f0 0x7f88cc153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:09.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.659+0000 7f88cb400640 1 -- 172.21.15.28:0/2283164329 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1866133051 0 0) 0x7f88cc0c1a30 con 0x7f88cc0c15d0 2024-02-21T23:14:09.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.659+0000 7f88cb400640 1 -- 172.21.15.28:0/2283164329 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88b4003520 con 0x7f88cc0c15d0 2024-02-21T23:14:09.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.659+0000 7f88cb400640 1 -- 172.21.15.28:0/2283164329 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3528312003 0 0) 0x7f88b4003520 con 0x7f88cc0c15d0 2024-02-21T23:14:09.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.659+0000 7f88cb400640 1 -- 172.21.15.28:0/2283164329 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f88cc0c11f0 msgr2=0x7f88cc153950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:14:09.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.659+0000 7f88cb400640 1 --2- 172.21.15.28:0/2283164329 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f88cc0c11f0 0x7f88cc153950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:09.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.659+0000 7f88cb400640 1 -- 172.21.15.28:0/2283164329 >> v1:172.21.15.81:6789/0 conn(0x7f88cc15b6d0 legacy=0x7f88cc15dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:09.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.659+0000 7f88cb400640 1 -- 172.21.15.28:0/2283164329 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88cc153150 con 0x7f88cc0c15d0 2024-02-21T23:14:09.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.659+0000 7f88cb400640 1 -- 172.21.15.28:0/2283164329 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f88b8001670 con 0x7f88cc0c15d0 2024-02-21T23:14:09.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.659+0000 7f88cb400640 1 -- 172.21.15.28:0/2283164329 >> v1:172.21.15.28:6789/0 conn(0x7f88cc0c15d0 legacy=0x7f88cc06dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:09.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.659+0000 7f88cb400640 1 --2- 172.21.15.28:0/2283164329 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f88cc0c11f0 0x7f88b4003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:09.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.659+0000 7f88cb400640 1 --2- 172.21.15.28:0/2283164329 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88b4003fd0 0x7f88b4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:09.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.659+0000 7f88cb400640 1 --2- 172.21.15.28:0/2283164329 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f88b4004d40 0x7f88b40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:09.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.659+0000 7f88cb400640 1 -- 172.21.15.28:0/2283164329 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88cc0a5ec0 con 0x7f88b4003fd0 2024-02-21T23:14:09.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.659+0000 7f88cb400640 1 -- 172.21.15.28:0/2283164329 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88b400bc50 con 0x7f88b4004d40 2024-02-21T23:14:09.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.659+0000 7f88cb400640 1 -- 172.21.15.28:0/2283164329 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88b400be50 con 0x7f88cc0c11f0 2024-02-21T23:14:09.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.660+0000 7f88cbe00640 1 --2- 172.21.15.28:0/2283164329 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88b4003fd0 0x7f88b4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:09.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.660+0000 7f88d0a00640 1 --2- 172.21.15.28:0/2283164329 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f88cc0c11f0 0x7f88b4003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:09.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.660+0000 7f88d1400640 1 --2- 172.21.15.28:0/2283164329 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f88b4004d40 0x7f88b40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:09.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.660+0000 7f88cbe00640 1 -- 172.21.15.28:0/2283164329 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f88cc0c11f0 msgr2=0x7f88b4003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:14:09.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.660+0000 7f88cbe00640 1 --2- 172.21.15.28:0/2283164329 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f88cc0c11f0 0x7f88b4003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:09.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.660+0000 7f88cbe00640 1 -- 172.21.15.28:0/2283164329 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f88b4004d40 msgr2=0x7f88b40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:14:09.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.660+0000 7f88cbe00640 1 --2- 172.21.15.28:0/2283164329 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f88b4004d40 0x7f88b40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:09.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.660+0000 7f88cbe00640 1 -- 172.21.15.28:0/2283164329 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88b4003520 con 0x7f88b4003fd0 2024-02-21T23:14:09.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.661+0000 7f88cbe00640 1 --2- 172.21.15.28:0/2283164329 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88b4003fd0 0x7f88b4004800 secure :-1 s=READY pgs=2454 cs=0 l=1 rev1=1 crypto rx=0x7f88b800c110 tx=0x7f88b4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d303c12feaf65ac server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:09.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.661+0000 7f88cb400640 1 -- 172.21.15.28:0/2283164329 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b8007c40 con 0x7f88b4003fd0 2024-02-21T23:14:09.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.661+0000 7f88cb400640 1 -- 172.21.15.28:0/2283164329 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88b8030070 con 0x7f88b4003fd0 2024-02-21T23:14:09.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.662+0000 7f88cb400640 1 -- 172.21.15.28:0/2283164329 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b8005d70 con 0x7f88b4003fd0 2024-02-21T23:14:09.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.662+0000 7f88d3400640 1 -- 172.21.15.28:0/2283164329 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88b4003fd0 msgr2=0x7f88b4004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:09.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.662+0000 7f88d3400640 1 --2- 172.21.15.28:0/2283164329 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88b4003fd0 0x7f88b4004800 secure :-1 s=READY pgs=2454 cs=0 l=1 rev1=1 crypto rx=0x7f88b800c110 tx=0x7f88b4005830 comp rx=0 tx=0).stop 2024-02-21T23:14:09.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.662+0000 7f88d1400640 1 -- 172.21.15.28:0/2283164329 reap_dead start 2024-02-21T23:14:09.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.662+0000 7f88d3400640 1 -- 172.21.15.28:0/2283164329 shutdown_connections 2024-02-21T23:14:09.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.662+0000 7f88d3400640 1 -- 172.21.15.28:0/2283164329 >> 172.21.15.28:0/2283164329 conn(0x7f88cc0c67f0 msgr2=0x7f88cc0c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:14:09.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.662+0000 7f88d3400640 1 -- 172.21.15.28:0/2283164329 shutdown_connections 2024-02-21T23:14:09.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.662+0000 7f88d3400640 1 -- 172.21.15.28:0/2283164329 wait complete. 2024-02-21T23:14:09.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.664+0000 7f88d3400640 1 Processor -- start 2024-02-21T23:14:09.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.664+0000 7f88d3400640 1 -- start start 2024-02-21T23:14:09.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.665+0000 7f88d3400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f88b4003fd0 0x7f88cc1454b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:09.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.665+0000 7f88d3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88cc143880 0x7f88cc0a63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:09.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.665+0000 7f88d3400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f88cc0ab6c0 0x7f88cc0a68f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:09.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.665+0000 7f88d3400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88cc0c4c00 con 0x7f88cc143880 2024-02-21T23:14:09.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.665+0000 7f88d3400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88cc14d1c0 con 0x7f88b4003fd0 2024-02-21T23:14:09.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.665+0000 7f88d3400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88cc0a6110 con 0x7f88cc0ab6c0 2024-02-21T23:14:09.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.665+0000 7f88cbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88cc143880 0x7f88cc0a63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:09.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.665+0000 7f88cbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88cc143880 0x7f88cc0a63b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39322/0 (socket says 172.21.15.28:39322) 2024-02-21T23:14:09.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.665+0000 7f88cbe00640 1 -- 172.21.15.28:0/1846357773 learned_addr learned my addr 172.21.15.28:0/1846357773 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:14:09.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.666+0000 7f88d0a00640 1 --2- 172.21.15.28:0/1846357773 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f88b4003fd0 0x7f88cc1454b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:09.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.666+0000 7f88d1400640 1 --2- 172.21.15.28:0/1846357773 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f88cc0ab6c0 0x7f88cc0a68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:09.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.666+0000 7f88cbe00640 1 -- 172.21.15.28:0/1846357773 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f88cc0ab6c0 msgr2=0x7f88cc0a68f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:09.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.666+0000 7f88cbe00640 1 --2- 172.21.15.28:0/1846357773 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f88cc0ab6c0 0x7f88cc0a68f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:09.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.666+0000 7f88cbe00640 1 -- 172.21.15.28:0/1846357773 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f88b4003fd0 msgr2=0x7f88cc1454b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:09.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.666+0000 7f88cbe00640 1 --2- 172.21.15.28:0/1846357773 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f88b4003fd0 0x7f88cc1454b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:09.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.666+0000 7f88cbe00640 1 -- 172.21.15.28:0/1846357773 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88cc1435a0 con 0x7f88cc143880 2024-02-21T23:14:09.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.666+0000 7f88d1400640 1 --2- 172.21.15.28:0/1846357773 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f88cc0ab6c0 0x7f88cc0a68f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:14:09.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.666+0000 7f88cbe00640 1 --2- 172.21.15.28:0/1846357773 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88cc143880 0x7f88cc0a63b0 secure :-1 s=READY pgs=2455 cs=0 l=1 rev1=1 crypto rx=0x7f88b800c230 tx=0x7f88b802e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:09.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.667+0000 7f88c8c00640 1 -- 172.21.15.28:0/1846357773 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b8046070 con 0x7f88cc143880 2024-02-21T23:14:09.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.667+0000 7f88c8c00640 1 -- 172.21.15.28:0/1846357773 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88b80049e0 con 0x7f88cc143880 2024-02-21T23:14:09.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.667+0000 7f88d3400640 1 -- 172.21.15.28:0/1846357773 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88cc0abf50 con 0x7f88cc143880 2024-02-21T23:14:09.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.667+0000 7f88c8c00640 1 -- 172.21.15.28:0/1846357773 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b8004db0 con 0x7f88cc143880 2024-02-21T23:14:09.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.667+0000 7f88d3400640 1 -- 172.21.15.28:0/1846357773 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88cc0aa3a0 con 0x7f88cc143880 2024-02-21T23:14:09.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.668+0000 7f88d3400640 1 -- 172.21.15.28:0/1846357773 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88cc0abd40 con 0x7f88cc143880 2024-02-21T23:14:09.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.670+0000 7f88c8c00640 1 -- 172.21.15.28:0/1846357773 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f88b8007070 con 0x7f88cc143880 2024-02-21T23:14:09.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.671+0000 7f88c8c00640 1 --2- 172.21.15.28:0/1846357773 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f88ac06b500 0x7f88ac06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:09.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.671+0000 7f88c8c00640 1 -- 172.21.15.28:0/1846357773 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f88b80ab790 con 0x7f88cc143880 2024-02-21T23:14:09.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.672+0000 7f88d0a00640 1 --2- 172.21.15.28:0/1846357773 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f88ac06b500 0x7f88ac06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:09.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.673+0000 7f88d0a00640 1 --2- 172.21.15.28:0/1846357773 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f88ac06b500 0x7f88ac06d9c0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f88c0009a10 tx=0x7f88c0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:09.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.676+0000 7f88c8c00640 1 -- 172.21.15.28:0/1846357773 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f88b8079de0 con 0x7f88cc143880 2024-02-21T23:14:09.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.921+0000 7f88d3400640 1 -- 172.21.15.28:0/1846357773 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f88cc067f30 con 0x7f88cc143880 2024-02-21T23:14:09.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.921+0000 7f88c8c00640 1 -- 172.21.15.28:0/1846357773 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 237 v237) v1 ==== 95+0+4176 (secure 0 0 0) 0x7f88b807de80 con 0x7f88cc143880 2024-02-21T23:14:09.910 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:14:09.910 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":237,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7265,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":224}],"filesystems":[{"mdsmap":{"epoch":237,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:14:08.638941+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":209,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":8287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8287":{"gid":8287,"name":"d","rank":1,"incarnation":235,"state":"up:reconnect","state_seq":13,"addr":"172.21.15.81:6835/1607587459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1607587459},{"type":"v1","addr":"172.21.15.81:6835","nonce":1607587459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:14:09.910 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 237 2024-02-21T23:14:09.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.924+0000 7f88d3400640 1 -- 172.21.15.28:0/1846357773 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f88ac06b500 msgr2=0x7f88ac06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:09.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.924+0000 7f88d3400640 1 --2- 172.21.15.28:0/1846357773 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f88ac06b500 0x7f88ac06d9c0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f88c0009a10 tx=0x7f88c0008040 comp rx=0 tx=0).stop 2024-02-21T23:14:09.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.924+0000 7f88d3400640 1 -- 172.21.15.28:0/1846357773 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88cc143880 msgr2=0x7f88cc0a63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:09.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.924+0000 7f88d3400640 1 --2- 172.21.15.28:0/1846357773 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88cc143880 0x7f88cc0a63b0 secure :-1 s=READY pgs=2455 cs=0 l=1 rev1=1 crypto rx=0x7f88b800c230 tx=0x7f88b802e070 comp rx=0 tx=0).stop 2024-02-21T23:14:09.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.924+0000 7f88d3400640 1 -- 172.21.15.28:0/1846357773 shutdown_connections 2024-02-21T23:14:09.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.924+0000 7f88d3400640 1 --2- 172.21.15.28:0/1846357773 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f88ac06b500 0x7f88ac06d9c0 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:09.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.924+0000 7f88d3400640 1 --2- 172.21.15.28:0/1846357773 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f88cc0ab6c0 0x7f88cc0a68f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:09.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.924+0000 7f88d3400640 1 --2- 172.21.15.28:0/1846357773 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88cc143880 0x7f88cc0a63b0 unknown :-1 s=CLOSED pgs=2455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:09.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.924+0000 7f88d3400640 1 --2- 172.21.15.28:0/1846357773 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f88b4003fd0 0x7f88cc1454b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:09.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.924+0000 7f88d3400640 1 -- 172.21.15.28:0/1846357773 >> 172.21.15.28:0/1846357773 conn(0x7f88cc0c67f0 msgr2=0x7f88cc0c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:14:09.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.925+0000 7f88d3400640 1 -- 172.21.15.28:0/1846357773 shutdown_connections 2024-02-21T23:14:09.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:09.925+0000 7f88d3400640 1 -- 172.21.15.28:0/1846357773 wait complete. 2024-02-21T23:14:10.337 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 237, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:14:08.638941+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 209, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 8287}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8287': {'gid': 8287, 'name': 'd', 'rank': 1, 'incarnation': 235, 'state': 'up:reconnect', 'state_seq': 13, 'addr': '172.21.15.81:6835/1607587459', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6834', 'nonce': 1607587459}, {'type': 'v1', 'addr': '172.21.15.81:6835', 'nonce': 1607587459}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:14:10.337 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8287:up:reconnect 2024-02-21T23:14:11.338 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:14:11.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.584+0000 7f99fca00640 1 Processor -- start 2024-02-21T23:14:11.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.584+0000 7f99fca00640 1 -- start start 2024-02-21T23:14:11.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.586+0000 7f99fca00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f99f8153950 0x7f99f80c15d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:11.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.586+0000 7f99fca00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99f8152e70 con 0x7f99f80c11f0 2024-02-21T23:14:11.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.586+0000 7f99fca00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99f8153150 con 0x7f99f815b6d0 2024-02-21T23:14:11.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.586+0000 7f99fca00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99f80a5ec0 con 0x7f99f8153950 2024-02-21T23:14:11.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.586+0000 7f99f5e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f99f80c11f0 0x7f99f806dce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:49976/0 (socket says 172.21.15.28:49976) 2024-02-21T23:14:11.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.586+0000 7f99f5e00640 1 -- 172.21.15.28:0/4144775737 learned_addr learned my addr 172.21.15.28:0/4144775737 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:14:11.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.587+0000 7f99efe00640 1 --2- 172.21.15.28:0/4144775737 >> v2:172.21.15.81:3300/0 conn(0x7f99f8153950 0x7f99f80c15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:11.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.587+0000 7f99f5400640 1 -- 172.21.15.28:0/4144775737 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2145307414 0 0) 0x7f99f8152e70 con 0x7f99f80c11f0 2024-02-21T23:14:11.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.587+0000 7f99f5400640 1 -- 172.21.15.28:0/4144775737 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99dc003520 con 0x7f99f80c11f0 2024-02-21T23:14:11.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.587+0000 7f99f5400640 1 -- 172.21.15.28:0/4144775737 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2869419023 0 0) 0x7f99dc003520 con 0x7f99f80c11f0 2024-02-21T23:14:11.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.587+0000 7f99efe00640 1 -- 172.21.15.28:0/4144775737 >> v1:172.21.15.81:6789/0 conn(0x7f99f815b6d0 legacy=0x7f99f815dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:11.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.587+0000 7f99efe00640 1 -- 172.21.15.28:0/4144775737 >> v1:172.21.15.28:6789/0 conn(0x7f99f80c11f0 legacy=0x7f99f806dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:11.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.587+0000 7f99efe00640 1 -- 172.21.15.28:0/4144775737 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99f8153430 con 0x7f99f8153950 2024-02-21T23:14:11.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.589+0000 7f99efe00640 1 --2- 172.21.15.28:0/4144775737 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f99f8153950 0x7f99f80c15d0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f99e00097a0 tx=0x7f99e002b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c0ed1790179ad602 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:11.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.589+0000 7f99f5400640 1 -- 172.21.15.28:0/4144775737 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e0037070 con 0x7f99f8153950 2024-02-21T23:14:11.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.589+0000 7f99f5400640 1 -- 172.21.15.28:0/4144775737 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99e0030510 con 0x7f99f8153950 2024-02-21T23:14:11.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.589+0000 7f99f5400640 1 -- 172.21.15.28:0/4144775737 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e8001670 con 0x7f99f8153950 2024-02-21T23:14:11.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.590+0000 7f99fca00640 1 -- 172.21.15.28:0/4144775737 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f99f8153950 msgr2=0x7f99f80c15d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:11.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.590+0000 7f99fca00640 1 --2- 172.21.15.28:0/4144775737 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f99f8153950 0x7f99f80c15d0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f99e00097a0 tx=0x7f99e002b790 comp rx=0 tx=0).stop 2024-02-21T23:14:11.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.590+0000 7f99fca00640 1 -- 172.21.15.28:0/4144775737 shutdown_connections 2024-02-21T23:14:11.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.590+0000 7f99fca00640 1 --2- 172.21.15.28:0/4144775737 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f99f8153950 0x7f99f80c15d0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:11.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.590+0000 7f99fca00640 1 -- 172.21.15.28:0/4144775737 >> 172.21.15.28:0/4144775737 conn(0x7f99f80c67f0 msgr2=0x7f99f80c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:14:11.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.590+0000 7f99fca00640 1 -- 172.21.15.28:0/4144775737 shutdown_connections 2024-02-21T23:14:11.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.590+0000 7f99fca00640 1 -- 172.21.15.28:0/4144775737 wait complete. 2024-02-21T23:14:11.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.593+0000 7f99fca00640 1 Processor -- start 2024-02-21T23:14:11.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.593+0000 7f99fca00640 1 -- start start 2024-02-21T23:14:11.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.594+0000 7f99fca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99f80c11f0 0x7f99f81446a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:11.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.595+0000 7f99fca00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f99f8153950 0x7f99f80a68e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:11.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.595+0000 7f99fca00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f99f815b6d0 0x7f99f80a6e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:11.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.595+0000 7f99fca00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99f80a6110 con 0x7f99f80c11f0 2024-02-21T23:14:11.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.595+0000 7f99fca00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99f80c4c00 con 0x7f99f815b6d0 2024-02-21T23:14:11.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.595+0000 7f99fca00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99f80c0d90 con 0x7f99f8153950 2024-02-21T23:14:11.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.595+0000 7f99f5e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99f80c11f0 0x7f99f81446a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:11.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.595+0000 7f99f5e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99f80c11f0 0x7f99f81446a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39326/0 (socket says 172.21.15.28:39326) 2024-02-21T23:14:11.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.595+0000 7f99f5e00640 1 -- 172.21.15.28:0/1869132742 learned_addr learned my addr 172.21.15.28:0/1869132742 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:14:11.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.595+0000 7f99efe00640 1 --2- 172.21.15.28:0/1869132742 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f99f8153950 0x7f99f80a68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:11.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.595+0000 7f99f6800640 1 --2- 172.21.15.28:0/1869132742 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f99f815b6d0 0x7f99f80a6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:11.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.595+0000 7f99f5e00640 1 -- 172.21.15.28:0/1869132742 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f99f8153950 msgr2=0x7f99f80a68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:11.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.595+0000 7f99f5e00640 1 --2- 172.21.15.28:0/1869132742 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f99f8153950 0x7f99f80a68e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:11.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.595+0000 7f99f5e00640 1 -- 172.21.15.28:0/1869132742 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f99f815b6d0 msgr2=0x7f99f80a6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:11.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.595+0000 7f99f5e00640 1 --2- 172.21.15.28:0/1869132742 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f99f815b6d0 0x7f99f80a6e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:11.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.595+0000 7f99f5e00640 1 -- 172.21.15.28:0/1869132742 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99f81e9510 con 0x7f99f80c11f0 2024-02-21T23:14:11.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.595+0000 7f99efe00640 1 --2- 172.21.15.28:0/1869132742 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f99f8153950 0x7f99f80a68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:14:11.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.595+0000 7f99f6800640 1 --2- 172.21.15.28:0/1869132742 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f99f815b6d0 0x7f99f80a6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:14:11.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.596+0000 7f99f5e00640 1 --2- 172.21.15.28:0/1869132742 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99f80c11f0 0x7f99f81446a0 secure :-1 s=READY pgs=2457 cs=0 l=1 rev1=1 crypto rx=0x7f99e800c520 tx=0x7f99e800c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:11.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.596+0000 7f99ee000640 1 -- 172.21.15.28:0/1869132742 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e8017070 con 0x7f99f80c11f0 2024-02-21T23:14:11.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.596+0000 7f99ee000640 1 -- 172.21.15.28:0/1869132742 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99e8005d00 con 0x7f99f80c11f0 2024-02-21T23:14:11.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.596+0000 7f99ee000640 1 -- 172.21.15.28:0/1869132742 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e80072b0 con 0x7f99f80c11f0 2024-02-21T23:14:11.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.596+0000 7f99fca00640 1 -- 172.21.15.28:0/1869132742 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99f8144be0 con 0x7f99f80c11f0 2024-02-21T23:14:11.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.596+0000 7f99fca00640 1 -- 172.21.15.28:0/1869132742 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99f80a81d0 con 0x7f99f80c11f0 2024-02-21T23:14:11.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.598+0000 7f99fca00640 1 -- 172.21.15.28:0/1869132742 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99f8067f30 con 0x7f99f80c11f0 2024-02-21T23:14:11.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.598+0000 7f99ee000640 1 -- 172.21.15.28:0/1869132742 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f99e801a070 con 0x7f99f80c11f0 2024-02-21T23:14:11.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.600+0000 7f99ee000640 1 --2- 172.21.15.28:0/1869132742 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f99d406b500 0x7f99d406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:11.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.600+0000 7f99ee000640 1 -- 172.21.15.28:0/1869132742 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f99e80898b0 con 0x7f99f80c11f0 2024-02-21T23:14:11.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.601+0000 7f99efe00640 1 --2- 172.21.15.28:0/1869132742 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f99d406b500 0x7f99d406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:11.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.602+0000 7f99efe00640 1 --2- 172.21.15.28:0/1869132742 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f99d406b500 0x7f99d406d9c0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f99e0034000 tx=0x7f99e002f000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:11.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.604+0000 7f99ee000640 1 -- 172.21.15.28:0/1869132742 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f99f81e9510 con 0x7f99f80c11f0 2024-02-21T23:14:11.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.848+0000 7f99fca00640 1 -- 172.21.15.28:0/1869132742 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f99f80c1a90 con 0x7f99f80c11f0 2024-02-21T23:14:11.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.849+0000 7f99ee000640 1 -- 172.21.15.28:0/1869132742 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 240 v240) v1 ==== 95+0+4175 (secure 0 0 0) 0x7f99e8058540 con 0x7f99f80c11f0 2024-02-21T23:14:11.838 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:14:11.838 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":240,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7265,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":224}],"filesystems":[{"mdsmap":{"epoch":240,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:14:11.669846+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":209,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":8287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8287":{"gid":8287,"name":"d","rank":1,"incarnation":235,"state":"up:stopping","state_seq":15,"addr":"172.21.15.81:6835/1607587459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1607587459},{"type":"v1","addr":"172.21.15.81:6835","nonce":1607587459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:14:11.838 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 240 2024-02-21T23:14:11.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.852+0000 7f99fca00640 1 -- 172.21.15.28:0/1869132742 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f99d406b500 msgr2=0x7f99d406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:11.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.852+0000 7f99fca00640 1 --2- 172.21.15.28:0/1869132742 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f99d406b500 0x7f99d406d9c0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f99e0034000 tx=0x7f99e002f000 comp rx=0 tx=0).stop 2024-02-21T23:14:11.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.852+0000 7f99fca00640 1 -- 172.21.15.28:0/1869132742 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99f80c11f0 msgr2=0x7f99f81446a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:11.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.852+0000 7f99fca00640 1 --2- 172.21.15.28:0/1869132742 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99f80c11f0 0x7f99f81446a0 secure :-1 s=READY pgs=2457 cs=0 l=1 rev1=1 crypto rx=0x7f99e800c520 tx=0x7f99e800c9e0 comp rx=0 tx=0).stop 2024-02-21T23:14:11.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.852+0000 7f99fca00640 1 -- 172.21.15.28:0/1869132742 shutdown_connections 2024-02-21T23:14:11.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.852+0000 7f99fca00640 1 --2- 172.21.15.28:0/1869132742 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f99d406b500 0x7f99d406d9c0 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:11.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.852+0000 7f99fca00640 1 --2- 172.21.15.28:0/1869132742 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f99f8153950 0x7f99f80a68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:11.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.852+0000 7f99fca00640 1 --2- 172.21.15.28:0/1869132742 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f99f815b6d0 0x7f99f80a6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:11.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.852+0000 7f99fca00640 1 --2- 172.21.15.28:0/1869132742 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99f80c11f0 0x7f99f81446a0 unknown :-1 s=CLOSED pgs=2457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:11.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.852+0000 7f99fca00640 1 -- 172.21.15.28:0/1869132742 >> 172.21.15.28:0/1869132742 conn(0x7f99f80c67f0 msgr2=0x7f99f806e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:14:11.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.852+0000 7f99fca00640 1 -- 172.21.15.28:0/1869132742 shutdown_connections 2024-02-21T23:14:11.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:11.852+0000 7f99fca00640 1 -- 172.21.15.28:0/1869132742 wait complete. 2024-02-21T23:14:12.264 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 240, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:14:11.669846+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 209, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 8287}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8287': {'gid': 8287, 'name': 'd', 'rank': 1, 'incarnation': 235, 'state': 'up:stopping', 'state_seq': 15, 'addr': '172.21.15.81:6835/1607587459', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6834', 'nonce': 1607587459}, {'type': 'v1', 'addr': '172.21.15.81:6835', 'nonce': 1607587459}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:14:12.264 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8287:up:stopping 2024-02-21T23:14:13.264 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:14:13.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.499+0000 7fea79200640 1 Processor -- start 2024-02-21T23:14:13.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.500+0000 7fea79200640 1 -- start start 2024-02-21T23:14:13.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.501+0000 7fea79200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fea74151cd0 0x7fea740c4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:13.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.501+0000 7fea79200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fea74155980 0x7fea74155e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:13.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.501+0000 7fea79200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea740c36b0 con 0x7fea7414f240 2024-02-21T23:14:13.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.501+0000 7fea79200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea740c50e0 con 0x7fea74151cd0 2024-02-21T23:14:13.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.501+0000 7fea79200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea740c3990 con 0x7fea74155980 2024-02-21T23:14:13.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.502+0000 7fea73400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fea7414f240 0x7fea740c7600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:49978/0 (socket says 172.21.15.28:49978) 2024-02-21T23:14:13.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.502+0000 7fea73400640 1 -- 172.21.15.28:0/3592388124 learned_addr learned my addr 172.21.15.28:0/3592388124 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:14:13.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.502+0000 7fea72a00640 1 --2- 172.21.15.28:0/3592388124 >> v2:172.21.15.28:3300/0 conn(0x7fea74151cd0 0x7fea740c4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:13.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.502+0000 7fea73e00640 1 --2- 172.21.15.28:0/3592388124 >> v2:172.21.15.81:3300/0 conn(0x7fea74155980 0x7fea74155e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:13.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.502+0000 7fea72000640 1 -- 172.21.15.28:0/3592388124 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1198038517 0 0) 0x7fea740c36b0 con 0x7fea7414f240 2024-02-21T23:14:13.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.503+0000 7fea72000640 1 -- 172.21.15.28:0/3592388124 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea58003520 con 0x7fea7414f240 2024-02-21T23:14:13.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.503+0000 7fea72a00640 1 -- 172.21.15.28:0/3592388124 >> v2:172.21.15.81:3300/0 conn(0x7fea74155980 msgr2=0x7fea74155e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:14:13.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.503+0000 7fea72a00640 1 --2- 172.21.15.28:0/3592388124 >> v2:172.21.15.81:3300/0 conn(0x7fea74155980 0x7fea74155e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:13.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.503+0000 7fea72a00640 1 -- 172.21.15.28:0/3592388124 >> v1:172.21.15.28:6789/0 conn(0x7fea7414f240 legacy=0x7fea740c7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:13.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.503+0000 7fea72a00640 1 -- 172.21.15.28:0/3592388124 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea740c3c70 con 0x7fea74151cd0 2024-02-21T23:14:13.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.504+0000 7fea72a00640 1 --2- 172.21.15.28:0/3592388124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea74151cd0 0x7fea740c4ba0 secure :-1 s=READY pgs=2459 cs=0 l=1 rev1=1 crypto rx=0x7fea74072160 tx=0x7fea6002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28fc633dffbe1743 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:13.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.504+0000 7fea72000640 1 -- 172.21.15.28:0/3592388124 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea600315a0 con 0x7fea74151cd0 2024-02-21T23:14:13.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.504+0000 7fea72000640 1 -- 172.21.15.28:0/3592388124 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea60037070 con 0x7fea74151cd0 2024-02-21T23:14:13.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.504+0000 7fea72000640 1 -- 172.21.15.28:0/3592388124 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea60031960 con 0x7fea74151cd0 2024-02-21T23:14:13.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.505+0000 7fea79200640 1 -- 172.21.15.28:0/3592388124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea74151cd0 msgr2=0x7fea740c4ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:13.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.505+0000 7fea79200640 1 --2- 172.21.15.28:0/3592388124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea74151cd0 0x7fea740c4ba0 secure :-1 s=READY pgs=2459 cs=0 l=1 rev1=1 crypto rx=0x7fea74072160 tx=0x7fea6002b840 comp rx=0 tx=0).stop 2024-02-21T23:14:13.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.505+0000 7fea79200640 1 -- 172.21.15.28:0/3592388124 shutdown_connections 2024-02-21T23:14:13.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.505+0000 7fea79200640 1 --2- 172.21.15.28:0/3592388124 >> v2:172.21.15.81:3300/0 conn(0x7fea74155980 0x7fea74155e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:13.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.505+0000 7fea79200640 1 --2- 172.21.15.28:0/3592388124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea74151cd0 0x7fea740c4ba0 unknown :-1 s=CLOSED pgs=2459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:13.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.505+0000 7fea79200640 1 -- 172.21.15.28:0/3592388124 >> 172.21.15.28:0/3592388124 conn(0x7fea74067050 msgr2=0x7fea740a84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:14:13.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.505+0000 7fea79200640 1 -- 172.21.15.28:0/3592388124 shutdown_connections 2024-02-21T23:14:13.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.505+0000 7fea79200640 1 -- 172.21.15.28:0/3592388124 wait complete. 2024-02-21T23:14:13.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.507+0000 7fea79200640 1 Processor -- start 2024-02-21T23:14:13.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.507+0000 7fea79200640 1 -- start start 2024-02-21T23:14:13.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.508+0000 7fea79200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea7414f240 0x7fea740c0d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:13.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.508+0000 7fea79200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fea74151cd0 0x7fea741e73f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:13.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.508+0000 7fea79200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fea74155980 0x7fea741e7930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:13.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.508+0000 7fea79200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea7406f2e0 con 0x7fea7414f240 2024-02-21T23:14:13.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.508+0000 7fea79200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea7414f620 con 0x7fea74151cd0 2024-02-21T23:14:13.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.508+0000 7fea79200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea7406f460 con 0x7fea74155980 2024-02-21T23:14:13.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.508+0000 7fea73400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea7414f240 0x7fea740c0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:13.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.508+0000 7fea73400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea7414f240 0x7fea740c0d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39350/0 (socket says 172.21.15.28:39350) 2024-02-21T23:14:13.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.508+0000 7fea73400640 1 -- 172.21.15.28:0/1326747637 learned_addr learned my addr 172.21.15.28:0/1326747637 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:14:13.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.509+0000 7fea72a00640 1 --2- 172.21.15.28:0/1326747637 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fea74151cd0 0x7fea741e73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:13.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.509+0000 7fea73e00640 1 --2- 172.21.15.28:0/1326747637 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fea74155980 0x7fea741e7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:13.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.509+0000 7fea73400640 1 -- 172.21.15.28:0/1326747637 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fea74155980 msgr2=0x7fea741e7930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:13.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.509+0000 7fea73400640 1 --2- 172.21.15.28:0/1326747637 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fea74155980 0x7fea741e7930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:13.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.509+0000 7fea73400640 1 -- 172.21.15.28:0/1326747637 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fea74151cd0 msgr2=0x7fea741e73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:13.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.509+0000 7fea73400640 1 --2- 172.21.15.28:0/1326747637 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fea74151cd0 0x7fea741e73f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:13.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.509+0000 7fea73400640 1 -- 172.21.15.28:0/1326747637 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea740acb90 con 0x7fea7414f240 2024-02-21T23:14:13.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.509+0000 7fea72a00640 1 --2- 172.21.15.28:0/1326747637 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fea74151cd0 0x7fea741e73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:14:13.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.509+0000 7fea73400640 1 --2- 172.21.15.28:0/1326747637 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea7414f240 0x7fea740c0d30 secure :-1 s=READY pgs=2460 cs=0 l=1 rev1=1 crypto rx=0x7fea6400c460 tx=0x7fea6400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:13.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.509+0000 7fea6b400640 1 -- 172.21.15.28:0/1326747637 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea64017070 con 0x7fea7414f240 2024-02-21T23:14:13.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.510+0000 7fea6b400640 1 -- 172.21.15.28:0/1326747637 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea64005ce0 con 0x7fea7414f240 2024-02-21T23:14:13.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.510+0000 7fea6b400640 1 -- 172.21.15.28:0/1326747637 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea64007290 con 0x7fea7414f240 2024-02-21T23:14:13.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.510+0000 7fea79200640 1 -- 172.21.15.28:0/1326747637 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea741eae70 con 0x7fea7414f240 2024-02-21T23:14:13.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.510+0000 7fea79200640 1 -- 172.21.15.28:0/1326747637 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea740c1270 con 0x7fea7414f240 2024-02-21T23:14:13.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.511+0000 7fea79200640 1 -- 172.21.15.28:0/1326747637 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea7404f290 con 0x7fea7414f240 2024-02-21T23:14:13.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.512+0000 7fea6b400640 1 -- 172.21.15.28:0/1326747637 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fea6401a070 con 0x7fea7414f240 2024-02-21T23:14:13.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.513+0000 7fea6b400640 1 --2- 172.21.15.28:0/1326747637 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fea5406b4d0 0x7fea5406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:13.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.513+0000 7fea6b400640 1 -- 172.21.15.28:0/1326747637 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fea640897e0 con 0x7fea7414f240 2024-02-21T23:14:13.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.514+0000 7fea72a00640 1 --2- 172.21.15.28:0/1326747637 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fea5406b4d0 0x7fea5406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:13.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.515+0000 7fea72a00640 1 --2- 172.21.15.28:0/1326747637 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fea5406b4d0 0x7fea5406d990 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fea60036000 tx=0x7fea60008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:13.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.518+0000 7fea6b400640 1 -- 172.21.15.28:0/1326747637 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fea740acb90 con 0x7fea7414f240 2024-02-21T23:14:13.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.781+0000 7fea79200640 1 -- 172.21.15.28:0/1326747637 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fea740c4830 con 0x7fea7414f240 2024-02-21T23:14:13.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.782+0000 7fea6b400640 1 -- 172.21.15.28:0/1326747637 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 240 v240) v1 ==== 95+0+4175 (secure 0 0 0) 0x7fea64058470 con 0x7fea7414f240 2024-02-21T23:14:13.771 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:14:13.771 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":240,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7265,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":224}],"filesystems":[{"mdsmap":{"epoch":240,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:14:11.669846+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":209,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":8287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8287":{"gid":8287,"name":"d","rank":1,"incarnation":235,"state":"up:stopping","state_seq":15,"addr":"172.21.15.81:6835/1607587459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1607587459},{"type":"v1","addr":"172.21.15.81:6835","nonce":1607587459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:14:13.771 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 240 2024-02-21T23:14:13.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.785+0000 7fea79200640 1 -- 172.21.15.28:0/1326747637 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fea5406b4d0 msgr2=0x7fea5406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:13.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.785+0000 7fea79200640 1 --2- 172.21.15.28:0/1326747637 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fea5406b4d0 0x7fea5406d990 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fea60036000 tx=0x7fea60008030 comp rx=0 tx=0).stop 2024-02-21T23:14:13.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.785+0000 7fea79200640 1 -- 172.21.15.28:0/1326747637 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea7414f240 msgr2=0x7fea740c0d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:13.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.785+0000 7fea79200640 1 --2- 172.21.15.28:0/1326747637 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea7414f240 0x7fea740c0d30 secure :-1 s=READY pgs=2460 cs=0 l=1 rev1=1 crypto rx=0x7fea6400c460 tx=0x7fea6400c920 comp rx=0 tx=0).stop 2024-02-21T23:14:13.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.785+0000 7fea79200640 1 -- 172.21.15.28:0/1326747637 shutdown_connections 2024-02-21T23:14:13.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.785+0000 7fea79200640 1 --2- 172.21.15.28:0/1326747637 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fea5406b4d0 0x7fea5406d990 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:13.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.785+0000 7fea79200640 1 --2- 172.21.15.28:0/1326747637 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fea74155980 0x7fea741e7930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:13.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.785+0000 7fea79200640 1 --2- 172.21.15.28:0/1326747637 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fea74151cd0 0x7fea741e73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:13.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.785+0000 7fea79200640 1 --2- 172.21.15.28:0/1326747637 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea7414f240 0x7fea740c0d30 unknown :-1 s=CLOSED pgs=2460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:13.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.785+0000 7fea79200640 1 -- 172.21.15.28:0/1326747637 >> 172.21.15.28:0/1326747637 conn(0x7fea74067050 msgr2=0x7fea740a84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:14:13.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.785+0000 7fea79200640 1 -- 172.21.15.28:0/1326747637 shutdown_connections 2024-02-21T23:14:13.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:13.786+0000 7fea79200640 1 -- 172.21.15.28:0/1326747637 wait complete. 2024-02-21T23:14:14.205 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 240, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:14:11.669846+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 209, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 8287}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8287': {'gid': 8287, 'name': 'd', 'rank': 1, 'incarnation': 235, 'state': 'up:stopping', 'state_seq': 15, 'addr': '172.21.15.81:6835/1607587459', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6834', 'nonce': 1607587459}, {'type': 'v1', 'addr': '172.21.15.81:6835', 'nonce': 1607587459}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:14:14.206 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8287:up:stopping 2024-02-21T23:14:15.207 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:14:15.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.438+0000 7f8607200640 1 Processor -- start 2024-02-21T23:14:15.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.438+0000 7f8607200640 1 -- start start 2024-02-21T23:14:15.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.440+0000 7f8607200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f86080c64d0 0x7f86080c8940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:15.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.440+0000 7f8607200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86080c2c20 con 0x7f86080c8560 2024-02-21T23:14:15.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.440+0000 7f8607200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8608155100 con 0x7f86080c7ef0 2024-02-21T23:14:15.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.440+0000 7f8607200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86081553e0 con 0x7f86080c64d0 2024-02-21T23:14:15.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.440+0000 7f8605e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f86080c8560 0x7f86080c4d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41176/0 (socket says 172.21.15.28:41176) 2024-02-21T23:14:15.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.440+0000 7f8605e00640 1 -- 172.21.15.28:0/2574014447 learned_addr learned my addr 172.21.15.28:0/2574014447 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:14:15.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.440+0000 7f8606800640 1 --2- 172.21.15.28:0/2574014447 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f86080c64d0 0x7f86080c8940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:15.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.440+0000 7f8604a00640 1 -- 172.21.15.28:0/2574014447 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3927837143 0 0) 0x7f86080c2c20 con 0x7f86080c8560 2024-02-21T23:14:15.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.440+0000 7f8604a00640 1 -- 172.21.15.28:0/2574014447 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85ec003520 con 0x7f86080c8560 2024-02-21T23:14:15.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.441+0000 7f8604a00640 1 -- 172.21.15.28:0/2574014447 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3778183007 0 0) 0x7f85ec003520 con 0x7f86080c8560 2024-02-21T23:14:15.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.441+0000 7f8604a00640 1 -- 172.21.15.28:0/2574014447 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f86080c64d0 msgr2=0x7f86080c8940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:14:15.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.441+0000 7f8604a00640 1 --2- 172.21.15.28:0/2574014447 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f86080c64d0 0x7f86080c8940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:15.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.441+0000 7f8604a00640 1 -- 172.21.15.28:0/2574014447 >> v1:172.21.15.81:6789/0 conn(0x7f86080c7ef0 legacy=0x7f86080c54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:15.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.441+0000 7f8604a00640 1 -- 172.21.15.28:0/2574014447 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86081556c0 con 0x7f86080c8560 2024-02-21T23:14:15.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.441+0000 7f8604a00640 1 -- 172.21.15.28:0/2574014447 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f85f8001670 con 0x7f86080c8560 2024-02-21T23:14:15.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.441+0000 7f8604a00640 1 -- 172.21.15.28:0/2574014447 >> v1:172.21.15.28:6789/0 conn(0x7f86080c8560 legacy=0x7f86080c4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:15.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.441+0000 7f8604a00640 1 --2- 172.21.15.28:0/2574014447 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f86080c64d0 0x7f85ec0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:15.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.441+0000 7f8604a00640 1 --2- 172.21.15.28:0/2574014447 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f85ec003f60 0x7f85ec004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:15.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.441+0000 7f8604a00640 1 --2- 172.21.15.28:0/2574014447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85ec004d40 0x7f85ec0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:15.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.441+0000 7f8604a00640 1 -- 172.21.15.28:0/2574014447 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86081553e0 con 0x7f85ec004d40 2024-02-21T23:14:15.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.441+0000 7f8604a00640 1 -- 172.21.15.28:0/2574014447 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85ec00bb50 con 0x7f85ec003f60 2024-02-21T23:14:15.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.441+0000 7f8604a00640 1 -- 172.21.15.28:0/2574014447 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85ec00bd50 con 0x7f86080c64d0 2024-02-21T23:14:15.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.441+0000 7f8606800640 1 --2- 172.21.15.28:0/2574014447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85ec004d40 0x7f85ec0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:15.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.442+0000 7f8605e00640 1 --2- 172.21.15.28:0/2574014447 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f86080c64d0 0x7f85ec0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:15.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.442+0000 7f8605400640 1 --2- 172.21.15.28:0/2574014447 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f85ec003f60 0x7f85ec004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:15.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.442+0000 7f8606800640 1 -- 172.21.15.28:0/2574014447 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f86080c64d0 msgr2=0x7f85ec0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:14:15.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.442+0000 7f8606800640 1 --2- 172.21.15.28:0/2574014447 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f86080c64d0 0x7f85ec0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:15.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.442+0000 7f8606800640 1 -- 172.21.15.28:0/2574014447 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f85ec003f60 msgr2=0x7f85ec004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:14:15.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.442+0000 7f8606800640 1 --2- 172.21.15.28:0/2574014447 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f85ec003f60 0x7f85ec004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:15.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.442+0000 7f8606800640 1 -- 172.21.15.28:0/2574014447 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85ec003520 con 0x7f85ec004d40 2024-02-21T23:14:15.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.444+0000 7f8606800640 1 --2- 172.21.15.28:0/2574014447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85ec004d40 0x7f85ec0051c0 secure :-1 s=READY pgs=2462 cs=0 l=1 rev1=1 crypto rx=0x7f8600009fc0 tx=0x7f85ec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15baffa378dee7b4 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:15.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.444+0000 7f8604a00640 1 -- 172.21.15.28:0/2574014447 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8600039070 con 0x7f85ec004d40 2024-02-21T23:14:15.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.444+0000 7f8604a00640 1 -- 172.21.15.28:0/2574014447 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f860002cc20 con 0x7f85ec004d40 2024-02-21T23:14:15.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.444+0000 7f8604a00640 1 -- 172.21.15.28:0/2574014447 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8600009c10 con 0x7f85ec004d40 2024-02-21T23:14:15.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.444+0000 7f8607200640 1 -- 172.21.15.28:0/2574014447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85ec004d40 msgr2=0x7f85ec0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:15.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.444+0000 7f8607200640 1 --2- 172.21.15.28:0/2574014447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85ec004d40 0x7f85ec0051c0 secure :-1 s=READY pgs=2462 cs=0 l=1 rev1=1 crypto rx=0x7f8600009fc0 tx=0x7f85ec005830 comp rx=0 tx=0).stop 2024-02-21T23:14:15.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.444+0000 7f8606800640 1 -- 172.21.15.28:0/2574014447 reap_dead start 2024-02-21T23:14:15.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.445+0000 7f8607200640 1 -- 172.21.15.28:0/2574014447 shutdown_connections 2024-02-21T23:14:15.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.445+0000 7f8607200640 1 -- 172.21.15.28:0/2574014447 >> 172.21.15.28:0/2574014447 conn(0x7f860806ae70 msgr2=0x7f860806b280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:14:15.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.445+0000 7f8607200640 1 -- 172.21.15.28:0/2574014447 shutdown_connections 2024-02-21T23:14:15.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.445+0000 7f8607200640 1 -- 172.21.15.28:0/2574014447 wait complete. 2024-02-21T23:14:15.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.446+0000 7f8607200640 1 Processor -- start 2024-02-21T23:14:15.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.447+0000 7f8607200640 1 -- start start 2024-02-21T23:14:15.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.447+0000 7f8607200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f85ec004d40 0x7f86080a8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:15.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.447+0000 7f8607200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f86080a9a50 0x7f86081edd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:15.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.447+0000 7f8607200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86080a8fa0 0x7f86081ee290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:15.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.447+0000 7f8607200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8608071a90 con 0x7f86080a8fa0 2024-02-21T23:14:15.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.447+0000 7f8607200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8608154db0 con 0x7f85ec004d40 2024-02-21T23:14:15.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.447+0000 7f8607200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8608071c10 con 0x7f86080a9a50 2024-02-21T23:14:15.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.448+0000 7f8606800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86080a8fa0 0x7f86081ee290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:15.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.448+0000 7f8606800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86080a8fa0 0x7f86081ee290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34220/0 (socket says 172.21.15.28:34220) 2024-02-21T23:14:15.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.448+0000 7f8606800640 1 -- 172.21.15.28:0/2777317659 learned_addr learned my addr 172.21.15.28:0/2777317659 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:14:15.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.448+0000 7f8605400640 1 --2- 172.21.15.28:0/2777317659 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f86080a9a50 0x7f86081edd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:15.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.448+0000 7f8605e00640 1 --2- 172.21.15.28:0/2777317659 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f85ec004d40 0x7f86080a8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:15.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.448+0000 7f8606800640 1 -- 172.21.15.28:0/2777317659 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f86080a9a50 msgr2=0x7f86081edd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:15.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.448+0000 7f8606800640 1 --2- 172.21.15.28:0/2777317659 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f86080a9a50 0x7f86081edd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:15.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.448+0000 7f8606800640 1 -- 172.21.15.28:0/2777317659 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f85ec004d40 msgr2=0x7f86080a8a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:15.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.448+0000 7f8606800640 1 --2- 172.21.15.28:0/2777317659 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f85ec004d40 0x7f86080a8a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:15.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.448+0000 7f8606800640 1 -- 172.21.15.28:0/2777317659 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8608158a30 con 0x7f86080a8fa0 2024-02-21T23:14:15.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.449+0000 7f8606800640 1 --2- 172.21.15.28:0/2777317659 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86080a8fa0 0x7f86081ee290 secure :-1 s=READY pgs=2463 cs=0 l=1 rev1=1 crypto rx=0x7f860000e230 tx=0x7f8600009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:15.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.449+0000 7f85fe000640 1 -- 172.21.15.28:0/2777317659 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8600047070 con 0x7f86080a8fa0 2024-02-21T23:14:15.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.449+0000 7f85fe000640 1 -- 172.21.15.28:0/2777317659 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8600034d40 con 0x7f86080a8fa0 2024-02-21T23:14:15.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.449+0000 7f85fe000640 1 -- 172.21.15.28:0/2777317659 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8600039040 con 0x7f86080a8fa0 2024-02-21T23:14:15.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.449+0000 7f8607200640 1 -- 172.21.15.28:0/2777317659 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8608158d10 con 0x7f86080a8fa0 2024-02-21T23:14:15.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.449+0000 7f8607200640 1 -- 172.21.15.28:0/2777317659 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86080a9ea0 con 0x7f86080a8fa0 2024-02-21T23:14:15.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.451+0000 7f8607200640 1 -- 172.21.15.28:0/2777317659 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f860804f290 con 0x7f86080a8fa0 2024-02-21T23:14:15.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.451+0000 7f85fe000640 1 -- 172.21.15.28:0/2777317659 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8600003990 con 0x7f86080a8fa0 2024-02-21T23:14:15.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.452+0000 7f85fe000640 1 --2- 172.21.15.28:0/2777317659 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f85e806b500 0x7f85e806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:15.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.452+0000 7f85fe000640 1 -- 172.21.15.28:0/2777317659 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f86000ac0c0 con 0x7f86080a8fa0 2024-02-21T23:14:15.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.453+0000 7f8605e00640 1 --2- 172.21.15.28:0/2777317659 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f85e806b500 0x7f85e806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:15.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.454+0000 7f8605e00640 1 --2- 172.21.15.28:0/2777317659 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f85e806b500 0x7f85e806d9c0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f85f8001eb0 tx=0x7f85f8001d80 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:15.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.461+0000 7f85fe000640 1 -- 172.21.15.28:0/2777317659 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f860007a820 con 0x7f86080a8fa0 2024-02-21T23:14:15.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.709+0000 7f8607200640 1 -- 172.21.15.28:0/2777317659 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f86080c1f40 con 0x7f86080a8fa0 2024-02-21T23:14:15.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.710+0000 7f85fe000640 1 -- 172.21.15.28:0/2777317659 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 240 v240) v1 ==== 95+0+4175 (secure 0 0 0) 0x7f860007e8c0 con 0x7f86080a8fa0 2024-02-21T23:14:15.699 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:14:15.699 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":240,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7265,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":224}],"filesystems":[{"mdsmap":{"epoch":240,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:14:11.669846+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":209,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":8287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8287":{"gid":8287,"name":"d","rank":1,"incarnation":235,"state":"up:stopping","state_seq":15,"addr":"172.21.15.81:6835/1607587459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1607587459},{"type":"v1","addr":"172.21.15.81:6835","nonce":1607587459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:14:15.699 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 240 2024-02-21T23:14:15.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.713+0000 7f8607200640 1 -- 172.21.15.28:0/2777317659 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f85e806b500 msgr2=0x7f85e806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:15.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.713+0000 7f8607200640 1 --2- 172.21.15.28:0/2777317659 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f85e806b500 0x7f85e806d9c0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f85f8001eb0 tx=0x7f85f8001d80 comp rx=0 tx=0).stop 2024-02-21T23:14:15.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.713+0000 7f8607200640 1 -- 172.21.15.28:0/2777317659 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86080a8fa0 msgr2=0x7f86081ee290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:15.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.713+0000 7f8607200640 1 --2- 172.21.15.28:0/2777317659 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86080a8fa0 0x7f86081ee290 secure :-1 s=READY pgs=2463 cs=0 l=1 rev1=1 crypto rx=0x7f860000e230 tx=0x7f8600009750 comp rx=0 tx=0).stop 2024-02-21T23:14:15.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.713+0000 7f8607200640 1 -- 172.21.15.28:0/2777317659 shutdown_connections 2024-02-21T23:14:15.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.713+0000 7f8607200640 1 --2- 172.21.15.28:0/2777317659 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f85e806b500 0x7f85e806d9c0 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:15.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.713+0000 7f8607200640 1 --2- 172.21.15.28:0/2777317659 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f86080a9a50 0x7f86081edd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:15.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.713+0000 7f8607200640 1 --2- 172.21.15.28:0/2777317659 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86080a8fa0 0x7f86081ee290 unknown :-1 s=CLOSED pgs=2463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:15.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.713+0000 7f8607200640 1 --2- 172.21.15.28:0/2777317659 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f85ec004d40 0x7f86080a8a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:15.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.713+0000 7f8607200640 1 -- 172.21.15.28:0/2777317659 >> 172.21.15.28:0/2777317659 conn(0x7f860806ae70 msgr2=0x7f860806b280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:14:15.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.713+0000 7f8607200640 1 -- 172.21.15.28:0/2777317659 shutdown_connections 2024-02-21T23:14:15.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:15.713+0000 7f8607200640 1 -- 172.21.15.28:0/2777317659 wait complete. 2024-02-21T23:14:16.124 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 240, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:14:11.669846+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 209, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 8287}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8287': {'gid': 8287, 'name': 'd', 'rank': 1, 'incarnation': 235, 'state': 'up:stopping', 'state_seq': 15, 'addr': '172.21.15.81:6835/1607587459', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6834', 'nonce': 1607587459}, {'type': 'v1', 'addr': '172.21.15.81:6835', 'nonce': 1607587459}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:14:16.124 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8287:up:stopping 2024-02-21T23:14:17.126 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:14:17.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.372+0000 7f77bf200640 1 Processor -- start 2024-02-21T23:14:17.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.372+0000 7f77bf200640 1 -- start start 2024-02-21T23:14:17.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.373+0000 7f77bf200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f77c00aff90 0x7f77c00c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:17.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.374+0000 7f77bf200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f77c006dbf0 0x7f77c0158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:17.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.374+0000 7f77bf200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77c00c74c0 con 0x7f77c00c5720 2024-02-21T23:14:17.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.374+0000 7f77bf200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77c00c7270 con 0x7f77c00aff90 2024-02-21T23:14:17.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.374+0000 7f77bf200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77c0158630 con 0x7f77c006dbf0 2024-02-21T23:14:17.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.374+0000 7f77bde00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f77c00aff90 0x7f77c00c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:17.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.374+0000 7f77bde00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f77c00aff90 0x7f77c00c51e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34222/0 (socket says 172.21.15.28:34222) 2024-02-21T23:14:17.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.374+0000 7f77bde00640 1 -- 172.21.15.28:0/4259265768 learned_addr learned my addr 172.21.15.28:0/4259265768 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:14:17.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.374+0000 7f77b5400640 1 --2- 172.21.15.28:0/4259265768 >> v2:172.21.15.81:3300/0 conn(0x7f77c006dbf0 0x7f77c0158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:17.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.374+0000 7f77bde00640 1 -- 172.21.15.28:0/4259265768 >> v2:172.21.15.81:3300/0 conn(0x7f77c006dbf0 msgr2=0x7f77c0158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:14:17.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.374+0000 7f77bde00640 1 --2- 172.21.15.28:0/4259265768 >> v2:172.21.15.81:3300/0 conn(0x7f77c006dbf0 0x7f77c0158ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:17.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.374+0000 7f77bde00640 1 -- 172.21.15.28:0/4259265768 >> v1:172.21.15.81:6789/0 conn(0x7f77c00c5720 legacy=0x7f77c006c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:14:17.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.375+0000 7f77bde00640 1 -- 172.21.15.28:0/4259265768 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77c00c77a0 con 0x7f77c00aff90 2024-02-21T23:14:17.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.376+0000 7f77bde00640 1 --2- 172.21.15.28:0/4259265768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77c00aff90 0x7f77c00c51e0 secure :-1 s=READY pgs=2464 cs=0 l=1 rev1=1 crypto rx=0x7f77c00c73f0 tx=0x7f77b002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57e649e1c0616217 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:17.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.376+0000 7f77bd400640 1 -- 172.21.15.28:0/4259265768 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77b002ca70 con 0x7f77c00aff90 2024-02-21T23:14:17.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.376+0000 7f77bd400640 1 -- 172.21.15.28:0/4259265768 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77b002cc10 con 0x7f77c00aff90 2024-02-21T23:14:17.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.376+0000 7f77bd400640 1 -- 172.21.15.28:0/4259265768 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77b00343f0 con 0x7f77c00aff90 2024-02-21T23:14:17.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.377+0000 7f77bf200640 1 -- 172.21.15.28:0/4259265768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77c00aff90 msgr2=0x7f77c00c51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:17.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.377+0000 7f77bf200640 1 --2- 172.21.15.28:0/4259265768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77c00aff90 0x7f77c00c51e0 secure :-1 s=READY pgs=2464 cs=0 l=1 rev1=1 crypto rx=0x7f77c00c73f0 tx=0x7f77b002bed0 comp rx=0 tx=0).stop 2024-02-21T23:14:17.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.377+0000 7f77bf200640 1 -- 172.21.15.28:0/4259265768 shutdown_connections 2024-02-21T23:14:17.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.377+0000 7f77bf200640 1 --2- 172.21.15.28:0/4259265768 >> v2:172.21.15.81:3300/0 conn(0x7f77c006dbf0 0x7f77c0158ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:17.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.377+0000 7f77bf200640 1 --2- 172.21.15.28:0/4259265768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77c00aff90 0x7f77c00c51e0 unknown :-1 s=CLOSED pgs=2464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:17.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.377+0000 7f77bf200640 1 -- 172.21.15.28:0/4259265768 >> 172.21.15.28:0/4259265768 conn(0x7f77c00ac370 msgr2=0x7f77c00ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:14:17.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.377+0000 7f77bf200640 1 -- 172.21.15.28:0/4259265768 shutdown_connections 2024-02-21T23:14:17.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.377+0000 7f77bf200640 1 -- 172.21.15.28:0/4259265768 wait complete. 2024-02-21T23:14:17.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.380+0000 7f77bf200640 1 Processor -- start 2024-02-21T23:14:17.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.380+0000 7f77bf200640 1 -- start start 2024-02-21T23:14:17.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.382+0000 7f77bf200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f77c006dbf0 0x7f77c01eb4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:17.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.382+0000 7f77bf200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77c00aff90 0x7f77c01515e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:17.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.382+0000 7f77bf200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f77c00c5720 0x7f77c00c1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:17.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.382+0000 7f77bf200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77c00c8760 con 0x7f77c00aff90 2024-02-21T23:14:17.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.382+0000 7f77bf200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77c00be0c0 con 0x7f77c006dbf0 2024-02-21T23:14:17.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.382+0000 7f77bf200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77c00c88e0 con 0x7f77c00c5720 2024-02-21T23:14:17.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.382+0000 7f77b5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77c00aff90 0x7f77c01515e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:17.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.382+0000 7f77b5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77c00aff90 0x7f77c01515e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34226/0 (socket says 172.21.15.28:34226) 2024-02-21T23:14:17.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.382+0000 7f77b5400640 1 -- 172.21.15.28:0/4126377718 learned_addr learned my addr 172.21.15.28:0/4126377718 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:14:17.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.382+0000 7f77be800640 1 --2- 172.21.15.28:0/4126377718 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f77c00c5720 0x7f77c00c1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:17.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.382+0000 7f77bde00640 1 --2- 172.21.15.28:0/4126377718 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f77c006dbf0 0x7f77c01eb4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:17.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.382+0000 7f77b5400640 1 -- 172.21.15.28:0/4126377718 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f77c00c5720 msgr2=0x7f77c00c1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:17.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.382+0000 7f77b5400640 1 --2- 172.21.15.28:0/4126377718 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f77c00c5720 0x7f77c00c1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:17.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.382+0000 7f77b5400640 1 -- 172.21.15.28:0/4126377718 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f77c006dbf0 msgr2=0x7f77c01eb4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:17.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.382+0000 7f77b5400640 1 --2- 172.21.15.28:0/4126377718 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f77c006dbf0 0x7f77c01eb4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:17.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.382+0000 7f77b5400640 1 -- 172.21.15.28:0/4126377718 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77c0154a00 con 0x7f77c00aff90 2024-02-21T23:14:17.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.383+0000 7f77b5400640 1 --2- 172.21.15.28:0/4126377718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77c00aff90 0x7f77c01515e0 secure :-1 s=READY pgs=2465 cs=0 l=1 rev1=1 crypto rx=0x7f77a800d7d0 tx=0x7f77a800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:17.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.383+0000 7f77b6a00640 1 -- 172.21.15.28:0/4126377718 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77a800b660 con 0x7f77c00aff90 2024-02-21T23:14:17.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.383+0000 7f77b6a00640 1 -- 172.21.15.28:0/4126377718 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77a8019070 con 0x7f77c00aff90 2024-02-21T23:14:17.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.383+0000 7f77b6a00640 1 -- 172.21.15.28:0/4126377718 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77a800ba20 con 0x7f77c00aff90 2024-02-21T23:14:17.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.383+0000 7f77bf200640 1 -- 172.21.15.28:0/4126377718 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77c0152af0 con 0x7f77c00aff90 2024-02-21T23:14:17.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.383+0000 7f77bf200640 1 -- 172.21.15.28:0/4126377718 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77c01530b0 con 0x7f77c00aff90 2024-02-21T23:14:17.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.385+0000 7f77bf200640 1 -- 172.21.15.28:0/4126377718 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77c004f290 con 0x7f77c00aff90 2024-02-21T23:14:17.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.386+0000 7f77b6a00640 1 -- 172.21.15.28:0/4126377718 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f77a800bbc0 con 0x7f77c00aff90 2024-02-21T23:14:17.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.387+0000 7f77b6a00640 1 --2- 172.21.15.28:0/4126377718 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f779c06b550 0x7f779c06da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:17.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.387+0000 7f77b6a00640 1 -- 172.21.15.28:0/4126377718 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f77a808a630 con 0x7f77c00aff90 2024-02-21T23:14:17.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.388+0000 7f77bde00640 1 --2- 172.21.15.28:0/4126377718 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f779c06b550 0x7f779c06da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:17.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.389+0000 7f77bde00640 1 --2- 172.21.15.28:0/4126377718 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f779c06b550 0x7f779c06da10 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f77b0035000 tx=0x7f77b0030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:17.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.391+0000 7f77b6a00640 1 -- 172.21.15.28:0/4126377718 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f77a8059440 con 0x7f77c00aff90 2024-02-21T23:14:17.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.630+0000 7f77bf200640 1 -- 172.21.15.28:0/4126377718 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f77c01572a0 con 0x7f77c00aff90 2024-02-21T23:14:17.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.631+0000 7f77b6a00640 1 -- 172.21.15.28:0/4126377718 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 240 v240) v1 ==== 95+0+4175 (secure 0 0 0) 0x7f77a8058d40 con 0x7f77c00aff90 2024-02-21T23:14:17.620 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:14:17.620 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":240,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7265,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":224}],"filesystems":[{"mdsmap":{"epoch":240,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:14:11.669846+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":209,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":8287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8287":{"gid":8287,"name":"d","rank":1,"incarnation":235,"state":"up:stopping","state_seq":15,"addr":"172.21.15.81:6835/1607587459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1607587459},{"type":"v1","addr":"172.21.15.81:6835","nonce":1607587459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:14:17.621 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 240 2024-02-21T23:14:17.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.634+0000 7f77bf200640 1 -- 172.21.15.28:0/4126377718 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f779c06b550 msgr2=0x7f779c06da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:17.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.634+0000 7f77bf200640 1 --2- 172.21.15.28:0/4126377718 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f779c06b550 0x7f779c06da10 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f77b0035000 tx=0x7f77b0030000 comp rx=0 tx=0).stop 2024-02-21T23:14:17.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.635+0000 7f77bf200640 1 -- 172.21.15.28:0/4126377718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77c00aff90 msgr2=0x7f77c01515e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:17.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.635+0000 7f77bf200640 1 --2- 172.21.15.28:0/4126377718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77c00aff90 0x7f77c01515e0 secure :-1 s=READY pgs=2465 cs=0 l=1 rev1=1 crypto rx=0x7f77a800d7d0 tx=0x7f77a800dc90 comp rx=0 tx=0).stop 2024-02-21T23:14:17.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.635+0000 7f77bf200640 1 -- 172.21.15.28:0/4126377718 shutdown_connections 2024-02-21T23:14:17.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.635+0000 7f77bf200640 1 --2- 172.21.15.28:0/4126377718 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f779c06b550 0x7f779c06da10 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:17.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.635+0000 7f77bf200640 1 --2- 172.21.15.28:0/4126377718 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f77c00c5720 0x7f77c00c1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:17.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.635+0000 7f77bf200640 1 --2- 172.21.15.28:0/4126377718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77c00aff90 0x7f77c01515e0 unknown :-1 s=CLOSED pgs=2465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:17.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.635+0000 7f77bf200640 1 --2- 172.21.15.28:0/4126377718 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f77c006dbf0 0x7f77c01eb4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:17.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.635+0000 7f77bf200640 1 -- 172.21.15.28:0/4126377718 >> 172.21.15.28:0/4126377718 conn(0x7f77c00ac370 msgr2=0x7f77c00ac750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:14:17.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.635+0000 7f77bf200640 1 -- 172.21.15.28:0/4126377718 shutdown_connections 2024-02-21T23:14:17.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:17.635+0000 7f77bf200640 1 -- 172.21.15.28:0/4126377718 wait complete. 2024-02-21T23:14:18.047 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 240, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:14:11.669846+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 209, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 8287}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8287': {'gid': 8287, 'name': 'd', 'rank': 1, 'incarnation': 235, 'state': 'up:stopping', 'state_seq': 15, 'addr': '172.21.15.81:6835/1607587459', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6834', 'nonce': 1607587459}, {'type': 'v1', 'addr': '172.21.15.81:6835', 'nonce': 1607587459}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:14:18.047 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8287:up:stopping 2024-02-21T23:14:19.048 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:14:19.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.296+0000 7f2267e00640 1 Processor -- start 2024-02-21T23:14:19.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.297+0000 7f2267e00640 1 -- start start 2024-02-21T23:14:19.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.299+0000 7f2267e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2260151f60 0x7f22600c11f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:19.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.299+0000 7f2267e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f22600c1730 0x7f226006dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:19.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.299+0000 7f2267e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f226006f230 con 0x7f226006e220 2024-02-21T23:14:19.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.299+0000 7f2267e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2260071a60 con 0x7f2260151f60 2024-02-21T23:14:19.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.299+0000 7f2267e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2260071c90 con 0x7f22600c1730 2024-02-21T23:14:19.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.299+0000 7f2265400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2260151f60 0x7f22600c11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:19.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.299+0000 7f2265400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2260151f60 0x7f22600c11f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34228/0 (socket says 172.21.15.28:34228) 2024-02-21T23:14:19.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.299+0000 7f2265400640 1 -- 172.21.15.28:0/3240888070 learned_addr learned my addr 172.21.15.28:0/3240888070 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:14:19.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.299+0000 7f2264a00640 1 --2- 172.21.15.28:0/3240888070 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f22600c1730 0x7f226006dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:19.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.300+0000 7f2265400640 1 -- 172.21.15.28:0/3240888070 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f22600c1730 msgr2=0x7f226006dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:14:19.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.300+0000 7f2265400640 1 --2- 172.21.15.28:0/3240888070 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f22600c1730 0x7f226006dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:19.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.300+0000 7f2265400640 1 -- 172.21.15.28:0/3240888070 >> v1:172.21.15.81:6789/0 conn(0x7f226006e220 legacy=0x7f2260154a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:14:19.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.300+0000 7f2265400640 1 -- 172.21.15.28:0/3240888070 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2260071f70 con 0x7f2260151f60 2024-02-21T23:14:19.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.301+0000 7f2265400640 1 --2- 172.21.15.28:0/3240888070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2260151f60 0x7f22600c11f0 secure :-1 s=READY pgs=2466 cs=0 l=1 rev1=1 crypto rx=0x7f2254009870 tx=0x7f225402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca39542fcaaa0426 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:19.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.301+0000 7f225fe00640 1 -- 172.21.15.28:0/3240888070 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2254035020 con 0x7f2260151f60 2024-02-21T23:14:19.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.301+0000 7f225fe00640 1 -- 172.21.15.28:0/3240888070 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2254031640 con 0x7f2260151f60 2024-02-21T23:14:19.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.301+0000 7f225fe00640 1 -- 172.21.15.28:0/3240888070 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f225402d040 con 0x7f2260151f60 2024-02-21T23:14:19.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.302+0000 7f2267e00640 1 -- 172.21.15.28:0/3240888070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2260151f60 msgr2=0x7f22600c11f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:19.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.302+0000 7f2267e00640 1 --2- 172.21.15.28:0/3240888070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2260151f60 0x7f22600c11f0 secure :-1 s=READY pgs=2466 cs=0 l=1 rev1=1 crypto rx=0x7f2254009870 tx=0x7f225402b8f0 comp rx=0 tx=0).stop 2024-02-21T23:14:19.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.302+0000 7f2267e00640 1 -- 172.21.15.28:0/3240888070 shutdown_connections 2024-02-21T23:14:19.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.302+0000 7f2267e00640 1 --2- 172.21.15.28:0/3240888070 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f22600c1730 0x7f226006dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:19.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.302+0000 7f2267e00640 1 --2- 172.21.15.28:0/3240888070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2260151f60 0x7f22600c11f0 unknown :-1 s=CLOSED pgs=2466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:19.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.302+0000 7f2267e00640 1 -- 172.21.15.28:0/3240888070 >> 172.21.15.28:0/3240888070 conn(0x7f22600c67f0 msgr2=0x7f22600c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:14:19.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.302+0000 7f2267e00640 1 -- 172.21.15.28:0/3240888070 shutdown_connections 2024-02-21T23:14:19.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.302+0000 7f2267e00640 1 -- 172.21.15.28:0/3240888070 wait complete. 2024-02-21T23:14:19.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.304+0000 7f2267e00640 1 Processor -- start 2024-02-21T23:14:19.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.304+0000 7f2267e00640 1 -- start start 2024-02-21T23:14:19.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.305+0000 7f2267e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f226006e220 0x7f226014abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:19.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.305+0000 7f2267e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f22600c1730 0x7f22601eb520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:19.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.305+0000 7f2267e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2260151f60 0x7f22601ee380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:19.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.305+0000 7f2267e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22600a6090 con 0x7f2260151f60 2024-02-21T23:14:19.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.305+0000 7f2267e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2260152340 con 0x7f22600c1730 2024-02-21T23:14:19.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.305+0000 7f2267e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22600a6210 con 0x7f226006e220 2024-02-21T23:14:19.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.305+0000 7f2265e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2260151f60 0x7f22601ee380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:19.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.305+0000 7f2265e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2260151f60 0x7f22601ee380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34244/0 (socket says 172.21.15.28:34244) 2024-02-21T23:14:19.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.305+0000 7f2265e00640 1 -- 172.21.15.28:0/88373565 learned_addr learned my addr 172.21.15.28:0/88373565 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:14:19.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.305+0000 7f2265400640 1 --2- 172.21.15.28:0/88373565 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f226006e220 0x7f226014abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:19.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.306+0000 7f2264a00640 1 --2- 172.21.15.28:0/88373565 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f22600c1730 0x7f22601eb520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:19.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.306+0000 7f2265e00640 1 -- 172.21.15.28:0/88373565 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f226006e220 msgr2=0x7f226014abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:19.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.306+0000 7f2265e00640 1 --2- 172.21.15.28:0/88373565 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f226006e220 0x7f226014abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:19.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.306+0000 7f2265e00640 1 -- 172.21.15.28:0/88373565 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f22600c1730 msgr2=0x7f22601eb520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:19.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.306+0000 7f2265e00640 1 --2- 172.21.15.28:0/88373565 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f22600c1730 0x7f22601eb520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:19.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.306+0000 7f2265e00640 1 -- 172.21.15.28:0/88373565 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22600a82c0 con 0x7f2260151f60 2024-02-21T23:14:19.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.306+0000 7f2264a00640 1 --2- 172.21.15.28:0/88373565 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f22600c1730 0x7f22601eb520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:14:19.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.306+0000 7f2265e00640 1 --2- 172.21.15.28:0/88373565 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2260151f60 0x7f22601ee380 secure :-1 s=READY pgs=2467 cs=0 l=1 rev1=1 crypto rx=0x7f225800c7b0 tx=0x7f225800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:19.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.307+0000 7f225d600640 1 -- 172.21.15.28:0/88373565 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2258004b60 con 0x7f2260151f60 2024-02-21T23:14:19.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.307+0000 7f225d600640 1 -- 172.21.15.28:0/88373565 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2258004d00 con 0x7f2260151f60 2024-02-21T23:14:19.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.307+0000 7f2267e00640 1 -- 172.21.15.28:0/88373565 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f226014e200 con 0x7f2260151f60 2024-02-21T23:14:19.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.307+0000 7f225d600640 1 -- 172.21.15.28:0/88373565 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f225800aa90 con 0x7f2260151f60 2024-02-21T23:14:19.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.307+0000 7f2267e00640 1 -- 172.21.15.28:0/88373565 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22600a6f50 con 0x7f2260151f60 2024-02-21T23:14:19.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.308+0000 7f2267e00640 1 -- 172.21.15.28:0/88373565 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22601eb050 con 0x7f2260151f60 2024-02-21T23:14:19.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.310+0000 7f225d600640 1 -- 172.21.15.28:0/88373565 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f225800ac30 con 0x7f2260151f60 2024-02-21T23:14:19.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.311+0000 7f225d600640 1 --2- 172.21.15.28:0/88373565 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f224006b4d0 0x7f224006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:19.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.311+0000 7f225d600640 1 -- 172.21.15.28:0/88373565 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f225808a6c0 con 0x7f2260151f60 2024-02-21T23:14:19.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.312+0000 7f2265400640 1 --2- 172.21.15.28:0/88373565 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f224006b4d0 0x7f224006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:19.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.313+0000 7f2265400640 1 --2- 172.21.15.28:0/88373565 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f224006b4d0 0x7f224006d990 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f2254036000 tx=0x7f225403c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:19.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.316+0000 7f225d600640 1 -- 172.21.15.28:0/88373565 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f2258020070 con 0x7f2260151f60 2024-02-21T23:14:19.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.553+0000 7f2267e00640 1 -- 172.21.15.28:0/88373565 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2260067f30 con 0x7f2260151f60 2024-02-21T23:14:19.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.554+0000 7f225d600640 1 -- 172.21.15.28:0/88373565 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 240 v240) v1 ==== 95+0+4175 (secure 0 0 0) 0x7f2258059350 con 0x7f2260151f60 2024-02-21T23:14:19.541 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:14:19.542 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":240,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7265,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":224}],"filesystems":[{"mdsmap":{"epoch":240,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:14:11.669846+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":209,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":8287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8287":{"gid":8287,"name":"d","rank":1,"incarnation":235,"state":"up:stopping","state_seq":15,"addr":"172.21.15.81:6835/1607587459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1607587459},{"type":"v1","addr":"172.21.15.81:6835","nonce":1607587459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:14:19.542 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 240 2024-02-21T23:14:19.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.557+0000 7f2267e00640 1 -- 172.21.15.28:0/88373565 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f224006b4d0 msgr2=0x7f224006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:19.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.557+0000 7f2267e00640 1 --2- 172.21.15.28:0/88373565 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f224006b4d0 0x7f224006d990 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f2254036000 tx=0x7f225403c040 comp rx=0 tx=0).stop 2024-02-21T23:14:19.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.557+0000 7f2267e00640 1 -- 172.21.15.28:0/88373565 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2260151f60 msgr2=0x7f22601ee380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:19.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.557+0000 7f2267e00640 1 --2- 172.21.15.28:0/88373565 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2260151f60 0x7f22601ee380 secure :-1 s=READY pgs=2467 cs=0 l=1 rev1=1 crypto rx=0x7f225800c7b0 tx=0x7f225800cc70 comp rx=0 tx=0).stop 2024-02-21T23:14:19.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.558+0000 7f2267e00640 1 -- 172.21.15.28:0/88373565 shutdown_connections 2024-02-21T23:14:19.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.558+0000 7f2267e00640 1 --2- 172.21.15.28:0/88373565 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f224006b4d0 0x7f224006d990 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:19.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.558+0000 7f2267e00640 1 --2- 172.21.15.28:0/88373565 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2260151f60 0x7f22601ee380 unknown :-1 s=CLOSED pgs=2467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:19.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.558+0000 7f2267e00640 1 --2- 172.21.15.28:0/88373565 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f22600c1730 0x7f22601eb520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:19.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.558+0000 7f2267e00640 1 --2- 172.21.15.28:0/88373565 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f226006e220 0x7f226014abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:19.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.558+0000 7f2267e00640 1 -- 172.21.15.28:0/88373565 >> 172.21.15.28:0/88373565 conn(0x7f22600c67f0 msgr2=0x7f22600c3b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:14:19.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.558+0000 7f2267e00640 1 -- 172.21.15.28:0/88373565 shutdown_connections 2024-02-21T23:14:19.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:19.558+0000 7f2267e00640 1 -- 172.21.15.28:0/88373565 wait complete. 2024-02-21T23:14:19.967 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 240, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:14:11.669846+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 209, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 8287}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8287': {'gid': 8287, 'name': 'd', 'rank': 1, 'incarnation': 235, 'state': 'up:stopping', 'state_seq': 15, 'addr': '172.21.15.81:6835/1607587459', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6834', 'nonce': 1607587459}, {'type': 'v1', 'addr': '172.21.15.81:6835', 'nonce': 1607587459}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:14:19.967 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8287:up:stopping 2024-02-21T23:14:20.968 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:14:21.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:21.204+0000 7f1a84200640 1 Processor -- start 2024-02-21T23:14:21.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:21.204+0000 7f1a84200640 1 -- start start 2024-02-21T23:14:21.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:21.205+0000 7f1a84200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f1a7c152ab0 0x7f1a7c0c7fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:21.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:21.206+0000 7f1a84200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a7c0a8180 con 0x7f1a7c158850 2024-02-21T23:14:21.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:21.206+0000 7f1a84200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a7c0c33b0 con 0x7f1a7c0c6710 2024-02-21T23:14:21.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:21.206+0000 7f1a84200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a7c0c3ce0 con 0x7f1a7c152ab0 2024-02-21T23:14:21.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:21.206+0000 7f1a82200640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f1a7c158850 0x7f1a7c158cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41188/0 (socket says 172.21.15.28:41188) 2024-02-21T23:14:21.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:21.206+0000 7f1a82200640 1 -- 172.21.15.28:0/3849221372 learned_addr learned my addr 172.21.15.28:0/3849221372 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:14:21.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:21.206+0000 7f1a81800640 1 --2- 172.21.15.28:0/3849221372 >> v2:172.21.15.81:3300/0 conn(0x7f1a7c152ab0 0x7f1a7c0c7fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:21.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:21.207+0000 7f1a7be00640 1 -- 172.21.15.28:0/3849221372 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1566594364 0 0) 0x7f1a74001110 con 0x7f1a7c158850 2024-02-21T23:14:21.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:21.207+0000 7f1a7be00640 1 -- 172.21.15.28:0/3849221372 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a60003520 con 0x7f1a7c158850 2024-02-21T23:14:21.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:21.207+0000 7f1a81800640 1 -- 172.21.15.28:0/3849221372 >> v1:172.21.15.81:6789/0 conn(0x7f1a7c0c6710 legacy=0x7f1a7c0c5210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:14:21.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:21.207+0000 7f1a81800640 1 -- 172.21.15.28:0/3849221372 >> v1:172.21.15.28:6789/0 conn(0x7f1a7c158850 legacy=0x7f1a7c158cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:21.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:21.207+0000 7f1a81800640 1 -- 172.21.15.28:0/3849221372 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a7c0c3690 con 0x7f1a7c152ab0 2024-02-21T23:14:21.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:21.208+0000 7f1a81800640 1 --2- 172.21.15.28:0/3849221372 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a7c152ab0 0x7f1a7c0c7fe0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c0097a0 tx=0x7f1a6c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3f94d7918b56cab9 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:21.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:21.209+0000 7f1a7be00640 1 -- 172.21.15.28:0/3849221372 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a6c035020 con 0x7f1a7c152ab0 2024-02-21T23:14:21.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:21.209+0000 7f1a7be00640 1 -- 172.21.15.28:0/3849221372 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a6c031830 con 0x7f1a7c152ab0 2024-02-21T23:14:21.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:21.209+0000 7f1a7be00640 1 -- 172.21.15.28:0/3849221372 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a6c031be0 con 0x7f1a7c152ab0 2024-02-21T23:14:21.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:21.209+0000 7f1a84200640 1 -- 172.21.15.28:0/3849221372 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a7c152ab0 msgr2=0x7f1a7c0c7fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:21.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:21.209+0000 7f1a84200640 1 --2- 172.21.15.28:0/3849221372 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a7c152ab0 0x7f1a7c0c7fe0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c0097a0 tx=0x7f1a6c02b8d0 comp rx=0 tx=0).stop 2024-02-21T23:14:21.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:21.210+0000 7f1a84200640 1 -- 172.21.15.28:0/3849221372 shutdown_connections 2024-02-21T23:14:21.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:21.210+0000 7f1a84200640 1 --2- 172.21.15.28:0/3849221372 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a7c152ab0 0x7f1a7c0c7fe0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:21.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:21.210+0000 7f1a84200640 1 -- 172.21.15.28:0/3849221372 >> 172.21.15.28:0/3849221372 conn(0x7f1a7c0c8810 msgr2=0x7f1a7c0c8bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:14:21.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:21.210+0000 7f1a84200640 1 -- 172.21.15.28:0/3849221372 shutdown_connections 2024-02-21T23:14:21.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:21.210+0000 7f1a84200640 1 -- 172.21.15.28:0/3849221372 wait complete. 2024-02-21T23:14:21.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:21.212+0000 7f1a84200640 1 Processor -- start 2024-02-21T23:14:21.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:21.213+0000 7f1a84200640 1 -- start start 2024-02-21T23:14:21.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:21.213+0000 7f1a84200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1a7c0c6710 0x7f1a7c1eed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:21.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:21.213+0000 7f1a84200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a7c152ab0 0x7f1a7c0aac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:21.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:21.214+0000 7f1a84200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a7c158850 0x7f1a7c14d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:21.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:21.214+0000 7f1a84200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a7c159550 con 0x7f1a7c158850 2024-02-21T23:14:21.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:21.214+0000 7f1a84200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a7c1593d0 con 0x7f1a7c152ab0 2024-02-21T23:14:21.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:21.214+0000 7f1a84200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a7c0c5920 con 0x7f1a7c0c6710 2024-02-21T23:14:21.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:21.214+0000 7f1a82200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a7c158850 0x7f1a7c14d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:21.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:21.214+0000 7f1a82200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a7c158850 0x7f1a7c14d5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34254/0 (socket says 172.21.15.28:34254) 2024-02-21T23:14:21.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:21.214+0000 7f1a82200640 1 -- 172.21.15.28:0/2029855831 learned_addr learned my addr 172.21.15.28:0/2029855831 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:14:21.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:21.214+0000 7f1a80e00640 1 --2- 172.21.15.28:0/2029855831 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a7c152ab0 0x7f1a7c0aac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:21.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:21.214+0000 7f1a81800640 1 --2- 172.21.15.28:0/2029855831 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1a7c0c6710 0x7f1a7c1eed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:21.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:21.214+0000 7f1a82200640 1 -- 172.21.15.28:0/2029855831 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1a7c0c6710 msgr2=0x7f1a7c1eed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:21.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:21.214+0000 7f1a82200640 1 --2- 172.21.15.28:0/2029855831 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1a7c0c6710 0x7f1a7c1eed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:21.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:21.214+0000 7f1a82200640 1 -- 172.21.15.28:0/2029855831 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a7c152ab0 msgr2=0x7f1a7c0aac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:21.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:21.214+0000 7f1a82200640 1 --2- 172.21.15.28:0/2029855831 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a7c152ab0 0x7f1a7c0aac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:21.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:21.214+0000 7f1a82200640 1 -- 172.21.15.28:0/2029855831 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a7c06fc40 con 0x7f1a7c158850 2024-02-21T23:14:21.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:21.215+0000 7f1a82200640 1 --2- 172.21.15.28:0/2029855831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a7c158850 0x7f1a7c14d5a0 secure :-1 s=READY pgs=2469 cs=0 l=1 rev1=1 crypto rx=0x7f1a7400c3a0 tx=0x7f1a7400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:21.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:21.215+0000 7f1a79600640 1 -- 172.21.15.28:0/2029855831 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a74017070 con 0x7f1a7c158850 2024-02-21T23:14:21.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:21.215+0000 7f1a79600640 1 -- 172.21.15.28:0/2029855831 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a74005ce0 con 0x7f1a7c158850 2024-02-21T23:14:21.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:21.215+0000 7f1a84200640 1 -- 172.21.15.28:0/2029855831 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a7c150790 con 0x7f1a7c158850 2024-02-21T23:14:21.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:21.215+0000 7f1a79600640 1 -- 172.21.15.28:0/2029855831 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a740072f0 con 0x7f1a7c158850 2024-02-21T23:14:21.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:21.215+0000 7f1a84200640 1 -- 172.21.15.28:0/2029855831 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a7c06ff00 con 0x7f1a7c158850 2024-02-21T23:14:21.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:21.217+0000 7f1a84200640 1 -- 172.21.15.28:0/2029855831 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a7c070100 con 0x7f1a7c158850 2024-02-21T23:14:21.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:21.217+0000 7f1a79600640 1 -- 172.21.15.28:0/2029855831 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1a7401a070 con 0x7f1a7c158850 2024-02-21T23:14:21.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:21.218+0000 7f1a79600640 1 --2- 172.21.15.28:0/2029855831 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1a5c06b4d0 0x7f1a5c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:21.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:21.218+0000 7f1a79600640 1 -- 172.21.15.28:0/2029855831 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f1a74089740 con 0x7f1a7c158850 2024-02-21T23:14:21.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:21.219+0000 7f1a81800640 1 --2- 172.21.15.28:0/2029855831 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1a5c06b4d0 0x7f1a5c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:21.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:21.220+0000 7f1a81800640 1 --2- 172.21.15.28:0/2029855831 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1a5c06b4d0 0x7f1a5c06d990 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c036000 tx=0x7f1a6c008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:21.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:21.223+0000 7f1a79600640 1 -- 172.21.15.28:0/2029855831 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1a7c070100 con 0x7f1a7c158850 2024-02-21T23:14:21.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:21.469+0000 7f1a84200640 1 -- 172.21.15.28:0/2029855831 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a7c067f30 con 0x7f1a7c158850 2024-02-21T23:14:21.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:21.469+0000 7f1a79600640 1 -- 172.21.15.28:0/2029855831 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 240 v240) v1 ==== 95+0+4175 (secure 0 0 0) 0x7f1a7c0a8180 con 0x7f1a7c158850 2024-02-21T23:14:21.457 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:14:21.457 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":240,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7265,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":224}],"filesystems":[{"mdsmap":{"epoch":240,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:14:11.669846+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":209,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":8287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8287":{"gid":8287,"name":"d","rank":1,"incarnation":235,"state":"up:stopping","state_seq":15,"addr":"172.21.15.81:6835/1607587459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1607587459},{"type":"v1","addr":"172.21.15.81:6835","nonce":1607587459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:14:21.458 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 240 2024-02-21T23:14:21.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:21.472+0000 7f1a84200640 1 -- 172.21.15.28:0/2029855831 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1a5c06b4d0 msgr2=0x7f1a5c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:21.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:21.472+0000 7f1a84200640 1 --2- 172.21.15.28:0/2029855831 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1a5c06b4d0 0x7f1a5c06d990 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c036000 tx=0x7f1a6c008030 comp rx=0 tx=0).stop 2024-02-21T23:14:21.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:21.472+0000 7f1a84200640 1 -- 172.21.15.28:0/2029855831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a7c158850 msgr2=0x7f1a7c14d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:21.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:21.472+0000 7f1a84200640 1 --2- 172.21.15.28:0/2029855831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a7c158850 0x7f1a7c14d5a0 secure :-1 s=READY pgs=2469 cs=0 l=1 rev1=1 crypto rx=0x7f1a7400c3a0 tx=0x7f1a7400c860 comp rx=0 tx=0).stop 2024-02-21T23:14:21.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:21.473+0000 7f1a84200640 1 -- 172.21.15.28:0/2029855831 shutdown_connections 2024-02-21T23:14:21.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:21.473+0000 7f1a84200640 1 --2- 172.21.15.28:0/2029855831 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1a5c06b4d0 0x7f1a5c06d990 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:21.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:21.473+0000 7f1a84200640 1 --2- 172.21.15.28:0/2029855831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a7c158850 0x7f1a7c14d5a0 unknown :-1 s=CLOSED pgs=2469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:21.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:21.473+0000 7f1a84200640 1 --2- 172.21.15.28:0/2029855831 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a7c152ab0 0x7f1a7c0aac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:21.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:21.473+0000 7f1a84200640 1 --2- 172.21.15.28:0/2029855831 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1a7c0c6710 0x7f1a7c1eed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:21.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:21.473+0000 7f1a84200640 1 -- 172.21.15.28:0/2029855831 >> 172.21.15.28:0/2029855831 conn(0x7f1a7c0c8810 msgr2=0x7f1a7c150de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:14:21.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:21.473+0000 7f1a84200640 1 -- 172.21.15.28:0/2029855831 shutdown_connections 2024-02-21T23:14:21.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:21.473+0000 7f1a84200640 1 -- 172.21.15.28:0/2029855831 wait complete. 2024-02-21T23:14:21.883 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 240, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:14:11.669846+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 209, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 8287}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8287': {'gid': 8287, 'name': 'd', 'rank': 1, 'incarnation': 235, 'state': 'up:stopping', 'state_seq': 15, 'addr': '172.21.15.81:6835/1607587459', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6834', 'nonce': 1607587459}, {'type': 'v1', 'addr': '172.21.15.81:6835', 'nonce': 1607587459}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:14:21.884 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8287:up:stopping 2024-02-21T23:14:22.885 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:14:23.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.121+0000 7f9c39c00640 1 Processor -- start 2024-02-21T23:14:23.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.121+0000 7f9c39c00640 1 -- start start 2024-02-21T23:14:23.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.123+0000 7f9c39c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f9c340abe40 0x7f9c3406be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:23.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.123+0000 7f9c39c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9c340acd10 0x7f9c34159760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:23.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.123+0000 7f9c39c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9c3406c3c0 0x7f9c34159ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:23.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.123+0000 7f9c39c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c3415d940 con 0x7f9c3406c3c0 2024-02-21T23:14:23.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.123+0000 7f9c39c00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c3415d380 con 0x7f9c340abe40 2024-02-21T23:14:23.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.123+0000 7f9c39c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c340ac220 con 0x7f9c340acd10 2024-02-21T23:14:23.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.124+0000 7f9c2be00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9c340acd10 0x7f9c34159760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:23.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.124+0000 7f9c33e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f9c340abe40 0x7f9c3406be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:23.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.124+0000 7f9c2be00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9c340acd10 0x7f9c34159760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3301/0 says I am v2:172.21.15.28:34972/0 (socket says 172.21.15.28:34972) 2024-02-21T23:14:23.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.124+0000 7f9c2be00640 1 -- 172.21.15.28:0/3702779304 learned_addr learned my addr 172.21.15.28:0/3702779304 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:14:23.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.124+0000 7f9c39200640 1 --2- 172.21.15.28:0/3702779304 >> v2:172.21.15.28:3300/0 conn(0x7f9c3406c3c0 0x7f9c34159ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:23.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.125+0000 7f9c2be00640 1 -- 172.21.15.28:0/3702779304 >> v2:172.21.15.81:3300/0 conn(0x7f9c340abe40 msgr2=0x7f9c3406be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:14:23.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.125+0000 7f9c2be00640 1 --2- 172.21.15.28:0/3702779304 >> v2:172.21.15.81:3300/0 conn(0x7f9c340abe40 0x7f9c3406be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:23.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.125+0000 7f9c2be00640 1 -- 172.21.15.28:0/3702779304 >> v2:172.21.15.28:3300/0 conn(0x7f9c3406c3c0 msgr2=0x7f9c34159ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:14:23.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.125+0000 7f9c2be00640 1 --2- 172.21.15.28:0/3702779304 >> v2:172.21.15.28:3300/0 conn(0x7f9c3406c3c0 0x7f9c34159ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:23.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.125+0000 7f9c2be00640 1 -- 172.21.15.28:0/3702779304 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c340ac6d0 con 0x7f9c340acd10 2024-02-21T23:14:23.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.125+0000 7f9c33e00640 1 --2- 172.21.15.28:0/3702779304 >> v2:172.21.15.81:3300/0 conn(0x7f9c340abe40 0x7f9c3406be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-21T23:14:23.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.126+0000 7f9c2be00640 1 --2- 172.21.15.28:0/3702779304 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9c340acd10 0x7f9c34159760 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9c34073730 tx=0x7f9c1c02b890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf77686f12f86262 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:23.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.127+0000 7f9c33400640 1 -- 172.21.15.28:0/3702779304 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c1c0317f0 con 0x7f9c340acd10 2024-02-21T23:14:23.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.127+0000 7f9c33400640 1 -- 172.21.15.28:0/3702779304 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c1c037070 con 0x7f9c340acd10 2024-02-21T23:14:23.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.127+0000 7f9c33400640 1 -- 172.21.15.28:0/3702779304 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c1c031bb0 con 0x7f9c340acd10 2024-02-21T23:14:23.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.128+0000 7f9c39c00640 1 -- 172.21.15.28:0/3702779304 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9c340acd10 msgr2=0x7f9c34159760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:23.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.128+0000 7f9c39c00640 1 --2- 172.21.15.28:0/3702779304 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9c340acd10 0x7f9c34159760 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9c34073730 tx=0x7f9c1c02b890 comp rx=0 tx=0).stop 2024-02-21T23:14:23.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.128+0000 7f9c39c00640 1 -- 172.21.15.28:0/3702779304 shutdown_connections 2024-02-21T23:14:23.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.128+0000 7f9c39c00640 1 --2- 172.21.15.28:0/3702779304 >> v2:172.21.15.28:3300/0 conn(0x7f9c3406c3c0 0x7f9c34159ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:23.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.128+0000 7f9c39c00640 1 --2- 172.21.15.28:0/3702779304 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9c340acd10 0x7f9c34159760 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:23.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.128+0000 7f9c39c00640 1 --2- 172.21.15.28:0/3702779304 >> v2:172.21.15.81:3300/0 conn(0x7f9c340abe40 0x7f9c3406be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:23.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.128+0000 7f9c39c00640 1 -- 172.21.15.28:0/3702779304 >> 172.21.15.28:0/3702779304 conn(0x7f9c34071910 msgr2=0x7f9c34071cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:14:23.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.128+0000 7f9c39c00640 1 -- 172.21.15.28:0/3702779304 shutdown_connections 2024-02-21T23:14:23.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.128+0000 7f9c39c00640 1 -- 172.21.15.28:0/3702779304 wait complete. 2024-02-21T23:14:23.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.131+0000 7f9c39c00640 1 Processor -- start 2024-02-21T23:14:23.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.131+0000 7f9c39c00640 1 -- start start 2024-02-21T23:14:23.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.133+0000 7f9c39c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9c3406c3c0 0x7f9c340c1e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:23.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.133+0000 7f9c39c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c340abe40 0x7f9c3415c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:23.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.133+0000 7f9c39c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c340acd10 0x7f9c3414aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:23.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.133+0000 7f9c39c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c34072c80 con 0x7f9c340acd10 2024-02-21T23:14:23.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.133+0000 7f9c39c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c3415d5d0 con 0x7f9c340abe40 2024-02-21T23:14:23.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.133+0000 7f9c39c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c3415a1e0 con 0x7f9c3406c3c0 2024-02-21T23:14:23.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.133+0000 7f9c39200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c340acd10 0x7f9c3414aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:23.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.133+0000 7f9c39200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c340acd10 0x7f9c3414aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34278/0 (socket says 172.21.15.28:34278) 2024-02-21T23:14:23.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.133+0000 7f9c39200640 1 -- 172.21.15.28:0/3756388966 learned_addr learned my addr 172.21.15.28:0/3756388966 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:14:23.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.133+0000 7f9c33e00640 1 --2- 172.21.15.28:0/3756388966 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9c3406c3c0 0x7f9c340c1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:23.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.133+0000 7f9c2be00640 1 --2- 172.21.15.28:0/3756388966 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c340abe40 0x7f9c3415c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:23.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.134+0000 7f9c39200640 1 -- 172.21.15.28:0/3756388966 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9c3406c3c0 msgr2=0x7f9c340c1e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:23.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.134+0000 7f9c39200640 1 --2- 172.21.15.28:0/3756388966 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9c3406c3c0 0x7f9c340c1e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:23.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.134+0000 7f9c39200640 1 -- 172.21.15.28:0/3756388966 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c340abe40 msgr2=0x7f9c3415c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:23.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.134+0000 7f9c39200640 1 --2- 172.21.15.28:0/3756388966 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c340abe40 0x7f9c3415c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:23.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.134+0000 7f9c39200640 1 -- 172.21.15.28:0/3756388966 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c340be540 con 0x7f9c340acd10 2024-02-21T23:14:23.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.134+0000 7f9c39200640 1 --2- 172.21.15.28:0/3756388966 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c340acd10 0x7f9c3414aaf0 secure :-1 s=READY pgs=2470 cs=0 l=1 rev1=1 crypto rx=0x7f9c2c00efc0 tx=0x7f9c2c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:23.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.134+0000 7f9c30c00640 1 -- 172.21.15.28:0/3756388966 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c2c018020 con 0x7f9c340acd10 2024-02-21T23:14:23.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.135+0000 7f9c30c00640 1 -- 172.21.15.28:0/3756388966 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c2c014480 con 0x7f9c340acd10 2024-02-21T23:14:23.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.135+0000 7f9c39c00640 1 -- 172.21.15.28:0/3756388966 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c341ef050 con 0x7f9c340acd10 2024-02-21T23:14:23.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.135+0000 7f9c30c00640 1 -- 172.21.15.28:0/3756388966 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c2c014870 con 0x7f9c340acd10 2024-02-21T23:14:23.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.135+0000 7f9c39c00640 1 -- 172.21.15.28:0/3756388966 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c341ef610 con 0x7f9c340acd10 2024-02-21T23:14:23.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.136+0000 7f9c39c00640 1 -- 172.21.15.28:0/3756388966 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c3404f290 con 0x7f9c340acd10 2024-02-21T23:14:23.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.137+0000 7f9c30c00640 1 -- 172.21.15.28:0/3756388966 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9c2c005d10 con 0x7f9c340acd10 2024-02-21T23:14:23.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.138+0000 7f9c30c00640 1 --2- 172.21.15.28:0/3756388966 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9c1006b4d0 0x7f9c1006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:23.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.138+0000 7f9c30c00640 1 -- 172.21.15.28:0/3756388966 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f9c2c08aef0 con 0x7f9c340acd10 2024-02-21T23:14:23.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.139+0000 7f9c33e00640 1 --2- 172.21.15.28:0/3756388966 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9c1006b4d0 0x7f9c1006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:23.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.140+0000 7f9c33e00640 1 --2- 172.21.15.28:0/3756388966 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9c1006b4d0 0x7f9c1006d990 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f9c240099b0 tx=0x7f9c24008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:23.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.143+0000 7f9c30c00640 1 -- 172.21.15.28:0/3756388966 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f9c2c022070 con 0x7f9c340acd10 2024-02-21T23:14:23.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.578+0000 7f9c39c00640 1 -- 172.21.15.28:0/3756388966 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9c34156710 con 0x7f9c340acd10 2024-02-21T23:14:23.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.579+0000 7f9c30c00640 1 -- 172.21.15.28:0/3756388966 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 240 v240) v1 ==== 95+0+4175 (secure 0 0 0) 0x7f9c340be540 con 0x7f9c340acd10 2024-02-21T23:14:23.567 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:14:23.567 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":240,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7265,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":224}],"filesystems":[{"mdsmap":{"epoch":240,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:14:11.669846+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":209,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":8287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8287":{"gid":8287,"name":"d","rank":1,"incarnation":235,"state":"up:stopping","state_seq":15,"addr":"172.21.15.81:6835/1607587459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1607587459},{"type":"v1","addr":"172.21.15.81:6835","nonce":1607587459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:14:23.567 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 240 2024-02-21T23:14:23.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.583+0000 7f9c39c00640 1 -- 172.21.15.28:0/3756388966 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9c1006b4d0 msgr2=0x7f9c1006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:23.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.583+0000 7f9c39c00640 1 --2- 172.21.15.28:0/3756388966 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9c1006b4d0 0x7f9c1006d990 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f9c240099b0 tx=0x7f9c24008040 comp rx=0 tx=0).stop 2024-02-21T23:14:23.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.583+0000 7f9c39c00640 1 -- 172.21.15.28:0/3756388966 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c340acd10 msgr2=0x7f9c3414aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:23.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.583+0000 7f9c39c00640 1 --2- 172.21.15.28:0/3756388966 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c340acd10 0x7f9c3414aaf0 secure :-1 s=READY pgs=2470 cs=0 l=1 rev1=1 crypto rx=0x7f9c2c00efc0 tx=0x7f9c2c00c490 comp rx=0 tx=0).stop 2024-02-21T23:14:23.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.583+0000 7f9c39c00640 1 -- 172.21.15.28:0/3756388966 shutdown_connections 2024-02-21T23:14:23.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.583+0000 7f9c39c00640 1 --2- 172.21.15.28:0/3756388966 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9c1006b4d0 0x7f9c1006d990 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:23.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.583+0000 7f9c39c00640 1 --2- 172.21.15.28:0/3756388966 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c340acd10 0x7f9c3414aaf0 unknown :-1 s=CLOSED pgs=2470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:23.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.583+0000 7f9c39c00640 1 --2- 172.21.15.28:0/3756388966 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c340abe40 0x7f9c3415c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:23.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.583+0000 7f9c39c00640 1 --2- 172.21.15.28:0/3756388966 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9c3406c3c0 0x7f9c340c1e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:23.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.583+0000 7f9c39c00640 1 -- 172.21.15.28:0/3756388966 >> 172.21.15.28:0/3756388966 conn(0x7f9c34071910 msgr2=0x7f9c340c6360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:14:23.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.583+0000 7f9c39c00640 1 -- 172.21.15.28:0/3756388966 shutdown_connections 2024-02-21T23:14:23.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:23.583+0000 7f9c39c00640 1 -- 172.21.15.28:0/3756388966 wait complete. 2024-02-21T23:14:23.994 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 240, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:14:11.669846+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 209, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 8287}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8287': {'gid': 8287, 'name': 'd', 'rank': 1, 'incarnation': 235, 'state': 'up:stopping', 'state_seq': 15, 'addr': '172.21.15.81:6835/1607587459', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6834', 'nonce': 1607587459}, {'type': 'v1', 'addr': '172.21.15.81:6835', 'nonce': 1607587459}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:14:23.994 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8287:up:stopping 2024-02-21T23:14:24.995 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:14:25.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.233+0000 7f0f34a00640 1 Processor -- start 2024-02-21T23:14:25.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.233+0000 7f0f34a00640 1 -- start start 2024-02-21T23:14:25.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.235+0000 7f0f34a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0f30152500 0x7f0f301558c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:25.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.235+0000 7f0f34a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0f300ca050 0x7f0f300bf6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:25.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.235+0000 7f0f34a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f301528e0 con 0x7f0f300bfc00 2024-02-21T23:14:25.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.235+0000 7f0f34a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f300c86e0 con 0x7f0f300ca050 2024-02-21T23:14:25.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.235+0000 7f0f34a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f300c8980 con 0x7f0f30152500 2024-02-21T23:14:25.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.235+0000 7f0f2ea00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0f300ca050 0x7f0f300bf6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:25.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.235+0000 7f0f2ea00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0f300ca050 0x7f0f300bf6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32864/0 (socket says 172.21.15.28:32864) 2024-02-21T23:14:25.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.235+0000 7f0f2ea00640 1 -- 172.21.15.28:0/2635291632 learned_addr learned my addr 172.21.15.28:0/2635291632 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:14:25.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.236+0000 7f0f2f400640 1 --2- 172.21.15.28:0/2635291632 >> v2:172.21.15.81:3300/0 conn(0x7f0f30152500 0x7f0f301558c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:25.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.236+0000 7f0f2ea00640 1 -- 172.21.15.28:0/2635291632 >> v2:172.21.15.81:3300/0 conn(0x7f0f30152500 msgr2=0x7f0f301558c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:14:25.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.236+0000 7f0f2ea00640 1 --2- 172.21.15.28:0/2635291632 >> v2:172.21.15.81:3300/0 conn(0x7f0f30152500 0x7f0f301558c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:25.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.236+0000 7f0f2ea00640 1 -- 172.21.15.28:0/2635291632 >> v1:172.21.15.81:6789/0 conn(0x7f0f300bfc00 legacy=0x7f0f3006bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:14:25.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.236+0000 7f0f2ea00640 1 -- 172.21.15.28:0/2635291632 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f300c8c60 con 0x7f0f300ca050 2024-02-21T23:14:25.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.236+0000 7f0f2f400640 1 --2- 172.21.15.28:0/2635291632 >> v2:172.21.15.81:3300/0 conn(0x7f0f30152500 0x7f0f301558c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:14:25.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.237+0000 7f0f2ea00640 1 --2- 172.21.15.28:0/2635291632 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f300ca050 0x7f0f300bf6c0 secure :-1 s=READY pgs=2471 cs=0 l=1 rev1=1 crypto rx=0x7f0f300ad430 tx=0x7f0f1c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27a961c8a4008970 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:25.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.237+0000 7f0f2e000640 1 -- 172.21.15.28:0/2635291632 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f1c02e070 con 0x7f0f300ca050 2024-02-21T23:14:25.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.237+0000 7f0f2e000640 1 -- 172.21.15.28:0/2635291632 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f1c037070 con 0x7f0f300ca050 2024-02-21T23:14:25.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.237+0000 7f0f2e000640 1 -- 172.21.15.28:0/2635291632 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f1c031a00 con 0x7f0f300ca050 2024-02-21T23:14:25.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.238+0000 7f0f34a00640 1 -- 172.21.15.28:0/2635291632 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f300ca050 msgr2=0x7f0f300bf6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:25.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.238+0000 7f0f34a00640 1 --2- 172.21.15.28:0/2635291632 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f300ca050 0x7f0f300bf6c0 secure :-1 s=READY pgs=2471 cs=0 l=1 rev1=1 crypto rx=0x7f0f300ad430 tx=0x7f0f1c02b880 comp rx=0 tx=0).stop 2024-02-21T23:14:25.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.238+0000 7f0f34a00640 1 -- 172.21.15.28:0/2635291632 shutdown_connections 2024-02-21T23:14:25.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.238+0000 7f0f34a00640 1 --2- 172.21.15.28:0/2635291632 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f300ca050 0x7f0f300bf6c0 unknown :-1 s=CLOSED pgs=2471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:25.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.238+0000 7f0f34a00640 1 --2- 172.21.15.28:0/2635291632 >> v2:172.21.15.81:3300/0 conn(0x7f0f30152500 0x7f0f301558c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:25.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.238+0000 7f0f34a00640 1 -- 172.21.15.28:0/2635291632 >> 172.21.15.28:0/2635291632 conn(0x7f0f300ac450 msgr2=0x7f0f300ac860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:14:25.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.239+0000 7f0f34a00640 1 -- 172.21.15.28:0/2635291632 shutdown_connections 2024-02-21T23:14:25.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.239+0000 7f0f34a00640 1 -- 172.21.15.28:0/2635291632 wait complete. 2024-02-21T23:14:25.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.240+0000 7f0f34a00640 1 Processor -- start 2024-02-21T23:14:25.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.240+0000 7f0f34a00640 1 -- start start 2024-02-21T23:14:25.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.241+0000 7f0f34a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f300bfc00 0x7f0f300ab6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:25.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.241+0000 7f0f34a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f300ca050 0x7f0f30158aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:25.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.241+0000 7f0f34a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0f30152500 0x7f0f30158fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:25.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.241+0000 7f0f34a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f3006c300 con 0x7f0f300ca050 2024-02-21T23:14:25.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.241+0000 7f0f34a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f300c9220 con 0x7f0f300bfc00 2024-02-21T23:14:25.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.241+0000 7f0f34a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f3006c480 con 0x7f0f30152500 2024-02-21T23:14:25.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.242+0000 7f0f2ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f300ca050 0x7f0f30158aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:25.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.242+0000 7f0f2ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f300ca050 0x7f0f30158aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32866/0 (socket says 172.21.15.28:32866) 2024-02-21T23:14:25.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.242+0000 7f0f2ea00640 1 -- 172.21.15.28:0/2215711532 learned_addr learned my addr 172.21.15.28:0/2215711532 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:14:25.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.242+0000 7f0f2f400640 1 --2- 172.21.15.28:0/2215711532 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f300bfc00 0x7f0f300ab6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:25.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.242+0000 7f0f2fe00640 1 --2- 172.21.15.28:0/2215711532 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0f30152500 0x7f0f30158fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:25.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.242+0000 7f0f2ea00640 1 -- 172.21.15.28:0/2215711532 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0f30152500 msgr2=0x7f0f30158fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:25.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.242+0000 7f0f2ea00640 1 --2- 172.21.15.28:0/2215711532 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0f30152500 0x7f0f30158fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:25.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.242+0000 7f0f2ea00640 1 -- 172.21.15.28:0/2215711532 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f300bfc00 msgr2=0x7f0f300ab6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:25.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.242+0000 7f0f2ea00640 1 --2- 172.21.15.28:0/2215711532 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f300bfc00 0x7f0f300ab6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:25.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.242+0000 7f0f2ea00640 1 -- 172.21.15.28:0/2215711532 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f30156a50 con 0x7f0f300ca050 2024-02-21T23:14:25.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.243+0000 7f0f2ea00640 1 --2- 172.21.15.28:0/2215711532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f300ca050 0x7f0f30158aa0 secure :-1 s=READY pgs=2472 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c02b8c0 tx=0x7f0f1c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:25.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.243+0000 7f0f27400640 1 -- 172.21.15.28:0/2215711532 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f1c032880 con 0x7f0f300ca050 2024-02-21T23:14:25.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.243+0000 7f0f27400640 1 -- 172.21.15.28:0/2215711532 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f1c032a20 con 0x7f0f300ca050 2024-02-21T23:14:25.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.243+0000 7f0f34a00640 1 -- 172.21.15.28:0/2215711532 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f30156d30 con 0x7f0f300ca050 2024-02-21T23:14:25.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.243+0000 7f0f27400640 1 -- 172.21.15.28:0/2215711532 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f1c032e10 con 0x7f0f300ca050 2024-02-21T23:14:25.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.243+0000 7f0f34a00640 1 -- 172.21.15.28:0/2215711532 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f300abbf0 con 0x7f0f300ca050 2024-02-21T23:14:25.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.244+0000 7f0f34a00640 1 -- 172.21.15.28:0/2215711532 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f3004f290 con 0x7f0f300ca050 2024-02-21T23:14:25.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.247+0000 7f0f27400640 1 -- 172.21.15.28:0/2215711532 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0f1c0453f0 con 0x7f0f300ca050 2024-02-21T23:14:25.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.248+0000 7f0f27400640 1 --2- 172.21.15.28:0/2215711532 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0f1006b4d0 0x7f0f1006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:25.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.248+0000 7f0f27400640 1 -- 172.21.15.28:0/2215711532 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f0f1c0ac3d0 con 0x7f0f300ca050 2024-02-21T23:14:25.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.249+0000 7f0f2f400640 1 --2- 172.21.15.28:0/2215711532 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0f1006b4d0 0x7f0f1006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:25.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.250+0000 7f0f2f400640 1 --2- 172.21.15.28:0/2215711532 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0f1006b4d0 0x7f0f1006d990 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f0f30156860 tx=0x7f0f20008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:25.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.253+0000 7f0f27400640 1 -- 172.21.15.28:0/2215711532 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f0f1c040070 con 0x7f0f300ca050 2024-02-21T23:14:25.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.500+0000 7f0f34a00640 1 -- 172.21.15.28:0/2215711532 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0f300c5160 con 0x7f0f300ca050 2024-02-21T23:14:25.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.501+0000 7f0f27400640 1 -- 172.21.15.28:0/2215711532 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 240 v240) v1 ==== 95+0+4175 (secure 0 0 0) 0x7f0f1c07b180 con 0x7f0f300ca050 2024-02-21T23:14:25.489 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:14:25.489 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":240,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7265,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":224}],"filesystems":[{"mdsmap":{"epoch":240,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:14:11.669846+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":209,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":8287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8287":{"gid":8287,"name":"d","rank":1,"incarnation":235,"state":"up:stopping","state_seq":15,"addr":"172.21.15.81:6835/1607587459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1607587459},{"type":"v1","addr":"172.21.15.81:6835","nonce":1607587459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:14:25.489 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 240 2024-02-21T23:14:25.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.503+0000 7f0f34a00640 1 -- 172.21.15.28:0/2215711532 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0f1006b4d0 msgr2=0x7f0f1006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:25.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.503+0000 7f0f34a00640 1 --2- 172.21.15.28:0/2215711532 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0f1006b4d0 0x7f0f1006d990 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f0f30156860 tx=0x7f0f20008040 comp rx=0 tx=0).stop 2024-02-21T23:14:25.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.503+0000 7f0f34a00640 1 -- 172.21.15.28:0/2215711532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f300ca050 msgr2=0x7f0f30158aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:25.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.503+0000 7f0f34a00640 1 --2- 172.21.15.28:0/2215711532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f300ca050 0x7f0f30158aa0 secure :-1 s=READY pgs=2472 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c02b8c0 tx=0x7f0f1c0046c0 comp rx=0 tx=0).stop 2024-02-21T23:14:25.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.504+0000 7f0f34a00640 1 -- 172.21.15.28:0/2215711532 shutdown_connections 2024-02-21T23:14:25.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.504+0000 7f0f34a00640 1 --2- 172.21.15.28:0/2215711532 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0f1006b4d0 0x7f0f1006d990 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:25.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.504+0000 7f0f34a00640 1 --2- 172.21.15.28:0/2215711532 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0f30152500 0x7f0f30158fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:25.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.504+0000 7f0f34a00640 1 --2- 172.21.15.28:0/2215711532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f300ca050 0x7f0f30158aa0 unknown :-1 s=CLOSED pgs=2472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:25.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.504+0000 7f0f34a00640 1 --2- 172.21.15.28:0/2215711532 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f300bfc00 0x7f0f300ab6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:25.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.504+0000 7f0f34a00640 1 -- 172.21.15.28:0/2215711532 >> 172.21.15.28:0/2215711532 conn(0x7f0f300ac450 msgr2=0x7f0f30150e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:14:25.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.504+0000 7f0f34a00640 1 -- 172.21.15.28:0/2215711532 shutdown_connections 2024-02-21T23:14:25.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:25.504+0000 7f0f34a00640 1 -- 172.21.15.28:0/2215711532 wait complete. 2024-02-21T23:14:25.913 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 240, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:14:11.669846+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 209, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 8287}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8287': {'gid': 8287, 'name': 'd', 'rank': 1, 'incarnation': 235, 'state': 'up:stopping', 'state_seq': 15, 'addr': '172.21.15.81:6835/1607587459', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6834', 'nonce': 1607587459}, {'type': 'v1', 'addr': '172.21.15.81:6835', 'nonce': 1607587459}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:14:25.913 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8287:up:stopping 2024-02-21T23:14:26.914 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:14:27.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.163+0000 7fda91200640 1 Processor -- start 2024-02-21T23:14:27.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.163+0000 7fda91200640 1 -- start start 2024-02-21T23:14:27.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.164+0000 7fda91200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fda8c06dbf0 0x7fda8c158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:27.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.164+0000 7fda91200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fda8c155960 0x7fda8c159000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:27.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.164+0000 7fda91200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda8c0c74c0 con 0x7fda8c0aff90 2024-02-21T23:14:27.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.164+0000 7fda91200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda8c0c7270 con 0x7fda8c06dbf0 2024-02-21T23:14:27.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.164+0000 7fda91200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda8c158630 con 0x7fda8c155960 2024-02-21T23:14:27.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.164+0000 7fda82a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fda8c06dbf0 0x7fda8c158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:27.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.164+0000 7fda82a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fda8c06dbf0 0x7fda8c158ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32874/0 (socket says 172.21.15.28:32874) 2024-02-21T23:14:27.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.164+0000 7fda82a00640 1 -- 172.21.15.28:0/1678521976 learned_addr learned my addr 172.21.15.28:0/1678521976 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:14:27.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.165+0000 7fda8be00640 1 --2- 172.21.15.28:0/1678521976 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fda8c155960 0x7fda8c159000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:27.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.165+0000 7fda82a00640 1 -- 172.21.15.28:0/1678521976 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fda8c155960 msgr2=0x7fda8c159000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:14:27.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.165+0000 7fda82a00640 1 --2- 172.21.15.28:0/1678521976 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fda8c155960 0x7fda8c159000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:27.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.165+0000 7fda82a00640 1 -- 172.21.15.28:0/1678521976 >> v1:172.21.15.28:6789/0 conn(0x7fda8c0aff90 legacy=0x7fda8c0c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:27.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.165+0000 7fda82a00640 1 -- 172.21.15.28:0/1678521976 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda8c0c77a0 con 0x7fda8c06dbf0 2024-02-21T23:14:27.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.167+0000 7fda82a00640 1 --2- 172.21.15.28:0/1678521976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda8c06dbf0 0x7fda8c158ac0 secure :-1 s=READY pgs=2474 cs=0 l=1 rev1=1 crypto rx=0x7fda74009870 tx=0x7fda7402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8aa4eb8d7837d929 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:27.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.167+0000 7fda8aa00640 1 -- 172.21.15.28:0/1678521976 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda74034070 con 0x7fda8c06dbf0 2024-02-21T23:14:27.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.167+0000 7fda8aa00640 1 -- 172.21.15.28:0/1678521976 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda74030840 con 0x7fda8c06dbf0 2024-02-21T23:14:27.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.167+0000 7fda8aa00640 1 -- 172.21.15.28:0/1678521976 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda74030c10 con 0x7fda8c06dbf0 2024-02-21T23:14:27.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.167+0000 7fda91200640 1 -- 172.21.15.28:0/1678521976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda8c06dbf0 msgr2=0x7fda8c158ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:27.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.167+0000 7fda91200640 1 --2- 172.21.15.28:0/1678521976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda8c06dbf0 0x7fda8c158ac0 secure :-1 s=READY pgs=2474 cs=0 l=1 rev1=1 crypto rx=0x7fda74009870 tx=0x7fda7402b980 comp rx=0 tx=0).stop 2024-02-21T23:14:27.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.168+0000 7fda91200640 1 -- 172.21.15.28:0/1678521976 shutdown_connections 2024-02-21T23:14:27.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.168+0000 7fda91200640 1 --2- 172.21.15.28:0/1678521976 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fda8c155960 0x7fda8c159000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:27.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.168+0000 7fda91200640 1 --2- 172.21.15.28:0/1678521976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda8c06dbf0 0x7fda8c158ac0 unknown :-1 s=CLOSED pgs=2474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:27.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.168+0000 7fda91200640 1 -- 172.21.15.28:0/1678521976 >> 172.21.15.28:0/1678521976 conn(0x7fda8c0ac370 msgr2=0x7fda8c0ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:14:27.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.168+0000 7fda91200640 1 -- 172.21.15.28:0/1678521976 shutdown_connections 2024-02-21T23:14:27.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.168+0000 7fda91200640 1 -- 172.21.15.28:0/1678521976 wait complete. 2024-02-21T23:14:27.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.169+0000 7fda91200640 1 Processor -- start 2024-02-21T23:14:27.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.169+0000 7fda91200640 1 -- start start 2024-02-21T23:14:27.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.170+0000 7fda91200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda8c06dbf0 0x7fda8c0c3e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:27.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.170+0000 7fda91200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fda8c0aff90 0x7fda8c0c2540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:27.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.170+0000 7fda91200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda8c155960 0x7fda8c0c2a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:27.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.170+0000 7fda91200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda8c0c58f0 con 0x7fda8c155960 2024-02-21T23:14:27.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.170+0000 7fda91200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda8c0be0c0 con 0x7fda8c06dbf0 2024-02-21T23:14:27.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.170+0000 7fda91200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda8c0c5a70 con 0x7fda8c0aff90 2024-02-21T23:14:27.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.171+0000 7fda8be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda8c155960 0x7fda8c0c2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:27.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.171+0000 7fda8be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda8c155960 0x7fda8c0c2a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32880/0 (socket says 172.21.15.28:32880) 2024-02-21T23:14:27.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.171+0000 7fda8be00640 1 -- 172.21.15.28:0/424528695 learned_addr learned my addr 172.21.15.28:0/424528695 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:14:27.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.171+0000 7fda82a00640 1 --2- 172.21.15.28:0/424528695 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fda8c0aff90 0x7fda8c0c2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:27.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.171+0000 7fda8b400640 1 --2- 172.21.15.28:0/424528695 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda8c06dbf0 0x7fda8c0c3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:27.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.171+0000 7fda8be00640 1 -- 172.21.15.28:0/424528695 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fda8c0aff90 msgr2=0x7fda8c0c2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:27.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.171+0000 7fda8be00640 1 --2- 172.21.15.28:0/424528695 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fda8c0aff90 0x7fda8c0c2540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:27.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.171+0000 7fda8be00640 1 -- 172.21.15.28:0/424528695 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda8c06dbf0 msgr2=0x7fda8c0c3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:27.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.171+0000 7fda8be00640 1 --2- 172.21.15.28:0/424528695 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda8c06dbf0 0x7fda8c0c3e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:27.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.171+0000 7fda8be00640 1 -- 172.21.15.28:0/424528695 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda8c1eb720 con 0x7fda8c155960 2024-02-21T23:14:27.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.172+0000 7fda8be00640 1 --2- 172.21.15.28:0/424528695 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda8c155960 0x7fda8c0c2a80 secure :-1 s=READY pgs=2475 cs=0 l=1 rev1=1 crypto rx=0x7fda8400d5d0 tx=0x7fda8400da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:27.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.172+0000 7fda83e00640 1 -- 172.21.15.28:0/424528695 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda840125f0 con 0x7fda8c155960 2024-02-21T23:14:27.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.172+0000 7fda83e00640 1 -- 172.21.15.28:0/424528695 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda84019070 con 0x7fda8c155960 2024-02-21T23:14:27.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.172+0000 7fda83e00640 1 -- 172.21.15.28:0/424528695 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda840129f0 con 0x7fda8c155960 2024-02-21T23:14:27.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.172+0000 7fda91200640 1 -- 172.21.15.28:0/424528695 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda8c0c1a60 con 0x7fda8c155960 2024-02-21T23:14:27.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.172+0000 7fda91200640 1 -- 172.21.15.28:0/424528695 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda8c0c2020 con 0x7fda8c155960 2024-02-21T23:14:27.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.173+0000 7fda91200640 1 -- 172.21.15.28:0/424528695 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda8c04f290 con 0x7fda8c155960 2024-02-21T23:14:27.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.175+0000 7fda83e00640 1 -- 172.21.15.28:0/424528695 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fda84012b90 con 0x7fda8c155960 2024-02-21T23:14:27.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.176+0000 7fda83e00640 1 --2- 172.21.15.28:0/424528695 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fda6806b4d0 0x7fda6806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:27.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.177+0000 7fda83e00640 1 -- 172.21.15.28:0/424528695 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fda8408a840 con 0x7fda8c155960 2024-02-21T23:14:27.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.177+0000 7fda8b400640 1 --2- 172.21.15.28:0/424528695 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fda6806b4d0 0x7fda6806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:27.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.178+0000 7fda8b400640 1 --2- 172.21.15.28:0/424528695 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fda6806b4d0 0x7fda6806d990 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fda7c007fd0 tx=0x7fda7c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:27.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.181+0000 7fda83e00640 1 -- 172.21.15.28:0/424528695 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fda84059450 con 0x7fda8c155960 2024-02-21T23:14:27.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.439+0000 7fda91200640 1 -- 172.21.15.28:0/424528695 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fda8c06d9d0 con 0x7fda8c155960 2024-02-21T23:14:27.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.440+0000 7fda83e00640 1 -- 172.21.15.28:0/424528695 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 240 v240) v1 ==== 95+0+4175 (secure 0 0 0) 0x7fda84058d50 con 0x7fda8c155960 2024-02-21T23:14:27.428 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:14:27.428 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":240,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7265,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":224}],"filesystems":[{"mdsmap":{"epoch":240,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:14:11.669846+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":209,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":8287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8287":{"gid":8287,"name":"d","rank":1,"incarnation":235,"state":"up:stopping","state_seq":15,"addr":"172.21.15.81:6835/1607587459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1607587459},{"type":"v1","addr":"172.21.15.81:6835","nonce":1607587459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:14:27.428 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 240 2024-02-21T23:14:27.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.444+0000 7fda91200640 1 -- 172.21.15.28:0/424528695 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fda6806b4d0 msgr2=0x7fda6806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:27.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.444+0000 7fda91200640 1 --2- 172.21.15.28:0/424528695 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fda6806b4d0 0x7fda6806d990 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fda7c007fd0 tx=0x7fda7c009040 comp rx=0 tx=0).stop 2024-02-21T23:14:27.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.444+0000 7fda91200640 1 -- 172.21.15.28:0/424528695 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda8c155960 msgr2=0x7fda8c0c2a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:27.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.444+0000 7fda91200640 1 --2- 172.21.15.28:0/424528695 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda8c155960 0x7fda8c0c2a80 secure :-1 s=READY pgs=2475 cs=0 l=1 rev1=1 crypto rx=0x7fda8400d5d0 tx=0x7fda8400da90 comp rx=0 tx=0).stop 2024-02-21T23:14:27.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.444+0000 7fda91200640 1 -- 172.21.15.28:0/424528695 shutdown_connections 2024-02-21T23:14:27.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.444+0000 7fda91200640 1 --2- 172.21.15.28:0/424528695 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fda6806b4d0 0x7fda6806d990 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:27.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.444+0000 7fda91200640 1 --2- 172.21.15.28:0/424528695 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fda8c0aff90 0x7fda8c0c2540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:27.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.444+0000 7fda91200640 1 --2- 172.21.15.28:0/424528695 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda8c155960 0x7fda8c0c2a80 unknown :-1 s=CLOSED pgs=2475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:27.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.444+0000 7fda91200640 1 --2- 172.21.15.28:0/424528695 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda8c06dbf0 0x7fda8c0c3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:27.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.444+0000 7fda91200640 1 -- 172.21.15.28:0/424528695 >> 172.21.15.28:0/424528695 conn(0x7fda8c0ac370 msgr2=0x7fda8c1571b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:14:27.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.444+0000 7fda91200640 1 -- 172.21.15.28:0/424528695 shutdown_connections 2024-02-21T23:14:27.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:27.444+0000 7fda91200640 1 -- 172.21.15.28:0/424528695 wait complete. 2024-02-21T23:14:27.864 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 240, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:14:11.669846+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 209, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 8287}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8287': {'gid': 8287, 'name': 'd', 'rank': 1, 'incarnation': 235, 'state': 'up:stopping', 'state_seq': 15, 'addr': '172.21.15.81:6835/1607587459', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6834', 'nonce': 1607587459}, {'type': 'v1', 'addr': '172.21.15.81:6835', 'nonce': 1607587459}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:14:27.864 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8287:up:stopping 2024-02-21T23:14:28.865 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:14:29.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.104+0000 7fb84e000640 1 Processor -- start 2024-02-21T23:14:29.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.105+0000 7fb84e000640 1 -- start start 2024-02-21T23:14:29.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.106+0000 7fb84e000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fb848155960 0x7fb8481608a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:29.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.106+0000 7fb84e000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8480c74c0 con 0x7fb84806dbf0 2024-02-21T23:14:29.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.106+0000 7fb84e000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8480c77a0 con 0x7fb8480aff90 2024-02-21T23:14:29.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.106+0000 7fb84e000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8480c7270 con 0x7fb848155960 2024-02-21T23:14:29.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.106+0000 7fb847e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb84806dbf0 0x7fb848158ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:52354/0 (socket says 172.21.15.28:52354) 2024-02-21T23:14:29.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.106+0000 7fb847e00640 1 -- 172.21.15.28:0/1922996680 learned_addr learned my addr 172.21.15.28:0/1922996680 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:14:29.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.107+0000 7fb84d600640 1 --2- 172.21.15.28:0/1922996680 >> v2:172.21.15.81:3300/0 conn(0x7fb848155960 0x7fb8481608a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:29.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.107+0000 7fb847400640 1 -- 172.21.15.28:0/1922996680 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2338891432 0 0) 0x7fb8480c74c0 con 0x7fb84806dbf0 2024-02-21T23:14:29.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.107+0000 7fb847400640 1 -- 172.21.15.28:0/1922996680 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb82c003520 con 0x7fb84806dbf0 2024-02-21T23:14:29.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.107+0000 7fb847400640 1 -- 172.21.15.28:0/1922996680 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1305261501 0 0) 0x7fb82c003520 con 0x7fb84806dbf0 2024-02-21T23:14:29.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.107+0000 7fb847400640 1 -- 172.21.15.28:0/1922996680 >> v2:172.21.15.81:3300/0 conn(0x7fb848155960 msgr2=0x7fb8481608a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:14:29.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.107+0000 7fb847400640 1 --2- 172.21.15.28:0/1922996680 >> v2:172.21.15.81:3300/0 conn(0x7fb848155960 0x7fb8481608a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:29.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.107+0000 7fb847400640 1 -- 172.21.15.28:0/1922996680 >> v1:172.21.15.81:6789/0 conn(0x7fb8480aff90 legacy=0x7fb8480c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:29.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.107+0000 7fb847400640 1 -- 172.21.15.28:0/1922996680 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8480c7a80 con 0x7fb84806dbf0 2024-02-21T23:14:29.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.107+0000 7fb847400640 1 -- 172.21.15.28:0/1922996680 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fb834001670 con 0x7fb84806dbf0 2024-02-21T23:14:29.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.107+0000 7fb847400640 1 -- 172.21.15.28:0/1922996680 >> v1:172.21.15.28:6789/0 conn(0x7fb84806dbf0 legacy=0x7fb848158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:29.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.107+0000 7fb847400640 1 --2- 172.21.15.28:0/1922996680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb82c003a00 0x7fb82c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:29.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.107+0000 7fb847400640 1 --2- 172.21.15.28:0/1922996680 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb82c004800 0x7fb82c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:29.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.107+0000 7fb847400640 1 --2- 172.21.15.28:0/1922996680 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb82c0051c0 0x7fb82c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:29.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.107+0000 7fb847400640 1 -- 172.21.15.28:0/1922996680 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8480c7270 con 0x7fb82c003a00 2024-02-21T23:14:29.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.107+0000 7fb847400640 1 -- 172.21.15.28:0/1922996680 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb82c00c0c0 con 0x7fb82c004800 2024-02-21T23:14:29.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.107+0000 7fb847400640 1 -- 172.21.15.28:0/1922996680 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb82c00c2c0 con 0x7fb82c0051c0 2024-02-21T23:14:29.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.108+0000 7fb84cc00640 1 --2- 172.21.15.28:0/1922996680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb82c003a00 0x7fb82c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:29.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.108+0000 7fb847e00640 1 --2- 172.21.15.28:0/1922996680 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb82c004800 0x7fb82c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:29.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.108+0000 7fb84d600640 1 --2- 172.21.15.28:0/1922996680 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb82c0051c0 0x7fb82c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:29.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.108+0000 7fb84cc00640 1 -- 172.21.15.28:0/1922996680 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb82c0051c0 msgr2=0x7fb82c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:14:29.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.108+0000 7fb84cc00640 1 --2- 172.21.15.28:0/1922996680 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb82c0051c0 0x7fb82c00b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:29.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.108+0000 7fb84cc00640 1 -- 172.21.15.28:0/1922996680 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb82c004800 msgr2=0x7fb82c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:14:29.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.108+0000 7fb84cc00640 1 --2- 172.21.15.28:0/1922996680 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb82c004800 0x7fb82c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:29.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.108+0000 7fb84cc00640 1 -- 172.21.15.28:0/1922996680 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb82c003520 con 0x7fb82c003a00 2024-02-21T23:14:29.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.108+0000 7fb847e00640 1 --2- 172.21.15.28:0/1922996680 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb82c004800 0x7fb82c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:14:29.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.108+0000 7fb84d600640 1 -- 172.21.15.28:0/1922996680 reap_dead start 2024-02-21T23:14:29.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.109+0000 7fb84cc00640 1 --2- 172.21.15.28:0/1922996680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb82c003a00 0x7fb82c003e20 secure :-1 s=READY pgs=2477 cs=0 l=1 rev1=1 crypto rx=0x7fb838001e50 tx=0x7fb82c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68899d35233b43cb server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:29.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.110+0000 7fb847400640 1 -- 172.21.15.28:0/1922996680 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb838001a70 con 0x7fb82c003a00 2024-02-21T23:14:29.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.110+0000 7fb847400640 1 -- 172.21.15.28:0/1922996680 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb838003b70 con 0x7fb82c003a00 2024-02-21T23:14:29.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.110+0000 7fb847400640 1 -- 172.21.15.28:0/1922996680 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8380050d0 con 0x7fb82c003a00 2024-02-21T23:14:29.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.110+0000 7fb84e000640 1 -- 172.21.15.28:0/1922996680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb82c003a00 msgr2=0x7fb82c003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:29.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.110+0000 7fb84e000640 1 --2- 172.21.15.28:0/1922996680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb82c003a00 0x7fb82c003e20 secure :-1 s=READY pgs=2477 cs=0 l=1 rev1=1 crypto rx=0x7fb838001e50 tx=0x7fb82c005830 comp rx=0 tx=0).stop 2024-02-21T23:14:29.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.110+0000 7fb84e000640 1 -- 172.21.15.28:0/1922996680 shutdown_connections 2024-02-21T23:14:29.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.110+0000 7fb84e000640 1 --2- 172.21.15.28:0/1922996680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb82c003a00 0x7fb82c003e20 unknown :-1 s=CLOSED pgs=2477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:29.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.110+0000 7fb84e000640 1 -- 172.21.15.28:0/1922996680 >> 172.21.15.28:0/1922996680 conn(0x7fb8480ac370 msgr2=0x7fb8480ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:14:29.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.110+0000 7fb84e000640 1 -- 172.21.15.28:0/1922996680 shutdown_connections 2024-02-21T23:14:29.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.110+0000 7fb84e000640 1 -- 172.21.15.28:0/1922996680 wait complete. 2024-02-21T23:14:29.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.112+0000 7fb84e000640 1 Processor -- start 2024-02-21T23:14:29.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.112+0000 7fb84e000640 1 -- start start 2024-02-21T23:14:29.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.113+0000 7fb84e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb82c003a00 0x7fb8480ae870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:29.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.113+0000 7fb84e000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb84814fd30 0x7fb84814eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:29.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.113+0000 7fb84e000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb84814f7e0 0x7fb84814f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:29.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.113+0000 7fb84e000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8480c8760 con 0x7fb82c003a00 2024-02-21T23:14:29.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.113+0000 7fb84e000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb848158630 con 0x7fb84814f7e0 2024-02-21T23:14:29.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.113+0000 7fb84e000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8480c88e0 con 0x7fb84814fd30 2024-02-21T23:14:29.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.113+0000 7fb84cc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb82c003a00 0x7fb8480ae870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:29.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.113+0000 7fb84cc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb82c003a00 0x7fb8480ae870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32896/0 (socket says 172.21.15.28:32896) 2024-02-21T23:14:29.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.113+0000 7fb84cc00640 1 -- 172.21.15.28:0/3660521694 learned_addr learned my addr 172.21.15.28:0/3660521694 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:14:29.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.114+0000 7fb847e00640 1 --2- 172.21.15.28:0/3660521694 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb84814fd30 0x7fb84814eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:29.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.114+0000 7fb84cc00640 1 -- 172.21.15.28:0/3660521694 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb84814fd30 msgr2=0x7fb84814eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:29.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.114+0000 7fb84cc00640 1 --2- 172.21.15.28:0/3660521694 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb84814fd30 0x7fb84814eb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:29.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.114+0000 7fb84cc00640 1 -- 172.21.15.28:0/3660521694 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb84814f7e0 msgr2=0x7fb84814f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:29.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.114+0000 7fb84cc00640 1 --2- 172.21.15.28:0/3660521694 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb84814f7e0 0x7fb84814f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:29.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.114+0000 7fb84cc00640 1 -- 172.21.15.28:0/3660521694 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8480c2100 con 0x7fb82c003a00 2024-02-21T23:14:29.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.114+0000 7fb84cc00640 1 --2- 172.21.15.28:0/3660521694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb82c003a00 0x7fb8480ae870 secure :-1 s=READY pgs=2478 cs=0 l=1 rev1=1 crypto rx=0x7fb83802c040 tx=0x7fb8380014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:29.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.114+0000 7fb844c00640 1 -- 172.21.15.28:0/3660521694 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb838007db0 con 0x7fb82c003a00 2024-02-21T23:14:29.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.114+0000 7fb844c00640 1 -- 172.21.15.28:0/3660521694 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb838003970 con 0x7fb82c003a00 2024-02-21T23:14:29.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.114+0000 7fb84e000640 1 -- 172.21.15.28:0/3660521694 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb84814f5a0 con 0x7fb82c003a00 2024-02-21T23:14:29.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.115+0000 7fb844c00640 1 -- 172.21.15.28:0/3660521694 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb838006d50 con 0x7fb82c003a00 2024-02-21T23:14:29.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.115+0000 7fb84e000640 1 -- 172.21.15.28:0/3660521694 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8481eb3f0 con 0x7fb82c003a00 2024-02-21T23:14:29.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.116+0000 7fb84e000640 1 -- 172.21.15.28:0/3660521694 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb84804f290 con 0x7fb82c003a00 2024-02-21T23:14:29.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.118+0000 7fb844c00640 1 -- 172.21.15.28:0/3660521694 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb838007030 con 0x7fb82c003a00 2024-02-21T23:14:29.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.119+0000 7fb844c00640 1 --2- 172.21.15.28:0/3660521694 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb82806b500 0x7fb82806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:29.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.119+0000 7fb844c00640 1 -- 172.21.15.28:0/3660521694 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(211..211 src has 1..211) v4 ==== 6274+0+0 (secure 0 0 0) 0x7fb8380abf30 con 0x7fb82c003a00 2024-02-21T23:14:29.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.119+0000 7fb847e00640 1 --2- 172.21.15.28:0/3660521694 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb82806b500 0x7fb82806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:29.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.120+0000 7fb847e00640 1 --2- 172.21.15.28:0/3660521694 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb82806b500 0x7fb82806d9c0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fb8340017e0 tx=0x7fb834009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:29.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.124+0000 7fb844c00640 1 -- 172.21.15.28:0/3660521694 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb838033020 con 0x7fb82c003a00 2024-02-21T23:14:29.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.368+0000 7fb84e000640 1 -- 172.21.15.28:0/3660521694 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb84806d9d0 con 0x7fb82c003a00 2024-02-21T23:14:29.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.369+0000 7fb844c00640 1 -- 172.21.15.28:0/3660521694 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 240 v240) v1 ==== 95+0+4175 (secure 0 0 0) 0x7fb83807ab60 con 0x7fb82c003a00 2024-02-21T23:14:29.357 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:14:29.357 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":240,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7265,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":224}],"filesystems":[{"mdsmap":{"epoch":240,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:14:11.669846+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":209,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":8287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8287":{"gid":8287,"name":"d","rank":1,"incarnation":235,"state":"up:stopping","state_seq":15,"addr":"172.21.15.81:6835/1607587459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1607587459},{"type":"v1","addr":"172.21.15.81:6835","nonce":1607587459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:14:29.358 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 240 2024-02-21T23:14:29.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.372+0000 7fb84e000640 1 -- 172.21.15.28:0/3660521694 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb82806b500 msgr2=0x7fb82806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:29.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.372+0000 7fb84e000640 1 --2- 172.21.15.28:0/3660521694 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb82806b500 0x7fb82806d9c0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fb8340017e0 tx=0x7fb834009040 comp rx=0 tx=0).stop 2024-02-21T23:14:29.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.372+0000 7fb84e000640 1 -- 172.21.15.28:0/3660521694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb82c003a00 msgr2=0x7fb8480ae870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:29.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.372+0000 7fb84e000640 1 --2- 172.21.15.28:0/3660521694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb82c003a00 0x7fb8480ae870 secure :-1 s=READY pgs=2478 cs=0 l=1 rev1=1 crypto rx=0x7fb83802c040 tx=0x7fb8380014a0 comp rx=0 tx=0).stop 2024-02-21T23:14:29.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.372+0000 7fb84e000640 1 -- 172.21.15.28:0/3660521694 shutdown_connections 2024-02-21T23:14:29.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.372+0000 7fb84e000640 1 --2- 172.21.15.28:0/3660521694 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb82806b500 0x7fb82806d9c0 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:29.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.372+0000 7fb84e000640 1 --2- 172.21.15.28:0/3660521694 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb84814fd30 0x7fb84814eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:29.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.372+0000 7fb84e000640 1 --2- 172.21.15.28:0/3660521694 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb84814f7e0 0x7fb84814f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:29.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.372+0000 7fb84e000640 1 --2- 172.21.15.28:0/3660521694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb82c003a00 0x7fb8480ae870 unknown :-1 s=CLOSED pgs=2478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:29.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.372+0000 7fb84e000640 1 -- 172.21.15.28:0/3660521694 >> 172.21.15.28:0/3660521694 conn(0x7fb8480ac370 msgr2=0x7fb848156e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:14:29.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.373+0000 7fb84e000640 1 -- 172.21.15.28:0/3660521694 shutdown_connections 2024-02-21T23:14:29.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:29.373+0000 7fb84e000640 1 -- 172.21.15.28:0/3660521694 wait complete. 2024-02-21T23:14:29.784 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 240, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:14:11.669846+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 209, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 8287}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8287': {'gid': 8287, 'name': 'd', 'rank': 1, 'incarnation': 235, 'state': 'up:stopping', 'state_seq': 15, 'addr': '172.21.15.81:6835/1607587459', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6834', 'nonce': 1607587459}, {'type': 'v1', 'addr': '172.21.15.81:6835', 'nonce': 1607587459}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:14:29.784 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8287:up:stopping 2024-02-21T23:14:30.785 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:14:31.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.019+0000 7f342f200640 1 Processor -- start 2024-02-21T23:14:31.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.020+0000 7f342f200640 1 -- start start 2024-02-21T23:14:31.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.021+0000 7f342f200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f34300aff90 0x7f34300c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:31.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.021+0000 7f342f200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f343006dbf0 0x7f3430158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:31.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.021+0000 7f342f200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f34300c5720 0x7f3430159000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:31.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.021+0000 7f342f200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34300c7270 con 0x7f34300c5720 2024-02-21T23:14:31.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.021+0000 7f342f200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3430158630 con 0x7f34300aff90 2024-02-21T23:14:31.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.021+0000 7f342f200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34300c74c0 con 0x7f343006dbf0 2024-02-21T23:14:31.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.022+0000 7f342e800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f34300c5720 0x7f3430159000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:31.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.022+0000 7f342e800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f34300c5720 0x7f3430159000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32906/0 (socket says 172.21.15.28:32906) 2024-02-21T23:14:31.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.022+0000 7f342e800640 1 -- 172.21.15.28:0/4219829880 learned_addr learned my addr 172.21.15.28:0/4219829880 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:14:31.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.022+0000 7f342d400640 1 --2- 172.21.15.28:0/4219829880 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f343006dbf0 0x7f3430158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:31.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.022+0000 7f342de00640 1 --2- 172.21.15.28:0/4219829880 >> v2:172.21.15.81:3300/0 conn(0x7f34300aff90 0x7f34300c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:31.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.022+0000 7f342e800640 1 -- 172.21.15.28:0/4219829880 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f343006dbf0 msgr2=0x7f3430158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:14:31.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.022+0000 7f342e800640 1 --2- 172.21.15.28:0/4219829880 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f343006dbf0 0x7f3430158ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:31.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.022+0000 7f342e800640 1 -- 172.21.15.28:0/4219829880 >> v2:172.21.15.81:3300/0 conn(0x7f34300aff90 msgr2=0x7f34300c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:14:31.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.022+0000 7f342e800640 1 --2- 172.21.15.28:0/4219829880 >> v2:172.21.15.81:3300/0 conn(0x7f34300aff90 0x7f34300c51e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:31.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.022+0000 7f342e800640 1 -- 172.21.15.28:0/4219829880 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34300c77a0 con 0x7f34300c5720 2024-02-21T23:14:31.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.022+0000 7f342d400640 1 --2- 172.21.15.28:0/4219829880 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f343006dbf0 0x7f3430158ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:14:31.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.023+0000 7f342e800640 1 --2- 172.21.15.28:0/4219829880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34300c5720 0x7f3430159000 secure :-1 s=READY pgs=2479 cs=0 l=1 rev1=1 crypto rx=0x7f34300aa1b0 tx=0x7f342802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8c85e96f2566df9 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:31.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.024+0000 7f342ca00640 1 -- 172.21.15.28:0/4219829880 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f342802ca80 con 0x7f34300c5720 2024-02-21T23:14:31.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.024+0000 7f342ca00640 1 -- 172.21.15.28:0/4219829880 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f342802cc20 con 0x7f34300c5720 2024-02-21T23:14:31.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.024+0000 7f342ca00640 1 -- 172.21.15.28:0/4219829880 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34280343f0 con 0x7f34300c5720 2024-02-21T23:14:31.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.024+0000 7f342f200640 1 -- 172.21.15.28:0/4219829880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34300c5720 msgr2=0x7f3430159000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:31.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.024+0000 7f342f200640 1 --2- 172.21.15.28:0/4219829880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34300c5720 0x7f3430159000 secure :-1 s=READY pgs=2479 cs=0 l=1 rev1=1 crypto rx=0x7f34300aa1b0 tx=0x7f342802bee0 comp rx=0 tx=0).stop 2024-02-21T23:14:31.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.025+0000 7f342f200640 1 -- 172.21.15.28:0/4219829880 shutdown_connections 2024-02-21T23:14:31.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.025+0000 7f342f200640 1 --2- 172.21.15.28:0/4219829880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34300c5720 0x7f3430159000 unknown :-1 s=CLOSED pgs=2479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:31.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.025+0000 7f342f200640 1 --2- 172.21.15.28:0/4219829880 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f343006dbf0 0x7f3430158ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:31.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.025+0000 7f342f200640 1 --2- 172.21.15.28:0/4219829880 >> v2:172.21.15.81:3300/0 conn(0x7f34300aff90 0x7f34300c51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:31.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.025+0000 7f342f200640 1 -- 172.21.15.28:0/4219829880 >> 172.21.15.28:0/4219829880 conn(0x7f34300ac370 msgr2=0x7f34300ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:14:31.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.025+0000 7f342f200640 1 -- 172.21.15.28:0/4219829880 shutdown_connections 2024-02-21T23:14:31.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.025+0000 7f342f200640 1 -- 172.21.15.28:0/4219829880 wait complete. 2024-02-21T23:14:31.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.027+0000 7f342f200640 1 Processor -- start 2024-02-21T23:14:31.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.027+0000 7f342f200640 1 -- start start 2024-02-21T23:14:31.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.028+0000 7f342f200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f343006dbf0 0x7f34301ead70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:31.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.028+0000 7f342f200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f34300aff90 0x7f3430147ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:31.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.028+0000 7f342f200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f34300c5720 0x7f3430147ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:31.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.028+0000 7f342f200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34300c8760 con 0x7f343006dbf0 2024-02-21T23:14:31.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.028+0000 7f342f200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34300be0c0 con 0x7f34300c5720 2024-02-21T23:14:31.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.028+0000 7f342f200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34300c88e0 con 0x7f34300aff90 2024-02-21T23:14:31.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.029+0000 7f342de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f343006dbf0 0x7f34301ead70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:31.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.029+0000 7f342de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f343006dbf0 0x7f34301ead70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32910/0 (socket says 172.21.15.28:32910) 2024-02-21T23:14:31.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.029+0000 7f342de00640 1 -- 172.21.15.28:0/1613938263 learned_addr learned my addr 172.21.15.28:0/1613938263 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:14:31.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.029+0000 7f342d400640 1 --2- 172.21.15.28:0/1613938263 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f34300aff90 0x7f3430147ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:31.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.029+0000 7f342e800640 1 --2- 172.21.15.28:0/1613938263 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f34300c5720 0x7f3430147ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:31.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.029+0000 7f342de00640 1 -- 172.21.15.28:0/1613938263 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f34300aff90 msgr2=0x7f3430147ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:31.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.029+0000 7f342de00640 1 --2- 172.21.15.28:0/1613938263 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f34300aff90 0x7f3430147ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:31.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.029+0000 7f342de00640 1 -- 172.21.15.28:0/1613938263 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f34300c5720 msgr2=0x7f3430147ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:31.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.029+0000 7f342de00640 1 --2- 172.21.15.28:0/1613938263 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f34300c5720 0x7f3430147ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:31.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.029+0000 7f342de00640 1 -- 172.21.15.28:0/1613938263 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34300c3ba0 con 0x7f343006dbf0 2024-02-21T23:14:31.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.030+0000 7f342de00640 1 --2- 172.21.15.28:0/1613938263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f343006dbf0 0x7f34301ead70 secure :-1 s=READY pgs=2480 cs=0 l=1 rev1=1 crypto rx=0x7f342000c820 tx=0x7f342000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:31.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.030+0000 7f3426000640 1 -- 172.21.15.28:0/1613938263 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3420015070 con 0x7f343006dbf0 2024-02-21T23:14:31.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.030+0000 7f3426000640 1 -- 172.21.15.28:0/1613938263 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3420004240 con 0x7f343006dbf0 2024-02-21T23:14:31.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.030+0000 7f3426000640 1 -- 172.21.15.28:0/1613938263 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f342000ce10 con 0x7f343006dbf0 2024-02-21T23:14:31.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.030+0000 7f342f200640 1 -- 172.21.15.28:0/1613938263 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34300c3e80 con 0x7f343006dbf0 2024-02-21T23:14:31.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.030+0000 7f342f200640 1 -- 172.21.15.28:0/1613938263 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34300c41b0 con 0x7f343006dbf0 2024-02-21T23:14:31.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.031+0000 7f342f200640 1 -- 172.21.15.28:0/1613938263 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f343004f290 con 0x7f343006dbf0 2024-02-21T23:14:31.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.032+0000 7f3426000640 1 -- 172.21.15.28:0/1613938263 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3420007830 con 0x7f343006dbf0 2024-02-21T23:14:31.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.034+0000 7f3426000640 1 --2- 172.21.15.28:0/1613938263 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f341006b4d0 0x7f341006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:31.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.034+0000 7f3426000640 1 -- 172.21.15.28:0/1613938263 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(211..211 src has 1..211) v4 ==== 6274+0+0 (secure 0 0 0) 0x7f342008a330 con 0x7f343006dbf0 2024-02-21T23:14:31.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.034+0000 7f342d400640 1 --2- 172.21.15.28:0/1613938263 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f341006b4d0 0x7f341006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:31.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.035+0000 7f342d400640 1 --2- 172.21.15.28:0/1613938263 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f341006b4d0 0x7f341006d990 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f341c006fd0 tx=0x7f341c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:31.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.039+0000 7f3426000640 1 -- 172.21.15.28:0/1613938263 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3420058f60 con 0x7f343006dbf0 2024-02-21T23:14:31.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.279+0000 7f342f200640 1 -- 172.21.15.28:0/1613938263 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3430156bb0 con 0x7f343006dbf0 2024-02-21T23:14:31.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.280+0000 7f3426000640 1 -- 172.21.15.28:0/1613938263 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 240 v240) v1 ==== 95+0+4175 (secure 0 0 0) 0x7f3420058860 con 0x7f343006dbf0 2024-02-21T23:14:31.267 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:14:31.267 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":240,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7265,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":224}],"filesystems":[{"mdsmap":{"epoch":240,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:14:11.669846+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":209,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":8287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8287":{"gid":8287,"name":"d","rank":1,"incarnation":235,"state":"up:stopping","state_seq":15,"addr":"172.21.15.81:6835/1607587459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1607587459},{"type":"v1","addr":"172.21.15.81:6835","nonce":1607587459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:14:31.268 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 240 2024-02-21T23:14:31.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.283+0000 7f342f200640 1 -- 172.21.15.28:0/1613938263 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f341006b4d0 msgr2=0x7f341006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:31.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.283+0000 7f342f200640 1 --2- 172.21.15.28:0/1613938263 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f341006b4d0 0x7f341006d990 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f341c006fd0 tx=0x7f341c008040 comp rx=0 tx=0).stop 2024-02-21T23:14:31.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.283+0000 7f342f200640 1 -- 172.21.15.28:0/1613938263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f343006dbf0 msgr2=0x7f34301ead70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:31.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.283+0000 7f342f200640 1 --2- 172.21.15.28:0/1613938263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f343006dbf0 0x7f34301ead70 secure :-1 s=READY pgs=2480 cs=0 l=1 rev1=1 crypto rx=0x7f342000c820 tx=0x7f342000cce0 comp rx=0 tx=0).stop 2024-02-21T23:14:31.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.283+0000 7f342f200640 1 -- 172.21.15.28:0/1613938263 shutdown_connections 2024-02-21T23:14:31.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.283+0000 7f342f200640 1 --2- 172.21.15.28:0/1613938263 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f341006b4d0 0x7f341006d990 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:31.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.283+0000 7f342f200640 1 --2- 172.21.15.28:0/1613938263 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f34300aff90 0x7f3430147ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:31.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.283+0000 7f342f200640 1 --2- 172.21.15.28:0/1613938263 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f34300c5720 0x7f3430147ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:31.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.283+0000 7f342f200640 1 --2- 172.21.15.28:0/1613938263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f343006dbf0 0x7f34301ead70 unknown :-1 s=CLOSED pgs=2480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:31.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.283+0000 7f342f200640 1 -- 172.21.15.28:0/1613938263 >> 172.21.15.28:0/1613938263 conn(0x7f34300ac370 msgr2=0x7f34300ac750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:14:31.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.283+0000 7f342f200640 1 -- 172.21.15.28:0/1613938263 shutdown_connections 2024-02-21T23:14:31.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:31.283+0000 7f342f200640 1 -- 172.21.15.28:0/1613938263 wait complete. 2024-02-21T23:14:31.693 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 240, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:14:11.669846+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 209, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 8287}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8287': {'gid': 8287, 'name': 'd', 'rank': 1, 'incarnation': 235, 'state': 'up:stopping', 'state_seq': 15, 'addr': '172.21.15.81:6835/1607587459', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6834', 'nonce': 1607587459}, {'type': 'v1', 'addr': '172.21.15.81:6835', 'nonce': 1607587459}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:14:31.693 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8287:up:stopping 2024-02-21T23:14:32.694 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:14:32.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.944+0000 7f65c5200640 1 Processor -- start 2024-02-21T23:14:32.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.944+0000 7f65c5200640 1 -- start start 2024-02-21T23:14:32.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.946+0000 7f65c5200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f65c00a5fd0 0x7f65c006dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:32.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.947+0000 7f65c5200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65c00c2760 con 0x7f65c0150450 2024-02-21T23:14:32.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.947+0000 7f65c5200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65c0156310 con 0x7f65c006ee40 2024-02-21T23:14:32.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.947+0000 7f65c5200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65c00c22c0 con 0x7f65c00a5fd0 2024-02-21T23:14:32.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.947+0000 7f65bf400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f65c0150450 0x7f65c01536f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:52356/0 (socket says 172.21.15.28:52356) 2024-02-21T23:14:32.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.947+0000 7f65bf400640 1 -- 172.21.15.28:0/272065070 learned_addr learned my addr 172.21.15.28:0/272065070 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:14:32.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.947+0000 7f65bfe00640 1 --2- 172.21.15.28:0/272065070 >> v2:172.21.15.81:3300/0 conn(0x7f65c00a5fd0 0x7f65c006dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:32.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.947+0000 7f65bea00640 1 -- 172.21.15.28:0/272065070 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2293503657 0 0) 0x7f65c00c2760 con 0x7f65c0150450 2024-02-21T23:14:32.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.947+0000 7f65bea00640 1 -- 172.21.15.28:0/272065070 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65a4003520 con 0x7f65c0150450 2024-02-21T23:14:32.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.948+0000 7f65bea00640 1 -- 172.21.15.28:0/272065070 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3520322659 0 0) 0x7f65a4003520 con 0x7f65c0150450 2024-02-21T23:14:32.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.948+0000 7f65bea00640 1 -- 172.21.15.28:0/272065070 >> v2:172.21.15.81:3300/0 conn(0x7f65c00a5fd0 msgr2=0x7f65c006dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:14:32.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.948+0000 7f65bea00640 1 --2- 172.21.15.28:0/272065070 >> v2:172.21.15.81:3300/0 conn(0x7f65c00a5fd0 0x7f65c006dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:32.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.948+0000 7f65bea00640 1 -- 172.21.15.28:0/272065070 >> v1:172.21.15.81:6789/0 conn(0x7f65c006ee40 legacy=0x7f65c0156af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:32.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.948+0000 7f65bea00640 1 -- 172.21.15.28:0/272065070 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65c00c29d0 con 0x7f65c0150450 2024-02-21T23:14:32.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.948+0000 7f65bea00640 1 -- 172.21.15.28:0/272065070 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f65b0001670 con 0x7f65c0150450 2024-02-21T23:14:32.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.948+0000 7f65bea00640 1 -- 172.21.15.28:0/272065070 >> v1:172.21.15.28:6789/0 conn(0x7f65c0150450 legacy=0x7f65c01536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:32.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.948+0000 7f65bea00640 1 --2- 172.21.15.28:0/272065070 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f65a4003a00 0x7f65a4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:32.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.948+0000 7f65bea00640 1 --2- 172.21.15.28:0/272065070 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65a4004800 0x7f65a4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:32.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.948+0000 7f65bea00640 1 --2- 172.21.15.28:0/272065070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65a40051c0 0x7f65a400fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:32.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.948+0000 7f65bea00640 1 -- 172.21.15.28:0/272065070 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65a40104a0 con 0x7f65a40051c0 2024-02-21T23:14:32.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.948+0000 7f65bea00640 1 -- 172.21.15.28:0/272065070 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65a4010650 con 0x7f65a4004800 2024-02-21T23:14:32.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.948+0000 7f65bea00640 1 -- 172.21.15.28:0/272065070 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65a4010850 con 0x7f65a4003a00 2024-02-21T23:14:32.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.948+0000 7f65bfe00640 1 --2- 172.21.15.28:0/272065070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65a40051c0 0x7f65a400fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:32.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.949+0000 7f65bf400640 1 --2- 172.21.15.28:0/272065070 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f65a4003a00 0x7f65a4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:32.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.949+0000 7f65b7e00640 1 --2- 172.21.15.28:0/272065070 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65a4004800 0x7f65a4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:32.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.949+0000 7f65bfe00640 1 -- 172.21.15.28:0/272065070 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f65a4003a00 msgr2=0x7f65a4003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:14:32.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.949+0000 7f65bfe00640 1 --2- 172.21.15.28:0/272065070 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f65a4003a00 0x7f65a4003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:32.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.949+0000 7f65bfe00640 1 -- 172.21.15.28:0/272065070 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65a4004800 msgr2=0x7f65a4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:14:32.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.949+0000 7f65bfe00640 1 --2- 172.21.15.28:0/272065070 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65a4004800 0x7f65a4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:32.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.949+0000 7f65bfe00640 1 -- 172.21.15.28:0/272065070 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65a4003520 con 0x7f65a40051c0 2024-02-21T23:14:32.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.949+0000 7f65bf400640 1 --2- 172.21.15.28:0/272065070 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f65a4003a00 0x7f65a4003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:14:32.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.950+0000 7f65bfe00640 1 -- 172.21.15.28:0/272065070 reap_dead start 2024-02-21T23:14:32.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.950+0000 7f65bfe00640 1 --2- 172.21.15.28:0/272065070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65a40051c0 0x7f65a400fdd0 secure :-1 s=READY pgs=2482 cs=0 l=1 rev1=1 crypto rx=0x7f65b8009fc0 tx=0x7f65a4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=501b0ece00670a50 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:32.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.950+0000 7f65bea00640 1 -- 172.21.15.28:0/272065070 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b8009c10 con 0x7f65a40051c0 2024-02-21T23:14:32.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.950+0000 7f65bea00640 1 -- 172.21.15.28:0/272065070 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65b802cca0 con 0x7f65a40051c0 2024-02-21T23:14:32.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.950+0000 7f65bea00640 1 -- 172.21.15.28:0/272065070 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b8003bd0 con 0x7f65a40051c0 2024-02-21T23:14:32.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.951+0000 7f65c5200640 1 -- 172.21.15.28:0/272065070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65a40051c0 msgr2=0x7f65a400fdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:32.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.951+0000 7f65c5200640 1 --2- 172.21.15.28:0/272065070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65a40051c0 0x7f65a400fdd0 secure :-1 s=READY pgs=2482 cs=0 l=1 rev1=1 crypto rx=0x7f65b8009fc0 tx=0x7f65a4005830 comp rx=0 tx=0).stop 2024-02-21T23:14:32.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.951+0000 7f65c5200640 1 -- 172.21.15.28:0/272065070 shutdown_connections 2024-02-21T23:14:32.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.951+0000 7f65c5200640 1 --2- 172.21.15.28:0/272065070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65a40051c0 0x7f65a400fdd0 unknown :-1 s=CLOSED pgs=2482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:32.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.951+0000 7f65c5200640 1 -- 172.21.15.28:0/272065070 >> 172.21.15.28:0/272065070 conn(0x7f65c006b630 msgr2=0x7f65c00c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:14:32.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.951+0000 7f65c5200640 1 -- 172.21.15.28:0/272065070 shutdown_connections 2024-02-21T23:14:32.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.951+0000 7f65c5200640 1 -- 172.21.15.28:0/272065070 wait complete. 2024-02-21T23:14:32.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.953+0000 7f65c5200640 1 Processor -- start 2024-02-21T23:14:32.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.953+0000 7f65c5200640 1 -- start start 2024-02-21T23:14:32.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.954+0000 7f65c5200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65a40051c0 0x7f65c01e9c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:32.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.954+0000 7f65c5200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65c01495b0 0x7f65c0148870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:32.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.954+0000 7f65c5200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f65c0149ae0 0x7f65c0148db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:32.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.954+0000 7f65c5200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65c01536f0 con 0x7f65a40051c0 2024-02-21T23:14:32.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.954+0000 7f65c5200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65c0156660 con 0x7f65c01495b0 2024-02-21T23:14:32.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.954+0000 7f65c5200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65c0153870 con 0x7f65c0149ae0 2024-02-21T23:14:32.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.954+0000 7f65bf400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65a40051c0 0x7f65c01e9c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:32.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.954+0000 7f65bf400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65a40051c0 0x7f65c01e9c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32936/0 (socket says 172.21.15.28:32936) 2024-02-21T23:14:32.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.954+0000 7f65bf400640 1 -- 172.21.15.28:0/2404664041 learned_addr learned my addr 172.21.15.28:0/2404664041 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:14:32.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.955+0000 7f65bf400640 1 -- 172.21.15.28:0/2404664041 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f65c0149ae0 msgr2=0x7f65c0148db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:32.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.955+0000 7f65b7e00640 1 --2- 172.21.15.28:0/2404664041 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65c01495b0 0x7f65c0148870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:32.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.955+0000 7f65bf400640 1 --2- 172.21.15.28:0/2404664041 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f65c0149ae0 0x7f65c0148db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:32.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.955+0000 7f65bf400640 1 -- 172.21.15.28:0/2404664041 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65c01495b0 msgr2=0x7f65c0148870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:32.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.955+0000 7f65bf400640 1 --2- 172.21.15.28:0/2404664041 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65c01495b0 0x7f65c0148870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:32.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.955+0000 7f65bf400640 1 -- 172.21.15.28:0/2404664041 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65c01ea4a0 con 0x7f65a40051c0 2024-02-21T23:14:32.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.955+0000 7f65bf400640 1 --2- 172.21.15.28:0/2404664041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65a40051c0 0x7f65c01e9c60 secure :-1 s=READY pgs=2483 cs=0 l=1 rev1=1 crypto rx=0x7f65b000e780 tx=0x7f65b000ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:32.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.956+0000 7f65b7400640 1 -- 172.21.15.28:0/2404664041 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b000c660 con 0x7f65a40051c0 2024-02-21T23:14:32.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.956+0000 7f65b7400640 1 -- 172.21.15.28:0/2404664041 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65b0018070 con 0x7f65a40051c0 2024-02-21T23:14:32.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.956+0000 7f65b7400640 1 -- 172.21.15.28:0/2404664041 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b000ca00 con 0x7f65a40051c0 2024-02-21T23:14:32.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.956+0000 7f65c5200640 1 -- 172.21.15.28:0/2404664041 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65c01ea780 con 0x7f65a40051c0 2024-02-21T23:14:32.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.956+0000 7f65c5200640 1 -- 172.21.15.28:0/2404664041 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65c01ea1a0 con 0x7f65a40051c0 2024-02-21T23:14:32.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.957+0000 7f65c5200640 1 -- 172.21.15.28:0/2404664041 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65c004f290 con 0x7f65a40051c0 2024-02-21T23:14:32.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.958+0000 7f65b7400640 1 -- 172.21.15.28:0/2404664041 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f65b000cba0 con 0x7f65a40051c0 2024-02-21T23:14:32.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.959+0000 7f65b7400640 1 --2- 172.21.15.28:0/2404664041 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f659c06b500 0x7f659c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:32.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.959+0000 7f65b7400640 1 -- 172.21.15.28:0/2404664041 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(211..211 src has 1..211) v4 ==== 6274+0+0 (secure 0 0 0) 0x7f65b008a140 con 0x7f65a40051c0 2024-02-21T23:14:32.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.960+0000 7f65b7e00640 1 --2- 172.21.15.28:0/2404664041 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f659c06b500 0x7f659c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:32.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.961+0000 7f65b7e00640 1 --2- 172.21.15.28:0/2404664041 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f659c06b500 0x7f659c06d9c0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f65a8007fd0 tx=0x7f65a8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:32.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:32.964+0000 7f65b7400640 1 -- 172.21.15.28:0/2404664041 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f65b0058e10 con 0x7f65a40051c0 2024-02-21T23:14:33.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:33.279+0000 7f65c5200640 1 -- 172.21.15.28:0/2404664041 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f65c00c1f10 con 0x7f65a40051c0 2024-02-21T23:14:33.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:33.280+0000 7f65b7400640 1 -- 172.21.15.28:0/2404664041 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 240 v240) v1 ==== 95+0+4175 (secure 0 0 0) 0x7f65b0058710 con 0x7f65a40051c0 2024-02-21T23:14:33.267 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:14:33.268 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":240,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7265,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":224}],"filesystems":[{"mdsmap":{"epoch":240,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:14:11.669846+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":209,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":8287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8287":{"gid":8287,"name":"d","rank":1,"incarnation":235,"state":"up:stopping","state_seq":15,"addr":"172.21.15.81:6835/1607587459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1607587459},{"type":"v1","addr":"172.21.15.81:6835","nonce":1607587459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:14:33.268 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 240 2024-02-21T23:14:33.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:33.283+0000 7f65c5200640 1 -- 172.21.15.28:0/2404664041 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f659c06b500 msgr2=0x7f659c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:33.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:33.283+0000 7f65c5200640 1 --2- 172.21.15.28:0/2404664041 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f659c06b500 0x7f659c06d9c0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f65a8007fd0 tx=0x7f65a8009040 comp rx=0 tx=0).stop 2024-02-21T23:14:33.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:33.283+0000 7f65c5200640 1 -- 172.21.15.28:0/2404664041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65a40051c0 msgr2=0x7f65c01e9c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:33.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:33.283+0000 7f65c5200640 1 --2- 172.21.15.28:0/2404664041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65a40051c0 0x7f65c01e9c60 secure :-1 s=READY pgs=2483 cs=0 l=1 rev1=1 crypto rx=0x7f65b000e780 tx=0x7f65b000ec40 comp rx=0 tx=0).stop 2024-02-21T23:14:33.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:33.283+0000 7f65c5200640 1 -- 172.21.15.28:0/2404664041 shutdown_connections 2024-02-21T23:14:33.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:33.283+0000 7f65c5200640 1 --2- 172.21.15.28:0/2404664041 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f659c06b500 0x7f659c06d9c0 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:33.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:33.283+0000 7f65c5200640 1 --2- 172.21.15.28:0/2404664041 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f65c0149ae0 0x7f65c0148db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:33.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:33.283+0000 7f65c5200640 1 --2- 172.21.15.28:0/2404664041 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f65c01495b0 0x7f65c0148870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:33.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:33.283+0000 7f65c5200640 1 --2- 172.21.15.28:0/2404664041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65a40051c0 0x7f65c01e9c60 unknown :-1 s=CLOSED pgs=2483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:33.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:33.283+0000 7f65c5200640 1 -- 172.21.15.28:0/2404664041 >> 172.21.15.28:0/2404664041 conn(0x7f65c006b630 msgr2=0x7f65c00c2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:14:33.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:33.283+0000 7f65c5200640 1 -- 172.21.15.28:0/2404664041 shutdown_connections 2024-02-21T23:14:33.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:33.283+0000 7f65c5200640 1 -- 172.21.15.28:0/2404664041 wait complete. 2024-02-21T23:14:33.692 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 240, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:14:11.669846+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 209, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 8287}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8287': {'gid': 8287, 'name': 'd', 'rank': 1, 'incarnation': 235, 'state': 'up:stopping', 'state_seq': 15, 'addr': '172.21.15.81:6835/1607587459', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6834', 'nonce': 1607587459}, {'type': 'v1', 'addr': '172.21.15.81:6835', 'nonce': 1607587459}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:14:33.692 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8287:up:stopping 2024-02-21T23:14:34.693 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:14:34.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:34.934+0000 7fee82a00640 1 Processor -- start 2024-02-21T23:14:34.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:34.935+0000 7fee82a00640 1 -- start start 2024-02-21T23:14:34.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:34.936+0000 7fee82a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fee7c06e3d0 0x7fee7c14b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:34.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:34.936+0000 7fee82a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee7c0a8b40 con 0x7fee7c0c0fe0 2024-02-21T23:14:34.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:34.936+0000 7fee82a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee7c06e7d0 con 0x7fee7c06dff0 2024-02-21T23:14:34.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:34.936+0000 7fee82a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee7c0a8e00 con 0x7fee7c06e3d0 2024-02-21T23:14:34.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:34.936+0000 7fee82000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fee7c06e3d0 0x7fee7c14b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:34.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:34.936+0000 7fee82000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fee7c06e3d0 0x7fee7c14b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41910/0 (socket says 172.21.15.28:41910) 2024-02-21T23:14:34.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:34.936+0000 7fee82000640 1 -- 172.21.15.28:0/3257685972 learned_addr learned my addr 172.21.15.28:0/3257685972 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:14:34.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:34.937+0000 7fee82000640 1 -- 172.21.15.28:0/3257685972 >> v1:172.21.15.81:6789/0 conn(0x7fee7c06dff0 legacy=0x7fee7c071350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:14:34.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:34.937+0000 7fee82000640 1 -- 172.21.15.28:0/3257685972 >> v1:172.21.15.28:6789/0 conn(0x7fee7c0c0fe0 legacy=0x7fee7c158580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:34.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:34.937+0000 7fee82000640 1 -- 172.21.15.28:0/3257685972 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee7c0a90e0 con 0x7fee7c06e3d0 2024-02-21T23:14:34.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:34.938+0000 7fee82000640 1 --2- 172.21.15.28:0/3257685972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee7c06e3d0 0x7fee7c14b2b0 secure :-1 s=READY pgs=2485 cs=0 l=1 rev1=1 crypto rx=0x7fee74009e50 tx=0x7fee7402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c0aee7cd3349e34 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:34.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:34.938+0000 7fee7be00640 1 -- 172.21.15.28:0/3257685972 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee7402cb20 con 0x7fee7c06e3d0 2024-02-21T23:14:34.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:34.938+0000 7fee7be00640 1 -- 172.21.15.28:0/3257685972 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee7402ccc0 con 0x7fee7c06e3d0 2024-02-21T23:14:34.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:34.938+0000 7fee7be00640 1 -- 172.21.15.28:0/3257685972 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee74034610 con 0x7fee7c06e3d0 2024-02-21T23:14:34.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:34.939+0000 7fee82a00640 1 -- 172.21.15.28:0/3257685972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee7c06e3d0 msgr2=0x7fee7c14b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:34.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:34.939+0000 7fee82a00640 1 --2- 172.21.15.28:0/3257685972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee7c06e3d0 0x7fee7c14b2b0 secure :-1 s=READY pgs=2485 cs=0 l=1 rev1=1 crypto rx=0x7fee74009e50 tx=0x7fee7402bf80 comp rx=0 tx=0).stop 2024-02-21T23:14:34.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:34.939+0000 7fee82a00640 1 -- 172.21.15.28:0/3257685972 shutdown_connections 2024-02-21T23:14:34.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:34.939+0000 7fee82a00640 1 --2- 172.21.15.28:0/3257685972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee7c06e3d0 0x7fee7c14b2b0 unknown :-1 s=CLOSED pgs=2485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:34.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:34.939+0000 7fee82a00640 1 -- 172.21.15.28:0/3257685972 >> 172.21.15.28:0/3257685972 conn(0x7fee7c06b630 msgr2=0x7fee7c0c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:14:34.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:34.939+0000 7fee82a00640 1 -- 172.21.15.28:0/3257685972 shutdown_connections 2024-02-21T23:14:34.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:34.939+0000 7fee82a00640 1 -- 172.21.15.28:0/3257685972 wait complete. 2024-02-21T23:14:34.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:34.941+0000 7fee82a00640 1 Processor -- start 2024-02-21T23:14:34.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:34.941+0000 7fee82a00640 1 -- start start 2024-02-21T23:14:34.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:34.942+0000 7fee82a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee7c06dff0 0x7fee7c154530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:34.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:34.942+0000 7fee82a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee7c06e3d0 0x7fee7c154a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:34.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:34.942+0000 7fee82a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fee7c0c0fe0 0x7fee7c14a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:34.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:34.942+0000 7fee82a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee7c073520 con 0x7fee7c06e3d0 2024-02-21T23:14:34.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:34.942+0000 7fee82a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee7c0731b0 con 0x7fee7c06dff0 2024-02-21T23:14:34.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:34.942+0000 7fee82a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee7c14b7f0 con 0x7fee7c0c0fe0 2024-02-21T23:14:34.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:34.942+0000 7fee80c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee7c06e3d0 0x7fee7c154a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:34.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:34.942+0000 7fee80c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee7c06e3d0 0x7fee7c154a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41916/0 (socket says 172.21.15.28:41916) 2024-02-21T23:14:34.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:34.942+0000 7fee80c00640 1 -- 172.21.15.28:0/2516080854 learned_addr learned my addr 172.21.15.28:0/2516080854 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:14:34.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:34.942+0000 7fee81600640 1 --2- 172.21.15.28:0/2516080854 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee7c06dff0 0x7fee7c154530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:34.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:34.943+0000 7fee82000640 1 --2- 172.21.15.28:0/2516080854 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fee7c0c0fe0 0x7fee7c14a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:34.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:34.943+0000 7fee80c00640 1 -- 172.21.15.28:0/2516080854 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fee7c0c0fe0 msgr2=0x7fee7c14a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:34.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:34.943+0000 7fee80c00640 1 --2- 172.21.15.28:0/2516080854 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fee7c0c0fe0 0x7fee7c14a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:34.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:34.943+0000 7fee80c00640 1 -- 172.21.15.28:0/2516080854 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee7c06dff0 msgr2=0x7fee7c154530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:34.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:34.943+0000 7fee80c00640 1 --2- 172.21.15.28:0/2516080854 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee7c06dff0 0x7fee7c154530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:34.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:34.943+0000 7fee80c00640 1 -- 172.21.15.28:0/2516080854 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee7c154110 con 0x7fee7c06e3d0 2024-02-21T23:14:34.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:34.943+0000 7fee81600640 1 --2- 172.21.15.28:0/2516080854 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee7c06dff0 0x7fee7c154530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:14:34.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:34.943+0000 7fee80c00640 1 --2- 172.21.15.28:0/2516080854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee7c06e3d0 0x7fee7c154a70 secure :-1 s=READY pgs=2486 cs=0 l=1 rev1=1 crypto rx=0x7fee6800c430 tx=0x7fee6800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:34.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:34.944+0000 7fee79600640 1 -- 172.21.15.28:0/2516080854 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee680052d0 con 0x7fee7c06e3d0 2024-02-21T23:14:34.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:34.944+0000 7fee79600640 1 -- 172.21.15.28:0/2516080854 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee68005490 con 0x7fee7c06e3d0 2024-02-21T23:14:34.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:34.944+0000 7fee82a00640 1 -- 172.21.15.28:0/2516080854 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee7c150440 con 0x7fee7c06e3d0 2024-02-21T23:14:34.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:34.944+0000 7fee79600640 1 -- 172.21.15.28:0/2516080854 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee68007260 con 0x7fee7c06e3d0 2024-02-21T23:14:34.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:34.944+0000 7fee82a00640 1 -- 172.21.15.28:0/2516080854 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee7c14e070 con 0x7fee7c06e3d0 2024-02-21T23:14:34.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:34.945+0000 7fee82a00640 1 -- 172.21.15.28:0/2516080854 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee7c04f290 con 0x7fee7c06e3d0 2024-02-21T23:14:34.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:34.947+0000 7fee79600640 1 -- 172.21.15.28:0/2516080854 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fee68007400 con 0x7fee7c06e3d0 2024-02-21T23:14:34.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:34.948+0000 7fee79600640 1 --2- 172.21.15.28:0/2516080854 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fee5c06b4d0 0x7fee5c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:34.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:34.948+0000 7fee79600640 1 -- 172.21.15.28:0/2516080854 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(211..211 src has 1..211) v4 ==== 6274+0+0 (secure 0 0 0) 0x7fee68089940 con 0x7fee7c06e3d0 2024-02-21T23:14:34.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:34.949+0000 7fee81600640 1 --2- 172.21.15.28:0/2516080854 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fee5c06b4d0 0x7fee5c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:34.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:34.950+0000 7fee81600640 1 --2- 172.21.15.28:0/2516080854 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fee5c06b4d0 0x7fee5c06d990 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fee70006fd0 tx=0x7fee70008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:34.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:34.953+0000 7fee79600640 1 -- 172.21.15.28:0/2516080854 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fee68058570 con 0x7fee7c06e3d0 2024-02-21T23:14:35.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:35.193+0000 7fee82a00640 1 -- 172.21.15.28:0/2516080854 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fee7c06ecb0 con 0x7fee7c06e3d0 2024-02-21T23:14:35.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:35.195+0000 7fee79600640 1 -- 172.21.15.28:0/2516080854 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 240 v240) v1 ==== 95+0+4175 (secure 0 0 0) 0x7fee68057e70 con 0x7fee7c06e3d0 2024-02-21T23:14:35.182 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:14:35.182 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":240,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7265,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":224}],"filesystems":[{"mdsmap":{"epoch":240,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:14:11.669846+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":209,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":8287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8287":{"gid":8287,"name":"d","rank":1,"incarnation":235,"state":"up:stopping","state_seq":15,"addr":"172.21.15.81:6835/1607587459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1607587459},{"type":"v1","addr":"172.21.15.81:6835","nonce":1607587459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:14:35.182 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 240 2024-02-21T23:14:35.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:35.197+0000 7fee82a00640 1 -- 172.21.15.28:0/2516080854 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fee5c06b4d0 msgr2=0x7fee5c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:35.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:35.197+0000 7fee82a00640 1 --2- 172.21.15.28:0/2516080854 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fee5c06b4d0 0x7fee5c06d990 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fee70006fd0 tx=0x7fee70008040 comp rx=0 tx=0).stop 2024-02-21T23:14:35.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:35.197+0000 7fee82a00640 1 -- 172.21.15.28:0/2516080854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee7c06e3d0 msgr2=0x7fee7c154a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:35.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:35.197+0000 7fee82a00640 1 --2- 172.21.15.28:0/2516080854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee7c06e3d0 0x7fee7c154a70 secure :-1 s=READY pgs=2486 cs=0 l=1 rev1=1 crypto rx=0x7fee6800c430 tx=0x7fee6800c8f0 comp rx=0 tx=0).stop 2024-02-21T23:14:35.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:35.197+0000 7fee82a00640 1 -- 172.21.15.28:0/2516080854 shutdown_connections 2024-02-21T23:14:35.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:35.197+0000 7fee82a00640 1 --2- 172.21.15.28:0/2516080854 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fee5c06b4d0 0x7fee5c06d990 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:35.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:35.197+0000 7fee82a00640 1 --2- 172.21.15.28:0/2516080854 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fee7c0c0fe0 0x7fee7c14a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:35.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:35.197+0000 7fee82a00640 1 --2- 172.21.15.28:0/2516080854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee7c06e3d0 0x7fee7c154a70 unknown :-1 s=CLOSED pgs=2486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:35.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:35.197+0000 7fee82a00640 1 --2- 172.21.15.28:0/2516080854 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee7c06dff0 0x7fee7c154530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:35.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:35.197+0000 7fee82a00640 1 -- 172.21.15.28:0/2516080854 >> 172.21.15.28:0/2516080854 conn(0x7fee7c06b630 msgr2=0x7fee7c0c3750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:14:35.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:35.198+0000 7fee82a00640 1 -- 172.21.15.28:0/2516080854 shutdown_connections 2024-02-21T23:14:35.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:35.198+0000 7fee82a00640 1 -- 172.21.15.28:0/2516080854 wait complete. 2024-02-21T23:14:35.607 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 240, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:14:11.669846+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 209, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 8287}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8287': {'gid': 8287, 'name': 'd', 'rank': 1, 'incarnation': 235, 'state': 'up:stopping', 'state_seq': 15, 'addr': '172.21.15.81:6835/1607587459', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6834', 'nonce': 1607587459}, {'type': 'v1', 'addr': '172.21.15.81:6835', 'nonce': 1607587459}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:14:35.607 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8287:up:stopping 2024-02-21T23:14:36.608 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:14:36.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:36.846+0000 7ffb45a00640 1 Processor -- start 2024-02-21T23:14:36.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:36.847+0000 7ffb45a00640 1 -- start start 2024-02-21T23:14:36.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:36.848+0000 7ffb45a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7ffb40155fa0 0x7ffb400714a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:36.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:36.848+0000 7ffb45a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffb4006db00 0x7ffb400c1c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:36.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:36.848+0000 7ffb45a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb400c3210 con 0x7ffb400c2180 2024-02-21T23:14:36.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:36.848+0000 7ffb45a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb400c3820 con 0x7ffb40155fa0 2024-02-21T23:14:36.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:36.848+0000 7ffb45a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4006df80 con 0x7ffb4006db00 2024-02-21T23:14:36.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:36.849+0000 7ffb3ea00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7ffb40155fa0 0x7ffb400714a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:36.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:36.849+0000 7ffb3ea00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7ffb40155fa0 0x7ffb400714a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.28:49320/0 (socket says 172.21.15.28:49320) 2024-02-21T23:14:36.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:36.849+0000 7ffb3ea00640 1 -- 172.21.15.28:0/3838495907 learned_addr learned my addr 172.21.15.28:0/3838495907 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:14:36.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:36.849+0000 7ffb3e000640 1 --2- 172.21.15.28:0/3838495907 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffb4006db00 0x7ffb400c1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:36.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:36.850+0000 7ffb3ea00640 1 -- 172.21.15.28:0/3838495907 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffb4006db00 msgr2=0x7ffb400c1c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:14:36.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:36.850+0000 7ffb3ea00640 1 --2- 172.21.15.28:0/3838495907 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffb4006db00 0x7ffb400c1c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:36.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:36.850+0000 7ffb3ea00640 1 -- 172.21.15.28:0/3838495907 >> v1:172.21.15.81:6789/0 conn(0x7ffb400c2180 legacy=0x7ffb400c3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:36.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:36.850+0000 7ffb3ea00640 1 -- 172.21.15.28:0/3838495907 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb400c2ae0 con 0x7ffb40155fa0 2024-02-21T23:14:36.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:36.850+0000 7ffb3e000640 1 --2- 172.21.15.28:0/3838495907 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffb4006db00 0x7ffb400c1c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-21T23:14:36.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:36.851+0000 7ffb3ea00640 1 --2- 172.21.15.28:0/3838495907 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb40155fa0 0x7ffb400714a0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7ffb30009870 tx=0x7ffb3002b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a17c8886e7defd36 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:36.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:36.852+0000 7ffb3d600640 1 -- 172.21.15.28:0/3838495907 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb300317c0 con 0x7ffb40155fa0 2024-02-21T23:14:36.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:36.852+0000 7ffb3d600640 1 -- 172.21.15.28:0/3838495907 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb30037070 con 0x7ffb40155fa0 2024-02-21T23:14:36.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:36.852+0000 7ffb3d600640 1 -- 172.21.15.28:0/3838495907 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb30031b80 con 0x7ffb40155fa0 2024-02-21T23:14:36.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:36.853+0000 7ffb45a00640 1 -- 172.21.15.28:0/3838495907 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb40155fa0 msgr2=0x7ffb400714a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:36.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:36.853+0000 7ffb45a00640 1 --2- 172.21.15.28:0/3838495907 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb40155fa0 0x7ffb400714a0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7ffb30009870 tx=0x7ffb3002b860 comp rx=0 tx=0).stop 2024-02-21T23:14:36.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:36.853+0000 7ffb45a00640 1 -- 172.21.15.28:0/3838495907 shutdown_connections 2024-02-21T23:14:36.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:36.853+0000 7ffb45a00640 1 --2- 172.21.15.28:0/3838495907 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffb4006db00 0x7ffb400c1c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:36.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:36.853+0000 7ffb45a00640 1 --2- 172.21.15.28:0/3838495907 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb40155fa0 0x7ffb400714a0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:36.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:36.853+0000 7ffb45a00640 1 -- 172.21.15.28:0/3838495907 >> 172.21.15.28:0/3838495907 conn(0x7ffb400aa5c0 msgr2=0x7ffb400aa9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:14:36.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:36.853+0000 7ffb45a00640 1 -- 172.21.15.28:0/3838495907 shutdown_connections 2024-02-21T23:14:36.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:36.853+0000 7ffb45a00640 1 -- 172.21.15.28:0/3838495907 wait complete. 2024-02-21T23:14:36.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:36.855+0000 7ffb45a00640 1 Processor -- start 2024-02-21T23:14:36.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:36.855+0000 7ffb45a00640 1 -- start start 2024-02-21T23:14:36.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:36.856+0000 7ffb45a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb4006db00 0x7ffb4014de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:36.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:36.856+0000 7ffb45a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffb400c2180 0x7ffb401ee3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:36.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:36.856+0000 7ffb45a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb40155fa0 0x7ffb4014bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:36.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:36.856+0000 7ffb45a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb400c6710 con 0x7ffb4006db00 2024-02-21T23:14:36.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:36.856+0000 7ffb45a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb400c34b0 con 0x7ffb40155fa0 2024-02-21T23:14:36.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:36.856+0000 7ffb45a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb40156380 con 0x7ffb400c2180 2024-02-21T23:14:36.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:36.856+0000 7ffb3ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb4006db00 0x7ffb4014de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:36.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:36.856+0000 7ffb3ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb4006db00 0x7ffb4014de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41932/0 (socket says 172.21.15.28:41932) 2024-02-21T23:14:36.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:36.856+0000 7ffb3ea00640 1 -- 172.21.15.28:0/2189186422 learned_addr learned my addr 172.21.15.28:0/2189186422 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:14:36.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:36.856+0000 7ffb3e000640 1 --2- 172.21.15.28:0/2189186422 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffb400c2180 0x7ffb401ee3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:36.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:36.856+0000 7ffb3f400640 1 --2- 172.21.15.28:0/2189186422 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb40155fa0 0x7ffb4014bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:36.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:36.857+0000 7ffb3ea00640 1 -- 172.21.15.28:0/2189186422 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffb400c2180 msgr2=0x7ffb401ee3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:36.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:36.857+0000 7ffb3ea00640 1 --2- 172.21.15.28:0/2189186422 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffb400c2180 0x7ffb401ee3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:36.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:36.857+0000 7ffb3ea00640 1 -- 172.21.15.28:0/2189186422 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb40155fa0 msgr2=0x7ffb4014bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:36.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:36.857+0000 7ffb3ea00640 1 --2- 172.21.15.28:0/2189186422 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb40155fa0 0x7ffb4014bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:36.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:36.857+0000 7ffb3ea00640 1 -- 172.21.15.28:0/2189186422 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb4014c950 con 0x7ffb4006db00 2024-02-21T23:14:36.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:36.857+0000 7ffb3ea00640 1 --2- 172.21.15.28:0/2189186422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb4006db00 0x7ffb4014de20 secure :-1 s=READY pgs=2487 cs=0 l=1 rev1=1 crypto rx=0x7ffb300090b0 tx=0x7ffb300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:36.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:36.857+0000 7ffb36a00640 1 -- 172.21.15.28:0/2189186422 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb30032650 con 0x7ffb4006db00 2024-02-21T23:14:36.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:36.858+0000 7ffb36a00640 1 -- 172.21.15.28:0/2189186422 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb300327f0 con 0x7ffb4006db00 2024-02-21T23:14:36.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:36.858+0000 7ffb36a00640 1 -- 172.21.15.28:0/2189186422 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb30032be0 con 0x7ffb4006db00 2024-02-21T23:14:36.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:36.858+0000 7ffb45a00640 1 -- 172.21.15.28:0/2189186422 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb401566c0 con 0x7ffb4006db00 2024-02-21T23:14:36.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:36.858+0000 7ffb45a00640 1 -- 172.21.15.28:0/2189186422 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb40159370 con 0x7ffb4006db00 2024-02-21T23:14:36.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:36.859+0000 7ffb45a00640 1 -- 172.21.15.28:0/2189186422 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb40068640 con 0x7ffb4006db00 2024-02-21T23:14:36.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:36.860+0000 7ffb36a00640 1 -- 172.21.15.28:0/2189186422 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffb3004c060 con 0x7ffb4006db00 2024-02-21T23:14:36.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:36.862+0000 7ffb36a00640 1 --2- 172.21.15.28:0/2189186422 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ffb2006b4d0 0x7ffb2006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:36.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:36.862+0000 7ffb36a00640 1 -- 172.21.15.28:0/2189186422 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(211..211 src has 1..211) v4 ==== 6274+0+0 (secure 0 0 0) 0x7ffb300aba90 con 0x7ffb4006db00 2024-02-21T23:14:36.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:36.862+0000 7ffb3e000640 1 --2- 172.21.15.28:0/2189186422 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ffb2006b4d0 0x7ffb2006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:36.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:36.864+0000 7ffb3e000640 1 --2- 172.21.15.28:0/2189186422 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ffb2006b4d0 0x7ffb2006d990 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7ffb400a87f0 tx=0x7ffb2c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:36.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:36.865+0000 7ffb36a00640 1 -- 172.21.15.28:0/2189186422 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ffb30040070 con 0x7ffb4006db00 2024-02-21T23:14:37.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:37.107+0000 7ffb45a00640 1 -- 172.21.15.28:0/2189186422 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffb400c2790 con 0x7ffb4006db00 2024-02-21T23:14:37.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:37.108+0000 7ffb36a00640 1 -- 172.21.15.28:0/2189186422 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 240 v240) v1 ==== 95+0+4175 (secure 0 0 0) 0x7ffb3007a6c0 con 0x7ffb4006db00 2024-02-21T23:14:37.096 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:14:37.096 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":240,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7265,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":224}],"filesystems":[{"mdsmap":{"epoch":240,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:14:11.669846+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":209,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":8287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8287":{"gid":8287,"name":"d","rank":1,"incarnation":235,"state":"up:stopping","state_seq":15,"addr":"172.21.15.81:6835/1607587459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1607587459},{"type":"v1","addr":"172.21.15.81:6835","nonce":1607587459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:14:37.097 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 240 2024-02-21T23:14:37.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:37.111+0000 7ffb45a00640 1 -- 172.21.15.28:0/2189186422 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ffb2006b4d0 msgr2=0x7ffb2006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:37.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:37.111+0000 7ffb45a00640 1 --2- 172.21.15.28:0/2189186422 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ffb2006b4d0 0x7ffb2006d990 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7ffb400a87f0 tx=0x7ffb2c008040 comp rx=0 tx=0).stop 2024-02-21T23:14:37.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:37.111+0000 7ffb45a00640 1 -- 172.21.15.28:0/2189186422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb4006db00 msgr2=0x7ffb4014de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:37.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:37.111+0000 7ffb45a00640 1 --2- 172.21.15.28:0/2189186422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb4006db00 0x7ffb4014de20 secure :-1 s=READY pgs=2487 cs=0 l=1 rev1=1 crypto rx=0x7ffb300090b0 tx=0x7ffb300046c0 comp rx=0 tx=0).stop 2024-02-21T23:14:37.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:37.111+0000 7ffb45a00640 1 -- 172.21.15.28:0/2189186422 shutdown_connections 2024-02-21T23:14:37.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:37.111+0000 7ffb45a00640 1 --2- 172.21.15.28:0/2189186422 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ffb2006b4d0 0x7ffb2006d990 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:37.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:37.111+0000 7ffb45a00640 1 --2- 172.21.15.28:0/2189186422 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffb400c2180 0x7ffb401ee3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:37.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:37.111+0000 7ffb45a00640 1 --2- 172.21.15.28:0/2189186422 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffb40155fa0 0x7ffb4014bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:37.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:37.111+0000 7ffb45a00640 1 --2- 172.21.15.28:0/2189186422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb4006db00 0x7ffb4014de20 unknown :-1 s=CLOSED pgs=2487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:37.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:37.111+0000 7ffb45a00640 1 -- 172.21.15.28:0/2189186422 >> 172.21.15.28:0/2189186422 conn(0x7ffb400aa5c0 msgr2=0x7ffb40154400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:14:37.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:37.111+0000 7ffb45a00640 1 -- 172.21.15.28:0/2189186422 shutdown_connections 2024-02-21T23:14:37.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:37.112+0000 7ffb45a00640 1 -- 172.21.15.28:0/2189186422 wait complete. 2024-02-21T23:14:37.520 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 240, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:14:11.669846+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 209, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 8287}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8287': {'gid': 8287, 'name': 'd', 'rank': 1, 'incarnation': 235, 'state': 'up:stopping', 'state_seq': 15, 'addr': '172.21.15.81:6835/1607587459', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6834', 'nonce': 1607587459}, {'type': 'v1', 'addr': '172.21.15.81:6835', 'nonce': 1607587459}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:14:37.521 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8287:up:stopping 2024-02-21T23:14:38.511 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-21T23:14:38.519 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-21T23:14:38.521 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:14:38.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.787+0000 7fc92ca00640 1 Processor -- start 2024-02-21T23:14:38.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.788+0000 7fc92ca00640 1 -- start start 2024-02-21T23:14:38.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.789+0000 7fc92ca00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc928151cd0 0x7fc9280c4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:38.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.789+0000 7fc92ca00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9280c36b0 con 0x7fc92814f240 2024-02-21T23:14:38.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.789+0000 7fc92ca00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9280c3990 con 0x7fc928155980 2024-02-21T23:14:38.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.789+0000 7fc92ca00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9280c50e0 con 0x7fc928151cd0 2024-02-21T23:14:38.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.790+0000 7fc927400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fc92814f240 0x7fc9280c7600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:44712/0 (socket says 172.21.15.28:44712) 2024-02-21T23:14:38.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.790+0000 7fc927400640 1 -- 172.21.15.28:0/142009357 learned_addr learned my addr 172.21.15.28:0/142009357 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:14:38.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.790+0000 7fc926000640 1 -- 172.21.15.28:0/142009357 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4252036008 0 0) 0x7fc9280c36b0 con 0x7fc92814f240 2024-02-21T23:14:38.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.790+0000 7fc926a00640 1 --2- 172.21.15.28:0/142009357 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc928151cd0 0x7fc9280c4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:38.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.790+0000 7fc926000640 1 -- 172.21.15.28:0/142009357 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc90c003520 con 0x7fc92814f240 2024-02-21T23:14:38.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.791+0000 7fc926000640 1 -- 172.21.15.28:0/142009357 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2727429800 0 0) 0x7fc90c003520 con 0x7fc92814f240 2024-02-21T23:14:38.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.791+0000 7fc926000640 1 -- 172.21.15.28:0/142009357 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc928151cd0 msgr2=0x7fc9280c4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:14:38.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.791+0000 7fc926000640 1 --2- 172.21.15.28:0/142009357 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc928151cd0 0x7fc9280c4ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:38.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.791+0000 7fc926000640 1 -- 172.21.15.28:0/142009357 >> v1:172.21.15.81:6789/0 conn(0x7fc928155980 legacy=0x7fc92815c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:38.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.791+0000 7fc926000640 1 -- 172.21.15.28:0/142009357 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9280c3c70 con 0x7fc92814f240 2024-02-21T23:14:38.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.791+0000 7fc926000640 1 -- 172.21.15.28:0/142009357 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fc9180019f0 con 0x7fc92814f240 2024-02-21T23:14:38.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.791+0000 7fc926000640 1 -- 172.21.15.28:0/142009357 >> v1:172.21.15.28:6789/0 conn(0x7fc92814f240 legacy=0x7fc9280c7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:38.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.791+0000 7fc926000640 1 --2- 172.21.15.28:0/142009357 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc90c0039e0 0x7fc90c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:38.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.791+0000 7fc926000640 1 --2- 172.21.15.28:0/142009357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc90c004800 0x7fc90c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:38.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.791+0000 7fc926000640 1 --2- 172.21.15.28:0/142009357 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc928151cd0 0x7fc90c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:38.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.791+0000 7fc926000640 1 -- 172.21.15.28:0/142009357 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9280c50e0 con 0x7fc90c004800 2024-02-21T23:14:38.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.791+0000 7fc926000640 1 -- 172.21.15.28:0/142009357 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc90c00dc80 con 0x7fc90c0039e0 2024-02-21T23:14:38.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.791+0000 7fc926000640 1 -- 172.21.15.28:0/142009357 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc90c00de80 con 0x7fc928151cd0 2024-02-21T23:14:38.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.791+0000 7fc926a00640 1 --2- 172.21.15.28:0/142009357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc90c004800 0x7fc90c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:38.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.792+0000 7fc927400640 1 --2- 172.21.15.28:0/142009357 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc90c0039e0 0x7fc90c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:38.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.792+0000 7fc927e00640 1 --2- 172.21.15.28:0/142009357 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc928151cd0 0x7fc90c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:38.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.792+0000 7fc926a00640 1 -- 172.21.15.28:0/142009357 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc928151cd0 msgr2=0x7fc90c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:14:38.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.792+0000 7fc926a00640 1 --2- 172.21.15.28:0/142009357 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc928151cd0 0x7fc90c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:38.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.792+0000 7fc926a00640 1 -- 172.21.15.28:0/142009357 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc90c0039e0 msgr2=0x7fc90c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:14:38.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.792+0000 7fc926a00640 1 --2- 172.21.15.28:0/142009357 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc90c0039e0 0x7fc90c003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:38.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.792+0000 7fc926a00640 1 -- 172.21.15.28:0/142009357 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc90c003520 con 0x7fc90c004800 2024-02-21T23:14:38.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.793+0000 7fc926a00640 1 --2- 172.21.15.28:0/142009357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc90c004800 0x7fc90c004c80 secure :-1 s=READY pgs=2489 cs=0 l=1 rev1=1 crypto rx=0x7fc910009a90 tx=0x7fc90c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93fc847a7ecb5e40 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:38.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.793+0000 7fc926000640 1 -- 172.21.15.28:0/142009357 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc910039070 con 0x7fc90c004800 2024-02-21T23:14:38.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.793+0000 7fc926000640 1 -- 172.21.15.28:0/142009357 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc91002c680 con 0x7fc90c004800 2024-02-21T23:14:38.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.794+0000 7fc926000640 1 -- 172.21.15.28:0/142009357 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc910009690 con 0x7fc90c004800 2024-02-21T23:14:38.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.794+0000 7fc92ca00640 1 -- 172.21.15.28:0/142009357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc90c004800 msgr2=0x7fc90c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:38.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.794+0000 7fc92ca00640 1 --2- 172.21.15.28:0/142009357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc90c004800 0x7fc90c004c80 secure :-1 s=READY pgs=2489 cs=0 l=1 rev1=1 crypto rx=0x7fc910009a90 tx=0x7fc90c005830 comp rx=0 tx=0).stop 2024-02-21T23:14:38.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.794+0000 7fc927e00640 1 -- 172.21.15.28:0/142009357 reap_dead start 2024-02-21T23:14:38.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.794+0000 7fc92ca00640 1 -- 172.21.15.28:0/142009357 shutdown_connections 2024-02-21T23:14:38.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.794+0000 7fc92ca00640 1 -- 172.21.15.28:0/142009357 >> 172.21.15.28:0/142009357 conn(0x7fc928067050 msgr2=0x7fc9280a84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:14:38.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.794+0000 7fc92ca00640 1 -- 172.21.15.28:0/142009357 shutdown_connections 2024-02-21T23:14:38.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.794+0000 7fc92ca00640 1 -- 172.21.15.28:0/142009357 wait complete. 2024-02-21T23:14:38.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.796+0000 7fc92ca00640 1 Processor -- start 2024-02-21T23:14:38.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.796+0000 7fc92ca00640 1 -- start start 2024-02-21T23:14:38.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.797+0000 7fc92ca00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc90c004800 0x7fc9280ac6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:38.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.797+0000 7fc92ca00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc92814cba0 0x7fc92806c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:38.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.797+0000 7fc92ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9280c5280 0x7fc9281edf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:38.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.797+0000 7fc92ca00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc92806f2e0 con 0x7fc9280c5280 2024-02-21T23:14:38.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.797+0000 7fc92ca00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc92814f620 con 0x7fc92814cba0 2024-02-21T23:14:38.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.797+0000 7fc92ca00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc92806f460 con 0x7fc90c004800 2024-02-21T23:14:38.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.797+0000 7fc927e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9280c5280 0x7fc9281edf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:38.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.797+0000 7fc927e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9280c5280 0x7fc9281edf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41956/0 (socket says 172.21.15.28:41956) 2024-02-21T23:14:38.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.797+0000 7fc927e00640 1 -- 172.21.15.28:0/799336340 learned_addr learned my addr 172.21.15.28:0/799336340 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:14:38.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.798+0000 7fc927400640 1 --2- 172.21.15.28:0/799336340 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc90c004800 0x7fc9280ac6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:38.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.798+0000 7fc926a00640 1 --2- 172.21.15.28:0/799336340 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc92814cba0 0x7fc92806c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:38.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.798+0000 7fc927e00640 1 -- 172.21.15.28:0/799336340 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc90c004800 msgr2=0x7fc9280ac6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:38.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.798+0000 7fc927e00640 1 --2- 172.21.15.28:0/799336340 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc90c004800 0x7fc9280ac6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:38.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.798+0000 7fc927e00640 1 -- 172.21.15.28:0/799336340 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc92814cba0 msgr2=0x7fc92806c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:38.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.798+0000 7fc927e00640 1 --2- 172.21.15.28:0/799336340 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc92814cba0 0x7fc92806c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:38.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.798+0000 7fc927e00640 1 -- 172.21.15.28:0/799336340 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc92806cbe0 con 0x7fc9280c5280 2024-02-21T23:14:38.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.798+0000 7fc926a00640 1 --2- 172.21.15.28:0/799336340 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc92814cba0 0x7fc92806c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:14:38.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.798+0000 7fc927e00640 1 --2- 172.21.15.28:0/799336340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9280c5280 0x7fc9281edf20 secure :-1 s=READY pgs=2490 cs=0 l=1 rev1=1 crypto rx=0x7fc9280c1970 tx=0x7fc92000f960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:38.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.798+0000 7fc91f400640 1 -- 172.21.15.28:0/799336340 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9200053a0 con 0x7fc9280c5280 2024-02-21T23:14:38.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.799+0000 7fc91f400640 1 -- 172.21.15.28:0/799336340 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc920018070 con 0x7fc9280c5280 2024-02-21T23:14:38.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.799+0000 7fc91f400640 1 -- 172.21.15.28:0/799336340 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc920003830 con 0x7fc9280c5280 2024-02-21T23:14:38.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.799+0000 7fc92ca00640 1 -- 172.21.15.28:0/799336340 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9280acc30 con 0x7fc9280c5280 2024-02-21T23:14:38.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.799+0000 7fc92ca00640 1 -- 172.21.15.28:0/799336340 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9280aa500 con 0x7fc9280c5280 2024-02-21T23:14:38.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.800+0000 7fc92ca00640 1 -- 172.21.15.28:0/799336340 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc92804f290 con 0x7fc9280c5280 2024-02-21T23:14:38.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.801+0000 7fc91f400640 1 -- 172.21.15.28:0/799336340 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc9200039d0 con 0x7fc9280c5280 2024-02-21T23:14:38.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.803+0000 7fc91f400640 1 --2- 172.21.15.28:0/799336340 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc90406b4d0 0x7fc90406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:38.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.803+0000 7fc91f400640 1 -- 172.21.15.28:0/799336340 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(211..211 src has 1..211) v4 ==== 6274+0+0 (secure 0 0 0) 0x7fc92008a0b0 con 0x7fc9280c5280 2024-02-21T23:14:38.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.803+0000 7fc927400640 1 --2- 172.21.15.28:0/799336340 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc90406b4d0 0x7fc90406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:38.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.805+0000 7fc927400640 1 --2- 172.21.15.28:0/799336340 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc90406b4d0 0x7fc90406d990 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7fc9180013d0 tx=0x7fc918006040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:38.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:38.808+0000 7fc91f400640 1 -- 172.21.15.28:0/799336340 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc92806cbe0 con 0x7fc9280c5280 2024-02-21T23:14:39.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:39.052+0000 7fc92ca00640 1 -- 172.21.15.28:0/799336340 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc928151ab0 con 0x7fc9280c5280 2024-02-21T23:14:39.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:39.053+0000 7fc91f400640 1 -- 172.21.15.28:0/799336340 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 240 v240) v1 ==== 95+0+4175 (secure 0 0 0) 0x7fc920058bc0 con 0x7fc9280c5280 2024-02-21T23:14:39.041 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:14:39.041 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":240,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7265,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":224}],"filesystems":[{"mdsmap":{"epoch":240,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:14:11.669846+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":209,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":8287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8287":{"gid":8287,"name":"d","rank":1,"incarnation":235,"state":"up:stopping","state_seq":15,"addr":"172.21.15.81:6835/1607587459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1607587459},{"type":"v1","addr":"172.21.15.81:6835","nonce":1607587459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:14:39.041 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 240 2024-02-21T23:14:39.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:39.056+0000 7fc92ca00640 1 -- 172.21.15.28:0/799336340 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc90406b4d0 msgr2=0x7fc90406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:39.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:39.056+0000 7fc92ca00640 1 --2- 172.21.15.28:0/799336340 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc90406b4d0 0x7fc90406d990 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7fc9180013d0 tx=0x7fc918006040 comp rx=0 tx=0).stop 2024-02-21T23:14:39.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:39.056+0000 7fc92ca00640 1 -- 172.21.15.28:0/799336340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9280c5280 msgr2=0x7fc9281edf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:39.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:39.056+0000 7fc92ca00640 1 --2- 172.21.15.28:0/799336340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9280c5280 0x7fc9281edf20 secure :-1 s=READY pgs=2490 cs=0 l=1 rev1=1 crypto rx=0x7fc9280c1970 tx=0x7fc92000f960 comp rx=0 tx=0).stop 2024-02-21T23:14:39.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:39.057+0000 7fc92ca00640 1 -- 172.21.15.28:0/799336340 shutdown_connections 2024-02-21T23:14:39.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:39.057+0000 7fc92ca00640 1 --2- 172.21.15.28:0/799336340 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc90406b4d0 0x7fc90406d990 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:39.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:39.057+0000 7fc92ca00640 1 --2- 172.21.15.28:0/799336340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9280c5280 0x7fc9281edf20 unknown :-1 s=CLOSED pgs=2490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:39.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:39.057+0000 7fc92ca00640 1 --2- 172.21.15.28:0/799336340 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc92814cba0 0x7fc92806c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:39.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:39.057+0000 7fc92ca00640 1 --2- 172.21.15.28:0/799336340 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc90c004800 0x7fc9280ac6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:39.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:39.057+0000 7fc92ca00640 1 -- 172.21.15.28:0/799336340 >> 172.21.15.28:0/799336340 conn(0x7fc928067050 msgr2=0x7fc9280c0ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:14:39.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:39.057+0000 7fc92ca00640 1 -- 172.21.15.28:0/799336340 shutdown_connections 2024-02-21T23:14:39.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:39.057+0000 7fc92ca00640 1 -- 172.21.15.28:0/799336340 wait complete. 2024-02-21T23:14:39.465 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 240, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:14:11.669846+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 209, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 8287}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8287': {'gid': 8287, 'name': 'd', 'rank': 1, 'incarnation': 235, 'state': 'up:stopping', 'state_seq': 15, 'addr': '172.21.15.81:6835/1607587459', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6834', 'nonce': 1607587459}, {'type': 'v1', 'addr': '172.21.15.81:6835', 'nonce': 1607587459}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:14:39.466 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8287:up:stopping 2024-02-21T23:14:40.467 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:14:40.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.701+0000 7f9966e00640 1 Processor -- start 2024-02-21T23:14:40.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.701+0000 7f9966e00640 1 -- start start 2024-02-21T23:14:40.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.703+0000 7f9966e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f996006fe00 0x7f99600c6e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:40.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.703+0000 7f9966e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f99600701e0 0x7f9960151de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:40.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.703+0000 7f9966e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99600c5910 con 0x7f99600a6d30 2024-02-21T23:14:40.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.703+0000 7f9966e00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99600c5bf0 con 0x7f996006fe00 2024-02-21T23:14:40.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.703+0000 7f9966e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99600c5ed0 con 0x7f99600701e0 2024-02-21T23:14:40.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.703+0000 7f9957e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f99600a6d30 0x7f99600c26e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:44720/0 (socket says 172.21.15.28:44720) 2024-02-21T23:14:40.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.703+0000 7f9957e00640 1 -- 172.21.15.28:0/227590188 learned_addr learned my addr 172.21.15.28:0/227590188 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:14:40.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.704+0000 7f9964e00640 1 --2- 172.21.15.28:0/227590188 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f99600701e0 0x7f9960151de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:40.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.704+0000 7f995fe00640 1 --2- 172.21.15.28:0/227590188 >> v2:172.21.15.81:3300/0 conn(0x7f996006fe00 0x7f99600c6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:40.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.704+0000 7f995f400640 1 -- 172.21.15.28:0/227590188 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1368373581 0 0) 0x7f99600c5910 con 0x7f99600a6d30 2024-02-21T23:14:40.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.704+0000 7f995f400640 1 -- 172.21.15.28:0/227590188 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9944005730 con 0x7f99600a6d30 2024-02-21T23:14:40.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.704+0000 7f995f400640 1 -- 172.21.15.28:0/227590188 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1292618770 0 0) 0x7f9944005730 con 0x7f99600a6d30 2024-02-21T23:14:40.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.704+0000 7f995f400640 1 -- 172.21.15.28:0/227590188 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f99600701e0 msgr2=0x7f9960151de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:14:40.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.704+0000 7f995f400640 1 --2- 172.21.15.28:0/227590188 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f99600701e0 0x7f9960151de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:40.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.704+0000 7f995f400640 1 -- 172.21.15.28:0/227590188 >> v2:172.21.15.81:3300/0 conn(0x7f996006fe00 msgr2=0x7f99600c6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:14:40.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.704+0000 7f995f400640 1 --2- 172.21.15.28:0/227590188 >> v2:172.21.15.81:3300/0 conn(0x7f996006fe00 0x7f99600c6e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:40.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.704+0000 7f995f400640 1 -- 172.21.15.28:0/227590188 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99600c61b0 con 0x7f99600a6d30 2024-02-21T23:14:40.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.704+0000 7f995f400640 1 -- 172.21.15.28:0/227590188 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f9948001670 con 0x7f99600a6d30 2024-02-21T23:14:40.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.704+0000 7f995f400640 1 -- 172.21.15.28:0/227590188 >> v1:172.21.15.28:6789/0 conn(0x7f99600a6d30 legacy=0x7f99600c26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:40.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.704+0000 7f995f400640 1 --2- 172.21.15.28:0/227590188 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9944005bc0 0x7f9944005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:40.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.704+0000 7f995f400640 1 --2- 172.21.15.28:0/227590188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9944006a10 0x7f9944006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:40.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.704+0000 7f995f400640 1 --2- 172.21.15.28:0/227590188 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f99600701e0 0x7f99440073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:40.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.705+0000 7f995f400640 1 -- 172.21.15.28:0/227590188 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99600c5bf0 con 0x7f9944006a10 2024-02-21T23:14:40.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.705+0000 7f995f400640 1 -- 172.21.15.28:0/227590188 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99600c5ed0 con 0x7f9944005bc0 2024-02-21T23:14:40.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.705+0000 7f995f400640 1 -- 172.21.15.28:0/227590188 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f994400dbd0 con 0x7f99600701e0 2024-02-21T23:14:40.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.705+0000 7f9957e00640 1 --2- 172.21.15.28:0/227590188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9944006a10 0x7f9944006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:40.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.705+0000 7f995fe00640 1 --2- 172.21.15.28:0/227590188 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9944005bc0 0x7f9944005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:40.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.705+0000 7f9964e00640 1 --2- 172.21.15.28:0/227590188 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f99600701e0 0x7f99440073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:40.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.705+0000 7f9957e00640 1 -- 172.21.15.28:0/227590188 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f99600701e0 msgr2=0x7f99440073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:14:40.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.705+0000 7f9957e00640 1 --2- 172.21.15.28:0/227590188 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f99600701e0 0x7f99440073d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:40.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.705+0000 7f9957e00640 1 -- 172.21.15.28:0/227590188 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9944005bc0 msgr2=0x7f9944005fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:14:40.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.705+0000 7f9957e00640 1 --2- 172.21.15.28:0/227590188 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9944005bc0 0x7f9944005fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:40.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.705+0000 7f9957e00640 1 -- 172.21.15.28:0/227590188 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9944005730 con 0x7f9944006a10 2024-02-21T23:14:40.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.707+0000 7f9957e00640 1 --2- 172.21.15.28:0/227590188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9944006a10 0x7f9944006e90 secure :-1 s=READY pgs=2492 cs=0 l=1 rev1=1 crypto rx=0x7f994800c240 tx=0x7f9944007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=111679e670e0834f server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:40.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.707+0000 7f995f400640 1 -- 172.21.15.28:0/227590188 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9948007d30 con 0x7f9944006a10 2024-02-21T23:14:40.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.707+0000 7f995f400640 1 -- 172.21.15.28:0/227590188 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9948030070 con 0x7f9944006a10 2024-02-21T23:14:40.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.707+0000 7f995f400640 1 -- 172.21.15.28:0/227590188 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f994802cd60 con 0x7f9944006a10 2024-02-21T23:14:40.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.707+0000 7f9966e00640 1 -- 172.21.15.28:0/227590188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9944006a10 msgr2=0x7f9944006e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:40.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.707+0000 7f9966e00640 1 --2- 172.21.15.28:0/227590188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9944006a10 0x7f9944006e90 secure :-1 s=READY pgs=2492 cs=0 l=1 rev1=1 crypto rx=0x7f994800c240 tx=0x7f9944007a40 comp rx=0 tx=0).stop 2024-02-21T23:14:40.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.707+0000 7f9964e00640 1 -- 172.21.15.28:0/227590188 reap_dead start 2024-02-21T23:14:40.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.707+0000 7f9966e00640 1 -- 172.21.15.28:0/227590188 shutdown_connections 2024-02-21T23:14:40.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.707+0000 7f9966e00640 1 -- 172.21.15.28:0/227590188 >> 172.21.15.28:0/227590188 conn(0x7f996006c5b0 msgr2=0x7f99600c7800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:14:40.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.708+0000 7f9966e00640 1 -- 172.21.15.28:0/227590188 shutdown_connections 2024-02-21T23:14:40.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.708+0000 7f9966e00640 1 -- 172.21.15.28:0/227590188 wait complete. 2024-02-21T23:14:40.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.710+0000 7f9966e00640 1 Processor -- start 2024-02-21T23:14:40.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.711+0000 7f9966e00640 1 -- start start 2024-02-21T23:14:40.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.712+0000 7f9966e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9944006a10 0x7f99601eb1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:40.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.712+0000 7f9966e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99601d7f60 0x7f99601d8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:40.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.712+0000 7f9966e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f99601512b0 0x7f99601ee590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:40.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.712+0000 7f9966e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99600c2f70 con 0x7f99601d7f60 2024-02-21T23:14:40.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.712+0000 7f9966e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99600c2df0 con 0x7f9944006a10 2024-02-21T23:14:40.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.712+0000 7f9966e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9960151de0 con 0x7f99601512b0 2024-02-21T23:14:40.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.713+0000 7f9957e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99601d7f60 0x7f99601d8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:40.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.713+0000 7f9957e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99601d7f60 0x7f99601d8af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41980/0 (socket says 172.21.15.28:41980) 2024-02-21T23:14:40.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.713+0000 7f9957e00640 1 -- 172.21.15.28:0/3910476555 learned_addr learned my addr 172.21.15.28:0/3910476555 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:14:40.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.713+0000 7f995fe00640 1 --2- 172.21.15.28:0/3910476555 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9944006a10 0x7f99601eb1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:40.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.713+0000 7f9964e00640 1 --2- 172.21.15.28:0/3910476555 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f99601512b0 0x7f99601ee590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:40.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.713+0000 7f9957e00640 1 -- 172.21.15.28:0/3910476555 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f99601512b0 msgr2=0x7f99601ee590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:40.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.713+0000 7f9957e00640 1 --2- 172.21.15.28:0/3910476555 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f99601512b0 0x7f99601ee590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:40.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.713+0000 7f9957e00640 1 -- 172.21.15.28:0/3910476555 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9944006a10 msgr2=0x7f99601eb1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:40.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.713+0000 7f9957e00640 1 --2- 172.21.15.28:0/3910476555 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9944006a10 0x7f99601eb1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:40.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.713+0000 7f9957e00640 1 -- 172.21.15.28:0/3910476555 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99601eddd0 con 0x7f99601d7f60 2024-02-21T23:14:40.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.713+0000 7f995fe00640 1 --2- 172.21.15.28:0/3910476555 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9944006a10 0x7f99601eb1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:14:40.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.713+0000 7f9957e00640 1 --2- 172.21.15.28:0/3910476555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99601d7f60 0x7f99601d8af0 secure :-1 s=READY pgs=2493 cs=0 l=1 rev1=1 crypto rx=0x7f994800c240 tx=0x7f99600c5ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:40.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.714+0000 7f995cc00640 1 -- 172.21.15.28:0/3910476555 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9948045070 con 0x7f99601d7f60 2024-02-21T23:14:40.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.714+0000 7f995cc00640 1 -- 172.21.15.28:0/3910476555 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9948004bf0 con 0x7f99601d7f60 2024-02-21T23:14:40.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.714+0000 7f995cc00640 1 -- 172.21.15.28:0/3910476555 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9948001c70 con 0x7f99601d7f60 2024-02-21T23:14:40.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.714+0000 7f9966e00640 1 -- 172.21.15.28:0/3910476555 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99600abcb0 con 0x7f99601d7f60 2024-02-21T23:14:40.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.714+0000 7f9966e00640 1 -- 172.21.15.28:0/3910476555 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99601ee070 con 0x7f99601d7f60 2024-02-21T23:14:40.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.715+0000 7f9966e00640 1 -- 172.21.15.28:0/3910476555 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9960068640 con 0x7f99601d7f60 2024-02-21T23:14:40.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.716+0000 7f995cc00640 1 -- 172.21.15.28:0/3910476555 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9948007070 con 0x7f99601d7f60 2024-02-21T23:14:40.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.717+0000 7f995cc00640 1 --2- 172.21.15.28:0/3910476555 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f993c06b4d0 0x7f993c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:40.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.718+0000 7f995cc00640 1 -- 172.21.15.28:0/3910476555 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(211..211 src has 1..211) v4 ==== 6274+0+0 (secure 0 0 0) 0x7f99480ab5c0 con 0x7f99601d7f60 2024-02-21T23:14:40.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.718+0000 7f995fe00640 1 --2- 172.21.15.28:0/3910476555 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f993c06b4d0 0x7f993c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:40.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.719+0000 7f995fe00640 1 --2- 172.21.15.28:0/3910476555 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f993c06b4d0 0x7f993c06d990 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f99600aa210 tx=0x7f9950008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:40.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.722+0000 7f995cc00640 1 -- 172.21.15.28:0/3910476555 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f9948079c20 con 0x7f99601d7f60 2024-02-21T23:14:40.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.963+0000 7f9966e00640 1 -- 172.21.15.28:0/3910476555 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f996006b020 con 0x7f99601d7f60 2024-02-21T23:14:40.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.964+0000 7f995cc00640 1 -- 172.21.15.28:0/3910476555 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 240 v240) v1 ==== 95+0+4175 (secure 0 0 0) 0x7f994807dcc0 con 0x7f99601d7f60 2024-02-21T23:14:40.953 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:14:40.953 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":240,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7265,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":224}],"filesystems":[{"mdsmap":{"epoch":240,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:14:11.669846+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":209,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":8287},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8287":{"gid":8287,"name":"d","rank":1,"incarnation":235,"state":"up:stopping","state_seq":15,"addr":"172.21.15.81:6835/1607587459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1607587459},{"type":"v1","addr":"172.21.15.81:6835","nonce":1607587459}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:14:40.953 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 240 2024-02-21T23:14:40.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.967+0000 7f9966e00640 1 -- 172.21.15.28:0/3910476555 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f993c06b4d0 msgr2=0x7f993c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:40.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.967+0000 7f9966e00640 1 --2- 172.21.15.28:0/3910476555 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f993c06b4d0 0x7f993c06d990 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f99600aa210 tx=0x7f9950008040 comp rx=0 tx=0).stop 2024-02-21T23:14:40.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.967+0000 7f9966e00640 1 -- 172.21.15.28:0/3910476555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99601d7f60 msgr2=0x7f99601d8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:40.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.967+0000 7f9966e00640 1 --2- 172.21.15.28:0/3910476555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99601d7f60 0x7f99601d8af0 secure :-1 s=READY pgs=2493 cs=0 l=1 rev1=1 crypto rx=0x7f994800c240 tx=0x7f99600c5ae0 comp rx=0 tx=0).stop 2024-02-21T23:14:40.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.967+0000 7f9966e00640 1 -- 172.21.15.28:0/3910476555 shutdown_connections 2024-02-21T23:14:40.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.967+0000 7f9966e00640 1 --2- 172.21.15.28:0/3910476555 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f993c06b4d0 0x7f993c06d990 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:40.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.967+0000 7f9966e00640 1 --2- 172.21.15.28:0/3910476555 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f99601512b0 0x7f99601ee590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:40.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.967+0000 7f9966e00640 1 --2- 172.21.15.28:0/3910476555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99601d7f60 0x7f99601d8af0 unknown :-1 s=CLOSED pgs=2493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:40.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.967+0000 7f9966e00640 1 --2- 172.21.15.28:0/3910476555 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9944006a10 0x7f99601eb1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:40.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.967+0000 7f9966e00640 1 -- 172.21.15.28:0/3910476555 >> 172.21.15.28:0/3910476555 conn(0x7f996006c5b0 msgr2=0x7f99600c4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:14:40.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.967+0000 7f9966e00640 1 -- 172.21.15.28:0/3910476555 shutdown_connections 2024-02-21T23:14:40.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:40.967+0000 7f9966e00640 1 -- 172.21.15.28:0/3910476555 wait complete. 2024-02-21T23:14:41.379 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 240, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:14:11.669846+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 209, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 8287}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8287': {'gid': 8287, 'name': 'd', 'rank': 1, 'incarnation': 235, 'state': 'up:stopping', 'state_seq': 15, 'addr': '172.21.15.81:6835/1607587459', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6834', 'nonce': 1607587459}, {'type': 'v1', 'addr': '172.21.15.81:6835', 'nonce': 1607587459}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:14:41.379 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8287:up:stopping 2024-02-21T23:14:42.380 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:14:42.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.613+0000 7f0fbec00640 1 Processor -- start 2024-02-21T23:14:42.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.614+0000 7f0fbec00640 1 -- start start 2024-02-21T23:14:42.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.615+0000 7f0fbec00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0fb814f0b0 0x7f0fb8155180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:42.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.615+0000 7f0fbec00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0fb80a6160 0x7f0fb80c2010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:42.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.615+0000 7f0fbec00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fb8155760 con 0x7f0fb80c2550 2024-02-21T23:14:42.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.616+0000 7f0fbec00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fb8151d30 con 0x7f0fb814f0b0 2024-02-21T23:14:42.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.616+0000 7f0fbec00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fb8155a40 con 0x7f0fb80a6160 2024-02-21T23:14:42.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.616+0000 7f0fbd800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0fb814f0b0 0x7f0fb8155180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:42.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.616+0000 7f0fbd800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0fb814f0b0 0x7f0fb8155180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41984/0 (socket says 172.21.15.28:41984) 2024-02-21T23:14:42.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.616+0000 7f0fbd800640 1 -- 172.21.15.28:0/674885834 learned_addr learned my addr 172.21.15.28:0/674885834 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:14:42.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.616+0000 7f0fbce00640 1 --2- 172.21.15.28:0/674885834 >> v2:172.21.15.81:3300/0 conn(0x7f0fb80a6160 0x7f0fb80c2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:42.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.617+0000 7f0fbd800640 1 -- 172.21.15.28:0/674885834 >> v2:172.21.15.81:3300/0 conn(0x7f0fb80a6160 msgr2=0x7f0fb80c2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:14:42.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.617+0000 7f0fbd800640 1 --2- 172.21.15.28:0/674885834 >> v2:172.21.15.81:3300/0 conn(0x7f0fb80a6160 0x7f0fb80c2010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:42.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.617+0000 7f0fbd800640 1 -- 172.21.15.28:0/674885834 >> v1:172.21.15.81:6789/0 conn(0x7f0fb80c2550 legacy=0x7f0fb81521c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:14:42.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.617+0000 7f0fbd800640 1 -- 172.21.15.28:0/674885834 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fb8155d20 con 0x7f0fb814f0b0 2024-02-21T23:14:42.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.618+0000 7f0fbd800640 1 --2- 172.21.15.28:0/674885834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0fb814f0b0 0x7f0fb8155180 secure :-1 s=READY pgs=2494 cs=0 l=1 rev1=1 crypto rx=0x7f0fb8151eb0 tx=0x7f0fac02b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b320eb039fe5722 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:42.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.618+0000 7f0fb7e00640 1 -- 172.21.15.28:0/674885834 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fac035020 con 0x7f0fb814f0b0 2024-02-21T23:14:42.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.618+0000 7f0fb7e00640 1 -- 172.21.15.28:0/674885834 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fac02d040 con 0x7f0fb814f0b0 2024-02-21T23:14:42.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.618+0000 7f0fb7e00640 1 -- 172.21.15.28:0/674885834 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fac031a80 con 0x7f0fb814f0b0 2024-02-21T23:14:42.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.619+0000 7f0fbec00640 1 -- 172.21.15.28:0/674885834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0fb814f0b0 msgr2=0x7f0fb8155180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:42.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.619+0000 7f0fbec00640 1 --2- 172.21.15.28:0/674885834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0fb814f0b0 0x7f0fb8155180 secure :-1 s=READY pgs=2494 cs=0 l=1 rev1=1 crypto rx=0x7f0fb8151eb0 tx=0x7f0fac02b900 comp rx=0 tx=0).stop 2024-02-21T23:14:42.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.619+0000 7f0fbec00640 1 -- 172.21.15.28:0/674885834 shutdown_connections 2024-02-21T23:14:42.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.619+0000 7f0fbec00640 1 --2- 172.21.15.28:0/674885834 >> v2:172.21.15.81:3300/0 conn(0x7f0fb80a6160 0x7f0fb80c2010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:42.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.619+0000 7f0fbec00640 1 --2- 172.21.15.28:0/674885834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0fb814f0b0 0x7f0fb8155180 unknown :-1 s=CLOSED pgs=2494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:42.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.619+0000 7f0fbec00640 1 -- 172.21.15.28:0/674885834 >> 172.21.15.28:0/674885834 conn(0x7f0fb806b630 msgr2=0x7f0fb80c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:14:42.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.619+0000 7f0fbec00640 1 -- 172.21.15.28:0/674885834 shutdown_connections 2024-02-21T23:14:42.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.619+0000 7f0fbec00640 1 -- 172.21.15.28:0/674885834 wait complete. 2024-02-21T23:14:42.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.621+0000 7f0fbec00640 1 Processor -- start 2024-02-21T23:14:42.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.622+0000 7f0fbec00640 1 -- start start 2024-02-21T23:14:42.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.623+0000 7f0fbec00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0fb80a6160 0x7f0fb814ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:42.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.623+0000 7f0fbec00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0fb80c2550 0x7f0fb8150c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:42.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.623+0000 7f0fbec00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0fb814f0b0 0x7f0fb81e8790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:42.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.623+0000 7f0fbec00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb80a7f70 con 0x7f0fb80c2550 2024-02-21T23:14:42.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.623+0000 7f0fbec00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb80a68d0 con 0x7f0fb814f0b0 2024-02-21T23:14:42.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.623+0000 7f0fbec00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb80a80f0 con 0x7f0fb80a6160 2024-02-21T23:14:42.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.623+0000 7f0fbce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0fb80c2550 0x7f0fb8150c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:42.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.623+0000 7f0fbce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0fb80c2550 0x7f0fb8150c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41994/0 (socket says 172.21.15.28:41994) 2024-02-21T23:14:42.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.623+0000 7f0fbce00640 1 -- 172.21.15.28:0/3421543653 learned_addr learned my addr 172.21.15.28:0/3421543653 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:14:42.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.624+0000 7f0fbd800640 1 --2- 172.21.15.28:0/3421543653 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0fb80a6160 0x7f0fb814ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:42.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.624+0000 7f0fbe200640 1 --2- 172.21.15.28:0/3421543653 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0fb814f0b0 0x7f0fb81e8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:42.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.624+0000 7f0fbce00640 1 -- 172.21.15.28:0/3421543653 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0fb80a6160 msgr2=0x7f0fb814ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:42.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.624+0000 7f0fbce00640 1 --2- 172.21.15.28:0/3421543653 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0fb80a6160 0x7f0fb814ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:42.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.624+0000 7f0fbce00640 1 -- 172.21.15.28:0/3421543653 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0fb814f0b0 msgr2=0x7f0fb81e8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:42.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.624+0000 7f0fbce00640 1 --2- 172.21.15.28:0/3421543653 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0fb814f0b0 0x7f0fb81e8790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:42.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.624+0000 7f0fbce00640 1 -- 172.21.15.28:0/3421543653 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fb81ecbf0 con 0x7f0fb80c2550 2024-02-21T23:14:42.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.624+0000 7f0fbd800640 1 --2- 172.21.15.28:0/3421543653 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0fb80a6160 0x7f0fb814ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:14:42.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.624+0000 7f0fbe200640 1 --2- 172.21.15.28:0/3421543653 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0fb814f0b0 0x7f0fb81e8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:14:42.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.624+0000 7f0fbce00640 1 --2- 172.21.15.28:0/3421543653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0fb80c2550 0x7f0fb8150c60 secure :-1 s=READY pgs=2495 cs=0 l=1 rev1=1 crypto rx=0x7f0fa400d5f0 tx=0x7f0fa400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:42.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.624+0000 7f0fb5600640 1 -- 172.21.15.28:0/3421543653 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fa40125f0 con 0x7f0fb80c2550 2024-02-21T23:14:42.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.624+0000 7f0fb5600640 1 -- 172.21.15.28:0/3421543653 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fa4019070 con 0x7f0fb80c2550 2024-02-21T23:14:42.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.624+0000 7f0fb5600640 1 -- 172.21.15.28:0/3421543653 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fa40129d0 con 0x7f0fb80c2550 2024-02-21T23:14:42.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.624+0000 7f0fbec00640 1 -- 172.21.15.28:0/3421543653 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fb81ebe30 con 0x7f0fb80c2550 2024-02-21T23:14:42.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.625+0000 7f0fbec00640 1 -- 172.21.15.28:0/3421543653 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fb81ec3f0 con 0x7f0fb80c2550 2024-02-21T23:14:42.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.626+0000 7f0fbec00640 1 -- 172.21.15.28:0/3421543653 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fb804f290 con 0x7f0fb80c2550 2024-02-21T23:14:42.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.628+0000 7f0fb5600640 1 -- 172.21.15.28:0/3421543653 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0fa4012b70 con 0x7f0fb80c2550 2024-02-21T23:14:42.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.629+0000 7f0fb5600640 1 --2- 172.21.15.28:0/3421543653 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0f9806b4d0 0x7f0f9806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:42.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.629+0000 7f0fb5600640 1 -- 172.21.15.28:0/3421543653 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(211..211 src has 1..211) v4 ==== 6274+0+0 (secure 0 0 0) 0x7f0fa408a7a0 con 0x7f0fb80c2550 2024-02-21T23:14:42.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.629+0000 7f0fbd800640 1 --2- 172.21.15.28:0/3421543653 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0f9806b4d0 0x7f0f9806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:42.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.630+0000 7f0fbd800640 1 --2- 172.21.15.28:0/3421543653 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0f9806b4d0 0x7f0f9806d990 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f0fac036000 tx=0x7f0fac008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:42.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.633+0000 7f0fb5600640 1 -- 172.21.15.28:0/3421543653 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f0fa4059470 con 0x7f0fb80c2550 2024-02-21T23:14:42.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.887+0000 7f0fbec00640 1 -- 172.21.15.28:0/3421543653 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0fb80ac3c0 con 0x7f0fb80c2550 2024-02-21T23:14:42.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.888+0000 7f0fb5600640 1 -- 172.21.15.28:0/3421543653 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 241 v241) v1 ==== 95+0+3381 (secure 0 0 0) 0x7f0fa4058d70 con 0x7f0fb80c2550 2024-02-21T23:14:42.877 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:14:42.877 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":241,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":241,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:14:42.637256+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:replay","state_seq":1,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:14:42.877 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 241 2024-02-21T23:14:42.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.891+0000 7f0fbec00640 1 -- 172.21.15.28:0/3421543653 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0f9806b4d0 msgr2=0x7f0f9806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:42.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.891+0000 7f0fbec00640 1 --2- 172.21.15.28:0/3421543653 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0f9806b4d0 0x7f0f9806d990 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f0fac036000 tx=0x7f0fac008030 comp rx=0 tx=0).stop 2024-02-21T23:14:42.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.891+0000 7f0fbec00640 1 -- 172.21.15.28:0/3421543653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0fb80c2550 msgr2=0x7f0fb8150c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:42.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.891+0000 7f0fbec00640 1 --2- 172.21.15.28:0/3421543653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0fb80c2550 0x7f0fb8150c60 secure :-1 s=READY pgs=2495 cs=0 l=1 rev1=1 crypto rx=0x7f0fa400d5f0 tx=0x7f0fa400dab0 comp rx=0 tx=0).stop 2024-02-21T23:14:42.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.891+0000 7f0fbec00640 1 -- 172.21.15.28:0/3421543653 shutdown_connections 2024-02-21T23:14:42.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.891+0000 7f0fbec00640 1 --2- 172.21.15.28:0/3421543653 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0f9806b4d0 0x7f0f9806d990 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:42.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.891+0000 7f0fbec00640 1 --2- 172.21.15.28:0/3421543653 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0fb814f0b0 0x7f0fb81e8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:42.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.891+0000 7f0fbec00640 1 --2- 172.21.15.28:0/3421543653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0fb80c2550 0x7f0fb8150c60 unknown :-1 s=CLOSED pgs=2495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:42.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.891+0000 7f0fbec00640 1 --2- 172.21.15.28:0/3421543653 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0fb80a6160 0x7f0fb814ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:42.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.891+0000 7f0fbec00640 1 -- 172.21.15.28:0/3421543653 >> 172.21.15.28:0/3421543653 conn(0x7f0fb806b630 msgr2=0x7f0fb80c1780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:14:42.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.891+0000 7f0fbec00640 1 -- 172.21.15.28:0/3421543653 shutdown_connections 2024-02-21T23:14:42.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:42.891+0000 7f0fbec00640 1 -- 172.21.15.28:0/3421543653 wait complete. 2024-02-21T23:14:43.302 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 241, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:14:42.637256+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:14:43.302 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:replay 2024-02-21T23:14:44.304 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:14:44.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.553+0000 7f2052800640 1 Processor -- start 2024-02-21T23:14:44.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.553+0000 7f2052800640 1 -- start start 2024-02-21T23:14:44.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.555+0000 7f2052800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f204c150450 0x7f204c0a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:44.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.555+0000 7f2052800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f204c156af0 0x7f204c06dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:44.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.555+0000 7f2052800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f204c0c2760 con 0x7f204c06ee40 2024-02-21T23:14:44.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.555+0000 7f2052800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f204c0c22c0 con 0x7f204c150450 2024-02-21T23:14:44.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.555+0000 7f2052800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f204c156660 con 0x7f204c156af0 2024-02-21T23:14:44.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.555+0000 7f2051400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f204c150450 0x7f204c0a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:44.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.555+0000 7f2051400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f204c150450 0x7f204c0a5fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42000/0 (socket says 172.21.15.28:42000) 2024-02-21T23:14:44.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.555+0000 7f2051400640 1 -- 172.21.15.28:0/2882669503 learned_addr learned my addr 172.21.15.28:0/2882669503 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:14:44.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.556+0000 7f2051e00640 1 --2- 172.21.15.28:0/2882669503 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f204c156af0 0x7f204c06dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:44.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.556+0000 7f2051400640 1 -- 172.21.15.28:0/2882669503 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f204c156af0 msgr2=0x7f204c06dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:14:44.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.556+0000 7f2051400640 1 --2- 172.21.15.28:0/2882669503 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f204c156af0 0x7f204c06dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:44.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.556+0000 7f2051400640 1 -- 172.21.15.28:0/2882669503 >> v1:172.21.15.81:6789/0 conn(0x7f204c06ee40 legacy=0x7f204c1536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:44.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.556+0000 7f2051400640 1 -- 172.21.15.28:0/2882669503 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f204c156310 con 0x7f204c150450 2024-02-21T23:14:44.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.558+0000 7f2051400640 1 --2- 172.21.15.28:0/2882669503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f204c150450 0x7f204c0a5fd0 secure :-1 s=READY pgs=2496 cs=0 l=1 rev1=1 crypto rx=0x7f2040009870 tx=0x7f204002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=358de4dd97223ac2 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:44.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.558+0000 7f204be00640 1 -- 172.21.15.28:0/2882669503 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2040034070 con 0x7f204c150450 2024-02-21T23:14:44.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.558+0000 7f204be00640 1 -- 172.21.15.28:0/2882669503 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2040030840 con 0x7f204c150450 2024-02-21T23:14:44.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.558+0000 7f204be00640 1 -- 172.21.15.28:0/2882669503 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2040030c10 con 0x7f204c150450 2024-02-21T23:14:44.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.558+0000 7f2052800640 1 -- 172.21.15.28:0/2882669503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f204c150450 msgr2=0x7f204c0a5fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:44.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.558+0000 7f2052800640 1 --2- 172.21.15.28:0/2882669503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f204c150450 0x7f204c0a5fd0 secure :-1 s=READY pgs=2496 cs=0 l=1 rev1=1 crypto rx=0x7f2040009870 tx=0x7f204002b980 comp rx=0 tx=0).stop 2024-02-21T23:14:44.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.559+0000 7f2052800640 1 -- 172.21.15.28:0/2882669503 shutdown_connections 2024-02-21T23:14:44.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.559+0000 7f2052800640 1 --2- 172.21.15.28:0/2882669503 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f204c156af0 0x7f204c06dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:44.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.559+0000 7f2052800640 1 --2- 172.21.15.28:0/2882669503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f204c150450 0x7f204c0a5fd0 unknown :-1 s=CLOSED pgs=2496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:44.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.559+0000 7f2052800640 1 -- 172.21.15.28:0/2882669503 >> 172.21.15.28:0/2882669503 conn(0x7f204c06b630 msgr2=0x7f204c0c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:14:44.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.559+0000 7f2052800640 1 -- 172.21.15.28:0/2882669503 shutdown_connections 2024-02-21T23:14:44.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.559+0000 7f2052800640 1 -- 172.21.15.28:0/2882669503 wait complete. 2024-02-21T23:14:44.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.560+0000 7f2052800640 1 Processor -- start 2024-02-21T23:14:44.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.561+0000 7f2052800640 1 -- start start 2024-02-21T23:14:44.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.561+0000 7f2052800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f204c06ee40 0x7f204c0a6d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:44.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.561+0000 7f2052800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f204c150450 0x7f204c1453f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:44.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.561+0000 7f2052800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f204c156af0 0x7f204c145930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:44.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.561+0000 7f2052800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f204c156f80 con 0x7f204c150450 2024-02-21T23:14:44.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.561+0000 7f2052800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f204c0c2cb0 con 0x7f204c156af0 2024-02-21T23:14:44.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.561+0000 7f2052800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f204c157100 con 0x7f204c06ee40 2024-02-21T23:14:44.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.562+0000 7f2050a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f204c150450 0x7f204c1453f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:44.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.562+0000 7f2050a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f204c150450 0x7f204c1453f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42010/0 (socket says 172.21.15.28:42010) 2024-02-21T23:14:44.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.562+0000 7f2050a00640 1 -- 172.21.15.28:0/2806490654 learned_addr learned my addr 172.21.15.28:0/2806490654 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:14:44.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.562+0000 7f2051e00640 1 --2- 172.21.15.28:0/2806490654 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f204c156af0 0x7f204c145930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:44.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.562+0000 7f2051400640 1 --2- 172.21.15.28:0/2806490654 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f204c06ee40 0x7f204c0a6d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:44.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.562+0000 7f2050a00640 1 -- 172.21.15.28:0/2806490654 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f204c06ee40 msgr2=0x7f204c0a6d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:44.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.562+0000 7f2050a00640 1 --2- 172.21.15.28:0/2806490654 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f204c06ee40 0x7f204c0a6d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:44.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.562+0000 7f2050a00640 1 -- 172.21.15.28:0/2806490654 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f204c156af0 msgr2=0x7f204c145930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:44.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.562+0000 7f2050a00640 1 --2- 172.21.15.28:0/2806490654 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f204c156af0 0x7f204c145930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:44.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.562+0000 7f2050a00640 1 -- 172.21.15.28:0/2806490654 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f204c072f70 con 0x7f204c150450 2024-02-21T23:14:44.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.562+0000 7f2051e00640 1 --2- 172.21.15.28:0/2806490654 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f204c156af0 0x7f204c145930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:14:44.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.562+0000 7f2051400640 1 --2- 172.21.15.28:0/2806490654 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f204c06ee40 0x7f204c0a6d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:14:44.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.563+0000 7f2050a00640 1 --2- 172.21.15.28:0/2806490654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f204c150450 0x7f204c1453f0 secure :-1 s=READY pgs=2497 cs=0 l=1 rev1=1 crypto rx=0x7f203800c430 tx=0x7f203800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:44.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.563+0000 7f2049600640 1 -- 172.21.15.28:0/2806490654 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20380052d0 con 0x7f204c150450 2024-02-21T23:14:44.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.563+0000 7f2049600640 1 -- 172.21.15.28:0/2806490654 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2038005490 con 0x7f204c150450 2024-02-21T23:14:44.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.563+0000 7f2049600640 1 -- 172.21.15.28:0/2806490654 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2038007260 con 0x7f204c150450 2024-02-21T23:14:44.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.563+0000 7f2052800640 1 -- 172.21.15.28:0/2806490654 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f204c1ebaf0 con 0x7f204c150450 2024-02-21T23:14:44.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.563+0000 7f2052800640 1 -- 172.21.15.28:0/2806490654 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f204c0a7260 con 0x7f204c150450 2024-02-21T23:14:44.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.564+0000 7f2052800640 1 -- 172.21.15.28:0/2806490654 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f204c04f290 con 0x7f204c150450 2024-02-21T23:14:44.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.565+0000 7f2049600640 1 -- 172.21.15.28:0/2806490654 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2038007400 con 0x7f204c150450 2024-02-21T23:14:44.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.566+0000 7f2049600640 1 --2- 172.21.15.28:0/2806490654 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f202c06b4d0 0x7f202c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:44.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.566+0000 7f2049600640 1 -- 172.21.15.28:0/2806490654 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f2038089a10 con 0x7f204c150450 2024-02-21T23:14:44.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.567+0000 7f2051400640 1 --2- 172.21.15.28:0/2806490654 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f202c06b4d0 0x7f202c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:44.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.568+0000 7f2051400640 1 --2- 172.21.15.28:0/2806490654 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f202c06b4d0 0x7f202c06d990 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f2040036000 tx=0x7f204003b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:44.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.571+0000 7f2049600640 1 -- 172.21.15.28:0/2806490654 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f2038058560 con 0x7f204c150450 2024-02-21T23:14:44.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.812+0000 7f2052800640 1 -- 172.21.15.28:0/2806490654 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f204c0c0be0 con 0x7f204c150450 2024-02-21T23:14:44.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.813+0000 7f2049600640 1 -- 172.21.15.28:0/2806490654 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 243 v243) v1 ==== 95+0+3385 (secure 0 0 0) 0x7f2038057e60 con 0x7f204c150450 2024-02-21T23:14:44.801 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:14:44.801 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":243,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":243,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:14:43.651162+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:reconnect","state_seq":22,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:14:44.802 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 243 2024-02-21T23:14:44.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.815+0000 7f2052800640 1 -- 172.21.15.28:0/2806490654 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f202c06b4d0 msgr2=0x7f202c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:44.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.815+0000 7f2052800640 1 --2- 172.21.15.28:0/2806490654 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f202c06b4d0 0x7f202c06d990 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f2040036000 tx=0x7f204003b040 comp rx=0 tx=0).stop 2024-02-21T23:14:44.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.815+0000 7f2052800640 1 -- 172.21.15.28:0/2806490654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f204c150450 msgr2=0x7f204c1453f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:44.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.815+0000 7f2052800640 1 --2- 172.21.15.28:0/2806490654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f204c150450 0x7f204c1453f0 secure :-1 s=READY pgs=2497 cs=0 l=1 rev1=1 crypto rx=0x7f203800c430 tx=0x7f203800c8f0 comp rx=0 tx=0).stop 2024-02-21T23:14:44.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.815+0000 7f2052800640 1 -- 172.21.15.28:0/2806490654 shutdown_connections 2024-02-21T23:14:44.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.815+0000 7f2052800640 1 --2- 172.21.15.28:0/2806490654 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f202c06b4d0 0x7f202c06d990 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:44.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.815+0000 7f2052800640 1 --2- 172.21.15.28:0/2806490654 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f204c156af0 0x7f204c145930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:44.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.815+0000 7f2052800640 1 --2- 172.21.15.28:0/2806490654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f204c150450 0x7f204c1453f0 unknown :-1 s=CLOSED pgs=2497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:44.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.815+0000 7f2052800640 1 --2- 172.21.15.28:0/2806490654 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f204c06ee40 0x7f204c0a6d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:44.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.815+0000 7f2052800640 1 -- 172.21.15.28:0/2806490654 >> 172.21.15.28:0/2806490654 conn(0x7f204c06b630 msgr2=0x7f204c14e660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:14:44.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.816+0000 7f2052800640 1 -- 172.21.15.28:0/2806490654 shutdown_connections 2024-02-21T23:14:44.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:44.816+0000 7f2052800640 1 -- 172.21.15.28:0/2806490654 wait complete. 2024-02-21T23:14:45.226 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 243, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:14:43.651162+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:reconnect', 'state_seq': 22, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:14:45.227 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:reconnect 2024-02-21T23:14:46.227 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:14:46.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.466+0000 7f77dfe00640 1 Processor -- start 2024-02-21T23:14:46.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.467+0000 7f77dfe00640 1 -- start start 2024-02-21T23:14:46.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.468+0000 7f77dfe00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f77e00c7890 0x7f77e006b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:46.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.468+0000 7f77dfe00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f77e006b760 0x7f77e0150030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:46.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.468+0000 7f77dfe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f77e0150570 0x7f77e0152c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:46.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.468+0000 7f77dfe00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77e0070200 con 0x7f77e0150570 2024-02-21T23:14:46.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.468+0000 7f77dfe00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77e006bc70 con 0x7f77e00c7890 2024-02-21T23:14:46.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.468+0000 7f77dfe00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77e006f440 con 0x7f77e006b760 2024-02-21T23:14:46.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.468+0000 7f77df400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f77e0150570 0x7f77e0152c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:46.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.468+0000 7f77df400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f77e0150570 0x7f77e0152c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57898/0 (socket says 172.21.15.28:57898) 2024-02-21T23:14:46.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.468+0000 7f77df400640 1 -- 172.21.15.28:0/385611423 learned_addr learned my addr 172.21.15.28:0/385611423 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:14:46.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.469+0000 7f77de000640 1 --2- 172.21.15.28:0/385611423 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f77e006b760 0x7f77e0150030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:46.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.469+0000 7f77dea00640 1 --2- 172.21.15.28:0/385611423 >> v2:172.21.15.81:3300/0 conn(0x7f77e00c7890 0x7f77e006b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:46.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.469+0000 7f77df400640 1 -- 172.21.15.28:0/385611423 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f77e006b760 msgr2=0x7f77e0150030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:14:46.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.469+0000 7f77df400640 1 --2- 172.21.15.28:0/385611423 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f77e006b760 0x7f77e0150030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:46.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.469+0000 7f77df400640 1 -- 172.21.15.28:0/385611423 >> v2:172.21.15.81:3300/0 conn(0x7f77e00c7890 msgr2=0x7f77e006b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:14:46.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.469+0000 7f77df400640 1 --2- 172.21.15.28:0/385611423 >> v2:172.21.15.81:3300/0 conn(0x7f77e00c7890 0x7f77e006b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:46.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.469+0000 7f77df400640 1 -- 172.21.15.28:0/385611423 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77e006f720 con 0x7f77e0150570 2024-02-21T23:14:46.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.470+0000 7f77df400640 1 --2- 172.21.15.28:0/385611423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77e0150570 0x7f77e0152c30 secure :-1 s=READY pgs=2498 cs=0 l=1 rev1=1 crypto rx=0x7f77d8009870 tx=0x7f77d802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d41b2eb538c5cb06 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:46.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.470+0000 7f77dd600640 1 -- 172.21.15.28:0/385611423 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77d80317c0 con 0x7f77e0150570 2024-02-21T23:14:46.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.470+0000 7f77dd600640 1 -- 172.21.15.28:0/385611423 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77d8037070 con 0x7f77e0150570 2024-02-21T23:14:46.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.470+0000 7f77dd600640 1 -- 172.21.15.28:0/385611423 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77d8031b80 con 0x7f77e0150570 2024-02-21T23:14:46.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.471+0000 7f77dfe00640 1 -- 172.21.15.28:0/385611423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77e0150570 msgr2=0x7f77e0152c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:46.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.471+0000 7f77dfe00640 1 --2- 172.21.15.28:0/385611423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77e0150570 0x7f77e0152c30 secure :-1 s=READY pgs=2498 cs=0 l=1 rev1=1 crypto rx=0x7f77d8009870 tx=0x7f77d802b860 comp rx=0 tx=0).stop 2024-02-21T23:14:46.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.471+0000 7f77dfe00640 1 -- 172.21.15.28:0/385611423 shutdown_connections 2024-02-21T23:14:46.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.471+0000 7f77dfe00640 1 --2- 172.21.15.28:0/385611423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77e0150570 0x7f77e0152c30 unknown :-1 s=CLOSED pgs=2498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:46.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.471+0000 7f77dfe00640 1 --2- 172.21.15.28:0/385611423 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f77e006b760 0x7f77e0150030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:46.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.471+0000 7f77dfe00640 1 --2- 172.21.15.28:0/385611423 >> v2:172.21.15.81:3300/0 conn(0x7f77e00c7890 0x7f77e006b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:46.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.471+0000 7f77dfe00640 1 -- 172.21.15.28:0/385611423 >> 172.21.15.28:0/385611423 conn(0x7f77e0067050 msgr2=0x7f77e00a84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:14:46.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.471+0000 7f77dfe00640 1 -- 172.21.15.28:0/385611423 shutdown_connections 2024-02-21T23:14:46.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.471+0000 7f77dfe00640 1 -- 172.21.15.28:0/385611423 wait complete. 2024-02-21T23:14:46.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.473+0000 7f77dfe00640 1 Processor -- start 2024-02-21T23:14:46.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.473+0000 7f77dfe00640 1 -- start start 2024-02-21T23:14:46.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.474+0000 7f77dfe00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f77e006b760 0x7f77e01eb020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:46.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.474+0000 7f77dfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77e00c7890 0x7f77e00c6040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:46.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.474+0000 7f77dfe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f77e0150570 0x7f77e00aad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:46.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.474+0000 7f77dfe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77e00c4d10 con 0x7f77e00c7890 2024-02-21T23:14:46.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.474+0000 7f77dfe00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77e006fce0 con 0x7f77e0150570 2024-02-21T23:14:46.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.474+0000 7f77dfe00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77e00c4e90 con 0x7f77e006b760 2024-02-21T23:14:46.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.474+0000 7f77de000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77e00c7890 0x7f77e00c6040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:46.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.474+0000 7f77de000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77e00c7890 0x7f77e00c6040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57900/0 (socket says 172.21.15.28:57900) 2024-02-21T23:14:46.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.474+0000 7f77de000640 1 -- 172.21.15.28:0/1637230638 learned_addr learned my addr 172.21.15.28:0/1637230638 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:14:46.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.475+0000 7f77dea00640 1 --2- 172.21.15.28:0/1637230638 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f77e006b760 0x7f77e01eb020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:46.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.475+0000 7f77df400640 1 --2- 172.21.15.28:0/1637230638 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f77e0150570 0x7f77e00aad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:46.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.475+0000 7f77de000640 1 -- 172.21.15.28:0/1637230638 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f77e006b760 msgr2=0x7f77e01eb020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:46.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.475+0000 7f77de000640 1 --2- 172.21.15.28:0/1637230638 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f77e006b760 0x7f77e01eb020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:46.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.475+0000 7f77de000640 1 -- 172.21.15.28:0/1637230638 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f77e0150570 msgr2=0x7f77e00aad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:46.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.475+0000 7f77de000640 1 --2- 172.21.15.28:0/1637230638 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f77e0150570 0x7f77e00aad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:46.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.475+0000 7f77de000640 1 -- 172.21.15.28:0/1637230638 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77e00c0f40 con 0x7f77e00c7890 2024-02-21T23:14:46.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.475+0000 7f77de000640 1 --2- 172.21.15.28:0/1637230638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77e00c7890 0x7f77e00c6040 secure :-1 s=READY pgs=2499 cs=0 l=1 rev1=1 crypto rx=0x7f77cc00c820 tx=0x7f77cc00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:46.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.475+0000 7f77d7400640 1 -- 172.21.15.28:0/1637230638 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77cc015070 con 0x7f77e00c7890 2024-02-21T23:14:46.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.476+0000 7f77d7400640 1 -- 172.21.15.28:0/1637230638 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77cc004220 con 0x7f77e00c7890 2024-02-21T23:14:46.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.476+0000 7f77d7400640 1 -- 172.21.15.28:0/1637230638 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77cc00ce10 con 0x7f77e00c7890 2024-02-21T23:14:46.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.476+0000 7f77dfe00640 1 -- 172.21.15.28:0/1637230638 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77e01521b0 con 0x7f77e00c7890 2024-02-21T23:14:46.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.476+0000 7f77dfe00640 1 -- 172.21.15.28:0/1637230638 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77e00abfa0 con 0x7f77e00c7890 2024-02-21T23:14:46.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.477+0000 7f77dfe00640 1 -- 172.21.15.28:0/1637230638 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77e004f290 con 0x7f77e00c7890 2024-02-21T23:14:46.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.478+0000 7f77d7400640 1 -- 172.21.15.28:0/1637230638 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f77cc0077d0 con 0x7f77e00c7890 2024-02-21T23:14:46.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.479+0000 7f77d7400640 1 --2- 172.21.15.28:0/1637230638 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f77c006b4d0 0x7f77c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:46.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.479+0000 7f77d7400640 1 -- 172.21.15.28:0/1637230638 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f77cc08a600 con 0x7f77e00c7890 2024-02-21T23:14:46.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.480+0000 7f77dea00640 1 --2- 172.21.15.28:0/1637230638 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f77c006b4d0 0x7f77c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:46.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.481+0000 7f77dea00640 1 --2- 172.21.15.28:0/1637230638 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f77c006b4d0 0x7f77c006d990 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f77d0007130 tx=0x7f77d0007060 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:46.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.483+0000 7f77d7400640 1 -- 172.21.15.28:0/1637230638 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f77e00c0f40 con 0x7f77e00c7890 2024-02-21T23:14:46.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.726+0000 7f77dfe00640 1 -- 172.21.15.28:0/1637230638 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f77e00c1ab0 con 0x7f77e00c7890 2024-02-21T23:14:46.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.727+0000 7f77d7400640 1 -- 172.21.15.28:0/1637230638 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 246 v246) v1 ==== 95+0+3384 (secure 0 0 0) 0x7f77cc0591e0 con 0x7f77e00c7890 2024-02-21T23:14:46.716 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:14:46.716 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":246,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":246,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:14:46.655515+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:14:46.716 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 246 2024-02-21T23:14:46.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.730+0000 7f77dfe00640 1 -- 172.21.15.28:0/1637230638 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f77c006b4d0 msgr2=0x7f77c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:46.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.730+0000 7f77dfe00640 1 --2- 172.21.15.28:0/1637230638 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f77c006b4d0 0x7f77c006d990 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f77d0007130 tx=0x7f77d0007060 comp rx=0 tx=0).stop 2024-02-21T23:14:46.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.730+0000 7f77dfe00640 1 -- 172.21.15.28:0/1637230638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77e00c7890 msgr2=0x7f77e00c6040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:46.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.730+0000 7f77dfe00640 1 --2- 172.21.15.28:0/1637230638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77e00c7890 0x7f77e00c6040 secure :-1 s=READY pgs=2499 cs=0 l=1 rev1=1 crypto rx=0x7f77cc00c820 tx=0x7f77cc00cce0 comp rx=0 tx=0).stop 2024-02-21T23:14:46.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.730+0000 7f77dfe00640 1 -- 172.21.15.28:0/1637230638 shutdown_connections 2024-02-21T23:14:46.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.730+0000 7f77dfe00640 1 --2- 172.21.15.28:0/1637230638 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f77c006b4d0 0x7f77c006d990 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:46.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.730+0000 7f77dfe00640 1 --2- 172.21.15.28:0/1637230638 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f77e0150570 0x7f77e00aad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:46.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.730+0000 7f77dfe00640 1 --2- 172.21.15.28:0/1637230638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77e00c7890 0x7f77e00c6040 unknown :-1 s=CLOSED pgs=2499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:46.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.730+0000 7f77dfe00640 1 --2- 172.21.15.28:0/1637230638 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f77e006b760 0x7f77e01eb020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:46.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.730+0000 7f77dfe00640 1 -- 172.21.15.28:0/1637230638 >> 172.21.15.28:0/1637230638 conn(0x7f77e0067050 msgr2=0x7f77e014afc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:14:46.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.730+0000 7f77dfe00640 1 -- 172.21.15.28:0/1637230638 shutdown_connections 2024-02-21T23:14:46.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:46.730+0000 7f77dfe00640 1 -- 172.21.15.28:0/1637230638 wait complete. 2024-02-21T23:14:47.140 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 246, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:14:46.655515+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:14:47.140 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:14:48.141 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:14:48.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.372+0000 7fbeeb200640 1 Processor -- start 2024-02-21T23:14:48.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.372+0000 7fbeeb200640 1 -- start start 2024-02-21T23:14:48.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.373+0000 7fbeeb200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbeec150450 0x7fbeec0a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:48.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.373+0000 7fbeeb200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbeec06ee40 0x7fbeec1536f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:48.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.373+0000 7fbeeb200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbeec0c2760 con 0x7fbeec156af0 2024-02-21T23:14:48.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.374+0000 7fbeeb200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbeec0c22c0 con 0x7fbeec06ee40 2024-02-21T23:14:48.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.374+0000 7fbeeb200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbeec156660 con 0x7fbeec150450 2024-02-21T23:14:48.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.374+0000 7fbee9400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbeec06ee40 0x7fbeec1536f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:48.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.374+0000 7fbee9400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbeec06ee40 0x7fbeec1536f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57902/0 (socket says 172.21.15.28:57902) 2024-02-21T23:14:48.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.374+0000 7fbee9400640 1 -- 172.21.15.28:0/2932697743 learned_addr learned my addr 172.21.15.28:0/2932697743 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:14:48.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.374+0000 7fbee9e00640 1 --2- 172.21.15.28:0/2932697743 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbeec150450 0x7fbeec0a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:48.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.374+0000 7fbee9400640 1 -- 172.21.15.28:0/2932697743 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbeec150450 msgr2=0x7fbeec0a5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:14:48.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.374+0000 7fbee9400640 1 --2- 172.21.15.28:0/2932697743 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbeec150450 0x7fbeec0a5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:48.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.374+0000 7fbee9400640 1 -- 172.21.15.28:0/2932697743 >> v1:172.21.15.81:6789/0 conn(0x7fbeec156af0 legacy=0x7fbeec06dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:14:48.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.374+0000 7fbee9400640 1 -- 172.21.15.28:0/2932697743 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeec153c30 con 0x7fbeec06ee40 2024-02-21T23:14:48.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.376+0000 7fbee9400640 1 --2- 172.21.15.28:0/2932697743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbeec06ee40 0x7fbeec1536f0 secure :-1 s=READY pgs=2500 cs=0 l=1 rev1=1 crypto rx=0x7fbed8009870 tx=0x7fbed802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a091136a26e18c88 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:48.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.376+0000 7fbee8a00640 1 -- 172.21.15.28:0/2932697743 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbed8035020 con 0x7fbeec06ee40 2024-02-21T23:14:48.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.376+0000 7fbee8a00640 1 -- 172.21.15.28:0/2932697743 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbed8031640 con 0x7fbeec06ee40 2024-02-21T23:14:48.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.376+0000 7fbee8a00640 1 -- 172.21.15.28:0/2932697743 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbed802d040 con 0x7fbeec06ee40 2024-02-21T23:14:48.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.377+0000 7fbeeb200640 1 -- 172.21.15.28:0/2932697743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbeec06ee40 msgr2=0x7fbeec1536f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:48.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.377+0000 7fbeeb200640 1 --2- 172.21.15.28:0/2932697743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbeec06ee40 0x7fbeec1536f0 secure :-1 s=READY pgs=2500 cs=0 l=1 rev1=1 crypto rx=0x7fbed8009870 tx=0x7fbed802b8f0 comp rx=0 tx=0).stop 2024-02-21T23:14:48.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.377+0000 7fbeeb200640 1 -- 172.21.15.28:0/2932697743 shutdown_connections 2024-02-21T23:14:48.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.377+0000 7fbeeb200640 1 --2- 172.21.15.28:0/2932697743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbeec06ee40 0x7fbeec1536f0 unknown :-1 s=CLOSED pgs=2500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:48.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.377+0000 7fbeeb200640 1 --2- 172.21.15.28:0/2932697743 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbeec150450 0x7fbeec0a5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:48.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.377+0000 7fbeeb200640 1 -- 172.21.15.28:0/2932697743 >> 172.21.15.28:0/2932697743 conn(0x7fbeec06b630 msgr2=0x7fbeec0c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:14:48.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.377+0000 7fbeeb200640 1 -- 172.21.15.28:0/2932697743 shutdown_connections 2024-02-21T23:14:48.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.377+0000 7fbeeb200640 1 -- 172.21.15.28:0/2932697743 wait complete. 2024-02-21T23:14:48.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.379+0000 7fbeeb200640 1 Processor -- start 2024-02-21T23:14:48.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.380+0000 7fbeeb200640 1 -- start start 2024-02-21T23:14:48.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.381+0000 7fbeeb200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbeec06ee40 0x7fbeec1ee910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:48.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.381+0000 7fbeeb200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbeec150450 0x7fbeec0c5950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:48.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.381+0000 7fbeeb200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbeec156af0 0x7fbeec1eac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:48.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.381+0000 7fbeeb200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbeec0c2cb0 con 0x7fbeec150450 2024-02-21T23:14:48.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.381+0000 7fbeeb200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbeec0c29d0 con 0x7fbeec06ee40 2024-02-21T23:14:48.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.381+0000 7fbeeb200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbeec06e700 con 0x7fbeec156af0 2024-02-21T23:14:48.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.381+0000 7fbee9400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbeec150450 0x7fbeec0c5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:48.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.381+0000 7fbee9400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbeec150450 0x7fbeec0c5950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57918/0 (socket says 172.21.15.28:57918) 2024-02-21T23:14:48.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.381+0000 7fbee9400640 1 -- 172.21.15.28:0/1228242300 learned_addr learned my addr 172.21.15.28:0/1228242300 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:14:48.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.381+0000 7fbee9e00640 1 --2- 172.21.15.28:0/1228242300 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbeec06ee40 0x7fbeec1ee910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:48.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.381+0000 7fbeea800640 1 --2- 172.21.15.28:0/1228242300 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbeec156af0 0x7fbeec1eac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:48.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.382+0000 7fbee9400640 1 -- 172.21.15.28:0/1228242300 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbeec156af0 msgr2=0x7fbeec1eac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:48.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.382+0000 7fbee9400640 1 --2- 172.21.15.28:0/1228242300 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbeec156af0 0x7fbeec1eac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:48.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.382+0000 7fbee9400640 1 -- 172.21.15.28:0/1228242300 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbeec06ee40 msgr2=0x7fbeec1ee910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:48.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.382+0000 7fbee9400640 1 --2- 172.21.15.28:0/1228242300 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbeec06ee40 0x7fbeec1ee910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:48.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.382+0000 7fbee9400640 1 -- 172.21.15.28:0/1228242300 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeec072d80 con 0x7fbeec150450 2024-02-21T23:14:48.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.382+0000 7fbee9400640 1 --2- 172.21.15.28:0/1228242300 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbeec150450 0x7fbeec0c5950 secure :-1 s=READY pgs=2501 cs=0 l=1 rev1=1 crypto rx=0x7fbed80097c0 tx=0x7fbed80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:48.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.382+0000 7fbee2000640 1 -- 172.21.15.28:0/1228242300 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbed8040070 con 0x7fbeec150450 2024-02-21T23:14:48.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.382+0000 7fbee2000640 1 -- 172.21.15.28:0/1228242300 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbed80399f0 con 0x7fbeec150450 2024-02-21T23:14:48.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.382+0000 7fbee2000640 1 -- 172.21.15.28:0/1228242300 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbed8035040 con 0x7fbeec150450 2024-02-21T23:14:48.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.382+0000 7fbeeb200640 1 -- 172.21.15.28:0/1228242300 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbeec0727c0 con 0x7fbeec150450 2024-02-21T23:14:48.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.383+0000 7fbeeb200640 1 -- 172.21.15.28:0/1228242300 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbeec071780 con 0x7fbeec150450 2024-02-21T23:14:48.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.384+0000 7fbeeb200640 1 -- 172.21.15.28:0/1228242300 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbeec04f290 con 0x7fbeec150450 2024-02-21T23:14:48.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.385+0000 7fbee2000640 1 -- 172.21.15.28:0/1228242300 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbed804b060 con 0x7fbeec150450 2024-02-21T23:14:48.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.386+0000 7fbee2000640 1 --2- 172.21.15.28:0/1228242300 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbecc06b4d0 0x7fbecc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:48.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.386+0000 7fbee2000640 1 -- 172.21.15.28:0/1228242300 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fbed80abf70 con 0x7fbeec150450 2024-02-21T23:14:48.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.387+0000 7fbee9e00640 1 --2- 172.21.15.28:0/1228242300 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbecc06b4d0 0x7fbecc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:48.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.388+0000 7fbee9e00640 1 --2- 172.21.15.28:0/1228242300 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbecc06b4d0 0x7fbecc06d990 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fbedc006fd0 tx=0x7fbedc008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:48.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.391+0000 7fbee2000640 1 -- 172.21.15.28:0/1228242300 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fbed807ab50 con 0x7fbeec150450 2024-02-21T23:14:48.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.630+0000 7fbeeb200640 1 -- 172.21.15.28:0/1228242300 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbeec0c1f10 con 0x7fbeec150450 2024-02-21T23:14:48.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.631+0000 7fbee2000640 1 -- 172.21.15.28:0/1228242300 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 246 v246) v1 ==== 95+0+3384 (secure 0 0 0) 0x7fbed807a450 con 0x7fbeec150450 2024-02-21T23:14:48.620 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:14:48.620 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":246,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":246,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:14:46.655515+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:14:48.620 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 246 2024-02-21T23:14:48.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.634+0000 7fbeeb200640 1 -- 172.21.15.28:0/1228242300 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbecc06b4d0 msgr2=0x7fbecc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:48.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.634+0000 7fbeeb200640 1 --2- 172.21.15.28:0/1228242300 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbecc06b4d0 0x7fbecc06d990 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fbedc006fd0 tx=0x7fbedc008040 comp rx=0 tx=0).stop 2024-02-21T23:14:48.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.634+0000 7fbeeb200640 1 -- 172.21.15.28:0/1228242300 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbeec150450 msgr2=0x7fbeec0c5950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:48.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.634+0000 7fbeeb200640 1 --2- 172.21.15.28:0/1228242300 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbeec150450 0x7fbeec0c5950 secure :-1 s=READY pgs=2501 cs=0 l=1 rev1=1 crypto rx=0x7fbed80097c0 tx=0x7fbed80046c0 comp rx=0 tx=0).stop 2024-02-21T23:14:48.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.634+0000 7fbeeb200640 1 -- 172.21.15.28:0/1228242300 shutdown_connections 2024-02-21T23:14:48.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.634+0000 7fbeeb200640 1 --2- 172.21.15.28:0/1228242300 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbecc06b4d0 0x7fbecc06d990 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:48.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.634+0000 7fbeeb200640 1 --2- 172.21.15.28:0/1228242300 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbeec156af0 0x7fbeec1eac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:48.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.634+0000 7fbeeb200640 1 --2- 172.21.15.28:0/1228242300 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbeec150450 0x7fbeec0c5950 unknown :-1 s=CLOSED pgs=2501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:48.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.634+0000 7fbeeb200640 1 --2- 172.21.15.28:0/1228242300 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbeec06ee40 0x7fbeec1ee910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:48.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.634+0000 7fbeeb200640 1 -- 172.21.15.28:0/1228242300 >> 172.21.15.28:0/1228242300 conn(0x7fbeec06b630 msgr2=0x7fbeec0c2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:14:48.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.634+0000 7fbeeb200640 1 -- 172.21.15.28:0/1228242300 shutdown_connections 2024-02-21T23:14:48.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:48.634+0000 7fbeeb200640 1 -- 172.21.15.28:0/1228242300 wait complete. 2024-02-21T23:14:49.043 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 246, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:14:46.655515+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:14:49.043 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:14:50.044 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:14:50.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.290+0000 7fa6cf800640 1 Processor -- start 2024-02-21T23:14:50.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.290+0000 7fa6cf800640 1 -- start start 2024-02-21T23:14:50.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.293+0000 7fa6cf800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fa6c80a5f40 0x7fa6c806dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:50.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.293+0000 7fa6cf800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa6c80c0270 0x7fa6c80c06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:50.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.293+0000 7fa6cf800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6c806e750 con 0x7fa6c80c3490 2024-02-21T23:14:50.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.293+0000 7fa6cf800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6c806eff0 con 0x7fa6c80c0270 2024-02-21T23:14:50.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.293+0000 7fa6cf800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6c80bfde0 con 0x7fa6c80a5f40 2024-02-21T23:14:50.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.293+0000 7fa6cce00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa6c80c3490 0x7fa6c8152140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60110/0 (socket says 172.21.15.28:60110) 2024-02-21T23:14:50.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.293+0000 7fa6cce00640 1 -- 172.21.15.28:0/3133008235 learned_addr learned my addr 172.21.15.28:0/3133008235 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:14:50.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.293+0000 7fa6cd800640 1 --2- 172.21.15.28:0/3133008235 >> v2:172.21.15.28:3300/0 conn(0x7fa6c80c0270 0x7fa6c80c06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:50.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.293+0000 7fa6c7e00640 1 --2- 172.21.15.28:0/3133008235 >> v2:172.21.15.81:3300/0 conn(0x7fa6c80a5f40 0x7fa6c806dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:50.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.293+0000 7fa6c7400640 1 -- 172.21.15.28:0/3133008235 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 63039464 0 0) 0x7fa6c806e750 con 0x7fa6c80c3490 2024-02-21T23:14:50.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.293+0000 7fa6c7400640 1 -- 172.21.15.28:0/3133008235 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6b0003520 con 0x7fa6c80c3490 2024-02-21T23:14:50.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.294+0000 7fa6cd800640 1 -- 172.21.15.28:0/3133008235 >> v2:172.21.15.81:3300/0 conn(0x7fa6c80a5f40 msgr2=0x7fa6c806dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:14:50.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.294+0000 7fa6cd800640 1 --2- 172.21.15.28:0/3133008235 >> v2:172.21.15.81:3300/0 conn(0x7fa6c80a5f40 0x7fa6c806dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:50.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.294+0000 7fa6cd800640 1 -- 172.21.15.28:0/3133008235 >> v1:172.21.15.28:6789/0 conn(0x7fa6c80c3490 legacy=0x7fa6c8152140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:50.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.294+0000 7fa6cd800640 1 -- 172.21.15.28:0/3133008235 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6c806ea30 con 0x7fa6c80c0270 2024-02-21T23:14:50.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.295+0000 7fa6cd800640 1 --2- 172.21.15.28:0/3133008235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6c80c0270 0x7fa6c80c06e0 secure :-1 s=READY pgs=2503 cs=0 l=1 rev1=1 crypto rx=0x7fa6c80c7610 tx=0x7fa6c002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53ee6cc9c4982d15 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:50.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.295+0000 7fa6c7400640 1 -- 172.21.15.28:0/3133008235 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c0039070 con 0x7fa6c80c0270 2024-02-21T23:14:50.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.295+0000 7fa6c7400640 1 -- 172.21.15.28:0/3133008235 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6c002cbf0 con 0x7fa6c80c0270 2024-02-21T23:14:50.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.295+0000 7fa6c7400640 1 -- 172.21.15.28:0/3133008235 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c0034440 con 0x7fa6c80c0270 2024-02-21T23:14:50.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.296+0000 7fa6cf800640 1 -- 172.21.15.28:0/3133008235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6c80c0270 msgr2=0x7fa6c80c06e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:50.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.296+0000 7fa6cf800640 1 --2- 172.21.15.28:0/3133008235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6c80c0270 0x7fa6c80c06e0 secure :-1 s=READY pgs=2503 cs=0 l=1 rev1=1 crypto rx=0x7fa6c80c7610 tx=0x7fa6c002be40 comp rx=0 tx=0).stop 2024-02-21T23:14:50.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.296+0000 7fa6cf800640 1 -- 172.21.15.28:0/3133008235 shutdown_connections 2024-02-21T23:14:50.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.296+0000 7fa6cf800640 1 --2- 172.21.15.28:0/3133008235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6c80c0270 0x7fa6c80c06e0 unknown :-1 s=CLOSED pgs=2503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:50.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.296+0000 7fa6cf800640 1 --2- 172.21.15.28:0/3133008235 >> v2:172.21.15.81:3300/0 conn(0x7fa6c80a5f40 0x7fa6c806dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:50.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.296+0000 7fa6cf800640 1 -- 172.21.15.28:0/3133008235 >> 172.21.15.28:0/3133008235 conn(0x7fa6c80c67f0 msgr2=0x7fa6c80c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:14:50.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.296+0000 7fa6cf800640 1 -- 172.21.15.28:0/3133008235 shutdown_connections 2024-02-21T23:14:50.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.296+0000 7fa6cf800640 1 -- 172.21.15.28:0/3133008235 wait complete. 2024-02-21T23:14:50.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.297+0000 7fa6cf800640 1 Processor -- start 2024-02-21T23:14:50.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.298+0000 7fa6cf800640 1 -- start start 2024-02-21T23:14:50.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.299+0000 7fa6cf800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa6c80a5f40 0x7fa6c80a77f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:50.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.299+0000 7fa6cf800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa6c80c0270 0x7fa6c80ab5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:50.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.299+0000 7fa6cf800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6c80c3490 0x7fa6c81eae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:50.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.299+0000 7fa6cf800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c80c0c20 con 0x7fa6c80c3490 2024-02-21T23:14:50.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.299+0000 7fa6cf800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c806e4f0 con 0x7fa6c80c0270 2024-02-21T23:14:50.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.299+0000 7fa6cf800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c80c1c70 con 0x7fa6c80a5f40 2024-02-21T23:14:50.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.299+0000 7fa6cd800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6c80c3490 0x7fa6c81eae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:50.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.299+0000 7fa6cd800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6c80c3490 0x7fa6c81eae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57934/0 (socket says 172.21.15.28:57934) 2024-02-21T23:14:50.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.299+0000 7fa6cd800640 1 -- 172.21.15.28:0/1101202875 learned_addr learned my addr 172.21.15.28:0/1101202875 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:14:50.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.299+0000 7fa6cce00640 1 --2- 172.21.15.28:0/1101202875 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa6c80a5f40 0x7fa6c80a77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:50.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.299+0000 7fa6c7e00640 1 --2- 172.21.15.28:0/1101202875 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa6c80c0270 0x7fa6c80ab5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:50.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.299+0000 7fa6cd800640 1 -- 172.21.15.28:0/1101202875 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa6c80a5f40 msgr2=0x7fa6c80a77f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:50.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.299+0000 7fa6cd800640 1 --2- 172.21.15.28:0/1101202875 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa6c80a5f40 0x7fa6c80a77f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:50.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.299+0000 7fa6cd800640 1 -- 172.21.15.28:0/1101202875 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa6c80c0270 msgr2=0x7fa6c80ab5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:50.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.299+0000 7fa6cd800640 1 --2- 172.21.15.28:0/1101202875 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa6c80c0270 0x7fa6c80ab5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:50.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.299+0000 7fa6cd800640 1 -- 172.21.15.28:0/1101202875 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6c8144960 con 0x7fa6c80c3490 2024-02-21T23:14:50.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.299+0000 7fa6c7e00640 1 --2- 172.21.15.28:0/1101202875 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa6c80c0270 0x7fa6c80ab5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:14:50.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.299+0000 7fa6cce00640 1 --2- 172.21.15.28:0/1101202875 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa6c80a5f40 0x7fa6c80a77f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:14:50.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.300+0000 7fa6cd800640 1 --2- 172.21.15.28:0/1101202875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6c80c3490 0x7fa6c81eae40 secure :-1 s=READY pgs=2504 cs=0 l=1 rev1=1 crypto rx=0x7fa6c0009e80 tx=0x7fa6c0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:50.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.300+0000 7fa6c4c00640 1 -- 172.21.15.28:0/1101202875 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c0039050 con 0x7fa6c80c3490 2024-02-21T23:14:50.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.300+0000 7fa6c4c00640 1 -- 172.21.15.28:0/1101202875 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6c0045070 con 0x7fa6c80c3490 2024-02-21T23:14:50.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.300+0000 7fa6c4c00640 1 -- 172.21.15.28:0/1101202875 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c0009450 con 0x7fa6c80c3490 2024-02-21T23:14:50.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.300+0000 7fa6cf800640 1 -- 172.21.15.28:0/1101202875 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6c81eec30 con 0x7fa6c80c3490 2024-02-21T23:14:50.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.300+0000 7fa6cf800640 1 -- 172.21.15.28:0/1101202875 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6c8144d10 con 0x7fa6c80c3490 2024-02-21T23:14:50.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.301+0000 7fa6cf800640 1 -- 172.21.15.28:0/1101202875 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6c81eea20 con 0x7fa6c80c3490 2024-02-21T23:14:50.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.302+0000 7fa6c4c00640 1 -- 172.21.15.28:0/1101202875 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa6c003e850 con 0x7fa6c80c3490 2024-02-21T23:14:50.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.303+0000 7fa6c4c00640 1 --2- 172.21.15.28:0/1101202875 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa6a806b4d0 0x7fa6a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:50.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.303+0000 7fa6c4c00640 1 -- 172.21.15.28:0/1101202875 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fa6c00ad110 con 0x7fa6c80c3490 2024-02-21T23:14:50.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.304+0000 7fa6cce00640 1 --2- 172.21.15.28:0/1101202875 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa6a806b4d0 0x7fa6a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:50.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.305+0000 7fa6cce00640 1 --2- 172.21.15.28:0/1101202875 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa6a806b4d0 0x7fa6a806d990 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc005880 tx=0x7fa6bc009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:50.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.308+0000 7fa6c4c00640 1 -- 172.21.15.28:0/1101202875 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fa6c007be20 con 0x7fa6c80c3490 2024-02-21T23:14:50.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.546+0000 7fa6cf800640 1 -- 172.21.15.28:0/1101202875 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6c8067f30 con 0x7fa6c80c3490 2024-02-21T23:14:50.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.547+0000 7fa6c4c00640 1 -- 172.21.15.28:0/1101202875 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 246 v246) v1 ==== 95+0+3384 (secure 0 0 0) 0x7fa6c007b720 con 0x7fa6c80c3490 2024-02-21T23:14:50.536 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:14:50.536 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":246,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":246,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:14:46.655515+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:14:50.536 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 246 2024-02-21T23:14:50.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.550+0000 7fa6cf800640 1 -- 172.21.15.28:0/1101202875 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa6a806b4d0 msgr2=0x7fa6a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:50.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.550+0000 7fa6cf800640 1 --2- 172.21.15.28:0/1101202875 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa6a806b4d0 0x7fa6a806d990 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc005880 tx=0x7fa6bc009040 comp rx=0 tx=0).stop 2024-02-21T23:14:50.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.550+0000 7fa6cf800640 1 -- 172.21.15.28:0/1101202875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6c80c3490 msgr2=0x7fa6c81eae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:50.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.550+0000 7fa6cf800640 1 --2- 172.21.15.28:0/1101202875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6c80c3490 0x7fa6c81eae40 secure :-1 s=READY pgs=2504 cs=0 l=1 rev1=1 crypto rx=0x7fa6c0009e80 tx=0x7fa6c0004a40 comp rx=0 tx=0).stop 2024-02-21T23:14:50.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.550+0000 7fa6cf800640 1 -- 172.21.15.28:0/1101202875 shutdown_connections 2024-02-21T23:14:50.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.550+0000 7fa6cf800640 1 --2- 172.21.15.28:0/1101202875 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa6a806b4d0 0x7fa6a806d990 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:50.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.550+0000 7fa6cf800640 1 --2- 172.21.15.28:0/1101202875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6c80c3490 0x7fa6c81eae40 unknown :-1 s=CLOSED pgs=2504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:50.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.550+0000 7fa6cf800640 1 --2- 172.21.15.28:0/1101202875 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa6c80c0270 0x7fa6c80ab5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:50.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.550+0000 7fa6cf800640 1 --2- 172.21.15.28:0/1101202875 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa6c80a5f40 0x7fa6c80a77f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:50.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.550+0000 7fa6cf800640 1 -- 172.21.15.28:0/1101202875 >> 172.21.15.28:0/1101202875 conn(0x7fa6c80c67f0 msgr2=0x7fa6c814f2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:14:50.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.550+0000 7fa6cf800640 1 -- 172.21.15.28:0/1101202875 shutdown_connections 2024-02-21T23:14:50.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:50.550+0000 7fa6cf800640 1 -- 172.21.15.28:0/1101202875 wait complete. 2024-02-21T23:14:50.960 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 246, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:14:46.655515+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:14:50.961 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:14:51.962 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:14:52.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.202+0000 7f3e0a800640 1 Processor -- start 2024-02-21T23:14:52.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.202+0000 7f3e0a800640 1 -- start start 2024-02-21T23:14:52.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.203+0000 7f3e0a800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3e0406ee40 0x7f3e040a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:52.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.203+0000 7f3e0a800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e040c2760 con 0x7f3e04150450 2024-02-21T23:14:52.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.203+0000 7f3e0a800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e04156310 con 0x7f3e04156af0 2024-02-21T23:14:52.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.203+0000 7f3e0a800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e040c22c0 con 0x7f3e0406ee40 2024-02-21T23:14:52.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.204+0000 7f3e09400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f3e04150450 0x7f3e041536f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60122/0 (socket says 172.21.15.28:60122) 2024-02-21T23:14:52.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.204+0000 7f3e09400640 1 -- 172.21.15.28:0/324796896 learned_addr learned my addr 172.21.15.28:0/324796896 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:14:52.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.204+0000 7f3e08a00640 1 --2- 172.21.15.28:0/324796896 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3e0406ee40 0x7f3e040a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:52.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.205+0000 7f3e03e00640 1 -- 172.21.15.28:0/324796896 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 175766950 0 0) 0x7f3e040c2760 con 0x7f3e04150450 2024-02-21T23:14:52.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.205+0000 7f3e03e00640 1 -- 172.21.15.28:0/324796896 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3de8003520 con 0x7f3e04150450 2024-02-21T23:14:52.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.205+0000 7f3e03e00640 1 -- 172.21.15.28:0/324796896 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4063056882 0 0) 0x7f3e04156310 con 0x7f3e04156af0 2024-02-21T23:14:52.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.205+0000 7f3e03e00640 1 -- 172.21.15.28:0/324796896 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e040c2760 con 0x7f3e04156af0 2024-02-21T23:14:52.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.205+0000 7f3e03e00640 1 -- 172.21.15.28:0/324796896 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1827557253 0 0) 0x7f3de8003520 con 0x7f3e04150450 2024-02-21T23:14:52.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.205+0000 7f3e03e00640 1 -- 172.21.15.28:0/324796896 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3e0406ee40 msgr2=0x7f3e040a5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:14:52.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.205+0000 7f3e03e00640 1 --2- 172.21.15.28:0/324796896 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3e0406ee40 0x7f3e040a5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:52.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.205+0000 7f3e03e00640 1 -- 172.21.15.28:0/324796896 >> v1:172.21.15.81:6789/0 conn(0x7f3e04156af0 legacy=0x7f3e0406dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:52.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.205+0000 7f3e03e00640 1 -- 172.21.15.28:0/324796896 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e040c29d0 con 0x7f3e04150450 2024-02-21T23:14:52.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.205+0000 7f3e03e00640 1 -- 172.21.15.28:0/324796896 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f3df40019f0 con 0x7f3e04150450 2024-02-21T23:14:52.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.205+0000 7f3e03e00640 1 -- 172.21.15.28:0/324796896 >> v1:172.21.15.28:6789/0 conn(0x7f3e04150450 legacy=0x7f3e041536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:52.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.205+0000 7f3e03e00640 1 --2- 172.21.15.28:0/324796896 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3de8003b70 0x7f3de8003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:52.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.205+0000 7f3e03e00640 1 --2- 172.21.15.28:0/324796896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3de8004800 0x7f3de8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:52.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.205+0000 7f3e03e00640 1 --2- 172.21.15.28:0/324796896 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3de80051c0 0x7f3de800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:52.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.205+0000 7f3e03e00640 1 -- 172.21.15.28:0/324796896 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3de800bfb0 con 0x7f3de8004800 2024-02-21T23:14:52.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.205+0000 7f3e03e00640 1 -- 172.21.15.28:0/324796896 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3de800c130 con 0x7f3de80051c0 2024-02-21T23:14:52.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.205+0000 7f3e03e00640 1 -- 172.21.15.28:0/324796896 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3de800c330 con 0x7f3de8003b70 2024-02-21T23:14:52.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.206+0000 7f3e08a00640 1 --2- 172.21.15.28:0/324796896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3de8004800 0x7f3de8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:52.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.206+0000 7f3e09e00640 1 --2- 172.21.15.28:0/324796896 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3de80051c0 0x7f3de800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:52.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.206+0000 7f3e09400640 1 --2- 172.21.15.28:0/324796896 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3de8003b70 0x7f3de8003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:52.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.206+0000 7f3e08a00640 1 -- 172.21.15.28:0/324796896 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3de8003b70 msgr2=0x7f3de8003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:14:52.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.206+0000 7f3e08a00640 1 --2- 172.21.15.28:0/324796896 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3de8003b70 0x7f3de8003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:52.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.206+0000 7f3e08a00640 1 -- 172.21.15.28:0/324796896 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3de80051c0 msgr2=0x7f3de800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:14:52.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.206+0000 7f3e08a00640 1 --2- 172.21.15.28:0/324796896 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3de80051c0 0x7f3de800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:52.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.206+0000 7f3e08a00640 1 -- 172.21.15.28:0/324796896 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3de8003520 con 0x7f3de8004800 2024-02-21T23:14:52.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.207+0000 7f3e08a00640 1 --2- 172.21.15.28:0/324796896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3de8004800 0x7f3de8004c80 secure :-1 s=READY pgs=2506 cs=0 l=1 rev1=1 crypto rx=0x7f3e040ac700 tx=0x7f3de8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c371fd19d3869c5 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:52.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.208+0000 7f3e03e00640 1 -- 172.21.15.28:0/324796896 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df0039070 con 0x7f3de8004800 2024-02-21T23:14:52.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.208+0000 7f3e03e00640 1 -- 172.21.15.28:0/324796896 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3df002c580 con 0x7f3de8004800 2024-02-21T23:14:52.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.208+0000 7f3e03e00640 1 -- 172.21.15.28:0/324796896 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df0009650 con 0x7f3de8004800 2024-02-21T23:14:52.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.208+0000 7f3e0a800640 1 -- 172.21.15.28:0/324796896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3de8004800 msgr2=0x7f3de8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:52.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.208+0000 7f3e0a800640 1 --2- 172.21.15.28:0/324796896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3de8004800 0x7f3de8004c80 secure :-1 s=READY pgs=2506 cs=0 l=1 rev1=1 crypto rx=0x7f3e040ac700 tx=0x7f3de8005830 comp rx=0 tx=0).stop 2024-02-21T23:14:52.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.208+0000 7f3e09e00640 1 -- 172.21.15.28:0/324796896 reap_dead start 2024-02-21T23:14:52.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.208+0000 7f3e0a800640 1 -- 172.21.15.28:0/324796896 shutdown_connections 2024-02-21T23:14:52.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.208+0000 7f3e0a800640 1 -- 172.21.15.28:0/324796896 >> 172.21.15.28:0/324796896 conn(0x7f3e0406b630 msgr2=0x7f3e040c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:14:52.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.208+0000 7f3e0a800640 1 -- 172.21.15.28:0/324796896 shutdown_connections 2024-02-21T23:14:52.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.209+0000 7f3e0a800640 1 -- 172.21.15.28:0/324796896 wait complete. 2024-02-21T23:14:52.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.211+0000 7f3e0a800640 1 Processor -- start 2024-02-21T23:14:52.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.211+0000 7f3e0a800640 1 -- start start 2024-02-21T23:14:52.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.212+0000 7f3e0a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3de8004800 0x7f3e0414cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:52.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.212+0000 7f3e0a800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3e041536d0 0x7f3e0413f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:52.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.212+0000 7f3e0a800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3e040a9120 0x7f3e0413fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:52.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.212+0000 7f3e0a800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e0406e700 con 0x7f3de8004800 2024-02-21T23:14:52.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.212+0000 7f3e0a800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e04156660 con 0x7f3e040a9120 2024-02-21T23:14:52.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.212+0000 7f3e0a800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e0406e880 con 0x7f3e041536d0 2024-02-21T23:14:52.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.213+0000 7f3e09400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3de8004800 0x7f3e0414cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:52.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.213+0000 7f3e09400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3de8004800 0x7f3e0414cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57948/0 (socket says 172.21.15.28:57948) 2024-02-21T23:14:52.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.213+0000 7f3e09400640 1 -- 172.21.15.28:0/3209245476 learned_addr learned my addr 172.21.15.28:0/3209245476 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:14:52.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.213+0000 7f3e09e00640 1 --2- 172.21.15.28:0/3209245476 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3e040a9120 0x7f3e0413fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:52.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.213+0000 7f3e08a00640 1 --2- 172.21.15.28:0/3209245476 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3e041536d0 0x7f3e0413f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:52.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.213+0000 7f3e09400640 1 -- 172.21.15.28:0/3209245476 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3e041536d0 msgr2=0x7f3e0413f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:52.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.213+0000 7f3e09400640 1 --2- 172.21.15.28:0/3209245476 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3e041536d0 0x7f3e0413f590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:52.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.213+0000 7f3e09400640 1 -- 172.21.15.28:0/3209245476 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3e040a9120 msgr2=0x7f3e0413fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:52.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.213+0000 7f3e09400640 1 --2- 172.21.15.28:0/3209245476 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3e040a9120 0x7f3e0413fad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:52.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.213+0000 7f3e09400640 1 -- 172.21.15.28:0/3209245476 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e04140010 con 0x7f3de8004800 2024-02-21T23:14:52.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.214+0000 7f3e09400640 1 --2- 172.21.15.28:0/3209245476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3de8004800 0x7f3e0414cc90 secure :-1 s=READY pgs=2507 cs=0 l=1 rev1=1 crypto rx=0x7f3df40079c0 tx=0x7f3df400e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:52.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.214+0000 7f3e01600640 1 -- 172.21.15.28:0/3209245476 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df400ee50 con 0x7f3de8004800 2024-02-21T23:14:52.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.214+0000 7f3e01600640 1 -- 172.21.15.28:0/3209245476 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3df4018070 con 0x7f3de8004800 2024-02-21T23:14:52.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.214+0000 7f3e01600640 1 -- 172.21.15.28:0/3209245476 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df4013660 con 0x7f3de8004800 2024-02-21T23:14:52.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.214+0000 7f3e0a800640 1 -- 172.21.15.28:0/3209245476 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e041e8b50 con 0x7f3de8004800 2024-02-21T23:14:52.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.214+0000 7f3e0a800640 1 -- 172.21.15.28:0/3209245476 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e041e9110 con 0x7f3de8004800 2024-02-21T23:14:52.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.215+0000 7f3e0a800640 1 -- 172.21.15.28:0/3209245476 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e0404f290 con 0x7f3de8004800 2024-02-21T23:14:52.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.216+0000 7f3e01600640 1 -- 172.21.15.28:0/3209245476 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3df4013800 con 0x7f3de8004800 2024-02-21T23:14:52.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.217+0000 7f3e01600640 1 --2- 172.21.15.28:0/3209245476 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3de406b4d0 0x7f3de406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:52.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.218+0000 7f3e01600640 1 -- 172.21.15.28:0/3209245476 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f3df408aac0 con 0x7f3de8004800 2024-02-21T23:14:52.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.218+0000 7f3e08a00640 1 --2- 172.21.15.28:0/3209245476 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3de406b4d0 0x7f3de406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:52.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.219+0000 7f3e08a00640 1 --2- 172.21.15.28:0/3209245476 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3de406b4d0 0x7f3de406d990 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f3df0036000 tx=0x7f3df0030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:52.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.222+0000 7f3e01600640 1 -- 172.21.15.28:0/3209245476 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3df40596a0 con 0x7f3de8004800 2024-02-21T23:14:52.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.458+0000 7f3e0a800640 1 -- 172.21.15.28:0/3209245476 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3e04156310 con 0x7f3de8004800 2024-02-21T23:14:52.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.459+0000 7f3e01600640 1 -- 172.21.15.28:0/3209245476 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 246 v246) v1 ==== 95+0+3384 (secure 0 0 0) 0x7f3df4058fa0 con 0x7f3de8004800 2024-02-21T23:14:52.447 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:14:52.448 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":246,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":246,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:14:46.655515+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:14:52.448 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 246 2024-02-21T23:14:52.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.461+0000 7f3e0a800640 1 -- 172.21.15.28:0/3209245476 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3de406b4d0 msgr2=0x7f3de406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:52.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.461+0000 7f3e0a800640 1 --2- 172.21.15.28:0/3209245476 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3de406b4d0 0x7f3de406d990 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f3df0036000 tx=0x7f3df0030000 comp rx=0 tx=0).stop 2024-02-21T23:14:52.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.461+0000 7f3e0a800640 1 -- 172.21.15.28:0/3209245476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3de8004800 msgr2=0x7f3e0414cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:52.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.461+0000 7f3e0a800640 1 --2- 172.21.15.28:0/3209245476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3de8004800 0x7f3e0414cc90 secure :-1 s=READY pgs=2507 cs=0 l=1 rev1=1 crypto rx=0x7f3df40079c0 tx=0x7f3df400e530 comp rx=0 tx=0).stop 2024-02-21T23:14:52.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.462+0000 7f3e0a800640 1 -- 172.21.15.28:0/3209245476 shutdown_connections 2024-02-21T23:14:52.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.462+0000 7f3e0a800640 1 --2- 172.21.15.28:0/3209245476 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3de406b4d0 0x7f3de406d990 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:52.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.462+0000 7f3e0a800640 1 --2- 172.21.15.28:0/3209245476 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3e041536d0 0x7f3e0413f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:52.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.462+0000 7f3e0a800640 1 --2- 172.21.15.28:0/3209245476 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3e040a9120 0x7f3e0413fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:52.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.462+0000 7f3e0a800640 1 --2- 172.21.15.28:0/3209245476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3de8004800 0x7f3e0414cc90 unknown :-1 s=CLOSED pgs=2507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:52.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.462+0000 7f3e0a800640 1 -- 172.21.15.28:0/3209245476 >> 172.21.15.28:0/3209245476 conn(0x7f3e0406b630 msgr2=0x7f3e040c64e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:14:52.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.462+0000 7f3e0a800640 1 -- 172.21.15.28:0/3209245476 shutdown_connections 2024-02-21T23:14:52.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:52.462+0000 7f3e0a800640 1 -- 172.21.15.28:0/3209245476 wait complete. 2024-02-21T23:14:52.871 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 246, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:14:46.655515+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:14:52.871 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:14:53.872 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:14:54.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.094+0000 7f1f1f200640 1 Processor -- start 2024-02-21T23:14:54.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.094+0000 7f1f1f200640 1 -- start start 2024-02-21T23:14:54.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.096+0000 7f1f1f200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1f20150450 0x7f1f200a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:54.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.096+0000 7f1f1f200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1f2006ee40 0x7f1f201536f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:54.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.096+0000 7f1f1f200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f200c2760 con 0x7f1f20156af0 2024-02-21T23:14:54.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.096+0000 7f1f1f200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f200c22c0 con 0x7f1f2006ee40 2024-02-21T23:14:54.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.096+0000 7f1f1f200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f20156660 con 0x7f1f20150450 2024-02-21T23:14:54.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.096+0000 7f1f1d400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1f2006ee40 0x7f1f201536f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:54.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.096+0000 7f1f1d400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1f2006ee40 0x7f1f201536f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57958/0 (socket says 172.21.15.28:57958) 2024-02-21T23:14:54.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.096+0000 7f1f1d400640 1 -- 172.21.15.28:0/3069645656 learned_addr learned my addr 172.21.15.28:0/3069645656 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:14:54.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.096+0000 7f1f1e800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f1f20156af0 0x7f1f2006dff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60136/0 (socket says 172.21.15.28:60136) 2024-02-21T23:14:54.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.097+0000 7f1f1de00640 1 --2- 172.21.15.28:0/3069645656 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1f20150450 0x7f1f200a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:54.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.097+0000 7f1f1d400640 1 -- 172.21.15.28:0/3069645656 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1f20150450 msgr2=0x7f1f200a5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:14:54.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.097+0000 7f1f1d400640 1 --2- 172.21.15.28:0/3069645656 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1f20150450 0x7f1f200a5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:54.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.097+0000 7f1f1d400640 1 -- 172.21.15.28:0/3069645656 >> v1:172.21.15.28:6789/0 conn(0x7f1f20156af0 legacy=0x7f1f2006dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:54.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.097+0000 7f1f1d400640 1 -- 172.21.15.28:0/3069645656 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f20153c30 con 0x7f1f2006ee40 2024-02-21T23:14:54.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.098+0000 7f1f1d400640 1 --2- 172.21.15.28:0/3069645656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f2006ee40 0x7f1f201536f0 secure :-1 s=READY pgs=2509 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c009870 tx=0x7f1f0c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22eefe666544629d server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:54.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.098+0000 7f1f1ca00640 1 -- 172.21.15.28:0/3069645656 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f0c034070 con 0x7f1f2006ee40 2024-02-21T23:14:54.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.098+0000 7f1f1ca00640 1 -- 172.21.15.28:0/3069645656 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f0c030840 con 0x7f1f2006ee40 2024-02-21T23:14:54.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.098+0000 7f1f1ca00640 1 -- 172.21.15.28:0/3069645656 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f0c030c10 con 0x7f1f2006ee40 2024-02-21T23:14:54.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.099+0000 7f1f1f200640 1 -- 172.21.15.28:0/3069645656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f2006ee40 msgr2=0x7f1f201536f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:54.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.099+0000 7f1f1f200640 1 --2- 172.21.15.28:0/3069645656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f2006ee40 0x7f1f201536f0 secure :-1 s=READY pgs=2509 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c009870 tx=0x7f1f0c02b980 comp rx=0 tx=0).stop 2024-02-21T23:14:54.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.099+0000 7f1f1f200640 1 -- 172.21.15.28:0/3069645656 shutdown_connections 2024-02-21T23:14:54.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.099+0000 7f1f1f200640 1 --2- 172.21.15.28:0/3069645656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f2006ee40 0x7f1f201536f0 unknown :-1 s=CLOSED pgs=2509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:54.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.099+0000 7f1f1f200640 1 --2- 172.21.15.28:0/3069645656 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1f20150450 0x7f1f200a5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:54.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.099+0000 7f1f1f200640 1 -- 172.21.15.28:0/3069645656 >> 172.21.15.28:0/3069645656 conn(0x7f1f2006b630 msgr2=0x7f1f200c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:14:54.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.099+0000 7f1f1f200640 1 -- 172.21.15.28:0/3069645656 shutdown_connections 2024-02-21T23:14:54.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.099+0000 7f1f1f200640 1 -- 172.21.15.28:0/3069645656 wait complete. 2024-02-21T23:14:54.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.101+0000 7f1f1f200640 1 Processor -- start 2024-02-21T23:14:54.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.101+0000 7f1f1f200640 1 -- start start 2024-02-21T23:14:54.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.102+0000 7f1f1f200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f2006ee40 0x7f1f200c5b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:54.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.102+0000 7f1f1f200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1f20150450 0x7f1f20072930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:54.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.102+0000 7f1f1f200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1f20156af0 0x7f1f20071070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:54.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.102+0000 7f1f1f200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f200c2cb0 con 0x7f1f2006ee40 2024-02-21T23:14:54.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.102+0000 7f1f1f200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f200c29d0 con 0x7f1f20150450 2024-02-21T23:14:54.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.102+0000 7f1f1f200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f2006e700 con 0x7f1f20156af0 2024-02-21T23:14:54.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.102+0000 7f1f1de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f2006ee40 0x7f1f200c5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:54.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.102+0000 7f1f1de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f2006ee40 0x7f1f200c5b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57966/0 (socket says 172.21.15.28:57966) 2024-02-21T23:14:54.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.102+0000 7f1f1de00640 1 -- 172.21.15.28:0/747302020 learned_addr learned my addr 172.21.15.28:0/747302020 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:14:54.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.103+0000 7f1f1e800640 1 --2- 172.21.15.28:0/747302020 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1f20156af0 0x7f1f20071070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:54.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.103+0000 7f1f1d400640 1 --2- 172.21.15.28:0/747302020 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1f20150450 0x7f1f20072930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:54.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.103+0000 7f1f1de00640 1 -- 172.21.15.28:0/747302020 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1f20156af0 msgr2=0x7f1f20071070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:54.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.103+0000 7f1f1de00640 1 --2- 172.21.15.28:0/747302020 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1f20156af0 0x7f1f20071070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:54.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.103+0000 7f1f1de00640 1 -- 172.21.15.28:0/747302020 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1f20150450 msgr2=0x7f1f20072930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:54.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.103+0000 7f1f1de00640 1 --2- 172.21.15.28:0/747302020 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1f20150450 0x7f1f20072930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:54.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.103+0000 7f1f1de00640 1 -- 172.21.15.28:0/747302020 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f201eac10 con 0x7f1f2006ee40 2024-02-21T23:14:54.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.103+0000 7f1f1d400640 1 --2- 172.21.15.28:0/747302020 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1f20150450 0x7f1f20072930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:14:54.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.103+0000 7f1f1de00640 1 --2- 172.21.15.28:0/747302020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f2006ee40 0x7f1f200c5b30 secure :-1 s=READY pgs=2510 cs=0 l=1 rev1=1 crypto rx=0x7f1f1000d800 tx=0x7f1f1000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:54.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.103+0000 7f1f16000640 1 -- 172.21.15.28:0/747302020 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f1000b6c0 con 0x7f1f2006ee40 2024-02-21T23:14:54.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.103+0000 7f1f16000640 1 -- 172.21.15.28:0/747302020 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f10019070 con 0x7f1f2006ee40 2024-02-21T23:14:54.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.103+0000 7f1f16000640 1 -- 172.21.15.28:0/747302020 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f1000ba80 con 0x7f1f2006ee40 2024-02-21T23:14:54.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.104+0000 7f1f1f200640 1 -- 172.21.15.28:0/747302020 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f200c6130 con 0x7f1f2006ee40 2024-02-21T23:14:54.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.104+0000 7f1f1f200640 1 -- 172.21.15.28:0/747302020 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f201eb1b0 con 0x7f1f2006ee40 2024-02-21T23:14:54.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.105+0000 7f1f1f200640 1 -- 172.21.15.28:0/747302020 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f20152d50 con 0x7f1f2006ee40 2024-02-21T23:14:54.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.106+0000 7f1f16000640 1 -- 172.21.15.28:0/747302020 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1f1000bc20 con 0x7f1f2006ee40 2024-02-21T23:14:54.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.107+0000 7f1f16000640 1 --2- 172.21.15.28:0/747302020 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1f0006b4d0 0x7f1f0006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:54.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.107+0000 7f1f16000640 1 -- 172.21.15.28:0/747302020 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f1f1008aa70 con 0x7f1f2006ee40 2024-02-21T23:14:54.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.108+0000 7f1f1d400640 1 --2- 172.21.15.28:0/747302020 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1f0006b4d0 0x7f1f0006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:54.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.109+0000 7f1f1d400640 1 --2- 172.21.15.28:0/747302020 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1f0006b4d0 0x7f1f0006d990 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c036000 tx=0x7f1f0c03b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:54.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.117+0000 7f1f16000640 1 -- 172.21.15.28:0/747302020 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1f20152d50 con 0x7f1f2006ee40 2024-02-21T23:14:54.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.387+0000 7f1f1f200640 1 -- 172.21.15.28:0/747302020 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f2004f290 con 0x7f1f2006ee40 2024-02-21T23:14:54.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.388+0000 7f1f16000640 1 -- 172.21.15.28:0/747302020 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 246 v246) v1 ==== 95+0+3384 (secure 0 0 0) 0x7f1f10059530 con 0x7f1f2006ee40 2024-02-21T23:14:54.375 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:14:54.375 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":246,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":246,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:14:46.655515+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:14:54.376 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 246 2024-02-21T23:14:54.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.390+0000 7f1f1f200640 1 -- 172.21.15.28:0/747302020 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1f0006b4d0 msgr2=0x7f1f0006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:54.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.390+0000 7f1f1f200640 1 --2- 172.21.15.28:0/747302020 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1f0006b4d0 0x7f1f0006d990 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c036000 tx=0x7f1f0c03b040 comp rx=0 tx=0).stop 2024-02-21T23:14:54.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.390+0000 7f1f1f200640 1 -- 172.21.15.28:0/747302020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f2006ee40 msgr2=0x7f1f200c5b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:54.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.390+0000 7f1f1f200640 1 --2- 172.21.15.28:0/747302020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f2006ee40 0x7f1f200c5b30 secure :-1 s=READY pgs=2510 cs=0 l=1 rev1=1 crypto rx=0x7f1f1000d800 tx=0x7f1f1000dcc0 comp rx=0 tx=0).stop 2024-02-21T23:14:54.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.390+0000 7f1f1f200640 1 -- 172.21.15.28:0/747302020 shutdown_connections 2024-02-21T23:14:54.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.390+0000 7f1f1f200640 1 --2- 172.21.15.28:0/747302020 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1f0006b4d0 0x7f1f0006d990 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:54.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.390+0000 7f1f1f200640 1 --2- 172.21.15.28:0/747302020 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1f20156af0 0x7f1f20071070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:54.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.390+0000 7f1f1f200640 1 --2- 172.21.15.28:0/747302020 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1f20150450 0x7f1f20072930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:54.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.390+0000 7f1f1f200640 1 --2- 172.21.15.28:0/747302020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f2006ee40 0x7f1f200c5b30 unknown :-1 s=CLOSED pgs=2510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:54.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.390+0000 7f1f1f200640 1 -- 172.21.15.28:0/747302020 >> 172.21.15.28:0/747302020 conn(0x7f1f2006b630 msgr2=0x7f1f200c3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:14:54.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.391+0000 7f1f1f200640 1 -- 172.21.15.28:0/747302020 shutdown_connections 2024-02-21T23:14:54.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:54.391+0000 7f1f1f200640 1 -- 172.21.15.28:0/747302020 wait complete. 2024-02-21T23:14:54.801 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 246, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:14:46.655515+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:14:54.801 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:14:55.802 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:14:56.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.039+0000 7fe692a00640 1 Processor -- start 2024-02-21T23:14:56.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.040+0000 7fe692a00640 1 -- start start 2024-02-21T23:14:56.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.041+0000 7fe692a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe68c0aff90 0x7fe68c0c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:56.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.041+0000 7fe692a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe68c0c5720 0x7fe68c15f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:56.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.041+0000 7fe692a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe68c0c74c0 con 0x7fe68c06dbf0 2024-02-21T23:14:56.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.041+0000 7fe692a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe68c0c7270 con 0x7fe68c0c5720 2024-02-21T23:14:56.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.041+0000 7fe692a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe68c158630 con 0x7fe68c0aff90 2024-02-21T23:14:56.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.041+0000 7fe692000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe68c0c5720 0x7fe68c15f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:56.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.041+0000 7fe692000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe68c0c5720 0x7fe68c15f890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49524/0 (socket says 172.21.15.28:49524) 2024-02-21T23:14:56.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.041+0000 7fe692000640 1 -- 172.21.15.28:0/664906323 learned_addr learned my addr 172.21.15.28:0/664906323 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:14:56.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.042+0000 7fe691600640 1 --2- 172.21.15.28:0/664906323 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe68c0aff90 0x7fe68c0c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:56.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.042+0000 7fe692000640 1 -- 172.21.15.28:0/664906323 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe68c0aff90 msgr2=0x7fe68c0c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:14:56.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.042+0000 7fe692000640 1 --2- 172.21.15.28:0/664906323 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe68c0aff90 0x7fe68c0c51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:56.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.042+0000 7fe692000640 1 -- 172.21.15.28:0/664906323 >> v1:172.21.15.81:6789/0 conn(0x7fe68c06dbf0 legacy=0x7fe68c158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:14:56.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.042+0000 7fe692000640 1 -- 172.21.15.28:0/664906323 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe68c0c77a0 con 0x7fe68c0c5720 2024-02-21T23:14:56.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.044+0000 7fe692000640 1 --2- 172.21.15.28:0/664906323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe68c0c5720 0x7fe68c15f890 secure :-1 s=READY pgs=2511 cs=0 l=1 rev1=1 crypto rx=0x7fe684009870 tx=0x7fe68402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=179e998b7c6b1eb4 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:56.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.044+0000 7fe68be00640 1 -- 172.21.15.28:0/664906323 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe684034070 con 0x7fe68c0c5720 2024-02-21T23:14:56.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.044+0000 7fe68be00640 1 -- 172.21.15.28:0/664906323 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe684030840 con 0x7fe68c0c5720 2024-02-21T23:14:56.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.044+0000 7fe68be00640 1 -- 172.21.15.28:0/664906323 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe684030c10 con 0x7fe68c0c5720 2024-02-21T23:14:56.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.045+0000 7fe692a00640 1 -- 172.21.15.28:0/664906323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe68c0c5720 msgr2=0x7fe68c15f890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:56.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.045+0000 7fe692a00640 1 --2- 172.21.15.28:0/664906323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe68c0c5720 0x7fe68c15f890 secure :-1 s=READY pgs=2511 cs=0 l=1 rev1=1 crypto rx=0x7fe684009870 tx=0x7fe68402b980 comp rx=0 tx=0).stop 2024-02-21T23:14:56.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.045+0000 7fe692a00640 1 -- 172.21.15.28:0/664906323 shutdown_connections 2024-02-21T23:14:56.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.045+0000 7fe692a00640 1 --2- 172.21.15.28:0/664906323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe68c0c5720 0x7fe68c15f890 unknown :-1 s=CLOSED pgs=2511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:56.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.045+0000 7fe692a00640 1 --2- 172.21.15.28:0/664906323 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe68c0aff90 0x7fe68c0c51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:56.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.045+0000 7fe692a00640 1 -- 172.21.15.28:0/664906323 >> 172.21.15.28:0/664906323 conn(0x7fe68c0ac370 msgr2=0x7fe68c0ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:14:56.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.045+0000 7fe692a00640 1 -- 172.21.15.28:0/664906323 shutdown_connections 2024-02-21T23:14:56.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.045+0000 7fe692a00640 1 -- 172.21.15.28:0/664906323 wait complete. 2024-02-21T23:14:56.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.047+0000 7fe692a00640 1 Processor -- start 2024-02-21T23:14:56.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.047+0000 7fe692a00640 1 -- start start 2024-02-21T23:14:56.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.048+0000 7fe692a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe68c06dbf0 0x7fe68c0c1080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:56.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.048+0000 7fe692a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe68c0aff90 0x7fe68c154b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:56.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.048+0000 7fe692a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe68c0c5720 0x7fe68c0c0060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:56.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.048+0000 7fe692a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe68c0c8760 con 0x7fe68c0aff90 2024-02-21T23:14:56.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.048+0000 7fe692a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe68c0be0c0 con 0x7fe68c06dbf0 2024-02-21T23:14:56.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.048+0000 7fe692a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe68c0c88e0 con 0x7fe68c0c5720 2024-02-21T23:14:56.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.048+0000 7fe690c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe68c0aff90 0x7fe68c154b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:56.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.048+0000 7fe690c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe68c0aff90 0x7fe68c154b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49534/0 (socket says 172.21.15.28:49534) 2024-02-21T23:14:56.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.048+0000 7fe690c00640 1 -- 172.21.15.28:0/1951459598 learned_addr learned my addr 172.21.15.28:0/1951459598 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:14:56.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.048+0000 7fe691600640 1 --2- 172.21.15.28:0/1951459598 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe68c06dbf0 0x7fe68c0c1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:56.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.048+0000 7fe692000640 1 --2- 172.21.15.28:0/1951459598 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe68c0c5720 0x7fe68c0c0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:56.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.048+0000 7fe690c00640 1 -- 172.21.15.28:0/1951459598 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe68c0c5720 msgr2=0x7fe68c0c0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:56.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.048+0000 7fe690c00640 1 --2- 172.21.15.28:0/1951459598 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe68c0c5720 0x7fe68c0c0060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:56.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.048+0000 7fe690c00640 1 -- 172.21.15.28:0/1951459598 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe68c06dbf0 msgr2=0x7fe68c0c1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:56.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.048+0000 7fe690c00640 1 --2- 172.21.15.28:0/1951459598 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe68c06dbf0 0x7fe68c0c1080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:56.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.048+0000 7fe690c00640 1 -- 172.21.15.28:0/1951459598 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe68c1eb1d0 con 0x7fe68c0aff90 2024-02-21T23:14:56.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.048+0000 7fe691600640 1 --2- 172.21.15.28:0/1951459598 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe68c06dbf0 0x7fe68c0c1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:14:56.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.049+0000 7fe690c00640 1 --2- 172.21.15.28:0/1951459598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe68c0aff90 0x7fe68c154b70 secure :-1 s=READY pgs=2512 cs=0 l=1 rev1=1 crypto rx=0x7fe67800c400 tx=0x7fe67800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:56.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.049+0000 7fe689600640 1 -- 172.21.15.28:0/1951459598 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6780052d0 con 0x7fe68c0aff90 2024-02-21T23:14:56.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.049+0000 7fe689600640 1 -- 172.21.15.28:0/1951459598 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe678005490 con 0x7fe68c0aff90 2024-02-21T23:14:56.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.049+0000 7fe689600640 1 -- 172.21.15.28:0/1951459598 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe678007260 con 0x7fe68c0aff90 2024-02-21T23:14:56.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.049+0000 7fe692a00640 1 -- 172.21.15.28:0/1951459598 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe68c1eb4b0 con 0x7fe68c0aff90 2024-02-21T23:14:56.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.049+0000 7fe692a00640 1 -- 172.21.15.28:0/1951459598 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe68c1eba70 con 0x7fe68c0aff90 2024-02-21T23:14:56.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.050+0000 7fe692a00640 1 -- 172.21.15.28:0/1951459598 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe68c04f290 con 0x7fe68c0aff90 2024-02-21T23:14:56.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.051+0000 7fe689600640 1 -- 172.21.15.28:0/1951459598 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe678007400 con 0x7fe68c0aff90 2024-02-21T23:14:56.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.053+0000 7fe689600640 1 --2- 172.21.15.28:0/1951459598 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe66c06b4d0 0x7fe66c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:56.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.053+0000 7fe689600640 1 -- 172.21.15.28:0/1951459598 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fe678089910 con 0x7fe68c0aff90 2024-02-21T23:14:56.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.054+0000 7fe691600640 1 --2- 172.21.15.28:0/1951459598 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe66c06b4d0 0x7fe66c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:56.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.055+0000 7fe691600640 1 --2- 172.21.15.28:0/1951459598 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe66c06b4d0 0x7fe66c06d990 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7fe680007330 tx=0x7fe68000b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:56.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.058+0000 7fe689600640 1 -- 172.21.15.28:0/1951459598 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe678058590 con 0x7fe68c0aff90 2024-02-21T23:14:56.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.321+0000 7fe692a00640 1 -- 172.21.15.28:0/1951459598 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe68c0c4e70 con 0x7fe68c0aff90 2024-02-21T23:14:56.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.323+0000 7fe689600640 1 -- 172.21.15.28:0/1951459598 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 246 v246) v1 ==== 95+0+3384 (secure 0 0 0) 0x7fe678057e90 con 0x7fe68c0aff90 2024-02-21T23:14:56.310 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:14:56.310 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":246,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":246,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:14:46.655515+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:14:56.310 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 246 2024-02-21T23:14:56.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.326+0000 7fe692a00640 1 -- 172.21.15.28:0/1951459598 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe66c06b4d0 msgr2=0x7fe66c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:56.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.326+0000 7fe692a00640 1 --2- 172.21.15.28:0/1951459598 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe66c06b4d0 0x7fe66c06d990 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7fe680007330 tx=0x7fe68000b040 comp rx=0 tx=0).stop 2024-02-21T23:14:56.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.326+0000 7fe692a00640 1 -- 172.21.15.28:0/1951459598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe68c0aff90 msgr2=0x7fe68c154b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:56.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.326+0000 7fe692a00640 1 --2- 172.21.15.28:0/1951459598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe68c0aff90 0x7fe68c154b70 secure :-1 s=READY pgs=2512 cs=0 l=1 rev1=1 crypto rx=0x7fe67800c400 tx=0x7fe67800c8c0 comp rx=0 tx=0).stop 2024-02-21T23:14:56.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.326+0000 7fe692a00640 1 -- 172.21.15.28:0/1951459598 shutdown_connections 2024-02-21T23:14:56.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.326+0000 7fe692a00640 1 --2- 172.21.15.28:0/1951459598 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe66c06b4d0 0x7fe66c06d990 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:56.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.326+0000 7fe692a00640 1 --2- 172.21.15.28:0/1951459598 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe68c0c5720 0x7fe68c0c0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:56.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.326+0000 7fe692a00640 1 --2- 172.21.15.28:0/1951459598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe68c0aff90 0x7fe68c154b70 unknown :-1 s=CLOSED pgs=2512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:56.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.326+0000 7fe692a00640 1 --2- 172.21.15.28:0/1951459598 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe68c06dbf0 0x7fe68c0c1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:56.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.327+0000 7fe692a00640 1 -- 172.21.15.28:0/1951459598 >> 172.21.15.28:0/1951459598 conn(0x7fe68c0ac370 msgr2=0x7fe68c0ac750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:14:56.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.327+0000 7fe692a00640 1 -- 172.21.15.28:0/1951459598 shutdown_connections 2024-02-21T23:14:56.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:56.327+0000 7fe692a00640 1 -- 172.21.15.28:0/1951459598 wait complete. 2024-02-21T23:14:56.737 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 246, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:14:46.655515+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:14:56.737 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:14:57.737 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:14:57.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.982+0000 7f7c0a600640 1 Processor -- start 2024-02-21T23:14:57.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.982+0000 7f7c0a600640 1 -- start start 2024-02-21T23:14:57.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.984+0000 7f7c0a600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7c0414ef60 0x7f7c04158ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:57.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.984+0000 7f7c0a600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c04159410 con 0x7f7c040c8a10 2024-02-21T23:14:57.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.984+0000 7f7c0a600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c04155a00 con 0x7f7c040c6740 2024-02-21T23:14:57.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.984+0000 7f7c0a600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c04155ce0 con 0x7f7c0414ef60 2024-02-21T23:14:57.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.984+0000 7f7c09c00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f7c040c8a10 0x7f7c0415f850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34562/0 (socket says 172.21.15.28:34562) 2024-02-21T23:14:57.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.984+0000 7f7c09c00640 1 -- 172.21.15.28:0/2520435734 learned_addr learned my addr 172.21.15.28:0/2520435734 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:14:57.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.985+0000 7f7c09200640 1 --2- 172.21.15.28:0/2520435734 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7c0414ef60 0x7f7c04158ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:57.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.985+0000 7f7c03400640 1 -- 172.21.15.28:0/2520435734 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 588216356 0 0) 0x7f7c04159410 con 0x7f7c040c8a10 2024-02-21T23:14:57.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.985+0000 7f7c03400640 1 -- 172.21.15.28:0/2520435734 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bec003520 con 0x7f7c040c8a10 2024-02-21T23:14:57.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.985+0000 7f7c03400640 1 -- 172.21.15.28:0/2520435734 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1502106904 0 0) 0x7f7bec003520 con 0x7f7c040c8a10 2024-02-21T23:14:57.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.985+0000 7f7c03400640 1 -- 172.21.15.28:0/2520435734 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7c0414ef60 msgr2=0x7f7c04158ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:14:57.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.985+0000 7f7c03400640 1 --2- 172.21.15.28:0/2520435734 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7c0414ef60 0x7f7c04158ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:57.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.985+0000 7f7c03400640 1 -- 172.21.15.28:0/2520435734 >> v1:172.21.15.81:6789/0 conn(0x7f7c040c6740 legacy=0x7f7c04154fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:57.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.985+0000 7f7c03400640 1 -- 172.21.15.28:0/2520435734 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c041586d0 con 0x7f7c040c8a10 2024-02-21T23:14:57.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.985+0000 7f7c03400640 1 -- 172.21.15.28:0/2520435734 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f7bfc0019f0 con 0x7f7c040c8a10 2024-02-21T23:14:57.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.985+0000 7f7c03400640 1 -- 172.21.15.28:0/2520435734 >> v1:172.21.15.28:6789/0 conn(0x7f7c040c8a10 legacy=0x7f7c0415f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:57.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.985+0000 7f7c03400640 1 --2- 172.21.15.28:0/2520435734 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7c0414ef60 0x7f7bec0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:57.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.986+0000 7f7c03400640 1 --2- 172.21.15.28:0/2520435734 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7bec003f60 0x7f7bec004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:57.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.986+0000 7f7c03400640 1 --2- 172.21.15.28:0/2520435734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bec004d40 0x7f7bec0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:57.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.986+0000 7f7c03400640 1 -- 172.21.15.28:0/2520435734 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c04155ce0 con 0x7f7bec004d40 2024-02-21T23:14:57.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.986+0000 7f7c03400640 1 -- 172.21.15.28:0/2520435734 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bec00baa0 con 0x7f7bec003f60 2024-02-21T23:14:57.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.986+0000 7f7c03400640 1 -- 172.21.15.28:0/2520435734 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bec00bca0 con 0x7f7c0414ef60 2024-02-21T23:14:57.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.986+0000 7f7c09c00640 1 --2- 172.21.15.28:0/2520435734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bec004d40 0x7f7bec0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:57.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.986+0000 7f7c09200640 1 --2- 172.21.15.28:0/2520435734 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7c0414ef60 0x7f7bec0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:57.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.986+0000 7f7c03e00640 1 --2- 172.21.15.28:0/2520435734 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7bec003f60 0x7f7bec004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:57.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.986+0000 7f7c09c00640 1 -- 172.21.15.28:0/2520435734 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7c0414ef60 msgr2=0x7f7bec0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:14:57.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.986+0000 7f7c09c00640 1 --2- 172.21.15.28:0/2520435734 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7c0414ef60 0x7f7bec0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:57.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.986+0000 7f7c09c00640 1 -- 172.21.15.28:0/2520435734 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7bec003f60 msgr2=0x7f7bec004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:14:57.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.986+0000 7f7c09c00640 1 --2- 172.21.15.28:0/2520435734 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7bec003f60 0x7f7bec004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:57.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.986+0000 7f7c09c00640 1 -- 172.21.15.28:0/2520435734 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bec003520 con 0x7f7bec004d40 2024-02-21T23:14:57.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.988+0000 7f7c09c00640 1 --2- 172.21.15.28:0/2520435734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bec004d40 0x7f7bec0051c0 secure :-1 s=READY pgs=2514 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc00a840 tx=0x7f7bec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d1ed15a700b9c3f server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:57.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.988+0000 7f7c03400640 1 -- 172.21.15.28:0/2520435734 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bfc00a460 con 0x7f7bec004d40 2024-02-21T23:14:57.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.988+0000 7f7c03400640 1 -- 172.21.15.28:0/2520435734 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bfc034650 con 0x7f7bec004d40 2024-02-21T23:14:57.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.988+0000 7f7c03400640 1 -- 172.21.15.28:0/2520435734 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bfc034a80 con 0x7f7bec004d40 2024-02-21T23:14:57.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.988+0000 7f7c0a600640 1 -- 172.21.15.28:0/2520435734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bec004d40 msgr2=0x7f7bec0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:57.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.988+0000 7f7c0a600640 1 --2- 172.21.15.28:0/2520435734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bec004d40 0x7f7bec0051c0 secure :-1 s=READY pgs=2514 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc00a840 tx=0x7f7bec005830 comp rx=0 tx=0).stop 2024-02-21T23:14:57.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.988+0000 7f7c09c00640 1 -- 172.21.15.28:0/2520435734 reap_dead start 2024-02-21T23:14:57.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.989+0000 7f7c0a600640 1 -- 172.21.15.28:0/2520435734 shutdown_connections 2024-02-21T23:14:57.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.989+0000 7f7c0a600640 1 -- 172.21.15.28:0/2520435734 >> 172.21.15.28:0/2520435734 conn(0x7f7c0406b260 msgr2=0x7f7c0406b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:14:57.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.989+0000 7f7c0a600640 1 -- 172.21.15.28:0/2520435734 shutdown_connections 2024-02-21T23:14:57.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.989+0000 7f7c0a600640 1 -- 172.21.15.28:0/2520435734 wait complete. 2024-02-21T23:14:57.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.990+0000 7f7c0a600640 1 Processor -- start 2024-02-21T23:14:57.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.991+0000 7f7c0a600640 1 -- start start 2024-02-21T23:14:57.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.992+0000 7f7c0a600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bec004d40 0x7f7c04070fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:57.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.992+0000 7f7c0a600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7c041ee810 0x7f7c0414d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:57.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.992+0000 7f7c0a600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7c041567d0 0x7f7c04156c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:57.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.992+0000 7f7c0a600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c041556c0 con 0x7f7bec004d40 2024-02-21T23:14:57.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.992+0000 7f7c0a600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0414f340 con 0x7f7c041ee810 2024-02-21T23:14:57.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.992+0000 7f7c0a600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c040c2a00 con 0x7f7c041567d0 2024-02-21T23:14:57.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.992+0000 7f7c09200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bec004d40 0x7f7c04070fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:57.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.992+0000 7f7c09200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bec004d40 0x7f7c04070fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49560/0 (socket says 172.21.15.28:49560) 2024-02-21T23:14:57.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.992+0000 7f7c09200640 1 -- 172.21.15.28:0/2333422303 learned_addr learned my addr 172.21.15.28:0/2333422303 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:14:57.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.992+0000 7f7c03e00640 1 --2- 172.21.15.28:0/2333422303 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7c041ee810 0x7f7c0414d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:57.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.992+0000 7f7c09c00640 1 --2- 172.21.15.28:0/2333422303 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7c041567d0 0x7f7c04156c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:57.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.992+0000 7f7c09200640 1 -- 172.21.15.28:0/2333422303 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7c041567d0 msgr2=0x7f7c04156c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:57.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.992+0000 7f7c09200640 1 --2- 172.21.15.28:0/2333422303 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7c041567d0 0x7f7c04156c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:57.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.992+0000 7f7c09200640 1 -- 172.21.15.28:0/2333422303 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7c041ee810 msgr2=0x7f7c0414d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:57.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.992+0000 7f7c09200640 1 --2- 172.21.15.28:0/2333422303 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7c041ee810 0x7f7c0414d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:57.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.993+0000 7f7c09200640 1 -- 172.21.15.28:0/2333422303 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c041eed10 con 0x7f7bec004d40 2024-02-21T23:14:57.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.993+0000 7f7c09200640 1 --2- 172.21.15.28:0/2333422303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bec004d40 0x7f7c04070fd0 secure :-1 s=READY pgs=2515 cs=0 l=1 rev1=1 crypto rx=0x7f7bf80119d0 tx=0x7f7bf8011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:57.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.993+0000 7f7c00c00640 1 -- 172.21.15.28:0/2333422303 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bf800fb50 con 0x7f7bec004d40 2024-02-21T23:14:57.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.993+0000 7f7c00c00640 1 -- 172.21.15.28:0/2333422303 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bf800fcf0 con 0x7f7bec004d40 2024-02-21T23:14:57.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.993+0000 7f7c00c00640 1 -- 172.21.15.28:0/2333422303 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bf800fea0 con 0x7f7bec004d40 2024-02-21T23:14:57.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.993+0000 7f7c0a600640 1 -- 172.21.15.28:0/2333422303 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c0406f1d0 con 0x7f7bec004d40 2024-02-21T23:14:57.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.994+0000 7f7c0a600640 1 -- 172.21.15.28:0/2333422303 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c0406f790 con 0x7f7bec004d40 2024-02-21T23:14:57.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.995+0000 7f7c0a600640 1 -- 172.21.15.28:0/2333422303 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c0404f290 con 0x7f7bec004d40 2024-02-21T23:14:57.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.996+0000 7f7c00c00640 1 -- 172.21.15.28:0/2333422303 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7bf8012040 con 0x7f7bec004d40 2024-02-21T23:14:57.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.997+0000 7f7c00c00640 1 --2- 172.21.15.28:0/2333422303 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7be406b500 0x7f7be406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:57.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.997+0000 7f7c00c00640 1 -- 172.21.15.28:0/2333422303 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f7bf808ab70 con 0x7f7bec004d40 2024-02-21T23:14:57.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.998+0000 7f7c03e00640 1 --2- 172.21.15.28:0/2333422303 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7be406b500 0x7f7be406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:57.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:57.999+0000 7f7c03e00640 1 --2- 172.21.15.28:0/2333422303 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7be406b500 0x7f7be406d9c0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f7c0414e0b0 tx=0x7f7bf0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:57.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:58.001+0000 7f7c00c00640 1 -- 172.21.15.28:0/2333422303 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7bf8019030 con 0x7f7bec004d40 2024-02-21T23:14:58.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:58.240+0000 7f7c0a600640 1 -- 172.21.15.28:0/2333422303 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7c040a80e0 con 0x7f7bec004d40 2024-02-21T23:14:58.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:58.241+0000 7f7c00c00640 1 -- 172.21.15.28:0/2333422303 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 246 v246) v1 ==== 95+0+3384 (secure 0 0 0) 0x7f7bf80596c0 con 0x7f7bec004d40 2024-02-21T23:14:58.229 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:14:58.229 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":246,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":246,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:14:46.655515+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:14:58.229 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 246 2024-02-21T23:14:58.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:58.245+0000 7f7c0a600640 1 -- 172.21.15.28:0/2333422303 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7be406b500 msgr2=0x7f7be406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:58.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:58.245+0000 7f7c0a600640 1 --2- 172.21.15.28:0/2333422303 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7be406b500 0x7f7be406d9c0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f7c0414e0b0 tx=0x7f7bf0009040 comp rx=0 tx=0).stop 2024-02-21T23:14:58.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:58.245+0000 7f7c0a600640 1 -- 172.21.15.28:0/2333422303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bec004d40 msgr2=0x7f7c04070fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:58.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:58.245+0000 7f7c0a600640 1 --2- 172.21.15.28:0/2333422303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bec004d40 0x7f7c04070fd0 secure :-1 s=READY pgs=2515 cs=0 l=1 rev1=1 crypto rx=0x7f7bf80119d0 tx=0x7f7bf8011e90 comp rx=0 tx=0).stop 2024-02-21T23:14:58.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:58.245+0000 7f7c0a600640 1 -- 172.21.15.28:0/2333422303 shutdown_connections 2024-02-21T23:14:58.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:58.245+0000 7f7c0a600640 1 --2- 172.21.15.28:0/2333422303 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7be406b500 0x7f7be406d9c0 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:58.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:58.245+0000 7f7c0a600640 1 --2- 172.21.15.28:0/2333422303 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7c041567d0 0x7f7c04156c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:58.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:58.245+0000 7f7c0a600640 1 --2- 172.21.15.28:0/2333422303 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7c041ee810 0x7f7c0414d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:58.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:58.245+0000 7f7c0a600640 1 --2- 172.21.15.28:0/2333422303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bec004d40 0x7f7c04070fd0 unknown :-1 s=CLOSED pgs=2515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:58.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:58.245+0000 7f7c0a600640 1 -- 172.21.15.28:0/2333422303 >> 172.21.15.28:0/2333422303 conn(0x7f7c0406b260 msgr2=0x7f7c040c2f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:14:58.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:58.245+0000 7f7c0a600640 1 -- 172.21.15.28:0/2333422303 shutdown_connections 2024-02-21T23:14:58.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:58.245+0000 7f7c0a600640 1 -- 172.21.15.28:0/2333422303 wait complete. 2024-02-21T23:14:58.659 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 246, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:14:46.655515+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:14:58.659 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:14:59.660 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:14:59.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:59.909+0000 7f3707000640 1 Processor -- start 2024-02-21T23:14:59.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:59.910+0000 7f3707000640 1 -- start start 2024-02-21T23:14:59.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:59.912+0000 7f3707000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f37000c5510 0x7f37000a7e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:59.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:59.912+0000 7f3707000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f37000c7320 0x7f37001521d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:59.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:59.912+0000 7f3707000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3700152710 0x7f3700155040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:59.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:59.912+0000 7f3707000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37000a8380 con 0x7f3700152710 2024-02-21T23:14:59.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:59.912+0000 7f3707000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37000c8040 con 0x7f37000c5510 2024-02-21T23:14:59.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:59.912+0000 7f3707000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37000c85c0 con 0x7f37000c7320 2024-02-21T23:14:59.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:59.912+0000 7f3705000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3700152710 0x7f3700155040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:59.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:59.913+0000 7f3705000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3700152710 0x7f3700155040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49570/0 (socket says 172.21.15.28:49570) 2024-02-21T23:14:59.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:59.913+0000 7f3705000640 1 -- 172.21.15.28:0/3953813258 learned_addr learned my addr 172.21.15.28:0/3953813258 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:14:59.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:59.913+0000 7f36f7400640 1 --2- 172.21.15.28:0/3953813258 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f37000c7320 0x7f37001521d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:59.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:59.913+0000 7f36ffe00640 1 --2- 172.21.15.28:0/3953813258 >> v2:172.21.15.81:3300/0 conn(0x7f37000c5510 0x7f37000a7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:59.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:59.913+0000 7f3705000640 1 -- 172.21.15.28:0/3953813258 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f37000c7320 msgr2=0x7f37001521d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:14:59.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:59.913+0000 7f3705000640 1 --2- 172.21.15.28:0/3953813258 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f37000c7320 0x7f37001521d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:59.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:59.913+0000 7f3705000640 1 -- 172.21.15.28:0/3953813258 >> v2:172.21.15.81:3300/0 conn(0x7f37000c5510 msgr2=0x7f37000a7e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:14:59.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:59.913+0000 7f3705000640 1 --2- 172.21.15.28:0/3953813258 >> v2:172.21.15.81:3300/0 conn(0x7f37000c5510 0x7f37000a7e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:59.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:59.913+0000 7f3705000640 1 -- 172.21.15.28:0/3953813258 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37000c82e0 con 0x7f3700152710 2024-02-21T23:14:59.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:59.914+0000 7f3705000640 1 --2- 172.21.15.28:0/3953813258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3700152710 0x7f3700155040 secure :-1 s=READY pgs=2516 cs=0 l=1 rev1=1 crypto rx=0x7f37000c9890 tx=0x7f36f802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7938ac748aa33bc3 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:59.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:59.915+0000 7f36ff400640 1 -- 172.21.15.28:0/3953813258 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36f8035020 con 0x7f3700152710 2024-02-21T23:14:59.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:59.915+0000 7f36ff400640 1 -- 172.21.15.28:0/3953813258 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36f8031870 con 0x7f3700152710 2024-02-21T23:14:59.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:59.915+0000 7f36ff400640 1 -- 172.21.15.28:0/3953813258 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36f8031c40 con 0x7f3700152710 2024-02-21T23:14:59.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:59.915+0000 7f3707000640 1 -- 172.21.15.28:0/3953813258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3700152710 msgr2=0x7f3700155040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:59.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:59.915+0000 7f3707000640 1 --2- 172.21.15.28:0/3953813258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3700152710 0x7f3700155040 secure :-1 s=READY pgs=2516 cs=0 l=1 rev1=1 crypto rx=0x7f37000c9890 tx=0x7f36f802b910 comp rx=0 tx=0).stop 2024-02-21T23:14:59.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:59.915+0000 7f3707000640 1 -- 172.21.15.28:0/3953813258 shutdown_connections 2024-02-21T23:14:59.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:59.915+0000 7f3707000640 1 --2- 172.21.15.28:0/3953813258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3700152710 0x7f3700155040 unknown :-1 s=CLOSED pgs=2516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:59.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:59.915+0000 7f3707000640 1 --2- 172.21.15.28:0/3953813258 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f37000c7320 0x7f37001521d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:59.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:59.915+0000 7f3707000640 1 --2- 172.21.15.28:0/3953813258 >> v2:172.21.15.81:3300/0 conn(0x7f37000c5510 0x7f37000a7e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:59.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:59.915+0000 7f3707000640 1 -- 172.21.15.28:0/3953813258 >> 172.21.15.28:0/3953813258 conn(0x7f37000c8810 msgr2=0x7f37000c8bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:14:59.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:59.916+0000 7f3707000640 1 -- 172.21.15.28:0/3953813258 shutdown_connections 2024-02-21T23:14:59.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:59.916+0000 7f3707000640 1 -- 172.21.15.28:0/3953813258 wait complete. 2024-02-21T23:14:59.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:59.918+0000 7f3707000640 1 Processor -- start 2024-02-21T23:14:59.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:59.919+0000 7f3707000640 1 -- start start 2024-02-21T23:14:59.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:59.920+0000 7f3707000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f37000c5510 0x7f3700158600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:59.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:59.920+0000 7f3707000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f37000c7320 0x7f3700158b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:59.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:59.920+0000 7f3707000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3700152710 0x7f37001ee4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:59.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:59.920+0000 7f3707000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37000c58f0 con 0x7f3700152710 2024-02-21T23:14:59.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:59.920+0000 7f3707000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37000c6cb0 con 0x7f37000c7320 2024-02-21T23:14:59.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:59.920+0000 7f3707000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37000c5a70 con 0x7f37000c5510 2024-02-21T23:14:59.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:59.920+0000 7f3705000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3700152710 0x7f37001ee4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:59.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:59.920+0000 7f3705000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3700152710 0x7f37001ee4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49578/0 (socket says 172.21.15.28:49578) 2024-02-21T23:14:59.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:59.920+0000 7f3705000640 1 -- 172.21.15.28:0/2954946526 learned_addr learned my addr 172.21.15.28:0/2954946526 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:14:59.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:59.921+0000 7f3705000640 1 -- 172.21.15.28:0/2954946526 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f37000c5510 msgr2=0x7f3700158600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:59.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:59.921+0000 7f3705000640 1 --2- 172.21.15.28:0/2954946526 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f37000c5510 0x7f3700158600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:59.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:59.921+0000 7f3705000640 1 -- 172.21.15.28:0/2954946526 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f37000c7320 msgr2=0x7f3700158b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:14:59.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:59.921+0000 7f3705000640 1 --2- 172.21.15.28:0/2954946526 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f37000c7320 0x7f3700158b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:14:59.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:59.921+0000 7f3705000640 1 -- 172.21.15.28:0/2954946526 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37001e1520 con 0x7f3700152710 2024-02-21T23:14:59.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:59.921+0000 7f3705000640 1 --2- 172.21.15.28:0/2954946526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3700152710 0x7f37001ee4d0 secure :-1 s=READY pgs=2517 cs=0 l=1 rev1=1 crypto rx=0x7f36f802b950 tx=0x7f36f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:59.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:59.921+0000 7f36fcc00640 1 -- 172.21.15.28:0/2954946526 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36f8040070 con 0x7f3700152710 2024-02-21T23:14:59.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:59.922+0000 7f36fcc00640 1 -- 172.21.15.28:0/2954946526 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36f8030c50 con 0x7f3700152710 2024-02-21T23:14:59.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:59.922+0000 7f36fcc00640 1 -- 172.21.15.28:0/2954946526 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36f8035040 con 0x7f3700152710 2024-02-21T23:14:59.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:59.922+0000 7f3707000640 1 -- 172.21.15.28:0/2954946526 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37001e1240 con 0x7f3700152710 2024-02-21T23:14:59.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:59.922+0000 7f3707000640 1 -- 172.21.15.28:0/2954946526 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37001df3e0 con 0x7f3700152710 2024-02-21T23:14:59.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:59.923+0000 7f3707000640 1 -- 172.21.15.28:0/2954946526 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3700068640 con 0x7f3700152710 2024-02-21T23:14:59.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:59.924+0000 7f36fcc00640 1 -- 172.21.15.28:0/2954946526 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f36f8038020 con 0x7f3700152710 2024-02-21T23:14:59.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:59.925+0000 7f36fcc00640 1 --2- 172.21.15.28:0/2954946526 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f36dc06b4d0 0x7f36dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:14:59.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:59.926+0000 7f36fcc00640 1 -- 172.21.15.28:0/2954946526 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f36f80ac340 con 0x7f3700152710 2024-02-21T23:14:59.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:59.926+0000 7f36ffe00640 1 --2- 172.21.15.28:0/2954946526 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f36dc06b4d0 0x7f36dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:14:59.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:59.927+0000 7f36ffe00640 1 --2- 172.21.15.28:0/2954946526 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f36dc06b4d0 0x7f36dc06d990 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f36f0006fd0 tx=0x7f36f0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:14:59.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:14:59.929+0000 7f36fcc00640 1 -- 172.21.15.28:0/2954946526 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f36f807af30 con 0x7f3700152710 2024-02-21T23:15:00.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:00.169+0000 7f3707000640 1 -- 172.21.15.28:0/2954946526 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f37000c6710 con 0x7f3700152710 2024-02-21T23:15:00.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:00.170+0000 7f36fcc00640 1 -- 172.21.15.28:0/2954946526 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 246 v246) v1 ==== 95+0+3384 (secure 0 0 0) 0x7f36f807a830 con 0x7f3700152710 2024-02-21T23:15:00.157 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:15:00.157 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":246,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":246,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:14:46.655515+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:15:00.158 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 246 2024-02-21T23:15:00.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:00.172+0000 7f3707000640 1 -- 172.21.15.28:0/2954946526 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f36dc06b4d0 msgr2=0x7f36dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:00.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:00.172+0000 7f3707000640 1 --2- 172.21.15.28:0/2954946526 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f36dc06b4d0 0x7f36dc06d990 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f36f0006fd0 tx=0x7f36f0008040 comp rx=0 tx=0).stop 2024-02-21T23:15:00.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:00.172+0000 7f3707000640 1 -- 172.21.15.28:0/2954946526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3700152710 msgr2=0x7f37001ee4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:00.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:00.172+0000 7f3707000640 1 --2- 172.21.15.28:0/2954946526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3700152710 0x7f37001ee4d0 secure :-1 s=READY pgs=2517 cs=0 l=1 rev1=1 crypto rx=0x7f36f802b950 tx=0x7f36f80046c0 comp rx=0 tx=0).stop 2024-02-21T23:15:00.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:00.173+0000 7f3707000640 1 -- 172.21.15.28:0/2954946526 shutdown_connections 2024-02-21T23:15:00.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:00.173+0000 7f3707000640 1 --2- 172.21.15.28:0/2954946526 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f36dc06b4d0 0x7f36dc06d990 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:00.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:00.173+0000 7f3707000640 1 --2- 172.21.15.28:0/2954946526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3700152710 0x7f37001ee4d0 unknown :-1 s=CLOSED pgs=2517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:00.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:00.173+0000 7f3707000640 1 --2- 172.21.15.28:0/2954946526 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f37000c7320 0x7f3700158b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:00.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:00.173+0000 7f3707000640 1 --2- 172.21.15.28:0/2954946526 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f37000c5510 0x7f3700158600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:00.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:00.173+0000 7f3707000640 1 -- 172.21.15.28:0/2954946526 >> 172.21.15.28:0/2954946526 conn(0x7f37000c8810 msgr2=0x7f370006c770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:15:00.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:00.173+0000 7f3707000640 1 -- 172.21.15.28:0/2954946526 shutdown_connections 2024-02-21T23:15:00.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:00.173+0000 7f3707000640 1 -- 172.21.15.28:0/2954946526 wait complete. 2024-02-21T23:15:00.583 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 246, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:14:46.655515+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:15:00.583 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:15:01.584 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:15:01.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:01.826+0000 7f9bd2000640 1 Processor -- start 2024-02-21T23:15:01.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:01.826+0000 7f9bd2000640 1 -- start start 2024-02-21T23:15:01.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:01.828+0000 7f9bd2000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9bcc156d90 0x7f9bcc06be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:01.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:01.828+0000 7f9bd2000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bcc0aca70 con 0x7f9bcc156360 2024-02-21T23:15:01.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:01.828+0000 7f9bd2000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bcc0acd50 con 0x7f9bcc0ac020 2024-02-21T23:15:01.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:01.828+0000 7f9bd2000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bcc0ac820 con 0x7f9bcc156d90 2024-02-21T23:15:01.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:01.828+0000 7f9bcbe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f9bcc156360 0x7f9bcc1534a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34572/0 (socket says 172.21.15.28:34572) 2024-02-21T23:15:01.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:01.828+0000 7f9bcbe00640 1 -- 172.21.15.28:0/4201641072 learned_addr learned my addr 172.21.15.28:0/4201641072 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:15:01.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:01.828+0000 7f9bd0c00640 1 --2- 172.21.15.28:0/4201641072 >> v2:172.21.15.28:3300/0 conn(0x7f9bcc156d90 0x7f9bcc06be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:01.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:01.829+0000 7f9bcb400640 1 -- 172.21.15.28:0/4201641072 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4269878632 0 0) 0x7f9bcc0aca70 con 0x7f9bcc156360 2024-02-21T23:15:01.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:01.829+0000 7f9bcb400640 1 -- 172.21.15.28:0/4201641072 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bb4003520 con 0x7f9bcc156360 2024-02-21T23:15:01.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:01.829+0000 7f9bd0c00640 1 -- 172.21.15.28:0/4201641072 >> v1:172.21.15.81:6789/0 conn(0x7f9bcc0ac020 legacy=0x7f9bcc15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:01.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:01.829+0000 7f9bcb400640 1 -- 172.21.15.28:0/4201641072 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3820376827 0 0) 0x7f9bb4003520 con 0x7f9bcc156360 2024-02-21T23:15:01.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:01.829+0000 7f9bd0c00640 1 -- 172.21.15.28:0/4201641072 >> v1:172.21.15.28:6789/0 conn(0x7f9bcc156360 legacy=0x7f9bcc1534a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:01.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:01.829+0000 7f9bd0c00640 1 -- 172.21.15.28:0/4201641072 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bcc152f80 con 0x7f9bcc156d90 2024-02-21T23:15:01.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:01.831+0000 7f9bd0c00640 1 --2- 172.21.15.28:0/4201641072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bcc156d90 0x7f9bcc06be80 secure :-1 s=READY pgs=2519 cs=0 l=1 rev1=1 crypto rx=0x7f9bc00097a0 tx=0x7f9bc002b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66845e155e69a278 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:01.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:01.831+0000 7f9bcb400640 1 -- 172.21.15.28:0/4201641072 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc0037070 con 0x7f9bcc156d90 2024-02-21T23:15:01.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:01.831+0000 7f9bcb400640 1 -- 172.21.15.28:0/4201641072 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bc0030510 con 0x7f9bcc156d90 2024-02-21T23:15:01.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:01.831+0000 7f9bcb400640 1 -- 172.21.15.28:0/4201641072 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb8001690 con 0x7f9bcc156d90 2024-02-21T23:15:01.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:01.832+0000 7f9bd2000640 1 -- 172.21.15.28:0/4201641072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bcc156d90 msgr2=0x7f9bcc06be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:01.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:01.832+0000 7f9bd2000640 1 --2- 172.21.15.28:0/4201641072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bcc156d90 0x7f9bcc06be80 secure :-1 s=READY pgs=2519 cs=0 l=1 rev1=1 crypto rx=0x7f9bc00097a0 tx=0x7f9bc002b710 comp rx=0 tx=0).stop 2024-02-21T23:15:01.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:01.832+0000 7f9bd2000640 1 -- 172.21.15.28:0/4201641072 shutdown_connections 2024-02-21T23:15:01.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:01.832+0000 7f9bd2000640 1 --2- 172.21.15.28:0/4201641072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bcc156d90 0x7f9bcc06be80 unknown :-1 s=CLOSED pgs=2519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:01.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:01.832+0000 7f9bd2000640 1 -- 172.21.15.28:0/4201641072 >> 172.21.15.28:0/4201641072 conn(0x7f9bcc071910 msgr2=0x7f9bcc071cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:15:01.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:01.832+0000 7f9bd2000640 1 -- 172.21.15.28:0/4201641072 shutdown_connections 2024-02-21T23:15:01.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:01.832+0000 7f9bd2000640 1 -- 172.21.15.28:0/4201641072 wait complete. 2024-02-21T23:15:01.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:01.833+0000 7f9bd2000640 1 Processor -- start 2024-02-21T23:15:01.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:01.834+0000 7f9bd2000640 1 -- start start 2024-02-21T23:15:01.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:01.834+0000 7f9bd2000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bcc0ac020 0x7f9bcc14cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:01.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:01.834+0000 7f9bd2000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9bcc156360 0x7f9bcc0c5b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:01.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:01.835+0000 7f9bd2000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9bcc156d90 0x7f9bcc1eab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:01.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:01.835+0000 7f9bd2000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bcc153d30 con 0x7f9bcc0ac020 2024-02-21T23:15:01.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:01.835+0000 7f9bd2000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bcc153bb0 con 0x7f9bcc156d90 2024-02-21T23:15:01.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:01.835+0000 7f9bd2000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bcc157170 con 0x7f9bcc156360 2024-02-21T23:15:01.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:01.835+0000 7f9bd0c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bcc0ac020 0x7f9bcc14cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:01.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:01.835+0000 7f9bd0c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bcc0ac020 0x7f9bcc14cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49600/0 (socket says 172.21.15.28:49600) 2024-02-21T23:15:01.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:01.835+0000 7f9bd0c00640 1 -- 172.21.15.28:0/568669335 learned_addr learned my addr 172.21.15.28:0/568669335 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:15:01.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:01.835+0000 7f9bd0c00640 1 -- 172.21.15.28:0/568669335 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9bcc156360 msgr2=0x7f9bcc0c5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:01.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:01.835+0000 7f9bd0c00640 1 --2- 172.21.15.28:0/568669335 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9bcc156360 0x7f9bcc0c5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:01.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:01.835+0000 7f9bd0c00640 1 -- 172.21.15.28:0/568669335 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9bcc156d90 msgr2=0x7f9bcc1eab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:01.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:01.835+0000 7f9bd0c00640 1 --2- 172.21.15.28:0/568669335 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9bcc156d90 0x7f9bcc1eab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:01.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:01.835+0000 7f9bd0c00640 1 -- 172.21.15.28:0/568669335 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bcc06c290 con 0x7f9bcc0ac020 2024-02-21T23:15:01.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:01.836+0000 7f9bd0c00640 1 --2- 172.21.15.28:0/568669335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bcc0ac020 0x7f9bcc14cf00 secure :-1 s=READY pgs=2520 cs=0 l=1 rev1=1 crypto rx=0x7f9bc002b750 tx=0x7f9bc00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:01.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:01.836+0000 7f9bc8c00640 1 -- 172.21.15.28:0/568669335 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc0042070 con 0x7f9bcc0ac020 2024-02-21T23:15:01.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:01.836+0000 7f9bc8c00640 1 -- 172.21.15.28:0/568669335 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bc003e870 con 0x7f9bcc0ac020 2024-02-21T23:15:01.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:01.836+0000 7f9bc8c00640 1 -- 172.21.15.28:0/568669335 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc0037040 con 0x7f9bcc0ac020 2024-02-21T23:15:01.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:01.836+0000 7f9bd2000640 1 -- 172.21.15.28:0/568669335 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bcc06c530 con 0x7f9bcc0ac020 2024-02-21T23:15:01.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:01.836+0000 7f9bd2000640 1 -- 172.21.15.28:0/568669335 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bcc1dd040 con 0x7f9bcc0ac020 2024-02-21T23:15:01.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:01.838+0000 7f9bd2000640 1 -- 172.21.15.28:0/568669335 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bcc04f290 con 0x7f9bcc0ac020 2024-02-21T23:15:01.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:01.838+0000 7f9bc8c00640 1 -- 172.21.15.28:0/568669335 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9bc0045070 con 0x7f9bcc0ac020 2024-02-21T23:15:01.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:01.840+0000 7f9bc8c00640 1 --2- 172.21.15.28:0/568669335 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9bac06b4d0 0x7f9bac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:01.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:01.840+0000 7f9bc8c00640 1 -- 172.21.15.28:0/568669335 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f9bc00ac140 con 0x7f9bcc0ac020 2024-02-21T23:15:01.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:01.841+0000 7f9bcbe00640 1 --2- 172.21.15.28:0/568669335 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9bac06b4d0 0x7f9bac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:01.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:01.842+0000 7f9bcbe00640 1 --2- 172.21.15.28:0/568669335 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9bac06b4d0 0x7f9bac06d990 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc1e2240 tx=0x7f9bb80010c0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:01.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:01.844+0000 7f9bc8c00640 1 -- 172.21.15.28:0/568669335 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f9bc007adc0 con 0x7f9bcc0ac020 2024-02-21T23:15:02.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:02.081+0000 7f9bd2000640 1 -- 172.21.15.28:0/568669335 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9bcc0c29e0 con 0x7f9bcc0ac020 2024-02-21T23:15:02.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:02.082+0000 7f9bc8c00640 1 -- 172.21.15.28:0/568669335 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 246 v246) v1 ==== 95+0+3384 (secure 0 0 0) 0x7f9bc007a6c0 con 0x7f9bcc0ac020 2024-02-21T23:15:02.070 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:15:02.071 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":246,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":246,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:14:46.655515+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:15:02.071 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 246 2024-02-21T23:15:02.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:02.085+0000 7f9bd2000640 1 -- 172.21.15.28:0/568669335 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9bac06b4d0 msgr2=0x7f9bac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:02.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:02.085+0000 7f9bd2000640 1 --2- 172.21.15.28:0/568669335 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9bac06b4d0 0x7f9bac06d990 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc1e2240 tx=0x7f9bb80010c0 comp rx=0 tx=0).stop 2024-02-21T23:15:02.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:02.086+0000 7f9bd2000640 1 -- 172.21.15.28:0/568669335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bcc0ac020 msgr2=0x7f9bcc14cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:02.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:02.086+0000 7f9bd2000640 1 --2- 172.21.15.28:0/568669335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bcc0ac020 0x7f9bcc14cf00 secure :-1 s=READY pgs=2520 cs=0 l=1 rev1=1 crypto rx=0x7f9bc002b750 tx=0x7f9bc00046c0 comp rx=0 tx=0).stop 2024-02-21T23:15:02.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:02.086+0000 7f9bd2000640 1 -- 172.21.15.28:0/568669335 shutdown_connections 2024-02-21T23:15:02.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:02.086+0000 7f9bd2000640 1 --2- 172.21.15.28:0/568669335 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9bac06b4d0 0x7f9bac06d990 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:02.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:02.086+0000 7f9bd2000640 1 --2- 172.21.15.28:0/568669335 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9bcc156360 0x7f9bcc0c5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:02.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:02.086+0000 7f9bd2000640 1 --2- 172.21.15.28:0/568669335 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9bcc156d90 0x7f9bcc1eab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:02.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:02.086+0000 7f9bd2000640 1 --2- 172.21.15.28:0/568669335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bcc0ac020 0x7f9bcc14cf00 unknown :-1 s=CLOSED pgs=2520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:02.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:02.086+0000 7f9bd2000640 1 -- 172.21.15.28:0/568669335 >> 172.21.15.28:0/568669335 conn(0x7f9bcc071910 msgr2=0x7f9bcc0c0370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:15:02.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:02.086+0000 7f9bd2000640 1 -- 172.21.15.28:0/568669335 shutdown_connections 2024-02-21T23:15:02.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:02.086+0000 7f9bd2000640 1 -- 172.21.15.28:0/568669335 wait complete. 2024-02-21T23:15:02.495 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 246, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:14:46.655515+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:15:02.495 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:15:03.496 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:15:03.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.734+0000 7fcfa2400640 1 Processor -- start 2024-02-21T23:15:03.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.735+0000 7fcfa2400640 1 -- start start 2024-02-21T23:15:03.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.736+0000 7fcfa2400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcf9c0c7be0 0x7fcf9c159dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:03.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.736+0000 7fcfa2400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf9c159610 con 0x7fcf9c1568c0 2024-02-21T23:15:03.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.736+0000 7fcfa2400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf9c15d400 con 0x7fcf9c0c2a90 2024-02-21T23:15:03.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.736+0000 7fcfa2400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9c0c6080 con 0x7fcf9c0c7be0 2024-02-21T23:15:03.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.737+0000 7fcf9b400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fcf9c1568c0 0x7fcf9c15a8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34574/0 (socket says 172.21.15.28:34574) 2024-02-21T23:15:03.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.737+0000 7fcf9b400640 1 -- 172.21.15.28:0/2988791483 learned_addr learned my addr 172.21.15.28:0/2988791483 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:15:03.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.737+0000 7fcf9aa00640 1 --2- 172.21.15.28:0/2988791483 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcf9c0c7be0 0x7fcf9c159dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:03.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.737+0000 7fcf9a000640 1 -- 172.21.15.28:0/2988791483 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4290873515 0 0) 0x7fcf9c159610 con 0x7fcf9c1568c0 2024-02-21T23:15:03.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.737+0000 7fcf9a000640 1 -- 172.21.15.28:0/2988791483 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf7c003520 con 0x7fcf9c1568c0 2024-02-21T23:15:03.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.738+0000 7fcf9a000640 1 -- 172.21.15.28:0/2988791483 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 20671253 0 0) 0x7fcf7c003520 con 0x7fcf9c1568c0 2024-02-21T23:15:03.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.738+0000 7fcf9a000640 1 -- 172.21.15.28:0/2988791483 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcf9c0c7be0 msgr2=0x7fcf9c159dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:15:03.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.738+0000 7fcf9a000640 1 --2- 172.21.15.28:0/2988791483 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcf9c0c7be0 0x7fcf9c159dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:03.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.738+0000 7fcf9a000640 1 -- 172.21.15.28:0/2988791483 >> v1:172.21.15.81:6789/0 conn(0x7fcf9c0c2a90 legacy=0x7fcf9c06d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:03.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.738+0000 7fcf9a000640 1 -- 172.21.15.28:0/2988791483 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf9c15d6e0 con 0x7fcf9c1568c0 2024-02-21T23:15:03.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.738+0000 7fcf9a000640 1 -- 172.21.15.28:0/2988791483 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fcf8c001670 con 0x7fcf9c1568c0 2024-02-21T23:15:03.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.738+0000 7fcf9a000640 1 -- 172.21.15.28:0/2988791483 >> v1:172.21.15.28:6789/0 conn(0x7fcf9c1568c0 legacy=0x7fcf9c15a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:03.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.738+0000 7fcf9a000640 1 --2- 172.21.15.28:0/2988791483 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcf9c0c7be0 0x7fcf7c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:03.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.738+0000 7fcf9a000640 1 --2- 172.21.15.28:0/2988791483 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf7c003fb0 0x7fcf7c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:03.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.738+0000 7fcf9a000640 1 --2- 172.21.15.28:0/2988791483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf7c004d40 0x7fcf7c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:03.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.738+0000 7fcf9a000640 1 -- 172.21.15.28:0/2988791483 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9c0c6080 con 0x7fcf7c004d40 2024-02-21T23:15:03.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.738+0000 7fcf9a000640 1 -- 172.21.15.28:0/2988791483 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf7c00bb00 con 0x7fcf7c003fb0 2024-02-21T23:15:03.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.738+0000 7fcf9a000640 1 -- 172.21.15.28:0/2988791483 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf7c00bd00 con 0x7fcf9c0c7be0 2024-02-21T23:15:03.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.738+0000 7fcf9be00640 1 --2- 172.21.15.28:0/2988791483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf7c004d40 0x7fcf7c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:03.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.738+0000 7fcf9b400640 1 --2- 172.21.15.28:0/2988791483 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcf9c0c7be0 0x7fcf7c0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:03.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.739+0000 7fcf9aa00640 1 --2- 172.21.15.28:0/2988791483 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf7c003fb0 0x7fcf7c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:03.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.739+0000 7fcf9be00640 1 -- 172.21.15.28:0/2988791483 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcf9c0c7be0 msgr2=0x7fcf7c0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:15:03.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.739+0000 7fcf9be00640 1 --2- 172.21.15.28:0/2988791483 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcf9c0c7be0 0x7fcf7c0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:03.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.739+0000 7fcf9be00640 1 -- 172.21.15.28:0/2988791483 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf7c003fb0 msgr2=0x7fcf7c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:15:03.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.739+0000 7fcf9be00640 1 --2- 172.21.15.28:0/2988791483 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf7c003fb0 0x7fcf7c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:03.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.739+0000 7fcf9be00640 1 -- 172.21.15.28:0/2988791483 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf7c003520 con 0x7fcf7c004d40 2024-02-21T23:15:03.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.740+0000 7fcf9be00640 1 --2- 172.21.15.28:0/2988791483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf7c004d40 0x7fcf7c0051c0 secure :-1 s=READY pgs=2522 cs=0 l=1 rev1=1 crypto rx=0x7fcf94001e20 tx=0x7fcf7c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df793e8fa4133b58 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:03.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.741+0000 7fcf9a000640 1 -- 172.21.15.28:0/2988791483 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf94001a70 con 0x7fcf7c004d40 2024-02-21T23:15:03.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.741+0000 7fcf9a000640 1 -- 172.21.15.28:0/2988791483 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf94003b70 con 0x7fcf7c004d40 2024-02-21T23:15:03.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.741+0000 7fcf9a000640 1 -- 172.21.15.28:0/2988791483 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf940050d0 con 0x7fcf7c004d40 2024-02-21T23:15:03.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.741+0000 7fcfa2400640 1 -- 172.21.15.28:0/2988791483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf7c004d40 msgr2=0x7fcf7c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:03.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.741+0000 7fcfa2400640 1 --2- 172.21.15.28:0/2988791483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf7c004d40 0x7fcf7c0051c0 secure :-1 s=READY pgs=2522 cs=0 l=1 rev1=1 crypto rx=0x7fcf94001e20 tx=0x7fcf7c005830 comp rx=0 tx=0).stop 2024-02-21T23:15:03.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.741+0000 7fcf9be00640 1 -- 172.21.15.28:0/2988791483 reap_dead start 2024-02-21T23:15:03.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.742+0000 7fcfa2400640 1 -- 172.21.15.28:0/2988791483 shutdown_connections 2024-02-21T23:15:03.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.742+0000 7fcfa2400640 1 -- 172.21.15.28:0/2988791483 >> 172.21.15.28:0/2988791483 conn(0x7fcf9c0c8810 msgr2=0x7fcf9c0c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:15:03.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.742+0000 7fcfa2400640 1 -- 172.21.15.28:0/2988791483 shutdown_connections 2024-02-21T23:15:03.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.742+0000 7fcfa2400640 1 -- 172.21.15.28:0/2988791483 wait complete. 2024-02-21T23:15:03.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.743+0000 7fcfa2400640 1 Processor -- start 2024-02-21T23:15:03.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.744+0000 7fcfa2400640 1 -- start start 2024-02-21T23:15:03.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.745+0000 7fcfa2400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf7c004d40 0x7fcf9c0ad1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:03.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.745+0000 7fcfa2400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcf9c1f25a0 0x7fcf9c1f2a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:03.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.745+0000 7fcfa2400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf9c1f2f60 0x7fcf9c0ab4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:03.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.745+0000 7fcfa2400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9c0c3490 con 0x7fcf7c004d40 2024-02-21T23:15:03.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.745+0000 7fcfa2400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9c15d9c0 con 0x7fcf9c1f2f60 2024-02-21T23:15:03.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.745+0000 7fcfa2400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9c159940 con 0x7fcf9c1f25a0 2024-02-21T23:15:03.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.745+0000 7fcf9b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf7c004d40 0x7fcf9c0ad1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:03.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.745+0000 7fcf9b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf7c004d40 0x7fcf9c0ad1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49610/0 (socket says 172.21.15.28:49610) 2024-02-21T23:15:03.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.745+0000 7fcf9b400640 1 -- 172.21.15.28:0/2307803041 learned_addr learned my addr 172.21.15.28:0/2307803041 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:15:03.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.745+0000 7fcf9aa00640 1 --2- 172.21.15.28:0/2307803041 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcf9c1f25a0 0x7fcf9c1f2a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:03.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.745+0000 7fcf9be00640 1 --2- 172.21.15.28:0/2307803041 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf9c1f2f60 0x7fcf9c0ab4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:03.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.745+0000 7fcf9b400640 1 -- 172.21.15.28:0/2307803041 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcf9c1f25a0 msgr2=0x7fcf9c1f2a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:03.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.745+0000 7fcf9b400640 1 --2- 172.21.15.28:0/2307803041 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcf9c1f25a0 0x7fcf9c1f2a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:03.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.745+0000 7fcf9b400640 1 -- 172.21.15.28:0/2307803041 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf9c1f2f60 msgr2=0x7fcf9c0ab4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:03.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.745+0000 7fcf9b400640 1 --2- 172.21.15.28:0/2307803041 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf9c1f2f60 0x7fcf9c0ab4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:03.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.745+0000 7fcf9b400640 1 -- 172.21.15.28:0/2307803041 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf9c1eee30 con 0x7fcf7c004d40 2024-02-21T23:15:03.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.746+0000 7fcf9b400640 1 --2- 172.21.15.28:0/2307803041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf7c004d40 0x7fcf9c0ad1a0 secure :-1 s=READY pgs=2523 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c00e7f0 tx=0x7fcf8c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:03.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.746+0000 7fcf93400640 1 -- 172.21.15.28:0/2307803041 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf8c00c870 con 0x7fcf7c004d40 2024-02-21T23:15:03.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.746+0000 7fcf93400640 1 -- 172.21.15.28:0/2307803041 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf8c018070 con 0x7fcf7c004d40 2024-02-21T23:15:03.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.746+0000 7fcf93400640 1 -- 172.21.15.28:0/2307803041 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf8c00cc70 con 0x7fcf7c004d40 2024-02-21T23:15:03.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.746+0000 7fcfa2400640 1 -- 172.21.15.28:0/2307803041 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf9c1ef0f0 con 0x7fcf7c004d40 2024-02-21T23:15:03.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.746+0000 7fcfa2400640 1 -- 172.21.15.28:0/2307803041 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf9c1ef6b0 con 0x7fcf7c004d40 2024-02-21T23:15:03.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.747+0000 7fcfa2400640 1 -- 172.21.15.28:0/2307803041 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf9c068640 con 0x7fcf7c004d40 2024-02-21T23:15:03.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.748+0000 7fcf93400640 1 -- 172.21.15.28:0/2307803041 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcf8c01d460 con 0x7fcf7c004d40 2024-02-21T23:15:03.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.750+0000 7fcf93400640 1 --2- 172.21.15.28:0/2307803041 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcf7806b500 0x7fcf7806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:03.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.750+0000 7fcf93400640 1 -- 172.21.15.28:0/2307803041 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fcf8c08a400 con 0x7fcf7c004d40 2024-02-21T23:15:03.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.750+0000 7fcf9aa00640 1 --2- 172.21.15.28:0/2307803041 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcf7806b500 0x7fcf7806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:03.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.752+0000 7fcf9aa00640 1 --2- 172.21.15.28:0/2307803041 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcf7806b500 0x7fcf7806d9c0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c158d10 tx=0x7fcf84008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:03.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.755+0000 7fcf93400640 1 -- 172.21.15.28:0/2307803041 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fcf8c058fe0 con 0x7fcf7c004d40 2024-02-21T23:15:03.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.996+0000 7fcfa2400640 1 -- 172.21.15.28:0/2307803041 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcf9c0c79c0 con 0x7fcf7c004d40 2024-02-21T23:15:03.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:03.997+0000 7fcf93400640 1 -- 172.21.15.28:0/2307803041 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 246 v246) v1 ==== 95+0+3384 (secure 0 0 0) 0x7fcf8c0588e0 con 0x7fcf7c004d40 2024-02-21T23:15:03.985 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:15:03.985 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":246,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":246,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:14:46.655515+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:15:03.985 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 246 2024-02-21T23:15:03.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:04.000+0000 7fcfa2400640 1 -- 172.21.15.28:0/2307803041 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcf7806b500 msgr2=0x7fcf7806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:03.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:04.000+0000 7fcfa2400640 1 --2- 172.21.15.28:0/2307803041 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcf7806b500 0x7fcf7806d9c0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c158d10 tx=0x7fcf84008040 comp rx=0 tx=0).stop 2024-02-21T23:15:03.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:04.000+0000 7fcfa2400640 1 -- 172.21.15.28:0/2307803041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf7c004d40 msgr2=0x7fcf9c0ad1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:03.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:04.000+0000 7fcfa2400640 1 --2- 172.21.15.28:0/2307803041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf7c004d40 0x7fcf9c0ad1a0 secure :-1 s=READY pgs=2523 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c00e7f0 tx=0x7fcf8c00ecb0 comp rx=0 tx=0).stop 2024-02-21T23:15:03.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:04.000+0000 7fcfa2400640 1 -- 172.21.15.28:0/2307803041 shutdown_connections 2024-02-21T23:15:03.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:04.000+0000 7fcfa2400640 1 --2- 172.21.15.28:0/2307803041 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcf7806b500 0x7fcf7806d9c0 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:03.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:04.000+0000 7fcfa2400640 1 --2- 172.21.15.28:0/2307803041 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcf9c1f25a0 0x7fcf9c1f2a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:03.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:04.000+0000 7fcfa2400640 1 --2- 172.21.15.28:0/2307803041 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf9c1f2f60 0x7fcf9c0ab4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:03.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:04.000+0000 7fcfa2400640 1 --2- 172.21.15.28:0/2307803041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf7c004d40 0x7fcf9c0ad1a0 unknown :-1 s=CLOSED pgs=2523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:03.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:04.000+0000 7fcfa2400640 1 -- 172.21.15.28:0/2307803041 >> 172.21.15.28:0/2307803041 conn(0x7fcf9c0c8810 msgr2=0x7fcf9c15c110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:15:03.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:04.000+0000 7fcfa2400640 1 -- 172.21.15.28:0/2307803041 shutdown_connections 2024-02-21T23:15:03.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:04.001+0000 7fcfa2400640 1 -- 172.21.15.28:0/2307803041 wait complete. 2024-02-21T23:15:04.412 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 246, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:14:46.655515+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:15:04.412 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:15:05.413 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:15:05.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.655+0000 7f2ce4a00640 1 Processor -- start 2024-02-21T23:15:05.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.655+0000 7f2ce4a00640 1 -- start start 2024-02-21T23:15:05.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.657+0000 7f2ce4a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f2ce00c11f0 0x7f2ce0153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:05.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.657+0000 7f2ce4a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2ce00c15d0 0x7f2ce006dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:05.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.657+0000 7f2ce4a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2ce0156350 0x7f2ce0156780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:05.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.657+0000 7f2ce4a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ce00c1a60 con 0x7f2ce00c15d0 2024-02-21T23:15:05.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.657+0000 7f2ce4a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ce00a5ec0 con 0x7f2ce00c11f0 2024-02-21T23:15:05.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.657+0000 7f2ce4a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce0152e70 con 0x7f2ce0156350 2024-02-21T23:15:05.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.657+0000 7f2cdd400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2ce00c15d0 0x7f2ce006dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:05.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.657+0000 7f2cdd400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2ce00c15d0 0x7f2ce006dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53750/0 (socket says 172.21.15.28:53750) 2024-02-21T23:15:05.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.657+0000 7f2cdd400640 1 -- 172.21.15.28:0/3317434755 learned_addr learned my addr 172.21.15.28:0/3317434755 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:15:05.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.657+0000 7f2cdde00640 1 --2- 172.21.15.28:0/3317434755 >> v2:172.21.15.81:3300/0 conn(0x7f2ce00c11f0 0x7f2ce0153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:05.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.658+0000 7f2cde800640 1 --2- 172.21.15.28:0/3317434755 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2ce0156350 0x7f2ce0156780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:05.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.658+0000 7f2cdd400640 1 -- 172.21.15.28:0/3317434755 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2ce0156350 msgr2=0x7f2ce0156780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:15:05.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.658+0000 7f2cdd400640 1 --2- 172.21.15.28:0/3317434755 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2ce0156350 0x7f2ce0156780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:05.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.658+0000 7f2cdd400640 1 -- 172.21.15.28:0/3317434755 >> v2:172.21.15.81:3300/0 conn(0x7f2ce00c11f0 msgr2=0x7f2ce0153950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:15:05.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.658+0000 7f2cdd400640 1 --2- 172.21.15.28:0/3317434755 >> v2:172.21.15.81:3300/0 conn(0x7f2ce00c11f0 0x7f2ce0153950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:05.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.658+0000 7f2cdd400640 1 -- 172.21.15.28:0/3317434755 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ce0153150 con 0x7f2ce00c15d0 2024-02-21T23:15:05.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.661+0000 7f2cdd400640 1 --2- 172.21.15.28:0/3317434755 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ce00c15d0 0x7f2ce006dce0 secure :-1 s=READY pgs=2524 cs=0 l=1 rev1=1 crypto rx=0x7f2ce00c7590 tx=0x7f2ccc02b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b68a78b81fcfd59e server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:05.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.661+0000 7f2cdca00640 1 -- 172.21.15.28:0/3317434755 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ccc038070 con 0x7f2ce00c15d0 2024-02-21T23:15:05.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.661+0000 7f2cdca00640 1 -- 172.21.15.28:0/3317434755 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ccc02f050 con 0x7f2ce00c15d0 2024-02-21T23:15:05.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.661+0000 7f2cdca00640 1 -- 172.21.15.28:0/3317434755 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ccc031970 con 0x7f2ce00c15d0 2024-02-21T23:15:05.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.662+0000 7f2ce4a00640 1 -- 172.21.15.28:0/3317434755 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ce00c15d0 msgr2=0x7f2ce006dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:05.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.662+0000 7f2ce4a00640 1 --2- 172.21.15.28:0/3317434755 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ce00c15d0 0x7f2ce006dce0 secure :-1 s=READY pgs=2524 cs=0 l=1 rev1=1 crypto rx=0x7f2ce00c7590 tx=0x7f2ccc02b7f0 comp rx=0 tx=0).stop 2024-02-21T23:15:05.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.662+0000 7f2ce4a00640 1 -- 172.21.15.28:0/3317434755 shutdown_connections 2024-02-21T23:15:05.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.662+0000 7f2ce4a00640 1 --2- 172.21.15.28:0/3317434755 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2ce0156350 0x7f2ce0156780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:05.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.662+0000 7f2ce4a00640 1 --2- 172.21.15.28:0/3317434755 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ce00c15d0 0x7f2ce006dce0 unknown :-1 s=CLOSED pgs=2524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:05.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.662+0000 7f2ce4a00640 1 --2- 172.21.15.28:0/3317434755 >> v2:172.21.15.81:3300/0 conn(0x7f2ce00c11f0 0x7f2ce0153950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:05.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.662+0000 7f2ce4a00640 1 -- 172.21.15.28:0/3317434755 >> 172.21.15.28:0/3317434755 conn(0x7f2ce00c67f0 msgr2=0x7f2ce00c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:15:05.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.662+0000 7f2ce4a00640 1 -- 172.21.15.28:0/3317434755 shutdown_connections 2024-02-21T23:15:05.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.662+0000 7f2ce4a00640 1 -- 172.21.15.28:0/3317434755 wait complete. 2024-02-21T23:15:05.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.665+0000 7f2ce4a00640 1 Processor -- start 2024-02-21T23:15:05.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.665+0000 7f2ce4a00640 1 -- start start 2024-02-21T23:15:05.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.667+0000 7f2ce4a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ce00c11f0 0x7f2ce0140d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:05.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.667+0000 7f2ce4a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ce00c15d0 0x7f2ce01ecc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:05.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.667+0000 7f2ce4a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2ce0156350 0x7f2ce0154b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:05.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.667+0000 7f2ce4a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce00c4c00 con 0x7f2ce00c11f0 2024-02-21T23:15:05.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.667+0000 7f2ce4a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce014d1c0 con 0x7f2ce00c15d0 2024-02-21T23:15:05.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.667+0000 7f2ce4a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce00a6110 con 0x7f2ce0156350 2024-02-21T23:15:05.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.667+0000 7f2cdde00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ce00c11f0 0x7f2ce0140d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:05.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.667+0000 7f2cdde00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ce00c11f0 0x7f2ce0140d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53760/0 (socket says 172.21.15.28:53760) 2024-02-21T23:15:05.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.667+0000 7f2cdde00640 1 -- 172.21.15.28:0/3390224467 learned_addr learned my addr 172.21.15.28:0/3390224467 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:15:05.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.667+0000 7f2cdd400640 1 --2- 172.21.15.28:0/3390224467 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ce00c15d0 0x7f2ce01ecc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:05.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.668+0000 7f2cde800640 1 --2- 172.21.15.28:0/3390224467 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2ce0156350 0x7f2ce0154b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:05.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.668+0000 7f2cdde00640 1 -- 172.21.15.28:0/3390224467 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2ce0156350 msgr2=0x7f2ce0154b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:05.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.668+0000 7f2cdde00640 1 --2- 172.21.15.28:0/3390224467 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2ce0156350 0x7f2ce0154b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:05.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.668+0000 7f2cdde00640 1 -- 172.21.15.28:0/3390224467 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ce00c15d0 msgr2=0x7f2ce01ecc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:05.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.668+0000 7f2cdde00640 1 --2- 172.21.15.28:0/3390224467 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ce00c15d0 0x7f2ce01ecc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:05.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.668+0000 7f2cdde00640 1 -- 172.21.15.28:0/3390224467 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ce01412e0 con 0x7f2ce00c11f0 2024-02-21T23:15:05.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.668+0000 7f2cdd400640 1 --2- 172.21.15.28:0/3390224467 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ce00c15d0 0x7f2ce01ecc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:15:05.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.668+0000 7f2cdde00640 1 --2- 172.21.15.28:0/3390224467 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ce00c11f0 0x7f2ce0140d00 secure :-1 s=READY pgs=2525 cs=0 l=1 rev1=1 crypto rx=0x7f2cd000e870 tx=0x7f2cd000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:05.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.669+0000 7f2cd6000640 1 -- 172.21.15.28:0/3390224467 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd0017070 con 0x7f2ce00c11f0 2024-02-21T23:15:05.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.669+0000 7f2cd6000640 1 -- 172.21.15.28:0/3390224467 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cd000ee60 con 0x7f2ce00c11f0 2024-02-21T23:15:05.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.669+0000 7f2cd6000640 1 -- 172.21.15.28:0/3390224467 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd000cb80 con 0x7f2ce00c11f0 2024-02-21T23:15:05.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.669+0000 7f2ce4a00640 1 -- 172.21.15.28:0/3390224467 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ce01415c0 con 0x7f2ce00c11f0 2024-02-21T23:15:05.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.669+0000 7f2ce4a00640 1 -- 172.21.15.28:0/3390224467 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ce01ec6e0 con 0x7f2ce00c11f0 2024-02-21T23:15:05.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.670+0000 7f2ce4a00640 1 -- 172.21.15.28:0/3390224467 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ce0067f30 con 0x7f2ce00c11f0 2024-02-21T23:15:05.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.671+0000 7f2cd6000640 1 -- 172.21.15.28:0/3390224467 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2cd001a070 con 0x7f2ce00c11f0 2024-02-21T23:15:05.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.672+0000 7f2cd6000640 1 --2- 172.21.15.28:0/3390224467 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2cc006b4d0 0x7f2cc006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:05.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.672+0000 7f2cd6000640 1 -- 172.21.15.28:0/3390224467 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f2cd008a280 con 0x7f2ce00c11f0 2024-02-21T23:15:05.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.673+0000 7f2cdd400640 1 --2- 172.21.15.28:0/3390224467 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2cc006b4d0 0x7f2cc006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:05.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.674+0000 7f2cdd400640 1 --2- 172.21.15.28:0/3390224467 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2cc006b4d0 0x7f2cc006d990 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc035000 tx=0x7f2ccc008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:05.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.677+0000 7f2cd6000640 1 -- 172.21.15.28:0/3390224467 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f2cd0058f00 con 0x7f2ce00c11f0 2024-02-21T23:15:05.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.919+0000 7f2ce4a00640 1 -- 172.21.15.28:0/3390224467 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2ce00c0960 con 0x7f2ce00c11f0 2024-02-21T23:15:05.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.920+0000 7f2cd6000640 1 -- 172.21.15.28:0/3390224467 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 246 v246) v1 ==== 95+0+3384 (secure 0 0 0) 0x7f2cd0058800 con 0x7f2ce00c11f0 2024-02-21T23:15:05.907 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:15:05.907 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":246,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":246,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:14:46.655515+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:15:05.908 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 246 2024-02-21T23:15:05.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.923+0000 7f2ce4a00640 1 -- 172.21.15.28:0/3390224467 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2cc006b4d0 msgr2=0x7f2cc006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:05.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.923+0000 7f2ce4a00640 1 --2- 172.21.15.28:0/3390224467 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2cc006b4d0 0x7f2cc006d990 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc035000 tx=0x7f2ccc008030 comp rx=0 tx=0).stop 2024-02-21T23:15:05.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.923+0000 7f2ce4a00640 1 -- 172.21.15.28:0/3390224467 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ce00c11f0 msgr2=0x7f2ce0140d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:05.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.923+0000 7f2ce4a00640 1 --2- 172.21.15.28:0/3390224467 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ce00c11f0 0x7f2ce0140d00 secure :-1 s=READY pgs=2525 cs=0 l=1 rev1=1 crypto rx=0x7f2cd000e870 tx=0x7f2cd000ed30 comp rx=0 tx=0).stop 2024-02-21T23:15:05.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.923+0000 7f2ce4a00640 1 -- 172.21.15.28:0/3390224467 shutdown_connections 2024-02-21T23:15:05.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.923+0000 7f2ce4a00640 1 --2- 172.21.15.28:0/3390224467 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2cc006b4d0 0x7f2cc006d990 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:05.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.923+0000 7f2ce4a00640 1 --2- 172.21.15.28:0/3390224467 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2ce0156350 0x7f2ce0154b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:05.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.923+0000 7f2ce4a00640 1 --2- 172.21.15.28:0/3390224467 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ce00c15d0 0x7f2ce01ecc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:05.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.923+0000 7f2ce4a00640 1 --2- 172.21.15.28:0/3390224467 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ce00c11f0 0x7f2ce0140d00 unknown :-1 s=CLOSED pgs=2525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:05.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.923+0000 7f2ce4a00640 1 -- 172.21.15.28:0/3390224467 >> 172.21.15.28:0/3390224467 conn(0x7f2ce00c67f0 msgr2=0x7f2ce0148bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:15:05.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.923+0000 7f2ce4a00640 1 -- 172.21.15.28:0/3390224467 shutdown_connections 2024-02-21T23:15:05.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:05.923+0000 7f2ce4a00640 1 -- 172.21.15.28:0/3390224467 wait complete. 2024-02-21T23:15:06.331 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 246, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:14:46.655515+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:15:06.331 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:15:07.332 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:15:07.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.559+0000 7ff0b7200640 1 Processor -- start 2024-02-21T23:15:07.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.560+0000 7ff0b7200640 1 -- start start 2024-02-21T23:15:07.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.561+0000 7ff0b7200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7ff0b00c3490 0x7ff0b006dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:07.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.561+0000 7ff0b7200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff0b00c0270 0x7ff0b00c0700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:07.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.561+0000 7ff0b7200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0b00c0c40 con 0x7ff0b00a5f40 2024-02-21T23:15:07.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.561+0000 7ff0b7200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0b006eff0 con 0x7ff0b00c3490 2024-02-21T23:15:07.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.561+0000 7ff0b7200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b00bfde0 con 0x7ff0b00c0270 2024-02-21T23:15:07.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.562+0000 7ff0afe00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7ff0b00c3490 0x7ff0b006dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:07.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.562+0000 7ff0b5200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff0b00c0270 0x7ff0b00c0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:07.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.562+0000 7ff0afe00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7ff0b00c3490 0x7ff0b006dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.28:34536/0 (socket says 172.21.15.28:34536) 2024-02-21T23:15:07.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.562+0000 7ff0a7e00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7ff0b00a5f40 0x7ff0b0152140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.28:39226/0 (socket says 172.21.15.28:39226) 2024-02-21T23:15:07.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.562+0000 7ff0afe00640 1 -- 172.21.15.28:0/2944562134 learned_addr learned my addr 172.21.15.28:0/2944562134 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:15:07.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.563+0000 7ff0afe00640 1 -- 172.21.15.28:0/2944562134 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff0b00c0270 msgr2=0x7ff0b00c0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:15:07.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.563+0000 7ff0afe00640 1 --2- 172.21.15.28:0/2944562134 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff0b00c0270 0x7ff0b00c0700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:07.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.563+0000 7ff0afe00640 1 -- 172.21.15.28:0/2944562134 >> v1:172.21.15.81:6789/0 conn(0x7ff0b00a5f40 legacy=0x7ff0b0152140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:07.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.563+0000 7ff0af400640 1 -- 172.21.15.28:0/2944562134 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1024041136 0 0) 0x7ff0b00c0c40 con 0x7ff0b00a5f40 2024-02-21T23:15:07.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.563+0000 7ff0afe00640 1 -- 172.21.15.28:0/2944562134 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0b006e750 con 0x7ff0b00c3490 2024-02-21T23:15:07.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.564+0000 7ff0afe00640 1 --2- 172.21.15.28:0/2944562134 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0b00c3490 0x7ff0b006dfb0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7ff0a0009870 tx=0x7ff0a002b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=406bd35a4c536eb1 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:07.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.565+0000 7ff0af400640 1 -- 172.21.15.28:0/2944562134 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a0035020 con 0x7ff0b00c3490 2024-02-21T23:15:07.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.565+0000 7ff0af400640 1 -- 172.21.15.28:0/2944562134 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0a0031640 con 0x7ff0b00c3490 2024-02-21T23:15:07.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.565+0000 7ff0af400640 1 -- 172.21.15.28:0/2944562134 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a002d040 con 0x7ff0b00c3490 2024-02-21T23:15:07.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.565+0000 7ff0b7200640 1 -- 172.21.15.28:0/2944562134 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0b00c3490 msgr2=0x7ff0b006dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:07.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.565+0000 7ff0b7200640 1 --2- 172.21.15.28:0/2944562134 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0b00c3490 0x7ff0b006dfb0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7ff0a0009870 tx=0x7ff0a002b8f0 comp rx=0 tx=0).stop 2024-02-21T23:15:07.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.566+0000 7ff0b7200640 1 -- 172.21.15.28:0/2944562134 shutdown_connections 2024-02-21T23:15:07.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.566+0000 7ff0b7200640 1 --2- 172.21.15.28:0/2944562134 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff0b00c0270 0x7ff0b00c0700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:07.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.566+0000 7ff0b7200640 1 --2- 172.21.15.28:0/2944562134 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0b00c3490 0x7ff0b006dfb0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:07.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.566+0000 7ff0b7200640 1 -- 172.21.15.28:0/2944562134 >> 172.21.15.28:0/2944562134 conn(0x7ff0b00c67f0 msgr2=0x7ff0b00c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:15:07.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.566+0000 7ff0b7200640 1 -- 172.21.15.28:0/2944562134 shutdown_connections 2024-02-21T23:15:07.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.566+0000 7ff0b7200640 1 -- 172.21.15.28:0/2944562134 wait complete. 2024-02-21T23:15:07.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.568+0000 7ff0b7200640 1 Processor -- start 2024-02-21T23:15:07.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.568+0000 7ff0b7200640 1 -- start start 2024-02-21T23:15:07.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.569+0000 7ff0b7200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0b00a5f40 0x7ff0b0149bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:07.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.569+0000 7ff0b7200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0b00c0270 0x7ff0b00722d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:07.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.569+0000 7ff0b7200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff0b00c3490 0x7ff0b01ee3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:07.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.569+0000 7ff0b7200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b006ed10 con 0x7ff0b00a5f40 2024-02-21T23:15:07.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.569+0000 7ff0b7200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b006e4f0 con 0x7ff0b00c0270 2024-02-21T23:15:07.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.569+0000 7ff0b7200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b00c1c70 con 0x7ff0b00c3490 2024-02-21T23:15:07.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.569+0000 7ff0afe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0b00a5f40 0x7ff0b0149bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:07.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.569+0000 7ff0afe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0b00a5f40 0x7ff0b0149bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53772/0 (socket says 172.21.15.28:53772) 2024-02-21T23:15:07.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.569+0000 7ff0afe00640 1 -- 172.21.15.28:0/4165091850 learned_addr learned my addr 172.21.15.28:0/4165091850 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:15:07.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.569+0000 7ff0a7e00640 1 --2- 172.21.15.28:0/4165091850 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0b00c0270 0x7ff0b00722d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:07.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.569+0000 7ff0b5200640 1 --2- 172.21.15.28:0/4165091850 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff0b00c3490 0x7ff0b01ee3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:07.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.570+0000 7ff0afe00640 1 -- 172.21.15.28:0/4165091850 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff0b00c3490 msgr2=0x7ff0b01ee3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:07.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.570+0000 7ff0afe00640 1 --2- 172.21.15.28:0/4165091850 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff0b00c3490 0x7ff0b01ee3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:07.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.570+0000 7ff0afe00640 1 -- 172.21.15.28:0/4165091850 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0b00c0270 msgr2=0x7ff0b00722d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:07.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.570+0000 7ff0afe00640 1 --2- 172.21.15.28:0/4165091850 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0b00c0270 0x7ff0b00722d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:07.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.570+0000 7ff0afe00640 1 -- 172.21.15.28:0/4165091850 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0b01498d0 con 0x7ff0b00a5f40 2024-02-21T23:15:07.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.570+0000 7ff0b5200640 1 --2- 172.21.15.28:0/4165091850 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff0b00c3490 0x7ff0b01ee3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:15:07.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.570+0000 7ff0afe00640 1 --2- 172.21.15.28:0/4165091850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0b00a5f40 0x7ff0b0149bb0 secure :-1 s=READY pgs=2526 cs=0 l=1 rev1=1 crypto rx=0x7ff0a0000c00 tx=0x7ff0a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:07.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.570+0000 7ff0acc00640 1 -- 172.21.15.28:0/4165091850 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a0040070 con 0x7ff0b00a5f40 2024-02-21T23:15:07.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.570+0000 7ff0acc00640 1 -- 172.21.15.28:0/4165091850 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0a00399f0 con 0x7ff0b00a5f40 2024-02-21T23:15:07.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.570+0000 7ff0acc00640 1 -- 172.21.15.28:0/4165091850 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a0035040 con 0x7ff0b00a5f40 2024-02-21T23:15:07.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.570+0000 7ff0b7200640 1 -- 172.21.15.28:0/4165091850 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0b01dff90 con 0x7ff0b00a5f40 2024-02-21T23:15:07.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.571+0000 7ff0b7200640 1 -- 172.21.15.28:0/4165091850 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0b01edec0 con 0x7ff0b00a5f40 2024-02-21T23:15:07.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.572+0000 7ff0b7200640 1 -- 172.21.15.28:0/4165091850 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0b0067f30 con 0x7ff0b00a5f40 2024-02-21T23:15:07.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.573+0000 7ff0acc00640 1 -- 172.21.15.28:0/4165091850 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff0a004b060 con 0x7ff0b00a5f40 2024-02-21T23:15:07.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.574+0000 7ff0acc00640 1 --2- 172.21.15.28:0/4165091850 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff08c06b500 0x7ff08c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:07.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.574+0000 7ff0acc00640 1 -- 172.21.15.28:0/4165091850 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7ff0a00abc60 con 0x7ff0b00a5f40 2024-02-21T23:15:07.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.574+0000 7ff0a7e00640 1 --2- 172.21.15.28:0/4165091850 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff08c06b500 0x7ff08c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:07.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.575+0000 7ff0a7e00640 1 --2- 172.21.15.28:0/4165091850 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff08c06b500 0x7ff08c06d9c0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7ff0b014b2b0 tx=0x7ff098007b50 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:07.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.578+0000 7ff0acc00640 1 -- 172.21.15.28:0/4165091850 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ff0a007a840 con 0x7ff0b00a5f40 2024-02-21T23:15:07.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.843+0000 7ff0b7200640 1 -- 172.21.15.28:0/4165091850 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff0b006dd00 con 0x7ff0b00a5f40 2024-02-21T23:15:07.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.844+0000 7ff0acc00640 1 -- 172.21.15.28:0/4165091850 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 246 v246) v1 ==== 95+0+3384 (secure 0 0 0) 0x7ff0a0080070 con 0x7ff0b00a5f40 2024-02-21T23:15:07.832 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:15:07.832 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":246,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":246,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:14:46.655515+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:15:07.832 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 246 2024-02-21T23:15:07.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.847+0000 7ff0b7200640 1 -- 172.21.15.28:0/4165091850 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff08c06b500 msgr2=0x7ff08c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:07.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.847+0000 7ff0b7200640 1 --2- 172.21.15.28:0/4165091850 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff08c06b500 0x7ff08c06d9c0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7ff0b014b2b0 tx=0x7ff098007b50 comp rx=0 tx=0).stop 2024-02-21T23:15:07.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.847+0000 7ff0b7200640 1 -- 172.21.15.28:0/4165091850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0b00a5f40 msgr2=0x7ff0b0149bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:07.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.847+0000 7ff0b7200640 1 --2- 172.21.15.28:0/4165091850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0b00a5f40 0x7ff0b0149bb0 secure :-1 s=READY pgs=2526 cs=0 l=1 rev1=1 crypto rx=0x7ff0a0000c00 tx=0x7ff0a00046c0 comp rx=0 tx=0).stop 2024-02-21T23:15:07.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.848+0000 7ff0b7200640 1 -- 172.21.15.28:0/4165091850 shutdown_connections 2024-02-21T23:15:07.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.848+0000 7ff0b7200640 1 --2- 172.21.15.28:0/4165091850 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff08c06b500 0x7ff08c06d9c0 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:07.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.848+0000 7ff0b7200640 1 --2- 172.21.15.28:0/4165091850 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff0b00c3490 0x7ff0b01ee3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:07.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.848+0000 7ff0b7200640 1 --2- 172.21.15.28:0/4165091850 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff0b00c0270 0x7ff0b00722d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:07.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.848+0000 7ff0b7200640 1 --2- 172.21.15.28:0/4165091850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0b00a5f40 0x7ff0b0149bb0 unknown :-1 s=CLOSED pgs=2526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:07.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.848+0000 7ff0b7200640 1 -- 172.21.15.28:0/4165091850 >> 172.21.15.28:0/4165091850 conn(0x7ff0b00c67f0 msgr2=0x7ff0b014a830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:15:07.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.848+0000 7ff0b7200640 1 -- 172.21.15.28:0/4165091850 shutdown_connections 2024-02-21T23:15:07.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:07.848+0000 7ff0b7200640 1 -- 172.21.15.28:0/4165091850 wait complete. 2024-02-21T23:15:08.262 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 246, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:14:46.655515+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:15:08.262 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:15:08.650 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-21T23:15:08.658 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-21T23:15:09.263 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:15:09.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.512+0000 7f54d1800640 1 Processor -- start 2024-02-21T23:15:09.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.513+0000 7f54d1800640 1 -- start start 2024-02-21T23:15:09.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.514+0000 7f54d1800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f54cc06ee40 0x7f54cc0a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:09.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.514+0000 7f54d1800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54cc0c2760 con 0x7f54cc156af0 2024-02-21T23:15:09.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.514+0000 7f54d1800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54cc156310 con 0x7f54cc150450 2024-02-21T23:15:09.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.514+0000 7f54d1800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54cc0c22c0 con 0x7f54cc06ee40 2024-02-21T23:15:09.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.515+0000 7f54d0e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f54cc156af0 0x7f54cc06dff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:54284/0 (socket says 172.21.15.28:54284) 2024-02-21T23:15:09.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.515+0000 7f54d0e00640 1 -- 172.21.15.28:0/2234361371 learned_addr learned my addr 172.21.15.28:0/2234361371 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:15:09.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.515+0000 7f54c3e00640 1 --2- 172.21.15.28:0/2234361371 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f54cc06ee40 0x7f54cc0a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:09.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.515+0000 7f54cb400640 1 -- 172.21.15.28:0/2234361371 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 423030019 0 0) 0x7f54cc0c2760 con 0x7f54cc156af0 2024-02-21T23:15:09.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.515+0000 7f54cb400640 1 -- 172.21.15.28:0/2234361371 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54b0003520 con 0x7f54cc156af0 2024-02-21T23:15:09.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.516+0000 7f54cb400640 1 -- 172.21.15.28:0/2234361371 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3667939314 0 0) 0x7f54b0003520 con 0x7f54cc156af0 2024-02-21T23:15:09.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.516+0000 7f54cb400640 1 -- 172.21.15.28:0/2234361371 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f54cc06ee40 msgr2=0x7f54cc0a5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:15:09.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.516+0000 7f54cb400640 1 --2- 172.21.15.28:0/2234361371 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f54cc06ee40 0x7f54cc0a5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:09.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.516+0000 7f54cb400640 1 -- 172.21.15.28:0/2234361371 >> v1:172.21.15.81:6789/0 conn(0x7f54cc150450 legacy=0x7f54cc1536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:09.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.516+0000 7f54cb400640 1 -- 172.21.15.28:0/2234361371 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54cc0c29d0 con 0x7f54cc156af0 2024-02-21T23:15:09.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.516+0000 7f54cb400640 1 -- 172.21.15.28:0/2234361371 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f54c40019f0 con 0x7f54cc156af0 2024-02-21T23:15:09.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.516+0000 7f54cb400640 1 -- 172.21.15.28:0/2234361371 >> v1:172.21.15.28:6789/0 conn(0x7f54cc156af0 legacy=0x7f54cc06dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:09.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.516+0000 7f54cb400640 1 --2- 172.21.15.28:0/2234361371 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54b00039e0 0x7f54b0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:09.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.516+0000 7f54cb400640 1 --2- 172.21.15.28:0/2234361371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54b0004800 0x7f54b0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:09.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.516+0000 7f54cb400640 1 --2- 172.21.15.28:0/2234361371 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f54b00051c0 0x7f54b000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:09.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.516+0000 7f54cb400640 1 -- 172.21.15.28:0/2234361371 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54b000bfb0 con 0x7f54b0004800 2024-02-21T23:15:09.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.516+0000 7f54cb400640 1 -- 172.21.15.28:0/2234361371 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54b000c130 con 0x7f54b00039e0 2024-02-21T23:15:09.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.516+0000 7f54cb400640 1 -- 172.21.15.28:0/2234361371 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54b000c300 con 0x7f54b00051c0 2024-02-21T23:15:09.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.516+0000 7f54c3e00640 1 --2- 172.21.15.28:0/2234361371 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f54cc06ee40 0x7f54cc0a5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-21T23:15:09.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.516+0000 7f54c3e00640 1 --2- 172.21.15.28:0/2234361371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54b0004800 0x7f54b0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:09.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.517+0000 7f54d0e00640 1 --2- 172.21.15.28:0/2234361371 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f54b00051c0 0x7f54b000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:09.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.517+0000 7f54cbe00640 1 --2- 172.21.15.28:0/2234361371 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54b00039e0 0x7f54b0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:09.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.517+0000 7f54c3e00640 1 -- 172.21.15.28:0/2234361371 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f54b00051c0 msgr2=0x7f54b000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:15:09.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.517+0000 7f54c3e00640 1 --2- 172.21.15.28:0/2234361371 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f54b00051c0 0x7f54b000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:09.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.517+0000 7f54c3e00640 1 -- 172.21.15.28:0/2234361371 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54b00039e0 msgr2=0x7f54b0003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:15:09.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.517+0000 7f54c3e00640 1 --2- 172.21.15.28:0/2234361371 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54b00039e0 0x7f54b0003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:09.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.517+0000 7f54c3e00640 1 -- 172.21.15.28:0/2234361371 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54b0003520 con 0x7f54b0004800 2024-02-21T23:15:09.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.518+0000 7f54c3e00640 1 --2- 172.21.15.28:0/2234361371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54b0004800 0x7f54b0004c80 secure :-1 s=READY pgs=2528 cs=0 l=1 rev1=1 crypto rx=0x7f54b4009b00 tx=0x7f54b0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2d4b944ec4e4f83 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:09.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.518+0000 7f54cb400640 1 -- 172.21.15.28:0/2234361371 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b40097d0 con 0x7f54b0004800 2024-02-21T23:15:09.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.518+0000 7f54cb400640 1 -- 172.21.15.28:0/2234361371 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54b4004030 con 0x7f54b0004800 2024-02-21T23:15:09.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.518+0000 7f54cb400640 1 -- 172.21.15.28:0/2234361371 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b4004400 con 0x7f54b0004800 2024-02-21T23:15:09.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.519+0000 7f54d1800640 1 -- 172.21.15.28:0/2234361371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54b0004800 msgr2=0x7f54b0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:09.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.519+0000 7f54d1800640 1 --2- 172.21.15.28:0/2234361371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54b0004800 0x7f54b0004c80 secure :-1 s=READY pgs=2528 cs=0 l=1 rev1=1 crypto rx=0x7f54b4009b00 tx=0x7f54b0005830 comp rx=0 tx=0).stop 2024-02-21T23:15:09.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.519+0000 7f54d0e00640 1 -- 172.21.15.28:0/2234361371 reap_dead start 2024-02-21T23:15:09.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.519+0000 7f54d1800640 1 -- 172.21.15.28:0/2234361371 shutdown_connections 2024-02-21T23:15:09.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.519+0000 7f54d1800640 1 -- 172.21.15.28:0/2234361371 >> 172.21.15.28:0/2234361371 conn(0x7f54cc06b630 msgr2=0x7f54cc0c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:15:09.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.519+0000 7f54d1800640 1 -- 172.21.15.28:0/2234361371 shutdown_connections 2024-02-21T23:15:09.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.519+0000 7f54d1800640 1 -- 172.21.15.28:0/2234361371 wait complete. 2024-02-21T23:15:09.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.521+0000 7f54d1800640 1 Processor -- start 2024-02-21T23:15:09.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.521+0000 7f54d1800640 1 -- start start 2024-02-21T23:15:09.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.522+0000 7f54d1800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f54b0004800 0x7f54cc14bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:09.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.522+0000 7f54d1800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54cc155e50 0x7f54cc1ea6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:09.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.522+0000 7f54d1800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54cc147a30 0x7f54cc1eac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:09.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.522+0000 7f54d1800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54cc06e700 con 0x7f54cc147a30 2024-02-21T23:15:09.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.522+0000 7f54d1800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54cc156660 con 0x7f54cc155e50 2024-02-21T23:15:09.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.522+0000 7f54d1800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54cc06e880 con 0x7f54b0004800 2024-02-21T23:15:09.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.522+0000 7f54d0e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54cc147a30 0x7f54cc1eac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:09.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.522+0000 7f54d0e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54cc147a30 0x7f54cc1eac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53794/0 (socket says 172.21.15.28:53794) 2024-02-21T23:15:09.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.522+0000 7f54d0e00640 1 -- 172.21.15.28:0/2412441316 learned_addr learned my addr 172.21.15.28:0/2412441316 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:15:09.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.522+0000 7f54cbe00640 1 --2- 172.21.15.28:0/2412441316 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f54b0004800 0x7f54cc14bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:09.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.522+0000 7f54c3e00640 1 --2- 172.21.15.28:0/2412441316 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54cc155e50 0x7f54cc1ea6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:09.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.523+0000 7f54d0e00640 1 -- 172.21.15.28:0/2412441316 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f54b0004800 msgr2=0x7f54cc14bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:09.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.523+0000 7f54d0e00640 1 --2- 172.21.15.28:0/2412441316 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f54b0004800 0x7f54cc14bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:09.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.523+0000 7f54d0e00640 1 -- 172.21.15.28:0/2412441316 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54cc155e50 msgr2=0x7f54cc1ea6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:09.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.523+0000 7f54d0e00640 1 --2- 172.21.15.28:0/2412441316 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54cc155e50 0x7f54cc1ea6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:09.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.523+0000 7f54d0e00640 1 -- 172.21.15.28:0/2412441316 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54cc147750 con 0x7f54cc147a30 2024-02-21T23:15:09.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.523+0000 7f54d0e00640 1 --2- 172.21.15.28:0/2412441316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54cc147a30 0x7f54cc1eac10 secure :-1 s=READY pgs=2529 cs=0 l=1 rev1=1 crypto rx=0x7f54c4007960 tx=0x7f54c400e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:09.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.523+0000 7f54c8c00640 1 -- 172.21.15.28:0/2412441316 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54c400ec40 con 0x7f54cc147a30 2024-02-21T23:15:09.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.523+0000 7f54c8c00640 1 -- 172.21.15.28:0/2412441316 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54c400ede0 con 0x7f54cc147a30 2024-02-21T23:15:09.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.523+0000 7f54c8c00640 1 -- 172.21.15.28:0/2412441316 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54c4014670 con 0x7f54cc147a30 2024-02-21T23:15:09.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.523+0000 7f54d1800640 1 -- 172.21.15.28:0/2412441316 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54cc1edc40 con 0x7f54cc147a30 2024-02-21T23:15:09.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.523+0000 7f54d1800640 1 -- 172.21.15.28:0/2412441316 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54cc1ee200 con 0x7f54cc147a30 2024-02-21T23:15:09.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.525+0000 7f54d1800640 1 -- 172.21.15.28:0/2412441316 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54cc04f290 con 0x7f54cc147a30 2024-02-21T23:15:09.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.525+0000 7f54c8c00640 1 -- 172.21.15.28:0/2412441316 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f54c4014a20 con 0x7f54cc147a30 2024-02-21T23:15:09.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.526+0000 7f54c8c00640 1 --2- 172.21.15.28:0/2412441316 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f54a806b550 0x7f54a806da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:09.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.526+0000 7f54c8c00640 1 -- 172.21.15.28:0/2412441316 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f54c408a450 con 0x7f54cc147a30 2024-02-21T23:15:09.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.527+0000 7f54cbe00640 1 --2- 172.21.15.28:0/2412441316 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f54a806b550 0x7f54a806da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:09.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.528+0000 7f54cbe00640 1 --2- 172.21.15.28:0/2412441316 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f54a806b550 0x7f54a806da10 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f54bc000f60 tx=0x7f54bc000ef0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:09.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.531+0000 7f54c8c00640 1 -- 172.21.15.28:0/2412441316 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f54c40590b0 con 0x7f54cc147a30 2024-02-21T23:15:09.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.812+0000 7f54d1800640 1 -- 172.21.15.28:0/2412441316 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f54cc156310 con 0x7f54cc147a30 2024-02-21T23:15:09.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.813+0000 7f54c8c00640 1 -- 172.21.15.28:0/2412441316 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 246 v246) v1 ==== 95+0+3384 (secure 0 0 0) 0x7f54c40589b0 con 0x7f54cc147a30 2024-02-21T23:15:09.800 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:15:09.801 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":246,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":246,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:14:46.655515+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:15:09.801 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 246 2024-02-21T23:15:09.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.816+0000 7f54d1800640 1 -- 172.21.15.28:0/2412441316 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f54a806b550 msgr2=0x7f54a806da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:09.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.816+0000 7f54d1800640 1 --2- 172.21.15.28:0/2412441316 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f54a806b550 0x7f54a806da10 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f54bc000f60 tx=0x7f54bc000ef0 comp rx=0 tx=0).stop 2024-02-21T23:15:09.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.816+0000 7f54d1800640 1 -- 172.21.15.28:0/2412441316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54cc147a30 msgr2=0x7f54cc1eac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:09.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.816+0000 7f54d1800640 1 --2- 172.21.15.28:0/2412441316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54cc147a30 0x7f54cc1eac10 secure :-1 s=READY pgs=2529 cs=0 l=1 rev1=1 crypto rx=0x7f54c4007960 tx=0x7f54c400e530 comp rx=0 tx=0).stop 2024-02-21T23:15:09.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.816+0000 7f54d1800640 1 -- 172.21.15.28:0/2412441316 shutdown_connections 2024-02-21T23:15:09.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.816+0000 7f54d1800640 1 --2- 172.21.15.28:0/2412441316 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f54a806b550 0x7f54a806da10 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:09.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.816+0000 7f54d1800640 1 --2- 172.21.15.28:0/2412441316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54cc147a30 0x7f54cc1eac10 unknown :-1 s=CLOSED pgs=2529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:09.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.816+0000 7f54d1800640 1 --2- 172.21.15.28:0/2412441316 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f54cc155e50 0x7f54cc1ea6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:09.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.816+0000 7f54d1800640 1 --2- 172.21.15.28:0/2412441316 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f54b0004800 0x7f54cc14bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:09.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.816+0000 7f54d1800640 1 -- 172.21.15.28:0/2412441316 >> 172.21.15.28:0/2412441316 conn(0x7f54cc06b630 msgr2=0x7f54cc14db80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:15:09.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.816+0000 7f54d1800640 1 -- 172.21.15.28:0/2412441316 shutdown_connections 2024-02-21T23:15:09.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:09.816+0000 7f54d1800640 1 -- 172.21.15.28:0/2412441316 wait complete. 2024-02-21T23:15:10.227 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 246, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:14:46.655515+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:15:10.227 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:15:11.228 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:15:11.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.467+0000 7f73c7e00640 1 Processor -- start 2024-02-21T23:15:11.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.467+0000 7f73c7e00640 1 -- start start 2024-02-21T23:15:11.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.469+0000 7f73c7e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f73c8150590 0x7f73c806be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:11.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.469+0000 7f73c7e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f73c806c3c0 0x7f73c8156a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:11.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.469+0000 7f73c7e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73c80a8a10 con 0x7f73c80a84b0 2024-02-21T23:15:11.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.469+0000 7f73c7e00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73c8072700 con 0x7f73c8150590 2024-02-21T23:15:11.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.469+0000 7f73c7e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73c80729e0 con 0x7f73c806c3c0 2024-02-21T23:15:11.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.469+0000 7f73c6000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f73c80a84b0 0x7f73c8156320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:54296/0 (socket says 172.21.15.28:54296) 2024-02-21T23:15:11.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.469+0000 7f73c6000640 1 -- 172.21.15.28:0/2999849536 learned_addr learned my addr 172.21.15.28:0/2999849536 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:15:11.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.470+0000 7f73c7400640 1 --2- 172.21.15.28:0/2999849536 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f73c806c3c0 0x7f73c8156a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:11.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.470+0000 7f73c6a00640 1 --2- 172.21.15.28:0/2999849536 >> v2:172.21.15.81:3300/0 conn(0x7f73c8150590 0x7f73c806be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:11.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.470+0000 7f73c5600640 1 -- 172.21.15.28:0/2999849536 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3780352053 0 0) 0x7f73c80a8a10 con 0x7f73c80a84b0 2024-02-21T23:15:11.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.470+0000 7f73c5600640 1 -- 172.21.15.28:0/2999849536 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73ac003520 con 0x7f73c80a84b0 2024-02-21T23:15:11.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.470+0000 7f73c5600640 1 -- 172.21.15.28:0/2999849536 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2448598405 0 0) 0x7f73ac003520 con 0x7f73c80a84b0 2024-02-21T23:15:11.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.470+0000 7f73c5600640 1 -- 172.21.15.28:0/2999849536 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f73c806c3c0 msgr2=0x7f73c8156a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:15:11.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.470+0000 7f73c5600640 1 --2- 172.21.15.28:0/2999849536 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f73c806c3c0 0x7f73c8156a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:11.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.470+0000 7f73c5600640 1 -- 172.21.15.28:0/2999849536 >> v2:172.21.15.81:3300/0 conn(0x7f73c8150590 msgr2=0x7f73c806be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:15:11.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.470+0000 7f73c5600640 1 --2- 172.21.15.28:0/2999849536 >> v2:172.21.15.81:3300/0 conn(0x7f73c8150590 0x7f73c806be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:11.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.470+0000 7f73c5600640 1 -- 172.21.15.28:0/2999849536 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73c80a8c80 con 0x7f73c80a84b0 2024-02-21T23:15:11.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.470+0000 7f73c5600640 1 -- 172.21.15.28:0/2999849536 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f73b4001670 con 0x7f73c80a84b0 2024-02-21T23:15:11.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.470+0000 7f73c5600640 1 -- 172.21.15.28:0/2999849536 >> v1:172.21.15.28:6789/0 conn(0x7f73c80a84b0 legacy=0x7f73c8156320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:11.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.470+0000 7f73c5600640 1 --2- 172.21.15.28:0/2999849536 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73ac0039b0 0x7f73ac003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:11.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.470+0000 7f73c5600640 1 --2- 172.21.15.28:0/2999849536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f73ac004800 0x7f73ac004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:11.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.470+0000 7f73c5600640 1 --2- 172.21.15.28:0/2999849536 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f73c806c3c0 0x7f73ac0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:11.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.470+0000 7f73c5600640 1 -- 172.21.15.28:0/2999849536 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73c8072700 con 0x7f73ac004800 2024-02-21T23:15:11.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.470+0000 7f73c5600640 1 -- 172.21.15.28:0/2999849536 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73c80729e0 con 0x7f73ac0039b0 2024-02-21T23:15:11.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.470+0000 7f73c5600640 1 -- 172.21.15.28:0/2999849536 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73ac00ba10 con 0x7f73c806c3c0 2024-02-21T23:15:11.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.471+0000 7f73c6000640 1 --2- 172.21.15.28:0/2999849536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f73ac004800 0x7f73ac004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:11.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.471+0000 7f73c6a00640 1 --2- 172.21.15.28:0/2999849536 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73ac0039b0 0x7f73ac003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:11.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.471+0000 7f73c7400640 1 --2- 172.21.15.28:0/2999849536 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f73c806c3c0 0x7f73ac0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:11.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.471+0000 7f73c6000640 1 -- 172.21.15.28:0/2999849536 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f73c806c3c0 msgr2=0x7f73ac0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:15:11.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.471+0000 7f73c6000640 1 --2- 172.21.15.28:0/2999849536 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f73c806c3c0 0x7f73ac0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:11.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.471+0000 7f73c6000640 1 -- 172.21.15.28:0/2999849536 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73ac0039b0 msgr2=0x7f73ac003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:15:11.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.471+0000 7f73c6000640 1 --2- 172.21.15.28:0/2999849536 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73ac0039b0 0x7f73ac003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:11.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.471+0000 7f73c6000640 1 -- 172.21.15.28:0/2999849536 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73ac003520 con 0x7f73ac004800 2024-02-21T23:15:11.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.473+0000 7f73c6000640 1 --2- 172.21.15.28:0/2999849536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f73ac004800 0x7f73ac004c80 secure :-1 s=READY pgs=2531 cs=0 l=1 rev1=1 crypto rx=0x7f73b400c190 tx=0x7f73ac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7adeeb8d96fb8754 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:11.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.473+0000 7f73c5600640 1 -- 172.21.15.28:0/2999849536 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73b4007c80 con 0x7f73ac004800 2024-02-21T23:15:11.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.473+0000 7f73c5600640 1 -- 172.21.15.28:0/2999849536 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73b4030070 con 0x7f73ac004800 2024-02-21T23:15:11.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.473+0000 7f73c5600640 1 -- 172.21.15.28:0/2999849536 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73b4005d30 con 0x7f73ac004800 2024-02-21T23:15:11.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.474+0000 7f73c7e00640 1 -- 172.21.15.28:0/2999849536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f73ac004800 msgr2=0x7f73ac004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:11.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.474+0000 7f73c7e00640 1 --2- 172.21.15.28:0/2999849536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f73ac004800 0x7f73ac004c80 secure :-1 s=READY pgs=2531 cs=0 l=1 rev1=1 crypto rx=0x7f73b400c190 tx=0x7f73ac005830 comp rx=0 tx=0).stop 2024-02-21T23:15:11.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.474+0000 7f73c7400640 1 -- 172.21.15.28:0/2999849536 reap_dead start 2024-02-21T23:15:11.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.474+0000 7f73c7e00640 1 -- 172.21.15.28:0/2999849536 shutdown_connections 2024-02-21T23:15:11.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.474+0000 7f73c7e00640 1 -- 172.21.15.28:0/2999849536 >> 172.21.15.28:0/2999849536 conn(0x7f73c8071910 msgr2=0x7f73c8071cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:15:11.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.474+0000 7f73c7e00640 1 -- 172.21.15.28:0/2999849536 shutdown_connections 2024-02-21T23:15:11.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.474+0000 7f73c7e00640 1 -- 172.21.15.28:0/2999849536 wait complete. 2024-02-21T23:15:11.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.476+0000 7f73c7e00640 1 Processor -- start 2024-02-21T23:15:11.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.476+0000 7f73c7e00640 1 -- start start 2024-02-21T23:15:11.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.477+0000 7f73c7e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73ac004800 0x7f73c81527b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:11.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.477+0000 7f73c7e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f73c81520f0 0x7f73c81df890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:11.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.477+0000 7f73c7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f73c80c1040 0x7f73c8142750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:11.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.477+0000 7f73c7e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73c80c2d60 con 0x7f73c80c1040 2024-02-21T23:15:11.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.477+0000 7f73c7e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73c80c2be0 con 0x7f73ac004800 2024-02-21T23:15:11.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.477+0000 7f73c7e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73c80c3340 con 0x7f73c81520f0 2024-02-21T23:15:11.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.477+0000 7f73c7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f73c80c1040 0x7f73c8142750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:11.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.478+0000 7f73c7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f73c80c1040 0x7f73c8142750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53808/0 (socket says 172.21.15.28:53808) 2024-02-21T23:15:11.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.478+0000 7f73c7400640 1 -- 172.21.15.28:0/4024904878 learned_addr learned my addr 172.21.15.28:0/4024904878 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:15:11.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.478+0000 7f73c6a00640 1 --2- 172.21.15.28:0/4024904878 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73ac004800 0x7f73c81527b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:11.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.478+0000 7f73c6000640 1 --2- 172.21.15.28:0/4024904878 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f73c81520f0 0x7f73c81df890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:11.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.478+0000 7f73c7400640 1 -- 172.21.15.28:0/4024904878 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f73c81520f0 msgr2=0x7f73c81df890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:11.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.478+0000 7f73c7400640 1 --2- 172.21.15.28:0/4024904878 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f73c81520f0 0x7f73c81df890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:11.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.478+0000 7f73c7400640 1 -- 172.21.15.28:0/4024904878 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73ac004800 msgr2=0x7f73c81527b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:11.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.478+0000 7f73c7400640 1 --2- 172.21.15.28:0/4024904878 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73ac004800 0x7f73c81527b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:11.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.478+0000 7f73c7400640 1 -- 172.21.15.28:0/4024904878 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73c81eb250 con 0x7f73c80c1040 2024-02-21T23:15:11.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.478+0000 7f73c6000640 1 --2- 172.21.15.28:0/4024904878 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f73c81520f0 0x7f73c81df890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:15:11.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.478+0000 7f73c6a00640 1 --2- 172.21.15.28:0/4024904878 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73ac004800 0x7f73c81527b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:15:11.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.478+0000 7f73c7400640 1 --2- 172.21.15.28:0/4024904878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f73c80c1040 0x7f73c8142750 secure :-1 s=READY pgs=2532 cs=0 l=1 rev1=1 crypto rx=0x7f73c806c8c0 tx=0x7f73c00115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:11.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.479+0000 7f73bea00640 1 -- 172.21.15.28:0/4024904878 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73c0011e10 con 0x7f73c80c1040 2024-02-21T23:15:11.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.479+0000 7f73bea00640 1 -- 172.21.15.28:0/4024904878 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73c001b070 con 0x7f73c80c1040 2024-02-21T23:15:11.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.479+0000 7f73bea00640 1 -- 172.21.15.28:0/4024904878 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73c0003990 con 0x7f73c80c1040 2024-02-21T23:15:11.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.479+0000 7f73c7e00640 1 -- 172.21.15.28:0/4024904878 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73c81eb530 con 0x7f73c80c1040 2024-02-21T23:15:11.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.479+0000 7f73c7e00640 1 -- 172.21.15.28:0/4024904878 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73c81dfdd0 con 0x7f73c80c1040 2024-02-21T23:15:11.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.480+0000 7f73c7e00640 1 -- 172.21.15.28:0/4024904878 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73c804f290 con 0x7f73c80c1040 2024-02-21T23:15:11.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.482+0000 7f73bea00640 1 -- 172.21.15.28:0/4024904878 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f73c0005d10 con 0x7f73c80c1040 2024-02-21T23:15:11.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.483+0000 7f73bea00640 1 --2- 172.21.15.28:0/4024904878 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f73a806b4d0 0x7f73a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:11.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.483+0000 7f73bea00640 1 -- 172.21.15.28:0/4024904878 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f73c008a820 con 0x7f73c80c1040 2024-02-21T23:15:11.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.484+0000 7f73c6a00640 1 --2- 172.21.15.28:0/4024904878 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f73a806b4d0 0x7f73a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:11.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.485+0000 7f73c6a00640 1 --2- 172.21.15.28:0/4024904878 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f73a806b4d0 0x7f73a806d990 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f73c81e8570 tx=0x7f73b8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:11.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.493+0000 7f73bea00640 1 -- 172.21.15.28:0/4024904878 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f73c0014350 con 0x7f73c80c1040 2024-02-21T23:15:11.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.740+0000 7f73c7e00640 1 -- 172.21.15.28:0/4024904878 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f73c80c28e0 con 0x7f73c80c1040 2024-02-21T23:15:11.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.741+0000 7f73bea00640 1 -- 172.21.15.28:0/4024904878 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 246 v246) v1 ==== 95+0+3384 (secure 0 0 0) 0x7f73c0059400 con 0x7f73c80c1040 2024-02-21T23:15:11.728 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:15:11.728 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":246,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":246,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:14:46.655515+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:15:11.729 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 246 2024-02-21T23:15:11.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.744+0000 7f73c7e00640 1 -- 172.21.15.28:0/4024904878 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f73a806b4d0 msgr2=0x7f73a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:11.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.744+0000 7f73c7e00640 1 --2- 172.21.15.28:0/4024904878 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f73a806b4d0 0x7f73a806d990 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f73c81e8570 tx=0x7f73b8008040 comp rx=0 tx=0).stop 2024-02-21T23:15:11.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.744+0000 7f73c7e00640 1 -- 172.21.15.28:0/4024904878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f73c80c1040 msgr2=0x7f73c8142750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:11.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.744+0000 7f73c7e00640 1 --2- 172.21.15.28:0/4024904878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f73c80c1040 0x7f73c8142750 secure :-1 s=READY pgs=2532 cs=0 l=1 rev1=1 crypto rx=0x7f73c806c8c0 tx=0x7f73c00115a0 comp rx=0 tx=0).stop 2024-02-21T23:15:11.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.744+0000 7f73c7e00640 1 -- 172.21.15.28:0/4024904878 shutdown_connections 2024-02-21T23:15:11.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.744+0000 7f73c7e00640 1 --2- 172.21.15.28:0/4024904878 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f73a806b4d0 0x7f73a806d990 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:11.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.744+0000 7f73c7e00640 1 --2- 172.21.15.28:0/4024904878 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f73c81520f0 0x7f73c81df890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:11.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.744+0000 7f73c7e00640 1 --2- 172.21.15.28:0/4024904878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f73c80c1040 0x7f73c8142750 unknown :-1 s=CLOSED pgs=2532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:11.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.744+0000 7f73c7e00640 1 --2- 172.21.15.28:0/4024904878 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73ac004800 0x7f73c81527b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:11.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.744+0000 7f73c7e00640 1 -- 172.21.15.28:0/4024904878 >> 172.21.15.28:0/4024904878 conn(0x7f73c8071910 msgr2=0x7f73c80abaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:15:11.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.744+0000 7f73c7e00640 1 -- 172.21.15.28:0/4024904878 shutdown_connections 2024-02-21T23:15:11.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:11.744+0000 7f73c7e00640 1 -- 172.21.15.28:0/4024904878 wait complete. 2024-02-21T23:15:12.154 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 246, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:14:46.655515+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:15:12.155 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:15:13.156 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:15:13.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.399+0000 7fa786e00640 1 Processor -- start 2024-02-21T23:15:13.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.399+0000 7fa786e00640 1 -- start start 2024-02-21T23:15:13.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.400+0000 7fa786e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fa7800aff90 0x7fa7800c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:13.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.401+0000 7fa786e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7800c74c0 con 0x7fa78006dbf0 2024-02-21T23:15:13.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.401+0000 7fa786e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7800c77a0 con 0x7fa7800c5720 2024-02-21T23:15:13.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.401+0000 7fa786e00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7800c7270 con 0x7fa7800aff90 2024-02-21T23:15:13.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.401+0000 7fa785000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa78006dbf0 0x7fa780158ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:54312/0 (socket says 172.21.15.28:54312) 2024-02-21T23:15:13.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.401+0000 7fa785000640 1 -- 172.21.15.28:0/127167901 learned_addr learned my addr 172.21.15.28:0/127167901 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:15:13.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.401+0000 7fa785a00640 1 --2- 172.21.15.28:0/127167901 >> v2:172.21.15.81:3300/0 conn(0x7fa7800aff90 0x7fa7800c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:13.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.402+0000 7fa77fe00640 1 -- 172.21.15.28:0/127167901 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 596436175 0 0) 0x7fa7800c74c0 con 0x7fa78006dbf0 2024-02-21T23:15:13.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.402+0000 7fa77fe00640 1 -- 172.21.15.28:0/127167901 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa768003520 con 0x7fa78006dbf0 2024-02-21T23:15:13.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.402+0000 7fa77fe00640 1 -- 172.21.15.28:0/127167901 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4288914846 0 0) 0x7fa768003520 con 0x7fa78006dbf0 2024-02-21T23:15:13.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.402+0000 7fa77fe00640 1 -- 172.21.15.28:0/127167901 >> v2:172.21.15.81:3300/0 conn(0x7fa7800aff90 msgr2=0x7fa7800c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:15:13.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.402+0000 7fa77fe00640 1 --2- 172.21.15.28:0/127167901 >> v2:172.21.15.81:3300/0 conn(0x7fa7800aff90 0x7fa7800c51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:13.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.402+0000 7fa77fe00640 1 -- 172.21.15.28:0/127167901 >> v1:172.21.15.81:6789/0 conn(0x7fa7800c5720 legacy=0x7fa78015f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:13.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.402+0000 7fa77fe00640 1 -- 172.21.15.28:0/127167901 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7800c7a80 con 0x7fa78006dbf0 2024-02-21T23:15:13.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.402+0000 7fa77fe00640 1 -- 172.21.15.28:0/127167901 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fa76c001670 con 0x7fa78006dbf0 2024-02-21T23:15:13.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.402+0000 7fa77fe00640 1 -- 172.21.15.28:0/127167901 >> v1:172.21.15.28:6789/0 conn(0x7fa78006dbf0 legacy=0x7fa780158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:13.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.402+0000 7fa77fe00640 1 --2- 172.21.15.28:0/127167901 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa7680039b0 0x7fa768003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:13.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.402+0000 7fa77fe00640 1 --2- 172.21.15.28:0/127167901 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa768004800 0x7fa768004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:13.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.402+0000 7fa77fe00640 1 --2- 172.21.15.28:0/127167901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7680051c0 0x7fa76800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:13.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.402+0000 7fa77fe00640 1 -- 172.21.15.28:0/127167901 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7800c7270 con 0x7fa7680051c0 2024-02-21T23:15:13.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.402+0000 7fa77fe00640 1 -- 172.21.15.28:0/127167901 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa76800c010 con 0x7fa768004800 2024-02-21T23:15:13.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.402+0000 7fa77fe00640 1 -- 172.21.15.28:0/127167901 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa76800c210 con 0x7fa7680039b0 2024-02-21T23:15:13.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.403+0000 7fa786400640 1 --2- 172.21.15.28:0/127167901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7680051c0 0x7fa76800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:13.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.403+0000 7fa785a00640 1 --2- 172.21.15.28:0/127167901 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa7680039b0 0x7fa768003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:13.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.403+0000 7fa785000640 1 --2- 172.21.15.28:0/127167901 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa768004800 0x7fa768004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:13.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.403+0000 7fa786400640 1 -- 172.21.15.28:0/127167901 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa7680039b0 msgr2=0x7fa768003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:15:13.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.403+0000 7fa786400640 1 --2- 172.21.15.28:0/127167901 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa7680039b0 0x7fa768003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:13.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.403+0000 7fa786400640 1 -- 172.21.15.28:0/127167901 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa768004800 msgr2=0x7fa768004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:15:13.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.403+0000 7fa786400640 1 --2- 172.21.15.28:0/127167901 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa768004800 0x7fa768004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:13.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.403+0000 7fa786400640 1 -- 172.21.15.28:0/127167901 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7800c7a80 con 0x7fa7680051c0 2024-02-21T23:15:13.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.403+0000 7fa785a00640 1 --2- 172.21.15.28:0/127167901 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa7680039b0 0x7fa768003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:15:13.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.403+0000 7fa785000640 1 --2- 172.21.15.28:0/127167901 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa768004800 0x7fa768004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:15:13.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.405+0000 7fa786400640 1 -- 172.21.15.28:0/127167901 reap_dead start 2024-02-21T23:15:13.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.405+0000 7fa786400640 1 --2- 172.21.15.28:0/127167901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7680051c0 0x7fa76800b9c0 secure :-1 s=READY pgs=2534 cs=0 l=1 rev1=1 crypto rx=0x7fa778001e50 tx=0x7fa768005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc311ac586e8a269 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:13.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.405+0000 7fa77fe00640 1 -- 172.21.15.28:0/127167901 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa778001a70 con 0x7fa7680051c0 2024-02-21T23:15:13.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.405+0000 7fa77fe00640 1 -- 172.21.15.28:0/127167901 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa778003b90 con 0x7fa7680051c0 2024-02-21T23:15:13.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.405+0000 7fa77fe00640 1 -- 172.21.15.28:0/127167901 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7780050d0 con 0x7fa7680051c0 2024-02-21T23:15:13.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.406+0000 7fa786e00640 1 -- 172.21.15.28:0/127167901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7680051c0 msgr2=0x7fa76800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:13.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.406+0000 7fa786e00640 1 --2- 172.21.15.28:0/127167901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7680051c0 0x7fa76800b9c0 secure :-1 s=READY pgs=2534 cs=0 l=1 rev1=1 crypto rx=0x7fa778001e50 tx=0x7fa768005830 comp rx=0 tx=0).stop 2024-02-21T23:15:13.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.406+0000 7fa786e00640 1 -- 172.21.15.28:0/127167901 shutdown_connections 2024-02-21T23:15:13.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.406+0000 7fa786e00640 1 --2- 172.21.15.28:0/127167901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7680051c0 0x7fa76800b9c0 unknown :-1 s=CLOSED pgs=2534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:13.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.406+0000 7fa786e00640 1 -- 172.21.15.28:0/127167901 >> 172.21.15.28:0/127167901 conn(0x7fa7800ac370 msgr2=0x7fa7800ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:15:13.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.406+0000 7fa786e00640 1 -- 172.21.15.28:0/127167901 shutdown_connections 2024-02-21T23:15:13.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.406+0000 7fa786e00640 1 -- 172.21.15.28:0/127167901 wait complete. 2024-02-21T23:15:13.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.408+0000 7fa786e00640 1 Processor -- start 2024-02-21T23:15:13.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.408+0000 7fa786e00640 1 -- start start 2024-02-21T23:15:13.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.409+0000 7fa786e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7680051c0 0x7fa7801edf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:13.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.409+0000 7fa786e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa7801ee480 0x7fa7801519e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:13.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.409+0000 7fa786e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa780151f20 0x7fa7800c21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:13.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.409+0000 7fa786e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7800c8760 con 0x7fa7680051c0 2024-02-21T23:15:13.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.409+0000 7fa786e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa780158630 con 0x7fa780151f20 2024-02-21T23:15:13.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.409+0000 7fa786e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7800c88e0 con 0x7fa7801ee480 2024-02-21T23:15:13.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.409+0000 7fa785a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7680051c0 0x7fa7801edf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:13.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.409+0000 7fa785a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7680051c0 0x7fa7801edf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53834/0 (socket says 172.21.15.28:53834) 2024-02-21T23:15:13.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.409+0000 7fa785a00640 1 -- 172.21.15.28:0/2211788928 learned_addr learned my addr 172.21.15.28:0/2211788928 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:15:13.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.409+0000 7fa785000640 1 --2- 172.21.15.28:0/2211788928 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa7801ee480 0x7fa7801519e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:13.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.410+0000 7fa786400640 1 --2- 172.21.15.28:0/2211788928 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa780151f20 0x7fa7800c21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:13.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.410+0000 7fa785a00640 1 -- 172.21.15.28:0/2211788928 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa7801ee480 msgr2=0x7fa7801519e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:13.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.410+0000 7fa785a00640 1 --2- 172.21.15.28:0/2211788928 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa7801ee480 0x7fa7801519e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:13.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.410+0000 7fa785a00640 1 -- 172.21.15.28:0/2211788928 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa780151f20 msgr2=0x7fa7800c21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:13.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.410+0000 7fa785a00640 1 --2- 172.21.15.28:0/2211788928 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa780151f20 0x7fa7800c21d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:13.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.410+0000 7fa785a00640 1 -- 172.21.15.28:0/2211788928 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7800c02f0 con 0x7fa7680051c0 2024-02-21T23:15:13.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.410+0000 7fa785000640 1 --2- 172.21.15.28:0/2211788928 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa7801ee480 0x7fa7801519e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:15:13.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.410+0000 7fa785a00640 1 --2- 172.21.15.28:0/2211788928 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7680051c0 0x7fa7801edf40 secure :-1 s=READY pgs=2535 cs=0 l=1 rev1=1 crypto rx=0x7fa774011570 tx=0x7fa774011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:13.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.410+0000 7fa77d600640 1 -- 172.21.15.28:0/2211788928 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa774015870 con 0x7fa7680051c0 2024-02-21T23:15:13.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.410+0000 7fa77d600640 1 -- 172.21.15.28:0/2211788928 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa774015a10 con 0x7fa7680051c0 2024-02-21T23:15:13.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.410+0000 7fa77d600640 1 -- 172.21.15.28:0/2211788928 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa774015dc0 con 0x7fa7680051c0 2024-02-21T23:15:13.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.410+0000 7fa786e00640 1 -- 172.21.15.28:0/2211788928 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7800c05d0 con 0x7fa7680051c0 2024-02-21T23:15:13.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.411+0000 7fa786e00640 1 -- 172.21.15.28:0/2211788928 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7800c19d0 con 0x7fa7680051c0 2024-02-21T23:15:13.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.412+0000 7fa786e00640 1 -- 172.21.15.28:0/2211788928 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa78004f290 con 0x7fa7680051c0 2024-02-21T23:15:13.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.413+0000 7fa77d600640 1 -- 172.21.15.28:0/2211788928 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa77400f460 con 0x7fa7680051c0 2024-02-21T23:15:13.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.414+0000 7fa77d600640 1 --2- 172.21.15.28:0/2211788928 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa76006b4d0 0x7fa76006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:13.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.414+0000 7fa77d600640 1 -- 172.21.15.28:0/2211788928 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fa77408ae00 con 0x7fa7680051c0 2024-02-21T23:15:13.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.415+0000 7fa785000640 1 --2- 172.21.15.28:0/2211788928 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa76006b4d0 0x7fa76006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:13.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.416+0000 7fa785000640 1 --2- 172.21.15.28:0/2211788928 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa76006b4d0 0x7fa76006d990 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fa76c0011e0 tx=0x7fa76c001cf0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:13.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.419+0000 7fa77d600640 1 -- 172.21.15.28:0/2211788928 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fa774019030 con 0x7fa7680051c0 2024-02-21T23:15:13.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.740+0000 7fa786e00640 1 -- 172.21.15.28:0/2211788928 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa7800c1490 con 0x7fa7680051c0 2024-02-21T23:15:13.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.740+0000 7fa77d600640 1 -- 172.21.15.28:0/2211788928 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 246 v246) v1 ==== 95+0+3384 (secure 0 0 0) 0x7fa774059950 con 0x7fa7680051c0 2024-02-21T23:15:13.728 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:15:13.728 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":246,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":246,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:14:46.655515+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:15:13.728 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 246 2024-02-21T23:15:13.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.744+0000 7fa786e00640 1 -- 172.21.15.28:0/2211788928 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa76006b4d0 msgr2=0x7fa76006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:13.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.744+0000 7fa786e00640 1 --2- 172.21.15.28:0/2211788928 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa76006b4d0 0x7fa76006d990 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fa76c0011e0 tx=0x7fa76c001cf0 comp rx=0 tx=0).stop 2024-02-21T23:15:13.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.744+0000 7fa786e00640 1 -- 172.21.15.28:0/2211788928 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7680051c0 msgr2=0x7fa7801edf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:13.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.744+0000 7fa786e00640 1 --2- 172.21.15.28:0/2211788928 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7680051c0 0x7fa7801edf40 secure :-1 s=READY pgs=2535 cs=0 l=1 rev1=1 crypto rx=0x7fa774011570 tx=0x7fa774011a30 comp rx=0 tx=0).stop 2024-02-21T23:15:13.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.744+0000 7fa786e00640 1 -- 172.21.15.28:0/2211788928 shutdown_connections 2024-02-21T23:15:13.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.744+0000 7fa786e00640 1 --2- 172.21.15.28:0/2211788928 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa76006b4d0 0x7fa76006d990 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:13.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.744+0000 7fa786e00640 1 --2- 172.21.15.28:0/2211788928 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa7801ee480 0x7fa7801519e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:13.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.744+0000 7fa786e00640 1 --2- 172.21.15.28:0/2211788928 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa780151f20 0x7fa7800c21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:13.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.744+0000 7fa786e00640 1 --2- 172.21.15.28:0/2211788928 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7680051c0 0x7fa7801edf40 unknown :-1 s=CLOSED pgs=2535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:13.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.744+0000 7fa786e00640 1 -- 172.21.15.28:0/2211788928 >> 172.21.15.28:0/2211788928 conn(0x7fa7800ac370 msgr2=0x7fa7800ac750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:15:13.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.744+0000 7fa786e00640 1 -- 172.21.15.28:0/2211788928 shutdown_connections 2024-02-21T23:15:13.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:13.744+0000 7fa786e00640 1 -- 172.21.15.28:0/2211788928 wait complete. 2024-02-21T23:15:14.156 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 246, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:14:46.655515+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:15:14.156 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:15:15.157 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:15:15.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.398+0000 7f3360c00640 1 Processor -- start 2024-02-21T23:15:15.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.399+0000 7f3360c00640 1 -- start start 2024-02-21T23:15:15.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.400+0000 7f3360c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f335c0aff90 0x7f335c0c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:15.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.400+0000 7f3360c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f335c06dbf0 0x7f335c158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:15.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.400+0000 7f3360c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f335c0c5720 0x7f335c159000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:15.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.400+0000 7f3360c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f335c0c7270 con 0x7f335c06dbf0 2024-02-21T23:15:15.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.400+0000 7f3360c00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f335c158630 con 0x7f335c0c5720 2024-02-21T23:15:15.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.400+0000 7f3360c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f335c0c74c0 con 0x7f335c0aff90 2024-02-21T23:15:15.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.401+0000 7f335aa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f335c06dbf0 0x7f335c158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:15.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.401+0000 7f335aa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f335c06dbf0 0x7f335c158ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39918/0 (socket says 172.21.15.28:39918) 2024-02-21T23:15:15.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.401+0000 7f335aa00640 1 -- 172.21.15.28:0/396770225 learned_addr learned my addr 172.21.15.28:0/396770225 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:15:15.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.401+0000 7f335b400640 1 --2- 172.21.15.28:0/396770225 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f335c0aff90 0x7f335c0c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:15.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.401+0000 7f335be00640 1 --2- 172.21.15.28:0/396770225 >> v2:172.21.15.81:3300/0 conn(0x7f335c0c5720 0x7f335c159000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:15.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.401+0000 7f335aa00640 1 -- 172.21.15.28:0/396770225 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f335c0aff90 msgr2=0x7f335c0c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:15:15.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.401+0000 7f335aa00640 1 --2- 172.21.15.28:0/396770225 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f335c0aff90 0x7f335c0c51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:15.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.401+0000 7f335aa00640 1 -- 172.21.15.28:0/396770225 >> v2:172.21.15.81:3300/0 conn(0x7f335c0c5720 msgr2=0x7f335c159000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:15:15.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.401+0000 7f335aa00640 1 --2- 172.21.15.28:0/396770225 >> v2:172.21.15.81:3300/0 conn(0x7f335c0c5720 0x7f335c159000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:15.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.401+0000 7f335aa00640 1 -- 172.21.15.28:0/396770225 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f335c0c77a0 con 0x7f335c06dbf0 2024-02-21T23:15:15.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.401+0000 7f335be00640 1 --2- 172.21.15.28:0/396770225 >> v2:172.21.15.81:3300/0 conn(0x7f335c0c5720 0x7f335c159000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:15:15.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.403+0000 7f335aa00640 1 --2- 172.21.15.28:0/396770225 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f335c06dbf0 0x7f335c158ac0 secure :-1 s=READY pgs=2536 cs=0 l=1 rev1=1 crypto rx=0x7f335c0ca6e0 tx=0x7f334802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1f5982dfde03a9e server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:15.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.403+0000 7f335a000640 1 -- 172.21.15.28:0/396770225 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3348035020 con 0x7f335c06dbf0 2024-02-21T23:15:15.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.403+0000 7f335a000640 1 -- 172.21.15.28:0/396770225 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3348031870 con 0x7f335c06dbf0 2024-02-21T23:15:15.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.403+0000 7f335a000640 1 -- 172.21.15.28:0/396770225 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3348031c40 con 0x7f335c06dbf0 2024-02-21T23:15:15.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.404+0000 7f3360c00640 1 -- 172.21.15.28:0/396770225 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f335c06dbf0 msgr2=0x7f335c158ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:15.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.404+0000 7f3360c00640 1 --2- 172.21.15.28:0/396770225 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f335c06dbf0 0x7f335c158ac0 secure :-1 s=READY pgs=2536 cs=0 l=1 rev1=1 crypto rx=0x7f335c0ca6e0 tx=0x7f334802b910 comp rx=0 tx=0).stop 2024-02-21T23:15:15.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.404+0000 7f3360c00640 1 -- 172.21.15.28:0/396770225 shutdown_connections 2024-02-21T23:15:15.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.404+0000 7f3360c00640 1 --2- 172.21.15.28:0/396770225 >> v2:172.21.15.81:3300/0 conn(0x7f335c0c5720 0x7f335c159000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:15.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.404+0000 7f3360c00640 1 --2- 172.21.15.28:0/396770225 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f335c06dbf0 0x7f335c158ac0 unknown :-1 s=CLOSED pgs=2536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:15.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.404+0000 7f3360c00640 1 --2- 172.21.15.28:0/396770225 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f335c0aff90 0x7f335c0c51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:15.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.404+0000 7f3360c00640 1 -- 172.21.15.28:0/396770225 >> 172.21.15.28:0/396770225 conn(0x7f335c0ac370 msgr2=0x7f335c0ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:15:15.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.404+0000 7f3360c00640 1 -- 172.21.15.28:0/396770225 shutdown_connections 2024-02-21T23:15:15.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.404+0000 7f3360c00640 1 -- 172.21.15.28:0/396770225 wait complete. 2024-02-21T23:15:15.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.406+0000 7f3360c00640 1 Processor -- start 2024-02-21T23:15:15.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.406+0000 7f3360c00640 1 -- start start 2024-02-21T23:15:15.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.407+0000 7f3360c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f335c06dbf0 0x7f335c1516c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:15.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.407+0000 7f3360c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f335c0aff90 0x7f335c14e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:15.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.407+0000 7f3360c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f335c0c5720 0x7f335c14e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:15.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.407+0000 7f3360c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f335c0c8760 con 0x7f335c0aff90 2024-02-21T23:15:15.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.407+0000 7f3360c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f335c0be0c0 con 0x7f335c0c5720 2024-02-21T23:15:15.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.407+0000 7f3360c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f335c0c88e0 con 0x7f335c06dbf0 2024-02-21T23:15:15.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.407+0000 7f335aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f335c0aff90 0x7f335c14e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:15.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.407+0000 7f335aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f335c0aff90 0x7f335c14e190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39932/0 (socket says 172.21.15.28:39932) 2024-02-21T23:15:15.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.407+0000 7f335aa00640 1 -- 172.21.15.28:0/1976746287 learned_addr learned my addr 172.21.15.28:0/1976746287 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:15:15.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.407+0000 7f335be00640 1 --2- 172.21.15.28:0/1976746287 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f335c0c5720 0x7f335c14e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:15.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.408+0000 7f335b400640 1 --2- 172.21.15.28:0/1976746287 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f335c06dbf0 0x7f335c1516c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:15.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.408+0000 7f335aa00640 1 -- 172.21.15.28:0/1976746287 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f335c06dbf0 msgr2=0x7f335c1516c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:15.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.408+0000 7f335aa00640 1 --2- 172.21.15.28:0/1976746287 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f335c06dbf0 0x7f335c1516c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:15.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.408+0000 7f335aa00640 1 -- 172.21.15.28:0/1976746287 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f335c0c5720 msgr2=0x7f335c14e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:15.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.408+0000 7f335aa00640 1 --2- 172.21.15.28:0/1976746287 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f335c0c5720 0x7f335c14e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:15.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.408+0000 7f335aa00640 1 -- 172.21.15.28:0/1976746287 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f335c1eab90 con 0x7f335c0aff90 2024-02-21T23:15:15.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.408+0000 7f335b400640 1 --2- 172.21.15.28:0/1976746287 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f335c06dbf0 0x7f335c1516c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:15:15.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.408+0000 7f335aa00640 1 --2- 172.21.15.28:0/1976746287 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f335c0aff90 0x7f335c14e190 secure :-1 s=READY pgs=2537 cs=0 l=1 rev1=1 crypto rx=0x7f3348000c00 tx=0x7f33480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:15.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.408+0000 7f3353400640 1 -- 172.21.15.28:0/1976746287 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f334803ad60 con 0x7f335c0aff90 2024-02-21T23:15:15.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.408+0000 7f3353400640 1 -- 172.21.15.28:0/1976746287 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33480443f0 con 0x7f335c0aff90 2024-02-21T23:15:15.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.408+0000 7f3360c00640 1 -- 172.21.15.28:0/1976746287 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f335c1eae70 con 0x7f335c0aff90 2024-02-21T23:15:15.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.408+0000 7f3353400640 1 -- 172.21.15.28:0/1976746287 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33480447c0 con 0x7f335c0aff90 2024-02-21T23:15:15.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.409+0000 7f3360c00640 1 -- 172.21.15.28:0/1976746287 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f335c151c00 con 0x7f335c0aff90 2024-02-21T23:15:15.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.410+0000 7f3360c00640 1 -- 172.21.15.28:0/1976746287 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f335c04f290 con 0x7f335c0aff90 2024-02-21T23:15:15.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.411+0000 7f3353400640 1 -- 172.21.15.28:0/1976746287 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3348038020 con 0x7f335c0aff90 2024-02-21T23:15:15.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.412+0000 7f3353400640 1 --2- 172.21.15.28:0/1976746287 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f333c06b4d0 0x7f333c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:15.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.413+0000 7f3353400640 1 -- 172.21.15.28:0/1976746287 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f33480ac6f0 con 0x7f335c0aff90 2024-02-21T23:15:15.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.413+0000 7f335b400640 1 --2- 172.21.15.28:0/1976746287 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f333c06b4d0 0x7f333c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:15.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.414+0000 7f335b400640 1 --2- 172.21.15.28:0/1976746287 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f333c06b4d0 0x7f333c06d990 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f334c009f70 tx=0x7f334c00b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:15.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.416+0000 7f3353400640 1 -- 172.21.15.28:0/1976746287 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f334803f070 con 0x7f335c0aff90 2024-02-21T23:15:15.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.659+0000 7f3360c00640 1 -- 172.21.15.28:0/1976746287 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f335c0b0610 con 0x7f335c0aff90 2024-02-21T23:15:15.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.659+0000 7f3353400640 1 -- 172.21.15.28:0/1976746287 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 246 v246) v1 ==== 95+0+3384 (secure 0 0 0) 0x7f334807b240 con 0x7f335c0aff90 2024-02-21T23:15:15.648 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:15:15.648 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":246,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":246,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:14:46.655515+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:15:15.648 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 246 2024-02-21T23:15:15.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.662+0000 7f3360c00640 1 -- 172.21.15.28:0/1976746287 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f333c06b4d0 msgr2=0x7f333c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:15.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.662+0000 7f3360c00640 1 --2- 172.21.15.28:0/1976746287 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f333c06b4d0 0x7f333c06d990 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f334c009f70 tx=0x7f334c00b040 comp rx=0 tx=0).stop 2024-02-21T23:15:15.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.662+0000 7f3360c00640 1 -- 172.21.15.28:0/1976746287 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f335c0aff90 msgr2=0x7f335c14e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:15.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.662+0000 7f3360c00640 1 --2- 172.21.15.28:0/1976746287 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f335c0aff90 0x7f335c14e190 secure :-1 s=READY pgs=2537 cs=0 l=1 rev1=1 crypto rx=0x7f3348000c00 tx=0x7f33480046c0 comp rx=0 tx=0).stop 2024-02-21T23:15:15.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.662+0000 7f3360c00640 1 -- 172.21.15.28:0/1976746287 shutdown_connections 2024-02-21T23:15:15.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.662+0000 7f3360c00640 1 --2- 172.21.15.28:0/1976746287 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f333c06b4d0 0x7f333c06d990 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:15.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.662+0000 7f3360c00640 1 --2- 172.21.15.28:0/1976746287 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f335c0c5720 0x7f335c14e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:15.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.662+0000 7f3360c00640 1 --2- 172.21.15.28:0/1976746287 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f335c0aff90 0x7f335c14e190 unknown :-1 s=CLOSED pgs=2537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:15.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.662+0000 7f3360c00640 1 --2- 172.21.15.28:0/1976746287 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f335c06dbf0 0x7f335c1516c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:15.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.662+0000 7f3360c00640 1 -- 172.21.15.28:0/1976746287 >> 172.21.15.28:0/1976746287 conn(0x7f335c0ac370 msgr2=0x7f335c0ac750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:15:15.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.662+0000 7f3360c00640 1 -- 172.21.15.28:0/1976746287 shutdown_connections 2024-02-21T23:15:15.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:15.663+0000 7f3360c00640 1 -- 172.21.15.28:0/1976746287 wait complete. 2024-02-21T23:15:16.080 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 246, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:14:46.655515+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:15:16.081 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:15:17.082 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:15:17.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:17.315+0000 7f6485a00640 1 Processor -- start 2024-02-21T23:15:17.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:17.315+0000 7f6485a00640 1 -- start start 2024-02-21T23:15:17.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:17.317+0000 7f6485a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f64800c7be0 0x7f6480156f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:17.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:17.317+0000 7f6485a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64801564a0 con 0x7f64800c5a00 2024-02-21T23:15:17.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:17.317+0000 7f6485a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64800c5020 con 0x7f6480070ad0 2024-02-21T23:15:17.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:17.317+0000 7f6485a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64800c2f90 con 0x7f64800c7be0 2024-02-21T23:15:17.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:17.317+0000 7f647e000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f64800c7be0 0x7f6480156f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:17.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:17.317+0000 7f647ea00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f64800c5a00 0x7f64800c4910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34238/0 (socket says 172.21.15.28:34238) 2024-02-21T23:15:17.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:17.317+0000 7f647ea00640 1 -- 172.21.15.28:0/954065201 learned_addr learned my addr 172.21.15.28:0/954065201 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:15:17.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:17.318+0000 7f647e000640 1 -- 172.21.15.28:0/954065201 >> v1:172.21.15.81:6789/0 conn(0x7f6480070ad0 legacy=0x7f6480070f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:15:17.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:17.318+0000 7f647e000640 1 -- 172.21.15.28:0/954065201 >> v1:172.21.15.28:6789/0 conn(0x7f64800c5a00 legacy=0x7f64800c4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:17.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:17.318+0000 7f647d600640 1 -- 172.21.15.28:0/954065201 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 908342997 0 0) 0x7f64801564a0 con 0x7f64800c5a00 2024-02-21T23:15:17.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:17.318+0000 7f647e000640 1 -- 172.21.15.28:0/954065201 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6480071650 con 0x7f64800c7be0 2024-02-21T23:15:17.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:17.319+0000 7f647e000640 1 --2- 172.21.15.28:0/954065201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64800c7be0 0x7f6480156f40 secure :-1 s=READY pgs=2539 cs=0 l=1 rev1=1 crypto rx=0x7f646c0097a0 tx=0x7f646c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5498c0ce9ef97d8 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:17.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:17.320+0000 7f647d600640 1 -- 172.21.15.28:0/954065201 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f646c0315a0 con 0x7f64800c7be0 2024-02-21T23:15:17.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:17.320+0000 7f647d600640 1 -- 172.21.15.28:0/954065201 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f646c037070 con 0x7f64800c7be0 2024-02-21T23:15:17.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:17.320+0000 7f647d600640 1 -- 172.21.15.28:0/954065201 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f646c031960 con 0x7f64800c7be0 2024-02-21T23:15:17.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:17.320+0000 7f6485a00640 1 -- 172.21.15.28:0/954065201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64800c7be0 msgr2=0x7f6480156f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:17.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:17.320+0000 7f6485a00640 1 --2- 172.21.15.28:0/954065201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64800c7be0 0x7f6480156f40 secure :-1 s=READY pgs=2539 cs=0 l=1 rev1=1 crypto rx=0x7f646c0097a0 tx=0x7f646c02b840 comp rx=0 tx=0).stop 2024-02-21T23:15:17.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:17.320+0000 7f6485a00640 1 -- 172.21.15.28:0/954065201 shutdown_connections 2024-02-21T23:15:17.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:17.320+0000 7f6485a00640 1 --2- 172.21.15.28:0/954065201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64800c7be0 0x7f6480156f40 unknown :-1 s=CLOSED pgs=2539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:17.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:17.320+0000 7f6485a00640 1 -- 172.21.15.28:0/954065201 >> 172.21.15.28:0/954065201 conn(0x7f64800c8810 msgr2=0x7f64800c8bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:15:17.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:17.320+0000 7f6485a00640 1 -- 172.21.15.28:0/954065201 shutdown_connections 2024-02-21T23:15:17.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:17.320+0000 7f6485a00640 1 -- 172.21.15.28:0/954065201 wait complete. 2024-02-21T23:15:17.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:17.322+0000 7f6485a00640 1 Processor -- start 2024-02-21T23:15:17.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:17.322+0000 7f6485a00640 1 -- start start 2024-02-21T23:15:17.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:17.323+0000 7f6485a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6480070ad0 0x7f6480144530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:17.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:17.323+0000 7f6485a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64800c5a00 0x7f6480144a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:17.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:17.323+0000 7f6485a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64800c7be0 0x7f64801df630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:17.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:17.323+0000 7f6485a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6480156a20 con 0x7f64800c7be0 2024-02-21T23:15:17.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:17.323+0000 7f6485a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6480156740 con 0x7f64800c5a00 2024-02-21T23:15:17.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:17.323+0000 7f6485a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64800c5de0 con 0x7f6480070ad0 2024-02-21T23:15:17.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:17.324+0000 7f647f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64800c7be0 0x7f64801df630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:17.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:17.324+0000 7f647f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64800c7be0 0x7f64801df630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39954/0 (socket says 172.21.15.28:39954) 2024-02-21T23:15:17.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:17.324+0000 7f647f400640 1 -- 172.21.15.28:0/87232761 learned_addr learned my addr 172.21.15.28:0/87232761 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:15:17.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:17.324+0000 7f647e000640 1 --2- 172.21.15.28:0/87232761 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64800c5a00 0x7f6480144a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:17.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:17.324+0000 7f647ea00640 1 --2- 172.21.15.28:0/87232761 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6480070ad0 0x7f6480144530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:17.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:17.324+0000 7f647f400640 1 -- 172.21.15.28:0/87232761 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6480070ad0 msgr2=0x7f6480144530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:17.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:17.324+0000 7f647f400640 1 --2- 172.21.15.28:0/87232761 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6480070ad0 0x7f6480144530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:17.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:17.324+0000 7f647f400640 1 -- 172.21.15.28:0/87232761 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64800c5a00 msgr2=0x7f6480144a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:17.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:17.324+0000 7f647f400640 1 --2- 172.21.15.28:0/87232761 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64800c5a00 0x7f6480144a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:17.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:17.324+0000 7f647f400640 1 -- 172.21.15.28:0/87232761 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64800ad9b0 con 0x7f64800c7be0 2024-02-21T23:15:17.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:17.325+0000 7f647f400640 1 --2- 172.21.15.28:0/87232761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64800c7be0 0x7f64801df630 secure :-1 s=READY pgs=2540 cs=0 l=1 rev1=1 crypto rx=0x7f647800b970 tx=0x7f647800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:17.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:17.325+0000 7f6476a00640 1 -- 172.21.15.28:0/87232761 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6478019070 con 0x7f64800c7be0 2024-02-21T23:15:17.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:17.325+0000 7f6476a00640 1 -- 172.21.15.28:0/87232761 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f647800cb40 con 0x7f64800c7be0 2024-02-21T23:15:17.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:17.325+0000 7f6476a00640 1 -- 172.21.15.28:0/87232761 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6478014440 con 0x7f64800c7be0 2024-02-21T23:15:17.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:17.325+0000 7f6485a00640 1 -- 172.21.15.28:0/87232761 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64800adc90 con 0x7f64800c7be0 2024-02-21T23:15:17.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:17.325+0000 7f6485a00640 1 -- 172.21.15.28:0/87232761 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f648014e4d0 con 0x7f64800c7be0 2024-02-21T23:15:17.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:17.326+0000 7f6485a00640 1 -- 172.21.15.28:0/87232761 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6480068640 con 0x7f64800c7be0 2024-02-21T23:15:17.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:17.327+0000 7f6476a00640 1 -- 172.21.15.28:0/87232761 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f647801c070 con 0x7f64800c7be0 2024-02-21T23:15:17.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:17.328+0000 7f6476a00640 1 --2- 172.21.15.28:0/87232761 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f646006b4d0 0x7f646006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:17.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:17.328+0000 7f6476a00640 1 -- 172.21.15.28:0/87232761 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f647808a150 con 0x7f64800c7be0 2024-02-21T23:15:17.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:17.329+0000 7f647ea00640 1 --2- 172.21.15.28:0/87232761 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f646006b4d0 0x7f646006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:17.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:17.330+0000 7f647ea00640 1 --2- 172.21.15.28:0/87232761 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f646006b4d0 0x7f646006d990 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f6470000f60 tx=0x7f6470007c00 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:17.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:17.333+0000 7f6476a00640 1 -- 172.21.15.28:0/87232761 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f6478058ca0 con 0x7f64800c7be0 2024-02-21T23:15:17.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:17.622+0000 7f6485a00640 1 -- 172.21.15.28:0/87232761 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f64800c79c0 con 0x7f64800c7be0 2024-02-21T23:15:17.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:17.623+0000 7f6476a00640 1 -- 172.21.15.28:0/87232761 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 246 v246) v1 ==== 95+0+3384 (secure 0 0 0) 0x7f64780585a0 con 0x7f64800c7be0 2024-02-21T23:15:17.612 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:15:17.612 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":246,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":246,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:14:46.655515+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:15:17.612 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 246 2024-02-21T23:15:17.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:17.626+0000 7f6485a00640 1 -- 172.21.15.28:0/87232761 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f646006b4d0 msgr2=0x7f646006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:17.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:17.626+0000 7f6485a00640 1 --2- 172.21.15.28:0/87232761 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f646006b4d0 0x7f646006d990 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f6470000f60 tx=0x7f6470007c00 comp rx=0 tx=0).stop 2024-02-21T23:15:17.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:17.626+0000 7f6485a00640 1 -- 172.21.15.28:0/87232761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64800c7be0 msgr2=0x7f64801df630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:17.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:17.626+0000 7f6485a00640 1 --2- 172.21.15.28:0/87232761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64800c7be0 0x7f64801df630 secure :-1 s=READY pgs=2540 cs=0 l=1 rev1=1 crypto rx=0x7f647800b970 tx=0x7f647800be30 comp rx=0 tx=0).stop 2024-02-21T23:15:17.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:17.626+0000 7f6485a00640 1 -- 172.21.15.28:0/87232761 shutdown_connections 2024-02-21T23:15:17.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:17.626+0000 7f6485a00640 1 --2- 172.21.15.28:0/87232761 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f646006b4d0 0x7f646006d990 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:17.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:17.626+0000 7f6485a00640 1 --2- 172.21.15.28:0/87232761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64800c7be0 0x7f64801df630 unknown :-1 s=CLOSED pgs=2540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:17.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:17.626+0000 7f6485a00640 1 --2- 172.21.15.28:0/87232761 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64800c5a00 0x7f6480144a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:17.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:17.626+0000 7f6485a00640 1 --2- 172.21.15.28:0/87232761 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6480070ad0 0x7f6480144530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:17.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:17.626+0000 7f6485a00640 1 -- 172.21.15.28:0/87232761 >> 172.21.15.28:0/87232761 conn(0x7f64800c8810 msgr2=0x7f64800c39a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:15:17.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:17.626+0000 7f6485a00640 1 -- 172.21.15.28:0/87232761 shutdown_connections 2024-02-21T23:15:17.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:17.626+0000 7f6485a00640 1 -- 172.21.15.28:0/87232761 wait complete. 2024-02-21T23:15:18.038 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 246, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:14:46.655515+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:15:18.039 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:15:19.040 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:15:19.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:19.281+0000 7f5817000640 1 Processor -- start 2024-02-21T23:15:19.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:19.281+0000 7f5817000640 1 -- start start 2024-02-21T23:15:19.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:19.282+0000 7f5817000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f58100c11f0 0x7f5810153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:19.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:19.282+0000 7f5817000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5810152e70 con 0x7f58100c15d0 2024-02-21T23:15:19.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:19.282+0000 7f5817000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5810153150 con 0x7f581015b6d0 2024-02-21T23:15:19.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:19.282+0000 7f5817000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58100c1a60 con 0x7f58100c11f0 2024-02-21T23:15:19.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:19.283+0000 7f580f400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f58100c15d0 0x7f581006dce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34246/0 (socket says 172.21.15.28:34246) 2024-02-21T23:15:19.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:19.283+0000 7f580f400640 1 -- 172.21.15.28:0/2095305361 learned_addr learned my addr 172.21.15.28:0/2095305361 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:15:19.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:19.283+0000 7f580fe00640 1 --2- 172.21.15.28:0/2095305361 >> v2:172.21.15.28:3300/0 conn(0x7f58100c11f0 0x7f5810153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:19.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:19.283+0000 7f580fe00640 1 -- 172.21.15.28:0/2095305361 >> v1:172.21.15.81:6789/0 conn(0x7f581015b6d0 legacy=0x7f581015dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:15:19.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:19.284+0000 7f580fe00640 1 -- 172.21.15.28:0/2095305361 >> v1:172.21.15.28:6789/0 conn(0x7f58100c15d0 legacy=0x7f581006dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:19.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:19.284+0000 7f580fe00640 1 -- 172.21.15.28:0/2095305361 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5810153430 con 0x7f58100c11f0 2024-02-21T23:15:19.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:19.285+0000 7f580fe00640 1 --2- 172.21.15.28:0/2095305361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58100c11f0 0x7f5810153950 secure :-1 s=READY pgs=2542 cs=0 l=1 rev1=1 crypto rx=0x7f5800009d90 tx=0x7f580002bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e21803d9a4ed236c server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:19.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:19.285+0000 7f580ea00640 1 -- 172.21.15.28:0/2095305361 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5800039070 con 0x7f58100c11f0 2024-02-21T23:15:19.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:19.285+0000 7f580ea00640 1 -- 172.21.15.28:0/2095305361 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f580002cb40 con 0x7f58100c11f0 2024-02-21T23:15:19.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:19.285+0000 7f580ea00640 1 -- 172.21.15.28:0/2095305361 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5800034440 con 0x7f58100c11f0 2024-02-21T23:15:19.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:19.286+0000 7f5817000640 1 -- 172.21.15.28:0/2095305361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58100c11f0 msgr2=0x7f5810153950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:19.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:19.286+0000 7f5817000640 1 --2- 172.21.15.28:0/2095305361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58100c11f0 0x7f5810153950 secure :-1 s=READY pgs=2542 cs=0 l=1 rev1=1 crypto rx=0x7f5800009d90 tx=0x7f580002bd90 comp rx=0 tx=0).stop 2024-02-21T23:15:19.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:19.286+0000 7f5817000640 1 -- 172.21.15.28:0/2095305361 shutdown_connections 2024-02-21T23:15:19.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:19.286+0000 7f5817000640 1 --2- 172.21.15.28:0/2095305361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58100c11f0 0x7f5810153950 unknown :-1 s=CLOSED pgs=2542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:19.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:19.286+0000 7f5817000640 1 -- 172.21.15.28:0/2095305361 >> 172.21.15.28:0/2095305361 conn(0x7f58100c67f0 msgr2=0x7f58100c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:15:19.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:19.286+0000 7f5817000640 1 -- 172.21.15.28:0/2095305361 shutdown_connections 2024-02-21T23:15:19.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:19.286+0000 7f5817000640 1 -- 172.21.15.28:0/2095305361 wait complete. 2024-02-21T23:15:19.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:19.287+0000 7f5817000640 1 Processor -- start 2024-02-21T23:15:19.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:19.288+0000 7f5817000640 1 -- start start 2024-02-21T23:15:19.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:19.288+0000 7f5817000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f58100c11f0 0x7f58101e8e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:19.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:19.289+0000 7f5817000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58100c15d0 0x7f5810144e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:19.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:19.289+0000 7f5817000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f581015b6d0 0x7f58101497d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:19.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:19.289+0000 7f5817000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58100c4c00 con 0x7f581015b6d0 2024-02-21T23:15:19.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:19.289+0000 7f5817000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58100a5ec0 con 0x7f58100c15d0 2024-02-21T23:15:19.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:19.289+0000 7f5817000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58100a6110 con 0x7f58100c11f0 2024-02-21T23:15:19.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:19.289+0000 7f5815000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f581015b6d0 0x7f58101497d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:19.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:19.289+0000 7f5815000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f581015b6d0 0x7f58101497d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39962/0 (socket says 172.21.15.28:39962) 2024-02-21T23:15:19.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:19.289+0000 7f5815000640 1 -- 172.21.15.28:0/3949935800 learned_addr learned my addr 172.21.15.28:0/3949935800 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:15:19.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:19.289+0000 7f580f400640 1 --2- 172.21.15.28:0/3949935800 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58100c15d0 0x7f5810144e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:19.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:19.289+0000 7f580fe00640 1 --2- 172.21.15.28:0/3949935800 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f58100c11f0 0x7f58101e8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:19.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:19.289+0000 7f5815000640 1 -- 172.21.15.28:0/3949935800 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f58100c11f0 msgr2=0x7f58101e8e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:19.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:19.289+0000 7f5815000640 1 --2- 172.21.15.28:0/3949935800 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f58100c11f0 0x7f58101e8e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:19.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:19.289+0000 7f5815000640 1 -- 172.21.15.28:0/3949935800 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58100c15d0 msgr2=0x7f5810144e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:19.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:19.289+0000 7f5815000640 1 --2- 172.21.15.28:0/3949935800 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58100c15d0 0x7f5810144e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:19.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:19.289+0000 7f5815000640 1 -- 172.21.15.28:0/3949935800 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5810145360 con 0x7f581015b6d0 2024-02-21T23:15:19.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:19.289+0000 7f580f400640 1 --2- 172.21.15.28:0/3949935800 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58100c15d0 0x7f5810144e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:15:19.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:19.290+0000 7f5815000640 1 --2- 172.21.15.28:0/3949935800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f581015b6d0 0x7f58101497d0 secure :-1 s=READY pgs=2543 cs=0 l=1 rev1=1 crypto rx=0x7f580800c440 tx=0x7f580800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:19.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:19.290+0000 7f5807e00640 1 -- 172.21.15.28:0/3949935800 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5808011790 con 0x7f581015b6d0 2024-02-21T23:15:19.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:19.290+0000 7f5807e00640 1 -- 172.21.15.28:0/3949935800 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5808017070 con 0x7f581015b6d0 2024-02-21T23:15:19.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:19.290+0000 7f5807e00640 1 -- 172.21.15.28:0/3949935800 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5808011b70 con 0x7f581015b6d0 2024-02-21T23:15:19.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:19.290+0000 7f5817000640 1 -- 172.21.15.28:0/3949935800 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58101e9460 con 0x7f581015b6d0 2024-02-21T23:15:19.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:19.290+0000 7f5817000640 1 -- 172.21.15.28:0/3949935800 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58101e9a20 con 0x7f581015b6d0 2024-02-21T23:15:19.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:19.292+0000 7f5817000640 1 -- 172.21.15.28:0/3949935800 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5810067f30 con 0x7f581015b6d0 2024-02-21T23:15:19.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:19.292+0000 7f5807e00640 1 -- 172.21.15.28:0/3949935800 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5808011d10 con 0x7f581015b6d0 2024-02-21T23:15:19.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:19.294+0000 7f5807e00640 1 --2- 172.21.15.28:0/3949935800 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f57ec06b4d0 0x7f57ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:19.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:19.294+0000 7f5807e00640 1 -- 172.21.15.28:0/3949935800 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f580808a200 con 0x7f581015b6d0 2024-02-21T23:15:19.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:19.294+0000 7f580fe00640 1 --2- 172.21.15.28:0/3949935800 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f57ec06b4d0 0x7f57ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:19.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:19.296+0000 7f580fe00640 1 --2- 172.21.15.28:0/3949935800 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f57ec06b4d0 0x7f57ec06d990 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f5800036000 tx=0x7f5800030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:19.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:19.299+0000 7f5807e00640 1 -- 172.21.15.28:0/3949935800 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f5810145360 con 0x7f581015b6d0 2024-02-21T23:15:19.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:19.543+0000 7f5817000640 1 -- 172.21.15.28:0/3949935800 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f581006f0e0 con 0x7f581015b6d0 2024-02-21T23:15:19.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:19.544+0000 7f5807e00640 1 -- 172.21.15.28:0/3949935800 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7f5808058e80 con 0x7f581015b6d0 2024-02-21T23:15:19.532 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:15:19.532 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:15:19.533 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:15:19.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:19.546+0000 7f5817000640 1 -- 172.21.15.28:0/3949935800 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f57ec06b4d0 msgr2=0x7f57ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:19.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:19.546+0000 7f5817000640 1 --2- 172.21.15.28:0/3949935800 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f57ec06b4d0 0x7f57ec06d990 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f5800036000 tx=0x7f5800030000 comp rx=0 tx=0).stop 2024-02-21T23:15:19.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:19.546+0000 7f5817000640 1 -- 172.21.15.28:0/3949935800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f581015b6d0 msgr2=0x7f58101497d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:19.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:19.546+0000 7f5817000640 1 --2- 172.21.15.28:0/3949935800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f581015b6d0 0x7f58101497d0 secure :-1 s=READY pgs=2543 cs=0 l=1 rev1=1 crypto rx=0x7f580800c440 tx=0x7f580800c900 comp rx=0 tx=0).stop 2024-02-21T23:15:19.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:19.547+0000 7f5817000640 1 -- 172.21.15.28:0/3949935800 shutdown_connections 2024-02-21T23:15:19.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:19.547+0000 7f5817000640 1 --2- 172.21.15.28:0/3949935800 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f57ec06b4d0 0x7f57ec06d990 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:19.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:19.547+0000 7f5817000640 1 --2- 172.21.15.28:0/3949935800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f581015b6d0 0x7f58101497d0 unknown :-1 s=CLOSED pgs=2543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:19.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:19.547+0000 7f5817000640 1 --2- 172.21.15.28:0/3949935800 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f58100c15d0 0x7f5810144e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:19.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:19.547+0000 7f5817000640 1 --2- 172.21.15.28:0/3949935800 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f58100c11f0 0x7f58101e8e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:19.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:19.547+0000 7f5817000640 1 -- 172.21.15.28:0/3949935800 >> 172.21.15.28:0/3949935800 conn(0x7f58100c67f0 msgr2=0x7f58100a7df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:15:19.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:19.547+0000 7f5817000640 1 -- 172.21.15.28:0/3949935800 shutdown_connections 2024-02-21T23:15:19.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:19.547+0000 7f5817000640 1 -- 172.21.15.28:0/3949935800 wait complete. 2024-02-21T23:15:19.959 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:15:19.959 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:15:20.960 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:15:21.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.194+0000 7ff828c00640 1 Processor -- start 2024-02-21T23:15:21.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.195+0000 7ff828c00640 1 -- start start 2024-02-21T23:15:21.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.196+0000 7ff828c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff8240c6710 0x7ff8240c7fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:21.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.196+0000 7ff828c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7ff824158850 0x7ff824158cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:21.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.196+0000 7ff828c00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8240a8180 con 0x7ff824152ab0 2024-02-21T23:15:21.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.196+0000 7ff828c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8240c3ce0 con 0x7ff8240c6710 2024-02-21T23:15:21.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.196+0000 7ff828c00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8240c33b0 con 0x7ff824158850 2024-02-21T23:15:21.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.196+0000 7ff819400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff8240c6710 0x7ff8240c7fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:21.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.196+0000 7ff819400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff8240c6710 0x7ff8240c7fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39972/0 (socket says 172.21.15.28:39972) 2024-02-21T23:15:21.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.196+0000 7ff819400640 1 -- 172.21.15.28:0/1718375817 learned_addr learned my addr 172.21.15.28:0/1718375817 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:15:21.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.197+0000 7ff822800640 1 --2- 172.21.15.28:0/1718375817 >> v2:172.21.15.81:3300/0 conn(0x7ff824158850 0x7ff824158cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:21.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.197+0000 7ff819400640 1 -- 172.21.15.28:0/1718375817 >> v2:172.21.15.81:3300/0 conn(0x7ff824158850 msgr2=0x7ff824158cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:15:21.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.197+0000 7ff819400640 1 --2- 172.21.15.28:0/1718375817 >> v2:172.21.15.81:3300/0 conn(0x7ff824158850 0x7ff824158cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:21.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.197+0000 7ff819400640 1 -- 172.21.15.28:0/1718375817 >> v1:172.21.15.81:6789/0 conn(0x7ff824152ab0 legacy=0x7ff8240c5210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:21.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.197+0000 7ff819400640 1 -- 172.21.15.28:0/1718375817 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8240c3690 con 0x7ff8240c6710 2024-02-21T23:15:21.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.197+0000 7ff822800640 1 --2- 172.21.15.28:0/1718375817 >> v2:172.21.15.81:3300/0 conn(0x7ff824158850 0x7ff824158cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:15:21.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.198+0000 7ff819400640 1 --2- 172.21.15.28:0/1718375817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8240c6710 0x7ff8240c7fe0 secure :-1 s=READY pgs=2544 cs=0 l=1 rev1=1 crypto rx=0x7ff824072d70 tx=0x7ff80c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da68062ddd3ec3d1 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:21.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.199+0000 7ff821400640 1 -- 172.21.15.28:0/1718375817 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff80c034070 con 0x7ff8240c6710 2024-02-21T23:15:21.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.199+0000 7ff821400640 1 -- 172.21.15.28:0/1718375817 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff80c030860 con 0x7ff8240c6710 2024-02-21T23:15:21.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.199+0000 7ff821400640 1 -- 172.21.15.28:0/1718375817 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff80c030c30 con 0x7ff8240c6710 2024-02-21T23:15:21.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.199+0000 7ff828c00640 1 -- 172.21.15.28:0/1718375817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8240c6710 msgr2=0x7ff8240c7fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:21.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.199+0000 7ff828c00640 1 --2- 172.21.15.28:0/1718375817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8240c6710 0x7ff8240c7fe0 secure :-1 s=READY pgs=2544 cs=0 l=1 rev1=1 crypto rx=0x7ff824072d70 tx=0x7ff80c02b9a0 comp rx=0 tx=0).stop 2024-02-21T23:15:21.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.199+0000 7ff828c00640 1 -- 172.21.15.28:0/1718375817 shutdown_connections 2024-02-21T23:15:21.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.199+0000 7ff828c00640 1 --2- 172.21.15.28:0/1718375817 >> v2:172.21.15.81:3300/0 conn(0x7ff824158850 0x7ff824158cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:21.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.199+0000 7ff828c00640 1 --2- 172.21.15.28:0/1718375817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8240c6710 0x7ff8240c7fe0 unknown :-1 s=CLOSED pgs=2544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:21.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.199+0000 7ff828c00640 1 -- 172.21.15.28:0/1718375817 >> 172.21.15.28:0/1718375817 conn(0x7ff8240c8810 msgr2=0x7ff8240c8bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:15:21.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.200+0000 7ff828c00640 1 -- 172.21.15.28:0/1718375817 shutdown_connections 2024-02-21T23:15:21.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.200+0000 7ff828c00640 1 -- 172.21.15.28:0/1718375817 wait complete. 2024-02-21T23:15:21.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.201+0000 7ff828c00640 1 Processor -- start 2024-02-21T23:15:21.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.201+0000 7ff828c00640 1 -- start start 2024-02-21T23:15:21.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.202+0000 7ff828c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff8240c6710 0x7ff824070cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:21.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.202+0000 7ff828c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff824152ab0 0x7ff824071200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:21.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.202+0000 7ff828c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff824158850 0x7ff824147560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:21.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.202+0000 7ff828c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8240c5aa0 con 0x7ff824152ab0 2024-02-21T23:15:21.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.202+0000 7ff828c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8240c5920 con 0x7ff8240c6710 2024-02-21T23:15:21.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.202+0000 7ff828c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff824159200 con 0x7ff824158850 2024-02-21T23:15:21.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.202+0000 7ff819400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff824152ab0 0x7ff824071200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:21.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.203+0000 7ff819400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff824152ab0 0x7ff824071200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39974/0 (socket says 172.21.15.28:39974) 2024-02-21T23:15:21.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.203+0000 7ff819400640 1 -- 172.21.15.28:0/3288435011 learned_addr learned my addr 172.21.15.28:0/3288435011 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:15:21.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.203+0000 7ff819400640 1 -- 172.21.15.28:0/3288435011 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff824158850 msgr2=0x7ff824147560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:21.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.203+0000 7ff822800640 1 --2- 172.21.15.28:0/3288435011 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff824158850 0x7ff824147560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:21.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.203+0000 7ff821e00640 1 --2- 172.21.15.28:0/3288435011 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff8240c6710 0x7ff824070cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:21.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.203+0000 7ff819400640 1 --2- 172.21.15.28:0/3288435011 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff824158850 0x7ff824147560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:21.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.203+0000 7ff819400640 1 -- 172.21.15.28:0/3288435011 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff8240c6710 msgr2=0x7ff824070cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:21.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.203+0000 7ff819400640 1 --2- 172.21.15.28:0/3288435011 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff8240c6710 0x7ff824070cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:21.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.203+0000 7ff819400640 1 -- 172.21.15.28:0/3288435011 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff824070590 con 0x7ff824152ab0 2024-02-21T23:15:21.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.203+0000 7ff821e00640 1 --2- 172.21.15.28:0/3288435011 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff8240c6710 0x7ff824070cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:15:21.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.203+0000 7ff819400640 1 --2- 172.21.15.28:0/3288435011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff824152ab0 0x7ff824071200 secure :-1 s=READY pgs=2545 cs=0 l=1 rev1=1 crypto rx=0x7ff80c02b9e0 tx=0x7ff80c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:21.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.204+0000 7ff81aa00640 1 -- 172.21.15.28:0/3288435011 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff80c048070 con 0x7ff824152ab0 2024-02-21T23:15:21.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.204+0000 7ff81aa00640 1 -- 172.21.15.28:0/3288435011 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff80c039d90 con 0x7ff824152ab0 2024-02-21T23:15:21.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.204+0000 7ff81aa00640 1 -- 172.21.15.28:0/3288435011 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff80c034040 con 0x7ff824152ab0 2024-02-21T23:15:21.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.204+0000 7ff828c00640 1 -- 172.21.15.28:0/3288435011 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8241eb640 con 0x7ff824152ab0 2024-02-21T23:15:21.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.204+0000 7ff828c00640 1 -- 172.21.15.28:0/3288435011 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff82414b970 con 0x7ff824152ab0 2024-02-21T23:15:21.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.205+0000 7ff828c00640 1 -- 172.21.15.28:0/3288435011 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff824067f30 con 0x7ff824152ab0 2024-02-21T23:15:21.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.207+0000 7ff81aa00640 1 -- 172.21.15.28:0/3288435011 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff80c04b070 con 0x7ff824152ab0 2024-02-21T23:15:21.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.208+0000 7ff81aa00640 1 --2- 172.21.15.28:0/3288435011 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff80006b4d0 0x7ff80006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:21.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.208+0000 7ff81aa00640 1 -- 172.21.15.28:0/3288435011 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7ff80c0ac6d0 con 0x7ff824152ab0 2024-02-21T23:15:21.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.209+0000 7ff821e00640 1 --2- 172.21.15.28:0/3288435011 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff80006b4d0 0x7ff80006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:21.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.210+0000 7ff821e00640 1 --2- 172.21.15.28:0/3288435011 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff80006b4d0 0x7ff80006d990 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7ff8140012e0 tx=0x7ff814009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:21.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.214+0000 7ff81aa00640 1 -- 172.21.15.28:0/3288435011 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ff80c07b2b0 con 0x7ff824152ab0 2024-02-21T23:15:21.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.455+0000 7ff828c00640 1 -- 172.21.15.28:0/3288435011 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff8240c6190 con 0x7ff824152ab0 2024-02-21T23:15:21.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.456+0000 7ff81aa00640 1 -- 172.21.15.28:0/3288435011 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7ff80c07abb0 con 0x7ff824152ab0 2024-02-21T23:15:21.445 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:15:21.445 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:15:21.445 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:15:21.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.459+0000 7ff828c00640 1 -- 172.21.15.28:0/3288435011 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff80006b4d0 msgr2=0x7ff80006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:21.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.459+0000 7ff828c00640 1 --2- 172.21.15.28:0/3288435011 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff80006b4d0 0x7ff80006d990 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7ff8140012e0 tx=0x7ff814009040 comp rx=0 tx=0).stop 2024-02-21T23:15:21.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.459+0000 7ff828c00640 1 -- 172.21.15.28:0/3288435011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff824152ab0 msgr2=0x7ff824071200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:21.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.459+0000 7ff828c00640 1 --2- 172.21.15.28:0/3288435011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff824152ab0 0x7ff824071200 secure :-1 s=READY pgs=2545 cs=0 l=1 rev1=1 crypto rx=0x7ff80c02b9e0 tx=0x7ff80c004730 comp rx=0 tx=0).stop 2024-02-21T23:15:21.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.459+0000 7ff828c00640 1 -- 172.21.15.28:0/3288435011 shutdown_connections 2024-02-21T23:15:21.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.459+0000 7ff828c00640 1 --2- 172.21.15.28:0/3288435011 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff80006b4d0 0x7ff80006d990 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:21.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.459+0000 7ff828c00640 1 --2- 172.21.15.28:0/3288435011 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff824158850 0x7ff824147560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:21.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.459+0000 7ff828c00640 1 --2- 172.21.15.28:0/3288435011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff824152ab0 0x7ff824071200 unknown :-1 s=CLOSED pgs=2545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:21.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.459+0000 7ff828c00640 1 --2- 172.21.15.28:0/3288435011 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff8240c6710 0x7ff824070cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:21.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.459+0000 7ff828c00640 1 -- 172.21.15.28:0/3288435011 >> 172.21.15.28:0/3288435011 conn(0x7ff8240c8810 msgr2=0x7ff8241535e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:15:21.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.460+0000 7ff828c00640 1 -- 172.21.15.28:0/3288435011 shutdown_connections 2024-02-21T23:15:21.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:21.460+0000 7ff828c00640 1 -- 172.21.15.28:0/3288435011 wait complete. 2024-02-21T23:15:21.869 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:15:21.869 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:15:22.870 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:15:23.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.119+0000 7f0842400640 1 Processor -- start 2024-02-21T23:15:23.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.119+0000 7f0842400640 1 -- start start 2024-02-21T23:15:23.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.122+0000 7f0842400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f083c06dbf0 0x7f083c158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:23.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.122+0000 7f0842400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f083c0c74c0 con 0x7f083c0aff90 2024-02-21T23:15:23.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.122+0000 7f0842400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f083c0c77a0 con 0x7f083c155960 2024-02-21T23:15:23.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.122+0000 7f0842400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f083c0c7270 con 0x7f083c06dbf0 2024-02-21T23:15:23.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.122+0000 7f0841000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f083c0aff90 0x7f083c0c51e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34260/0 (socket says 172.21.15.28:34260) 2024-02-21T23:15:23.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.122+0000 7f0841000640 1 -- 172.21.15.28:0/180033093 learned_addr learned my addr 172.21.15.28:0/180033093 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:15:23.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.123+0000 7f083b400640 1 -- 172.21.15.28:0/180033093 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2745850489 0 0) 0x7f083c0c74c0 con 0x7f083c0aff90 2024-02-21T23:15:23.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.123+0000 7f083be00640 1 --2- 172.21.15.28:0/180033093 >> v2:172.21.15.81:3300/0 conn(0x7f083c06dbf0 0x7f083c158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:23.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.123+0000 7f083b400640 1 -- 172.21.15.28:0/180033093 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0824003520 con 0x7f083c0aff90 2024-02-21T23:15:23.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.123+0000 7f083b400640 1 -- 172.21.15.28:0/180033093 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3434299351 0 0) 0x7f0824003520 con 0x7f083c0aff90 2024-02-21T23:15:23.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.123+0000 7f083b400640 1 -- 172.21.15.28:0/180033093 >> v2:172.21.15.81:3300/0 conn(0x7f083c06dbf0 msgr2=0x7f083c158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:15:23.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.123+0000 7f083b400640 1 --2- 172.21.15.28:0/180033093 >> v2:172.21.15.81:3300/0 conn(0x7f083c06dbf0 0x7f083c158ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:23.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.123+0000 7f083b400640 1 -- 172.21.15.28:0/180033093 >> v1:172.21.15.81:6789/0 conn(0x7f083c155960 legacy=0x7f083c15f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:15:23.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.123+0000 7f083b400640 1 -- 172.21.15.28:0/180033093 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f083c0c7a80 con 0x7f083c0aff90 2024-02-21T23:15:23.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.123+0000 7f083b400640 1 -- 172.21.15.28:0/180033093 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f0830001670 con 0x7f083c0aff90 2024-02-21T23:15:23.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.124+0000 7f083b400640 1 -- 172.21.15.28:0/180033093 >> v1:172.21.15.28:6789/0 conn(0x7f083c0aff90 legacy=0x7f083c0c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:23.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.124+0000 7f083b400640 1 --2- 172.21.15.28:0/180033093 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f08240039b0 0x7f0824003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:23.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.124+0000 7f083b400640 1 --2- 172.21.15.28:0/180033093 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0824004800 0x7f0824004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:23.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.124+0000 7f083b400640 1 --2- 172.21.15.28:0/180033093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08240051c0 0x7f082400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:23.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.124+0000 7f083b400640 1 -- 172.21.15.28:0/180033093 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f083c0c7270 con 0x7f08240051c0 2024-02-21T23:15:23.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.124+0000 7f083b400640 1 -- 172.21.15.28:0/180033093 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f082400c060 con 0x7f0824004800 2024-02-21T23:15:23.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.124+0000 7f083b400640 1 -- 172.21.15.28:0/180033093 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f082400c260 con 0x7f08240039b0 2024-02-21T23:15:23.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.124+0000 7f0841a00640 1 --2- 172.21.15.28:0/180033093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08240051c0 0x7f082400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:23.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.124+0000 7f0841000640 1 --2- 172.21.15.28:0/180033093 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f08240039b0 0x7f0824003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:23.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.124+0000 7f083be00640 1 --2- 172.21.15.28:0/180033093 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0824004800 0x7f0824004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:23.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.125+0000 7f0841a00640 1 -- 172.21.15.28:0/180033093 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f08240039b0 msgr2=0x7f0824003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:15:23.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.125+0000 7f0841a00640 1 --2- 172.21.15.28:0/180033093 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f08240039b0 0x7f0824003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:23.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.125+0000 7f0841a00640 1 -- 172.21.15.28:0/180033093 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0824004800 msgr2=0x7f0824004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:15:23.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.125+0000 7f0841a00640 1 --2- 172.21.15.28:0/180033093 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0824004800 0x7f0824004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:23.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.125+0000 7f0841a00640 1 -- 172.21.15.28:0/180033093 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0824003520 con 0x7f08240051c0 2024-02-21T23:15:23.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.125+0000 7f083be00640 1 --2- 172.21.15.28:0/180033093 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0824004800 0x7f0824004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:15:23.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.126+0000 7f0841a00640 1 -- 172.21.15.28:0/180033093 reap_dead start 2024-02-21T23:15:23.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.127+0000 7f0841a00640 1 --2- 172.21.15.28:0/180033093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08240051c0 0x7f082400b9c0 secure :-1 s=READY pgs=2547 cs=0 l=1 rev1=1 crypto rx=0x7f0834001e50 tx=0x7f0824005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faac67ec2f07e321 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:23.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.127+0000 7f083b400640 1 -- 172.21.15.28:0/180033093 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0834001a70 con 0x7f08240051c0 2024-02-21T23:15:23.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.127+0000 7f083b400640 1 -- 172.21.15.28:0/180033093 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0834003b90 con 0x7f08240051c0 2024-02-21T23:15:23.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.127+0000 7f083b400640 1 -- 172.21.15.28:0/180033093 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08340050d0 con 0x7f08240051c0 2024-02-21T23:15:23.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.127+0000 7f0842400640 1 -- 172.21.15.28:0/180033093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08240051c0 msgr2=0x7f082400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:23.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.127+0000 7f0842400640 1 --2- 172.21.15.28:0/180033093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08240051c0 0x7f082400b9c0 secure :-1 s=READY pgs=2547 cs=0 l=1 rev1=1 crypto rx=0x7f0834001e50 tx=0x7f0824005830 comp rx=0 tx=0).stop 2024-02-21T23:15:23.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.128+0000 7f0842400640 1 -- 172.21.15.28:0/180033093 shutdown_connections 2024-02-21T23:15:23.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.128+0000 7f0842400640 1 --2- 172.21.15.28:0/180033093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08240051c0 0x7f082400b9c0 unknown :-1 s=CLOSED pgs=2547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:23.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.128+0000 7f0842400640 1 -- 172.21.15.28:0/180033093 >> 172.21.15.28:0/180033093 conn(0x7f083c0ac370 msgr2=0x7f083c0ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:15:23.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.128+0000 7f0842400640 1 -- 172.21.15.28:0/180033093 shutdown_connections 2024-02-21T23:15:23.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.128+0000 7f0842400640 1 -- 172.21.15.28:0/180033093 wait complete. 2024-02-21T23:15:23.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.129+0000 7f0842400640 1 Processor -- start 2024-02-21T23:15:23.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.130+0000 7f0842400640 1 -- start start 2024-02-21T23:15:23.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.130+0000 7f0842400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f08240051c0 0x7f083c0c4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:23.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.130+0000 7f0842400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f083c151000 0x7f083c154040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:23.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.130+0000 7f0842400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f083c0c37e0 0x7f083c0c08d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:23.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.130+0000 7f0842400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f083c0c8760 con 0x7f083c0c37e0 2024-02-21T23:15:23.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.130+0000 7f0842400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f083c158630 con 0x7f083c151000 2024-02-21T23:15:23.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.130+0000 7f0842400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f083c0c88e0 con 0x7f08240051c0 2024-02-21T23:15:23.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.131+0000 7f0841a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f083c0c37e0 0x7f083c0c08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:23.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.131+0000 7f0841a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f083c0c37e0 0x7f083c0c08d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39992/0 (socket says 172.21.15.28:39992) 2024-02-21T23:15:23.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.131+0000 7f0841a00640 1 -- 172.21.15.28:0/1569429889 learned_addr learned my addr 172.21.15.28:0/1569429889 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:15:23.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.131+0000 7f0841000640 1 --2- 172.21.15.28:0/1569429889 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f08240051c0 0x7f083c0c4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:23.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.131+0000 7f083be00640 1 --2- 172.21.15.28:0/1569429889 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f083c151000 0x7f083c154040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:23.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.131+0000 7f0841a00640 1 -- 172.21.15.28:0/1569429889 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f08240051c0 msgr2=0x7f083c0c4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:23.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.131+0000 7f0841a00640 1 --2- 172.21.15.28:0/1569429889 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f08240051c0 0x7f083c0c4590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:23.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.131+0000 7f0841a00640 1 -- 172.21.15.28:0/1569429889 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f083c151000 msgr2=0x7f083c154040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:23.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.131+0000 7f0841a00640 1 --2- 172.21.15.28:0/1569429889 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f083c151000 0x7f083c154040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:23.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.131+0000 7f0841a00640 1 -- 172.21.15.28:0/1569429889 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f083c0bed60 con 0x7f083c0c37e0 2024-02-21T23:15:23.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.132+0000 7f0841a00640 1 --2- 172.21.15.28:0/1569429889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f083c0c37e0 0x7f083c0c08d0 secure :-1 s=READY pgs=2548 cs=0 l=1 rev1=1 crypto rx=0x7f0834001f70 tx=0x7f08340014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:23.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.132+0000 7f0838c00640 1 -- 172.21.15.28:0/1569429889 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0834006d50 con 0x7f083c0c37e0 2024-02-21T23:15:23.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.132+0000 7f0838c00640 1 -- 172.21.15.28:0/1569429889 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0834007030 con 0x7f083c0c37e0 2024-02-21T23:15:23.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.132+0000 7f0838c00640 1 -- 172.21.15.28:0/1569429889 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08340073e0 con 0x7f083c0c37e0 2024-02-21T23:15:23.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.132+0000 7f0842400640 1 -- 172.21.15.28:0/1569429889 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f083c0bf040 con 0x7f083c0c37e0 2024-02-21T23:15:23.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.132+0000 7f0842400640 1 -- 172.21.15.28:0/1569429889 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f083c0be360 con 0x7f083c0c37e0 2024-02-21T23:15:23.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.133+0000 7f0842400640 1 -- 172.21.15.28:0/1569429889 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f083c0c5f80 con 0x7f083c0c37e0 2024-02-21T23:15:23.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.134+0000 7f0838c00640 1 -- 172.21.15.28:0/1569429889 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0834007790 con 0x7f083c0c37e0 2024-02-21T23:15:23.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.135+0000 7f0838c00640 1 --2- 172.21.15.28:0/1569429889 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f081c06b4d0 0x7f081c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:23.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.135+0000 7f0838c00640 1 -- 172.21.15.28:0/1569429889 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f08340ac4d0 con 0x7f083c0c37e0 2024-02-21T23:15:23.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.136+0000 7f0841000640 1 --2- 172.21.15.28:0/1569429889 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f081c06b4d0 0x7f081c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:23.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.137+0000 7f0841000640 1 --2- 172.21.15.28:0/1569429889 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f081c06b4d0 0x7f081c06d990 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f0830001eb0 tx=0x7f0830009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:23.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.144+0000 7f0838c00640 1 -- 172.21.15.28:0/1569429889 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f083c0c5f80 con 0x7f083c0c37e0 2024-02-21T23:15:23.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.380+0000 7f0842400640 1 -- 172.21.15.28:0/1569429889 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f083c04f290 con 0x7f083c0c37e0 2024-02-21T23:15:23.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.382+0000 7f0838c00640 1 -- 172.21.15.28:0/1569429889 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7f0834049030 con 0x7f083c0c37e0 2024-02-21T23:15:23.371 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:15:23.371 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:15:23.371 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:15:23.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.384+0000 7f0842400640 1 -- 172.21.15.28:0/1569429889 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f081c06b4d0 msgr2=0x7f081c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:23.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.384+0000 7f0842400640 1 --2- 172.21.15.28:0/1569429889 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f081c06b4d0 0x7f081c06d990 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f0830001eb0 tx=0x7f0830009040 comp rx=0 tx=0).stop 2024-02-21T23:15:23.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.384+0000 7f0842400640 1 -- 172.21.15.28:0/1569429889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f083c0c37e0 msgr2=0x7f083c0c08d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:23.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.384+0000 7f0842400640 1 --2- 172.21.15.28:0/1569429889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f083c0c37e0 0x7f083c0c08d0 secure :-1 s=READY pgs=2548 cs=0 l=1 rev1=1 crypto rx=0x7f0834001f70 tx=0x7f08340014a0 comp rx=0 tx=0).stop 2024-02-21T23:15:23.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.384+0000 7f0842400640 1 -- 172.21.15.28:0/1569429889 shutdown_connections 2024-02-21T23:15:23.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.384+0000 7f0842400640 1 --2- 172.21.15.28:0/1569429889 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f081c06b4d0 0x7f081c06d990 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:23.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.384+0000 7f0842400640 1 --2- 172.21.15.28:0/1569429889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f083c0c37e0 0x7f083c0c08d0 unknown :-1 s=CLOSED pgs=2548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:23.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.384+0000 7f0842400640 1 --2- 172.21.15.28:0/1569429889 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f083c151000 0x7f083c154040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:23.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.384+0000 7f0842400640 1 --2- 172.21.15.28:0/1569429889 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f08240051c0 0x7f083c0c4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:23.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.384+0000 7f0842400640 1 -- 172.21.15.28:0/1569429889 >> 172.21.15.28:0/1569429889 conn(0x7f083c0ac370 msgr2=0x7f083c151ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:15:23.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.385+0000 7f0842400640 1 -- 172.21.15.28:0/1569429889 shutdown_connections 2024-02-21T23:15:23.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:23.385+0000 7f0842400640 1 -- 172.21.15.28:0/1569429889 wait complete. 2024-02-21T23:15:23.794 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:15:23.795 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:15:24.796 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:15:25.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.050+0000 7fd7e2e00640 1 Processor -- start 2024-02-21T23:15:25.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.050+0000 7fd7e2e00640 1 -- start start 2024-02-21T23:15:25.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.052+0000 7fd7e2e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd7dc0c11f0 0x7fd7dc153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:25.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.052+0000 7fd7e2e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7dc152e70 con 0x7fd7dc15b6d0 2024-02-21T23:15:25.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.052+0000 7fd7e2e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7dc153150 con 0x7fd7dc0c15d0 2024-02-21T23:15:25.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.052+0000 7fd7e2e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7dc0c1a60 con 0x7fd7dc0c11f0 2024-02-21T23:15:25.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.052+0000 7fd7e0e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd7dc15b6d0 0x7fd7dc15dac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:40240/0 (socket says 172.21.15.28:40240) 2024-02-21T23:15:25.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.052+0000 7fd7e0e00640 1 -- 172.21.15.28:0/1999151347 learned_addr learned my addr 172.21.15.28:0/1999151347 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:15:25.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.052+0000 7fd7dbe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd7dc0c11f0 0x7fd7dc153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:25.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.053+0000 7fd7daa00640 1 -- 172.21.15.28:0/1999151347 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2250003676 0 0) 0x7fd7dc152e70 con 0x7fd7dc15b6d0 2024-02-21T23:15:25.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.053+0000 7fd7daa00640 1 -- 172.21.15.28:0/1999151347 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7c0003520 con 0x7fd7dc15b6d0 2024-02-21T23:15:25.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.053+0000 7fd7dbe00640 1 -- 172.21.15.28:0/1999151347 >> v1:172.21.15.81:6789/0 conn(0x7fd7dc0c15d0 legacy=0x7fd7dc06dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:25.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.053+0000 7fd7dbe00640 1 -- 172.21.15.28:0/1999151347 >> v1:172.21.15.28:6789/0 conn(0x7fd7dc15b6d0 legacy=0x7fd7dc15dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:25.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.053+0000 7fd7dbe00640 1 -- 172.21.15.28:0/1999151347 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7dc153430 con 0x7fd7dc0c11f0 2024-02-21T23:15:25.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.054+0000 7fd7dbe00640 1 --2- 172.21.15.28:0/1999151347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7dc0c11f0 0x7fd7dc153950 secure :-1 s=READY pgs=2550 cs=0 l=1 rev1=1 crypto rx=0x7fd7cc009d90 tx=0x7fd7cc02be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4185d5f26d5f418 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:25.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.055+0000 7fd7daa00640 1 -- 172.21.15.28:0/1999151347 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7cc039070 con 0x7fd7dc0c11f0 2024-02-21T23:15:25.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.055+0000 7fd7daa00640 1 -- 172.21.15.28:0/1999151347 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7cc02cbd0 con 0x7fd7dc0c11f0 2024-02-21T23:15:25.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.055+0000 7fd7daa00640 1 -- 172.21.15.28:0/1999151347 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7cc034440 con 0x7fd7dc0c11f0 2024-02-21T23:15:25.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.056+0000 7fd7e2e00640 1 -- 172.21.15.28:0/1999151347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7dc0c11f0 msgr2=0x7fd7dc153950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:25.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.056+0000 7fd7e2e00640 1 --2- 172.21.15.28:0/1999151347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7dc0c11f0 0x7fd7dc153950 secure :-1 s=READY pgs=2550 cs=0 l=1 rev1=1 crypto rx=0x7fd7cc009d90 tx=0x7fd7cc02be20 comp rx=0 tx=0).stop 2024-02-21T23:15:25.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.056+0000 7fd7e2e00640 1 -- 172.21.15.28:0/1999151347 shutdown_connections 2024-02-21T23:15:25.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.056+0000 7fd7e2e00640 1 --2- 172.21.15.28:0/1999151347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7dc0c11f0 0x7fd7dc153950 unknown :-1 s=CLOSED pgs=2550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:25.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.056+0000 7fd7e2e00640 1 -- 172.21.15.28:0/1999151347 >> 172.21.15.28:0/1999151347 conn(0x7fd7dc0c67f0 msgr2=0x7fd7dc0c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:15:25.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.056+0000 7fd7e2e00640 1 -- 172.21.15.28:0/1999151347 shutdown_connections 2024-02-21T23:15:25.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.056+0000 7fd7e2e00640 1 -- 172.21.15.28:0/1999151347 wait complete. 2024-02-21T23:15:25.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.058+0000 7fd7e2e00640 1 Processor -- start 2024-02-21T23:15:25.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.058+0000 7fd7e2e00640 1 -- start start 2024-02-21T23:15:25.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.059+0000 7fd7e2e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7dc0c11f0 0x7fd7dc147040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:25.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.059+0000 7fd7e2e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd7dc0c15d0 0x7fd7dc144a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:25.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.059+0000 7fd7e2e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd7dc15b6d0 0x7fd7dc144fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:25.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.059+0000 7fd7e2e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7dc0c4c00 con 0x7fd7dc0c11f0 2024-02-21T23:15:25.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.059+0000 7fd7e2e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7dc0a5ec0 con 0x7fd7dc15b6d0 2024-02-21T23:15:25.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.059+0000 7fd7e2e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7dc0a6110 con 0x7fd7dc0c15d0 2024-02-21T23:15:25.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.059+0000 7fd7dbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7dc0c11f0 0x7fd7dc147040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:25.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.060+0000 7fd7dbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7dc0c11f0 0x7fd7dc147040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44520/0 (socket says 172.21.15.28:44520) 2024-02-21T23:15:25.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.060+0000 7fd7dbe00640 1 -- 172.21.15.28:0/2391015753 learned_addr learned my addr 172.21.15.28:0/2391015753 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:15:25.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.060+0000 7fd7e0e00640 1 --2- 172.21.15.28:0/2391015753 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd7dc15b6d0 0x7fd7dc144fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:25.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.060+0000 7fd7dbe00640 1 -- 172.21.15.28:0/2391015753 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd7dc0c15d0 msgr2=0x7fd7dc144a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:25.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.060+0000 7fd7db400640 1 --2- 172.21.15.28:0/2391015753 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd7dc0c15d0 0x7fd7dc144a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:25.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.060+0000 7fd7dbe00640 1 --2- 172.21.15.28:0/2391015753 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd7dc0c15d0 0x7fd7dc144a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:25.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.060+0000 7fd7dbe00640 1 -- 172.21.15.28:0/2391015753 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd7dc15b6d0 msgr2=0x7fd7dc144fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:25.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.060+0000 7fd7dbe00640 1 --2- 172.21.15.28:0/2391015753 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd7dc15b6d0 0x7fd7dc144fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:25.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.060+0000 7fd7dbe00640 1 -- 172.21.15.28:0/2391015753 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7dc0c3360 con 0x7fd7dc0c11f0 2024-02-21T23:15:25.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.060+0000 7fd7e0e00640 1 --2- 172.21.15.28:0/2391015753 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd7dc15b6d0 0x7fd7dc144fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:15:25.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.060+0000 7fd7db400640 1 --2- 172.21.15.28:0/2391015753 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd7dc0c15d0 0x7fd7dc144a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:15:25.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.060+0000 7fd7dbe00640 1 --2- 172.21.15.28:0/2391015753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7dc0c11f0 0x7fd7dc147040 secure :-1 s=READY pgs=2551 cs=0 l=1 rev1=1 crypto rx=0x7fd7cc02c430 tx=0x7fd7cc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:25.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.061+0000 7fd7d3e00640 1 -- 172.21.15.28:0/2391015753 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7cc039050 con 0x7fd7dc0c11f0 2024-02-21T23:15:25.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.061+0000 7fd7d3e00640 1 -- 172.21.15.28:0/2391015753 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7cc045070 con 0x7fd7dc0c11f0 2024-02-21T23:15:25.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.061+0000 7fd7d3e00640 1 -- 172.21.15.28:0/2391015753 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7cc003e10 con 0x7fd7dc0c11f0 2024-02-21T23:15:25.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.061+0000 7fd7e2e00640 1 -- 172.21.15.28:0/2391015753 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7dc0a6e90 con 0x7fd7dc0c11f0 2024-02-21T23:15:25.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.061+0000 7fd7e2e00640 1 -- 172.21.15.28:0/2391015753 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7dc0a7430 con 0x7fd7dc0c11f0 2024-02-21T23:15:25.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.062+0000 7fd7e2e00640 1 -- 172.21.15.28:0/2391015753 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7dc067f30 con 0x7fd7dc0c11f0 2024-02-21T23:15:25.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.063+0000 7fd7d3e00640 1 -- 172.21.15.28:0/2391015753 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd7cc04a460 con 0x7fd7dc0c11f0 2024-02-21T23:15:25.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.064+0000 7fd7d3e00640 1 --2- 172.21.15.28:0/2391015753 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd7bc06b500 0x7fd7bc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:25.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.064+0000 7fd7d3e00640 1 -- 172.21.15.28:0/2391015753 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fd7cc0ace10 con 0x7fd7dc0c11f0 2024-02-21T23:15:25.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.065+0000 7fd7db400640 1 --2- 172.21.15.28:0/2391015753 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd7bc06b500 0x7fd7bc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:25.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.066+0000 7fd7db400640 1 --2- 172.21.15.28:0/2391015753 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd7bc06b500 0x7fd7bc06d9c0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc147870 tx=0x7fd7c8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:25.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.069+0000 7fd7d3e00640 1 -- 172.21.15.28:0/2391015753 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd7cc0b0030 con 0x7fd7dc0c11f0 2024-02-21T23:15:25.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.309+0000 7fd7e2e00640 1 -- 172.21.15.28:0/2391015753 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd7dc0c6240 con 0x7fd7dc0c11f0 2024-02-21T23:15:25.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.310+0000 7fd7d3e00640 1 -- 172.21.15.28:0/2391015753 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7fd7cc07bbd0 con 0x7fd7dc0c11f0 2024-02-21T23:15:25.298 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:15:25.298 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:15:25.298 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:15:25.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.312+0000 7fd7e2e00640 1 -- 172.21.15.28:0/2391015753 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd7bc06b500 msgr2=0x7fd7bc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:25.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.313+0000 7fd7e2e00640 1 --2- 172.21.15.28:0/2391015753 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd7bc06b500 0x7fd7bc06d9c0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc147870 tx=0x7fd7c8009040 comp rx=0 tx=0).stop 2024-02-21T23:15:25.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.313+0000 7fd7e2e00640 1 -- 172.21.15.28:0/2391015753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7dc0c11f0 msgr2=0x7fd7dc147040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:25.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.313+0000 7fd7e2e00640 1 --2- 172.21.15.28:0/2391015753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7dc0c11f0 0x7fd7dc147040 secure :-1 s=READY pgs=2551 cs=0 l=1 rev1=1 crypto rx=0x7fd7cc02c430 tx=0x7fd7cc004a40 comp rx=0 tx=0).stop 2024-02-21T23:15:25.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.313+0000 7fd7e2e00640 1 -- 172.21.15.28:0/2391015753 shutdown_connections 2024-02-21T23:15:25.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.313+0000 7fd7e2e00640 1 --2- 172.21.15.28:0/2391015753 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd7bc06b500 0x7fd7bc06d9c0 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:25.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.313+0000 7fd7e2e00640 1 --2- 172.21.15.28:0/2391015753 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd7dc0c15d0 0x7fd7dc144a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:25.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.313+0000 7fd7e2e00640 1 --2- 172.21.15.28:0/2391015753 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd7dc15b6d0 0x7fd7dc144fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:25.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.313+0000 7fd7e2e00640 1 --2- 172.21.15.28:0/2391015753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7dc0c11f0 0x7fd7dc147040 unknown :-1 s=CLOSED pgs=2551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:25.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.313+0000 7fd7e2e00640 1 -- 172.21.15.28:0/2391015753 >> 172.21.15.28:0/2391015753 conn(0x7fd7dc0c67f0 msgr2=0x7fd7dc0bfc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:15:25.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.313+0000 7fd7e2e00640 1 -- 172.21.15.28:0/2391015753 shutdown_connections 2024-02-21T23:15:25.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:25.313+0000 7fd7e2e00640 1 -- 172.21.15.28:0/2391015753 wait complete. 2024-02-21T23:15:25.723 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:15:25.723 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:15:26.724 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:15:26.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:26.962+0000 7f8c28a00640 1 Processor -- start 2024-02-21T23:15:26.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:26.962+0000 7f8c28a00640 1 -- start start 2024-02-21T23:15:26.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:26.964+0000 7f8c28a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8c2406dff0 0x7f8c2406e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:26.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:26.964+0000 7f8c28a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8c24158580 0x7f8c2414b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:26.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:26.964+0000 7f8c28a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c240a8b40 con 0x7f8c240c0fe0 2024-02-21T23:15:26.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:26.964+0000 7f8c28a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c240a8e00 con 0x7f8c2406dff0 2024-02-21T23:15:26.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:26.964+0000 7f8c28a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c240a90e0 con 0x7f8c24158580 2024-02-21T23:15:26.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:26.964+0000 7f8c22800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8c2406dff0 0x7f8c2406e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:26.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:26.964+0000 7f8c22800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8c2406dff0 0x7f8c2406e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44536/0 (socket says 172.21.15.28:44536) 2024-02-21T23:15:26.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:26.964+0000 7f8c22800640 1 -- 172.21.15.28:0/331882928 learned_addr learned my addr 172.21.15.28:0/331882928 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:15:26.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:26.965+0000 7f8c23200640 1 --2- 172.21.15.28:0/331882928 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8c24158580 0x7f8c2414b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:26.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:26.965+0000 7f8c22800640 1 -- 172.21.15.28:0/331882928 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8c24158580 msgr2=0x7f8c2414b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:15:26.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:26.965+0000 7f8c22800640 1 --2- 172.21.15.28:0/331882928 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8c24158580 0x7f8c2414b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:26.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:26.965+0000 7f8c22800640 1 -- 172.21.15.28:0/331882928 >> v1:172.21.15.28:6789/0 conn(0x7f8c240c0fe0 legacy=0x7f8c24071350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:26.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:26.965+0000 7f8c22800640 1 -- 172.21.15.28:0/331882928 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c24072ed0 con 0x7f8c2406dff0 2024-02-21T23:15:26.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:26.965+0000 7f8c23200640 1 --2- 172.21.15.28:0/331882928 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8c24158580 0x7f8c2414b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:15:26.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:26.966+0000 7f8c22800640 1 --2- 172.21.15.28:0/331882928 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c2406dff0 0x7f8c2406e3d0 secure :-1 s=READY pgs=2553 cs=0 l=1 rev1=1 crypto rx=0x7f8c14009e50 tx=0x7f8c1402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28e3246b8d4c36d0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:26.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:26.967+0000 7f8c21400640 1 -- 172.21.15.28:0/331882928 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c1402caf0 con 0x7f8c2406dff0 2024-02-21T23:15:26.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:26.967+0000 7f8c21400640 1 -- 172.21.15.28:0/331882928 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c1402cc90 con 0x7f8c2406dff0 2024-02-21T23:15:26.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:26.967+0000 7f8c21400640 1 -- 172.21.15.28:0/331882928 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c14034610 con 0x7f8c2406dff0 2024-02-21T23:15:26.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:26.967+0000 7f8c28a00640 1 -- 172.21.15.28:0/331882928 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c2406dff0 msgr2=0x7f8c2406e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:26.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:26.967+0000 7f8c28a00640 1 --2- 172.21.15.28:0/331882928 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c2406dff0 0x7f8c2406e3d0 secure :-1 s=READY pgs=2553 cs=0 l=1 rev1=1 crypto rx=0x7f8c14009e50 tx=0x7f8c1402bf50 comp rx=0 tx=0).stop 2024-02-21T23:15:26.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:26.967+0000 7f8c28a00640 1 -- 172.21.15.28:0/331882928 shutdown_connections 2024-02-21T23:15:26.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:26.967+0000 7f8c28a00640 1 --2- 172.21.15.28:0/331882928 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8c24158580 0x7f8c2414b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:26.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:26.967+0000 7f8c28a00640 1 --2- 172.21.15.28:0/331882928 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c2406dff0 0x7f8c2406e3d0 unknown :-1 s=CLOSED pgs=2553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:26.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:26.967+0000 7f8c28a00640 1 -- 172.21.15.28:0/331882928 >> 172.21.15.28:0/331882928 conn(0x7f8c2406b630 msgr2=0x7f8c240c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:15:26.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:26.967+0000 7f8c28a00640 1 -- 172.21.15.28:0/331882928 shutdown_connections 2024-02-21T23:15:26.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:26.968+0000 7f8c28a00640 1 -- 172.21.15.28:0/331882928 wait complete. 2024-02-21T23:15:26.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:26.969+0000 7f8c28a00640 1 Processor -- start 2024-02-21T23:15:26.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:26.969+0000 7f8c28a00640 1 -- start start 2024-02-21T23:15:26.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:26.970+0000 7f8c28a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8c2406dff0 0x7f8c24148ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:26.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:26.970+0000 7f8c28a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c240c0fe0 0x7f8c24149020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:26.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:26.970+0000 7f8c28a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c24158580 0x7f8c241479f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:26.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:26.970+0000 7f8c28a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c241589d0 con 0x7f8c24158580 2024-02-21T23:15:26.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:26.970+0000 7f8c28a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c24073520 con 0x7f8c240c0fe0 2024-02-21T23:15:26.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:26.970+0000 7f8c28a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c24158b50 con 0x7f8c2406dff0 2024-02-21T23:15:26.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:26.970+0000 7f8c23200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c24158580 0x7f8c241479f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:26.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:26.971+0000 7f8c23200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c24158580 0x7f8c241479f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44538/0 (socket says 172.21.15.28:44538) 2024-02-21T23:15:26.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:26.971+0000 7f8c23200640 1 -- 172.21.15.28:0/2547812812 learned_addr learned my addr 172.21.15.28:0/2547812812 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:15:26.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:26.971+0000 7f8c22800640 1 --2- 172.21.15.28:0/2547812812 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8c2406dff0 0x7f8c24148ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:26.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:26.971+0000 7f8c21e00640 1 --2- 172.21.15.28:0/2547812812 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c240c0fe0 0x7f8c24149020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:26.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:26.971+0000 7f8c23200640 1 -- 172.21.15.28:0/2547812812 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8c2406dff0 msgr2=0x7f8c24148ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:26.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:26.971+0000 7f8c23200640 1 --2- 172.21.15.28:0/2547812812 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8c2406dff0 0x7f8c24148ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:26.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:26.971+0000 7f8c23200640 1 -- 172.21.15.28:0/2547812812 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c240c0fe0 msgr2=0x7f8c24149020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:26.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:26.971+0000 7f8c23200640 1 --2- 172.21.15.28:0/2547812812 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c240c0fe0 0x7f8c24149020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:26.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:26.971+0000 7f8c23200640 1 -- 172.21.15.28:0/2547812812 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c240ab920 con 0x7f8c24158580 2024-02-21T23:15:26.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:26.971+0000 7f8c23200640 1 --2- 172.21.15.28:0/2547812812 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c24158580 0x7f8c241479f0 secure :-1 s=READY pgs=2554 cs=0 l=1 rev1=1 crypto rx=0x7f8c1c00c7f0 tx=0x7f8c1c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:26.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:26.972+0000 7f8c1aa00640 1 -- 172.21.15.28:0/2547812812 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c1c015070 con 0x7f8c24158580 2024-02-21T23:15:26.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:26.972+0000 7f8c1aa00640 1 -- 172.21.15.28:0/2547812812 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c1c007820 con 0x7f8c24158580 2024-02-21T23:15:26.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:26.972+0000 7f8c1aa00640 1 -- 172.21.15.28:0/2547812812 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c1c007c10 con 0x7f8c24158580 2024-02-21T23:15:26.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:26.972+0000 7f8c28a00640 1 -- 172.21.15.28:0/2547812812 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c240a9ee0 con 0x7f8c24158580 2024-02-21T23:15:26.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:26.972+0000 7f8c28a00640 1 -- 172.21.15.28:0/2547812812 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c240aa4a0 con 0x7f8c24158580 2024-02-21T23:15:26.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:26.973+0000 7f8c28a00640 1 -- 172.21.15.28:0/2547812812 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c2404f290 con 0x7f8c24158580 2024-02-21T23:15:26.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:26.975+0000 7f8c1aa00640 1 -- 172.21.15.28:0/2547812812 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8c1c018070 con 0x7f8c24158580 2024-02-21T23:15:26.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:26.976+0000 7f8c1aa00640 1 --2- 172.21.15.28:0/2547812812 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8c0406b4d0 0x7f8c0406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:26.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:26.976+0000 7f8c1aa00640 1 -- 172.21.15.28:0/2547812812 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f8c1c08a5a0 con 0x7f8c24158580 2024-02-21T23:15:26.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:26.976+0000 7f8c22800640 1 --2- 172.21.15.28:0/2547812812 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8c0406b4d0 0x7f8c0406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:26.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:26.978+0000 7f8c22800640 1 --2- 172.21.15.28:0/2547812812 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8c0406b4d0 0x7f8c0406d990 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f8c14035000 tx=0x7f8c14030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:26.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:26.980+0000 7f8c1aa00640 1 -- 172.21.15.28:0/2547812812 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f8c1c059220 con 0x7f8c24158580 2024-02-21T23:15:27.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:27.367+0000 7f8c28a00640 1 -- 172.21.15.28:0/2547812812 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8c2406ecb0 con 0x7f8c24158580 2024-02-21T23:15:27.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:27.368+0000 7f8c1aa00640 1 -- 172.21.15.28:0/2547812812 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7f8c1c058b20 con 0x7f8c24158580 2024-02-21T23:15:27.360 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:15:27.360 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:15:27.361 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:15:27.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:27.371+0000 7f8c28a00640 1 -- 172.21.15.28:0/2547812812 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8c0406b4d0 msgr2=0x7f8c0406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:27.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:27.371+0000 7f8c28a00640 1 --2- 172.21.15.28:0/2547812812 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8c0406b4d0 0x7f8c0406d990 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f8c14035000 tx=0x7f8c14030000 comp rx=0 tx=0).stop 2024-02-21T23:15:27.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:27.371+0000 7f8c28a00640 1 -- 172.21.15.28:0/2547812812 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c24158580 msgr2=0x7f8c241479f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:27.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:27.371+0000 7f8c28a00640 1 --2- 172.21.15.28:0/2547812812 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c24158580 0x7f8c241479f0 secure :-1 s=READY pgs=2554 cs=0 l=1 rev1=1 crypto rx=0x7f8c1c00c7f0 tx=0x7f8c1c00ccb0 comp rx=0 tx=0).stop 2024-02-21T23:15:27.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:27.371+0000 7f8c28a00640 1 -- 172.21.15.28:0/2547812812 shutdown_connections 2024-02-21T23:15:27.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:27.371+0000 7f8c28a00640 1 --2- 172.21.15.28:0/2547812812 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8c0406b4d0 0x7f8c0406d990 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:27.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:27.371+0000 7f8c28a00640 1 --2- 172.21.15.28:0/2547812812 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c24158580 0x7f8c241479f0 unknown :-1 s=CLOSED pgs=2554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:27.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:27.371+0000 7f8c28a00640 1 --2- 172.21.15.28:0/2547812812 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8c240c0fe0 0x7f8c24149020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:27.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:27.371+0000 7f8c28a00640 1 --2- 172.21.15.28:0/2547812812 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8c2406dff0 0x7f8c24148ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:27.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:27.371+0000 7f8c28a00640 1 -- 172.21.15.28:0/2547812812 >> 172.21.15.28:0/2547812812 conn(0x7f8c2406b630 msgr2=0x7f8c24151510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:15:27.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:27.371+0000 7f8c28a00640 1 -- 172.21.15.28:0/2547812812 shutdown_connections 2024-02-21T23:15:27.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:27.371+0000 7f8c28a00640 1 -- 172.21.15.28:0/2547812812 wait complete. 2024-02-21T23:15:27.782 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:15:27.783 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:15:28.784 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:15:29.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.057+0000 7fa9d7e00640 1 Processor -- start 2024-02-21T23:15:29.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.057+0000 7fa9d7e00640 1 -- start start 2024-02-21T23:15:29.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.058+0000 7fa9d7e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fa9d80aff90 0x7fa9d80c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:29.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.058+0000 7fa9d7e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa9d806dbf0 0x7fa9d8158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:29.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.058+0000 7fa9d7e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9d80c74c0 con 0x7fa9d80c5720 2024-02-21T23:15:29.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.058+0000 7fa9d7e00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9d80c7270 con 0x7fa9d80aff90 2024-02-21T23:15:29.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.058+0000 7fa9d7e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d8158630 con 0x7fa9d806dbf0 2024-02-21T23:15:29.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.059+0000 7fa9d7400640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fa9d80c5720 0x7fa9d806c270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.28:36694/0 (socket says 172.21.15.28:36694) 2024-02-21T23:15:29.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.059+0000 7fa9d7400640 1 -- 172.21.15.28:0/3692399693 learned_addr learned my addr 172.21.15.28:0/3692399693 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:15:29.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.059+0000 7fa9d6000640 1 --2- 172.21.15.28:0/3692399693 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa9d806dbf0 0x7fa9d8158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:29.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.060+0000 7fa9d6a00640 1 --2- 172.21.15.28:0/3692399693 >> v2:172.21.15.81:3300/0 conn(0x7fa9d80aff90 0x7fa9d80c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:29.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.060+0000 7fa9d6000640 1 -- 172.21.15.28:0/3692399693 >> v2:172.21.15.81:3300/0 conn(0x7fa9d80aff90 msgr2=0x7fa9d80c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:15:29.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.060+0000 7fa9d6000640 1 --2- 172.21.15.28:0/3692399693 >> v2:172.21.15.81:3300/0 conn(0x7fa9d80aff90 0x7fa9d80c51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:29.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.060+0000 7fa9d6000640 1 -- 172.21.15.28:0/3692399693 >> v1:172.21.15.81:6789/0 conn(0x7fa9d80c5720 legacy=0x7fa9d806c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:29.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.060+0000 7fa9d6000640 1 -- 172.21.15.28:0/3692399693 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9d80c77a0 con 0x7fa9d806dbf0 2024-02-21T23:15:29.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.060+0000 7fa9d5600640 1 -- 172.21.15.28:0/3692399693 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2407286035 0 0) 0x7fa9d80c74c0 con 0x7fa9d80c5720 2024-02-21T23:15:29.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.062+0000 7fa9d6000640 1 --2- 172.21.15.28:0/3692399693 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa9d806dbf0 0x7fa9d8158ac0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa9c40097a0 tx=0x7fa9c402b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23622866750cc8b9 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:29.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.062+0000 7fa9d5600640 1 -- 172.21.15.28:0/3692399693 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9c4035020 con 0x7fa9d806dbf0 2024-02-21T23:15:29.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.062+0000 7fa9d5600640 1 -- 172.21.15.28:0/3692399693 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9c4031810 con 0x7fa9d806dbf0 2024-02-21T23:15:29.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.062+0000 7fa9d5600640 1 -- 172.21.15.28:0/3692399693 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9c4031be0 con 0x7fa9d806dbf0 2024-02-21T23:15:29.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.063+0000 7fa9d7e00640 1 -- 172.21.15.28:0/3692399693 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa9d806dbf0 msgr2=0x7fa9d8158ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:29.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.063+0000 7fa9d7e00640 1 --2- 172.21.15.28:0/3692399693 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa9d806dbf0 0x7fa9d8158ac0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa9c40097a0 tx=0x7fa9c402b8b0 comp rx=0 tx=0).stop 2024-02-21T23:15:29.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.063+0000 7fa9d7e00640 1 -- 172.21.15.28:0/3692399693 shutdown_connections 2024-02-21T23:15:29.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.063+0000 7fa9d7e00640 1 --2- 172.21.15.28:0/3692399693 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa9d806dbf0 0x7fa9d8158ac0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:29.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.063+0000 7fa9d7e00640 1 --2- 172.21.15.28:0/3692399693 >> v2:172.21.15.81:3300/0 conn(0x7fa9d80aff90 0x7fa9d80c51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:29.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.063+0000 7fa9d7e00640 1 -- 172.21.15.28:0/3692399693 >> 172.21.15.28:0/3692399693 conn(0x7fa9d80ac370 msgr2=0x7fa9d80ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:15:29.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.063+0000 7fa9d7e00640 1 -- 172.21.15.28:0/3692399693 shutdown_connections 2024-02-21T23:15:29.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.063+0000 7fa9d7e00640 1 -- 172.21.15.28:0/3692399693 wait complete. 2024-02-21T23:15:29.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.065+0000 7fa9d7e00640 1 Processor -- start 2024-02-21T23:15:29.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.065+0000 7fa9d7e00640 1 -- start start 2024-02-21T23:15:29.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.067+0000 7fa9d7e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa9d806dbf0 0x7fa9d80c3e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:29.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.067+0000 7fa9d7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9d80aff90 0x7fa9d80bccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:29.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.067+0000 7fa9d7e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa9d80c5720 0x7fa9d80bd230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:29.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.067+0000 7fa9d7e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d80c8760 con 0x7fa9d80aff90 2024-02-21T23:15:29.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.067+0000 7fa9d7e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d80be0c0 con 0x7fa9d806dbf0 2024-02-21T23:15:29.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.067+0000 7fa9d7e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d80c88e0 con 0x7fa9d80c5720 2024-02-21T23:15:29.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.067+0000 7fa9d6000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9d80aff90 0x7fa9d80bccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:29.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.067+0000 7fa9d6000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9d80aff90 0x7fa9d80bccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44550/0 (socket says 172.21.15.28:44550) 2024-02-21T23:15:29.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.067+0000 7fa9d6000640 1 -- 172.21.15.28:0/3082263553 learned_addr learned my addr 172.21.15.28:0/3082263553 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:15:29.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.067+0000 7fa9d6a00640 1 --2- 172.21.15.28:0/3082263553 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa9d806dbf0 0x7fa9d80c3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:29.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.067+0000 7fa9d7400640 1 --2- 172.21.15.28:0/3082263553 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa9d80c5720 0x7fa9d80bd230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:29.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.067+0000 7fa9d6000640 1 -- 172.21.15.28:0/3082263553 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa9d80c5720 msgr2=0x7fa9d80bd230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:29.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.067+0000 7fa9d6000640 1 --2- 172.21.15.28:0/3082263553 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa9d80c5720 0x7fa9d80bd230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:29.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.067+0000 7fa9d6000640 1 -- 172.21.15.28:0/3082263553 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa9d806dbf0 msgr2=0x7fa9d80c3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:29.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.067+0000 7fa9d6000640 1 --2- 172.21.15.28:0/3082263553 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa9d806dbf0 0x7fa9d80c3e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:29.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.068+0000 7fa9d6000640 1 -- 172.21.15.28:0/3082263553 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9d80bd770 con 0x7fa9d80aff90 2024-02-21T23:15:29.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.068+0000 7fa9d6000640 1 --2- 172.21.15.28:0/3082263553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9d80aff90 0x7fa9d80bccf0 secure :-1 s=READY pgs=2555 cs=0 l=1 rev1=1 crypto rx=0x7fa9c40098c0 tx=0x7fa9c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:29.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.068+0000 7fa9cea00640 1 -- 172.21.15.28:0/3082263553 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9c40453f0 con 0x7fa9d80aff90 2024-02-21T23:15:29.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.068+0000 7fa9cea00640 1 -- 172.21.15.28:0/3082263553 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9c4045590 con 0x7fa9d80aff90 2024-02-21T23:15:29.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.068+0000 7fa9d7e00640 1 -- 172.21.15.28:0/3082263553 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9d81eaf40 con 0x7fa9d80aff90 2024-02-21T23:15:29.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.068+0000 7fa9cea00640 1 -- 172.21.15.28:0/3082263553 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9c4045980 con 0x7fa9d80aff90 2024-02-21T23:15:29.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.068+0000 7fa9d7e00640 1 -- 172.21.15.28:0/3082263553 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9d80c1b80 con 0x7fa9d80aff90 2024-02-21T23:15:29.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.070+0000 7fa9d7e00640 1 -- 172.21.15.28:0/3082263553 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9d804f290 con 0x7fa9d80aff90 2024-02-21T23:15:29.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.071+0000 7fa9cea00640 1 -- 172.21.15.28:0/3082263553 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa9c404c060 con 0x7fa9d80aff90 2024-02-21T23:15:29.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.071+0000 7fa9cea00640 1 --2- 172.21.15.28:0/3082263553 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa9b806b4d0 0x7fa9b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:29.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.072+0000 7fa9cea00640 1 -- 172.21.15.28:0/3082263553 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fa9c40ac370 con 0x7fa9d80aff90 2024-02-21T23:15:29.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.072+0000 7fa9d6a00640 1 --2- 172.21.15.28:0/3082263553 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa9b806b4d0 0x7fa9b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:29.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.073+0000 7fa9d6a00640 1 --2- 172.21.15.28:0/3082263553 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa9b806b4d0 0x7fa9b806d990 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fa9c8006fd0 tx=0x7fa9c8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:29.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.076+0000 7fa9cea00640 1 -- 172.21.15.28:0/3082263553 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fa9c4040070 con 0x7fa9d80aff90 2024-02-21T23:15:29.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.323+0000 7fa9d7e00640 1 -- 172.21.15.28:0/3082263553 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa9d806d9d0 con 0x7fa9d80aff90 2024-02-21T23:15:29.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.324+0000 7fa9cea00640 1 -- 172.21.15.28:0/3082263553 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7fa9c407aff0 con 0x7fa9d80aff90 2024-02-21T23:15:29.312 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:15:29.312 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:15:29.312 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:15:29.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.327+0000 7fa9d7e00640 1 -- 172.21.15.28:0/3082263553 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa9b806b4d0 msgr2=0x7fa9b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:29.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.327+0000 7fa9d7e00640 1 --2- 172.21.15.28:0/3082263553 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa9b806b4d0 0x7fa9b806d990 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fa9c8006fd0 tx=0x7fa9c8008040 comp rx=0 tx=0).stop 2024-02-21T23:15:29.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.327+0000 7fa9d7e00640 1 -- 172.21.15.28:0/3082263553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9d80aff90 msgr2=0x7fa9d80bccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:29.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.327+0000 7fa9d7e00640 1 --2- 172.21.15.28:0/3082263553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9d80aff90 0x7fa9d80bccf0 secure :-1 s=READY pgs=2555 cs=0 l=1 rev1=1 crypto rx=0x7fa9c40098c0 tx=0x7fa9c40046c0 comp rx=0 tx=0).stop 2024-02-21T23:15:29.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.327+0000 7fa9d7e00640 1 -- 172.21.15.28:0/3082263553 shutdown_connections 2024-02-21T23:15:29.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.327+0000 7fa9d7e00640 1 --2- 172.21.15.28:0/3082263553 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa9b806b4d0 0x7fa9b806d990 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:29.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.327+0000 7fa9d7e00640 1 --2- 172.21.15.28:0/3082263553 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa9d80c5720 0x7fa9d80bd230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:29.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.327+0000 7fa9d7e00640 1 --2- 172.21.15.28:0/3082263553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9d80aff90 0x7fa9d80bccf0 unknown :-1 s=CLOSED pgs=2555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:29.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.327+0000 7fa9d7e00640 1 --2- 172.21.15.28:0/3082263553 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa9d806dbf0 0x7fa9d80c3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:29.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.327+0000 7fa9d7e00640 1 -- 172.21.15.28:0/3082263553 >> 172.21.15.28:0/3082263553 conn(0x7fa9d80ac370 msgr2=0x7fa9d80c9f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:15:29.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.328+0000 7fa9d7e00640 1 -- 172.21.15.28:0/3082263553 shutdown_connections 2024-02-21T23:15:29.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:29.328+0000 7fa9d7e00640 1 -- 172.21.15.28:0/3082263553 wait complete. 2024-02-21T23:15:29.744 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:15:29.744 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:15:30.745 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:15:30.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:30.991+0000 7fd4efe00640 1 Processor -- start 2024-02-21T23:15:30.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:30.992+0000 7fd4efe00640 1 -- start start 2024-02-21T23:15:30.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:30.993+0000 7fd4efe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd4f006dbf0 0x7fd4f0158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:30.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:30.993+0000 7fd4efe00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd4f0155960 0x7fd4f0159000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:30.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:30.993+0000 7fd4efe00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4f00c74c0 con 0x7fd4f00aff90 2024-02-21T23:15:30.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:30.993+0000 7fd4efe00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4f00c7270 con 0x7fd4f006dbf0 2024-02-21T23:15:30.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:30.993+0000 7fd4efe00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f0158630 con 0x7fd4f0155960 2024-02-21T23:15:30.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:30.993+0000 7fd4eea00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd4f00aff90 0x7fd4f00c51e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:40260/0 (socket says 172.21.15.28:40260) 2024-02-21T23:15:30.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:30.993+0000 7fd4eea00640 1 -- 172.21.15.28:0/3508967049 learned_addr learned my addr 172.21.15.28:0/3508967049 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:15:30.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:30.993+0000 7fd4ee000640 1 --2- 172.21.15.28:0/3508967049 >> v2:172.21.15.28:3300/0 conn(0x7fd4f006dbf0 0x7fd4f0158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:30.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:30.994+0000 7fd4ef400640 1 --2- 172.21.15.28:0/3508967049 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd4f0155960 0x7fd4f0159000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:30.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:30.994+0000 7fd4ed600640 1 -- 172.21.15.28:0/3508967049 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1821386941 0 0) 0x7fd4f00c74c0 con 0x7fd4f00aff90 2024-02-21T23:15:30.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:30.994+0000 7fd4ed600640 1 -- 172.21.15.28:0/3508967049 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4d4003520 con 0x7fd4f00aff90 2024-02-21T23:15:30.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:30.994+0000 7fd4ee000640 1 -- 172.21.15.28:0/3508967049 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd4f0155960 msgr2=0x7fd4f0159000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:15:30.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:30.994+0000 7fd4ee000640 1 --2- 172.21.15.28:0/3508967049 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd4f0155960 0x7fd4f0159000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:30.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:30.994+0000 7fd4ee000640 1 -- 172.21.15.28:0/3508967049 >> v1:172.21.15.28:6789/0 conn(0x7fd4f00aff90 legacy=0x7fd4f00c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:30.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:30.994+0000 7fd4ee000640 1 -- 172.21.15.28:0/3508967049 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4f00c77a0 con 0x7fd4f006dbf0 2024-02-21T23:15:30.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:30.995+0000 7fd4ee000640 1 --2- 172.21.15.28:0/3508967049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4f006dbf0 0x7fd4f0158ac0 secure :-1 s=READY pgs=2557 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc0097a0 tx=0x7fd4dc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=333727c3074aff33 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:30.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:30.996+0000 7fd4ed600640 1 -- 172.21.15.28:0/3508967049 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4dc035020 con 0x7fd4f006dbf0 2024-02-21T23:15:30.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:30.996+0000 7fd4ed600640 1 -- 172.21.15.28:0/3508967049 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4dc031830 con 0x7fd4f006dbf0 2024-02-21T23:15:30.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:30.996+0000 7fd4ed600640 1 -- 172.21.15.28:0/3508967049 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4dc031c00 con 0x7fd4f006dbf0 2024-02-21T23:15:30.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:30.996+0000 7fd4efe00640 1 -- 172.21.15.28:0/3508967049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4f006dbf0 msgr2=0x7fd4f0158ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:30.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:30.996+0000 7fd4efe00640 1 --2- 172.21.15.28:0/3508967049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4f006dbf0 0x7fd4f0158ac0 secure :-1 s=READY pgs=2557 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc0097a0 tx=0x7fd4dc02b8d0 comp rx=0 tx=0).stop 2024-02-21T23:15:30.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:30.996+0000 7fd4efe00640 1 -- 172.21.15.28:0/3508967049 shutdown_connections 2024-02-21T23:15:30.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:30.996+0000 7fd4efe00640 1 --2- 172.21.15.28:0/3508967049 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd4f0155960 0x7fd4f0159000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:30.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:30.996+0000 7fd4efe00640 1 --2- 172.21.15.28:0/3508967049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4f006dbf0 0x7fd4f0158ac0 unknown :-1 s=CLOSED pgs=2557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:30.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:30.996+0000 7fd4efe00640 1 -- 172.21.15.28:0/3508967049 >> 172.21.15.28:0/3508967049 conn(0x7fd4f00ac370 msgr2=0x7fd4f00ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:15:30.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:30.996+0000 7fd4efe00640 1 -- 172.21.15.28:0/3508967049 shutdown_connections 2024-02-21T23:15:30.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:30.996+0000 7fd4efe00640 1 -- 172.21.15.28:0/3508967049 wait complete. 2024-02-21T23:15:30.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:30.999+0000 7fd4efe00640 1 Processor -- start 2024-02-21T23:15:30.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:30.999+0000 7fd4efe00640 1 -- start start 2024-02-21T23:15:30.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:31.001+0000 7fd4efe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4f006dbf0 0x7fd4f00be240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:30.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:31.001+0000 7fd4efe00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd4f00aff90 0x7fd4f0154750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:30.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:31.001+0000 7fd4efe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd4f0155960 0x7fd4f01ee550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:30.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:31.001+0000 7fd4efe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f00c58f0 con 0x7fd4f006dbf0 2024-02-21T23:15:30.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:31.001+0000 7fd4efe00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f00be0c0 con 0x7fd4f0155960 2024-02-21T23:15:30.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:31.001+0000 7fd4efe00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f00c5a70 con 0x7fd4f00aff90 2024-02-21T23:15:30.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:31.001+0000 7fd4eea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4f006dbf0 0x7fd4f00be240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:30.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:31.001+0000 7fd4eea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4f006dbf0 0x7fd4f00be240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44568/0 (socket says 172.21.15.28:44568) 2024-02-21T23:15:30.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:31.001+0000 7fd4eea00640 1 -- 172.21.15.28:0/3162931987 learned_addr learned my addr 172.21.15.28:0/3162931987 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:15:30.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:31.001+0000 7fd4ee000640 1 --2- 172.21.15.28:0/3162931987 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd4f00aff90 0x7fd4f0154750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:30.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:31.002+0000 7fd4ef400640 1 --2- 172.21.15.28:0/3162931987 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd4f0155960 0x7fd4f01ee550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:30.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:31.002+0000 7fd4eea00640 1 -- 172.21.15.28:0/3162931987 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd4f00aff90 msgr2=0x7fd4f0154750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:30.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:31.002+0000 7fd4eea00640 1 --2- 172.21.15.28:0/3162931987 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd4f00aff90 0x7fd4f0154750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:30.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:31.002+0000 7fd4eea00640 1 -- 172.21.15.28:0/3162931987 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd4f0155960 msgr2=0x7fd4f01ee550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:30.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:31.002+0000 7fd4eea00640 1 --2- 172.21.15.28:0/3162931987 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd4f0155960 0x7fd4f01ee550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:30.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:31.002+0000 7fd4eea00640 1 -- 172.21.15.28:0/3162931987 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4f01eb300 con 0x7fd4f006dbf0 2024-02-21T23:15:30.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:31.002+0000 7fd4ef400640 1 --2- 172.21.15.28:0/3162931987 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd4f0155960 0x7fd4f01ee550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:15:30.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:31.002+0000 7fd4eea00640 1 --2- 172.21.15.28:0/3162931987 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4f006dbf0 0x7fd4f00be240 secure :-1 s=READY pgs=2558 cs=0 l=1 rev1=1 crypto rx=0x7fd4e000c460 tx=0x7fd4e000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:30.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:31.002+0000 7fd4e7400640 1 -- 172.21.15.28:0/3162931987 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4e0017070 con 0x7fd4f006dbf0 2024-02-21T23:15:30.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:31.002+0000 7fd4e7400640 1 -- 172.21.15.28:0/3162931987 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4e0005ce0 con 0x7fd4f006dbf0 2024-02-21T23:15:30.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:31.002+0000 7fd4e7400640 1 -- 172.21.15.28:0/3162931987 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4e0007290 con 0x7fd4f006dbf0 2024-02-21T23:15:30.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:31.003+0000 7fd4efe00640 1 -- 172.21.15.28:0/3162931987 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4f01edd50 con 0x7fd4f006dbf0 2024-02-21T23:15:30.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:31.003+0000 7fd4efe00640 1 -- 172.21.15.28:0/3162931987 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4f0154c90 con 0x7fd4f006dbf0 2024-02-21T23:15:30.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:31.004+0000 7fd4efe00640 1 -- 172.21.15.28:0/3162931987 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4f004f290 con 0x7fd4f006dbf0 2024-02-21T23:15:30.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:31.005+0000 7fd4e7400640 1 -- 172.21.15.28:0/3162931987 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd4e001a070 con 0x7fd4f006dbf0 2024-02-21T23:15:30.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:31.007+0000 7fd4e7400640 1 --2- 172.21.15.28:0/3162931987 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd4d006b500 0x7fd4d006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:30.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:31.007+0000 7fd4e7400640 1 -- 172.21.15.28:0/3162931987 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fd4e00897e0 con 0x7fd4f006dbf0 2024-02-21T23:15:30.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:31.007+0000 7fd4ee000640 1 --2- 172.21.15.28:0/3162931987 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd4d006b500 0x7fd4d006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:30.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:31.008+0000 7fd4ee000640 1 --2- 172.21.15.28:0/3162931987 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd4d006b500 0x7fd4d006d9c0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc036000 tx=0x7fd4dc008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:30.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:31.012+0000 7fd4e7400640 1 -- 172.21.15.28:0/3162931987 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd4e00583c0 con 0x7fd4f006dbf0 2024-02-21T23:15:31.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:31.250+0000 7fd4efe00640 1 -- 172.21.15.28:0/3162931987 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd4f006d9d0 con 0x7fd4f006dbf0 2024-02-21T23:15:31.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:31.251+0000 7fd4e7400640 1 -- 172.21.15.28:0/3162931987 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7fd4e0057cc0 con 0x7fd4f006dbf0 2024-02-21T23:15:31.238 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:15:31.238 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:15:31.239 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:15:31.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:31.253+0000 7fd4efe00640 1 -- 172.21.15.28:0/3162931987 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd4d006b500 msgr2=0x7fd4d006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:31.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:31.253+0000 7fd4efe00640 1 --2- 172.21.15.28:0/3162931987 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd4d006b500 0x7fd4d006d9c0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc036000 tx=0x7fd4dc008030 comp rx=0 tx=0).stop 2024-02-21T23:15:31.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:31.253+0000 7fd4efe00640 1 -- 172.21.15.28:0/3162931987 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4f006dbf0 msgr2=0x7fd4f00be240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:31.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:31.253+0000 7fd4efe00640 1 --2- 172.21.15.28:0/3162931987 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4f006dbf0 0x7fd4f00be240 secure :-1 s=READY pgs=2558 cs=0 l=1 rev1=1 crypto rx=0x7fd4e000c460 tx=0x7fd4e000c920 comp rx=0 tx=0).stop 2024-02-21T23:15:31.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:31.253+0000 7fd4efe00640 1 -- 172.21.15.28:0/3162931987 shutdown_connections 2024-02-21T23:15:31.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:31.253+0000 7fd4efe00640 1 --2- 172.21.15.28:0/3162931987 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd4d006b500 0x7fd4d006d9c0 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:31.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:31.253+0000 7fd4efe00640 1 --2- 172.21.15.28:0/3162931987 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd4f00aff90 0x7fd4f0154750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:31.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:31.253+0000 7fd4efe00640 1 --2- 172.21.15.28:0/3162931987 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd4f0155960 0x7fd4f01ee550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:31.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:31.253+0000 7fd4efe00640 1 --2- 172.21.15.28:0/3162931987 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd4f006dbf0 0x7fd4f00be240 unknown :-1 s=CLOSED pgs=2558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:31.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:31.253+0000 7fd4efe00640 1 -- 172.21.15.28:0/3162931987 >> 172.21.15.28:0/3162931987 conn(0x7fd4f00ac370 msgr2=0x7fd4f0157040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:15:31.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:31.254+0000 7fd4efe00640 1 -- 172.21.15.28:0/3162931987 shutdown_connections 2024-02-21T23:15:31.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:31.254+0000 7fd4efe00640 1 -- 172.21.15.28:0/3162931987 wait complete. 2024-02-21T23:15:31.660 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:15:31.660 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:15:32.662 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:15:32.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.899+0000 7fc10f200640 1 Processor -- start 2024-02-21T23:15:32.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.899+0000 7fc10f200640 1 -- start start 2024-02-21T23:15:32.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.900+0000 7fc10f200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc1080aff90 0x7fc1080c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:32.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.900+0000 7fc10f200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fc1080c5720 0x7fc10815f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:32.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.900+0000 7fc10f200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1080c74c0 con 0x7fc10806dbf0 2024-02-21T23:15:32.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.901+0000 7fc10f200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1080c7270 con 0x7fc1080c5720 2024-02-21T23:15:32.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.901+0000 7fc10f200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc108158630 con 0x7fc1080aff90 2024-02-21T23:15:32.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.901+0000 7fc10d400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fc10806dbf0 0x7fc108158ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:40262/0 (socket says 172.21.15.28:40262) 2024-02-21T23:15:32.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.901+0000 7fc10d400640 1 -- 172.21.15.28:0/2878648481 learned_addr learned my addr 172.21.15.28:0/2878648481 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:15:32.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.901+0000 7fc10de00640 1 --2- 172.21.15.28:0/2878648481 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc1080aff90 0x7fc1080c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:32.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.901+0000 7fc10e800640 1 --2- 172.21.15.28:0/2878648481 >> v2:172.21.15.81:3300/0 conn(0x7fc1080c5720 0x7fc10815f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:32.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.901+0000 7fc10ca00640 1 -- 172.21.15.28:0/2878648481 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4109005082 0 0) 0x7fc1080c74c0 con 0x7fc10806dbf0 2024-02-21T23:15:32.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.902+0000 7fc10ca00640 1 -- 172.21.15.28:0/2878648481 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0f4003520 con 0x7fc10806dbf0 2024-02-21T23:15:32.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.902+0000 7fc10ca00640 1 -- 172.21.15.28:0/2878648481 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1427982927 0 0) 0x7fc0f4003520 con 0x7fc10806dbf0 2024-02-21T23:15:32.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.902+0000 7fc10ca00640 1 -- 172.21.15.28:0/2878648481 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc1080aff90 msgr2=0x7fc1080c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:15:32.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.902+0000 7fc10ca00640 1 --2- 172.21.15.28:0/2878648481 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc1080aff90 0x7fc1080c51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:32.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.902+0000 7fc10ca00640 1 -- 172.21.15.28:0/2878648481 >> v2:172.21.15.81:3300/0 conn(0x7fc1080c5720 msgr2=0x7fc10815f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:15:32.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.902+0000 7fc10ca00640 1 --2- 172.21.15.28:0/2878648481 >> v2:172.21.15.81:3300/0 conn(0x7fc1080c5720 0x7fc10815f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:32.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.902+0000 7fc10ca00640 1 -- 172.21.15.28:0/2878648481 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1080c77a0 con 0x7fc10806dbf0 2024-02-21T23:15:32.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.902+0000 7fc10ca00640 1 -- 172.21.15.28:0/2878648481 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fc0f8001670 con 0x7fc10806dbf0 2024-02-21T23:15:32.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.902+0000 7fc10ca00640 1 -- 172.21.15.28:0/2878648481 >> v1:172.21.15.28:6789/0 conn(0x7fc10806dbf0 legacy=0x7fc108158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:32.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.902+0000 7fc10ca00640 1 --2- 172.21.15.28:0/2878648481 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc1080aff90 0x7fc0f40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:32.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.902+0000 7fc10ca00640 1 --2- 172.21.15.28:0/2878648481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0f4003f80 0x7fc0f4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:32.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.902+0000 7fc10ca00640 1 --2- 172.21.15.28:0/2878648481 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0f4004d40 0x7fc0f40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:32.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.902+0000 7fc10ca00640 1 -- 172.21.15.28:0/2878648481 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1080c7270 con 0x7fc0f4003f80 2024-02-21T23:15:32.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.902+0000 7fc10ca00640 1 -- 172.21.15.28:0/2878648481 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc108158630 con 0x7fc0f4004d40 2024-02-21T23:15:32.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.902+0000 7fc10ca00640 1 -- 172.21.15.28:0/2878648481 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f400b9c0 con 0x7fc1080aff90 2024-02-21T23:15:32.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.902+0000 7fc10d400640 1 --2- 172.21.15.28:0/2878648481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0f4003f80 0x7fc0f4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:32.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.903+0000 7fc10d400640 1 -- 172.21.15.28:0/2878648481 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc1080aff90 msgr2=0x7fc0f40039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:15:32.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.903+0000 7fc10d400640 1 --2- 172.21.15.28:0/2878648481 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc1080aff90 0x7fc0f40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:32.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.903+0000 7fc10d400640 1 -- 172.21.15.28:0/2878648481 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0f4004d40 msgr2=0x7fc0f40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:15:32.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.903+0000 7fc10d400640 1 --2- 172.21.15.28:0/2878648481 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0f4004d40 0x7fc0f40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:32.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.903+0000 7fc10d400640 1 -- 172.21.15.28:0/2878648481 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0f4003520 con 0x7fc0f4003f80 2024-02-21T23:15:32.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.904+0000 7fc10d400640 1 --2- 172.21.15.28:0/2878648481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0f4003f80 0x7fc0f4004800 secure :-1 s=READY pgs=2560 cs=0 l=1 rev1=1 crypto rx=0x7fc0f800c1c0 tx=0x7fc0f4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbb83456c60d254 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:32.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.904+0000 7fc10ca00640 1 -- 172.21.15.28:0/2878648481 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0f8007c80 con 0x7fc0f4003f80 2024-02-21T23:15:32.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.904+0000 7fc10ca00640 1 -- 172.21.15.28:0/2878648481 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0f8030070 con 0x7fc0f4003f80 2024-02-21T23:15:32.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.904+0000 7fc10ca00640 1 -- 172.21.15.28:0/2878648481 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0f8005d70 con 0x7fc0f4003f80 2024-02-21T23:15:32.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.905+0000 7fc10f200640 1 -- 172.21.15.28:0/2878648481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0f4003f80 msgr2=0x7fc0f4004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:32.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.905+0000 7fc10f200640 1 --2- 172.21.15.28:0/2878648481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0f4003f80 0x7fc0f4004800 secure :-1 s=READY pgs=2560 cs=0 l=1 rev1=1 crypto rx=0x7fc0f800c1c0 tx=0x7fc0f4005830 comp rx=0 tx=0).stop 2024-02-21T23:15:32.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.905+0000 7fc10e800640 1 -- 172.21.15.28:0/2878648481 reap_dead start 2024-02-21T23:15:32.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.905+0000 7fc10f200640 1 -- 172.21.15.28:0/2878648481 shutdown_connections 2024-02-21T23:15:32.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.905+0000 7fc10f200640 1 -- 172.21.15.28:0/2878648481 >> 172.21.15.28:0/2878648481 conn(0x7fc1080ac370 msgr2=0x7fc1080ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:15:32.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.905+0000 7fc10f200640 1 -- 172.21.15.28:0/2878648481 shutdown_connections 2024-02-21T23:15:32.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.905+0000 7fc10f200640 1 -- 172.21.15.28:0/2878648481 wait complete. 2024-02-21T23:15:32.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.907+0000 7fc10f200640 1 Processor -- start 2024-02-21T23:15:32.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.907+0000 7fc10f200640 1 -- start start 2024-02-21T23:15:32.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.908+0000 7fc10f200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0f4003f80 0x7fc1080c1220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:32.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.908+0000 7fc10f200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1081519d0 0x7fc108148680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:32.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.908+0000 7fc10f200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc108148bc0 0x7fc1081ebb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:32.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.908+0000 7fc10f200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1080c8760 con 0x7fc1081519d0 2024-02-21T23:15:32.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.908+0000 7fc10f200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1080be0c0 con 0x7fc0f4003f80 2024-02-21T23:15:32.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.908+0000 7fc10f200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1080c88e0 con 0x7fc108148bc0 2024-02-21T23:15:32.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.908+0000 7fc10d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1081519d0 0x7fc108148680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:32.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.908+0000 7fc10d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1081519d0 0x7fc108148680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44582/0 (socket says 172.21.15.28:44582) 2024-02-21T23:15:32.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.908+0000 7fc10d400640 1 -- 172.21.15.28:0/3662387383 learned_addr learned my addr 172.21.15.28:0/3662387383 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:15:32.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.909+0000 7fc10e800640 1 --2- 172.21.15.28:0/3662387383 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc108148bc0 0x7fc1081ebb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:32.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.909+0000 7fc10de00640 1 --2- 172.21.15.28:0/3662387383 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0f4003f80 0x7fc1080c1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:32.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.909+0000 7fc10d400640 1 -- 172.21.15.28:0/3662387383 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc108148bc0 msgr2=0x7fc1081ebb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:32.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.909+0000 7fc10d400640 1 --2- 172.21.15.28:0/3662387383 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc108148bc0 0x7fc1081ebb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:32.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.909+0000 7fc10d400640 1 -- 172.21.15.28:0/3662387383 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0f4003f80 msgr2=0x7fc1080c1220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:32.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.909+0000 7fc10d400640 1 --2- 172.21.15.28:0/3662387383 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0f4003f80 0x7fc1080c1220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:32.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.909+0000 7fc10d400640 1 -- 172.21.15.28:0/3662387383 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc108154980 con 0x7fc1081519d0 2024-02-21T23:15:32.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.909+0000 7fc10d400640 1 --2- 172.21.15.28:0/3662387383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1081519d0 0x7fc108148680 secure :-1 s=READY pgs=2561 cs=0 l=1 rev1=1 crypto rx=0x7fc0f800c160 tx=0x7fc0f8001970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:32.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.910+0000 7fc0fe000640 1 -- 172.21.15.28:0/3662387383 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0f8001bc0 con 0x7fc1081519d0 2024-02-21T23:15:32.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.910+0000 7fc0fe000640 1 -- 172.21.15.28:0/3662387383 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0f8004c10 con 0x7fc1081519d0 2024-02-21T23:15:32.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.910+0000 7fc0fe000640 1 -- 172.21.15.28:0/3662387383 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0f8007070 con 0x7fc1081519d0 2024-02-21T23:15:32.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.910+0000 7fc10f200640 1 -- 172.21.15.28:0/3662387383 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1081ee9e0 con 0x7fc1081519d0 2024-02-21T23:15:32.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.910+0000 7fc10f200640 1 -- 172.21.15.28:0/3662387383 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc108149040 con 0x7fc1081519d0 2024-02-21T23:15:32.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.911+0000 7fc10f200640 1 -- 172.21.15.28:0/3662387383 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc10804f290 con 0x7fc1081519d0 2024-02-21T23:15:32.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.912+0000 7fc0fe000640 1 -- 172.21.15.28:0/3662387383 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc0f8007210 con 0x7fc1081519d0 2024-02-21T23:15:32.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.913+0000 7fc0fe000640 1 --2- 172.21.15.28:0/3662387383 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc0ec06b4d0 0x7fc0ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:32.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.914+0000 7fc0fe000640 1 -- 172.21.15.28:0/3662387383 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fc0f80ab6a0 con 0x7fc1081519d0 2024-02-21T23:15:32.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.914+0000 7fc10de00640 1 --2- 172.21.15.28:0/3662387383 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc0ec06b4d0 0x7fc0ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:32.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.915+0000 7fc10de00640 1 --2- 172.21.15.28:0/3662387383 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc0ec06b4d0 0x7fc0ec06d990 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fc1040072b0 tx=0x7fc10400d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:32.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:32.918+0000 7fc0fe000640 1 -- 172.21.15.28:0/3662387383 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc0f807a400 con 0x7fc1081519d0 2024-02-21T23:15:33.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:33.167+0000 7fc10f200640 1 -- 172.21.15.28:0/3662387383 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc1080c19e0 con 0x7fc1081519d0 2024-02-21T23:15:33.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:33.168+0000 7fc0fe000640 1 -- 172.21.15.28:0/3662387383 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7fc0f8079d00 con 0x7fc1081519d0 2024-02-21T23:15:33.155 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:15:33.156 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:15:33.156 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:15:33.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:33.171+0000 7fc10f200640 1 -- 172.21.15.28:0/3662387383 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc0ec06b4d0 msgr2=0x7fc0ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:33.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:33.171+0000 7fc10f200640 1 --2- 172.21.15.28:0/3662387383 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc0ec06b4d0 0x7fc0ec06d990 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fc1040072b0 tx=0x7fc10400d040 comp rx=0 tx=0).stop 2024-02-21T23:15:33.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:33.171+0000 7fc10f200640 1 -- 172.21.15.28:0/3662387383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1081519d0 msgr2=0x7fc108148680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:33.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:33.171+0000 7fc10f200640 1 --2- 172.21.15.28:0/3662387383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1081519d0 0x7fc108148680 secure :-1 s=READY pgs=2561 cs=0 l=1 rev1=1 crypto rx=0x7fc0f800c160 tx=0x7fc0f8001970 comp rx=0 tx=0).stop 2024-02-21T23:15:33.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:33.171+0000 7fc10f200640 1 -- 172.21.15.28:0/3662387383 shutdown_connections 2024-02-21T23:15:33.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:33.171+0000 7fc10f200640 1 --2- 172.21.15.28:0/3662387383 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc0ec06b4d0 0x7fc0ec06d990 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:33.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:33.171+0000 7fc10f200640 1 --2- 172.21.15.28:0/3662387383 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc108148bc0 0x7fc1081ebb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:33.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:33.171+0000 7fc10f200640 1 --2- 172.21.15.28:0/3662387383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1081519d0 0x7fc108148680 unknown :-1 s=CLOSED pgs=2561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:33.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:33.171+0000 7fc10f200640 1 --2- 172.21.15.28:0/3662387383 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc0f4003f80 0x7fc1080c1220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:33.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:33.171+0000 7fc10f200640 1 -- 172.21.15.28:0/3662387383 >> 172.21.15.28:0/3662387383 conn(0x7fc1080ac370 msgr2=0x7fc1080ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:15:33.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:33.172+0000 7fc10f200640 1 -- 172.21.15.28:0/3662387383 shutdown_connections 2024-02-21T23:15:33.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:33.172+0000 7fc10f200640 1 -- 172.21.15.28:0/3662387383 wait complete. 2024-02-21T23:15:33.585 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:15:33.585 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:15:34.586 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:15:34.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:34.825+0000 7f2ffc200640 1 Processor -- start 2024-02-21T23:15:34.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:34.826+0000 7f2ffc200640 1 -- start start 2024-02-21T23:15:34.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:34.827+0000 7f2ffc200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f2ff406ecb0 0x7f2ff4156db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:34.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:34.827+0000 7f2ffc200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2ff406f090 0x7f2ff40c2330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:34.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:34.827+0000 7f2ffc200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2ff40c2870 0x7f2ff40a8a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:34.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:34.827+0000 7f2ffc200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ff4150b60 con 0x7f2ff40c2870 2024-02-21T23:15:34.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:34.827+0000 7f2ffc200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ff406f6b0 con 0x7f2ff406ecb0 2024-02-21T23:15:34.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:34.827+0000 7f2ffc200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff406f990 con 0x7f2ff406f090 2024-02-21T23:15:34.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:34.828+0000 7f2ffa200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2ff40c2870 0x7f2ff40a8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:34.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:34.828+0000 7f2ffa200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2ff40c2870 0x7f2ff40a8a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44590/0 (socket says 172.21.15.28:44590) 2024-02-21T23:15:34.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:34.828+0000 7f2ffa200640 1 -- 172.21.15.28:0/3904266859 learned_addr learned my addr 172.21.15.28:0/3904266859 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:15:34.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:34.828+0000 7f2ff8e00640 1 --2- 172.21.15.28:0/3904266859 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2ff406f090 0x7f2ff40c2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:34.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:34.828+0000 7f2ff9800640 1 --2- 172.21.15.28:0/3904266859 >> v2:172.21.15.81:3300/0 conn(0x7f2ff406ecb0 0x7f2ff4156db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:34.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:34.828+0000 7f2ffa200640 1 -- 172.21.15.28:0/3904266859 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2ff406f090 msgr2=0x7f2ff40c2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:15:34.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:34.828+0000 7f2ffa200640 1 --2- 172.21.15.28:0/3904266859 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2ff406f090 0x7f2ff40c2330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:34.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:34.828+0000 7f2ffa200640 1 -- 172.21.15.28:0/3904266859 >> v2:172.21.15.81:3300/0 conn(0x7f2ff406ecb0 msgr2=0x7f2ff4156db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:15:34.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:34.828+0000 7f2ffa200640 1 --2- 172.21.15.28:0/3904266859 >> v2:172.21.15.81:3300/0 conn(0x7f2ff406ecb0 0x7f2ff4156db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:34.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:34.828+0000 7f2ffa200640 1 -- 172.21.15.28:0/3904266859 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ff406fc70 con 0x7f2ff40c2870 2024-02-21T23:15:34.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:34.830+0000 7f2ffa200640 1 --2- 172.21.15.28:0/3904266859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ff40c2870 0x7f2ff40a8a70 secure :-1 s=READY pgs=2562 cs=0 l=1 rev1=1 crypto rx=0x7f2ff4067290 tx=0x7f2fec02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79298ada4811b730 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:34.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:34.830+0000 7f2ff3e00640 1 -- 172.21.15.28:0/3904266859 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fec02ca70 con 0x7f2ff40c2870 2024-02-21T23:15:34.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:34.830+0000 7f2ff3e00640 1 -- 172.21.15.28:0/3904266859 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fec02cc10 con 0x7f2ff40c2870 2024-02-21T23:15:34.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:34.830+0000 7f2ff3e00640 1 -- 172.21.15.28:0/3904266859 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fec0343f0 con 0x7f2ff40c2870 2024-02-21T23:15:34.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:34.831+0000 7f2ffc200640 1 -- 172.21.15.28:0/3904266859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ff40c2870 msgr2=0x7f2ff40a8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:34.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:34.831+0000 7f2ffc200640 1 --2- 172.21.15.28:0/3904266859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ff40c2870 0x7f2ff40a8a70 secure :-1 s=READY pgs=2562 cs=0 l=1 rev1=1 crypto rx=0x7f2ff4067290 tx=0x7f2fec02bed0 comp rx=0 tx=0).stop 2024-02-21T23:15:34.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:34.831+0000 7f2ffc200640 1 -- 172.21.15.28:0/3904266859 shutdown_connections 2024-02-21T23:15:34.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:34.831+0000 7f2ffc200640 1 --2- 172.21.15.28:0/3904266859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ff40c2870 0x7f2ff40a8a70 unknown :-1 s=CLOSED pgs=2562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:34.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:34.831+0000 7f2ffc200640 1 --2- 172.21.15.28:0/3904266859 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2ff406f090 0x7f2ff40c2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:34.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:34.831+0000 7f2ffc200640 1 --2- 172.21.15.28:0/3904266859 >> v2:172.21.15.81:3300/0 conn(0x7f2ff406ecb0 0x7f2ff4156db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:34.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:34.831+0000 7f2ffc200640 1 -- 172.21.15.28:0/3904266859 >> 172.21.15.28:0/3904266859 conn(0x7f2ff40c3bb0 msgr2=0x7f2ff40c3fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:15:34.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:34.831+0000 7f2ffc200640 1 -- 172.21.15.28:0/3904266859 shutdown_connections 2024-02-21T23:15:34.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:34.831+0000 7f2ffc200640 1 -- 172.21.15.28:0/3904266859 wait complete. 2024-02-21T23:15:34.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:34.832+0000 7f2ffc200640 1 Processor -- start 2024-02-21T23:15:34.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:34.833+0000 7f2ffc200640 1 -- start start 2024-02-21T23:15:34.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:34.833+0000 7f2ffc200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ff406ecb0 0x7f2ff41ee990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:34.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:34.833+0000 7f2ffc200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2ff406f090 0x7f2ff40aabf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:34.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:34.833+0000 7f2ffc200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ff40c2870 0x7f2ff41eb0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:34.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:34.834+0000 7f2ffc200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff406e210 con 0x7f2ff406ecb0 2024-02-21T23:15:34.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:34.834+0000 7f2ffc200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff40a9e20 con 0x7f2ff40c2870 2024-02-21T23:15:34.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:34.834+0000 7f2ffc200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff406e390 con 0x7f2ff406f090 2024-02-21T23:15:34.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:34.834+0000 7f2ff9800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ff406ecb0 0x7f2ff41ee990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:34.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:34.834+0000 7f2ff9800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ff406ecb0 0x7f2ff41ee990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44604/0 (socket says 172.21.15.28:44604) 2024-02-21T23:15:34.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:34.834+0000 7f2ff9800640 1 -- 172.21.15.28:0/803227909 learned_addr learned my addr 172.21.15.28:0/803227909 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:15:34.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:34.834+0000 7f2ff9800640 1 -- 172.21.15.28:0/803227909 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2ff406f090 msgr2=0x7f2ff40aabf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:34.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:34.834+0000 7f2ff9800640 1 --2- 172.21.15.28:0/803227909 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2ff406f090 0x7f2ff40aabf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:34.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:34.834+0000 7f2ff9800640 1 -- 172.21.15.28:0/803227909 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ff40c2870 msgr2=0x7f2ff41eb0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:34.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:34.834+0000 7f2ffa200640 1 --2- 172.21.15.28:0/803227909 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ff40c2870 0x7f2ff41eb0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:34.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:34.834+0000 7f2ff9800640 1 --2- 172.21.15.28:0/803227909 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ff40c2870 0x7f2ff41eb0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:34.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:34.834+0000 7f2ff9800640 1 -- 172.21.15.28:0/803227909 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ff414dad0 con 0x7f2ff406ecb0 2024-02-21T23:15:34.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:34.835+0000 7f2ff9800640 1 --2- 172.21.15.28:0/803227909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ff406ecb0 0x7f2ff41ee990 secure :-1 s=READY pgs=2563 cs=0 l=1 rev1=1 crypto rx=0x7f2fe800d800 tx=0x7f2fe800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:34.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:34.835+0000 7f2ff1600640 1 -- 172.21.15.28:0/803227909 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe8018020 con 0x7f2ff406ecb0 2024-02-21T23:15:34.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:34.835+0000 7f2ff1600640 1 -- 172.21.15.28:0/803227909 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fe80096a0 con 0x7f2ff406ecb0 2024-02-21T23:15:34.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:34.835+0000 7f2ff1600640 1 -- 172.21.15.28:0/803227909 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe8009a70 con 0x7f2ff406ecb0 2024-02-21T23:15:34.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:34.835+0000 7f2ffc200640 1 -- 172.21.15.28:0/803227909 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ff414ddb0 con 0x7f2ff406ecb0 2024-02-21T23:15:34.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:34.835+0000 7f2ffc200640 1 -- 172.21.15.28:0/803227909 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ff40ab130 con 0x7f2ff406ecb0 2024-02-21T23:15:34.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:34.836+0000 7f2ffc200640 1 -- 172.21.15.28:0/803227909 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ff4074ab0 con 0x7f2ff406ecb0 2024-02-21T23:15:34.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:34.838+0000 7f2ff1600640 1 -- 172.21.15.28:0/803227909 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2fe801b020 con 0x7f2ff406ecb0 2024-02-21T23:15:34.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:34.838+0000 7f2ff1600640 1 --2- 172.21.15.28:0/803227909 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2fd406b4d0 0x7f2fd406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:34.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:34.839+0000 7f2ff1600640 1 -- 172.21.15.28:0/803227909 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f2fe808aa00 con 0x7f2ff406ecb0 2024-02-21T23:15:34.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:34.839+0000 7f2ff8e00640 1 --2- 172.21.15.28:0/803227909 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2fd406b4d0 0x7f2fd406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:34.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:34.840+0000 7f2ff8e00640 1 --2- 172.21.15.28:0/803227909 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2fd406b4d0 0x7f2fd406d990 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f2ff40abe20 tx=0x7f2fe0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:34.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:34.844+0000 7f2ff1600640 1 -- 172.21.15.28:0/803227909 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f2ff4074ab0 con 0x7f2ff406ecb0 2024-02-21T23:15:35.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:35.083+0000 7f2ffc200640 1 -- 172.21.15.28:0/803227909 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2ff4068640 con 0x7f2ff406ecb0 2024-02-21T23:15:35.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:35.084+0000 7f2ff1600640 1 -- 172.21.15.28:0/803227909 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7f2fe8022020 con 0x7f2ff406ecb0 2024-02-21T23:15:35.072 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:15:35.072 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:15:35.072 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:15:35.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:35.087+0000 7f2ffc200640 1 -- 172.21.15.28:0/803227909 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2fd406b4d0 msgr2=0x7f2fd406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:35.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:35.087+0000 7f2ffc200640 1 --2- 172.21.15.28:0/803227909 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2fd406b4d0 0x7f2fd406d990 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f2ff40abe20 tx=0x7f2fe0008040 comp rx=0 tx=0).stop 2024-02-21T23:15:35.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:35.087+0000 7f2ffc200640 1 -- 172.21.15.28:0/803227909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ff406ecb0 msgr2=0x7f2ff41ee990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:35.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:35.087+0000 7f2ffc200640 1 --2- 172.21.15.28:0/803227909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ff406ecb0 0x7f2ff41ee990 secure :-1 s=READY pgs=2563 cs=0 l=1 rev1=1 crypto rx=0x7f2fe800d800 tx=0x7f2fe800dcc0 comp rx=0 tx=0).stop 2024-02-21T23:15:35.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:35.087+0000 7f2ffc200640 1 -- 172.21.15.28:0/803227909 shutdown_connections 2024-02-21T23:15:35.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:35.087+0000 7f2ffc200640 1 --2- 172.21.15.28:0/803227909 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2fd406b4d0 0x7f2fd406d990 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:35.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:35.087+0000 7f2ffc200640 1 --2- 172.21.15.28:0/803227909 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2ff406f090 0x7f2ff40aabf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:35.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:35.087+0000 7f2ffc200640 1 --2- 172.21.15.28:0/803227909 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ff40c2870 0x7f2ff41eb0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:35.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:35.087+0000 7f2ffc200640 1 --2- 172.21.15.28:0/803227909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ff406ecb0 0x7f2ff41ee990 unknown :-1 s=CLOSED pgs=2563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:35.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:35.087+0000 7f2ffc200640 1 -- 172.21.15.28:0/803227909 >> 172.21.15.28:0/803227909 conn(0x7f2ff40c3bb0 msgr2=0x7f2ff40c3f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:15:35.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:35.087+0000 7f2ffc200640 1 -- 172.21.15.28:0/803227909 shutdown_connections 2024-02-21T23:15:35.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:35.088+0000 7f2ffc200640 1 -- 172.21.15.28:0/803227909 wait complete. 2024-02-21T23:15:35.501 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:15:35.502 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:15:36.503 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:15:36.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.735+0000 7f59a1800640 1 Processor -- start 2024-02-21T23:15:36.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.736+0000 7f59a1800640 1 -- start start 2024-02-21T23:15:36.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.737+0000 7f59a1800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f599c155960 0x7f599c1608a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:36.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.737+0000 7f59a1800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f599c0c74c0 con 0x7f599c06dbf0 2024-02-21T23:15:36.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.737+0000 7f59a1800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f599c0c77a0 con 0x7f599c0aff90 2024-02-21T23:15:36.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.737+0000 7f59a1800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f599c0c7270 con 0x7f599c155960 2024-02-21T23:15:36.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.738+0000 7f599b400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f599c06dbf0 0x7f599c158ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42800/0 (socket says 172.21.15.28:42800) 2024-02-21T23:15:36.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.738+0000 7f599b400640 1 -- 172.21.15.28:0/3437713463 learned_addr learned my addr 172.21.15.28:0/3437713463 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:15:36.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.738+0000 7f59a0e00640 1 --2- 172.21.15.28:0/3437713463 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f599c155960 0x7f599c1608a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:36.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.738+0000 7f599aa00640 1 -- 172.21.15.28:0/3437713463 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 538983461 0 0) 0x7f599c0c74c0 con 0x7f599c06dbf0 2024-02-21T23:15:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.738+0000 7f599aa00640 1 -- 172.21.15.28:0/3437713463 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5980003520 con 0x7f599c06dbf0 2024-02-21T23:15:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.739+0000 7f599aa00640 1 -- 172.21.15.28:0/3437713463 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3099219789 0 0) 0x7f5980003520 con 0x7f599c06dbf0 2024-02-21T23:15:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.739+0000 7f599aa00640 1 -- 172.21.15.28:0/3437713463 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f599c155960 msgr2=0x7f599c1608a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:15:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.739+0000 7f599aa00640 1 --2- 172.21.15.28:0/3437713463 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f599c155960 0x7f599c1608a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.739+0000 7f599aa00640 1 -- 172.21.15.28:0/3437713463 >> v1:172.21.15.81:6789/0 conn(0x7f599c0aff90 legacy=0x7f599c0c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.739+0000 7f599aa00640 1 -- 172.21.15.28:0/3437713463 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f599c0c7a80 con 0x7f599c06dbf0 2024-02-21T23:15:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.739+0000 7f599aa00640 1 -- 172.21.15.28:0/3437713463 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f5984001670 con 0x7f599c06dbf0 2024-02-21T23:15:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.739+0000 7f599aa00640 1 -- 172.21.15.28:0/3437713463 >> v1:172.21.15.28:6789/0 conn(0x7f599c06dbf0 legacy=0x7f599c158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.739+0000 7f599aa00640 1 --2- 172.21.15.28:0/3437713463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59800039e0 0x7f5980003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.739+0000 7f599aa00640 1 --2- 172.21.15.28:0/3437713463 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5980004800 0x7f5980004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.739+0000 7f599aa00640 1 --2- 172.21.15.28:0/3437713463 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f599c155960 0x7f59800051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.739+0000 7f599aa00640 1 -- 172.21.15.28:0/3437713463 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f599c0c7270 con 0x7f59800039e0 2024-02-21T23:15:36.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.739+0000 7f599aa00640 1 -- 172.21.15.28:0/3437713463 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f598000ba70 con 0x7f5980004800 2024-02-21T23:15:36.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.739+0000 7f599aa00640 1 -- 172.21.15.28:0/3437713463 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f598000bc70 con 0x7f599c155960 2024-02-21T23:15:36.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.739+0000 7f599be00640 1 --2- 172.21.15.28:0/3437713463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59800039e0 0x7f5980003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:36.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.739+0000 7f599b400640 1 --2- 172.21.15.28:0/3437713463 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5980004800 0x7f5980004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:36.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.740+0000 7f59a0e00640 1 --2- 172.21.15.28:0/3437713463 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f599c155960 0x7f59800051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:36.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.740+0000 7f599be00640 1 -- 172.21.15.28:0/3437713463 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f599c155960 msgr2=0x7f59800051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:15:36.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.740+0000 7f599be00640 1 --2- 172.21.15.28:0/3437713463 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f599c155960 0x7f59800051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:36.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.740+0000 7f599be00640 1 -- 172.21.15.28:0/3437713463 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5980004800 msgr2=0x7f5980004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:15:36.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.740+0000 7f599be00640 1 --2- 172.21.15.28:0/3437713463 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5980004800 0x7f5980004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:36.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.740+0000 7f599be00640 1 -- 172.21.15.28:0/3437713463 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5980003520 con 0x7f59800039e0 2024-02-21T23:15:36.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.741+0000 7f599be00640 1 --2- 172.21.15.28:0/3437713463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59800039e0 0x7f5980003e00 secure :-1 s=READY pgs=2565 cs=0 l=1 rev1=1 crypto rx=0x7f598c001720 tx=0x7f5980005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28d79f5824dec4a1 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:36.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.741+0000 7f599aa00640 1 -- 172.21.15.28:0/3437713463 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f598c001370 con 0x7f59800039e0 2024-02-21T23:15:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.741+0000 7f599aa00640 1 -- 172.21.15.28:0/3437713463 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f598c0052d0 con 0x7f59800039e0 2024-02-21T23:15:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.741+0000 7f599aa00640 1 -- 172.21.15.28:0/3437713463 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f598c003830 con 0x7f59800039e0 2024-02-21T23:15:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.742+0000 7f59a1800640 1 -- 172.21.15.28:0/3437713463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59800039e0 msgr2=0x7f5980003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.742+0000 7f59a1800640 1 --2- 172.21.15.28:0/3437713463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59800039e0 0x7f5980003e00 secure :-1 s=READY pgs=2565 cs=0 l=1 rev1=1 crypto rx=0x7f598c001720 tx=0x7f5980005830 comp rx=0 tx=0).stop 2024-02-21T23:15:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.742+0000 7f59a0e00640 1 -- 172.21.15.28:0/3437713463 reap_dead start 2024-02-21T23:15:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.742+0000 7f59a1800640 1 -- 172.21.15.28:0/3437713463 shutdown_connections 2024-02-21T23:15:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.742+0000 7f59a1800640 1 -- 172.21.15.28:0/3437713463 >> 172.21.15.28:0/3437713463 conn(0x7f599c0ac370 msgr2=0x7f599c0ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:15:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.742+0000 7f59a1800640 1 -- 172.21.15.28:0/3437713463 shutdown_connections 2024-02-21T23:15:36.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.742+0000 7f59a1800640 1 -- 172.21.15.28:0/3437713463 wait complete. 2024-02-21T23:15:36.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.745+0000 7f59a1800640 1 Processor -- start 2024-02-21T23:15:36.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.745+0000 7f59a1800640 1 -- start start 2024-02-21T23:15:36.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.747+0000 7f59a1800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f59800039e0 0x7f599c1ee9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:36.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.747+0000 7f59a1800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f599c1eb460 0x7f599c0be350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:36.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.747+0000 7f59a1800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f599c0aeb80 0x7f599c1518c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:36.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.747+0000 7f59a1800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f599c0c58f0 con 0x7f599c1eb460 2024-02-21T23:15:36.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.747+0000 7f59a1800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f599c158630 con 0x7f59800039e0 2024-02-21T23:15:36.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.747+0000 7f59a1800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f599c0c5a70 con 0x7f599c0aeb80 2024-02-21T23:15:36.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.747+0000 7f599b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f599c1eb460 0x7f599c0be350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:36.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.747+0000 7f599b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f599c1eb460 0x7f599c0be350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55232/0 (socket says 172.21.15.28:55232) 2024-02-21T23:15:36.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.747+0000 7f599b400640 1 -- 172.21.15.28:0/1215609889 learned_addr learned my addr 172.21.15.28:0/1215609889 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:15:36.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.747+0000 7f599be00640 1 --2- 172.21.15.28:0/1215609889 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f59800039e0 0x7f599c1ee9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:36.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.747+0000 7f59a0e00640 1 --2- 172.21.15.28:0/1215609889 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f599c0aeb80 0x7f599c1518c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:36.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.748+0000 7f599b400640 1 -- 172.21.15.28:0/1215609889 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f599c0aeb80 msgr2=0x7f599c1518c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:36.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.748+0000 7f599b400640 1 --2- 172.21.15.28:0/1215609889 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f599c0aeb80 0x7f599c1518c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:36.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.748+0000 7f599b400640 1 -- 172.21.15.28:0/1215609889 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f59800039e0 msgr2=0x7f599c1ee9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:36.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.748+0000 7f599b400640 1 --2- 172.21.15.28:0/1215609889 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f59800039e0 0x7f599c1ee9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:36.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.748+0000 7f599b400640 1 -- 172.21.15.28:0/1215609889 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f599c0be890 con 0x7f599c1eb460 2024-02-21T23:15:36.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.748+0000 7f599be00640 1 --2- 172.21.15.28:0/1215609889 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f59800039e0 0x7f599c1ee9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:15:36.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.748+0000 7f599b400640 1 --2- 172.21.15.28:0/1215609889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f599c1eb460 0x7f599c0be350 secure :-1 s=READY pgs=2566 cs=0 l=1 rev1=1 crypto rx=0x7f5984001200 tx=0x7f598400eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:36.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.748+0000 7f5993e00640 1 -- 172.21.15.28:0/1215609889 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f598400c6c0 con 0x7f599c1eb460 2024-02-21T23:15:36.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.748+0000 7f5993e00640 1 -- 172.21.15.28:0/1215609889 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5984018070 con 0x7f599c1eb460 2024-02-21T23:15:36.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.748+0000 7f5993e00640 1 -- 172.21.15.28:0/1215609889 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f598400cac0 con 0x7f599c1eb460 2024-02-21T23:15:36.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.748+0000 7f59a1800640 1 -- 172.21.15.28:0/1215609889 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f599c0c3960 con 0x7f599c1eb460 2024-02-21T23:15:36.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.749+0000 7f59a1800640 1 -- 172.21.15.28:0/1215609889 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f599c1eac60 con 0x7f599c1eb460 2024-02-21T23:15:36.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.750+0000 7f59a1800640 1 -- 172.21.15.28:0/1215609889 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f599c0c3e80 con 0x7f599c1eb460 2024-02-21T23:15:36.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.751+0000 7f5993e00640 1 -- 172.21.15.28:0/1215609889 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f598401d460 con 0x7f599c1eb460 2024-02-21T23:15:36.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.752+0000 7f5993e00640 1 --2- 172.21.15.28:0/1215609889 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f597806b500 0x7f597806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:36.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.752+0000 7f5993e00640 1 -- 172.21.15.28:0/1215609889 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f598408a530 con 0x7f599c1eb460 2024-02-21T23:15:36.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.753+0000 7f599be00640 1 --2- 172.21.15.28:0/1215609889 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f597806b500 0x7f597806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:36.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.754+0000 7f599be00640 1 --2- 172.21.15.28:0/1215609889 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f597806b500 0x7f597806d9c0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f598c033000 tx=0x7f598c009030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:36.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:36.757+0000 7f5993e00640 1 -- 172.21.15.28:0/1215609889 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f599c0c3e80 con 0x7f599c1eb460 2024-02-21T23:15:36.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:37.004+0000 7f59a1800640 1 -- 172.21.15.28:0/1215609889 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f599c1eaa50 con 0x7f599c1eb460 2024-02-21T23:15:36.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:37.005+0000 7f5993e00640 1 -- 172.21.15.28:0/1215609889 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7f5984059110 con 0x7f599c1eb460 2024-02-21T23:15:36.992 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:15:36.992 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:15:36.993 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:15:36.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:37.008+0000 7f59a1800640 1 -- 172.21.15.28:0/1215609889 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f597806b500 msgr2=0x7f597806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:36.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:37.008+0000 7f59a1800640 1 --2- 172.21.15.28:0/1215609889 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f597806b500 0x7f597806d9c0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f598c033000 tx=0x7f598c009030 comp rx=0 tx=0).stop 2024-02-21T23:15:36.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:37.008+0000 7f59a1800640 1 -- 172.21.15.28:0/1215609889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f599c1eb460 msgr2=0x7f599c0be350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:36.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:37.008+0000 7f59a1800640 1 --2- 172.21.15.28:0/1215609889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f599c1eb460 0x7f599c0be350 secure :-1 s=READY pgs=2566 cs=0 l=1 rev1=1 crypto rx=0x7f5984001200 tx=0x7f598400eb90 comp rx=0 tx=0).stop 2024-02-21T23:15:36.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:37.008+0000 7f59a1800640 1 -- 172.21.15.28:0/1215609889 shutdown_connections 2024-02-21T23:15:36.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:37.008+0000 7f59a1800640 1 --2- 172.21.15.28:0/1215609889 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f597806b500 0x7f597806d9c0 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:36.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:37.008+0000 7f59a1800640 1 --2- 172.21.15.28:0/1215609889 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f599c0aeb80 0x7f599c1518c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:36.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:37.008+0000 7f59a1800640 1 --2- 172.21.15.28:0/1215609889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f599c1eb460 0x7f599c0be350 unknown :-1 s=CLOSED pgs=2566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:36.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:37.008+0000 7f59a1800640 1 --2- 172.21.15.28:0/1215609889 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f59800039e0 0x7f599c1ee9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:36.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:37.008+0000 7f59a1800640 1 -- 172.21.15.28:0/1215609889 >> 172.21.15.28:0/1215609889 conn(0x7f599c0ac370 msgr2=0x7f599c157e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:15:36.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:37.008+0000 7f59a1800640 1 -- 172.21.15.28:0/1215609889 shutdown_connections 2024-02-21T23:15:36.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:37.008+0000 7f59a1800640 1 -- 172.21.15.28:0/1215609889 wait complete. 2024-02-21T23:15:37.420 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:15:37.420 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:15:38.421 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:15:38.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.667+0000 7f7259800640 1 Processor -- start 2024-02-21T23:15:38.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.667+0000 7f7259800640 1 -- start start 2024-02-21T23:15:38.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.668+0000 7f7259800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7254072430 0x7f72540c3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:38.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.668+0000 7f7259800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72540c6560 con 0x7f7254155860 2024-02-21T23:15:38.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.668+0000 7f7259800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72540c5cc0 con 0x7f72540c3280 2024-02-21T23:15:38.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.668+0000 7f7259800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72540c5fa0 con 0x7f7254072430 2024-02-21T23:15:38.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.669+0000 7f7253400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f7254155860 0x7f725415c5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42802/0 (socket says 172.21.15.28:42802) 2024-02-21T23:15:38.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.669+0000 7f7253400640 1 -- 172.21.15.28:0/2792522869 learned_addr learned my addr 172.21.15.28:0/2792522869 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:15:38.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.669+0000 7f7252000640 1 --2- 172.21.15.28:0/2792522869 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7254072430 0x7f72540c3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:38.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.670+0000 7f7251600640 1 -- 172.21.15.28:0/2792522869 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3022919197 0 0) 0x7f72540c6560 con 0x7f7254155860 2024-02-21T23:15:38.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.670+0000 7f7251600640 1 -- 172.21.15.28:0/2792522869 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7238005730 con 0x7f7254155860 2024-02-21T23:15:38.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.670+0000 7f7251600640 1 -- 172.21.15.28:0/2792522869 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1075134851 0 0) 0x7f72540c5cc0 con 0x7f72540c3280 2024-02-21T23:15:38.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.670+0000 7f7252000640 1 -- 172.21.15.28:0/2792522869 >> v1:172.21.15.81:6789/0 conn(0x7f72540c3280 legacy=0x7f72541526e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:38.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.670+0000 7f7252000640 1 -- 172.21.15.28:0/2792522869 >> v1:172.21.15.28:6789/0 conn(0x7f7254155860 legacy=0x7f725415c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:38.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.670+0000 7f7252000640 1 -- 172.21.15.28:0/2792522869 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72540c6280 con 0x7f7254072430 2024-02-21T23:15:38.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.671+0000 7f7252000640 1 --2- 172.21.15.28:0/2792522869 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7254072430 0x7f72540c3660 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f72400097d0 tx=0x7f724002b770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=62527a82a089de52 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:38.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.672+0000 7f7251600640 1 -- 172.21.15.28:0/2792522869 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7240037070 con 0x7f7254072430 2024-02-21T23:15:38.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.672+0000 7f7251600640 1 -- 172.21.15.28:0/2792522869 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7240030710 con 0x7f7254072430 2024-02-21T23:15:38.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.672+0000 7f7251600640 1 -- 172.21.15.28:0/2792522869 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7240030ae0 con 0x7f7254072430 2024-02-21T23:15:38.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.673+0000 7f7259800640 1 -- 172.21.15.28:0/2792522869 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7254072430 msgr2=0x7f72540c3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:38.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.673+0000 7f7259800640 1 --2- 172.21.15.28:0/2792522869 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7254072430 0x7f72540c3660 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f72400097d0 tx=0x7f724002b770 comp rx=0 tx=0).stop 2024-02-21T23:15:38.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.673+0000 7f7259800640 1 -- 172.21.15.28:0/2792522869 shutdown_connections 2024-02-21T23:15:38.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.673+0000 7f7259800640 1 --2- 172.21.15.28:0/2792522869 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7254072430 0x7f72540c3660 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:38.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.673+0000 7f7259800640 1 -- 172.21.15.28:0/2792522869 >> 172.21.15.28:0/2792522869 conn(0x7f72540c67f0 msgr2=0x7f72540c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:15:38.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.673+0000 7f7259800640 1 -- 172.21.15.28:0/2792522869 shutdown_connections 2024-02-21T23:15:38.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.673+0000 7f7259800640 1 -- 172.21.15.28:0/2792522869 wait complete. 2024-02-21T23:15:38.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.675+0000 7f7259800640 1 Processor -- start 2024-02-21T23:15:38.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.675+0000 7f7259800640 1 -- start start 2024-02-21T23:15:38.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.676+0000 7f7259800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7254072430 0x7f72540a63c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:38.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.676+0000 7f7259800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72540c3280 0x7f72540abae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:38.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.676+0000 7f7259800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7254155860 0x7f72541eafb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:38.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.676+0000 7f7259800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72540c5aa0 con 0x7f72540c3280 2024-02-21T23:15:38.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.676+0000 7f7259800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72540c5920 con 0x7f7254155860 2024-02-21T23:15:38.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.676+0000 7f7259800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72540733e0 con 0x7f7254072430 2024-02-21T23:15:38.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.676+0000 7f7252000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72540c3280 0x7f72540abae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:38.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.676+0000 7f7252000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72540c3280 0x7f72540abae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55244/0 (socket says 172.21.15.28:55244) 2024-02-21T23:15:38.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.676+0000 7f7252000640 1 -- 172.21.15.28:0/2411411518 learned_addr learned my addr 172.21.15.28:0/2411411518 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:15:38.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.676+0000 7f7252a00640 1 --2- 172.21.15.28:0/2411411518 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7254072430 0x7f72540a63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:38.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.676+0000 7f7253400640 1 --2- 172.21.15.28:0/2411411518 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7254155860 0x7f72541eafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:38.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.677+0000 7f7252000640 1 -- 172.21.15.28:0/2411411518 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7254072430 msgr2=0x7f72540a63c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:38.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.677+0000 7f7252000640 1 --2- 172.21.15.28:0/2411411518 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7254072430 0x7f72540a63c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:38.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.677+0000 7f7252000640 1 -- 172.21.15.28:0/2411411518 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7254155860 msgr2=0x7f72541eafb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:38.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.677+0000 7f7252000640 1 --2- 172.21.15.28:0/2411411518 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7254155860 0x7f72541eafb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:38.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.677+0000 7f7252000640 1 -- 172.21.15.28:0/2411411518 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72540ac230 con 0x7f72540c3280 2024-02-21T23:15:38.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.677+0000 7f7252000640 1 --2- 172.21.15.28:0/2411411518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72540c3280 0x7f72540abae0 secure :-1 s=READY pgs=2568 cs=0 l=1 rev1=1 crypto rx=0x7f7240009720 tx=0x7f72400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:38.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.677+0000 7f724aa00640 1 -- 172.21.15.28:0/2411411518 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7240042070 con 0x7f72540c3280 2024-02-21T23:15:38.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.678+0000 7f724aa00640 1 -- 172.21.15.28:0/2411411518 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7240004220 con 0x7f72540c3280 2024-02-21T23:15:38.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.678+0000 7f724aa00640 1 -- 172.21.15.28:0/2411411518 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7240037040 con 0x7f72540c3280 2024-02-21T23:15:38.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.678+0000 7f7259800640 1 -- 172.21.15.28:0/2411411518 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72540ac510 con 0x7f72540c3280 2024-02-21T23:15:38.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.678+0000 7f7259800640 1 -- 172.21.15.28:0/2411411518 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72541ea770 con 0x7f72540c3280 2024-02-21T23:15:38.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.679+0000 7f7259800640 1 -- 172.21.15.28:0/2411411518 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7254067f30 con 0x7f72540c3280 2024-02-21T23:15:38.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.680+0000 7f724aa00640 1 -- 172.21.15.28:0/2411411518 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f724003a040 con 0x7f72540c3280 2024-02-21T23:15:38.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.681+0000 7f724aa00640 1 --2- 172.21.15.28:0/2411411518 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f723406b4d0 0x7f723406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:38.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.681+0000 7f724aa00640 1 -- 172.21.15.28:0/2411411518 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f72400abd20 con 0x7f72540c3280 2024-02-21T23:15:38.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.682+0000 7f7252a00640 1 --2- 172.21.15.28:0/2411411518 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f723406b4d0 0x7f723406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:38.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.683+0000 7f7252a00640 1 --2- 172.21.15.28:0/2411411518 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f723406b4d0 0x7f723406d990 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f7244000f60 tx=0x7f7244009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:38.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.686+0000 7f724aa00640 1 -- 172.21.15.28:0/2411411518 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f724007a900 con 0x7f72540c3280 2024-02-21T23:15:38.791 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-21T23:15:38.797 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-21T23:15:38.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.932+0000 7f7259800640 1 -- 172.21.15.28:0/2411411518 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f725406f2d0 con 0x7f72540c3280 2024-02-21T23:15:38.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.933+0000 7f724aa00640 1 -- 172.21.15.28:0/2411411518 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7f724007a200 con 0x7f72540c3280 2024-02-21T23:15:38.920 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:15:38.920 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:15:38.920 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:15:38.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.935+0000 7f7259800640 1 -- 172.21.15.28:0/2411411518 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f723406b4d0 msgr2=0x7f723406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:38.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.935+0000 7f7259800640 1 --2- 172.21.15.28:0/2411411518 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f723406b4d0 0x7f723406d990 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f7244000f60 tx=0x7f7244009040 comp rx=0 tx=0).stop 2024-02-21T23:15:38.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.935+0000 7f7259800640 1 -- 172.21.15.28:0/2411411518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72540c3280 msgr2=0x7f72540abae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:38.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.935+0000 7f7259800640 1 --2- 172.21.15.28:0/2411411518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72540c3280 0x7f72540abae0 secure :-1 s=READY pgs=2568 cs=0 l=1 rev1=1 crypto rx=0x7f7240009720 tx=0x7f72400046c0 comp rx=0 tx=0).stop 2024-02-21T23:15:38.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.935+0000 7f7259800640 1 -- 172.21.15.28:0/2411411518 shutdown_connections 2024-02-21T23:15:38.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.935+0000 7f7259800640 1 --2- 172.21.15.28:0/2411411518 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f723406b4d0 0x7f723406d990 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:38.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.935+0000 7f7259800640 1 --2- 172.21.15.28:0/2411411518 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7254155860 0x7f72541eafb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:38.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.935+0000 7f7259800640 1 --2- 172.21.15.28:0/2411411518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72540c3280 0x7f72540abae0 unknown :-1 s=CLOSED pgs=2568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:38.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.935+0000 7f7259800640 1 --2- 172.21.15.28:0/2411411518 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7254072430 0x7f72540a63c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:38.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.935+0000 7f7259800640 1 -- 172.21.15.28:0/2411411518 >> 172.21.15.28:0/2411411518 conn(0x7f72540c67f0 msgr2=0x7f7254155c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:15:38.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.935+0000 7f7259800640 1 -- 172.21.15.28:0/2411411518 shutdown_connections 2024-02-21T23:15:38.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:38.935+0000 7f7259800640 1 -- 172.21.15.28:0/2411411518 wait complete. 2024-02-21T23:15:39.346 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:15:39.346 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:15:40.346 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:15:40.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.586+0000 7f15e7000640 1 Processor -- start 2024-02-21T23:15:40.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.586+0000 7f15e7000640 1 -- start start 2024-02-21T23:15:40.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.588+0000 7f15e7000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f15e006b0e0 0x7f15e006b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:40.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.588+0000 7f15e7000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f15e006f950 0x7f15e0151f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:40.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.588+0000 7f15e7000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f15e0072330 0x7f15e01524b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:40.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.588+0000 7f15e7000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15e0072880 con 0x7f15e006b0e0 2024-02-21T23:15:40.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.588+0000 7f15e7000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15e00c0a80 con 0x7f15e0072330 2024-02-21T23:15:40.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.588+0000 7f15e7000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15e00c0d60 con 0x7f15e006f950 2024-02-21T23:15:40.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.588+0000 7f15e5c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f15e006b0e0 0x7f15e006b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:40.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.588+0000 7f15e5c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f15e006b0e0 0x7f15e006b4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55252/0 (socket says 172.21.15.28:55252) 2024-02-21T23:15:40.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.588+0000 7f15e5c00640 1 -- 172.21.15.28:0/1934336992 learned_addr learned my addr 172.21.15.28:0/1934336992 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:15:40.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.589+0000 7f15e5c00640 1 -- 172.21.15.28:0/1934336992 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f15e006f950 msgr2=0x7f15e0151f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:15:40.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.589+0000 7f15e5c00640 1 --2- 172.21.15.28:0/1934336992 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f15e006f950 0x7f15e0151f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:40.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.589+0000 7f15e5c00640 1 -- 172.21.15.28:0/1934336992 >> v2:172.21.15.81:3300/0 conn(0x7f15e0072330 msgr2=0x7f15e01524b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:15:40.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.589+0000 7f15e5c00640 1 --2- 172.21.15.28:0/1934336992 >> v2:172.21.15.81:3300/0 conn(0x7f15e0072330 0x7f15e01524b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:40.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.589+0000 7f15e5c00640 1 -- 172.21.15.28:0/1934336992 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15e0072ad0 con 0x7f15e006b0e0 2024-02-21T23:15:40.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.590+0000 7f15e5c00640 1 --2- 172.21.15.28:0/1934336992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15e006b0e0 0x7f15e006b4c0 secure :-1 s=READY pgs=2569 cs=0 l=1 rev1=1 crypto rx=0x7f15e0073540 tx=0x7f15d402b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae25a6fa9b5733bb server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:40.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.590+0000 7f15dfe00640 1 -- 172.21.15.28:0/1934336992 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d40317f0 con 0x7f15e006b0e0 2024-02-21T23:15:40.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.590+0000 7f15dfe00640 1 -- 172.21.15.28:0/1934336992 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15d4037070 con 0x7f15e006b0e0 2024-02-21T23:15:40.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.591+0000 7f15dfe00640 1 -- 172.21.15.28:0/1934336992 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d4031bb0 con 0x7f15e006b0e0 2024-02-21T23:15:40.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.591+0000 7f15e7000640 1 -- 172.21.15.28:0/1934336992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15e006b0e0 msgr2=0x7f15e006b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:40.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.591+0000 7f15e7000640 1 --2- 172.21.15.28:0/1934336992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15e006b0e0 0x7f15e006b4c0 secure :-1 s=READY pgs=2569 cs=0 l=1 rev1=1 crypto rx=0x7f15e0073540 tx=0x7f15d402b890 comp rx=0 tx=0).stop 2024-02-21T23:15:40.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.591+0000 7f15e7000640 1 -- 172.21.15.28:0/1934336992 shutdown_connections 2024-02-21T23:15:40.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.591+0000 7f15e7000640 1 --2- 172.21.15.28:0/1934336992 >> v2:172.21.15.81:3300/0 conn(0x7f15e0072330 0x7f15e01524b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:40.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.591+0000 7f15e7000640 1 --2- 172.21.15.28:0/1934336992 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f15e006f950 0x7f15e0151f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:40.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.591+0000 7f15e7000640 1 --2- 172.21.15.28:0/1934336992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15e006b0e0 0x7f15e006b4c0 unknown :-1 s=CLOSED pgs=2569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:40.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.591+0000 7f15e7000640 1 -- 172.21.15.28:0/1934336992 >> 172.21.15.28:0/1934336992 conn(0x7f15e006e8c0 msgr2=0x7f15e006bf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:15:40.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.591+0000 7f15e7000640 1 -- 172.21.15.28:0/1934336992 shutdown_connections 2024-02-21T23:15:40.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.591+0000 7f15e7000640 1 -- 172.21.15.28:0/1934336992 wait complete. 2024-02-21T23:15:40.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.593+0000 7f15e7000640 1 Processor -- start 2024-02-21T23:15:40.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.593+0000 7f15e7000640 1 -- start start 2024-02-21T23:15:40.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.594+0000 7f15e7000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15e006b0e0 0x7f15e0142350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:40.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.594+0000 7f15e7000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f15e006f950 0x7f15e00a91a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:40.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.594+0000 7f15e7000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f15e0072330 0x7f15e00c7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:40.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.594+0000 7f15e7000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e0155020 con 0x7f15e006b0e0 2024-02-21T23:15:40.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.594+0000 7f15e7000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e0154ea0 con 0x7f15e0072330 2024-02-21T23:15:40.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.594+0000 7f15e7000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15e01551a0 con 0x7f15e006f950 2024-02-21T23:15:40.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.594+0000 7f15e5c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15e006b0e0 0x7f15e0142350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:40.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.594+0000 7f15e5c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15e006b0e0 0x7f15e0142350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55268/0 (socket says 172.21.15.28:55268) 2024-02-21T23:15:40.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.594+0000 7f15e5c00640 1 -- 172.21.15.28:0/1063985599 learned_addr learned my addr 172.21.15.28:0/1063985599 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:15:40.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.595+0000 7f15e5c00640 1 -- 172.21.15.28:0/1063985599 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f15e006f950 msgr2=0x7f15e00a91a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:40.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.595+0000 7f15e5c00640 1 --2- 172.21.15.28:0/1063985599 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f15e006f950 0x7f15e00a91a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:40.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.595+0000 7f15e5c00640 1 -- 172.21.15.28:0/1063985599 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f15e0072330 msgr2=0x7f15e00c7ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:40.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.595+0000 7f15e6600640 1 --2- 172.21.15.28:0/1063985599 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f15e0072330 0x7f15e00c7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:40.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.595+0000 7f15e5c00640 1 --2- 172.21.15.28:0/1063985599 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f15e0072330 0x7f15e00c7ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:40.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.595+0000 7f15e5c00640 1 -- 172.21.15.28:0/1063985599 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15e00c7370 con 0x7f15e006b0e0 2024-02-21T23:15:40.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.595+0000 7f15e5c00640 1 --2- 172.21.15.28:0/1063985599 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15e006b0e0 0x7f15e0142350 secure :-1 s=READY pgs=2570 cs=0 l=1 rev1=1 crypto rx=0x7f15d402b8d0 tx=0x7f15d40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:40.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.596+0000 7f15dd600640 1 -- 172.21.15.28:0/1063985599 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d4032890 con 0x7f15e006b0e0 2024-02-21T23:15:40.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.596+0000 7f15dd600640 1 -- 172.21.15.28:0/1063985599 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15d402e070 con 0x7f15e006b0e0 2024-02-21T23:15:40.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.596+0000 7f15dd600640 1 -- 172.21.15.28:0/1063985599 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d4032c70 con 0x7f15e006b0e0 2024-02-21T23:15:40.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.596+0000 7f15e7000640 1 -- 172.21.15.28:0/1063985599 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15e01eaba0 con 0x7f15e006b0e0 2024-02-21T23:15:40.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.596+0000 7f15e7000640 1 -- 172.21.15.28:0/1063985599 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15e01eee90 con 0x7f15e006b0e0 2024-02-21T23:15:40.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.597+0000 7f15e7000640 1 -- 172.21.15.28:0/1063985599 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15e004f290 con 0x7f15e006b0e0 2024-02-21T23:15:40.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.598+0000 7f15dd600640 1 -- 172.21.15.28:0/1063985599 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f15d40453f0 con 0x7f15e006b0e0 2024-02-21T23:15:40.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.599+0000 7f15dd600640 1 --2- 172.21.15.28:0/1063985599 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f15c006b550 0x7f15c006da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:40.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.599+0000 7f15dd600640 1 -- 172.21.15.28:0/1063985599 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f15d40ac700 con 0x7f15e006b0e0 2024-02-21T23:15:40.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.600+0000 7f15e5200640 1 --2- 172.21.15.28:0/1063985599 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f15c006b550 0x7f15c006da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:40.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.601+0000 7f15e5200640 1 --2- 172.21.15.28:0/1063985599 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f15c006b550 0x7f15c006da10 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f15e014bed0 tx=0x7f15cc006c90 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:40.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.608+0000 7f15dd600640 1 -- 172.21.15.28:0/1063985599 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f15d4040070 con 0x7f15e006b0e0 2024-02-21T23:15:40.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.847+0000 7f15e7000640 1 -- 172.21.15.28:0/1063985599 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f15e00c1c30 con 0x7f15e006b0e0 2024-02-21T23:15:40.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.848+0000 7f15dd600640 1 -- 172.21.15.28:0/1063985599 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7f15d407b360 con 0x7f15e006b0e0 2024-02-21T23:15:40.836 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:15:40.836 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:15:40.836 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:15:40.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.851+0000 7f15e7000640 1 -- 172.21.15.28:0/1063985599 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f15c006b550 msgr2=0x7f15c006da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:40.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.851+0000 7f15e7000640 1 --2- 172.21.15.28:0/1063985599 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f15c006b550 0x7f15c006da10 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f15e014bed0 tx=0x7f15cc006c90 comp rx=0 tx=0).stop 2024-02-21T23:15:40.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.851+0000 7f15e7000640 1 -- 172.21.15.28:0/1063985599 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15e006b0e0 msgr2=0x7f15e0142350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:40.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.851+0000 7f15e7000640 1 --2- 172.21.15.28:0/1063985599 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15e006b0e0 0x7f15e0142350 secure :-1 s=READY pgs=2570 cs=0 l=1 rev1=1 crypto rx=0x7f15d402b8d0 tx=0x7f15d40046c0 comp rx=0 tx=0).stop 2024-02-21T23:15:40.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.851+0000 7f15e7000640 1 -- 172.21.15.28:0/1063985599 shutdown_connections 2024-02-21T23:15:40.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.851+0000 7f15e7000640 1 --2- 172.21.15.28:0/1063985599 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f15c006b550 0x7f15c006da10 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:40.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.851+0000 7f15e7000640 1 --2- 172.21.15.28:0/1063985599 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f15e006f950 0x7f15e00a91a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:40.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.851+0000 7f15e7000640 1 --2- 172.21.15.28:0/1063985599 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f15e0072330 0x7f15e00c7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:40.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.851+0000 7f15e7000640 1 --2- 172.21.15.28:0/1063985599 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15e006b0e0 0x7f15e0142350 unknown :-1 s=CLOSED pgs=2570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:40.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.851+0000 7f15e7000640 1 -- 172.21.15.28:0/1063985599 >> 172.21.15.28:0/1063985599 conn(0x7f15e006e8c0 msgr2=0x7f15e0150550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:15:40.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.851+0000 7f15e7000640 1 -- 172.21.15.28:0/1063985599 shutdown_connections 2024-02-21T23:15:40.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:40.851+0000 7f15e7000640 1 -- 172.21.15.28:0/1063985599 wait complete. 2024-02-21T23:15:41.263 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:15:41.263 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:15:42.264 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:15:42.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:42.505+0000 7f04cf400640 1 Processor -- start 2024-02-21T23:15:42.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:42.506+0000 7f04cf400640 1 -- start start 2024-02-21T23:15:42.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:42.507+0000 7f04cf400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f04c813e620 0x7f04c8159070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:42.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:42.507+0000 7f04cf400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04c81551b0 con 0x7f04c80bf990 2024-02-21T23:15:42.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:42.507+0000 7f04cf400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04c81595b0 con 0x7f04c80c6460 2024-02-21T23:15:42.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:42.507+0000 7f04cf400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04c80c71d0 con 0x7f04c813e620 2024-02-21T23:15:42.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:42.508+0000 7f04cca00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f04c813e620 0x7f04c8159070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:42.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:42.508+0000 7f04cca00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f04c813e620 0x7f04c8159070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55280/0 (socket says 172.21.15.28:55280) 2024-02-21T23:15:42.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:42.508+0000 7f04cca00640 1 -- 172.21.15.28:0/2862620123 learned_addr learned my addr 172.21.15.28:0/2862620123 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:15:42.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:42.508+0000 7f04cd400640 1 --1- 172.21.15.28:0/2862620123 >> v1:172.21.15.28:6789/0 conn(0x7f04c80bf990 0x7f04c80bfdc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42818/0 (socket says 172.21.15.28:42818) 2024-02-21T23:15:42.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:42.508+0000 7f04cca00640 1 -- 172.21.15.28:0/2862620123 >> v1:172.21.15.81:6789/0 conn(0x7f04c80c6460 legacy=0x7f04c80ca8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:15:42.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:42.508+0000 7f04cca00640 1 -- 172.21.15.28:0/2862620123 >> v1:172.21.15.28:6789/0 conn(0x7f04c80bf990 legacy=0x7f04c80bfdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:42.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:42.508+0000 7f04cca00640 1 -- 172.21.15.28:0/2862620123 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04c8155420 con 0x7f04c813e620 2024-02-21T23:15:42.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:42.508+0000 7f04c7400640 1 -- 172.21.15.28:0/2862620123 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 720384209 0 0) 0x7f04c81551b0 con 0x7f04c80bf990 2024-02-21T23:15:42.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:42.510+0000 7f04cca00640 1 --2- 172.21.15.28:0/2862620123 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04c813e620 0x7f04c8159070 secure :-1 s=READY pgs=2572 cs=0 l=1 rev1=1 crypto rx=0x7f04bc009870 tx=0x7f04bc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de776ca0eb1f3e39 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:42.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:42.510+0000 7f04c7400640 1 -- 172.21.15.28:0/2862620123 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04bc035020 con 0x7f04c813e620 2024-02-21T23:15:42.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:42.510+0000 7f04c7400640 1 -- 172.21.15.28:0/2862620123 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04bc031870 con 0x7f04c813e620 2024-02-21T23:15:42.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:42.510+0000 7f04c7400640 1 -- 172.21.15.28:0/2862620123 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04bc031c40 con 0x7f04c813e620 2024-02-21T23:15:42.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:42.510+0000 7f04cf400640 1 -- 172.21.15.28:0/2862620123 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04c813e620 msgr2=0x7f04c8159070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:42.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:42.510+0000 7f04cf400640 1 --2- 172.21.15.28:0/2862620123 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04c813e620 0x7f04c8159070 secure :-1 s=READY pgs=2572 cs=0 l=1 rev1=1 crypto rx=0x7f04bc009870 tx=0x7f04bc02b910 comp rx=0 tx=0).stop 2024-02-21T23:15:42.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:42.511+0000 7f04cf400640 1 -- 172.21.15.28:0/2862620123 shutdown_connections 2024-02-21T23:15:42.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:42.511+0000 7f04cf400640 1 --2- 172.21.15.28:0/2862620123 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04c813e620 0x7f04c8159070 unknown :-1 s=CLOSED pgs=2572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:42.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:42.511+0000 7f04cf400640 1 -- 172.21.15.28:0/2862620123 >> 172.21.15.28:0/2862620123 conn(0x7f04c808b1c0 msgr2=0x7f04c80725a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:15:42.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:42.511+0000 7f04cf400640 1 -- 172.21.15.28:0/2862620123 shutdown_connections 2024-02-21T23:15:42.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:42.511+0000 7f04cf400640 1 -- 172.21.15.28:0/2862620123 wait complete. 2024-02-21T23:15:42.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:42.512+0000 7f04cf400640 1 Processor -- start 2024-02-21T23:15:42.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:42.513+0000 7f04cf400640 1 -- start start 2024-02-21T23:15:42.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:42.513+0000 7f04cf400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04c80bf990 0x7f04c81ee400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:42.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:42.513+0000 7f04cf400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f04c80c6460 0x7f04c80c3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:42.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:42.513+0000 7f04cf400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f04c813e620 0x7f04c8138000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:42.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:42.513+0000 7f04cf400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04c8155cc0 con 0x7f04c80bf990 2024-02-21T23:15:42.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:42.513+0000 7f04cf400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04c81559e0 con 0x7f04c80c6460 2024-02-21T23:15:42.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:42.513+0000 7f04cf400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04c813ea00 con 0x7f04c813e620 2024-02-21T23:15:42.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:42.514+0000 7f04cca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04c80bf990 0x7f04c81ee400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:42.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:42.514+0000 7f04cca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04c80bf990 0x7f04c81ee400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55282/0 (socket says 172.21.15.28:55282) 2024-02-21T23:15:42.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:42.514+0000 7f04cca00640 1 -- 172.21.15.28:0/199484803 learned_addr learned my addr 172.21.15.28:0/199484803 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:15:42.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:42.514+0000 7f04cca00640 1 -- 172.21.15.28:0/199484803 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f04c813e620 msgr2=0x7f04c8138000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:42.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:42.514+0000 7f04cd400640 1 --2- 172.21.15.28:0/199484803 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f04c813e620 0x7f04c8138000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:42.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:42.514+0000 7f04c7e00640 1 --2- 172.21.15.28:0/199484803 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f04c80c6460 0x7f04c80c3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:42.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:42.514+0000 7f04cca00640 1 --2- 172.21.15.28:0/199484803 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f04c813e620 0x7f04c8138000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:42.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:42.514+0000 7f04cca00640 1 -- 172.21.15.28:0/199484803 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f04c80c6460 msgr2=0x7f04c80c3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:42.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:42.514+0000 7f04cca00640 1 --2- 172.21.15.28:0/199484803 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f04c80c6460 0x7f04c80c3d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:42.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:42.514+0000 7f04cca00640 1 -- 172.21.15.28:0/199484803 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04c80c3820 con 0x7f04c80bf990 2024-02-21T23:15:42.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:42.515+0000 7f04cca00640 1 --2- 172.21.15.28:0/199484803 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04c80bf990 0x7f04c81ee400 secure :-1 s=READY pgs=2573 cs=0 l=1 rev1=1 crypto rx=0x7f04bc0097c0 tx=0x7f04bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:42.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:42.515+0000 7f04c4c00640 1 -- 172.21.15.28:0/199484803 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04bc03ad60 con 0x7f04c80bf990 2024-02-21T23:15:42.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:42.515+0000 7f04c4c00640 1 -- 172.21.15.28:0/199484803 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04bc0443f0 con 0x7f04c80bf990 2024-02-21T23:15:42.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:42.515+0000 7f04c4c00640 1 -- 172.21.15.28:0/199484803 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04bc0447e0 con 0x7f04c80bf990 2024-02-21T23:15:42.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:42.515+0000 7f04cf400640 1 -- 172.21.15.28:0/199484803 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04c81e22b0 con 0x7f04c80bf990 2024-02-21T23:15:42.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:42.515+0000 7f04cf400640 1 -- 172.21.15.28:0/199484803 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04c81eed40 con 0x7f04c80bf990 2024-02-21T23:15:42.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:42.517+0000 7f04cf400640 1 -- 172.21.15.28:0/199484803 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04c80c39f0 con 0x7f04c80bf990 2024-02-21T23:15:42.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:42.518+0000 7f04c4c00640 1 -- 172.21.15.28:0/199484803 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f04bc04b060 con 0x7f04c80bf990 2024-02-21T23:15:42.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:42.519+0000 7f04c4c00640 1 --2- 172.21.15.28:0/199484803 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f04a806b4d0 0x7f04a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:42.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:42.519+0000 7f04c4c00640 1 -- 172.21.15.28:0/199484803 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f04bc0ac610 con 0x7f04c80bf990 2024-02-21T23:15:42.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:42.519+0000 7f04c7e00640 1 --2- 172.21.15.28:0/199484803 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f04a806b4d0 0x7f04a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:42.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:42.521+0000 7f04c7e00640 1 --2- 172.21.15.28:0/199484803 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f04a806b4d0 0x7f04a806d990 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f04c8152360 tx=0x7f04b4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:42.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:42.523+0000 7f04c4c00640 1 -- 172.21.15.28:0/199484803 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f04bc03f070 con 0x7f04c80bf990 2024-02-21T23:15:42.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:42.844+0000 7f04cf400640 1 -- 172.21.15.28:0/199484803 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f04c8067f30 con 0x7f04c80bf990 2024-02-21T23:15:42.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:42.845+0000 7f04c4c00640 1 -- 172.21.15.28:0/199484803 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7f04bc02f320 con 0x7f04c80bf990 2024-02-21T23:15:42.832 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:15:42.832 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:15:42.832 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:15:42.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:42.847+0000 7f04cf400640 1 -- 172.21.15.28:0/199484803 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f04a806b4d0 msgr2=0x7f04a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:42.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:42.847+0000 7f04cf400640 1 --2- 172.21.15.28:0/199484803 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f04a806b4d0 0x7f04a806d990 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f04c8152360 tx=0x7f04b4008040 comp rx=0 tx=0).stop 2024-02-21T23:15:42.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:42.847+0000 7f04cf400640 1 -- 172.21.15.28:0/199484803 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04c80bf990 msgr2=0x7f04c81ee400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:42.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:42.847+0000 7f04cf400640 1 --2- 172.21.15.28:0/199484803 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04c80bf990 0x7f04c81ee400 secure :-1 s=READY pgs=2573 cs=0 l=1 rev1=1 crypto rx=0x7f04bc0097c0 tx=0x7f04bc0046c0 comp rx=0 tx=0).stop 2024-02-21T23:15:42.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:42.847+0000 7f04cf400640 1 -- 172.21.15.28:0/199484803 shutdown_connections 2024-02-21T23:15:42.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:42.847+0000 7f04cf400640 1 --2- 172.21.15.28:0/199484803 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f04a806b4d0 0x7f04a806d990 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:42.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:42.847+0000 7f04cf400640 1 --2- 172.21.15.28:0/199484803 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f04c813e620 0x7f04c8138000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:42.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:42.847+0000 7f04cf400640 1 --2- 172.21.15.28:0/199484803 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f04c80c6460 0x7f04c80c3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:42.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:42.847+0000 7f04cf400640 1 --2- 172.21.15.28:0/199484803 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04c80bf990 0x7f04c81ee400 unknown :-1 s=CLOSED pgs=2573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:42.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:42.847+0000 7f04cf400640 1 -- 172.21.15.28:0/199484803 >> 172.21.15.28:0/199484803 conn(0x7f04c808b1c0 msgr2=0x7f04c80c7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:15:42.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:42.847+0000 7f04cf400640 1 -- 172.21.15.28:0/199484803 shutdown_connections 2024-02-21T23:15:42.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:42.847+0000 7f04cf400640 1 -- 172.21.15.28:0/199484803 wait complete. 2024-02-21T23:15:43.255 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:15:43.255 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:15:44.256 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:15:44.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.498+0000 7ff18f200640 1 Processor -- start 2024-02-21T23:15:44.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.498+0000 7ff18f200640 1 -- start start 2024-02-21T23:15:44.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.499+0000 7ff18f200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff1900aff90 0x7ff1900c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:44.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.499+0000 7ff18f200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff1900c5720 0x7ff19015f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:44.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.499+0000 7ff18f200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1900c74c0 con 0x7ff19006dbf0 2024-02-21T23:15:44.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.499+0000 7ff18f200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1900c7270 con 0x7ff1900aff90 2024-02-21T23:15:44.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.499+0000 7ff18f200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff190158630 con 0x7ff1900c5720 2024-02-21T23:15:44.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.500+0000 7ff18de00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff1900aff90 0x7ff1900c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:44.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.500+0000 7ff18de00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff1900aff90 0x7ff1900c51e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55292/0 (socket says 172.21.15.28:55292) 2024-02-21T23:15:44.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.500+0000 7ff18de00640 1 -- 172.21.15.28:0/336772842 learned_addr learned my addr 172.21.15.28:0/336772842 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:15:44.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.500+0000 7ff18de00640 1 -- 172.21.15.28:0/336772842 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff1900c5720 msgr2=0x7ff19015f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:15:44.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.500+0000 7ff18e800640 1 --2- 172.21.15.28:0/336772842 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff1900c5720 0x7ff19015f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:44.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.500+0000 7ff18de00640 1 --2- 172.21.15.28:0/336772842 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff1900c5720 0x7ff19015f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:44.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.500+0000 7ff18de00640 1 -- 172.21.15.28:0/336772842 >> v1:172.21.15.28:6789/0 conn(0x7ff19006dbf0 legacy=0x7ff190158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:44.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.500+0000 7ff18de00640 1 -- 172.21.15.28:0/336772842 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1900c77a0 con 0x7ff1900aff90 2024-02-21T23:15:44.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.500+0000 7ff18e800640 1 --2- 172.21.15.28:0/336772842 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff1900c5720 0x7ff19015f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:15:44.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.502+0000 7ff18de00640 1 --2- 172.21.15.28:0/336772842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1900aff90 0x7ff1900c51e0 secure :-1 s=READY pgs=2575 cs=0 l=1 rev1=1 crypto rx=0x7ff180009870 tx=0x7ff18002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c21fd64c67e02d94 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:44.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.503+0000 7ff18d400640 1 -- 172.21.15.28:0/336772842 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff180034070 con 0x7ff1900aff90 2024-02-21T23:15:44.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.503+0000 7ff18d400640 1 -- 172.21.15.28:0/336772842 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff180030840 con 0x7ff1900aff90 2024-02-21T23:15:44.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.503+0000 7ff18d400640 1 -- 172.21.15.28:0/336772842 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff180030c10 con 0x7ff1900aff90 2024-02-21T23:15:44.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.503+0000 7ff18f200640 1 -- 172.21.15.28:0/336772842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1900aff90 msgr2=0x7ff1900c51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:44.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.503+0000 7ff18f200640 1 --2- 172.21.15.28:0/336772842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1900aff90 0x7ff1900c51e0 secure :-1 s=READY pgs=2575 cs=0 l=1 rev1=1 crypto rx=0x7ff180009870 tx=0x7ff18002b980 comp rx=0 tx=0).stop 2024-02-21T23:15:44.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.503+0000 7ff18f200640 1 -- 172.21.15.28:0/336772842 shutdown_connections 2024-02-21T23:15:44.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.503+0000 7ff18f200640 1 --2- 172.21.15.28:0/336772842 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff1900c5720 0x7ff19015f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:44.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.503+0000 7ff18f200640 1 --2- 172.21.15.28:0/336772842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1900aff90 0x7ff1900c51e0 unknown :-1 s=CLOSED pgs=2575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:44.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.503+0000 7ff18f200640 1 -- 172.21.15.28:0/336772842 >> 172.21.15.28:0/336772842 conn(0x7ff1900ac370 msgr2=0x7ff1900ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:15:44.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.503+0000 7ff18f200640 1 -- 172.21.15.28:0/336772842 shutdown_connections 2024-02-21T23:15:44.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.503+0000 7ff18f200640 1 -- 172.21.15.28:0/336772842 wait complete. 2024-02-21T23:15:44.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.505+0000 7ff18f200640 1 Processor -- start 2024-02-21T23:15:44.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.505+0000 7ff18f200640 1 -- start start 2024-02-21T23:15:44.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.506+0000 7ff18f200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff19006dbf0 0x7ff1900ae740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:44.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.506+0000 7ff18f200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff1900aff90 0x7ff1901ebbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:44.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.506+0000 7ff18f200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1900c5720 0x7ff19014e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:44.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.506+0000 7ff18f200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1900c8760 con 0x7ff1900c5720 2024-02-21T23:15:44.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.506+0000 7ff18f200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1900be0c0 con 0x7ff19006dbf0 2024-02-21T23:15:44.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.506+0000 7ff18f200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1900c88e0 con 0x7ff1900aff90 2024-02-21T23:15:44.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.506+0000 7ff18e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1900c5720 0x7ff19014e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:44.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.507+0000 7ff18e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1900c5720 0x7ff19014e3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55304/0 (socket says 172.21.15.28:55304) 2024-02-21T23:15:44.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.507+0000 7ff18e800640 1 -- 172.21.15.28:0/3881244583 learned_addr learned my addr 172.21.15.28:0/3881244583 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:15:44.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.507+0000 7ff18de00640 1 --2- 172.21.15.28:0/3881244583 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff19006dbf0 0x7ff1900ae740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:44.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.507+0000 7ff18e800640 1 -- 172.21.15.28:0/3881244583 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff1900aff90 msgr2=0x7ff1901ebbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:44.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.507+0000 7ff185400640 1 --2- 172.21.15.28:0/3881244583 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff1900aff90 0x7ff1901ebbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:44.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.507+0000 7ff18e800640 1 --2- 172.21.15.28:0/3881244583 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff1900aff90 0x7ff1901ebbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:44.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.507+0000 7ff18e800640 1 -- 172.21.15.28:0/3881244583 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff19006dbf0 msgr2=0x7ff1900ae740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:44.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.507+0000 7ff18e800640 1 --2- 172.21.15.28:0/3881244583 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff19006dbf0 0x7ff1900ae740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:44.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.507+0000 7ff18e800640 1 -- 172.21.15.28:0/3881244583 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff190154600 con 0x7ff1900c5720 2024-02-21T23:15:44.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.507+0000 7ff18de00640 1 --2- 172.21.15.28:0/3881244583 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff19006dbf0 0x7ff1900ae740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:15:44.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.507+0000 7ff185400640 1 --2- 172.21.15.28:0/3881244583 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff1900aff90 0x7ff1901ebbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:15:44.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.507+0000 7ff18e800640 1 --2- 172.21.15.28:0/3881244583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1900c5720 0x7ff19014e3b0 secure :-1 s=READY pgs=2576 cs=0 l=1 rev1=1 crypto rx=0x7ff18800cd80 tx=0x7ff18800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:44.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.508+0000 7ff186a00640 1 -- 172.21.15.28:0/3881244583 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff18800f680 con 0x7ff1900c5720 2024-02-21T23:15:44.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.508+0000 7ff186a00640 1 -- 172.21.15.28:0/3881244583 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff18800f840 con 0x7ff1900c5720 2024-02-21T23:15:44.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.508+0000 7ff18f200640 1 -- 172.21.15.28:0/3881244583 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff190153790 con 0x7ff1900c5720 2024-02-21T23:15:44.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.508+0000 7ff186a00640 1 -- 172.21.15.28:0/3881244583 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff18800fc30 con 0x7ff1900c5720 2024-02-21T23:15:44.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.508+0000 7ff18f200640 1 -- 172.21.15.28:0/3881244583 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1901548c0 con 0x7ff1900c5720 2024-02-21T23:15:44.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.509+0000 7ff18f200640 1 -- 172.21.15.28:0/3881244583 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff19004f290 con 0x7ff1900c5720 2024-02-21T23:15:44.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.510+0000 7ff186a00640 1 -- 172.21.15.28:0/3881244583 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff1880213f0 con 0x7ff1900c5720 2024-02-21T23:15:44.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.512+0000 7ff186a00640 1 --2- 172.21.15.28:0/3881244583 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff16c06b4d0 0x7ff16c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:44.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.512+0000 7ff186a00640 1 -- 172.21.15.28:0/3881244583 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7ff18808b2a0 con 0x7ff1900c5720 2024-02-21T23:15:44.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.512+0000 7ff18de00640 1 --2- 172.21.15.28:0/3881244583 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff16c06b4d0 0x7ff16c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:44.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.514+0000 7ff18de00640 1 --2- 172.21.15.28:0/3881244583 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff16c06b4d0 0x7ff16c06d990 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7ff180036000 tx=0x7ff18003b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:44.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.516+0000 7ff186a00640 1 -- 172.21.15.28:0/3881244583 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ff188014030 con 0x7ff1900c5720 2024-02-21T23:15:44.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.782+0000 7ff18f200640 1 -- 172.21.15.28:0/3881244583 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff19006d9d0 con 0x7ff1900c5720 2024-02-21T23:15:44.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.783+0000 7ff186a00640 1 -- 172.21.15.28:0/3881244583 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7ff1880597a0 con 0x7ff1900c5720 2024-02-21T23:15:44.770 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:15:44.770 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:15:44.770 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:15:44.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.785+0000 7ff18f200640 1 -- 172.21.15.28:0/3881244583 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff16c06b4d0 msgr2=0x7ff16c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:44.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.785+0000 7ff18f200640 1 --2- 172.21.15.28:0/3881244583 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff16c06b4d0 0x7ff16c06d990 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7ff180036000 tx=0x7ff18003b040 comp rx=0 tx=0).stop 2024-02-21T23:15:44.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.785+0000 7ff18f200640 1 -- 172.21.15.28:0/3881244583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1900c5720 msgr2=0x7ff19014e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:44.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.785+0000 7ff18f200640 1 --2- 172.21.15.28:0/3881244583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1900c5720 0x7ff19014e3b0 secure :-1 s=READY pgs=2576 cs=0 l=1 rev1=1 crypto rx=0x7ff18800cd80 tx=0x7ff18800a650 comp rx=0 tx=0).stop 2024-02-21T23:15:44.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.786+0000 7ff18f200640 1 -- 172.21.15.28:0/3881244583 shutdown_connections 2024-02-21T23:15:44.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.786+0000 7ff18f200640 1 --2- 172.21.15.28:0/3881244583 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff16c06b4d0 0x7ff16c06d990 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:44.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.786+0000 7ff18f200640 1 --2- 172.21.15.28:0/3881244583 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff1900aff90 0x7ff1901ebbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:44.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.786+0000 7ff18f200640 1 --2- 172.21.15.28:0/3881244583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1900c5720 0x7ff19014e3b0 unknown :-1 s=CLOSED pgs=2576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:44.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.786+0000 7ff18f200640 1 --2- 172.21.15.28:0/3881244583 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff19006dbf0 0x7ff1900ae740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:44.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.786+0000 7ff18f200640 1 -- 172.21.15.28:0/3881244583 >> 172.21.15.28:0/3881244583 conn(0x7ff1900ac370 msgr2=0x7ff1900ac750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:15:44.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.786+0000 7ff18f200640 1 -- 172.21.15.28:0/3881244583 shutdown_connections 2024-02-21T23:15:44.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:44.786+0000 7ff18f200640 1 -- 172.21.15.28:0/3881244583 wait complete. 2024-02-21T23:15:45.200 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:15:45.200 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:15:46.201 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:15:46.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.432+0000 7f0a36c00640 1 Processor -- start 2024-02-21T23:15:46.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.432+0000 7f0a36c00640 1 -- start start 2024-02-21T23:15:46.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.434+0000 7f0a36c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0a300c11f0 0x7f0a30153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:46.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.434+0000 7f0a36c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0a3015b6d0 0x7f0a3015dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:46.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.434+0000 7f0a36c00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a30152e70 con 0x7f0a300c15d0 2024-02-21T23:15:46.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.434+0000 7f0a36c00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a300c1a60 con 0x7f0a300c11f0 2024-02-21T23:15:46.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.434+0000 7f0a36c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a300a5ec0 con 0x7f0a3015b6d0 2024-02-21T23:15:46.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.435+0000 7f0a2fe00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0a300c11f0 0x7f0a30153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:46.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.435+0000 7f0a34c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0a3015b6d0 0x7f0a3015dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:46.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.435+0000 7f0a2fe00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0a300c11f0 0x7f0a30153950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.28:60504/0 (socket says 172.21.15.28:60504) 2024-02-21T23:15:46.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.435+0000 7f0a2fe00640 1 -- 172.21.15.28:0/2267359522 learned_addr learned my addr 172.21.15.28:0/2267359522 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:15:46.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.435+0000 7f0a34c00640 1 -- 172.21.15.28:0/2267359522 >> v2:172.21.15.81:3300/0 conn(0x7f0a300c11f0 msgr2=0x7f0a30153950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:15:46.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.435+0000 7f0a34c00640 1 --2- 172.21.15.28:0/2267359522 >> v2:172.21.15.81:3300/0 conn(0x7f0a300c11f0 0x7f0a30153950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:46.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.435+0000 7f0a34c00640 1 -- 172.21.15.28:0/2267359522 >> v1:172.21.15.81:6789/0 conn(0x7f0a300c15d0 legacy=0x7f0a3006dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:46.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.435+0000 7f0a34c00640 1 -- 172.21.15.28:0/2267359522 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a30153150 con 0x7f0a3015b6d0 2024-02-21T23:15:46.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.437+0000 7f0a34c00640 1 --2- 172.21.15.28:0/2267359522 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0a3015b6d0 0x7f0a3015dac0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f0a280097a0 tx=0x7f0a2802b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e2b9e0baa7d4b13 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:46.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.437+0000 7f0a2f400640 1 -- 172.21.15.28:0/2267359522 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a28039070 con 0x7f0a3015b6d0 2024-02-21T23:15:46.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.437+0000 7f0a2f400640 1 -- 172.21.15.28:0/2267359522 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a280316f0 con 0x7f0a3015b6d0 2024-02-21T23:15:46.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.437+0000 7f0a2f400640 1 -- 172.21.15.28:0/2267359522 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a28031ac0 con 0x7f0a3015b6d0 2024-02-21T23:15:46.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.438+0000 7f0a36c00640 1 -- 172.21.15.28:0/2267359522 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0a3015b6d0 msgr2=0x7f0a3015dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:46.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.438+0000 7f0a36c00640 1 --2- 172.21.15.28:0/2267359522 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0a3015b6d0 0x7f0a3015dac0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f0a280097a0 tx=0x7f0a2802b790 comp rx=0 tx=0).stop 2024-02-21T23:15:46.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.438+0000 7f0a36c00640 1 -- 172.21.15.28:0/2267359522 shutdown_connections 2024-02-21T23:15:46.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.438+0000 7f0a36c00640 1 --2- 172.21.15.28:0/2267359522 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0a3015b6d0 0x7f0a3015dac0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:46.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.438+0000 7f0a36c00640 1 --2- 172.21.15.28:0/2267359522 >> v2:172.21.15.81:3300/0 conn(0x7f0a300c11f0 0x7f0a30153950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:46.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.438+0000 7f0a36c00640 1 -- 172.21.15.28:0/2267359522 >> 172.21.15.28:0/2267359522 conn(0x7f0a300c67f0 msgr2=0x7f0a300c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:15:46.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.438+0000 7f0a36c00640 1 -- 172.21.15.28:0/2267359522 shutdown_connections 2024-02-21T23:15:46.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.438+0000 7f0a36c00640 1 -- 172.21.15.28:0/2267359522 wait complete. 2024-02-21T23:15:46.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.440+0000 7f0a36c00640 1 Processor -- start 2024-02-21T23:15:46.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.440+0000 7f0a36c00640 1 -- start start 2024-02-21T23:15:46.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.441+0000 7f0a36c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a300c11f0 0x7f0a300a7ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:46.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.441+0000 7f0a36c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0a300c15d0 0x7f0a30151e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:46.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.441+0000 7f0a36c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a3015b6d0 0x7f0a30142860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:46.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.441+0000 7f0a36c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a300c4c00 con 0x7f0a300c11f0 2024-02-21T23:15:46.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.441+0000 7f0a36c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a3014d1c0 con 0x7f0a3015b6d0 2024-02-21T23:15:46.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.441+0000 7f0a36c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a300a6110 con 0x7f0a300c15d0 2024-02-21T23:15:46.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.441+0000 7f0a2fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a300c11f0 0x7f0a300a7ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:46.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.442+0000 7f0a2fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a300c11f0 0x7f0a300a7ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59300/0 (socket says 172.21.15.28:59300) 2024-02-21T23:15:46.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.442+0000 7f0a2fe00640 1 -- 172.21.15.28:0/1758159251 learned_addr learned my addr 172.21.15.28:0/1758159251 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:15:46.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.442+0000 7f0a27400640 1 --2- 172.21.15.28:0/1758159251 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0a300c15d0 0x7f0a30151e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:46.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.442+0000 7f0a34c00640 1 --2- 172.21.15.28:0/1758159251 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a3015b6d0 0x7f0a30142860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:46.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.442+0000 7f0a2fe00640 1 -- 172.21.15.28:0/1758159251 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0a300c15d0 msgr2=0x7f0a30151e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:46.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.442+0000 7f0a2fe00640 1 --2- 172.21.15.28:0/1758159251 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0a300c15d0 0x7f0a30151e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:46.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.442+0000 7f0a2fe00640 1 -- 172.21.15.28:0/1758159251 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a3015b6d0 msgr2=0x7f0a30142860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:46.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.442+0000 7f0a2fe00640 1 --2- 172.21.15.28:0/1758159251 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a3015b6d0 0x7f0a30142860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:46.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.442+0000 7f0a2fe00640 1 -- 172.21.15.28:0/1758159251 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a300a63d0 con 0x7f0a300c11f0 2024-02-21T23:15:46.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.442+0000 7f0a2fe00640 1 --2- 172.21.15.28:0/1758159251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a300c11f0 0x7f0a300a7ec0 secure :-1 s=READY pgs=2577 cs=0 l=1 rev1=1 crypto rx=0x7f0a2000efc0 tx=0x7f0a2000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:46.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.443+0000 7f0a2cc00640 1 -- 172.21.15.28:0/1758159251 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a2000cda0 con 0x7f0a300c11f0 2024-02-21T23:15:46.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.443+0000 7f0a2cc00640 1 -- 172.21.15.28:0/1758159251 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a20019070 con 0x7f0a300c11f0 2024-02-21T23:15:46.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.443+0000 7f0a2cc00640 1 -- 172.21.15.28:0/1758159251 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a20014680 con 0x7f0a300c11f0 2024-02-21T23:15:46.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.443+0000 7f0a36c00640 1 -- 172.21.15.28:0/1758159251 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a30148ca0 con 0x7f0a300c11f0 2024-02-21T23:15:46.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.443+0000 7f0a36c00640 1 -- 172.21.15.28:0/1758159251 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a300a6670 con 0x7f0a300c11f0 2024-02-21T23:15:46.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.444+0000 7f0a36c00640 1 -- 172.21.15.28:0/1758159251 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a30067f30 con 0x7f0a300c11f0 2024-02-21T23:15:46.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.446+0000 7f0a2cc00640 1 -- 172.21.15.28:0/1758159251 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0a20005d10 con 0x7f0a300c11f0 2024-02-21T23:15:46.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.447+0000 7f0a2cc00640 1 --2- 172.21.15.28:0/1758159251 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0a0c06b4d0 0x7f0a0c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:46.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.447+0000 7f0a2cc00640 1 -- 172.21.15.28:0/1758159251 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f0a2008aa40 con 0x7f0a300c11f0 2024-02-21T23:15:46.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.448+0000 7f0a27400640 1 --2- 172.21.15.28:0/1758159251 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0a0c06b4d0 0x7f0a0c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:46.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.449+0000 7f0a27400640 1 --2- 172.21.15.28:0/1758159251 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0a0c06b4d0 0x7f0a0c06d990 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f0a300a8400 tx=0x7f0a18009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:46.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.452+0000 7f0a2cc00640 1 -- 172.21.15.28:0/1758159251 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f0a20010040 con 0x7f0a300c11f0 2024-02-21T23:15:46.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.690+0000 7f0a36c00640 1 -- 172.21.15.28:0/1758159251 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0a300a5c30 con 0x7f0a300c11f0 2024-02-21T23:15:46.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.691+0000 7f0a2cc00640 1 -- 172.21.15.28:0/1758159251 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7f0a20059620 con 0x7f0a300c11f0 2024-02-21T23:15:46.678 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:15:46.678 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:15:46.678 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:15:46.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.694+0000 7f0a36c00640 1 -- 172.21.15.28:0/1758159251 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0a0c06b4d0 msgr2=0x7f0a0c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:46.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.694+0000 7f0a36c00640 1 --2- 172.21.15.28:0/1758159251 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0a0c06b4d0 0x7f0a0c06d990 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f0a300a8400 tx=0x7f0a18009040 comp rx=0 tx=0).stop 2024-02-21T23:15:46.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.694+0000 7f0a36c00640 1 -- 172.21.15.28:0/1758159251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a300c11f0 msgr2=0x7f0a300a7ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:46.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.694+0000 7f0a36c00640 1 --2- 172.21.15.28:0/1758159251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a300c11f0 0x7f0a300a7ec0 secure :-1 s=READY pgs=2577 cs=0 l=1 rev1=1 crypto rx=0x7f0a2000efc0 tx=0x7f0a2000c490 comp rx=0 tx=0).stop 2024-02-21T23:15:46.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.694+0000 7f0a36c00640 1 -- 172.21.15.28:0/1758159251 shutdown_connections 2024-02-21T23:15:46.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.694+0000 7f0a36c00640 1 --2- 172.21.15.28:0/1758159251 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0a0c06b4d0 0x7f0a0c06d990 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:46.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.694+0000 7f0a36c00640 1 --2- 172.21.15.28:0/1758159251 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0a300c15d0 0x7f0a30151e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:46.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.694+0000 7f0a36c00640 1 --2- 172.21.15.28:0/1758159251 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a3015b6d0 0x7f0a30142860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:46.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.694+0000 7f0a36c00640 1 --2- 172.21.15.28:0/1758159251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a300c11f0 0x7f0a300a7ec0 unknown :-1 s=CLOSED pgs=2577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:46.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.694+0000 7f0a36c00640 1 -- 172.21.15.28:0/1758159251 >> 172.21.15.28:0/1758159251 conn(0x7f0a300c67f0 msgr2=0x7f0a30149f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:15:46.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.694+0000 7f0a36c00640 1 -- 172.21.15.28:0/1758159251 shutdown_connections 2024-02-21T23:15:46.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:46.694+0000 7f0a36c00640 1 -- 172.21.15.28:0/1758159251 wait complete. 2024-02-21T23:15:47.105 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:15:47.105 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:15:48.106 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:15:48.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.405+0000 7fcea4c00640 1 Processor -- start 2024-02-21T23:15:48.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.405+0000 7fcea4c00640 1 -- start start 2024-02-21T23:15:48.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.407+0000 7fcea4c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcea0150030 0x7fcea0150490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:48.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.407+0000 7fcea4c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fcea0152c30 0x7fcea01530a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:48.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.407+0000 7fcea4c00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcea0070200 con 0x7fcea00c7890 2024-02-21T23:15:48.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.407+0000 7fcea4c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcea006bc70 con 0x7fcea0150030 2024-02-21T23:15:48.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.407+0000 7fcea4c00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcea006f440 con 0x7fcea0152c30 2024-02-21T23:15:48.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.408+0000 7fce9ea00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcea0150030 0x7fcea0150490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:48.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.408+0000 7fce9ea00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcea0150030 0x7fcea0150490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59306/0 (socket says 172.21.15.28:59306) 2024-02-21T23:15:48.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.408+0000 7fce9ea00640 1 -- 172.21.15.28:0/389083304 learned_addr learned my addr 172.21.15.28:0/389083304 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:15:48.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.408+0000 7fce9fe00640 1 --2- 172.21.15.28:0/389083304 >> v2:172.21.15.81:3300/0 conn(0x7fcea0152c30 0x7fcea01530a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:48.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.408+0000 7fce9ea00640 1 -- 172.21.15.28:0/389083304 >> v2:172.21.15.81:3300/0 conn(0x7fcea0152c30 msgr2=0x7fcea01530a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:15:48.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.408+0000 7fce9ea00640 1 --2- 172.21.15.28:0/389083304 >> v2:172.21.15.81:3300/0 conn(0x7fcea0152c30 0x7fcea01530a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:48.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.408+0000 7fce9ea00640 1 -- 172.21.15.28:0/389083304 >> v1:172.21.15.81:6789/0 conn(0x7fcea00c7890 legacy=0x7fcea006b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:15:48.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.408+0000 7fce9ea00640 1 -- 172.21.15.28:0/389083304 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcea006f720 con 0x7fcea0150030 2024-02-21T23:15:48.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.408+0000 7fce9fe00640 1 --2- 172.21.15.28:0/389083304 >> v2:172.21.15.81:3300/0 conn(0x7fcea0152c30 0x7fcea01530a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:15:48.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.409+0000 7fce9ea00640 1 --2- 172.21.15.28:0/389083304 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcea0150030 0x7fcea0150490 secure :-1 s=READY pgs=2578 cs=0 l=1 rev1=1 crypto rx=0x7fcea00a8e10 tx=0x7fce8c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1524fc2409e06be server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:48.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.410+0000 7fce9e000640 1 -- 172.21.15.28:0/389083304 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce8c02e070 con 0x7fcea0150030 2024-02-21T23:15:48.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.410+0000 7fce9e000640 1 -- 172.21.15.28:0/389083304 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce8c037070 con 0x7fcea0150030 2024-02-21T23:15:48.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.410+0000 7fce9e000640 1 -- 172.21.15.28:0/389083304 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce8c031a00 con 0x7fcea0150030 2024-02-21T23:15:48.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.410+0000 7fcea4c00640 1 -- 172.21.15.28:0/389083304 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcea0150030 msgr2=0x7fcea0150490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:48.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.410+0000 7fcea4c00640 1 --2- 172.21.15.28:0/389083304 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcea0150030 0x7fcea0150490 secure :-1 s=READY pgs=2578 cs=0 l=1 rev1=1 crypto rx=0x7fcea00a8e10 tx=0x7fce8c02b880 comp rx=0 tx=0).stop 2024-02-21T23:15:48.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.410+0000 7fcea4c00640 1 -- 172.21.15.28:0/389083304 shutdown_connections 2024-02-21T23:15:48.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.410+0000 7fcea4c00640 1 --2- 172.21.15.28:0/389083304 >> v2:172.21.15.81:3300/0 conn(0x7fcea0152c30 0x7fcea01530a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:48.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.410+0000 7fcea4c00640 1 --2- 172.21.15.28:0/389083304 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcea0150030 0x7fcea0150490 unknown :-1 s=CLOSED pgs=2578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:48.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.410+0000 7fcea4c00640 1 -- 172.21.15.28:0/389083304 >> 172.21.15.28:0/389083304 conn(0x7fcea0067050 msgr2=0x7fcea00a84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:15:48.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.411+0000 7fcea4c00640 1 -- 172.21.15.28:0/389083304 shutdown_connections 2024-02-21T23:15:48.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.411+0000 7fcea4c00640 1 -- 172.21.15.28:0/389083304 wait complete. 2024-02-21T23:15:48.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.413+0000 7fcea4c00640 1 Processor -- start 2024-02-21T23:15:48.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.413+0000 7fcea4c00640 1 -- start start 2024-02-21T23:15:48.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.415+0000 7fcea4c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcea00c7890 0x7fcea00c0d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:48.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.415+0000 7fcea4c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcea0150030 0x7fcea0145c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:48.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.415+0000 7fcea4c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcea0152c30 0x7fcea01461d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:48.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.415+0000 7fcea4c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea006b930 con 0x7fcea0152c30 2024-02-21T23:15:48.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.415+0000 7fcea4c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea006fce0 con 0x7fcea00c7890 2024-02-21T23:15:48.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.415+0000 7fcea4c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcea00c4d10 con 0x7fcea0150030 2024-02-21T23:15:48.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.415+0000 7fce9fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcea0152c30 0x7fcea01461d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:48.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.415+0000 7fce9fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcea0152c30 0x7fcea01461d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59316/0 (socket says 172.21.15.28:59316) 2024-02-21T23:15:48.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.415+0000 7fce9fe00640 1 -- 172.21.15.28:0/4226182358 learned_addr learned my addr 172.21.15.28:0/4226182358 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:15:48.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.416+0000 7fce9ea00640 1 --2- 172.21.15.28:0/4226182358 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcea0150030 0x7fcea0145c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:48.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.416+0000 7fce9f400640 1 --2- 172.21.15.28:0/4226182358 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcea00c7890 0x7fcea00c0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:48.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.416+0000 7fce9fe00640 1 -- 172.21.15.28:0/4226182358 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcea0150030 msgr2=0x7fcea0145c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:48.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.416+0000 7fce9fe00640 1 --2- 172.21.15.28:0/4226182358 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcea0150030 0x7fcea0145c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:48.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.416+0000 7fce9fe00640 1 -- 172.21.15.28:0/4226182358 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcea00c7890 msgr2=0x7fcea00c0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:48.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.416+0000 7fce9fe00640 1 --2- 172.21.15.28:0/4226182358 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcea00c7890 0x7fcea00c0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:48.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.416+0000 7fce9fe00640 1 -- 172.21.15.28:0/4226182358 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcea0147a50 con 0x7fcea0152c30 2024-02-21T23:15:48.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.416+0000 7fce9fe00640 1 --2- 172.21.15.28:0/4226182358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcea0152c30 0x7fcea01461d0 secure :-1 s=READY pgs=2579 cs=0 l=1 rev1=1 crypto rx=0x7fce9800cd50 tx=0x7fce9800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:48.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.417+0000 7fce97400640 1 -- 172.21.15.28:0/4226182358 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce9800f680 con 0x7fcea0152c30 2024-02-21T23:15:48.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.417+0000 7fce97400640 1 -- 172.21.15.28:0/4226182358 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce9800f840 con 0x7fcea0152c30 2024-02-21T23:15:48.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.417+0000 7fce97400640 1 -- 172.21.15.28:0/4226182358 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce9800fc30 con 0x7fcea0152c30 2024-02-21T23:15:48.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.417+0000 7fcea4c00640 1 -- 172.21.15.28:0/4226182358 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcea01eb230 con 0x7fcea0152c30 2024-02-21T23:15:48.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.417+0000 7fcea4c00640 1 -- 172.21.15.28:0/4226182358 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcea00c23a0 con 0x7fcea0152c30 2024-02-21T23:15:48.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.418+0000 7fcea4c00640 1 -- 172.21.15.28:0/4226182358 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcea004f290 con 0x7fcea0152c30 2024-02-21T23:15:48.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.419+0000 7fce97400640 1 -- 172.21.15.28:0/4226182358 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fce9800e3e0 con 0x7fcea0152c30 2024-02-21T23:15:48.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.420+0000 7fce97400640 1 --2- 172.21.15.28:0/4226182358 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fce8006b4d0 0x7fce8006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:48.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.420+0000 7fce97400640 1 -- 172.21.15.28:0/4226182358 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fce9808b180 con 0x7fcea0152c30 2024-02-21T23:15:48.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.421+0000 7fce9f400640 1 --2- 172.21.15.28:0/4226182358 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fce8006b4d0 0x7fce8006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:48.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.422+0000 7fce9f400640 1 --2- 172.21.15.28:0/4226182358 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fce8006b4d0 0x7fce8006d990 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fce90007fd0 tx=0x7fce90009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:48.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.425+0000 7fce97400640 1 -- 172.21.15.28:0/4226182358 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fce98014030 con 0x7fcea0152c30 2024-02-21T23:15:48.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.760+0000 7fcea4c00640 1 -- 172.21.15.28:0/4226182358 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcea006b610 con 0x7fcea0152c30 2024-02-21T23:15:48.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.760+0000 7fce97400640 1 -- 172.21.15.28:0/4226182358 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7fce98059e00 con 0x7fcea0152c30 2024-02-21T23:15:48.748 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:15:48.748 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:15:48.749 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:15:48.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.763+0000 7fcea4c00640 1 -- 172.21.15.28:0/4226182358 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fce8006b4d0 msgr2=0x7fce8006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:48.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.763+0000 7fcea4c00640 1 --2- 172.21.15.28:0/4226182358 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fce8006b4d0 0x7fce8006d990 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fce90007fd0 tx=0x7fce90009040 comp rx=0 tx=0).stop 2024-02-21T23:15:48.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.763+0000 7fcea4c00640 1 -- 172.21.15.28:0/4226182358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcea0152c30 msgr2=0x7fcea01461d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:48.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.763+0000 7fcea4c00640 1 --2- 172.21.15.28:0/4226182358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcea0152c30 0x7fcea01461d0 secure :-1 s=READY pgs=2579 cs=0 l=1 rev1=1 crypto rx=0x7fce9800cd50 tx=0x7fce9800a650 comp rx=0 tx=0).stop 2024-02-21T23:15:48.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.763+0000 7fcea4c00640 1 -- 172.21.15.28:0/4226182358 shutdown_connections 2024-02-21T23:15:48.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.763+0000 7fcea4c00640 1 --2- 172.21.15.28:0/4226182358 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fce8006b4d0 0x7fce8006d990 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:48.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.763+0000 7fcea4c00640 1 --2- 172.21.15.28:0/4226182358 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcea0150030 0x7fcea0145c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:48.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.763+0000 7fcea4c00640 1 --2- 172.21.15.28:0/4226182358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcea0152c30 0x7fcea01461d0 unknown :-1 s=CLOSED pgs=2579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:48.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.763+0000 7fcea4c00640 1 --2- 172.21.15.28:0/4226182358 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcea00c7890 0x7fcea00c0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:48.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.763+0000 7fcea4c00640 1 -- 172.21.15.28:0/4226182358 >> 172.21.15.28:0/4226182358 conn(0x7fcea0067050 msgr2=0x7fcea0152130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:15:48.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.763+0000 7fcea4c00640 1 -- 172.21.15.28:0/4226182358 shutdown_connections 2024-02-21T23:15:48.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:48.763+0000 7fcea4c00640 1 -- 172.21.15.28:0/4226182358 wait complete. 2024-02-21T23:15:49.171 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:15:49.172 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:15:50.173 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:15:50.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.405+0000 7f1ff6200640 1 Processor -- start 2024-02-21T23:15:50.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.405+0000 7f1ff6200640 1 -- start start 2024-02-21T23:15:50.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.407+0000 7f1ff6200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f1ff00c8a80 0x7f1ff0156c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:50.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.407+0000 7f1ff6200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1ff00c8e60 0x7f1ff0153640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:50.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.407+0000 7f1ff6200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ff0153b80 con 0x7f1ff01540b0 2024-02-21T23:15:50.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.407+0000 7f1ff6200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ff00c92e0 con 0x7f1ff00c8a80 2024-02-21T23:15:50.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.407+0000 7f1ff6200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff00c8000 con 0x7f1ff00c8e60 2024-02-21T23:15:50.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.408+0000 7f1ff5800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f1ff01540b0 0x7f1ff00705a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55084/0 (socket says 172.21.15.28:55084) 2024-02-21T23:15:50.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.408+0000 7f1ff5800640 1 -- 172.21.15.28:0/1218769702 learned_addr learned my addr 172.21.15.28:0/1218769702 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:15:50.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.408+0000 7f1ff4e00640 1 --2- 172.21.15.28:0/1218769702 >> v2:172.21.15.81:3300/0 conn(0x7f1ff00c8a80 0x7f1ff0156c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:50.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.408+0000 7f1fefe00640 1 --2- 172.21.15.28:0/1218769702 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1ff00c8e60 0x7f1ff0153640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:50.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.408+0000 7f1fef400640 1 -- 172.21.15.28:0/1218769702 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2024543133 0 0) 0x7f1fe8001490 con 0x7f1ff01540b0 2024-02-21T23:15:50.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.408+0000 7f1fef400640 1 -- 172.21.15.28:0/1218769702 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fd8003520 con 0x7f1ff01540b0 2024-02-21T23:15:50.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.408+0000 7f1fef400640 1 -- 172.21.15.28:0/1218769702 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 988947871 0 0) 0x7f1fd8003520 con 0x7f1ff01540b0 2024-02-21T23:15:50.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.408+0000 7f1fef400640 1 -- 172.21.15.28:0/1218769702 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1ff00c8e60 msgr2=0x7f1ff0153640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:15:50.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.408+0000 7f1fef400640 1 --2- 172.21.15.28:0/1218769702 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1ff00c8e60 0x7f1ff0153640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:50.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.408+0000 7f1fef400640 1 -- 172.21.15.28:0/1218769702 >> v2:172.21.15.81:3300/0 conn(0x7f1ff00c8a80 msgr2=0x7f1ff0156c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:15:50.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.408+0000 7f1fef400640 1 --2- 172.21.15.28:0/1218769702 >> v2:172.21.15.81:3300/0 conn(0x7f1ff00c8a80 0x7f1ff0156c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:50.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.409+0000 7f1fef400640 1 -- 172.21.15.28:0/1218769702 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ff00c82e0 con 0x7f1ff01540b0 2024-02-21T23:15:50.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.409+0000 7f1fef400640 1 -- 172.21.15.28:0/1218769702 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f1fe8001bc0 con 0x7f1ff01540b0 2024-02-21T23:15:50.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.409+0000 7f1fef400640 1 -- 172.21.15.28:0/1218769702 >> v1:172.21.15.28:6789/0 conn(0x7f1ff01540b0 legacy=0x7f1ff00705a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:50.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.409+0000 7f1fef400640 1 --2- 172.21.15.28:0/1218769702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fd80039b0 0x7f1fd8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:50.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.409+0000 7f1fef400640 1 --2- 172.21.15.28:0/1218769702 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1ff00c8e60 0x7f1fd8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:50.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.409+0000 7f1fef400640 1 --2- 172.21.15.28:0/1218769702 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1fd8004d40 0x7f1fd80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:50.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.409+0000 7f1fef400640 1 -- 172.21.15.28:0/1218769702 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff00c92e0 con 0x7f1fd80039b0 2024-02-21T23:15:50.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.409+0000 7f1fef400640 1 -- 172.21.15.28:0/1218769702 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff00c8000 con 0x7f1fd8004d40 2024-02-21T23:15:50.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.409+0000 7f1fef400640 1 -- 172.21.15.28:0/1218769702 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd800b9c0 con 0x7f1ff00c8e60 2024-02-21T23:15:50.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.409+0000 7f1ff4e00640 1 --2- 172.21.15.28:0/1218769702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fd80039b0 0x7f1fd8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:50.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.409+0000 7f1ff5800640 1 --2- 172.21.15.28:0/1218769702 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1fd8004d40 0x7f1fd80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:50.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.409+0000 7f1ff4e00640 1 -- 172.21.15.28:0/1218769702 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1ff00c8e60 msgr2=0x7f1fd8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:15:50.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.409+0000 7f1ff4e00640 1 --2- 172.21.15.28:0/1218769702 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1ff00c8e60 0x7f1fd8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:50.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.409+0000 7f1ff4e00640 1 -- 172.21.15.28:0/1218769702 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1fd8004d40 msgr2=0x7f1fd80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:15:50.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.409+0000 7f1ff4e00640 1 --2- 172.21.15.28:0/1218769702 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1fd8004d40 0x7f1fd80051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:50.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.409+0000 7f1ff4e00640 1 -- 172.21.15.28:0/1218769702 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fd8003520 con 0x7f1fd80039b0 2024-02-21T23:15:50.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.411+0000 7f1ff4e00640 1 --2- 172.21.15.28:0/1218769702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fd80039b0 0x7f1fd8003dd0 secure :-1 s=READY pgs=2581 cs=0 l=1 rev1=1 crypto rx=0x7f1fe4009ac0 tx=0x7f1fd8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b89bf58d2bbf2df5 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:50.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.411+0000 7f1fef400640 1 -- 172.21.15.28:0/1218769702 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fe4039070 con 0x7f1fd80039b0 2024-02-21T23:15:50.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.411+0000 7f1fef400640 1 -- 172.21.15.28:0/1218769702 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fe402c710 con 0x7f1fd80039b0 2024-02-21T23:15:50.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.411+0000 7f1fef400640 1 -- 172.21.15.28:0/1218769702 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fe4009690 con 0x7f1fd80039b0 2024-02-21T23:15:50.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.411+0000 7f1ff6200640 1 -- 172.21.15.28:0/1218769702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fd80039b0 msgr2=0x7f1fd8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:50.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.411+0000 7f1ff6200640 1 --2- 172.21.15.28:0/1218769702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fd80039b0 0x7f1fd8003dd0 secure :-1 s=READY pgs=2581 cs=0 l=1 rev1=1 crypto rx=0x7f1fe4009ac0 tx=0x7f1fd8005830 comp rx=0 tx=0).stop 2024-02-21T23:15:50.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.411+0000 7f1ff5800640 1 -- 172.21.15.28:0/1218769702 reap_dead start 2024-02-21T23:15:50.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.412+0000 7f1ff6200640 1 -- 172.21.15.28:0/1218769702 shutdown_connections 2024-02-21T23:15:50.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.412+0000 7f1ff6200640 1 -- 172.21.15.28:0/1218769702 >> 172.21.15.28:0/1218769702 conn(0x7f1ff006b260 msgr2=0x7f1ff006b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:15:50.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.412+0000 7f1ff6200640 1 -- 172.21.15.28:0/1218769702 shutdown_connections 2024-02-21T23:15:50.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.412+0000 7f1ff6200640 1 -- 172.21.15.28:0/1218769702 wait complete. 2024-02-21T23:15:50.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.414+0000 7f1ff6200640 1 Processor -- start 2024-02-21T23:15:50.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.414+0000 7f1ff6200640 1 -- start start 2024-02-21T23:15:50.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.416+0000 7f1ff6200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1fd80039b0 0x7f1ff014e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:50.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.416+0000 7f1ff6200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ff014ced0 0x7f1ff0151e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:50.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.416+0000 7f1ff6200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1ff01ee2d0 0x7f1ff01ee750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:50.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.416+0000 7f1ff6200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff0156770 con 0x7f1ff014ced0 2024-02-21T23:15:50.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.416+0000 7f1ff6200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff0156490 con 0x7f1fd80039b0 2024-02-21T23:15:50.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.416+0000 7f1ff6200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff00c7c90 con 0x7f1ff01ee2d0 2024-02-21T23:15:50.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.416+0000 7f1fefe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ff014ced0 0x7f1ff0151e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:50.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.416+0000 7f1fefe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ff014ced0 0x7f1ff0151e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59332/0 (socket says 172.21.15.28:59332) 2024-02-21T23:15:50.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.416+0000 7f1fefe00640 1 -- 172.21.15.28:0/1300621992 learned_addr learned my addr 172.21.15.28:0/1300621992 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:15:50.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.416+0000 7f1ff4e00640 1 --2- 172.21.15.28:0/1300621992 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1fd80039b0 0x7f1ff014e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:50.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.416+0000 7f1ff5800640 1 --2- 172.21.15.28:0/1300621992 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1ff01ee2d0 0x7f1ff01ee750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:50.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.416+0000 7f1fefe00640 1 -- 172.21.15.28:0/1300621992 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1ff01ee2d0 msgr2=0x7f1ff01ee750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:50.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.416+0000 7f1fefe00640 1 --2- 172.21.15.28:0/1300621992 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1ff01ee2d0 0x7f1ff01ee750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:50.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.416+0000 7f1fefe00640 1 -- 172.21.15.28:0/1300621992 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1fd80039b0 msgr2=0x7f1ff014e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:50.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.416+0000 7f1fefe00640 1 --2- 172.21.15.28:0/1300621992 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1fd80039b0 0x7f1ff014e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:50.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.416+0000 7f1fefe00640 1 -- 172.21.15.28:0/1300621992 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ff014f930 con 0x7f1ff014ced0 2024-02-21T23:15:50.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.417+0000 7f1fefe00640 1 --2- 172.21.15.28:0/1300621992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ff014ced0 0x7f1ff0151e20 secure :-1 s=READY pgs=2582 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc00e7f0 tx=0x7f1fdc00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:50.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.417+0000 7f1fecc00640 1 -- 172.21.15.28:0/1300621992 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc017070 con 0x7f1ff014ced0 2024-02-21T23:15:50.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.417+0000 7f1fecc00640 1 -- 172.21.15.28:0/1300621992 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fdc00c940 con 0x7f1ff014ced0 2024-02-21T23:15:50.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.417+0000 7f1fecc00640 1 -- 172.21.15.28:0/1300621992 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc00cd10 con 0x7f1ff014ced0 2024-02-21T23:15:50.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.417+0000 7f1ff6200640 1 -- 172.21.15.28:0/1300621992 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ff014d440 con 0x7f1ff014ced0 2024-02-21T23:15:50.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.417+0000 7f1ff6200640 1 -- 172.21.15.28:0/1300621992 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ff014f130 con 0x7f1ff014ced0 2024-02-21T23:15:50.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.419+0000 7f1ff6200640 1 -- 172.21.15.28:0/1300621992 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ff004f290 con 0x7f1ff014ced0 2024-02-21T23:15:50.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.420+0000 7f1fecc00640 1 -- 172.21.15.28:0/1300621992 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1fdc01a070 con 0x7f1ff014ced0 2024-02-21T23:15:50.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.420+0000 7f1fecc00640 1 --2- 172.21.15.28:0/1300621992 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1fd006b4d0 0x7f1fd006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:50.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.421+0000 7f1fecc00640 1 -- 172.21.15.28:0/1300621992 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f1fdc08a350 con 0x7f1ff014ced0 2024-02-21T23:15:50.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.421+0000 7f1ff4e00640 1 --2- 172.21.15.28:0/1300621992 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1fd006b4d0 0x7f1fd006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:50.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.423+0000 7f1ff4e00640 1 --2- 172.21.15.28:0/1300621992 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1fd006b4d0 0x7f1fd006d990 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f1fe4036000 tx=0x7f1fe4030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:50.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.425+0000 7f1fecc00640 1 -- 172.21.15.28:0/1300621992 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1fdc058f40 con 0x7f1ff014ced0 2024-02-21T23:15:50.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.665+0000 7f1ff6200640 1 -- 172.21.15.28:0/1300621992 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1ff00c5f30 con 0x7f1ff014ced0 2024-02-21T23:15:50.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.666+0000 7f1fecc00640 1 -- 172.21.15.28:0/1300621992 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7f1fdc058840 con 0x7f1ff014ced0 2024-02-21T23:15:50.655 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:15:50.655 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:15:50.655 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:15:50.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.669+0000 7f1ff6200640 1 -- 172.21.15.28:0/1300621992 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1fd006b4d0 msgr2=0x7f1fd006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:50.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.669+0000 7f1ff6200640 1 --2- 172.21.15.28:0/1300621992 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1fd006b4d0 0x7f1fd006d990 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f1fe4036000 tx=0x7f1fe4030000 comp rx=0 tx=0).stop 2024-02-21T23:15:50.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.669+0000 7f1ff6200640 1 -- 172.21.15.28:0/1300621992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ff014ced0 msgr2=0x7f1ff0151e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:50.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.669+0000 7f1ff6200640 1 --2- 172.21.15.28:0/1300621992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ff014ced0 0x7f1ff0151e20 secure :-1 s=READY pgs=2582 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc00e7f0 tx=0x7f1fdc00ecb0 comp rx=0 tx=0).stop 2024-02-21T23:15:50.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.669+0000 7f1ff6200640 1 -- 172.21.15.28:0/1300621992 shutdown_connections 2024-02-21T23:15:50.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.669+0000 7f1ff6200640 1 --2- 172.21.15.28:0/1300621992 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1fd006b4d0 0x7f1fd006d990 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:50.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.669+0000 7f1ff6200640 1 --2- 172.21.15.28:0/1300621992 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1ff01ee2d0 0x7f1ff01ee750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:50.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.669+0000 7f1ff6200640 1 --2- 172.21.15.28:0/1300621992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ff014ced0 0x7f1ff0151e20 unknown :-1 s=CLOSED pgs=2582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:50.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.669+0000 7f1ff6200640 1 --2- 172.21.15.28:0/1300621992 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1fd80039b0 0x7f1ff014e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:50.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.669+0000 7f1ff6200640 1 -- 172.21.15.28:0/1300621992 >> 172.21.15.28:0/1300621992 conn(0x7f1ff006b260 msgr2=0x7f1ff00c2350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:15:50.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.669+0000 7f1ff6200640 1 -- 172.21.15.28:0/1300621992 shutdown_connections 2024-02-21T23:15:50.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:50.669+0000 7f1ff6200640 1 -- 172.21.15.28:0/1300621992 wait complete. 2024-02-21T23:15:51.079 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:15:51.079 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:15:52.080 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:15:52.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.323+0000 7f7f8a800640 1 Processor -- start 2024-02-21T23:15:52.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.324+0000 7f7f8a800640 1 -- start start 2024-02-21T23:15:52.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.325+0000 7f7f8a800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7f840aff90 0x7f7f840c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:52.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.325+0000 7f7f8a800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f840c74c0 con 0x7f7f8406dbf0 2024-02-21T23:15:52.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.325+0000 7f7f8a800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f840c77a0 con 0x7f7f840c5720 2024-02-21T23:15:52.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.325+0000 7f7f8a800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f840c7270 con 0x7f7f840aff90 2024-02-21T23:15:52.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.325+0000 7f7f88a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f7f8406dbf0 0x7f7f84158ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55090/0 (socket says 172.21.15.28:55090) 2024-02-21T23:15:52.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.325+0000 7f7f88a00640 1 -- 172.21.15.28:0/3383457614 learned_addr learned my addr 172.21.15.28:0/3383457614 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:15:52.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.326+0000 7f7f89400640 1 --2- 172.21.15.28:0/3383457614 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7f840aff90 0x7f7f840c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:52.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.326+0000 7f7f83e00640 1 -- 172.21.15.28:0/3383457614 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 484722271 0 0) 0x7f7f840c74c0 con 0x7f7f8406dbf0 2024-02-21T23:15:52.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.326+0000 7f7f83e00640 1 -- 172.21.15.28:0/3383457614 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f6c003520 con 0x7f7f8406dbf0 2024-02-21T23:15:52.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.326+0000 7f7f83e00640 1 -- 172.21.15.28:0/3383457614 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2681005617 0 0) 0x7f7f6c003520 con 0x7f7f8406dbf0 2024-02-21T23:15:52.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.326+0000 7f7f83e00640 1 -- 172.21.15.28:0/3383457614 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7f840aff90 msgr2=0x7f7f840c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:15:52.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.326+0000 7f7f83e00640 1 --2- 172.21.15.28:0/3383457614 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7f840aff90 0x7f7f840c51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:52.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.326+0000 7f7f83e00640 1 -- 172.21.15.28:0/3383457614 >> v1:172.21.15.81:6789/0 conn(0x7f7f840c5720 legacy=0x7f7f8415f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:52.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.326+0000 7f7f83e00640 1 -- 172.21.15.28:0/3383457614 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f840c7a80 con 0x7f7f8406dbf0 2024-02-21T23:15:52.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.326+0000 7f7f83e00640 1 -- 172.21.15.28:0/3383457614 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f7f70001670 con 0x7f7f8406dbf0 2024-02-21T23:15:52.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.326+0000 7f7f83e00640 1 -- 172.21.15.28:0/3383457614 >> v1:172.21.15.28:6789/0 conn(0x7f7f8406dbf0 legacy=0x7f7f84158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:52.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.327+0000 7f7f83e00640 1 --2- 172.21.15.28:0/3383457614 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7f840aff90 0x7f7f6c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:52.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.327+0000 7f7f83e00640 1 --2- 172.21.15.28:0/3383457614 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f6c003f60 0x7f7f6c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:52.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.327+0000 7f7f83e00640 1 --2- 172.21.15.28:0/3383457614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f6c004d40 0x7f7f6c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:52.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.327+0000 7f7f83e00640 1 -- 172.21.15.28:0/3383457614 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f840c7270 con 0x7f7f6c004d40 2024-02-21T23:15:52.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.327+0000 7f7f83e00640 1 -- 172.21.15.28:0/3383457614 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f6c00baa0 con 0x7f7f6c003f60 2024-02-21T23:15:52.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.327+0000 7f7f83e00640 1 -- 172.21.15.28:0/3383457614 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f6c00bca0 con 0x7f7f840aff90 2024-02-21T23:15:52.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.327+0000 7f7f89e00640 1 --2- 172.21.15.28:0/3383457614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f6c004d40 0x7f7f6c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:52.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.327+0000 7f7f89400640 1 --2- 172.21.15.28:0/3383457614 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7f840aff90 0x7f7f6c0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:52.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.327+0000 7f7f88a00640 1 --2- 172.21.15.28:0/3383457614 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f6c003f60 0x7f7f6c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:52.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.327+0000 7f7f89e00640 1 -- 172.21.15.28:0/3383457614 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7f840aff90 msgr2=0x7f7f6c0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:15:52.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.327+0000 7f7f89e00640 1 --2- 172.21.15.28:0/3383457614 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7f840aff90 0x7f7f6c0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:52.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.327+0000 7f7f89e00640 1 -- 172.21.15.28:0/3383457614 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f6c003f60 msgr2=0x7f7f6c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:15:52.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.327+0000 7f7f89e00640 1 --2- 172.21.15.28:0/3383457614 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f6c003f60 0x7f7f6c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:52.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.327+0000 7f7f89e00640 1 -- 172.21.15.28:0/3383457614 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f6c003520 con 0x7f7f6c004d40 2024-02-21T23:15:52.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.327+0000 7f7f88a00640 1 --2- 172.21.15.28:0/3383457614 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f6c003f60 0x7f7f6c004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:15:52.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.328+0000 7f7f89e00640 1 --2- 172.21.15.28:0/3383457614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f6c004d40 0x7f7f6c0051c0 secure :-1 s=READY pgs=2584 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c001e50 tx=0x7f7f6c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a7951dabbf7244d server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:52.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.329+0000 7f7f83e00640 1 -- 172.21.15.28:0/3383457614 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f7c001a70 con 0x7f7f6c004d40 2024-02-21T23:15:52.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.329+0000 7f7f83e00640 1 -- 172.21.15.28:0/3383457614 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f7c003b70 con 0x7f7f6c004d40 2024-02-21T23:15:52.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.329+0000 7f7f83e00640 1 -- 172.21.15.28:0/3383457614 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f7c0050d0 con 0x7f7f6c004d40 2024-02-21T23:15:52.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.329+0000 7f7f8a800640 1 -- 172.21.15.28:0/3383457614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f6c004d40 msgr2=0x7f7f6c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:52.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.329+0000 7f7f8a800640 1 --2- 172.21.15.28:0/3383457614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f6c004d40 0x7f7f6c0051c0 secure :-1 s=READY pgs=2584 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c001e50 tx=0x7f7f6c005830 comp rx=0 tx=0).stop 2024-02-21T23:15:52.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.329+0000 7f7f89e00640 1 -- 172.21.15.28:0/3383457614 reap_dead start 2024-02-21T23:15:52.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.330+0000 7f7f8a800640 1 -- 172.21.15.28:0/3383457614 shutdown_connections 2024-02-21T23:15:52.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.330+0000 7f7f8a800640 1 -- 172.21.15.28:0/3383457614 >> 172.21.15.28:0/3383457614 conn(0x7f7f840ac370 msgr2=0x7f7f840ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:15:52.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.330+0000 7f7f8a800640 1 -- 172.21.15.28:0/3383457614 shutdown_connections 2024-02-21T23:15:52.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.330+0000 7f7f8a800640 1 -- 172.21.15.28:0/3383457614 wait complete. 2024-02-21T23:15:52.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.331+0000 7f7f8a800640 1 Processor -- start 2024-02-21T23:15:52.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.332+0000 7f7f8a800640 1 -- start start 2024-02-21T23:15:52.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.333+0000 7f7f8a800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f6c004d40 0x7f7f84151690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:52.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.333+0000 7f7f8a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f84154c30 0x7f7f840c2900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:52.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.333+0000 7f7f8a800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7f84151bd0 0x7f7f840c2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:52.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.333+0000 7f7f8a800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f840c8760 con 0x7f7f84154c30 2024-02-21T23:15:52.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.333+0000 7f7f8a800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f84158630 con 0x7f7f6c004d40 2024-02-21T23:15:52.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.333+0000 7f7f8a800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f840c88e0 con 0x7f7f84151bd0 2024-02-21T23:15:52.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.333+0000 7f7f88a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f84154c30 0x7f7f840c2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:52.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.333+0000 7f7f88a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f84154c30 0x7f7f840c2900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59350/0 (socket says 172.21.15.28:59350) 2024-02-21T23:15:52.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.333+0000 7f7f88a00640 1 -- 172.21.15.28:0/2776465294 learned_addr learned my addr 172.21.15.28:0/2776465294 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:15:52.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.333+0000 7f7f89400640 1 --2- 172.21.15.28:0/2776465294 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f6c004d40 0x7f7f84151690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:52.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.333+0000 7f7f89e00640 1 --2- 172.21.15.28:0/2776465294 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7f84151bd0 0x7f7f840c2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:52.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.333+0000 7f7f88a00640 1 -- 172.21.15.28:0/2776465294 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7f84151bd0 msgr2=0x7f7f840c2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:52.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.333+0000 7f7f88a00640 1 --2- 172.21.15.28:0/2776465294 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7f84151bd0 0x7f7f840c2e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:52.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.333+0000 7f7f88a00640 1 -- 172.21.15.28:0/2776465294 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f6c004d40 msgr2=0x7f7f84151690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:52.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.333+0000 7f7f88a00640 1 --2- 172.21.15.28:0/2776465294 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f6c004d40 0x7f7f84151690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:52.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.333+0000 7f7f88a00640 1 -- 172.21.15.28:0/2776465294 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f840c3ef0 con 0x7f7f84154c30 2024-02-21T23:15:52.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.333+0000 7f7f89400640 1 --2- 172.21.15.28:0/2776465294 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f6c004d40 0x7f7f84151690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:15:52.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.334+0000 7f7f88a00640 1 --2- 172.21.15.28:0/2776465294 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f84154c30 0x7f7f840c2900 secure :-1 s=READY pgs=2585 cs=0 l=1 rev1=1 crypto rx=0x7f7f7000d8e0 tx=0x7f7f7000dda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:52.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.334+0000 7f7f81600640 1 -- 172.21.15.28:0/2776465294 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f70019070 con 0x7f7f84154c30 2024-02-21T23:15:52.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.334+0000 7f7f81600640 1 -- 172.21.15.28:0/2776465294 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f700074e0 con 0x7f7f84154c30 2024-02-21T23:15:52.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.334+0000 7f7f8a800640 1 -- 172.21.15.28:0/2776465294 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f840c3930 con 0x7f7f84154c30 2024-02-21T23:15:52.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.334+0000 7f7f81600640 1 -- 172.21.15.28:0/2776465294 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f70007690 con 0x7f7f84154c30 2024-02-21T23:15:52.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.334+0000 7f7f8a800640 1 -- 172.21.15.28:0/2776465294 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f840c23e0 con 0x7f7f84154c30 2024-02-21T23:15:52.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.335+0000 7f7f8a800640 1 -- 172.21.15.28:0/2776465294 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f8404f290 con 0x7f7f84154c30 2024-02-21T23:15:52.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.336+0000 7f7f81600640 1 -- 172.21.15.28:0/2776465294 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7f7001c070 con 0x7f7f84154c30 2024-02-21T23:15:52.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.337+0000 7f7f81600640 1 --2- 172.21.15.28:0/2776465294 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7f6406b4d0 0x7f7f6406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:52.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.337+0000 7f7f81600640 1 -- 172.21.15.28:0/2776465294 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f7f7008a180 con 0x7f7f84154c30 2024-02-21T23:15:52.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.338+0000 7f7f89400640 1 --2- 172.21.15.28:0/2776465294 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7f6406b4d0 0x7f7f6406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:52.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.339+0000 7f7f89400640 1 --2- 172.21.15.28:0/2776465294 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7f6406b4d0 0x7f7f6406d990 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f7f78009a00 tx=0x7f7f78008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:52.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.342+0000 7f7f81600640 1 -- 172.21.15.28:0/2776465294 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7f70058d60 con 0x7f7f84154c30 2024-02-21T23:15:52.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.582+0000 7f7f8a800640 1 -- 172.21.15.28:0/2776465294 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7f8406d9d0 con 0x7f7f84154c30 2024-02-21T23:15:52.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.582+0000 7f7f81600640 1 -- 172.21.15.28:0/2776465294 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7f7f70058660 con 0x7f7f84154c30 2024-02-21T23:15:52.571 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:15:52.571 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:15:52.571 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:15:52.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.585+0000 7f7f8a800640 1 -- 172.21.15.28:0/2776465294 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7f6406b4d0 msgr2=0x7f7f6406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:52.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.585+0000 7f7f8a800640 1 --2- 172.21.15.28:0/2776465294 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7f6406b4d0 0x7f7f6406d990 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f7f78009a00 tx=0x7f7f78008040 comp rx=0 tx=0).stop 2024-02-21T23:15:52.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.585+0000 7f7f8a800640 1 -- 172.21.15.28:0/2776465294 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f84154c30 msgr2=0x7f7f840c2900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:52.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.585+0000 7f7f8a800640 1 --2- 172.21.15.28:0/2776465294 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f84154c30 0x7f7f840c2900 secure :-1 s=READY pgs=2585 cs=0 l=1 rev1=1 crypto rx=0x7f7f7000d8e0 tx=0x7f7f7000dda0 comp rx=0 tx=0).stop 2024-02-21T23:15:52.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.585+0000 7f7f8a800640 1 -- 172.21.15.28:0/2776465294 shutdown_connections 2024-02-21T23:15:52.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.585+0000 7f7f8a800640 1 --2- 172.21.15.28:0/2776465294 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7f6406b4d0 0x7f7f6406d990 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:52.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.585+0000 7f7f8a800640 1 --2- 172.21.15.28:0/2776465294 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7f84151bd0 0x7f7f840c2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:52.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.585+0000 7f7f8a800640 1 --2- 172.21.15.28:0/2776465294 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f84154c30 0x7f7f840c2900 unknown :-1 s=CLOSED pgs=2585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:52.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.585+0000 7f7f8a800640 1 --2- 172.21.15.28:0/2776465294 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7f6c004d40 0x7f7f84151690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:52.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.585+0000 7f7f8a800640 1 -- 172.21.15.28:0/2776465294 >> 172.21.15.28:0/2776465294 conn(0x7f7f840ac370 msgr2=0x7f7f84156300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:15:52.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.585+0000 7f7f8a800640 1 -- 172.21.15.28:0/2776465294 shutdown_connections 2024-02-21T23:15:52.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:52.585+0000 7f7f8a800640 1 -- 172.21.15.28:0/2776465294 wait complete. 2024-02-21T23:15:52.994 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:15:52.994 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:15:53.995 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:15:54.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.231+0000 7f8b4d200640 1 Processor -- start 2024-02-21T23:15:54.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.231+0000 7f8b4d200640 1 -- start start 2024-02-21T23:15:54.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.233+0000 7f8b4d200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f8b480c4b00 0x7f8b48158cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:54.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.233+0000 7f8b4d200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b480c68d0 con 0x7f8b480c4590 2024-02-21T23:15:54.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.233+0000 7f8b4d200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b480c6bb0 con 0x7f8b48152770 2024-02-21T23:15:54.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.233+0000 7f8b4d200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b480c7200 con 0x7f8b480c4b00 2024-02-21T23:15:54.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.233+0000 7f8b3fe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f8b480c4590 0x7f8b481585c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55102/0 (socket says 172.21.15.28:55102) 2024-02-21T23:15:54.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.233+0000 7f8b3fe00640 1 -- 172.21.15.28:0/3483350603 learned_addr learned my addr 172.21.15.28:0/3483350603 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:15:54.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.234+0000 7f8b46800640 1 --2- 172.21.15.28:0/3483350603 >> v2:172.21.15.81:3300/0 conn(0x7f8b480c4b00 0x7f8b48158cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:54.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.234+0000 7f8b45400640 1 -- 172.21.15.28:0/3483350603 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2807191066 0 0) 0x7f8b480c68d0 con 0x7f8b480c4590 2024-02-21T23:15:54.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.234+0000 7f8b45400640 1 -- 172.21.15.28:0/3483350603 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b2c003520 con 0x7f8b480c4590 2024-02-21T23:15:54.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.234+0000 7f8b45400640 1 -- 172.21.15.28:0/3483350603 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1102240154 0 0) 0x7f8b2c003520 con 0x7f8b480c4590 2024-02-21T23:15:54.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.234+0000 7f8b45400640 1 -- 172.21.15.28:0/3483350603 >> v2:172.21.15.81:3300/0 conn(0x7f8b480c4b00 msgr2=0x7f8b48158cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:15:54.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.234+0000 7f8b45400640 1 --2- 172.21.15.28:0/3483350603 >> v2:172.21.15.81:3300/0 conn(0x7f8b480c4b00 0x7f8b48158cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:54.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.234+0000 7f8b45400640 1 -- 172.21.15.28:0/3483350603 >> v1:172.21.15.81:6789/0 conn(0x7f8b48152770 legacy=0x7f8b480c31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:54.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.234+0000 7f8b45400640 1 -- 172.21.15.28:0/3483350603 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b480c6e90 con 0x7f8b480c4590 2024-02-21T23:15:54.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.234+0000 7f8b45400640 1 -- 172.21.15.28:0/3483350603 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f8b30001670 con 0x7f8b480c4590 2024-02-21T23:15:54.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.234+0000 7f8b45400640 1 -- 172.21.15.28:0/3483350603 >> v1:172.21.15.28:6789/0 conn(0x7f8b480c4590 legacy=0x7f8b481585c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:54.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.234+0000 7f8b45400640 1 --2- 172.21.15.28:0/3483350603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b2c0039b0 0x7f8b2c003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:54.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.235+0000 7f8b45400640 1 --2- 172.21.15.28:0/3483350603 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8b2c004800 0x7f8b2c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:54.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.235+0000 7f8b45400640 1 --2- 172.21.15.28:0/3483350603 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b2c0051c0 0x7f8b2c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:54.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.235+0000 7f8b45400640 1 -- 172.21.15.28:0/3483350603 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b480c7200 con 0x7f8b2c0039b0 2024-02-21T23:15:54.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.235+0000 7f8b45400640 1 -- 172.21.15.28:0/3483350603 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b2c00c090 con 0x7f8b2c0051c0 2024-02-21T23:15:54.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.235+0000 7f8b45400640 1 -- 172.21.15.28:0/3483350603 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b2c00c290 con 0x7f8b2c004800 2024-02-21T23:15:54.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.235+0000 7f8b45e00640 1 --2- 172.21.15.28:0/3483350603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b2c0039b0 0x7f8b2c003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:54.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.235+0000 7f8b3fe00640 1 --2- 172.21.15.28:0/3483350603 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8b2c004800 0x7f8b2c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:54.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.235+0000 7f8b46800640 1 --2- 172.21.15.28:0/3483350603 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b2c0051c0 0x7f8b2c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:54.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.235+0000 7f8b45e00640 1 -- 172.21.15.28:0/3483350603 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8b2c004800 msgr2=0x7f8b2c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:15:54.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.235+0000 7f8b45e00640 1 --2- 172.21.15.28:0/3483350603 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8b2c004800 0x7f8b2c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:54.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.236+0000 7f8b45e00640 1 -- 172.21.15.28:0/3483350603 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b2c0051c0 msgr2=0x7f8b2c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:15:54.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.236+0000 7f8b45e00640 1 --2- 172.21.15.28:0/3483350603 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b2c0051c0 0x7f8b2c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:54.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.236+0000 7f8b45e00640 1 -- 172.21.15.28:0/3483350603 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b2c003520 con 0x7f8b2c0039b0 2024-02-21T23:15:54.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.236+0000 7f8b46800640 1 -- 172.21.15.28:0/3483350603 reap_dead start 2024-02-21T23:15:54.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.237+0000 7f8b45e00640 1 --2- 172.21.15.28:0/3483350603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b2c0039b0 0x7f8b2c003df0 secure :-1 s=READY pgs=2587 cs=0 l=1 rev1=1 crypto rx=0x7f8b38001750 tx=0x7f8b2c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e1377d707f4ffd5 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:54.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.237+0000 7f8b45400640 1 -- 172.21.15.28:0/3483350603 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b38001370 con 0x7f8b2c0039b0 2024-02-21T23:15:54.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.237+0000 7f8b45400640 1 -- 172.21.15.28:0/3483350603 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b380052d0 con 0x7f8b2c0039b0 2024-02-21T23:15:54.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.237+0000 7f8b45400640 1 -- 172.21.15.28:0/3483350603 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b38003830 con 0x7f8b2c0039b0 2024-02-21T23:15:54.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.237+0000 7f8b4d200640 1 -- 172.21.15.28:0/3483350603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b2c0039b0 msgr2=0x7f8b2c003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:54.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.237+0000 7f8b4d200640 1 --2- 172.21.15.28:0/3483350603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b2c0039b0 0x7f8b2c003df0 secure :-1 s=READY pgs=2587 cs=0 l=1 rev1=1 crypto rx=0x7f8b38001750 tx=0x7f8b2c005830 comp rx=0 tx=0).stop 2024-02-21T23:15:54.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.238+0000 7f8b4d200640 1 -- 172.21.15.28:0/3483350603 shutdown_connections 2024-02-21T23:15:54.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.238+0000 7f8b4d200640 1 --2- 172.21.15.28:0/3483350603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b2c0039b0 0x7f8b2c003df0 unknown :-1 s=CLOSED pgs=2587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:54.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.238+0000 7f8b4d200640 1 -- 172.21.15.28:0/3483350603 >> 172.21.15.28:0/3483350603 conn(0x7f8b480aa550 msgr2=0x7f8b480aa930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:15:54.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.238+0000 7f8b4d200640 1 -- 172.21.15.28:0/3483350603 shutdown_connections 2024-02-21T23:15:54.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.238+0000 7f8b4d200640 1 -- 172.21.15.28:0/3483350603 wait complete. 2024-02-21T23:15:54.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.240+0000 7f8b4d200640 1 Processor -- start 2024-02-21T23:15:54.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.241+0000 7f8b4d200640 1 -- start start 2024-02-21T23:15:54.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.242+0000 7f8b4d200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b2c0039b0 0x7f8b48147c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:54.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.242+0000 7f8b4d200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b480ad430 0x7f8b48154180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:54.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.242+0000 7f8b4d200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8b481edfd0 0x7f8b48144770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:54.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.242+0000 7f8b4d200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b480c9150 con 0x7f8b480ad430 2024-02-21T23:15:54.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.242+0000 7f8b4d200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b480c8fd0 con 0x7f8b2c0039b0 2024-02-21T23:15:54.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.242+0000 7f8b4d200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b480c31a0 con 0x7f8b481edfd0 2024-02-21T23:15:54.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.243+0000 7f8b3fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b480ad430 0x7f8b48154180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:54.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.243+0000 7f8b3fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b480ad430 0x7f8b48154180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59372/0 (socket says 172.21.15.28:59372) 2024-02-21T23:15:54.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.243+0000 7f8b3fe00640 1 -- 172.21.15.28:0/1354663635 learned_addr learned my addr 172.21.15.28:0/1354663635 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:15:54.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.243+0000 7f8b45e00640 1 --2- 172.21.15.28:0/1354663635 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b2c0039b0 0x7f8b48147c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:54.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.243+0000 7f8b46800640 1 --2- 172.21.15.28:0/1354663635 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8b481edfd0 0x7f8b48144770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:54.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.243+0000 7f8b3fe00640 1 -- 172.21.15.28:0/1354663635 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8b481edfd0 msgr2=0x7f8b48144770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:54.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.243+0000 7f8b3fe00640 1 --2- 172.21.15.28:0/1354663635 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8b481edfd0 0x7f8b48144770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:54.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.243+0000 7f8b3fe00640 1 -- 172.21.15.28:0/1354663635 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b2c0039b0 msgr2=0x7f8b48147c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:54.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.243+0000 7f8b3fe00640 1 --2- 172.21.15.28:0/1354663635 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b2c0039b0 0x7f8b48147c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:54.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.243+0000 7f8b3fe00640 1 -- 172.21.15.28:0/1354663635 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b481457c0 con 0x7f8b480ad430 2024-02-21T23:15:54.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.243+0000 7f8b46800640 1 --2- 172.21.15.28:0/1354663635 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8b481edfd0 0x7f8b48144770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:15:54.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.244+0000 7f8b3fe00640 1 --2- 172.21.15.28:0/1354663635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b480ad430 0x7f8b48154180 secure :-1 s=READY pgs=2588 cs=0 l=1 rev1=1 crypto rx=0x7f8b3000e730 tx=0x7f8b3000ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:54.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.244+0000 7f8b3e000640 1 -- 172.21.15.28:0/1354663635 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b3000ee40 con 0x7f8b480ad430 2024-02-21T23:15:54.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.244+0000 7f8b3e000640 1 -- 172.21.15.28:0/1354663635 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b30018070 con 0x7f8b480ad430 2024-02-21T23:15:54.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.244+0000 7f8b3e000640 1 -- 172.21.15.28:0/1354663635 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b3000ca30 con 0x7f8b480ad430 2024-02-21T23:15:54.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.244+0000 7f8b4d200640 1 -- 172.21.15.28:0/1354663635 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b48151fb0 con 0x7f8b480ad430 2024-02-21T23:15:54.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.244+0000 7f8b4d200640 1 -- 172.21.15.28:0/1354663635 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b48145330 con 0x7f8b480ad430 2024-02-21T23:15:54.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.246+0000 7f8b4d200640 1 -- 172.21.15.28:0/1354663635 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b48068640 con 0x7f8b480ad430 2024-02-21T23:15:54.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.246+0000 7f8b3e000640 1 -- 172.21.15.28:0/1354663635 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8b3000cbd0 con 0x7f8b480ad430 2024-02-21T23:15:54.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.247+0000 7f8b3e000640 1 --2- 172.21.15.28:0/1354663635 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8b2406b500 0x7f8b2406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:54.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.247+0000 7f8b3e000640 1 -- 172.21.15.28:0/1354663635 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f8b30089e50 con 0x7f8b480ad430 2024-02-21T23:15:54.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.248+0000 7f8b45e00640 1 --2- 172.21.15.28:0/1354663635 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8b2406b500 0x7f8b2406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:54.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.249+0000 7f8b45e00640 1 --2- 172.21.15.28:0/1354663635 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8b2406b500 0x7f8b2406d9c0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f8b38001750 tx=0x7f8b38009030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:54.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.252+0000 7f8b3e000640 1 -- 172.21.15.28:0/1354663635 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f8b30058ad0 con 0x7f8b480ad430 2024-02-21T23:15:54.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.499+0000 7f8b4d200640 1 -- 172.21.15.28:0/1354663635 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b480a8560 con 0x7f8b480ad430 2024-02-21T23:15:54.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.500+0000 7f8b3e000640 1 -- 172.21.15.28:0/1354663635 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7f8b300583d0 con 0x7f8b480ad430 2024-02-21T23:15:54.488 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:15:54.488 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:15:54.489 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:15:54.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.503+0000 7f8b4d200640 1 -- 172.21.15.28:0/1354663635 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8b2406b500 msgr2=0x7f8b2406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:54.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.503+0000 7f8b4d200640 1 --2- 172.21.15.28:0/1354663635 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8b2406b500 0x7f8b2406d9c0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f8b38001750 tx=0x7f8b38009030 comp rx=0 tx=0).stop 2024-02-21T23:15:54.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.503+0000 7f8b4d200640 1 -- 172.21.15.28:0/1354663635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b480ad430 msgr2=0x7f8b48154180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:54.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.503+0000 7f8b4d200640 1 --2- 172.21.15.28:0/1354663635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b480ad430 0x7f8b48154180 secure :-1 s=READY pgs=2588 cs=0 l=1 rev1=1 crypto rx=0x7f8b3000e730 tx=0x7f8b3000ebf0 comp rx=0 tx=0).stop 2024-02-21T23:15:54.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.503+0000 7f8b4d200640 1 -- 172.21.15.28:0/1354663635 shutdown_connections 2024-02-21T23:15:54.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.503+0000 7f8b4d200640 1 --2- 172.21.15.28:0/1354663635 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8b2406b500 0x7f8b2406d9c0 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:54.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.503+0000 7f8b4d200640 1 --2- 172.21.15.28:0/1354663635 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8b481edfd0 0x7f8b48144770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:54.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.503+0000 7f8b4d200640 1 --2- 172.21.15.28:0/1354663635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b480ad430 0x7f8b48154180 unknown :-1 s=CLOSED pgs=2588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:54.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.503+0000 7f8b4d200640 1 --2- 172.21.15.28:0/1354663635 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b2c0039b0 0x7f8b48147c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:54.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.503+0000 7f8b4d200640 1 -- 172.21.15.28:0/1354663635 >> 172.21.15.28:0/1354663635 conn(0x7f8b480aa550 msgr2=0x7f8b480c5230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:15:54.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.503+0000 7f8b4d200640 1 -- 172.21.15.28:0/1354663635 shutdown_connections 2024-02-21T23:15:54.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:54.503+0000 7f8b4d200640 1 -- 172.21.15.28:0/1354663635 wait complete. 2024-02-21T23:15:54.915 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:15:54.915 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:15:55.916 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:15:56.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.151+0000 7f9f59a00640 1 Processor -- start 2024-02-21T23:15:56.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.151+0000 7f9f59a00640 1 -- start start 2024-02-21T23:15:56.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.153+0000 7f9f59a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9f540aff90 0x7f9f540c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:56.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.153+0000 7f9f59a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f540c74c0 con 0x7f9f5406dbf0 2024-02-21T23:15:56.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.153+0000 7f9f59a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f540c77a0 con 0x7f9f540c5720 2024-02-21T23:15:56.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.153+0000 7f9f59a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f540c7270 con 0x7f9f540aff90 2024-02-21T23:15:56.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.154+0000 7f9f53400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f9f5406dbf0 0x7f9f54158ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34654/0 (socket says 172.21.15.28:34654) 2024-02-21T23:15:56.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.154+0000 7f9f53400640 1 -- 172.21.15.28:0/3327161140 learned_addr learned my addr 172.21.15.28:0/3327161140 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:15:56.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.154+0000 7f9f53e00640 1 --2- 172.21.15.28:0/3327161140 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9f540aff90 0x7f9f540c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:56.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.154+0000 7f9f52a00640 1 -- 172.21.15.28:0/3327161140 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 199367500 0 0) 0x7f9f540c74c0 con 0x7f9f5406dbf0 2024-02-21T23:15:56.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.154+0000 7f9f52a00640 1 -- 172.21.15.28:0/3327161140 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f38003520 con 0x7f9f5406dbf0 2024-02-21T23:15:56.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.155+0000 7f9f52a00640 1 -- 172.21.15.28:0/3327161140 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3849818912 0 0) 0x7f9f38003520 con 0x7f9f5406dbf0 2024-02-21T23:15:56.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.155+0000 7f9f52a00640 1 -- 172.21.15.28:0/3327161140 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9f540aff90 msgr2=0x7f9f540c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:15:56.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.155+0000 7f9f52a00640 1 --2- 172.21.15.28:0/3327161140 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9f540aff90 0x7f9f540c51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:56.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.155+0000 7f9f52a00640 1 -- 172.21.15.28:0/3327161140 >> v1:172.21.15.81:6789/0 conn(0x7f9f540c5720 legacy=0x7f9f5415f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:56.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.155+0000 7f9f52a00640 1 -- 172.21.15.28:0/3327161140 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f540c7a80 con 0x7f9f5406dbf0 2024-02-21T23:15:56.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.155+0000 7f9f52a00640 1 -- 172.21.15.28:0/3327161140 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f9f40001670 con 0x7f9f5406dbf0 2024-02-21T23:15:56.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.155+0000 7f9f52a00640 1 -- 172.21.15.28:0/3327161140 >> v1:172.21.15.28:6789/0 conn(0x7f9f5406dbf0 legacy=0x7f9f54158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:56.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.155+0000 7f9f52a00640 1 --2- 172.21.15.28:0/3327161140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f380039e0 0x7f9f38003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:56.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.155+0000 7f9f52a00640 1 --2- 172.21.15.28:0/3327161140 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f38004800 0x7f9f38004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:56.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.155+0000 7f9f52a00640 1 --2- 172.21.15.28:0/3327161140 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9f540aff90 0x7f9f380051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:56.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.155+0000 7f9f52a00640 1 -- 172.21.15.28:0/3327161140 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f540c7270 con 0x7f9f380039e0 2024-02-21T23:15:56.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.155+0000 7f9f52a00640 1 -- 172.21.15.28:0/3327161140 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f3800ba70 con 0x7f9f38004800 2024-02-21T23:15:56.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.155+0000 7f9f52a00640 1 -- 172.21.15.28:0/3327161140 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f3800bc70 con 0x7f9f540aff90 2024-02-21T23:15:56.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.155+0000 7f9f53e00640 1 --2- 172.21.15.28:0/3327161140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f380039e0 0x7f9f38003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:56.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.156+0000 7f9f59000640 1 --2- 172.21.15.28:0/3327161140 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9f540aff90 0x7f9f380051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:56.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.156+0000 7f9f53400640 1 --2- 172.21.15.28:0/3327161140 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f38004800 0x7f9f38004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:56.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.156+0000 7f9f53e00640 1 -- 172.21.15.28:0/3327161140 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9f540aff90 msgr2=0x7f9f380051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:15:56.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.156+0000 7f9f53e00640 1 --2- 172.21.15.28:0/3327161140 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9f540aff90 0x7f9f380051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:56.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.156+0000 7f9f53e00640 1 -- 172.21.15.28:0/3327161140 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f38004800 msgr2=0x7f9f38004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:15:56.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.156+0000 7f9f53e00640 1 --2- 172.21.15.28:0/3327161140 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f38004800 0x7f9f38004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:56.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.156+0000 7f9f53e00640 1 -- 172.21.15.28:0/3327161140 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f38003520 con 0x7f9f380039e0 2024-02-21T23:15:56.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.157+0000 7f9f53e00640 1 --2- 172.21.15.28:0/3327161140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f380039e0 0x7f9f38003e00 secure :-1 s=READY pgs=2590 cs=0 l=1 rev1=1 crypto rx=0x7f9f44009a90 tx=0x7f9f38005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=becd933259c2fa0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:56.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.157+0000 7f9f52a00640 1 -- 172.21.15.28:0/3327161140 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f44039070 con 0x7f9f380039e0 2024-02-21T23:15:56.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.157+0000 7f9f52a00640 1 -- 172.21.15.28:0/3327161140 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f4402c660 con 0x7f9f380039e0 2024-02-21T23:15:56.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.157+0000 7f9f52a00640 1 -- 172.21.15.28:0/3327161140 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f44009690 con 0x7f9f380039e0 2024-02-21T23:15:56.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.158+0000 7f9f59a00640 1 -- 172.21.15.28:0/3327161140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f380039e0 msgr2=0x7f9f38003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:56.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.158+0000 7f9f59a00640 1 --2- 172.21.15.28:0/3327161140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f380039e0 0x7f9f38003e00 secure :-1 s=READY pgs=2590 cs=0 l=1 rev1=1 crypto rx=0x7f9f44009a90 tx=0x7f9f38005830 comp rx=0 tx=0).stop 2024-02-21T23:15:56.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.158+0000 7f9f59000640 1 -- 172.21.15.28:0/3327161140 reap_dead start 2024-02-21T23:15:56.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.158+0000 7f9f59a00640 1 -- 172.21.15.28:0/3327161140 shutdown_connections 2024-02-21T23:15:56.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.158+0000 7f9f59a00640 1 -- 172.21.15.28:0/3327161140 >> 172.21.15.28:0/3327161140 conn(0x7f9f540ac370 msgr2=0x7f9f540ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:15:56.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.158+0000 7f9f59a00640 1 -- 172.21.15.28:0/3327161140 shutdown_connections 2024-02-21T23:15:56.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.158+0000 7f9f59a00640 1 -- 172.21.15.28:0/3327161140 wait complete. 2024-02-21T23:15:56.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.160+0000 7f9f59a00640 1 Processor -- start 2024-02-21T23:15:56.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.160+0000 7f9f59a00640 1 -- start start 2024-02-21T23:15:56.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.162+0000 7f9f59a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f380039e0 0x7f9f540ab5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:56.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.162+0000 7f9f59a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9f540ae2e0 0x7f9f5414ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:56.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.162+0000 7f9f59a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f5414f290 0x7f9f541edca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:56.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.162+0000 7f9f59a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f540c8760 con 0x7f9f380039e0 2024-02-21T23:15:56.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.162+0000 7f9f59a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f54158630 con 0x7f9f5414f290 2024-02-21T23:15:56.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.162+0000 7f9f59a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f540c88e0 con 0x7f9f540ae2e0 2024-02-21T23:15:56.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.162+0000 7f9f53e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f380039e0 0x7f9f540ab5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:56.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.162+0000 7f9f53e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f380039e0 0x7f9f540ab5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39688/0 (socket says 172.21.15.28:39688) 2024-02-21T23:15:56.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.162+0000 7f9f53e00640 1 -- 172.21.15.28:0/1349566417 learned_addr learned my addr 172.21.15.28:0/1349566417 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:15:56.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.162+0000 7f9f53400640 1 --2- 172.21.15.28:0/1349566417 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9f540ae2e0 0x7f9f5414ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:56.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.162+0000 7f9f59000640 1 --2- 172.21.15.28:0/1349566417 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f5414f290 0x7f9f541edca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:56.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.162+0000 7f9f53e00640 1 -- 172.21.15.28:0/1349566417 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9f540ae2e0 msgr2=0x7f9f5414ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:56.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.162+0000 7f9f53e00640 1 --2- 172.21.15.28:0/1349566417 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9f540ae2e0 0x7f9f5414ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:56.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.162+0000 7f9f53e00640 1 -- 172.21.15.28:0/1349566417 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f5414f290 msgr2=0x7f9f541edca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:56.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.162+0000 7f9f53e00640 1 --2- 172.21.15.28:0/1349566417 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f5414f290 0x7f9f541edca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:56.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.162+0000 7f9f53e00640 1 -- 172.21.15.28:0/1349566417 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f541eb610 con 0x7f9f380039e0 2024-02-21T23:15:56.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.163+0000 7f9f53e00640 1 --2- 172.21.15.28:0/1349566417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f380039e0 0x7f9f540ab5d0 secure :-1 s=READY pgs=2591 cs=0 l=1 rev1=1 crypto rx=0x7f9f44009a90 tx=0x7f9f44004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:56.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.163+0000 7f9f4be00640 1 -- 172.21.15.28:0/1349566417 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f44039050 con 0x7f9f380039e0 2024-02-21T23:15:56.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.163+0000 7f9f4be00640 1 -- 172.21.15.28:0/1349566417 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f44045070 con 0x7f9f380039e0 2024-02-21T23:15:56.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.163+0000 7f9f4be00640 1 -- 172.21.15.28:0/1349566417 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f44003830 con 0x7f9f380039e0 2024-02-21T23:15:56.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.163+0000 7f9f59a00640 1 -- 172.21.15.28:0/1349566417 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f540aeb30 con 0x7f9f380039e0 2024-02-21T23:15:56.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.163+0000 7f9f59a00640 1 -- 172.21.15.28:0/1349566417 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f541eab30 con 0x7f9f380039e0 2024-02-21T23:15:56.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.165+0000 7f9f59a00640 1 -- 172.21.15.28:0/1349566417 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f5404f290 con 0x7f9f380039e0 2024-02-21T23:15:56.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.165+0000 7f9f4be00640 1 -- 172.21.15.28:0/1349566417 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9f440039d0 con 0x7f9f380039e0 2024-02-21T23:15:56.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.166+0000 7f9f4be00640 1 --2- 172.21.15.28:0/1349566417 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9f3406b4d0 0x7f9f3406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:56.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.166+0000 7f9f4be00640 1 -- 172.21.15.28:0/1349566417 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f9f440abfd0 con 0x7f9f380039e0 2024-02-21T23:15:56.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.167+0000 7f9f53400640 1 --2- 172.21.15.28:0/1349566417 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9f3406b4d0 0x7f9f3406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:56.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.168+0000 7f9f53400640 1 --2- 172.21.15.28:0/1349566417 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9f3406b4d0 0x7f9f3406d990 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f9f40001200 tx=0x7f9f40009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:56.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.170+0000 7f9f4be00640 1 -- 172.21.15.28:0/1349566417 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f9f4404e020 con 0x7f9f380039e0 2024-02-21T23:15:56.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.418+0000 7f9f59a00640 1 -- 172.21.15.28:0/1349566417 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9f541524f0 con 0x7f9f380039e0 2024-02-21T23:15:56.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.419+0000 7f9f4be00640 1 -- 172.21.15.28:0/1349566417 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7f9f4407ac70 con 0x7f9f380039e0 2024-02-21T23:15:56.408 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:15:56.408 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:15:56.408 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:15:56.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.422+0000 7f9f59a00640 1 -- 172.21.15.28:0/1349566417 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9f3406b4d0 msgr2=0x7f9f3406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:56.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.422+0000 7f9f59a00640 1 --2- 172.21.15.28:0/1349566417 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9f3406b4d0 0x7f9f3406d990 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f9f40001200 tx=0x7f9f40009040 comp rx=0 tx=0).stop 2024-02-21T23:15:56.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.422+0000 7f9f59a00640 1 -- 172.21.15.28:0/1349566417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f380039e0 msgr2=0x7f9f540ab5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:56.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.422+0000 7f9f59a00640 1 --2- 172.21.15.28:0/1349566417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f380039e0 0x7f9f540ab5d0 secure :-1 s=READY pgs=2591 cs=0 l=1 rev1=1 crypto rx=0x7f9f44009a90 tx=0x7f9f44004730 comp rx=0 tx=0).stop 2024-02-21T23:15:56.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.422+0000 7f9f59a00640 1 -- 172.21.15.28:0/1349566417 shutdown_connections 2024-02-21T23:15:56.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.422+0000 7f9f59a00640 1 --2- 172.21.15.28:0/1349566417 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9f3406b4d0 0x7f9f3406d990 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:56.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.422+0000 7f9f59a00640 1 --2- 172.21.15.28:0/1349566417 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9f540ae2e0 0x7f9f5414ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:56.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.422+0000 7f9f59a00640 1 --2- 172.21.15.28:0/1349566417 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f5414f290 0x7f9f541edca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:56.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.422+0000 7f9f59a00640 1 --2- 172.21.15.28:0/1349566417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f380039e0 0x7f9f540ab5d0 unknown :-1 s=CLOSED pgs=2591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:56.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.422+0000 7f9f59a00640 1 -- 172.21.15.28:0/1349566417 >> 172.21.15.28:0/1349566417 conn(0x7f9f540ac370 msgr2=0x7f9f540ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:15:56.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.422+0000 7f9f59a00640 1 -- 172.21.15.28:0/1349566417 shutdown_connections 2024-02-21T23:15:56.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:56.422+0000 7f9f59a00640 1 -- 172.21.15.28:0/1349566417 wait complete. 2024-02-21T23:15:56.832 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:15:56.832 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:15:57.833 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:15:58.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.071+0000 7ffaaa000640 1 Processor -- start 2024-02-21T23:15:58.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.071+0000 7ffaaa000640 1 -- start start 2024-02-21T23:15:58.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.073+0000 7ffaaa000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ffaa40c11f0 0x7ffaa4153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:58.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.073+0000 7ffaaa000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7ffaa40c15d0 0x7ffaa406dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:58.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.073+0000 7ffaaa000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaa4152e70 con 0x7ffaa4156350 2024-02-21T23:15:58.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.073+0000 7ffaaa000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffaa40c1a60 con 0x7ffaa40c11f0 2024-02-21T23:15:58.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.073+0000 7ffaaa000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffaa40a5ec0 con 0x7ffaa40c15d0 2024-02-21T23:15:58.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.073+0000 7ffaa3400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ffaa40c11f0 0x7ffaa4153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:58.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.073+0000 7ffaa3400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ffaa40c11f0 0x7ffaa4153950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39694/0 (socket says 172.21.15.28:39694) 2024-02-21T23:15:58.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.073+0000 7ffaa3400640 1 -- 172.21.15.28:0/2410800666 learned_addr learned my addr 172.21.15.28:0/2410800666 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:15:58.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.074+0000 7ffaa2a00640 1 --2- 172.21.15.28:0/2410800666 >> v2:172.21.15.81:3300/0 conn(0x7ffaa40c15d0 0x7ffaa406dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:58.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.074+0000 7ffaa3400640 1 -- 172.21.15.28:0/2410800666 >> v2:172.21.15.81:3300/0 conn(0x7ffaa40c15d0 msgr2=0x7ffaa406dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:15:58.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.074+0000 7ffaa3400640 1 --2- 172.21.15.28:0/2410800666 >> v2:172.21.15.81:3300/0 conn(0x7ffaa40c15d0 0x7ffaa406dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:58.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.074+0000 7ffaa3400640 1 -- 172.21.15.28:0/2410800666 >> v1:172.21.15.81:6789/0 conn(0x7ffaa4156350 legacy=0x7ffaa4156760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:15:58.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.074+0000 7ffaa3400640 1 -- 172.21.15.28:0/2410800666 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaa4153150 con 0x7ffaa40c11f0 2024-02-21T23:15:58.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.074+0000 7ffaa2a00640 1 --2- 172.21.15.28:0/2410800666 >> v2:172.21.15.81:3300/0 conn(0x7ffaa40c15d0 0x7ffaa406dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:15:58.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.075+0000 7ffaa3400640 1 --2- 172.21.15.28:0/2410800666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffaa40c11f0 0x7ffaa4153950 secure :-1 s=READY pgs=2592 cs=0 l=1 rev1=1 crypto rx=0x7ffaa40c24a0 tx=0x7ffa9402b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cff551b0a5cd0cb server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:58.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.075+0000 7ffaa2000640 1 -- 172.21.15.28:0/2410800666 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa94038070 con 0x7ffaa40c11f0 2024-02-21T23:15:58.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.075+0000 7ffaa2000640 1 -- 172.21.15.28:0/2410800666 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa9402f050 con 0x7ffaa40c11f0 2024-02-21T23:15:58.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.075+0000 7ffaa2000640 1 -- 172.21.15.28:0/2410800666 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa94031970 con 0x7ffaa40c11f0 2024-02-21T23:15:58.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.076+0000 7ffaaa000640 1 -- 172.21.15.28:0/2410800666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffaa40c11f0 msgr2=0x7ffaa4153950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:58.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.076+0000 7ffaaa000640 1 --2- 172.21.15.28:0/2410800666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffaa40c11f0 0x7ffaa4153950 secure :-1 s=READY pgs=2592 cs=0 l=1 rev1=1 crypto rx=0x7ffaa40c24a0 tx=0x7ffa9402b7f0 comp rx=0 tx=0).stop 2024-02-21T23:15:58.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.076+0000 7ffaaa000640 1 -- 172.21.15.28:0/2410800666 shutdown_connections 2024-02-21T23:15:58.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.076+0000 7ffaaa000640 1 --2- 172.21.15.28:0/2410800666 >> v2:172.21.15.81:3300/0 conn(0x7ffaa40c15d0 0x7ffaa406dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:58.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.076+0000 7ffaaa000640 1 --2- 172.21.15.28:0/2410800666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffaa40c11f0 0x7ffaa4153950 unknown :-1 s=CLOSED pgs=2592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:58.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.076+0000 7ffaaa000640 1 -- 172.21.15.28:0/2410800666 >> 172.21.15.28:0/2410800666 conn(0x7ffaa40c67f0 msgr2=0x7ffaa40c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:15:58.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.076+0000 7ffaaa000640 1 -- 172.21.15.28:0/2410800666 shutdown_connections 2024-02-21T23:15:58.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.076+0000 7ffaaa000640 1 -- 172.21.15.28:0/2410800666 wait complete. 2024-02-21T23:15:58.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.079+0000 7ffaaa000640 1 Processor -- start 2024-02-21T23:15:58.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.079+0000 7ffaaa000640 1 -- start start 2024-02-21T23:15:58.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.081+0000 7ffaaa000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffaa40c11f0 0x7ffaa40a7c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:58.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.081+0000 7ffaaa000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffaa40c15d0 0x7ffaa413fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:58.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.081+0000 7ffaaa000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffaa4156350 0x7ffaa4140110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:58.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.081+0000 7ffaaa000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa40c4c00 con 0x7ffaa40c15d0 2024-02-21T23:15:58.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.081+0000 7ffaaa000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa414d1c0 con 0x7ffaa40c11f0 2024-02-21T23:15:58.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.081+0000 7ffaaa000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa40a6110 con 0x7ffaa4156350 2024-02-21T23:15:58.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.081+0000 7ffaa2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffaa40c15d0 0x7ffaa413fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:58.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.081+0000 7ffaa2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffaa40c15d0 0x7ffaa413fbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39710/0 (socket says 172.21.15.28:39710) 2024-02-21T23:15:58.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.081+0000 7ffaa2a00640 1 -- 172.21.15.28:0/1389101631 learned_addr learned my addr 172.21.15.28:0/1389101631 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:15:58.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.081+0000 7ffaa3400640 1 --2- 172.21.15.28:0/1389101631 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffaa40c11f0 0x7ffaa40a7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:58.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.081+0000 7ffaa3e00640 1 --2- 172.21.15.28:0/1389101631 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffaa4156350 0x7ffaa4140110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:58.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.082+0000 7ffaa2a00640 1 -- 172.21.15.28:0/1389101631 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffaa4156350 msgr2=0x7ffaa4140110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:58.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.082+0000 7ffaa2a00640 1 --2- 172.21.15.28:0/1389101631 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffaa4156350 0x7ffaa4140110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:58.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.082+0000 7ffaa2a00640 1 -- 172.21.15.28:0/1389101631 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffaa40c11f0 msgr2=0x7ffaa40a7c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:58.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.082+0000 7ffaa2a00640 1 --2- 172.21.15.28:0/1389101631 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffaa40c11f0 0x7ffaa40a7c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:58.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.082+0000 7ffaa2a00640 1 -- 172.21.15.28:0/1389101631 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaa4072910 con 0x7ffaa40c15d0 2024-02-21T23:15:58.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.082+0000 7ffaa3e00640 1 --2- 172.21.15.28:0/1389101631 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffaa4156350 0x7ffaa4140110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:15:58.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.082+0000 7ffaa2a00640 1 --2- 172.21.15.28:0/1389101631 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffaa40c15d0 0x7ffaa413fbd0 secure :-1 s=READY pgs=2593 cs=0 l=1 rev1=1 crypto rx=0x7ffa9000c820 tx=0x7ffa9000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:58.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.082+0000 7ffa9b400640 1 -- 172.21.15.28:0/1389101631 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa90015070 con 0x7ffaa40c15d0 2024-02-21T23:15:58.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.082+0000 7ffa9b400640 1 -- 172.21.15.28:0/1389101631 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa90007a30 con 0x7ffaa40c15d0 2024-02-21T23:15:58.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.082+0000 7ffa9b400640 1 -- 172.21.15.28:0/1389101631 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa90007e00 con 0x7ffaa40c15d0 2024-02-21T23:15:58.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.082+0000 7ffaaa000640 1 -- 172.21.15.28:0/1389101631 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaa4072bf0 con 0x7ffaa40c15d0 2024-02-21T23:15:58.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.083+0000 7ffaaa000640 1 -- 172.21.15.28:0/1389101631 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaa41ee460 con 0x7ffaa40c15d0 2024-02-21T23:15:58.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.084+0000 7ffaaa000640 1 -- 172.21.15.28:0/1389101631 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffaa4067f30 con 0x7ffaa40c15d0 2024-02-21T23:15:58.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.086+0000 7ffa9b400640 1 -- 172.21.15.28:0/1389101631 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffa90018070 con 0x7ffaa40c15d0 2024-02-21T23:15:58.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.087+0000 7ffa9b400640 1 --2- 172.21.15.28:0/1389101631 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ffa8406b4d0 0x7ffa8406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:58.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.087+0000 7ffa9b400640 1 -- 172.21.15.28:0/1389101631 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7ffa9008a4c0 con 0x7ffaa40c15d0 2024-02-21T23:15:58.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.087+0000 7ffaa3400640 1 --2- 172.21.15.28:0/1389101631 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ffa8406b4d0 0x7ffa8406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:58.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.089+0000 7ffaa3400640 1 --2- 172.21.15.28:0/1389101631 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ffa8406b4d0 0x7ffa8406d990 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7ffa94035000 tx=0x7ffa94008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:58.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.092+0000 7ffa9b400640 1 -- 172.21.15.28:0/1389101631 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ffa90059010 con 0x7ffaa40c15d0 2024-02-21T23:15:58.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.324+0000 7ffaaa000640 1 -- 172.21.15.28:0/1389101631 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffaa40a5c30 con 0x7ffaa40c15d0 2024-02-21T23:15:58.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.325+0000 7ffa9b400640 1 -- 172.21.15.28:0/1389101631 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7ffa90058910 con 0x7ffaa40c15d0 2024-02-21T23:15:58.314 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:15:58.314 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:15:58.314 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:15:58.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.327+0000 7ffaaa000640 1 -- 172.21.15.28:0/1389101631 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ffa8406b4d0 msgr2=0x7ffa8406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:58.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.327+0000 7ffaaa000640 1 --2- 172.21.15.28:0/1389101631 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ffa8406b4d0 0x7ffa8406d990 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7ffa94035000 tx=0x7ffa94008030 comp rx=0 tx=0).stop 2024-02-21T23:15:58.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.328+0000 7ffaaa000640 1 -- 172.21.15.28:0/1389101631 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffaa40c15d0 msgr2=0x7ffaa413fbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:58.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.328+0000 7ffaaa000640 1 --2- 172.21.15.28:0/1389101631 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffaa40c15d0 0x7ffaa413fbd0 secure :-1 s=READY pgs=2593 cs=0 l=1 rev1=1 crypto rx=0x7ffa9000c820 tx=0x7ffa9000cc20 comp rx=0 tx=0).stop 2024-02-21T23:15:58.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.328+0000 7ffaaa000640 1 -- 172.21.15.28:0/1389101631 shutdown_connections 2024-02-21T23:15:58.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.328+0000 7ffaaa000640 1 --2- 172.21.15.28:0/1389101631 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ffa8406b4d0 0x7ffa8406d990 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:58.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.328+0000 7ffaaa000640 1 --2- 172.21.15.28:0/1389101631 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ffaa4156350 0x7ffaa4140110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:58.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.328+0000 7ffaaa000640 1 --2- 172.21.15.28:0/1389101631 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffaa40c15d0 0x7ffaa413fbd0 unknown :-1 s=CLOSED pgs=2593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:58.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.328+0000 7ffaaa000640 1 --2- 172.21.15.28:0/1389101631 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffaa40c11f0 0x7ffaa40a7c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:58.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.328+0000 7ffaaa000640 1 -- 172.21.15.28:0/1389101631 >> 172.21.15.28:0/1389101631 conn(0x7ffaa40c67f0 msgr2=0x7ffaa406dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:15:58.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.328+0000 7ffaaa000640 1 -- 172.21.15.28:0/1389101631 shutdown_connections 2024-02-21T23:15:58.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:58.328+0000 7ffaaa000640 1 -- 172.21.15.28:0/1389101631 wait complete. 2024-02-21T23:15:58.735 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:15:58.735 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:15:59.737 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:15:59.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:59.972+0000 7f14a9c00640 1 Processor -- start 2024-02-21T23:15:59.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:59.972+0000 7f14a9c00640 1 -- start start 2024-02-21T23:15:59.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:59.973+0000 7f14a9c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f14a40c54c0 0x7f14a40a7f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:59.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:59.973+0000 7f14a9c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f14a40c6ef0 0x7f14a4152560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:59.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:59.973+0000 7f14a9c00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14a40c6820 con 0x7f14a4152aa0 2024-02-21T23:15:59.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:59.973+0000 7f14a9c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14a40c3de0 con 0x7f14a40c54c0 2024-02-21T23:15:59.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:59.974+0000 7f14a9c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14a40c7fe0 con 0x7f14a40c6ef0 2024-02-21T23:15:59.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:59.974+0000 7f14a2a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f14a40c54c0 0x7f14a40a7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:59.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:59.974+0000 7f14a2a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f14a40c54c0 0x7f14a40a7f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39714/0 (socket says 172.21.15.28:39714) 2024-02-21T23:15:59.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:59.974+0000 7f14a2a00640 1 -- 172.21.15.28:0/2928430066 learned_addr learned my addr 172.21.15.28:0/2928430066 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:15:59.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:59.974+0000 7f14a2000640 1 --2- 172.21.15.28:0/2928430066 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f14a40c6ef0 0x7f14a4152560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:59.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:59.974+0000 7f14a2a00640 1 -- 172.21.15.28:0/2928430066 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f14a40c6ef0 msgr2=0x7f14a4152560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:15:59.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:59.974+0000 7f14a2a00640 1 --2- 172.21.15.28:0/2928430066 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f14a40c6ef0 0x7f14a4152560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:59.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:59.974+0000 7f14a2a00640 1 -- 172.21.15.28:0/2928430066 >> v1:172.21.15.81:6789/0 conn(0x7f14a4152aa0 legacy=0x7f14a4158940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:59.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:59.975+0000 7f14a2a00640 1 -- 172.21.15.28:0/2928430066 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14a40c6b00 con 0x7f14a40c54c0 2024-02-21T23:15:59.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:59.976+0000 7f14a2a00640 1 --2- 172.21.15.28:0/2928430066 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14a40c54c0 0x7f14a40a7f40 secure :-1 s=READY pgs=2594 cs=0 l=1 rev1=1 crypto rx=0x7f1494009e50 tx=0x7f149402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3324a46a88babe0b server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:59.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:59.976+0000 7f14a1600640 1 -- 172.21.15.28:0/2928430066 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f149402ca80 con 0x7f14a40c54c0 2024-02-21T23:15:59.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:59.976+0000 7f14a1600640 1 -- 172.21.15.28:0/2928430066 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f149402cc20 con 0x7f14a40c54c0 2024-02-21T23:15:59.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:59.976+0000 7f14a1600640 1 -- 172.21.15.28:0/2928430066 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14940343f0 con 0x7f14a40c54c0 2024-02-21T23:15:59.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:59.977+0000 7f14a9c00640 1 -- 172.21.15.28:0/2928430066 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14a40c54c0 msgr2=0x7f14a40a7f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:59.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:59.977+0000 7f14a9c00640 1 --2- 172.21.15.28:0/2928430066 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14a40c54c0 0x7f14a40a7f40 secure :-1 s=READY pgs=2594 cs=0 l=1 rev1=1 crypto rx=0x7f1494009e50 tx=0x7f149402bee0 comp rx=0 tx=0).stop 2024-02-21T23:15:59.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:59.977+0000 7f14a9c00640 1 -- 172.21.15.28:0/2928430066 shutdown_connections 2024-02-21T23:15:59.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:59.977+0000 7f14a9c00640 1 --2- 172.21.15.28:0/2928430066 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f14a40c6ef0 0x7f14a4152560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:59.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:59.977+0000 7f14a9c00640 1 --2- 172.21.15.28:0/2928430066 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14a40c54c0 0x7f14a40a7f40 unknown :-1 s=CLOSED pgs=2594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:59.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:59.977+0000 7f14a9c00640 1 -- 172.21.15.28:0/2928430066 >> 172.21.15.28:0/2928430066 conn(0x7f14a40c8810 msgr2=0x7f14a40c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:15:59.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:59.977+0000 7f14a9c00640 1 -- 172.21.15.28:0/2928430066 shutdown_connections 2024-02-21T23:15:59.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:59.977+0000 7f14a9c00640 1 -- 172.21.15.28:0/2928430066 wait complete. 2024-02-21T23:15:59.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:59.980+0000 7f14a9c00640 1 Processor -- start 2024-02-21T23:15:59.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:59.980+0000 7f14a9c00640 1 -- start start 2024-02-21T23:15:59.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:59.981+0000 7f14a9c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f14a40c54c0 0x7f14a40accb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:59.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:59.981+0000 7f14a9c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14a40c6ef0 0x7f14a41eaf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:59.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:59.981+0000 7f14a9c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f14a4152aa0 0x7f14a41551f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:59.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:59.981+0000 7f14a9c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a40c58a0 con 0x7f14a40c6ef0 2024-02-21T23:15:59.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:59.981+0000 7f14a9c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a40c85a0 con 0x7f14a4152aa0 2024-02-21T23:15:59.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:59.981+0000 7f14a9c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14a40c5a20 con 0x7f14a40c54c0 2024-02-21T23:15:59.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:59.982+0000 7f14a2000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14a40c6ef0 0x7f14a41eaf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:59.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:59.982+0000 7f14a2000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14a40c6ef0 0x7f14a41eaf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39722/0 (socket says 172.21.15.28:39722) 2024-02-21T23:15:59.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:59.982+0000 7f14a2000640 1 -- 172.21.15.28:0/454133924 learned_addr learned my addr 172.21.15.28:0/454133924 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:15:59.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:59.982+0000 7f14a2a00640 1 --2- 172.21.15.28:0/454133924 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f14a40c54c0 0x7f14a40accb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:59.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:59.982+0000 7f14a3400640 1 --2- 172.21.15.28:0/454133924 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f14a4152aa0 0x7f14a41551f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:59.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:59.982+0000 7f14a2000640 1 -- 172.21.15.28:0/454133924 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f14a40c54c0 msgr2=0x7f14a40accb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:59.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:59.982+0000 7f14a2000640 1 --2- 172.21.15.28:0/454133924 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f14a40c54c0 0x7f14a40accb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:59.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:59.982+0000 7f14a2000640 1 -- 172.21.15.28:0/454133924 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f14a4152aa0 msgr2=0x7f14a41551f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:15:59.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:59.982+0000 7f14a2000640 1 --2- 172.21.15.28:0/454133924 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f14a4152aa0 0x7f14a41551f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:15:59.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:59.982+0000 7f14a2000640 1 -- 172.21.15.28:0/454133924 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14a4158630 con 0x7f14a40c6ef0 2024-02-21T23:15:59.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:59.982+0000 7f14a2a00640 1 --2- 172.21.15.28:0/454133924 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f14a40c54c0 0x7f14a40accb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:15:59.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:59.982+0000 7f14a3400640 1 --2- 172.21.15.28:0/454133924 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f14a4152aa0 0x7f14a41551f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:15:59.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:59.983+0000 7f14a2000640 1 --2- 172.21.15.28:0/454133924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14a40c6ef0 0x7f14a41eaf00 secure :-1 s=READY pgs=2595 cs=0 l=1 rev1=1 crypto rx=0x7f148c00d5f0 tx=0x7f148c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:59.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:59.983+0000 7f149aa00640 1 -- 172.21.15.28:0/454133924 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f148c0125f0 con 0x7f14a40c6ef0 2024-02-21T23:15:59.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:59.983+0000 7f149aa00640 1 -- 172.21.15.28:0/454133924 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f148c019070 con 0x7f14a40c6ef0 2024-02-21T23:15:59.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:59.983+0000 7f149aa00640 1 -- 172.21.15.28:0/454133924 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f148c0129d0 con 0x7f14a40c6ef0 2024-02-21T23:15:59.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:59.983+0000 7f14a9c00640 1 -- 172.21.15.28:0/454133924 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14a41580b0 con 0x7f14a40c6ef0 2024-02-21T23:15:59.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:59.983+0000 7f14a9c00640 1 -- 172.21.15.28:0/454133924 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14a41570d0 con 0x7f14a40c6ef0 2024-02-21T23:15:59.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:59.984+0000 7f14a9c00640 1 -- 172.21.15.28:0/454133924 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14a4068640 con 0x7f14a40c6ef0 2024-02-21T23:15:59.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:59.985+0000 7f149aa00640 1 -- 172.21.15.28:0/454133924 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f148c012b70 con 0x7f14a40c6ef0 2024-02-21T23:15:59.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:59.986+0000 7f149aa00640 1 --2- 172.21.15.28:0/454133924 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f148006b4d0 0x7f148006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:15:59.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:59.986+0000 7f149aa00640 1 -- 172.21.15.28:0/454133924 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f148c08a8a0 con 0x7f14a40c6ef0 2024-02-21T23:15:59.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:59.987+0000 7f14a2a00640 1 --2- 172.21.15.28:0/454133924 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f148006b4d0 0x7f148006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:15:59.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:59.988+0000 7f14a2a00640 1 --2- 172.21.15.28:0/454133924 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f148006b4d0 0x7f148006d990 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f1494035000 tx=0x7f1494030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:15:59.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:15:59.995+0000 7f149aa00640 1 -- 172.21.15.28:0/454133924 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f148c059480 con 0x7f14a40c6ef0 2024-02-21T23:16:00.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:00.236+0000 7f14a9c00640 1 -- 172.21.15.28:0/454133924 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f14a40c7840 con 0x7f14a40c6ef0 2024-02-21T23:16:00.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:00.237+0000 7f149aa00640 1 -- 172.21.15.28:0/454133924 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7f148c058d80 con 0x7f14a40c6ef0 2024-02-21T23:16:00.225 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:16:00.225 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:16:00.225 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:16:00.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:00.239+0000 7f14a9c00640 1 -- 172.21.15.28:0/454133924 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f148006b4d0 msgr2=0x7f148006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:00.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:00.239+0000 7f14a9c00640 1 --2- 172.21.15.28:0/454133924 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f148006b4d0 0x7f148006d990 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f1494035000 tx=0x7f1494030000 comp rx=0 tx=0).stop 2024-02-21T23:16:00.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:00.239+0000 7f14a9c00640 1 -- 172.21.15.28:0/454133924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14a40c6ef0 msgr2=0x7f14a41eaf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:00.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:00.239+0000 7f14a9c00640 1 --2- 172.21.15.28:0/454133924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14a40c6ef0 0x7f14a41eaf00 secure :-1 s=READY pgs=2595 cs=0 l=1 rev1=1 crypto rx=0x7f148c00d5f0 tx=0x7f148c00dab0 comp rx=0 tx=0).stop 2024-02-21T23:16:00.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:00.239+0000 7f14a9c00640 1 -- 172.21.15.28:0/454133924 shutdown_connections 2024-02-21T23:16:00.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:00.239+0000 7f14a9c00640 1 --2- 172.21.15.28:0/454133924 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f148006b4d0 0x7f148006d990 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:00.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:00.239+0000 7f14a9c00640 1 --2- 172.21.15.28:0/454133924 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f14a4152aa0 0x7f14a41551f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:00.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:00.239+0000 7f14a9c00640 1 --2- 172.21.15.28:0/454133924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14a40c6ef0 0x7f14a41eaf00 unknown :-1 s=CLOSED pgs=2595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:00.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:00.239+0000 7f14a9c00640 1 --2- 172.21.15.28:0/454133924 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f14a40c54c0 0x7f14a40accb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:00.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:00.239+0000 7f14a9c00640 1 -- 172.21.15.28:0/454133924 >> 172.21.15.28:0/454133924 conn(0x7f14a40c8810 msgr2=0x7f14a40c8f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:16:00.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:00.240+0000 7f14a9c00640 1 -- 172.21.15.28:0/454133924 shutdown_connections 2024-02-21T23:16:00.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:00.240+0000 7f14a9c00640 1 -- 172.21.15.28:0/454133924 wait complete. 2024-02-21T23:16:00.651 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:16:00.651 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:16:01.652 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:16:01.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:01.885+0000 7f1a8f400640 1 Processor -- start 2024-02-21T23:16:01.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:01.886+0000 7f1a8f400640 1 -- start start 2024-02-21T23:16:01.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:01.887+0000 7f1a8f400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1a880af970 0x7f1a880c7350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:01.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:01.887+0000 7f1a8f400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f1a88155ca0 0x7f1a8806cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:01.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:01.887+0000 7f1a8f400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a880c5e60 con 0x7f1a880c7890 2024-02-21T23:16:01.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:01.887+0000 7f1a8f400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a881561a0 con 0x7f1a880af970 2024-02-21T23:16:01.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:01.887+0000 7f1a8f400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a880c6140 con 0x7f1a88155ca0 2024-02-21T23:16:01.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:01.888+0000 7f1a8ca00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1a880af970 0x7f1a880c7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:01.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:01.888+0000 7f1a8ca00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1a880af970 0x7f1a880c7350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39736/0 (socket says 172.21.15.28:39736) 2024-02-21T23:16:01.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:01.888+0000 7f1a8ca00640 1 -- 172.21.15.28:0/3526282995 learned_addr learned my addr 172.21.15.28:0/3526282995 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:16:01.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:01.888+0000 7f1a8d400640 1 --1- 172.21.15.28:0/3526282995 >> v1:172.21.15.28:6789/0 conn(0x7f1a880c7890 0x7f1a8806d3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34668/0 (socket says 172.21.15.28:34668) 2024-02-21T23:16:01.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:01.888+0000 7f1a87e00640 1 --2- 172.21.15.28:0/3526282995 >> v2:172.21.15.81:3300/0 conn(0x7f1a88155ca0 0x7f1a8806cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:01.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:01.888+0000 7f1a8ca00640 1 -- 172.21.15.28:0/3526282995 >> v2:172.21.15.81:3300/0 conn(0x7f1a88155ca0 msgr2=0x7f1a8806cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:16:01.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:01.888+0000 7f1a8ca00640 1 --2- 172.21.15.28:0/3526282995 >> v2:172.21.15.81:3300/0 conn(0x7f1a88155ca0 0x7f1a8806cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:01.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:01.888+0000 7f1a8ca00640 1 -- 172.21.15.28:0/3526282995 >> v1:172.21.15.28:6789/0 conn(0x7f1a880c7890 legacy=0x7f1a8806d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:01.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:01.888+0000 7f1a8ca00640 1 -- 172.21.15.28:0/3526282995 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a880c6420 con 0x7f1a880af970 2024-02-21T23:16:01.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:01.888+0000 7f1a87400640 1 -- 172.21.15.28:0/3526282995 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2313894738 0 0) 0x7f1a880c5e60 con 0x7f1a880c7890 2024-02-21T23:16:01.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:01.889+0000 7f1a8ca00640 1 --2- 172.21.15.28:0/3526282995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a880af970 0x7f1a880c7350 secure :-1 s=READY pgs=2597 cs=0 l=1 rev1=1 crypto rx=0x7f1a880c6dc0 tx=0x7f1a7c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f7fdea069e3d40a server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:01.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:01.890+0000 7f1a87400640 1 -- 172.21.15.28:0/3526282995 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a7c035020 con 0x7f1a880af970 2024-02-21T23:16:01.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:01.890+0000 7f1a87400640 1 -- 172.21.15.28:0/3526282995 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a7c031870 con 0x7f1a880af970 2024-02-21T23:16:01.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:01.890+0000 7f1a87400640 1 -- 172.21.15.28:0/3526282995 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a7c031c40 con 0x7f1a880af970 2024-02-21T23:16:01.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:01.891+0000 7f1a8f400640 1 -- 172.21.15.28:0/3526282995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a880af970 msgr2=0x7f1a880c7350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:01.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:01.891+0000 7f1a8f400640 1 --2- 172.21.15.28:0/3526282995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a880af970 0x7f1a880c7350 secure :-1 s=READY pgs=2597 cs=0 l=1 rev1=1 crypto rx=0x7f1a880c6dc0 tx=0x7f1a7c02b910 comp rx=0 tx=0).stop 2024-02-21T23:16:01.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:01.891+0000 7f1a8f400640 1 -- 172.21.15.28:0/3526282995 shutdown_connections 2024-02-21T23:16:01.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:01.891+0000 7f1a8f400640 1 --2- 172.21.15.28:0/3526282995 >> v2:172.21.15.81:3300/0 conn(0x7f1a88155ca0 0x7f1a8806cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:01.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:01.891+0000 7f1a8f400640 1 --2- 172.21.15.28:0/3526282995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a880af970 0x7f1a880c7350 unknown :-1 s=CLOSED pgs=2597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:01.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:01.891+0000 7f1a8f400640 1 -- 172.21.15.28:0/3526282995 >> 172.21.15.28:0/3526282995 conn(0x7f1a880725a0 msgr2=0x7f1a880729b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:16:01.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:01.891+0000 7f1a8f400640 1 -- 172.21.15.28:0/3526282995 shutdown_connections 2024-02-21T23:16:01.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:01.891+0000 7f1a8f400640 1 -- 172.21.15.28:0/3526282995 wait complete. 2024-02-21T23:16:01.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:01.893+0000 7f1a8f400640 1 Processor -- start 2024-02-21T23:16:01.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:01.893+0000 7f1a8f400640 1 -- start start 2024-02-21T23:16:01.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:01.894+0000 7f1a8f400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a880af970 0x7f1a8814efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:01.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:01.894+0000 7f1a8f400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1a880c7890 0x7f1a8814dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:01.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:01.894+0000 7f1a8f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a88155ca0 0x7f1a88158790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:01.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:01.894+0000 7f1a8f400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a880afd50 con 0x7f1a88155ca0 2024-02-21T23:16:01.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:01.894+0000 7f1a8f400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a8806daf0 con 0x7f1a880af970 2024-02-21T23:16:01.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:01.894+0000 7f1a8f400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a880afed0 con 0x7f1a880c7890 2024-02-21T23:16:01.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:01.894+0000 7f1a8d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a88155ca0 0x7f1a88158790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:01.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:01.894+0000 7f1a8d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a88155ca0 0x7f1a88158790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39742/0 (socket says 172.21.15.28:39742) 2024-02-21T23:16:01.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:01.894+0000 7f1a8d400640 1 -- 172.21.15.28:0/3691997961 learned_addr learned my addr 172.21.15.28:0/3691997961 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:16:01.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:01.894+0000 7f1a87e00640 1 --2- 172.21.15.28:0/3691997961 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1a880c7890 0x7f1a8814dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:01.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:01.894+0000 7f1a8ca00640 1 --2- 172.21.15.28:0/3691997961 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a880af970 0x7f1a8814efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:01.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:01.894+0000 7f1a8d400640 1 -- 172.21.15.28:0/3691997961 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1a880c7890 msgr2=0x7f1a8814dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:01.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:01.894+0000 7f1a8d400640 1 --2- 172.21.15.28:0/3691997961 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1a880c7890 0x7f1a8814dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:01.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:01.894+0000 7f1a8d400640 1 -- 172.21.15.28:0/3691997961 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a880af970 msgr2=0x7f1a8814efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:01.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:01.894+0000 7f1a8d400640 1 --2- 172.21.15.28:0/3691997961 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a880af970 0x7f1a8814efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:01.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:01.895+0000 7f1a8d400640 1 -- 172.21.15.28:0/3691997961 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a8814ea20 con 0x7f1a88155ca0 2024-02-21T23:16:01.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:01.895+0000 7f1a87e00640 1 --2- 172.21.15.28:0/3691997961 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1a880c7890 0x7f1a8814dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:16:01.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:01.895+0000 7f1a8ca00640 1 --2- 172.21.15.28:0/3691997961 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a880af970 0x7f1a8814efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:16:01.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:01.895+0000 7f1a8d400640 1 --2- 172.21.15.28:0/3691997961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a88155ca0 0x7f1a88158790 secure :-1 s=READY pgs=2598 cs=0 l=1 rev1=1 crypto rx=0x7f1a8000c7a0 tx=0x7f1a8000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:01.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:01.895+0000 7f1a84c00640 1 -- 172.21.15.28:0/3691997961 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a80018070 con 0x7f1a88155ca0 2024-02-21T23:16:01.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:01.895+0000 7f1a84c00640 1 -- 172.21.15.28:0/3691997961 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a8000a640 con 0x7f1a88155ca0 2024-02-21T23:16:01.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:01.895+0000 7f1a8f400640 1 -- 172.21.15.28:0/3691997961 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a8814fa90 con 0x7f1a88155ca0 2024-02-21T23:16:01.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:01.895+0000 7f1a84c00640 1 -- 172.21.15.28:0/3691997961 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a8000aab0 con 0x7f1a88155ca0 2024-02-21T23:16:01.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:01.895+0000 7f1a8f400640 1 -- 172.21.15.28:0/3691997961 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a88157820 con 0x7f1a88155ca0 2024-02-21T23:16:01.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:01.896+0000 7f1a8f400640 1 -- 172.21.15.28:0/3691997961 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a88157650 con 0x7f1a88155ca0 2024-02-21T23:16:01.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:01.898+0000 7f1a84c00640 1 -- 172.21.15.28:0/3691997961 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1a8001b070 con 0x7f1a88155ca0 2024-02-21T23:16:01.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:01.899+0000 7f1a84c00640 1 --2- 172.21.15.28:0/3691997961 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1a6806b4d0 0x7f1a6806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:01.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:01.899+0000 7f1a84c00640 1 -- 172.21.15.28:0/3691997961 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f1a80089a40 con 0x7f1a88155ca0 2024-02-21T23:16:01.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:01.900+0000 7f1a8ca00640 1 --2- 172.21.15.28:0/3691997961 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1a6806b4d0 0x7f1a6806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:01.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:01.901+0000 7f1a8ca00640 1 --2- 172.21.15.28:0/3691997961 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1a6806b4d0 0x7f1a6806d990 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f1a7c036000 tx=0x7f1a7c008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:01.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:01.903+0000 7f1a84c00640 1 -- 172.21.15.28:0/3691997961 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1a800586c0 con 0x7f1a88155ca0 2024-02-21T23:16:02.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:02.163+0000 7f1a8f400640 1 -- 172.21.15.28:0/3691997961 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a8814ffb0 con 0x7f1a88155ca0 2024-02-21T23:16:02.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:02.163+0000 7f1a84c00640 1 -- 172.21.15.28:0/3691997961 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7f1a80089e20 con 0x7f1a88155ca0 2024-02-21T23:16:02.152 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:16:02.152 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:16:02.152 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:16:02.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:02.166+0000 7f1a8f400640 1 -- 172.21.15.28:0/3691997961 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1a6806b4d0 msgr2=0x7f1a6806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:02.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:02.166+0000 7f1a8f400640 1 --2- 172.21.15.28:0/3691997961 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1a6806b4d0 0x7f1a6806d990 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f1a7c036000 tx=0x7f1a7c008030 comp rx=0 tx=0).stop 2024-02-21T23:16:02.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:02.166+0000 7f1a8f400640 1 -- 172.21.15.28:0/3691997961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a88155ca0 msgr2=0x7f1a88158790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:02.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:02.166+0000 7f1a8f400640 1 --2- 172.21.15.28:0/3691997961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a88155ca0 0x7f1a88158790 secure :-1 s=READY pgs=2598 cs=0 l=1 rev1=1 crypto rx=0x7f1a8000c7a0 tx=0x7f1a8000cc60 comp rx=0 tx=0).stop 2024-02-21T23:16:02.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:02.167+0000 7f1a8f400640 1 -- 172.21.15.28:0/3691997961 shutdown_connections 2024-02-21T23:16:02.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:02.167+0000 7f1a8f400640 1 --2- 172.21.15.28:0/3691997961 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1a6806b4d0 0x7f1a6806d990 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:02.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:02.167+0000 7f1a8f400640 1 --2- 172.21.15.28:0/3691997961 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1a880c7890 0x7f1a8814dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:02.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:02.167+0000 7f1a8f400640 1 --2- 172.21.15.28:0/3691997961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a88155ca0 0x7f1a88158790 unknown :-1 s=CLOSED pgs=2598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:02.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:02.167+0000 7f1a8f400640 1 --2- 172.21.15.28:0/3691997961 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1a880af970 0x7f1a8814efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:02.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:02.167+0000 7f1a8f400640 1 -- 172.21.15.28:0/3691997961 >> 172.21.15.28:0/3691997961 conn(0x7f1a880725a0 msgr2=0x7f1a880c08c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:16:02.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:02.167+0000 7f1a8f400640 1 -- 172.21.15.28:0/3691997961 shutdown_connections 2024-02-21T23:16:02.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:02.167+0000 7f1a8f400640 1 -- 172.21.15.28:0/3691997961 wait complete. 2024-02-21T23:16:02.575 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:16:02.575 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:16:03.577 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:16:03.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:03.812+0000 7fa831e00640 1 Processor -- start 2024-02-21T23:16:03.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:03.812+0000 7fa831e00640 1 -- start start 2024-02-21T23:16:03.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:03.814+0000 7fa831e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa82c0a9930 0x7fa82c0a6f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:03.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:03.814+0000 7fa831e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa82c0c1a40 0x7fa82c152bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:03.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:03.814+0000 7fa831e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa82c156c50 con 0x7fa82c0c2120 2024-02-21T23:16:03.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:03.814+0000 7fa831e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa82c072a40 con 0x7fa82c0a9930 2024-02-21T23:16:03.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:03.814+0000 7fa831e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa82c156ef0 con 0x7fa82c0c1a40 2024-02-21T23:16:03.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:03.814+0000 7fa830a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa82c0a9930 0x7fa82c0a6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:03.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:03.814+0000 7fa830a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa82c0a9930 0x7fa82c0a6f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39754/0 (socket says 172.21.15.28:39754) 2024-02-21T23:16:03.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:03.814+0000 7fa830a00640 1 -- 172.21.15.28:0/148623723 learned_addr learned my addr 172.21.15.28:0/148623723 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:16:03.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:03.814+0000 7fa82be00640 1 --2- 172.21.15.28:0/148623723 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa82c0c1a40 0x7fa82c152bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:03.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:03.815+0000 7fa830a00640 1 -- 172.21.15.28:0/148623723 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa82c0c1a40 msgr2=0x7fa82c152bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:16:03.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:03.815+0000 7fa830a00640 1 --2- 172.21.15.28:0/148623723 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa82c0c1a40 0x7fa82c152bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:03.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:03.815+0000 7fa830a00640 1 -- 172.21.15.28:0/148623723 >> v1:172.21.15.28:6789/0 conn(0x7fa82c0c2120 legacy=0x7fa82c14fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:03.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:03.815+0000 7fa830a00640 1 -- 172.21.15.28:0/148623723 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa82c073450 con 0x7fa82c0a9930 2024-02-21T23:16:03.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:03.816+0000 7fa830a00640 1 --2- 172.21.15.28:0/148623723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa82c0a9930 0x7fa82c0a6f50 secure :-1 s=READY pgs=2600 cs=0 l=1 rev1=1 crypto rx=0x7fa82000ded0 tx=0x7fa820034230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12202b55337959d0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:03.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:03.816+0000 7fa82b400640 1 -- 172.21.15.28:0/148623723 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa820034dd0 con 0x7fa82c0a9930 2024-02-21T23:16:03.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:03.816+0000 7fa82b400640 1 -- 172.21.15.28:0/148623723 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa82003c3f0 con 0x7fa82c0a9930 2024-02-21T23:16:03.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:03.816+0000 7fa82b400640 1 -- 172.21.15.28:0/148623723 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa82003c7c0 con 0x7fa82c0a9930 2024-02-21T23:16:03.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:03.817+0000 7fa831e00640 1 -- 172.21.15.28:0/148623723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa82c0a9930 msgr2=0x7fa82c0a6f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:03.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:03.817+0000 7fa831e00640 1 --2- 172.21.15.28:0/148623723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa82c0a9930 0x7fa82c0a6f50 secure :-1 s=READY pgs=2600 cs=0 l=1 rev1=1 crypto rx=0x7fa82000ded0 tx=0x7fa820034230 comp rx=0 tx=0).stop 2024-02-21T23:16:03.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:03.817+0000 7fa831e00640 1 -- 172.21.15.28:0/148623723 shutdown_connections 2024-02-21T23:16:03.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:03.817+0000 7fa831e00640 1 --2- 172.21.15.28:0/148623723 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa82c0c1a40 0x7fa82c152bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:03.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:03.817+0000 7fa831e00640 1 --2- 172.21.15.28:0/148623723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa82c0a9930 0x7fa82c0a6f50 unknown :-1 s=CLOSED pgs=2600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:03.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:03.817+0000 7fa831e00640 1 -- 172.21.15.28:0/148623723 >> 172.21.15.28:0/148623723 conn(0x7fa82c0c7720 msgr2=0x7fa82c0c7b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:16:03.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:03.817+0000 7fa831e00640 1 -- 172.21.15.28:0/148623723 shutdown_connections 2024-02-21T23:16:03.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:03.817+0000 7fa831e00640 1 -- 172.21.15.28:0/148623723 wait complete. 2024-02-21T23:16:03.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:03.820+0000 7fa831e00640 1 Processor -- start 2024-02-21T23:16:03.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:03.820+0000 7fa831e00640 1 -- start start 2024-02-21T23:16:03.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:03.821+0000 7fa831e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa82c0a9930 0x7fa82c155150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:03.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:03.821+0000 7fa831e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa82c0c1a40 0x7fa82c0bdc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:03.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:03.821+0000 7fa831e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa82c0c2120 0x7fa82c1e8980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:03.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:03.821+0000 7fa831e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa82c1505b0 con 0x7fa82c0c2120 2024-02-21T23:16:03.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:03.821+0000 7fa831e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa82c0a9f20 con 0x7fa82c0a9930 2024-02-21T23:16:03.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:03.821+0000 7fa831e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa82c150730 con 0x7fa82c0c1a40 2024-02-21T23:16:03.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:03.822+0000 7fa831400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa82c0c2120 0x7fa82c1e8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:03.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:03.822+0000 7fa831400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa82c0c2120 0x7fa82c1e8980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39770/0 (socket says 172.21.15.28:39770) 2024-02-21T23:16:03.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:03.822+0000 7fa831400640 1 -- 172.21.15.28:0/1271374170 learned_addr learned my addr 172.21.15.28:0/1271374170 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:16:03.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:03.822+0000 7fa830a00640 1 --2- 172.21.15.28:0/1271374170 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa82c0a9930 0x7fa82c155150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:03.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:03.822+0000 7fa82be00640 1 --2- 172.21.15.28:0/1271374170 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa82c0c1a40 0x7fa82c0bdc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:03.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:03.822+0000 7fa831400640 1 -- 172.21.15.28:0/1271374170 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa82c0c1a40 msgr2=0x7fa82c0bdc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:03.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:03.822+0000 7fa831400640 1 --2- 172.21.15.28:0/1271374170 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa82c0c1a40 0x7fa82c0bdc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:03.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:03.822+0000 7fa831400640 1 -- 172.21.15.28:0/1271374170 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa82c0a9930 msgr2=0x7fa82c155150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:03.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:03.822+0000 7fa831400640 1 --2- 172.21.15.28:0/1271374170 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa82c0a9930 0x7fa82c155150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:03.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:03.822+0000 7fa831400640 1 -- 172.21.15.28:0/1271374170 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa82c0ad470 con 0x7fa82c0c2120 2024-02-21T23:16:03.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:03.823+0000 7fa831400640 1 --2- 172.21.15.28:0/1271374170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa82c0c2120 0x7fa82c1e8980 secure :-1 s=READY pgs=2601 cs=0 l=1 rev1=1 crypto rx=0x7fa82400c7d0 tx=0x7fa82400cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:03.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:03.823+0000 7fa828c00640 1 -- 172.21.15.28:0/1271374170 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa824003b70 con 0x7fa82c0c2120 2024-02-21T23:16:03.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:03.823+0000 7fa828c00640 1 -- 172.21.15.28:0/1271374170 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa824003d30 con 0x7fa82c0c2120 2024-02-21T23:16:03.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:03.823+0000 7fa828c00640 1 -- 172.21.15.28:0/1271374170 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa82400aa90 con 0x7fa82c0c2120 2024-02-21T23:16:03.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:03.823+0000 7fa831e00640 1 -- 172.21.15.28:0/1271374170 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa82c0ad750 con 0x7fa82c0c2120 2024-02-21T23:16:03.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:03.823+0000 7fa831e00640 1 -- 172.21.15.28:0/1271374170 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa82c1e8180 con 0x7fa82c0c2120 2024-02-21T23:16:03.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:03.825+0000 7fa831e00640 1 -- 172.21.15.28:0/1271374170 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa82c04f290 con 0x7fa82c0c2120 2024-02-21T23:16:03.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:03.827+0000 7fa828c00640 1 -- 172.21.15.28:0/1271374170 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa82400ac30 con 0x7fa82c0c2120 2024-02-21T23:16:03.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:03.828+0000 7fa828c00640 1 --2- 172.21.15.28:0/1271374170 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa80c06b4d0 0x7fa80c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:03.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:03.828+0000 7fa828c00640 1 -- 172.21.15.28:0/1271374170 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fa82408a460 con 0x7fa82c0c2120 2024-02-21T23:16:03.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:03.828+0000 7fa830a00640 1 --2- 172.21.15.28:0/1271374170 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa80c06b4d0 0x7fa80c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:03.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:03.830+0000 7fa830a00640 1 --2- 172.21.15.28:0/1271374170 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa80c06b4d0 0x7fa80c06d990 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7fa82003d000 tx=0x7fa820038000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:03.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:03.832+0000 7fa828c00640 1 -- 172.21.15.28:0/1271374170 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fa824058fb0 con 0x7fa82c0c2120 2024-02-21T23:16:04.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:04.076+0000 7fa831e00640 1 -- 172.21.15.28:0/1271374170 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa82c0745b0 con 0x7fa82c0c2120 2024-02-21T23:16:04.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:04.077+0000 7fa828c00640 1 -- 172.21.15.28:0/1271374170 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7fa8240588b0 con 0x7fa82c0c2120 2024-02-21T23:16:04.065 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:16:04.065 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:16:04.065 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:16:04.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:04.080+0000 7fa831e00640 1 -- 172.21.15.28:0/1271374170 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa80c06b4d0 msgr2=0x7fa80c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:04.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:04.080+0000 7fa831e00640 1 --2- 172.21.15.28:0/1271374170 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa80c06b4d0 0x7fa80c06d990 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7fa82003d000 tx=0x7fa820038000 comp rx=0 tx=0).stop 2024-02-21T23:16:04.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:04.080+0000 7fa831e00640 1 -- 172.21.15.28:0/1271374170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa82c0c2120 msgr2=0x7fa82c1e8980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:04.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:04.080+0000 7fa831e00640 1 --2- 172.21.15.28:0/1271374170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa82c0c2120 0x7fa82c1e8980 secure :-1 s=READY pgs=2601 cs=0 l=1 rev1=1 crypto rx=0x7fa82400c7d0 tx=0x7fa82400cc90 comp rx=0 tx=0).stop 2024-02-21T23:16:04.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:04.080+0000 7fa831e00640 1 -- 172.21.15.28:0/1271374170 shutdown_connections 2024-02-21T23:16:04.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:04.080+0000 7fa831e00640 1 --2- 172.21.15.28:0/1271374170 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa80c06b4d0 0x7fa80c06d990 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:04.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:04.080+0000 7fa831e00640 1 --2- 172.21.15.28:0/1271374170 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa82c0c1a40 0x7fa82c0bdc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:04.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:04.080+0000 7fa831e00640 1 --2- 172.21.15.28:0/1271374170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa82c0c2120 0x7fa82c1e8980 unknown :-1 s=CLOSED pgs=2601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:04.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:04.080+0000 7fa831e00640 1 --2- 172.21.15.28:0/1271374170 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa82c0a9930 0x7fa82c155150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:04.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:04.080+0000 7fa831e00640 1 -- 172.21.15.28:0/1271374170 >> 172.21.15.28:0/1271374170 conn(0x7fa82c0c7720 msgr2=0x7fa82c14dcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:16:04.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:04.080+0000 7fa831e00640 1 -- 172.21.15.28:0/1271374170 shutdown_connections 2024-02-21T23:16:04.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:04.080+0000 7fa831e00640 1 -- 172.21.15.28:0/1271374170 wait complete. 2024-02-21T23:16:04.499 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:16:04.500 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:16:05.501 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:16:05.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.734+0000 7f2cf2400640 1 Processor -- start 2024-02-21T23:16:05.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.734+0000 7f2cf2400640 1 -- start start 2024-02-21T23:16:05.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.736+0000 7f2cf2400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2cec0aff90 0x7f2cec0c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:05.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.736+0000 7f2cf2400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cec0c74c0 con 0x7f2cec06dbf0 2024-02-21T23:16:05.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.736+0000 7f2cf2400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cec0c77a0 con 0x7f2cec0c5720 2024-02-21T23:16:05.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.736+0000 7f2cf2400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cec0c7270 con 0x7f2cec0aff90 2024-02-21T23:16:05.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.737+0000 7f2cebe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f2cec06dbf0 0x7f2cec158ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34932/0 (socket says 172.21.15.28:34932) 2024-02-21T23:16:05.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.737+0000 7f2cebe00640 1 -- 172.21.15.28:0/4043644415 learned_addr learned my addr 172.21.15.28:0/4043644415 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:16:05.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.737+0000 7f2cf1000640 1 --2- 172.21.15.28:0/4043644415 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2cec0aff90 0x7f2cec0c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:05.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.737+0000 7f2ceb400640 1 -- 172.21.15.28:0/4043644415 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2311951651 0 0) 0x7f2cec0c74c0 con 0x7f2cec06dbf0 2024-02-21T23:16:05.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.737+0000 7f2ceb400640 1 -- 172.21.15.28:0/4043644415 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cd4003520 con 0x7f2cec06dbf0 2024-02-21T23:16:05.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.737+0000 7f2ceb400640 1 -- 172.21.15.28:0/4043644415 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3773612859 0 0) 0x7f2cd4003520 con 0x7f2cec06dbf0 2024-02-21T23:16:05.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.738+0000 7f2ceb400640 1 -- 172.21.15.28:0/4043644415 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2cec0aff90 msgr2=0x7f2cec0c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:16:05.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.738+0000 7f2ceb400640 1 --2- 172.21.15.28:0/4043644415 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2cec0aff90 0x7f2cec0c51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:05.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.738+0000 7f2ceb400640 1 -- 172.21.15.28:0/4043644415 >> v1:172.21.15.81:6789/0 conn(0x7f2cec0c5720 legacy=0x7f2cec15f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:05.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.738+0000 7f2ceb400640 1 -- 172.21.15.28:0/4043644415 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cec0c7a80 con 0x7f2cec06dbf0 2024-02-21T23:16:05.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.738+0000 7f2ceb400640 1 -- 172.21.15.28:0/4043644415 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f2cd8001670 con 0x7f2cec06dbf0 2024-02-21T23:16:05.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.738+0000 7f2ceb400640 1 -- 172.21.15.28:0/4043644415 >> v1:172.21.15.28:6789/0 conn(0x7f2cec06dbf0 legacy=0x7f2cec158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:05.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.738+0000 7f2ceb400640 1 --2- 172.21.15.28:0/4043644415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cd40039e0 0x7f2cd4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:05.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.738+0000 7f2ceb400640 1 --2- 172.21.15.28:0/4043644415 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2cec0aff90 0x7f2cd4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:05.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.738+0000 7f2ceb400640 1 --2- 172.21.15.28:0/4043644415 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2cd4004d40 0x7f2cd40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:05.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.738+0000 7f2ceb400640 1 -- 172.21.15.28:0/4043644415 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cec0c7270 con 0x7f2cd40039e0 2024-02-21T23:16:05.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.738+0000 7f2ceb400640 1 -- 172.21.15.28:0/4043644415 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd400baa0 con 0x7f2cd4004d40 2024-02-21T23:16:05.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.738+0000 7f2ceb400640 1 -- 172.21.15.28:0/4043644415 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd400bca0 con 0x7f2cec0aff90 2024-02-21T23:16:05.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.738+0000 7f2cf1000640 1 --2- 172.21.15.28:0/4043644415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cd40039e0 0x7f2cd4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:05.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.738+0000 7f2cebe00640 1 --2- 172.21.15.28:0/4043644415 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2cec0aff90 0x7f2cd4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:05.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.738+0000 7f2cf1a00640 1 --2- 172.21.15.28:0/4043644415 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2cd4004d40 0x7f2cd40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:05.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.739+0000 7f2cf1000640 1 -- 172.21.15.28:0/4043644415 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2cec0aff90 msgr2=0x7f2cd4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:16:05.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.739+0000 7f2cf1000640 1 --2- 172.21.15.28:0/4043644415 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2cec0aff90 0x7f2cd4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:05.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.739+0000 7f2cf1000640 1 -- 172.21.15.28:0/4043644415 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2cd4004d40 msgr2=0x7f2cd40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:16:05.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.739+0000 7f2cf1000640 1 --2- 172.21.15.28:0/4043644415 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2cd4004d40 0x7f2cd40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:05.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.739+0000 7f2cf1000640 1 -- 172.21.15.28:0/4043644415 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cd4003520 con 0x7f2cd40039e0 2024-02-21T23:16:05.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.740+0000 7f2cf1000640 1 --2- 172.21.15.28:0/4043644415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cd40039e0 0x7f2cd4003e00 secure :-1 s=READY pgs=2603 cs=0 l=1 rev1=1 crypto rx=0x7f2ce0009a90 tx=0x7f2cd4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5c08b85711162e5 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:05.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.740+0000 7f2ceb400640 1 -- 172.21.15.28:0/4043644415 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ce0039070 con 0x7f2cd40039e0 2024-02-21T23:16:05.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.740+0000 7f2ceb400640 1 -- 172.21.15.28:0/4043644415 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ce002c680 con 0x7f2cd40039e0 2024-02-21T23:16:05.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.740+0000 7f2ceb400640 1 -- 172.21.15.28:0/4043644415 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ce0009690 con 0x7f2cd40039e0 2024-02-21T23:16:05.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.741+0000 7f2cf2400640 1 -- 172.21.15.28:0/4043644415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cd40039e0 msgr2=0x7f2cd4003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:05.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.741+0000 7f2cf2400640 1 --2- 172.21.15.28:0/4043644415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cd40039e0 0x7f2cd4003e00 secure :-1 s=READY pgs=2603 cs=0 l=1 rev1=1 crypto rx=0x7f2ce0009a90 tx=0x7f2cd4005830 comp rx=0 tx=0).stop 2024-02-21T23:16:05.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.741+0000 7f2cf1a00640 1 -- 172.21.15.28:0/4043644415 reap_dead start 2024-02-21T23:16:05.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.741+0000 7f2cf2400640 1 -- 172.21.15.28:0/4043644415 shutdown_connections 2024-02-21T23:16:05.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.741+0000 7f2cf2400640 1 -- 172.21.15.28:0/4043644415 >> 172.21.15.28:0/4043644415 conn(0x7f2cec0ac370 msgr2=0x7f2cec0ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:16:05.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.741+0000 7f2cf2400640 1 -- 172.21.15.28:0/4043644415 shutdown_connections 2024-02-21T23:16:05.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.741+0000 7f2cf2400640 1 -- 172.21.15.28:0/4043644415 wait complete. 2024-02-21T23:16:05.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.743+0000 7f2cf2400640 1 Processor -- start 2024-02-21T23:16:05.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.743+0000 7f2cf2400640 1 -- start start 2024-02-21T23:16:05.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.744+0000 7f2cf2400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cd40039e0 0x7f2cec0aac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:05.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.744+0000 7f2cf2400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2cec1df750 0x7f2cec0c1b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:05.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.744+0000 7f2cf2400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2cec1ded70 0x7f2cec1ee180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:05.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.744+0000 7f2cf2400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cec0c8760 con 0x7f2cd40039e0 2024-02-21T23:16:05.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.744+0000 7f2cf2400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cec158630 con 0x7f2cec1df750 2024-02-21T23:16:05.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.744+0000 7f2cf2400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cec0c88e0 con 0x7f2cec1ded70 2024-02-21T23:16:05.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.744+0000 7f2cf1000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cd40039e0 0x7f2cec0aac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:05.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.744+0000 7f2cf1000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cd40039e0 0x7f2cec0aac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36346/0 (socket says 172.21.15.28:36346) 2024-02-21T23:16:05.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.744+0000 7f2cf1000640 1 -- 172.21.15.28:0/1656581096 learned_addr learned my addr 172.21.15.28:0/1656581096 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:16:05.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.744+0000 7f2cebe00640 1 --2- 172.21.15.28:0/1656581096 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2cec1df750 0x7f2cec0c1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:05.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.744+0000 7f2cf1a00640 1 --2- 172.21.15.28:0/1656581096 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2cec1ded70 0x7f2cec1ee180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:05.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.745+0000 7f2cf1000640 1 -- 172.21.15.28:0/1656581096 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2cec1ded70 msgr2=0x7f2cec1ee180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:05.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.745+0000 7f2cf1000640 1 --2- 172.21.15.28:0/1656581096 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2cec1ded70 0x7f2cec1ee180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:05.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.745+0000 7f2cf1000640 1 -- 172.21.15.28:0/1656581096 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2cec1df750 msgr2=0x7f2cec0c1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:05.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.745+0000 7f2cf1000640 1 --2- 172.21.15.28:0/1656581096 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2cec1df750 0x7f2cec0c1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:05.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.745+0000 7f2cf1000640 1 -- 172.21.15.28:0/1656581096 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cec0c2390 con 0x7f2cd40039e0 2024-02-21T23:16:05.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.745+0000 7f2cf1a00640 1 --2- 172.21.15.28:0/1656581096 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2cec1ded70 0x7f2cec1ee180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:16:05.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.745+0000 7f2cf1000640 1 --2- 172.21.15.28:0/1656581096 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cd40039e0 0x7f2cec0aac10 secure :-1 s=READY pgs=2604 cs=0 l=1 rev1=1 crypto rx=0x7f2ce0009bb0 tx=0x7f2ce0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:05.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.745+0000 7f2ce8c00640 1 -- 172.21.15.28:0/1656581096 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ce0039050 con 0x7f2cd40039e0 2024-02-21T23:16:05.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.745+0000 7f2ce8c00640 1 -- 172.21.15.28:0/1656581096 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ce0045070 con 0x7f2cd40039e0 2024-02-21T23:16:05.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.745+0000 7f2ce8c00640 1 -- 172.21.15.28:0/1656581096 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ce0003850 con 0x7f2cd40039e0 2024-02-21T23:16:05.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.745+0000 7f2cf2400640 1 -- 172.21.15.28:0/1656581096 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cec0c2670 con 0x7f2cd40039e0 2024-02-21T23:16:05.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.746+0000 7f2cf2400640 1 -- 172.21.15.28:0/1656581096 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cec1de570 con 0x7f2cd40039e0 2024-02-21T23:16:05.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.747+0000 7f2cf2400640 1 -- 172.21.15.28:0/1656581096 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cec04f290 con 0x7f2cd40039e0 2024-02-21T23:16:05.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.747+0000 7f2ce8c00640 1 -- 172.21.15.28:0/1656581096 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2ce00039f0 con 0x7f2cd40039e0 2024-02-21T23:16:05.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.748+0000 7f2ce8c00640 1 --2- 172.21.15.28:0/1656581096 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2ccc06b4d0 0x7f2ccc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:05.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.749+0000 7f2ce8c00640 1 -- 172.21.15.28:0/1656581096 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f2ce00ac640 con 0x7f2cd40039e0 2024-02-21T23:16:05.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.749+0000 7f2cebe00640 1 --2- 172.21.15.28:0/1656581096 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2ccc06b4d0 0x7f2ccc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:05.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.750+0000 7f2cebe00640 1 --2- 172.21.15.28:0/1656581096 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2ccc06b4d0 0x7f2ccc06d990 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f2cd8001050 tx=0x7f2cd8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:05.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.753+0000 7f2ce8c00640 1 -- 172.21.15.28:0/1656581096 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f2ce007b250 con 0x7f2cd40039e0 2024-02-21T23:16:05.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.996+0000 7f2cf2400640 1 -- 172.21.15.28:0/1656581096 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2cec0acc40 con 0x7f2cd40039e0 2024-02-21T23:16:05.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:05.998+0000 7f2ce8c00640 1 -- 172.21.15.28:0/1656581096 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7f2ce007ab50 con 0x7f2cd40039e0 2024-02-21T23:16:05.985 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:16:05.985 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:16:05.986 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:16:05.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:06.001+0000 7f2cf2400640 1 -- 172.21.15.28:0/1656581096 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2ccc06b4d0 msgr2=0x7f2ccc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:05.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:06.001+0000 7f2cf2400640 1 --2- 172.21.15.28:0/1656581096 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2ccc06b4d0 0x7f2ccc06d990 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f2cd8001050 tx=0x7f2cd8009040 comp rx=0 tx=0).stop 2024-02-21T23:16:05.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:06.001+0000 7f2cf2400640 1 -- 172.21.15.28:0/1656581096 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cd40039e0 msgr2=0x7f2cec0aac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:05.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:06.001+0000 7f2cf2400640 1 --2- 172.21.15.28:0/1656581096 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cd40039e0 0x7f2cec0aac10 secure :-1 s=READY pgs=2604 cs=0 l=1 rev1=1 crypto rx=0x7f2ce0009bb0 tx=0x7f2ce0004730 comp rx=0 tx=0).stop 2024-02-21T23:16:05.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:06.001+0000 7f2cf2400640 1 -- 172.21.15.28:0/1656581096 shutdown_connections 2024-02-21T23:16:05.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:06.001+0000 7f2cf2400640 1 --2- 172.21.15.28:0/1656581096 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2ccc06b4d0 0x7f2ccc06d990 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:05.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:06.001+0000 7f2cf2400640 1 --2- 172.21.15.28:0/1656581096 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2cec1ded70 0x7f2cec1ee180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:05.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:06.001+0000 7f2cf2400640 1 --2- 172.21.15.28:0/1656581096 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2cec1df750 0x7f2cec0c1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:05.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:06.001+0000 7f2cf2400640 1 --2- 172.21.15.28:0/1656581096 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cd40039e0 0x7f2cec0aac10 unknown :-1 s=CLOSED pgs=2604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:05.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:06.001+0000 7f2cf2400640 1 -- 172.21.15.28:0/1656581096 >> 172.21.15.28:0/1656581096 conn(0x7f2cec0ac370 msgr2=0x7f2cec0c8ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:16:05.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:06.001+0000 7f2cf2400640 1 -- 172.21.15.28:0/1656581096 shutdown_connections 2024-02-21T23:16:05.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:06.001+0000 7f2cf2400640 1 -- 172.21.15.28:0/1656581096 wait complete. 2024-02-21T23:16:06.411 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:16:06.411 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:16:07.413 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:16:07.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.651+0000 7fcdfbe00640 1 Processor -- start 2024-02-21T23:16:07.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.652+0000 7fcdfbe00640 1 -- start start 2024-02-21T23:16:07.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.653+0000 7fcdfbe00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcdfc06b0e0 0x7fcdfc06b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:07.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.653+0000 7fcdfbe00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fcdfc152680 0x7fcdfc154ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:07.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.654+0000 7fcdfbe00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdfc0c0a80 con 0x7fcdfc06f950 2024-02-21T23:16:07.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.654+0000 7fcdfbe00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdfc072880 con 0x7fcdfc152680 2024-02-21T23:16:07.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.654+0000 7fcdfbe00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdfc0c0d60 con 0x7fcdfc06b0e0 2024-02-21T23:16:07.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.654+0000 7fcdfb400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fcdfc152680 0x7fcdfc154ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:07.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.654+0000 7fcdfaa00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcdfc06b0e0 0x7fcdfc06b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:07.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.654+0000 7fcdfb400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fcdfc152680 0x7fcdfc154ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.28:47158/0 (socket says 172.21.15.28:47158) 2024-02-21T23:16:07.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.654+0000 7fcdfa000640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fcdfc06f950 0x7fcdfc151f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.28:55986/0 (socket says 172.21.15.28:55986) 2024-02-21T23:16:07.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.654+0000 7fcdfaa00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcdfc06b0e0 0x7fcdfc06b4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3301/0 says I am v2:172.21.15.28:47266/0 (socket says 172.21.15.28:47266) 2024-02-21T23:16:07.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.654+0000 7fcdfb400640 1 -- 172.21.15.28:0/779440974 learned_addr learned my addr 172.21.15.28:0/779440974 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:16:07.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.655+0000 7fcdfb400640 1 -- 172.21.15.28:0/779440974 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcdfc06b0e0 msgr2=0x7fcdfc06b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:16:07.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.655+0000 7fcdfb400640 1 --2- 172.21.15.28:0/779440974 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcdfc06b0e0 0x7fcdfc06b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:07.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.655+0000 7fcdfb400640 1 -- 172.21.15.28:0/779440974 >> v1:172.21.15.81:6789/0 conn(0x7fcdfc06f950 legacy=0x7fcdfc151f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:07.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.655+0000 7fcdfb400640 1 -- 172.21.15.28:0/779440974 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdfc072ad0 con 0x7fcdfc152680 2024-02-21T23:16:07.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.656+0000 7fcdfb400640 1 --2- 172.21.15.28:0/779440974 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcdfc152680 0x7fcdfc154ea0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fcdf4009e50 tx=0x7fcdf402bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ece5ad56022d9a6 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:07.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.657+0000 7fcdf9600640 1 -- 172.21.15.28:0/779440974 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf402ca70 con 0x7fcdfc152680 2024-02-21T23:16:07.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.657+0000 7fcdf9600640 1 -- 172.21.15.28:0/779440974 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdf402cc10 con 0x7fcdfc152680 2024-02-21T23:16:07.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.657+0000 7fcdf9600640 1 -- 172.21.15.28:0/779440974 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf40343f0 con 0x7fcdfc152680 2024-02-21T23:16:07.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.658+0000 7fcdfbe00640 1 -- 172.21.15.28:0/779440974 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcdfc152680 msgr2=0x7fcdfc154ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:07.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.658+0000 7fcdfbe00640 1 --2- 172.21.15.28:0/779440974 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcdfc152680 0x7fcdfc154ea0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fcdf4009e50 tx=0x7fcdf402bed0 comp rx=0 tx=0).stop 2024-02-21T23:16:07.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.658+0000 7fcdfbe00640 1 -- 172.21.15.28:0/779440974 shutdown_connections 2024-02-21T23:16:07.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.658+0000 7fcdfbe00640 1 --2- 172.21.15.28:0/779440974 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcdfc152680 0x7fcdfc154ea0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:07.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.658+0000 7fcdfbe00640 1 --2- 172.21.15.28:0/779440974 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcdfc06b0e0 0x7fcdfc06b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:07.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.658+0000 7fcdfbe00640 1 -- 172.21.15.28:0/779440974 >> 172.21.15.28:0/779440974 conn(0x7fcdfc06e8c0 msgr2=0x7fcdfc06bf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:16:07.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.658+0000 7fcdfbe00640 1 -- 172.21.15.28:0/779440974 shutdown_connections 2024-02-21T23:16:07.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.658+0000 7fcdfbe00640 1 -- 172.21.15.28:0/779440974 wait complete. 2024-02-21T23:16:07.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.660+0000 7fcdfbe00640 1 Processor -- start 2024-02-21T23:16:07.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.660+0000 7fcdfbe00640 1 -- start start 2024-02-21T23:16:07.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.661+0000 7fcdfbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdfc06b0e0 0x7fcdfc0aa6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:07.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.661+0000 7fcdfbe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcdfc06f950 0x7fcdfc1eee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:07.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.661+0000 7fcdfbe00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcdfc152680 0x7fcdfc1441b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:07.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.661+0000 7fcdfbe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdfc0724b0 con 0x7fcdfc06b0e0 2024-02-21T23:16:07.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.661+0000 7fcdfbe00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdfc072330 con 0x7fcdfc06f950 2024-02-21T23:16:07.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.661+0000 7fcdfbe00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdfc072630 con 0x7fcdfc152680 2024-02-21T23:16:07.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.661+0000 7fcdfaa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdfc06b0e0 0x7fcdfc0aa6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:07.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.661+0000 7fcdfaa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdfc06b0e0 0x7fcdfc0aa6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36348/0 (socket says 172.21.15.28:36348) 2024-02-21T23:16:07.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.661+0000 7fcdfaa00640 1 -- 172.21.15.28:0/2959868000 learned_addr learned my addr 172.21.15.28:0/2959868000 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:16:07.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.662+0000 7fcdfa000640 1 --2- 172.21.15.28:0/2959868000 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcdfc06f950 0x7fcdfc1eee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:07.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.662+0000 7fcdfb400640 1 --2- 172.21.15.28:0/2959868000 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcdfc152680 0x7fcdfc1441b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:07.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.662+0000 7fcdfaa00640 1 -- 172.21.15.28:0/2959868000 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcdfc152680 msgr2=0x7fcdfc1441b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:07.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.662+0000 7fcdfaa00640 1 --2- 172.21.15.28:0/2959868000 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcdfc152680 0x7fcdfc1441b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:07.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.662+0000 7fcdfaa00640 1 -- 172.21.15.28:0/2959868000 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcdfc06f950 msgr2=0x7fcdfc1eee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:07.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.662+0000 7fcdfaa00640 1 --2- 172.21.15.28:0/2959868000 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcdfc06f950 0x7fcdfc1eee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:07.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.662+0000 7fcdfaa00640 1 -- 172.21.15.28:0/2959868000 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdfc146b50 con 0x7fcdfc06b0e0 2024-02-21T23:16:07.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.662+0000 7fcdfb400640 1 --2- 172.21.15.28:0/2959868000 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcdfc152680 0x7fcdfc1441b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:16:07.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.662+0000 7fcdfaa00640 1 --2- 172.21.15.28:0/2959868000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdfc06b0e0 0x7fcdfc0aa6c0 secure :-1 s=READY pgs=2605 cs=0 l=1 rev1=1 crypto rx=0x7fcdec00e870 tx=0x7fcdec00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:07.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.663+0000 7fcdf2a00640 1 -- 172.21.15.28:0/2959868000 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdec017070 con 0x7fcdfc06b0e0 2024-02-21T23:16:07.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.663+0000 7fcdf2a00640 1 -- 172.21.15.28:0/2959868000 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdec00c940 con 0x7fcdfc06b0e0 2024-02-21T23:16:07.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.663+0000 7fcdf2a00640 1 -- 172.21.15.28:0/2959868000 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdec00cd30 con 0x7fcdfc06b0e0 2024-02-21T23:16:07.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.663+0000 7fcdfbe00640 1 -- 172.21.15.28:0/2959868000 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdfc147a00 con 0x7fcdfc06b0e0 2024-02-21T23:16:07.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.663+0000 7fcdfbe00640 1 -- 172.21.15.28:0/2959868000 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdfc1eb0c0 con 0x7fcdfc06b0e0 2024-02-21T23:16:07.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.664+0000 7fcdfbe00640 1 -- 172.21.15.28:0/2959868000 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdfc1eaeb0 con 0x7fcdfc06b0e0 2024-02-21T23:16:07.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.665+0000 7fcdf2a00640 1 -- 172.21.15.28:0/2959868000 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcdec01a070 con 0x7fcdfc06b0e0 2024-02-21T23:16:07.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.666+0000 7fcdf2a00640 1 --2- 172.21.15.28:0/2959868000 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcddc06b4d0 0x7fcddc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:07.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.667+0000 7fcdf2a00640 1 -- 172.21.15.28:0/2959868000 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fcdec08a360 con 0x7fcdfc06b0e0 2024-02-21T23:16:07.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.667+0000 7fcdfa000640 1 --2- 172.21.15.28:0/2959868000 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcddc06b4d0 0x7fcddc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:07.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.668+0000 7fcdfa000640 1 --2- 172.21.15.28:0/2959868000 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcddc06b4d0 0x7fcddc06d990 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fcde8007fd0 tx=0x7fcde8007c40 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:07.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.670+0000 7fcdf2a00640 1 -- 172.21.15.28:0/2959868000 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fcdec058f40 con 0x7fcdfc06b0e0 2024-02-21T23:16:07.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.915+0000 7fcdfbe00640 1 -- 172.21.15.28:0/2959868000 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcdfc04f290 con 0x7fcdfc06b0e0 2024-02-21T23:16:07.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.916+0000 7fcdf2a00640 1 -- 172.21.15.28:0/2959868000 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7fcdec058840 con 0x7fcdfc06b0e0 2024-02-21T23:16:07.903 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:16:07.903 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:16:07.903 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:16:07.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.918+0000 7fcdfbe00640 1 -- 172.21.15.28:0/2959868000 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcddc06b4d0 msgr2=0x7fcddc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:07.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.918+0000 7fcdfbe00640 1 --2- 172.21.15.28:0/2959868000 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcddc06b4d0 0x7fcddc06d990 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fcde8007fd0 tx=0x7fcde8007c40 comp rx=0 tx=0).stop 2024-02-21T23:16:07.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.918+0000 7fcdfbe00640 1 -- 172.21.15.28:0/2959868000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdfc06b0e0 msgr2=0x7fcdfc0aa6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:07.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.918+0000 7fcdfbe00640 1 --2- 172.21.15.28:0/2959868000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdfc06b0e0 0x7fcdfc0aa6c0 secure :-1 s=READY pgs=2605 cs=0 l=1 rev1=1 crypto rx=0x7fcdec00e870 tx=0x7fcdec00ed30 comp rx=0 tx=0).stop 2024-02-21T23:16:07.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.918+0000 7fcdfbe00640 1 -- 172.21.15.28:0/2959868000 shutdown_connections 2024-02-21T23:16:07.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.918+0000 7fcdfbe00640 1 --2- 172.21.15.28:0/2959868000 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcddc06b4d0 0x7fcddc06d990 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:07.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.918+0000 7fcdfbe00640 1 --2- 172.21.15.28:0/2959868000 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcdfc152680 0x7fcdfc1441b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:07.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.918+0000 7fcdfbe00640 1 --2- 172.21.15.28:0/2959868000 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcdfc06f950 0x7fcdfc1eee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:07.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.918+0000 7fcdfbe00640 1 --2- 172.21.15.28:0/2959868000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdfc06b0e0 0x7fcdfc0aa6c0 unknown :-1 s=CLOSED pgs=2605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:07.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.918+0000 7fcdfbe00640 1 -- 172.21.15.28:0/2959868000 >> 172.21.15.28:0/2959868000 conn(0x7fcdfc06e8c0 msgr2=0x7fcdfc0c0fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:16:07.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.919+0000 7fcdfbe00640 1 -- 172.21.15.28:0/2959868000 shutdown_connections 2024-02-21T23:16:07.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:07.919+0000 7fcdfbe00640 1 -- 172.21.15.28:0/2959868000 wait complete. 2024-02-21T23:16:08.329 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:16:08.330 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:16:08.937 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-21T23:16:08.946 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-21T23:16:09.331 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:16:09.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.558+0000 7f9f45800640 1 Processor -- start 2024-02-21T23:16:09.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.559+0000 7f9f45800640 1 -- start start 2024-02-21T23:16:09.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.560+0000 7f9f45800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f9f40151f60 0x7f9f400c11f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:09.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.560+0000 7f9f45800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9f40072ca0 0x7f9f4015bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:09.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.560+0000 7f9f45800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f4006f230 con 0x7f9f400c1730 2024-02-21T23:16:09.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.560+0000 7f9f45800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f40071a60 con 0x7f9f40151f60 2024-02-21T23:16:09.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.560+0000 7f9f45800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f40071c90 con 0x7f9f40072ca0 2024-02-21T23:16:09.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.561+0000 7f9f3ea00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f9f40151f60 0x7f9f400c11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:09.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.561+0000 7f9f3f400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9f40072ca0 0x7f9f4015bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:09.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.561+0000 7f9f3f400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9f40072ca0 0x7f9f4015bd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3301/0 says I am v2:172.21.15.28:47276/0 (socket says 172.21.15.28:47276) 2024-02-21T23:16:09.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.561+0000 7f9f3f400640 1 -- 172.21.15.28:0/3990593622 learned_addr learned my addr 172.21.15.28:0/3990593622 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:16:09.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.562+0000 7f9f3f400640 1 -- 172.21.15.28:0/3990593622 >> v2:172.21.15.81:3300/0 conn(0x7f9f40151f60 msgr2=0x7f9f400c11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:16:09.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.562+0000 7f9f3f400640 1 --2- 172.21.15.28:0/3990593622 >> v2:172.21.15.81:3300/0 conn(0x7f9f40151f60 0x7f9f400c11f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:09.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.562+0000 7f9f3f400640 1 -- 172.21.15.28:0/3990593622 >> v1:172.21.15.81:6789/0 conn(0x7f9f400c1730 legacy=0x7f9f4006dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:09.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.562+0000 7f9f3f400640 1 -- 172.21.15.28:0/3990593622 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f40071f70 con 0x7f9f40072ca0 2024-02-21T23:16:09.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.563+0000 7f9f3f400640 1 --2- 172.21.15.28:0/3990593622 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9f40072ca0 0x7f9f4015bd50 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f9f38009e50 tx=0x7f9f3802bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20e7b1d292f8c366 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:09.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.564+0000 7f9f3d600640 1 -- 172.21.15.28:0/3990593622 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f3802cb00 con 0x7f9f40072ca0 2024-02-21T23:16:09.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.564+0000 7f9f3d600640 1 -- 172.21.15.28:0/3990593622 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f3802cca0 con 0x7f9f40072ca0 2024-02-21T23:16:09.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.564+0000 7f9f3d600640 1 -- 172.21.15.28:0/3990593622 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f38034610 con 0x7f9f40072ca0 2024-02-21T23:16:09.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.564+0000 7f9f45800640 1 -- 172.21.15.28:0/3990593622 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9f40072ca0 msgr2=0x7f9f4015bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:09.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.564+0000 7f9f45800640 1 --2- 172.21.15.28:0/3990593622 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9f40072ca0 0x7f9f4015bd50 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f9f38009e50 tx=0x7f9f3802bf60 comp rx=0 tx=0).stop 2024-02-21T23:16:09.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.564+0000 7f9f45800640 1 -- 172.21.15.28:0/3990593622 shutdown_connections 2024-02-21T23:16:09.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.564+0000 7f9f45800640 1 --2- 172.21.15.28:0/3990593622 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9f40072ca0 0x7f9f4015bd50 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:09.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.564+0000 7f9f45800640 1 --2- 172.21.15.28:0/3990593622 >> v2:172.21.15.81:3300/0 conn(0x7f9f40151f60 0x7f9f400c11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:09.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.564+0000 7f9f45800640 1 -- 172.21.15.28:0/3990593622 >> 172.21.15.28:0/3990593622 conn(0x7f9f400c67f0 msgr2=0x7f9f400c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:16:09.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.564+0000 7f9f45800640 1 -- 172.21.15.28:0/3990593622 shutdown_connections 2024-02-21T23:16:09.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.564+0000 7f9f45800640 1 -- 172.21.15.28:0/3990593622 wait complete. 2024-02-21T23:16:09.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.566+0000 7f9f45800640 1 Processor -- start 2024-02-21T23:16:09.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.566+0000 7f9f45800640 1 -- start start 2024-02-21T23:16:09.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.567+0000 7f9f45800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f40072ca0 0x7f9f401ee940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:09.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.567+0000 7f9f45800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f400c1730 0x7f9f400a83e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:09.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.567+0000 7f9f45800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9f40151f60 0x7f9f4014c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:09.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.567+0000 7f9f45800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f400a6090 con 0x7f9f400c1730 2024-02-21T23:16:09.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.567+0000 7f9f45800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f40152340 con 0x7f9f40072ca0 2024-02-21T23:16:09.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.567+0000 7f9f45800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f400a6210 con 0x7f9f40151f60 2024-02-21T23:16:09.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.567+0000 7f9f3e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f400c1730 0x7f9f400a83e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:09.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.568+0000 7f9f3e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f400c1730 0x7f9f400a83e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36364/0 (socket says 172.21.15.28:36364) 2024-02-21T23:16:09.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.568+0000 7f9f3e000640 1 -- 172.21.15.28:0/2505415069 learned_addr learned my addr 172.21.15.28:0/2505415069 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:16:09.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.568+0000 7f9f3ea00640 1 --2- 172.21.15.28:0/2505415069 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f40072ca0 0x7f9f401ee940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:09.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.568+0000 7f9f3f400640 1 --2- 172.21.15.28:0/2505415069 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9f40151f60 0x7f9f4014c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:09.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.568+0000 7f9f3e000640 1 -- 172.21.15.28:0/2505415069 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9f40151f60 msgr2=0x7f9f4014c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:09.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.568+0000 7f9f3e000640 1 --2- 172.21.15.28:0/2505415069 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9f40151f60 0x7f9f4014c290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:09.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.568+0000 7f9f3e000640 1 -- 172.21.15.28:0/2505415069 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f40072ca0 msgr2=0x7f9f401ee940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:09.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.568+0000 7f9f3e000640 1 --2- 172.21.15.28:0/2505415069 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f40072ca0 0x7f9f401ee940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:09.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.568+0000 7f9f3e000640 1 -- 172.21.15.28:0/2505415069 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f400a8d60 con 0x7f9f400c1730 2024-02-21T23:16:09.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.568+0000 7f9f3ea00640 1 --2- 172.21.15.28:0/2505415069 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f40072ca0 0x7f9f401ee940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:16:09.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.568+0000 7f9f3e000640 1 --2- 172.21.15.28:0/2505415069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f400c1730 0x7f9f400a83e0 secure :-1 s=READY pgs=2606 cs=0 l=1 rev1=1 crypto rx=0x7f9f2c00c470 tx=0x7f9f2c001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:09.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.569+0000 7f9f36a00640 1 -- 172.21.15.28:0/2505415069 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f2c0052d0 con 0x7f9f400c1730 2024-02-21T23:16:09.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.569+0000 7f9f36a00640 1 -- 172.21.15.28:0/2505415069 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f2c005470 con 0x7f9f400c1730 2024-02-21T23:16:09.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.569+0000 7f9f36a00640 1 -- 172.21.15.28:0/2505415069 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f2c007260 con 0x7f9f400c1730 2024-02-21T23:16:09.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.569+0000 7f9f45800640 1 -- 172.21.15.28:0/2505415069 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f400a9040 con 0x7f9f400c1730 2024-02-21T23:16:09.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.569+0000 7f9f45800640 1 -- 172.21.15.28:0/2505415069 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f4014d1f0 con 0x7f9f400c1730 2024-02-21T23:16:09.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.570+0000 7f9f45800640 1 -- 172.21.15.28:0/2505415069 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f40067f30 con 0x7f9f400c1730 2024-02-21T23:16:09.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.571+0000 7f9f36a00640 1 -- 172.21.15.28:0/2505415069 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9f2c007400 con 0x7f9f400c1730 2024-02-21T23:16:09.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.572+0000 7f9f36a00640 1 --2- 172.21.15.28:0/2505415069 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9f2006b4d0 0x7f9f2006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:09.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.572+0000 7f9f36a00640 1 -- 172.21.15.28:0/2505415069 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f9f2c0899b0 con 0x7f9f400c1730 2024-02-21T23:16:09.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.573+0000 7f9f3ea00640 1 --2- 172.21.15.28:0/2505415069 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9f2006b4d0 0x7f9f2006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:09.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.574+0000 7f9f3ea00640 1 --2- 172.21.15.28:0/2505415069 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9f2006b4d0 0x7f9f2006d990 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f9f3000b240 tx=0x7f9f3000b3a0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:09.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.576+0000 7f9f36a00640 1 -- 172.21.15.28:0/2505415069 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f9f2c058590 con 0x7f9f400c1730 2024-02-21T23:16:09.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.819+0000 7f9f45800640 1 -- 172.21.15.28:0/2505415069 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9f400a9210 con 0x7f9f400c1730 2024-02-21T23:16:09.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.819+0000 7f9f36a00640 1 -- 172.21.15.28:0/2505415069 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7f9f2c057e90 con 0x7f9f400c1730 2024-02-21T23:16:09.807 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:16:09.807 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:16:09.807 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:16:09.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.822+0000 7f9f45800640 1 -- 172.21.15.28:0/2505415069 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9f2006b4d0 msgr2=0x7f9f2006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:09.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.822+0000 7f9f45800640 1 --2- 172.21.15.28:0/2505415069 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9f2006b4d0 0x7f9f2006d990 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f9f3000b240 tx=0x7f9f3000b3a0 comp rx=0 tx=0).stop 2024-02-21T23:16:09.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.822+0000 7f9f45800640 1 -- 172.21.15.28:0/2505415069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f400c1730 msgr2=0x7f9f400a83e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:09.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.822+0000 7f9f45800640 1 --2- 172.21.15.28:0/2505415069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f400c1730 0x7f9f400a83e0 secure :-1 s=READY pgs=2606 cs=0 l=1 rev1=1 crypto rx=0x7f9f2c00c470 tx=0x7f9f2c001e70 comp rx=0 tx=0).stop 2024-02-21T23:16:09.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.822+0000 7f9f45800640 1 -- 172.21.15.28:0/2505415069 shutdown_connections 2024-02-21T23:16:09.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.822+0000 7f9f45800640 1 --2- 172.21.15.28:0/2505415069 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9f2006b4d0 0x7f9f2006d990 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:09.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.822+0000 7f9f45800640 1 --2- 172.21.15.28:0/2505415069 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9f40151f60 0x7f9f4014c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:09.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.822+0000 7f9f45800640 1 --2- 172.21.15.28:0/2505415069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f400c1730 0x7f9f400a83e0 unknown :-1 s=CLOSED pgs=2606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:09.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.822+0000 7f9f45800640 1 --2- 172.21.15.28:0/2505415069 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f40072ca0 0x7f9f401ee940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:09.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.822+0000 7f9f45800640 1 -- 172.21.15.28:0/2505415069 >> 172.21.15.28:0/2505415069 conn(0x7f9f400c67f0 msgr2=0x7f9f400c3b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:16:09.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.822+0000 7f9f45800640 1 -- 172.21.15.28:0/2505415069 shutdown_connections 2024-02-21T23:16:09.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:09.822+0000 7f9f45800640 1 -- 172.21.15.28:0/2505415069 wait complete. 2024-02-21T23:16:10.230 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:16:10.230 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:16:11.231 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:16:11.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.469+0000 7f7142800640 1 Processor -- start 2024-02-21T23:16:11.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.469+0000 7f7142800640 1 -- start start 2024-02-21T23:16:11.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.471+0000 7f7142800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f713c153950 0x7f713c0c15d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:11.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.471+0000 7f7142800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f713c15b6d0 0x7f713c15dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:11.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.471+0000 7f7142800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f713c152e70 con 0x7f713c0c11f0 2024-02-21T23:16:11.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.471+0000 7f7142800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f713c0a5ec0 con 0x7f713c15b6d0 2024-02-21T23:16:11.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.471+0000 7f7142800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f713c153150 con 0x7f713c153950 2024-02-21T23:16:11.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.471+0000 7f713be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f713c15b6d0 0x7f713c15dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:11.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.471+0000 7f713be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f713c15b6d0 0x7f713c15dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36366/0 (socket says 172.21.15.28:36366) 2024-02-21T23:16:11.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.471+0000 7f713be00640 1 -- 172.21.15.28:0/1403070071 learned_addr learned my addr 172.21.15.28:0/1403070071 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:16:11.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.471+0000 7f7132a00640 1 --2- 172.21.15.28:0/1403070071 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f713c153950 0x7f713c0c15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:11.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.472+0000 7f713be00640 1 -- 172.21.15.28:0/1403070071 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f713c153950 msgr2=0x7f713c0c15d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:16:11.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.472+0000 7f713be00640 1 --2- 172.21.15.28:0/1403070071 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f713c153950 0x7f713c0c15d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:11.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.472+0000 7f713be00640 1 -- 172.21.15.28:0/1403070071 >> v1:172.21.15.81:6789/0 conn(0x7f713c0c11f0 legacy=0x7f713c06dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:16:11.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.472+0000 7f713be00640 1 -- 172.21.15.28:0/1403070071 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f713c153430 con 0x7f713c15b6d0 2024-02-21T23:16:11.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.473+0000 7f713be00640 1 --2- 172.21.15.28:0/1403070071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f713c15b6d0 0x7f713c15dac0 secure :-1 s=READY pgs=2607 cs=0 l=1 rev1=1 crypto rx=0x7f7134009e50 tx=0x7f713402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54521511608368a6 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:11.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.473+0000 7f713aa00640 1 -- 172.21.15.28:0/1403070071 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f713402c9e0 con 0x7f713c15b6d0 2024-02-21T23:16:11.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.473+0000 7f713aa00640 1 -- 172.21.15.28:0/1403070071 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f713402cb80 con 0x7f713c15b6d0 2024-02-21T23:16:11.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.473+0000 7f713aa00640 1 -- 172.21.15.28:0/1403070071 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71340343f0 con 0x7f713c15b6d0 2024-02-21T23:16:11.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.474+0000 7f7142800640 1 -- 172.21.15.28:0/1403070071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f713c15b6d0 msgr2=0x7f713c15dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:11.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.474+0000 7f7142800640 1 --2- 172.21.15.28:0/1403070071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f713c15b6d0 0x7f713c15dac0 secure :-1 s=READY pgs=2607 cs=0 l=1 rev1=1 crypto rx=0x7f7134009e50 tx=0x7f713402be40 comp rx=0 tx=0).stop 2024-02-21T23:16:11.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.474+0000 7f7142800640 1 -- 172.21.15.28:0/1403070071 shutdown_connections 2024-02-21T23:16:11.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.474+0000 7f7142800640 1 --2- 172.21.15.28:0/1403070071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f713c15b6d0 0x7f713c15dac0 unknown :-1 s=CLOSED pgs=2607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:11.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.474+0000 7f7142800640 1 --2- 172.21.15.28:0/1403070071 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f713c153950 0x7f713c0c15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:11.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.474+0000 7f7142800640 1 -- 172.21.15.28:0/1403070071 >> 172.21.15.28:0/1403070071 conn(0x7f713c0c67f0 msgr2=0x7f713c0c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:16:11.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.474+0000 7f7142800640 1 -- 172.21.15.28:0/1403070071 shutdown_connections 2024-02-21T23:16:11.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.474+0000 7f7142800640 1 -- 172.21.15.28:0/1403070071 wait complete. 2024-02-21T23:16:11.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.476+0000 7f7142800640 1 Processor -- start 2024-02-21T23:16:11.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.476+0000 7f7142800640 1 -- start start 2024-02-21T23:16:11.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.477+0000 7f7142800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f713c0c11f0 0x7f713c142740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:11.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.477+0000 7f7142800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f713c153950 0x7f713c143c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:11.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.477+0000 7f7142800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f713c15b6d0 0x7f713c1ebfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:11.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.477+0000 7f7142800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f713c0a6110 con 0x7f713c0c11f0 2024-02-21T23:16:11.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.477+0000 7f7142800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f713c0c4c00 con 0x7f713c15b6d0 2024-02-21T23:16:11.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.477+0000 7f7142800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f713c0c0d90 con 0x7f713c153950 2024-02-21T23:16:11.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.478+0000 7f713b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f713c0c11f0 0x7f713c142740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:11.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.478+0000 7f713b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f713c0c11f0 0x7f713c142740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36370/0 (socket says 172.21.15.28:36370) 2024-02-21T23:16:11.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.478+0000 7f713b400640 1 -- 172.21.15.28:0/1655550744 learned_addr learned my addr 172.21.15.28:0/1655550744 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:16:11.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.478+0000 7f7132a00640 1 --2- 172.21.15.28:0/1655550744 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f713c153950 0x7f713c143c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:11.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.478+0000 7f713be00640 1 --2- 172.21.15.28:0/1655550744 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f713c15b6d0 0x7f713c1ebfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:11.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.478+0000 7f713b400640 1 -- 172.21.15.28:0/1655550744 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f713c153950 msgr2=0x7f713c143c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:11.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.478+0000 7f713b400640 1 --2- 172.21.15.28:0/1655550744 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f713c153950 0x7f713c143c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:11.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.478+0000 7f713b400640 1 -- 172.21.15.28:0/1655550744 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f713c15b6d0 msgr2=0x7f713c1ebfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:11.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.478+0000 7f713b400640 1 --2- 172.21.15.28:0/1655550744 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f713c15b6d0 0x7f713c1ebfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:11.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.478+0000 7f713b400640 1 -- 172.21.15.28:0/1655550744 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f713c0a7020 con 0x7f713c0c11f0 2024-02-21T23:16:11.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.478+0000 7f7132a00640 1 --2- 172.21.15.28:0/1655550744 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f713c153950 0x7f713c143c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:16:11.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.479+0000 7f713b400640 1 --2- 172.21.15.28:0/1655550744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f713c0c11f0 0x7f713c142740 secure :-1 s=READY pgs=2608 cs=0 l=1 rev1=1 crypto rx=0x7f712c00b420 tx=0x7f712c00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:11.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.479+0000 7f7133e00640 1 -- 172.21.15.28:0/1655550744 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f712c011830 con 0x7f713c0c11f0 2024-02-21T23:16:11.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.479+0000 7f7133e00640 1 -- 172.21.15.28:0/1655550744 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f712c016070 con 0x7f713c0c11f0 2024-02-21T23:16:11.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.479+0000 7f7133e00640 1 -- 172.21.15.28:0/1655550744 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f712c011c10 con 0x7f713c0c11f0 2024-02-21T23:16:11.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.479+0000 7f7142800640 1 -- 172.21.15.28:0/1655550744 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f713c144500 con 0x7f713c0c11f0 2024-02-21T23:16:11.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.479+0000 7f7142800640 1 -- 172.21.15.28:0/1655550744 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f713c1ecb70 con 0x7f713c0c11f0 2024-02-21T23:16:11.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.480+0000 7f7142800640 1 -- 172.21.15.28:0/1655550744 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f713c067f30 con 0x7f713c0c11f0 2024-02-21T23:16:11.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.482+0000 7f7133e00640 1 -- 172.21.15.28:0/1655550744 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f712c01b460 con 0x7f713c0c11f0 2024-02-21T23:16:11.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.483+0000 7f7133e00640 1 --2- 172.21.15.28:0/1655550744 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f711806b4d0 0x7f711806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:11.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.483+0000 7f7133e00640 1 -- 172.21.15.28:0/1655550744 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f712c089ff0 con 0x7f713c0c11f0 2024-02-21T23:16:11.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.484+0000 7f7132a00640 1 --2- 172.21.15.28:0/1655550744 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f711806b4d0 0x7f711806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:11.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.485+0000 7f7132a00640 1 --2- 172.21.15.28:0/1655550744 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f711806b4d0 0x7f711806d990 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f713c144dd0 tx=0x7f7124008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:11.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.487+0000 7f7133e00640 1 -- 172.21.15.28:0/1655550744 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f712c020020 con 0x7f713c0c11f0 2024-02-21T23:16:11.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.727+0000 7f7142800640 1 -- 172.21.15.28:0/1655550744 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f713c0a71f0 con 0x7f713c0c11f0 2024-02-21T23:16:11.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.728+0000 7f7133e00640 1 -- 172.21.15.28:0/1655550744 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7f712c058ba0 con 0x7f713c0c11f0 2024-02-21T23:16:11.715 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:16:11.715 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:16:11.715 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:16:11.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.730+0000 7f7142800640 1 -- 172.21.15.28:0/1655550744 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f711806b4d0 msgr2=0x7f711806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:11.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.730+0000 7f7142800640 1 --2- 172.21.15.28:0/1655550744 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f711806b4d0 0x7f711806d990 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f713c144dd0 tx=0x7f7124008040 comp rx=0 tx=0).stop 2024-02-21T23:16:11.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.730+0000 7f7142800640 1 -- 172.21.15.28:0/1655550744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f713c0c11f0 msgr2=0x7f713c142740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:11.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.730+0000 7f7142800640 1 --2- 172.21.15.28:0/1655550744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f713c0c11f0 0x7f713c142740 secure :-1 s=READY pgs=2608 cs=0 l=1 rev1=1 crypto rx=0x7f712c00b420 tx=0x7f712c00b8e0 comp rx=0 tx=0).stop 2024-02-21T23:16:11.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.731+0000 7f7142800640 1 -- 172.21.15.28:0/1655550744 shutdown_connections 2024-02-21T23:16:11.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.731+0000 7f7142800640 1 --2- 172.21.15.28:0/1655550744 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f711806b4d0 0x7f711806d990 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:11.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.731+0000 7f7142800640 1 --2- 172.21.15.28:0/1655550744 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f713c153950 0x7f713c143c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:11.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.731+0000 7f7142800640 1 --2- 172.21.15.28:0/1655550744 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f713c15b6d0 0x7f713c1ebfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:11.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.731+0000 7f7142800640 1 --2- 172.21.15.28:0/1655550744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f713c0c11f0 0x7f713c142740 unknown :-1 s=CLOSED pgs=2608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:11.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.731+0000 7f7142800640 1 -- 172.21.15.28:0/1655550744 >> 172.21.15.28:0/1655550744 conn(0x7f713c0c67f0 msgr2=0x7f713c0c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:16:11.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.731+0000 7f7142800640 1 -- 172.21.15.28:0/1655550744 shutdown_connections 2024-02-21T23:16:11.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:11.731+0000 7f7142800640 1 -- 172.21.15.28:0/1655550744 wait complete. 2024-02-21T23:16:12.139 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:16:12.139 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:16:13.141 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:16:13.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.386+0000 7f313f200640 1 Processor -- start 2024-02-21T23:16:13.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.386+0000 7f313f200640 1 -- start start 2024-02-21T23:16:13.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.387+0000 7f313f200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f31381526b0 0x7f31381559c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:13.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.387+0000 7f313f200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f31380ca940 0x7f31380cadc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:13.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.387+0000 7f313f200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31380c6f30 con 0x7f31380c76f0 2024-02-21T23:16:13.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.387+0000 7f313f200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31380c61b0 con 0x7f31381526b0 2024-02-21T23:16:13.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.387+0000 7f313f200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31380c6450 con 0x7f31380ca940 2024-02-21T23:16:13.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.388+0000 7f313d200640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f31380c76f0 0x7f31380c7b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34938/0 (socket says 172.21.15.28:34938) 2024-02-21T23:16:13.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.388+0000 7f313d200640 1 -- 172.21.15.28:0/3429214473 learned_addr learned my addr 172.21.15.28:0/3429214473 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:16:13.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.388+0000 7f312f400640 1 --2- 172.21.15.28:0/3429214473 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f31380ca940 0x7f31380cadc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:13.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.388+0000 7f3137e00640 1 --2- 172.21.15.28:0/3429214473 >> v2:172.21.15.81:3300/0 conn(0x7f31381526b0 0x7f31381559c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:13.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.388+0000 7f3137400640 1 -- 172.21.15.28:0/3429214473 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2809149298 0 0) 0x7f31380c6f30 con 0x7f31380c76f0 2024-02-21T23:16:13.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.388+0000 7f3137400640 1 -- 172.21.15.28:0/3429214473 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f311c003520 con 0x7f31380c76f0 2024-02-21T23:16:13.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.389+0000 7f3137400640 1 -- 172.21.15.28:0/3429214473 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2589671994 0 0) 0x7f311c003520 con 0x7f31380c76f0 2024-02-21T23:16:13.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.389+0000 7f3137400640 1 -- 172.21.15.28:0/3429214473 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f31380ca940 msgr2=0x7f31380cadc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:16:13.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.389+0000 7f3137400640 1 --2- 172.21.15.28:0/3429214473 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f31380ca940 0x7f31380cadc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:13.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.389+0000 7f3137400640 1 -- 172.21.15.28:0/3429214473 >> v2:172.21.15.81:3300/0 conn(0x7f31381526b0 msgr2=0x7f31381559c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:16:13.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.389+0000 7f3137400640 1 --2- 172.21.15.28:0/3429214473 >> v2:172.21.15.81:3300/0 conn(0x7f31381526b0 0x7f31381559c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:13.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.389+0000 7f3137400640 1 -- 172.21.15.28:0/3429214473 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31380c8290 con 0x7f31380c76f0 2024-02-21T23:16:13.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.389+0000 7f3137400640 1 -- 172.21.15.28:0/3429214473 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f3130001670 con 0x7f31380c76f0 2024-02-21T23:16:13.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.389+0000 7f3137400640 1 -- 172.21.15.28:0/3429214473 >> v1:172.21.15.28:6789/0 conn(0x7f31380c76f0 legacy=0x7f31380c7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:13.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.389+0000 7f3137400640 1 --2- 172.21.15.28:0/3429214473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f311c0039b0 0x7f311c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:13.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.389+0000 7f3137400640 1 --2- 172.21.15.28:0/3429214473 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f31380ca940 0x7f311c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:13.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.389+0000 7f3137400640 1 --2- 172.21.15.28:0/3429214473 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f311c004d40 0x7f311c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:13.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.389+0000 7f3137400640 1 -- 172.21.15.28:0/3429214473 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31380c6450 con 0x7f311c0039b0 2024-02-21T23:16:13.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.389+0000 7f3137400640 1 -- 172.21.15.28:0/3429214473 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f311c00b9c0 con 0x7f311c004d40 2024-02-21T23:16:13.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.389+0000 7f3137400640 1 -- 172.21.15.28:0/3429214473 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f311c00bb40 con 0x7f31380ca940 2024-02-21T23:16:13.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.389+0000 7f3137e00640 1 --2- 172.21.15.28:0/3429214473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f311c0039b0 0x7f311c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:13.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.389+0000 7f312f400640 1 --2- 172.21.15.28:0/3429214473 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f31380ca940 0x7f311c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:13.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.390+0000 7f3137e00640 1 -- 172.21.15.28:0/3429214473 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f31380ca940 msgr2=0x7f311c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:16:13.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.390+0000 7f3137e00640 1 --2- 172.21.15.28:0/3429214473 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f31380ca940 0x7f311c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:13.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.390+0000 7f3137e00640 1 -- 172.21.15.28:0/3429214473 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f311c004d40 msgr2=0x7f311c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:16:13.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.390+0000 7f3137e00640 1 --2- 172.21.15.28:0/3429214473 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f311c004d40 0x7f311c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:13.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.390+0000 7f3137e00640 1 -- 172.21.15.28:0/3429214473 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f311c003520 con 0x7f311c0039b0 2024-02-21T23:16:13.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.390+0000 7f312f400640 1 --2- 172.21.15.28:0/3429214473 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f31380ca940 0x7f311c004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:16:13.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.391+0000 7f3137e00640 1 --2- 172.21.15.28:0/3429214473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f311c0039b0 0x7f311c003dd0 secure :-1 s=READY pgs=2610 cs=0 l=1 rev1=1 crypto rx=0x7f3128009ac0 tx=0x7f311c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f5ccf5b4ef18dd6 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:13.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.391+0000 7f3137400640 1 -- 172.21.15.28:0/3429214473 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3128039070 con 0x7f311c0039b0 2024-02-21T23:16:13.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.391+0000 7f3137400640 1 -- 172.21.15.28:0/3429214473 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f312802c710 con 0x7f311c0039b0 2024-02-21T23:16:13.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.391+0000 7f3137400640 1 -- 172.21.15.28:0/3429214473 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3128009690 con 0x7f311c0039b0 2024-02-21T23:16:13.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.392+0000 7f313f200640 1 -- 172.21.15.28:0/3429214473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f311c0039b0 msgr2=0x7f311c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:13.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.392+0000 7f313f200640 1 --2- 172.21.15.28:0/3429214473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f311c0039b0 0x7f311c003dd0 secure :-1 s=READY pgs=2610 cs=0 l=1 rev1=1 crypto rx=0x7f3128009ac0 tx=0x7f311c005830 comp rx=0 tx=0).stop 2024-02-21T23:16:13.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.392+0000 7f313d200640 1 -- 172.21.15.28:0/3429214473 reap_dead start 2024-02-21T23:16:13.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.392+0000 7f313f200640 1 -- 172.21.15.28:0/3429214473 shutdown_connections 2024-02-21T23:16:13.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.392+0000 7f313f200640 1 -- 172.21.15.28:0/3429214473 >> 172.21.15.28:0/3429214473 conn(0x7f313806d3d0 msgr2=0x7f313808afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:16:13.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.392+0000 7f313f200640 1 -- 172.21.15.28:0/3429214473 shutdown_connections 2024-02-21T23:16:13.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.392+0000 7f313f200640 1 -- 172.21.15.28:0/3429214473 wait complete. 2024-02-21T23:16:13.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.393+0000 7f313f200640 1 Processor -- start 2024-02-21T23:16:13.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.394+0000 7f313f200640 1 -- start start 2024-02-21T23:16:13.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.394+0000 7f313f200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f311c0039b0 0x7f313814f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:13.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.394+0000 7f313f200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31380c8560 0x7f31380ca040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:13.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.395+0000 7f313f200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f313814e340 0x7f313814e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:13.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.395+0000 7f313f200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31380c71d0 con 0x7f31380c8560 2024-02-21T23:16:13.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.395+0000 7f313f200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31380c69f0 con 0x7f313814e340 2024-02-21T23:16:13.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.395+0000 7f313f200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31380c5c90 con 0x7f311c0039b0 2024-02-21T23:16:13.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.395+0000 7f312f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31380c8560 0x7f31380ca040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:13.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.395+0000 7f312f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31380c8560 0x7f31380ca040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36398/0 (socket says 172.21.15.28:36398) 2024-02-21T23:16:13.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.395+0000 7f312f400640 1 -- 172.21.15.28:0/1169663688 learned_addr learned my addr 172.21.15.28:0/1169663688 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:16:13.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.395+0000 7f3137e00640 1 --2- 172.21.15.28:0/1169663688 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f311c0039b0 0x7f313814f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:13.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.395+0000 7f312f400640 1 -- 172.21.15.28:0/1169663688 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f311c0039b0 msgr2=0x7f313814f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:13.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.395+0000 7f313d200640 1 --2- 172.21.15.28:0/1169663688 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f313814e340 0x7f313814e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:13.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.395+0000 7f312f400640 1 --2- 172.21.15.28:0/1169663688 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f311c0039b0 0x7f313814f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:13.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.395+0000 7f312f400640 1 -- 172.21.15.28:0/1169663688 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f313814e340 msgr2=0x7f313814e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:13.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.395+0000 7f312f400640 1 --2- 172.21.15.28:0/1169663688 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f313814e340 0x7f313814e7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:13.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.395+0000 7f312f400640 1 -- 172.21.15.28:0/1169663688 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f313814ff20 con 0x7f31380c8560 2024-02-21T23:16:13.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.395+0000 7f3137e00640 1 --2- 172.21.15.28:0/1169663688 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f311c0039b0 0x7f313814f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:16:13.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.395+0000 7f313d200640 1 --2- 172.21.15.28:0/1169663688 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f313814e340 0x7f313814e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:16:13.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.396+0000 7f312f400640 1 --2- 172.21.15.28:0/1169663688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31380c8560 0x7f31380ca040 secure :-1 s=READY pgs=2611 cs=0 l=1 rev1=1 crypto rx=0x7f31200101c0 tx=0x7f3120010680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:13.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.396+0000 7f3134c00640 1 -- 172.21.15.28:0/1169663688 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f312001a030 con 0x7f31380c8560 2024-02-21T23:16:13.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.396+0000 7f3134c00640 1 -- 172.21.15.28:0/1169663688 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3120003990 con 0x7f31380c8560 2024-02-21T23:16:13.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.396+0000 7f3134c00640 1 -- 172.21.15.28:0/1169663688 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3120003d60 con 0x7f31380c8560 2024-02-21T23:16:13.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.396+0000 7f313f200640 1 -- 172.21.15.28:0/1169663688 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31381eaf20 con 0x7f31380c8560 2024-02-21T23:16:13.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.396+0000 7f313f200640 1 -- 172.21.15.28:0/1169663688 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31380ca580 con 0x7f31380c8560 2024-02-21T23:16:13.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.398+0000 7f313f200640 1 -- 172.21.15.28:0/1169663688 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f313806ba30 con 0x7f31380c8560 2024-02-21T23:16:13.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.399+0000 7f3134c00640 1 -- 172.21.15.28:0/1169663688 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f312001d070 con 0x7f31380c8560 2024-02-21T23:16:13.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.399+0000 7f3134c00640 1 --2- 172.21.15.28:0/1169663688 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f311406b500 0x7f311406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:13.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.400+0000 7f3134c00640 1 -- 172.21.15.28:0/1169663688 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f312008b3a0 con 0x7f31380c8560 2024-02-21T23:16:13.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.400+0000 7f3137e00640 1 --2- 172.21.15.28:0/1169663688 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f311406b500 0x7f311406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:13.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.401+0000 7f3137e00640 1 --2- 172.21.15.28:0/1169663688 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f311406b500 0x7f311406d9c0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f3128036000 tx=0x7f3128030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:13.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.405+0000 7f3134c00640 1 -- 172.21.15.28:0/1169663688 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f313806ba30 con 0x7f31380c8560 2024-02-21T23:16:13.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.644+0000 7f313f200640 1 -- 172.21.15.28:0/1169663688 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3138067f30 con 0x7f31380c8560 2024-02-21T23:16:13.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.645+0000 7f3134c00640 1 -- 172.21.15.28:0/1169663688 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7f312001de40 con 0x7f31380c8560 2024-02-21T23:16:13.632 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:16:13.632 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:16:13.633 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:16:13.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.648+0000 7f313f200640 1 -- 172.21.15.28:0/1169663688 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f311406b500 msgr2=0x7f311406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:13.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.648+0000 7f313f200640 1 --2- 172.21.15.28:0/1169663688 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f311406b500 0x7f311406d9c0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f3128036000 tx=0x7f3128030000 comp rx=0 tx=0).stop 2024-02-21T23:16:13.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.648+0000 7f313f200640 1 -- 172.21.15.28:0/1169663688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31380c8560 msgr2=0x7f31380ca040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:13.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.648+0000 7f313f200640 1 --2- 172.21.15.28:0/1169663688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31380c8560 0x7f31380ca040 secure :-1 s=READY pgs=2611 cs=0 l=1 rev1=1 crypto rx=0x7f31200101c0 tx=0x7f3120010680 comp rx=0 tx=0).stop 2024-02-21T23:16:13.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.649+0000 7f313f200640 1 -- 172.21.15.28:0/1169663688 shutdown_connections 2024-02-21T23:16:13.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.649+0000 7f313f200640 1 --2- 172.21.15.28:0/1169663688 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f311406b500 0x7f311406d9c0 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:13.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.649+0000 7f313f200640 1 --2- 172.21.15.28:0/1169663688 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f313814e340 0x7f313814e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:13.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.649+0000 7f313f200640 1 --2- 172.21.15.28:0/1169663688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31380c8560 0x7f31380ca040 unknown :-1 s=CLOSED pgs=2611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:13.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.649+0000 7f313f200640 1 --2- 172.21.15.28:0/1169663688 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f311c0039b0 0x7f313814f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:13.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.649+0000 7f313f200640 1 -- 172.21.15.28:0/1169663688 >> 172.21.15.28:0/1169663688 conn(0x7f313806d3d0 msgr2=0x7f3138153430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:16:13.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.649+0000 7f313f200640 1 -- 172.21.15.28:0/1169663688 shutdown_connections 2024-02-21T23:16:13.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:13.649+0000 7f313f200640 1 -- 172.21.15.28:0/1169663688 wait complete. 2024-02-21T23:16:14.058 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:16:14.059 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:16:15.059 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:16:15.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.298+0000 7f7cb7e00640 1 Processor -- start 2024-02-21T23:16:15.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.299+0000 7f7cb7e00640 1 -- start start 2024-02-21T23:16:15.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.300+0000 7f7cb7e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f7cb8150450 0x7f7cb80a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:15.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.300+0000 7f7cb7e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7cb806ee40 0x7f7cb81536f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:15.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.301+0000 7f7cb7e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cb80c2760 con 0x7f7cb8156af0 2024-02-21T23:16:15.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.301+0000 7f7cb7e00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cb80c22c0 con 0x7f7cb8150450 2024-02-21T23:16:15.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.301+0000 7f7cb7e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb8156660 con 0x7f7cb806ee40 2024-02-21T23:16:15.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.301+0000 7f7cb7400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f7cb8156af0 0x7f7cb806dff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50274/0 (socket says 172.21.15.28:50274) 2024-02-21T23:16:15.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.301+0000 7f7cb7400640 1 -- 172.21.15.28:0/3246899579 learned_addr learned my addr 172.21.15.28:0/3246899579 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:16:15.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.301+0000 7f7cb6a00640 1 --2- 172.21.15.28:0/3246899579 >> v2:172.21.15.81:3300/0 conn(0x7f7cb8150450 0x7f7cb80a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:15.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.301+0000 7f7cae000640 1 --2- 172.21.15.28:0/3246899579 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7cb806ee40 0x7f7cb81536f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:15.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.301+0000 7f7cb6000640 1 -- 172.21.15.28:0/3246899579 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1923796838 0 0) 0x7f7cb80c2760 con 0x7f7cb8156af0 2024-02-21T23:16:15.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.301+0000 7f7cb6000640 1 -- 172.21.15.28:0/3246899579 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c9c003520 con 0x7f7cb8156af0 2024-02-21T23:16:15.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.302+0000 7f7cb6000640 1 -- 172.21.15.28:0/3246899579 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1486867360 0 0) 0x7f7c9c003520 con 0x7f7cb8156af0 2024-02-21T23:16:15.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.302+0000 7f7cb6000640 1 -- 172.21.15.28:0/3246899579 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7cb806ee40 msgr2=0x7f7cb81536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:16:15.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.302+0000 7f7cb6000640 1 --2- 172.21.15.28:0/3246899579 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7cb806ee40 0x7f7cb81536f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:15.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.302+0000 7f7cb6000640 1 -- 172.21.15.28:0/3246899579 >> v2:172.21.15.81:3300/0 conn(0x7f7cb8150450 msgr2=0x7f7cb80a5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:16:15.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.302+0000 7f7cb6000640 1 --2- 172.21.15.28:0/3246899579 >> v2:172.21.15.81:3300/0 conn(0x7f7cb8150450 0x7f7cb80a5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:15.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.302+0000 7f7cb6000640 1 -- 172.21.15.28:0/3246899579 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cb8153c30 con 0x7f7cb8156af0 2024-02-21T23:16:15.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.302+0000 7f7cb6000640 1 -- 172.21.15.28:0/3246899579 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f7cb00019f0 con 0x7f7cb8156af0 2024-02-21T23:16:15.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.302+0000 7f7cb6000640 1 -- 172.21.15.28:0/3246899579 >> v1:172.21.15.28:6789/0 conn(0x7f7cb8156af0 legacy=0x7f7cb806dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:15.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.302+0000 7f7cb6000640 1 --2- 172.21.15.28:0/3246899579 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7cb806ee40 0x7f7c9c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:15.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.302+0000 7f7cb6000640 1 --2- 172.21.15.28:0/3246899579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c9c003f80 0x7f7c9c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:15.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.302+0000 7f7cb6000640 1 --2- 172.21.15.28:0/3246899579 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7c9c004d40 0x7f7c9c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:15.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.302+0000 7f7cb6000640 1 -- 172.21.15.28:0/3246899579 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb8156660 con 0x7f7c9c003f80 2024-02-21T23:16:15.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.302+0000 7f7cb6000640 1 -- 172.21.15.28:0/3246899579 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c9c00b9c0 con 0x7f7c9c004d40 2024-02-21T23:16:15.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.302+0000 7f7cb6000640 1 -- 172.21.15.28:0/3246899579 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c9c00bb40 con 0x7f7cb806ee40 2024-02-21T23:16:15.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.302+0000 7f7cae000640 1 --2- 172.21.15.28:0/3246899579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c9c003f80 0x7f7c9c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:15.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.303+0000 7f7cb6a00640 1 --2- 172.21.15.28:0/3246899579 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7cb806ee40 0x7f7c9c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:15.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.303+0000 7f7cae000640 1 -- 172.21.15.28:0/3246899579 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7cb806ee40 msgr2=0x7f7c9c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:16:15.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.303+0000 7f7cae000640 1 --2- 172.21.15.28:0/3246899579 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7cb806ee40 0x7f7c9c0039b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:15.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.303+0000 7f7cb7400640 1 --2- 172.21.15.28:0/3246899579 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7c9c004d40 0x7f7c9c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:15.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.303+0000 7f7cae000640 1 -- 172.21.15.28:0/3246899579 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7c9c004d40 msgr2=0x7f7c9c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:16:15.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.303+0000 7f7cae000640 1 --2- 172.21.15.28:0/3246899579 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7c9c004d40 0x7f7c9c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:15.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.303+0000 7f7cae000640 1 -- 172.21.15.28:0/3246899579 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c9c003520 con 0x7f7c9c003f80 2024-02-21T23:16:15.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.303+0000 7f7cb7400640 1 --2- 172.21.15.28:0/3246899579 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7c9c004d40 0x7f7c9c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:16:15.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.304+0000 7f7cae000640 1 --2- 172.21.15.28:0/3246899579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c9c003f80 0x7f7c9c004800 secure :-1 s=READY pgs=2613 cs=0 l=1 rev1=1 crypto rx=0x7f7ca0009af0 tx=0x7f7c9c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0cb949c27e881d0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:15.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.304+0000 7f7cb6000640 1 -- 172.21.15.28:0/3246899579 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ca0039070 con 0x7f7c9c003f80 2024-02-21T23:16:15.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.304+0000 7f7cb6000640 1 -- 172.21.15.28:0/3246899579 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ca002c6b0 con 0x7f7c9c003f80 2024-02-21T23:16:15.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.304+0000 7f7cb6000640 1 -- 172.21.15.28:0/3246899579 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ca0009690 con 0x7f7c9c003f80 2024-02-21T23:16:15.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.305+0000 7f7cb7e00640 1 -- 172.21.15.28:0/3246899579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c9c003f80 msgr2=0x7f7c9c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:15.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.305+0000 7f7cb7e00640 1 --2- 172.21.15.28:0/3246899579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c9c003f80 0x7f7c9c004800 secure :-1 s=READY pgs=2613 cs=0 l=1 rev1=1 crypto rx=0x7f7ca0009af0 tx=0x7f7c9c005830 comp rx=0 tx=0).stop 2024-02-21T23:16:15.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.305+0000 7f7cb7400640 1 -- 172.21.15.28:0/3246899579 reap_dead start 2024-02-21T23:16:15.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.305+0000 7f7cb7e00640 1 -- 172.21.15.28:0/3246899579 shutdown_connections 2024-02-21T23:16:15.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.305+0000 7f7cb7e00640 1 -- 172.21.15.28:0/3246899579 >> 172.21.15.28:0/3246899579 conn(0x7f7cb806b630 msgr2=0x7f7cb80c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:16:15.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.305+0000 7f7cb7e00640 1 -- 172.21.15.28:0/3246899579 shutdown_connections 2024-02-21T23:16:15.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.305+0000 7f7cb7e00640 1 -- 172.21.15.28:0/3246899579 wait complete. 2024-02-21T23:16:15.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.307+0000 7f7cb7e00640 1 Processor -- start 2024-02-21T23:16:15.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.307+0000 7f7cb7e00640 1 -- start start 2024-02-21T23:16:15.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.308+0000 7f7cb7e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7c9c003f80 0x7f7cb80bfba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:15.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.308+0000 7f7cb7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cb80c0780 0x7f7cb8149650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:15.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.308+0000 7f7cb7e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7cb81dc5f0 0x7f7cb8149b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:15.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.308+0000 7f7cb7e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb80c2cb0 con 0x7f7cb80c0780 2024-02-21T23:16:15.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.308+0000 7f7cb7e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb80c29d0 con 0x7f7cb81dc5f0 2024-02-21T23:16:15.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.308+0000 7f7cb7e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb806e700 con 0x7f7c9c003f80 2024-02-21T23:16:15.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.308+0000 7f7cae000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cb80c0780 0x7f7cb8149650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:15.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.308+0000 7f7cae000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cb80c0780 0x7f7cb8149650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37514/0 (socket says 172.21.15.28:37514) 2024-02-21T23:16:15.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.308+0000 7f7cae000640 1 -- 172.21.15.28:0/3146833304 learned_addr learned my addr 172.21.15.28:0/3146833304 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:16:15.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.309+0000 7f7cb6a00640 1 --2- 172.21.15.28:0/3146833304 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7c9c003f80 0x7f7cb80bfba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:15.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.309+0000 7f7cae000640 1 -- 172.21.15.28:0/3146833304 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7c9c003f80 msgr2=0x7f7cb80bfba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:15.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.309+0000 7f7cae000640 1 --2- 172.21.15.28:0/3146833304 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7c9c003f80 0x7f7cb80bfba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:15.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.309+0000 7f7cb7400640 1 --2- 172.21.15.28:0/3146833304 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7cb81dc5f0 0x7f7cb8149b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:15.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.309+0000 7f7cae000640 1 -- 172.21.15.28:0/3146833304 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7cb81dc5f0 msgr2=0x7f7cb8149b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:15.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.309+0000 7f7cae000640 1 --2- 172.21.15.28:0/3146833304 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7cb81dc5f0 0x7f7cb8149b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:15.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.309+0000 7f7cae000640 1 -- 172.21.15.28:0/3146833304 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cb814a0d0 con 0x7f7cb80c0780 2024-02-21T23:16:15.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.309+0000 7f7cae000640 1 --2- 172.21.15.28:0/3146833304 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cb80c0780 0x7f7cb8149650 secure :-1 s=READY pgs=2614 cs=0 l=1 rev1=1 crypto rx=0x7f7ca0037040 tx=0x7f7ca0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:15.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.309+0000 7f7caf400640 1 -- 172.21.15.28:0/3146833304 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ca0042020 con 0x7f7cb80c0780 2024-02-21T23:16:15.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.310+0000 7f7caf400640 1 -- 172.21.15.28:0/3146833304 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ca003a040 con 0x7f7cb80c0780 2024-02-21T23:16:15.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.310+0000 7f7caf400640 1 -- 172.21.15.28:0/3146833304 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ca0039070 con 0x7f7cb80c0780 2024-02-21T23:16:15.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.310+0000 7f7cb7e00640 1 -- 172.21.15.28:0/3146833304 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cb81df150 con 0x7f7cb80c0780 2024-02-21T23:16:15.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.310+0000 7f7cb7e00640 1 -- 172.21.15.28:0/3146833304 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cb8072950 con 0x7f7cb80c0780 2024-02-21T23:16:15.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.311+0000 7f7cb7e00640 1 -- 172.21.15.28:0/3146833304 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cb804f290 con 0x7f7cb80c0780 2024-02-21T23:16:15.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.312+0000 7f7caf400640 1 -- 172.21.15.28:0/3146833304 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7ca0045020 con 0x7f7cb80c0780 2024-02-21T23:16:15.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.313+0000 7f7caf400640 1 --2- 172.21.15.28:0/3146833304 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7c9406b500 0x7f7c9406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:15.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.313+0000 7f7caf400640 1 -- 172.21.15.28:0/3146833304 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f7ca00ab850 con 0x7f7cb80c0780 2024-02-21T23:16:15.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.313+0000 7f7cb6a00640 1 --2- 172.21.15.28:0/3146833304 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7c9406b500 0x7f7c9406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:15.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.315+0000 7f7cb6a00640 1 --2- 172.21.15.28:0/3146833304 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7c9406b500 0x7f7c9406d9c0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f7ca8009990 tx=0x7f7ca8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:15.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.318+0000 7f7caf400640 1 -- 172.21.15.28:0/3146833304 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7ca004c070 con 0x7f7cb80c0780 2024-02-21T23:16:15.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.562+0000 7f7cb7e00640 1 -- 172.21.15.28:0/3146833304 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7cb80c0be0 con 0x7f7cb80c0780 2024-02-21T23:16:15.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.562+0000 7f7caf400640 1 -- 172.21.15.28:0/3146833304 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7f7ca007a430 con 0x7f7cb80c0780 2024-02-21T23:16:15.550 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:16:15.550 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:16:15.550 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:16:15.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.565+0000 7f7cb7e00640 1 -- 172.21.15.28:0/3146833304 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7c9406b500 msgr2=0x7f7c9406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:15.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.565+0000 7f7cb7e00640 1 --2- 172.21.15.28:0/3146833304 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7c9406b500 0x7f7c9406d9c0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f7ca8009990 tx=0x7f7ca8008040 comp rx=0 tx=0).stop 2024-02-21T23:16:15.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.565+0000 7f7cb7e00640 1 -- 172.21.15.28:0/3146833304 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cb80c0780 msgr2=0x7f7cb8149650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:15.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.565+0000 7f7cb7e00640 1 --2- 172.21.15.28:0/3146833304 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cb80c0780 0x7f7cb8149650 secure :-1 s=READY pgs=2614 cs=0 l=1 rev1=1 crypto rx=0x7f7ca0037040 tx=0x7f7ca0004730 comp rx=0 tx=0).stop 2024-02-21T23:16:15.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.565+0000 7f7cb7e00640 1 -- 172.21.15.28:0/3146833304 shutdown_connections 2024-02-21T23:16:15.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.565+0000 7f7cb7e00640 1 --2- 172.21.15.28:0/3146833304 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7c9406b500 0x7f7c9406d9c0 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:15.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.565+0000 7f7cb7e00640 1 --2- 172.21.15.28:0/3146833304 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7cb81dc5f0 0x7f7cb8149b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:15.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.565+0000 7f7cb7e00640 1 --2- 172.21.15.28:0/3146833304 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cb80c0780 0x7f7cb8149650 unknown :-1 s=CLOSED pgs=2614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:15.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.565+0000 7f7cb7e00640 1 --2- 172.21.15.28:0/3146833304 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7c9c003f80 0x7f7cb80bfba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:15.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.565+0000 7f7cb7e00640 1 -- 172.21.15.28:0/3146833304 >> 172.21.15.28:0/3146833304 conn(0x7f7cb806b630 msgr2=0x7f7cb80a6e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:16:15.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.565+0000 7f7cb7e00640 1 -- 172.21.15.28:0/3146833304 shutdown_connections 2024-02-21T23:16:15.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:15.565+0000 7f7cb7e00640 1 -- 172.21.15.28:0/3146833304 wait complete. 2024-02-21T23:16:15.973 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:16:15.974 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:16:16.974 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:16:17.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.215+0000 7f9732800640 1 Processor -- start 2024-02-21T23:16:17.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.215+0000 7f9732800640 1 -- start start 2024-02-21T23:16:17.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.217+0000 7f9732800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f972c14f0b0 0x7f972c155180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:17.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.217+0000 7f9732800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f972c0a6160 0x7f972c0c2010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:17.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.217+0000 7f9732800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f972c155760 con 0x7f972c0c2550 2024-02-21T23:16:17.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.217+0000 7f9732800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f972c151d30 con 0x7f972c14f0b0 2024-02-21T23:16:17.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.217+0000 7f9732800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f972c155a40 con 0x7f972c0a6160 2024-02-21T23:16:17.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.217+0000 7f9731400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f972c14f0b0 0x7f972c155180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:17.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.217+0000 7f9731400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f972c14f0b0 0x7f972c155180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37516/0 (socket says 172.21.15.28:37516) 2024-02-21T23:16:17.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.217+0000 7f9731e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f972c0c2550 0x7f972c1521c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50290/0 (socket says 172.21.15.28:50290) 2024-02-21T23:16:17.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.217+0000 7f9731400640 1 -- 172.21.15.28:0/3584751767 learned_addr learned my addr 172.21.15.28:0/3584751767 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:16:17.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.218+0000 7f9730a00640 1 --2- 172.21.15.28:0/3584751767 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f972c0a6160 0x7f972c0c2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:17.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.218+0000 7f9731400640 1 -- 172.21.15.28:0/3584751767 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f972c0a6160 msgr2=0x7f972c0c2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:16:17.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.218+0000 7f9731400640 1 --2- 172.21.15.28:0/3584751767 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f972c0a6160 0x7f972c0c2010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:17.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.218+0000 7f9731400640 1 -- 172.21.15.28:0/3584751767 >> v1:172.21.15.28:6789/0 conn(0x7f972c0c2550 legacy=0x7f972c1521c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:17.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.218+0000 7f9731400640 1 -- 172.21.15.28:0/3584751767 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f972c155d20 con 0x7f972c14f0b0 2024-02-21T23:16:17.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.218+0000 7f9730a00640 1 --2- 172.21.15.28:0/3584751767 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f972c0a6160 0x7f972c0c2010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:16:17.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.218+0000 7f972be00640 1 -- 172.21.15.28:0/3584751767 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2658400979 0 0) 0x7f972c155760 con 0x7f972c0c2550 2024-02-21T23:16:17.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.219+0000 7f9731400640 1 --2- 172.21.15.28:0/3584751767 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f972c14f0b0 0x7f972c155180 secure :-1 s=READY pgs=2616 cs=0 l=1 rev1=1 crypto rx=0x7f9720009870 tx=0x7f972002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=248f2a91d42b6b5b server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:17.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.219+0000 7f972be00640 1 -- 172.21.15.28:0/3584751767 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9720034070 con 0x7f972c14f0b0 2024-02-21T23:16:17.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.219+0000 7f972be00640 1 -- 172.21.15.28:0/3584751767 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9720030840 con 0x7f972c14f0b0 2024-02-21T23:16:17.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.219+0000 7f972be00640 1 -- 172.21.15.28:0/3584751767 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9720030c10 con 0x7f972c14f0b0 2024-02-21T23:16:17.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.220+0000 7f9732800640 1 -- 172.21.15.28:0/3584751767 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f972c14f0b0 msgr2=0x7f972c155180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:17.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.220+0000 7f9732800640 1 --2- 172.21.15.28:0/3584751767 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f972c14f0b0 0x7f972c155180 secure :-1 s=READY pgs=2616 cs=0 l=1 rev1=1 crypto rx=0x7f9720009870 tx=0x7f972002b980 comp rx=0 tx=0).stop 2024-02-21T23:16:17.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.220+0000 7f9732800640 1 -- 172.21.15.28:0/3584751767 shutdown_connections 2024-02-21T23:16:17.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.220+0000 7f9732800640 1 --2- 172.21.15.28:0/3584751767 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f972c0a6160 0x7f972c0c2010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:17.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.220+0000 7f9732800640 1 --2- 172.21.15.28:0/3584751767 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f972c14f0b0 0x7f972c155180 unknown :-1 s=CLOSED pgs=2616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:17.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.220+0000 7f9732800640 1 -- 172.21.15.28:0/3584751767 >> 172.21.15.28:0/3584751767 conn(0x7f972c06b630 msgr2=0x7f972c0c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:16:17.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.221+0000 7f9732800640 1 -- 172.21.15.28:0/3584751767 shutdown_connections 2024-02-21T23:16:17.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.221+0000 7f9732800640 1 -- 172.21.15.28:0/3584751767 wait complete. 2024-02-21T23:16:17.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.222+0000 7f9732800640 1 Processor -- start 2024-02-21T23:16:17.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.222+0000 7f9732800640 1 -- start start 2024-02-21T23:16:17.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.223+0000 7f9732800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f972c0a6160 0x7f972c14fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:17.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.223+0000 7f9732800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f972c0c2550 0x7f972c0c00b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:17.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.223+0000 7f9732800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f972c14f0b0 0x7f972c148280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:17.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.223+0000 7f9732800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f972c0a7f70 con 0x7f972c0c2550 2024-02-21T23:16:17.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.223+0000 7f9732800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f972c0a68d0 con 0x7f972c0a6160 2024-02-21T23:16:17.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.223+0000 7f9732800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f972c0a80f0 con 0x7f972c14f0b0 2024-02-21T23:16:17.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.224+0000 7f9730a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f972c0c2550 0x7f972c0c00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:17.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.224+0000 7f9730a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f972c0c2550 0x7f972c0c00b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37524/0 (socket says 172.21.15.28:37524) 2024-02-21T23:16:17.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.224+0000 7f9730a00640 1 -- 172.21.15.28:0/1200808104 learned_addr learned my addr 172.21.15.28:0/1200808104 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:16:17.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.224+0000 7f9731e00640 1 --2- 172.21.15.28:0/1200808104 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f972c14f0b0 0x7f972c148280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:17.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.224+0000 7f9731400640 1 --2- 172.21.15.28:0/1200808104 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f972c0a6160 0x7f972c14fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:17.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.224+0000 7f9730a00640 1 -- 172.21.15.28:0/1200808104 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f972c14f0b0 msgr2=0x7f972c148280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:17.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.224+0000 7f9730a00640 1 --2- 172.21.15.28:0/1200808104 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f972c14f0b0 0x7f972c148280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:17.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.224+0000 7f9730a00640 1 -- 172.21.15.28:0/1200808104 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f972c0a6160 msgr2=0x7f972c14fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:17.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.224+0000 7f9730a00640 1 --2- 172.21.15.28:0/1200808104 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f972c0a6160 0x7f972c14fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:17.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.224+0000 7f9730a00640 1 -- 172.21.15.28:0/1200808104 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f972c14e3e0 con 0x7f972c0c2550 2024-02-21T23:16:17.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.224+0000 7f9731e00640 1 --2- 172.21.15.28:0/1200808104 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f972c14f0b0 0x7f972c148280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:16:17.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.224+0000 7f9731400640 1 --2- 172.21.15.28:0/1200808104 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f972c0a6160 0x7f972c14fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:16:17.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.225+0000 7f9730a00640 1 --2- 172.21.15.28:0/1200808104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f972c0c2550 0x7f972c0c00b0 secure :-1 s=READY pgs=2617 cs=0 l=1 rev1=1 crypto rx=0x7f971800c7f0 tx=0x7f971800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:17.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.225+0000 7f9729600640 1 -- 172.21.15.28:0/1200808104 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9718015070 con 0x7f972c0c2550 2024-02-21T23:16:17.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.225+0000 7f9729600640 1 -- 172.21.15.28:0/1200808104 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9718007a30 con 0x7f972c0c2550 2024-02-21T23:16:17.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.225+0000 7f9732800640 1 -- 172.21.15.28:0/1200808104 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f972c1eaed0 con 0x7f972c0c2550 2024-02-21T23:16:17.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.225+0000 7f9729600640 1 -- 172.21.15.28:0/1200808104 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9718007e00 con 0x7f972c0c2550 2024-02-21T23:16:17.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.225+0000 7f9732800640 1 -- 172.21.15.28:0/1200808104 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f972c0c05f0 con 0x7f972c0c2550 2024-02-21T23:16:17.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.226+0000 7f9732800640 1 -- 172.21.15.28:0/1200808104 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f972c1eacc0 con 0x7f972c0c2550 2024-02-21T23:16:17.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.227+0000 7f9729600640 1 -- 172.21.15.28:0/1200808104 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9718018070 con 0x7f972c0c2550 2024-02-21T23:16:17.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.229+0000 7f9729600640 1 --2- 172.21.15.28:0/1200808104 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f970c06b4d0 0x7f970c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:17.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.229+0000 7f9729600640 1 -- 172.21.15.28:0/1200808104 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f971808a4a0 con 0x7f972c0c2550 2024-02-21T23:16:17.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.229+0000 7f9731400640 1 --2- 172.21.15.28:0/1200808104 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f970c06b4d0 0x7f970c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:17.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.230+0000 7f9731400640 1 --2- 172.21.15.28:0/1200808104 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f970c06b4d0 0x7f970c06d990 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f9720036000 tx=0x7f9720008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:17.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.234+0000 7f9729600640 1 -- 172.21.15.28:0/1200808104 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f9718059080 con 0x7f972c0c2550 2024-02-21T23:16:17.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.493+0000 7f9732800640 1 -- 172.21.15.28:0/1200808104 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f972c04f290 con 0x7f972c0c2550 2024-02-21T23:16:17.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.494+0000 7f9729600640 1 -- 172.21.15.28:0/1200808104 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7f9718058980 con 0x7f972c0c2550 2024-02-21T23:16:17.482 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:16:17.482 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:16:17.482 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:16:17.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.497+0000 7f9732800640 1 -- 172.21.15.28:0/1200808104 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f970c06b4d0 msgr2=0x7f970c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:17.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.497+0000 7f9732800640 1 --2- 172.21.15.28:0/1200808104 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f970c06b4d0 0x7f970c06d990 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f9720036000 tx=0x7f9720008030 comp rx=0 tx=0).stop 2024-02-21T23:16:17.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.497+0000 7f9732800640 1 -- 172.21.15.28:0/1200808104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f972c0c2550 msgr2=0x7f972c0c00b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:17.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.497+0000 7f9732800640 1 --2- 172.21.15.28:0/1200808104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f972c0c2550 0x7f972c0c00b0 secure :-1 s=READY pgs=2617 cs=0 l=1 rev1=1 crypto rx=0x7f971800c7f0 tx=0x7f971800cbf0 comp rx=0 tx=0).stop 2024-02-21T23:16:17.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.497+0000 7f9732800640 1 -- 172.21.15.28:0/1200808104 shutdown_connections 2024-02-21T23:16:17.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.497+0000 7f9732800640 1 --2- 172.21.15.28:0/1200808104 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f970c06b4d0 0x7f970c06d990 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:17.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.497+0000 7f9732800640 1 --2- 172.21.15.28:0/1200808104 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f972c14f0b0 0x7f972c148280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:17.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.497+0000 7f9732800640 1 --2- 172.21.15.28:0/1200808104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f972c0c2550 0x7f972c0c00b0 unknown :-1 s=CLOSED pgs=2617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:17.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.497+0000 7f9732800640 1 --2- 172.21.15.28:0/1200808104 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f972c0a6160 0x7f972c14fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:17.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.497+0000 7f9732800640 1 -- 172.21.15.28:0/1200808104 >> 172.21.15.28:0/1200808104 conn(0x7f972c06b630 msgr2=0x7f972c149390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:16:17.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.497+0000 7f9732800640 1 -- 172.21.15.28:0/1200808104 shutdown_connections 2024-02-21T23:16:17.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:17.497+0000 7f9732800640 1 -- 172.21.15.28:0/1200808104 wait complete. 2024-02-21T23:16:17.907 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:16:17.907 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:16:18.909 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:16:19.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.138+0000 7fd5f7e00640 1 Processor -- start 2024-02-21T23:16:19.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.138+0000 7fd5f7e00640 1 -- start start 2024-02-21T23:16:19.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.141+0000 7fd5f7e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd5f814ef60 0x7fd5f8158ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:19.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.141+0000 7fd5f7e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd5f80c6740 0x7fd5f8154fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:19.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.141+0000 7fd5f7e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5f8159410 con 0x7fd5f81554f0 2024-02-21T23:16:19.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.141+0000 7fd5f6a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd5f814ef60 0x7fd5f8158ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:19.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.141+0000 7fd5f6a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd5f814ef60 0x7fd5f8158ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37536/0 (socket says 172.21.15.28:37536) 2024-02-21T23:16:19.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.141+0000 7fd5f6a00640 1 -- 172.21.15.28:0/1559941255 learned_addr learned my addr 172.21.15.28:0/1559941255 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:16:19.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.141+0000 7fd5f7e00640 1 -- 172.21.15.28:0/1559941255 --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5f8155a00 con 0x7fd5f814ef60 2024-02-21T23:16:19.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.141+0000 7fd5f7e00640 1 -- 172.21.15.28:0/1559941255 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5f8155ce0 con 0x7fd5f80c6740 2024-02-21T23:16:19.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.142+0000 7fd5efe00640 1 --2- 172.21.15.28:0/1559941255 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd5f80c6740 0x7fd5f8154fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:19.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.142+0000 7fd5f6a00640 1 -- 172.21.15.28:0/1559941255 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd5f80c6740 msgr2=0x7fd5f8154fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:16:19.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.142+0000 7fd5f6a00640 1 --2- 172.21.15.28:0/1559941255 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd5f80c6740 0x7fd5f8154fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:19.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.142+0000 7fd5f6a00640 1 -- 172.21.15.28:0/1559941255 >> v1:172.21.15.81:6789/0 conn(0x7fd5f81554f0 legacy=0x7fd5f80c8a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:19.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.142+0000 7fd5f6a00640 1 -- 172.21.15.28:0/1559941255 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5f81586d0 con 0x7fd5f814ef60 2024-02-21T23:16:19.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.143+0000 7fd5f6a00640 1 --2- 172.21.15.28:0/1559941255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5f814ef60 0x7fd5f8158ed0 secure :-1 s=READY pgs=2618 cs=0 l=1 rev1=1 crypto rx=0x7fd5e8009870 tx=0x7fd5e802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f06428433d91b515 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:19.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.144+0000 7fd5f6000640 1 -- 172.21.15.28:0/1559941255 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5e8034070 con 0x7fd5f814ef60 2024-02-21T23:16:19.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.144+0000 7fd5f6000640 1 -- 172.21.15.28:0/1559941255 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5e8030840 con 0x7fd5f814ef60 2024-02-21T23:16:19.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.144+0000 7fd5f6000640 1 -- 172.21.15.28:0/1559941255 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5e8030c10 con 0x7fd5f814ef60 2024-02-21T23:16:19.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.145+0000 7fd5f7e00640 1 -- 172.21.15.28:0/1559941255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5f814ef60 msgr2=0x7fd5f8158ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:19.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.145+0000 7fd5f7e00640 1 --2- 172.21.15.28:0/1559941255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5f814ef60 0x7fd5f8158ed0 secure :-1 s=READY pgs=2618 cs=0 l=1 rev1=1 crypto rx=0x7fd5e8009870 tx=0x7fd5e802b980 comp rx=0 tx=0).stop 2024-02-21T23:16:19.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.145+0000 7fd5f7e00640 1 -- 172.21.15.28:0/1559941255 shutdown_connections 2024-02-21T23:16:19.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.145+0000 7fd5f7e00640 1 --2- 172.21.15.28:0/1559941255 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd5f80c6740 0x7fd5f8154fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:19.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.145+0000 7fd5f7e00640 1 --2- 172.21.15.28:0/1559941255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5f814ef60 0x7fd5f8158ed0 unknown :-1 s=CLOSED pgs=2618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:19.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.145+0000 7fd5f7e00640 1 -- 172.21.15.28:0/1559941255 >> 172.21.15.28:0/1559941255 conn(0x7fd5f806b260 msgr2=0x7fd5f806b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:16:19.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.145+0000 7fd5f7e00640 1 -- 172.21.15.28:0/1559941255 shutdown_connections 2024-02-21T23:16:19.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.145+0000 7fd5f7e00640 1 -- 172.21.15.28:0/1559941255 wait complete. 2024-02-21T23:16:19.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.147+0000 7fd5f7e00640 1 Processor -- start 2024-02-21T23:16:19.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.147+0000 7fd5f7e00640 1 -- start start 2024-02-21T23:16:19.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.148+0000 7fd5f7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5f80c6740 0x7fd5f806f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:19.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.148+0000 7fd5f7e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd5f814ef60 0x7fd5f81577f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:19.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.148+0000 7fd5f7e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5f81554f0 0x7fd5f8151ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:19.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.148+0000 7fd5f7e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5f80c9120 con 0x7fd5f80c6740 2024-02-21T23:16:19.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.148+0000 7fd5f7e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5f814f340 con 0x7fd5f81554f0 2024-02-21T23:16:19.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.148+0000 7fd5f7e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5f80c92a0 con 0x7fd5f814ef60 2024-02-21T23:16:19.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.148+0000 7fd5f6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5f80c6740 0x7fd5f806f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:19.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.148+0000 7fd5f6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5f80c6740 0x7fd5f806f130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37552/0 (socket says 172.21.15.28:37552) 2024-02-21T23:16:19.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.148+0000 7fd5f6a00640 1 -- 172.21.15.28:0/1646540783 learned_addr learned my addr 172.21.15.28:0/1646540783 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:16:19.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.148+0000 7fd5efe00640 1 --2- 172.21.15.28:0/1646540783 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd5f814ef60 0x7fd5f81577f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:19.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.148+0000 7fd5f6a00640 1 -- 172.21.15.28:0/1646540783 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd5f814ef60 msgr2=0x7fd5f81577f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:19.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.148+0000 7fd5f6a00640 1 --2- 172.21.15.28:0/1646540783 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd5f814ef60 0x7fd5f81577f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:19.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.148+0000 7fd5f6a00640 1 -- 172.21.15.28:0/1646540783 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5f81554f0 msgr2=0x7fd5f8151ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:19.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.148+0000 7fd5f7400640 1 --2- 172.21.15.28:0/1646540783 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5f81554f0 0x7fd5f8151ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:19.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.149+0000 7fd5f6a00640 1 --2- 172.21.15.28:0/1646540783 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5f81554f0 0x7fd5f8151ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:19.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.149+0000 7fd5f6a00640 1 -- 172.21.15.28:0/1646540783 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5f81efed0 con 0x7fd5f80c6740 2024-02-21T23:16:19.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.149+0000 7fd5efe00640 1 --2- 172.21.15.28:0/1646540783 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd5f814ef60 0x7fd5f81577f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:16:19.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.149+0000 7fd5f7400640 1 --2- 172.21.15.28:0/1646540783 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5f81554f0 0x7fd5f8151ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:16:19.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.149+0000 7fd5f6a00640 1 --2- 172.21.15.28:0/1646540783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5f80c6740 0x7fd5f806f130 secure :-1 s=READY pgs=2619 cs=0 l=1 rev1=1 crypto rx=0x7fd5e802b9c0 tx=0x7fd5e8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:19.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.149+0000 7fd5eea00640 1 -- 172.21.15.28:0/1646540783 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5e8048070 con 0x7fd5f80c6740 2024-02-21T23:16:19.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.149+0000 7fd5eea00640 1 -- 172.21.15.28:0/1646540783 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5e8030640 con 0x7fd5f80c6740 2024-02-21T23:16:19.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.149+0000 7fd5eea00640 1 -- 172.21.15.28:0/1646540783 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5e8034040 con 0x7fd5f80c6740 2024-02-21T23:16:19.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.149+0000 7fd5f7e00640 1 -- 172.21.15.28:0/1646540783 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5f8156d50 con 0x7fd5f80c6740 2024-02-21T23:16:19.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.150+0000 7fd5f7e00640 1 -- 172.21.15.28:0/1646540783 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5f8156ff0 con 0x7fd5f80c6740 2024-02-21T23:16:19.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.151+0000 7fd5f7e00640 1 -- 172.21.15.28:0/1646540783 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5f804f290 con 0x7fd5f80c6740 2024-02-21T23:16:19.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.152+0000 7fd5eea00640 1 -- 172.21.15.28:0/1646540783 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd5e804b070 con 0x7fd5f80c6740 2024-02-21T23:16:19.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.153+0000 7fd5eea00640 1 --2- 172.21.15.28:0/1646540783 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd5d406b4d0 0x7fd5d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:19.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.153+0000 7fd5eea00640 1 -- 172.21.15.28:0/1646540783 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fd5e80ac330 con 0x7fd5f80c6740 2024-02-21T23:16:19.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.153+0000 7fd5efe00640 1 --2- 172.21.15.28:0/1646540783 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd5d406b4d0 0x7fd5d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:19.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.154+0000 7fd5efe00640 1 --2- 172.21.15.28:0/1646540783 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd5d406b4d0 0x7fd5d406d990 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7fd5f8070c70 tx=0x7fd5e0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:19.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.157+0000 7fd5eea00640 1 -- 172.21.15.28:0/1646540783 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd5e807b0d0 con 0x7fd5f80c6740 2024-02-21T23:16:19.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.402+0000 7fd5f7e00640 1 -- 172.21.15.28:0/1646540783 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd5f80c80d0 con 0x7fd5f80c6740 2024-02-21T23:16:19.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.403+0000 7fd5eea00640 1 -- 172.21.15.28:0/1646540783 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7fd5e807a9d0 con 0x7fd5f80c6740 2024-02-21T23:16:19.391 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:16:19.391 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:16:19.391 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:16:19.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.406+0000 7fd5f7e00640 1 -- 172.21.15.28:0/1646540783 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd5d406b4d0 msgr2=0x7fd5d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:19.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.406+0000 7fd5f7e00640 1 --2- 172.21.15.28:0/1646540783 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd5d406b4d0 0x7fd5d406d990 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7fd5f8070c70 tx=0x7fd5e0008040 comp rx=0 tx=0).stop 2024-02-21T23:16:19.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.406+0000 7fd5f7e00640 1 -- 172.21.15.28:0/1646540783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5f80c6740 msgr2=0x7fd5f806f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:19.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.406+0000 7fd5f7e00640 1 --2- 172.21.15.28:0/1646540783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5f80c6740 0x7fd5f806f130 secure :-1 s=READY pgs=2619 cs=0 l=1 rev1=1 crypto rx=0x7fd5e802b9c0 tx=0x7fd5e8004730 comp rx=0 tx=0).stop 2024-02-21T23:16:19.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.406+0000 7fd5f7e00640 1 -- 172.21.15.28:0/1646540783 shutdown_connections 2024-02-21T23:16:19.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.407+0000 7fd5f7e00640 1 --2- 172.21.15.28:0/1646540783 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd5d406b4d0 0x7fd5d406d990 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:19.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.407+0000 7fd5f7e00640 1 --2- 172.21.15.28:0/1646540783 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd5f814ef60 0x7fd5f81577f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:19.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.407+0000 7fd5f7e00640 1 --2- 172.21.15.28:0/1646540783 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd5f81554f0 0x7fd5f8151ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:19.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.407+0000 7fd5f7e00640 1 --2- 172.21.15.28:0/1646540783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5f80c6740 0x7fd5f806f130 unknown :-1 s=CLOSED pgs=2619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:19.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.407+0000 7fd5f7e00640 1 -- 172.21.15.28:0/1646540783 >> 172.21.15.28:0/1646540783 conn(0x7fd5f806b260 msgr2=0x7fd5f814d290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:16:19.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.407+0000 7fd5f7e00640 1 -- 172.21.15.28:0/1646540783 shutdown_connections 2024-02-21T23:16:19.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:19.407+0000 7fd5f7e00640 1 -- 172.21.15.28:0/1646540783 wait complete. 2024-02-21T23:16:19.814 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:16:19.814 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:16:20.815 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:16:21.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.054+0000 7fd121c00640 1 Processor -- start 2024-02-21T23:16:21.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.055+0000 7fd121c00640 1 -- start start 2024-02-21T23:16:21.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.056+0000 7fd121c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd11c14f170 0x7fd11c0a90e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:21.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.057+0000 7fd121c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd11c0707f0 0x7fd11c152560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:21.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.057+0000 7fd121c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fd11c155950 0x7fd11c0721d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:21.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.057+0000 7fd121c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd11c14f550 con 0x7fd11c14f170 2024-02-21T23:16:21.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.057+0000 7fd121c00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd11c0a9620 con 0x7fd11c155950 2024-02-21T23:16:21.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.057+0000 7fd121c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd11c152aa0 con 0x7fd11c0707f0 2024-02-21T23:16:21.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.057+0000 7fd11be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd11c14f170 0x7fd11c0a90e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:21.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.057+0000 7fd11be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd11c14f170 0x7fd11c0a90e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37556/0 (socket says 172.21.15.28:37556) 2024-02-21T23:16:21.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.057+0000 7fd11be00640 1 -- 172.21.15.28:0/1109426793 learned_addr learned my addr 172.21.15.28:0/1109426793 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:16:21.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.057+0000 7fd113e00640 1 --2- 172.21.15.28:0/1109426793 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd11c0707f0 0x7fd11c152560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:21.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.057+0000 7fd121200640 1 --2- 172.21.15.28:0/1109426793 >> v2:172.21.15.81:3300/0 conn(0x7fd11c155950 0x7fd11c0721d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:21.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.058+0000 7fd11be00640 1 -- 172.21.15.28:0/1109426793 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd11c0707f0 msgr2=0x7fd11c152560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:16:21.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.058+0000 7fd11be00640 1 --2- 172.21.15.28:0/1109426793 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd11c0707f0 0x7fd11c152560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:21.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.058+0000 7fd11be00640 1 -- 172.21.15.28:0/1109426793 >> v2:172.21.15.81:3300/0 conn(0x7fd11c155950 msgr2=0x7fd11c0721d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:16:21.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.058+0000 7fd11be00640 1 --2- 172.21.15.28:0/1109426793 >> v2:172.21.15.81:3300/0 conn(0x7fd11c155950 0x7fd11c0721d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:21.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.058+0000 7fd11be00640 1 -- 172.21.15.28:0/1109426793 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd11c0a7550 con 0x7fd11c14f170 2024-02-21T23:16:21.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.059+0000 7fd11be00640 1 --2- 172.21.15.28:0/1109426793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd11c14f170 0x7fd11c0a90e0 secure :-1 s=READY pgs=2620 cs=0 l=1 rev1=1 crypto rx=0x7fd11c070d60 tx=0x7fd10c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ea09af9c9bb7dce server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:21.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.060+0000 7fd11b400640 1 -- 172.21.15.28:0/1109426793 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd10c035020 con 0x7fd11c14f170 2024-02-21T23:16:21.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.060+0000 7fd11b400640 1 -- 172.21.15.28:0/1109426793 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd10c031870 con 0x7fd11c14f170 2024-02-21T23:16:21.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.060+0000 7fd11b400640 1 -- 172.21.15.28:0/1109426793 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd10c031c40 con 0x7fd11c14f170 2024-02-21T23:16:21.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.060+0000 7fd121c00640 1 -- 172.21.15.28:0/1109426793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd11c14f170 msgr2=0x7fd11c0a90e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:21.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.060+0000 7fd121c00640 1 --2- 172.21.15.28:0/1109426793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd11c14f170 0x7fd11c0a90e0 secure :-1 s=READY pgs=2620 cs=0 l=1 rev1=1 crypto rx=0x7fd11c070d60 tx=0x7fd10c02b910 comp rx=0 tx=0).stop 2024-02-21T23:16:21.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.061+0000 7fd121c00640 1 -- 172.21.15.28:0/1109426793 shutdown_connections 2024-02-21T23:16:21.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.061+0000 7fd121c00640 1 --2- 172.21.15.28:0/1109426793 >> v2:172.21.15.81:3300/0 conn(0x7fd11c155950 0x7fd11c0721d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:21.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.061+0000 7fd121c00640 1 --2- 172.21.15.28:0/1109426793 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd11c0707f0 0x7fd11c152560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:21.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.061+0000 7fd121c00640 1 --2- 172.21.15.28:0/1109426793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd11c14f170 0x7fd11c0a90e0 unknown :-1 s=CLOSED pgs=2620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:21.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.061+0000 7fd121c00640 1 -- 172.21.15.28:0/1109426793 >> 172.21.15.28:0/1109426793 conn(0x7fd11c0c7720 msgr2=0x7fd11c0c7b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:16:21.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.061+0000 7fd121c00640 1 -- 172.21.15.28:0/1109426793 shutdown_connections 2024-02-21T23:16:21.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.061+0000 7fd121c00640 1 -- 172.21.15.28:0/1109426793 wait complete. 2024-02-21T23:16:21.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.062+0000 7fd121c00640 1 Processor -- start 2024-02-21T23:16:21.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.063+0000 7fd121c00640 1 -- start start 2024-02-21T23:16:21.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.063+0000 7fd121c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd11c0707f0 0x7fd11c14c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:21.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.064+0000 7fd121c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd11c14f170 0x7fd11c1eb5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:21.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.064+0000 7fd121c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd11c155950 0x7fd11c15dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:21.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.064+0000 7fd121c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd11c155430 con 0x7fd11c14f170 2024-02-21T23:16:21.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.064+0000 7fd121c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd11c155150 con 0x7fd11c0707f0 2024-02-21T23:16:21.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.064+0000 7fd121c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd11c0a7b10 con 0x7fd11c155950 2024-02-21T23:16:21.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.064+0000 7fd113e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd11c14f170 0x7fd11c1eb5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:21.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.064+0000 7fd113e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd11c14f170 0x7fd11c1eb5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37564/0 (socket says 172.21.15.28:37564) 2024-02-21T23:16:21.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.064+0000 7fd113e00640 1 -- 172.21.15.28:0/3384690931 learned_addr learned my addr 172.21.15.28:0/3384690931 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:16:21.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.064+0000 7fd11be00640 1 --2- 172.21.15.28:0/3384690931 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd11c0707f0 0x7fd11c14c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:21.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.064+0000 7fd121200640 1 --2- 172.21.15.28:0/3384690931 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd11c155950 0x7fd11c15dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:21.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.065+0000 7fd113e00640 1 -- 172.21.15.28:0/3384690931 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd11c155950 msgr2=0x7fd11c15dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:21.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.065+0000 7fd113e00640 1 --2- 172.21.15.28:0/3384690931 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd11c155950 0x7fd11c15dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:21.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.065+0000 7fd113e00640 1 -- 172.21.15.28:0/3384690931 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd11c0707f0 msgr2=0x7fd11c14c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:21.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.065+0000 7fd113e00640 1 --2- 172.21.15.28:0/3384690931 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd11c0707f0 0x7fd11c14c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:21.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.065+0000 7fd113e00640 1 -- 172.21.15.28:0/3384690931 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd11c14e480 con 0x7fd11c14f170 2024-02-21T23:16:21.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.065+0000 7fd113e00640 1 --2- 172.21.15.28:0/3384690931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd11c14f170 0x7fd11c1eb5a0 secure :-1 s=READY pgs=2621 cs=0 l=1 rev1=1 crypto rx=0x7fd10400d7d0 tx=0x7fd10400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:21.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.065+0000 7fd118c00640 1 -- 172.21.15.28:0/3384690931 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1040096a0 con 0x7fd11c14f170 2024-02-21T23:16:21.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.065+0000 7fd118c00640 1 -- 172.21.15.28:0/3384690931 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd104019070 con 0x7fd11c14f170 2024-02-21T23:16:21.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.065+0000 7fd121c00640 1 -- 172.21.15.28:0/3384690931 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd11c14dec0 con 0x7fd11c14f170 2024-02-21T23:16:21.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.065+0000 7fd118c00640 1 -- 172.21.15.28:0/3384690931 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd104009a60 con 0x7fd11c14f170 2024-02-21T23:16:21.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.065+0000 7fd121c00640 1 -- 172.21.15.28:0/3384690931 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd11c1eada0 con 0x7fd11c14f170 2024-02-21T23:16:21.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.067+0000 7fd121c00640 1 -- 172.21.15.28:0/3384690931 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd11c04f290 con 0x7fd11c14f170 2024-02-21T23:16:21.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.067+0000 7fd118c00640 1 -- 172.21.15.28:0/3384690931 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd104009c00 con 0x7fd11c14f170 2024-02-21T23:16:21.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.068+0000 7fd118c00640 1 --2- 172.21.15.28:0/3384690931 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd0f806b4d0 0x7fd0f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:21.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.068+0000 7fd118c00640 1 -- 172.21.15.28:0/3384690931 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fd10408a950 con 0x7fd11c14f170 2024-02-21T23:16:21.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.069+0000 7fd11be00640 1 --2- 172.21.15.28:0/3384690931 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd0f806b4d0 0x7fd0f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:21.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.070+0000 7fd11be00640 1 --2- 172.21.15.28:0/3384690931 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd0f806b4d0 0x7fd0f806d990 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7fd10c036000 tx=0x7fd10c03c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:21.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.073+0000 7fd118c00640 1 -- 172.21.15.28:0/3384690931 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd1040594a0 con 0x7fd11c14f170 2024-02-21T23:16:21.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.312+0000 7fd121c00640 1 -- 172.21.15.28:0/3384690931 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd11c070560 con 0x7fd11c14f170 2024-02-21T23:16:21.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.313+0000 7fd118c00640 1 -- 172.21.15.28:0/3384690931 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7fd104058da0 con 0x7fd11c14f170 2024-02-21T23:16:21.300 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:16:21.300 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:16:21.301 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:16:21.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.315+0000 7fd121c00640 1 -- 172.21.15.28:0/3384690931 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd0f806b4d0 msgr2=0x7fd0f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:21.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.315+0000 7fd121c00640 1 --2- 172.21.15.28:0/3384690931 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd0f806b4d0 0x7fd0f806d990 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7fd10c036000 tx=0x7fd10c03c040 comp rx=0 tx=0).stop 2024-02-21T23:16:21.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.315+0000 7fd121c00640 1 -- 172.21.15.28:0/3384690931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd11c14f170 msgr2=0x7fd11c1eb5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:21.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.315+0000 7fd121c00640 1 --2- 172.21.15.28:0/3384690931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd11c14f170 0x7fd11c1eb5a0 secure :-1 s=READY pgs=2621 cs=0 l=1 rev1=1 crypto rx=0x7fd10400d7d0 tx=0x7fd10400dc90 comp rx=0 tx=0).stop 2024-02-21T23:16:21.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.316+0000 7fd121c00640 1 -- 172.21.15.28:0/3384690931 shutdown_connections 2024-02-21T23:16:21.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.316+0000 7fd121c00640 1 --2- 172.21.15.28:0/3384690931 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd0f806b4d0 0x7fd0f806d990 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:21.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.316+0000 7fd121c00640 1 --2- 172.21.15.28:0/3384690931 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd11c155950 0x7fd11c15dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:21.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.316+0000 7fd121c00640 1 --2- 172.21.15.28:0/3384690931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd11c14f170 0x7fd11c1eb5a0 unknown :-1 s=CLOSED pgs=2621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:21.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.316+0000 7fd121c00640 1 --2- 172.21.15.28:0/3384690931 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd11c0707f0 0x7fd11c14c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:21.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.316+0000 7fd121c00640 1 -- 172.21.15.28:0/3384690931 >> 172.21.15.28:0/3384690931 conn(0x7fd11c0c7720 msgr2=0x7fd11c149880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:16:21.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.316+0000 7fd121c00640 1 -- 172.21.15.28:0/3384690931 shutdown_connections 2024-02-21T23:16:21.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:21.316+0000 7fd121c00640 1 -- 172.21.15.28:0/3384690931 wait complete. 2024-02-21T23:16:21.725 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:16:21.725 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:16:22.727 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:16:22.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.968+0000 7fb78be00640 1 Processor -- start 2024-02-21T23:16:22.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.968+0000 7fb78be00640 1 -- start start 2024-02-21T23:16:22.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.970+0000 7fb78be00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb78c0aff90 0x7fb78c0c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:22.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.970+0000 7fb78be00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fb78c0c5720 0x7fb78c15f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:22.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.970+0000 7fb78be00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb78c0c74c0 con 0x7fb78c06dbf0 2024-02-21T23:16:22.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.970+0000 7fb78be00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb78c0c7270 con 0x7fb78c0c5720 2024-02-21T23:16:22.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.970+0000 7fb78be00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb78c158630 con 0x7fb78c0aff90 2024-02-21T23:16:22.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.970+0000 7fb78a000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb78c06dbf0 0x7fb78c158ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50298/0 (socket says 172.21.15.28:50298) 2024-02-21T23:16:22.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.970+0000 7fb78a000640 1 -- 172.21.15.28:0/140952286 learned_addr learned my addr 172.21.15.28:0/140952286 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:16:22.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.971+0000 7fb78b400640 1 --2- 172.21.15.28:0/140952286 >> v2:172.21.15.81:3300/0 conn(0x7fb78c0c5720 0x7fb78c15f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:22.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.971+0000 7fb78aa00640 1 --2- 172.21.15.28:0/140952286 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb78c0aff90 0x7fb78c0c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:22.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.971+0000 7fb789600640 1 -- 172.21.15.28:0/140952286 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 570450653 0 0) 0x7fb78c0c74c0 con 0x7fb78c06dbf0 2024-02-21T23:16:22.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.971+0000 7fb789600640 1 -- 172.21.15.28:0/140952286 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb770003520 con 0x7fb78c06dbf0 2024-02-21T23:16:22.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.971+0000 7fb789600640 1 -- 172.21.15.28:0/140952286 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 270995535 0 0) 0x7fb770003520 con 0x7fb78c06dbf0 2024-02-21T23:16:22.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.971+0000 7fb789600640 1 -- 172.21.15.28:0/140952286 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb78c0aff90 msgr2=0x7fb78c0c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:16:22.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.971+0000 7fb789600640 1 --2- 172.21.15.28:0/140952286 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb78c0aff90 0x7fb78c0c51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:22.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.971+0000 7fb789600640 1 -- 172.21.15.28:0/140952286 >> v2:172.21.15.81:3300/0 conn(0x7fb78c0c5720 msgr2=0x7fb78c15f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:16:22.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.971+0000 7fb789600640 1 --2- 172.21.15.28:0/140952286 >> v2:172.21.15.81:3300/0 conn(0x7fb78c0c5720 0x7fb78c15f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:22.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.971+0000 7fb789600640 1 -- 172.21.15.28:0/140952286 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb78c0c77a0 con 0x7fb78c06dbf0 2024-02-21T23:16:22.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.971+0000 7fb789600640 1 -- 172.21.15.28:0/140952286 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fb778001670 con 0x7fb78c06dbf0 2024-02-21T23:16:22.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.971+0000 7fb78aa00640 1 --2- 172.21.15.28:0/140952286 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb78c0aff90 0x7fb78c0c51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:16:22.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.972+0000 7fb789600640 1 -- 172.21.15.28:0/140952286 >> v1:172.21.15.28:6789/0 conn(0x7fb78c06dbf0 legacy=0x7fb78c158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:22.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.972+0000 7fb789600640 1 --2- 172.21.15.28:0/140952286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7700039b0 0x7fb770003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:22.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.972+0000 7fb789600640 1 --2- 172.21.15.28:0/140952286 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb770004800 0x7fb770004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:22.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.972+0000 7fb789600640 1 --2- 172.21.15.28:0/140952286 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb78c0aff90 0x7fb7700051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:22.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.972+0000 7fb789600640 1 -- 172.21.15.28:0/140952286 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb78c0c7270 con 0x7fb7700039b0 2024-02-21T23:16:22.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.972+0000 7fb789600640 1 -- 172.21.15.28:0/140952286 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb78c158630 con 0x7fb770004800 2024-02-21T23:16:22.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.972+0000 7fb789600640 1 -- 172.21.15.28:0/140952286 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb77000b9f0 con 0x7fb78c0aff90 2024-02-21T23:16:22.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.972+0000 7fb78aa00640 1 --2- 172.21.15.28:0/140952286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7700039b0 0x7fb770003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:22.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.972+0000 7fb78b400640 1 --2- 172.21.15.28:0/140952286 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb78c0aff90 0x7fb7700051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:22.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.972+0000 7fb78a000640 1 --2- 172.21.15.28:0/140952286 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb770004800 0x7fb770004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:22.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.973+0000 7fb78aa00640 1 -- 172.21.15.28:0/140952286 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb78c0aff90 msgr2=0x7fb7700051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:16:22.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.973+0000 7fb78aa00640 1 --2- 172.21.15.28:0/140952286 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb78c0aff90 0x7fb7700051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:22.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.973+0000 7fb78aa00640 1 -- 172.21.15.28:0/140952286 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb770004800 msgr2=0x7fb770004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:16:22.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.973+0000 7fb78aa00640 1 --2- 172.21.15.28:0/140952286 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb770004800 0x7fb770004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:22.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.973+0000 7fb78aa00640 1 -- 172.21.15.28:0/140952286 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb770003520 con 0x7fb7700039b0 2024-02-21T23:16:22.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.973+0000 7fb78a000640 1 --2- 172.21.15.28:0/140952286 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb770004800 0x7fb770004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:16:22.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.974+0000 7fb78aa00640 1 --2- 172.21.15.28:0/140952286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7700039b0 0x7fb770003dd0 secure :-1 s=READY pgs=2623 cs=0 l=1 rev1=1 crypto rx=0x7fb77c0098a0 tx=0x7fb770005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65dc2f7f1df56870 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:22.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.974+0000 7fb789600640 1 -- 172.21.15.28:0/140952286 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb77c00f040 con 0x7fb7700039b0 2024-02-21T23:16:22.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.974+0000 7fb789600640 1 -- 172.21.15.28:0/140952286 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb77c004230 con 0x7fb7700039b0 2024-02-21T23:16:22.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.974+0000 7fb789600640 1 -- 172.21.15.28:0/140952286 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb77c003610 con 0x7fb7700039b0 2024-02-21T23:16:22.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.974+0000 7fb78be00640 1 -- 172.21.15.28:0/140952286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7700039b0 msgr2=0x7fb770003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:22.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.974+0000 7fb78be00640 1 --2- 172.21.15.28:0/140952286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7700039b0 0x7fb770003dd0 secure :-1 s=READY pgs=2623 cs=0 l=1 rev1=1 crypto rx=0x7fb77c0098a0 tx=0x7fb770005830 comp rx=0 tx=0).stop 2024-02-21T23:16:22.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.974+0000 7fb78b400640 1 -- 172.21.15.28:0/140952286 reap_dead start 2024-02-21T23:16:22.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.975+0000 7fb78be00640 1 -- 172.21.15.28:0/140952286 shutdown_connections 2024-02-21T23:16:22.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.975+0000 7fb78be00640 1 -- 172.21.15.28:0/140952286 >> 172.21.15.28:0/140952286 conn(0x7fb78c0ac370 msgr2=0x7fb78c0ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:16:22.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.975+0000 7fb78be00640 1 -- 172.21.15.28:0/140952286 shutdown_connections 2024-02-21T23:16:22.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.975+0000 7fb78be00640 1 -- 172.21.15.28:0/140952286 wait complete. 2024-02-21T23:16:22.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.976+0000 7fb78be00640 1 Processor -- start 2024-02-21T23:16:22.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.977+0000 7fb78be00640 1 -- start start 2024-02-21T23:16:22.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.977+0000 7fb78be00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb7700039b0 0x7fb78c0aedc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:22.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.978+0000 7fb78be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb78c0c46d0 0x7fb78c0bfc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:22.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.978+0000 7fb78be00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb78c147940 0x7fb78c147dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:22.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.978+0000 7fb78be00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb78c0c8760 con 0x7fb78c0c46d0 2024-02-21T23:16:22.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.978+0000 7fb78be00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb78c0be0c0 con 0x7fb78c147940 2024-02-21T23:16:22.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.978+0000 7fb78be00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb78c0c88e0 con 0x7fb7700039b0 2024-02-21T23:16:22.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.978+0000 7fb78a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb78c0c46d0 0x7fb78c0bfc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:22.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.978+0000 7fb78a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb78c0c46d0 0x7fb78c0bfc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37578/0 (socket says 172.21.15.28:37578) 2024-02-21T23:16:22.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.978+0000 7fb78a000640 1 -- 172.21.15.28:0/4173687179 learned_addr learned my addr 172.21.15.28:0/4173687179 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:16:22.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.978+0000 7fb78aa00640 1 --2- 172.21.15.28:0/4173687179 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb7700039b0 0x7fb78c0aedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:22.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.978+0000 7fb78b400640 1 --2- 172.21.15.28:0/4173687179 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb78c147940 0x7fb78c147dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:22.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.978+0000 7fb78a000640 1 -- 172.21.15.28:0/4173687179 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb7700039b0 msgr2=0x7fb78c0aedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:22.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.978+0000 7fb78a000640 1 --2- 172.21.15.28:0/4173687179 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb7700039b0 0x7fb78c0aedc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:22.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.978+0000 7fb78a000640 1 -- 172.21.15.28:0/4173687179 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb78c147940 msgr2=0x7fb78c147dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:22.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.978+0000 7fb78a000640 1 --2- 172.21.15.28:0/4173687179 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb78c147940 0x7fb78c147dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:22.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.978+0000 7fb78a000640 1 -- 172.21.15.28:0/4173687179 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb78c0acc00 con 0x7fb78c0c46d0 2024-02-21T23:16:22.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.979+0000 7fb78a000640 1 --2- 172.21.15.28:0/4173687179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb78c0c46d0 0x7fb78c0bfc10 secure :-1 s=READY pgs=2624 cs=0 l=1 rev1=1 crypto rx=0x7fb77800d790 tx=0x7fb77800dc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:22.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.979+0000 7fb782a00640 1 -- 172.21.15.28:0/4173687179 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb778019070 con 0x7fb78c0c46d0 2024-02-21T23:16:22.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.979+0000 7fb782a00640 1 -- 172.21.15.28:0/4173687179 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7780074e0 con 0x7fb78c0c46d0 2024-02-21T23:16:22.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.979+0000 7fb782a00640 1 -- 172.21.15.28:0/4173687179 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb77800dea0 con 0x7fb78c0c46d0 2024-02-21T23:16:22.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.979+0000 7fb78be00640 1 -- 172.21.15.28:0/4173687179 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb78c0bef30 con 0x7fb78c0c46d0 2024-02-21T23:16:22.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.979+0000 7fb78be00640 1 -- 172.21.15.28:0/4173687179 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb78c0bf4b0 con 0x7fb78c0c46d0 2024-02-21T23:16:22.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.981+0000 7fb78be00640 1 -- 172.21.15.28:0/4173687179 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb78c1570d0 con 0x7fb78c0c46d0 2024-02-21T23:16:22.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.981+0000 7fb782a00640 1 -- 172.21.15.28:0/4173687179 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb7780049e0 con 0x7fb78c0c46d0 2024-02-21T23:16:22.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.983+0000 7fb782a00640 1 --2- 172.21.15.28:0/4173687179 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb76c06b4d0 0x7fb76c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:22.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.983+0000 7fb782a00640 1 -- 172.21.15.28:0/4173687179 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fb778089ef0 con 0x7fb78c0c46d0 2024-02-21T23:16:22.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.983+0000 7fb78aa00640 1 --2- 172.21.15.28:0/4173687179 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb76c06b4d0 0x7fb76c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:22.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.984+0000 7fb78aa00640 1 --2- 172.21.15.28:0/4173687179 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb76c06b4d0 0x7fb76c06d990 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7fb77c035000 tx=0x7fb77c03a040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:22.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:22.992+0000 7fb782a00640 1 -- 172.21.15.28:0/4173687179 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb78c1570d0 con 0x7fb78c0c46d0 2024-02-21T23:16:23.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:23.238+0000 7fb78be00640 1 -- 172.21.15.28:0/4173687179 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb78c04f290 con 0x7fb78c0c46d0 2024-02-21T23:16:23.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:23.239+0000 7fb782a00640 1 -- 172.21.15.28:0/4173687179 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7fb778058a50 con 0x7fb78c0c46d0 2024-02-21T23:16:23.227 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:16:23.227 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:16:23.227 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:16:23.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:23.242+0000 7fb78be00640 1 -- 172.21.15.28:0/4173687179 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb76c06b4d0 msgr2=0x7fb76c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:23.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:23.242+0000 7fb78be00640 1 --2- 172.21.15.28:0/4173687179 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb76c06b4d0 0x7fb76c06d990 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7fb77c035000 tx=0x7fb77c03a040 comp rx=0 tx=0).stop 2024-02-21T23:16:23.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:23.242+0000 7fb78be00640 1 -- 172.21.15.28:0/4173687179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb78c0c46d0 msgr2=0x7fb78c0bfc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:23.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:23.242+0000 7fb78be00640 1 --2- 172.21.15.28:0/4173687179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb78c0c46d0 0x7fb78c0bfc10 secure :-1 s=READY pgs=2624 cs=0 l=1 rev1=1 crypto rx=0x7fb77800d790 tx=0x7fb77800dc50 comp rx=0 tx=0).stop 2024-02-21T23:16:23.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:23.242+0000 7fb78be00640 1 -- 172.21.15.28:0/4173687179 shutdown_connections 2024-02-21T23:16:23.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:23.242+0000 7fb78be00640 1 --2- 172.21.15.28:0/4173687179 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb76c06b4d0 0x7fb76c06d990 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:23.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:23.242+0000 7fb78be00640 1 --2- 172.21.15.28:0/4173687179 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb78c147940 0x7fb78c147dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:23.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:23.242+0000 7fb78be00640 1 --2- 172.21.15.28:0/4173687179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb78c0c46d0 0x7fb78c0bfc10 unknown :-1 s=CLOSED pgs=2624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:23.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:23.242+0000 7fb78be00640 1 --2- 172.21.15.28:0/4173687179 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb7700039b0 0x7fb78c0aedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:23.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:23.242+0000 7fb78be00640 1 -- 172.21.15.28:0/4173687179 >> 172.21.15.28:0/4173687179 conn(0x7fb78c0ac370 msgr2=0x7fb78c0c8ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:16:23.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:23.242+0000 7fb78be00640 1 -- 172.21.15.28:0/4173687179 shutdown_connections 2024-02-21T23:16:23.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:23.242+0000 7fb78be00640 1 -- 172.21.15.28:0/4173687179 wait complete. 2024-02-21T23:16:23.656 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:16:23.656 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:16:24.657 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:16:24.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:24.898+0000 7f9dd8e00640 1 Processor -- start 2024-02-21T23:16:24.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:24.899+0000 7f9dd8e00640 1 -- start start 2024-02-21T23:16:24.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:24.901+0000 7f9dd8e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9dd40aff90 0x7f9dd40c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:24.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:24.901+0000 7f9dd8e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9dd40c5720 0x7f9dd415f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:24.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:24.901+0000 7f9dd8e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dd40c74c0 con 0x7f9dd406dbf0 2024-02-21T23:16:24.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:24.901+0000 7f9dd8e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9dd40c7270 con 0x7f9dd40aff90 2024-02-21T23:16:24.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:24.901+0000 7f9dd8e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd4158630 con 0x7f9dd40c5720 2024-02-21T23:16:24.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:24.901+0000 7f9dd3400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9dd40aff90 0x7f9dd40c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:24.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:24.901+0000 7f9dd3400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9dd40aff90 0x7f9dd40c51e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59260/0 (socket says 172.21.15.28:59260) 2024-02-21T23:16:24.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:24.901+0000 7f9dd3400640 1 -- 172.21.15.28:0/3957398068 learned_addr learned my addr 172.21.15.28:0/3957398068 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:16:24.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:24.902+0000 7f9dd3e00640 1 --2- 172.21.15.28:0/3957398068 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9dd40c5720 0x7f9dd415f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:24.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:24.902+0000 7f9dd3400640 1 -- 172.21.15.28:0/3957398068 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9dd40c5720 msgr2=0x7f9dd415f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:16:24.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:24.902+0000 7f9dd3400640 1 --2- 172.21.15.28:0/3957398068 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9dd40c5720 0x7f9dd415f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:24.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:24.902+0000 7f9dd3400640 1 -- 172.21.15.28:0/3957398068 >> v1:172.21.15.81:6789/0 conn(0x7f9dd406dbf0 legacy=0x7f9dd4158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:16:24.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:24.902+0000 7f9dd3400640 1 -- 172.21.15.28:0/3957398068 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dd40c77a0 con 0x7f9dd40aff90 2024-02-21T23:16:24.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:24.903+0000 7f9dd3400640 1 --2- 172.21.15.28:0/3957398068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9dd40aff90 0x7f9dd40c51e0 secure :-1 s=READY pgs=2625 cs=0 l=1 rev1=1 crypto rx=0x7f9dc4009870 tx=0x7f9dc402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8c5044b1a2858fc server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:24.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:24.903+0000 7f9dd2a00640 1 -- 172.21.15.28:0/3957398068 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc4034070 con 0x7f9dd40aff90 2024-02-21T23:16:24.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:24.903+0000 7f9dd2a00640 1 -- 172.21.15.28:0/3957398068 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dc4030840 con 0x7f9dd40aff90 2024-02-21T23:16:24.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:24.903+0000 7f9dd2a00640 1 -- 172.21.15.28:0/3957398068 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc4030c10 con 0x7f9dd40aff90 2024-02-21T23:16:24.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:24.904+0000 7f9dd8e00640 1 -- 172.21.15.28:0/3957398068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9dd40aff90 msgr2=0x7f9dd40c51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:24.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:24.904+0000 7f9dd8e00640 1 --2- 172.21.15.28:0/3957398068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9dd40aff90 0x7f9dd40c51e0 secure :-1 s=READY pgs=2625 cs=0 l=1 rev1=1 crypto rx=0x7f9dc4009870 tx=0x7f9dc402b980 comp rx=0 tx=0).stop 2024-02-21T23:16:24.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:24.904+0000 7f9dd8e00640 1 -- 172.21.15.28:0/3957398068 shutdown_connections 2024-02-21T23:16:24.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:24.904+0000 7f9dd8e00640 1 --2- 172.21.15.28:0/3957398068 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9dd40c5720 0x7f9dd415f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:24.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:24.904+0000 7f9dd8e00640 1 --2- 172.21.15.28:0/3957398068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9dd40aff90 0x7f9dd40c51e0 unknown :-1 s=CLOSED pgs=2625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:24.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:24.904+0000 7f9dd8e00640 1 -- 172.21.15.28:0/3957398068 >> 172.21.15.28:0/3957398068 conn(0x7f9dd40ac370 msgr2=0x7f9dd40ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:16:24.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:24.904+0000 7f9dd8e00640 1 -- 172.21.15.28:0/3957398068 shutdown_connections 2024-02-21T23:16:24.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:24.904+0000 7f9dd8e00640 1 -- 172.21.15.28:0/3957398068 wait complete. 2024-02-21T23:16:24.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:24.906+0000 7f9dd8e00640 1 Processor -- start 2024-02-21T23:16:24.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:24.906+0000 7f9dd8e00640 1 -- start start 2024-02-21T23:16:24.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:24.908+0000 7f9dd8e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9dd406dbf0 0x7f9dd41edf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:24.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:24.908+0000 7f9dd8e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9dd40aff90 0x7f9dd40ae700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:24.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:24.908+0000 7f9dd8e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9dd40c5720 0x7f9dd414ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:24.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:24.908+0000 7f9dd8e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd40c8760 con 0x7f9dd40aff90 2024-02-21T23:16:24.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:24.908+0000 7f9dd8e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd40be0c0 con 0x7f9dd40c5720 2024-02-21T23:16:24.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:24.908+0000 7f9dd8e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd40c88e0 con 0x7f9dd406dbf0 2024-02-21T23:16:24.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:24.908+0000 7f9dcbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9dd40aff90 0x7f9dd40ae700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:24.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:24.908+0000 7f9dcbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9dd40aff90 0x7f9dd40ae700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59264/0 (socket says 172.21.15.28:59264) 2024-02-21T23:16:24.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:24.908+0000 7f9dcbe00640 1 -- 172.21.15.28:0/2534673741 learned_addr learned my addr 172.21.15.28:0/2534673741 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:16:24.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:24.909+0000 7f9dd3e00640 1 --2- 172.21.15.28:0/2534673741 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9dd40c5720 0x7f9dd414ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:24.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:24.909+0000 7f9dd3400640 1 --2- 172.21.15.28:0/2534673741 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9dd406dbf0 0x7f9dd41edf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:24.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:24.909+0000 7f9dcbe00640 1 -- 172.21.15.28:0/2534673741 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9dd406dbf0 msgr2=0x7f9dd41edf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:24.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:24.909+0000 7f9dcbe00640 1 --2- 172.21.15.28:0/2534673741 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9dd406dbf0 0x7f9dd41edf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:24.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:24.909+0000 7f9dcbe00640 1 -- 172.21.15.28:0/2534673741 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9dd40c5720 msgr2=0x7f9dd414ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:24.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:24.909+0000 7f9dcbe00640 1 --2- 172.21.15.28:0/2534673741 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9dd40c5720 0x7f9dd414ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:24.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:24.909+0000 7f9dcbe00640 1 -- 172.21.15.28:0/2534673741 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dd41ee9a0 con 0x7f9dd40aff90 2024-02-21T23:16:24.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:24.909+0000 7f9dcbe00640 1 --2- 172.21.15.28:0/2534673741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9dd40aff90 0x7f9dd40ae700 secure :-1 s=READY pgs=2626 cs=0 l=1 rev1=1 crypto rx=0x7f9dbc00c440 tx=0x7f9dbc001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:24.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:24.910+0000 7f9dcb400640 1 -- 172.21.15.28:0/2534673741 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dbc0052d0 con 0x7f9dd40aff90 2024-02-21T23:16:24.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:24.910+0000 7f9dcb400640 1 -- 172.21.15.28:0/2534673741 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dbc005470 con 0x7f9dd40aff90 2024-02-21T23:16:24.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:24.910+0000 7f9dcb400640 1 -- 172.21.15.28:0/2534673741 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dbc007240 con 0x7f9dd40aff90 2024-02-21T23:16:24.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:24.910+0000 7f9dd8e00640 1 -- 172.21.15.28:0/2534673741 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dd40ae2b0 con 0x7f9dd40aff90 2024-02-21T23:16:24.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:24.910+0000 7f9dd8e00640 1 -- 172.21.15.28:0/2534673741 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dd41e11e0 con 0x7f9dd40aff90 2024-02-21T23:16:24.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:24.911+0000 7f9dd8e00640 1 -- 172.21.15.28:0/2534673741 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dd404f290 con 0x7f9dd40aff90 2024-02-21T23:16:24.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:24.912+0000 7f9dcb400640 1 -- 172.21.15.28:0/2534673741 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9dbc0073e0 con 0x7f9dd40aff90 2024-02-21T23:16:24.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:24.913+0000 7f9dcb400640 1 --2- 172.21.15.28:0/2534673741 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9db006b4d0 0x7f9db006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:24.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:24.913+0000 7f9dcb400640 1 -- 172.21.15.28:0/2534673741 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f9dbc089910 con 0x7f9dd40aff90 2024-02-21T23:16:24.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:24.913+0000 7f9dd3400640 1 --2- 172.21.15.28:0/2534673741 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9db006b4d0 0x7f9db006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:24.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:24.915+0000 7f9dd3400640 1 --2- 172.21.15.28:0/2534673741 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9db006b4d0 0x7f9db006d990 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f9dc4036000 tx=0x7f9dc403b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:24.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:24.921+0000 7f9dcb400640 1 -- 172.21.15.28:0/2534673741 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f9dbc0584f0 con 0x7f9dd40aff90 2024-02-21T23:16:25.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:25.184+0000 7f9dd8e00640 1 -- 172.21.15.28:0/2534673741 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9dd406b4a0 con 0x7f9dd40aff90 2024-02-21T23:16:25.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:25.185+0000 7f9dcb400640 1 -- 172.21.15.28:0/2534673741 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7f9dbc057df0 con 0x7f9dd40aff90 2024-02-21T23:16:25.174 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:16:25.174 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:16:25.174 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:16:25.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:25.188+0000 7f9dd8e00640 1 -- 172.21.15.28:0/2534673741 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9db006b4d0 msgr2=0x7f9db006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:25.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:25.188+0000 7f9dd8e00640 1 --2- 172.21.15.28:0/2534673741 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9db006b4d0 0x7f9db006d990 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f9dc4036000 tx=0x7f9dc403b040 comp rx=0 tx=0).stop 2024-02-21T23:16:25.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:25.188+0000 7f9dd8e00640 1 -- 172.21.15.28:0/2534673741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9dd40aff90 msgr2=0x7f9dd40ae700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:25.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:25.188+0000 7f9dd8e00640 1 --2- 172.21.15.28:0/2534673741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9dd40aff90 0x7f9dd40ae700 secure :-1 s=READY pgs=2626 cs=0 l=1 rev1=1 crypto rx=0x7f9dbc00c440 tx=0x7f9dbc001e70 comp rx=0 tx=0).stop 2024-02-21T23:16:25.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:25.188+0000 7f9dd8e00640 1 -- 172.21.15.28:0/2534673741 shutdown_connections 2024-02-21T23:16:25.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:25.188+0000 7f9dd8e00640 1 --2- 172.21.15.28:0/2534673741 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9db006b4d0 0x7f9db006d990 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:25.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:25.188+0000 7f9dd8e00640 1 --2- 172.21.15.28:0/2534673741 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9dd40c5720 0x7f9dd414ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:25.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:25.188+0000 7f9dd8e00640 1 --2- 172.21.15.28:0/2534673741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9dd40aff90 0x7f9dd40ae700 unknown :-1 s=CLOSED pgs=2626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:25.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:25.188+0000 7f9dd8e00640 1 --2- 172.21.15.28:0/2534673741 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9dd406dbf0 0x7f9dd41edf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:25.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:25.188+0000 7f9dd8e00640 1 -- 172.21.15.28:0/2534673741 >> 172.21.15.28:0/2534673741 conn(0x7f9dd40ac370 msgr2=0x7f9dd40ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:16:25.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:25.188+0000 7f9dd8e00640 1 -- 172.21.15.28:0/2534673741 shutdown_connections 2024-02-21T23:16:25.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:25.188+0000 7f9dd8e00640 1 -- 172.21.15.28:0/2534673741 wait complete. 2024-02-21T23:16:25.602 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:16:25.602 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:16:26.603 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:16:26.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:26.846+0000 7f1350a00640 1 Processor -- start 2024-02-21T23:16:26.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:26.846+0000 7f1350a00640 1 -- start start 2024-02-21T23:16:26.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:26.848+0000 7f1350a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f134c156d90 0x7f134c06be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:26.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:26.848+0000 7f1350a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f134c156360 0x7f134c1534a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:26.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:26.848+0000 7f1350a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f134c0aca70 con 0x7f134c0ac020 2024-02-21T23:16:26.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:26.848+0000 7f1350a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f134c0ac820 con 0x7f134c156360 2024-02-21T23:16:26.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:26.848+0000 7f1350a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f134c0acd50 con 0x7f134c156d90 2024-02-21T23:16:26.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:26.849+0000 7f1342a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f134c156360 0x7f134c1534a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:26.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:26.849+0000 7f1342a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f134c156360 0x7f134c1534a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59272/0 (socket says 172.21.15.28:59272) 2024-02-21T23:16:26.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:26.849+0000 7f1342a00640 1 -- 172.21.15.28:0/821061364 learned_addr learned my addr 172.21.15.28:0/821061364 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:16:26.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:26.849+0000 7f134b400640 1 --2- 172.21.15.28:0/821061364 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f134c156d90 0x7f134c06be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:26.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:26.849+0000 7f1342a00640 1 -- 172.21.15.28:0/821061364 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f134c156d90 msgr2=0x7f134c06be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:16:26.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:26.849+0000 7f1342a00640 1 --2- 172.21.15.28:0/821061364 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f134c156d90 0x7f134c06be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:26.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:26.849+0000 7f1342a00640 1 -- 172.21.15.28:0/821061364 >> v1:172.21.15.81:6789/0 conn(0x7f134c0ac020 legacy=0x7f134c06e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:26.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:26.849+0000 7f1342a00640 1 -- 172.21.15.28:0/821061364 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f134c152f80 con 0x7f134c156360 2024-02-21T23:16:26.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:26.851+0000 7f1342a00640 1 --2- 172.21.15.28:0/821061364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f134c156360 0x7f134c1534a0 secure :-1 s=READY pgs=2627 cs=0 l=1 rev1=1 crypto rx=0x7f1334009e50 tx=0x7f133402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccef595d37edce95 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:26.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:26.852+0000 7f134aa00640 1 -- 172.21.15.28:0/821061364 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f133402c9f0 con 0x7f134c156360 2024-02-21T23:16:26.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:26.852+0000 7f134aa00640 1 -- 172.21.15.28:0/821061364 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f133402cb90 con 0x7f134c156360 2024-02-21T23:16:26.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:26.852+0000 7f134aa00640 1 -- 172.21.15.28:0/821061364 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13340343f0 con 0x7f134c156360 2024-02-21T23:16:26.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:26.852+0000 7f1350a00640 1 -- 172.21.15.28:0/821061364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f134c156360 msgr2=0x7f134c1534a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:26.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:26.852+0000 7f1350a00640 1 --2- 172.21.15.28:0/821061364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f134c156360 0x7f134c1534a0 secure :-1 s=READY pgs=2627 cs=0 l=1 rev1=1 crypto rx=0x7f1334009e50 tx=0x7f133402be50 comp rx=0 tx=0).stop 2024-02-21T23:16:26.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:26.852+0000 7f1350a00640 1 -- 172.21.15.28:0/821061364 shutdown_connections 2024-02-21T23:16:26.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:26.852+0000 7f1350a00640 1 --2- 172.21.15.28:0/821061364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f134c156360 0x7f134c1534a0 unknown :-1 s=CLOSED pgs=2627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:26.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:26.852+0000 7f1350a00640 1 --2- 172.21.15.28:0/821061364 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f134c156d90 0x7f134c06be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:26.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:26.852+0000 7f1350a00640 1 -- 172.21.15.28:0/821061364 >> 172.21.15.28:0/821061364 conn(0x7f134c071910 msgr2=0x7f134c071cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:16:26.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:26.853+0000 7f1350a00640 1 -- 172.21.15.28:0/821061364 shutdown_connections 2024-02-21T23:16:26.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:26.853+0000 7f1350a00640 1 -- 172.21.15.28:0/821061364 wait complete. 2024-02-21T23:16:26.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:26.854+0000 7f1350a00640 1 Processor -- start 2024-02-21T23:16:26.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:26.854+0000 7f1350a00640 1 -- start start 2024-02-21T23:16:26.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:26.855+0000 7f1350a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f134c0ac020 0x7f134c14b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:26.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:26.855+0000 7f1350a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f134c156360 0x7f134c1497d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:26.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:26.855+0000 7f1350a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f134c156d90 0x7f134c1540f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:26.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:26.855+0000 7f1350a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f134c1572f0 con 0x7f134c156360 2024-02-21T23:16:26.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:26.855+0000 7f1350a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f134c157170 con 0x7f134c156d90 2024-02-21T23:16:26.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:26.855+0000 7f1350a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f134c06c3c0 con 0x7f134c0ac020 2024-02-21T23:16:26.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:26.856+0000 7f1342a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f134c156360 0x7f134c1497d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:26.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:26.856+0000 7f1342a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f134c156360 0x7f134c1497d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59280/0 (socket says 172.21.15.28:59280) 2024-02-21T23:16:26.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:26.856+0000 7f1342a00640 1 -- 172.21.15.28:0/4146920280 learned_addr learned my addr 172.21.15.28:0/4146920280 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:16:26.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:26.856+0000 7f134b400640 1 --2- 172.21.15.28:0/4146920280 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f134c0ac020 0x7f134c14b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:26.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:26.856+0000 7f134be00640 1 --2- 172.21.15.28:0/4146920280 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f134c156d90 0x7f134c1540f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:26.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:26.856+0000 7f1342a00640 1 -- 172.21.15.28:0/4146920280 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f134c0ac020 msgr2=0x7f134c14b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:26.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:26.856+0000 7f1342a00640 1 --2- 172.21.15.28:0/4146920280 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f134c0ac020 0x7f134c14b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:26.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:26.856+0000 7f1342a00640 1 -- 172.21.15.28:0/4146920280 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f134c156d90 msgr2=0x7f134c1540f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:26.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:26.856+0000 7f1342a00640 1 --2- 172.21.15.28:0/4146920280 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f134c156d90 0x7f134c1540f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:26.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:26.856+0000 7f1342a00640 1 -- 172.21.15.28:0/4146920280 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f134c149d10 con 0x7f134c156360 2024-02-21T23:16:26.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:26.857+0000 7f1342a00640 1 --2- 172.21.15.28:0/4146920280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f134c156360 0x7f134c1497d0 secure :-1 s=READY pgs=2628 cs=0 l=1 rev1=1 crypto rx=0x7f1334036040 tx=0x7f1334004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:26.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:26.857+0000 7f1343e00640 1 -- 172.21.15.28:0/4146920280 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1334044020 con 0x7f134c156360 2024-02-21T23:16:26.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:26.857+0000 7f1343e00640 1 -- 172.21.15.28:0/4146920280 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1334040470 con 0x7f134c156360 2024-02-21T23:16:26.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:26.857+0000 7f1343e00640 1 -- 172.21.15.28:0/4146920280 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f133402c9f0 con 0x7f134c156360 2024-02-21T23:16:26.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:26.857+0000 7f1350a00640 1 -- 172.21.15.28:0/4146920280 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f134c0c62e0 con 0x7f134c156360 2024-02-21T23:16:26.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:26.857+0000 7f1350a00640 1 -- 172.21.15.28:0/4146920280 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f134c0c6000 con 0x7f134c156360 2024-02-21T23:16:26.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:26.859+0000 7f1350a00640 1 -- 172.21.15.28:0/4146920280 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f134c06c1e0 con 0x7f134c156360 2024-02-21T23:16:26.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:26.859+0000 7f1343e00640 1 -- 172.21.15.28:0/4146920280 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1334047020 con 0x7f134c156360 2024-02-21T23:16:26.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:26.860+0000 7f1343e00640 1 --2- 172.21.15.28:0/4146920280 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f132806b4d0 0x7f132806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:26.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:26.860+0000 7f1343e00640 1 -- 172.21.15.28:0/4146920280 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f13340acf90 con 0x7f134c156360 2024-02-21T23:16:26.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:26.861+0000 7f134b400640 1 --2- 172.21.15.28:0/4146920280 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f132806b4d0 0x7f132806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:26.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:26.862+0000 7f134b400640 1 --2- 172.21.15.28:0/4146920280 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f132806b4d0 0x7f132806d990 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f134c0c2f00 tx=0x7f133c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:26.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:26.864+0000 7f1343e00640 1 -- 172.21.15.28:0/4146920280 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f134c06c1e0 con 0x7f134c156360 2024-02-21T23:16:27.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:27.146+0000 7f1350a00640 1 -- 172.21.15.28:0/4146920280 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f134c04f290 con 0x7f134c156360 2024-02-21T23:16:27.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:27.147+0000 7f1343e00640 1 -- 172.21.15.28:0/4146920280 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7f133404e030 con 0x7f134c156360 2024-02-21T23:16:27.135 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:16:27.135 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:16:27.135 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:16:27.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:27.150+0000 7f1350a00640 1 -- 172.21.15.28:0/4146920280 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f132806b4d0 msgr2=0x7f132806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:27.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:27.150+0000 7f1350a00640 1 --2- 172.21.15.28:0/4146920280 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f132806b4d0 0x7f132806d990 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f134c0c2f00 tx=0x7f133c008040 comp rx=0 tx=0).stop 2024-02-21T23:16:27.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:27.150+0000 7f1350a00640 1 -- 172.21.15.28:0/4146920280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f134c156360 msgr2=0x7f134c1497d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:27.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:27.150+0000 7f1350a00640 1 --2- 172.21.15.28:0/4146920280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f134c156360 0x7f134c1497d0 secure :-1 s=READY pgs=2628 cs=0 l=1 rev1=1 crypto rx=0x7f1334036040 tx=0x7f1334004a40 comp rx=0 tx=0).stop 2024-02-21T23:16:27.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:27.150+0000 7f1350a00640 1 -- 172.21.15.28:0/4146920280 shutdown_connections 2024-02-21T23:16:27.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:27.150+0000 7f1350a00640 1 --2- 172.21.15.28:0/4146920280 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f132806b4d0 0x7f132806d990 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:27.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:27.150+0000 7f1350a00640 1 --2- 172.21.15.28:0/4146920280 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f134c156d90 0x7f134c1540f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:27.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:27.150+0000 7f1350a00640 1 --2- 172.21.15.28:0/4146920280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f134c156360 0x7f134c1497d0 unknown :-1 s=CLOSED pgs=2628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:27.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:27.150+0000 7f1350a00640 1 --2- 172.21.15.28:0/4146920280 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f134c0ac020 0x7f134c14b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:27.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:27.150+0000 7f1350a00640 1 -- 172.21.15.28:0/4146920280 >> 172.21.15.28:0/4146920280 conn(0x7f134c071910 msgr2=0x7f134c1533a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:16:27.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:27.150+0000 7f1350a00640 1 -- 172.21.15.28:0/4146920280 shutdown_connections 2024-02-21T23:16:27.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:27.150+0000 7f1350a00640 1 -- 172.21.15.28:0/4146920280 wait complete. 2024-02-21T23:16:27.562 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:16:27.562 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:16:28.563 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:16:28.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:28.809+0000 7f767f200640 1 Processor -- start 2024-02-21T23:16:28.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:28.809+0000 7f767f200640 1 -- start start 2024-02-21T23:16:28.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:28.810+0000 7f767f200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7680150450 0x7f76800a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:28.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:28.810+0000 7f767f200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76800c2760 con 0x7f7680156af0 2024-02-21T23:16:28.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:28.810+0000 7f767f200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7680156310 con 0x7f768006ee40 2024-02-21T23:16:28.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:28.810+0000 7f767f200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76800c22c0 con 0x7f7680150450 2024-02-21T23:16:28.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:28.811+0000 7f767de00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7680150450 0x7f76800a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:28.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:28.811+0000 7f767e800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f7680156af0 0x7f768006dff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:48536/0 (socket says 172.21.15.28:48536) 2024-02-21T23:16:28.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:28.811+0000 7f767e800640 1 -- 172.21.15.28:0/3393472711 learned_addr learned my addr 172.21.15.28:0/3393472711 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:16:28.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:28.811+0000 7f767de00640 1 -- 172.21.15.28:0/3393472711 >> v1:172.21.15.81:6789/0 conn(0x7f768006ee40 legacy=0x7f76801536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:16:28.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:28.811+0000 7f767de00640 1 -- 172.21.15.28:0/3393472711 >> v1:172.21.15.28:6789/0 conn(0x7f7680156af0 legacy=0x7f768006dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:28.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:28.811+0000 7f767de00640 1 -- 172.21.15.28:0/3393472711 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76800c29d0 con 0x7f7680150450 2024-02-21T23:16:28.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:28.811+0000 7f767ca00640 1 -- 172.21.15.28:0/3393472711 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 776442475 0 0) 0x7f76800c2760 con 0x7f7680156af0 2024-02-21T23:16:28.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:28.812+0000 7f767de00640 1 --2- 172.21.15.28:0/3393472711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7680150450 0x7f76800a5fd0 secure :-1 s=READY pgs=2630 cs=0 l=1 rev1=1 crypto rx=0x7f7670009d90 tx=0x7f767002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb3816ad51d423e2 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:28.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:28.813+0000 7f767ca00640 1 -- 172.21.15.28:0/3393472711 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7670039070 con 0x7f7680150450 2024-02-21T23:16:28.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:28.813+0000 7f767ca00640 1 -- 172.21.15.28:0/3393472711 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f767002cc60 con 0x7f7680150450 2024-02-21T23:16:28.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:28.813+0000 7f767ca00640 1 -- 172.21.15.28:0/3393472711 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7670034660 con 0x7f7680150450 2024-02-21T23:16:28.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:28.813+0000 7f767f200640 1 -- 172.21.15.28:0/3393472711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7680150450 msgr2=0x7f76800a5fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:28.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:28.813+0000 7f767f200640 1 --2- 172.21.15.28:0/3393472711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7680150450 0x7f76800a5fd0 secure :-1 s=READY pgs=2630 cs=0 l=1 rev1=1 crypto rx=0x7f7670009d90 tx=0x7f767002beb0 comp rx=0 tx=0).stop 2024-02-21T23:16:28.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:28.813+0000 7f767f200640 1 -- 172.21.15.28:0/3393472711 shutdown_connections 2024-02-21T23:16:28.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:28.813+0000 7f767f200640 1 --2- 172.21.15.28:0/3393472711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7680150450 0x7f76800a5fd0 unknown :-1 s=CLOSED pgs=2630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:28.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:28.813+0000 7f767f200640 1 -- 172.21.15.28:0/3393472711 >> 172.21.15.28:0/3393472711 conn(0x7f768006b630 msgr2=0x7f76800c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:16:28.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:28.813+0000 7f767f200640 1 -- 172.21.15.28:0/3393472711 shutdown_connections 2024-02-21T23:16:28.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:28.813+0000 7f767f200640 1 -- 172.21.15.28:0/3393472711 wait complete. 2024-02-21T23:16:28.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:28.815+0000 7f767f200640 1 Processor -- start 2024-02-21T23:16:28.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:28.815+0000 7f767f200640 1 -- start start 2024-02-21T23:16:28.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:28.816+0000 7f767f200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f768006ee40 0x7f76800aba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:28.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:28.816+0000 7f767f200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7680150450 0x7f768014cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:28.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:28.816+0000 7f767f200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7680156af0 0x7f76801eaca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:28.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:28.816+0000 7f767f200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f768006e700 con 0x7f7680150450 2024-02-21T23:16:28.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:28.816+0000 7f767f200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7680156660 con 0x7f768006ee40 2024-02-21T23:16:28.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:28.816+0000 7f767f200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f768006e880 con 0x7f7680156af0 2024-02-21T23:16:28.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:28.816+0000 7f767d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7680150450 0x7f768014cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:28.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:28.816+0000 7f767d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7680150450 0x7f768014cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59286/0 (socket says 172.21.15.28:59286) 2024-02-21T23:16:28.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:28.816+0000 7f767d400640 1 -- 172.21.15.28:0/2288192622 learned_addr learned my addr 172.21.15.28:0/2288192622 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:16:28.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:28.817+0000 7f767de00640 1 --2- 172.21.15.28:0/2288192622 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f768006ee40 0x7f76800aba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:28.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:28.817+0000 7f767e800640 1 --2- 172.21.15.28:0/2288192622 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7680156af0 0x7f76801eaca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:28.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:28.817+0000 7f767d400640 1 -- 172.21.15.28:0/2288192622 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7680156af0 msgr2=0x7f76801eaca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:28.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:28.817+0000 7f767d400640 1 --2- 172.21.15.28:0/2288192622 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7680156af0 0x7f76801eaca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:28.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:28.817+0000 7f767d400640 1 -- 172.21.15.28:0/2288192622 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f768006ee40 msgr2=0x7f76800aba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:28.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:28.817+0000 7f767d400640 1 --2- 172.21.15.28:0/2288192622 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f768006ee40 0x7f76800aba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:28.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:28.817+0000 7f767d400640 1 -- 172.21.15.28:0/2288192622 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76801eeb20 con 0x7f7680150450 2024-02-21T23:16:28.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:28.817+0000 7f767de00640 1 --2- 172.21.15.28:0/2288192622 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f768006ee40 0x7f76800aba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:16:28.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:28.817+0000 7f767d400640 1 --2- 172.21.15.28:0/2288192622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7680150450 0x7f768014cac0 secure :-1 s=READY pgs=2631 cs=0 l=1 rev1=1 crypto rx=0x7f766c00c430 tx=0x7f766c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:28.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:28.817+0000 7f7676000640 1 -- 172.21.15.28:0/2288192622 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f766c011790 con 0x7f7680150450 2024-02-21T23:16:28.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:28.818+0000 7f7676000640 1 -- 172.21.15.28:0/2288192622 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f766c017070 con 0x7f7680150450 2024-02-21T23:16:28.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:28.818+0000 7f7676000640 1 -- 172.21.15.28:0/2288192622 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f766c011bb0 con 0x7f7680150450 2024-02-21T23:16:28.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:28.818+0000 7f767f200640 1 -- 172.21.15.28:0/2288192622 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7680071d60 con 0x7f7680150450 2024-02-21T23:16:28.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:28.818+0000 7f767f200640 1 -- 172.21.15.28:0/2288192622 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f768014d000 con 0x7f7680150450 2024-02-21T23:16:28.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:28.819+0000 7f767f200640 1 -- 172.21.15.28:0/2288192622 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76801ee8b0 con 0x7f7680150450 2024-02-21T23:16:28.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:28.820+0000 7f7676000640 1 -- 172.21.15.28:0/2288192622 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f766c01c460 con 0x7f7680150450 2024-02-21T23:16:28.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:28.821+0000 7f7676000640 1 --2- 172.21.15.28:0/2288192622 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f766006b4d0 0x7f766006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:28.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:28.821+0000 7f7676000640 1 -- 172.21.15.28:0/2288192622 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f766c08a3d0 con 0x7f7680150450 2024-02-21T23:16:28.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:28.821+0000 7f767de00640 1 --2- 172.21.15.28:0/2288192622 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f766006b4d0 0x7f766006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:28.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:28.822+0000 7f767de00640 1 --2- 172.21.15.28:0/2288192622 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f766006b4d0 0x7f766006d990 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f7670036000 tx=0x7f7670030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:28.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:28.831+0000 7f7676000640 1 -- 172.21.15.28:0/2288192622 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f766c058fb0 con 0x7f7680150450 2024-02-21T23:16:29.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:29.072+0000 7f767f200640 1 -- 172.21.15.28:0/2288192622 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f768004f290 con 0x7f7680150450 2024-02-21T23:16:29.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:29.072+0000 7f7676000640 1 -- 172.21.15.28:0/2288192622 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7f766c0588b0 con 0x7f7680150450 2024-02-21T23:16:29.061 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:16:29.061 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:16:29.061 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:16:29.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:29.075+0000 7f767f200640 1 -- 172.21.15.28:0/2288192622 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f766006b4d0 msgr2=0x7f766006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:29.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:29.075+0000 7f767f200640 1 --2- 172.21.15.28:0/2288192622 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f766006b4d0 0x7f766006d990 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f7670036000 tx=0x7f7670030000 comp rx=0 tx=0).stop 2024-02-21T23:16:29.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:29.075+0000 7f767f200640 1 -- 172.21.15.28:0/2288192622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7680150450 msgr2=0x7f768014cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:29.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:29.075+0000 7f767f200640 1 --2- 172.21.15.28:0/2288192622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7680150450 0x7f768014cac0 secure :-1 s=READY pgs=2631 cs=0 l=1 rev1=1 crypto rx=0x7f766c00c430 tx=0x7f766c00c8f0 comp rx=0 tx=0).stop 2024-02-21T23:16:29.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:29.075+0000 7f767f200640 1 -- 172.21.15.28:0/2288192622 shutdown_connections 2024-02-21T23:16:29.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:29.075+0000 7f767f200640 1 --2- 172.21.15.28:0/2288192622 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f766006b4d0 0x7f766006d990 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:29.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:29.075+0000 7f767f200640 1 --2- 172.21.15.28:0/2288192622 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7680156af0 0x7f76801eaca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:29.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:29.075+0000 7f767f200640 1 --2- 172.21.15.28:0/2288192622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7680150450 0x7f768014cac0 unknown :-1 s=CLOSED pgs=2631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:29.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:29.075+0000 7f767f200640 1 --2- 172.21.15.28:0/2288192622 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f768006ee40 0x7f76800aba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:29.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:29.075+0000 7f767f200640 1 -- 172.21.15.28:0/2288192622 >> 172.21.15.28:0/2288192622 conn(0x7f768006b630 msgr2=0x7f768014f060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:16:29.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:29.075+0000 7f767f200640 1 -- 172.21.15.28:0/2288192622 shutdown_connections 2024-02-21T23:16:29.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:29.076+0000 7f767f200640 1 -- 172.21.15.28:0/2288192622 wait complete. 2024-02-21T23:16:29.488 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:16:29.488 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:16:30.489 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:16:30.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.725+0000 7f2ad0600640 1 Processor -- start 2024-02-21T23:16:30.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.725+0000 7f2ad0600640 1 -- start start 2024-02-21T23:16:30.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.728+0000 7f2ad0600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2ac80c4900 0x7f2ac80c7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:30.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.728+0000 7f2ad0600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ac8070f90 con 0x7f2ac81502f0 2024-02-21T23:16:30.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.728+0000 7f2ad0600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ac8071270 con 0x7f2ac8070a60 2024-02-21T23:16:30.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.728+0000 7f2ad0600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac806ff30 con 0x7f2ac80c4900 2024-02-21T23:16:30.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.728+0000 7f2acdc00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f2ac81502f0 0x7f2ac8153850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:48546/0 (socket says 172.21.15.28:48546) 2024-02-21T23:16:30.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.728+0000 7f2acdc00640 1 -- 172.21.15.28:0/3951580280 learned_addr learned my addr 172.21.15.28:0/3951580280 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:16:30.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.729+0000 7f2ace600640 1 --2- 172.21.15.28:0/3951580280 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2ac80c4900 0x7f2ac80c7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:30.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.729+0000 7f2ac7e00640 1 -- 172.21.15.28:0/3951580280 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2654999016 0 0) 0x7f2ac8070f90 con 0x7f2ac81502f0 2024-02-21T23:16:30.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.729+0000 7f2ac7e00640 1 -- 172.21.15.28:0/3951580280 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2aac003520 con 0x7f2ac81502f0 2024-02-21T23:16:30.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.729+0000 7f2ac7e00640 1 -- 172.21.15.28:0/3951580280 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1285116931 0 0) 0x7f2aac003520 con 0x7f2ac81502f0 2024-02-21T23:16:30.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.729+0000 7f2ac7e00640 1 -- 172.21.15.28:0/3951580280 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2ac80c4900 msgr2=0x7f2ac80c7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:16:30.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.729+0000 7f2ac7e00640 1 --2- 172.21.15.28:0/3951580280 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2ac80c4900 0x7f2ac80c7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:30.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.729+0000 7f2ac7e00640 1 -- 172.21.15.28:0/3951580280 >> v1:172.21.15.81:6789/0 conn(0x7f2ac8070a60 legacy=0x7f2ac8156bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:30.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.729+0000 7f2ac7e00640 1 -- 172.21.15.28:0/3951580280 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ac8071550 con 0x7f2ac81502f0 2024-02-21T23:16:30.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.729+0000 7f2ac7e00640 1 -- 172.21.15.28:0/3951580280 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f2ab8001670 con 0x7f2ac81502f0 2024-02-21T23:16:30.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.729+0000 7f2ac7e00640 1 -- 172.21.15.28:0/3951580280 >> v1:172.21.15.28:6789/0 conn(0x7f2ac81502f0 legacy=0x7f2ac8153850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:30.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.730+0000 7f2ac7e00640 1 --2- 172.21.15.28:0/3951580280 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2ac80c4900 0x7f2aac0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:30.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.730+0000 7f2ac7e00640 1 --2- 172.21.15.28:0/3951580280 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2aac003fb0 0x7f2aac004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:30.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.730+0000 7f2ac7e00640 1 --2- 172.21.15.28:0/3951580280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2aac004d40 0x7f2aac0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:30.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.730+0000 7f2ac7e00640 1 -- 172.21.15.28:0/3951580280 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aac00bad0 con 0x7f2aac004d40 2024-02-21T23:16:30.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.730+0000 7f2ac7e00640 1 -- 172.21.15.28:0/3951580280 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aac00bc80 con 0x7f2aac003fb0 2024-02-21T23:16:30.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.730+0000 7f2ac7e00640 1 -- 172.21.15.28:0/3951580280 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2aac00be80 con 0x7f2ac80c4900 2024-02-21T23:16:30.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.730+0000 7f2ace600640 1 --2- 172.21.15.28:0/3951580280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2aac004d40 0x7f2aac0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:30.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.730+0000 7f2acdc00640 1 --2- 172.21.15.28:0/3951580280 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2ac80c4900 0x7f2aac0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:30.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.730+0000 7f2acd200640 1 --2- 172.21.15.28:0/3951580280 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2aac003fb0 0x7f2aac004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:30.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.730+0000 7f2ace600640 1 -- 172.21.15.28:0/3951580280 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2ac80c4900 msgr2=0x7f2aac0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:16:30.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.730+0000 7f2ace600640 1 --2- 172.21.15.28:0/3951580280 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2ac80c4900 0x7f2aac0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:30.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.730+0000 7f2ace600640 1 -- 172.21.15.28:0/3951580280 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2aac003fb0 msgr2=0x7f2aac004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:16:30.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.730+0000 7f2ace600640 1 --2- 172.21.15.28:0/3951580280 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2aac003fb0 0x7f2aac004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:30.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.731+0000 7f2ace600640 1 -- 172.21.15.28:0/3951580280 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aac003520 con 0x7f2aac004d40 2024-02-21T23:16:30.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.731+0000 7f2acdc00640 1 --2- 172.21.15.28:0/3951580280 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2ac80c4900 0x7f2aac0039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:16:30.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.732+0000 7f2ace600640 1 --2- 172.21.15.28:0/3951580280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2aac004d40 0x7f2aac0051c0 secure :-1 s=READY pgs=2633 cs=0 l=1 rev1=1 crypto rx=0x7f2ac0009ac0 tx=0x7f2aac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be2df33afcc25052 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:30.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.732+0000 7f2ac7e00640 1 -- 172.21.15.28:0/3951580280 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac0039070 con 0x7f2aac004d40 2024-02-21T23:16:30.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.732+0000 7f2ac7e00640 1 -- 172.21.15.28:0/3951580280 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ac002c6b0 con 0x7f2aac004d40 2024-02-21T23:16:30.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.732+0000 7f2ac7e00640 1 -- 172.21.15.28:0/3951580280 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac0009690 con 0x7f2aac004d40 2024-02-21T23:16:30.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.733+0000 7f2ad0600640 1 -- 172.21.15.28:0/3951580280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2aac004d40 msgr2=0x7f2aac0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:30.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.733+0000 7f2ad0600640 1 --2- 172.21.15.28:0/3951580280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2aac004d40 0x7f2aac0051c0 secure :-1 s=READY pgs=2633 cs=0 l=1 rev1=1 crypto rx=0x7f2ac0009ac0 tx=0x7f2aac005830 comp rx=0 tx=0).stop 2024-02-21T23:16:30.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.733+0000 7f2ace600640 1 -- 172.21.15.28:0/3951580280 reap_dead start 2024-02-21T23:16:30.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.733+0000 7f2ad0600640 1 -- 172.21.15.28:0/3951580280 shutdown_connections 2024-02-21T23:16:30.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.733+0000 7f2ad0600640 1 -- 172.21.15.28:0/3951580280 >> 172.21.15.28:0/3951580280 conn(0x7f2ac80c8810 msgr2=0x7f2ac80c8bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:16:30.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.733+0000 7f2ad0600640 1 -- 172.21.15.28:0/3951580280 shutdown_connections 2024-02-21T23:16:30.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.733+0000 7f2ad0600640 1 -- 172.21.15.28:0/3951580280 wait complete. 2024-02-21T23:16:30.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.735+0000 7f2ad0600640 1 Processor -- start 2024-02-21T23:16:30.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.735+0000 7f2ad0600640 1 -- start start 2024-02-21T23:16:30.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.736+0000 7f2ad0600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2aac004d40 0x7f2ac80abb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:30.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.736+0000 7f2ad0600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ac8140890 0x7f2ac80ad6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:30.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.736+0000 7f2ad0600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2ac806cb30 0x7f2ac806cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:30.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.736+0000 7f2ad0600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac81572d0 con 0x7f2aac004d40 2024-02-21T23:16:30.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.736+0000 7f2ad0600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac81566a0 con 0x7f2ac8140890 2024-02-21T23:16:30.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.736+0000 7f2ad0600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac81506d0 con 0x7f2ac806cb30 2024-02-21T23:16:30.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.736+0000 7f2acdc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2aac004d40 0x7f2ac80abb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:30.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.736+0000 7f2acdc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2aac004d40 0x7f2ac80abb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59298/0 (socket says 172.21.15.28:59298) 2024-02-21T23:16:30.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.736+0000 7f2acdc00640 1 -- 172.21.15.28:0/2871789240 learned_addr learned my addr 172.21.15.28:0/2871789240 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:16:30.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.737+0000 7f2ace600640 1 --2- 172.21.15.28:0/2871789240 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2ac806cb30 0x7f2ac806cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:30.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.737+0000 7f2acd200640 1 --2- 172.21.15.28:0/2871789240 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ac8140890 0x7f2ac80ad6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:30.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.737+0000 7f2acdc00640 1 -- 172.21.15.28:0/2871789240 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2ac806cb30 msgr2=0x7f2ac806cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:30.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.737+0000 7f2acdc00640 1 --2- 172.21.15.28:0/2871789240 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2ac806cb30 0x7f2ac806cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:30.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.737+0000 7f2acdc00640 1 -- 172.21.15.28:0/2871789240 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ac8140890 msgr2=0x7f2ac80ad6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:30.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.737+0000 7f2acdc00640 1 --2- 172.21.15.28:0/2871789240 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ac8140890 0x7f2ac80ad6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:30.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.737+0000 7f2acdc00640 1 -- 172.21.15.28:0/2871789240 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ac80c1ce0 con 0x7f2aac004d40 2024-02-21T23:16:30.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.737+0000 7f2acd200640 1 --2- 172.21.15.28:0/2871789240 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ac8140890 0x7f2ac80ad6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:16:30.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.737+0000 7f2acdc00640 1 --2- 172.21.15.28:0/2871789240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2aac004d40 0x7f2ac80abb30 secure :-1 s=READY pgs=2634 cs=0 l=1 rev1=1 crypto rx=0x7f2ab8000fb0 tx=0x7f2ab800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:30.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.737+0000 7f2ac5600640 1 -- 172.21.15.28:0/2871789240 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab800c8d0 con 0x7f2aac004d40 2024-02-21T23:16:30.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.737+0000 7f2ac5600640 1 -- 172.21.15.28:0/2871789240 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ab8018070 con 0x7f2aac004d40 2024-02-21T23:16:30.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.737+0000 7f2ad0600640 1 -- 172.21.15.28:0/2871789240 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ac80c1fc0 con 0x7f2aac004d40 2024-02-21T23:16:30.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.737+0000 7f2ac5600640 1 -- 172.21.15.28:0/2871789240 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab800ccb0 con 0x7f2aac004d40 2024-02-21T23:16:30.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.737+0000 7f2ad0600640 1 -- 172.21.15.28:0/2871789240 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ac80c2580 con 0x7f2aac004d40 2024-02-21T23:16:30.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.739+0000 7f2ad0600640 1 -- 172.21.15.28:0/2871789240 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ac80c6040 con 0x7f2aac004d40 2024-02-21T23:16:30.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.740+0000 7f2ac5600640 1 -- 172.21.15.28:0/2871789240 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2ab801d460 con 0x7f2aac004d40 2024-02-21T23:16:30.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.741+0000 7f2ac5600640 1 --2- 172.21.15.28:0/2871789240 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2aa806b4d0 0x7f2aa806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:30.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.741+0000 7f2ac5600640 1 -- 172.21.15.28:0/2871789240 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f2ab808a4b0 con 0x7f2aac004d40 2024-02-21T23:16:30.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.741+0000 7f2acd200640 1 --2- 172.21.15.28:0/2871789240 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2aa806b4d0 0x7f2aa806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:30.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.743+0000 7f2acd200640 1 --2- 172.21.15.28:0/2871789240 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2aa806b4d0 0x7f2aa806d990 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f2ab4007fd0 tx=0x7f2ab4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:30.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.745+0000 7f2ac5600640 1 -- 172.21.15.28:0/2871789240 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f2ac80c6040 con 0x7f2aac004d40 2024-02-21T23:16:30.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.977+0000 7f2ad0600640 1 -- 172.21.15.28:0/2871789240 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2ac80c2ab0 con 0x7f2aac004d40 2024-02-21T23:16:30.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.979+0000 7f2ac5600640 1 -- 172.21.15.28:0/2871789240 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7f2ac80c2ab0 con 0x7f2aac004d40 2024-02-21T23:16:30.967 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:16:30.967 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:16:30.967 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:16:30.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.981+0000 7f2ad0600640 1 -- 172.21.15.28:0/2871789240 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2aa806b4d0 msgr2=0x7f2aa806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:30.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.981+0000 7f2ad0600640 1 --2- 172.21.15.28:0/2871789240 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2aa806b4d0 0x7f2aa806d990 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f2ab4007fd0 tx=0x7f2ab4009040 comp rx=0 tx=0).stop 2024-02-21T23:16:30.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.981+0000 7f2ad0600640 1 -- 172.21.15.28:0/2871789240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2aac004d40 msgr2=0x7f2ac80abb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:30.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.981+0000 7f2ad0600640 1 --2- 172.21.15.28:0/2871789240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2aac004d40 0x7f2ac80abb30 secure :-1 s=READY pgs=2634 cs=0 l=1 rev1=1 crypto rx=0x7f2ab8000fb0 tx=0x7f2ab800ecd0 comp rx=0 tx=0).stop 2024-02-21T23:16:30.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.981+0000 7f2ad0600640 1 -- 172.21.15.28:0/2871789240 shutdown_connections 2024-02-21T23:16:30.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.981+0000 7f2ad0600640 1 --2- 172.21.15.28:0/2871789240 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2aa806b4d0 0x7f2aa806d990 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:30.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.981+0000 7f2ad0600640 1 --2- 172.21.15.28:0/2871789240 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2ac806cb30 0x7f2ac806cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:30.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.981+0000 7f2ad0600640 1 --2- 172.21.15.28:0/2871789240 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2ac8140890 0x7f2ac80ad6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:30.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.981+0000 7f2ad0600640 1 --2- 172.21.15.28:0/2871789240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2aac004d40 0x7f2ac80abb30 unknown :-1 s=CLOSED pgs=2634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:30.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.981+0000 7f2ad0600640 1 -- 172.21.15.28:0/2871789240 >> 172.21.15.28:0/2871789240 conn(0x7f2ac80c8810 msgr2=0x7f2ac814f540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:16:30.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.981+0000 7f2ad0600640 1 -- 172.21.15.28:0/2871789240 shutdown_connections 2024-02-21T23:16:30.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:30.981+0000 7f2ad0600640 1 -- 172.21.15.28:0/2871789240 wait complete. 2024-02-21T23:16:31.388 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:16:31.389 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:16:32.390 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:16:32.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.631+0000 7f83efe00640 1 Processor -- start 2024-02-21T23:16:32.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.631+0000 7f83efe00640 1 -- start start 2024-02-21T23:16:32.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.633+0000 7f83efe00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f83f0150450 0x7f83f00a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:32.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.633+0000 7f83efe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f83f006ee40 0x7f83f01536f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:32.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.633+0000 7f83efe00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83f00c2760 con 0x7f83f0156af0 2024-02-21T23:16:32.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.633+0000 7f83efe00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83f00c22c0 con 0x7f83f006ee40 2024-02-21T23:16:32.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.633+0000 7f83efe00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83f0156660 con 0x7f83f0150450 2024-02-21T23:16:32.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.633+0000 7f83e6000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f83f006ee40 0x7f83f01536f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:32.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.633+0000 7f83e6000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f83f006ee40 0x7f83f01536f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59314/0 (socket says 172.21.15.28:59314) 2024-02-21T23:16:32.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.633+0000 7f83e6000640 1 -- 172.21.15.28:0/2232951210 learned_addr learned my addr 172.21.15.28:0/2232951210 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:16:32.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.634+0000 7f83eea00640 1 --2- 172.21.15.28:0/2232951210 >> v2:172.21.15.81:3300/0 conn(0x7f83f0150450 0x7f83f00a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:32.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.634+0000 7f83e6000640 1 -- 172.21.15.28:0/2232951210 >> v2:172.21.15.81:3300/0 conn(0x7f83f0150450 msgr2=0x7f83f00a5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:16:32.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.634+0000 7f83e6000640 1 --2- 172.21.15.28:0/2232951210 >> v2:172.21.15.81:3300/0 conn(0x7f83f0150450 0x7f83f00a5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:32.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.634+0000 7f83e6000640 1 -- 172.21.15.28:0/2232951210 >> v1:172.21.15.28:6789/0 conn(0x7f83f0156af0 legacy=0x7f83f006dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:32.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.634+0000 7f83e6000640 1 -- 172.21.15.28:0/2232951210 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83f0153c30 con 0x7f83f006ee40 2024-02-21T23:16:32.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.635+0000 7f83e6000640 1 --2- 172.21.15.28:0/2232951210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83f006ee40 0x7f83f01536f0 secure :-1 s=READY pgs=2636 cs=0 l=1 rev1=1 crypto rx=0x7f83f006f600 tx=0x7f83d802b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac8d2056985a5ab0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:32.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.635+0000 7f83ee000640 1 -- 172.21.15.28:0/2232951210 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d8034070 con 0x7f83f006ee40 2024-02-21T23:16:32.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.635+0000 7f83ee000640 1 -- 172.21.15.28:0/2232951210 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83d8030870 con 0x7f83f006ee40 2024-02-21T23:16:32.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.635+0000 7f83ee000640 1 -- 172.21.15.28:0/2232951210 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d8030c40 con 0x7f83f006ee40 2024-02-21T23:16:32.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.636+0000 7f83efe00640 1 -- 172.21.15.28:0/2232951210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83f006ee40 msgr2=0x7f83f01536f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:32.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.636+0000 7f83efe00640 1 --2- 172.21.15.28:0/2232951210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83f006ee40 0x7f83f01536f0 secure :-1 s=READY pgs=2636 cs=0 l=1 rev1=1 crypto rx=0x7f83f006f600 tx=0x7f83d802b9b0 comp rx=0 tx=0).stop 2024-02-21T23:16:32.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.636+0000 7f83efe00640 1 -- 172.21.15.28:0/2232951210 shutdown_connections 2024-02-21T23:16:32.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.636+0000 7f83efe00640 1 --2- 172.21.15.28:0/2232951210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83f006ee40 0x7f83f01536f0 unknown :-1 s=CLOSED pgs=2636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:32.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.636+0000 7f83efe00640 1 --2- 172.21.15.28:0/2232951210 >> v2:172.21.15.81:3300/0 conn(0x7f83f0150450 0x7f83f00a5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:32.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.636+0000 7f83efe00640 1 -- 172.21.15.28:0/2232951210 >> 172.21.15.28:0/2232951210 conn(0x7f83f006b630 msgr2=0x7f83f00c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:16:32.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.636+0000 7f83efe00640 1 -- 172.21.15.28:0/2232951210 shutdown_connections 2024-02-21T23:16:32.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.636+0000 7f83efe00640 1 -- 172.21.15.28:0/2232951210 wait complete. 2024-02-21T23:16:32.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.639+0000 7f83efe00640 1 Processor -- start 2024-02-21T23:16:32.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.639+0000 7f83efe00640 1 -- start start 2024-02-21T23:16:32.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.641+0000 7f83efe00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f83f006ee40 0x7f83f00c40f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:32.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.641+0000 7f83efe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f83f0150450 0x7f83f00c3200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:32.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.641+0000 7f83efe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83f0156af0 0x7f83f0072110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:32.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.641+0000 7f83efe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83f00c2cb0 con 0x7f83f0156af0 2024-02-21T23:16:32.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.641+0000 7f83efe00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83f00c29d0 con 0x7f83f0150450 2024-02-21T23:16:32.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.641+0000 7f83efe00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83f006e700 con 0x7f83f006ee40 2024-02-21T23:16:32.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.641+0000 7f83ef400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83f0156af0 0x7f83f0072110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:32.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.641+0000 7f83ef400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83f0156af0 0x7f83f0072110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59320/0 (socket says 172.21.15.28:59320) 2024-02-21T23:16:32.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.641+0000 7f83ef400640 1 -- 172.21.15.28:0/1979520532 learned_addr learned my addr 172.21.15.28:0/1979520532 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:16:32.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.641+0000 7f83eea00640 1 --2- 172.21.15.28:0/1979520532 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f83f006ee40 0x7f83f00c40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:32.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.641+0000 7f83e6000640 1 --2- 172.21.15.28:0/1979520532 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f83f0150450 0x7f83f00c3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:32.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.642+0000 7f83ef400640 1 -- 172.21.15.28:0/1979520532 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f83f006ee40 msgr2=0x7f83f00c40f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:32.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.642+0000 7f83ef400640 1 --2- 172.21.15.28:0/1979520532 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f83f006ee40 0x7f83f00c40f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:32.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.642+0000 7f83ef400640 1 -- 172.21.15.28:0/1979520532 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f83f0150450 msgr2=0x7f83f00c3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:32.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.642+0000 7f83ef400640 1 --2- 172.21.15.28:0/1979520532 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f83f0150450 0x7f83f00c3200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:32.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.642+0000 7f83ef400640 1 -- 172.21.15.28:0/1979520532 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83f00c4630 con 0x7f83f0156af0 2024-02-21T23:16:32.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.642+0000 7f83ef400640 1 --2- 172.21.15.28:0/1979520532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83f0156af0 0x7f83f0072110 secure :-1 s=READY pgs=2637 cs=0 l=1 rev1=1 crypto rx=0x7f83e800c7d0 tx=0x7f83e800cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:32.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.642+0000 7f83e7400640 1 -- 172.21.15.28:0/1979520532 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83e8003b70 con 0x7f83f0156af0 2024-02-21T23:16:32.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.642+0000 7f83e7400640 1 -- 172.21.15.28:0/1979520532 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83e8003d30 con 0x7f83f0156af0 2024-02-21T23:16:32.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.642+0000 7f83efe00640 1 -- 172.21.15.28:0/1979520532 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83f0071070 con 0x7f83f0156af0 2024-02-21T23:16:32.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.642+0000 7f83e7400640 1 -- 172.21.15.28:0/1979520532 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83e800aa90 con 0x7f83f0156af0 2024-02-21T23:16:32.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.643+0000 7f83efe00640 1 -- 172.21.15.28:0/1979520532 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83f0071630 con 0x7f83f0156af0 2024-02-21T23:16:32.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.644+0000 7f83efe00640 1 -- 172.21.15.28:0/1979520532 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83f004f290 con 0x7f83f0156af0 2024-02-21T23:16:32.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.645+0000 7f83e7400640 1 -- 172.21.15.28:0/1979520532 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f83e800ac30 con 0x7f83f0156af0 2024-02-21T23:16:32.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.646+0000 7f83e7400640 1 --2- 172.21.15.28:0/1979520532 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f83cc06b4d0 0x7f83cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:32.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.646+0000 7f83e7400640 1 -- 172.21.15.28:0/1979520532 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f83e808a3d0 con 0x7f83f0156af0 2024-02-21T23:16:32.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.646+0000 7f83eea00640 1 --2- 172.21.15.28:0/1979520532 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f83cc06b4d0 0x7f83cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:32.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.648+0000 7f83eea00640 1 --2- 172.21.15.28:0/1979520532 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f83cc06b4d0 0x7f83cc06d990 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f83e0006fd0 tx=0x7f83e0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:32.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.655+0000 7f83e7400640 1 -- 172.21.15.28:0/1979520532 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f83e8058fb0 con 0x7f83f0156af0 2024-02-21T23:16:32.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.896+0000 7f83efe00640 1 -- 172.21.15.28:0/1979520532 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f83f006e9a0 con 0x7f83f0156af0 2024-02-21T23:16:32.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.897+0000 7f83e7400640 1 -- 172.21.15.28:0/1979520532 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7f83e80588b0 con 0x7f83f0156af0 2024-02-21T23:16:32.885 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:16:32.885 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:16:32.886 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:16:32.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.899+0000 7f83efe00640 1 -- 172.21.15.28:0/1979520532 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f83cc06b4d0 msgr2=0x7f83cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:32.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.900+0000 7f83efe00640 1 --2- 172.21.15.28:0/1979520532 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f83cc06b4d0 0x7f83cc06d990 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f83e0006fd0 tx=0x7f83e0008040 comp rx=0 tx=0).stop 2024-02-21T23:16:32.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.900+0000 7f83efe00640 1 -- 172.21.15.28:0/1979520532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83f0156af0 msgr2=0x7f83f0072110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:32.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.900+0000 7f83efe00640 1 --2- 172.21.15.28:0/1979520532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83f0156af0 0x7f83f0072110 secure :-1 s=READY pgs=2637 cs=0 l=1 rev1=1 crypto rx=0x7f83e800c7d0 tx=0x7f83e800cc90 comp rx=0 tx=0).stop 2024-02-21T23:16:32.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.900+0000 7f83efe00640 1 -- 172.21.15.28:0/1979520532 shutdown_connections 2024-02-21T23:16:32.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.900+0000 7f83efe00640 1 --2- 172.21.15.28:0/1979520532 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f83cc06b4d0 0x7f83cc06d990 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:32.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.900+0000 7f83efe00640 1 --2- 172.21.15.28:0/1979520532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83f0156af0 0x7f83f0072110 unknown :-1 s=CLOSED pgs=2637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:32.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.900+0000 7f83efe00640 1 --2- 172.21.15.28:0/1979520532 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f83f0150450 0x7f83f00c3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:32.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.900+0000 7f83efe00640 1 --2- 172.21.15.28:0/1979520532 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f83f006ee40 0x7f83f00c40f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:32.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.900+0000 7f83efe00640 1 -- 172.21.15.28:0/1979520532 >> 172.21.15.28:0/1979520532 conn(0x7f83f006b630 msgr2=0x7f83f0153080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:16:32.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.900+0000 7f83efe00640 1 -- 172.21.15.28:0/1979520532 shutdown_connections 2024-02-21T23:16:32.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:32.900+0000 7f83efe00640 1 -- 172.21.15.28:0/1979520532 wait complete. 2024-02-21T23:16:33.307 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:16:33.308 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:16:34.309 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:16:34.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.547+0000 7f604c200640 1 Processor -- start 2024-02-21T23:16:34.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.547+0000 7f604c200640 1 -- start start 2024-02-21T23:16:34.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.548+0000 7f604c200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f60440c5a80 0x7f60441538d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:34.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.548+0000 7f604c200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f604406f550 con 0x7f60440c7b20 2024-02-21T23:16:34.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.548+0000 7f604c200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60440c8090 con 0x7f60440c7400 2024-02-21T23:16:34.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.548+0000 7f604c200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6044153e10 con 0x7f60440c5a80 2024-02-21T23:16:34.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.549+0000 7f6048e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f60440c7b20 0x7f6044154340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:48550/0 (socket says 172.21.15.28:48550) 2024-02-21T23:16:34.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.549+0000 7f6048e00640 1 -- 172.21.15.28:0/2621484991 learned_addr learned my addr 172.21.15.28:0/2621484991 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:16:34.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.549+0000 7f6049800640 1 --2- 172.21.15.28:0/2621484991 >> v2:172.21.15.28:3300/0 conn(0x7f60440c5a80 0x7f60441538d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:34.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.549+0000 7f6043e00640 1 -- 172.21.15.28:0/2621484991 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3673622423 0 0) 0x7f604406f550 con 0x7f60440c7b20 2024-02-21T23:16:34.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.549+0000 7f6043e00640 1 -- 172.21.15.28:0/2621484991 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6028003520 con 0x7f60440c7b20 2024-02-21T23:16:34.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.550+0000 7f6049800640 1 -- 172.21.15.28:0/2621484991 >> v1:172.21.15.81:6789/0 conn(0x7f60440c7400 legacy=0x7f6044070a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:16:34.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.550+0000 7f6049800640 1 -- 172.21.15.28:0/2621484991 >> v1:172.21.15.28:6789/0 conn(0x7f60440c7b20 legacy=0x7f6044154340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:34.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.550+0000 7f6049800640 1 -- 172.21.15.28:0/2621484991 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60440c6920 con 0x7f60440c5a80 2024-02-21T23:16:34.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.551+0000 7f6049800640 1 --2- 172.21.15.28:0/2621484991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60440c5a80 0x7f60441538d0 secure :-1 s=READY pgs=2639 cs=0 l=1 rev1=1 crypto rx=0x7f6034009d90 tx=0x7f603402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a0580a4966d2275 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:34.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.551+0000 7f6043e00640 1 -- 172.21.15.28:0/2621484991 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6034039070 con 0x7f60440c5a80 2024-02-21T23:16:34.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.551+0000 7f6043e00640 1 -- 172.21.15.28:0/2621484991 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f603402cd00 con 0x7f60440c5a80 2024-02-21T23:16:34.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.551+0000 7f6043e00640 1 -- 172.21.15.28:0/2621484991 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6034034660 con 0x7f60440c5a80 2024-02-21T23:16:34.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.552+0000 7f604c200640 1 -- 172.21.15.28:0/2621484991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60440c5a80 msgr2=0x7f60441538d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:34.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.552+0000 7f604c200640 1 --2- 172.21.15.28:0/2621484991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60440c5a80 0x7f60441538d0 secure :-1 s=READY pgs=2639 cs=0 l=1 rev1=1 crypto rx=0x7f6034009d90 tx=0x7f603402bf50 comp rx=0 tx=0).stop 2024-02-21T23:16:34.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.552+0000 7f604c200640 1 -- 172.21.15.28:0/2621484991 shutdown_connections 2024-02-21T23:16:34.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.552+0000 7f604c200640 1 --2- 172.21.15.28:0/2621484991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60440c5a80 0x7f60441538d0 unknown :-1 s=CLOSED pgs=2639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:34.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.552+0000 7f604c200640 1 -- 172.21.15.28:0/2621484991 >> 172.21.15.28:0/2621484991 conn(0x7f60440c8810 msgr2=0x7f60440c8bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:16:34.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.552+0000 7f604c200640 1 -- 172.21.15.28:0/2621484991 shutdown_connections 2024-02-21T23:16:34.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.552+0000 7f604c200640 1 -- 172.21.15.28:0/2621484991 wait complete. 2024-02-21T23:16:34.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.553+0000 7f604c200640 1 Processor -- start 2024-02-21T23:16:34.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.554+0000 7f604c200640 1 -- start start 2024-02-21T23:16:34.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.555+0000 7f604c200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60440c5a80 0x7f60440bd830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:34.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.555+0000 7f604c200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f60440c7400 0x7f60440be7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:34.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.555+0000 7f604c200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f60440c7b20 0x7f60441dc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:34.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.555+0000 7f604c200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6044154a50 con 0x7f60440c5a80 2024-02-21T23:16:34.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.555+0000 7f604c200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60440c6ee0 con 0x7f60440c7400 2024-02-21T23:16:34.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.555+0000 7f604c200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6044071120 con 0x7f60440c7b20 2024-02-21T23:16:34.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.556+0000 7f6049800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60440c5a80 0x7f60440bd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:34.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.556+0000 7f6049800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60440c5a80 0x7f60440bd830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59342/0 (socket says 172.21.15.28:59342) 2024-02-21T23:16:34.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.556+0000 7f6049800640 1 -- 172.21.15.28:0/1595125376 learned_addr learned my addr 172.21.15.28:0/1595125376 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:16:34.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.556+0000 7f6048e00640 1 --2- 172.21.15.28:0/1595125376 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f60440c7400 0x7f60440be7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:34.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.556+0000 7f604a200640 1 --2- 172.21.15.28:0/1595125376 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f60440c7b20 0x7f60441dc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:34.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.556+0000 7f6049800640 1 -- 172.21.15.28:0/1595125376 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f60440c7b20 msgr2=0x7f60441dc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:34.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.556+0000 7f6049800640 1 --2- 172.21.15.28:0/1595125376 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f60440c7b20 0x7f60441dc4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:34.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.556+0000 7f6049800640 1 -- 172.21.15.28:0/1595125376 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f60440c7400 msgr2=0x7f60440be7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:34.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.556+0000 7f6049800640 1 --2- 172.21.15.28:0/1595125376 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f60440c7400 0x7f60440be7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:34.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.556+0000 7f6049800640 1 -- 172.21.15.28:0/1595125376 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60440bfd00 con 0x7f60440c5a80 2024-02-21T23:16:34.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.556+0000 7f6048e00640 1 --2- 172.21.15.28:0/1595125376 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f60440c7400 0x7f60440be7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:16:34.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.556+0000 7f6049800640 1 --2- 172.21.15.28:0/1595125376 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60440c5a80 0x7f60440bd830 secure :-1 s=READY pgs=2640 cs=0 l=1 rev1=1 crypto rx=0x7f6034009d90 tx=0x7f6034004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:34.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.557+0000 7f6041600640 1 -- 172.21.15.28:0/1595125376 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6034039050 con 0x7f60440c5a80 2024-02-21T23:16:34.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.557+0000 7f6041600640 1 -- 172.21.15.28:0/1595125376 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6034045070 con 0x7f60440c5a80 2024-02-21T23:16:34.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.557+0000 7f6041600640 1 -- 172.21.15.28:0/1595125376 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f603403e850 con 0x7f60440c5a80 2024-02-21T23:16:34.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.557+0000 7f604c200640 1 -- 172.21.15.28:0/1595125376 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60440bdf80 con 0x7f60440c5a80 2024-02-21T23:16:34.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.557+0000 7f604c200640 1 -- 172.21.15.28:0/1595125376 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60441df960 con 0x7f60440c5a80 2024-02-21T23:16:34.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.559+0000 7f603aa00640 1 -- 172.21.15.28:0/1595125376 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6018005290 con 0x7f60440c5a80 2024-02-21T23:16:34.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.559+0000 7f6041600640 1 -- 172.21.15.28:0/1595125376 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f603403e9f0 con 0x7f60440c5a80 2024-02-21T23:16:34.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.560+0000 7f6041600640 1 --2- 172.21.15.28:0/1595125376 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f602406b4d0 0x7f602406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:34.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.560+0000 7f6041600640 1 -- 172.21.15.28:0/1595125376 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f60340ad250 con 0x7f60440c5a80 2024-02-21T23:16:34.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.561+0000 7f6048e00640 1 --2- 172.21.15.28:0/1595125376 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f602406b4d0 0x7f602406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:34.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.562+0000 7f6048e00640 1 --2- 172.21.15.28:0/1595125376 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f602406b4d0 0x7f602406d990 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f60440c2bb0 tx=0x7f6030009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:34.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.565+0000 7f6041600640 1 -- 172.21.15.28:0/1595125376 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f6034048360 con 0x7f60440c5a80 2024-02-21T23:16:34.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.805+0000 7f603aa00640 1 -- 172.21.15.28:0/1595125376 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6018005580 con 0x7f60440c5a80 2024-02-21T23:16:34.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.806+0000 7f6041600640 1 -- 172.21.15.28:0/1595125376 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7f603407c370 con 0x7f60440c5a80 2024-02-21T23:16:34.795 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:16:34.795 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:16:34.795 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:16:34.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.810+0000 7f603aa00640 1 -- 172.21.15.28:0/1595125376 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f602406b4d0 msgr2=0x7f602406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:34.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.810+0000 7f603aa00640 1 --2- 172.21.15.28:0/1595125376 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f602406b4d0 0x7f602406d990 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f60440c2bb0 tx=0x7f6030009040 comp rx=0 tx=0).stop 2024-02-21T23:16:34.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.810+0000 7f603aa00640 1 -- 172.21.15.28:0/1595125376 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60440c5a80 msgr2=0x7f60440bd830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:34.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.810+0000 7f603aa00640 1 --2- 172.21.15.28:0/1595125376 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60440c5a80 0x7f60440bd830 secure :-1 s=READY pgs=2640 cs=0 l=1 rev1=1 crypto rx=0x7f6034009d90 tx=0x7f6034004a40 comp rx=0 tx=0).stop 2024-02-21T23:16:34.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.810+0000 7f603aa00640 1 -- 172.21.15.28:0/1595125376 shutdown_connections 2024-02-21T23:16:34.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.810+0000 7f603aa00640 1 --2- 172.21.15.28:0/1595125376 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f602406b4d0 0x7f602406d990 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:34.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.810+0000 7f603aa00640 1 --2- 172.21.15.28:0/1595125376 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f60440c7b20 0x7f60441dc4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:34.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.810+0000 7f603aa00640 1 --2- 172.21.15.28:0/1595125376 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f60440c7400 0x7f60440be7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:34.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.810+0000 7f603aa00640 1 --2- 172.21.15.28:0/1595125376 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60440c5a80 0x7f60440bd830 unknown :-1 s=CLOSED pgs=2640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:34.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.810+0000 7f603aa00640 1 -- 172.21.15.28:0/1595125376 >> 172.21.15.28:0/1595125376 conn(0x7f60440c8810 msgr2=0x7f60440c8bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:16:34.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.810+0000 7f603aa00640 1 -- 172.21.15.28:0/1595125376 shutdown_connections 2024-02-21T23:16:34.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:34.810+0000 7f603aa00640 1 -- 172.21.15.28:0/1595125376 wait complete. 2024-02-21T23:16:35.230 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:16:35.231 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:16:36.231 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:16:36.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.471+0000 7f1987600640 1 Processor -- start 2024-02-21T23:16:36.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.471+0000 7f1987600640 1 -- start start 2024-02-21T23:16:36.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.473+0000 7f1987600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f19801526b0 0x7f19801559c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:36.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.473+0000 7f1987600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19800c6f30 con 0x7f19800cadc0 2024-02-21T23:16:36.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.473+0000 7f1987600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19800c6450 con 0x7f19800ca940 2024-02-21T23:16:36.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.473+0000 7f1987600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19800c61b0 con 0x7f19801526b0 2024-02-21T23:16:36.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.473+0000 7f1984c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f19801526b0 0x7f19801559c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:36.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.474+0000 7f1984c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f19801526b0 0x7f19801559c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51378/0 (socket says 172.21.15.28:51378) 2024-02-21T23:16:36.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.474+0000 7f1984c00640 1 -- 172.21.15.28:0/958807169 learned_addr learned my addr 172.21.15.28:0/958807169 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:16:36.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.474+0000 7f1984c00640 1 -- 172.21.15.28:0/958807169 >> v1:172.21.15.81:6789/0 conn(0x7f19800ca940 legacy=0x7f19800c76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:16:36.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.474+0000 7f1984c00640 1 -- 172.21.15.28:0/958807169 >> v1:172.21.15.28:6789/0 conn(0x7f19800cadc0 legacy=0x7f198015f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:36.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.474+0000 7f1984c00640 1 -- 172.21.15.28:0/958807169 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19800c6710 con 0x7f19801526b0 2024-02-21T23:16:36.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.474+0000 7f197f400640 1 -- 172.21.15.28:0/958807169 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2734891564 0 0) 0x7f19800c6f30 con 0x7f19800cadc0 2024-02-21T23:16:36.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.476+0000 7f1984c00640 1 --2- 172.21.15.28:0/958807169 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19801526b0 0x7f19801559c0 secure :-1 s=READY pgs=2642 cs=0 l=1 rev1=1 crypto rx=0x7f1970009870 tx=0x7f197002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee2d14dfeaba6f32 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:36.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.476+0000 7f197f400640 1 -- 172.21.15.28:0/958807169 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1970035020 con 0x7f19801526b0 2024-02-21T23:16:36.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.476+0000 7f197f400640 1 -- 172.21.15.28:0/958807169 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1970031870 con 0x7f19801526b0 2024-02-21T23:16:36.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.476+0000 7f197f400640 1 -- 172.21.15.28:0/958807169 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1970031c40 con 0x7f19801526b0 2024-02-21T23:16:36.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.477+0000 7f1987600640 1 -- 172.21.15.28:0/958807169 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19801526b0 msgr2=0x7f19801559c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:36.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.477+0000 7f1987600640 1 --2- 172.21.15.28:0/958807169 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19801526b0 0x7f19801559c0 secure :-1 s=READY pgs=2642 cs=0 l=1 rev1=1 crypto rx=0x7f1970009870 tx=0x7f197002b910 comp rx=0 tx=0).stop 2024-02-21T23:16:36.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.477+0000 7f1987600640 1 -- 172.21.15.28:0/958807169 shutdown_connections 2024-02-21T23:16:36.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.477+0000 7f1987600640 1 --2- 172.21.15.28:0/958807169 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19801526b0 0x7f19801559c0 unknown :-1 s=CLOSED pgs=2642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:36.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.477+0000 7f1987600640 1 -- 172.21.15.28:0/958807169 >> 172.21.15.28:0/958807169 conn(0x7f198006d3d0 msgr2=0x7f198008afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:16:36.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.477+0000 7f1987600640 1 -- 172.21.15.28:0/958807169 shutdown_connections 2024-02-21T23:16:36.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.477+0000 7f1987600640 1 -- 172.21.15.28:0/958807169 wait complete. 2024-02-21T23:16:36.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.479+0000 7f1987600640 1 Processor -- start 2024-02-21T23:16:36.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.479+0000 7f1987600640 1 -- start start 2024-02-21T23:16:36.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.480+0000 7f1987600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19800ca940 0x7f19801de5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:36.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.480+0000 7f1987600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f19800cadc0 0x7f19800c9110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:36.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.480+0000 7f1987600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f19801526b0 0x7f198014ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:36.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.480+0000 7f1987600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19800c5c90 con 0x7f19800ca940 2024-02-21T23:16:36.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.480+0000 7f1987600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19800c71d0 con 0x7f19801526b0 2024-02-21T23:16:36.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.480+0000 7f1987600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19800c4e70 con 0x7f19800cadc0 2024-02-21T23:16:36.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.480+0000 7f1984c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19800ca940 0x7f19801de5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:36.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.480+0000 7f1984c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19800ca940 0x7f19801de5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51386/0 (socket says 172.21.15.28:51386) 2024-02-21T23:16:36.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.480+0000 7f1984c00640 1 -- 172.21.15.28:0/2361259960 learned_addr learned my addr 172.21.15.28:0/2361259960 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:16:36.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.480+0000 7f197fe00640 1 --2- 172.21.15.28:0/2361259960 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f19800cadc0 0x7f19800c9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:36.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.480+0000 7f1984c00640 1 -- 172.21.15.28:0/2361259960 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f19800cadc0 msgr2=0x7f19800c9110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:36.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.481+0000 7f1984c00640 1 --2- 172.21.15.28:0/2361259960 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f19800cadc0 0x7f19800c9110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:36.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.481+0000 7f1984c00640 1 -- 172.21.15.28:0/2361259960 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f19801526b0 msgr2=0x7f198014ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:36.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.481+0000 7f1985600640 1 --2- 172.21.15.28:0/2361259960 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f19801526b0 0x7f198014ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:36.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.481+0000 7f1984c00640 1 --2- 172.21.15.28:0/2361259960 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f19801526b0 0x7f198014ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:36.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.481+0000 7f1984c00640 1 -- 172.21.15.28:0/2361259960 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19800c9b30 con 0x7f19800ca940 2024-02-21T23:16:36.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.481+0000 7f197fe00640 1 --2- 172.21.15.28:0/2361259960 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f19800cadc0 0x7f19800c9110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:16:36.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.481+0000 7f1985600640 1 --2- 172.21.15.28:0/2361259960 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f19801526b0 0x7f198014ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:16:36.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.481+0000 7f1984c00640 1 --2- 172.21.15.28:0/2361259960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19800ca940 0x7f19801de5a0 secure :-1 s=READY pgs=2643 cs=0 l=1 rev1=1 crypto rx=0x7f1970009870 tx=0x7f19700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:36.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.481+0000 7f197cc00640 1 -- 172.21.15.28:0/2361259960 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f197003ad60 con 0x7f19800ca940 2024-02-21T23:16:36.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.481+0000 7f197cc00640 1 -- 172.21.15.28:0/2361259960 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19700443f0 con 0x7f19800ca940 2024-02-21T23:16:36.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.481+0000 7f1987600640 1 -- 172.21.15.28:0/2361259960 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19801deae0 con 0x7f19800ca940 2024-02-21T23:16:36.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.481+0000 7f197cc00640 1 -- 172.21.15.28:0/2361259960 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19700447e0 con 0x7f19800ca940 2024-02-21T23:16:36.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.482+0000 7f1987600640 1 -- 172.21.15.28:0/2361259960 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19800c9650 con 0x7f19800ca940 2024-02-21T23:16:36.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.483+0000 7f1987600640 1 -- 172.21.15.28:0/2361259960 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1980067f30 con 0x7f19800ca940 2024-02-21T23:16:36.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.484+0000 7f197cc00640 1 -- 172.21.15.28:0/2361259960 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f197004b060 con 0x7f19800ca940 2024-02-21T23:16:36.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.485+0000 7f197cc00640 1 --2- 172.21.15.28:0/2361259960 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f196006b4d0 0x7f196006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:36.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.486+0000 7f197cc00640 1 -- 172.21.15.28:0/2361259960 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f19700ac630 con 0x7f19800ca940 2024-02-21T23:16:36.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.486+0000 7f197fe00640 1 --2- 172.21.15.28:0/2361259960 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f196006b4d0 0x7f196006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:36.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.487+0000 7f197fe00640 1 --2- 172.21.15.28:0/2361259960 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f196006b4d0 0x7f196006d990 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f19800c87e0 tx=0x7f196c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:36.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.489+0000 7f197cc00640 1 -- 172.21.15.28:0/2361259960 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f197003f070 con 0x7f19800ca940 2024-02-21T23:16:36.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.735+0000 7f1987600640 1 -- 172.21.15.28:0/2361259960 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1980151350 con 0x7f19800ca940 2024-02-21T23:16:36.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.736+0000 7f197cc00640 1 -- 172.21.15.28:0/2361259960 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7f197002f320 con 0x7f19800ca940 2024-02-21T23:16:36.725 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:16:36.725 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:16:36.725 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:16:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.739+0000 7f1987600640 1 -- 172.21.15.28:0/2361259960 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f196006b4d0 msgr2=0x7f196006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.739+0000 7f1987600640 1 --2- 172.21.15.28:0/2361259960 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f196006b4d0 0x7f196006d990 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f19800c87e0 tx=0x7f196c009040 comp rx=0 tx=0).stop 2024-02-21T23:16:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.739+0000 7f1987600640 1 -- 172.21.15.28:0/2361259960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19800ca940 msgr2=0x7f19801de5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.739+0000 7f1987600640 1 --2- 172.21.15.28:0/2361259960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19800ca940 0x7f19801de5a0 secure :-1 s=READY pgs=2643 cs=0 l=1 rev1=1 crypto rx=0x7f1970009870 tx=0x7f19700046c0 comp rx=0 tx=0).stop 2024-02-21T23:16:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.740+0000 7f1987600640 1 -- 172.21.15.28:0/2361259960 shutdown_connections 2024-02-21T23:16:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.740+0000 7f1987600640 1 --2- 172.21.15.28:0/2361259960 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f196006b4d0 0x7f196006d990 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.740+0000 7f1987600640 1 --2- 172.21.15.28:0/2361259960 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f19800cadc0 0x7f19800c9110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:36.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.740+0000 7f1987600640 1 --2- 172.21.15.28:0/2361259960 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f19801526b0 0x7f198014ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:36.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.740+0000 7f1987600640 1 --2- 172.21.15.28:0/2361259960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19800ca940 0x7f19801de5a0 unknown :-1 s=CLOSED pgs=2643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:36.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.740+0000 7f1987600640 1 -- 172.21.15.28:0/2361259960 >> 172.21.15.28:0/2361259960 conn(0x7f198006d3d0 msgr2=0x7f1980150460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:16:36.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.740+0000 7f1987600640 1 -- 172.21.15.28:0/2361259960 shutdown_connections 2024-02-21T23:16:36.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:36.740+0000 7f1987600640 1 -- 172.21.15.28:0/2361259960 wait complete. 2024-02-21T23:16:37.148 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:16:37.149 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:16:38.150 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:16:38.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:38.386+0000 7f3aec600640 1 Processor -- start 2024-02-21T23:16:38.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:38.386+0000 7f3aec600640 1 -- start start 2024-02-21T23:16:38.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:38.388+0000 7f3aec600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3ae40c11f0 0x7f3ae4153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:38.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:38.388+0000 7f3aec600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ae4152e70 con 0x7f3ae40c15d0 2024-02-21T23:16:38.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:38.388+0000 7f3aec600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ae4153150 con 0x7f3ae415b6d0 2024-02-21T23:16:38.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:38.388+0000 7f3aec600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ae40c1a60 con 0x7f3ae40c11f0 2024-02-21T23:16:38.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:38.388+0000 7f3ae9200640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f3ae40c15d0 0x7f3ae406dce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:45644/0 (socket says 172.21.15.28:45644) 2024-02-21T23:16:38.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:38.388+0000 7f3ae9200640 1 -- 172.21.15.28:0/874903786 learned_addr learned my addr 172.21.15.28:0/874903786 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:16:38.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:38.388+0000 7f3ae9c00640 1 --2- 172.21.15.28:0/874903786 >> v2:172.21.15.28:3300/0 conn(0x7f3ae40c11f0 0x7f3ae4153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:38.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:38.389+0000 7f3ae9c00640 1 -- 172.21.15.28:0/874903786 >> v1:172.21.15.81:6789/0 conn(0x7f3ae415b6d0 legacy=0x7f3ae415dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:16:38.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:38.389+0000 7f3ae9c00640 1 -- 172.21.15.28:0/874903786 >> v1:172.21.15.28:6789/0 conn(0x7f3ae40c15d0 legacy=0x7f3ae406dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:38.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:38.389+0000 7f3ae3e00640 1 -- 172.21.15.28:0/874903786 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3299472431 0 0) 0x7f3ae4152e70 con 0x7f3ae40c15d0 2024-02-21T23:16:38.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:38.389+0000 7f3ae9c00640 1 -- 172.21.15.28:0/874903786 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ae4153430 con 0x7f3ae40c11f0 2024-02-21T23:16:38.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:38.391+0000 7f3ae9c00640 1 --2- 172.21.15.28:0/874903786 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ae40c11f0 0x7f3ae4153950 secure :-1 s=READY pgs=2645 cs=0 l=1 rev1=1 crypto rx=0x7f3ad8009d90 tx=0x7f3ad802be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2f5dfa2ec8645d server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:38.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:38.391+0000 7f3ae3e00640 1 -- 172.21.15.28:0/874903786 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ad8039070 con 0x7f3ae40c11f0 2024-02-21T23:16:38.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:38.391+0000 7f3ae3e00640 1 -- 172.21.15.28:0/874903786 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ad802cbd0 con 0x7f3ae40c11f0 2024-02-21T23:16:38.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:38.391+0000 7f3ae3e00640 1 -- 172.21.15.28:0/874903786 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ad8034440 con 0x7f3ae40c11f0 2024-02-21T23:16:38.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:38.392+0000 7f3aec600640 1 -- 172.21.15.28:0/874903786 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ae40c11f0 msgr2=0x7f3ae4153950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:38.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:38.392+0000 7f3aec600640 1 --2- 172.21.15.28:0/874903786 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ae40c11f0 0x7f3ae4153950 secure :-1 s=READY pgs=2645 cs=0 l=1 rev1=1 crypto rx=0x7f3ad8009d90 tx=0x7f3ad802be20 comp rx=0 tx=0).stop 2024-02-21T23:16:38.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:38.392+0000 7f3aec600640 1 -- 172.21.15.28:0/874903786 shutdown_connections 2024-02-21T23:16:38.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:38.392+0000 7f3aec600640 1 --2- 172.21.15.28:0/874903786 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ae40c11f0 0x7f3ae4153950 unknown :-1 s=CLOSED pgs=2645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:38.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:38.392+0000 7f3aec600640 1 -- 172.21.15.28:0/874903786 >> 172.21.15.28:0/874903786 conn(0x7f3ae40c67f0 msgr2=0x7f3ae40c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:16:38.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:38.392+0000 7f3aec600640 1 -- 172.21.15.28:0/874903786 shutdown_connections 2024-02-21T23:16:38.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:38.392+0000 7f3aec600640 1 -- 172.21.15.28:0/874903786 wait complete. 2024-02-21T23:16:38.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:38.393+0000 7f3aec600640 1 Processor -- start 2024-02-21T23:16:38.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:38.394+0000 7f3aec600640 1 -- start start 2024-02-21T23:16:38.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:38.394+0000 7f3aec600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ae40c11f0 0x7f3ae41eb8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:38.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:38.395+0000 7f3aec600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3ae40c15d0 0x7f3ae4146c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:38.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:38.395+0000 7f3aec600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3ae415b6d0 0x7f3ae40bfc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:38.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:38.395+0000 7f3aec600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae40c4c00 con 0x7f3ae40c11f0 2024-02-21T23:16:38.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:38.395+0000 7f3aec600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae40a5ec0 con 0x7f3ae40c15d0 2024-02-21T23:16:38.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:38.395+0000 7f3aec600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae40a6110 con 0x7f3ae415b6d0 2024-02-21T23:16:38.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:38.395+0000 7f3ae9c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ae40c11f0 0x7f3ae41eb8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:38.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:38.395+0000 7f3ae9c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ae40c11f0 0x7f3ae41eb8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51404/0 (socket says 172.21.15.28:51404) 2024-02-21T23:16:38.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:38.395+0000 7f3ae9c00640 1 -- 172.21.15.28:0/3289321944 learned_addr learned my addr 172.21.15.28:0/3289321944 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:16:38.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:38.395+0000 7f3ae9c00640 1 -- 172.21.15.28:0/3289321944 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3ae415b6d0 msgr2=0x7f3ae40bfc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:38.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:38.395+0000 7f3ae9c00640 1 --2- 172.21.15.28:0/3289321944 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3ae415b6d0 0x7f3ae40bfc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:38.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:38.395+0000 7f3ae9c00640 1 -- 172.21.15.28:0/3289321944 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3ae40c15d0 msgr2=0x7f3ae4146c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:38.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:38.395+0000 7f3ae9c00640 1 --2- 172.21.15.28:0/3289321944 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3ae40c15d0 0x7f3ae4146c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:38.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:38.395+0000 7f3ae9c00640 1 -- 172.21.15.28:0/3289321944 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ae4147380 con 0x7f3ae40c11f0 2024-02-21T23:16:38.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:38.396+0000 7f3ae9c00640 1 --2- 172.21.15.28:0/3289321944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ae40c11f0 0x7f3ae41eb8d0 secure :-1 s=READY pgs=2646 cs=0 l=1 rev1=1 crypto rx=0x7f3ad8009eb0 tx=0x7f3ad8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:38.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:38.396+0000 7f3ae1600640 1 -- 172.21.15.28:0/3289321944 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ad8039050 con 0x7f3ae40c11f0 2024-02-21T23:16:38.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:38.396+0000 7f3ae1600640 1 -- 172.21.15.28:0/3289321944 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ad8045070 con 0x7f3ae40c11f0 2024-02-21T23:16:38.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:38.396+0000 7f3aec600640 1 -- 172.21.15.28:0/3289321944 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ae4072c20 con 0x7f3ae40c11f0 2024-02-21T23:16:38.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:38.396+0000 7f3ae1600640 1 -- 172.21.15.28:0/3289321944 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ad803ea70 con 0x7f3ae40c11f0 2024-02-21T23:16:38.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:38.396+0000 7f3aec600640 1 -- 172.21.15.28:0/3289321944 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ae40731e0 con 0x7f3ae40c11f0 2024-02-21T23:16:38.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:38.397+0000 7f3aec600640 1 -- 172.21.15.28:0/3289321944 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ae4067f30 con 0x7f3ae40c11f0 2024-02-21T23:16:38.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:38.398+0000 7f3ae1600640 1 -- 172.21.15.28:0/3289321944 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3ad802ca90 con 0x7f3ae40c11f0 2024-02-21T23:16:38.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:38.399+0000 7f3ae1600640 1 --2- 172.21.15.28:0/3289321944 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3ac406b4d0 0x7f3ac406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:38.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:38.399+0000 7f3ae1600640 1 -- 172.21.15.28:0/3289321944 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f3ad80ad270 con 0x7f3ae40c11f0 2024-02-21T23:16:38.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:38.400+0000 7f3ae9200640 1 --2- 172.21.15.28:0/3289321944 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3ac406b4d0 0x7f3ac406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:38.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:38.401+0000 7f3ae9200640 1 --2- 172.21.15.28:0/3289321944 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3ac406b4d0 0x7f3ac406d990 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f3ad0000c00 tx=0x7f3ad00079c0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:38.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:38.404+0000 7f3ae1600640 1 -- 172.21.15.28:0/3289321944 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3ae4147380 con 0x7f3ae40c11f0 2024-02-21T23:16:38.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:38.643+0000 7f3aec600640 1 -- 172.21.15.28:0/3289321944 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3ae40c0960 con 0x7f3ae40c11f0 2024-02-21T23:16:38.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:38.644+0000 7f3ae1600640 1 -- 172.21.15.28:0/3289321944 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7f3ad807be50 con 0x7f3ae40c11f0 2024-02-21T23:16:38.632 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:16:38.632 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:16:38.632 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:16:38.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:38.646+0000 7f3aec600640 1 -- 172.21.15.28:0/3289321944 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3ac406b4d0 msgr2=0x7f3ac406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:38.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:38.646+0000 7f3aec600640 1 --2- 172.21.15.28:0/3289321944 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3ac406b4d0 0x7f3ac406d990 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f3ad0000c00 tx=0x7f3ad00079c0 comp rx=0 tx=0).stop 2024-02-21T23:16:38.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:38.646+0000 7f3aec600640 1 -- 172.21.15.28:0/3289321944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ae40c11f0 msgr2=0x7f3ae41eb8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:38.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:38.646+0000 7f3aec600640 1 --2- 172.21.15.28:0/3289321944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ae40c11f0 0x7f3ae41eb8d0 secure :-1 s=READY pgs=2646 cs=0 l=1 rev1=1 crypto rx=0x7f3ad8009eb0 tx=0x7f3ad8004a40 comp rx=0 tx=0).stop 2024-02-21T23:16:38.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:38.646+0000 7f3aec600640 1 -- 172.21.15.28:0/3289321944 shutdown_connections 2024-02-21T23:16:38.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:38.646+0000 7f3aec600640 1 --2- 172.21.15.28:0/3289321944 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3ac406b4d0 0x7f3ac406d990 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:38.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:38.646+0000 7f3aec600640 1 --2- 172.21.15.28:0/3289321944 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3ae415b6d0 0x7f3ae40bfc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:38.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:38.646+0000 7f3aec600640 1 --2- 172.21.15.28:0/3289321944 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3ae40c15d0 0x7f3ae4146c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:38.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:38.646+0000 7f3aec600640 1 --2- 172.21.15.28:0/3289321944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ae40c11f0 0x7f3ae41eb8d0 unknown :-1 s=CLOSED pgs=2646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:38.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:38.646+0000 7f3aec600640 1 -- 172.21.15.28:0/3289321944 >> 172.21.15.28:0/3289321944 conn(0x7f3ae40c67f0 msgr2=0x7f3ae40c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:16:38.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:38.647+0000 7f3aec600640 1 -- 172.21.15.28:0/3289321944 shutdown_connections 2024-02-21T23:16:38.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:38.647+0000 7f3aec600640 1 -- 172.21.15.28:0/3289321944 wait complete. 2024-02-21T23:16:39.054 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:16:39.055 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:16:39.092 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-21T23:16:39.101 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-21T23:16:40.056 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:16:40.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.296+0000 7f5edae00640 1 Processor -- start 2024-02-21T23:16:40.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.296+0000 7f5edae00640 1 -- start start 2024-02-21T23:16:40.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.298+0000 7f5edae00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f5ed406dbf0 0x7f5ed4158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:40.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.298+0000 7f5edae00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5ed4155960 0x7f5ed4159000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:40.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.298+0000 7f5edae00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ed40c74c0 con 0x7f5ed40aff90 2024-02-21T23:16:40.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.298+0000 7f5edae00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ed40c7270 con 0x7f5ed4155960 2024-02-21T23:16:40.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.298+0000 7f5edae00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ed4158630 con 0x7f5ed406dbf0 2024-02-21T23:16:40.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.298+0000 7f5eda400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5ed4155960 0x7f5ed4159000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:40.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.298+0000 7f5eda400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5ed4155960 0x7f5ed4159000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51406/0 (socket says 172.21.15.28:51406) 2024-02-21T23:16:40.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.298+0000 7f5eda400640 1 -- 172.21.15.28:0/4246828668 learned_addr learned my addr 172.21.15.28:0/4246828668 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:16:40.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.299+0000 7f5ed9000640 1 --2- 172.21.15.28:0/4246828668 >> v2:172.21.15.81:3300/0 conn(0x7f5ed406dbf0 0x7f5ed4158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:40.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.299+0000 7f5eda400640 1 -- 172.21.15.28:0/4246828668 >> v2:172.21.15.81:3300/0 conn(0x7f5ed406dbf0 msgr2=0x7f5ed4158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:16:40.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.299+0000 7f5eda400640 1 --2- 172.21.15.28:0/4246828668 >> v2:172.21.15.81:3300/0 conn(0x7f5ed406dbf0 0x7f5ed4158ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:40.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.299+0000 7f5eda400640 1 -- 172.21.15.28:0/4246828668 >> v1:172.21.15.81:6789/0 conn(0x7f5ed40aff90 legacy=0x7f5ed40c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:16:40.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.299+0000 7f5eda400640 1 -- 172.21.15.28:0/4246828668 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ed40c77a0 con 0x7f5ed4155960 2024-02-21T23:16:40.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.300+0000 7f5eda400640 1 --2- 172.21.15.28:0/4246828668 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ed4155960 0x7f5ed4159000 secure :-1 s=READY pgs=2647 cs=0 l=1 rev1=1 crypto rx=0x7f5ed40cb8c0 tx=0x7f5ecc02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb1df5f14b872df8 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:40.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.300+0000 7f5ed3e00640 1 -- 172.21.15.28:0/4246828668 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ecc02ca80 con 0x7f5ed4155960 2024-02-21T23:16:40.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.300+0000 7f5ed3e00640 1 -- 172.21.15.28:0/4246828668 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ecc02cc20 con 0x7f5ed4155960 2024-02-21T23:16:40.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.300+0000 7f5ed3e00640 1 -- 172.21.15.28:0/4246828668 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ecc0343f0 con 0x7f5ed4155960 2024-02-21T23:16:40.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.301+0000 7f5edae00640 1 -- 172.21.15.28:0/4246828668 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ed4155960 msgr2=0x7f5ed4159000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:40.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.301+0000 7f5edae00640 1 --2- 172.21.15.28:0/4246828668 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ed4155960 0x7f5ed4159000 secure :-1 s=READY pgs=2647 cs=0 l=1 rev1=1 crypto rx=0x7f5ed40cb8c0 tx=0x7f5ecc02bee0 comp rx=0 tx=0).stop 2024-02-21T23:16:40.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.301+0000 7f5edae00640 1 -- 172.21.15.28:0/4246828668 shutdown_connections 2024-02-21T23:16:40.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.301+0000 7f5edae00640 1 --2- 172.21.15.28:0/4246828668 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ed4155960 0x7f5ed4159000 unknown :-1 s=CLOSED pgs=2647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:40.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.301+0000 7f5edae00640 1 --2- 172.21.15.28:0/4246828668 >> v2:172.21.15.81:3300/0 conn(0x7f5ed406dbf0 0x7f5ed4158ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:40.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.301+0000 7f5edae00640 1 -- 172.21.15.28:0/4246828668 >> 172.21.15.28:0/4246828668 conn(0x7f5ed40ac370 msgr2=0x7f5ed40ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:16:40.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.301+0000 7f5edae00640 1 -- 172.21.15.28:0/4246828668 shutdown_connections 2024-02-21T23:16:40.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.301+0000 7f5edae00640 1 -- 172.21.15.28:0/4246828668 wait complete. 2024-02-21T23:16:40.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.303+0000 7f5edae00640 1 Processor -- start 2024-02-21T23:16:40.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.303+0000 7f5edae00640 1 -- start start 2024-02-21T23:16:40.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.304+0000 7f5edae00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5ed406dbf0 0x7f5ed40ae740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:40.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.304+0000 7f5edae00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5ed40aff90 0x7f5ed4152d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:40.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.304+0000 7f5edae00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ed4155960 0x7f5ed40c38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:40.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.304+0000 7f5edae00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed40c58f0 con 0x7f5ed4155960 2024-02-21T23:16:40.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.304+0000 7f5edae00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed40be0c0 con 0x7f5ed406dbf0 2024-02-21T23:16:40.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.304+0000 7f5edae00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed40c5a70 con 0x7f5ed40aff90 2024-02-21T23:16:40.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.304+0000 7f5eda400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ed4155960 0x7f5ed40c38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:40.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.304+0000 7f5eda400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ed4155960 0x7f5ed40c38b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51412/0 (socket says 172.21.15.28:51412) 2024-02-21T23:16:40.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.304+0000 7f5eda400640 1 -- 172.21.15.28:0/2381705566 learned_addr learned my addr 172.21.15.28:0/2381705566 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:16:40.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.304+0000 7f5ed9000640 1 --2- 172.21.15.28:0/2381705566 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5ed40aff90 0x7f5ed4152d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:40.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.304+0000 7f5ed9a00640 1 --2- 172.21.15.28:0/2381705566 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5ed406dbf0 0x7f5ed40ae740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:40.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.305+0000 7f5eda400640 1 -- 172.21.15.28:0/2381705566 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5ed40aff90 msgr2=0x7f5ed4152d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:40.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.305+0000 7f5eda400640 1 --2- 172.21.15.28:0/2381705566 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5ed40aff90 0x7f5ed4152d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:40.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.305+0000 7f5eda400640 1 -- 172.21.15.28:0/2381705566 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5ed406dbf0 msgr2=0x7f5ed40ae740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:40.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.305+0000 7f5eda400640 1 --2- 172.21.15.28:0/2381705566 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5ed406dbf0 0x7f5ed40ae740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:40.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.305+0000 7f5eda400640 1 -- 172.21.15.28:0/2381705566 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ed40af480 con 0x7f5ed4155960 2024-02-21T23:16:40.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.305+0000 7f5eda400640 1 --2- 172.21.15.28:0/2381705566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ed4155960 0x7f5ed40c38b0 secure :-1 s=READY pgs=2648 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc036040 tx=0x7f5ecc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:40.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.306+0000 7f5ed1600640 1 -- 172.21.15.28:0/2381705566 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ecc043020 con 0x7f5ed4155960 2024-02-21T23:16:40.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.306+0000 7f5ed1600640 1 -- 172.21.15.28:0/2381705566 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ecc03e450 con 0x7f5ed4155960 2024-02-21T23:16:40.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.306+0000 7f5edae00640 1 -- 172.21.15.28:0/2381705566 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ed414f370 con 0x7f5ed4155960 2024-02-21T23:16:40.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.306+0000 7f5ed1600640 1 -- 172.21.15.28:0/2381705566 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ecc009510 con 0x7f5ed4155960 2024-02-21T23:16:40.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.306+0000 7f5edae00640 1 -- 172.21.15.28:0/2381705566 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ed414ebe0 con 0x7f5ed4155960 2024-02-21T23:16:40.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.307+0000 7f5edae00640 1 -- 172.21.15.28:0/2381705566 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ed404f290 con 0x7f5ed4155960 2024-02-21T23:16:40.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.309+0000 7f5ed1600640 1 -- 172.21.15.28:0/2381705566 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5ecc046020 con 0x7f5ed4155960 2024-02-21T23:16:40.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.310+0000 7f5ed1600640 1 --2- 172.21.15.28:0/2381705566 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5eb406b4d0 0x7f5eb406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:40.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.311+0000 7f5ed1600640 1 -- 172.21.15.28:0/2381705566 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f5ecc0accf0 con 0x7f5ed4155960 2024-02-21T23:16:40.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.311+0000 7f5ed9a00640 1 --2- 172.21.15.28:0/2381705566 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5eb406b4d0 0x7f5eb406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:40.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.312+0000 7f5ed9a00640 1 --2- 172.21.15.28:0/2381705566 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5eb406b4d0 0x7f5eb406d990 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f5ec8004ea0 tx=0x7f5ec8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:40.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.315+0000 7f5ed1600640 1 -- 172.21.15.28:0/2381705566 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f5ecc04d030 con 0x7f5ed4155960 2024-02-21T23:16:40.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.555+0000 7f5edae00640 1 -- 172.21.15.28:0/2381705566 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ed406d9d0 con 0x7f5ed4155960 2024-02-21T23:16:40.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.556+0000 7f5ed1600640 1 -- 172.21.15.28:0/2381705566 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7f5ecc07b9f0 con 0x7f5ed4155960 2024-02-21T23:16:40.544 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:16:40.544 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:16:40.544 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:16:40.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.559+0000 7f5edae00640 1 -- 172.21.15.28:0/2381705566 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5eb406b4d0 msgr2=0x7f5eb406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:40.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.559+0000 7f5edae00640 1 --2- 172.21.15.28:0/2381705566 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5eb406b4d0 0x7f5eb406d990 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f5ec8004ea0 tx=0x7f5ec8009040 comp rx=0 tx=0).stop 2024-02-21T23:16:40.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.559+0000 7f5edae00640 1 -- 172.21.15.28:0/2381705566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ed4155960 msgr2=0x7f5ed40c38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:40.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.559+0000 7f5edae00640 1 --2- 172.21.15.28:0/2381705566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ed4155960 0x7f5ed40c38b0 secure :-1 s=READY pgs=2648 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc036040 tx=0x7f5ecc004a40 comp rx=0 tx=0).stop 2024-02-21T23:16:40.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.559+0000 7f5edae00640 1 -- 172.21.15.28:0/2381705566 shutdown_connections 2024-02-21T23:16:40.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.559+0000 7f5edae00640 1 --2- 172.21.15.28:0/2381705566 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f5eb406b4d0 0x7f5eb406d990 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:40.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.559+0000 7f5edae00640 1 --2- 172.21.15.28:0/2381705566 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f5ed40aff90 0x7f5ed4152d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:40.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.559+0000 7f5edae00640 1 --2- 172.21.15.28:0/2381705566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ed4155960 0x7f5ed40c38b0 unknown :-1 s=CLOSED pgs=2648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:40.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.559+0000 7f5edae00640 1 --2- 172.21.15.28:0/2381705566 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5ed406dbf0 0x7f5ed40ae740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:40.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.559+0000 7f5edae00640 1 -- 172.21.15.28:0/2381705566 >> 172.21.15.28:0/2381705566 conn(0x7f5ed40ac370 msgr2=0x7f5ed40c50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:16:40.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.559+0000 7f5edae00640 1 -- 172.21.15.28:0/2381705566 shutdown_connections 2024-02-21T23:16:40.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:40.560+0000 7f5edae00640 1 -- 172.21.15.28:0/2381705566 wait complete. 2024-02-21T23:16:40.969 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:16:40.969 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:16:41.970 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:16:42.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.208+0000 7f3b8f000640 1 Processor -- start 2024-02-21T23:16:42.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.208+0000 7f3b8f000640 1 -- start start 2024-02-21T23:16:42.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.209+0000 7f3b8f000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f3b8814d510 0x7f3b880c66a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:42.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.209+0000 7f3b8f000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3b880c3af0 0x7f3b8806f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:42.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.209+0000 7f3b8f000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b880c48f0 con 0x7f3b880c6be0 2024-02-21T23:16:42.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.209+0000 7f3b8f000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b880c4bc0 con 0x7f3b880c3af0 2024-02-21T23:16:42.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.209+0000 7f3b8f000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b880c4ea0 con 0x7f3b8814d510 2024-02-21T23:16:42.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.209+0000 7f3b87400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3b880c3af0 0x7f3b8806f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:42.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.210+0000 7f3b87400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3b880c3af0 0x7f3b8806f2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51422/0 (socket says 172.21.15.28:51422) 2024-02-21T23:16:42.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.210+0000 7f3b87400640 1 -- 172.21.15.28:0/231054788 learned_addr learned my addr 172.21.15.28:0/231054788 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:16:42.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.210+0000 7f3b87e00640 1 --2- 172.21.15.28:0/231054788 >> v2:172.21.15.81:3300/0 conn(0x7f3b8814d510 0x7f3b880c66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:42.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.210+0000 7f3b87400640 1 -- 172.21.15.28:0/231054788 >> v2:172.21.15.81:3300/0 conn(0x7f3b8814d510 msgr2=0x7f3b880c66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:16:42.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.210+0000 7f3b87400640 1 --2- 172.21.15.28:0/231054788 >> v2:172.21.15.81:3300/0 conn(0x7f3b8814d510 0x7f3b880c66a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:42.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.210+0000 7f3b87400640 1 -- 172.21.15.28:0/231054788 >> v1:172.21.15.81:6789/0 conn(0x7f3b880c6be0 legacy=0x7f3b8806f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:16:42.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.210+0000 7f3b87400640 1 -- 172.21.15.28:0/231054788 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b88153b30 con 0x7f3b880c3af0 2024-02-21T23:16:42.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.211+0000 7f3b87400640 1 --2- 172.21.15.28:0/231054788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b880c3af0 0x7f3b8806f2c0 secure :-1 s=READY pgs=2649 cs=0 l=1 rev1=1 crypto rx=0x7f3b8806e560 tx=0x7f3b7002b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8dc2cc2fa8f73c4 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:42.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.211+0000 7f3b86a00640 1 -- 172.21.15.28:0/231054788 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b70038070 con 0x7f3b880c3af0 2024-02-21T23:16:42.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.212+0000 7f3b86a00640 1 -- 172.21.15.28:0/231054788 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b7002f050 con 0x7f3b880c3af0 2024-02-21T23:16:42.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.212+0000 7f3b86a00640 1 -- 172.21.15.28:0/231054788 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b70031970 con 0x7f3b880c3af0 2024-02-21T23:16:42.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.212+0000 7f3b8f000640 1 -- 172.21.15.28:0/231054788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b880c3af0 msgr2=0x7f3b8806f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:42.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.212+0000 7f3b8f000640 1 --2- 172.21.15.28:0/231054788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b880c3af0 0x7f3b8806f2c0 secure :-1 s=READY pgs=2649 cs=0 l=1 rev1=1 crypto rx=0x7f3b8806e560 tx=0x7f3b7002b7f0 comp rx=0 tx=0).stop 2024-02-21T23:16:42.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.212+0000 7f3b8f000640 1 -- 172.21.15.28:0/231054788 shutdown_connections 2024-02-21T23:16:42.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.212+0000 7f3b8f000640 1 --2- 172.21.15.28:0/231054788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b880c3af0 0x7f3b8806f2c0 unknown :-1 s=CLOSED pgs=2649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:42.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.212+0000 7f3b8f000640 1 --2- 172.21.15.28:0/231054788 >> v2:172.21.15.81:3300/0 conn(0x7f3b8814d510 0x7f3b880c66a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:42.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.212+0000 7f3b8f000640 1 -- 172.21.15.28:0/231054788 >> 172.21.15.28:0/231054788 conn(0x7f3b880aa550 msgr2=0x7f3b880aa960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:16:42.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.212+0000 7f3b8f000640 1 -- 172.21.15.28:0/231054788 shutdown_connections 2024-02-21T23:16:42.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.212+0000 7f3b8f000640 1 -- 172.21.15.28:0/231054788 wait complete. 2024-02-21T23:16:42.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.214+0000 7f3b8f000640 1 Processor -- start 2024-02-21T23:16:42.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.214+0000 7f3b8f000640 1 -- start start 2024-02-21T23:16:42.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.215+0000 7f3b8f000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b880c3af0 0x7f3b8814b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:42.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.215+0000 7f3b8f000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3b880c6be0 0x7f3b88152820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:42.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.215+0000 7f3b8f000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b8814d510 0x7f3b881ee940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:42.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.215+0000 7f3b8f000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b88070090 con 0x7f3b8814d510 2024-02-21T23:16:42.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.215+0000 7f3b8f000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b8806ff10 con 0x7f3b880c3af0 2024-02-21T23:16:42.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.215+0000 7f3b8f000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b880c3fa0 con 0x7f3b880c6be0 2024-02-21T23:16:42.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.215+0000 7f3b8d000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b8814d510 0x7f3b881ee940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:42.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.215+0000 7f3b8d000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b8814d510 0x7f3b881ee940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51436/0 (socket says 172.21.15.28:51436) 2024-02-21T23:16:42.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.215+0000 7f3b8d000640 1 -- 172.21.15.28:0/2190936809 learned_addr learned my addr 172.21.15.28:0/2190936809 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:16:42.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.216+0000 7f3b87e00640 1 --2- 172.21.15.28:0/2190936809 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b880c3af0 0x7f3b8814b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:42.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.216+0000 7f3b87400640 1 --2- 172.21.15.28:0/2190936809 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3b880c6be0 0x7f3b88152820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:42.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.216+0000 7f3b8d000640 1 -- 172.21.15.28:0/2190936809 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3b880c6be0 msgr2=0x7f3b88152820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:42.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.216+0000 7f3b8d000640 1 --2- 172.21.15.28:0/2190936809 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3b880c6be0 0x7f3b88152820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:42.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.216+0000 7f3b8d000640 1 -- 172.21.15.28:0/2190936809 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b880c3af0 msgr2=0x7f3b8814b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:42.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.216+0000 7f3b8d000640 1 --2- 172.21.15.28:0/2190936809 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b880c3af0 0x7f3b8814b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:42.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.216+0000 7f3b8d000640 1 -- 172.21.15.28:0/2190936809 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b88148950 con 0x7f3b8814d510 2024-02-21T23:16:42.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.216+0000 7f3b8d000640 1 --2- 172.21.15.28:0/2190936809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b8814d510 0x7f3b881ee940 secure :-1 s=READY pgs=2650 cs=0 l=1 rev1=1 crypto rx=0x7f3b8000c440 tx=0x7f3b8000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:42.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.217+0000 7f3b7fe00640 1 -- 172.21.15.28:0/2190936809 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b80011790 con 0x7f3b8814d510 2024-02-21T23:16:42.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.217+0000 7f3b7fe00640 1 -- 172.21.15.28:0/2190936809 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b80017070 con 0x7f3b8814d510 2024-02-21T23:16:42.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.217+0000 7f3b7fe00640 1 -- 172.21.15.28:0/2190936809 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b80011b70 con 0x7f3b8814d510 2024-02-21T23:16:42.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.217+0000 7f3b8f000640 1 -- 172.21.15.28:0/2190936809 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b88148c30 con 0x7f3b8814d510 2024-02-21T23:16:42.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.217+0000 7f3b8f000640 1 -- 172.21.15.28:0/2190936809 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b881df410 con 0x7f3b8814d510 2024-02-21T23:16:42.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.218+0000 7f3b8f000640 1 -- 172.21.15.28:0/2190936809 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b88068640 con 0x7f3b8814d510 2024-02-21T23:16:42.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.219+0000 7f3b7fe00640 1 -- 172.21.15.28:0/2190936809 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3b80011d10 con 0x7f3b8814d510 2024-02-21T23:16:42.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.220+0000 7f3b7fe00640 1 --2- 172.21.15.28:0/2190936809 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3b6406b4d0 0x7f3b6406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:42.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.220+0000 7f3b7fe00640 1 -- 172.21.15.28:0/2190936809 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f3b8008a290 con 0x7f3b8814d510 2024-02-21T23:16:42.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.221+0000 7f3b87e00640 1 --2- 172.21.15.28:0/2190936809 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3b6406b4d0 0x7f3b6406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:42.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.222+0000 7f3b87e00640 1 --2- 172.21.15.28:0/2190936809 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3b6406b4d0 0x7f3b6406d990 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f3b780099c0 tx=0x7f3b78003990 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:42.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.224+0000 7f3b7fe00640 1 -- 172.21.15.28:0/2190936809 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3b80058e80 con 0x7f3b8814d510 2024-02-21T23:16:42.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.472+0000 7f3b8f000640 1 -- 172.21.15.28:0/2190936809 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3b880c3200 con 0x7f3b8814d510 2024-02-21T23:16:42.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.473+0000 7f3b7fe00640 1 -- 172.21.15.28:0/2190936809 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7f3b80058780 con 0x7f3b8814d510 2024-02-21T23:16:42.461 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:16:42.461 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:16:42.461 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:16:42.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.476+0000 7f3b8f000640 1 -- 172.21.15.28:0/2190936809 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3b6406b4d0 msgr2=0x7f3b6406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:42.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.476+0000 7f3b8f000640 1 --2- 172.21.15.28:0/2190936809 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3b6406b4d0 0x7f3b6406d990 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f3b780099c0 tx=0x7f3b78003990 comp rx=0 tx=0).stop 2024-02-21T23:16:42.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.476+0000 7f3b8f000640 1 -- 172.21.15.28:0/2190936809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b8814d510 msgr2=0x7f3b881ee940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:42.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.476+0000 7f3b8f000640 1 --2- 172.21.15.28:0/2190936809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b8814d510 0x7f3b881ee940 secure :-1 s=READY pgs=2650 cs=0 l=1 rev1=1 crypto rx=0x7f3b8000c440 tx=0x7f3b8000c900 comp rx=0 tx=0).stop 2024-02-21T23:16:42.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.476+0000 7f3b8f000640 1 -- 172.21.15.28:0/2190936809 shutdown_connections 2024-02-21T23:16:42.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.476+0000 7f3b8f000640 1 --2- 172.21.15.28:0/2190936809 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3b6406b4d0 0x7f3b6406d990 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:42.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.476+0000 7f3b8f000640 1 --2- 172.21.15.28:0/2190936809 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3b880c6be0 0x7f3b88152820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:42.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.476+0000 7f3b8f000640 1 --2- 172.21.15.28:0/2190936809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b8814d510 0x7f3b881ee940 unknown :-1 s=CLOSED pgs=2650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:42.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.476+0000 7f3b8f000640 1 --2- 172.21.15.28:0/2190936809 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3b880c3af0 0x7f3b8814b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:42.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.476+0000 7f3b8f000640 1 -- 172.21.15.28:0/2190936809 >> 172.21.15.28:0/2190936809 conn(0x7f3b880aa550 msgr2=0x7f3b880c1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:16:42.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.476+0000 7f3b8f000640 1 -- 172.21.15.28:0/2190936809 shutdown_connections 2024-02-21T23:16:42.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:42.476+0000 7f3b8f000640 1 -- 172.21.15.28:0/2190936809 wait complete. 2024-02-21T23:16:42.886 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:16:42.886 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:16:43.887 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:16:44.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.126+0000 7f6a79a00640 1 Processor -- start 2024-02-21T23:16:44.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.127+0000 7f6a79a00640 1 -- start start 2024-02-21T23:16:44.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.128+0000 7f6a79a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6a7414e8d0 0x7f6a7406c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:44.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.128+0000 7f6a79a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f6a740724f0 0x7f6a74151f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:44.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.128+0000 7f6a79a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6a740c1860 0x7f6a740718c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:44.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.128+0000 7f6a79a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a74155510 con 0x7f6a7414e8d0 2024-02-21T23:16:44.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.128+0000 7f6a79a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a741557f0 con 0x7f6a740724f0 2024-02-21T23:16:44.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.128+0000 7f6a79a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a74151480 con 0x7f6a740c1860 2024-02-21T23:16:44.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.129+0000 7f6a73e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6a7414e8d0 0x7f6a7406c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:44.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.129+0000 7f6a73e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6a7414e8d0 0x7f6a7406c360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51444/0 (socket says 172.21.15.28:51444) 2024-02-21T23:16:44.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.129+0000 7f6a73e00640 1 -- 172.21.15.28:0/569166387 learned_addr learned my addr 172.21.15.28:0/569166387 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:16:44.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.129+0000 7f6a73400640 1 --2- 172.21.15.28:0/569166387 >> v2:172.21.15.81:3300/0 conn(0x7f6a740724f0 0x7f6a74151f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:44.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.129+0000 7f6a79000640 1 --2- 172.21.15.28:0/569166387 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6a740c1860 0x7f6a740718c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:44.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.129+0000 7f6a73e00640 1 -- 172.21.15.28:0/569166387 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6a740c1860 msgr2=0x7f6a740718c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:16:44.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.129+0000 7f6a73e00640 1 --2- 172.21.15.28:0/569166387 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6a740c1860 0x7f6a740718c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:44.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.129+0000 7f6a73e00640 1 -- 172.21.15.28:0/569166387 >> v2:172.21.15.81:3300/0 conn(0x7f6a740724f0 msgr2=0x7f6a74151f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:16:44.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.129+0000 7f6a73e00640 1 --2- 172.21.15.28:0/569166387 >> v2:172.21.15.81:3300/0 conn(0x7f6a740724f0 0x7f6a74151f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:44.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.129+0000 7f6a73e00640 1 -- 172.21.15.28:0/569166387 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a74151760 con 0x7f6a7414e8d0 2024-02-21T23:16:44.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.129+0000 7f6a73400640 1 --2- 172.21.15.28:0/569166387 >> v2:172.21.15.81:3300/0 conn(0x7f6a740724f0 0x7f6a74151f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:16:44.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.130+0000 7f6a73e00640 1 --2- 172.21.15.28:0/569166387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a7414e8d0 0x7f6a7406c360 secure :-1 s=READY pgs=2651 cs=0 l=1 rev1=1 crypto rx=0x7f6a740678c0 tx=0x7f6a5c02dfb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0eb154a4fdbb7a4 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:44.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.131+0000 7f6a72a00640 1 -- 172.21.15.28:0/569166387 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a5c03b070 con 0x7f6a7414e8d0 2024-02-21T23:16:44.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.131+0000 7f6a72a00640 1 -- 172.21.15.28:0/569166387 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a5c02ed60 con 0x7f6a7414e8d0 2024-02-21T23:16:44.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.131+0000 7f6a72a00640 1 -- 172.21.15.28:0/569166387 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a5c036660 con 0x7f6a7414e8d0 2024-02-21T23:16:44.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.131+0000 7f6a79a00640 1 -- 172.21.15.28:0/569166387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a7414e8d0 msgr2=0x7f6a7406c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:44.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.131+0000 7f6a79a00640 1 --2- 172.21.15.28:0/569166387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a7414e8d0 0x7f6a7406c360 secure :-1 s=READY pgs=2651 cs=0 l=1 rev1=1 crypto rx=0x7f6a740678c0 tx=0x7f6a5c02dfb0 comp rx=0 tx=0).stop 2024-02-21T23:16:44.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.131+0000 7f6a79a00640 1 -- 172.21.15.28:0/569166387 shutdown_connections 2024-02-21T23:16:44.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.131+0000 7f6a79a00640 1 --2- 172.21.15.28:0/569166387 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6a740c1860 0x7f6a740718c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:44.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.131+0000 7f6a79a00640 1 --2- 172.21.15.28:0/569166387 >> v2:172.21.15.81:3300/0 conn(0x7f6a740724f0 0x7f6a74151f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:44.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.131+0000 7f6a79a00640 1 --2- 172.21.15.28:0/569166387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a7414e8d0 0x7f6a7406c360 unknown :-1 s=CLOSED pgs=2651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:44.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.131+0000 7f6a79a00640 1 -- 172.21.15.28:0/569166387 >> 172.21.15.28:0/569166387 conn(0x7f6a740c3960 msgr2=0x7f6a740c3d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:16:44.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.132+0000 7f6a79a00640 1 -- 172.21.15.28:0/569166387 shutdown_connections 2024-02-21T23:16:44.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.132+0000 7f6a79a00640 1 -- 172.21.15.28:0/569166387 wait complete. 2024-02-21T23:16:44.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.133+0000 7f6a79a00640 1 Processor -- start 2024-02-21T23:16:44.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.133+0000 7f6a79a00640 1 -- start start 2024-02-21T23:16:44.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.134+0000 7f6a79a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6a740724f0 0x7f6a741eb270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:44.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.134+0000 7f6a79a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6a740c1860 0x7f6a740c4a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:44.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.134+0000 7f6a79a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a7414e8d0 0x7f6a740c0290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:44.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.134+0000 7f6a79a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a7414ee30 con 0x7f6a7414e8d0 2024-02-21T23:16:44.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.134+0000 7f6a79a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a7414ecb0 con 0x7f6a740724f0 2024-02-21T23:16:44.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.134+0000 7f6a79a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a740c1cb0 con 0x7f6a740c1860 2024-02-21T23:16:44.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.135+0000 7f6a79000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a7414e8d0 0x7f6a740c0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:44.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.135+0000 7f6a79000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a7414e8d0 0x7f6a740c0290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51452/0 (socket says 172.21.15.28:51452) 2024-02-21T23:16:44.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.135+0000 7f6a79000640 1 -- 172.21.15.28:0/877996088 learned_addr learned my addr 172.21.15.28:0/877996088 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:16:44.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.135+0000 7f6a73e00640 1 --2- 172.21.15.28:0/877996088 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6a740724f0 0x7f6a741eb270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:44.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.135+0000 7f6a73400640 1 --2- 172.21.15.28:0/877996088 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6a740c1860 0x7f6a740c4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:44.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.135+0000 7f6a79000640 1 -- 172.21.15.28:0/877996088 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6a740c1860 msgr2=0x7f6a740c4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:44.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.135+0000 7f6a79000640 1 --2- 172.21.15.28:0/877996088 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6a740c1860 0x7f6a740c4a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:44.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.135+0000 7f6a79000640 1 -- 172.21.15.28:0/877996088 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6a740724f0 msgr2=0x7f6a741eb270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:44.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.135+0000 7f6a79000640 1 --2- 172.21.15.28:0/877996088 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6a740724f0 0x7f6a741eb270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:44.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.135+0000 7f6a79000640 1 -- 172.21.15.28:0/877996088 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a7414a950 con 0x7f6a7414e8d0 2024-02-21T23:16:44.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.135+0000 7f6a73400640 1 --2- 172.21.15.28:0/877996088 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6a740c1860 0x7f6a740c4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:16:44.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.136+0000 7f6a79000640 1 --2- 172.21.15.28:0/877996088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a7414e8d0 0x7f6a740c0290 secure :-1 s=READY pgs=2652 cs=0 l=1 rev1=1 crypto rx=0x7f6a6c00c550 tx=0x7f6a6c00ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:44.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.136+0000 7f6a6be00640 1 -- 172.21.15.28:0/877996088 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a6c015070 con 0x7f6a7414e8d0 2024-02-21T23:16:44.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.136+0000 7f6a6be00640 1 -- 172.21.15.28:0/877996088 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a6c004220 con 0x7f6a7414e8d0 2024-02-21T23:16:44.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.136+0000 7f6a79a00640 1 -- 172.21.15.28:0/877996088 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a7414d300 con 0x7f6a7414e8d0 2024-02-21T23:16:44.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.136+0000 7f6a6be00640 1 -- 172.21.15.28:0/877996088 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a6c003610 con 0x7f6a7414e8d0 2024-02-21T23:16:44.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.136+0000 7f6a79a00640 1 -- 172.21.15.28:0/877996088 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a741eb7b0 con 0x7f6a7414e8d0 2024-02-21T23:16:44.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.137+0000 7f6a79a00640 1 -- 172.21.15.28:0/877996088 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a7414d0f0 con 0x7f6a7414e8d0 2024-02-21T23:16:44.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.138+0000 7f6a6be00640 1 -- 172.21.15.28:0/877996088 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6a6c0037b0 con 0x7f6a7414e8d0 2024-02-21T23:16:44.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.139+0000 7f6a6be00640 1 --2- 172.21.15.28:0/877996088 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6a5006b4d0 0x7f6a5006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:44.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.139+0000 7f6a6be00640 1 -- 172.21.15.28:0/877996088 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f6a6c08a300 con 0x7f6a7414e8d0 2024-02-21T23:16:44.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.140+0000 7f6a73e00640 1 --2- 172.21.15.28:0/877996088 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6a5006b4d0 0x7f6a5006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:44.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.141+0000 7f6a73e00640 1 --2- 172.21.15.28:0/877996088 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6a5006b4d0 0x7f6a5006d990 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c038000 tx=0x7f6a5c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:44.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.149+0000 7f6a6be00640 1 -- 172.21.15.28:0/877996088 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f6a6c058e50 con 0x7f6a7414e8d0 2024-02-21T23:16:44.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.390+0000 7f6a79a00640 1 -- 172.21.15.28:0/877996088 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6a7404f290 con 0x7f6a7414e8d0 2024-02-21T23:16:44.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.391+0000 7f6a6be00640 1 -- 172.21.15.28:0/877996088 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7f6a6c08a0c0 con 0x7f6a7414e8d0 2024-02-21T23:16:44.378 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:16:44.379 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:16:44.379 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:16:44.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.394+0000 7f6a79a00640 1 -- 172.21.15.28:0/877996088 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6a5006b4d0 msgr2=0x7f6a5006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:44.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.394+0000 7f6a79a00640 1 --2- 172.21.15.28:0/877996088 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6a5006b4d0 0x7f6a5006d990 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c038000 tx=0x7f6a5c008040 comp rx=0 tx=0).stop 2024-02-21T23:16:44.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.394+0000 7f6a79a00640 1 -- 172.21.15.28:0/877996088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a7414e8d0 msgr2=0x7f6a740c0290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:44.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.395+0000 7f6a79a00640 1 --2- 172.21.15.28:0/877996088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a7414e8d0 0x7f6a740c0290 secure :-1 s=READY pgs=2652 cs=0 l=1 rev1=1 crypto rx=0x7f6a6c00c550 tx=0x7f6a6c00ca10 comp rx=0 tx=0).stop 2024-02-21T23:16:44.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.395+0000 7f6a79a00640 1 -- 172.21.15.28:0/877996088 shutdown_connections 2024-02-21T23:16:44.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.395+0000 7f6a79a00640 1 --2- 172.21.15.28:0/877996088 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f6a5006b4d0 0x7f6a5006d990 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:44.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.395+0000 7f6a79a00640 1 --2- 172.21.15.28:0/877996088 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6a740c1860 0x7f6a740c4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:44.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.395+0000 7f6a79a00640 1 --2- 172.21.15.28:0/877996088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a7414e8d0 0x7f6a740c0290 unknown :-1 s=CLOSED pgs=2652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:44.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.395+0000 7f6a79a00640 1 --2- 172.21.15.28:0/877996088 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6a740724f0 0x7f6a741eb270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:44.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.395+0000 7f6a79a00640 1 -- 172.21.15.28:0/877996088 >> 172.21.15.28:0/877996088 conn(0x7f6a740c3960 msgr2=0x7f6a7414b500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:16:44.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.395+0000 7f6a79a00640 1 -- 172.21.15.28:0/877996088 shutdown_connections 2024-02-21T23:16:44.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:44.395+0000 7f6a79a00640 1 -- 172.21.15.28:0/877996088 wait complete. 2024-02-21T23:16:44.809 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:16:44.810 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:16:45.811 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:16:46.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.036+0000 7fd3eec00640 1 Processor -- start 2024-02-21T23:16:46.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.036+0000 7fd3eec00640 1 -- start start 2024-02-21T23:16:46.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.038+0000 7fd3eec00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fd3e80aff90 0x7fd3e80c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:46.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.038+0000 7fd3eec00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3e80c74c0 con 0x7fd3e80c5720 2024-02-21T23:16:46.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.038+0000 7fd3eec00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3e80c77a0 con 0x7fd3e806dbf0 2024-02-21T23:16:46.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.038+0000 7fd3eec00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3e80c7270 con 0x7fd3e80aff90 2024-02-21T23:16:46.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.038+0000 7fd3ee200640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd3e80c5720 0x7fd3e815f890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:51684/0 (socket says 172.21.15.28:51684) 2024-02-21T23:16:46.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.038+0000 7fd3ee200640 1 -- 172.21.15.28:0/1700072974 learned_addr learned my addr 172.21.15.28:0/1700072974 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:16:46.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.038+0000 7fd3ed800640 1 --2- 172.21.15.28:0/1700072974 >> v2:172.21.15.81:3300/0 conn(0x7fd3e80aff90 0x7fd3e80c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:46.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.039+0000 7fd3e7e00640 1 -- 172.21.15.28:0/1700072974 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4034331863 0 0) 0x7fd3e80c74c0 con 0x7fd3e80c5720 2024-02-21T23:16:46.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.039+0000 7fd3e7e00640 1 -- 172.21.15.28:0/1700072974 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3d0003520 con 0x7fd3e80c5720 2024-02-21T23:16:46.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.039+0000 7fd3e7e00640 1 -- 172.21.15.28:0/1700072974 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3155266102 0 0) 0x7fd3d0003520 con 0x7fd3e80c5720 2024-02-21T23:16:46.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.039+0000 7fd3e7e00640 1 -- 172.21.15.28:0/1700072974 >> v2:172.21.15.81:3300/0 conn(0x7fd3e80aff90 msgr2=0x7fd3e80c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:16:46.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.039+0000 7fd3e7e00640 1 --2- 172.21.15.28:0/1700072974 >> v2:172.21.15.81:3300/0 conn(0x7fd3e80aff90 0x7fd3e80c51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:46.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.039+0000 7fd3e7e00640 1 -- 172.21.15.28:0/1700072974 >> v1:172.21.15.81:6789/0 conn(0x7fd3e806dbf0 legacy=0x7fd3e8158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:46.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.039+0000 7fd3e7e00640 1 -- 172.21.15.28:0/1700072974 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3e80c7a80 con 0x7fd3e80c5720 2024-02-21T23:16:46.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.039+0000 7fd3e7e00640 1 -- 172.21.15.28:0/1700072974 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fd3e00019f0 con 0x7fd3e80c5720 2024-02-21T23:16:46.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.039+0000 7fd3e7e00640 1 -- 172.21.15.28:0/1700072974 >> v1:172.21.15.28:6789/0 conn(0x7fd3e80c5720 legacy=0x7fd3e815f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:46.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.039+0000 7fd3e7e00640 1 --2- 172.21.15.28:0/1700072974 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd3d0003a00 0x7fd3d0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:46.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.039+0000 7fd3e7e00640 1 --2- 172.21.15.28:0/1700072974 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd3d0004800 0x7fd3d0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:46.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.039+0000 7fd3e7e00640 1 --2- 172.21.15.28:0/1700072974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3d00051c0 0x7fd3d000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:46.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.039+0000 7fd3e7e00640 1 -- 172.21.15.28:0/1700072974 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e80c7270 con 0x7fd3d00051c0 2024-02-21T23:16:46.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.039+0000 7fd3e7e00640 1 -- 172.21.15.28:0/1700072974 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d000c0c0 con 0x7fd3d0004800 2024-02-21T23:16:46.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.039+0000 7fd3e7e00640 1 -- 172.21.15.28:0/1700072974 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d000c2c0 con 0x7fd3d0003a00 2024-02-21T23:16:46.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.040+0000 7fd3ee200640 1 --2- 172.21.15.28:0/1700072974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3d00051c0 0x7fd3d000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:46.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.040+0000 7fd3ece00640 1 --2- 172.21.15.28:0/1700072974 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd3d0004800 0x7fd3d0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:46.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.040+0000 7fd3ed800640 1 --2- 172.21.15.28:0/1700072974 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd3d0003a00 0x7fd3d0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:46.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.040+0000 7fd3ee200640 1 -- 172.21.15.28:0/1700072974 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd3d0003a00 msgr2=0x7fd3d0003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:16:46.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.040+0000 7fd3ee200640 1 --2- 172.21.15.28:0/1700072974 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd3d0003a00 0x7fd3d0003e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:46.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.040+0000 7fd3ee200640 1 -- 172.21.15.28:0/1700072974 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd3d0004800 msgr2=0x7fd3d0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:16:46.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.040+0000 7fd3ee200640 1 --2- 172.21.15.28:0/1700072974 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd3d0004800 0x7fd3d0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:46.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.040+0000 7fd3ee200640 1 -- 172.21.15.28:0/1700072974 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3d0003520 con 0x7fd3d00051c0 2024-02-21T23:16:46.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.041+0000 7fd3ee200640 1 -- 172.21.15.28:0/1700072974 reap_dead start 2024-02-21T23:16:46.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.041+0000 7fd3ee200640 1 --2- 172.21.15.28:0/1700072974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3d00051c0 0x7fd3d000b9c0 secure :-1 s=READY pgs=2654 cs=0 l=1 rev1=1 crypto rx=0x7fd3e000a7b0 tx=0x7fd3d0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17e3ec3209c0369c server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:46.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.042+0000 7fd3e7e00640 1 -- 172.21.15.28:0/1700072974 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e000a3d0 con 0x7fd3d00051c0 2024-02-21T23:16:46.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.042+0000 7fd3e7e00640 1 -- 172.21.15.28:0/1700072974 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3e0034650 con 0x7fd3d00051c0 2024-02-21T23:16:46.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.042+0000 7fd3e7e00640 1 -- 172.21.15.28:0/1700072974 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e0034a40 con 0x7fd3d00051c0 2024-02-21T23:16:46.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.042+0000 7fd3eec00640 1 -- 172.21.15.28:0/1700072974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3d00051c0 msgr2=0x7fd3d000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:46.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.042+0000 7fd3eec00640 1 --2- 172.21.15.28:0/1700072974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3d00051c0 0x7fd3d000b9c0 secure :-1 s=READY pgs=2654 cs=0 l=1 rev1=1 crypto rx=0x7fd3e000a7b0 tx=0x7fd3d0005830 comp rx=0 tx=0).stop 2024-02-21T23:16:46.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.043+0000 7fd3eec00640 1 -- 172.21.15.28:0/1700072974 shutdown_connections 2024-02-21T23:16:46.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.043+0000 7fd3eec00640 1 --2- 172.21.15.28:0/1700072974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3d00051c0 0x7fd3d000b9c0 unknown :-1 s=CLOSED pgs=2654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:46.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.043+0000 7fd3eec00640 1 -- 172.21.15.28:0/1700072974 >> 172.21.15.28:0/1700072974 conn(0x7fd3e80ac370 msgr2=0x7fd3e80ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:16:46.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.043+0000 7fd3eec00640 1 -- 172.21.15.28:0/1700072974 shutdown_connections 2024-02-21T23:16:46.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.043+0000 7fd3eec00640 1 -- 172.21.15.28:0/1700072974 wait complete. 2024-02-21T23:16:46.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.044+0000 7fd3eec00640 1 Processor -- start 2024-02-21T23:16:46.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.044+0000 7fd3eec00640 1 -- start start 2024-02-21T23:16:46.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.045+0000 7fd3eec00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd3d00051c0 0x7fd3e8153990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:46.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.045+0000 7fd3eec00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd3e81543f0 0x7fd3e80ae740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:46.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.045+0000 7fd3eec00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3e80c4140 0x7fd3e80c45c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:46.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.045+0000 7fd3eec00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e80c8760 con 0x7fd3e80c4140 2024-02-21T23:16:46.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.045+0000 7fd3eec00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e8158630 con 0x7fd3e81543f0 2024-02-21T23:16:46.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.045+0000 7fd3eec00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e80c88e0 con 0x7fd3d00051c0 2024-02-21T23:16:46.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.046+0000 7fd3ee200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3e80c4140 0x7fd3e80c45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:46.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.046+0000 7fd3ee200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3e80c4140 0x7fd3e80c45c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60456/0 (socket says 172.21.15.28:60456) 2024-02-21T23:16:46.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.046+0000 7fd3ee200640 1 -- 172.21.15.28:0/1590654919 learned_addr learned my addr 172.21.15.28:0/1590654919 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:16:46.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.046+0000 7fd3ed800640 1 --2- 172.21.15.28:0/1590654919 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd3d00051c0 0x7fd3e8153990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:46.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.046+0000 7fd3ece00640 1 --2- 172.21.15.28:0/1590654919 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd3e81543f0 0x7fd3e80ae740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:46.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.046+0000 7fd3ee200640 1 -- 172.21.15.28:0/1590654919 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd3d00051c0 msgr2=0x7fd3e8153990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:46.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.046+0000 7fd3ee200640 1 --2- 172.21.15.28:0/1590654919 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd3d00051c0 0x7fd3e8153990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:46.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.046+0000 7fd3ee200640 1 -- 172.21.15.28:0/1590654919 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd3e81543f0 msgr2=0x7fd3e80ae740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:46.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.046+0000 7fd3ee200640 1 --2- 172.21.15.28:0/1590654919 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd3e81543f0 0x7fd3e80ae740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:46.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.046+0000 7fd3ee200640 1 -- 172.21.15.28:0/1590654919 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3e8154eb0 con 0x7fd3e80c4140 2024-02-21T23:16:46.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.047+0000 7fd3ee200640 1 --2- 172.21.15.28:0/1590654919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3e80c4140 0x7fd3e80c45c0 secure :-1 s=READY pgs=2655 cs=0 l=1 rev1=1 crypto rx=0x7fd3e000a8a0 tx=0x7fd3e0001cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:46.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.047+0000 7fd3e5600640 1 -- 172.21.15.28:0/1590654919 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e0044070 con 0x7fd3e80c4140 2024-02-21T23:16:46.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.047+0000 7fd3e5600640 1 -- 172.21.15.28:0/1590654919 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3e0004e20 con 0x7fd3e80c4140 2024-02-21T23:16:46.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.047+0000 7fd3e5600640 1 -- 172.21.15.28:0/1590654919 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e0039040 con 0x7fd3e80c4140 2024-02-21T23:16:46.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.047+0000 7fd3eec00640 1 -- 172.21.15.28:0/1590654919 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3e80be950 con 0x7fd3e80c4140 2024-02-21T23:16:46.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.047+0000 7fd3eec00640 1 -- 172.21.15.28:0/1590654919 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3e80bebf0 con 0x7fd3e80c4140 2024-02-21T23:16:46.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.048+0000 7fd3eec00640 1 -- 172.21.15.28:0/1590654919 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3e804f290 con 0x7fd3e80c4140 2024-02-21T23:16:46.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.049+0000 7fd3e5600640 1 -- 172.21.15.28:0/1590654919 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd3e0047070 con 0x7fd3e80c4140 2024-02-21T23:16:46.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.050+0000 7fd3e5600640 1 --2- 172.21.15.28:0/1590654919 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd3c806b4d0 0x7fd3c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:46.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.050+0000 7fd3e5600640 1 -- 172.21.15.28:0/1590654919 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fd3e00abd30 con 0x7fd3e80c4140 2024-02-21T23:16:46.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.051+0000 7fd3ed800640 1 --2- 172.21.15.28:0/1590654919 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd3c806b4d0 0x7fd3c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:46.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.052+0000 7fd3ed800640 1 --2- 172.21.15.28:0/1590654919 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd3c806b4d0 0x7fd3c806d990 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc009a10 tx=0x7fd3dc008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:46.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.054+0000 7fd3e5600640 1 -- 172.21.15.28:0/1590654919 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd3e007a2c0 con 0x7fd3e80c4140 2024-02-21T23:16:46.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.295+0000 7fd3eec00640 1 -- 172.21.15.28:0/1590654919 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd3e806d9d0 con 0x7fd3e80c4140 2024-02-21T23:16:46.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.296+0000 7fd3e5600640 1 -- 172.21.15.28:0/1590654919 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7fd3e007e360 con 0x7fd3e80c4140 2024-02-21T23:16:46.283 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:16:46.283 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:16:46.283 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:16:46.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.299+0000 7fd3eec00640 1 -- 172.21.15.28:0/1590654919 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd3c806b4d0 msgr2=0x7fd3c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:46.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.299+0000 7fd3eec00640 1 --2- 172.21.15.28:0/1590654919 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd3c806b4d0 0x7fd3c806d990 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc009a10 tx=0x7fd3dc008040 comp rx=0 tx=0).stop 2024-02-21T23:16:46.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.299+0000 7fd3eec00640 1 -- 172.21.15.28:0/1590654919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3e80c4140 msgr2=0x7fd3e80c45c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:46.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.299+0000 7fd3eec00640 1 --2- 172.21.15.28:0/1590654919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3e80c4140 0x7fd3e80c45c0 secure :-1 s=READY pgs=2655 cs=0 l=1 rev1=1 crypto rx=0x7fd3e000a8a0 tx=0x7fd3e0001cf0 comp rx=0 tx=0).stop 2024-02-21T23:16:46.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.299+0000 7fd3eec00640 1 -- 172.21.15.28:0/1590654919 shutdown_connections 2024-02-21T23:16:46.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.299+0000 7fd3eec00640 1 --2- 172.21.15.28:0/1590654919 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd3c806b4d0 0x7fd3c806d990 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:46.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.299+0000 7fd3eec00640 1 --2- 172.21.15.28:0/1590654919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3e80c4140 0x7fd3e80c45c0 unknown :-1 s=CLOSED pgs=2655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:46.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.299+0000 7fd3eec00640 1 --2- 172.21.15.28:0/1590654919 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd3e81543f0 0x7fd3e80ae740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:46.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.299+0000 7fd3eec00640 1 --2- 172.21.15.28:0/1590654919 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd3d00051c0 0x7fd3e8153990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:46.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.299+0000 7fd3eec00640 1 -- 172.21.15.28:0/1590654919 >> 172.21.15.28:0/1590654919 conn(0x7fd3e80ac370 msgr2=0x7fd3e80ac750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:16:46.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.299+0000 7fd3eec00640 1 -- 172.21.15.28:0/1590654919 shutdown_connections 2024-02-21T23:16:46.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:46.299+0000 7fd3eec00640 1 -- 172.21.15.28:0/1590654919 wait complete. 2024-02-21T23:16:46.710 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:16:46.710 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:16:47.712 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:16:47.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.942+0000 7f6292800640 1 Processor -- start 2024-02-21T23:16:47.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.942+0000 7f6292800640 1 -- start start 2024-02-21T23:16:47.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.944+0000 7f6292800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f628c0c4d20 0x7f628c0a9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:47.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.944+0000 7f6292800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f628c06b6d0 con 0x7f628c06bad0 2024-02-21T23:16:47.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.944+0000 7f6292800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f628c06fc80 con 0x7f628c06f130 2024-02-21T23:16:47.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.944+0000 7f6292800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f628c06ff60 con 0x7f628c0c4d20 2024-02-21T23:16:47.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.945+0000 7f6290a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f628c06bad0 0x7f628c15d900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:51686/0 (socket says 172.21.15.28:51686) 2024-02-21T23:16:47.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.945+0000 7f6290a00640 1 -- 172.21.15.28:0/2671083619 learned_addr learned my addr 172.21.15.28:0/2671083619 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:16:47.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.945+0000 7f6291e00640 1 --2- 172.21.15.28:0/2671083619 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f628c0c4d20 0x7f628c0a9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:47.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.945+0000 7f628be00640 1 -- 172.21.15.28:0/2671083619 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3028614120 0 0) 0x7f628c06b6d0 con 0x7f628c06bad0 2024-02-21T23:16:47.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.945+0000 7f628be00640 1 -- 172.21.15.28:0/2671083619 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6274003520 con 0x7f628c06bad0 2024-02-21T23:16:47.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.946+0000 7f628be00640 1 -- 172.21.15.28:0/2671083619 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2466978440 0 0) 0x7f6274003520 con 0x7f628c06bad0 2024-02-21T23:16:47.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.946+0000 7f628be00640 1 -- 172.21.15.28:0/2671083619 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f628c0c4d20 msgr2=0x7f628c0a9eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:16:47.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.946+0000 7f628be00640 1 --2- 172.21.15.28:0/2671083619 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f628c0c4d20 0x7f628c0a9eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:47.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.946+0000 7f628be00640 1 -- 172.21.15.28:0/2671083619 >> v1:172.21.15.81:6789/0 conn(0x7f628c06f130 legacy=0x7f628c158580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:47.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.946+0000 7f628be00640 1 -- 172.21.15.28:0/2671083619 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f628c070240 con 0x7f628c06bad0 2024-02-21T23:16:47.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.946+0000 7f628be00640 1 -- 172.21.15.28:0/2671083619 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f6278001670 con 0x7f628c06bad0 2024-02-21T23:16:47.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.946+0000 7f628be00640 1 -- 172.21.15.28:0/2671083619 >> v1:172.21.15.28:6789/0 conn(0x7f628c06bad0 legacy=0x7f628c15d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:47.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.946+0000 7f628be00640 1 --2- 172.21.15.28:0/2671083619 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f628c0c4d20 0x7f62740039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:47.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.946+0000 7f628be00640 1 --2- 172.21.15.28:0/2671083619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6274003fb0 0x7f6274004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:47.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.946+0000 7f628be00640 1 --2- 172.21.15.28:0/2671083619 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6274004d40 0x7f62740051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:47.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.946+0000 7f628be00640 1 -- 172.21.15.28:0/2671083619 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f628c06ff60 con 0x7f6274003fb0 2024-02-21T23:16:47.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.946+0000 7f628be00640 1 -- 172.21.15.28:0/2671083619 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f627400bbb0 con 0x7f6274004d40 2024-02-21T23:16:47.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.946+0000 7f628be00640 1 -- 172.21.15.28:0/2671083619 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f627400bdb0 con 0x7f628c0c4d20 2024-02-21T23:16:47.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.946+0000 7f6290a00640 1 --2- 172.21.15.28:0/2671083619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6274003fb0 0x7f6274004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:47.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.947+0000 7f6291400640 1 --2- 172.21.15.28:0/2671083619 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f628c0c4d20 0x7f62740039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:47.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.947+0000 7f6291e00640 1 --2- 172.21.15.28:0/2671083619 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6274004d40 0x7f62740051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:47.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.947+0000 7f6290a00640 1 -- 172.21.15.28:0/2671083619 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f628c0c4d20 msgr2=0x7f62740039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:16:47.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.947+0000 7f6290a00640 1 --2- 172.21.15.28:0/2671083619 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f628c0c4d20 0x7f62740039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:47.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.947+0000 7f6290a00640 1 -- 172.21.15.28:0/2671083619 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6274004d40 msgr2=0x7f62740051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:16:47.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.947+0000 7f6290a00640 1 --2- 172.21.15.28:0/2671083619 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6274004d40 0x7f62740051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:47.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.947+0000 7f6290a00640 1 -- 172.21.15.28:0/2671083619 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6274003520 con 0x7f6274003fb0 2024-02-21T23:16:47.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.949+0000 7f6290a00640 1 --2- 172.21.15.28:0/2671083619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6274003fb0 0x7f6274004800 secure :-1 s=READY pgs=2657 cs=0 l=1 rev1=1 crypto rx=0x7f627800c0e0 tx=0x7f6274005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0665fba80977141 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:47.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.949+0000 7f628be00640 1 -- 172.21.15.28:0/2671083619 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6278007c40 con 0x7f6274003fb0 2024-02-21T23:16:47.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.949+0000 7f628be00640 1 -- 172.21.15.28:0/2671083619 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6278030070 con 0x7f6274003fb0 2024-02-21T23:16:47.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.949+0000 7f628be00640 1 -- 172.21.15.28:0/2671083619 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6278005d30 con 0x7f6274003fb0 2024-02-21T23:16:47.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.950+0000 7f6292800640 1 -- 172.21.15.28:0/2671083619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6274003fb0 msgr2=0x7f6274004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:47.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.950+0000 7f6292800640 1 --2- 172.21.15.28:0/2671083619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6274003fb0 0x7f6274004800 secure :-1 s=READY pgs=2657 cs=0 l=1 rev1=1 crypto rx=0x7f627800c0e0 tx=0x7f6274005830 comp rx=0 tx=0).stop 2024-02-21T23:16:47.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.950+0000 7f6291e00640 1 -- 172.21.15.28:0/2671083619 reap_dead start 2024-02-21T23:16:47.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.950+0000 7f6292800640 1 -- 172.21.15.28:0/2671083619 shutdown_connections 2024-02-21T23:16:47.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.950+0000 7f6292800640 1 -- 172.21.15.28:0/2671083619 >> 172.21.15.28:0/2671083619 conn(0x7f628c067050 msgr2=0x7f628c0a84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:16:47.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.950+0000 7f6292800640 1 -- 172.21.15.28:0/2671083619 shutdown_connections 2024-02-21T23:16:47.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.950+0000 7f6292800640 1 -- 172.21.15.28:0/2671083619 wait complete. 2024-02-21T23:16:47.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.952+0000 7f6292800640 1 Processor -- start 2024-02-21T23:16:47.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.952+0000 7f6292800640 1 -- start start 2024-02-21T23:16:47.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.953+0000 7f6292800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6274003fb0 0x7f628c1540f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:47.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.953+0000 7f6292800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f628c1eeb20 0x7f628c151020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:47.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.953+0000 7f6292800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f628c1dfb00 0x7f628c1dff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:47.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.953+0000 7f6292800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f628c0a9990 con 0x7f628c1eeb20 2024-02-21T23:16:47.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.953+0000 7f6292800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f628c0a96b0 con 0x7f628c1dfb00 2024-02-21T23:16:47.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.953+0000 7f6292800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f628c158c90 con 0x7f6274003fb0 2024-02-21T23:16:47.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.953+0000 7f6290a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f628c1eeb20 0x7f628c151020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:47.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.953+0000 7f6290a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f628c1eeb20 0x7f628c151020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60478/0 (socket says 172.21.15.28:60478) 2024-02-21T23:16:47.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.953+0000 7f6290a00640 1 -- 172.21.15.28:0/1288534430 learned_addr learned my addr 172.21.15.28:0/1288534430 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:16:47.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.953+0000 7f6291400640 1 --2- 172.21.15.28:0/1288534430 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6274003fb0 0x7f628c1540f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:47.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.953+0000 7f6291e00640 1 --2- 172.21.15.28:0/1288534430 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f628c1dfb00 0x7f628c1dff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:47.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.954+0000 7f6290a00640 1 -- 172.21.15.28:0/1288534430 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6274003fb0 msgr2=0x7f628c1540f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:47.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.954+0000 7f6290a00640 1 --2- 172.21.15.28:0/1288534430 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6274003fb0 0x7f628c1540f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:47.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.954+0000 7f6290a00640 1 -- 172.21.15.28:0/1288534430 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f628c1dfb00 msgr2=0x7f628c1dff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:47.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.954+0000 7f6290a00640 1 --2- 172.21.15.28:0/1288534430 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f628c1dfb00 0x7f628c1dff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:47.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.954+0000 7f6290a00640 1 -- 172.21.15.28:0/1288534430 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f628c1532d0 con 0x7f628c1eeb20 2024-02-21T23:16:47.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.954+0000 7f6290a00640 1 --2- 172.21.15.28:0/1288534430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f628c1eeb20 0x7f628c151020 secure :-1 s=READY pgs=2658 cs=0 l=1 rev1=1 crypto rx=0x7f627802c480 tx=0x7f628c0c1a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:47.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.954+0000 7f6289600640 1 -- 172.21.15.28:0/1288534430 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6278045070 con 0x7f628c1eeb20 2024-02-21T23:16:47.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.954+0000 7f6289600640 1 -- 172.21.15.28:0/1288534430 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6278004bf0 con 0x7f628c1eeb20 2024-02-21T23:16:47.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.954+0000 7f6292800640 1 -- 172.21.15.28:0/1288534430 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f628c1ee040 con 0x7f628c1eeb20 2024-02-21T23:16:47.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.954+0000 7f6289600640 1 -- 172.21.15.28:0/1288534430 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6278001ba0 con 0x7f628c1eeb20 2024-02-21T23:16:47.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.955+0000 7f6292800640 1 -- 172.21.15.28:0/1288534430 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f628c154630 con 0x7f628c1eeb20 2024-02-21T23:16:47.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.956+0000 7f6292800640 1 -- 172.21.15.28:0/1288534430 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f628c152c20 con 0x7f628c1eeb20 2024-02-21T23:16:47.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.957+0000 7f6289600640 1 -- 172.21.15.28:0/1288534430 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6278007070 con 0x7f628c1eeb20 2024-02-21T23:16:47.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.958+0000 7f6289600640 1 --2- 172.21.15.28:0/1288534430 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f626c06b500 0x7f626c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:47.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.958+0000 7f6289600640 1 -- 172.21.15.28:0/1288534430 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f62780ab570 con 0x7f628c1eeb20 2024-02-21T23:16:47.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.958+0000 7f6291400640 1 --2- 172.21.15.28:0/1288534430 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f626c06b500 0x7f626c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:47.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.960+0000 7f6291400640 1 --2- 172.21.15.28:0/1288534430 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f626c06b500 0x7f626c06d9c0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f6280000c00 tx=0x7f6280009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:47.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:47.962+0000 7f6289600640 1 -- 172.21.15.28:0/1288534430 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f628c152c20 con 0x7f628c1eeb20 2024-02-21T23:16:48.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:48.199+0000 7f6292800640 1 -- 172.21.15.28:0/1288534430 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f628c04f290 con 0x7f628c1eeb20 2024-02-21T23:16:48.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:48.200+0000 7f6289600640 1 -- 172.21.15.28:0/1288534430 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7f627807a1d0 con 0x7f628c1eeb20 2024-02-21T23:16:48.187 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:16:48.187 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:16:48.187 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:16:48.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:48.202+0000 7f6292800640 1 -- 172.21.15.28:0/1288534430 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f626c06b500 msgr2=0x7f626c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:48.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:48.202+0000 7f6292800640 1 --2- 172.21.15.28:0/1288534430 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f626c06b500 0x7f626c06d9c0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f6280000c00 tx=0x7f6280009040 comp rx=0 tx=0).stop 2024-02-21T23:16:48.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:48.202+0000 7f6292800640 1 -- 172.21.15.28:0/1288534430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f628c1eeb20 msgr2=0x7f628c151020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:48.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:48.202+0000 7f6292800640 1 --2- 172.21.15.28:0/1288534430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f628c1eeb20 0x7f628c151020 secure :-1 s=READY pgs=2658 cs=0 l=1 rev1=1 crypto rx=0x7f627802c480 tx=0x7f628c0c1a50 comp rx=0 tx=0).stop 2024-02-21T23:16:48.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:48.203+0000 7f6292800640 1 -- 172.21.15.28:0/1288534430 shutdown_connections 2024-02-21T23:16:48.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:48.203+0000 7f6292800640 1 --2- 172.21.15.28:0/1288534430 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f626c06b500 0x7f626c06d9c0 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:48.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:48.203+0000 7f6292800640 1 --2- 172.21.15.28:0/1288534430 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f628c1dfb00 0x7f628c1dff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:48.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:48.203+0000 7f6292800640 1 --2- 172.21.15.28:0/1288534430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f628c1eeb20 0x7f628c151020 unknown :-1 s=CLOSED pgs=2658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:48.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:48.203+0000 7f6292800640 1 --2- 172.21.15.28:0/1288534430 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f6274003fb0 0x7f628c1540f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:48.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:48.203+0000 7f6292800640 1 -- 172.21.15.28:0/1288534430 >> 172.21.15.28:0/1288534430 conn(0x7f628c067050 msgr2=0x7f628c0c3a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:16:48.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:48.203+0000 7f6292800640 1 -- 172.21.15.28:0/1288534430 shutdown_connections 2024-02-21T23:16:48.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:48.203+0000 7f6292800640 1 -- 172.21.15.28:0/1288534430 wait complete. 2024-02-21T23:16:48.611 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:16:48.611 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:16:49.612 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:16:49.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.850+0000 7f8135400640 1 Processor -- start 2024-02-21T23:16:49.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.850+0000 7f8135400640 1 -- start start 2024-02-21T23:16:49.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.851+0000 7f8135400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f81300aff90 0x7f81300c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:49.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.851+0000 7f8135400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f81300c5720 0x7f813015f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:49.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.852+0000 7f8135400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81300c74c0 con 0x7f813006dbf0 2024-02-21T23:16:49.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.852+0000 7f8135400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81300c7270 con 0x7f81300aff90 2024-02-21T23:16:49.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.852+0000 7f8135400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8130158630 con 0x7f81300c5720 2024-02-21T23:16:49.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.852+0000 7f812f400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f813006dbf0 0x7f8130158ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:51696/0 (socket says 172.21.15.28:51696) 2024-02-21T23:16:49.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.852+0000 7f812f400640 1 -- 172.21.15.28:0/1777985604 learned_addr learned my addr 172.21.15.28:0/1777985604 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:16:49.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.852+0000 7f8134a00640 1 --2- 172.21.15.28:0/1777985604 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f81300c5720 0x7f813015f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:49.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.852+0000 7f812fe00640 1 --2- 172.21.15.28:0/1777985604 >> v2:172.21.15.81:3300/0 conn(0x7f81300aff90 0x7f81300c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:49.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.852+0000 7f812ea00640 1 -- 172.21.15.28:0/1777985604 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1892360116 0 0) 0x7f81300c74c0 con 0x7f813006dbf0 2024-02-21T23:16:49.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.853+0000 7f812ea00640 1 -- 172.21.15.28:0/1777985604 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8114003520 con 0x7f813006dbf0 2024-02-21T23:16:49.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.853+0000 7f812ea00640 1 -- 172.21.15.28:0/1777985604 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1107156736 0 0) 0x7f8114003520 con 0x7f813006dbf0 2024-02-21T23:16:49.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.853+0000 7f812ea00640 1 -- 172.21.15.28:0/1777985604 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f81300c5720 msgr2=0x7f813015f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:16:49.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.853+0000 7f812ea00640 1 --2- 172.21.15.28:0/1777985604 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f81300c5720 0x7f813015f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:49.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.853+0000 7f812ea00640 1 -- 172.21.15.28:0/1777985604 >> v2:172.21.15.81:3300/0 conn(0x7f81300aff90 msgr2=0x7f81300c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:16:49.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.853+0000 7f812ea00640 1 --2- 172.21.15.28:0/1777985604 >> v2:172.21.15.81:3300/0 conn(0x7f81300aff90 0x7f81300c51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:49.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.853+0000 7f812ea00640 1 -- 172.21.15.28:0/1777985604 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81300c77a0 con 0x7f813006dbf0 2024-02-21T23:16:49.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.853+0000 7f812ea00640 1 -- 172.21.15.28:0/1777985604 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f811c001670 con 0x7f813006dbf0 2024-02-21T23:16:49.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.853+0000 7f812ea00640 1 -- 172.21.15.28:0/1777985604 >> v1:172.21.15.28:6789/0 conn(0x7f813006dbf0 legacy=0x7f8130158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:49.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.853+0000 7f812ea00640 1 --2- 172.21.15.28:0/1777985604 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f81300c5720 0x7f81140039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:49.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.853+0000 7f812ea00640 1 --2- 172.21.15.28:0/1777985604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8114003f80 0x7f8114004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:49.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.853+0000 7f812ea00640 1 --2- 172.21.15.28:0/1777985604 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8114004d40 0x7f81140051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:49.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.853+0000 7f812ea00640 1 -- 172.21.15.28:0/1777985604 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81300c7270 con 0x7f8114003f80 2024-02-21T23:16:49.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.853+0000 7f812ea00640 1 -- 172.21.15.28:0/1777985604 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8130158630 con 0x7f8114004d40 2024-02-21T23:16:49.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.853+0000 7f812ea00640 1 -- 172.21.15.28:0/1777985604 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f811400b9c0 con 0x7f81300c5720 2024-02-21T23:16:49.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.853+0000 7f812f400640 1 --2- 172.21.15.28:0/1777985604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8114003f80 0x7f8114004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:49.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.854+0000 7f812fe00640 1 --2- 172.21.15.28:0/1777985604 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f81300c5720 0x7f81140039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:49.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.854+0000 7f8134a00640 1 --2- 172.21.15.28:0/1777985604 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8114004d40 0x7f81140051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:49.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.854+0000 7f812f400640 1 -- 172.21.15.28:0/1777985604 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f81300c5720 msgr2=0x7f81140039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:16:49.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.854+0000 7f812f400640 1 --2- 172.21.15.28:0/1777985604 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f81300c5720 0x7f81140039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:49.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.854+0000 7f812f400640 1 -- 172.21.15.28:0/1777985604 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8114004d40 msgr2=0x7f81140051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:16:49.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.854+0000 7f812f400640 1 --2- 172.21.15.28:0/1777985604 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8114004d40 0x7f81140051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:49.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.854+0000 7f812f400640 1 -- 172.21.15.28:0/1777985604 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8114003520 con 0x7f8114003f80 2024-02-21T23:16:49.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.854+0000 7f8134a00640 1 --2- 172.21.15.28:0/1777985604 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8114004d40 0x7f81140051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:16:49.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.855+0000 7f812f400640 1 --2- 172.21.15.28:0/1777985604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8114003f80 0x7f8114004800 secure :-1 s=READY pgs=2660 cs=0 l=1 rev1=1 crypto rx=0x7f811c00c190 tx=0x7f8114005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a934ffd2083d2375 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:49.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.855+0000 7f812ea00640 1 -- 172.21.15.28:0/1777985604 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f811c007c80 con 0x7f8114003f80 2024-02-21T23:16:49.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.855+0000 7f812ea00640 1 -- 172.21.15.28:0/1777985604 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f811c030070 con 0x7f8114003f80 2024-02-21T23:16:49.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.855+0000 7f812ea00640 1 -- 172.21.15.28:0/1777985604 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f811c005d70 con 0x7f8114003f80 2024-02-21T23:16:49.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.856+0000 7f8135400640 1 -- 172.21.15.28:0/1777985604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8114003f80 msgr2=0x7f8114004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:49.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.856+0000 7f8135400640 1 --2- 172.21.15.28:0/1777985604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8114003f80 0x7f8114004800 secure :-1 s=READY pgs=2660 cs=0 l=1 rev1=1 crypto rx=0x7f811c00c190 tx=0x7f8114005830 comp rx=0 tx=0).stop 2024-02-21T23:16:49.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.856+0000 7f8134a00640 1 -- 172.21.15.28:0/1777985604 reap_dead start 2024-02-21T23:16:49.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.856+0000 7f8135400640 1 -- 172.21.15.28:0/1777985604 shutdown_connections 2024-02-21T23:16:49.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.856+0000 7f8135400640 1 -- 172.21.15.28:0/1777985604 >> 172.21.15.28:0/1777985604 conn(0x7f81300ac370 msgr2=0x7f81300ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:16:49.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.856+0000 7f8135400640 1 -- 172.21.15.28:0/1777985604 shutdown_connections 2024-02-21T23:16:49.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.856+0000 7f8135400640 1 -- 172.21.15.28:0/1777985604 wait complete. 2024-02-21T23:16:49.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.858+0000 7f8135400640 1 Processor -- start 2024-02-21T23:16:49.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.858+0000 7f8135400640 1 -- start start 2024-02-21T23:16:49.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.859+0000 7f8135400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8114003f80 0x7f8130157100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:49.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.859+0000 7f8135400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81301ede20 0x7f81301eae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:49.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.859+0000 7f8135400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81300c4750 0x7f81301eb3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:49.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.859+0000 7f8135400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81300c8760 con 0x7f81300c4750 2024-02-21T23:16:49.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.859+0000 7f8135400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81300be0c0 con 0x7f81301ede20 2024-02-21T23:16:49.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.859+0000 7f8135400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81300c88e0 con 0x7f8114003f80 2024-02-21T23:16:49.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.859+0000 7f8134a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81300c4750 0x7f81301eb3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:49.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.859+0000 7f8134a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81300c4750 0x7f81301eb3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60494/0 (socket says 172.21.15.28:60494) 2024-02-21T23:16:49.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.859+0000 7f8134a00640 1 -- 172.21.15.28:0/1267179098 learned_addr learned my addr 172.21.15.28:0/1267179098 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:16:49.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.859+0000 7f812fe00640 1 --2- 172.21.15.28:0/1267179098 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8114003f80 0x7f8130157100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:49.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.860+0000 7f812f400640 1 --2- 172.21.15.28:0/1267179098 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81301ede20 0x7f81301eae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:49.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.860+0000 7f8134a00640 1 -- 172.21.15.28:0/1267179098 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8114003f80 msgr2=0x7f8130157100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:49.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.860+0000 7f8134a00640 1 --2- 172.21.15.28:0/1267179098 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8114003f80 0x7f8130157100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:49.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.860+0000 7f8134a00640 1 -- 172.21.15.28:0/1267179098 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81301ede20 msgr2=0x7f81301eae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:49.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.860+0000 7f8134a00640 1 --2- 172.21.15.28:0/1267179098 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81301ede20 0x7f81301eae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:49.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.860+0000 7f8134a00640 1 -- 172.21.15.28:0/1267179098 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81301ee3c0 con 0x7f81300c4750 2024-02-21T23:16:49.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.860+0000 7f8134a00640 1 --2- 172.21.15.28:0/1267179098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81300c4750 0x7f81301eb3b0 secure :-1 s=READY pgs=2661 cs=0 l=1 rev1=1 crypto rx=0x7f81300c73f0 tx=0x7f812800f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:49.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.860+0000 7f8127e00640 1 -- 172.21.15.28:0/1267179098 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f812801c070 con 0x7f81300c4750 2024-02-21T23:16:49.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.860+0000 7f8127e00640 1 -- 172.21.15.28:0/1267179098 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8128017400 con 0x7f81300c4750 2024-02-21T23:16:49.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.860+0000 7f8135400640 1 -- 172.21.15.28:0/1267179098 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81301ee6a0 con 0x7f81300c4750 2024-02-21T23:16:49.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.860+0000 7f8127e00640 1 -- 172.21.15.28:0/1267179098 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81280177d0 con 0x7f81300c4750 2024-02-21T23:16:49.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.861+0000 7f8135400640 1 -- 172.21.15.28:0/1267179098 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8130157640 con 0x7f81300c4750 2024-02-21T23:16:49.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.862+0000 7f8135400640 1 -- 172.21.15.28:0/1267179098 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f813004f290 con 0x7f81300c4750 2024-02-21T23:16:49.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.864+0000 7f8127e00640 1 -- 172.21.15.28:0/1267179098 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f812801f070 con 0x7f81300c4750 2024-02-21T23:16:49.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.865+0000 7f8127e00640 1 --2- 172.21.15.28:0/1267179098 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f811006b4d0 0x7f811006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:49.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.865+0000 7f8127e00640 1 -- 172.21.15.28:0/1267179098 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f812808a540 con 0x7f81300c4750 2024-02-21T23:16:49.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.865+0000 7f812fe00640 1 --2- 172.21.15.28:0/1267179098 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f811006b4d0 0x7f811006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:49.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.867+0000 7f812fe00640 1 --2- 172.21.15.28:0/1267179098 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f811006b4d0 0x7f811006d990 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f8120009a10 tx=0x7f8120008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:49.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:49.870+0000 7f8127e00640 1 -- 172.21.15.28:0/1267179098 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f8128012070 con 0x7f81300c4750 2024-02-21T23:16:50.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:50.107+0000 7f8135400640 1 -- 172.21.15.28:0/1267179098 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f813006d9d0 con 0x7f81300c4750 2024-02-21T23:16:50.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:50.108+0000 7f8127e00640 1 -- 172.21.15.28:0/1267179098 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7f8128059120 con 0x7f81300c4750 2024-02-21T23:16:50.095 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:16:50.095 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:16:50.095 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:16:50.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:50.110+0000 7f8135400640 1 -- 172.21.15.28:0/1267179098 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f811006b4d0 msgr2=0x7f811006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:50.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:50.110+0000 7f8135400640 1 --2- 172.21.15.28:0/1267179098 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f811006b4d0 0x7f811006d990 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f8120009a10 tx=0x7f8120008040 comp rx=0 tx=0).stop 2024-02-21T23:16:50.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:50.110+0000 7f8135400640 1 -- 172.21.15.28:0/1267179098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81300c4750 msgr2=0x7f81301eb3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:50.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:50.110+0000 7f8135400640 1 --2- 172.21.15.28:0/1267179098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81300c4750 0x7f81301eb3b0 secure :-1 s=READY pgs=2661 cs=0 l=1 rev1=1 crypto rx=0x7f81300c73f0 tx=0x7f812800f4a0 comp rx=0 tx=0).stop 2024-02-21T23:16:50.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:50.111+0000 7f8135400640 1 -- 172.21.15.28:0/1267179098 shutdown_connections 2024-02-21T23:16:50.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:50.111+0000 7f8135400640 1 --2- 172.21.15.28:0/1267179098 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f811006b4d0 0x7f811006d990 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:50.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:50.111+0000 7f8135400640 1 --2- 172.21.15.28:0/1267179098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81300c4750 0x7f81301eb3b0 unknown :-1 s=CLOSED pgs=2661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:50.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:50.111+0000 7f8135400640 1 --2- 172.21.15.28:0/1267179098 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f81301ede20 0x7f81301eae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:50.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:50.111+0000 7f8135400640 1 --2- 172.21.15.28:0/1267179098 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8114003f80 0x7f8130157100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:50.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:50.111+0000 7f8135400640 1 -- 172.21.15.28:0/1267179098 >> 172.21.15.28:0/1267179098 conn(0x7f81300ac370 msgr2=0x7f81300ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:16:50.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:50.111+0000 7f8135400640 1 -- 172.21.15.28:0/1267179098 shutdown_connections 2024-02-21T23:16:50.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:50.111+0000 7f8135400640 1 -- 172.21.15.28:0/1267179098 wait complete. 2024-02-21T23:16:50.521 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:16:50.521 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:16:51.522 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:16:51.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:51.754+0000 7f7e39e00640 1 Processor -- start 2024-02-21T23:16:51.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:51.754+0000 7f7e39e00640 1 -- start start 2024-02-21T23:16:51.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:51.755+0000 7f7e39e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7e3406ee40 0x7f7e340a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:51.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:51.755+0000 7f7e39e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7e34156af0 0x7f7e3406dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:51.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:51.756+0000 7f7e39e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e340c2760 con 0x7f7e34150450 2024-02-21T23:16:51.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:51.756+0000 7f7e39e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e340c22c0 con 0x7f7e3406ee40 2024-02-21T23:16:51.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:51.756+0000 7f7e39e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e34156660 con 0x7f7e34156af0 2024-02-21T23:16:51.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:51.756+0000 7f7e33e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7e3406ee40 0x7f7e340a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:51.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:51.756+0000 7f7e33e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7e3406ee40 0x7f7e340a5fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60496/0 (socket says 172.21.15.28:60496) 2024-02-21T23:16:51.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:51.756+0000 7f7e33e00640 1 -- 172.21.15.28:0/1439338445 learned_addr learned my addr 172.21.15.28:0/1439338445 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:16:51.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:51.756+0000 7f7e38a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f7e34150450 0x7f7e341536f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:51698/0 (socket says 172.21.15.28:51698) 2024-02-21T23:16:51.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:51.756+0000 7f7e33e00640 1 -- 172.21.15.28:0/1439338445 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7e34156af0 msgr2=0x7f7e3406dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:16:51.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:51.756+0000 7f7e39400640 1 --2- 172.21.15.28:0/1439338445 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7e34156af0 0x7f7e3406dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:51.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:51.756+0000 7f7e33e00640 1 --2- 172.21.15.28:0/1439338445 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7e34156af0 0x7f7e3406dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:51.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:51.756+0000 7f7e33e00640 1 -- 172.21.15.28:0/1439338445 >> v1:172.21.15.28:6789/0 conn(0x7f7e34150450 legacy=0x7f7e341536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:51.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:51.756+0000 7f7e33e00640 1 -- 172.21.15.28:0/1439338445 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e34156310 con 0x7f7e3406ee40 2024-02-21T23:16:51.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:51.756+0000 7f7e33400640 1 -- 172.21.15.28:0/1439338445 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1180955539 0 0) 0x7f7e340c2760 con 0x7f7e34150450 2024-02-21T23:16:51.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:51.756+0000 7f7e39400640 1 --2- 172.21.15.28:0/1439338445 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7e34156af0 0x7f7e3406dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:16:51.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:51.758+0000 7f7e33e00640 1 --2- 172.21.15.28:0/1439338445 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e3406ee40 0x7f7e340a5fd0 secure :-1 s=READY pgs=2663 cs=0 l=1 rev1=1 crypto rx=0x7f7e20009870 tx=0x7f7e2002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c3ac5413beff53b server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:51.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:51.758+0000 7f7e33400640 1 -- 172.21.15.28:0/1439338445 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e20034070 con 0x7f7e3406ee40 2024-02-21T23:16:51.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:51.758+0000 7f7e33400640 1 -- 172.21.15.28:0/1439338445 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e20030840 con 0x7f7e3406ee40 2024-02-21T23:16:51.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:51.758+0000 7f7e33400640 1 -- 172.21.15.28:0/1439338445 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e20030c10 con 0x7f7e3406ee40 2024-02-21T23:16:51.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:51.758+0000 7f7e39e00640 1 -- 172.21.15.28:0/1439338445 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e3406ee40 msgr2=0x7f7e340a5fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:51.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:51.758+0000 7f7e39e00640 1 --2- 172.21.15.28:0/1439338445 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e3406ee40 0x7f7e340a5fd0 secure :-1 s=READY pgs=2663 cs=0 l=1 rev1=1 crypto rx=0x7f7e20009870 tx=0x7f7e2002b980 comp rx=0 tx=0).stop 2024-02-21T23:16:51.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:51.759+0000 7f7e39e00640 1 -- 172.21.15.28:0/1439338445 shutdown_connections 2024-02-21T23:16:51.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:51.759+0000 7f7e39e00640 1 --2- 172.21.15.28:0/1439338445 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7e34156af0 0x7f7e3406dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:51.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:51.759+0000 7f7e39e00640 1 --2- 172.21.15.28:0/1439338445 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e3406ee40 0x7f7e340a5fd0 unknown :-1 s=CLOSED pgs=2663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:51.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:51.759+0000 7f7e39e00640 1 -- 172.21.15.28:0/1439338445 >> 172.21.15.28:0/1439338445 conn(0x7f7e3406b630 msgr2=0x7f7e340c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:16:51.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:51.759+0000 7f7e39e00640 1 -- 172.21.15.28:0/1439338445 shutdown_connections 2024-02-21T23:16:51.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:51.759+0000 7f7e39e00640 1 -- 172.21.15.28:0/1439338445 wait complete. 2024-02-21T23:16:51.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:51.761+0000 7f7e39e00640 1 Processor -- start 2024-02-21T23:16:51.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:51.761+0000 7f7e39e00640 1 -- start start 2024-02-21T23:16:51.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:51.762+0000 7f7e39e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7e3406ee40 0x7f7e340c53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:51.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:51.762+0000 7f7e39e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e34150450 0x7f7e3414a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:51.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:51.762+0000 7f7e39e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e34156af0 0x7f7e3414ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:51.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:51.762+0000 7f7e39e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e34156f80 con 0x7f7e34156af0 2024-02-21T23:16:51.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:51.762+0000 7f7e39e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e340c2cb0 con 0x7f7e34150450 2024-02-21T23:16:51.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:51.762+0000 7f7e39e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e34157100 con 0x7f7e3406ee40 2024-02-21T23:16:51.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:51.762+0000 7f7e39400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e34156af0 0x7f7e3414ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:51.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:51.762+0000 7f7e39400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e34156af0 0x7f7e3414ac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60510/0 (socket says 172.21.15.28:60510) 2024-02-21T23:16:51.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:51.762+0000 7f7e39400640 1 -- 172.21.15.28:0/4183923462 learned_addr learned my addr 172.21.15.28:0/4183923462 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:16:51.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:51.762+0000 7f7e38a00640 1 --2- 172.21.15.28:0/4183923462 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7e3406ee40 0x7f7e340c53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:51.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:51.762+0000 7f7e33e00640 1 --2- 172.21.15.28:0/4183923462 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e34150450 0x7f7e3414a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:51.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:51.763+0000 7f7e39400640 1 -- 172.21.15.28:0/4183923462 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7e3406ee40 msgr2=0x7f7e340c53b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:51.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:51.763+0000 7f7e39400640 1 --2- 172.21.15.28:0/4183923462 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7e3406ee40 0x7f7e340c53b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:51.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:51.763+0000 7f7e39400640 1 -- 172.21.15.28:0/4183923462 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e34150450 msgr2=0x7f7e3414a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:51.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:51.763+0000 7f7e39400640 1 --2- 172.21.15.28:0/4183923462 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e34150450 0x7f7e3414a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:51.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:51.763+0000 7f7e39400640 1 -- 172.21.15.28:0/4183923462 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e341eb5d0 con 0x7f7e34156af0 2024-02-21T23:16:51.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:51.763+0000 7f7e39400640 1 --2- 172.21.15.28:0/4183923462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e34156af0 0x7f7e3414ac30 secure :-1 s=READY pgs=2664 cs=0 l=1 rev1=1 crypto rx=0x7f7e2c00cd80 tx=0x7f7e2c00a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:51.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:51.763+0000 7f7e30c00640 1 -- 172.21.15.28:0/4183923462 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e2c00f680 con 0x7f7e34156af0 2024-02-21T23:16:51.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:51.763+0000 7f7e30c00640 1 -- 172.21.15.28:0/4183923462 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e2c00f840 con 0x7f7e34156af0 2024-02-21T23:16:51.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:51.763+0000 7f7e30c00640 1 -- 172.21.15.28:0/4183923462 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e2c00fc30 con 0x7f7e34156af0 2024-02-21T23:16:51.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:51.763+0000 7f7e39e00640 1 -- 172.21.15.28:0/4183923462 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e340abd40 con 0x7f7e34156af0 2024-02-21T23:16:51.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:51.764+0000 7f7e39e00640 1 -- 172.21.15.28:0/4183923462 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e340ac300 con 0x7f7e34156af0 2024-02-21T23:16:51.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:51.765+0000 7f7e39e00640 1 -- 172.21.15.28:0/4183923462 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e3404f290 con 0x7f7e34156af0 2024-02-21T23:16:51.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:51.766+0000 7f7e30c00640 1 -- 172.21.15.28:0/4183923462 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7e2c00e3e0 con 0x7f7e34156af0 2024-02-21T23:16:51.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:51.767+0000 7f7e30c00640 1 --2- 172.21.15.28:0/4183923462 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7e1406b4d0 0x7f7e1406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:51.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:51.767+0000 7f7e30c00640 1 -- 172.21.15.28:0/4183923462 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f7e2c08b270 con 0x7f7e34156af0 2024-02-21T23:16:51.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:51.767+0000 7f7e38a00640 1 --2- 172.21.15.28:0/4183923462 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7e1406b4d0 0x7f7e1406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:51.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:51.769+0000 7f7e38a00640 1 --2- 172.21.15.28:0/4183923462 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7e1406b4d0 0x7f7e1406d990 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f7e28000f60 tx=0x7f7e28007b50 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:51.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:51.772+0000 7f7e30c00640 1 -- 172.21.15.28:0/4183923462 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7e2c014030 con 0x7f7e34156af0 2024-02-21T23:16:51.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:52.011+0000 7f7e39e00640 1 -- 172.21.15.28:0/4183923462 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e340c3b40 con 0x7f7e34156af0 2024-02-21T23:16:51.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:52.012+0000 7f7e30c00640 1 -- 172.21.15.28:0/4183923462 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7f7e2c059e50 con 0x7f7e34156af0 2024-02-21T23:16:51.999 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:16:51.999 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:16:52.000 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:16:52.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:52.015+0000 7f7e39e00640 1 -- 172.21.15.28:0/4183923462 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7e1406b4d0 msgr2=0x7f7e1406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:52.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:52.015+0000 7f7e39e00640 1 --2- 172.21.15.28:0/4183923462 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7e1406b4d0 0x7f7e1406d990 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f7e28000f60 tx=0x7f7e28007b50 comp rx=0 tx=0).stop 2024-02-21T23:16:52.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:52.015+0000 7f7e39e00640 1 -- 172.21.15.28:0/4183923462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e34156af0 msgr2=0x7f7e3414ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:52.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:52.015+0000 7f7e39e00640 1 --2- 172.21.15.28:0/4183923462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e34156af0 0x7f7e3414ac30 secure :-1 s=READY pgs=2664 cs=0 l=1 rev1=1 crypto rx=0x7f7e2c00cd80 tx=0x7f7e2c00a650 comp rx=0 tx=0).stop 2024-02-21T23:16:52.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:52.015+0000 7f7e39e00640 1 -- 172.21.15.28:0/4183923462 shutdown_connections 2024-02-21T23:16:52.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:52.015+0000 7f7e39e00640 1 --2- 172.21.15.28:0/4183923462 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7e1406b4d0 0x7f7e1406d990 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:52.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:52.015+0000 7f7e39e00640 1 --2- 172.21.15.28:0/4183923462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e34156af0 0x7f7e3414ac30 unknown :-1 s=CLOSED pgs=2664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:52.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:52.015+0000 7f7e39e00640 1 --2- 172.21.15.28:0/4183923462 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e34150450 0x7f7e3414a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:52.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:52.015+0000 7f7e39e00640 1 --2- 172.21.15.28:0/4183923462 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7e3406ee40 0x7f7e340c53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:52.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:52.015+0000 7f7e39e00640 1 -- 172.21.15.28:0/4183923462 >> 172.21.15.28:0/4183923462 conn(0x7f7e3406b630 msgr2=0x7f7e340c64e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:16:52.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:52.015+0000 7f7e39e00640 1 -- 172.21.15.28:0/4183923462 shutdown_connections 2024-02-21T23:16:52.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:52.015+0000 7f7e39e00640 1 -- 172.21.15.28:0/4183923462 wait complete. 2024-02-21T23:16:52.428 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:16:52.428 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:16:53.430 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:16:53.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.663+0000 7fdbb6800640 1 Processor -- start 2024-02-21T23:16:53.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.663+0000 7fdbb6800640 1 -- start start 2024-02-21T23:16:53.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.664+0000 7fdbb6800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fdbb0150450 0x7fdbb00a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:53.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.664+0000 7fdbb6800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdbb0156af0 0x7fdbb006dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:53.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.665+0000 7fdbb6800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbb00c2760 con 0x7fdbb006ee40 2024-02-21T23:16:53.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.665+0000 7fdbb6800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbb00c22c0 con 0x7fdbb0150450 2024-02-21T23:16:53.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.665+0000 7fdbb6800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbb0156660 con 0x7fdbb0156af0 2024-02-21T23:16:53.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.665+0000 7fdbb5400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fdbb0150450 0x7fdbb00a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:53.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.665+0000 7fdbb5e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdbb0156af0 0x7fdbb006dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:53.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.665+0000 7fdbb5400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fdbb0150450 0x7fdbb00a5fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.28:52012/0 (socket says 172.21.15.28:52012) 2024-02-21T23:16:53.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.665+0000 7fdbb5400640 1 -- 172.21.15.28:0/3918319839 learned_addr learned my addr 172.21.15.28:0/3918319839 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:16:53.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.666+0000 7fdbb5400640 1 -- 172.21.15.28:0/3918319839 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdbb0156af0 msgr2=0x7fdbb006dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:16:53.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.666+0000 7fdbb5400640 1 --2- 172.21.15.28:0/3918319839 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdbb0156af0 0x7fdbb006dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:53.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.666+0000 7fdbb5400640 1 -- 172.21.15.28:0/3918319839 >> v1:172.21.15.81:6789/0 conn(0x7fdbb006ee40 legacy=0x7fdbb01536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:53.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.666+0000 7fdbb5400640 1 -- 172.21.15.28:0/3918319839 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbb0156310 con 0x7fdbb0150450 2024-02-21T23:16:53.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.667+0000 7fdbb5400640 1 --2- 172.21.15.28:0/3918319839 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdbb0150450 0x7fdbb00a5fd0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fdba4009870 tx=0x7fdba402b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e76aa95a79882ac server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:53.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.668+0000 7fdbafe00640 1 -- 172.21.15.28:0/3918319839 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdba4034070 con 0x7fdbb0150450 2024-02-21T23:16:53.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.668+0000 7fdbafe00640 1 -- 172.21.15.28:0/3918319839 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdba4030840 con 0x7fdbb0150450 2024-02-21T23:16:53.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.668+0000 7fdbafe00640 1 -- 172.21.15.28:0/3918319839 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdba4030c10 con 0x7fdbb0150450 2024-02-21T23:16:53.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.669+0000 7fdbb6800640 1 -- 172.21.15.28:0/3918319839 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdbb0150450 msgr2=0x7fdbb00a5fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:53.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.669+0000 7fdbb6800640 1 --2- 172.21.15.28:0/3918319839 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdbb0150450 0x7fdbb00a5fd0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fdba4009870 tx=0x7fdba402b980 comp rx=0 tx=0).stop 2024-02-21T23:16:53.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.669+0000 7fdbb6800640 1 -- 172.21.15.28:0/3918319839 shutdown_connections 2024-02-21T23:16:53.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.669+0000 7fdbb6800640 1 --2- 172.21.15.28:0/3918319839 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdbb0156af0 0x7fdbb006dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:53.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.669+0000 7fdbb6800640 1 --2- 172.21.15.28:0/3918319839 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdbb0150450 0x7fdbb00a5fd0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:53.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.669+0000 7fdbb6800640 1 -- 172.21.15.28:0/3918319839 >> 172.21.15.28:0/3918319839 conn(0x7fdbb006b630 msgr2=0x7fdbb00c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:16:53.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.669+0000 7fdbb6800640 1 -- 172.21.15.28:0/3918319839 shutdown_connections 2024-02-21T23:16:53.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.669+0000 7fdbb6800640 1 -- 172.21.15.28:0/3918319839 wait complete. 2024-02-21T23:16:53.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.671+0000 7fdbb6800640 1 Processor -- start 2024-02-21T23:16:53.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.671+0000 7fdbb6800640 1 -- start start 2024-02-21T23:16:53.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.673+0000 7fdbb6800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbb006ee40 0x7fdbb0149280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:53.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.673+0000 7fdbb6800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdbb0150450 0x7fdbb01ebf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:53.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.673+0000 7fdbb6800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdbb0156af0 0x7fdbb01452c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:53.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.673+0000 7fdbb6800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbb0156f80 con 0x7fdbb006ee40 2024-02-21T23:16:53.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.673+0000 7fdbb6800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbb00c2cb0 con 0x7fdbb0150450 2024-02-21T23:16:53.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.673+0000 7fdbb6800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbb0157100 con 0x7fdbb0156af0 2024-02-21T23:16:53.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.673+0000 7fdbb5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbb006ee40 0x7fdbb0149280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:53.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.673+0000 7fdbb5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbb006ee40 0x7fdbb0149280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60512/0 (socket says 172.21.15.28:60512) 2024-02-21T23:16:53.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.673+0000 7fdbb5400640 1 -- 172.21.15.28:0/3281610210 learned_addr learned my addr 172.21.15.28:0/3281610210 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:16:53.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.673+0000 7fdbb5e00640 1 --2- 172.21.15.28:0/3281610210 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdbb0156af0 0x7fdbb01452c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:53.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.674+0000 7fdbb4a00640 1 --2- 172.21.15.28:0/3281610210 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdbb0150450 0x7fdbb01ebf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:53.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.674+0000 7fdbb5400640 1 -- 172.21.15.28:0/3281610210 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdbb0156af0 msgr2=0x7fdbb01452c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:53.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.674+0000 7fdbb5400640 1 --2- 172.21.15.28:0/3281610210 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdbb0156af0 0x7fdbb01452c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:53.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.674+0000 7fdbb5400640 1 -- 172.21.15.28:0/3281610210 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdbb0150450 msgr2=0x7fdbb01ebf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:53.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.674+0000 7fdbb5400640 1 --2- 172.21.15.28:0/3281610210 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdbb0150450 0x7fdbb01ebf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:53.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.674+0000 7fdbb5400640 1 -- 172.21.15.28:0/3281610210 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbb00c0a60 con 0x7fdbb006ee40 2024-02-21T23:16:53.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.674+0000 7fdbb4a00640 1 --2- 172.21.15.28:0/3281610210 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdbb0150450 0x7fdbb01ebf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:16:53.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.674+0000 7fdbb5e00640 1 --2- 172.21.15.28:0/3281610210 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdbb0156af0 0x7fdbb01452c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:16:53.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.674+0000 7fdbb5400640 1 --2- 172.21.15.28:0/3281610210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbb006ee40 0x7fdbb0149280 secure :-1 s=READY pgs=2665 cs=0 l=1 rev1=1 crypto rx=0x7fdba4000c00 tx=0x7fdba4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:53.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.674+0000 7fdbad600640 1 -- 172.21.15.28:0/3281610210 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdba4048070 con 0x7fdbb006ee40 2024-02-21T23:16:53.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.675+0000 7fdbad600640 1 -- 172.21.15.28:0/3281610210 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdba4030640 con 0x7fdbb006ee40 2024-02-21T23:16:53.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.675+0000 7fdbad600640 1 -- 172.21.15.28:0/3281610210 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdba4034040 con 0x7fdbb006ee40 2024-02-21T23:16:53.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.675+0000 7fdbb6800640 1 -- 172.21.15.28:0/3281610210 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbb00c4190 con 0x7fdbb006ee40 2024-02-21T23:16:53.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.675+0000 7fdbb6800640 1 -- 172.21.15.28:0/3281610210 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbb014ba40 con 0x7fdbb006ee40 2024-02-21T23:16:53.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.676+0000 7fdbb6800640 1 -- 172.21.15.28:0/3281610210 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbb004f290 con 0x7fdbb006ee40 2024-02-21T23:16:53.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.677+0000 7fdbad600640 1 -- 172.21.15.28:0/3281610210 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdba4037070 con 0x7fdbb006ee40 2024-02-21T23:16:53.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.678+0000 7fdbad600640 1 --2- 172.21.15.28:0/3281610210 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdb9006b500 0x7fdb9006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:53.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.678+0000 7fdbad600640 1 -- 172.21.15.28:0/3281610210 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fdba40ac080 con 0x7fdbb006ee40 2024-02-21T23:16:53.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.679+0000 7fdbb4a00640 1 --2- 172.21.15.28:0/3281610210 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdb9006b500 0x7fdb9006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:53.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.680+0000 7fdbb4a00640 1 --2- 172.21.15.28:0/3281610210 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdb9006b500 0x7fdb9006d9c0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7fdbb01562e0 tx=0x7fdb9c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:53.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.684+0000 7fdbad600640 1 -- 172.21.15.28:0/3281610210 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fdba407ac60 con 0x7fdbb006ee40 2024-02-21T23:16:53.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.928+0000 7fdbb6800640 1 -- 172.21.15.28:0/3281610210 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdbb00c1f10 con 0x7fdbb006ee40 2024-02-21T23:16:53.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.929+0000 7fdbad600640 1 -- 172.21.15.28:0/3281610210 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7fdba407a560 con 0x7fdbb006ee40 2024-02-21T23:16:53.917 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:16:53.917 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:16:53.917 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:16:53.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.931+0000 7fdbb6800640 1 -- 172.21.15.28:0/3281610210 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdb9006b500 msgr2=0x7fdb9006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:53.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.932+0000 7fdbb6800640 1 --2- 172.21.15.28:0/3281610210 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdb9006b500 0x7fdb9006d9c0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7fdbb01562e0 tx=0x7fdb9c009040 comp rx=0 tx=0).stop 2024-02-21T23:16:53.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.932+0000 7fdbb6800640 1 -- 172.21.15.28:0/3281610210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbb006ee40 msgr2=0x7fdbb0149280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:53.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.932+0000 7fdbb6800640 1 --2- 172.21.15.28:0/3281610210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbb006ee40 0x7fdbb0149280 secure :-1 s=READY pgs=2665 cs=0 l=1 rev1=1 crypto rx=0x7fdba4000c00 tx=0x7fdba4004730 comp rx=0 tx=0).stop 2024-02-21T23:16:53.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.932+0000 7fdbb6800640 1 -- 172.21.15.28:0/3281610210 shutdown_connections 2024-02-21T23:16:53.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.932+0000 7fdbb6800640 1 --2- 172.21.15.28:0/3281610210 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdb9006b500 0x7fdb9006d9c0 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:53.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.932+0000 7fdbb6800640 1 --2- 172.21.15.28:0/3281610210 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdbb0156af0 0x7fdbb01452c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:53.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.932+0000 7fdbb6800640 1 --2- 172.21.15.28:0/3281610210 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdbb0150450 0x7fdbb01ebf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:53.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.932+0000 7fdbb6800640 1 --2- 172.21.15.28:0/3281610210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbb006ee40 0x7fdbb0149280 unknown :-1 s=CLOSED pgs=2665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:53.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.932+0000 7fdbb6800640 1 -- 172.21.15.28:0/3281610210 >> 172.21.15.28:0/3281610210 conn(0x7fdbb006b630 msgr2=0x7fdbb01dc2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:16:53.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.932+0000 7fdbb6800640 1 -- 172.21.15.28:0/3281610210 shutdown_connections 2024-02-21T23:16:53.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:53.932+0000 7fdbb6800640 1 -- 172.21.15.28:0/3281610210 wait complete. 2024-02-21T23:16:54.341 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:16:54.341 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:16:55.342 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:16:55.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:55.580+0000 7f609c800640 1 Processor -- start 2024-02-21T23:16:55.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:55.580+0000 7f609c800640 1 -- start start 2024-02-21T23:16:55.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:55.582+0000 7f609c800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f60940c11f0 0x7f6094153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:55.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:55.582+0000 7f609c800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6094152e70 con 0x7f609415b6d0 2024-02-21T23:16:55.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:55.582+0000 7f609c800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6094153150 con 0x7f60940c15d0 2024-02-21T23:16:55.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:55.582+0000 7f609c800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60940c1a60 con 0x7f60940c11f0 2024-02-21T23:16:55.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:55.582+0000 7f6099e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f60940c11f0 0x7f6094153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:55.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:55.582+0000 7f6099e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f60940c11f0 0x7f6094153950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44246/0 (socket says 172.21.15.28:44246) 2024-02-21T23:16:55.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:55.582+0000 7f609a800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f609415b6d0 0x7f609415dac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:51540/0 (socket says 172.21.15.28:51540) 2024-02-21T23:16:55.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:55.582+0000 7f6099e00640 1 -- 172.21.15.28:0/2598610782 learned_addr learned my addr 172.21.15.28:0/2598610782 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:16:55.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:55.583+0000 7f6099e00640 1 -- 172.21.15.28:0/2598610782 >> v1:172.21.15.81:6789/0 conn(0x7f60940c15d0 legacy=0x7f609406dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:16:55.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:55.583+0000 7f6099e00640 1 -- 172.21.15.28:0/2598610782 >> v1:172.21.15.28:6789/0 conn(0x7f609415b6d0 legacy=0x7f609415dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:55.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:55.583+0000 7f6099e00640 1 -- 172.21.15.28:0/2598610782 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6094153430 con 0x7f60940c11f0 2024-02-21T23:16:55.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:55.583+0000 7f6098a00640 1 -- 172.21.15.28:0/2598610782 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1918366774 0 0) 0x7f6094152e70 con 0x7f609415b6d0 2024-02-21T23:16:55.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:55.584+0000 7f6099e00640 1 --2- 172.21.15.28:0/2598610782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60940c11f0 0x7f6094153950 secure :-1 s=READY pgs=2667 cs=0 l=1 rev1=1 crypto rx=0x7f6088009870 tx=0x7f608802b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b79fa175c7207 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:55.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:55.584+0000 7f6098a00640 1 -- 172.21.15.28:0/2598610782 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60880317d0 con 0x7f60940c11f0 2024-02-21T23:16:55.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:55.584+0000 7f6098a00640 1 -- 172.21.15.28:0/2598610782 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6088037070 con 0x7f60940c11f0 2024-02-21T23:16:55.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:55.584+0000 7f6098a00640 1 -- 172.21.15.28:0/2598610782 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6088031b90 con 0x7f60940c11f0 2024-02-21T23:16:55.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:55.585+0000 7f609c800640 1 -- 172.21.15.28:0/2598610782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60940c11f0 msgr2=0x7f6094153950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:55.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:55.585+0000 7f609c800640 1 --2- 172.21.15.28:0/2598610782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60940c11f0 0x7f6094153950 secure :-1 s=READY pgs=2667 cs=0 l=1 rev1=1 crypto rx=0x7f6088009870 tx=0x7f608802b870 comp rx=0 tx=0).stop 2024-02-21T23:16:55.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:55.585+0000 7f609c800640 1 -- 172.21.15.28:0/2598610782 shutdown_connections 2024-02-21T23:16:55.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:55.585+0000 7f609c800640 1 --2- 172.21.15.28:0/2598610782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60940c11f0 0x7f6094153950 unknown :-1 s=CLOSED pgs=2667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:55.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:55.585+0000 7f609c800640 1 -- 172.21.15.28:0/2598610782 >> 172.21.15.28:0/2598610782 conn(0x7f60940c67f0 msgr2=0x7f60940c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:16:55.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:55.585+0000 7f609c800640 1 -- 172.21.15.28:0/2598610782 shutdown_connections 2024-02-21T23:16:55.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:55.585+0000 7f609c800640 1 -- 172.21.15.28:0/2598610782 wait complete. 2024-02-21T23:16:55.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:55.587+0000 7f609c800640 1 Processor -- start 2024-02-21T23:16:55.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:55.587+0000 7f609c800640 1 -- start start 2024-02-21T23:16:55.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:55.588+0000 7f609c800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60940c11f0 0x7f60940bfe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:55.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:55.588+0000 7f609c800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f60940c15d0 0x7f60941eb8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:55.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:55.588+0000 7f609c800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f609415b6d0 0x7f60941ebe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:55.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:55.588+0000 7f609c800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60940c4c00 con 0x7f60940c11f0 2024-02-21T23:16:55.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:55.588+0000 7f609c800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60940a5ec0 con 0x7f609415b6d0 2024-02-21T23:16:55.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:55.588+0000 7f609c800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60940a6110 con 0x7f60940c15d0 2024-02-21T23:16:55.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:55.588+0000 7f6099e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60940c11f0 0x7f60940bfe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:55.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:55.588+0000 7f6099e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60940c11f0 0x7f60940bfe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44258/0 (socket says 172.21.15.28:44258) 2024-02-21T23:16:55.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:55.588+0000 7f6099e00640 1 -- 172.21.15.28:0/3664135510 learned_addr learned my addr 172.21.15.28:0/3664135510 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:16:55.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:55.588+0000 7f609a800640 1 --2- 172.21.15.28:0/3664135510 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f609415b6d0 0x7f60941ebe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:55.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:55.588+0000 7f6099400640 1 --2- 172.21.15.28:0/3664135510 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f60940c15d0 0x7f60941eb8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:55.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:55.588+0000 7f6099e00640 1 -- 172.21.15.28:0/3664135510 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f60940c15d0 msgr2=0x7f60941eb8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:55.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:55.588+0000 7f6099e00640 1 --2- 172.21.15.28:0/3664135510 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f60940c15d0 0x7f60941eb8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:55.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:55.588+0000 7f6099e00640 1 -- 172.21.15.28:0/3664135510 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f609415b6d0 msgr2=0x7f60941ebe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:55.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:55.588+0000 7f6099e00640 1 --2- 172.21.15.28:0/3664135510 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f609415b6d0 0x7f60941ebe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:55.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:55.588+0000 7f6099e00640 1 -- 172.21.15.28:0/3664135510 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6094072cf0 con 0x7f60940c11f0 2024-02-21T23:16:55.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:55.589+0000 7f6099e00640 1 --2- 172.21.15.28:0/3664135510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60940c11f0 0x7f60940bfe90 secure :-1 s=READY pgs=2668 cs=0 l=1 rev1=1 crypto rx=0x7f60880097c0 tx=0x7f60880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:55.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:55.589+0000 7f608e000640 1 -- 172.21.15.28:0/3664135510 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6088032870 con 0x7f60940c11f0 2024-02-21T23:16:55.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:55.589+0000 7f608e000640 1 -- 172.21.15.28:0/3664135510 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6088032a10 con 0x7f60940c11f0 2024-02-21T23:16:55.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:55.589+0000 7f608e000640 1 -- 172.21.15.28:0/3664135510 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6088032e00 con 0x7f60940c11f0 2024-02-21T23:16:55.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:55.589+0000 7f609c800640 1 -- 172.21.15.28:0/3664135510 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60941ef420 con 0x7f60940c11f0 2024-02-21T23:16:55.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:55.589+0000 7f609c800640 1 -- 172.21.15.28:0/3664135510 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60940c03d0 con 0x7f60940c11f0 2024-02-21T23:16:55.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:55.590+0000 7f609c800640 1 -- 172.21.15.28:0/3664135510 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60940c0d90 con 0x7f60940c11f0 2024-02-21T23:16:55.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:55.592+0000 7f608e000640 1 -- 172.21.15.28:0/3664135510 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f608804c060 con 0x7f60940c11f0 2024-02-21T23:16:55.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:55.593+0000 7f608e000640 1 --2- 172.21.15.28:0/3664135510 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f607806b4d0 0x7f607806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:55.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:55.593+0000 7f608e000640 1 -- 172.21.15.28:0/3664135510 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f60880abc30 con 0x7f60940c11f0 2024-02-21T23:16:55.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:55.594+0000 7f6099400640 1 --2- 172.21.15.28:0/3664135510 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f607806b4d0 0x7f607806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:55.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:55.595+0000 7f6099400640 1 --2- 172.21.15.28:0/3664135510 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f607806b4d0 0x7f607806d990 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f608400a750 tx=0x7f6084008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:55.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:55.598+0000 7f608e000640 1 -- 172.21.15.28:0/3664135510 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f60940c0d90 con 0x7f60940c11f0 2024-02-21T23:16:55.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:55.840+0000 7f609c800640 1 -- 172.21.15.28:0/3664135510 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6094067f30 con 0x7f60940c11f0 2024-02-21T23:16:55.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:55.841+0000 7f608e000640 1 -- 172.21.15.28:0/3664135510 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7f6088040070 con 0x7f60940c11f0 2024-02-21T23:16:55.828 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:16:55.829 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:16:55.829 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:16:55.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:55.843+0000 7f609c800640 1 -- 172.21.15.28:0/3664135510 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f607806b4d0 msgr2=0x7f607806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:55.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:55.843+0000 7f609c800640 1 --2- 172.21.15.28:0/3664135510 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f607806b4d0 0x7f607806d990 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f608400a750 tx=0x7f6084008040 comp rx=0 tx=0).stop 2024-02-21T23:16:55.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:55.844+0000 7f609c800640 1 -- 172.21.15.28:0/3664135510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60940c11f0 msgr2=0x7f60940bfe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:55.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:55.844+0000 7f609c800640 1 --2- 172.21.15.28:0/3664135510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60940c11f0 0x7f60940bfe90 secure :-1 s=READY pgs=2668 cs=0 l=1 rev1=1 crypto rx=0x7f60880097c0 tx=0x7f60880046c0 comp rx=0 tx=0).stop 2024-02-21T23:16:55.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:55.844+0000 7f609c800640 1 -- 172.21.15.28:0/3664135510 shutdown_connections 2024-02-21T23:16:55.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:55.844+0000 7f609c800640 1 --2- 172.21.15.28:0/3664135510 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f607806b4d0 0x7f607806d990 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:55.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:55.844+0000 7f609c800640 1 --2- 172.21.15.28:0/3664135510 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f60940c15d0 0x7f60941eb8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:55.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:55.844+0000 7f609c800640 1 --2- 172.21.15.28:0/3664135510 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f609415b6d0 0x7f60941ebe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:55.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:55.844+0000 7f609c800640 1 --2- 172.21.15.28:0/3664135510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60940c11f0 0x7f60940bfe90 unknown :-1 s=CLOSED pgs=2668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:55.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:55.844+0000 7f609c800640 1 -- 172.21.15.28:0/3664135510 >> 172.21.15.28:0/3664135510 conn(0x7f60940c67f0 msgr2=0x7f60940c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:16:55.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:55.844+0000 7f609c800640 1 -- 172.21.15.28:0/3664135510 shutdown_connections 2024-02-21T23:16:55.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:55.844+0000 7f609c800640 1 -- 172.21.15.28:0/3664135510 wait complete. 2024-02-21T23:16:56.252 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:16:56.253 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:16:57.254 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:16:57.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.492+0000 7fb944600640 1 Processor -- start 2024-02-21T23:16:57.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.493+0000 7fb944600640 1 -- start start 2024-02-21T23:16:57.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.494+0000 7fb944600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb93c0c11f0 0x7fb93c153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:57.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.494+0000 7fb944600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fb93c15b6d0 0x7fb93c15dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:57.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.494+0000 7fb944600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb93c0c1a30 con 0x7fb93c0c15d0 2024-02-21T23:16:57.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.494+0000 7fb944600640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb93c0a5ec0 con 0x7fb93c15b6d0 2024-02-21T23:16:57.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.494+0000 7fb944600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb93c152e70 con 0x7fb93c0c11f0 2024-02-21T23:16:57.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.495+0000 7fb941c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb93c0c11f0 0x7fb93c153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:57.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.495+0000 7fb942600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fb93c15b6d0 0x7fb93c15dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:57.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.495+0000 7fb941c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb93c0c11f0 0x7fb93c153950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3301/0 says I am v2:172.21.15.28:41670/0 (socket says 172.21.15.28:41670) 2024-02-21T23:16:57.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.495+0000 7fb941c00640 1 -- 172.21.15.28:0/3189140140 learned_addr learned my addr 172.21.15.28:0/3189140140 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:16:57.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.496+0000 7fb941c00640 1 -- 172.21.15.28:0/3189140140 >> v2:172.21.15.81:3300/0 conn(0x7fb93c15b6d0 msgr2=0x7fb93c15dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:16:57.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.496+0000 7fb941c00640 1 --2- 172.21.15.28:0/3189140140 >> v2:172.21.15.81:3300/0 conn(0x7fb93c15b6d0 0x7fb93c15dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:57.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.496+0000 7fb941c00640 1 -- 172.21.15.28:0/3189140140 >> v1:172.21.15.81:6789/0 conn(0x7fb93c0c15d0 legacy=0x7fb93c06dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:57.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.496+0000 7fb941c00640 1 -- 172.21.15.28:0/3189140140 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb93c153150 con 0x7fb93c0c11f0 2024-02-21T23:16:57.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.497+0000 7fb941c00640 1 --2- 172.21.15.28:0/3189140140 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb93c0c11f0 0x7fb93c153950 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fb930009870 tx=0x7fb93002b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7eaba36df24051f9 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:57.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.497+0000 7fb93be00640 1 -- 172.21.15.28:0/3189140140 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9300317c0 con 0x7fb93c0c11f0 2024-02-21T23:16:57.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.498+0000 7fb93be00640 1 -- 172.21.15.28:0/3189140140 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb930037070 con 0x7fb93c0c11f0 2024-02-21T23:16:57.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.498+0000 7fb93be00640 1 -- 172.21.15.28:0/3189140140 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb930031b80 con 0x7fb93c0c11f0 2024-02-21T23:16:57.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.498+0000 7fb944600640 1 -- 172.21.15.28:0/3189140140 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb93c0c11f0 msgr2=0x7fb93c153950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:57.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.498+0000 7fb944600640 1 --2- 172.21.15.28:0/3189140140 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb93c0c11f0 0x7fb93c153950 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fb930009870 tx=0x7fb93002b860 comp rx=0 tx=0).stop 2024-02-21T23:16:57.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.498+0000 7fb944600640 1 -- 172.21.15.28:0/3189140140 shutdown_connections 2024-02-21T23:16:57.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.498+0000 7fb944600640 1 --2- 172.21.15.28:0/3189140140 >> v2:172.21.15.81:3300/0 conn(0x7fb93c15b6d0 0x7fb93c15dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:57.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.498+0000 7fb944600640 1 --2- 172.21.15.28:0/3189140140 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb93c0c11f0 0x7fb93c153950 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:57.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.498+0000 7fb944600640 1 -- 172.21.15.28:0/3189140140 >> 172.21.15.28:0/3189140140 conn(0x7fb93c0c67f0 msgr2=0x7fb93c0c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:16:57.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.498+0000 7fb944600640 1 -- 172.21.15.28:0/3189140140 shutdown_connections 2024-02-21T23:16:57.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.498+0000 7fb944600640 1 -- 172.21.15.28:0/3189140140 wait complete. 2024-02-21T23:16:57.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.500+0000 7fb944600640 1 Processor -- start 2024-02-21T23:16:57.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.501+0000 7fb944600640 1 -- start start 2024-02-21T23:16:57.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.502+0000 7fb944600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb93c0c11f0 0x7fb93c141b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:57.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.502+0000 7fb944600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb93c0c15d0 0x7fb93c0bfc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:57.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.502+0000 7fb944600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb93c15b6d0 0x7fb93c0714f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:57.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.502+0000 7fb944600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb93c0c4c00 con 0x7fb93c15b6d0 2024-02-21T23:16:57.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.502+0000 7fb944600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb93c14d1c0 con 0x7fb93c0c15d0 2024-02-21T23:16:57.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.502+0000 7fb944600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb93c0a6110 con 0x7fb93c0c11f0 2024-02-21T23:16:57.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.502+0000 7fb942600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb93c15b6d0 0x7fb93c0714f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:57.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.502+0000 7fb942600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb93c15b6d0 0x7fb93c0714f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44260/0 (socket says 172.21.15.28:44260) 2024-02-21T23:16:57.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.502+0000 7fb942600640 1 -- 172.21.15.28:0/4171593308 learned_addr learned my addr 172.21.15.28:0/4171593308 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:16:57.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.502+0000 7fb941c00640 1 --2- 172.21.15.28:0/4171593308 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb93c0c11f0 0x7fb93c141b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:57.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.502+0000 7fb941200640 1 --2- 172.21.15.28:0/4171593308 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb93c0c15d0 0x7fb93c0bfc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:57.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.503+0000 7fb942600640 1 -- 172.21.15.28:0/4171593308 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb93c0c11f0 msgr2=0x7fb93c141b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:57.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.503+0000 7fb942600640 1 --2- 172.21.15.28:0/4171593308 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb93c0c11f0 0x7fb93c141b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:57.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.503+0000 7fb942600640 1 -- 172.21.15.28:0/4171593308 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb93c0c15d0 msgr2=0x7fb93c0bfc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:57.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.503+0000 7fb942600640 1 --2- 172.21.15.28:0/4171593308 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb93c0c15d0 0x7fb93c0bfc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:57.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.503+0000 7fb942600640 1 -- 172.21.15.28:0/4171593308 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb93c1eb830 con 0x7fb93c15b6d0 2024-02-21T23:16:57.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.503+0000 7fb942600640 1 --2- 172.21.15.28:0/4171593308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb93c15b6d0 0x7fb93c0714f0 secure :-1 s=READY pgs=2669 cs=0 l=1 rev1=1 crypto rx=0x7fb93400efc0 tx=0x7fb93400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:57.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.503+0000 7fb939600640 1 -- 172.21.15.28:0/4171593308 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb934018020 con 0x7fb93c15b6d0 2024-02-21T23:16:57.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.504+0000 7fb939600640 1 -- 172.21.15.28:0/4171593308 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb934014480 con 0x7fb93c15b6d0 2024-02-21T23:16:57.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.504+0000 7fb939600640 1 -- 172.21.15.28:0/4171593308 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb934009e60 con 0x7fb93c15b6d0 2024-02-21T23:16:57.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.504+0000 7fb944600640 1 -- 172.21.15.28:0/4171593308 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb93c1eb550 con 0x7fb93c15b6d0 2024-02-21T23:16:57.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.504+0000 7fb944600640 1 -- 172.21.15.28:0/4171593308 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb93c142090 con 0x7fb93c15b6d0 2024-02-21T23:16:57.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.505+0000 7fb944600640 1 -- 172.21.15.28:0/4171593308 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb93c067f30 con 0x7fb93c15b6d0 2024-02-21T23:16:57.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.506+0000 7fb939600640 1 -- 172.21.15.28:0/4171593308 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb934005d10 con 0x7fb93c15b6d0 2024-02-21T23:16:57.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.507+0000 7fb939600640 1 --2- 172.21.15.28:0/4171593308 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb91c06b4d0 0x7fb91c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:57.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.507+0000 7fb939600640 1 -- 172.21.15.28:0/4171593308 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fb93408b1f0 con 0x7fb93c15b6d0 2024-02-21T23:16:57.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.508+0000 7fb941c00640 1 --2- 172.21.15.28:0/4171593308 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb91c06b4d0 0x7fb91c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:57.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.509+0000 7fb941c00640 1 --2- 172.21.15.28:0/4171593308 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb91c06b4d0 0x7fb91c06d990 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fb930036000 tx=0x7fb93003d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:57.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.512+0000 7fb939600640 1 -- 172.21.15.28:0/4171593308 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb934022030 con 0x7fb93c15b6d0 2024-02-21T23:16:57.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.781+0000 7fb944600640 1 -- 172.21.15.28:0/4171593308 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb93c0c0960 con 0x7fb93c15b6d0 2024-02-21T23:16:57.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.782+0000 7fb939600640 1 -- 172.21.15.28:0/4171593308 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7fb934010040 con 0x7fb93c15b6d0 2024-02-21T23:16:57.770 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:16:57.770 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:16:57.770 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:16:57.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.785+0000 7fb944600640 1 -- 172.21.15.28:0/4171593308 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb91c06b4d0 msgr2=0x7fb91c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:57.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.785+0000 7fb944600640 1 --2- 172.21.15.28:0/4171593308 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb91c06b4d0 0x7fb91c06d990 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fb930036000 tx=0x7fb93003d040 comp rx=0 tx=0).stop 2024-02-21T23:16:57.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.785+0000 7fb944600640 1 -- 172.21.15.28:0/4171593308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb93c15b6d0 msgr2=0x7fb93c0714f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:57.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.785+0000 7fb944600640 1 --2- 172.21.15.28:0/4171593308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb93c15b6d0 0x7fb93c0714f0 secure :-1 s=READY pgs=2669 cs=0 l=1 rev1=1 crypto rx=0x7fb93400efc0 tx=0x7fb93400c490 comp rx=0 tx=0).stop 2024-02-21T23:16:57.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.785+0000 7fb944600640 1 -- 172.21.15.28:0/4171593308 shutdown_connections 2024-02-21T23:16:57.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.785+0000 7fb944600640 1 --2- 172.21.15.28:0/4171593308 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb91c06b4d0 0x7fb91c06d990 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:57.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.785+0000 7fb944600640 1 --2- 172.21.15.28:0/4171593308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb93c15b6d0 0x7fb93c0714f0 unknown :-1 s=CLOSED pgs=2669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:57.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.785+0000 7fb944600640 1 --2- 172.21.15.28:0/4171593308 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb93c0c15d0 0x7fb93c0bfc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:57.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.785+0000 7fb944600640 1 --2- 172.21.15.28:0/4171593308 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb93c0c11f0 0x7fb93c141b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:57.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.785+0000 7fb944600640 1 -- 172.21.15.28:0/4171593308 >> 172.21.15.28:0/4171593308 conn(0x7fb93c0c67f0 msgr2=0x7fb93c1519f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:16:57.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.785+0000 7fb944600640 1 -- 172.21.15.28:0/4171593308 shutdown_connections 2024-02-21T23:16:57.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:57.785+0000 7fb944600640 1 -- 172.21.15.28:0/4171593308 wait complete. 2024-02-21T23:16:58.198 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:16:58.198 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:16:59.199 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:16:59.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.440+0000 7f072b200640 1 Processor -- start 2024-02-21T23:16:59.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.440+0000 7f072b200640 1 -- start start 2024-02-21T23:16:59.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.441+0000 7f072b200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f07240c5510 0x7f07240a7e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:59.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.441+0000 7f072b200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0724155040 0x7f0724155440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:59.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.441+0000 7f072b200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07240c8040 con 0x7f07240c7320 2024-02-21T23:16:59.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.441+0000 7f072b200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07240a8380 con 0x7f07240c5510 2024-02-21T23:16:59.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.441+0000 7f072b200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07240c85c0 con 0x7f0724155040 2024-02-21T23:16:59.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.441+0000 7f0723e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f07240c5510 0x7f07240a7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:59.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.441+0000 7f0723e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f07240c5510 0x7f07240a7e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44268/0 (socket says 172.21.15.28:44268) 2024-02-21T23:16:59.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.441+0000 7f0723e00640 1 -- 172.21.15.28:0/4034610506 learned_addr learned my addr 172.21.15.28:0/4034610506 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:16:59.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.442+0000 7f0729200640 1 --2- 172.21.15.28:0/4034610506 >> v2:172.21.15.81:3300/0 conn(0x7f0724155040 0x7f0724155440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:59.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.442+0000 7f0723e00640 1 -- 172.21.15.28:0/4034610506 >> v2:172.21.15.81:3300/0 conn(0x7f0724155040 msgr2=0x7f0724155440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:16:59.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.442+0000 7f0723e00640 1 --2- 172.21.15.28:0/4034610506 >> v2:172.21.15.81:3300/0 conn(0x7f0724155040 0x7f0724155440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:59.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.442+0000 7f0723e00640 1 -- 172.21.15.28:0/4034610506 >> v1:172.21.15.81:6789/0 conn(0x7f07240c7320 legacy=0x7f07241521d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:16:59.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.442+0000 7f0723e00640 1 -- 172.21.15.28:0/4034610506 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07240c82e0 con 0x7f07240c5510 2024-02-21T23:16:59.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.442+0000 7f0729200640 1 --2- 172.21.15.28:0/4034610506 >> v2:172.21.15.81:3300/0 conn(0x7f0724155040 0x7f0724155440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:16:59.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.443+0000 7f0723e00640 1 --2- 172.21.15.28:0/4034610506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07240c5510 0x7f07240a7e40 secure :-1 s=READY pgs=2670 cs=0 l=1 rev1=1 crypto rx=0x7f07240c9890 tx=0x7f071402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37bdb0f72e648a85 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:59.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.443+0000 7f0722a00640 1 -- 172.21.15.28:0/4034610506 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f071402ca70 con 0x7f07240c5510 2024-02-21T23:16:59.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.443+0000 7f0722a00640 1 -- 172.21.15.28:0/4034610506 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f071402cc10 con 0x7f07240c5510 2024-02-21T23:16:59.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.443+0000 7f0722a00640 1 -- 172.21.15.28:0/4034610506 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07140343f0 con 0x7f07240c5510 2024-02-21T23:16:59.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.444+0000 7f072b200640 1 -- 172.21.15.28:0/4034610506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07240c5510 msgr2=0x7f07240a7e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:59.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.444+0000 7f072b200640 1 --2- 172.21.15.28:0/4034610506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07240c5510 0x7f07240a7e40 secure :-1 s=READY pgs=2670 cs=0 l=1 rev1=1 crypto rx=0x7f07240c9890 tx=0x7f071402bed0 comp rx=0 tx=0).stop 2024-02-21T23:16:59.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.444+0000 7f072b200640 1 -- 172.21.15.28:0/4034610506 shutdown_connections 2024-02-21T23:16:59.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.444+0000 7f072b200640 1 --2- 172.21.15.28:0/4034610506 >> v2:172.21.15.81:3300/0 conn(0x7f0724155040 0x7f0724155440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:59.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.444+0000 7f072b200640 1 --2- 172.21.15.28:0/4034610506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07240c5510 0x7f07240a7e40 unknown :-1 s=CLOSED pgs=2670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:59.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.444+0000 7f072b200640 1 -- 172.21.15.28:0/4034610506 >> 172.21.15.28:0/4034610506 conn(0x7f07240c8810 msgr2=0x7f07240c8bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:16:59.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.444+0000 7f072b200640 1 -- 172.21.15.28:0/4034610506 shutdown_connections 2024-02-21T23:16:59.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.445+0000 7f072b200640 1 -- 172.21.15.28:0/4034610506 wait complete. 2024-02-21T23:16:59.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.446+0000 7f072b200640 1 Processor -- start 2024-02-21T23:16:59.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.446+0000 7f072b200640 1 -- start start 2024-02-21T23:16:59.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.447+0000 7f072b200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f07240c5510 0x7f0724156570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:59.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.447+0000 7f072b200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f07240c7320 0x7f07241e1b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:59.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.447+0000 7f072b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0724155040 0x7f07241e20a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:59.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.447+0000 7f072b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07240c58f0 con 0x7f0724155040 2024-02-21T23:16:59.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.447+0000 7f072b200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07240c6cb0 con 0x7f07240c5510 2024-02-21T23:16:59.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.447+0000 7f072b200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07240c5a70 con 0x7f07240c7320 2024-02-21T23:16:59.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.448+0000 7f0729200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0724155040 0x7f07241e20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:59.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.448+0000 7f0729200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0724155040 0x7f07241e20a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44272/0 (socket says 172.21.15.28:44272) 2024-02-21T23:16:59.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.448+0000 7f0729200640 1 -- 172.21.15.28:0/585733610 learned_addr learned my addr 172.21.15.28:0/585733610 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:16:59.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.448+0000 7f0723400640 1 --2- 172.21.15.28:0/585733610 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f07240c7320 0x7f07241e1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:59.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.448+0000 7f0723e00640 1 --2- 172.21.15.28:0/585733610 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f07240c5510 0x7f0724156570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:59.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.448+0000 7f0729200640 1 -- 172.21.15.28:0/585733610 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f07240c7320 msgr2=0x7f07241e1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:59.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.448+0000 7f0729200640 1 --2- 172.21.15.28:0/585733610 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f07240c7320 0x7f07241e1b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:59.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.448+0000 7f0729200640 1 -- 172.21.15.28:0/585733610 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f07240c5510 msgr2=0x7f0724156570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:59.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.448+0000 7f0729200640 1 --2- 172.21.15.28:0/585733610 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f07240c5510 0x7f0724156570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:59.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.448+0000 7f0729200640 1 -- 172.21.15.28:0/585733610 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f072406fee0 con 0x7f0724155040 2024-02-21T23:16:59.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.448+0000 7f0729200640 1 --2- 172.21.15.28:0/585733610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0724155040 0x7f07241e20a0 secure :-1 s=READY pgs=2671 cs=0 l=1 rev1=1 crypto rx=0x7f071c00c7f0 tx=0x7f071c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:59.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.449+0000 7f071be00640 1 -- 172.21.15.28:0/585733610 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f071c015020 con 0x7f0724155040 2024-02-21T23:16:59.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.449+0000 7f071be00640 1 -- 172.21.15.28:0/585733610 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f071c004230 con 0x7f0724155040 2024-02-21T23:16:59.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.449+0000 7f072b200640 1 -- 172.21.15.28:0/585733610 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07240701c0 con 0x7f0724155040 2024-02-21T23:16:59.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.449+0000 7f071be00640 1 -- 172.21.15.28:0/585733610 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f071c003610 con 0x7f0724155040 2024-02-21T23:16:59.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.449+0000 7f072b200640 1 -- 172.21.15.28:0/585733610 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0724070780 con 0x7f0724155040 2024-02-21T23:16:59.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.450+0000 7f072b200640 1 -- 172.21.15.28:0/585733610 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0724068640 con 0x7f0724155040 2024-02-21T23:16:59.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.451+0000 7f071be00640 1 -- 172.21.15.28:0/585733610 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f071c018020 con 0x7f0724155040 2024-02-21T23:16:59.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.452+0000 7f071be00640 1 --2- 172.21.15.28:0/585733610 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f070006b4d0 0x7f070006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:16:59.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.452+0000 7f071be00640 1 -- 172.21.15.28:0/585733610 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f071c08a6b0 con 0x7f0724155040 2024-02-21T23:16:59.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.453+0000 7f0723e00640 1 --2- 172.21.15.28:0/585733610 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f070006b4d0 0x7f070006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:16:59.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.454+0000 7f0723e00640 1 --2- 172.21.15.28:0/585733610 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f070006b4d0 0x7f070006d990 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f0714035000 tx=0x7f0714030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:16:59.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.457+0000 7f071be00640 1 -- 172.21.15.28:0/585733610 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f071c01f020 con 0x7f0724155040 2024-02-21T23:16:59.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.695+0000 7f072b200640 1 -- 172.21.15.28:0/585733610 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f07240c6710 con 0x7f0724155040 2024-02-21T23:16:59.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.696+0000 7f071be00640 1 -- 172.21.15.28:0/585733610 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7f071c0593f0 con 0x7f0724155040 2024-02-21T23:16:59.685 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:16:59.685 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:16:59.685 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:16:59.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.700+0000 7f072b200640 1 -- 172.21.15.28:0/585733610 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f070006b4d0 msgr2=0x7f070006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:59.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.700+0000 7f072b200640 1 --2- 172.21.15.28:0/585733610 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f070006b4d0 0x7f070006d990 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f0714035000 tx=0x7f0714030000 comp rx=0 tx=0).stop 2024-02-21T23:16:59.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.700+0000 7f072b200640 1 -- 172.21.15.28:0/585733610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0724155040 msgr2=0x7f07241e20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:16:59.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.700+0000 7f072b200640 1 --2- 172.21.15.28:0/585733610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0724155040 0x7f07241e20a0 secure :-1 s=READY pgs=2671 cs=0 l=1 rev1=1 crypto rx=0x7f071c00c7f0 tx=0x7f071c00ccb0 comp rx=0 tx=0).stop 2024-02-21T23:16:59.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.700+0000 7f072b200640 1 -- 172.21.15.28:0/585733610 shutdown_connections 2024-02-21T23:16:59.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.700+0000 7f072b200640 1 --2- 172.21.15.28:0/585733610 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f070006b4d0 0x7f070006d990 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:59.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.700+0000 7f072b200640 1 --2- 172.21.15.28:0/585733610 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f07240c7320 0x7f07241e1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:59.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.700+0000 7f072b200640 1 --2- 172.21.15.28:0/585733610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0724155040 0x7f07241e20a0 unknown :-1 s=CLOSED pgs=2671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:59.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.700+0000 7f072b200640 1 --2- 172.21.15.28:0/585733610 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f07240c5510 0x7f0724156570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:16:59.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.700+0000 7f072b200640 1 -- 172.21.15.28:0/585733610 >> 172.21.15.28:0/585733610 conn(0x7f07240c8810 msgr2=0x7f0724150940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:16:59.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.700+0000 7f072b200640 1 -- 172.21.15.28:0/585733610 shutdown_connections 2024-02-21T23:16:59.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:16:59.700+0000 7f072b200640 1 -- 172.21.15.28:0/585733610 wait complete. 2024-02-21T23:17:00.118 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:17:00.118 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:17:01.119 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:17:01.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.345+0000 7fbd0c200640 1 Processor -- start 2024-02-21T23:17:01.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.346+0000 7fbd0c200640 1 -- start start 2024-02-21T23:17:01.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.347+0000 7fbd0c200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbd041569d0 0x7fbd040c3280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:01.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.347+0000 7fbd0c200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd04159900 con 0x7fbd040c4fe0 2024-02-21T23:17:01.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.347+0000 7fbd0c200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd04159be0 con 0x7fbd04159400 2024-02-21T23:17:01.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.347+0000 7fbd0c200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd04159ec0 con 0x7fbd041569d0 2024-02-21T23:17:01.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.348+0000 7fbd08e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fbd040c4fe0 0x7fbd0406e0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:51550/0 (socket says 172.21.15.28:51550) 2024-02-21T23:17:01.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.348+0000 7fbd08e00640 1 -- 172.21.15.28:0/965944316 learned_addr learned my addr 172.21.15.28:0/965944316 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:17:01.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.348+0000 7fbd09800640 1 --2- 172.21.15.28:0/965944316 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbd041569d0 0x7fbd040c3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:01.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.349+0000 7fbd03e00640 1 -- 172.21.15.28:0/965944316 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1609534695 0 0) 0x7fbd04159900 con 0x7fbd040c4fe0 2024-02-21T23:17:01.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.349+0000 7fbd03e00640 1 -- 172.21.15.28:0/965944316 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbcec003520 con 0x7fbd040c4fe0 2024-02-21T23:17:01.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.349+0000 7fbd09800640 1 -- 172.21.15.28:0/965944316 >> v1:172.21.15.81:6789/0 conn(0x7fbd04159400 legacy=0x7fbd0406e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:01.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.349+0000 7fbd09800640 1 -- 172.21.15.28:0/965944316 >> v1:172.21.15.28:6789/0 conn(0x7fbd040c4fe0 legacy=0x7fbd0406e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:01.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.349+0000 7fbd03e00640 1 -- 172.21.15.28:0/965944316 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 809944071 0 0) 0x7fbcec003520 con 0x7fbd040c4fe0 2024-02-21T23:17:01.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.349+0000 7fbd09800640 1 -- 172.21.15.28:0/965944316 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd0415a1a0 con 0x7fbd041569d0 2024-02-21T23:17:01.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.351+0000 7fbd09800640 1 --2- 172.21.15.28:0/965944316 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbd041569d0 0x7fbd040c3280 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fbcf80097d0 tx=0x7fbcf802b770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=476a92292d9a8ab4 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:01.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.351+0000 7fbd03e00640 1 -- 172.21.15.28:0/965944316 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcf8037070 con 0x7fbd041569d0 2024-02-21T23:17:01.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.352+0000 7fbd03e00640 1 -- 172.21.15.28:0/965944316 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcf8030510 con 0x7fbd041569d0 2024-02-21T23:17:01.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.352+0000 7fbd03e00640 1 -- 172.21.15.28:0/965944316 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcf0001670 con 0x7fbd041569d0 2024-02-21T23:17:01.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.352+0000 7fbd0c200640 1 -- 172.21.15.28:0/965944316 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbd041569d0 msgr2=0x7fbd040c3280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:01.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.352+0000 7fbd0c200640 1 --2- 172.21.15.28:0/965944316 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbd041569d0 0x7fbd040c3280 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fbcf80097d0 tx=0x7fbcf802b770 comp rx=0 tx=0).stop 2024-02-21T23:17:01.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.352+0000 7fbd0c200640 1 -- 172.21.15.28:0/965944316 shutdown_connections 2024-02-21T23:17:01.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.352+0000 7fbd0c200640 1 --2- 172.21.15.28:0/965944316 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbd041569d0 0x7fbd040c3280 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:01.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.352+0000 7fbd0c200640 1 -- 172.21.15.28:0/965944316 >> 172.21.15.28:0/965944316 conn(0x7fbd040c67f0 msgr2=0x7fbd040c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:01.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.352+0000 7fbd0c200640 1 -- 172.21.15.28:0/965944316 shutdown_connections 2024-02-21T23:17:01.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.352+0000 7fbd0c200640 1 -- 172.21.15.28:0/965944316 wait complete. 2024-02-21T23:17:01.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.354+0000 7fbd0c200640 1 Processor -- start 2024-02-21T23:17:01.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.354+0000 7fbd0c200640 1 -- start start 2024-02-21T23:17:01.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.355+0000 7fbd0c200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbd040c4fe0 0x7fbd04149580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:01.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.355+0000 7fbd0c200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbd041569d0 0x7fbd04152940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:01.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.355+0000 7fbd0c200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd04159400 0x7fbd04152e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:01.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.355+0000 7fbd0c200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd040c3940 con 0x7fbd04159400 2024-02-21T23:17:01.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.355+0000 7fbd0c200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd040c37c0 con 0x7fbd040c4fe0 2024-02-21T23:17:01.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.355+0000 7fbd0c200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd040c3ac0 con 0x7fbd041569d0 2024-02-21T23:17:01.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.355+0000 7fbd0a200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd04159400 0x7fbd04152e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:01.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.356+0000 7fbd0a200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd04159400 0x7fbd04152e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44276/0 (socket says 172.21.15.28:44276) 2024-02-21T23:17:01.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.356+0000 7fbd0a200640 1 -- 172.21.15.28:0/3869727844 learned_addr learned my addr 172.21.15.28:0/3869727844 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:01.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.356+0000 7fbd08e00640 1 --2- 172.21.15.28:0/3869727844 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbd041569d0 0x7fbd04152940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:01.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.356+0000 7fbd09800640 1 --2- 172.21.15.28:0/3869727844 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbd040c4fe0 0x7fbd04149580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:01.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.356+0000 7fbd0a200640 1 -- 172.21.15.28:0/3869727844 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbd041569d0 msgr2=0x7fbd04152940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:01.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.356+0000 7fbd0a200640 1 --2- 172.21.15.28:0/3869727844 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbd041569d0 0x7fbd04152940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:01.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.356+0000 7fbd0a200640 1 -- 172.21.15.28:0/3869727844 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbd040c4fe0 msgr2=0x7fbd04149580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:01.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.356+0000 7fbd0a200640 1 --2- 172.21.15.28:0/3869727844 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbd040c4fe0 0x7fbd04149580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:01.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.356+0000 7fbd0a200640 1 -- 172.21.15.28:0/3869727844 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd0414a0e0 con 0x7fbd04159400 2024-02-21T23:17:01.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.356+0000 7fbd0a200640 1 --2- 172.21.15.28:0/3869727844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd04159400 0x7fbd04152e80 secure :-1 s=READY pgs=2673 cs=0 l=1 rev1=1 crypto rx=0x7fbcfc00c680 tx=0x7fbcfc00cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:01.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.357+0000 7fbd01600640 1 -- 172.21.15.28:0/3869727844 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcfc003b70 con 0x7fbd04159400 2024-02-21T23:17:01.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.357+0000 7fbd01600640 1 -- 172.21.15.28:0/3869727844 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcfc003d10 con 0x7fbd04159400 2024-02-21T23:17:01.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.357+0000 7fbd0c200640 1 -- 172.21.15.28:0/3869727844 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd0414a3c0 con 0x7fbd04159400 2024-02-21T23:17:01.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.357+0000 7fbd01600640 1 -- 172.21.15.28:0/3869727844 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcfc005310 con 0x7fbd04159400 2024-02-21T23:17:01.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.357+0000 7fbd0c200640 1 -- 172.21.15.28:0/3869727844 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd040a9c30 con 0x7fbd04159400 2024-02-21T23:17:01.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.358+0000 7fbd0c200640 1 -- 172.21.15.28:0/3869727844 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd04067f30 con 0x7fbd04159400 2024-02-21T23:17:01.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.359+0000 7fbd01600640 1 -- 172.21.15.28:0/3869727844 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbcfc0054b0 con 0x7fbd04159400 2024-02-21T23:17:01.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.360+0000 7fbd01600640 1 --2- 172.21.15.28:0/3869727844 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbce406b4d0 0x7fbce406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:01.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.361+0000 7fbd01600640 1 -- 172.21.15.28:0/3869727844 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fbcfc08a360 con 0x7fbd04159400 2024-02-21T23:17:01.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.361+0000 7fbd09800640 1 --2- 172.21.15.28:0/3869727844 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbce406b4d0 0x7fbce406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:01.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.362+0000 7fbd09800640 1 --2- 172.21.15.28:0/3869727844 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbce406b4d0 0x7fbce406d990 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7fbcf8034000 tx=0x7fbcf8008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:01.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.365+0000 7fbd01600640 1 -- 172.21.15.28:0/3869727844 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fbcfc058eb0 con 0x7fbd04159400 2024-02-21T23:17:01.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.665+0000 7fbd0c200640 1 -- 172.21.15.28:0/3869727844 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbd0406f1d0 con 0x7fbd04159400 2024-02-21T23:17:01.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.666+0000 7fbd01600640 1 -- 172.21.15.28:0/3869727844 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7fbcfc0587b0 con 0x7fbd04159400 2024-02-21T23:17:01.654 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:17:01.655 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:17:01.655 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:17:01.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.669+0000 7fbd0c200640 1 -- 172.21.15.28:0/3869727844 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbce406b4d0 msgr2=0x7fbce406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:01.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.669+0000 7fbd0c200640 1 --2- 172.21.15.28:0/3869727844 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbce406b4d0 0x7fbce406d990 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7fbcf8034000 tx=0x7fbcf8008030 comp rx=0 tx=0).stop 2024-02-21T23:17:01.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.669+0000 7fbd0c200640 1 -- 172.21.15.28:0/3869727844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd04159400 msgr2=0x7fbd04152e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:01.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.669+0000 7fbd0c200640 1 --2- 172.21.15.28:0/3869727844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd04159400 0x7fbd04152e80 secure :-1 s=READY pgs=2673 cs=0 l=1 rev1=1 crypto rx=0x7fbcfc00c680 tx=0x7fbcfc00cb40 comp rx=0 tx=0).stop 2024-02-21T23:17:01.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.670+0000 7fbd0c200640 1 -- 172.21.15.28:0/3869727844 shutdown_connections 2024-02-21T23:17:01.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.670+0000 7fbd0c200640 1 --2- 172.21.15.28:0/3869727844 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbce406b4d0 0x7fbce406d990 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:01.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.670+0000 7fbd0c200640 1 --2- 172.21.15.28:0/3869727844 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbd041569d0 0x7fbd04152940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:01.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.670+0000 7fbd0c200640 1 --2- 172.21.15.28:0/3869727844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd04159400 0x7fbd04152e80 unknown :-1 s=CLOSED pgs=2673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:01.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.670+0000 7fbd0c200640 1 --2- 172.21.15.28:0/3869727844 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbd040c4fe0 0x7fbd04149580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:01.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.670+0000 7fbd0c200640 1 -- 172.21.15.28:0/3869727844 >> 172.21.15.28:0/3869727844 conn(0x7fbd040c67f0 msgr2=0x7fbd040c0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:01.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.670+0000 7fbd0c200640 1 -- 172.21.15.28:0/3869727844 shutdown_connections 2024-02-21T23:17:01.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:01.670+0000 7fbd0c200640 1 -- 172.21.15.28:0/3869727844 wait complete. 2024-02-21T23:17:02.079 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:17:02.079 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:17:03.081 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:17:03.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.311+0000 7f8806400640 1 Processor -- start 2024-02-21T23:17:03.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.312+0000 7f8806400640 1 -- start start 2024-02-21T23:17:03.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.314+0000 7f8806400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8800151e30 0x7f88000a7690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:03.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.314+0000 7f8806400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f88000a7bd0 0x7f880006e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:03.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.314+0000 7f8806400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88000c2790 con 0x7f880014b750 2024-02-21T23:17:03.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.314+0000 7f8806400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88001519a0 con 0x7f88000a7bd0 2024-02-21T23:17:03.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.314+0000 7f8806400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8800152290 con 0x7f8800151e30 2024-02-21T23:17:03.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.314+0000 7f8805a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f88000a7bd0 0x7f880006e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:03.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.314+0000 7f8805a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f88000a7bd0 0x7f880006e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44280/0 (socket says 172.21.15.28:44280) 2024-02-21T23:17:03.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.314+0000 7f8805a00640 1 -- 172.21.15.28:0/2316873760 learned_addr learned my addr 172.21.15.28:0/2316873760 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:03.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.315+0000 7f87f7e00640 1 --2- 172.21.15.28:0/2316873760 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8800151e30 0x7f88000a7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:03.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.315+0000 7f8805a00640 1 -- 172.21.15.28:0/2316873760 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8800151e30 msgr2=0x7f88000a7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:03.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.315+0000 7f8805a00640 1 --2- 172.21.15.28:0/2316873760 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8800151e30 0x7f88000a7690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:03.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.315+0000 7f8805a00640 1 -- 172.21.15.28:0/2316873760 >> v1:172.21.15.81:6789/0 conn(0x7f880014b750 legacy=0x7f8800155300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:03.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.315+0000 7f8805a00640 1 -- 172.21.15.28:0/2316873760 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88000c2a70 con 0x7f88000a7bd0 2024-02-21T23:17:03.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.316+0000 7f8805a00640 1 --2- 172.21.15.28:0/2316873760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88000a7bd0 0x7f880006e8f0 secure :-1 s=READY pgs=2674 cs=0 l=1 rev1=1 crypto rx=0x7f87f8009e50 tx=0x7f87f802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fccf96f74e5812f server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:03.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.316+0000 7f87ffe00640 1 -- 172.21.15.28:0/2316873760 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f802c9e0 con 0x7f88000a7bd0 2024-02-21T23:17:03.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.316+0000 7f87ffe00640 1 -- 172.21.15.28:0/2316873760 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87f802cb80 con 0x7f88000a7bd0 2024-02-21T23:17:03.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.316+0000 7f87ffe00640 1 -- 172.21.15.28:0/2316873760 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f80343f0 con 0x7f88000a7bd0 2024-02-21T23:17:03.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.317+0000 7f8806400640 1 -- 172.21.15.28:0/2316873760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88000a7bd0 msgr2=0x7f880006e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:03.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.317+0000 7f8806400640 1 --2- 172.21.15.28:0/2316873760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88000a7bd0 0x7f880006e8f0 secure :-1 s=READY pgs=2674 cs=0 l=1 rev1=1 crypto rx=0x7f87f8009e50 tx=0x7f87f802be40 comp rx=0 tx=0).stop 2024-02-21T23:17:03.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.317+0000 7f8806400640 1 -- 172.21.15.28:0/2316873760 shutdown_connections 2024-02-21T23:17:03.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.317+0000 7f8806400640 1 --2- 172.21.15.28:0/2316873760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88000a7bd0 0x7f880006e8f0 unknown :-1 s=CLOSED pgs=2674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:03.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.317+0000 7f8806400640 1 --2- 172.21.15.28:0/2316873760 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8800151e30 0x7f88000a7690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:03.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.317+0000 7f8806400640 1 -- 172.21.15.28:0/2316873760 >> 172.21.15.28:0/2316873760 conn(0x7f880006b630 msgr2=0x7f88000c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:03.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.317+0000 7f8806400640 1 -- 172.21.15.28:0/2316873760 shutdown_connections 2024-02-21T23:17:03.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.317+0000 7f8806400640 1 -- 172.21.15.28:0/2316873760 wait complete. 2024-02-21T23:17:03.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.319+0000 7f8806400640 1 Processor -- start 2024-02-21T23:17:03.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.319+0000 7f8806400640 1 -- start start 2024-02-21T23:17:03.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.320+0000 7f8806400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88000a7bd0 0x7f88001432a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:03.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.320+0000 7f8806400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f880014b750 0x7f88000aab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:03.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.320+0000 7f8806400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8800151e30 0x7f88000c5900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:03.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.320+0000 7f8806400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88000c09c0 con 0x7f88000a7bd0 2024-02-21T23:17:03.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.320+0000 7f8806400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88001512e0 con 0x7f8800151e30 2024-02-21T23:17:03.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.320+0000 7f8806400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88000c0b40 con 0x7f880014b750 2024-02-21T23:17:03.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.320+0000 7f8805000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88000a7bd0 0x7f88001432a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:03.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.320+0000 7f8805000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88000a7bd0 0x7f88001432a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44292/0 (socket says 172.21.15.28:44292) 2024-02-21T23:17:03.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.320+0000 7f8805000640 1 -- 172.21.15.28:0/3183452737 learned_addr learned my addr 172.21.15.28:0/3183452737 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:03.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.320+0000 7f87f7e00640 1 --2- 172.21.15.28:0/3183452737 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f880014b750 0x7f88000aab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:03.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.320+0000 7f8805a00640 1 --2- 172.21.15.28:0/3183452737 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8800151e30 0x7f88000c5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:03.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.320+0000 7f8805000640 1 -- 172.21.15.28:0/3183452737 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f880014b750 msgr2=0x7f88000aab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:03.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.320+0000 7f8805000640 1 --2- 172.21.15.28:0/3183452737 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f880014b750 0x7f88000aab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:03.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.320+0000 7f8805000640 1 -- 172.21.15.28:0/3183452737 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8800151e30 msgr2=0x7f88000c5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:03.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.320+0000 7f8805000640 1 --2- 172.21.15.28:0/3183452737 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8800151e30 0x7f88000c5900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:03.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.321+0000 7f8805000640 1 -- 172.21.15.28:0/3183452737 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f880014e650 con 0x7f88000a7bd0 2024-02-21T23:17:03.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.321+0000 7f87f7e00640 1 --2- 172.21.15.28:0/3183452737 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f880014b750 0x7f88000aab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:17:03.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.321+0000 7f8805a00640 1 --2- 172.21.15.28:0/3183452737 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8800151e30 0x7f88000c5900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:17:03.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.321+0000 7f8805000640 1 --2- 172.21.15.28:0/3183452737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88000a7bd0 0x7f88001432a0 secure :-1 s=READY pgs=2675 cs=0 l=1 rev1=1 crypto rx=0x7f87f000c460 tx=0x7f87f000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:03.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.321+0000 7f87fd600640 1 -- 172.21.15.28:0/3183452737 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f0016020 con 0x7f88000a7bd0 2024-02-21T23:17:03.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.321+0000 7f87fd600640 1 -- 172.21.15.28:0/3183452737 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87f0012870 con 0x7f88000a7bd0 2024-02-21T23:17:03.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.321+0000 7f87fd600640 1 -- 172.21.15.28:0/3183452737 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f0012c60 con 0x7f88000a7bd0 2024-02-21T23:17:03.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.321+0000 7f8806400640 1 -- 172.21.15.28:0/3183452737 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88000aa160 con 0x7f88000a7bd0 2024-02-21T23:17:03.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.321+0000 7f8806400640 1 -- 172.21.15.28:0/3183452737 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88001437e0 con 0x7f88000a7bd0 2024-02-21T23:17:03.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.323+0000 7f8806400640 1 -- 172.21.15.28:0/3183452737 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f880014f150 con 0x7f88000a7bd0 2024-02-21T23:17:03.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.324+0000 7f87fd600640 1 -- 172.21.15.28:0/3183452737 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f87f0019020 con 0x7f88000a7bd0 2024-02-21T23:17:03.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.325+0000 7f87fd600640 1 --2- 172.21.15.28:0/3183452737 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f87dc06b4d0 0x7f87dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:03.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.325+0000 7f87fd600640 1 -- 172.21.15.28:0/3183452737 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f87f008a570 con 0x7f88000a7bd0 2024-02-21T23:17:03.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.325+0000 7f87f7e00640 1 --2- 172.21.15.28:0/3183452737 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f87dc06b4d0 0x7f87dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:03.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.326+0000 7f87f7e00640 1 --2- 172.21.15.28:0/3183452737 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f87dc06b4d0 0x7f87dc06d990 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f880013b3b0 tx=0x7f87e8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:03.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.330+0000 7f87fd600640 1 -- 172.21.15.28:0/3183452737 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f880014f150 con 0x7f88000a7bd0 2024-02-21T23:17:03.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.568+0000 7f8806400640 1 -- 172.21.15.28:0/3183452737 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f880004f290 con 0x7f88000a7bd0 2024-02-21T23:17:03.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.569+0000 7f87fd600640 1 -- 172.21.15.28:0/3183452737 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7f87f0020020 con 0x7f88000a7bd0 2024-02-21T23:17:03.558 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:17:03.558 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:17:03.561 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:17:03.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.571+0000 7f8806400640 1 -- 172.21.15.28:0/3183452737 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f87dc06b4d0 msgr2=0x7f87dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:03.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.571+0000 7f8806400640 1 --2- 172.21.15.28:0/3183452737 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f87dc06b4d0 0x7f87dc06d990 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f880013b3b0 tx=0x7f87e8008040 comp rx=0 tx=0).stop 2024-02-21T23:17:03.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.571+0000 7f8806400640 1 -- 172.21.15.28:0/3183452737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88000a7bd0 msgr2=0x7f88001432a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:03.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.571+0000 7f8806400640 1 --2- 172.21.15.28:0/3183452737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88000a7bd0 0x7f88001432a0 secure :-1 s=READY pgs=2675 cs=0 l=1 rev1=1 crypto rx=0x7f87f000c460 tx=0x7f87f000c920 comp rx=0 tx=0).stop 2024-02-21T23:17:03.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.571+0000 7f8806400640 1 -- 172.21.15.28:0/3183452737 shutdown_connections 2024-02-21T23:17:03.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.571+0000 7f8806400640 1 --2- 172.21.15.28:0/3183452737 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f87dc06b4d0 0x7f87dc06d990 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:03.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.571+0000 7f8806400640 1 --2- 172.21.15.28:0/3183452737 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f880014b750 0x7f88000aab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:03.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.571+0000 7f8806400640 1 --2- 172.21.15.28:0/3183452737 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8800151e30 0x7f88000c5900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:03.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.571+0000 7f8806400640 1 --2- 172.21.15.28:0/3183452737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88000a7bd0 0x7f88001432a0 unknown :-1 s=CLOSED pgs=2675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:03.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.571+0000 7f8806400640 1 -- 172.21.15.28:0/3183452737 >> 172.21.15.28:0/3183452737 conn(0x7f880006b630 msgr2=0x7f88000bf930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:03.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.571+0000 7f8806400640 1 -- 172.21.15.28:0/3183452737 shutdown_connections 2024-02-21T23:17:03.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:03.571+0000 7f8806400640 1 -- 172.21.15.28:0/3183452737 wait complete. 2024-02-21T23:17:03.985 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:17:03.985 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:17:04.987 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:17:05.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.230+0000 7f5744400640 1 Processor -- start 2024-02-21T23:17:05.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.230+0000 7f5744400640 1 -- start start 2024-02-21T23:17:05.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.232+0000 7f5744400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f573c072eb0 0x7f573c1591a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:05.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.232+0000 7f5744400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f573c0727a0 0x7f573c0c5770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:05.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.232+0000 7f5744400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f573c0c13b0 con 0x7f573c0c4830 2024-02-21T23:17:05.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.232+0000 7f5744400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f573c0a7370 con 0x7f573c0727a0 2024-02-21T23:17:05.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.232+0000 7f5744400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f573c155c50 con 0x7f573c072eb0 2024-02-21T23:17:05.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.232+0000 7f5741000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f573c072eb0 0x7f573c1591a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:05.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.232+0000 7f5742400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f573c0727a0 0x7f573c0c5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:05.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.232+0000 7f5742400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f573c0727a0 0x7f573c0c5770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.28:52932/0 (socket says 172.21.15.28:52932) 2024-02-21T23:17:05.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.232+0000 7f5742400640 1 -- 172.21.15.28:0/2220177883 learned_addr learned my addr 172.21.15.28:0/2220177883 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:05.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.233+0000 7f5741000640 1 -- 172.21.15.28:0/2220177883 >> v2:172.21.15.81:3300/0 conn(0x7f573c0727a0 msgr2=0x7f573c0c5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:05.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.233+0000 7f5741000640 1 --2- 172.21.15.28:0/2220177883 >> v2:172.21.15.81:3300/0 conn(0x7f573c0727a0 0x7f573c0c5770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:05.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.233+0000 7f5741000640 1 -- 172.21.15.28:0/2220177883 >> v1:172.21.15.81:6789/0 conn(0x7f573c0c4830 legacy=0x7f573c0c0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:05.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.233+0000 7f5741000640 1 -- 172.21.15.28:0/2220177883 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f573c155f30 con 0x7f573c072eb0 2024-02-21T23:17:05.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.233+0000 7f5742400640 1 --2- 172.21.15.28:0/2220177883 >> v2:172.21.15.81:3300/0 conn(0x7f573c0727a0 0x7f573c0c5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-21T23:17:05.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.234+0000 7f5741000640 1 --2- 172.21.15.28:0/2220177883 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f573c072eb0 0x7f573c1591a0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f57280097a0 tx=0x7f572802b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad4f004905fe2f10 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:05.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.235+0000 7f573be00640 1 -- 172.21.15.28:0/2220177883 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5728039070 con 0x7f573c072eb0 2024-02-21T23:17:05.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.235+0000 7f573be00640 1 -- 172.21.15.28:0/2220177883 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57280316f0 con 0x7f573c072eb0 2024-02-21T23:17:05.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.235+0000 7f573be00640 1 -- 172.21.15.28:0/2220177883 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5728031ac0 con 0x7f573c072eb0 2024-02-21T23:17:05.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.236+0000 7f5744400640 1 -- 172.21.15.28:0/2220177883 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f573c072eb0 msgr2=0x7f573c1591a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:05.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.236+0000 7f5744400640 1 --2- 172.21.15.28:0/2220177883 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f573c072eb0 0x7f573c1591a0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f57280097a0 tx=0x7f572802b790 comp rx=0 tx=0).stop 2024-02-21T23:17:05.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.236+0000 7f5744400640 1 -- 172.21.15.28:0/2220177883 shutdown_connections 2024-02-21T23:17:05.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.236+0000 7f5744400640 1 --2- 172.21.15.28:0/2220177883 >> v2:172.21.15.81:3300/0 conn(0x7f573c0727a0 0x7f573c0c5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:05.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.236+0000 7f5744400640 1 --2- 172.21.15.28:0/2220177883 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f573c072eb0 0x7f573c1591a0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:05.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.236+0000 7f5744400640 1 -- 172.21.15.28:0/2220177883 >> 172.21.15.28:0/2220177883 conn(0x7f573c06c5b0 msgr2=0x7f573c0c7870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:05.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.236+0000 7f5744400640 1 -- 172.21.15.28:0/2220177883 shutdown_connections 2024-02-21T23:17:05.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.236+0000 7f5744400640 1 -- 172.21.15.28:0/2220177883 wait complete. 2024-02-21T23:17:05.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.238+0000 7f5744400640 1 Processor -- start 2024-02-21T23:17:05.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.238+0000 7f5744400640 1 -- start start 2024-02-21T23:17:05.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.239+0000 7f5744400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f573c0727a0 0x7f573c149440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:05.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.239+0000 7f5744400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f573c072eb0 0x7f573c14b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:05.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.239+0000 7f5744400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f573c0c4830 0x7f573c14d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:05.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.239+0000 7f5744400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f573c0c4c10 con 0x7f573c072eb0 2024-02-21T23:17:05.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.239+0000 7f5744400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f573c1589a0 con 0x7f573c0c4830 2024-02-21T23:17:05.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.239+0000 7f5744400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f573c0c5cb0 con 0x7f573c0727a0 2024-02-21T23:17:05.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.239+0000 7f5741000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f573c072eb0 0x7f573c14b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:05.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.239+0000 7f5741a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f573c0727a0 0x7f573c149440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:05.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.239+0000 7f5741000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f573c072eb0 0x7f573c14b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35888/0 (socket says 172.21.15.28:35888) 2024-02-21T23:17:05.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.239+0000 7f5741000640 1 -- 172.21.15.28:0/746649616 learned_addr learned my addr 172.21.15.28:0/746649616 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:05.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.240+0000 7f5742400640 1 --2- 172.21.15.28:0/746649616 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f573c0c4830 0x7f573c14d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:05.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.240+0000 7f5741000640 1 -- 172.21.15.28:0/746649616 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f573c0727a0 msgr2=0x7f573c149440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:05.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.240+0000 7f5741000640 1 --2- 172.21.15.28:0/746649616 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f573c0727a0 0x7f573c149440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:05.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.240+0000 7f5741000640 1 -- 172.21.15.28:0/746649616 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f573c0c4830 msgr2=0x7f573c14d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:05.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.240+0000 7f5741000640 1 --2- 172.21.15.28:0/746649616 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f573c0c4830 0x7f573c14d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:05.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.240+0000 7f5741000640 1 -- 172.21.15.28:0/746649616 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f573c14b8e0 con 0x7f573c072eb0 2024-02-21T23:17:05.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.241+0000 7f5741000640 1 --2- 172.21.15.28:0/746649616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f573c072eb0 0x7f573c14b0a0 secure :-1 s=READY pgs=2676 cs=0 l=1 rev1=1 crypto rx=0x7f5728000c00 tx=0x7f57280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:05.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.241+0000 7f5739600640 1 -- 172.21.15.28:0/746649616 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5728043070 con 0x7f573c072eb0 2024-02-21T23:17:05.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.241+0000 7f5739600640 1 -- 172.21.15.28:0/746649616 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5728034480 con 0x7f573c072eb0 2024-02-21T23:17:05.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.241+0000 7f5739600640 1 -- 172.21.15.28:0/746649616 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5728039040 con 0x7f573c072eb0 2024-02-21T23:17:05.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.241+0000 7f5744400640 1 -- 172.21.15.28:0/746649616 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f573c14bbc0 con 0x7f573c072eb0 2024-02-21T23:17:05.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.241+0000 7f5744400640 1 -- 172.21.15.28:0/746649616 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f573c14b5e0 con 0x7f573c072eb0 2024-02-21T23:17:05.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.243+0000 7f5744400640 1 -- 172.21.15.28:0/746649616 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f573c068640 con 0x7f573c072eb0 2024-02-21T23:17:05.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.244+0000 7f5739600640 1 -- 172.21.15.28:0/746649616 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5728046070 con 0x7f573c072eb0 2024-02-21T23:17:05.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.245+0000 7f5739600640 1 --2- 172.21.15.28:0/746649616 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f571c06b500 0x7f571c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:05.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.245+0000 7f5739600640 1 -- 172.21.15.28:0/746649616 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f57280abe30 con 0x7f573c072eb0 2024-02-21T23:17:05.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.246+0000 7f5741a00640 1 --2- 172.21.15.28:0/746649616 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f571c06b500 0x7f571c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:05.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.247+0000 7f5741a00640 1 --2- 172.21.15.28:0/746649616 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f571c06b500 0x7f571c06d9c0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f573c154e50 tx=0x7f5730009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:05.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.250+0000 7f5739600640 1 -- 172.21.15.28:0/746649616 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f57280316c0 con 0x7f573c072eb0 2024-02-21T23:17:05.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.502+0000 7f5744400640 1 -- 172.21.15.28:0/746649616 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f573c0c0f50 con 0x7f573c072eb0 2024-02-21T23:17:05.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.503+0000 7f5739600640 1 -- 172.21.15.28:0/746649616 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7f572807a460 con 0x7f573c072eb0 2024-02-21T23:17:05.491 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:17:05.492 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:17:05.492 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:17:05.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.506+0000 7f5744400640 1 -- 172.21.15.28:0/746649616 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f571c06b500 msgr2=0x7f571c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:05.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.506+0000 7f5744400640 1 --2- 172.21.15.28:0/746649616 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f571c06b500 0x7f571c06d9c0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f573c154e50 tx=0x7f5730009040 comp rx=0 tx=0).stop 2024-02-21T23:17:05.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.506+0000 7f5744400640 1 -- 172.21.15.28:0/746649616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f573c072eb0 msgr2=0x7f573c14b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:05.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.506+0000 7f5744400640 1 --2- 172.21.15.28:0/746649616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f573c072eb0 0x7f573c14b0a0 secure :-1 s=READY pgs=2676 cs=0 l=1 rev1=1 crypto rx=0x7f5728000c00 tx=0x7f57280046c0 comp rx=0 tx=0).stop 2024-02-21T23:17:05.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.506+0000 7f5744400640 1 -- 172.21.15.28:0/746649616 shutdown_connections 2024-02-21T23:17:05.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.506+0000 7f5744400640 1 --2- 172.21.15.28:0/746649616 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f571c06b500 0x7f571c06d9c0 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:05.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.506+0000 7f5744400640 1 --2- 172.21.15.28:0/746649616 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f573c0c4830 0x7f573c14d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:05.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.506+0000 7f5744400640 1 --2- 172.21.15.28:0/746649616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f573c072eb0 0x7f573c14b0a0 unknown :-1 s=CLOSED pgs=2676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:05.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.506+0000 7f5744400640 1 --2- 172.21.15.28:0/746649616 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f573c0727a0 0x7f573c149440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:05.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.506+0000 7f5744400640 1 -- 172.21.15.28:0/746649616 >> 172.21.15.28:0/746649616 conn(0x7f573c06c5b0 msgr2=0x7f573c156340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:05.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.507+0000 7f5744400640 1 -- 172.21.15.28:0/746649616 shutdown_connections 2024-02-21T23:17:05.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:05.507+0000 7f5744400640 1 -- 172.21.15.28:0/746649616 wait complete. 2024-02-21T23:17:05.919 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:17:05.919 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:17:06.920 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:17:07.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.158+0000 7fca35400640 1 Processor -- start 2024-02-21T23:17:07.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.158+0000 7fca35400640 1 -- start start 2024-02-21T23:17:07.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.159+0000 7fca35400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fca300c49c0 0x7fca30156db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:07.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.159+0000 7fca35400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fca3006f590 0x7fca30072180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:07.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.159+0000 7fca35400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca300a8820 con 0x7fca3006ae70 2024-02-21T23:17:07.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.159+0000 7fca35400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca300a8de0 con 0x7fca300c49c0 2024-02-21T23:17:07.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.159+0000 7fca35400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca30156360 con 0x7fca3006f590 2024-02-21T23:17:07.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.160+0000 7fca2fe00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fca300c49c0 0x7fca30156db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:07.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.160+0000 7fca2f400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fca3006f590 0x7fca30072180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:07.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.160+0000 7fca2f400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fca3006f590 0x7fca30072180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3301/0 says I am v2:172.21.15.28:46048/0 (socket says 172.21.15.28:46048) 2024-02-21T23:17:07.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.160+0000 7fca2f400640 1 -- 172.21.15.28:0/348899349 learned_addr learned my addr 172.21.15.28:0/348899349 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:07.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.161+0000 7fca2f400640 1 -- 172.21.15.28:0/348899349 >> v2:172.21.15.81:3300/0 conn(0x7fca300c49c0 msgr2=0x7fca30156db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:07.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.161+0000 7fca2f400640 1 --2- 172.21.15.28:0/348899349 >> v2:172.21.15.81:3300/0 conn(0x7fca300c49c0 0x7fca30156db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:07.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.161+0000 7fca2f400640 1 -- 172.21.15.28:0/348899349 >> v1:172.21.15.81:6789/0 conn(0x7fca3006ae70 legacy=0x7fca3006b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:07.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.161+0000 7fca2f400640 1 -- 172.21.15.28:0/348899349 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca300a8b00 con 0x7fca3006f590 2024-02-21T23:17:07.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.162+0000 7fca2f400640 1 --2- 172.21.15.28:0/348899349 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fca3006f590 0x7fca30072180 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fca1c009870 tx=0x7fca1c02b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c6ff29941816e2a server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:07.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.163+0000 7fca2ea00640 1 -- 172.21.15.28:0/348899349 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca1c0317c0 con 0x7fca3006f590 2024-02-21T23:17:07.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.163+0000 7fca2ea00640 1 -- 172.21.15.28:0/348899349 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca1c037070 con 0x7fca3006f590 2024-02-21T23:17:07.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.163+0000 7fca2ea00640 1 -- 172.21.15.28:0/348899349 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca1c031b80 con 0x7fca3006f590 2024-02-21T23:17:07.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.163+0000 7fca35400640 1 -- 172.21.15.28:0/348899349 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fca3006f590 msgr2=0x7fca30072180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:07.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.163+0000 7fca35400640 1 --2- 172.21.15.28:0/348899349 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fca3006f590 0x7fca30072180 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fca1c009870 tx=0x7fca1c02b860 comp rx=0 tx=0).stop 2024-02-21T23:17:07.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.163+0000 7fca35400640 1 -- 172.21.15.28:0/348899349 shutdown_connections 2024-02-21T23:17:07.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.163+0000 7fca35400640 1 --2- 172.21.15.28:0/348899349 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fca3006f590 0x7fca30072180 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:07.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.163+0000 7fca35400640 1 --2- 172.21.15.28:0/348899349 >> v2:172.21.15.81:3300/0 conn(0x7fca300c49c0 0x7fca30156db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:07.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.163+0000 7fca35400640 1 -- 172.21.15.28:0/348899349 >> 172.21.15.28:0/348899349 conn(0x7fca3006e8c0 msgr2=0x7fca3006bf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:07.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.163+0000 7fca35400640 1 -- 172.21.15.28:0/348899349 shutdown_connections 2024-02-21T23:17:07.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.164+0000 7fca35400640 1 -- 172.21.15.28:0/348899349 wait complete. 2024-02-21T23:17:07.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.165+0000 7fca35400640 1 Processor -- start 2024-02-21T23:17:07.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.165+0000 7fca35400640 1 -- start start 2024-02-21T23:17:07.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.166+0000 7fca35400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca3006ae70 0x7fca300c7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:07.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.166+0000 7fca35400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fca3006f590 0x7fca30150fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:07.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.166+0000 7fca35400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fca300c49c0 0x7fca300c6450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:07.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.166+0000 7fca35400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca30156890 con 0x7fca3006ae70 2024-02-21T23:17:07.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.166+0000 7fca35400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca301572f0 con 0x7fca3006f590 2024-02-21T23:17:07.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.166+0000 7fca35400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca300726c0 con 0x7fca300c49c0 2024-02-21T23:17:07.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.166+0000 7fca2fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca3006ae70 0x7fca300c7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:07.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.166+0000 7fca2fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca3006ae70 0x7fca300c7bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35892/0 (socket says 172.21.15.28:35892) 2024-02-21T23:17:07.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.167+0000 7fca2fe00640 1 -- 172.21.15.28:0/1033262650 learned_addr learned my addr 172.21.15.28:0/1033262650 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:07.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.167+0000 7fca2fe00640 1 -- 172.21.15.28:0/1033262650 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fca300c49c0 msgr2=0x7fca300c6450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:07.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.167+0000 7fca2f400640 1 --2- 172.21.15.28:0/1033262650 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fca3006f590 0x7fca30150fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:07.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.167+0000 7fca34a00640 1 --2- 172.21.15.28:0/1033262650 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fca300c49c0 0x7fca300c6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:07.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.167+0000 7fca2fe00640 1 --2- 172.21.15.28:0/1033262650 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fca300c49c0 0x7fca300c6450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:07.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.167+0000 7fca2fe00640 1 -- 172.21.15.28:0/1033262650 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fca3006f590 msgr2=0x7fca30150fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:07.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.167+0000 7fca2fe00640 1 --2- 172.21.15.28:0/1033262650 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fca3006f590 0x7fca30150fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:07.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.167+0000 7fca2fe00640 1 -- 172.21.15.28:0/1033262650 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca30151500 con 0x7fca3006ae70 2024-02-21T23:17:07.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.167+0000 7fca2f400640 1 --2- 172.21.15.28:0/1033262650 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fca3006f590 0x7fca30150fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:17:07.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.167+0000 7fca34a00640 1 --2- 172.21.15.28:0/1033262650 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fca300c49c0 0x7fca300c6450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:17:07.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.167+0000 7fca2fe00640 1 --2- 172.21.15.28:0/1033262650 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca3006ae70 0x7fca300c7bd0 secure :-1 s=READY pgs=2677 cs=0 l=1 rev1=1 crypto rx=0x7fca2000e8a0 tx=0x7fca2000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:07.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.168+0000 7fca27e00640 1 -- 172.21.15.28:0/1033262650 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca20017070 con 0x7fca3006ae70 2024-02-21T23:17:07.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.168+0000 7fca27e00640 1 -- 172.21.15.28:0/1033262650 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca2000c940 con 0x7fca3006ae70 2024-02-21T23:17:07.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.168+0000 7fca27e00640 1 -- 172.21.15.28:0/1033262650 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca2000cd30 con 0x7fca3006ae70 2024-02-21T23:17:07.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.168+0000 7fca35400640 1 -- 172.21.15.28:0/1033262650 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca30150b90 con 0x7fca3006ae70 2024-02-21T23:17:07.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.168+0000 7fca35400640 1 -- 172.21.15.28:0/1033262650 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca301508b0 con 0x7fca3006ae70 2024-02-21T23:17:07.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.169+0000 7fca35400640 1 -- 172.21.15.28:0/1033262650 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca3004f290 con 0x7fca3006ae70 2024-02-21T23:17:07.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.170+0000 7fca27e00640 1 -- 172.21.15.28:0/1033262650 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fca2001a070 con 0x7fca3006ae70 2024-02-21T23:17:07.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.171+0000 7fca27e00640 1 --2- 172.21.15.28:0/1033262650 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fca1006b4d0 0x7fca1006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:07.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.171+0000 7fca27e00640 1 -- 172.21.15.28:0/1033262650 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fca2008a430 con 0x7fca3006ae70 2024-02-21T23:17:07.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.172+0000 7fca2f400640 1 --2- 172.21.15.28:0/1033262650 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fca1006b4d0 0x7fca1006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:07.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.173+0000 7fca2f400640 1 --2- 172.21.15.28:0/1033262650 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fca1006b4d0 0x7fca1006d990 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fca1c036000 tx=0x7fca1c008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:07.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.176+0000 7fca27e00640 1 -- 172.21.15.28:0/1033262650 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fca20058ef0 con 0x7fca3006ae70 2024-02-21T23:17:07.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.417+0000 7fca35400640 1 -- 172.21.15.28:0/1033262650 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fca3006f370 con 0x7fca3006ae70 2024-02-21T23:17:07.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.418+0000 7fca27e00640 1 -- 172.21.15.28:0/1033262650 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7fca200587f0 con 0x7fca3006ae70 2024-02-21T23:17:07.406 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:17:07.406 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:17:07.406 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:17:07.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.421+0000 7fca35400640 1 -- 172.21.15.28:0/1033262650 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fca1006b4d0 msgr2=0x7fca1006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:07.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.421+0000 7fca35400640 1 --2- 172.21.15.28:0/1033262650 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fca1006b4d0 0x7fca1006d990 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fca1c036000 tx=0x7fca1c008030 comp rx=0 tx=0).stop 2024-02-21T23:17:07.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.421+0000 7fca35400640 1 -- 172.21.15.28:0/1033262650 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca3006ae70 msgr2=0x7fca300c7bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:07.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.421+0000 7fca35400640 1 --2- 172.21.15.28:0/1033262650 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca3006ae70 0x7fca300c7bd0 secure :-1 s=READY pgs=2677 cs=0 l=1 rev1=1 crypto rx=0x7fca2000e8a0 tx=0x7fca2000ed60 comp rx=0 tx=0).stop 2024-02-21T23:17:07.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.421+0000 7fca35400640 1 -- 172.21.15.28:0/1033262650 shutdown_connections 2024-02-21T23:17:07.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.421+0000 7fca35400640 1 --2- 172.21.15.28:0/1033262650 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fca1006b4d0 0x7fca1006d990 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:07.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.421+0000 7fca35400640 1 --2- 172.21.15.28:0/1033262650 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fca300c49c0 0x7fca300c6450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:07.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.421+0000 7fca35400640 1 --2- 172.21.15.28:0/1033262650 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fca3006f590 0x7fca30150fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:07.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.421+0000 7fca35400640 1 --2- 172.21.15.28:0/1033262650 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca3006ae70 0x7fca300c7bd0 unknown :-1 s=CLOSED pgs=2677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:07.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.421+0000 7fca35400640 1 -- 172.21.15.28:0/1033262650 >> 172.21.15.28:0/1033262650 conn(0x7fca3006e8c0 msgr2=0x7fca3006bf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:07.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.421+0000 7fca35400640 1 -- 172.21.15.28:0/1033262650 shutdown_connections 2024-02-21T23:17:07.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:07.421+0000 7fca35400640 1 -- 172.21.15.28:0/1033262650 wait complete. 2024-02-21T23:17:07.833 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:17:07.834 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:17:08.834 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:17:09.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.068+0000 7fd186000640 1 Processor -- start 2024-02-21T23:17:09.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.068+0000 7fd186000640 1 -- start start 2024-02-21T23:17:09.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.070+0000 7fd186000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fd1800aff90 0x7fd1800c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:09.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.070+0000 7fd186000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd1800c5720 0x7fd18015f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:09.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.070+0000 7fd186000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1800c74c0 con 0x7fd18006dbf0 2024-02-21T23:17:09.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.070+0000 7fd186000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1800c7270 con 0x7fd1800c5720 2024-02-21T23:17:09.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.070+0000 7fd186000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd180158630 con 0x7fd1800aff90 2024-02-21T23:17:09.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.070+0000 7fd185600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd1800c5720 0x7fd18015f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:09.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.070+0000 7fd185600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd1800c5720 0x7fd18015f890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35894/0 (socket says 172.21.15.28:35894) 2024-02-21T23:17:09.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.070+0000 7fd185600640 1 -- 172.21.15.28:0/4183640691 learned_addr learned my addr 172.21.15.28:0/4183640691 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:09.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.071+0000 7fd184c00640 1 --2- 172.21.15.28:0/4183640691 >> v2:172.21.15.81:3300/0 conn(0x7fd1800aff90 0x7fd1800c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:09.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.071+0000 7fd185600640 1 -- 172.21.15.28:0/4183640691 >> v2:172.21.15.81:3300/0 conn(0x7fd1800aff90 msgr2=0x7fd1800c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:09.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.071+0000 7fd185600640 1 --2- 172.21.15.28:0/4183640691 >> v2:172.21.15.81:3300/0 conn(0x7fd1800aff90 0x7fd1800c51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:09.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.071+0000 7fd185600640 1 -- 172.21.15.28:0/4183640691 >> v1:172.21.15.81:6789/0 conn(0x7fd18006dbf0 legacy=0x7fd180158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:09.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.071+0000 7fd185600640 1 -- 172.21.15.28:0/4183640691 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1800c77a0 con 0x7fd1800c5720 2024-02-21T23:17:09.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.071+0000 7fd184c00640 1 --2- 172.21.15.28:0/4183640691 >> v2:172.21.15.81:3300/0 conn(0x7fd1800aff90 0x7fd1800c51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:17:09.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.072+0000 7fd185600640 1 --2- 172.21.15.28:0/4183640691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1800c5720 0x7fd18015f890 secure :-1 s=READY pgs=2678 cs=0 l=1 rev1=1 crypto rx=0x7fd1800c73f0 tx=0x7fd17802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75543a5504341c2a server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:09.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.072+0000 7fd17f400640 1 -- 172.21.15.28:0/4183640691 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd17802ca70 con 0x7fd1800c5720 2024-02-21T23:17:09.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.072+0000 7fd17f400640 1 -- 172.21.15.28:0/4183640691 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd17802cc10 con 0x7fd1800c5720 2024-02-21T23:17:09.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.072+0000 7fd17f400640 1 -- 172.21.15.28:0/4183640691 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1780343f0 con 0x7fd1800c5720 2024-02-21T23:17:09.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.073+0000 7fd186000640 1 -- 172.21.15.28:0/4183640691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1800c5720 msgr2=0x7fd18015f890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:09.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.073+0000 7fd186000640 1 --2- 172.21.15.28:0/4183640691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1800c5720 0x7fd18015f890 secure :-1 s=READY pgs=2678 cs=0 l=1 rev1=1 crypto rx=0x7fd1800c73f0 tx=0x7fd17802bed0 comp rx=0 tx=0).stop 2024-02-21T23:17:09.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.073+0000 7fd186000640 1 -- 172.21.15.28:0/4183640691 shutdown_connections 2024-02-21T23:17:09.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.073+0000 7fd186000640 1 --2- 172.21.15.28:0/4183640691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1800c5720 0x7fd18015f890 unknown :-1 s=CLOSED pgs=2678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:09.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.073+0000 7fd186000640 1 --2- 172.21.15.28:0/4183640691 >> v2:172.21.15.81:3300/0 conn(0x7fd1800aff90 0x7fd1800c51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:09.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.073+0000 7fd186000640 1 -- 172.21.15.28:0/4183640691 >> 172.21.15.28:0/4183640691 conn(0x7fd1800ac370 msgr2=0x7fd1800ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:09.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.073+0000 7fd186000640 1 -- 172.21.15.28:0/4183640691 shutdown_connections 2024-02-21T23:17:09.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.073+0000 7fd186000640 1 -- 172.21.15.28:0/4183640691 wait complete. 2024-02-21T23:17:09.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.075+0000 7fd186000640 1 Processor -- start 2024-02-21T23:17:09.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.075+0000 7fd186000640 1 -- start start 2024-02-21T23:17:09.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.076+0000 7fd186000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd18006dbf0 0x7fd1800c4320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:09.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.076+0000 7fd186000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd1800aff90 0x7fd180154040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:09.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.076+0000 7fd186000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1800c5720 0x7fd1800c2010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:09.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.076+0000 7fd186000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1800c8760 con 0x7fd1800c5720 2024-02-21T23:17:09.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.076+0000 7fd186000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1800be0c0 con 0x7fd18006dbf0 2024-02-21T23:17:09.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.076+0000 7fd186000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1800c88e0 con 0x7fd1800aff90 2024-02-21T23:17:09.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.076+0000 7fd185600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1800c5720 0x7fd1800c2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:09.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.076+0000 7fd185600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1800c5720 0x7fd1800c2010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35904/0 (socket says 172.21.15.28:35904) 2024-02-21T23:17:09.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.076+0000 7fd185600640 1 -- 172.21.15.28:0/2537512404 learned_addr learned my addr 172.21.15.28:0/2537512404 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:09.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.076+0000 7fd184c00640 1 --2- 172.21.15.28:0/2537512404 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd18006dbf0 0x7fd1800c4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:09.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.077+0000 7fd17fe00640 1 --2- 172.21.15.28:0/2537512404 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd1800aff90 0x7fd180154040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:09.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.077+0000 7fd185600640 1 -- 172.21.15.28:0/2537512404 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd1800aff90 msgr2=0x7fd180154040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:09.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.077+0000 7fd185600640 1 --2- 172.21.15.28:0/2537512404 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd1800aff90 0x7fd180154040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:09.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.077+0000 7fd185600640 1 -- 172.21.15.28:0/2537512404 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd18006dbf0 msgr2=0x7fd1800c4320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:09.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.077+0000 7fd185600640 1 --2- 172.21.15.28:0/2537512404 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd18006dbf0 0x7fd1800c4320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:09.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.077+0000 7fd185600640 1 -- 172.21.15.28:0/2537512404 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1800ad5b0 con 0x7fd1800c5720 2024-02-21T23:17:09.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.077+0000 7fd185600640 1 --2- 172.21.15.28:0/2537512404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1800c5720 0x7fd1800c2010 secure :-1 s=READY pgs=2679 cs=0 l=1 rev1=1 crypto rx=0x7fd178036040 tx=0x7fd178004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:09.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.077+0000 7fd17cc00640 1 -- 172.21.15.28:0/2537512404 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd178043020 con 0x7fd1800c5720 2024-02-21T23:17:09.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.077+0000 7fd17cc00640 1 -- 172.21.15.28:0/2537512404 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd17803e450 con 0x7fd1800c5720 2024-02-21T23:17:09.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.077+0000 7fd186000640 1 -- 172.21.15.28:0/2537512404 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1800ad890 con 0x7fd1800c5720 2024-02-21T23:17:09.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.078+0000 7fd17cc00640 1 -- 172.21.15.28:0/2537512404 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd17803e860 con 0x7fd1800c5720 2024-02-21T23:17:09.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.078+0000 7fd186000640 1 -- 172.21.15.28:0/2537512404 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1800ade50 con 0x7fd1800c5720 2024-02-21T23:17:09.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.079+0000 7fd186000640 1 -- 172.21.15.28:0/2537512404 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd18004f290 con 0x7fd1800c5720 2024-02-21T23:17:09.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.080+0000 7fd17cc00640 1 -- 172.21.15.28:0/2537512404 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd178046020 con 0x7fd1800c5720 2024-02-21T23:17:09.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.081+0000 7fd17cc00640 1 --2- 172.21.15.28:0/2537512404 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd16006b4d0 0x7fd16006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:09.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.081+0000 7fd17cc00640 1 -- 172.21.15.28:0/2537512404 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fd1780acc50 con 0x7fd1800c5720 2024-02-21T23:17:09.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.081+0000 7fd184c00640 1 --2- 172.21.15.28:0/2537512404 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd16006b4d0 0x7fd16006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:09.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.083+0000 7fd184c00640 1 --2- 172.21.15.28:0/2537512404 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd16006b4d0 0x7fd16006d990 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fd1800c5580 tx=0x7fd174008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:09.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.085+0000 7fd17cc00640 1 -- 172.21.15.28:0/2537512404 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fd1780b2020 con 0x7fd1800c5720 2024-02-21T23:17:09.224 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-21T23:17:09.230 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-21T23:17:09.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.400+0000 7fd186000640 1 -- 172.21.15.28:0/2537512404 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd1801559e0 con 0x7fd1800c5720 2024-02-21T23:17:09.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.401+0000 7fd17cc00640 1 -- 172.21.15.28:0/2537512404 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7fd17804d030 con 0x7fd1800c5720 2024-02-21T23:17:09.389 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:17:09.389 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:17:09.389 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:17:09.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.404+0000 7fd186000640 1 -- 172.21.15.28:0/2537512404 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd16006b4d0 msgr2=0x7fd16006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:09.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.404+0000 7fd186000640 1 --2- 172.21.15.28:0/2537512404 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd16006b4d0 0x7fd16006d990 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fd1800c5580 tx=0x7fd174008040 comp rx=0 tx=0).stop 2024-02-21T23:17:09.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.404+0000 7fd186000640 1 -- 172.21.15.28:0/2537512404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1800c5720 msgr2=0x7fd1800c2010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:09.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.404+0000 7fd186000640 1 --2- 172.21.15.28:0/2537512404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1800c5720 0x7fd1800c2010 secure :-1 s=READY pgs=2679 cs=0 l=1 rev1=1 crypto rx=0x7fd178036040 tx=0x7fd178004a40 comp rx=0 tx=0).stop 2024-02-21T23:17:09.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.404+0000 7fd186000640 1 -- 172.21.15.28:0/2537512404 shutdown_connections 2024-02-21T23:17:09.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.404+0000 7fd186000640 1 --2- 172.21.15.28:0/2537512404 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fd16006b4d0 0x7fd16006d990 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:09.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.404+0000 7fd186000640 1 --2- 172.21.15.28:0/2537512404 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fd1800aff90 0x7fd180154040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:09.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.404+0000 7fd186000640 1 --2- 172.21.15.28:0/2537512404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1800c5720 0x7fd1800c2010 unknown :-1 s=CLOSED pgs=2679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:09.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.404+0000 7fd186000640 1 --2- 172.21.15.28:0/2537512404 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd18006dbf0 0x7fd1800c4320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:09.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.404+0000 7fd186000640 1 -- 172.21.15.28:0/2537512404 >> 172.21.15.28:0/2537512404 conn(0x7fd1800ac370 msgr2=0x7fd1800ac750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:09.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.404+0000 7fd186000640 1 -- 172.21.15.28:0/2537512404 shutdown_connections 2024-02-21T23:17:09.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:09.404+0000 7fd186000640 1 -- 172.21.15.28:0/2537512404 wait complete. 2024-02-21T23:17:09.814 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:17:09.814 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:17:10.815 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:17:11.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:11.081+0000 7fb2eb200640 1 Processor -- start 2024-02-21T23:17:11.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:11.082+0000 7fb2eb200640 1 -- start start 2024-02-21T23:17:11.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:11.083+0000 7fb2eb200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb2e4152500 0x7fb2e41558c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:11.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:11.083+0000 7fb2eb200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2e41528e0 con 0x7fb2e406bbf0 2024-02-21T23:17:11.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:11.083+0000 7fb2eb200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2e40c86e0 con 0x7fb2e40ca050 2024-02-21T23:17:11.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:11.083+0000 7fb2eb200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2e40c8980 con 0x7fb2e4152500 2024-02-21T23:17:11.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:11.084+0000 7fb2e9e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb2e4152500 0x7fb2e41558c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:11.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:11.084+0000 7fb2ea800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb2e406bbf0 0x7fb2e406bff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59802/0 (socket says 172.21.15.28:59802) 2024-02-21T23:17:11.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:11.084+0000 7fb2ea800640 1 -- 172.21.15.28:0/3412543190 learned_addr learned my addr 172.21.15.28:0/3412543190 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:17:11.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:11.084+0000 7fb2e9e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb2e4152500 0x7fb2e41558c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35912/0 (socket says 172.21.15.28:35912) 2024-02-21T23:17:11.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:11.084+0000 7fb2e9e00640 1 -- 172.21.15.28:0/3412543190 >> v1:172.21.15.81:6789/0 conn(0x7fb2e40ca050 legacy=0x7fb2e40bf6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:11.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:11.084+0000 7fb2e9e00640 1 -- 172.21.15.28:0/3412543190 >> v1:172.21.15.28:6789/0 conn(0x7fb2e406bbf0 legacy=0x7fb2e406bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:11.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:11.084+0000 7fb2e9e00640 1 -- 172.21.15.28:0/3412543190 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2e40c8c60 con 0x7fb2e4152500 2024-02-21T23:17:11.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:11.084+0000 7fb2e8a00640 1 -- 172.21.15.28:0/3412543190 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1277464956 0 0) 0x7fb2e41528e0 con 0x7fb2e406bbf0 2024-02-21T23:17:11.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:11.085+0000 7fb2e9e00640 1 --2- 172.21.15.28:0/3412543190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2e4152500 0x7fb2e41558c0 secure :-1 s=READY pgs=2681 cs=0 l=1 rev1=1 crypto rx=0x7fb2d80097a0 tx=0x7fb2d802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d5d6dabcc2324ae server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:11.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:11.086+0000 7fb2e8a00640 1 -- 172.21.15.28:0/3412543190 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d80315a0 con 0x7fb2e4152500 2024-02-21T23:17:11.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:11.086+0000 7fb2e8a00640 1 -- 172.21.15.28:0/3412543190 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2d8037070 con 0x7fb2e4152500 2024-02-21T23:17:11.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:11.086+0000 7fb2e8a00640 1 -- 172.21.15.28:0/3412543190 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d8031960 con 0x7fb2e4152500 2024-02-21T23:17:11.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:11.086+0000 7fb2eb200640 1 -- 172.21.15.28:0/3412543190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2e4152500 msgr2=0x7fb2e41558c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:11.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:11.086+0000 7fb2eb200640 1 --2- 172.21.15.28:0/3412543190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2e4152500 0x7fb2e41558c0 secure :-1 s=READY pgs=2681 cs=0 l=1 rev1=1 crypto rx=0x7fb2d80097a0 tx=0x7fb2d802b840 comp rx=0 tx=0).stop 2024-02-21T23:17:11.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:11.086+0000 7fb2eb200640 1 -- 172.21.15.28:0/3412543190 shutdown_connections 2024-02-21T23:17:11.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:11.086+0000 7fb2eb200640 1 --2- 172.21.15.28:0/3412543190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2e4152500 0x7fb2e41558c0 unknown :-1 s=CLOSED pgs=2681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:11.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:11.086+0000 7fb2eb200640 1 -- 172.21.15.28:0/3412543190 >> 172.21.15.28:0/3412543190 conn(0x7fb2e40ac450 msgr2=0x7fb2e40ac860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:11.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:11.086+0000 7fb2eb200640 1 -- 172.21.15.28:0/3412543190 shutdown_connections 2024-02-21T23:17:11.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:11.087+0000 7fb2eb200640 1 -- 172.21.15.28:0/3412543190 wait complete. 2024-02-21T23:17:11.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:11.089+0000 7fb2eb200640 1 Processor -- start 2024-02-21T23:17:11.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:11.089+0000 7fb2eb200640 1 -- start start 2024-02-21T23:17:11.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:11.091+0000 7fb2eb200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2e406bbf0 0x7fb2e41df610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:11.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:11.091+0000 7fb2eb200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2e40ca050 0x7fb2e414ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:11.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:11.091+0000 7fb2eb200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb2e4152500 0x7fb2e414f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:11.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:11.091+0000 7fb2eb200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e40bfdd0 con 0x7fb2e40ca050 2024-02-21T23:17:11.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:11.091+0000 7fb2eb200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e40c9220 con 0x7fb2e406bbf0 2024-02-21T23:17:11.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:11.091+0000 7fb2eb200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e40bff50 con 0x7fb2e4152500 2024-02-21T23:17:11.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:11.091+0000 7fb2e9400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2e40ca050 0x7fb2e414ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:11.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:11.091+0000 7fb2e9400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2e40ca050 0x7fb2e414ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35924/0 (socket says 172.21.15.28:35924) 2024-02-21T23:17:11.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:11.091+0000 7fb2e9400640 1 -- 172.21.15.28:0/537593886 learned_addr learned my addr 172.21.15.28:0/537593886 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:11.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:11.092+0000 7fb2ea800640 1 --2- 172.21.15.28:0/537593886 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb2e4152500 0x7fb2e414f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:11.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:11.092+0000 7fb2e9e00640 1 --2- 172.21.15.28:0/537593886 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2e406bbf0 0x7fb2e41df610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:11.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:11.092+0000 7fb2e9400640 1 -- 172.21.15.28:0/537593886 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb2e4152500 msgr2=0x7fb2e414f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:11.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:11.092+0000 7fb2e9400640 1 --2- 172.21.15.28:0/537593886 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb2e4152500 0x7fb2e414f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:11.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:11.092+0000 7fb2e9400640 1 -- 172.21.15.28:0/537593886 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2e406bbf0 msgr2=0x7fb2e41df610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:11.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:11.092+0000 7fb2e9400640 1 --2- 172.21.15.28:0/537593886 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2e406bbf0 0x7fb2e41df610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:11.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:11.092+0000 7fb2e9400640 1 -- 172.21.15.28:0/537593886 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2e40b0950 con 0x7fb2e40ca050 2024-02-21T23:17:11.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:11.092+0000 7fb2e9400640 1 --2- 172.21.15.28:0/537593886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2e40ca050 0x7fb2e414ef30 secure :-1 s=READY pgs=2682 cs=0 l=1 rev1=1 crypto rx=0x7fb2d400b3f0 tx=0x7fb2d400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:11.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:11.092+0000 7fb2de000640 1 -- 172.21.15.28:0/537593886 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d4015020 con 0x7fb2e40ca050 2024-02-21T23:17:11.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:11.092+0000 7fb2de000640 1 -- 172.21.15.28:0/537593886 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2d4011830 con 0x7fb2e40ca050 2024-02-21T23:17:11.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:11.092+0000 7fb2de000640 1 -- 172.21.15.28:0/537593886 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d4011c20 con 0x7fb2e40ca050 2024-02-21T23:17:11.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:11.092+0000 7fb2eb200640 1 -- 172.21.15.28:0/537593886 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2e41ef000 con 0x7fb2e40ca050 2024-02-21T23:17:11.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:11.093+0000 7fb2eb200640 1 -- 172.21.15.28:0/537593886 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2e4148830 con 0x7fb2e40ca050 2024-02-21T23:17:11.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:11.094+0000 7fb2eb200640 1 -- 172.21.15.28:0/537593886 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2e41def00 con 0x7fb2e40ca050 2024-02-21T23:17:11.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:11.097+0000 7fb2de000640 1 -- 172.21.15.28:0/537593886 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb2d4018020 con 0x7fb2e40ca050 2024-02-21T23:17:11.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:11.097+0000 7fb2de000640 1 --2- 172.21.15.28:0/537593886 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb2c806b4d0 0x7fb2c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:11.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:11.097+0000 7fb2de000640 1 -- 172.21.15.28:0/537593886 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fb2d408a160 con 0x7fb2e40ca050 2024-02-21T23:17:11.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:11.098+0000 7fb2e9e00640 1 --2- 172.21.15.28:0/537593886 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb2c806b4d0 0x7fb2c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:11.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:11.099+0000 7fb2e9e00640 1 --2- 172.21.15.28:0/537593886 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb2c806b4d0 0x7fb2c806d990 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7fb2d8036000 tx=0x7fb2d803d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:11.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:11.103+0000 7fb2de000640 1 -- 172.21.15.28:0/537593886 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb2d401f030 con 0x7fb2e40ca050 2024-02-21T23:17:11.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:11.339+0000 7fb2eb200640 1 -- 172.21.15.28:0/537593886 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb2e404f290 con 0x7fb2e40ca050 2024-02-21T23:17:11.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:11.340+0000 7fb2de000640 1 -- 172.21.15.28:0/537593886 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7fb2d40172c0 con 0x7fb2e40ca050 2024-02-21T23:17:11.329 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:17:11.329 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:17:11.329 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:17:11.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:11.342+0000 7fb2eb200640 1 -- 172.21.15.28:0/537593886 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb2c806b4d0 msgr2=0x7fb2c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:11.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:11.342+0000 7fb2eb200640 1 --2- 172.21.15.28:0/537593886 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb2c806b4d0 0x7fb2c806d990 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7fb2d8036000 tx=0x7fb2d803d040 comp rx=0 tx=0).stop 2024-02-21T23:17:11.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:11.342+0000 7fb2eb200640 1 -- 172.21.15.28:0/537593886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2e40ca050 msgr2=0x7fb2e414ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:11.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:11.342+0000 7fb2eb200640 1 --2- 172.21.15.28:0/537593886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2e40ca050 0x7fb2e414ef30 secure :-1 s=READY pgs=2682 cs=0 l=1 rev1=1 crypto rx=0x7fb2d400b3f0 tx=0x7fb2d400b8b0 comp rx=0 tx=0).stop 2024-02-21T23:17:11.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:11.343+0000 7fb2eb200640 1 -- 172.21.15.28:0/537593886 shutdown_connections 2024-02-21T23:17:11.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:11.343+0000 7fb2eb200640 1 --2- 172.21.15.28:0/537593886 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb2c806b4d0 0x7fb2c806d990 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:11.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:11.343+0000 7fb2eb200640 1 --2- 172.21.15.28:0/537593886 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb2e4152500 0x7fb2e414f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:11.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:11.343+0000 7fb2eb200640 1 --2- 172.21.15.28:0/537593886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2e40ca050 0x7fb2e414ef30 unknown :-1 s=CLOSED pgs=2682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:11.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:11.343+0000 7fb2eb200640 1 --2- 172.21.15.28:0/537593886 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb2e406bbf0 0x7fb2e41df610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:11.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:11.343+0000 7fb2eb200640 1 -- 172.21.15.28:0/537593886 >> 172.21.15.28:0/537593886 conn(0x7fb2e40ac450 msgr2=0x7fb2e40abb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:11.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:11.343+0000 7fb2eb200640 1 -- 172.21.15.28:0/537593886 shutdown_connections 2024-02-21T23:17:11.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:11.343+0000 7fb2eb200640 1 -- 172.21.15.28:0/537593886 wait complete. 2024-02-21T23:17:11.751 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:17:11.752 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:17:12.753 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:17:12.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:12.992+0000 7f56f3e00640 1 Processor -- start 2024-02-21T23:17:12.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:12.992+0000 7f56f3e00640 1 -- start start 2024-02-21T23:17:12.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:12.994+0000 7f56f3e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f56f40aff90 0x7f56f40c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:12.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:12.994+0000 7f56f3e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f56f406dbf0 0x7f56f4158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:12.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:12.994+0000 7f56f3e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f56f40c5720 0x7f56f4159000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:12.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:12.994+0000 7f56f3e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56f40c7270 con 0x7f56f40aff90 2024-02-21T23:17:12.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:12.994+0000 7f56f3e00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56f4158630 con 0x7f56f406dbf0 2024-02-21T23:17:12.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:12.994+0000 7f56f3e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56f40c74c0 con 0x7f56f40c5720 2024-02-21T23:17:12.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:12.994+0000 7f56f2a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f56f40aff90 0x7f56f40c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:12.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:12.994+0000 7f56f2a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f56f40aff90 0x7f56f40c51e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35940/0 (socket says 172.21.15.28:35940) 2024-02-21T23:17:12.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:12.994+0000 7f56f2a00640 1 -- 172.21.15.28:0/196260701 learned_addr learned my addr 172.21.15.28:0/196260701 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:12.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:12.995+0000 7f56f2a00640 1 -- 172.21.15.28:0/196260701 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f56f40c5720 msgr2=0x7f56f4159000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:12.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:12.995+0000 7f56f2a00640 1 --2- 172.21.15.28:0/196260701 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f56f40c5720 0x7f56f4159000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:12.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:12.995+0000 7f56f2a00640 1 -- 172.21.15.28:0/196260701 >> v2:172.21.15.81:3300/0 conn(0x7f56f406dbf0 msgr2=0x7f56f4158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:12.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:12.995+0000 7f56f2a00640 1 --2- 172.21.15.28:0/196260701 >> v2:172.21.15.81:3300/0 conn(0x7f56f406dbf0 0x7f56f4158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:12.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:12.995+0000 7f56f2a00640 1 -- 172.21.15.28:0/196260701 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56f40c77a0 con 0x7f56f40aff90 2024-02-21T23:17:12.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:12.996+0000 7f56f2a00640 1 --2- 172.21.15.28:0/196260701 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56f40aff90 0x7f56f40c51e0 secure :-1 s=READY pgs=2683 cs=0 l=1 rev1=1 crypto rx=0x7f56f40cb8c0 tx=0x7f56e402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6ac634cc4a887f3 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:12.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:12.996+0000 7f56f2000640 1 -- 172.21.15.28:0/196260701 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56e4035020 con 0x7f56f40aff90 2024-02-21T23:17:12.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:12.996+0000 7f56f2000640 1 -- 172.21.15.28:0/196260701 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56e4031870 con 0x7f56f40aff90 2024-02-21T23:17:12.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:12.996+0000 7f56f2000640 1 -- 172.21.15.28:0/196260701 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56e4031c40 con 0x7f56f40aff90 2024-02-21T23:17:12.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:12.997+0000 7f56f3e00640 1 -- 172.21.15.28:0/196260701 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56f40aff90 msgr2=0x7f56f40c51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:12.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:12.997+0000 7f56f3e00640 1 --2- 172.21.15.28:0/196260701 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56f40aff90 0x7f56f40c51e0 secure :-1 s=READY pgs=2683 cs=0 l=1 rev1=1 crypto rx=0x7f56f40cb8c0 tx=0x7f56e402b910 comp rx=0 tx=0).stop 2024-02-21T23:17:12.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:12.997+0000 7f56f3e00640 1 -- 172.21.15.28:0/196260701 shutdown_connections 2024-02-21T23:17:12.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:12.997+0000 7f56f3e00640 1 --2- 172.21.15.28:0/196260701 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f56f40c5720 0x7f56f4159000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:12.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:12.997+0000 7f56f3e00640 1 --2- 172.21.15.28:0/196260701 >> v2:172.21.15.81:3300/0 conn(0x7f56f406dbf0 0x7f56f4158ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:12.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:12.997+0000 7f56f3e00640 1 --2- 172.21.15.28:0/196260701 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56f40aff90 0x7f56f40c51e0 unknown :-1 s=CLOSED pgs=2683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:12.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:12.997+0000 7f56f3e00640 1 -- 172.21.15.28:0/196260701 >> 172.21.15.28:0/196260701 conn(0x7f56f40ac370 msgr2=0x7f56f40ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:12.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:12.997+0000 7f56f3e00640 1 -- 172.21.15.28:0/196260701 shutdown_connections 2024-02-21T23:17:12.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:12.997+0000 7f56f3e00640 1 -- 172.21.15.28:0/196260701 wait complete. 2024-02-21T23:17:12.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:12.999+0000 7f56f3e00640 1 Processor -- start 2024-02-21T23:17:12.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:12.999+0000 7f56f3e00640 1 -- start start 2024-02-21T23:17:12.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:13.000+0000 7f56f3e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56f406dbf0 0x7f56f40c4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:12.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:13.000+0000 7f56f3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56f40aff90 0x7f56f40c2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:12.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:13.000+0000 7f56f3e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f56f40c5720 0x7f56f41ee200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:12.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:13.000+0000 7f56f3e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56f40c8760 con 0x7f56f40aff90 2024-02-21T23:17:12.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:13.000+0000 7f56f3e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56f40be0c0 con 0x7f56f406dbf0 2024-02-21T23:17:12.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:13.000+0000 7f56f3e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56f40c88e0 con 0x7f56f40c5720 2024-02-21T23:17:12.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:13.000+0000 7f56ea000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56f40aff90 0x7f56f40c2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:12.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:13.001+0000 7f56ea000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56f40aff90 0x7f56f40c2310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35956/0 (socket says 172.21.15.28:35956) 2024-02-21T23:17:12.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:13.001+0000 7f56ea000640 1 -- 172.21.15.28:0/2032619287 learned_addr learned my addr 172.21.15.28:0/2032619287 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:12.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:13.001+0000 7f56f2a00640 1 --2- 172.21.15.28:0/2032619287 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56f406dbf0 0x7f56f40c4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:12.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:13.001+0000 7f56f3400640 1 --2- 172.21.15.28:0/2032619287 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f56f40c5720 0x7f56f41ee200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:12.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:13.001+0000 7f56ea000640 1 -- 172.21.15.28:0/2032619287 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f56f40c5720 msgr2=0x7f56f41ee200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:12.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:13.001+0000 7f56ea000640 1 --2- 172.21.15.28:0/2032619287 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f56f40c5720 0x7f56f41ee200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:12.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:13.001+0000 7f56ea000640 1 -- 172.21.15.28:0/2032619287 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56f406dbf0 msgr2=0x7f56f40c4090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:12.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:13.001+0000 7f56ea000640 1 --2- 172.21.15.28:0/2032619287 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56f406dbf0 0x7f56f40c4090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:12.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:13.001+0000 7f56ea000640 1 -- 172.21.15.28:0/2032619287 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56f4151c00 con 0x7f56f40aff90 2024-02-21T23:17:12.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:13.001+0000 7f56ea000640 1 --2- 172.21.15.28:0/2032619287 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56f40aff90 0x7f56f40c2310 secure :-1 s=READY pgs=2684 cs=0 l=1 rev1=1 crypto rx=0x7f56dc00b450 tx=0x7f56dc00b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:12.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:13.002+0000 7f56ebe00640 1 -- 172.21.15.28:0/2032619287 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56dc015020 con 0x7f56f40aff90 2024-02-21T23:17:12.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:13.002+0000 7f56ebe00640 1 -- 172.21.15.28:0/2032619287 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56dc011890 con 0x7f56f40aff90 2024-02-21T23:17:12.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:13.002+0000 7f56f3e00640 1 -- 172.21.15.28:0/2032619287 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56f41eec80 con 0x7f56f40aff90 2024-02-21T23:17:12.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:13.002+0000 7f56ebe00640 1 -- 172.21.15.28:0/2032619287 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56dc011c80 con 0x7f56f40aff90 2024-02-21T23:17:12.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:13.002+0000 7f56f3e00640 1 -- 172.21.15.28:0/2032619287 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56f40be240 con 0x7f56f40aff90 2024-02-21T23:17:12.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:13.003+0000 7f56f3e00640 1 -- 172.21.15.28:0/2032619287 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56f404f290 con 0x7f56f40aff90 2024-02-21T23:17:12.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:13.005+0000 7f56ebe00640 1 -- 172.21.15.28:0/2032619287 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f56dc018020 con 0x7f56f40aff90 2024-02-21T23:17:12.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:13.006+0000 7f56ebe00640 1 --2- 172.21.15.28:0/2032619287 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f56d006b4d0 0x7f56d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:12.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:13.006+0000 7f56ebe00640 1 -- 172.21.15.28:0/2032619287 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f56dc08a600 con 0x7f56f40aff90 2024-02-21T23:17:12.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:13.006+0000 7f56f2a00640 1 --2- 172.21.15.28:0/2032619287 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f56d006b4d0 0x7f56d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:12.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:13.008+0000 7f56f2a00640 1 --2- 172.21.15.28:0/2032619287 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f56d006b4d0 0x7f56d006d990 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f56e4036000 tx=0x7f56e403c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:12.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:13.011+0000 7f56ebe00640 1 -- 172.21.15.28:0/2032619287 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f56dc01f020 con 0x7f56f40aff90 2024-02-21T23:17:13.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:13.242+0000 7f56f3e00640 1 -- 172.21.15.28:0/2032619287 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f56f406d9d0 con 0x7f56f40aff90 2024-02-21T23:17:13.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:13.243+0000 7f56ebe00640 1 -- 172.21.15.28:0/2032619287 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7f56dc00d040 con 0x7f56f40aff90 2024-02-21T23:17:13.230 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:17:13.230 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:17:13.231 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:17:13.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:13.245+0000 7f56f3e00640 1 -- 172.21.15.28:0/2032619287 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f56d006b4d0 msgr2=0x7f56d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:13.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:13.246+0000 7f56f3e00640 1 --2- 172.21.15.28:0/2032619287 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f56d006b4d0 0x7f56d006d990 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f56e4036000 tx=0x7f56e403c040 comp rx=0 tx=0).stop 2024-02-21T23:17:13.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:13.246+0000 7f56f3e00640 1 -- 172.21.15.28:0/2032619287 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56f40aff90 msgr2=0x7f56f40c2310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:13.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:13.246+0000 7f56f3e00640 1 --2- 172.21.15.28:0/2032619287 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56f40aff90 0x7f56f40c2310 secure :-1 s=READY pgs=2684 cs=0 l=1 rev1=1 crypto rx=0x7f56dc00b450 tx=0x7f56dc00b910 comp rx=0 tx=0).stop 2024-02-21T23:17:13.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:13.246+0000 7f56f3e00640 1 -- 172.21.15.28:0/2032619287 shutdown_connections 2024-02-21T23:17:13.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:13.246+0000 7f56f3e00640 1 --2- 172.21.15.28:0/2032619287 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f56d006b4d0 0x7f56d006d990 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:13.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:13.246+0000 7f56f3e00640 1 --2- 172.21.15.28:0/2032619287 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f56f40c5720 0x7f56f41ee200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:13.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:13.246+0000 7f56f3e00640 1 --2- 172.21.15.28:0/2032619287 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56f40aff90 0x7f56f40c2310 unknown :-1 s=CLOSED pgs=2684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:13.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:13.246+0000 7f56f3e00640 1 --2- 172.21.15.28:0/2032619287 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56f406dbf0 0x7f56f40c4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:13.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:13.246+0000 7f56f3e00640 1 -- 172.21.15.28:0/2032619287 >> 172.21.15.28:0/2032619287 conn(0x7f56f40ac370 msgr2=0x7f56f40ac750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:13.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:13.246+0000 7f56f3e00640 1 -- 172.21.15.28:0/2032619287 shutdown_connections 2024-02-21T23:17:13.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:13.246+0000 7f56f3e00640 1 -- 172.21.15.28:0/2032619287 wait complete. 2024-02-21T23:17:13.655 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:17:13.655 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:17:14.656 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:17:14.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.892+0000 7fc21ae00640 1 Processor -- start 2024-02-21T23:17:14.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.892+0000 7fc21ae00640 1 -- start start 2024-02-21T23:17:14.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.894+0000 7fc21ae00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fc21413e620 0x7fc214159070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:14.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.894+0000 7fc21ae00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2141551b0 con 0x7fc2140bf990 2024-02-21T23:17:14.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.894+0000 7fc21ae00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2141595b0 con 0x7fc2140c6460 2024-02-21T23:17:14.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.894+0000 7fc21ae00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2140c71d0 con 0x7fc21413e620 2024-02-21T23:17:14.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.894+0000 7fc218e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fc2140bf990 0x7fc2140bfdc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:51998/0 (socket says 172.21.15.28:51998) 2024-02-21T23:17:14.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.894+0000 7fc218e00640 1 -- 172.21.15.28:0/3886517331 learned_addr learned my addr 172.21.15.28:0/3886517331 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:17:14.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.895+0000 7fc213e00640 1 --2- 172.21.15.28:0/3886517331 >> v2:172.21.15.81:3300/0 conn(0x7fc21413e620 0x7fc214159070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:14.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.895+0000 7fc212a00640 1 -- 172.21.15.28:0/3886517331 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2974423338 0 0) 0x7fc2141551b0 con 0x7fc2140bf990 2024-02-21T23:17:14.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.895+0000 7fc212a00640 1 -- 172.21.15.28:0/3886517331 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1f8003520 con 0x7fc2140bf990 2024-02-21T23:17:14.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.895+0000 7fc212a00640 1 -- 172.21.15.28:0/3886517331 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1503938438 0 0) 0x7fc1f8003520 con 0x7fc2140bf990 2024-02-21T23:17:14.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.895+0000 7fc212a00640 1 -- 172.21.15.28:0/3886517331 >> v2:172.21.15.81:3300/0 conn(0x7fc21413e620 msgr2=0x7fc214159070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:14.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.895+0000 7fc212a00640 1 --2- 172.21.15.28:0/3886517331 >> v2:172.21.15.81:3300/0 conn(0x7fc21413e620 0x7fc214159070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:14.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.895+0000 7fc212a00640 1 -- 172.21.15.28:0/3886517331 >> v1:172.21.15.81:6789/0 conn(0x7fc2140c6460 legacy=0x7fc2140ca8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:14.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.895+0000 7fc212a00640 1 -- 172.21.15.28:0/3886517331 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc214155420 con 0x7fc2140bf990 2024-02-21T23:17:14.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.895+0000 7fc212a00640 1 -- 172.21.15.28:0/3886517331 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fc204001670 con 0x7fc2140bf990 2024-02-21T23:17:14.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.895+0000 7fc212a00640 1 -- 172.21.15.28:0/3886517331 >> v1:172.21.15.28:6789/0 conn(0x7fc2140bf990 legacy=0x7fc2140bfdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:14.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.895+0000 7fc212a00640 1 --2- 172.21.15.28:0/3886517331 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc1f80039b0 0x7fc1f8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:14.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.896+0000 7fc212a00640 1 --2- 172.21.15.28:0/3886517331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1f8004800 0x7fc1f8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:14.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.896+0000 7fc212a00640 1 --2- 172.21.15.28:0/3886517331 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc1f80051c0 0x7fc1f800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:14.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.896+0000 7fc212a00640 1 -- 172.21.15.28:0/3886517331 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2140c71d0 con 0x7fc1f8004800 2024-02-21T23:17:14.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.896+0000 7fc212a00640 1 -- 172.21.15.28:0/3886517331 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f800c160 con 0x7fc1f80051c0 2024-02-21T23:17:14.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.896+0000 7fc212a00640 1 -- 172.21.15.28:0/3886517331 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f800c360 con 0x7fc1f80039b0 2024-02-21T23:17:14.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.896+0000 7fc213400640 1 --2- 172.21.15.28:0/3886517331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1f8004800 0x7fc1f8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:14.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.896+0000 7fc213e00640 1 --2- 172.21.15.28:0/3886517331 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc1f80039b0 0x7fc1f8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:14.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.896+0000 7fc218e00640 1 --2- 172.21.15.28:0/3886517331 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc1f80051c0 0x7fc1f800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:14.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.896+0000 7fc213400640 1 -- 172.21.15.28:0/3886517331 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc1f80039b0 msgr2=0x7fc1f8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:14.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.896+0000 7fc213400640 1 --2- 172.21.15.28:0/3886517331 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc1f80039b0 0x7fc1f8003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:14.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.896+0000 7fc213400640 1 -- 172.21.15.28:0/3886517331 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc1f80051c0 msgr2=0x7fc1f800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:14.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.896+0000 7fc213400640 1 --2- 172.21.15.28:0/3886517331 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc1f80051c0 0x7fc1f800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:14.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.896+0000 7fc213400640 1 -- 172.21.15.28:0/3886517331 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1f8003520 con 0x7fc1f8004800 2024-02-21T23:17:14.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.896+0000 7fc218e00640 1 --2- 172.21.15.28:0/3886517331 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc1f80051c0 0x7fc1f800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:17:14.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.896+0000 7fc218e00640 1 -- 172.21.15.28:0/3886517331 reap_dead start 2024-02-21T23:17:14.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.897+0000 7fc213400640 1 --2- 172.21.15.28:0/3886517331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1f8004800 0x7fc1f8004c80 secure :-1 s=READY pgs=2686 cs=0 l=1 rev1=1 crypto rx=0x7fc1fc001750 tx=0x7fc1f8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2eebef2bcb975a26 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:14.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.898+0000 7fc212a00640 1 -- 172.21.15.28:0/3886517331 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1fc001370 con 0x7fc1f8004800 2024-02-21T23:17:14.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.898+0000 7fc212a00640 1 -- 172.21.15.28:0/3886517331 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1fc0052d0 con 0x7fc1f8004800 2024-02-21T23:17:14.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.898+0000 7fc212a00640 1 -- 172.21.15.28:0/3886517331 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1fc003830 con 0x7fc1f8004800 2024-02-21T23:17:14.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.898+0000 7fc21ae00640 1 -- 172.21.15.28:0/3886517331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1f8004800 msgr2=0x7fc1f8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:14.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.898+0000 7fc21ae00640 1 --2- 172.21.15.28:0/3886517331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1f8004800 0x7fc1f8004c80 secure :-1 s=READY pgs=2686 cs=0 l=1 rev1=1 crypto rx=0x7fc1fc001750 tx=0x7fc1f8005830 comp rx=0 tx=0).stop 2024-02-21T23:17:14.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.898+0000 7fc21ae00640 1 -- 172.21.15.28:0/3886517331 shutdown_connections 2024-02-21T23:17:14.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.898+0000 7fc21ae00640 1 --2- 172.21.15.28:0/3886517331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1f8004800 0x7fc1f8004c80 unknown :-1 s=CLOSED pgs=2686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:14.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.898+0000 7fc21ae00640 1 -- 172.21.15.28:0/3886517331 >> 172.21.15.28:0/3886517331 conn(0x7fc21408b1c0 msgr2=0x7fc2140725a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:14.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.899+0000 7fc21ae00640 1 -- 172.21.15.28:0/3886517331 shutdown_connections 2024-02-21T23:17:14.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.899+0000 7fc21ae00640 1 -- 172.21.15.28:0/3886517331 wait complete. 2024-02-21T23:17:14.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.900+0000 7fc21ae00640 1 Processor -- start 2024-02-21T23:17:14.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.900+0000 7fc21ae00640 1 -- start start 2024-02-21T23:17:14.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.901+0000 7fc21ae00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc1f8004800 0x7fc2141e1f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:14.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.901+0000 7fc21ae00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc21413b330 0x7fc2141edfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:14.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.901+0000 7fc21ae00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc2141ee4e0 0x7fc21413a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:14.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.901+0000 7fc21ae00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc214155cc0 con 0x7fc21413b330 2024-02-21T23:17:14.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.901+0000 7fc21ae00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2141559e0 con 0x7fc1f8004800 2024-02-21T23:17:14.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.901+0000 7fc21ae00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc21413ea00 con 0x7fc2141ee4e0 2024-02-21T23:17:14.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.902+0000 7fc213400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc21413b330 0x7fc2141edfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:14.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.902+0000 7fc213400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc21413b330 0x7fc2141edfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39210/0 (socket says 172.21.15.28:39210) 2024-02-21T23:17:14.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.902+0000 7fc213400640 1 -- 172.21.15.28:0/133129015 learned_addr learned my addr 172.21.15.28:0/133129015 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:14.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.902+0000 7fc213e00640 1 --2- 172.21.15.28:0/133129015 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc1f8004800 0x7fc2141e1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:14.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.902+0000 7fc218e00640 1 --2- 172.21.15.28:0/133129015 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc2141ee4e0 0x7fc21413a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:14.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.902+0000 7fc213400640 1 -- 172.21.15.28:0/133129015 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc2141ee4e0 msgr2=0x7fc21413a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:14.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.902+0000 7fc213400640 1 --2- 172.21.15.28:0/133129015 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc2141ee4e0 0x7fc21413a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:14.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.902+0000 7fc213400640 1 -- 172.21.15.28:0/133129015 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc1f8004800 msgr2=0x7fc2141e1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:14.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.902+0000 7fc213400640 1 --2- 172.21.15.28:0/133129015 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc1f8004800 0x7fc2141e1f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:14.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.902+0000 7fc213400640 1 -- 172.21.15.28:0/133129015 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2141ee990 con 0x7fc21413b330 2024-02-21T23:17:14.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.902+0000 7fc218e00640 1 --2- 172.21.15.28:0/133129015 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc2141ee4e0 0x7fc21413a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:17:14.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.902+0000 7fc213400640 1 --2- 172.21.15.28:0/133129015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc21413b330 0x7fc2141edfa0 secure :-1 s=READY pgs=2687 cs=0 l=1 rev1=1 crypto rx=0x7fc1fc005040 tx=0x7fc1fc001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:14.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.903+0000 7fc20be00640 1 -- 172.21.15.28:0/133129015 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1fc041020 con 0x7fc21413b330 2024-02-21T23:17:14.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.903+0000 7fc20be00640 1 -- 172.21.15.28:0/133129015 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1fc005480 con 0x7fc21413b330 2024-02-21T23:17:14.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.903+0000 7fc20be00640 1 -- 172.21.15.28:0/133129015 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1fc007a30 con 0x7fc21413b330 2024-02-21T23:17:14.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.903+0000 7fc21ae00640 1 -- 172.21.15.28:0/133129015 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2141565a0 con 0x7fc21413b330 2024-02-21T23:17:14.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.903+0000 7fc21ae00640 1 -- 172.21.15.28:0/133129015 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2141e17f0 con 0x7fc21413b330 2024-02-21T23:17:14.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.904+0000 7fc21ae00640 1 -- 172.21.15.28:0/133129015 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc214067f30 con 0x7fc21413b330 2024-02-21T23:17:14.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.905+0000 7fc20be00640 1 -- 172.21.15.28:0/133129015 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc1fc044020 con 0x7fc21413b330 2024-02-21T23:17:14.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.906+0000 7fc20be00640 1 --2- 172.21.15.28:0/133129015 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc1f006b500 0x7fc1f006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:14.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.906+0000 7fc20be00640 1 -- 172.21.15.28:0/133129015 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fc1fc0abe80 con 0x7fc21413b330 2024-02-21T23:17:14.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.907+0000 7fc213e00640 1 --2- 172.21.15.28:0/133129015 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc1f006b500 0x7fc1f006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:14.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.908+0000 7fc213e00640 1 --2- 172.21.15.28:0/133129015 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc1f006b500 0x7fc1f006d9c0 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7fc21406a6d0 tx=0x7fc20c00b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:14.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:14.911+0000 7fc20be00640 1 -- 172.21.15.28:0/133129015 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc1fc04b030 con 0x7fc21413b330 2024-02-21T23:17:15.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:15.224+0000 7fc21ae00640 1 -- 172.21.15.28:0/133129015 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc214154ad0 con 0x7fc21413b330 2024-02-21T23:17:15.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:15.225+0000 7fc20be00640 1 -- 172.21.15.28:0/133129015 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7fc1fc0303c0 con 0x7fc21413b330 2024-02-21T23:17:15.213 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:17:15.213 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:17:15.213 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:17:15.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:15.228+0000 7fc21ae00640 1 -- 172.21.15.28:0/133129015 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc1f006b500 msgr2=0x7fc1f006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:15.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:15.228+0000 7fc21ae00640 1 --2- 172.21.15.28:0/133129015 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc1f006b500 0x7fc1f006d9c0 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7fc21406a6d0 tx=0x7fc20c00b040 comp rx=0 tx=0).stop 2024-02-21T23:17:15.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:15.228+0000 7fc21ae00640 1 -- 172.21.15.28:0/133129015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc21413b330 msgr2=0x7fc2141edfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:15.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:15.228+0000 7fc21ae00640 1 --2- 172.21.15.28:0/133129015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc21413b330 0x7fc2141edfa0 secure :-1 s=READY pgs=2687 cs=0 l=1 rev1=1 crypto rx=0x7fc1fc005040 tx=0x7fc1fc001120 comp rx=0 tx=0).stop 2024-02-21T23:17:15.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:15.228+0000 7fc21ae00640 1 -- 172.21.15.28:0/133129015 shutdown_connections 2024-02-21T23:17:15.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:15.228+0000 7fc21ae00640 1 --2- 172.21.15.28:0/133129015 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc1f006b500 0x7fc1f006d9c0 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:15.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:15.228+0000 7fc21ae00640 1 --2- 172.21.15.28:0/133129015 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc2141ee4e0 0x7fc21413a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:15.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:15.228+0000 7fc21ae00640 1 --2- 172.21.15.28:0/133129015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc21413b330 0x7fc2141edfa0 unknown :-1 s=CLOSED pgs=2687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:15.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:15.228+0000 7fc21ae00640 1 --2- 172.21.15.28:0/133129015 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc1f8004800 0x7fc2141e1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:15.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:15.228+0000 7fc21ae00640 1 -- 172.21.15.28:0/133129015 >> 172.21.15.28:0/133129015 conn(0x7fc21408b1c0 msgr2=0x7fc2140c7350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:15.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:15.228+0000 7fc21ae00640 1 -- 172.21.15.28:0/133129015 shutdown_connections 2024-02-21T23:17:15.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:15.228+0000 7fc21ae00640 1 -- 172.21.15.28:0/133129015 wait complete. 2024-02-21T23:17:15.643 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:17:15.644 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:17:16.645 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:17:16.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.880+0000 7fe6db200640 1 Processor -- start 2024-02-21T23:17:16.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.880+0000 7fe6db200640 1 -- start start 2024-02-21T23:17:16.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.882+0000 7fe6db200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe6dc155960 0x7fe6dc1608a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:16.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.882+0000 7fe6db200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6dc0c74c0 con 0x7fe6dc06dbf0 2024-02-21T23:17:16.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.882+0000 7fe6db200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6dc0c77a0 con 0x7fe6dc0aff90 2024-02-21T23:17:16.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.882+0000 7fe6db200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6dc0c7270 con 0x7fe6dc155960 2024-02-21T23:17:16.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.882+0000 7fe6d3e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe6dc06dbf0 0x7fe6dc158ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:52014/0 (socket says 172.21.15.28:52014) 2024-02-21T23:17:16.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.882+0000 7fe6d3e00640 1 -- 172.21.15.28:0/3000603997 learned_addr learned my addr 172.21.15.28:0/3000603997 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:17:16.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.883+0000 7fe6da800640 1 --2- 172.21.15.28:0/3000603997 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe6dc155960 0x7fe6dc1608a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:16.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.883+0000 7fe6d9400640 1 -- 172.21.15.28:0/3000603997 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 6074668 0 0) 0x7fe6dc0c74c0 con 0x7fe6dc06dbf0 2024-02-21T23:17:16.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.883+0000 7fe6d9400640 1 -- 172.21.15.28:0/3000603997 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6c0003520 con 0x7fe6dc06dbf0 2024-02-21T23:17:16.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.883+0000 7fe6d9400640 1 -- 172.21.15.28:0/3000603997 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2875011802 0 0) 0x7fe6c0003520 con 0x7fe6dc06dbf0 2024-02-21T23:17:16.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.883+0000 7fe6d9400640 1 -- 172.21.15.28:0/3000603997 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe6dc155960 msgr2=0x7fe6dc1608a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:16.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.883+0000 7fe6d9400640 1 --2- 172.21.15.28:0/3000603997 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe6dc155960 0x7fe6dc1608a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:16.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.883+0000 7fe6d9400640 1 -- 172.21.15.28:0/3000603997 >> v1:172.21.15.81:6789/0 conn(0x7fe6dc0aff90 legacy=0x7fe6dc0c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:16.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.883+0000 7fe6d9400640 1 -- 172.21.15.28:0/3000603997 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6dc0c7a80 con 0x7fe6dc06dbf0 2024-02-21T23:17:16.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.883+0000 7fe6d9400640 1 -- 172.21.15.28:0/3000603997 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fe6c4001670 con 0x7fe6dc06dbf0 2024-02-21T23:17:16.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.884+0000 7fe6d9400640 1 -- 172.21.15.28:0/3000603997 >> v1:172.21.15.28:6789/0 conn(0x7fe6dc06dbf0 legacy=0x7fe6dc158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:16.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.884+0000 7fe6d9400640 1 --2- 172.21.15.28:0/3000603997 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe6c00039e0 0x7fe6c0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:16.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.884+0000 7fe6d9400640 1 --2- 172.21.15.28:0/3000603997 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe6dc155960 0x7fe6c0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:16.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.884+0000 7fe6d9400640 1 --2- 172.21.15.28:0/3000603997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6c0004d40 0x7fe6c00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:16.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.884+0000 7fe6d9400640 1 -- 172.21.15.28:0/3000603997 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6dc0c7270 con 0x7fe6c0004d40 2024-02-21T23:17:16.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.884+0000 7fe6d9400640 1 -- 172.21.15.28:0/3000603997 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c000ba70 con 0x7fe6c00039e0 2024-02-21T23:17:16.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.884+0000 7fe6d9400640 1 -- 172.21.15.28:0/3000603997 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c000bc70 con 0x7fe6dc155960 2024-02-21T23:17:16.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.884+0000 7fe6da800640 1 --2- 172.21.15.28:0/3000603997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6c0004d40 0x7fe6c00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:16.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.884+0000 7fe6d3e00640 1 --2- 172.21.15.28:0/3000603997 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe6dc155960 0x7fe6c0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:16.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.884+0000 7fe6d9e00640 1 --2- 172.21.15.28:0/3000603997 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe6c00039e0 0x7fe6c0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:16.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.884+0000 7fe6da800640 1 -- 172.21.15.28:0/3000603997 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe6dc155960 msgr2=0x7fe6c0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:16.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.884+0000 7fe6da800640 1 --2- 172.21.15.28:0/3000603997 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe6dc155960 0x7fe6c0004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:16.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.884+0000 7fe6da800640 1 -- 172.21.15.28:0/3000603997 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe6c00039e0 msgr2=0x7fe6c0003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:16.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.884+0000 7fe6da800640 1 --2- 172.21.15.28:0/3000603997 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe6c00039e0 0x7fe6c0003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:16.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.884+0000 7fe6da800640 1 -- 172.21.15.28:0/3000603997 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6c0003520 con 0x7fe6c0004d40 2024-02-21T23:17:16.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.884+0000 7fe6d9e00640 1 --2- 172.21.15.28:0/3000603997 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe6c00039e0 0x7fe6c0003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:17:16.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.885+0000 7fe6da800640 1 --2- 172.21.15.28:0/3000603997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6c0004d40 0x7fe6c00051c0 secure :-1 s=READY pgs=2689 cs=0 l=1 rev1=1 crypto rx=0x7fe6d4009f70 tx=0x7fe6c0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8eb4eb64d4c3181f server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:16.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.886+0000 7fe6d9400640 1 -- 172.21.15.28:0/3000603997 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d4039070 con 0x7fe6c0004d40 2024-02-21T23:17:16.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.886+0000 7fe6d9400640 1 -- 172.21.15.28:0/3000603997 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6d402cc90 con 0x7fe6c0004d40 2024-02-21T23:17:16.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.886+0000 7fe6d9400640 1 -- 172.21.15.28:0/3000603997 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d4009bf0 con 0x7fe6c0004d40 2024-02-21T23:17:16.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.886+0000 7fe6db200640 1 -- 172.21.15.28:0/3000603997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6c0004d40 msgr2=0x7fe6c00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:16.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.886+0000 7fe6db200640 1 --2- 172.21.15.28:0/3000603997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6c0004d40 0x7fe6c00051c0 secure :-1 s=READY pgs=2689 cs=0 l=1 rev1=1 crypto rx=0x7fe6d4009f70 tx=0x7fe6c0005830 comp rx=0 tx=0).stop 2024-02-21T23:17:16.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.886+0000 7fe6da800640 1 -- 172.21.15.28:0/3000603997 reap_dead start 2024-02-21T23:17:16.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.887+0000 7fe6db200640 1 -- 172.21.15.28:0/3000603997 shutdown_connections 2024-02-21T23:17:16.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.887+0000 7fe6db200640 1 -- 172.21.15.28:0/3000603997 >> 172.21.15.28:0/3000603997 conn(0x7fe6dc0ac370 msgr2=0x7fe6dc0ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:16.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.887+0000 7fe6db200640 1 -- 172.21.15.28:0/3000603997 shutdown_connections 2024-02-21T23:17:16.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.887+0000 7fe6db200640 1 -- 172.21.15.28:0/3000603997 wait complete. 2024-02-21T23:17:16.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.889+0000 7fe6db200640 1 Processor -- start 2024-02-21T23:17:16.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.890+0000 7fe6db200640 1 -- start start 2024-02-21T23:17:16.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.891+0000 7fe6db200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe6c0004d40 0x7fe6dc151b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:16.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.891+0000 7fe6db200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe6dc0c0ea0 0x7fe6dc0c8760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:16.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.891+0000 7fe6db200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6dc0c8ca0 0x7fe6dc1ee6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:16.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.891+0000 7fe6db200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6dc0c58f0 con 0x7fe6dc0c8ca0 2024-02-21T23:17:16.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.891+0000 7fe6db200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6dc158630 con 0x7fe6dc0c0ea0 2024-02-21T23:17:16.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.891+0000 7fe6db200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6dc0c5a70 con 0x7fe6c0004d40 2024-02-21T23:17:16.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.891+0000 7fe6da800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6dc0c8ca0 0x7fe6dc1ee6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:16.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.891+0000 7fe6da800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6dc0c8ca0 0x7fe6dc1ee6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39238/0 (socket says 172.21.15.28:39238) 2024-02-21T23:17:16.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.891+0000 7fe6da800640 1 -- 172.21.15.28:0/3116645573 learned_addr learned my addr 172.21.15.28:0/3116645573 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:16.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.892+0000 7fe6d3e00640 1 --2- 172.21.15.28:0/3116645573 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe6dc0c0ea0 0x7fe6dc0c8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:16.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.892+0000 7fe6da800640 1 -- 172.21.15.28:0/3116645573 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe6c0004d40 msgr2=0x7fe6dc151b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:16.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.892+0000 7fe6da800640 1 --2- 172.21.15.28:0/3116645573 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe6c0004d40 0x7fe6dc151b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:16.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.892+0000 7fe6da800640 1 -- 172.21.15.28:0/3116645573 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe6dc0c0ea0 msgr2=0x7fe6dc0c8760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:16.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.892+0000 7fe6da800640 1 --2- 172.21.15.28:0/3116645573 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe6dc0c0ea0 0x7fe6dc0c8760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:16.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.892+0000 7fe6da800640 1 -- 172.21.15.28:0/3116645573 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6dc152070 con 0x7fe6dc0c8ca0 2024-02-21T23:17:16.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.892+0000 7fe6da800640 1 --2- 172.21.15.28:0/3116645573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6dc0c8ca0 0x7fe6dc1ee6a0 secure :-1 s=READY pgs=2690 cs=0 l=1 rev1=1 crypto rx=0x7fe6d4037040 tx=0x7fe6d4009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:16.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.892+0000 7fe6d2000640 1 -- 172.21.15.28:0/3116645573 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d4043020 con 0x7fe6dc0c8ca0 2024-02-21T23:17:16.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.893+0000 7fe6d2000640 1 -- 172.21.15.28:0/3116645573 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6d4034d40 con 0x7fe6dc0c8ca0 2024-02-21T23:17:16.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.893+0000 7fe6db200640 1 -- 172.21.15.28:0/3116645573 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6dc0c3da0 con 0x7fe6dc0c8ca0 2024-02-21T23:17:16.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.893+0000 7fe6d2000640 1 -- 172.21.15.28:0/3116645573 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d4039040 con 0x7fe6dc0c8ca0 2024-02-21T23:17:16.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.893+0000 7fe6db200640 1 -- 172.21.15.28:0/3116645573 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6dc1eaeb0 con 0x7fe6dc0c8ca0 2024-02-21T23:17:16.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.894+0000 7fe6db200640 1 -- 172.21.15.28:0/3116645573 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6dc152530 con 0x7fe6dc0c8ca0 2024-02-21T23:17:16.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.895+0000 7fe6d2000640 1 -- 172.21.15.28:0/3116645573 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe6d4046020 con 0x7fe6dc0c8ca0 2024-02-21T23:17:16.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.897+0000 7fe6d2000640 1 --2- 172.21.15.28:0/3116645573 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe6b806b500 0x7fe6b806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:16.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.897+0000 7fe6d2000640 1 -- 172.21.15.28:0/3116645573 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fe6d40ac7e0 con 0x7fe6dc0c8ca0 2024-02-21T23:17:16.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.897+0000 7fe6d9e00640 1 --2- 172.21.15.28:0/3116645573 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe6b806b500 0x7fe6b806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:16.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.899+0000 7fe6d9e00640 1 --2- 172.21.15.28:0/3116645573 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe6b806b500 0x7fe6b806d9c0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc1e8ae0 tx=0x7fe6cc009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:16.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:16.901+0000 7fe6d2000640 1 -- 172.21.15.28:0/3116645573 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fe6dc152530 con 0x7fe6dc0c8ca0 2024-02-21T23:17:17.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:17.145+0000 7fe6db200640 1 -- 172.21.15.28:0/3116645573 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe6dc1eaca0 con 0x7fe6dc0c8ca0 2024-02-21T23:17:17.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:17.146+0000 7fe6d2000640 1 -- 172.21.15.28:0/3116645573 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7fe6d404d030 con 0x7fe6dc0c8ca0 2024-02-21T23:17:17.134 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:17:17.134 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:17:17.134 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:17:17.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:17.149+0000 7fe6db200640 1 -- 172.21.15.28:0/3116645573 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe6b806b500 msgr2=0x7fe6b806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:17.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:17.149+0000 7fe6db200640 1 --2- 172.21.15.28:0/3116645573 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe6b806b500 0x7fe6b806d9c0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc1e8ae0 tx=0x7fe6cc009040 comp rx=0 tx=0).stop 2024-02-21T23:17:17.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:17.149+0000 7fe6db200640 1 -- 172.21.15.28:0/3116645573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6dc0c8ca0 msgr2=0x7fe6dc1ee6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:17.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:17.149+0000 7fe6db200640 1 --2- 172.21.15.28:0/3116645573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6dc0c8ca0 0x7fe6dc1ee6a0 secure :-1 s=READY pgs=2690 cs=0 l=1 rev1=1 crypto rx=0x7fe6d4037040 tx=0x7fe6d4009750 comp rx=0 tx=0).stop 2024-02-21T23:17:17.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:17.149+0000 7fe6db200640 1 -- 172.21.15.28:0/3116645573 shutdown_connections 2024-02-21T23:17:17.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:17.149+0000 7fe6db200640 1 --2- 172.21.15.28:0/3116645573 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe6b806b500 0x7fe6b806d9c0 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:17.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:17.149+0000 7fe6db200640 1 --2- 172.21.15.28:0/3116645573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6dc0c8ca0 0x7fe6dc1ee6a0 unknown :-1 s=CLOSED pgs=2690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:17.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:17.149+0000 7fe6db200640 1 --2- 172.21.15.28:0/3116645573 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe6dc0c0ea0 0x7fe6dc0c8760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:17.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:17.149+0000 7fe6db200640 1 --2- 172.21.15.28:0/3116645573 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe6c0004d40 0x7fe6dc151b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:17.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:17.149+0000 7fe6db200640 1 -- 172.21.15.28:0/3116645573 >> 172.21.15.28:0/3116645573 conn(0x7fe6dc0ac370 msgr2=0x7fe6dc0c50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:17.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:17.149+0000 7fe6db200640 1 -- 172.21.15.28:0/3116645573 shutdown_connections 2024-02-21T23:17:17.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:17.150+0000 7fe6db200640 1 -- 172.21.15.28:0/3116645573 wait complete. 2024-02-21T23:17:17.558 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:17:17.558 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:17:18.559 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:17:18.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:18.789+0000 7f1cd3e00640 1 Processor -- start 2024-02-21T23:17:18.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:18.789+0000 7f1cd3e00640 1 -- start start 2024-02-21T23:17:18.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:18.791+0000 7f1cd3e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f1cd40c8560 0x7f1cd40c7ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:18.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:18.791+0000 7f1cd3e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1cd40c64d0 0x7f1cd40c8940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:18.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:18.791+0000 7f1cd3e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cd40c2c20 con 0x7f1cd40c4d90 2024-02-21T23:17:18.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:18.791+0000 7f1cd3e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cd40c6910 con 0x7f1cd40c64d0 2024-02-21T23:17:18.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:18.791+0000 7f1cd3e00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cd4155100 con 0x7f1cd40c8560 2024-02-21T23:17:18.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:18.791+0000 7f1cd2000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1cd40c64d0 0x7f1cd40c8940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:18.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:18.792+0000 7f1cd2000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1cd40c64d0 0x7f1cd40c8940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39242/0 (socket says 172.21.15.28:39242) 2024-02-21T23:17:18.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:18.792+0000 7f1cd2000640 1 -- 172.21.15.28:0/1359196350 learned_addr learned my addr 172.21.15.28:0/1359196350 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:18.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:18.792+0000 7f1cd2a00640 1 --2- 172.21.15.28:0/1359196350 >> v2:172.21.15.81:3300/0 conn(0x7f1cd40c8560 0x7f1cd40c7ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:18.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:18.793+0000 7f1cd1600640 1 -- 172.21.15.28:0/1359196350 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1538645272 0 0) 0x7f1cd40c2c20 con 0x7f1cd40c4d90 2024-02-21T23:17:18.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:18.793+0000 7f1cd1600640 1 -- 172.21.15.28:0/1359196350 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cb8003520 con 0x7f1cd40c4d90 2024-02-21T23:17:18.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:18.793+0000 7f1cd2000640 1 -- 172.21.15.28:0/1359196350 >> v2:172.21.15.81:3300/0 conn(0x7f1cd40c8560 msgr2=0x7f1cd40c7ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:18.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:18.793+0000 7f1cd2000640 1 --2- 172.21.15.28:0/1359196350 >> v2:172.21.15.81:3300/0 conn(0x7f1cd40c8560 0x7f1cd40c7ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:18.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:18.793+0000 7f1cd2000640 1 -- 172.21.15.28:0/1359196350 >> v1:172.21.15.81:6789/0 conn(0x7f1cd40c4d90 legacy=0x7f1cd40c5200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:18.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:18.793+0000 7f1cd2000640 1 -- 172.21.15.28:0/1359196350 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cd41553e0 con 0x7f1cd40c64d0 2024-02-21T23:17:18.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:18.794+0000 7f1cd2000640 1 --2- 172.21.15.28:0/1359196350 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cd40c64d0 0x7f1cd40c8940 secure :-1 s=READY pgs=2691 cs=0 l=1 rev1=1 crypto rx=0x7f1cd40aa1c0 tx=0x7f1cc002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9993d3a48a0ac420 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:18.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:18.795+0000 7f1cd1600640 1 -- 172.21.15.28:0/1359196350 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cc0034070 con 0x7f1cd40c64d0 2024-02-21T23:17:18.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:18.795+0000 7f1cd1600640 1 -- 172.21.15.28:0/1359196350 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cc0030860 con 0x7f1cd40c64d0 2024-02-21T23:17:18.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:18.795+0000 7f1cd1600640 1 -- 172.21.15.28:0/1359196350 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cc0030c10 con 0x7f1cd40c64d0 2024-02-21T23:17:18.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:18.795+0000 7f1cd3e00640 1 -- 172.21.15.28:0/1359196350 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cd40c64d0 msgr2=0x7f1cd40c8940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:18.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:18.795+0000 7f1cd3e00640 1 --2- 172.21.15.28:0/1359196350 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cd40c64d0 0x7f1cd40c8940 secure :-1 s=READY pgs=2691 cs=0 l=1 rev1=1 crypto rx=0x7f1cd40aa1c0 tx=0x7f1cc002b9a0 comp rx=0 tx=0).stop 2024-02-21T23:17:18.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:18.796+0000 7f1cd3e00640 1 -- 172.21.15.28:0/1359196350 shutdown_connections 2024-02-21T23:17:18.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:18.796+0000 7f1cd3e00640 1 --2- 172.21.15.28:0/1359196350 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cd40c64d0 0x7f1cd40c8940 unknown :-1 s=CLOSED pgs=2691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:18.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:18.796+0000 7f1cd3e00640 1 --2- 172.21.15.28:0/1359196350 >> v2:172.21.15.81:3300/0 conn(0x7f1cd40c8560 0x7f1cd40c7ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:18.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:18.796+0000 7f1cd3e00640 1 -- 172.21.15.28:0/1359196350 >> 172.21.15.28:0/1359196350 conn(0x7f1cd406ae70 msgr2=0x7f1cd406b280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:18.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:18.796+0000 7f1cd3e00640 1 -- 172.21.15.28:0/1359196350 shutdown_connections 2024-02-21T23:17:18.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:18.796+0000 7f1cd3e00640 1 -- 172.21.15.28:0/1359196350 wait complete. 2024-02-21T23:17:18.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:18.797+0000 7f1cd3e00640 1 Processor -- start 2024-02-21T23:17:18.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:18.798+0000 7f1cd3e00640 1 -- start start 2024-02-21T23:17:18.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:18.798+0000 7f1cd3e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1cd40c4d90 0x7f1cd41ee5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:18.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:18.798+0000 7f1cd3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cd40c64d0 0x7f1cd406c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:18.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:18.799+0000 7f1cd3e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1cd40c8560 0x7f1cd414d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:18.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:18.799+0000 7f1cd3e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd4154db0 con 0x7f1cd40c64d0 2024-02-21T23:17:18.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:18.799+0000 7f1cd3e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd41559a0 con 0x7f1cd40c8560 2024-02-21T23:17:18.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:18.799+0000 7f1cd3e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd40c5910 con 0x7f1cd40c4d90 2024-02-21T23:17:18.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:18.799+0000 7f1cd2000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cd40c64d0 0x7f1cd406c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:18.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:18.799+0000 7f1cd2000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cd40c64d0 0x7f1cd406c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39246/0 (socket says 172.21.15.28:39246) 2024-02-21T23:17:18.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:18.799+0000 7f1cd2000640 1 -- 172.21.15.28:0/2603733367 learned_addr learned my addr 172.21.15.28:0/2603733367 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:18.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:18.799+0000 7f1cd2000640 1 -- 172.21.15.28:0/2603733367 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1cd40c4d90 msgr2=0x7f1cd41ee5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:18.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:18.799+0000 7f1cd2000640 1 --2- 172.21.15.28:0/2603733367 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1cd40c4d90 0x7f1cd41ee5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:18.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:18.799+0000 7f1cd2000640 1 -- 172.21.15.28:0/2603733367 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1cd40c8560 msgr2=0x7f1cd414d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:18.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:18.799+0000 7f1cd2000640 1 --2- 172.21.15.28:0/2603733367 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1cd40c8560 0x7f1cd414d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:18.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:18.799+0000 7f1cd2000640 1 -- 172.21.15.28:0/2603733367 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cd4157100 con 0x7f1cd40c64d0 2024-02-21T23:17:18.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:18.800+0000 7f1cd2000640 1 --2- 172.21.15.28:0/2603733367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cd40c64d0 0x7f1cd406c9b0 secure :-1 s=READY pgs=2692 cs=0 l=1 rev1=1 crypto rx=0x7f1cc002b9e0 tx=0x7f1cc0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:18.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:18.800+0000 7f1ccaa00640 1 -- 172.21.15.28:0/2603733367 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cc0048070 con 0x7f1cd40c64d0 2024-02-21T23:17:18.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:18.800+0000 7f1ccaa00640 1 -- 172.21.15.28:0/2603733367 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cc0030e20 con 0x7f1cd40c64d0 2024-02-21T23:17:18.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:18.800+0000 7f1ccaa00640 1 -- 172.21.15.28:0/2603733367 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cc0034040 con 0x7f1cd40c64d0 2024-02-21T23:17:18.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:18.800+0000 7f1cd3e00640 1 -- 172.21.15.28:0/2603733367 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cd40a83b0 con 0x7f1cd40c64d0 2024-02-21T23:17:18.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:18.800+0000 7f1cd3e00640 1 -- 172.21.15.28:0/2603733367 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cd406cef0 con 0x7f1cd40c64d0 2024-02-21T23:17:18.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:18.802+0000 7f1cd3e00640 1 -- 172.21.15.28:0/2603733367 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cd404f290 con 0x7f1cd40c64d0 2024-02-21T23:17:18.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:18.804+0000 7f1ccaa00640 1 -- 172.21.15.28:0/2603733367 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1cc0037070 con 0x7f1cd40c64d0 2024-02-21T23:17:18.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:18.804+0000 7f1ccaa00640 1 --2- 172.21.15.28:0/2603733367 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1cb406b4d0 0x7f1cb406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:18.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:18.805+0000 7f1ccaa00640 1 -- 172.21.15.28:0/2603733367 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f1cc00ac100 con 0x7f1cd40c64d0 2024-02-21T23:17:18.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:18.805+0000 7f1cd2a00640 1 --2- 172.21.15.28:0/2603733367 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1cb406b4d0 0x7f1cb406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:18.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:18.806+0000 7f1cd2a00640 1 --2- 172.21.15.28:0/2603733367 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1cb406b4d0 0x7f1cb406d990 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f1cc4006a90 tx=0x7f1cc4006a20 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:18.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:18.810+0000 7f1ccaa00640 1 -- 172.21.15.28:0/2603733367 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f1cc007ae00 con 0x7f1cd40c64d0 2024-02-21T23:17:19.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:19.085+0000 7f1cd3e00640 1 -- 172.21.15.28:0/2603733367 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1cd40c1f40 con 0x7f1cd40c64d0 2024-02-21T23:17:19.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:19.086+0000 7f1ccaa00640 1 -- 172.21.15.28:0/2603733367 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7f1cc007a700 con 0x7f1cd40c64d0 2024-02-21T23:17:19.074 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:17:19.074 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:17:19.074 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:17:19.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:19.089+0000 7f1cd3e00640 1 -- 172.21.15.28:0/2603733367 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1cb406b4d0 msgr2=0x7f1cb406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:19.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:19.089+0000 7f1cd3e00640 1 --2- 172.21.15.28:0/2603733367 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1cb406b4d0 0x7f1cb406d990 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f1cc4006a90 tx=0x7f1cc4006a20 comp rx=0 tx=0).stop 2024-02-21T23:17:19.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:19.089+0000 7f1cd3e00640 1 -- 172.21.15.28:0/2603733367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cd40c64d0 msgr2=0x7f1cd406c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:19.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:19.089+0000 7f1cd3e00640 1 --2- 172.21.15.28:0/2603733367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cd40c64d0 0x7f1cd406c9b0 secure :-1 s=READY pgs=2692 cs=0 l=1 rev1=1 crypto rx=0x7f1cc002b9e0 tx=0x7f1cc0004730 comp rx=0 tx=0).stop 2024-02-21T23:17:19.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:19.089+0000 7f1cd3e00640 1 -- 172.21.15.28:0/2603733367 shutdown_connections 2024-02-21T23:17:19.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:19.089+0000 7f1cd3e00640 1 --2- 172.21.15.28:0/2603733367 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f1cb406b4d0 0x7f1cb406d990 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:19.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:19.089+0000 7f1cd3e00640 1 --2- 172.21.15.28:0/2603733367 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1cd40c8560 0x7f1cd414d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:19.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:19.089+0000 7f1cd3e00640 1 --2- 172.21.15.28:0/2603733367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cd40c64d0 0x7f1cd406c9b0 unknown :-1 s=CLOSED pgs=2692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:19.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:19.089+0000 7f1cd3e00640 1 --2- 172.21.15.28:0/2603733367 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1cd40c4d90 0x7f1cd41ee5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:19.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:19.089+0000 7f1cd3e00640 1 -- 172.21.15.28:0/2603733367 >> 172.21.15.28:0/2603733367 conn(0x7f1cd406ae70 msgr2=0x7f1cd40c5200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:19.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:19.089+0000 7f1cd3e00640 1 -- 172.21.15.28:0/2603733367 shutdown_connections 2024-02-21T23:17:19.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:19.090+0000 7f1cd3e00640 1 -- 172.21.15.28:0/2603733367 wait complete. 2024-02-21T23:17:19.500 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:17:19.500 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:17:20.501 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:17:20.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:20.740+0000 7fc904a00640 1 Processor -- start 2024-02-21T23:17:20.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:20.740+0000 7fc904a00640 1 -- start start 2024-02-21T23:17:20.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:20.741+0000 7fc904a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc900152ab0 0x7fc9000c7fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:20.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:20.741+0000 7fc904a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc9000c6710 0x7fc9000c5210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:20.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:20.741+0000 7fc904a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fc9000c5750 0x7fc900158850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:20.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:20.741+0000 7fc904a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9000c3ce0 con 0x7fc9000c6710 2024-02-21T23:17:20.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:20.741+0000 7fc904a00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9000a8180 con 0x7fc9000c5750 2024-02-21T23:17:20.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:20.741+0000 7fc904a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9000c33b0 con 0x7fc900152ab0 2024-02-21T23:17:20.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:20.742+0000 7fc8fd400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc9000c6710 0x7fc9000c5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:20.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:20.742+0000 7fc8fd400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc9000c6710 0x7fc9000c5210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39252/0 (socket says 172.21.15.28:39252) 2024-02-21T23:17:20.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:20.742+0000 7fc8fd400640 1 -- 172.21.15.28:0/4013563373 learned_addr learned my addr 172.21.15.28:0/4013563373 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:20.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:20.742+0000 7fc8fe800640 1 --2- 172.21.15.28:0/4013563373 >> v2:172.21.15.81:3300/0 conn(0x7fc9000c5750 0x7fc900158850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:20.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:20.742+0000 7fc8fde00640 1 --2- 172.21.15.28:0/4013563373 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc900152ab0 0x7fc9000c7fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:20.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:20.742+0000 7fc8fd400640 1 -- 172.21.15.28:0/4013563373 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc900152ab0 msgr2=0x7fc9000c7fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:20.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:20.742+0000 7fc8fd400640 1 --2- 172.21.15.28:0/4013563373 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc900152ab0 0x7fc9000c7fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:20.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:20.742+0000 7fc8fd400640 1 -- 172.21.15.28:0/4013563373 >> v2:172.21.15.81:3300/0 conn(0x7fc9000c5750 msgr2=0x7fc900158850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:20.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:20.742+0000 7fc8fd400640 1 --2- 172.21.15.28:0/4013563373 >> v2:172.21.15.81:3300/0 conn(0x7fc9000c5750 0x7fc900158850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:20.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:20.742+0000 7fc8fd400640 1 -- 172.21.15.28:0/4013563373 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9000c3690 con 0x7fc9000c6710 2024-02-21T23:17:20.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:20.742+0000 7fc8fe800640 1 --2- 172.21.15.28:0/4013563373 >> v2:172.21.15.81:3300/0 conn(0x7fc9000c5750 0x7fc900158850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:17:20.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:20.745+0000 7fc8fd400640 1 --2- 172.21.15.28:0/4013563373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9000c6710 0x7fc9000c5210 secure :-1 s=READY pgs=2693 cs=0 l=1 rev1=1 crypto rx=0x7fc9000c3030 tx=0x7fc8e802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cb3c7c1cb30632c server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:20.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:20.745+0000 7fc8fca00640 1 -- 172.21.15.28:0/4013563373 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e8035020 con 0x7fc9000c6710 2024-02-21T23:17:20.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:20.745+0000 7fc8fca00640 1 -- 172.21.15.28:0/4013563373 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8e8031870 con 0x7fc9000c6710 2024-02-21T23:17:20.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:20.745+0000 7fc8fca00640 1 -- 172.21.15.28:0/4013563373 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e8031c40 con 0x7fc9000c6710 2024-02-21T23:17:20.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:20.746+0000 7fc904a00640 1 -- 172.21.15.28:0/4013563373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9000c6710 msgr2=0x7fc9000c5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:20.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:20.746+0000 7fc904a00640 1 --2- 172.21.15.28:0/4013563373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9000c6710 0x7fc9000c5210 secure :-1 s=READY pgs=2693 cs=0 l=1 rev1=1 crypto rx=0x7fc9000c3030 tx=0x7fc8e802b910 comp rx=0 tx=0).stop 2024-02-21T23:17:20.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:20.746+0000 7fc904a00640 1 -- 172.21.15.28:0/4013563373 shutdown_connections 2024-02-21T23:17:20.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:20.746+0000 7fc904a00640 1 --2- 172.21.15.28:0/4013563373 >> v2:172.21.15.81:3300/0 conn(0x7fc9000c5750 0x7fc900158850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:20.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:20.746+0000 7fc904a00640 1 --2- 172.21.15.28:0/4013563373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9000c6710 0x7fc9000c5210 unknown :-1 s=CLOSED pgs=2693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:20.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:20.746+0000 7fc904a00640 1 --2- 172.21.15.28:0/4013563373 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc900152ab0 0x7fc9000c7fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:20.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:20.746+0000 7fc904a00640 1 -- 172.21.15.28:0/4013563373 >> 172.21.15.28:0/4013563373 conn(0x7fc9000c8810 msgr2=0x7fc9000c8bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:20.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:20.746+0000 7fc904a00640 1 -- 172.21.15.28:0/4013563373 shutdown_connections 2024-02-21T23:17:20.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:20.746+0000 7fc904a00640 1 -- 172.21.15.28:0/4013563373 wait complete. 2024-02-21T23:17:20.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:20.748+0000 7fc904a00640 1 Processor -- start 2024-02-21T23:17:20.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:20.748+0000 7fc904a00640 1 -- start start 2024-02-21T23:17:20.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:20.749+0000 7fc904a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc9000c5750 0x7fc90014fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:20.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:20.749+0000 7fc904a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc9000c6710 0x7fc90014d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:20.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:20.749+0000 7fc904a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc900152ab0 0x7fc900150e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:20.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:20.749+0000 7fc904a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc900158f10 con 0x7fc900152ab0 2024-02-21T23:17:20.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:20.749+0000 7fc904a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc900158d90 con 0x7fc9000c6710 2024-02-21T23:17:20.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:20.749+0000 7fc904a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc900159090 con 0x7fc9000c5750 2024-02-21T23:17:20.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:20.749+0000 7fc8fe800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc900152ab0 0x7fc900150e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:20.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:20.749+0000 7fc8fe800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc900152ab0 0x7fc900150e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39264/0 (socket says 172.21.15.28:39264) 2024-02-21T23:17:20.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:20.749+0000 7fc8fe800640 1 -- 172.21.15.28:0/679919248 learned_addr learned my addr 172.21.15.28:0/679919248 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:20.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:20.749+0000 7fc8fde00640 1 --2- 172.21.15.28:0/679919248 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc9000c5750 0x7fc90014fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:20.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:20.749+0000 7fc8fd400640 1 --2- 172.21.15.28:0/679919248 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc9000c6710 0x7fc90014d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:20.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:20.750+0000 7fc8fe800640 1 -- 172.21.15.28:0/679919248 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc9000c5750 msgr2=0x7fc90014fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:20.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:20.750+0000 7fc8fe800640 1 --2- 172.21.15.28:0/679919248 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc9000c5750 0x7fc90014fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:20.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:20.750+0000 7fc8fe800640 1 -- 172.21.15.28:0/679919248 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc9000c6710 msgr2=0x7fc90014d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:20.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:20.750+0000 7fc8fe800640 1 --2- 172.21.15.28:0/679919248 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc9000c6710 0x7fc90014d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:20.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:20.750+0000 7fc8fe800640 1 -- 172.21.15.28:0/679919248 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc90014f0e0 con 0x7fc900152ab0 2024-02-21T23:17:20.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:20.750+0000 7fc8fe800640 1 --2- 172.21.15.28:0/679919248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc900152ab0 0x7fc900150e40 secure :-1 s=READY pgs=2694 cs=0 l=1 rev1=1 crypto rx=0x7fc8f8007e10 tx=0x7fc8f800d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:20.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:20.750+0000 7fc8f6000640 1 -- 172.21.15.28:0/679919248 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f801a070 con 0x7fc900152ab0 2024-02-21T23:17:20.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:20.750+0000 7fc8f6000640 1 -- 172.21.15.28:0/679919248 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8f8003b90 con 0x7fc900152ab0 2024-02-21T23:17:20.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:20.750+0000 7fc8f6000640 1 -- 172.21.15.28:0/679919248 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f8004030 con 0x7fc900152ab0 2024-02-21T23:17:20.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:20.750+0000 7fc904a00640 1 -- 172.21.15.28:0/679919248 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc90014f3c0 con 0x7fc900152ab0 2024-02-21T23:17:20.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:20.750+0000 7fc904a00640 1 -- 172.21.15.28:0/679919248 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc90014f980 con 0x7fc900152ab0 2024-02-21T23:17:20.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:20.752+0000 7fc904a00640 1 -- 172.21.15.28:0/679919248 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc900067f30 con 0x7fc900152ab0 2024-02-21T23:17:20.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:20.753+0000 7fc8f6000640 1 -- 172.21.15.28:0/679919248 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc8f801d070 con 0x7fc900152ab0 2024-02-21T23:17:20.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:20.754+0000 7fc8f6000640 1 --2- 172.21.15.28:0/679919248 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc8dc06b4d0 0x7fc8dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:20.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:20.754+0000 7fc8f6000640 1 -- 172.21.15.28:0/679919248 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fc8f808a780 con 0x7fc900152ab0 2024-02-21T23:17:20.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:20.754+0000 7fc8fde00640 1 --2- 172.21.15.28:0/679919248 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc8dc06b4d0 0x7fc8dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:20.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:20.756+0000 7fc8fde00640 1 --2- 172.21.15.28:0/679919248 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc8dc06b4d0 0x7fc8dc06d990 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fc8f0009a30 tx=0x7fc8f0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:20.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:20.763+0000 7fc8f6000640 1 -- 172.21.15.28:0/679919248 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc8f8010070 con 0x7fc900152ab0 2024-02-21T23:17:21.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:21.014+0000 7fc904a00640 1 -- 172.21.15.28:0/679919248 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc900150940 con 0x7fc900152ab0 2024-02-21T23:17:21.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:21.015+0000 7fc8f6000640 1 -- 172.21.15.28:0/679919248 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7fc8f8059370 con 0x7fc900152ab0 2024-02-21T23:17:21.002 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:17:21.002 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:17:21.002 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:17:21.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:21.018+0000 7fc904a00640 1 -- 172.21.15.28:0/679919248 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc8dc06b4d0 msgr2=0x7fc8dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:21.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:21.018+0000 7fc904a00640 1 --2- 172.21.15.28:0/679919248 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc8dc06b4d0 0x7fc8dc06d990 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fc8f0009a30 tx=0x7fc8f0008040 comp rx=0 tx=0).stop 2024-02-21T23:17:21.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:21.018+0000 7fc904a00640 1 -- 172.21.15.28:0/679919248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc900152ab0 msgr2=0x7fc900150e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:21.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:21.018+0000 7fc904a00640 1 --2- 172.21.15.28:0/679919248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc900152ab0 0x7fc900150e40 secure :-1 s=READY pgs=2694 cs=0 l=1 rev1=1 crypto rx=0x7fc8f8007e10 tx=0x7fc8f800d4a0 comp rx=0 tx=0).stop 2024-02-21T23:17:21.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:21.018+0000 7fc904a00640 1 -- 172.21.15.28:0/679919248 shutdown_connections 2024-02-21T23:17:21.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:21.018+0000 7fc904a00640 1 --2- 172.21.15.28:0/679919248 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc8dc06b4d0 0x7fc8dc06d990 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:21.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:21.018+0000 7fc904a00640 1 --2- 172.21.15.28:0/679919248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc900152ab0 0x7fc900150e40 unknown :-1 s=CLOSED pgs=2694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:21.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:21.018+0000 7fc904a00640 1 --2- 172.21.15.28:0/679919248 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc9000c6710 0x7fc90014d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:21.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:21.018+0000 7fc904a00640 1 --2- 172.21.15.28:0/679919248 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc9000c5750 0x7fc90014fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:21.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:21.018+0000 7fc904a00640 1 -- 172.21.15.28:0/679919248 >> 172.21.15.28:0/679919248 conn(0x7fc9000c8810 msgr2=0x7fc9000c7c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:21.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:21.018+0000 7fc904a00640 1 -- 172.21.15.28:0/679919248 shutdown_connections 2024-02-21T23:17:21.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:21.018+0000 7fc904a00640 1 -- 172.21.15.28:0/679919248 wait complete. 2024-02-21T23:17:21.428 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:17:21.428 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:17:22.429 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:17:22.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.663+0000 7f86dfe00640 1 Processor -- start 2024-02-21T23:17:22.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.663+0000 7f86dfe00640 1 -- start start 2024-02-21T23:17:22.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.665+0000 7f86dfe00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f86e006dbf0 0x7f86e0158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:22.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.665+0000 7f86dfe00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f86e0155960 0x7f86e0159000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:22.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.665+0000 7f86dfe00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86e00c74c0 con 0x7f86e00aff90 2024-02-21T23:17:22.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.665+0000 7f86dfe00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86e00c7270 con 0x7f86e0155960 2024-02-21T23:17:22.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.665+0000 7f86dfe00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86e0158630 con 0x7f86e006dbf0 2024-02-21T23:17:22.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.665+0000 7f86dea00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f86e00aff90 0x7f86e00c51e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:52030/0 (socket says 172.21.15.28:52030) 2024-02-21T23:17:22.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.665+0000 7f86dea00640 1 -- 172.21.15.28:0/3077376754 learned_addr learned my addr 172.21.15.28:0/3077376754 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:17:22.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.666+0000 7f86d7e00640 1 --2- 172.21.15.28:0/3077376754 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f86e006dbf0 0x7f86e0158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:22.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.666+0000 7f86df400640 1 --2- 172.21.15.28:0/3077376754 >> v2:172.21.15.81:3300/0 conn(0x7f86e0155960 0x7f86e0159000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:22.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.666+0000 7f86de000640 1 -- 172.21.15.28:0/3077376754 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 858030347 0 0) 0x7f86e00c74c0 con 0x7f86e00aff90 2024-02-21T23:17:22.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.666+0000 7f86de000640 1 -- 172.21.15.28:0/3077376754 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86c4003520 con 0x7f86e00aff90 2024-02-21T23:17:22.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.666+0000 7f86de000640 1 -- 172.21.15.28:0/3077376754 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1757388754 0 0) 0x7f86c4003520 con 0x7f86e00aff90 2024-02-21T23:17:22.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.666+0000 7f86de000640 1 -- 172.21.15.28:0/3077376754 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f86e006dbf0 msgr2=0x7f86e0158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:22.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.666+0000 7f86de000640 1 --2- 172.21.15.28:0/3077376754 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f86e006dbf0 0x7f86e0158ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:22.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.666+0000 7f86de000640 1 -- 172.21.15.28:0/3077376754 >> v2:172.21.15.81:3300/0 conn(0x7f86e0155960 msgr2=0x7f86e0159000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:22.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.666+0000 7f86de000640 1 --2- 172.21.15.28:0/3077376754 >> v2:172.21.15.81:3300/0 conn(0x7f86e0155960 0x7f86e0159000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:22.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.666+0000 7f86de000640 1 -- 172.21.15.28:0/3077376754 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86e00c77a0 con 0x7f86e00aff90 2024-02-21T23:17:22.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.666+0000 7f86de000640 1 -- 172.21.15.28:0/3077376754 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f86d0001670 con 0x7f86e00aff90 2024-02-21T23:17:22.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.666+0000 7f86de000640 1 -- 172.21.15.28:0/3077376754 >> v1:172.21.15.28:6789/0 conn(0x7f86e00aff90 legacy=0x7f86e00c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:22.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.666+0000 7f86de000640 1 --2- 172.21.15.28:0/3077376754 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86c40039b0 0x7f86c4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:22.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.666+0000 7f86de000640 1 --2- 172.21.15.28:0/3077376754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86c4004800 0x7f86c4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:22.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.666+0000 7f86de000640 1 --2- 172.21.15.28:0/3077376754 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f86e006dbf0 0x7f86c40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:22.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.666+0000 7f86de000640 1 -- 172.21.15.28:0/3077376754 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86e00c7270 con 0x7f86c4004800 2024-02-21T23:17:22.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.666+0000 7f86de000640 1 -- 172.21.15.28:0/3077376754 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86e0158630 con 0x7f86c40039b0 2024-02-21T23:17:22.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.666+0000 7f86de000640 1 -- 172.21.15.28:0/3077376754 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86c400b9c0 con 0x7f86e006dbf0 2024-02-21T23:17:22.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.667+0000 7f86d7e00640 1 --2- 172.21.15.28:0/3077376754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86c4004800 0x7f86c4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:22.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.667+0000 7f86dea00640 1 --2- 172.21.15.28:0/3077376754 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86c40039b0 0x7f86c4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:22.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.667+0000 7f86df400640 1 --2- 172.21.15.28:0/3077376754 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f86e006dbf0 0x7f86c40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:22.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.667+0000 7f86d7e00640 1 -- 172.21.15.28:0/3077376754 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f86e006dbf0 msgr2=0x7f86c40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:22.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.667+0000 7f86d7e00640 1 --2- 172.21.15.28:0/3077376754 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f86e006dbf0 0x7f86c40051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:22.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.667+0000 7f86d7e00640 1 -- 172.21.15.28:0/3077376754 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86c40039b0 msgr2=0x7f86c4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:22.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.667+0000 7f86d7e00640 1 --2- 172.21.15.28:0/3077376754 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86c40039b0 0x7f86c4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:22.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.667+0000 7f86d7e00640 1 -- 172.21.15.28:0/3077376754 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86c4003520 con 0x7f86c4004800 2024-02-21T23:17:22.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.667+0000 7f86dea00640 1 --2- 172.21.15.28:0/3077376754 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86c40039b0 0x7f86c4003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:17:22.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.669+0000 7f86d7e00640 1 --2- 172.21.15.28:0/3077376754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86c4004800 0x7f86c4004c80 secure :-1 s=READY pgs=2696 cs=0 l=1 rev1=1 crypto rx=0x7f86c8009ac0 tx=0x7f86c4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=415c956045349764 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:22.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.669+0000 7f86de000640 1 -- 172.21.15.28:0/3077376754 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c8039070 con 0x7f86c4004800 2024-02-21T23:17:22.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.669+0000 7f86de000640 1 -- 172.21.15.28:0/3077376754 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86c802c6b0 con 0x7f86c4004800 2024-02-21T23:17:22.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.669+0000 7f86de000640 1 -- 172.21.15.28:0/3077376754 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c8009690 con 0x7f86c4004800 2024-02-21T23:17:22.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.670+0000 7f86dfe00640 1 -- 172.21.15.28:0/3077376754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86c4004800 msgr2=0x7f86c4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:22.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.670+0000 7f86dfe00640 1 --2- 172.21.15.28:0/3077376754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86c4004800 0x7f86c4004c80 secure :-1 s=READY pgs=2696 cs=0 l=1 rev1=1 crypto rx=0x7f86c8009ac0 tx=0x7f86c4005830 comp rx=0 tx=0).stop 2024-02-21T23:17:22.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.670+0000 7f86df400640 1 -- 172.21.15.28:0/3077376754 reap_dead start 2024-02-21T23:17:22.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.670+0000 7f86dfe00640 1 -- 172.21.15.28:0/3077376754 shutdown_connections 2024-02-21T23:17:22.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.670+0000 7f86dfe00640 1 -- 172.21.15.28:0/3077376754 >> 172.21.15.28:0/3077376754 conn(0x7f86e00ac370 msgr2=0x7f86e00ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:22.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.670+0000 7f86dfe00640 1 -- 172.21.15.28:0/3077376754 shutdown_connections 2024-02-21T23:17:22.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.670+0000 7f86dfe00640 1 -- 172.21.15.28:0/3077376754 wait complete. 2024-02-21T23:17:22.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.672+0000 7f86dfe00640 1 Processor -- start 2024-02-21T23:17:22.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.672+0000 7f86dfe00640 1 -- start start 2024-02-21T23:17:22.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.673+0000 7f86dfe00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f86c4004800 0x7f86e00c1030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:22.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.673+0000 7f86dfe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86e0152130 0x7f86e00c0310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:22.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.673+0000 7f86dfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86e00c0850 0x7f86e0151690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:22.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.673+0000 7f86dfe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86e00c58f0 con 0x7f86e00c0850 2024-02-21T23:17:22.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.673+0000 7f86dfe00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86e00be0c0 con 0x7f86e0152130 2024-02-21T23:17:22.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.673+0000 7f86dfe00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86e00c5a70 con 0x7f86c4004800 2024-02-21T23:17:22.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.673+0000 7f86df400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86e00c0850 0x7f86e0151690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:22.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.673+0000 7f86df400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86e00c0850 0x7f86e0151690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39290/0 (socket says 172.21.15.28:39290) 2024-02-21T23:17:22.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.673+0000 7f86df400640 1 -- 172.21.15.28:0/3504672610 learned_addr learned my addr 172.21.15.28:0/3504672610 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:22.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.673+0000 7f86dea00640 1 --2- 172.21.15.28:0/3504672610 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f86c4004800 0x7f86e00c1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:22.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.674+0000 7f86d7e00640 1 --2- 172.21.15.28:0/3504672610 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86e0152130 0x7f86e00c0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:22.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.674+0000 7f86df400640 1 -- 172.21.15.28:0/3504672610 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f86c4004800 msgr2=0x7f86e00c1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:22.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.674+0000 7f86df400640 1 --2- 172.21.15.28:0/3504672610 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f86c4004800 0x7f86e00c1030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:22.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.674+0000 7f86df400640 1 -- 172.21.15.28:0/3504672610 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86e0152130 msgr2=0x7f86e00c0310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:22.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.674+0000 7f86df400640 1 --2- 172.21.15.28:0/3504672610 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86e0152130 0x7f86e00c0310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:22.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.674+0000 7f86df400640 1 -- 172.21.15.28:0/3504672610 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86e01eafc0 con 0x7f86e00c0850 2024-02-21T23:17:22.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.674+0000 7f86df400640 1 --2- 172.21.15.28:0/3504672610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86e00c0850 0x7f86e0151690 secure :-1 s=READY pgs=2697 cs=0 l=1 rev1=1 crypto rx=0x7f86e00ca6e0 tx=0x7f86d800f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:22.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.674+0000 7f86d6a00640 1 -- 172.21.15.28:0/3504672610 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86d8016620 con 0x7f86e00c0850 2024-02-21T23:17:22.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.674+0000 7f86d6a00640 1 -- 172.21.15.28:0/3504672610 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86d801b070 con 0x7f86e00c0850 2024-02-21T23:17:22.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.674+0000 7f86d6a00640 1 -- 172.21.15.28:0/3504672610 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86d8016a20 con 0x7f86e00c0850 2024-02-21T23:17:22.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.674+0000 7f86dfe00640 1 -- 172.21.15.28:0/3504672610 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86e01eb2a0 con 0x7f86e00c0850 2024-02-21T23:17:22.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.675+0000 7f86dfe00640 1 -- 172.21.15.28:0/3504672610 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86e00be5c0 con 0x7f86e00c0850 2024-02-21T23:17:22.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.676+0000 7f86dfe00640 1 -- 172.21.15.28:0/3504672610 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86e004f290 con 0x7f86e00c0850 2024-02-21T23:17:22.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.677+0000 7f86d6a00640 1 -- 172.21.15.28:0/3504672610 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f86d8005d10 con 0x7f86e00c0850 2024-02-21T23:17:22.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.678+0000 7f86d6a00640 1 --2- 172.21.15.28:0/3504672610 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f86bc06b500 0x7f86bc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:22.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.678+0000 7f86d6a00640 1 -- 172.21.15.28:0/3504672610 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f86d808b0d0 con 0x7f86e00c0850 2024-02-21T23:17:22.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.679+0000 7f86dea00640 1 --2- 172.21.15.28:0/3504672610 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f86bc06b500 0x7f86bc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:22.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.680+0000 7f86dea00640 1 --2- 172.21.15.28:0/3504672610 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f86bc06b500 0x7f86bc06d9c0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f86d0005a50 tx=0x7f86d0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:22.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.682+0000 7f86d6a00640 1 -- 172.21.15.28:0/3504672610 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f86d8025020 con 0x7f86e00c0850 2024-02-21T23:17:22.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.996+0000 7f86dfe00640 1 -- 172.21.15.28:0/3504672610 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f86e00be790 con 0x7f86e00c0850 2024-02-21T23:17:22.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:22.997+0000 7f86d6a00640 1 -- 172.21.15.28:0/3504672610 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7f86d8059c20 con 0x7f86e00c0850 2024-02-21T23:17:22.984 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:17:22.985 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:17:22.985 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:17:22.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:23.000+0000 7f86dfe00640 1 -- 172.21.15.28:0/3504672610 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f86bc06b500 msgr2=0x7f86bc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:22.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:23.000+0000 7f86dfe00640 1 --2- 172.21.15.28:0/3504672610 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f86bc06b500 0x7f86bc06d9c0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f86d0005a50 tx=0x7f86d0009040 comp rx=0 tx=0).stop 2024-02-21T23:17:22.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:23.000+0000 7f86dfe00640 1 -- 172.21.15.28:0/3504672610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86e00c0850 msgr2=0x7f86e0151690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:22.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:23.000+0000 7f86dfe00640 1 --2- 172.21.15.28:0/3504672610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86e00c0850 0x7f86e0151690 secure :-1 s=READY pgs=2697 cs=0 l=1 rev1=1 crypto rx=0x7f86e00ca6e0 tx=0x7f86d800f520 comp rx=0 tx=0).stop 2024-02-21T23:17:22.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:23.000+0000 7f86dfe00640 1 -- 172.21.15.28:0/3504672610 shutdown_connections 2024-02-21T23:17:22.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:23.000+0000 7f86dfe00640 1 --2- 172.21.15.28:0/3504672610 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f86bc06b500 0x7f86bc06d9c0 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:22.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:23.000+0000 7f86dfe00640 1 --2- 172.21.15.28:0/3504672610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86e00c0850 0x7f86e0151690 unknown :-1 s=CLOSED pgs=2697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:22.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:23.000+0000 7f86dfe00640 1 --2- 172.21.15.28:0/3504672610 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86e0152130 0x7f86e00c0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:22.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:23.000+0000 7f86dfe00640 1 --2- 172.21.15.28:0/3504672610 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f86c4004800 0x7f86e00c1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:22.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:23.000+0000 7f86dfe00640 1 -- 172.21.15.28:0/3504672610 >> 172.21.15.28:0/3504672610 conn(0x7f86e00ac370 msgr2=0x7f86e00ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:22.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:23.000+0000 7f86dfe00640 1 -- 172.21.15.28:0/3504672610 shutdown_connections 2024-02-21T23:17:22.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:23.000+0000 7f86dfe00640 1 -- 172.21.15.28:0/3504672610 wait complete. 2024-02-21T23:17:23.407 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:17:23.407 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:17:24.410 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:17:24.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.642+0000 7fdce5800640 1 Processor -- start 2024-02-21T23:17:24.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.642+0000 7fdce5800640 1 -- start start 2024-02-21T23:17:24.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.644+0000 7fdce5800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdce0153950 0x7fdce00c15d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:24.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.644+0000 7fdce5800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdce0152e70 con 0x7fdce015b6d0 2024-02-21T23:17:24.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.644+0000 7fdce5800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdce0153150 con 0x7fdce00c11f0 2024-02-21T23:17:24.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.644+0000 7fdce5800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdce00a5ec0 con 0x7fdce0153950 2024-02-21T23:17:24.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.644+0000 7fdcdf400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fdce015b6d0 0x7fdce015dac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:52032/0 (socket says 172.21.15.28:52032) 2024-02-21T23:17:24.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.644+0000 7fdcdf400640 1 -- 172.21.15.28:0/2838006226 learned_addr learned my addr 172.21.15.28:0/2838006226 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:17:24.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.645+0000 7fdcde000640 1 --2- 172.21.15.28:0/2838006226 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdce0153950 0x7fdce00c15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:24.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.645+0000 7fdcdd600640 1 -- 172.21.15.28:0/2838006226 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2911166746 0 0) 0x7fdce0152e70 con 0x7fdce015b6d0 2024-02-21T23:17:24.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.645+0000 7fdcdd600640 1 -- 172.21.15.28:0/2838006226 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcc4003520 con 0x7fdce015b6d0 2024-02-21T23:17:24.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.645+0000 7fdcdd600640 1 -- 172.21.15.28:0/2838006226 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1343662119 0 0) 0x7fdcc4003520 con 0x7fdce015b6d0 2024-02-21T23:17:24.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.645+0000 7fdcdd600640 1 -- 172.21.15.28:0/2838006226 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdce0153950 msgr2=0x7fdce00c15d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:24.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.645+0000 7fdcdd600640 1 --2- 172.21.15.28:0/2838006226 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdce0153950 0x7fdce00c15d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:24.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.645+0000 7fdcdd600640 1 -- 172.21.15.28:0/2838006226 >> v1:172.21.15.81:6789/0 conn(0x7fdce00c11f0 legacy=0x7fdce006dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:24.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.645+0000 7fdcdd600640 1 -- 172.21.15.28:0/2838006226 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdce0153430 con 0x7fdce015b6d0 2024-02-21T23:17:24.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.645+0000 7fdcdd600640 1 -- 172.21.15.28:0/2838006226 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fdcd8001670 con 0x7fdce015b6d0 2024-02-21T23:17:24.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.645+0000 7fdcdd600640 1 -- 172.21.15.28:0/2838006226 >> v1:172.21.15.28:6789/0 conn(0x7fdce015b6d0 legacy=0x7fdce015dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:24.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.645+0000 7fdcdd600640 1 --2- 172.21.15.28:0/2838006226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcc40039e0 0x7fdcc4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:24.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.646+0000 7fdcdd600640 1 --2- 172.21.15.28:0/2838006226 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdce0153950 0x7fdcc4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:24.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.646+0000 7fdcdd600640 1 --2- 172.21.15.28:0/2838006226 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdcc4004d40 0x7fdcc40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:24.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.646+0000 7fdcdd600640 1 -- 172.21.15.28:0/2838006226 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdce00a5ec0 con 0x7fdcc40039e0 2024-02-21T23:17:24.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.646+0000 7fdcdd600640 1 -- 172.21.15.28:0/2838006226 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc400baa0 con 0x7fdcc4004d40 2024-02-21T23:17:24.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.646+0000 7fdcdd600640 1 -- 172.21.15.28:0/2838006226 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc400bca0 con 0x7fdce0153950 2024-02-21T23:17:24.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.646+0000 7fdcdea00640 1 --2- 172.21.15.28:0/2838006226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcc40039e0 0x7fdcc4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:24.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.646+0000 7fdcde000640 1 --2- 172.21.15.28:0/2838006226 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdce0153950 0x7fdcc4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:24.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.646+0000 7fdcdf400640 1 --2- 172.21.15.28:0/2838006226 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdcc4004d40 0x7fdcc40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:24.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.646+0000 7fdcdea00640 1 -- 172.21.15.28:0/2838006226 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdce0153950 msgr2=0x7fdcc4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:24.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.646+0000 7fdcdea00640 1 --2- 172.21.15.28:0/2838006226 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdce0153950 0x7fdcc4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:24.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.646+0000 7fdcdea00640 1 -- 172.21.15.28:0/2838006226 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdcc4004d40 msgr2=0x7fdcc40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:24.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.646+0000 7fdcdea00640 1 --2- 172.21.15.28:0/2838006226 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdcc4004d40 0x7fdcc40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:24.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.646+0000 7fdcdea00640 1 -- 172.21.15.28:0/2838006226 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcc4003520 con 0x7fdcc40039e0 2024-02-21T23:17:24.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.646+0000 7fdcdf400640 1 --2- 172.21.15.28:0/2838006226 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdcc4004d40 0x7fdcc40051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:17:24.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.648+0000 7fdcdea00640 1 --2- 172.21.15.28:0/2838006226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcc40039e0 0x7fdcc4003de0 secure :-1 s=READY pgs=2699 cs=0 l=1 rev1=1 crypto rx=0x7fdcd0002010 tx=0x7fdcc4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=267ba09de2edb0cc server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:24.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.648+0000 7fdcdd600640 1 -- 172.21.15.28:0/2838006226 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcd0039070 con 0x7fdcc40039e0 2024-02-21T23:17:24.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.648+0000 7fdcdd600640 1 -- 172.21.15.28:0/2838006226 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcd002c770 con 0x7fdcc40039e0 2024-02-21T23:17:24.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.648+0000 7fdcdd600640 1 -- 172.21.15.28:0/2838006226 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcd0001be0 con 0x7fdcc40039e0 2024-02-21T23:17:24.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.648+0000 7fdce5800640 1 -- 172.21.15.28:0/2838006226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcc40039e0 msgr2=0x7fdcc4003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:24.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.648+0000 7fdce5800640 1 --2- 172.21.15.28:0/2838006226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcc40039e0 0x7fdcc4003de0 secure :-1 s=READY pgs=2699 cs=0 l=1 rev1=1 crypto rx=0x7fdcd0002010 tx=0x7fdcc4005830 comp rx=0 tx=0).stop 2024-02-21T23:17:24.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.648+0000 7fdcdf400640 1 -- 172.21.15.28:0/2838006226 reap_dead start 2024-02-21T23:17:24.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.649+0000 7fdce5800640 1 -- 172.21.15.28:0/2838006226 shutdown_connections 2024-02-21T23:17:24.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.649+0000 7fdce5800640 1 -- 172.21.15.28:0/2838006226 >> 172.21.15.28:0/2838006226 conn(0x7fdce00c67f0 msgr2=0x7fdce00c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:24.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.649+0000 7fdce5800640 1 -- 172.21.15.28:0/2838006226 shutdown_connections 2024-02-21T23:17:24.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.649+0000 7fdce5800640 1 -- 172.21.15.28:0/2838006226 wait complete. 2024-02-21T23:17:24.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.650+0000 7fdce5800640 1 Processor -- start 2024-02-21T23:17:24.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.651+0000 7fdce5800640 1 -- start start 2024-02-21T23:17:24.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.651+0000 7fdce5800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdcc40039e0 0x7fdce0145330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:24.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.651+0000 7fdce5800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdce0144bb0 0x7fdce01493d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:24.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.651+0000 7fdce5800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdce0142720 0x7fdce0142bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:24.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.651+0000 7fdce5800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdce00a6110 con 0x7fdce0144bb0 2024-02-21T23:17:24.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.651+0000 7fdce5800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdce00c4c00 con 0x7fdcc40039e0 2024-02-21T23:17:24.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.651+0000 7fdce5800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdce00c0d90 con 0x7fdce0142720 2024-02-21T23:17:24.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.652+0000 7fdcde000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdce0144bb0 0x7fdce01493d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:24.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.652+0000 7fdcde000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdce0144bb0 0x7fdce01493d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39308/0 (socket says 172.21.15.28:39308) 2024-02-21T23:17:24.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.652+0000 7fdcde000640 1 -- 172.21.15.28:0/2742400273 learned_addr learned my addr 172.21.15.28:0/2742400273 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:24.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.652+0000 7fdcdf400640 1 --2- 172.21.15.28:0/2742400273 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdce0142720 0x7fdce0142bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:24.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.652+0000 7fdcdea00640 1 --2- 172.21.15.28:0/2742400273 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdcc40039e0 0x7fdce0145330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:24.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.652+0000 7fdcde000640 1 -- 172.21.15.28:0/2742400273 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdce0142720 msgr2=0x7fdce0142bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:24.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.652+0000 7fdcde000640 1 --2- 172.21.15.28:0/2742400273 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdce0142720 0x7fdce0142bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:24.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.652+0000 7fdcde000640 1 -- 172.21.15.28:0/2742400273 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdcc40039e0 msgr2=0x7fdce0145330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:24.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.652+0000 7fdcde000640 1 --2- 172.21.15.28:0/2742400273 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdcc40039e0 0x7fdce0145330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:24.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.652+0000 7fdcde000640 1 -- 172.21.15.28:0/2742400273 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdce01471e0 con 0x7fdce0144bb0 2024-02-21T23:17:24.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.652+0000 7fdcdea00640 1 --2- 172.21.15.28:0/2742400273 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdcc40039e0 0x7fdce0145330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:17:24.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.653+0000 7fdcde000640 1 --2- 172.21.15.28:0/2742400273 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdce0144bb0 0x7fdce01493d0 secure :-1 s=READY pgs=2700 cs=0 l=1 rev1=1 crypto rx=0x7fdcc8011600 tx=0x7fdcc8011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:24.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.653+0000 7fdcd6a00640 1 -- 172.21.15.28:0/2742400273 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc800f920 con 0x7fdce0144bb0 2024-02-21T23:17:24.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.653+0000 7fdcd6a00640 1 -- 172.21.15.28:0/2742400273 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcc800fac0 con 0x7fdce0144bb0 2024-02-21T23:17:24.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.653+0000 7fdcd6a00640 1 -- 172.21.15.28:0/2742400273 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc800fe70 con 0x7fdce0144bb0 2024-02-21T23:17:24.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.653+0000 7fdce5800640 1 -- 172.21.15.28:0/2742400273 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdce01474c0 con 0x7fdce0144bb0 2024-02-21T23:17:24.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.653+0000 7fdce5800640 1 -- 172.21.15.28:0/2742400273 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdce00aa730 con 0x7fdce0144bb0 2024-02-21T23:17:24.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.654+0000 7fdce5800640 1 -- 172.21.15.28:0/2742400273 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdce0067f30 con 0x7fdce0144bb0 2024-02-21T23:17:24.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.655+0000 7fdcd6a00640 1 -- 172.21.15.28:0/2742400273 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdcc8015460 con 0x7fdce0144bb0 2024-02-21T23:17:24.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.657+0000 7fdcd6a00640 1 --2- 172.21.15.28:0/2742400273 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdcbc06b500 0x7fdcbc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:24.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.657+0000 7fdcd6a00640 1 -- 172.21.15.28:0/2742400273 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fdcc808ad00 con 0x7fdce0144bb0 2024-02-21T23:17:24.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.657+0000 7fdcdea00640 1 --2- 172.21.15.28:0/2742400273 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdcbc06b500 0x7fdcbc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:24.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.658+0000 7fdcdea00640 1 --2- 172.21.15.28:0/2742400273 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdcbc06b500 0x7fdcbc06d9c0 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7fdcd0036000 tx=0x7fdcd0030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:24.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.662+0000 7fdcd6a00640 1 -- 172.21.15.28:0/2742400273 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fdcc8019030 con 0x7fdce0144bb0 2024-02-21T23:17:24.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.897+0000 7fdce5800640 1 -- 172.21.15.28:0/2742400273 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdce006f850 con 0x7fdce0144bb0 2024-02-21T23:17:24.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.898+0000 7fdcd6a00640 1 -- 172.21.15.28:0/2742400273 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7fdcc80598e0 con 0x7fdce0144bb0 2024-02-21T23:17:24.885 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:17:24.885 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:17:24.885 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:17:24.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.900+0000 7fdce5800640 1 -- 172.21.15.28:0/2742400273 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdcbc06b500 msgr2=0x7fdcbc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:24.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.900+0000 7fdce5800640 1 --2- 172.21.15.28:0/2742400273 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdcbc06b500 0x7fdcbc06d9c0 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7fdcd0036000 tx=0x7fdcd0030000 comp rx=0 tx=0).stop 2024-02-21T23:17:24.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.900+0000 7fdce5800640 1 -- 172.21.15.28:0/2742400273 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdce0144bb0 msgr2=0x7fdce01493d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:24.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.900+0000 7fdce5800640 1 --2- 172.21.15.28:0/2742400273 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdce0144bb0 0x7fdce01493d0 secure :-1 s=READY pgs=2700 cs=0 l=1 rev1=1 crypto rx=0x7fdcc8011600 tx=0x7fdcc8011ac0 comp rx=0 tx=0).stop 2024-02-21T23:17:24.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.900+0000 7fdce5800640 1 -- 172.21.15.28:0/2742400273 shutdown_connections 2024-02-21T23:17:24.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.900+0000 7fdce5800640 1 --2- 172.21.15.28:0/2742400273 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fdcbc06b500 0x7fdcbc06d9c0 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:24.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.900+0000 7fdce5800640 1 --2- 172.21.15.28:0/2742400273 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fdce0142720 0x7fdce0142bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:24.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.900+0000 7fdce5800640 1 --2- 172.21.15.28:0/2742400273 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdce0144bb0 0x7fdce01493d0 unknown :-1 s=CLOSED pgs=2700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:24.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.900+0000 7fdce5800640 1 --2- 172.21.15.28:0/2742400273 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdcc40039e0 0x7fdce0145330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:24.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.900+0000 7fdce5800640 1 -- 172.21.15.28:0/2742400273 >> 172.21.15.28:0/2742400273 conn(0x7fdce00c67f0 msgr2=0x7fdce01441d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:24.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.901+0000 7fdce5800640 1 -- 172.21.15.28:0/2742400273 shutdown_connections 2024-02-21T23:17:24.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:24.901+0000 7fdce5800640 1 -- 172.21.15.28:0/2742400273 wait complete. 2024-02-21T23:17:25.309 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:17:25.309 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:17:26.311 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:17:26.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.572+0000 7f032be00640 1 Processor -- start 2024-02-21T23:17:26.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.572+0000 7f032be00640 1 -- start start 2024-02-21T23:17:26.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.573+0000 7f032be00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f032c155960 0x7f032c1608a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:26.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.573+0000 7f032be00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f032c0c74c0 con 0x7f032c0aff90 2024-02-21T23:17:26.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.573+0000 7f032be00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f032c0c77a0 con 0x7f032c06dbf0 2024-02-21T23:17:26.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.573+0000 7f032be00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f032c0c7270 con 0x7f032c155960 2024-02-21T23:17:26.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.574+0000 7f032aa00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f032c0aff90 0x7f032c0c51e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:33576/0 (socket says 172.21.15.28:33576) 2024-02-21T23:17:26.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.574+0000 7f032aa00640 1 -- 172.21.15.28:0/1824655361 learned_addr learned my addr 172.21.15.28:0/1824655361 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:17:26.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.574+0000 7f032b400640 1 --2- 172.21.15.28:0/1824655361 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f032c155960 0x7f032c1608a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:26.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.574+0000 7f0329600640 1 -- 172.21.15.28:0/1824655361 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3273141045 0 0) 0x7f032c0c74c0 con 0x7f032c0aff90 2024-02-21T23:17:26.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.574+0000 7f0329600640 1 -- 172.21.15.28:0/1824655361 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0310003520 con 0x7f032c0aff90 2024-02-21T23:17:26.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.575+0000 7f0329600640 1 -- 172.21.15.28:0/1824655361 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3933442596 0 0) 0x7f0310003520 con 0x7f032c0aff90 2024-02-21T23:17:26.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.575+0000 7f0329600640 1 -- 172.21.15.28:0/1824655361 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f032c155960 msgr2=0x7f032c1608a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:26.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.575+0000 7f0329600640 1 --2- 172.21.15.28:0/1824655361 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f032c155960 0x7f032c1608a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:26.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.575+0000 7f0329600640 1 -- 172.21.15.28:0/1824655361 >> v1:172.21.15.81:6789/0 conn(0x7f032c06dbf0 legacy=0x7f032c158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:26.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.575+0000 7f0329600640 1 -- 172.21.15.28:0/1824655361 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f032c0c7a80 con 0x7f032c0aff90 2024-02-21T23:17:26.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.575+0000 7f0329600640 1 -- 172.21.15.28:0/1824655361 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f031c001670 con 0x7f032c0aff90 2024-02-21T23:17:26.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.575+0000 7f0329600640 1 -- 172.21.15.28:0/1824655361 >> v1:172.21.15.28:6789/0 conn(0x7f032c0aff90 legacy=0x7f032c0c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:26.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.575+0000 7f0329600640 1 --2- 172.21.15.28:0/1824655361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03100039e0 0x7f0310003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:26.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.575+0000 7f0329600640 1 --2- 172.21.15.28:0/1824655361 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f032c155960 0x7f0310004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:26.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.575+0000 7f0329600640 1 --2- 172.21.15.28:0/1824655361 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0310004d40 0x7f03100051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:26.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.575+0000 7f0329600640 1 -- 172.21.15.28:0/1824655361 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f032c0c7270 con 0x7f03100039e0 2024-02-21T23:17:26.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.575+0000 7f0329600640 1 -- 172.21.15.28:0/1824655361 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f031000baa0 con 0x7f0310004d40 2024-02-21T23:17:26.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.575+0000 7f0329600640 1 -- 172.21.15.28:0/1824655361 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f031000bca0 con 0x7f032c155960 2024-02-21T23:17:26.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.575+0000 7f032aa00640 1 --2- 172.21.15.28:0/1824655361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03100039e0 0x7f0310003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:26.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.576+0000 7f032a000640 1 --2- 172.21.15.28:0/1824655361 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f032c155960 0x7f0310004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:26.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.576+0000 7f032b400640 1 --2- 172.21.15.28:0/1824655361 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0310004d40 0x7f03100051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:26.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.576+0000 7f032aa00640 1 -- 172.21.15.28:0/1824655361 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f032c155960 msgr2=0x7f0310004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:26.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.576+0000 7f032aa00640 1 --2- 172.21.15.28:0/1824655361 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f032c155960 0x7f0310004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:26.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.576+0000 7f032aa00640 1 -- 172.21.15.28:0/1824655361 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0310004d40 msgr2=0x7f03100051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:26.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.576+0000 7f032aa00640 1 --2- 172.21.15.28:0/1824655361 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0310004d40 0x7f03100051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:26.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.576+0000 7f032aa00640 1 -- 172.21.15.28:0/1824655361 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0310003520 con 0x7f03100039e0 2024-02-21T23:17:26.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.576+0000 7f032a000640 1 --2- 172.21.15.28:0/1824655361 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f032c155960 0x7f0310004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:17:26.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.577+0000 7f032aa00640 1 --2- 172.21.15.28:0/1824655361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03100039e0 0x7f0310003e00 secure :-1 s=READY pgs=2702 cs=0 l=1 rev1=1 crypto rx=0x7f031c00c160 tx=0x7f0310005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88995c3bf689b501 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:26.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.577+0000 7f0329600640 1 -- 172.21.15.28:0/1824655361 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f031c007c60 con 0x7f03100039e0 2024-02-21T23:17:26.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.577+0000 7f0329600640 1 -- 172.21.15.28:0/1824655361 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f031c030070 con 0x7f03100039e0 2024-02-21T23:17:26.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.577+0000 7f0329600640 1 -- 172.21.15.28:0/1824655361 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f031c005d50 con 0x7f03100039e0 2024-02-21T23:17:26.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.578+0000 7f032be00640 1 -- 172.21.15.28:0/1824655361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03100039e0 msgr2=0x7f0310003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:26.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.578+0000 7f032be00640 1 --2- 172.21.15.28:0/1824655361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03100039e0 0x7f0310003e00 secure :-1 s=READY pgs=2702 cs=0 l=1 rev1=1 crypto rx=0x7f031c00c160 tx=0x7f0310005830 comp rx=0 tx=0).stop 2024-02-21T23:17:26.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.578+0000 7f032b400640 1 -- 172.21.15.28:0/1824655361 reap_dead start 2024-02-21T23:17:26.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.578+0000 7f032be00640 1 -- 172.21.15.28:0/1824655361 shutdown_connections 2024-02-21T23:17:26.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.578+0000 7f032be00640 1 -- 172.21.15.28:0/1824655361 >> 172.21.15.28:0/1824655361 conn(0x7f032c0ac370 msgr2=0x7f032c0ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:26.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.578+0000 7f032be00640 1 -- 172.21.15.28:0/1824655361 shutdown_connections 2024-02-21T23:17:26.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.578+0000 7f032be00640 1 -- 172.21.15.28:0/1824655361 wait complete. 2024-02-21T23:17:26.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.580+0000 7f032be00640 1 Processor -- start 2024-02-21T23:17:26.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.580+0000 7f032be00640 1 -- start start 2024-02-21T23:17:26.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.581+0000 7f032be00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f03100039e0 0x7f032c0c3500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:26.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.581+0000 7f032be00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f032c0bed60 0x7f032c1ee410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:26.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.581+0000 7f032be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f032c1ee950 0x7f032c151690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:26.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.581+0000 7f032be00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f032c0c58f0 con 0x7f032c1ee950 2024-02-21T23:17:26.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.581+0000 7f032be00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f032c158630 con 0x7f03100039e0 2024-02-21T23:17:26.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.581+0000 7f032be00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f032c0c5a70 con 0x7f032c0bed60 2024-02-21T23:17:26.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.581+0000 7f032b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f032c1ee950 0x7f032c151690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:26.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.581+0000 7f032b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f032c1ee950 0x7f032c151690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41152/0 (socket says 172.21.15.28:41152) 2024-02-21T23:17:26.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.581+0000 7f032b400640 1 -- 172.21.15.28:0/2979089067 learned_addr learned my addr 172.21.15.28:0/2979089067 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:26.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.581+0000 7f032aa00640 1 --2- 172.21.15.28:0/2979089067 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f03100039e0 0x7f032c0c3500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:26.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.581+0000 7f032a000640 1 --2- 172.21.15.28:0/2979089067 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f032c0bed60 0x7f032c1ee410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:26.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.581+0000 7f032b400640 1 -- 172.21.15.28:0/2979089067 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f032c0bed60 msgr2=0x7f032c1ee410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:26.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.581+0000 7f032b400640 1 --2- 172.21.15.28:0/2979089067 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f032c0bed60 0x7f032c1ee410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:26.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.581+0000 7f032b400640 1 -- 172.21.15.28:0/2979089067 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f03100039e0 msgr2=0x7f032c0c3500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:26.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.581+0000 7f032b400640 1 --2- 172.21.15.28:0/2979089067 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f03100039e0 0x7f032c0c3500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:26.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.581+0000 7f032b400640 1 -- 172.21.15.28:0/2979089067 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f032c1eee90 con 0x7f032c1ee950 2024-02-21T23:17:26.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.582+0000 7f032b400640 1 --2- 172.21.15.28:0/2979089067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f032c1ee950 0x7f032c151690 secure :-1 s=READY pgs=2703 cs=0 l=1 rev1=1 crypto rx=0x7f0324009fd0 tx=0x7f032400f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:26.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.582+0000 7f0322a00640 1 -- 172.21.15.28:0/2979089067 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f032401c070 con 0x7f032c1ee950 2024-02-21T23:17:26.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.582+0000 7f0322a00640 1 -- 172.21.15.28:0/2979089067 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0324017400 con 0x7f032c1ee950 2024-02-21T23:17:26.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.582+0000 7f0322a00640 1 -- 172.21.15.28:0/2979089067 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03240177f0 con 0x7f032c1ee950 2024-02-21T23:17:26.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.582+0000 7f032be00640 1 -- 172.21.15.28:0/2979089067 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f032c0c05f0 con 0x7f032c1ee950 2024-02-21T23:17:26.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.582+0000 7f032be00640 1 -- 172.21.15.28:0/2979089067 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f032c0c0bb0 con 0x7f032c1ee950 2024-02-21T23:17:26.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.583+0000 7f032be00640 1 -- 172.21.15.28:0/2979089067 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f032c04f290 con 0x7f032c1ee950 2024-02-21T23:17:26.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.585+0000 7f0322a00640 1 -- 172.21.15.28:0/2979089067 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f032401f070 con 0x7f032c1ee950 2024-02-21T23:17:26.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.586+0000 7f0322a00640 1 --2- 172.21.15.28:0/2979089067 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f030c06b4d0 0x7f030c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:26.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.587+0000 7f0322a00640 1 -- 172.21.15.28:0/2979089067 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f032408a330 con 0x7f032c1ee950 2024-02-21T23:17:26.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.587+0000 7f032aa00640 1 --2- 172.21.15.28:0/2979089067 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f030c06b4d0 0x7f030c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:26.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.588+0000 7f032aa00640 1 --2- 172.21.15.28:0/2979089067 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f030c06b4d0 0x7f030c06d990 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f03100057e0 tx=0x7f031c009030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:26.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.591+0000 7f0322a00640 1 -- 172.21.15.28:0/2979089067 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f0324012070 con 0x7f032c1ee950 2024-02-21T23:17:26.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.882+0000 7f032be00640 1 -- 172.21.15.28:0/2979089067 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f032c0af490 con 0x7f032c1ee950 2024-02-21T23:17:26.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.883+0000 7f0322a00640 1 -- 172.21.15.28:0/2979089067 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7f0324058e80 con 0x7f032c1ee950 2024-02-21T23:17:26.870 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:17:26.870 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:17:26.871 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:17:26.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.886+0000 7f032be00640 1 -- 172.21.15.28:0/2979089067 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f030c06b4d0 msgr2=0x7f030c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:26.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.886+0000 7f032be00640 1 --2- 172.21.15.28:0/2979089067 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f030c06b4d0 0x7f030c06d990 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f03100057e0 tx=0x7f031c009030 comp rx=0 tx=0).stop 2024-02-21T23:17:26.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.886+0000 7f032be00640 1 -- 172.21.15.28:0/2979089067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f032c1ee950 msgr2=0x7f032c151690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:26.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.886+0000 7f032be00640 1 --2- 172.21.15.28:0/2979089067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f032c1ee950 0x7f032c151690 secure :-1 s=READY pgs=2703 cs=0 l=1 rev1=1 crypto rx=0x7f0324009fd0 tx=0x7f032400f4a0 comp rx=0 tx=0).stop 2024-02-21T23:17:26.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.886+0000 7f032be00640 1 -- 172.21.15.28:0/2979089067 shutdown_connections 2024-02-21T23:17:26.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.886+0000 7f032be00640 1 --2- 172.21.15.28:0/2979089067 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f030c06b4d0 0x7f030c06d990 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:26.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.886+0000 7f032be00640 1 --2- 172.21.15.28:0/2979089067 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f032c0bed60 0x7f032c1ee410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:26.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.886+0000 7f032be00640 1 --2- 172.21.15.28:0/2979089067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f032c1ee950 0x7f032c151690 unknown :-1 s=CLOSED pgs=2703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:26.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.886+0000 7f032be00640 1 --2- 172.21.15.28:0/2979089067 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f03100039e0 0x7f032c0c3500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:26.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.886+0000 7f032be00640 1 -- 172.21.15.28:0/2979089067 >> 172.21.15.28:0/2979089067 conn(0x7f032c0ac370 msgr2=0x7f032c0ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:26.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.886+0000 7f032be00640 1 -- 172.21.15.28:0/2979089067 shutdown_connections 2024-02-21T23:17:26.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:26.886+0000 7f032be00640 1 -- 172.21.15.28:0/2979089067 wait complete. 2024-02-21T23:17:27.296 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:17:27.296 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:17:28.297 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:17:28.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.555+0000 7f2189800640 1 Processor -- start 2024-02-21T23:17:28.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.555+0000 7f2189800640 1 -- start start 2024-02-21T23:17:28.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.557+0000 7f2189800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f218406dbf0 0x7f2184158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:28.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.557+0000 7f2189800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2184155960 0x7f2184159000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:28.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.557+0000 7f2189800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21840c74c0 con 0x7f21840aff90 2024-02-21T23:17:28.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.557+0000 7f2189800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21840c7270 con 0x7f218406dbf0 2024-02-21T23:17:28.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.557+0000 7f2189800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2184158630 con 0x7f2184155960 2024-02-21T23:17:28.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.557+0000 7f217be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f218406dbf0 0x7f2184158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:28.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.557+0000 7f217be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f218406dbf0 0x7f2184158ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41158/0 (socket says 172.21.15.28:41158) 2024-02-21T23:17:28.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.557+0000 7f217be00640 1 -- 172.21.15.28:0/583763666 learned_addr learned my addr 172.21.15.28:0/583763666 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:28.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.557+0000 7f2188e00640 1 --2- 172.21.15.28:0/583763666 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2184155960 0x7f2184159000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:28.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.558+0000 7f217be00640 1 -- 172.21.15.28:0/583763666 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2184155960 msgr2=0x7f2184159000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:28.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.558+0000 7f217be00640 1 --2- 172.21.15.28:0/583763666 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2184155960 0x7f2184159000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:28.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.558+0000 7f217be00640 1 -- 172.21.15.28:0/583763666 >> v1:172.21.15.81:6789/0 conn(0x7f21840aff90 legacy=0x7f21840c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:28.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.558+0000 7f217be00640 1 -- 172.21.15.28:0/583763666 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21840c77a0 con 0x7f218406dbf0 2024-02-21T23:17:28.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.559+0000 7f217be00640 1 --2- 172.21.15.28:0/583763666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f218406dbf0 0x7f2184158ac0 secure :-1 s=READY pgs=2704 cs=0 l=1 rev1=1 crypto rx=0x7f216c009870 tx=0x7f216c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9cc504ec22a6f9c server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:28.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.559+0000 7f2183400640 1 -- 172.21.15.28:0/583763666 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f216c034070 con 0x7f218406dbf0 2024-02-21T23:17:28.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.559+0000 7f2183400640 1 -- 172.21.15.28:0/583763666 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f216c030840 con 0x7f218406dbf0 2024-02-21T23:17:28.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.559+0000 7f2183400640 1 -- 172.21.15.28:0/583763666 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f216c030c10 con 0x7f218406dbf0 2024-02-21T23:17:28.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.560+0000 7f2189800640 1 -- 172.21.15.28:0/583763666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f218406dbf0 msgr2=0x7f2184158ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:28.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.560+0000 7f2189800640 1 --2- 172.21.15.28:0/583763666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f218406dbf0 0x7f2184158ac0 secure :-1 s=READY pgs=2704 cs=0 l=1 rev1=1 crypto rx=0x7f216c009870 tx=0x7f216c02b980 comp rx=0 tx=0).stop 2024-02-21T23:17:28.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.560+0000 7f2189800640 1 -- 172.21.15.28:0/583763666 shutdown_connections 2024-02-21T23:17:28.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.560+0000 7f2189800640 1 --2- 172.21.15.28:0/583763666 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2184155960 0x7f2184159000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:28.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.560+0000 7f2189800640 1 --2- 172.21.15.28:0/583763666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f218406dbf0 0x7f2184158ac0 unknown :-1 s=CLOSED pgs=2704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:28.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.560+0000 7f2189800640 1 -- 172.21.15.28:0/583763666 >> 172.21.15.28:0/583763666 conn(0x7f21840ac370 msgr2=0x7f21840ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:28.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.560+0000 7f2189800640 1 -- 172.21.15.28:0/583763666 shutdown_connections 2024-02-21T23:17:28.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.560+0000 7f2189800640 1 -- 172.21.15.28:0/583763666 wait complete. 2024-02-21T23:17:28.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.563+0000 7f2189800640 1 Processor -- start 2024-02-21T23:17:28.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.563+0000 7f2189800640 1 -- start start 2024-02-21T23:17:28.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.564+0000 7f2189800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f218406dbf0 0x7f21840c3db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:28.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.564+0000 7f2189800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f21840aff90 0x7f21840c0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:28.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.565+0000 7f2189800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2184155960 0x7f21841ee020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:28.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.565+0000 7f2189800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21840c58f0 con 0x7f218406dbf0 2024-02-21T23:17:28.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.565+0000 7f2189800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21840be0c0 con 0x7f2184155960 2024-02-21T23:17:28.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.565+0000 7f2189800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21840c5a70 con 0x7f21840aff90 2024-02-21T23:17:28.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.565+0000 7f2183e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f218406dbf0 0x7f21840c3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:28.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.565+0000 7f2183e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f218406dbf0 0x7f21840c3db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41168/0 (socket says 172.21.15.28:41168) 2024-02-21T23:17:28.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.565+0000 7f2183e00640 1 -- 172.21.15.28:0/2958439075 learned_addr learned my addr 172.21.15.28:0/2958439075 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:28.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.565+0000 7f217be00640 1 --2- 172.21.15.28:0/2958439075 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f21840aff90 0x7f21840c0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:28.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.565+0000 7f2188e00640 1 --2- 172.21.15.28:0/2958439075 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2184155960 0x7f21841ee020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:28.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.565+0000 7f2183e00640 1 -- 172.21.15.28:0/2958439075 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f21840aff90 msgr2=0x7f21840c0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:28.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.565+0000 7f2183e00640 1 --2- 172.21.15.28:0/2958439075 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f21840aff90 0x7f21840c0cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:28.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.565+0000 7f2183e00640 1 -- 172.21.15.28:0/2958439075 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2184155960 msgr2=0x7f21841ee020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:28.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.565+0000 7f2183e00640 1 --2- 172.21.15.28:0/2958439075 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2184155960 0x7f21841ee020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:28.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.565+0000 7f2183e00640 1 -- 172.21.15.28:0/2958439075 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21840c1210 con 0x7f218406dbf0 2024-02-21T23:17:28.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.565+0000 7f2188e00640 1 --2- 172.21.15.28:0/2958439075 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2184155960 0x7f21841ee020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:17:28.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.566+0000 7f2183e00640 1 --2- 172.21.15.28:0/2958439075 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f218406dbf0 0x7f21840c3db0 secure :-1 s=READY pgs=2705 cs=0 l=1 rev1=1 crypto rx=0x7f217400c450 tx=0x7f217400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:28.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.566+0000 7f2180c00640 1 -- 172.21.15.28:0/2958439075 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2174017070 con 0x7f218406dbf0 2024-02-21T23:17:28.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.566+0000 7f2180c00640 1 -- 172.21.15.28:0/2958439075 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21740052d0 con 0x7f218406dbf0 2024-02-21T23:17:28.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.566+0000 7f2189800640 1 -- 172.21.15.28:0/2958439075 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21841ee860 con 0x7f218406dbf0 2024-02-21T23:17:28.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.566+0000 7f2180c00640 1 -- 172.21.15.28:0/2958439075 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2174003830 con 0x7f218406dbf0 2024-02-21T23:17:28.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.566+0000 7f2189800640 1 -- 172.21.15.28:0/2958439075 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21841eeb00 con 0x7f218406dbf0 2024-02-21T23:17:28.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.568+0000 7f2189800640 1 -- 172.21.15.28:0/2958439075 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f218404f290 con 0x7f218406dbf0 2024-02-21T23:17:28.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.569+0000 7f2180c00640 1 -- 172.21.15.28:0/2958439075 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f217401a070 con 0x7f218406dbf0 2024-02-21T23:17:28.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.570+0000 7f2180c00640 1 --2- 172.21.15.28:0/2958439075 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f216006b4d0 0x7f216006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:28.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.570+0000 7f2180c00640 1 -- 172.21.15.28:0/2958439075 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f21740897f0 con 0x7f218406dbf0 2024-02-21T23:17:28.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.570+0000 7f217be00640 1 --2- 172.21.15.28:0/2958439075 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f216006b4d0 0x7f216006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:28.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.572+0000 7f217be00640 1 --2- 172.21.15.28:0/2958439075 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f216006b4d0 0x7f216006d990 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f216c036000 tx=0x7f216c03b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:28.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.575+0000 7f2180c00640 1 -- 172.21.15.28:0/2958439075 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f21840c1210 con 0x7f218406dbf0 2024-02-21T23:17:28.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.812+0000 7f2189800640 1 -- 172.21.15.28:0/2958439075 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f21840af490 con 0x7f218406dbf0 2024-02-21T23:17:28.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.813+0000 7f2180c00640 1 -- 172.21.15.28:0/2958439075 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7f2174058470 con 0x7f218406dbf0 2024-02-21T23:17:28.925 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:17:28.925 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:17:28.926 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:17:28.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.816+0000 7f2189800640 1 -- 172.21.15.28:0/2958439075 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f216006b4d0 msgr2=0x7f216006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:28.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.816+0000 7f2189800640 1 --2- 172.21.15.28:0/2958439075 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f216006b4d0 0x7f216006d990 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f216c036000 tx=0x7f216c03b040 comp rx=0 tx=0).stop 2024-02-21T23:17:28.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.816+0000 7f2189800640 1 -- 172.21.15.28:0/2958439075 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f218406dbf0 msgr2=0x7f21840c3db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:28.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.816+0000 7f2189800640 1 --2- 172.21.15.28:0/2958439075 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f218406dbf0 0x7f21840c3db0 secure :-1 s=READY pgs=2705 cs=0 l=1 rev1=1 crypto rx=0x7f217400c450 tx=0x7f217400c910 comp rx=0 tx=0).stop 2024-02-21T23:17:28.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.816+0000 7f2189800640 1 -- 172.21.15.28:0/2958439075 shutdown_connections 2024-02-21T23:17:28.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.816+0000 7f2189800640 1 --2- 172.21.15.28:0/2958439075 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f216006b4d0 0x7f216006d990 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:28.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.816+0000 7f2189800640 1 --2- 172.21.15.28:0/2958439075 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f21840aff90 0x7f21840c0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:28.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.816+0000 7f2189800640 1 --2- 172.21.15.28:0/2958439075 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2184155960 0x7f21841ee020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:28.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.816+0000 7f2189800640 1 --2- 172.21.15.28:0/2958439075 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f218406dbf0 0x7f21840c3db0 unknown :-1 s=CLOSED pgs=2705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:28.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.816+0000 7f2189800640 1 -- 172.21.15.28:0/2958439075 >> 172.21.15.28:0/2958439075 conn(0x7f21840ac370 msgr2=0x7f21840ac750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:28.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.816+0000 7f2189800640 1 -- 172.21.15.28:0/2958439075 shutdown_connections 2024-02-21T23:17:28.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:28.816+0000 7f2189800640 1 -- 172.21.15.28:0/2958439075 wait complete. 2024-02-21T23:17:29.331 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:17:29.331 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:17:30.332 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:17:30.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.572+0000 7fc302e00640 1 Processor -- start 2024-02-21T23:17:30.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.572+0000 7fc302e00640 1 -- start start 2024-02-21T23:17:30.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.574+0000 7fc302e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc2fc0c11f0 0x7fc2fc153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:30.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.574+0000 7fc302e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fc2fc0c15d0 0x7fc2fc06dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:30.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.574+0000 7fc302e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2fc0c1a30 con 0x7fc2fc156350 2024-02-21T23:17:30.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.574+0000 7fc302e00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2fc0a5ec0 con 0x7fc2fc0c15d0 2024-02-21T23:17:30.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.574+0000 7fc302e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2fc152e70 con 0x7fc2fc0c11f0 2024-02-21T23:17:30.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.574+0000 7fc2fbe00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc2fc0c11f0 0x7fc2fc153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:30.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.574+0000 7fc2f3400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fc2fc0c15d0 0x7fc2fc06dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:30.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.575+0000 7fc2fbe00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc2fc0c11f0 0x7fc2fc153950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3301/0 says I am v2:172.21.15.28:56738/0 (socket says 172.21.15.28:56738) 2024-02-21T23:17:30.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.575+0000 7fc2fbe00640 1 -- 172.21.15.28:0/954638778 learned_addr learned my addr 172.21.15.28:0/954638778 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:30.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.575+0000 7fc2f3400640 1 -- 172.21.15.28:0/954638778 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc2fc0c11f0 msgr2=0x7fc2fc153950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:30.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.575+0000 7fc2f3400640 1 --2- 172.21.15.28:0/954638778 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc2fc0c11f0 0x7fc2fc153950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:30.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.575+0000 7fc2f3400640 1 -- 172.21.15.28:0/954638778 >> v1:172.21.15.81:6789/0 conn(0x7fc2fc156350 legacy=0x7fc2fc1567a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:30.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.575+0000 7fc2f3400640 1 -- 172.21.15.28:0/954638778 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2fc153150 con 0x7fc2fc0c15d0 2024-02-21T23:17:30.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.575+0000 7fc2fbe00640 1 --2- 172.21.15.28:0/954638778 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc2fc0c11f0 0x7fc2fc153950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-21T23:17:30.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.577+0000 7fc2f3400640 1 --2- 172.21.15.28:0/954638778 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2fc0c15d0 0x7fc2fc06dce0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fc2e40097a0 tx=0x7fc2e402b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=74a85775dd6c6a04 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:30.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.577+0000 7fc2fb400640 1 -- 172.21.15.28:0/954638778 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2e4039070 con 0x7fc2fc0c15d0 2024-02-21T23:17:30.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.577+0000 7fc2fb400640 1 -- 172.21.15.28:0/954638778 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2e40316f0 con 0x7fc2fc0c15d0 2024-02-21T23:17:30.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.577+0000 7fc2fb400640 1 -- 172.21.15.28:0/954638778 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2e4031ac0 con 0x7fc2fc0c15d0 2024-02-21T23:17:30.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.578+0000 7fc302e00640 1 -- 172.21.15.28:0/954638778 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2fc0c15d0 msgr2=0x7fc2fc06dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:30.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.578+0000 7fc302e00640 1 --2- 172.21.15.28:0/954638778 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2fc0c15d0 0x7fc2fc06dce0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fc2e40097a0 tx=0x7fc2e402b790 comp rx=0 tx=0).stop 2024-02-21T23:17:30.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.578+0000 7fc302e00640 1 -- 172.21.15.28:0/954638778 shutdown_connections 2024-02-21T23:17:30.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.578+0000 7fc302e00640 1 --2- 172.21.15.28:0/954638778 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2fc0c15d0 0x7fc2fc06dce0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:30.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.578+0000 7fc302e00640 1 --2- 172.21.15.28:0/954638778 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc2fc0c11f0 0x7fc2fc153950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:30.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.578+0000 7fc302e00640 1 -- 172.21.15.28:0/954638778 >> 172.21.15.28:0/954638778 conn(0x7fc2fc0c67f0 msgr2=0x7fc2fc0c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:30.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.579+0000 7fc302e00640 1 -- 172.21.15.28:0/954638778 shutdown_connections 2024-02-21T23:17:30.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.579+0000 7fc302e00640 1 -- 172.21.15.28:0/954638778 wait complete. 2024-02-21T23:17:30.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.581+0000 7fc302e00640 1 Processor -- start 2024-02-21T23:17:30.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.581+0000 7fc302e00640 1 -- start start 2024-02-21T23:17:30.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.582+0000 7fc302e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2fc0c11f0 0x7fc2fc0bfe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:30.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.582+0000 7fc302e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc2fc0c15d0 0x7fc2fc071d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:30.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.582+0000 7fc302e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2fc156350 0x7fc2fc1eb150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:30.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.582+0000 7fc302e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2fc0c4c00 con 0x7fc2fc0c11f0 2024-02-21T23:17:30.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.582+0000 7fc302e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2fc14d1c0 con 0x7fc2fc156350 2024-02-21T23:17:30.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.582+0000 7fc302e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2fc0a6110 con 0x7fc2fc0c15d0 2024-02-21T23:17:30.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.582+0000 7fc2fbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2fc0c11f0 0x7fc2fc0bfe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:30.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.582+0000 7fc2fbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2fc0c11f0 0x7fc2fc0bfe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41180/0 (socket says 172.21.15.28:41180) 2024-02-21T23:17:30.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.582+0000 7fc2fbe00640 1 -- 172.21.15.28:0/3199923612 learned_addr learned my addr 172.21.15.28:0/3199923612 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:30.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.583+0000 7fc300e00640 1 --2- 172.21.15.28:0/3199923612 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2fc156350 0x7fc2fc1eb150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:30.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.583+0000 7fc2f3400640 1 --2- 172.21.15.28:0/3199923612 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc2fc0c15d0 0x7fc2fc071d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:30.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.583+0000 7fc2fbe00640 1 -- 172.21.15.28:0/3199923612 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc2fc0c15d0 msgr2=0x7fc2fc071d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:30.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.583+0000 7fc2fbe00640 1 --2- 172.21.15.28:0/3199923612 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc2fc0c15d0 0x7fc2fc071d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:30.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.583+0000 7fc2fbe00640 1 -- 172.21.15.28:0/3199923612 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2fc156350 msgr2=0x7fc2fc1eb150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:30.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.583+0000 7fc2fbe00640 1 --2- 172.21.15.28:0/3199923612 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2fc156350 0x7fc2fc1eb150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:30.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.583+0000 7fc2fbe00640 1 -- 172.21.15.28:0/3199923612 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2fc0a72b0 con 0x7fc2fc0c11f0 2024-02-21T23:17:30.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.583+0000 7fc2f3400640 1 --2- 172.21.15.28:0/3199923612 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc2fc0c15d0 0x7fc2fc071d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:17:30.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.583+0000 7fc2fbe00640 1 --2- 172.21.15.28:0/3199923612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2fc0c11f0 0x7fc2fc0bfe90 secure :-1 s=READY pgs=2706 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec00e9e0 tx=0x7fc2ec00eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:30.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.583+0000 7fc2f8c00640 1 -- 172.21.15.28:0/3199923612 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2ec017070 con 0x7fc2fc0c11f0 2024-02-21T23:17:30.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.584+0000 7fc2f8c00640 1 -- 172.21.15.28:0/3199923612 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2ec00c9d0 con 0x7fc2fc0c11f0 2024-02-21T23:17:30.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.584+0000 7fc2f8c00640 1 -- 172.21.15.28:0/3199923612 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2ec00cd80 con 0x7fc2fc0c11f0 2024-02-21T23:17:30.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.584+0000 7fc302e00640 1 -- 172.21.15.28:0/3199923612 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2fc0727c0 con 0x7fc2fc0c11f0 2024-02-21T23:17:30.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.584+0000 7fc302e00640 1 -- 172.21.15.28:0/3199923612 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2fc0c03d0 con 0x7fc2fc0c11f0 2024-02-21T23:17:30.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.585+0000 7fc302e00640 1 -- 172.21.15.28:0/3199923612 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2fc067f30 con 0x7fc2fc0c11f0 2024-02-21T23:17:30.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.586+0000 7fc2f8c00640 1 -- 172.21.15.28:0/3199923612 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc2ec01a070 con 0x7fc2fc0c11f0 2024-02-21T23:17:30.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.587+0000 7fc2f8c00640 1 --2- 172.21.15.28:0/3199923612 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc2d806b4d0 0x7fc2d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:30.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.587+0000 7fc2f8c00640 1 -- 172.21.15.28:0/3199923612 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fc2ec08a770 con 0x7fc2fc0c11f0 2024-02-21T23:17:30.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.588+0000 7fc2f3400640 1 --2- 172.21.15.28:0/3199923612 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc2d806b4d0 0x7fc2d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:30.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.589+0000 7fc2f3400640 1 --2- 172.21.15.28:0/3199923612 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc2d806b4d0 0x7fc2d806d990 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7fc2e4036000 tx=0x7fc2e4008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:30.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.595+0000 7fc2f8c00640 1 -- 172.21.15.28:0/3199923612 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fc2ec0592d0 con 0x7fc2fc0c11f0 2024-02-21T23:17:30.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.843+0000 7fc302e00640 1 -- 172.21.15.28:0/3199923612 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc2fc0c0960 con 0x7fc2fc0c11f0 2024-02-21T23:17:30.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.844+0000 7fc2f8c00640 1 -- 172.21.15.28:0/3199923612 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7fc2ec058bd0 con 0x7fc2fc0c11f0 2024-02-21T23:17:30.831 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:17:30.831 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:17:30.831 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:17:30.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.847+0000 7fc302e00640 1 -- 172.21.15.28:0/3199923612 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc2d806b4d0 msgr2=0x7fc2d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:30.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.847+0000 7fc302e00640 1 --2- 172.21.15.28:0/3199923612 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc2d806b4d0 0x7fc2d806d990 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7fc2e4036000 tx=0x7fc2e4008030 comp rx=0 tx=0).stop 2024-02-21T23:17:30.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.847+0000 7fc302e00640 1 -- 172.21.15.28:0/3199923612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2fc0c11f0 msgr2=0x7fc2fc0bfe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:30.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.847+0000 7fc302e00640 1 --2- 172.21.15.28:0/3199923612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2fc0c11f0 0x7fc2fc0bfe90 secure :-1 s=READY pgs=2706 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec00e9e0 tx=0x7fc2ec00eea0 comp rx=0 tx=0).stop 2024-02-21T23:17:30.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.847+0000 7fc302e00640 1 -- 172.21.15.28:0/3199923612 shutdown_connections 2024-02-21T23:17:30.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.847+0000 7fc302e00640 1 --2- 172.21.15.28:0/3199923612 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fc2d806b4d0 0x7fc2d806d990 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:30.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.847+0000 7fc302e00640 1 --2- 172.21.15.28:0/3199923612 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fc2fc0c15d0 0x7fc2fc071d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:30.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.847+0000 7fc302e00640 1 --2- 172.21.15.28:0/3199923612 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc2fc156350 0x7fc2fc1eb150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:30.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.847+0000 7fc302e00640 1 --2- 172.21.15.28:0/3199923612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2fc0c11f0 0x7fc2fc0bfe90 unknown :-1 s=CLOSED pgs=2706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:30.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.847+0000 7fc302e00640 1 -- 172.21.15.28:0/3199923612 >> 172.21.15.28:0/3199923612 conn(0x7fc2fc0c67f0 msgr2=0x7fc2fc14b230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:30.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.847+0000 7fc302e00640 1 -- 172.21.15.28:0/3199923612 shutdown_connections 2024-02-21T23:17:30.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:30.847+0000 7fc302e00640 1 -- 172.21.15.28:0/3199923612 wait complete. 2024-02-21T23:17:31.256 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:17:31.257 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:17:32.257 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:17:32.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.498+0000 7fbf0d200640 1 Processor -- start 2024-02-21T23:17:32.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.498+0000 7fbf0d200640 1 -- start start 2024-02-21T23:17:32.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.499+0000 7fbf0d200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbf08153950 0x7fbf080c15d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:32.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.499+0000 7fbf0d200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbf0815b6d0 0x7fbf0815dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:32.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.500+0000 7fbf0d200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf08152e70 con 0x7fbf080c11f0 2024-02-21T23:17:32.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.500+0000 7fbf0d200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf080a5ec0 con 0x7fbf08153950 2024-02-21T23:17:32.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.500+0000 7fbf0d200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf08153150 con 0x7fbf0815b6d0 2024-02-21T23:17:32.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.500+0000 7fbf05400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbf08153950 0x7fbf080c15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:32.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.500+0000 7fbf05400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbf08153950 0x7fbf080c15d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41196/0 (socket says 172.21.15.28:41196) 2024-02-21T23:17:32.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.500+0000 7fbf05400640 1 -- 172.21.15.28:0/341909831 learned_addr learned my addr 172.21.15.28:0/341909831 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:32.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.500+0000 7fbf06800640 1 --2- 172.21.15.28:0/341909831 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbf0815b6d0 0x7fbf0815dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:32.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.500+0000 7fbf05400640 1 -- 172.21.15.28:0/341909831 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbf0815b6d0 msgr2=0x7fbf0815dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:32.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.501+0000 7fbf05400640 1 --2- 172.21.15.28:0/341909831 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbf0815b6d0 0x7fbf0815dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:32.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.501+0000 7fbf05400640 1 -- 172.21.15.28:0/341909831 >> v1:172.21.15.81:6789/0 conn(0x7fbf080c11f0 legacy=0x7fbf0806dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:32.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.501+0000 7fbf05400640 1 -- 172.21.15.28:0/341909831 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf08153430 con 0x7fbf08153950 2024-02-21T23:17:32.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.502+0000 7fbf05400640 1 --2- 172.21.15.28:0/341909831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf08153950 0x7fbf080c15d0 secure :-1 s=READY pgs=2707 cs=0 l=1 rev1=1 crypto rx=0x7fbef4009870 tx=0x7fbef402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1b0747e08a1e00f server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:32.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.502+0000 7fbf04a00640 1 -- 172.21.15.28:0/341909831 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef40317c0 con 0x7fbf08153950 2024-02-21T23:17:32.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.502+0000 7fbf04a00640 1 -- 172.21.15.28:0/341909831 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbef4037070 con 0x7fbf08153950 2024-02-21T23:17:32.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.502+0000 7fbf04a00640 1 -- 172.21.15.28:0/341909831 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef4031b80 con 0x7fbf08153950 2024-02-21T23:17:32.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.503+0000 7fbf0d200640 1 -- 172.21.15.28:0/341909831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf08153950 msgr2=0x7fbf080c15d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:32.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.503+0000 7fbf0d200640 1 --2- 172.21.15.28:0/341909831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf08153950 0x7fbf080c15d0 secure :-1 s=READY pgs=2707 cs=0 l=1 rev1=1 crypto rx=0x7fbef4009870 tx=0x7fbef402b860 comp rx=0 tx=0).stop 2024-02-21T23:17:32.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.503+0000 7fbf0d200640 1 -- 172.21.15.28:0/341909831 shutdown_connections 2024-02-21T23:17:32.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.503+0000 7fbf0d200640 1 --2- 172.21.15.28:0/341909831 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbf0815b6d0 0x7fbf0815dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:32.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.503+0000 7fbf0d200640 1 --2- 172.21.15.28:0/341909831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf08153950 0x7fbf080c15d0 unknown :-1 s=CLOSED pgs=2707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:32.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.503+0000 7fbf0d200640 1 -- 172.21.15.28:0/341909831 >> 172.21.15.28:0/341909831 conn(0x7fbf080c67f0 msgr2=0x7fbf080c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:32.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.503+0000 7fbf0d200640 1 -- 172.21.15.28:0/341909831 shutdown_connections 2024-02-21T23:17:32.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.503+0000 7fbf0d200640 1 -- 172.21.15.28:0/341909831 wait complete. 2024-02-21T23:17:32.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.505+0000 7fbf0d200640 1 Processor -- start 2024-02-21T23:17:32.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.505+0000 7fbf0d200640 1 -- start start 2024-02-21T23:17:32.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.506+0000 7fbf0d200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbf080c11f0 0x7fbf081ea850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:32.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.506+0000 7fbf0d200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf08153950 0x7fbf0814e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:32.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.506+0000 7fbf0d200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf0815b6d0 0x7fbf081ee6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:32.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.506+0000 7fbf0d200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf080a6110 con 0x7fbf0815b6d0 2024-02-21T23:17:32.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.506+0000 7fbf0d200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf080c4c00 con 0x7fbf08153950 2024-02-21T23:17:32.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.506+0000 7fbf0d200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf080c0d90 con 0x7fbf080c11f0 2024-02-21T23:17:32.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.506+0000 7fbf06800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf0815b6d0 0x7fbf081ee6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:32.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.506+0000 7fbf06800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf0815b6d0 0x7fbf081ee6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41204/0 (socket says 172.21.15.28:41204) 2024-02-21T23:17:32.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.506+0000 7fbf06800640 1 -- 172.21.15.28:0/86752291 learned_addr learned my addr 172.21.15.28:0/86752291 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:32.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.506+0000 7fbf05e00640 1 --2- 172.21.15.28:0/86752291 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbf080c11f0 0x7fbf081ea850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:32.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.506+0000 7fbf05400640 1 --2- 172.21.15.28:0/86752291 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf08153950 0x7fbf0814e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:32.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.507+0000 7fbf06800640 1 -- 172.21.15.28:0/86752291 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbf080c11f0 msgr2=0x7fbf081ea850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:32.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.507+0000 7fbf06800640 1 --2- 172.21.15.28:0/86752291 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbf080c11f0 0x7fbf081ea850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:32.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.507+0000 7fbf06800640 1 -- 172.21.15.28:0/86752291 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf08153950 msgr2=0x7fbf0814e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:32.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.507+0000 7fbf06800640 1 --2- 172.21.15.28:0/86752291 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf08153950 0x7fbf0814e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:32.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.507+0000 7fbf06800640 1 -- 172.21.15.28:0/86752291 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf080a7ca0 con 0x7fbf0815b6d0 2024-02-21T23:17:32.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.507+0000 7fbf06800640 1 --2- 172.21.15.28:0/86752291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf0815b6d0 0x7fbf081ee6e0 secure :-1 s=READY pgs=2708 cs=0 l=1 rev1=1 crypto rx=0x7fbf00007e40 tx=0x7fbf0000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:32.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.507+0000 7fbefe000640 1 -- 172.21.15.28:0/86752291 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf00014620 con 0x7fbf0815b6d0 2024-02-21T23:17:32.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.507+0000 7fbefe000640 1 -- 172.21.15.28:0/86752291 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf00019070 con 0x7fbf0815b6d0 2024-02-21T23:17:32.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.507+0000 7fbefe000640 1 -- 172.21.15.28:0/86752291 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf00014a20 con 0x7fbf0815b6d0 2024-02-21T23:17:32.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.507+0000 7fbf0d200640 1 -- 172.21.15.28:0/86752291 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf080a7f40 con 0x7fbf0815b6d0 2024-02-21T23:17:32.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.508+0000 7fbf0d200640 1 -- 172.21.15.28:0/86752291 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf081edee0 con 0x7fbf0815b6d0 2024-02-21T23:17:32.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.509+0000 7fbf0d200640 1 -- 172.21.15.28:0/86752291 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf08067f30 con 0x7fbf0815b6d0 2024-02-21T23:17:32.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.510+0000 7fbefe000640 1 -- 172.21.15.28:0/86752291 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbf00014bc0 con 0x7fbf0815b6d0 2024-02-21T23:17:32.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.511+0000 7fbefe000640 1 --2- 172.21.15.28:0/86752291 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbee806b4d0 0x7fbee806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:32.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.511+0000 7fbefe000640 1 -- 172.21.15.28:0/86752291 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fbf0008adb0 con 0x7fbf0815b6d0 2024-02-21T23:17:32.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.511+0000 7fbf05e00640 1 --2- 172.21.15.28:0/86752291 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbee806b4d0 0x7fbee806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:32.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.512+0000 7fbf05e00640 1 --2- 172.21.15.28:0/86752291 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbee806b4d0 0x7fbee806d990 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7fbef8006fd0 tx=0x7fbef8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:32.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.515+0000 7fbefe000640 1 -- 172.21.15.28:0/86752291 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fbf0008f070 con 0x7fbf0815b6d0 2024-02-21T23:17:32.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.762+0000 7fbf0d200640 1 -- 172.21.15.28:0/86752291 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf080c0960 con 0x7fbf0815b6d0 2024-02-21T23:17:32.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.762+0000 7fbefe000640 1 -- 172.21.15.28:0/86752291 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7fbf00059900 con 0x7fbf0815b6d0 2024-02-21T23:17:32.750 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:17:32.750 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:17:32.750 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:17:32.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.765+0000 7fbf0d200640 1 -- 172.21.15.28:0/86752291 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbee806b4d0 msgr2=0x7fbee806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:32.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.765+0000 7fbf0d200640 1 --2- 172.21.15.28:0/86752291 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbee806b4d0 0x7fbee806d990 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7fbef8006fd0 tx=0x7fbef8008040 comp rx=0 tx=0).stop 2024-02-21T23:17:32.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.765+0000 7fbf0d200640 1 -- 172.21.15.28:0/86752291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf0815b6d0 msgr2=0x7fbf081ee6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:32.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.765+0000 7fbf0d200640 1 --2- 172.21.15.28:0/86752291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf0815b6d0 0x7fbf081ee6e0 secure :-1 s=READY pgs=2708 cs=0 l=1 rev1=1 crypto rx=0x7fbf00007e40 tx=0x7fbf0000e4b0 comp rx=0 tx=0).stop 2024-02-21T23:17:32.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.766+0000 7fbf0d200640 1 -- 172.21.15.28:0/86752291 shutdown_connections 2024-02-21T23:17:32.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.766+0000 7fbf0d200640 1 --2- 172.21.15.28:0/86752291 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbee806b4d0 0x7fbee806d990 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:32.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.766+0000 7fbf0d200640 1 --2- 172.21.15.28:0/86752291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf0815b6d0 0x7fbf081ee6e0 unknown :-1 s=CLOSED pgs=2708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:32.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.766+0000 7fbf0d200640 1 --2- 172.21.15.28:0/86752291 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf08153950 0x7fbf0814e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:32.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.766+0000 7fbf0d200640 1 --2- 172.21.15.28:0/86752291 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbf080c11f0 0x7fbf081ea850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:32.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.766+0000 7fbf0d200640 1 -- 172.21.15.28:0/86752291 >> 172.21.15.28:0/86752291 conn(0x7fbf080c67f0 msgr2=0x7fbf080c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:32.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.766+0000 7fbf0d200640 1 -- 172.21.15.28:0/86752291 shutdown_connections 2024-02-21T23:17:32.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:32.766+0000 7fbf0d200640 1 -- 172.21.15.28:0/86752291 wait complete. 2024-02-21T23:17:33.175 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:17:33.176 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:17:34.177 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:17:34.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.416+0000 7fed7f200640 1 Processor -- start 2024-02-21T23:17:34.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.417+0000 7fed7f200640 1 -- start start 2024-02-21T23:17:34.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.418+0000 7fed7f200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fed8006dff0 0x7fed8006e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:34.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.418+0000 7fed7f200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fed800c0fe0 0x7fed80071350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:34.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.418+0000 7fed7f200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fed80158580 0x7fed8014b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:34.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.418+0000 7fed7f200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed800a8b40 con 0x7fed800c0fe0 2024-02-21T23:17:34.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.418+0000 7fed7f200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed800a8e00 con 0x7fed80158580 2024-02-21T23:17:34.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.418+0000 7fed7f200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed800a90e0 con 0x7fed8006dff0 2024-02-21T23:17:34.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.418+0000 7fed75400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fed800c0fe0 0x7fed80071350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:34.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.418+0000 7fed75400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fed800c0fe0 0x7fed80071350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41206/0 (socket says 172.21.15.28:41206) 2024-02-21T23:17:34.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.418+0000 7fed75400640 1 -- 172.21.15.28:0/3580406215 learned_addr learned my addr 172.21.15.28:0/3580406215 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:34.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.419+0000 7fed75400640 1 -- 172.21.15.28:0/3580406215 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fed8006dff0 msgr2=0x7fed8006e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:34.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.419+0000 7fed7de00640 1 --2- 172.21.15.28:0/3580406215 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fed8006dff0 0x7fed8006e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:34.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.419+0000 7fed75400640 1 --2- 172.21.15.28:0/3580406215 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fed8006dff0 0x7fed8006e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:34.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.419+0000 7fed75400640 1 -- 172.21.15.28:0/3580406215 >> v2:172.21.15.81:3300/0 conn(0x7fed80158580 msgr2=0x7fed8014b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:34.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.419+0000 7fed7e800640 1 --2- 172.21.15.28:0/3580406215 >> v2:172.21.15.81:3300/0 conn(0x7fed80158580 0x7fed8014b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:34.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.419+0000 7fed75400640 1 --2- 172.21.15.28:0/3580406215 >> v2:172.21.15.81:3300/0 conn(0x7fed80158580 0x7fed8014b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:34.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.419+0000 7fed75400640 1 -- 172.21.15.28:0/3580406215 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed80072ed0 con 0x7fed800c0fe0 2024-02-21T23:17:34.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.419+0000 7fed7de00640 1 --2- 172.21.15.28:0/3580406215 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fed8006dff0 0x7fed8006e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:17:34.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.419+0000 7fed7e800640 1 --2- 172.21.15.28:0/3580406215 >> v2:172.21.15.81:3300/0 conn(0x7fed80158580 0x7fed8014b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:17:34.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.420+0000 7fed75400640 1 --2- 172.21.15.28:0/3580406215 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed800c0fe0 0x7fed80071350 secure :-1 s=READY pgs=2709 cs=0 l=1 rev1=1 crypto rx=0x7fed8006f270 tx=0x7fed6802b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e77a17ac6386886 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:34.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.421+0000 7fed7d400640 1 -- 172.21.15.28:0/3580406215 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed68035020 con 0x7fed800c0fe0 2024-02-21T23:17:34.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.421+0000 7fed7d400640 1 -- 172.21.15.28:0/3580406215 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed6802d040 con 0x7fed800c0fe0 2024-02-21T23:17:34.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.421+0000 7fed7d400640 1 -- 172.21.15.28:0/3580406215 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed68031a80 con 0x7fed800c0fe0 2024-02-21T23:17:34.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.421+0000 7fed7f200640 1 -- 172.21.15.28:0/3580406215 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed800c0fe0 msgr2=0x7fed80071350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:34.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.421+0000 7fed7f200640 1 --2- 172.21.15.28:0/3580406215 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed800c0fe0 0x7fed80071350 secure :-1 s=READY pgs=2709 cs=0 l=1 rev1=1 crypto rx=0x7fed8006f270 tx=0x7fed6802b900 comp rx=0 tx=0).stop 2024-02-21T23:17:34.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.422+0000 7fed7f200640 1 -- 172.21.15.28:0/3580406215 shutdown_connections 2024-02-21T23:17:34.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.422+0000 7fed7f200640 1 --2- 172.21.15.28:0/3580406215 >> v2:172.21.15.81:3300/0 conn(0x7fed80158580 0x7fed8014b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:34.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.422+0000 7fed7f200640 1 --2- 172.21.15.28:0/3580406215 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed800c0fe0 0x7fed80071350 unknown :-1 s=CLOSED pgs=2709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:34.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.422+0000 7fed7f200640 1 --2- 172.21.15.28:0/3580406215 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fed8006dff0 0x7fed8006e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:34.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.422+0000 7fed7f200640 1 -- 172.21.15.28:0/3580406215 >> 172.21.15.28:0/3580406215 conn(0x7fed8006b630 msgr2=0x7fed800c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:34.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.422+0000 7fed7f200640 1 -- 172.21.15.28:0/3580406215 shutdown_connections 2024-02-21T23:17:34.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.422+0000 7fed7f200640 1 -- 172.21.15.28:0/3580406215 wait complete. 2024-02-21T23:17:34.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.424+0000 7fed7f200640 1 Processor -- start 2024-02-21T23:17:34.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.425+0000 7fed7f200640 1 -- start start 2024-02-21T23:17:34.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.426+0000 7fed7f200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fed8006dff0 0x7fed801df050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:34.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.426+0000 7fed7f200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed800c0fe0 0x7fed80151fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:34.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.426+0000 7fed7f200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed80158580 0x7fed801eacb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:34.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.426+0000 7fed7f200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed80071890 con 0x7fed80158580 2024-02-21T23:17:34.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.426+0000 7fed7f200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed80073520 con 0x7fed800c0fe0 2024-02-21T23:17:34.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.426+0000 7fed7f200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed80158a10 con 0x7fed8006dff0 2024-02-21T23:17:34.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.427+0000 7fed7e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed80158580 0x7fed801eacb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:34.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.427+0000 7fed7e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed80158580 0x7fed801eacb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41210/0 (socket says 172.21.15.28:41210) 2024-02-21T23:17:34.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.427+0000 7fed7e800640 1 -- 172.21.15.28:0/778496892 learned_addr learned my addr 172.21.15.28:0/778496892 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:34.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.427+0000 7fed7de00640 1 --2- 172.21.15.28:0/778496892 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fed8006dff0 0x7fed801df050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:34.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.427+0000 7fed75400640 1 --2- 172.21.15.28:0/778496892 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed800c0fe0 0x7fed80151fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:34.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.427+0000 7fed7e800640 1 -- 172.21.15.28:0/778496892 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fed8006dff0 msgr2=0x7fed801df050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:34.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.427+0000 7fed7e800640 1 --2- 172.21.15.28:0/778496892 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fed8006dff0 0x7fed801df050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:34.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.427+0000 7fed7e800640 1 -- 172.21.15.28:0/778496892 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed800c0fe0 msgr2=0x7fed80151fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:34.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.427+0000 7fed7e800640 1 --2- 172.21.15.28:0/778496892 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed800c0fe0 0x7fed80151fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:34.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.427+0000 7fed7e800640 1 -- 172.21.15.28:0/778496892 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed801eecd0 con 0x7fed80158580 2024-02-21T23:17:34.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.427+0000 7fed7de00640 1 --2- 172.21.15.28:0/778496892 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fed8006dff0 0x7fed801df050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:17:34.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.427+0000 7fed75400640 1 --2- 172.21.15.28:0/778496892 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed800c0fe0 0x7fed80151fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:17:34.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.428+0000 7fed7e800640 1 --2- 172.21.15.28:0/778496892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed80158580 0x7fed801eacb0 secure :-1 s=READY pgs=2710 cs=0 l=1 rev1=1 crypto rx=0x7fed7800c7f0 tx=0x7fed7800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:34.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.428+0000 7fed76a00640 1 -- 172.21.15.28:0/778496892 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed78015070 con 0x7fed80158580 2024-02-21T23:17:34.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.428+0000 7fed76a00640 1 -- 172.21.15.28:0/778496892 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed78007820 con 0x7fed80158580 2024-02-21T23:17:34.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.428+0000 7fed76a00640 1 -- 172.21.15.28:0/778496892 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed78007c50 con 0x7fed80158580 2024-02-21T23:17:34.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.428+0000 7fed7f200640 1 -- 172.21.15.28:0/778496892 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed801eefb0 con 0x7fed80158580 2024-02-21T23:17:34.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.428+0000 7fed7f200640 1 -- 172.21.15.28:0/778496892 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed80072140 con 0x7fed80158580 2024-02-21T23:17:34.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.429+0000 7fed7f200640 1 -- 172.21.15.28:0/778496892 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed8004f290 con 0x7fed80158580 2024-02-21T23:17:34.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.430+0000 7fed76a00640 1 -- 172.21.15.28:0/778496892 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fed78018070 con 0x7fed80158580 2024-02-21T23:17:34.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.431+0000 7fed76a00640 1 --2- 172.21.15.28:0/778496892 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fed5c06b4d0 0x7fed5c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:34.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.431+0000 7fed76a00640 1 -- 172.21.15.28:0/778496892 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fed7808a6a0 con 0x7fed80158580 2024-02-21T23:17:34.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.432+0000 7fed7de00640 1 --2- 172.21.15.28:0/778496892 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fed5c06b4d0 0x7fed5c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:34.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.433+0000 7fed7de00640 1 --2- 172.21.15.28:0/778496892 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fed5c06b4d0 0x7fed5c06d990 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7fed801e06e0 tx=0x7fed70008470 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:34.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.441+0000 7fed76a00640 1 -- 172.21.15.28:0/778496892 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fed78059280 con 0x7fed80158580 2024-02-21T23:17:34.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.684+0000 7fed7f200640 1 -- 172.21.15.28:0/778496892 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fed8006ecb0 con 0x7fed80158580 2024-02-21T23:17:34.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.685+0000 7fed76a00640 1 -- 172.21.15.28:0/778496892 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7fed78058b80 con 0x7fed80158580 2024-02-21T23:17:34.673 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:17:34.673 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:17:34.673 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:17:34.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.688+0000 7fed7f200640 1 -- 172.21.15.28:0/778496892 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fed5c06b4d0 msgr2=0x7fed5c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:34.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.688+0000 7fed7f200640 1 --2- 172.21.15.28:0/778496892 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fed5c06b4d0 0x7fed5c06d990 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7fed801e06e0 tx=0x7fed70008470 comp rx=0 tx=0).stop 2024-02-21T23:17:34.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.688+0000 7fed7f200640 1 -- 172.21.15.28:0/778496892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed80158580 msgr2=0x7fed801eacb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:34.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.688+0000 7fed7f200640 1 --2- 172.21.15.28:0/778496892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed80158580 0x7fed801eacb0 secure :-1 s=READY pgs=2710 cs=0 l=1 rev1=1 crypto rx=0x7fed7800c7f0 tx=0x7fed7800ccb0 comp rx=0 tx=0).stop 2024-02-21T23:17:34.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.688+0000 7fed7f200640 1 -- 172.21.15.28:0/778496892 shutdown_connections 2024-02-21T23:17:34.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.688+0000 7fed7f200640 1 --2- 172.21.15.28:0/778496892 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fed5c06b4d0 0x7fed5c06d990 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:34.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.688+0000 7fed7f200640 1 --2- 172.21.15.28:0/778496892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed80158580 0x7fed801eacb0 unknown :-1 s=CLOSED pgs=2710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:34.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.688+0000 7fed7f200640 1 --2- 172.21.15.28:0/778496892 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed800c0fe0 0x7fed80151fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:34.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.688+0000 7fed7f200640 1 --2- 172.21.15.28:0/778496892 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fed8006dff0 0x7fed801df050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:34.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.688+0000 7fed7f200640 1 -- 172.21.15.28:0/778496892 >> 172.21.15.28:0/778496892 conn(0x7fed8006b630 msgr2=0x7fed800c26a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:34.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.688+0000 7fed7f200640 1 -- 172.21.15.28:0/778496892 shutdown_connections 2024-02-21T23:17:34.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:34.689+0000 7fed7f200640 1 -- 172.21.15.28:0/778496892 wait complete. 2024-02-21T23:17:35.100 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 247, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-02-21T23:13:29.725909+0000', 'modified': '2024-02-21T23:15:17.646309+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 212, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 0, 'in': [0, 1], 'up': {'mds_0': 8296, 'mds_1': 7265}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7265': {'gid': 7265, 'name': 'b', 'rank': 1, 'incarnation': 241, 'state': 'up:stopping', 'state_seq': 24, 'addr': '172.21.15.81:6837/3779716966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.81:6836', 'nonce': 3779716966}, {'type': 'v1', 'addr': '172.21.15.81:6837', 'nonce': 3779716966}]}, 'join_fscid': -1, 'laggy_since': '2024-02-21T23:15:17.646309+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8296': {'gid': 8296, 'name': 'c', 'rank': 0, 'incarnation': 225, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3142544451', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3142544451}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3142544451}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [34], 'metadata_pool': 33, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-21T23:17:35.100 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7265:up:stopping 2024-02-21T23:17:36.102 DEBUG:tasks.cephfs.filesystem:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 16, "epoch": 247, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 16, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-02-21T23:13:29.725909+0000", "damaged": [], "data_pools": [ 34 ], "enabled": true, "epoch": 247, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_7265": { "addr": "172.21.15.81:6837/3779716966", "addrs": { "addrvec": [ { "addr": "172.21.15.81:6836", "nonce": 3779716966, "type": "v2" }, { "addr": "172.21.15.81:6837", "nonce": 3779716966, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7265, "incarnation": 241, "join_fscid": -1, "laggy_since": "2024-02-21T23:15:17.646309+0000", "name": "b", "rank": 1, "state": "up:stopping", "state_seq": 24 }, "gid_8296": { "addr": "172.21.15.28:6835/3142544451", "addrs": { "addrvec": [ { "addr": "172.21.15.28:6834", "nonce": 3142544451, "type": "v2" }, { "addr": "172.21.15.28:6835", "nonce": 3142544451, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8296, "incarnation": 225, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 212, "max_file_size": 1099511627776, "max_mds": 0, "max_xattr_size": 65536, "metadata_pool": 33, "modified": "2024-02-21T23:15:17.646309+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8296, "mds_1": 7265 } } } ], "standbys": [] } 2024-02-21T23:17:36.102 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-02-21T23:17:36.102 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-21T23:17:36.218 DEBUG:teuthology.orchestra.run.smithi081:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-02-21T23:17:36.340 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-02-21T23:17:36.340 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-21T23:17:36.340 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-21T23:17:36.400 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054608,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-02-21T23:17:36.400 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-02-21T23:17:36.401 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-21T23:17:36.457 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053833,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498812k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249404k,nr_inodes=812351,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498812k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-02-21T23:17:36.457 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:17:36.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.680+0000 7f7fe3200640 1 Processor -- start 2024-02-21T23:17:36.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.680+0000 7f7fe3200640 1 -- start start 2024-02-21T23:17:36.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.682+0000 7f7fe3200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7fe40aff90 0x7f7fe40c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:36.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.682+0000 7f7fe3200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f7fe40c5720 0x7f7fe415f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:36.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.682+0000 7f7fe3200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fe40c74c0 con 0x7f7fe406dbf0 2024-02-21T23:17:36.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.682+0000 7f7fe3200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fe40c7270 con 0x7f7fe40aff90 2024-02-21T23:17:36.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.682+0000 7f7fe3200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fe4158630 con 0x7f7fe40c5720 2024-02-21T23:17:36.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.682+0000 7f7fe1e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7fe40aff90 0x7f7fe40c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:36.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.682+0000 7f7fe1400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f7fe406dbf0 0x7f7fe4158ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59810/0 (socket says 172.21.15.28:59810) 2024-02-21T23:17:36.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.682+0000 7f7fe1400640 1 -- 172.21.15.28:0/3920288344 learned_addr learned my addr 172.21.15.28:0/3920288344 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:17:36.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.683+0000 7f7fe2800640 1 --2- 172.21.15.28:0/3920288344 >> v2:172.21.15.81:3300/0 conn(0x7f7fe40c5720 0x7f7fe415f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:36.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.683+0000 7f7fe1e00640 1 -- 172.21.15.28:0/3920288344 >> v2:172.21.15.81:3300/0 conn(0x7f7fe40c5720 msgr2=0x7f7fe415f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:36.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.683+0000 7f7fe1e00640 1 --2- 172.21.15.28:0/3920288344 >> v2:172.21.15.81:3300/0 conn(0x7f7fe40c5720 0x7f7fe415f890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:36.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.683+0000 7f7fe1e00640 1 -- 172.21.15.28:0/3920288344 >> v1:172.21.15.28:6789/0 conn(0x7f7fe406dbf0 legacy=0x7f7fe4158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:36.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.683+0000 7f7fe1e00640 1 -- 172.21.15.28:0/3920288344 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fe40c77a0 con 0x7f7fe40aff90 2024-02-21T23:17:36.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.683+0000 7f7fe0a00640 1 -- 172.21.15.28:0/3920288344 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 256336674 0 0) 0x7f7fe40c74c0 con 0x7f7fe406dbf0 2024-02-21T23:17:36.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.684+0000 7f7fe1e00640 1 --2- 172.21.15.28:0/3920288344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fe40aff90 0x7f7fe40c51e0 secure :-1 s=READY pgs=2712 cs=0 l=1 rev1=1 crypto rx=0x7f7fe40c73f0 tx=0x7f7fd402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a1dc1a726b3c1e5 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:36.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.684+0000 7f7fe0a00640 1 -- 172.21.15.28:0/3920288344 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fd4039070 con 0x7f7fe40aff90 2024-02-21T23:17:36.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.684+0000 7f7fe0a00640 1 -- 172.21.15.28:0/3920288344 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fd402cc60 con 0x7f7fe40aff90 2024-02-21T23:17:36.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.685+0000 7f7fe0a00640 1 -- 172.21.15.28:0/3920288344 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fd4034660 con 0x7f7fe40aff90 2024-02-21T23:17:36.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.685+0000 7f7fe3200640 1 -- 172.21.15.28:0/3920288344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fe40aff90 msgr2=0x7f7fe40c51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:36.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.685+0000 7f7fe3200640 1 --2- 172.21.15.28:0/3920288344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fe40aff90 0x7f7fe40c51e0 secure :-1 s=READY pgs=2712 cs=0 l=1 rev1=1 crypto rx=0x7f7fe40c73f0 tx=0x7f7fd402beb0 comp rx=0 tx=0).stop 2024-02-21T23:17:36.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.685+0000 7f7fe3200640 1 -- 172.21.15.28:0/3920288344 shutdown_connections 2024-02-21T23:17:36.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.685+0000 7f7fe3200640 1 --2- 172.21.15.28:0/3920288344 >> v2:172.21.15.81:3300/0 conn(0x7f7fe40c5720 0x7f7fe415f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:36.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.685+0000 7f7fe3200640 1 --2- 172.21.15.28:0/3920288344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fe40aff90 0x7f7fe40c51e0 unknown :-1 s=CLOSED pgs=2712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:36.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.685+0000 7f7fe3200640 1 -- 172.21.15.28:0/3920288344 >> 172.21.15.28:0/3920288344 conn(0x7f7fe40ac370 msgr2=0x7f7fe40ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:36.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.685+0000 7f7fe3200640 1 -- 172.21.15.28:0/3920288344 shutdown_connections 2024-02-21T23:17:36.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.685+0000 7f7fe3200640 1 -- 172.21.15.28:0/3920288344 wait complete. 2024-02-21T23:17:36.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.687+0000 7f7fe3200640 1 Processor -- start 2024-02-21T23:17:36.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.687+0000 7f7fe3200640 1 -- start start 2024-02-21T23:17:36.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.688+0000 7f7fe3200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7fe406dbf0 0x7f7fe40c0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:36.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.688+0000 7f7fe3200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fe40aff90 0x7f7fe4156570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:36.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.688+0000 7f7fe3200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7fe40c5720 0x7f7fe40c3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:36.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.688+0000 7f7fe3200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe40c8760 con 0x7f7fe40aff90 2024-02-21T23:17:36.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.688+0000 7f7fe3200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe40be0c0 con 0x7f7fe406dbf0 2024-02-21T23:17:36.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.688+0000 7f7fe3200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe40c88e0 con 0x7f7fe40c5720 2024-02-21T23:17:36.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.688+0000 7f7fe1400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fe40aff90 0x7f7fe4156570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:36.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.688+0000 7f7fe1400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fe40aff90 0x7f7fe4156570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59618/0 (socket says 172.21.15.28:59618) 2024-02-21T23:17:36.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.689+0000 7f7fe1400640 1 -- 172.21.15.28:0/1475490381 learned_addr learned my addr 172.21.15.28:0/1475490381 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:36.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.689+0000 7f7fe1e00640 1 --2- 172.21.15.28:0/1475490381 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7fe406dbf0 0x7f7fe40c0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:36.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.689+0000 7f7fe1400640 1 -- 172.21.15.28:0/1475490381 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7fe40c5720 msgr2=0x7f7fe40c3670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:36.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.689+0000 7f7fe1400640 1 --2- 172.21.15.28:0/1475490381 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7fe40c5720 0x7f7fe40c3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:36.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.689+0000 7f7fe1400640 1 -- 172.21.15.28:0/1475490381 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7fe406dbf0 msgr2=0x7f7fe40c0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:36.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.689+0000 7f7fe1400640 1 --2- 172.21.15.28:0/1475490381 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7fe406dbf0 0x7f7fe40c0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:36.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.689+0000 7f7fe1400640 1 -- 172.21.15.28:0/1475490381 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fe4151f30 con 0x7f7fe40aff90 2024-02-21T23:17:36.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.689+0000 7f7fe1e00640 1 --2- 172.21.15.28:0/1475490381 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7fe406dbf0 0x7f7fe40c0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:17:36.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.689+0000 7f7fe1400640 1 --2- 172.21.15.28:0/1475490381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fe40aff90 0x7f7fe4156570 secure :-1 s=READY pgs=2713 cs=0 l=1 rev1=1 crypto rx=0x7f7fd000c4b0 tx=0x7f7fd000c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:36.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.689+0000 7f7fda000640 1 -- 172.21.15.28:0/1475490381 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fd0018070 con 0x7f7fe40aff90 2024-02-21T23:17:36.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.690+0000 7f7fda000640 1 -- 172.21.15.28:0/1475490381 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fd0007500 con 0x7f7fe40aff90 2024-02-21T23:17:36.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.690+0000 7f7fda000640 1 -- 172.21.15.28:0/1475490381 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fd0004a20 con 0x7f7fe40aff90 2024-02-21T23:17:36.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.690+0000 7f7fe3200640 1 -- 172.21.15.28:0/1475490381 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fe41517a0 con 0x7f7fe40aff90 2024-02-21T23:17:36.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.690+0000 7f7fe3200640 1 -- 172.21.15.28:0/1475490381 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fe40c46b0 con 0x7f7fe40aff90 2024-02-21T23:17:36.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.691+0000 7f7fe3200640 1 -- 172.21.15.28:0/1475490381 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fe404f290 con 0x7f7fe40aff90 2024-02-21T23:17:36.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.692+0000 7f7fda000640 1 -- 172.21.15.28:0/1475490381 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7fd001b070 con 0x7f7fe40aff90 2024-02-21T23:17:36.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.693+0000 7f7fda000640 1 --2- 172.21.15.28:0/1475490381 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7fc406b4d0 0x7f7fc406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:36.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.693+0000 7f7fda000640 1 -- 172.21.15.28:0/1475490381 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f7fd00897d0 con 0x7f7fe40aff90 2024-02-21T23:17:36.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.693+0000 7f7fe1e00640 1 --2- 172.21.15.28:0/1475490381 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7fc406b4d0 0x7f7fc406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:36.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.695+0000 7f7fe1e00640 1 --2- 172.21.15.28:0/1475490381 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7fc406b4d0 0x7f7fc406d990 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f7fd4036000 tx=0x7f7fd4030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:36.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.697+0000 7f7fda000640 1 -- 172.21.15.28:0/1475490381 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7fd0058470 con 0x7f7fe40aff90 2024-02-21T23:17:36.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.938+0000 7f7fe3200640 1 -- 172.21.15.28:0/1475490381 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7fe406d9d0 con 0x7f7fe40aff90 2024-02-21T23:17:36.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.939+0000 7f7fda000640 1 -- 172.21.15.28:0/1475490381 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7f7fd0057d70 con 0x7f7fe40aff90 2024-02-21T23:17:36.927 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:17:36.927 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:17:36.927 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:17:36.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.941+0000 7f7fe3200640 1 -- 172.21.15.28:0/1475490381 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7fc406b4d0 msgr2=0x7f7fc406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:36.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.941+0000 7f7fe3200640 1 --2- 172.21.15.28:0/1475490381 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7fc406b4d0 0x7f7fc406d990 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f7fd4036000 tx=0x7f7fd4030000 comp rx=0 tx=0).stop 2024-02-21T23:17:36.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.941+0000 7f7fe3200640 1 -- 172.21.15.28:0/1475490381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fe40aff90 msgr2=0x7f7fe4156570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:36.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.941+0000 7f7fe3200640 1 --2- 172.21.15.28:0/1475490381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fe40aff90 0x7f7fe4156570 secure :-1 s=READY pgs=2713 cs=0 l=1 rev1=1 crypto rx=0x7f7fd000c4b0 tx=0x7f7fd000c970 comp rx=0 tx=0).stop 2024-02-21T23:17:36.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.941+0000 7f7fe3200640 1 -- 172.21.15.28:0/1475490381 shutdown_connections 2024-02-21T23:17:36.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.941+0000 7f7fe3200640 1 --2- 172.21.15.28:0/1475490381 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7fc406b4d0 0x7f7fc406d990 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:36.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.941+0000 7f7fe3200640 1 --2- 172.21.15.28:0/1475490381 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7fe40c5720 0x7f7fe40c3670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:36.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.941+0000 7f7fe3200640 1 --2- 172.21.15.28:0/1475490381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fe40aff90 0x7f7fe4156570 unknown :-1 s=CLOSED pgs=2713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:36.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.942+0000 7f7fe3200640 1 --2- 172.21.15.28:0/1475490381 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7fe406dbf0 0x7f7fe40c0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:36.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.942+0000 7f7fe3200640 1 -- 172.21.15.28:0/1475490381 >> 172.21.15.28:0/1475490381 conn(0x7f7fe40ac370 msgr2=0x7f7fe40c8ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:36.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.942+0000 7f7fe3200640 1 -- 172.21.15.28:0/1475490381 shutdown_connections 2024-02-21T23:17:36.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:36.942+0000 7f7fe3200640 1 -- 172.21.15.28:0/1475490381 wait complete. 2024-02-21T23:17:36.950 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:17:37.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.174+0000 7fa5dec00640 1 Processor -- start 2024-02-21T23:17:37.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.174+0000 7fa5dec00640 1 -- start start 2024-02-21T23:17:37.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.176+0000 7fa5dec00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa5d806dbf0 0x7fa5d8158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:37.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.176+0000 7fa5dec00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5d80c74c0 con 0x7fa5d8155960 2024-02-21T23:17:37.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.176+0000 7fa5dec00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5d80c77a0 con 0x7fa5d80aff90 2024-02-21T23:17:37.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.176+0000 7fa5dec00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d80c7270 con 0x7fa5d806dbf0 2024-02-21T23:17:37.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.176+0000 7fa5de200640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa5d8155960 0x7fa5d815f890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59812/0 (socket says 172.21.15.28:59812) 2024-02-21T23:17:37.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.176+0000 7fa5de200640 1 -- 172.21.15.28:0/997442141 learned_addr learned my addr 172.21.15.28:0/997442141 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:17:37.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.176+0000 7fa5dce00640 1 --2- 172.21.15.28:0/997442141 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa5d806dbf0 0x7fa5d8158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:37.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.177+0000 7fa5d7e00640 1 -- 172.21.15.28:0/997442141 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 125815522 0 0) 0x7fa5d80c74c0 con 0x7fa5d8155960 2024-02-21T23:17:37.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.177+0000 7fa5d7e00640 1 -- 172.21.15.28:0/997442141 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5c0003520 con 0x7fa5d8155960 2024-02-21T23:17:37.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.177+0000 7fa5d7e00640 1 -- 172.21.15.28:0/997442141 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3188756192 0 0) 0x7fa5c0003520 con 0x7fa5d8155960 2024-02-21T23:17:37.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.177+0000 7fa5d7e00640 1 -- 172.21.15.28:0/997442141 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa5d806dbf0 msgr2=0x7fa5d8158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:37.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.177+0000 7fa5d7e00640 1 --2- 172.21.15.28:0/997442141 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa5d806dbf0 0x7fa5d8158ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:37.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.177+0000 7fa5d7e00640 1 -- 172.21.15.28:0/997442141 >> v1:172.21.15.81:6789/0 conn(0x7fa5d80aff90 legacy=0x7fa5d80c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:37.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.177+0000 7fa5d7e00640 1 -- 172.21.15.28:0/997442141 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5d80c7a80 con 0x7fa5d8155960 2024-02-21T23:17:37.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.177+0000 7fa5d7e00640 1 -- 172.21.15.28:0/997442141 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fa5d00019f0 con 0x7fa5d8155960 2024-02-21T23:17:37.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.177+0000 7fa5d7e00640 1 -- 172.21.15.28:0/997442141 >> v1:172.21.15.28:6789/0 conn(0x7fa5d8155960 legacy=0x7fa5d815f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:37.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.177+0000 7fa5d7e00640 1 --2- 172.21.15.28:0/997442141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5c00039e0 0x7fa5c0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:37.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.177+0000 7fa5d7e00640 1 --2- 172.21.15.28:0/997442141 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa5c0004800 0x7fa5c0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:37.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.177+0000 7fa5d7e00640 1 --2- 172.21.15.28:0/997442141 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa5d806dbf0 0x7fa5c00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:37.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.177+0000 7fa5d7e00640 1 -- 172.21.15.28:0/997442141 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d80c7270 con 0x7fa5c00039e0 2024-02-21T23:17:37.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.177+0000 7fa5d7e00640 1 -- 172.21.15.28:0/997442141 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c000ba70 con 0x7fa5c0004800 2024-02-21T23:17:37.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.177+0000 7fa5d7e00640 1 -- 172.21.15.28:0/997442141 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c000bc70 con 0x7fa5d806dbf0 2024-02-21T23:17:37.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.178+0000 7fa5dd800640 1 --2- 172.21.15.28:0/997442141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5c00039e0 0x7fa5c0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:37.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.178+0000 7fa5de200640 1 --2- 172.21.15.28:0/997442141 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa5d806dbf0 0x7fa5c00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:37.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.178+0000 7fa5dce00640 1 --2- 172.21.15.28:0/997442141 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa5c0004800 0x7fa5c0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:37.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.178+0000 7fa5dd800640 1 -- 172.21.15.28:0/997442141 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa5d806dbf0 msgr2=0x7fa5c00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:37.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.178+0000 7fa5dd800640 1 --2- 172.21.15.28:0/997442141 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa5d806dbf0 0x7fa5c00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:37.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.178+0000 7fa5dd800640 1 -- 172.21.15.28:0/997442141 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa5c0004800 msgr2=0x7fa5c0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:37.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.178+0000 7fa5dd800640 1 --2- 172.21.15.28:0/997442141 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa5c0004800 0x7fa5c0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:37.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.178+0000 7fa5dd800640 1 -- 172.21.15.28:0/997442141 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5c0003520 con 0x7fa5c00039e0 2024-02-21T23:17:37.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.178+0000 7fa5dce00640 1 --2- 172.21.15.28:0/997442141 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa5c0004800 0x7fa5c0004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:17:37.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.178+0000 7fa5de200640 1 --2- 172.21.15.28:0/997442141 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa5d806dbf0 0x7fa5c00051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:17:37.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.179+0000 7fa5dd800640 1 --2- 172.21.15.28:0/997442141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5c00039e0 0x7fa5c0003e00 secure :-1 s=READY pgs=2715 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc001720 tx=0x7fa5c0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68b2970d26ed0465 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:37.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.179+0000 7fa5d7e00640 1 -- 172.21.15.28:0/997442141 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5cc001370 con 0x7fa5c00039e0 2024-02-21T23:17:37.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.180+0000 7fa5d7e00640 1 -- 172.21.15.28:0/997442141 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5cc0052d0 con 0x7fa5c00039e0 2024-02-21T23:17:37.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.180+0000 7fa5d7e00640 1 -- 172.21.15.28:0/997442141 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5cc003830 con 0x7fa5c00039e0 2024-02-21T23:17:37.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.180+0000 7fa5dec00640 1 -- 172.21.15.28:0/997442141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5c00039e0 msgr2=0x7fa5c0003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:37.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.180+0000 7fa5dec00640 1 --2- 172.21.15.28:0/997442141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5c00039e0 0x7fa5c0003e00 secure :-1 s=READY pgs=2715 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc001720 tx=0x7fa5c0005830 comp rx=0 tx=0).stop 2024-02-21T23:17:37.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.180+0000 7fa5de200640 1 -- 172.21.15.28:0/997442141 reap_dead start 2024-02-21T23:17:37.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.180+0000 7fa5dec00640 1 -- 172.21.15.28:0/997442141 shutdown_connections 2024-02-21T23:17:37.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.180+0000 7fa5dec00640 1 -- 172.21.15.28:0/997442141 >> 172.21.15.28:0/997442141 conn(0x7fa5d80ac370 msgr2=0x7fa5d80ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:37.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.180+0000 7fa5dec00640 1 -- 172.21.15.28:0/997442141 shutdown_connections 2024-02-21T23:17:37.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.180+0000 7fa5dec00640 1 -- 172.21.15.28:0/997442141 wait complete. 2024-02-21T23:17:37.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.182+0000 7fa5dec00640 1 Processor -- start 2024-02-21T23:17:37.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.182+0000 7fa5dec00640 1 -- start start 2024-02-21T23:17:37.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.183+0000 7fa5dec00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5c00039e0 0x7fa5d814f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:37.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.183+0000 7fa5dec00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa5d80beaf0 0x7fa5d81ee130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:37.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.183+0000 7fa5dec00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa5d80ae740 0x7fa5d81eb2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:37.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.183+0000 7fa5dec00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d80c58f0 con 0x7fa5c00039e0 2024-02-21T23:17:37.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.183+0000 7fa5dec00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d8158630 con 0x7fa5d80ae740 2024-02-21T23:17:37.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.183+0000 7fa5dec00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d80c5a70 con 0x7fa5d80beaf0 2024-02-21T23:17:37.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.183+0000 7fa5dd800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5c00039e0 0x7fa5d814f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:37.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.183+0000 7fa5dd800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5c00039e0 0x7fa5d814f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59642/0 (socket says 172.21.15.28:59642) 2024-02-21T23:17:37.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.183+0000 7fa5dd800640 1 -- 172.21.15.28:0/1594390531 learned_addr learned my addr 172.21.15.28:0/1594390531 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:37.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.184+0000 7fa5dd800640 1 -- 172.21.15.28:0/1594390531 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa5d80beaf0 msgr2=0x7fa5d81ee130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:37.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.184+0000 7fa5dd800640 1 --2- 172.21.15.28:0/1594390531 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa5d80beaf0 0x7fa5d81ee130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:37.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.184+0000 7fa5dd800640 1 -- 172.21.15.28:0/1594390531 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa5d80ae740 msgr2=0x7fa5d81eb2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:37.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.184+0000 7fa5dd800640 1 --2- 172.21.15.28:0/1594390531 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa5d80ae740 0x7fa5d81eb2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:37.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.184+0000 7fa5dd800640 1 -- 172.21.15.28:0/1594390531 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5d81ee810 con 0x7fa5c00039e0 2024-02-21T23:17:37.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.184+0000 7fa5dd800640 1 --2- 172.21.15.28:0/1594390531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5c00039e0 0x7fa5d814f5c0 secure :-1 s=READY pgs=2716 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc005040 tx=0x7fa5cc001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:37.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.184+0000 7fa5d5600640 1 -- 172.21.15.28:0/1594390531 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5cc041070 con 0x7fa5c00039e0 2024-02-21T23:17:37.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.185+0000 7fa5d5600640 1 -- 172.21.15.28:0/1594390531 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5cc005480 con 0x7fa5c00039e0 2024-02-21T23:17:37.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.185+0000 7fa5d5600640 1 -- 172.21.15.28:0/1594390531 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5cc007a30 con 0x7fa5c00039e0 2024-02-21T23:17:37.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.185+0000 7fa5dec00640 1 -- 172.21.15.28:0/1594390531 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5d81eeaf0 con 0x7fa5c00039e0 2024-02-21T23:17:37.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.185+0000 7fa5dec00640 1 -- 172.21.15.28:0/1594390531 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5d81eadd0 con 0x7fa5c00039e0 2024-02-21T23:17:37.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.186+0000 7fa5dec00640 1 -- 172.21.15.28:0/1594390531 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5d81ef010 con 0x7fa5c00039e0 2024-02-21T23:17:37.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.187+0000 7fa5d5600640 1 -- 172.21.15.28:0/1594390531 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa5cc044070 con 0x7fa5c00039e0 2024-02-21T23:17:37.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.188+0000 7fa5d5600640 1 --2- 172.21.15.28:0/1594390531 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa5b806b4d0 0x7fa5b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:37.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.189+0000 7fa5d5600640 1 -- 172.21.15.28:0/1594390531 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7fa5cc0ac4c0 con 0x7fa5c00039e0 2024-02-21T23:17:37.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.189+0000 7fa5dce00640 1 --2- 172.21.15.28:0/1594390531 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa5b806b4d0 0x7fa5b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:37.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.190+0000 7fa5dce00640 1 --2- 172.21.15.28:0/1594390531 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa5b806b4d0 0x7fa5b806d990 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7fa5d80bddf0 tx=0x7fa5c4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:37.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.193+0000 7fa5d5600640 1 -- 172.21.15.28:0/1594390531 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fa5d81ef010 con 0x7fa5c00039e0 2024-02-21T23:17:37.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.397+0000 7fa5dec00640 1 -- 172.21.15.28:0/1594390531 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa5d81eabc0 con 0x7fa5c00039e0 2024-02-21T23:17:37.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.398+0000 7fa5d5600640 1 -- 172.21.15.28:0/1594390531 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v212) v1 ==== 74+0+17227 (secure 0 0 0) 0x7fa5cc0052a0 con 0x7fa5c00039e0 2024-02-21T23:17:37.388 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:17:37.388 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":212,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:14:42.637299+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:13:27.612392+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"207","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_data","create_time":"2024-02-21T23:13:28.744308+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"208","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"34.b","mappings":[{"from":0,"to":1}]},{"pgid":"34.f","mappings":[{"from":0,"to":1}]},{"pgid":"34.31","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.81:6835/1607587459":"2024-02-22T23:14:42.637238+0000","172.21.15.28:6833/1864206570":"2024-02-22T23:14:07.624525+0000","172.21.15.81:6834/1607587459":"2024-02-22T23:14:42.637238+0000","172.21.15.28:6832/1864206570":"2024-02-22T23:14:07.624525+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:17:37.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.401+0000 7fa5dec00640 1 -- 172.21.15.28:0/1594390531 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa5b806b4d0 msgr2=0x7fa5b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:37.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.401+0000 7fa5dec00640 1 --2- 172.21.15.28:0/1594390531 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa5b806b4d0 0x7fa5b806d990 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7fa5d80bddf0 tx=0x7fa5c4008040 comp rx=0 tx=0).stop 2024-02-21T23:17:37.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.401+0000 7fa5dec00640 1 -- 172.21.15.28:0/1594390531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5c00039e0 msgr2=0x7fa5d814f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:37.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.401+0000 7fa5dec00640 1 --2- 172.21.15.28:0/1594390531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5c00039e0 0x7fa5d814f5c0 secure :-1 s=READY pgs=2716 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc005040 tx=0x7fa5cc001120 comp rx=0 tx=0).stop 2024-02-21T23:17:37.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.402+0000 7fa5dec00640 1 -- 172.21.15.28:0/1594390531 shutdown_connections 2024-02-21T23:17:37.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.402+0000 7fa5dec00640 1 --2- 172.21.15.28:0/1594390531 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa5b806b4d0 0x7fa5b806d990 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:37.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.402+0000 7fa5dec00640 1 --2- 172.21.15.28:0/1594390531 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa5d80beaf0 0x7fa5d81ee130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:37.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.402+0000 7fa5dec00640 1 --2- 172.21.15.28:0/1594390531 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa5d80ae740 0x7fa5d81eb2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:37.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.402+0000 7fa5dec00640 1 --2- 172.21.15.28:0/1594390531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5c00039e0 0x7fa5d814f5c0 unknown :-1 s=CLOSED pgs=2716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:37.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.402+0000 7fa5dec00640 1 -- 172.21.15.28:0/1594390531 >> 172.21.15.28:0/1594390531 conn(0x7fa5d80ac370 msgr2=0x7fa5d80ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:37.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.402+0000 7fa5dec00640 1 -- 172.21.15.28:0/1594390531 shutdown_connections 2024-02-21T23:17:37.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.402+0000 7fa5dec00640 1 -- 172.21.15.28:0/1594390531 wait complete. 2024-02-21T23:17:37.412 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-21T23:02:30.728329+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 33, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-21T23:13:27.612392+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '207', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 34, 'pool_name': 'cephfs_data', 'create_time': '2024-02-21T23:13:28.744308+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '208', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-21T23:17:37.413 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-21T23:17:37.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.641+0000 7f7d8e600640 1 Processor -- start 2024-02-21T23:17:37.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.641+0000 7f7d8e600640 1 -- start start 2024-02-21T23:17:37.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.643+0000 7f7d8e600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f7d880c5cc0 0x7f7d880c6c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:37.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.643+0000 7f7d8e600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7d880c3030 0x7f7d880c3460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:37.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.643+0000 7f7d8e600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d88159580 con 0x7f7d880c7590 2024-02-21T23:17:37.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.643+0000 7f7d8e600640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d880c6230 con 0x7f7d880c5cc0 2024-02-21T23:17:37.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.643+0000 7f7d8e600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d881587c0 con 0x7f7d880c3030 2024-02-21T23:17:37.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.643+0000 7f7d86a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f7d880c5cc0 0x7f7d880c6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:37.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.643+0000 7f7d87e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7d880c3030 0x7f7d880c3460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:37.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.644+0000 7f7d86a00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f7d880c5cc0 0x7f7d880c6c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.28:41622/0 (socket says 172.21.15.28:41622) 2024-02-21T23:17:37.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.644+0000 7f7d86a00640 1 -- 172.21.15.28:0/3820070553 learned_addr learned my addr 172.21.15.28:0/3820070553 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:37.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.644+0000 7f7d87e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7d880c3030 0x7f7d880c3460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3301/0 says I am v2:172.21.15.28:43788/0 (socket says 172.21.15.28:43788) 2024-02-21T23:17:37.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.644+0000 7f7d86a00640 1 -- 172.21.15.28:0/3820070553 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7d880c3030 msgr2=0x7f7d880c3460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:37.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.644+0000 7f7d86a00640 1 --2- 172.21.15.28:0/3820070553 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7d880c3030 0x7f7d880c3460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:37.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.644+0000 7f7d86a00640 1 -- 172.21.15.28:0/3820070553 >> v1:172.21.15.81:6789/0 conn(0x7f7d880c7590 legacy=0x7f7d881524e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:37.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.644+0000 7f7d86a00640 1 -- 172.21.15.28:0/3820070553 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d88158aa0 con 0x7f7d880c5cc0 2024-02-21T23:17:37.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.645+0000 7f7d87e00640 1 --2- 172.21.15.28:0/3820070553 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7d880c3030 0x7f7d880c3460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-21T23:17:37.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.646+0000 7f7d86a00640 1 --2- 172.21.15.28:0/3820070553 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7d880c5cc0 0x7f7d880c6c60 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f7d74009870 tx=0x7f7d7402b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=580535059ba484f9 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:37.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.647+0000 7f7d86000640 1 -- 172.21.15.28:0/3820070553 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d74035020 con 0x7f7d880c5cc0 2024-02-21T23:17:37.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.647+0000 7f7d86000640 1 -- 172.21.15.28:0/3820070553 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d74031640 con 0x7f7d880c5cc0 2024-02-21T23:17:37.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.647+0000 7f7d86000640 1 -- 172.21.15.28:0/3820070553 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d7402d040 con 0x7f7d880c5cc0 2024-02-21T23:17:37.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.647+0000 7f7d8e600640 1 -- 172.21.15.28:0/3820070553 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7d880c5cc0 msgr2=0x7f7d880c6c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:37.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.647+0000 7f7d8e600640 1 --2- 172.21.15.28:0/3820070553 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7d880c5cc0 0x7f7d880c6c60 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f7d74009870 tx=0x7f7d7402b8f0 comp rx=0 tx=0).stop 2024-02-21T23:17:37.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.648+0000 7f7d8e600640 1 -- 172.21.15.28:0/3820070553 shutdown_connections 2024-02-21T23:17:37.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.648+0000 7f7d8e600640 1 --2- 172.21.15.28:0/3820070553 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7d880c3030 0x7f7d880c3460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:37.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.648+0000 7f7d8e600640 1 --2- 172.21.15.28:0/3820070553 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7d880c5cc0 0x7f7d880c6c60 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:37.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.648+0000 7f7d8e600640 1 -- 172.21.15.28:0/3820070553 >> 172.21.15.28:0/3820070553 conn(0x7f7d8808afe0 msgr2=0x7f7d880725a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:37.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.648+0000 7f7d8e600640 1 -- 172.21.15.28:0/3820070553 shutdown_connections 2024-02-21T23:17:37.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.648+0000 7f7d8e600640 1 -- 172.21.15.28:0/3820070553 wait complete. 2024-02-21T23:17:37.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.651+0000 7f7d8e600640 1 Processor -- start 2024-02-21T23:17:37.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.651+0000 7f7d8e600640 1 -- start start 2024-02-21T23:17:37.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.653+0000 7f7d8e600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7d880c3030 0x7f7d88144340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:37.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.653+0000 7f7d8e600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7d880c5cc0 0x7f7d88156340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:37.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.653+0000 7f7d8e600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d880c7590 0x7f7d88157db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:37.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.654+0000 7f7d8e600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d880c6720 con 0x7f7d880c7590 2024-02-21T23:17:37.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.654+0000 7f7d8e600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d88159060 con 0x7f7d880c5cc0 2024-02-21T23:17:37.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.654+0000 7f7d8e600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d880c68a0 con 0x7f7d880c3030 2024-02-21T23:17:37.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.654+0000 7f7d87e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d880c7590 0x7f7d88157db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:37.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.654+0000 7f7d87e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d880c7590 0x7f7d88157db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59658/0 (socket says 172.21.15.28:59658) 2024-02-21T23:17:37.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.654+0000 7f7d87e00640 1 -- 172.21.15.28:0/2113243203 learned_addr learned my addr 172.21.15.28:0/2113243203 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:37.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.654+0000 7f7d87400640 1 --2- 172.21.15.28:0/2113243203 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7d880c3030 0x7f7d88144340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:37.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.654+0000 7f7d86a00640 1 --2- 172.21.15.28:0/2113243203 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7d880c5cc0 0x7f7d88156340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:37.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.655+0000 7f7d87e00640 1 -- 172.21.15.28:0/2113243203 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7d880c3030 msgr2=0x7f7d88144340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:37.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.655+0000 7f7d87e00640 1 --2- 172.21.15.28:0/2113243203 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7d880c3030 0x7f7d88144340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:37.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.655+0000 7f7d87e00640 1 -- 172.21.15.28:0/2113243203 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7d880c5cc0 msgr2=0x7f7d88156340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:37.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.655+0000 7f7d87e00640 1 --2- 172.21.15.28:0/2113243203 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7d880c5cc0 0x7f7d88156340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:37.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.655+0000 7f7d87e00640 1 -- 172.21.15.28:0/2113243203 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d88153980 con 0x7f7d880c7590 2024-02-21T23:17:37.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.655+0000 7f7d87e00640 1 --2- 172.21.15.28:0/2113243203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d880c7590 0x7f7d88157db0 secure :-1 s=READY pgs=2717 cs=0 l=1 rev1=1 crypto rx=0x7f7d8000efc0 tx=0x7f7d80009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:37.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.655+0000 7f7d7f400640 1 -- 172.21.15.28:0/2113243203 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d80018030 con 0x7f7d880c7590 2024-02-21T23:17:37.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.655+0000 7f7d7f400640 1 -- 172.21.15.28:0/2113243203 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d8000cde0 con 0x7f7d880c7590 2024-02-21T23:17:37.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.655+0000 7f7d8e600640 1 -- 172.21.15.28:0/2113243203 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d88144880 con 0x7f7d880c7590 2024-02-21T23:17:37.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.655+0000 7f7d7f400640 1 -- 172.21.15.28:0/2113243203 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d80003bb0 con 0x7f7d880c7590 2024-02-21T23:17:37.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.656+0000 7f7d8e600640 1 -- 172.21.15.28:0/2113243203 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d88156e70 con 0x7f7d880c7590 2024-02-21T23:17:37.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.657+0000 7f7d8e600640 1 -- 172.21.15.28:0/2113243203 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d88067f30 con 0x7f7d880c7590 2024-02-21T23:17:37.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.658+0000 7f7d7f400640 1 -- 172.21.15.28:0/2113243203 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7d80005d10 con 0x7f7d880c7590 2024-02-21T23:17:37.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.659+0000 7f7d7f400640 1 --2- 172.21.15.28:0/2113243203 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7d6806b4d0 0x7f7d6806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:37.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.659+0000 7f7d7f400640 1 -- 172.21.15.28:0/2113243203 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f7d8008aec0 con 0x7f7d880c7590 2024-02-21T23:17:37.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.659+0000 7f7d87400640 1 --2- 172.21.15.28:0/2113243203 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7d6806b4d0 0x7f7d6806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:37.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.661+0000 7f7d87400640 1 --2- 172.21.15.28:0/2113243203 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7d6806b4d0 0x7f7d6806d990 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f7d78000f60 tx=0x7f7d78009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:37.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.664+0000 7f7d7f400640 1 -- 172.21.15.28:0/2113243203 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7d80022070 con 0x7f7d880c7590 2024-02-21T23:17:37.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.863+0000 7f7d8e600640 1 -- 172.21.15.28:0/2113243203 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7d88152630 con 0x7f7d880c7590 2024-02-21T23:17:37.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.864+0000 7f7d7f400640 1 -- 172.21.15.28:0/2113243203 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v212) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7d88144880 con 0x7f7d880c7590 2024-02-21T23:17:37.852 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-02-21T23:17:37.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.866+0000 7f7d8e600640 1 -- 172.21.15.28:0/2113243203 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7d6806b4d0 msgr2=0x7f7d6806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:37.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.866+0000 7f7d8e600640 1 --2- 172.21.15.28:0/2113243203 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7d6806b4d0 0x7f7d6806d990 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f7d78000f60 tx=0x7f7d78009040 comp rx=0 tx=0).stop 2024-02-21T23:17:37.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.866+0000 7f7d8e600640 1 -- 172.21.15.28:0/2113243203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d880c7590 msgr2=0x7f7d88157db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:37.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.866+0000 7f7d8e600640 1 --2- 172.21.15.28:0/2113243203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d880c7590 0x7f7d88157db0 secure :-1 s=READY pgs=2717 cs=0 l=1 rev1=1 crypto rx=0x7f7d8000efc0 tx=0x7f7d80009f90 comp rx=0 tx=0).stop 2024-02-21T23:17:37.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.867+0000 7f7d8e600640 1 -- 172.21.15.28:0/2113243203 shutdown_connections 2024-02-21T23:17:37.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.867+0000 7f7d8e600640 1 --2- 172.21.15.28:0/2113243203 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7d6806b4d0 0x7f7d6806d990 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:37.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.867+0000 7f7d8e600640 1 --2- 172.21.15.28:0/2113243203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d880c7590 0x7f7d88157db0 unknown :-1 s=CLOSED pgs=2717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:37.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.867+0000 7f7d8e600640 1 --2- 172.21.15.28:0/2113243203 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7d880c5cc0 0x7f7d88156340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:37.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.867+0000 7f7d8e600640 1 --2- 172.21.15.28:0/2113243203 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7d880c3030 0x7f7d88144340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:37.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.867+0000 7f7d8e600640 1 -- 172.21.15.28:0/2113243203 >> 172.21.15.28:0/2113243203 conn(0x7f7d8808afe0 msgr2=0x7f7d8814ff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:37.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.867+0000 7f7d8e600640 1 -- 172.21.15.28:0/2113243203 shutdown_connections 2024-02-21T23:17:37.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:37.867+0000 7f7d8e600640 1 -- 172.21.15.28:0/2113243203 wait complete. 2024-02-21T23:17:37.882 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-21T23:17:38.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.124+0000 7ff513000640 1 Processor -- start 2024-02-21T23:17:38.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.124+0000 7ff513000640 1 -- start start 2024-02-21T23:17:38.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.126+0000 7ff513000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7ff50c13e0c0 0x7ff50c0c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:38.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.126+0000 7ff513000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff50c06dbc0 0x7ff50c06c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:38.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.126+0000 7ff513000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff50c158550 con 0x7ff50c0c76f0 2024-02-21T23:17:38.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.126+0000 7ff513000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff50c0c89a0 con 0x7ff50c13e0c0 2024-02-21T23:17:38.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.126+0000 7ff513000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff50c1587d0 con 0x7ff50c06dbc0 2024-02-21T23:17:38.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.126+0000 7ff512600640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff50c0c76f0 0x7ff50c06c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59818/0 (socket says 172.21.15.28:59818) 2024-02-21T23:17:38.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.126+0000 7ff512600640 1 -- 172.21.15.28:0/2387307287 learned_addr learned my addr 172.21.15.28:0/2387307287 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:17:38.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.127+0000 7ff511200640 1 --2- 172.21.15.28:0/2387307287 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff50c06dbc0 0x7ff50c06c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:38.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.127+0000 7ff511c00640 1 --2- 172.21.15.28:0/2387307287 >> v2:172.21.15.81:3300/0 conn(0x7ff50c13e0c0 0x7ff50c0c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:38.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.127+0000 7ff50be00640 1 -- 172.21.15.28:0/2387307287 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2348984824 0 0) 0x7ff504001490 con 0x7ff50c0c76f0 2024-02-21T23:17:38.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.127+0000 7ff50be00640 1 -- 172.21.15.28:0/2387307287 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4f4003520 con 0x7ff50c0c76f0 2024-02-21T23:17:38.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.127+0000 7ff511200640 1 -- 172.21.15.28:0/2387307287 >> v2:172.21.15.81:3300/0 conn(0x7ff50c13e0c0 msgr2=0x7ff50c0c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:38.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.127+0000 7ff511200640 1 --2- 172.21.15.28:0/2387307287 >> v2:172.21.15.81:3300/0 conn(0x7ff50c13e0c0 0x7ff50c0c71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:38.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.127+0000 7ff511200640 1 -- 172.21.15.28:0/2387307287 >> v1:172.21.15.28:6789/0 conn(0x7ff50c0c76f0 legacy=0x7ff50c06c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:38.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.127+0000 7ff50be00640 1 -- 172.21.15.28:0/2387307287 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 503169700 0 0) 0x7ff4f4003520 con 0x7ff50c0c76f0 2024-02-21T23:17:38.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.127+0000 7ff511200640 1 -- 172.21.15.28:0/2387307287 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff50c158ab0 con 0x7ff50c06dbc0 2024-02-21T23:17:38.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.129+0000 7ff511200640 1 --2- 172.21.15.28:0/2387307287 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff50c06dbc0 0x7ff50c06c280 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc0097a0 tx=0x7ff4fc02b710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c130324381cafa1 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:38.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.129+0000 7ff50be00640 1 -- 172.21.15.28:0/2387307287 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4fc037070 con 0x7ff50c06dbc0 2024-02-21T23:17:38.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.129+0000 7ff50be00640 1 -- 172.21.15.28:0/2387307287 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4fc030510 con 0x7ff50c06dbc0 2024-02-21T23:17:38.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.129+0000 7ff50be00640 1 -- 172.21.15.28:0/2387307287 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff504001bc0 con 0x7ff50c06dbc0 2024-02-21T23:17:38.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.130+0000 7ff513000640 1 -- 172.21.15.28:0/2387307287 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff50c06dbc0 msgr2=0x7ff50c06c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:38.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.130+0000 7ff513000640 1 --2- 172.21.15.28:0/2387307287 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff50c06dbc0 0x7ff50c06c280 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc0097a0 tx=0x7ff4fc02b710 comp rx=0 tx=0).stop 2024-02-21T23:17:38.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.130+0000 7ff513000640 1 -- 172.21.15.28:0/2387307287 shutdown_connections 2024-02-21T23:17:38.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.130+0000 7ff513000640 1 --2- 172.21.15.28:0/2387307287 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff50c06dbc0 0x7ff50c06c280 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:38.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.130+0000 7ff513000640 1 --2- 172.21.15.28:0/2387307287 >> v2:172.21.15.81:3300/0 conn(0x7ff50c13e0c0 0x7ff50c0c71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:38.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.130+0000 7ff513000640 1 -- 172.21.15.28:0/2387307287 >> 172.21.15.28:0/2387307287 conn(0x7ff50c073110 msgr2=0x7ff50c0ac490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:38.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.130+0000 7ff513000640 1 -- 172.21.15.28:0/2387307287 shutdown_connections 2024-02-21T23:17:38.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.130+0000 7ff513000640 1 -- 172.21.15.28:0/2387307287 wait complete. 2024-02-21T23:17:38.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.132+0000 7ff513000640 1 Processor -- start 2024-02-21T23:17:38.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.132+0000 7ff513000640 1 -- start start 2024-02-21T23:17:38.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.133+0000 7ff513000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff50c06dbc0 0x7ff50c157640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:38.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.133+0000 7ff513000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff50c0c76f0 0x7ff50c133da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:38.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.133+0000 7ff513000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff50c13e0c0 0x7ff50c1eaaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:38.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.133+0000 7ff513000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff50c06ced0 con 0x7ff50c13e0c0 2024-02-21T23:17:38.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.133+0000 7ff513000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff50c159070 con 0x7ff50c0c76f0 2024-02-21T23:17:38.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.133+0000 7ff513000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff50c13e520 con 0x7ff50c06dbc0 2024-02-21T23:17:38.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.133+0000 7ff512600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff50c13e0c0 0x7ff50c1eaaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:38.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.133+0000 7ff512600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff50c13e0c0 0x7ff50c1eaaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59666/0 (socket says 172.21.15.28:59666) 2024-02-21T23:17:38.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.133+0000 7ff512600640 1 -- 172.21.15.28:0/2563475172 learned_addr learned my addr 172.21.15.28:0/2563475172 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:38.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.133+0000 7ff511c00640 1 --2- 172.21.15.28:0/2563475172 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff50c06dbc0 0x7ff50c157640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:38.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.133+0000 7ff511200640 1 --2- 172.21.15.28:0/2563475172 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff50c0c76f0 0x7ff50c133da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:38.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.134+0000 7ff512600640 1 -- 172.21.15.28:0/2563475172 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff50c06dbc0 msgr2=0x7ff50c157640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:38.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.134+0000 7ff512600640 1 --2- 172.21.15.28:0/2563475172 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff50c06dbc0 0x7ff50c157640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:38.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.134+0000 7ff512600640 1 -- 172.21.15.28:0/2563475172 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff50c0c76f0 msgr2=0x7ff50c133da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:38.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.134+0000 7ff512600640 1 --2- 172.21.15.28:0/2563475172 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff50c0c76f0 0x7ff50c133da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:38.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.134+0000 7ff512600640 1 -- 172.21.15.28:0/2563475172 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff50c1edf40 con 0x7ff50c13e0c0 2024-02-21T23:17:38.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.134+0000 7ff512600640 1 --2- 172.21.15.28:0/2563475172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff50c13e0c0 0x7ff50c1eaaa0 secure :-1 s=READY pgs=2719 cs=0 l=1 rev1=1 crypto rx=0x7ff504001a90 tx=0x7ff50400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:38.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.134+0000 7ff509600640 1 -- 172.21.15.28:0/2563475172 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff504012400 con 0x7ff50c13e0c0 2024-02-21T23:17:38.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.134+0000 7ff509600640 1 -- 172.21.15.28:0/2563475172 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5040125c0 con 0x7ff50c13e0c0 2024-02-21T23:17:38.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.134+0000 7ff513000640 1 -- 172.21.15.28:0/2563475172 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff50c1ee220 con 0x7ff50c13e0c0 2024-02-21T23:17:38.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.134+0000 7ff509600640 1 -- 172.21.15.28:0/2563475172 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5040129b0 con 0x7ff50c13e0c0 2024-02-21T23:17:38.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.134+0000 7ff513000640 1 -- 172.21.15.28:0/2563475172 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff50c1ee7e0 con 0x7ff50c13e0c0 2024-02-21T23:17:38.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.136+0000 7ff513000640 1 -- 172.21.15.28:0/2563475172 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff50c04f290 con 0x7ff50c13e0c0 2024-02-21T23:17:38.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.137+0000 7ff509600640 1 -- 172.21.15.28:0/2563475172 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff504012b50 con 0x7ff50c13e0c0 2024-02-21T23:17:38.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.139+0000 7ff509600640 1 --2- 172.21.15.28:0/2563475172 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff4f006b4d0 0x7ff4f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:38.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.139+0000 7ff509600640 1 -- 172.21.15.28:0/2563475172 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7ff50408ad80 con 0x7ff50c13e0c0 2024-02-21T23:17:38.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.139+0000 7ff511c00640 1 --2- 172.21.15.28:0/2563475172 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff4f006b4d0 0x7ff4f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:38.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.140+0000 7ff511c00640 1 --2- 172.21.15.28:0/2563475172 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff4f006b4d0 0x7ff4f006d990 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7ff500009a30 tx=0x7ff500008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:38.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.144+0000 7ff509600640 1 -- 172.21.15.28:0/2563475172 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ff50c158550 con 0x7ff50c13e0c0 2024-02-21T23:17:38.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.348+0000 7ff513000640 1 -- 172.21.15.28:0/2563475172 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff50c06d9a0 con 0x7ff50c13e0c0 2024-02-21T23:17:38.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.348+0000 7ff509600640 1 -- 172.21.15.28:0/2563475172 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v212) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff5040182b0 con 0x7ff50c13e0c0 2024-02-21T23:17:38.337 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-02-21T23:17:38.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.351+0000 7ff513000640 1 -- 172.21.15.28:0/2563475172 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff4f006b4d0 msgr2=0x7ff4f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:38.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.351+0000 7ff513000640 1 --2- 172.21.15.28:0/2563475172 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff4f006b4d0 0x7ff4f006d990 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7ff500009a30 tx=0x7ff500008040 comp rx=0 tx=0).stop 2024-02-21T23:17:38.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.351+0000 7ff513000640 1 -- 172.21.15.28:0/2563475172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff50c13e0c0 msgr2=0x7ff50c1eaaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:38.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.351+0000 7ff513000640 1 --2- 172.21.15.28:0/2563475172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff50c13e0c0 0x7ff50c1eaaa0 secure :-1 s=READY pgs=2719 cs=0 l=1 rev1=1 crypto rx=0x7ff504001a90 tx=0x7ff50400a430 comp rx=0 tx=0).stop 2024-02-21T23:17:38.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.351+0000 7ff513000640 1 -- 172.21.15.28:0/2563475172 shutdown_connections 2024-02-21T23:17:38.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.351+0000 7ff513000640 1 --2- 172.21.15.28:0/2563475172 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff4f006b4d0 0x7ff4f006d990 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:38.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.351+0000 7ff513000640 1 --2- 172.21.15.28:0/2563475172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff50c13e0c0 0x7ff50c1eaaa0 unknown :-1 s=CLOSED pgs=2719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:38.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.351+0000 7ff513000640 1 --2- 172.21.15.28:0/2563475172 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff50c0c76f0 0x7ff50c133da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:38.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.351+0000 7ff513000640 1 --2- 172.21.15.28:0/2563475172 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff50c06dbc0 0x7ff50c157640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:38.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.351+0000 7ff513000640 1 -- 172.21.15.28:0/2563475172 >> 172.21.15.28:0/2563475172 conn(0x7ff50c073110 msgr2=0x7ff50c13c9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:38.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.351+0000 7ff513000640 1 -- 172.21.15.28:0/2563475172 shutdown_connections 2024-02-21T23:17:38.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.351+0000 7ff513000640 1 -- 172.21.15.28:0/2563475172 wait complete. 2024-02-21T23:17:38.359 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-21T23:17:38.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.584+0000 7f520be00640 1 Processor -- start 2024-02-21T23:17:38.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.584+0000 7f520be00640 1 -- start start 2024-02-21T23:17:38.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.586+0000 7f520be00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f520c13e0c0 0x7f520c0c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:38.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.586+0000 7f520be00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f520c158550 con 0x7f520c0c76f0 2024-02-21T23:17:38.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.586+0000 7f520be00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f520c1587d0 con 0x7f520c06dbc0 2024-02-21T23:17:38.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.586+0000 7f520be00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f520c0c89a0 con 0x7f520c13e0c0 2024-02-21T23:17:38.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.586+0000 7f520b400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f520c0c76f0 0x7f520c06c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59832/0 (socket says 172.21.15.28:59832) 2024-02-21T23:17:38.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.586+0000 7f520b400640 1 -- 172.21.15.28:0/1208923555 learned_addr learned my addr 172.21.15.28:0/1208923555 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:17:38.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.586+0000 7f520aa00640 1 --2- 172.21.15.28:0/1208923555 >> v2:172.21.15.81:3300/0 conn(0x7f520c13e0c0 0x7f520c0c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:38.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.587+0000 7f5209600640 1 -- 172.21.15.28:0/1208923555 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2707755787 0 0) 0x7f5204001490 con 0x7f520c0c76f0 2024-02-21T23:17:38.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.587+0000 7f5209600640 1 -- 172.21.15.28:0/1208923555 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51f0003520 con 0x7f520c0c76f0 2024-02-21T23:17:38.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.587+0000 7f520aa00640 1 -- 172.21.15.28:0/1208923555 >> v1:172.21.15.81:6789/0 conn(0x7f520c06dbc0 legacy=0x7f520c06c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:38.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.587+0000 7f520aa00640 1 -- 172.21.15.28:0/1208923555 >> v1:172.21.15.28:6789/0 conn(0x7f520c0c76f0 legacy=0x7f520c06c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:38.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.587+0000 7f5209600640 1 -- 172.21.15.28:0/1208923555 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2191430730 0 0) 0x7f51f0003520 con 0x7f520c0c76f0 2024-02-21T23:17:38.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.587+0000 7f520aa00640 1 -- 172.21.15.28:0/1208923555 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f520c158ab0 con 0x7f520c13e0c0 2024-02-21T23:17:38.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.589+0000 7f520aa00640 1 --2- 172.21.15.28:0/1208923555 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f520c13e0c0 0x7f520c0c71b0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f51fc0097a0 tx=0x7f51fc02b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5052fa1fadfbc498 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:38.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.589+0000 7f5209600640 1 -- 172.21.15.28:0/1208923555 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51fc037070 con 0x7f520c13e0c0 2024-02-21T23:17:38.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.589+0000 7f5209600640 1 -- 172.21.15.28:0/1208923555 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51fc030510 con 0x7f520c13e0c0 2024-02-21T23:17:38.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.589+0000 7f5209600640 1 -- 172.21.15.28:0/1208923555 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5204001bc0 con 0x7f520c13e0c0 2024-02-21T23:17:38.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.590+0000 7f520be00640 1 -- 172.21.15.28:0/1208923555 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f520c13e0c0 msgr2=0x7f520c0c71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:38.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.590+0000 7f520be00640 1 --2- 172.21.15.28:0/1208923555 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f520c13e0c0 0x7f520c0c71b0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f51fc0097a0 tx=0x7f51fc02b710 comp rx=0 tx=0).stop 2024-02-21T23:17:38.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.590+0000 7f520be00640 1 -- 172.21.15.28:0/1208923555 shutdown_connections 2024-02-21T23:17:38.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.590+0000 7f520be00640 1 --2- 172.21.15.28:0/1208923555 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f520c13e0c0 0x7f520c0c71b0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:38.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.590+0000 7f520be00640 1 -- 172.21.15.28:0/1208923555 >> 172.21.15.28:0/1208923555 conn(0x7f520c073110 msgr2=0x7f520c0ac490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:38.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.590+0000 7f520be00640 1 -- 172.21.15.28:0/1208923555 shutdown_connections 2024-02-21T23:17:38.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.590+0000 7f520be00640 1 -- 172.21.15.28:0/1208923555 wait complete. 2024-02-21T23:17:38.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.592+0000 7f520be00640 1 Processor -- start 2024-02-21T23:17:38.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.592+0000 7f520be00640 1 -- start start 2024-02-21T23:17:38.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.593+0000 7f520be00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f520c06dbc0 0x7f520c13af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:38.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.593+0000 7f520be00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f520c0c76f0 0x7f520c0add40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:38.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.593+0000 7f520be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f520c13e0c0 0x7f520c139360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:38.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.593+0000 7f520be00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f520c13e520 con 0x7f520c13e0c0 2024-02-21T23:17:38.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.593+0000 7f520be00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f520c159070 con 0x7f520c06dbc0 2024-02-21T23:17:38.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.593+0000 7f520be00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f520c13e6a0 con 0x7f520c0c76f0 2024-02-21T23:17:38.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.593+0000 7f520b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f520c13e0c0 0x7f520c139360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:38.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.593+0000 7f520b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f520c13e0c0 0x7f520c139360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59682/0 (socket says 172.21.15.28:59682) 2024-02-21T23:17:38.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.593+0000 7f520b400640 1 -- 172.21.15.28:0/1136737507 learned_addr learned my addr 172.21.15.28:0/1136737507 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:38.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.594+0000 7f520aa00640 1 --2- 172.21.15.28:0/1136737507 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f520c06dbc0 0x7f520c13af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:38.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.594+0000 7f520a000640 1 --2- 172.21.15.28:0/1136737507 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f520c0c76f0 0x7f520c0add40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:38.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.594+0000 7f520b400640 1 -- 172.21.15.28:0/1136737507 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f520c0c76f0 msgr2=0x7f520c0add40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:38.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.594+0000 7f520b400640 1 --2- 172.21.15.28:0/1136737507 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f520c0c76f0 0x7f520c0add40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:38.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.594+0000 7f520b400640 1 -- 172.21.15.28:0/1136737507 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f520c06dbc0 msgr2=0x7f520c13af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:38.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.594+0000 7f520b400640 1 --2- 172.21.15.28:0/1136737507 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f520c06dbc0 0x7f520c13af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:38.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.594+0000 7f520b400640 1 -- 172.21.15.28:0/1136737507 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f520c13b480 con 0x7f520c13e0c0 2024-02-21T23:17:38.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.594+0000 7f520aa00640 1 --2- 172.21.15.28:0/1136737507 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f520c06dbc0 0x7f520c13af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:17:38.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.594+0000 7f520b400640 1 --2- 172.21.15.28:0/1136737507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f520c13e0c0 0x7f520c139360 secure :-1 s=READY pgs=2721 cs=0 l=1 rev1=1 crypto rx=0x7f520400cc40 tx=0x7f520400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:38.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.595+0000 7f5203400640 1 -- 172.21.15.28:0/1136737507 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f520400ce50 con 0x7f520c13e0c0 2024-02-21T23:17:38.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.595+0000 7f5203400640 1 -- 172.21.15.28:0/1136737507 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5204012400 con 0x7f520c13e0c0 2024-02-21T23:17:38.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.595+0000 7f5203400640 1 -- 172.21.15.28:0/1136737507 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5204012810 con 0x7f520c13e0c0 2024-02-21T23:17:38.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.595+0000 7f520be00640 1 -- 172.21.15.28:0/1136737507 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f520c139f10 con 0x7f520c13e0c0 2024-02-21T23:17:38.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.595+0000 7f520be00640 1 -- 172.21.15.28:0/1136737507 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f520c0ae280 con 0x7f520c13e0c0 2024-02-21T23:17:38.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.596+0000 7f520be00640 1 -- 172.21.15.28:0/1136737507 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f520c13a2c0 con 0x7f520c13e0c0 2024-02-21T23:17:38.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.598+0000 7f5203400640 1 -- 172.21.15.28:0/1136737507 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f52040129b0 con 0x7f520c13e0c0 2024-02-21T23:17:38.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.600+0000 7f5203400640 1 --2- 172.21.15.28:0/1136737507 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f51ec06b670 0x7f51ec06db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:38.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.600+0000 7f5203400640 1 -- 172.21.15.28:0/1136737507 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f520408ad10 con 0x7f520c13e0c0 2024-02-21T23:17:38.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.600+0000 7f520aa00640 1 --2- 172.21.15.28:0/1136737507 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f51ec06b670 0x7f51ec06db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:38.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.601+0000 7f520aa00640 1 --2- 172.21.15.28:0/1136737507 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f51ec06b670 0x7f51ec06db30 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f51fc034000 tx=0x7f51fc008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:38.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.604+0000 7f5203400640 1 -- 172.21.15.28:0/1136737507 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f5204090020 con 0x7f520c13e0c0 2024-02-21T23:17:38.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.811+0000 7f520be00640 1 -- 172.21.15.28:0/1136737507 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f520c04f290 con 0x7f520c13e0c0 2024-02-21T23:17:38.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.812+0000 7f5203400640 1 -- 172.21.15.28:0/1136737507 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v212) v1 ==== 100+0+11 (secure 0 0 0) 0x7f520c158550 con 0x7f520c13e0c0 2024-02-21T23:17:38.800 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-02-21T23:17:38.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.815+0000 7f520be00640 1 -- 172.21.15.28:0/1136737507 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f51ec06b670 msgr2=0x7f51ec06db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:38.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.815+0000 7f520be00640 1 --2- 172.21.15.28:0/1136737507 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f51ec06b670 0x7f51ec06db30 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f51fc034000 tx=0x7f51fc008030 comp rx=0 tx=0).stop 2024-02-21T23:17:38.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.815+0000 7f520be00640 1 -- 172.21.15.28:0/1136737507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f520c13e0c0 msgr2=0x7f520c139360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:38.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.815+0000 7f520be00640 1 --2- 172.21.15.28:0/1136737507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f520c13e0c0 0x7f520c139360 secure :-1 s=READY pgs=2721 cs=0 l=1 rev1=1 crypto rx=0x7f520400cc40 tx=0x7f520400a430 comp rx=0 tx=0).stop 2024-02-21T23:17:38.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.815+0000 7f520be00640 1 -- 172.21.15.28:0/1136737507 shutdown_connections 2024-02-21T23:17:38.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.815+0000 7f520be00640 1 --2- 172.21.15.28:0/1136737507 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f51ec06b670 0x7f51ec06db30 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:38.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.815+0000 7f520be00640 1 --2- 172.21.15.28:0/1136737507 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f520c0c76f0 0x7f520c0add40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:38.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.815+0000 7f520be00640 1 --2- 172.21.15.28:0/1136737507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f520c13e0c0 0x7f520c139360 unknown :-1 s=CLOSED pgs=2721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:38.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.815+0000 7f520be00640 1 --2- 172.21.15.28:0/1136737507 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f520c06dbc0 0x7f520c13af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:38.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.815+0000 7f520be00640 1 -- 172.21.15.28:0/1136737507 >> 172.21.15.28:0/1136737507 conn(0x7f520c073110 msgr2=0x7f520c13c150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:38.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.815+0000 7f520be00640 1 -- 172.21.15.28:0/1136737507 shutdown_connections 2024-02-21T23:17:38.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:38.815+0000 7f520be00640 1 -- 172.21.15.28:0/1136737507 wait complete. 2024-02-21T23:17:38.822 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:17:39.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.051+0000 7f4ebbe00640 1 Processor -- start 2024-02-21T23:17:39.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.052+0000 7f4ebbe00640 1 -- start start 2024-02-21T23:17:39.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.053+0000 7f4ebbe00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4ebc0aff90 0x7f4ebc0c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:39.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.053+0000 7f4ebbe00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ebc0c74c0 con 0x7f4ebc06dbf0 2024-02-21T23:17:39.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.053+0000 7f4ebbe00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ebc0c77a0 con 0x7f4ebc0c5720 2024-02-21T23:17:39.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.053+0000 7f4ebbe00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ebc0c7270 con 0x7f4ebc0aff90 2024-02-21T23:17:39.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.054+0000 7f4eba000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f4ebc06dbf0 0x7f4ebc158ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59838/0 (socket says 172.21.15.28:59838) 2024-02-21T23:17:39.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.054+0000 7f4eba000640 1 -- 172.21.15.28:0/1747554585 learned_addr learned my addr 172.21.15.28:0/1747554585 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:17:39.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.054+0000 7f4ebaa00640 1 --2- 172.21.15.28:0/1747554585 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4ebc0aff90 0x7f4ebc0c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:39.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.054+0000 7f4eb9600640 1 -- 172.21.15.28:0/1747554585 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3030784251 0 0) 0x7f4ebc0c74c0 con 0x7f4ebc06dbf0 2024-02-21T23:17:39.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.054+0000 7f4eb9600640 1 -- 172.21.15.28:0/1747554585 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ea0003520 con 0x7f4ebc06dbf0 2024-02-21T23:17:39.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.055+0000 7f4ebaa00640 1 -- 172.21.15.28:0/1747554585 >> v1:172.21.15.81:6789/0 conn(0x7f4ebc0c5720 legacy=0x7f4ebc15f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:39.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.055+0000 7f4ebaa00640 1 -- 172.21.15.28:0/1747554585 >> v1:172.21.15.28:6789/0 conn(0x7f4ebc06dbf0 legacy=0x7f4ebc158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:39.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.055+0000 7f4eb9600640 1 -- 172.21.15.28:0/1747554585 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 385042598 0 0) 0x7f4ea0003520 con 0x7f4ebc06dbf0 2024-02-21T23:17:39.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.055+0000 7f4ebaa00640 1 -- 172.21.15.28:0/1747554585 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ebc0c7a80 con 0x7f4ebc0aff90 2024-02-21T23:17:39.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.056+0000 7f4ebaa00640 1 --2- 172.21.15.28:0/1747554585 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4ebc0aff90 0x7f4ebc0c51e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f4eac0097d0 tx=0x7f4eac02b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e86703f7e4d4168 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:39.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.056+0000 7f4eb9600640 1 -- 172.21.15.28:0/1747554585 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eac037070 con 0x7f4ebc0aff90 2024-02-21T23:17:39.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.057+0000 7f4eb9600640 1 -- 172.21.15.28:0/1747554585 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4eac030510 con 0x7f4ebc0aff90 2024-02-21T23:17:39.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.057+0000 7f4eb9600640 1 -- 172.21.15.28:0/1747554585 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea8001670 con 0x7f4ebc0aff90 2024-02-21T23:17:39.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.057+0000 7f4ebbe00640 1 -- 172.21.15.28:0/1747554585 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4ebc0aff90 msgr2=0x7f4ebc0c51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:39.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.057+0000 7f4ebbe00640 1 --2- 172.21.15.28:0/1747554585 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4ebc0aff90 0x7f4ebc0c51e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f4eac0097d0 tx=0x7f4eac02b800 comp rx=0 tx=0).stop 2024-02-21T23:17:39.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.057+0000 7f4ebbe00640 1 -- 172.21.15.28:0/1747554585 shutdown_connections 2024-02-21T23:17:39.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.057+0000 7f4ebbe00640 1 --2- 172.21.15.28:0/1747554585 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4ebc0aff90 0x7f4ebc0c51e0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:39.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.057+0000 7f4ebbe00640 1 -- 172.21.15.28:0/1747554585 >> 172.21.15.28:0/1747554585 conn(0x7f4ebc0ac370 msgr2=0x7f4ebc0ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:39.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.057+0000 7f4ebbe00640 1 -- 172.21.15.28:0/1747554585 shutdown_connections 2024-02-21T23:17:39.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.057+0000 7f4ebbe00640 1 -- 172.21.15.28:0/1747554585 wait complete. 2024-02-21T23:17:39.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.059+0000 7f4ebbe00640 1 Processor -- start 2024-02-21T23:17:39.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.059+0000 7f4ebbe00640 1 -- start start 2024-02-21T23:17:39.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.060+0000 7f4ebbe00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4ebc06dbf0 0x7f4ebc1eaea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:39.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.060+0000 7f4ebbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ebc0aff90 0x7f4ebc0c20d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:39.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.060+0000 7f4ebbe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ebc0c5720 0x7f4ebc1ee320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:39.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.060+0000 7f4ebbe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ebc0c8760 con 0x7f4ebc0aff90 2024-02-21T23:17:39.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.060+0000 7f4ebbe00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ebc158630 con 0x7f4ebc0c5720 2024-02-21T23:17:39.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.060+0000 7f4ebbe00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ebc0c88e0 con 0x7f4ebc06dbf0 2024-02-21T23:17:39.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.060+0000 7f4eba000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ebc0aff90 0x7f4ebc0c20d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:39.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.060+0000 7f4eba000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ebc0aff90 0x7f4ebc0c20d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59696/0 (socket says 172.21.15.28:59696) 2024-02-21T23:17:39.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.060+0000 7f4eba000640 1 -- 172.21.15.28:0/2020082120 learned_addr learned my addr 172.21.15.28:0/2020082120 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:39.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.061+0000 7f4ebb400640 1 --2- 172.21.15.28:0/2020082120 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ebc0c5720 0x7f4ebc1ee320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:39.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.061+0000 7f4ebaa00640 1 --2- 172.21.15.28:0/2020082120 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4ebc06dbf0 0x7f4ebc1eaea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:39.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.061+0000 7f4eba000640 1 -- 172.21.15.28:0/2020082120 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4ebc06dbf0 msgr2=0x7f4ebc1eaea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:39.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.061+0000 7f4eba000640 1 --2- 172.21.15.28:0/2020082120 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4ebc06dbf0 0x7f4ebc1eaea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:39.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.061+0000 7f4eba000640 1 -- 172.21.15.28:0/2020082120 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ebc0c5720 msgr2=0x7f4ebc1ee320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:39.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.061+0000 7f4eba000640 1 --2- 172.21.15.28:0/2020082120 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ebc0c5720 0x7f4ebc1ee320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:39.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.061+0000 7f4eba000640 1 -- 172.21.15.28:0/2020082120 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ebc0c2610 con 0x7f4ebc0aff90 2024-02-21T23:17:39.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.061+0000 7f4eba000640 1 --2- 172.21.15.28:0/2020082120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ebc0aff90 0x7f4ebc0c20d0 secure :-1 s=READY pgs=2723 cs=0 l=1 rev1=1 crypto rx=0x7f4ea800c6f0 tx=0x7f4ea800cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:39.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.062+0000 7f4eb2a00640 1 -- 172.21.15.28:0/2020082120 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea8005ad0 con 0x7f4ebc0aff90 2024-02-21T23:17:39.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.062+0000 7f4eb2a00640 1 -- 172.21.15.28:0/2020082120 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ea8005c70 con 0x7f4ebc0aff90 2024-02-21T23:17:39.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.062+0000 7f4eb2a00640 1 -- 172.21.15.28:0/2020082120 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea8007270 con 0x7f4ebc0aff90 2024-02-21T23:17:39.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.062+0000 7f4ebbe00640 1 -- 172.21.15.28:0/2020082120 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ebc1ee860 con 0x7f4ebc0aff90 2024-02-21T23:17:39.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.062+0000 7f4ebbe00640 1 -- 172.21.15.28:0/2020082120 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ebc154e70 con 0x7f4ebc0aff90 2024-02-21T23:17:39.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.063+0000 7f4ebbe00640 1 -- 172.21.15.28:0/2020082120 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ebc04f290 con 0x7f4ebc0aff90 2024-02-21T23:17:39.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.064+0000 7f4eb2a00640 1 -- 172.21.15.28:0/2020082120 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4ea8007410 con 0x7f4ebc0aff90 2024-02-21T23:17:39.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.065+0000 7f4eb2a00640 1 --2- 172.21.15.28:0/2020082120 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4e9c06b500 0x7f4e9c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:39.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.065+0000 7f4eb2a00640 1 -- 172.21.15.28:0/2020082120 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f4ea808a2d0 con 0x7f4ebc0aff90 2024-02-21T23:17:39.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.066+0000 7f4ebaa00640 1 --2- 172.21.15.28:0/2020082120 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4e9c06b500 0x7f4e9c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:39.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.067+0000 7f4ebaa00640 1 --2- 172.21.15.28:0/2020082120 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4e9c06b500 0x7f4e9c06d9c0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f4eac034000 tx=0x7f4eac008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:39.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.069+0000 7f4eb2a00640 1 -- 172.21.15.28:0/2020082120 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f4ea8058eb0 con 0x7f4ebc0aff90 2024-02-21T23:17:39.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.315+0000 7f4ebbe00640 1 -- 172.21.15.28:0/2020082120 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4ebc06d9d0 con 0x7f4ebc0aff90 2024-02-21T23:17:39.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.316+0000 7f4eb2a00640 1 -- 172.21.15.28:0/2020082120 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7f4ea80587b0 con 0x7f4ebc0aff90 2024-02-21T23:17:39.304 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:17:39.304 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:17:39.304 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:17:39.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.318+0000 7f4ebbe00640 1 -- 172.21.15.28:0/2020082120 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4e9c06b500 msgr2=0x7f4e9c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:39.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.318+0000 7f4ebbe00640 1 --2- 172.21.15.28:0/2020082120 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4e9c06b500 0x7f4e9c06d9c0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f4eac034000 tx=0x7f4eac008030 comp rx=0 tx=0).stop 2024-02-21T23:17:39.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.318+0000 7f4ebbe00640 1 -- 172.21.15.28:0/2020082120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ebc0aff90 msgr2=0x7f4ebc0c20d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:39.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.318+0000 7f4ebbe00640 1 --2- 172.21.15.28:0/2020082120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ebc0aff90 0x7f4ebc0c20d0 secure :-1 s=READY pgs=2723 cs=0 l=1 rev1=1 crypto rx=0x7f4ea800c6f0 tx=0x7f4ea800cbb0 comp rx=0 tx=0).stop 2024-02-21T23:17:39.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.318+0000 7f4ebbe00640 1 -- 172.21.15.28:0/2020082120 shutdown_connections 2024-02-21T23:17:39.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.318+0000 7f4ebbe00640 1 --2- 172.21.15.28:0/2020082120 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f4e9c06b500 0x7f4e9c06d9c0 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:39.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.318+0000 7f4ebbe00640 1 --2- 172.21.15.28:0/2020082120 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ebc0c5720 0x7f4ebc1ee320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:39.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.318+0000 7f4ebbe00640 1 --2- 172.21.15.28:0/2020082120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ebc0aff90 0x7f4ebc0c20d0 unknown :-1 s=CLOSED pgs=2723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:39.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.318+0000 7f4ebbe00640 1 --2- 172.21.15.28:0/2020082120 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f4ebc06dbf0 0x7f4ebc1eaea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:39.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.318+0000 7f4ebbe00640 1 -- 172.21.15.28:0/2020082120 >> 172.21.15.28:0/2020082120 conn(0x7f4ebc0ac370 msgr2=0x7f4ebc157170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:39.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.318+0000 7f4ebbe00640 1 -- 172.21.15.28:0/2020082120 shutdown_connections 2024-02-21T23:17:39.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.319+0000 7f4ebbe00640 1 -- 172.21.15.28:0/2020082120 wait complete. 2024-02-21T23:17:39.328 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:17:39.363 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-21T23:17:39.382 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-21T23:17:39.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.587+0000 7f67f3200640 1 Processor -- start 2024-02-21T23:17:39.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.588+0000 7f67f3200640 1 -- start start 2024-02-21T23:17:39.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.589+0000 7f67f3200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f67f40aff90 0x7f67f40c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:39.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.589+0000 7f67f3200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f67f40c5720 0x7f67f415f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:39.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.589+0000 7f67f3200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67f40c74c0 con 0x7f67f406dbf0 2024-02-21T23:17:39.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.589+0000 7f67f3200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67f40c7270 con 0x7f67f40aff90 2024-02-21T23:17:39.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.589+0000 7f67f3200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67f4158630 con 0x7f67f40c5720 2024-02-21T23:17:39.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.589+0000 7f67f1e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f67f40aff90 0x7f67f40c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:39.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.589+0000 7f67f1400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f67f406dbf0 0x7f67f4158ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59844/0 (socket says 172.21.15.28:59844) 2024-02-21T23:17:39.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.589+0000 7f67f1400640 1 -- 172.21.15.28:0/3077447710 learned_addr learned my addr 172.21.15.28:0/3077447710 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:17:39.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.590+0000 7f67f2800640 1 --2- 172.21.15.28:0/3077447710 >> v2:172.21.15.81:3300/0 conn(0x7f67f40c5720 0x7f67f415f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:39.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.590+0000 7f67f1e00640 1 -- 172.21.15.28:0/3077447710 >> v2:172.21.15.81:3300/0 conn(0x7f67f40c5720 msgr2=0x7f67f415f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:39.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.590+0000 7f67f1e00640 1 --2- 172.21.15.28:0/3077447710 >> v2:172.21.15.81:3300/0 conn(0x7f67f40c5720 0x7f67f415f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:39.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.590+0000 7f67f1e00640 1 -- 172.21.15.28:0/3077447710 >> v1:172.21.15.28:6789/0 conn(0x7f67f406dbf0 legacy=0x7f67f4158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:39.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.590+0000 7f67f1e00640 1 -- 172.21.15.28:0/3077447710 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67f40c77a0 con 0x7f67f40aff90 2024-02-21T23:17:39.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.590+0000 7f67f0a00640 1 -- 172.21.15.28:0/3077447710 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2645302846 0 0) 0x7f67f40c74c0 con 0x7f67f406dbf0 2024-02-21T23:17:39.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.590+0000 7f67f2800640 1 --2- 172.21.15.28:0/3077447710 >> v2:172.21.15.81:3300/0 conn(0x7f67f40c5720 0x7f67f415f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:17:39.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.592+0000 7f67f1e00640 1 --2- 172.21.15.28:0/3077447710 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67f40aff90 0x7f67f40c51e0 secure :-1 s=READY pgs=2725 cs=0 l=1 rev1=1 crypto rx=0x7f67f40c73f0 tx=0x7f67e402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9020b251e4b5e116 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:39.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.592+0000 7f67f0a00640 1 -- 172.21.15.28:0/3077447710 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67e4035020 con 0x7f67f40aff90 2024-02-21T23:17:39.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.592+0000 7f67f0a00640 1 -- 172.21.15.28:0/3077447710 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67e4031830 con 0x7f67f40aff90 2024-02-21T23:17:39.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.592+0000 7f67f0a00640 1 -- 172.21.15.28:0/3077447710 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67e4031c00 con 0x7f67f40aff90 2024-02-21T23:17:39.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.593+0000 7f67f3200640 1 -- 172.21.15.28:0/3077447710 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67f40aff90 msgr2=0x7f67f40c51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:39.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.593+0000 7f67f3200640 1 --2- 172.21.15.28:0/3077447710 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67f40aff90 0x7f67f40c51e0 secure :-1 s=READY pgs=2725 cs=0 l=1 rev1=1 crypto rx=0x7f67f40c73f0 tx=0x7f67e402b8d0 comp rx=0 tx=0).stop 2024-02-21T23:17:39.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.593+0000 7f67f3200640 1 -- 172.21.15.28:0/3077447710 shutdown_connections 2024-02-21T23:17:39.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.593+0000 7f67f3200640 1 --2- 172.21.15.28:0/3077447710 >> v2:172.21.15.81:3300/0 conn(0x7f67f40c5720 0x7f67f415f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:39.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.593+0000 7f67f3200640 1 --2- 172.21.15.28:0/3077447710 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67f40aff90 0x7f67f40c51e0 unknown :-1 s=CLOSED pgs=2725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:39.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.593+0000 7f67f3200640 1 -- 172.21.15.28:0/3077447710 >> 172.21.15.28:0/3077447710 conn(0x7f67f40ac370 msgr2=0x7f67f40ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:39.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.593+0000 7f67f3200640 1 -- 172.21.15.28:0/3077447710 shutdown_connections 2024-02-21T23:17:39.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.593+0000 7f67f3200640 1 -- 172.21.15.28:0/3077447710 wait complete. 2024-02-21T23:17:39.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.596+0000 7f67f3200640 1 Processor -- start 2024-02-21T23:17:39.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.596+0000 7f67f3200640 1 -- start start 2024-02-21T23:17:39.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.598+0000 7f67f3200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67f406dbf0 0x7f67f41eae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:39.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.598+0000 7f67f3200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f67f40aff90 0x7f67f414b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:39.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.598+0000 7f67f3200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f67f40c5720 0x7f67f40ae4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:39.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.598+0000 7f67f3200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67f40c8760 con 0x7f67f406dbf0 2024-02-21T23:17:39.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.598+0000 7f67f3200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67f40be0c0 con 0x7f67f40c5720 2024-02-21T23:17:39.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.598+0000 7f67f3200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67f40c88e0 con 0x7f67f40aff90 2024-02-21T23:17:39.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.598+0000 7f67f1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67f406dbf0 0x7f67f41eae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:39.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.599+0000 7f67f1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67f406dbf0 0x7f67f41eae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59720/0 (socket says 172.21.15.28:59720) 2024-02-21T23:17:39.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.599+0000 7f67f1e00640 1 -- 172.21.15.28:0/2498983615 learned_addr learned my addr 172.21.15.28:0/2498983615 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:39.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.599+0000 7f67f1400640 1 --2- 172.21.15.28:0/2498983615 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f67f40aff90 0x7f67f414b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:39.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.599+0000 7f67f2800640 1 --2- 172.21.15.28:0/2498983615 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f67f40c5720 0x7f67f40ae4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:39.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.599+0000 7f67f1e00640 1 -- 172.21.15.28:0/2498983615 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f67f40aff90 msgr2=0x7f67f414b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:39.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.599+0000 7f67f1e00640 1 --2- 172.21.15.28:0/2498983615 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f67f40aff90 0x7f67f414b630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:39.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.599+0000 7f67f1e00640 1 -- 172.21.15.28:0/2498983615 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f67f40c5720 msgr2=0x7f67f40ae4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:39.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.599+0000 7f67f1e00640 1 --2- 172.21.15.28:0/2498983615 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f67f40c5720 0x7f67f40ae4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:39.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.599+0000 7f67f1e00640 1 -- 172.21.15.28:0/2498983615 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67f41545e0 con 0x7f67f406dbf0 2024-02-21T23:17:39.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.599+0000 7f67f2800640 1 --2- 172.21.15.28:0/2498983615 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f67f40c5720 0x7f67f40ae4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:17:39.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.599+0000 7f67f1e00640 1 --2- 172.21.15.28:0/2498983615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67f406dbf0 0x7f67f41eae70 secure :-1 s=READY pgs=2726 cs=0 l=1 rev1=1 crypto rx=0x7f67e4009860 tx=0x7f67e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:39.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.600+0000 7f67ea000640 1 -- 172.21.15.28:0/2498983615 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67e4035050 con 0x7f67f406dbf0 2024-02-21T23:17:39.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.600+0000 7f67ea000640 1 -- 172.21.15.28:0/2498983615 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67e4004230 con 0x7f67f406dbf0 2024-02-21T23:17:39.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.600+0000 7f67ea000640 1 -- 172.21.15.28:0/2498983615 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67e403b800 con 0x7f67f406dbf0 2024-02-21T23:17:39.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.600+0000 7f67f3200640 1 -- 172.21.15.28:0/2498983615 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67f41548c0 con 0x7f67f406dbf0 2024-02-21T23:17:39.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.600+0000 7f67f3200640 1 -- 172.21.15.28:0/2498983615 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67f40c1e20 con 0x7f67f406dbf0 2024-02-21T23:17:39.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.601+0000 7f67f3200640 1 -- 172.21.15.28:0/2498983615 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67f404f290 con 0x7f67f406dbf0 2024-02-21T23:17:39.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.602+0000 7f67ea000640 1 -- 172.21.15.28:0/2498983615 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f67e4038020 con 0x7f67f406dbf0 2024-02-21T23:17:39.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.603+0000 7f67ea000640 1 --2- 172.21.15.28:0/2498983615 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f67d406b4d0 0x7f67d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:39.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.603+0000 7f67ea000640 1 -- 172.21.15.28:0/2498983615 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f67e40ac320 con 0x7f67f406dbf0 2024-02-21T23:17:39.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.604+0000 7f67f1400640 1 --2- 172.21.15.28:0/2498983615 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f67d406b4d0 0x7f67d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:39.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.605+0000 7f67f1400640 1 --2- 172.21.15.28:0/2498983615 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f67d406b4d0 0x7f67d406d990 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f67f40af400 tx=0x7f67e0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:39.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.607+0000 7f67ea000640 1 -- 172.21.15.28:0/2498983615 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f67e4040070 con 0x7f67f406dbf0 2024-02-21T23:17:39.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.809+0000 7f67f3200640 1 -- 172.21.15.28:0/2498983615 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f67f406d9d0 con 0x7f67f406dbf0 2024-02-21T23:17:39.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.811+0000 7f67ea000640 1 -- 172.21.15.28:0/2498983615 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v212) v1 ==== 74+0+17227 (secure 0 0 0) 0x7f67e407af00 con 0x7f67f406dbf0 2024-02-21T23:17:39.799 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:17:39.799 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":212,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:14:42.637299+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:13:27.612392+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"207","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_data","create_time":"2024-02-21T23:13:28.744308+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"208","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"34.b","mappings":[{"from":0,"to":1}]},{"pgid":"34.f","mappings":[{"from":0,"to":1}]},{"pgid":"34.31","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.81:6835/1607587459":"2024-02-22T23:14:42.637238+0000","172.21.15.28:6833/1864206570":"2024-02-22T23:14:07.624525+0000","172.21.15.81:6834/1607587459":"2024-02-22T23:14:42.637238+0000","172.21.15.28:6832/1864206570":"2024-02-22T23:14:07.624525+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:17:39.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.813+0000 7f67f3200640 1 -- 172.21.15.28:0/2498983615 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f67d406b4d0 msgr2=0x7f67d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:39.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.813+0000 7f67f3200640 1 --2- 172.21.15.28:0/2498983615 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f67d406b4d0 0x7f67d406d990 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f67f40af400 tx=0x7f67e0009040 comp rx=0 tx=0).stop 2024-02-21T23:17:39.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.813+0000 7f67f3200640 1 -- 172.21.15.28:0/2498983615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67f406dbf0 msgr2=0x7f67f41eae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:39.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.813+0000 7f67f3200640 1 --2- 172.21.15.28:0/2498983615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67f406dbf0 0x7f67f41eae70 secure :-1 s=READY pgs=2726 cs=0 l=1 rev1=1 crypto rx=0x7f67e4009860 tx=0x7f67e40046c0 comp rx=0 tx=0).stop 2024-02-21T23:17:39.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.813+0000 7f67f3200640 1 -- 172.21.15.28:0/2498983615 shutdown_connections 2024-02-21T23:17:39.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.813+0000 7f67f3200640 1 --2- 172.21.15.28:0/2498983615 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f67d406b4d0 0x7f67d406d990 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:39.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.813+0000 7f67f3200640 1 --2- 172.21.15.28:0/2498983615 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f67f40aff90 0x7f67f414b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:39.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.813+0000 7f67f3200640 1 --2- 172.21.15.28:0/2498983615 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f67f40c5720 0x7f67f40ae4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:39.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.813+0000 7f67f3200640 1 --2- 172.21.15.28:0/2498983615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67f406dbf0 0x7f67f41eae70 unknown :-1 s=CLOSED pgs=2726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:39.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.813+0000 7f67f3200640 1 -- 172.21.15.28:0/2498983615 >> 172.21.15.28:0/2498983615 conn(0x7f67f40ac370 msgr2=0x7f67f40ac750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:39.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.813+0000 7f67f3200640 1 -- 172.21.15.28:0/2498983615 shutdown_connections 2024-02-21T23:17:39.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:39.814+0000 7f67f3200640 1 -- 172.21.15.28:0/2498983615 wait complete. 2024-02-21T23:17:39.824 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-02-21T23:17:39.824 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:17:40.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.077+0000 7f318b200640 1 Processor -- start 2024-02-21T23:17:40.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.077+0000 7f318b200640 1 -- start start 2024-02-21T23:17:40.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.079+0000 7f318b200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3184155960 0x7f31841608a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:40.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.079+0000 7f318b200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31840c74c0 con 0x7f318406dbf0 2024-02-21T23:17:40.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.079+0000 7f318b200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31840c77a0 con 0x7f31840aff90 2024-02-21T23:17:40.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.079+0000 7f318b200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31840c7270 con 0x7f3184155960 2024-02-21T23:17:40.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.079+0000 7f318a800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3184155960 0x7f31841608a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:40.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.079+0000 7f3189400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f318406dbf0 0x7f3184158ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59846/0 (socket says 172.21.15.28:59846) 2024-02-21T23:17:40.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.079+0000 7f3189400640 1 -- 172.21.15.28:0/2577901635 learned_addr learned my addr 172.21.15.28:0/2577901635 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:17:40.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.080+0000 7f318a800640 1 -- 172.21.15.28:0/2577901635 >> v1:172.21.15.81:6789/0 conn(0x7f31840aff90 legacy=0x7f31840c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:40.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.080+0000 7f318a800640 1 -- 172.21.15.28:0/2577901635 >> v1:172.21.15.28:6789/0 conn(0x7f318406dbf0 legacy=0x7f3184158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:40.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.080+0000 7f318a800640 1 -- 172.21.15.28:0/2577901635 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31840c7a80 con 0x7f3184155960 2024-02-21T23:17:40.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.081+0000 7f318a800640 1 --2- 172.21.15.28:0/2577901635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3184155960 0x7f31841608a0 secure :-1 s=READY pgs=2728 cs=0 l=1 rev1=1 crypto rx=0x7f3180009d90 tx=0x7f318002bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=305e2648a8cedf5e server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:40.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.081+0000 7f3188a00640 1 -- 172.21.15.28:0/2577901635 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3180039070 con 0x7f3184155960 2024-02-21T23:17:40.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.081+0000 7f3188a00640 1 -- 172.21.15.28:0/2577901635 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f318002ccf0 con 0x7f3184155960 2024-02-21T23:17:40.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.081+0000 7f3188a00640 1 -- 172.21.15.28:0/2577901635 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3180034660 con 0x7f3184155960 2024-02-21T23:17:40.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.081+0000 7f318b200640 1 -- 172.21.15.28:0/2577901635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3184155960 msgr2=0x7f31841608a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:40.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.082+0000 7f318b200640 1 --2- 172.21.15.28:0/2577901635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3184155960 0x7f31841608a0 secure :-1 s=READY pgs=2728 cs=0 l=1 rev1=1 crypto rx=0x7f3180009d90 tx=0x7f318002bf40 comp rx=0 tx=0).stop 2024-02-21T23:17:40.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.082+0000 7f318b200640 1 -- 172.21.15.28:0/2577901635 shutdown_connections 2024-02-21T23:17:40.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.082+0000 7f318b200640 1 --2- 172.21.15.28:0/2577901635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3184155960 0x7f31841608a0 unknown :-1 s=CLOSED pgs=2728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:40.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.082+0000 7f318b200640 1 -- 172.21.15.28:0/2577901635 >> 172.21.15.28:0/2577901635 conn(0x7f31840ac370 msgr2=0x7f31840ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:40.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.082+0000 7f318b200640 1 -- 172.21.15.28:0/2577901635 shutdown_connections 2024-02-21T23:17:40.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.082+0000 7f318b200640 1 -- 172.21.15.28:0/2577901635 wait complete. 2024-02-21T23:17:40.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.083+0000 7f318b200640 1 Processor -- start 2024-02-21T23:17:40.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.084+0000 7f318b200640 1 -- start start 2024-02-21T23:17:40.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.084+0000 7f318b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f318406dbf0 0x7f31840c4290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:40.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.085+0000 7f318b200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f31840aff90 0x7f31840ae740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:40.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.085+0000 7f318b200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3184155960 0x7f31840c1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:40.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.085+0000 7f318b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31840c58f0 con 0x7f318406dbf0 2024-02-21T23:17:40.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.085+0000 7f318b200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3184158630 con 0x7f3184155960 2024-02-21T23:17:40.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.085+0000 7f318b200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31840c5a70 con 0x7f31840aff90 2024-02-21T23:17:40.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.085+0000 7f3189e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f318406dbf0 0x7f31840c4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:40.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.085+0000 7f3189e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f318406dbf0 0x7f31840c4290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59736/0 (socket says 172.21.15.28:59736) 2024-02-21T23:17:40.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.085+0000 7f3189e00640 1 -- 172.21.15.28:0/2078003993 learned_addr learned my addr 172.21.15.28:0/2078003993 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:40.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.085+0000 7f3189400640 1 --2- 172.21.15.28:0/2078003993 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f31840aff90 0x7f31840ae740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:40.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.085+0000 7f318a800640 1 --2- 172.21.15.28:0/2078003993 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3184155960 0x7f31840c1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:40.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.085+0000 7f3189e00640 1 -- 172.21.15.28:0/2078003993 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f31840aff90 msgr2=0x7f31840ae740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:40.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.085+0000 7f3189e00640 1 --2- 172.21.15.28:0/2078003993 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f31840aff90 0x7f31840ae740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:40.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.085+0000 7f3189e00640 1 -- 172.21.15.28:0/2078003993 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3184155960 msgr2=0x7f31840c1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:40.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.085+0000 7f3189e00640 1 --2- 172.21.15.28:0/2078003993 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3184155960 0x7f31840c1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:40.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.085+0000 7f3189e00640 1 -- 172.21.15.28:0/2078003993 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3184154d60 con 0x7f318406dbf0 2024-02-21T23:17:40.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.085+0000 7f3189400640 1 --2- 172.21.15.28:0/2078003993 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f31840aff90 0x7f31840ae740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:17:40.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.085+0000 7f318a800640 1 --2- 172.21.15.28:0/2078003993 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3184155960 0x7f31840c1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:17:40.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.086+0000 7f3189e00640 1 --2- 172.21.15.28:0/2078003993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f318406dbf0 0x7f31840c4290 secure :-1 s=READY pgs=2729 cs=0 l=1 rev1=1 crypto rx=0x7f317800c450 tx=0x7f317800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:40.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.086+0000 7f317e000640 1 -- 172.21.15.28:0/2078003993 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31780052d0 con 0x7f318406dbf0 2024-02-21T23:17:40.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.086+0000 7f317e000640 1 -- 172.21.15.28:0/2078003993 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3178005470 con 0x7f318406dbf0 2024-02-21T23:17:40.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.086+0000 7f317e000640 1 -- 172.21.15.28:0/2078003993 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3178012870 con 0x7f318406dbf0 2024-02-21T23:17:40.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.086+0000 7f318b200640 1 -- 172.21.15.28:0/2078003993 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3184154880 con 0x7f318406dbf0 2024-02-21T23:17:40.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.086+0000 7f318b200640 1 -- 172.21.15.28:0/2078003993 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31841539a0 con 0x7f318406dbf0 2024-02-21T23:17:40.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.087+0000 7f318b200640 1 -- 172.21.15.28:0/2078003993 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3184153790 con 0x7f318406dbf0 2024-02-21T23:17:40.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.089+0000 7f317e000640 1 -- 172.21.15.28:0/2078003993 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3178012a10 con 0x7f318406dbf0 2024-02-21T23:17:40.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.090+0000 7f317e000640 1 --2- 172.21.15.28:0/2078003993 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f316806b500 0x7f316806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:40.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.091+0000 7f317e000640 1 -- 172.21.15.28:0/2078003993 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f3178089ff0 con 0x7f318406dbf0 2024-02-21T23:17:40.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.091+0000 7f3189400640 1 --2- 172.21.15.28:0/2078003993 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f316806b500 0x7f316806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:40.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.092+0000 7f3189400640 1 --2- 172.21.15.28:0/2078003993 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f316806b500 0x7f316806d9c0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f3184157820 tx=0x7f3174009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:40.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.095+0000 7f317e000640 1 -- 172.21.15.28:0/2078003993 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3178058c70 con 0x7f318406dbf0 2024-02-21T23:17:40.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.334+0000 7f318b200640 1 -- 172.21.15.28:0/2078003993 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f318406d9d0 con 0x7f318406dbf0 2024-02-21T23:17:40.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.335+0000 7f317e000640 1 -- 172.21.15.28:0/2078003993 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7f3178058570 con 0x7f318406dbf0 2024-02-21T23:17:40.323 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:17:40.323 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:17:40.324 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:17:40.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.338+0000 7f318b200640 1 -- 172.21.15.28:0/2078003993 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f316806b500 msgr2=0x7f316806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:40.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.338+0000 7f318b200640 1 --2- 172.21.15.28:0/2078003993 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f316806b500 0x7f316806d9c0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f3184157820 tx=0x7f3174009040 comp rx=0 tx=0).stop 2024-02-21T23:17:40.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.338+0000 7f318b200640 1 -- 172.21.15.28:0/2078003993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f318406dbf0 msgr2=0x7f31840c4290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:40.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.338+0000 7f318b200640 1 --2- 172.21.15.28:0/2078003993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f318406dbf0 0x7f31840c4290 secure :-1 s=READY pgs=2729 cs=0 l=1 rev1=1 crypto rx=0x7f317800c450 tx=0x7f317800c910 comp rx=0 tx=0).stop 2024-02-21T23:17:40.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.338+0000 7f318b200640 1 -- 172.21.15.28:0/2078003993 shutdown_connections 2024-02-21T23:17:40.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.338+0000 7f318b200640 1 --2- 172.21.15.28:0/2078003993 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f316806b500 0x7f316806d9c0 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:40.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.338+0000 7f318b200640 1 --2- 172.21.15.28:0/2078003993 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f31840aff90 0x7f31840ae740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:40.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.338+0000 7f318b200640 1 --2- 172.21.15.28:0/2078003993 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3184155960 0x7f31840c1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:40.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.338+0000 7f318b200640 1 --2- 172.21.15.28:0/2078003993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f318406dbf0 0x7f31840c4290 unknown :-1 s=CLOSED pgs=2729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:40.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.338+0000 7f318b200640 1 -- 172.21.15.28:0/2078003993 >> 172.21.15.28:0/2078003993 conn(0x7f31840ac370 msgr2=0x7f31840ac750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:40.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.338+0000 7f318b200640 1 -- 172.21.15.28:0/2078003993 shutdown_connections 2024-02-21T23:17:40.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.338+0000 7f318b200640 1 -- 172.21.15.28:0/2078003993 wait complete. 2024-02-21T23:17:40.344 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:17:40.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.562+0000 7f2d42800640 1 Processor -- start 2024-02-21T23:17:40.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.562+0000 7f2d42800640 1 -- start start 2024-02-21T23:17:40.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.563+0000 7f2d42800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f2d3c0aff90 0x7f2d3c0c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:40.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.563+0000 7f2d42800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2d3c06dbf0 0x7f2d3c158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:40.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.563+0000 7f2d42800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d3c0c74c0 con 0x7f2d3c0c5720 2024-02-21T23:17:40.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.563+0000 7f2d42800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d3c0c7270 con 0x7f2d3c06dbf0 2024-02-21T23:17:40.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.563+0000 7f2d42800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d3c158630 con 0x7f2d3c0aff90 2024-02-21T23:17:40.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.564+0000 7f2d41e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f2d3c0c5720 0x7f2d3c06c270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59850/0 (socket says 172.21.15.28:59850) 2024-02-21T23:17:40.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.564+0000 7f2d41e00640 1 -- 172.21.15.28:0/2465131892 learned_addr learned my addr 172.21.15.28:0/2465131892 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:17:40.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.564+0000 7f2d40a00640 1 --2- 172.21.15.28:0/2465131892 >> v2:172.21.15.28:3300/0 conn(0x7f2d3c06dbf0 0x7f2d3c158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:40.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.564+0000 7f2d41400640 1 --2- 172.21.15.28:0/2465131892 >> v2:172.21.15.81:3300/0 conn(0x7f2d3c0aff90 0x7f2d3c0c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:40.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.564+0000 7f2d3be00640 1 -- 172.21.15.28:0/2465131892 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3310897228 0 0) 0x7f2d3c0c74c0 con 0x7f2d3c0c5720 2024-02-21T23:17:40.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.564+0000 7f2d3be00640 1 -- 172.21.15.28:0/2465131892 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d24003520 con 0x7f2d3c0c5720 2024-02-21T23:17:40.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.564+0000 7f2d40a00640 1 -- 172.21.15.28:0/2465131892 >> v2:172.21.15.81:3300/0 conn(0x7f2d3c0aff90 msgr2=0x7f2d3c0c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:40.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.564+0000 7f2d40a00640 1 --2- 172.21.15.28:0/2465131892 >> v2:172.21.15.81:3300/0 conn(0x7f2d3c0aff90 0x7f2d3c0c51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:40.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.564+0000 7f2d40a00640 1 -- 172.21.15.28:0/2465131892 >> v1:172.21.15.28:6789/0 conn(0x7f2d3c0c5720 legacy=0x7f2d3c06c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:40.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.564+0000 7f2d40a00640 1 -- 172.21.15.28:0/2465131892 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d3c0c77a0 con 0x7f2d3c06dbf0 2024-02-21T23:17:40.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.564+0000 7f2d41400640 1 --2- 172.21.15.28:0/2465131892 >> v2:172.21.15.81:3300/0 conn(0x7f2d3c0aff90 0x7f2d3c0c51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:17:40.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.566+0000 7f2d40a00640 1 --2- 172.21.15.28:0/2465131892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d3c06dbf0 0x7f2d3c158ac0 secure :-1 s=READY pgs=2731 cs=0 l=1 rev1=1 crypto rx=0x7f2d3c0c73f0 tx=0x7f2d2802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f643fab078f29743 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:40.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.566+0000 7f2d3be00640 1 -- 172.21.15.28:0/2465131892 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d28035020 con 0x7f2d3c06dbf0 2024-02-21T23:17:40.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.566+0000 7f2d3be00640 1 -- 172.21.15.28:0/2465131892 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d28031830 con 0x7f2d3c06dbf0 2024-02-21T23:17:40.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.566+0000 7f2d3be00640 1 -- 172.21.15.28:0/2465131892 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d28031be0 con 0x7f2d3c06dbf0 2024-02-21T23:17:40.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.567+0000 7f2d42800640 1 -- 172.21.15.28:0/2465131892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d3c06dbf0 msgr2=0x7f2d3c158ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:40.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.567+0000 7f2d42800640 1 --2- 172.21.15.28:0/2465131892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d3c06dbf0 0x7f2d3c158ac0 secure :-1 s=READY pgs=2731 cs=0 l=1 rev1=1 crypto rx=0x7f2d3c0c73f0 tx=0x7f2d2802b8d0 comp rx=0 tx=0).stop 2024-02-21T23:17:40.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.567+0000 7f2d42800640 1 -- 172.21.15.28:0/2465131892 shutdown_connections 2024-02-21T23:17:40.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.567+0000 7f2d42800640 1 --2- 172.21.15.28:0/2465131892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d3c06dbf0 0x7f2d3c158ac0 unknown :-1 s=CLOSED pgs=2731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:40.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.567+0000 7f2d42800640 1 --2- 172.21.15.28:0/2465131892 >> v2:172.21.15.81:3300/0 conn(0x7f2d3c0aff90 0x7f2d3c0c51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:40.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.567+0000 7f2d42800640 1 -- 172.21.15.28:0/2465131892 >> 172.21.15.28:0/2465131892 conn(0x7f2d3c0ac370 msgr2=0x7f2d3c0ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:40.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.567+0000 7f2d42800640 1 -- 172.21.15.28:0/2465131892 shutdown_connections 2024-02-21T23:17:40.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.567+0000 7f2d42800640 1 -- 172.21.15.28:0/2465131892 wait complete. 2024-02-21T23:17:40.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.568+0000 7f2d42800640 1 Processor -- start 2024-02-21T23:17:40.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.569+0000 7f2d42800640 1 -- start start 2024-02-21T23:17:40.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.569+0000 7f2d42800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2d3c06dbf0 0x7f2d3c1edfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:40.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.569+0000 7f2d42800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2d3c0aff90 0x7f2d3c0be570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:40.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.569+0000 7f2d42800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d3c0c5720 0x7f2d3c0beab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:40.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.569+0000 7f2d42800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d3c0c8760 con 0x7f2d3c0c5720 2024-02-21T23:17:40.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.569+0000 7f2d42800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d3c0be0c0 con 0x7f2d3c06dbf0 2024-02-21T23:17:40.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.569+0000 7f2d42800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d3c0c88e0 con 0x7f2d3c0aff90 2024-02-21T23:17:40.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.570+0000 7f2d41e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d3c0c5720 0x7f2d3c0beab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:40.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.570+0000 7f2d41e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d3c0c5720 0x7f2d3c0beab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59752/0 (socket says 172.21.15.28:59752) 2024-02-21T23:17:40.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.570+0000 7f2d41e00640 1 -- 172.21.15.28:0/3093767998 learned_addr learned my addr 172.21.15.28:0/3093767998 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:40.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.570+0000 7f2d41400640 1 --2- 172.21.15.28:0/3093767998 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2d3c06dbf0 0x7f2d3c1edfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:40.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.570+0000 7f2d40a00640 1 --2- 172.21.15.28:0/3093767998 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2d3c0aff90 0x7f2d3c0be570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:40.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.570+0000 7f2d41e00640 1 -- 172.21.15.28:0/3093767998 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2d3c0aff90 msgr2=0x7f2d3c0be570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:40.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.570+0000 7f2d41e00640 1 --2- 172.21.15.28:0/3093767998 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2d3c0aff90 0x7f2d3c0be570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:40.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.570+0000 7f2d41e00640 1 -- 172.21.15.28:0/3093767998 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2d3c06dbf0 msgr2=0x7f2d3c1edfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:40.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.570+0000 7f2d41e00640 1 --2- 172.21.15.28:0/3093767998 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2d3c06dbf0 0x7f2d3c1edfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:40.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.570+0000 7f2d41e00640 1 -- 172.21.15.28:0/3093767998 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d3c0c3fc0 con 0x7f2d3c0c5720 2024-02-21T23:17:40.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.571+0000 7f2d41e00640 1 --2- 172.21.15.28:0/3093767998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d3c0c5720 0x7f2d3c0beab0 secure :-1 s=READY pgs=2732 cs=0 l=1 rev1=1 crypto rx=0x7f2d3400cad0 tx=0x7f2d3400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:40.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.571+0000 7f2d39600640 1 -- 172.21.15.28:0/3093767998 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d34017070 con 0x7f2d3c0c5720 2024-02-21T23:17:40.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.571+0000 7f2d39600640 1 -- 172.21.15.28:0/3093767998 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d3400ac30 con 0x7f2d3c0c5720 2024-02-21T23:17:40.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.571+0000 7f2d39600640 1 -- 172.21.15.28:0/3093767998 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d34012400 con 0x7f2d3c0c5720 2024-02-21T23:17:40.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.571+0000 7f2d42800640 1 -- 172.21.15.28:0/3093767998 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d3c0c4890 con 0x7f2d3c0c5720 2024-02-21T23:17:40.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.571+0000 7f2d42800640 1 -- 172.21.15.28:0/3093767998 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d3c0cb440 con 0x7f2d3c0c5720 2024-02-21T23:17:40.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.573+0000 7f2d42800640 1 -- 172.21.15.28:0/3093767998 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d3c0acaf0 con 0x7f2d3c0c5720 2024-02-21T23:17:40.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.573+0000 7f2d39600640 1 -- 172.21.15.28:0/3093767998 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2d3401a070 con 0x7f2d3c0c5720 2024-02-21T23:17:40.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.574+0000 7f2d39600640 1 --2- 172.21.15.28:0/3093767998 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2d1c06b4d0 0x7f2d1c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:40.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.574+0000 7f2d39600640 1 -- 172.21.15.28:0/3093767998 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f2d3408a1c0 con 0x7f2d3c0c5720 2024-02-21T23:17:40.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.575+0000 7f2d41400640 1 --2- 172.21.15.28:0/3093767998 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2d1c06b4d0 0x7f2d1c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:40.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.576+0000 7f2d41400640 1 --2- 172.21.15.28:0/3093767998 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2d1c06b4d0 0x7f2d1c06d990 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f2d3c0cafd0 tx=0x7f2d30008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:40.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.579+0000 7f2d39600640 1 -- 172.21.15.28:0/3093767998 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f2d3c0acaf0 con 0x7f2d3c0c5720 2024-02-21T23:17:40.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.850+0000 7f2d42800640 1 -- 172.21.15.28:0/3093767998 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2d3c0c4a60 con 0x7f2d3c0c5720 2024-02-21T23:17:40.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.851+0000 7f2d39600640 1 -- 172.21.15.28:0/3093767998 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+3432 (secure 0 0 0) 0x7f2d3c0c4a60 con 0x7f2d3c0c5720 2024-02-21T23:17:40.839 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:17:40.839 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":247,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-21T23:13:29.725909+0000","modified":"2024-02-21T23:15:17.646309+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":212,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":0,"in":[0,1],"up":{"mds_0":8296,"mds_1":7265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7265":{"gid":7265,"name":"b","rank":1,"incarnation":241,"state":"up:stopping","state_seq":24,"addr":"172.21.15.81:6837/3779716966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":3779716966},{"type":"v1","addr":"172.21.15.81:6837","nonce":3779716966}]},"join_fscid":-1,"laggy_since":"2024-02-21T23:15:17.646309+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8296":{"gid":8296,"name":"c","rank":0,"incarnation":225,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3142544451","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3142544451},{"type":"v1","addr":"172.21.15.28:6835","nonce":3142544451}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[34],"metadata_pool":33,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-02-21T23:17:40.839 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 247 2024-02-21T23:17:40.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.853+0000 7f2d42800640 1 -- 172.21.15.28:0/3093767998 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2d1c06b4d0 msgr2=0x7f2d1c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:40.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.853+0000 7f2d42800640 1 --2- 172.21.15.28:0/3093767998 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2d1c06b4d0 0x7f2d1c06d990 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f2d3c0cafd0 tx=0x7f2d30008040 comp rx=0 tx=0).stop 2024-02-21T23:17:40.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.853+0000 7f2d42800640 1 -- 172.21.15.28:0/3093767998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d3c0c5720 msgr2=0x7f2d3c0beab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:40.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.853+0000 7f2d42800640 1 --2- 172.21.15.28:0/3093767998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d3c0c5720 0x7f2d3c0beab0 secure :-1 s=READY pgs=2732 cs=0 l=1 rev1=1 crypto rx=0x7f2d3400cad0 tx=0x7f2d3400cf90 comp rx=0 tx=0).stop 2024-02-21T23:17:40.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.853+0000 7f2d42800640 1 -- 172.21.15.28:0/3093767998 shutdown_connections 2024-02-21T23:17:40.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.853+0000 7f2d42800640 1 --2- 172.21.15.28:0/3093767998 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2d1c06b4d0 0x7f2d1c06d990 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:40.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.853+0000 7f2d42800640 1 --2- 172.21.15.28:0/3093767998 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2d3c0aff90 0x7f2d3c0be570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:40.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.853+0000 7f2d42800640 1 --2- 172.21.15.28:0/3093767998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d3c0c5720 0x7f2d3c0beab0 unknown :-1 s=CLOSED pgs=2732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:40.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.853+0000 7f2d42800640 1 --2- 172.21.15.28:0/3093767998 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2d3c06dbf0 0x7f2d3c1edfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:40.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.853+0000 7f2d42800640 1 -- 172.21.15.28:0/3093767998 >> 172.21.15.28:0/3093767998 conn(0x7f2d3c0ac370 msgr2=0x7f2d3c0c9f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:40.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.854+0000 7f2d42800640 1 -- 172.21.15.28:0/3093767998 shutdown_connections 2024-02-21T23:17:40.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:40.854+0000 7f2d42800640 1 -- 172.21.15.28:0/3093767998 wait complete. 2024-02-21T23:17:40.861 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:17:41.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.089+0000 7f631c000640 1 Processor -- start 2024-02-21T23:17:41.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.089+0000 7f631c000640 1 -- start start 2024-02-21T23:17:41.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.091+0000 7f631c000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6314152ab0 0x7f63140c7fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:41.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.091+0000 7f631c000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f6314158850 0x7f6314158cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:41.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.091+0000 7f631c000640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63140a8180 con 0x7f63140c6710 2024-02-21T23:17:41.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.091+0000 7f631c000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63140c3ce0 con 0x7f6314152ab0 2024-02-21T23:17:41.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.091+0000 7f631c000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63140c33b0 con 0x7f6314158850 2024-02-21T23:17:41.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.091+0000 7f6319600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6314152ab0 0x7f63140c7fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:41.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.091+0000 7f6319600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6314152ab0 0x7f63140c7fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59768/0 (socket says 172.21.15.28:59768) 2024-02-21T23:17:41.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.091+0000 7f6319600640 1 -- 172.21.15.28:0/2349796387 learned_addr learned my addr 172.21.15.28:0/2349796387 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:41.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.092+0000 7f631a000640 1 --2- 172.21.15.28:0/2349796387 >> v2:172.21.15.81:3300/0 conn(0x7f6314158850 0x7f6314158cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:41.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.092+0000 7f6319600640 1 -- 172.21.15.28:0/2349796387 >> v2:172.21.15.81:3300/0 conn(0x7f6314158850 msgr2=0x7f6314158cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:41.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.092+0000 7f6319600640 1 --2- 172.21.15.28:0/2349796387 >> v2:172.21.15.81:3300/0 conn(0x7f6314158850 0x7f6314158cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:41.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.092+0000 7f6319600640 1 -- 172.21.15.28:0/2349796387 >> v1:172.21.15.81:6789/0 conn(0x7f63140c6710 legacy=0x7f63140c5210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:41.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.092+0000 7f6319600640 1 -- 172.21.15.28:0/2349796387 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63140c3690 con 0x7f6314152ab0 2024-02-21T23:17:41.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.093+0000 7f6319600640 1 --2- 172.21.15.28:0/2349796387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6314152ab0 0x7f63140c7fe0 secure :-1 s=READY pgs=2733 cs=0 l=1 rev1=1 crypto rx=0x7f6314072d70 tx=0x7f630802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce8e13d3e93a9ba5 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:41.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.093+0000 7f6313e00640 1 -- 172.21.15.28:0/2349796387 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6308035020 con 0x7f6314152ab0 2024-02-21T23:17:41.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.093+0000 7f6313e00640 1 -- 172.21.15.28:0/2349796387 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6308031870 con 0x7f6314152ab0 2024-02-21T23:17:41.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.093+0000 7f6313e00640 1 -- 172.21.15.28:0/2349796387 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6308031c40 con 0x7f6314152ab0 2024-02-21T23:17:41.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.094+0000 7f631c000640 1 -- 172.21.15.28:0/2349796387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6314152ab0 msgr2=0x7f63140c7fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:41.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.094+0000 7f631c000640 1 --2- 172.21.15.28:0/2349796387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6314152ab0 0x7f63140c7fe0 secure :-1 s=READY pgs=2733 cs=0 l=1 rev1=1 crypto rx=0x7f6314072d70 tx=0x7f630802b910 comp rx=0 tx=0).stop 2024-02-21T23:17:41.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.094+0000 7f631c000640 1 -- 172.21.15.28:0/2349796387 shutdown_connections 2024-02-21T23:17:41.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.094+0000 7f631c000640 1 --2- 172.21.15.28:0/2349796387 >> v2:172.21.15.81:3300/0 conn(0x7f6314158850 0x7f6314158cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:41.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.094+0000 7f631c000640 1 --2- 172.21.15.28:0/2349796387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6314152ab0 0x7f63140c7fe0 unknown :-1 s=CLOSED pgs=2733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:41.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.094+0000 7f631c000640 1 -- 172.21.15.28:0/2349796387 >> 172.21.15.28:0/2349796387 conn(0x7f63140c8810 msgr2=0x7f63140c8bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:41.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.094+0000 7f631c000640 1 -- 172.21.15.28:0/2349796387 shutdown_connections 2024-02-21T23:17:41.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.094+0000 7f631c000640 1 -- 172.21.15.28:0/2349796387 wait complete. 2024-02-21T23:17:41.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.096+0000 7f631c000640 1 Processor -- start 2024-02-21T23:17:41.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.096+0000 7f631c000640 1 -- start start 2024-02-21T23:17:41.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.097+0000 7f631c000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f63140c6710 0x7f631406fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:41.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.097+0000 7f631c000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6314152ab0 0x7f63141ee350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:41.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.097+0000 7f631c000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6314158850 0x7f6314156ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:41.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.097+0000 7f631c000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63140c5aa0 con 0x7f6314152ab0 2024-02-21T23:17:41.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.097+0000 7f631c000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63140c5920 con 0x7f6314158850 2024-02-21T23:17:41.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.097+0000 7f631c000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6314159200 con 0x7f63140c6710 2024-02-21T23:17:41.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.097+0000 7f6318c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6314152ab0 0x7f63141ee350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:41.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.097+0000 7f6318c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6314152ab0 0x7f63141ee350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59776/0 (socket says 172.21.15.28:59776) 2024-02-21T23:17:41.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.097+0000 7f6318c00640 1 -- 172.21.15.28:0/3822953498 learned_addr learned my addr 172.21.15.28:0/3822953498 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:41.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.097+0000 7f6318c00640 1 -- 172.21.15.28:0/3822953498 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f63140c6710 msgr2=0x7f631406fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:41.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.097+0000 7f6318c00640 1 --2- 172.21.15.28:0/3822953498 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f63140c6710 0x7f631406fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:41.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.097+0000 7f6318c00640 1 -- 172.21.15.28:0/3822953498 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6314158850 msgr2=0x7f6314156ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:41.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.097+0000 7f6318c00640 1 --2- 172.21.15.28:0/3822953498 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6314158850 0x7f6314156ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:41.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.097+0000 7f6318c00640 1 -- 172.21.15.28:0/3822953498 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63140aa160 con 0x7f6314152ab0 2024-02-21T23:17:41.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.098+0000 7f6318c00640 1 --2- 172.21.15.28:0/3822953498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6314152ab0 0x7f63141ee350 secure :-1 s=READY pgs=2734 cs=0 l=1 rev1=1 crypto rx=0x7f630000c430 tx=0x7f630000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:41.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.098+0000 7f6311600640 1 -- 172.21.15.28:0/3822953498 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6300011790 con 0x7f6314152ab0 2024-02-21T23:17:41.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.098+0000 7f6311600640 1 -- 172.21.15.28:0/3822953498 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6300017070 con 0x7f6314152ab0 2024-02-21T23:17:41.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.098+0000 7f6311600640 1 -- 172.21.15.28:0/3822953498 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6300011bb0 con 0x7f6314152ab0 2024-02-21T23:17:41.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.098+0000 7f631c000640 1 -- 172.21.15.28:0/3822953498 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63141eeb90 con 0x7f6314152ab0 2024-02-21T23:17:41.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.098+0000 7f631c000640 1 -- 172.21.15.28:0/3822953498 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6314157920 con 0x7f6314152ab0 2024-02-21T23:17:41.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.099+0000 7f631c000640 1 -- 172.21.15.28:0/3822953498 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6314067f30 con 0x7f6314152ab0 2024-02-21T23:17:41.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.100+0000 7f6311600640 1 -- 172.21.15.28:0/3822953498 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f630001c460 con 0x7f6314152ab0 2024-02-21T23:17:41.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.101+0000 7f6311600640 1 --2- 172.21.15.28:0/3822953498 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f62f406b4d0 0x7f62f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:41.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.101+0000 7f6311600640 1 -- 172.21.15.28:0/3822953498 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f630008a380 con 0x7f6314152ab0 2024-02-21T23:17:41.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.102+0000 7f6319600640 1 --2- 172.21.15.28:0/3822953498 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f62f406b4d0 0x7f62f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:41.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.103+0000 7f6319600640 1 --2- 172.21.15.28:0/3822953498 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f62f406b4d0 0x7f62f406d990 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f6308036000 tx=0x7f6308030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:41.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.105+0000 7f6311600640 1 -- 172.21.15.28:0/3822953498 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f6300058f60 con 0x7f6314152ab0 2024-02-21T23:17:41.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.306+0000 7f631c000640 1 -- 172.21.15.28:0/3822953498 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6314154f00 con 0x7f6314152ab0 2024-02-21T23:17:41.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.310+0000 7f6311600640 1 -- 172.21.15.28:0/3822953498 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v212) v1 ==== 74+0+17227 (secure 0 0 0) 0x7f6300058860 con 0x7f6314152ab0 2024-02-21T23:17:41.298 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:17:41.298 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":212,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:14:42.637299+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"cephfs_metadata","create_time":"2024-02-21T23:13:27.612392+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"207","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_data","create_time":"2024-02-21T23:13:28.744308+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"208","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"34.b","mappings":[{"from":0,"to":1}]},{"pgid":"34.f","mappings":[{"from":0,"to":1}]},{"pgid":"34.31","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.81:6835/1607587459":"2024-02-22T23:14:42.637238+0000","172.21.15.28:6833/1864206570":"2024-02-22T23:14:07.624525+0000","172.21.15.81:6834/1607587459":"2024-02-22T23:14:42.637238+0000","172.21.15.28:6832/1864206570":"2024-02-22T23:14:07.624525+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:17:41.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.312+0000 7f631c000640 1 -- 172.21.15.28:0/3822953498 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f62f406b4d0 msgr2=0x7f62f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:41.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.312+0000 7f631c000640 1 --2- 172.21.15.28:0/3822953498 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f62f406b4d0 0x7f62f406d990 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f6308036000 tx=0x7f6308030000 comp rx=0 tx=0).stop 2024-02-21T23:17:41.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.312+0000 7f631c000640 1 -- 172.21.15.28:0/3822953498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6314152ab0 msgr2=0x7f63141ee350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:41.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.312+0000 7f631c000640 1 --2- 172.21.15.28:0/3822953498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6314152ab0 0x7f63141ee350 secure :-1 s=READY pgs=2734 cs=0 l=1 rev1=1 crypto rx=0x7f630000c430 tx=0x7f630000c8f0 comp rx=0 tx=0).stop 2024-02-21T23:17:41.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.312+0000 7f631c000640 1 -- 172.21.15.28:0/3822953498 shutdown_connections 2024-02-21T23:17:41.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.312+0000 7f631c000640 1 --2- 172.21.15.28:0/3822953498 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f62f406b4d0 0x7f62f406d990 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:41.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.312+0000 7f631c000640 1 --2- 172.21.15.28:0/3822953498 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6314158850 0x7f6314156ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:41.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.312+0000 7f631c000640 1 --2- 172.21.15.28:0/3822953498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6314152ab0 0x7f63141ee350 unknown :-1 s=CLOSED pgs=2734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:41.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.312+0000 7f631c000640 1 --2- 172.21.15.28:0/3822953498 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f63140c6710 0x7f631406fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:41.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.312+0000 7f631c000640 1 -- 172.21.15.28:0/3822953498 >> 172.21.15.28:0/3822953498 conn(0x7f63140c8810 msgr2=0x7f6314150d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:41.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.312+0000 7f631c000640 1 -- 172.21.15.28:0/3822953498 shutdown_connections 2024-02-21T23:17:41.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.312+0000 7f631c000640 1 -- 172.21.15.28:0/3822953498 wait complete. 2024-02-21T23:17:41.321 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-02-21T23:17:41.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.554+0000 7f56d4c00640 1 Processor -- start 2024-02-21T23:17:41.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.554+0000 7f56d4c00640 1 -- start start 2024-02-21T23:17:41.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.556+0000 7f56d4c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f56d0152740 0x7f56d00c6a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:41.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.556+0000 7f56d4c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f56d00c5260 0x7f56d00c56f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:41.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.556+0000 7f56d4c00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56d00c6640 con 0x7f56d00bf880 2024-02-21T23:17:41.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.556+0000 7f56d4c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56d0155fc0 con 0x7f56d00c5260 2024-02-21T23:17:41.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.556+0000 7f56d4c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56d0155200 con 0x7f56d0152740 2024-02-21T23:17:41.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.556+0000 7f56cfe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f56d00c5260 0x7f56d00c56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:41.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.556+0000 7f56cfe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f56d00c5260 0x7f56d00c56f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59784/0 (socket says 172.21.15.28:59784) 2024-02-21T23:17:41.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.556+0000 7f56cfe00640 1 -- 172.21.15.28:0/3603017288 learned_addr learned my addr 172.21.15.28:0/3603017288 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:41.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.556+0000 7f56cf400640 1 --2- 172.21.15.28:0/3603017288 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f56d0152740 0x7f56d00c6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:41.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.557+0000 7f56cfe00640 1 -- 172.21.15.28:0/3603017288 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f56d0152740 msgr2=0x7f56d00c6a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:41.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.557+0000 7f56cfe00640 1 --2- 172.21.15.28:0/3603017288 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f56d0152740 0x7f56d00c6a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:41.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.557+0000 7f56cfe00640 1 -- 172.21.15.28:0/3603017288 >> v1:172.21.15.81:6789/0 conn(0x7f56d00bf880 legacy=0x7f56d00c72f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:41.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.557+0000 7f56cfe00640 1 -- 172.21.15.28:0/3603017288 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56d01554e0 con 0x7f56d00c5260 2024-02-21T23:17:41.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.558+0000 7f56cfe00640 1 --2- 172.21.15.28:0/3603017288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56d00c5260 0x7f56d00c56f0 secure :-1 s=READY pgs=2735 cs=0 l=1 rev1=1 crypto rx=0x7f56c8009e50 tx=0x7f56c802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b52a499181e17f94 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:41.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.558+0000 7f56cea00640 1 -- 172.21.15.28:0/3603017288 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56c802c9e0 con 0x7f56d00c5260 2024-02-21T23:17:41.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.558+0000 7f56cea00640 1 -- 172.21.15.28:0/3603017288 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56c802cb80 con 0x7f56d00c5260 2024-02-21T23:17:41.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.558+0000 7f56cea00640 1 -- 172.21.15.28:0/3603017288 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56c80343f0 con 0x7f56d00c5260 2024-02-21T23:17:41.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.559+0000 7f56d4c00640 1 -- 172.21.15.28:0/3603017288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56d00c5260 msgr2=0x7f56d00c56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:41.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.559+0000 7f56d4c00640 1 --2- 172.21.15.28:0/3603017288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56d00c5260 0x7f56d00c56f0 secure :-1 s=READY pgs=2735 cs=0 l=1 rev1=1 crypto rx=0x7f56c8009e50 tx=0x7f56c802be40 comp rx=0 tx=0).stop 2024-02-21T23:17:41.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.559+0000 7f56d4c00640 1 -- 172.21.15.28:0/3603017288 shutdown_connections 2024-02-21T23:17:41.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.559+0000 7f56d4c00640 1 --2- 172.21.15.28:0/3603017288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56d00c5260 0x7f56d00c56f0 unknown :-1 s=CLOSED pgs=2735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:41.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.559+0000 7f56d4c00640 1 --2- 172.21.15.28:0/3603017288 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f56d0152740 0x7f56d00c6a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:41.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.559+0000 7f56d4c00640 1 -- 172.21.15.28:0/3603017288 >> 172.21.15.28:0/3603017288 conn(0x7f56d0067080 msgr2=0x7f56d00725a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:41.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.559+0000 7f56d4c00640 1 -- 172.21.15.28:0/3603017288 shutdown_connections 2024-02-21T23:17:41.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.559+0000 7f56d4c00640 1 -- 172.21.15.28:0/3603017288 wait complete. 2024-02-21T23:17:41.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.562+0000 7f56d4c00640 1 Processor -- start 2024-02-21T23:17:41.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.562+0000 7f56d4c00640 1 -- start start 2024-02-21T23:17:41.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.563+0000 7f56d4c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56d00bf880 0x7f56d01453f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:41.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.563+0000 7f56d4c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56d00c5260 0x7f56d014abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:41.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.563+0000 7f56d4c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f56d0152740 0x7f56d0154720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:41.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.563+0000 7f56d4c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56d00c7a00 con 0x7f56d00c5260 2024-02-21T23:17:41.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.563+0000 7f56d4c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56d0155aa0 con 0x7f56d00bf880 2024-02-21T23:17:41.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.563+0000 7f56d4c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56d00c7b80 con 0x7f56d0152740 2024-02-21T23:17:41.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.564+0000 7f56c7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56d00c5260 0x7f56d014abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:41.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.564+0000 7f56c7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56d00c5260 0x7f56d014abc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59792/0 (socket says 172.21.15.28:59792) 2024-02-21T23:17:41.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.564+0000 7f56c7e00640 1 -- 172.21.15.28:0/2614478887 learned_addr learned my addr 172.21.15.28:0/2614478887 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:41.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.564+0000 7f56cf400640 1 --2- 172.21.15.28:0/2614478887 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56d00bf880 0x7f56d01453f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:41.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.564+0000 7f56cfe00640 1 --2- 172.21.15.28:0/2614478887 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f56d0152740 0x7f56d0154720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:41.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.564+0000 7f56c7e00640 1 -- 172.21.15.28:0/2614478887 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f56d0152740 msgr2=0x7f56d0154720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:41.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.564+0000 7f56c7e00640 1 --2- 172.21.15.28:0/2614478887 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f56d0152740 0x7f56d0154720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:41.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.564+0000 7f56c7e00640 1 -- 172.21.15.28:0/2614478887 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56d00bf880 msgr2=0x7f56d01453f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:41.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.564+0000 7f56c7e00640 1 --2- 172.21.15.28:0/2614478887 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56d00bf880 0x7f56d01453f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:41.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.564+0000 7f56c7e00640 1 -- 172.21.15.28:0/2614478887 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56d00c8e50 con 0x7f56d00c5260 2024-02-21T23:17:41.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.565+0000 7f56c7e00640 1 --2- 172.21.15.28:0/2614478887 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56d00c5260 0x7f56d014abc0 secure :-1 s=READY pgs=2736 cs=0 l=1 rev1=1 crypto rx=0x7f56b800c430 tx=0x7f56b800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:41.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.565+0000 7f56c7400640 1 -- 172.21.15.28:0/2614478887 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b8011790 con 0x7f56d00c5260 2024-02-21T23:17:41.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.565+0000 7f56c7400640 1 -- 172.21.15.28:0/2614478887 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56b8017070 con 0x7f56d00c5260 2024-02-21T23:17:41.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.565+0000 7f56c7400640 1 -- 172.21.15.28:0/2614478887 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b8011bb0 con 0x7f56d00c5260 2024-02-21T23:17:41.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.565+0000 7f56d4c00640 1 -- 172.21.15.28:0/2614478887 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56d00c9130 con 0x7f56d00c5260 2024-02-21T23:17:41.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.565+0000 7f56d4c00640 1 -- 172.21.15.28:0/2614478887 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56d014b400 con 0x7f56d00c5260 2024-02-21T23:17:41.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.567+0000 7f56d4c00640 1 -- 172.21.15.28:0/2614478887 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56d00b0c00 con 0x7f56d00c5260 2024-02-21T23:17:41.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.568+0000 7f56c7400640 1 -- 172.21.15.28:0/2614478887 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f56b801c460 con 0x7f56d00c5260 2024-02-21T23:17:41.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.569+0000 7f56c7400640 1 --2- 172.21.15.28:0/2614478887 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f56ac06b4d0 0x7f56ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:41.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.569+0000 7f56c7400640 1 -- 172.21.15.28:0/2614478887 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(212..212 src has 1..212) v4 ==== 6360+0+0 (secure 0 0 0) 0x7f56b808a230 con 0x7f56d00c5260 2024-02-21T23:17:41.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.569+0000 7f56cf400640 1 --2- 172.21.15.28:0/2614478887 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f56ac06b4d0 0x7f56ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:41.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.571+0000 7f56cf400640 1 --2- 172.21.15.28:0/2614478887 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f56ac06b4d0 0x7f56ac06d990 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f56c00079a0 tx=0x7f56c0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:41.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.573+0000 7f56c7400640 1 -- 172.21.15.28:0/2614478887 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f56d00b0c00 con 0x7f56d00c5260 2024-02-21T23:17:41.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.877+0000 7f56d4c00640 1 -- 172.21.15.28:0/2614478887 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f56d004f290 con 0x7f56d00c5260 2024-02-21T23:17:41.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.882+0000 7f56c7400640 1 -- 172.21.15.28:0/2614478887 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v248) v1 ==== 161+0+0 (secure 0 0 0) 0x7f56b8058eb0 con 0x7f56d00c5260 2024-02-21T23:17:41.870 INFO:teuthology.orchestra.run.smithi028.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-02-21T23:17:41.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.886+0000 7f56d4c00640 1 -- 172.21.15.28:0/2614478887 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f56ac06b4d0 msgr2=0x7f56ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:41.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.886+0000 7f56d4c00640 1 --2- 172.21.15.28:0/2614478887 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f56ac06b4d0 0x7f56ac06d990 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f56c00079a0 tx=0x7f56c0008040 comp rx=0 tx=0).stop 2024-02-21T23:17:41.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.886+0000 7f56d4c00640 1 -- 172.21.15.28:0/2614478887 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56d00c5260 msgr2=0x7f56d014abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:41.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.886+0000 7f56d4c00640 1 --2- 172.21.15.28:0/2614478887 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56d00c5260 0x7f56d014abc0 secure :-1 s=READY pgs=2736 cs=0 l=1 rev1=1 crypto rx=0x7f56b800c430 tx=0x7f56b800c8f0 comp rx=0 tx=0).stop 2024-02-21T23:17:41.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.886+0000 7f56d4c00640 1 -- 172.21.15.28:0/2614478887 shutdown_connections 2024-02-21T23:17:41.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.886+0000 7f56d4c00640 1 --2- 172.21.15.28:0/2614478887 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f56ac06b4d0 0x7f56ac06d990 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:41.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.886+0000 7f56d4c00640 1 --2- 172.21.15.28:0/2614478887 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f56d0152740 0x7f56d0154720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:41.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.886+0000 7f56d4c00640 1 --2- 172.21.15.28:0/2614478887 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56d00c5260 0x7f56d014abc0 unknown :-1 s=CLOSED pgs=2736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:41.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.886+0000 7f56d4c00640 1 --2- 172.21.15.28:0/2614478887 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f56d00bf880 0x7f56d01453f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:41.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.886+0000 7f56d4c00640 1 -- 172.21.15.28:0/2614478887 >> 172.21.15.28:0/2614478887 conn(0x7f56d0067080 msgr2=0x7f56d00c72f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:41.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.886+0000 7f56d4c00640 1 -- 172.21.15.28:0/2614478887 shutdown_connections 2024-02-21T23:17:41.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:41.886+0000 7f56d4c00640 1 -- 172.21.15.28:0/2614478887 wait complete. 2024-02-21T23:17:41.886 INFO:tasks.ceph.mds.c.smithi028.stderr: -9432> 2024-02-21T23:13:34.633+0000 7fc87a200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-21T23:17:41.886 INFO:tasks.ceph.mds.c.smithi028.stderr: -9133> 2024-02-21T23:13:39.634+0000 7fc87a200640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-02-21T23:17:41.893 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-02-21T23:17:41.960 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-02-21T23:17:42.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.103+0000 7f187b000640 1 Processor -- start 2024-02-21T23:17:42.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.103+0000 7f187b000640 1 -- start start 2024-02-21T23:17:42.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.105+0000 7f187b000640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f18740c4960 0x7f18740c6220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:42.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.105+0000 7f187b000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1874158dc0 0x7f1874152930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:42.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.105+0000 7f187b000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18740c20d0 con 0x7f18741524f0 2024-02-21T23:17:42.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.105+0000 7f187b000640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18740a62e0 con 0x7f18740c4960 2024-02-21T23:17:42.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.105+0000 7f187b000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18740c1960 con 0x7f1874158dc0 2024-02-21T23:17:42.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.106+0000 7f186be00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f18741524f0 0x7f18741586b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59860/0 (socket says 172.21.15.28:59860) 2024-02-21T23:17:42.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.106+0000 7f186be00640 1 -- 172.21.15.28:0/2229830917 learned_addr learned my addr 172.21.15.28:0/2229830917 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:17:42.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.106+0000 7f1873e00640 1 --2- 172.21.15.28:0/2229830917 >> v2:172.21.15.81:3300/0 conn(0x7f18740c4960 0x7f18740c6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:42.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.106+0000 7f1879000640 1 --2- 172.21.15.28:0/2229830917 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1874158dc0 0x7f1874152930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:42.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.106+0000 7f1873400640 1 -- 172.21.15.28:0/2229830917 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2541251061 0 0) 0x7f18740c20d0 con 0x7f18741524f0 2024-02-21T23:17:42.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.106+0000 7f1873400640 1 -- 172.21.15.28:0/2229830917 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1858003520 con 0x7f18741524f0 2024-02-21T23:17:42.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.106+0000 7f1873e00640 1 -- 172.21.15.28:0/2229830917 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1874158dc0 msgr2=0x7f1874152930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:42.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.106+0000 7f1873e00640 1 --2- 172.21.15.28:0/2229830917 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1874158dc0 0x7f1874152930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:42.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.106+0000 7f1873e00640 1 -- 172.21.15.28:0/2229830917 >> v1:172.21.15.28:6789/0 conn(0x7f18741524f0 legacy=0x7f18741586b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:42.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.106+0000 7f1873400640 1 -- 172.21.15.28:0/2229830917 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2547061137 0 0) 0x7f1858003520 con 0x7f18741524f0 2024-02-21T23:17:42.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.106+0000 7f1873e00640 1 -- 172.21.15.28:0/2229830917 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18740c1290 con 0x7f18740c4960 2024-02-21T23:17:42.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.108+0000 7f1873e00640 1 --2- 172.21.15.28:0/2229830917 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f18740c4960 0x7f18740c6220 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f18640097a0 tx=0x7f186402b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c6d73dcd1edb20b5 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:42.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.108+0000 7f1873400640 1 -- 172.21.15.28:0/2229830917 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1864037070 con 0x7f18740c4960 2024-02-21T23:17:42.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.109+0000 7f1873400640 1 -- 172.21.15.28:0/2229830917 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1864030510 con 0x7f18740c4960 2024-02-21T23:17:42.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.109+0000 7f1873400640 1 -- 172.21.15.28:0/2229830917 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f185c001670 con 0x7f18740c4960 2024-02-21T23:17:42.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.109+0000 7f187b000640 1 -- 172.21.15.28:0/2229830917 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f18740c4960 msgr2=0x7f18740c6220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:42.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.109+0000 7f187b000640 1 --2- 172.21.15.28:0/2229830917 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f18740c4960 0x7f18740c6220 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f18640097a0 tx=0x7f186402b710 comp rx=0 tx=0).stop 2024-02-21T23:17:42.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.109+0000 7f187b000640 1 -- 172.21.15.28:0/2229830917 shutdown_connections 2024-02-21T23:17:42.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.109+0000 7f187b000640 1 --2- 172.21.15.28:0/2229830917 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f1874158dc0 0x7f1874152930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:42.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.109+0000 7f187b000640 1 --2- 172.21.15.28:0/2229830917 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f18740c4960 0x7f18740c6220 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:42.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.109+0000 7f187b000640 1 -- 172.21.15.28:0/2229830917 >> 172.21.15.28:0/2229830917 conn(0x7f18740a85a0 msgr2=0x7f18740a8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:42.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.109+0000 7f187b000640 1 -- 172.21.15.28:0/2229830917 shutdown_connections 2024-02-21T23:17:42.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.109+0000 7f187b000640 1 -- 172.21.15.28:0/2229830917 wait complete. 2024-02-21T23:17:42.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.111+0000 7f187b000640 1 Processor -- start 2024-02-21T23:17:42.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.111+0000 7f187b000640 1 -- start start 2024-02-21T23:17:42.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.112+0000 7f187b000640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f18740c4960 0x7f1874145d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:42.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.112+0000 7f187b000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f18741524f0 0x7f18740ab5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:42.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.112+0000 7f187b000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1874158dc0 0x7f18741e7a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:42.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.112+0000 7f187b000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18740c1bb0 con 0x7f1874158dc0 2024-02-21T23:17:42.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.112+0000 7f187b000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1874159250 con 0x7f18741524f0 2024-02-21T23:17:42.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.112+0000 7f187b000640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18740a5f20 con 0x7f18740c4960 2024-02-21T23:17:42.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.112+0000 7f1879000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1874158dc0 0x7f18741e7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:42.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.112+0000 7f1879000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1874158dc0 0x7f18741e7a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59800/0 (socket says 172.21.15.28:59800) 2024-02-21T23:17:42.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.112+0000 7f1879000640 1 -- 172.21.15.28:0/2509491091 learned_addr learned my addr 172.21.15.28:0/2509491091 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:42.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.113+0000 7f186be00640 1 --2- 172.21.15.28:0/2509491091 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f18741524f0 0x7f18740ab5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:42.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.113+0000 7f1879000640 1 -- 172.21.15.28:0/2509491091 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f18740c4960 msgr2=0x7f1874145d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:42.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.113+0000 7f1873e00640 1 --2- 172.21.15.28:0/2509491091 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f18740c4960 0x7f1874145d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:42.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.113+0000 7f1879000640 1 --2- 172.21.15.28:0/2509491091 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f18740c4960 0x7f1874145d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:42.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.113+0000 7f1879000640 1 -- 172.21.15.28:0/2509491091 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f18741524f0 msgr2=0x7f18740ab5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:42.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.113+0000 7f1879000640 1 --2- 172.21.15.28:0/2509491091 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f18741524f0 0x7f18740ab5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:42.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.113+0000 7f1879000640 1 -- 172.21.15.28:0/2509491091 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1874072600 con 0x7f1874158dc0 2024-02-21T23:17:42.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.113+0000 7f186be00640 1 --2- 172.21.15.28:0/2509491091 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f18741524f0 0x7f18740ab5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:17:42.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.113+0000 7f1879000640 1 --2- 172.21.15.28:0/2509491091 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1874158dc0 0x7f18741e7a00 secure :-1 s=READY pgs=2740 cs=0 l=1 rev1=1 crypto rx=0x7f186c009d30 tx=0x7f186c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:42.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.114+0000 7f1870c00640 1 -- 172.21.15.28:0/2509491091 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f186c018020 con 0x7f1874158dc0 2024-02-21T23:17:42.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.114+0000 7f1870c00640 1 -- 172.21.15.28:0/2509491091 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f186c014480 con 0x7f1874158dc0 2024-02-21T23:17:42.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.114+0000 7f1870c00640 1 -- 172.21.15.28:0/2509491091 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f186c014870 con 0x7f1874158dc0 2024-02-21T23:17:42.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.114+0000 7f187b000640 1 -- 172.21.15.28:0/2509491091 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1874072040 con 0x7f1874158dc0 2024-02-21T23:17:42.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.114+0000 7f187b000640 1 -- 172.21.15.28:0/2509491091 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18740a9fe0 con 0x7f1874158dc0 2024-02-21T23:17:42.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.115+0000 7f187b000640 1 -- 172.21.15.28:0/2509491091 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1874067f30 con 0x7f1874158dc0 2024-02-21T23:17:42.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.116+0000 7f1870c00640 1 -- 172.21.15.28:0/2509491091 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f186c005d10 con 0x7f1874158dc0 2024-02-21T23:17:42.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.117+0000 7f1870c00640 1 --2- 172.21.15.28:0/2509491091 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f185006b4d0 0x7f185006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:42.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.117+0000 7f1870c00640 1 -- 172.21.15.28:0/2509491091 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f186c08aff0 con 0x7f1874158dc0 2024-02-21T23:17:42.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.118+0000 7f1873e00640 1 --2- 172.21.15.28:0/2509491091 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f185006b4d0 0x7f185006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:42.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.119+0000 7f1873e00640 1 --2- 172.21.15.28:0/2509491091 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f185006b4d0 0x7f185006d990 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f1864034000 tx=0x7f1864008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:42.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.121+0000 7f1870c00640 1 -- 172.21.15.28:0/2509491091 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f186c022070 con 0x7f1874158dc0 2024-02-21T23:17:42.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.361+0000 7f187b000640 1 -- 172.21.15.28:0/2509491091 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f1874152b80 con 0x7f1874158dc0 2024-02-21T23:17:42.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.885+0000 7f1870c00640 1 -- 172.21.15.28:0/2509491091 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v249) v1 ==== 104+0+0 (secure 0 0 0) 0x7f186c010040 con 0x7f1874158dc0 2024-02-21T23:17:42.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.889+0000 7f187b000640 1 -- 172.21.15.28:0/2509491091 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f185006b4d0 msgr2=0x7f185006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:42.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.889+0000 7f187b000640 1 --2- 172.21.15.28:0/2509491091 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f185006b4d0 0x7f185006d990 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f1864034000 tx=0x7f1864008030 comp rx=0 tx=0).stop 2024-02-21T23:17:42.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.889+0000 7f187b000640 1 -- 172.21.15.28:0/2509491091 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1874158dc0 msgr2=0x7f18741e7a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:42.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.889+0000 7f187b000640 1 --2- 172.21.15.28:0/2509491091 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1874158dc0 0x7f18741e7a00 secure :-1 s=READY pgs=2740 cs=0 l=1 rev1=1 crypto rx=0x7f186c009d30 tx=0x7f186c00c490 comp rx=0 tx=0).stop 2024-02-21T23:17:42.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.889+0000 7f187b000640 1 -- 172.21.15.28:0/2509491091 shutdown_connections 2024-02-21T23:17:42.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.889+0000 7f187b000640 1 --2- 172.21.15.28:0/2509491091 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f185006b4d0 0x7f185006d990 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:42.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.889+0000 7f187b000640 1 --2- 172.21.15.28:0/2509491091 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1874158dc0 0x7f18741e7a00 unknown :-1 s=CLOSED pgs=2740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:42.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.889+0000 7f187b000640 1 --2- 172.21.15.28:0/2509491091 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f18741524f0 0x7f18740ab5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:42.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.889+0000 7f187b000640 1 --2- 172.21.15.28:0/2509491091 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f18740c4960 0x7f1874145d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:42.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.889+0000 7f187b000640 1 -- 172.21.15.28:0/2509491091 >> 172.21.15.28:0/2509491091 conn(0x7f18740a85a0 msgr2=0x7f18740a8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:42.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.889+0000 7f187b000640 1 -- 172.21.15.28:0/2509491091 shutdown_connections 2024-02-21T23:17:42.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:42.889+0000 7f187b000640 1 -- 172.21.15.28:0/2509491091 wait complete. 2024-02-21T23:17:42.897 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-02-21T23:17:43.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:43.135+0000 7fb57b200640 1 Processor -- start 2024-02-21T23:17:43.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:43.136+0000 7fb57b200640 1 -- start start 2024-02-21T23:17:43.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:43.137+0000 7fb57b200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb5741501c0 0x7fb5740a96c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:43.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:43.137+0000 7fb57b200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb5740a9c00 0x7fb57406e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:43.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:43.137+0000 7fb57b200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb574152fd0 con 0x7fb5740c1010 2024-02-21T23:17:43.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:43.137+0000 7fb57b200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5741532b0 con 0x7fb5740a9c00 2024-02-21T23:17:43.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:43.137+0000 7fb57b200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb574153590 con 0x7fb5741501c0 2024-02-21T23:17:43.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:43.138+0000 7fb579400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb5740a9c00 0x7fb57406e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:43.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:43.138+0000 7fb579400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb5740a9c00 0x7fb57406e330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59802/0 (socket says 172.21.15.28:59802) 2024-02-21T23:17:43.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:43.138+0000 7fb579400640 1 -- 172.21.15.28:0/2025976087 learned_addr learned my addr 172.21.15.28:0/2025976087 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:43.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:43.138+0000 7fb579e00640 1 --2- 172.21.15.28:0/2025976087 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb5741501c0 0x7fb5740a96c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:43.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:43.138+0000 7fb579400640 1 -- 172.21.15.28:0/2025976087 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb5741501c0 msgr2=0x7fb5740a96c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:43.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:43.138+0000 7fb579400640 1 --2- 172.21.15.28:0/2025976087 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb5741501c0 0x7fb5740a96c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:43.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:43.138+0000 7fb579400640 1 -- 172.21.15.28:0/2025976087 >> v1:172.21.15.81:6789/0 conn(0x7fb5740c1010 legacy=0x7fb57406e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:43.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:43.138+0000 7fb579400640 1 -- 172.21.15.28:0/2025976087 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb574153870 con 0x7fb5740a9c00 2024-02-21T23:17:43.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:43.138+0000 7fb579e00640 1 --2- 172.21.15.28:0/2025976087 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb5741501c0 0x7fb5740a96c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:17:43.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:43.140+0000 7fb579400640 1 --2- 172.21.15.28:0/2025976087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5740a9c00 0x7fb57406e330 secure :-1 s=READY pgs=2741 cs=0 l=1 rev1=1 crypto rx=0x7fb564009870 tx=0x7fb56402b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=294f28a772b02497 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:43.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:43.140+0000 7fb578a00640 1 -- 172.21.15.28:0/2025976087 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb564038070 con 0x7fb5740a9c00 2024-02-21T23:17:43.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:43.140+0000 7fb578a00640 1 -- 172.21.15.28:0/2025976087 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb564031740 con 0x7fb5740a9c00 2024-02-21T23:17:43.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:43.140+0000 7fb578a00640 1 -- 172.21.15.28:0/2025976087 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb56402f050 con 0x7fb5740a9c00 2024-02-21T23:17:43.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:43.141+0000 7fb57b200640 1 -- 172.21.15.28:0/2025976087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5740a9c00 msgr2=0x7fb57406e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:43.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:43.141+0000 7fb57b200640 1 --2- 172.21.15.28:0/2025976087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5740a9c00 0x7fb57406e330 secure :-1 s=READY pgs=2741 cs=0 l=1 rev1=1 crypto rx=0x7fb564009870 tx=0x7fb56402b7e0 comp rx=0 tx=0).stop 2024-02-21T23:17:43.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:43.141+0000 7fb57b200640 1 -- 172.21.15.28:0/2025976087 shutdown_connections 2024-02-21T23:17:43.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:43.141+0000 7fb57b200640 1 --2- 172.21.15.28:0/2025976087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5740a9c00 0x7fb57406e330 unknown :-1 s=CLOSED pgs=2741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:43.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:43.141+0000 7fb57b200640 1 --2- 172.21.15.28:0/2025976087 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb5741501c0 0x7fb5740a96c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:43.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:43.141+0000 7fb57b200640 1 -- 172.21.15.28:0/2025976087 >> 172.21.15.28:0/2025976087 conn(0x7fb5740c6610 msgr2=0x7fb5740c69f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:43.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:43.141+0000 7fb57b200640 1 -- 172.21.15.28:0/2025976087 shutdown_connections 2024-02-21T23:17:43.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:43.141+0000 7fb57b200640 1 -- 172.21.15.28:0/2025976087 wait complete. 2024-02-21T23:17:43.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:43.143+0000 7fb57b200640 1 Processor -- start 2024-02-21T23:17:43.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:43.143+0000 7fb57b200640 1 -- start start 2024-02-21T23:17:43.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:43.144+0000 7fb57b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5740a9c00 0x7fb574145dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:43.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:43.144+0000 7fb57b200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5740c1010 0x7fb574154d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:43.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:43.144+0000 7fb57b200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb5741501c0 0x7fb5741517b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:43.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:43.144+0000 7fb57b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5740c0c20 con 0x7fb5740a9c00 2024-02-21T23:17:43.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:43.144+0000 7fb57b200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb57406ef80 con 0x7fb5740c1010 2024-02-21T23:17:43.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:43.144+0000 7fb57b200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5740c7340 con 0x7fb5741501c0 2024-02-21T23:17:43.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:43.144+0000 7fb579e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5740a9c00 0x7fb574145dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:43.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:43.144+0000 7fb579e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5740a9c00 0x7fb574145dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59804/0 (socket says 172.21.15.28:59804) 2024-02-21T23:17:43.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:43.144+0000 7fb579e00640 1 -- 172.21.15.28:0/3672737569 learned_addr learned my addr 172.21.15.28:0/3672737569 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:43.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:43.144+0000 7fb579400640 1 --2- 172.21.15.28:0/3672737569 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5740c1010 0x7fb574154d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:43.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:43.144+0000 7fb57a800640 1 --2- 172.21.15.28:0/3672737569 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb5741501c0 0x7fb5741517b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:43.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:43.144+0000 7fb579e00640 1 -- 172.21.15.28:0/3672737569 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb5741501c0 msgr2=0x7fb5741517b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:43.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:43.144+0000 7fb579e00640 1 --2- 172.21.15.28:0/3672737569 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb5741501c0 0x7fb5741517b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:43.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:43.144+0000 7fb579e00640 1 -- 172.21.15.28:0/3672737569 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5740c1010 msgr2=0x7fb574154d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:43.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:43.144+0000 7fb579e00640 1 --2- 172.21.15.28:0/3672737569 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5740c1010 0x7fb574154d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:43.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:43.144+0000 7fb579e00640 1 -- 172.21.15.28:0/3672737569 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb574073160 con 0x7fb5740a9c00 2024-02-21T23:17:43.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:43.145+0000 7fb57a800640 1 --2- 172.21.15.28:0/3672737569 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb5741501c0 0x7fb5741517b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:17:43.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:43.145+0000 7fb579400640 1 --2- 172.21.15.28:0/3672737569 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5740c1010 0x7fb574154d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:17:43.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:43.145+0000 7fb579e00640 1 --2- 172.21.15.28:0/3672737569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5740a9c00 0x7fb574145dc0 secure :-1 s=READY pgs=2742 cs=0 l=1 rev1=1 crypto rx=0x7fb56c00ce30 tx=0x7fb56c00a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:43.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:43.145+0000 7fb56a000640 1 -- 172.21.15.28:0/3672737569 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb56c00f890 con 0x7fb5740a9c00 2024-02-21T23:17:43.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:43.145+0000 7fb56a000640 1 -- 172.21.15.28:0/3672737569 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb56c00fa30 con 0x7fb5740a9c00 2024-02-21T23:17:43.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:43.145+0000 7fb57b200640 1 -- 172.21.15.28:0/3672737569 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb574145ae0 con 0x7fb5740a9c00 2024-02-21T23:17:43.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:43.145+0000 7fb56a000640 1 -- 172.21.15.28:0/3672737569 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb56c00fe00 con 0x7fb5740a9c00 2024-02-21T23:17:43.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:43.145+0000 7fb57b200640 1 -- 172.21.15.28:0/3672737569 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb574151cf0 con 0x7fb5740a9c00 2024-02-21T23:17:43.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:43.146+0000 7fb57b200640 1 -- 172.21.15.28:0/3672737569 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb574072c70 con 0x7fb5740a9c00 2024-02-21T23:17:43.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:43.147+0000 7fb56a000640 1 -- 172.21.15.28:0/3672737569 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb56c018400 con 0x7fb5740a9c00 2024-02-21T23:17:43.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:43.148+0000 7fb56a000640 1 --2- 172.21.15.28:0/3672737569 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb55806b4d0 0x7fb55806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:43.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:43.148+0000 7fb56a000640 1 -- 172.21.15.28:0/3672737569 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fb56c08b2d0 con 0x7fb5740a9c00 2024-02-21T23:17:43.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:43.149+0000 7fb579400640 1 --2- 172.21.15.28:0/3672737569 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb55806b4d0 0x7fb55806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:43.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:43.150+0000 7fb579400640 1 --2- 172.21.15.28:0/3672737569 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb55806b4d0 0x7fb55806d990 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fb564035000 tx=0x7fb564008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:43.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:43.153+0000 7fb56a000640 1 -- 172.21.15.28:0/3672737569 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb56c014030 con 0x7fb5740a9c00 2024-02-21T23:17:43.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:43.353+0000 7fb57b200640 1 -- 172.21.15.28:0/3672737569 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb57404f290 con 0x7fb5740a9c00 2024-02-21T23:17:44.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.211+0000 7fb56a000640 1 -- 172.21.15.28:0/3672737569 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v214) v1 ==== 181+0+0 (secure 0 0 0) 0x7fb56c00ae20 con 0x7fb5740a9c00 2024-02-21T23:17:44.200 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' removed 2024-02-21T23:17:44.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.215+0000 7fb57b200640 1 -- 172.21.15.28:0/3672737569 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb55806b4d0 msgr2=0x7fb55806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:44.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.215+0000 7fb57b200640 1 --2- 172.21.15.28:0/3672737569 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb55806b4d0 0x7fb55806d990 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fb564035000 tx=0x7fb564008030 comp rx=0 tx=0).stop 2024-02-21T23:17:44.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.215+0000 7fb57b200640 1 -- 172.21.15.28:0/3672737569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5740a9c00 msgr2=0x7fb574145dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:44.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.215+0000 7fb57b200640 1 --2- 172.21.15.28:0/3672737569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5740a9c00 0x7fb574145dc0 secure :-1 s=READY pgs=2742 cs=0 l=1 rev1=1 crypto rx=0x7fb56c00ce30 tx=0x7fb56c00a650 comp rx=0 tx=0).stop 2024-02-21T23:17:44.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.215+0000 7fb57b200640 1 -- 172.21.15.28:0/3672737569 shutdown_connections 2024-02-21T23:17:44.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.215+0000 7fb57b200640 1 --2- 172.21.15.28:0/3672737569 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb55806b4d0 0x7fb55806d990 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:44.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.215+0000 7fb57b200640 1 --2- 172.21.15.28:0/3672737569 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb5741501c0 0x7fb5741517b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:44.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.215+0000 7fb57b200640 1 --2- 172.21.15.28:0/3672737569 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5740c1010 0x7fb574154d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:44.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.215+0000 7fb57b200640 1 --2- 172.21.15.28:0/3672737569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5740a9c00 0x7fb574145dc0 unknown :-1 s=CLOSED pgs=2742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:44.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.215+0000 7fb57b200640 1 -- 172.21.15.28:0/3672737569 >> 172.21.15.28:0/3672737569 conn(0x7fb5740c6610 msgr2=0x7fb57414eb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:44.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.215+0000 7fb57b200640 1 -- 172.21.15.28:0/3672737569 shutdown_connections 2024-02-21T23:17:44.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.215+0000 7fb57b200640 1 -- 172.21.15.28:0/3672737569 wait complete. 2024-02-21T23:17:44.233 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-02-21T23:17:44.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.443+0000 7f94ddc00640 1 Processor -- start 2024-02-21T23:17:44.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.443+0000 7f94ddc00640 1 -- start start 2024-02-21T23:17:44.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.445+0000 7f94ddc00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f94d81501c0 0x7f94d80a96c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:44.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.445+0000 7f94ddc00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f94d80a9c00 0x7f94d806e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:44.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.445+0000 7f94ddc00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94d8152fd0 con 0x7f94d80c1010 2024-02-21T23:17:44.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.445+0000 7f94ddc00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94d81532b0 con 0x7f94d81501c0 2024-02-21T23:17:44.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.445+0000 7f94ddc00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94d8153590 con 0x7f94d80a9c00 2024-02-21T23:17:44.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.445+0000 7f94d7e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f94d81501c0 0x7f94d80a96c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:44.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.445+0000 7f94d7e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f94d81501c0 0x7f94d80a96c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59814/0 (socket says 172.21.15.28:59814) 2024-02-21T23:17:44.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.445+0000 7f94d7e00640 1 -- 172.21.15.28:0/4149417282 learned_addr learned my addr 172.21.15.28:0/4149417282 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:44.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.446+0000 7f94d7e00640 1 -- 172.21.15.28:0/4149417282 >> v2:172.21.15.81:3300/0 conn(0x7f94d80a9c00 msgr2=0x7f94d806e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:44.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.446+0000 7f94d7e00640 1 --2- 172.21.15.28:0/4149417282 >> v2:172.21.15.81:3300/0 conn(0x7f94d80a9c00 0x7f94d806e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:44.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.446+0000 7f94d7e00640 1 -- 172.21.15.28:0/4149417282 >> v1:172.21.15.81:6789/0 conn(0x7f94d80c1010 legacy=0x7f94d806e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:44.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.446+0000 7f94d7e00640 1 -- 172.21.15.28:0/4149417282 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94d8153870 con 0x7f94d81501c0 2024-02-21T23:17:44.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.447+0000 7f94d7e00640 1 --2- 172.21.15.28:0/4149417282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94d81501c0 0x7f94d80a96c0 secure :-1 s=READY pgs=2743 cs=0 l=1 rev1=1 crypto rx=0x7f94d80c41d0 tx=0x7f94c802b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b82bb4b61a1e28b6 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:44.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.447+0000 7f94d7400640 1 -- 172.21.15.28:0/4149417282 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c8038070 con 0x7f94d81501c0 2024-02-21T23:17:44.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.447+0000 7f94d7400640 1 -- 172.21.15.28:0/4149417282 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94c802f050 con 0x7f94d81501c0 2024-02-21T23:17:44.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.447+0000 7f94d7400640 1 -- 172.21.15.28:0/4149417282 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c8031970 con 0x7f94d81501c0 2024-02-21T23:17:44.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.448+0000 7f94ddc00640 1 -- 172.21.15.28:0/4149417282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94d81501c0 msgr2=0x7f94d80a96c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:44.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.448+0000 7f94ddc00640 1 --2- 172.21.15.28:0/4149417282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94d81501c0 0x7f94d80a96c0 secure :-1 s=READY pgs=2743 cs=0 l=1 rev1=1 crypto rx=0x7f94d80c41d0 tx=0x7f94c802b7f0 comp rx=0 tx=0).stop 2024-02-21T23:17:44.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.448+0000 7f94ddc00640 1 -- 172.21.15.28:0/4149417282 shutdown_connections 2024-02-21T23:17:44.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.448+0000 7f94ddc00640 1 --2- 172.21.15.28:0/4149417282 >> v2:172.21.15.81:3300/0 conn(0x7f94d80a9c00 0x7f94d806e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:44.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.448+0000 7f94ddc00640 1 --2- 172.21.15.28:0/4149417282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94d81501c0 0x7f94d80a96c0 unknown :-1 s=CLOSED pgs=2743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:44.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.448+0000 7f94ddc00640 1 -- 172.21.15.28:0/4149417282 >> 172.21.15.28:0/4149417282 conn(0x7f94d80c6610 msgr2=0x7f94d80c69f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:44.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.448+0000 7f94ddc00640 1 -- 172.21.15.28:0/4149417282 shutdown_connections 2024-02-21T23:17:44.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.448+0000 7f94ddc00640 1 -- 172.21.15.28:0/4149417282 wait complete. 2024-02-21T23:17:44.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.451+0000 7f94ddc00640 1 Processor -- start 2024-02-21T23:17:44.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.451+0000 7f94ddc00640 1 -- start start 2024-02-21T23:17:44.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.452+0000 7f94ddc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94d80a9c00 0x7f94d8144e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:44.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.452+0000 7f94ddc00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f94d80c1010 0x7f94d814e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:44.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.452+0000 7f94ddc00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f94d81501c0 0x7f94d8156700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:44.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.452+0000 7f94ddc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94d80c0c20 con 0x7f94d80a9c00 2024-02-21T23:17:44.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.452+0000 7f94ddc00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94d806ef80 con 0x7f94d80c1010 2024-02-21T23:17:44.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.452+0000 7f94ddc00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94d80c7340 con 0x7f94d81501c0 2024-02-21T23:17:44.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.452+0000 7f94d7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94d80a9c00 0x7f94d8144e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:44.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.452+0000 7f94d7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94d80a9c00 0x7f94d8144e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59818/0 (socket says 172.21.15.28:59818) 2024-02-21T23:17:44.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.452+0000 7f94d7e00640 1 -- 172.21.15.28:0/2716196942 learned_addr learned my addr 172.21.15.28:0/2716196942 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:44.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.453+0000 7f94cfe00640 1 --2- 172.21.15.28:0/2716196942 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f94d80c1010 0x7f94d814e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:44.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.453+0000 7f94d7e00640 1 -- 172.21.15.28:0/2716196942 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f94d81501c0 msgr2=0x7f94d8156700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:44.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.453+0000 7f94d7e00640 1 --2- 172.21.15.28:0/2716196942 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f94d81501c0 0x7f94d8156700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:44.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.453+0000 7f94d7e00640 1 -- 172.21.15.28:0/2716196942 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f94d80c1010 msgr2=0x7f94d814e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:44.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.453+0000 7f94d7e00640 1 --2- 172.21.15.28:0/2716196942 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f94d80c1010 0x7f94d814e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:44.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.453+0000 7f94d7e00640 1 -- 172.21.15.28:0/2716196942 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94d81edf90 con 0x7f94d80a9c00 2024-02-21T23:17:44.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.453+0000 7f94cfe00640 1 --2- 172.21.15.28:0/2716196942 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f94d80c1010 0x7f94d814e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:17:44.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.453+0000 7f94d7e00640 1 --2- 172.21.15.28:0/2716196942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94d80a9c00 0x7f94d8144e80 secure :-1 s=READY pgs=2744 cs=0 l=1 rev1=1 crypto rx=0x7f94c802b830 tx=0x7f94c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:44.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.453+0000 7f94d4c00640 1 -- 172.21.15.28:0/2716196942 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c8043070 con 0x7f94d80a9c00 2024-02-21T23:17:44.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.454+0000 7f94d4c00640 1 -- 172.21.15.28:0/2716196942 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94c803f890 con 0x7f94d80a9c00 2024-02-21T23:17:44.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.454+0000 7f94d4c00640 1 -- 172.21.15.28:0/2716196942 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c8038040 con 0x7f94d80a9c00 2024-02-21T23:17:44.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.454+0000 7f94ddc00640 1 -- 172.21.15.28:0/2716196942 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94d81ee270 con 0x7f94d80a9c00 2024-02-21T23:17:44.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.454+0000 7f94ddc00640 1 -- 172.21.15.28:0/2716196942 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94d814e750 con 0x7f94d80a9c00 2024-02-21T23:17:44.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.455+0000 7f94ddc00640 1 -- 172.21.15.28:0/2716196942 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94d80c2b00 con 0x7f94d80a9c00 2024-02-21T23:17:44.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.456+0000 7f94d4c00640 1 -- 172.21.15.28:0/2716196942 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f94c8046070 con 0x7f94d80a9c00 2024-02-21T23:17:44.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.457+0000 7f94d4c00640 1 --2- 172.21.15.28:0/2716196942 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f94b406b500 0x7f94b406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:44.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.457+0000 7f94d4c00640 1 -- 172.21.15.28:0/2716196942 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6094+0+0 (secure 0 0 0) 0x7f94c80ac4c0 con 0x7f94d80a9c00 2024-02-21T23:17:44.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.457+0000 7f94cfe00640 1 --2- 172.21.15.28:0/2716196942 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f94b406b500 0x7f94b406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:44.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.458+0000 7f94cfe00640 1 --2- 172.21.15.28:0/2716196942 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f94b406b500 0x7f94b406d9c0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f94d814c550 tx=0x7f94c0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:44.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.461+0000 7f94d4c00640 1 -- 172.21.15.28:0/2716196942 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f94d80c2b00 con 0x7f94d80a9c00 2024-02-21T23:17:44.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:44.666+0000 7f94ddc00640 1 -- 172.21.15.28:0/2716196942 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f94d814ea10 con 0x7f94d80a9c00 2024-02-21T23:17:45.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.223+0000 7f94d4c00640 1 -- 172.21.15.28:0/2716196942 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v215) v1 ==== 169+0+0 (secure 0 0 0) 0x7f94c807b240 con 0x7f94d80a9c00 2024-02-21T23:17:45.211 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' removed 2024-02-21T23:17:45.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.227+0000 7f94ddc00640 1 -- 172.21.15.28:0/2716196942 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f94b406b500 msgr2=0x7f94b406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:45.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.227+0000 7f94ddc00640 1 --2- 172.21.15.28:0/2716196942 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f94b406b500 0x7f94b406d9c0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f94d814c550 tx=0x7f94c0008040 comp rx=0 tx=0).stop 2024-02-21T23:17:45.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.227+0000 7f94ddc00640 1 -- 172.21.15.28:0/2716196942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94d80a9c00 msgr2=0x7f94d8144e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:45.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.227+0000 7f94ddc00640 1 --2- 172.21.15.28:0/2716196942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94d80a9c00 0x7f94d8144e80 secure :-1 s=READY pgs=2744 cs=0 l=1 rev1=1 crypto rx=0x7f94c802b830 tx=0x7f94c80046c0 comp rx=0 tx=0).stop 2024-02-21T23:17:45.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.231+0000 7f94ddc00640 1 -- 172.21.15.28:0/2716196942 shutdown_connections 2024-02-21T23:17:45.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.231+0000 7f94ddc00640 1 --2- 172.21.15.28:0/2716196942 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f94b406b500 0x7f94b406d9c0 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:45.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.231+0000 7f94ddc00640 1 --2- 172.21.15.28:0/2716196942 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f94d81501c0 0x7f94d8156700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:45.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.231+0000 7f94ddc00640 1 --2- 172.21.15.28:0/2716196942 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f94d80c1010 0x7f94d814e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:45.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.231+0000 7f94ddc00640 1 --2- 172.21.15.28:0/2716196942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94d80a9c00 0x7f94d8144e80 unknown :-1 s=CLOSED pgs=2744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:45.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.231+0000 7f94ddc00640 1 -- 172.21.15.28:0/2716196942 >> 172.21.15.28:0/2716196942 conn(0x7f94d80c6610 msgr2=0x7f94d806e330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:45.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.235+0000 7f94ddc00640 1 -- 172.21.15.28:0/2716196942 shutdown_connections 2024-02-21T23:17:45.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.235+0000 7f94ddc00640 1 -- 172.21.15.28:0/2716196942 wait complete. 2024-02-21T23:17:45.247 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-02-21T23:17:45.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.459+0000 7fb0bb800640 1 Processor -- start 2024-02-21T23:17:45.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.460+0000 7fb0bb800640 1 -- start start 2024-02-21T23:17:45.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.461+0000 7fb0bb800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb0b406f250 0x7fb0b414eb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:45.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.461+0000 7fb0bb800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fb0b40c17d0 0x7fb0b406ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:45.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.461+0000 7fb0bb800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0b40a60f0 con 0x7fb0b406d120 2024-02-21T23:17:45.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.461+0000 7fb0bb800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0b4152180 con 0x7fb0b40c17d0 2024-02-21T23:17:45.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.461+0000 7fb0bb800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b40c1390 con 0x7fb0b406f250 2024-02-21T23:17:45.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.462+0000 7fb0b8e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb0b406f250 0x7fb0b414eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:45.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.462+0000 7fb0b8e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb0b406f250 0x7fb0b414eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3301/0 says I am v2:172.21.15.28:54960/0 (socket says 172.21.15.28:54960) 2024-02-21T23:17:45.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.462+0000 7fb0b8e00640 1 -- 172.21.15.28:0/1268127854 learned_addr learned my addr 172.21.15.28:0/1268127854 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:45.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.462+0000 7fb0b3e00640 1 --2- 172.21.15.28:0/1268127854 >> v2:172.21.15.81:3300/0 conn(0x7fb0b40c17d0 0x7fb0b406ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:45.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.463+0000 7fb0b8e00640 1 -- 172.21.15.28:0/1268127854 >> v2:172.21.15.81:3300/0 conn(0x7fb0b40c17d0 msgr2=0x7fb0b406ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:45.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.463+0000 7fb0b8e00640 1 --2- 172.21.15.28:0/1268127854 >> v2:172.21.15.81:3300/0 conn(0x7fb0b40c17d0 0x7fb0b406ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:45.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.463+0000 7fb0b8e00640 1 -- 172.21.15.28:0/1268127854 >> v1:172.21.15.81:6789/0 conn(0x7fb0b406d120 legacy=0x7fb0b406d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:45.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.463+0000 7fb0b8e00640 1 -- 172.21.15.28:0/1268127854 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0b406dcc0 con 0x7fb0b406f250 2024-02-21T23:17:45.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.463+0000 7fb0b3400640 1 -- 172.21.15.28:0/1268127854 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4215994078 0 0) 0x7fb0b40a60f0 con 0x7fb0b406d120 2024-02-21T23:17:45.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.464+0000 7fb0b8e00640 1 --2- 172.21.15.28:0/1268127854 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb0b406f250 0x7fb0b414eb00 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb0a8009870 tx=0x7fb0a802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4afd6c965bd3ba86 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:45.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.465+0000 7fb0b3400640 1 -- 172.21.15.28:0/1268127854 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a8035020 con 0x7fb0b406f250 2024-02-21T23:17:45.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.465+0000 7fb0b3400640 1 -- 172.21.15.28:0/1268127854 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0a8031640 con 0x7fb0b406f250 2024-02-21T23:17:45.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.465+0000 7fb0b3400640 1 -- 172.21.15.28:0/1268127854 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a802d040 con 0x7fb0b406f250 2024-02-21T23:17:45.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.465+0000 7fb0bb800640 1 -- 172.21.15.28:0/1268127854 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb0b406f250 msgr2=0x7fb0b414eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:45.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.465+0000 7fb0bb800640 1 --2- 172.21.15.28:0/1268127854 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb0b406f250 0x7fb0b414eb00 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb0a8009870 tx=0x7fb0a802b8f0 comp rx=0 tx=0).stop 2024-02-21T23:17:45.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.465+0000 7fb0bb800640 1 -- 172.21.15.28:0/1268127854 shutdown_connections 2024-02-21T23:17:45.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.465+0000 7fb0bb800640 1 --2- 172.21.15.28:0/1268127854 >> v2:172.21.15.81:3300/0 conn(0x7fb0b40c17d0 0x7fb0b406ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:45.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.465+0000 7fb0bb800640 1 --2- 172.21.15.28:0/1268127854 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb0b406f250 0x7fb0b414eb00 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:45.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.465+0000 7fb0bb800640 1 -- 172.21.15.28:0/1268127854 >> 172.21.15.28:0/1268127854 conn(0x7fb0b40a85a0 msgr2=0x7fb0b40a8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:45.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.465+0000 7fb0bb800640 1 -- 172.21.15.28:0/1268127854 shutdown_connections 2024-02-21T23:17:45.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.465+0000 7fb0bb800640 1 -- 172.21.15.28:0/1268127854 wait complete. 2024-02-21T23:17:45.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.467+0000 7fb0bb800640 1 Processor -- start 2024-02-21T23:17:45.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.467+0000 7fb0bb800640 1 -- start start 2024-02-21T23:17:45.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.468+0000 7fb0bb800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0b406d120 0x7fb0b40a9720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:45.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.468+0000 7fb0bb800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0b406f250 0x7fb0b40aa6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:45.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.468+0000 7fb0bb800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb0b40c17d0 0x7fb0b4147ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:45.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.468+0000 7fb0bb800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b40c1ea0 con 0x7fb0b406f250 2024-02-21T23:17:45.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.468+0000 7fb0bb800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b40c1d20 con 0x7fb0b406d120 2024-02-21T23:17:45.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.468+0000 7fb0bb800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b40c2020 con 0x7fb0b40c17d0 2024-02-21T23:17:45.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.469+0000 7fb0b3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0b406f250 0x7fb0b40aa6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:45.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.469+0000 7fb0b3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0b406f250 0x7fb0b40aa6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35170/0 (socket says 172.21.15.28:35170) 2024-02-21T23:17:45.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.469+0000 7fb0b3e00640 1 -- 172.21.15.28:0/370514216 learned_addr learned my addr 172.21.15.28:0/370514216 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:45.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.469+0000 7fb0b8e00640 1 --2- 172.21.15.28:0/370514216 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0b406d120 0x7fb0b40a9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:45.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.469+0000 7fb0b9800640 1 --2- 172.21.15.28:0/370514216 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb0b40c17d0 0x7fb0b4147ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:45.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.469+0000 7fb0b3e00640 1 -- 172.21.15.28:0/370514216 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb0b40c17d0 msgr2=0x7fb0b4147ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:45.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.469+0000 7fb0b3e00640 1 --2- 172.21.15.28:0/370514216 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb0b40c17d0 0x7fb0b4147ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:45.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.469+0000 7fb0b3e00640 1 -- 172.21.15.28:0/370514216 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0b406d120 msgr2=0x7fb0b40a9720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:45.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.469+0000 7fb0b3e00640 1 --2- 172.21.15.28:0/370514216 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0b406d120 0x7fb0b40a9720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:45.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.469+0000 7fb0b3e00640 1 -- 172.21.15.28:0/370514216 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0b40a94d0 con 0x7fb0b406f250 2024-02-21T23:17:45.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.470+0000 7fb0b3e00640 1 --2- 172.21.15.28:0/370514216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0b406f250 0x7fb0b40aa6a0 secure :-1 s=READY pgs=2745 cs=0 l=1 rev1=1 crypto rx=0x7fb0a000e870 tx=0x7fb0a000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:45.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.470+0000 7fb0b0c00640 1 -- 172.21.15.28:0/370514216 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a0017070 con 0x7fb0b406f250 2024-02-21T23:17:45.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.470+0000 7fb0b0c00640 1 -- 172.21.15.28:0/370514216 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0a000c940 con 0x7fb0b406f250 2024-02-21T23:17:45.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.470+0000 7fb0bb800640 1 -- 172.21.15.28:0/370514216 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0b41ee2f0 con 0x7fb0b406f250 2024-02-21T23:17:45.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.470+0000 7fb0b0c00640 1 -- 172.21.15.28:0/370514216 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a000cd30 con 0x7fb0b406f250 2024-02-21T23:17:45.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.470+0000 7fb0bb800640 1 -- 172.21.15.28:0/370514216 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0b41ee8b0 con 0x7fb0b406f250 2024-02-21T23:17:45.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.471+0000 7fb0bb800640 1 -- 172.21.15.28:0/370514216 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0b406c560 con 0x7fb0b406f250 2024-02-21T23:17:45.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.472+0000 7fb0b0c00640 1 -- 172.21.15.28:0/370514216 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb0a001a070 con 0x7fb0b406f250 2024-02-21T23:17:45.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.473+0000 7fb0b0c00640 1 --2- 172.21.15.28:0/370514216 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb09406b500 0x7fb09406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:45.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.473+0000 7fb0b0c00640 1 -- 172.21.15.28:0/370514216 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 5593+0+0 (secure 0 0 0) 0x7fb0a008a450 con 0x7fb0b406f250 2024-02-21T23:17:45.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.474+0000 7fb0b8e00640 1 --2- 172.21.15.28:0/370514216 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb09406b500 0x7fb09406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:45.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.475+0000 7fb0b8e00640 1 --2- 172.21.15.28:0/370514216 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb09406b500 0x7fb09406d9c0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fb0a8036000 tx=0x7fb0a803c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:45.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.478+0000 7fb0b0c00640 1 -- 172.21.15.28:0/370514216 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb0b406c560 con 0x7fb0b406f250 2024-02-21T23:17:45.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.695+0000 7fb0bb800640 1 -- 172.21.15.28:0/370514216 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fb0b40c43e0 con 0x7fb0b406f250 2024-02-21T23:17:45.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.702+0000 7fb0b0c00640 1 -- 172.21.15.28:0/370514216 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v64) v1 ==== 104+0+0 (secure 0 0 0) 0x7fb0b40c43e0 con 0x7fb0b406f250 2024-02-21T23:17:45.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.705+0000 7fb0bb800640 1 -- 172.21.15.28:0/370514216 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb09406b500 msgr2=0x7fb09406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:45.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.705+0000 7fb0bb800640 1 --2- 172.21.15.28:0/370514216 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb09406b500 0x7fb09406d9c0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fb0a8036000 tx=0x7fb0a803c040 comp rx=0 tx=0).stop 2024-02-21T23:17:45.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.705+0000 7fb0bb800640 1 -- 172.21.15.28:0/370514216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0b406f250 msgr2=0x7fb0b40aa6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:45.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.705+0000 7fb0bb800640 1 --2- 172.21.15.28:0/370514216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0b406f250 0x7fb0b40aa6a0 secure :-1 s=READY pgs=2745 cs=0 l=1 rev1=1 crypto rx=0x7fb0a000e870 tx=0x7fb0a000ed30 comp rx=0 tx=0).stop 2024-02-21T23:17:45.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.705+0000 7fb0bb800640 1 -- 172.21.15.28:0/370514216 shutdown_connections 2024-02-21T23:17:45.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.705+0000 7fb0bb800640 1 --2- 172.21.15.28:0/370514216 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb09406b500 0x7fb09406d9c0 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:45.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.705+0000 7fb0bb800640 1 --2- 172.21.15.28:0/370514216 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb0b40c17d0 0x7fb0b4147ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:45.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.705+0000 7fb0bb800640 1 --2- 172.21.15.28:0/370514216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0b406f250 0x7fb0b40aa6a0 unknown :-1 s=CLOSED pgs=2745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:45.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.705+0000 7fb0bb800640 1 --2- 172.21.15.28:0/370514216 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb0b406d120 0x7fb0b40a9720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:45.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.705+0000 7fb0bb800640 1 -- 172.21.15.28:0/370514216 >> 172.21.15.28:0/370514216 conn(0x7fb0b40a85a0 msgr2=0x7fb0b4071ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:45.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.706+0000 7fb0bb800640 1 -- 172.21.15.28:0/370514216 shutdown_connections 2024-02-21T23:17:45.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.706+0000 7fb0bb800640 1 -- 172.21.15.28:0/370514216 wait complete. 2024-02-21T23:17:45.713 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_failover.TestClusterResize.test_down_twice' 2024-02-21T23:17:45.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.942+0000 7f7c08c00640 1 Processor -- start 2024-02-21T23:17:45.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.943+0000 7f7c08c00640 1 -- start start 2024-02-21T23:17:45.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.943+0000 7f7c08c00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f7c0406dff0 0x7f7c0406e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:45.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.943+0000 7f7c08c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7c040c0fe0 0x7f7c04071350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:45.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.943+0000 7f7c08c00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c040a8b40 con 0x7f7c04158580 2024-02-21T23:17:45.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.943+0000 7f7c08c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c040a8e00 con 0x7f7c040c0fe0 2024-02-21T23:17:45.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.943+0000 7f7c08c00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c040a90e0 con 0x7f7c0406dff0 2024-02-21T23:17:45.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.943+0000 7f7c02a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7c040c0fe0 0x7f7c04071350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:45.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.943+0000 7f7c02a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7c040c0fe0 0x7f7c04071350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35172/0 (socket says 172.21.15.28:35172) 2024-02-21T23:17:45.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.943+0000 7f7c02a00640 1 -- 172.21.15.28:0/4101943562 learned_addr learned my addr 172.21.15.28:0/4101943562 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:45.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.944+0000 7f7c03400640 1 --2- 172.21.15.28:0/4101943562 >> v2:172.21.15.81:3300/0 conn(0x7f7c0406dff0 0x7f7c0406e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:45.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.944+0000 7f7c02a00640 1 -- 172.21.15.28:0/4101943562 >> v2:172.21.15.81:3300/0 conn(0x7f7c0406dff0 msgr2=0x7f7c0406e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:45.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.944+0000 7f7c02a00640 1 --2- 172.21.15.28:0/4101943562 >> v2:172.21.15.81:3300/0 conn(0x7f7c0406dff0 0x7f7c0406e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:45.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.944+0000 7f7c02a00640 1 -- 172.21.15.28:0/4101943562 >> v1:172.21.15.81:6789/0 conn(0x7f7c04158580 legacy=0x7f7c0414b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:45.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.944+0000 7f7c02a00640 1 -- 172.21.15.28:0/4101943562 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c04072ed0 con 0x7f7c040c0fe0 2024-02-21T23:17:45.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.945+0000 7f7c02a00640 1 --2- 172.21.15.28:0/4101943562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c040c0fe0 0x7f7c04071350 secure :-1 s=READY pgs=2746 cs=0 l=1 rev1=1 crypto rx=0x7f7c0406be50 tx=0x7f7bf002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21d80c4fd6c0e37e server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:45.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.945+0000 7f7c02000640 1 -- 172.21.15.28:0/4101943562 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bf0035020 con 0x7f7c040c0fe0 2024-02-21T23:17:45.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.945+0000 7f7c02000640 1 -- 172.21.15.28:0/4101943562 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bf0031870 con 0x7f7c040c0fe0 2024-02-21T23:17:45.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.945+0000 7f7c02000640 1 -- 172.21.15.28:0/4101943562 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bf0031c40 con 0x7f7c040c0fe0 2024-02-21T23:17:45.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.946+0000 7f7c08c00640 1 -- 172.21.15.28:0/4101943562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c040c0fe0 msgr2=0x7f7c04071350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:45.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.946+0000 7f7c08c00640 1 --2- 172.21.15.28:0/4101943562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c040c0fe0 0x7f7c04071350 secure :-1 s=READY pgs=2746 cs=0 l=1 rev1=1 crypto rx=0x7f7c0406be50 tx=0x7f7bf002b910 comp rx=0 tx=0).stop 2024-02-21T23:17:45.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.946+0000 7f7c08c00640 1 -- 172.21.15.28:0/4101943562 shutdown_connections 2024-02-21T23:17:45.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.946+0000 7f7c08c00640 1 --2- 172.21.15.28:0/4101943562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c040c0fe0 0x7f7c04071350 unknown :-1 s=CLOSED pgs=2746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:45.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.946+0000 7f7c08c00640 1 --2- 172.21.15.28:0/4101943562 >> v2:172.21.15.81:3300/0 conn(0x7f7c0406dff0 0x7f7c0406e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:45.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.946+0000 7f7c08c00640 1 -- 172.21.15.28:0/4101943562 >> 172.21.15.28:0/4101943562 conn(0x7f7c0406b630 msgr2=0x7f7c040c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:45.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.946+0000 7f7c08c00640 1 -- 172.21.15.28:0/4101943562 shutdown_connections 2024-02-21T23:17:45.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.946+0000 7f7c08c00640 1 -- 172.21.15.28:0/4101943562 wait complete. 2024-02-21T23:17:45.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.949+0000 7f7c08c00640 1 Processor -- start 2024-02-21T23:17:45.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.949+0000 7f7c08c00640 1 -- start start 2024-02-21T23:17:45.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.950+0000 7f7c08c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7c0406dff0 0x7f7c04147270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:45.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.950+0000 7f7c08c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c040c0fe0 0x7f7c040ab290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:45.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.950+0000 7f7c08c00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7c04158580 0x7f7c040ab7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:45.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.950+0000 7f7c08c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c04071890 con 0x7f7c040c0fe0 2024-02-21T23:17:45.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.950+0000 7f7c08c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c04073520 con 0x7f7c0406dff0 2024-02-21T23:17:45.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.950+0000 7f7c08c00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c04158980 con 0x7f7c04158580 2024-02-21T23:17:45.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.950+0000 7f7c02a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c040c0fe0 0x7f7c040ab290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:45.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.950+0000 7f7c02a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c040c0fe0 0x7f7c040ab290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35186/0 (socket says 172.21.15.28:35186) 2024-02-21T23:17:45.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.950+0000 7f7c02a00640 1 -- 172.21.15.28:0/1029060658 learned_addr learned my addr 172.21.15.28:0/1029060658 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:45.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.950+0000 7f7c03400640 1 --2- 172.21.15.28:0/1029060658 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7c0406dff0 0x7f7c04147270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:45.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.950+0000 7f7c02a00640 1 -- 172.21.15.28:0/1029060658 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7c04158580 msgr2=0x7f7c040ab7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:45.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.950+0000 7f7c03e00640 1 --2- 172.21.15.28:0/1029060658 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7c04158580 0x7f7c040ab7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:45.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.951+0000 7f7c02a00640 1 --2- 172.21.15.28:0/1029060658 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7c04158580 0x7f7c040ab7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:45.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.951+0000 7f7c02a00640 1 -- 172.21.15.28:0/1029060658 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7c0406dff0 msgr2=0x7f7c04147270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:45.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.951+0000 7f7c02a00640 1 --2- 172.21.15.28:0/1029060658 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7c0406dff0 0x7f7c04147270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:45.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.951+0000 7f7c02a00640 1 -- 172.21.15.28:0/1029060658 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c04148430 con 0x7f7c040c0fe0 2024-02-21T23:17:45.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.951+0000 7f7c03400640 1 --2- 172.21.15.28:0/1029060658 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7c0406dff0 0x7f7c04147270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:17:45.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.951+0000 7f7c02a00640 1 --2- 172.21.15.28:0/1029060658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c040c0fe0 0x7f7c040ab290 secure :-1 s=READY pgs=2747 cs=0 l=1 rev1=1 crypto rx=0x7f7bf002b950 tx=0x7f7bf00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:45.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.951+0000 7f7bfb400640 1 -- 172.21.15.28:0/1029060658 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bf0040070 con 0x7f7c040c0fe0 2024-02-21T23:17:45.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.951+0000 7f7bfb400640 1 -- 172.21.15.28:0/1029060658 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bf003a420 con 0x7f7c040c0fe0 2024-02-21T23:17:45.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.951+0000 7f7bfb400640 1 -- 172.21.15.28:0/1029060658 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bf0035040 con 0x7f7c040c0fe0 2024-02-21T23:17:45.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.951+0000 7f7c08c00640 1 -- 172.21.15.28:0/1029060658 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c041eef50 con 0x7f7c040c0fe0 2024-02-21T23:17:45.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.951+0000 7f7c08c00640 1 -- 172.21.15.28:0/1029060658 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c041489d0 con 0x7f7c040c0fe0 2024-02-21T23:17:45.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.954+0000 7f7c08c00640 1 -- 172.21.15.28:0/1029060658 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c0404f290 con 0x7f7c040c0fe0 2024-02-21T23:17:45.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.955+0000 7f7bfb400640 1 -- 172.21.15.28:0/1029060658 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7bf0038020 con 0x7f7c040c0fe0 2024-02-21T23:17:45.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.956+0000 7f7bfb400640 1 --2- 172.21.15.28:0/1029060658 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7be406b4d0 0x7f7be406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:45.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.956+0000 7f7bfb400640 1 -- 172.21.15.28:0/1029060658 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f7bf00ac200 con 0x7f7c040c0fe0 2024-02-21T23:17:45.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.956+0000 7f7c03400640 1 --2- 172.21.15.28:0/1029060658 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7be406b4d0 0x7f7be406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:45.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.957+0000 7f7c03400640 1 --2- 172.21.15.28:0/1029060658 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7be406b4d0 0x7f7be406d990 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f7bf400b820 tx=0x7f7bf4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:45.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:45.961+0000 7f7bfb400640 1 -- 172.21.15.28:0/1029060658 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f7bf007b0e0 con 0x7f7c040c0fe0 2024-02-21T23:17:46.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:46.235+0000 7f7c08c00640 1 -- 172.21.15.28:0/1029060658 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterResize.test_down_twice"]} v 0) v1 -- 0x7f7c0406ecb0 con 0x7f7c040c0fe0 2024-02-21T23:17:47.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:47.230+0000 7f7bfb400640 1 -- 172.21.15.28:0/1029060658 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_failover.TestClusterResize.test_down_twice"]}]=0 v812) v1 ==== 139+0+0 (secure 0 0 0) 0x7f7bf007a9e0 con 0x7f7c040c0fe0 2024-02-21T23:17:47.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:47.233+0000 7f7c08c00640 1 -- 172.21.15.28:0/1029060658 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7be406b4d0 msgr2=0x7f7be406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:47.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:47.233+0000 7f7c08c00640 1 --2- 172.21.15.28:0/1029060658 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7be406b4d0 0x7f7be406d990 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f7bf400b820 tx=0x7f7bf4008040 comp rx=0 tx=0).stop 2024-02-21T23:17:47.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:47.233+0000 7f7c08c00640 1 -- 172.21.15.28:0/1029060658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c040c0fe0 msgr2=0x7f7c040ab290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:47.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:47.233+0000 7f7c08c00640 1 --2- 172.21.15.28:0/1029060658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c040c0fe0 0x7f7c040ab290 secure :-1 s=READY pgs=2747 cs=0 l=1 rev1=1 crypto rx=0x7f7bf002b950 tx=0x7f7bf00046c0 comp rx=0 tx=0).stop 2024-02-21T23:17:47.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:47.234+0000 7f7c08c00640 1 -- 172.21.15.28:0/1029060658 shutdown_connections 2024-02-21T23:17:47.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:47.234+0000 7f7c08c00640 1 --2- 172.21.15.28:0/1029060658 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7be406b4d0 0x7f7be406d990 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:47.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:47.234+0000 7f7c08c00640 1 --2- 172.21.15.28:0/1029060658 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7c04158580 0x7f7c040ab7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:47.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:47.234+0000 7f7c08c00640 1 --2- 172.21.15.28:0/1029060658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c040c0fe0 0x7f7c040ab290 unknown :-1 s=CLOSED pgs=2747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:47.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:47.234+0000 7f7c08c00640 1 --2- 172.21.15.28:0/1029060658 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7c0406dff0 0x7f7c04147270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:47.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:47.234+0000 7f7c08c00640 1 -- 172.21.15.28:0/1029060658 >> 172.21.15.28:0/1029060658 conn(0x7f7c0406b630 msgr2=0x7f7c04151380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:47.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:47.234+0000 7f7c08c00640 1 -- 172.21.15.28:0/1029060658 shutdown_connections 2024-02-21T23:17:47.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:47.234+0000 7f7c08c00640 1 -- 172.21.15.28:0/1029060658 wait complete. 2024-02-21T23:17:47.243 INFO:tasks.cephfs_test_runner:That marking a FS down twice does not wipe old_max_mds. ... ERROR 2024-02-21T23:17:47.244 INFO:tasks.cephfs_test_runner: 2024-02-21T23:17:47.244 INFO:tasks.cephfs_test_runner:====================================================================== 2024-02-21T23:17:47.244 INFO:tasks.cephfs_test_runner:ERROR: test_down_twice (tasks.cephfs.test_failover.TestClusterResize) 2024-02-21T23:17:47.244 INFO:tasks.cephfs_test_runner:That marking a FS down twice does not wipe old_max_mds. 2024-02-21T23:17:47.244 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-02-21T23:17:47.244 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-02-21T23:17:47.244 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_3bfc837d7b45ce8e095d92fa9c342964cea5e95f/qa/tasks/cephfs/test_failover.py", line 274, in test_down_twice 2024-02-21T23:17:47.244 INFO:tasks.cephfs_test_runner: self.fs.wait_for_daemons() 2024-02-21T23:17:47.244 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_3bfc837d7b45ce8e095d92fa9c342964cea5e95f/qa/tasks/cephfs/filesystem.py", line 1240, in wait_for_daemons 2024-02-21T23:17:47.244 INFO:tasks.cephfs_test_runner: raise RuntimeError("Timed out waiting for MDS daemons to become healthy") 2024-02-21T23:17:47.244 INFO:tasks.cephfs_test_runner:RuntimeError: Timed out waiting for MDS daemons to become healthy 2024-02-21T23:17:47.244 INFO:tasks.cephfs_test_runner: 2024-02-21T23:17:47.244 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-02-21T23:17:47.244 INFO:tasks.cephfs_test_runner:Ran 12 tests in 847.833s 2024-02-21T23:17:47.245 INFO:tasks.cephfs_test_runner: 2024-02-21T23:17:47.245 INFO:tasks.cephfs_test_runner:FAILED (errors=1) 2024-02-21T23:17:47.245 INFO:tasks.cephfs_test_runner: 2024-02-21T23:17:47.245 INFO:tasks.cephfs_test_runner:====================================================================== 2024-02-21T23:17:47.245 INFO:tasks.cephfs_test_runner:ERROR: test_down_twice (tasks.cephfs.test_failover.TestClusterResize) 2024-02-21T23:17:47.245 INFO:tasks.cephfs_test_runner:That marking a FS down twice does not wipe old_max_mds. 2024-02-21T23:17:47.245 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-02-21T23:17:47.245 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-02-21T23:17:47.245 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_3bfc837d7b45ce8e095d92fa9c342964cea5e95f/qa/tasks/cephfs/test_failover.py", line 274, in test_down_twice 2024-02-21T23:17:47.245 INFO:tasks.cephfs_test_runner: self.fs.wait_for_daemons() 2024-02-21T23:17:47.245 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_3bfc837d7b45ce8e095d92fa9c342964cea5e95f/qa/tasks/cephfs/filesystem.py", line 1240, in wait_for_daemons 2024-02-21T23:17:47.245 INFO:tasks.cephfs_test_runner: raise RuntimeError("Timed out waiting for MDS daemons to become healthy") 2024-02-21T23:17:47.245 INFO:tasks.cephfs_test_runner:RuntimeError: Timed out waiting for MDS daemons to become healthy 2024-02-21T23:17:47.245 INFO:tasks.cephfs_test_runner: 2024-02-21T23:17:47.246 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_3bfc837d7b45ce8e095d92fa9c342964cea5e95f/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_down_twice (tasks.cephfs.test_failover.TestClusterResize) 2024-02-21T23:17:47.466 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=e380bb7cc8b84d30afe72a3769c040e0 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_3bfc837d7b45ce8e095d92fa9c342964cea5e95f/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_down_twice (tasks.cephfs.test_failover.TestClusterResize) 2024-02-21T23:17:47.468 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-02-21T23:17:47.477 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-02-21T23:17:47.488 INFO:tasks.kclient:Unmounting kernel clients... 2024-02-21T23:17:47.488 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-21T23:17:47.489 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-21T23:17:47.552 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054608,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-02-21T23:17:47.552 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-02-21T23:17:47.552 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-21T23:17:47.609 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053833,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498812k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249404k,nr_inodes=812351,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498812k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 2024-02-21T23:17:47.610 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:17:47.610 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-02-21T23:17:47.671 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-02-21T23:17:47.672 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:17:47.672 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-02-21T23:17:47.773 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:17:47.774 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link delete ceph-brx 2024-02-21T23:17:47.906 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:17:47.907 DEBUG:teuthology.orchestra.run.smithi081:> ip netns list 2024-02-21T23:17:47.962 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-02-21T23:17:47.962 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:17:47.963 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-02-21T23:17:48.052 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-21T23:17:48.052 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip link delete ceph-brx 2024-02-21T23:17:48.157 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-02-21T23:17:48.166 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-02-21T23:17:48.166 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-21T23:17:48.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.403+0000 7f8a2c600640 1 Processor -- start 2024-02-21T23:17:48.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.404+0000 7f8a2c600640 1 -- start start 2024-02-21T23:17:48.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.405+0000 7f8a2c600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8a240c11f0 0x7f8a24153950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:48.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.405+0000 7f8a2c600640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f8a240c15d0 0x7f8a2406dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:48.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.405+0000 7f8a2c600640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a24152e70 con 0x7f8a24156350 2024-02-21T23:17:48.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.405+0000 7f8a2c600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a240c1a60 con 0x7f8a240c11f0 2024-02-21T23:17:48.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.405+0000 7f8a2c600640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a240a5ec0 con 0x7f8a240c15d0 2024-02-21T23:17:48.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.405+0000 7f8a29c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8a240c11f0 0x7f8a24153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:48.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.406+0000 7f8a29c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8a240c11f0 0x7f8a24153950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35202/0 (socket says 172.21.15.28:35202) 2024-02-21T23:17:48.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.406+0000 7f8a29c00640 1 -- 172.21.15.28:0/1284838488 learned_addr learned my addr 172.21.15.28:0/1284838488 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:48.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.406+0000 7f8a29200640 1 --2- 172.21.15.28:0/1284838488 >> v2:172.21.15.81:3300/0 conn(0x7f8a240c15d0 0x7f8a2406dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:48.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.406+0000 7f8a29c00640 1 -- 172.21.15.28:0/1284838488 >> v2:172.21.15.81:3300/0 conn(0x7f8a240c15d0 msgr2=0x7f8a2406dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:48.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.406+0000 7f8a29c00640 1 --2- 172.21.15.28:0/1284838488 >> v2:172.21.15.81:3300/0 conn(0x7f8a240c15d0 0x7f8a2406dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:48.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.406+0000 7f8a29c00640 1 -- 172.21.15.28:0/1284838488 >> v1:172.21.15.81:6789/0 conn(0x7f8a24156350 legacy=0x7f8a24156760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:48.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.406+0000 7f8a29c00640 1 -- 172.21.15.28:0/1284838488 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a24153150 con 0x7f8a240c11f0 2024-02-21T23:17:48.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.408+0000 7f8a29c00640 1 --2- 172.21.15.28:0/1284838488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a240c11f0 0x7f8a24153950 secure :-1 s=READY pgs=2748 cs=0 l=1 rev1=1 crypto rx=0x7f8a240c24a0 tx=0x7f8a1802b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4627142646c38e2 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:48.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.408+0000 7f8a23e00640 1 -- 172.21.15.28:0/1284838488 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a18038070 con 0x7f8a240c11f0 2024-02-21T23:17:48.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.408+0000 7f8a23e00640 1 -- 172.21.15.28:0/1284838488 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a1802f050 con 0x7f8a240c11f0 2024-02-21T23:17:48.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.408+0000 7f8a23e00640 1 -- 172.21.15.28:0/1284838488 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a18031970 con 0x7f8a240c11f0 2024-02-21T23:17:48.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.409+0000 7f8a2c600640 1 -- 172.21.15.28:0/1284838488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a240c11f0 msgr2=0x7f8a24153950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:48.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.409+0000 7f8a2c600640 1 --2- 172.21.15.28:0/1284838488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a240c11f0 0x7f8a24153950 secure :-1 s=READY pgs=2748 cs=0 l=1 rev1=1 crypto rx=0x7f8a240c24a0 tx=0x7f8a1802b7f0 comp rx=0 tx=0).stop 2024-02-21T23:17:48.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.409+0000 7f8a2c600640 1 -- 172.21.15.28:0/1284838488 shutdown_connections 2024-02-21T23:17:48.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.409+0000 7f8a2c600640 1 --2- 172.21.15.28:0/1284838488 >> v2:172.21.15.81:3300/0 conn(0x7f8a240c15d0 0x7f8a2406dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:48.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.409+0000 7f8a2c600640 1 --2- 172.21.15.28:0/1284838488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a240c11f0 0x7f8a24153950 unknown :-1 s=CLOSED pgs=2748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:48.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.409+0000 7f8a2c600640 1 -- 172.21.15.28:0/1284838488 >> 172.21.15.28:0/1284838488 conn(0x7f8a240c67f0 msgr2=0x7f8a240c6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:48.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.409+0000 7f8a2c600640 1 -- 172.21.15.28:0/1284838488 shutdown_connections 2024-02-21T23:17:48.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.409+0000 7f8a2c600640 1 -- 172.21.15.28:0/1284838488 wait complete. 2024-02-21T23:17:48.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.411+0000 7f8a2c600640 1 Processor -- start 2024-02-21T23:17:48.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.412+0000 7f8a2c600640 1 -- start start 2024-02-21T23:17:48.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.413+0000 7f8a2c600640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8a240c11f0 0x7f8a2413ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:48.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.413+0000 7f8a2c600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8a240c15d0 0x7f8a240a76d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:48.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.413+0000 7f8a2c600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a24156350 0x7f8a240a6660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:48.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.413+0000 7f8a2c600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a240c4c00 con 0x7f8a24156350 2024-02-21T23:17:48.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.413+0000 7f8a2c600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a2414d1c0 con 0x7f8a240c15d0 2024-02-21T23:17:48.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.413+0000 7f8a2c600640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a240a6110 con 0x7f8a240c11f0 2024-02-21T23:17:48.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.413+0000 7f8a2a600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a24156350 0x7f8a240a6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:48.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.413+0000 7f8a2a600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a24156350 0x7f8a240a6660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35206/0 (socket says 172.21.15.28:35206) 2024-02-21T23:17:48.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.413+0000 7f8a2a600640 1 -- 172.21.15.28:0/343999830 learned_addr learned my addr 172.21.15.28:0/343999830 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:48.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.414+0000 7f8a2a600640 1 -- 172.21.15.28:0/343999830 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8a240c11f0 msgr2=0x7f8a2413ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:48.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.414+0000 7f8a2a600640 1 --2- 172.21.15.28:0/343999830 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8a240c11f0 0x7f8a2413ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:48.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.414+0000 7f8a2a600640 1 -- 172.21.15.28:0/343999830 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8a240c15d0 msgr2=0x7f8a240a76d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:48.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.414+0000 7f8a2a600640 1 --2- 172.21.15.28:0/343999830 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8a240c15d0 0x7f8a240a76d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:48.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.414+0000 7f8a2a600640 1 -- 172.21.15.28:0/343999830 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a24150780 con 0x7f8a24156350 2024-02-21T23:17:48.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.414+0000 7f8a2a600640 1 --2- 172.21.15.28:0/343999830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a24156350 0x7f8a240a6660 secure :-1 s=READY pgs=2749 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c00c7b0 tx=0x7f8a1c00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:48.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.414+0000 7f8a21600640 1 -- 172.21.15.28:0/343999830 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a1c004b60 con 0x7f8a24156350 2024-02-21T23:17:48.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.414+0000 7f8a21600640 1 -- 172.21.15.28:0/343999830 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a1c004d00 con 0x7f8a24156350 2024-02-21T23:17:48.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.414+0000 7f8a21600640 1 -- 172.21.15.28:0/343999830 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a1c00aa90 con 0x7f8a24156350 2024-02-21T23:17:48.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.414+0000 7f8a2c600640 1 -- 172.21.15.28:0/343999830 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a241f3580 con 0x7f8a24156350 2024-02-21T23:17:48.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.415+0000 7f8a2c600640 1 -- 172.21.15.28:0/343999830 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a241f2ce0 con 0x7f8a24156350 2024-02-21T23:17:48.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.416+0000 7f8a2c600640 1 -- 172.21.15.28:0/343999830 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a24067f30 con 0x7f8a24156350 2024-02-21T23:17:48.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.417+0000 7f8a21600640 1 -- 172.21.15.28:0/343999830 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8a1c00ac30 con 0x7f8a24156350 2024-02-21T23:17:48.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.418+0000 7f8a21600640 1 --2- 172.21.15.28:0/343999830 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8a0406b500 0x7f8a0406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:48.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.418+0000 7f8a21600640 1 -- 172.21.15.28:0/343999830 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f8a1c08a480 con 0x7f8a24156350 2024-02-21T23:17:48.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.418+0000 7f8a29c00640 1 --2- 172.21.15.28:0/343999830 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8a0406b500 0x7f8a0406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:48.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.420+0000 7f8a29c00640 1 --2- 172.21.15.28:0/343999830 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8a0406b500 0x7f8a0406d9c0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f8a18035000 tx=0x7f8a18030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:48.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.423+0000 7f8a21600640 1 -- 172.21.15.28:0/343999830 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f8a1c020070 con 0x7f8a24156350 2024-02-21T23:17:48.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.618+0000 7f8a2c600640 1 -- 172.21.15.28:0/343999830 --> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8a240a5c30 con 0x7f8a0406b500 2024-02-21T23:17:48.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.620+0000 7f8a21600640 1 -- 172.21.15.28:0/343999830 <== mgr.4101 v2:172.21.15.81:6832/1848538577 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13593 (secure 0 0 0) 0x7f8a240a5c30 con 0x7f8a0406b500 2024-02-21T23:17:48.607 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:17:48.608 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-21T23:17:48.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.623+0000 7f8a2c600640 1 -- 172.21.15.28:0/343999830 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8a0406b500 msgr2=0x7f8a0406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:48.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.623+0000 7f8a2c600640 1 --2- 172.21.15.28:0/343999830 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8a0406b500 0x7f8a0406d9c0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f8a18035000 tx=0x7f8a18030000 comp rx=0 tx=0).stop 2024-02-21T23:17:48.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.623+0000 7f8a2c600640 1 -- 172.21.15.28:0/343999830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a24156350 msgr2=0x7f8a240a6660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:48.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.623+0000 7f8a2c600640 1 --2- 172.21.15.28:0/343999830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a24156350 0x7f8a240a6660 secure :-1 s=READY pgs=2749 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c00c7b0 tx=0x7f8a1c00cc70 comp rx=0 tx=0).stop 2024-02-21T23:17:48.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.623+0000 7f8a2c600640 1 -- 172.21.15.28:0/343999830 shutdown_connections 2024-02-21T23:17:48.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.623+0000 7f8a2c600640 1 --2- 172.21.15.28:0/343999830 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f8a0406b500 0x7f8a0406d9c0 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:48.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.623+0000 7f8a2c600640 1 --2- 172.21.15.28:0/343999830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a24156350 0x7f8a240a6660 unknown :-1 s=CLOSED pgs=2749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:48.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.623+0000 7f8a2c600640 1 --2- 172.21.15.28:0/343999830 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8a240c15d0 0x7f8a240a76d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:48.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.623+0000 7f8a2c600640 1 --2- 172.21.15.28:0/343999830 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f8a240c11f0 0x7f8a2413ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:48.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.623+0000 7f8a2c600640 1 -- 172.21.15.28:0/343999830 >> 172.21.15.28:0/343999830 conn(0x7f8a240c67f0 msgr2=0x7f8a24151400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:48.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.623+0000 7f8a2c600640 1 -- 172.21.15.28:0/343999830 shutdown_connections 2024-02-21T23:17:48.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.623+0000 7f8a2c600640 1 -- 172.21.15.28:0/343999830 wait complete. 2024-02-21T23:17:48.630 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":666,"stamp":"2024-02-21T23:17:46.999122+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":103,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":287560,"kb_used_data":21456,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754687160,"statfs":{"total":773094113280,"available":772799651840,"internally_reserved":0,"allocated":21970944,"data_stored":11876986,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":272354865},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-3816,"num_objects":-41,"num_object_clones":0,"num_object_copies":-82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-41,"num_whiteouts":0,"num_read":-55,"num_read_kb":-61,"num_write":-85,"num_write_kb":-26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002124"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":535,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-02-21T23:17:45.612378+0000","last_change":"2024-02-21T23:02:32.289335+0000","last_active":"2024-02-21T23:17:45.612378+0000","last_peered":"2024-02-21T23:17:45.612378+0000","last_clean":"2024-02-21T23:17:45.612378+0000","last_became_active":"2024-02-21T23:02:32.286913+0000","last_became_peered":"2024-02-21T23:02:32.286913+0000","last_unstale":"2024-02-21T23:17:45.612378+0000","last_undegraded":"2024-02-21T23:17:45.612378+0000","last_fullsized":"2024-02-21T23:17:45.612378+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_clean_scrub_stamp":"2024-02-21T23:02:31.172569+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:36:23.987374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575033,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33900,"kb_used_data":2720,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337940,"statfs":{"total":96636764160,"available":96602050560,"internally_reserved":0,"allocated":2785280,"data_stored":1923483,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575034,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33856,"kb_used_data":2672,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337984,"statfs":{"total":96636764160,"available":96602095616,"internally_reserved":0,"allocated":2736128,"data_stored":1339381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575033,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33836,"kb_used_data":2656,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338004,"statfs":{"total":96636764160,"available":96602116096,"internally_reserved":0,"allocated":2719744,"data_stored":1333115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575034,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33856,"kb_used_data":2672,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337984,"statfs":{"total":96636764160,"available":96602095616,"internally_reserved":0,"allocated":2736128,"data_stored":1339381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575034,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38016,"kb_used_data":2672,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333824,"statfs":{"total":96636764160,"available":96597835776,"internally_reserved":0,"allocated":2736128,"data_stored":1339381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575035,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38016,"kb_used_data":2672,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333824,"statfs":{"total":96636764160,"available":96597835776,"internally_reserved":0,"allocated":2736128,"data_stored":1339381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575033,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38016,"kb_used_data":2672,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333824,"statfs":{"total":96636764160,"available":96597835776,"internally_reserved":0,"allocated":2736128,"data_stored":1339381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575033,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38064,"kb_used_data":2720,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333776,"statfs":{"total":96636764160,"available":96597786624,"internally_reserved":0,"allocated":2785280,"data_stored":1923483,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-21T23:17:48.631 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-21T23:17:48.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.856+0000 7f6052400640 1 Processor -- start 2024-02-21T23:17:48.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.856+0000 7f6052400640 1 -- start start 2024-02-21T23:17:48.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.857+0000 7f6052400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f604c150450 0x7f604c0a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:48.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.858+0000 7f6052400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f604c06ee40 0x7f604c1536f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:48.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.858+0000 7f6052400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f604c0c2760 con 0x7f604c156af0 2024-02-21T23:17:48.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.858+0000 7f6052400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f604c0c22c0 con 0x7f604c150450 2024-02-21T23:17:48.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.858+0000 7f6052400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f604c156660 con 0x7f604c06ee40 2024-02-21T23:17:48.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.858+0000 7f6051a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f604c156af0 0x7f604c06dff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:54354/0 (socket says 172.21.15.28:54354) 2024-02-21T23:17:48.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.858+0000 7f6051a00640 1 -- 172.21.15.28:0/3532612663 learned_addr learned my addr 172.21.15.28:0/3532612663 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:17:48.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.858+0000 7f604be00640 1 --2- 172.21.15.28:0/3532612663 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f604c06ee40 0x7f604c1536f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:48.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.858+0000 7f6051000640 1 --2- 172.21.15.28:0/3532612663 >> v2:172.21.15.81:3300/0 conn(0x7f604c150450 0x7f604c0a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:48.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.858+0000 7f604b400640 1 -- 172.21.15.28:0/3532612663 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2612981919 0 0) 0x7f604c0c2760 con 0x7f604c156af0 2024-02-21T23:17:48.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.859+0000 7f604b400640 1 -- 172.21.15.28:0/3532612663 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6034003520 con 0x7f604c156af0 2024-02-21T23:17:48.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.859+0000 7f604b400640 1 -- 172.21.15.28:0/3532612663 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2043853634 0 0) 0x7f6034003520 con 0x7f604c156af0 2024-02-21T23:17:48.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.859+0000 7f604b400640 1 -- 172.21.15.28:0/3532612663 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f604c06ee40 msgr2=0x7f604c1536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:48.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.859+0000 7f604b400640 1 --2- 172.21.15.28:0/3532612663 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f604c06ee40 0x7f604c1536f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:48.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.859+0000 7f604b400640 1 -- 172.21.15.28:0/3532612663 >> v2:172.21.15.81:3300/0 conn(0x7f604c150450 msgr2=0x7f604c0a5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:48.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.859+0000 7f604b400640 1 --2- 172.21.15.28:0/3532612663 >> v2:172.21.15.81:3300/0 conn(0x7f604c150450 0x7f604c0a5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:48.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.859+0000 7f604b400640 1 -- 172.21.15.28:0/3532612663 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f604c153c30 con 0x7f604c156af0 2024-02-21T23:17:48.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.859+0000 7f604b400640 1 -- 172.21.15.28:0/3532612663 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7f60440019f0 con 0x7f604c156af0 2024-02-21T23:17:48.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.859+0000 7f604be00640 1 --2- 172.21.15.28:0/3532612663 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f604c06ee40 0x7f604c1536f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-21T23:17:48.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.859+0000 7f604b400640 1 -- 172.21.15.28:0/3532612663 >> v1:172.21.15.28:6789/0 conn(0x7f604c156af0 legacy=0x7f604c06dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:48.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.859+0000 7f604b400640 1 --2- 172.21.15.28:0/3532612663 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f60340039b0 0x7f6034003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:48.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.859+0000 7f604b400640 1 --2- 172.21.15.28:0/3532612663 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f604c06ee40 0x7f6034004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:48.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.859+0000 7f604b400640 1 --2- 172.21.15.28:0/3532612663 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6034004d40 0x7f60340051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:48.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.859+0000 7f604b400640 1 -- 172.21.15.28:0/3532612663 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f604c156660 con 0x7f6034004d40 2024-02-21T23:17:48.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.859+0000 7f604b400640 1 -- 172.21.15.28:0/3532612663 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f603400b9c0 con 0x7f60340039b0 2024-02-21T23:17:48.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.859+0000 7f604b400640 1 -- 172.21.15.28:0/3532612663 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f603400bb40 con 0x7f604c06ee40 2024-02-21T23:17:48.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.860+0000 7f6051a00640 1 --2- 172.21.15.28:0/3532612663 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6034004d40 0x7f60340051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:48.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.860+0000 7f6051000640 1 --2- 172.21.15.28:0/3532612663 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f60340039b0 0x7f6034003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:48.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.860+0000 7f604be00640 1 --2- 172.21.15.28:0/3532612663 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f604c06ee40 0x7f6034004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:48.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.860+0000 7f6051a00640 1 -- 172.21.15.28:0/3532612663 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f604c06ee40 msgr2=0x7f6034004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:48.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.860+0000 7f6051a00640 1 --2- 172.21.15.28:0/3532612663 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f604c06ee40 0x7f6034004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:48.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.860+0000 7f6051a00640 1 -- 172.21.15.28:0/3532612663 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f60340039b0 msgr2=0x7f6034003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:48.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.860+0000 7f6051a00640 1 --2- 172.21.15.28:0/3532612663 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f60340039b0 0x7f6034003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:48.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.860+0000 7f6051a00640 1 -- 172.21.15.28:0/3532612663 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6034003520 con 0x7f6034004d40 2024-02-21T23:17:48.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.861+0000 7f6051a00640 1 --2- 172.21.15.28:0/3532612663 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6034004d40 0x7f60340051c0 secure :-1 s=READY pgs=2751 cs=0 l=1 rev1=1 crypto rx=0x7f604400a920 tx=0x7f6034005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16694a87db937bf4 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:48.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.862+0000 7f604b400640 1 -- 172.21.15.28:0/3532612663 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f604400a4c0 con 0x7f6034004d40 2024-02-21T23:17:48.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.862+0000 7f604b400640 1 -- 172.21.15.28:0/3532612663 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6044034650 con 0x7f6034004d40 2024-02-21T23:17:48.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.862+0000 7f604b400640 1 -- 172.21.15.28:0/3532612663 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6044034a40 con 0x7f6034004d40 2024-02-21T23:17:48.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.862+0000 7f6052400640 1 -- 172.21.15.28:0/3532612663 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6034004d40 msgr2=0x7f60340051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:48.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.862+0000 7f6052400640 1 --2- 172.21.15.28:0/3532612663 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6034004d40 0x7f60340051c0 secure :-1 s=READY pgs=2751 cs=0 l=1 rev1=1 crypto rx=0x7f604400a920 tx=0x7f6034005830 comp rx=0 tx=0).stop 2024-02-21T23:17:48.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.862+0000 7f6051a00640 1 -- 172.21.15.28:0/3532612663 reap_dead start 2024-02-21T23:17:48.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.862+0000 7f6052400640 1 -- 172.21.15.28:0/3532612663 shutdown_connections 2024-02-21T23:17:48.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.862+0000 7f6052400640 1 -- 172.21.15.28:0/3532612663 >> 172.21.15.28:0/3532612663 conn(0x7f604c06b630 msgr2=0x7f604c0c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:48.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.862+0000 7f6052400640 1 -- 172.21.15.28:0/3532612663 shutdown_connections 2024-02-21T23:17:48.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.862+0000 7f6052400640 1 -- 172.21.15.28:0/3532612663 wait complete. 2024-02-21T23:17:48.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.864+0000 7f6052400640 1 Processor -- start 2024-02-21T23:17:48.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.864+0000 7f6052400640 1 -- start start 2024-02-21T23:17:48.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.865+0000 7f6052400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6034004d40 0x7f604c0c54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:48.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.865+0000 7f6052400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f604c1ea3b0 0x7f604c1e7650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:48.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.865+0000 7f6052400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f604c14a6f0 0x7f604c147490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:48.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.865+0000 7f6052400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f604c0c2cb0 con 0x7f604c14a6f0 2024-02-21T23:17:48.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.865+0000 7f6052400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f604c0c29d0 con 0x7f6034004d40 2024-02-21T23:17:48.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.865+0000 7f6052400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f604c06e700 con 0x7f604c1ea3b0 2024-02-21T23:17:48.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.865+0000 7f6051a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f604c14a6f0 0x7f604c147490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:48.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.866+0000 7f6051a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f604c14a6f0 0x7f604c147490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35226/0 (socket says 172.21.15.28:35226) 2024-02-21T23:17:48.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.866+0000 7f6051a00640 1 -- 172.21.15.28:0/961596016 learned_addr learned my addr 172.21.15.28:0/961596016 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:48.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.866+0000 7f6051000640 1 --2- 172.21.15.28:0/961596016 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6034004d40 0x7f604c0c54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:48.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.866+0000 7f604be00640 1 --2- 172.21.15.28:0/961596016 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f604c1ea3b0 0x7f604c1e7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:48.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.866+0000 7f6051a00640 1 -- 172.21.15.28:0/961596016 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f604c1ea3b0 msgr2=0x7f604c1e7650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:48.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.866+0000 7f6051a00640 1 --2- 172.21.15.28:0/961596016 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f604c1ea3b0 0x7f604c1e7650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:48.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.866+0000 7f6051a00640 1 -- 172.21.15.28:0/961596016 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6034004d40 msgr2=0x7f604c0c54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:48.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.866+0000 7f6051a00640 1 --2- 172.21.15.28:0/961596016 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6034004d40 0x7f604c0c54e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:48.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.866+0000 7f6051a00640 1 -- 172.21.15.28:0/961596016 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f604c1e6f40 con 0x7f604c14a6f0 2024-02-21T23:17:48.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.866+0000 7f6051a00640 1 --2- 172.21.15.28:0/961596016 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f604c14a6f0 0x7f604c147490 secure :-1 s=READY pgs=2752 cs=0 l=1 rev1=1 crypto rx=0x7f604402ccb0 tx=0x7f604c070ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:48.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.866+0000 7f6048c00640 1 -- 172.21.15.28:0/961596016 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6044045070 con 0x7f604c14a6f0 2024-02-21T23:17:48.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.867+0000 7f6048c00640 1 -- 172.21.15.28:0/961596016 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f604403a040 con 0x7f604c14a6f0 2024-02-21T23:17:48.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.867+0000 7f6048c00640 1 -- 172.21.15.28:0/961596016 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6044002000 con 0x7f604c14a6f0 2024-02-21T23:17:48.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.867+0000 7f6052400640 1 -- 172.21.15.28:0/961596016 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f604c1e98d0 con 0x7f604c14a6f0 2024-02-21T23:17:48.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.867+0000 7f6052400640 1 -- 172.21.15.28:0/961596016 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f604c0c5a20 con 0x7f604c14a6f0 2024-02-21T23:17:48.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.868+0000 7f6052400640 1 -- 172.21.15.28:0/961596016 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f604c04f290 con 0x7f604c14a6f0 2024-02-21T23:17:48.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.869+0000 7f6048c00640 1 -- 172.21.15.28:0/961596016 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6044048070 con 0x7f604c14a6f0 2024-02-21T23:17:48.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.870+0000 7f6048c00640 1 --2- 172.21.15.28:0/961596016 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f602c06b4a0 0x7f602c06d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:48.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.870+0000 7f6048c00640 1 -- 172.21.15.28:0/961596016 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f60440abc20 con 0x7f604c14a6f0 2024-02-21T23:17:48.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.871+0000 7f6051000640 1 --2- 172.21.15.28:0/961596016 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f602c06b4a0 0x7f602c06d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:48.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.872+0000 7f6051000640 1 --2- 172.21.15.28:0/961596016 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f602c06b4a0 0x7f602c06d960 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f6040009a60 tx=0x7f6040008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:48.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:48.875+0000 7f6048c00640 1 -- 172.21.15.28:0/961596016 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f604407a420 con 0x7f604c14a6f0 2024-02-21T23:17:49.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.072+0000 7f6052400640 1 -- 172.21.15.28:0/961596016 --> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f604c06be20 con 0x7f602c06b4a0 2024-02-21T23:17:49.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.075+0000 7f6048c00640 1 -- 172.21.15.28:0/961596016 <== mgr.4101 v2:172.21.15.81:6832/1848538577 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13593 (secure 0 0 0) 0x7f604c06be20 con 0x7f602c06b4a0 2024-02-21T23:17:49.063 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:17:49.064 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-21T23:17:49.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.078+0000 7f6052400640 1 -- 172.21.15.28:0/961596016 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f602c06b4a0 msgr2=0x7f602c06d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:49.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.078+0000 7f6052400640 1 --2- 172.21.15.28:0/961596016 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f602c06b4a0 0x7f602c06d960 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f6040009a60 tx=0x7f6040008040 comp rx=0 tx=0).stop 2024-02-21T23:17:49.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.078+0000 7f6052400640 1 -- 172.21.15.28:0/961596016 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f604c14a6f0 msgr2=0x7f604c147490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:49.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.078+0000 7f6052400640 1 --2- 172.21.15.28:0/961596016 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f604c14a6f0 0x7f604c147490 secure :-1 s=READY pgs=2752 cs=0 l=1 rev1=1 crypto rx=0x7f604402ccb0 tx=0x7f604c070ec0 comp rx=0 tx=0).stop 2024-02-21T23:17:49.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.078+0000 7f6052400640 1 -- 172.21.15.28:0/961596016 shutdown_connections 2024-02-21T23:17:49.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.078+0000 7f6052400640 1 --2- 172.21.15.28:0/961596016 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f602c06b4a0 0x7f602c06d960 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:49.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.078+0000 7f6052400640 1 --2- 172.21.15.28:0/961596016 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f604c1ea3b0 0x7f604c1e7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:49.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.078+0000 7f6052400640 1 --2- 172.21.15.28:0/961596016 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f604c14a6f0 0x7f604c147490 unknown :-1 s=CLOSED pgs=2752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:49.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.078+0000 7f6052400640 1 --2- 172.21.15.28:0/961596016 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6034004d40 0x7f604c0c54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:49.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.078+0000 7f6052400640 1 -- 172.21.15.28:0/961596016 >> 172.21.15.28:0/961596016 conn(0x7f604c06b630 msgr2=0x7f604c153440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:49.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.078+0000 7f6052400640 1 -- 172.21.15.28:0/961596016 shutdown_connections 2024-02-21T23:17:49.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.078+0000 7f6052400640 1 -- 172.21.15.28:0/961596016 wait complete. 2024-02-21T23:17:49.085 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":667,"stamp":"2024-02-21T23:17:48.999714+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":103,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":287560,"kb_used_data":21456,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754687160,"statfs":{"total":773094113280,"available":772799651840,"internally_reserved":0,"allocated":21970944,"data_stored":11876986,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":272354865},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-3816,"num_objects":-41,"num_object_clones":0,"num_object_copies":-82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-41,"num_whiteouts":0,"num_read":-55,"num_read_kb":-61,"num_write":-85,"num_write_kb":-26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.114810"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":535,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-02-21T23:17:45.612378+0000","last_change":"2024-02-21T23:02:32.289335+0000","last_active":"2024-02-21T23:17:45.612378+0000","last_peered":"2024-02-21T23:17:45.612378+0000","last_clean":"2024-02-21T23:17:45.612378+0000","last_became_active":"2024-02-21T23:02:32.286913+0000","last_became_peered":"2024-02-21T23:02:32.286913+0000","last_unstale":"2024-02-21T23:17:45.612378+0000","last_undegraded":"2024-02-21T23:17:45.612378+0000","last_fullsized":"2024-02-21T23:17:45.612378+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_clean_scrub_stamp":"2024-02-21T23:02:31.172569+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:36:23.987374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575033,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33900,"kb_used_data":2720,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337940,"statfs":{"total":96636764160,"available":96602050560,"internally_reserved":0,"allocated":2785280,"data_stored":1923483,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575034,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33856,"kb_used_data":2672,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337984,"statfs":{"total":96636764160,"available":96602095616,"internally_reserved":0,"allocated":2736128,"data_stored":1339381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575033,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33836,"kb_used_data":2656,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338004,"statfs":{"total":96636764160,"available":96602116096,"internally_reserved":0,"allocated":2719744,"data_stored":1333115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575034,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33856,"kb_used_data":2672,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337984,"statfs":{"total":96636764160,"available":96602095616,"internally_reserved":0,"allocated":2736128,"data_stored":1339381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575034,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38016,"kb_used_data":2672,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333824,"statfs":{"total":96636764160,"available":96597835776,"internally_reserved":0,"allocated":2736128,"data_stored":1339381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575035,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38016,"kb_used_data":2672,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333824,"statfs":{"total":96636764160,"available":96597835776,"internally_reserved":0,"allocated":2736128,"data_stored":1339381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575033,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38016,"kb_used_data":2672,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333824,"statfs":{"total":96636764160,"available":96597835776,"internally_reserved":0,"allocated":2736128,"data_stored":1339381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575033,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38064,"kb_used_data":2720,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333776,"statfs":{"total":96636764160,"available":96597786624,"internally_reserved":0,"allocated":2785280,"data_stored":1923483,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-21T23:17:49.086 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-02-21T23:17:49.086 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-21T23:17:49.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.314+0000 7fa3f6800640 1 Processor -- start 2024-02-21T23:17:49.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.314+0000 7fa3f6800640 1 -- start start 2024-02-21T23:17:49.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.315+0000 7fa3f6800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa3f006ee40 0x7fa3f00a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:49.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.316+0000 7fa3f6800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3f00c2760 con 0x7fa3f0156af0 2024-02-21T23:17:49.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.316+0000 7fa3f6800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3f0156310 con 0x7fa3f0150450 2024-02-21T23:17:49.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.316+0000 7fa3f6800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3f00c22c0 con 0x7fa3f006ee40 2024-02-21T23:17:49.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.316+0000 7fa3f4a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa3f006ee40 0x7fa3f00a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:49.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.316+0000 7fa3f5e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa3f0156af0 0x7fa3f006dff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:54366/0 (socket says 172.21.15.28:54366) 2024-02-21T23:17:49.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.316+0000 7fa3f5e00640 1 -- 172.21.15.28:0/831791169 learned_addr learned my addr 172.21.15.28:0/831791169 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:17:49.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.316+0000 7fa3f4a00640 1 -- 172.21.15.28:0/831791169 >> v1:172.21.15.81:6789/0 conn(0x7fa3f0150450 legacy=0x7fa3f01536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:49.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.316+0000 7fa3f4a00640 1 -- 172.21.15.28:0/831791169 >> v1:172.21.15.28:6789/0 conn(0x7fa3f0156af0 legacy=0x7fa3f006dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:49.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.316+0000 7fa3f4a00640 1 -- 172.21.15.28:0/831791169 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3f00c29d0 con 0x7fa3f006ee40 2024-02-21T23:17:49.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.318+0000 7fa3f4a00640 1 --2- 172.21.15.28:0/831791169 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3f006ee40 0x7fa3f00a5fd0 secure :-1 s=READY pgs=2754 cs=0 l=1 rev1=1 crypto rx=0x7fa3dc0097a0 tx=0x7fa3dc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f619bb64ce58a8c5 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:49.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.318+0000 7fa3efe00640 1 -- 172.21.15.28:0/831791169 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3dc035020 con 0x7fa3f006ee40 2024-02-21T23:17:49.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.318+0000 7fa3efe00640 1 -- 172.21.15.28:0/831791169 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3dc031830 con 0x7fa3f006ee40 2024-02-21T23:17:49.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.318+0000 7fa3efe00640 1 -- 172.21.15.28:0/831791169 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3dc031c00 con 0x7fa3f006ee40 2024-02-21T23:17:49.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.319+0000 7fa3f6800640 1 -- 172.21.15.28:0/831791169 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3f006ee40 msgr2=0x7fa3f00a5fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:49.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.319+0000 7fa3f6800640 1 --2- 172.21.15.28:0/831791169 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3f006ee40 0x7fa3f00a5fd0 secure :-1 s=READY pgs=2754 cs=0 l=1 rev1=1 crypto rx=0x7fa3dc0097a0 tx=0x7fa3dc02b8d0 comp rx=0 tx=0).stop 2024-02-21T23:17:49.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.319+0000 7fa3f6800640 1 -- 172.21.15.28:0/831791169 shutdown_connections 2024-02-21T23:17:49.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.319+0000 7fa3f6800640 1 --2- 172.21.15.28:0/831791169 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3f006ee40 0x7fa3f00a5fd0 unknown :-1 s=CLOSED pgs=2754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:49.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.319+0000 7fa3f6800640 1 -- 172.21.15.28:0/831791169 >> 172.21.15.28:0/831791169 conn(0x7fa3f006b630 msgr2=0x7fa3f00c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:49.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.319+0000 7fa3f6800640 1 -- 172.21.15.28:0/831791169 shutdown_connections 2024-02-21T23:17:49.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.319+0000 7fa3f6800640 1 -- 172.21.15.28:0/831791169 wait complete. 2024-02-21T23:17:49.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.320+0000 7fa3f6800640 1 Processor -- start 2024-02-21T23:17:49.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.321+0000 7fa3f6800640 1 -- start start 2024-02-21T23:17:49.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.321+0000 7fa3f6800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3f006ee40 0x7fa3f014c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:49.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.321+0000 7fa3f6800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa3f0150450 0x7fa3f01e9f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:49.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.321+0000 7fa3f6800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa3f0156af0 0x7fa3f01ea450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:49.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.321+0000 7fa3f6800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f006e700 con 0x7fa3f006ee40 2024-02-21T23:17:49.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.321+0000 7fa3f6800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f0156660 con 0x7fa3f0156af0 2024-02-21T23:17:49.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.321+0000 7fa3f6800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f006e880 con 0x7fa3f0150450 2024-02-21T23:17:49.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.322+0000 7fa3f5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3f006ee40 0x7fa3f014c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:49.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.322+0000 7fa3f5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3f006ee40 0x7fa3f014c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35242/0 (socket says 172.21.15.28:35242) 2024-02-21T23:17:49.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.322+0000 7fa3f5400640 1 -- 172.21.15.28:0/3592211497 learned_addr learned my addr 172.21.15.28:0/3592211497 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:49.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.322+0000 7fa3f5e00640 1 --2- 172.21.15.28:0/3592211497 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa3f0156af0 0x7fa3f01ea450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:49.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.322+0000 7fa3f5400640 1 -- 172.21.15.28:0/3592211497 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa3f0150450 msgr2=0x7fa3f01e9f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:49.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.322+0000 7fa3f5400640 1 --2- 172.21.15.28:0/3592211497 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa3f0150450 0x7fa3f01e9f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:49.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.322+0000 7fa3f5400640 1 -- 172.21.15.28:0/3592211497 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa3f0156af0 msgr2=0x7fa3f01ea450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:49.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.322+0000 7fa3f5400640 1 --2- 172.21.15.28:0/3592211497 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa3f0156af0 0x7fa3f01ea450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:49.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.322+0000 7fa3f5400640 1 -- 172.21.15.28:0/3592211497 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3f0144a30 con 0x7fa3f006ee40 2024-02-21T23:17:49.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.323+0000 7fa3f5400640 1 --2- 172.21.15.28:0/3592211497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3f006ee40 0x7fa3f014c9b0 secure :-1 s=READY pgs=2755 cs=0 l=1 rev1=1 crypto rx=0x7fa3e400c430 tx=0x7fa3e400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:49.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.323+0000 7fa3ed600640 1 -- 172.21.15.28:0/3592211497 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e4011790 con 0x7fa3f006ee40 2024-02-21T23:17:49.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.323+0000 7fa3ed600640 1 -- 172.21.15.28:0/3592211497 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3e4017070 con 0x7fa3f006ee40 2024-02-21T23:17:49.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.323+0000 7fa3ed600640 1 -- 172.21.15.28:0/3592211497 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e4011bb0 con 0x7fa3f006ee40 2024-02-21T23:17:49.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.323+0000 7fa3f6800640 1 -- 172.21.15.28:0/3592211497 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3f0142aa0 con 0x7fa3f006ee40 2024-02-21T23:17:49.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.323+0000 7fa3f6800640 1 -- 172.21.15.28:0/3592211497 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3f0144f10 con 0x7fa3f006ee40 2024-02-21T23:17:49.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.324+0000 7fa3f6800640 1 -- 172.21.15.28:0/3592211497 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3f01ea990 con 0x7fa3f006ee40 2024-02-21T23:17:49.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.325+0000 7fa3ed600640 1 -- 172.21.15.28:0/3592211497 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa3e401c460 con 0x7fa3f006ee40 2024-02-21T23:17:49.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.326+0000 7fa3ed600640 1 --2- 172.21.15.28:0/3592211497 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa3d006b520 0x7fa3d006d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:49.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.326+0000 7fa3ed600640 1 -- 172.21.15.28:0/3592211497 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 5593+0+0 (secure 0 0 0) 0x7fa3e4089ff0 con 0x7fa3f006ee40 2024-02-21T23:17:49.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.327+0000 7fa3f4a00640 1 --2- 172.21.15.28:0/3592211497 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa3d006b520 0x7fa3d006d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:49.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.328+0000 7fa3f4a00640 1 --2- 172.21.15.28:0/3592211497 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa3d006b520 0x7fa3d006d9e0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7fa3dc036000 tx=0x7fa3dc030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:49.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.331+0000 7fa3ed600640 1 -- 172.21.15.28:0/3592211497 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fa3f01ea990 con 0x7fa3f006ee40 2024-02-21T23:17:49.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.527+0000 7fa3f6800640 1 -- 172.21.15.28:0/3592211497 --> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa3f0148e20 con 0x7fa3d006b520 2024-02-21T23:17:49.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.529+0000 7fa3ed600640 1 -- 172.21.15.28:0/3592211497 <== mgr.4101 v2:172.21.15.81:6832/1848538577 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13593 (secure 0 0 0) 0x7fa3f0148e20 con 0x7fa3d006b520 2024-02-21T23:17:49.517 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:17:49.517 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-21T23:17:49.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.532+0000 7fa3f6800640 1 -- 172.21.15.28:0/3592211497 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa3d006b520 msgr2=0x7fa3d006d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:49.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.532+0000 7fa3f6800640 1 --2- 172.21.15.28:0/3592211497 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa3d006b520 0x7fa3d006d9e0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7fa3dc036000 tx=0x7fa3dc030000 comp rx=0 tx=0).stop 2024-02-21T23:17:49.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.532+0000 7fa3f6800640 1 -- 172.21.15.28:0/3592211497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3f006ee40 msgr2=0x7fa3f014c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:49.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.532+0000 7fa3f6800640 1 --2- 172.21.15.28:0/3592211497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3f006ee40 0x7fa3f014c9b0 secure :-1 s=READY pgs=2755 cs=0 l=1 rev1=1 crypto rx=0x7fa3e400c430 tx=0x7fa3e400c8f0 comp rx=0 tx=0).stop 2024-02-21T23:17:49.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.532+0000 7fa3f6800640 1 -- 172.21.15.28:0/3592211497 shutdown_connections 2024-02-21T23:17:49.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.532+0000 7fa3f6800640 1 --2- 172.21.15.28:0/3592211497 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa3d006b520 0x7fa3d006d9e0 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:49.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.532+0000 7fa3f6800640 1 --2- 172.21.15.28:0/3592211497 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa3f0150450 0x7fa3f01e9f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:49.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.532+0000 7fa3f6800640 1 --2- 172.21.15.28:0/3592211497 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa3f0156af0 0x7fa3f01ea450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:49.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.532+0000 7fa3f6800640 1 --2- 172.21.15.28:0/3592211497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3f006ee40 0x7fa3f014c9b0 unknown :-1 s=CLOSED pgs=2755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:49.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.532+0000 7fa3f6800640 1 -- 172.21.15.28:0/3592211497 >> 172.21.15.28:0/3592211497 conn(0x7fa3f006b630 msgr2=0x7fa3f0143330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:49.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.532+0000 7fa3f6800640 1 -- 172.21.15.28:0/3592211497 shutdown_connections 2024-02-21T23:17:49.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.532+0000 7fa3f6800640 1 -- 172.21.15.28:0/3592211497 wait complete. 2024-02-21T23:17:49.539 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":667,"stamp":"2024-02-21T23:17:48.999714+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":103,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":287560,"kb_used_data":21456,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754687160,"statfs":{"total":773094113280,"available":772799651840,"internally_reserved":0,"allocated":21970944,"data_stored":11876986,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":272354865},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-3816,"num_objects":-41,"num_object_clones":0,"num_object_copies":-82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-41,"num_whiteouts":0,"num_read":-55,"num_read_kb":-61,"num_write":-85,"num_write_kb":-26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.114810"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":535,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-02-21T23:17:45.612378+0000","last_change":"2024-02-21T23:02:32.289335+0000","last_active":"2024-02-21T23:17:45.612378+0000","last_peered":"2024-02-21T23:17:45.612378+0000","last_clean":"2024-02-21T23:17:45.612378+0000","last_became_active":"2024-02-21T23:02:32.286913+0000","last_became_peered":"2024-02-21T23:02:32.286913+0000","last_unstale":"2024-02-21T23:17:45.612378+0000","last_undegraded":"2024-02-21T23:17:45.612378+0000","last_fullsized":"2024-02-21T23:17:45.612378+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_clean_scrub_stamp":"2024-02-21T23:02:31.172569+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:36:23.987374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575033,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33900,"kb_used_data":2720,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337940,"statfs":{"total":96636764160,"available":96602050560,"internally_reserved":0,"allocated":2785280,"data_stored":1923483,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575034,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33856,"kb_used_data":2672,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337984,"statfs":{"total":96636764160,"available":96602095616,"internally_reserved":0,"allocated":2736128,"data_stored":1339381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575033,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33836,"kb_used_data":2656,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338004,"statfs":{"total":96636764160,"available":96602116096,"internally_reserved":0,"allocated":2719744,"data_stored":1333115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575034,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33856,"kb_used_data":2672,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337984,"statfs":{"total":96636764160,"available":96602095616,"internally_reserved":0,"allocated":2736128,"data_stored":1339381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575034,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38016,"kb_used_data":2672,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333824,"statfs":{"total":96636764160,"available":96597835776,"internally_reserved":0,"allocated":2736128,"data_stored":1339381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575035,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38016,"kb_used_data":2672,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333824,"statfs":{"total":96636764160,"available":96597835776,"internally_reserved":0,"allocated":2736128,"data_stored":1339381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575033,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38016,"kb_used_data":2672,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333824,"statfs":{"total":96636764160,"available":96597835776,"internally_reserved":0,"allocated":2736128,"data_stored":1339381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575033,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38064,"kb_used_data":2720,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333776,"statfs":{"total":96636764160,"available":96597786624,"internally_reserved":0,"allocated":2785280,"data_stored":1923483,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-21T23:17:49.540 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-21T23:17:49.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.775+0000 7ff3cb200640 1 Processor -- start 2024-02-21T23:17:49.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.776+0000 7ff3cb200640 1 -- start start 2024-02-21T23:17:49.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.777+0000 7ff3cb200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7ff3cc0aff90 0x7ff3cc0c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:49.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.777+0000 7ff3cb200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3cc0c74c0 con 0x7ff3cc0c5720 2024-02-21T23:17:49.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.777+0000 7ff3cb200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3cc0c77a0 con 0x7ff3cc06dbf0 2024-02-21T23:17:49.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.777+0000 7ff3cb200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3cc0c7270 con 0x7ff3cc0aff90 2024-02-21T23:17:49.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.777+0000 7ff3ca800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff3cc0c5720 0x7ff3cc15f890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:54376/0 (socket says 172.21.15.28:54376) 2024-02-21T23:17:49.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.777+0000 7ff3ca800640 1 -- 172.21.15.28:0/2288996731 learned_addr learned my addr 172.21.15.28:0/2288996731 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:17:49.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.778+0000 7ff3c9e00640 1 --2- 172.21.15.28:0/2288996731 >> v2:172.21.15.81:3300/0 conn(0x7ff3cc0aff90 0x7ff3cc0c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:49.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.778+0000 7ff3c8a00640 1 -- 172.21.15.28:0/2288996731 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 430844306 0 0) 0x7ff3cc0c74c0 con 0x7ff3cc0c5720 2024-02-21T23:17:49.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.778+0000 7ff3c8a00640 1 -- 172.21.15.28:0/2288996731 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3b0003520 con 0x7ff3cc0c5720 2024-02-21T23:17:49.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.778+0000 7ff3c8a00640 1 -- 172.21.15.28:0/2288996731 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4289775491 0 0) 0x7ff3b0003520 con 0x7ff3cc0c5720 2024-02-21T23:17:49.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.778+0000 7ff3c8a00640 1 -- 172.21.15.28:0/2288996731 >> v2:172.21.15.81:3300/0 conn(0x7ff3cc0aff90 msgr2=0x7ff3cc0c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:49.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.778+0000 7ff3c8a00640 1 --2- 172.21.15.28:0/2288996731 >> v2:172.21.15.81:3300/0 conn(0x7ff3cc0aff90 0x7ff3cc0c51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:49.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.778+0000 7ff3c8a00640 1 -- 172.21.15.28:0/2288996731 >> v1:172.21.15.81:6789/0 conn(0x7ff3cc06dbf0 legacy=0x7ff3cc158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:49.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.778+0000 7ff3c8a00640 1 -- 172.21.15.28:0/2288996731 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3cc0c7a80 con 0x7ff3cc0c5720 2024-02-21T23:17:49.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.778+0000 7ff3c8a00640 1 -- 172.21.15.28:0/2288996731 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7ff3c40019f0 con 0x7ff3cc0c5720 2024-02-21T23:17:49.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.778+0000 7ff3c8a00640 1 -- 172.21.15.28:0/2288996731 >> v1:172.21.15.28:6789/0 conn(0x7ff3cc0c5720 legacy=0x7ff3cc15f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:49.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.778+0000 7ff3c8a00640 1 --2- 172.21.15.28:0/2288996731 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3b0003a00 0x7ff3b0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:49.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.778+0000 7ff3c8a00640 1 --2- 172.21.15.28:0/2288996731 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff3b0004800 0x7ff3b0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:49.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.778+0000 7ff3c8a00640 1 --2- 172.21.15.28:0/2288996731 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3b00051c0 0x7ff3b000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:49.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.778+0000 7ff3c8a00640 1 -- 172.21.15.28:0/2288996731 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3cc0c7270 con 0x7ff3b0003a00 2024-02-21T23:17:49.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.778+0000 7ff3c8a00640 1 -- 172.21.15.28:0/2288996731 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b000c0c0 con 0x7ff3b00051c0 2024-02-21T23:17:49.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.778+0000 7ff3c8a00640 1 -- 172.21.15.28:0/2288996731 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b000c2c0 con 0x7ff3b0004800 2024-02-21T23:17:49.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.778+0000 7ff3c9e00640 1 --2- 172.21.15.28:0/2288996731 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3b0003a00 0x7ff3b0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:49.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.779+0000 7ff3c9e00640 1 -- 172.21.15.28:0/2288996731 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff3b0004800 msgr2=0x7ff3b0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:49.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.779+0000 7ff3c9e00640 1 --2- 172.21.15.28:0/2288996731 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff3b0004800 0x7ff3b0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:49.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.779+0000 7ff3c9e00640 1 -- 172.21.15.28:0/2288996731 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3b00051c0 msgr2=0x7ff3b000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:49.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.779+0000 7ff3c9e00640 1 --2- 172.21.15.28:0/2288996731 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3b00051c0 0x7ff3b000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:49.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.779+0000 7ff3c9e00640 1 -- 172.21.15.28:0/2288996731 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3b0003520 con 0x7ff3b0003a00 2024-02-21T23:17:49.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.779+0000 7ff3ca800640 1 -- 172.21.15.28:0/2288996731 reap_dead start 2024-02-21T23:17:49.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.780+0000 7ff3c9e00640 1 --2- 172.21.15.28:0/2288996731 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3b0003a00 0x7ff3b0003e20 secure :-1 s=READY pgs=2757 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc009ac0 tx=0x7ff3b0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a780f38167fb33b server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:49.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.780+0000 7ff3c8a00640 1 -- 172.21.15.28:0/2288996731 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3bc039070 con 0x7ff3b0003a00 2024-02-21T23:17:49.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.780+0000 7ff3c8a00640 1 -- 172.21.15.28:0/2288996731 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3bc02c6d0 con 0x7ff3b0003a00 2024-02-21T23:17:49.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.780+0000 7ff3c8a00640 1 -- 172.21.15.28:0/2288996731 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3bc009690 con 0x7ff3b0003a00 2024-02-21T23:17:49.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.781+0000 7ff3cb200640 1 -- 172.21.15.28:0/2288996731 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3b0003a00 msgr2=0x7ff3b0003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:49.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.781+0000 7ff3cb200640 1 --2- 172.21.15.28:0/2288996731 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3b0003a00 0x7ff3b0003e20 secure :-1 s=READY pgs=2757 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc009ac0 tx=0x7ff3b0005830 comp rx=0 tx=0).stop 2024-02-21T23:17:49.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.781+0000 7ff3cb200640 1 -- 172.21.15.28:0/2288996731 shutdown_connections 2024-02-21T23:17:49.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.781+0000 7ff3cb200640 1 --2- 172.21.15.28:0/2288996731 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3b0003a00 0x7ff3b0003e20 unknown :-1 s=CLOSED pgs=2757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:49.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.781+0000 7ff3cb200640 1 -- 172.21.15.28:0/2288996731 >> 172.21.15.28:0/2288996731 conn(0x7ff3cc0ac370 msgr2=0x7ff3cc0ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:49.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.781+0000 7ff3cb200640 1 -- 172.21.15.28:0/2288996731 shutdown_connections 2024-02-21T23:17:49.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.781+0000 7ff3cb200640 1 -- 172.21.15.28:0/2288996731 wait complete. 2024-02-21T23:17:49.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.783+0000 7ff3cb200640 1 Processor -- start 2024-02-21T23:17:49.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.783+0000 7ff3cb200640 1 -- start start 2024-02-21T23:17:49.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.784+0000 7ff3cb200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3b0003a00 0x7ff3cc0aab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:49.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.784+0000 7ff3cb200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3cc1eb480 0x7ff3cc1edd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:49.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.784+0000 7ff3cb200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff3cc1ee2c0 0x7ff3cc151be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:49.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.784+0000 7ff3cb200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3cc0c8760 con 0x7ff3b0003a00 2024-02-21T23:17:49.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.784+0000 7ff3cb200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3cc158630 con 0x7ff3cc1eb480 2024-02-21T23:17:49.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.784+0000 7ff3cb200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3cc0c88e0 con 0x7ff3cc1ee2c0 2024-02-21T23:17:49.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.784+0000 7ff3c9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3b0003a00 0x7ff3cc0aab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:49.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.784+0000 7ff3c9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3b0003a00 0x7ff3cc0aab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35262/0 (socket says 172.21.15.28:35262) 2024-02-21T23:17:49.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.784+0000 7ff3c9e00640 1 -- 172.21.15.28:0/2342873703 learned_addr learned my addr 172.21.15.28:0/2342873703 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:49.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.785+0000 7ff3ca800640 1 --2- 172.21.15.28:0/2342873703 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff3cc1ee2c0 0x7ff3cc151be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:49.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.785+0000 7ff3c9e00640 1 -- 172.21.15.28:0/2342873703 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff3cc1ee2c0 msgr2=0x7ff3cc151be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:49.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.785+0000 7ff3c9400640 1 --2- 172.21.15.28:0/2342873703 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3cc1eb480 0x7ff3cc1edd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:49.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.785+0000 7ff3c9e00640 1 --2- 172.21.15.28:0/2342873703 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff3cc1ee2c0 0x7ff3cc151be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:49.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.785+0000 7ff3c9e00640 1 -- 172.21.15.28:0/2342873703 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3cc1eb480 msgr2=0x7ff3cc1edd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:49.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.785+0000 7ff3c9e00640 1 --2- 172.21.15.28:0/2342873703 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3cc1eb480 0x7ff3cc1edd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:49.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.785+0000 7ff3c9e00640 1 -- 172.21.15.28:0/2342873703 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3cc0c3ce0 con 0x7ff3b0003a00 2024-02-21T23:17:49.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.785+0000 7ff3c9400640 1 --2- 172.21.15.28:0/2342873703 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3cc1eb480 0x7ff3cc1edd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:17:49.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.785+0000 7ff3c9e00640 1 --2- 172.21.15.28:0/2342873703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3b0003a00 0x7ff3cc0aab40 secure :-1 s=READY pgs=2758 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc037040 tx=0x7ff3bc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:49.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.785+0000 7ff3c2000640 1 -- 172.21.15.28:0/2342873703 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3bc039050 con 0x7ff3b0003a00 2024-02-21T23:17:49.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.785+0000 7ff3c2000640 1 -- 172.21.15.28:0/2342873703 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3bc045070 con 0x7ff3b0003a00 2024-02-21T23:17:49.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.785+0000 7ff3cb200640 1 -- 172.21.15.28:0/2342873703 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3cc1ea9a0 con 0x7ff3b0003a00 2024-02-21T23:17:49.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.785+0000 7ff3c2000640 1 -- 172.21.15.28:0/2342873703 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3bc003a70 con 0x7ff3b0003a00 2024-02-21T23:17:49.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.786+0000 7ff3cb200640 1 -- 172.21.15.28:0/2342873703 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3cc0c4a20 con 0x7ff3b0003a00 2024-02-21T23:17:49.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.787+0000 7ff3cb200640 1 -- 172.21.15.28:0/2342873703 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3cc1ea790 con 0x7ff3b0003a00 2024-02-21T23:17:49.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.788+0000 7ff3c2000640 1 -- 172.21.15.28:0/2342873703 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff3bc003c10 con 0x7ff3b0003a00 2024-02-21T23:17:49.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.788+0000 7ff3c2000640 1 --2- 172.21.15.28:0/2342873703 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff3a806b4d0 0x7ff3a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:49.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.789+0000 7ff3c2000640 1 -- 172.21.15.28:0/2342873703 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 5593+0+0 (secure 0 0 0) 0x7ff3bc0abc80 con 0x7ff3b0003a00 2024-02-21T23:17:49.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.789+0000 7ff3c9400640 1 --2- 172.21.15.28:0/2342873703 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff3a806b4d0 0x7ff3a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:49.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.790+0000 7ff3c9400640 1 --2- 172.21.15.28:0/2342873703 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff3a806b4d0 0x7ff3a806d990 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7ff3cc0c1700 tx=0x7ff3b4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:49.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.793+0000 7ff3c2000640 1 -- 172.21.15.28:0/2342873703 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7ff3bc04e020 con 0x7ff3b0003a00 2024-02-21T23:17:49.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.994+0000 7ff3cb200640 1 -- 172.21.15.28:0/2342873703 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff3cc04f290 con 0x7ff3b0003a00 2024-02-21T23:17:49.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.996+0000 7ff3c2000640 1 -- 172.21.15.28:0/2342873703 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v215) v1 ==== 74+0+13633 (secure 0 0 0) 0x7ff3bc07ab60 con 0x7ff3b0003a00 2024-02-21T23:17:49.983 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:17:49.983 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":215,"fsid":"d8a1daf7-6c0d-46c2-b6f1-4a4413a6e5ac","created":"2024-02-21T23:02:17.606995+0000","modified":"2024-02-21T23:17:45.213430+0000","last_up_change":"2024-02-21T23:02:30.169021+0000","last_in_change":"2024-02-21T23:02:21.651243+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":34,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-21T23:02:30.728329+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f6ca1d98-5d2a-413d-870c-d7b9da1ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6801","nonce":2069258563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6803","nonce":2069258563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6807","nonce":2069258563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2069258563},{"type":"v1","addr":"172.21.15.28:6805","nonce":2069258563}]},"public_addr":"172.21.15.28:6801/2069258563","cluster_addr":"172.21.15.28:6803/2069258563","heartbeat_back_addr":"172.21.15.28:6807/2069258563","heartbeat_front_addr":"172.21.15.28:6805/2069258563","state":["exists","up"]},{"osd":1,"uuid":"2a25dd0c-4c28-4a86-a506-f195f981de12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6810","nonce":2193666734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6811","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6813","nonce":2193666734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6818","nonce":2193666734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":2193666734},{"type":"v1","addr":"172.21.15.28:6815","nonce":2193666734}]},"public_addr":"172.21.15.28:6810/2193666734","cluster_addr":"172.21.15.28:6813/2193666734","heartbeat_back_addr":"172.21.15.28:6818/2193666734","heartbeat_front_addr":"172.21.15.28:6815/2193666734","state":["exists","up"]},{"osd":2,"uuid":"96493c37-efc1-481f-8e86-972e59ccac3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6809","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6812","nonce":160098884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6817","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6819","nonce":160098884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6823","nonce":160098884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":160098884},{"type":"v1","addr":"172.21.15.28:6821","nonce":160098884}]},"public_addr":"172.21.15.28:6812/160098884","cluster_addr":"172.21.15.28:6819/160098884","heartbeat_back_addr":"172.21.15.28:6823/160098884","heartbeat_front_addr":"172.21.15.28:6821/160098884","state":["exists","up"]},{"osd":3,"uuid":"6654ebd8-3251-43f7-8730-4db82e869c98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6825","nonce":3053508071}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6827","nonce":3053508071}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6831","nonce":3053508071}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":3053508071},{"type":"v1","addr":"172.21.15.28:6829","nonce":3053508071}]},"public_addr":"172.21.15.28:6825/3053508071","cluster_addr":"172.21.15.28:6827/3053508071","heartbeat_back_addr":"172.21.15.28:6831/3053508071","heartbeat_front_addr":"172.21.15.28:6829/3053508071","state":["exists","up"]},{"osd":4,"uuid":"6ac7806e-a5b0-4786-898a-b31607939493","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":210,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6821","nonce":2091269704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6824","nonce":2091269704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6829","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6830","nonce":2091269704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6825","nonce":2091269704},{"type":"v1","addr":"172.21.15.81:6827","nonce":2091269704}]},"public_addr":"172.21.15.81:6821/2091269704","cluster_addr":"172.21.15.81:6824/2091269704","heartbeat_back_addr":"172.21.15.81:6830/2091269704","heartbeat_front_addr":"172.21.15.81:6827/2091269704","state":["exists","up"]},{"osd":5,"uuid":"ca6f7bb1-5003-4a86-b23b-eb6b23e377a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6809","nonce":568898662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6811","nonce":568898662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6815","nonce":568898662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":568898662},{"type":"v1","addr":"172.21.15.81:6813","nonce":568898662}]},"public_addr":"172.21.15.81:6809/568898662","cluster_addr":"172.21.15.81:6811/568898662","heartbeat_back_addr":"172.21.15.81:6815/568898662","heartbeat_front_addr":"172.21.15.81:6813/568898662","state":["exists","up"]},{"osd":6,"uuid":"5231b268-6822-4ba3-8e00-4b374f580251","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6801","nonce":3369511790}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6803","nonce":3369511790}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6807","nonce":3369511790}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":3369511790},{"type":"v1","addr":"172.21.15.81:6805","nonce":3369511790}]},"public_addr":"172.21.15.81:6801/3369511790","cluster_addr":"172.21.15.81:6803/3369511790","heartbeat_back_addr":"172.21.15.81:6807/3369511790","heartbeat_front_addr":"172.21.15.81:6805/3369511790","state":["exists","up"]},{"osd":7,"uuid":"4b2df9db-d659-4a19-a88d-ea4d759259dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6817","nonce":3022012384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6820","nonce":3022012384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6831","nonce":3022012384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":3022012384},{"type":"v1","addr":"172.21.15.81:6826","nonce":3022012384}]},"public_addr":"172.21.15.81:6817/3022012384","cluster_addr":"172.21.15.81:6820/3022012384","heartbeat_back_addr":"172.21.15.81:6831/3022012384","heartbeat_front_addr":"172.21.15.81:6826/3022012384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.226350+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.893979+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.993126+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.959962+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:27.279035+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.900787+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-21T23:02:26.975282+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.81:6836/3779716966":"2024-02-22T23:17:41.880112+0000","172.21.15.28:6834/3142544451":"2024-02-22T23:17:41.880137+0000","172.21.15.81:6837/3779716966":"2024-02-22T23:17:41.880112+0000","172.21.15.81:6835/1607587459":"2024-02-22T23:14:42.637238+0000","172.21.15.28:6833/1864206570":"2024-02-22T23:14:07.624525+0000","172.21.15.28:6835/3142544451":"2024-02-22T23:17:41.880137+0000","172.21.15.81:6834/1607587459":"2024-02-22T23:14:42.637238+0000","172.21.15.28:6832/1864206570":"2024-02-22T23:14:07.624525+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-21T23:17:49.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.998+0000 7ff3cb200640 1 -- 172.21.15.28:0/2342873703 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff3a806b4d0 msgr2=0x7ff3a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:49.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.998+0000 7ff3cb200640 1 --2- 172.21.15.28:0/2342873703 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff3a806b4d0 0x7ff3a806d990 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7ff3cc0c1700 tx=0x7ff3b4009040 comp rx=0 tx=0).stop 2024-02-21T23:17:49.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.999+0000 7ff3cb200640 1 -- 172.21.15.28:0/2342873703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3b0003a00 msgr2=0x7ff3cc0aab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:49.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.999+0000 7ff3cb200640 1 --2- 172.21.15.28:0/2342873703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3b0003a00 0x7ff3cc0aab40 secure :-1 s=READY pgs=2758 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc037040 tx=0x7ff3bc004730 comp rx=0 tx=0).stop 2024-02-21T23:17:49.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.999+0000 7ff3cb200640 1 -- 172.21.15.28:0/2342873703 shutdown_connections 2024-02-21T23:17:49.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.999+0000 7ff3cb200640 1 --2- 172.21.15.28:0/2342873703 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff3a806b4d0 0x7ff3a806d990 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:49.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.999+0000 7ff3cb200640 1 --2- 172.21.15.28:0/2342873703 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff3cc1ee2c0 0x7ff3cc151be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:49.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.999+0000 7ff3cb200640 1 --2- 172.21.15.28:0/2342873703 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff3cc1eb480 0x7ff3cc1edd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:49.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.999+0000 7ff3cb200640 1 --2- 172.21.15.28:0/2342873703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3b0003a00 0x7ff3cc0aab40 unknown :-1 s=CLOSED pgs=2758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:49.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.999+0000 7ff3cb200640 1 -- 172.21.15.28:0/2342873703 >> 172.21.15.28:0/2342873703 conn(0x7ff3cc0ac370 msgr2=0x7ff3cc0ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:49.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.999+0000 7ff3cb200640 1 -- 172.21.15.28:0/2342873703 shutdown_connections 2024-02-21T23:17:49.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:49.999+0000 7ff3cb200640 1 -- 172.21.15.28:0/2342873703 wait complete. 2024-02-21T23:17:51.008 INFO:tasks.ceph:Scrubbing osd.0 2024-02-21T23:17:51.008 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-02-21T23:17:51.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.259+0000 7f21b7200640 1 Processor -- start 2024-02-21T23:17:51.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.259+0000 7f21b7200640 1 -- start start 2024-02-21T23:17:51.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.260+0000 7f21b7200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f21b806cfa0 0x7f21b80ca540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:51.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.260+0000 7f21b7200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f21b8158460 0x7f21b80c6290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:51.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.261+0000 7f21b7200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f21b80c8f70 0x7f21b81522c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:51.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.261+0000 7f21b7200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21b80c8ae0 con 0x7f21b80c8f70 2024-02-21T23:17:51.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.261+0000 7f21b7200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21b80caf40 con 0x7f21b806cfa0 2024-02-21T23:17:51.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.261+0000 7f21b7200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21b81588f0 con 0x7f21b8158460 2024-02-21T23:17:51.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.261+0000 7f21b5e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f21b806cfa0 0x7f21b80ca540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:51.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.261+0000 7f21b6800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f21b80c8f70 0x7f21b81522c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:51.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.261+0000 7f21b5e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f21b806cfa0 0x7f21b80ca540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.28:38896/0 (socket says 172.21.15.28:38896) 2024-02-21T23:17:51.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.261+0000 7f21b5400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f21b8158460 0x7f21b80c6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:51.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.261+0000 7f21b6800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f21b80c8f70 0x7f21b81522c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35266/0 (socket says 172.21.15.28:35266) 2024-02-21T23:17:51.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.261+0000 7f21b5e00640 1 -- 172.21.15.28:0/3436654041 learned_addr learned my addr 172.21.15.28:0/3436654041 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:51.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.262+0000 7f21b5e00640 1 -- 172.21.15.28:0/3436654041 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f21b8158460 msgr2=0x7f21b80c6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:51.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.262+0000 7f21b5e00640 1 --2- 172.21.15.28:0/3436654041 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f21b8158460 0x7f21b80c6290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:51.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.262+0000 7f21b5e00640 1 -- 172.21.15.28:0/3436654041 >> v2:172.21.15.28:3300/0 conn(0x7f21b80c8f70 msgr2=0x7f21b81522c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:51.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.262+0000 7f21b5e00640 1 --2- 172.21.15.28:0/3436654041 >> v2:172.21.15.28:3300/0 conn(0x7f21b80c8f70 0x7f21b81522c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:51.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.262+0000 7f21b5e00640 1 -- 172.21.15.28:0/3436654041 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21b80c87b0 con 0x7f21b806cfa0 2024-02-21T23:17:51.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.262+0000 7f21b6800640 1 --2- 172.21.15.28:0/3436654041 >> v2:172.21.15.28:3300/0 conn(0x7f21b80c8f70 0x7f21b81522c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-21T23:17:51.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.263+0000 7f21b5e00640 1 --2- 172.21.15.28:0/3436654041 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f21b806cfa0 0x7f21b80ca540 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f21b80aa1b0 tx=0x7f21a802b7e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=22498f3d8d914bc server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:51.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.264+0000 7f21b4a00640 1 -- 172.21.15.28:0/3436654041 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a8038070 con 0x7f21b806cfa0 2024-02-21T23:17:51.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.264+0000 7f21b4a00640 1 -- 172.21.15.28:0/3436654041 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21a8031740 con 0x7f21b806cfa0 2024-02-21T23:17:51.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.264+0000 7f21b4a00640 1 -- 172.21.15.28:0/3436654041 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a802f050 con 0x7f21b806cfa0 2024-02-21T23:17:51.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.264+0000 7f21b7200640 1 -- 172.21.15.28:0/3436654041 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f21b806cfa0 msgr2=0x7f21b80ca540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:51.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.264+0000 7f21b7200640 1 --2- 172.21.15.28:0/3436654041 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f21b806cfa0 0x7f21b80ca540 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f21b80aa1b0 tx=0x7f21a802b7e0 comp rx=0 tx=0).stop 2024-02-21T23:17:51.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.265+0000 7f21b7200640 1 -- 172.21.15.28:0/3436654041 shutdown_connections 2024-02-21T23:17:51.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.265+0000 7f21b7200640 1 --2- 172.21.15.28:0/3436654041 >> v2:172.21.15.28:3300/0 conn(0x7f21b80c8f70 0x7f21b81522c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:51.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.265+0000 7f21b7200640 1 --2- 172.21.15.28:0/3436654041 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f21b8158460 0x7f21b80c6290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:51.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.265+0000 7f21b7200640 1 --2- 172.21.15.28:0/3436654041 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f21b806cfa0 0x7f21b80ca540 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:51.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.265+0000 7f21b7200640 1 -- 172.21.15.28:0/3436654041 >> 172.21.15.28:0/3436654041 conn(0x7f21b80725a0 msgr2=0x7f21b80729b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:51.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.265+0000 7f21b7200640 1 -- 172.21.15.28:0/3436654041 shutdown_connections 2024-02-21T23:17:51.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.265+0000 7f21b7200640 1 -- 172.21.15.28:0/3436654041 wait complete. 2024-02-21T23:17:51.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.266+0000 7f21b7200640 1 Processor -- start 2024-02-21T23:17:51.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.266+0000 7f21b7200640 1 -- start start 2024-02-21T23:17:51.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.267+0000 7f21b7200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f21b806cfa0 0x7f21b814ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:51.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.267+0000 7f21b7200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21b80c8f70 0x7f21b81ea570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:51.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.267+0000 7f21b7200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f21b8158460 0x7f21b81497c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:51.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.267+0000 7f21b7200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21b80c93c0 con 0x7f21b80c8f70 2024-02-21T23:17:51.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.267+0000 7f21b7200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21b80c6c90 con 0x7f21b8158460 2024-02-21T23:17:51.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.267+0000 7f21b7200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21b80c9540 con 0x7f21b806cfa0 2024-02-21T23:17:51.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.268+0000 7f21b5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21b80c8f70 0x7f21b81ea570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:51.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.268+0000 7f21b5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21b80c8f70 0x7f21b81ea570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35268/0 (socket says 172.21.15.28:35268) 2024-02-21T23:17:51.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.268+0000 7f21b5400640 1 -- 172.21.15.28:0/1408431378 learned_addr learned my addr 172.21.15.28:0/1408431378 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:51.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.268+0000 7f21b5e00640 1 --2- 172.21.15.28:0/1408431378 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f21b806cfa0 0x7f21b814ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:51.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.268+0000 7f21b6800640 1 --2- 172.21.15.28:0/1408431378 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f21b8158460 0x7f21b81497c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:51.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.268+0000 7f21b5400640 1 -- 172.21.15.28:0/1408431378 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f21b806cfa0 msgr2=0x7f21b814ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:51.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.268+0000 7f21b5400640 1 --2- 172.21.15.28:0/1408431378 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f21b806cfa0 0x7f21b814ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:51.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.268+0000 7f21b5400640 1 -- 172.21.15.28:0/1408431378 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f21b8158460 msgr2=0x7f21b81497c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:51.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.268+0000 7f21b5400640 1 --2- 172.21.15.28:0/1408431378 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f21b8158460 0x7f21b81497c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:51.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.268+0000 7f21b5400640 1 -- 172.21.15.28:0/1408431378 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21b80ac640 con 0x7f21b80c8f70 2024-02-21T23:17:51.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.268+0000 7f21b5e00640 1 --2- 172.21.15.28:0/1408431378 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f21b806cfa0 0x7f21b814ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:17:51.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.268+0000 7f21b5400640 1 --2- 172.21.15.28:0/1408431378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21b80c8f70 0x7f21b81ea570 secure :-1 s=READY pgs=2759 cs=0 l=1 rev1=1 crypto rx=0x7f21a400e9b0 tx=0x7f21a400edb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:51.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.269+0000 7f21ae000640 1 -- 172.21.15.28:0/1408431378 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a4017070 con 0x7f21b80c8f70 2024-02-21T23:17:51.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.269+0000 7f21ae000640 1 -- 172.21.15.28:0/1408431378 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21a400c940 con 0x7f21b80c8f70 2024-02-21T23:17:51.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.269+0000 7f21ae000640 1 -- 172.21.15.28:0/1408431378 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a400cd30 con 0x7f21b80c8f70 2024-02-21T23:17:51.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.269+0000 7f21b7200640 1 -- 172.21.15.28:0/1408431378 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21b814d2a0 con 0x7f21b80c8f70 2024-02-21T23:17:51.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.269+0000 7f21b7200640 1 -- 172.21.15.28:0/1408431378 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21b814a6b0 con 0x7f21b80c8f70 2024-02-21T23:17:51.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.270+0000 7f21b7200640 1 -- 172.21.15.28:0/1408431378 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f21b804f290 con 0x7f21b80c8f70 2024-02-21T23:17:51.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.271+0000 7f21ae000640 1 -- 172.21.15.28:0/1408431378 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f21a401a070 con 0x7f21b80c8f70 2024-02-21T23:17:51.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.272+0000 7f21ae000640 1 --2- 172.21.15.28:0/1408431378 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f219806b500 0x7f219806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:51.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.272+0000 7f21ae000640 1 -- 172.21.15.28:0/1408431378 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f21a408a0c0 con 0x7f21b80c8f70 2024-02-21T23:17:51.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.273+0000 7f21ae000640 1 --2- 172.21.15.28:0/1408431378 >> [v2:172.21.15.28:6800/2069258563,v1:172.21.15.28:6801/2069258563] conn(0x7f2198071b80 0x7f2198073fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:51.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.273+0000 7f21ae000640 1 -- 172.21.15.28:0/1408431378 --> [v2:172.21.15.28:6800/2069258563,v1:172.21.15.28:6801/2069258563] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f21a4015e50 con 0x7f2198071b80 2024-02-21T23:17:51.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.273+0000 7f21ae000640 1 -- 172.21.15.28:0/1408431378 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=215) v2 ==== 24+0+0 (secure 0 0 0) 0x7f21a4091050 con 0x7f21b80c8f70 2024-02-21T23:17:51.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.273+0000 7f21b5e00640 1 --2- 172.21.15.28:0/1408431378 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f219806b500 0x7f219806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:51.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.273+0000 7f21b6800640 1 --2- 172.21.15.28:0/1408431378 >> [v2:172.21.15.28:6800/2069258563,v1:172.21.15.28:6801/2069258563] conn(0x7f2198071b80 0x7f2198073fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:51.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.274+0000 7f21b6800640 1 --2- 172.21.15.28:0/1408431378 >> [v2:172.21.15.28:6800/2069258563,v1:172.21.15.28:6801/2069258563] conn(0x7f2198071b80 0x7f2198073fc0 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:51.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.274+0000 7f21b5e00640 1 --2- 172.21.15.28:0/1408431378 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f219806b500 0x7f219806d9c0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f21a8035000 tx=0x7f21a8008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:51.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.274+0000 7f21ae000640 1 -- 172.21.15.28:0/1408431378 <== osd.0 v2:172.21.15.28:6800/2069258563 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f21a4015e50 con 0x7f2198071b80 2024-02-21T23:17:51.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.289+0000 7f21b7200640 1 -- 172.21.15.28:0/1408431378 --> [v2:172.21.15.28:6800/2069258563,v1:172.21.15.28:6801/2069258563] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f21b8157bf0 con 0x7f2198071b80 2024-02-21T23:17:51.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.289+0000 7f21ae000640 1 -- 172.21.15.28:0/1408431378 <== osd.0 v2:172.21.15.28:6800/2069258563 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f21b8157bf0 con 0x7f2198071b80 2024-02-21T23:17:51.277 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-21T23:17:51.277 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-02-21T23:17:51.277 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-21T23:17:51.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.290+0000 7f21b7200640 1 -- 172.21.15.28:0/1408431378 >> [v2:172.21.15.28:6800/2069258563,v1:172.21.15.28:6801/2069258563] conn(0x7f2198071b80 msgr2=0x7f2198073fc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:51.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.290+0000 7f21b7200640 1 --2- 172.21.15.28:0/1408431378 >> [v2:172.21.15.28:6800/2069258563,v1:172.21.15.28:6801/2069258563] conn(0x7f2198071b80 0x7f2198073fc0 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:51.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.290+0000 7f21b7200640 1 -- 172.21.15.28:0/1408431378 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f219806b500 msgr2=0x7f219806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:51.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.290+0000 7f21b7200640 1 --2- 172.21.15.28:0/1408431378 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f219806b500 0x7f219806d9c0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f21a8035000 tx=0x7f21a8008030 comp rx=0 tx=0).stop 2024-02-21T23:17:51.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.290+0000 7f21b7200640 1 -- 172.21.15.28:0/1408431378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21b80c8f70 msgr2=0x7f21b81ea570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:51.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.290+0000 7f21b7200640 1 --2- 172.21.15.28:0/1408431378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21b80c8f70 0x7f21b81ea570 secure :-1 s=READY pgs=2759 cs=0 l=1 rev1=1 crypto rx=0x7f21a400e9b0 tx=0x7f21a400edb0 comp rx=0 tx=0).stop 2024-02-21T23:17:51.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.290+0000 7f21b6800640 1 -- 172.21.15.28:0/1408431378 reap_dead start 2024-02-21T23:17:51.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.291+0000 7f21b7200640 1 -- 172.21.15.28:0/1408431378 shutdown_connections 2024-02-21T23:17:51.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.291+0000 7f21b7200640 1 -- 172.21.15.28:0/1408431378 >> 172.21.15.28:0/1408431378 conn(0x7f21b80725a0 msgr2=0x7f21b80ada60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:51.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.291+0000 7f21b7200640 1 -- 172.21.15.28:0/1408431378 shutdown_connections 2024-02-21T23:17:51.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.291+0000 7f21b7200640 1 -- 172.21.15.28:0/1408431378 wait complete. 2024-02-21T23:17:51.296 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-02-21T23:17:51.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.523+0000 7fa08be00640 1 Processor -- start 2024-02-21T23:17:51.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.523+0000 7fa08be00640 1 -- start start 2024-02-21T23:17:51.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.525+0000 7fa08be00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa08c14ba10 0x7fa08c0c44b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:51.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.525+0000 7fa08be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa08c0730e0 0x7fa08c06f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:51.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.525+0000 7fa08be00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fa08c0c49f0 0x7fa08c06f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:51.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.525+0000 7fa08be00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa08c1529c0 con 0x7fa08c0730e0 2024-02-21T23:17:51.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.525+0000 7fa08be00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa08c072c60 con 0x7fa08c0c49f0 2024-02-21T23:17:51.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.525+0000 7fa08be00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa08c1526e0 con 0x7fa08c14ba10 2024-02-21T23:17:51.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.526+0000 7fa08aa00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa08c14ba10 0x7fa08c0c44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:51.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.526+0000 7fa08aa00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa08c14ba10 0x7fa08c0c44b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3301/0 says I am v2:172.21.15.28:55050/0 (socket says 172.21.15.28:55050) 2024-02-21T23:17:51.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.526+0000 7fa08aa00640 1 -- 172.21.15.28:0/1389523181 learned_addr learned my addr 172.21.15.28:0/1389523181 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:51.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.526+0000 7fa082000640 1 --2- 172.21.15.28:0/1389523181 >> v2:172.21.15.28:3300/0 conn(0x7fa08c0730e0 0x7fa08c06f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:51.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.526+0000 7fa08b400640 1 --2- 172.21.15.28:0/1389523181 >> v2:172.21.15.81:3300/0 conn(0x7fa08c0c49f0 0x7fa08c06f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:51.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.526+0000 7fa082000640 1 -- 172.21.15.28:0/1389523181 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa08c14ba10 msgr2=0x7fa08c0c44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:51.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.526+0000 7fa082000640 1 --2- 172.21.15.28:0/1389523181 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa08c14ba10 0x7fa08c0c44b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:51.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.526+0000 7fa082000640 1 -- 172.21.15.28:0/1389523181 >> v2:172.21.15.81:3300/0 conn(0x7fa08c0c49f0 msgr2=0x7fa08c06f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:51.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.526+0000 7fa082000640 1 --2- 172.21.15.28:0/1389523181 >> v2:172.21.15.81:3300/0 conn(0x7fa08c0c49f0 0x7fa08c06f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:51.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.526+0000 7fa082000640 1 -- 172.21.15.28:0/1389523181 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa08c0c2150 con 0x7fa08c0730e0 2024-02-21T23:17:51.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.526+0000 7fa08aa00640 1 --2- 172.21.15.28:0/1389523181 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa08c14ba10 0x7fa08c0c44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-21T23:17:51.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.528+0000 7fa082000640 1 --2- 172.21.15.28:0/1389523181 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa08c0730e0 0x7fa08c06f080 secure :-1 s=READY pgs=2760 cs=0 l=1 rev1=1 crypto rx=0x7fa0740097a0 tx=0x7fa07402b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebc84c77a70b0508 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:51.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.528+0000 7fa08a000640 1 -- 172.21.15.28:0/1389523181 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa074039070 con 0x7fa08c0730e0 2024-02-21T23:17:51.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.528+0000 7fa08a000640 1 -- 172.21.15.28:0/1389523181 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0740316f0 con 0x7fa08c0730e0 2024-02-21T23:17:51.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.528+0000 7fa08a000640 1 -- 172.21.15.28:0/1389523181 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa074031ac0 con 0x7fa08c0730e0 2024-02-21T23:17:51.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.528+0000 7fa08be00640 1 -- 172.21.15.28:0/1389523181 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa08c0730e0 msgr2=0x7fa08c06f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:51.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.528+0000 7fa08be00640 1 --2- 172.21.15.28:0/1389523181 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa08c0730e0 0x7fa08c06f080 secure :-1 s=READY pgs=2760 cs=0 l=1 rev1=1 crypto rx=0x7fa0740097a0 tx=0x7fa07402b790 comp rx=0 tx=0).stop 2024-02-21T23:17:51.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.529+0000 7fa08be00640 1 -- 172.21.15.28:0/1389523181 shutdown_connections 2024-02-21T23:17:51.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.529+0000 7fa08be00640 1 --2- 172.21.15.28:0/1389523181 >> v2:172.21.15.81:3300/0 conn(0x7fa08c0c49f0 0x7fa08c06f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:51.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.529+0000 7fa08be00640 1 --2- 172.21.15.28:0/1389523181 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa08c0730e0 0x7fa08c06f080 unknown :-1 s=CLOSED pgs=2760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:51.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.529+0000 7fa08be00640 1 --2- 172.21.15.28:0/1389523181 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa08c14ba10 0x7fa08c0c44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:51.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.529+0000 7fa08be00640 1 -- 172.21.15.28:0/1389523181 >> 172.21.15.28:0/1389523181 conn(0x7fa08c06c160 msgr2=0x7fa08c06c540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:51.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.529+0000 7fa08be00640 1 -- 172.21.15.28:0/1389523181 shutdown_connections 2024-02-21T23:17:51.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.529+0000 7fa08be00640 1 -- 172.21.15.28:0/1389523181 wait complete. 2024-02-21T23:17:51.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.530+0000 7fa08be00640 1 Processor -- start 2024-02-21T23:17:51.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.530+0000 7fa08be00640 1 -- start start 2024-02-21T23:17:51.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.531+0000 7fa08be00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa08c0730e0 0x7fa08c1ea750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:51.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.531+0000 7fa08be00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa08c0c49f0 0x7fa08c13c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:51.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.531+0000 7fa08be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa08c14ba10 0x7fa08c149950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:51.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.531+0000 7fa08be00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa08c14bdf0 con 0x7fa08c14ba10 2024-02-21T23:17:51.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.531+0000 7fa08be00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa08c0c1c60 con 0x7fa08c0730e0 2024-02-21T23:17:51.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.531+0000 7fa08be00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa08c14bf70 con 0x7fa08c0c49f0 2024-02-21T23:17:51.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.532+0000 7fa08b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa08c14ba10 0x7fa08c149950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:51.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.532+0000 7fa08b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa08c14ba10 0x7fa08c149950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35284/0 (socket says 172.21.15.28:35284) 2024-02-21T23:17:51.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.532+0000 7fa08b400640 1 -- 172.21.15.28:0/2788503142 learned_addr learned my addr 172.21.15.28:0/2788503142 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:51.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.532+0000 7fa082000640 1 --2- 172.21.15.28:0/2788503142 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa08c0c49f0 0x7fa08c13c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:51.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.532+0000 7fa08aa00640 1 --2- 172.21.15.28:0/2788503142 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa08c0730e0 0x7fa08c1ea750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:51.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.532+0000 7fa08b400640 1 -- 172.21.15.28:0/2788503142 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa08c0c49f0 msgr2=0x7fa08c13c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:51.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.532+0000 7fa08b400640 1 --2- 172.21.15.28:0/2788503142 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa08c0c49f0 0x7fa08c13c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:51.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.532+0000 7fa08b400640 1 -- 172.21.15.28:0/2788503142 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa08c0730e0 msgr2=0x7fa08c1ea750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:51.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.532+0000 7fa08b400640 1 --2- 172.21.15.28:0/2788503142 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa08c0730e0 0x7fa08c1ea750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:51.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.532+0000 7fa08b400640 1 -- 172.21.15.28:0/2788503142 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa08c147ca0 con 0x7fa08c14ba10 2024-02-21T23:17:51.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.532+0000 7fa08b400640 1 --2- 172.21.15.28:0/2788503142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa08c14ba10 0x7fa08c149950 secure :-1 s=READY pgs=2761 cs=0 l=1 rev1=1 crypto rx=0x7fa08400ef00 tx=0x7fa08400b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:51.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.533+0000 7fa083400640 1 -- 172.21.15.28:0/2788503142 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa084018020 con 0x7fa08c14ba10 2024-02-21T23:17:51.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.533+0000 7fa083400640 1 -- 172.21.15.28:0/2788503142 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa084014400 con 0x7fa08c14ba10 2024-02-21T23:17:51.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.533+0000 7fa083400640 1 -- 172.21.15.28:0/2788503142 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0840147f0 con 0x7fa08c14ba10 2024-02-21T23:17:51.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.533+0000 7fa08be00640 1 -- 172.21.15.28:0/2788503142 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa08c1479c0 con 0x7fa08c14ba10 2024-02-21T23:17:51.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.533+0000 7fa08be00640 1 -- 172.21.15.28:0/2788503142 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa08c1eac90 con 0x7fa08c14ba10 2024-02-21T23:17:51.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.534+0000 7fa08be00640 1 -- 172.21.15.28:0/2788503142 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa08c04f290 con 0x7fa08c14ba10 2024-02-21T23:17:51.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.536+0000 7fa083400640 1 -- 172.21.15.28:0/2788503142 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa084005d10 con 0x7fa08c14ba10 2024-02-21T23:17:51.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.537+0000 7fa083400640 1 --2- 172.21.15.28:0/2788503142 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa06806b500 0x7fa06806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:51.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.537+0000 7fa083400640 1 -- 172.21.15.28:0/2788503142 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 5593+0+0 (secure 0 0 0) 0x7fa08408a800 con 0x7fa08c14ba10 2024-02-21T23:17:51.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.537+0000 7fa08aa00640 1 --2- 172.21.15.28:0/2788503142 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa06806b500 0x7fa06806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:51.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.538+0000 7fa08aa00640 1 --2- 172.21.15.28:0/2788503142 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa06806b500 0x7fa06806d9c0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7fa07c009ac0 tx=0x7fa07c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:51.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.541+0000 7fa083400640 1 -- 172.21.15.28:0/2788503142 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fa084022070 con 0x7fa08c14ba10 2024-02-21T23:17:51.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.746+0000 7fa08be00640 1 -- 172.21.15.28:0/2788503142 --> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7fa08c072ec0 con 0x7fa06806b500 2024-02-21T23:17:51.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.747+0000 7fa083400640 1 -- 172.21.15.28:0/2788503142 <== mgr.4101 v2:172.21.15.81:6832/1848538577 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fa08c072ec0 con 0x7fa06806b500 2024-02-21T23:17:51.734 INFO:teuthology.orchestra.run.smithi028.stderr:instructed osd(s) 0 to deep-scrub 2024-02-21T23:17:51.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.750+0000 7fa08be00640 1 -- 172.21.15.28:0/2788503142 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa06806b500 msgr2=0x7fa06806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:51.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.750+0000 7fa08be00640 1 --2- 172.21.15.28:0/2788503142 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa06806b500 0x7fa06806d9c0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7fa07c009ac0 tx=0x7fa07c008040 comp rx=0 tx=0).stop 2024-02-21T23:17:51.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.750+0000 7fa08be00640 1 -- 172.21.15.28:0/2788503142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa08c14ba10 msgr2=0x7fa08c149950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:51.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.750+0000 7fa08be00640 1 --2- 172.21.15.28:0/2788503142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa08c14ba10 0x7fa08c149950 secure :-1 s=READY pgs=2761 cs=0 l=1 rev1=1 crypto rx=0x7fa08400ef00 tx=0x7fa08400b480 comp rx=0 tx=0).stop 2024-02-21T23:17:51.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.750+0000 7fa08be00640 1 -- 172.21.15.28:0/2788503142 shutdown_connections 2024-02-21T23:17:51.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.750+0000 7fa08be00640 1 --2- 172.21.15.28:0/2788503142 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa06806b500 0x7fa06806d9c0 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:51.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.750+0000 7fa08be00640 1 --2- 172.21.15.28:0/2788503142 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa08c0c49f0 0x7fa08c13c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:51.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.750+0000 7fa08be00640 1 --2- 172.21.15.28:0/2788503142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa08c14ba10 0x7fa08c149950 unknown :-1 s=CLOSED pgs=2761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:51.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.750+0000 7fa08be00640 1 --2- 172.21.15.28:0/2788503142 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa08c0730e0 0x7fa08c1ea750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:51.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.750+0000 7fa08be00640 1 -- 172.21.15.28:0/2788503142 >> 172.21.15.28:0/2788503142 conn(0x7fa08c06c160 msgr2=0x7fa08c06c540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:51.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.750+0000 7fa08be00640 1 -- 172.21.15.28:0/2788503142 shutdown_connections 2024-02-21T23:17:51.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.750+0000 7fa08be00640 1 -- 172.21.15.28:0/2788503142 wait complete. 2024-02-21T23:17:51.758 INFO:tasks.ceph:Scrubbing osd.1 2024-02-21T23:17:51.758 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-02-21T23:17:51.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.997+0000 7fcf53400640 1 Processor -- start 2024-02-21T23:17:51.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.997+0000 7fcf53400640 1 -- start start 2024-02-21T23:17:51.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.999+0000 7fcf53400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcf4c1525c0 0x7fcf4c0cb390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:51.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.999+0000 7fcf53400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fcf4c159010 0x7fcf4c0c6a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:51.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.999+0000 7fcf53400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf4c155120 con 0x7fcf4c0c6f40 2024-02-21T23:17:51.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.999+0000 7fcf53400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf4c155390 con 0x7fcf4c159010 2024-02-21T23:17:51.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:51.999+0000 7fcf53400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf4c155670 con 0x7fcf4c1525c0 2024-02-21T23:17:51.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.000+0000 7fcf50a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcf4c1525c0 0x7fcf4c0cb390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:51.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.000+0000 7fcf4be00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fcf4c159010 0x7fcf4c0c6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:51.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.000+0000 7fcf50a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcf4c1525c0 0x7fcf4c0cb390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3301/0 says I am v2:172.21.15.28:55064/0 (socket says 172.21.15.28:55064) 2024-02-21T23:17:51.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.000+0000 7fcf50a00640 1 -- 172.21.15.28:0/4169051901 learned_addr learned my addr 172.21.15.28:0/4169051901 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:51.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.001+0000 7fcf50a00640 1 -- 172.21.15.28:0/4169051901 >> v2:172.21.15.81:3300/0 conn(0x7fcf4c159010 msgr2=0x7fcf4c0c6a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:51.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.001+0000 7fcf50a00640 1 --2- 172.21.15.28:0/4169051901 >> v2:172.21.15.81:3300/0 conn(0x7fcf4c159010 0x7fcf4c0c6a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:51.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.001+0000 7fcf50a00640 1 -- 172.21.15.28:0/4169051901 >> v1:172.21.15.81:6789/0 conn(0x7fcf4c0c6f40 legacy=0x7fcf4c0c3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:51.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.001+0000 7fcf50a00640 1 -- 172.21.15.28:0/4169051901 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf4c155950 con 0x7fcf4c1525c0 2024-02-21T23:17:51.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.001+0000 7fcf4b400640 1 -- 172.21.15.28:0/4169051901 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3720027052 0 0) 0x7fcf4c155120 con 0x7fcf4c0c6f40 2024-02-21T23:17:51.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.001+0000 7fcf4be00640 1 --2- 172.21.15.28:0/4169051901 >> v2:172.21.15.81:3300/0 conn(0x7fcf4c159010 0x7fcf4c0c6a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-21T23:17:51.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.002+0000 7fcf50a00640 1 --2- 172.21.15.28:0/4169051901 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcf4c1525c0 0x7fcf4c0cb390 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fcf40009870 tx=0x7fcf4002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33461696afd242e5 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:51.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.003+0000 7fcf4b400640 1 -- 172.21.15.28:0/4169051901 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf40034070 con 0x7fcf4c1525c0 2024-02-21T23:17:51.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.003+0000 7fcf4b400640 1 -- 172.21.15.28:0/4169051901 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf40030840 con 0x7fcf4c1525c0 2024-02-21T23:17:51.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.003+0000 7fcf4b400640 1 -- 172.21.15.28:0/4169051901 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf40030c10 con 0x7fcf4c1525c0 2024-02-21T23:17:51.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.004+0000 7fcf53400640 1 -- 172.21.15.28:0/4169051901 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcf4c1525c0 msgr2=0x7fcf4c0cb390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:51.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.004+0000 7fcf53400640 1 --2- 172.21.15.28:0/4169051901 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcf4c1525c0 0x7fcf4c0cb390 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fcf40009870 tx=0x7fcf4002b980 comp rx=0 tx=0).stop 2024-02-21T23:17:51.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.004+0000 7fcf53400640 1 -- 172.21.15.28:0/4169051901 shutdown_connections 2024-02-21T23:17:51.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.004+0000 7fcf53400640 1 --2- 172.21.15.28:0/4169051901 >> v2:172.21.15.81:3300/0 conn(0x7fcf4c159010 0x7fcf4c0c6a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:51.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.004+0000 7fcf53400640 1 --2- 172.21.15.28:0/4169051901 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcf4c1525c0 0x7fcf4c0cb390 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:51.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.004+0000 7fcf53400640 1 -- 172.21.15.28:0/4169051901 >> 172.21.15.28:0/4169051901 conn(0x7fcf4c0729c0 msgr2=0x7fcf4c072dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:51.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.004+0000 7fcf53400640 1 -- 172.21.15.28:0/4169051901 shutdown_connections 2024-02-21T23:17:51.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.004+0000 7fcf53400640 1 -- 172.21.15.28:0/4169051901 wait complete. 2024-02-21T23:17:51.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.006+0000 7fcf53400640 1 Processor -- start 2024-02-21T23:17:51.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.006+0000 7fcf53400640 1 -- start start 2024-02-21T23:17:51.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.008+0000 7fcf53400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf4c0c6f40 0x7fcf4c1eed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:51.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.008+0000 7fcf53400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcf4c1525c0 0x7fcf4c0c9a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:51.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.008+0000 7fcf53400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf4c148450 0x7fcf4c1eb440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:51.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.008+0000 7fcf53400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf4c1594a0 con 0x7fcf4c148450 2024-02-21T23:17:51.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.008+0000 7fcf53400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf4c0caff0 con 0x7fcf4c0c6f40 2024-02-21T23:17:51.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.008+0000 7fcf53400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf4c155e70 con 0x7fcf4c1525c0 2024-02-21T23:17:51.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.008+0000 7fcf51400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf4c148450 0x7fcf4c1eb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:51.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.008+0000 7fcf51400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf4c148450 0x7fcf4c1eb440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35290/0 (socket says 172.21.15.28:35290) 2024-02-21T23:17:51.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.008+0000 7fcf51400640 1 -- 172.21.15.28:0/3893893832 learned_addr learned my addr 172.21.15.28:0/3893893832 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:51.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.008+0000 7fcf4be00640 1 --2- 172.21.15.28:0/3893893832 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcf4c1525c0 0x7fcf4c0c9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:51.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.008+0000 7fcf50a00640 1 --2- 172.21.15.28:0/3893893832 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf4c0c6f40 0x7fcf4c1eed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:51.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.009+0000 7fcf51400640 1 -- 172.21.15.28:0/3893893832 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcf4c1525c0 msgr2=0x7fcf4c0c9a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:51.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.009+0000 7fcf51400640 1 --2- 172.21.15.28:0/3893893832 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcf4c1525c0 0x7fcf4c0c9a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:51.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.009+0000 7fcf51400640 1 -- 172.21.15.28:0/3893893832 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf4c0c6f40 msgr2=0x7fcf4c1eed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:51.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.009+0000 7fcf51400640 1 --2- 172.21.15.28:0/3893893832 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcf4c0c6f40 0x7fcf4c1eed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:51.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.009+0000 7fcf51400640 1 -- 172.21.15.28:0/3893893832 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf4c157830 con 0x7fcf4c148450 2024-02-21T23:17:51.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.009+0000 7fcf4be00640 1 --2- 172.21.15.28:0/3893893832 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fcf4c1525c0 0x7fcf4c0c9a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:17:51.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.009+0000 7fcf51400640 1 --2- 172.21.15.28:0/3893893832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf4c148450 0x7fcf4c1eb440 secure :-1 s=READY pgs=2762 cs=0 l=1 rev1=1 crypto rx=0x7fcf4400c760 tx=0x7fcf4400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:51.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.009+0000 7fcf48c00640 1 -- 172.21.15.28:0/3893893832 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf44003da0 con 0x7fcf4c148450 2024-02-21T23:17:51.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.009+0000 7fcf48c00640 1 -- 172.21.15.28:0/3893893832 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf44005070 con 0x7fcf4c148450 2024-02-21T23:17:51.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.009+0000 7fcf48c00640 1 -- 172.21.15.28:0/3893893832 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf44005460 con 0x7fcf4c148450 2024-02-21T23:17:51.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.009+0000 7fcf53400640 1 -- 172.21.15.28:0/3893893832 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf4c157b10 con 0x7fcf4c148450 2024-02-21T23:17:51.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.009+0000 7fcf53400640 1 -- 172.21.15.28:0/3893893832 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf4c1eaf20 con 0x7fcf4c148450 2024-02-21T23:17:52.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.010+0000 7fcf53400640 1 -- 172.21.15.28:0/3893893832 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fcf4c067f30 con 0x7fcf4c148450 2024-02-21T23:17:52.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.012+0000 7fcf48c00640 1 -- 172.21.15.28:0/3893893832 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcf44005600 con 0x7fcf4c148450 2024-02-21T23:17:52.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.013+0000 7fcf48c00640 1 --2- 172.21.15.28:0/3893893832 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcf2c06b4d0 0x7fcf2c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:52.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.013+0000 7fcf48c00640 1 -- 172.21.15.28:0/3893893832 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 5593+0+0 (secure 0 0 0) 0x7fcf4408ab00 con 0x7fcf4c148450 2024-02-21T23:17:52.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.014+0000 7fcf48c00640 1 --2- 172.21.15.28:0/3893893832 >> [v2:172.21.15.28:6808/2193666734,v1:172.21.15.28:6810/2193666734] conn(0x7fcf2c071b50 0x7fcf2c073f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:52.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.014+0000 7fcf48c00640 1 -- 172.21.15.28:0/3893893832 --> [v2:172.21.15.28:6808/2193666734,v1:172.21.15.28:6810/2193666734] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcf2c074660 con 0x7fcf2c071b50 2024-02-21T23:17:52.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.014+0000 7fcf48c00640 1 -- 172.21.15.28:0/3893893832 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=215) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcf4408aeb0 con 0x7fcf4c148450 2024-02-21T23:17:52.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.014+0000 7fcf50a00640 1 --2- 172.21.15.28:0/3893893832 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcf2c06b4d0 0x7fcf2c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:52.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.014+0000 7fcf4be00640 1 --2- 172.21.15.28:0/3893893832 >> [v2:172.21.15.28:6808/2193666734,v1:172.21.15.28:6810/2193666734] conn(0x7fcf2c071b50 0x7fcf2c073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:52.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.015+0000 7fcf4be00640 1 --2- 172.21.15.28:0/3893893832 >> [v2:172.21.15.28:6808/2193666734,v1:172.21.15.28:6810/2193666734] conn(0x7fcf2c071b50 0x7fcf2c073f90 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:52.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.015+0000 7fcf50a00640 1 --2- 172.21.15.28:0/3893893832 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcf2c06b4d0 0x7fcf2c06d990 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7fcf40036000 tx=0x7fcf4003b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:52.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.015+0000 7fcf48c00640 1 -- 172.21.15.28:0/3893893832 <== osd.1 v2:172.21.15.28:6808/2193666734 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fcf2c074660 con 0x7fcf2c071b50 2024-02-21T23:17:52.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.030+0000 7fcf53400640 1 -- 172.21.15.28:0/3893893832 --> [v2:172.21.15.28:6808/2193666734,v1:172.21.15.28:6810/2193666734] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fcf4c1507e0 con 0x7fcf2c071b50 2024-02-21T23:17:52.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.031+0000 7fcf48c00640 1 -- 172.21.15.28:0/3893893832 <== osd.1 v2:172.21.15.28:6808/2193666734 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fcf4c1507e0 con 0x7fcf2c071b50 2024-02-21T23:17:52.019 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-21T23:17:52.019 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-02-21T23:17:52.019 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-21T23:17:52.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.031+0000 7fcf53400640 1 -- 172.21.15.28:0/3893893832 >> [v2:172.21.15.28:6808/2193666734,v1:172.21.15.28:6810/2193666734] conn(0x7fcf2c071b50 msgr2=0x7fcf2c073f90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:52.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.031+0000 7fcf53400640 1 --2- 172.21.15.28:0/3893893832 >> [v2:172.21.15.28:6808/2193666734,v1:172.21.15.28:6810/2193666734] conn(0x7fcf2c071b50 0x7fcf2c073f90 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:52.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.031+0000 7fcf53400640 1 -- 172.21.15.28:0/3893893832 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcf2c06b4d0 msgr2=0x7fcf2c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:52.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.031+0000 7fcf53400640 1 --2- 172.21.15.28:0/3893893832 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fcf2c06b4d0 0x7fcf2c06d990 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7fcf40036000 tx=0x7fcf4003b040 comp rx=0 tx=0).stop 2024-02-21T23:17:52.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.031+0000 7fcf53400640 1 -- 172.21.15.28:0/3893893832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf4c148450 msgr2=0x7fcf4c1eb440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:52.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.031+0000 7fcf53400640 1 --2- 172.21.15.28:0/3893893832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf4c148450 0x7fcf4c1eb440 secure :-1 s=READY pgs=2762 cs=0 l=1 rev1=1 crypto rx=0x7fcf4400c760 tx=0x7fcf4400cc20 comp rx=0 tx=0).stop 2024-02-21T23:17:52.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.031+0000 7fcf51400640 1 -- 172.21.15.28:0/3893893832 reap_dead start 2024-02-21T23:17:52.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.031+0000 7fcf53400640 1 -- 172.21.15.28:0/3893893832 shutdown_connections 2024-02-21T23:17:52.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.031+0000 7fcf53400640 1 -- 172.21.15.28:0/3893893832 >> 172.21.15.28:0/3893893832 conn(0x7fcf4c0729c0 msgr2=0x7fcf4c14e3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:52.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.032+0000 7fcf53400640 1 -- 172.21.15.28:0/3893893832 shutdown_connections 2024-02-21T23:17:52.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.032+0000 7fcf53400640 1 -- 172.21.15.28:0/3893893832 wait complete. 2024-02-21T23:17:52.038 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-02-21T23:17:52.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.270+0000 7fda8fe00640 1 Processor -- start 2024-02-21T23:17:52.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.270+0000 7fda8fe00640 1 -- start start 2024-02-21T23:17:52.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.271+0000 7fda8fe00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fda880c8010 0x7fda88158f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:52.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.271+0000 7fda8fe00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda880c6110 con 0x7fda881525d0 2024-02-21T23:17:52.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.271+0000 7fda8fe00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda880c33b0 con 0x7fda880a7f40 2024-02-21T23:17:52.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.271+0000 7fda8fe00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda880c3ce0 con 0x7fda880c8010 2024-02-21T23:17:52.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.272+0000 7fda8d400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fda881525d0 0x7fda880c6930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:54386/0 (socket says 172.21.15.28:54386) 2024-02-21T23:17:52.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.272+0000 7fda8d400640 1 -- 172.21.15.28:0/1040825172 learned_addr learned my addr 172.21.15.28:0/1040825172 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:17:52.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.272+0000 7fda87e00640 1 -- 172.21.15.28:0/1040825172 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 306858293 0 0) 0x7fda880c6110 con 0x7fda881525d0 2024-02-21T23:17:52.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.272+0000 7fda87e00640 1 -- 172.21.15.28:0/1040825172 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda6c003520 con 0x7fda881525d0 2024-02-21T23:17:52.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.272+0000 7fda8de00640 1 --2- 172.21.15.28:0/1040825172 >> v2:172.21.15.81:3300/0 conn(0x7fda880c8010 0x7fda88158f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:52.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.273+0000 7fda87e00640 1 -- 172.21.15.28:0/1040825172 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3873905366 0 0) 0x7fda6c003520 con 0x7fda881525d0 2024-02-21T23:17:52.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.273+0000 7fda87e00640 1 -- 172.21.15.28:0/1040825172 >> v2:172.21.15.81:3300/0 conn(0x7fda880c8010 msgr2=0x7fda88158f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:52.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.273+0000 7fda87e00640 1 --2- 172.21.15.28:0/1040825172 >> v2:172.21.15.81:3300/0 conn(0x7fda880c8010 0x7fda88158f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:52.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.273+0000 7fda87e00640 1 -- 172.21.15.28:0/1040825172 >> v1:172.21.15.81:6789/0 conn(0x7fda880a7f40 legacy=0x7fda881587f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:52.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.273+0000 7fda87e00640 1 -- 172.21.15.28:0/1040825172 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda880c3690 con 0x7fda881525d0 2024-02-21T23:17:52.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.273+0000 7fda87e00640 1 -- 172.21.15.28:0/1040825172 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fda78001670 con 0x7fda881525d0 2024-02-21T23:17:52.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.273+0000 7fda87e00640 1 -- 172.21.15.28:0/1040825172 >> v1:172.21.15.28:6789/0 conn(0x7fda881525d0 legacy=0x7fda880c6930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:52.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.273+0000 7fda87e00640 1 --2- 172.21.15.28:0/1040825172 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda6c003a00 0x7fda6c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:52.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.273+0000 7fda87e00640 1 --2- 172.21.15.28:0/1040825172 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fda6c004800 0x7fda6c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:52.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.273+0000 7fda87e00640 1 --2- 172.21.15.28:0/1040825172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda6c005210 0x7fda6c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:52.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.273+0000 7fda87e00640 1 -- 172.21.15.28:0/1040825172 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda880c3ce0 con 0x7fda6c005210 2024-02-21T23:17:52.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.273+0000 7fda87e00640 1 -- 172.21.15.28:0/1040825172 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda6c00c140 con 0x7fda6c003a00 2024-02-21T23:17:52.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.273+0000 7fda87e00640 1 -- 172.21.15.28:0/1040825172 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda6c00c340 con 0x7fda6c004800 2024-02-21T23:17:52.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.273+0000 7fda8de00640 1 --2- 172.21.15.28:0/1040825172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda6c005210 0x7fda6c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:52.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.273+0000 7fda8d400640 1 --2- 172.21.15.28:0/1040825172 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda6c003a00 0x7fda6c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:52.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.274+0000 7fda8ca00640 1 --2- 172.21.15.28:0/1040825172 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fda6c004800 0x7fda6c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:52.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.274+0000 7fda8de00640 1 -- 172.21.15.28:0/1040825172 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fda6c004800 msgr2=0x7fda6c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:52.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.274+0000 7fda8de00640 1 --2- 172.21.15.28:0/1040825172 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fda6c004800 0x7fda6c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:52.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.274+0000 7fda8de00640 1 -- 172.21.15.28:0/1040825172 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda6c003a00 msgr2=0x7fda6c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:52.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.274+0000 7fda8de00640 1 --2- 172.21.15.28:0/1040825172 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda6c003a00 0x7fda6c003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:52.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.274+0000 7fda8de00640 1 -- 172.21.15.28:0/1040825172 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda6c003520 con 0x7fda6c005210 2024-02-21T23:17:52.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.274+0000 7fda8d400640 1 --2- 172.21.15.28:0/1040825172 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda6c003a00 0x7fda6c003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-21T23:17:52.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.275+0000 7fda8de00640 1 -- 172.21.15.28:0/1040825172 reap_dead start 2024-02-21T23:17:52.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.275+0000 7fda8de00640 1 --2- 172.21.15.28:0/1040825172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda6c005210 0x7fda6c00b9c0 secure :-1 s=READY pgs=2764 cs=0 l=1 rev1=1 crypto rx=0x7fda80007330 tx=0x7fda6c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ce27078f9043652 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:52.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.275+0000 7fda87e00640 1 -- 172.21.15.28:0/1040825172 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda80009be0 con 0x7fda6c005210 2024-02-21T23:17:52.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.275+0000 7fda87e00640 1 -- 172.21.15.28:0/1040825172 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda8002ccd0 con 0x7fda6c005210 2024-02-21T23:17:52.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.275+0000 7fda87e00640 1 -- 172.21.15.28:0/1040825172 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda80003bd0 con 0x7fda6c005210 2024-02-21T23:17:52.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.276+0000 7fda8fe00640 1 -- 172.21.15.28:0/1040825172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda6c005210 msgr2=0x7fda6c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:52.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.276+0000 7fda8fe00640 1 --2- 172.21.15.28:0/1040825172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda6c005210 0x7fda6c00b9c0 secure :-1 s=READY pgs=2764 cs=0 l=1 rev1=1 crypto rx=0x7fda80007330 tx=0x7fda6c005830 comp rx=0 tx=0).stop 2024-02-21T23:17:52.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.276+0000 7fda8fe00640 1 -- 172.21.15.28:0/1040825172 shutdown_connections 2024-02-21T23:17:52.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.276+0000 7fda8fe00640 1 --2- 172.21.15.28:0/1040825172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda6c005210 0x7fda6c00b9c0 unknown :-1 s=CLOSED pgs=2764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:52.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.276+0000 7fda8fe00640 1 -- 172.21.15.28:0/1040825172 >> 172.21.15.28:0/1040825172 conn(0x7fda880c8810 msgr2=0x7fda880c8bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:52.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.276+0000 7fda8fe00640 1 -- 172.21.15.28:0/1040825172 shutdown_connections 2024-02-21T23:17:52.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.276+0000 7fda8fe00640 1 -- 172.21.15.28:0/1040825172 wait complete. 2024-02-21T23:17:52.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.278+0000 7fda8fe00640 1 Processor -- start 2024-02-21T23:17:52.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.278+0000 7fda8fe00640 1 -- start start 2024-02-21T23:17:52.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.279+0000 7fda8fe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda6c005210 0x7fda8814af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:52.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.279+0000 7fda8fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda88153730 0x7fda881ee430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:52.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.279+0000 7fda8fe00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fda881ee970 0x7fda88070340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:52.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.279+0000 7fda8fe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda88152b30 con 0x7fda88153730 2024-02-21T23:17:52.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.279+0000 7fda8fe00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda881529b0 con 0x7fda6c005210 2024-02-21T23:17:52.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.279+0000 7fda8fe00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda880c6930 con 0x7fda881ee970 2024-02-21T23:17:52.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.279+0000 7fda8ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda88153730 0x7fda881ee430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:52.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.279+0000 7fda8ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda88153730 0x7fda881ee430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35310/0 (socket says 172.21.15.28:35310) 2024-02-21T23:17:52.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.279+0000 7fda8ca00640 1 -- 172.21.15.28:0/1845303432 learned_addr learned my addr 172.21.15.28:0/1845303432 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:52.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.279+0000 7fda8d400640 1 --2- 172.21.15.28:0/1845303432 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda6c005210 0x7fda8814af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:52.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.279+0000 7fda8de00640 1 --2- 172.21.15.28:0/1845303432 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fda881ee970 0x7fda88070340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:52.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.279+0000 7fda8ca00640 1 -- 172.21.15.28:0/1845303432 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fda881ee970 msgr2=0x7fda88070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:52.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.279+0000 7fda8ca00640 1 --2- 172.21.15.28:0/1845303432 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fda881ee970 0x7fda88070340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:52.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.279+0000 7fda8ca00640 1 -- 172.21.15.28:0/1845303432 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda6c005210 msgr2=0x7fda8814af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:52.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.279+0000 7fda8ca00640 1 --2- 172.21.15.28:0/1845303432 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda6c005210 0x7fda8814af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:52.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.279+0000 7fda8ca00640 1 -- 172.21.15.28:0/1845303432 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda8806cfe0 con 0x7fda88153730 2024-02-21T23:17:52.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.279+0000 7fda8d400640 1 --2- 172.21.15.28:0/1845303432 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda6c005210 0x7fda8814af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:17:52.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.280+0000 7fda8ca00640 1 --2- 172.21.15.28:0/1845303432 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda88153730 0x7fda881ee430 secure :-1 s=READY pgs=2765 cs=0 l=1 rev1=1 crypto rx=0x7fda7400e4c0 tx=0x7fda7400e980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:52.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.280+0000 7fda85600640 1 -- 172.21.15.28:0/1845303432 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda740135f0 con 0x7fda88153730 2024-02-21T23:17:52.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.280+0000 7fda85600640 1 -- 172.21.15.28:0/1845303432 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda7401a070 con 0x7fda88153730 2024-02-21T23:17:52.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.280+0000 7fda85600640 1 -- 172.21.15.28:0/1845303432 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda740139d0 con 0x7fda88153730 2024-02-21T23:17:52.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.280+0000 7fda8fe00640 1 -- 172.21.15.28:0/1845303432 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda881edf50 con 0x7fda88153730 2024-02-21T23:17:52.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.280+0000 7fda8fe00640 1 -- 172.21.15.28:0/1845303432 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda8806ddd0 con 0x7fda88153730 2024-02-21T23:17:52.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.281+0000 7fda8fe00640 1 -- 172.21.15.28:0/1845303432 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda881ee120 con 0x7fda88153730 2024-02-21T23:17:52.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.282+0000 7fda85600640 1 -- 172.21.15.28:0/1845303432 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fda74013b70 con 0x7fda88153730 2024-02-21T23:17:52.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.284+0000 7fda85600640 1 --2- 172.21.15.28:0/1845303432 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fda6806b4d0 0x7fda6806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:52.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.284+0000 7fda85600640 1 -- 172.21.15.28:0/1845303432 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 5593+0+0 (secure 0 0 0) 0x7fda7408a0c0 con 0x7fda88153730 2024-02-21T23:17:52.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.284+0000 7fda8d400640 1 --2- 172.21.15.28:0/1845303432 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fda6806b4d0 0x7fda6806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:52.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.286+0000 7fda8d400640 1 --2- 172.21.15.28:0/1845303432 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fda6806b4d0 0x7fda6806d990 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7fda780017e0 tx=0x7fda78001430 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:52.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.288+0000 7fda85600640 1 -- 172.21.15.28:0/1845303432 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fda74024020 con 0x7fda88153730 2024-02-21T23:17:52.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.489+0000 7fda8fe00640 1 -- 172.21.15.28:0/1845303432 --> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7fda881eed50 con 0x7fda6806b4d0 2024-02-21T23:17:52.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.491+0000 7fda85600640 1 -- 172.21.15.28:0/1845303432 <== mgr.4101 v2:172.21.15.81:6832/1848538577 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fda78005d40 con 0x7fda6806b4d0 2024-02-21T23:17:52.478 INFO:teuthology.orchestra.run.smithi028.stderr:instructed osd(s) 1 to deep-scrub 2024-02-21T23:17:52.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.493+0000 7fda8fe00640 1 -- 172.21.15.28:0/1845303432 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fda6806b4d0 msgr2=0x7fda6806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:52.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.493+0000 7fda8fe00640 1 --2- 172.21.15.28:0/1845303432 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fda6806b4d0 0x7fda6806d990 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7fda780017e0 tx=0x7fda78001430 comp rx=0 tx=0).stop 2024-02-21T23:17:52.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.493+0000 7fda8fe00640 1 -- 172.21.15.28:0/1845303432 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda88153730 msgr2=0x7fda881ee430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:52.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.493+0000 7fda8fe00640 1 --2- 172.21.15.28:0/1845303432 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda88153730 0x7fda881ee430 secure :-1 s=READY pgs=2765 cs=0 l=1 rev1=1 crypto rx=0x7fda7400e4c0 tx=0x7fda7400e980 comp rx=0 tx=0).stop 2024-02-21T23:17:52.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.493+0000 7fda8fe00640 1 -- 172.21.15.28:0/1845303432 shutdown_connections 2024-02-21T23:17:52.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.493+0000 7fda8fe00640 1 --2- 172.21.15.28:0/1845303432 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fda6806b4d0 0x7fda6806d990 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:52.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.493+0000 7fda8fe00640 1 --2- 172.21.15.28:0/1845303432 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fda881ee970 0x7fda88070340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:52.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.493+0000 7fda8fe00640 1 --2- 172.21.15.28:0/1845303432 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda88153730 0x7fda881ee430 unknown :-1 s=CLOSED pgs=2765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:52.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.494+0000 7fda8fe00640 1 --2- 172.21.15.28:0/1845303432 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fda6c005210 0x7fda8814af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:52.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.494+0000 7fda8fe00640 1 -- 172.21.15.28:0/1845303432 >> 172.21.15.28:0/1845303432 conn(0x7fda880c8810 msgr2=0x7fda8814f960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:52.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.494+0000 7fda8fe00640 1 -- 172.21.15.28:0/1845303432 shutdown_connections 2024-02-21T23:17:52.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.494+0000 7fda8fe00640 1 -- 172.21.15.28:0/1845303432 wait complete. 2024-02-21T23:17:52.500 INFO:tasks.ceph:Scrubbing osd.2 2024-02-21T23:17:52.501 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-02-21T23:17:52.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.737+0000 7fbf2fe00640 1 Processor -- start 2024-02-21T23:17:52.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.737+0000 7fbf2fe00640 1 -- start start 2024-02-21T23:17:52.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.739+0000 7fbf2fe00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbf301501c0 0x7fbf300a96c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:52.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.739+0000 7fbf2fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbf300a9c00 0x7fbf3006e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:52.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.739+0000 7fbf2fe00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fbf300c1010 0x7fbf3006e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:52.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.739+0000 7fbf2fe00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf30152fd0 con 0x7fbf300a9c00 2024-02-21T23:17:52.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.739+0000 7fbf2fe00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf301532b0 con 0x7fbf300c1010 2024-02-21T23:17:52.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.739+0000 7fbf2fe00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf30153590 con 0x7fbf301501c0 2024-02-21T23:17:52.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.739+0000 7fbf2e000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbf300a9c00 0x7fbf3006e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:52.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.739+0000 7fbf2e000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbf300a9c00 0x7fbf3006e330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35316/0 (socket says 172.21.15.28:35316) 2024-02-21T23:17:52.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.739+0000 7fbf2e000640 1 -- 172.21.15.28:0/1200167614 learned_addr learned my addr 172.21.15.28:0/1200167614 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:52.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.739+0000 7fbf2f400640 1 --2- 172.21.15.28:0/1200167614 >> v2:172.21.15.81:3300/0 conn(0x7fbf300c1010 0x7fbf3006e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:52.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.740+0000 7fbf2e000640 1 -- 172.21.15.28:0/1200167614 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbf301501c0 msgr2=0x7fbf300a96c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:52.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.740+0000 7fbf2ea00640 1 --2- 172.21.15.28:0/1200167614 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbf301501c0 0x7fbf300a96c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:52.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.740+0000 7fbf2e000640 1 --2- 172.21.15.28:0/1200167614 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbf301501c0 0x7fbf300a96c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:52.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.740+0000 7fbf2e000640 1 -- 172.21.15.28:0/1200167614 >> v2:172.21.15.81:3300/0 conn(0x7fbf300c1010 msgr2=0x7fbf3006e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:52.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.740+0000 7fbf2e000640 1 --2- 172.21.15.28:0/1200167614 >> v2:172.21.15.81:3300/0 conn(0x7fbf300c1010 0x7fbf3006e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:52.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.740+0000 7fbf2e000640 1 -- 172.21.15.28:0/1200167614 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf30153870 con 0x7fbf300a9c00 2024-02-21T23:17:52.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.740+0000 7fbf2f400640 1 --2- 172.21.15.28:0/1200167614 >> v2:172.21.15.81:3300/0 conn(0x7fbf300c1010 0x7fbf3006e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:17:52.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.740+0000 7fbf2ea00640 1 --2- 172.21.15.28:0/1200167614 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbf301501c0 0x7fbf300a96c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:17:52.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.741+0000 7fbf2e000640 1 --2- 172.21.15.28:0/1200167614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf300a9c00 0x7fbf3006e330 secure :-1 s=READY pgs=2766 cs=0 l=1 rev1=1 crypto rx=0x7fbf300c14a0 tx=0x7fbf1c02b800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f5b0020415f1d1f server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:52.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.741+0000 7fbf2d600640 1 -- 172.21.15.28:0/1200167614 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf1c031760 con 0x7fbf300a9c00 2024-02-21T23:17:52.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.741+0000 7fbf2d600640 1 -- 172.21.15.28:0/1200167614 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf1c037070 con 0x7fbf300a9c00 2024-02-21T23:17:52.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.741+0000 7fbf2d600640 1 -- 172.21.15.28:0/1200167614 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf1c031b20 con 0x7fbf300a9c00 2024-02-21T23:17:52.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.742+0000 7fbf2fe00640 1 -- 172.21.15.28:0/1200167614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf300a9c00 msgr2=0x7fbf3006e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:52.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.742+0000 7fbf2fe00640 1 --2- 172.21.15.28:0/1200167614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf300a9c00 0x7fbf3006e330 secure :-1 s=READY pgs=2766 cs=0 l=1 rev1=1 crypto rx=0x7fbf300c14a0 tx=0x7fbf1c02b800 comp rx=0 tx=0).stop 2024-02-21T23:17:52.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.742+0000 7fbf2fe00640 1 -- 172.21.15.28:0/1200167614 shutdown_connections 2024-02-21T23:17:52.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.742+0000 7fbf2fe00640 1 --2- 172.21.15.28:0/1200167614 >> v2:172.21.15.81:3300/0 conn(0x7fbf300c1010 0x7fbf3006e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:52.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.742+0000 7fbf2fe00640 1 --2- 172.21.15.28:0/1200167614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf300a9c00 0x7fbf3006e330 unknown :-1 s=CLOSED pgs=2766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:52.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.742+0000 7fbf2fe00640 1 --2- 172.21.15.28:0/1200167614 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbf301501c0 0x7fbf300a96c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:52.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.742+0000 7fbf2fe00640 1 -- 172.21.15.28:0/1200167614 >> 172.21.15.28:0/1200167614 conn(0x7fbf300c6610 msgr2=0x7fbf300c69f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:52.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.742+0000 7fbf2fe00640 1 -- 172.21.15.28:0/1200167614 shutdown_connections 2024-02-21T23:17:52.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.742+0000 7fbf2fe00640 1 -- 172.21.15.28:0/1200167614 wait complete. 2024-02-21T23:17:52.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.744+0000 7fbf2fe00640 1 Processor -- start 2024-02-21T23:17:52.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.744+0000 7fbf2fe00640 1 -- start start 2024-02-21T23:17:52.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.745+0000 7fbf2fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf300a9c00 0x7fbf301eaf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:52.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.745+0000 7fbf2fe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf300c1010 0x7fbf300bfcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:52.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.745+0000 7fbf2fe00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbf301501c0 0x7fbf30148f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:52.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.745+0000 7fbf2fe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf300c7340 con 0x7fbf300a9c00 2024-02-21T23:17:52.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.745+0000 7fbf2fe00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf300c0c20 con 0x7fbf300c1010 2024-02-21T23:17:52.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.745+0000 7fbf2fe00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf301505a0 con 0x7fbf301501c0 2024-02-21T23:17:52.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.746+0000 7fbf2ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf300a9c00 0x7fbf301eaf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:52.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.746+0000 7fbf2ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf300a9c00 0x7fbf301eaf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35324/0 (socket says 172.21.15.28:35324) 2024-02-21T23:17:52.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.746+0000 7fbf2ea00640 1 -- 172.21.15.28:0/2905235557 learned_addr learned my addr 172.21.15.28:0/2905235557 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:52.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.746+0000 7fbf2f400640 1 --2- 172.21.15.28:0/2905235557 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbf301501c0 0x7fbf30148f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:52.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.746+0000 7fbf2e000640 1 --2- 172.21.15.28:0/2905235557 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf300c1010 0x7fbf300bfcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:52.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.746+0000 7fbf2ea00640 1 -- 172.21.15.28:0/2905235557 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbf301501c0 msgr2=0x7fbf30148f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:52.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.746+0000 7fbf2ea00640 1 --2- 172.21.15.28:0/2905235557 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fbf301501c0 0x7fbf30148f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:52.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.746+0000 7fbf2ea00640 1 -- 172.21.15.28:0/2905235557 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf300c1010 msgr2=0x7fbf300bfcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:52.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.746+0000 7fbf2ea00640 1 --2- 172.21.15.28:0/2905235557 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf300c1010 0x7fbf300bfcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:52.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.746+0000 7fbf2ea00640 1 -- 172.21.15.28:0/2905235557 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf3014b130 con 0x7fbf300a9c00 2024-02-21T23:17:52.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.747+0000 7fbf2ea00640 1 --2- 172.21.15.28:0/2905235557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf300a9c00 0x7fbf301eaf60 secure :-1 s=READY pgs=2767 cs=0 l=1 rev1=1 crypto rx=0x7fbf2000c7f0 tx=0x7fbf2000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:52.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.747+0000 7fbf26a00640 1 -- 172.21.15.28:0/2905235557 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf20015070 con 0x7fbf300a9c00 2024-02-21T23:17:52.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.747+0000 7fbf26a00640 1 -- 172.21.15.28:0/2905235557 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf20007a30 con 0x7fbf300a9c00 2024-02-21T23:17:52.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.747+0000 7fbf26a00640 1 -- 172.21.15.28:0/2905235557 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf20007e00 con 0x7fbf300a9c00 2024-02-21T23:17:52.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.747+0000 7fbf2fe00640 1 -- 172.21.15.28:0/2905235557 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf3014b410 con 0x7fbf300a9c00 2024-02-21T23:17:52.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.747+0000 7fbf2fe00640 1 -- 172.21.15.28:0/2905235557 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf300716b0 con 0x7fbf300a9c00 2024-02-21T23:17:52.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.748+0000 7fbf2fe00640 1 -- 172.21.15.28:0/2905235557 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fbf30071a20 con 0x7fbf300a9c00 2024-02-21T23:17:52.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.749+0000 7fbf26a00640 1 -- 172.21.15.28:0/2905235557 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbf20018070 con 0x7fbf300a9c00 2024-02-21T23:17:52.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.750+0000 7fbf26a00640 1 --2- 172.21.15.28:0/2905235557 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbf1006b500 0x7fbf1006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:52.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.750+0000 7fbf26a00640 1 -- 172.21.15.28:0/2905235557 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 5593+0+0 (secure 0 0 0) 0x7fbf2008a1e0 con 0x7fbf300a9c00 2024-02-21T23:17:52.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.750+0000 7fbf26a00640 1 --2- 172.21.15.28:0/2905235557 >> [v2:172.21.15.28:6809/160098884,v1:172.21.15.28:6812/160098884] conn(0x7fbf10071b80 0x7fbf10073fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:52.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.750+0000 7fbf26a00640 1 -- 172.21.15.28:0/2905235557 --> [v2:172.21.15.28:6809/160098884,v1:172.21.15.28:6812/160098884] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbf10074690 con 0x7fbf10071b80 2024-02-21T23:17:52.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.750+0000 7fbf26a00640 1 -- 172.21.15.28:0/2905235557 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=215) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbf20091050 con 0x7fbf300a9c00 2024-02-21T23:17:52.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.751+0000 7fbf2f400640 1 --2- 172.21.15.28:0/2905235557 >> [v2:172.21.15.28:6809/160098884,v1:172.21.15.28:6812/160098884] conn(0x7fbf10071b80 0x7fbf10073fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:52.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.751+0000 7fbf2e000640 1 --2- 172.21.15.28:0/2905235557 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbf1006b500 0x7fbf1006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:52.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.751+0000 7fbf2f400640 1 --2- 172.21.15.28:0/2905235557 >> [v2:172.21.15.28:6809/160098884,v1:172.21.15.28:6812/160098884] conn(0x7fbf10071b80 0x7fbf10073fc0 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:52.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.752+0000 7fbf26a00640 1 -- 172.21.15.28:0/2905235557 <== osd.2 v2:172.21.15.28:6809/160098884 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fbf10074690 con 0x7fbf10071b80 2024-02-21T23:17:52.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.752+0000 7fbf2e000640 1 --2- 172.21.15.28:0/2905235557 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbf1006b500 0x7fbf1006d9c0 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7fbf1c036000 tx=0x7fbf1c03d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:52.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.767+0000 7fbf2fe00640 1 -- 172.21.15.28:0/2905235557 --> [v2:172.21.15.28:6809/160098884,v1:172.21.15.28:6812/160098884] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fbf3004f290 con 0x7fbf10071b80 2024-02-21T23:17:52.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.767+0000 7fbf26a00640 1 -- 172.21.15.28:0/2905235557 <== osd.2 v2:172.21.15.28:6809/160098884 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fbf3004f290 con 0x7fbf10071b80 2024-02-21T23:17:52.755 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-21T23:17:52.755 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-02-21T23:17:52.755 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-21T23:17:52.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.768+0000 7fbf2fe00640 1 -- 172.21.15.28:0/2905235557 >> [v2:172.21.15.28:6809/160098884,v1:172.21.15.28:6812/160098884] conn(0x7fbf10071b80 msgr2=0x7fbf10073fc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:52.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.768+0000 7fbf2fe00640 1 --2- 172.21.15.28:0/2905235557 >> [v2:172.21.15.28:6809/160098884,v1:172.21.15.28:6812/160098884] conn(0x7fbf10071b80 0x7fbf10073fc0 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:52.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.768+0000 7fbf2fe00640 1 -- 172.21.15.28:0/2905235557 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbf1006b500 msgr2=0x7fbf1006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:52.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.768+0000 7fbf2fe00640 1 --2- 172.21.15.28:0/2905235557 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fbf1006b500 0x7fbf1006d9c0 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7fbf1c036000 tx=0x7fbf1c03d040 comp rx=0 tx=0).stop 2024-02-21T23:17:52.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.768+0000 7fbf2fe00640 1 -- 172.21.15.28:0/2905235557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf300a9c00 msgr2=0x7fbf301eaf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:52.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.768+0000 7fbf2fe00640 1 --2- 172.21.15.28:0/2905235557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf300a9c00 0x7fbf301eaf60 secure :-1 s=READY pgs=2767 cs=0 l=1 rev1=1 crypto rx=0x7fbf2000c7f0 tx=0x7fbf2000cbf0 comp rx=0 tx=0).stop 2024-02-21T23:17:52.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.768+0000 7fbf2f400640 1 -- 172.21.15.28:0/2905235557 reap_dead start 2024-02-21T23:17:52.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.768+0000 7fbf2fe00640 1 -- 172.21.15.28:0/2905235557 shutdown_connections 2024-02-21T23:17:52.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.768+0000 7fbf2fe00640 1 -- 172.21.15.28:0/2905235557 >> 172.21.15.28:0/2905235557 conn(0x7fbf300c6610 msgr2=0x7fbf3014e250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:52.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.768+0000 7fbf2fe00640 1 -- 172.21.15.28:0/2905235557 shutdown_connections 2024-02-21T23:17:52.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:52.768+0000 7fbf2fe00640 1 -- 172.21.15.28:0/2905235557 wait complete. 2024-02-21T23:17:52.773 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-02-21T23:17:52.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.007+0000 7fee75e00640 1 Processor -- start 2024-02-21T23:17:52.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.007+0000 7fee75e00640 1 -- start start 2024-02-21T23:17:52.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.009+0000 7fee75e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fee7006eca0 0x7fee7006dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:52.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.009+0000 7fee75e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee70155860 con 0x7fee70073140 2024-02-21T23:17:52.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.009+0000 7fee75e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee70151830 con 0x7fee7014ec20 2024-02-21T23:17:52.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.009+0000 7fee75e00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee70151600 con 0x7fee7006eca0 2024-02-21T23:17:52.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.009+0000 7fee6fe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fee70073140 0x7fee700c0ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:54396/0 (socket says 172.21.15.28:54396) 2024-02-21T23:17:52.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.009+0000 7fee6fe00640 1 -- 172.21.15.28:0/2383406134 learned_addr learned my addr 172.21.15.28:0/2383406134 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:17:52.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.009+0000 7fee6f400640 1 -- 172.21.15.28:0/2383406134 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2879068906 0 0) 0x7fee70155860 con 0x7fee70073140 2024-02-21T23:17:52.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.009+0000 7fee6f400640 1 -- 172.21.15.28:0/2383406134 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee54003440 con 0x7fee70073140 2024-02-21T23:17:52.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.010+0000 7fee75400640 1 --2- 172.21.15.28:0/2383406134 >> v2:172.21.15.81:3300/0 conn(0x7fee7006eca0 0x7fee7006dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:52.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.010+0000 7fee6f400640 1 -- 172.21.15.28:0/2383406134 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 304182473 0 0) 0x7fee54003440 con 0x7fee70073140 2024-02-21T23:17:52.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.010+0000 7fee6f400640 1 -- 172.21.15.28:0/2383406134 >> v2:172.21.15.81:3300/0 conn(0x7fee7006eca0 msgr2=0x7fee7006dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:52.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.010+0000 7fee6f400640 1 --2- 172.21.15.28:0/2383406134 >> v2:172.21.15.81:3300/0 conn(0x7fee7006eca0 0x7fee7006dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:53.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.010+0000 7fee6f400640 1 -- 172.21.15.28:0/2383406134 >> v1:172.21.15.81:6789/0 conn(0x7fee7014ec20 legacy=0x7fee70152030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:53.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.010+0000 7fee6f400640 1 -- 172.21.15.28:0/2383406134 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee70155530 con 0x7fee70073140 2024-02-21T23:17:53.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.010+0000 7fee6f400640 1 -- 172.21.15.28:0/2383406134 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fee58001670 con 0x7fee70073140 2024-02-21T23:17:53.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.010+0000 7fee6f400640 1 -- 172.21.15.28:0/2383406134 >> v1:172.21.15.28:6789/0 conn(0x7fee70073140 legacy=0x7fee700c0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:53.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.010+0000 7fee6f400640 1 --2- 172.21.15.28:0/2383406134 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fee54003aa0 0x7fee54003ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:53.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.010+0000 7fee6f400640 1 --2- 172.21.15.28:0/2383406134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee54008900 0x7fee5400ad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:53.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.010+0000 7fee6f400640 1 --2- 172.21.15.28:0/2383406134 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee5400b610 0x7fee5400dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:53.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.010+0000 7fee6f400640 1 -- 172.21.15.28:0/2383406134 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee5400e380 con 0x7fee54008900 2024-02-21T23:17:53.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.010+0000 7fee6f400640 1 -- 172.21.15.28:0/2383406134 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee5400e530 con 0x7fee5400b610 2024-02-21T23:17:53.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.010+0000 7fee6f400640 1 -- 172.21.15.28:0/2383406134 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee5400e730 con 0x7fee54003aa0 2024-02-21T23:17:53.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.010+0000 7fee6fe00640 1 --2- 172.21.15.28:0/2383406134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee54008900 0x7fee5400ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:53.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.011+0000 7fee74a00640 1 --2- 172.21.15.28:0/2383406134 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fee54003aa0 0x7fee54003ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:53.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.011+0000 7fee75400640 1 --2- 172.21.15.28:0/2383406134 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee5400b610 0x7fee5400dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:53.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.011+0000 7fee6fe00640 1 -- 172.21.15.28:0/2383406134 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fee54003aa0 msgr2=0x7fee54003ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:53.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.011+0000 7fee6fe00640 1 --2- 172.21.15.28:0/2383406134 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fee54003aa0 0x7fee54003ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:53.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.011+0000 7fee6fe00640 1 -- 172.21.15.28:0/2383406134 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee5400b610 msgr2=0x7fee5400dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:53.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.011+0000 7fee6fe00640 1 --2- 172.21.15.28:0/2383406134 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee5400b610 0x7fee5400dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:53.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.011+0000 7fee6fe00640 1 -- 172.21.15.28:0/2383406134 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee54003440 con 0x7fee54008900 2024-02-21T23:17:53.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.011+0000 7fee75400640 1 -- 172.21.15.28:0/2383406134 reap_dead start 2024-02-21T23:17:53.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.012+0000 7fee6fe00640 1 --2- 172.21.15.28:0/2383406134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee54008900 0x7fee5400ad70 secure :-1 s=READY pgs=2769 cs=0 l=1 rev1=1 crypto rx=0x7fee5800c210 tx=0x7fee54003730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddd78f8aeec71339 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:53.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.012+0000 7fee6f400640 1 -- 172.21.15.28:0/2383406134 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee58007d30 con 0x7fee54008900 2024-02-21T23:17:53.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.012+0000 7fee6f400640 1 -- 172.21.15.28:0/2383406134 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee58030070 con 0x7fee54008900 2024-02-21T23:17:53.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.012+0000 7fee6f400640 1 -- 172.21.15.28:0/2383406134 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee5802cda0 con 0x7fee54008900 2024-02-21T23:17:53.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.013+0000 7fee75e00640 1 -- 172.21.15.28:0/2383406134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee54008900 msgr2=0x7fee5400ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:53.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.013+0000 7fee75e00640 1 --2- 172.21.15.28:0/2383406134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee54008900 0x7fee5400ad70 secure :-1 s=READY pgs=2769 cs=0 l=1 rev1=1 crypto rx=0x7fee5800c210 tx=0x7fee54003730 comp rx=0 tx=0).stop 2024-02-21T23:17:53.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.013+0000 7fee75e00640 1 -- 172.21.15.28:0/2383406134 shutdown_connections 2024-02-21T23:17:53.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.013+0000 7fee75e00640 1 --2- 172.21.15.28:0/2383406134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee54008900 0x7fee5400ad70 unknown :-1 s=CLOSED pgs=2769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:53.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.013+0000 7fee75e00640 1 -- 172.21.15.28:0/2383406134 >> 172.21.15.28:0/2383406134 conn(0x7fee700a8320 msgr2=0x7fee700a8730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:53.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.013+0000 7fee75e00640 1 -- 172.21.15.28:0/2383406134 shutdown_connections 2024-02-21T23:17:53.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.013+0000 7fee75e00640 1 -- 172.21.15.28:0/2383406134 wait complete. 2024-02-21T23:17:53.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.015+0000 7fee75e00640 1 Processor -- start 2024-02-21T23:17:53.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.015+0000 7fee75e00640 1 -- start start 2024-02-21T23:17:53.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.016+0000 7fee75e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fee54008900 0x7fee7014c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:53.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.016+0000 7fee75e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee70099a50 0x7fee700abd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:53.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.016+0000 7fee75e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee7014caa0 0x7fee700a9580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:53.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.016+0000 7fee75e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee7014f000 con 0x7fee70099a50 2024-02-21T23:17:53.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.016+0000 7fee75e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee70151ba0 con 0x7fee7014caa0 2024-02-21T23:17:53.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.016+0000 7fee75e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee700c0ee0 con 0x7fee54008900 2024-02-21T23:17:53.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.016+0000 7fee6fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee70099a50 0x7fee700abd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:53.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.016+0000 7fee6fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee70099a50 0x7fee700abd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35330/0 (socket says 172.21.15.28:35330) 2024-02-21T23:17:53.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.016+0000 7fee6fe00640 1 -- 172.21.15.28:0/577794661 learned_addr learned my addr 172.21.15.28:0/577794661 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:53.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.017+0000 7fee74a00640 1 --2- 172.21.15.28:0/577794661 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fee54008900 0x7fee7014c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:53.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.017+0000 7fee75400640 1 --2- 172.21.15.28:0/577794661 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee7014caa0 0x7fee700a9580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:53.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.017+0000 7fee6fe00640 1 -- 172.21.15.28:0/577794661 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fee54008900 msgr2=0x7fee7014c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:53.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.017+0000 7fee6fe00640 1 --2- 172.21.15.28:0/577794661 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fee54008900 0x7fee7014c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:53.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.017+0000 7fee6fe00640 1 -- 172.21.15.28:0/577794661 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee7014caa0 msgr2=0x7fee700a9580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:53.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.017+0000 7fee6fe00640 1 --2- 172.21.15.28:0/577794661 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee7014caa0 0x7fee700a9580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:53.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.017+0000 7fee6fe00640 1 -- 172.21.15.28:0/577794661 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee7009c0c0 con 0x7fee70099a50 2024-02-21T23:17:53.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.017+0000 7fee6fe00640 1 --2- 172.21.15.28:0/577794661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee70099a50 0x7fee700abd90 secure :-1 s=READY pgs=2770 cs=0 l=1 rev1=1 crypto rx=0x7fee5800c210 tx=0x7fee5802e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:53.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.018+0000 7fee6cc00640 1 -- 172.21.15.28:0/577794661 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee58046070 con 0x7fee70099a50 2024-02-21T23:17:53.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.018+0000 7fee6cc00640 1 -- 172.21.15.28:0/577794661 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee58000ed0 con 0x7fee70099a50 2024-02-21T23:17:53.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.018+0000 7fee6cc00640 1 -- 172.21.15.28:0/577794661 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee58004e50 con 0x7fee70099a50 2024-02-21T23:17:53.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.018+0000 7fee75e00640 1 -- 172.21.15.28:0/577794661 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee701ee730 con 0x7fee70099a50 2024-02-21T23:17:53.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.018+0000 7fee75e00640 1 -- 172.21.15.28:0/577794661 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee700a7990 con 0x7fee70099a50 2024-02-21T23:17:53.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.019+0000 7fee75e00640 1 -- 172.21.15.28:0/577794661 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee700c73f0 con 0x7fee70099a50 2024-02-21T23:17:53.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.020+0000 7fee6cc00640 1 -- 172.21.15.28:0/577794661 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fee58007070 con 0x7fee70099a50 2024-02-21T23:17:53.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.021+0000 7fee6cc00640 1 --2- 172.21.15.28:0/577794661 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fee4c06b670 0x7fee4c06db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:53.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.021+0000 7fee6cc00640 1 -- 172.21.15.28:0/577794661 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 5593+0+0 (secure 0 0 0) 0x7fee580ab540 con 0x7fee70099a50 2024-02-21T23:17:53.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.022+0000 7fee74a00640 1 --2- 172.21.15.28:0/577794661 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fee4c06b670 0x7fee4c06db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:53.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.023+0000 7fee74a00640 1 --2- 172.21.15.28:0/577794661 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fee4c06b670 0x7fee4c06db30 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7fee60007fd0 tx=0x7fee60009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:53.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.026+0000 7fee6cc00640 1 -- 172.21.15.28:0/577794661 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fee700c73f0 con 0x7fee70099a50 2024-02-21T23:17:53.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.226+0000 7fee75e00640 1 -- 172.21.15.28:0/577794661 --> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7fee7004f290 con 0x7fee4c06b670 2024-02-21T23:17:53.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.227+0000 7fee6cc00640 1 -- 172.21.15.28:0/577794661 <== mgr.4101 v2:172.21.15.81:6832/1848538577 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fee7004f290 con 0x7fee4c06b670 2024-02-21T23:17:53.214 INFO:teuthology.orchestra.run.smithi028.stderr:instructed osd(s) 2 to deep-scrub 2024-02-21T23:17:53.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.230+0000 7fee75e00640 1 -- 172.21.15.28:0/577794661 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fee4c06b670 msgr2=0x7fee4c06db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:53.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.230+0000 7fee75e00640 1 --2- 172.21.15.28:0/577794661 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fee4c06b670 0x7fee4c06db30 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7fee60007fd0 tx=0x7fee60009040 comp rx=0 tx=0).stop 2024-02-21T23:17:53.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.230+0000 7fee75e00640 1 -- 172.21.15.28:0/577794661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee70099a50 msgr2=0x7fee700abd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:53.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.230+0000 7fee75e00640 1 --2- 172.21.15.28:0/577794661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee70099a50 0x7fee700abd90 secure :-1 s=READY pgs=2770 cs=0 l=1 rev1=1 crypto rx=0x7fee5800c210 tx=0x7fee5802e070 comp rx=0 tx=0).stop 2024-02-21T23:17:53.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.230+0000 7fee75e00640 1 -- 172.21.15.28:0/577794661 shutdown_connections 2024-02-21T23:17:53.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.230+0000 7fee75e00640 1 --2- 172.21.15.28:0/577794661 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fee4c06b670 0x7fee4c06db30 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:53.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.230+0000 7fee75e00640 1 --2- 172.21.15.28:0/577794661 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee7014caa0 0x7fee700a9580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:53.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.230+0000 7fee75e00640 1 --2- 172.21.15.28:0/577794661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee70099a50 0x7fee700abd90 unknown :-1 s=CLOSED pgs=2770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:53.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.230+0000 7fee75e00640 1 --2- 172.21.15.28:0/577794661 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fee54008900 0x7fee7014c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:53.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.230+0000 7fee75e00640 1 -- 172.21.15.28:0/577794661 >> 172.21.15.28:0/577794661 conn(0x7fee700a8320 msgr2=0x7fee700c40d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:53.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.230+0000 7fee75e00640 1 -- 172.21.15.28:0/577794661 shutdown_connections 2024-02-21T23:17:53.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.231+0000 7fee75e00640 1 -- 172.21.15.28:0/577794661 wait complete. 2024-02-21T23:17:53.238 INFO:tasks.ceph:Scrubbing osd.3 2024-02-21T23:17:53.238 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-02-21T23:17:53.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.473+0000 7fa7b7e00640 1 Processor -- start 2024-02-21T23:17:53.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.473+0000 7fa7b7e00640 1 -- start start 2024-02-21T23:17:53.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.475+0000 7fa7b7e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa7b80c1010 0x7fa7b806e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:53.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.475+0000 7fa7b7e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa7b806e870 0x7fa7b815d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:53.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.475+0000 7fa7b7e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7b8152fd0 con 0x7fa7b81501c0 2024-02-21T23:17:53.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.475+0000 7fa7b7e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7b81532b0 con 0x7fa7b806e870 2024-02-21T23:17:53.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.475+0000 7fa7b7e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b8153590 con 0x7fa7b80c1010 2024-02-21T23:17:53.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.475+0000 7fa7b7400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa7b806e870 0x7fa7b815d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:53.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.476+0000 7fa7b7400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa7b806e870 0x7fa7b815d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35344/0 (socket says 172.21.15.28:35344) 2024-02-21T23:17:53.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.476+0000 7fa7b7400640 1 -- 172.21.15.28:0/2754792416 learned_addr learned my addr 172.21.15.28:0/2754792416 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:53.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.476+0000 7fa7afe00640 1 --2- 172.21.15.28:0/2754792416 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa7b80c1010 0x7fa7b806e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:53.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.476+0000 7fa7b7400640 1 -- 172.21.15.28:0/2754792416 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa7b80c1010 msgr2=0x7fa7b806e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:53.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.476+0000 7fa7b7400640 1 --2- 172.21.15.28:0/2754792416 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa7b80c1010 0x7fa7b806e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:53.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.476+0000 7fa7b7400640 1 -- 172.21.15.28:0/2754792416 >> v1:172.21.15.81:6789/0 conn(0x7fa7b81501c0 legacy=0x7fa7b80a96c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:53.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.476+0000 7fa7b7400640 1 -- 172.21.15.28:0/2754792416 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7b8153870 con 0x7fa7b806e870 2024-02-21T23:17:53.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.478+0000 7fa7b7400640 1 --2- 172.21.15.28:0/2754792416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7b806e870 0x7fa7b815d5b0 secure :-1 s=READY pgs=2771 cs=0 l=1 rev1=1 crypto rx=0x7fa7b0009e50 tx=0x7fa7b002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f9e177c69b9de59 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:53.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.478+0000 7fa7b6000640 1 -- 172.21.15.28:0/2754792416 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b002ca70 con 0x7fa7b806e870 2024-02-21T23:17:53.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.478+0000 7fa7b6000640 1 -- 172.21.15.28:0/2754792416 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7b002cc10 con 0x7fa7b806e870 2024-02-21T23:17:53.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.478+0000 7fa7b6000640 1 -- 172.21.15.28:0/2754792416 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b00343f0 con 0x7fa7b806e870 2024-02-21T23:17:53.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.478+0000 7fa7b7e00640 1 -- 172.21.15.28:0/2754792416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7b806e870 msgr2=0x7fa7b815d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:53.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.478+0000 7fa7b7e00640 1 --2- 172.21.15.28:0/2754792416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7b806e870 0x7fa7b815d5b0 secure :-1 s=READY pgs=2771 cs=0 l=1 rev1=1 crypto rx=0x7fa7b0009e50 tx=0x7fa7b002bed0 comp rx=0 tx=0).stop 2024-02-21T23:17:53.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.479+0000 7fa7b7e00640 1 -- 172.21.15.28:0/2754792416 shutdown_connections 2024-02-21T23:17:53.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.479+0000 7fa7b7e00640 1 --2- 172.21.15.28:0/2754792416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7b806e870 0x7fa7b815d5b0 unknown :-1 s=CLOSED pgs=2771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:53.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.479+0000 7fa7b7e00640 1 --2- 172.21.15.28:0/2754792416 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa7b80c1010 0x7fa7b806e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:53.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.479+0000 7fa7b7e00640 1 -- 172.21.15.28:0/2754792416 >> 172.21.15.28:0/2754792416 conn(0x7fa7b80c6610 msgr2=0x7fa7b80c69f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:53.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.479+0000 7fa7b7e00640 1 -- 172.21.15.28:0/2754792416 shutdown_connections 2024-02-21T23:17:53.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.479+0000 7fa7b7e00640 1 -- 172.21.15.28:0/2754792416 wait complete. 2024-02-21T23:17:53.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.480+0000 7fa7b7e00640 1 Processor -- start 2024-02-21T23:17:53.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.481+0000 7fa7b7e00640 1 -- start start 2024-02-21T23:17:53.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.481+0000 7fa7b7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7b806e870 0x7fa7b814c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:53.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.481+0000 7fa7b7e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa7b80c1010 0x7fa7b8072a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:53.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.481+0000 7fa7b7e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa7b81501c0 0x7fa7b80c5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:53.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.481+0000 7fa7b7e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b80c7340 con 0x7fa7b806e870 2024-02-21T23:17:53.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.481+0000 7fa7b7e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b80c0c20 con 0x7fa7b80c1010 2024-02-21T23:17:53.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.481+0000 7fa7b7e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b81505a0 con 0x7fa7b81501c0 2024-02-21T23:17:53.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.482+0000 7fa7b6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7b806e870 0x7fa7b814c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:53.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.482+0000 7fa7b6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7b806e870 0x7fa7b814c0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35356/0 (socket says 172.21.15.28:35356) 2024-02-21T23:17:53.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.482+0000 7fa7b6a00640 1 -- 172.21.15.28:0/3780549321 learned_addr learned my addr 172.21.15.28:0/3780549321 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:53.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.482+0000 7fa7b6a00640 1 -- 172.21.15.28:0/3780549321 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa7b81501c0 msgr2=0x7fa7b80c5550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:53.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.482+0000 7fa7b6a00640 1 --2- 172.21.15.28:0/3780549321 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fa7b81501c0 0x7fa7b80c5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:53.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.482+0000 7fa7b6a00640 1 -- 172.21.15.28:0/3780549321 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa7b80c1010 msgr2=0x7fa7b8072a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:53.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.482+0000 7fa7b6a00640 1 --2- 172.21.15.28:0/3780549321 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa7b80c1010 0x7fa7b8072a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:53.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.482+0000 7fa7b6a00640 1 -- 172.21.15.28:0/3780549321 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7b8152c60 con 0x7fa7b806e870 2024-02-21T23:17:53.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.483+0000 7fa7b6a00640 1 --2- 172.21.15.28:0/3780549321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7b806e870 0x7fa7b814c0a0 secure :-1 s=READY pgs=2772 cs=0 l=1 rev1=1 crypto rx=0x7fa7a800c450 tx=0x7fa7a800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:53.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.483+0000 7fa7aea00640 1 -- 172.21.15.28:0/3780549321 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7a80052d0 con 0x7fa7b806e870 2024-02-21T23:17:53.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.483+0000 7fa7aea00640 1 -- 172.21.15.28:0/3780549321 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7a8005470 con 0x7fa7b806e870 2024-02-21T23:17:53.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.483+0000 7fa7aea00640 1 -- 172.21.15.28:0/3780549321 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7a8012870 con 0x7fa7b806e870 2024-02-21T23:17:53.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.483+0000 7fa7b7e00640 1 -- 172.21.15.28:0/3780549321 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7b81e6fc0 con 0x7fa7b806e870 2024-02-21T23:17:53.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.483+0000 7fa7b7e00640 1 -- 172.21.15.28:0/3780549321 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7b8071120 con 0x7fa7b806e870 2024-02-21T23:17:53.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.484+0000 7fa7b7e00640 1 -- 172.21.15.28:0/3780549321 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa7b806f6b0 con 0x7fa7b806e870 2024-02-21T23:17:53.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.485+0000 7fa7aea00640 1 -- 172.21.15.28:0/3780549321 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa7a8012a10 con 0x7fa7b806e870 2024-02-21T23:17:53.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.486+0000 7fa7aea00640 1 --2- 172.21.15.28:0/3780549321 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa79406b4d0 0x7fa79406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:53.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.486+0000 7fa7aea00640 1 -- 172.21.15.28:0/3780549321 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 5593+0+0 (secure 0 0 0) 0x7fa7a8089db0 con 0x7fa7b806e870 2024-02-21T23:17:53.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.486+0000 7fa7aea00640 1 --2- 172.21.15.28:0/3780549321 >> [v2:172.21.15.28:6824/3053508071,v1:172.21.15.28:6825/3053508071] conn(0x7fa794071b50 0x7fa794073f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:53.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.486+0000 7fa7aea00640 1 -- 172.21.15.28:0/3780549321 --> [v2:172.21.15.28:6824/3053508071,v1:172.21.15.28:6825/3053508071] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa794074660 con 0x7fa794071b50 2024-02-21T23:17:53.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.486+0000 7fa7aea00640 1 -- 172.21.15.28:0/3780549321 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=215) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa7a808c3d0 con 0x7fa7b806e870 2024-02-21T23:17:53.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.487+0000 7fa7b7400640 1 --2- 172.21.15.28:0/3780549321 >> [v2:172.21.15.28:6824/3053508071,v1:172.21.15.28:6825/3053508071] conn(0x7fa794071b50 0x7fa794073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:53.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.487+0000 7fa7afe00640 1 --2- 172.21.15.28:0/3780549321 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa79406b4d0 0x7fa79406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:53.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.487+0000 7fa7b7400640 1 --2- 172.21.15.28:0/3780549321 >> [v2:172.21.15.28:6824/3053508071,v1:172.21.15.28:6825/3053508071] conn(0x7fa794071b50 0x7fa794073f90 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:53.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.488+0000 7fa7afe00640 1 --2- 172.21.15.28:0/3780549321 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa79406b4d0 0x7fa79406d990 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7fa7b80c0860 tx=0x7fa7a0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:53.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.488+0000 7fa7aea00640 1 -- 172.21.15.28:0/3780549321 <== osd.3 v2:172.21.15.28:6824/3053508071 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fa794074660 con 0x7fa794071b50 2024-02-21T23:17:53.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.504+0000 7fa7b7e00640 1 -- 172.21.15.28:0/3780549321 --> [v2:172.21.15.28:6824/3053508071,v1:172.21.15.28:6825/3053508071] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fa7b804f290 con 0x7fa794071b50 2024-02-21T23:17:53.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.504+0000 7fa7aea00640 1 -- 172.21.15.28:0/3780549321 <== osd.3 v2:172.21.15.28:6824/3053508071 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fa7b804f290 con 0x7fa794071b50 2024-02-21T23:17:53.491 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-21T23:17:53.492 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-02-21T23:17:53.492 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-21T23:17:53.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.504+0000 7fa7b7e00640 1 -- 172.21.15.28:0/3780549321 >> [v2:172.21.15.28:6824/3053508071,v1:172.21.15.28:6825/3053508071] conn(0x7fa794071b50 msgr2=0x7fa794073f90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:53.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.504+0000 7fa7b7e00640 1 --2- 172.21.15.28:0/3780549321 >> [v2:172.21.15.28:6824/3053508071,v1:172.21.15.28:6825/3053508071] conn(0x7fa794071b50 0x7fa794073f90 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:53.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.505+0000 7fa7b7e00640 1 -- 172.21.15.28:0/3780549321 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa79406b4d0 msgr2=0x7fa79406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:53.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.505+0000 7fa7b7e00640 1 --2- 172.21.15.28:0/3780549321 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fa79406b4d0 0x7fa79406d990 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7fa7b80c0860 tx=0x7fa7a0008040 comp rx=0 tx=0).stop 2024-02-21T23:17:53.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.505+0000 7fa7b7e00640 1 -- 172.21.15.28:0/3780549321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7b806e870 msgr2=0x7fa7b814c0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:53.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.505+0000 7fa7b7e00640 1 --2- 172.21.15.28:0/3780549321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7b806e870 0x7fa7b814c0a0 secure :-1 s=READY pgs=2772 cs=0 l=1 rev1=1 crypto rx=0x7fa7a800c450 tx=0x7fa7a800c910 comp rx=0 tx=0).stop 2024-02-21T23:17:53.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.505+0000 7fa7b7400640 1 -- 172.21.15.28:0/3780549321 reap_dead start 2024-02-21T23:17:53.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.505+0000 7fa7b7e00640 1 -- 172.21.15.28:0/3780549321 shutdown_connections 2024-02-21T23:17:53.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.505+0000 7fa7b7e00640 1 -- 172.21.15.28:0/3780549321 >> 172.21.15.28:0/3780549321 conn(0x7fa7b80c6610 msgr2=0x7fa7b80c69f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:53.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.505+0000 7fa7b7e00640 1 -- 172.21.15.28:0/3780549321 shutdown_connections 2024-02-21T23:17:53.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.505+0000 7fa7b7e00640 1 -- 172.21.15.28:0/3780549321 wait complete. 2024-02-21T23:17:53.511 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-02-21T23:17:53.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.734+0000 7f2f73e00640 1 Processor -- start 2024-02-21T23:17:53.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.734+0000 7f2f73e00640 1 -- start start 2024-02-21T23:17:53.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.735+0000 7f2f73e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2f7406d410 0x7f2f740c1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:53.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.735+0000 7f2f73e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f2f74155260 0x7f2f741556f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:53.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.736+0000 7f2f73e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f74072e90 con 0x7f2f74148700 2024-02-21T23:17:53.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.736+0000 7f2f73e00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f740c0ee0 con 0x7f2f74155260 2024-02-21T23:17:53.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.736+0000 7f2f73e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f74073100 con 0x7f2f7406d410 2024-02-21T23:17:53.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.736+0000 7f2f72a00640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f2f74148700 0x7f2f740c13e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.28:34048/0 (socket says 172.21.15.28:34048) 2024-02-21T23:17:53.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.736+0000 7f2f72a00640 1 -- 172.21.15.28:0/1284783821 learned_addr learned my addr 172.21.15.28:0/1284783821 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:17:53.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.736+0000 7f2f72000640 1 --2- 172.21.15.28:0/1284783821 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2f7406d410 0x7f2f740c1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:53.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.736+0000 7f2f73400640 1 --2- 172.21.15.28:0/1284783821 >> v2:172.21.15.81:3300/0 conn(0x7f2f74155260 0x7f2f741556f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:53.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.737+0000 7f2f71600640 1 -- 172.21.15.28:0/1284783821 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1643594623 0 0) 0x7f2f74072e90 con 0x7f2f74148700 2024-02-21T23:17:53.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.737+0000 7f2f71600640 1 -- 172.21.15.28:0/1284783821 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f58003520 con 0x7f2f74148700 2024-02-21T23:17:53.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.737+0000 7f2f73400640 1 -- 172.21.15.28:0/1284783821 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2f7406d410 msgr2=0x7f2f740c1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:53.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.737+0000 7f2f73400640 1 --2- 172.21.15.28:0/1284783821 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2f7406d410 0x7f2f740c1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:53.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.737+0000 7f2f73400640 1 -- 172.21.15.28:0/1284783821 >> v1:172.21.15.81:6789/0 conn(0x7f2f74148700 legacy=0x7f2f740c13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:53.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.737+0000 7f2f73400640 1 -- 172.21.15.28:0/1284783821 --> v2:172.21.15.81:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f740733e0 con 0x7f2f74155260 2024-02-21T23:17:53.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.737+0000 7f2f72000640 1 --2- 172.21.15.28:0/1284783821 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2f7406d410 0x7f2f740c1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-21T23:17:53.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.739+0000 7f2f73400640 1 --2- 172.21.15.28:0/1284783821 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2f74155260 0x7f2f741556f0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c0097a0 tx=0x7f2f6c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a209010f7227aebb server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:53.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.739+0000 7f2f71600640 1 -- 172.21.15.28:0/1284783821 <== mon.1 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f6c035020 con 0x7f2f74155260 2024-02-21T23:17:53.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.739+0000 7f2f71600640 1 -- 172.21.15.28:0/1284783821 <== mon.1 v2:172.21.15.81:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f6c031830 con 0x7f2f74155260 2024-02-21T23:17:53.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.739+0000 7f2f71600640 1 -- 172.21.15.28:0/1284783821 <== mon.1 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f6c031c00 con 0x7f2f74155260 2024-02-21T23:17:53.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.740+0000 7f2f73e00640 1 -- 172.21.15.28:0/1284783821 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2f74155260 msgr2=0x7f2f741556f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:53.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.740+0000 7f2f73e00640 1 --2- 172.21.15.28:0/1284783821 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2f74155260 0x7f2f741556f0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c0097a0 tx=0x7f2f6c02b8d0 comp rx=0 tx=0).stop 2024-02-21T23:17:53.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.740+0000 7f2f73e00640 1 -- 172.21.15.28:0/1284783821 shutdown_connections 2024-02-21T23:17:53.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.740+0000 7f2f73e00640 1 --2- 172.21.15.28:0/1284783821 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2f74155260 0x7f2f741556f0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:53.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.740+0000 7f2f73e00640 1 --2- 172.21.15.28:0/1284783821 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2f7406d410 0x7f2f740c1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:53.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.740+0000 7f2f73e00640 1 -- 172.21.15.28:0/1284783821 >> 172.21.15.28:0/1284783821 conn(0x7f2f740a8320 msgr2=0x7f2f740a8730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:53.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.740+0000 7f2f73e00640 1 -- 172.21.15.28:0/1284783821 shutdown_connections 2024-02-21T23:17:53.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.740+0000 7f2f73e00640 1 -- 172.21.15.28:0/1284783821 wait complete. 2024-02-21T23:17:53.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.742+0000 7f2f73e00640 1 Processor -- start 2024-02-21T23:17:53.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.743+0000 7f2f73e00640 1 -- start start 2024-02-21T23:17:53.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.744+0000 7f2f73e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2f7406d410 0x7f2f740bfa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:53.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.744+0000 7f2f73e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f74148700 0x7f2f741ead40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:53.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.744+0000 7f2f73e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2f74155260 0x7f2f741eb280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:53.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.744+0000 7f2f73e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f740a9080 con 0x7f2f74148700 2024-02-21T23:17:53.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.744+0000 7f2f73e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f74148ae0 con 0x7f2f7406d410 2024-02-21T23:17:53.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.744+0000 7f2f73e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f7406d9f0 con 0x7f2f74155260 2024-02-21T23:17:53.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.744+0000 7f2f72000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f74148700 0x7f2f741ead40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:53.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.744+0000 7f2f72000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f74148700 0x7f2f741ead40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35358/0 (socket says 172.21.15.28:35358) 2024-02-21T23:17:53.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.744+0000 7f2f72000640 1 -- 172.21.15.28:0/514632195 learned_addr learned my addr 172.21.15.28:0/514632195 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:53.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.745+0000 7f2f73400640 1 --2- 172.21.15.28:0/514632195 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2f74155260 0x7f2f741eb280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:53.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.745+0000 7f2f72a00640 1 --2- 172.21.15.28:0/514632195 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2f7406d410 0x7f2f740bfa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:53.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.745+0000 7f2f72000640 1 -- 172.21.15.28:0/514632195 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2f74155260 msgr2=0x7f2f741eb280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:53.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.745+0000 7f2f72000640 1 --2- 172.21.15.28:0/514632195 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2f74155260 0x7f2f741eb280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:53.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.745+0000 7f2f72000640 1 -- 172.21.15.28:0/514632195 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2f7406d410 msgr2=0x7f2f740bfa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:53.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.745+0000 7f2f72000640 1 --2- 172.21.15.28:0/514632195 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2f7406d410 0x7f2f740bfa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:53.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.745+0000 7f2f72000640 1 -- 172.21.15.28:0/514632195 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f7414c120 con 0x7f2f74148700 2024-02-21T23:17:53.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.745+0000 7f2f72000640 1 --2- 172.21.15.28:0/514632195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f74148700 0x7f2f741ead40 secure :-1 s=READY pgs=2773 cs=0 l=1 rev1=1 crypto rx=0x7f2f6000e960 tx=0x7f2f6000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:53.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.745+0000 7f2f6aa00640 1 -- 172.21.15.28:0/514632195 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f60017070 con 0x7f2f74148700 2024-02-21T23:17:53.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.746+0000 7f2f6aa00640 1 -- 172.21.15.28:0/514632195 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f6000ca00 con 0x7f2f74148700 2024-02-21T23:17:53.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.746+0000 7f2f73e00640 1 -- 172.21.15.28:0/514632195 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f7414b5a0 con 0x7f2f74148700 2024-02-21T23:17:53.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.746+0000 7f2f6aa00640 1 -- 172.21.15.28:0/514632195 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f6000cdd0 con 0x7f2f74148700 2024-02-21T23:17:53.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.746+0000 7f2f73e00640 1 -- 172.21.15.28:0/514632195 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f7414be40 con 0x7f2f74148700 2024-02-21T23:17:53.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.747+0000 7f2f73e00640 1 -- 172.21.15.28:0/514632195 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f740c73f0 con 0x7f2f74148700 2024-02-21T23:17:53.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.748+0000 7f2f6aa00640 1 -- 172.21.15.28:0/514632195 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2f6001a070 con 0x7f2f74148700 2024-02-21T23:17:53.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.749+0000 7f2f6aa00640 1 --2- 172.21.15.28:0/514632195 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2f5406b4d0 0x7f2f5406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:53.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.749+0000 7f2f6aa00640 1 -- 172.21.15.28:0/514632195 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f2f6008a070 con 0x7f2f74148700 2024-02-21T23:17:53.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.749+0000 7f2f72a00640 1 --2- 172.21.15.28:0/514632195 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2f5406b4d0 0x7f2f5406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:53.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.750+0000 7f2f72a00640 1 --2- 172.21.15.28:0/514632195 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2f5406b4d0 0x7f2f5406d990 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f2f64001560 tx=0x7f2f64009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:53.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.753+0000 7f2f6aa00640 1 -- 172.21.15.28:0/514632195 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f2f740c73f0 con 0x7f2f74148700 2024-02-21T23:17:53.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.957+0000 7f2f73e00640 1 -- 172.21.15.28:0/514632195 --> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f2f7404f290 con 0x7f2f5406b4d0 2024-02-21T23:17:53.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.958+0000 7f2f6aa00640 1 -- 172.21.15.28:0/514632195 <== mgr.4101 v2:172.21.15.81:6832/1848538577 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f2f7404f290 con 0x7f2f5406b4d0 2024-02-21T23:17:53.946 INFO:teuthology.orchestra.run.smithi028.stderr:instructed osd(s) 3 to deep-scrub 2024-02-21T23:17:53.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.961+0000 7f2f73e00640 1 -- 172.21.15.28:0/514632195 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2f5406b4d0 msgr2=0x7f2f5406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:53.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.961+0000 7f2f73e00640 1 --2- 172.21.15.28:0/514632195 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2f5406b4d0 0x7f2f5406d990 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f2f64001560 tx=0x7f2f64009040 comp rx=0 tx=0).stop 2024-02-21T23:17:53.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.961+0000 7f2f73e00640 1 -- 172.21.15.28:0/514632195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f74148700 msgr2=0x7f2f741ead40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:53.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.961+0000 7f2f73e00640 1 --2- 172.21.15.28:0/514632195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f74148700 0x7f2f741ead40 secure :-1 s=READY pgs=2773 cs=0 l=1 rev1=1 crypto rx=0x7f2f6000e960 tx=0x7f2f6000ee20 comp rx=0 tx=0).stop 2024-02-21T23:17:53.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.962+0000 7f2f73e00640 1 -- 172.21.15.28:0/514632195 shutdown_connections 2024-02-21T23:17:53.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.962+0000 7f2f73e00640 1 --2- 172.21.15.28:0/514632195 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f2f5406b4d0 0x7f2f5406d990 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:53.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.962+0000 7f2f73e00640 1 --2- 172.21.15.28:0/514632195 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f2f74155260 0x7f2f741eb280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:53.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.962+0000 7f2f73e00640 1 --2- 172.21.15.28:0/514632195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f74148700 0x7f2f741ead40 unknown :-1 s=CLOSED pgs=2773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:53.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.962+0000 7f2f73e00640 1 --2- 172.21.15.28:0/514632195 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2f7406d410 0x7f2f740bfa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:53.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.962+0000 7f2f73e00640 1 -- 172.21.15.28:0/514632195 >> 172.21.15.28:0/514632195 conn(0x7f2f740a8320 msgr2=0x7f2f74154290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:53.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.962+0000 7f2f73e00640 1 -- 172.21.15.28:0/514632195 shutdown_connections 2024-02-21T23:17:53.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:53.962+0000 7f2f73e00640 1 -- 172.21.15.28:0/514632195 wait complete. 2024-02-21T23:17:53.967 INFO:tasks.ceph:Scrubbing osd.4 2024-02-21T23:17:53.967 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-02-21T23:17:54.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.207+0000 7fe80be00640 1 Processor -- start 2024-02-21T23:17:54.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.207+0000 7fe80be00640 1 -- start start 2024-02-21T23:17:54.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.209+0000 7fe80be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe80c14f190 0x7fe80c06f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:54.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.209+0000 7fe80be00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe80c0c49e0 0x7fe80c072ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:54.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.209+0000 7fe80be00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe80c14f570 con 0x7fe80c06fec0 2024-02-21T23:17:54.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.209+0000 7fe80be00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe80c0c31d0 con 0x7fe80c14f190 2024-02-21T23:17:54.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.209+0000 7fe80be00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe80c0c34b0 con 0x7fe80c0c49e0 2024-02-21T23:17:54.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.210+0000 7fe80aa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe80c14f190 0x7fe80c06f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:54.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.210+0000 7fe80a000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe80c06fec0 0x7fe80c072590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:54408/0 (socket says 172.21.15.28:54408) 2024-02-21T23:17:54.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.210+0000 7fe80a000640 1 -- 172.21.15.28:0/984409847 learned_addr learned my addr 172.21.15.28:0/984409847 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:17:54.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.210+0000 7fe80aa00640 1 -- 172.21.15.28:0/984409847 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe80c0c49e0 msgr2=0x7fe80c072ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:54.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.210+0000 7fe80b400640 1 --2- 172.21.15.28:0/984409847 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe80c0c49e0 0x7fe80c072ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:54.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.210+0000 7fe80aa00640 1 --2- 172.21.15.28:0/984409847 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe80c0c49e0 0x7fe80c072ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:54.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.210+0000 7fe80aa00640 1 -- 172.21.15.28:0/984409847 >> v1:172.21.15.28:6789/0 conn(0x7fe80c06fec0 legacy=0x7fe80c072590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:54.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.210+0000 7fe80aa00640 1 -- 172.21.15.28:0/984409847 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe80c0c3790 con 0x7fe80c14f190 2024-02-21T23:17:54.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.210+0000 7fe809600640 1 -- 172.21.15.28:0/984409847 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4107196421 0 0) 0x7fe7f8001110 con 0x7fe80c06fec0 2024-02-21T23:17:54.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.211+0000 7fe80aa00640 1 --2- 172.21.15.28:0/984409847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe80c14f190 0x7fe80c06f980 secure :-1 s=READY pgs=2775 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc0097a0 tx=0x7fe7fc02b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc1b503e59f9024d server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:54.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.212+0000 7fe809600640 1 -- 172.21.15.28:0/984409847 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7fc039070 con 0x7fe80c14f190 2024-02-21T23:17:54.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.212+0000 7fe809600640 1 -- 172.21.15.28:0/984409847 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7fc0316f0 con 0x7fe80c14f190 2024-02-21T23:17:54.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.212+0000 7fe809600640 1 -- 172.21.15.28:0/984409847 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7fc031ac0 con 0x7fe80c14f190 2024-02-21T23:17:54.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.212+0000 7fe80be00640 1 -- 172.21.15.28:0/984409847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe80c14f190 msgr2=0x7fe80c06f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:54.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.212+0000 7fe80be00640 1 --2- 172.21.15.28:0/984409847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe80c14f190 0x7fe80c06f980 secure :-1 s=READY pgs=2775 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc0097a0 tx=0x7fe7fc02b790 comp rx=0 tx=0).stop 2024-02-21T23:17:54.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.212+0000 7fe80be00640 1 -- 172.21.15.28:0/984409847 shutdown_connections 2024-02-21T23:17:54.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.212+0000 7fe80be00640 1 --2- 172.21.15.28:0/984409847 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe80c0c49e0 0x7fe80c072ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:54.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.212+0000 7fe80be00640 1 --2- 172.21.15.28:0/984409847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe80c14f190 0x7fe80c06f980 unknown :-1 s=CLOSED pgs=2775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:54.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.212+0000 7fe80be00640 1 -- 172.21.15.28:0/984409847 >> 172.21.15.28:0/984409847 conn(0x7fe80c066e40 msgr2=0x7fe80c0a84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:54.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.213+0000 7fe80be00640 1 -- 172.21.15.28:0/984409847 shutdown_connections 2024-02-21T23:17:54.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.213+0000 7fe80be00640 1 -- 172.21.15.28:0/984409847 wait complete. 2024-02-21T23:17:54.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.214+0000 7fe80be00640 1 Processor -- start 2024-02-21T23:17:54.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.214+0000 7fe80be00640 1 -- start start 2024-02-21T23:17:54.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.215+0000 7fe80be00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe80c06fec0 0x7fe80c0c0d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:54.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.215+0000 7fe80be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe80c0c49e0 0x7fe80c1ea5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:54.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.215+0000 7fe80be00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe80c14f190 0x7fe80c1eab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:54.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.215+0000 7fe80be00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe80c0c4f30 con 0x7fe80c0c49e0 2024-02-21T23:17:54.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.215+0000 7fe80be00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe80c0731e0 con 0x7fe80c14f190 2024-02-21T23:17:54.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.215+0000 7fe80be00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe80c0c50b0 con 0x7fe80c06fec0 2024-02-21T23:17:54.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.215+0000 7fe80a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe80c0c49e0 0x7fe80c1ea5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:54.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.216+0000 7fe80a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe80c0c49e0 0x7fe80c1ea5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35374/0 (socket says 172.21.15.28:35374) 2024-02-21T23:17:54.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.216+0000 7fe80a000640 1 -- 172.21.15.28:0/3895428961 learned_addr learned my addr 172.21.15.28:0/3895428961 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:54.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.216+0000 7fe80a000640 1 -- 172.21.15.28:0/3895428961 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe80c06fec0 msgr2=0x7fe80c0c0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:54.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.216+0000 7fe80aa00640 1 --2- 172.21.15.28:0/3895428961 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe80c06fec0 0x7fe80c0c0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:54.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.216+0000 7fe80a000640 1 --2- 172.21.15.28:0/3895428961 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fe80c06fec0 0x7fe80c0c0d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:54.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.216+0000 7fe80a000640 1 -- 172.21.15.28:0/3895428961 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe80c14f190 msgr2=0x7fe80c1eab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:54.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.216+0000 7fe80a000640 1 --2- 172.21.15.28:0/3895428961 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe80c14f190 0x7fe80c1eab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:54.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.216+0000 7fe80a000640 1 -- 172.21.15.28:0/3895428961 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe80c141760 con 0x7fe80c0c49e0 2024-02-21T23:17:54.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.216+0000 7fe80a000640 1 --2- 172.21.15.28:0/3895428961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe80c0c49e0 0x7fe80c1ea5c0 secure :-1 s=READY pgs=2776 cs=0 l=1 rev1=1 crypto rx=0x7fe7f800c420 tx=0x7fe7f800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:54.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.216+0000 7fe802a00640 1 -- 172.21.15.28:0/3895428961 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f80074e0 con 0x7fe80c0c49e0 2024-02-21T23:17:54.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.217+0000 7fe802a00640 1 -- 172.21.15.28:0/3895428961 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7f80047e0 con 0x7fe80c0c49e0 2024-02-21T23:17:54.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.217+0000 7fe802a00640 1 -- 172.21.15.28:0/3895428961 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f8004c10 con 0x7fe80c0c49e0 2024-02-21T23:17:54.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.217+0000 7fe80be00640 1 -- 172.21.15.28:0/3895428961 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe80c1edbd0 con 0x7fe80c0c49e0 2024-02-21T23:17:54.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.217+0000 7fe80be00640 1 -- 172.21.15.28:0/3895428961 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe80c1ee190 con 0x7fe80c0c49e0 2024-02-21T23:17:54.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.218+0000 7fe80be00640 1 -- 172.21.15.28:0/3895428961 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe80c04f290 con 0x7fe80c0c49e0 2024-02-21T23:17:54.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.219+0000 7fe802a00640 1 -- 172.21.15.28:0/3895428961 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe7f8011070 con 0x7fe80c0c49e0 2024-02-21T23:17:54.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.220+0000 7fe802a00640 1 --2- 172.21.15.28:0/3895428961 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe7ec06b4d0 0x7fe7ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:54.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.220+0000 7fe802a00640 1 -- 172.21.15.28:0/3895428961 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 5593+0+0 (secure 0 0 0) 0x7fe7f8089d20 con 0x7fe80c0c49e0 2024-02-21T23:17:54.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.220+0000 7fe802a00640 1 --2- 172.21.15.28:0/3895428961 >> [v2:172.21.15.81:6819/2091269704,v1:172.21.15.81:6821/2091269704] conn(0x7fe7ec071b50 0x7fe7ec073f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:54.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.220+0000 7fe802a00640 1 -- 172.21.15.28:0/3895428961 --> [v2:172.21.15.81:6819/2091269704,v1:172.21.15.81:6821/2091269704] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe7f800ee50 con 0x7fe7ec071b50 2024-02-21T23:17:54.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.220+0000 7fe802a00640 1 -- 172.21.15.28:0/3895428961 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=215) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe7f808e030 con 0x7fe80c0c49e0 2024-02-21T23:17:54.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.220+0000 7fe80aa00640 1 --2- 172.21.15.28:0/3895428961 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe7ec06b4d0 0x7fe7ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:54.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.221+0000 7fe80b400640 1 --2- 172.21.15.28:0/3895428961 >> [v2:172.21.15.81:6819/2091269704,v1:172.21.15.81:6821/2091269704] conn(0x7fe7ec071b50 0x7fe7ec073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:54.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.222+0000 7fe80aa00640 1 --2- 172.21.15.28:0/3895428961 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe7ec06b4d0 0x7fe7ec06d990 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc036000 tx=0x7fe7fc008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:54.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.222+0000 7fe80b400640 1 --2- 172.21.15.28:0/3895428961 >> [v2:172.21.15.81:6819/2091269704,v1:172.21.15.81:6821/2091269704] conn(0x7fe7ec071b50 0x7fe7ec073f90 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:54.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.224+0000 7fe802a00640 1 -- 172.21.15.28:0/3895428961 <== osd.4 v2:172.21.15.81:6819/2091269704 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fe7f800ee50 con 0x7fe7ec071b50 2024-02-21T23:17:54.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.239+0000 7fe80be00640 1 -- 172.21.15.28:0/3895428961 --> [v2:172.21.15.81:6819/2091269704,v1:172.21.15.81:6821/2091269704] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fe80c0c47c0 con 0x7fe7ec071b50 2024-02-21T23:17:54.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.240+0000 7fe802a00640 1 -- 172.21.15.28:0/3895428961 <== osd.4 v2:172.21.15.81:6819/2091269704 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fe80c0c47c0 con 0x7fe7ec071b50 2024-02-21T23:17:54.228 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-21T23:17:54.228 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-02-21T23:17:54.228 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-21T23:17:54.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.241+0000 7fe80be00640 1 -- 172.21.15.28:0/3895428961 >> [v2:172.21.15.81:6819/2091269704,v1:172.21.15.81:6821/2091269704] conn(0x7fe7ec071b50 msgr2=0x7fe7ec073f90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:54.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.241+0000 7fe80be00640 1 --2- 172.21.15.28:0/3895428961 >> [v2:172.21.15.81:6819/2091269704,v1:172.21.15.81:6821/2091269704] conn(0x7fe7ec071b50 0x7fe7ec073f90 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:54.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.241+0000 7fe80be00640 1 -- 172.21.15.28:0/3895428961 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe7ec06b4d0 msgr2=0x7fe7ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:54.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.241+0000 7fe80be00640 1 --2- 172.21.15.28:0/3895428961 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fe7ec06b4d0 0x7fe7ec06d990 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc036000 tx=0x7fe7fc008030 comp rx=0 tx=0).stop 2024-02-21T23:17:54.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.241+0000 7fe80be00640 1 -- 172.21.15.28:0/3895428961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe80c0c49e0 msgr2=0x7fe80c1ea5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:54.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.241+0000 7fe80be00640 1 --2- 172.21.15.28:0/3895428961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe80c0c49e0 0x7fe80c1ea5c0 secure :-1 s=READY pgs=2776 cs=0 l=1 rev1=1 crypto rx=0x7fe7f800c420 tx=0x7fe7f800c8e0 comp rx=0 tx=0).stop 2024-02-21T23:17:54.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.241+0000 7fe80b400640 1 -- 172.21.15.28:0/3895428961 reap_dead start 2024-02-21T23:17:54.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.241+0000 7fe80be00640 1 -- 172.21.15.28:0/3895428961 shutdown_connections 2024-02-21T23:17:54.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.241+0000 7fe80be00640 1 -- 172.21.15.28:0/3895428961 >> 172.21.15.28:0/3895428961 conn(0x7fe80c066e40 msgr2=0x7fe80c0a84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:54.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.242+0000 7fe80be00640 1 -- 172.21.15.28:0/3895428961 shutdown_connections 2024-02-21T23:17:54.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.242+0000 7fe80be00640 1 -- 172.21.15.28:0/3895428961 wait complete. 2024-02-21T23:17:54.246 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-02-21T23:17:54.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.475+0000 7fb923400640 1 Processor -- start 2024-02-21T23:17:54.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.475+0000 7fb923400640 1 -- start start 2024-02-21T23:17:54.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.476+0000 7fb923400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb91c156c80 0x7fb91c06e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:54.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.476+0000 7fb923400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb91c159fd0 con 0x7fb91c072e60 2024-02-21T23:17:54.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.476+0000 7fb923400640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb91c0c5db0 con 0x7fb91c0c11f0 2024-02-21T23:17:54.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.476+0000 7fb923400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb91c0c5b60 con 0x7fb91c156c80 2024-02-21T23:17:54.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.477+0000 7fb91be00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb91c072e60 0x7fb91c0c4ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:54416/0 (socket says 172.21.15.28:54416) 2024-02-21T23:17:54.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.477+0000 7fb91be00640 1 -- 172.21.15.28:0/1200838249 learned_addr learned my addr 172.21.15.28:0/1200838249 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:17:54.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.477+0000 7fb920a00640 1 --2- 172.21.15.28:0/1200838249 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb91c156c80 0x7fb91c06e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:54.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.477+0000 7fb91b400640 1 -- 172.21.15.28:0/1200838249 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 537470321 0 0) 0x7fb91c159fd0 con 0x7fb91c072e60 2024-02-21T23:17:54.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.477+0000 7fb91b400640 1 -- 172.21.15.28:0/1200838249 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb900003520 con 0x7fb91c072e60 2024-02-21T23:17:54.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.478+0000 7fb91b400640 1 -- 172.21.15.28:0/1200838249 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4214338127 0 0) 0x7fb900003520 con 0x7fb91c072e60 2024-02-21T23:17:54.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.478+0000 7fb91b400640 1 -- 172.21.15.28:0/1200838249 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb91c156c80 msgr2=0x7fb91c06e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:54.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.478+0000 7fb91b400640 1 --2- 172.21.15.28:0/1200838249 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb91c156c80 0x7fb91c06e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:54.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.478+0000 7fb91b400640 1 -- 172.21.15.28:0/1200838249 >> v1:172.21.15.81:6789/0 conn(0x7fb91c0c11f0 legacy=0x7fb91c0c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:54.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.478+0000 7fb91b400640 1 -- 172.21.15.28:0/1200838249 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb91c15a290 con 0x7fb91c072e60 2024-02-21T23:17:54.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.478+0000 7fb91b400640 1 -- 172.21.15.28:0/1200838249 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 389107730 0 0) 0x7fb908001670 con 0x7fb91c072e60 2024-02-21T23:17:54.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.478+0000 7fb91b400640 1 -- 172.21.15.28:0/1200838249 >> v1:172.21.15.28:6789/0 conn(0x7fb91c072e60 legacy=0x7fb91c0c4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:54.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.478+0000 7fb91b400640 1 --2- 172.21.15.28:0/1200838249 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9000039e0 0x7fb900003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:54.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.478+0000 7fb91b400640 1 --2- 172.21.15.28:0/1200838249 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb900004800 0x7fb900004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:54.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.478+0000 7fb91b400640 1 --2- 172.21.15.28:0/1200838249 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb91c156c80 0x7fb9000051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:54.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.478+0000 7fb91b400640 1 -- 172.21.15.28:0/1200838249 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb91c0c5b60 con 0x7fb9000039e0 2024-02-21T23:17:54.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.478+0000 7fb91b400640 1 -- 172.21.15.28:0/1200838249 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb90000bac0 con 0x7fb900004800 2024-02-21T23:17:54.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.478+0000 7fb91b400640 1 -- 172.21.15.28:0/1200838249 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb90000bcc0 con 0x7fb91c156c80 2024-02-21T23:17:54.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.478+0000 7fb920a00640 1 --2- 172.21.15.28:0/1200838249 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9000039e0 0x7fb900003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:54.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.478+0000 7fb91be00640 1 --2- 172.21.15.28:0/1200838249 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb900004800 0x7fb900004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:54.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.478+0000 7fb920a00640 1 -- 172.21.15.28:0/1200838249 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb91c156c80 msgr2=0x7fb9000051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:54.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.478+0000 7fb920a00640 1 --2- 172.21.15.28:0/1200838249 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb91c156c80 0x7fb9000051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:54.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.479+0000 7fb920a00640 1 -- 172.21.15.28:0/1200838249 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb900004800 msgr2=0x7fb900004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:54.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.479+0000 7fb920a00640 1 --2- 172.21.15.28:0/1200838249 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb900004800 0x7fb900004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:54.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.479+0000 7fb920a00640 1 -- 172.21.15.28:0/1200838249 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb900003520 con 0x7fb9000039e0 2024-02-21T23:17:54.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.479+0000 7fb91be00640 1 --2- 172.21.15.28:0/1200838249 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb900004800 0x7fb900004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:17:54.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.480+0000 7fb920a00640 1 --2- 172.21.15.28:0/1200838249 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9000039e0 0x7fb900003e20 secure :-1 s=READY pgs=2778 cs=0 l=1 rev1=1 crypto rx=0x7fb90c009ac0 tx=0x7fb900005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da901f24ed9c6060 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:54.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.480+0000 7fb91b400640 1 -- 172.21.15.28:0/1200838249 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb90c039070 con 0x7fb9000039e0 2024-02-21T23:17:54.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.480+0000 7fb91b400640 1 -- 172.21.15.28:0/1200838249 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb90c02c710 con 0x7fb9000039e0 2024-02-21T23:17:54.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.480+0000 7fb91b400640 1 -- 172.21.15.28:0/1200838249 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb90c009690 con 0x7fb9000039e0 2024-02-21T23:17:54.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.480+0000 7fb923400640 1 -- 172.21.15.28:0/1200838249 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9000039e0 msgr2=0x7fb900003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:54.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.480+0000 7fb923400640 1 --2- 172.21.15.28:0/1200838249 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9000039e0 0x7fb900003e20 secure :-1 s=READY pgs=2778 cs=0 l=1 rev1=1 crypto rx=0x7fb90c009ac0 tx=0x7fb900005830 comp rx=0 tx=0).stop 2024-02-21T23:17:54.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.481+0000 7fb921400640 1 -- 172.21.15.28:0/1200838249 reap_dead start 2024-02-21T23:17:54.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.481+0000 7fb923400640 1 -- 172.21.15.28:0/1200838249 shutdown_connections 2024-02-21T23:17:54.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.481+0000 7fb923400640 1 -- 172.21.15.28:0/1200838249 >> 172.21.15.28:0/1200838249 conn(0x7fb91c06b820 msgr2=0x7fb91c0c67f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:54.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.481+0000 7fb923400640 1 -- 172.21.15.28:0/1200838249 shutdown_connections 2024-02-21T23:17:54.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.481+0000 7fb923400640 1 -- 172.21.15.28:0/1200838249 wait complete. 2024-02-21T23:17:54.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.482+0000 7fb923400640 1 Processor -- start 2024-02-21T23:17:54.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.483+0000 7fb923400640 1 -- start start 2024-02-21T23:17:54.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.483+0000 7fb923400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9000039e0 0x7fb91c0c2be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:54.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.483+0000 7fb923400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb91c0c37b0 0x7fb91c1e5e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:54.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.483+0000 7fb923400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb91c1f3000 0x7fb91c1ef220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:54.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.483+0000 7fb923400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb91c06e840 con 0x7fb9000039e0 2024-02-21T23:17:54.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.483+0000 7fb923400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb91c0c0e90 con 0x7fb91c0c37b0 2024-02-21T23:17:54.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.483+0000 7fb923400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb91c06e9c0 con 0x7fb91c1f3000 2024-02-21T23:17:54.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.484+0000 7fb920a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9000039e0 0x7fb91c0c2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:54.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.484+0000 7fb920a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9000039e0 0x7fb91c0c2be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35380/0 (socket says 172.21.15.28:35380) 2024-02-21T23:17:54.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.484+0000 7fb920a00640 1 -- 172.21.15.28:0/3186245793 learned_addr learned my addr 172.21.15.28:0/3186245793 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:54.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.484+0000 7fb921400640 1 --2- 172.21.15.28:0/3186245793 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb91c1f3000 0x7fb91c1ef220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:54.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.484+0000 7fb91be00640 1 --2- 172.21.15.28:0/3186245793 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb91c0c37b0 0x7fb91c1e5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:54.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.484+0000 7fb920a00640 1 -- 172.21.15.28:0/3186245793 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb91c1f3000 msgr2=0x7fb91c1ef220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:54.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.484+0000 7fb920a00640 1 --2- 172.21.15.28:0/3186245793 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb91c1f3000 0x7fb91c1ef220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:54.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.484+0000 7fb920a00640 1 -- 172.21.15.28:0/3186245793 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb91c0c37b0 msgr2=0x7fb91c1e5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:54.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.484+0000 7fb920a00640 1 --2- 172.21.15.28:0/3186245793 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb91c0c37b0 0x7fb91c1e5e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:54.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.484+0000 7fb920a00640 1 -- 172.21.15.28:0/3186245793 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb91c0c3120 con 0x7fb9000039e0 2024-02-21T23:17:54.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.485+0000 7fb920a00640 1 --2- 172.21.15.28:0/3186245793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9000039e0 0x7fb91c0c2be0 secure :-1 s=READY pgs=2779 cs=0 l=1 rev1=1 crypto rx=0x7fb90c037040 tx=0x7fb90c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:54.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.485+0000 7fb918c00640 1 -- 172.21.15.28:0/3186245793 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb90c039050 con 0x7fb9000039e0 2024-02-21T23:17:54.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.485+0000 7fb918c00640 1 -- 172.21.15.28:0/3186245793 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb90c045070 con 0x7fb9000039e0 2024-02-21T23:17:54.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.485+0000 7fb918c00640 1 -- 172.21.15.28:0/3186245793 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb90c003a70 con 0x7fb9000039e0 2024-02-21T23:17:54.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.485+0000 7fb923400640 1 -- 172.21.15.28:0/3186245793 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb91c0a8290 con 0x7fb9000039e0 2024-02-21T23:17:54.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.485+0000 7fb923400640 1 -- 172.21.15.28:0/3186245793 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb91c0aa9e0 con 0x7fb9000039e0 2024-02-21T23:17:54.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.486+0000 7fb923400640 1 -- 172.21.15.28:0/3186245793 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb91c06f400 con 0x7fb9000039e0 2024-02-21T23:17:54.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.487+0000 7fb918c00640 1 -- 172.21.15.28:0/3186245793 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb90c003c10 con 0x7fb9000039e0 2024-02-21T23:17:54.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.488+0000 7fb918c00640 1 --2- 172.21.15.28:0/3186245793 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb8fc06b4d0 0x7fb8fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:54.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.488+0000 7fb918c00640 1 -- 172.21.15.28:0/3186245793 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 5593+0+0 (secure 0 0 0) 0x7fb90c0abc10 con 0x7fb9000039e0 2024-02-21T23:17:54.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.489+0000 7fb91be00640 1 --2- 172.21.15.28:0/3186245793 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb8fc06b4d0 0x7fb8fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:54.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.490+0000 7fb91be00640 1 --2- 172.21.15.28:0/3186245793 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb8fc06b4d0 0x7fb8fc06d990 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7fb91c1f0350 tx=0x7fb908009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:54.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.493+0000 7fb918c00640 1 -- 172.21.15.28:0/3186245793 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fb90c04e020 con 0x7fb9000039e0 2024-02-21T23:17:54.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.690+0000 7fb923400640 1 -- 172.21.15.28:0/3186245793 --> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7fb91c04f370 con 0x7fb8fc06b4d0 2024-02-21T23:17:54.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.691+0000 7fb918c00640 1 -- 172.21.15.28:0/3186245793 <== mgr.4101 v2:172.21.15.81:6832/1848538577 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fb91c04f370 con 0x7fb8fc06b4d0 2024-02-21T23:17:54.679 INFO:teuthology.orchestra.run.smithi028.stderr:instructed osd(s) 4 to deep-scrub 2024-02-21T23:17:54.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.694+0000 7fb923400640 1 -- 172.21.15.28:0/3186245793 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb8fc06b4d0 msgr2=0x7fb8fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:54.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.694+0000 7fb923400640 1 --2- 172.21.15.28:0/3186245793 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb8fc06b4d0 0x7fb8fc06d990 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7fb91c1f0350 tx=0x7fb908009040 comp rx=0 tx=0).stop 2024-02-21T23:17:54.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.694+0000 7fb923400640 1 -- 172.21.15.28:0/3186245793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9000039e0 msgr2=0x7fb91c0c2be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:54.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.694+0000 7fb923400640 1 --2- 172.21.15.28:0/3186245793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9000039e0 0x7fb91c0c2be0 secure :-1 s=READY pgs=2779 cs=0 l=1 rev1=1 crypto rx=0x7fb90c037040 tx=0x7fb90c004730 comp rx=0 tx=0).stop 2024-02-21T23:17:54.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.694+0000 7fb923400640 1 -- 172.21.15.28:0/3186245793 shutdown_connections 2024-02-21T23:17:54.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.694+0000 7fb923400640 1 --2- 172.21.15.28:0/3186245793 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb8fc06b4d0 0x7fb8fc06d990 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:54.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.694+0000 7fb923400640 1 --2- 172.21.15.28:0/3186245793 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb91c1f3000 0x7fb91c1ef220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:54.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.694+0000 7fb923400640 1 --2- 172.21.15.28:0/3186245793 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb91c0c37b0 0x7fb91c1e5e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:54.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.694+0000 7fb923400640 1 --2- 172.21.15.28:0/3186245793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9000039e0 0x7fb91c0c2be0 unknown :-1 s=CLOSED pgs=2779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:54.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.694+0000 7fb923400640 1 -- 172.21.15.28:0/3186245793 >> 172.21.15.28:0/3186245793 conn(0x7fb91c06b820 msgr2=0x7fb91c0713b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:54.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.695+0000 7fb923400640 1 -- 172.21.15.28:0/3186245793 shutdown_connections 2024-02-21T23:17:54.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.695+0000 7fb923400640 1 -- 172.21.15.28:0/3186245793 wait complete. 2024-02-21T23:17:54.701 INFO:tasks.ceph:Scrubbing osd.5 2024-02-21T23:17:54.702 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-02-21T23:17:54.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.945+0000 7f7e19200640 1 Processor -- start 2024-02-21T23:17:54.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.945+0000 7f7e19200640 1 -- start start 2024-02-21T23:17:54.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.947+0000 7f7e19200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7e1406cfa0 0x7f7e140ca540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:54.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.947+0000 7f7e19200640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f7e14158460 0x7f7e140c6290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:54.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.947+0000 7f7e19200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e140caf40 con 0x7f7e140c8f70 2024-02-21T23:17:54.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.947+0000 7f7e19200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e140c8ae0 con 0x7f7e1406cfa0 2024-02-21T23:17:54.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.947+0000 7f7e19200640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e140c87b0 con 0x7f7e14158460 2024-02-21T23:17:54.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.947+0000 7f7e13400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7e1406cfa0 0x7f7e140ca540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:54.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.947+0000 7f7e13400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7e1406cfa0 0x7f7e140ca540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49562/0 (socket says 172.21.15.28:49562) 2024-02-21T23:17:54.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.947+0000 7f7e13400640 1 -- 172.21.15.28:0/1467649427 learned_addr learned my addr 172.21.15.28:0/1467649427 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:54.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.947+0000 7f7e0be00640 1 --2- 172.21.15.28:0/1467649427 >> v2:172.21.15.81:3300/0 conn(0x7f7e14158460 0x7f7e140c6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:54.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.948+0000 7f7e13400640 1 -- 172.21.15.28:0/1467649427 >> v2:172.21.15.81:3300/0 conn(0x7f7e14158460 msgr2=0x7f7e140c6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:54.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.948+0000 7f7e13400640 1 --2- 172.21.15.28:0/1467649427 >> v2:172.21.15.81:3300/0 conn(0x7f7e14158460 0x7f7e140c6290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:54.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.948+0000 7f7e13400640 1 -- 172.21.15.28:0/1467649427 >> v1:172.21.15.28:6789/0 conn(0x7f7e140c8f70 legacy=0x7f7e141522c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:54.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.948+0000 7f7e13400640 1 -- 172.21.15.28:0/1467649427 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e141588f0 con 0x7f7e1406cfa0 2024-02-21T23:17:54.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.949+0000 7f7e13400640 1 --2- 172.21.15.28:0/1467649427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e1406cfa0 0x7f7e140ca540 secure :-1 s=READY pgs=2781 cs=0 l=1 rev1=1 crypto rx=0x7f7e1406e0c0 tx=0x7f7e0402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f9dd70d08ad8b7e server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:54.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.949+0000 7f7e12a00640 1 -- 172.21.15.28:0/1467649427 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e0402e070 con 0x7f7e1406cfa0 2024-02-21T23:17:54.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.949+0000 7f7e12a00640 1 -- 172.21.15.28:0/1467649427 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e04037070 con 0x7f7e1406cfa0 2024-02-21T23:17:54.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.949+0000 7f7e12a00640 1 -- 172.21.15.28:0/1467649427 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e04031a00 con 0x7f7e1406cfa0 2024-02-21T23:17:54.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.950+0000 7f7e19200640 1 -- 172.21.15.28:0/1467649427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e1406cfa0 msgr2=0x7f7e140ca540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:54.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.950+0000 7f7e19200640 1 --2- 172.21.15.28:0/1467649427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e1406cfa0 0x7f7e140ca540 secure :-1 s=READY pgs=2781 cs=0 l=1 rev1=1 crypto rx=0x7f7e1406e0c0 tx=0x7f7e0402b880 comp rx=0 tx=0).stop 2024-02-21T23:17:54.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.950+0000 7f7e19200640 1 -- 172.21.15.28:0/1467649427 shutdown_connections 2024-02-21T23:17:54.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.950+0000 7f7e19200640 1 --2- 172.21.15.28:0/1467649427 >> v2:172.21.15.81:3300/0 conn(0x7f7e14158460 0x7f7e140c6290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:54.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.950+0000 7f7e19200640 1 --2- 172.21.15.28:0/1467649427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e1406cfa0 0x7f7e140ca540 unknown :-1 s=CLOSED pgs=2781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:54.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.950+0000 7f7e19200640 1 -- 172.21.15.28:0/1467649427 >> 172.21.15.28:0/1467649427 conn(0x7f7e140725a0 msgr2=0x7f7e140729b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:54.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.950+0000 7f7e19200640 1 -- 172.21.15.28:0/1467649427 shutdown_connections 2024-02-21T23:17:54.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.950+0000 7f7e19200640 1 -- 172.21.15.28:0/1467649427 wait complete. 2024-02-21T23:17:54.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.952+0000 7f7e19200640 1 Processor -- start 2024-02-21T23:17:54.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.952+0000 7f7e19200640 1 -- start start 2024-02-21T23:17:54.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.953+0000 7f7e19200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e1406cfa0 0x7f7e14153fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:54.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.953+0000 7f7e19200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7e140c8f70 0x7f7e140ae720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:54.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.953+0000 7f7e19200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e14158460 0x7f7e140aec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:54.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.953+0000 7f7e19200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e140c6c90 con 0x7f7e1406cfa0 2024-02-21T23:17:54.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.953+0000 7f7e19200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e140c69b0 con 0x7f7e14158460 2024-02-21T23:17:54.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.953+0000 7f7e19200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e141529d0 con 0x7f7e140c8f70 2024-02-21T23:17:54.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.953+0000 7f7e13400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e1406cfa0 0x7f7e14153fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:54.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.953+0000 7f7e13400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e1406cfa0 0x7f7e14153fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49570/0 (socket says 172.21.15.28:49570) 2024-02-21T23:17:54.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.953+0000 7f7e13400640 1 -- 172.21.15.28:0/1913139361 learned_addr learned my addr 172.21.15.28:0/1913139361 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:54.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.954+0000 7f7e13e00640 1 --2- 172.21.15.28:0/1913139361 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e14158460 0x7f7e140aec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:54.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.954+0000 7f7e13400640 1 -- 172.21.15.28:0/1913139361 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7e140c8f70 msgr2=0x7f7e140ae720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:54.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.954+0000 7f7e0be00640 1 --2- 172.21.15.28:0/1913139361 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7e140c8f70 0x7f7e140ae720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:54.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.954+0000 7f7e13400640 1 --2- 172.21.15.28:0/1913139361 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7e140c8f70 0x7f7e140ae720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:54.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.954+0000 7f7e13400640 1 -- 172.21.15.28:0/1913139361 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e14158460 msgr2=0x7f7e140aec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:54.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.954+0000 7f7e13400640 1 --2- 172.21.15.28:0/1913139361 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e14158460 0x7f7e140aec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:54.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.954+0000 7f7e13400640 1 -- 172.21.15.28:0/1913139361 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e141de7a0 con 0x7f7e1406cfa0 2024-02-21T23:17:54.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.954+0000 7f7e13e00640 1 --2- 172.21.15.28:0/1913139361 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7e14158460 0x7f7e140aec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:17:54.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.954+0000 7f7e0be00640 1 --2- 172.21.15.28:0/1913139361 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f7e140c8f70 0x7f7e140ae720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:17:54.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.954+0000 7f7e13400640 1 --2- 172.21.15.28:0/1913139361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e1406cfa0 0x7f7e14153fa0 secure :-1 s=READY pgs=2782 cs=0 l=1 rev1=1 crypto rx=0x7f7e0402b8c0 tx=0x7f7e040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:54.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.954+0000 7f7e0b400640 1 -- 172.21.15.28:0/1913139361 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e04032670 con 0x7f7e1406cfa0 2024-02-21T23:17:54.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.954+0000 7f7e0b400640 1 -- 172.21.15.28:0/1913139361 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e04032850 con 0x7f7e1406cfa0 2024-02-21T23:17:54.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.954+0000 7f7e0b400640 1 -- 172.21.15.28:0/1913139361 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e04032c40 con 0x7f7e1406cfa0 2024-02-21T23:17:54.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.954+0000 7f7e19200640 1 -- 172.21.15.28:0/1913139361 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e140aa8b0 con 0x7f7e1406cfa0 2024-02-21T23:17:54.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.955+0000 7f7e19200640 1 -- 172.21.15.28:0/1913139361 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e140aab50 con 0x7f7e1406cfa0 2024-02-21T23:17:54.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.956+0000 7f7e19200640 1 -- 172.21.15.28:0/1913139361 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7e1404f290 con 0x7f7e1406cfa0 2024-02-21T23:17:54.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.957+0000 7f7e0b400640 1 -- 172.21.15.28:0/1913139361 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7e040453f0 con 0x7f7e1406cfa0 2024-02-21T23:17:54.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.957+0000 7f7e0b400640 1 --2- 172.21.15.28:0/1913139361 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7df006b4d0 0x7f7df006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:54.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.958+0000 7f7e0b400640 1 -- 172.21.15.28:0/1913139361 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f7e040abfa0 con 0x7f7e1406cfa0 2024-02-21T23:17:54.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.958+0000 7f7e0b400640 1 --2- 172.21.15.28:0/1913139361 >> [v2:172.21.15.81:6808/568898662,v1:172.21.15.81:6809/568898662] conn(0x7f7df0071b50 0x7f7df0073f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:54.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.958+0000 7f7e0b400640 1 -- 172.21.15.28:0/1913139361 --> [v2:172.21.15.81:6808/568898662,v1:172.21.15.81:6809/568898662] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7e04008f00 con 0x7f7df0071b50 2024-02-21T23:17:54.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.958+0000 7f7e0b400640 1 -- 172.21.15.28:0/1913139361 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=215) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7e040ae5c0 con 0x7f7e1406cfa0 2024-02-21T23:17:54.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.958+0000 7f7e0be00640 1 --2- 172.21.15.28:0/1913139361 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7df006b4d0 0x7f7df006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:54.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.958+0000 7f7e13e00640 1 --2- 172.21.15.28:0/1913139361 >> [v2:172.21.15.81:6808/568898662,v1:172.21.15.81:6809/568898662] conn(0x7f7df0071b50 0x7f7df0073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:54.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.960+0000 7f7e0be00640 1 --2- 172.21.15.28:0/1913139361 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7df006b4d0 0x7f7df006d990 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f7dfc006fd0 tx=0x7f7dfc008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:54.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.960+0000 7f7e13e00640 1 --2- 172.21.15.28:0/1913139361 >> [v2:172.21.15.81:6808/568898662,v1:172.21.15.81:6809/568898662] conn(0x7f7df0071b50 0x7f7df0073f90 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:54.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.962+0000 7f7e0b400640 1 -- 172.21.15.28:0/1913139361 <== osd.5 v2:172.21.15.81:6808/568898662 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f7e04008f00 con 0x7f7df0071b50 2024-02-21T23:17:54.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.977+0000 7f7e19200640 1 -- 172.21.15.28:0/1913139361 --> [v2:172.21.15.81:6808/568898662,v1:172.21.15.81:6809/568898662] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f7e14072980 con 0x7f7df0071b50 2024-02-21T23:17:54.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.978+0000 7f7e0b400640 1 -- 172.21.15.28:0/1913139361 <== osd.5 v2:172.21.15.81:6808/568898662 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f7e14072980 con 0x7f7df0071b50 2024-02-21T23:17:54.965 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-21T23:17:54.965 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-02-21T23:17:54.965 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-21T23:17:54.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.978+0000 7f7e19200640 1 -- 172.21.15.28:0/1913139361 >> [v2:172.21.15.81:6808/568898662,v1:172.21.15.81:6809/568898662] conn(0x7f7df0071b50 msgr2=0x7f7df0073f90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:54.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.978+0000 7f7e19200640 1 --2- 172.21.15.28:0/1913139361 >> [v2:172.21.15.81:6808/568898662,v1:172.21.15.81:6809/568898662] conn(0x7f7df0071b50 0x7f7df0073f90 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:54.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.978+0000 7f7e19200640 1 -- 172.21.15.28:0/1913139361 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7df006b4d0 msgr2=0x7f7df006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:54.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.978+0000 7f7e19200640 1 --2- 172.21.15.28:0/1913139361 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f7df006b4d0 0x7f7df006d990 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f7dfc006fd0 tx=0x7f7dfc008040 comp rx=0 tx=0).stop 2024-02-21T23:17:54.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.978+0000 7f7e19200640 1 -- 172.21.15.28:0/1913139361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e1406cfa0 msgr2=0x7f7e14153fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:54.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.978+0000 7f7e19200640 1 --2- 172.21.15.28:0/1913139361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e1406cfa0 0x7f7e14153fa0 secure :-1 s=READY pgs=2782 cs=0 l=1 rev1=1 crypto rx=0x7f7e0402b8c0 tx=0x7f7e040046c0 comp rx=0 tx=0).stop 2024-02-21T23:17:54.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.978+0000 7f7e13e00640 1 -- 172.21.15.28:0/1913139361 reap_dead start 2024-02-21T23:17:54.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.979+0000 7f7e19200640 1 -- 172.21.15.28:0/1913139361 shutdown_connections 2024-02-21T23:17:54.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.979+0000 7f7e19200640 1 -- 172.21.15.28:0/1913139361 >> 172.21.15.28:0/1913139361 conn(0x7f7e140725a0 msgr2=0x7f7e140c61e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:54.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.979+0000 7f7e19200640 1 -- 172.21.15.28:0/1913139361 shutdown_connections 2024-02-21T23:17:54.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:54.979+0000 7f7e19200640 1 -- 172.21.15.28:0/1913139361 wait complete. 2024-02-21T23:17:54.984 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-02-21T23:17:55.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.212+0000 7fefec800640 1 Processor -- start 2024-02-21T23:17:55.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.213+0000 7fefec800640 1 -- start start 2024-02-21T23:17:55.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.214+0000 7fefec800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fefe406fe40 0x7fefe4150370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:55.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.214+0000 7fefec800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7fefe40ad760 0x7fefe41508b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:55.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.214+0000 7fefec800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefe406f560 con 0x7fefe406b230 2024-02-21T23:17:55.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.214+0000 7fefec800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefe4072520 con 0x7fefe406fe40 2024-02-21T23:17:55.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.214+0000 7fefec800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefe406f7e0 con 0x7fefe40ad760 2024-02-21T23:17:55.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.214+0000 7fefe9e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fefe406fe40 0x7fefe4150370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:55.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.214+0000 7fefe9e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fefe406fe40 0x7fefe4150370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49580/0 (socket says 172.21.15.28:49580) 2024-02-21T23:17:55.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.214+0000 7fefe9e00640 1 -- 172.21.15.28:0/1773519257 learned_addr learned my addr 172.21.15.28:0/1773519257 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:55.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.215+0000 7fefe9e00640 1 -- 172.21.15.28:0/1773519257 >> v2:172.21.15.81:3300/0 conn(0x7fefe40ad760 msgr2=0x7fefe41508b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:55.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.215+0000 7fefe9400640 1 --2- 172.21.15.28:0/1773519257 >> v2:172.21.15.81:3300/0 conn(0x7fefe40ad760 0x7fefe41508b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:55.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.215+0000 7fefe9e00640 1 --2- 172.21.15.28:0/1773519257 >> v2:172.21.15.81:3300/0 conn(0x7fefe40ad760 0x7fefe41508b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:55.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.215+0000 7fefe9e00640 1 -- 172.21.15.28:0/1773519257 >> v1:172.21.15.81:6789/0 conn(0x7fefe406b230 legacy=0x7fefe4153090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:55.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.215+0000 7fefe9e00640 1 -- 172.21.15.28:0/1773519257 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefe406b920 con 0x7fefe406fe40 2024-02-21T23:17:55.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.215+0000 7fefe9400640 1 --2- 172.21.15.28:0/1773519257 >> v2:172.21.15.81:3300/0 conn(0x7fefe40ad760 0x7fefe41508b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:17:55.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.216+0000 7fefe9e00640 1 --2- 172.21.15.28:0/1773519257 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefe406fe40 0x7fefe4150370 secure :-1 s=READY pgs=2783 cs=0 l=1 rev1=1 crypto rx=0x7fefe4071010 tx=0x7fefd802b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56de87fcfbeba93 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:55.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.216+0000 7fefe8a00640 1 -- 172.21.15.28:0/1773519257 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefd8038070 con 0x7fefe406fe40 2024-02-21T23:17:55.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.216+0000 7fefe8a00640 1 -- 172.21.15.28:0/1773519257 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefd802f050 con 0x7fefe406fe40 2024-02-21T23:17:55.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.216+0000 7fefe8a00640 1 -- 172.21.15.28:0/1773519257 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefd8031970 con 0x7fefe406fe40 2024-02-21T23:17:55.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.217+0000 7fefec800640 1 -- 172.21.15.28:0/1773519257 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefe406fe40 msgr2=0x7fefe4150370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:55.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.217+0000 7fefec800640 1 --2- 172.21.15.28:0/1773519257 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefe406fe40 0x7fefe4150370 secure :-1 s=READY pgs=2783 cs=0 l=1 rev1=1 crypto rx=0x7fefe4071010 tx=0x7fefd802b7f0 comp rx=0 tx=0).stop 2024-02-21T23:17:55.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.217+0000 7fefec800640 1 -- 172.21.15.28:0/1773519257 shutdown_connections 2024-02-21T23:17:55.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.217+0000 7fefec800640 1 --2- 172.21.15.28:0/1773519257 >> v2:172.21.15.81:3300/0 conn(0x7fefe40ad760 0x7fefe41508b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:55.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.217+0000 7fefec800640 1 --2- 172.21.15.28:0/1773519257 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefe406fe40 0x7fefe4150370 unknown :-1 s=CLOSED pgs=2783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:55.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.217+0000 7fefec800640 1 -- 172.21.15.28:0/1773519257 >> 172.21.15.28:0/1773519257 conn(0x7fefe40a9770 msgr2=0x7fefe40a9b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:55.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.217+0000 7fefec800640 1 -- 172.21.15.28:0/1773519257 shutdown_connections 2024-02-21T23:17:55.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.217+0000 7fefec800640 1 -- 172.21.15.28:0/1773519257 wait complete. 2024-02-21T23:17:55.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.219+0000 7fefec800640 1 Processor -- start 2024-02-21T23:17:55.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.219+0000 7fefec800640 1 -- start start 2024-02-21T23:17:55.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.220+0000 7fefec800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fefe406b230 0x7fefe41ee700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:55.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.220+0000 7fefec800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fefe406fe40 0x7fefe40c43a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:55.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.220+0000 7fefec800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefe414f4e0 0x7fefe40c48e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:55.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.220+0000 7fefec800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefe4072e00 con 0x7fefe414f4e0 2024-02-21T23:17:55.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.220+0000 7fefec800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefe4072770 con 0x7fefe406b230 2024-02-21T23:17:55.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.220+0000 7fefec800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefe4070220 con 0x7fefe406fe40 2024-02-21T23:17:55.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.220+0000 7fefea800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefe414f4e0 0x7fefe40c48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:55.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.220+0000 7fefea800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefe414f4e0 0x7fefe40c48e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49590/0 (socket says 172.21.15.28:49590) 2024-02-21T23:17:55.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.220+0000 7fefea800640 1 -- 172.21.15.28:0/1267942295 learned_addr learned my addr 172.21.15.28:0/1267942295 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:55.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.221+0000 7fefe9e00640 1 --2- 172.21.15.28:0/1267942295 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fefe406b230 0x7fefe41ee700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:55.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.221+0000 7fefe9400640 1 --2- 172.21.15.28:0/1267942295 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fefe406fe40 0x7fefe40c43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:55.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.221+0000 7fefea800640 1 -- 172.21.15.28:0/1267942295 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fefe406fe40 msgr2=0x7fefe40c43a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:55.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.221+0000 7fefea800640 1 --2- 172.21.15.28:0/1267942295 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fefe406fe40 0x7fefe40c43a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:55.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.221+0000 7fefea800640 1 -- 172.21.15.28:0/1267942295 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fefe406b230 msgr2=0x7fefe41ee700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:55.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.221+0000 7fefea800640 1 --2- 172.21.15.28:0/1267942295 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fefe406b230 0x7fefe41ee700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:55.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.221+0000 7fefea800640 1 -- 172.21.15.28:0/1267942295 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefe40c3ec0 con 0x7fefe414f4e0 2024-02-21T23:17:55.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.221+0000 7fefe9400640 1 --2- 172.21.15.28:0/1267942295 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fefe406fe40 0x7fefe40c43a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:17:55.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.221+0000 7fefe9e00640 1 --2- 172.21.15.28:0/1267942295 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fefe406b230 0x7fefe41ee700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:17:55.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.221+0000 7fefea800640 1 --2- 172.21.15.28:0/1267942295 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefe414f4e0 0x7fefe40c48e0 secure :-1 s=READY pgs=2784 cs=0 l=1 rev1=1 crypto rx=0x7fefe000c700 tx=0x7fefe000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:55.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.221+0000 7fefde000640 1 -- 172.21.15.28:0/1267942295 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefe0016020 con 0x7fefe414f4e0 2024-02-21T23:17:55.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.221+0000 7fefde000640 1 -- 172.21.15.28:0/1267942295 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefe000aa50 con 0x7fefe414f4e0 2024-02-21T23:17:55.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.221+0000 7fefec800640 1 -- 172.21.15.28:0/1267942295 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefe41ef140 con 0x7fefe414f4e0 2024-02-21T23:17:55.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.222+0000 7fefde000640 1 -- 172.21.15.28:0/1267942295 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefe000ce10 con 0x7fefe414f4e0 2024-02-21T23:17:55.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.222+0000 7fefec800640 1 -- 172.21.15.28:0/1267942295 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefe4154780 con 0x7fefe414f4e0 2024-02-21T23:17:55.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.223+0000 7fefec800640 1 -- 172.21.15.28:0/1267942295 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefe41549c0 con 0x7fefe414f4e0 2024-02-21T23:17:55.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.224+0000 7fefde000640 1 -- 172.21.15.28:0/1267942295 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fefe0019020 con 0x7fefe414f4e0 2024-02-21T23:17:55.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.225+0000 7fefde000640 1 --2- 172.21.15.28:0/1267942295 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fefc806b4d0 0x7fefc806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:55.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.225+0000 7fefde000640 1 -- 172.21.15.28:0/1267942295 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 5593+0+0 (secure 0 0 0) 0x7fefe008a3c0 con 0x7fefe414f4e0 2024-02-21T23:17:55.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.225+0000 7fefe9e00640 1 --2- 172.21.15.28:0/1267942295 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fefc806b4d0 0x7fefc806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:55.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.227+0000 7fefe9e00640 1 --2- 172.21.15.28:0/1267942295 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fefc806b4d0 0x7fefc806d990 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7fefd8035000 tx=0x7fefd8008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:55.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.229+0000 7fefde000640 1 -- 172.21.15.28:0/1267942295 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7fefe0020020 con 0x7fefe414f4e0 2024-02-21T23:17:55.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.436+0000 7fefec800640 1 -- 172.21.15.28:0/1267942295 --> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7fefe40c4e20 con 0x7fefc806b4d0 2024-02-21T23:17:55.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.437+0000 7fefde000640 1 -- 172.21.15.28:0/1267942295 <== mgr.4101 v2:172.21.15.81:6832/1848538577 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fefe40c4e20 con 0x7fefc806b4d0 2024-02-21T23:17:55.424 INFO:teuthology.orchestra.run.smithi028.stderr:instructed osd(s) 5 to deep-scrub 2024-02-21T23:17:55.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.439+0000 7fefec800640 1 -- 172.21.15.28:0/1267942295 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fefc806b4d0 msgr2=0x7fefc806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:55.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.439+0000 7fefec800640 1 --2- 172.21.15.28:0/1267942295 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fefc806b4d0 0x7fefc806d990 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7fefd8035000 tx=0x7fefd8008030 comp rx=0 tx=0).stop 2024-02-21T23:17:55.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.439+0000 7fefec800640 1 -- 172.21.15.28:0/1267942295 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefe414f4e0 msgr2=0x7fefe40c48e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:55.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.439+0000 7fefec800640 1 --2- 172.21.15.28:0/1267942295 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefe414f4e0 0x7fefe40c48e0 secure :-1 s=READY pgs=2784 cs=0 l=1 rev1=1 crypto rx=0x7fefe000c700 tx=0x7fefe000cbc0 comp rx=0 tx=0).stop 2024-02-21T23:17:55.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.440+0000 7fefec800640 1 -- 172.21.15.28:0/1267942295 shutdown_connections 2024-02-21T23:17:55.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.440+0000 7fefec800640 1 --2- 172.21.15.28:0/1267942295 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fefc806b4d0 0x7fefc806d990 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:55.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.440+0000 7fefec800640 1 --2- 172.21.15.28:0/1267942295 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fefe406fe40 0x7fefe40c43a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:55.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.440+0000 7fefec800640 1 --2- 172.21.15.28:0/1267942295 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefe414f4e0 0x7fefe40c48e0 unknown :-1 s=CLOSED pgs=2784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:55.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.440+0000 7fefec800640 1 --2- 172.21.15.28:0/1267942295 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fefe406b230 0x7fefe41ee700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:55.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.440+0000 7fefec800640 1 -- 172.21.15.28:0/1267942295 >> 172.21.15.28:0/1267942295 conn(0x7fefe40a9770 msgr2=0x7fefe40c0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:55.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.440+0000 7fefec800640 1 -- 172.21.15.28:0/1267942295 shutdown_connections 2024-02-21T23:17:55.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.440+0000 7fefec800640 1 -- 172.21.15.28:0/1267942295 wait complete. 2024-02-21T23:17:55.447 INFO:tasks.ceph:Scrubbing osd.6 2024-02-21T23:17:55.447 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-02-21T23:17:55.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.690+0000 7ff6d0a00640 1 Processor -- start 2024-02-21T23:17:55.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.690+0000 7ff6d0a00640 1 -- start start 2024-02-21T23:17:55.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.692+0000 7ff6d0a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff6cc06cfa0 0x7ff6cc0ca540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:55.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.692+0000 7ff6d0a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6cc0caf40 con 0x7ff6cc158460 2024-02-21T23:17:55.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.692+0000 7ff6d0a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6cc1588f0 con 0x7ff6cc0c6290 2024-02-21T23:17:55.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.692+0000 7ff6d0a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6cc0c8ae0 con 0x7ff6cc06cfa0 2024-02-21T23:17:55.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.692+0000 7ff6ca800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff6cc06cfa0 0x7ff6cc0ca540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:55.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.692+0000 7ff6c3e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff6cc158460 0x7ff6cc0c8f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:57120/0 (socket says 172.21.15.28:57120) 2024-02-21T23:17:55.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.692+0000 7ff6c3e00640 1 -- 172.21.15.28:0/81613944 learned_addr learned my addr 172.21.15.28:0/81613944 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:17:55.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.693+0000 7ff6ca800640 1 -- 172.21.15.28:0/81613944 >> v1:172.21.15.81:6789/0 conn(0x7ff6cc0c6290 legacy=0x7ff6cc1522c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:55.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.693+0000 7ff6ca800640 1 -- 172.21.15.28:0/81613944 >> v1:172.21.15.28:6789/0 conn(0x7ff6cc158460 legacy=0x7ff6cc0c8f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:55.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.693+0000 7ff6ca800640 1 -- 172.21.15.28:0/81613944 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6cc0c87b0 con 0x7ff6cc06cfa0 2024-02-21T23:17:55.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.693+0000 7ff6c9e00640 1 -- 172.21.15.28:0/81613944 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3666241937 0 0) 0x7ff6cc0caf40 con 0x7ff6cc158460 2024-02-21T23:17:55.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.694+0000 7ff6ca800640 1 --2- 172.21.15.28:0/81613944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6cc06cfa0 0x7ff6cc0ca540 secure :-1 s=READY pgs=2786 cs=0 l=1 rev1=1 crypto rx=0x7ff6bc0097a0 tx=0x7ff6bc02b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f96261e76e9025e server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:55.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.694+0000 7ff6c9e00640 1 -- 172.21.15.28:0/81613944 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6bc038070 con 0x7ff6cc06cfa0 2024-02-21T23:17:55.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.694+0000 7ff6c9e00640 1 -- 172.21.15.28:0/81613944 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6bc031700 con 0x7ff6cc06cfa0 2024-02-21T23:17:55.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.694+0000 7ff6c9e00640 1 -- 172.21.15.28:0/81613944 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6bc031ad0 con 0x7ff6cc06cfa0 2024-02-21T23:17:55.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.695+0000 7ff6d0a00640 1 -- 172.21.15.28:0/81613944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6cc06cfa0 msgr2=0x7ff6cc0ca540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:55.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.695+0000 7ff6d0a00640 1 --2- 172.21.15.28:0/81613944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6cc06cfa0 0x7ff6cc0ca540 secure :-1 s=READY pgs=2786 cs=0 l=1 rev1=1 crypto rx=0x7ff6bc0097a0 tx=0x7ff6bc02b7a0 comp rx=0 tx=0).stop 2024-02-21T23:17:55.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.695+0000 7ff6d0a00640 1 -- 172.21.15.28:0/81613944 shutdown_connections 2024-02-21T23:17:55.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.695+0000 7ff6d0a00640 1 --2- 172.21.15.28:0/81613944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6cc06cfa0 0x7ff6cc0ca540 unknown :-1 s=CLOSED pgs=2786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:55.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.695+0000 7ff6d0a00640 1 -- 172.21.15.28:0/81613944 >> 172.21.15.28:0/81613944 conn(0x7ff6cc0725a0 msgr2=0x7ff6cc0729b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:55.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.695+0000 7ff6d0a00640 1 -- 172.21.15.28:0/81613944 shutdown_connections 2024-02-21T23:17:55.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.695+0000 7ff6d0a00640 1 -- 172.21.15.28:0/81613944 wait complete. 2024-02-21T23:17:55.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.697+0000 7ff6d0a00640 1 Processor -- start 2024-02-21T23:17:55.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.697+0000 7ff6d0a00640 1 -- start start 2024-02-21T23:17:55.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.698+0000 7ff6d0a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6cc06cfa0 0x7ff6cc155130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:55.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.698+0000 7ff6d0a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff6cc0c6290 0x7ff6cc1eb060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:55.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.698+0000 7ff6d0a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff6cc158460 0x7ff6cc1eb5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:55.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.698+0000 7ff6d0a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6cc0c6c90 con 0x7ff6cc06cfa0 2024-02-21T23:17:55.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.698+0000 7ff6d0a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6cc0c69b0 con 0x7ff6cc0c6290 2024-02-21T23:17:55.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.698+0000 7ff6d0a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6cc0c66a0 con 0x7ff6cc158460 2024-02-21T23:17:55.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.698+0000 7ff6ca800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6cc06cfa0 0x7ff6cc155130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:55.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.698+0000 7ff6ca800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6cc06cfa0 0x7ff6cc155130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49616/0 (socket says 172.21.15.28:49616) 2024-02-21T23:17:55.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.698+0000 7ff6ca800640 1 -- 172.21.15.28:0/2191400037 learned_addr learned my addr 172.21.15.28:0/2191400037 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:55.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.699+0000 7ff6c3e00640 1 --2- 172.21.15.28:0/2191400037 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff6cc0c6290 0x7ff6cc1eb060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:55.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.699+0000 7ff6ca800640 1 -- 172.21.15.28:0/2191400037 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff6cc158460 msgr2=0x7ff6cc1eb5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:55.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.699+0000 7ff6cb200640 1 --2- 172.21.15.28:0/2191400037 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff6cc158460 0x7ff6cc1eb5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:55.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.699+0000 7ff6ca800640 1 --2- 172.21.15.28:0/2191400037 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff6cc158460 0x7ff6cc1eb5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:55.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.699+0000 7ff6ca800640 1 -- 172.21.15.28:0/2191400037 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff6cc0c6290 msgr2=0x7ff6cc1eb060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:55.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.699+0000 7ff6ca800640 1 --2- 172.21.15.28:0/2191400037 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff6cc0c6290 0x7ff6cc1eb060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:55.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.699+0000 7ff6ca800640 1 -- 172.21.15.28:0/2191400037 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6cc1ebae0 con 0x7ff6cc06cfa0 2024-02-21T23:17:55.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.699+0000 7ff6c3e00640 1 --2- 172.21.15.28:0/2191400037 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff6cc0c6290 0x7ff6cc1eb060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:17:55.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.699+0000 7ff6cb200640 1 --2- 172.21.15.28:0/2191400037 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7ff6cc158460 0x7ff6cc1eb5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:17:55.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.699+0000 7ff6ca800640 1 --2- 172.21.15.28:0/2191400037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6cc06cfa0 0x7ff6cc155130 secure :-1 s=READY pgs=2787 cs=0 l=1 rev1=1 crypto rx=0x7ff6bc0097a0 tx=0x7ff6bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:55.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.699+0000 7ff6c2a00640 1 -- 172.21.15.28:0/2191400037 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6bc043070 con 0x7ff6cc06cfa0 2024-02-21T23:17:55.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.699+0000 7ff6c2a00640 1 -- 172.21.15.28:0/2191400037 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6bc033a30 con 0x7ff6cc06cfa0 2024-02-21T23:17:55.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.699+0000 7ff6c2a00640 1 -- 172.21.15.28:0/2191400037 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6bc038040 con 0x7ff6cc06cfa0 2024-02-21T23:17:55.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.699+0000 7ff6d0a00640 1 -- 172.21.15.28:0/2191400037 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6cc1eab60 con 0x7ff6cc06cfa0 2024-02-21T23:17:55.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.700+0000 7ff6d0a00640 1 -- 172.21.15.28:0/2191400037 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6cc1ee840 con 0x7ff6cc06cfa0 2024-02-21T23:17:55.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.701+0000 7ff6d0a00640 1 -- 172.21.15.28:0/2191400037 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff6cc04f290 con 0x7ff6cc06cfa0 2024-02-21T23:17:55.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.702+0000 7ff6c2a00640 1 -- 172.21.15.28:0/2191400037 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff6bc046070 con 0x7ff6cc06cfa0 2024-02-21T23:17:55.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.703+0000 7ff6c2a00640 1 --2- 172.21.15.28:0/2191400037 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff6a806b4d0 0x7ff6a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:55.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.703+0000 7ff6c2a00640 1 -- 172.21.15.28:0/2191400037 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 5593+0+0 (secure 0 0 0) 0x7ff6bc0ac0c0 con 0x7ff6cc06cfa0 2024-02-21T23:17:55.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.703+0000 7ff6c2a00640 1 --2- 172.21.15.28:0/2191400037 >> [v2:172.21.15.81:6800/3369511790,v1:172.21.15.81:6801/3369511790] conn(0x7ff6a8071b50 0x7ff6a8073f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:55.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.703+0000 7ff6c2a00640 1 -- 172.21.15.28:0/2191400037 --> [v2:172.21.15.81:6800/3369511790,v1:172.21.15.81:6801/3369511790] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff6bc007e60 con 0x7ff6a8071b50 2024-02-21T23:17:55.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.703+0000 7ff6c2a00640 1 -- 172.21.15.28:0/2191400037 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=215) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff6bc0b3050 con 0x7ff6cc06cfa0 2024-02-21T23:17:55.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.703+0000 7ff6c3e00640 1 --2- 172.21.15.28:0/2191400037 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff6a806b4d0 0x7ff6a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:55.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.704+0000 7ff6cb200640 1 --2- 172.21.15.28:0/2191400037 >> [v2:172.21.15.81:6800/3369511790,v1:172.21.15.81:6801/3369511790] conn(0x7ff6a8071b50 0x7ff6a8073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:55.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.704+0000 7ff6c3e00640 1 --2- 172.21.15.28:0/2191400037 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff6a806b4d0 0x7ff6a806d990 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc0ca040 tx=0x7ff6b4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:55.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.705+0000 7ff6cb200640 1 --2- 172.21.15.28:0/2191400037 >> [v2:172.21.15.81:6800/3369511790,v1:172.21.15.81:6801/3369511790] conn(0x7ff6a8071b50 0x7ff6a8073f90 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:55.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.707+0000 7ff6c2a00640 1 -- 172.21.15.28:0/2191400037 <== osd.6 v2:172.21.15.81:6800/3369511790 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff6bc007e60 con 0x7ff6a8071b50 2024-02-21T23:17:55.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.722+0000 7ff6d0a00640 1 -- 172.21.15.28:0/2191400037 --> [v2:172.21.15.81:6800/3369511790,v1:172.21.15.81:6801/3369511790] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7ff6cc0afbc0 con 0x7ff6a8071b50 2024-02-21T23:17:55.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.723+0000 7ff6c2a00640 1 -- 172.21.15.28:0/2191400037 <== osd.6 v2:172.21.15.81:6800/3369511790 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7ff6cc0afbc0 con 0x7ff6a8071b50 2024-02-21T23:17:55.710 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-21T23:17:55.711 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-02-21T23:17:55.711 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-21T23:17:55.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.724+0000 7ff6d0a00640 1 -- 172.21.15.28:0/2191400037 >> [v2:172.21.15.81:6800/3369511790,v1:172.21.15.81:6801/3369511790] conn(0x7ff6a8071b50 msgr2=0x7ff6a8073f90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:55.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.724+0000 7ff6d0a00640 1 --2- 172.21.15.28:0/2191400037 >> [v2:172.21.15.81:6800/3369511790,v1:172.21.15.81:6801/3369511790] conn(0x7ff6a8071b50 0x7ff6a8073f90 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:55.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.724+0000 7ff6d0a00640 1 -- 172.21.15.28:0/2191400037 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff6a806b4d0 msgr2=0x7ff6a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:55.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.724+0000 7ff6d0a00640 1 --2- 172.21.15.28:0/2191400037 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7ff6a806b4d0 0x7ff6a806d990 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc0ca040 tx=0x7ff6b4009040 comp rx=0 tx=0).stop 2024-02-21T23:17:55.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.724+0000 7ff6d0a00640 1 -- 172.21.15.28:0/2191400037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6cc06cfa0 msgr2=0x7ff6cc155130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:55.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.724+0000 7ff6d0a00640 1 --2- 172.21.15.28:0/2191400037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6cc06cfa0 0x7ff6cc155130 secure :-1 s=READY pgs=2787 cs=0 l=1 rev1=1 crypto rx=0x7ff6bc0097a0 tx=0x7ff6bc0046c0 comp rx=0 tx=0).stop 2024-02-21T23:17:55.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.724+0000 7ff6cb200640 1 -- 172.21.15.28:0/2191400037 reap_dead start 2024-02-21T23:17:55.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.724+0000 7ff6d0a00640 1 -- 172.21.15.28:0/2191400037 shutdown_connections 2024-02-21T23:17:55.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.724+0000 7ff6d0a00640 1 -- 172.21.15.28:0/2191400037 >> 172.21.15.28:0/2191400037 conn(0x7ff6cc0725a0 msgr2=0x7ff6cc153550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:55.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.724+0000 7ff6d0a00640 1 -- 172.21.15.28:0/2191400037 shutdown_connections 2024-02-21T23:17:55.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.725+0000 7ff6d0a00640 1 -- 172.21.15.28:0/2191400037 wait complete. 2024-02-21T23:17:55.733 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-02-21T23:17:55.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.969+0000 7f802ec00640 1 Processor -- start 2024-02-21T23:17:55.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.970+0000 7f802ec00640 1 -- start start 2024-02-21T23:17:55.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.971+0000 7f802ec00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f8028149830 0x7f80280a8e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:55.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.971+0000 7f802ec00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f80280c43e0 0x7f80280c5a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:55.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.971+0000 7f802ec00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8028149c10 con 0x7f802806d410 2024-02-21T23:17:55.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.971+0000 7f802ec00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80281537d0 con 0x7f80280c43e0 2024-02-21T23:17:55.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.971+0000 7f802ec00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8028153ab0 con 0x7f8028149830 2024-02-21T23:17:55.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.972+0000 7f802ce00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f802806d410 0x7f80280c0ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:57126/0 (socket says 172.21.15.28:57126) 2024-02-21T23:17:55.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.972+0000 7f802ce00640 1 -- 172.21.15.28:0/1948334375 learned_addr learned my addr 172.21.15.28:0/1948334375 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:17:55.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.972+0000 7f802e200640 1 --2- 172.21.15.28:0/1948334375 >> v2:172.21.15.28:3300/0 conn(0x7f80280c43e0 0x7f80280c5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:55.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.972+0000 7f802d800640 1 --2- 172.21.15.28:0/1948334375 >> v2:172.21.15.81:3300/0 conn(0x7f8028149830 0x7f80280a8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:55.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.972+0000 7f8027e00640 1 -- 172.21.15.28:0/1948334375 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 237556178 0 0) 0x7f8028149c10 con 0x7f802806d410 2024-02-21T23:17:55.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.972+0000 7f8027e00640 1 -- 172.21.15.28:0/1948334375 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8010003520 con 0x7f802806d410 2024-02-21T23:17:55.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.972+0000 7f802e200640 1 -- 172.21.15.28:0/1948334375 >> v2:172.21.15.81:3300/0 conn(0x7f8028149830 msgr2=0x7f80280a8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:55.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.972+0000 7f802e200640 1 --2- 172.21.15.28:0/1948334375 >> v2:172.21.15.81:3300/0 conn(0x7f8028149830 0x7f80280a8e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:55.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.972+0000 7f802e200640 1 -- 172.21.15.28:0/1948334375 >> v1:172.21.15.28:6789/0 conn(0x7f802806d410 legacy=0x7f80280c0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:55.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.972+0000 7f802e200640 1 -- 172.21.15.28:0/1948334375 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80280c4fa0 con 0x7f80280c43e0 2024-02-21T23:17:55.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.974+0000 7f802e200640 1 --2- 172.21.15.28:0/1948334375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80280c43e0 0x7f80280c5a80 secure :-1 s=READY pgs=2789 cs=0 l=1 rev1=1 crypto rx=0x7f802806f7c0 tx=0x7f802002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9572fa03096b7cb server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:55.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.974+0000 7f8027e00640 1 -- 172.21.15.28:0/1948334375 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8020039070 con 0x7f80280c43e0 2024-02-21T23:17:55.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.974+0000 7f8027e00640 1 -- 172.21.15.28:0/1948334375 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f802002cc80 con 0x7f80280c43e0 2024-02-21T23:17:55.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.974+0000 7f8027e00640 1 -- 172.21.15.28:0/1948334375 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8020034660 con 0x7f80280c43e0 2024-02-21T23:17:55.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.974+0000 7f802ec00640 1 -- 172.21.15.28:0/1948334375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80280c43e0 msgr2=0x7f80280c5a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:55.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.974+0000 7f802ec00640 1 --2- 172.21.15.28:0/1948334375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80280c43e0 0x7f80280c5a80 secure :-1 s=READY pgs=2789 cs=0 l=1 rev1=1 crypto rx=0x7f802806f7c0 tx=0x7f802002bed0 comp rx=0 tx=0).stop 2024-02-21T23:17:55.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.974+0000 7f802ec00640 1 -- 172.21.15.28:0/1948334375 shutdown_connections 2024-02-21T23:17:55.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.974+0000 7f802ec00640 1 --2- 172.21.15.28:0/1948334375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80280c43e0 0x7f80280c5a80 unknown :-1 s=CLOSED pgs=2789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:55.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.974+0000 7f802ec00640 1 --2- 172.21.15.28:0/1948334375 >> v2:172.21.15.81:3300/0 conn(0x7f8028149830 0x7f80280a8e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:55.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.974+0000 7f802ec00640 1 -- 172.21.15.28:0/1948334375 >> 172.21.15.28:0/1948334375 conn(0x7f80280a8320 msgr2=0x7f80280a8730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:55.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.975+0000 7f802ec00640 1 -- 172.21.15.28:0/1948334375 shutdown_connections 2024-02-21T23:17:55.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.975+0000 7f802ec00640 1 -- 172.21.15.28:0/1948334375 wait complete. 2024-02-21T23:17:55.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.976+0000 7f802ec00640 1 Processor -- start 2024-02-21T23:17:55.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.977+0000 7f802ec00640 1 -- start start 2024-02-21T23:17:55.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.977+0000 7f802ec00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f802806d410 0x7f80281dea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:55.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.977+0000 7f802ec00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80280c43e0 0x7f80281edeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:55.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.977+0000 7f802ec00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8028149830 0x7f8028071e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:55.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.977+0000 7f802ec00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80280c15f0 con 0x7f80280c43e0 2024-02-21T23:17:55.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.977+0000 7f802ec00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80280c5560 con 0x7f8028149830 2024-02-21T23:17:55.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.977+0000 7f802ec00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80280c1770 con 0x7f802806d410 2024-02-21T23:17:55.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.978+0000 7f802ce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80280c43e0 0x7f80281edeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:55.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.978+0000 7f802ce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80280c43e0 0x7f80281edeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49634/0 (socket says 172.21.15.28:49634) 2024-02-21T23:17:55.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.978+0000 7f802ce00640 1 -- 172.21.15.28:0/1833682925 learned_addr learned my addr 172.21.15.28:0/1833682925 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:55.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.978+0000 7f802e200640 1 --2- 172.21.15.28:0/1833682925 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8028149830 0x7f8028071e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:55.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.978+0000 7f802d800640 1 --2- 172.21.15.28:0/1833682925 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f802806d410 0x7f80281dea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:55.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.978+0000 7f802ce00640 1 -- 172.21.15.28:0/1833682925 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f802806d410 msgr2=0x7f80281dea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:55.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.978+0000 7f802ce00640 1 --2- 172.21.15.28:0/1833682925 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f802806d410 0x7f80281dea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:55.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.978+0000 7f802ce00640 1 -- 172.21.15.28:0/1833682925 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8028149830 msgr2=0x7f8028071e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:55.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.978+0000 7f802ce00640 1 --2- 172.21.15.28:0/1833682925 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8028149830 0x7f8028071e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:55.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.978+0000 7f802ce00640 1 -- 172.21.15.28:0/1833682925 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80281ee7f0 con 0x7f80280c43e0 2024-02-21T23:17:55.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.978+0000 7f802e200640 1 --2- 172.21.15.28:0/1833682925 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8028149830 0x7f8028071e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:17:55.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.979+0000 7f802ce00640 1 --2- 172.21.15.28:0/1833682925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80280c43e0 0x7f80281edeb0 secure :-1 s=READY pgs=2790 cs=0 l=1 rev1=1 crypto rx=0x7f801400c430 tx=0x7f801400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:55.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.979+0000 7f8025600640 1 -- 172.21.15.28:0/1833682925 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8014017070 con 0x7f80280c43e0 2024-02-21T23:17:55.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.979+0000 7f8025600640 1 -- 172.21.15.28:0/1833682925 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8014005d00 con 0x7f80280c43e0 2024-02-21T23:17:55.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.979+0000 7f8025600640 1 -- 172.21.15.28:0/1833682925 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8014007310 con 0x7f80280c43e0 2024-02-21T23:17:55.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.979+0000 7f802ec00640 1 -- 172.21.15.28:0/1833682925 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80280ac020 con 0x7f80280c43e0 2024-02-21T23:17:55.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.979+0000 7f802ec00640 1 -- 172.21.15.28:0/1833682925 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80281ebb40 con 0x7f80280c43e0 2024-02-21T23:17:55.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.981+0000 7f802ec00640 1 -- 172.21.15.28:0/1833682925 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80280c73f0 con 0x7f80280c43e0 2024-02-21T23:17:55.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.981+0000 7f8025600640 1 -- 172.21.15.28:0/1833682925 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f801401a070 con 0x7f80280c43e0 2024-02-21T23:17:55.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.982+0000 7f8025600640 1 --2- 172.21.15.28:0/1833682925 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f800806b500 0x7f800806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:55.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.982+0000 7f8025600640 1 -- 172.21.15.28:0/1833682925 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f80140895c0 con 0x7f80280c43e0 2024-02-21T23:17:55.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.983+0000 7f802d800640 1 --2- 172.21.15.28:0/1833682925 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f800806b500 0x7f800806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:55.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.984+0000 7f802d800640 1 --2- 172.21.15.28:0/1833682925 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f800806b500 0x7f800806d9c0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f801c006fd0 tx=0x7f801c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:55.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:55.987+0000 7f8025600640 1 -- 172.21.15.28:0/1833682925 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f80280c73f0 con 0x7f80280c43e0 2024-02-21T23:17:56.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.186+0000 7f802ec00640 1 -- 172.21.15.28:0/1833682925 --> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7f802804f290 con 0x7f800806b500 2024-02-21T23:17:56.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.187+0000 7f8025600640 1 -- 172.21.15.28:0/1833682925 <== mgr.4101 v2:172.21.15.81:6832/1848538577 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f802804f290 con 0x7f800806b500 2024-02-21T23:17:56.174 INFO:teuthology.orchestra.run.smithi028.stderr:instructed osd(s) 6 to deep-scrub 2024-02-21T23:17:56.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.189+0000 7f802ec00640 1 -- 172.21.15.28:0/1833682925 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f800806b500 msgr2=0x7f800806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:56.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.189+0000 7f802ec00640 1 --2- 172.21.15.28:0/1833682925 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f800806b500 0x7f800806d9c0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f801c006fd0 tx=0x7f801c008040 comp rx=0 tx=0).stop 2024-02-21T23:17:56.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.189+0000 7f802ec00640 1 -- 172.21.15.28:0/1833682925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80280c43e0 msgr2=0x7f80281edeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:56.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.189+0000 7f802ec00640 1 --2- 172.21.15.28:0/1833682925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80280c43e0 0x7f80281edeb0 secure :-1 s=READY pgs=2790 cs=0 l=1 rev1=1 crypto rx=0x7f801400c430 tx=0x7f801400c8f0 comp rx=0 tx=0).stop 2024-02-21T23:17:56.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.189+0000 7f802ec00640 1 -- 172.21.15.28:0/1833682925 shutdown_connections 2024-02-21T23:17:56.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.189+0000 7f802ec00640 1 --2- 172.21.15.28:0/1833682925 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f800806b500 0x7f800806d9c0 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:56.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.189+0000 7f802ec00640 1 --2- 172.21.15.28:0/1833682925 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8028149830 0x7f8028071e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:56.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.189+0000 7f802ec00640 1 --2- 172.21.15.28:0/1833682925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80280c43e0 0x7f80281edeb0 unknown :-1 s=CLOSED pgs=2790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:56.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.189+0000 7f802ec00640 1 --2- 172.21.15.28:0/1833682925 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f802806d410 0x7f80281dea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:56.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.189+0000 7f802ec00640 1 -- 172.21.15.28:0/1833682925 >> 172.21.15.28:0/1833682925 conn(0x7f80280a8320 msgr2=0x7f80280bf9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:56.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.190+0000 7f802ec00640 1 -- 172.21.15.28:0/1833682925 shutdown_connections 2024-02-21T23:17:56.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.190+0000 7f802ec00640 1 -- 172.21.15.28:0/1833682925 wait complete. 2024-02-21T23:17:56.197 INFO:tasks.ceph:Scrubbing osd.7 2024-02-21T23:17:56.197 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-02-21T23:17:56.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.432+0000 7fb3a0a00640 1 Processor -- start 2024-02-21T23:17:56.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.433+0000 7fb3a0a00640 1 -- start start 2024-02-21T23:17:56.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.434+0000 7fb3a0a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb39c06ae70 0x7fb39c06b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:56.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.434+0000 7fb3a0a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb39c155870 0x7fb39c15c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:56.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.434+0000 7fb3a0a00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb39c1527d0 con 0x7fb39c0c2a90 2024-02-21T23:17:56.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.434+0000 7fb3a0a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb39c0724f0 con 0x7fb39c155870 2024-02-21T23:17:56.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.434+0000 7fb3a0a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb39c0727d0 con 0x7fb39c06ae70 2024-02-21T23:17:56.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.435+0000 7fb39b200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb39c155870 0x7fb39c15c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:56.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.435+0000 7fb39b200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb39c155870 0x7fb39c15c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49638/0 (socket says 172.21.15.28:49638) 2024-02-21T23:17:56.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.435+0000 7fb39b200640 1 -- 172.21.15.28:0/2223368792 learned_addr learned my addr 172.21.15.28:0/2223368792 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:56.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.435+0000 7fb39a800640 1 --2- 172.21.15.28:0/2223368792 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb39c06ae70 0x7fb39c06b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:56.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.435+0000 7fb39b200640 1 -- 172.21.15.28:0/2223368792 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb39c06ae70 msgr2=0x7fb39c06b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:56.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.435+0000 7fb39b200640 1 --2- 172.21.15.28:0/2223368792 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb39c06ae70 0x7fb39c06b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:56.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.435+0000 7fb39b200640 1 -- 172.21.15.28:0/2223368792 >> v1:172.21.15.81:6789/0 conn(0x7fb39c0c2a90 legacy=0x7fb39c155160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:56.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.435+0000 7fb39b200640 1 -- 172.21.15.28:0/2223368792 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb39c06f720 con 0x7fb39c155870 2024-02-21T23:17:56.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.436+0000 7fb39b200640 1 --2- 172.21.15.28:0/2223368792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb39c155870 0x7fb39c15c3f0 secure :-1 s=READY pgs=2791 cs=0 l=1 rev1=1 crypto rx=0x7fb394009e50 tx=0x7fb39402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78063bdce271283e server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:56.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.437+0000 7fb399e00640 1 -- 172.21.15.28:0/2223368792 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb39402cb00 con 0x7fb39c155870 2024-02-21T23:17:56.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.437+0000 7fb399e00640 1 -- 172.21.15.28:0/2223368792 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb39402cca0 con 0x7fb39c155870 2024-02-21T23:17:56.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.437+0000 7fb399e00640 1 -- 172.21.15.28:0/2223368792 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb394034610 con 0x7fb39c155870 2024-02-21T23:17:56.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.437+0000 7fb3a0a00640 1 -- 172.21.15.28:0/2223368792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb39c155870 msgr2=0x7fb39c15c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:56.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.437+0000 7fb3a0a00640 1 --2- 172.21.15.28:0/2223368792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb39c155870 0x7fb39c15c3f0 secure :-1 s=READY pgs=2791 cs=0 l=1 rev1=1 crypto rx=0x7fb394009e50 tx=0x7fb39402bf60 comp rx=0 tx=0).stop 2024-02-21T23:17:56.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.438+0000 7fb3a0a00640 1 -- 172.21.15.28:0/2223368792 shutdown_connections 2024-02-21T23:17:56.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.438+0000 7fb3a0a00640 1 --2- 172.21.15.28:0/2223368792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb39c155870 0x7fb39c15c3f0 unknown :-1 s=CLOSED pgs=2791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:56.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.438+0000 7fb3a0a00640 1 --2- 172.21.15.28:0/2223368792 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb39c06ae70 0x7fb39c06b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:56.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.438+0000 7fb3a0a00640 1 -- 172.21.15.28:0/2223368792 >> 172.21.15.28:0/2223368792 conn(0x7fb39c0c6610 msgr2=0x7fb39c0c69f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:56.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.438+0000 7fb3a0a00640 1 -- 172.21.15.28:0/2223368792 shutdown_connections 2024-02-21T23:17:56.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.438+0000 7fb3a0a00640 1 -- 172.21.15.28:0/2223368792 wait complete. 2024-02-21T23:17:56.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.440+0000 7fb3a0a00640 1 Processor -- start 2024-02-21T23:17:56.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.440+0000 7fb3a0a00640 1 -- start start 2024-02-21T23:17:56.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.441+0000 7fb3a0a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb39c06ae70 0x7fb39c0bfcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:56.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.441+0000 7fb3a0a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb39c0c2a90 0x7fb39c13a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:56.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.441+0000 7fb3a0a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb39c155870 0x7fb39c1ee290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:56.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.441+0000 7fb3a0a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb39c155cf0 con 0x7fb39c06ae70 2024-02-21T23:17:56.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.441+0000 7fb3a0a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb39c0c2240 con 0x7fb39c0c2a90 2024-02-21T23:17:56.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.441+0000 7fb3a0a00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb39c155e70 con 0x7fb39c155870 2024-02-21T23:17:56.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.441+0000 7fb39a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb39c06ae70 0x7fb39c0bfcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:56.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.441+0000 7fb39a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb39c06ae70 0x7fb39c0bfcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49648/0 (socket says 172.21.15.28:49648) 2024-02-21T23:17:56.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.441+0000 7fb39a800640 1 -- 172.21.15.28:0/3902485436 learned_addr learned my addr 172.21.15.28:0/3902485436 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:56.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.442+0000 7fb393e00640 1 --2- 172.21.15.28:0/3902485436 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb39c0c2a90 0x7fb39c13a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:56.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.442+0000 7fb39b200640 1 --2- 172.21.15.28:0/3902485436 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb39c155870 0x7fb39c1ee290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:56.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.442+0000 7fb39a800640 1 -- 172.21.15.28:0/3902485436 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb39c155870 msgr2=0x7fb39c1ee290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:56.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.442+0000 7fb39a800640 1 --2- 172.21.15.28:0/3902485436 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7fb39c155870 0x7fb39c1ee290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:56.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.442+0000 7fb39a800640 1 -- 172.21.15.28:0/3902485436 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb39c0c2a90 msgr2=0x7fb39c13a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:56.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.442+0000 7fb39a800640 1 --2- 172.21.15.28:0/3902485436 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb39c0c2a90 0x7fb39c13a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:56.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.442+0000 7fb39a800640 1 -- 172.21.15.28:0/3902485436 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb39c153650 con 0x7fb39c06ae70 2024-02-21T23:17:56.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.442+0000 7fb39a800640 1 --2- 172.21.15.28:0/3902485436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb39c06ae70 0x7fb39c0bfcb0 secure :-1 s=READY pgs=2792 cs=0 l=1 rev1=1 crypto rx=0x7fb38c00d830 tx=0x7fb38c00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:56.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.443+0000 7fb392a00640 1 -- 172.21.15.28:0/3902485436 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb38c00b6c0 con 0x7fb39c06ae70 2024-02-21T23:17:56.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.443+0000 7fb392a00640 1 -- 172.21.15.28:0/3902485436 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb38c019070 con 0x7fb39c06ae70 2024-02-21T23:17:56.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.443+0000 7fb392a00640 1 -- 172.21.15.28:0/3902485436 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb38c00ba80 con 0x7fb39c06ae70 2024-02-21T23:17:56.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.443+0000 7fb3a0a00640 1 -- 172.21.15.28:0/3902485436 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb39c153930 con 0x7fb39c06ae70 2024-02-21T23:17:56.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.443+0000 7fb3a0a00640 1 -- 172.21.15.28:0/3902485436 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb39c0c01f0 con 0x7fb39c06ae70 2024-02-21T23:17:56.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.444+0000 7fb3a0a00640 1 -- 172.21.15.28:0/3902485436 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb39c14f8e0 con 0x7fb39c06ae70 2024-02-21T23:17:56.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.445+0000 7fb392a00640 1 -- 172.21.15.28:0/3902485436 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb38c01e460 con 0x7fb39c06ae70 2024-02-21T23:17:56.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.446+0000 7fb392a00640 1 --2- 172.21.15.28:0/3902485436 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb37806b4d0 0x7fb37806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:56.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.446+0000 7fb392a00640 1 -- 172.21.15.28:0/3902485436 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 5593+0+0 (secure 0 0 0) 0x7fb38c08a880 con 0x7fb39c06ae70 2024-02-21T23:17:56.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.446+0000 7fb392a00640 1 --2- 172.21.15.28:0/3902485436 >> [v2:172.21.15.81:6816/3022012384,v1:172.21.15.81:6817/3022012384] conn(0x7fb378071b50 0x7fb378073f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:56.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.446+0000 7fb392a00640 1 -- 172.21.15.28:0/3902485436 --> [v2:172.21.15.81:6816/3022012384,v1:172.21.15.81:6817/3022012384] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb38c001d30 con 0x7fb378071b50 2024-02-21T23:17:56.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.446+0000 7fb392a00640 1 -- 172.21.15.28:0/3902485436 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=215) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb38c08f2a0 con 0x7fb39c06ae70 2024-02-21T23:17:56.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.446+0000 7fb393e00640 1 --2- 172.21.15.28:0/3902485436 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb37806b4d0 0x7fb37806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:56.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.447+0000 7fb39b200640 1 --2- 172.21.15.28:0/3902485436 >> [v2:172.21.15.81:6816/3022012384,v1:172.21.15.81:6817/3022012384] conn(0x7fb378071b50 0x7fb378073f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:56.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.448+0000 7fb393e00640 1 --2- 172.21.15.28:0/3902485436 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb37806b4d0 0x7fb37806d990 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fb39c15fb20 tx=0x7fb384009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:56.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.448+0000 7fb39b200640 1 --2- 172.21.15.28:0/3902485436 >> [v2:172.21.15.81:6816/3022012384,v1:172.21.15.81:6817/3022012384] conn(0x7fb378071b50 0x7fb378073f90 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:56.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.450+0000 7fb392a00640 1 -- 172.21.15.28:0/3902485436 <== osd.7 v2:172.21.15.81:6816/3022012384 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fb38c001d30 con 0x7fb378071b50 2024-02-21T23:17:56.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.465+0000 7fb3a0a00640 1 -- 172.21.15.28:0/3902485436 --> [v2:172.21.15.81:6816/3022012384,v1:172.21.15.81:6817/3022012384] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fb39c04f290 con 0x7fb378071b50 2024-02-21T23:17:56.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.466+0000 7fb392a00640 1 -- 172.21.15.28:0/3902485436 <== osd.7 v2:172.21.15.81:6816/3022012384 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fb39c04f290 con 0x7fb378071b50 2024-02-21T23:17:56.453 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-21T23:17:56.453 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-02-21T23:17:56.453 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-21T23:17:56.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.466+0000 7fb3a0a00640 1 -- 172.21.15.28:0/3902485436 >> [v2:172.21.15.81:6816/3022012384,v1:172.21.15.81:6817/3022012384] conn(0x7fb378071b50 msgr2=0x7fb378073f90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:56.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.466+0000 7fb3a0a00640 1 --2- 172.21.15.28:0/3902485436 >> [v2:172.21.15.81:6816/3022012384,v1:172.21.15.81:6817/3022012384] conn(0x7fb378071b50 0x7fb378073f90 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:56.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.466+0000 7fb3a0a00640 1 -- 172.21.15.28:0/3902485436 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb37806b4d0 msgr2=0x7fb37806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:56.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.466+0000 7fb3a0a00640 1 --2- 172.21.15.28:0/3902485436 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7fb37806b4d0 0x7fb37806d990 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fb39c15fb20 tx=0x7fb384009040 comp rx=0 tx=0).stop 2024-02-21T23:17:56.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.466+0000 7fb3a0a00640 1 -- 172.21.15.28:0/3902485436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb39c06ae70 msgr2=0x7fb39c0bfcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:56.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.466+0000 7fb3a0a00640 1 --2- 172.21.15.28:0/3902485436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb39c06ae70 0x7fb39c0bfcb0 secure :-1 s=READY pgs=2792 cs=0 l=1 rev1=1 crypto rx=0x7fb38c00d830 tx=0x7fb38c00dcf0 comp rx=0 tx=0).stop 2024-02-21T23:17:56.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.466+0000 7fb39b200640 1 -- 172.21.15.28:0/3902485436 reap_dead start 2024-02-21T23:17:56.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.466+0000 7fb3a0a00640 1 -- 172.21.15.28:0/3902485436 shutdown_connections 2024-02-21T23:17:56.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.466+0000 7fb3a0a00640 1 -- 172.21.15.28:0/3902485436 >> 172.21.15.28:0/3902485436 conn(0x7fb39c0c6610 msgr2=0x7fb39c074360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:56.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.467+0000 7fb3a0a00640 1 -- 172.21.15.28:0/3902485436 shutdown_connections 2024-02-21T23:17:56.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.467+0000 7fb3a0a00640 1 -- 172.21.15.28:0/3902485436 wait complete. 2024-02-21T23:17:56.475 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-02-21T23:17:56.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.714+0000 7f3d7e200640 1 Processor -- start 2024-02-21T23:17:56.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.714+0000 7f3d7e200640 1 -- start start 2024-02-21T23:17:56.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.716+0000 7f3d7e200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3d78156480 0x7f3d78154380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:56.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.716+0000 7f3d7e200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3d780c6710 0x7f3d78070a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:56.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.716+0000 7f3d7e200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d780c6cd0 con 0x7f3d78150860 2024-02-21T23:17:56.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.716+0000 7f3d7e200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d78157400 con 0x7f3d780c6710 2024-02-21T23:17:56.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.716+0000 7f3d7e200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d780c78e0 con 0x7f3d78156480 2024-02-21T23:17:56.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.716+0000 7f3d77400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f3d78150860 0x7f3d780c7d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:57134/0 (socket says 172.21.15.28:57134) 2024-02-21T23:17:56.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.717+0000 7f3d77400640 1 -- 172.21.15.28:0/3051748481 learned_addr learned my addr 172.21.15.28:0/3051748481 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:17:56.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.717+0000 7f3d77e00640 1 --2- 172.21.15.28:0/3051748481 >> v2:172.21.15.28:3300/0 conn(0x7f3d780c6710 0x7f3d78070a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:56.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.717+0000 7f3d76a00640 1 --2- 172.21.15.28:0/3051748481 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3d78156480 0x7f3d78154380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:56.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.717+0000 7f3d77e00640 1 -- 172.21.15.28:0/3051748481 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3d78156480 msgr2=0x7f3d78154380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:56.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.717+0000 7f3d77e00640 1 --2- 172.21.15.28:0/3051748481 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3d78156480 0x7f3d78154380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:56.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.717+0000 7f3d77e00640 1 -- 172.21.15.28:0/3051748481 >> v1:172.21.15.28:6789/0 conn(0x7f3d78150860 legacy=0x7f3d780c7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:56.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.717+0000 7f3d77e00640 1 -- 172.21.15.28:0/3051748481 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d780c6fb0 con 0x7f3d780c6710 2024-02-21T23:17:56.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.717+0000 7f3d76000640 1 -- 172.21.15.28:0/3051748481 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1133854155 0 0) 0x7f3d780c6cd0 con 0x7f3d78150860 2024-02-21T23:17:56.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.718+0000 7f3d77e00640 1 --2- 172.21.15.28:0/3051748481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d780c6710 0x7f3d78070a20 secure :-1 s=READY pgs=2794 cs=0 l=1 rev1=1 crypto rx=0x7f3d70009d90 tx=0x7f3d7002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=779e87e73360825d server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:56.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.718+0000 7f3d76000640 1 -- 172.21.15.28:0/3051748481 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d70039070 con 0x7f3d780c6710 2024-02-21T23:17:56.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.719+0000 7f3d76000640 1 -- 172.21.15.28:0/3051748481 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d7002cc50 con 0x7f3d780c6710 2024-02-21T23:17:56.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.719+0000 7f3d76000640 1 -- 172.21.15.28:0/3051748481 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d70034440 con 0x7f3d780c6710 2024-02-21T23:17:56.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.719+0000 7f3d7e200640 1 -- 172.21.15.28:0/3051748481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d780c6710 msgr2=0x7f3d78070a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:56.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.719+0000 7f3d7e200640 1 --2- 172.21.15.28:0/3051748481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d780c6710 0x7f3d78070a20 secure :-1 s=READY pgs=2794 cs=0 l=1 rev1=1 crypto rx=0x7f3d70009d90 tx=0x7f3d7002bea0 comp rx=0 tx=0).stop 2024-02-21T23:17:56.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.719+0000 7f3d7e200640 1 -- 172.21.15.28:0/3051748481 shutdown_connections 2024-02-21T23:17:56.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.719+0000 7f3d7e200640 1 --2- 172.21.15.28:0/3051748481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d780c6710 0x7f3d78070a20 unknown :-1 s=CLOSED pgs=2794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:56.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.719+0000 7f3d7e200640 1 --2- 172.21.15.28:0/3051748481 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3d78156480 0x7f3d78154380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:56.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.719+0000 7f3d7e200640 1 -- 172.21.15.28:0/3051748481 >> 172.21.15.28:0/3051748481 conn(0x7f3d780c8810 msgr2=0x7f3d780c8bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:56.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.720+0000 7f3d7e200640 1 -- 172.21.15.28:0/3051748481 shutdown_connections 2024-02-21T23:17:56.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.720+0000 7f3d7e200640 1 -- 172.21.15.28:0/3051748481 wait complete. 2024-02-21T23:17:56.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.721+0000 7f3d7e200640 1 Processor -- start 2024-02-21T23:17:56.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.721+0000 7f3d7e200640 1 -- start start 2024-02-21T23:17:56.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.722+0000 7f3d7e200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d780c6710 0x7f3d780aaf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:56.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.722+0000 7f3d7e200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3d78150860 0x7f3d78153310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:56.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.722+0000 7f3d7e200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d78156480 0x7f3d78145750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:56.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.722+0000 7f3d7e200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d781548c0 con 0x7f3d780c6710 2024-02-21T23:17:56.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.722+0000 7f3d7e200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d780c7570 con 0x7f3d78156480 2024-02-21T23:17:56.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.722+0000 7f3d7e200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d780c32f0 con 0x7f3d78150860 2024-02-21T23:17:56.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.722+0000 7f3d77400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d780c6710 0x7f3d780aaf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:56.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.723+0000 7f3d77400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d780c6710 0x7f3d780aaf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49664/0 (socket says 172.21.15.28:49664) 2024-02-21T23:17:56.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.723+0000 7f3d77400640 1 -- 172.21.15.28:0/3721584002 learned_addr learned my addr 172.21.15.28:0/3721584002 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:56.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.723+0000 7f3d77400640 1 -- 172.21.15.28:0/3721584002 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3d78150860 msgr2=0x7f3d78153310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:56.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.723+0000 7f3d77400640 1 --2- 172.21.15.28:0/3721584002 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3d78150860 0x7f3d78153310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:56.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.723+0000 7f3d77400640 1 -- 172.21.15.28:0/3721584002 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d78156480 msgr2=0x7f3d78145750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:56.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.723+0000 7f3d77400640 1 --2- 172.21.15.28:0/3721584002 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d78156480 0x7f3d78145750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:56.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.723+0000 7f3d77400640 1 -- 172.21.15.28:0/3721584002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d781e90d0 con 0x7f3d780c6710 2024-02-21T23:17:56.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.723+0000 7f3d77400640 1 --2- 172.21.15.28:0/3721584002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d780c6710 0x7f3d780aaf20 secure :-1 s=READY pgs=2795 cs=0 l=1 rev1=1 crypto rx=0x7f3d6800c430 tx=0x7f3d6800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:56.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.723+0000 7f3d6f400640 1 -- 172.21.15.28:0/3721584002 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d680074e0 con 0x7f3d780c6710 2024-02-21T23:17:56.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.724+0000 7f3d6f400640 1 -- 172.21.15.28:0/3721584002 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d680047e0 con 0x7f3d780c6710 2024-02-21T23:17:56.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.724+0000 7f3d6f400640 1 -- 172.21.15.28:0/3721584002 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d68004bd0 con 0x7f3d780c6710 2024-02-21T23:17:56.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.724+0000 7f3d7e200640 1 -- 172.21.15.28:0/3721584002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d78153b50 con 0x7f3d780c6710 2024-02-21T23:17:56.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.724+0000 7f3d7e200640 1 -- 172.21.15.28:0/3721584002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d78150e20 con 0x7f3d780c6710 2024-02-21T23:17:56.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.725+0000 7f3d7e200640 1 -- 172.21.15.28:0/3721584002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d780c5c90 con 0x7f3d780c6710 2024-02-21T23:17:56.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.726+0000 7f3d6f400640 1 -- 172.21.15.28:0/3721584002 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3d68011070 con 0x7f3d780c6710 2024-02-21T23:17:56.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.727+0000 7f3d6f400640 1 --2- 172.21.15.28:0/3721584002 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3d5406b500 0x7f3d5406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:56.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.727+0000 7f3d6f400640 1 -- 172.21.15.28:0/3721584002 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f3d68089cf0 con 0x7f3d780c6710 2024-02-21T23:17:56.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.727+0000 7f3d76a00640 1 --2- 172.21.15.28:0/3721584002 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3d5406b500 0x7f3d5406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:56.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.729+0000 7f3d76a00640 1 --2- 172.21.15.28:0/3721584002 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3d5406b500 0x7f3d5406d9c0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f3d780c4970 tx=0x7f3d60008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:56.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.732+0000 7f3d6f400640 1 -- 172.21.15.28:0/3721584002 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f3d780c5c90 con 0x7f3d780c6710 2024-02-21T23:17:56.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.933+0000 7f3d7e200640 1 -- 172.21.15.28:0/3721584002 --> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f3d7814f6c0 con 0x7f3d5406b500 2024-02-21T23:17:56.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.934+0000 7f3d6f400640 1 -- 172.21.15.28:0/3721584002 <== mgr.4101 v2:172.21.15.81:6832/1848538577 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f3d7814f6c0 con 0x7f3d5406b500 2024-02-21T23:17:56.921 INFO:teuthology.orchestra.run.smithi028.stderr:instructed osd(s) 7 to deep-scrub 2024-02-21T23:17:56.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.936+0000 7f3d7e200640 1 -- 172.21.15.28:0/3721584002 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3d5406b500 msgr2=0x7f3d5406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:56.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.936+0000 7f3d7e200640 1 --2- 172.21.15.28:0/3721584002 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3d5406b500 0x7f3d5406d9c0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f3d780c4970 tx=0x7f3d60008040 comp rx=0 tx=0).stop 2024-02-21T23:17:56.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.936+0000 7f3d7e200640 1 -- 172.21.15.28:0/3721584002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d780c6710 msgr2=0x7f3d780aaf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:56.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.936+0000 7f3d7e200640 1 --2- 172.21.15.28:0/3721584002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d780c6710 0x7f3d780aaf20 secure :-1 s=READY pgs=2795 cs=0 l=1 rev1=1 crypto rx=0x7f3d6800c430 tx=0x7f3d6800c8f0 comp rx=0 tx=0).stop 2024-02-21T23:17:56.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.937+0000 7f3d7e200640 1 -- 172.21.15.28:0/3721584002 shutdown_connections 2024-02-21T23:17:56.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.937+0000 7f3d7e200640 1 --2- 172.21.15.28:0/3721584002 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f3d5406b500 0x7f3d5406d9c0 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:56.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.937+0000 7f3d7e200640 1 --2- 172.21.15.28:0/3721584002 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f3d78150860 0x7f3d78153310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:56.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.937+0000 7f3d7e200640 1 --2- 172.21.15.28:0/3721584002 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3d78156480 0x7f3d78145750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:56.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.937+0000 7f3d7e200640 1 --2- 172.21.15.28:0/3721584002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d780c6710 0x7f3d780aaf20 unknown :-1 s=CLOSED pgs=2795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:56.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.937+0000 7f3d7e200640 1 -- 172.21.15.28:0/3721584002 >> 172.21.15.28:0/3721584002 conn(0x7f3d780c8810 msgr2=0x7f3d780c7d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:56.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.937+0000 7f3d7e200640 1 -- 172.21.15.28:0/3721584002 shutdown_connections 2024-02-21T23:17:56.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:56.937+0000 7f3d7e200640 1 -- 172.21.15.28:0/3721584002 wait complete. 2024-02-21T23:17:56.943 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-21T23:17:57.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.176+0000 7f1993e00640 1 Processor -- start 2024-02-21T23:17:57.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.176+0000 7f1993e00640 1 -- start start 2024-02-21T23:17:57.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.177+0000 7f1993e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f19940aff90 0x7f19940c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:57.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.177+0000 7f1993e00640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f19940c5720 0x7f199415f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:57.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.177+0000 7f1993e00640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19940c74c0 con 0x7f199406dbf0 2024-02-21T23:17:57.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.177+0000 7f1993e00640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19940c7270 con 0x7f19940c5720 2024-02-21T23:17:57.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.177+0000 7f1993e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1994158630 con 0x7f19940aff90 2024-02-21T23:17:57.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.178+0000 7f1992a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f19940aff90 0x7f19940c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:57.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.178+0000 7f1992a00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f19940aff90 0x7f19940c51e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3301/0 says I am v2:172.21.15.28:47038/0 (socket says 172.21.15.28:47038) 2024-02-21T23:17:57.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.178+0000 7f1992a00640 1 -- 172.21.15.28:0/2827084414 learned_addr learned my addr 172.21.15.28:0/2827084414 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:57.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.178+0000 7f1993400640 1 --2- 172.21.15.28:0/2827084414 >> v2:172.21.15.81:3300/0 conn(0x7f19940c5720 0x7f199415f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:57.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.179+0000 7f1992a00640 1 -- 172.21.15.28:0/2827084414 >> v2:172.21.15.81:3300/0 conn(0x7f19940c5720 msgr2=0x7f199415f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:17:57.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.179+0000 7f1992a00640 1 --2- 172.21.15.28:0/2827084414 >> v2:172.21.15.81:3300/0 conn(0x7f19940c5720 0x7f199415f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:57.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.179+0000 7f1992a00640 1 -- 172.21.15.28:0/2827084414 >> v1:172.21.15.81:6789/0 conn(0x7f199406dbf0 legacy=0x7f1994158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:57.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.179+0000 7f1992a00640 1 -- 172.21.15.28:0/2827084414 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19940c77a0 con 0x7f19940aff90 2024-02-21T23:17:57.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.180+0000 7f1992a00640 1 --2- 172.21.15.28:0/2827084414 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f19940aff90 0x7f19940c51e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f1984009870 tx=0x7f198402b7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a8c3333d2ba7c0ee server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:57.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.181+0000 7f1992000640 1 -- 172.21.15.28:0/2827084414 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1984037070 con 0x7f19940aff90 2024-02-21T23:17:57.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.181+0000 7f1992000640 1 -- 172.21.15.28:0/2827084414 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1984004230 con 0x7f19940aff90 2024-02-21T23:17:57.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.181+0000 7f1992000640 1 -- 172.21.15.28:0/2827084414 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1984003610 con 0x7f19940aff90 2024-02-21T23:17:57.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.182+0000 7f1993e00640 1 -- 172.21.15.28:0/2827084414 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f19940aff90 msgr2=0x7f19940c51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:57.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.182+0000 7f1993e00640 1 --2- 172.21.15.28:0/2827084414 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f19940aff90 0x7f19940c51e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f1984009870 tx=0x7f198402b7e0 comp rx=0 tx=0).stop 2024-02-21T23:17:57.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.182+0000 7f1993e00640 1 -- 172.21.15.28:0/2827084414 shutdown_connections 2024-02-21T23:17:57.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.182+0000 7f1993e00640 1 --2- 172.21.15.28:0/2827084414 >> v2:172.21.15.81:3300/0 conn(0x7f19940c5720 0x7f199415f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:57.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.182+0000 7f1993e00640 1 --2- 172.21.15.28:0/2827084414 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f19940aff90 0x7f19940c51e0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:57.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.182+0000 7f1993e00640 1 -- 172.21.15.28:0/2827084414 >> 172.21.15.28:0/2827084414 conn(0x7f19940ac370 msgr2=0x7f19940ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:57.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.182+0000 7f1993e00640 1 -- 172.21.15.28:0/2827084414 shutdown_connections 2024-02-21T23:17:57.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.182+0000 7f1993e00640 1 -- 172.21.15.28:0/2827084414 wait complete. 2024-02-21T23:17:57.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.184+0000 7f1993e00640 1 Processor -- start 2024-02-21T23:17:57.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.184+0000 7f1993e00640 1 -- start start 2024-02-21T23:17:57.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.185+0000 7f1993e00640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f199406dbf0 0x7f19940bf250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:57.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.185+0000 7f1993e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19940aff90 0x7f19940aecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:57.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.185+0000 7f1993e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f19940c5720 0x7f19941eb4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:57.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.185+0000 7f1993e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19940c8760 con 0x7f19940aff90 2024-02-21T23:17:57.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.185+0000 7f1993e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19940be0c0 con 0x7f19940c5720 2024-02-21T23:17:57.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.185+0000 7f1993e00640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19940c88e0 con 0x7f199406dbf0 2024-02-21T23:17:57.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.186+0000 7f198be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19940aff90 0x7f19940aecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:57.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.186+0000 7f198be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19940aff90 0x7f19940aecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49678/0 (socket says 172.21.15.28:49678) 2024-02-21T23:17:57.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.186+0000 7f198be00640 1 -- 172.21.15.28:0/2531769130 learned_addr learned my addr 172.21.15.28:0/2531769130 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:17:57.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.186+0000 7f1992a00640 1 --2- 172.21.15.28:0/2531769130 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f199406dbf0 0x7f19940bf250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:57.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.186+0000 7f1993400640 1 --2- 172.21.15.28:0/2531769130 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f19940c5720 0x7f19941eb4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:57.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.186+0000 7f198be00640 1 -- 172.21.15.28:0/2531769130 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f199406dbf0 msgr2=0x7f19940bf250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:57.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.186+0000 7f198be00640 1 --2- 172.21.15.28:0/2531769130 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f199406dbf0 0x7f19940bf250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:57.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.186+0000 7f198be00640 1 -- 172.21.15.28:0/2531769130 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f19940c5720 msgr2=0x7f19941eb4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:57.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.186+0000 7f198be00640 1 --2- 172.21.15.28:0/2531769130 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f19940c5720 0x7f19941eb4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:57.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.186+0000 7f198be00640 1 -- 172.21.15.28:0/2531769130 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19941543d0 con 0x7f19940aff90 2024-02-21T23:17:57.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.186+0000 7f1993400640 1 --2- 172.21.15.28:0/2531769130 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f19940c5720 0x7f19941eb4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:17:57.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.187+0000 7f198be00640 1 --2- 172.21.15.28:0/2531769130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19940aff90 0x7f19940aecc0 secure :-1 s=READY pgs=2796 cs=0 l=1 rev1=1 crypto rx=0x7f197c00c780 tx=0x7f197c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:57.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.187+0000 7f198b400640 1 -- 172.21.15.28:0/2531769130 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f197c017070 con 0x7f19940aff90 2024-02-21T23:17:57.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.187+0000 7f198b400640 1 -- 172.21.15.28:0/2531769130 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f197c005290 con 0x7f19940aff90 2024-02-21T23:17:57.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.187+0000 7f198b400640 1 -- 172.21.15.28:0/2531769130 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f197c003830 con 0x7f19940aff90 2024-02-21T23:17:57.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.187+0000 7f1993e00640 1 -- 172.21.15.28:0/2531769130 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19941ead10 con 0x7f19940aff90 2024-02-21T23:17:57.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.187+0000 7f1993e00640 1 -- 172.21.15.28:0/2531769130 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19940af200 con 0x7f19940aff90 2024-02-21T23:17:57.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.188+0000 7f1993e00640 1 -- 172.21.15.28:0/2531769130 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f199404f290 con 0x7f19940aff90 2024-02-21T23:17:57.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.189+0000 7f198b400640 1 -- 172.21.15.28:0/2531769130 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f197c01a070 con 0x7f19940aff90 2024-02-21T23:17:57.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.190+0000 7f198b400640 1 --2- 172.21.15.28:0/2531769130 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f197006b4d0 0x7f197006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:17:57.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.190+0000 7f198b400640 1 -- 172.21.15.28:0/2531769130 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f197c0896f0 con 0x7f19940aff90 2024-02-21T23:17:57.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.191+0000 7f1992a00640 1 --2- 172.21.15.28:0/2531769130 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f197006b4d0 0x7f197006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:17:57.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.192+0000 7f1992a00640 1 --2- 172.21.15.28:0/2531769130 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f197006b4d0 0x7f197006d990 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f1984034000 tx=0x7f1984008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:17:57.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.195+0000 7f198b400640 1 -- 172.21.15.28:0/2531769130 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f19941ead10 con 0x7f19940aff90 2024-02-21T23:17:57.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.398+0000 7f1993e00640 1 -- 172.21.15.28:0/2531769130 --> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f19940c50e0 con 0x7f197006b4d0 2024-02-21T23:17:57.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.400+0000 7f198b400640 1 -- 172.21.15.28:0/2531769130 <== mgr.4101 v2:172.21.15.81:6832/1848538577 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13454 (secure 0 0 0) 0x7f19940c50e0 con 0x7f197006b4d0 2024-02-21T23:17:57.388 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:17:57.388 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-21T23:17:57.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.403+0000 7f1993e00640 1 -- 172.21.15.28:0/2531769130 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f197006b4d0 msgr2=0x7f197006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:57.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.403+0000 7f1993e00640 1 --2- 172.21.15.28:0/2531769130 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f197006b4d0 0x7f197006d990 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f1984034000 tx=0x7f1984008030 comp rx=0 tx=0).stop 2024-02-21T23:17:57.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.403+0000 7f1993e00640 1 -- 172.21.15.28:0/2531769130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19940aff90 msgr2=0x7f19940aecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:17:57.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.403+0000 7f1993e00640 1 --2- 172.21.15.28:0/2531769130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19940aff90 0x7f19940aecc0 secure :-1 s=READY pgs=2796 cs=0 l=1 rev1=1 crypto rx=0x7f197c00c780 tx=0x7f197c00cc40 comp rx=0 tx=0).stop 2024-02-21T23:17:57.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.403+0000 7f1993e00640 1 -- 172.21.15.28:0/2531769130 shutdown_connections 2024-02-21T23:17:57.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.403+0000 7f1993e00640 1 --2- 172.21.15.28:0/2531769130 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f197006b4d0 0x7f197006d990 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:57.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.403+0000 7f1993e00640 1 --2- 172.21.15.28:0/2531769130 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f19940c5720 0x7f19941eb4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:57.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.403+0000 7f1993e00640 1 --2- 172.21.15.28:0/2531769130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19940aff90 0x7f19940aecc0 unknown :-1 s=CLOSED pgs=2796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:57.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.403+0000 7f1993e00640 1 --2- 172.21.15.28:0/2531769130 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f199406dbf0 0x7f19940bf250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:17:57.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.403+0000 7f1993e00640 1 -- 172.21.15.28:0/2531769130 >> 172.21.15.28:0/2531769130 conn(0x7f19940ac370 msgr2=0x7f19940ca7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:17:57.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.403+0000 7f1993e00640 1 -- 172.21.15.28:0/2531769130 shutdown_connections 2024-02-21T23:17:57.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:17:57.403+0000 7f1993e00640 1 -- 172.21.15.28:0/2531769130 wait complete. 2024-02-21T23:17:57.412 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":671,"stamp":"2024-02-21T23:17:57.002038+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":287608,"kb_used_data":21504,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754687112,"statfs":{"total":773094113280,"available":772799602688,"internally_reserved":0,"allocated":22020096,"data_stored":11895784,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":272354865},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.775829"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":535,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-02-21T23:17:45.612378+0000","last_change":"2024-02-21T23:02:32.289335+0000","last_active":"2024-02-21T23:17:45.612378+0000","last_peered":"2024-02-21T23:17:45.612378+0000","last_clean":"2024-02-21T23:17:45.612378+0000","last_became_active":"2024-02-21T23:02:32.286913+0000","last_became_peered":"2024-02-21T23:02:32.286913+0000","last_unstale":"2024-02-21T23:17:45.612378+0000","last_undegraded":"2024-02-21T23:17:45.612378+0000","last_fullsized":"2024-02-21T23:17:45.612378+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-21T23:02:31.172569+0000","last_clean_scrub_stamp":"2024-02-21T23:02:31.172569+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-22T23:36:23.987374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575035,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33916,"kb_used_data":2736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337924,"statfs":{"total":96636764160,"available":96602034176,"internally_reserved":0,"allocated":2801664,"data_stored":1929749,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575036,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33856,"kb_used_data":2672,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337984,"statfs":{"total":96636764160,"available":96602095616,"internally_reserved":0,"allocated":2736128,"data_stored":1339381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575035,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33856,"kb_used_data":2672,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337984,"statfs":{"total":96636764160,"available":96602095616,"internally_reserved":0,"allocated":2736128,"data_stored":1339381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575036,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33856,"kb_used_data":2672,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337984,"statfs":{"total":96636764160,"available":96602095616,"internally_reserved":0,"allocated":2736128,"data_stored":1339381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575036,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38016,"kb_used_data":2672,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333824,"statfs":{"total":96636764160,"available":96597835776,"internally_reserved":0,"allocated":2736128,"data_stored":1339381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575037,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38016,"kb_used_data":2672,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333824,"statfs":{"total":96636764160,"available":96597835776,"internally_reserved":0,"allocated":2736128,"data_stored":1339381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575035,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38016,"kb_used_data":2672,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333824,"statfs":{"total":96636764160,"available":96597835776,"internally_reserved":0,"allocated":2736128,"data_stored":1339381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575035,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38076,"kb_used_data":2736,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333764,"statfs":{"total":96636764160,"available":96597774336,"internally_reserved":0,"allocated":2801664,"data_stored":1929749,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-21T23:17:57.415 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-02-21T23:02:31.172569+0000 time.struct_time(tm_year=2024, tm_mon=2, tm_mday=21, tm_hour=23, tm_min=2, tm_sec=31, tm_wday=2, tm_yday=52, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=2, tm_mday=21, tm_hour=23, tm_min=17, tm_sec=50, tm_wday=2, tm_yday=52, tm_isdst=0) 2024-02-21T23:17:57.415 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-02-21T23:18:09.502 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-21T23:18:09.511 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-21T23:18:17.415 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-21T23:18:17.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.657+0000 7f0cd2400640 1 Processor -- start 2024-02-21T23:18:17.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.658+0000 7f0cd2400640 1 -- start start 2024-02-21T23:18:17.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.659+0000 7f0cd2400640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f0ccc06dbf0 0x7f0ccc158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:18:17.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.659+0000 7f0cd2400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0ccc155960 0x7f0ccc159000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:18:17.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.659+0000 7f0cd2400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ccc0c74c0 con 0x7f0ccc0aff90 2024-02-21T23:18:17.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.659+0000 7f0cd2400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ccc0c7270 con 0x7f0ccc155960 2024-02-21T23:18:17.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.659+0000 7f0cd2400640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ccc158630 con 0x7f0ccc06dbf0 2024-02-21T23:18:17.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.659+0000 7f0cd1000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f0ccc0aff90 0x7f0ccc0c51e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42628/0 (socket says 172.21.15.28:42628) 2024-02-21T23:18:17.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.659+0000 7f0cd1000640 1 -- 172.21.15.28:0/686785385 learned_addr learned my addr 172.21.15.28:0/686785385 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-02-21T23:18:17.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.660+0000 7f0cd1a00640 1 --2- 172.21.15.28:0/686785385 >> v2:172.21.15.28:3300/0 conn(0x7f0ccc155960 0x7f0ccc159000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:18:17.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.660+0000 7f0ccbe00640 1 --2- 172.21.15.28:0/686785385 >> v2:172.21.15.81:3300/0 conn(0x7f0ccc06dbf0 0x7f0ccc158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:18:17.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.660+0000 7f0ccb400640 1 -- 172.21.15.28:0/686785385 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1896888611 0 0) 0x7f0ccc0c74c0 con 0x7f0ccc0aff90 2024-02-21T23:18:17.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.660+0000 7f0cd1a00640 1 -- 172.21.15.28:0/686785385 >> v2:172.21.15.81:3300/0 conn(0x7f0ccc06dbf0 msgr2=0x7f0ccc158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:18:17.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.660+0000 7f0cd1a00640 1 --2- 172.21.15.28:0/686785385 >> v2:172.21.15.81:3300/0 conn(0x7f0ccc06dbf0 0x7f0ccc158ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:18:17.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.660+0000 7f0cd1a00640 1 -- 172.21.15.28:0/686785385 >> v1:172.21.15.28:6789/0 conn(0x7f0ccc0aff90 legacy=0x7f0ccc0c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:18:17.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.660+0000 7f0cd1a00640 1 -- 172.21.15.28:0/686785385 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ccc0c77a0 con 0x7f0ccc155960 2024-02-21T23:18:17.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.661+0000 7f0cd1a00640 1 --2- 172.21.15.28:0/686785385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ccc155960 0x7f0ccc159000 secure :-1 s=READY pgs=2798 cs=0 l=1 rev1=1 crypto rx=0x7f0ccc0cb8c0 tx=0x7f0cc402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abc0138d6f89ccc4 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:18:17.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.662+0000 7f0ccb400640 1 -- 172.21.15.28:0/686785385 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cc4035020 con 0x7f0ccc155960 2024-02-21T23:18:17.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.662+0000 7f0ccb400640 1 -- 172.21.15.28:0/686785385 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cc4031830 con 0x7f0ccc155960 2024-02-21T23:18:17.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.662+0000 7f0ccb400640 1 -- 172.21.15.28:0/686785385 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cc4031c00 con 0x7f0ccc155960 2024-02-21T23:18:17.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.662+0000 7f0cd2400640 1 -- 172.21.15.28:0/686785385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ccc155960 msgr2=0x7f0ccc159000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:18:17.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.662+0000 7f0cd2400640 1 --2- 172.21.15.28:0/686785385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ccc155960 0x7f0ccc159000 secure :-1 s=READY pgs=2798 cs=0 l=1 rev1=1 crypto rx=0x7f0ccc0cb8c0 tx=0x7f0cc402b8d0 comp rx=0 tx=0).stop 2024-02-21T23:18:17.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.663+0000 7f0cd2400640 1 -- 172.21.15.28:0/686785385 shutdown_connections 2024-02-21T23:18:17.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.663+0000 7f0cd2400640 1 --2- 172.21.15.28:0/686785385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ccc155960 0x7f0ccc159000 unknown :-1 s=CLOSED pgs=2798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:18:17.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.663+0000 7f0cd2400640 1 --2- 172.21.15.28:0/686785385 >> v2:172.21.15.81:3300/0 conn(0x7f0ccc06dbf0 0x7f0ccc158ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:18:17.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.663+0000 7f0cd2400640 1 -- 172.21.15.28:0/686785385 >> 172.21.15.28:0/686785385 conn(0x7f0ccc0ac370 msgr2=0x7f0ccc0ac780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:18:17.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.663+0000 7f0cd2400640 1 -- 172.21.15.28:0/686785385 shutdown_connections 2024-02-21T23:18:17.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.663+0000 7f0cd2400640 1 -- 172.21.15.28:0/686785385 wait complete. 2024-02-21T23:18:17.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.664+0000 7f0cd2400640 1 Processor -- start 2024-02-21T23:18:17.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.665+0000 7f0cd2400640 1 -- start start 2024-02-21T23:18:17.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.666+0000 7f0cd2400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ccc06dbf0 0x7f0ccc1ee190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:18:17.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.666+0000 7f0cd2400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ccc0aff90 0x7f0ccc0c2950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:18:17.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.666+0000 7f0cd2400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0ccc155960 0x7f0ccc0c2e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:18:17.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.666+0000 7f0cd2400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ccc0c58f0 con 0x7f0ccc0aff90 2024-02-21T23:18:17.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.666+0000 7f0cd2400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ccc0be0c0 con 0x7f0ccc06dbf0 2024-02-21T23:18:17.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.666+0000 7f0cd2400640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ccc0c5a70 con 0x7f0ccc155960 2024-02-21T23:18:17.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.666+0000 7f0ccbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ccc0aff90 0x7f0ccc0c2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:18:17.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.666+0000 7f0ccbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ccc0aff90 0x7f0ccc0c2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36024/0 (socket says 172.21.15.28:36024) 2024-02-21T23:18:17.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.666+0000 7f0ccbe00640 1 -- 172.21.15.28:0/716697607 learned_addr learned my addr 172.21.15.28:0/716697607 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:18:17.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.666+0000 7f0cd1000640 1 --2- 172.21.15.28:0/716697607 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ccc06dbf0 0x7f0ccc1ee190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:18:17.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.666+0000 7f0cd1a00640 1 --2- 172.21.15.28:0/716697607 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0ccc155960 0x7f0ccc0c2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:18:17.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.666+0000 7f0ccbe00640 1 -- 172.21.15.28:0/716697607 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0ccc155960 msgr2=0x7f0ccc0c2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:18:17.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.666+0000 7f0ccbe00640 1 --2- 172.21.15.28:0/716697607 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0ccc155960 0x7f0ccc0c2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:18:17.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.666+0000 7f0ccbe00640 1 -- 172.21.15.28:0/716697607 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ccc06dbf0 msgr2=0x7f0ccc1ee190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:18:17.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.666+0000 7f0ccbe00640 1 --2- 172.21.15.28:0/716697607 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ccc06dbf0 0x7f0ccc1ee190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:18:17.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.666+0000 7f0ccbe00640 1 -- 172.21.15.28:0/716697607 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ccc1eee20 con 0x7f0ccc0aff90 2024-02-21T23:18:17.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.667+0000 7f0ccbe00640 1 --2- 172.21.15.28:0/716697607 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ccc0aff90 0x7f0ccc0c2950 secure :-1 s=READY pgs=2799 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc009fd0 tx=0x7f0cbc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:18:17.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.667+0000 7f0cc8c00640 1 -- 172.21.15.28:0/716697607 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cbc014620 con 0x7f0ccc0aff90 2024-02-21T23:18:17.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.667+0000 7f0cc8c00640 1 -- 172.21.15.28:0/716697607 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cbc019070 con 0x7f0ccc0aff90 2024-02-21T23:18:17.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.667+0000 7f0cd2400640 1 -- 172.21.15.28:0/716697607 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ccc0c4070 con 0x7f0ccc0aff90 2024-02-21T23:18:17.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.667+0000 7f0cc8c00640 1 -- 172.21.15.28:0/716697607 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cbc014a00 con 0x7f0ccc0aff90 2024-02-21T23:18:17.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.667+0000 7f0cd2400640 1 -- 172.21.15.28:0/716697607 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ccc0be240 con 0x7f0ccc0aff90 2024-02-21T23:18:17.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.669+0000 7f0cd2400640 1 -- 172.21.15.28:0/716697607 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ccc0ab360 con 0x7f0ccc0aff90 2024-02-21T23:18:17.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.669+0000 7f0cc8c00640 1 -- 172.21.15.28:0/716697607 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0cbc014ba0 con 0x7f0ccc0aff90 2024-02-21T23:18:17.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.670+0000 7f0cc8c00640 1 --2- 172.21.15.28:0/716697607 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0ca806b4d0 0x7f0ca806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:18:17.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.670+0000 7f0cc8c00640 1 -- 172.21.15.28:0/716697607 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f0cbc08aac0 con 0x7f0ccc0aff90 2024-02-21T23:18:17.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.671+0000 7f0cd1000640 1 --2- 172.21.15.28:0/716697607 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0ca806b4d0 0x7f0ca806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:18:17.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.672+0000 7f0cd1000640 1 --2- 172.21.15.28:0/716697607 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0ca806b4d0 0x7f0ca806d990 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f0cb4000f60 tx=0x7f0cb4007b50 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:18:17.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.675+0000 7f0cc8c00640 1 -- 172.21.15.28:0/716697607 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f0ccc0ab360 con 0x7f0ccc0aff90 2024-02-21T23:18:17.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.869+0000 7f0cd2400640 1 -- 172.21.15.28:0/716697607 --> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0ccc04f290 con 0x7f0ca806b4d0 2024-02-21T23:18:17.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.872+0000 7f0cc8c00640 1 -- 172.21.15.28:0/716697607 <== mgr.4101 v2:172.21.15.81:6832/1848538577 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13464 (secure 0 0 0) 0x7f0ccc04f290 con 0x7f0ca806b4d0 2024-02-21T23:18:17.860 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:18:17.860 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-21T23:18:17.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.875+0000 7f0cd2400640 1 -- 172.21.15.28:0/716697607 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0ca806b4d0 msgr2=0x7f0ca806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:18:17.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.875+0000 7f0cd2400640 1 --2- 172.21.15.28:0/716697607 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0ca806b4d0 0x7f0ca806d990 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f0cb4000f60 tx=0x7f0cb4007b50 comp rx=0 tx=0).stop 2024-02-21T23:18:17.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.875+0000 7f0cd2400640 1 -- 172.21.15.28:0/716697607 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ccc0aff90 msgr2=0x7f0ccc0c2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:18:17.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.875+0000 7f0cd2400640 1 --2- 172.21.15.28:0/716697607 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ccc0aff90 0x7f0ccc0c2950 secure :-1 s=READY pgs=2799 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc009fd0 tx=0x7f0cbc00e4b0 comp rx=0 tx=0).stop 2024-02-21T23:18:17.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.875+0000 7f0cd2400640 1 -- 172.21.15.28:0/716697607 shutdown_connections 2024-02-21T23:18:17.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.875+0000 7f0cd2400640 1 --2- 172.21.15.28:0/716697607 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f0ca806b4d0 0x7f0ca806d990 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:18:17.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.875+0000 7f0cd2400640 1 --2- 172.21.15.28:0/716697607 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f0ccc155960 0x7f0ccc0c2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:18:17.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.875+0000 7f0cd2400640 1 --2- 172.21.15.28:0/716697607 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ccc0aff90 0x7f0ccc0c2950 unknown :-1 s=CLOSED pgs=2799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:18:17.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.875+0000 7f0cd2400640 1 --2- 172.21.15.28:0/716697607 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0ccc06dbf0 0x7f0ccc1ee190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:18:17.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.875+0000 7f0cd2400640 1 -- 172.21.15.28:0/716697607 >> 172.21.15.28:0/716697607 conn(0x7f0ccc0ac370 msgr2=0x7f0ccc0cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:18:17.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.875+0000 7f0cd2400640 1 -- 172.21.15.28:0/716697607 shutdown_connections 2024-02-21T23:18:17.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:17.875+0000 7f0cd2400640 1 -- 172.21.15.28:0/716697607 wait complete. 2024-02-21T23:18:17.883 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":681,"stamp":"2024-02-21T23:18:17.007896+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":287600,"kb_used_data":21504,"kb_used_omap":12,"kb_used_meta":265971,"kb_avail":754687120,"statfs":{"total":773094113280,"available":772799610880,"internally_reserved":0,"allocated":22020096,"data_stored":11895784,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12751,"internal_metadata":272354865},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003579"},"pg_stats":[{"pgid":"1.0","version":"215'119","reported_seq":546,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-02-21T23:17:57.867453+0000","last_change":"2024-02-21T23:17:57.867523+0000","last_active":"2024-02-21T23:17:57.867453+0000","last_peered":"2024-02-21T23:17:57.867453+0000","last_clean":"2024-02-21T23:17:57.867453+0000","last_became_active":"2024-02-21T23:02:32.286913+0000","last_became_peered":"2024-02-21T23:02:32.286913+0000","last_unstale":"2024-02-21T23:17:57.867453+0000","last_undegraded":"2024-02-21T23:17:57.867453+0000","last_fullsized":"2024-02-21T23:17:57.867453+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"215'119","last_scrub_stamp":"2024-02-21T23:17:57.867278+0000","last_deep_scrub":"215'119","last_deep_scrub_stamp":"2024-02-21T23:17:57.867278+0000","last_clean_scrub_stamp":"2024-02-21T23:17:57.867278+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-23T08:27:42.203211+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65096,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575039,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33912,"kb_used_data":2736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337928,"statfs":{"total":96636764160,"available":96602038272,"internally_reserved":0,"allocated":2801664,"data_stored":1929749,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575040,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33856,"kb_used_data":2672,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337984,"statfs":{"total":96636764160,"available":96602095616,"internally_reserved":0,"allocated":2736128,"data_stored":1339381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575039,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33856,"kb_used_data":2672,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337984,"statfs":{"total":96636764160,"available":96602095616,"internally_reserved":0,"allocated":2736128,"data_stored":1339381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575040,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33856,"kb_used_data":2672,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337984,"statfs":{"total":96636764160,"available":96602095616,"internally_reserved":0,"allocated":2736128,"data_stored":1339381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575040,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38016,"kb_used_data":2672,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333824,"statfs":{"total":96636764160,"available":96597835776,"internally_reserved":0,"allocated":2736128,"data_stored":1339381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575041,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38016,"kb_used_data":2672,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333824,"statfs":{"total":96636764160,"available":96597835776,"internally_reserved":0,"allocated":2736128,"data_stored":1339381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575039,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":38016,"kb_used_data":2672,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333824,"statfs":{"total":96636764160,"available":96597835776,"internally_reserved":0,"allocated":2736128,"data_stored":1339381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575039,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38072,"kb_used_data":2736,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333768,"statfs":{"total":96636764160,"available":96597778432,"internally_reserved":0,"allocated":2801664,"data_stored":1929749,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32548,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-21T23:18:17.884 DEBUG:teuthology.orchestra.run.smithi028:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-02-21T23:18:18.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.103+0000 7f9f0a800640 1 Processor -- start 2024-02-21T23:18:18.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.103+0000 7f9f0a800640 1 -- start start 2024-02-21T23:18:18.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.104+0000 7f9f0a800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9f04150240 0x7f9f040c12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:18:18.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.104+0000 7f9f0a800640 1 --2- >> v2:172.21.15.81:3300/0 conn(0x7f9f040c2c10 0x7f9f0406ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:18:18.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.104+0000 7f9f0a800640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f0406e220 con 0x7f9f040c4580 2024-02-21T23:18:18.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.104+0000 7f9f0a800640 1 -- --> v2:172.21.15.81:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f04150620 con 0x7f9f040c2c10 2024-02-21T23:18:18.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.104+0000 7f9f0a800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f04153a90 con 0x7f9f04150240 2024-02-21T23:18:18.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.105+0000 7f9f09400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9f04150240 0x7f9f040c12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:18:18.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.105+0000 7f9f09400640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9f04150240 0x7f9f040c12c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3301/0 says I am v2:172.21.15.28:50072/0 (socket says 172.21.15.28:50072) 2024-02-21T23:18:18.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.105+0000 7f9f09400640 1 -- 172.21.15.28:0/433097140 learned_addr learned my addr 172.21.15.28:0/433097140 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:18:18.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.105+0000 7f9f09e00640 1 --2- 172.21.15.28:0/433097140 >> v2:172.21.15.81:3300/0 conn(0x7f9f040c2c10 0x7f9f0406ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:18:18.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.106+0000 7f9f09400640 1 -- 172.21.15.28:0/433097140 >> v2:172.21.15.81:3300/0 conn(0x7f9f040c2c10 msgr2=0x7f9f0406ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:18:18.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.106+0000 7f9f09400640 1 --2- 172.21.15.28:0/433097140 >> v2:172.21.15.81:3300/0 conn(0x7f9f040c2c10 0x7f9f0406ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:18:18.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.106+0000 7f9f09400640 1 -- 172.21.15.28:0/433097140 >> v1:172.21.15.81:6789/0 conn(0x7f9f040c4580 legacy=0x7f9f0406db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:18:18.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.106+0000 7f9f09400640 1 -- 172.21.15.28:0/433097140 --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f0406f780 con 0x7f9f04150240 2024-02-21T23:18:18.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.107+0000 7f9f09400640 1 --2- 172.21.15.28:0/433097140 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9f04150240 0x7f9f040c12c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9ef8009870 tx=0x7f9ef802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d86048f3d1436ec server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:18:18.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.107+0000 7f9f03e00640 1 -- 172.21.15.28:0/433097140 <== mon.2 v2:172.21.15.81:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ef8035020 con 0x7f9f04150240 2024-02-21T23:18:18.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.107+0000 7f9f03e00640 1 -- 172.21.15.28:0/433097140 <== mon.2 v2:172.21.15.81:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ef8031640 con 0x7f9f04150240 2024-02-21T23:18:18.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.107+0000 7f9f03e00640 1 -- 172.21.15.28:0/433097140 <== mon.2 v2:172.21.15.81:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ef802d040 con 0x7f9f04150240 2024-02-21T23:18:18.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.108+0000 7f9f0a800640 1 -- 172.21.15.28:0/433097140 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9f04150240 msgr2=0x7f9f040c12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:18:18.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.108+0000 7f9f0a800640 1 --2- 172.21.15.28:0/433097140 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9f04150240 0x7f9f040c12c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9ef8009870 tx=0x7f9ef802b8f0 comp rx=0 tx=0).stop 2024-02-21T23:18:18.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.108+0000 7f9f0a800640 1 -- 172.21.15.28:0/433097140 shutdown_connections 2024-02-21T23:18:18.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.108+0000 7f9f0a800640 1 --2- 172.21.15.28:0/433097140 >> v2:172.21.15.81:3300/0 conn(0x7f9f040c2c10 0x7f9f0406ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:18:18.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.108+0000 7f9f0a800640 1 --2- 172.21.15.28:0/433097140 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9f04150240 0x7f9f040c12c0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:18:18.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.108+0000 7f9f0a800640 1 -- 172.21.15.28:0/433097140 >> 172.21.15.28:0/433097140 conn(0x7f9f040a8530 msgr2=0x7f9f040a8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:18:18.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.108+0000 7f9f0a800640 1 -- 172.21.15.28:0/433097140 shutdown_connections 2024-02-21T23:18:18.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.108+0000 7f9f0a800640 1 -- 172.21.15.28:0/433097140 wait complete. 2024-02-21T23:18:18.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.110+0000 7f9f0a800640 1 Processor -- start 2024-02-21T23:18:18.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.110+0000 7f9f0a800640 1 -- start start 2024-02-21T23:18:18.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.111+0000 7f9f0a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f040c2c10 0x7f9f041e16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:18:18.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.111+0000 7f9f0a800640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9f040c4580 0x7f9f04151e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:18:18.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.111+0000 7f9f0a800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f04150240 0x7f9f04152350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:18:18.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.111+0000 7f9f0a800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f040c4a10 con 0x7f9f040c2c10 2024-02-21T23:18:18.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.111+0000 7f9f0a800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f04153720 con 0x7f9f04150240 2024-02-21T23:18:18.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.111+0000 7f9f0a800640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f040c4b90 con 0x7f9f040c4580 2024-02-21T23:18:18.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.111+0000 7f9f09400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f040c2c10 0x7f9f041e16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:18:18.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.111+0000 7f9f09400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f040c2c10 0x7f9f041e16a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36026/0 (socket says 172.21.15.28:36026) 2024-02-21T23:18:18.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.111+0000 7f9f09400640 1 -- 172.21.15.28:0/3221733943 learned_addr learned my addr 172.21.15.28:0/3221733943 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:18:18.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.112+0000 7f9f09400640 1 -- 172.21.15.28:0/3221733943 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9f040c4580 msgr2=0x7f9f04151e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:18:18.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.112+0000 7f9f09400640 1 --2- 172.21.15.28:0/3221733943 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9f040c4580 0x7f9f04151e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:18:18.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.112+0000 7f9f09400640 1 -- 172.21.15.28:0/3221733943 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f04150240 msgr2=0x7f9f04152350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:18:18.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.112+0000 7f9f09400640 1 --2- 172.21.15.28:0/3221733943 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f04150240 0x7f9f04152350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:18:18.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.112+0000 7f9f09400640 1 -- 172.21.15.28:0/3221733943 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f041eb2c0 con 0x7f9f040c2c10 2024-02-21T23:18:18.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.112+0000 7f9f09400640 1 --2- 172.21.15.28:0/3221733943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f040c2c10 0x7f9f041e16a0 secure :-1 s=READY pgs=2800 cs=0 l=1 rev1=1 crypto rx=0x7f9ef802b930 tx=0x7f9ef80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:18:18.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.112+0000 7f9f01600640 1 -- 172.21.15.28:0/3221733943 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ef8040070 con 0x7f9f040c2c10 2024-02-21T23:18:18.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.112+0000 7f9f01600640 1 -- 172.21.15.28:0/3221733943 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ef803e070 con 0x7f9f040c2c10 2024-02-21T23:18:18.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.112+0000 7f9f01600640 1 -- 172.21.15.28:0/3221733943 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ef8035040 con 0x7f9f040c2c10 2024-02-21T23:18:18.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.112+0000 7f9f0a800640 1 -- 172.21.15.28:0/3221733943 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f041eb5a0 con 0x7f9f040c2c10 2024-02-21T23:18:18.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.113+0000 7f9f0a800640 1 -- 172.21.15.28:0/3221733943 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f040a9a60 con 0x7f9f040c2c10 2024-02-21T23:18:18.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.114+0000 7f9f0a800640 1 -- 172.21.15.28:0/3221733943 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f0404f290 con 0x7f9f040c2c10 2024-02-21T23:18:18.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.115+0000 7f9f01600640 1 -- 172.21.15.28:0/3221733943 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9ef8038020 con 0x7f9f040c2c10 2024-02-21T23:18:18.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.116+0000 7f9f01600640 1 --2- 172.21.15.28:0/3221733943 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9ee806b4d0 0x7f9ee806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:18:18.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.116+0000 7f9f01600640 1 -- 172.21.15.28:0/3221733943 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 5593+0+0 (secure 0 0 0) 0x7f9ef80abaf0 con 0x7f9f040c2c10 2024-02-21T23:18:18.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.117+0000 7f9f08a00640 1 --2- 172.21.15.28:0/3221733943 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9ee806b4d0 0x7f9ee806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:18:18.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.118+0000 7f9f08a00640 1 --2- 172.21.15.28:0/3221733943 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9ee806b4d0 0x7f9ee806d990 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f9ef4000f60 tx=0x7f9ef40079d0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:18:18.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.120+0000 7f9f01600640 1 -- 172.21.15.28:0/3221733943 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7f9ef807a9d0 con 0x7f9f040c2c10 2024-02-21T23:18:18.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.312+0000 7f9f0a800640 1 -- 172.21.15.28:0/3221733943 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f9f040a9c60 con 0x7f9f040c2c10 2024-02-21T23:18:18.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.317+0000 7f9f01600640 1 -- 172.21.15.28:0/3221733943 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v65)=0 v65) v1 ==== 123+0+0 (secure 0 0 0) 0x7f9ef807a2d0 con 0x7f9f040c2c10 2024-02-21T23:18:18.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.318+0000 7f9f01600640 1 -- 172.21.15.28:0/3221733943 <== mon.0 v2:172.21.15.28:3300/0 8 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f9ef80b1080 con 0x7f9f040c2c10 2024-02-21T23:18:18.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.321+0000 7f9f0a800640 1 -- 172.21.15.28:0/3221733943 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9ee806b4d0 msgr2=0x7f9ee806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:18:18.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.321+0000 7f9f0a800640 1 --2- 172.21.15.28:0/3221733943 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9ee806b4d0 0x7f9ee806d990 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f9ef4000f60 tx=0x7f9ef40079d0 comp rx=0 tx=0).stop 2024-02-21T23:18:18.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.321+0000 7f9f0a800640 1 -- 172.21.15.28:0/3221733943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f040c2c10 msgr2=0x7f9f041e16a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:18:18.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.321+0000 7f9f0a800640 1 --2- 172.21.15.28:0/3221733943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f040c2c10 0x7f9f041e16a0 secure :-1 s=READY pgs=2800 cs=0 l=1 rev1=1 crypto rx=0x7f9ef802b930 tx=0x7f9ef80046c0 comp rx=0 tx=0).stop 2024-02-21T23:18:18.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.321+0000 7f9f0a800640 1 -- 172.21.15.28:0/3221733943 shutdown_connections 2024-02-21T23:18:18.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.321+0000 7f9f0a800640 1 --2- 172.21.15.28:0/3221733943 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7f9ee806b4d0 0x7f9ee806d990 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:18:18.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.321+0000 7f9f0a800640 1 --2- 172.21.15.28:0/3221733943 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7f9f040c4580 0x7f9f04151e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:18:18.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.321+0000 7f9f0a800640 1 --2- 172.21.15.28:0/3221733943 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f04150240 0x7f9f04152350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:18:18.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.321+0000 7f9f0a800640 1 --2- 172.21.15.28:0/3221733943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f040c2c10 0x7f9f041e16a0 unknown :-1 s=CLOSED pgs=2800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:18:18.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.321+0000 7f9f0a800640 1 -- 172.21.15.28:0/3221733943 >> 172.21.15.28:0/3221733943 conn(0x7f9f040a8530 msgr2=0x7f9f0406ae70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:18:18.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.321+0000 7f9f0a800640 1 -- 172.21.15.28:0/3221733943 shutdown_connections 2024-02-21T23:18:18.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.321+0000 7f9f0a800640 1 -- 172.21.15.28:0/3221733943 wait complete. 2024-02-21T23:18:18.327 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-02-21T23:18:18.327 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-21T23:18:18.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.553+0000 7faedf200640 1 Processor -- start 2024-02-21T23:18:18.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.553+0000 7faedf200640 1 -- start start 2024-02-21T23:18:18.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.554+0000 7faedf200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7faee006ee40 0x7faee00a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:18:18.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.554+0000 7faedf200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faee0156af0 0x7faee006dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:18:18.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.554+0000 7faedf200640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faee00c2760 con 0x7faee0150450 2024-02-21T23:18:18.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.554+0000 7faedf200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faee00c22c0 con 0x7faee006ee40 2024-02-21T23:18:18.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.554+0000 7faedf200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faee0156660 con 0x7faee0156af0 2024-02-21T23:18:18.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.555+0000 7faedd400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7faee006ee40 0x7faee00a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:18:18.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.555+0000 7faedd400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7faee006ee40 0x7faee00a5fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36036/0 (socket says 172.21.15.28:36036) 2024-02-21T23:18:18.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.555+0000 7faedd400640 1 -- 172.21.15.28:0/1399928498 learned_addr learned my addr 172.21.15.28:0/1399928498 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:18:18.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.555+0000 7faede800640 1 --2- 172.21.15.28:0/1399928498 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faee0156af0 0x7faee006dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:18:18.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.555+0000 7faedd400640 1 -- 172.21.15.28:0/1399928498 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faee0156af0 msgr2=0x7faee006dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-21T23:18:18.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.555+0000 7faedd400640 1 --2- 172.21.15.28:0/1399928498 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faee0156af0 0x7faee006dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:18:18.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.556+0000 7faedd400640 1 -- 172.21.15.28:0/1399928498 >> v1:172.21.15.81:6789/0 conn(0x7faee0150450 legacy=0x7faee01536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:18:18.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.556+0000 7faedd400640 1 -- 172.21.15.28:0/1399928498 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faee0156310 con 0x7faee006ee40 2024-02-21T23:18:18.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.557+0000 7faedd400640 1 --2- 172.21.15.28:0/1399928498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faee006ee40 0x7faee00a5fd0 secure :-1 s=READY pgs=2801 cs=0 l=1 rev1=1 crypto rx=0x7faecc009870 tx=0x7faecc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7381d87ad58f109 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:18:18.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.558+0000 7faedca00640 1 -- 172.21.15.28:0/1399928498 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faecc034070 con 0x7faee006ee40 2024-02-21T23:18:18.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.558+0000 7faedca00640 1 -- 172.21.15.28:0/1399928498 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7faecc030840 con 0x7faee006ee40 2024-02-21T23:18:18.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.558+0000 7faedca00640 1 -- 172.21.15.28:0/1399928498 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faecc030c30 con 0x7faee006ee40 2024-02-21T23:18:18.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.558+0000 7faedf200640 1 -- 172.21.15.28:0/1399928498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faee006ee40 msgr2=0x7faee00a5fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:18:18.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.558+0000 7faedf200640 1 --2- 172.21.15.28:0/1399928498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faee006ee40 0x7faee00a5fd0 secure :-1 s=READY pgs=2801 cs=0 l=1 rev1=1 crypto rx=0x7faecc009870 tx=0x7faecc02b980 comp rx=0 tx=0).stop 2024-02-21T23:18:18.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.559+0000 7faedf200640 1 -- 172.21.15.28:0/1399928498 shutdown_connections 2024-02-21T23:18:18.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.559+0000 7faedf200640 1 --2- 172.21.15.28:0/1399928498 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faee0156af0 0x7faee006dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:18:18.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.559+0000 7faedf200640 1 --2- 172.21.15.28:0/1399928498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faee006ee40 0x7faee00a5fd0 unknown :-1 s=CLOSED pgs=2801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:18:18.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.559+0000 7faedf200640 1 -- 172.21.15.28:0/1399928498 >> 172.21.15.28:0/1399928498 conn(0x7faee006b630 msgr2=0x7faee00c6500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:18:18.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.559+0000 7faedf200640 1 -- 172.21.15.28:0/1399928498 shutdown_connections 2024-02-21T23:18:18.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.559+0000 7faedf200640 1 -- 172.21.15.28:0/1399928498 wait complete. 2024-02-21T23:18:18.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.560+0000 7faedf200640 1 Processor -- start 2024-02-21T23:18:18.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.560+0000 7faedf200640 1 -- start start 2024-02-21T23:18:18.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.561+0000 7faedf200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faee006ee40 0x7faee0155040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:18:18.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.561+0000 7faedf200640 1 --2- >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faee0150450 0x7faee00c56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:18:18.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.561+0000 7faedf200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faee0156af0 0x7faee00c5be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:18:18.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.561+0000 7faedf200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faee0156f80 con 0x7faee006ee40 2024-02-21T23:18:18.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.561+0000 7faedf200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faee00c2cb0 con 0x7faee0156af0 2024-02-21T23:18:18.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.561+0000 7faedf200640 1 -- --> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faee0157100 con 0x7faee0150450 2024-02-21T23:18:18.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.562+0000 7faedde00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faee006ee40 0x7faee0155040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:18:18.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.562+0000 7faedde00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faee006ee40 0x7faee0155040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36050/0 (socket says 172.21.15.28:36050) 2024-02-21T23:18:18.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.562+0000 7faedde00640 1 -- 172.21.15.28:0/3107879345 learned_addr learned my addr 172.21.15.28:0/3107879345 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-02-21T23:18:18.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.562+0000 7faede800640 1 --2- 172.21.15.28:0/3107879345 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faee0156af0 0x7faee00c5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:18:18.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.562+0000 7faedd400640 1 --2- 172.21.15.28:0/3107879345 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faee0150450 0x7faee00c56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:18:18.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.562+0000 7faedde00640 1 -- 172.21.15.28:0/3107879345 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faee0150450 msgr2=0x7faee00c56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:18:18.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.562+0000 7faedde00640 1 --2- 172.21.15.28:0/3107879345 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faee0150450 0x7faee00c56a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:18:18.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.562+0000 7faedde00640 1 -- 172.21.15.28:0/3107879345 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faee0156af0 msgr2=0x7faee00c5be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:18:18.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.562+0000 7faedde00640 1 --2- 172.21.15.28:0/3107879345 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faee0156af0 0x7faee00c5be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:18:18.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.562+0000 7faedde00640 1 -- 172.21.15.28:0/3107879345 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faee01eba20 con 0x7faee006ee40 2024-02-21T23:18:18.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.562+0000 7faede800640 1 --2- 172.21.15.28:0/3107879345 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faee0156af0 0x7faee00c5be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-21T23:18:18.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.563+0000 7faedde00640 1 --2- 172.21.15.28:0/3107879345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faee006ee40 0x7faee0155040 secure :-1 s=READY pgs=2802 cs=0 l=1 rev1=1 crypto rx=0x7faed000c430 tx=0x7faed000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:18:18.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.563+0000 7faed6000640 1 -- 172.21.15.28:0/3107879345 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed00052d0 con 0x7faee006ee40 2024-02-21T23:18:18.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.563+0000 7faed6000640 1 -- 172.21.15.28:0/3107879345 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7faed0005490 con 0x7faee006ee40 2024-02-21T23:18:18.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.563+0000 7faed6000640 1 -- 172.21.15.28:0/3107879345 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed0007280 con 0x7faee006ee40 2024-02-21T23:18:18.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.563+0000 7faedf200640 1 -- 172.21.15.28:0/3107879345 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faee00c48e0 con 0x7faee006ee40 2024-02-21T23:18:18.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.563+0000 7faedf200640 1 -- 172.21.15.28:0/3107879345 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faee00c4ea0 con 0x7faee006ee40 2024-02-21T23:18:18.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.564+0000 7faedf200640 1 -- 172.21.15.28:0/3107879345 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faee004f290 con 0x7faee006ee40 2024-02-21T23:18:18.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.565+0000 7faed6000640 1 -- 172.21.15.28:0/3107879345 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faed0007420 con 0x7faee006ee40 2024-02-21T23:18:18.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.566+0000 7faed6000640 1 --2- 172.21.15.28:0/3107879345 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7faec006b4d0 0x7faec006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-21T23:18:18.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.566+0000 7faed6000640 1 -- 172.21.15.28:0/3107879345 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 5593+0+0 (secure 0 0 0) 0x7faed0089620 con 0x7faee006ee40 2024-02-21T23:18:18.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.567+0000 7faedd400640 1 --2- 172.21.15.28:0/3107879345 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7faec006b4d0 0x7faec006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-21T23:18:18.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.568+0000 7faedd400640 1 --2- 172.21.15.28:0/3107879345 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7faec006b4d0 0x7faec006d990 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7faecc036000 tx=0x7faecc03b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-21T23:18:18.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.570+0000 7faed6000640 1 -- 172.21.15.28:0/3107879345 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183725 (secure 0 0 0) 0x7faed0058580 con 0x7faee006ee40 2024-02-21T23:18:18.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.799+0000 7faedf200640 1 -- 172.21.15.28:0/3107879345 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faee00c1f10 con 0x7faee006ee40 2024-02-21T23:18:18.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.800+0000 7faed6000640 1 -- 172.21.15.28:0/3107879345 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 249 v249) v1 ==== 95+0+1329 (secure 0 0 0) 0x7faed0057e80 con 0x7faee006ee40 2024-02-21T23:18:18.788 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-21T23:18:18.788 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":249,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11850,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2326968847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2326968847},{"type":"v1","addr":"172.21.15.28:6835","nonce":2326968847}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":249}],"filesystems":[]} 2024-02-21T23:18:18.788 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 249 2024-02-21T23:18:18.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.802+0000 7faedf200640 1 -- 172.21.15.28:0/3107879345 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7faec006b4d0 msgr2=0x7faec006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:18:18.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.802+0000 7faedf200640 1 --2- 172.21.15.28:0/3107879345 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7faec006b4d0 0x7faec006d990 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7faecc036000 tx=0x7faecc03b040 comp rx=0 tx=0).stop 2024-02-21T23:18:18.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.802+0000 7faedf200640 1 -- 172.21.15.28:0/3107879345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faee006ee40 msgr2=0x7faee0155040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-21T23:18:18.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.802+0000 7faedf200640 1 --2- 172.21.15.28:0/3107879345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faee006ee40 0x7faee0155040 secure :-1 s=READY pgs=2802 cs=0 l=1 rev1=1 crypto rx=0x7faed000c430 tx=0x7faed000c8f0 comp rx=0 tx=0).stop 2024-02-21T23:18:18.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.802+0000 7faedf200640 1 -- 172.21.15.28:0/3107879345 shutdown_connections 2024-02-21T23:18:18.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.802+0000 7faedf200640 1 --2- 172.21.15.28:0/3107879345 >> [v2:172.21.15.81:6832/1848538577,v1:172.21.15.81:6833/1848538577] conn(0x7faec006b4d0 0x7faec006d990 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:18:18.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.802+0000 7faedf200640 1 --2- 172.21.15.28:0/3107879345 >> [v2:172.21.15.81:3301/0,v1:172.21.15.81:6790/0] conn(0x7faee0150450 0x7faee00c56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:18:18.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.802+0000 7faedf200640 1 --2- 172.21.15.28:0/3107879345 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faee0156af0 0x7faee00c5be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:18:18.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.802+0000 7faedf200640 1 --2- 172.21.15.28:0/3107879345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faee006ee40 0x7faee0155040 unknown :-1 s=CLOSED pgs=2802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-21T23:18:18.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.802+0000 7faedf200640 1 -- 172.21.15.28:0/3107879345 >> 172.21.15.28:0/3107879345 conn(0x7faee006b630 msgr2=0x7faee00c01c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-21T23:18:18.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.802+0000 7faedf200640 1 -- 172.21.15.28:0/3107879345 shutdown_connections 2024-02-21T23:18:18.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-02-21T23:18:18.802+0000 7faedf200640 1 -- 172.21.15.28:0/3107879345 wait complete. 2024-02-21T23:18:18.810 DEBUG:tasks.cephfs.filesystem:already dead... 2024-02-21T23:18:18.810 INFO:teuthology.misc:Shutting down mds daemons... 2024-02-21T23:18:18.810 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-02-21T23:18:18.810 INFO:teuthology.orchestra.run:waiting for 300 2024-02-21T23:18:18.828 INFO:tasks.ceph.mds.a:Stopped 2024-02-21T23:18:18.828 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-02-21T23:18:18.828 INFO:teuthology.orchestra.run:waiting for 300 2024-02-21T23:18:18.900 INFO:tasks.ceph.mds.c:Stopped 2024-02-21T23:18:18.900 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-02-21T23:18:18.901 INFO:teuthology.orchestra.run:waiting for 300 2024-02-21T23:18:18.920 INFO:tasks.ceph.mds.b:Stopped 2024-02-21T23:18:18.920 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-02-21T23:18:18.921 INFO:teuthology.orchestra.run:waiting for 300 2024-02-21T23:18:18.988 INFO:tasks.ceph.mds.d:Stopped 2024-02-21T23:18:18.988 INFO:teuthology.misc:Shutting down osd daemons... 2024-02-21T23:18:18.989 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-02-21T23:18:18.989 INFO:teuthology.orchestra.run:waiting for 300 2024-02-21T23:18:19.060 INFO:tasks.ceph.osd.0:Stopped 2024-02-21T23:18:19.060 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-02-21T23:18:19.060 INFO:teuthology.orchestra.run:waiting for 300 2024-02-21T23:18:19.126 INFO:tasks.ceph.osd.1:Stopped 2024-02-21T23:18:19.127 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-02-21T23:18:19.127 INFO:teuthology.orchestra.run:waiting for 300 2024-02-21T23:18:19.183 INFO:tasks.ceph.osd.2:Stopped 2024-02-21T23:18:19.183 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-02-21T23:18:19.183 INFO:teuthology.orchestra.run:waiting for 300 2024-02-21T23:18:19.252 INFO:tasks.ceph.osd.3:Stopped 2024-02-21T23:18:19.253 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-02-21T23:18:19.253 INFO:teuthology.orchestra.run:waiting for 300 2024-02-21T23:18:19.323 INFO:tasks.ceph.osd.4:Stopped 2024-02-21T23:18:19.323 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-02-21T23:18:19.324 INFO:teuthology.orchestra.run:waiting for 300 2024-02-21T23:18:19.386 INFO:tasks.ceph.osd.5:Stopped 2024-02-21T23:18:19.387 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-02-21T23:18:19.387 INFO:teuthology.orchestra.run:waiting for 300 2024-02-21T23:18:19.460 INFO:tasks.ceph.osd.6:Stopped 2024-02-21T23:18:19.460 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-02-21T23:18:19.460 INFO:teuthology.orchestra.run:waiting for 300 2024-02-21T23:18:19.517 INFO:tasks.ceph.osd.7:Stopped 2024-02-21T23:18:19.517 INFO:teuthology.misc:Shutting down mgr daemons... 2024-02-21T23:18:19.518 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-02-21T23:18:19.518 INFO:teuthology.orchestra.run:waiting for 300 2024-02-21T23:18:19.607 INFO:tasks.ceph.mgr.y:Stopped 2024-02-21T23:18:19.608 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-02-21T23:18:19.608 INFO:teuthology.orchestra.run:waiting for 300 2024-02-21T23:18:19.691 INFO:tasks.ceph.mgr.x:Stopped 2024-02-21T23:18:19.692 INFO:teuthology.misc:Shutting down mon daemons... 2024-02-21T23:18:19.692 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-02-21T23:18:19.692 INFO:teuthology.orchestra.run:waiting for 300 2024-02-21T23:18:19.741 INFO:tasks.ceph.mon.a:Stopped 2024-02-21T23:18:19.742 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-02-21T23:18:19.742 INFO:teuthology.orchestra.run:waiting for 300 2024-02-21T23:18:19.770 INFO:tasks.ceph.mon.b:Stopped 2024-02-21T23:18:19.770 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-02-21T23:18:19.770 INFO:teuthology.orchestra.run:waiting for 300 2024-02-21T23:18:19.856 INFO:tasks.ceph.mon.c:Stopped 2024-02-21T23:18:19.857 INFO:tasks.ceph:Checking cluster log for badness... 2024-02-21T23:18:19.857 DEBUG:teuthology.orchestra.run.smithi028:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'Replacing daemon mds' | egrep -v '\(MDS_INSUFFICIENT_STANDBY\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(FS_DEGRADED\)' | head -n 1 2024-02-21T23:18:19.969 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi028.front.sepia.ceph.com 2024-02-21T23:18:19.969 DEBUG:teuthology.orchestra.run.smithi028:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-02-21T23:18:20.220 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi028.front.sepia.ceph.com 2024-02-21T23:18:20.221 DEBUG:teuthology.orchestra.run.smithi028:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-02-21T23:18:20.381 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi028.front.sepia.ceph.com 2024-02-21T23:18:20.381 DEBUG:teuthology.orchestra.run.smithi028:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-02-21T23:18:20.559 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi028.front.sepia.ceph.com 2024-02-21T23:18:20.560 DEBUG:teuthology.orchestra.run.smithi028:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-02-21T23:18:20.734 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi081.front.sepia.ceph.com 2024-02-21T23:18:20.735 DEBUG:teuthology.orchestra.run.smithi081:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-02-21T23:18:21.021 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi081.front.sepia.ceph.com 2024-02-21T23:18:21.022 DEBUG:teuthology.orchestra.run.smithi081:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-02-21T23:18:21.175 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi081.front.sepia.ceph.com 2024-02-21T23:18:21.176 DEBUG:teuthology.orchestra.run.smithi081:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-02-21T23:18:21.326 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi081.front.sepia.ceph.com 2024-02-21T23:18:21.326 DEBUG:teuthology.orchestra.run.smithi081:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-02-21T23:18:21.482 INFO:tasks.ceph:Archiving mon data... 2024-02-21T23:18:21.483 DEBUG:teuthology.misc:Transferring archived files from smithi028:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/leonidus-2024-02-21_22:31:03-fs-wip-lusov-qdb-distro-default-smithi/7570010/data/mon.a.tgz 2024-02-21T23:18:21.483 DEBUG:teuthology.orchestra.run.smithi028:> mktemp 2024-02-21T23:18:21.542 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/tmp.LJhTkJhInd 2024-02-21T23:18:21.543 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.LJhTkJhInd 2024-02-21T23:18:21.773 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 0666 /tmp/tmp.LJhTkJhInd 2024-02-21T23:18:21.933 DEBUG:teuthology.orchestra.remote:smithi028:/tmp/tmp.LJhTkJhInd is 495KB 2024-02-21T23:18:22.050 DEBUG:teuthology.orchestra.run.smithi028:> rm -fr /tmp/tmp.LJhTkJhInd 2024-02-21T23:18:22.109 DEBUG:teuthology.misc:Transferring archived files from smithi081:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/leonidus-2024-02-21_22:31:03-fs-wip-lusov-qdb-distro-default-smithi/7570010/data/mon.b.tgz 2024-02-21T23:18:22.110 DEBUG:teuthology.orchestra.run.smithi081:> mktemp 2024-02-21T23:18:22.167 INFO:teuthology.orchestra.run.smithi081.stdout:/tmp/tmp.tO0aKsNBfC 2024-02-21T23:18:22.168 DEBUG:teuthology.orchestra.run.smithi081:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.tO0aKsNBfC 2024-02-21T23:18:22.396 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 0666 /tmp/tmp.tO0aKsNBfC 2024-02-21T23:18:22.539 DEBUG:teuthology.orchestra.remote:smithi081:/tmp/tmp.tO0aKsNBfC is 500KB 2024-02-21T23:18:22.651 DEBUG:teuthology.orchestra.run.smithi081:> rm -fr /tmp/tmp.tO0aKsNBfC 2024-02-21T23:18:22.706 DEBUG:teuthology.misc:Transferring archived files from smithi081:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/leonidus-2024-02-21_22:31:03-fs-wip-lusov-qdb-distro-default-smithi/7570010/data/mon.c.tgz 2024-02-21T23:18:22.707 DEBUG:teuthology.orchestra.run.smithi081:> mktemp 2024-02-21T23:18:22.756 INFO:teuthology.orchestra.run.smithi081.stdout:/tmp/tmp.puuJEhgn6R 2024-02-21T23:18:22.757 DEBUG:teuthology.orchestra.run.smithi081:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.puuJEhgn6R 2024-02-21T23:18:22.960 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 0666 /tmp/tmp.puuJEhgn6R 2024-02-21T23:18:23.106 DEBUG:teuthology.orchestra.remote:smithi081:/tmp/tmp.puuJEhgn6R is 500KB 2024-02-21T23:18:23.216 DEBUG:teuthology.orchestra.run.smithi081:> rm -fr /tmp/tmp.puuJEhgn6R 2024-02-21T23:18:23.270 INFO:tasks.ceph:Cleaning ceph cluster... 2024-02-21T23:18:23.270 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-02-21T23:18:23.278 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-02-21T23:18:23.439 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-02-21T23:18:23.496 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-02-21T23:18:23.497 INFO:tasks.ceph:Archiving crash dumps... 2024-02-21T23:18:23.499 DEBUG:teuthology.misc:Transferring archived files from smithi028:/var/lib/ceph/crash to /home/teuthworker/archive/leonidus-2024-02-21_22:31:03-fs-wip-lusov-qdb-distro-default-smithi/7570010/remote/smithi028/crash 2024-02-21T23:18:23.499 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-02-21T23:18:23.600 DEBUG:teuthology.misc:Transferring archived files from smithi081:/var/lib/ceph/crash to /home/teuthworker/archive/leonidus-2024-02-21_22:31:03-fs-wip-lusov-qdb-distro-default-smithi/7570010/remote/smithi081/crash 2024-02-21T23:18:23.601 DEBUG:teuthology.orchestra.run.smithi081:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-02-21T23:18:23.698 INFO:tasks.ceph:Shutting down logrotate 2024-02-21T23:18:23.699 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-21T23:18:23.707 DEBUG:teuthology.orchestra.run.smithi081:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-02-21T23:18:23.840 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-02-21T23:18:23.939 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-02-21T23:18:24.039 INFO:tasks.ceph:Compressing logs... 2024-02-21T23:18:24.040 DEBUG:teuthology.orchestra.run.smithi028:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-21T23:18:24.048 DEBUG:teuthology.orchestra.run.smithi081:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-21T23:18:24.135 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-02-21T23:18:24.135 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36640.log 2024-02-21T23:18:24.137 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36800.log 2024-02-21T23:18:24.138 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.36640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36760.log 2024-02-21T23:18:24.138 INFO:teuthology.orchestra.run.smithi081.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.36640.log.gz 2024-02-21T23:18:24.139 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.36800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36840.log 2024-02-21T23:18:24.139 INFO:teuthology.orchestra.run.smithi081.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.36800.log.gz 2024-02-21T23:18:24.140 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.36760.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-02-21T23:18:24.140 INFO:teuthology.orchestra.run.smithi081.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.36760.log.gz 2024-02-21T23:18:24.141 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.36840.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-02-21T23:18:24.141 INFO:teuthology.orchestra.run.smithi081.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.36840.log.gz 2024-02-21T23:18:24.142 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-02-21T23:18:24.142 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47750.log 2024-02-21T23:18:24.143 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-02-21T23:18:24.144 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59135.log 2024-02-21T23:18:24.145 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-02-21T23:18:24.146 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40740.log 2024-02-21T23:18:24.146 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36720.log 2024-02-21T23:18:24.147 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.47750.log: /var/log/ceph/ceph-client.admin.59135.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.59135.log.gz 2024-02-21T23:18:24.147 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47750.log.gz 2024-02-21T23:18:24.147 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73678.log 2024-02-21T23:18:24.149 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.40740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55896.log 2024-02-21T23:18:24.149 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40740.log.gz 2024-02-21T23:18:24.149 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-02-21T23:18:24.149 INFO:teuthology.orchestra.run.smithi081.stderr: 93.9% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-02-21T23:18:24.149 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.36720.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.36720.log.gz 2024-02-21T23:18:24.151 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.73678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50570.log 2024-02-21T23:18:24.151 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.73678.log.gz 2024-02-21T23:18:24.151 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.55896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52638.log 2024-02-21T23:18:24.151 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55896.log.gz 2024-02-21T23:18:24.152 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36920.log 2024-02-21T23:18:24.152 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.50570.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62122.log 2024-02-21T23:18:24.153 INFO:teuthology.orchestra.run.smithi028.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.50570.log.gz 2024-02-21T23:18:24.153 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-02-21T23:18:24.154 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.52638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75218.log 2024-02-21T23:18:24.154 INFO:teuthology.orchestra.run.smithi028.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.52638.log.gz 2024-02-21T23:18:24.155 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.36920.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.36920.log.gz 2024-02-21T23:18:24.156 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.62122.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.62122.log.gz 2024-02-21T23:18:24.156 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57618.log 2024-02-21T23:18:24.156 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.75218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44339.log 2024-02-21T23:18:24.156 INFO:teuthology.orchestra.run.smithi028.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.75218.log.gz 2024-02-21T23:18:24.157 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-02-21T23:18:24.157 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.57618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81987.log 2024-02-21T23:18:24.158 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57618.log.gz 2024-02-21T23:18:24.160 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36880.log 2024-02-21T23:18:24.160 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.44339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49400.log 2024-02-21T23:18:24.160 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44339.log.gz 2024-02-21T23:18:24.161 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.81987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84520.log 2024-02-21T23:18:24.161 INFO:teuthology.orchestra.run.smithi028.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.81987.log.gz 2024-02-21T23:18:24.161 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.49400.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49400.log.gz 2024-02-21T23:18:24.161 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69679.log 2024-02-21T23:18:24.163 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.36880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36680.log 92.9% -- replaced with /var/log/ceph/ceph.log.gz 2024-02-21T23:18:24.164 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-02-21T23:18:24.164 INFO:teuthology.orchestra.run.smithi081.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.36880.log.gz 2024-02-21T23:18:24.165 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.84520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83704.log 2024-02-21T23:18:24.165 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.84520.log.gz 2024-02-21T23:18:24.165 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.69679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58642.log 2024-02-21T23:18:24.165 INFO:teuthology.orchestra.run.smithi028.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.69679.log.gz 2024-02-21T23:18:24.165 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.83704.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.83704.log.gz 2024-02-21T23:18:24.165 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71436.log 2024-02-21T23:18:24.166 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.58642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51500.log 2024-02-21T23:18:24.166 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.58642.log.gz 2024-02-21T23:18:24.167 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-02-21T23:18:24.168 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.71436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63710.log 2024-02-21T23:18:24.168 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.71436.log.gz 2024-02-21T23:18:24.169 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.51500.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-02-21T23:18:24.169 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51500.log.gz 2024-02-21T23:18:24.170 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-mds.d.log: /var/log/ceph/ceph-client.admin.36680.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.36680.log.gz 2024-02-21T23:18:24.170 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.41242.log 2024-02-21T23:18:24.170 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.63710.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.63710.log.gz 2024-02-21T23:18:24.170 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53351.log 2024-02-21T23:18:24.171 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-02-21T23:18:24.171 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55242.log 2024-02-21T23:18:24.172 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.53351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62579.log 2024-02-21T23:18:24.173 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53351.log.gz 2024-02-21T23:18:24.173 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.55242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56603.log 2024-02-21T23:18:24.174 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.62579.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55242.log.gz 2024-02-21T23:18:24.175 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59818.log 2024-02-21T23:18:24.175 INFO:teuthology.orchestra.run.smithi028.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.62579.log.gz 2024-02-21T23:18:24.176 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.1.41242.log: /var/log/ceph/ceph-mon.c.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.41242.log.gz 2024-02-21T23:18:24.176 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.56603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61614.log 2024-02-21T23:18:24.177 INFO:teuthology.orchestra.run.smithi028.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.56603.log.gz 2024-02-21T23:18:24.177 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.59818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74368.log 2024-02-21T23:18:24.178 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.59818.log.gz 2024-02-21T23:18:24.179 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.61614.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.61614.log.gz 2024-02-21T23:18:24.179 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43925.log 2024-02-21T23:18:24.180 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.74368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60982.log 2024-02-21T23:18:24.180 INFO:teuthology.orchestra.run.smithi028.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.74368.log.gz 2024-02-21T23:18:24.181 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.43925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67186.log 2024-02-21T23:18:24.182 INFO:teuthology.orchestra.run.smithi028.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43925.log.gz 2024-02-21T23:18:24.183 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.60982.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.60982.log.gz 2024-02-21T23:18:24.183 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67144.log 2024-02-21T23:18:24.184 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.67186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83494.log 2024-02-21T23:18:24.184 INFO:teuthology.orchestra.run.smithi028.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.67186.log.gz 2024-02-21T23:18:24.185 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.67144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49584.log 2024-02-21T23:18:24.185 INFO:teuthology.orchestra.run.smithi028.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.67144.log.gz 2024-02-21T23:18:24.187 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.83494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51752.log 2024-02-21T23:18:24.187 INFO:teuthology.orchestra.run.smithi028.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.83494.log.gz 2024-02-21T23:18:24.188 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.49584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51944.log 2024-02-21T23:18:24.188 INFO:teuthology.orchestra.run.smithi028.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.49584.log.gz 2024-02-21T23:18:24.189 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.51752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78543.log 2024-02-21T23:18:24.189 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51752.log.gz 2024-02-21T23:18:24.191 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.51944.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75714.log 2024-02-21T23:18:24.191 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51944.log.gz 2024-02-21T23:18:24.192 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.78543.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.78543.log.gz 2024-02-21T23:18:24.192 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65036.log 2024-02-21T23:18:24.193 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.75714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77799.log 2024-02-21T23:18:24.193 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.75714.log.gz 2024-02-21T23:18:24.194 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.65036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79220.log 2024-02-21T23:18:24.194 INFO:teuthology.orchestra.run.smithi028.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.65036.log.gz 2024-02-21T23:18:24.195 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.77799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81507.log 88.5% 2024-02-21T23:18:24.195 INFO:teuthology.orchestra.run.smithi028.stderr: -- replaced with /var/log/ceph/ceph-client.admin.77799.log.gz 2024-02-21T23:18:24.197 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.79220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86650.log 2024-02-21T23:18:24.197 INFO:teuthology.orchestra.run.smithi028.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.79220.log.gz 2024-02-21T23:18:24.198 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.81507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82492.log 2024-02-21T23:18:24.199 INFO:teuthology.orchestra.run.smithi028.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.81507.log.gz 2024-02-21T23:18:24.199 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.86650.log: gzip -5 89.1% --verbose -- replaced with /var/log/ceph/ceph-client.admin.86650.log.gz 2024-02-21T23:18:24.199 INFO:teuthology.orchestra.run.smithi028.stderr: -- /var/log/ceph/ceph-client.admin.76406.log 2024-02-21T23:18:24.201 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.82492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68623.log 2024-02-21T23:18:24.201 INFO:teuthology.orchestra.run.smithi028.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.82492.log.gz 2024-02-21T23:18:24.202 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.76406.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.76406.log.gz 2024-02-21T23:18:24.202 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64928.log 2024-02-21T23:18:24.203 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.68623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60940.log 2024-02-21T23:18:24.204 INFO:teuthology.orchestra.run.smithi028.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.68623.log.gz 2024-02-21T23:18:24.205 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.64928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68833.log 2024-02-21T23:18:24.205 INFO:teuthology.orchestra.run.smithi028.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.64928.log.gz 2024-02-21T23:18:24.206 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.60940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80073.log 2024-02-21T23:18:24.206 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.60940.log.gz 2024-02-21T23:18:24.207 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.68833.log: gzip -5 --verbose -- 88.0% /var/log/ceph/ceph-client.admin.76322.log 2024-02-21T23:18:24.207 INFO:teuthology.orchestra.run.smithi028.stderr: -- replaced with /var/log/ceph/ceph-client.admin.68833.log.gz 2024-02-21T23:18:24.208 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.80073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44507.log 2024-02-21T23:18:24.209 INFO:teuthology.orchestra.run.smithi028.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.80073.log.gz 2024-02-21T23:18:24.210 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.76322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44381.log 2024-02-21T23:18:24.210 INFO:teuthology.orchestra.run.smithi028.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.76322.log.gz 2024-02-21T23:18:24.211 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.44507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80614.log 2024-02-21T23:18:24.212 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44507.log.gz 2024-02-21T23:18:24.213 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.44381.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44381.log.gz 2024-02-21T23:18:24.213 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86860.log 2024-02-21T23:18:24.214 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.80614.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.80614.log.gz 2024-02-21T23:18:24.214 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71520.log 2024-02-21T23:18:24.215 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.86860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74593.log 2024-02-21T23:18:24.215 INFO:teuthology.orchestra.run.smithi028.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.86860.log.gz 2024-02-21T23:18:24.216 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.71520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61521.log 2024-02-21T23:18:24.217 INFO:teuthology.orchestra.run.smithi028.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.71520.log.gz 2024-02-21T23:18:24.218 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.74593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43585.log 2024-02-21T23:18:24.219 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.61521.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.74593.log.gz 2024-02-21T23:18:24.219 INFO:teuthology.orchestra.run.smithi028.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.61521.log.gz 2024-02-21T23:18:24.219 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57324.log 2024-02-21T23:18:24.220 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.43585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75261.log 2024-02-21T23:18:24.221 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.43585.log.gz 2024-02-21T23:18:24.222 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.57324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64544.log 2024-02-21T23:18:24.222 INFO:teuthology.orchestra.run.smithi028.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.57324.log.gz 2024-02-21T23:18:24.223 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.75261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55103.log 2024-02-21T23:18:24.223 INFO:teuthology.orchestra.run.smithi028.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.75261.log.gz 2024-02-21T23:18:24.224 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.64544.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.64544.log.gz 2024-02-21T23:18:24.230 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64736.log 2024-02-21T23:18:24.230 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.55103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78585.log 2024-02-21T23:18:24.230 INFO:teuthology.orchestra.run.smithi028.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.55103.log.gz 2024-02-21T23:18:24.230 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.64736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73720.log 2024-02-21T23:18:24.230 INFO:teuthology.orchestra.run.smithi028.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.64736.log.gz 2024-02-21T23:18:24.230 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.78585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85438.log 2024-02-21T23:18:24.230 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.78585.log.gz 2024-02-21T23:18:24.231 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.73720.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.73720.log.gz 2024-02-21T23:18:24.231 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47487.log 2024-02-21T23:18:24.232 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.85438.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59500.log 2024-02-21T23:18:24.232 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.85438.log.gz 2024-02-21T23:18:24.233 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.47487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69507.log 2024-02-21T23:18:24.233 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47487.log.gz 2024-02-21T23:18:24.234 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.59500.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85270.log 2024-02-21T23:18:24.234 INFO:teuthology.orchestra.run.smithi028.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.59500.log.gz 2024-02-21T23:18:24.236 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.69507.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.69507.log.gz 2024-02-21T23:18:24.236 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83452.log 2024-02-21T23:18:24.237 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.85270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52070.log 2024-02-21T23:18:24.237 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.85270.log.gz 2024-02-21T23:18:24.238 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.83452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41614.log 2024-02-21T23:18:24.238 INFO:teuthology.orchestra.run.smithi028.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.83452.log.gz 2024-02-21T23:18:24.239 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.52070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48167.log 2024-02-21T23:18:24.240 INFO:teuthology.orchestra.run.smithi028.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.52070.log.gz 2024-02-21T23:18:24.241 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.41614.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41614.log.gz 2024-02-21T23:18:24.241 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51454.log 2024-02-21T23:18:24.241 INFO:teuthology.orchestra.run.smithi081.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-02-21T23:18:24.242 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.48167.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54354.log 2024-02-21T23:18:24.243 INFO:teuthology.orchestra.run.smithi028.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.48167.log.gz 2024-02-21T23:18:24.245 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.51454.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.51454.log.gz 2024-02-21T23:18:24.245 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68371.log 2024-02-21T23:18:24.246 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.54354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46164.log 2024-02-21T23:18:24.246 INFO:teuthology.orchestra.run.smithi028.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54354.log.gz 2024-02-21T23:18:24.246 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.68371.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44717.log 2024-02-21T23:18:24.247 INFO:teuthology.orchestra.run.smithi028.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.68371.log.gz 2024-02-21T23:18:24.247 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.46164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74411.log 2024-02-21T23:18:24.248 INFO:teuthology.orchestra.run.smithi028.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46164.log.gz 2024-02-21T23:18:24.249 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.44717.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44717.log.gzgzip 2024-02-21T23:18:24.249 INFO:teuthology.orchestra.run.smithi028.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.83326.log 2024-02-21T23:18:24.250 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.74411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44885.log 2024-02-21T23:18:24.250 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.74411.log.gz 2024-02-21T23:18:24.251 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.83326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53141.log 2024-02-21T23:18:24.251 INFO:teuthology.orchestra.run.smithi028.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.83326.log.gz 2024-02-21T23:18:24.253 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.44885.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47961.log 2024-02-21T23:18:24.253 INFO:teuthology.orchestra.run.smithi028.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.44885.log.gz 2024-02-21T23:18:24.254 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.53141.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.53141.log.gz 2024-02-21T23:18:24.255 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53099.log 2024-02-21T23:18:24.256 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.47961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76808.log 2024-02-21T23:18:24.256 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.47961.log.gz 2024-02-21T23:18:24.257 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.53099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45139.log 2024-02-21T23:18:24.257 INFO:teuthology.orchestra.run.smithi028.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53099.log.gz 2024-02-21T23:18:24.258 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.76808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41232.log 2024-02-21T23:18:24.259 INFO:teuthology.orchestra.run.smithi028.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.76808.log.gz 2024-02-21T23:18:24.260 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.45139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53267.log 2024-02-21T23:18:24.260 INFO:teuthology.orchestra.run.smithi028.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.45139.log.gz 2024-02-21T23:18:24.261 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.41232.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53790.log 2024-02-21T23:18:24.261 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41232.log.gz 2024-02-21T23:18:24.262 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.53267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69595.log 2024-02-21T23:18:24.262 INFO:teuthology.orchestra.run.smithi028.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53267.log.gz 2024-02-21T23:18:24.263 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.53790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81684.log 2024-02-21T23:18:24.264 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53790.log.gz 2024-02-21T23:18:24.265 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.69595.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.69595.log.gz 2024-02-21T23:18:24.265 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76682.log 2024-02-21T23:18:24.266 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.81684.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60291.log 2024-02-21T23:18:24.266 INFO:teuthology.orchestra.run.smithi028.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.81684.log.gz 2024-02-21T23:18:24.267 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.76682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62400.log 2024-02-21T23:18:24.267 INFO:teuthology.orchestra.run.smithi028.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.76682.log.gz 2024-02-21T23:18:24.268 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.60291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87112.log 2024-02-21T23:18:24.269 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.60291.log.gz 2024-02-21T23:18:24.270 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.62400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68875.log 2024-02-21T23:18:24.270 INFO:teuthology.orchestra.run.smithi028.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.62400.log.gz 2024-02-21T23:18:24.271 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.87112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81837.log 2024-02-21T23:18:24.271 INFO:teuthology.orchestra.run.smithi028.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.87112.log.gz 2024-02-21T23:18:24.272 INFO:teuthology.orchestra.run.smithi081.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-02-21T23:18:24.273 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.68875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59710.log 2024-02-21T23:18:24.273 INFO:teuthology.orchestra.run.smithi028.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.68875.log.gz 2024-02-21T23:18:24.274 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.81837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83746.log 2024-02-21T23:18:24.274 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.59710.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.81837.log.gz 2024-02-21T23:18:24.275 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52028.log 2024-02-21T23:18:24.276 INFO:teuthology.orchestra.run.smithi028.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.59710.log.gz 2024-02-21T23:18:24.276 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.83746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85918.log 2024-02-21T23:18:24.276 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.83746.log.gz 2024-02-21T23:18:24.277 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.52028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46905.log 2024-02-21T23:18:24.278 INFO:teuthology.orchestra.run.smithi028.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.52028.log.gz 2024-02-21T23:18:24.279 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.85918.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.85918.log.gz 2024-02-21T23:18:24.279 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62014.log 2024-02-21T23:18:24.280 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.46905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60165.log 2024-02-21T23:18:24.280 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46905.log.gz 2024-02-21T23:18:24.281 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.62014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61786.log 2024-02-21T23:18:24.282 INFO:teuthology.orchestra.run.smithi028.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.62014.log.gz 2024-02-21T23:18:24.283 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.60165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86818.log 2024-02-21T23:18:24.283 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.60165.log.gz 2024-02-21T23:18:24.284 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.61786.log: gzip 89.4% -5 -- replaced with /var/log/ceph/ceph-client.admin.61786.log.gz --verbose 2024-02-21T23:18:24.284 INFO:teuthology.orchestra.run.smithi028.stderr: -- /var/log/ceph/ceph-client.admin.57576.log 2024-02-21T23:18:24.285 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.86818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47193.log 2024-02-21T23:18:24.286 INFO:teuthology.orchestra.run.smithi028.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.86818.log.gz 2024-02-21T23:18:24.286 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.57576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64376.log 2024-02-21T23:18:24.287 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.57576.log.gz 2024-02-21T23:18:24.288 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.47193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75880.log 2024-02-21T23:18:24.288 INFO:teuthology.orchestra.run.smithi028.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.47193.log.gz 2024-02-21T23:18:24.289 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.64376.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.64376.log.gz 2024-02-21T23:18:24.289 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86380.log 2024-02-21T23:18:24.290 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.75880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66028.log 2024-02-21T23:18:24.291 INFO:teuthology.orchestra.run.smithi028.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.75880.log.gz 2024-02-21T23:18:24.292 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.86380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87275.log 2024-02-21T23:18:24.292 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.86380.log.gz 2024-02-21T23:18:24.293 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.66028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48603.log 2024-02-21T23:18:24.293 INFO:teuthology.orchestra.run.smithi028.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.66028.log.gz 2024-02-21T23:18:24.294 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.87275.log: 88.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.87275.log.gz -5 2024-02-21T23:18:24.294 INFO:teuthology.orchestra.run.smithi028.stderr: --verbose -- /var/log/ceph/ceph-client.admin.45602.log 2024-02-21T23:18:24.295 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.48603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66735.log 2024-02-21T23:18:24.296 INFO:teuthology.orchestra.run.smithi028.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.48603.log.gz 2024-02-21T23:18:24.297 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.45602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83056.log 2024-02-21T23:18:24.297 INFO:teuthology.orchestra.run.smithi028.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45602.log.gz 2024-02-21T23:18:24.298 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.66735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59668.log 2024-02-21T23:18:24.298 INFO:teuthology.orchestra.run.smithi028.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.66735.log.gz 2024-02-21T23:18:24.299 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.83056.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.83056.log.gz 2024-02-21T23:18:24.300 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58600.log 2024-02-21T23:18:24.301 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.59668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68665.log 2024-02-21T23:18:24.301 INFO:teuthology.orchestra.run.smithi028.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.59668.log.gz 2024-02-21T23:18:24.302 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.58600.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51902.log 2024-02-21T23:18:24.302 INFO:teuthology.orchestra.run.smithi028.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.58600.log.gz 2024-02-21T23:18:24.303 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.68665.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60898.log 2024-02-21T23:18:24.304 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.68665.log.gz 2024-02-21T23:18:24.305 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.51902.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51902.log.gz 2024-02-21T23:18:24.305 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79947.log 2024-02-21T23:18:24.306 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.60898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50341.log 2024-02-21T23:18:24.306 INFO:teuthology.orchestra.run.smithi028.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.60898.log.gz 2024-02-21T23:18:24.307 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.79947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74990.log 2024-02-21T23:18:24.308 INFO:teuthology.orchestra.run.smithi028.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.79947.log.gz 2024-02-21T23:18:24.309 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.50341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44675.log 2024-02-21T23:18:24.309 INFO:teuthology.orchestra.run.smithi028.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.50341.log.gz 2024-02-21T23:18:24.310 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.74990.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.74990.log.gz 2024-02-21T23:18:24.310 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43247.log 2024-02-21T23:18:24.311 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.44675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42977.log 2024-02-21T23:18:24.312 INFO:teuthology.orchestra.run.smithi028.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44675.log.gz 2024-02-21T23:18:24.312 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.43247.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69241.log 2024-02-21T23:18:24.313 INFO:teuthology.orchestra.run.smithi028.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.43247.log.gz 2024-02-21T23:18:24.314 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.42977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51063.log 2024-02-21T23:18:24.314 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42977.log.gz 2024-02-21T23:18:24.315 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.69241.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.69241.log.gz 2024-02-21T23:18:24.315 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77102.log 2024-02-21T23:18:24.316 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.51063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66563.log 2024-02-21T23:18:24.317 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51063.log.gz 2024-02-21T23:18:24.317 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.77102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88610.log 2024-02-21T23:18:24.318 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.77102.log.gz 2024-02-21T23:18:24.319 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.66563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59179.log 2024-02-21T23:18:24.319 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.66563.log.gz 2024-02-21T23:18:24.320 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.88610.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76892.log 88.6% 2024-02-21T23:18:24.320 INFO:teuthology.orchestra.run.smithi028.stderr: -- replaced with /var/log/ceph/ceph-client.admin.88610.log.gz 2024-02-21T23:18:24.321 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.59179.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86044.log 2024-02-21T23:18:24.322 INFO:teuthology.orchestra.run.smithi028.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.59179.log.gz 2024-02-21T23:18:24.322 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.76892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51584.log 2024-02-21T23:18:24.323 INFO:teuthology.orchestra.run.smithi028.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.76892.log.gz 2024-02-21T23:18:24.324 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.86044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69038.log 2024-02-21T23:18:24.325 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.86044.log.gz 2024-02-21T23:18:24.325 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.51584.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51584.log.gz 2024-02-21T23:18:24.326 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72832.log 2024-02-21T23:18:24.327 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.69038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67805.log 2024-02-21T23:18:24.327 INFO:teuthology.orchestra.run.smithi028.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.69038.log.gz 2024-02-21T23:18:24.328 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.72832.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70916.log 2024-02-21T23:18:24.328 INFO:teuthology.orchestra.run.smithi028.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.72832.log.gz 2024-02-21T23:18:24.330 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.67805.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83200.log 2024-02-21T23:18:24.330 INFO:teuthology.orchestra.run.smithi028.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.67805.log.gz 2024-02-21T23:18:24.331 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.70916.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.70916.log.gz 2024-02-21T23:18:24.331 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64208.log 2024-02-21T23:18:24.332 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.83200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54312.log 2024-02-21T23:18:24.333 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.83200.log.gz 2024-02-21T23:18:24.334 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.64208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55452.log 2024-02-21T23:18:24.334 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.64208.log.gz 2024-02-21T23:18:24.335 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.54312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81771.log 2024-02-21T23:18:24.335 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.54312.log.gz 2024-02-21T23:18:24.336 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.55452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60645.log 2024-02-21T23:18:24.337 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55452.log.gz 2024-02-21T23:18:24.338 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.81771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57278.log 2024-02-21T23:18:24.338 INFO:teuthology.orchestra.run.smithi028.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.81771.log.gz 2024-02-21T23:18:24.339 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.60645.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84622.log 2024-02-21T23:18:24.341 INFO:teuthology.orchestra.run.smithi028.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.60645.log.gz 2024-02-21T23:18:24.341 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.57278.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49124.log 2024-02-21T23:18:24.341 INFO:teuthology.orchestra.run.smithi028.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.57278.log.gz 2024-02-21T23:18:24.342 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.84622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70622.log 2024-02-21T23:18:24.343 INFO:teuthology.orchestra.run.smithi028.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.84622.log.gz 2024-02-21T23:18:24.343 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.49124.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49124.log.gz 2024-02-21T23:18:24.344 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78219.log 2024-02-21T23:18:24.345 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.70622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62206.log 2024-02-21T23:18:24.345 INFO:teuthology.orchestra.run.smithi028.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.70622.log.gz 2024-02-21T23:18:24.346 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.78219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82552.log 2024-02-21T23:18:24.346 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.78219.log.gz 2024-02-21T23:18:24.347 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.62206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54228.log 2024-02-21T23:18:24.348 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.82552.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.62206.log.gz 2024-02-21T23:18:24.349 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79514.log 2024-02-21T23:18:24.349 INFO:teuthology.orchestra.run.smithi028.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.82552.log.gz 2024-02-21T23:18:24.351 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.54228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83536.log 2024-02-21T23:18:24.351 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.54228.log.gz 2024-02-21T23:18:24.351 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.79514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70209.log 2024-02-21T23:18:24.351 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.79514.log.gz 2024-02-21T23:18:24.352 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.83536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53477.log 2024-02-21T23:18:24.352 INFO:teuthology.orchestra.run.smithi028.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.83536.log.gz 2024-02-21T23:18:24.354 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.70209.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.70209.log.gz 2024-02-21T23:18:24.354 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77673.log 2024-02-21T23:18:24.354 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.53477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75032.log 2024-02-21T23:18:24.355 INFO:teuthology.orchestra.run.smithi028.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.53477.log.gz 2024-02-21T23:18:24.355 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.77673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68287.log 2024-02-21T23:18:24.356 INFO:teuthology.orchestra.run.smithi028.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.77673.log.gz 2024-02-21T23:18:24.357 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.75032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46584.log 2024-02-21T23:18:24.357 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.75032.log.gz 2024-02-21T23:18:24.358 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.68287.log: 88.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.68287.log.gz 2024-02-21T23:18:24.358 INFO:teuthology.orchestra.run.smithi028.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.86086.log 2024-02-21T23:18:24.359 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.46584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42631.log 2024-02-21T23:18:24.360 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46584.log.gz 2024-02-21T23:18:24.360 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.86086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65524.log 2024-02-21T23:18:24.361 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.86086.log.gz 2024-02-21T23:18:24.362 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.42631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44927.log 2024-02-21T23:18:24.365 INFO:teuthology.orchestra.run.smithi028.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.42631.log.gz 2024-02-21T23:18:24.365 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.65524.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.65524.log.gz 2024-02-21T23:18:24.366 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68581.log 2024-02-21T23:18:24.366 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.44927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48775.log 2024-02-21T23:18:24.366 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44927.log.gz 2024-02-21T23:18:24.368 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.68581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58012.log 2024-02-21T23:18:24.368 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.68581.log.gz 2024-02-21T23:18:24.368 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.48775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84310.log 2024-02-21T23:18:24.368 INFO:teuthology.orchestra.run.smithi028.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.48775.log.gz 2024-02-21T23:18:24.369 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.58012.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.58012.log.gz 2024-02-21T23:18:24.369 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86170.log 2024-02-21T23:18:24.370 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.84310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86422.log 2024-02-21T23:18:24.370 INFO:teuthology.orchestra.run.smithi028.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.84310.log.gz 2024-02-21T23:18:24.371 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.86170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44189.log 2024-02-21T23:18:24.372 INFO:teuthology.orchestra.run.smithi028.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.86170.log.gz 2024-02-21T23:18:24.373 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.86422.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.86422.log.gz 2024-02-21T23:18:24.373 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77589.log 2024-02-21T23:18:24.374 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.44189.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64502.log 2024-02-21T23:18:24.374 INFO:teuthology.orchestra.run.smithi028.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.44189.log.gz 2024-02-21T23:18:24.375 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.77589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58180.log 2024-02-21T23:18:24.375 INFO:teuthology.orchestra.run.smithi028.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.77589.log.gz 2024-02-21T23:18:24.376 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.64502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44801.log 2024-02-21T23:18:24.377 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.64502.log.gz 2024-02-21T23:18:24.378 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.58180.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.58180.log.gz 2024-02-21T23:18:24.378 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44183.log 2024-02-21T23:18:24.379 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.44801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45936.log 2024-02-21T23:18:24.379 INFO:teuthology.orchestra.run.smithi028.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44801.log.gz 2024-02-21T23:18:24.380 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.44183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42409.log 2024-02-21T23:18:24.381 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44183.log.gz 2024-02-21T23:18:24.382 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.45936.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51147.log 2024-02-21T23:18:24.382 INFO:teuthology.orchestra.run.smithi028.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.45936.log.gz 2024-02-21T23:18:24.383 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.42409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43121.log 2024-02-21T23:18:24.383 INFO:teuthology.orchestra.run.smithi028.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.42409.log.gz 2024-02-21T23:18:24.384 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.51147.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51147.log.gz 2024-02-21T23:18:24.384 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51794.log 2024-02-21T23:18:24.386 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.43121.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55620.log 2024-02-21T23:18:24.386 INFO:teuthology.orchestra.run.smithi028.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43121.log.gz 2024-02-21T23:18:24.387 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.51794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46062.log 2024-02-21T23:18:24.387 INFO:teuthology.orchestra.run.smithi028.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.51794.log.gz 2024-02-21T23:18:24.388 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.55620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76490.log 2024-02-21T23:18:24.389 INFO:teuthology.orchestra.run.smithi028.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55620.log.gz 2024-02-21T23:18:24.390 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.46062.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46062.log.gz 2024-02-21T23:18:24.390 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68539.log 2024-02-21T23:18:24.391 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.76490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62893.log 2024-02-21T23:18:24.391 INFO:teuthology.orchestra.run.smithi028.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.76490.log.gz 2024-02-21T23:18:24.392 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.68539.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88526.log 2024-02-21T23:18:24.392 INFO:teuthology.orchestra.run.smithi028.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.68539.log.gz 2024-02-21T23:18:24.393 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.62893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42815.log 2024-02-21T23:18:24.394 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.62893.log.gz 2024-02-21T23:18:24.395 INFO:teuthology.orchestra.run.smithi081.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-02-21T23:18:24.395 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.88526.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.88526.log.gz 2024-02-21T23:18:24.396 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49987.log 2024-02-21T23:18:24.396 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.42815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43163.log 2024-02-21T23:18:24.397 INFO:teuthology.orchestra.run.smithi028.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.42815.log.gz 2024-02-21T23:18:24.398 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.49987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60856.log 2024-02-21T23:18:24.398 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.49987.log.gz 2024-02-21T23:18:24.399 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.43163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74182.log 2024-02-21T23:18:24.399 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43163.log.gz 2024-02-21T23:18:24.400 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.60856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74738.log 2024-02-21T23:18:24.400 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.60856.log.gz 2024-02-21T23:18:24.402 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.74182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78926.log 2024-02-21T23:18:24.402 INFO:teuthology.orchestra.run.smithi028.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.74182.log.gz 2024-02-21T23:18:24.403 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.74738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73548.log 2024-02-21T23:18:24.404 INFO:teuthology.orchestra.run.smithi028.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.74738.log.gz 2024-02-21T23:18:24.404 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.78926.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.78926.log.gz 2024-02-21T23:18:24.404 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70832.log 2024-02-21T23:18:24.405 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.73548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70958.log 2024-02-21T23:18:24.406 INFO:teuthology.orchestra.run.smithi028.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.73548.log.gz 2024-02-21T23:18:24.406 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.70832.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63440.log 2024-02-21T23:18:24.407 INFO:teuthology.orchestra.run.smithi028.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.70832.log.gz 2024-02-21T23:18:24.408 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.70958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44423.log 2024-02-21T23:18:24.411 INFO:teuthology.orchestra.run.smithi028.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.70958.log.gz 2024-02-21T23:18:24.411 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.63440.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.63440.log.gz 2024-02-21T23:18:24.411 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41360.log 2024-02-21T23:18:24.412 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.44423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47403.log 2024-02-21T23:18:24.412 INFO:teuthology.orchestra.run.smithi028.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44423.log.gz 2024-02-21T23:18:24.415 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.41360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54396.log 2024-02-21T23:18:24.415 INFO:teuthology.orchestra.run.smithi028.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41360.log.gz 2024-02-21T23:18:24.415 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.47403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84184.log 2024-02-21T23:18:24.416 INFO:teuthology.orchestra.run.smithi028.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47403.log.gz 2024-02-21T23:18:24.416 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.54396.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.54396.log.gz 2024-02-21T23:18:24.416 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82240.log 2024-02-21T23:18:24.418 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.84184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73455.log 2024-02-21T23:18:24.418 INFO:teuthology.orchestra.run.smithi028.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.84184.log.gz 2024-02-21T23:18:24.418 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.82240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81032.log 2024-02-21T23:18:24.418 INFO:teuthology.orchestra.run.smithi028.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.82240.log.gz 2024-02-21T23:18:24.419 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.73455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64292.log 2024-02-21T23:18:24.419 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.73455.log.gz 2024-02-21T23:18:24.420 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.81032.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.81032.log.gz 2024-02-21T23:18:24.420 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79370.log 2024-02-21T23:18:24.421 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.64292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59274.log 2024-02-21T23:18:24.421 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.64292.log.gz 2024-02-21T23:18:24.422 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.79370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84916.log 2024-02-21T23:18:24.422 INFO:teuthology.orchestra.run.smithi028.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.79370.log.gz 2024-02-21T23:18:24.423 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.59274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43923.log 2024-02-21T23:18:24.424 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.59274.log.gz 2024-02-21T23:18:24.425 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.84916.log: 89.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.84916.log.gz -5 2024-02-21T23:18:24.425 INFO:teuthology.orchestra.run.smithi028.stderr: --verbose -- /var/log/ceph/ceph-client.admin.61744.log 2024-02-21T23:18:24.426 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.43923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80241.log 2024-02-21T23:18:24.426 INFO:teuthology.orchestra.run.smithi028.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43923.log.gz 2024-02-21T23:18:24.427 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.61744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55746.log 2024-02-21T23:18:24.427 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.61744.log.gz 2024-02-21T23:18:24.429 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.80241.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54438.log 2024-02-21T23:18:24.429 INFO:teuthology.orchestra.run.smithi028.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.80241.log.gz 2024-02-21T23:18:24.430 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.55746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75634.log 2024-02-21T23:18:24.431 INFO:teuthology.orchestra.run.smithi028.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.55746.log.gz 2024-02-21T23:18:24.431 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.54438.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83932.log 2024-02-21T23:18:24.431 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54438.log.gz 2024-02-21T23:18:24.433 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.75634.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69326.log 2024-02-21T23:18:24.433 INFO:teuthology.orchestra.run.smithi028.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.75634.log.gz 2024-02-21T23:18:24.434 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.83932.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.83932.log.gz 2024-02-21T23:18:24.434 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79779.log 2024-02-21T23:18:24.435 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.69326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82846.log 2024-02-21T23:18:24.436 INFO:teuthology.orchestra.run.smithi028.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.69326.log.gz 2024-02-21T23:18:24.436 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.79779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58516.log 2024-02-21T23:18:24.437 INFO:teuthology.orchestra.run.smithi028.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.79779.log.gz 2024-02-21T23:18:24.438 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.82846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52386.log 2024-02-21T23:18:24.438 INFO:teuthology.orchestra.run.smithi028.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.82846.log.gz 2024-02-21T23:18:24.439 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.58516.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.58516.log.gz 2024-02-21T23:18:24.440 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75471.log 2024-02-21T23:18:24.441 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.52386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54270.log 2024-02-21T23:18:24.441 INFO:teuthology.orchestra.run.smithi028.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.52386.log.gz 2024-02-21T23:18:24.442 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.75471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84478.log 2024-02-21T23:18:24.442 INFO:teuthology.orchestra.run.smithi028.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.75471.log.gz 2024-02-21T23:18:24.443 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.54270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84352.log 2024-02-21T23:18:24.444 INFO:teuthology.orchestra.run.smithi028.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.54270.log.gz 2024-02-21T23:18:24.445 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.84478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43584.log 2024-02-21T23:18:24.445 INFO:teuthology.orchestra.run.smithi028.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.84478.log.gz 2024-02-21T23:18:24.446 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.84352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57534.log 2024-02-21T23:18:24.446 INFO:teuthology.orchestra.run.smithi028.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.84352.log.gz 2024-02-21T23:18:24.447 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.43584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63314.log 2024-02-21T23:18:24.448 INFO:teuthology.orchestra.run.smithi028.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43584.log.gz 2024-02-21T23:18:24.448 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.57534.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75795.log 2024-02-21T23:18:24.449 INFO:teuthology.orchestra.run.smithi028.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.57534.log.gz 2024-02-21T23:18:24.450 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.63314.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67336.log 2024-02-21T23:18:24.450 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.75795.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.63314.log.gz 2024-02-21T23:18:24.451 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.75795.log.gz 2024-02-21T23:18:24.451 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67847.log 2024-02-21T23:18:24.452 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.67336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87317.log 2024-02-21T23:18:24.453 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.67336.log.gz 2024-02-21T23:18:24.453 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.67847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88778.log 2024-02-21T23:18:24.454 INFO:teuthology.orchestra.run.smithi028.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.67847.log.gz 2024-02-21T23:18:24.455 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.87317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64820.log 2024-02-21T23:18:24.456 INFO:teuthology.orchestra.run.smithi028.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.87317.log.gz 2024-02-21T23:18:24.456 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.88778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43205.log 2024-02-21T23:18:24.456 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.88778.log.gz 2024-02-21T23:18:24.458 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.64820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80902.log 2024-02-21T23:18:24.458 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.64820.log.gz 2024-02-21T23:18:24.459 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.43205.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75513.log 2024-02-21T23:18:24.460 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43205.log.gz 2024-02-21T23:18:24.460 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.80902.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69999.log 2024-02-21T23:18:24.460 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.80902.log.gz 2024-02-21T23:18:24.461 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.75513.log: 88.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.75513.log.gz -5 2024-02-21T23:18:24.461 INFO:teuthology.orchestra.run.smithi028.stderr: --verbose -- /var/log/ceph/ceph-client.admin.57910.log 2024-02-21T23:18:24.463 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.69999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78712.log 2024-02-21T23:18:24.463 INFO:teuthology.orchestra.run.smithi028.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.69999.log.gz 2024-02-21T23:18:24.463 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.57910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43589.log 2024-02-21T23:18:24.464 INFO:teuthology.orchestra.run.smithi028.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.57910.log.gz 2024-02-21T23:18:24.465 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.78712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54888.log 2024-02-21T23:18:24.466 INFO:teuthology.orchestra.run.smithi028.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.78712.log.gz 2024-02-21T23:18:24.466 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.43589.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43589.log.gz 2024-02-21T23:18:24.466 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48645.log 2024-02-21T23:18:24.467 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.54888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60020.log 2024-02-21T23:18:24.468 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54888.log.gz 2024-02-21T23:18:24.468 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.48645.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45810.log 2024-02-21T23:18:24.469 INFO:teuthology.orchestra.run.smithi028.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48645.log.gz 2024-02-21T23:18:24.470 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.60020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54606.log 2024-02-21T23:18:24.470 INFO:teuthology.orchestra.run.smithi028.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.60020.log.gz 2024-02-21T23:18:24.471 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.45810.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69763.log 2024-02-21T23:18:24.471 INFO:teuthology.orchestra.run.smithi028.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.45810.log.gz 2024-02-21T23:18:24.472 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.54606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54846.log 2024-02-21T23:18:24.472 INFO:teuthology.orchestra.run.smithi028.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.54606.log.gz 2024-02-21T23:18:24.473 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.69763.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62080.log 2024-02-21T23:18:24.474 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.69763.log.gz 2024-02-21T23:18:24.475 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.54846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53519.log 2024-02-21T23:18:24.475 INFO:teuthology.orchestra.run.smithi028.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.54846.log.gz 2024-02-21T23:18:24.476 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.62080.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.62080.log.gz 2024-02-21T23:18:24.476 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81945.log 2024-02-21T23:18:24.477 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.53519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76598.log 2024-02-21T23:18:24.478 INFO:teuthology.orchestra.run.smithi028.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.53519.log.gz 2024-02-21T23:18:24.478 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.81945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41188.log 2024-02-21T23:18:24.479 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.81945.log.gz 2024-02-21T23:18:24.480 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.76598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71000.log 2024-02-21T23:18:24.480 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.76598.log.gz 2024-02-21T23:18:24.481 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.41188.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41188.log.gz 2024-02-21T23:18:24.482 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42773.log 2024-02-21T23:18:24.483 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.71000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84958.log 2024-02-21T23:18:24.483 INFO:teuthology.orchestra.run.smithi028.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.71000.log.gz 2024-02-21T23:18:24.484 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.42773.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54186.log 2024-02-21T23:18:24.484 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42773.log.gz 2024-02-21T23:18:24.485 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.84958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47708.log 2024-02-21T23:18:24.485 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.84958.log.gz 2024-02-21T23:18:24.486 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.54186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85228.log 2024-02-21T23:18:24.487 INFO:teuthology.orchestra.run.smithi028.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.54186.log.gz 2024-02-21T23:18:24.488 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.47708.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47708.log.gz 2024-02-21T23:18:24.488 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71993.log 2024-02-21T23:18:24.489 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.85228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46668.log 2024-02-21T23:18:24.489 INFO:teuthology.orchestra.run.smithi028.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.85228.log.gz 2024-02-21T23:18:24.490 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.71993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81213.log 2024-02-21T23:18:24.490 INFO:teuthology.orchestra.run.smithi028.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.71993.log.gz 2024-02-21T23:18:24.492 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.46668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72916.log 2024-02-21T23:18:24.492 INFO:teuthology.orchestra.run.smithi028.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.46668.log.gz 2024-02-21T23:18:24.493 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.81213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52596.log 2024-02-21T23:18:24.493 INFO:teuthology.orchestra.run.smithi028.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.81213.log.gz 2024-02-21T23:18:24.494 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.72916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55368.log 2024-02-21T23:18:24.494 INFO:teuthology.orchestra.run.smithi028.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.72916.log.gz 2024-02-21T23:18:24.495 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.52596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56519.log 2024-02-21T23:18:24.496 INFO:teuthology.orchestra.run.smithi028.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52596.log.gz 2024-02-21T23:18:24.497 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.55368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71940.log 2024-02-21T23:18:24.497 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55368.log.gz 2024-02-21T23:18:24.498 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.56519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67763.log 2024-02-21T23:18:24.498 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.56519.log.gz 2024-02-21T23:18:24.499 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.71940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58306.log 2024-02-21T23:18:24.500 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.71940.log.gz 2024-02-21T23:18:24.500 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.67763.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41764.log 2024-02-21T23:18:24.501 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.67763.log.gz 2024-02-21T23:18:24.502 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.58306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66828.log 2024-02-21T23:18:24.502 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.58306.log.gz 2024-02-21T23:18:24.503 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.41764.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41764.log.gz 2024-02-21T23:18:24.503 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86212.log 2024-02-21T23:18:24.504 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.66828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48087.log 2024-02-21T23:18:24.505 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.66828.log.gz 2024-02-21T23:18:24.505 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.86212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72435.log 2024-02-21T23:18:24.506 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.86212.log.gz 2024-02-21T23:18:24.507 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.48087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50257.log 2024-02-21T23:18:24.507 INFO:teuthology.orchestra.run.smithi028.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.48087.log.gz 2024-02-21T23:18:24.508 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.72435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46374.log 2024-02-21T23:18:24.508 INFO:teuthology.orchestra.run.smithi028.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.72435.log.gz 2024-02-21T23:18:24.509 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.50257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49166.log 2024-02-21T23:18:24.510 INFO:teuthology.orchestra.run.smithi028.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.50257.log.gz 2024-02-21T23:18:24.511 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.46374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77210.log 2024-02-21T23:18:24.511 INFO:teuthology.orchestra.run.smithi028.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46374.log.gz 2024-02-21T23:18:24.512 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.49166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55812.log 2024-02-21T23:18:24.512 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49166.log.gz 2024-02-21T23:18:24.513 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.77210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78462.log 2024-02-21T23:18:24.514 INFO:teuthology.orchestra.run.smithi028.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.77210.log.gz 2024-02-21T23:18:24.515 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.55812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57784.log 2024-02-21T23:18:24.515 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.55812.log.gz 2024-02-21T23:18:24.516 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.78462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55494.log 2024-02-21T23:18:24.516 INFO:teuthology.orchestra.run.smithi028.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.78462.log.gz 2024-02-21T23:18:24.517 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.57784.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81083.log 2024-02-21T23:18:24.518 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57784.log.gz 2024-02-21T23:18:24.519 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.55494.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55494.log.gz 2024-02-21T23:18:24.519 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52512.log 2024-02-21T23:18:24.520 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.81083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87459.log 2024-02-21T23:18:24.520 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.81083.log.gz 2024-02-21T23:18:24.521 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.52512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74659.log 2024-02-21T23:18:24.522 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52512.log.gz 2024-02-21T23:18:24.523 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.87459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85000.log 2024-02-21T23:18:24.523 INFO:teuthology.orchestra.run.smithi028.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.87459.log.gz 2024-02-21T23:18:24.524 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.74659.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46248.log 2024-02-21T23:18:24.524 INFO:teuthology.orchestra.run.smithi028.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.74659.log.gz 2024-02-21T23:18:24.525 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.85000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49842.log 2024-02-21T23:18:24.526 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.85000.log.gz 2024-02-21T23:18:24.526 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.46248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42497.log 2024-02-21T23:18:24.527 INFO:teuthology.orchestra.run.smithi028.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.46248.log.gz 2024-02-21T23:18:24.528 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.49842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44549.log 2024-02-21T23:18:24.529 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.42497.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.49842.log.gz 2024-02-21T23:18:24.529 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56687.log 2024-02-21T23:18:24.529 INFO:teuthology.orchestra.run.smithi028.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.42497.log.gz 2024-02-21T23:18:24.530 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.44549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47919.log 2024-02-21T23:18:24.531 INFO:teuthology.orchestra.run.smithi028.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44549.log.gz 2024-02-21T23:18:24.531 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.56687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88442.log 2024-02-21T23:18:24.532 INFO:teuthology.orchestra.run.smithi028.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.56687.log.gz 2024-02-21T23:18:24.533 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.47919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79989.log 2024-02-21T23:18:24.533 INFO:teuthology.orchestra.run.smithi028.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47919.log.gz 2024-02-21T23:18:24.534 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.88442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77925.log 2024-02-21T23:18:24.534 INFO:teuthology.orchestra.run.smithi028.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.88442.log.gz 2024-02-21T23:18:24.535 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.79989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80694.log 2024-02-21T23:18:24.536 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.79989.log.gz 2024-02-21T23:18:24.536 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.77925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86254.log 2024-02-21T23:18:24.537 INFO:teuthology.orchestra.run.smithi028.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.77925.log.gz 2024-02-21T23:18:24.538 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.80694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84226.log 2024-02-21T23:18:24.538 INFO:teuthology.orchestra.run.smithi028.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.80694.log.gz 2024-02-21T23:18:24.539 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.86254.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.86254.log.gz 2024-02-21T23:18:24.539 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80860.log 2024-02-21T23:18:24.540 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.84226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74864.log 2024-02-21T23:18:24.541 INFO:teuthology.orchestra.run.smithi028.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.84226.log.gz 2024-02-21T23:18:24.541 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.80860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73506.log 2024-02-21T23:18:24.542 INFO:teuthology.orchestra.run.smithi028.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.80860.log.gz 2024-02-21T23:18:24.543 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.74864.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65566.log 2024-02-21T23:18:24.543 INFO:teuthology.orchestra.run.smithi028.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.74864.log.gz 2024-02-21T23:18:24.544 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.73506.log: 87.9%gzip -- replaced with /var/log/ceph/ceph-client.admin.73506.log.gz -5 2024-02-21T23:18:24.544 INFO:teuthology.orchestra.run.smithi028.stderr: --verbose -- /var/log/ceph/ceph-client.admin.79821.log 2024-02-21T23:18:24.545 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.65566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66605.log 2024-02-21T23:18:24.546 INFO:teuthology.orchestra.run.smithi028.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.65566.log.gz 2024-02-21T23:18:24.546 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.79821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56252.log 2024-02-21T23:18:24.547 INFO:teuthology.orchestra.run.smithi028.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.79821.log.gz 2024-02-21T23:18:24.548 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.66605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45011.log 2024-02-21T23:18:24.548 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.66605.log.gz 2024-02-21T23:18:24.549 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.56252.log: gzip -5 --verbose 88.3% -- /var/log/ceph/ceph-client.admin.73198.log -- replaced with /var/log/ceph/ceph-client.admin.56252.log.gz 2024-02-21T23:18:24.550 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-02-21T23:18:24.551 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.45011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85858.log 2024-02-21T23:18:24.551 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.73198.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.45011.log.gz 2024-02-21T23:18:24.552 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60543.log 2024-02-21T23:18:24.552 INFO:teuthology.orchestra.run.smithi028.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.73198.log.gz 2024-02-21T23:18:24.553 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.85858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70297.log 2024-02-21T23:18:24.553 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.85858.log.gz 2024-02-21T23:18:24.555 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.60543.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.60543.log.gz 2024-02-21T23:18:24.555 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64334.log 2024-02-21T23:18:24.556 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.70297.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50696.log 2024-02-21T23:18:24.556 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.70297.log.gz 2024-02-21T23:18:24.557 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.64334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69637.log 2024-02-21T23:18:24.557 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.64334.log.gz 2024-02-21T23:18:24.558 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.50696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70496.log 2024-02-21T23:18:24.559 INFO:teuthology.orchestra.run.smithi028.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50696.log.gz 2024-02-21T23:18:24.560 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.69637.log: gzip -5 --verbose -- 88.7% /var/log/ceph/ceph-client.admin.59860.log 2024-02-21T23:18:24.560 INFO:teuthology.orchestra.run.smithi028.stderr: -- replaced with /var/log/ceph/ceph-client.admin.69637.log.gz 2024-02-21T23:18:24.561 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.70496.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-02-21T23:18:24.561 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.70496.log.gz 2024-02-21T23:18:24.562 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.59860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70454.log 2024-02-21T23:18:24.562 INFO:teuthology.orchestra.run.smithi028.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.59860.log.gz 2024-02-21T23:18:24.563 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57826.log 2024-02-21T23:18:24.565 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.70454.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.70454.log.gz 2024-02-21T23:18:24.565 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46710.log 2024-02-21T23:18:24.566 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.57826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47235.log 2024-02-21T23:18:24.566 INFO:teuthology.orchestra.run.smithi028.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.57826.log.gz 2024-02-21T23:18:24.567 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.46710.log: gzip 87.8% -- replaced with /var/log/ceph/ceph-client.admin.46710.log.gz 2024-02-21T23:18:24.568 INFO:teuthology.orchestra.run.smithi028.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.78177.log 2024-02-21T23:18:24.569 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.47235.log: gzip -5 88.3% --verbose -- replaced with /var/log/ceph/ceph-client.admin.47235.log.gz 2024-02-21T23:18:24.569 INFO:teuthology.orchestra.run.smithi028.stderr: -- /var/log/ceph/ceph-client.admin.42546.log 2024-02-21T23:18:24.570 INFO:teuthology.orchestra.run.smithi028.stderr: 89.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-02-21T23:18:24.570 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.78177.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.78177.log.gz 2024-02-21T23:18:24.570 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67294.log 2024-02-21T23:18:24.571 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.42546.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73037.log 2024-02-21T23:18:24.572 INFO:teuthology.orchestra.run.smithi028.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.42546.log.gz 2024-02-21T23:18:24.572 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.67294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50654.log 2024-02-21T23:18:24.573 INFO:teuthology.orchestra.run.smithi028.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.67294.log.gz 2024-02-21T23:18:24.574 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.73037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80031.log 2024-02-21T23:18:24.574 INFO:teuthology.orchestra.run.smithi028.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.73037.log.gz 2024-02-21T23:18:24.575 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.50654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44465.log 2024-02-21T23:18:24.575 INFO:teuthology.orchestra.run.smithi028.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50654.log.gz 2024-02-21T23:18:24.576 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.80031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80493.log 2024-02-21T23:18:24.576 INFO:teuthology.orchestra.run.smithi028.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.80031.log.gz 2024-02-21T23:18:24.578 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.44465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67060.log 2024-02-21T23:18:24.578 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44465.log.gz 2024-02-21T23:18:24.579 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.80493.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.80493.log.gz 2024-02-21T23:18:24.579 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64862.log 2024-02-21T23:18:24.580 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.67060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55284.log 2024-02-21T23:18:24.580 INFO:teuthology.orchestra.run.smithi028.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.67060.log.gz 2024-02-21T23:18:24.581 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.64862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79010.log 2024-02-21T23:18:24.582 INFO:teuthology.orchestra.run.smithi028.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.64862.log.gz 2024-02-21T23:18:24.583 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.55284.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.55284.log.gz 2024-02-21T23:18:24.583 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43933.log 2024-02-21T23:18:24.584 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.79010.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88400.log 2024-02-21T23:18:24.584 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.79010.log.gz 2024-02-21T23:18:24.585 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.43933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81339.log 2024-02-21T23:18:24.585 INFO:teuthology.orchestra.run.smithi028.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43933.log.gz 2024-02-21T23:18:24.586 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.88400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63146.log 2024-02-21T23:18:24.586 INFO:teuthology.orchestra.run.smithi028.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.88400.log.gz 2024-02-21T23:18:24.588 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.81339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53309.log 2024-02-21T23:18:24.589 INFO:teuthology.orchestra.run.smithi028.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.81339.log.gz 2024-02-21T23:18:24.589 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.63146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77547.log 2024-02-21T23:18:24.589 INFO:teuthology.orchestra.run.smithi028.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.63146.log.gz 2024-02-21T23:18:24.590 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.53309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54564.log 2024-02-21T23:18:24.590 INFO:teuthology.orchestra.run.smithi028.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53309.log.gz 2024-02-21T23:18:24.591 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.77547.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70748.log 2024-02-21T23:18:24.592 INFO:teuthology.orchestra.run.smithi028.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.77547.log.gz 2024-02-21T23:18:24.592 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.54564.log: gzip 86.9% -- replaced with /var/log/ceph/ceph-client.admin.54564.log.gz -5 2024-02-21T23:18:24.593 INFO:teuthology.orchestra.run.smithi028.stderr: --verbose -- /var/log/ceph/ceph-client.admin.69456.log 2024-02-21T23:18:24.594 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.70748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77276.log 2024-02-21T23:18:24.595 INFO:teuthology.orchestra.run.smithi028.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.70748.log.gz 2024-02-21T23:18:24.595 INFO:teuthology.orchestra.run.smithi081.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-02-21T23:18:24.596 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.69456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66786.log 2024-02-21T23:18:24.596 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.69456.log.gz 2024-02-21T23:18:24.596 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.77276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65986.log 2024-02-21T23:18:24.597 INFO:teuthology.orchestra.run.smithi028.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.77276.log.gz 2024-02-21T23:18:24.598 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.66786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50780.log 2024-02-21T23:18:24.598 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.66786.log.gz 2024-02-21T23:18:24.599 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.65986.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.65986.log.gz 2024-02-21T23:18:24.599 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76766.log 2024-02-21T23:18:24.600 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.50780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80157.log 2024-02-21T23:18:24.600 INFO:teuthology.orchestra.run.smithi028.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50780.log.gz 2024-02-21T23:18:24.601 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.76766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64670.log 2024-02-21T23:18:24.601 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.76766.log.gz 2024-02-21T23:18:24.602 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.80157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69889.log 2024-02-21T23:18:24.603 INFO:teuthology.orchestra.run.smithi028.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.80157.log.gz 2024-02-21T23:18:24.604 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.64670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44969.log 2024-02-21T23:18:24.604 INFO:teuthology.orchestra.run.smithi028.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.64670.log.gz 2024-02-21T23:18:24.605 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.69889.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79328.log 2024-02-21T23:18:24.605 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.69889.log.gz 2024-02-21T23:18:24.606 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.44969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57366.log 2024-02-21T23:18:24.607 INFO:teuthology.orchestra.run.smithi028.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44969.log.gz 2024-02-21T23:18:24.608 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.79328.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.79328.log.gz 2024-02-21T23:18:24.608 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85774.log 2024-02-21T23:18:24.609 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.57366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72165.log 2024-02-21T23:18:24.609 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.57366.log.gz 2024-02-21T23:18:24.610 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.85774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68707.log 2024-02-21T23:18:24.610 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.85774.log.gz 2024-02-21T23:18:24.611 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.72165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54931.log 2024-02-21T23:18:24.612 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.72165.log.gz 2024-02-21T23:18:24.613 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.68707.log: 88.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.68707.log.gz 2024-02-21T23:18:24.613 INFO:teuthology.orchestra.run.smithi028.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.88736.log 2024-02-21T23:18:24.614 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.54931.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54931.log.gz 2024-02-21T23:18:24.614 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60772.log 2024-02-21T23:18:24.615 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.88736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42857.log 2024-02-21T23:18:24.615 INFO:teuthology.orchestra.run.smithi028.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.88736.log.gz 2024-02-21T23:18:24.616 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.60772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61433.log 2024-02-21T23:18:24.616 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.60772.log.gz 2024-02-21T23:18:24.617 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.42857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73240.log 2024-02-21T23:18:24.618 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.42857.log.gz 2024-02-21T23:18:24.619 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.61433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53225.log 2024-02-21T23:18:24.619 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.61433.log.gz 2024-02-21T23:18:24.620 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.73240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57019.log 2024-02-21T23:18:24.620 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.73240.log.gz 2024-02-21T23:18:24.621 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.53225.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78051.log 2024-02-21T23:18:24.621 INFO:teuthology.orchestra.run.smithi081.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-02-21T23:18:24.622 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.53225.log.gz 2024-02-21T23:18:24.622 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.57019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81903.log 2024-02-21T23:18:24.623 INFO:teuthology.orchestra.run.smithi028.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.57019.log.gz 2024-02-21T23:18:24.624 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.78051.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.78051.log.gz 2024-02-21T23:18:24.624 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68329.log 2024-02-21T23:18:24.625 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.81903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68227.log 2024-02-21T23:18:24.625 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.81903.log.gz 2024-02-21T23:18:24.626 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.68329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51626.log 2024-02-21T23:18:24.626 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.68329.log.gz 2024-02-21T23:18:24.627 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.68227.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83578.log 2024-02-21T23:18:24.627 INFO:teuthology.orchestra.run.smithi028.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.68227.log.gz 2024-02-21T23:18:24.628 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.51626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50981.log 2024-02-21T23:18:24.629 INFO:teuthology.orchestra.run.smithi028.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.51626.log.gz 2024-02-21T23:18:24.630 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.83578.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.83578.log.gz 2024-02-21T23:18:24.630 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49800.log 2024-02-21T23:18:24.631 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.50981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61391.log 2024-02-21T23:18:24.631 INFO:teuthology.orchestra.run.smithi028.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.50981.log.gz 2024-02-21T23:18:24.632 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.49800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77144.log 2024-02-21T23:18:24.633 INFO:teuthology.orchestra.run.smithi028.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49800.log.gz 2024-02-21T23:18:24.634 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.61391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71145.log 2024-02-21T23:18:24.634 INFO:teuthology.orchestra.run.smithi028.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.61391.log.gz 2024-02-21T23:18:24.635 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.77144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85960.log 2024-02-21T23:18:24.635 INFO:teuthology.orchestra.run.smithi028.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.77144.log.gz 2024-02-21T23:18:24.636 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.71145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78884.log 2024-02-21T23:18:24.636 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.71145.log.gz 2024-02-21T23:18:24.641 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.85960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49945.log 2024-02-21T23:18:24.641 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.85960.log.gz 2024-02-21T23:18:24.641 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.78884.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.78884.log.gz 2024-02-21T23:18:24.641 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48687.log 2024-02-21T23:18:24.641 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.49945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65818.log 2024-02-21T23:18:24.641 INFO:teuthology.orchestra.run.smithi028.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.49945.log.gz 2024-02-21T23:18:24.642 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.48687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76280.log 2024-02-21T23:18:24.642 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48687.log.gz 2024-02-21T23:18:24.643 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.65818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43928.log 2024-02-21T23:18:24.643 INFO:teuthology.orchestra.run.smithi028.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.65818.log.gz 2024-02-21T23:18:24.644 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.76280.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.76280.log.gz 2024-02-21T23:18:24.644 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47659.log 2024-02-21T23:18:24.645 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.43928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82324.log 2024-02-21T23:18:24.646 INFO:teuthology.orchestra.run.smithi028.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43928.log.gz 2024-02-21T23:18:24.647 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.47659.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84268.log 2024-02-21T23:18:24.647 INFO:teuthology.orchestra.run.smithi028.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.47659.log.gz 2024-02-21T23:18:24.648 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.82324.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.82324.log.gz 2024-02-21T23:18:24.648 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75837.log 2024-02-21T23:18:24.650 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.84268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79598.log 2024-02-21T23:18:24.650 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.84268.log.gz 2024-02-21T23:18:24.651 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.75837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51105.log 2024-02-21T23:18:24.651 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.75837.log.gz 2024-02-21T23:18:24.652 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.79598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77841.log 2024-02-21T23:18:24.653 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.79598.log.gz 2024-02-21T23:18:24.654 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.51105.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86608.log 2024-02-21T23:18:24.654 INFO:teuthology.orchestra.run.smithi028.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.51105.log.gz 2024-02-21T23:18:24.655 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.77841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56477.log 2024-02-21T23:18:24.655 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.77841.log.gz 2024-02-21T23:18:24.656 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.86608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50029.log 2024-02-21T23:18:24.656 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.86608.log.gz 2024-02-21T23:18:24.657 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.56477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81125.log 2024-02-21T23:18:24.657 INFO:teuthology.orchestra.run.smithi028.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.56477.log.gz 2024-02-21T23:18:24.658 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.50029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66916.log 2024-02-21T23:18:24.659 INFO:teuthology.orchestra.run.smithi028.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.50029.log.gz 2024-02-21T23:18:24.659 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.81125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61477.log 2024-02-21T23:18:24.659 INFO:teuthology.orchestra.run.smithi028.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.81125.log.gz 2024-02-21T23:18:24.661 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.66916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74308.log 2024-02-21T23:18:24.661 INFO:teuthology.orchestra.run.smithi028.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.66916.log.gz 2024-02-21T23:18:24.662 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.61477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58348.log 2024-02-21T23:18:24.662 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.61477.log.gz 2024-02-21T23:18:24.663 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.74308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74224.log 2024-02-21T23:18:24.663 INFO:teuthology.orchestra.run.smithi028.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.74308.log.gz 2024-02-21T23:18:24.664 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.58348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87732.log 2024-02-21T23:18:24.665 INFO:teuthology.orchestra.run.smithi028.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.58348.log.gz 2024-02-21T23:18:24.666 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.74224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75429.log 2024-02-21T23:18:24.666 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.74224.log.gz 2024-02-21T23:18:24.667 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.87732.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.87732.log.gz 2024-02-21T23:18:24.667 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64250.log 2024-02-21T23:18:24.668 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.75429.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55704.log 2024-02-21T23:18:24.668 INFO:teuthology.orchestra.run.smithi028.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.75429.log.gz 2024-02-21T23:18:24.670 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.64250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49082.log 2024-02-21T23:18:24.670 INFO:teuthology.orchestra.run.smithi028.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.64250.log.gz 2024-02-21T23:18:24.671 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.55704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81615.log 2024-02-21T23:18:24.671 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.49082.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.55704.log.gz 2024-02-21T23:18:24.672 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49082.log.gz 2024-02-21T23:18:24.672 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87503.log 2024-02-21T23:18:24.673 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.81615.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47614.log 2024-02-21T23:18:24.673 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.81615.log.gz 2024-02-21T23:18:24.674 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.87503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85732.log 2024-02-21T23:18:24.675 INFO:teuthology.orchestra.run.smithi028.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.87503.log.gz 2024-02-21T23:18:24.676 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.47614.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46458.log 2024-02-21T23:18:24.676 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.47614.log.gz 2024-02-21T23:18:24.677 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.85732.log: 88.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.85732.log.gz 2024-02-21T23:18:24.677 INFO:teuthology.orchestra.run.smithi028.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.72077.log 2024-02-21T23:18:24.678 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.46458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82408.log 2024-02-21T23:18:24.678 INFO:teuthology.orchestra.run.smithi028.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46458.log.gz 2024-02-21T23:18:24.679 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.72077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63356.log 2024-02-21T23:18:24.680 INFO:teuthology.orchestra.run.smithi028.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.72077.log.gz 2024-02-21T23:18:24.681 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.82408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52554.log 2024-02-21T23:18:24.681 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.82408.log.gz 2024-02-21T23:18:24.682 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.63356.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.63356.log.gz 2024-02-21T23:18:24.683 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75074.log 2024-02-21T23:18:24.683 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.52554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58726.log 2024-02-21T23:18:24.684 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52554.log.gz 2024-02-21T23:18:24.685 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.75074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50822.log 2024-02-21T23:18:24.685 INFO:teuthology.orchestra.run.smithi028.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.75074.log.gz 2024-02-21T23:18:24.686 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.58726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42452.log 2024-02-21T23:18:24.687 INFO:teuthology.orchestra.run.smithi028.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.58726.log.gz 2024-02-21T23:18:24.687 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.50822.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50822.log.gz 2024-02-21T23:18:24.687 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83620.log 2024-02-21T23:18:24.688 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.42452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56431.log 2024-02-21T23:18:24.689 INFO:teuthology.orchestra.run.smithi028.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.42452.log.gz 2024-02-21T23:18:24.691 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.83620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76112.log 2024-02-21T23:18:24.691 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.83620.log.gz 2024-02-21T23:18:24.691 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.56431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50071.log 2024-02-21T23:18:24.692 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.76112.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.56431.log.gz 2024-02-21T23:18:24.692 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.76112.log.gz 2024-02-21T23:18:24.693 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44297.log 2024-02-21T23:18:24.694 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.50071.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53874.log 2024-02-21T23:18:24.694 INFO:teuthology.orchestra.run.smithi028.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.50071.log.gz 2024-02-21T23:18:24.695 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.44297.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56771.log 2024-02-21T23:18:24.695 INFO:teuthology.orchestra.run.smithi028.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44297.log.gz 2024-02-21T23:18:24.697 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.53874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58966.log 2024-02-21T23:18:24.697 INFO:teuthology.orchestra.run.smithi028.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53874.log.gz 2024-02-21T23:18:24.698 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.56771.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.56771.log.gz 2024-02-21T23:18:24.698 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48333.log 2024-02-21T23:18:24.699 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.58966.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57741.log 2024-02-21T23:18:24.699 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.58966.log.gz 2024-02-21T23:18:24.700 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.48333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40983.log 2024-02-21T23:18:24.701 INFO:teuthology.orchestra.run.smithi028.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.48333.log.gz 2024-02-21T23:18:24.702 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.57741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51370.log 2024-02-21T23:18:24.702 INFO:teuthology.orchestra.run.smithi028.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.57741.log.gz 2024-02-21T23:18:24.703 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.40983.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.40983.log.gz 2024-02-21T23:18:24.703 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79640.log 2024-02-21T23:18:24.704 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.51370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68917.log 2024-02-21T23:18:24.705 INFO:teuthology.orchestra.run.smithi028.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51370.log.gz 2024-02-21T23:18:24.706 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.79640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56208.log 2024-02-21T23:18:24.706 INFO:teuthology.orchestra.run.smithi028.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.79640.log.gz 2024-02-21T23:18:24.707 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.68917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75345.log 2024-02-21T23:18:24.707 INFO:teuthology.orchestra.run.smithi028.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.68917.log.gz 2024-02-21T23:18:24.708 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.56208.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.56208.log.gz 2024-02-21T23:18:24.708 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54084.log 2024-02-21T23:18:24.709 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.75345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76850.log 2024-02-21T23:18:24.710 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.75345.log.gz 2024-02-21T23:18:24.711 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.54084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52112.log 2024-02-21T23:18:24.711 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.54084.log.gz 2024-02-21T23:18:24.712 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.76850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46752.log 2024-02-21T23:18:24.713 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.76850.log.gz 2024-02-21T23:18:24.713 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.52112.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52112.log.gz 2024-02-21T23:18:24.714 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55938.log 2024-02-21T23:18:24.715 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.46752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55536.log 2024-02-21T23:18:24.715 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46752.log.gz 2024-02-21T23:18:24.716 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.55938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61024.log 2024-02-21T23:18:24.716 INFO:teuthology.orchestra.run.smithi028.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55938.log.gz 2024-02-21T23:18:24.717 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.55536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50215.log 2024-02-21T23:18:24.717 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55536.log.gz 2024-02-21T23:18:24.718 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.61024.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.61024.log.gz 2024-02-21T23:18:24.719 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71478.log 2024-02-21T23:18:24.719 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.50215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42731.log 2024-02-21T23:18:24.720 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50215.log.gz 2024-02-21T23:18:24.721 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.71478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54000.log 2024-02-21T23:18:24.721 INFO:teuthology.orchestra.run.smithi028.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.71478.log.gz 2024-02-21T23:18:24.722 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.42731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76070.log 2024-02-21T23:18:24.722 INFO:teuthology.orchestra.run.smithi028.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42731.log.gz 2024-02-21T23:18:24.723 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.54000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45642.log 2024-02-21T23:18:24.723 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54000.log.gz 2024-02-21T23:18:24.724 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.76070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82366.log 2024-02-21T23:18:24.725 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.76070.log.gz 2024-02-21T23:18:24.725 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.45642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83242.log 2024-02-21T23:18:24.726 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45642.log.gz 2024-02-21T23:18:24.727 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.82366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65089.log 2024-02-21T23:18:24.727 INFO:teuthology.orchestra.run.smithi028.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.82366.log.gz 2024-02-21T23:18:24.728 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.83242.log: gzip 89.3% -- replaced with /var/log/ceph/ceph-client.admin.83242.log.gz -5 2024-02-21T23:18:24.728 INFO:teuthology.orchestra.run.smithi028.stderr: --verbose -- /var/log/ceph/ceph-client.admin.57408.log 2024-02-21T23:18:24.729 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.65089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78968.log 2024-02-21T23:18:24.730 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.65089.log.gz 2024-02-21T23:18:24.731 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.57408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55662.log 2024-02-21T23:18:24.731 INFO:teuthology.orchestra.run.smithi028.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57408.log.gz 2024-02-21T23:18:24.732 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.78968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83662.log 2024-02-21T23:18:24.732 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.78968.log.gz 2024-02-21T23:18:24.733 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.55662.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.55662.log.gz 2024-02-21T23:18:24.733 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67378.log 2024-02-21T23:18:24.734 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.83662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43913.log 2024-02-21T23:18:24.734 INFO:teuthology.orchestra.run.smithi028.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.83662.log.gz 2024-02-21T23:18:24.735 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.67378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82198.log 2024-02-21T23:18:24.736 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.67378.log.gz 2024-02-21T23:18:24.737 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.43913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65608.log 2024-02-21T23:18:24.737 INFO:teuthology.orchestra.run.smithi028.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43913.log.gz 2024-02-21T23:18:24.738 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.82198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87774.log 2024-02-21T23:18:24.738 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.82198.log.gz 2024-02-21T23:18:24.739 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.65608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57679.log 2024-02-21T23:18:24.740 INFO:teuthology.orchestra.run.smithi028.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.65608.log.gz 2024-02-21T23:18:24.740 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.87774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61349.log 2024-02-21T23:18:24.741 INFO:teuthology.orchestra.run.smithi028.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.87774.log.gz 2024-02-21T23:18:24.742 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.57679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88274.log 2024-02-21T23:18:24.742 INFO:teuthology.orchestra.run.smithi028.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.57679.log.gz 2024-02-21T23:18:24.743 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.61349.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.61349.log.gz 2024-02-21T23:18:24.743 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72622.log 2024-02-21T23:18:24.744 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.88274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49208.log 2024-02-21T23:18:24.745 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.88274.log.gz 2024-02-21T23:18:24.745 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.72622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72664.log 2024-02-21T23:18:24.746 INFO:teuthology.orchestra.run.smithi028.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.72622.log.gz 2024-02-21T23:18:24.747 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.49208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85126.log 2024-02-21T23:18:24.747 INFO:teuthology.orchestra.run.smithi028.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.49208.log.gz 2024-02-21T23:18:24.748 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.72664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48826.log 2024-02-21T23:18:24.748 INFO:teuthology.orchestra.run.smithi028.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.72664.log.gz 2024-02-21T23:18:24.749 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.85126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84580.log 2024-02-21T23:18:24.749 INFO:teuthology.orchestra.run.smithi028.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.85126.log.gz 2024-02-21T23:18:24.751 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.48826.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48826.log.gz 2024-02-21T23:18:24.751 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56166.log 2024-02-21T23:18:24.752 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.84580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88963.log 2024-02-21T23:18:24.752 INFO:teuthology.orchestra.run.smithi028.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.84580.log.gz 2024-02-21T23:18:24.753 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.56166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68101.log 2024-02-21T23:18:24.754 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.56166.log.gz 2024-02-21T23:18:24.755 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.88963.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.88963.log.gz 2024-02-21T23:18:24.755 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63482.log 2024-02-21T23:18:24.756 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.68101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69805.log 2024-02-21T23:18:24.756 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.68101.log.gz 2024-02-21T23:18:24.757 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.63482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53057.log 2024-02-21T23:18:24.757 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.63482.log.gz 2024-02-21T23:18:24.758 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.69805.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88190.log 2024-02-21T23:18:24.758 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.69805.log.gz 2024-02-21T23:18:24.760 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.53057.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75116.log 2024-02-21T23:18:24.760 INFO:teuthology.orchestra.run.smithi028.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.53057.log.gz 2024-02-21T23:18:24.761 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.88190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46416.log 2024-02-21T23:18:24.761 INFO:teuthology.orchestra.run.smithi028.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.88190.log.gz 2024-02-21T23:18:24.762 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.75116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86944.log 2024-02-21T23:18:24.762 INFO:teuthology.orchestra.run.smithi028.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.75116.log.gz 2024-02-21T23:18:24.763 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.46416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41446.log 2024-02-21T23:18:24.763 INFO:teuthology.orchestra.run.smithi028.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46416.log.gz 2024-02-21T23:18:24.765 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.86944.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88232.log 2024-02-21T23:18:24.765 INFO:teuthology.orchestra.run.smithi028.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.86944.log.gz 2024-02-21T23:18:24.766 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.41446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47361.log 2024-02-21T23:18:24.766 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.41446.log.gz 2024-02-21T23:18:24.767 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.88232.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78628.log 2024-02-21T23:18:24.767 INFO:teuthology.orchestra.run.smithi028.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.88232.log.gz 2024-02-21T23:18:24.768 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.47361.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-02-21T23:18:24.768 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.47361.log.gz 2024-02-21T23:18:24.769 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.78628.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82678.log 2024-02-21T23:18:24.769 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.78628.log.gz 2024-02-21T23:18:24.771 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57194.log 2024-02-21T23:18:24.772 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.82678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57970.log 2024-02-21T23:18:24.772 INFO:teuthology.orchestra.run.smithi028.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.82678.log.gz 2024-02-21T23:18:24.773 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.57194.log: 88.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.57194.log.gz 2024-02-21T23:18:24.773 INFO:teuthology.orchestra.run.smithi028.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.43019.log 2024-02-21T23:18:24.775 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.57970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62164.log 2024-02-21T23:18:24.775 INFO:teuthology.orchestra.run.smithi028.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.57970.log.gz 2024-02-21T23:18:24.776 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.43019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87191.log 2024-02-21T23:18:24.776 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43019.log.gz 2024-02-21T23:18:24.777 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.62164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70538.log 2024-02-21T23:18:24.777 INFO:teuthology.orchestra.run.smithi028.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.62164.log.gz 2024-02-21T23:18:24.778 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.87191.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46020.log 2024-02-21T23:18:24.779 INFO:teuthology.orchestra.run.smithi028.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.87191.log.gz 2024-02-21T23:18:24.780 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.70538.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62809.log 2024-02-21T23:18:24.780 INFO:teuthology.orchestra.run.smithi028.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.70538.log.gz 2024-02-21T23:18:24.781 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.46020.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46020.log.gz 2024-02-21T23:18:24.781 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55059.log 2024-02-21T23:18:24.782 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.62809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41036.log 2024-02-21T23:18:24.782 INFO:teuthology.orchestra.run.smithi028.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.62809.log.gz 2024-02-21T23:18:24.783 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.55059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73283.log 2024-02-21T23:18:24.784 INFO:teuthology.orchestra.run.smithi028.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.55059.log.gz 2024-02-21T23:18:24.785 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.41036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65235.log 2024-02-21T23:18:24.785 INFO:teuthology.orchestra.run.smithi028.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41036.log.gz 2024-02-21T23:18:24.786 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.73283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72035.log 2024-02-21T23:18:24.786 INFO:teuthology.orchestra.run.smithi028.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.73283.log.gz 2024-02-21T23:18:24.787 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.65235.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63230.log 2024-02-21T23:18:24.787 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.65235.log.gz 2024-02-21T23:18:24.789 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.72035.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87689.log 2024-02-21T23:18:24.789 INFO:teuthology.orchestra.run.smithi028.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.72035.log.gz 2024-02-21T23:18:24.790 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.63230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87629.log 2024-02-21T23:18:24.790 INFO:teuthology.orchestra.run.smithi028.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.63230.log.gz 2024-02-21T23:18:24.791 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.87689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48459.log 2024-02-21T23:18:24.791 INFO:teuthology.orchestra.run.smithi028.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.87689.log.gz 2024-02-21T23:18:24.792 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.87629.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54042.log 2024-02-21T23:18:24.792 INFO:teuthology.orchestra.run.smithi028.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.87629.log.gz 2024-02-21T23:18:24.793 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.48459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86338.log 2024-02-21T23:18:24.793 INFO:teuthology.orchestra.run.smithi028.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.48459.log.gz 2024-02-21T23:18:24.794 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.54042.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65902.log 2024-02-21T23:18:24.795 INFO:teuthology.orchestra.run.smithi028.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54042.log.gz 2024-02-21T23:18:24.796 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.86338.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87028.log 2024-02-21T23:18:24.796 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.86338.log.gz 2024-02-21T23:18:24.797 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.65902.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72351.log 2024-02-21T23:18:24.797 INFO:teuthology.orchestra.run.smithi028.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.65902.log.gz 2024-02-21T23:18:24.798 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.87028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62706.log 2024-02-21T23:18:24.798 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.87028.log.gz 2024-02-21T23:18:24.799 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.72351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83368.log 2024-02-21T23:18:24.800 INFO:teuthology.orchestra.run.smithi028.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.72351.log.gz 2024-02-21T23:18:24.801 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.62706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59223.log 2024-02-21T23:18:24.801 INFO:teuthology.orchestra.run.smithi028.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.62706.log.gz 2024-02-21T23:18:24.802 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.83368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58432.log 2024-02-21T23:18:24.803 INFO:teuthology.orchestra.run.smithi028.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.83368.log.gz 2024-02-21T23:18:24.803 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.59223.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87233.log 2024-02-21T23:18:24.804 INFO:teuthology.orchestra.run.smithi028.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.59223.log.gz 2024-02-21T23:18:24.805 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.58432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70348.log 2024-02-21T23:18:24.805 INFO:teuthology.orchestra.run.smithi028.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.58432.log.gz 2024-02-21T23:18:24.806 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.87233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68143.log 2024-02-21T23:18:24.806 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.87233.log.gz 2024-02-21T23:18:24.807 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.70348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80199.log 2024-02-21T23:18:24.807 INFO:teuthology.orchestra.run.smithi028.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.70348.log.gz 2024-02-21T23:18:24.808 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.68143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61103.log 2024-02-21T23:18:24.809 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.68143.log.gz 2024-02-21T23:18:24.810 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.80199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88820.log 2024-02-21T23:18:24.810 INFO:teuthology.orchestra.run.smithi028.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.80199.log.gz 2024-02-21T23:18:24.811 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.61103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63020.log 2024-02-21T23:18:24.811 INFO:teuthology.orchestra.run.smithi028.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.61103.log.gz 2024-02-21T23:18:24.812 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.88820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58264.log 2024-02-21T23:18:24.812 INFO:teuthology.orchestra.run.smithi028.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.88820.log.gz 2024-02-21T23:18:24.813 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.63020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50901.log 2024-02-21T23:18:24.814 INFO:teuthology.orchestra.run.smithi028.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.63020.log.gz 2024-02-21T23:18:24.814 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.58264.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.58264.log.gz 2024-02-21T23:18:24.814 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65193.log 2024-02-21T23:18:24.815 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.50901.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59626.log 2024-02-21T23:18:24.816 INFO:teuthology.orchestra.run.smithi028.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.50901.log.gz 2024-02-21T23:18:24.817 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.65193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57152.log 2024-02-21T23:18:24.817 INFO:teuthology.orchestra.run.smithi028.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.65193.log.gz 2024-02-21T23:18:24.818 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.59626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71042.log 2024-02-21T23:18:24.818 INFO:teuthology.orchestra.run.smithi028.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.59626.log.gz 2024-02-21T23:18:24.819 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.57152.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.57152.log.gz 2024-02-21T23:18:24.819 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69118.log 2024-02-21T23:18:24.820 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.71042.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.31041.log 2024-02-21T23:18:24.820 INFO:teuthology.orchestra.run.smithi028.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.71042.log.gz 2024-02-21T23:18:24.821 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.69118.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74780.log 2024-02-21T23:18:24.821 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.69118.log.gz 2024-02-21T23:18:24.822 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph.tmp-client.admin.31041.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.67420.log -- replaced with /var/log/ceph/ceph.tmp-client.admin.31041.log.gz 2024-02-21T23:18:24.822 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-02-21T23:18:24.823 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.74780.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.74780.log.gz 2024-02-21T23:18:24.824 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69549.log 2024-02-21T23:18:24.825 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.67420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46794.log 2024-02-21T23:18:24.825 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.67420.log.gz 2024-02-21T23:18:24.826 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.69549.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.69549.log.gz 2024-02-21T23:18:24.826 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58885.log 2024-02-21T23:18:24.827 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.46794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67913.log 2024-02-21T23:18:24.827 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46794.log.gz 2024-02-21T23:18:24.828 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.58885.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.58885.log.gz 2024-02-21T23:18:24.828 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87980.log 2024-02-21T23:18:24.829 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.67913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53748.log 2024-02-21T23:18:24.830 INFO:teuthology.orchestra.run.smithi028.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.67913.log.gz 2024-02-21T23:18:24.831 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.87980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54685.log 2024-02-21T23:18:24.831 INFO:teuthology.orchestra.run.smithi028.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.87980.log.gz 2024-02-21T23:18:24.832 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.53748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83098.log 2024-02-21T23:18:24.832 INFO:teuthology.orchestra.run.smithi028.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.53748.log.gz 2024-02-21T23:18:24.833 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.54685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43596.log 2024-02-21T23:18:24.834 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.54685.log.gz 2024-02-21T23:18:24.834 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.83098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85354.log 2024-02-21T23:18:24.835 INFO:teuthology.orchestra.run.smithi028.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.83098.log.gz 2024-02-21T23:18:24.836 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.43596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78303.log 2024-02-21T23:18:24.836 INFO:teuthology.orchestra.run.smithi028.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.43596.log.gz 2024-02-21T23:18:24.837 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.85354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41656.log 2024-02-21T23:18:24.838 INFO:teuthology.orchestra.run.smithi028.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.85354.log.gz 2024-02-21T23:18:24.839 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.78303.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53183.log 2024-02-21T23:18:24.839 INFO:teuthology.orchestra.run.smithi028.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.78303.log.gz 2024-02-21T23:18:24.840 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.41656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41318.log 2024-02-21T23:18:24.846 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41656.log.gz 2024-02-21T23:18:24.846 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.53183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69847.log 2024-02-21T23:18:24.846 INFO:teuthology.orchestra.run.smithi028.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.53183.log.gz 2024-02-21T23:18:24.846 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.41318.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41318.log.gz 2024-02-21T23:18:24.846 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50738.log 2024-02-21T23:18:24.846 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.69847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65398.log 2024-02-21T23:18:24.847 INFO:teuthology.orchestra.run.smithi028.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.69847.log.gz 2024-02-21T23:18:24.847 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.50738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66521.log 2024-02-21T23:18:24.847 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50738.log.gz 2024-02-21T23:18:24.847 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.65398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72790.log 2024-02-21T23:18:24.849 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.65398.log.gz 2024-02-21T23:18:24.849 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.66521.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.66521.log.gz 2024-02-21T23:18:24.849 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57450.log 2024-02-21T23:18:24.849 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.72790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78093.log 2024-02-21T23:18:24.849 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.72790.log.gz 2024-02-21T23:18:24.850 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.57450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84436.log 2024-02-21T23:18:24.850 INFO:teuthology.orchestra.run.smithi028.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.57450.log.gz 2024-02-21T23:18:24.850 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.78093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77384.log 2024-02-21T23:18:24.851 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.78093.log.gz 2024-02-21T23:18:24.851 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.84436.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.84436.log.gz 2024-02-21T23:18:24.852 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76448.log 2024-02-21T23:18:24.852 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.77384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79136.log 2024-02-21T23:18:24.853 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.76448.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.77384.log.gz 2024-02-21T23:18:24.854 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69412.log 2024-02-21T23:18:24.854 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.76448.log.gz 2024-02-21T23:18:24.855 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.79136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71102.log 2024-02-21T23:18:24.855 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.79136.log.gz 2024-02-21T23:18:24.856 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.69412.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43583.log 2024-02-21T23:18:24.856 INFO:teuthology.orchestra.run.smithi028.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.69412.log.gz 2024-02-21T23:18:24.857 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.71102.log: 88.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.71102.log.gz 2024-02-21T23:18:24.857 INFO:teuthology.orchestra.run.smithi028.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.82804.log 2024-02-21T23:18:24.858 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.43583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48731.log 2024-02-21T23:18:24.858 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43583.log.gz 2024-02-21T23:18:24.859 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.82804.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.82804.log.gz 2024-02-21T23:18:24.859 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79052.log 2024-02-21T23:18:24.860 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.48731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43373.log 2024-02-21T23:18:24.861 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48731.log.gz 2024-02-21T23:18:24.862 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.79052.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.79052.log.gz 2024-02-21T23:18:24.862 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72119.log 2024-02-21T23:18:24.863 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.43373.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81423.log 2024-02-21T23:18:24.863 INFO:teuthology.orchestra.run.smithi028.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43373.log.gz 2024-02-21T23:18:24.864 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.72119.log: gzip 88.9% -- replaced with /var/log/ceph/ceph-client.admin.72119.log.gz -5 2024-02-21T23:18:24.864 INFO:teuthology.orchestra.run.smithi028.stderr: --verbose -- /var/log/ceph/ceph-client.admin.80325.log 2024-02-21T23:18:24.865 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.81423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58222.log 2024-02-21T23:18:24.865 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.81423.log.gz 2024-02-21T23:18:24.866 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.80325.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.80325.log.gz 2024-02-21T23:18:24.866 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87070.log 2024-02-21T23:18:24.867 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.58222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85396.log 2024-02-21T23:18:24.868 INFO:teuthology.orchestra.run.smithi028.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.58222.log.gz 2024-02-21T23:18:24.869 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.87070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64628.log 2024-02-21T23:18:24.869 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.87070.log.gz 2024-02-21T23:18:24.870 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.85396.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63951.log 2024-02-21T23:18:24.870 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.85396.log.gz 2024-02-21T23:18:24.871 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.64628.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80409.log 2024-02-21T23:18:24.872 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.64628.log.gz 2024-02-21T23:18:24.873 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.63951.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86548.log 2024-02-21T23:18:24.873 INFO:teuthology.orchestra.run.smithi028.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.63951.log.gz 2024-02-21T23:18:24.874 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.80409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71814.log 2024-02-21T23:18:24.874 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.80409.log.gz 2024-02-21T23:18:24.876 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.86548.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.86548.log.gz 2024-02-21T23:18:24.876 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59542.log 2024-02-21T23:18:24.877 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.71814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74038.log 2024-02-21T23:18:24.877 INFO:teuthology.orchestra.run.smithi028.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.71814.log.gz 2024-02-21T23:18:24.878 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.59542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60375.log 2024-02-21T23:18:24.879 INFO:teuthology.orchestra.run.smithi028.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.59542.log.gz 2024-02-21T23:18:24.880 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.74038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53958.log 2024-02-21T23:18:24.880 INFO:teuthology.orchestra.run.smithi028.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.74038.log.gz 2024-02-21T23:18:24.881 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.60375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74266.log 2024-02-21T23:18:24.881 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.60375.log.gz 2024-02-21T23:18:24.882 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.53958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82972.log 2024-02-21T23:18:24.883 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53958.log.gz 2024-02-21T23:18:24.883 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.74266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54973.log 2024-02-21T23:18:24.884 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.74266.log.gz 2024-02-21T23:18:24.885 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.82972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84706.log 2024-02-21T23:18:24.885 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.82972.log.gz 2024-02-21T23:18:24.886 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.54973.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60585.log 2024-02-21T23:18:24.886 INFO:teuthology.orchestra.run.smithi028.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.54973.log.gz 2024-02-21T23:18:24.888 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.84706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66070.log 2024-02-21T23:18:24.888 INFO:teuthology.orchestra.run.smithi028.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.84706.log.gz 2024-02-21T23:18:24.889 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.60585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64164.log 2024-02-21T23:18:24.889 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.60585.log.gz 2024-02-21T23:18:24.890 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.66070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53015.log 2024-02-21T23:18:24.890 INFO:teuthology.orchestra.run.smithi028.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.66070.log.gz 2024-02-21T23:18:24.891 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.64164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76934.log 2024-02-21T23:18:24.892 INFO:teuthology.orchestra.run.smithi028.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.64164.log.gz 2024-02-21T23:18:24.893 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.53015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80115.log 2024-02-21T23:18:24.893 INFO:teuthology.orchestra.run.smithi028.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.53015.log.gz 2024-02-21T23:18:24.894 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.76934.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73846.log 2024-02-21T23:18:24.894 INFO:teuthology.orchestra.run.smithi028.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.76934.log.gz 2024-02-21T23:18:24.895 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.80115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61660.log 2024-02-21T23:18:24.896 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.80115.log.gz 2024-02-21T23:18:24.897 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.73846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56813.log 2024-02-21T23:18:24.897 INFO:teuthology.orchestra.run.smithi028.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.73846.log.gz 2024-02-21T23:18:24.898 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.61660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41698.log 2024-02-21T23:18:24.898 INFO:teuthology.orchestra.run.smithi028.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.61660.log.gz 2024-02-21T23:18:24.900 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.56813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68059.log 2024-02-21T23:18:24.900 INFO:teuthology.orchestra.run.smithi028.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.56813.log.gz 2024-02-21T23:18:24.901 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.41698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59902.log 2024-02-21T23:18:24.901 INFO:teuthology.orchestra.run.smithi028.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.41698.log.gz 2024-02-21T23:18:24.902 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.68059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55326.log 2024-02-21T23:18:24.903 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.68059.log.gz 2024-02-21T23:18:24.904 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.59902.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45726.log 2024-02-21T23:18:24.905 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.55326.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.59902.log.gz 2024-02-21T23:18:24.905 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58474.log 2024-02-21T23:18:24.906 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.55326.log.gz 2024-02-21T23:18:24.906 INFO:teuthology.orchestra.run.smithi028.stderr:gzip/var/log/ceph/ceph-client.admin.45726.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.71730.log 2024-02-21T23:18:24.907 INFO:teuthology.orchestra.run.smithi028.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45726.log.gz 2024-02-21T23:18:24.908 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.58474.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78135.log 2024-02-21T23:18:24.908 INFO:teuthology.orchestra.run.smithi028.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.58474.log.gz 2024-02-21T23:18:24.909 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.71730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51412.log 2024-02-21T23:18:24.909 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.71730.log.gz 2024-02-21T23:18:24.911 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.78135.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62851.log 2024-02-21T23:18:24.911 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.78135.log.gz 2024-02-21T23:18:24.912 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.51412.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67018.log 2024-02-21T23:18:24.912 INFO:teuthology.orchestra.run.smithi028.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51412.log.gz 2024-02-21T23:18:24.914 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.62851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65482.log 2024-02-21T23:18:24.914 INFO:teuthology.orchestra.run.smithi028.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.62851.log.gz 2024-02-21T23:18:24.915 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.67018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85522.log 2024-02-21T23:18:24.915 INFO:teuthology.orchestra.run.smithi028.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.67018.log.gz 2024-02-21T23:18:24.916 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.65482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42191.log 2024-02-21T23:18:24.916 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.65482.log.gz 2024-02-21T23:18:24.918 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.85522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78800.log 2024-02-21T23:18:24.918 INFO:teuthology.orchestra.run.smithi028.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.85522.log.gz 2024-02-21T23:18:24.919 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.42191.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-02-21T23:18:24.920 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.78800.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.78800.log.gz 2024-02-21T23:18:24.921 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77318.log 2024-02-21T23:18:24.921 INFO:teuthology.orchestra.run.smithi028.stderr: 92.1% -- replaced with /var/log/ceph/ceph-client.admin.42191.log.gz 2024-02-21T23:18:24.922 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79905.log 2024-02-21T23:18:24.923 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-osd.3.log: /var/log/ceph/ceph-client.admin.77318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78009.log 2024-02-21T23:18:24.924 INFO:teuthology.orchestra.run.smithi028.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.77318.log.gz 2024-02-21T23:18:24.925 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.79905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84832.log 2024-02-21T23:18:24.925 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.79905.log.gz 2024-02-21T23:18:24.926 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.78009.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65440.log 2024-02-21T23:18:24.926 INFO:teuthology.orchestra.run.smithi028.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.78009.log.gz 2024-02-21T23:18:24.928 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.84832.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55410.log 2024-02-21T23:18:24.928 INFO:teuthology.orchestra.run.smithi028.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.84832.log.gz 2024-02-21T23:18:24.929 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.65440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59584.log 2024-02-21T23:18:24.929 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.65440.log.gz 2024-02-21T23:18:24.930 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.55410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52680.log 2024-02-21T23:18:24.931 INFO:teuthology.orchestra.run.smithi028.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55410.log.gz 2024-02-21T23:18:24.932 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.59584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72580.log 2024-02-21T23:18:24.932 INFO:teuthology.orchestra.run.smithi028.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.59584.log.gz 2024-02-21T23:18:24.933 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.52680.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49316.log 2024-02-21T23:18:24.933 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52680.log.gz 2024-02-21T23:18:24.934 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.72580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60459.log 2024-02-21T23:18:24.935 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.72580.log.gz 2024-02-21T23:18:24.936 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.49316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66976.log 2024-02-21T23:18:24.936 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49316.log.gz 2024-02-21T23:18:24.937 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.60459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70125.log 2024-02-21T23:18:24.937 INFO:teuthology.orchestra.run.smithi028.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.60459.log.gz 2024-02-21T23:18:24.938 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.66976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48998.log 2024-02-21T23:18:24.939 INFO:teuthology.orchestra.run.smithi028.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.66976.log.gz 2024-02-21T23:18:24.940 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.70125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64036.log 2024-02-21T23:18:24.940 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.70125.log.gz 2024-02-21T23:18:24.941 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.48998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83140.log 2024-02-21T23:18:24.941 INFO:teuthology.orchestra.run.smithi028.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48998.log.gz 2024-02-21T23:18:24.942 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.64036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59051.log 2024-02-21T23:18:24.943 INFO:teuthology.orchestra.run.smithi028.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.64036.log.gz 2024-02-21T23:18:24.944 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.83140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48417.log 2024-02-21T23:18:24.944 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.83140.log.gz 2024-02-21T23:18:24.945 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.59051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72477.log 2024-02-21T23:18:24.946 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.59051.log.gz 2024-02-21T23:18:24.947 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.48417.log: gzip 88.4% -5 -- replaced with /var/log/ceph/ceph-client.admin.48417.log.gz --verbose 2024-02-21T23:18:24.947 INFO:teuthology.orchestra.run.smithi028.stderr: -- /var/log/ceph/ceph-client.admin.86902.log 2024-02-21T23:18:24.948 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.72477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43289.log 2024-02-21T23:18:24.948 INFO:teuthology.orchestra.run.smithi028.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.72477.log.gz/var/log/ceph/ceph-client.admin.86902.log: 2024-02-21T23:18:24.949 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.86902.log.gz 2024-02-21T23:18:24.949 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49358.log 2024-02-21T23:18:24.950 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.43289.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51668.log 2024-02-21T23:18:24.950 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.49358.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.43289.log.gz 2024-02-21T23:18:24.951 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88064.log 2024-02-21T23:18:24.951 INFO:teuthology.orchestra.run.smithi028.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.49358.log.gz 2024-02-21T23:18:24.952 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.51668.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51668.log.gz 2024-02-21T23:18:24.953 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85168.log 2024-02-21T23:18:24.953 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.88064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88652.log 2024-02-21T23:18:24.954 INFO:teuthology.orchestra.run.smithi028.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.88064.log.gz 2024-02-21T23:18:24.955 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.85168.log: gzip 88.3% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.85168.log.gz -- 2024-02-21T23:18:24.955 INFO:teuthology.orchestra.run.smithi028.stderr: /var/log/ceph/ceph-client.admin.66647.log 2024-02-21T23:18:24.956 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.88652.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.88652.log.gz 2024-02-21T23:18:24.956 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58054.log 2024-02-21T23:18:24.957 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.66647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73117.log 2024-02-21T23:18:24.958 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.66647.log.gz 2024-02-21T23:18:24.959 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.58054.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43331.log 2024-02-21T23:18:24.959 INFO:teuthology.orchestra.run.smithi028.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.58054.log.gz 2024-02-21T23:18:24.960 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.73117.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.73117.log.gz 2024-02-21T23:18:24.960 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67655.log 2024-02-21T23:18:24.961 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.43331.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79863.log 2024-02-21T23:18:24.961 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.67655.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.43331.log.gz 2024-02-21T23:18:24.962 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88484.log 2024-02-21T23:18:24.962 INFO:teuthology.orchestra.run.smithi028.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.67655.log.gz 2024-02-21T23:18:24.963 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.79863.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.79863.log.gz 2024-02-21T23:18:24.963 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52196.log 2024-02-21T23:18:24.964 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.88484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48247.log 2024-02-21T23:18:24.965 INFO:teuthology.orchestra.run.smithi028.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.88484.log.gz 2024-02-21T23:18:24.966 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.52196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61828.log 2024-02-21T23:18:24.966 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52196.log.gz 2024-02-21T23:18:24.967 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.48247.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48247.log.gz 2024-02-21T23:18:24.967 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71604.log 2024-02-21T23:18:24.968 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.61828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67955.log 2024-02-21T23:18:24.969 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.61828.log.gz 2024-02-21T23:18:24.969 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.71604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81465.log 2024-02-21T23:18:24.969 INFO:teuthology.orchestra.run.smithi028.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.71604.log.gz 2024-02-21T23:18:24.971 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.67955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48045.log 2024-02-21T23:18:24.971 INFO:teuthology.orchestra.run.smithi028.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.67955.log.gz 2024-02-21T23:18:24.971 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.81465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82930.log 2024-02-21T23:18:24.972 INFO:teuthology.orchestra.run.smithi028.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.81465.log.gz 2024-02-21T23:18:24.973 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.48045.log: 88.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.48045.log.gz -5 2024-02-21T23:18:24.973 INFO:teuthology.orchestra.run.smithi028.stderr: --verbose -- /var/log/ceph/ceph-client.admin.68185.log 2024-02-21T23:18:24.974 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.82930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46500.log 2024-02-21T23:18:24.974 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.82930.log.gz 2024-02-21T23:18:24.975 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.68185.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64078.log 2024-02-21T23:18:24.976 INFO:teuthology.orchestra.run.smithi028.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.68185.log.gz 2024-02-21T23:18:24.976 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.46500.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78670.log 2024-02-21T23:18:24.977 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.46500.log.gz 2024-02-21T23:18:24.978 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.64078.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.64078.log.gz 2024-02-21T23:18:24.978 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58390.log 2024-02-21T23:18:24.979 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.78670.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.78670.log.gz 2024-02-21T23:18:24.979 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72748.log 2024-02-21T23:18:24.980 INFO:teuthology.orchestra.run.smithi028.stderr:gzip/var/log/ceph/ceph-client.admin.58390.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.59458.log 2024-02-21T23:18:24.980 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.72748.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.58390.log.gz 2024-02-21T23:18:24.981 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose 87.6% -- /var/log/ceph/ceph-client.admin.45894.log -- replaced with /var/log/ceph/ceph-client.admin.72748.log.gz 2024-02-21T23:18:24.981 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-02-21T23:18:24.982 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.59458.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.59458.log.gz 2024-02-21T23:18:24.983 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66112.log 2024-02-21T23:18:24.983 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.45894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78756.log 2024-02-21T23:18:24.984 INFO:teuthology.orchestra.run.smithi028.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.45894.log.gz 2024-02-21T23:18:24.985 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.66112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80776.log 2024-02-21T23:18:24.985 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.66112.log.gz 2024-02-21T23:18:24.986 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.78756.log: 88.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.78756.log.gz 2024-02-21T23:18:24.986 INFO:teuthology.orchestra.run.smithi028.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41488.log 2024-02-21T23:18:24.987 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.80776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50443.log 2024-02-21T23:18:24.988 INFO:teuthology.orchestra.run.smithi028.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.80776.log.gz 2024-02-21T23:18:24.988 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.41488.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41488.log.gz 2024-02-21T23:18:24.989 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43915.log 2024-02-21T23:18:24.989 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.50443.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.50443.log.gz 2024-02-21T23:18:24.990 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43061.log 2024-02-21T23:18:24.990 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.43915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56082.log 2024-02-21T23:18:24.991 INFO:teuthology.orchestra.run.smithi028.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.43915.log.gz 2024-02-21T23:18:24.992 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.43061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73930.log 2024-02-21T23:18:24.992 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43061.log.gz 2024-02-21T23:18:24.993 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.56082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81573.log 2024-02-21T23:18:24.993 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.56082.log.gz 2024-02-21T23:18:24.994 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.73930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61870.log 2024-02-21T23:18:24.995 INFO:teuthology.orchestra.run.smithi028.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.73930.log.gz 2024-02-21T23:18:24.995 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.81573.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63650.log 2024-02-21T23:18:24.996 INFO:teuthology.orchestra.run.smithi028.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.81573.log.gz 2024-02-21T23:18:24.997 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.61870.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.61870.log.gz 2024-02-21T23:18:24.997 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75555.log 2024-02-21T23:18:24.998 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.63650.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.63650.log.gz 2024-02-21T23:18:24.998 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66196.log 2024-02-21T23:18:24.999 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.75555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88358.log 2024-02-21T23:18:24.999 INFO:teuthology.orchestra.run.smithi028.stderr: 88.1%/var/log/ceph/ceph-client.admin.66196.log: -- replaced with /var/log/ceph/ceph-client.admin.75555.log.gz 2024-02-21T23:18:25.000 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50528.log 2024-02-21T23:18:25.000 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.66196.log.gz 2024-02-21T23:18:25.001 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.88358.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.88358.log.gz 2024-02-21T23:18:25.002 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76976.log 2024-02-21T23:18:25.003 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.50528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51189.log 2024-02-21T23:18:25.003 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50528.log.gz 2024-02-21T23:18:25.004 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.76976.log: 88.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.76976.log.gz -5 2024-02-21T23:18:25.004 INFO:teuthology.orchestra.run.smithi028.stderr: --verbose -- /var/log/ceph/ceph-client.admin.87545.log 2024-02-21T23:18:25.005 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.51189.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63993.log 2024-02-21T23:18:25.005 INFO:teuthology.orchestra.run.smithi028.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.51189.log.gz 2024-02-21T23:18:25.006 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.87545.log: gzip 88.1% -- replaced with /var/log/ceph/ceph-client.admin.87545.log.gz -5 2024-02-21T23:18:25.007 INFO:teuthology.orchestra.run.smithi028.stderr: --verbose -- /var/log/ceph/ceph-client.admin.53706.log 2024-02-21T23:18:25.007 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.63993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43415.log 2024-02-21T23:18:25.008 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.53706.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.63993.log.gz 2024-02-21T23:18:25.009 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46104.log 2024-02-21T23:18:25.009 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53706.log.gz 2024-02-21T23:18:25.010 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.43415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85816.log 2024-02-21T23:18:25.010 INFO:teuthology.orchestra.run.smithi028.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.43415.log.gz 2024-02-21T23:18:25.011 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.46104.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74906.log 2024-02-21T23:18:25.011 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46104.log.gz 2024-02-21T23:18:25.012 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.85816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71646.log 2024-02-21T23:18:25.013 INFO:teuthology.orchestra.run.smithi028.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.85816.log.gz 2024-02-21T23:18:25.014 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.74906.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78261.log 2024-02-21T23:18:25.014 INFO:teuthology.orchestra.run.smithi028.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.74906.log.gz 2024-02-21T23:18:25.015 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.71646.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.71646.log.gz 2024-02-21T23:18:25.015 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66417.log 2024-02-21T23:18:25.016 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.78261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66479.log 2024-02-21T23:18:25.016 INFO:teuthology.orchestra.run.smithi028.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.78261.log.gz 2024-02-21T23:18:25.017 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.66417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44185.log 2024-02-21T23:18:25.017 INFO:teuthology.orchestra.run.smithi028.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.66417.log.gz 2024-02-21T23:18:25.018 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.66479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79737.log 2024-02-21T23:18:25.019 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.66479.log.gz 2024-02-21T23:18:25.019 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.44185.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80367.log 2024-02-21T23:18:25.020 INFO:teuthology.orchestra.run.smithi028.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44185.log.gz 2024-02-21T23:18:25.020 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.79737.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41139.log 2024-02-21T23:18:25.021 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.79737.log.gz 2024-02-21T23:18:25.022 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.80367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65860.log 2024-02-21T23:18:25.022 INFO:teuthology.orchestra.run.smithi028.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.80367.log.gz 2024-02-21T23:18:25.023 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.41139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75922.log 2024-02-21T23:18:25.023 INFO:teuthology.orchestra.run.smithi028.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41139.log.gz 2024-02-21T23:18:25.024 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.65860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66238.log 2024-02-21T23:18:25.025 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.65860.log.gz 2024-02-21T23:18:25.025 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.75922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64778.log 2024-02-21T23:18:25.026 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.75922.log.gz 2024-02-21T23:18:25.027 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.66238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45978.log 2024-02-21T23:18:25.027 INFO:teuthology.orchestra.run.smithi028.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.66238.log.gz 2024-02-21T23:18:25.028 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.64778.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.64778.log.gz 2024-02-21T23:18:25.028 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47277.log 2024-02-21T23:18:25.029 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.45978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57492.log 2024-02-21T23:18:25.029 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45978.log.gz 2024-02-21T23:18:25.030 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.47277.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.47277.log.gz 2024-02-21T23:18:25.030 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86692.log 2024-02-21T23:18:25.031 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.57492.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.57492.log.gz 2024-02-21T23:18:25.032 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56645.log 2024-02-21T23:18:25.032 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.86692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75387.log 2024-02-21T23:18:25.033 INFO:teuthology.orchestra.run.smithi028.stderr: 88.9%/var/log/ceph/ceph-client.admin.56645.log: -- replaced with /var/log/ceph/ceph-client.admin.86692.log.gz 2024-02-21T23:18:25.034 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46206.log 2024-02-21T23:18:25.034 INFO:teuthology.orchestra.run.smithi028.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.56645.log.gz 2024-02-21T23:18:25.035 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.75387.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80988.log 2024-02-21T23:18:25.035 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.75387.log.gz 2024-02-21T23:18:25.036 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.46206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76008.log 2024-02-21T23:18:25.036 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46206.log.gz 2024-02-21T23:18:25.037 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.80988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67547.log 2024-02-21T23:18:25.038 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.80988.log.gz 2024-02-21T23:18:25.039 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.76008.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.76008.log.gz 2024-02-21T23:18:25.039 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63566.log 2024-02-21T23:18:25.040 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.67547.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.67547.log.gz 2024-02-21T23:18:25.040 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71898.log 2024-02-21T23:18:25.041 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.63566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84748.log 2024-02-21T23:18:25.041 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.63566.log.gz 2024-02-21T23:18:25.042 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.71898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45852.log 2024-02-21T23:18:25.042 INFO:teuthology.orchestra.run.smithi028.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.71898.log.gz 2024-02-21T23:18:25.043 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.84748.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.84748.log.gz 2024-02-21T23:18:25.043 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79556.log 2024-02-21T23:18:25.049 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.45852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66317.log 2024-02-21T23:18:25.049 INFO:teuthology.orchestra.run.smithi028.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45852.log.gz 2024-02-21T23:18:25.050 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.79556.log: gzip 88.5% -5 -- replaced with /var/log/ceph/ceph-client.admin.79556.log.gz --verbose 2024-02-21T23:18:25.050 INFO:teuthology.orchestra.run.smithi028.stderr: -- /var/log/ceph/ceph-client.admin.81729.log 2024-02-21T23:18:25.050 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.66317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60417.log 2024-02-21T23:18:25.050 INFO:teuthology.orchestra.run.smithi028.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.66317.log.gz 2024-02-21T23:18:25.050 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.81729.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.81729.log.gz 2024-02-21T23:18:25.050 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74822.log 2024-02-21T23:18:25.050 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.60417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66691.log 2024-02-21T23:18:25.051 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.74822.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.60417.log.gz 2024-02-21T23:18:25.051 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- 88.0% /var/log/ceph/ceph-client.admin.66154.log 2024-02-21T23:18:25.051 INFO:teuthology.orchestra.run.smithi028.stderr: -- replaced with /var/log/ceph/ceph-client.admin.74822.log.gz 2024-02-21T23:18:25.054 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.66691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57868.log 2024-02-21T23:18:25.054 INFO:teuthology.orchestra.run.smithi028.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.66691.log.gz 2024-02-21T23:18:25.054 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.66154.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.66154.log.gz 2024-02-21T23:18:25.055 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82072.log 2024-02-21T23:18:25.056 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.57868.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.57868.log.gz 2024-02-21T23:18:25.056 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51860.log 2024-02-21T23:18:25.058 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.82072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71314.log 2024-02-21T23:18:25.058 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.82072.log.gz 2024-02-21T23:18:25.059 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.51860.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51860.log.gz 2024-02-21T23:18:25.059 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48501.log 2024-02-21T23:18:25.060 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.71314.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51710.log 2024-02-21T23:18:25.060 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.71314.log.gz 2024-02-21T23:18:25.061 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.48501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55578.log 2024-02-21T23:18:25.061 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48501.log.gz 2024-02-21T23:18:25.062 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.51710.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51710.log.gz 2024-02-21T23:18:25.062 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82720.log 2024-02-21T23:18:25.063 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.55578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60814.log 2024-02-21T23:18:25.064 INFO:teuthology.orchestra.run.smithi028.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.55578.log.gz 2024-02-21T23:18:25.064 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.82720.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.82720.log.gz 2024-02-21T23:18:25.065 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58684.log 2024-02-21T23:18:25.066 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.60814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77715.log 2024-02-21T23:18:25.066 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.60814.log.gz 2024-02-21T23:18:25.067 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.58684.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59776.log 2024-02-21T23:18:25.067 INFO:teuthology.orchestra.run.smithi028.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.58684.log.gz 2024-02-21T23:18:25.068 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.77715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72706.log 2024-02-21T23:18:25.068 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.77715.log.gz 2024-02-21T23:18:25.069 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.59776.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.59776.log.gz 2024-02-21T23:18:25.069 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82156.log 2024-02-21T23:18:25.070 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.72706.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.72706.log.gz 2024-02-21T23:18:25.071 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77505.log 2024-02-21T23:18:25.072 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.82156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77967.log 2024-02-21T23:18:25.072 INFO:teuthology.orchestra.run.smithi028.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.82156.log.gz 2024-02-21T23:18:25.073 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.77505.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75964.log 2024-02-21T23:18:25.073 INFO:teuthology.orchestra.run.smithi028.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.77505.log.gz 2024-02-21T23:18:25.074 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.77967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48956.log 2024-02-21T23:18:25.074 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.77967.log.gz 2024-02-21T23:18:25.076 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.75964.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-02-21T23:18:25.076 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.75964.log.gz 2024-02-21T23:18:25.076 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.48956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70083.log 2024-02-21T23:18:25.077 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.48956.log.gz 2024-02-21T23:18:25.078 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49442.log 2024-02-21T23:18:25.079 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.70083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57236.log 2024-02-21T23:18:25.079 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.70083.log.gz 2024-02-21T23:18:25.081 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.49442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56855.log 2024-02-21T23:18:25.081 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.57236.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.49442.log.gz 2024-02-21T23:18:25.082 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79430.log 2024-02-21T23:18:25.082 INFO:teuthology.orchestra.run.smithi028.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57236.log.gz 2024-02-21T23:18:25.083 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.56855.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84016.log 2024-02-21T23:18:25.083 INFO:teuthology.orchestra.run.smithi028.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.56855.log.gz 2024-02-21T23:18:25.084 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.79430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82282.log 2024-02-21T23:18:25.084 INFO:teuthology.orchestra.run.smithi028.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.79430.log.gz 2024-02-21T23:18:25.085 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.84016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68749.log 2024-02-21T23:18:25.086 INFO:teuthology.orchestra.run.smithi028.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.84016.log.gz 2024-02-21T23:18:25.087 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.82282.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51231.log 2024-02-21T23:18:25.087 INFO:teuthology.orchestra.run.smithi028.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.82282.log.gz 2024-02-21T23:18:25.088 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.68749.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63608.log 2024-02-21T23:18:25.088 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.68749.log.gz 2024-02-21T23:18:25.089 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.51231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88568.log 2024-02-21T23:18:25.089 INFO:teuthology.orchestra.run.smithi028.stderr: 94.6% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-02-21T23:18:25.090 INFO:teuthology.orchestra.run.smithi028.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.51231.log.gz 2024-02-21T23:18:25.090 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.63608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87858.log 2024-02-21T23:18:25.091 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.88568.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.63608.log.gz 2024-02-21T23:18:25.092 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60501.log 2024-02-21T23:18:25.092 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.88568.log.gz 2024-02-21T23:18:25.093 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.87858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50113.log 2024-02-21T23:18:25.093 INFO:teuthology.orchestra.run.smithi028.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.87858.log.gz 2024-02-21T23:18:25.094 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.60501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83974.log 88.5% 2024-02-21T23:18:25.094 INFO:teuthology.orchestra.run.smithi028.stderr: -- replaced with /var/log/ceph/ceph-client.admin.60501.log.gz 2024-02-21T23:18:25.095 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.50113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46989.log 2024-02-21T23:18:25.096 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.83974.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50113.log.gz 2024-02-21T23:18:25.096 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70664.log 2024-02-21T23:18:25.096 INFO:teuthology.orchestra.run.smithi028.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.83974.log.gz 2024-02-21T23:18:25.097 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.46989.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46989.log.gz 2024-02-21T23:18:25.098 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82888.log 2024-02-21T23:18:25.098 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.70664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80451.log 2024-02-21T23:18:25.099 INFO:teuthology.orchestra.run.smithi028.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.70664.log.gz 2024-02-21T23:18:25.100 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.82888.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.82888.log.gz 2024-02-21T23:18:25.100 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49250.log 2024-02-21T23:18:25.101 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.80451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83410.log 2024-02-21T23:18:25.101 INFO:teuthology.orchestra.run.smithi028.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.80451.log.gz 2024-02-21T23:18:25.102 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.49250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62292.log 2024-02-21T23:18:25.103 INFO:teuthology.orchestra.run.smithi028.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.49250.log.gz 2024-02-21T23:18:25.103 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.83410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59416.log 2024-02-21T23:18:25.104 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.83410.log.gz 2024-02-21T23:18:25.105 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.62292.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.62292.log.gz 2024-02-21T23:18:25.105 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76154.log 2024-02-21T23:18:25.106 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.59416.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.59416.log.gz 2024-02-21T23:18:25.106 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52428.log 2024-02-21T23:18:25.107 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56729.log 2024-02-21T23:18:25.108 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.76154.log: /var/log/ceph/ceph-client.admin.52428.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.76154.log.gz 2024-02-21T23:18:25.108 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52428.log.gz 2024-02-21T23:18:25.109 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87816.log 2024-02-21T23:18:25.109 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.56729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71206.log 2024-02-21T23:18:25.110 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.56729.log.gz 2024-02-21T23:18:25.110 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.87816.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.87816.log.gz 2024-02-21T23:18:25.111 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51319.log 2024-02-21T23:18:25.111 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.71206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67462.log 2024-02-21T23:18:25.112 INFO:teuthology.orchestra.run.smithi028.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.71206.log.gz 2024-02-21T23:18:25.113 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.51319.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51319.log.gz 2024-02-21T23:18:25.113 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86128.log 2024-02-21T23:18:25.114 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.67462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70874.log 2024-02-21T23:18:25.115 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.86128.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.67462.log.gz 2024-02-21T23:18:25.115 INFO:teuthology.orchestra.run.smithi028.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.86128.log.gz 2024-02-21T23:18:25.115 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70253.log 2024-02-21T23:18:25.116 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.70874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77018.log 2024-02-21T23:18:25.116 INFO:teuthology.orchestra.run.smithi028.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.70874.log.gz 2024-02-21T23:18:25.117 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.70253.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.70253.log.gz 2024-02-21T23:18:25.117 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84058.log 2024-02-21T23:18:25.118 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.77018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50612.log 2024-02-21T23:18:25.118 INFO:teuthology.orchestra.run.smithi028.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.77018.log.gz 2024-02-21T23:18:25.119 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.84058.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.84058.log.gz 2024-02-21T23:18:25.119 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49542.log 2024-02-21T23:18:25.120 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41404.log 2024-02-21T23:18:25.121 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.50612.log: /var/log/ceph/ceph-client.admin.49542.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.50612.log.gz 2024-02-21T23:18:25.122 INFO:teuthology.orchestra.run.smithi028.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49542.log.gz 2024-02-21T23:18:25.122 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63870.log 2024-02-21T23:18:25.122 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.41404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62664.log 2024-02-21T23:18:25.123 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41404.log.gz 2024-02-21T23:18:25.124 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.63870.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.63870.log.gz 2024-02-21T23:18:25.124 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86464.log 2024-02-21T23:18:25.124 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.62664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71248.log 2024-02-21T23:18:25.125 INFO:teuthology.orchestra.run.smithi028.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.62664.log.gz 2024-02-21T23:18:25.126 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.86464.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.86464.log.gz 2024-02-21T23:18:25.126 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48003.log 2024-02-21T23:18:25.127 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.71248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67228.log 2024-02-21T23:18:25.128 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.48003.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.71248.log.gz 2024-02-21T23:18:25.128 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48003.log.gz 2024-02-21T23:18:25.128 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61306.log 2024-02-21T23:18:25.129 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.67228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52154.log 2024-02-21T23:18:25.130 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.61306.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.67228.log.gz 2024-02-21T23:18:25.130 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60249.log 2024-02-21T23:18:25.131 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.61306.log.gz 2024-02-21T23:18:25.131 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.52154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53916.log 2024-02-21T23:18:25.132 INFO:teuthology.orchestra.run.smithi028.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52154.log.gz 2024-02-21T23:18:25.133 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.60249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68959.log 2024-02-21T23:18:25.133 INFO:teuthology.orchestra.run.smithi028.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.60249.log.gz 2024-02-21T23:18:25.133 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.53916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52470.log 2024-02-21T23:18:25.134 INFO:teuthology.orchestra.run.smithi028.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.53916.log.gz 2024-02-21T23:18:25.135 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.68959.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.68959.log.gz 2024-02-21T23:18:25.135 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81171.log 2024-02-21T23:18:25.136 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.52470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50173.log 2024-02-21T23:18:25.136 INFO:teuthology.orchestra.run.smithi028.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52470.log.gz 2024-02-21T23:18:25.137 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.81171.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.81171.log.gz 2024-02-21T23:18:25.137 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65295.log 2024-02-21T23:18:25.138 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.50173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82450.log 2024-02-21T23:18:25.138 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6%/var/log/ceph/ceph-client.admin.65295.log: -- replaced with /var/log/ceph/ceph-client.admin.50173.log.gz 2024-02-21T23:18:25.139 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79262.log 2024-02-21T23:18:25.139 INFO:teuthology.orchestra.run.smithi028.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.65295.log.gz 2024-02-21T23:18:25.140 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.82450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41572.log 2024-02-21T23:18:25.141 INFO:teuthology.orchestra.run.smithi028.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.82450.log.gz 2024-02-21T23:18:25.142 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.79262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74080.log 2024-02-21T23:18:25.142 INFO:teuthology.orchestra.run.smithi028.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.79262.log.gz 2024-02-21T23:18:25.143 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.41572.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88694.log 2024-02-21T23:18:25.143 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41572.log.gz 2024-02-21T23:18:25.144 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.74080.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.74080.log.gz 2024-02-21T23:18:25.144 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70390.log 2024-02-21T23:18:25.145 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.88694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52238.log 2024-02-21T23:18:25.146 INFO:teuthology.orchestra.run.smithi028.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.88694.log.gz 2024-02-21T23:18:25.146 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.70390.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.70390.log.gz 2024-02-21T23:18:25.146 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72874.log 2024-02-21T23:18:25.147 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.52238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71688.log 2024-02-21T23:18:25.148 INFO:teuthology.orchestra.run.smithi028.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52238.log.gz 2024-02-21T23:18:25.148 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.72874.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.72874.log.gz 2024-02-21T23:18:25.149 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84142.log 2024-02-21T23:18:25.149 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.71688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72537.log 2024-02-21T23:18:25.150 INFO:teuthology.orchestra.run.smithi028.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.71688.log.gz 2024-02-21T23:18:25.151 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.84142.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.84142.log.gz 2024-02-21T23:18:25.151 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63524.log 2024-02-21T23:18:25.152 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.72537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62978.log 2024-02-21T23:18:25.152 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.72537.log.gz 2024-02-21T23:18:25.153 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.63524.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.63524.log.gz 2024-02-21T23:18:25.153 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83014.log 2024-02-21T23:18:25.154 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.62978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80818.log 2024-02-21T23:18:25.154 INFO:teuthology.orchestra.run.smithi028.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.62978.log.gz 2024-02-21T23:18:25.155 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.83014.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.83014.log.gz 2024-02-21T23:18:25.155 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75158.log 2024-02-21T23:18:25.156 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.80818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61702.log 2024-02-21T23:18:25.157 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.80818.log.gz 2024-02-21T23:18:25.157 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.75158.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.75158.log.gz 2024-02-21T23:18:25.158 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45684.log 2024-02-21T23:18:25.158 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.61702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65734.log 2024-02-21T23:18:25.159 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.61702.log.gz 2024-02-21T23:18:25.160 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.45684.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.45684.log.gz 2024-02-21T23:18:25.160 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46947.log 2024-02-21T23:18:25.160 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.65734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88148.log 2024-02-21T23:18:25.161 INFO:teuthology.orchestra.run.smithi028.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.65734.log.gz 2024-02-21T23:18:25.162 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.46947.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46947.log.gz 2024-02-21T23:18:25.162 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70580.log 2024-02-21T23:18:25.162 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.88148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55854.log 2024-02-21T23:18:25.163 INFO:teuthology.orchestra.run.smithi028.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.88148.log.gz 2024-02-21T23:18:25.164 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.70580.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.70580.log.gz 2024-02-21T23:18:25.164 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51275.log 2024-02-21T23:18:25.165 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.55854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46332.log 2024-02-21T23:18:25.165 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.55854.log.gz 2024-02-21T23:18:25.166 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.51275.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51275.log.gz 2024-02-21T23:18:25.166 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53832.log 2024-02-21T23:18:25.167 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.46332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63062.log 2024-02-21T23:18:25.168 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46332.log.gz 2024-02-21T23:18:25.168 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.53832.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53832.log.gz 2024-02-21T23:18:25.169 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68455.log 2024-02-21T23:18:25.169 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.63062.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87587.log 2024-02-21T23:18:25.170 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.68455.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.63062.log.gz 2024-02-21T23:18:25.171 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56038.log 2024-02-21T23:18:25.171 INFO:teuthology.orchestra.run.smithi028.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.68455.log.gz 2024-02-21T23:18:25.171 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.87587.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43597.log 2024-02-21T23:18:25.172 INFO:teuthology.orchestra.run.smithi028.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.87587.log.gz 2024-02-21T23:18:25.173 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.56038.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.56038.log.gz 2024-02-21T23:18:25.173 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54126.log 2024-02-21T23:18:25.174 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.43597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59317.log 2024-02-21T23:18:25.174 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.54126.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43597.log.gz 2024-02-21T23:18:25.175 INFO:teuthology.orchestra.run.smithi028.stderr:gzip 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54126.log.gz -5 2024-02-21T23:18:25.175 INFO:teuthology.orchestra.run.smithi028.stderr: --verbose -- /var/log/ceph/ceph-client.admin.52300.log 2024-02-21T23:18:25.176 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.59317.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.59317.log.gz 2024-02-21T23:18:25.176 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80535.log 2024-02-21T23:18:25.177 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.52300.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-02-21T23:18:25.178 INFO:teuthology.orchestra.run.smithi028.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52300.log.gz 2024-02-21T23:18:25.179 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.80535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56897.log 2024-02-21T23:18:25.179 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.80535.log.gz 2024-02-21T23:18:25.180 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63790.log 2024-02-21T23:18:25.181 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.56897.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50299.log 2024-02-21T23:18:25.181 INFO:teuthology.orchestra.run.smithi028.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.56897.log.gz 2024-02-21T23:18:25.182 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.63790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44591.log 2024-02-21T23:18:25.182 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.63790.log.gz 2024-02-21T23:18:25.184 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.50299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85690.log 2024-02-21T23:18:25.184 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50299.log.gz 2024-02-21T23:18:25.185 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.44591.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83890.log 2024-02-21T23:18:25.185 INFO:teuthology.orchestra.run.smithi028.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.44591.log.gz 2024-02-21T23:18:25.186 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.85690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73636.log 2024-02-21T23:18:25.187 INFO:teuthology.orchestra.run.smithi028.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.85690.log.gz 2024-02-21T23:18:25.188 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.83890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73594.log 2024-02-21T23:18:25.188 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.83890.log.gz 2024-02-21T23:18:25.189 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.73636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77631.log 2024-02-21T23:18:25.189 INFO:teuthology.orchestra.run.smithi028.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.73636.log.gz 2024-02-21T23:18:25.190 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.73594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42588.log 2024-02-21T23:18:25.190 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.73594.log.gz 2024-02-21T23:18:25.191 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.77631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84874.log 2024-02-21T23:18:25.192 INFO:teuthology.orchestra.run.smithi028.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.77631.log.gz 2024-02-21T23:18:25.192 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.42588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72958.log 2024-02-21T23:18:25.193 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42588.log.gz 2024-02-21T23:18:25.194 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.84874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81297.log 2024-02-21T23:18:25.194 INFO:teuthology.orchestra.run.smithi028.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.84874.log.gz 2024-02-21T23:18:25.196 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.72958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83284.log 2024-02-21T23:18:25.196 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.72958.log.gz 2024-02-21T23:18:25.196 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.81297.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86002.log 2024-02-21T23:18:25.196 INFO:teuthology.orchestra.run.smithi028.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.81297.log.gz 2024-02-21T23:18:25.198 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.83284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82030.log 2024-02-21T23:18:25.198 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.83284.log.gz 2024-02-21T23:18:25.199 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.86002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72207.log 2024-02-21T23:18:25.199 INFO:teuthology.orchestra.run.smithi028.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.86002.log.gz 2024-02-21T23:18:25.200 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.82030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53622.log 2024-02-21T23:18:25.200 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.82030.log.gz 2024-02-21T23:18:25.201 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.72207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55015.log 2024-02-21T23:18:25.202 INFO:teuthology.orchestra.run.smithi028.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.72207.log.gz 2024-02-21T23:18:25.203 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.53622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41530.log 2024-02-21T23:18:25.203 INFO:teuthology.orchestra.run.smithi028.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53622.log.gz 2024-02-21T23:18:25.204 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.55015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64120.log 2024-02-21T23:18:25.204 INFO:teuthology.orchestra.run.smithi028.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.55015.log.gz 2024-02-21T23:18:25.205 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.41530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62537.log 2024-02-21T23:18:25.205 INFO:teuthology.orchestra.run.smithi028.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.41530.log.gz 2024-02-21T23:18:25.206 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.64120.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71772.log 2024-02-21T23:18:25.207 INFO:teuthology.orchestra.run.smithi028.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.64120.log.gz 2024-02-21T23:18:25.207 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.62537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61264.log 2024-02-21T23:18:25.208 INFO:teuthology.orchestra.run.smithi028.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.62537.log.gz 2024-02-21T23:18:25.209 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.71772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87359.log 2024-02-21T23:18:25.209 INFO:teuthology.orchestra.run.smithi028.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.71772.log.gz 2024-02-21T23:18:25.210 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.61264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58558.log 2024-02-21T23:18:25.210 INFO:teuthology.orchestra.run.smithi028.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.61264.log.gz 2024-02-21T23:18:25.211 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.87359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69721.log 2024-02-21T23:18:25.212 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.87359.log.gz 2024-02-21T23:18:25.213 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.58558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53579.log 2024-02-21T23:18:25.213 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.58558.log.gz 2024-02-21T23:18:25.214 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.69721.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86296.log 2024-02-21T23:18:25.214 INFO:teuthology.orchestra.run.smithi028.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.69721.log.gz 2024-02-21T23:18:25.215 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.53579.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86986.log 2024-02-21T23:18:25.215 INFO:teuthology.orchestra.run.smithi028.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.53579.log.gz 2024-02-21T23:18:25.216 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.86296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65776.log 2024-02-21T23:18:25.217 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.86296.log.gz 2024-02-21T23:18:25.218 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.86986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53435.log 2024-02-21T23:18:25.218 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.86986.log.gz 2024-02-21T23:18:25.219 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.65776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48914.log 2024-02-21T23:18:25.219 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.65776.log.gz 2024-02-21T23:18:25.220 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.53435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63104.log 2024-02-21T23:18:25.220 INFO:teuthology.orchestra.run.smithi028.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53435.log.gz 2024-02-21T23:18:25.222 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.48914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54480.log 2024-02-21T23:18:25.222 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48914.log.gz 2024-02-21T23:18:25.223 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.63104.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60333.log 2024-02-21T23:18:25.223 INFO:teuthology.orchestra.run.smithi028.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.63104.log.gz 2024-02-21T23:18:25.225 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.54480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51986.log 2024-02-21T23:18:25.225 INFO:teuthology.orchestra.run.smithi028.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.54480.log.gz 2024-02-21T23:18:25.225 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.60333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62334.log 2024-02-21T23:18:25.225 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.60333.log.gz 2024-02-21T23:18:25.227 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.51986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88316.log 2024-02-21T23:18:25.227 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51986.log.gz 2024-02-21T23:18:25.228 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.62334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50486.log 2024-02-21T23:18:25.228 INFO:teuthology.orchestra.run.smithi028.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.62334.log.gz 2024-02-21T23:18:25.229 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.88316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67589.log 2024-02-21T23:18:25.229 INFO:teuthology.orchestra.run.smithi028.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.88316.log.gz 2024-02-21T23:18:25.231 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.50486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56296.log 2024-02-21T23:18:25.231 INFO:teuthology.orchestra.run.smithi028.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.50486.log.gz 2024-02-21T23:18:25.232 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.67589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71856.log 2024-02-21T23:18:25.232 INFO:teuthology.orchestra.run.smithi028.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.67589.log.gz 2024-02-21T23:18:25.233 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.56296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49692.log 2024-02-21T23:18:25.233 INFO:teuthology.orchestra.run.smithi028.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.56296.log.gz 2024-02-21T23:18:25.234 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.71856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85480.log 2024-02-21T23:18:25.234 INFO:teuthology.orchestra.run.smithi028.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.71856.log.gz 2024-02-21T23:18:25.236 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.49692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49734.log 2024-02-21T23:18:25.236 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49692.log.gz 2024-02-21T23:18:25.237 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.85480.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-02-21T23:18:25.237 INFO:teuthology.orchestra.run.smithi028.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.85480.log.gz 2024-02-21T23:18:25.238 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.49734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48289.log 2024-02-21T23:18:25.239 INFO:teuthology.orchestra.run.smithi028.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.49734.log.gz 2024-02-21T23:18:25.239 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64460.log 2024-02-21T23:18:25.241 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.48289.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68497.log 2024-02-21T23:18:25.241 INFO:teuthology.orchestra.run.smithi028.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.48289.log.gz 2024-02-21T23:18:25.242 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.64460.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.64460.log.gz 2024-02-21T23:18:25.242 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88881.log 2024-02-21T23:18:25.243 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.68497.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.68497.log.gz 2024-02-21T23:18:25.244 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph.log.gz 2024-02-21T23:18:25.244 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42689.log 2024-02-21T23:18:25.245 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.88881.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52936.log 2024-02-21T23:18:25.245 INFO:teuthology.orchestra.run.smithi028.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.88881.log.gz 2024-02-21T23:18:25.246 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.42689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43588.log 2024-02-21T23:18:25.246 INFO:teuthology.orchestra.run.smithi028.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42689.log.gz 2024-02-21T23:18:25.248 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.52936.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86734.log 2024-02-21T23:18:25.248 INFO:teuthology.orchestra.run.smithi028.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52936.log.gz 2024-02-21T23:18:25.249 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.43588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53664.log 2024-02-21T23:18:25.249 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43588.log.gz 2024-02-21T23:18:25.250 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.86734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56957.log 2024-02-21T23:18:25.250 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.86734.log.gz 2024-02-21T23:18:25.251 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.53664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47793.log 2024-02-21T23:18:25.251 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53664.log.gz 2024-02-21T23:18:25.256 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.56957.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-02-21T23:18:25.256 INFO:teuthology.orchestra.run.smithi028.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.56957.log.gz 2024-02-21T23:18:25.256 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.47793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70041.log 2024-02-21T23:18:25.256 INFO:teuthology.orchestra.run.smithi028.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.47793.log.gz 2024-02-21T23:18:25.257 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65944.log 2024-02-21T23:18:25.257 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.70041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68413.log 2024-02-21T23:18:25.257 INFO:teuthology.orchestra.run.smithi028.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.70041.log.gz 2024-02-21T23:18:25.258 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.65944.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.65944.log.gz 2024-02-21T23:18:25.258 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43594.log 2024-02-21T23:18:25.259 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.68413.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.68413.log.gz 2024-02-21T23:18:25.260 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56561.log 2024-02-21T23:18:25.261 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.43594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41078.log 2024-02-21T23:18:25.261 INFO:teuthology.orchestra.run.smithi028.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43594.log.gz 2024-02-21T23:18:25.262 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.56561.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.56561.log.gz 2024-02-21T23:18:25.263 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5/var/log/ceph/ceph-client.admin.41078.log: --verbose -- /var/log/ceph/ceph-client.admin.47877.log 2024-02-21T23:18:25.263 INFO:teuthology.orchestra.run.smithi028.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41078.log.gz 2024-02-21T23:18:25.264 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80944.log 2024-02-21T23:18:25.266 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.47877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76196.log 2024-02-21T23:18:25.266 INFO:teuthology.orchestra.run.smithi028.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.47877.log.gz 2024-02-21T23:18:25.267 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.80944.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.80944.log.gz 2024-02-21T23:18:25.268 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79178.log 2024-02-21T23:18:25.268 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.76196.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.76196.log.gz 2024-02-21T23:18:25.269 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69368.log 2024-02-21T23:18:25.270 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.79178.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-02-21T23:18:25.270 INFO:teuthology.orchestra.run.smithi028.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.79178.log.gz 2024-02-21T23:18:25.272 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.69368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46542.log 2024-02-21T23:18:25.272 INFO:teuthology.orchestra.run.smithi028.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.69368.log.gz 2024-02-21T23:18:25.274 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-mds.c.log: /var/log/ceph/ceph-client.admin.46542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60207.log 2024-02-21T23:18:25.274 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46542.log.gz 2024-02-21T23:18:25.275 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47529.log 2024-02-21T23:18:25.277 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.60207.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.60207.log.gz 2024-02-21T23:18:25.278 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82636.log 2024-02-21T23:18:25.278 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.47529.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.47529.log.gz 2024-02-21T23:18:25.279 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74505.log 2024-02-21T23:18:25.280 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.82636.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.82636.log.gz 2024-02-21T23:18:25.282 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88921.log/var/log/ceph/ceph-client.admin.74505.log: 2024-02-21T23:18:25.282 INFO:teuthology.orchestra.run.smithi028.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.74505.log.gz 2024-02-21T23:18:25.283 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40836.log 2024-02-21T23:18:25.284 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.88921.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.88921.log.gz 2024-02-21T23:18:25.285 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44633.log 2024-02-21T23:18:25.286 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.40836.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.40836.log.gz 2024-02-21T23:18:25.287 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44759.log 2024-02-21T23:18:25.288 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.44633.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44633.log.gz 2024-02-21T23:18:25.288 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54522.log 2024-02-21T23:18:25.289 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.44759.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44759.log.gz 2024-02-21T23:18:25.290 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37354.log 2024-02-21T23:18:25.291 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.54522.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54522.log.gz 2024-02-21T23:18:25.292 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76364.log 2024-02-21T23:18:25.293 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.37354.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.37354.log.gz 2024-02-21T23:18:25.293 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55154.log 2024-02-21T23:18:25.296 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.76364.log: 89.2%gzip -5 --verbose --/var/log/ceph/ceph-client.admin.55154.log: /var/log/ceph/ceph-client.admin.79094.log 2024-02-21T23:18:25.296 INFO:teuthology.orchestra.run.smithi028.stderr: -- replaced with /var/log/ceph/ceph-client.admin.76364.log.gz 2024-02-21T23:18:25.296 INFO:teuthology.orchestra.run.smithi028.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.55154.log.gz 2024-02-21T23:18:25.298 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71562.log 2024-02-21T23:18:25.299 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.79094.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.79094.log.gz 2024-02-21T23:18:25.300 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47115.log 2024-02-21T23:18:25.301 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.71562.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.71562.log.gz 2024-02-21T23:18:25.301 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45768.log 2024-02-21T23:18:25.303 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.47115.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.47115.log.gz 2024-02-21T23:18:25.304 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.45768.log: /var/log/ceph/ceph-client.admin.72393.log 2024-02-21T23:18:25.304 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45768.log.gz 2024-02-21T23:18:25.305 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80283.log 2024-02-21T23:18:25.307 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.72393.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.72393.log.gz 2024-02-21T23:18:25.308 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.80283.log: /var/log/ceph/ceph-client.admin.65132.log 2024-02-21T23:18:25.308 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.80283.log.gz 2024-02-21T23:18:25.309 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56124.log 2024-02-21T23:18:25.310 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.65132.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.65132.log.gz 2024-02-21T23:18:25.311 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73888.log 2024-02-21T23:18:25.312 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.56124.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.56124.log.gz 2024-02-21T23:18:25.313 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76238.log 2024-02-21T23:18:25.314 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.73888.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.73888.log.gz 2024-02-21T23:18:25.315 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76640.log 2024-02-21T23:18:25.316 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.76238.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.76238.log.gz 2024-02-21T23:18:25.317 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.76640.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.76640.log.gz 2024-02-21T23:18:25.318 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43927.log 2024-02-21T23:18:25.319 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85042.log 2024-02-21T23:18:25.322 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.43927.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43927.log.gz 2024-02-21T23:18:25.322 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.85042.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87401.log 2024-02-21T23:18:25.322 INFO:teuthology.orchestra.run.smithi028.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.85042.log.gz 2024-02-21T23:18:25.323 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37247.log 2024-02-21T23:18:25.325 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.87401.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.87401.log.gz 2024-02-21T23:18:25.326 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63188.log/var/log/ceph/ceph-client.admin.37247.log: 2024-02-21T23:18:25.326 INFO:teuthology.orchestra.run.smithi028.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37247.log.gz 2024-02-21T23:18:25.327 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46290.log 2024-02-21T23:18:25.328 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.63188.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.63188.log.gz 2024-02-21T23:18:25.330 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.46290.log: /var/log/ceph/ceph-client.admin.47073.log 2024-02-21T23:18:25.330 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46290.log.gz 2024-02-21T23:18:25.331 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69199.log 2024-02-21T23:18:25.332 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.47073.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47073.log.gz 2024-02-21T23:18:25.333 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65650.log 2024-02-21T23:18:25.334 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.69199.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.69199.log.gz 2024-02-21T23:18:25.335 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60062.log 2024-02-21T23:18:25.336 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.65650.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.65650.log.gz 2024-02-21T23:18:25.337 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78842.log 2024-02-21T23:18:25.338 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.60062.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.60062.log.gz 2024-02-21T23:18:25.339 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.78842.log: /var/log/ceph/ceph-client.admin.76532.log 2024-02-21T23:18:25.340 INFO:teuthology.orchestra.run.smithi028.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.78842.log.gz 2024-02-21T23:18:25.341 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77757.log 2024-02-21T23:18:25.342 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.76532.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.76532.log.gz 2024-02-21T23:18:25.343 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.77757.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73996.log 2024-02-21T23:18:25.343 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.77757.log.gz 2024-02-21T23:18:25.344 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81381.log 2024-02-21T23:18:25.346 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.73996.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.73996.log.gz 2024-02-21T23:18:25.347 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5/var/log/ceph/ceph-client.admin.81381.log: --verbose -- /var/log/ceph/ceph-client.admin.85606.log 2024-02-21T23:18:25.347 INFO:teuthology.orchestra.run.smithi028.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.81381.log.gz 2024-02-21T23:18:25.348 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69284.log 2024-02-21T23:18:25.350 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.85606.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.85606.log.gz 2024-02-21T23:18:25.351 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57110.log 2024-02-21T23:18:25.351 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.69284.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.69284.log.gz 2024-02-21T23:18:25.352 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71394.log 2024-02-21T23:18:25.354 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.57110.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.57110.log.gz 2024-02-21T23:18:25.355 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49650.log 2024-02-21T23:18:25.355 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.71394.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.71394.log.gz 2024-02-21T23:18:25.356 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61954.log 2024-02-21T23:18:25.358 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.49650.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49650.log.gz 2024-02-21T23:18:25.358 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.61954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56389.log 2024-02-21T23:18:25.360 INFO:teuthology.orchestra.run.smithi028.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.61954.log.gz 2024-02-21T23:18:25.360 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53393.log 2024-02-21T23:18:25.362 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.56389.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.56389.log.gz 2024-02-21T23:18:25.362 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36986.log 2024-02-21T23:18:25.363 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.53393.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.53393.log.gz 2024-02-21T23:18:25.364 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52855.log 2024-02-21T23:18:25.365 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.36986.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.36986.log.gz 2024-02-21T23:18:25.366 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.52855.log: -- /var/log/ceph/ceph-client.admin.47571.log 2024-02-21T23:18:25.367 INFO:teuthology.orchestra.run.smithi028.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52855.log.gz 2024-02-21T23:18:25.367 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85084.log 2024-02-21T23:18:25.369 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.47571.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.47571.log.gz 2024-02-21T23:18:25.370 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.85084.log: /var/log/ceph/ceph-client.admin.77060.log 2024-02-21T23:18:25.370 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.85084.log.gz 2024-02-21T23:18:25.371 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47319.log 2024-02-21T23:18:25.373 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.77060.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.77060.log.gz 2024-02-21T23:18:25.374 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83788.log 2024-02-21T23:18:25.375 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.47319.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47319.log.gz 2024-02-21T23:18:25.376 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70790.log 2024-02-21T23:18:25.377 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.83788.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.83788.log.gz 2024-02-21T23:18:25.378 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52344.log/var/log/ceph/ceph-client.admin.70790.log: 2024-02-21T23:18:25.378 INFO:teuthology.orchestra.run.smithi028.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.70790.log.gz 2024-02-21T23:18:25.379 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47031.log 2024-02-21T23:18:25.381 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.52344.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52344.log.gz 2024-02-21T23:18:25.381 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41276.log 2024-02-21T23:18:25.382 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.47031.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.47031.log.gz 2024-02-21T23:18:25.383 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82114.log 2024-02-21T23:18:25.384 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.41276.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41276.log.gz 2024-02-21T23:18:25.385 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65692.log/var/log/ceph/ceph-client.admin.82114.log: 2024-02-21T23:18:25.386 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.82114.log.gz 2024-02-21T23:18:25.387 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47445.log 2024-02-21T23:18:25.388 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.65692.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.65692.log.gz 2024-02-21T23:18:25.389 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51542.log/var/log/ceph/ceph-client.admin.47445.log: 2024-02-21T23:18:25.390 INFO:teuthology.orchestra.run.smithi028.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.47445.log.gz 2024-02-21T23:18:25.391 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56347.log 2024-02-21T23:18:25.393 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.51542.log: 91.1% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-02-21T23:18:25.393 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51542.log.gz 2024-02-21T23:18:25.393 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73411.log 2024-02-21T23:18:25.393 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.56347.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.56347.log.gz 2024-02-21T23:18:25.394 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58096.log 2024-02-21T23:18:25.395 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.73411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58805.log 2024-02-21T23:18:25.396 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.73411.log.gz 2024-02-21T23:18:25.397 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.58096.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.58096.log.gz 2024-02-21T23:18:25.397 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48543.log 2024-02-21T23:18:25.398 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.58805.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.58805.log.gz 2024-02-21T23:18:25.399 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84100.log 2024-02-21T23:18:25.400 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.48543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.45453.log 2024-02-21T23:18:25.400 INFO:teuthology.orchestra.run.smithi028.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.48543.log.gz 2024-02-21T23:18:25.402 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.84100.log: gzip 87.7% -- replaced with /var/log/ceph/ceph-client.admin.84100.log.gz 2024-02-21T23:18:25.402 INFO:teuthology.orchestra.run.smithi028.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.84394.log 2024-02-21T23:18:25.402 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.0.45453.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.45453.log.gz 2024-02-21T23:18:25.403 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64970.log 2024-02-21T23:18:25.404 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.84394.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73804.log 2024-02-21T23:18:25.405 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.84394.log.gz 2024-02-21T23:18:25.406 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.64970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83830.log 2024-02-21T23:18:25.406 INFO:teuthology.orchestra.run.smithi028.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.64970.log.gz 2024-02-21T23:18:25.407 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.73804.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.73804.log.gz 2024-02-21T23:18:25.407 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85648.log 2024-02-21T23:18:25.409 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.83830.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74122.log 2024-02-21T23:18:25.409 INFO:teuthology.orchestra.run.smithi028.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.83830.log.gz 2024-02-21T23:18:25.410 INFO:teuthology.orchestra.run.smithi028.stderr: 90.3% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-02-21T23:18:25.410 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.85648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55980.log 2024-02-21T23:18:25.410 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.85648.log.gz 2024-02-21T23:18:25.411 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.74122.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85312.log 2024-02-21T23:18:25.411 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.55980.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.74122.log.gz 2024-02-21T23:18:25.412 INFO:teuthology.orchestra.run.smithi028.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.55980.log.gz 2024-02-21T23:18:25.412 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82762.log 2024-02-21T23:18:25.413 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.85312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48375.log 2024-02-21T23:18:25.414 INFO:teuthology.orchestra.run.smithi028.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.85312.log.gz 2024-02-21T23:18:25.414 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.82762.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.82762.log.gz 2024-02-21T23:18:25.415 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48868.log 2024-02-21T23:18:25.416 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.48375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73325.log 2024-02-21T23:18:25.416 INFO:teuthology.orchestra.run.smithi028.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48375.log.gz 2024-02-21T23:18:25.417 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.48868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79472.log 88.6% 2024-02-21T23:18:25.417 INFO:teuthology.orchestra.run.smithi028.stderr: -- replaced with /var/log/ceph/ceph-client.admin.48868.log.gz 2024-02-21T23:18:25.418 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.73325.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74551.log 2024-02-21T23:18:25.418 INFO:teuthology.orchestra.run.smithi028.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.73325.log.gz 2024-02-21T23:18:25.419 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.79472.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43929.log 2024-02-21T23:18:25.419 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.79472.log.gz 2024-02-21T23:18:25.420 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.74551.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.74551.log.gz 2024-02-21T23:18:25.421 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44843.log 2024-02-21T23:18:25.422 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.43929.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61183.log 2024-02-21T23:18:25.422 INFO:teuthology.orchestra.run.smithi028.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43929.log.gz 2024-02-21T23:18:25.423 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.44843.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44843.log.gz 2024-02-21T23:18:25.423 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67697.log 2024-02-21T23:18:25.424 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.61183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74948.log 2024-02-21T23:18:25.425 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.61183.log.gz 2024-02-21T23:18:25.426 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.67697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46626.log 2024-02-21T23:18:25.426 INFO:teuthology.orchestra.run.smithi028.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.67697.log.gz 2024-02-21T23:18:25.426 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.74948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59093.log 2024-02-21T23:18:25.427 INFO:teuthology.orchestra.run.smithi028.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.74948.log.gz 2024-02-21T23:18:25.428 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.46626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88022.log 2024-02-21T23:18:25.428 INFO:teuthology.orchestra.run.smithi028.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.46626.log.gz 2024-02-21T23:18:25.428 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.59093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77883.log 2024-02-21T23:18:25.429 INFO:teuthology.orchestra.run.smithi028.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.59093.log.gz 2024-02-21T23:18:25.430 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.88022.log: gzip 88.9% -5 --verbose -- /var/log/ceph/ceph-client.admin.86776.log 2024-02-21T23:18:25.430 INFO:teuthology.orchestra.run.smithi028.stderr: -- replaced with /var/log/ceph/ceph-client.admin.88022.log.gz 2024-02-21T23:18:25.431 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.77883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52722.log 2024-02-21T23:18:25.431 INFO:teuthology.orchestra.run.smithi028.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.77883.log.gz 2024-02-21T23:18:25.432 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.86776.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.86776.log.gz 2024-02-21T23:18:25.432 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42899.log 2024-02-21T23:18:25.433 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.52722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55196.log 2024-02-21T23:18:25.434 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.42899.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52722.log.gz 2024-02-21T23:18:25.435 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42899.log.gz 2024-02-21T23:18:25.435 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75303.log 2024-02-21T23:18:25.435 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.55196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73762.log 2024-02-21T23:18:25.436 INFO:teuthology.orchestra.run.smithi028.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.55196.log.gz 2024-02-21T23:18:25.437 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.75303.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72249.log 2024-02-21T23:18:25.437 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.75303.log.gz 2024-02-21T23:18:25.438 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.73762.log: 88.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67102.log 2024-02-21T23:18:25.438 INFO:teuthology.orchestra.run.smithi028.stderr: -- replaced with /var/log/ceph/ceph-client.admin.73762.log.gz 2024-02-21T23:18:25.439 INFO:teuthology.orchestra.run.smithi028.stderr:gzip/var/log/ceph/ceph-client.admin.72249.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.62453.log 2024-02-21T23:18:25.439 INFO:teuthology.orchestra.run.smithi028.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.72249.log.gz 2024-02-21T23:18:25.440 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.67102.log: gzip -5 --verbose 86.9% -- replaced with /var/log/ceph/ceph-client.admin.67102.log.gz 2024-02-21T23:18:25.441 INFO:teuthology.orchestra.run.smithi028.stderr: -- /var/log/ceph/ceph-client.admin.68791.log 2024-02-21T23:18:25.441 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.62453.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.62453.log.gz 2024-02-21T23:18:25.442 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86506.log 2024-02-21T23:18:25.443 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.68791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64418.log 2024-02-21T23:18:25.443 INFO:teuthology.orchestra.run.smithi028.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.68791.log.gz 2024-02-21T23:18:25.444 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.86506.log: 88.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59008.log 2024-02-21T23:18:25.444 INFO:teuthology.orchestra.run.smithi028.stderr: -- replaced with /var/log/ceph/ceph-client.admin.86506.log.gz 2024-02-21T23:18:25.445 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.64418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72291.log 2024-02-21T23:18:25.446 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.64418.log.gz 2024-02-21T23:18:25.446 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.59008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66874.log 2024-02-21T23:18:25.447 INFO:teuthology.orchestra.run.smithi028.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.59008.log.gz 2024-02-21T23:18:25.448 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.72291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50383.log 2024-02-21T23:18:25.448 INFO:teuthology.orchestra.run.smithi028.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.72291.log.gz 2024-02-21T23:18:25.449 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.66874.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.66874.log.gzgzip 2024-02-21T23:18:25.449 INFO:teuthology.orchestra.run.smithi028.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.63272.log 2024-02-21T23:18:25.450 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.50383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85564.log 2024-02-21T23:18:25.450 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50383.log.gz 2024-02-21T23:18:25.451 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.63272.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.63272.log.gz 2024-02-21T23:18:25.452 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58138.log 2024-02-21T23:18:25.453 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.85564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62936.log 2024-02-21T23:18:25.453 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.85564.log.gz 2024-02-21T23:18:25.454 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.58138.log: 88.9%gzip -- replaced with /var/log/ceph/ceph-client.admin.58138.log.gz 2024-02-21T23:18:25.454 INFO:teuthology.orchestra.run.smithi028.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.70167.log 2024-02-21T23:18:25.455 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.62936.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61572.log 2024-02-21T23:18:25.456 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.62936.log.gz 2024-02-21T23:18:25.461 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.70167.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76724.log 2024-02-21T23:18:25.461 INFO:teuthology.orchestra.run.smithi028.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.70167.log.gz 2024-02-21T23:18:25.462 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.61572.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.61572.log.gz 2024-02-21T23:18:25.462 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84664.log 2024-02-21T23:18:25.462 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.76724.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-02-21T23:18:25.462 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.76724.log.gz 2024-02-21T23:18:25.462 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.84664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44161.log 2024-02-21T23:18:25.462 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.84664.log.gz 2024-02-21T23:18:25.462 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60730.log 2024-02-21T23:18:25.463 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.44161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78382.log 2024-02-21T23:18:25.463 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44161.log.gz 2024-02-21T23:18:25.464 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.60730.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.60730.log.gz 2024-02-21T23:18:25.464 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70706.log 2024-02-21T23:18:25.465 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.78382.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.78382.log.gz 2024-02-21T23:18:25.465 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73367.log 2024-02-21T23:18:25.467 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.70706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47835.log 2024-02-21T23:18:25.467 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.70706.log.gz 2024-02-21T23:18:25.469 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.73367.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.73367.log.gz 2024-02-21T23:18:25.469 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62495.log 2024-02-21T23:18:25.470 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.47835.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61912.log 2024-02-21T23:18:25.471 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47835.log.gz/var/log/ceph/ceph-client.admin.62495.log: 2024-02-21T23:18:25.471 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54765.log 87.5% -- replaced with /var/log/ceph/ceph-client.admin.62495.log.gz 2024-02-21T23:18:25.471 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-02-21T23:18:25.472 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.61912.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.61912.log.gz 2024-02-21T23:18:25.473 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40919.log 2024-02-21T23:18:25.474 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.54765.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59367.log 2024-02-21T23:18:25.474 INFO:teuthology.orchestra.run.smithi028.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54765.log.gz 2024-02-21T23:18:25.476 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.40919.log: gzip 88.6% -5 -- replaced with /var/log/ceph/ceph-client.admin.40919.log.gz --verbose 2024-02-21T23:18:25.476 INFO:teuthology.orchestra.run.smithi028.stderr: -- /var/log/ceph/ceph-client.admin.84790.log 2024-02-21T23:18:25.477 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.59367.log: 88.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52793.log 2024-02-21T23:18:25.477 INFO:teuthology.orchestra.run.smithi028.stderr: -- replaced with /var/log/ceph/ceph-client.admin.59367.log.gz 2024-02-21T23:18:25.478 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.84790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82594.log 2024-02-21T23:18:25.478 INFO:teuthology.orchestra.run.smithi028.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.84790.log.gz 2024-02-21T23:18:25.480 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.52793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69932.log 2024-02-21T23:18:25.480 INFO:teuthology.orchestra.run.smithi028.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.52793.log.gz 2024-02-21T23:18:25.481 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.82594.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.82594.log.gz 2024-02-21T23:18:25.481 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81255.log 2024-02-21T23:18:25.483 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.69932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74463.log 2024-02-21T23:18:25.483 INFO:teuthology.orchestra.run.smithi028.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.69932.log.gz 2024-02-21T23:18:25.485 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.81255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88106.log 2024-02-21T23:18:25.485 INFO:teuthology.orchestra.run.smithi028.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.81255.log.gz 2024-02-21T23:18:25.485 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.74463.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.74463.log.gz 2024-02-21T23:18:25.486 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64586.log 2024-02-21T23:18:25.487 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.88106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63398.log 2024-02-21T23:18:25.487 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.88106.log.gz 2024-02-21T23:18:25.488 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.64586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60688.log 2024-02-21T23:18:25.488 INFO:teuthology.orchestra.run.smithi028.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.64586.log.gz 2024-02-21T23:18:25.489 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.63398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49040.log 2024-02-21T23:18:25.490 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.63398.log.gz 2024-02-21T23:18:25.491 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.60688.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.60688.log.gz 2024-02-21T23:18:25.492 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.49040.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49040.log.gz 2024-02-21T23:18:25.536 INFO:teuthology.orchestra.run.smithi028.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-02-21T23:18:25.780 INFO:teuthology.orchestra.run.smithi081.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-02-21T23:18:25.860 INFO:teuthology.orchestra.run.smithi081.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-02-21T23:18:25.864 INFO:teuthology.orchestra.run.smithi081.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-02-21T23:18:25.896 INFO:teuthology.orchestra.run.smithi081.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-02-21T23:18:25.900 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-02-21T23:18:25.900 INFO:teuthology.orchestra.run.smithi081.stderr:real 0m1.804s 2024-02-21T23:18:25.900 INFO:teuthology.orchestra.run.smithi081.stderr:user 0m7.543s 2024-02-21T23:18:25.900 INFO:teuthology.orchestra.run.smithi081.stderr:sys 0m0.683s 2024-02-21T23:18:26.511 INFO:teuthology.orchestra.run.smithi028.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-02-21T23:18:26.798 INFO:teuthology.orchestra.run.smithi028.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-02-21T23:18:26.862 INFO:teuthology.orchestra.run.smithi028.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-02-21T23:18:26.938 INFO:teuthology.orchestra.run.smithi028.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-02-21T23:18:26.941 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-02-21T23:18:26.942 INFO:teuthology.orchestra.run.smithi028.stderr:real 0m2.843s 2024-02-21T23:18:26.942 INFO:teuthology.orchestra.run.smithi028.stderr:user 0m8.275s 2024-02-21T23:18:26.942 INFO:teuthology.orchestra.run.smithi028.stderr:sys 0m3.457s 2024-02-21T23:18:26.943 INFO:tasks.ceph:Archiving logs... 2024-02-21T23:18:26.943 DEBUG:teuthology.misc:Transferring archived files from smithi028:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-02-21_22:31:03-fs-wip-lusov-qdb-distro-default-smithi/7570010/remote/smithi028/log 2024-02-21T23:18:26.944 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-21T23:18:28.995 DEBUG:teuthology.misc:Transferring archived files from smithi081:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-02-21_22:31:03-fs-wip-lusov-qdb-distro-default-smithi/7570010/remote/smithi081/log 2024-02-21T23:18:28.996 DEBUG:teuthology.orchestra.run.smithi081:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-21T23:18:30.206 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-02-21T23:18:30.216 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-02-21T23:18:30.217 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-21T23:18:30.225 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-21T23:18:30.381 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-02-21T23:18:30.390 INFO:teuthology.task.clock:Checking final clock skew... 2024-02-21T23:18:30.390 DEBUG:teuthology.orchestra.run.smithi028:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-21T23:18:30.398 DEBUG:teuthology.orchestra.run.smithi081:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-21T23:18:30.447 INFO:teuthology.orchestra.run.smithi028.stderr:bash: line 1: ntpq: command not found 2024-02-21T23:18:30.452 INFO:teuthology.orchestra.run.smithi081.stderr:bash: line 1: ntpq: command not found 2024-02-21T23:18:30.456 INFO:teuthology.orchestra.run.smithi028.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-21T23:18:30.457 INFO:teuthology.orchestra.run.smithi028.stdout:=============================================================================== 2024-02-21T23:18:30.457 INFO:teuthology.orchestra.run.smithi028.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 100 +482us[ +481us] +/- 103ms 2024-02-21T23:18:30.457 INFO:teuthology.orchestra.run.smithi028.stdout:^* hv02.front.sepia.ceph.com 3 7 377 99 -34us[ -36us] +/- 53ms 2024-02-21T23:18:30.457 INFO:teuthology.orchestra.run.smithi028.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 35 -389us[ -389us] +/- 83ms 2024-02-21T23:18:30.457 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-02-21T23:18:30.461 INFO:teuthology.orchestra.run.smithi081.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-21T23:18:30.462 INFO:teuthology.orchestra.run.smithi081.stdout:=============================================================================== 2024-02-21T23:18:30.462 INFO:teuthology.orchestra.run.smithi081.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 35 +518us[ +518us] +/- 104ms 2024-02-21T23:18:30.462 INFO:teuthology.orchestra.run.smithi081.stdout:^* hv02.front.sepia.ceph.com 3 6 377 35 -3ns[+2352ns] +/- 54ms 2024-02-21T23:18:30.462 INFO:teuthology.orchestra.run.smithi081.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 36 -362us[ -360us] +/- 83ms 2024-02-21T23:18:30.462 INFO:teuthology.orchestra.run.smithi081.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-02-21T23:18:30.463 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-02-21T23:18:30.473 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-02-21T23:18:30.473 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-02-21T23:18:30.486 DEBUG:teuthology.orchestra.run.smithi028:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-21T23:18:30.756 DEBUG:teuthology.orchestra.run.smithi081:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-21T23:18:30.962 DEBUG:teuthology.orchestra.run.smithi028:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-21T23:18:31.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-21T23:18:31.074 DEBUG:teuthology.orchestra.run.smithi081:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-21T23:18:31.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-21T23:18:31.177 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-02-21T23:18:31.188 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-02-21T23:18:31.247 INFO:teuthology.task.internal:Duration was 1560.072766 seconds 2024-02-21T23:18:31.248 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-02-21T23:18:31.268 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-02-21T23:18:31.269 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-21T23:18:31.278 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-21T23:18:31.441 INFO:teuthology.orchestra.run.smithi081.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-21T23:18:31.454 INFO:teuthology.orchestra.run.smithi028.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-21T23:18:31.911 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-02-21T23:18:31.912 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi028.front.sepia.ceph.com 2024-02-21T23:18:31.912 DEBUG:teuthology.orchestra.run.smithi028:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-21T23:18:31.982 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi081.front.sepia.ceph.com 2024-02-21T23:18:31.982 DEBUG:teuthology.orchestra.run.smithi081:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-21T23:18:32.057 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-02-21T23:18:32.058 DEBUG:teuthology.orchestra.run.smithi028:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-21T23:18:32.066 DEBUG:teuthology.orchestra.run.smithi081:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-21T23:18:32.164 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-02-21T23:18:32.165 DEBUG:teuthology.orchestra.run.smithi028:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-21T23:18:32.170 DEBUG:teuthology.orchestra.run.smithi081:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-21T23:18:32.396 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-02-21T23:18:32.408 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-02-21T23:18:32.408 DEBUG:teuthology.orchestra.run.smithi028:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-21T23:18:32.507 DEBUG:teuthology.orchestra.run.smithi081:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-21T23:18:32.609 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-02-21T23:18:32.619 DEBUG:teuthology.orchestra.run.smithi028:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-21T23:18:32.628 DEBUG:teuthology.orchestra.run.smithi081:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-21T23:18:32.738 INFO:teuthology.orchestra.run.smithi081.stdout:kernel.core_pattern = core 2024-02-21T23:18:32.738 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern = core 2024-02-21T23:18:32.793 DEBUG:teuthology.orchestra.run.smithi028:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-21T23:18:32.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-21T23:18:32.845 DEBUG:teuthology.orchestra.run.smithi081:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-21T23:18:32.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-21T23:18:32.899 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-02-21T23:18:32.912 INFO:teuthology.task.internal:Transferring archived files... 2024-02-21T23:18:32.912 DEBUG:teuthology.misc:Transferring archived files from smithi028:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-02-21_22:31:03-fs-wip-lusov-qdb-distro-default-smithi/7570010/remote/smithi028 2024-02-21T23:18:32.913 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-21T23:18:33.024 DEBUG:teuthology.misc:Transferring archived files from smithi081:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-02-21_22:31:03-fs-wip-lusov-qdb-distro-default-smithi/7570010/remote/smithi081 2024-02-21T23:18:33.024 DEBUG:teuthology.orchestra.run.smithi081:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-21T23:18:33.131 INFO:teuthology.task.internal:Removing archive directory... 2024-02-21T23:18:33.132 DEBUG:teuthology.orchestra.run.smithi028:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-21T23:18:33.140 DEBUG:teuthology.orchestra.run.smithi081:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-21T23:18:33.197 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-02-21T23:18:33.209 INFO:teuthology.task.internal:Not uploading archives. 2024-02-21T23:18:33.209 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-02-21T23:18:33.224 INFO:teuthology.task.internal:Tidying up after the test... 2024-02-21T23:18:33.225 DEBUG:teuthology.orchestra.run.smithi028:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-21T23:18:33.233 DEBUG:teuthology.orchestra.run.smithi081:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-21T23:18:33.283 INFO:teuthology.orchestra.run.smithi028.stdout: 395173 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 21 23:18 /home/ubuntu/cephtest 2024-02-21T23:18:33.289 INFO:teuthology.orchestra.run.smithi081.stdout: 395173 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 21 23:18 /home/ubuntu/cephtest 2024-02-21T23:18:33.292 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-02-21T23:18:33.303 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-02-21T23:18:33.331 INFO:teuthology.nuke:Checking targets against current locks 2024-02-21T23:18:33.359 DEBUG:teuthology.nuke:shortname: smithi028 2024-02-21T23:18:33.360 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-21T23:18:33.384 DEBUG:teuthology.nuke:shortname: smithi081 2024-02-21T23:18:33.384 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-21T23:18:33.388 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi028.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-02-21_22:31:03-fs-wip-lusov-qdb-distro-default-smithi/7570010', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-21 22:40:53.153709', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCUSNVaQl5WeiAT0gHEsvl0ygugXPVSClauMigugq2UJ+dAgMEqmIvsZoNWBzBashC9pE97t51XpgAkk2r59kVg='} 2024-02-21T23:18:33.413 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi081.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-02-21_22:31:03-fs-wip-lusov-qdb-distro-default-smithi/7570010', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-21 22:40:53.151886', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMCOnqfcgZ62VPscyDatzXpS3IE3CO1tSuvz7xPZ2DxeiVsJvN6mJONmlC48XG7K18AySCjdQ+ADLKEeyfcr+TU='} 2024-02-21T23:18:33.446 INFO:teuthology.orchestra.console.smithi028:Power off 2024-02-21T23:18:33.447 DEBUG:teuthology.orchestra.console.smithi028:pexpect command: ipmitool -H smithi028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-21T23:18:33.474 DEBUG:teuthology.orchestra.console.smithi028:power off output: Chassis Power Control: Down/Off 2024-02-21T23:18:33.474 DEBUG:teuthology.orchestra.console.smithi028:pexpect command: ipmitool -H smithi028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-21T23:18:33.490 INFO:teuthology.orchestra.console.smithi081:Power off 2024-02-21T23:18:33.491 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-21T23:18:33.501 DEBUG:teuthology.orchestra.console.smithi028:check power output: Chassis Power is on 2024-02-21T23:18:33.513 DEBUG:teuthology.orchestra.console.smithi081:power off output: Chassis Power Control: Down/Off 2024-02-21T23:18:33.513 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-21T23:18:33.534 DEBUG:teuthology.orchestra.console.smithi081:check power output: Chassis Power is on 2024-02-21T23:18:37.502 DEBUG:teuthology.orchestra.console.smithi028:pexpect command: ipmitool -H smithi028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-21T23:18:37.535 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-21T23:18:37.613 DEBUG:teuthology.orchestra.console.smithi028:check power output: Chassis Power is on 2024-02-21T23:18:37.650 DEBUG:teuthology.orchestra.console.smithi081:check power output: Chassis Power is on 2024-02-21T23:18:41.614 DEBUG:teuthology.orchestra.console.smithi028:pexpect command: ipmitool -H smithi028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-21T23:18:41.651 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-21T23:18:41.727 DEBUG:teuthology.orchestra.console.smithi028:check power output: Chassis Power is off 2024-02-21T23:18:41.773 DEBUG:teuthology.orchestra.console.smithi081:check power output: Chassis Power is on 2024-02-21T23:18:41.873 INFO:teuthology.orchestra.console.smithi028:Power off completed 2024-02-21T23:18:45.776 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-21T23:18:45.887 DEBUG:teuthology.orchestra.console.smithi081:check power output: Chassis Power is off 2024-02-21T23:18:45.988 INFO:teuthology.orchestra.console.smithi081:Power off completed 2024-02-21T23:18:46.097 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 1130e053228021b1be9cac260b5ed52c5a9027c5 client.1-kernel-sha1: 1130e053228021b1be9cac260b5ed52c5a9027c5 description: fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} duration: 1560.072766304016 failure_reason: 'Test failure: test_down_twice (tasks.cephfs.test_failover.TestClusterResize)' flavor: default owner: scheduled_leonidus@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=e380bb7cc8b84d30afe72a3769c040e0 status: fail success: false 2024-02-21T23:18:46.098 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-21T23:18:46.186 INFO:teuthology.run:FAIL